diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 00000000..7c32d5f7 --- /dev/null +++ b/.gitattributes @@ -0,0 +1 @@ +*.jar filter=lfs diff=lfs merge=lfs -text diff --git a/LICENSE b/LICENSE deleted file mode 100644 index 7bf9011a..00000000 --- a/LICENSE +++ /dev/null @@ -1,201 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright 2017, The TensorFlow Authors. - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. diff --git a/PUBLICATIONS.md b/PUBLICATIONS.md deleted file mode 100644 index f8ac7eed..00000000 --- a/PUBLICATIONS.md +++ /dev/null @@ -1,1114 +0,0 @@ -# List of publications using Lingvo. - - - -## Translation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-[1] - -Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, - Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, - L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, - N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, - G. Corrado, M. Hughes, and J. Dean, “Google's neural machine translation - system: Bridging the gap between human and machine translation,” tech. rep., - 2016. -[ pdf ] - -
-[2] - -M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorat, - F. Viégas, M. Wattenberg, G. Corrado, M. Hughes, and J. Dean, - “Google's multilingual neural machine translation system: Enabling - zero-shot translation,” Transactions of the Association for - Computational Linguistics, vol. 5, pp. 339--351, 2017. -[ DOI | -pdf ] - -
-[3] - -A. Eriguchi, M. Johnson, O. Firat, H. Kazawa, and W. Macherey, “Zero-shot - cross-lingual classification using multilingual neural machine translation,” - arXiv preprint arXiv:1809.04686, 2018. -[ pdf ] - -
-[4] - -A. Bapna, M. X. Chen, O. Firat, Y. Cao, and Y. Wu, “Training deeper neural - machine translation models with transparent attention,” in Proc. - Conference on Empirical Methods in Natural Language Processing (EMNLP), - 2018. -[ pdf ] - -
-[5] - -C. Cherry, G. Foster, A. Bapna, O. Firat, and W. Macherey, “Revisiting - character-based neural machine translation with capacity and compression,” - in Proc. Conference on Empirical Methods in Natural Language Processing - (EMNLP), 2018. -[ pdf ] - -
-[6] - -M. X. Chen, O. Firat, A. Bapna, M. Johnson, W. Macherey, G. Foster, L. Jones, - M. Schuster, N. Shazeer, N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, - Z. Chen, Y. Wu, and M. Hughes, “The Best of Both Worlds: Combining Recent - Advances in Neural Machine Translation,” in Proc. Annual Meeting of - the Association for Computational Linguistics (ACL), 2018. -[ pdf ] - -
-[7] - -J. Kuczmarski and M. Johnson, “Gender-aware natural language translation,” - 2018. -[ pdf ] - -
-[8] - -R. Aharoni, M. Johnson, and O. Firat, “Massively multilingual neural machine - translation,” 2019. -[ pdf ] - -
-[9] - -J. Luo, Y. Cao, and R. Barzilay, “Neural decipherment via minimum-cost flow: - From ugaritic to linear b,” 2019. -[ http ] - -
-[10] - -N. Arivazhagan, C. Cherry, W. Macherey, C.-C. Chiu, S. Yavuz, R. Pang, W. Li, - and C. Raffel, “Monotonic infinite lookback attention for simultaneous - machine translation,” in Proc. Annual Meeting of the Association for - Computational Linguistics (ACL), 2019. -[ http ] - -
-[11] - -M. Freitag, I. Caswell, and S. Roy, “Ape at scale and its implications on mt - evaluation biases,” 2019. -[ pdf | -http ] - -
-[12] - -N. Arivazhagan, A. Bapna, O. Firat, D. Lepikhin, M. Johnson, M. Krikun, M. X. - Chen, Y. Cao, G. Foster, C. Cherry, W. Macherey, Z. Chen, and Y. Wu, - “Massively multilingual neural machine translation in the wild: Findings and - challenges,” 2019. -[ arXiv | -http ] - -
-[13] - -Y. Huang, Y. Cheng, A. Bapna, O. Firat, M. X. Chen, D. Chen, H. Lee, J. Ngiam, - Q. V. Le, Y. Wu, and Z. Chen, “Gpipe: Efficient training of giant neural - networks using pipeline parallelism,” in Advances in Neural Information - Processing Systems, 2019. -[ http ] - -
- -## Speech recognition - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-[1] - -C.-C.Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, - A. Kannan, R. J. Weiss, K. Rao, K. Gonina, N. Jaitly, B. Li, J. Chorowski, - and M. Bacchiani, “State-of-the-art speech recognition with - sequence-to-sequence models,” in Proc. IEEE International Conference - on Acoustics, Speech, and Signal Processing (ICASSP), 2018. -[ pdf ] - -
-[2] - -S. Toshniwal, T. N. Sainath, R. J. Weiss, B. Li, P. Moreno, E. Weinstein, and - K. Rao, “Multilingual speech recognition with a single end-to-end model,” - in Proc. IEEE International Conference on Acoustics, Speech, and - Signal Processing (ICASSP), 2018. -[ pdf ] - -
-[3] - -B. Li, T. N. Sainath, K. Sim, M. Bacchiani, E. Weinstein, P. Nguyen, Z. Chen, - Y. Wu, and K. Rao, “Multi-Dialect Speech Recognition With a Single - Sequence-to-Sequence Model,” in Proc. IEEE International Conference - on Acoustics, Speech, and Signal Processing (ICASSP), 2018. -[ pdf ] - -
-[4] - -T. N. Sainath, P. Prabhavalkar, S. Kumar, S. Lee, A. Kannan, D. Rybach, - V. Schogol, P. Nguyen, B. Li, Y. Wu, Z. Chen, and C. C. Chiu, “No Need for - a Lexicon? Evaluating the Value of the Pronunciation Lexica in End-to-End - Models,” in Proc. IEEE International Conference on Acoustics, - Speech, and Signal Processing (ICASSP), 2018. -[ pdf ] - -
-[5] - -D. Lawson, C. C. Chiu, G. Tucker, C. Raffel, K. Swersky, and N. Jaitly, - “Learning hard alignments with variational inference,” in Proc. - IEEE International Conference on Acoustics, Speech, and Signal Processing - (ICASSP), 2018. -[ pdf ] - -
-[6] - -A. Kannan, Y. Wu, P. Nguyen, T. N. Sainath, Z. Chen, and R. Prabhavalkar, “An - analysis of incorporating an external language model into a - sequence-to-sequence model,” in Proc. IEEE International Conference - on Acoustics, Speech, and Signal Processing (ICASSP), 2018. -[ pdf ] - -
-[7] - -R. Prabhavalkar, T. N. Sainath, Y. Wu, P. Nguyen, Z. Chen, C. C. Chiu, and - A. Kannan, “Minimum Word Error Rate Training for Attention-based - Sequence-to-sequence Models,” in Proc. IEEE International Conference - on Acoustics, Speech, and Signal Processing (ICASSP), 2018. -[ pdf ] - -
-[8] - -T. N. Sainath, C. C. Chiu, R. Prabhavalkar, A. Kannan, Y. Wu, P. Nguyen, and - Z. C. Z, “Improving the Performance of Online Neural Transducer Models,” - in Proc. IEEE International Conference on Acoustics, Speech, and - Signal Processing (ICASSP), 2018. -[ pdf ] - -
-[9] - -C. C. Chiu and C. Raffel, “Monotonic Chunkwise Attention,” in Proc. - International Conference on Learning Representations (ICLR), 2018. -[ pdf ] - -
-[10] - -I. Williams, A. Kannan, P. Aleksic, D. Rybach, and T. N. S. TN, “Contextual - Speech Recognition in End-to-End Neural Network Systems using Beam Search,” - in Proc. Interspeech, 2018. -[ pdf ] - -
-[11] - -C. C. Chiu, A. Tripathi, K. Chou, C. Co, N. Jaitly, D. Jaunzeikare, A. Kannan, - P. Nguyen, H. Sak, A. Sankar, J. Tansuwan, N. Wan, Y. Wu, and X. Zhang, - “Speech recognition for medical conversations,” in Proc. - Interspeech, 2018. -[ pdf ] - -
-[12] - -R. Pang, T. N. Sainath, R. Prabhavalkar, S. Gupta, Y. Wu, S. Zhang, and C. C. - Chiu, “Compression of End-to-End Models,” in Proc. Interspeech, - 2018. -[ pdf ] - -
-[13] - -S. Toshniwal, A. Kannan, C. C. Chiu, Y. Wu, T. N. Sainath, and K. Livescu, “A - comparison of techniques for language model integration in encoder-decoder - speech recognition,” in Proc. IEEE Spoken Language Technology - Workshop (SLT), 2018. -[ pdf ] - -
-[14] - -G. Pundak, T. N. Sainath, R. Prabhavalkar, A. Kannan, and D. Zhao, “Deep - context: End-to-end contextual speech recognition,” in Proc. IEEE - Spoken Language Technology Workshop (SLT), 2018. -[ pdf ] - -
-[15] - -B. Li, Y. Zhang, T. N. Sainath, Y. Wu, and W. Chan, “Bytes are all you need: - End-to-end multilingual speech recognition and synthesis with bytes,” in - Proc. IEEE International Conference on Acoustics, Speech, and Signal - Processing (ICASSP), 2019. -[ pdf ] - -
-[16] - -J. Guo, T. N. Sainath, and R. J. Weiss, “A spelling correction model for - end-to-end speech recognition,” in Proc. IEEE International - Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2019. -[ pdf ] - -
-[17] - -U. Alon, G. Pundak, and T. N. Sainath, “Contextual speech recognition with - difficult negative training examples,” in Proc. IEEE International - Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2019. -[ pdf ] - -
-[18] - -Y. Qin, N. Carlini, I. Goodfellow, G. Cottrell, and C. Raffel, “Imperceptible, - robust, and targeted adversarial examples for automatic speech recognition,” - in Proc. International Conference on Machine Learning (ICML), 2019. -[ pdf ] - -
-[19] - -D. S. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, - “SpecAugment: A Simple Data Augmentation Method for Automatic Speech - Recognition,” in arXiv, 2019. -[ pdf ] - -
-[20] - -B. Li, T. N. Sainath, R. Pang, and Z. Wu, “Semi-supervised training for - end-to-end models via weak distillation,” in Proc. IEEE International - Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2019. -[ pdf ] - -
-[21] - -S.-Y. Chang, R. Prabhavalkar, Y. He, T. N. Sainath, and G. Simko, “Joint - endpointing and decoding with end-to-end models,” in Proc. IEEE - International Conference on Acoustics, Speech, and Signal Processing - (ICASSP), 2019. -[ pdf ] - -
-[22] - -J. Heymann, K. C. Sim, and B. Li, “Improving ctc using stimulated learning for - sequence modeling,” in Proc. IEEE International Conference on - Acoustics, Speech, and Signal Processing (ICASSP), 2019. -[ pdf ] - -
-[23] - -A. Bruguier, R. Prabhavalkar, G. Pundak, and T. N. Sainath, “Phoebe: - Pronunciation-aware contextualization for end-to-end speech recognition,” in - Proc. IEEE International Conference on Acoustics, Speech, and Signal - Processing (ICASSP), 2019. -[ pdf ] - -
-[24] - -Y. He, T. N. Sainath, R. Prabhavalkar, I. McGraw, R. Alvarez, D. Zhao, - D. Rybach, A. Kannan, Y. Wu, R. Pang, Q. Liang, D. Bhatia, Y. Shangguan, - B. Li, G. Pundak, K. C. Sim, T. Bagby, S.-Y. Chang, K. Rao, and - A. Gruenstein, “Streaming end-to-end speech recognition for mobile - devices,” in Proc. IEEE International Conference on Acoustics, - Speech, and Signal Processing (ICASSP), 2019. -[ pdf ] - -
-[25] - -K. Irie, R. Prabhavalkar, A. Kannan, A. Bruguier, D. Rybach, and P. Nguyen, - “On the choice of modeling unit for sequence-to-sequence speech - recognition,” in Proc. Interspeech, 2019. -[ pdf ] - -
-[26] - -C. Peyser, H. Zhang, T. N. Sainath, and Z. Wu, “Improving Performance of - End-to-End ASR on Numeric Sequences,” in Proc. Interspeech, 2019. -[ pdf ] - -
-[27] - -D. Zhao, T. N. Sainath, D. Rybach, D. Bhatia, B. Li, and R. Pang, - “Shallow-fusion end-to-end contextual biasing,” in Proc. Interspeech, - 2019. -[ pdf ] - -
-[28] - -T. N. Sainath, R. Pang, D. Rybach, Y. He, R. Prabhavalkar, W. Li, M. Visontai, - Q. Liang, T. Strohman, Y. Wu, I. McGraw, and C.-C. Chiu, “Two-pass - end-to-end speech recognition,” in Proc. Interspeech, 2019. -[ pdf ] - -
-[29] - -C.-C. Chiu, W. Han, Y. Zhang, R. Pang, S. Kishchenko, P. Nguyen, A. Narayanan, - H. Liao, S. Zhang, A. Kannan, R. Prabhavalkar, Z. Chen, T. Sainath, and - Y. Wu, “A comparison of end-to-end models for long-form speech - recognition,” 2019. -[ pdf ] - -
-[30] - -A. Narayanan, R. Prabhavalkar, C. Chiu, D. Rybach, T. Sainath, and T. Strohman, - “Recognizing long-form speech using streaming end-to-end models,” 2019. -[ pdf ] - -
-[31] - -T. N. Sainath, R. Pang, R. Weiss, Y. He, C.-C. Chiu, and T. Strohman, “An - attention-based joint acoustic and text on-device end-to-end model,” in - Proc. IEEE International Conference on Acoustics, Speech, and Signal - Processing (ICASSP), 2020. - -
-[32] - -Z. Lu, L. Cao, Y. Zhang, C.-C. Chiu, and J. Fan, “Speech sentiment analysis - via pre-trained features from end-to-end asr models,” in Proc. IEEE - International Conference on Acoustics, Speech, and Signal Processing - (ICASSP), 2020. - -
-[33] - -D. Park, Y. Zhang, C.-C. Chiu, Y. Chen, B. Li, W. Chan, Q. Le, and Y. Wu, - “Specaugment on large scale datasets,” in Proc. IEEE International - Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020. -[ pdf ] - -
-[34] - -T. Sainath, Y. He, B. Li, A. Narayanan, R. Pang, A. Bruguier, S. yiin Chang, - W. Li, R. Alvarez, Z. Chen, C. cheng Chiu, D. Garcia, A. Gruenstein, K. Hu, - M. Jin, A. Kannan, Q. Liang, I. McGraw, C. Peyser, R. Prabhavalkar, - G. Pundak, D. Rybach, Y. Shangguan, Y. Sheth, T. Strohman, M. Visontai, - Y. Wu, Y. Zhang, and D. Zhao, “A streaming on-device end-to-end model - surpassing server-side conventional model quality and latency,” in - Proc. IEEE International Conference on Acoustics, Speech, and Signal - Processing (ICASSP), 2020. - -
-[35] - -A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, - Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer - for speech recognition,” in Proc. Interspeech, 2020. -[ pdf ] - -
-[36] - -W. Han, Z. Zhang, Y. Zhang, J. Yu, C.-C. Chiu, J. Qin, A. Gulati, R. Pang, and - Y. Wu, “Contextnet: Improving convolutional neural networks for automatic - speech recognition with global context,” in Proc. Interspeech, 2020. -[ pdf ] - -
-[37] - -W. Li, J. Qin, C.-C. Chiu, R. Pang, and Y. He, “Parallel rescoring with - transformer for streaming on-device speech recognition,” in Proc. - Interspeech, 2020. - -
-[38] - -D. S. Park, Y. Zhang, Y. Jia, W. Han, C.-C. Chiu, B. Li, Y. Wu, and Q. V. Le, - “Improved noisy student training for automatic speech recognition,” in - Proc. Interspeech, 2020. -[ pdf ] - -
- -## Language understanding - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-[1] - -A. Kannan, K. Chen, D. Jaunzeikare, and A. Rajkomar, “Semi-Supervised - Learning for Information Extraction from Dialogue,” in Proc. - Interspeech, 2018. -[ pdf ] - -
-[2] - -S. Yavuz, C. C. Chiu, P. Nguyen, and Y. Wu, “CaLcs: Continuously - Approximating Longest Common Subsequence for Sequence Level Optimization,” - in Proc. Conference on Empirical Methods in Natural Language Processing - (EMNLP), 2018. -[ pdf ] - -
-[3] - -P. Haghani, A. Narayanan, M. Bacchiani, G. Chuang, N. Gaur, P. Moreno, - R. Prabhavalkar, Z. Qu, and A. Waters, “From Audio to Semantics: Approaches - to End-to-End Spoken Language Understanding,” in Proc. IEEE Spoken - Language Technology Workshop (SLT), 2018. -[ pdf ] - -
-[4] - -M. X. Chen, B. N. Lee, G. Bansal, Y. Cao, S. Zhang, J. Lu, J. Tsay, Y. Wang, - A. M. Dai, Z. Chen, T. Sohn, and Y. Wu, “Gmail smart compose: Real-time - assisted writing,” in Proceedings of the 25th ACM SIGKDD International - Conference on Knowledge Discovery & Data Mining, Association for Computing - Machinery, 2019. -[ pdf | -http ] - -
- -## Speech synthesis - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-[1] - -J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, - Y. Zhang, Y. Wang, R. Skerry-Ryan, R. A. Saurous, Y. Agiomyrgiannakis, and - Y. Wu, “Natural TTS synthesis by conditioning WaveNet on mel spectrogram - predictions,” in Proc. IEEE International Conference on Acoustics, - Speech, and Signal Processing (ICASSP), 2018. -[ sound examples | -pdf ] - -
-[2] - -J. Chorowski, R. J. Weiss, R. A. Saurous, and S. Bengio, “On using - backpropagation for speech texture generation and voice conversion,” in - Proc. IEEE International Conference on Acoustics, Speech, and Signal - Processing (ICASSP), 2018. -[ sound examples | -pdf ] - -
-[3] - -Y. Jia, Y. Zhang, R. J. Weiss, Q. Wang, J. Shen, F. Ren, Z. Chen, P. Nguyen, - R. Pang, I. Lopez-Moreno, and Y. Wu, “Transfer learning from speaker - verification to multispeaker text-to-speech synthesis,” in Advances in - Neural Information Processing Systems, 2018. -[ sound examples | -pdf ] - -
-[4] - -W. N. Hsu, Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Y. Wang, Y. Cao, Y. Jia, - Z. Chen, J. Shen, P. Nguyen, and R. Pang, “Hierarchical generative modeling - for controllable speech synthesis,” in Proc. International Conference - on Learning Representations (ICLR), 2019. -[ sound examples | -pdf ] - -
-[5] - -W. N. Hsu, Y. Zhang, R. J. Weiss, Y. A. Chung, Y. Wang, Y. Wu, and J. Glass, - “Disentangling correlated speaker and noise for speech synthesis via data - augmentation and adversarial factorization,” in NeurIPS 2018 Workshop - on Interpretability and Robustness in Audio, Speech, and Language, 2018. -[ pdf ] - -
-[6] - -H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu, - “LibriTTS: A corpus derived from LibriSpeech for text-to-speech,” in - Proc. Interspeech, 2019. -[ data | -pdf ] - -
-[7] - -F. Biadsy, R. J. Weiss, P. Moreno, D. Kanvesky, and Y. Jia, “Parrotron: An - end-to-end speech-to-speech conversion model and its applications to - hearing-impaired speech and speech separation,” in Proc. Interspeech, - 2019. -[ sound examples | -pdf ] - -
-[8] - -Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Z. Chen, R. J. Skerry-Ryan, Y. Jia, - A. Rosenberg, and B. Ramabhadran, “Learning to speak fluently in a foreign - language: Multilingual speech synthesis and cross-language voice cloning,” - in Proc. Interspeech, 2019. -[ sound examples | -pdf ] - -
-[9] - -G. Sun, Y. Zhang, R. J. Weiss, Y. Cao, H. Zen, and Y. Wu, “Fully-hierarchical - fine-grained prosody modeling for interpretable speech synthesis,” in - Proc. IEEE International Conference on Acoustics, Speech, and Signal - Processing (ICASSP), 2020. -[ sound examples | -pdf ] - -
-[10] - -G. Sun, Y. Zhang, R. J. Weiss, Y. Cao, H. Zen, A. Rosenberg, B. Ramabhadran, - and Y. Wu, “Generating diverse and natural text-to-speech samples using a - quantized fine-grained VAE and auto-regressive prosody prior,” in - Proc. IEEE International Conference on Acoustics, Speech, and Signal - Processing (ICASSP), 2020. -[ sound examples | -pdf ] - -
- -## Speech translation - - - - - - - - - - - - - - - - - - - - - - -
-[1] - -R. J. Weiss, J. Chorowski, N. Jaitly, Y. Wu, and Z. Chen, - “Sequence-to-sequence models can directly translate foreign speech,” in - Proc. Interspeech, 2017. -[ pdf ] - -
-[2] - -Y. Jia, M. Johnson, W. Macherey, R. J. Weiss, Y. Cao, C. C. Chiu, N. Ari, - S. Laurenzo, and Y. Wu, “Leveraging weakly supervised data to improve - end-to-end speech-to-text translation,” in Proc. IEEE International - Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2019. -[ pdf ] - -
-[3] - -Y. Jia, R. J. Weiss, F. Biadsy, W. Macherey, M. Johnson, Z. Chen, and Y. Wu, - “Direct speech-to-speech translation with a sequence-to-sequence model,” in - Proc. Interspeech, 2019. -[ sound examples | -pdf ] - -
- -## Optimization - - - - - - - - - - - - - - - - - - - - - - -
-[1] - -R. Anil, V. Gupta, T. Koren, K. Regan, and Y. Singer, “Second order - optimization made practical,” arXiv preprint arXiv:2002.09018, 2020. -[ pdf ] - -
-[2] - -N. Agarwal, R. Anil, E. Hazan, T. Koren, and C. Zhang, “Disentangling adaptive - gradient methods from learning rates,” arXiv preprint - arXiv:2002.11803, 2020. -[ pdf ] - -
-[3] - -R. Anil, V. Gupta, T. Koren, and Y. Singer, “Memory efficient adaptive - optimization,” in Advances in Neural Information Processing Systems, - pp. 9749--9758, 2019. -[ pdf ] - -
\ No newline at end of file diff --git a/README.md b/README.md index f2fdc64a..b289511d 100644 --- a/README.md +++ b/README.md @@ -1,314 +1,19 @@ -# People's Speech Data Pipelines (Originally a fork of Lingvo) +# People's Speech Data Pipelines -[![PyPI](https://badge.fury.io/py/lingvo.svg)](https://badge.fury.io/py/lingvo) -[![Python](https://img.shields.io/pypi/pyversions/lingvo)](https://badge.fury.io/py/tensorflow) +Installation -[![Documentation](https://img.shields.io/badge/api-reference-blue.svg)](https://tensorflow.github.io/lingvo) - -[![License](https://img.shields.io/github/license/tensorflow/lingvo)](LICENSE) - -## What is it? - -Lingvo is a framework for building neural networks in Tensorflow, particularly -sequence models. - -A list of publications using Lingvo can be found [here](PUBLICATIONS.md). - -## Table of Contents - -* [Releases](#releases) - * [Major breaking changes](#major-breaking-changes) -* [Quick start](#quick-start) - * [Installation](#installation) - * [Running the MNIST image model](#running-the-mnist-image-model) - * [Running the machine translation model](#running-the-machine-translation-model) - * [Running the 3d object detection model](#running-the-3d-object-detection-model) -* [Models](#models) - * [Automatic Speech Recognition](#automatic-speech-recognition) - * [Car](#car) - * [Image](#image) - * [Language Modelling](#language-modelling) - * [Machine Translation](#machine-translation) -* [References](#references) -* [License](#license) - -## Releases - -PyPI Version | Commit ------------- | ---------------------------------------- -0.7.2 | b05642fe386ee79e0d88aa083565c9a93428519e - -
-Older releases -

- -PyPI Version | Commit ------------- | ------ - -Details for older releases are unavailable. - -

- -### Major breaking changes - -#### HEAD - -* **General** - * NestedMap Flatten/Pack/Transform/Filter etc now expand descendent dicts - as well. - * Subclasses of BaseLayer extending from `abc.ABCMeta` should now extend - `base_layer.ABCLayerMeta` instead. - * Trying to call self.CreateChild outside of `__init__` now raises an - error. - * `base_layer.initializer` has been removed. Subclasses no longer need to - decorate their `__init__` function. - * Trying to call self.CreateVariable outside of `__init__` or - `_CreateLayerVariables` now raises an error. - * It is no longer possible to access self.vars or self.theta inside of - `__init__`. Refactor by moving the variable creation and access to - `_CreateLayerVariables`. The variable scope is set automatically according to - the layer name in `_CreateLayerVariables`. - -
-Older releases -

- -Details for older releases are available. - -

- -## Quick start - -### Installation - -There are two ways to set up Lingvo: installing a fixed version through pip, or -cloning the repository and building it with bazel. Docker configurations are -provided for each case. - -If you would just like to use the framework as-is, it is easiest to just install -it through pip. This makes it possible to develop and train custom models using -a frozen version of the Lingvo framework. However, it is difficult to modify the -framework code or implement new custom ops. - -If you would like to develop the framework further and potentially contribute -pull requests, you should avoid using pip and clone the repository instead. - -**pip:** - -The [Lingvo pip package](https://pypi.org/project/lingvo) can be installed with -`pip3 install lingvo`. - -See the -[codelab](https://colab.research.google.com/github/tensorflow/lingvo/blob/master/codelabs/introduction.ipynb) -for how to get started with the pip package. - -**From sources:** - -The prerequisites are: - -* a TensorFlow 2.2 [installation](https://www.tensorflow.org/install/), -* a `C++` compiler (only g++ 7.3 is officially supported), and -* the bazel build system. - -Refer to [docker/dev.dockerfile](docker/dev.dockerfile) for a set of working -requirements. - -`git clone` the repository, then use bazel to build and run targets directly. -The `python -m module` commands in the codelab need to be mapped onto `bazel -run` commands. - -**docker:** - -Docker configurations are available for both situations. Instructions can be -found in the comments on the top of each file. - -* [lib.dockerfile](docker/lib.dockerfile) has the Lingvo pip package - preinstalled. -* [dev.dockerfile](docker/dev.dockerfile) can be used to build Lingvo from - sources. - -[How to install docker.](https://docs.docker.com/install/linux/docker-ce/ubuntu/) - -### Running the MNIST image model - -#### Preparing the input data - -**pip:** - -```shell -mkdir -p /tmp/mnist -python3 -m lingvo.tools.keras2ckpt --dataset=mnist -``` - -**bazel:** - -```shell -mkdir -p /tmp/mnist -bazel run -c opt //lingvo/tools:keras2ckpt -- --dataset=mnist -``` - -The following files will be created in `/tmp/mnist`: - -* `mnist.data-00000-of-00001`: 53MB. -* `mnist.index`: 241 bytes. - -#### Running the model - -**pip:** - -```shell -cd /tmp/mnist -curl -O https://raw.githubusercontent.com/tensorflow/lingvo/master/lingvo/tasks/image/params/mnist.py -python3 -m lingvo.trainer --run_locally=cpu --mode=sync --model=mnist.LeNet5 --logdir=/tmp/mnist/log -``` - -**bazel:** - -```shell -(cpu) bazel build -c opt //lingvo:trainer -(gpu) bazel build -c opt --config=cuda //lingvo:trainer -bazel-bin/lingvo/trainer --run_locally=cpu --mode=sync --model=image.mnist.LeNet5 --logdir=/tmp/mnist/log --logtostderr -``` - -After about 20 seconds, the loss should drop below 0.3 and a checkpoint will be -saved, like below. Kill the trainer with Ctrl+C. ``` -trainer.py:518] step: 205, steps/sec: 11.64 ... loss:0.25747201 ... -checkpointer.py:115] Save checkpoint -checkpointer.py:117] Save checkpoint done: /tmp/mnist/log/train/ckpt-00000205 +sudo apt-get install git-lfs +# Set up a virtual environment of some sort +pip install numpy Cython +python setup.py develop +cp galvasr2/*.jar $(python -c "import pyspark; print(pyspark.__path__[0])")/jars ``` -Some artifacts will be produced in `/tmp/mnist/log/control`: - -* `params.txt`: hyper-parameters. -* `model_analysis.txt`: model sizes for each layer. -* `train.pbtxt`: the training `tf.GraphDef`. -* `events.*`: a tensorboard events file. - -As well as in `/tmp/mnist/log/train`: - -* `checkpoint`: a text file containing information about the checkpoint files. -* `ckpt-*`: the checkpoint files. +Run forced alignment pipeline. -Now, let's evaluate the model on the "Test" dataset. In the normal training -setup the trainer and evaler should be run at the same time as two separate -processes. - -**pip:** - -```shell -python3 -m lingvo.trainer --job=evaler_test --run_locally=cpu --mode=sync --model=mnist.LeNet5 --logdir=/tmp/mnist/log ``` - -**bazel:** - -```shell -bazel-bin/lingvo/trainer --job=evaler_test --run_locally=cpu --mode=sync --model=image.mnist.LeNet5 --logdir=/tmp/mnist/log --logtostderr +python galvasr2/align/spark/align_cuda_decoder.py --stage=0 ``` -Kill the job with Ctrl+C when it starts waiting for a new checkpoint. - -``` -base_runner.py:177] No new check point is found: /tmp/mnist/log/train/ckpt-00000205 -``` - -The evaluation accuracy can be found slightly earlier in the logs. - -``` -base_runner.py:111] eval_test: step: 205, acc5: 0.99775392, accuracy: 0.94150388, ..., loss: 0.20770954, ... -``` - -### Running the machine translation model - -To run a more elaborate model, you'll need a cluster with GPUs. Please refer to -[`third_party/py/lingvo/tasks/mt/README.md`](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/mt/README.md) -for more information. - -### Running the 3d object detection model - -To run the StarNet model using CloudTPUs on GCP, please refer to -[`third_party/py/lingvo/tasks/car/README.md`](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/car/README.md). - -## Models - -### Automatic Speech Recogition - -* [asr.librispeech.Librispeech960Grapheme](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/asr/params/librispeech.py)1,2 -* [asr.librispeech.Librispeech960Wpm](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/asr/params/librispeech.py)1,2 - -### Car - -* [car.kitti.StarNetCarModel0701](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/car/params/kitti.py)3 -* [car.kitti.StarNetPedCycModel0704](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/car/params/kitti.py)3 -* [car.waymo.StarNetVehicle](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/car/params/waymo.py)3 -* [car.waymo.StarNetPed](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/car/params/waymo.py)3 - -### Image - -* [image.mnist.LeNet5](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/image/params/mnist.py)4 - -### Language Modelling - -* [lm.one_billion_wds.WordLevelOneBwdsSimpleSampledSoftmax](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/lm/params/one_billion_wds.py)5 - -### Machine Translation - -* [mt.wmt14_en_de.WmtEnDeTransformerBase](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/mt/params/wmt14_en_de.py)6 -* [mt.wmt14_en_de.WmtEnDeRNMT](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/mt/params/wmt14_en_de.py)6 -* [mt.wmtm16_en_de.WmtCaptionEnDeTransformer](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/mt/params/wmtm16_en_de.py)6 - - - -\[1]: [Listen, Attend and Spell](https://arxiv.org/pdf/1508.01211.pdf). William -Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. ICASSP 2016. - -\[2]: [End-to-end Continuous Speech Recognition using Attention-based Recurrent -NN: First Results](https://arxiv.org/pdf/1412.1602.pdf). Jan Chorowski, Dzmitry -Bahdanau, Kyunghyun Cho, and Yoshua Bengio. arXiv 2014. - -\[3]: -[StarNet: Targeted Computation for Object Detection in Point Clouds](https://arxiv.org/pdf/1908.11069.pdf). -Jiquan Ngiam, Benjamin Caine, Wei Han, Brandon Yang, Yuning Chai, Pei Sun, Yin -Zhou, Xi Yi, Ouais Alsharif, Patrick Nguyen, Zhifeng Chen, Jonathon Shlens, and -Vijay Vasudevan. arXiv 2019. - -\[4]: -[Gradient-based learning applied to document recognition](http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf). -Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. IEEE 1998. - -\[5]: -[Exploring the Limits of Language Modeling](https://arxiv.org/pdf/1602.02410.pdf). -Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. -arXiv, 2016. - -\[6]: [The Best of Both Worlds: Combining Recent Advances in Neural Machine -Translation](http://aclweb.org/anthology/P18-1008). Mia X. Chen, Orhan Firat, -Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike -Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz -Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. ACL 2018. - - - -## References - -* [API Docs](https://tensorflow.github.io/lingvo/) -* [Codelab](https://colab.research.google.com/github/tensorflow/lingvo/blob/master/codelabs/introduction.ipynb) - -Please cite this [paper](https://arxiv.org/abs/1902.08295) when referencing -Lingvo. - -``` -@misc{shen2019lingvo, - title={Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling}, - author={Jonathan Shen and Patrick Nguyen and Yonghui Wu and Zhifeng Chen and others}, - year={2019}, - eprint={1902.08295}, - archivePrefix={arXiv}, - primaryClass={cs.LG} -} -``` - -## License - -[Apache License 2.0](LICENSE) \ No newline at end of file diff --git a/codelabs/introduction.ipynb b/codelabs/introduction.ipynb deleted file mode 100644 index 259c95a8..00000000 --- a/codelabs/introduction.ipynb +++ /dev/null @@ -1,658 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "view-in-github" - }, - "source": [ - "\u003ca href=\"https://colab.research.google.com/github/tensorflow/lingvo/blob/master/codelabs/introduction.ipynb\" target=\"_parent\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\u003c/a\u003e" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "od97_nvR82Qo" - }, - "source": [ - "# Introduction to Lingvo\n", - "\n", - "This codelab will guide you through the implementation of a **sequence-to-sequence** model using [**Lingvo**](https://github.com/tensorflow/lingvo).\n", - "\n", - "**Sequence-to-sequence** models map input sequences of arbitrary length to\n", - "output sequences of arbitrary length. Example uses of sequence-to-sequence\n", - "models include machine translation, which maps a sequence of words from one\n", - "language into a sequence of words in another language with the same meaning;\n", - "speech recognition, which maps a sequence of acoustic features into a sequence\n", - "of words; and text summarization, which\n", - "maps a sequence of words into a shorter sequence which conveys the same meaning.\n", - "\n", - "In this codelab, you will create a model which restores punctuation and\n", - "capitalization to text which has been lowercased and stripped of punctuation.\n", - "For example, given the following text:\n", - "\n", - "\u003e she asked do you know the way to san jose\n", - "\n", - "The model will output the following properly-punctuated-and-capitalized text:\n", - "\n", - "\u003e She asked, \"Do you know the way to San Jose\"?\n", - "\n", - "We will train an RNMT+ model based off of [\"The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation. (Chen et al., 2018)\"](https://arxiv.org/abs/1804.09849)." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "4bPGJjZkLBdz" - }, - "source": [ - "## Table of Contents\n", - "\n", - "In Colab, click `[View]-\u003e[Table of contents]` on the menu bar." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "bnpyjXz1OFAu" - }, - "source": [ - "## Prerequisites\n", - "\n", - "The main goal of this codelab is to teach you how to define and train sequence-to-sequence models in Lingvo. We do not aim to teach either Python or Tensorflow, and no sophisticated Python or Tensorflow programming will be required. However, the following will be helpful in understanding this codelab:\n", - "\n", - "- Familiarity with high-level machine learning primitives, in particular,\n", - " recurrent neural networks, LSTMs, and attention.\n", - "- Comfort reading and writing simple Python code. In particular, you should\n", - " know how to define simple classes and how inheritance works.\n", - "- Basic knowledge of the Tensorflow training workflow. If you have trained\n", - " simple Tensorflow models before (e.g., via another codelab), you should know\n", - " enough for this codelab.\n", - "\n", - "### Resources\n", - "\n", - "- [Introduction to RNNs and LSTMs](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)\n", - "- [Official Tensorflow Tutorials](https://www.tensorflow.org/tutorials/)\n", - "- For more advanced topics or to get a deeper understanding of Lingvo beyond this codelab, see the [paper](https://arxiv.org/abs/1902.08295)." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "coa-U7N6Kunb" - }, - "source": [ - "## Learning Objectives\n", - "\n", - "This codelab will teach you the following:\n", - "\n", - "- How to generate input data for training a sequence-to-sequence model in Lingvo.\n", - "- How models are specified and configured in Lingvo, by adapting a pre-existing model architecture for machine translation.\n", - "- How to use the trained model for inference.\n", - "\n", - "This codelab does not:\n", - "\n", - "- Teach you how to design a model for solving specific tasks.\n", - "- Provide a state-of-the-art model for the punctuator task." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "_mtJSYor9d6d" - }, - "source": [ - "## Environment Setup\n", - "\n", - "To start with, we need to connect this Colab notebook with Lingvo.\n", - "\n", - "```shell\n", - "mkdir -p /tmp/lingvo_codelab \u0026\u0026 cd /tmp/lingvo_codelab\n", - "pip3 install lingvo\n", - "python3 -m lingvo.ipython_kernel\n", - "```\n", - "\n", - "Finally, on the top right hand side of this Colab notebook, open the dropdown beside \"CONNECT\" and select \"Connect to local runtime...\", enter `http://localhost:8888` and press CONNECT.\n", - "\n", - "You should now see the words \"CONNECTED\" and be able to execute the following cell." - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "2JRfZTumHxpY" - }, - "outputs": [], - "source": [ - "import lingvo" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "lkN9qajvNtEs" - }, - "source": [ - "## Input Pipeline\n", - "\n", - "In order to train a sequence-to-sequence model, we need a set of pairs of source\n", - "and target sequences. For this codelab, our source sequences will be\n", - "text which has been lowercased and had its punctuation removed, and the target\n", - "sequences will be the original sentences, with their original casing and\n", - "punctuation.\n", - "\n", - "Since neural networks require numeric inputs, we will also need a tokenization scheme mapping the sequence of characters to a sequence of numbers. In this codelab, we will use a pre-trained word-piece model.\n", - "\n", - "### Download Raw Input\n", - "\n", - "We will use the [Brown Corpus](http://www.nltk.org/nltk_data) as the source of our training data. Run the following cell to download and preprocess the dataset. The script will generate `train.txt` and `test.txt` containing the training and test data at an 80:20 split with individual sentences on each line." - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "Asfie_a8YmN3" - }, - "outputs": [], - "source": [ - "!python3 -m lingvo.tasks.punctuator.tools.download_brown_corpus --outdir=/tmp/punctuator_data\n", - "!curl -O https://raw.githubusercontent.com/tensorflow/lingvo/master/lingvo/tasks/punctuator/params/brown_corpus_wpm.16000.vocab" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "BHq87NF-8lgD" - }, - "source": [ - "### Define an Input Generator\n", - "\n", - "In order to train a model, we need an input generator that provides structured mini-batches of source-target pairs. The input generator handles all the processing necessary to generate numeric data that can be fed to the model. This includes:\n", - "\n", - "- reading examples from the data source in random order, where the data source may be split across multiple files;\n", - "- processing the data -- for our task this involves generating a \"source\" sentence by converting all characters to lower-case and removing punctuation, and then tokenizing both the source and target sequences into integer tokens; and\n", - "- batching together examples by padding them to the same length. Multiple buckets of different lengths may be used to avoid inefficiency from padding a short input to a very long length.\n", - "\n", - "Fortunately, the majority of this is handled in the background by Lingvo. We only need to specify how the data should be processed.\n", - "\n", - "Input generators are subclasses of *BaseInputGenerator* found in [lingvo/core/base_input_generator.py](https://github.com/tensorflow/lingvo/blob/master/lingvo/core/base_input_generator.py) and have the following structure:\n", - "\n", - "- a `Params` classmethod that returns a default Params object for configuring the input generator. Experiment configurations inside Lingvo are controlled using these Params objects.\n", - "- an `_InputBatch` method that returns a [`NestedMap`](https://github.com/tensorflow/lingvo/blob/3344e201719961183d88713784ccbae447f5c52a/lingvo/core/py_utils.py#L392) containing the input batch. `NestedMap` is an arbitrarily nested map structure used throughout Lingvo.\n", - "- an optional `_PreprocessInputBatch` method that preprocesses the batch returned by `_InputBatch`.\n", - "\n", - "Here is an example of the input generator for the Punctuator task, found in [lingvo/tasks/punctuator/input_generator.py](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/punctuator/input_generator.py).\n", - "\n", - "Run the cell below to write the file to disk." - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "T2BzBCqE_yvt" - }, - "outputs": [], - "source": [ - "%%writefile input_generator.py\n", - "import string\n", - "import lingvo.compat as tf\n", - "from lingvo.core import base_input_generator\n", - "from lingvo.core import base_layer\n", - "from lingvo.core import generic_input\n", - "from lingvo.core import py_utils\n", - "from lingvo.core import tokenizers\n", - "\n", - "\n", - "class PunctuatorInput(base_input_generator.BaseSequenceInputGenerator):\n", - " \"\"\"Reads text line by line and processes them for the punctuator task.\"\"\"\n", - "\n", - " @classmethod\n", - " def Params(cls):\n", - " \"\"\"Defaults params for PunctuatorInput.\"\"\"\n", - " p = super(PunctuatorInput, cls).Params()\n", - " p.tokenizer = tokenizers.WpmTokenizer.Params()\n", - " return p\n", - "\n", - " def _ProcessLine(self, line):\n", - " \"\"\"A single-text-line processor.\n", - " Gets a string tensor representing a line of text that have been read from\n", - " the input file, and splits it to graphemes (characters).\n", - " We use original characters as the target labels, and the lowercased and\n", - " punctuation-removed characters as the source labels.\n", - " Args:\n", - " line: a 1D string tensor.\n", - " Returns:\n", - " A list of tensors, in the expected order by __init__.\n", - " \"\"\"\n", - " # Tokenize the input into integer ids.\n", - " # tgt_ids has the start-of-sentence token prepended, and tgt_labels has the\n", - " # end-of-sentence token appended.\n", - " tgt_ids, tgt_labels, tgt_paddings = self.StringsToIds(\n", - " tf.convert_to_tensor([line]))\n", - "\n", - " def Normalize(line):\n", - " # Lowercase and remove punctuation.\n", - " line = line.lower().translate(None, string.punctuation.encode('utf-8'))\n", - " # Convert multiple consecutive spaces to a single one.\n", - " line = b' '.join(line.split())\n", - " return line\n", - "\n", - " normalized_line = tf.py_func(Normalize, [line], tf.string, stateful=False)\n", - " _, src_labels, src_paddings = self.StringsToIds(\n", - " tf.convert_to_tensor([normalized_line]), is_source=True)\n", - " # The model expects the source without a start-of-sentence token.\n", - " src_ids = src_labels\n", - "\n", - " # Compute the length for bucketing.\n", - " bucket_key = tf.cast(\n", - " tf.round(\n", - " tf.maximum(\n", - " tf.reduce_sum(1.0 - src_paddings),\n", - " tf.reduce_sum(1.0 - tgt_paddings))), tf.int32)\n", - " tgt_weights = 1.0 - tgt_paddings\n", - "\n", - " # Return tensors in an order consistent with __init__.\n", - " out_tensors = [\n", - " src_ids, src_paddings, tgt_ids, tgt_paddings, tgt_labels, tgt_weights\n", - " ]\n", - " return [tf.squeeze(t, axis=0) for t in out_tensors], bucket_key\n", - "\n", - " def _DataSourceFromFilePattern(self, file_pattern):\n", - " \"\"\"Create the input processing op.\n", - " Args:\n", - " file_pattern: The file pattern to use as input.\n", - " Returns:\n", - " an operation that when executed, calls `_ProcessLine` on a line read\n", - " from `file_pattern`.\n", - " \"\"\"\n", - " return generic_input.GenericInput(\n", - " file_pattern=file_pattern,\n", - " processor=self._ProcessLine,\n", - " # Pad dimension 0 to the same length.\n", - " dynamic_padding_dimensions=[0] * 6,\n", - " # The constant values to use for padding each of the outputs.\n", - " dynamic_padding_constants=[0, 1, 0, 1, 0, 0],\n", - " **self.CommonInputOpArgs())\n", - "\n", - " def __init__(self, params):\n", - " super(PunctuatorInput, self).__init__(params)\n", - "\n", - " # Build the input processing graph.\n", - " (self._src_ids, self._src_paddings, self._tgt_ids, self._tgt_paddings,\n", - " self._tgt_labels,\n", - " self._tgt_weights), self._bucket_keys = self._BuildDataSource()\n", - "\n", - " def InfeedBatchSize(self):\n", - " return tf.shape(self._src_ids)[0]\n", - "\n", - " def _InputBatch(self):\n", - " \"\"\"Returns a single batch as a `.NestedMap` to be passed to the model.\"\"\"\n", - " ret = py_utils.NestedMap()\n", - "\n", - " ret.bucket_keys = self._bucket_keys\n", - "\n", - " ret.src = py_utils.NestedMap()\n", - " ret.src.ids = tf.cast(self._src_ids, dtype=tf.int32)\n", - " ret.src.paddings = self._src_paddings\n", - "\n", - " ret.tgt = py_utils.NestedMap()\n", - " ret.tgt.ids = self._tgt_ids\n", - " ret.tgt.labels = tf.cast(self._tgt_labels, dtype=tf.int32)\n", - " ret.tgt.weights = self._tgt_weights\n", - " ret.tgt.paddings = self._tgt_paddings\n", - "\n", - " return ret" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "lrwT1f0IfPca" - }, - "source": [ - "## Model Definition\n", - "\n", - "Next, we need to define the network structure for the task. The network is a nested structure of layers. Most classes in Lingvo are subclasses of *BaseLayer* found in [lingvo/core/base_layer.py](https://github.com/tensorflow/lingvo/blob/560f838bd576c7b911df379121eb58252b6ae326/lingvo/core/base_layer.py#L150) and inherit the following:\n", - "\n", - "- a Params classmethod that returns a default [Params](https://github.com/tensorflow/lingvo/blob/3344e201719961183d88713784ccbae447f5c52a/lingvo/core/hyperparams.py#L151) object for configuring the class. In addition to hyperparameters, the Params object can also contain Params objects for configuring child layers. Some of the properties present in all Params objects include:\n", - " - `cls`: the python class that the Params object is associated with. This can be used to construct an instance of the class;\n", - " - `name`: the name of this layer;\n", - " - `dtype`: the default dtype to use when creating variables.\n", - "- The `__init__` constructor. All child layers and variables should be created here.\n", - "- A `CreateVariable` method that is called to create variables.\n", - "- A `CreateChild` method that is called to create child layers.\n", - "- A `FProp` method that implements forward propagation through the layer.\n", - "\n", - "As a reference, many examples of layers can be found in [lingvo/core/layers.py](https://github.com/tensorflow/lingvo/blob/master/lingvo/core/layers.py), [lingvo/core/attention.py](https://github.com/tensorflow/lingvo/blob/master/lingvo/core/attention.py), and [lingvo/core/rnn_layers.py](https://github.com/tensorflow/lingvo/blob/master/lingvo/core/rnn_layers.py).\n", - "\n", - "\u0026nbsp;\n", - "\n", - "The root layer for the network should be a subclass of `BaseTask` found in [lingvo/core/base_model.py](https://github.com/tensorflow/lingvo/blob/918c584f057481717eff6e1e29ae028aeab3d165/lingvo/core/base_model.py#L79), and implements the following:\n", - "\n", - "- A `ComputePredictions` method that takes the current variable values (`theta`) and `input_batch` and returns the network predictions.\n", - "- A `ComputeLoss` method that takes `theta`, `input_batch`, and the `predictions` returned from `ComputePredictions` and returns a dictionary of scalar metrics, one of which should be `loss`. These scalar metrics are exported to TensorBoard as summaries.\n", - "- An optional `Decode` method for creating a separate graph for decoding. For example, training and evaluation might use teacher forcing while decoding might not.\n", - "- An optional `Inference` method that returns a graph with feeds and fetches that can be used together with a saved checkpoint for inference. This differs from `Decode` in that it can be fed data directly instead of using data from the input generator.\n", - "\n", - "\u0026nbsp;\n", - "\n", - "This codelab uses the existing networks from [lingvo/tasks/punctuator/model.py](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/punctuator/model.py), which is derived from the networks in [lingvo/tasks/mt/model.py](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/mt/model.py) with an added `Inference` method for the punctuator task. The actual logic lies mostly in [lingvo/tasks/mt/encoder.py](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/mt/encoder.py) and [lingvo/tasks/mt/decoder.py](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/mt/decoder.py)." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "9ItaOgXNpql0" - }, - "source": [ - "## Model Configuration\n", - "\n", - "After defining the input generator and the network, we need to create an model configuration with the specific hyperparameters to use for our model.\n", - "\n", - "Since there is only a single task, we create a subclass of `SingleTaskModelParams` found in [lingvo/core/base_model_params.py](https://github.com/tensorflow/lingvo/blob/4747cf80a7e6cf58211aa899bae854820a3b42f6/lingvo/core/base_model_params.py#L47). It has the following structure:\n", - "\n", - "- The `Train`/`Dev`/`Test` methods configure the input generator for the respective datasets.\n", - "- The `Task` method configures the network.\n", - "\n", - "The following cell contains the configuration that will be used in this codelab. It can also be found in [lingvo/tasks/punctuator/params/codelab.py](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/punctuator/params/codelab.py). The network configuration in the `Task` classmethod is delegated to [lingvo/tasks/mt/params/base_config.py](https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/mt/params/base_config.py).\n", - "\n", - "Run the cell below to write the file to disk." - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "3CjruUvXY5we" - }, - "outputs": [], - "source": [ - "%%writefile codelab.py\n", - "import input_generator\n", - "import os\n", - "from lingvo import model_registry\n", - "import lingvo.compat as tf\n", - "from lingvo.core import base_model_params\n", - "from lingvo.tasks.mt import base_config\n", - "from lingvo.tasks.punctuator import model\n", - "\n", - "\n", - "# This base class defines parameters for the input generator for a specific\n", - "# dataset. Specific network architectures will be implemented in subclasses.\n", - "class BrownCorpusWPM(base_model_params.SingleTaskModelParams):\n", - " \"\"\"Brown Corpus data with a Word-Piece Model tokenizer.\"\"\"\n", - "\n", - " # Generated using\n", - " # lingvo/tasks/punctuator/tools:download_brown_corpus.\n", - " _DATADIR = '/tmp/punctuator_data'\n", - " _VOCAB_FILE = 'brown_corpus_wpm.16000.vocab'\n", - " # _VOCAB_SIZE needs to be a multiple of 16 because we use a sharded softmax\n", - " # with 16 shards.\n", - " _VOCAB_SIZE = 16000\n", - "\n", - " def Train(self):\n", - " p = input_generator.PunctuatorInput.Params()\n", - " p.file_pattern = 'text:' + os.path.join(self._DATADIR, 'train.txt')\n", - " p.file_random_seed = 0 # Do not use a fixed seed.\n", - " p.file_parallelism = 1 # We only have a single input file.\n", - "\n", - " # The bucket upper bound specifies how to split the input into buckets. We\n", - " # train on sequences up to maximum bucket size and discard longer examples.\n", - " p.bucket_upper_bound = [10, 20, 30, 60, 120]\n", - "\n", - " # The bucket batch limit determines how many examples are there in each\n", - " # batch during training. We reduce the batch size for the buckets that\n", - " # have higher upper bound (batches that consist of longer sequences)\n", - " # in order to prevent out of memory issues.\n", - " # Note that this hyperparameter varies widely based on the model and\n", - " # language. Larger models may warrant smaller batches in order to fit in\n", - " # memory, for example; and ideographical languages like Chinese may benefit\n", - " # from more buckets.\n", - " p.bucket_batch_limit = [512, 256, 160, 80, 40]\n", - "\n", - " p.tokenizer.vocab_filepath = self._VOCAB_FILE\n", - " p.tokenizer.vocab_size = self._VOCAB_SIZE\n", - " p.tokenizer.pad_to_max_length = False\n", - "\n", - " # Set the tokenizer max length slightly longer than the largest bucket to\n", - " # discard examples that are longer than we allow.\n", - " p.source_max_length = p.bucket_upper_bound[-1] + 2\n", - " p.target_max_length = p.bucket_upper_bound[-1] + 2\n", - " return p\n", - "\n", - " # There is also a Dev method for dev set params, but we don't have a dev set.\n", - " def Test(self):\n", - " p = input_generator.PunctuatorInput.Params()\n", - " p.file_pattern = 'text:' + os.path.join(self._DATADIR, 'test.txt')\n", - " p.file_random_seed = 27182818 # Fix random seed for testing.\n", - " # The following two parameters are important if there's more than one input\n", - " # file. For this codelab it doesn't actually matter.\n", - " p.file_parallelism = 1 # Avoid randomness in testing.\n", - " # In order to make exactly one pass over the dev/test sets, we set buffer\n", - " # size to 1. Greater numbers may cause inaccurate dev/test scores.\n", - " p.file_buffer_size = 1\n", - "\n", - " p.bucket_upper_bound = [10, 20, 30, 60, 120, 200]\n", - " p.bucket_batch_limit = [16] * 4 + [4] * 2\n", - "\n", - " p.tokenizer.vocab_filepath = self._VOCAB_FILE\n", - " p.tokenizer.vocab_size = self._VOCAB_SIZE\n", - " p.tokenizer.pad_to_max_length = False\n", - "\n", - " p.source_max_length = p.bucket_upper_bound[-1] + 2\n", - " p.target_max_length = p.bucket_upper_bound[-1] + 2\n", - " return p\n", - "\n", - "\n", - "# This decorator registers the model in the Lingvo model registry.\n", - "# This file is lingvo/tasks/punctuator/params/codelab.py,\n", - "# so the model will be registered as punctuator.codelab.RNMTModel.\n", - "@model_registry.RegisterSingleTaskModel\n", - "class RNMTModel(BrownCorpusWPM):\n", - " \"\"\"RNMT+ Model.\"\"\"\n", - "\n", - " def Task(self):\n", - " p = base_config.SetupRNMTParams(\n", - " model.RNMTModel.Params(),\n", - " name='punctuator_rnmt',\n", - " vocab_size=self._VOCAB_SIZE,\n", - " embedding_dim=1024,\n", - " hidden_dim=1024,\n", - " num_heads=4,\n", - " num_encoder_layers=6,\n", - " num_decoder_layers=8,\n", - " learning_rate=1e-4,\n", - " l2_regularizer_weight=1e-5,\n", - " lr_warmup_steps=500,\n", - " lr_decay_start=400000,\n", - " lr_decay_end=1200000,\n", - " lr_min=0.5,\n", - " ls_uncertainty=0.1,\n", - " atten_dropout_prob=0.3,\n", - " residual_dropout_prob=0.3,\n", - " adam_beta2=0.98,\n", - " adam_epsilon=1e-6,\n", - " )\n", - " p.eval.samples_per_summary = 2466\n", - " return p" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "ooJYcOA4tcXW" - }, - "source": [ - "## Model Training\n", - "\n", - "The following cell trains the model. Note that this will require approximately 2.5GB of space in `logdir`." - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "91eOF9Sqy1iG" - }, - "outputs": [], - "source": [ - "# Start tensorboard (access at http://localhost:6006)\n", - "import os\n", - "os.system('lsof -t -i:6006 || tensorboard --logdir=/tmp/punctuator \u0026')\n", - "\n", - "!python3 -m lingvo.trainer --model=codelab.RNMTModel --mode=sync --logdir=/tmp/punctuator --saver_max_to_keep=2 --noenable_asserts --run_locally=gpu" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "sOQMKZiuwNKr" - }, - "source": [ - "The following cell evaluates the model. In Lingvo, evaluation is meant to be run alongside training as a separate process that periodically looks for the latest checkpoint and evaluates it. There is only one process in Colab so running this cell will evaluate the current checkpoint then it will block indefinitely waiting for the next checkpoint." - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "leWsSPQ6L__H" - }, - "outputs": [], - "source": [ - "!python3 -m lingvo.trainer --model=codelab.RNMTModel --job=evaler_test --logdir=/tmp/punctuator --run_locally=cpu" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "wwqXVIaG2hA_" - }, - "source": [ - "There is also a Decoder job that can be run the same way. The difference between the Evaler and Decoder varies by model." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "g1V7f-Oxw9vp" - }, - "source": [ - "## Model Inference\n", - "\n", - "After the model has been trained for around 10-20k steps (a few hours on GPU), its inference graph can be used to interact with the model using arbitrary inputs." - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "cellView": "both", - "colab": {}, - "colab_type": "code", - "id": "7KU0zeUTKS9Z" - }, - "outputs": [], - "source": [ - "import string\n", - "\n", - "from lingvo import compat as tf\n", - "from lingvo import model_imports\n", - "from lingvo import model_registry\n", - "from lingvo.core import inference_graph_exporter\n", - "from lingvo.core import predictor\n", - "from lingvo.core.ops.hyps_pb2 import Hypothesis\n", - "\n", - "tf.flags.FLAGS.mark_as_parsed()\n", - "\n", - "\n", - "src = \"she asked do you know the way to san jose\" #@param {type:'string'}\n", - "src = src.lower().translate(str.maketrans('', '', string.punctuation))\n", - "print(src)\n", - "\n", - "checkpoint = tf.train.latest_checkpoint('/tmp/punctuator/train')\n", - "print('Using checkpoint %s' % checkpoint)\n", - "\n", - "# Run inference\n", - "params = model_registry.GetParams('codelab.RNMTModel', 'Test')\n", - "inference_graph = inference_graph_exporter.InferenceGraphExporter.Export(params)\n", - "pred = predictor.Predictor(inference_graph, \n", - " checkpoint=checkpoint, \n", - " device_type='cpu')\n", - "src_ids, decoded, scores, hyps = pred.Run(\n", - " ['src_ids', 'topk_decoded', 'topk_scores', 'topk_hyps'], src_strings=[src])\n", - "# print(src_ids[0])\n", - "for text, score in zip(decoded[0].tolist(), scores[0].tolist()):\n", - " print(\"%.5f: %s\" % (score, text))\n", - "# for i, hyp in enumerate(hyps[0]):\n", - "# print(\"=======hyp %d=======\" % i)\n", - "# print(Hypothesis().FromString(hyp))" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "vBDiPV07mFmX" - }, - "source": [ - "Footnote: One might wonder why our result places the question mark outside the quotation. This happens because the Brown corpus follows a 1959 US Patent Office precedure of transliterating texts into punch cards, where the closing question mark is always punched before the punctuation, so that the punctuation mark occurs at the end of the sentence. See [this link](http://clu.uni.no/icame/manuals/BROWN/INDEX.HTM) for more details. Our result is just following this pattern in our training data." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "_hW7aTI8fM4O" - }, - "source": [ - "## Additional Resources\n", - "\n", - "For more advanced topics or to get a deeper understanding of Lingvo beyond this codelab, see the [paper](https://arxiv.org/abs/1902.08295)." - ] - } - ], - "metadata": { - "colab": { - "collapsed_sections": [], - "name": "Introduction to Lingvo", - "provenance": [], - "toc_visible": true - }, - "kernelspec": { - "display_name": "Python 3", - "name": "python3" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} diff --git a/docker/dev.dockerfile b/docker/dev.dockerfile deleted file mode 100644 index ac3eeea1..00000000 --- a/docker/dev.dockerfile +++ /dev/null @@ -1,115 +0,0 @@ -# Run the following commands in order: -# -# LINGVO_DIR="/tmp/lingvo" # (change to the cloned lingvo directory, e.g. "$HOME/lingvo") -# LINGVO_DEVICE="gpu" # (Leave empty to build and run CPU only docker) -# docker build --tag tensorflow:lingvo $(test "$LINGVO_DEVICE" = "gpu" && echo "--build-arg base_image=nvidia/cuda:10.0-cudnn7-runtime-ubuntu18.04") - < "$LINGVO_DIR/docker/dev.dockerfile" -# docker run --rm $(test "$LINGVO_DEVICE" = "gpu" && echo "--runtime=nvidia") -it -v ${LINGVO_DIR}:/tmp/lingvo -v ${HOME}/.gitconfig:/home/${USER}/.gitconfig:ro -p 6006:6006 -p 8888:8888 --name lingvo tensorflow:lingvo bash -# -# Test that everything worked: -# -# bazel test -c opt --test_output=streamed //lingvo:trainer_test //lingvo:models_test - -ARG cpu_base_image="ubuntu:18.04" -ARG base_image=$cpu_base_image -FROM $base_image - -LABEL maintainer="Lingvo team " - -# Re-declare args because the args declared before FROM can't be used in any -# instruction after a FROM. -ARG cpu_base_image="ubuntu:18.04" -ARG base_image=$cpu_base_image - -# Pick up some TF dependencies -RUN apt-get update && apt-get install -y --no-install-recommends software-properties-common -RUN apt-get update && apt-get install -y --no-install-recommends \ - aria2 \ - build-essential \ - curl \ - dirmngr \ - git \ - gpg-agent \ - less \ - libfreetype6-dev \ - libhdf5-serial-dev \ - libpng-dev \ - libzmq3-dev \ - lsof \ - pkg-config \ - rename \ - rsync \ - sox \ - unzip \ - vim \ - && \ - apt-get clean && \ - rm -rf /var/lib/apt/lists/* - -# Install python 3.7 -RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BA6932366A755776 -RUN echo "deb http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic main" > /etc/apt/sources.list.d/deadsnakes-ppa-bionic.list -RUN apt-get update && apt-get install -y python3.7 -RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 1000 -# bazel assumes the python executable is "python". -RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.7 1000 - -RUN curl -O https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py && rm get-pip.py - -ARG bazel_version=0.28.1 -# This is to install bazel, for development purposes. -ENV BAZEL_VERSION ${bazel_version} -RUN mkdir /bazel && \ - cd /bazel && \ - curl -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" -fSsL -O https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh && \ - curl -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" -fSsL -o /bazel/LICENSE.txt https://raw.githubusercontent.com/bazelbuild/bazel/master/LICENSE && \ - chmod +x bazel-*.sh && \ - ./bazel-$BAZEL_VERSION-installer-linux-x86_64.sh && \ - cd / && \ - rm -f /bazel/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh - -ARG pip_dependencies=' \ - apache-beam[gcp]>=2.8 \ - contextlib2 \ - dataclasses \ - google-api-python-client \ - h5py \ - ipykernel \ - jupyter \ - jupyter_http_over_ws \ - matplotlib \ - model-pruning-google-research \ - numpy \ - oauth2client \ - pandas \ - Pillow \ - pyyaml \ - recommonmark \ - scikit-learn==0.20.3 \ - scipy \ - sklearn \ - sphinx \ - sphinx_rtd_theme \ - sympy \ - waymo-open-dataset-tf-2-2-0' - -RUN pip3 --no-cache-dir install $pip_dependencies -RUN python3 -m ipykernel.kernelspec - -# The latest tensorflow requires CUDA 10 compatible nvidia drivers (410.xx). -# If you are unable to update your drivers, an alternative is to compile -# tensorflow from source instead of installing from pip. -# Ensure we install the correct version by uninstalling first. -RUN pip3 uninstall -y tensorflow tensorflow-gpu tf-nightly tf-nightly-gpu -RUN pip3 --no-cache-dir install tensorflow-gpu==2.2.0 - -RUN jupyter serverextension enable --py jupyter_http_over_ws - -# TensorBoard -EXPOSE 6006 - -# Jupyter -EXPOSE 8888 - -WORKDIR "/tmp/lingvo" - -CMD ["/bin/bash"] diff --git a/docker/lib.dockerfile b/docker/lib.dockerfile deleted file mode 100644 index b28c9550..00000000 --- a/docker/lib.dockerfile +++ /dev/null @@ -1,51 +0,0 @@ -# CPU only: -# docker build --tag tensorflow:lingvo_lib - < docker/lib.dockerfile -# docker run --rm -it -p 6006:6006 -p 8888:8888 --name lingvo tensorflow:lingvo_lib bash -# -# With GPU support: -# docker build --tag tensorflow:lingvo_lib_gpu --build-arg base_image=nvidia/cuda:10.0-cudnn7-runtime-ubuntu18.04 - < docker/lib.dockerfile -# docker run --runtime=nvidia --rm -it -p 6006:6006 -p 8888:8888 --name lingvo tensorflow:lingvo_lib_gpu bash - -ARG cpu_base_image="ubuntu:18.04" -ARG base_image=$cpu_base_image -FROM $base_image - -LABEL maintainer="Lingvo Bot " - -# Re-declare args because the args declared before FROM can't be used in any -# instruction after a FROM. -ARG cpu_base_image="ubuntu:18.04" -ARG base_image=$cpu_base_image - -# Pick up some TF dependencies -RUN apt-get update && apt-get install -y --no-install-recommends software-properties-common -RUN apt-get update && apt-get install -y --no-install-recommends \ - build-essential \ - curl \ - git \ - less \ - lsof \ - pkg-config \ - python3-distutils \ - rsync \ - sox \ - unzip \ - vim \ - && \ - apt-get clean && \ - rm -rf /var/lib/apt/lists/* - -RUN curl -O https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py && rm get-pip.py - -RUN pip3 --no-cache-dir install lingvo - -RUN python3 -m ipykernel.kernelspec -RUN jupyter serverextension enable --py jupyter_http_over_ws - -# TensorBoard -EXPOSE 6006 - -# Jupyter -EXPOSE 8888 - -CMD ["/bin/bash"] diff --git a/docker/run_distributed.py b/docker/run_distributed.py deleted file mode 100755 index 181ea5eb..00000000 --- a/docker/run_distributed.py +++ /dev/null @@ -1,223 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Runs code on a fleet of machines. - -This runs the lingvo code on a fleet of docker for demonstration and testing -purposes. We assume the following: -* There is a running container -* There is a shared volume in /sharedfs. In reality, this would be something -like an NFS or HDFS mount. - -The script is run on the host and only requires python and the docker binary -to be installed. - -We run two "clusters": one for training, and one for decoding. The trainer -jobs (controller, trainer_client/worker or trainer/ps) are connected to -each other, whereas the decoder jobs are independent, only reading from -the shared filesystem. The trainer jobs are configured via a cluster spec -flag, whereas the decoder jobs are configured with individual flags. -""" - -import os -from pipes import quote as shell_quote -import shutil -import subprocess -import sys - -_SYNC_TRAIN_CLUSTER_SPEC = { - "worker": [ - "worker0:43222", - "worker1:43222", - "worker2:43222", - ], - "controller": ["controller:43214",], - "trainer_client": ["trainer_client:24601"], -} - -_ASYNC_TRAIN_CLUSTER_SPEC = { - "trainer": [ - "trainer0:43222", - "trainer1:43222", - "trainer2:43222", - ], - "ps": [ - "ps0:43221", - "ps1:43221", - ], - "controller": ["controller:43214",], -} - -DECODE_CLUSTER_SPEC = { - "evaler_test": ["evaler_test:23487"], - "decoder_test": ["decoder_test:24679"], -} - -MODEL = "image.mnist.LeNet5" -DATADIR = "/tmp/mnist" -TRAIN_MODE = "sync" - -TRAIN_CLUSTER_SPEC = ( - _SYNC_TRAIN_CLUSTER_SPEC - if TRAIN_MODE == "sync" else _ASYNC_TRAIN_CLUSTER_SPEC) - -DOCKER_BIN = "/usr/bin/docker" -# All that is required is that we have pip installed tensorflow. -DOCKER_IMAGE_NAME = "tensorflow:lingvo" -# This was created using -# bazel build -c opt //lingvo:trainer.par -# cp bazel-bin/lingvo/trainer.par . -# Since /tmp/lingvo is mounted, we can see it. -# TODO(drpng): hard-wiring below. -TRAINER_PACKAGE = "/tmp/lingvo/trainer.par" -DRY_RUN = False -NETWORK_NAME = "tf-net" - -SHARED_FS_MOUNTPOINT = "/tmp/sharedfs" - - -def _RunDocker(args): - print("Running: docker %s" % args) - if DRY_RUN: - return 0 - ret = subprocess.call([DOCKER_BIN] + args) - return ret - - -def _RunDockerOrDie(args): - ret = _RunDocker(args) - if ret != 0: - sys.stderr.write("Failed to run: %s\n" % ret) - sys.stderr.flush() - sys.exit(ret) - - -def _ExecInDocker(container_name, - cmd_array, - workdir=None, - logfile=None, - detach=False): - """Execute in docker container.""" - if not workdir: - workdir = "/tmp" - opts = ["-t", "-w", workdir] - if detach: - opts += ["-d"] - # TODO(drpng): avoid quoting hell. - base_cmd = ["exec"] + opts + [container_name] - if logfile: - # The logfile is in the container. - cmd = " ".join(shell_quote(x) for x in cmd_array) - cmd += " >& %s" % logfile - full_cmd = base_cmd + ["bash", "-c", cmd] - else: - full_cmd = base_cmd + cmd_array - ret = _RunDocker(full_cmd) - if ret != 0: - sys.stderr.write( - "Failed to exec within %s: %s" % (container_name, cmd_array)) - sys.exit(ret) - - -def _Machine(machine_port): - # From host:port to host. - return machine_port[:machine_port.index(":")] - - -def Cleanup(): - specs = list(TRAIN_CLUSTER_SPEC.values()) + list(DECODE_CLUSTER_SPEC.values()) - for job_machines in specs: - machines = [_Machine(x) for x in job_machines] - _RunDocker(["stop", "-t", "0"] + machines) - _RunDocker(["network", "rm", NETWORK_NAME]) - shutil.rmtree(SHARED_FS_MOUNTPOINT, ignore_errors=True) - - -def InitFiles(): - os.mkdir(SHARED_FS_MOUNTPOINT, 0o1777) - # Create these directories so that we own them, not root. - os.mkdir(SHARED_FS_MOUNTPOINT + "/log", 0o1777) - os.mkdir(SHARED_FS_MOUNTPOINT + "/log/train", 0o1777) - os.mkdir(SHARED_FS_MOUNTPOINT + "/log/decoder_test", 0o1777) - os.mkdir(SHARED_FS_MOUNTPOINT + "/log/eval_test", 0o1777) - - -def InitNetwork(): - _RunDockerOrDie(["network", "create", "--driver", "bridge", NETWORK_NAME]) - - -def StartFleet(): - specs = list(TRAIN_CLUSTER_SPEC.values()) + list(DECODE_CLUSTER_SPEC.values()) - for job_machines in specs: - for machine_port in job_machines: - machine_name = _Machine(machine_port) - _RunDockerOrDie([ - "run", "--rm", "--name", machine_name, "-dit", "--network", - NETWORK_NAME, "-v", ":".join([SHARED_FS_MOUNTPOINT] * 2), "-v", - ":".join([DATADIR] * 2 + ["ro"]), DOCKER_IMAGE_NAME, "bash" - ]) - - -def MakeFlagClusterSpec(cluster_spec): - job_specs = [] - for job_name in sorted(cluster_spec.keys()): - job_specs += [job_name + "=" + ",".join(cluster_spec[job_name])] - flag_spec = "@".join(job_specs) - return flag_spec - - -def CopyTrainerToSharedMount(): - shutil.copy(TRAINER_PACKAGE, SHARED_FS_MOUNTPOINT + "/trainer.par") - - -def InstallAndStartProcess(cluster_spec): - """Unpacks the trainer and kick off training.""" - cluster_spec_flag = MakeFlagClusterSpec(cluster_spec) - for job_name, machines in cluster_spec.items(): - task_idx = 0 - for machine_port in machines: - machine_name = _Machine(machine_port) - _ExecInDocker( - machine_name, [ - os.path.join(SHARED_FS_MOUNTPOINT, "trainer.par"), - "--cluster_spec=%s" % cluster_spec_flag, - "--job=%s" % job_name, - "--task=%d" % task_idx, - "--mode=%s" % TRAIN_MODE, - "--logtostderr", - "--model=%s" % MODEL, - "--logdir=%s/log" % SHARED_FS_MOUNTPOINT, - ], - workdir="/tmp", - logfile="%s/%s.%d.log" % (SHARED_FS_MOUNTPOINT, job_name, task_idx), - detach=True) - task_idx += 1 - - -def main(): - Cleanup() - InitFiles() - InitNetwork() - StartFleet() - CopyTrainerToSharedMount() - InstallAndStartProcess(TRAIN_CLUSTER_SPEC) - for role in sorted(DECODE_CLUSTER_SPEC.keys()): - # Each decode process is its own spec. - machine_spec = DECODE_CLUSTER_SPEC[role] - InstallAndStartProcess({role: machine_spec}) - - -if __name__ == "__main__": - main() diff --git a/docs/apidoc/conf.py b/docs/apidoc/conf.py deleted file mode 100644 index 9edba272..00000000 --- a/docs/apidoc/conf.py +++ /dev/null @@ -1,217 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Configuration file for the Sphinx documentation builder. - -This file does only contain a selection of the most common options. For a -full list see the documentation: -http://www.sphinx-doc.org/en/master/config -""" - -# -- Path setup -------------------------------------------------------------- - -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. -# -# import os -# import sys -# sys.path.insert(0, u'/tmp/lingvo/lingvo') - -# -- Project information ----------------------------------------------------- - -project = u'Lingvo' -copyright = u'2018' -author = u'' - -# The short X.Y version -version = u'' -# The full version, including alpha/beta/rc tags -release = u'' - -# -- General configuration --------------------------------------------------- - -# If your documentation needs a minimal Sphinx version, state it here. -# -# needs_sphinx = '1.0' - -# Add any Sphinx extension module names here, as strings. They can be -# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom -# ones. -extensions = [ - 'sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinx.ext.mathjax', - 'sphinx.ext.napoleon', 'sphinx.ext.todo', 'sphinx.ext.viewcode' -] - -autodoc_default_flags = [ - 'members', 'undoc-members', 'private-members', 'show-inheritance' -] -autodoc_member_order = 'bysource' -napoleon_google_docstring = True -default_role = 'py:obj' -intersphinx_mapping = { - 'python': ('https://docs.python.org/3.7', None), - 'numpy': ('http://docs.scipy.org/doc/numpy/', None), -} - -# Add any paths that contain templates here, relative to this directory. -templates_path = ['_templates'] - - - -# The suffix(es) of source filenames. -# You can specify multiple suffix as a list of string: -from docutils.transforms import Transform -from recommonmark.parser import CommonMarkParser -source_parsers = { - '.md': CommonMarkParser, -} -source_suffix = ['.rst', '.md'] - -# The master toctree document. -master_doc = 'index' - -# The language for content autogenerated by Sphinx. Refer to documentation -# for a list of supported languages. -# -# This is also used if you do content translation via gettext catalogs. -# Usually you set "language" from the command line for these cases. -language = 'en' - -# List of patterns, relative to source directory, that match files and -# directories to ignore when looking for source files. -# This pattern also affects html_static_path and html_extra_path . -exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] - -# The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'sphinx' - -# -- Options for HTML output ------------------------------------------------- - -# The theme to use for HTML and HTML Help pages. See the documentation for -# a list of builtin themes. -# -html_theme = 'sphinx_rtd_theme' - -# Theme options are theme-specific and customize the look and feel of a theme -# further. For a list of options available for each theme, see the -# documentation. -# -# html_theme_options = {} - -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = [] - -# Custom sidebar templates, must be a dictionary that maps document names -# to template names. -# -# The default sidebars (for documents that don't match any pattern) are -# defined by theme itself. Builtin themes are using these templates by -# default: ``['localtoc.html', 'relations.html', 'sourcelink.html', -# 'searchbox.html']``. -# -# html_sidebars = {} - -# -- Options for HTMLHelp output --------------------------------------------- - -# Output file base name for HTML help builder. -htmlhelp_basename = 'lingvodoc' - -# -- Options for LaTeX output ------------------------------------------------ - -latex_elements = { - # The paper size ('letterpaper' or 'a4paper'). - # - # 'papersize': 'letterpaper', - - # The font size ('10pt', '11pt' or '12pt'). - # - # 'pointsize': '10pt', - - # Additional stuff for the LaTeX preamble. - # - # 'preamble': '', - - # Latex figure (float) alignment - # - # 'figure_align': 'htbp', -} - -# Grouping the document tree into LaTeX files. List of tuples -# (source start file, target name, title, -# author, documentclass [howto, manual, or own class]). -latex_documents = [ - (master_doc, 'lingvo.tex', u'Lingvo Documentation', u'', 'manual'), -] - -# -- Options for manual page output ------------------------------------------ - -# One entry per manual page. List of tuples -# (source start file, name, description, authors, manual section). -man_pages = [(master_doc, 'lingvo', u'Lingvo Documentation', [author], 1)] - -# -- Options for Texinfo output ---------------------------------------------- - -# Grouping the document tree into Texinfo files. List of tuples -# (source start file, target name, title, author, -# dir menu entry, description, category) -texinfo_documents = [ - (master_doc, 'lingvo', u'Lingvo Documentation', author, 'Lingvo', - 'One line description of project.', 'Miscellaneous'), -] - -# -- Options for Epub output ------------------------------------------------- - -# Bibliographic Dublin Core info. -epub_title = project -epub_author = author -epub_publisher = author -epub_copyright = copyright - -# The unique identifier of the text. This can be a ISBN number -# or the project homepage. -# -# epub_identifier = '' - -# A unique identification for the text. -# -# epub_uid = '' - -# A list of files that should not be packed into the epub file. -epub_exclude_files = ['search.html'] - -# -- Extension configuration ------------------------------------------------- - -# -- Options for todo extension ---------------------------------------------- - -# If true, `todo` and `todoList` produce output, else they produce nothing. -todo_include_todos = True - - -class ResetFlags(Transform): - - default_priority = 999 - - def apply(self): - from absl import flags # pylint: disable=g-import-not-at-top - for flag in list(flags.FLAGS): - if flag not in ('showprefixforinfo',): - delattr(flags.FLAGS, flag) - - -def setup(app): - app.add_transform(ResetFlags) diff --git a/docs/apidoc/index.rst b/docs/apidoc/index.rst deleted file mode 100644 index 385c88c6..00000000 --- a/docs/apidoc/index.rst +++ /dev/null @@ -1,17 +0,0 @@ -.. lingvo documentation master file, created by - sphinx-quickstart on Thu Aug 30 16:01:44 2018. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -Welcome to lingvo's documentation! -================================== - -.. include:: modules.rst - - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` diff --git a/docs/generate_apidoc.sh b/docs/generate_apidoc.sh deleted file mode 100755 index d59839fc..00000000 --- a/docs/generate_apidoc.sh +++ /dev/null @@ -1,45 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -e -OUTDIR=/tmp/lingvo_apidoc - -# Run this from inside docker. -bazel build -c opt \ - //lingvo/core/ops:x_ops.so \ - //lingvo/tasks/car/ops:car_ops.so \ - //lingvo/core:inference_graph_pb2.py \ - //lingvo/core:hyperparams_pb2.py \ - //lingvo/core/ops:hyps_pb2.py \ - //lingvo/core/ops:record_py_pb2 \ - //lingvo/core/ops:versioned_file_set_py_pb2 \ - //lingvo/tasks/mt:text_input_pb2.py \ - 2>&1 -cp -f bazel-bin/lingvo/core/ops/x_ops.so lingvo/core/ops -cp -f bazel-bin/lingvo/tasks/car/ops/car_ops.so lingvo/tasks/car/ops -cp -f bazel-genfiles/lingvo/core/inference_graph_pb2.py lingvo/core -cp -f bazel-genfiles/lingvo/core/hyperparams_pb2.py lingvo/core -cp -f bazel-genfiles/lingvo/core/ops/hyps_pb2.py lingvo/core/ops -cp -f bazel-genfiles/lingvo/core/ops/record_pb2.py lingvo/core/ops -cp -f bazel-genfiles/lingvo/core/ops/versioned_file_set_pb2.py lingvo/core/ops -cp -f bazel-genfiles/lingvo/tasks/mt/text_input_pb2.py lingvo/tasks/mt -sphinx-apidoc -o "$OUTDIR" -efPM --implicit-namespaces lingvo/ $(find . -name '*_test.py') -cp docs/apidoc/{conf.py,index.rst} "$OUTDIR" -(export PYTHONPATH="$(pwd)" && cd "$OUTDIR" && sphinx-build -b html -T -j auto . build) -rm -f lingvo/core/{inference_graph_pb2.py,hyperparams_pb2.py} -rm -f lingvo/core/ops/{x_ops.so,hyps_pb2.py,record_pb2.py,versioned_file_set_pb2.py} -rm -f lingvo/tasks/car/ops/car_ops.so -rm -f lingvo/tasks/mt/text_input_pb2.py diff --git a/docs/single/build.sh b/docs/single/build.sh deleted file mode 100755 index 2ccd2e21..00000000 --- a/docs/single/build.sh +++ /dev/null @@ -1,35 +0,0 @@ -#!/bin/bash -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -cd "${0%/*}" # Change to script directory. - -if ! command -v sphinx-build &> /dev/null -then - echo "sphinx could not be found. Please install with 'pip3 install sphinx sphinx_rtd_theme recommonmark.'" - exit -fi - -cleanup() { - rm -rf single.py* _build __pycache__ - exit -} -trap cleanup ERR INT TERM - -echo "Paste the docstring to check (end with ctrl+D)." -cat > single.py - -sphinx-build -b singlehtml -q -T . _build -cleanup diff --git a/docs/single/conf.py b/docs/single/conf.py deleted file mode 100644 index cd99c06b..00000000 --- a/docs/single/conf.py +++ /dev/null @@ -1,52 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Configuration file for the Sphinx documentation builder.""" - -import os -import sys -from recommonmark.parser import CommonMarkParser - -# enable autodoc to load local modules -sys.path.insert(0, os.path.abspath('.')) - -project = 'Lingvo' -copyright = '2020' # pylint: disable=redefined-builtin -author = '' -extensions = [ - 'sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinx.ext.mathjax', - 'sphinx.ext.napoleon', 'sphinx.ext.todo', 'sphinx.ext.viewcode' -] -autodoc_default_flags = [ - 'members', 'undoc-members', 'private-members', 'show-inheritance' -] -autodoc_member_order = 'bysource' -napoleon_google_docstring = True -default_role = 'py:obj' -intersphinx_mapping = { - 'python': ('https://docs.python.org/3.7', None), - 'numpy': ('http://docs.scipy.org/doc/numpy/', None), -} -templates_path = ['_templates'] -source_parsers = { - '.md': CommonMarkParser, -} -source_suffix = ['.rst', '.md'] -exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] -pygments_style = 'sphinx' -html_theme = 'sphinx_rtd_theme' -html_static_path = [] -html_theme_options = {'nosidebar': True} -todo_include_todos = True diff --git a/docs/single/index.rst b/docs/single/index.rst deleted file mode 100644 index b8de17c4..00000000 --- a/docs/single/index.rst +++ /dev/null @@ -1,5 +0,0 @@ -single -========= - -.. automodule:: single - :members: diff --git a/lingvo/__init__.py b/galvasr2/__init__.py similarity index 100% rename from lingvo/__init__.py rename to galvasr2/__init__.py diff --git a/lingvo/core/__init__.py b/galvasr2/align/__init__.py similarity index 100% rename from lingvo/core/__init__.py rename to galvasr2/align/__init__.py diff --git a/galvasr2/align/audio.py b/galvasr2/align/audio.py index ec020403..08905ebc 100644 --- a/galvasr2/align/audio.py +++ b/galvasr2/align/audio.py @@ -7,7 +7,7 @@ import numpy as np from webrtcvad import Vad -from utils import LimitingPool +from galvasr2.align.utils import LimitingPool DEFAULT_RATE = 16000 DEFAULT_CHANNELS = 1 diff --git a/galvasr2/align/dsalign_main.py b/galvasr2/align/dsalign_main.py index 7a6e16cd..fef2cf81 100644 --- a/galvasr2/align/dsalign_main.py +++ b/galvasr2/align/dsalign_main.py @@ -2,20 +2,18 @@ import json import logging import argparse -import deepspeech import subprocess import os.path as path import numpy as np import textdistance import multiprocessing from collections import Counter -from search import FuzzySearch +from galvasr2.align.search import FuzzySearch from glob import glob -from text import Alphabet, TextCleaner, levenshtein, similarity -from utils import enweight, log_progress -from audio import DEFAULT_RATE, read_frames_from_file, vad_split -from generate_lm import convert_and_filter_topk, build_lm -from generate_package import create_bundle +from galvasr2.align.text import Alphabet, TextCleaner, levenshtein, similarity +from galvasr2.align.utils import enweight, log_progress +from galvasr2.align.audio import DEFAULT_RATE, read_frames_from_file, vad_split +from galvasr2.align.generate_lm import convert_and_filter_topk, build_lm BEAM_WIDTH = 500 LM_ALPHA = 1 @@ -80,13 +78,7 @@ def read_script(script_path): def init_stt(output_graph_path, scorer_path): - # global model? How are we supposed to do multiprocessing then? - global model - model = deepspeech.Model(output_graph_path) - # It definitely seems very reasonable to adapt the - # ctc_decoder_with_kenlm tensorflow operator to use this... - model.enableExternalScorer(scorer_path) - logging.debug("Process {}: Loaded models".format(os.getpid())) + raise Exception("Dead code") def stt(sample): diff --git a/galvasr2/align/search.py b/galvasr2/align/search.py index 3e7a30e0..9f0079c0 100644 --- a/galvasr2/align/search.py +++ b/galvasr2/align/search.py @@ -1,5 +1,5 @@ from collections import Counter -from text import ngrams, similarity +from galvasr2.align.text import ngrams, similarity from galvasr2.align.smith_waterman import sw_align as sw_align_sped_up diff --git a/lingvo/core/steps/__init__.py b/galvasr2/align/spark/__init__.py similarity index 100% rename from lingvo/core/steps/__init__.py rename to galvasr2/align/spark/__init__.py diff --git a/galvasr2/align/spark/align_cuda_decoder.py b/galvasr2/align/spark/align_cuda_decoder.py index ccd0c241..baaf7117 100644 --- a/galvasr2/align/spark/align_cuda_decoder.py +++ b/galvasr2/align/spark/align_cuda_decoder.py @@ -4,6 +4,7 @@ from copy import deepcopy from concurrent.futures import ThreadPoolExecutor +import glob import shlex import subprocess import logging @@ -18,8 +19,8 @@ import pyspark import pyspark.sql.functions as F import pyspark.sql.types as T -import tensorflow as tf +import galvasr2 from galvasr2.align.spark.align_lib import ( create_audio_segments_udf, create_audio_segment_names_udf, @@ -35,11 +36,12 @@ from galvasr2.align.spark.dsalign_lib import prepare_align_udf from galvasr2.align.spark.schemas import ARCHIVE_ORG_SCHEMA from galvasr2.utils import find_runfiles -import dsalign_main +from galvasr2.align import dsalign_main FLAGS = flags.FLAGS flags.DEFINE_integer("stage", 2, "") +flags.DEFINE_integer("end_stage", sys.maxsize, "") flags.DEFINE_string( "input_catalogue", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS.jsonl.gz", @@ -145,14 +147,19 @@ def main(argv): mem_bytes = os.sysconf("SC_PAGE_SIZE") * os.sysconf( "SC_PHYS_PAGES" ) # e.g. 4015976448 - mem_gib = int((mem_bytes / (1024.0 ** 3)) * 0.9) - tar_jar = os.path.join( - find_runfiles(), "__main__/galvasr2/spark/tar_spark_datasource.jar" - ) + mem_gib = int((mem_bytes / (1024.0**3)) * 0.9) + (jar_path,) = galvasr2.__path__ + jars = ",".join(glob.glob(os.path.join(jar_path, "*.jar"))) + print("GALVEZ:jars=", jars) + os.makedirs("/tmp/spark-events", exist_ok=True) spark = ( - pyspark.sql.SparkSession.builder.master(f"local[{os.cpu_count() - 1}]") + pyspark.sql.SparkSession.builder.master(f"local[{os.cpu_count()}]") .config("spark.eventLog.enabled", "true") - .config("spark.eventLog.dir", "/spark-events") + # .config("spark.eventLog.dir", "/spark-events") + .config( + "spark.hadoop.fs.AbstractFileSystem.gs.impl", + "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS", + ) .config("spark.sql.execution.arrow.pyspark.enabled", "true") .config( "spark.driver.extraJavaOptions", @@ -163,9 +170,9 @@ def main(argv): "-Dio.netty.tryReflectionSetAccessible=true", ) .config("spark.driver.memory", f"{mem_gib}g") - .config("spark.history.fs.logDirectory", "/spark-events") + # .config("spark.history.fs.logDirectory", "/spark-events") .config("spark.sql.execution.arrow.maxRecordsPerBatch", "1") - .config("spark.jars", tar_jar) + .config("spark.jars", jars) .config("spark.local.dir", "/mnt/disks/spark-scratch/") .getOrCreate() ) @@ -173,6 +180,28 @@ def main(argv): logging.getLogger("py4j").setLevel(logging.ERROR) catalogue_df = load_audio_id_text_id_mapping(spark, FLAGS.input_catalogue) + audio_paths = F.concat( + F.lit(FLAGS.input_gcs_path), + F.lit("/"), + F.col("identifier"), + F.lit("/"), + F.col("audio_document_id"), + ) + srt_paths = F.concat( + F.lit(FLAGS.input_gcs_path), + F.lit("/"), + F.col("identifier"), + F.lit("/"), + F.col("text_document_id"), + ) + temp_catalogue_df = catalogue_df.withColumn("audio_paths", audio_paths).withColumn( + "srt_paths", srt_paths + ) + # print("GALVEZ:schema=") + # temp_catalogue_df.printSchema() + temp_catalogue_df.toPandas().to_json( + "audio_id_text_id_mapping.json", orient="records", lines=True + ) _, licenseurl_df = load_audio_and_text_dfs(spark, FLAGS.input_catalogue) licenseurl_df = licenseurl_df.select( @@ -194,7 +223,7 @@ def main(argv): os.makedirs(FLAGS.work_dir, exist_ok=True) wav_scp = os.path.join(FLAGS.work_dir, "wav.scp") ctm_out_dir = os.path.join(FLAGS.work_dir, "decoder_ctm_dir") - if FLAGS.stage <= 0: + if FLAGS.stage <= 0 and FLAGS.end_stage >= 0: catalogue_df = catalogue_df.cache() # catalogue_df.write.mode("overwrite").format("csv").options(header="true").save(key_int_mapping) training_sample_rows = catalogue_df.collect() @@ -223,7 +252,7 @@ def main(argv): # Can get 266x RTF with this configuration. Keep it? # bath size of 100 and num channels of 100 works just fine - if FLAGS.stage <= 1: + if FLAGS.stage <= 1 and FLAGS.end_stage >= 1: if not FLAGS.work_dir.startswith("gs://"): os.makedirs(ctm_out_dir, exist_ok=True) with TemporaryMountDirectory( @@ -282,7 +311,7 @@ def run_gpu(posix_wav_scp_shard, gpu_number): executor.shutdown(wait=True) alignments_dir = os.path.join(FLAGS.alignments_work_dir, "alignments_json_jul_28") - if FLAGS.stage <= 2: + if FLAGS.stage <= 2 and FLAGS.end_stage >= 2: # TODO: Add options to DSAlign here dsalign_args = dsalign_main.parse_args( ["--output-wer", "--output-cer"] @@ -342,7 +371,7 @@ def run_gpu(posix_wav_scp_shard, gpu_number): manifest_dir = os.path.join(FLAGS.work_dir, "dataset_manifest") tars_dir = os.path.join(FLAGS.work_dir, "dataset_tars") - if FLAGS.stage <= 3: + if FLAGS.stage <= 3 and FLAGS.end_stage >= 3: duplicate_data_path = "gs://the-peoples-speech-west-europe/forced-aligner/data_deduplication/data_deduplication_v2_lines.json" duplicates_df = spark.read.format("json").load(duplicate_data_path) @@ -559,7 +588,7 @@ def run_gpu(posix_wav_scp_shard, gpu_number): repartitioned_tars_dir = os.path.join(FLAGS.work_dir, "repartitioned_dataset_tars") tmp_tars_dir = os.path.join(FLAGS.work_dir, "repartitioned_dataset_tmp_dir") - if FLAGS.stage <= 4: + if FLAGS.stage <= 4 and FLAGS.end_stage >= 4: tars_df = spark.read.format("tar").load(tars_dir) # .limit(100) number_of_rows = tars_df.count() @@ -617,7 +646,7 @@ def run_gpu(posix_wav_scp_shard, gpu_number): FLAGS.work_dir, "dataset_manifest_nemo_single" ) - if FLAGS.stage <= 5: + if FLAGS.stage <= 5 and FLAGS.end_stage >= 5: json_df = spark.read.format("json").load(manifest_dir) nemo_df = json_df.select( F.explode( @@ -654,7 +683,7 @@ def run_gpu(posix_wav_scp_shard, gpu_number): single_manifest_dir = os.path.join(FLAGS.work_dir, "dataset_manifest_single") single_tar_dir = os.path.join(FLAGS.work_dir, "dataset_tars_single") # Create single tar file and single json file - if FLAGS.stage <= 6: + if FLAGS.stage <= 6 and FLAGS.end_stage >= 6: json_df = spark.read.format("json").load(manifest_dir) json_df.coalesce(1).write.format("json").mode("overwrite").save( single_manifest_dir @@ -666,16 +695,3 @@ def run_gpu(posix_wav_scp_shard, gpu_number): if __name__ == "__main__": app.run(main) - - -# GALVEZ: [ -# Row(key='\x1d\x0c\x00\x00`\x00\x00\x00urts_ca9_04-55168-gov_uscourts_ca9_04-55168_2005-07-11_mp3_0.flac'), -# Row(key='�\x03\x00\x00`\x00\x00\x00urts_ca9_04-55168-gov_uscourts_ca9_04-55168_2005-07-11_mp3_1.flac'), -# Row(key='X\x05\x00\x00`\x00\x00\x00urts_ca9_04-55168-gov_uscourts_ca9_04-55168_2005-07-11_mp3_2.flac'), -# Row(key='�\x01\x00\x00`\x00\x00\x00urts_ca9_04-55168-gov_uscourts_ca9_04-55168_2005-07-11_mp3_3.flac'), -# Row(key='�\r\x00\x00`\x00\x00\x00urts_ca9_04-55168-gov_uscourts_ca9_04-55168_2005-07-11_mp3_4.flac'), -# Row(key='�\x03\x00\x00`\x00\x00\x00urts_ca9_04-55168-gov_uscourts_ca9_04-55168_2005-07-11_mp3_5.flac'), -# Row(key='&\x04\x00\x00`\x00\x00\x00urts_ca9_04-55168-gov_uscourts_ca9_04-55168_2005-07-11_mp3_6.flac'), -# Row(key='�\x00\x00\x00`\x00\x00\x00urts_ca9_04-55168-gov_uscourts_ca9_04-55168_2005-07-11_mp3_7.flac'), -# Row(key='�\x07\x00\x00`\x00\x00\x00urts_ca9_04-55168-gov_uscourts_ca9_04-55168_2005-07-11_mp3_8.flac'), -# Row(key=';\x0e\x00\x00`\x00\x00\x00urts_ca9_04-55168-gov_uscourts_ca9_04-55168_2005-07-11_mp3_9.flac')] diff --git a/galvasr2/align/spark/align_lib.py b/galvasr2/align/spark/align_lib.py index b8c8b7b3..d728fa20 100644 --- a/galvasr2/align/spark/align_lib.py +++ b/galvasr2/align/spark/align_lib.py @@ -16,7 +16,6 @@ from typing import List, Tuple import wave -import ds_ctcdecoder from ftfy import fix_text, guess_bytes import langid import numpy as np @@ -25,11 +24,10 @@ from pydub import AudioSegment import pyspark from matching.games import HospitalResident -import tensorflow as tf +import srt import tqdm import re -import srt from pyspark.sql.functions import col, pandas_udf import pyspark.sql.types as T @@ -317,109 +315,6 @@ def vad( GENERATE_LM_OUTPUT_SCHEMA = StructType([StructField("path", StringType())]) -def prepare_generate_lm_udf(kenlm_path: str, debug_work_dir: str, alphabet_path: str): - # @pandas_udf(GENERATE_LM_OUTPUT_SCHEMA) - # TODO: Need to sort the log_probabilities by int64_uttid (right?) - def generate_lm( - grouping_key: Tuple[np.str, np.str], data_df: pd.DataFrame - ) -> pd.DataFrame: - ( - identifier, - text_document_id, - ) = grouping_key - identifier = str(identifier) - text_document_id = str(text_document_id) - - transcript = data_df.transcript[0] - with tempfile.NamedTemporaryFile("w+t", dir=debug_work_dir) as input_txt: - input_txt.write(transcript) - input_txt.flush() - os.makedirs(os.path.join(debug_work_dir, identifier), exist_ok=True) - scorer_path = os.path.join( - debug_work_dir, identifier, text_document_id + ".scorer" - ) - data_lower, vocab_str = convert_and_filter_topk( - scorer_path, input_txt.name, 500000 - ) - build_lm( - scorer_path, - kenlm_path, - 5, - "85%", - "0|0|1", - True, - 255, - 8, - "trie", - data_lower, - vocab_str, - ) - os.remove(scorer_path + "." + "lower.txt.gz") - os.remove(scorer_path + "." + "lm.arpa") - os.remove(scorer_path + "." + "lm_filtered.arpa") - - create_bundle( - alphabet_path, - scorer_path + "." + "lm.binary", - scorer_path + "." + "vocab-500000.txt", - scorer_path, - False, - 0.931289039105002, - 1.1834137581510284, - ) - os.remove(scorer_path + "." + "lm.binary") - os.remove(scorer_path + "." + "vocab-500000.txt") - - with open(alphabet_path) as fh: - num_output_symbols = len(fh.readlines()) + 1 - assert num_output_symbols == 32, f"GALVEZ:{num_output_symbols}" - transcripts = [] - - id_to_symbol = {} - with open(alphabet_path) as fh: - for i, line in enumerate(fh): - id_to_symbol[i] = line.rstrip() - id_to_symbol[31] = "blank" - - for row in data_df.itertuples(): - log_probabilities = row.log_probabilities.reshape(-1, num_output_symbols) - probabilities = np.exp(log_probabilities) - # np.exp(probabilities, out=probabilities) - np.testing.assert_allclose(probabilities.sum(axis=1), 1.0, atol=1e-3) - # simple_decoder_output = [] - # for t in range(probabilities.shape[0]): - # best = np.argmax(probabilities[t,:]) - # print(np.max(probabilities[t,:])) - # if (id_to_symbol[best] != "blank"): - # simple_decoder_output.append(id_to_symbol[best]) - - # print("GALVEZ simple output:", "".join(simple_decoder_output)) - - cutoff_prob = 1.0 - cutoff_top_n = 100 - scorer = ds_ctcdecoder.Scorer() - result = scorer.init( - scorer_path.encode("utf-8"), alphabet_path.encode("utf-8") - ) - scorer.set_utf8_mode(False) - assert result == 0, result - alphabet = ds_ctcdecoder.Alphabet() - result = alphabet.init(alphabet_path.encode("utf-8")) - assert not scorer.is_utf8_mode() - assert result == 0, result - scorer = None - outputs = ds_ctcdecoder.ctc_beam_search_decoder( - probabilities, alphabet, 100, cutoff_prob, cutoff_top_n, scorer - ) - print(f"GALVEZ:output={outputs[0][1]}") - print(f"GALVEZ:length={probabilities.shape[0] * 30. / 1000.}") - transcripts.append(outputs[0][1]) - - return pd.DataFrame({"path": pd.Series(transcripts)}) - - return generate_lm - - def load_audio_and_text_dfs(spark, input_catalogue_path: str): df = spark.read.format("json").schema(ARCHIVE_ORG_SCHEMA).load(input_catalogue_path) diff --git a/galvasr2/align/spark/align_lib_test.py b/galvasr2/align/spark/align_lib_test.py index 147e1b8a..e8a450ca 100644 --- a/galvasr2/align/spark/align_lib_test.py +++ b/galvasr2/align/spark/align_lib_test.py @@ -128,7 +128,7 @@ def setUpClass(cls): mem_bytes = os.sysconf("SC_PAGE_SIZE") * os.sysconf( "SC_PHYS_PAGES" ) # e.g. 4015976448 - mem_gib = int((mem_bytes / (1024.0 ** 3)) * 0.9) + mem_gib = int((mem_bytes / (1024.0**3)) * 0.9) tar_jar = os.path.join( find_runfiles(), "__main__/galvasr2/spark/tar_spark_datasource.jar" ) diff --git a/galvasr2/align/spark/dsalign_lib.py b/galvasr2/align/spark/dsalign_lib.py index ceda82e9..280756db 100644 --- a/galvasr2/align/spark/dsalign_lib.py +++ b/galvasr2/align/spark/dsalign_lib.py @@ -6,12 +6,12 @@ from typing import Dict, List, Tuple from collections import Counter -from search import FuzzySearch +from galvasr2.align.search import FuzzySearch import textdistance from glob import glob -from text import Alphabet, TextCleaner, levenshtein, similarity -from utils import enweight -from dsalign_main import ALGORITHMS, NAMED_NUMBERS +from galvasr2.align.text import Alphabet, TextCleaner, levenshtein, similarity +from galvasr2.align.utils import enweight +from galvasr2.align.dsalign_main import ALGORITHMS, NAMED_NUMBERS import logging diff --git a/galvasr2/align/stats.py b/galvasr2/align/stats.py index 32018d88..841b3204 100644 --- a/galvasr2/align/stats.py +++ b/galvasr2/align/stats.py @@ -6,7 +6,7 @@ from pickle import load, dump from collections import Counter from datetime import timedelta -from utils import log_progress +from galvasr2.align.utils import log_progress def fail(message, code=1): diff --git a/galvasr2/align/text.py b/galvasr2/align/text.py index 3fc2819f..7dae72e0 100644 --- a/galvasr2/align/text.py +++ b/galvasr2/align/text.py @@ -3,7 +3,7 @@ import codecs from six.moves import range from collections import Counter -from utils import enweight +from galvasr2.align.utils import enweight class Alphabet(object): diff --git a/galvasr2/gcs-connector-hadoop3-2.2.6-shaded.jar b/galvasr2/gcs-connector-hadoop3-2.2.6-shaded.jar new file mode 100644 index 00000000..7833ddfb --- /dev/null +++ b/galvasr2/gcs-connector-hadoop3-2.2.6-shaded.jar @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9b6f1be75928da71fe8926c8bc74b9a4e3570e4f12df3880ec79e586118b0e97 +size 33831057 diff --git a/galvasr2/tar_spark_datasource.jar b/galvasr2/tar_spark_datasource.jar new file mode 100755 index 00000000..ea2ef162 --- /dev/null +++ b/galvasr2/tar_spark_datasource.jar @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0a8775725e31eced0ebdff2b7afe2ab400d5f0fd063d1905083adcc2f34454fd +size 17333 diff --git a/lingvo/.style.yapf b/lingvo/.style.yapf deleted file mode 100644 index 7c93ab7b..00000000 --- a/lingvo/.style.yapf +++ /dev/null @@ -1,3 +0,0 @@ -[style] -based_on_style = google -indent_width = 2 \ No newline at end of file diff --git a/lingvo/BUILD b/lingvo/BUILD deleted file mode 100644 index bf8f795c..00000000 --- a/lingvo/BUILD +++ /dev/null @@ -1,311 +0,0 @@ -load( - "//lingvo:lingvo.bzl", - "lingvo_py_binary", -) - -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) - -exports_files(["LICENSE"]) - -config_setting( - name = "cuda", - values = {"define": "using_cuda=true"}, -) - -py_library( - name = "base_runner", - srcs = ["base_runner.py"], - srcs_version = "PY3", - deps = [ - ":base_trial", - ":compat", - "//lingvo/core:cluster_factory", - "//lingvo/core:early_stop", - "//lingvo/core:py_utils", - # Implicit tensorflow py proto dependency. - ], -) - -py_library( - name = "base_trial", - srcs = ["base_trial.py"], - srcs_version = "PY3", - deps = [ - "//lingvo/core:hyperparams", - ], -) - -py_library( - name = "compat", - srcs = ["compat.py"], - srcs_version = "PY3", - deps = [ - # Implicit absl.flags dependency. - # Implicit absl.logging dependency. - # Implicit tensorflow dependency. - ], -) - -py_test( - name = "compat_test", - srcs = ["compat_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":compat", - ], -) - -py_library( - name = "model_imports_no_params", - srcs = ["model_imports.py"], - srcs_version = "PY3", -) - -# Depend on this for access to the model registry with params for all tasks as -# transitive deps. Only py_binary should depend on this target. -py_library( - name = "model_imports", - srcs_version = "PY3", - deps = [ - ":model_imports_no_params", - "//lingvo/tasks:all_params", - ], -) - -py_test( - name = "model_import_test", - srcs = ["model_import_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":compat", - ":model_imports_no_params", - ], -) - -py_library( - name = "models_test_helper", - testonly = 1, - srcs = ["models_test_helper.py"], - srcs_version = "PY3", - deps = [ - ":compat", - "//lingvo/core:base_input_generator", - "//lingvo/core:base_model", - "//lingvo/core:bn_layers", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - ], -) - -py_test( - name = "models_test", - srcs = ["models_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":compat", - ":model_imports", - ":model_registry", - ":model_registry_test_lib", - ":models_test_helper", - "//lingvo/core:base_model", - ], -) - -py_library( - name = "model_registry", - srcs = ["model_registry.py"], - srcs_version = "PY3", - deps = [ - ":compat", - ":model_imports_no_params", - "//lingvo/core:base_model_params", - ], -) - -py_test( - name = "model_registry_test", - srcs = ["model_registry_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [":model_registry_test_lib"], -) - -py_library( - name = "model_registry_test_lib", - testonly = 1, - srcs = ["model_registry_test.py"], - srcs_version = "PY3", - deps = [ - ":compat", - ":model_registry", - "//lingvo/core:base_input_generator", - "//lingvo/core:base_model", - "//lingvo/core:base_model_params", - "//lingvo/core:test_utils", - ], -) - -py_library( - name = "datasets_lib", - srcs = ["datasets.py"], - srcs_version = "PY3", -) - -py_library( - name = "trainer_lib", - srcs = ["trainer.py"], - srcs_version = "PY3", - deps = [ - ":base_trial", - ":compat", - ":datasets_lib", - ":executor_lib", - ":model_imports_no_params", - ":model_registry", - # Implicit network file system dependency. - "//lingvo:base_runner", - # Implicit IPython dependency. - "//lingvo/core:base_layer", - "//lingvo/core:base_model", - "//lingvo/core:base_model_params", - "//lingvo/core:checkpointer_lib", - "//lingvo/core:cluster_factory", - "//lingvo/core:inference_graph_exporter", - "//lingvo/core:metrics", - "//lingvo/core:py_utils", - "//lingvo/core:summary_utils", - # Implicit numpy dependency. - # Implicit tensorflow grpc dependency. - ], -) - -lingvo_py_binary( - name = "trainer", - srcs = [":trainer_lib"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":model_imports", - ":trainer_lib", - ], -) - -py_test( - name = "trainer_test", - size = "large", - timeout = "long", - srcs = ["trainer_test.py"], - python_version = "PY3", - shard_count = 12, - srcs_version = "PY3", - tags = [ - "noasan", - "nomsan", - "notsan", - "optonly", - ], - deps = [ - ":base_trial", - ":compat", - ":model_registry", - ":trainer_lib", - # Implicit absl.testing.flagsaver dependency. - "//lingvo/core:base_input_generator", - "//lingvo/core:base_model", - "//lingvo/core:base_model_params", - "//lingvo/core:hyperparams", - "//lingvo/core:inference_graph_py_pb2", - "//lingvo/core:test_utils", - "//lingvo/core:trainer_test_utils", - "//lingvo/tasks/image:input_generator", - "//lingvo/tasks/image/params:mnist", # build_cleaner: keep - "//lingvo/tasks/punctuator/params:codelab", # build_cleaner: keep - # Implicit numpy dependency. - ], -) - -py_library( - name = "trainer_test_lib", - testonly = 1, - srcs = ["trainer_test.py"], - srcs_version = "PY3", - tags = [ - "noasan", - "nomsan", - "notsan", - "optonly", - ], - deps = [ - ":base_trial", - ":compat", - ":model_registry", - ":trainer_lib", - "//lingvo/core:base_input_generator", - "//lingvo/core:base_layer", - "//lingvo/core:base_model", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - "//lingvo/core:trainer_test_utils", - "//lingvo/tasks/image:input_generator", - "//lingvo/tasks/image/params:mnist", - "//lingvo/tasks/punctuator/params:codelab", - # Implicit numpy dependency. - ], -) - -lingvo_py_binary( - name = "ipython_kernel", - srcs = ["ipython_kernel.py"], - data = [ - "//lingvo/tasks/punctuator/tools:download_brown_corpus", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":model_imports", # build_cleaner: keep - ":trainer_lib", # build_cleaner: keep - # Implicit absl.app dependency. - "//lingvo/core:predictor_lib", # build_cleaner: keep - "//lingvo/core/ops:hyps_py_pb2", # build_cleaner: keep - ], -) - -genrule( - name = "tf_dot_protos", - srcs = [], - outs = ["tf_protos.tar"], - cmd = - "$(location //lingvo/tools:" + - "generate_tf_dot_protos) $(location " + - "//lingvo/tools:generate_proto_def) $(@D)", - tools = [ - "//lingvo/tools:generate_proto_def", - "//lingvo/tools:generate_tf_dot_protos", - ], -) - -py_library( - name = "executor_lib", - srcs = ["executor.py"], - srcs_version = "PY3", - deps = [ - ":compat", - # Implicit network file system dependency. - "//lingvo:base_runner", - "//lingvo/core:base_model", - "//lingvo/core:checkpointer_lib", - "//lingvo/core:cluster_factory", - "//lingvo/core:ml_perf_log", - "//lingvo/core:multitask_model", - "//lingvo/core:py_utils", - "//lingvo/core:task_scheduler", - # Implicit tensorflow grpc dependency. - ], -) diff --git a/lingvo/base_runner.py b/lingvo/base_runner.py deleted file mode 100644 index 71c24931..00000000 --- a/lingvo/base_runner.py +++ /dev/null @@ -1,424 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Base class for all jobs.""" - -import os -import time -import traceback - -from lingvo import base_trial -import lingvo.compat as tf -from lingvo.core import cluster_factory -from lingvo.core import early_stop -from lingvo.core import py_utils - - -class BaseRunner: - """Base class for all jobs.""" - - def __init__(self, - params, - model_task_name, - logdir, - tf_master, - trial=base_trial.NoOpTrial()): - """Construct a new BaseRunner. - - Args: - params: Params object containing model configuration. - model_task_name: String name of the task this runner should execute for - multitask models only. See flag for details. - logdir: String path to the log directory to output to. - tf_master: String path to the master job, e.g. 'local'. - trial: An optional hyperparameter trial. Used by Vizier studies. - """ - p = params.Copy() - # Set in subclasses. - self._job_name = '' - - self._params = trial.OverrideModelParams(p) - tf.logging.info('=' * 60) - for line in self.params.ToText().split('\n'): - tf.logging.info('%s', line) - tf.logging.info('=' * 60) - tf.logging.info('FLAGS') - for attribute in dir(tf.flags.FLAGS): - tf.logging.info("%s=%s", attribute, getattr(tf.flags.FLAGS, attribute)) - tf.logging.info('=' * 60) - - self._logdir = logdir - self._tf_master = tf_master - self._model_task_name = model_task_name - self._trial = trial - # If the runner is conducting a Vizier trial, scope all the variables - # (e.g., global_step) by the trial id so that we do not share states across - # trials. - self._container_id = self._trial.Name() - self._should_report_metrics = False - - # To early terminate a runner, we set max_steps here and that will trigger - # appropriate ShouldStop behavior in the threads. This is used by Vizier - # to early stop a trial. - self._max_steps = None - - self.params.cluster.logdir = logdir - self._cluster = cluster_factory.Cluster(self.params.cluster) - self._train_dir = os.path.join(self._logdir, 'train') - tf.io.gfile.makedirs(self._train_dir) - self._graph = tf.Graph() - self._summary_writer = None - self._initialize_tables = None - self._dequeue_thread_complete = False - - early_stop.MetricHistory.SetLogdirInMetricHistories(p, logdir) - self._early_stop = None - if p.train.early_stop and p.train.early_stop.window: - self._early_stop = early_stop.EarlyStop(p.train.early_stop) - with self._graph.as_default(): - self._early_stop.FProp(None) - - self._SetStatusMessage('Starting ...') - - @property - def params(self): - return self._params - - def _InVizierStudy(self): - return not isinstance(self._trial, base_trial.NoOpTrial) - - def _FormatStatusMessage(self, message, retrying): - if self._trial.Name(): - message = 'Trial:{} {}'.format(self._trial.Name(), message) - if retrying: - message = f'Job {self._job_name}: Retrying as expected\n{message}' - return message - - def _SetStatusMessage(self, message, retrying=False): - """Update the status message for this task.""" - tf.logging.info(self._FormatStatusMessage(message, retrying)) - - def _ShouldStop(self, sess, step): - """Check if the runner should stop.""" - if step >= self.params.train.max_steps: - tf.logging.info('ShouldStop: step:%6d params.train.max_steps:%6d', - step, self.params.train.max_steps) - return True - - if self._max_steps and step >= self._max_steps: - tf.logging.info('ShouldStop: step:%6d _max_steps:%6d', step, - self._max_steps) - return True - - if self._early_stop and self._early_stop.Stop(sess): - tf.logging.info('ShouldStop: Early stopping.') - return True - - return False - - def _WriteToLog(self, text, logdir, filename): - """Logs `text` and saves it under `logdir/filename`.""" - with tf.io.gfile.GFile(os.path.join(logdir, filename), 'w') as f: - f.write(text) - - if self._summary_writer is not None: - # Force newlines to be rendered correctly by Markdown. - text = text.replace('\n', ' \n') - self._summary_writer.add_summary( - tf.Summary(value=[ - tf.Summary.Value( - tag=filename, tensor=tf.make_tensor_proto([text])) - ])) - - def _WaitUntilInit(self, sess, start_up_delay_steps=None): - """Wait until the model is ready.""" - # Wait a fix amount of time at start. - time.sleep(30) - - @py_utils.Retry(retry_value=(tf.errors.FailedPreconditionError,)) - def RetryLoop(): - try: - global_step = sess.run(py_utils.GetGlobalStep()) - except tf.errors.FailedPreconditionError as e: - tf.logging.info('%s: Probably the expected race on global_step: %s', - self._job_name, e) - raise - msg = 'step:%6d' % global_step - self._SetStatusMessage(msg) - if start_up_delay_steps: - if global_step < start_up_delay_steps: - msg = 'global step (%d) has not reached start up delay steps (%d)' % ( - global_step, self._start_up_delay_steps) - tf.logging.info('%s: %s', self._job_name, msg) - raise tf.errors.FailedPreconditionError( - node_def=None, op=None, message=msg) - return global_step - - return RetryLoop() - - @py_utils.Retry( - initial_delay_sec=1, delay_growth_factor=1.5, max_delay_sec=300) - def _FindNewCheckpoint(self, prev_path, sess): - """Returns the path to a new checkpoint, or raises RuntimeError.""" - if self._trial.ShouldStop() or self._ShouldStop(sess, 0): - return None - path = tf.train.latest_checkpoint(self._train_dir) - if not path: - msg = 'No check point is found in %s' % self._train_dir - tf.logging.info('%s: %s', self._job_name, msg) - raise RuntimeError(msg) - if path == prev_path: - msg = 'No new check point is found: %s' % path - tf.logging.info('%s: %s', self._job_name, msg) - raise RuntimeError(msg) - return path - - @py_utils.Retry(max_retries=0) - def _RunLoop(self, job_name, loop_func, loop_args=()): - """Runs `loop_func`, retrying on expected errors. - - Args: - job_name: string job name. - loop_func: callable to run and retry on expected errors. - loop_args: list or tuple of arguments to be passed to the loop_func. - """ - try: - tf.logging.info('%s started.', job_name) - loop_func(*loop_args) - tf.logging.info('%s done.', job_name) - return - except Exception as e: # pylint:disable=broad-except - fatal_error_msgs = [ - 'Compilation failure', - 'Run-time shape mismatch for TPUExecute argument' - ] - if any([x in str(e) for x in fatal_error_msgs]): - # Fatal error if failing to compile graph on TPU. - retry = False - elif isinstance(e, tf.errors.AbortedError): - # AbortedError: is thrown when processes restarts. - retry = True - if self._InVizierStudy(): - # With Vizier studies, we want to avoid retrying under some error - # conditions, these are captured here. - # Do not retry (via raise/retry) if AbortedError with RecvTensor - # message. This can happen if there are memory issues. - if ('The same RecvTensor (WorkerServiceImpl) request was received ' - 'twice' in str(e)): - retry = False - tf.logging.info('%s done (infeasible error).', job_name) - elif isinstance(e, tf.errors.OutOfRangeError): - # OutOfRangeError: Test/dev datasets are exhausted. - retry = self._cluster.do_eval - elif isinstance(e, tf.errors.InvalidArgumentError): - # InvalidArgumentError: variables were not initialized. Comes from - # ResourceVariableOp. - retry = True - # Do not retry within Vizier study when NaNs cause InvalidArgumentError. - if self._InVizierStudy(): - if 'Tensor had NaN values' in str(e): - retry = False - tf.logging.info('%s done (infeasible result due to NaN values).', - job_name) - elif isinstance( - e, py_utils.transient_tf_errors + - (tf.errors.DataLossError, tf.errors.CancelledError)): - # Retry on these errors. - # FailedPreconditionError: variables are not initialized. - # DataLossError: Race condition between evaler and trainer when saving - # or removing checkpoints. - # CancelledError: Node was closed (on TPU). - if isinstance(e, tf.errors.FailedPreconditionError): - # GALV: This error frequently was due to global_step not - # being initialized, which as far as I can tell is a hard - # error, so I disabling retrying in this case. - retry = False - else: - retry = True - else: - retry = False - - if retry: - # Retry indefinitely (error should be transient). - self._SetStatusMessage( - '%s exception: %s\n' % (job_name, e), retrying=True) - - for msg in traceback.format_exc().split('\n'): - tf.logging.vlog(1, msg) - - raise - else: - # Allow the job to complete on errors that are unlikely to be transient, - # e.g. caused by a mis-configured model. - if self._should_report_metrics: - self._trial.ReportDone( - infeasible=True, infeasible_reason='Fatal error encountered.') - tf.logging.error('%s done (fatal error): %s', job_name, type(e)) - - self._SetStatusMessage('%s exception: %s\n' % (job_name, e)) - - # Prints the error message line by line to avoid message cropping. - msgv = traceback.format_exc().split('\n') - for msg in msgv: - tf.logging.error(msg) - - # Check if we are potentially running within an experiment. If so, - # the worker should continue to the next trial instead of terminating - # the process. - if self._InVizierStudy(): - return - - # tf.logging.fatal prints out stack traces. Typically, that's not - # useful at all here. Here we want to exit the program - # definitively. Because LOG(QFATAL) is not easily available via - # python so far, we need a way to exit the program directly. - # Because sys.exit(1) must be called from the main thread, and does - # not cancel non-daemon threads anyway, we use os._exit instead. - # Because tf.logging.error() may return before the flush is complete, - # we need an extra sleep before exit. - time.sleep(15) - # GALV: print() statements will not be flushed by os._exit, so - # I flush sys.stdouthere! - import sys; sys.stdout.flush() - os._exit(1) # pylint: disable=protected-access - - def _DequeueThreadComplete(self): - self._dequeue_thread_complete = True - return - - def _LoopEnqueue(self, op, session_override=None): - """Runs the enqueue op in a loop.""" - p = self.params - sess = session_override or self._GetSession() - - with tf.container(self._container_id), sess: - if self._initialize_tables is not None: - sess.run(self._initialize_tables) - gsteps = py_utils.GetGlobalStep() - local_enqueue_steps = 0 - - # Global enqueue steps measures how many global steps have data enqueued - # for already. We use this to terminate; note that the enqueue op may - # hang in session.run if we do not terminate with this check. - global_enqueue_steps = None - - tf.logging.info('params.train.max_steps: %d, enqueue_max_steps: %d', - p.train.max_steps, p.train.enqueue_max_steps) - while True: - if self._dequeue_thread_complete: - tf.logging.info( - 'LoopEnqueue done since consuming thread is done.') - return - - global_step = sess.run(gsteps) - if global_enqueue_steps is None: - global_enqueue_steps = global_step - if local_enqueue_steps % 1000 == 0: - tf.logging.info( - 'Current global_enqueue_steps: %d, ' - 'local_enqueue_steps: %d, global_step: %d', global_enqueue_steps, - local_enqueue_steps, global_step) - - if py_utils.use_tpu(): - global_steps_with_available_data = int(global_enqueue_steps // - p.train.tpu_steps_per_loop * - p.train.tpu_steps_per_loop) - else: - global_steps_with_available_data = global_enqueue_steps - - if (self._ShouldStop(sess, global_steps_with_available_data) or - self._ShouldStop(sess, global_step)): - tf.logging.info('Done. ShouldStop is True.') - tf.logging.info('Enqueue loop sleeping') - time.sleep(15) - continue - if (p.train.enqueue_max_steps > 0 and - local_enqueue_steps >= p.train.enqueue_max_steps): - tf.logging.info('Done. train.enqueue_max_steps reached.') - tf.logging.info('Enqueue loop sleeping') - time.sleep(15) - continue - local_enqueue_steps += 1 - - # There are tpu_infeed_parallelism parallel threads enqueuing. - # We account for all of them when updating global_enqueue_steps. - global_enqueue_steps += p.input.tpu_infeed_parallelism - - sess.run([op]) - - def _GetSession(self, **kwargs): - graph = kwargs.pop('graph', self._graph) - return tf.Session( - self._tf_master, graph=graph, config=py_utils.SessionConfig(**kwargs)) - - def GetTrainDir(self): - return self._train_dir - - @classmethod - def _GetTtlDir(cls, path, duration): - """Returns a path to a time-limited directory under dir if required.""" - del duration - return path - - def _CreateSummaryWriter(self, logdir): - """Creates and returns a tf summary writer.""" - return tf.summary.FileWriter(logdir) - - def _WriteSummaries(self, - summary_writer, - job_name, - global_step, - summaries, - text_filename=None): - """Construct the summary and write them to the summary writer. - - Args: - summary_writer: The summary writer to use. - job_name: The name of the job that tries to write this summary. - global_step: The checkpoint used for eval is generated at this step. - summaries: a dict from keys to `tf.Summary` protobuf messages. - text_filename: If not None, writes the summary to the text file. - """ - status_metrics = [] - for name, summary in sorted(summaries.items()): - if not isinstance(summary, tf.summary.Summary): - tf.logging.warning( - 'Non tf.Summary args passed to _WriteSummaries, skipping: ' - 'job:%s name:%s @%s', job_name, name, global_step) - continue - summary_writer.add_summary(summary, global_step) - if summary.value: - for value in summary.value: - if value.HasField('simple_value'): - tf.logging.info('%s summary on checkpoint@%d %s = %.8g', - job_name, global_step, value.tag, - value.simple_value) - status_metrics.append('%s: %.8g' % (value.tag, value.simple_value)) - early_stop.MetricHistory.ConditionalAppend(job_name, value.tag, - global_step, - value.simple_value) - else: - tf.logging.info('%s summary on checkpoint@%d %s', job_name, - global_step, value.tag) - summary_writer.flush() - self._SetStatusMessage('%s: step:%6d, %s' % - (job_name, global_step, ', '.join(status_metrics))) - if text_filename is not None: - with tf.io.gfile.GFile(text_filename, 'w') as f: - f.write('\n'.join(status_metrics)) - - def _ExportMetrics(self, **kwargs): - """Exports metrics externally.""" - pass diff --git a/lingvo/base_trial.py b/lingvo/base_trial.py deleted file mode 100644 index 3664bbe2..00000000 --- a/lingvo/base_trial.py +++ /dev/null @@ -1,126 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Defines trials for parameter exploration.""" - -import time - -from lingvo.core import hyperparams - - -class Trial: - """Base class for a trial.""" - - @classmethod - def Params(cls): - """Default parameters for a trial.""" - p = hyperparams.Params() - p.Define( - 'report_interval_seconds', 600, - 'Interval between reporting trial results and checking for early ' - 'stopping.') - p.Define('vizier_objective_metric_key', 'loss', - 'Which eval metric to use as the "objective value" for tuning.') - p.Define( - 'report_during_training', False, - 'Whether to report objective metrics during the training process.') - return p - - def __init__(self, params): - self._params = params.Copy() - self._next_report_time = time.time() - - @property - def report_interval_seconds(self): - return self._params.report_interval_seconds - - @property - def objective_metric_key(self): - return self._params.vizier_objective_metric_key - - def Name(self): - raise NotImplementedError('Abstract method') - - def OverrideModelParams(self, model_params): - """Modifies `model_params` according to trial params. - - Through this method a `Trial` may tweak model hyperparams (e.g., learning - rate, shape, depth, or width of networks). - - Args: - model_params: the original model hyperparams. - - Returns: - The modified `model_params`. - """ - raise NotImplementedError('Abstract method') - - def ShouldStop(self): - """Returns whether the trial should stop.""" - raise NotImplementedError('Abstract method') - - def ReportDone(self, infeasible=False, infeasible_reason=''): - """Report that the trial is completed.""" - raise NotImplementedError('Abstract method') - - def ShouldStopAndMaybeReport(self, global_step, metrics_dict): - """Returns whether the trial should stop. - - Args: - global_step: The global step counter. - metrics_dict: If not None, contains the metric should be - reported. If None, do nothing but returns whether the - trial should stop. - """ - if not metrics_dict or not self._params.report_during_training: - return self.ShouldStop() - if time.time() < self._next_report_time: - return False - self._next_report_time = time.time() + self.report_interval_seconds - return self._DoReportTrainingProgress(global_step, metrics_dict) - - def _DoReportTrainingProgress(self, global_step, metrics_dict): - raise NotImplementedError('Abstract method') - - def ReportEvalMeasure(self, global_step, metrics_dict, checkpoint_path): - """Reports eval measurement and returns whether the trial should stop.""" - raise NotImplementedError('Abstract method') - - -class NoOpTrial(Trial): - """A Trial implementation that does nothing.""" - - def __init__(self): - super().__init__(Trial.Params()) - - def Name(self): - return '' - - def OverrideModelParams(self, model_params): - return model_params - - def ShouldStop(self): - return False - - def ReportDone(self, infeasible=False, infeasible_reason=''): - return False - - def ShouldStopAndMaybeReport(self, global_step, metrics_dict): - del global_step, metrics_dict # Unused - return False - - def ReportEvalMeasure(self, global_step, metrics_dict, checkpoint_path): - del global_step, metrics_dict, checkpoint_path # Unused - return False diff --git a/lingvo/compat.py b/lingvo/compat.py deleted file mode 100644 index 06901c6f..00000000 --- a/lingvo/compat.py +++ /dev/null @@ -1,411 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""The compatible tensorflow library.""" - -import tensorflow.compat.v1 as tf1 -from tensorflow.compat.v2 import * # pylint:disable=wildcard-import, g-bad-import-order - -# Import absl.flags and absl.logging to overwrite the Tensorflow ones. -# This is the intended behavior in TF 2.0. -# pylint:disable=g-bad-import-order, unused-import, g-import-not-at-top -from absl import flags -from absl import logging -# pylint: disable=g-direct-tensorflow-import - -from tensorflow.core.protobuf import config_pb2 -from tensorflow.python.data.ops import dataset_ops -from tensorflow.python.data.util import random_seed -from tensorflow.python.framework import function as _function_lib -from tensorflow.python.framework import ops -from tensorflow.python.ops import array_ops -from tensorflow.python.ops import check_ops -from tensorflow.python.ops import embedding_ops -from tensorflow.python.ops import functional_ops -from tensorflow.python.ops import gen_dataset_ops -from tensorflow.python.ops import gen_io_ops -from tensorflow.python.ops import inplace_ops -from tensorflow.python.ops import math_ops -from tensorflow.python.platform import app -from tensorflow.python.util import module_wrapper as _module_wrapper -# pylint: enable=g-direct-tensorflow-import -# pylint: enable=unused-import, g-bad-import-order, g-import-not-at-top - -tf1.disable_eager_execution() - -if tf1.executing_eagerly(): - logging.warning("Lingvo does not support eager execution yet. Please disable " - "eager execution with tf.compat.v1.disable_eager_execution() " - "or proceed at your own risk.") - - -def _clone_module(m): - """Shallow clone of module `m`.""" - if isinstance(m, _module_wrapper.TFModuleWrapper): - # pylint: disable=protected-access - return _module_wrapper.TFModuleWrapper( - wrapped=_clone_module(m._tfmw_wrapped_module), - module_name=m._tfmw_module_name, - public_apis=m._tfmw_public_apis, - deprecation=m._tfmw_print_deprecation_warnings, - has_lite=m._tfmw_has_lite) - # pylint: enable=protected-access - out = type(m)(m.__name__, m.__doc__) - out.__dict__.update(m.__dict__) - return out - -# Aliases to a few routines lingvo libraries uses often. -Defun = _function_lib.Defun -While = functional_ops.While -If = functional_ops.If -InplaceUpdate = inplace_ops.alias_inplace_update -Empty = inplace_ops.empty -EmptyLike = inplace_ops.empty_like - -# pylint: disable=undefined-variable, used-before-assignment -# Move this V2 symbol here to avoid being overwritten by its following V1 -# version. -where_v2 = where -while_loop_v2 = while_loop - -# Import the local V2 module to maker sure the following V1 overwritting never -# applies to the global module and symbol. -data = _clone_module(data) -graph_util = _clone_module(graph_util) -image = _clone_module(image) -io = _clone_module(io) -losses = _clone_module(keras.losses) -metrics = _clone_module(keras.metrics) -nn = _clone_module(nn) -saved_model = _clone_module(saved_model) -strings = _clone_module(strings) -summary = _clone_module(summary) -test = _clone_module(test) -train = _clone_module(train) - -# By default, with TF2 enabled and (eager execution or tf.function), -# `tf.data` API will choose the stateful implementation for methods -# `tf.data.Dataset.shuffle()`, `tf.data.Dataset.cache()` and -# `tf.data.Dataset.list_files()`. which is not compatible with -# `tf.data.make_one_shot_iterator` in TF2 (see b/162270607). -# Here is a stateless implementation of `shuffle`, `cache` and -# `list_files` to resolve the TF2 imcompatibility issue. - -# Note that, these methods are meant for internal use only. Please don't use -# it unless you know exactly what you do. - - -class _CacheDataset(dataset_ops.UnaryUnchangedStructureDataset): - """A `Dataset` that caches elements of its input.""" - - def __init__(self, input_dataset, filename): - """Caches the elements in the dataset.""" - self._input_dataset = input_dataset - self._filename = ops.convert_to_tensor( - filename, dtype=string, name="filename") - variant_tensor = gen_dataset_ops.cache_dataset( - input_dataset._variant_tensor, # pylint: disable=protected-access - filename=self._filename, - **self._flat_structure) - super(_CacheDataset, self).__init__(input_dataset, variant_tensor) - - -class _ShuffleDataset(dataset_ops.UnaryUnchangedStructureDataset): - """A `Dataset` that randomly shuffles the elements of its input.""" - - def __init__(self, - input_dataset, - buffer_size, - seed=None, - reshuffle_each_iteration=None): - """Randomly shuffles the elements of this dataset.""" - self._input_dataset = input_dataset - self._buffer_size = ops.convert_to_tensor( - buffer_size, dtype=int64, name="buffer_size") - self._seed, self._seed2 = random_seed.get_seed(seed) - if reshuffle_each_iteration is None: - reshuffle_each_iteration = True - self._reshuffle_each_iteration = reshuffle_each_iteration - - variant_tensor = gen_dataset_ops.shuffle_dataset( - input_dataset._variant_tensor, # pylint: disable=protected-access - buffer_size=self._buffer_size, - seed=self._seed, - seed2=self._seed2, - reshuffle_each_iteration=self._reshuffle_each_iteration, - **self._flat_structure) - super(_ShuffleDataset, self).__init__(input_dataset, variant_tensor) - - -def stateless_shuffle_dataset(buffer_size, - seed=None, - reshuffle_each_iteration=None): - """Randomly shuffles the elements of the dataset based on a stateless shuffle implementation. - - This method returns a stateless ShuffleDataset unconditionally. It can be - used with `dataset.apply()` to obtain a stateless shuffled dataset, which - supports the TF1 compatibility API `tf.data.make_one_shot_iterator()` in TF2. - Example: - >>> dataset = tf.data.Dataset.range(3) - >>> dataset = dataset.apply( - ... stateless_shuffle_dataset((3, reshuffle_each_iteration=True)) - - Args: - buffer_size: A `tf.int64` scalar `tf.Tensor`, representing the number of - elements from this dataset from which the new dataset will sample. - seed: (Optional.) A `tf.int64` scalar `tf.Tensor`, representing the random - seed that will be used to create the distribution. See - `tf.random.set_seed` for behavior. - reshuffle_each_iteration: (Optional.) A boolean, which if true indicates - that the dataset should be pseudorandomly reshuffled each time it is - iterated over. (Defaults to `True`.) - - Returns: - Dataset: A `Dataset`. - """ - - def _apply_fn(dataset): - out_dataset = dataset_ops.DatasetV1Adapter( - _ShuffleDataset(dataset, buffer_size, seed, reshuffle_each_iteration)) - return out_dataset - - return _apply_fn - - -def stateless_cache_dataset(filename=""): - """Caches the elements in the dataset based on a stateless cache implementation. - - This method returns a stateless CacheDataset unconditionally. It can be - used with `dataset.apply()` to obtain a stateless cached dataset, which - supports the TF1 compatibility API `tf.data.make_one_shot_iterator()` in TF2. - - Example: - >>> dataset = tf.data.Dataset.range(3) - >>> dataset = dataset.apply(stateless_cache_dataset()) - - - Args: - filename: A `tf.string` scalar `tf.Tensor`, representing the name of a - directory on the filesystem to use for caching elements in this Dataset. - If a filename is not provided, the dataset will be cached in memory. - - Returns: - Dataset: A `Dataset`. - """ - - def _apply_fn(dataset): - out_dataset = dataset_ops.DatasetV1Adapter(_CacheDataset(dataset, filename)) - return out_dataset - - return _apply_fn - - -def stateless_list_files(file_pattern, shuffle=None, seed=None): - """A dataset of all files matching one or more glob patterns. - - Note that, if `shuffle` is not None, it will use a stateless shuffle - implementation. Then the returned dataset supports the TF1 compatibility API - `tf.data.make_one_shot_iterator()` in TF2. - - Example: - >>> dataset = tf.stateless_list_files("some_file_pattern") - - Args: - file_pattern: A string, a list of strings, or a `tf.Tensor` of string type - (scalar or vector), representing the filename glob (i.e. shell wildcard) - pattern(s) that will be matched. - shuffle: (Optional.) If `True`, the file names will be shuffled randomly - based on a stateless implementation. Defaults to `True`. - seed: (Optional.) A `tf.int64` scalar `tf.Tensor`, representing the random - seed that will be used to create the distribution. See - `tf.random.set_seed` for behavior. - - Returns: - Dataset: A `Dataset` of strings corresponding to file names. - """ - with ops.name_scope("list_files"): - if shuffle is None: - shuffle = True - file_pattern = ops.convert_to_tensor( - file_pattern, dtype=string, name="file_pattern") - matching_files = gen_io_ops.matching_files(file_pattern) - - # Raise an exception if `file_pattern` does not match any files. - condition = math_ops.greater( - array_ops.shape(matching_files)[0], 0, name="match_not_empty") - message = math_ops.add( - "No files matched pattern: ", - strings.reduce_join(file_pattern, separator=", "), - name="message") - - assert_not_empty = debugging.Assert( - condition, [message], summarize=1, name="assert_not_empty") - with control_dependencies([assert_not_empty]): - matching_files = identity(matching_files) - - dataset = data.Dataset.from_tensor_slices(matching_files) - if shuffle: - buffer_size = math_ops.maximum( - shape(matching_files, out_type=dtypes.int64)[0], 1) - # Use stateless shuffled dataset - dataset = dataset.apply(stateless_shuffle_dataset(buffer_size, seed=seed)) - return dataset -# pylint: enable=undefined-variable, used-before-assignment - -# TF 1.x symbols used in the codebase. -# To keep this list short, please use TF 2.x API whenever applicable. -# Only use TF 1.x API if it has no 2.x equivalent. -# pylint: disable=undefined-variable -add_to_collection = tf1.add_to_collection -all_variables = tf1.global_variables -# The following asserts can be directly replaced with TF2 `tf.debugging.*` -# after TF2/eager is enabled. -assert_integer = tf1.assert_integer -assert_positive = tf1.assert_positive -assert_type = tf1.assert_type -assert_scalar = tf1.assert_scalar -assign = tf1.assign -assign_add = tf1.assign_add -assign_sub = tf1.assign_sub -AUTO_REUSE = tf1.AUTO_REUSE -container = tf1.container -data.Dataset = tf1.data.Dataset -data.TFRecordDataset = tf1.data.TFRecordDataset -device = tf1.device -Dimension = tf1.Dimension -disable_eager_execution = tf1.disable_eager_execution -div = tf1.div -enable_eager_execution = tf1.enable_eager_execution -floor_div = tf1.floor_div -get_collection = tf1.get_collection -get_collection_ref = tf1.get_collection_ref -get_default_graph = tf1.get_default_graph -get_local_variable = tf1.get_local_variable -get_seed = tf1.get_seed -get_variable = tf1.get_variable -get_variable_scope = tf1.get_variable_scope -global_variables = tf1.global_variables -global_variables_initializer = tf1.global_variables_initializer -gradients = tf1.gradients -graph_util.convert_variables_to_constants = ( - tf1.graph_util.convert_variables_to_constants) -graph_util.extract_sub_graph = tf1.graph_util.extract_sub_graph -GraphDef = tf1.GraphDef -GraphKeys = tf1.GraphKeys -GraphOptions = tf1.GraphOptions -group = tf1.group -image.resize_bilinear = tf1.image.resize_bilinear -image.resize_images = tf1.image.resize_images -image.resize_nearest_neighbor = tf1.image.resize_nearest_neighbor -initialize_all_tables = tf1.initialize_all_tables -InteractiveSession = tf1.InteractiveSession -io.tf_record_iterator = tf1.io.tf_record_iterator -layers = tf1.layers -local_variables_initializer = tf1.local_variables_initializer -losses.absolute_difference = tf1.losses.absolute_difference -losses.add_loss = tf1.losses.add_loss -losses.compute_weighted_loss = tf1.losses.compute_weighted_loss -losses.get_regularization_loss = tf1.losses.get_regularization_loss -losses.huber_loss = tf1.losses.huber_loss -losses.mean_squared_error = tf1.losses.mean_squared_error -losses.Reduction.MEAN = tf1.losses.Reduction.MEAN -losses.Reduction.SUM = tf1.losses.Reduction.SUM -losses.sigmoid_cross_entropy = tf1.losses.sigmoid_cross_entropy -losses.softmax_cross_entropy = tf1.losses.softmax_cross_entropy -losses.sparse_softmax_cross_entropy = (tf1.losses.sparse_softmax_cross_entropy) -make_template = tf1.make_template -metrics.accuracy = tf1.metrics.accuracy -metrics.auc = tf1.metrics.auc -metrics.precision = tf1.metrics.precision -metrics.recall = tf1.metrics.recall -moving_average_variables = tf1.moving_average_variables -multinomial = tf1.multinomial -name_scope = tf1.name_scope -OptimizerOptions = tf1.OptimizerOptions -placeholder = tf1.placeholder -placeholder_with_default = tf1.placeholder_with_default -Print = tf1.Print -py_func = tf1.py_func -python_io = tf1.python_io -report_uninitialized_variables = tf1.report_uninitialized_variables -reset_default_graph = tf1.reset_default_graph -resource_loader = tf1.resource_loader -RunMetadata = tf1.RunMetadata -RunOptions = tf1.RunOptions -saved_model.build_signature_def = tf1.saved_model.build_signature_def -saved_model.Builder = tf1.saved_model.Builder -saved_model.load = tf1.saved_model.load -saved_model.loader = tf1.saved_model.loader -saved_model.signature_constants = tf1.saved_model.signature_constants -saved_model.simple_save = tf1.saved_model.simple_save -saved_model.tag_constants = tf1.saved_model.tag_constants -saved_model.utils = tf1.saved_model.utils -Session = tf1.Session -sparse_to_dense = tf1.sparse_to_dense -string_split = tf1.string_split -strings.reduce_join = tf1.reduce_join -strings.split = tf1.strings.split -Summary = tf1.Summary -if tf1.summary is not None: - # tf.summary are not supported on TPU so we sometimes set tf.summary to None - # to prohibit the direct use of it. - # It is safe to skip copying tf.summary members in such cases. - summary.audio = tf1.summary.audio - summary.FileWriter = tf1.summary.FileWriter - summary.histogram = tf1.summary.histogram - summary.image = tf1.summary.image - summary.merge = tf1.summary.merge - summary.merge_all = tf1.summary.merge_all - summary.scalar = tf1.summary.scalar - summary.Summary = tf1.summary.Summary - summary.Summary.FromString = tf1.summary.Summary.FromString -tables_initializer = tf1.tables_initializer -test.compute_gradient_error = tf1.test.compute_gradient_error -test.get_temp_dir = tf1.test.get_temp_dir -test.mock = tf1.test.mock -tpu = tf1.tpu -train.AdadeltaOptimizer = tf1.train.AdadeltaOptimizer -train.AdagradOptimizer = tf1.train.AdagradOptimizer -train.AdamOptimizer = tf1.train.AdamOptimizer -train.export_meta_graph = tf1.train.export_meta_graph -train.get_or_create_global_step = tf1.train.get_or_create_global_step -train.get_global_step = tf1.train.get_global_step -train.GradientDescentOptimizer = tf1.train.GradientDescentOptimizer -train.MomentumOptimizer = tf1.train.MomentumOptimizer -train.MonitoredTrainingSession = tf1.train.MonitoredTrainingSession -train.NewCheckpointReader = tf1.train.NewCheckpointReader -train.Optimizer = tf1.train.Optimizer -train.RMSPropOptimizer = tf1.train.RMSPropOptimizer -train.Saver = tf1.train.Saver -train.SaverDef = tf1.train.SaverDef -train.summary_iterator = tf1.train.summary_iterator -trainable_variables = tf1.trainable_variables -Variable = tf1.Variable -variables_initializer = tf1.variables_initializer -VariableScope = tf1.VariableScope -variance_scaling_initializer = tf1.variance_scaling_initializer -variable_scope = tf1.variable_scope -where = tf1.where -while_loop = tf1.while_loop -wrap_function = tf1.wrap_function - -# Explicit 1.x symbol import. -data.make_initializable_iterator = dataset_ops.make_initializable_iterator -data.make_one_shot_iterator = dataset_ops.make_one_shot_iterator -# For `nn.embedding_lookup`, v2 doesn't have the arg 'partition_strategy' in -# the API, and uses 'partition_strategy="div"' by default; -# while v1 uses 'partition_strategy="mod"' by default. Keep this for now. -nn.embedding_lookup = embedding_ops.embedding_lookup -# pylint: enable=undefined-variable diff --git a/lingvo/compat_test.py b/lingvo/compat_test.py deleted file mode 100644 index 09f1150d..00000000 --- a/lingvo/compat_test.py +++ /dev/null @@ -1,53 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for compat.py.""" - -import lingvo.compat as tf -import tensorflow.compat.v1 as tf1 -import tensorflow.compat.v2 as tf2 -from tensorflow.python.framework import function # pylint:disable=g-direct-tensorflow-import - - -class CompatTest(tf.test.TestCase): - - def testSomeTFSymbols(self): - self.assertIsNotNone(tf.logging) - self.assertIsNotNone(tf.flags) - self.assertIs(tf.Defun, function.Defun) - - def testDoesNotModifyTF2(self): - modules_no_overwritten = [ - (tf2.data, tf1.data), - (tf2.graph_util, tf1.graph_util), - (tf2.image, tf1.image), - (tf2.initializers, tf1.initializers), - (tf2.io, tf1.io), - (tf2.losses, tf1.losses), - (tf2.metrics, tf1.metrics), - (tf2.nn, tf1.nn), - (tf2.random, tf1.random), - (tf2.saved_model, tf1.saved_model), - (tf2.strings, tf1.strings), - (tf2.summary, tf1.summary), - (tf2.test, tf1.test), - (tf2.train, tf1.train), - ] - for modules in modules_no_overwritten: - self.assertIsNot(modules[0], modules[1]) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/BUILD b/lingvo/core/BUILD deleted file mode 100644 index 425bb52b..00000000 --- a/lingvo/core/BUILD +++ /dev/null @@ -1,2154 +0,0 @@ -load( - "//lingvo:lingvo.bzl", - "lingvo_cuda_py_test", - "lingvo_proto_cc", - "lingvo_proto_py", - "lingvo_py_binary", -) - -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -py_library( - name = "attention", - srcs = ["attention.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":layers", - ":py_utils", - ":quant_utils", - ":symbolic", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -lingvo_cuda_py_test( - name = "attention_test", - srcs = ["attention_test.py"], - python_version = "PY3", - shard_count = 5, - deps = [ - ":attention", - ":py_utils", - ":quant_utils", - ":test_utils", - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "base_decoder", - srcs = ["base_decoder.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":beam_search_helper", - ":py_utils", - ":target_sequence_sampler", - "//lingvo:compat", - ], -) - -py_library( - name = "batch_major_attention", - srcs = ["batch_major_attention.py"], - srcs_version = "PY3", - deps = [ - ":attention_util", - ":base_layer", - ":builder", - ":conv_layers_builder", - ":gpipe", - ":hyperparams", - ":layers", - ":layers_with_attention", - ":moe_layers", - ":py_utils", - ":symbolic", - ":tshape", - "//lingvo:compat", - ], -) - -py_library( - name = "batch_utils", - srcs = ["batch_utils.py"], - srcs_version = "PY3", - deps = [ - ":cluster_factory", - ":py_utils", - ], -) - -py_test( - name = "batch_utils_test", - srcs = ["batch_utils_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":batch_utils", - ":cluster_factory", - # Implicit absl.testing.flagsaver dependency. - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - ], -) - -py_library( - name = "beam_search_helper", - srcs = [ - "beam_search_helper.py", - ], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":py_utils", - "//lingvo:compat", - "//lingvo/core/ops", - ], -) - -py_test( - name = "beam_search_helper_test", - srcs = ["beam_search_helper_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":beam_search_helper", - ":py_utils", - ":test_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "bfloat16_variables", - srcs = ["bfloat16_variables.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - ], -) - -py_test( - name = "bfloat16_variables_test", - srcs = ["bfloat16_variables_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":bfloat16_variables", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "constants", - srcs = ["constants.py"], - srcs_version = "PY3", -) - -py_library( - name = "conformer_layer", - srcs = ["conformer_layer.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:batch_major_attention", - "//lingvo/core:bn_layers", - "//lingvo/core:conv_layers_with_time_padding", - "//lingvo/core:layers", - "//lingvo/core:layers_with_attention", - ], -) - -py_test( - name = "conformer_layer_test", - srcs = ["conformer_layer_test.py"], - python_version = "PY3", - shard_count = 2, - srcs_version = "PY3", - deps = [ - ":conformer_layer", - # Implicit absl.testing.flagsaver dependency. - # Implicit absl.testing.parameterized dependency. - "//lingvo/core:bn_layers", - "//lingvo/core:test_utils", - ], -) - -py_library( - name = "target_sequence_sampler", - srcs = [ - "target_sequence_sampler.py", - ], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":py_utils", - ":recurrent", - "//lingvo:compat", - ], -) - -py_test( - name = "target_sequence_sampler_test", - srcs = ["target_sequence_sampler_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":py_utils", - ":target_sequence_sampler", - ":test_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "base_input_generator", - srcs = ["base_input_generator.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":batch_utils", - ":datasource", - ":hyperparams", - ":input_generator_helper", - ":inspect_utils", - ":py_utils", - ":tokenizers", - "//lingvo:compat", - "//lingvo/core/ops", - ], -) - -py_test( - name = "base_input_generator_test", - srcs = ["base_input_generator_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":base_input_generator", - ":cluster_factory", - ":hyperparams", - ":py_utils", - ":test_utils", - # Implicit absl.testing.flagsaver dependency. - "//lingvo:compat", - # Implicit mock dependency. - # Implicit numpy dependency. - ], -) - -py_test( - name = "batch_major_attention_test", - size = "large", - srcs = ["batch_major_attention_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":attention_util", - ":batch_major_attention", - ":py_utils", - ":test_utils", - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "base_layer", - srcs = ["base_layer.py"], - srcs_version = "PY3", - deps = [ - ":cluster_factory", - ":hyperparams", - ":py_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "bn_layers", - srcs = ["bn_layers.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":py_utils", - ":summary_utils", - "//lingvo:compat", - ], -) - -py_test( - name = "base_layer_test", - size = "small", - srcs = ["base_layer_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":base_layer", - ":hyperparams", - ":py_utils", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "reversible_layers", - srcs = ["reversible_layers.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":py_utils", - "//lingvo:compat", - ], -) - -py_test( - name = "reversible_layers_test", - srcs = ["reversible_layers_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":layers", - ":py_utils", - ":reversible_layers", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "decoder_lib", - srcs = ["decoder_lib.py"], - srcs_version = "PY3", - deps = [ - ], -) - -py_library( - name = "base_model", - srcs = ["base_model.py"], - srcs_version = "PY3", - deps = [ - ":base_input_generator", - ":base_layer", - ":build_data", - ":cluster_factory", - ":early_stop", - ":hyperparams", - ":learner", - ":optimizer", - ":py_utils", - ":schedule", - ":summary_utils", - ":task_scheduler", - "//lingvo/core:decoder_lib", - # Implicit model_pruning.pruning dependency. - "//lingvo:compat", - ], -) - -py_test( - name = "base_model_test", - srcs = ["base_model_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":base_decoder", - ":base_input_generator", - ":base_layer", - ":base_model", - ":base_model_params", - ":distillation_task", - ":hyperparams", - ":layers", - ":learner", - ":py_utils", - ":task_scheduler", - ":test_utils", - # Implicit absl.testing.flagsaver dependency. - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "base_model_params", - srcs = ["base_model_params.py"], - srcs_version = "PY3", - deps = [ - ":base_input_generator", - ":base_model", - ":hyperparams", - "//lingvo:datasets_lib", - ], -) - -py_test( - name = "base_model_params_test", - srcs = ["base_model_params_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":base_model_params", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "build_data", - srcs = ["build_data.py"], - srcs_version = "PY3", - deps = [ - ":hyperparams", - # Implicit build_data dependency. - ], -) - -py_library( - name = "builder", - srcs = ["builder.py"], - srcs_version = "PY3", - deps = [ - ":builder_layers", - ":hyperparams", - ":layers", - ":py_utils", - ":tshape", - "//lingvo:compat", - ], -) - -py_library( - name = "builder_layers", - srcs = ["builder_layers.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":computation_cost", - ":py_utils", - ":recurrent", - ":summary_utils", - ":symbolic", - ":tshape", - "//lingvo:compat", - ], -) - -py_test( - name = "builder_layers_test", - srcs = ["builder_layers_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":builder_layers", - ":layers", - ":py_utils", - ":test_utils", - ":tshape", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_test( - name = "builder_test", - srcs = ["builder_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":builder", - ":cluster_factory", - ":layers", - ":py_utils", - ":test_utils", - ":tshape", - "//lingvo:compat", - # Implicit numpy dependency. - # Implicit sympy dependency. - ], -) - -py_library( - name = "cluster", - srcs = ["cluster.py"], - srcs_version = "PY3", - deps = [ - ":hyperparams", - ":py_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "cluster_factory", - srcs = ["cluster_factory.py"], - srcs_version = "PY3", - deps = [ - "//lingvo/core:cluster", - ], -) - -py_test( - name = "cluster_test", - size = "small", - srcs = ["cluster_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":cluster_factory", - ":test_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "datasource", - srcs = ["datasource.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":py_utils", - "//lingvo:compat", - ], -) - -py_test( - name = "datasource_test", - size = "small", - srcs = ["datasource_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":datasource", - ":py_utils", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "matrix_functions", - srcs = ["matrix_functions.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core/ops", - # Implicit numpy dependency. - ], -) - -py_test( - name = "matrix_functions_test", - size = "small", - srcs = ["matrix_functions_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":distributed_shampoo", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "distributed_shampoo", - srcs = ["distributed_shampoo.py"], - srcs_version = "PY3", - deps = [ - ":matrix_functions", - "//lingvo:compat", - "//lingvo/core/ops", - # Implicit numpy dependency. - ], -) - -py_test( - name = "distributed_shampoo_test", - size = "small", - srcs = ["distributed_shampoo_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":distributed_shampoo", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "early_stop", - srcs = ["early_stop.py"], - srcs_version = "PY3", - deps = [ - ":hyperparams", - "//lingvo:compat", - "//lingvo/core/ops", - ], -) - -py_test( - name = "early_stop_test", - size = "small", - srcs = ["early_stop_test.py"], - data = [ - "//lingvo/core/ops/testdata:best_step_testdata", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":early_stop", - ":hyperparams", - ":test_helper", - ":test_utils", - "//lingvo:compat", - ], -) - -lingvo_proto_cc( - name = "inference_graph_proto", - src = "inference_graph.proto", -) - -lingvo_proto_py( - name = "inference_graph_py_pb2", - src = "inference_graph.proto", - deps = [":inference_graph_proto"], -) - -py_library( - name = "hyperparams", - srcs = ["hyperparams.py"], - srcs_version = "PY3", - deps = [ - ":hyperparams_py_pb2", - ":symbolic", - # Implicit python proto dependency. - # Implicit dataclasses dependency. - "//lingvo:compat", - ], -) - -lingvo_proto_cc( - name = "hyperparams_proto", - src = "hyperparams.proto", -) - -lingvo_proto_py( - name = "hyperparams_py_pb2", - src = "hyperparams.proto", - deps = [":hyperparams_proto"], -) - -py_test( - name = "hyperparams_test", - srcs = ["hyperparams_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":hyperparams", - ":hyperparams_py_pb2", - ":symbolic", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "inference_graph_exporter", - srcs = ["inference_graph_exporter.py"], - srcs_version = "PY3", - deps = [ - ":base_model", - ":bfloat16_variables", - ":inference_graph_py_pb2", - ":py_utils", - # Implicit python proto dependency. - "//lingvo:compat", - # Implicit six dependency. - ], -) - -py_test( - name = "inference_graph_exporter_test", - srcs = ["inference_graph_exporter_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":base_input_generator", - ":base_model", - ":base_model_params", - ":inference_graph_exporter", - ":inference_graph_py_pb2", - ":predictor_lib", - ":py_utils", - ":test_utils", - "//lingvo:compat", - "//lingvo:model_registry", - ], -) - -py_library( - name = "input_generator_helper", - srcs = ["input_generator_helper.py"], - srcs_version = "PY3", - deps = [ - ":py_utils", - "//lingvo:compat", - ], -) - -py_test( - name = "input_generator_helper_test", - srcs = ["input_generator_helper_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":input_generator_helper", - ":test_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "inspect_utils", - srcs = ["inspect_utils.py"], - srcs_version = "PY3", -) - -py_test( - name = "inspect_utils_test", - srcs = ["inspect_utils_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":hyperparams", - ":inspect_utils", - ":test_utils", - ], -) - -py_library( - name = "layers", - srcs = ["layers.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":bn_layers", - ":builder_layers", - ":computation_cost", - ":constants", - ":conv_layers_with_time_padding", - ":pruning_utils", - ":py_utils", - ":quant_utils", - ":recurrent", - ":schedule", - ":summary_utils", - ":symbolic", - ":tshape", - "//lingvo:compat", - # Implicit numpy dependency. - # Implicit sympy dependency. - ], -) - -py_library( - name = "learner", - srcs = ["learner.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":gradient_combiner", - ":optimizer", - ":py_utils", - ":schedule", - ":summary_utils", - "//lingvo:compat", - ], -) - -py_test( - name = "learner_test", - srcs = ["learner_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":base_layer", - ":learner", - ":optimizer", - ":py_utils", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "conv_layers_with_time_padding", - srcs = ["conv_layers_with_time_padding.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":bn_layers", - ":py_utils", - ":symbolic", - ":tshape", - "//lingvo:compat", - ], -) - -py_library( - name = "conv_layers_builder", - srcs = ["conv_layers_builder.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":builder", - ":builder_layers", - ":conv_layers_with_time_padding", - ":py_utils", - # Implicit absl.flags dependency. - "//lingvo:compat", - ], -) - -lingvo_cuda_py_test( - name = "layers_test", - srcs = ["layers_test.py"], - python_version = "PY3", - shard_count = 20, - deps = [":layers_test_lib"], -) - -lingvo_cuda_py_test( - name = "conv_layers_builder_test", - srcs = ["conv_layers_builder_test.py"], - python_version = "PY3", - deps = [ - ":bn_layers", - ":conv_layers_builder", - ":conv_layers_with_time_padding", - ":layers", - ":test_utils", - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -lingvo_cuda_py_test( - name = "conv_layers_with_time_padding_test", - srcs = ["conv_layers_with_time_padding_test.py"], - python_version = "PY3", - deps = [":layers_test_lib"], -) - -py_library( - name = "layers_test_lib", - testonly = 1, - srcs = ["layers_test.py"], - srcs_version = "PY3", - deps = [ - ":cluster_factory", - ":gpipe", - ":layers", - ":py_utils", - ":quant_utils", - ":symbolic", - ":test_utils", - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "saver", - srcs = ["saver.py"], - srcs_version = "PY3", - deps = [ - ":py_utils", - # Implicit python proto dependency. - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "schedule", - srcs = ["schedule.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":early_stop", - ":py_utils", - "//lingvo:compat", - "//lingvo/core/ops", - ], -) - -py_test( - name = "saver_test", - srcs = ["saver_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":cluster_factory", - ":py_utils", - ":saver", - ":test_utils", - "//lingvo:compat", - "//lingvo/tasks/image/params:mnist", - # Implicit numpy dependency. - ], -) - -py_test( - name = "schedule_test", - size = "small", - timeout = "moderate", - srcs = ["schedule_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":cluster_factory", - ":early_stop", - ":schedule", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "metrics", - srcs = ["metrics.py"], - srcs_version = "PY3", - deps = [ - ":hyperparams", - ":plot", - ":py_utils", - ":scorers", - "//lingvo:compat", - # Implicit numpy dependency. - # Implicit sklearn dependency. # buildcleaner: keep - ], -) - -py_library( - name = "ml_perf_bleu_metric", - srcs = ["ml_perf_bleu_metric.py"], - srcs_version = "PY3", - deps = [ - ":metrics", - "//lingvo:compat", - # Implicit numpy dependency. - # Implicit six dependency. - ], -) - -py_test( - name = "ml_perf_bleu_metric_test", - srcs = ["ml_perf_bleu_metric_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ml_perf_bleu_metric", - ":test_utils", - "//lingvo:compat", - ], -) - -py_test( - name = "metrics_test", - srcs = ["metrics_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":metrics", - ":py_utils", - ":test_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "multitask_model", - srcs = ["multitask_model.py"], - srcs_version = "PY3", - deps = [ - ":base_model", - ":py_utils", - "//lingvo:compat", - ], -) - -py_test( - name = "multitask_model_test", - srcs = ["multitask_model_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":base_input_generator", - ":base_layer", - ":base_model", - ":base_model_params", - ":hyperparams", - ":multitask_model", - ":py_utils", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "adagraft", - srcs = [ - "adagraft.py", - ], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core/ops", - # Implicit numpy dependency. - ], -) - -py_test( - name = "adagraft_test", - size = "small", - srcs = ["adagraft_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":adagraft", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "optimizer", - srcs = ["optimizer.py"], - srcs_version = "PY3", - deps = [ - ":adagraft", - ":base_layer", - ":distributed_shampoo", - ":py_utils", - ":summary_utils", - "//lingvo:compat", - ], -) - -py_test( - name = "optimizer_test", - size = "small", - srcs = ["optimizer_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":layers", - ":optimizer", - ":py_utils", - ":test_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "plot", - srcs = ["plot.py"], - srcs_version = "PY3", - deps = [ - ":py_utils", - # Implicit Tkinter dependency. # buildcleaner: keep - "//lingvo:compat", - # Implicit matplotlib dependency. - # Implicit numpy dependency. - # Implicit six dependency. - ], -) - -py_test( - name = "plot_test", - srcs = ["plot_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":plot", - ":test_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "predictor_lib", - srcs = ["predictor.py"], - srcs_version = "PY3", - deps = [ - ":inference_graph_py_pb2", - ":py_utils", - # Implicit python proto dependency. - # Implicit IPython dependency. - "//lingvo:compat", - "//lingvo:model_imports_no_params", - ], -) - -lingvo_py_binary( - name = "predictor", - srcs = ["predictor.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":inference_graph_py_pb2", - ":py_utils", - # Implicit python proto dependency. - "//lingvo:compat", - "//lingvo:model_imports_no_params", - ], -) - -py_test( - name = "predictor_test", - size = "medium", - srcs = ["predictor_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":base_input_generator", - ":base_model", - ":inference_graph_exporter", - ":inference_graph_py_pb2", - ":predictor_lib", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "predictor_runner_base", - srcs = ["predictor_runner_base.py"], - srcs_version = "PY3", - deps = [ - ":predictor_lib", - ":py_utils", - # Implicit absl.flags dependency. - # Implicit concurrent.futures dependency. - "//lingvo:compat", - # Implicit six dependency. - ], -) - -py_library( - name = "pruning_utils", - srcs = ["pruning_utils.py"], - srcs_version = "PY3", - deps = [ - # Implicit model_pruning.pruning dependency. - "//lingvo:compat", - ], -) - -py_library( - name = "py_utils", - srcs = ["py_utils.py"], - srcs_version = "PY3", - deps = [ - ":hyperparams", - ":retry", - ":symbolic", - ":tshape", - "//lingvo:compat", - "//lingvo/core/ops", - # Implicit numpy dependency. - # Implicit six dependency. - # Implicit tensorflow py proto dependency. - ], -) - -py_test( - name = "py_utils_test", - size = "medium", - srcs = ["py_utils_test.py"], - data = [ - "//lingvo/core/testdata:lenet_test_model", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":base_layer", - ":builder_layers", - ":cluster_factory", - ":hyperparams", - ":py_utils", - ":recurrent", - ":symbolic", - ":test_helper", - ":test_utils", - # Implicit absl.testing.flagsaver dependency. - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - "//lingvo:model_registry", - "//lingvo/tasks/image/params:mnist", - # Implicit mock dependency. - # Implicit numpy dependency. - ], -) - -py_library( - name = "quant_utils", - srcs = ["quant_utils.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":hyperparams", - ":py_utils", - ":summary_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_test( - name = "quant_utils_test", - size = "small", - srcs = ["quant_utils_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":py_utils", - ":quant_utils", - ":test_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "layers_with_gpipe", - srcs = ["layers_with_gpipe.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":batch_major_attention", - ":gpipe", - ":layers", - ":layers_with_attention", - ":py_utils", - ":tshape", - "//lingvo:compat", - ], -) - -py_test( - name = "layers_with_gpipe_test", - size = "large", - srcs = ["layers_with_gpipe_test.py"], - python_version = "PY3", - shard_count = 15, - srcs_version = "PY3", - deps = [ - ":layers_with_gpipe", - ":py_utils", - ":test_utils", - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "spectrum_augmenter", - srcs = ["spectrum_augmenter.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":py_utils", - "//lingvo:compat", - ], -) - -py_test( - name = "spectrum_augmenter_test", - srcs = ["spectrum_augmenter_test.py"], - python_version = "PY3", - shard_count = 1, - srcs_version = "PY3", - deps = [ - ":spectrum_augmenter", - ":test_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "spectrum_augmenter_on_device", - srcs = ["spectrum_augmenter_on_device.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":py_utils", - ":spectrum_augmenter", - "//lingvo:compat", - ], -) - -py_test( - name = "spectrum_augmenter_on_device_test", - srcs = ["spectrum_augmenter_on_device_test.py"], - python_version = "PY3", - shard_count = 1, - srcs_version = "PY3", - deps = [ - ":spectrum_augmenter", - ":spectrum_augmenter_on_device", - ":test_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "generic_input", - srcs = ["generic_input.py"], - srcs_version = "PY3", - deps = [ - ":py_utils", - "//lingvo:compat", - "//lingvo/core/ops", - # Implicit tensorflow python framework_for_generated_wrappers dependency. - ], -) - -py_test( - name = "generic_input_test", - srcs = ["generic_input_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":generic_input", - ":py_utils", - ":test_utils", - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "gpipe", - srcs = ["gpipe.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":builder_layers", - ":py_utils", - ":recurrent", - ":tshape", - "//lingvo:compat", - ], -) - -py_test( - name = "gpipe_test", - srcs = ["gpipe_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":base_layer", - ":gpipe", - ":layers", - ":py_utils", - ":test_utils", - ":tshape", - "//lingvo:compat", - ], -) - -py_library( - name = "recurrent", - srcs = ["recurrent.py"], - srcs_version = "PY3", - deps = [ - ":cluster_factory", - ":constants", - ":py_utils", - ":sendrecv", - ":symbolic", - "//lingvo:compat", - ], -) - -py_test( - name = "recurrent_test", - srcs = ["recurrent_test.py"], - python_version = "PY3", - shard_count = 6, - srcs_version = "PY3", - deps = [":recurrent_test_lib"], -) - -py_library( - name = "recurrent_test_lib", - testonly = 1, - srcs = ["recurrent_test.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":py_utils", - ":recurrent", - ":symbolic", - ":test_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -lingvo_cuda_py_test( - name = "recurrent_gpu_test", - srcs = ["recurrent_gpu_test.py"], - python_version = "PY3", - shard_count = 6, - deps = [ - ":py_utils", - ":recurrent", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "attention_util", - srcs = ["attention_util.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":layers", - ":py_utils", - ":summary_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "retry", - srcs = ["retry.py"], - srcs_version = "PY3", - deps = [ - # Implicit retry dependency. - ], -) - -py_test( - name = "attention_util_test", - srcs = ["attention_util_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":attention_util", - ":py_utils", - ":test_utils", - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_test( - name = "retry_test", - srcs = ["retry_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":retry", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "self_attention_layer", - srcs = ["self_attention_layer.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":batch_major_attention", - ":py_utils", - ], -) - -py_library( - name = "sendrecv", - srcs = ["sendrecv.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - ], -) - -py_test( - name = "self_attention_layer_test", - timeout = "long", - srcs = ["self_attention_layer_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":batch_major_attention", - ":py_utils", - ":self_attention_layer", - ":test_utils", - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_test( - name = "sendrecv_test", - srcs = ["sendrecv_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":sendrecv", - ":test_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "summary_utils", - srcs = ["summary_utils.py"], - srcs_version = "PY3", - deps = [ - ":cluster_factory", - ":plot", - ":py_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "task_scheduler", - srcs = ["task_scheduler.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":early_stop", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_test( - name = "task_scheduler_test", - srcs = ["task_scheduler_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":early_stop", - ":task_scheduler", - ":test_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "test_helper", - srcs = ["test_helper.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - ], -) - -py_library( - name = "test_utils", - srcs = ["test_utils.py"], - srcs_version = "PY3", - deps = [ - ":cluster_factory", - ":py_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_test( - name = "test_utils_test", - srcs = ["test_utils_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "trainer_test_utils", - srcs = ["trainer_test_utils.py"], - srcs_version = "PY3", - deps = [ - ":base_input_generator", - ":base_model", - ":py_utils", - ":summary_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "test_trainer_utils", - srcs = ["test_trainer_utils.py"], - srcs_version = "PY3", - deps = [ - ":test_utils", - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - "//lingvo:trainer_lib", - ], -) - -py_library( - name = "wpm_encoder", - srcs = ["wpm_encoder.py"], - srcs_version = "PY3", - deps = [ - ":py_utils", - "//lingvo:compat", - "//lingvo/core/ops", - # Implicit six dependency. - ], -) - -py_test( - name = "wpm_encoder_test", - srcs = ["wpm_encoder_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":test_utils", - ":wpm_encoder", - "//lingvo:compat", - ], -) - -py_library( - name = "tokenizers", - srcs = ["tokenizers.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":py_utils", - ":wpm_encoder", - "//lingvo:compat", - "//lingvo/core/ops", - # Implicit numpy dependency. - ], -) - -py_library( - name = "tpu_summary", - srcs = ["tpu_summary.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - ], -) - -py_test( - name = "wpm_tokenizer_test", - srcs = ["wpm_tokenizer_test.py"], - data = [ - "//lingvo/tasks/mt:wpm_ende", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":test_helper", - ":test_utils", - ":tokenizers", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "rnn_cell", - srcs = ["rnn_cell.py"], - srcs_version = "PY3", - deps = [ - ":hyperparams", - ":layers", - ":pruning_utils", - ":py_utils", - ":quant_utils", - ":summary_utils", - "//lingvo:compat", - ], -) - -lingvo_cuda_py_test( - name = "rnn_cell_test", - srcs = ["rnn_cell_test.py"], - python_version = "PY3", - deps = [ - ":py_utils", - ":quant_utils", - ":rnn_cell", - ":test_utils", - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "layers_with_attention", - srcs = ["layers_with_attention.py"], - srcs_version = "PY3", - deps = [ - ":attention", - ":base_layer", - ":layers", - ":py_utils", - ":symbolic", - "//lingvo:compat", - ], -) - -py_test( - name = "layers_with_attention_test", - size = "large", - srcs = ["layers_with_attention_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":attention", - ":layers", - ":layers_with_attention", - ":py_utils", - ":test_utils", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "rnn_layers", - srcs = ["rnn_layers.py"], - srcs_version = "PY3", - deps = [ - ":attention", - ":base_layer", - ":layers", - ":py_utils", - ":quant_utils", - ":recurrent", - ":rnn_cell", - "//lingvo:compat", - ], -) - -lingvo_cuda_py_test( - name = "rnn_layers_test", - size = "large", - srcs = ["rnn_layers_test.py"], - python_version = "PY3", - shard_count = 15, - deps = [":rnn_layers_test_lib"], -) - -py_library( - name = "rnn_layers_test_lib", - testonly = 1, - srcs = ["rnn_layers_test.py"], - srcs_version = "PY3", - deps = [ - ":attention", - ":base_layer", - ":cluster_factory", - ":layers_with_attention", - ":py_utils", - ":rnn_cell", - ":rnn_layers", - ":test_utils", - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "lstm_frnn_layer", - srcs = ["lstm_frnn_layer.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":recurrent", - ":rnn_cell", - ":rnn_layers", - # Implicit absl.flags dependency. - "//lingvo:compat", - ], -) - -py_test( - name = "lstm_frnn_layer_test", - srcs = ["lstm_frnn_layer_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":lstm_frnn_layer", - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - ], -) - -py_library( - name = "scorers", - srcs = ["scorers.py"], - srcs_version = "PY3", - deps = [ - # Implicit six dependency. - ], -) - -py_test( - name = "scorers_test", - srcs = ["scorers_test.py"], - data = ["//lingvo/core/ops/testdata:wmt"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":scorers", - ":test_helper", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "model_helper", - srcs = ["model_helper.py"], - srcs_version = "PY3", - deps = [ - ":rnn_layers", - "//lingvo:compat", - ], -) - -py_library( - name = "moe_layers", - srcs = ["moe_layers.py"], - deps = [ - ":base_layer", - ":py_utils", - ":tpu_summary", - "//lingvo:compat", - ], -) - -py_library( - name = "tshape", - srcs = ["tshape.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - # Implicit sympy dependency. - ], -) - -py_test( - name = "tpu_summary_test", - srcs = ["tpu_summary_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":test_utils", - ":tpu_summary", - "//lingvo:compat", - ], -) - -py_test( - name = "tshape_test", - srcs = ["tshape_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":test_utils", - ":tshape", - "//lingvo:compat", - ], -) - -py_library( - name = "symbolic", - srcs = ["symbolic.py"], - srcs_version = "PY3", - deps = [ - # Implicit sympy dependency. - ], -) - -py_test( - name = "symbolic_test", - srcs = ["symbolic_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":symbolic", - ":test_utils", - "//lingvo:compat", - # Implicit sympy dependency. - ], -) - -py_library( - name = "insertion", - srcs = ["insertion.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":py_utils", - "//lingvo:compat", - ], -) - -lingvo_cuda_py_test( - name = "insertion_test", - srcs = ["insertion_test.py"], - python_version = "PY3", - deps = [ - ":insertion", - ":test_utils", - ], -) - -py_library( - name = "computation_cost", - srcs = ["computation_cost.py"], - srcs_version = "PY3", - deps = [ - ":bn_layers", - "//lingvo:compat", - ], -) - -py_library( - name = "checkpointer_lib", - srcs = ["checkpointer.py"], - srcs_version = "PY3", - deps = [ - ":cluster_factory", - ":py_utils", - "//lingvo:compat", - # Implicit six dependency. - ], -) - -py_library( - name = "program_lib", - srcs = ["program.py"], - srcs_version = "PY3", - deps = [ - ":base_model", - ":checkpointer_lib", - ":cluster_factory", - ":hyperparams", - ":metrics", - ":ml_perf_log", - ":py_utils", - ":summary_utils", - "//lingvo:compat", - # Implicit tensorflow grpc dependency. - ], -) - -py_library( - name = "step", - srcs = ["step.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":builder_layers", - ":py_utils", - ":recurrent", - "//lingvo:compat", - ], -) - -py_test( - name = "step_test", - size = "small", - srcs = ["step_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":builder_layers", - ":py_utils", - ":step", - ":test_utils", - "//lingvo:compat", - ], -) - -py_test( - name = "summary_utils_test", - srcs = ["summary_utils_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":cluster_factory", - ":summary_utils", - ":test_utils", - "//lingvo:compat", - ], -) - -py_test( - name = "checkpointer_test", - srcs = ["checkpointer_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":base_input_generator", - ":base_model", - ":checkpointer_lib", - ":py_utils", - ":test_utils", - "//lingvo:compat", - ], -) - -py_test( - name = "ema_test", - srcs = ["ema_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":base_decoder", - ":base_input_generator", - ":base_model", - ":checkpointer_lib", - ":layers", - ":py_utils", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "ml_perf_tokenizer", - srcs = ["ml_perf_tokenizer.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":py_utils", - ":tokenizers", - "//lingvo:compat", - "//lingvo/core/ops", - # Implicit numpy dependency. - ], -) - -py_library( - name = "ml_perf_log", - srcs = ["ml_perf_log.py"], - srcs_version = "PY3", -) - -py_test( - name = "ml_perf_log_test", - srcs = ["ml_perf_log_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ml_perf_log", - "//lingvo:compat", - ], -) - -py_library( - name = "egdd", - srcs = [ - "egdd.py", - ], - srcs_version = "PY3", - deps = [ - # Implicit numpy dependency. - # Implicit tensorflow dependency. - ], -) - -py_test( - name = "egdd_test", - size = "large", - srcs = ["egdd_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":egdd", - ":test_utils", - "//lingvo:compat", - ], -) - -py_library( - name = "distillation_task", - srcs = ["distillation_task.py"], - srcs_version = "PY3", - deps = [ - ":base_model", - ":py_utils", - ":schedule", - "//lingvo:compat", - ], -) - -py_library( - name = "gradient_combiner", - srcs = ["gradient_combiner.py"], - srcs_version = "PY3", - deps = [":base_layer"], -) - -py_library( - name = "tpu_embedding_layers", - srcs = ["tpu_embedding_layers.py"], - srcs_version = "PY3", - deps = [ - ":base_layer", - ":py_utils", - "//lingvo:compat", - ], -) diff --git a/lingvo/core/adagraft.py b/lingvo/core/adagraft.py deleted file mode 100644 index 0809b194..00000000 --- a/lingvo/core/adagraft.py +++ /dev/null @@ -1,185 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""AdaGraft optimizer https://arxiv.org/abs/2002.11803 .""" - -import lingvo.compat as tf - - -class AdaGraftOptimizer(tf.train.Optimizer): - """Optimizer which combines per-layer direction and magnitude from two optimizers. - - Disentangling Adaptive Gradient Methods from Learning Rates - Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, Cyril Zhang - https://arxiv.org/abs/2002.11803 - """ - - def __init__(self, - learning_rate, - magnitude_optimizer, - direction_optimizer, - diagnostic=False, - use_global_norm=False, - name="AdaGraft"): - """Construct a new AdaGraft optimizer. - - Args: - learning_rate: A `Tensor` or a floating point value. The learning rate. - magnitude_optimizer: Child Optimizer to inherit step sizes. - direction_optimizer: Child Optimizer to inherit step directions. - diagnostic: Whether to record per-tensor step norms. - use_global_norm: Graft global l2 norms rather than per-layer. - name: Optional name prefix for the operations created when applying - gradients. - """ - super().__init__(False, name) - self._learning_rate = learning_rate - self.magnitude_optimizer = magnitude_optimizer - self.direction_optimizer = direction_optimizer - self.diagnostic = diagnostic - self.use_global_norm = use_global_norm - - def _create_slots(self, var_list): - self.magnitude_optimizer._create_slots(var_list) # pylint: disable=protected-access - self.direction_optimizer._create_slots(var_list) # pylint: disable=protected-access - - for v in var_list: - with tf.ops.colocate_with(v): - self._zeros_slot(v, "scratch_copy", self._name) - if self.diagnostic or self.use_global_norm: - self._get_or_make_slot(v, tf.constant(0.0), "m_step_norm", self._name) - self._get_or_make_slot(v, tf.constant(0.0), "d_step_norm", self._name) - - def _prepare(self): - self.magnitude_optimizer._prepare() # pylint: disable=protected-access - self.direction_optimizer._prepare() # pylint: disable=protected-access - - learning_rate = self._call_if_callable(self._learning_rate) - self._learning_rate_tensor = tf.convert_to_tensor( - learning_rate, name="learning_rate") - - if self.use_global_norm: # create list of all vars for global _finish - self._variables = [] - - def _apply_dense(self, grad, var): # pylint: disable=g-doc-args - return self._internal_apply_dense( - grad, - var, - self.magnitude_optimizer._apply_dense, # pylint: disable=protected-access - self.direction_optimizer._apply_dense) # pylint: disable=protected-access - - def _resource_apply_dense(self, grad, var): - return self._internal_apply_dense( - grad, - var, - self.magnitude_optimizer._resource_apply_dense, # pylint: disable=protected-access - self.direction_optimizer._resource_apply_dense) # pylint: disable=protected-access - - def _internal_apply_dense(self, grad, var, magnitude_optimizer_apply_fn, - direction_optimizer_apply_fn): # pylint: disable=g-doc-args - """Main optimization logic of AdaGraft, which calls the child optimizers. - - Args: - grad: Tensor containing gradients. - var: Tensor containing parameter values. - magnitude_optimizer_apply_fn: Apply magnitude optimizer. - direction_optimizer_apply_fn: Apply direction optimizer. - - Returns: - The final update op, which increments var by the grafted step. - - Pseudocode: - - Copy weights into scratch space 'scratch_copy'. - - Run magnitude_optimizer in-place. - - Use scratch copy to figure out how far we moved ('magnitude_step'). - - Copy weights back. - - Run direction_optimizer in-place. - - Move weights along the line segment with scratch_copy. - """ - - if self.use_global_norm: - self._variables.append(var) - - # Slot with current parameter values - scratch_slot = self.get_slot(var, "scratch_copy") - old_var = tf.assign(scratch_slot, var) - - with tf.control_dependencies([old_var]): - m_updated_var = magnitude_optimizer_apply_fn(grad, var) # pylint: disable=protected-access - - # Run magnitude optimizer and compute the norm of the update. - with tf.control_dependencies([m_updated_var]): - m_step = var - old_var - m_step_norm = tf.norm(m_step) - if self.diagnostic or self.use_global_norm: - m_step_norm = tf.assign(self.get_slot(var, "m_step_norm"), m_step_norm) - - # Run direction optimizer and compute its norm, and the direction. - with tf.control_dependencies([m_step_norm]): - flushed_var = tf.assign(var, old_var) - with tf.control_dependencies([flushed_var]): - d_updated_var = direction_optimizer_apply_fn(grad, var) # pylint: disable=protected-access - - # Run an update of the direction optimizer with magnitude optimizer norm. - with tf.control_dependencies([d_updated_var]): - d_step = var - old_var - d_step_norm = tf.norm(d_step) - if self.diagnostic or self.use_global_norm: - d_step_norm = tf.assign(self.get_slot(var, "d_step_norm"), d_step_norm) - if self.use_global_norm: - flushed_var = tf.assign(var, old_var) - with tf.control_dependencies([d_step_norm, flushed_var]): - return tf.assign(scratch_slot, d_step) - step = tf.where( - tf.greater(d_step_norm, 0), - (m_step_norm / tf.maximum(d_step_norm, 1e-30)) * d_step, - tf.zeros_like(d_step)) - return tf.assign(var, old_var + self._learning_rate_tensor * step) - - def _finish(self, update_ops, name_scope): - with tf.control_dependencies(update_ops): - ops1 = self.magnitude_optimizer._finish([], name_scope + "_m") # pylint: disable=protected-access - ops2 = self.direction_optimizer._finish([], name_scope + "_d") # pylint: disable=protected-access - - if self.use_global_norm: # apply global grafting - with tf.control_dependencies([ops1, ops2]): - m_global_norm = tf.Variable(0.) - d_global_norm = tf.Variable(0.) - for var in self._variables: - m_step_norm = self.get_slot(var, "m_step_norm") - d_step_norm = self.get_slot(var, "d_step_norm") - tf.assign_add(m_global_norm, m_step_norm**2) - tf.assign_add(d_global_norm, d_step_norm**2) - - multiplier = tf.sqrt(m_global_norm / tf.maximum(d_global_norm, 1e-30)) - - step_ops = [] - for var in self._variables: - d_step = self.get_slot(var, "scratch_copy") - step = tf.where( - tf.greater(d_step_norm, 0), multiplier * d_step, - tf.zeros_like(d_step)) - step_op = tf.assign_add(var, self._learning_rate_tensor * step) - step_ops.append(step_op) - return tf.group(*step_ops, name=name_scope) - - return tf.group(*([ops1, ops2] + update_ops), name=name_scope) - - # Sparse gradients are not handled currently and is part of future work. - def _resource_apply_sparse(self, grad_values, var, grad_indices): - return tf.no_op() - - def _apply_sparse(self, grad, var): - return tf.no_op() diff --git a/lingvo/core/adagraft_test.py b/lingvo/core/adagraft_test.py deleted file mode 100644 index 5a468a9d..00000000 --- a/lingvo/core/adagraft_test.py +++ /dev/null @@ -1,83 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -# Lint as: python3 -"""Tests for AdaGraft.""" - -import lingvo.compat as tf -from lingvo.core import adagraft -from lingvo.core import test_utils - - -class AdagraftTest(test_utils.TestCase): - - def test_identity(self): - # AdaGraft(1, opt, opt) should do the same thing as opt. - opt1 = tf.train.AdamOptimizer(0.5, beta1=0.5, beta2=0.5) - opt2 = tf.train.AdamOptimizer(0.5, beta1=0.5, beta2=0.5) - opt3 = tf.train.AdamOptimizer(0.5, beta1=0.5, beta2=0.5) - opt = adagraft.AdaGraftOptimizer(1.0, opt1, opt2) - with self.cached_session(): - var0 = tf.Variable(2.0, name="var0") - var1 = tf.Variable(3.0, name="var1") - loss = (var0 - 1) * (var0 - 1) + (var1 - 1) * (var1 - 1) - o = opt.minimize(loss) - oo = opt3.minimize(loss) - self.evaluate(tf.global_variables_initializer()) - self.evaluate(o) - l1 = self.evaluate([loss, var0, var1]) - print(l1) - self.evaluate([tf.assign(var0, 2.0), tf.assign(var1, 3.0)]) - self.evaluate(oo) - l2 = self.evaluate([loss, var0, var1]) - print(l2) - self.assertAllClose(l1, l2) - - def test_step(self): - """Tests grafting of Adam and SGD steps. - - Derivation of one step of Adam and SGD: - Gradient value is [2,4]. - Adam Derivation: - Lr_1 = 0.5(1-0.6)^(0.5)/(1-0.5) = 0.63245553203 - Does not matter - m_1 = 0.5*G = [1,2] - v_1 = 0.4*G^2 = [1.6,6.4] - AdamStep = Lr_1*m_1/(sqrt{v_1}+eps) = [0.5, 0.5] - Normalized AdamStep = [1.0, 1.0] - SGDStep = [0.6, 1.2] Norm = [0.6, 1.2] - TotalStep = 0.9*[0.6, 1.2] - NewVar = [1.46, 1.92] - """ - opt1 = tf.train.GradientDescentOptimizer(0.3) - opt2 = tf.train.AdamOptimizer(0.5, beta1=0.5, beta2=0.6) - opt = adagraft.AdaGraftOptimizer(0.9, opt1, opt2) - with self.cached_session(): - var0 = tf.Variable(2.0, name="var0") - var1 = tf.Variable(3.0, name="var1") - loss = (var0 - 1) * (var0 - 1) + (var1 - 1) * (var1 - 1) - o = opt.minimize(loss) - self.evaluate(tf.global_variables_initializer()) - - correct_values = [[1.058, 1.46, 1.92], [0.22387284, 1.2116001, 1.4232]] - - for i in range(2): - self.evaluate(o) - step_values = self.evaluate([loss, var0, var1]) - print(step_values) - self.assertAllClose(correct_values[i], step_values) - - -if __name__ == "__main__": - tf.test.main() diff --git a/lingvo/core/attention.py b/lingvo/core/attention.py deleted file mode 100644 index a1a87893..00000000 --- a/lingvo/core/attention.py +++ /dev/null @@ -1,3277 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Attention models.""" - -import math -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import layers -from lingvo.core import py_utils -from lingvo.core import quant_utils -from lingvo.core import symbolic - -import numpy as np - -from tensorflow.python.ops import inplace_ops # pylint:disable=g-direct-tensorflow-import - - -# Currently, quantization statistics cannot be accumulated across arbitrary -# defuns, so we allow them to be disabled. A potentially more robust fix is -# to save and merge the attention state across the defun boundary as is -# done in recurrent.py. -def _ConditionalCallDefun(cond, f, inputs): - if not cond: - return f(inputs) - return py_utils.CallDefun(f, inputs) - - -def _ApplyAttentionDropout(params, x, global_step): - """Apply attention dropout according to the given parameters. - - If `params.atten_dropout_deterministic` is set to True, the dropout will be - fully deterministic. - - Args: - params: The parameters of attention layer. - x: A float Tensor on which to apply dropout. - global_step: Required for deterministic dropout. - - Returns: - A Tensor with the same shape as `x`. - """ - if params.atten_dropout_prob == 0: - return x - - if params.atten_dropout_deterministic: - seeds = py_utils.GenerateStepSeedPair(params, global_step) - return py_utils.DeterministicDropout(x, 1.0 - params.atten_dropout_prob, - seeds) - else: - return tf.nn.dropout( - x, rate=params.atten_dropout_prob, seed=params.random_seed) - - -def SafeCumprod(x, *args, **kwargs): - """Computes cumprod of x in logspace using cumsum to avoid underflow. - - The cumprod function and its gradient can result in numerical instabilities - when its argument has very small and/or zero values. As long as the argument - is all positive, we can instead compute the cumulative product as - exp(cumsum(log(x))). This function can be called identically to - tf.math.cumprod. - - Args: - x: Tensor to take the cumulative product of. - *args: Passed on to cumsum; these are identical to those in cumprod. - **kwargs: Passed on to cumsum; these are identical to those in cumprod. - - Returns: - Cumulative product of x. - """ - with tf.name_scope(None, 'SafeCumprod', [x]): - x = tf.convert_to_tensor(x, name='x') - tiny = np.finfo(x.dtype.as_numpy_dtype).tiny - return tf.exp( - py_utils.CumSum( - tf.math.log(tf.clip_by_value(x, tiny, 1)), *args, **kwargs)) - - -# pyformat: disable -def MonotonicAttentionProb(p_choose_i, previous_attention, mode): - """Compute monotonic attention distribution from choosing probabilities. - - Monotonic attention implies that the input sequence is processed in an - explicitly left-to-right manner when generating the output sequence. In - addition, once an input sequence element is attended to at a given output - timestep, elements occurring before it cannot be attended to at subsequent - output timesteps. This function generates attention distributions according - to these assumptions. For more information, see `Online and Linear-Time - Attention by Enforcing Monotonic Alignments`. - - Args: - p_choose_i: Probability of choosing input sequence/memory element i. Should - be of shape (batch_size, input_sequence_length), and should all be in the - range [0, 1]. - previous_attention: The attention distribution from the previous output - timestep. Should be of shape (batch_size, input_sequence_length). For - the first output timestep, preevious_attention[n] should be [1, 0, 0, ..., - 0] for all n in [0, ... batch_size - 1]. - mode: How to compute the attention distribution. Must be one of `recursive`, - `parallel`, or `hard`. - - * recursive: uses tf.scan to recursively compute the distribution. This is - slowest but is exact, general, and does not suffer from numerical - instabilities. - * parallel: uses parallelized cumulative-sum and cumulative-product - operations to compute a closed-form solution to the recurrence relation - defining the attention distribution. This makes it more efficient than - 'recursive', but it requires numerical checks which make the - distribution non-exact. This can be a problem in particular when - input_sequence_length is long and/or p_choose_i has entries very close - to 0 or 1. - * hard: requires that the probabilities in p_choose_i are all either 0 or - 1, and subsequently uses a more efficient and exact solution. - - Returns: - A tensor of shape (batch_size, input_sequence_length) representing the - attention distributions for each sequence in the batch. - - Raises: - ValueError: mode is not one of 'recursive', 'parallel', 'hard'. - """ - # pyformat: enable - # Force things to be tensors - p_choose_i = tf.convert_to_tensor(p_choose_i, name='p_choose_i') - previous_attention = tf.convert_to_tensor( - previous_attention, name='previous_attention') - if mode == 'recursive': - batch_size = py_utils.GetShape(p_choose_i)[0] - tf.logging.info(batch_size) - # Compute [1, 1 - p_choose_i[0], 1 - p_choose_i[1], ..., 1 - p_choose_i[-2]] - shifted_1mp_choose_i = tf.concat( - [tf.ones((batch_size, 1)), 1 - p_choose_i[:, :-1]], 1) - # Compute attention distribution recursively as - # q[i] = (1 - p_choose_i[i - 1])*q[i - 1] + previous_attention[i] - # attention[i] = p_choose_i[i]*q[i] - attention = p_choose_i * tf.transpose( - tf.scan( - # Need to use reshape to remind TF of the shape between loop - # iterations. - lambda x, yz: tf.reshape(yz[0] * x + yz[1], (batch_size,)), - # Loop variables yz[0] and yz[1] - [ - tf.transpose(shifted_1mp_choose_i), - tf.transpose(previous_attention) - ], - # Initial value of x is just zeros - tf.zeros((batch_size,)))) - elif mode == 'parallel': - # SafeCumprod computes cumprod in logspace with numeric checks - cumprod_1mp_choose_i = SafeCumprod(1 - p_choose_i, axis=1, exclusive=True) - # Compute recurrence relation solution - attention = p_choose_i * cumprod_1mp_choose_i * py_utils.CumSum( - previous_attention / - # Clip cumprod_1mp to avoid divide-by-zero - tf.clip_by_value(cumprod_1mp_choose_i, 1e-10, 1.), - axis=1) - elif mode == 'hard': - # Remove any probabilities before the index chosen last time step - p_choose_i *= tf.cumsum(previous_attention, axis=1) - # Now, use exclusive cumprod to remove probabilities after the first - # chosen index, like so: - # p_choose_i = [0, 0, 0, 1, 1, 0, 1, 1] - # cumprod(1 - p_choose_i, exclusive=True) = [1, 1, 1, 1, 0, 0, 0, 0] - # Product of above: [0, 0, 0, 1, 0, 0, 0, 0] - attention = p_choose_i * tf.math.cumprod( - 1 - p_choose_i, axis=1, exclusive=True) - else: - raise ValueError("mode must be 'recursive', 'parallel', or 'hard'.") - return attention - - -class BaseAttentionLayer(quant_utils.QuantizableLayer): - """A base class for all attention layers.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('atten_dropout_prob', 0.0, - 'Probability at which we apply dropout to the attention weights.') - p.Define( - 'atten_dropout_deterministic', False, - 'Whether to dropout in a fully deterministic way, which is more ' - 'suitable for TPU.') - p.Define('packed_input', False, - 'If True, each training example may pack multiple sequences.') - - p.qdomain.Define('softmax', None, 'QDomain for the internal softmax.') - p.qdomain.Define( - 'fullyconnected', None, 'Fully connected layers are fed ' - 'into activation functions which have known input ranges') - - return p - - def __init__(self, params): - """Constructs a BaseAttentionLayer object.""" - if not params.name: - raise ValueError('params.name is not set.') - super().__init__(params) - - self._source_init_done = False - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - self.TrackQTensor('logits', domain='fullyconnected') - - def InitForSourcePacked(self, - theta, - source_vecs, - source_contexts, - source_padding, - source_segment_id=None): - """Initialize attention for the given source vectors. - - Must set `_source_init_done` to True in the function. - - Note: `source_segment_id`, if present, should always have the same shape as - `source_padding`. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - source_vecs: A single tensor of shape [time, batch_size, source_dim]. - source_contexts: A single tensor of shape [time, batch_size, some_dim]. - source_padding: A tensor of shape [time, batch_size]. - source_segment_id: A tensor of shape [time, batch_size]. source_segment_id - is not None for packed inputs where one training example may pack - multiple sequences. - - Returns: - A `.NestedMap` object to be passed to ComputeContextVectorWithSource. - The internal structure of the return value should be considered an - implementation detail of the attention mechanism and should not be - inspected or modified by its callers. - """ - self._source_init_done = True - self._packed_src = self.PackSource(theta, source_vecs, source_contexts, - source_padding, source_segment_id) - return self._packed_src - - def PackSource(self, - theta, - source_vecs, - source_contexts, - source_padding, - source_segment_id=None): - """Packs source vectors. - - Does not change attention state. - - Note: `source_segment_id`, if present, should always have the same shape as - `source_padding`. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - source_vecs: A single tensor of shape [time, batch_size, source_dim]. - source_contexts: A single tensor of shape [time, batch_size, some_dim]. - source_padding: A tensor of shape [time, batch_size]. - source_segment_id: A tensor of shape [time, batch_size]. source_segment_id - is not None for packed inputs where one training example may pack - multiple sequences. - - Returns: - A `.NestedMap` object to be passed to ComputeContextVectorWithSource. - The internal structure of the return value should be considered an - implementation detail of the attention mechanism and should not be - inspected or modified by its callers. - """ - raise NotImplementedError('Abstract method.') - - def ComputeContextVectorWithSource(self, - theta, - packed_src, - query_vec, - attention_state=None, - per_step_source_padding=None, - query_segment_id=None): - """Computes the context vector given the current query output. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - packed_src: A `.NestedMap` object returned by PackSource or - InitForSourcePacked. - query_vec: a tensor of shape [batch_size, query_dim]. - attention_state: previous attention state. - per_step_source_padding: Source sequence padding to apply at this step. If - not None, it should have shape [target_batch_size, source_length]. - query_segment_id: a tensor of shape [batch_size]. - - Returns: - A tuple of 3 elements. - - - The attention context vector: [batch_size, context_dim] - - The attention probability vector: [batch_size, time] - - The new attention mechanism state: possibly nested tuple of tensors - with dimensions [target_batch, ...] - """ - raise NotImplementedError('Abstract method.') - - def ComputeContextVector(self, - theta, - query_vec, - attention_state=None, - per_step_source_padding=None, - query_segment_id=None): - """Computes the context vector given the current query output. - - Unlike `ComputeContextVectorWithSource` which explicitly asks for the packed - source tensors, `ComputeContextVector` uses the class' internal variables. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: a tensor of shape [batch_size, query_dim]. - attention_state: previous attention state. - per_step_source_padding: Source sequence padding to apply at this step. If - not None, it should be of shape [target_batch_size, source_length]. - query_segment_id: a tensor of shape [batch_size]. - - Returns: - A tuple of 3 elements. - - - The attention context vector. - - The attention probability vector. - - The new attention mechanism state: possibly nested tuple of tensors with - dimensions [target_batch, ...] - """ - assert self._source_init_done - return self.ComputeContextVectorWithSource(theta, self._packed_src, - query_vec, attention_state, - per_step_source_padding, - query_segment_id) - - def GetInitializationSourceState(self): - """Gets the attention initialization state. - - The base class only preserves the `concated_source_vecs`, - `concated_source_contexts` and `source_padding`. If subclasses use more - state than this and need to interact with inference code that must - fetch and reload state, this and `SetInitializationSourceState` must - be overridden. - - Returns: - A `.NestedMap` of Tensors that can be preserved and reset via - `SetInitializationSourceState()` at a later point. This allows, for - example, for attention computations to span session runs. - """ - assert self._source_init_done - return self._packed_src - - def SetInitializationSourceState(self, new_init_state): - """Sets the attention initialization state. - - Args: - new_init_state: A `.NestedMap` matching what was returned from - `GetInitializationSourceState`, which will return this layer to that - initialization state. - """ - self._source_init_done = True - self._packed_src = new_init_state.DeepCopy() - - def _PaddedSoftmax(self, logits, padding, narrow_to_asym_bit_depth=False): - """Performs a softmax as if padding were applied after exponentiation. - - The default implementation uses numerical techniques to approximate this - with a standard `tf.nn.softmax` (using large negative logits for padded - values). It defers to a `Defun` that may be replaced on low-range - implementations with a version that is numerically correct. - - Args: - logits: Logits. - padding: Padding (must be the same shape as logits). - narrow_to_asym_bit_depth: Narrows the bit depth, removing the upper limit - value. This is to accommodate certain interpreters that would cover a 0 - .... 2**bits - 1 range for quantization. - - Returns: - Result of the softmax. - """ - fns = self.fns - - if logits.dtype.is_complex: - logits = tf.abs(logits) - assert logits.dtype.is_floating - assert hasattr(logits.dtype, 'max') - very_negative_logits = ( - tf.ones_like(logits) * logits.dtype.max * - tf.constant(-0.7, dtype=logits.dtype)) - if self.do_eval: - very_negative_logits = self.QTensor('logits', very_negative_logits) - padded_logits = tf.where(padding > 0.0, very_negative_logits, logits) - # TFLite hardcodes the range of qsoftmax, setting explicitly to avoid - # incompatible concats. - return fns.qsoftmax( - padded_logits, - qdomain='softmax', - narrow_to_asym_bit_depth=narrow_to_asym_bit_depth) - - def _UpdatePaddingWithPackedInputMask(self, padding, source_segment_ids, - query_segment_ids): - """Creates an attention mask based on source and query segment ids. - - This creates a mask that removes invalid attention, where the query vector - might assign some weight to neighboring sequences in a packed input example. - Assumes `n = target_batch // source_batch`. - - Args: - padding: Padding for logits, a tensor of shape [time, n, source_batch]. - source_segment_ids: a tensor of shape [time, source_batch]. - query_segment_ids: a tensor of shape [target_batch]. - - Returns: - Logits with mask applied. - """ - # Generating packed input mask for attention padding. - source_segment_ids = tf.expand_dims(source_segment_ids, 1) - query_segment_ids = tf.reshape( - query_segment_ids, - [1, -1, py_utils.GetShape(source_segment_ids)[2]]) - padding = tf.where( - tf.equal(source_segment_ids, query_segment_ids), padding, - tf.ones_like(padding)) - return padding - - -class AdditiveAttention(BaseAttentionLayer): - """Implements additive attention (also known as "Bahdanau Attention"). - - Described in: - - Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio. - "Neural Machine Translation by Jointly Learning to Align and Translate." - ICLR 2015. - https://arxiv.org/abs/1409.0473 - """ - - @classmethod - def Params(cls): - """Params for this `AdditiveAttention` class.""" - p = super().Params() - p.Define('source_dim', 0, 'Number of source nodes.') - p.Define('query_dim', 0, 'Number of query nodes.') - p.Define('hidden_dim', 0, 'Number of hidden nodes.') - # Fill in reasonable default for params init - p.params_init = py_utils.WeightInit.GaussianSqrtDim() - p.Define( - 'same_batch_size', False, - 'True iff the source and target sequence has the same batch size.') - return p - - def __init__(self, params): - """Constructs an `AdditiveAttention` object.""" - super().__init__(params) - p = self.params - - def AttenProbs(inputs): - """Generates probs.""" - source_batch = py_utils.GetShape(inputs.source_padding)[1] - target_batch = py_utils.GetShape(inputs.per_step_source_padding)[0] - multiplier = target_batch // source_batch - - # Shape of summed is [sl, tb/sb, sb, hidden_dim]. - summed = tf.tanh(inputs.concated_source_vecs + inputs.query_vec_reshaped) - # logits is of shape [sl * tb/sb * sb, 1]. Computes dot product - # between v with every rows in 'summed'. Then we reshape the - # result to be of shape [sl, tb/sb, sb]. - # - # Another equivalent way is to do: - # logits = tf.reduce_sum(summed * - # tf.reshape(v, [1, 1, 1, hidden_dim]), 3) - logits = py_utils.Matmul( - tf.reshape(summed, [-1, p.hidden_dim]), - tf.reshape(inputs.v, [p.hidden_dim, 1])) - logits = tf.reshape(logits, tf.shape(summed)[:3]) - # Take out the padding states. - # _source_padding is of shape [source_length, source_batch]. - # reshaped to [source_length, 1, source_batch]. - # per_step_source_padding is reshaped to the same but with 'multiplier' - # for the second dim. - source_padding = tf.expand_dims(inputs.source_padding, 1) - per_step_source_padding = tf.reshape( - tf.transpose(inputs.per_step_source_padding), - [-1, multiplier, source_batch]) - source_padding += per_step_source_padding - - if p.packed_input: - source_padding = self._UpdatePaddingWithPackedInputMask( - source_padding, inputs.source_segment_id, inputs.query_segment_id) - # Reshape logits to a matrix of shape [target_batch, source_length] and - # takes the softmax to compute the probabilities. - logits = tf.transpose(tf.reshape(logits, [-1, target_batch])) - source_padding = tf.transpose( - tf.reshape(source_padding, [-1, target_batch])) - probs = self._PaddedSoftmax(logits, source_padding) - return probs - - # Adds the atten function into the graph's library. - def Atten(v, w, source_padding, source_segment_id, concated_source_vecs, - concated_source_contexts, query_vec, query_segment_id, - per_step_source_padding, global_step): - """Computes the attention context vector. - - Args: - v: hidden weight. [hidden_dim, 1]. - w: query weight. [query_dim, hidden_dim]. - source_padding: [source_length, source_batch]. - source_segment_id: [source_lentgh, source_batch] - concated_source_vecs: [source_length, source_batch, hidden_dim]. - concated_source_contexts: [source_batch, source_length, context_dim] - query_vec: [target_batch, query_dim] - query_segment_id: [target_batch] - per_step_source_padding: [target_batch, source_length] - global_step: Required for deterministic dropout. - Note: concated_source_vecs are the vectors that are used to compute the - attention score between the query_vec and each concated_source_vec. The - concated_source_contexts are the vectors that compose the result. The - attention context vector is computed as a weighted average of the - concated_source_contexts, using the scores that were computed using - concated_source_vecs. - - Returns: - attention context vectors and probabilities. - """ - source_batch = py_utils.GetShape(concated_source_vecs)[1] - target_batch = py_utils.GetShape(query_vec)[0] - multiplier = target_batch // source_batch - # concated_source_vecs is reshaped to - # [source_length, 1, source_batch, hidden_dims] - concated_source_vecs = tf.expand_dims(concated_source_vecs, 1) - query_vec_transformed = py_utils.Matmul(query_vec, w) - - # query_vec is reshaped to - # [1, target_batch/source_batch, source_batch, hidden_dims]. - query_vec_reshaped = tf.reshape( - query_vec_transformed, [1, multiplier, source_batch, p.hidden_dim]) - # probs is of shape [target_batch, source_length] - probs = py_utils.CallDefun( - AttenProbs, - py_utils.NestedMap( - concated_source_vecs=concated_source_vecs, - source_padding=source_padding, - query_vec_reshaped=query_vec_reshaped, - v=v, - per_step_source_padding=per_step_source_padding, - source_segment_id=source_segment_id, - query_segment_id=query_segment_id)) - probs.set_shape(per_step_source_padding.shape) - - # Apply dropout to weights if applicable. - if not self.do_eval: - probs = _ApplyAttentionDropout(p, probs, global_step) - - # Reshape probs to be of shape - # [target_batch/source_batch, source_batch, source_length] - probs_reshaped = tf.reshape(probs, [multiplier, source_batch, -1]) - # Transpose probs to be of shape - # [source_batch, target_batch/source_batch, source_length] - probs_reshaped = tf.transpose(probs_reshaped, [1, 0, 2]) - # Batched matmul - # [source_batch, target_batch/source_batch, source_length] * - # [source_batch, source_length, context_dim] = - # [source_batch, target_batch/source_batch, context_dim] - summed = tf.matmul(probs_reshaped, concated_source_contexts) - - # summed is of shape - # [target_batch/source_batch, source_batch, context_dim] - summed = tf.transpose(summed, [1, 0, 2]) - - return tf.reshape(summed, [target_batch, -1]), probs - - # The source batch size equals to the target batch size. - def AttenSameBatchSize(v, w, source_padding, source_segment_id, - concated_source_vecs, concated_source_contexts, - query_vec, query_segment_id, per_step_source_padding, - global_step): - """Computes the attention context vector. - - Args: - v: hidden weight. [hidden_dim]. - w: query weight. [query_dim, hidden_dim]. - source_padding: [sl, b] - source_segment_id: [sl, b] - concated_source_vecs: [sl, b, hidden_dim]. - concated_source_contexts: [b, sl, context_dim] - query_vec: [b, query_dim] - query_segment_id: [b] - per_step_source_padding: [b, sl] - global_step: Required for deterministic dropout. - - Returns: - attention context vectors and probabilities. - """ - # TODO(jiaye): support dropout - if p.atten_dropout_prob != 0: - raise NotImplementedError('dropout is not supported') - del global_step - - # [b, hidden_dim] - query_vec = py_utils.Matmul(query_vec, w) - # [sl, b] - def AttenProbs(inputs): - """Calculates atten probs with padding.""" - # tf.tanh(x+y) shape [sl, b, hidden_dim] - summed = tf.tanh(inputs.x + inputs.y) - # [-1, hidden_dim] * [hidden_dim, 1] = [-1, 1] - res = py_utils.Matmul( - tf.reshape(summed, [-1, p.hidden_dim]), tf.expand_dims(inputs.v, 1)) - # Reshape res to [sl, b] - logits = tf.reshape(res, tf.shape(summed)[:2]) - # Take out the padding states. _source_padding is of shape [sl, b]. - source_padding = inputs.source_padding + tf.transpose( - inputs.per_step_source_padding) - - if p.packed_input: - source_padding = self._UpdatePaddingWithPackedInputMask( - tf.expand_dims(source_padding, 1), inputs.source_segment_id, - inputs.query_segment_id) - source_padding = tf.squeeze(source_padding, 1) - # [b, sl] - source_padding = tf.transpose(source_padding) - logits = tf.transpose(logits) - # softmax to compute the probabilities. [b, sl] - probs = self._PaddedSoftmax(logits, source_padding) - return probs - - probs = py_utils.CallDefun( - AttenProbs, - py_utils.NestedMap( - x=concated_source_vecs, - source_padding=source_padding, - y=query_vec, - v=v, - per_step_source_padding=per_step_source_padding, - source_segment_id=source_segment_id, - query_segment_id=query_segment_id)) - probs.set_shape(per_step_source_padding.shape) - - # contexts[i, :] is a weighted (probs[i, :]) average of - # concated_source_vecs[i, :, :]. - # Reshaped probs is of shape [b, 1, sl] - reshaped_probs = tf.expand_dims(probs, 1) - # [b, 1, sl] * [b, sl, context_dim] = [b, 1, context_dim] - contexts = tf.matmul(reshaped_probs, concated_source_contexts) - # Reshaped context is of shape [b, context_dim] - contexts = tf.squeeze(contexts, axis=1) - return contexts, probs - - if p.same_batch_size: - self._ctx_vec = AttenSameBatchSize - else: - self._ctx_vec = Atten - - def EncodeSource(src_w, vecs, ctxs): - """Prepares source vec and ctx.""" - time, batch = py_utils.GetShape(vecs, 2) - ctxs = py_utils.HasShape(ctxs, [time, batch, -1]) - # source_dim can be a symbolic expression. - transformed_vecs = tf.reshape( - py_utils.Matmul( - tf.reshape(vecs, [-1, symbolic.ToStatic(p.source_dim)]), src_w), - [time, batch, -1]) - transformed_vecs = tf.identity( - transformed_vecs, name='source_vecs_projected') - transposed_ctxs = tf.transpose(ctxs, [1, 0, 2]) - transposed_ctxs = tf.identity(transposed_ctxs, name='source_ctx') - return transformed_vecs, transposed_ctxs - - self._encode_source = EncodeSource - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - pc = py_utils.WeightParams( - shape=[p.source_dim, p.hidden_dim], - init=p.params_init, - dtype=p.dtype, - collections=['AdditiveAttention_vars']) - self.CreateVariable('source_var', pc, self.AddGlobalVN) - - pc = py_utils.WeightParams( - shape=[p.query_dim, p.hidden_dim], - init=p.params_init, - dtype=p.dtype, - collections=['AdditiveAttention_vars']) - self.CreateVariable('query_var', pc, self.AddGlobalVN) - - pc = py_utils.WeightParams( - shape=[p.hidden_dim], - init=p.params_init, - dtype=p.dtype, - collections=['AdditiveAttention_vars']) - self.CreateVariable('hidden_var', pc, self.AddGlobalVN) - - def PackSource(self, - theta, - source_vecs, - source_contexts, - source_padding, - source_segment_id=None): - """Packs source vectors. - - Does not change attention state. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - source_vecs: A single tensor of shape [time, batch_size, source_dim]. - source_contexts: A single tensor of shape [time, batch_size, some_dim]. - source_padding: A tensor of shape [time, batch_size]. - source_segment_id: A tensor of shape [time, batch_size]. - - Returns: - A NestedMap containing the packed source. - """ - with tf.name_scope(self.params.name): - if source_segment_id is None: - source_segment_id = tf.zeros_like(source_padding) - - (concated_source_vecs, concated_source_contexts) = ( - self._encode_source(theta.source_var, source_vecs, source_contexts)) - return py_utils.NestedMap( - # [time, batch_size, hidden_dim]. - source_vecs=concated_source_vecs, - # [batch_size, time, context_dim]. - # Note the mismatch between `source_vecs` and `source_contexts`. In - # `source_vecs`, time is the first dim, while it is the second dim in - # `source_contexts`. - source_contexts=concated_source_contexts, - # [time, batch_size]. - source_padding=source_padding, - # [time, batch_size]. - source_segment_id=source_segment_id) - - def ZeroAttentionState(self, source_length, decoder_batch_size): - p = self.params - # This is just a dummy state. The first dimension of the state has to match - # decoder_batch_size. - zs = tf.zeros([decoder_batch_size, 1], dtype=py_utils.FPropDtype(p)) - return zs - - def ComputeContextVectorWithSource(self, - theta, - packed_src, - query_vec, - attention_state=None, - per_step_source_padding=None, - query_segment_id=None): - """Computes the context vector given the current query output. - - Note: `packed_src.source_vecs` are the vectors that are used to compute the - attention score between the `query_vec` and each `packed_src.source_vecs`. - The `packed_src.source_contexts` are the vectors that compose the result. - The attention context vector is computed as a weighted average of the - `packed_src.source_contexts`, using the scores that were computed using - `packed_src.source_vecs`. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - packed_src: A `.NestedMap` object returned by PackSource or - InitForSourcePacked. - query_vec: a tensor of shape [batch_size, query_dim]. - attention_state: previous attention state. It is not used in - `AdditiveAttention`, and is simply passed through. - per_step_source_padding: Source sequence padding to apply at this step. If - not None, it should be of shape [target_batch_size, source_length]. - query_segment_id: a tensor of shape [batch_size] - - Returns: - A tuple of 3 elements. - - - The attention context vector: [batch_size, context_dim] - - The attention probability vector: [batch_size, time] - - The new attention mechanism state: possibly nested tuple of tensors with - dimensions [target_batch, ...] - """ - p = self.params - concated_source_vecs = packed_src.source_vecs - concated_source_contexts = packed_src.source_contexts - source_padding = packed_src.source_padding - source_segment_id = packed_src.source_segment_id - query_batch_size = py_utils.GetShape(query_vec)[0] - source_length = py_utils.GetShape(source_padding)[0] - if per_step_source_padding is None: - zero = tf.constant(0.0, dtype=query_vec.dtype) - per_step_source_padding = tf.fill([query_batch_size, source_length], zero) - per_step_source_padding = py_utils.HasShape( - per_step_source_padding, [query_batch_size, source_length]) - hidden = py_utils.AddPerStepVN(p, theta.hidden_var) - query = py_utils.AddPerStepVN(p, theta.query_var) - - if source_segment_id is None: - source_segment_id = tf.zeros_like(source_padding) - if query_segment_id is None: - query_segment_id = tf.zeros( - tf.shape(query_vec)[0], dtype=source_padding.dtype) - - ctx_vec, prob = self._ctx_vec(hidden, query, source_padding, - source_segment_id, concated_source_vecs, - concated_source_contexts, query_vec, - query_segment_id, per_step_source_padding, - theta.global_step) - - return ctx_vec, prob, attention_state - - -class DotProductAttention(BaseAttentionLayer): - """Implements dot-product attention (also known as "Luong Attention"). - - Described in: - - Minh-Thang Luong, Hieu Pham, Christopher D. Manning. - "Effective Approaches to Attention-based Neural Machine Translation." - EMNLP 2015. - https://arxiv.org/abs/1508.04025 - """ - - @classmethod - def Params(cls): - """Params for `DotProductAttention`.""" - p = super().Params() - p.Define('source_dim', 0, 'Number of source nodes.') - p.Define('query_dim', 0, 'Number of query nodes.') - p.Define('hidden_dim', 0, 'Number of hidden nodes.') - p.Define( - 'use_dim_scale', True, 'Whether or not to use per_dim_scale to scale ' - 'the individual dims when calculating attention probabilities. It can ' - 'increase training stability when set to False.') - return p - - def __init__(self, params): - """Constructs a DotProductAttention object.""" - super().__init__(params) - p = self.params - # TODO(yonghui): relax these constraints. - assert p.source_dim == p.query_dim - assert p.source_dim == p.hidden_dim - - def AttenProbs(inputs): - """Main attention function. - - target_batch = source_batch * n where n is an integer >= 1. - In this case inputs.query_vec contains: - ------------------------- - | instance 1 | - | instance 2 | - 0 | ... | - | instance source_batch | - ------------------------- - | instance 1 | - | instance 2 | - 1 | ... | - | instance source_batch | - ------------------------- - ... - ------------------------- - | instance 1 | - | instance 2 | - n-1 | ... | - | instance source_batch | - ------------------------- - One use case is beam search where n = beam size. - - Args: - inputs: a NestedMap containing: - - per_dim_scale: [source_dim], a vec to scale individual dims. - - source_padding: [time, source_batch]. - - concated_source_vecs: [source_batch, time, source_dim]. - - query_vec: [target_batch, source_dim]. - - per_step_source_padding: [target_batch, source_length] - - source_segment_id: [time, source_batch]. - - query_segment_id: [target_batch]. - - Returns: - logits [target_batch, source_time]. - """ - source_padding = tf.transpose(inputs.source_padding) - concated_source_vecs = inputs.concated_source_vecs - - logit_scale = tf.stop_gradient( - tf.math.rsqrt( - tf.cast( - py_utils.GetShape(inputs.query_vec)[1], - dtype=py_utils.FPropDtype(p)))) - source_batch = py_utils.GetShape(concated_source_vecs)[0] - target_batch = py_utils.GetShape(inputs.query_vec)[0] - query_vec = inputs.query_vec * inputs.per_dim_scale - # The n here refers to the "n" described in the comment above. - n = target_batch // source_batch - query_vec = tf.reshape(query_vec, [n, source_batch, -1]) - # => [source_batch, source_dim, n] - query_vec = tf.transpose(query_vec, [1, 2, 0]) - # => [n, source_batch, source_sequence_len] - per_step_source_padding = tf.reshape(inputs.per_step_source_padding, - [n, source_batch, -1]) - # => [source_batch, source_sequence_len, n] - per_step_source_padding = tf.transpose(per_step_source_padding, [1, 2, 0]) - # Dot-product part. - # Calls batch_mat_mul since dim > 2 for per-instance matmul. - # [source_batch, time, source_dim] * [source_batch, source_dim, n] - # => [source_batch, time, n] - logits = tf.matmul(concated_source_vecs, query_vec) - logits *= logit_scale - # Exclude padding frames. - # [source_batch, time] => [source_batch, time, 1] - source_padding = tf.expand_dims(source_padding, 2) - source_padding += per_step_source_padding - if p.packed_input: - source_padding = tf.transpose(source_padding, [1, 2, 0]) - source_padding = self._UpdatePaddingWithPackedInputMask( - source_padding, inputs.source_segment_id, inputs.query_segment_id) - source_padding = tf.transpose(source_padding, [1, 2, 0]) - else: - source_padding = tf.transpose(source_padding, [2, 0, 1]) - - # => [n, source_batch, time] - logits = tf.transpose(logits, [2, 0, 1]) - - # => [n * source_batch, time]. - # This makes logits store content in the same order as query_vec. - logits = tf.reshape(logits, [target_batch, -1]) - source_padding = tf.reshape(source_padding, [target_batch, -1]) - probs = self._PaddedSoftmax(logits, source_padding) - return probs - - def Atten(per_dim_scale, source_padding, source_segment_id, - concated_source_vecs, concated_source_contexts, query_vec, - query_segment_id, per_step_source_padding, global_step): - """Main attention function. - - Args: - per_dim_scale: [source_dim], a vec to scale individual dims. - source_padding: [time, source_batch]. - source_segment_id: [time, source_batch]. - concated_source_vecs: [time, source_batch, source_dim]. - concated_source_contexts: [source_batch, time, context_dim]. - query_vec: [target_batch, source_dim]. - query_segment_id: [target_batch]. - per_step_source_padding: [target_batch, source_length] - global_step: Required for deterministic dropout. - Note: concated_source_vecs are the vectors that are used to compute the - attention score between the query_vec and each concated_source_vec. The - concated_source_contexts are the vectors that compose the result. The - attention context vector is computed as a weighted average of the - concated_source_contexts, using the scores that were computed using - concated_source_vecs. - - Returns: - Two tensors: - - - context_vector: [target_batch, context_dim]. - - probs: [target_batch, time]. - """ - py_utils.assert_shape_match([py_utils.GetShape(concated_source_vecs)[2]], - [py_utils.GetShape(query_vec)[1]]) - py_utils.assert_shape_match([py_utils.GetShape(concated_source_vecs)[2]], - [symbolic.ToStatic(p.source_dim)]) - source_batch = py_utils.GetShape(concated_source_vecs)[1] - target_batch = py_utils.GetShape(query_vec)[0] - n = target_batch // source_batch - concated_source_vecs = tf.transpose(concated_source_vecs, [1, 0, 2]) - concated_source_vecs = tf.identity( - concated_source_vecs, name='concated_source_vecs') - returned_probs = py_utils.CallDefun( - AttenProbs, - py_utils.NestedMap( - per_dim_scale=per_dim_scale, - source_padding=source_padding, - concated_source_vecs=concated_source_vecs, - query_vec=query_vec, - per_step_source_padding=per_step_source_padding, - source_segment_id=source_segment_id, - query_segment_id=query_segment_id)) - returned_probs.set_shape(per_step_source_padding.shape) - - # => [n, source_batch, time]. - probs = tf.reshape(returned_probs, [n, source_batch, -1]) - # => [source_batch, n, time]. - probs = tf.transpose(probs, [1, 0, 2]) - - # Apply dropout to weights if applicable. - if not self.do_eval: - probs = _ApplyAttentionDropout(p, probs, global_step) - - # Weight each frame with the probability and sum them. - # [source_batch, n, time] * [source_batch, time, context_dim] - # => [source_batch, n, context_dim]. - concated_source_contexts = tf.identity( - concated_source_contexts, name='concated_source_contexts') - context_vector = tf.matmul(probs, concated_source_contexts) - # => [n, source_batch, context_dim]. - context_vector = tf.transpose(context_vector, [1, 0, 2]) - # => [n * source_batch, context_dim]. - context_vector = tf.reshape(context_vector, [target_batch, -1]) - - return context_vector, returned_probs - - self._ctx_vec = Atten - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - - if p.use_dim_scale: - pc = py_utils.WeightParams( - shape=[p.hidden_dim], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=['DotProductAttention_vars']) - - self.CreateVariable('per_dim_scale', pc) - - def PackSource(self, - theta, - source_vecs, - source_contexts, - source_padding, - source_segment_id=None): - """Packs source vectors. - - Does not change attention state. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - source_vecs: A tensor of shape [time, source_batch, source_dim]. - source_contexts: A tensor of shape [time, source_batch, context_dim]. - source_padding: A tensor of shape [time, source_batch]. - source_segment_id: A tensor of shape [time, source_batch]. - - Returns: - A tuple (concated_source_vecs, concated_source_contexts, source_padding) - where `concated_source_vecs` is a tensor of shape [time, batch_size, - hidden_dim], `concated_source_contexts` is a tensor of shape - [batch_size, time, some_dim] and `source_padding` is a tensor of shape - [time, batch_size]. - """ - concated_source_vecs = tf.identity(source_vecs) - concated_source_contexts = tf.transpose(source_contexts, [1, 0, 2]) - if source_segment_id is None: - source_segment_id = tf.zeros_like(source_padding) - return py_utils.NestedMap( - # [time, batch_size, hidden_dim]. - source_vecs=concated_source_vecs, - # [batch_size, time, context_dim]. - # Note the mismatch between `source_vecs` and `source_contexts`. In - # `source_vecs`, time is the first dim, while it is the second dim in - # `source_contexts`. - source_contexts=concated_source_contexts, - # [time, batch_size]. - source_padding=source_padding, - # [time, batch_size]. - source_segment_id=source_segment_id) - - def ZeroAttentionState(self, source_length, decoder_batch_size): - p = self.params - # No states to keep track of currently. - return tf.zeros([decoder_batch_size, 1], dtype=p.dtype) - - def ComputeContextVectorWithSource(self, - theta, - packed_src, - query_vec, - attention_state=None, - per_step_source_padding=None, - query_segment_id=None): - """Computes the context vector given the current query output. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - packed_src: A `.NestedMap` object returned by PackSource or - InitForSourcePacked. - query_vec: a tensor of shape [target_batch, query_dim], where target_batch - = n * source_batch (e.g., n = num_hyps_per_beam in beamsearch). Along - the target_batch dimension, there are n groups of consecutive rows, each - group containing source_batch rows. - attention_state: previous attention state. It is not used in - AdditiveAttention, and is simply passed through. - per_step_source_padding: Source sequence padding to apply at this step. If - not None, it should be of shape [target_batch, source_length]. - query_segment_id: Query segment id with shape [target_batch]. - - Returns: - A tuple of 3 elements. - - - The attention context vector: [batch_size, context_dim] - - The attention probability vector: [batch_size, time] - - The new attention mechanism state: possibly nested tuple of tensors - with dimensions [target_batch, ...] - """ - concated_source_vecs = packed_src.source_vecs - concated_source_contexts = packed_src.source_contexts - - source_padding = packed_src.source_padding - source_segment_id = packed_src.source_segment_id - query_batch_size = py_utils.GetShape(query_vec)[0] - source_sequence_length = py_utils.GetShape(source_padding)[0] - if per_step_source_padding is None: - zero = tf.constant(0.0, dtype=query_vec.dtype) - per_step_source_padding = tf.fill( - [query_batch_size, source_sequence_length], zero) - per_step_source_padding = py_utils.HasShape( - per_step_source_padding, [query_batch_size, source_sequence_length]) - if source_segment_id is None: - source_segment_id = tf.zeros_like(source_padding) - if query_segment_id is None: - query_segment_id = tf.zeros( - py_utils.GetShape(query_vec)[0], dtype=source_padding.dtype) - - def ScaleFn(x): - return tf.nn.softplus(x) / tf.nn.softplus(tf.constant(0.0, dtype=x.dtype)) - - if self.params.use_dim_scale: - per_dim_scale_var = theta.per_dim_scale - else: - per_dim_scale_var = tf.constant(0.0, dtype=query_vec.dtype) - - ctx_vec, prob = self._ctx_vec( - ScaleFn(per_dim_scale_var), source_padding, source_segment_id, - concated_source_vecs, concated_source_contexts, query_vec, - query_segment_id, per_step_source_padding, theta.global_step) - return ctx_vec, prob, attention_state - - -def _RecursiveReshape(x, shape): - if x is None: - return None - elif isinstance(x, py_utils.NestedMap): - return x.Transform(lambda y: _RecursiveReshape(y, shape)) - else: - return tf.reshape(x, shape) if x.shape.ndims == 2 else x - - -class MultiHeadedAttention(BaseAttentionLayer, quant_utils.QuantizableLayer): - """Attention with multiple attention heads. - - Conceptually, the algorithm works as follows: - - 1. Source vectors (attention keys) are first projected to vectors of dim - p.hidden_dim. - 2. Query vectors are projected to vectors of dim p.hidden_dim as well. - 3. Context vectors (attention values) are not projected by default, unless - `enable_ctx_pre_proj` is True. - 4. Source vectors, query vectors and context vectors are all split into - p.num_attention_heads chunks. - 5. The inner atten mechanism is computed separately on each of the chunks. - 6. Attention contexts from each of the chunk are concatenated to form the - final context. - 7. Attention probs from each of the chunk are averaged to form the final - attention prob. - """ - - @classmethod - def Params(cls): - """Params for MultiHeadedAttention.""" - p = super().Params() - p.Define('source_dim', 0, 'Number of source nodes.') - p.Define('query_dim', 0, 'Number of query nodes.') - p.Define('context_dim', 0, 'Number of context nodes.') - p.Define('hidden_dim', 0, 'Number of hidden nodes.') - p.Define('num_attention_heads', 2, 'Num of attention heads.') - p.Define( - 'use_source_vec_as_attention_value', True, - 'Whether or not to use source_vec as the attention value as well.' - ' If True, we expect source_vec and source_contexts are the same.') - p.Define('enable_source_proj', True, - 'If False, source side linear projection is disabled.') - p.Define('enable_query_proj', True, - 'If False, query side linear projection is disabled.') - p.Define('inner_atten_params', DotProductAttention.Params(), - 'Params for underlying attention mechanism.') - p.Define( - 'enable_ctx_pre_proj', False, - 'If True, context is pre-projected before processing into' - ' hidden_dim.') - p.Define( - 'enable_ctx_post_proj', False, - 'If True, computed context is post projected into' - ' ctx_post_proj_dim.') - p.Define('ctx_post_proj_dim', 0, 'Number of post projection nodes.') - p.Define( - 'num_post_proj', 1, 'Number of post projections, usually the same as ' - 'number of tasks. Each task may choose to use one of the post ' - 'projection layers.') - p.Define( - 'proj_init', 'default', 'Initialization approach for projection ' - 'layers:' - 'uniform: Use uniform initialization. ' - 'default: Use the default Xavier initialization.') - p.Define( - 'attention_head_prob_index', -1, 'If > 0, instead of averaging ' - 'the probabilities of all attention heads when returning the ' - 'attention probability, instead return the selected index prob.') - - p.Define('use_bias', True, 'Whether to use bias for projection layer.') - p.Define('enable_per_dim_scale', True, - 'Whether to use per_dim_scale in inner_atten.') - - # Often the attention context output needs to be concated - # with tensors from another layer. This allows them to share - # quantization parameters. By convention, all attention layers - # need to include their context output vectors in this domain. - p.qdomain.Define('atten_context', None, - 'Quantization domain for attention context.') - - p.params_init = py_utils.WeightInit.Xavier(scale=1.0) - - return p - - def __init__(self, params): - """Constructs a MultiHeadedAttention object.""" - super().__init__(params) - p = self.params - assert symbolic.ToStatic(p.hidden_dim) % p.num_attention_heads == 0 - - if p.proj_init not in ('uniform', 'default'): - raise ValueError('Unknown proj_init: %s!' % p.proj_init) - - att_dim = p.hidden_dim // p.num_attention_heads - - att_p = p.inner_atten_params.Set( - source_dim=att_dim, - query_dim=att_dim, - hidden_dim=att_dim, - dtype=p.dtype, - atten_dropout_prob=p.atten_dropout_prob, - atten_dropout_deterministic=p.atten_dropout_deterministic, - packed_input=p.packed_input) - - if att_p.cls == DotProductAttention: - att_p.use_dim_scale = p.enable_per_dim_scale - - if not att_p.name: - att_p.name = 'inner_att' - self.CreateChild('atten', att_p) - if p.attention_head_prob_index >= 0: - assert p.attention_head_prob_index < p.num_attention_heads - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - - def InitProj(layer_dim, bias=False): - if p.proj_init == 'uniform': - # Note we also initialize bias with uniform distribution here, following - # the default Pytorch implementation: - # https://pytorch.org/docs/stable/nn.html#linear - proj_init = py_utils.WeightInit.Uniform(scale=np.sqrt(1.0 / layer_dim)) - elif p.proj_init == 'default': - proj_init = py_utils.WeightInit.Constant(0.0) if bias else p.params_init - return proj_init - - if p.use_bias: - pc_bias = py_utils.WeightParams( - shape=[p.hidden_dim], - init=InitProj(p.hidden_dim, bias=True), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - - if p.enable_source_proj: - pc = py_utils.WeightParams( - shape=[p.source_dim, p.hidden_dim], - init=InitProj(p.source_dim), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('source_proj', pc) - if p.use_bias: - self.CreateVariable('source_proj_b', pc_bias) - else: - assert p.source_dim == p.hidden_dim - - if p.enable_query_proj: - pc = py_utils.WeightParams( - shape=[p.query_dim, p.hidden_dim], - init=InitProj(p.query_dim), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('query_proj', pc) - if p.use_bias: - self.CreateVariable('query_proj_b', pc_bias) - else: - assert p.query_dim == p.hidden_dim - - if p.enable_ctx_pre_proj and not p.use_source_vec_as_attention_value: - assert p.context_dim - pc = py_utils.WeightParams( - shape=[p.context_dim, p.hidden_dim], - init=InitProj(p.context_dim), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('ctx_proj', pc) - if p.use_bias: - self.CreateVariable('ctx_proj_b', pc_bias) - - if p.enable_ctx_post_proj: - assert p.ctx_post_proj_dim - if p.num_post_proj == 1: - pc_shape = [p.hidden_dim, p.ctx_post_proj_dim] - pc_b_shape = [p.ctx_post_proj_dim] - elif p.num_post_proj > 1: - pc_shape = [p.hidden_dim, p.ctx_post_proj_dim, p.num_post_proj] - pc_b_shape = [p.ctx_post_proj_dim, p.num_post_proj] - else: - raise ValueError('num_post_proj must > 0!') - pc = py_utils.WeightParams( - shape=pc_shape, - init=InitProj(p.hidden_dim), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('ctx_post_proj', pc) - if p.use_bias: - pc_bias_post_proj = py_utils.WeightParams( - shape=pc_b_shape, - init=InitProj(p.ctx_post_proj_dim, bias=True), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('ctx_post_proj_b', pc_bias_post_proj) - - self.TrackQTensor('source_proj_matmul', 'source_proj_add', - 'query_proj_matmul', 'query_proj_add', - 'ctx_pre_proj_matmul', 'ctx_pre_proj_add') - # TODO(suderman): Remove the self.do_eval check below once brop quant within - # defun is fixed on the training side. This is less than ideal as-is because - # training will just trend to match downstream quant constraints vs force - # alignment. - self.TrackQTensor( - 'ctx_post_proj_matmul', 'ctx_post_proj_add', domain='atten_context') - - @classmethod - def SetOutputContextDim(cls, p, out_dim): - p.ctx_post_proj_dim = out_dim - - @py_utils.NameScopeDecorator('MultiHeadedAttention/PackSource') - def PackSource(self, - theta, - source_vecs, - source_contexts, - source_padding, - source_segment_id=None): - """Packs source vectors. - - Does not change attention state. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - source_vecs: A tensor of shape [time, source_batch, source_dim]. - source_contexts: A tensor of shape [time, source_batch, context_dim]. - source_padding: A tensor of shape [time, source_batch]. - source_segment_id: A tensor of shape [time, source_batch]. - - Returns: - A NestedMap representing packed src. It will have the same structure - as the one returned by the inner atten, except that source_batch will be - source_batch * num_heads. - """ - - p = self.params - fns = self.fns - if not p.enable_source_proj: - assert p.source_dim == p.hidden_dim - if not p.enable_query_proj: - assert p.query_dim == p.hidden_dim - with tf.name_scope('init__0'): - if p.use_source_vec_as_attention_value: - source_vecs = py_utils.HasShape(source_vecs, - py_utils.GetShape(source_contexts)) - time_steps, batch_size = py_utils.GetShape(source_padding, 2) - # source_projected shape [time * source_batch, hidden] - with tf.name_scope('init__0a'): - source_vec_depth = py_utils.GetShape(source_vecs)[2] - with tf.name_scope('init__0b'): - if p.enable_source_proj: - source_projected = ( - fns.qbatchmatmul( - tf.reshape(source_vecs, [-1, source_vec_depth]), - fns.qweight(theta.source_proj), - qt='source_proj_matmul')) - if p.use_bias: - source_projected = fns.qadd( - source_projected, - fns.qweight(theta.source_proj_b), - qt='source_proj_add') - else: - source_projected = tf.reshape(source_vecs, [-1, source_vec_depth]) - with tf.name_scope('init__1'): - num_heads = p.num_attention_heads - # => [time, source_batch * num_heads, hidden / num_heads] - source_projected = tf.reshape(source_projected, [ - time_steps, batch_size * num_heads, - symbolic.ToStatic(p.hidden_dim // num_heads) - ]) - source_projected = self.ProcessProjectionVec(theta, source_projected, - 'source') - if p.use_source_vec_as_attention_value: - source_contexts_reshaped = source_projected - else: - if p.enable_ctx_pre_proj: - source_contexts_projected = fns.qbatchmatmul( - tf.reshape(source_contexts, - [-1, py_utils.GetShape(source_contexts)[2]]), - fns.qweight(theta.ctx_proj), - qt='ctx_pre_proj_matmul') - if p.use_bias: - source_contexts_projected = fns.qadd( - source_contexts_projected, - fns.qweight(theta.ctx_proj_b), - qt='ctx_pre_proj_add') - else: - source_contexts_projected = source_contexts - - source_context_depth = py_utils.GetShape(source_contexts_projected)[-1] - source_contexts_reshaped = tf.reshape(source_contexts_projected, [ - time_steps, batch_size * num_heads, - source_context_depth // num_heads - ]) - source_contexts_projected = self.ProcessProjectionVec( - theta, source_contexts_projected, 'ctx') - - with tf.name_scope('init__2'): - source_padding_replicated = tf.reshape( - tf.tile( - tf.reshape(source_padding, [time_steps, batch_size, 1]), - [1, 1, num_heads]), [time_steps, batch_size * num_heads]) - if source_segment_id is None: - source_segment_id_repl = tf.zeros_like(source_padding_replicated) - else: - source_segment_id_repl = tf.reshape( - tf.tile( - tf.reshape(source_segment_id, [time_steps, batch_size, 1]), - [1, 1, num_heads]), [time_steps, batch_size * num_heads]) - - return self.atten.PackSource(theta.atten, source_projected, - source_contexts_reshaped, - source_padding_replicated, - source_segment_id_repl) - - @py_utils.NameScopeDecorator('MultiHeadedAttention/ExtendSourcePacked') - def ExtendSourcePacked(self, - theta, - new_source_vecs, - new_source_contexts, - new_source_paddings, - new_source_segment_ids, - cached_packed_src, - t=None): - """Extend cached source_vecs and source_contexts by one more timestep. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - new_source_vecs: A tensor of shape [source_batch, source_dim]. - new_source_contexts: A tensor of shape [source_batch, context_dim]. - new_source_vecs and new_source_contexts are source_vecs and - source_contexts for the new timestep to be extended. - new_source_paddings: If not None, a tensor of shape [source_batch]. - source_padding for the new timestep. - new_source_segment_ids: If not None, a tensor of shape [source_batch]. - source_segment_id for the new timestep. - cached_packed_src: a `.NestedMap` object, containing already preprocessed - source_vecs and source_contexts for the previous t-1 steps. To support - tf.while_loop on TPU (satisfying static shape requirement), instead of - using tf.concat to update the cached vectors, the time dimension of each - cached vector is fixed as the max_sequence_length and inplace - update op is used to update the information for each time step: - * source_vecs: A tensor of shape [max_sequence_length, source_batch, - hidden_dim]. [:t, :, :] contains valid preprocessed source_vecs in the - previous t - 1 timesteps, the rests are invalid data. - * source_contexts: A tensor of shape [max_sequence_length, source_batch, - hidden_dim]. [:t, :, :] contains valid preprocessed source_contexts in - the previous t - 1 timesteps, the rests are invalid data. - * source_padding: If not None, a tensor of shape [max_sequence_length, - source_batch, num_heads]. [:t, :, :] contains cached source padding - for the previous t - 1 timesteps, the rests are invalid data. - * source_segment_id: If not None, a tensor of shape - [max_sequence_length, source_batch, num_heads]. [:t, :, :] contains - cached source segment id for the previous t - 1 timesteps, the rests - are invalid data. - When t is None (not running on TPU or the while loop is unrolled): - * source_vecs: A tensor of shape [t - 1, source_batch, hidden_dim]. - * source_contexts: A tensor of shape [t - 1, source_batch, hidden_dim]. - * source_padding: If not None, a tensor of shape [t - 1, source_batch, - num_heads], cached source padding for the previous t - 1 timesteps. - * source_segment_id: If not None, a tensor of shape [t - 1, - source_batch, num_heads], cached source segment id for the previous t - - 1 timesteps. - t: a scalar, the current time step, 0-based. - - Returns: - Extended cached source_vecs, source_contexts, source_paddings, and - source_segment_ids. The time dimension of each cached state is fixed: - 'extended_source_vec' is of shape [max_sequence_length, batch_size, - num_heads * dim]; - 'extended_source_context' is of shape [max_sequence_length, batch_size, - num_heads * dim]; - 'source_padding' is of shape [max_sequence_length, batch_size, num_heads]; - 'source_segment_id' is of shape [max_sequence_length, batch_size, - num_heads]. - But only [:(t + 1), :, :] contains valid data. - If t is not given, - 'extended_source_vec' is of shape [t, batch_size, num_heads * dim]; - 'extended_source_context' is of shape [t, batch_size, num_heads * dim]; - 'source_padding' is of shape [t, batch_size, num_heads]; - 'source_segment_id' is of shape [t, batch_size, num_heads]. - """ - batch_size = py_utils.GetShape(new_source_vecs)[0] - if new_source_paddings is None: - new_source_paddings = tf.zeros([batch_size], dtype=new_source_vecs.dtype) - if new_source_segment_ids is None: - new_source_segment_ids = tf.zeros([batch_size], - dtype=new_source_vecs.dtype) - processed_packed_src = self.InitForSourcePacked( - theta, tf.expand_dims(new_source_vecs, 0), - tf.expand_dims(new_source_contexts, 0), - tf.expand_dims(new_source_paddings, 0), - tf.expand_dims(new_source_segment_ids, 0)) - extended_packed_src = py_utils.NestedMap() - for key in ('source_vecs', 'source_contexts', 'source_padding', - 'source_segment_id'): - if cached_packed_src.get(key, None) is None: - extended_packed_src[key] = None - else: - if t is not None: - processed = tf.reshape(processed_packed_src[key], [batch_size, -1]) - # Make sure t is a scaler instead of tensors having shape like [1,]. - # This could happen in cases where function is called by recurrent.py - # (for example target_sequence_sampler.) - t = tf.reshape(t, []) - extended_packed_src[key] = inplace_ops.alias_inplace_update( - cached_packed_src[key], t, processed) - else: - processed = tf.reshape(processed_packed_src[key], [1, batch_size, -1]) - extended_packed_src[key] = tf.concat( - [cached_packed_src[key], processed], axis=0) - return extended_packed_src - - @py_utils.NameScopeDecorator('MultiHeadedAttention/ZeroAttentionState') - def ZeroAttentionState(self, source_length, decoder_batch_size): - zero_att_state = self.atten.ZeroAttentionState( - source_length, decoder_batch_size * self.params.num_attention_heads) - # [batch * num_heads, length] => [batch, num_heads * length]. - zero_att_state = _RecursiveReshape(zero_att_state, [decoder_batch_size, -1]) - nested_map_zero_att_state = py_utils.NestedMap(inner=zero_att_state) - if self.params.attention_head_prob_index >= 0: - selected_prob_head = tf.zeros([decoder_batch_size, source_length]) - nested_map_zero_att_state[ - 'selected_attention_head_probs'] = selected_prob_head - return nested_map_zero_att_state - - def ProcessProjectionVec(self, theta, projection_vec, projection_type): - # no-op for this class but allows subclasses to override to process - # projected vectors. - return projection_vec - - @py_utils.NameScopeDecorator( - 'MultiHeadedAttention/ComputeContextVectorWithSource') - def ComputeContextVectorWithSource(self, - theta, - packed_src, - query_vec, - attention_state=None, - per_step_source_padding=None, - query_segment_id=None, - atten_idx=None): - """Computes the context vector given the current query output. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - packed_src: A `.NestedMap` object returned by PackSource or - InitForSourcePacked. - query_vec: a tensor of shape [target_batch, query_dim]. - attention_state: A NestedMap. 'inner' contains the inner attention - state. It is not used in AdditiveAttention, and is simply passed - through. Optionally, if attention_head_prob_index >= 0, then - 'selected_attention_head_probs' contains the selected attention - probability head. - per_step_source_padding: Source sequence padding to apply at this step. If - not None, it should be of shape [target_batch_size, source_length]. - query_segment_id: a tensor of shape [target_batch]. - atten_idx: If not None, then apply a different attention projection for - different samples in a batch, each of which may come from different - tasks. This is usually used in multi-task setting. A tensor of shape - [target_batch]. - Note: concated_source_vecs are the vectors that are used to compute the - attention score between the query_vec and each concated_source_vec. The - concated_source_contexts are the vectors that compose the result. The - attention context vector is computed as a weighted average of the - concated_source_contexts, using the scores that were computed using - concated_source_vecs. - - Returns: - A tuple of 3 elements. - - - The attention context vector: [batch_size, context_dim] - - The attention probability vector: [batch_size, time] - - The new attention mechanism state: A nested tuple of tensors with - dimensions [target_batch, ...]. See input 'attention_state' for - description of items in the nested tuple. - """ - p = self.params - fns = self.fns - source_padding = packed_src.source_padding - source_seq_len = py_utils.GetShape(source_padding)[0] - num_heads = p.num_attention_heads - batch_size = py_utils.GetShape(query_vec)[0] - static_inner_atten_dim = symbolic.ToStatic(p.hidden_dim // num_heads) - query_vec_projected_shape = [batch_size * num_heads, static_inner_atten_dim] - - if p.enable_query_proj: - query_vec_projected = fns.qbatchmatmul( - query_vec, fns.qweight(theta.query_proj), qt='query_proj_matmul') - if p.use_bias: - query_vec_projected = fns.qadd( - query_vec_projected, - fns.qweight(theta.query_proj_b), - qt='query_proj_add') - query_vec_projected = tf.reshape(query_vec_projected, - query_vec_projected_shape) - query_vec_projected = self.ProcessProjectionVec(theta, - query_vec_projected, - 'query') - else: - query_vec_projected = tf.reshape(query_vec, query_vec_projected_shape) - - query_batch_size = py_utils.GetShape(query_vec)[0] - if query_segment_id is None: - query_segment_id = tf.zeros( - query_batch_size * num_heads, dtype=source_padding.dtype) - else: - query_segment_id_repl = tf.tile( - tf.expand_dims(query_segment_id, 1), [1, num_heads]) - query_segment_id = tf.reshape(query_segment_id_repl, [-1]) - - if per_step_source_padding is None: - zero = tf.constant(0.0, dtype=query_vec.dtype) - per_step_source_padding = tf.fill([query_batch_size, source_seq_len], - zero) - per_step_source_padding = py_utils.HasShape( - per_step_source_padding, [query_batch_size, source_seq_len]) - per_step_source_padding = tf.reshape( - tf.tile(per_step_source_padding, [1, num_heads]), [-1, source_seq_len]) - attention_state = _RecursiveReshape(attention_state, - [batch_size * num_heads, -1]) - if isinstance(attention_state, py_utils.NestedMap): - if 'emit_probs' in attention_state: - inner_state = attention_state - elif 'inner' in attention_state: - inner_state = attention_state.inner - else: - inner_state = attention_state - ctx_vec, prob, new_inner_state = self.atten.ComputeContextVectorWithSource( - theta.atten, packed_src, query_vec_projected, inner_state, - per_step_source_padding, query_segment_id) - ctx_vec = tf.reshape(ctx_vec, [batch_size, -1]) - if p.enable_ctx_post_proj: - if atten_idx is None: - assert p.num_post_proj == 1, ( - 'atten_idx is None, this means there is no need to select ' - 'different post projections, and p.num_post_proj is supposed to be ' - '1. However you set p.num_post_proj=%s .' % p.num_post_proj) - ctx_vec = fns.qbatchmatmul( - ctx_vec, - fns.qweight(theta.ctx_post_proj), - qt='ctx_post_proj_matmul') - if p.use_bias: - ctx_vec = fns.qadd( - ctx_vec, - fns.qweight(theta.ctx_post_proj_b), - qt='ctx_post_proj_add') - else: - assert p.num_post_proj > 1, ( - 'atten_idx is not None, this means there are multiple post ' - 'projections, and p.num_post_proj is supposed to be > 1. However ' - 'you set p.num_post_proj=%s .' % p.num_post_proj) - bs_range = [tf.range(batch_size)] - select = tf.transpose(tf.concat([bs_range, [atten_idx]], axis=0)) - # => [batch, dim, num_langs] - ctx_vec = tf.einsum('ab,bcd->acd', ctx_vec, theta.ctx_post_proj) - if p.use_bias: - ctx_vec += tf.expand_dims(theta.ctx_post_proj_b, 0) - # => [batch, num_langs, dim] - ctx_vec = tf.transpose(ctx_vec, [0, 2, 1]) - # => [batch, dim] - ctx_vec = tf.gather_nd(ctx_vec, select) - ctx_vec = self.ProcessProjectionVec(theta, ctx_vec, 'ctx_post') - - # explicitly name this tensor for potential future reference - multi_headed_atten_prob = tf.reshape( - prob, [batch_size, num_heads, -1], name='multi_headed_atten_prob') - # TODO(laurenzo): Use a better named range function (we want to represent - # 0..1 probs). - prob = self.QRSoftmax(tf.reduce_mean(multi_headed_atten_prob, 1)) - if isinstance(attention_state, py_utils.NestedMap): - att_state = attention_state - if 'emit_probs' in attention_state: - att_state = new_inner_state - elif 'inner' in attention_state: - att_state.inner = new_inner_state - else: - att_state = new_inner_state - if p.attention_head_prob_index >= 0: - selected_prob_head = multi_headed_atten_prob[:, p. - attention_head_prob_index, :] - att_state.selected_attention_head_probs = selected_prob_head - att_state = _RecursiveReshape(att_state, [batch_size, -1]) - return ctx_vec, prob, att_state - - @py_utils.NameScopeDecorator( - 'MultiHeadedAttention/ComputeContextVectorWithAttenProbs') - def ComputeContextVectorWithAttenProbs(self, theta, packed_context, - atten_probs): - """Computes the context vector given the attention probailities. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - packed_context: Concated source contexts with shape [ batch_size * - num_heads, time, context_dim // num_heads]. - atten_probs: The attention probability vector: [batch_size * num_heads, - time]. - - Returns: - The attention context vector shaped [target_batch, source_dim]. - If p.enable_ctx_post_proj is false, source_dim = context_dim, - otherwise, source_dim = p.ctx_post_proj_dim. - """ - p = self.params - num_heads = p.num_attention_heads - # packed_context: [batch_size * num_head, num_style, - # hidden_dim / num_head] - # inp: [batch_size * num_head, num_style] - packed_context = py_utils.with_dependencies([ - py_utils.assert_shape_match([py_utils.GetShape(packed_context)[0]], - [py_utils.GetShape(atten_probs)[0]]) - ], packed_context) - b_size = py_utils.GetShape(packed_context)[0] // num_heads - ctx_vec = tf.reshape( - tf.matmul(tf.expand_dims(atten_probs, 1), packed_context), [b_size, -1]) - if p.enable_ctx_post_proj: - ctx_vec_proj = tf.matmul(ctx_vec, theta.ctx_post_proj) - ctx_vec_proj += theta.ctx_post_proj_b - ctx_vec_proj = self.ProcessProjectionVec(theta, ctx_vec_proj, 'ctx_post') - else: - ctx_vec_proj = ctx_vec - return ctx_vec_proj, ctx_vec - - def PackCachedSource(self, cached_src): - p = self.params - concated_source_vecs = cached_src.source_vecs - concated_source_contexts = cached_src.source_contexts - source_padding = cached_src.source_padding - source_segment_id = cached_src.source_segment_id - batch_size = py_utils.GetShape(concated_source_vecs)[1] - src_seq_len = py_utils.GetShape(concated_source_vecs)[0] - num_heads = p.num_attention_heads - packed_src = py_utils.NestedMap() - packed_src.source_vecs = tf.reshape( - concated_source_vecs, [src_seq_len, batch_size * num_heads, -1]) - # TODO(yonghui): Rewrite the following with just one transpose. - packed_src.source_contexts = tf.transpose( - tf.reshape(concated_source_contexts, - [src_seq_len, batch_size * num_heads, -1]), [1, 0, 2]) - if source_padding is not None: - packed_src.source_padding = tf.reshape( - source_padding, [src_seq_len, batch_size * num_heads]) - else: - packed_src.source_padding = tf.zeros( - [src_seq_len, batch_size * num_heads], dtype=py_utils.FPropDtype(p)) - if source_segment_id is None: - packed_src.source_segment_id = tf.zeros( - [src_seq_len, batch_size * num_heads], - dtype=packed_src.source_padding.dtype) - else: - packed_src.source_segment_id = tf.reshape( - source_segment_id, [src_seq_len, batch_size * num_heads]) - return packed_src - - @py_utils.NameScopeDecorator( - 'MultiHeadedAttention/ComputeContextVectorWithCachedSource') - def ComputeContextVectorWithCachedSource(self, - theta, - cached_src, - query_vec, - attention_state=None, - per_step_source_padding=None, - query_segment_id=None): - """Same as the ComputeContextVectorWithSource api above, except values ... - - in source_vecs, source_contexts and source_padding are ordered differently. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - cached_src: A `.NestedMap` object returned by ExtendSourcePacked. - query_vec: a tensor of shape [target_batch, query_dim]. - attention_state: previous attention state. It is not used in - AdditiveAttention, and is simply passed through. - per_step_source_padding: Source sequence padding to apply at this step. If - not None, it should be of shape [target_batch_size, source_length]. - query_segment_id: a tensor of shape [target_batch]. - - Returns: - A tuple of 3 tensors: - - - The attention context vector: [target_batch, source_dim] - - The attention probability vector: [target_batch, time] - - The new attention mechanism state: possibly nested tuple of tensors with - dimensions [target_batch....] - """ - return self.ComputeContextVectorWithSource( - theta, self.PackCachedSource(cached_src), query_vec, attention_state, - per_step_source_padding, query_segment_id) - - -class LocationSensitiveAttention(BaseAttentionLayer): - """An attention that also takes into account previously attended locations. - - See section 2.2 of this paper for a description of this technique: - http://papers.nips.cc/paper/5847-attention-based-models-for-speech-recognition.pdf - """ - - @classmethod - def Params(cls): - """Params for this LocationSensitiveAttention class.""" - p = super().Params() - p.Define('source_dim', 0, 'Number of source nodes.') - p.Define('location_filter_size', 0, - 'Location filter size, should be an odd number e.g. 31.') - p.Define('location_num_filters', 0, 'Number of location filters, e.g. 32.') - p.Define('query_dim', 0, 'Number of query nodes.') - p.Define('hidden_dim', 0, 'Number of hidden nodes.') - p.Define( - 'same_batch_size', False, - 'True iff the source and target sequence has the same batch size.') - p.Define( - 'location_features', ['PREV_PROBS'], - 'List signals to run the convolutions on. Possible options are: ' - 'PREV_PROBS, CUMULATIVE_PROBS.') - - # Often the attention context output needs to be concated - # with tensors from another layer. This allows them to share - # quantization parameters. By convention, all attention layers - # need to include their context output vectors in this domain. - p.qdomain.Define('atten_context', None, - 'Quantization domain for attention context.') - - # Fill in reasonable default for params init - p.params_init = py_utils.WeightInit.GaussianSqrtDim() - return p - - def __init__(self, params): - """Constructs an LocationSensitiveAttention object.""" - super().__init__(params) - p = self.params - self._is_quantized = p.qdomain.default is not None - assert not p.packed_input, ('Packed input is not supported yet for ' - 'LocationSensitiveAttention.') - - if p.atten_dropout_prob != 0: - raise NotImplementedError('dropout is not supported') - - def AttenLogits(inputs): - """Generates logits.""" - fns = self.fns - - def CollapseOutDim(x): - return tf.reshape(x, [-1, tf.shape(x)[-1]]) - - # => [sl, sb, hd] - location_feats = tf.transpose(inputs.location_feats, [2, 0, 1]) - location_hidden = fns.qmatmul( - CollapseOutDim(location_feats), inputs.location_var, qt='logits_mul') - - sl = py_utils.GetShape(location_feats)[0] - tb = py_utils.GetShape(location_feats)[1] - hd = py_utils.GetShape(inputs.location_var)[1] - location_hidden = tf.reshape(location_hidden, [sl, tb, hd]) - sb = py_utils.GetShape(inputs.query_vec_reshaped)[2] - bs_mult = py_utils.GetShape(inputs.query_vec_reshaped)[1] - location_hidden = tf.reshape(location_hidden, [sl, bs_mult, sb, hd]) - - # Shape of summed is [sl, tb/sb, sb, hidden_dim]. - summed = fns.qadd( - inputs.concated_source_vecs, - inputs.query_vec_reshaped, - qt='logits_add') - summed = fns.qadd(summed, location_hidden, qt='logits_bias') - summed = fns.qtanh(summed) - # logits is of shape [sl * tb/sb * sb, 1]. Computes dot product - # between v with every rows in 'summed'. Then we reshape the - # result to be of shape [sl, tb/sb, sb]. - logits = fns.qmatmul( - tf.reshape(summed, [-1, p.hidden_dim]), - tf.reshape(inputs.hidden_v, [p.hidden_dim, 1]), - qt='logits') - logits = tf.reshape(logits, py_utils.GetShape(summed)[:3]) - return logits - - def AttenLogitsSameBatchSize(inputs): - """Generates logits. - - Optimized code path for when the target and the source have the same batch - size. - - Args: - inputs: a NestedMap containing: - - concated_source_vecs: Tensor of shape [sl, batch, dim] - - query_vec_transformed: Tensor of shape [batch, dim] - - hidden_v: Tensor of shape [dim] - - location_feats: Tensor of shape [batch, location_feature_dim, sl] - - location_var: Tensor of shape [location_feature_dim, dim] - - Returns: - logits in the shape [sl, batch_size]. - """ - - def CollapseOutDim(x): - return tf.reshape(x, [-1, tf.shape(x)[-1]]) - - fns = self.fns - # => [sl, sb, hd] - location_feats = tf.transpose(inputs.location_feats, [2, 0, 1]) - location_hidden = fns.qmatmul( - CollapseOutDim(location_feats), inputs.location_var, qt='logits_mul') - sl = tf.shape(location_feats)[0] - tb = tf.shape(location_feats)[1] - hd = tf.shape(inputs.location_var)[1] - location_hidden = tf.reshape(location_hidden, [sl, tb, hd]) - - # Shape of summed is [sl, sb, hidden_dim]. - summed = fns.qadd( - inputs.concated_source_vecs, - tf.expand_dims(inputs.query_vec_transformed, 0), - qt='logits_add') - - summed = fns.qadd(summed, location_hidden, qt='logits_bias') - summed = fns.qtanh(summed) - - # logits is of shape [sl * sb, 1]. Computes dot product - # between v with every rows in 'summed'. Then we reshape the - # result to be of shape [sl, tb]. - logits = fns.qmatmul( - tf.reshape(summed, [-1, p.hidden_dim]), - tf.reshape(inputs.hidden_v, [p.hidden_dim, 1]), - qt='logits') - logits = tf.reshape(logits, py_utils.GetShape(summed)[:2]) - return logits - - def Atten(hidden_var, query_var, source_padding, concated_source_vecs, - concated_source_contexts, query_vec, attention_state, - location_filter_var, location_var, per_step_source_padding): - """Computes the attention context vector.""" - p = self.params - # attention_state shape [batch, len(p.location_features), slen] - # it contains previous and accumulated attention probabilites. - attention_state = py_utils.HasShape(attention_state, - [-1, len(p.location_features), -1]) - - fns = self.fns - location_feats = self._ApplyConv(attention_state, location_filter_var) - - # concated_source_vecs is of shape [sl, sb, dims] - # concated_source_contexts is of shape [sb, sl, context_dim] - # query_vec is of shape [tb, dims] - sb = py_utils.GetShape(concated_source_vecs)[1] - tb = py_utils.GetShape(query_vec)[0] - multiplier = tb // sb - # concated_source_vecs is reshaped to [sl, 1, sb, hidden_dims] - concated_source_vecs = tf.expand_dims(concated_source_vecs, 1) - query_vec_transformed = fns.qmatmul( - query_vec, query_var, qt='atten_matmul') - # query_vec is reshaped to [1, tb/sb, sb, hidden_dims]. - query_vec_reshaped = tf.reshape(query_vec_transformed, - [1, multiplier, sb, p.hidden_dim]) - # logits is of shape [sl, tb/sb, sb] - logits = _ConditionalCallDefun( - self._is_quantized, AttenLogits, - py_utils.NestedMap( - concated_source_vecs=concated_source_vecs, - query_vec_reshaped=query_vec_reshaped, - hidden_v=hidden_var, - location_feats=location_feats, - location_var=location_var)) - # Take out the padding states. - # _source_padding is of shape [sl, sb]. - # reshaped to [sl, 1, sb]. - source_padding = tf.expand_dims(source_padding, 1) - per_step_source_padding = tf.reshape( - tf.transpose(per_step_source_padding), [-1, multiplier, sb]) - - source_padding = self.QRPadding( - tf.add(source_padding, per_step_source_padding)) - - # Reshape logits to a matrix of shape [tb, sl] and takes the - # softmax to compute the probabilities. - logits = tf.transpose(tf.reshape(logits, [-1, tb])) - source_padding = tf.transpose(tf.reshape(source_padding, [-1, tb])) - probs = self._PaddedSoftmax( - logits, source_padding, narrow_to_asym_bit_depth=True) - # Reshape probs to be of shape [tb/sb, sb, sl]. - probs_reshaped = tf.reshape(probs, [multiplier, sb, -1]) - # Transpose probs to be of shape [sb, tb/sb, sl] - probs_reshaped = tf.transpose(probs_reshaped, [1, 0, 2]) - # [sb, tb/sb, sl] * [sb, sl, context_dim] = [sb, tb/sb, context_dim] - summed = fns.qbatchmatmul( - tf.cast(probs_reshaped, concated_source_contexts.dtype), - concated_source_contexts, - qt='atten_context') - # summed is of shape [tb/sb, sb, context_dim] - summed = tf.transpose(summed, [1, 0, 2]) - return tf.reshape(summed, [tb, -1]), probs - - def AttenSameBatchSize(hidden_var, query_var, source_padding, - concated_source_vecs, concated_source_contexts, - query_vec, attention_state, location_filter_var, - location_var, per_step_source_padding): - """Computes the attention context vector. - - Optimized code path for when source and target have the same batch size. - """ - del per_step_source_padding - p = self.params - # attention_state shape [batch, len(p.location_features), slen] - # it contains previous and accumulated attention probabilites. - attention_state = py_utils.HasShape(attention_state, - [-1, len(p.location_features), -1]) - - fns = self.fns - location_feats = self._ApplyConv(attention_state, location_filter_var) - query_vec_transformed = fns.qmatmul( - query_vec, query_var, qt='atten_matmul') - # logits is of shape [sl, sb] - logits = _ConditionalCallDefun( - not self._is_quantized, AttenLogitsSameBatchSize, - py_utils.NestedMap( - concated_source_vecs=concated_source_vecs, - query_vec_transformed=query_vec_transformed, - hidden_v=hidden_var, - location_feats=location_feats, - location_var=location_var)) - # => [sl, tb] - logits.set_shape(source_padding.shape) - # Reshape logits to a matrix of shape [tb, sl] and takes the - # softmax to compute the probabilities. - logits = tf.transpose(logits) - source_padding = tf.transpose(source_padding) - probs = self._PaddedSoftmax( - logits, source_padding, narrow_to_asym_bit_depth=True) - summed = fns.qbatchmatmul( - tf.cast(tf.expand_dims(probs, 1), concated_source_contexts.dtype), - concated_source_contexts, - qt='atten_context') - return tf.squeeze(summed, 1), probs - - if p.same_batch_size: - self._ctx_vec = AttenSameBatchSize - else: - self._ctx_vec = Atten - - def EncodeSource(src_w, vecs, ctxs): - fns = self.fns - time, batch = py_utils.GetShape(vecs, 2) - ctxs = py_utils.HasShape(ctxs, [time, batch, -1]) - transformed_vecs = tf.reshape( - fns.qmatmul( - tf.reshape(vecs, [-1, p.source_dim]), src_w, qt='encode_matmul'), - [time, batch, -1]) - transposed_ctxs = tf.transpose(ctxs, [1, 0, 2]) - return transformed_vecs, transposed_ctxs - - self._encode_source = EncodeSource - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - - pc = py_utils.WeightParams( - shape=[p.source_dim, p.hidden_dim], - init=p.params_init, - dtype=p.dtype, - collections=['LocationSensitiveAttention_vars']) - self.CreateVariable('source_var', pc, self.AddGlobalVN) - - pc = py_utils.WeightParams( - shape=[p.query_dim, p.hidden_dim], - init=p.params_init, - dtype=p.dtype, - collections=['LocationSensitiveAttention_vars']) - self.CreateVariable('query_var', pc, self.AddGlobalVN) - - pc = py_utils.WeightParams( - shape=[p.hidden_dim], - init=p.params_init, - dtype=p.dtype, - collections=['LocationSensitiveAttention_vars']) - self.CreateVariable('hidden_var', pc, self.AddGlobalVN) - - assert p.location_filter_size % 2 == 1 - assert p.location_num_filters > 0 - - location_filter_shape = [ - p.location_filter_size, - len(p.location_features), p.location_num_filters - ] - # TODO(yonghui): Don't hard code how params are initialized. - location_filter_pc = py_utils.WeightParams( - shape=location_filter_shape, - init=py_utils.WeightInit.Uniform(0.05), - dtype=p.dtype, - collections=['LocationSensitiveAttention_vars']) - self.CreateVariable('location_filter_var', location_filter_pc, - self.AddGlobalVN) - location_var_shape = [p.location_num_filters, p.hidden_dim] - location_pc = py_utils.WeightParams( - shape=location_var_shape, - init=py_utils.WeightInit.Uniform(0.05), - dtype=p.dtype, - collections=['LocationSensitiveAttention_vars']) - self.CreateVariable('location_var', location_pc, self.AddGlobalVN) - - self.TrackQTensor('atten_conv') - self.TrackQTensor('atten_context', domain='atten_context') - self.TrackQTensor( - 'atten_matmul', - 'logits_add', - 'encode_matmul', - 'logits_mul', - 'logits_bias', - domain='fullyconnected') - - def _ApplyConv(self, attention_state, location_filter_var): - """Applies the convolution on attention state.""" - p = self.params - fns = self.fns - attention_state_f32 = attention_state - location_filter_var_f32 = location_filter_var - if p.dtype != tf.float32: - attention_state_f32 = tf.cast(attention_state, tf.float32) - location_filter_var_f32 = tf.cast(location_filter_var, tf.float32) - data_format = 'NCW' - if not py_utils.use_xla(): - # NCW format is not supported on CPU. - attention_state_f32 = tf.transpose(attention_state_f32, [0, 2, 1]) - data_format = 'NWC' - location_feats = fns.qconv1d( - attention_state_f32, - location_filter_var_f32, - 1, - 'SAME', - data_format=data_format, - qt='atten_conv') - if not py_utils.use_xla(): - location_feats = tf.transpose(location_feats, [0, 2, 1]) - if p.dtype != tf.float32: - location_feats = tf.cast(location_feats, p.dtype) - # [sb, hd, sl] - return location_feats - - def PackSource(self, - theta, - source_vecs, - source_contexts, - source_padding, - source_segment_id=None): - with tf.name_scope(self.params.name): - if source_segment_id is None: - source_segment_id = tf.zeros_like(source_padding) - (concated_source_vecs, concated_source_contexts) = ( - self._encode_source( - self.QWeight(theta.source_var), source_vecs, source_contexts)) - return py_utils.NestedMap( - # [time, batch_size, hidden_dim]. - source_vecs=concated_source_vecs, - # [batch_size, time, context_dim]. - # Note the mismatch between `source_vecs` and `source_contexts`. In - # `source_vecs`, time is the first dim, while it is the second dim in - # `source_contexts`. - source_contexts=concated_source_contexts, - # [time, batch_size]. - source_padding=source_padding, - # [time, batch_size]. - source_segment_id=source_segment_id) - - def ZeroAttentionState(self, source_length, decoder_batch_size): - p = self.params - dtype = p.dtype.real_dtype - num_features = len(p.location_features) - with tf.name_scope(p.name): - state = tf.concat([ - tf.ones([decoder_batch_size, num_features, 1], dtype=dtype), - tf.zeros([decoder_batch_size, num_features, source_length - 1], - dtype=dtype) - ], 2) - - state = self.QRSoftmax(state) - return state - - def ComputeContextVectorWithSource(self, - theta, - packed_src, - query_vec, - attention_state=None, - per_step_source_padding=None, - query_segment_id=None): - """Computes the context vector given the current query output. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - packed_src: A `.NestedMap` object returned by PackSource or - InitForSourcePacked. - query_vec: a tensor of shape [batch_size, query_dim]. - attention_state: If `params().location_features == ['PREV_PROBS', - 'CUMULATIVE_PROBS']`, then `attention_state` is a tensor of shape - [batch_size, 2, src_len]. - - - attention_state[:, 0, :] contains previous attention probabilities. - - attention_state[:, 1, :] contains a sum over previous timesteps of - attention probabilities. - - per_step_source_padding: Source sequence padding to apply at this step. If - not None, it should be of shape [target_batch_size, source_length]. - query_segment_id: Query segment id with shape [batch_size]. - - Note: concated_source_vecs are the vectors that are used to compute the - attention score between the query_vec and each concated_source_vec. The - concated_source_contexts are the vectors that compose the result. The - attention context vector is computed as a weighted average of the - concated_source_contexts, using the scores that were computed using - concated_source_vecs. - - Returns: - A tuple of 3 elements. - - - The attention context vector: [batch_size, context_dim] - - The attention probability vector: [batch_size, time] - - The new attention mechanism state: possibly nested tuple of tensors with - dimensions [target_batch, ...] - """ - del query_segment_id - p = self.params - concated_source_vecs = packed_src.source_vecs - concated_source_contexts = packed_src.source_contexts - source_padding = packed_src.source_padding - if p.same_batch_size: - assert per_step_source_padding is None - query_batch_size = py_utils.GetShape(query_vec)[0] - source_length = py_utils.GetShape(source_padding)[0] - if per_step_source_padding is None: - zero = tf.constant(0.0, dtype=query_vec.dtype) - per_step_source_padding = tf.fill([query_batch_size, source_length], zero) - per_step_source_padding = py_utils.HasShape( - per_step_source_padding, [query_batch_size, source_length]) - - hidden = py_utils.AddPerStepVN(p, theta.hidden_var) - query = py_utils.AddPerStepVN(p, theta.query_var) - location_filter = py_utils.AddPerStepVN(p, theta.location_filter_var) - location = py_utils.AddPerStepVN(p, theta.location_var) - - ctx_vec, prob = self._ctx_vec(hidden, query, source_padding, - concated_source_vecs, - concated_source_contexts, query_vec, - attention_state, location_filter, location, - per_step_source_padding) - - new_feats = {'PREV_PROBS': prob} - if 'CUMULATIVE_PROBS' in p.location_features: - # Quantization must match the _PaddedSoftmax method. - cum_prob_index = p.location_features.index('CUMULATIVE_PROBS') - new_feats['CUMULATIVE_PROBS'] = self.QRSoftmax( - tf.add(prob, attention_state[:, cum_prob_index, :]), - narrow_to_asym_bit_depth=True) - new_attention_state = tf.stack([new_feats[f] for f in p.location_features], - axis=1) - return ctx_vec, prob, new_attention_state - - -def MergeSourcePaddingWithPerStepSourcePadding(source_padding, - per_step_source_padding, tb): - """Merges source padding with per-step source padding. - - Args: - source_padding: [sl, sb]. - per_step_source_padding: [tb, sl]. - tb: target batch size. - - Returns: - A tensor of shape [tb, sl]. - """ - # source_padding is of shape [sl, sb]. - sl = py_utils.GetShape(source_padding)[0] - sb = py_utils.GetShape(source_padding)[1] - - if per_step_source_padding is None: - zero = tf.constant(0.0, dtype=source_padding.dtype) - per_step_source_padding = tf.fill([tb, sl], zero) - per_step_source_padding = py_utils.HasShape(per_step_source_padding, [tb, sl]) - - # Transpose and reshape source_padding to [1, sb, sl]. - source_padding = tf.expand_dims(tf.transpose(source_padding), 0) - # Merge source_padding and per_step_source_padding. - source_padding = tf.maximum(source_padding, - tf.reshape(per_step_source_padding, [-1, sb, sl])) - return tf.reshape(source_padding, [tb, -1]) - - -class MonotonicAttention(BaseAttentionLayer): - """An attention mechanism which enforces monotonic alignments. - - This layer implements the monotonic attention mechanism described in - Online and Linear-Time Attention by Enforcing Mononotonic Alignments - (https://arxiv.org/abs/1704.00784). It is used in exactly the same way as - AdditiveAttention, but both the attention distribution and the energy function - are different. - - Rather than using a softmax, this mechanism feeds the attention energy into a - (hard or soft) sigmoid and treats the output as Bernoulli probabilities - representing the probability of attending to a given entry in the input - sequence, processed from left-to-right. Based on this interpretation, the - resulting distribution over input sequence entries is computed with a dynamic - program. The intended use is to train with soft sigmoids according to the - expected output (setting param hard_sigmoid=False), then use hard sigmoids at - test time to allow for online and linear-time decoding. To encourge the train - and test-time behavior to be similar, noise can optionally be added to the - sigmoid activations during training (param pre_sigmoid_noise). For the energy - function, rather than computing:: - - E = dot(v, tanh(dot(W, query) + dot(W, encoder_states))) - - it computes:: - - E = dot(g*v/||v||, tanh(dot(W, query) + dot(W, encoder_states) + b)) + r - - where g and r are scalars and b is a vector, and ||v|| is the L2 norm of v. - instead. These modifications address the fact that the sigmoids in the - monotonic attention mechanism are sensitive to offset and a bit harder to - train compared to the softmax function. It can be helpful to initialize the - energy bias scalar r to a negative value (param hidden_bias_init). - """ - - @classmethod - def Params(cls): - """Params for this MonotonicAttention class.""" - p = super().Params() - p.Define('source_dim', 0, 'Number of source nodes.') - p.Define('query_dim', 0, 'Number of query nodes.') - p.Define('hidden_dim', 0, 'Number of hidden nodes.') - p.Define('pre_sigmoid_noise', 0, 'Standard deviation of pre-sigmoid noise.') - p.Define('hidden_bias_init', -1, 'Initial value of hidden bias.') - p.Define('hard_sigmoid', False, 'Whether to use a hard sigmoid.') - # Fill in reasonable default for params init - p.params_init = py_utils.WeightInit.GaussianSqrtDim() - return p - - def __init__(self, params): - """Constructs an MonotonicAttention object.""" - super().__init__(params) - p = self.params - assert not p.packed_input, ('Packed input not supported for Monotonic ' - 'Attention.') - if p.atten_dropout_prob != 0: - raise NotImplementedError('dropout is not supported') - - # When running eval, don't add pre-sigmoid noise, and use a hard sigmoid to - # match behavior of online decoding. - if self.do_eval: - p.pre_sigmoid_noise = 0. - p.hard_sigmoid = True - - def EncodeSource(src_w, vecs, ctxs): - time, batch = py_utils.GetShape(vecs, 2) - ctxs = py_utils.HasShape(ctxs, [time, batch, -1]) - transformed_vecs = tf.reshape( - py_utils.Matmul(tf.reshape(vecs, [-1, p.source_dim]), src_w), - [time, batch, -1]) - transposed_ctxs = tf.transpose(ctxs, [1, 0, 2]) - return transformed_vecs, transposed_ctxs - - self._encode_source = EncodeSource - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - - # source is the weight matrix for the memory/encoder states - pc = py_utils.WeightParams( - shape=[p.source_dim, p.hidden_dim], - init=p.params_init, - dtype=p.dtype, - collections=['MonotonicAttention_vars']) - self.CreateVariable('source_var', pc, self.AddGlobalVN) - - # query is the weight matrix for the query/decoder RNN state - pc = py_utils.WeightParams( - shape=[p.query_dim, p.hidden_dim], - init=p.params_init, - dtype=p.dtype, - collections=['MonotonicAttention_vars']) - self.CreateVariable('query_var', pc, self.AddGlobalVN) - - # hidden is the pre-softmax vector which converts from tanh to scalar - pc = py_utils.WeightParams( - shape=[p.hidden_dim], - init=p.params_init, - dtype=p.dtype, - collections=['MonotonicAttention_vars']) - self.CreateVariable('hidden_var', pc, self.AddGlobalVN) - - # energy_bias is the bias vector which appears inside of tanh - # Initialize the bias vector to all zeros - pc = py_utils.WeightParams( - shape=[p.hidden_dim], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=['MonotonicAttention_vars']) - self.CreateVariable('energy_bias_var', pc) - - # hidden_scale is the weight normalization scale for hidden - # Initialize so that the initial scale is 1/sqrt(hidden_dim) - pc = py_utils.WeightParams( - shape=[], - init=py_utils.WeightInit.Constant(1 / np.sqrt(p.hidden_dim)), - dtype=p.dtype, - collections=['MonotonicAttention_vars']) - self.CreateVariable('hidden_scale_var', pc) - - # hidden_bias is the bias scalar applied before the sigmoid - # Use the hidden_bias_init hyperparam to set the initial value - pc = py_utils.WeightParams( - shape=[], - init=py_utils.WeightInit.Constant(p.hidden_bias_init), - dtype=p.dtype, - collections=['MonotonicAttention_vars']) - self.CreateVariable('hidden_bias_var', pc) - - def PackSource(self, - theta, - source_vecs, - source_contexts, - source_padding, - source_segment_id=None): - with tf.name_scope(self.params.name): - if source_segment_id is None: - source_segment_id = tf.zeros_like(source_padding) - (concated_source_vecs, concated_source_contexts) = ( - self._encode_source(theta.source_var, source_vecs, source_contexts)) - return py_utils.NestedMap( - # [time, batch_size, hidden_dim]. - source_vecs=concated_source_vecs, - # [batch_size, time, context_dim]. - # Note the mismatch between `source_vecs` and `source_contexts`. In - # `source_vecs`, time is the first dim, while it is the second dim in - # `source_contexts`. - source_contexts=concated_source_contexts, - # [time, batch_size]. - source_padding=source_padding, - # [time, batch_size]. - source_segment_id=source_segment_id) - - def ZeroAttentionState(self, source_length, decoder_batch_size): - p = self.params - dtype = p.dtype - with tf.name_scope(p.name): - # Set initial previous attention to [1, 0, ... 0] to avoid special-casing - emit_probs = tf.one_hot( - tf.zeros((decoder_batch_size,), dtype=tf.int32), - source_length, - dtype=dtype) - return py_utils.NestedMap(emit_probs=emit_probs) - - def ComputeProbabilities(self, theta, concated_source_vecs, - merged_source_padding, query_vec, attention_state): - """Computes probabilities of emissions.""" - - # concated_source_contexts is of shape [sb, sl, context_dim] - # query_vec is of shape [tb, dims] - sb = tf.shape(concated_source_vecs)[1] - tb = tf.shape(query_vec)[0] - multiplier = tb // sb - - p = self.params - - def AttenLogits(inputs): - """Computes logits from source, query, and variables. - - Args: - inputs: a NestedMap containing: - - concated_source_vecs: [sl, sb, hidden_dims]. - - query_vec: [tb, query_dim]. - - query_v: [query_dim, hidden_dim] - - energy_b: [hidden_dim]. - - hidden_v: [hidden_dim]. - - hidden_g: []. - - hidden_b: []. - - Returns: - logits shaped [tb, sl]. - """ - # Apply query matrix to query. Becomes [tb, hidden_dim]. - query_vec_transformed = py_utils.Matmul( - inputs.query_vec, inputs.query_v, name='query_transformation') - # query_vec is reshaped to [1, tb/sb, sb, hidden_dim]. - query_vec_reshaped = tf.reshape(query_vec_transformed, - [1, multiplier, sb, p.hidden_dim]) - - # [sl, 1, sb, hidden_dim]. - concated_source_vecs = tf.expand_dims(inputs.concated_source_vecs, 1) - energy_b = tf.reshape(inputs.energy_b, [1, 1, 1, -1]) - # Shape of summed is [sl, tb/sb, sb, hidden_dim]. - summed = tf.tanh(concated_source_vecs + query_vec_reshaped + energy_b) - hidden_v = inputs.hidden_g * tf.nn.l2_normalize(inputs.hidden_v, axis=0) - # logits is of shape [sl * tb/sb * sb, 1]. Computes dot product - # between v with every rows in 'summed'. Then we reshape the - # result to be of shape [sl, tb/sb, sb]. - # - # Another equivalent way is to do: - # logits = tf.reduce_sum(summed * - # tf.reshape(v, [1, 1, 1, hidden_dim]), 3) - logits = py_utils.Matmul( - tf.reshape(summed, [-1, p.hidden_dim]), - tf.reshape(hidden_v, [p.hidden_dim, 1])) - logits += inputs.hidden_b - # [tb, sl]. - logits = tf.transpose(tf.reshape(logits, [-1, tb]), [1, 0]) - return logits - - with tf.name_scope('logits'): - logits = py_utils.CallDefun( - AttenLogits, - py_utils.NestedMap( - concated_source_vecs=concated_source_vecs, - query_vec=query_vec, - query_v=theta.query_var, - energy_b=theta.energy_bias_var, - hidden_v=theta.hidden_var, - hidden_g=theta.hidden_scale_var, - hidden_b=theta.hidden_bias_var)) - - previous_attention = attention_state.emit_probs - with tf.name_scope('prob'): - if self.params.hard_sigmoid: - # If using a hard sigmoid, just compare against 0 - p_choose_i = tf.cast(tf.greater(logits, 0), logits.dtype) - # Never choose padded values. - p_choose_i = tf.where(merged_source_padding > 0.0, - tf.zeros_like(p_choose_i), p_choose_i) - # Compute probability distribution assuming hard probabilities - probs = MonotonicAttentionProb(p_choose_i, previous_attention, 'hard') - else: - # Compute pre-sigmoid noise. - activation_noise = tf.random.stateless_normal( - py_utils.GetShape(logits), - py_utils.GenerateStepSeedPair(p, theta.global_step), - dtype=logits.dtype) - # Compute sigmoid probabilities. - p_choose_i = tf.nn.sigmoid(logits + self.params.pre_sigmoid_noise * - activation_noise) - # Never choose padded values. - p_choose_i = tf.where(merged_source_padding > 0, - tf.zeros_like(p_choose_i), p_choose_i) - # Compute attention distribution - probs = MonotonicAttentionProb(p_choose_i, previous_attention, - 'parallel') - - # [tb, sl]. - return probs, py_utils.NestedMap(emit_probs=probs) - - def ComputeContextVectorWithSource(self, - theta, - packed_src, - query_vec, - attention_state, - per_step_source_padding=None, - query_segment_id=None): - """Computes the context vector given the current query output. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - packed_src: A `.NestedMap` object returned by PackSource or - InitForSourcePacked. - query_vec: a tensor of shape [batch_size, query_dim]. - attention_state: The attention probs computed at the previous timestep. - per_step_source_padding: Source sequence padding to apply at this step. If - not None, it should be of shape [target_batch_size, source_length]. - query_segment_id: a tensor of shape [batch_size]. - Note: concated_source_vecs are the vectors that are used to compute the - attention score between the query_vec and each concated_source_vec. The - concated_source_contexts are the vectors that compose the result. The - attention context vector is computed as a weighted average of the - concated_source_contexts, using the scores that were computed using - concated_source_vecs. - - Returns: - A tuple of 3 elements. - - - The attention context vector: [batch_size, context_dim] - - The attention probability vector: [batch_size, time] - - The attention probability vector: (again, to be interpreted as state). - """ - del query_segment_id - concated_source_vecs = packed_src.source_vecs - concated_source_contexts = packed_src.source_contexts - source_padding = packed_src.source_padding - sb = tf.shape(concated_source_vecs)[1] - tb = tf.shape(query_vec)[0] - multiplier = tb // sb - merged_source_padding = MergeSourcePaddingWithPerStepSourcePadding( - source_padding, per_step_source_padding, tb) - - probs, new_state = self.ComputeProbabilities(theta, concated_source_vecs, - merged_source_padding, - query_vec, attention_state) - - with tf.name_scope('sum'): - # Reshape probs to be of shape [tb/sb, sb, sl] - probs_reshaped = tf.reshape(probs, [multiplier, sb, -1]) - # Transpose probs to be of shape [sb, tb/sb, sl] - probs_reshaped = tf.transpose(probs_reshaped, [1, 0, 2]) - # Batched matmul - # [sb, tb/sb, sl] * [sb, sl, context_dim] = [sb, tb/sb, context_dim] - summed = tf.matmul(probs_reshaped, concated_source_contexts) - # summed is of shape [tb/sb, sb, context_dim] - summed = tf.transpose(summed, [1, 0, 2]) - ctx_vec = tf.reshape(summed, [tb, -1]) - - return ctx_vec, probs, new_state - - -class GmmMonotonicAttention(BaseAttentionLayer): - """A GMM-based monotonic attention module. - - Based on "Generating Sequences With Recurrent Neural Networks" by Alex Graves. - Eq [46-51] in https://arxiv.org/abs/1308.0850. - """ - - @classmethod - def Params(cls): - """Params for this MonotonicAttention class.""" - p = super().Params() - p.Define('source_dim', 0, 'Number of source nodes.') - p.Define('query_dim', 0, 'Number of query nodes.') - p.Define('hidden_dim', 128, - 'Number of hidden units for the MLP that predicts GMM params.') - p.Define('max_offset', -1, - 'Max offset to move attention pointer, Enabled only when > 0.') - p.Define('num_mixtures', 5, 'Number of location GMM components.') - p.Define( - 'normalize_probs', False, - 'Whether to normalize probabilities computed by GMM. Otherwise, ' - 'the attention weights (i.e. probabilities) may not add up to ' - '1.0.') - - # TODO(ngyuzh): find a good initialize for both TTS and ASR. Consider split - # the layer if it's very sensitive to the initialization - p.params_init = py_utils.WeightInit.Xavier(0.1) - return p - - def __init__(self, params): - """Constructs a GMM-based monotonic attention module.""" - super().__init__(params) - p = self.params - if p.atten_dropout_prob != 0: - raise NotImplementedError('dropout is not supported.') - - # TODO(ngyuzh): Compare Sigmoid and other activation functions. - ff_params = layers.FeedForwardNet.Params().Set( - name=p.name, - input_dim=p.query_dim, - hidden_layer_dims=[p.hidden_dim, p.num_mixtures * 3], - activation=['SIGMOID', 'NONE'], - params_init=p.params_init.Copy()) - self.CreateChild('GMM', ff_params) - - def ComputeProbs(encoder_positions, priors, means, variances): - """Computes the location GMM probabilities at all encoder positions. - - This function assumes that the first 2 dimensions of `priors`, `means`, - `variances`, and the return value: - `multiplier (target_batch / source_batch)` and `source_batch` are - transposed, and `encoder_positions` has only non-one dimensions. - - Args: - encoder_positions: [source_batch, source_length] - priors: [multiplier, source_batch, num_mixtures] - means: [multiplier, source_batch, num_mixtures] - variances: [multiplier, source_batch, num_mixtures] - - Returns: - Probabilities shaped [multiplier, source_batch, source_length]. - """ - # [multiplier, source_batch, 1, num_mixtures] - priors = tf.expand_dims(priors, 2) - means = tf.expand_dims(means, 2) - variances = tf.expand_dims(variances, 2) - epsilon = 1e-8 - - # [source_batch, source_length, 1] - encoder_positions = tf.expand_dims(encoder_positions, 2) - - # [multiplier, source_batch, source_length, num_mixtures] - probs = ((priors * tf.math.rsqrt(2 * np.pi * variances + epsilon)) * - tf.exp(-(encoder_positions - means)**2 / - (2 * variances + epsilon))) - - # [multiplier, source_batch, source_length] - return tf.reduce_sum(probs, axis=3) - - def Atten(source_padding, concated_source_vecs, concated_source_contexts, - query_vec, priors, means, variances, encoder_positions, - per_step_source_padding): - """Computes the attention context vector. - - Args: - source_padding: [source_length, source_batch] - concated_source_vecs: [source_length, source_batch, hidden_dim] - concated_source_contexts: [source_batch, source_length, context_dim] - query_vec: [target_batch, query_dim] - priors: [target_batch, num_mixtures] - means: [target_batch, num_mixtures] - variances: [target_batch, num_mixtures] - encoder_positions: [source_batch, source_length] - per_step_source_padding: [target_batch, source_length] - - Returns: - Tuple(context vector, atten probs): - - - context vector: [target_batch, context_dim] - - attention probabilities: [target_batch, source_length] - """ - # Note: shape [target_batch] can be converted to - # [multiplier, source_batch], not [source_batch, multiplier]. - p = self.params - source_batch = tf.shape(concated_source_vecs)[1] - target_batch = tf.shape(query_vec)[0] - multiplier = target_batch // source_batch - - # [multiplier, source_batch, num_mixtures] - priors = tf.reshape(priors, [multiplier, source_batch, p.num_mixtures]) - means = tf.reshape(means, [multiplier, source_batch, p.num_mixtures]) - variances = tf.reshape(variances, - [multiplier, source_batch, p.num_mixtures]) - - # [multiplier, source_batch, source_length] - probs = ComputeProbs(encoder_positions, priors, means, variances) - - # [source_batch, source_length] - source_padding = tf.transpose(source_padding) - - # [multiplier, source_batch, source_length] - per_step_source_padding = tf.reshape(per_step_source_padding, - [multiplier, source_batch, -1]) - source_padding += per_step_source_padding - source_padding = tf.minimum(source_padding, 1.0) - - # [multiplier, source_batch, source_length] - probs *= (1.0 - source_padding) - if p.normalize_probs: - probs /= tf.maximum(tf.reduce_sum(probs, axis=2, keepdims=True), 1e-12) - - # [source_batch, multiplier, source_length] - probs_transposed = tf.transpose(probs, [1, 0, 2]) - - # Matmul: - # [source_batch, multiplier, source_length] - # @ [source_batch, source_length, context_dim] - # -> [source_batch, multiplier, context_dim] - context_vector_transposed = tf.matmul(probs_transposed, - concated_source_contexts) - - # [multiplier, source_batch, context_dim] - context_vector = tf.transpose(context_vector_transposed, [1, 0, 2]) - - # [target_batch, context_dim], [target_batch, source_length] - return (tf.reshape(context_vector, [target_batch, -1]), - tf.reshape(probs, [target_batch, -1])) - - self._ctx_vec = Atten - - def EncodeSource(vecs, ctxs): - # TODO(ngyuzh): combine with content-base attention. - time, batch = py_utils.GetShape(vecs, 2) - ctxs = py_utils.HasShape(ctxs, [time, batch, -1]) - transposed_ctxs = tf.transpose(ctxs, [1, 0, 2]) - return vecs, transposed_ctxs - - self._encode_source = EncodeSource - - def PackSource(self, - theta, - source_vecs, - source_contexts, - source_padding, - source_segment_id=None): - with tf.name_scope(self.params.name): - if source_segment_id is None: - source_segment_id = tf.zeros_like(source_padding) - (concated_source_vecs, concated_source_contexts) = ( - self._encode_source(source_vecs, source_contexts)) - return py_utils.NestedMap( - # [source_length, source_batch, hidden_dim]. - source_vecs=concated_source_vecs, - # [source_batch, source_length, context_dim]. - # Note the mismatch between `source_vecs` and `source_contexts`. In - # `source_vecs`, `source_length` is the first dim, while it is the - # second dim in `source_contexts`. - source_contexts=concated_source_contexts, - # [source_length, source_batch]. - source_padding=source_padding, - # [source_length, source_batch]. - source_segment_id=source_segment_id) - - def ZeroAttentionState(self, source_length, decoder_batch_size): - p = self.params - - # [target_batch, num_mixtures] - position = tf.zeros([decoder_batch_size, p.num_mixtures], dtype=p.dtype) - position_offsets = tf.zeros([decoder_batch_size, p.num_mixtures], - dtype=p.dtype) - variances = tf.ones([decoder_batch_size, p.num_mixtures], dtype=p.dtype) - priors = tf.zeros([decoder_batch_size, p.num_mixtures], dtype=p.dtype) - - # [target_batch, num_mixtures, 4] - return tf.stack([position, position_offsets, variances, priors], axis=2) - - def ComputeContextVectorWithSource(self, - theta, - packed_src, - query_vec, - attention_state, - per_step_source_padding=None, - query_segment_id=None): - """Computes the context vector given the current query output. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - packed_src: A `.NestedMap` object returned by PackSource or - InitForSourcePacked. - query_vec: a tensor of shape [target_batch, query_dim]. - attention_state: previous attention state, a tensor of shape - [target_batch, num_mixtures, 4]. - - attention_state[:, :, 0] contains previous location - - attention_state[:, :, 1] contains previous offset. - - attention_state[:, :, 2] contains previous variance. - - attention_state[:, :, 3] contains previous prior. - per_step_source_padding: Source sequence padding to apply at this step. If - not None, it should be of shape [target_batch, source_length]. - query_segment_id: a tensor of shape [target_batch]. - Note: concated_source_vecs are the vectors that are used to compute the - attention score between the query_vec and each concated_source_vec. The - concated_source_contexts are the vectors that compose the result. The - attention context vector is computed as a weighted average of the - concated_source_contexts, using the scores that were computed using - concated_source_vecs. - - Returns: - A tuple of 3 elements. - - - The attention context vector: [target_batch, context_dim] - - The attention probability vector: [target_batch, source_length] - - The new attention state vector: [target_batch, num_mixtures, 4] - """ - del query_segment_id - p = self.params - concated_source_vecs = packed_src.source_vecs - concated_source_contexts = packed_src.source_contexts - source_padding = packed_src.source_padding - - target_batch = tf.shape(query_vec)[0] - source_length = tf.shape(source_padding)[0] - source_batch = tf.shape(source_padding)[1] - - # [target_batch, source_length] - if per_step_source_padding is None: - per_step_source_padding = tf.zeros([target_batch, source_length], - dtype=query_vec.dtype) - per_step_source_padding = py_utils.HasShape(per_step_source_padding, - [target_batch, source_length]) - - # [target_batch, num_mixtures * 3] - out = self.GMM.FProp(theta.GMM, query_vec) - - # [target_batch, num_mixtures] - priors_logits, position_offset_logits, log_variances = tf.split( - out, 3, axis=1, name='GMM') - - log_variances = tf.minimum(log_variances, layers.LOG_SCALE_CLAMP_BOUND) - variances = tf.exp(log_variances) - - priors = tf.nn.softmax(priors_logits) - - if p.max_offset > 0: - position_offset = tf.nn.sigmoid(position_offset_logits) - position_offset *= p.max_offset - else: - position_offset = tf.exp(position_offset_logits) - - new_position = attention_state[:, :, 0] + position_offset - - # Tile and reshape encoder_positions to [source_batch, source_length] - # so that it can be evaluated by locations GMMs in a vectorized way. - encoder_positions = tf.expand_dims( - tf.cast(tf.range(source_length), tf.float32), 0) - encoder_positions = tf.tile(encoder_positions, [source_batch, 1]) - - # [target_batch, context_dim], [target_batch, source_length] - ctx_vec, prob = self._ctx_vec(source_padding, concated_source_vecs, - concated_source_contexts, query_vec, priors, - new_position, variances, encoder_positions, - per_step_source_padding) - - # [target_batch, num_mixtures, 4] - new_atten_states = tf.stack( - [new_position, position_offset, variances, priors], axis=2) - - return ctx_vec, prob, new_atten_states - - -class MergerLayer(base_layer.BaseLayer): - """Merges a list of input tensors with various options into a single tensor. - - Implements a merger/combiner operator given a list of tensors. The merger - operator outputs a single tensor with the following options (merger_op): - - - atten: Applies attention over the set of input tensors given query vector. - - mean: Takes the mean of input tensors. - - concat: Concatenates the input tensors over the last dimension. - - sum: Sum up all the input tensors. - - weighted_sum: Use learnt weights to combine input tensors. - - gated_avg: Learnt input dependent gates are used to average tensors. - - This class is expected to be called by multi-source/multi-column models. - """ - - @classmethod - def Params(cls): - """Params for this MergerLayer class.""" - p = super().Params() - p.Define('merger_op', None, 'How to merge input tensors.') - p.Define('source_dim', 0, 'Number of source nodes.') - p.Define('query_dim', 0, 'Number of query nodes.') - p.Define('hidden_dim', 0, 'Number of hidden nodes.') - p.Define('attention_tpl', AdditiveAttention.Params(), - 'Attention used by the merger layer when merger_op is atten.') - p.Define( - 'pre_proj_input_dims', None, - 'If set, should be a list of depths for the tensors to be merged.' - ' Setting this will result in a pre-projection to source_dim' - ' before the merger.') - p.Define( - 'pre_proj_output_dims', None, - 'Should be a list of depths which the input tensors specified in ' - 'pre_proj_input_dims need to be projected to. Should match the length ' - 'of pre_proj_input_dims.') - p.Define( - 'proj_tpl', - layers.ProjectionLayer.Params().Set( - batch_norm=False, weight_norm=True, has_bias=True), - 'Configs template for the projection layer.') - p.Define('gated_avg_tpl', layers.GatedAverageLayer.Params(), - 'Configs template for the gated average layer.') - p.Define('num_sources', 0, 'If merger_op=weighted_sum, then must specify ' - 'num of sources.') - p.Define('post_proj', None, - 'Post projection for the merged context vector.') - return p - - # Merging operation keys supported by this layer. - MERGER_OPS = ['mean', 'atten', 'concat', 'sum', 'weighted_sum', 'gated_avg'] - - def __init__(self, params): - super().__init__(params) - p = self.params - if not p.name: - raise ValueError('Layer must have a specified name!') - if p.merger_op not in set(self.MERGER_OPS): - raise ValueError('Merger op must be one of: ', self.MERGER_OPS) - - if p.merger_op == 'atten': - atten_params = p.attention_tpl.Copy() - atten_params.source_dim = p.source_dim - atten_params.query_dim = p.query_dim - atten_params.hidden_dim = p.hidden_dim - atten_params.dtype = p.dtype - if atten_params.params_init is None: - atten_params.params_init = py_utils.WeightInit.Gaussian( - 1. / math.sqrt(atten_params.source_dim + atten_params.query_dim), - seed=p.random_seed) - self.CreateChild('atten', atten_params) - - if p.pre_proj_input_dims: - if not p.pre_proj_output_dims: - raise ValueError('Output dims should be specified for projection.') - if len(p.pre_proj_input_dims) != len(p.pre_proj_output_dims): - raise ValueError( - 'Output dims should be the same length as input dims. ' - 'Expected: %s obtained: %s' % - (len(p.pre_proj_input_dims), len(p.pre_proj_output_dims))) - pre_proj_params = [] - for i, (pre_proj_input_dim, pre_proj_output_dim) in enumerate( - zip(p.pre_proj_input_dims, p.pre_proj_output_dims)): - proj_p = p.proj_tpl.Copy() - proj_p.name = 'merger_pre_proj_%d' % i - proj_p.input_dim = pre_proj_input_dim - proj_p.output_dim = pre_proj_output_dim - pre_proj_params.append(proj_p) - self.CreateChildren('pre_proj', pre_proj_params) - - if p.merger_op == 'gated_avg': - assert p.num_sources > 0, ('For merger_op=gated_avg, must specify ' - 'num_sources > 0.') - params = p.gated_avg_tpl.Copy() - params.name = 'g_avg_merger' - params.num_nodes = p.source_dim - params.num_inputs = p.num_sources - self.CreateChild('gated_average', params) - - if p.post_proj: - self.CreateChild('post_proj', p.post_proj) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - - if p.merger_op == 'weighted_sum': - assert p.num_sources > 0, ('For merger_op=weighted_sum, must specify ' - 'num_sources > 0.') - params_init = py_utils.WeightInit.Constant(1.0 / p.num_sources) - # Weights to be learned. - pw = py_utils.WeightParams( - shape=[p.num_sources], - init=params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('sum_weight', pw) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - if 'atten' in self.children: - self.atten.InstantiateVariables() - if 'gated_average' in self.children: - self.gated_average.InstantiateVariables() - if 'pre_proj' in self.children: - for proj in self.pre_proj: - proj.InstantiateVariables() - super()._CreateChildrenVariables() - - def FProp(self, theta, inputs, query_vec=None): - """Combines the list of input tensors into a single tensor. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: A list of tensors of shape [..., hidden_dim] or [..., - [pre_proj_input_dims[i]]] if pre_proj_input_dims is specified. - query_vec: A tensor of shape [..., hidden_dim]. - - Returns: - A tensor of the same shape with input tensors. - - Raises: - ValueError: p.merger_op is not defined. - """ - p = self.params - n_sources = len(inputs) - - if p.pre_proj_input_dims and len(p.pre_proj_input_dims) != n_sources: - raise ValueError('pre_proj_input_dims must be specified for each input.') - - if n_sources == 1: - return inputs[0] - - # Pre-projection operation. - if p.pre_proj_input_dims: - for i in range(n_sources): - inputs[i] = self.pre_proj[i].FProp(theta.pre_proj[i], inputs[i]) - - tensor_pairs = list(zip(inputs[:-1], inputs[1:])) - if p.merger_op == 'mean': - # Simply take the mean, all dims must match. - with tf.control_dependencies([ - py_utils.assert_shape_match(tf.shape(t1), tf.shape(t2)) - for t1, t2 in tensor_pairs - ]): - output = tf.add_n(inputs) / n_sources - - elif p.merger_op == 'sum': - # Sum up all sources, all dims must match. - with tf.control_dependencies([ - py_utils.assert_shape_match(tf.shape(t1), tf.shape(t2)) - for t1, t2 in tensor_pairs - ]): - output = tf.add_n(inputs) - - elif p.merger_op == 'weighted_sum': - # Weighted sum of all sources, all dims must match. - # For weighted_sum, assume input is a list of rank 3 tensors - inputs = tf.stack(inputs) - inputs = py_utils.HasRank(inputs, 4) - - with tf.control_dependencies([ - py_utils.assert_shape_match(tf.shape(t1), tf.shape(t2)) - for t1, t2 in tensor_pairs - ]): - w = tf.expand_dims( - tf.expand_dims(tf.expand_dims(theta.sum_weight, 1), 1), 1) - w = tf.tile( - w, - [1, - tf.shape(inputs)[1], - tf.shape(inputs)[2], - tf.shape(inputs)[3]]) - output = tf.reduce_sum(inputs * w, axis=0) - - elif p.merger_op == 'atten': - # Apply attention over the concatenated tensor, all dims must match. - with tf.control_dependencies([ - py_utils.assert_shape_match(tf.shape(t1), tf.shape(t2)) - for t1, t2 in tensor_pairs - ]): - inputs = tf.stack(inputs, axis=0) - batch_size = tf.shape(inputs)[1] - paddings = tf.zeros([n_sources, batch_size], dtype=inputs.dtype) - self.atten.InitForSourcePacked(theta.atten, inputs, inputs, paddings) - output, _, _ = self.atten.ComputeContextVector( - theta.atten, tf.reshape(query_vec, [-1, p.query_dim])) - - elif p.merger_op == 'concat': - # Concatenate over the last dim, all dims but last must match. - with tf.control_dependencies([ - py_utils.assert_equal(tf.shape(t1)[:-1], - tf.shape(t2)[:-1]) for t1, t2 in tensor_pairs - ]): - output = tf.concat(inputs, axis=-1) - - elif p.merger_op == 'gated_avg': - output = self.gated_average.FProp(theta.gated_average, inputs) - - else: - raise ValueError('Unrecognized merge op!') - - if p.post_proj: - output = self.post_proj.FProp(theta.post_proj, output) - - return output - - -class MultiSourceAttention(BaseAttentionLayer): - """Attention with multiple source sub-attentions. - - It attends to multiple sources and uses one query as input to generates a - combined attention context. The dimension of the combined context vector is a - sum of all source context vectors. Each source attention has its separate - params and is associated with a source key. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('source_atten_tpls', None, - 'A list of (source_key, attention_param) ' - 'pairs.') - p.Define('source_dim', 0, 'Default source dimension.') - p.Define( - 'query_dim', 0, 'Number of query nodes. Child attention params ' - 'must have query_dim less or euqal than 0 or equal to this value.') - p.Define( - 'primary_source_key', 'source_0', 'Key for the primary source ' - 'whose attention probabilities will be used as an output.') - p.Define( - 'atten_merger_tpl', - MergerLayer.Params().Set( - params_init=py_utils.WeightInit.Uniform(0.04), merger_op='sum'), - 'Params to specify how to merge source attention vectors.') - return p - - def __init__(self, params): - """Constructs an MultiSourceAttention object.""" - super().__init__(params) - p = self.params - for source_key, atten_p in p.source_atten_tpls: - child_p = atten_p.Copy() - if child_p.query_dim <= 0: - child_p.query_dim = p.query_dim - else: - assert child_p.query_dim == p.query_dim - if child_p.source_dim <= 0: - child_p.source_dim = p.source_dim - self.CreateChild('atten_%s' % source_key, child_p) - - # Initialize source context vector merging layer. - merger_p = p.atten_merger_tpl.Copy() - merger_p.name = 'atten_merger' - merger_p.source_dim = p.source_dim - merger_p.query_dim = p.query_dim - self.CreateChild('atten_merger', merger_p) - - def PackSource(self, - theta, - source_vecs, - source_contexts, - source_padding, - source_segment_id=None): - p = self.params - with tf.name_scope(self.params.name): - packed_src = py_utils.NestedMap() - for source_key, _ in p.source_atten_tpls: - packed_src[source_key] = ( - self.children['atten_%s' % source_key].InitForSourcePacked( - theta['atten_%s' % source_key], source_vecs[source_key], - source_contexts[source_key], source_padding[source_key], - source_segment_id[source_key] if source_segment_id else None)) - return packed_src - - def ZeroAttentionState(self, source_seq_length, decoder_batch_size): - p = self.params - with tf.name_scope(self.params.name): - return py_utils.NestedMap({ - source_key: getattr(self, 'atten_%s' % source_key).ZeroAttentionState( - source_seq_length[source_key], decoder_batch_size) - for source_key, _ in p.source_atten_tpls - }) - - def ComputeContextVectorWithSource(self, - theta, - packed_src, - query_vec, - attention_state=None, - per_step_source_padding=None, - query_segment_id=None): - p = self.params - assert per_step_source_padding is None - with tf.name_scope(self.params.name): - result_map = py_utils.NestedMap() - for source_key, _ in p.source_atten_tpls: - result_map[source_key] = ( - self.children['atten_%s' % - source_key].ComputeContextVectorWithSource( - theta.get('atten_%s' % source_key), - packed_src[source_key], query_vec, - attention_state[source_key] - if attention_state else None, - per_step_source_padding, query_segment_id)) - return self._CombineContext(theta, result_map, query_vec) - - def _CombineContext(self, theta, context_map, query_vec): - ctxs = context_map.Flatten() - combined_context = ( - self.atten_merger.FProp(theta.atten_merger, [ctx for ctx, _, _ in ctxs], - query_vec)) - return ( - combined_context, - # Return atten_probs of the primary source. - # TODO(huk): Maybe return a NestedMap. - context_map[self.params.primary_source_key][1], - py_utils.NestedMap({ - src_key: context_map[src_key][2] - for src_key, _ in self.params.source_atten_tpls - })) diff --git a/lingvo/core/attention_test.py b/lingvo/core/attention_test.py deleted file mode 100644 index 40961894..00000000 --- a/lingvo/core/attention_test.py +++ /dev/null @@ -1,2377 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for attention.""" - -import math -from absl.testing import parameterized -import lingvo.compat as tf -from lingvo.core import attention -from lingvo.core import layers -from lingvo.core import py_utils -from lingvo.core import quant_utils -from lingvo.core import test_utils -import numpy as np - - -class AttentionTest(test_utils.TestCase, parameterized.TestCase): - """Test attention models.""" - - def _CheckStaticShapes(self, atten_vec, atten_prob, target_batch_size, - source_length, context_dim): - """Static shape must be set correctly for RNN beam search compatibility.""" - self.assertIsNotNone(atten_prob.shape.ndims) - self.assertEqual((target_batch_size, source_length), atten_prob.shape) - self.assertIsNotNone(atten_vec.shape.ndims) - self.assertEqual((target_batch_size, context_dim), atten_vec.shape) - - def _AdditiveAttentionInputs(self, packed_inputs=False, tgt_bs=6): - np.random.seed(12345) - source_vecs = tf.constant(np.random.rand(6, 3, 4), dtype=tf.float32) - source_contexts = tf.constant(np.random.rand(6, 3, 5), dtype=tf.float32) - source_padding = tf.transpose( - tf.constant( - [[0, 0, 1, 1, 0, 0], [1, 0, 0, 0, 1, 0], [0, 0, 1, 0, 1, 0]], - dtype=tf.float32)) - source_segment_id = tf.transpose( - tf.constant( - [[0, 0, 1, 1, 1, 1], [0, 0, 1, 1, 2, 2], [0, 1, 1, 1, 1, 2]], - dtype=tf.float32)) - query_vec = tf.constant(np.random.rand(tgt_bs, 7), dtype=tf.float32) - qsi = [0, 1, 1, 1, 2, 2] - query_segment_id = tf.constant(qsi[:tgt_bs], dtype=tf.float32) - - params = attention.AdditiveAttention.Params() - params.name = 'atten' - params.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - params.source_dim = 4 - params.query_dim = 7 - params.hidden_dim = 7 - params.vn.global_vn = False - params.vn.per_step_vn = False - params.packed_input = packed_inputs - tensors = (source_vecs, source_contexts, source_padding, source_segment_id, - query_vec, query_segment_id) - return params, tensors - - def testAdditiveAttention(self): - with self.session(use_gpu=True): - params, tensors = self._AdditiveAttentionInputs() - source_vecs, source_contexts, source_padding, _, query_vec, _ = tensors - atten = attention.AdditiveAttention(params) - self.assertLen(atten.vars.Flatten(), 3) - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec) - - self._CheckStaticShapes( - atten_vec, - atten_prob, - target_batch_size=query_vec.shape[0], - source_length=source_contexts.shape[0], - context_dim=source_contexts.shape[2]) - - # TODO(yonghui): add atten.vars for the variables attention model - # declares. - atten_vars = tf.get_collection('AdditiveAttention_vars') - self.assertLen(atten_vars, 3) - - self.evaluate(tf.global_variables_initializer()) - - all_vars = tf.trainable_variables() - for v in all_vars: - print(v.eval()) - - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - - print(['additive attention prob_out', np.array_repr(prob_out)]) - print(['additive attention atten_vec_out', np.array_repr(atten_vec_out)]) - - # pyformat: disable - # pylint: disable=bad-whitespace - expected_prob_out = [ - [0.2555742 , 0.24073002, 0. , 0. , 0.25412574, - 0.24957004], - [0. , 0.25394136, 0.24764746, 0.25480017, 0. , - 0.24361098], - [0.25094295, 0.2499937 , 0. , 0.24308342, 0. , - 0.25597993], - [0.25559244, 0.24070661, 0. , 0. , 0.25412717, - 0.24957375], - [0. , 0.25393167, 0.24765188, 0.25481117, 0. , - 0.24360526], - [0.25113183, 0.24990553, 0. , 0.24246082, 0. , - 0.25650182]] - - expected_atten_vec_out = [ - [0.49745506, 0.63471669, 0.49220526, 0.5683012 , 0.42753702], - [0.51502365, 0.56183743, 0.37644109, 0.87425125, 0.46182787], - [0.57862502, 0.44246522, 0.36931852, 0.41002905, 0.14327194], - [0.49745634, 0.63471717, 0.49220967, 0.56829125, 0.4275257 ], - [0.51501834, 0.56183696, 0.37644821, 0.87425053, 0.46182543], - [0.57893348, 0.44248882, 0.36938411, 0.41006744, 0.14328158]] - # pylint: enable=bad-whitespace - # pyformat: enable - - self.assertAllClose(expected_prob_out, prob_out) - self.assertAllClose(expected_atten_vec_out, atten_vec_out) - - def testAdditiveAttentionWithPackedInputs(self): - with self.session(use_gpu=True): - params, tensors = self._AdditiveAttentionInputs(packed_inputs=True) - (source_vecs, source_contexts, source_padding, source_segment_id, - query_vec, query_segment_id) = tensors - atten = attention.AdditiveAttention(params) - self.assertLen(atten.vars.Flatten(), 3) - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding, source_segment_id) - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec, query_segment_id=query_segment_id) - - # TODO(yonghui): add atten.vars for the variables attention model - # declares. - atten_vars = tf.get_collection('AdditiveAttention_vars') - self.assertLen(atten_vars, 3) - - self.evaluate(tf.global_variables_initializer()) - - all_vars = tf.trainable_variables() - for v in all_vars: - print(v.eval()) - - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - - self._CheckStaticShapes( - atten_vec, - atten_prob, - target_batch_size=query_vec.shape[0], - source_length=source_contexts.shape[0], - context_dim=source_contexts.shape[2]) - - print(['packed additive attention prob_out', np.array_repr(prob_out)]) - print([ - 'packed additive attention atten_vec_out', - np.array_repr(atten_vec_out) - ]) - - # pyformat: disable - # pylint: disable=bad-whitespace - expected_prob_out = [ - [0.51495469, 0.48504525, 0 , 0 , 0 , - 0 ], - [0 , 0 , 0.49288213, 0.50711787, 0 , - 0 ], - [0. , 0.5070073 , 0. , 0.4929927 , 0. , - 0 ], - [0. , 0 , 0. , 0. , 0.50451994, - 0.49548006], - [0. , 0. , 0. , 0. , 0. , - 1 ], - [0. , 0. , 0. , 0. , 0. , - 1 ]] - - expected_atten_vec_out = [ - [0.35256192, 0.68348885, 0.41128731, 0.48906463, 0.50537711], - [0.45880911, 0.6068666 , 0.59867024, 0.82797134, 0.33504993], - [0.54934788, 0.50335771, 0.26117462, 0.32834488, 0.16398546], - [0.64022166, 0.58665955, 0.571935 , 0.64637613, 0.35084069], - [0.27927336, 0.06444023, 0.19862361, 0.93168277, 0.85441357], - [0.95473474, 0.05225335, 0.57947171, 0.48049626, 0.02170898]] - # pylint: enable=bad-whitespace - # pyformat: enable - - self.assertAllClose(expected_prob_out, prob_out) - self.assertAllClose(expected_atten_vec_out, atten_vec_out) - - def testAdditiveAttentionDeterministicDropout(self): - with self.session(use_gpu=True): - params, tensors = self._AdditiveAttentionInputs() - source_vecs, source_contexts, source_padding, _, query_vec, _ = tensors - params.atten_dropout_prob = 0.5 - params.atten_dropout_deterministic = True - params.random_seed = 78924 - - atten = attention.AdditiveAttention(params) - self.assertLen(atten.vars.Flatten(), 3) - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec) - - self._CheckStaticShapes( - atten_vec, - atten_prob, - target_batch_size=query_vec.shape[0], - source_length=source_contexts.shape[0], - context_dim=source_contexts.shape[2]) - - atten_vars = tf.get_collection('AdditiveAttention_vars') - self.assertLen(atten_vars, 3) - - self.evaluate(tf.global_variables_initializer()) - - all_vars = tf.trainable_variables() - for v in all_vars: - print(v.eval()) - - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - - print('additive attention prob_out %r' % prob_out) - print('additive attention atten_vec_out %r' % atten_vec_out) - - expected_prob_out = [ - [0.51114839, 0.48146003, 0., 0., 0., 0.], - [0., 0.50788271, 0., 0.50960034, 0., 0.48722193], - [0., 0.49998739, 0., 0., 0., 0.51195991], - [0., 0.48141322, 0., 0., 0.50825435, 0.4991475], - [0., 0.50786334, 0.49530372, 0., 0., 0.48721054], - [0., 0.49981108, 0., 0., 0., 0.51300365], - ] - - expected_atten_vec_out = [ - [0.34995595, 0.67843682, 0.40824726, 0.4854497, 0.50164163], - [0.60576487, 0.80303985, 0.46480939, 1.3962903, 0.79863495], - [0.90196574, 0.47579059, 0.31802341, 0.34388986, 0.15836108], - [0.81517166, 0.90433061, 0.72681838, 1.02123988, 0.72982419], - [0.99326241, 0.83445895, 0.43935478, 1.26866817, 0.71197236], - [0.90281653, 0.47568679, 0.31862068, 0.34435683, 0.15833181], - ] - - self.assertAllClose(expected_prob_out, prob_out) - self.assertAllClose(expected_atten_vec_out, atten_vec_out) - - def _testSameBatchSize(self, same_batch_size, packed_inputs=False): - with self.session(use_gpu=True, graph=tf.Graph()): - tf.random.set_seed(398847392) - params, tensors = self._AdditiveAttentionInputs(packed_inputs, tgt_bs=3) - source_vecs, source_contexts, source_padding, _, query_vec, _ = tensors - params.same_batch_size = same_batch_size - - atten = attention.AdditiveAttention(params) - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec) - - self._CheckStaticShapes( - atten_vec, - atten_prob, - target_batch_size=query_vec.shape[0], - source_length=source_contexts.shape[0], - context_dim=source_contexts.shape[2]) - - self.assertLen(atten.vars.Flatten(), 3) - - self.evaluate(tf.global_variables_initializer()) - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - return atten_vec_out, prob_out - - def testAdditiveAttentionSameBatchSize(self): - vec0, prob0 = self._testSameBatchSize(False) - vec1, prob1 = self._testSameBatchSize(True) - self.assertAllClose(vec0, vec1) - self.assertAllClose(prob0, prob1) - - def testAdditiveAttentionSameBatchSizePackedInputs(self): - vec0, prob0 = self._testSameBatchSize(False, True) - vec1, prob1 = self._testSameBatchSize(True, True) - self.assertAllClose(vec0, vec1) - self.assertAllClose(prob0, prob1) - - def testAdditiveAttentionSmallerHiddenLayer(self): - with self.session(use_gpu=True): - np.random.seed(12345) - source_vecs = tf.constant(np.random.rand(6, 3, 4), dtype=tf.float32) - source_contexts = tf.constant(np.random.rand(6, 3, 5), dtype=tf.float32) - source_padding = tf.transpose( - tf.constant( - [[0, 0, 1, 1, 0, 0], [1, 0, 0, 0, 1, 0], [0, 0, 1, 0, 1, 0]], - dtype=tf.float32)) - query_vec = tf.constant(np.random.rand(6, 7), dtype=tf.float32) - - params = attention.AdditiveAttention.Params() - params.name = 'atten' - params.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - params.source_dim = 4 - params.query_dim = 7 - params.hidden_dim = 5 - params.vn.global_vn = False - params.vn.per_step_vn = False - - atten = attention.AdditiveAttention(params) - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec) - - self._CheckStaticShapes( - atten_vec, - atten_prob, - target_batch_size=query_vec.shape[0], - source_length=source_contexts.shape[0], - context_dim=source_contexts.shape[2]) - - atten_vars = tf.get_collection('AdditiveAttention_vars') - self.assertLen(atten_vars, 3) - - self.evaluate(tf.global_variables_initializer()) - - all_vars = tf.trainable_variables() - for v in all_vars: - print(v.eval()) - - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - - print(['prob_out smaller hidden layer', np.array_repr(prob_out)]) - print( - ['atten_vec_out smaller hidden layer', - np.array_repr(atten_vec_out)]) - - # pyformat: disable - # pylint: disable=bad-whitespace - expected_prob_out = [ - [0.25242305, 0.24356601, 0. , 0. , 0.25346902, - 0.25054196], - [0. , 0.25230604, 0.24693871, 0.25406054, 0. , - 0.24669473], - [0.2501823 , 0.24922216, 0. , 0.24693316, 0. , - 0.25366238], - [0.25267059, 0.24300526, 0. , 0. , 0.25369659, - 0.25062758], - [0. , 0.25272119, 0.24642748, 0.25435579, 0. , - 0.24649554], - [0.25044653, 0.24924593, 0. , 0.24560687, 0. , - 0.25470066]] - - expected_atten_vec_out = [ - [0.49746257, 0.63428223, 0.4914251 , 0.57035601, 0.42964566], - [0.51383036, 0.55960417, 0.37601081, 0.87443453, 0.46342701], - [0.57660079, 0.44147781, 0.36953348, 0.41017395, 0.14293665], - [0.49755943, 0.63429612, 0.49157569, 0.57015073, 0.42933062], - [0.51371205, 0.55982226, 0.37590009, 0.87454152, 0.4633899 ], - [0.57732767, 0.44161472, 0.36958888, 0.41019297, 0.14298658]] - # pylint: enable=bad-whitespace - # pyformat: enable - - self.assertAllClose(expected_prob_out, prob_out) - self.assertAllClose(expected_atten_vec_out, atten_vec_out) - - def testAdditiveAttentionFp16NoNaN(self): - with self.session(use_gpu=True): - np.random.seed(12345) - source_vecs = tf.constant(np.random.rand(6, 3, 4), dtype=tf.float16) - source_contexts = tf.constant(np.random.rand(6, 3, 5), dtype=tf.float16) - source_padding = tf.transpose( - tf.constant( - [[0, 0, 1, 1, 0, 0], [1, 0, 0, 0, 1, 0], [0, 0, 1, 0, 1, 0]], - dtype=tf.float16)) - query_vec = tf.constant(np.random.rand(6, 7), dtype=tf.float16) - - params = attention.AdditiveAttention.Params() - params.dtype = tf.float16 - params.name = 'atten' - params.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - params.source_dim = 4 - params.query_dim = 7 - params.hidden_dim = 7 - params.vn.global_vn = False - params.vn.per_step_vn = False - - atten = attention.AdditiveAttention(params) - self.assertLen(atten.vars.Flatten(), 3) - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec) - - self._CheckStaticShapes( - atten_vec, - atten_prob, - target_batch_size=query_vec.shape[0], - source_length=source_contexts.shape[0], - context_dim=source_contexts.shape[2]) - - # TODO(yonghui): add atten.vars for the variables attention model - # declares. - atten_vars = tf.get_collection('AdditiveAttention_vars') - self.assertLen(atten_vars, 3) - - self.evaluate(tf.global_variables_initializer()) - - all_vars = tf.trainable_variables() - for v in all_vars: - print(v.eval()) - - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - print(atten_vec_out.dtype) - print(prob_out.dtype) - self.assertTrue(np.all(np.isfinite(atten_vec_out))) - self.assertTrue(np.all(np.isfinite(prob_out))) - - def testAdditiveAttentionVN64bits(self): - with self.session(use_gpu=True): - np.random.seed(12345) - source_vecs = tf.constant(np.random.rand(5, 3, 4), dtype=tf.float64) - source_contexts = tf.constant(np.random.rand(5, 3, 5), dtype=tf.float64) - source_padding = tf.transpose( - tf.constant([[0, 0, 1, 1, 0], [1, 0, 0, 0, 1], [0, 0, 1, 0, 1]], - dtype=tf.float64)) - query_vec = tf.constant(np.random.rand(6, 7), dtype=tf.float64) - - params = attention.AdditiveAttention.Params() - params.name = 'atten' - params.dtype = tf.float64 - params.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - params.source_dim = 4 - params.query_dim = 7 - params.hidden_dim = 5 - params.vn.global_vn = True - params.vn.per_step_vn = True - params.vn.scale = 1.0 - params.vn.seed = 54321 - - atten = attention.AdditiveAttention(params) - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec) - - self._CheckStaticShapes( - atten_vec, - atten_prob, - target_batch_size=query_vec.shape[0], - source_length=source_contexts.shape[0], - context_dim=source_contexts.shape[2]) - - atten_vars = tf.get_collection('AdditiveAttention_vars') - self.assertLen(atten_vars, 3) - - self.evaluate(tf.global_variables_initializer()) - - all_vars = tf.trainable_variables() - for v in all_vars: - print(v.eval()) - - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - - print(['prob_out with vn:', np.array_repr(prob_out)]) - print(['atten_vec_out with vn:', np.array_repr(atten_vec_out)]) - - # pyformat: disable - # pylint: disable=bad-whitespace - expected_prob_out =[ - [ 0.43249266, 0.18638571, 0. , 0. , 0.38112162], - [ 0. , 0.32589137, 0.3505654 , 0.32354323, 0. ], - [ 0.26777833, 0.43991441, 0. , 0.29230726, 0. ], - [ 0.34583678, 0.32633085, 0. , 0. , 0.32783237], - [ 0. , 0.32734872, 0.34749836, 0.32515292, 0. ], - [ 0.33614176, 0.33607175, 0. , 0.32778649, 0. ] - ] - expected_atten_vec_out = [ - [ 0.56117282, 0.37872234, 0.42109472, 0.38981267, 0.45946841], - [ 0.85743407, 0.37325286, 0.66322611, 0.69286686, 0.141359 ], - [ 0.7377786 , 0.42298519, 0.39970782, 0.67703222, 0.4157012 ], - [ 0.51011499, 0.35817489, 0.47894328, 0.41259201, 0.54384056], - [ 0.85716326, 0.37340558, 0.66250852, 0.69187486, 0.14179651], - [ 0.78078121, 0.45646575, 0.4052385 , 0.68248276, 0.43502425] - ] - # pylint: enable=bad-whitespace - # pyformat: enable - - self.assertAllClose(expected_prob_out, prob_out) - self.assertAllClose(expected_atten_vec_out, atten_vec_out) - - def _DotProductAttention(self, packed_inputs): - with self.session(use_gpu=True): - np.random.seed(12345) - # source_vecs_p, source_contexts_p, source_padding_p, query_vec_p are used - # for both TensorFlow and numpy computation. - source_vecs_p = [np.random.rand(3, 4) for _ in range(6)] - source_vecs = tf.stack( - [tf.constant(x, dtype=tf.float32) for x in source_vecs_p]) - source_contexts_p = [np.random.rand(3, 5) for _ in range(6)] - source_contexts = tf.stack( - [tf.constant(x, dtype=tf.float32) for x in source_contexts_p]) - source_padding_p = [[0, 0, 1, 1, 0, 0], [1, 0, 0, 0, 1, 0], - [0, 0, 1, 0, 1, 0]] - source_padding = tf.transpose( - tf.constant(source_padding_p, dtype=tf.float32)) - query_vec_p = np.random.rand(6, 4) - query_vec = tf.constant(query_vec_p, dtype=tf.float32) - query_segment_id_p = [0, 1, 1, 1, 2, 2] - source_segment_id_p = [[0, 0, 1, 1, 1, 1], [0, 0, 1, 1, 2, 2], - [0, 1, 1, 1, 1, 2]] - source_segment_id = None - query_segment_id = None - if packed_inputs: - source_segment_id = tf.transpose( - tf.constant(source_segment_id_p, dtype=tf.float32)) - query_segment_id = tf.constant(query_segment_id_p, dtype=tf.float32) - params = attention.DotProductAttention.Params() - params.name = 'dotproduct_atten' - params.source_dim = 4 - params.query_dim = 4 - params.hidden_dim = 4 - params.packed_input = packed_inputs - atten = attention.DotProductAttention(params) - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding, source_segment_id) - self.evaluate(tf.global_variables_initializer()) - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec, query_segment_id=query_segment_id) - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - # Use numpy to perform the same computation to generate expected results. - source_vecs_p = np.array(source_vecs_p) - # Dot-product part. - expected_logit = np.array([ - np.dot(source_vecs_p[:, i % 3, :], query_vec_p[i, :]) - for i in range(6) - ]) / math.sqrt(4) - elexp = np.exp(expected_logit) - source_padding_p = np.array(source_padding_p) - elexp *= (1 - np.tile(source_padding_p, (2, 1))) - if packed_inputs: - # Manually constructed packed input mask. - mask = np.asarray([[1, 1, 0, 0, 0, 0], [0, 0, 1, 1, 0, 0], - [0, 1, 1, 1, 1, 0], [0, 0, 1, 1, 1, 1], - [0, 0, 0, 0, 1, 1], [0, 0, 0, 0, 0, 1]]) - elexp *= mask - expected_prob_out = elexp / np.expand_dims(np.sum(elexp, axis=1), axis=1) - expanded_epo = np.expand_dims(expected_prob_out, axis=2) - source_contexts_p = np.array(source_contexts_p) - expected_atten_vec_out = np.array([ - np.sum( - source_contexts_p[:, i % 3, :] * expanded_epo[i, :, :], axis=0) - for i in range(6) - ]) - - print(['additive attention prob_out', np.array_repr(prob_out)]) - print(['additive attention atten_vec_out', np.array_repr(atten_vec_out)]) - - self.assertAllClose(expected_prob_out, prob_out) - self.assertAllClose(expected_atten_vec_out, atten_vec_out) - - def testDotProductAttention(self): - self._DotProductAttention(packed_inputs=False) - - def testDotProductAttentionPackedInput(self): - self._DotProductAttention(packed_inputs=True) - - def _MultiHeadedAttentionInputs(self, source_dim=4, dtype=tf.float32): - np.random.seed(6348575) - # source_vecs_p, source_contexts_p, source_padding_p, query_vec_p are used - # for both TensorFlow and numpy computation. - source_vecs_p = [np.random.rand(3, source_dim) for _ in range(6)] - source_vecs = tf.stack([tf.constant(x, dtype=dtype) for x in source_vecs_p]) - source_contexts_p = [np.random.rand(3, 6) for _ in range(6)] - source_contexts = tf.stack( - [tf.constant(x, dtype=dtype) for x in source_contexts_p]) - source_padding_p = [[0, 0, 1, 1, 0, 0], [1, 0, 0, 0, 1, 0], - [0, 0, 1, 0, 1, 0]] - source_padding = tf.transpose(tf.constant(source_padding_p, dtype=dtype)) - query_vec_p = np.random.rand(6, 4) - query_vec = tf.constant(query_vec_p, dtype=dtype) - query_segment_id_p = [0, 1, 1, 1, 2, 2] - source_segment_id_p = [[0, 0, 1, 1, 1, 1], [0, 0, 1, 1, 2, 2], - [0, 1, 1, 1, 1, 2]] - source_segment_id = tf.transpose( - tf.constant(source_segment_id_p, dtype=dtype)) - query_segment_id = tf.constant(query_segment_id_p, dtype=dtype) - return (source_vecs, source_contexts, source_padding, source_padding_p, - query_vec, source_segment_id, query_segment_id) - - def testMultiHeadedAttentionDotProductWithFeedinProbs(self): - with self.session(use_gpu=True): - (source_vecs, source_contexts, source_padding, _, _, _, - _) = self._MultiHeadedAttentionInputs() - iap = attention.DotProductAttention.Params() - iap.name = 'dot_atten' - params = attention.MultiHeadedAttention.Params().Set( - name='multihead_atten', - source_dim=4, - query_dim=4, - hidden_dim=4, - inner_atten_params=iap, - num_attention_heads=2, - use_source_vec_as_attention_value=False) - atten = params.Instantiate() - packed_src = atten.InitForSourcePacked(atten.theta, source_vecs, - source_contexts, source_padding) - self.evaluate(tf.global_variables_initializer()) - atten_probs = tf.constant([[1.0] + [0.0] * 5] * 3 * 2, dtype=tf.float32) - atten_vec_proj, atten_vec = atten.ComputeContextVectorWithAttenProbs( - atten.theta, packed_src.source_contexts, atten_probs) - atten_vec_proj, atten_vec, packed_context = self.evaluate( - [atten_vec_proj, atten_vec, packed_src.source_contexts]) - self.assertAllClose( - atten_vec, - np.reshape(np.transpose(packed_context, (0, 2, 1)), [3, 6, 6])[:, :, - 0]) - self.assertAllClose([2.5694468, 4.36386967, 3.24537992], - np.sum(atten_vec_proj, axis=1)) - - def _testMultiHeadedAttentionExtendCachedSourceVecsHelper( - self, additive_atten, dtype, fprop_dtype): - # source_batch:3, target_batch:6. Test n = 2 case. - use_gpu = (dtype == tf.float32 and fprop_dtype == tf.float32) - with self.session(use_gpu=use_gpu): - (source_vecs, source_contexts, source_padding, _, query_vec, - source_seg_id, - query_seg_id) = self._MultiHeadedAttentionInputs(dtype=fprop_dtype) - if additive_atten: - iap = attention.AdditiveAttention.Params() - iap.name = 'add_atten' - else: - iap = attention.DotProductAttention.Params() - iap.name = 'dot_atten' - params = attention.MultiHeadedAttention.Params().Set( - name='multihead_atten', - dtype=dtype, - fprop_dtype=fprop_dtype, - source_dim=4, - query_dim=4, - hidden_dim=4, - inner_atten_params=iap, - num_attention_heads=2, - use_source_vec_as_attention_value=False, - packed_input=True) - atten = params.Instantiate() - theta = atten.theta - packed_src1 = atten.InitForSourcePacked(theta, source_vecs, - source_contexts, source_padding, - source_seg_id) - cached_src = py_utils.NestedMap( - source_vecs=tf.zeros([0, 3, 4], dtype=packed_src1.source_vecs.dtype), - source_contexts=tf.zeros([0, 3, 6], - dtype=packed_src1.source_contexts.dtype), - source_padding=tf.zeros([0, 3, 2], - dtype=packed_src1.source_padding.dtype), - source_segment_id=tf.zeros([0, 3, 2], - dtype=packed_src1.source_segment_id.dtype)) - for i in range(6): - cached_src = atten.ExtendSourcePacked(theta, source_vecs[i, :, :], - source_contexts[i, :, :], - source_padding[i, :], - source_seg_id[i, :], cached_src) - packed_src2 = atten.PackCachedSource(cached_src) - self.evaluate(tf.global_variables_initializer()) - - atten_vec_1, prob_1, _ = atten.ComputeContextVectorWithSource( - theta, packed_src1, query_vec, query_segment_id=query_seg_id) - atten_vec_2, prob_2, _ = atten.ComputeContextVectorWithCachedSource( - theta, cached_src, query_vec, query_segment_id=query_seg_id) - - packed_src1_v, packed_src2_v, cached_src_v = self.evaluate( - [packed_src1, packed_src2, cached_src]) - tf.logging.info('packed_src1=%s', packed_src1_v) - tf.logging.info('packed_src2=%s', packed_src2_v) - tf.logging.info('cached_src=%s', cached_src_v) - self.assertAllClose(packed_src1_v.source_vecs, packed_src2_v.source_vecs) - self.assertAllClose(packed_src1_v.source_contexts, - packed_src2_v.source_contexts) - self.assertAllClose(packed_src1_v.source_padding, - packed_src2_v.source_padding) - self.assertAllClose(packed_src1_v.source_segment_id, - packed_src2_v.source_segment_id) - atten_vec1_v, prob1_v, atten_vec2_v, prob2_v = self.evaluate( - [atten_vec_1, prob_1, atten_vec_2, prob_2]) - self.assertAllClose(prob1_v, prob2_v) - self.assertAllClose(atten_vec1_v, atten_vec2_v) - - def testMultiHeadedAttentionExtendCachedSourceVecsAdditiveFloat32(self): - self._testMultiHeadedAttentionExtendCachedSourceVecsHelper( - additive_atten=True, dtype=tf.float32, fprop_dtype=tf.float32) - - def testMultiHeadedAttentionExtendCachedSourceVecsAdditiveFloat32Float16( - self): - self._testMultiHeadedAttentionExtendCachedSourceVecsHelper( - additive_atten=True, dtype=tf.float32, fprop_dtype=tf.float16) - - def testMultiHeadedAttentionExtendCachedSourceVecsDotFloat32(self): - self._testMultiHeadedAttentionExtendCachedSourceVecsHelper( - additive_atten=False, dtype=tf.float32, fprop_dtype=tf.float32) - - def testMultiHeadedAttentionExtendCachedSourceVecsDotFloat32Float16(self): - self._testMultiHeadedAttentionExtendCachedSourceVecsHelper( - additive_atten=False, dtype=tf.float32, fprop_dtype=tf.float16) - - def _testMultiHeadedAttentionExtendCachedSourceVecsNoPaddingsHelper( - self, additive_attention=False): - # source_batch:3, target_batch:6. Test n = 2 case. - with self.session(use_gpu=True): - (source_vecs, source_contexts, _, _, query_vec, _, - _) = self._MultiHeadedAttentionInputs() - source_padding = tf.zeros([6, 3]) - if additive_attention: - iap = attention.AdditiveAttention.Params() - iap.name = 'add_atten' - else: - iap = attention.DotProductAttention.Params() - iap.name = 'dot_atten' - params = attention.MultiHeadedAttention.Params().Set( - name='multihead_atten', - source_dim=4, - query_dim=4, - hidden_dim=4, - inner_atten_params=iap, - num_attention_heads=2, - use_source_vec_as_attention_value=False) - atten = params.Instantiate() - packed_src1 = atten.InitForSourcePacked(atten.theta, source_vecs, - source_contexts, source_padding) - cached_src = py_utils.NestedMap( - source_vecs=tf.zeros([0, 3, 4], dtype=packed_src1.source_vecs.dtype), - source_contexts=tf.zeros([0, 3, 6], - dtype=packed_src1.source_contexts.dtype), - source_padding=None, - source_seg_id=None) - for i in range(6): - cached_src = atten.ExtendSourcePacked(atten.theta, source_vecs[i, :, :], - source_contexts[i, :, :], None, - None, cached_src) - packed_src2 = atten.PackCachedSource(cached_src) - self.evaluate(tf.global_variables_initializer()) - - atten_vec_1, prob_1, _ = atten.ComputeContextVectorWithSource( - atten.theta, packed_src1, query_vec) - atten_vec_2, prob_2, _ = atten.ComputeContextVectorWithCachedSource( - atten.theta, cached_src, query_vec) - - (source_vec1_v, source_context1_v, source_vec2_v, source_context2_v, - atten_vec1_v, prob1_v, atten_vec2_v, prob2_v) = self.evaluate([ - packed_src1.source_vecs, packed_src1.source_contexts, - packed_src2.source_vecs, packed_src2.source_contexts, atten_vec_1, - prob_1, atten_vec_2, prob_2 - ]) - self.assertAllClose(source_vec1_v, source_vec2_v) - self.assertAllClose(source_context1_v, source_context2_v) - self.assertAllClose(atten_vec1_v, atten_vec2_v) - self.assertAllClose(prob1_v, prob2_v) - - def testMultiHeadedDotAttentionExtendCachedSourceVecsNoPaddings(self): - self._testMultiHeadedAttentionExtendCachedSourceVecsNoPaddingsHelper(False) - - def testMultiHeadedAddAttentionExtendCachedSourceVecsNoPaddings(self): - self._testMultiHeadedAttentionExtendCachedSourceVecsNoPaddingsHelper(True) - - def testMultiHeadedAttentionDotProduct(self): - # source_batch:3, target_batch:6. Test n = 2 case. - with self.session(use_gpu=True): - (source_vecs, source_contexts, source_padding, source_padding_p, - query_vec, _, _) = self._MultiHeadedAttentionInputs() - iap = attention.DotProductAttention.Params() - iap.name = 'dot_atten' - params = attention.MultiHeadedAttention.Params().Set( - name='multihead_atten', - source_dim=4, - query_dim=4, - hidden_dim=4, - inner_atten_params=iap, - num_attention_heads=2, - use_source_vec_as_attention_value=False) - atten = params.Instantiate() - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - self.evaluate(tf.global_variables_initializer()) - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec) - - self._CheckStaticShapes( - atten_vec, - atten_prob, - target_batch_size=query_vec.shape[0], - source_length=source_contexts.shape[0], - context_dim=source_contexts.shape[2]) - - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - print('atten_vec_out', np.sum(atten_vec_out, axis=1)) - self.assertAllClose([ - 2.84679317, 2.36924601, 3.54831171, 2.86487937, 2.3537426, 3.54308939 - ], np.sum(atten_vec_out, axis=1)) - print('atten_vec_out', atten_vec_out) - print('prob_out', prob_out) - t_batch_size = 6 - s_batch_size = 3 - for i in range(t_batch_size): - # Test to make sure we didn't mess up indexing. - s_index = i % s_batch_size - atten.InitForSourcePacked(atten.theta, - source_vecs[:, s_index:s_index + 1, :], - source_contexts[:, s_index:s_index + 1, :], - source_padding[:, s_index:s_index + 1]) - atten_vec_i, prob_i, _ = atten.ComputeContextVector( - atten.theta, query_vec[i:i + 1]) - atten_vec_i_out, prob_i_out = self.evaluate([atten_vec_i, prob_i]) - self.assertAllClose(prob_i_out, prob_out[i:i + 1]) - self.assertAllClose(atten_vec_i_out, atten_vec_out[i:i + 1]) - padding_i = source_padding_p[s_index] - # Check to make sure prob exists only on valid timesteps. - self.assertEqual(0.0, np.sum(padding_i * prob_i_out)) - - def testMultiHeadedAttentionDotProductNoPerDimScaleNoBias(self): - # source_batch:3, target_batch:6. Test n = 2 case. - with self.session(use_gpu=True): - (source_vecs, source_contexts, source_padding, source_padding_p, - query_vec, _, _) = self._MultiHeadedAttentionInputs() - iap = attention.DotProductAttention.Params() - iap.name = 'dot_atten' - params = attention.MultiHeadedAttention.Params().Set( - name='multihead_atten', - source_dim=4, - query_dim=4, - hidden_dim=4, - inner_atten_params=iap, - num_attention_heads=2, - use_source_vec_as_attention_value=False, - enable_per_dim_scale=False, - use_bias=False) - atten = params.Instantiate() - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - self.evaluate(tf.global_variables_initializer()) - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec) - - self._CheckStaticShapes( - atten_vec, - atten_prob, - target_batch_size=query_vec.shape[0], - source_length=source_contexts.shape[0], - context_dim=source_contexts.shape[2]) - - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - print('atten_vec_out', np.sum(atten_vec_out, axis=1)) - self.assertAllClose([ - 2.84679317, 2.36924601, 3.54831171, 2.86487937, 2.3537426, 3.54308939 - ], np.sum(atten_vec_out, axis=1)) - print('atten_vec_out', atten_vec_out) - print('prob_out', prob_out) - t_batch_size = 6 - s_batch_size = 3 - for i in range(t_batch_size): - # Test to make sure we didn't mess up indexing. - s_index = i % s_batch_size - atten.InitForSourcePacked(atten.theta, - source_vecs[:, s_index:s_index + 1, :], - source_contexts[:, s_index:s_index + 1, :], - source_padding[:, s_index:s_index + 1]) - atten_vec_i, prob_i, _ = atten.ComputeContextVector( - atten.theta, query_vec[i:i + 1]) - atten_vec_i_out, prob_i_out = self.evaluate([atten_vec_i, prob_i]) - self.assertAllClose(prob_i_out, prob_out[i:i + 1]) - self.assertAllClose(atten_vec_i_out, atten_vec_out[i:i + 1]) - padding_i = source_padding_p[s_index] - # Check to make sure prob exists only on valid timesteps. - self.assertEqual(0.0, np.sum(padding_i * prob_i_out)) - - def testMultiHeadedAttentionDotProductPackedInput(self): - # source_batch:3, target_batch:6. Test n = 2 case. - with self.session(use_gpu=True): - (source_vecs, source_contexts, source_padding, source_padding_p, - query_vec, source_seg_id, - query_seg_id) = self._MultiHeadedAttentionInputs() - iap = attention.DotProductAttention.Params() - iap.name = 'dot_atten' - params = attention.MultiHeadedAttention.Params().Set( - name='multihead_atten', - source_dim=4, - query_dim=4, - hidden_dim=4, - inner_atten_params=iap, - num_attention_heads=2, - use_source_vec_as_attention_value=False, - packed_input=True) - atten = params.Instantiate() - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding, source_seg_id) - self.evaluate(tf.global_variables_initializer()) - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec, query_segment_id=query_seg_id) - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - print('atten_vec_out', np.sum(atten_vec_out, axis=1)) - self.assertAllClose( - [2.565648, 2.268182, 3.739031, 3.093884, 2.770367, 3.580353], - np.sum(atten_vec_out, axis=1)) - print('atten_vec_out', atten_vec_out) - print('prob_out', prob_out) - t_batch_size = 6 - s_batch_size = 3 - for i in range(t_batch_size): - # Test to make sure we didn't mess up indexing. - s_index = i % s_batch_size - src_seg_id = source_seg_id[:, s_index:s_index + 1] - atten.InitForSourcePacked(atten.theta, - source_vecs[:, s_index:s_index + 1, :], - source_contexts[:, s_index:s_index + 1, :], - source_padding[:, s_index:s_index + 1], - src_seg_id) - qry_seg_id = query_seg_id[i:i + 1] - atten_vec_i, prob_i, _ = atten.ComputeContextVector( - atten.theta, query_vec[i:i + 1], query_segment_id=qry_seg_id) - atten_vec_i_out, prob_i_out = self.evaluate([atten_vec_i, prob_i]) - self.assertAllClose(prob_i_out, prob_out[i:i + 1]) - self.assertAllClose(atten_vec_i_out, atten_vec_out[i:i + 1]) - padding_i = source_padding_p[s_index] - # Check to make sure prob exists only on valid timesteps. - self.assertEqual(0.0, np.sum(padding_i * prob_i_out)) - - def testMultiHeadedAttentionDotProductDeterministicDropout(self): - # source_batch:3, target_batch:6. Test n = 2 case. - with self.session(use_gpu=True): - source_vecs, source_contexts, source_padding, _, query_vec, _, _ = ( - self._MultiHeadedAttentionInputs()) - iap = attention.DotProductAttention.Params() - iap.name = 'dot_atten' - params = attention.MultiHeadedAttention.Params().Set( - name='multihead_atten', - source_dim=4, - query_dim=4, - hidden_dim=4, - inner_atten_params=iap, - num_attention_heads=2, - atten_dropout_prob=0.5, - atten_dropout_deterministic=True, - random_seed=7249528, - use_source_vec_as_attention_value=False) - atten = params.Instantiate() - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - atten_state = atten.ZeroAttentionState(2, 6) - print('atten_state:', atten_state) - - atten_vec, atten_prob, atten_state = atten.ComputeContextVector( - atten.theta, query_vec, atten_state) - - self.evaluate(tf.global_variables_initializer()) - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - - print('atten_vec_out', np.sum(atten_vec_out, axis=1)) - self.assertAllClose( - [4.152538, 3.423376, 4.782746, 4.238819, 1.785596, 3.239613], - np.sum(atten_vec_out, axis=1)) - - print('atten_vec_out %r' % atten_vec_out) - print('prob_out %r' % prob_out) - - expected_prob_out = [ - [0.235899, 0.26425028, 0., 0., 0.2534047, 0.24644604], - [0., 0.28664297, 0.2365877, 0.24659133, 0., 0.23017803], - [0.2340432, 0.26618454, 0., 0.257336, 0., 0.24243627], - [0.25304407, 0.25804248, 0., 0., 0.24191463, 0.24699883], - [0., 0.24431466, 0.25021935, 0.24959373, 0., 0.2558723], - [0.2792741, 0.21746796, 0., 0.25468093, 0., 0.24857706] - ] - expected_atten_vec_out = [ - [ - 0.87741864, 0.73626477, 0.90619636, 0.64786565, 0.6392238, - 0.3455683 - ], - [0.39735186, 0.6100546, 1.1262057, 0.43140602, 0.29049844, 0.5678591], - [ - 0.90514874, 1.1781378, 0.82465374, 0.64402366, 0.8315829, - 0.39919922 - ], - [ - 0.7801036, 0.92708683, 0.8839339, 0.70208144, 0.56675506, - 0.37885764 - ], - [0.1725478, 0.02231066, 0.06681296, 0.5807549, 0.25445306, 0.6887169], - [0.46632785, 0.9006454, 0.8054941, 0.4738411, 0.30304378, 0.2902605] - ] - self.assertAllClose(expected_prob_out, prob_out) - self.assertAllClose(expected_atten_vec_out, atten_vec_out) - - def testMultiHeadedAttentionMonotonic(self): - # source_batch:3, target_batch:6. Test n = 2 case. - with self.session(use_gpu=True): - (source_vecs, source_contexts, source_padding, source_padding_p, - query_vec, _, _) = self._MultiHeadedAttentionInputs() - iap = attention.MonotonicAttention.Params() - iap.name = 'mono_atten' - params = attention.MultiHeadedAttention.Params().Set( - name='multihead_atten', - source_dim=4, - query_dim=4, - hidden_dim=4, - inner_atten_params=iap, - num_attention_heads=2, - use_source_vec_as_attention_value=False) - atten = params.Instantiate() - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - # [batch * 2 heads, time] - atten_init_state = self._attentionStateWithRandomEmitProbabilities( - atten, 12, 6) - print('atten_init_state', atten_init_state) - self.evaluate(tf.global_variables_initializer()) - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec, atten_init_state) - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - print('atten_vec_out', np.sum(atten_vec_out, axis=1)) - self.assertAllClose( - [1.494033, 1.120422, 1.699309, 1.508609, 1.1329, 1.670303], - np.sum(atten_vec_out, axis=1)) - print('atten_vec_out', atten_vec_out) - print('prob_out', prob_out) - t_batch_size = 6 - s_batch_size = 3 - for i in range(t_batch_size): - # Test to make sure we didn't mess up indexing. - s_index = i % s_batch_size - atten.InitForSourcePacked(atten.theta, - source_vecs[:, s_index:s_index + 1, :], - source_contexts[:, s_index:s_index + 1, :], - source_padding[:, s_index:s_index + 1]) - j = i * 2 - sliced_atten_state = py_utils.NestedMap( - emit_probs=atten_init_state.emit_probs[j:j + 2]) - atten_vec_i, prob_i, _ = atten.ComputeContextVector( - atten.theta, query_vec[i:i + 1], sliced_atten_state) - atten_vec_i_out, prob_i_out = self.evaluate([atten_vec_i, prob_i]) - self.assertAllClose(prob_i_out, prob_out[i:i + 1]) - self.assertAllClose(atten_vec_i_out, atten_vec_out[i:i + 1]) - padding_i = source_padding_p[s_index] - # Check to make sure prob exists only on valid timesteps. - self.assertEqual(0.0, np.sum(padding_i * prob_i_out)) - - def testMultiHeadedAttentionDotProductWithAllProj(self): - # source_batch:3, target_batch:6. Test n = 2 case. - with self.session(use_gpu=True): - (source_vecs, source_contexts, source_padding, source_padding_p, - query_vec, _, _) = self._MultiHeadedAttentionInputs() - iap = attention.DotProductAttention.Params() - iap.name = 'dot_atten' - params = attention.MultiHeadedAttention.Params().Set( - name='multihead_atten', - source_dim=4, - query_dim=4, - hidden_dim=4, - inner_atten_params=iap, - num_attention_heads=2, - use_source_vec_as_attention_value=False, - enable_ctx_pre_proj=True, - enable_ctx_post_proj=True, - ctx_post_proj_dim=5, - context_dim=6) - atten = params.Instantiate() - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - self.evaluate(tf.global_variables_initializer()) - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec) - - self._CheckStaticShapes( - atten_vec, - atten_prob, - target_batch_size=query_vec.shape[0], - source_length=source_contexts.shape[0], - context_dim=params.ctx_post_proj_dim) - - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - print('atten_vec_out', np.sum(atten_vec_out, axis=1)) - self.assertAllClose([ - 1.356745, 0.65274805, 1.39460433, 1.34961343, 0.63025361, 1.41543126 - ], np.sum(atten_vec_out, axis=1)) - print('atten_vec_out', atten_vec_out) - print('prob_out', prob_out) - t_batch_size = 6 - s_batch_size = 3 - for i in range(t_batch_size): - # Test to make sure we didn't mess up indexing. - s_index = i % s_batch_size - atten.InitForSourcePacked(atten.theta, - source_vecs[:, s_index:s_index + 1, :], - source_contexts[:, s_index:s_index + 1, :], - source_padding[:, s_index:s_index + 1]) - atten_vec_i, prob_i, _ = atten.ComputeContextVector( - atten.theta, query_vec[i:i + 1]) - atten_vec_i_out, prob_i_out = self.evaluate([atten_vec_i, prob_i]) - self.assertAllClose(prob_i_out, prob_out[i:i + 1]) - self.assertAllClose(atten_vec_i_out, atten_vec_out[i:i + 1]) - padding_i = source_padding_p[s_index] - # Check to make sure prob exists only on valid timesteps. - self.assertEqual(0.0, np.sum(padding_i * prob_i_out)) - - def testMultiHeadedAttentionDotProductMultiPostProj(self): - # Test for multiple attention post-projection. - # source_batch:3, target_batch:6. Test n = 2 case. - with self.session(use_gpu=True): - (source_vecs, source_contexts, source_padding, _, query_vec, _, - _) = self._MultiHeadedAttentionInputs() - iap = attention.DotProductAttention.Params() - iap.name = 'dot_atten' - params = attention.MultiHeadedAttention.Params().Set( - name='multihead_atten', - source_dim=4, - query_dim=4, - hidden_dim=4, - inner_atten_params=iap, - num_attention_heads=2, - use_source_vec_as_attention_value=False, - enable_ctx_pre_proj=True, - enable_ctx_post_proj=True, - ctx_post_proj_dim=6, - context_dim=6, - num_post_proj=2) - atten = params.Instantiate() - packed_src = atten.InitForSourcePacked(atten.theta, source_vecs, - source_contexts, source_padding) - # Duplicate atten_idx n=2 times. - atten_idx = tf.constant([0, 1, 1] * 2, dtype=tf.int32) - self.evaluate(tf.global_variables_initializer()) - atten_vec, atten_prob, _ = atten.ComputeContextVectorWithSource( - atten.theta, packed_src, query_vec, atten_idx=atten_idx) - - self._CheckStaticShapes( - atten_vec, - atten_prob, - target_batch_size=query_vec.shape[0], - source_length=source_contexts.shape[0], - context_dim=source_contexts.shape[2]) - - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - self.assertAllClose([ - 0.66697717, 0.52266854, 0.7827165, 0.65693897, 0.51808167, 0.82977116 - ], np.sum(atten_vec_out, axis=1)) - print('atten_vec_out', atten_vec_out) - print('prob_out', prob_out) - - def _testMultiHeadedAttentionAdditiveHelper(self, - source_dim, - expected_vec, - packed_input=False): - # source_batch:3, target_batch:6. Test n = 2 case. - with self.session(use_gpu=True): - (source_vecs, source_contexts, source_padding, source_padding_p, - query_vec, source_seg_id, - query_seg_id) = self._MultiHeadedAttentionInputs(source_dim) - if not packed_input: - source_seg_id = None - query_seg_id = None - iap = attention.AdditiveAttention.Params() - iap.name = 'add_atten' - params = attention.MultiHeadedAttention.Params().Set( - name='multihead_atten', - source_dim=source_dim, - query_dim=4, - hidden_dim=4, - num_attention_heads=2, - inner_atten_params=iap, - use_source_vec_as_attention_value=False, - vn=py_utils.VariationalNoiseParams(0.0, False, False), - packed_input=packed_input) - atten = params.Instantiate() - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding, source_seg_id) - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec, query_segment_id=query_seg_id) - - self._CheckStaticShapes( - atten_vec, - atten_prob, - target_batch_size=query_vec.shape[0], - source_length=source_contexts.shape[0], - context_dim=source_contexts.shape[2]) - - self.evaluate(tf.global_variables_initializer()) - atten_vec_out, prob_out = self.evaluate([atten_vec, atten_prob]) - print('atten_vec_out', np.sum(atten_vec_out, axis=1)) - self.assertAllClose(expected_vec, np.sum(atten_vec_out, axis=1)) - print('atten_vec_out', atten_vec_out) - print('prob_out', prob_out) - t_batch_size = 6 - s_batch_size = 3 - for i in range(t_batch_size): - # Test to make sure we didn't mess up indexing. - s_index = i % s_batch_size - src_seg_id = None - if packed_input: - src_seg_id = source_seg_id[:, s_index:s_index + 1] - atten.InitForSourcePacked(atten.theta, - source_vecs[:, s_index:s_index + 1, :], - source_contexts[:, s_index:s_index + 1, :], - source_padding[:, s_index:s_index + 1], - src_seg_id) - qry_seg_id = None - if packed_input: - qry_seg_id = query_seg_id[i:i + 1] - atten_vec_i, prob_i, _ = atten.ComputeContextVector( - atten.theta, query_vec[i:i + 1], query_segment_id=qry_seg_id) - atten_vec_i_out, prob_i_out = self.evaluate([atten_vec_i, prob_i]) - self.assertAllClose(prob_i_out, prob_out[i:i + 1]) - self.assertAllClose(atten_vec_i_out, atten_vec_out[i:i + 1]) - padding_i = source_padding_p[s_index] - # Check to make sure prob exists only on valid timesteps. - self.assertEqual(0.0, np.sum(padding_i * prob_i_out)) - - def testMultiHeadedAttentionAttStateSingleProb(self): - # source_batch:3, target_batch:6. Test n = 2 case. - with self.session(use_gpu=True): - (source_vecs, source_contexts, source_padding, _, query_vec, _, - _) = self._MultiHeadedAttentionInputs() - iap = attention.DotProductAttention.Params() - iap.name = 'dot_atten' - params = attention.MultiHeadedAttention.Params().Set( - name='multihead_atten', - source_dim=4, - query_dim=4, - hidden_dim=4, - inner_atten_params=iap, - num_attention_heads=2, - use_source_vec_as_attention_value=False, - attention_head_prob_index=1) - atten = params.Instantiate() - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - atten_state = atten.ZeroAttentionState(2, 6) - print('atten_state:', atten_state) - - atten_vec, atten_prob, atten_state = atten.ComputeContextVector( - atten.theta, query_vec, atten_state) - self.evaluate(tf.global_variables_initializer()) - atten_vec_out, _, atten_state = self.evaluate( - [atten_vec, atten_prob, atten_state]) - print('atten_vec_out', np.sum(atten_vec_out, axis=1)) - print('atten_state', atten_state) - self.assertAllClose([ - 2.84679317, 2.36924601, 3.54831171, 2.86487937, 2.3537426, 3.54308939 - ], np.sum(atten_vec_out, axis=1)) - expected_prob_atten_state = ([ - [0.24530524, 0.24182455, 0, 0, 0.2497975, 0.2630728], - [0, 0.24692935, 0.23176268, 0.26929834, 0, 0.2520097], - [0.28280658, 0.23664463, 0, 0.18057014, 0, 0.29997864], - [0.21294391, 0.2421909, 0, 0, 0.2702513, 0.27461392], - [0, 0.25139052, 0.24466391, 0.25138932, 0, 0.25255626], - [0.25900277, 0.2514635, 0, 0.23059677, 0, 0.25893703] - ]) - self.assertAllClose(expected_prob_atten_state, - atten_state.selected_attention_head_probs) - - def testMultiHeadedAttentionAdditive(self): - self._testMultiHeadedAttentionAdditiveHelper( - 4, [2.858081, 2.33295, 3.529434, 2.856466, 2.342262, 3.526487]) - - def testMultiHeadedAttentionAdditivePackedInput(self): - self._testMultiHeadedAttentionAdditiveHelper( - 4, [2.585192, 2.267683, 3.708972, 3.107646, 2.770367, 3.580353], - packed_input=True) - - def testMultiHeadedAttentionAdditiveUnequalDim(self): - self._testMultiHeadedAttentionAdditiveHelper( - 14, [3.189594, 2.462574, 2.912001, 3.19924, 2.462459, 2.909231]) - - def testLocationSensitiveAttention1(self): - with self.session(use_gpu=True): - np.random.seed(12345) - source_vecs = tf.stack([ - tf.constant(np.random.rand(3, 4), dtype=tf.float32) for _ in range(6) - ]) - source_contexts = tf.stack([ - tf.constant(np.random.rand(3, 5), dtype=tf.float32) for _ in range(6) - ]) - source_padding = tf.transpose( - tf.constant( - [[0, 0, 1, 1, 0, 0], [1, 0, 0, 0, 1, 0], [0, 0, 1, 0, 1, 0]], - dtype=tf.float32)) - query_vec = tf.constant(np.random.rand(6, 7), dtype=tf.float32) - - params = attention.LocationSensitiveAttention.Params() - params.name = 'loc_atten' - params.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - params.source_dim = 4 - params.query_dim = 7 - params.hidden_dim = 7 - params.vn.global_vn = False - params.vn.per_step_vn = False - params.location_filter_size = 3 - params.location_num_filters = 4 - - atten = attention.LocationSensitiveAttention(params) - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - - atten_init_state = tf.nn.softmax( - tf.constant( - np.random.rand(6, len(params.location_features), 6), - dtype=tf.float32)) - - atten_vec, atten_prob, atten_state = atten.ComputeContextVector( - atten.theta, query_vec, atten_init_state) - - atten_vars = tf.get_collection('LocationSensitiveAttention_vars') - self.assertLen(atten_vars, 5) - - self.evaluate(tf.global_variables_initializer()) - - atten_vec_out, prob_out, atten_init_state_out, atten_state_out = self.evaluate( - [atten_vec, atten_prob, atten_init_state, atten_state]) - - self.assertEqual(atten_init_state_out.shape, atten_state_out.shape) - - print(['additive attention prob_out', np.array_repr(prob_out)]) - print(['additive attention atten_vec_out', np.array_repr(atten_vec_out)]) - - # pyformat: disable - # pylint: disable=bad-whitespace - expected_prob_out = [ - [ 0.25557119, 0.2407331 , 0. , 0. , 0.25413439, - 0.24956135], - [ 0. , 0.2539435 , 0.24765202, 0.25480285, 0. , - 0.24360162], - [ 0.25094694, 0.25000173, 0. , 0.24308425, 0. , - 0.25596702], - [ 0.25559491, 0.24071115, 0. , 0. , 0.2541317 , - 0.24956223], - [ 0. , 0.25393987, 0.24765508, 0.25481141, 0. , - 0.24359357], - [ 0.25112614, 0.24990462, 0. , 0.24246819, 0. , - 0.25650105]] - expected_atten_vec_out = [ - [ 0.49745601, 0.63471878, 0.49220741, 0.56829882, 0.42753279], - [ 0.51502693, 0.56184328, 0.37644374, 0.87425017, 0.46182287], - [ 0.57862061, 0.44247472, 0.36931327, 0.41002682, 0.14327496], - [ 0.49745524, 0.63471991, 0.49221092, 0.56828701, 0.427522 ], - [ 0.51502484, 0.5618462 , 0.37644884, 0.87424958, 0.46181911], - [ 0.57893252, 0.44248456, 0.36938512, 0.4100675 , 0.14328022]] - # pyformat: enable - # pylint: enable=bad-whitespace - - self.assertAllClose(expected_prob_out, prob_out) - self.assertAllClose(expected_atten_vec_out, atten_vec_out) - - def testLocationSensitiveAttention2(self): - with self.session(use_gpu=True): - np.random.seed(12345) - source_vecs = tf.stack([ - tf.constant(np.random.rand(3, 4), dtype=tf.float32) for _ in range(6) - ]) - source_contexts = tf.stack([ - tf.constant(np.random.rand(3, 5), dtype=tf.float32) for _ in range(6) - ]) - source_padding = tf.transpose( - tf.constant( - [[0, 0, 1, 1, 0, 0], [1, 0, 0, 0, 1, 0], [0, 0, 1, 0, 1, 0]], - dtype=tf.float32)) - query_vec = tf.constant(np.random.rand(6, 7), dtype=tf.float32) - - params = attention.LocationSensitiveAttention.Params() - params.name = 'loc_atten' - params.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - params.source_dim = 4 - params.query_dim = 7 - params.hidden_dim = 7 - params.vn.global_vn = False - params.vn.per_step_vn = False - params.location_filter_size = 3 - params.location_num_filters = 4 - params.location_features = ['PREV_PROBS', 'CUMULATIVE_PROBS'] - - atten = attention.LocationSensitiveAttention(params) - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - - atten_init_state = atten.ZeroAttentionState(tf.shape(source_vecs)[0], 6) - - (unused_atten_vec, unused_atten_prob, - atten_state) = atten.ComputeContextVector(atten.theta, query_vec, - atten_init_state) - - atten_vars = tf.get_collection('LocationSensitiveAttention_vars') - self.assertLen(atten_vars, 5) - - self.evaluate(tf.global_variables_initializer()) - - atten_init_state_out, atten_state_out = self.evaluate( - [atten_init_state, atten_state]) - - self.assertEqual(atten_init_state_out.shape, atten_state_out.shape) - - def _testLocationSensitiveAttentionSameBatchSizeHelper( - self, same_batch_size, quantized=False): - with self.session(tf.Graph(), use_gpu=True): - np.random.seed(12345) - dtype = tf.float32 if quantized else tf.float64 - source_vecs = tf.constant(np.random.rand(6, 3, 4), dtype=dtype) - source_contexts = tf.constant(np.random.rand(6, 3, 5), dtype=dtype) - source_padding = tf.transpose( - tf.constant( - [[0, 0, 1, 1, 0, 0], [1, 0, 0, 0, 1, 0], [0, 0, 1, 0, 1, 0]], - dtype=dtype)) - - query_vec = tf.constant(np.random.rand(3, 7), dtype=dtype) - - params = attention.LocationSensitiveAttention.Params() - params.dtype = dtype - params.name = 'loc_atten' - params.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - params.source_dim = 4 - params.query_dim = 7 - params.hidden_dim = 7 - params.vn.global_vn = False - params.vn.per_step_vn = False - params.location_filter_size = 3 - params.location_num_filters = 4 - params.same_batch_size = same_batch_size - - if quantized: - cc_schedule = quant_utils.FakeQuantizationSchedule.Params().Set( - clip_start_step=0, - clip_end_step=13000, - quant_start_step=14000, - start_cap=8.0, - end_cap=1.0) - qdomain = quant_utils.SymmetricScheduledClipQDomain.Params().Set( - cc_schedule=cc_schedule.Copy()) - params.qdomain.default = qdomain.Copy() - params.qdomain.atten_context = qdomain.Copy() - - atten = attention.LocationSensitiveAttention(params) - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - - atten_init_state = tf.nn.softmax( - tf.constant( - np.random.rand(3, len(params.location_features), 6), dtype=dtype)) - - atten_vec, atten_prob, atten_state = atten.ComputeContextVector( - atten.theta, query_vec, atten_init_state) - - atten_vars = tf.get_collection('LocationSensitiveAttention_vars') - self.assertLen(atten_vars, 5) - - self.evaluate(tf.global_variables_initializer()) - - atten_vec_out, prob_out, atten_init_state_out, atten_state_out = self.evaluate( - [atten_vec, atten_prob, atten_init_state, atten_state]) - - self.assertEqual(atten_init_state_out.shape, atten_state_out.shape) - return atten_vec_out, prob_out, atten_init_state_out, atten_state_out - - def testLocationSensitiveAttentionSameBatchSize(self): - (atten_vec_out1, prob_out1, atten_init_state_out1, atten_state_out1) = ( - self._testLocationSensitiveAttentionSameBatchSizeHelper(True)) - (atten_vec_out2, prob_out2, atten_init_state_out2, atten_state_out2) = ( - self._testLocationSensitiveAttentionSameBatchSizeHelper(False)) - self.assertAllClose(atten_vec_out1, atten_vec_out2, rtol=1e-04, atol=1e-04) - self.assertAllClose(prob_out1, prob_out2, rtol=1e-04, atol=1e-04) - self.assertAllClose( - atten_init_state_out1, atten_init_state_out2, rtol=1e-04, atol=1e-04) - self.assertAllClose( - atten_state_out1, atten_state_out2, rtol=1e-04, atol=1e-04) - - def testLocationSensitiveAttentionQuantized(self): - (atten_vec_out1, prob_out1, atten_init_state_out1, atten_state_out1) = ( - self._testLocationSensitiveAttentionSameBatchSizeHelper(False, False)) - (atten_vec_out2, prob_out2, atten_init_state_out2, atten_state_out2) = ( - self._testLocationSensitiveAttentionSameBatchSizeHelper(False, True)) - self.assertAllClose(atten_vec_out1, atten_vec_out2, rtol=1e-02, atol=1e-02) - self.assertAllClose(prob_out1, prob_out2, rtol=1e-02, atol=1e-02) - self.assertAllClose( - atten_init_state_out1, atten_init_state_out2, rtol=1e-02, atol=1e-02) - self.assertAllClose( - atten_state_out1, atten_state_out2, rtol=1e-02, atol=1e-02) - - def testLocationSensitiveAttentionQuantizedSameBatch(self): - (atten_vec_out1, prob_out1, atten_init_state_out1, atten_state_out1) = ( - self._testLocationSensitiveAttentionSameBatchSizeHelper(True, False)) - (atten_vec_out2, prob_out2, atten_init_state_out2, atten_state_out2) = ( - self._testLocationSensitiveAttentionSameBatchSizeHelper(True, True)) - self.assertAllClose(atten_vec_out1, atten_vec_out2, rtol=1e-02, atol=1e-02) - self.assertAllClose(prob_out1, prob_out2, rtol=1e-02, atol=1e-02) - self.assertAllClose( - atten_init_state_out1, atten_init_state_out2, rtol=1e-02, atol=1e-02) - self.assertAllClose( - atten_state_out1, atten_state_out2, rtol=1e-02, atol=1e-02) - - def _attentionStateWithRandomEmitProbabilities(self, - atten, - batch_size, - time, - dtype=tf.float32): - atten_state = atten.ZeroAttentionState(time, batch_size) - atten_state.emit_probs = tf.nn.softmax( - tf.constant(np.random.rand(batch_size, time), dtype=dtype)) - return atten_state - - def testMonotonicAttention(self): - with self.session(use_gpu=True): - np.random.seed(12345) - batch_size = 3 - source_dim = 4 - context_dim = 5 - time = 6 - query_dim = 7 - source_vecs = tf.constant( - np.random.rand(time, batch_size, source_dim), dtype=tf.float32) - source_contexts = tf.constant( - np.random.rand(time, batch_size, context_dim), dtype=tf.float32) - source_padding = tf.transpose( - tf.constant( - [[0, 0, 1, 1, 0, 0], [1, 0, 0, 0, 1, 0], [0, 0, 1, 0, 1, 0]], - dtype=tf.float32)) - query_vec = tf.constant( - np.random.rand(batch_size, query_dim), dtype=tf.float32) - - params = attention.MonotonicAttention.Params() - params.name = 'monotonic_attention' - params.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - params.source_dim = source_dim - params.query_dim = query_dim - params.hidden_dim = query_dim - params.vn.global_vn = False - params.vn.per_step_vn = False - - atten = attention.MonotonicAttention(params) - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - - atten_init_state = self._attentionStateWithRandomEmitProbabilities( - atten, batch_size, time) - atten_vec, atten_prob, atten_state = atten.ComputeContextVector( - atten.theta, query_vec, atten_init_state) - - atten_vars = tf.get_collection('MonotonicAttention_vars') - self.assertLen(atten_vars, 6) - - self.evaluate(tf.global_variables_initializer()) - - atten_vec_out, prob_out, atten_init_state_out, atten_state_out = self.evaluate( - [atten_vec, atten_prob, atten_init_state, atten_state]) - - self.assertEqual(atten_init_state_out.emit_probs.shape, - atten_state_out.emit_probs.shape) - - print(['monotonic attention prob_out', np.array_repr(prob_out)]) - print(['monotonic attention atten_vec_out', np.array_repr(atten_vec_out)]) - - expected_prob_out = [[ - 0.03654566, 0.05925026, 0., 0., 0.20958641, 0.19560105 - ], [0., 0.09670404, 0.13182665, 0.13221622, 0., - 0.18074416], [0.04112773, 0.07072841, 0., 0.13837409, 0., 0.23935230]] - - expected_atten_vec_out = [ - [0.2937718, 0.30372939, 0.27034321, 0.31328040, 0.19393572], - [0.2553753, 0.26388022, 0.20429659, 0.47469878, 0.27512118], - [0.33394262, 0.1191523, 0.22405925, 0.21366173, 0.03946214] - ] - - self.assertAllClose(expected_prob_out, prob_out) - self.assertAllClose(expected_atten_vec_out, atten_vec_out) - - def testMonotonicAttentionHard(self): - with self.session(use_gpu=True): - batch_size = 3 - source_dim = 4 - context_dim = 5 - time = 6 - query_dim = 10 - source_vecs = tf.constant( - np.random.randn(time, batch_size, source_dim), dtype=tf.float32) - source_contexts = tf.constant( - np.random.randn(time, batch_size, context_dim), dtype=tf.float32) - source_padding = tf.zeros((time, batch_size), dtype=tf.float32) - query_vec = tf.constant( - np.random.randn(batch_size, query_dim), dtype=tf.float32) - - params = attention.MonotonicAttention.Params() - params.name = 'monotonic_attention' - params.params_init = py_utils.WeightInit.Gaussian(0.1) - params.source_dim = source_dim - params.query_dim = query_dim - params.hidden_dim = query_dim - params.vn.global_vn = False - params.vn.per_step_vn = False - params.hard_sigmoid = True - # To encourage some probabilities to be > 0 - params.hidden_bias_init = 0. - - atten = attention.MonotonicAttention(params) - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - - atten_init_state = atten.ZeroAttentionState(time, batch_size) - - _, atten_prob, atten_state = atten.ComputeContextVector( - atten.theta, query_vec, atten_init_state) - - atten_vars = tf.get_collection('MonotonicAttention_vars') - self.assertLen(atten_vars, 6) - - self.evaluate(tf.global_variables_initializer()) - - prob_out, atten_state_out = self.evaluate([atten_prob, atten_state]) - print(['hard monotonic prob', np.array_repr(prob_out)]) - # Make sure all probabilities are binary - self.assertTrue(np.all(np.logical_or(prob_out == 0, prob_out == 1))) - # Make sure either one index was attended or none were - prob_sum = np.sum(prob_out, 1) - self.assertTrue(np.all(np.logical_or(prob_sum == 1, prob_sum == 0))) - - query_vec = tf.constant( - np.random.randn(batch_size, query_dim), dtype=tf.float32) - # Feed state back in - _, atten_prob, atten_state = atten.ComputeContextVector( - atten.theta, query_vec, atten_state_out) - prob_out2 = self.evaluate(atten_prob) - print(['hard monotonic prob2', np.array_repr(prob_out2)]) - # Get indices of where attention was assigned at each output timestep - idx1 = np.argmax(prob_out, 1) - idx2 = np.argmax(prob_out2, 1) - # Either the index must have increased, or all probs were 0 - self.assertTrue( - np.all(np.logical_or(idx1 <= idx2, - np.sum(prob_out2, 1) == 0))) - - def testMonotonicAttentionBackProp(self): - with self.session(use_gpu=True) as sess: - # Use float64 dtype for numeric checks - dtype = tf.float64 - tf.random.set_seed(398847392) - np.random.seed(12345) - batch_size = 3 - source_dim = 4 - context_dim = 5 - time = 6 - query_dim = 7 - source_vecs = tf.constant( - np.random.rand(time, batch_size, source_dim), dtype=tf.float64) - source_contexts = tf.constant( - np.random.rand(time, batch_size, context_dim), dtype=tf.float64) - source_padding = tf.zeros((time, batch_size), dtype=tf.float64) - query_vec = tf.constant( - np.random.rand(batch_size, query_dim), dtype=tf.float64) - - params = attention.MonotonicAttention.Params() - params.name = 'monotonic_attention' - params.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - params.source_dim = source_dim - params.query_dim = query_dim - params.hidden_dim = query_dim - params.dtype = dtype - params.vn.global_vn = False - params.vn.per_step_vn = False - - atten = attention.MonotonicAttention(params) - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - - atten_init_state = self._attentionStateWithRandomEmitProbabilities( - atten, batch_size, time, dtype=dtype) - - atten_vec, _, _ = atten.ComputeContextVector(atten.theta, query_vec, - atten_init_state) - - loss = tf.reduce_sum(atten_vec) - - all_vars = tf.trainable_variables() - self.assertLen(all_vars, 6) - - grads = tf.gradients(loss, all_vars) - self.evaluate(tf.global_variables_initializer()) - sym_grads = [sg.eval() for sg in grads] - num_grads = [ - test_utils.ComputeNumericGradient(sess, loss, v) for v in all_vars - ] - - print(sym_grads) - print(num_grads) - - for sg, ng in zip(sym_grads, num_grads): - self.assertAllClose(sg, ng, rtol=1e-06, atol=1e-06) - - def _testPerStepSourcePaddingHelper(self, atten, depth=6, atten_state=None): - with self.session(use_gpu=True): - np.random.seed(505837249) - source_vecs = tf.stack([ - tf.constant(np.random.rand(2, depth), dtype=tf.float32) - for _ in range(6) - ]) - source_contexts = tf.stack([ - tf.constant(np.random.rand(2, depth), dtype=tf.float32) - for _ in range(6) - ]) - source_padding = tf.transpose( - tf.constant([[0, 0, 1, 1, 0, 0], [1, 0, 0, 0, 1, 0]], - dtype=tf.float32)) - query_vec = tf.constant(np.random.rand(2, depth), dtype=tf.float32) - query_vec = tf.concat([query_vec, query_vec], 0) - - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - # No per_step_padding. - atten_vec1, atten_prob1, _ = atten.ComputeContextVector( - atten.theta, - query_vec, - attention_state=atten_state, - per_step_source_padding=None) - per_step_padding = tf.constant( - [[0.0, 1.0, 1.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0], - [0.0, 1.0, 1.0, 0.0, 0.0, 1.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]], - dtype=tf.float32) - atten_vec2, atten_prob2, _ = atten.ComputeContextVector( - atten.theta, - query_vec, - attention_state=atten_state, - per_step_source_padding=per_step_padding) - - self.evaluate(tf.global_variables_initializer()) - atten_vec1_out, atten_prob1_out = self.evaluate([atten_vec1, atten_prob1]) - atten_vec2_out, atten_prob2_out = self.evaluate([atten_vec2, atten_prob2]) - print('atten_prob1_out', atten_prob1_out) - print('atten_prob2_out', atten_prob2_out) - print('atten_vec1_out', atten_vec1_out) - print('atten_vec2_out', atten_vec2_out) - self.assertAllClose(atten_prob1_out[:2], atten_prob1_out[2:]) - self.assertAllClose(atten_vec1_out[:2], atten_vec1_out[2:]) - self.assertAllClose(atten_prob1_out[1], atten_prob2_out[1]) - self.assertAllClose(atten_vec1_out[1], atten_vec2_out[1]) - self.assertAllClose(atten_prob1_out[3], atten_prob2_out[3]) - self.assertAllClose(atten_vec1_out[3], atten_vec2_out[3]) - self.assertAllClose(atten_prob2_out[1], atten_prob2_out[3]) - self.assertAllClose(atten_vec2_out[1], atten_vec2_out[3]) - self.assertGreater( - np.max(np.abs(atten_prob1_out[0] - atten_prob2_out[0])), 0.1) - self.assertGreater( - np.max(np.abs(atten_prob1_out[2] - atten_prob2_out[2])), 0.1) - self.assertGreater( - np.max(np.abs(atten_prob2_out[0] - atten_prob2_out[2])), 0.1) - return atten_prob2_out, atten_vec2_out - - def testPerStepSourcePaddingAdditiveAttention(self): - params = attention.AdditiveAttention.Params() - params.name = 'atten' - params.params_init = py_utils.WeightInit.Gaussian(0.1, 877374) - depth = 6 - params.source_dim = depth - params.query_dim = depth - params.hidden_dim = depth - params.vn.global_vn = False - params.vn.per_step_vn = False - atten = params.Instantiate() - prob_out, vec_out = self._testPerStepSourcePaddingHelper(atten, depth) - print('vec_out', np.array_repr(np.sum(vec_out, 1))) - self.assertAllClose([2.00084352, 3.2933836, 2.30622029, 3.2933836], - np.sum(vec_out, 1)) - self.assertAllClose([1.0, 1.0, 1.0, 1.0], np.sum(prob_out, 1)) - - def testPerStepSourcePaddingDotProductAttention(self): - params = attention.DotProductAttention.Params() - params.name = 'atten' - depth = 6 - params.source_dim = depth - params.query_dim = depth - params.hidden_dim = depth - params.vn.global_vn = False - params.vn.per_step_vn = False - atten = params.Instantiate() - prob_out, vec_out = self._testPerStepSourcePaddingHelper(atten, depth) - print('vec_out', np.array_repr(np.sum(vec_out, 1))) - self.assertAllClose([2.02671742, 3.38590097, 2.34964013, 3.38590097], - np.sum(vec_out, 1)) - self.assertAllClose([1.0, 1.0, 1.0, 1.0], np.sum(prob_out, 1)) - - def testPerStepSourcePaddingMultiHeadedAttention(self): - params = attention.MultiHeadedAttention.Params() - params.name = 'atten' - params.params_init = py_utils.WeightInit.Gaussian(0.1, 877374) - depth = 6 - params.source_dim = depth - params.query_dim = depth - params.hidden_dim = depth - params.vn.global_vn = False - params.vn.per_step_vn = False - atten = params.Instantiate() - prob_out, vec_out = self._testPerStepSourcePaddingHelper(atten, depth) - print('vec_out', np.array_repr(np.sum(vec_out, 1))) - self.assertAllClose([-0.006338, -0.025153, 0.041647, -0.025153], - np.sum(vec_out, 1)) - self.assertAllClose([1.0, 1.0, 1.0, 1.0], np.sum(prob_out, 1)) - - def testPerStepSourcePaddingLocationSensitiveAttention(self): - params = attention.LocationSensitiveAttention.Params() - params.name = 'atten' - params.params_init = py_utils.WeightInit.Gaussian(0.1, 877374) - depth = 6 - params.source_dim = depth - params.query_dim = depth - params.hidden_dim = depth - params.location_filter_size = 3 - params.location_num_filters = 4 - params.vn.global_vn = False - params.vn.per_step_vn = False - atten_state = tf.concat( - [tf.ones([4, 1], tf.float32), - tf.zeros([4, 5], tf.float32)], 1) - atten_state = tf.expand_dims(atten_state, 1) - atten = params.Instantiate() - prob_out, vec_out = self._testPerStepSourcePaddingHelper( - atten, depth, atten_state=atten_state) - print('vec_out', np.array_repr(np.sum(vec_out, 1))) - self.assertAllClose([2.001103, 3.293414, 2.306448, 3.293414], - np.sum(vec_out, 1)) - self.assertAllClose([1.0, 1.0, 1.0, 1.0], np.sum(prob_out, 1)) - - def testPerStepSourcePaddingMonotonicAttention(self): - params = attention.MonotonicAttention.Params() - params.name = 'atten' - params.params_init = py_utils.WeightInit.Gaussian(0.1, 877374) - depth = 6 - params.source_dim = depth - params.query_dim = depth - params.hidden_dim = depth - params.vn.global_vn = False - params.vn.per_step_vn = False - atten = params.Instantiate() - atten_state = atten.ZeroAttentionState(6, 4) - atten_state.emit_probs = tf.concat( - [tf.ones([4, 1], tf.float32), - tf.zeros([4, 5], tf.float32)], 1) - prob_out, vec_out = self._testPerStepSourcePaddingHelper( - atten, depth, atten_state=atten_state) - print('prob_out', np.array_repr(np.sum(prob_out, 1))) - print('vec_out', np.array_repr(np.sum(vec_out, 1))) - - def testGmmMonotonicAttentionDropout(self): - p = attention.GmmMonotonicAttention.Params().Set( - name='gmm_monotonic_attention', atten_dropout_prob=0.5) - with self.assertRaises(NotImplementedError): - p.Instantiate() - - def testGmmMonotonicAttention(self): - with self.session(use_gpu=True): - np.random.seed(12345) - source_vecs = tf.stack([ - tf.constant(np.random.rand(3, 4), dtype=tf.float32) for _ in range(6) - ]) - source_contexts = tf.stack([ - tf.constant(np.random.rand(3, 5), dtype=tf.float32) for _ in range(6) - ]) - source_padding = tf.transpose( - tf.constant( - [[0, 0, 1, 1, 0, 0], [1, 0, 0, 0, 1, 0], [0, 0, 1, 0, 1, 0]], - dtype=tf.float32)) - query_vec = tf.constant(np.random.rand(6, 7), dtype=tf.float32) - - params = attention.GmmMonotonicAttention.Params() - params.name = 'gmm_atten' - params.params_init = py_utils.WeightInit.Xavier(0.1) - params.source_dim = 4 - params.query_dim = 7 - params.hidden_dim = 7 - params.num_mixtures = 2 - params.vn.global_vn = False - params.vn.per_step_vn = False - - atten = params.Instantiate() - atten.InitForSourcePacked(atten.theta, source_vecs, source_contexts, - source_padding) - # target_batch=6 - atten_init_state = atten.ZeroAttentionState(tf.shape(source_vecs)[0], 6) - - atten_vec, atten_prob, atten_state = atten.ComputeContextVector( - atten.theta, query_vec, atten_init_state) - - self.evaluate(tf.global_variables_initializer()) - - atten_vec_out, prob_out, atten_init_state_out, atten_state_out = self.evaluate( - [atten_vec, atten_prob, atten_init_state, atten_state]) - - self.assertEqual(atten_init_state_out.shape, atten_state_out.shape) - self.assertEqual(atten_init_state_out.shape, (6, 2, 4)) - - print(['gmm attention prob_out', np.array_repr(prob_out)]) - print(['gmm attention atten_vec_out', np.array_repr(atten_vec_out)]) - - # pyformat: disable - # pylint: disable=bad-whitespace - expected_prob_out = [ - [ 2.45764434e-01, 3.97835493e-01, 0., 0., 4.25808690e-03, - 1.29864624e-04], - [ 0., 3.98021877e-01, 2.37964690e-01, 5.23146540e-02, 0., - 1.29256863e-04], - [ 2.46294901e-01, 3.97767872e-01, 0., 5.21243662e-02, 0., - 1.29372784e-04], - [ 2.45875627e-01, 3.97635251e-01, 0., 0., 4.27022483e-03, - 1.30706903e-04], - [ 0., 3.97709191e-01, 2.37897262e-01, 5.24106659e-02, 0., - 1.30714150e-04], - [ 2.46048093e-01, 3.97871077e-01, 0., 5.21884784e-02, 0., - 1.29211781e-04]] - expected_atten_vec_out = [ - [ 0.23010808, 0.43757612, 0.25150469, 0.3631629 , 0.37140277], - [ 0.54693544, 0.56182981, 0.21333349, 0.58108622, 0.21566363], - [ 0.4048025 , 0.53986353, 0.13288836, 0.22497796, 0.17450145], - [ 0.23008531, 0.4375343 , 0.25150725, 0.36303982, 0.37127423], - [ 0.54661846, 0.5615437 , 0.21332006, 0.58084518, 0.21558265], - [ 0.40484226, 0.53978455, 0.13283314, 0.22490481, 0.17447782]] - # pyformat: enable - # pylint: enable=bad-whitespace - - self.assertAllClose(expected_prob_out, prob_out) - self.assertAllClose(expected_atten_vec_out, atten_vec_out) - - def testMergerLayerMean(self): - with self.session(use_gpu=True): - np.random.seed(505837249) - depth = 4 - batch = 5 - n_sources = 3 - p_ctxs = [ - np.random.rand(batch, depth).astype('float32') - for _ in range(n_sources) - ] - ctxs = [tf.constant(ctx, dtype=tf.float32) for ctx in p_ctxs] - - p = attention.MergerLayer.Params() - p.name = 'merger_layer' - p.merger_op = 'mean' - p.source_dim = depth - merger = p.Instantiate() - - ctx = merger.FProp(merger.theta, ctxs) - self.evaluate(tf.global_variables_initializer()) - actual_ctx = self.evaluate([ctx])[0] - - expected_ctx = np.mean(p_ctxs, axis=0) - self.assertEqual(actual_ctx.shape, (batch, depth)) - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - - def testMergerLayerAdditiveAttention(self): - with self.session(use_gpu=True): - np.random.seed(505837249) - depth = 4 - batch = 5 - query_dim = 7 - n_sources = 3 - ctxs = [ - tf.constant(np.random.rand(batch, depth), dtype=tf.float32) - for _ in range(n_sources) - ] - query_vec = tf.constant( - np.random.rand(batch * 2, query_dim), dtype=tf.float32) - p = attention.MergerLayer.Params() - p.name = 'merger_layer' - p.merger_op = 'atten' - p.source_dim = depth - p.query_dim = query_dim - p.hidden_dim = depth - merger = p.Instantiate() - - ctx = merger.FProp(merger.theta, ctxs, query_vec) - self.evaluate(tf.global_variables_initializer()) - actual_ctx = self.evaluate(ctx) - - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [ 0.40796196, 0.50855637, 0.92564321, 0.72608167], - [ 0.34300309, 0.17305931, 0.64801621, 0.4161588 ], - [ 0.40570667, 0.28166312, 0.07109687, 0.07077176], - [ 0.44923055, 0.56033343, 0.70899796, 0.73256713], - [ 0.56362778, 0.42331296, 0.47032064, 0.76701462], - [ 0.40873578, 0.50516003, 0.92537481, 0.72435796], - [ 0.33702248, 0.17404726, 0.65101075, 0.41883218], - [ 0.40316698, 0.28128177, 0.0709244 , 0.07073996], - [ 0.44036126, 0.53640223, 0.68623006, 0.75264776], - [ 0.54324883, 0.42487082, 0.4616943 , 0.77234119]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertEqual(actual_ctx.shape, (batch * 2, depth)) - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - - def testMergerLayerDotProductAttention(self): - with self.session(use_gpu=True): - np.random.seed(505837249) - depth = 4 - batch = 5 - n_sources = 3 - ctxs = [ - tf.constant(np.random.rand(batch, depth), dtype=tf.float32) - for _ in range(n_sources) - ] - query_vec = tf.constant( - np.random.rand(batch * 2, depth), dtype=tf.float32) - p = attention.MergerLayer.Params() - p.name = 'merger_layer' - p.merger_op = 'atten' - p.source_dim = depth - p.query_dim = depth - p.hidden_dim = depth - p.attention_tpl = attention.DotProductAttention.Params() - merger = p.Instantiate() - - ctx = merger.FProp(merger.theta, ctxs, query_vec) - self.evaluate(tf.global_variables_initializer()) - actual_ctx = self.evaluate(ctx) - - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [ 0.40122974, 0.53032947, 0.92722446, 0.73408204], - [ 0.37834394, 0.16492322, 0.6284582 , 0.40583336], - [ 0.43172807, 0.28519249, 0.07334236, 0.07126588], - [ 0.48187545, 0.56433642, 0.7028234 , 0.77750808], - [ 0.59640014, 0.46689704, 0.47688526, 0.74523771], - [ 0.41653261, 0.50926942, 0.92638767, 0.74147904], - [ 0.34954029, 0.16965927, 0.64286244, 0.41876066], - [ 0.44629157, 0.28723121, 0.07451884, 0.07151417], - [ 0.509902 , 0.62019253, 0.75361776, 0.74199384], - [ 0.56122077, 0.42407531, 0.46921006, 0.76747787]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertEqual(actual_ctx.shape, (batch * 2, depth)) - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - - def testMergerLayerConcat(self): - with self.session(use_gpu=True): - np.random.seed(505837249) - depth = 4 - batch = 5 - n_sources = 3 - ctxs = [ - tf.constant(np.random.rand(batch, depth), dtype=tf.float32) - for _ in range(n_sources) - ] - p = attention.MergerLayer.Params() - p.name = 'merger_layer' - p.merger_op = 'concat' - p.source_dim = depth - merger = p.Instantiate() - - ctx = merger.FProp(merger.theta, ctxs) - self.evaluate(tf.global_variables_initializer()) - actual_ctx = self.evaluate([ctx])[0] - - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [ 0.1177848 , 0.94777811, 0.94537693, 0.6216979 , 0.51051533, - 0.5474115 , 0.93749231, 0.93760508, 0.5904724 , 0.05267439, - 0.89581013, 0.63010913], - [ 0.25139269, 0.13851869, 0.65362513, 0.57537138, 0.05093541, - 0.28593501, 0.84663856, 0.39284077, 0.79584485, 0.07670615, - 0.40381077, 0.26504567], - [ 0.1108813 , 0.23381528, 0.05560364, 0.06867393, 0.77289224, - 0.32918185, 0.10567363, 0.07876136, 0.35448784, 0.28477612, - 0.05394353, 0.06531866], - [ 0.82317245, 0.78475511, 0.82936037, 0.99494314, 0.07920805, - 0.02165302, 0.25108394, 0.92048419, 0.44413447, 0.81940264, - 0.98786688, 0.35846332], - [ 0.86243463, 0.75607926, 0.54042 , 0.58698255, 0.13624814, - 0.47994047, 0.28561282, 0.87185597, 0.66811442, 0.07942203, - 0.56781054, 0.83598584]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertEqual(actual_ctx.shape, (batch, n_sources * depth)) - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - - def testMergerLayerConcatPostProj(self): - with self.session(use_gpu=True): - np.random.seed(505837249) - depth = 4 - batch = 5 - n_sources = 3 - ctxs = [ - tf.constant(np.random.rand(batch, depth), dtype=tf.float32) - for _ in range(n_sources) - ] - p = attention.MergerLayer.Params() - p.name = 'merger_layer' - p.merger_op = 'concat' - p.source_dim = depth - # Post projection to a dimensionality of 4. - p.post_proj = layers.ProjectionLayer.Params().Set( - name='post_proj', - batch_norm=False, - weight_norm=False, - has_bias=True, - input_dim=12, - output_dim=4) - merger = p.Instantiate() - ctx = merger.FProp(merger.theta, ctxs) - self.evaluate(tf.global_variables_initializer()) - actual_ctx = self.evaluate([ctx])[0] - - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [0.05845007, 0.14603308, 2.099096 , 0.03618803], - [0.50603 , 0.1128372 , 1.0714196 , 0.3054366 ], - [0. , 0.17477296, 0. , 0. ], - [0.34721488, 0. , 0.9593564 , 0.6714128 ], - [0.012324 , 0. , 1.3537602 , 0.16794051]] - - # pyformat: enable - # pylint: enable=bad-whitespace - tf.logging.info(np.array_repr(actual_ctx)) - self.assertEqual(actual_ctx.shape, (batch, depth)) - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - - def testMergerLayerConcatPreProjections(self): - with self.session(use_gpu=True): - np.random.seed(505837249) - depth = 4 - batch = 5 - n_sources = 3 - ctxs = [ - tf.constant(np.random.rand(batch, depth), dtype=tf.float32) - for _ in range(n_sources) - ] - p = attention.MergerLayer.Params() - # We down project all of the sources to dimensionality 1. - p.pre_proj_input_dims = [4, 4, 4] - p.pre_proj_output_dims = [1, 1, 1] - p.name = 'merger_layer' - p.merger_op = 'concat' - p.source_dim = depth - merger = p.Instantiate() - - ctx = merger.FProp(merger.theta, ctxs) - self.evaluate(tf.global_variables_initializer()) - actual_ctx = self.evaluate([ctx])[0] - - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [ 0., 0.72890908, 0. ], - [ 0.4647972, 0.28266785, 0. ], - [ 0., 0.74580085, 0.09588336], - [ 0.46080768, 0., 0.66402191], - [ 0.19947493, 0.38837075, 0. ], - ] - # pyformat: enable - # pylint: enable=bad-whitespace - tf.logging.info(np.array_repr(actual_ctx)) - # The final context vector will have shape (5, 3) since each source - # has dimensionality 1 after the down projection above. - self.assertEqual(actual_ctx.shape, (batch, n_sources)) - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - - def testInvalidPreProjections(self): - with self.session(use_gpu=True): - np.random.seed(505837249) - depth = 4 - p = attention.MergerLayer.Params() - # We intentionally set output_dims to be of a different - # length. This should cause a ValueError to be raised - # during init. - p.pre_proj_input_dims = [4, 4, 4] - p.pre_proj_output_dims = [1, 1] - p.name = 'merger_layer' - p.merger_op = 'concat' - p.source_dim = depth - with self.assertRaisesRegex( - ValueError, 'Output dims should be the same length as input dims.*'): - _ = p.Instantiate() - - def testMergerLayerWeightedSum(self): - with self.session(use_gpu=True): - np.random.seed(505837249) - depth = 4 - batch = 2 - n_sources = 3 - ctxs = [[[1.0, 2.0, 3.0, 4.0], [2.0, 3.0, 4.0, 5.0]], - [[3.0, 4.0, 5.0, 6.0], [6.0, 7.0, 8.0, 9.0]], - [[4.0, 5.0, 6.0, 7.0], [7.0, 8.0, 1.0, 2.0]]] - p = attention.MergerLayer.Params() - p.name = 'merger_layer' - p.merger_op = 'weighted_sum' - p.source_dim = depth - p.num_sources = n_sources - merger = p.Instantiate() - - ctxs = [tf.expand_dims(i, 2) for i in ctxs] - ctx = tf.squeeze(merger.FProp(merger.theta, ctxs), 2) - self.evaluate(tf.global_variables_initializer()) - actual_ctx = self.evaluate(ctx) - - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [[ 2.66666675, 3.66666675, 4.66666698, 5.66666698], - [ 5.0, 6.0, 4.33333349, 5.33333349]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertEqual(actual_ctx.shape, (batch, depth)) - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - - def testMergerLayerGatedAvg(self): - with self.session(use_gpu=True): - np.random.seed(505837249) - depth = 4 - batch = 2 - n_sources = 3 - - inp_1 = np.asarray([[0.0, 0.0, 0.0, 0.0], [-1.0, -1.0, 1.0, 1.0]], - dtype=np.float32) - inp_2 = np.asarray([[1.0, 1.0, 1.0, 1.0], [-1.0, -1.0, 1.0, 1.0]], - dtype=np.float32) - inp_3 = np.asarray([[-1.0, -1.0, -1.0, -1.0], [-1.0, -1.0, 1.0, 1.0]], - dtype=np.float32) - p = attention.MergerLayer.Params() - p.name = 'merger_layer' - p.merger_op = 'gated_avg' - p.source_dim = depth - p.num_sources = n_sources - merger = p.Instantiate() - - ctx = merger.FProp(merger.theta, [inp_1, inp_2, inp_3]) - self.evaluate(tf.global_variables_initializer()) - actual_ctx = self.evaluate(ctx) - - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [ 0.365041, 0.365041, 0.365041, 0.365041], - [ -1.0, -1.0, 1.0 , 1.0]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertEqual(actual_ctx.shape, (batch, depth)) - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - - def testMultiSourceMultiHeadedAttention(self): - with self.session(use_gpu=True) as sess: - (source_vecs, source_contexts, source_padding, source_padding_p, - query_vec, _, _) = self._MultiHeadedAttentionInputs() - iap = attention.DotProductAttention.Params() - iap.name = 'dot_atten' - mha_params = attention.MultiHeadedAttention.Params().Set( - name='multihead_atten', - source_dim=4, - query_dim=4, - hidden_dim=4, - inner_atten_params=iap, - num_attention_heads=2, - use_source_vec_as_attention_value=False) - - # Single-source attention. - params = attention.MultiSourceAttention.Params().Set( - name='one_source_atten', - source_dim=4, - query_dim=4, - source_atten_tpls=[('src_1', mha_params)], - primary_source_key='src_1') - atten = params.Instantiate() - atten.InitForSourcePacked(atten.theta, - py_utils.NestedMap(src_1=source_vecs), - py_utils.NestedMap(src_1=source_contexts), - py_utils.NestedMap(src_1=source_padding)) - tf.global_variables_initializer().run() - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec) - self._CheckStaticShapes( - atten_vec, - atten_prob, - target_batch_size=query_vec.shape[0], - source_length=source_contexts.shape[0], - context_dim=source_contexts.shape[2]) - - atten_vec_out, prob_out = sess.run([atten_vec, atten_prob]) - print('atten_vec_out', np.sum(atten_vec_out, axis=1)) - - self.assertAllClose( - [2.8940253, 2.2901258, 3.5679011, 2.894734, 2.2989905, 3.5306041], - np.sum(atten_vec_out, axis=1)) - print('atten_vec_out', atten_vec_out) - print('prob_out', prob_out) - t_batch_size = 6 - s_batch_size = 3 - for i in range(t_batch_size): - # Test to make sure we didn't mess up indexing. - s_i = i % s_batch_size - atten.InitForSourcePacked( - atten.theta, py_utils.NestedMap(src_1=source_vecs[:, - s_i:s_i + 1, :]), - py_utils.NestedMap(src_1=source_contexts[:, s_i:s_i + 1, :]), - py_utils.NestedMap(src_1=source_padding[:, s_i:s_i + 1])) - atten_vec_i, prob_i, _ = atten.ComputeContextVector( - atten.theta, query_vec[i:i + 1]) - atten_vec_i_out, prob_i_out = sess.run([atten_vec_i, prob_i]) - self.assertAllClose(prob_i_out, prob_out[i:i + 1]) - self.assertAllClose(atten_vec_i_out, atten_vec_out[i:i + 1]) - padding_i = source_padding_p[s_i] - # Check to make sure prob exists only on valid timesteps. - self.assertEqual(0.0, np.sum(padding_i * prob_i_out)) - - # Two-source attention. - atten_merger_p = attention.MergerLayer.Params().Set( - params_init=py_utils.WeightInit.Uniform(0.04), - merger_op='concat', # concatenate attention - pre_proj_input_dims=[6, 6], - pre_proj_output_dims=[6, 6]) - params = attention.MultiSourceAttention.Params().Set( - name='two_source_atten', - source_dim=4, - query_dim=4, - source_atten_tpls=[('src_1', mha_params), - ('src_2', - mha_params.Copy().Set(name='multihead_atten2'))], - primary_source_key='src_1', - atten_merger_tpl=atten_merger_p) - atten = params.Instantiate() - - (source_vecs2, source_contexts2, source_padding2, source_padding_p, - query_vec, _, _) = self._MultiHeadedAttentionInputs() - atten.InitForSourcePacked( - atten.theta, - py_utils.NestedMap(src_1=source_vecs, src_2=source_vecs2), - py_utils.NestedMap(src_1=source_contexts, src_2=source_contexts2), - py_utils.NestedMap(src_1=source_padding, src_2=source_padding2)) - tf.global_variables_initializer().run() - atten_vec, atten_prob, _ = atten.ComputeContextVector( - atten.theta, query_vec) - self._CheckStaticShapes( - atten_vec, - atten_prob, - target_batch_size=query_vec.shape[0], - source_length=source_contexts.shape[0], - context_dim=2 * source_contexts.shape[2]) - - atten_vec_out, prob_out = sess.run([atten_vec, atten_prob]) - print('atten_vec_out', np.sum(atten_vec_out, axis=1)) - - self.assertAllClose( - [2.860059, 2.022061, 3.128138, 2.8762774, 2.103229, 3.1187325], - np.sum(atten_vec_out, axis=1)) - print('atten_vec_out', atten_vec_out) - print('prob_out', prob_out) - t_batch_size = 6 - s_batch_size = 3 - for i in range(t_batch_size): - # Test to make sure we didn't mess up indexing. - s_i = i % s_batch_size - atten.InitForSourcePacked( - atten.theta, - py_utils.NestedMap( - src_1=source_vecs[:, s_i:s_i + 1, :], - src_2=source_vecs2[:, s_i:s_i + 1, :]), - py_utils.NestedMap( - src_1=source_contexts[:, s_i:s_i + 1, :], - src_2=source_contexts2[:, s_i:s_i + 1, :]), - py_utils.NestedMap( - src_1=source_padding[:, s_i:s_i + 1], - src_2=source_padding2[:, s_i:s_i + 1])) - atten_vec_i, prob_i, _ = atten.ComputeContextVector( - atten.theta, query_vec[i:i + 1]) - atten_vec_i_out, prob_i_out = sess.run([atten_vec_i, prob_i]) - self.assertAllClose(prob_i_out, prob_out[i:i + 1]) - self.assertAllClose(atten_vec_i_out, atten_vec_out[i:i + 1]) - padding_i = source_padding_p[s_i] - # Check to make sure prob exists only on valid timesteps. - self.assertEqual(0.0, np.sum(padding_i * prob_i_out)) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/attention_util.py b/lingvo/core/attention_util.py deleted file mode 100644 index 3fbc9e9c..00000000 --- a/lingvo/core/attention_util.py +++ /dev/null @@ -1,664 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Attention related utils, e.g. relative positional embeddings.""" - -from lingvo import compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils -from lingvo.core import summary_utils - - -def ConvertToBlocks(x, block_size, padding_val=0.0): - """Turns a sequence to non overlapping blocks. - - Args: - x: a tensor of [batch, time, ...]. - block_size: int. Number of time frames in a block. - padding_val: float. value on the padded frames. - - Returns: - A tensor of [batch, num_blocks, block_size, ...], with necessary paddings, - where output[:, i, ...] are x[:, i*block_size:(i+1)*block_size, ...]. - """ - shape = py_utils.GetShape(x) - b, t = shape[:2] - if block_size < 1: - raise ValueError('block_size must be at least 1, got {}'.format(block_size)) - w = block_size - # Pad t to be a multiply of w. - num_blocks = (t + w - 1) // w - pad_to_length = num_blocks * w - padded = py_utils.PadSequenceDimension(x, pad_to_length, padding_val) - reshaped = tf.reshape(padded, [b, num_blocks, w] + shape[2:]) - return reshaped - - -def ExtractBlockContext(x, - block_size, - left_context, - right_context, - padding_val=0.0): - """Extracts temporal context for every block. - - Args: - x: a tensor of [batch, time, ...]. - block_size: int. Number of time frames in a block. - left_context: int. Left context size. - right_context: int. Right context size. - padding_val: float. value on the padded frames. - - Returns: - A tensor of [batch, num_blocks, context_size, ...], with necessary paddings, - where context_size = block_size + (left_context - 1) + right_context, - and output[:, i, ...] are x[:, start-left_context+1:end+right_context, ...], - start = i * block_size, end = (i + 1) * block_size. - """ - if block_size < 1: - raise ValueError('block_size must be at least 1, got {}'.format(block_size)) - if left_context < 1 or left_context > block_size + 1: - raise ValueError( - 'left_context must be at least 1 and at most block_size + 1 = {}, ' - 'got {}'.format(block_size + 1, left_context)) - if right_context < 0 or right_context > block_size: - raise ValueError( - 'right_context must be at least 0 and at most block_size = {}, ' - 'got {}'.format(block_size, right_context)) - - block = ConvertToBlocks(x, block_size, padding_val) - concat_list = [block] - - if left_context > 1: - if block_size == left_context - 1: - left_block = tf.roll(block, shift=1, axis=1) - else: - x_shift = tf.roll(x, shift=left_context - 1, axis=1) - x_shift_block = ConvertToBlocks(x_shift, block_size, padding_val) - left_block = x_shift_block[:, :, :left_context - 1:, ...] - concat_list = [left_block] + concat_list - - if right_context > 0: - if block_size == right_context: - right_block = tf.roll(block, shift=-1, axis=1) - else: - x_shift = tf.roll(x, shift=-right_context, axis=1) - x_shift_block = ConvertToBlocks(x_shift, block_size, padding_val) - right_block = x_shift_block[:, :, -right_context:, ...] - concat_list += [right_block] - - return tf.concat(concat_list, axis=2) - - -def MakeCausalPadding(seq_len, - block_size, - left_context, - right_context, - dtype=tf.float32): - """Makes the causal padding tensor for a full sequence. - - Args: - seq_len: int or scalar int tensor. Sequence length. - block_size: int. Number of time frames in a block. - left_context: int. Left context size. - right_context: int. Right context size. - dtype: tf.dtype, default is tf.float32. - - Returns: - A tensor of [num_blocks, block_size, context_size] taking values in {0, 1}, - where context_size = block_size + (left_context - 1) + right_context. - Element b, i, j is zero if in the b-th block, the i-th frame can access - the j-th frame in the context. - """ - seq_len = py_utils.with_dependencies([ - py_utils.assert_greater_equal( - seq_len, 1, message='seq_len must be at least 1') - ], seq_len) - - num_blocks = (seq_len + block_size - 1) // block_size - context_size = block_size + (left_context - 1) + right_context - - # [num_blocks, block_size]: source positions in the original sequence. - src_positions = tf.reshape( - tf.range(num_blocks * block_size), [num_blocks, block_size]) - # [num_blocks,]: source positions at the start of each block. - block_start_positions = tf.range(0, num_blocks * block_size, block_size) - # [context_size]: positions relative to the block start. - relative_context_positions = tf.range(context_size) - (left_context - 1) - - # [num_blocks, context_size]: target positions in the original sequence. - tgt_positions = ( - block_start_positions[:, tf.newaxis] + - relative_context_positions[tf.newaxis, :]) - # [num_blocks, block_size, context_size]: position differences between source- - # target pairs. - position_diff = src_positions[:, :, tf.newaxis] - tgt_positions[:, - tf.newaxis, :] - # [num_blocks, block_size, context_size]: if attention is allowed between - # source-target pairs. - valid_atten = tf.math.logical_and(-right_context <= position_diff, - position_diff < left_context) - - # [num_blocks, block_size]: if the source position is valid, not padded. - valid_src = src_positions < seq_len - # [num_blocks, context_size]: if the target position is valid, not padded. - valid_tgt = tf.math.logical_and(0 <= tgt_positions, tgt_positions < seq_len) - - valid_atten &= tf.math.logical_and(valid_src[:, :, tf.newaxis], - valid_tgt[:, tf.newaxis, :]) - - padding = 1.0 - tf.cast(valid_atten, dtype=dtype) - - return padding - - -def RelShift(x): - """Performs relative shift on 4D tensor (first 2 axis are batching dims). - - Given input of shape [?, ?, W, W], this does "relative shifting" for the - last two dims, s.t. output[b, n, i, j] = 0 if i > j else input[b, n, i, j-i] - - Args: - x: A Tensor of shape [?, ?, W, W] - - Returns: - A Tensor of the same shape as input with its content shifted (as described - above). - """ - b, n, w, _ = py_utils.GetShape(x) - x = py_utils.HasShape(x, [-1, -1, w, w]) - x = tf.pad(x, ((0, 0), (0, 0), (0, 0), (0, 1))) - x = tf.reshape(x, [b, n, w + 1, w]) - x = x[:, :, :w, :] - return x - - -def RelPositionBias(content, abs_pos_emb, skip_term_b=False): - """Compute relative position bias. - - This is a subroutine used by variants of self-attentions with relative - positional embedding. - - B: batch size - T: sequence length - N: num of attention heads. - H: per-head attention dimension. - - output[b][n][i][j] = content[b][i][n] x abs_pos_emb[i-j+T-1][n] - - Notice padding is supposed to be masked by the caller of this function. - - Args: - tensors of the following shapes: - content: [N, H] if skip_term_b else [B, T, N, H] - abs_pos_emb: [2T - 1, N, H], the absolute positional embedding. - abs_pos_emb[i] is the emb of relative distance i - (T-1). - skip_term_b: If to skip term_b in section 3.3 equation. - - Returns: - The attention logits tensor. [N, T, T] if skip_term_b else [B, N, T, T]. - """ - - if not skip_term_b: - b, t, n, h = py_utils.GetShape(content) - l = 2 * t - 1 - abs_pos_emb = py_utils.HasShape(abs_pos_emb, [l, n, h]) - else: - n, h = py_utils.GetShape(content) - l = py_utils.GetShape(abs_pos_emb)[0] - t = (l + 1) // 2 - - if not skip_term_b: - # [B, N, T, L=2T-1] - term_bd = tf.einsum('BTNH,LNH->BNTL', content, abs_pos_emb) - term_bd = tf.reshape(term_bd, [b, n, t * l], name='flatten') - # [B, N, T * (L + 1)]. - term_bd = tf.pad(term_bd, ((0, 0), (0, 0), (0, t))) - # [B, N, T, L + 1]. - term_bd = tf.reshape(term_bd, [b, n, t, l + 1], name='restore') - return term_bd[:, :, :, t - 1::-1] - else: - # [N, L=2T-1] - term_d = tf.einsum('NH,LNH->NL', content, abs_pos_emb) - # [N, T, L] - term_d = tf.tile(tf.expand_dims(term_d, axis=1), [1, t, 1], name='tile') - term_d = tf.reshape(term_d, [n, t * l]) - # [N, T * (L + 1)]. - term_d = tf.pad(term_d, ((0, 0), (0, t))) - # [N, T, L + 1]. - term_d = tf.reshape(term_d, [n, t, l + 1], name='restore') - return term_d[:, :, t - 1::-1] - - -def _AttenLogits(query, - key, - abs_pos_emb, - content_bias=None, - positional_bias=None, - skip_term_b=False): - """Attention logits from ... - - Transformer-XL(https://arxiv.org/pdf/1901.02860.pdf, section 3.3) version of - self attention with relative position embedding. - - Notice padding is supposed to be masked by the caller of this function. - - B: batch size - T: sequence length - N: num of attention heads. - H: per-head attention dimension. - - Args: - tensors of the following shapes: - query: [B, T, N, H] - key: [B, T, N, H] - abs_pos_emb: [2T - 1, N, H]. The sinusoid positional embedding from - https://arxiv.org/abs/1706.03762. abs_pos_emb[i] is the emb of relative - distance i - (T-1). - content_bias: [N, H] or None - positional_bias: [N, H] or None - skip_term_b: If to skip term_b in section 3.3 equation. - - Returns: - The attention logits tensor. [B, N, T, T] - """ - b, t, n, h = py_utils.GetShape(query) - - key = py_utils.HasShape(key, [b, t, n, h]) - if content_bias is not None: - content_bias = py_utils.HasShape(content_bias, [n, h]) - else: - content_bias = 0 - if positional_bias is not None: - positional_bias = py_utils.HasShape(positional_bias, [n, h]) - else: - positional_bias = 0 - - # [B, N, T, S=T] - with tf.name_scope('term_ac'): - term_ac = tf.einsum('BTNH,BSNH->BNTS', query + content_bias, key) - with tf.name_scope('term_bd'): - if skip_term_b: - content = positional_bias - else: - content = query + positional_bias - term_bd = RelPositionBias(content, abs_pos_emb, skip_term_b) - return term_ac + term_bd - - -def AttenLogitsTransformerXL(query, - key, - abs_pos_emb, - content_bias, - positional_bias, - skip_term_b=False): - """Attention logits from ... - - Transformer-XL(https://arxiv.org/pdf/1901.02860.pdf, section 3.3) version of - self attention with relative position embedding. - - Notice padding is supposed to be masked by the caller of this function. - - B: batch size - T: sequence length - N: num of attention heads. - H: per-head attention dimension. - - Args: - tensors of the following shapes: - query: [B, T, N, H] - key: [B, T, N, H] - abs_pos_emb: [2T - 1, N, H]. The sinusoid positional embedding from - https://arxiv.org/abs/1706.03762. abs_pos_emb[i] is the emb of relative - distance i - (T-1). - content_bias: [N, H] - positional_bias: [N, H] - skip_term_b: If to skip term_b in section 3.3 equation. - - Returns: - The attention logits tensor. [B, N, T, T] - """ - return _AttenLogits(query, key, abs_pos_emb, content_bias, positional_bias, - skip_term_b) - - -def AttenLogitsRPE(query, key, abs_pos_emb): - """Attention logits from ... - - https://arxiv.org/pdf/1803.02155.pdf with trainable rel position emb. - - Notice padding is supposed to be masked by the caller of this function. - - B: batch size - T: sequence length - N: num of attention heads. - H: per-head attention dimension. - - Args: - tensors of the following shapes: - query: [B, T, N, H] - key: [B, T, N, H] - abs_pos_emb: [2T - 1, N, H]. The trainable embdding. abs_pos_emb[i] is - the emb of relative distance i - (T-1). - - Returns: - The attention logits tensor. [B, N, T, T] - """ - return _AttenLogits(query, key, abs_pos_emb) - - -class KMeansClusteringForAtten(base_layer.BaseLayer): - """Implements k-means clustering with mini-batch updates. - - This is used in the implementation of https://arxiv.org/pdf/2003.05997. - - We use the following capital letters to denote shape parameters: - B = batch size - L = length of the input sequence (referred to as S or T elsewhere) - N = number of attention heads - H = dimensions of each attention head - K = number of clusters - """ - - @classmethod - def Params(cls): - """Params.""" - p = super().Params() - p.Define( - 'num_clusters', 0, 'Number of clusters, typically around the square' - ' root of the sequence length.') - p.Define('num_heads', 1, 'Num of attention heads.') - p.Define('dim_per_head', 0, 'Dimensions of each attention head.') - p.Define('decay', 0.999, 'The decay with which to update centroids.') - p.Define('epsilon', 1e-6, 'Tiny value to guard against divide by 0.') - p.Define( - 'apply_layer_norm', True, 'Whether to apply LayerNorm() on the ' - 'inputs first. If unset, caller must normalize first.') - return p - - def __init__(self, params): - """Constructs an instance which tracks its own set of centroids.""" - super().__init__(params) - p = self.params - assert p.num_clusters - assert p.dim_per_head - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - # The per-head centroids. Shape [N, K, H]. - means = py_utils.WeightParams( - shape=[p.num_heads, p.num_clusters, p.dim_per_head], - init=py_utils.WeightInit.Gaussian(), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('means', means) - - @classmethod - def LayerNorm(cls, x, epsilon=1e-6): - """Performs layer normalization on the last dimension of 'x'. - - This differs from layers.LayerNorm in that it fixes both scale and bias at - 0. - - Args: - x: An input tensor to be normalized. - epsilon: Tiny value used to guard against rsqrt of 0. - - Returns: - 'x' with its last dimension normalized. - """ - counts, means_ss, variance_ss, _, = tf.nn.sufficient_statistics( - x, axes=[-1], keepdims=True) - mean, variance = tf.nn.normalize_moments(counts, means_ss, variance_ss, - None) - return (x - mean) * tf.math.rsqrt(variance + epsilon) - - def FProp(self, theta, x, paddings=None, update=False): - """Computes distances of the given input 'x' to all centroids. - - This implementation applies layer normalization on 'x' internally first, - and the returned 'dists' is computed using the normalized 'x'. - - Args: - theta: A `.NestedMap` of weights' values of this layer. - x: A tensor of shape [B, L, N, H]. - paddings: If not None, a tensor of shape [B, L]. - update: bool, whether to update centroids using x. - - Returns: - dists: "distances" of the given input 'x' to all centroids. - Shape [B, L, N, K]. - k_means_loss: the average squared Euclidean distances to the closest - centroid, a scalar. - """ - p = self.params - if paddings is None: - paddings = tf.zeros_like(x[:, :, 0, 0]) - # Shape [B, L, 1, 1] - paddings_4d = paddings[:, :, None, None] - - if p.apply_layer_norm: - x = KMeansClusteringForAtten.LayerNorm(x, p.epsilon) - - # 'x' is normalized (but theta.means is not), we use negative dot product to - # approximate the Euclidean distance here. - dists = -tf.einsum('BLNH, NKH -> BLNK', x, theta.means) - - # For padded positions we update the distances to very large numbers. - very_large_dists = tf.ones_like(dists) * tf.constant( - 0.1, dtype=dists.dtype) * dists.dtype.max - paddings_tiled = tf.tile(paddings_4d, [1, 1, p.num_heads, p.num_clusters]) - dists = tf.where(paddings_tiled > 0.0, very_large_dists, dists) - - # Shape [B, L, N, K], the same as 'dists' above. - nearest_one_hot = tf.one_hot(tf.math.argmin(dists, axis=-1), p.num_clusters) - # Same shape as the input 'x'. - nearest_centroid = tf.einsum('BLNK, NKH -> BLNH', nearest_one_hot, - theta.means) - diff = tf.math.squared_difference(x, tf.stop_gradient(nearest_centroid)) - diff = py_utils.ApplyPadding(paddings_4d, diff) - diff = tf.math.reduce_mean(diff, axis=2) - - # The commitment loss which when back proped against encourages the 'x' - # values to commit to their chosen centroids. - k_means_loss = tf.math.reduce_sum(diff) / tf.math.reduce_sum(1.0 - paddings) - summary_utils.scalar('k_means/squared_distance_loss', k_means_loss) - - # TODO(zhouwk): investigate normalizing theta.means after each update. - means_norm = tf.norm(theta.means) - summary_utils.scalar('k_means/centroid_l2_norm/min', - tf.math.reduce_min(means_norm)) - summary_utils.scalar('k_means/centroid_l2_norm/mean', - tf.math.reduce_mean(means_norm)) - - if not update: - return dists, k_means_loss - - # To update the centroids (self.vars.means), we apply gradient descent on - # the mini-batch of input 'x', which yields the following: - # new_centroid = centroid + (1 - decay) * (x_mean - centroid) - # where x_mean is the average over all the input vectors closest to this - # centroid. - # - # Note that this approach is equivalent with backprop via - # loss = tf.math.reduce_mean( - # tf.math.squared_difference(tf.stop_gradient(x), nearest_centroid))) - # , except that here the learning rate is independently set via 'decay'. - - # Ensure that the padded positions are not used to update the centroids. - nearest_one_hot = py_utils.ApplyPadding(paddings_4d, nearest_one_hot) - - # Sum away batch and sequence length dimensions to get per cluster count. - # Shape: [N, K] - per_cluster_count = tf.reduce_sum(nearest_one_hot, axis=[0, 1]) - summary_utils.histogram('k_means/per_cluster_vec_count', per_cluster_count) - - # Sum of the input 'x' per each closest centroid. - sum_x = tf.einsum('BLNK, BLNH -> NKH', nearest_one_hot, x) - - if py_utils.use_tpu(): - per_cluster_count = tf.tpu.cross_replica_sum(per_cluster_count) - sum_x = tf.tpu.cross_replica_sum(sum_x) - - # If per_cluster_count for a cluster is 0, then 'nearest_one_hot' in that - # cluster's position will always be 0, hence 'sum_x' in that dimension will - # be 0. - new_means = sum_x / tf.maximum( - tf.constant(1.0, dtype=per_cluster_count.dtype), - tf.expand_dims(per_cluster_count, axis=-1)) - - # We use exponential moving average. TODO(zhouwk): investigate smooth this - # over an exponentially moving averaged per cluster count. - # - # Note that we intentionally do not normalize the means after this update - # as empirically this works better. - update_means_diff = (1.0 - p.decay) * (new_means - theta.means) - return py_utils.with_dependencies( - [tf.assign_add(self.vars.means, update_means_diff)], - dists), k_means_loss - - -def ComputeSparseAttention(q, k, v, sparsity_indices, paddings=None): - """Computes attention according to a sparsity pattern. - - We use the following capital letters to denote shape parameters: - B = batch size - S = length of the source sequence - T = length of the target sequence - N = number of attention heads - H = dimensions of each attention head - K = number of clusters - W = attention window (K <= S) - - The 'sparsity_indices' is a tensor of integral type where the last dimension - contains W indices (W is the attention window) for each corresponding position - along S in 'k' that the query is allowed to attend to. - - For example, if sparsity_indices[batch_idx, target time step, head_idx] = - [1, 7, 8], it means that token in the query attends to values with indices - 1, 7, and 8, and the attention window here is 3. - - The valid values in 'sparsity_indices' are [-1, S-1]. Note that the value -1 - is reserved to mean paddings, distinct from the value (S-1). - - For example, if W=S and 'sparsity_indices' contains range(S) on the last - dimension, this degenerates to the original full attention. - - We require that 'sparsity_indices' does not contain duplicates (except for -1 - to indicate paddings), but we do not require 'sparsity_indices' to be sorted. - - Note that this implementation is flexible and geneic but is not optimized for - time or space complexity. Please consider grouping queries that attend to the - same subset of values first for efficiency. - - Args: - q: (projected) queries, [B, T, N, H]; - k: (projected) keys, [B, S, N, H]; - v: (projected) values, [B, S, N, H]; - sparsity_indices: [B, T, N, W], where W is the attention window; - paddings: paddings for keys, [B, S] if not None. - - Returns: - output: the encoded output, [B, T, N, H]. - atten_probs: the attention weights, [B, T, N, S]. - """ - q = tf.convert_to_tensor(q) - k = tf.convert_to_tensor(k) - v = tf.convert_to_tensor(v) - sparsity_indices = tf.convert_to_tensor(sparsity_indices) - - k = py_utils.HasRank(k, 4) - _, source_length, _, dim_per_head = py_utils.GetShape(k, 4) - sparsity_indices = py_utils.HasRank(sparsity_indices, 4) - batch_size, target_length, num_heads, attention_window = py_utils.GetShape( - sparsity_indices, 4) - py_utils.assert_less_equal( - attention_window, source_length, - 'The provided sparsity_indices has attention window ' - ' > source length. This is likely an error.') - - # To prepare for gathering the relevant vectors from 'k', we prepare - # gather_idx of shape [B, T, N, W, 3] where the last dimension corresponds to - # slices in 'k' indexed by (batch index, source time step, head index), - # where the source length index comes from the original W dimension in - # 'sparsity_indices'. - seq_idx = tf.expand_dims(sparsity_indices, axis=-1) - # Overwrite the paddings -1 with valid gather indices (zeros). We will - # fix the logits with -inf in these positions later. - seq_idx = tf.where(seq_idx < 0, tf.zeros_like(seq_idx), seq_idx) - batch_idx = tf.reshape( - tf.range(0, batch_size, dtype=sparsity_indices.dtype), - [batch_size, 1, 1, 1, 1]) - batch_idx = tf.tile(batch_idx, - [1, target_length, num_heads, attention_window, 1]) - head_idx = tf.reshape( - tf.range(0, num_heads, dtype=sparsity_indices.dtype), - [1, 1, num_heads, 1, 1]) - head_idx = tf.tile(head_idx, - [batch_size, target_length, 1, attention_window, 1]) - # [B, T, N, W, 3], where last dimension is (batch index, source length index, - # head index). - gather_idx = tf.concat([batch_idx, seq_idx, head_idx], axis=-1) - - # Both the gathered k and v have shape [B, T, N, W, H] - k = tf.gather_nd(k, gather_idx) - v = tf.gather_nd(v, gather_idx) - - if paddings is None: - paddings = tf.zeros([batch_size, source_length]) - paddings = tf.convert_to_tensor(paddings) - paddings = tf.expand_dims(paddings, axis=-1) - # [B, S, N] - paddings = tf.tile(paddings, [1, 1, num_heads]) - # [B, T, N, W] - paddings = tf.gather_nd(paddings, gather_idx) - - logits = tf.einsum('BTNH, BTNWH -> BTNW', q, k) - logits *= tf.math.rsqrt(tf.cast(dim_per_head, q.dtype)) - - very_negative_logits = ( - tf.ones_like(logits) * logits.dtype.max * - tf.constant(-0.7, dtype=logits.dtype)) - padded_logits = tf.where( - tf.math.logical_or(sparsity_indices < 0, paddings > 0.0), - very_negative_logits, logits) - - # [B, T, N, W] - atten_probs = tf.nn.softmax(padded_logits, name='attention_weights') - atten_probs = tf.where(sparsity_indices < 0, tf.zeros_like(logits), - atten_probs) - output = tf.einsum('BTNW, BTNWH -> BTNH', atten_probs, v) - - # Scatter 'atten_probs' back into the original source length. - # [B, T, N, W, 1] - batch_idx = tf.tile( - tf.range(batch_size)[:, None, None, None, None], - [1, target_length, num_heads, attention_window, 1]) - # [B, T, N, W, 1] - target_seq_idx = tf.tile( - tf.range(target_length)[None, :, None, None, None], - [batch_size, 1, num_heads, attention_window, 1]) - # [B, T, N, W, 1] - head_idx = tf.tile( - tf.range(num_heads)[None, None, :, None, None], - [batch_size, target_length, 1, attention_window, 1]) - # seq_idx: [B, T, N, W, 1] - # [B, T, N, W, 4] - scatter_idx = tf.concat([batch_idx, target_seq_idx, head_idx, seq_idx], -1) - # [B, T, N, S] - scattered_probs = tf.scatter_nd( - scatter_idx, atten_probs, - [batch_size, target_length, num_heads, source_length]) - return output, scattered_probs diff --git a/lingvo/core/attention_util_test.py b/lingvo/core/attention_util_test.py deleted file mode 100644 index b1507574..00000000 --- a/lingvo/core/attention_util_test.py +++ /dev/null @@ -1,511 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for attention_util.""" - -from absl.testing import parameterized - -from lingvo import compat as tf -from lingvo.core import attention_util -from lingvo.core import test_utils - -import numpy as np - -FLAGS = tf.flags.FLAGS - - -class RelPositionBiasTest(test_utils.TestCase, parameterized.TestCase): - - def testBasic(self): - with self.session(): - t = 3 - # [BTNH]. - content = tf.linalg.diag(tf.ones([t]))[None, :, None, :] - # [LNH]. - abs_pos_emb = tf.reshape( - tf.range(t * (2 * t - 1), dtype=tf.float32), [2 * t - 1, 1, t]) - tf.logging.info('content=%s abs_pos_emb=%s', content.eval(), - abs_pos_emb.eval()) - self.assertAllClose([[[[6., 3., 0.], [10., 7., 4.], [14., 11., 8.]]]], - attention_util.RelPositionBias(content, - abs_pos_emb).eval()) - - -def OracleAttentionLogits(query, - key, - abs_pos_emb, - content_bias, - positional_bias, - skip_term_b=False): - """Computes expected attention logits using non-vectorized approach.""" - batch, seqlen, num_heads, _ = query.shape - tgtlen, srclen = seqlen, seqlen - - logits = np.zeros((batch, num_heads, tgtlen, srclen)) - - for b in range(batch): - for n in range(num_heads): - for i in range(tgtlen): - for j in range(srclen): - offset = seqlen - 1 - pos_emb = abs_pos_emb[i - j + offset] - logits[b][n][i][j] = np.dot(query[b][i][n], key[b][j][n]) - if not skip_term_b: - logits[b][n][i][j] += np.dot(query[b][i][n], pos_emb[n]) - if content_bias is not None: - logits[b][n][i][j] += np.dot(content_bias[n], key[b][j][n]) - if positional_bias is not None: - logits[b][n][i][j] += np.dot(positional_bias[n], pos_emb[n]) - return logits - - -class TransformerXLRelativeAttentionTest(test_utils.TestCase, - parameterized.TestCase): - - def setUp(self): - super().setUp() - self.input_dim = 32 - self.num_heads = 4 - self.batch = 4 - self.seqlen = 16 - - def _GetTestInputs(self): - np.random.seed(FLAGS.test_random_seed) - query = 3 * np.random.rand(self.batch, self.seqlen, self.num_heads, - self.input_dim).astype(np.float32) - key = 5 * np.random.rand(self.batch, self.seqlen, self.num_heads, - self.input_dim).astype(np.float32) - abs_pos_emb = 7 * np.random.rand(2 * self.seqlen - 1, self.num_heads, - self.input_dim).astype(np.float32) - content_bias = 11 * np.random.rand(self.num_heads, self.input_dim).astype( - np.float32) - positional_bias = 13 * np.random.rand(self.num_heads, - self.input_dim).astype(np.float32) - return query, key, abs_pos_emb, content_bias, positional_bias - - @parameterized.named_parameters( - ('Base', False), - ('Lite', True), - ) - def testTransformerXL(self, skip_term_b): - (query, key, abs_pos_emb, content_bias, - positional_bias) = self._GetTestInputs() - expected = OracleAttentionLogits(query, key, abs_pos_emb, content_bias, - positional_bias, skip_term_b) - actual_t = attention_util.AttenLogitsTransformerXL(query, key, abs_pos_emb, - content_bias, - positional_bias, - skip_term_b) - with self.session() as sess: - actual = sess.run(actual_t) - self.assertAllClose(expected, actual) - - def testRPE(self): - (query, key, abs_pos_emb, _, _) = self._GetTestInputs() - expected = OracleAttentionLogits(query, key, abs_pos_emb, None, None) - actual_t = attention_util.AttenLogitsRPE(query, key, abs_pos_emb) - with self.session() as sess: - actual = sess.run(actual_t) - self.assertAllClose(expected, actual) - - -class BlockUtilsTest(test_utils.TestCase, parameterized.TestCase): - - @parameterized.named_parameters( - ('single_block', 7), - ('one_frame_block', 1), - ('two_frame_blocks', 2), - ) - def testConvertToBlocks(self, block_size): - x_val = np.random.random([2, 6, 2, 3, 4]) - with self.session() as sess: - x = tf.convert_to_tensor(x_val, tf.float32) - x_blocks = attention_util.ConvertToBlocks(x, block_size) - x_blocks_val = sess.run(x_blocks) - # Check shape. - batch_size = x_val.shape[0] - other_dims = x_val.shape[2:] - num_blocks = int(np.ceil(x_val.shape[1] / float(block_size))) - expected_shape = (batch_size, num_blocks, block_size) + other_dims - self.assertAllEqual(expected_shape, x_blocks_val.shape) - - # Check values. - x_recover = x_blocks_val.reshape((x_blocks_val.shape[0], -1) + - x_blocks_val.shape[3:]) - x_recover = x_recover[:, :x_val.shape[1], ...] - self.assertAllClose(x_val, x_recover) - - @parameterized.named_parameters( - ('single_block', 7, 2, 1), - ('single_frame_context', 1, 1, 0), - ('other_case_1', 3, 4, 1), - ('other_case_2', 4, 2, 4), - ) - def testExtractBlockContext(self, block_size, left_context, right_context): - x_val = np.random.random([2, 6, 2, 3, 4]) - with self.session() as sess: - x = tf.convert_to_tensor(x_val, tf.float32) - x_context = attention_util.ExtractBlockContext(x, block_size, - left_context, - right_context) - x_context_val = sess.run(x_context) - # Check shape. - batch_size = x_val.shape[0] - other_dims = x_val.shape[2:] - num_blocks = int(np.ceil(x_val.shape[1] / float(block_size))) - context_size = block_size + left_context - 1 + right_context - expected_shape = (batch_size, num_blocks, context_size) + other_dims - self.assertAllEqual(expected_shape, x_context_val.shape) - - # Check values block by block. - for block_idx in range(num_blocks): - context_start = block_idx * block_size - left_context + 1 - context_end = (block_idx + 1) * block_size + right_context - slice_start = max(0, context_start) - slice_end = min(x_val.shape[1], context_end) - expected_val = x_val[:, slice_start:slice_end, ...] - actual_val = x_context_val[:, block_idx, ...] - # remove paddings - front_padding = slice_start - context_start - back_padding = context_end - slice_end - actual_val = actual_val[:, front_padding:context_size - back_padding, ...] - self.assertAllClose(expected_val, actual_val) - - def _getReferenceCausalPadding(self, seq_len, block_size, left_context, - right_context): - num_blocks = int(np.ceil(seq_len / float(block_size))) - context_size = block_size + left_context - 1 + right_context - padding = np.ones((num_blocks, block_size, context_size)) - - for i in range(num_blocks): - for j in range(block_size): - actual_src_pos = j + i * block_size - if actual_src_pos < seq_len: - for k in range(context_size): - actual_tgt_pos = k + i * block_size - (left_context - 1) - if 0 <= actual_tgt_pos and actual_tgt_pos < seq_len: - diff = actual_src_pos - actual_tgt_pos - if -right_context <= diff and diff < left_context: - padding[i, j, k] = 0 - - return padding - - @parameterized.named_parameters( - ('single_block', 6, 9, 2, 1), - ('single_frame_block', 6, 1, 2, 1), - ('single_frame_context', 6, 1, 1, 0), - ('other_case_1', 6, 3, 4, 1), - ('other_case_2', 6, 4, 2, 4), - ) - def testMakeCausalPadding(self, seq_len, block_size, left_context, - right_context): - with self.session() as sess: - seq_len_t = tf.convert_to_tensor(seq_len) - padding = attention_util.MakeCausalPadding(seq_len_t, block_size, - left_context, right_context) - padding_val = sess.run(padding) - - ref_padding = self._getReferenceCausalPadding(seq_len, block_size, - left_context, right_context) - self.assertAllEqual(ref_padding, padding_val) - - -class KMeansClusteringForAttenTest(test_utils.TestCase): - - def testFProp(self): - p = attention_util.KMeansClusteringForAtten.Params() - p.name = 'k_means' - p.num_clusters = 2 - p.dim_per_head = 4 - p.num_heads = 3 - batch_size = 5 - seq_length = 6 - x = np.random.rand(batch_size, seq_length, p.num_heads, - p.dim_per_head).astype(np.float32) - k_means = p.Instantiate() - - with self.session(): - dists, loss = k_means.FProp(k_means.theta, x, update=True) - self.evaluate(tf.global_variables_initializer()) - dists, loss = self.evaluate([dists, loss]) - self.assertEqual(dists.shape, - (batch_size, seq_length, p.num_heads, p.num_clusters)) - self.assertEqual(loss.shape, ()) - - def testFPropFixedInput(self): - p = attention_util.KMeansClusteringForAtten.Params() - p.name = 'k_means' - p.num_clusters = 3 - p.dim_per_head = 6 - p.num_heads = 4 - p.decay = 0.5 - k_means = p.Instantiate() - batch_size = 2 - seq_length = 5 - - with self.session(): - x = np.random.rand(batch_size, seq_length, p.num_heads, - p.dim_per_head).astype(np.float32) - self.evaluate(tf.global_variables_initializer()) - fixed_loss = None - for _ in range(10): - dists, loss = k_means.FProp(k_means.theta, x, update=False) - dists, loss = self.evaluate([dists, loss]) - if not fixed_loss: - fixed_loss = loss - else: - # If we do not update, the loss remain fixed. - self.assertEqual(loss, fixed_loss) - prev_loss = fixed_loss - self.evaluate(k_means.FProp(k_means.theta, x, update=True)) - for _ in range(5): - dists, loss = k_means.FProp(k_means.theta, x, update=True) - _, loss = self.evaluate([dists, loss]) - # If we update the centroids, the loss should strictly decrease. - self.assertGreater(prev_loss - loss, 1e-5) - prev_loss = loss - - def testFPropClustering(self): - p = attention_util.KMeansClusteringForAtten.Params() - p.name = 'k_means' - p.num_clusters = 2 - p.dim_per_head = 3 - p.num_heads = 2 - p.decay = 0.8 - k_means = p.Instantiate() - batch_size = 3 - seq_length = 3 - - with self.session() as sess: - self.evaluate(tf.global_variables_initializer()) - - def _GenInput(): - # We randomly generate inputs such that head 0 is clustered - # around (±1/√2, ±1/√2, ∓√2), while head 1 is clustered around - # (∓-√2, ±1/√2, ±1/√2). - noise = 0.05 * np.random.rand(batch_size, seq_length, p.num_heads, - p.dim_per_head).astype(np.float32) - x1 = np.random.binomial(1, 0.5, [batch_size, seq_length, 1, 1]) * 2 - 1 - x1 = np.tile( - np.array([1., 1., -1.], dtype=np.float32), - [batch_size, seq_length, 1, 1]) * x1 - x2 = np.random.binomial(1, 0.5, [batch_size, seq_length, 1, 1]) * 2 - 1 - x2 = np.tile( - np.array([-1., 1., 1.], dtype=np.float32), - [batch_size, seq_length, 1, 1]) * x2 - x = np.concatenate([x1, x2], axis=2) + noise - return x.astype(np.float32) - - for _ in range(25): - _, loss = sess.run( - k_means.FProp(k_means.theta, _GenInput(), update=True)) - final_means = k_means.theta.means.eval() - # We assert that the centroids are close to the true centers. - self.assertAllClose( - np.abs(final_means), [[[0.71, 0.71, 1.41], [0.71, 0.71, 1.41]], - [[1.41, 0.71, 0.71], [1.41, 0.71, 0.71]]], - rtol=0.03, - atol=0.03) - self.assertLess(loss, 0.005) - - def testFPropPadding(self): - p = attention_util.KMeansClusteringForAtten.Params() - p.name = 'k_means' - p.num_clusters = 2 - p.dim_per_head = 3 - p.num_heads = 1 - p.decay = 0.7 - k_means = p.Instantiate() - batch_size = 3 - seq_length = 5 - - with self.session() as sess: - self.evaluate(tf.global_variables_initializer()) - - def _GenInput(): - # We randomly generate inputs such that inputs are clustered - # around (±1/√2, ±1/√2, ∓√2) or (∓-√2, ±1/√2, ±1/√2) with one of them - # hidden by padding. - paddings = np.random.binomial(1, 0.5, [batch_size, seq_length]).astype( - np.float32) - x = np.expand_dims(np.expand_dims(paddings, axis=-1), axis=-1) - # When padding is 0, we generate (∓1, 1, ±1); when padding is 1, we - # generate (±1, 1, ∓1). - x = np.concatenate([2 * x - 1, np.ones_like(x), 1 - 2 * x], axis=-1) - x *= np.random.binomial(1, 0.5, [batch_size, seq_length, 1, 1]) * 2 - 1 - return x, paddings - - for _ in range(30): - x, paddings = _GenInput() - self.assertEqual(x.shape, - (batch_size, seq_length, p.num_heads, p.dim_per_head)) - self.assertEqual(paddings.shape, (batch_size, seq_length)) - _, loss1 = sess.run( - k_means.FProp(k_means.theta, x, paddings, update=True)) - means1 = k_means.theta.means.eval() - - # We reverse the padding to hide the other half. - for _ in range(40): - x, paddings = _GenInput() - _, loss2 = sess.run( - k_means.FProp(k_means.theta, x, 1.0 - paddings, update=True)) - means2 = k_means.theta.means.eval() - - # We compute the loss using the previous input centering on - # different centroids. The squared distance should be 3. - _, loss3 = sess.run( - k_means.FProp(k_means.theta, x, paddings, update=False)) - - self.assertAllClose( - np.abs(means1), [[[1.41, 0.71, 0.71], [1.41, 0.71, 0.71]]], - rtol=0.03, - atol=0.03) - self.assertLess(loss1, 1e-5) - self.assertAllClose( - np.abs(means2), [[[0.71, 0.71, 1.41], [0.71, 0.71, 1.41]]], - rtol=0.03, - atol=0.03) - self.assertLess(loss2, 1e-5) - - self.assertAllClose(loss3, 3.0, 1e-4, 1e-4) - - def testFPropClusteringEmptyCluster(self): - p = attention_util.KMeansClusteringForAtten.Params() - p.name = 'k_means' - p.num_clusters = 10 - p.dim_per_head = 3 - p.num_heads = 1 - p.decay = 0.8 - k_means = p.Instantiate() - batch_size = 3 - seq_length = 4 - - # All of our inputs are (1, 1, -1) - x = np.ones([batch_size, seq_length, 1, 1], dtype=np.float32) - x = np.concatenate([x, x, -x], axis=-1) - with self.session() as sess: - self.evaluate(tf.global_variables_initializer()) - for _ in range(30): - dists, loss = sess.run(k_means.FProp(k_means.theta, x, update=True)) - means = k_means.theta.means.eval() - idx = np.argmin(dists, axis=-1) - idx_1 = idx[0, 0, 0] - # We assert that 'dists' achieves minimum all at the same cluster. - self.assertAllEqual(idx, - np.array(idx_1 * np.ones([batch_size, seq_length, 1]))) - # We assert that at this index the centroid is close to (1/√2, 1/√2, -√2). - means = np.squeeze(means[:, idx_1, :]) - self.assertAllClose(means, [0.71, 0.71, -1.41], rtol=0.03, atol=0.03) - self.assertLess(loss, 1e-4) - - -class ComputeSparseAttention(test_utils.TestCase): - - def testBasics(self): - batch_size = 3 - source_length = 5 - target_length = 4 - num_heads = 2 - dim_per_head = 3 - q = np.random.rand(batch_size, target_length, num_heads, - dim_per_head).astype(np.float32) - k = np.random.rand(batch_size, source_length, num_heads, - dim_per_head).astype(np.float32) - v = np.random.rand(batch_size, source_length, num_heads, - dim_per_head).astype(np.float32) - # attention window = 2 - sparsity_indices = np.concatenate([ - np.zeros([batch_size, target_length, num_heads, 1], dtype=np.int32), - np.ones([batch_size, target_length, num_heads, 1], dtype=np.int32), - ], - axis=-1) - - with self.session() as sess: - out, probs = sess.run( - attention_util.ComputeSparseAttention(q, k, v, sparsity_indices)) - self.assertEqual(out.shape, - (batch_size, target_length, num_heads, dim_per_head)) - self.assertEqual(probs.shape, - (batch_size, target_length, num_heads, source_length)) - # attention weights sum to 1. - self.assertAllClose( - np.sum(probs, axis=-1), - np.ones([batch_size, target_length, num_heads])) - - # attention window = 4, but last two are always paddings. - sparsity_indices = np.concatenate([ - sparsity_indices, - -np.ones([batch_size, target_length, num_heads, 2], dtype=np.int32), - ], - axis=-1) - with self.session() as sess: - out2, probs2 = sess.run( - attention_util.ComputeSparseAttention(q, k, v, sparsity_indices)) - # We assert that the encoded outputs are the same as before, - # and the attention weights are 0 on the padded positions. - self.assertAllClose(out, out2) - self.assertAllClose(probs, probs2) - - # attention window = 4. - sparsity_indices = np.tile( - np.arange(4, dtype=np.int32), [batch_size, target_length, num_heads, 1]) - # but position 2 and 3 are padded. - paddings = np.tile([0., 0., 1., 1., 0.], [batch_size, 1]) - with self.session() as sess: - out3, probs3 = sess.run( - attention_util.ComputeSparseAttention(q, k, v, sparsity_indices, - paddings)) - # We assert that the encoded outputs and attention weights are the same - # as before. - self.assertAllClose(out2, out3) - self.assertAllClose(probs2, probs3) - - def testFullAttention(self): - batch_size = 4 - source_length = 7 - target_length = 6 - num_heads = 3 - dim_per_head = 5 - q = np.random.rand(batch_size, target_length, num_heads, - dim_per_head).astype(np.float32) - k = np.random.rand(batch_size, source_length, num_heads, - dim_per_head).astype(np.float32) - v = np.random.rand(batch_size, source_length, num_heads, - dim_per_head).astype(np.float32) - # attention window = source length, randomly permutated - # np.arange(source_length) - sparsity_indices = np.tile( - np.random.permutation(source_length).astype(np.int32), - [batch_size, target_length, num_heads, 1]) - - with self.session() as sess: - out, probs = sess.run( - attention_util.ComputeSparseAttention(q, k, v, sparsity_indices)) - - # compute full attention in numpy - expected_logit = np.einsum('BTNH, BSNH -> BTNS', q, k) - expected_logit /= np.sqrt(dim_per_head) - elexp = np.exp(expected_logit) - expected_probs = elexp / np.expand_dims(np.sum(elexp, axis=-1), axis=-1) - expected_output = np.einsum('BTNS, BSNH -> BTNH', expected_probs, v) - - # We assert that the output is close to the full attention, - # since our sparsity_indices is range(source_length) - self.assertAllClose(probs, expected_probs) - self.assertAllClose(out, expected_output) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/base_decoder.py b/lingvo/core/base_decoder.py deleted file mode 100644 index eb434285..00000000 --- a/lingvo/core/base_decoder.py +++ /dev/null @@ -1,394 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Common decoder interface.""" - -import collections - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import beam_search_helper -from lingvo.core import py_utils -from lingvo.core import target_sequence_sampler - -# metrics: Dict[Text, Tuple[float, float]] A dict of named metrics, which must -# include 'loss'. The value of the dict is (metric_val, count), where -# metric_val is the sum of the metric over all examples, and count is the -# number of examples seen. The mean value of the metric is metric_val/count. -# This is the first output of ComputeLoss. -# predictions: Union[Tensor, Dict[Text, Tensor], NestedMap] This is the output -# of ComputePredictions. -# per_sequence: Dict[Text, Tensor] This is the second output of ComputeLoss. -DecoderOutput = collections.namedtuple( - 'DecoderOutput', - ['metrics', 'predictions', 'per_sequence'], -) - - -class BaseDecoder(base_layer.BaseLayer): - """Base class for all decoders.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'packed_input', False, 'If True, decoder and all layers support ' - 'multiple examples in a single sequence.') - return p - - @classmethod - def UpdateTargetVocabSize(cls, p, vocab_size, wpm_model=None): - """Sets the vocab size and wpm model in the params. - - Args: - p: model params. - vocab_size: size of the vocabulary. - wpm_model: file name prefix pointing to a wordpiece model. - - Returns: - Model target vocabulary params updated with the vocab size and wpm model. - """ - raise NotImplementedError('Abstract method') - - def FProp(self, theta, encoder_outputs, targets): - """Decodes `targets` given encoded source. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - encoder_outputs: a NestedMap computed by encoder. - targets: A NestedMap containing additional inputs to the decoder, - such as the targets being predicted. - - Returns: - A DecoderOutput namedtuple. - """ - predictions = self.ComputePredictions(theta, encoder_outputs, targets) - metrics, per_sequence = self.ComputeLoss(theta, predictions, targets) - return DecoderOutput( - metrics=metrics, predictions=predictions, per_sequence=per_sequence) - - def ComputePredictions(self, theta, encoder_outputs, targets): - raise NotImplementedError('Abstract method: %s' % type(self)) - - def ComputeLoss(self, theta, predictions, targets): - raise NotImplementedError('Abstract method: %s' % type(self)) - - -class BaseBeamSearchDecoder(BaseDecoder): - """Decoder that does beam search.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('target_sos_id', 1, 'Id of the target sequence sos symbol.') - p.Define('target_eos_id', 2, 'Id of the target sequence eos symbol.') - # TODO(rpang): remove target_seq_len and use beam_search.target_seq_len - # instead. - p.Define('target_seq_len', 0, 'Target seq length.') - p.Define('beam_search', beam_search_helper.BeamSearchHelper.Params(), - 'BeamSearchHelper params.') - p.Define('greedy_search', beam_search_helper.GreedySearchHelper.Params(), - 'GreedySearchHelper params.') - p.Define('target_sequence_sampler', - target_sequence_sampler.TargetSequenceSampler.Params(), - 'TargetSequenceSampler params.') - p.Define( - 'bias_only_if_consistent', True, 'BeamSearchBiased bias is only' - 'applied if the hypothesis has been consistent with targets so far.') - return p - - @classmethod - def UpdateTargetVocabSize(cls, p, vocab_size, wpm_model=None): - """Sets the vocab size and wpm model in the params. - - Args: - p: model params. - vocab_size: size of the vocabulary. - wpm_model: file name prefix pointing to a wordpiece model. - - Returns: - Model target vocabulary params updated with the vocab size and wpm model. - """ - raise NotImplementedError('Abstract method') - - def __init__(self, params): - super().__init__(params) - p = self.params - p.beam_search.target_seq_len = p.target_seq_len - p.beam_search.target_sos_id = p.target_sos_id - p.beam_search.target_eos_id = p.target_eos_id - self.CreateChild('beam_search', p.beam_search) - p.greedy_search.target_seq_len = p.target_seq_len - p.greedy_search.target_sos_id = p.target_sos_id - p.greedy_search.target_eos_id = p.target_eos_id - self.CreateChild('greedy_search', p.greedy_search) - p.target_sequence_sampler.target_seq_len = p.target_seq_len - p.target_sequence_sampler.target_sos_id = p.target_sos_id - p.target_sequence_sampler.target_eos_id = p.target_eos_id - self.CreateChild('target_sequence_sampler', p.target_sequence_sampler) - - def AddExtraDecodingInfo(self, encoder_outputs, targets): - """Adds extra decoding information to encoded_outputs. - - Args: - encoder_outputs: a NestedMap computed by encoder. - targets: a NestedMap containing target input fields. - - Returns: - encoder_ouputs with extra information used for decoding. - """ - return encoder_outputs - - def BeamSearchDecode(self, encoder_outputs, num_hyps_per_beam_override=0): - """Performs beam search based decoding. - - Args: - encoder_outputs: the outputs of the encoder. - num_hyps_per_beam_override: If set to a value <= 0, this parameter is - ignored. If set to a value > 0, then this value will be used to override - p.num_hyps_per_beam. - - Returns: - `.BeamSearchDecodeOutput`, A namedtuple whose elements are tensors. - """ - return self.BeamSearchDecodeWithTheta(self.theta, encoder_outputs, - num_hyps_per_beam_override) - - def BeamSearchDecodeWithTheta(self, - theta, - encoder_outputs, - num_hyps_per_beam_override=0): - return self.beam_search.BeamSearchDecode(theta, encoder_outputs, - num_hyps_per_beam_override, - self._InitBeamSearchStateCallback, - self._PreBeamSearchStepCallback, - self._PostBeamSearchStepCallback) - - def GreedySearchDecode(self, encoder_outputs): - """Performs beam search based decoding. - - Args: - encoder_outputs: the outputs of the encoder. - - Returns: - greedy search decode output. - """ - return self.GreedySearchDecodeWithTheta(self.theta, encoder_outputs) - - def GreedySearchDecodeWithTheta(self, theta, encoder_outputs): - return self.greedy_search.GreedySearchDecode( - theta, encoder_outputs, - self._InitBeamSearchStateCallback, - self._PreBeamSearchStepCallback, - self._PostBeamSearchStepCallback) - - def SampleTargetSequences(self, theta, encoder_outputs, random_seed): - """Performs target sequence sampling. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - encoder_outputs: a NestedMap computed by encoder. - random_seed: a scalar int32 tensor representing the random seed. - - Returns: - A NestedMap containing the following tensors - - - 'ids': [batch, max_target_length] of int32, representing the target - sequence ids, not including target_sos_id, but maybe ending with - target_eos_id if target_eos_id is sampled. - - 'paddings': [batch, max_target_length] of 0/1, where 1 represents - a padded timestep. - """ - return self.target_sequence_sampler.Sample( - theta, encoder_outputs, random_seed, self._InitBeamSearchStateCallback, - self._PreBeamSearchStepCallback, self._PostBeamSearchStepCallback) - - def BeamSearchDecodeBiased(self, - encoder_outputs, - num_hyps_per_beam_override=0): - """Performs beam-search decoding while biasing towards provided targets. - - Args: - encoder_outputs: a NestedMap computed by encoder. Must include `targets`, - which is used to bias beam search. - num_hyps_per_beam_override: If set to a value <= 0, this parameter is - ignored. If set to a value > 0, then this value will be used to override - `p.num_hyps_per_beam`. - - Returns: - BeamSearchDecodeOutput, a namedtuple containing the decode results. - """ - p = self.params - - targets = encoder_outputs.targets - targets.weights *= (1.0 - targets.paddings) - - def PadToTargetSeqLen(tensor, constant): - length = tf.shape(tensor)[1] - pad = tf.maximum(0, p.beam_search.target_seq_len - length) - return tf.pad(tensor, [[0, 0], [0, pad]], constant_values=constant) - - targets.labels = PadToTargetSeqLen(targets.labels, 0) - targets.weights = PadToTargetSeqLen(targets.weights, 0) - - def InitBeamSearchStateCallback(theta, encoder_outputs, num_hyps_per_beam): - """Wrapper for adding bias to _InitBeamSearchStateCallback. - - Exapnds state to track consistency of hypothesis with provided target. - - Args: - theta: A NestedMap object containing weights' values of this layer and - its children layers. - encoder_outputs: A NestedMap computed by encoder. - num_hyps_per_beam: An int, number hyps to keep for source sentence. - - Returns: - initial_results: a `.NestedMap` of initial results. - states: a `.NestedMap` of initial model states that the client - would like to keep track of for each hyp. The states relevant here - are: - time_step: A scalar indicating current step (=0 for initial state) of - decoder. Must be provided and maintained by super. - consistent: A boolean tensor of shape [tgt_batch, 1] which tracks - whether each hypothesis has exactly matched - encoder_outputs.targets - so far. - """ - initial_results, states = self._InitBeamSearchStateCallback( - theta, encoder_outputs, num_hyps_per_beam) - assert hasattr(states, 'time_step') - if tf.is_tensor(encoder_outputs.padding): - batch_size = tf.shape(encoder_outputs.padding)[1] - else: # Required for multisource models. - batch_size = tf.shape(list(encoder_outputs.padding.values())[0])[1] - num_hyps = batch_size * num_hyps_per_beam - # states.consistent is initially all True - states.consistent = tf.ones([ - num_hyps, - ], dtype=tf.bool) - return initial_results, states - - def PreBeamSearchStepCallback(theta, encoder_outputs, step_ids, states, - num_hyps_per_beam, *args, **kwargs): - """Wrapper for adding bias to _PreBeamSearchStateCallback. - - Biases results.log_probs towards provided encoder_outputs.targets. - - Args: - theta: a NestedMap of parameters. - encoder_outputs: a NestedMap computed by encoder. - step_ids: A tensor of shape [tgt_batch, 1]. - states: A `.NestedMap` of tensors representing states that the clients - would like to keep track of for each of the active hyps. - num_hyps_per_beam: Beam size. - *args: additional arguments to _PreBeamSearchStepCallback. - **kwargs: additional arguments to _PreBeamSearchStepCallback. - - Returns: - A tuple (results, out_states). - results: A `.NestedMap` of beam search results. - atten_probs: - The updated attention probs, of shape [tgt_batch, src_len]. - log_probs: - Log prob for each of the tokens in the target vocab. This is of - shape - [tgt_batch, vocab_size]. - out_states: a `.NestedMap` The updated states. The states relevant here - are: - time_step: A scalar indicating current step of decoder. Must be - provided and maintained by subclass. - consistent: A boolean vector of shape [tgt_batch, ] which tracks - whether each hypothesis has exactly matched - encoder_outputs.targets - so far. - """ - p = self.params - time_step = states.time_step - bs_results, out_states = self._PreBeamSearchStepCallback( - theta, encoder_outputs, step_ids, states, num_hyps_per_beam, *args, - **kwargs) - labels = encoder_outputs.targets.labels - weights = encoder_outputs.targets.weights - - def ApplyBias(): - """Bias and update log_probs and consistent.""" - - def TileForBeamAndFlatten(tensor): - tensor = tf.reshape(tensor, [1, -1]) # [1, src_batch] - tensor = tf.tile( - tensor, [num_hyps_per_beam, 1]) # [num_hyps_per_beam, src_batch] - tgt_batch = tf.shape(step_ids)[0] # num_hyps_per_beam*src_batch - return tf.reshape(tensor, [tgt_batch]) - - # Consistent if step_ids == labels from previous step - # TODO(navari): Consider updating consistent only if weights > 0. Then - # re-evaluate the need for bias_only_if_consistent=True. - # Note that prev_label is incorrrect for step 0 but is overridden later - prev_label = TileForBeamAndFlatten( - tf.gather(labels, tf.maximum(time_step - 1, 0), axis=1)) - is_step0 = tf.equal(time_step, 0) - local_consistence = tf.math.logical_or( - is_step0, tf.equal(prev_label, tf.squeeze(step_ids, 1))) - consistent = tf.math.logical_and(states.consistent, local_consistence) - - # get label, weight slices corresponding to current time_step - label = TileForBeamAndFlatten(tf.gather(labels, time_step, axis=1)) - weight = TileForBeamAndFlatten(tf.gather(weights, time_step, axis=1)) - if p.bias_only_if_consistent: - weight = weight * tf.cast(consistent, py_utils.FPropDtype(p)) - - # convert from dense label to sparse label probs - vocab_size = tf.shape(bs_results.log_probs)[1] - uncertainty = tf.constant(1e-10, py_utils.FPropDtype( - p)) # avoid 0 probs which may cause issues with log - label_probs = tf.one_hot( - label, - vocab_size, - on_value=1 - uncertainty, - off_value=uncertainty / - tf.cast(vocab_size - 1, py_utils.FPropDtype(p)), - dtype=py_utils.FPropDtype(p)) # [tgt_batch, vocab_size] - pred_probs = tf.exp(bs_results.log_probs) - - # interpolate predicted probs and label probs - weight = tf.expand_dims(weight, 1) - probs = py_utils.with_dependencies([ - py_utils.assert_less_equal(weight, 1.), - py_utils.assert_greater_equal(weight, 0.) - ], (1.0 - weight) * pred_probs + weight * label_probs) - return tf.math.log(probs), consistent - - def NoApplyBias(): - """No-op. Return original log_probs and consistent.""" - return bs_results.log_probs, states.consistent - - log_probs, consistent = tf.cond( - tf.reduce_all(tf.equal(weights, 0.0)), NoApplyBias, ApplyBias) - bs_results.log_probs = log_probs - out_states.consistent = consistent - - return bs_results, out_states - - return self.beam_search.BeamSearchDecode(self.theta, encoder_outputs, - num_hyps_per_beam_override, - InitBeamSearchStateCallback, - PreBeamSearchStepCallback, - self._PostBeamSearchStepCallback) - - def InferenceAdditionalEncoder(self, feeds): - """Generate an inference graph for the additional encoder.""" - return py_utils.NestedMap(), py_utils.NestedMap() diff --git a/lingvo/core/base_input_generator.py b/lingvo/core/base_input_generator.py deleted file mode 100644 index e527b144..00000000 --- a/lingvo/core/base_input_generator.py +++ /dev/null @@ -1,1205 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Input generators. - -There are three types of batch sizes: - -* Device split batch size: Defined by Params() and is the batch size - on each device/TPU core. BaseInputGenerator.params.batch_size and - BaseSequenceInputGenerator.params.bucket_batch_limit specify per-split batch - size. - -* GlobalBatchSize: number of examples in a global batch. - -* InfeedBatchSize: global_batch_size // num_infeed_hosts, where - num_infeed_hosts is cluster.num_tpu_hosts if using per-host infeed with TPU, - otherwise num_infeed_hosts is 1. - -TODO(rpang): Deal with on packed_inputs. -""" - -import inspect - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import batch_utils -from lingvo.core import datasource -from lingvo.core import hyperparams -from lingvo.core import input_generator_helper as ig_helper -from lingvo.core import inspect_utils -from lingvo.core import ops -from lingvo.core import py_utils -from lingvo.core import tokenizers -import tensorflow.compat.v1 as tf1 -import tensorflow.compat.v2 as tf2 - -# pylint: disable=g-direct-tensorflow-import -from tensorflow.python.ops import io_ops -from tensorflow.python.tpu import tpu_embedding as tpu_embedding_lib -from tensorflow.python.tpu import tpu_feed -# pylint: enable=g-direct-tensorflow-import - -DEFAULT_TOKENIZER_KEY = 'default' - - -class BaseInputGenerator(base_layer.BaseLayer): - """The abstract base input generator.""" - - @classmethod - def DefineInfeedParams(cls, p): - # TPU related infeed tuning. - # Supported use cases: - # - # Data parallelism (num_partitions=None) - # - single host (use_per_host_infeed=False, tpu_infeed_parallelism=1)) - # - multi host (use_per_host_infeed=False, tpu_infeed_parallelism>1) - # - per host (use_per_host_infeed=True) - # Model parallelism (num_partitions>1 where) - # - non-partitioned infeed (use_partitioned_infeed_queue=False): - # - Only first partition gets infeed (e.g. manual partition) - # - single host (use_per_host_infeed=False) - # - per host (use_per_host_infeed=True) - # - All partitions gets data parallel infeed (e.g. MoE) - # - single host not supported - # - per host (use_per_host_infeed=True, use_per_core_infeed=True) - # num_partitions should be set to number of partitions per replica - # - partitioned infeed (use_partitioned_infeed_queue=True) - # - single host (use_per_host_infeed=False) - # - per host (use_per_host_infeed=True) - # num_partitions should be set to number of partitions per replica - # and all partitions should exist on a single host - p.Define('use_per_host_infeed', False, - 'Whether run infeed op on each host.') - p.Define('use_per_core_infeed', False, - 'Whether to shard the infeed per TPU core instead of per replica') - p.Define('tpu_infeed_parallelism', 1, - 'Uses these many python threads to drive infeed concurrently.') - p.Define('use_partitioned_infeed_queue', False, 'Use partitioned infeed') - p.Define('num_partitions', None, 'Num partitions') - - @classmethod - def Params(cls): - """Defaults params for input generators.""" - p = super().Params() - p.name = 'input' - p.Define( - 'batch_size', 0, 'Batch size for a device split. This will be ' - 'scaled to match the accelarator hardware topology.') - p.Define( - 'num_samples', 0, - 'If non-zero, the dataset contains these many samples. ' - 'For test/eval dataset, if we want the test/evel job evaluate ' - 'the whole dataset, this param must be set precisely. Otherwise, ' - 'this param is optional.') - p.Define('resettable', False, - 'If True, the input generator must implement Reset().') - cls.DefineInfeedParams(p) - - p.Define('remote', hyperparams.Params(), - 'Params to configure remote input policy.') - pp = p.remote - pp.Define( - 'shardable_batch', True, - 'True if and only if this input generates simple batches whose 1st ' - 'dimension of every tensor in a batch is the batch dimension, and ' - 'other dimensions are always the same.') - pp.Define( - 'max_inflights_per_target', 32, 'The maximum number of ' - 'concurrent inflight remote input fetches per remote target.') - - return p - - def __init__(self, params): - super().__init__(params) - # parameter to tell the bprop one hot for all the files. - # TODO(ankurbpn): Initialize when using sources from mixed record yielders. - self._bprop_onehot = tf.constant([1], dtype=tf.float32) - # Each entry is a regular expression specifying the set of variables - # to bprop per data source. - self._bprop_variable_filters = [''] - # For TPU enqueue ops, we do not use graph collections, instead, we rely - # on this member variable. This is especially useful for - # executor-driven multiple programs, as we need more fine-grained - # access to drive the infeed for a specific program, rather than - # a single global collection across the graph. - self._tpu_infeed_op = None - # A list of InfeedQueues. - self._tpu_queues = [] - - # Set to true in GetProcessedInputBatch() (and thus _InputBatch()) - self._in_get_processed_input_batch = False - - def CommonInputOpArgs(self): - """Common input params.""" - return {} - - def GetBpropVariableFilters(self): - return self._bprop_variable_filters - - def GetInputSourceOneHot(self): - """Get the current bprop type of the input generator batch.""" - return self._bprop_onehot - - def GlobalBatchSize(self): - """Returns the total batch size (for stats), int or dynamic int tensor.""" - # Uses `InfeedBatchSize()` instead of calculating it from `p.batch_size` - # because the behavior would be overridden by subclasses. - global_batch_size = batch_utils.scale_infeed_to_global( - self.InfeedBatchSize(), self.params.use_per_host_infeed) - tf.logging.info('GlobalBatchSize {}'.format(global_batch_size)) - return global_batch_size - - def InfeedBatchSize(self): - """Returns the batch size of the input batch: int or dynamic int tensor.""" - batch_per_input = batch_utils.scale_split_to_infeed( - self.params.batch_size, self.params.use_per_host_infeed) - tf.logging.info('batch_per_input: %d', batch_per_input) - return batch_per_input - - def _InputBatch(self): - """The current input batch, not preprocessed. - - This is meant to be overridden by subclasses, but not called directly. - Callers should use `GetPreprocessedInputBatch()`. - - Returns: - A `.NestedMap` of input tensors. Each tensor's dim-0 must be the same - and denotes the batch dimension. - """ - raise NotImplementedError('Abstract method') - - def _PreprocessInputBatch(self, batch): - """Preprocesses input batch from _InputBatch. - - Args: - batch: A NestedMap containing input tensors in the format returned by - _InputBatch. - - Returns: - A NestedMap containing preprocessed inputs to feed to the model. - """ - return batch - - def GetPreprocessedInputBatch(self): - """Returns a NestedMap containing a preprocessed batch of inputs. - - These are the actual inputs fed to the model. - - Subclasses generally should not override this function directly. Instead, - override _InputBatch and maybe _PreprocessInputBatch. - """ - self._in_get_processed_input_batch = True - res = self._PreprocessInputBatch(self._InputBatch()) - self._in_get_processed_input_batch = False - return res - - @property - def tpu_number_of_shards(self): - p = self.params - cluster = self.cluster - num_tpu_hosts = cluster.num_tpu_hosts - num_infeed_hosts = num_tpu_hosts if p.use_per_host_infeed else 1 - shards = (cluster.total_worker_devices // num_infeed_hosts) - if p.use_partitioned_infeed_queue or not p.use_per_core_infeed: - shards = shards // cluster.num_devices_per_split - return shards - - def CreateTpuEnqueueOps(self): - """Create the host-side enqueue ops. - - This should be called in an outer non-TPU context. - """ - assert not self._tpu_queues, ('CreateTpuEnqueueOps should only be called ' - 'once.') - self._tpu_queues = [] - p = self.params - cluster = self.cluster - num_tpu_hosts = cluster.num_tpu_hosts - num_cores_per_host = cluster.total_worker_devices // num_tpu_hosts - tf.logging.info( - 'CreateTpuEnqueueOps num_splits_per_client={} ' - 'num_devices_per_split={} num_tpu_hosts={} use_per_host_infeed={}' - .format(cluster.num_splits_per_client, cluster.num_devices_per_split, - num_tpu_hosts, p.use_per_host_infeed)) - - assert num_tpu_hosts > 0, ('num_tpu_hosts: %d' % num_tpu_hosts) - if p.use_per_core_infeed: - if (not p.use_per_host_infeed) or p.use_partitioned_infeed_queue: - raise ValueError('use_per_core_infeed need to have use_per_host_infeed ' - 'but not use_partitioned_infeed_queue.') - if (cluster.num_devices_per_split > num_cores_per_host and - p.use_per_host_infeed): - tf.logging.fatal('Doesn\'t support per host infeed mode when ' - 'num_devices_per_split({}) > num_cores_per_host({}).' - 'Each host must be able to accommodate >= 1 split when ' - 'using per_host_infeed.'.format( - cluster.num_devices_per_split, num_cores_per_host)) - - shards = self.tpu_number_of_shards - tf.logging.info('shards {}'.format(shards)) - - input_ops_list = [] - tpu_embedding_collection = tf.get_collection(py_utils.TPU_EMBEDDING) - tpu_embedding = ( - tpu_embedding_collection[0] if tpu_embedding_collection else None) - - if num_tpu_hosts > 1 and tpu_embedding is not None: - if not p.use_per_host_infeed: - tf.logging.fatal( - 'TPU Embedding must be used with per_host_infeed with multiple ' - 'TPU host topologies.') - - tpu_emb_input_keys = ( - list(tpu_embedding.feature_to_config_dict.keys()) - if tpu_embedding is not None else []) - tf.logging.info('tpu_emb_input_keys: %r', tpu_emb_input_keys) - - num_infeed_hosts = num_tpu_hosts if p.use_per_host_infeed else 1 - tf.logging.info('num_infeed_hosts: %d', num_infeed_hosts) - for task_id in range(num_infeed_hosts): - host_device = '/task:{}/device:CPU:0'.format(task_id) - with tf.device(host_device): - self._batch = self.GetPreprocessedInputBatch() - if isinstance(self._batch, py_utils.NestedMap): - # Hack: bucket_keys and xxx.bucket_keys are not needed on TPU. - # Note that when MultiTaskData is used, bucket_keys will be at the - # second level of the dictionary. - self._batch = self._batch.FilterKeyVal( - lambda k, _: not k.endswith('bucket_keys')) - tf.logging.info('host_device: %s, batch: %r', host_device, self._batch) - - for k, x in self._batch.FlattenItems(): - assert x.shape.is_fully_defined(), ( - 'Shape must be fully defined: %s: %s' % (k, x)) - # TODO(cwhipkey): if it's a string (or other type not supported on - # TPU), drop it from feeding and on the other end add in an op that - # fails if used. - shapes = self._batch.Transform(lambda x: x.shape).Flatten() - dtypes = self._batch.Transform(lambda x: x.dtype).Flatten() - - tf.logging.info('host_device: %s infeed shapes: %r', host_device, - shapes) - tf.logging.info('host_device: %s infeed dtypes: %r', host_device, - dtypes) - - if p.use_partitioned_infeed_queue: - device_assignment = py_utils.GetTpuDeviceAssignment() - - host_device = device_assignment.host_device( - replica=0, job=tf.flags.FLAGS.tf_master) - host_id = int(host_device.split('/task:')[1].split('/device:')[0]) - tf.logging.info('host_id: {} host_device: {}'.format( - host_id, host_device)) - q = tpu_feed._PartitionedInfeedQueue( # pylint: disable=protected-access - number_of_tuple_elements=len(dtypes), - device_assignment=device_assignment, - host_id=host_id, - input_partition_dims=[ - [p.num_partitions] + [1] * (len(s) - 1) for s in shapes - ], - tuple_types=dtypes, - tuple_shapes=shapes) - else: - if p.use_per_core_infeed: - q = tpu_feed.InfeedQueue( - tuple_types=dtypes, - tuple_shapes=shapes, - number_of_partitions=p.num_partitions) - else: - q = tpu_feed.InfeedQueue(tuple_types=dtypes, tuple_shapes=shapes) - assert shards is not None - q.set_number_of_shards(shards) - - self._tpu_queues.append(q) - - if p.use_partitioned_infeed_queue: - input_ops = q.generate_enqueue_ops([self._batch.Flatten()]) - elif p.use_per_host_infeed: - # TODO(ylc/zhifengc): Add this to a policy module and test it. - def TPUOrdinalFunction(shard_index_in_host): - if p.use_per_core_infeed: - return shard_index_in_host - device_assignment = py_utils.GetTpuDeviceAssignment() - if device_assignment: - # We put both enqueue/dequeue ops at core 0 in each replica. - replica = device_assignment.lookup_replicas( - task_id, 0)[shard_index_in_host] # pylint: disable=cell-var-from-loop - return device_assignment.tpu_ordinal(replica=replica) - else: - return shard_index_in_host - - input_ops = q.split_inputs_and_generate_enqueue_ops( - self._batch.Flatten(), - placement_function=lambda x: host_device, # pylint: disable=cell-var-from-loop - tpu_ordinal_function=TPUOrdinalFunction) - else: - input_ops = q.split_inputs_and_generate_enqueue_ops( - self._batch.Flatten(), - device_assignment=py_utils.GetTpuDeviceAssignment()) - input_ops_list += input_ops - - tf.logging.info('input_ops_list %s', input_ops_list) - grouped_infeed_op = tf.group(*input_ops_list) - self._tpu_infeed_op = [] - for _ in range(p.tpu_infeed_parallelism): - self._tpu_infeed_op.append(grouped_infeed_op) - - def TpuDequeueBatch(self): - """Create TPU dequeue ops. - - This should only be called within a TPU context. - - Returns: - - A NestedMap of the input batch. - """ - assert self._tpu_queues, 'CreateTpuEnqueueOps must be called first.' - with tf.device(tf.tpu.core(0)): - # Note that the dequeue_tuple op on the TPU core - # only cares about the shape/types being dequeued - # which is why this is hard-coded to the first Queue. - tensors = self._tpu_queues[0].generate_dequeue_op() - return self._batch.Pack(tensors) - - def CreateTpuEmbeddingEnqueueOps(self): - """Creates the TpuEmbedding enqueue ops on the host. - - Note that this must be called after the instantiation of the - monolithic TPUEmbeddingLayer. - """ - p = self.params - cluster = self.cluster - num_tpu_hosts = cluster.num_tpu_hosts - num_infeed_hosts = num_tpu_hosts if p.use_per_host_infeed else 1 - - tpu_embedding_collection = tf.get_collection(py_utils.TPU_EMBEDDING) - tpu_embedding = ( - tpu_embedding_collection[0] if tpu_embedding_collection else None) - - enqueue_ops = [] - - if num_tpu_hosts > 1 and tpu_embedding is not None: - if not p.use_per_host_infeed: - tf.logging.fatal( - 'TPU Embedding must be used with per_host_infeed with multiple ' - 'TPU host topologies.') - tpu_emb_input_keys = ( - list(tpu_embedding.feature_to_config_dict.keys()) - if tpu_embedding is not None else []) - tf.logging.info('tpu_emb_input_keys: %r', tpu_emb_input_keys) - if not tpu_embedding: - return - - for task_id in range(num_infeed_hosts): - host_device = '/task:{}/device:CPU:0'.format(task_id) - with tf.device(host_device): - if isinstance(self._batch, py_utils.NestedMap): - # Hack: bucket_keys and xxx.bucket_keys are not needed on TPU. - # Note that when MultiTaskData is used, bucket_keys will be at the - # second level of the dictionary. - self._batch = self._batch.FilterKeyVal( - lambda k, _: not k.endswith('bucket_keys')) - tf.logging.info('host_device: %s, batch: %r', host_device, self._batch) - - enqueue_dict_per_core = [ - {} for _ in range(tpu_embedding.num_cores_per_host) - ] - num_cores_per_host = tpu_embedding.num_cores_per_host - for key in tpu_emb_input_keys: - feat = self._batch[key] - tpu_emb_feat_splitted = tf.split(feat, num_cores_per_host) - for core, split in enumerate(tpu_emb_feat_splitted): - # Dense to sparse. Note the assumption of a padding id. - sample_indices = tf.where(tf.not_equal(split, -1)) - embedding_indices = tf.gather_nd(split, sample_indices) - enqueue_data = tpu_embedding_lib.EnqueueData( - embedding_indices, sample_indices) - enqueue_dict_per_core[core][key] = enqueue_data - enqueue_ops += tpu_embedding.generate_enqueue_ops(enqueue_dict_per_core) - self._tpu_infeed_op.append(tf.group(*enqueue_ops)) - - @property - def tpu_infeed_op(self): - if self._tpu_infeed_op is not None: - return self._tpu_infeed_op - else: - raise ValueError('TPU infeed op not set. Call CreateTpuEnqueueOps first.') - - def SplitInputBatch(self, num_splits): - """Splits the current InputBatch into num_splits ways. - - Args: - num_splits: The number of splits. - - Returns: - A list of `.NestedMap`. Each `.NestedMap` represents the input - tensors in one split. - """ - assert num_splits >= 1 - - batch = self.GetPreprocessedInputBatch() - if num_splits == 1: - # Special case. No split is needed. - return [batch] - - assert not py_utils.use_tpu() - field_split = ig_helper.SplitTensors(batch.Flatten(), num_splits) - num_fields = len(field_split) - ret = [] - for j in range(num_splits): - split_flatten = [field_split[i][j] for i in range(num_fields)] - split = batch.Pack(split_flatten) - ret += [split] - return ret - - def Reset(self, tf_session): - """Reset the input-generator. - - Override so that the input_generator reproduces examples as if from a fresh - instantiation. - - Args: - tf_session: A tensorflow session. - """ - raise NotImplementedError() - - -class BaseInputGeneratorFromFiles(BaseInputGenerator): - """Base class for input generators that reads from files.""" - - @classmethod - def Params(cls): - """Defaults params for input generators.""" - p = super().Params() - p.Define( - # NOTE: file_pattern is deprecated. New params should use - # file_datasource instead. - # TODO(b/139345706) remove file_pattern parameter - 'file_pattern', - '', - 'A single file pattern string, a list of file pattern strings or a list' - ' of pairs or a list of tuples. Some of the cases may not be supported ' - 'Depending on the value of use_within_batch_mixing and use_chaining.' - 'In the later 2 cases, probablistic samples are from the inputs ' - 'proportional to their weights. Typically, values are binary ' - 'protocol buffers containing train/eval samples. Keys are not used.') - p.Define( - 'file_datasource', None, 'A DataSource describing the file sources ' - 'including any weights and bprop_variable_filters required.') - p.Define('file_random_seed', 301, - 'Random seed for shuffling the input data.') - p.Define( - 'file_buffer_size', 10000, - 'How many records are buffered for random shuffling. This param ' - 'affects how much RAM a train/test job needs. E.g., if an average ' - 'record is about 500KB, the buffer needs 5GB ram.') - p.Define( - 'file_buffer_size_in_seconds', 0, - 'If non-zero, keep enough records in the buffer to handle N seconds ' - 'worth of demand. E.g., if the training job is reading 1000 records ' - 'per second and this parameter is set to 10, the buffer is resized ' - 'to contain 10000 records. This parameter is useful when reading from ' - 'many data sources at different speeds, as it automatically tunes the ' - 'size of buffers to fit demand. The file_buffer_size parameter is an ' - 'upper bound to the buffer size.') - p.Define('file_parallelism', 16, 'How many files to read concurrently.') - p.Define( - 'bucket_adjust_every_n', 0, 'If non-zero, optimize the values of ' - 'bucket_upper_bound except the last one after every N records ' - 'based on the current input length distribution.') - p.Define( - 'flush_every_n', 0, 'If non-zero, flushes all batches buffered ' - 'so far every these many records are yielded.') - p.Define('num_batcher_threads', 1, 'Number of threads to use for input ' - 'record batcher.') - p.Define( - 'require_sequential_order', False, - 'If true, the input op is required to process the file glob as ' - 'well as the contents of each file in a deterministic sequential order.' - ' This is intended for unit tests. Setting this automatically disables ' - 'file_random_seed, file_buffer_size, file_parallelism, ' - 'num_batcher_threads, and requires a single file_pattern.') - p.Define( - 'repeat_count', -1, - 'Number of repetitions of a dataset before throwing OutOfRange error ' - 'when using require_sequential_order. Must only be set if ' - 'require_sequential_order is True.') - # TODO(b/139345706) when file_pattern is deleted use_within_batch_mixing - # will be specified by passing a WithinBatchMixingDataSource to - # p.file_datasource and this param should be deleted as well. - p.Define( - 'use_within_batch_mixing', False, 'Whether to mix records from ' - 'different input sources within batch or across batches (the ' - 'default option). This option only takes effect when file_pattern' - ' is a list of file patterns with weights.') - # TODO(b/139345706) when file_pattern is deleted use_chaining - # will be specified by passing a ChainingDataSource to - # p.file_datasource and this param should be deleted as well. - p.Define( - 'use_chaining', False, 'Whether to output records from ' - 'different input sources one after another, i.e., first all records ' - 'from a first input source, then all records from a second one, etc. ' - 'use_chaining does not guarantee that records from subsequent input ' - 'sources are placed in separate input batches. ' - 'This option only takes effect when file_pattern is a list of file ' - 'patterns.') - - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.use_per_host_infeed and p.file_random_seed != 0: - raise ValueError('file_random_seed needs to be 0 when ' - 'use_per_host_infeed == True.') - - assert not (p.file_pattern and p.file_datasource - ), 'Only one of file_pattern and data_source can be specified' - - # TODO(b/139345706) remove support for file_pattern - if not p.file_datasource: - if isinstance(p.file_pattern, str): - p.file_datasource = datasource.SimpleDataSource.Params().Set( - file_pattern=p.file_pattern) - elif isinstance(p.file_pattern, (list, tuple)): - if all([isinstance(x, str) for x in p.file_pattern]): - # While this violates the documentation and intended use, there are - # subclasses that have used a tuple of strings, rather than a list of - # string, weight tuples. Rather than treating lists and tuples - # differently, support both here until p.file_pattern is removed. - p.file_datasource = datasource.SimpleDataSource.Params().Set( - file_pattern=','.join(p.file_pattern)) - elif p.use_within_batch_mixing: - assert not p.use_chaining, "Can't both use chaining and mixing" - - if max(list(map(len, p.file_pattern))) >= 3: - # Within batch mixing doesn't work with backprop filters, i.e. when - # file_pattern param contains a list of - # tuples. - raise ValueError('Expected a list of pairs, got %s' % - p.file_pattern) - - file_patterns, weights = (list(x) for x in zip(*p.file_pattern)) - - p.file_datasource = datasource.WithinBatchMixingDataSource.Params( - ).Set( - file_patterns=file_patterns, weights=weights) - elif p.use_chaining: - p.file_datasource = datasource.ChainingDataSource.Params().Set( - file_patterns=p.file_pattern) - else: - # Otherwise fall back to MixByWeight-based approach. - file_patterns = [] - weights = [] - bprop_variable_filters = [] - for input_entry in p.file_pattern: - if isinstance(input_entry, str): - raise ValueError( - 'Should explicitly specify weights, got string: %s' % - input_entry) - file_pattern, weight = input_entry[:2] - file_patterns.append(file_pattern) - weights.append(weight) - bprop_variable_filter = input_entry[2] if len( - input_entry) > 2 else '' - bprop_variable_filters.append(bprop_variable_filter) - p.file_datasource = datasource.CrossBatchMixingDataSource.Params( - ).Set( - file_patterns=file_patterns, - weights=weights, - bprop_variable_filters=bprop_variable_filters) - else: - raise ValueError('Cannot parse p.file_pattern into a datasource.') - - self.CreateChild('datasource', p.file_datasource) - - def CommonInputOpArgs(self): - """Common input params.""" - p = self.params - args = super().CommonInputOpArgs() - if p.file_datasource and issubclass(p.file_datasource.cls, - datasource.ChainingDataSource): - # If a user provides a ChainingDataSource make sure that the - # param is set correctly when passed to the InputOp - p.use_chaining = True - args.update({ - 'file_random_seed': p.file_random_seed, - 'file_buffer_size': p.file_buffer_size, - 'file_parallelism': p.file_parallelism, - 'file_buffer_size_in_seconds': p.file_buffer_size_in_seconds, - 'bucket_adjust_every_n': p.bucket_adjust_every_n, - 'flush_every_n': p.flush_every_n, - 'num_threads': p.num_batcher_threads, - 'require_sequential_order': p.require_sequential_order, - 'repeat_count': p.repeat_count, - 'use_chaining': p.use_chaining, - }) - args.update(self._InputOpBucketingArgs()) - return args - - def _InputOpBucketingArgs(self): - return { - 'bucket_upper_bound': [1000000000], - 'bucket_batch_limit': [self.InfeedBatchSize()], - } - - # TODO(b/139345706): After p.file_pattern is deleted, the following functions - # _DataSourceFromFilePattern, _BuildDataSourceWithMetadata, _BuildDataSource - # can be deleted and functionality moved to using the DataSource directly. - def _DataSourceFromFilePattern(self, file_pattern, input_source_weights=None): - """Read and return input batch from a string file_pattern. - - Args: - file_pattern: A string file pattern. - input_source_weights: A list of float input source weights to control - input example mix in the batch. The records will be sampled from inputs - proportionally to these weights. Defaults to None which should be - treated as an empty list. - - Returns: - A tuple of tf.Tensors where the tensors which contain the input data and - have major dimension same as size of input batch. - """ - raise NotImplementedError() - - def _BuildDataSourceWithMetadata(self): - """Read and return input batch from `p.file_pattern`. - - `p.file_pattern` may be a string file_pattern or a - list of (file_pattern, weight, [bprop_variable_filter]) tuples. - bprop_variable_filter is optional. When bprop_variable_filter is used, - batches will always contain the examples from the same source. Otherwise, - examples from different sources may be mixed together. - - Returns: - A `.NestedMap` containing - - - data: a tuple of tf.Tensor or `.NestedMap` of - tf.Tensor same as `self._DataSourceFromFilePattern()` - - source_selected: a tensor of size [batch_size, number of data sources] - or None. - - selected_bprop: a tensor of size [number of data sources] or None. - - bprop_variable_filters: a list of bprop_variable filters for each source - or None. - - Raises: - ValueError: If file_datasource is not set - """ - p = self.params - if p.use_per_host_infeed and not self._in_get_processed_input_batch: - raise ValueError( - 'This input generator does not support p.use_per_host_infeed. ' - 'Please set it to False, or move the call to self._BuildDataSource() ' - 'from self.__init__() to self._InputBatch() for batches to be ' - 'correctly replicated per host.') - if not p.file_datasource and p.file_pattern: - # This is a workaround for subclasses which have defined - # their own data source-like functionality. - tf.logging.info( - 'Creating data source-like output from class %s using ' - 'file_pattern %s', self, p.file_pattern) - ret = py_utils.NestedMap() - ret.data = self._DataSourceFromFilePattern(p.file_pattern) - else: - tf.logging.info( - 'Building data source %s with params %s and ' - 'file_pattern %s', self.datasource, self.datasource.params, - p.file_pattern) - ret = self.datasource.BuildDataSource(self._DataSourceFromFilePattern) - if 'selected_bprop' in ret: - self._bprop_onehot = ret.selected_bprop - if 'bprop_variable_filters' in ret: - self._bprop_variable_filters = ret.bprop_variable_filters - if 'source_selected' not in ret: - ret.source_selected = None - return ret - - def _BuildDataSource(self): - """Read and return input batch from `p.file_pattern`. - - Same as _BuildDataSourceWithMetadata but does not return any metadata. - - Returns: - A tuple of tf.Tensor or `.NestedMap` of tf.Tensor same as - `self._DataSourceFromFilePattern()`. - - Raises: - ValueError: If unknown token type. - """ - return self._BuildDataSourceWithMetadata()['data'] - - -class BaseSequenceInputGenerator(BaseInputGeneratorFromFiles): - """The basic sequence input generator.""" - - @classmethod - def Params(cls): - """Defaults params for sequence input generators.""" - p = super().Params() - p.Delete('batch_size') - p.remote.shardable_batch = False - - # How input should be bucketized. - p.Define( - 'bucket_upper_bound', [2560], 'Bucketing scheme. Required to be' - 'a sorted list of integers. Examples that are longer than all bucket' - 'upper bounds are skipped.') - p.Define( - 'bucket_batch_limit', [8], - 'Desired per-split batch size per bucket. Scaled in ' - 'infeed_bucket_batch_size to the infeed size.' - 'Must be the same length as bucket_upper_bound.') - p.Define('source_max_length', None, - 'The maximum length of the source sequence.') - p.Define('target_max_length', 300, - 'The maximum length of the target sequence.') - p.Define('pad_to_max_seq_length', False, - 'If True, input tensors will be padded to max_length.') - p.Define('tokenizer', tokenizers.AsciiTokenizer.Params(), - 'Tokenizer params.') - p.Define( - 'tokenizer_dict', {}, - 'If multiple tokenizers are required, they can be accessed through ' - 'this dict via a key.') - return p - - def __init__(self, params): - super().__init__(params) - - p = self.params - - if p.tokenizer: - assert DEFAULT_TOKENIZER_KEY not in p.tokenizer_dict - p.tokenizer_dict[DEFAULT_TOKENIZER_KEY] = p.tokenizer - - self.tokenizer_dict = {} - for k, p in p.tokenizer_dict.items(): - if p: - name = '_tokenizer_' + k - self.CreateChild(name, p) - self.tokenizer_dict[k] = self.children[name] - else: - self.tokenizer_dict[k] = None - - if DEFAULT_TOKENIZER_KEY in self.tokenizer_dict: - self.tokenizer = self.tokenizer_dict[DEFAULT_TOKENIZER_KEY] - - @property # Adjust batch size according to the cluster spec. - def infeed_bucket_batch_limit(self): - """Returns the bucket batch limit for one infeed host.""" - p = self.params - cluster = self.cluster - infeed_bucket_batch_limit = [ - batch_utils.scale_split_to_infeed(b, p.use_per_host_infeed) - for b in p.bucket_batch_limit - ] - tf.logging.info( - 'infeed_bucket_batch_limit={} num_splits_per_client={} bucket_batch_limit={}' - .format(infeed_bucket_batch_limit, cluster.num_splits_per_client, - p.bucket_batch_limit)) - return infeed_bucket_batch_limit - - def InfeedBatchSize(self): - """Returns the batch size of one infeed pipeline. - - Override in subclass to provide dynamically shaped infeed batch size. - - If use_per_host_infeed is False then there is only one infeed pipeline and - then the GlobalBatchSize() and the InfeedBatchSize() is the same. - """ - buckets = self.infeed_bucket_batch_limit - if any(x != buckets[0] for x in buckets): - tf.logging.warning('Using max bucket batch limit but not all limits are ' - 'the same {}'.format(buckets)) - infeed_size = max(buckets) - tf.logging.info('InfeedBatchSize: %d', infeed_size) - return infeed_size - - def _InputOpBucketingArgs(self): - p = self.params - bucket_batch_limit = self.infeed_bucket_batch_limit - tf.logging.info('infeed_bucket_batch_limit %r', bucket_batch_limit) - return { - 'bucket_upper_bound': p.bucket_upper_bound, - 'bucket_batch_limit': bucket_batch_limit, - } - - def StringsToIds(self, - strs, - is_source=False, - external_max_length=None, - external_append_eos=None, - key=None, - languages=None): - """Tokenize strs into vocab ids. - - Args: - strs: A vector of strings. - is_source: A bool to indicate whether to use `source_max_length` to pad - 'strs'. - external_max_length: An int providing the max_length for strs. - external_append_eos: Bool or None. If None, will be ignored and - `params.append_eos` will be used. If bool, will determine if an eos - symbol will be added to tokens. - key: A string key in case the model has multiple tokenizers. - languages: A vector of str with the same length as `strs`. - - Returns: - A tuple (ids, labels, paddings) with the same shape [batch, maxlen]. - - - ids[i, j] is the input token id of i-th sample for j-th step. - - labels[i, j] is the target token id of i-th sample for j-th step. - - paddings[i, j] is 1 iff i-th sample's j-th step is padded. - - Raises: - ValueError: If unknown token type. - """ - p = self.params - - if external_max_length is not None: - maxlen = external_max_length - elif is_source: - maxlen = p.source_max_length - else: - maxlen = p.target_max_length - - key = key or DEFAULT_TOKENIZER_KEY - return self.tokenizer_dict[key].StringsToIds( - strs, maxlen, external_append_eos, languages=languages) - - def IdsToStrings(self, ids, lens, key=None): - """Converts ids back to strings. - - Args: - ids: A matrix of shape [batch, seqlen]. ids[i, :] is the i-th sample's - ids. - lens: A vector of shape [batch]. lens[i] is the sequence length of the - i-th sample. Only the first lens[i] tokens in ids[i, :] are valid tokens - for the i-th sequence. - key: A string key in case the model has multiple tokenizers. - - Returns: - sequences - A vector of shape [batch]. The converted string sequence. - - Raises: - ValueError: If unknown token type. - """ - key = key or DEFAULT_TOKENIZER_KEY - return self.tokenizer_dict[key].IdsToStrings(ids, lens) - - def Cast(self, v): - """Cast tensor dtype to fprop_dtype.""" - if not v.dtype.is_floating: - return v - return tf.cast(v, py_utils.FPropDtype(self.params)) - - -class BaseTinyDatasetInput(BaseInputGenerator): - """Input generator for tiny dataset which are stored in tf checkpoint. - - | Input batch (b: batch size, h: height, w: width, d: depth): - | raw: Samples. [b, h, w, d]. - | data: Preprocessed samples. [b, h, w, d]. - | label: Labels. [b]. - | weight: [b]. weight[i] is 1.0 if i-th sample is considered to - | be a real example. Otherwise, weight[i] is 0.0. - """ - - @classmethod - def Params(cls): - """Defaults params.""" - p = super().Params() - p.Define('ckpt', None, 'A TensorFlow checkpoint.') - p.Define('data', 'x_train', 'The tensor name in the ckpt.') - p.Define('data_dtype', tf.uint8, 'The tensor dtype in the ckpt.') - p.Define( - 'data_shape', (0, 0, 0), 'A tuple of ints. E.g., a tiny image ' - 'has the shape (height, weight, depth).') - p.Define('label', 'y_train', 'The tensor name in the ckpt.') - p.Define('label_dtype', tf.uint8, 'The tensor dtype in the ckpt.') - p.Define('repeat', True, 'If true, goes through the dataset repeatedly.') - p.use_per_host_infeed = True - return p - - def _InputBatch(self): - p = self.params - - @tf.function - def ReadData(): - x, y = io_ops.restore_v2(p.ckpt, [p.data, p.label], [''] * 2, - [p.data_dtype, p.label_dtype]) - # Always convert to float32. - return tf.cast(x, tf.float32), tf.cast(y, tf.float32) - - # Loads data and label into memory and keep it around. - data, label = ops.cached_call( - f=ReadData.get_concrete_function(), T=[tf.float32, tf.float32]) - b, shape = self.InfeedBatchSize(), list(p.data_shape) - data = tf.reshape(data, [-1] + shape) - label = tf.reshape(label, [-1]) - label = py_utils.HasShape(label, [tf.shape(data)[0]]) - sample_ids = ops.random_permutation_sequence( - num=p.num_samples, - batch=b, - repeat=p.repeat, - seed=p.random_seed if p.random_seed else 0) - n = tf.shape(sample_ids)[0] - raw = py_utils.PadOrTrimTo(tf.gather(data, sample_ids), [b] + shape) - ret = py_utils.NestedMap( - raw=raw, - data=self._Preprocess(raw), - label=py_utils.PadOrTrimTo(tf.gather(label, sample_ids), [b]), - weight=py_utils.PadOrTrimTo(tf.ones([n], dtype=tf.float32), [b])) - # Okay, so here is my problem. - if not py_utils.use_tpu(): - ret['sample_ids'] = sample_ids - return ret - - def _Preprocess(self, raw): - return raw - - -class BaseDataExampleInputGenerator(BaseInputGenerator): - """Base class for input generators that read Feature protos via tf.data.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_files', None, 'Delimited glob of input files.') - p.Define( - 'dataset_type', None, - 'A dataset class constructor such as tf.data.TFRecordDataset. ' - 'The class constructor must take a list of filenames and produce an ' - 'object that extends tf.data.Dataset.') - p.Define('randomize_order', True, 'Whether to randomize the order.') - p.Define('parallel_readers', 1, 'Number of parallel reader threads.') - p.Define('num_examples', -1, 'Number of examples (-1 for unlimited).') - p.Define( - 'num_epochs', -1, - 'Number of passes through the data to make (-1 for unlimited).' - '`tf.errors.OutOfRangeError` is thrown after the limit is reached.') - p.Define('randomize_shuffle_size', 500, - 'Size of the random shuffle buffer.') - p.remote.shardable_batch = False - return p - - def __init__(self, params): - super().__init__(params) - p = params - assert p.input_files, ( - 'input_files is required for a tf.data example input generator') - assert p.dataset_type, ( - 'dataset_type is required for a tf.data example input generator') - - def GetFeatureSpec(self): - """Subclasses must implement and return a feature spec. - - Returns: - NestedMap of features compatible with tf.io.parse_example. Default - implementation returns an empty dict. - """ - return {} - - def GetPreprocessedInputBatch(self): - p = self.params - - def ParseAndProcess(*cols): - """Parses a Tensorflow example into features.""" - # Assume either one or two column input. If one, then the record is - # assumed to be that column. If 2, then it is assumed to be a KV store - # and the record is the second column. - assert len(cols) in [ - 1, 2 - ], ('BaseExampleInputGenerator supports one or two column input') - record = cols[-1] - feature_spec = self.GetFeatureSpec() - features = py_utils.NestedMap(tf.io.parse_example(record, feature_spec)) - return self._PreprocessInputBatch(features) - - dataset_factory = p.dataset_type - dataset = ( - tf.data.Dataset.list_files( - p.input_files, shuffle=bool(p.randomize_order)).apply( - tf.data.experimental.parallel_interleave( - dataset_factory, - cycle_length=p.parallel_readers, - sloppy=p.randomize_order))) - - if p.randomize_order: - dataset = dataset.shuffle(p.randomize_shuffle_size) - dataset = dataset.take(p.num_examples) - dataset = dataset.repeat(p.num_epochs) - dataset = dataset.batch(self.InfeedBatchSize(), drop_remainder=True) - dataset = dataset.map( - ParseAndProcess, num_parallel_calls=p.parallel_readers) - dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE) - iterator = dataset.make_one_shot_iterator() - input_batch = iterator.get_next() - return input_batch - - -def DefineTFDataInput(name, func, ignore_args=None, map_args=None): - """Defines a new InputGenerator class from given tf.data pipeline. - - This function allows users to utilize existing tf.data pipelines which are - defined externally, without making binding boilerplates. - The generated InputGenerator behaves like a one-shot iterator of the given - pipeline. If the iterator is designed to be repeated, the returned - InputGenerator will work similarly. - This function generates `Params` automatically by analysing the given - pipeline's signature so that the behavior of the pipeline can be saved into - `Params`. - This function defines the InputGenerator class on the caller's module. To - avoid any confusion, the returned class have to be stored in the module-level - symbol with the same identifier with given `name`. - - Example: - >>> # A tf.data pipeline which returns a dict of Tensors. - >>> def my_dataset(begin=0, end=10): - ... ds = tf.data.Dataset.from_tensor_slices(tf.range(begin, end)) - ... return ds.map(lambda x: {'value': x}) - - >>> # Defines the InputGenerator class for my_dataset. - >>> MyInput = DefineTFDataInput('MyInput', my_dataset) - - >>> # Obtains Params of MyInput. - >>> p = MyInput.Params() - >>> assert p.args.begin == 0 - >>> assert p.args.end == 10 - - >>> # Instantiates the InputGenerator from Params. - >>> ig = p.Instantiate() - >>> assert isinstance(ig, MyInput) - - >>> # Obtains the data tensors. - - >>> # In TFv1: - >>> data = ig.GetPreprocessedInputBatch() - >>> with tf.Session() as sess: - ... values = sess.run(data) # {'value': 0} - ... values = sess.run(data) # {'value': 1} - ... values = sess.run(data) # {'value': 2} - - >>> # In TFv2: - >>> values = ig.GetPreprocessedInputBatch() # {'value': 0} - >>> values = ig.GetPreprocessedInputBatch() # {'value': 1} - >>> values = ig.GetPreprocessedInputBatch() # {'value': 2} - - Args: - name: A string, representing the name of the new InputGenerator class. - func: A callable to be analysed to generate the new InputGenerator. The - return value of `func` must be a single `tf.data.Dataset` which yields a - dict or its subclasses. The signature (parameter list) of `func` must have - all explicit parameters needed to configure the pipeline. `*args` and - `**kwargs` parameters would be ignored from defining `Params`. - ignore_args: A collection of strings, representing the set of parameter - names to be ignored from defining `Params`. - map_args: A {str: str} dict, representing mappings from existing fields in - `Params()` to `func`'s parameter. These mappings can be used to propagate - some particular Lingvo-specific options defined by others (typically by - super classes: `BaseInputGenerator` or `BaseLayer`) to the given function. - Each entry in the dict represents a `{func_param: layer_param}` pair such - that the `Params().layer_param` field will be mapped to the parameter - `func_param` of `func`. `func_param` won't be added into `Params().args` - to avoid duplicated definitions about the same parameters. - - Returns: - A new InputGenerator class that invokes `func` internally. The `Params()` - method of the returned class makes a new Params containing the `args` field - representing the parameters of `func`. The `GetPreprocessedInputBatch()` - method returns a `py_utils.NestedMap` representing the same dict of the - obtained data from the dataset. - """ - ignore_args = set(ignore_args if ignore_args is not None else ()) - map_args = dict(map_args if map_args is not None else {}) - - # Defines the class first as it will be required to call `super()`. - generated_cls = type(name, (BaseInputGenerator,), {}) - - @classmethod - def _Params(cls): - """Generates Params to configure the InputGenerator. - - This function analyses the signature of the given callable `func` and - defines corresponding fields into `Params` to the obtained function - parameters. - - Returns: - An `InstantiableParams` object representing the InputGenerator. It has the - `args` field which contains the set of parameters of `func`. - """ - # Keys in `map_args` will also be ignored. - actual_ignore_args = ignore_args | set(map_args.keys()) - - p = super(generated_cls, cls).Params() - - # Introduces a new group `args` to avoid confusion between `func`'s - # parameters and existing params defined by super classes. - # TODO(oday): For better UX, consider removing this nested field and add - # `func`s parameters to `p` directly. We need to make sure that there are no - # side effects by integrating `func`'s parameters and follows: - # - BaseInputGenerator.Params() - # - BaseLayer.Params() - # - InstantiableParams.cls - p.Define('args', hyperparams.Params(), 'Parameter list of the pipeline.') - inspect_utils.DefineParams(func, p.args, actual_ignore_args) - return p - - def _Init(self, params): - """Initializes the InputGenerator.""" - super(generated_cls, self).__init__(params) - p = self.params - - # We have to make the one-shot iterator only once as _InputBatch will be - # called repeatedly in TFv2. - overrides = {k: p.Get(v) for k, v in map_args.items()} - dataset = inspect_utils.CallWithParams(func, p.args, **overrides) - assert isinstance(dataset, (tf1.data.Dataset, tf2.data.Dataset)), ( - 'DefineTFDataInput must take a callable which returns a ' - '`tf.data.Dataset`. The given callable `%s` returned `%s`' % - (func, dataset)) - self.iterator = tf1.data.make_one_shot_iterator(dataset) - - def _InputBatch(self): - """Generates data tensors by invoking the pipeline.""" - - # TFv1: Returns Tensors which will be determined by Session.run(). - # TFv2: Returns Tensors with actual values. - data = self.iterator.get_next() - - # Converts dict to NestedMap to maintain consistency with existing - # functionalities in base_input_generator. - # TODO(oday): Consider mitigating this restriction. - assert isinstance(data, dict), ( - 'DefineTFDataInput accepts only datasets that returns a dict or its ' - 'subclasses.') - if not isinstance(data, py_utils.NestedMap): - data = py_utils.NestedMap.FromNestedDict(data) - - return data - - # Overrides member methods. - generated_cls.Params = _Params - generated_cls.__init__ = _Init - generated_cls._InputBatch = _InputBatch # pylint: disable=protected-access - - # Sets __module__ to the caller's module name for pickling and restoring from - # Params to work. - # See also the namedtuple's implementation for details. - module = inspect.stack()[1].frame.f_globals.get('__name__', '__main__') - generated_cls.__module__ = module - - return generated_cls diff --git a/lingvo/core/base_input_generator_test.py b/lingvo/core/base_input_generator_test.py deleted file mode 100644 index 6fd469fe..00000000 --- a/lingvo/core/base_input_generator_test.py +++ /dev/null @@ -1,477 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for base_input_generator.""" - -import os -import shutil -import tempfile -from absl.testing import flagsaver -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import cluster_factory -from lingvo.core import hyperparams -from lingvo.core import py_utils -from lingvo.core import test_utils -import mock -import numpy as np - - -import tensorflow.compat.v1 as tf1 -import tensorflow.compat.v2 as tf2 - - -def _CreateFakeTFRecordFiles(record_count=10): - tmpdir = tempfile.mkdtemp() - data_path = os.path.join(tmpdir, 'fake.tfrecord') - with tf.io.TFRecordWriter(data_path) as w: - for _ in range(record_count): - feature = { - 'audio': - tf.train.Feature( - float_list=tf.train.FloatList( - value=np.random.uniform(-1.0, 1.0, 48000))), - } - example = tf.train.Example(features=tf.train.Features(feature=feature)) - w.write(example.SerializeToString()) - return tmpdir, data_path - - -class BaseInputGeneratorBatchSizeTest(test_utils.TestCase): - - @flagsaver.flagsaver(xla_device='tpu', enable_asserts=False) - def testSingleHostInfeed(self): - with cluster_factory.ForTestingWorker(tpus=128): - p = base_input_generator.BaseInputGenerator.Params() - p.batch_size = 16 - p.use_per_host_infeed = False - input_generator = p.Instantiate() - - self.assertEqual(2048, input_generator.InfeedBatchSize()) - self.assertEqual(2048, input_generator.GlobalBatchSize()) - - @flagsaver.flagsaver(xla_device='tpu', enable_asserts=False) - def testPerHostInfeed(self): - with cluster_factory.ForTestingWorker(tpus=128, num_tpu_hosts=8): - p = base_input_generator.BaseInputGenerator.Params() - p.batch_size = 16 - p.use_per_host_infeed = True - input_generator = p.Instantiate() - - self.assertEqual(256, input_generator.InfeedBatchSize()) - self.assertEqual(2048, input_generator.GlobalBatchSize()) - - -class ToyInputGenerator(base_input_generator.BaseDataExampleInputGenerator): - - def GetFeatureSpec(self): - return {'audio': tf.io.FixedLenFeature([48000], tf.float32)} - - -class BaseExampleInputGeneratorTest(test_utils.TestCase): - - def setUp(self): - super().setUp() - tf.reset_default_graph() - - def tearDown(self): - super().tearDown() - if hasattr(self, '_tmpdir'): - shutil.rmtree(self._tmpdir) - - def testTfRecordFile(self): - p = ToyInputGenerator.Params() - p.batch_size = 2 - self._tmpdir, p.input_files = _CreateFakeTFRecordFiles() - p.dataset_type = tf.data.TFRecordDataset - p.randomize_order = False - p.parallel_readers = 1 - ig = p.Instantiate() - with self.session(graph=tf.get_default_graph()): - inputs = ig.GetPreprocessedInputBatch() - eval_inputs = self.evaluate(inputs) - input_shapes = eval_inputs.Transform(lambda t: t.shape) - self.assertEqual(input_shapes.audio, (2, 48000)) - - def testTfRecordFileLargeBatch(self): - p = ToyInputGenerator.Params() - p.batch_size = 200 - self._tmpdir, p.input_files = _CreateFakeTFRecordFiles() - p.dataset_type = tf.data.TFRecordDataset - p.randomize_order = False - p.parallel_readers = 1 - ig = p.Instantiate() - with self.session(graph=tf.get_default_graph()): - inputs = ig.GetPreprocessedInputBatch() - eval_inputs = self.evaluate(inputs) - input_shapes = eval_inputs.Transform(lambda t: t.shape) - self.assertEqual(input_shapes.audio, (200, 48000)) - - def testNumEpochs(self): - p = ToyInputGenerator.Params() - p.batch_size = 3 - p.num_epochs = 7 - self._tmpdir, p.input_files = _CreateFakeTFRecordFiles( - record_count=p.batch_size) - p.dataset_type = tf.data.TFRecordDataset - p.randomize_order = False - p.parallel_readers = 1 - ig = p.Instantiate() - with self.session(graph=tf.get_default_graph()): - inputs = ig.GetPreprocessedInputBatch() - for _ in range(p.num_epochs): - eval_inputs = self.evaluate(inputs) - self.assertEqual(eval_inputs.audio.shape, (p.batch_size, 48000)) - with self.assertRaisesRegex(tf.errors.OutOfRangeError, 'End of sequence'): - self.evaluate(inputs) - - def testRespectsInfeedBatchSize(self): - p = ToyInputGenerator.Params() - p.batch_size = 3 - self._tmpdir, p.input_files = _CreateFakeTFRecordFiles() - p.dataset_type = tf.data.TFRecordDataset - - ig = p.Instantiate() - batch = ig.GetPreprocessedInputBatch() - self.assertEqual(batch.audio.shape[0], p.batch_size) - self.assertEqual(p.batch_size, ig.InfeedBatchSize()) - - tf.reset_default_graph() - ig = p.Instantiate() - with mock.patch.object( - ig, 'InfeedBatchSize', return_value=42) as mock_method: - batch = ig.GetPreprocessedInputBatch() - self.assertEqual(batch.audio.shape[0], 42) - mock_method.assert_called() - - -# Dataset pipelines for TFDataInputTest. -def _TestDatasetFn(begin=0, end=10): - """Test tf.data pipeline.""" - ds = tf.data.Dataset.from_tensor_slices(tf.range(begin, end)) - return ds.map(lambda x: {'value': x}) - - -def _TestDatasetFnWithoutDefault(begin, end=10): - """Test tf.data pipeline with non-defaulted parameters.""" - ds = tf.data.Dataset.from_tensor_slices(tf.range(begin, end)) - return ds.map(lambda x: {'value': x}) - - -def _TestDatasetFnWithRepeat(begin=0, end=10): - """Test tf.data pipeline with repeat.""" - ds = tf.data.Dataset.from_tensor_slices(tf.range(begin, end)).repeat() - return ds.map(lambda x: {'value': x}) - - -def _TestDatasetFnV1(begin=0, end=10): - """Similar to _TestDatasetFn but returns TFv1's dataset explicitly.""" - ds = tf1.data.Dataset.from_tensor_slices(tf.range(begin, end)) - return ds.map(lambda x: {'value': x}) - - -def _TestDatasetFnV2(begin=0, end=10): - """Similar to _TestDatasetFn but returns TFv2's dataset explicitly.""" - ds = tf2.data.Dataset.from_tensor_slices(tf.range(begin, end)) - return ds.map(lambda x: {'value': x}) - - -class _TestDatasetClass: - """A class that generates tf.data by its member function.""" - - def __init__(self, begin): - self._begin = begin - - def DatasetFn(self, end=10): - ds = tf.data.Dataset.from_tensor_slices(tf.range(self._begin, end)) - return ds.map(lambda x: {'value': x}) - - -# A class object which will be instantiated at importing the module. -# It can be used in DefineTFDataInput(). -_TestDatasetObject = _TestDatasetClass(begin=0) - -# InputGenerators for TFDataInputTest. -_TestTFDataInput = base_input_generator.DefineTFDataInput( - '_TestTFDataInput', _TestDatasetFn) -_TestTFDataInputWithIgnoreArgs = base_input_generator.DefineTFDataInput( - '_TestTFDataInputWithIgnoreArgs', _TestDatasetFn, ignore_args=('begin',)) -_TestTFDataInputWithMapArgs = base_input_generator.DefineTFDataInput( - '_TestTFDataInputWithMapArgs', - _TestDatasetFn, - map_args={'end': 'num_samples'}) -_TestTFDataInputWithoutDefault = base_input_generator.DefineTFDataInput( - '_TestTFDataInputWithoutDefault', _TestDatasetFnWithoutDefault) -_TestTFDataInputWithRepeat = base_input_generator.DefineTFDataInput( - '_TestTFDataInputWithRepeat', _TestDatasetFnWithRepeat) -_TestTFDataInputWithBoundMethod = base_input_generator.DefineTFDataInput( - '_TestTFDataInputWithBoundMethod', _TestDatasetObject.DatasetFn) -_TestTFDataInputV1 = base_input_generator.DefineTFDataInput( - '_TestTFDataInputV1', _TestDatasetFnV1) -_TestTFDataInputV2 = base_input_generator.DefineTFDataInput( - '_TestTFDataInputV2', _TestDatasetFnV2) - - -class TFDataInputTest(test_utils.TestCase): - - def testModule(self): - self.assertEqual(_TestTFDataInput.__module__, '__main__') - self.assertEqual(_TestTFDataInputWithIgnoreArgs.__module__, '__main__') - self.assertEqual(_TestTFDataInputWithMapArgs.__module__, '__main__') - self.assertEqual(_TestTFDataInputWithoutDefault.__module__, '__main__') - self.assertEqual(_TestTFDataInputWithRepeat.__module__, '__main__') - self.assertEqual(_TestTFDataInputWithBoundMethod.__module__, '__main__') - self.assertEqual(_TestTFDataInputV1.__module__, '__main__') - self.assertEqual(_TestTFDataInputV2.__module__, '__main__') - - def testExample(self): - """Tests the example code in the function docstring.""" - p = _TestTFDataInput.Params() - self.assertIn('args', p) - self.assertIn('begin', p.args) - self.assertIn('end', p.args) - self.assertEqual(p.args.begin, 0) - self.assertEqual(p.args.end, 10) - - ig = p.Instantiate() - self.assertIsInstance(ig, _TestTFDataInput) - - with self.session(graph=tf.get_default_graph()) as sess: - data = ig.GetPreprocessedInputBatch() - self.assertIsInstance(data, py_utils.NestedMap) - self.assertIsInstance(data.value, tf.Tensor) - self.assertAllEqual(data.value.shape, ()) - self.assertEqual(data.value.dtype, tf.int32) - - # Consumes all data. - for i in range(p.args.begin, p.args.end): - self.assertEqual(sess.run(data).value, i) - - with self.assertRaises(tf.errors.OutOfRangeError): - sess.run(data) - - def testToFromProto(self): - """Similar to `testExample` but params will be restored from a proto.""" - serialized_proto = _TestTFDataInput.Params().ToProto() - p = hyperparams.Params.FromProto(serialized_proto) - self.assertIn('args', p) - self.assertIn('begin', p.args) - self.assertIn('end', p.args) - self.assertEqual(p.args.begin, 0) - self.assertEqual(p.args.end, 10) - - ig = p.Instantiate() - self.assertIsInstance(ig, _TestTFDataInput) - - with self.session(graph=tf.get_default_graph()) as sess: - data = ig.GetPreprocessedInputBatch() - self.assertIsInstance(data, py_utils.NestedMap) - self.assertIsInstance(data.value, tf.Tensor) - self.assertAllEqual(data.value.shape, ()) - self.assertEqual(data.value.dtype, tf.int32) - - # Consumes all data. - for i in range(p.args.begin, p.args.end): - self.assertEqual(sess.run(data).value, i) - - with self.assertRaises(tf.errors.OutOfRangeError): - sess.run(data) - - def testWithIgnoreArgs(self): - """Tests the `ignore_args` parameter.""" - p = _TestTFDataInputWithIgnoreArgs.Params() - self.assertIn('args', p) - self.assertNotIn('begin', p.args) - self.assertIn('end', p.args) - self.assertEqual(p.args.end, 10) - - ig = p.Instantiate() - self.assertIsInstance(ig, _TestTFDataInputWithIgnoreArgs) - - with self.session(graph=tf.get_default_graph()) as sess: - data = ig.GetPreprocessedInputBatch() - self.assertIsInstance(data, py_utils.NestedMap) - self.assertIsInstance(data.value, tf.Tensor) - self.assertAllEqual(data.value.shape, ()) - self.assertEqual(data.value.dtype, tf.int32) - - # Consumes all data. - for i in range(p.args.end): - self.assertEqual(sess.run(data).value, i) - - with self.assertRaises(tf.errors.OutOfRangeError): - sess.run(data) - - def testWithMapArgs(self): - """Tests the `map_args` parameter.""" - p = _TestTFDataInputWithMapArgs.Params() - self.assertIn('args', p) - self.assertIn('num_samples', p) # Defined by BaseInputGenerator. - self.assertIn('begin', p.args) - self.assertNotIn('end', p.args) - self.assertEqual(p.num_samples, 0) - self.assertEqual(p.args.begin, 0) - - p.num_samples = 20 - ig = p.Instantiate() - self.assertIsInstance(ig, _TestTFDataInputWithMapArgs) - - with self.session(graph=tf.get_default_graph()) as sess: - data = ig.GetPreprocessedInputBatch() - self.assertIsInstance(data, py_utils.NestedMap) - self.assertIsInstance(data.value, tf.Tensor) - self.assertAllEqual(data.value.shape, ()) - self.assertEqual(data.value.dtype, tf.int32) - - # Consumes all data. - for i in range(p.args.begin, p.num_samples): - self.assertEqual(sess.run(data).value, i) - - with self.assertRaises(tf.errors.OutOfRangeError): - sess.run(data) - - def testWithoutDefault(self): - """Tests parameters without defaults.""" - p = _TestTFDataInputWithoutDefault.Params() - self.assertIn('args', p) - self.assertIn('begin', p.args) - self.assertIn('end', p.args) - self.assertIsNone(p.args.begin) - self.assertEqual(p.args.end, 10) - - p.args.begin = 0 - ig = p.Instantiate() - self.assertIsInstance(ig, _TestTFDataInputWithoutDefault) - - with self.session(graph=tf.get_default_graph()) as sess: - data = ig.GetPreprocessedInputBatch() - self.assertIsInstance(data, py_utils.NestedMap) - self.assertIsInstance(data.value, tf.Tensor) - self.assertAllEqual(data.value.shape, ()) - self.assertEqual(data.value.dtype, tf.int32) - - # Consumes all data. - for i in range(p.args.begin, p.args.end): - self.assertEqual(sess.run(data).value, i) - - with self.assertRaises(tf.errors.OutOfRangeError): - sess.run(data) - - def testWithRepeat(self): - """Tests if the repeated dataset runs forever.""" - p = _TestTFDataInputWithRepeat.Params() - self.assertIn('args', p) - self.assertIn('begin', p.args) - self.assertIn('end', p.args) - self.assertEqual(p.args.begin, 0) - self.assertEqual(p.args.end, 10) - - ig = p.Instantiate() - self.assertIsInstance(ig, _TestTFDataInputWithRepeat) - - with self.session(graph=tf.get_default_graph()) as sess: - data = ig.GetPreprocessedInputBatch() - self.assertIsInstance(data, py_utils.NestedMap) - self.assertIsInstance(data.value, tf.Tensor) - self.assertAllEqual(data.value.shape, ()) - self.assertEqual(data.value.dtype, tf.int32) - - # Runs the dataset several times: it should not raise OutOfRangeError. - for _ in range(3): - for i in range(p.args.begin, p.args.end): - self.assertEqual(sess.run(data).value, i) - - def testWithBoundMethod(self): - """Tests pipeline defined by a bound method: member function with self.""" - p = _TestTFDataInputWithBoundMethod.Params() - self.assertIn('args', p) - self.assertNotIn('begin', p.args) - self.assertIn('end', p.args) - self.assertEqual(p.args.end, 10) - - ig = p.Instantiate() - self.assertIsInstance(ig, _TestTFDataInputWithBoundMethod) - - with self.session(graph=tf.get_default_graph()) as sess: - data = ig.GetPreprocessedInputBatch() - self.assertIsInstance(data, py_utils.NestedMap) - self.assertIsInstance(data.value, tf.Tensor) - self.assertAllEqual(data.value.shape, ()) - self.assertEqual(data.value.dtype, tf.int32) - - # Consumes all data. - for i in range(p.args.end): - self.assertEqual(sess.run(data).value, i) - - with self.assertRaises(tf.errors.OutOfRangeError): - sess.run(data) - - def testDatasetV1(self): - """Tests the TFv1 Dataset.""" - p = _TestTFDataInputV1.Params() - self.assertIn('args', p) - self.assertIn('begin', p.args) - self.assertIn('end', p.args) - self.assertEqual(p.args.begin, 0) - self.assertEqual(p.args.end, 10) - - ig = p.Instantiate() - self.assertIsInstance(ig, _TestTFDataInputV1) - - with self.session(graph=tf.get_default_graph()) as sess: - data = ig.GetPreprocessedInputBatch() - self.assertIsInstance(data, py_utils.NestedMap) - self.assertIsInstance(data.value, tf.Tensor) - self.assertAllEqual(data.value.shape, ()) - self.assertEqual(data.value.dtype, tf.int32) - - # Consumes all data. - for i in range(p.args.begin, p.args.end): - self.assertEqual(sess.run(data).value, i) - - with self.assertRaises(tf.errors.OutOfRangeError): - sess.run(data) - - def testDatasetV2(self): - """Tests the TFv2 Dataset.""" - p = _TestTFDataInputV2.Params() - self.assertIn('args', p) - self.assertIn('begin', p.args) - self.assertIn('end', p.args) - self.assertEqual(p.args.begin, 0) - self.assertEqual(p.args.end, 10) - - ig = p.Instantiate() - self.assertIsInstance(ig, _TestTFDataInputV2) - - # We keep the TFv1's Session here since v1/v2 behaviors would not coexist. - # TODO(oday): write TFv2-specific tests. - with self.session(graph=tf.get_default_graph()) as sess: - data = ig.GetPreprocessedInputBatch() - self.assertIsInstance(data, py_utils.NestedMap) - self.assertIsInstance(data.value, tf.Tensor) - self.assertAllEqual(data.value.shape, ()) - self.assertEqual(data.value.dtype, tf.int32) - - # Consumes all data. - for i in range(p.args.begin, p.args.end): - self.assertEqual(sess.run(data).value, i) - - with self.assertRaises(tf.errors.OutOfRangeError): - sess.run(data) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/base_layer.py b/lingvo/core/base_layer.py deleted file mode 100644 index 77501beb..00000000 --- a/lingvo/core/base_layer.py +++ /dev/null @@ -1,980 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Base class for all layers.""" - -import abc -import collections -import enum -import itertools -import re -import lingvo.compat as tf -from lingvo.core import cluster_factory -from lingvo.core import hyperparams -from lingvo.core import py_utils - -FLAGS = tf.flags.FLAGS - -_LAYER_STACK = py_utils.ThreadLocalStack() -_CREATE_VARIABLES_STACK = py_utils.ThreadLocalStack() - - -class Accumulator: - """Layers can register accumulators to persist step-level state. - - Accumulators must be represented by a Tensor of a fixed shape. The default - value must be supplied by overriding DefaultValue(). It is important that - the default tensor value is created on each call in order to avoid - accumulators leaking to different graphs. - - Accumulators can be enabled (default) or disabled by pairing - Disable()/Enable() calls. When disabled, the accumulator will only return - the default value and will silently drop calls to SetValue(). When computing - gradients that may touch accumulators, calls should be bracketed with - Disable()/Enable(). - - Care must be taken when manipulating accumulators across Defun boundaries. - Typically, values for all accumulators in a layer must be explicitly - retrieved and passed in to the Defun scope by calling - layer.GetAccumulatorValues(), marshalling into the Defun and setting them - via layer.SetAccumulatorValues(). The reverse must be done on return. - """ - - def __init__(self): - # None for initial value or the current Tensor value. - self._value = None - self._disable_count = 0 - - @property - def is_disabled(self): - """Whether the accumulator is disabled.""" - return self._disable_count > 0 - - def Disable(self): - """Disables the accumulator (must be balanced with Enable).""" - self._disable_count += 1 - - def Enable(self): - """Enables the accumulator (must balance a Disable).""" - assert self._disable_count > 0, 'Unbalanced Accumulator Enable/Disable' - self._disable_count -= 1 - - def GetValue(self): - """Gets the current value of the accumulator Tensor.""" - if self.is_disabled or self._value is None: - return self.DefaultValue() - else: - return self._value - - def SetValue(self, value): - """Sets the current value of the accumulator Tensor.""" - if not self.is_disabled: - self._value = value - - def Reset(self): - """Resets the accumulator to its default value.""" - if not self.is_disabled: - self._value = None - - def DefaultValue(self): - raise NotImplementedError('DefaultValue must be implemented') - - -def _BaseLayerInitWrapper(func): # pylint: disable=invalid-name - """A decorator for layer's __init__. - - Args: - func: The __init__ method of `BaseLayer`'s subclasses. - - Returns: - A decorator wrapper for layer's initializer. Note that this wrapper can - be called multiple times for the same layer instance, once for each - __init__() for classes on the class hierarchy. - """ - - def Wrapper(self, *args, **kwargs): - """Decorator wrapper fn.""" - stack = _LAYER_STACK.stack - if stack and stack[-1] is self: - # Short circuit if called multiple times (eg. super() chain). - func(self, *args, **kwargs) - return - - # Push back self (the current layer) to the stack. - stack_size = len(stack) - stack.append(self) - try: - # Calls the layer's real __init__ method. - func(self, *args, **kwargs) - if len(stack) > 1: - # Records the fact stack[-2] just created a sub-layer self. - stack[-2]._AutoAddChild(self) # pylint: disable=protected-access - finally: - # Pop out self (the current layer). - assert stack[-1] is self - stack.pop() - assert len(stack) == stack_size - - if not stack: - # Outermost layer just finished __init__. - if self.cluster.immediately_create_variables: - self.InstantiateVariables() - - return Wrapper - - -def RecursiveFindLayerParams(params): - """Returns all params that define a layer.""" - if not isinstance(params, hyperparams.Params): - return [] - layer_params = [] - if hasattr(params, 'cls') and issubclass(params.cls, BaseLayer): - layer_params.append(params) - for _, p in params.IterParams(): - if isinstance(p, (list, tuple)): - for item in p: - layer_params.extend(RecursiveFindLayerParams(item)) - elif isinstance(p, dict): - for item in p.items(): - layer_params.extend(RecursiveFindLayerParams(item)) - else: - layer_params.extend(RecursiveFindLayerParams(p)) - return layer_params - - -class BaseLayerMeta(type): - """Metaclass tracking child layers and variable initialization.""" - - # pylint: disable=bad-mcs-classmethod-argument - def __new__(mcs, name, bases, dct): - cls = super(BaseLayerMeta, mcs).__new__(mcs, name, bases, dct) - if '__init__' not in dct: - - def TrivialInit(self, params): - super(cls, self).__init__(params) # pylint: disable=bad-super-call - - cls.__init__ = TrivialInit - - cls.__init__ = _BaseLayerInitWrapper(cls.__init__) - return cls - # pylint: enable=bad-mcs-classmethod-argument - - def __call__(cls, *args, **kwargs): - self = super().__call__(*args, **kwargs) - # This happens after self.__init__() - # pylint: disable=protected-access - self._disable_create_child = True - self._VerifyChildren() - # pylint: enable=protected-access - return self - - -class ABCLayerMeta(BaseLayerMeta, abc.ABCMeta): - pass - - -# NamedTuple that records the metadata for creating a variable. -# For internal use only. Subclasses of BaseLayer should use -# self.CreateVariable() to create variables. -CreateVariableMeta = collections.namedtuple( - 'CreateVariableMeta', ['var_params', 'theta_fn', 'kwargs']) - - -class _CreateLayerVariablesStatus(enum.Enum): - NOT_CALLED = 1 - IN_PROGRESS = 2 - COMPLETED = 3 - - -LAYER_WT = 'layer_weight_variable' - - -class BaseLayer(tf.Module, metaclass=BaseLayerMeta): - r"""Base class for all the layer object. - - As this BaseLayer is a proper sub-class of tf.Module, it supports proper - tracking and reflection of key constituents such as variables and submodules. - - self.submodules returns a list of submodules that are reachable through - recursive member access from self. - - self.variables returns a list of Variables that are reachable through - recursive member access from self. - - self(\*args, \*\*kwargs) carries out computation on the input args and kwargs. - """ - - # Set to an inference driver name if this is an inference specialization - # class. - _INFERENCE_DRIVER_NAME = None - - @classmethod - def Params(cls): - """Returns the layer params.""" - p = hyperparams.InstantiableParams(cls) - p.Define('inference_driver_name', cls._INFERENCE_DRIVER_NAME, - 'Name of the inference driver used to construct this layer.') - p.Define('name', '', 'Name of this layer object.') - p.Define('dtype', tf.float32, 'Datatype to use.') - # None value will make FProp use dtype instead of fprop_dtype. - # TODO(lepikhin): all @tf.Defun should use p.fprop_dtype if it is set. - p.Define('fprop_dtype', None, 'Activations datatype to use.') - p.Define( - 'random_seed', None, 'Random seed for deterministic unittests. This ' - 'is inherited by child layers if they do not set a random_seed.') - p.Define('vn', py_utils.DefaultVN(), - 'How variational noise should be applied.') - p.Define( - 'params_init', py_utils.DefaultParamInit(), - 'How model weights should be initialized. Not to be confused with ' - 'hyperparams.') - # Makes additional alterations for graphs being used for inference. - p.Define('is_inference', None, 'True if in inference mode.') - # In addition to is_inference, indicate that the inference graph is - # for a single step. - p.Define( - 'allow_implicit_capture', None, - 'When using Defuns, code often asserts that the Defun does not ' - 'capture undeclared inputs. This eliminates a source of bugs ' - 'at the expense of making some kinds of models or utilities ' - 'hard/impossible to use. Setting this to True/False (versus None) ' - 'causes the setting to apply to this layer and its children.') - p.Define( - 'skip_lp_regularization', None, - 'If True, all variables in this layer will skip Lp regularization. ' - 'If None/False, only variables explicitly in the ' - 'SKIP_LP_REGULARIZATION collection will skip Lp regularization. ' - 'Also propagated to child layers with default settings (None).') - return p - - @staticmethod - def CopyBaseParams(from_params, to_params): - """Copies BaseLayer params from `from_params` to `to_params`.""" - assert issubclass(from_params.cls, BaseLayer) - assert issubclass(to_params.cls, BaseLayer) - # Copy-over the BaseLayer params. - if to_params.dtype == tf.float32: - to_params.dtype = from_params.dtype - if from_params.fprop_dtype is not None: - to_params.fprop_dtype = from_params.fprop_dtype - if to_params.random_seed is None: - to_params.random_seed = from_params.random_seed - if to_params.is_inference is None: - to_params.is_inference = from_params.is_inference - if to_params.allow_implicit_capture is None: - to_params.allow_implicit_capture = from_params.allow_implicit_capture - if to_params.skip_lp_regularization is None: - to_params.skip_lp_regularization = from_params.skip_lp_regularization - - # Only copy from base when vn config is using the default setting. - if to_params.vn == py_utils.DefaultVN(): - to_params.vn = from_params.vn.Copy() - - # TODO(rpang): derive to_params.params_init.seed from - # from_params.params_init.seed if it is specified in 'from_params' and not - # in 'to_params'. - if py_utils.IsDefaultParamInit(to_params.params_init): - # Copy over params_init as well. - to_params.params_init = from_params.params_init.Copy() - return to_params - - def __init__(self, params): - """Layer constructor. - - Args: - params: A params used to construct this layer. - """ - assert params.name, ( - 'Layer params for %s must have a "name"' % self.__class__.__name__) - - tf_module_name = params.name - tf_module_name = re.sub('[^a-zA-Z0-9_]+', '_', tf_module_name) - tf_module_name = 'bbf_' + self.__class__.__name__ + '_' + tf_module_name - py_utils.NestedMap.CheckKey(tf_module_name) - - # initialize the base class. - super().__init__(tf_module_name) - - # Note AutoTracking doesn't work properly due to its inability to walk - # through py_utils.NestedMap data structures which are used widely - # throughout the Lingvo codebase. Also there seems to be some performance - # hit in turning on auto-tracking in constructing graphs. For now, we - # disable auto-tracking. - # TODO(lingvo): Re-enable auto-tracking when fuller support is - # added for key data structures used in Lingvo, and performance issue is - # debugged more and understood better. - self._setattr_tracking = False - - self._parent = None - for parent in reversed(_LAYER_STACK.stack): - if parent is not self: - self._parent = parent - break - self._params = params.Copy() - tf.logging.debug('Creating layer %s with params: \n %s \n', - self.__class__.__name__, str(params)) - # Vars created by this layer. - self._private_vars = py_utils.NestedMap() - # Theta derived from this layer's vars. - self._private_theta = py_utils.NestedMap() - # Child layers created by this layer through CreateChild/CreateChildren. - self._private_children = py_utils.NestedMap() - # Child layers created by this layer. A well-formed layer should - # have self._private_children equals to self._children_list. I.e., - # all child layers are created using CreateChild/CreateChildren. - self._children_list = [] - # Extra theta's not directly correspond to any underlying vars. For example, - # the concatenated sharded variables. - self._extra_theta = py_utils.NestedMap() - # All registered accumulators. - self._private_accumulators = py_utils.NestedMap() - # Layer-private functions. Add with AddFunction. - self._private_fns = dict() - # Mapping from variable names to its symbolic shape. - # self._var_symbolic_shape_map['var_name'] will be a tuple of integers or - # symbolic expressions, one for each dimension of the variable. - self._var_symbolic_shape_map = {} - - self._is_variable_free = False - self._variables_to_create = {} - self._create_variables_status = _CreateLayerVariablesStatus.NOT_CALLED - - def SetVariableFree(self, value=True): - """Marks this layer as having no variables. - - Note that this status affects sublayers and child layers too. - - Args: - value: True to set layer as variable free. - """ - if self._create_variables_status != _CreateLayerVariablesStatus.NOT_CALLED: - raise ValueError( - 'Variable free status for %s must be set before InstantiateVariables().' - % self.params.cls) - if self._variables_to_create: - raise ValueError('Cannot set layer %s with variables as variable free.' % - self.params.cls) - self._is_variable_free = value - - def FPropDefaultTheta(self, *args, **kwargs): - """Calls `FProp`.""" - return self.FProp(self.theta, *args, **kwargs) - - def __call__(self, *args, **kwargs): - """Forwards call to FPropDefaultTheta.""" - return self.FPropDefaultTheta(*args, **kwargs) - - def FProp(self, theta, *args, **kwargs): - """Forward propagation. - - The central interface that subclasses should implement. The caller - calls `FProp` with a `theta` dictionary. E.g.:: - - foo = InstanceOfASubClassOfFoo(params) - y = foo.FProp(foo.theta, x) - - The implementation of `FProp()` computes a function given - the theta and the inputs. E.g.:: - - subs = self.children - inputs = args[0] - a0 = subs.linear.FProp(theta.linear, inputs) - a1 = subs.softmax.FProp(theta.softmax, a0) - # The same layer applied twice. - a2 = subs.linear.FProp(theta.linear, a1) - return a2 - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - *args: List args. - **kwargs: Keyward args. - """ - del theta - del args - del kwargs - raise NotImplementedError('Abstract method of %s' % self) - - @classmethod - def FPropMeta(cls, params, *args, **kwargs): - """Returns metadata about the `FProp` computation for this layer. - - **Experimental feature.** - Don't use or depend on it without consulting Lingvo authors. - - E.g.:: - - p = SomeComplexLayer.Params() - meta = p.cls.FPropMeta(p, tshape.Shape([128, 20, 50, 'channels'])) - - `meta.flops` gives an estimate count of floating point operations done by - one `FProp` given an input tensor of shape [128, 20, 50, channels]. - `meta.out_shapes` is a tuple of TShape, which tells you what shape - of tensors this layer will return. - - Args: - params: The param of a layer of this layer type. - *args: Corresponds to FProp with Tensors replaced by `TensorShape`. - **kwargs: Corresponds to FProp with Tensors replaced by `TensorShape`. - - Returns: - A `.NestedMap` with - - - flops - The estimated number of floating point operations incurred by - this fprop. - - out_shapes - A tuple of `TShape`. I.e., `out_shapes[i]` - represents the shape of the `i`-th returned tensor of the fprop. - """ - raise NotImplementedError('FPropMeta of %s' % cls) - - @property - def params(self): - """Returns the params upon which this layer is built.""" - return self._params - - @property - def cluster(self): - """Returns the current cluster configuration.""" - return cluster_factory.Current() - - @property - def do_eval(self): - return self.cluster.do_eval - - @property - def parent(self): - """None if self is the root layer, otherwise the parent layer of self.""" - return self._parent - - @property - def path(self): - """Returns a '.'-separated string with all layer names from the root.""" - if self.parent: - return self.parent.path + '.' + self.params.name - else: - return self.params.name - - @property - def layer_type(self): - """Returns layer type prefixed with 'lingvo.'.""" - return 'lingvo.' + self.__class__.__name__ - - @property - def children(self): - """Returns children layers of this layer in a `.NestedMap`.""" - return self._private_children - - @property - def global_step(self): - if self._create_variables_status == _CreateLayerVariablesStatus.NOT_CALLED: - raise ValueError( - 'Cannot access global_step before InstantiateVariables().') - return self._global_step - - def __getattr__(self, name): - """Returns the child layer of the given name.""" - if name == '_private_children': - # Raising AttributeError without custom message triggers normal python - # handling of __getattr__ AttributeError. - raise AttributeError() - if name in self._private_children: - return self._private_children[name] - elif (hasattr(type(self), name) and - isinstance(getattr(type(self), name), property)): - # There was an AttributeError raised by a property getter. - # Call property getter again directly to raise the same error. - return getattr(type(self), name).fget(self) - else: - raise AttributeError('%s is not a sub-layer of %s.' % (name, self)) - - def GetDescendant(self, path): - """Returns a descendant layer given the path. - - NOTE(yonghui): This GetDescendant is not complete. It is not able to descent - into list/tuple substructures. - - Args: - path: a comma separated string denoting a descendant of this layer. - - Returns: - The descendant layer. - - Raises: - KeyError: if the descendant is not found. - """ - sub = self - if path: - for k in path.split('.'): - if k not in sub.children: - raise KeyError('%s not found in %s' % (k, list(sub.children.keys()))) - sub = sub.children[k] - return sub - - @property - def vars(self): - """Returns variables of this layer and its children in a `.NestedMap`.""" - if self._is_variable_free: - return self._private_children.Transform(lambda _: py_utils.NestedMap()) - if self._create_variables_status == _CreateLayerVariablesStatus.NOT_CALLED: - raise ValueError( - 'Cannot access vars for layer %s before they have been created.' % - self.params.cls) - ret = self._private_children.Transform(lambda x: x.vars) - for k in self._private_vars.keys(): - ret[k] = self._private_vars[k] - return ret - - @property - def theta(self): - """Returns theta of this layer and its children in a `.NestedMap`.""" - if self._is_variable_free: - return self._private_children.Transform(lambda _: py_utils.NestedMap()) - if self._create_variables_status == _CreateLayerVariablesStatus.NOT_CALLED: - raise ValueError( - 'Cannot access theta for layer %s before they have been created.' % - self.params.cls) - ret = self._private_children.Transform(lambda x: x.theta) - - private_theta = self._private_theta - - if (self._params.fprop_dtype is not None and - self._params.fprop_dtype != self._params.dtype): - - def MaybeCastToFPropDtype(x): - if x is not None and x.dtype == self._params.dtype: - return tf.cast(x, self._params.fprop_dtype) - else: - return x - - private_theta = private_theta.Transform(MaybeCastToFPropDtype) - - ret.update(private_theta) - return ret - - @property - def accumulators(self): - """Returns `.NestedMap` of `Accumulator` instances for this and children.""" - ret = self._private_children.Transform(lambda x: x.accumulators) - for k, acc in self._private_accumulators.items(): - ret[k] = acc - return ret - - @property - def fns(self): - """Returns a read-only view of layer local functions. - - Functions can be accessed by index (['name']) or attribute notation - (`fns.foo`). - - Returns: - Read-only attribute accessible dict view of the layer's function library. - """ - return py_utils.ReadOnlyAttrDictView(self._private_fns) - - def AddFunction(self, name, f, replace=False): - """Adds a function to the layer's `fns` collection. - - This should be used to add op-like functions specific to the operation - of the layer and its children. Such functions should be added in `__init__` - and may either be raw python functions or TensorFlow Defuns. This - facility is just a mechanism for organizing them and having basic checks - on name collisions. - - Args: - name: The function name. It will be accessible as `self.fns.{name}`. - f: The function body. - replace: Whether to replace an existing function (default False). - - Raises: - AttributeError: If the function already exists and replace == False. - """ - py_utils.NestedMap.CheckKey(name) - if not replace: - if name in self._private_fns: - raise AttributeError( - 'Function "%s" is already defined on layer "%r"' % (name, self)) - self._private_fns[name] = f - - def _CheckName(self, name): - """Asserts name's validity.""" - py_utils.NestedMap.CheckKey(name) - assert name not in self._private_vars, ( - '%s exists in vars, %s' % (name, list(self._private_vars.keys()))) - assert name not in self._private_theta, ( - '%s exists in theta, %s' % (name, list(self._private_theta.keys()))) - assert name not in self._private_children, ('%s exists in children, %s' % ( - name, list(self._private_children.keys()))) - assert name not in self._private_accumulators, ( - '%s exists in global_accumulator: %s' % - (name, list(self._private_accumulators.keys()))) - - def _VariableCollections(self): - return [LAYER_WT, '%s_vars' % self.__class__.__name__] - - def RegisterAccumulator(self, name, acc): - """Registers an accumulator for this layer. - - An accumulator is used to propagate some state to a future point, - where it is acted on (typically as part of `PostTrainingStepUpdate`). This - mechanism allows for arbitrarily nested parts of a model to export state - back to the global scope. Accumulators must be specially handled - when crossing into `Defun` or recurrent scopes. By abstracting the - mechanism, it allows all such state to be handled uniformly and generically. - - Example (typically from `__init__`):: - - class MyAccumulator(base_layer.Accumulator): - def DefaultValue(self): - # [count, min, max] - return tf.convert_to_tensor([0.0, 0.0, 0.0]) - def Update(self, state1): - state0 = self.GetValue() - self.SetValue(tf.stack([ - state0[0] + state1[0], - tf.minimum(state0[1], state1[1]), - tf.maximum(state0[2], state1[2])])) - - self.RegisterAccumulator('mytracker', acc) - - Later, access the current value and update it:: - - acc = self.accumulators.mytracker - acc.Update(tf.convert_to_tensor([1.0, batch_min, batch_max])) - - Then, typically in `PostTrainingStepUpdate`:: - - acc = self.accumulator.mytracker - acc_value = acc.GetValue() - # Do something with the value. - acc.Reset() - - Args: - name: The accumulator name. Shares a namespace with children, vars and - extra theta. - acc: An `Accumulator` instance. - """ - self._CheckName(name) - self._private_accumulators[name] = acc - - def GetAccumulatorValues(self): - """Recursively gets values of all accumulators. - - Returns: - `.NestedMap` of Tensors for each registered accumulator. - """ - return self.accumulators.Transform(lambda acc: acc.GetValue()) - - def SetAccumulatorValues(self, new_values_nmap): - """Recursively sets the values of all accumulators from a map. - - Args: - new_values_nmap: `.NestedMap` of accumulator name:Tensor. - """ - accumulator_list = self.accumulators.Flatten() - value_list = new_values_nmap.Flatten() - for acc, value in zip(accumulator_list, value_list): - acc.SetValue(value) - - def GetVariableSymbolicShape(self, var_name): - """Returns the variable's symbolic shape.""" - return self._var_symbolic_shape_map.get(var_name, None) - - def CreateVariable(self, name, var_params, theta_fn=None, **kwargs): - """Create a variable of this layer according to the parameter `var_params`. - - E.g.:: - - def __init__(self, ...): # A layer's constructor - self.CreateVariable( - 'weight', py_utils.WeightParams(shape=[100, 100])) - - `theta_fn` is used to apply a simple transformation on the created - variable's value before used by the forward computation. E.g., to - add the global variational noise according to this layer's - parameter, one can do:: - - def __init__(self, ...): # A layer's constructor - self.CreateVariable( - name='weight', - var_params=py_utils.WeightParams(shape=[100, 100]), - theta_fn=self.AddGlobalVN) - - In some contexts, eg. TPU training, variables may not be created immediately - but rather the creation request will be cached and created later via a call - to layer.InstantiateVariables(). - - Args: - name: Variable name which is used as the key into vars/theta. - var_params: `Params` used to create the variable. - theta_fn: A python function that takes a variable's value and returns a - new value to be used later for computation. Its signature must be - (tf.Tensor) -> (tf.Tensor). - **kwargs: Keyword args passed to `.py_utils.CreateVariable`. - """ - if self._is_variable_free: - raise ValueError('Cannot create variable in variable free layer.') - if self._create_variables_status == _CreateLayerVariablesStatus.COMPLETED: - raise ValueError( - 'CreateVariable call after variable creation has completed! ' - 'CreateVariable should be called in __init__ or _CreateLayerVariables.' - ) - self._CheckName(name) - if (self.params.skip_lp_regularization and - py_utils.SKIP_LP_REGULARIZATION not in var_params.collections): - var_params = py_utils.WeightParams( - shape=var_params.shape, - dtype=var_params.dtype, - init=var_params.init, - collections=(var_params.collections + - [py_utils.SKIP_LP_REGULARIZATION])) - self._var_symbolic_shape_map[name] = var_params.shape - meta = CreateVariableMeta( - var_params=var_params.Copy(), - theta_fn=theta_fn, - kwargs=kwargs) - if self._create_variables_status == _CreateLayerVariablesStatus.IN_PROGRESS: - # If InstantiateVariables has been called, create variable immediately. - self._CreateVariableInternal(name, meta) - else: - # Otherwise cache the variable to be created. - self._variables_to_create[name] = meta - - def _CreateVariableInternal(self, name, meta): - """Immediately creates the variable described by `meta`. - - DO NOT OVERRIDE. For internal use only. Subclasses of BaseLayer should use - self.CreateVariable() to create variables. - - Args: - name: The variable name. - meta: A CreateVariableMeta describing the variable to be created. - """ - meta.kwargs.setdefault('default_seed', self.params.random_seed) - var = py_utils.CreateVariable(name, meta.var_params, **meta.kwargs) - self._private_vars[name] = var - if FLAGS.no_identity_on_vars: - value = var - else: - with tf.device(var.device): - value = tf.identity(var) - if meta.theta_fn is not None: - value = meta.theta_fn(value) - self._private_theta[name] = value - - def InstantiateVariables(self): - """Create variables for this layer and child layers. - - DO NOT OVERRIDE. Override self._CreateLayerVariables instead. - """ - if self._create_variables_status != _CreateLayerVariablesStatus.NOT_CALLED: - return - self._create_variables_status = _CreateLayerVariablesStatus.IN_PROGRESS - - stack_size = len(_CREATE_VARIABLES_STACK.stack) - _CREATE_VARIABLES_STACK.stack.append(self) - try: - self._global_step = py_utils.GetGlobalStep() - self._CreateChildrenVariables() - - if not self._is_variable_free: - self.AddExtraTheta('global_step', self._global_step) - with tf.variable_scope( - py_utils.SanitizeScopeKey(self.params.name), - auxiliary_name_scope=False): - for name, meta in list(self._variables_to_create.items()): - self._CreateVariableInternal(name, meta) - self._CreateLayerVariables() - finally: - assert _CREATE_VARIABLES_STACK.stack[-1] is self - _CREATE_VARIABLES_STACK.stack.pop() - assert len(_CREATE_VARIABLES_STACK.stack) == stack_size - - self._create_variables_status = _CreateLayerVariablesStatus.COMPLETED - - if not _CREATE_VARIABLES_STACK.stack: - # Outermost layer just finished InstantiateVariables. - self._VerifyVarsAndTheta() - - def _CreateChildrenVariables(self): - """Create variables for child layers. - - Should be rarely overridden, only in cases when control over the context of - children InstantiateVariables calls are needed. eg, if children variables - need to be created inside of a specific context manager. - - There are a few cases of this in the codebase marked as for backwards - compability. This is only to ensure that variable scopes remain compatible - through the code migration. New layers should not copy that pattern, and - instead follow the standard pattern of self.CreateChild() in __init__() and - self.CreateVariable() in _CreateLayerVariables(). If you are okay with - breaking old checkpoints, you can go ahead and delete those functions. - """ - with tf.variable_scope( - py_utils.SanitizeScopeKey(self.params.name), - auxiliary_name_scope=False): - for child in self._children_list: - if self._is_variable_free and not child._is_variable_free: # pylint: disable=protected-access - raise ValueError( - 'Variable free layer %s(%s) child %s(%s) has variables.' % - (self.params.name, self.params.cls, child.params.name, - child.params.cls)) - child.InstantiateVariables() - - def _CreateLayerVariables(self): - """Actually create variables for this layer. - - Subclasses should override this function. - - Variables are created inside of tf.variable_scope(self.params.name). - """ - pass - - def AddExtraTheta(self, theta_name, theta_value): - """Add extra `theta` that doesn't directly correspond to `vars`.""" - self._CheckName(theta_name) - self._private_theta[theta_name] = theta_value - self._extra_theta[theta_name] = theta_value - - def AddGlobalVN(self, value): - return py_utils.AddGlobalVN(self.params, value) - - def CreateChild(self, name, params): - """Create a sub layer. - - The created sub layer can be accessed by `name`. E.g.:: - - self.CreateChild('foo', ...) - self.foo.FProp... - - or:: - - self.children['foo'].Fprop... - self.children.foo.Fprop... - - Args: - name: Sub layer name which is used as the key into vars/theta. - params: `Hyperparams` object to instantiate a layer. - """ - if hasattr(self, '_disable_create_child') and self._disable_create_child: - raise ValueError('Attempting to call CreateChild outside of __init__.') - self._CheckName(name) - if not params.name: - params.name = name - p = self.CopyBaseParams(self.params, params.Copy()) - child = p.Instantiate() - self._private_children[name] = child - - def CreateChildren(self, name, params): - """Create a list or dict of sub layers. - - The created sub layer list can be accessed by `name`. E.g.:: - - self.CreateChildren('foo', ...) - self.foo[10].FProp... - - or:: - - self.children['foo'][10].Fprop... - self.children.foo[10].Fprop... - - Args: - name: The name for the sub layers, which is used as the key into - vars/theta. - params: a list or dict of `Hyperparams` objects to create. - """ - if hasattr(self, '_disable_create_child') and self._disable_create_child: - raise ValueError('Attempting to call CreateChildren outside of __init__.') - self._CheckName(name) - - uid = itertools.count() - - def Instantiate(p): - p = self.CopyBaseParams(self.params, p.Copy()) - if not p.name: - p.name = '%s_%d' % (name, next(uid)) - return p.Instantiate() - - self._private_children[name] = py_utils.NestedMap( - sub=params).Transform(Instantiate).sub - - def AddChild(self, name, children): - """Add existing layer or layers as sublayer.""" - for child in py_utils.Flatten(children): - assert isinstance(child, BaseLayer) - self._CheckName(name) - self._private_children[name] = children - - def _AutoAddChild(self, child): - """Record that a layer `child` is instantiated by this layer. - - This method should only be called internally by BaseLayerMeta. - - Args: - child: A sub-layer of this layer. - """ - self._children_list.append(child) - - def _VerifyChildren(self): - """Verify all children created by this layer are via `CreateChild(ren)`.""" - created_children = self._private_children.Flatten() - for v in self._children_list: - if v not in created_children: - tf.logging.info([ - (child.params.name, type(child)) for child in created_children - ]) - raise ValueError( - '%s is not created by BaseLayer.CreateChild(ren) in %r.' % - (v.params.name, self)) - - def _VerifyVarsAndTheta(self): - """Verify that vars and theta have the same nested structure.""" - for child in self._children_list: - child._VerifyVarsAndTheta() # pylint: disable=protected-access - - def MatchKeys(x, y): - assert len(x) <= len(y) - for k in x.keys(): - assert k in y, '%s not in %s.' % (k, y) - if isinstance(x[k], py_utils.NestedMap): - assert isinstance(y[k], py_utils.NestedMap), '%s is not a map' % y[k] - MatchKeys(x[k], y[k]) - - # NOTE: this check can be quadratically expensive. Maybe only - # enable this in unittests. - MatchKeys(self.vars, self.theta) - - # Make sure whatever not in self.vars are in self._extra_theta - for k in self.theta.keys(): - assert k in self.vars or k in self._extra_theta - - def PostTrainingStepUpdate(self, global_step): - """Returns a TF op which will be invoked at each training step. - - Subclasses of `BaseLayer` can implement this method. The method should - return a TF op to be invoked during training after gradients are applied. - - Args: - global_step: the global step. - """ - update_ops = [ - child.PostTrainingStepUpdate(global_step) - for child in self._private_children.Flatten() - ] - return tf.group(*update_ops) - - -def IsLayerParams(x): - return (isinstance(x, hyperparams.InstantiableParams) and - issubclass(x.cls, BaseLayer)) diff --git a/lingvo/core/base_layer_test.py b/lingvo/core/base_layer_test.py deleted file mode 100644 index 78ca2543..00000000 --- a/lingvo/core/base_layer_test.py +++ /dev/null @@ -1,326 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for base_layer.""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import hyperparams -from lingvo.core import py_utils -from lingvo.core import test_utils - - -class AddingAccumulator(base_layer.Accumulator): - - def DefaultValue(self): - return tf.convert_to_tensor(0.0) - - def Update(self, new_value): - self.SetValue(self.GetValue() + new_value) - - -def EvalAndFlatten(nmap): - return nmap.Transform(lambda x: x.eval()).FlattenItems() - - -class TestLayer(base_layer.BaseLayer): - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - self.CreateVariable( - 'w', - py_utils.WeightParams( - shape=[4, 4], - dtype=p.dtype, - init=p.params_init, - collections=[self.__class__.__name__ + '_vars'])) - self.CreateVariable( - 'b', - py_utils.WeightParams( - shape=[4], - dtype=p.dtype, - init=py_utils.WeightInit.Constant(), - collections=[ - self.__class__.__name__ + '_vars', - py_utils.SKIP_LP_REGULARIZATION - ])) - - -class BaseLayerTest(test_utils.TestCase): - - def testCopyBaseParams(self): - # CopyBaseParams should only overwrite vn setting when target use - # default vn config. - layer_base_p = base_layer.BaseLayer.Params() - from_param = layer_base_p.Copy() - to_param = layer_base_p.Copy() - from_param.vn.global_vn = True - from_param.random_seed = 1234 - from_param.skip_lp_regularization = True - # Target use default, overwrite. - base_layer.BaseLayer.CopyBaseParams(from_param, to_param) - self.assertTrue(to_param.vn.global_vn) - self.assertEqual(1234, to_param.random_seed) - self.assertTrue(to_param.skip_lp_regularization) - to_param = layer_base_p.Copy() - to_param.vn.per_step_vn = True - to_param.random_seed = 4321 - to_param.skip_lp_regularization = False - # Target does not use default, should not overwrite. - base_layer.BaseLayer.CopyBaseParams(from_param, to_param) - self.assertTrue(to_param.vn.per_step_vn) - self.assertFalse(to_param.vn.global_vn) - self.assertEqual(4321, to_param.random_seed) - self.assertFalse(to_param.skip_lp_regularization) - - def testCreateChildren(self): - layer_p = base_layer.BaseLayer.Params() - layer_p.name = 'test' - layer = layer_p.Instantiate() - layer._disable_create_child = False # pylint: disable=protected-access - layer.CreateChildren('a', [ - layer_p, - [layer_p, layer_p], - { - 'b': layer_p, - 'c': { - 'd': [layer_p, layer_p] - } - }, - ]) - self.assertEqual(len(layer.a), 3) - self.assertEqual(len(layer.a[1]), 2) - self.assertEqual(len(layer.a[2]), 2) - self.assertEqual(len(layer.a[2]['c']['d']), 2) - self.assertEqual(len(layer.vars.a), 3) - self.assertEqual(len(layer.vars.a[1]), 2) - self.assertEqual(len(layer.vars.a[2]), 2) - self.assertEqual(len(layer.vars.a[2]['c']['d']), 2) - self.assertEqual(len(layer.theta.a), 3) - self.assertEqual(len(layer.theta.a[1]), 2) - self.assertEqual(len(layer.theta.a[2]), 2) - self.assertEqual(len(layer.theta.a[2]['c']['d']), 2) - - def testCreateVariable(self): - layer_p = TestLayer.Params().Set(name='test') - layer = layer_p.Instantiate() - self.assertEqual('test/w/var:0', layer.vars.w.name) - self.assertEqual('test/b/var:0', layer.vars.b.name) - self.assertNotIn(layer.vars.w, - tf.get_collection(py_utils.SKIP_LP_REGULARIZATION)) - # 'b' always skips Lp regularization. - self.assertIn(layer.vars.b, - tf.get_collection(py_utils.SKIP_LP_REGULARIZATION)) - - def testCreateVariableSkipLpRegularization(self): - layer_p = TestLayer.Params().Set(name='test', skip_lp_regularization=True) - layer = layer_p.Instantiate() - self.assertIn(layer.vars.w, - tf.get_collection(py_utils.SKIP_LP_REGULARIZATION)) - self.assertIn(layer.vars.b, - tf.get_collection(py_utils.SKIP_LP_REGULARIZATION)) - - def testGetDescendant(self): - q = base_layer.BaseLayer.Params() - q.name = 'test' - # pylint: disable=protected-access - l = q.Instantiate() - p = base_layer.BaseLayer.Params() - l._disable_create_child = False - l.CreateChild('a', p) - l.CreateChild('b', p) - l.a._disable_create_child = False - l.a.CreateChild('c', p) - l.a.c._disable_create_child = False - l.a.c.CreateChild('d', p) - l.b._disable_create_child = False - l.b.CreateChild('e', p) - # pylint: enable=protected-access - self.assertEqual(l, l.GetDescendant('')) - self.assertEqual(l.a, l.GetDescendant('a')) - self.assertEqual(l.b, l.GetDescendant('b')) - self.assertEqual(l.a.c, l.GetDescendant('a.c')) - self.assertEqual(l.a.c.d, l.GetDescendant('a.c.d')) - self.assertEqual(l.b.e, l.GetDescendant('b.e')) - - def testCreateAccumulator(self): - layer_p = base_layer.BaseLayer.Params() - layer_p.name = 'test' - layer = layer_p.Instantiate() - layer._disable_create_child = False # pylint: disable=protected-access - layer.CreateChild('child', layer_p) - - # First accumulator should succeed. - acc1 = AddingAccumulator() - layer.RegisterAccumulator('acc1', acc1) - - # Name of existing child should fail. - with self.assertRaises(AssertionError): - layer.RegisterAccumulator('child', AddingAccumulator()) - - # Duplicate should fail. - with self.assertRaises(AssertionError): - layer.RegisterAccumulator('acc1', AddingAccumulator()) - - # Child with the same name should fail. - with self.assertRaises(AssertionError): - layer.CreateChild('acc1', layer_p) - - self.assertEqual(acc1, layer.accumulators.acc1) - - # Get of not created accumulator should fail. - with self.assertRaises(AttributeError): - layer.accumulators.notexist - - def testGetUpdateAccumulator(self): - with self.session(): - layer_p = base_layer.BaseLayer.Params() - layer_p.name = 'test' - layer = layer_p.Instantiate() - - layer.RegisterAccumulator('acc1', AddingAccumulator()) - - # Initial value. - self.assertEqual(0.0, layer.accumulators.acc1.GetValue().eval()) - - # Update/merge. - layer.accumulators.acc1.Update(1.0) - layer.accumulators.acc1.Update(1.0) - self.assertEqual(2.0, layer.accumulators.acc1.GetValue().eval()) - - # Reset. - layer.accumulators.Transform(lambda acc: acc.Reset()) - self.assertEqual(0.0, layer.accumulators.acc1.GetValue().eval()) - - def testAccumulatorDisableEnable(self): - with self.session(): - layer_p = base_layer.BaseLayer.Params() - layer_p.name = 'test' - layer = layer_p.Instantiate() - - layer.RegisterAccumulator('acc1', AddingAccumulator()) - layer.accumulators.acc1.Update(1.0) - - # Disable should force value to 0 and reject updates. - layer.accumulators.Transform(lambda acc: acc.Disable()) - self.assertEqual(0.0, layer.accumulators.acc1.GetValue().eval()) - layer.accumulators.acc1.Update(3.0) - self.assertEqual(0.0, layer.accumulators.acc1.GetValue().eval()) - layer.accumulators.Transform(lambda acc: acc.Enable()) - - # Should restore. - self.assertEqual(1.0, layer.accumulators.acc1.GetValue().eval()) - - def testGetSetAccumulatorValues(self): - with self.session(): - layer_p = base_layer.BaseLayer.Params() - layer_p.name = 'test' - layer1 = layer_p.Instantiate() - layer1._disable_create_child = False # pylint: disable=protected-access - layer1.CreateChild('layer1a', layer_p) - layer1.CreateChild('layer1b', layer_p) - layer1.layer1b._disable_create_child = False # pylint: disable=protected-access - layer1.layer1b.CreateChild('layer1b1', layer_p) - - # Create nested accumulators: - # acc1: layer1 - # acc2: layer1.layer1a - # acc3: layer1.layer1b.layer1b1 - layer1.RegisterAccumulator('acc1', AddingAccumulator()) - layer1.layer1a.RegisterAccumulator('acc2', AddingAccumulator()) - layer1.layer1b.layer1b1.RegisterAccumulator('acc3', AddingAccumulator()) - - # Pack with initial values. - initial_pack = layer1.GetAccumulatorValues() - initial_pack_eval = EvalAndFlatten(initial_pack) - print('Initial values pack =', initial_pack_eval) - self.assertEqual(initial_pack_eval, [('acc1', 0.0), ('layer1a.acc2', 0.0), - ('layer1b.layer1b1.acc3', 0.0)]) - - # Update to a new known state. - layer1.accumulators.acc1.Update(1.0) - layer1.layer1a.accumulators.acc2.Update(2.0) - layer1.layer1b.layer1b1.accumulators.acc3.Update(3.0) - updated_pack = layer1.GetAccumulatorValues() - updated_pack_eval = EvalAndFlatten(updated_pack) - print('Updated values pack =', updated_pack_eval) - self.assertEqual(updated_pack_eval, [('acc1', 1.0), ('layer1a.acc2', 2.0), - ('layer1b.layer1b1.acc3', 3.0)]) - - # Save and reset. - saved_pack = layer1.GetAccumulatorValues() - layer1.accumulators.Transform(lambda acc: acc.Reset()) - self.assertEqual(0.0, layer1.accumulators.acc1.GetValue().eval()) - self.assertEqual(0.0, layer1.layer1a.accumulators.acc2.GetValue().eval()) - self.assertEqual( - 0.0, - layer1.layer1b.layer1b1.accumulators.acc3.GetValue().eval()) - - # Set and check. - layer1.SetAccumulatorValues(saved_pack) - self.assertEqual(1.0, layer1.accumulators.acc1.GetValue().eval()) - self.assertEqual(2.0, layer1.layer1a.accumulators.acc2.GetValue().eval()) - self.assertEqual( - 3.0, - layer1.layer1b.layer1b1.accumulators.acc3.GetValue().eval()) - - def testAddFunction(self): - layer_p = base_layer.BaseLayer.Params() - layer_p.name = 'test' - layer = layer_p.Instantiate() - - layer.AddFunction('test1', lambda: 1) - with self.assertRaises(AttributeError): - layer.AddFunction('test1', lambda: 2) - self.assertEqual(1, layer.fns.test1()) - - def testAttributeErrorInPropertyGetter(self): - - class BadLayer(base_layer.BaseLayer): - - @classmethod - def Params(cls): - return super().Params() - - def __init__(self, params): - super().__init__(params) - - @property - def bad_property(self): - raise AttributeError('INTERNAL') - - layer_p = BadLayer.Params() - layer_p.name = 'test' - layer = layer_p.Instantiate() - - with self.assertRaisesRegex(AttributeError, 'bad_sub_layer'): - _ = layer.bad_sub_layer - - with self.assertRaisesRegex(AttributeError, 'INTERNAL'): - _ = layer.bad_property - - def testIsLayerParams(self): - self.assertTrue(base_layer.IsLayerParams(base_layer.BaseLayer.Params())) - self.assertTrue(base_layer.IsLayerParams(TestLayer.Params())) - self.assertFalse(base_layer.IsLayerParams(None)) - self.assertFalse(base_layer.IsLayerParams(hyperparams.Params())) - self.assertFalse( - base_layer.IsLayerParams( - hyperparams.InstantiableParams(base_layer.Accumulator))) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/base_model.py b/lingvo/core/base_model.py deleted file mode 100644 index 6ceecea7..00000000 --- a/lingvo/core/base_model.py +++ /dev/null @@ -1,1229 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Base model.""" - -import collections -import re - -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import base_layer -from lingvo.core import build_data -from lingvo.core import cluster_factory -from lingvo.core import early_stop -from lingvo.core import hyperparams -from lingvo.core import learner -from lingvo.core import optimizer -from lingvo.core import py_utils -from lingvo.core import schedule -from lingvo.core import summary_utils -from lingvo.core import task_scheduler -from lingvo.core import decoder_lib -from model_pruning.python import pruning - - -class DecodeFinalizeArgs( - collections.namedtuple('DecodeFinalizeArgs', - ['decode_out_path', 'decode_out'])): - """Arguments to BaseTask.DecodeFinalize(). - - Attributes: - decode_out_path: Path to where decoder outputs can be written. - decode_out: A list of key value pairs aggregated from return values of. - PostProcessDecodeOut(). - """ - - -class BaseTask(base_layer.BaseLayer): - """A single encoder/decoder task. - - One task usually consists of one InputGenerator, one train_op, - a list of eval_metrics, etc. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input', None, 'Input generator Params.') - p.Define('encoder', None, 'Encoder Params.') - p.Define('online_encoder', None, 'Online Encoder Params.') - p.Define('decoder', None, 'Decoder Params.') - p.Define( - 'task_global_step', False, - 'Whether or not to use task-specific global steps, which causes each ' - 'task to use its own global_step instead of the true global_step. ' - 'NOTE: this may be severely broken. Verify carefully!') - p.Define( - 'defer_global_step_update', False, - 'Whether or not to defer the global step update. This is used when ' - 'doing gradient accumulation, which update the global step only when ' - 'weights are updated. Currently this supports only true global step.') - p.Define('train', hyperparams.Params(), - 'Params to control how this task should be trained.') - - p.Define('ml_perf', hyperparams.Params(), 'MlPerf configuration.') - - tp = p.train - tp.Define( - 'start_up_delay_steps', 200, 'i-th replica starts training after ' - 'i*(i+1)/2*start_up_delay_steps steps') - tp.Define('max_steps', 4 * 10**6, 'Maximum number of training steps.') - tp.Define('tpu_steps_per_loop', 1000, 'The number of training steps per ' - 'training loop for TPUs.') - tp.Define( - 'vn_start_step', 200000000, - 'Step starting from which variational noise is added to ' - 'params values during training.') - tp.Define('vn_std', 0.0, 'Std of the variational noise.') - tp.Define('early_stop', early_stop.EarlyStop.Params(), - 'Early stopping based on dev-set performance.') - tp.Define( - 'ema_decay', 0.0, - 'If > 0, enable ExponentialMovingAverage during training ' - 'with the give decay. ' - 'Must be < 1. Disabled if <= 0.') - tp.Define( - 'ema_decay_moving_vars', None, - 'If True, include variables from collection "moving_vars" in ema.') - tp.Define( - 'init_from_checkpoint_rules', {}, - 'If not None, a dictionary with keys corresponding to a checkpoint ' - 'path and values corresponding to variable loading rules is expected. ' - 'Each key is expected to be a path to a checkpoint from which to ' - 'initialize part of the model. Variables are only loaded from this ' - 'path during initialization and will override values provided by ' - 'initialization.' - 'The corresponding values (loading_rules) are expected to be a tuple ' - 'consisting of two list: loading rules, and ignore rules, respectively.' - 'The first list (loading rules) contains the list of variables ' - 'which should be initialized from the checkpoint: each element in the ' - 'list is a pair of strings. The first element is a regex and the ' - 'second is a python format string. If a variable in the model matches ' - 'a regex, we rename using the format string to determine the ' - 'corresponding var in the checkpoint. Note that, it is an error if a ' - 'model variable matches multiple loading rules, for the same ' - 'checkpoint or across checkpoints.' - 'The second list (ignore rules) is a list of regexes which specify ' - 'variables in the model which should not be initialized using the ' - 'loading rules. Thus, if a variable in the model to be trained matches ' - 'one of the rules in the loading rules, as well as one of the regular ' - 'expressions in the ignore rules, the variable will not be initialized ' - 'from the checkpoint, but will instead be initialized from the ' - 'variable initalizer defined in the graph.' - 'Examples:' - '{"checkpoint_path": ([("(.*)", "%s")], [])} will initialize all the ' - 'model parameters from the checkpoint_path.') - tp.Define( - 'pruning_hparams_dict', None, 'Pruning related hyperparameters. A dict ' - 'with hyperparameter: value pairs. See google-research.model_pruning.') - tp.Define( - 'enqueue_max_steps', -1, 'Max enqueue steps. -1 meaning no limit.' - ' This flag should be set for unit-test only.') - tp.Define('save_interval_seconds', 60 * 10, - 'Generates a checkpoint roughly once every this many seconds.') - tp.Define('save_max_to_keep', 100, - 'Maximum number of recent checkpoints to keep.') - tp.Define('save_keep_checkpoint_every_n_hours', 0.5, - 'How often to keep a checkpoint.') - - tp.Define('summary_interval_steps', 100, - 'Generates a summary roughly once every this many steps.') - # The following params must mirror those in Learner.Params(). - # TODO(rpang): migrate existing params to use learner and - # delete legacy params. - # LINT.IfChange - tp.Define( - 'learner', None, 'One or a list of optimization programs. ' - 'If None, uses a Learner created from the legacy params ' - 'defined below: learning_rate, lr_schedule, optimizer, etc.') - tp.Define( - 'l2_regularizer_weight', None, - 'If not None, L2 regularization to apply to the weights. ' - 'Otherwise, disable L2 regularization.') - tp.Define( - 'l1_regularizer_weight', None, - 'If not None, L1 regularization to apply to the weights. ' - 'Otherwise, disable L1 regularization.') - tp.Define('learning_rate', 0.0, 'learning rate to use.') - tp.Define( - 'clip_gradient_norm_to_value', 0.0, - 'Clip gradient by global norm to this value. This is similar to ' - 'the bahaviour of tf.clip_by_global_norm, if you are looking for ' - 'tf.clip_by_norm refer to clip_gradient_single_norm_to_value. Note ' - 'these are mutually exclusive.') - tp.Define( - 'clip_gradient_single_norm_to_value', 0.0, - 'Clip gradient by single tensor norm to this value. This is ' - 'similar to the bahaviour of tf.clip_by_norm. Note this is mutually ' - 'exlusive to using clip_gradient_norm_to_value.') - tp.Define('grad_norm_to_clip_to_zero', 0.0, - 'Clip gradient to 0 if its norm exceeds this value.') - tp.Define('grad_norm_tracker', None, 'Params for GradNormTracker.') - tp.Define('optimizer', optimizer.Adam.Params(), 'Params for the optimizer.') - tp.Define('lr_schedule', schedule.ContinuousSchedule.Params(), - 'Learning rate decay schedule.') - tp.Define( - 'bprop_variable_filter', None, - 'If set, only backprop variables whose names partially match ' - 'this regexp (re.search).') - tp.Define( - 'bprop_variable_exclusion', None, - 'If set, do not backprop variables whose names partially match ' - 'this regexp (re.search).') - tp.Define( - 'grad_aggregation_method', tf.AggregationMethod.EXPERIMENTAL_TREE, - 'Specifies the method used to combine gradient terms. Accepted ' - 'values are constants defined in the class AggregationMethod.') - tp.Define( - 'gate_gradients', False, - 'If True, add a tuple around the gradients returned for an ' - 'operations. This avoids some race conditions.') - tp.Define('colocate_gradients_with_ops', True, - 'If True, try colocating gradients with the corresponding op.') - tp.Define('scale_gradients', True, - 'Whether to apply gradients adjustment and scaling.') - # LINT.ThenChange(learner.py) - p.Define('eval', hyperparams.Params(), - 'Params to control how this task should be evaled.') - ep = p.eval - ep.Define( - 'samples_per_summary', 1000, - 'If > 0, generates one summary after this many samples, at most. ' - 'If == 0 or the dataset has fewer examples, evaluate the whole set.') - ep.Define( - 'decoder_samples_per_summary', 0, - 'If > 0, each decoder summary will contain at most this many samples. ' - 'If == 0, defaults to `samples_per_summary` for ' - 'backwards compatibility.') - ep.Define( - 'load_checkpoint_from', None, - 'If not None, specifies a location for the checkpoint that ' - 'should be used for eval. One example format is a ' - 'checkpoint directory of a training run.') - ep.Define('start_eval_after', 0, - 'Start evaluation after specified number of steps.') - ep.Define('start_decoder_after', 0, - 'Only decode checkpoints after this step.') - return p - - @classmethod - def UpdateTargetVocabSize(cls, p, vocab_size, wpm_model=None): - """Updates params with the vocab size and wpm model. - - Args: - p: model params. - vocab_size: size of the vocabulary. - wpm_model: file name prefix pointing to a wordpiece model. - - Returns: - Model params updated with the vocab size and wpm model. - """ - dp = p.decoder - p.decoder = dp.cls.UpdateTargetVocabSize(dp, vocab_size, wpm_model) - return p - - def __init__(self, params): - assert issubclass(params.cls, BaseTask) - # Ensure global_step exists before calling super. - py_utils.GetOrCreateGlobalStepVar() - super().__init__(params) - - p = self.params - - self._encoder = None - self._online_encoder = None - self._decoder = None - - self._loss = None - self._num_predictions = None - self._train_op = None - self._post_train_ops = [] - self._eval_metrics = {} - self._per_example = {} - - # Create the gradient mask, - self._per_input_gradient_mask = None - - if p.task_global_step: - with tf.name_scope(None), tf.variable_scope( - py_utils.GetGlobalVariableScope()): - var_name = p.name + '_global_step' - # Create the variable immediately. - self._CreateVariableInternal( - var_name, - base_layer.CreateVariableMeta( - var_params=py_utils.WeightParams( - [], py_utils.WeightInit.Constant(0), tf.int64), - theta_fn=None, - kwargs=dict( - trainable=False, - collections=[tf.GraphKeys.GLOBAL_VARIABLES]))) - summary_utils.scalar(var_name, self._private_vars[var_name]) - self._global_step_var = self._private_vars[var_name] - else: - self._global_step_var = py_utils.GetOrCreateGlobalStepVar() - - if p.input: - # TODO(zhifengc): Consider a simpler way to ensure the input - # generator stops after one epoch. - if self.do_eval and p.eval: - seq_inp = issubclass(p.input.cls, - base_input_generator.BaseInputGeneratorFromFiles) - if p.input.num_samples == 0: - # Dataset size is unknown. Computes eval summary based on num_samples. - assert p.eval.samples_per_summary > 0 - elif (p.eval.samples_per_summary == 0) or (p.input.num_samples < - p.eval.samples_per_summary): - # If we know the dataset size and we want to evaluate the full - # set, we need to coordinate the input generator to flush out - # all samples so the evaler and decoder compute metrics on the - # whole set for each summary step. - if seq_inp: - p.input.flush_every_n = p.input.num_samples - p.eval.samples_per_summary = p.input.num_samples - if seq_inp and p.input.num_batcher_threads > 1: - tf.logging.warning( - 'input.num_batcher_threads > 1 inside eval mode. ' - 'The input generator may not iterate over exactly ' - 'one epoch per run') - tf.logging.info('input_params: %s', p.input) - input_params = self.cluster.PlaceInput(p.input) - - # For TPU training, we create the input generator in a - # different scope and AddChild it in later. - if 'skip_create_child' not in p.input: - self.CreateChild('input', input_params) - - tp = p.train - - # p.train can be None if this task is the teacher/student task in a - # DistillationTask. - if tp: - self._SetLearnerFromLegacyParams(tp) - if tp.learner is not None: - if isinstance(tp.learner, (list, tuple)): - self.CreateChildren('learners', tp.learner) - else: - self.CreateChildren('learners', [tp.learner]) - self._UpdateVnConfig() - - def InstantiateVariables(self): - with py_utils.GlobalStepContext( - tf.identity(self._global_step_var, name='global_step_tensor')): - super().InstantiateVariables() - - def _SetLearnerFromLegacyParams(self, tp): - """Sets tp.learner based on legacy params.""" - if tp.learner is not None: - return - tp.learner = learner.ExtractLearnerFromLegacyParams(tp) - - def ComputePredictions(self, theta, input_batch): - """Computes predictions for `input_batch`. - - The output can be in the form of probablistic distributions, e.g., softmax - logits for discrete outputs, mixture of logistics for continuous values, or - regression values. - - For training/evaluation, the output will be used for computing loss and - gradient updates, including comparing predicted distributions between - teacher and student for distillation. During inference the output can be - used to compute final outputs, perhaps with sampling. - - Args: - theta: A `.NestedMap` object containing variable values of this task. - input_batch: A `.NestedMap` object containing input tensors to this tower. - - Returns: - Predictions, either a single Tensor, a `.NestedMap`, or a namedtuple. - """ - raise NotImplementedError('Abstract method') - - def ComputeLoss(self, theta, predictions, input_batch): - """Computes loss and other metrics for the given predictions. - - Args: - theta: A `.NestedMap` object containing variable values of this task. - predictions: The output of `ComputePredictions`. - input_batch: A `.NestedMap` object containing input tensors to this tower. - - Returns: - (dict, dict): - - - A dict containing str keys and (metric, weight) pairs as values, where - one of the keys is expected to be 'loss'. - - A dict containing arbitrary tensors describing something about each - training example, where the first dimension of each tensor is the batch - index. - """ - raise NotImplementedError('Abstract method') - - def FilterPerExampleTensors(self, per_example): - """Return the per-example tensors ProcessFPropResults needs. - - By default we don't send any per-example tensors to ProcessFPropResults - because some may be expensive to compute. Implement this method to let - some of them pass through. - - Args: - per_example: A dict of tensors returned as per-example tensors from FProp. - - Returns: - A dict containing a subset of the key/value pairs in per_example. - """ - return {} - - def ProcessFPropResults(self, sess, global_step, metrics, per_example): - """Called once for each train loop. - - BaseModel.ProcessFPropResults is also called on each loop, so you - can put your implementation wherever it is most convenient for you. - - Be sure to implement BaseTask.FilterPerExampleTensors if you plan to use any - per-example tensors in this method. - - Args: - sess: a session. - global_step: task global step. Since ProcessFPropResults is called after - sess.run(train_op), this value will be 1 higher than the value in FProp. - metrics: the metrics dict returned by FPropTower. - per_example: the per_example dict returned by FPropTower. - """ - pass - - def FPropTower(self, theta, input_batch): - """Forward propagation through one tower of the model. - - Args: - theta: A `.NestedMap` object containing variable values of this task - copied to this tower's devices. - input_batch: A `.NestedMap` object containing input tensors to this tower. - - Returns: - (dict, dict): - - - A dict containing str keys and (metric, weight) pairs as values, where - one of the keys is expected to be 'loss'. - - A dict containing arbitrary tensors describing something about each - training example, where the first dimension of each tensor is the batch - index. - """ - predictions = self.ComputePredictions(theta, input_batch) - return self.ComputeLoss(theta, predictions, input_batch) - - def FProp(self, theta, input_batch): - """Forward propagation. - - This default `FProp` implementation here supports batch splitting in - synchronous and asynchronous training when sub-classes implement - `FPropTower`. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - input_batch: The input batch. A `NestedMap` of tensors. Or, if input batch - spiltting is used, a list of `NestedMap`, one for each split. - - Returns: - (dict, dict): - - - A dict containing str keys and (metric, weight) pairs as values, where - one of the keys is expected to be 'loss'. - - A dict containing arbitrary tensors describing something about each - training example, where the first dimension of each tensor is the batch - index. - """ - p = self.params - with tf.name_scope('fprop'), tf.name_scope(p.name): - # Always reset step seed at the start of a new global_step. - py_utils.ResetStepSeed() - metrics, per_example = self._FPropSplitInputBatch(theta, input_batch) - self._FPropResult(metrics, per_example) - return metrics, per_example - - def _FPropTpu(self, theta, input_batch): - with tf.name_scope('tower_0_0'): - metrics, per_example = self.FPropTower(theta, input_batch) - metrics = py_utils.WeightedAvgOfMetrics([metrics]) - return metrics, per_example - - def _FPropSplitInputBatch(self, theta, input_batch): - """Splits the input batch on the input device.""" - if py_utils.use_tpu(): - return self._FPropTpu(theta, input_batch) - - cluster = self.cluster - num_splits = cluster.num_splits_per_client - - if not isinstance(input_batch, list): - input_batch = [input_batch] - - assert len(input_batch) == num_splits, (len(input_batch), num_splits) - - # dev_list_per_replica[i][j] is the i-th worker's j-th device. - dev_list_per_replica = cluster.available_devices.tolist() - - # Asserts invariant of the total number of splits w.r.t., - # splits per worker. - splits_per_replica = cluster.num_splits_per_replica - assert num_splits == splits_per_replica * len(dev_list_per_replica), ( - num_splits, splits_per_replica, len(dev_list_per_replica)) - - all_metrics = [] - all_per_example_tensors = [] - with cluster: - for w_id, w_devs in enumerate(dev_list_per_replica): - # Make local copy of the vars, shard on devices for this worker. - theta_local = py_utils.CreateLocalTheta( - theta, w_devs, label='worker %d' % w_id) - - for s_id in range(splits_per_replica): - # s_id-th split for the w_id-th worker. - split_id = splits_per_replica * w_id + s_id - with cluster_factory.SetModelSplit(split_id) as c: - with tf.device(c.WorkerDeviceInModelSplit(0)): - with tf.name_scope('tower_%d_%d' % (w_id, s_id)): - batch = input_batch[split_id] - metrics, per_example = self.FPropTower(theta_local, batch) - all_metrics.append(metrics) - all_per_example_tensors.append(per_example) - return py_utils.WeightedAvgOfMetrics( - all_metrics), py_utils.ConcatPerExampleTensors(all_per_example_tensors) - - def _FPropResult(self, metrics, per_example): - # Adds stats about the input batch. - p = self._params - if p.input is not None: - metrics['num_samples_in_batch'] = (tf.convert_to_tensor( - self.input_generator.GlobalBatchSize()), tf.constant(1.0)) - # Generates summaries. - for name, (value, weight) in metrics.items(): - self.AddEvalMetric(name, value, weight) - per_example = self.FilterPerExampleTensors(per_example) - for name, value in per_example.items(): - self.AddPerExampleTensor(name, value) - # Loss. - self._loss, self._num_predictions = metrics['loss'] - self._loss = py_utils.CheckNumerics(self._loss) - self._metrics = metrics - summary_utils.scalar('num_predictions', self._num_predictions) - - def GetInputBatch(self): - """Gets an input batch.""" - if py_utils.use_tpu(): - return self.input_generator.TpuDequeueBatch() - else: - return self.input_generator.SplitInputBatch( - self.cluster.num_splits_per_client) - - def FPropDefaultTheta(self, input_batch=None): - """Calls `FProp` with this layer's parameters.""" - if input_batch is None: - input_batch = self.GetInputBatch() - return self.FProp(self.theta, input_batch) - - def AdjustGradients(self, vars_gradients): - """Allow for custom gradient manipulation prior to clipping.""" - tf.logging.info('BaseTask.AdjustGradients') - return vars_gradients - - def PostTrainingLoop(self): - self._post_training_loop_op = tf.group(*[ - opt.ApplyPostTrainingLoop(self._global_step_var) - for opt in self.learners - ]) - - def BProp(self): - self._BPropForVariables(self.vars) - - def _BPropGenTrainOps(self, vmap, metrics=None, add_summary=True): - """Populates the train_ops dictionary in a backwards pass.""" - metrics = metrics or self._metrics - - bprop_variable_filters = self.input_generator.GetBpropVariableFilters() - # Only compute the mask if the variable filters are not empty. - if bprop_variable_filters != [''] * len(bprop_variable_filters): - self._ComputeGradientMask(bprop_variable_filters) - train_ops = {} # mapping from op name to op. - gradient_mask = None - if self._per_input_gradient_mask: - # TODO(neerajgaur): Change this to use source_selected from input_batch. - onehot = self.input_generator.GetInputSourceOneHot() - gradient_mask = { - k: tf.tensordot(v, onehot, 1) - for k, v in self._per_input_gradient_mask.items() - } - all_losses = [] - for optimization in self.learners: - learner_name = optimization.params.name - (losses, train_ops['train/%s' % learner_name], - eval_metrics) = optimization.Apply( - metrics, - vmap, - gradient_mask=gradient_mask, - gradient_adjuster=self.AdjustGradients) - all_losses.extend(losses) - if add_summary: - for key, (value, weight) in eval_metrics.items(): - self.AddEvalMetric(key + '/' + learner_name, value, weight) - - relevant_bn_updates, _ = py_utils.FindRelevantBatchNormUpdates( - all_losses, tf.get_collection(py_utils.BATCH_NORM_UPDATES)) - train_ops['bn_updates'] = relevant_bn_updates - - var_update_ops = [ - tf.group(*tf.nest.flatten(train_ops), name='var_update_ops') - ] - # Post training step update. - with tf.control_dependencies(var_update_ops): - post_step_op = self.PostTrainingStepUpdate(self.global_step) - - train_ops = {} - with tf.control_dependencies([post_step_op]): - # Get the op to update the weight masks and thresholds - mask_update_op = self._GetMaskUpdateOp() - train_ops['mask_updates'] = mask_update_op - with tf.control_dependencies([mask_update_op]): - true_global_step = py_utils.GetOrCreateGlobalStepVar() - with tf.ops.colocate_with(true_global_step): - if self.params.defer_global_step_update: - increment_global_steps = true_global_step - else: - increment_global_steps = tf.assign_add(true_global_step, 1) - if self._global_step_var != true_global_step: - with tf.ops.colocate_with(self._global_step_var): - increment_global_steps = tf.group( - increment_global_steps, tf.assign_add(self._global_step_var, 1)) - train_ops['global_step'] = increment_global_steps - - # If we are using Tpu Embeddings, generate the monolithic send - # gradient op. - tpu_embedding_activations = tf.get_collection( - py_utils.TPU_EMBEDDING_ACTIVATIONS) - if tpu_embedding_activations: - tpu_embedding_activations_dict = tpu_embedding_activations[0] - tpu_embedding = tf.get_collection(py_utils.TPU_EMBEDDING)[0] - tpu_embedding_send_gradient_op = py_utils.ComputeTpuEmbeddingGradients( - self.loss, tpu_embedding_activations_dict, tpu_embedding) - train_ops['tpu_embedding'] = tpu_embedding_send_gradient_op - - for op_name, op in train_ops.items(): - assert op is not None, op_name - return train_ops - - def _BPropForVariables(self, vmap): - """Constructs the backward graph.""" - train_ops = self._BPropGenTrainOps(vmap) - - # TODO(rpang): try to structure _train_op as: - # tf.cond(skip_step, , ) - # so that we skip all other updates when a step is skipped. - with tf.control_dependencies( - [tf.group(*tf.nest.flatten(train_ops), name='train_ops')]): - self._train_op = tf.group(self._post_train_ops, name='bprop') - - def _ComputeGradientMask(self, bprop_variable_filters): - """Compute gradient mask for each variable and bprop_variable_filters. - - Note that per_input_gradient_mask[var][i] will be 1 if var matches - bprop_variable_filter[i], 0 otherwise. - - Args: - bprop_variable_filters: A list of regex bprop_variable_filters for each - file pattern. - """ - self._per_input_gradient_mask = py_utils.NestedMap() - all_vars = set(self.vars.Flatten()) - for var in all_vars: - self._per_input_gradient_mask[var.name] = ( - tf.zeros(len(bprop_variable_filters), dtype=tf.float32)) - for i in range(len(bprop_variable_filters)): - if re.search(bprop_variable_filters[i], var.name): - tf.logging.info( - 'Keep gradient after filtering, regex: %s var: %s' % - (bprop_variable_filters[i], var.name)) - self._per_input_gradient_mask[var.name] += ( - tf.one_hot(i, len(bprop_variable_filters), dtype=tf.float32)) - - def ApplyExponentialMovingAverage(self, ema): - """Wraps `self.train_op` with an op updating exponential moving average.""" - if (self._create_variables_status != - base_layer._CreateLayerVariablesStatus.COMPLETED): # pylint: disable=protected-access - raise ValueError( - 'ApplyExponentialMovingAverage called before InstantiateVariables!') - # TODO(rpang): raise an exception if this is called in the eval mode. - p = self.params - # We need to apply EMA to trainable and moving average variable of this - # Task, not just bprop vars, so that we create a shadow - # '/ExponentialMovingAverage' variable for every trainable and moving - # average variable. - all_vars = set(tf.trainable_variables()) | set( - tf.moving_average_variables()) - if p.train.ema_decay_moving_vars: - all_vars |= set(tf.get_collection('moving_vars')) - all_vars &= set(self.vars.Flatten()) - for var in all_vars: - tf.logging.debug('ApplyExponentialMovingAverage: %s', var.name) - with tf.name_scope('moving_average'): - self._post_train_ops.append(ema.apply(all_vars)) - - # TODO(blee): Rename Decode->DecodeWithDefaultTheta, DecodeWithTheta->Decode. - def Decode(self, input_batch): - """Constructs the inference graph for eval decoding. - - Args: - input_batch: The input batch. A `NestedMap` of tensors. Or, if input batch - spiltting is used, a list of `NestedMap`, one for each split. - - Returns: - a dict of Tensors as decoder output. - """ - return self.DecodeWithTheta(self.theta, input_batch) - - def DecodeWithTheta(self, theta, input_batch): - """Constructs the inference graph for eval decoding with theta. - - Args: - theta: A `.NestedMap` object containing variable values of this task. - input_batch: The input batch. A `NestedMap` of tensors. Or, if input batch - spiltting is used, a list of `NestedMap`, one for each split. - - Returns: - a dict of Tensors as decoder output. - """ - return {} - - def Inference(self): - """Constructs the inference graph. - - Each subgraph represents a public API for a part of the graph which can - be operated independently. By convention, the subgraph named 'default' - should perform end to end inference via the input generator. - - Note that having distinct subgraphs (e.g. 'encoder', 'decoder') is - not just a space optimization: when driving the graph externally in an - online fashion, evaluation often needs to be broken into pieces. In this - case, the graph will be constructed with only those pieces. - - Returns: - An `inference_graph_pb2.InferenceGraph` message. - """ - raise NotImplementedError('Abstract method') - - def CreateDecoderMetrics(self): - """Creates a dict of decoder metrics for `PostProcessDecodeOut` to update. - - Returns a dict mapping from string keys to `.BaseMetric` objects. - """ - pass - - def PostProcessDecodeOut(self, decode_out_dict, decode_metrics_dict): - """Post-processes decoder out and updates contents of `decode_metrics_dict`. - - Args: - decode_out_dict: A dictionary of Tensors fetched. - decode_metrics_dict: A dict mapping from string key to `.BaseMetric` - object as created by `CreateDecoderMetrics`. - - Returns: - output_key_value_pairs - a list of (key, value) pairs that can be saved - (i.e. of type str, bytes, or unicode). - """ - pass - - def DecodeFinalize(self, decode_finalize_args): - """Finalize any work for decoding. - - Args: - decode_finalize_args: A DecodeFinalizeArgs namedtuple. - """ - decode_out_path = decode_finalize_args.decode_out_path - decode_out = decode_finalize_args.decode_out - if decode_out: - decoder_lib.WriteKeyValuePairs(decode_out_path, decode_out) - - @property - def loss(self): - assert self._loss is not None, ('No loss is defined. Call FProp first.') - return self._loss - - @property - def train_op(self): - assert self._train_op is not None, ( - 'No train op is defined. Call BProp first.') - return self._train_op - - @property - def post_training_loop_op(self): - assert self._post_training_loop_op is not None, ( - 'No post_training_loop_op op is defined. Call PostTrainingLoop first.') - return self._post_training_loop_op - - @property - def global_step(self): - return self._global_step_var - - @property - def input_generator(self): - return self.input - - @property - def eval_metrics(self): - """Returns the evaluation metrics. - - Returns: - A map from metric name (a python string) to a tuple (value, weight). - Both value and weight are scalar Tensors. - """ - return self._eval_metrics - - @property - def per_example_tensors(self): - """Returns per-example outputs. - - Returns: - A map from tensor name (a python string) to a tensor, where the - first dimension is the batch index of the training example corresponding - to this output. - """ - return self._per_example - - def AddEvalMetric(self, name, value, weight): - """Adds a metric to the eval metrics. - - Args: - name: A python string. The name of the metric. - value: A scalar Tensor. - weight: A scalar Tensor. - - Raises: - ValueError: if `name` is already defined. - - """ - if name in self._eval_metrics: - raise ValueError('Metric %s has already been defined.' % name) - self._eval_metrics[name] = (value, weight) - - def AddPerExampleTensor(self, name, value): - if name in self._per_example: - raise ValueError('Metric %s has already been defined.' % name) - self._per_example[name] = value - - def _UpdateVnConfig(self): - """Update vn config from the various vn flags.""" - p = self.params - tp = p.train - if tp: - vn_enabled = ((tp.vn_std > 0) and p.vn and - (p.vn.global_vn or p.vn.per_step_vn)) - if self.do_eval or (not vn_enabled): - p.vn = py_utils.VariationalNoiseParams(None, False, False) - else: - # vn.scale is dependent on global_step. - p.vn.scale = tf.cast(self.global_step > tp.vn_start_step, - py_utils.FPropDtype(p)) * tp.vn_std - - def _GetMaskUpdateOp(self): - """Returns op to update masks and threshold variables for model pruning.""" - p = self.params - tp = p.train - mask_update_op = tf.no_op() - if tp.pruning_hparams_dict: - assert isinstance(tp.pruning_hparams_dict, dict) - pruning_hparams = pruning.get_pruning_hparams().override_from_dict( - tp.pruning_hparams_dict) - pruning_obj = pruning.Pruning( - pruning_hparams, global_step=self.global_step) - if self.cluster.add_summary: - pruning_obj.add_pruning_summaries() - mask_update_op = pruning_obj.conditional_mask_update_op() - return mask_update_op - - -class BaseModel(base_layer.BaseLayer): - """The abstract model class. All models are sub-class of this class.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'model', None, 'Which python function generates the param. It includes ' - 'the file name and lineno where the function is defined.') - p.Define( - 'cluster', cluster_factory.Cluster.Params(), - 'The training cluster. Individual layer may config differently' - ' based on training cluster it is running under.') - p.Define('input', None, 'Input generator Params.') - p.Define('build_data', build_data.BuildData(), 'Build data of this binary.') - p.Define('train', hyperparams.Params(), - 'Params to control how this model should be trained.') - tp = p.train - tp.Define( - 'start_up_delay_steps', 200, 'i-th replica starts training after ' - 'i*(i+1)/2*start_up_delay_steps steps') - tp.Define('max_steps', 4 * 10**6, 'Training max of 4M steps.') - tp.Define('tpu_steps_per_loop', 1000, 'The number of training steps per ' - 'training loop for TPUs.') - tp.Define( - 'ema_decay', 0.0, - 'If > 0, enable ExponentialMovingAverage during training ' - 'with the give decay. ' - 'Must be < 1. Disabled if <= 0. ' - 'Must be set consistent across all tasks.') - tp.Define( - 'ema_decay_moving_vars', None, - 'If True, include variables from collection "moving_vars" in ema. ' - 'Must be set consistent across all tasks.') - tp.Define('init_from_checkpoint_rules', {}, - 'See BaseTask documentation for details.') - tp.Define('early_stop', None, - 'Early stopping based on dev-set performance.') - tp.Define( - 'enqueue_max_steps', -1, 'Max enqueue steps. -1 meaning no limit.' - ' This flag should be set for unit-test only.') - tp.Define('save_interval_seconds', 60 * 10, - 'Generates a checkpoint roughly once every this many seconds.') - tp.Define('save_max_to_keep', 100, - 'Maximum number of recent checkpoints to keep.') - tp.Define('save_keep_checkpoint_every_n_hours', 0.5, - 'How often to keep a checkpoint.') - tp.Define('summary_interval_steps', 100, - 'Generates a checkpoint roughly once every this many steps.') - - return p - - def __init__(self, params): - """Initializes this Model.""" - assert issubclass(params.cls, BaseModel) - super().__init__(params) - tf.logging.info('Training parameters for %s: %s', params.cls, - self.params.train) - self._global_step_var = py_utils.GetOrCreateGlobalStepVar() - - tp = self.params.train - if tp.ema_decay > 0: - assert tp.ema_decay < 1.0 - self._ema = tf.train.ExponentialMovingAverage( - decay=tp.ema_decay, num_updates=self.global_step) - else: - self._ema = None - - @property - def global_step(self): - return self._global_step_var - - @property - def ema(self): - return self._ema - - @property - def variables_for_ema(self): - p = self.params - all_vars = set(tf.trainable_variables()) | set( - tf.moving_average_variables()) - if p.train.ema_decay_moving_vars: - all_vars |= set(tf.get_collection('moving_vars')) - all_vars &= set(self.vars.Flatten()) - for var in all_vars: - tf.logging.debug('variables_for_ema: %s', var.name) - return all_vars - - def ConstructFPropBPropGraph(self): - raise NotImplementedError('Abstract method') - - def ConstructFPropGraph(self): - raise NotImplementedError('Abstract method') - - def ConstructPostTrainingLoop(self): - raise NotImplementedError('Abstract method') - - @property - def tasks(self): - """Returns a list of all tasks.""" - raise NotImplementedError('Abstract method') - - def GetTask(self, task_name): - """Return the task associated with 'task_name'. - - Args: - task_name: string, the name of the model task to be returned. - - Returns: - An instance of `BaseTask`. - """ - raise NotImplementedError('Abstract method') - - def ProcessFPropResults(self, sess, global_step, metrics, per_example): - """Called once for each train loop. - - BaseTask.ProcessFPropResults is also called on each loop, so you - can put your implementation wherever it is most convenient for you. - - Be sure to implement BaseTask.FilterPerExampleTensors if you plan to use any - per-example tensors in this method. - - Args: - sess: a session. - global_step: model global step. Since ProcessFPropResults is called after - sess.run(train_op), this value will be 1 higher than the value in FProp. - metrics: the metrics dict returned by FPropTower. - per_example: the per_example dict returned by FPropTower. - """ - pass - - -class SingleTaskBase(BaseModel): - """Represents a single task from a model. - - Subclasses must create a Task in self._task by the end of __init__. - """ - - @property - def tasks(self): - return [self._task] - - def GetTask(self, task_name=None): - assert not task_name, 'Must not specify >task_name< for single-task model.' - return self._task - - def SampleTask(self, global_step): - return self._task - - def ConstructFPropBPropGraph(self): - if self.ema: - tf.logging.info('ApplyExponentialMovingAverage on %s', self._task) - self._task.ApplyExponentialMovingAverage(self.ema) - self._task.FPropDefaultTheta() - self._task.BProp() - - def ConstructFPropGraph(self): - self._task.FPropDefaultTheta() - - def ConstructPostTrainingLoop(self): - self._task.PostTrainingLoop() - - -class SingleTaskModel(SingleTaskBase): - """Model that consists of a single task.""" - - @classmethod - def Params(cls, task_params=None): - p = super().Params() - p.Define( - 'task', None, - '`InstantiableParams` object for a `BaseTask` or its derivatives.') - - if task_params is not None: - # Copy over model parameters from the task parameters. - p.task = task_params - base_layer.BaseLayer.CopyBaseParams(p.task, p) - tp = p.train - tp.start_up_delay_steps = p.task.train.start_up_delay_steps - tp.max_steps = p.task.train.max_steps - tp.tpu_steps_per_loop = p.task.train.tpu_steps_per_loop - # init_from_checkpoint_rules does not need to be copied. - tp.early_stop = p.task.train.early_stop - tp.enqueue_max_steps = p.task.train.enqueue_max_steps - tp.save_interval_seconds = p.task.train.save_interval_seconds - tp.save_max_to_keep = p.task.train.save_max_to_keep - tp.save_keep_checkpoint_every_n_hours = ( - p.task.train.save_keep_checkpoint_every_n_hours) - tp.summary_interval_steps = p.task.train.summary_interval_steps - - return p - - def __init__(self, params): - assert issubclass(params.cls, SingleTaskModel) - assert params.task - p = params.Copy() # Make a copy to avoid modifying the input. - p.name = p.name or p.task.name - p.task.name = p.task.name or p.name - if p.input: - assert not p.task.input - p.task.input = p.input - else: - assert p.task.input - p.input = p.task.input - p.train.ema_decay = p.task.train.ema_decay - p.train.ema_decay_moving_vars = p.task.train.ema_decay_moving_vars - - super().__init__(p) - - self.CreateChild('_task', self.params.task) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - self._task.InstantiateVariables() - super()._CreateChildrenVariables() - - -class MultiTaskSubModel(SingleTaskBase): - """'Model' consisting of a task from a multi-task model. - - The entire multi-task model is constructed, but otherwise this model - appears to be a SingleTaskModel consisting of just one of the multi-task - model's tasks. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'multi_task_sub_model' - p.Define('task_name', '', 'The name of the task to execute from the ' - 'enclosing model.') - return p - - def __init__(self, params, shared_model=None): - super().__init__(params) - p = self.params - self._model = shared_model - self._task = self._model.children.Get(p.task_name) - - -class MultiTaskModel(BaseModel): - """Model that consists of multiple tasks.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'task_params', hyperparams.Params(), - 'Params object mapping task name to `BaskTask`(or derivatives) ' - 'Params.') - p.Define( - 'task_probs', hyperparams.Params(), - 'Params object mapping task name to the relative likelihood the ' - 'task will be sampled during training.') - p.Define('task_schedule', None, 'Task schedule.') - p.Define( - 'task_global_step', False, - 'Whether or not to use task-specific global steps, which causes each ' - 'task to use its own global_step instead of the true global_step. ' - 'NOTE: this may be severely broken. Verify carefully!') - p.Define( - 'task_name_var_scope', True, - 'Whether or not to use the task name as a variable scope. Note that ' - 'this has been the default behavior for some time, but seems to be ' - 'redundant since the individual tasks scope by their `name`.') - return p - - @staticmethod - def TaskNames(params): - return sorted(task_name for task_name, _ in params.task_params.IterParams()) - - def __init__(self, params): - assert issubclass(params.cls, MultiTaskModel) - super().__init__(params) - p = self.params - assert len(p.task_params) > 1 - - sorted_task_params = sorted( - (task_name, task_params) - for task_name, task_params in p.task_params.IterParams()) - - # Pass input params to tasks. - assert isinstance(p.input, hyperparams.Params) - - for task_name, task_params in sorted_task_params: - assert isinstance(task_params, hyperparams.Params) - assert not task_params.input - try: - task_params.input = p.input.Get(task_name) - except AttributeError as e: - tf.logging.error( - 'Missing input params for task %s !' - 'Check that you have the correct datasets ' - 'passed to DefineMultitaskDatasets.', task_name) - raise e - - if p.task_global_step: - assert task_name == task_params.name, (task_name, task_params.name) - task_params.task_global_step = True - - assert set(dir(p.input)) == set(dir(p.task_params)) - - # For compatibility with older API (with p.task_probs) - if p.task_schedule is None: - p.task_schedule = task_scheduler.ConstantScheduler.Params() - p.task_schedule.task_probs = sorted(list(p.task_probs.IterParams())) - - if p.train.ema_decay > 0: - for task_name, task_params in sorted_task_params: - for field in ['ema_decay', 'ema_decay_moving_vars']: - if task_params.train.Get(field) != p.train.Get(field): - raise ValueError('Params did not match for field %s in task %s' % - (field, task_name)) - - # CreateChild copies over global configs in p to individual task params, - # which then gets propagated down to all sub-layers during - # BaseTask._PropagateDownGlobalConfigs(), or through sub-sequent CreateChild - # or CreateChildren calls. - with tf.name_scope(p.name): - for task_name, task_params in sorted_task_params: - self.CreateChild(task_name, task_params) - - self.CreateChild('task_schedule', p.task_schedule) - - def _CreateChildrenVariables(self): - with tf.name_scope(self.params.name): - for task_name, task in zip(self.task_names, self.tasks): - if self.params.task_name_var_scope: - with tf.variable_scope(task_name): - task.InstantiateVariables() - else: - task.InstantiateVariables() - self.task_schedule.InstantiateVariables() - super()._CreateChildrenVariables() - - @property - def task_names(self): - return MultiTaskModel.TaskNames(self.params) - - @property - def tasks(self): - return [self.children[name] for name in self.task_names] - - def GetTask(self, task_name): - assert task_name, 'Must specify >task_name< for multi-task model.' - return self.children[task_name] - - def SampleTask(self, global_step): - """Sample a task according self.task_schedule. - - `self.task_schedule.cur_probs` will also be updated. - - Args: - global_step: int. Current time step. - """ - sampled_task = self.task_schedule.Sample(global_step) - tf.logging.info('Sampled task: %s', sampled_task) - return self.children[sampled_task] - - def ConstructFPropBPropGraph(self): - for task_name in self.task_names: - with tf.name_scope(task_name): - task = self.GetTask(task_name) - if self.ema: - task.ApplyExponentialMovingAverage(self.ema) - task.FPropDefaultTheta() - task.BProp() - - def ConstructFPropGraph(self): - for task_name in self.task_names: - with tf.name_scope(task_name): - task = self.GetTask(task_name) - task.FPropDefaultTheta() diff --git a/lingvo/core/base_model_params.py b/lingvo/core/base_model_params.py deleted file mode 100644 index 23a81a59..00000000 --- a/lingvo/core/base_model_params.py +++ /dev/null @@ -1,109 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""BaseModelParams class definition.""" - -from lingvo import datasets -from lingvo.core import base_input_generator -from lingvo.core import base_model -from lingvo.core import hyperparams - - -class DatasetError(Exception): - """Dataset error exception class.""" - pass - - -class _BaseModelParams: - """Base class for storing model Params for a single experiment.""" - - def GetDatasetParams(self, dataset): - """Convenience function that returns the param for the given dataset name. - - Args: - dataset: A python string. Typically, 'Dev', 'Test', etc. - - Returns: - If there is a `cls.${dataset}` method defined, call that method to - generate a hyperparam for the input data. - - Raises: - DatasetError: if there is not a `${dataset}` method defined under `cls`. - """ - try: - f = getattr(self, dataset) - except AttributeError as e: - raise DatasetError( - str(e) + - '; available datasets are: %s' % datasets.GetDatasets(type(self))) - return f() - - -class SingleTaskModelParams(_BaseModelParams): - """Model Params for a `.SingleTaskModel`.""" - - def Train(self): - """Returns Params for the training dataset.""" - return base_input_generator.BaseSequenceInputGenerator.Params().Set( - name='Train') - - def Dev(self): - """Returns Params for the development dataset.""" - return base_input_generator.BaseSequenceInputGenerator.Params().Set( - name='Dev') - - def Test(self): - """Returns Params for the testing dataset.""" - return base_input_generator.BaseSequenceInputGenerator.Params().Set( - name='Test') - - def Task(self): - """Returns task params.""" - raise NotImplementedError('Abstract method') - - def Model(self): - """Returns model params. - - Emulates structure of `MultiTaskModelParams`. - """ - return base_model.SingleTaskModel.Params(self.Task()) - - def ProgramSchedule(self): - """Returns a schedule for the Executor.""" - raise NotImplementedError('Abstract method') - - -class MultiTaskModelParams(_BaseModelParams): - """Model Params for a `.MultiTaskModel`.""" - - def Train(self): - """Returns Params for the training dataset.""" - return hyperparams.Params() - - def Dev(self): - """Returns Params for the development dataset.""" - return hyperparams.Params() - - def Test(self): - """Returns Params for the testing dataset.""" - return hyperparams.Params() - - def Model(self): - """Returns model params.""" - raise NotImplementedError('Abstract method') - - def ProgramSchedule(self): - """Returns a schedule for the Executor.""" - raise NotImplementedError('Abstract method') diff --git a/lingvo/core/base_model_params_test.py b/lingvo/core/base_model_params_test.py deleted file mode 100644 index 1c00958f..00000000 --- a/lingvo/core/base_model_params_test.py +++ /dev/null @@ -1,43 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for base_model_params.""" - -import lingvo.compat as tf -from lingvo.core import base_model_params -from lingvo.core import test_utils - - -class BaseModelParamsTest(test_utils.TestCase): - - def testGetDatasetParams_SingleTaskModelParams(self): - dummy_model = base_model_params.SingleTaskModelParams() - self.assertEqual(dummy_model.Train(), dummy_model.GetDatasetParams('Train')) - self.assertEqual(dummy_model.Dev(), dummy_model.GetDatasetParams('Dev')) - self.assertEqual(dummy_model.Test(), dummy_model.GetDatasetParams('Test')) - with self.assertRaises(base_model_params.DatasetError): - dummy_model.GetDatasetParams('Invalid') - - def testGetDatasetParams_MultiTaskModelParams(self): - dummy_model = base_model_params.MultiTaskModelParams() - self.assertEqual(dummy_model.Train(), dummy_model.GetDatasetParams('Train')) - self.assertEqual(dummy_model.Dev(), dummy_model.GetDatasetParams('Dev')) - self.assertEqual(dummy_model.Test(), dummy_model.GetDatasetParams('Test')) - with self.assertRaises(base_model_params.DatasetError): - dummy_model.GetDatasetParams('Invalid') - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/base_model_test.py b/lingvo/core/base_model_test.py deleted file mode 100644 index 3f31ca16..00000000 --- a/lingvo/core/base_model_test.py +++ /dev/null @@ -1,528 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for base_model.""" - -from absl.testing import flagsaver -import lingvo.compat as tf -from lingvo.core import base_decoder -from lingvo.core import base_input_generator -from lingvo.core import base_layer -from lingvo.core import base_model -from lingvo.core import base_model_params -from lingvo.core import distillation_task -from lingvo.core import hyperparams -from lingvo.core import layers -from lingvo.core import learner -from lingvo.core import py_utils -from lingvo.core import task_scheduler -from lingvo.core import test_utils -import numpy as np - - -FLAGS = tf.flags.FLAGS - -_NUMPY_RANDOM_SEED = 9885784 - - -class TestTask(base_model.BaseTask): - - def __init__(self, params): - super().__init__(params) - self.CreateChild('x', layers.BatchNormLayer.Params().Set(name='x', dim=1)) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - self.CreateVariable( - 'a', - py_utils.WeightParams(shape=[], init=py_utils.WeightInit.Constant(0))) - self.CreateVariable( - 'b', - py_utils.WeightParams(shape=[], init=py_utils.WeightInit.Constant(0))) - - -class BaseTaskTest(test_utils.TestCase): - - @classmethod - def TestParams(cls): - p = TestTask.Params() - p.name = 'base_mdl' - p.encoder = base_layer.BaseLayer.Params() - p.encoder.name = 'encoder' - p.decoder = base_decoder.BaseDecoder.Params() - p.decoder.name = 'decoder' - return p - - def testInit(self): - _ = self.TestParams().Instantiate() - - @flagsaver.flagsaver - def testScaleGradients(self): - p = self.TestParams() - task = p.Instantiate() - var_a = task.theta.a - var_grads = py_utils.NestedMap( - a=py_utils.VarGrad(var_a, tf.ones_like(var_a))) - scaled_grads_map = task.learners[0].ScaleGradients(var_grads) - - FLAGS.enable_check_numerics = False - with self.session(): - self.evaluate(tf.global_variables_initializer()) - self.assertEqual(1.0, scaled_grads_map.grad_scale.eval()) - # The final gradient must be finite. - self.assertFalse( - tf.math.is_nan(scaled_grads_map.final_var_grads.a[1]).eval()) - self.assertTrue( - tf.math.is_finite(scaled_grads_map.final_var_grads.a[1]).eval()) - - @flagsaver.flagsaver - def testScaleGradientsInf(self): - FLAGS.enable_check_numerics = False - p = self.TestParams() - task = p.Instantiate() - var_a = task.theta.a - # Infinite gradient. - var_grads = py_utils.NestedMap(a=py_utils.VarGrad(var_a, tf.math.log(0.))) - scaled_grads_map = task.learners[0].ScaleGradients(var_grads) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - self.assertEqual(0., scaled_grads_map.grad_scale.eval()) - # The final gradient must be finite. - self.assertFalse( - tf.math.is_nan(scaled_grads_map.final_var_grads.a[1]).eval()) - self.assertTrue( - tf.math.is_finite(scaled_grads_map.final_var_grads.a[1]).eval()) - - @flagsaver.flagsaver - def testScaleGradientsNaN(self): - FLAGS.enable_check_numerics = False - p = self.TestParams() - task = p.Instantiate() - var_a = task.theta.a - # Make a NaN gradient. - var_grads = py_utils.NestedMap( - a=py_utils.VarGrad(var_a, 0. * tf.math.log(0.))) - scaled_grads_map = task.learners[0].ScaleGradients(var_grads) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - self.assertEqual(0., scaled_grads_map.grad_scale.eval()) - # The final gradient must be finite. - self.assertFalse( - tf.math.is_nan(scaled_grads_map.final_var_grads.a[1]).eval()) - self.assertTrue( - tf.math.is_finite(scaled_grads_map.final_var_grads.a[1]).eval()) - - @flagsaver.flagsaver - def testScaleGradientsCheckNumerics(self): - """ScaleGradients when enable_check_numerics=True.""" - FLAGS.enable_check_numerics = True - p = self.TestParams() - task = p.Instantiate() - var_a = task.theta.a - # Make a NaN gradient. - var_grads = py_utils.NestedMap( - a=py_utils.VarGrad(var_a, 0. * tf.math.log(0.))) - scaled_grads_map = task.learners[0].ScaleGradients(var_grads) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - self.assertEqual(0., scaled_grads_map.grad_scale.eval()) - # Fetching the gradient raises an exception with enable_check_numerics. - with self.assertRaisesRegex(tf.errors.InvalidArgumentError, - 'is not finite'): - _ = scaled_grads_map.final_var_grads.a[1].eval() - - def testScaleGradientsError(self): - p = self.TestParams() - p.train.clip_gradient_single_norm_to_value = 1.0 - p.train.clip_gradient_norm_to_value = 1.0 - task = p.Instantiate() - var_a = task.theta.a - var_grads = py_utils.NestedMap( - a=py_utils.VarGrad(var_a, tf.ones_like(var_a))) - self.assertRaises(ValueError, task.learners[0].ScaleGradients, var_grads) - - @flagsaver.flagsaver - def testScaleGradientsSingleTensorNorm(self): - p = self.TestParams() - p.train.clip_gradient_single_norm_to_value = 1.0 - p.train.clip_gradient_norm_to_value = None - task = p.Instantiate() - - var_a = task.theta.a - var_b = task.theta.b - var_grads = py_utils.NestedMap( - a=py_utils.VarGrad(var_a, - tf.ones_like(var_a) * 10.0), - b=py_utils.VarGrad(var_b, - tf.ones_like(var_b) * 0.5)) - scaled_grads_map = task.learners[0].ScaleGradients(var_grads) - - FLAGS.enable_check_numerics = False - with self.session(): - self.evaluate(tf.global_variables_initializer()) - - # Each variable is clipped indipendently to grad scale of 1. - self.assertAllClose(scaled_grads_map.final_var_grads.a[1].eval(), 1.0) - self.assertAllClose(scaled_grads_map.final_var_grads.b[1].eval(), 0.5) - - -class TeacherTask(base_model.BaseTask): - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - self.CreateVariable( - 'x', - py_utils.WeightParams(shape=[], init=py_utils.WeightInit.Constant(0))) - - def ComputePredictions(self, theta, input_batch): - return theta.x - - -class StudentTask(base_model.BaseTask): - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - self.CreateVariable( - 'x', - py_utils.WeightParams(shape=[], init=py_utils.WeightInit.Uniform())) - - def ComputePredictions(self, theta, input_batch): - return theta.x - - -class TestInputGenerator(base_input_generator.BaseSequenceInputGenerator): - - def InfeedBatchSize(self): - """Override BaseSequenceInputGenerator.""" - return 1 - - def _InputBatch(self): - return 0 - - -class DistillationTestTask(distillation_task.DistillationTask): - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'distillation_test' - p.teacher = TeacherTask.Params() - p.student = StudentTask.Params() - p.input = TestInputGenerator.Params() - p.train.learning_rate = 1e3 - p.teacher.train = None - p.teacher.eval = None - p.student.train = None - p.student.eval = None - return p - - def __init__(self, params): - super().__init__(params) - - def ComputeLoss(self, theta, predictions, input_batch): - return {'loss': (predictions.teacher - predictions.student, 1)}, {} - - -class DistillationTaskTest(test_utils.TestCase): - - def _GetVarValuesBeforeAndAfter(self, params, steps=10): - task = params.Instantiate() - self.assertIsNotNone(task.teacher.params.input) - self.assertIsNotNone(task.student.params.input) - metrics = task.FPropDefaultTheta()[0] - self.assertCountEqual(['loss', 'num_samples_in_batch'], - list(metrics.keys())) - task.BProp() - # Expected side effects of BProp(). - self.assertIsNotNone(task.train_op) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - - variables = {} - values_before_training = {} - values_after_training = {} - for child in ('teacher', 'student'): - variables[child] = { - k: v - for k, v in getattr(task, child).vars.FlattenItems() - } - values_before_training[child] = self.evaluate(variables[child]) - - # Train for a few steps. - for _ in range(10): - self.evaluate(task.train_op) - for child in ('teacher', 'student'): - values_after_training[child] = self.evaluate(variables[child]) - return values_before_training, values_after_training - - def testFProp(self): - values_before_training, values_after_training = ( - self._GetVarValuesBeforeAndAfter(DistillationTestTask.Params())) - for child in ('teacher', 'student'): - for k, v in values_after_training[child].items(): - print('Comparing variable %s' % k) - if child == 'teacher': - # Teacher vars should not change after training. - self.assertAllEqual(values_before_training[child][k], v) - else: - # Student vars should change after training. - self.assertNotAlmostEqual(values_before_training[child][k], v) - - def testFPropTeacherEnabled(self): - params = DistillationTestTask.Params() - params.train_teacher = True - params.distillation_loss_weight.value = 0.5 - values_before_training, values_after_training = ( - self._GetVarValuesBeforeAndAfter(params)) - for child in ('teacher', 'student'): - for k, v in values_after_training[child].items(): - print('Comparing variable %s' % k) - if child == 'teacher': - # Teacher vars should change after training. - self.assertNotAlmostEqual(values_before_training[child][k], v) - else: - # Student vars should change after training. - self.assertNotAlmostEqual(values_before_training[child][k], v) - - -class SingleTaskModelTest(test_utils.TestCase): - - def testInit(self): - p = base_model.SingleTaskModel.Params() - p.task = BaseTaskTest.TestParams() - p.task.input = base_input_generator.BaseSequenceInputGenerator.Params() - p.task.train.learner = (learner.Learner.Params().Set(name='loss')) - model = p.Instantiate() - self.assertEqual(model.params.name, model.GetTask().params.name) - self.assertEqual(model.params.task, model.GetTask().params) - self.assertEqual(len(model.tasks), 1) - self.assertEqual(model.tasks[0], model.GetTask()) - self.assertEqual(model.tasks[0], model.SampleTask(None)) - - def testExponentialMovingAverage(self): - p = base_model.SingleTaskModel.Params() - p.task = BaseTaskTest.TestParams() - p.task.input = base_input_generator.BaseSequenceInputGenerator.Params() - p.task.train.ema_decay = 0.9 - p.task.train.ema_decay_moving_vars = False - model = p.Instantiate() - task = model._task - task._train_op = tf.no_op() - task.ApplyExponentialMovingAverage(model.ema) - with tf.variable_scope('base_mdl', reuse=True): - beta = tf.get_variable('x/beta/var') - mean = tf.get_variable('x/moving_mean/var') - self.assertIsNotNone(model.ema.average(beta)) - self.assertIsNone(model.ema.average(mean)) - - def testExponentialMovingAverageIncludingMovingVars(self): - p = base_model.SingleTaskModel.Params() - p.task = BaseTaskTest.TestParams() - p.task.input = base_input_generator.BaseSequenceInputGenerator.Params() - p.task.train.ema_decay = 0.9 - p.task.train.ema_decay_moving_vars = True - p.task.input = base_input_generator.BaseSequenceInputGenerator.Params() - model = p.Instantiate() - task = model._task - task._train_op = tf.no_op() - task.ApplyExponentialMovingAverage(model.ema) - with tf.variable_scope('base_mdl', reuse=True): - beta = tf.get_variable('x/beta/var') - mean = tf.get_variable('x/moving_mean/var') - self.assertIsNotNone(model.ema.average(beta)) - self.assertIsNotNone(model.ema.average(mean)) - - -class MultiTaskModelTest(test_utils.TestCase): - - def testInitMissingInputParams(self): - p = base_model.MultiTaskModel.Params() - p.name = 'MultiTaskModel' - p0 = BaseTaskTest.TestParams() - p0.train.learner = (learner.Learner.Params().Set(name='loss')) - p1 = BaseTaskTest.TestParams() - p1.train.learner = (learner.Learner.Params().Set(name='loss')) - - p.input = base_model_params.MultiTaskModelParams().Train() - p.input.Define('a', - base_input_generator.BaseSequenceInputGenerator.Params(), '') - - p.task_params = hyperparams.Params() - p.task_params.Define('a', p0, '') - p.task_params.Define('b', p1, '') - - p.task_probs = hyperparams.Params() - p.task_probs.Define('a', 0.5, '') - p.task_probs.Define('b', 0.5, '') - self.assertRaises(AttributeError, p.Instantiate) - - def testInit(self): - p = base_model.MultiTaskModel.Params() - p.name = 'MultiTaskModel' - p0 = BaseTaskTest.TestParams() - p0.train.learner = (learner.Learner.Params().Set(name='loss')) - p1 = BaseTaskTest.TestParams() - p1.train.learner = (learner.Learner.Params().Set(name='loss')) - - p.input = base_model_params.MultiTaskModelParams().Train() - p.input.Define('a', - base_input_generator.BaseSequenceInputGenerator.Params(), '') - p.input.Define('b', - base_input_generator.BaseSequenceInputGenerator.Params(), '') - - p.task_params = hyperparams.Params() - p.task_params.Define('a', p0, '') - p.task_params.Define('b', p1, '') - - p.task_probs = hyperparams.Params() - p.task_probs.Define('a', 0.5, '') - p.task_probs.Define('b', 0.5, '') - - model = p.Instantiate() - self.assertEqual(len(model.tasks), 2) - self.assertEqual(set(model.task_names), {'a', 'b'}) - self.assertEqual(set(model.tasks), {model.GetTask('a'), model.GetTask('b')}) - self.assertEqual(model.params.task_params.a, model.GetTask('a').params) - self.assertEqual(model.params.task_params.b, model.GetTask('b').params) - - def _setUpTestSampleTask(self): - np.random.seed(_NUMPY_RANDOM_SEED) - - # define and initialize tasks, model and params - p = base_model.MultiTaskModel.Params() - p.name = 'MultiTaskModel' - p0 = BaseTaskTest.TestParams() - p1 = BaseTaskTest.TestParams() - - p.input = base_model_params.MultiTaskModelParams().Train() - p.input.Define('a', - base_input_generator.BaseSequenceInputGenerator.Params(), '') - p.input.Define('b', - base_input_generator.BaseSequenceInputGenerator.Params(), '') - - p.task_params = hyperparams.Params() - p.task_params.Define('a', p0, '') - p.task_params.Define('b', p1, '') - - return p - - def _testSampleTaskHelper(self, p): - model = p.Instantiate() - - task_to_id = {model.children['a']: 'a', model.children['b']: 'b'} - task_counts = {'a': 0, 'b': 0} - - # initialize tensorflow graph and global step - with self.session(): - self.evaluate(tf.global_variables_initializer()) - global_step = self.evaluate(model.global_step) - for _ in range(100): - task = model.SampleTask(global_step) - task_counts[task_to_id[task]] += 1 - - self.assertEqual(task_counts['a'], 83) - self.assertEqual(task_counts['b'], 17) - - def testSampleTaskSpecifiedWithoutScheduler(self): - """Expected distribution: 'a': 0.8 , 'b': 0.2.""" - p = self._setUpTestSampleTask() - - p.task_probs = hyperparams.Params() - p.task_probs.Define('a', 0.8, '') - p.task_probs.Define('b', 0.2, '') - - self._testSampleTaskHelper(p) - - def testSampleTask(self): - """Expected distribution: 'a': 0.8 , 'b': 0.2.""" - p = self._setUpTestSampleTask() - - p.task_schedule = task_scheduler.ConstantScheduler.Params() - p.task_schedule.task_probs = [('a', 0.8), ('b', 0.2)] - - self._testSampleTaskHelper(p) - - -class PostTrainingTask(base_model.BaseTask): - - def __init__(self, params): - super().__init__(params) - p = layers.FeedForwardNet.Params().Set( - name='ffn', - input_dim=10, - hidden_layer_dims=[20, 30], - batch_norm=True, - activation='TANH', - params_init=py_utils.WeightInit.Uniform(1.0)) - self.CreateChild('ffn', p) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - self.CreateVariable( - 'counter1', - py_utils.WeightParams(shape=[], init=py_utils.WeightInit.Constant(0))) - self.CreateVariable( - 'counter2', - py_utils.WeightParams(shape=[], init=py_utils.WeightInit.Constant(0))) - - def PostTrainingStepUpdate(self, global_step): - # We expect the training step to be done, so capture - # the value of counter1 into counter2. - return tf.assign(self.vars.counter2, self.vars.counter1) - - def ComputePredictions(self, theta, input_batch): - input_data = tf.random.normal([1, 10], dtype=tf.float32) + tf.cast( - input_batch, tf.float32) - add = tf.assign_add(self.vars.counter1, 1.) - input_data += add - result = self.ffn.FProp(theta.ffn, input_data) - return {'result': result} - - def ComputeLoss(self, theta, predictions, input_batch): - loss = tf.reduce_sum(predictions['result']) - return {'loss': (loss, 1)}, {} - - -class PostTrainingTest(test_utils.TestCase): - - @classmethod - def TestParams(cls): - p = PostTrainingTask.Params() - p.name = 'base_mdl' - p.input = TestInputGenerator.Params() - return p - - def testPost(self): - p = self.TestParams() - task = p.Instantiate() - task.FPropDefaultTheta() - task.BProp() - train_op = task.train_op - with self.session(): - self.evaluate(tf.global_variables_initializer()) - for _ in range(20): - self.evaluate(train_op) - c1, c2 = self.evaluate([task.vars.counter1, task.vars.counter2]) - # Both vars should have the same value if the PostTrainingStep - # happens after the training step. - self.assertEqual(c1, c2) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/batch_major_attention.py b/lingvo/core/batch_major_attention.py deleted file mode 100644 index 7c1dc4dd..00000000 --- a/lingvo/core/batch_major_attention.py +++ /dev/null @@ -1,4216 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Multi-headed attention layers for Transformer machine translation. - -[1] Attention is all you need. - https://arxiv.org/pdf/1706.03762.pdf Section 3. -""" - -import bisect -from lingvo import compat as tf -from lingvo.core import attention_util -from lingvo.core import base_layer -from lingvo.core import builder -from lingvo.core import conv_layers_builder as conv_layers -from lingvo.core import gpipe -from lingvo.core import hyperparams -from lingvo.core import layers -from lingvo.core import layers_with_attention -from lingvo.core import moe_layers -from lingvo.core import py_utils -from lingvo.core import symbolic -from lingvo.core import tshape -# pylint: disable=g-direct-tensorflow-import -from tensorflow.python.ops import inplace_ops - -# pylint: enable=g-direct-tensorflow-import - - -def CausalPadding(slen, dtype=tf.float32): - return 1 - tf.linalg.band_part(tf.ones([slen, slen], dtype=dtype), -1, 0) - - -def GetDtypeMin(dtype=tf.float32): - return tf.constant(-0.7, dtype=dtype) * dtype.max - - -def SegmentMask(segment_id, - source_segment_id, - dtype=tf.float32, - apply_dtype_min=True): - """Calculates a segment mask for attention. - - Args: - segment_id: [B, T] - source_segment_id: [B, S] - dtype: data type of generated mask. - apply_dtype_min: Outputs a 0/1 padding mask if set to False. This is needed - for GPipe layers to avoid nan issues. - - Returns: - segment_mask: [B, 1, T, S]: A mask that is ready to - be added to [B, N, T, S] attention logits. if apply_dtype_min is False, - outputs a 0/1 padding mask instead. - """ - if segment_id is None or source_segment_id is None: - return None - # Compute [B, T, S] = [B, T, 1] != [B, 1, S] - ret = tf.cast( - tf.not_equal( - tf.expand_dims(segment_id, 2), tf.expand_dims(source_segment_id, 1)), - dtype=dtype) - if apply_dtype_min: - ret *= GetDtypeMin(ret.dtype) - # [B, T, S] -> [B, 1, T, S] - return tf.expand_dims(ret, axis=1) - - -class PerDimScaleLayer(base_layer.BaseLayer): - """A layer to scale individual dims of the input.""" - - @classmethod - def Params(cls): - """Params for `PerDimScaleLayer`.""" - p = super().Params() - p.Define('dim', 0, 'Number of individual dims .') - return p - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - pc = py_utils.WeightParams( - shape=[p.dim], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('per_dim_scale', pc) - - def FProp(self, theta, inputs): - """Return theta.scale * inputs. - - Args: - theta: weights defined in this layer. - inputs: 4D tensor with shape [..., p.dim] - - Returns: - outpus: 4D tensor with shape [..., p.dim] - """ - p = self.params - dim = symbolic.ToStatic(p.dim) - inputs = py_utils.HasShape(inputs, [-1, -1, -1, dim]) - scale = tf.math.rsqrt(tf.cast(dim, inputs.dtype)) - scale *= tf.nn.softplus(theta.per_dim_scale) / tf.nn.softplus( - tf.constant(0.0, dtype=inputs.dtype)) - return inputs * scale - - @classmethod - def FPropMeta(cls, p, inputs): - py_utils.CheckShapes((inputs,)) - return py_utils.NestedMap( - flops=inputs.num_elements() * 5, out_shapes=(inputs,)) - - -class MultiHeadedProjectionLayer(base_layer.BaseLayer): - """Layer that computes multi heads projection. - - This layer is expected to be used within MultiHeadedAttention below. - """ - - @classmethod - def Params(cls): - """Params for MultiHeadedProjectionLayer.""" - p = super().Params() - p.Define('input_dim', 0, 'Input dimension.') - p.Define('num_heads', 0, 'Number of heads.') - p.Define('dim_per_head', 0, 'Size of each head.') - p.Define( - 'is_output_projection', False, - 'Whether it is out projection or not. If False, we use ' - '"BTD,DNH->BTNH" for query,key,value projection. Otherwise we use ' - '"BTNH,DNH->BTD" for output projection.') - p.Define('use_bias', True, 'If to add bias in projection.') - p.Define('xla_num_partitions', None, 'Number of SPMD partitions.') - return p - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - pc = py_utils.WeightParams( - shape=[p.input_dim, p.num_heads, p.dim_per_head], - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('w', pc) - if p.use_bias: - if p.is_output_projection: - pc_bias = py_utils.WeightParams( - shape=[p.input_dim], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - else: - pc_bias = py_utils.WeightParams( - shape=[p.num_heads, p.dim_per_head], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('b', pc_bias) - - def FProp(self, theta, inputs): - """Computes the multi headed projection for inputs. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: A tensor of shape [batch_size, time_steps, num_heads, - dim_per_head] or [batch_size, time_steps, hidden_size]. - - Returns: - The projected tensor with shape [[batch_size, time_steps, hidden_size] or - [batch_size, time_steps, num_heads, dim_per_head]. - """ - p = self.params - if p.xla_num_partitions: - theta.w = moe_layers.Split( - theta.w, 1, p.xla_num_partitions, use_sharding_op=True) - if p.is_output_projection: - inputs = py_utils.HasShape( - inputs, [-1, -1, p.num_heads, - symbolic.ToStatic(p.dim_per_head)]) - ret = tf.einsum('BTNH,DNH->BTD', inputs, theta.w) - else: - inputs = py_utils.HasShape( - inputs, [-1, -1, symbolic.ToStatic(p.input_dim)]) - ret = tf.einsum('BTD,DNH->BTNH', inputs, theta.w) - if p.use_bias: - if p.xla_num_partitions and not p.is_output_projection: - theta.b = moe_layers.Split( - theta.b, 0, p.xla_num_partitions, use_sharding_op=True) - ret += theta.b - return ret - - -class MultiHeadedAttention(base_layer.BaseLayer): - """Dot-product attention with multiple attention heads. - - This implementation heavily uses einsum (wrapped in py_utils.Einsum) to be - efficient on TPUs. We use the following capital letters to denote certain - tensor parameters. - - B = batch size - S = length of the key/value (source) - T = length of the query (target) - D = model dimension - N = number of attention heads - H = dimensions of each attention head. - - The algorithm is sketched as follows. Each intermediate tensor or weight - tensor is annotated with its shape. E.g., Wq, the weight tensor for query's - projection, its shape is [D, N, H]. - - Trainable weights: - Wq, Wk, Wv: [D, N, H] - Wout: [D, N, H] - - Input q:[B, T, D]; k:[B, S, D]; v:[B, S, D] - q_proj:[B, T, N, H] = einsum('BTD,DNH->BTNH', x, Wq) - k_proj:[B, S, N, H] = einsum('BSD,DNH->BSNH', x, Wk) - v_proj:[B, S, N, H] = einsum('BSD,DNH->BSNH', x, Wv) - logits:[B, N, T, S] = einsum('BTNH,BSNH->BNTS', q_proj, k_proj) / sqrt(H) - probs:[B, N, T, S] = softmax(logits) - context:[B, T, N, H] = einsum('BNTS,BSNH->BTNH', probs, v_proj) - Output y:[B, T, D] = einsum('BTNH,DNH>BTD', context, Wout) - """ - - @classmethod - def Params(cls): - """Params for _MultiHeadedAttention.""" - p = super().Params() - p.Define('input_dim', 0, 'Number of key nodes.') - p.Define('hidden_dim', 0, 'Number of hidden nodes.') - p.Define('num_heads', 1, 'Num of attention heads.') - p.Define('dropout_tpl', layers.DropoutLayer.Params(), - 'Params for dropout layer.') - p.Define( - 'enable_value_proj', True, 'Whether value v is pre-projected ' - ' before self attention or not.') - p.Define('enable_per_dim_scale', True, - 'Whether using per_dim_scale or scaling by a constant factor.') - p.Define('atten_dropout_prob', 0.0, - 'Probability at which we apply dropout to the attention weights.') - p.Define('proj_tpl', MultiHeadedProjectionLayer.Params(), 'Params for ' - 'projection layer.') - p.Define('packed_input', False, 'Whether there is packed input.') - p.Define('use_bias', True, 'Whether to use bias for projection layers.') - p.Define('xla_num_partitions', None, 'Number of SPMD partitions.') - p.Define( - 'enable_scaling_code_motion', False, 'Move scalings from the side ' - 'of T^2 to the side of T for better performance. This may result ' - 'in model quality drops when using bf16 for some models due to ' - 'different XLA fusion decisions.') - return p - - def __init__(self, params): - """Constructs a _MultiHeadedAttention object.""" - super().__init__(params) - p = self.params - assert p.input_dim, 'input_dim is {}'.format(p.input_dim) - assert p.hidden_dim, 'hidden_dim is {}'.format(p.hidden_dim) - assert (symbolic.IsExpr(p.hidden_dim) or p.hidden_dim % p.num_heads == 0), ( - f'hidden_dim: {p.hidden_dim} is not a multiple of num_heads: ' - f'{p.num_heads}.') - dim_per_head = p.hidden_dim // p.num_heads - - def ProjectInput(): - return p.proj_tpl.Copy().Set( - input_dim=p.input_dim, - num_heads=p.num_heads, - dim_per_head=dim_per_head, - use_bias=p.use_bias, - xla_num_partitions=p.xla_num_partitions) - - self.CreateChild('key', ProjectInput()) - self.CreateChild('query', ProjectInput()) - if p.enable_value_proj: - self.CreateChild('value', ProjectInput()) - if p.enable_per_dim_scale: - self.CreateChild('per_dim_scale', - PerDimScaleLayer.Params().Set(dim=dim_per_head)) - self.CreateChild('atten_dropout', - p.dropout_tpl.Set(keep_prob=1.0 - p.atten_dropout_prob)) - # Setting is_output_projection=True to set the projection direction - # from hidden dim to input dim. - self.CreateChild( - 'post', - p.proj_tpl.Copy().Set( - input_dim=p.input_dim, - num_heads=p.num_heads, - dim_per_head=dim_per_head, - is_output_projection=True, - use_bias=p.use_bias, - xla_num_partitions=p.xla_num_partitions)) - - def _AttenLogits(self, theta, query, key): - """Computes attention logits. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query: A Tensor of shape [B, T, N, H] - key: A Tensor of shape [B, T, N, H] - - Returns: - A Tensor of shape [B, N, T, S] - """ - return tf.einsum('BTNH,BSNH->BNTS', query, key) - - def _AttenLogitsOneStep(self, theta, query, key, time_step): - """Attention logits for one single target (query) step. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query: [B, N, H]. - key: [S, B, N, H] or [S, B, N*H/128, 128]. - time_step: Current time step. - - Returns: - A Tensor of shape [S, B, N] - """ - p = self.params - s, b, _, _ = py_utils.GetShape(key, 4) - n = p.num_heads - h = p.hidden_dim // n - - # [s, b, n] - return tf.einsum('BNH,SBNH->SBN', query, tf.reshape(key, [s, b, n, h])) - - def AttenProbs(self, - theta, - query, - key, - paddings, - segment_mask, - per_step_padding=None): - """Compute attention probability. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query: [B, T, N, H]. - key: [B, S, N, H]. - paddings: [B, S]. - segment_mask: [B, 1, T, S]: A mask that is applied to prevent attention - between different segments. This is already been converted into large - negative logits. Only applied if packed_input = True. - per_step_padding: A mask used by decoder self-attention to prevent - information flow from future (causal padding). It has shape [B, T, S] if - not None. - - Returns: - probs: [B, N, T, S]. - probs_sum: [B, N, T, 1]. - """ - key = py_utils.HasRank(key, 4) - b, s, n, h = py_utils.GetShape(key, 4) - query = py_utils.HasShape(query, [b, -1, n, h]) - t = py_utils.GetShape(query)[1] - if segment_mask is not None and self.params.packed_input: - segment_mask = py_utils.HasShape(segment_mask, [b, 1, t, s]) - - with tf.name_scope('logits'): - # Keep softmax computation in float32 otherwise the low precision can - # can lead to worse quality. - logits = tf.cast(self._AttenLogits(theta, query, key), tf.float32) - - # Apply segment mask. - if self.params.packed_input and segment_mask is not None: - # Paddings have been included in segment_mask. - padded_logits = logits + tf.cast(segment_mask, tf.float32) - else: - # Exclude padding frames. - paddings = py_utils.HasShape(paddings, [b, s]) - paddings = tf.tile(tf.reshape(paddings, [b, 1, 1, s]), [1, n, t, 1]) - if per_step_padding is not None: - per_step_padding = tf.tile( - tf.expand_dims(per_step_padding, 1), [1, n, 1, 1]) - paddings += per_step_padding - - very_negative_logits = ( - tf.ones_like(logits) * logits.dtype.max * - tf.constant(-0.7, dtype=logits.dtype)) - padded_logits = tf.where(paddings > 0.0, very_negative_logits, logits) - - if self.params.enable_scaling_code_motion: - # Split the softmax into two parts. Do the 1st part here; the 2nd part - # (scaling) is moved after _AttenContext for better performance. - probs = padded_logits - tf.stop_gradient( - tf.reduce_max(padded_logits, -1, True)) - probs = tf.cast(tf.exp(probs), key.dtype) - probs_sum = tf.reduce_sum(probs, -1, True) - else: - probs = tf.cast(tf.nn.softmax(padded_logits), key.dtype) - probs_sum = None - - probs = py_utils.HasShape(probs, [b, n, t, s]) - return probs, probs_sum - - def _AttenContext(self, theta, probs, value): - return tf.einsum('BNTS,BSNH->BTNH', probs, value) - - def _AttenContextOneStep(self, theta, probs, value, time_step): - s, b, _, _ = py_utils.GetShape(value, 4) - n = self.params.num_heads - h = self.params.hidden_dim // n - - return tf.einsum('SBN,SBNH->BNH', probs, tf.reshape(value, [s, b, n, h])) - - def _DotAtten(self, - theta, - query, - key, - value, - paddings, - segment_mask, - per_step_padding=None): - """Main attention function. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query: [B, T, N, H]. - key: [B, S, N, H]. - value: [B, S, N, H]. - paddings: [B, S]. - segment_mask: [B, 1, T, S]: A mask that is applied to prevent attention - between different segments. This is already been converted into large - negative logits. Only applied if packed_input = True. - per_step_padding: A mask used by decoder self-attention to prevent - information flow from future (causal padding). It has shape [B, T, S] if - not None. - - Returns: - encoded: [B, T, N, H]. - atten_probs: [B, N, T, S]. - """ - p = self.params - # Scale the query projection. - if p.enable_per_dim_scale: - query = self.per_dim_scale.FProp(theta.per_dim_scale, query) - else: - query *= (p.hidden_dim // p.num_heads)**-0.5 - - # Compute prob with shape [batch, heads, target_time, source_time]. - with tf.name_scope('probs'): - probs, probs_sum = self.AttenProbs(theta, query, key, paddings, - segment_mask, per_step_padding) - # Apply dropout to probs. - probs = self.atten_dropout.FProp(theta.atten_dropout, probs) - - # Compute the attention context vector. - with tf.name_scope('ctx'): - encoded = self._AttenContext(theta, probs, value) - if p.enable_scaling_code_motion: - # The 2nd part of the softamx --- scaling. - encoded = encoded / tf.transpose(probs_sum, [0, 2, 1, 3]) - - return encoded, probs - - def _DotAttenOneStep(self, - theta, - query, - key, - value, - paddings, - segment_mask, - per_step_padding=None, - time_step=None, - use_short_seq_opt=False): - """Dot attention function for queries with 1 time step. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query: [B, 1, N, H]. - key: [S, B, N, H] or [S, B, N*H/128, 128]. - value: [S, B, N, H] or [S, B, N*H/128, 128]. - paddings: [B, S]. - segment_mask: [B, 1, T, S]: A mask that is applied to prevent attention - between different segments. This is already been converted into large - negative logits. Only applied if packed_input = True. - per_step_padding: A mask used by decoder self-attention to prevent - information flow from future (causal padding). It has shape [B, 1, S] if - not None. - time_step: Current time step. - use_short_seq_opt: A bool, whether using short sequence optimization. - - Returns: - encoded: [B, 1, N, H]. - """ - p = self.params - # Scale the query projection. - if p.enable_per_dim_scale: - query = self.per_dim_scale.FProp(theta.per_dim_scale, query) - else: - query *= (p.hidden_dim // p.num_heads)**-0.5 - - key = py_utils.HasRank(key, 4) - - b, t, n, h = py_utils.GetShape(query, 4) - s, b, _, _ = py_utils.GetShape(key, 4) - paddings = py_utils.HasShape(paddings, [b, s]) - assert t == 1 - - if per_step_padding is not None: - paddings += tf.squeeze(per_step_padding, 1) - - query = tf.reshape(query, [b, n, h]) - pad = tf.reshape( - tf.tile(tf.expand_dims(tf.transpose(paddings), 2), [1, 1, n]), [s, -1]) - very_negative_logits = ( - tf.ones_like(pad) * query.dtype.max * - tf.constant(-0.7, dtype=query.dtype)) - - def _LongSeq(): - """For long sequence, directly apply to the entire tensor with padding.""" - logits = self._AttenLogitsOneStep(theta, query, key, time_step) - - logits = tf.reshape(logits, [s, -1]) - padded_logits = tf.where(pad > 0.0, very_negative_logits, logits) - probs = tf.nn.softmax(padded_logits, axis=0) - probs = tf.reshape(probs, [s, b, n]) - - encoded = self._AttenContextOneStep(theta, probs, value, time_step) - return tf.expand_dims(encoded, 1) - - def _ShortSeq(): - """For short sequence, using while loop for early exit.""" - - def _AttenStep(o, k, q, ts): - """Computes logits for attention prob for one step. - - Args: - o: the output logits of shape [S, B*N] - k: cached key of shape [S, B, N*H/128, 8] - q: query of shape [B, N, H] - ts: a scala tensor to represent time_step - - Returns: - Updated logits and time steps. - """ - ot = tf.reshape( - tf.reduce_sum(tf.reshape(tf.gather(k, ts), [-1, n, h]) * q, -1), - [-1]) - return inplace_ops.alias_inplace_update(o, ts, ot), k, q, ts + 1 - - logits, _, _, _ = tf.while_loop( - lambda _o, _k, _q, ts: ts <= time_step, - _AttenStep, - loop_vars=(inplace_ops.empty([s, b * n], query.dtype, - init=True), key, query, - tf.zeros([], tf.int32))) - - padded_logits = tf.where(pad > 0.0, very_negative_logits, logits) - probs = tf.nn.softmax(padded_logits, axis=0) - - def _DotStep(o, p, v, ts): - """Computes encoded activation. - - Args: - o: the output activation of shape [B, N, H] - p: probabiliy of shape [S, B*N] - v: cached value of shape [S, B, N*H/128, 8] - ts: a scala tensor to represent time_step - - Returns: - Updated output and time steps. - """ - return o + tf.reshape(tf.gather(p, ts), [-1, n, 1]) * tf.reshape( - tf.gather(v, ts), [-1, n, h]), p, v, ts + 1 - - encoded, _, _, _ = tf.while_loop( - lambda o, p, v, ts: ts <= time_step, - _DotStep, - loop_vars=(tf.zeros([b, n, h], - probs.dtype), probs, value, tf.zeros([], - tf.int32))) - return tf.expand_dims(encoded, 1) - - return _ShortSeq() if use_short_seq_opt else _LongSeq() - - def FProp(self, - theta, - query_vec, - key_vec, - value_vec, - paddings, - segment_mask=None, - per_step_padding=None): - """Computes the value vector given the current query output. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [B, T, D]. - key_vec: [B, S, D]. - value_vec: [B, S, D]. - paddings: [B, S]. - segment_mask: [B, 1, T, S]. A mask only applied if packed_input=True. - per_step_padding: A mask used by decoder self-attention to prevent - information flow from future (causal padding). It has shape [B, T, T] if - not None. - - Returns: - encoded: [B, T, D]. - atten_probs: [B, N, T, S]. - - Raises: - ValueError: If value projection is disabled. - """ - p = self.params - - # Project inputs to key, value and query, respectively has shape - # [B, S, N, H], [B, S, N, H], and [B, T, N, H]. - query_proj = self.query.FProp(theta.query, query_vec) - key_proj = self.key.FProp(theta.key, key_vec) - if p.enable_value_proj: - value_proj = self.value.FProp(theta.value, value_vec) - else: - h = p.num_heads - _, _, d = py_utils.GetShape(value_vec, 3) - dh = d // h - # TODO(b/119531146): Reshape is inefficient here. Use one-hot matmul - # avoids the data formatting. Change this back to reshape once XLA - # has optimized reshape performance. - rhs = tf.reshape( - tf.one_hot(tf.range(d) // dh, h, dtype=value_vec.dtype), - [d, h, 1]) * tf.reshape( - tf.one_hot(tf.range(d) % dh, dh, dtype=value_vec.dtype), - [d, 1, dh]) - value_proj = tf.einsum('BTD,DNH->BTNH', value_vec, rhs) - - if p.packed_input and not self.do_eval: - assert segment_mask is not None - encoded, atten_probs = self._DotAtten(theta, query_proj, key_proj, - value_proj, paddings, segment_mask, - per_step_padding) - # Post projection - encoded = self.post.FProp(theta.post, encoded) - return encoded, atten_probs - - def InitStates(self, theta, target_batch_size, target_max_length): - p = self.params - num_heads = p.num_heads - atten_dim = p.hidden_dim - if not atten_dim: # Check for Pathways as atten_tpl.hidden_dim is not set. - atten_dim = p.input_dim - dim_per_head = atten_dim // num_heads - # TODO(shafey): Determine if we want to make the cached shape 128 to - # avoid padding and more efficient interpolation in beamsearch. - return py_utils.NestedMap( - key=inplace_ops.empty( - shape=(target_max_length, target_batch_size, num_heads, - dim_per_head), - dtype=py_utils.FPropDtype(p), - init=True), - value=inplace_ops.empty( - shape=(target_max_length, target_batch_size, num_heads, - dim_per_head), - dtype=py_utils.FPropDtype(p), - init=True)) - - def ExtendStep(self, - theta, - query_vec, - cached_states, - paddings, - segment_mask, - per_step_padding, - time_step, - use_short_seq_opt=False): - """Computes the value vector given the query of the current step. - - This function is used by autoregressive decoding. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [B, 1, D]. - cached_states: A `.NestedMap` object containing tensors which are the - results of previous attentions, used for fast decoding. key - [T, B, - N, H]. value - [T, B, N, H]. - paddings: [B, T], or None if there is no padding. - segment_mask: [B, 1, T, S] or None. - per_step_padding: A mask used by decoder self-attention to prevent - information flow from future (causal padding). It has shape [B, 1, T] if - not None. - time_step: A scalar or tensor with [B], current decode step, 0-based. - if it's a scalar, all the time step are the same decode step. - if it's a tensor, it represents current decode step for each sample. - use_short_seq_opt: A bool, whether using short sequence optimization. - - Returns: - encoded: [B, 1, D]. - updated_key_vec: [T, B, N, H]. - updated_value_vec: [T, B, N, H]. - - Raises: - ValueError: If value projection is disabled. - """ - p = self.params - if not p.enable_value_proj: - raise ValueError('Value projection must be enabled for Transformer ' - 'machine translation.') - - time_step = tf.convert_to_tensor(time_step) - synced_time_step = (time_step.shape.ndims == 0) - t, b, n, h = py_utils.GetShape(cached_states.key, 4) - - # Project inputs to key, value and query. Each has shape [B, 1, N, H]. - new_key_proj = self.key.FProp(theta.key, query_vec) - new_value_proj = self.value.FProp(theta.value, query_vec) - query_proj = self.query.FProp(theta.query, query_vec) - - # Using a if condtion, in case it's more efficient to update the same index. - if synced_time_step: - # The extended_key and extended_value have shape [T, B, N, H]. - extended_key = inplace_ops.alias_inplace_update( - cached_states.key, time_step, tf.reshape(new_key_proj, [b, n, h])) - extended_value = inplace_ops.alias_inplace_update( - cached_states.value, time_step, tf.reshape(new_value_proj, [b, n, h])) - else: - # The extended_key and extended_value have shape [T, B, N, H]. - selected_indices = tf.range(b) + time_step * b - extended_key = inplace_ops.alias_inplace_update( - tf.reshape(cached_states.key, [-1, n, h]), selected_indices, - tf.reshape(new_key_proj, [b, n, h])) - extended_value = inplace_ops.alias_inplace_update( - tf.reshape(cached_states.value, [-1, n, h]), selected_indices, - tf.reshape(new_value_proj, [b, n, h])) - extended_key = tf.reshape(extended_key, [t, b, n, h]) - extended_value = tf.reshape(extended_value, [t, b, n, h]) - updated_state = py_utils.NestedMap(key=extended_key, value=extended_value) - - if paddings is None: - paddings = tf.zeros([b, t], dtype=query_vec.dtype) - - encoded = self._DotAttenOneStep( - theta, - query_proj, - extended_key, - extended_value, - paddings, - segment_mask, - per_step_padding, - time_step=time_step, - use_short_seq_opt=use_short_seq_opt) - - # Post projection. - encoded = self.post.FProp(theta.post, encoded) - return encoded, updated_state - - @classmethod - def FPropMeta(cls, p, *args): - # args[0]: [b, t, d], args[1]: [b, s, d], args[2]: [b, s, d], - # args[3]: [b, s], args[4]: [b, t, s] if not None - args = tuple(py_utils.Flatten(args)) - py_utils.CheckShapes(args) - b, t, d = args[0] - s = args[3][1] - n = p.num_heads - # O(b * t * s * d) computation for self-attention and there are four - # projection layers, two of which has O(b * t * d^2), the other two has - # O(b * s * d^2). Each multiple-sum took 2 flops. Approximately - # self_attention took 15 flops per element since softmax is expensive. - flops = 15 * b * t * s * d + 2 * 2 * (b * t * d * d + b * s * d * d) - return py_utils.NestedMap(flops=flops, out_shapes=(args[0], (b, n, t, s))) - - -class MultiHeadedAttentionXL(MultiHeadedAttention): - """Transformer-XL multiheaded attention with relative positional embedding. - - https://arxiv.org/pdf/1901.02860.pdf section 3.3. - - Notice this is only intended for self attention. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('rel_pos_emb_dim', None, - 'Dimension of relative positional embedding.') - p.Define('skip_term_b', False, - 'If True, skip term_b in the paper section 3.3.') - return p - - def __init__(self, params): - """Constructs a MultiHeadedAttentionXL object.""" - super().__init__(params) - params = self.params - - assert not params.packed_input, 'Packed input not implemented yet.' - - if params.rel_pos_emb_dim is None or params.rel_pos_emb_dim <= 0: - raise ValueError('Invalide rel_pos_emb_dim: %s' % params.rel_pos_emb_dim) - - emb_params = layers.PositionalEmbeddingLayer.Params().Set( - embedding_dim=params.rel_pos_emb_dim) - self.CreateChild('pos_emb', emb_params) - - # Projection layer for relative position encoding - dim_per_head = params.hidden_dim // params.num_heads - pos_proj_tpl = params.proj_tpl.Copy().Set( - input_dim=params.rel_pos_emb_dim, - num_heads=params.num_heads, - dim_per_head=dim_per_head, - use_bias=False) - self.CreateChild('pos_proj', pos_proj_tpl) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - params = self.params - - dim_per_head = params.hidden_dim // params.num_heads - u_pc = py_utils.WeightParams( - shape=[params.num_heads, dim_per_head], - init=py_utils.WeightInit.Constant(0.0), - dtype=params.dtype, - collections=[self.__class__.__name__ + '_vars']) - v_pc = py_utils.WeightParams( - shape=[params.num_heads, dim_per_head], - init=py_utils.WeightInit.Constant(0.0), - dtype=params.dtype, - collections=[self.__class__.__name__ + '_vars']) - - self.CreateVariable('u', u_pc) - self.CreateVariable('v', v_pc) - - def _AttenLogits(self, theta, query, key): - b, _, n, h = py_utils.GetShape(key, 4) - t = py_utils.GetShape(query)[1] - - # This layer only supports self attention. - key = py_utils.HasShape(key, [b, t, n, h]) - - # [1, 2T - 1] - pos = tf.expand_dims(tf.range(-(t - 1), t, name='relative_pos'), 0) - sin_emb = self.pos_emb.FPropWithPosition(theta.pos_emb, pos) - # [1, 2T - 1, N, H] - sin_emb = self.pos_proj.FProp(theta.pos_proj, sin_emb) - # [2T - 1, N, H] - sin_emb = tf.squeeze(sin_emb, 0) - - logits = attention_util.AttenLogitsTransformerXL(query, key, sin_emb, - theta.u, theta.v, - self.params.skip_term_b) - return logits - - def _AttenLogitsOneStep(self, theta, query, key, time_step): - """Attention logits for one single target (query) step. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query: [B, N, H]. - key: [S, B, N, H] or [S, B, N*H/128, 128]. - time_step: Current time step. - if it's a scalar, all the time step are the same decode step. - if it's a tensor, it represents current decode step for each sample. - - Returns: - A Tensor of shape [S, B, N] - """ - p = self.params - synced_time_step = (time_step.shape.ndims == 0) - s, b, _, _ = py_utils.GetShape(key, 4) - n = p.num_heads - h = p.hidden_dim // n - - # Transformer_XL relative attention. - if time_step is None: - raise ValueError('`time_step` can not be None when using relative ' - 'position encoding in attention.') - # term a and c. - logits = tf.einsum('BNH,SBNH->SBN', query + theta.u, - tf.reshape(key, [s, b, n, h])) - if synced_time_step: - position = tf.expand_dims(time_step - tf.range(s), 0) - else: - # [b, s] - position = ( - tf.expand_dims(time_step, -1) - - tf.tile(tf.expand_dims(tf.range(s), 0), [b, 1])) - # [b, s, emb_dim] - sin_emb = self.pos_emb.FPropWithPosition(theta.pos_emb, position) - # [b, s, n, h] - sin_emb = self.pos_proj.FProp(theta.pos_proj, sin_emb) - if synced_time_step: - # [s, n, h] - sin_emb = tf.squeeze(sin_emb, 0) - # term b an d. - if not p.skip_term_b: - logits += tf.einsum('BNH,SNH->SBN', query + theta.v, sin_emb) - else: - logits += tf.expand_dims(tf.einsum('NH,SNH->SN', theta.v, sin_emb), 1) - else: - # term b an d. - if not p.skip_term_b: - logits += tf.einsum('BNH,BSNH->SBN', query + theta.v, sin_emb) - else: - logits += tf.einsum('NH,BSNH->BSN', theta.v, sin_emb) - return logits - - def ExtendStep(self, - theta, - query_vec, - cached_states, - paddings, - segment_mask, - per_step_padding, - time_step, - use_short_seq_opt=False): - # TODO(jamesqin): support use_short_seq_opt for TransofrmerXL attention. - assert not use_short_seq_opt - return super().ExtendStep(theta, query_vec, cached_states, paddings, - segment_mask, per_step_padding, time_step, - use_short_seq_opt) - - -class MultiHeadedAttentionRPE(MultiHeadedAttention): - """Multiheaded attention with relative positional embedding ... - - See https://arxiv.org/pdf/1803.02155.pdf. - - Notice this is only intended for self attention. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('rel_pos_emb_dim', None, - 'Dimension of relative positional embedding.') - p.Define('rel_pos_radius', None, - 'Relative distance is clipped to [-radius, radius].') - p.Define('skip_value_emb', False, 'If skipping value positional embedding.') - p.Define( - 'use_global_emb', True, - 'If using global relative positional embedding. Only effective if ' - '`rel_pos_emb_tpl` is not None.') - return p - - def __init__(self, params): - """Constructs a MultiHeadedAttentionRPE object.""" - super().__init__(params) - params = self.params - - assert not params.packed_input, 'Packed input not implemented yet.' - - if not params.rel_pos_radius: - raise ValueError('Invalid rel_pos_radius: %s' % params.rel_pos_radius) - - if params.rel_pos_emb_dim is None: - rel_pos_emb_dim = params.hidden_dim - else: - rel_pos_emb_dim = params.rel_pos_emb_dim - - rel_pos_emb_tpl = layers.RelativePositionalEmbeddingLayer.Params().Set( - radius=params.rel_pos_radius, dim=rel_pos_emb_dim) - if rel_pos_emb_dim != params.hidden_dim: - # Projection layer for relative position encoding - dim_per_head = params.hidden_dim // params.num_heads - pos_proj_tpl = params.proj_tpl.Copy().Set( - input_dim=rel_pos_emb_dim, - num_heads=params.num_heads, - dim_per_head=dim_per_head, - use_bias=False) - else: - pos_proj_tpl = None - - self.CreateChild('key_emb', rel_pos_emb_tpl) - # Add projection layer if rel_pos_emb_dim is different from hidden_dim. - if pos_proj_tpl is not None: - self.CreateChild('key_pos_proj', pos_proj_tpl) - if not params.skip_value_emb: - self.CreateChild('value_emb', rel_pos_emb_tpl) - if pos_proj_tpl is not None: - self.CreateChild('value_pos_proj', pos_proj_tpl) - - def _CreateChildrenVariables(self): - with tf.variable_scope( - self.params.name, - reuse=tf.AUTO_REUSE if self.params.use_global_emb else False): - for child in ['key_emb', 'key_pos_proj', 'value_emb', 'value_pos_proj']: - if child in self.children: - self.children[child].InstantiateVariables() - super()._CreateChildrenVariables() - - def _RelativePositionValueEmb(self, theta, key): - """Gets relative positional value embedding. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - key: The attention key, a tensor of shape [batch, seqlen, dim] - - Returns: - Relative positional embedding, a Tensor of shape - [tgt_time=seqlen, src_time=seqlen, num_heads, attenion_dim] - """ - emb_layer = self.value_emb - emb_theta = theta.value_emb - - seqlen = py_utils.GetShape(key)[1] - src_time_indices = tf.tile(tf.expand_dims(tf.range(seqlen), 0), [seqlen, 1]) - tgt_time_indices = tf.tile( - tf.expand_dims(tf.range(seqlen), -1), [1, seqlen]) - - # [tgt_time=T, src_time=T, num_heads x hidden_dim] - pos_emb = emb_layer.FProp(emb_theta, src_time_indices - tgt_time_indices) - - params = self.params - num_heads = self.params.num_heads - tgt_time, src_time, _ = py_utils.GetShape(pos_emb) - - pos_proj_layer = 'value_pos_proj' - if hasattr(self, pos_proj_layer): - return getattr(self, pos_proj_layer).FProp( - getattr(theta, pos_proj_layer), pos_emb) - else: - return tf.reshape( - pos_emb, - [tgt_time, src_time, num_heads, params.hidden_dim // num_heads]) - - def _AttenLogits(self, theta, query, key): - # TODO(jamesqin): optimize it. - b, _, n, h = py_utils.GetShape(key, 4) - t = py_utils.GetShape(query)[1] - - # This layer only supports self attention. - key = py_utils.HasShape(key, [b, t, n, h]) - - # [1, 2T - 1] - pos = tf.expand_dims(tf.range(-(t - 1), t), 0) - # [1, 2T - 1, rel_pos_emb_dim] - abs_emb = self.key_emb.FProp(theta.key_emb, pos) - if hasattr(self, 'key_pos_proj'): - # [1, 2T - 1, N, H] - abs_emb = self.key_pos_proj.FProp(theta.key_pos_proj, abs_emb) - # [2T - 1, N, H] - abs_emb = tf.squeeze(abs_emb, 0) - else: - abs_emb = tf.reshape(abs_emb, [2 * t - 1, n, h]) - - return attention_util.AttenLogitsRPE(query, key, abs_emb) - - def _AttenLogitsOneStep(self, theta, query, key, time_step): - """Attention logits for one single target (query) step. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query: [B, N, H]. - key: [S, B, N, H] or [S, B, N*H/128, 128]. - time_step: Current time step. - - Returns: - A Tensor of shape [S, B, N] - """ - p = self.params - s, b, _, _ = py_utils.GetShape(key, 4) - n = p.num_heads - h = p.hidden_dim // n - - # Transformer_XL relative attention. - if time_step is None: - raise ValueError('`time_step` can not be None when using relative ' - 'position encoding in attention.') - # Gets positional embedding. - # [1, S] - rel_dists = tf.expand_dims(time_step - tf.range(s), 0) - # [1, S, rel_pos_emb_dim] - pos_emb = self.key_emb.FPropDefaultTheta(rel_dists) - if hasattr(self, 'key_pos_proj'): - # [1, S, N, H] - pos_emb = self.key_pos_proj.FProp(theta.key_pos_proj, pos_emb) - # [S, 1, N, H] - pos_emb = tf.transpose(pos_emb, [1, 0, 2, 3]) - else: - pos_emb = tf.reshape(pos_emb, [s, 1, n, h]) - return tf.einsum('BNH,SBNH->SBN', query, - tf.reshape(key, [s, b, n, h]) + pos_emb) - - def _AttenContext(self, theta, probs, value): - # TODO(jamesqin): optimize it. - encoded = tf.einsum('BNij,BjNH->BiNH', probs, value) - - if not self.params.skip_value_emb: - encoded += tf.einsum('BNij,ijNH->BiNH', probs, - self._RelativePositionValueEmb(theta, value)) - return encoded - - def _AttenContextOneStep(self, theta, probs, value, time_step): - s, b, _, _ = py_utils.GetShape(value, 4) - n = self.params.num_heads - h = self.params.hidden_dim // n - - logits = tf.einsum('SBN,SBNH->BNH', probs, tf.reshape(value, [s, b, n, h])) - - if not self.params.skip_value_emb: - # [1, S] - rel_dists = tf.expand_dims(time_step - tf.range(s), 0) - # [1, S, rel_pos_emb_dim] - pos_emb = self.value_emb.FProp(theta.value_emb, rel_dists) - if hasattr(self, 'value_pos_proj'): - # [1, S, N, H] - pos_emb = self.value_pos_proj.FProp(theta.value_pos_proj, pos_emb) - pos_emb = tf.squeeze(pos_emb, 0) - else: - pos_emb = tf.reshape(pos_emb, [s, n, h]) - logits += tf.einsum('SBN,SNH->BNH', probs, pos_emb) - return logits - - def ExtendStep(self, - theta, - query_vec, - cached_states, - paddings, - segment_mask, - per_step_padding, - time_step, - use_short_seq_opt=False): - # TODO(jamesqin): support use_short_seq_opt. - assert not use_short_seq_opt - return super().ExtendStep(theta, query_vec, cached_states, paddings, - segment_mask, per_step_padding, time_step, - use_short_seq_opt) - - @classmethod - def FPropMeta(cls, p, *args): - return NotImplementedError() - - -class LocalSelfAttention(MultiHeadedAttention): - """Dot-product causal self attention using a sliding window. - - We use the following capital letters to denote certain - tensor parameters. - - B = batch size - S=T= length of the key/value (source) and query (target) - D = model dimension - N = number of attention heads - H = dimensions of each attention head - W = block size - L = left context size, including left L-1 positions and self - R = right context size - F = L + R = context size of one position. - C = L + R + W - 1 = context size of a block of W positions. - U = ceiling(T/W). - - The key difference to base class is on calculating logits: - Base class: - 1) Compute the full S x T attention. - 2) Apply a S x T mask to enforce local attention window. - This implementation: - 1) Compute a W x C attention for each of the U blocks. Where the i-th - block has query[W*i:W*(i+1)] and key[W*(i-1)-L-1:W*(i+1)+R]. - 2) Apply a W x C mask for each block. - - Effectively, we reduce both time and space complexities for computing the - sliding window attention from O(S * T) to O(S * C). In practice we observe - reduced HBM usage on TPU but no speed gains. - - Note: Cross attention is not supported. As a result in speech models this - class can only be used for encoder. - - TODO(weihan): add masking based local attention to the base class. - - """ - - @classmethod - def Params(cls): - """Params for LocalSelfAttention.""" - p = super().Params() - p.Define( - 'block_size', None, 'Size of a processing block, if unset, default to ' - 'max(1, left_context-1).') - p.Define( - 'left_context', None, 'Number of left positions to attend ' - '(including current position).') - p.Define('right_context', 0, 'Number of right positions to attend.') - return p - - def __init__(self, params): - """Constructs a LocalSelfAttention object.""" - super().__init__(params) - - p = self.params - assert p.left_context >= 1, 'Left context should be at least one.' - assert not p.packed_input, 'Packed input not implemented yet.' - if p.block_size is None: - p.block_size = max(1, p.left_context - 1) - tf.logging.warning('block_size not set, use default value {}'.format( - p.block_size)) - - assert not p.packed_input, 'Packed input not implemented yet.' - - def _AttenLogits(self, theta, query, key): - return tf.einsum('BUTNH,BUSNH->BNUTS', query, key) - - def AttenProbs(self, - theta, - query, - key, - paddings, - segment_mask, - unused_per_step_padding=None): - """Compute attention probability. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query: [B, T, N, H]. - key: [B, S=T, N, H]. - paddings: [B, T]. - segment_mask: [B, 1, T, S] not used right now. - unused_per_step_padding: Not used. - - Returns: - logits: [B, U, N, W, 2 * W] - """ - p = self.params - key = py_utils.HasRank(key, 4) - b, t, n, h = py_utils.GetShape(key, 4) - paddings = py_utils.HasShape(paddings, [b, t]) - query = py_utils.HasShape(query, [b, t, n, h]) - - # -> [B, U, C, N, H] - key_block_context = attention_util.ExtractBlockContext( - key, - block_size=p.block_size, - left_context=p.left_context, - right_context=p.right_context) - _, u, c, _, _ = py_utils.GetShape(key_block_context) - - # -> [B, U, W, N, H] - query_blocks = attention_util.ConvertToBlocks( - query, block_size=p.block_size) - _, _, w, _, _ = py_utils.GetShape(query_blocks) - - # -> [B, U, C] - paddings_block_context = attention_util.ExtractBlockContext( - paddings, - block_size=p.block_size, - left_context=p.left_context, - right_context=p.right_context, - padding_val=1) - - # -> [B, N, U, W, C] - paddings = tf.tile( - tf.reshape(paddings_block_context, [b, 1, u, 1, c]), [1, n, 1, w, 1]) - - # Make local causal paddings. - # -> [U, W, C] - local_causal_padding = attention_util.MakeCausalPadding( - seq_len=t, - block_size=p.block_size, - left_context=p.left_context, - right_context=p.right_context, - dtype=paddings.dtype) - paddings += local_causal_padding - - # -> [B, N, U, W, C] - logits = self._AttenLogits(theta, query_blocks, key_block_context) - - very_negative_logits = ( - tf.ones_like(logits) * logits.dtype.max * - tf.constant(-0.7, dtype=logits.dtype)) - padded_logits = tf.where(paddings > 0.0, very_negative_logits, logits) - - probs = tf.nn.softmax(padded_logits) - return probs - - def _DotAtten(self, - theta, - query, - key, - value, - paddings, - segment_mask, - per_step_padding=None): - """Main attention function. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query: [B, T, N, H]. - key: [B, S=T, N, H]. - value: [B, S=T, N, H]. - paddings: [B, S=T]. - segment_mask: [B, 1, S=T, S=T]. - per_step_padding: A mask of shape [B, T, S=T] if not None. - - Returns: - encoded: [B, T, N, H]. - atten_probs: [B, N, T, S]. - """ - p = self.params - # Scale the query projection. - if p.enable_per_dim_scale: - query = self.per_dim_scale.FProp(theta.per_dim_scale, query) - else: - query *= (p.hidden_dim // p.num_heads)**-0.5 - t0 = py_utils.GetShape(query)[1] - - # -> [B, N, U, W, C] - probs = self.AttenProbs(theta, query, key, paddings, segment_mask, - per_step_padding) - - # Apply dropout to probs. - probs = self.atten_dropout.FProp(theta.atten_dropout, probs) - - # -> [B, U, C, N, H] - value_block_context = attention_util.ExtractBlockContext( - value, - block_size=p.block_size, - left_context=p.left_context, - right_context=p.right_context) - - # Compute the attention context vector. - # -> [B, U, W, N, H] - encoded = tf.einsum('BNUWC,BUCNH->BUWNH', probs, value_block_context) - b, u, w, n, h = py_utils.GetShape(encoded) - encoded = tf.reshape(encoded, [b, u * w, n, h]) - # Remove the extra time padding introduced by converting to blocks. - encoded = encoded[:, :t0, ...] - return encoded, probs - - def ExtendStep(self, - theta, - query_vec, - cached_states, - paddings, - segment_mask=None, - per_step_padding=None, - time_step=None, - use_short_seq_opt=False): - """Computes the value vector given the query of the current step. - - This function is used by autoregressive decoding. This function knows the - length of full sequence, thus it is different from StreamingExtendStep. - - Note: When the context window size is much smaller than target sequence - length, to make it run more efficent, T below can be just the window size. - Then, time_step should be the relative decode step and not bigger than T. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [B, 1, D]. - cached_states: A `.NestedMap` object containing tensors which are the - results of previous attentions, used for fast decoding. key - [T, B, - N, H]. value - [T, B, N, H]. - paddings: [B, T], or None if there is no padding. - segment_mask: [B, 1, T, S] or None. Not used right now. - per_step_padding: A mask used by decoder self-attention to prevent - information flow from future (causal padding). It has shape [B, 1, T] if - not None. Not used right now. - time_step: A scalar, the current decode step, 0-based. - use_short_seq_opt: A bool, whether using short sequence optimization. Not - supported right now. - - Returns: - encoded: [B, 1, D]. - updated_key_vec: [T, B, N, H]. - updated_value_vec: [T, B, N, H]. - - Raises: - ValueError: If right_context is non-zero. - NotImplementedError: If use_short_seq_opt is true. - """ - p = self.params - if p.right_context != 0: - raise ValueError( - 'Right context must be zero for autoregressive decoding.') - if use_short_seq_opt: - raise NotImplementedError('use_short_seq_opt is not supported yet.') - - # Make local causal paddings, which have shape [B, T]. - t, b, _, _ = py_utils.GetShape(cached_states.key, 4) - if paddings is None: - paddings = tf.zeros([b, t], dtype=query_vec.dtype) - position_diff = tf.tile(tf.range(t)[tf.newaxis, :], [b, 1]) - time_step - valid_atten = tf.math.logical_and(position_diff > -p.left_context, - position_diff <= 0) - local_causal_padding = 1.0 - tf.cast(valid_atten, dtype=query_vec.dtype) - paddings += local_causal_padding - - return super().ExtendStep(theta, query_vec, cached_states, paddings, - segment_mask, per_step_padding, time_step, - use_short_seq_opt) - - def zero_state(self, batch_size=1): - """Returns the initial state given the batch size. - - Args: - batch_size: the batch size. - - Returns: - state: The initial state for streaming inference. - """ - p = self.params - assert p.enable_value_proj, 'Value projection must be enabled.' - assert p.right_context == 0, ('StreamingExtendStep does not support look ' - 'ahead') - key_state = tf.zeros( - shape=[ - p.left_context, batch_size, p.num_heads, p.hidden_dim // p.num_heads - ], - dtype=tf.float32) - value_state = tf.zeros( - shape=[ - p.left_context, batch_size, p.num_heads, p.hidden_dim // p.num_heads - ], - dtype=tf.float32) - state = py_utils.NestedMap(key=key_state, value=value_state) - return state - - def StreamingExtendStep(self, query_vec, state, time_step): - """Computes the value vector given the query of the current step. - - This function doesn't know the length of full sequence, thus it is - different from ExtendStep. - - Args: - query_vec: A query vector of shape [B, 1, D]. - state: A `.NestedMap` object containing tensors {key, value} which are - results of previous attentions. key, value are of shape [T, B, N, H] - where T is the state size of this layer. - time_step: A tensor of shape [1] and type tf.int32. Note, we can not use - scalar tensor here because TfLiteConverter doesn't have good support of - it (b/138865275). - - Returns: - output: Output of the given query vector with shape [B, 1, D]. - state: updated state. - """ - p = self.params - assert p.enable_value_proj, 'Value projection must be enabled.' - assert p.right_context == 0, ('StreamingExtendStep does not support look ' - 'ahead') - query_vec = py_utils.with_dependencies([ - py_utils.assert_shape_match(tf.shape(query_vec), [-1, 1, p.input_dim]) - ], query_vec) - state.key = py_utils.with_dependencies([ - py_utils.assert_shape_match( - tf.shape(state.key), - [p.left_context, -1, p.num_heads, p.hidden_dim // p.num_heads]) - ], state.key) - state.value = py_utils.with_dependencies([ - py_utils.assert_shape_match( - tf.shape(state.value), - [p.left_context, -1, p.num_heads, p.hidden_dim // p.num_heads]) - ], state.value) - - t, b, n, h = py_utils.GetShape(state.key, 4) # t: context window size - - # Computes key, value projection and updates state. - new_key_proj = self.key.FProp(self.theta.key, query_vec) # [B, 1, N, H] - new_key_proj = tf.reshape(new_key_proj, [1, b, n, h]) - new_value_proj = self.key.FProp(self.theta.value, query_vec) # [B, 1, N, H] - new_value_proj = tf.reshape(new_value_proj, [1, b, n, h]) - state.key = tf.concat([state.key[1:, :, :, :], new_key_proj], axis=0) - state.value = tf.concat([state.value[1:, :, :, :], new_value_proj], axis=0) - - # For a time step less than the context window size, the time dimension of - # input of logits computation is equal to the time step (not a full context - # window). - t = tf.math.minimum(time_step[0] + 1, t) - key_input = state.key[-t:, :, :, :] - value_input = state.value[-t:, :, :, :] - - # Computes query projection. - query_proj = self.query.FProp(self.theta.query, query_vec) # [B, 1, N, H] - - # Scales the query projection. - if p.enable_per_dim_scale: - query_proj = self.per_dim_scale.FProp(self.theta.per_dim_scale, - query_proj) - else: - query_proj *= h**-0.5 - query_proj = tf.reshape(query_proj, [b, n, h]) - - # Computes attention outputs. - # TODO(wildstone): Replaces the einsum ops used below with mat mul to get - # rid of TfLite Flex ops. - logits = self._AttenLogitsOneStep(self.theta, query_proj, key_input, - t - 1) # [T, B, N] - logits = tf.reshape(logits, [t, -1]) - posteriors = tf.nn.softmax(logits, axis=0) - posteriors = tf.reshape(posteriors, [t, b, n]) - output = tf.einsum('TBN, TBNH->BNH', posteriors, value_input) - - # Post projection. - output = tf.expand_dims(output, 1) - output = self.post.FProp(self.theta.post, output) - return output, state - - @classmethod - def FPropMeta(cls, p, *args): - raise NotImplementedError() - - -class LocalSelfAttentionXL(LocalSelfAttention): - """Local causal version of transformer-xl self attention.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('rel_pos_emb_dim', None, - 'Dimension of relative positional embedding.') - p.Define('skip_term_b', False, - 'If True, skip term_b in the paper section 3.3.') - return p - - def __init__(self, params): - """Constructs a LocalSelfAttentionXL object.""" - super().__init__(params) - params = self.params - if params.rel_pos_emb_dim is None or params.rel_pos_emb_dim <= 0: - raise ValueError('Invalid rel_pos_emb_dim: %s' % params.rel_pos_emb_dim) - - emb_params = layers.PositionalEmbeddingLayer.Params().Set( - embedding_dim=params.rel_pos_emb_dim) - self.CreateChild('pos_emb', emb_params) - - # Projection layer for relative position encoding - dim_per_head = params.hidden_dim // params.num_heads - pos_proj_tpl = params.proj_tpl.Copy().Set( - input_dim=params.rel_pos_emb_dim, - num_heads=params.num_heads, - dim_per_head=dim_per_head, - use_bias=False) - self.CreateChild('pos_proj', pos_proj_tpl) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - params = self.params - - dim_per_head = params.hidden_dim // params.num_heads - u_pc = py_utils.WeightParams( - shape=[params.num_heads, dim_per_head], - init=py_utils.WeightInit.Constant(0.0), - dtype=params.dtype, - collections=[self.__class__.__name__ + '_vars']) - v_pc = py_utils.WeightParams( - shape=[params.num_heads, dim_per_head], - init=py_utils.WeightInit.Constant(0.0), - dtype=params.dtype, - collections=[self.__class__.__name__ + '_vars']) - - self.CreateVariable('u', u_pc) - self.CreateVariable('v', v_pc) - - def _AttenLogits(self, theta, query, key): - b, u, w, _, _ = py_utils.GetShape(query) - _, _, c, _, _ = py_utils.GetShape(key) - n = self.params.num_heads - l = self.params.left_context - r = self.params.right_context - f = l + r - # term a and c - term_ac = tf.einsum('BUTNH,BUSNH->BNUTS', query + theta.u, key) - - # term b and d - # [1, F] - pos = tf.expand_dims(tf.range(l - 1, -r - 1, -1), 0) - sin_emb = self.pos_emb.FPropWithPosition(theta.pos_emb, pos) - # [1, F, N, H] - sin_emb = self.pos_proj.FProp(theta.pos_proj, sin_emb) - # [F, N, H] - sin_emb = tf.squeeze(sin_emb, 0) - - p = self.params - if not p.skip_term_b: - # [B, N, U, W, F] - term_bd = tf.einsum('BUWNH,FNH->BNUWF', query + theta.v, sin_emb) - - # Perform relative shift in order to get [B, N, U, W, C] - # Pads the input to [B, N, U, C, C+1] - term_bd = tf.pad(term_bd, - ((0, 0), (0, 0), (0, 0), (0, c - w), (0, c + 1 - f))) - - # Reshapes to [B, N, U, C+1, C]. Note the output last dim is 1-smaller - # than the input, which "pushses" one element off to the next row for each - # row. The accumulated effect is row_i is right-shifted i steps (i>=0). - term_bd = tf.reshape(term_bd, [b, n, u, c + 1, c]) - - # Keeps useful slices. [B, N, U, W, C] - term_bd = tf.slice(term_bd, [0, 0, 0, 0, 0], [-1, -1, -1, w, -1]) - else: - # [N, F] - term_d = tf.einsum('NH,FNH->NF', theta.v, sin_emb) - # [N, W, F] - term_d = tf.tile(tf.expand_dims(term_d, 1), [1, w, 1]) - # [N, C, C+1] - term_d = tf.pad(term_d, ((0, 0), (0, c - w), (0, c + 1 - f))) - # [N, C+1, C] - term_d = tf.reshape(term_d, [n, c + 1, c]) - # Keeps useful slices. [N, W, C] - term_d = tf.slice(term_d, [0, 0, 0], [-1, w, -1]) - term_bd = tf.reshape(term_d, [1, n, 1, w, c]) - return term_ac + term_bd - - def _AttenLogitsOneStep(self, theta, query, key, time_step): - """Attention logits for one single target (query) step. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query: [B, N, H]. - key: [S, B, N, H] or [S, B, N*H/128, 128]. - time_step: Current time step. - - Returns: - A Tensor of shape [S, B, N] - """ - p = self.params - s, b, _, _ = py_utils.GetShape(key, 4) - n = p.num_heads - h = p.hidden_dim // n - - # Transformer_XL relative attention. - if time_step is None: - raise ValueError('`time_step` can not be None when using relative ' - 'position encoding in attention.') - # term a and c. - logits = tf.einsum('BNH,SBNH->SBN', query + theta.u, - tf.reshape(key, [s, b, n, h])) - position = tf.expand_dims(time_step - tf.range(s), 0) - # [1, s, emb_dim] - sin_emb = self.pos_emb.FPropWithPosition(theta.pos_emb, position) - sin_emb = self.pos_proj.FProp(theta.pos_proj, sin_emb) - # [s, n, h] - sin_emb = tf.squeeze(sin_emb, 0) - - # term b an d. - if not p.skip_term_b: - logits += tf.einsum('BNH,SNH->SBN', query + theta.v, sin_emb) - else: - logits += tf.expand_dims(tf.einsum('NH,SNH->SN', theta.v, sin_emb), 1) - return logits - - -class RoutingAttention(MultiHeadedAttention): - """"Implements a sparse attention based on k-means clustering. - - This is used in the routing transformer https://arxiv.org/pdf/2003.05997. - - This verison of multi-headed attention differs from the full attention - in that it uses k-means clusterting to cluster the queries and keys first, - and each query only attend to a subset of keys that are close to the centroid - closest to that query. As Euclidean distance is used to determine closeness, - we layer normalize queries and keys first so that closeness lead to a larger - dot product. - - TODO(zhouwk) This class is missing the following features: - * propagate clustering loss; - * supporting packed inputs; - * support attention dropout; - * support relative position encoding; - * support using local attention on some heads. - - We use the following capital letters to denote shape parameters: - B = batch size - S = length of the source sequence - T = length of the target sequence - N = number of attention heads - H = dimensions of each attention head - D = model dimension - - K = number of clusters - W = attention window - """ - - @classmethod - def Params(cls): - """Params.""" - p = super().Params() - p.Define( - 'num_clusters', 0, 'Number of clusters, typically around the square' - ' root of the sequence length.') - p.Define('attention_window', 0, 'The number of keys each query attends to.') - p.Define('clustering', attention_util.KMeansClusteringForAtten.Params(), - 'The params for a clustering layer.') - p.Define( - 'causal_masking', False, - 'Whether causal masking is enabled. When set, a query at position idx ' - 'is only allowed to attend to keys/values at positions <= idx.') - p.Define( - 'fast_path', True, - 'Whether to use a more efficient implementation. The fast path is ' - 'signanificantly faster by grouping queries when determining which ' - 'values to attend to (which might leave out some queries or duplicate ' - 'others); fast_path=False computes this per each query.') - p.Define( - 'query_group_size_factor', 1.2, - 'Only used when p.fast_path=True. When grouping queries, we make the ' - 'group size larger by this multiplier to not leave out any queries due ' - 'to potential cluster imbalance.') - return p - - def __init__(self, params): - """Constructs an instance of RoutingAttention.""" - super().__init__(params) - p = self.params - assert p.num_clusters - assert p.attention_window - assert not p.packed_input - - clustering_p = p.clustering - clustering_p.num_clusters = p.num_clusters - clustering_p.num_heads = p.num_heads - clustering_p.dim_per_head = p.hidden_dim // p.num_heads - # We normalize manually prior so that we can reuse the same normalized - # query/key to compute attention probs later. - clustering_p.apply_layer_norm = False - self.CreateChild('clustering', clustering_p) - - def _DotAtten(self, - theta, - query, - key, - value, - paddings, - segment_mask=None, - per_step_padding=None, - query_paddings=None): - """Computes the attention. - - Each query selects 'p.attention_window' number of keys to attend to. First - we find the closest centroid to that query, and we only allow that query to - attend to the 'p.attention_window' closest keys to that centroid. - - In order to use K-means, this implementation applies layer normalization - to both the queries and the keys, and uses the normalized results to compute - attention weights. - - When 'p.attention_window' is the source length, this should evalue to the - full attention (using layer normalized queries and keys). - - The caller should pass in the paddings for both 'key' and 'query' because - during training, when we update the clustering we need to know the paddings - for both. (For the inference path only 'key_paddings' is useful.) - - Args: - theta: A `.NestedMap` of the values of this layer's weights. - query: [B, T, N, H]. - key: [B, S, N, H]. - value: [B, S, N, H]. - paddings: [B, S], paddings for key. - segment_mask: must be None. - per_step_padding: must be None. Please use p.causal_masking. - query_paddings: [B, T], or None. - - Returns: - encoded: [B, T, N, H]. - atten_probs: [B, T, N, S]. - """ - p = self.params - if segment_mask is not None or per_step_padding is not None: - raise ValueError('Requires segment_mask=None and per_step_padding=None.') - key_paddings = paddings - b, t = py_utils.GetShape(query, 2) - if query_paddings is None: - query_paddings = tf.zeros([b, t], dtype=key_paddings.dtype) - - is_self_attention = (query is key) - # Whether to update the centroids. Only do this during training. - update = not self.do_eval - - query = attention_util.KMeansClusteringForAtten.LayerNorm(query) - # [B, T, N, K] - q_dists, _ = self.clustering.FProp( - theta.clustering, query, query_paddings, update=update) - - if is_self_attention: - key = query - k_dists = q_dists - else: - key = attention_util.KMeansClusteringForAtten.LayerNorm(key) - # [B, S, N, K] - k_dists, _ = self.clustering.FProp( - theta.clustering, key, key_paddings, update=update) - if p.fast_path: - return self._DotAttenFastPath(theta, query, key, value, q_dists, k_dists, - query_paddings, key_paddings) - else: - return self._DotAttenSlowPath(theta, query, key, value, q_dists, k_dists, - query_paddings, key_paddings) - - def InitStates(self, theta, target_batch_size, target_max_length): - """Initialize 'states' with .key, .value, and .key_dists.""" - p = self.params - states = super().InitStates(theta, target_batch_size, target_max_length) - states.key_dists = inplace_ops.empty( - shape=(target_max_length, target_batch_size, p.num_heads, - p.num_clusters), - dtype=py_utils.FPropDtype(p), - init=True) - return states - - def ExtendStep(self, - theta, - query_vec, - cached_states, - paddings, - time_step, - segment_mask=None, - per_step_padding=None, - use_short_seq_opt=False): - """Computes the value vector given the query of the current step. - - This function is used by autoregressive decoding. Used for self-attention - (hence S=T) with p.causal_masking is True. - - We compute the key/value/key_dists at `time_step` and cache the updated - full length results in `cache_states` to reduce duplicate computation. - - p.fast_path is ignored (as if p.fast_path=False) as at each step we only - compute for query of length 1. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [B, 1, D]. - cached_states: A `.NestedMap` object containing tensors which are the - results of previous attentions, used for fast decoding. It contains .key - and .value with shape [T, B, N, H], and .key_dists with shape [T, B, N, - K]. Note that they are all time-major. - paddings: [B, T], or None if there is no padding. - time_step: Scalar, the current decode step, 0-based. - segment_mask: must be None. - per_step_padding: must be None. We obey causal masking. - use_short_seq_opt: must be False. - - Returns: - encoded: [B, 1, D]. - updated_states: `.NestedMap` with .key, .value, .key_dists. - - Raises: - ValueError: If value projection is disabled. - """ - p = self.params - if not p.enable_value_proj: - raise ValueError('Value projection must be enabled: ' - 'set p.enable_value_proj = True.') - if not p.causal_masking: - raise ValueError('p.causal_masking must be true.') - if segment_mask is not None or per_step_padding is not None: - raise ValueError('Requires segment_mask=None and per_step_padding=None.') - if use_short_seq_opt: - raise ValueError('Requires use_short_seq_opt=False.') - if time_step is None: - raise ValueError('Requires valid time_step, not None.') - - t, b, n, h = py_utils.GetShape(cached_states.key, 4) - - # Project inputs to key, value and query. Each has shape [B, 1, N, H]. - key_proj = self.key.FProp(theta.key, query_vec) - value_proj = self.value.FProp(theta.value, query_vec) - query_proj = self.query.FProp(theta.query, query_vec) - - query_proj = attention_util.KMeansClusteringForAtten.LayerNorm(query_proj) - key_proj = attention_util.KMeansClusteringForAtten.LayerNorm(key_proj) - # [B, 1, N, K] - k_dists, _ = self.clustering.FProp(theta.clustering, key_proj) - - # The updated_key and extended_value have shape [T, B, N, H]. - updated_key = inplace_ops.alias_inplace_update( - cached_states.key, time_step, tf.reshape(key_proj, [b, n, h])) - updated_value = inplace_ops.alias_inplace_update( - cached_states.value, time_step, tf.reshape(value_proj, [b, n, h])) - # Shape [T, B, N, K] - updated_key_dists = inplace_ops.alias_inplace_update( - cached_states.key_dists, time_step, - tf.reshape(k_dists, [b, n, p.num_clusters])) - updated_states = py_utils.NestedMap( - key=updated_key, value=updated_value, key_dists=updated_key_dists) - - if paddings is None: - paddings = tf.zeros([b, t], dtype=query_vec.dtype) - # Apply causal padding. Shape [B, T] - paddings = tf.where( - tf.greater( - tf.tile(tf.range(t)[None, :], [b, 1]), tf.fill([b, t], time_step)), - tf.ones_like(paddings), paddings) - query_paddings = tf.zeros([b, 1], dtype=paddings.dtype) - - encoded = self._DotAttenOneStep( - theta, - query_proj, - updated_states, - query_paddings=query_paddings, - key_paddings=paddings, - time_step=time_step) - # Post projection. - encoded = self.post.FProp(theta.post, encoded) - return encoded, updated_states - - def _DotAttenOneStep(self, theta, query, states, query_paddings, key_paddings, - time_step): - """Dot attention function for queries with 1 time step. - - Called from ExtendStep(). Used for self-attention with p.causal_masking - is True. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query: [B, 1, N, H], already normalized. - states: .key and .value with shape [T, B, N, H], .key_dists with shape - [T, B, N, K]. .key is normalized. - query_paddings: [B, 1]. - key_paddings: [B, T]. - time_step: Scalar, the current decode step, 0-based. - - Returns: - encoded: [B, 1, N, H]. - """ - p = self.params - # [B, 1, N, K] - q_dists, _ = self.clustering.FProp(theta.clustering, query) - # [B, T, N, K] - k_dists = tf.transpose(states.key_dists, [1, 0, 2, 3]) - - very_large_dists = tf.ones_like(k_dists) * tf.constant( - 0.1, dtype=k_dists.dtype) * k_dists.dtype.max - paddings_tiled = tf.tile(key_paddings[:, :, None, None], - [1, 1, p.num_heads, p.num_clusters]) - k_dists = tf.where(paddings_tiled > 0.0, very_large_dists, k_dists) - - key = tf.transpose(states.key, [1, 0, 2, 3]) - value = tf.transpose(states.value, [1, 0, 2, 3]) - encoded, _ = self._DotAttenSlowPath( - theta, - query, - key, - value, - q_dists, - k_dists, - query_paddings, - key_paddings, - query_relative_position_shift=time_step) - return encoded - - def _DotAttenSlowPath(self, - theta, - query, - key, - value, - q_dists, - k_dists, - query_paddings, - key_paddings, - query_relative_position_shift=0): - """Computes the attention via the slow path. - - This implementation selects, on a per query basis, p.attention_window - number of keys/values to attend to. - - Args: - theta: A `.NestedMap` of the values of this layer's weights. - query: [B, T, N, H], already normalized. - key: [B, S, N, H], already normalized. - value: [B, S, N, H]. - q_dists: [B, T, N, K]. - k_dists: [B, S, N, K]. - query_paddings: [B, T]. - key_paddings: [B, S]. - query_relative_position_shift: scalar. The position (relative to key[0]) - of query[0]. This impacts relative position encoding (not yet - implemented) and causal masking. - - Returns: - encoded: [B, T, N, H]. - atten_probs: [B, T, N, S]. - """ - p = self.params - - # [B, N, K, S] - # If key is padded in a position, 'k_dists' is inf which ensures - # that we consider all non-padded keys even if some padded keys - # might appear closer. - k_dists = tf.transpose(k_dists, [0, 2, 3, 1]) - - # [B, N, K, W], for each centroid, the indices of closest key vecs. - # It's okay if W is so larger such that a padded index is included, - # because below in attention_util.ComputeSparseAttention() correctly - # handles 'paddings'. - _, closest_indices = tf.math.top_k(-k_dists, p.attention_window) - # [B, T, N, K], one hot encoded closest centroid for each query vec. - nearest_one_hot = tf.one_hot( - tf.math.argmin(q_dists, axis=-1), - p.num_clusters, - dtype=closest_indices.dtype) - - # For each query vec, we allow it to attend to those keys that are the - # W closest to its centroid, where W is the attention window. - sparsity_indices = tf.einsum('BTNK, BNKW -> BTNW', nearest_one_hot, - closest_indices) - if p.causal_masking: - batch_size, q_length, num_heads = py_utils.GetShape(query, 3) - query_positions = tf.range(q_length) + query_relative_position_shift - # [B, T, N, W] where the T dimension is range(T) - query_positions = tf.tile(query_positions[None, :, None, None], - [batch_size, 1, num_heads, p.attention_window]) - masked_indices = -tf.ones_like(sparsity_indices) - # Replace key positions in the future with -1 to indicate masking. - # - # Note that this is done after selecting top_k from 'k_dists', so for - # example if all the closest keys are in the future, we waste - # p.attention_window on padded keys when in theory we could have attended - # to further away keys that are not in the future (in order to achieve - # that we need to pick top_k from 'k_dists' differently for each query). - sparsity_indices = tf.where( - tf.math.greater(sparsity_indices, query_positions), masked_indices, - sparsity_indices) - - return attention_util.ComputeSparseAttention(query, key, value, - sparsity_indices, key_paddings) - - def _DotAttenFastPath(self, theta, query, key, value, q_dists, k_dists, - query_paddings, key_paddings): - """Computes the attention via the fast path. - - This implementation compute groups of queries, and for each group, - selects a set of p.attention_window number of keys/values that each - query in that group all attend to. - - There is no guarantee a query uniquely belong to a single group, although - via clustering this should likely be the case. When a query belong to - multiple groups, the attention is averaged post softmax; when a query - does not belong to any group, the attention result is zero. - - Args: - theta: A `.NestedMap` of the values of this layer's weights. - query: [B, T, N, H], already normalized. - key: [B, S, N, H], already normalized. - value: [B, S, N, H]. - q_dists: [B, T, N, K]. - k_dists: [B, S, N, K]. - query_paddings: [B, T]. - key_paddings: [B, S]. - - Returns: - encoded: [B, T, N, H]. - atten_probs: [B, T, N, S]. Note, N * S * T space complexity here. - """ - p = self.params - # [B, N, K, S] - # If key is padded in a position, 'k_dists' is inf which ensures - # that we consider all non-padded keys even if some padded keys - # might appear closer. - k_dists = tf.transpose(k_dists, [0, 2, 3, 1]) - - # [B, N, K, W], for each centroid, the indices of closest key vecs. - # closest_k may include padded positions. - _, closest_k = tf.math.top_k(-k_dists, p.attention_window) - - q_length = py_utils.GetShape(query, 2)[1] - k_length = py_utils.GetShape(key, 2)[1] - assert isinstance(q_length, int) - assert isinstance(k_length, int) - q_cluster_size = int(p.query_group_size_factor * q_length / p.num_clusters) - # Of shape [B, N, K, T] - q_dists = tf.transpose(q_dists, [0, 2, 3, 1]) - # closest_q of shape [B, N, K, V], where V = q_cluster_size - # closest_q may include padded positions. - _, closest_q = tf.math.top_k(-q_dists, q_cluster_size) - - def gather(v, indx): - """Gathers values from v. - - Args: - v: A tensor of shape [B, T, N, D] - indx: A tensor of shape [B, N, K, W] - - Returns: - A value of shape [B, N, K, W, D] - """ - # pylint: disable=invalid-name - B, _, N, _ = py_utils.GetShape(v, 4) - _, _, K, W = py_utils.GetShape(indx, 4) - # pylint: enable=invalid-name - batch_idx = tf.range(B)[:, None, None, None, None] - batch_idx = tf.tile(batch_idx, [1, N, K, W, 1]) - seq_idx = indx[:, :, :, :, None] - head_idx = tf.range(N)[None, :, None, None, None] - head_idx = tf.tile(head_idx, [B, 1, K, W, 1]) - gather_idx = tf.concat([batch_idx, seq_idx, head_idx], 4) - return tf.gather_nd(v, gather_idx) - - # c_ shorts for clustered. - _, _, num_heads, dim_per_head = py_utils.GetShape(query, 4) - # of shape [B, N, K, V, D] - c_query = gather(query, closest_q) - # of shape [B, N, K, W, D] - c_key, c_value = tf.split( - gather(tf.concat([key, value], -1), closest_k), 2, -1) - # of shape [B, N, K, W, 1] - c_key_paddings = gather( - # [B, T, N, 1] - tf.tile(key_paddings[:, :, None, None], [1, 1, num_heads, 1]), - closest_k) - # of shape [B, N, K, V, W] - is_key_padded = tf.tile( - tf.transpose(c_key_paddings, [0, 1, 2, 4, 3]), - [1, 1, 1, q_cluster_size, 1]) > 0.5 - if p.causal_masking: - # both position matrices of shape [B, N, K, V, W] - c_query_positions = tf.tile(closest_q[:, :, :, :, None], - [1, 1, 1, 1, p.attention_window]) - c_key_positions = tf.tile(closest_k[:, :, :, None, :], - [1, 1, 1, q_cluster_size, 1]) - # We pad the logit for future key positions relative to each query - is_key_padded = tf.math.logical_or( - is_key_padded, tf.math.greater(c_key_positions, c_query_positions)) - - logits = tf.einsum('BNKVD,BNKWD->BNKVW', c_query, c_key) - logits *= tf.math.rsqrt(tf.cast(dim_per_head, p.dtype)) - - very_negative_logits = ( - tf.ones_like(logits) * logits.dtype.max * - tf.constant(-0.7, dtype=logits.dtype)) - padded_logits = tf.where(is_key_padded, very_negative_logits, logits) - - c_atten_probs = tf.nn.softmax(padded_logits) - c_outputs = tf.einsum('BNKWD,BNKVW->BNKVD', c_value, c_atten_probs) - - def scatter(v, indx, seq_len): - """Scatters v according to indx. - - Args: - v: A tensor of shape [B, N, K, V, D]. - indx: A tensor of shape [B, N, K, V]. - seq_len: sequence length of the output. - - Returns: - output: A tensor of shape [B, T, N, D], where T = seq_len. - """ - # Need to scatter outputs back to the original shape. - # pylint: disable=invalid-name - B, N, K, V, D = py_utils.GetShape(v, 5) - # pylint: enable=invalid-name - # [B, N, K, V, 1] - batch_idx = tf.tile( - tf.range(B)[:, None, None, None, None], [1, N, K, V, 1]) - # [B, N, K, V, 1] - seq_idx = indx[:, :, :, :, None] - # [B, N, K, V, 1] - head_idx = tf.tile( - tf.range(N)[None, :, None, None, None], [B, 1, K, V, 1]) - scatter_idx = tf.concat([batch_idx, seq_idx, head_idx], 4) - scattered = tf.scatter_nd( - scatter_idx, tf.concat([v, tf.ones_like(v[:, :, :, :, :1])], -1), - [B, seq_len, N, D + 1]) - # We need to normaliz as one query vector may appear in multiple clusters. - scattered, den = tf.split(scattered, [v.shape.as_list()[-1], 1], -1) - # den = tf.squeeze(den, -1) - out = scattered / tf.maximum(tf.constant(1.0, dtype=den.dtype), - den) # [:, :, :, None]) - return out - - def scatter_atten_prob(c_atten_probs, closest_k, closest_q, k_length, - q_length): - """Scatters c_atten_probs. - - Args: - c_atten_probs: A tensor of shape [B, N, K, V, W]. - closest_k: A tensor of shape [B, N, K, W]. - closest_q: A tensor of shape [B, N, K, V]. - k_length: Length of the key vectors. - q_length: Length of the query vectors. - - Returns: - output: A tensor of shape [B, q_length, N, k_length]. - """ - # Need to scatter outputs back to the original shape. - # pylint: disable=invalid-name - B, N, K, V, W = py_utils.GetShape(c_atten_probs, 5) - # pylint: enable=invalid-name - # [B, N, K, V, W, 1] - batch_idx = tf.tile( - tf.range(B)[:, None, None, None, None, None], [1, N, K, V, W, 1]) - # [B, N, K, V, W, 1] - k_idx = tf.tile(closest_k[:, :, :, None, :, None], [1, 1, 1, V, 1, 1]) - q_idx = tf.tile(closest_q[:, :, :, :, None, None], [1, 1, 1, 1, W, 1]) - head_idx = tf.tile( - tf.range(N)[None, :, None, None, None, None], [B, 1, K, V, W, 1]) - scatter_idx = tf.concat([batch_idx, q_idx, head_idx, k_idx], 5) - scattered_prob = tf.scatter_nd(scatter_idx, c_atten_probs, - [B, q_length, N, k_length]) - - # We need to normalize the attention prob as one query vector may appear - # in multiple clusters. - # [B, N, K, V, 3] - times_idx = tf.concat([batch_idx, q_idx, head_idx], 5)[:, :, :, :, 0, :] - # [B, q_length, N] - times = tf.scatter_nd( - times_idx, tf.cast(tf.ones_like(closest_q), scattered_prob.dtype), - [B, q_length, N]) - times = tf.maximum(1.0, times[:, :, :, None]) - out = scattered_prob / times - return out - - out = scatter(c_outputs, closest_q, q_length) - out_prob = scatter_atten_prob(c_atten_probs, closest_k, closest_q, k_length, - q_length) - return out, out_prob - - -class MultiSourceAttention(base_layer.BaseLayer): - """Batch major attention with multiple source sub-attentions. - - It attends to multiple sources and uses one query as input to generates a - combined attention context. The dimension of the combined context vector is a - sum of all source context vectors. Each source attention has its separate - params and is associated with a source key. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('source_atten_tpls', None, - 'A list of (source_key, attention_param) pairs.') - p.Define('input_dim', 0, 'Default key dimension.') - p.Define('hidden_dim', 0, 'Default hidden dimension.') - p.Define( - 'primary_source_key', 'source_0', 'Key for the primary source ' - 'whose attention probabilities will be used as an output.') - p.Define('atten_merger_tpl', None, - 'Params to specify how to merge source attention vectors.') - return p - - def __init__(self, params): - """Constructs an MultiSourceAttention object.""" - super().__init__(params) - p = self.params - assert p.primary_source_key in [ - x for x, _ in p.source_atten_tpls - ], ('Source attention must have the primary source key.') - for source_key, atten_p in p.source_atten_tpls: - child_p = atten_p.Copy() - if child_p.hidden_dim <= 0: - child_p.hidden_dim = p.hidden_dim - if child_p.input_dim <= 0: - child_p.input_dim = p.input_dim - self.CreateChild('atten_%s' % source_key, child_p) - - # Initialize source context vector merging layer. - merger_p = p.atten_merger_tpl.Copy() - merger_p.name = 'atten_merger' - merger_p.source_dim = p.input_dim - merger_p.query_dim = p.input_dim - self.CreateChild('atten_merger', merger_p) - - def FProp(self, - theta, - query_vec, - key_vec, - value_vec, - paddings, - segment_mask=None, - per_step_padding=None): - p = self.params - with tf.name_scope(self.params.name): - result_map = py_utils.NestedMap() - for source_key, _ in p.source_atten_tpls: - result_map[source_key] = ( - self.children['atten_%s' % source_key].FProp( - theta.get('atten_%s' % source_key), query_vec, - key_vec[source_key], value_vec[source_key], - paddings[source_key], - segment_mask[source_key] if segment_mask else None, - per_step_padding)) - return self._CombineContext(theta, result_map, query_vec) - - def _CombineContext(self, theta, enc_map, query_vec): - encs = enc_map.Flatten() - combined_enc = ( - self.atten_merger.FProp(theta.atten_merger, [enc for enc, _ in encs], - query_vec)) - # Return atten_probs of the primary source. - return combined_enc, enc_map[self.params.primary_source_key][1] - - def AttenProbs(self, - theta, - query, - key, - paddings, - segment_mask, - per_step_padding=None): - primary_source_key = self.params.primary_source_key - child_name = 'atten_%s' % primary_source_key - return self.children[child_name].AttenProbs( - theta.get(child_name), query, key[primary_source_key], - paddings[primary_source_key], - segment_mask[primary_source_key] if segment_mask else None, - per_step_padding) - - -class TransformerAttentionLayer(base_layer.BaseLayer): - """Multiheaded attention sub-layer in Transformer layer. - - Input is first normalized using Layer Normalization. Output of layer - normalization is processed using multi-headed attention. And finally, the - output of the attention layer is combined with the residual connection. - - This layer will be used in the following two scenarios: - - 1. Multi-Headed Self-Attention, where attention keys, values (source_vecs) and - queries come from the same previous layer output. - 2. Masked Multi-Headed Self-Attention, where attention keys, values and - queries all come from the same previous layer output, but rightward - activations are masked to prevent information flow from future. This is the - use case for Transformer decoder self-attention layers. Can be activated by - setting is_masked flag of this layer. - 3. Multi-Headed Cross-Attention, where attention keys and values - (source_vecs) are coming from a different source (output of the encoder), - and queries coming from the previous layer outputs (decoder). - - We use the same capital letters to denote certain tensor parameters as - MultiHeadedAttention class. - - B = batch size - S = length of the key/value (source) - T = length of the query (target) - D = model dimension - N = number of attention heads - H = dimensions of each attention head. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', 0, 'Dimension of the transformer block input.') - p.Define('hidden_dim', 0, 'Dimension of the attention hidden dim.') - p.Define('num_heads', 8, 'Number of attention heads.') - p.Define( - 'is_masked', False, - 'If set, uses causal non local multiheaded attention.' - 'This option is not valid when atten_tpl is LocalSelfAttention ' - 'or its subclass(es).') - p.Define( - 'atten_dropout_prob', 0.0, - 'Probability at which we apply dropout to the attention probs. ' - 'This practically drops memory values at random positions.') - p.Define( - 'residual_dropout_prob', 0.0, - 'Probability at which we apply dropout to the residual layers, ' - 'such that, residual(x, y) = (x + dropout(y)).') - p.Define('add_unnormalized_input', True, - 'If set, uses unnormalized input in the residual add.') - p.Define('add_skip_connection', True, - 'If True, add input (or normalized input) to the output.') - p.Define('ln_tpl', layers.LayerNorm.Params(), - 'Layer norm default params. No layernorm if set to None.') - p.Define('atten_tpl', - MultiHeadedAttention.Params().Set(), - 'Multi-Headed Dot-Product Attention default params') - p.Define( - 'dropout_tpl', layers.DropoutLayer.Params(), - 'Residual dropout params template. keep_prop will be reset to ' - '(1.0 - residual_dropout_prob).') - return p - - @classmethod - def CommonParams(cls, - input_dim, - num_heads, - is_masked=False, - use_relative_atten=False, - relative_pos_emb_dim=None, - local_context=None, - left_context=None, - right_context=None, - dropout_prob=0.): - # pylint: disable=g-doc-args - """Returns a hyperparam for the most representative cases. - - CommonParams is not expected to be extended to an omnipotent/generic builder - method. Specific use cases should take the return value of it and apply - further customization. It should be kept lean and only extended cautiously - for very common cases. - """ - # pylint: enable=g-doc-args - if not use_relative_atten: - assert not relative_pos_emb_dim - else: - relative_pos_emb_dim = relative_pos_emb_dim or input_dim - - if local_context: - assert not left_context and not right_context, ( - 'local_context and (left_context, right_context) can not be set ' - 'at the same time.') - left_context = local_context + 1 # include 'self' position. - right_context = local_context - - p = cls.Params().Set( - input_dim=input_dim, - num_heads=num_heads, - is_masked=is_masked, - atten_dropout_prob=dropout_prob, - residual_dropout_prob=dropout_prob) - - is_local = left_context or right_context - if is_local: - atten_cls = ( - LocalSelfAttentionXL if use_relative_atten else LocalSelfAttention) - else: - atten_cls = ( - MultiHeadedAttentionXL - if use_relative_atten else MultiHeadedAttention) - p.atten_tpl = atten_cls.Params() - if use_relative_atten: - p.atten_tpl.rel_pos_emb_dim = relative_pos_emb_dim - if is_local: - p.atten_tpl.Set(left_context=left_context, right_context=right_context) - return p - - def _InitAttentionParams(self, atten_tpl): - """Returns an initialized transformer attention parameters.""" - p = self.params - params = atten_tpl.Copy() - params.name = 'multihead_atten' - params.input_dim = p.input_dim - params.hidden_dim = p.hidden_dim - params.num_heads = p.num_heads - params.atten_dropout_prob = p.atten_dropout_prob - return params - - def __init__(self, params): - super().__init__(params) - p = self.params - - if not p.hidden_dim: - p.hidden_dim = p.input_dim - - # Initialize attention. - params = self._InitAttentionParams(p.atten_tpl) - if p.is_masked and issubclass(params.cls, LocalSelfAttention): - tf.logging.warn('\'is_masked\' is not effective when used with ' - 'LocalSelfAttention and its subclass(es).') - self.CreateChild('atten', params) - - # Initialize attention layer normalization. - if p.ln_tpl: - params = p.ln_tpl.Copy() - params.name = 'atten_ln' - params.input_dim = p.input_dim - self.CreateChild('layer_norm', params) - - # Initialize residual dropout. - dropout_tpl = p.dropout_tpl.Copy() - dropout_tpl.keep_prob = (1.0 - p.residual_dropout_prob) - self.CreateChild('residual_dropout', dropout_tpl) - - def FProp(self, - theta, - query_vec, - source_vecs, - paddings, - per_step_padding_override=None, - segment_mask=None): - """Compute the result of Transformer attention layer. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [B, T, D]. - source_vecs: [B, S, D] (cross_attention) or None (self-attention). - paddings: [B, S]. - per_step_padding_override: [B, T, T] for self attention or [B, T, S] for - cross attention. - segment_mask: [B, 1, T, S]. - - Returns: - output: [B, T, D]. - atten_probs: [B, N, T, S]. - """ - p = self.params - b, t, _ = py_utils.GetShape(query_vec, 3) - unnormalized_query_vec = query_vec - - # Layer normalization. - if p.ln_tpl: - query_vec = self.layer_norm.FProp(theta.layer_norm, query_vec) - - # For self-attention: keys = queries. - if source_vecs is None: - source_vecs = query_vec - - # Generates mask, with shape [b, t, t]. - if per_step_padding_override is None: - if p.is_masked and segment_mask is None: - # causal padding. - per_step_padding = tf.tile( - tf.expand_dims(CausalPadding(t, dtype=query_vec.dtype), 0), - [b, 1, 1]) - else: - per_step_padding = None - else: - per_step_padding = per_step_padding_override - - # Multiheaded attention. - with tf.name_scope('atten'): - ctx_vec, atten_probs = self.atten.FProp( - theta.atten, - query_vec, # query - source_vecs, # key - source_vecs, # value - paddings, - segment_mask=segment_mask, - per_step_padding=per_step_padding) - - # Residual connection. - ctx_vec = self.residual_dropout.FProp(theta.residual_dropout, ctx_vec) - input_to_add = ( - unnormalized_query_vec if p.add_unnormalized_input else query_vec) - if p.add_skip_connection: - ctx_vec += input_to_add - return ctx_vec, atten_probs - - def InitStates(self, theta, target_batch_size, target_max_length): - return self.atten.InitStates(theta.atten, target_batch_size, - target_max_length) - - def ExtendStep(self, - theta, - query_vec, - cached_states, - time_step, - use_short_seq_opt=False): - """Compute the result and update cached states for the current step. - - This function is used by autoregressive decoding. This function knows the - length of full sequence, thus it is different from StreamingExtendStep. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [B, 1, D] - cached_states: A `.NestedMap` object containing tensors which are the - results of previous attentions, used for fast decoding. key - [T, B, - N, H]. value - [T, B, N, H]. - time_step: A scalar or tensor with [B], current decode step, 0-based. - if it's a scalar, all the time step are the same decode step. - if it's a tensor, it represents current decode step for each sample. - use_short_seq_opt: A bool, whether using short sequence optimization. - - Returns: - cur_output: [B, 1, D] - updated_states: A `.NestedMap` object containing the updated states. - key - [T, B, N, H]. - value - [T, B, N, H]. - - Raises: - ValueError: If not used as masked/causal self-attention. - """ - p = self.params - if not p.is_masked: - raise ValueError( - 'ExtendStep should be used only by masked/causal self-attention.') - - t, b, _, _ = py_utils.GetShape(cached_states.key, 4) - unnormalized_query_vec = query_vec - time_step = tf.convert_to_tensor(time_step) - - if time_step.shape.ndims == 0: - batch_time_step = tf.tile(tf.reshape(time_step, [-1]), [b]) - else: - batch_time_step = time_step - - # Generates mask, with shape [b, 1, t]. - zero_padding = tf.zeros([b, t], dtype=query_vec.dtype) - # [b, t] - per_step_padding = tf.where( - tf.less( - tf.tile(tf.expand_dims(tf.range(t), 0), [b, 1]), - tf.expand_dims(batch_time_step + 1, -1)), zero_padding, - tf.ones_like(zero_padding, dtype=query_vec.dtype)) - # [b, 1, t] - per_step_padding = tf.expand_dims(per_step_padding, 1) - - # Layer normalization. - if p.ln_tpl: - query_vec = self.layer_norm.FProp(theta.layer_norm, query_vec) - - # Multiheaded masked/causal self-attention. - ctx_vec, updated_states = self.atten.ExtendStep(theta.atten, query_vec, - cached_states, None, None, - per_step_padding, time_step, - use_short_seq_opt) - - # Residual connection. - ctx_vec = self.residual_dropout.FProp(theta.residual_dropout, ctx_vec) - input_to_add = ( - unnormalized_query_vec if p.add_unnormalized_input else query_vec) - if p.add_skip_connection: - ctx_vec += input_to_add - return ctx_vec, updated_states - - def zero_state(self, batch_size=1): - """Returns the initial state given the batch size. - - Args: - batch_size: the batch size. - - Returns: - state: The initial state for streaming inference. - """ - return self.atten.zero_state(batch_size) - - def StreamingExtendStep(self, query_vec, state, time_step): - """Computes the value vector given the query of the current step. - - Args: - query_vec: A query vector of shape [B, 1, D]. - state: A `.NestedMap` object containing tensors {key, value} which are - results of previous attentions. key, value are of shape [T, B, N, H] - where T is the context size of this layer. - time_step: A tensor of shape [1] and type tf.int32. Note, we can not use - scalar tensor here because TfLiteConverter doesn't have good support of - it (b/138865275). - - Returns: - output: Output of the given query vector with shape [B, 1, D]. - state: updated state. - """ - assert isinstance(self.atten, LocalSelfAttention) or isinstance( - self.atten, LocalSelfAttentionXL) - - p = self.params - unnormalized_query_vec = query_vec - if p.ln_tpl: - query_vec = self.layer_norm.FProp(self.theta.layer_norm, query_vec) - output, state = self.atten.StreamingExtendStep(query_vec, state, time_step) - - # Residual connection. - input_to_add = ( - unnormalized_query_vec if p.add_unnormalized_input else query_vec) - if p.add_skip_connection: - output += input_to_add - return output, state - - -class TransformerMultiSourceAttentionLayer(TransformerAttentionLayer): - """Batch major multi-source multi-headed attention. - - Only supports scenarios 3 described by comments on TransformerAttentionLayer: - - 3. Multi-source multi-headed cross-attention, where attention keys and values - (source_vecs) are coming from different sources (one of them is usually - the outputs of the encoder), and queries coming from the previous layer - outputs (decoder). Specifically, attention keys and values are NestedMaps - containing encodings of different sources. This corresponds to a - multi-source decoder-to-encoder attention mechanism, i.e., decoder attends - to encoder outputs and other sources. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_source', 0, 'Number of sources to attend to.') - p.Define( - 'primary_source_index', 0, 'Index of the primary source whose ' - 'attention probs will be returned.') - p.Define('multi_source_atten', MultiSourceAttention.Params(), - 'Multi-source attention params.') - # Only used for case 3. - p.is_masked = False - return p - - def _InitAttentionParams(self, atten_tpl): - """Returns an initialized multi-source transformer attention parameters.""" - p = self.params - source_atten_tpls = [] - # Set up each source attention. - for i in range(p.num_source): - src_key = 'source_%d' % i - src_atten = atten_tpl.Copy() - src_atten = super()._InitAttentionParams(src_atten) - src_atten.name = 'multihead_atten_%s' % src_key - source_atten_tpls.append((src_key, src_atten)) - - # Initialize multi-source attention. - msa = p.multi_source_atten.Copy() - msa.name = 'multi_source_atten' - msa.input_dim = p.input_dim - msa.hidden_dim = p.hidden_dim - msa.source_atten_tpls = source_atten_tpls - msa.primary_source_key = 'source_%d' % p.primary_source_index - return msa - - -class TransformerLayer(base_layer.BaseLayer): - """Transformer layer with multiheaded attention. - - Applies self-attention followed by a cross-attention and feed forward layer. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', 0, 'Dimension of the transformer block input.') - p.Define('output_dim', 0, 'Dimension of the transformer block output.') - p.Define('num_heads', None, 'Num of heads in self attention.') - p.Define('has_aux_atten', False, - 'If set, introduces a second attention layer') - p.Define('mask_self_atten', False, 'If True, use masked self-attention.') - p.Define('tr_atten_tpl', - TransformerAttentionLayer.Params().Set(), - 'Transformer Attention Layer params.') - p.Define( - 'tr_self_atten_tpl', None, - 'Attention template for self attention. If unset, use tr_atten_tpl.') - p.Define( - 'tr_fflayer_tpl', - layers_with_attention.TransformerFeedForwardLayer.Params().Set( - hidden_dim=2048), 'Transformer Feed-Forward Layer params.') - p.Define('packed_input', False, - 'If True, each training example may pack multiple sequences.') - return p - - @classmethod - def CommonParams(cls, - input_dim, - atten_num_heads, - atten_is_relative=False, - atten_local_context=None, - atten_left_context=None, - atten_right_context=None, - has_aux_atten=False, - mask_self_atten=False, - fflayer_hidden_dim=None, - fflayer_output_dim=None, - dropout_prob=0.): - # pylint: disable=g-doc-args - """Returns a hyperparam for the most representative cases. - - CommonParams is not expected to be extended to an omnipotent/generic builder - method. Specific use cases should take the return value of it and apply - further customization. It should be kept lean and only extended cautiously - for very common cases. - """ - # pylint: enable=g-doc-args - output_dim = fflayer_output_dim or input_dim - fflayer_hidden_dim = fflayer_hidden_dim or 4 * input_dim - # TODO(jamesqin): check how mask_self_atten work with local atten. - p = cls.Params().Set( - name='transformer_layer', - input_dim=input_dim, - output_dim=output_dim, - num_heads=atten_num_heads, - has_aux_atten=has_aux_atten, - mask_self_atten=mask_self_atten) - p.tr_self_atten_tpl = TransformerAttentionLayer.CommonParams( - input_dim, - atten_num_heads, - is_masked=mask_self_atten, - local_context=atten_local_context, - left_context=atten_left_context, - right_context=atten_right_context, - dropout_prob=dropout_prob) - p.tr_fflayer_tpl.Set( - hidden_dim=fflayer_hidden_dim, - residual_dropout_prob=dropout_prob, - relu_dropout_prob=dropout_prob) - return p - - @classmethod - def SetNumInputNodes(cls, p, num_input_nodes): - p.input_dim = num_input_nodes - - @classmethod - def NumOutputNodes(cls, p): - return p.output_dim - - def __init__(self, params): - super().__init__(params) - p = self.params - - # Initialize masked multi-headed self-attention - if p.tr_self_atten_tpl is not None: - self_atten_tpl = p.tr_self_atten_tpl - else: - self_atten_tpl = p.tr_atten_tpl - params = self_atten_tpl.Copy() - params.name = 'multihead_self_atten' - params.input_dim = p.input_dim - params.is_masked = p.mask_self_atten - if p.num_heads: - params.num_heads = p.num_heads - params.atten_tpl.packed_input = p.packed_input - self.CreateChild('self_atten', params) - - if p.has_aux_atten: - # Initialize multi-headed cross-attention - params = p.tr_atten_tpl.Copy() - params.name = 'multihead_cross_atten' - params.input_dim = p.input_dim - if p.num_heads: - params.num_heads = p.num_heads - params.atten_tpl.packed_input = p.packed_input - self.CreateChild('cross_atten', params) - - # Initialize feed-forward layer - params = p.tr_fflayer_tpl.Copy() - params.name = 'tr_fflayer' - params.input_dim = p.input_dim - params.output_dim = p.output_dim - self.CreateChild('fflayer', params) - - def _GetSourceBatchSize(self, aux_vec): - return py_utils.GetShape(aux_vec, 2)[0] - - def _GetSourceLength(self, aux_vec): - return py_utils.GetShape(aux_vec, 2)[1] - - def FProp(self, - theta, - query_vec, - paddings, - aux_vec=None, - aux_paddings=None, - per_step_padding_override=None, - segment_mask=None, - aux_segment_mask=None): - """Transformer decoder layer. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [target_batch, target_time, dim]. - paddings: [target_batch, target_time]. - aux_vec: [source_batch, source_time, dim]. - aux_paddings: [source_batch, source_time]. - per_step_padding_override: [target_batch, target_time, target_time]. - segment_mask: [target_batch, 1, target_time, target_time] - aux_segment_mask: [source_batch, 1, target_time, source_time] - - target_batch can be a multiple of source_batch, where samples in - target_batch are arranged in the order of [m, source_batch] where m = - target_batch / source_batch. - - Returns: - The fflayer output with shape [target_batch, target_time, dim]. - atten_probs: [B, N, T, S]. - """ - p = self.params - # First the self-attention layer. - if p.packed_input: - assert aux_segment_mask is not None, ('Need to specify aux_segment_mask ' - 'for packed input.') - with tf.name_scope('self_atten'): - atten_vec, atten_probs = self.self_atten.FProp( - theta.self_atten, - query_vec, - None, - paddings, - segment_mask=segment_mask, - per_step_padding_override=per_step_padding_override) - - if p.has_aux_atten: - with tf.name_scope('aux_atten'): - # Next the cross-attention layer. - target_batch, target_time, dim = py_utils.GetShape(query_vec, 3) - - source_batch = self._GetSourceBatchSize(aux_vec) - source_time = self._GetSourceLength(aux_vec) - - atten_vec = tf.reshape(atten_vec, [-1, source_batch, target_time, dim]) - atten_vec = tf.reshape( - tf.transpose(atten_vec, [1, 0, 2, 3]), [source_batch, -1, dim]) - atten_vec, atten_probs = self.cross_atten.FProp( - theta.cross_atten, - atten_vec, - aux_vec, - aux_paddings, - segment_mask=aux_segment_mask) - num_heads = py_utils.GetShape(atten_probs)[1] - atten_probs = tf.reshape( - atten_probs, - [source_batch, -1, num_heads, target_time, source_time]) - atten_probs = tf.transpose(atten_probs, [1, 0, 2, 3, 4]) - atten_probs = tf.reshape( - atten_probs, [target_batch, num_heads, target_time, source_time]) - atten_vec = tf.reshape(atten_vec, [source_batch, -1, target_time, dim]) - atten_vec = tf.transpose(atten_vec, [1, 0, 2, 3]) - atten_vec = tf.reshape(atten_vec, [target_batch, target_time, dim]) - - # Finally the feed-forward layer. - with tf.name_scope('fflayer'): - return self.fflayer.FProp(theta.fflayer, atten_vec, paddings), atten_probs - - def InitStates(self, theta, target_batch_size, target_max_length): - return self.self_atten.InitStates(theta.self_atten, target_batch_size, - target_max_length) - - def ExtendStep(self, - theta, - query_vec, - aux_vec, - aux_paddings, - cached_states, - time_step, - use_short_seq_opt=False): - """Transformer decoder layer, extend one step in autoregressive decoding. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [target_batch, 1, dim]. - aux_vec: [source_batch, source_time, dim] - aux_paddings: [source_batch, source_time] - cached_states: A `.NestedMap` object containing tensors which are the - results of previous attentions, used for fast decoding. key - - [target_time, target_batch, num_heads, dim_per_head]. value - - [target_time, target_batch, num_heads, dim_per_head]. - time_step: A scalar, the current decode step, 0-based. - use_short_seq_opt: A bool, whether using short sequence optimization. - - Returns: - cur_output: [target_batch, 1, dim] - updated_states: A `.NestedMap` object containing the updated states. - key - [target_time, target_batch, num_heads, dim_per_head]. - value - [target_time, target_batch, num_heads, dim_per_head]. - """ - target_batch, _, dim = py_utils.GetShape(query_vec, 3) - - # First the self-attention layer. - atten_vec, updated_states = self.self_atten.ExtendStep( - theta.self_atten, query_vec, cached_states, time_step, - use_short_seq_opt) - if self.params.has_aux_atten: - source_batch = self._GetSourceBatchSize(aux_vec) - # Next the cross-attention layer. - atten_vec = tf.reshape(atten_vec, [source_batch, -1, dim]) - atten_vec, _ = self.cross_atten.FProp(theta.cross_atten, atten_vec, - aux_vec, aux_paddings) - atten_vec = tf.reshape(atten_vec, [target_batch, 1, -1]) - - # Finally the feed-forward layer. - cur_output = self.fflayer.FProp( - theta.fflayer, atten_vec, - tf.zeros([target_batch, 1], dtype=atten_vec.dtype)) - return cur_output, updated_states - - -class MultiSourceTransformerLayer(TransformerLayer): - """Multi-source transformer layer with multiheaded attention. - - Multi-source attention is used for cross attention. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_source', 0, 'Number of sources to attend to.') - p.Define( - 'primary_source_index', 0, 'Index for the primary source ' - 'whose attention probabilities will be used as an output.') - return p - - def __init__(self, params): - assert issubclass(params.tr_atten_tpl.cls, - TransformerMultiSourceAttentionLayer) - # Set up multi-source attention layer - cross_atten_p = params.tr_atten_tpl - cross_atten_p.num_source = params.num_source - cross_atten_p.primary_source_index = params.primary_source_index - assert params.tr_self_atten_tpl - super().__init__(params) - - @property - def primary_source_key(self): - return 'source_%d' % self.params.primary_source_index - - def _GetSourceBatchSize(self, aux_vec): - return py_utils.GetShape(aux_vec[self.primary_source_key], 2)[0] - - def _GetSourceLength(self, aux_vec): - return py_utils.GetShape(aux_vec[self.primary_source_key], 2)[1] - - -# mt_attention_layer.MultiHeadedAttentionXL -ATTEN_TRANSFORMER_XL = 'transformer_xl' -# mt_attention_layer.MultiHeadedAttentionRPE -ATTEN_RPE = 'rpe' - - -def UseRelativeAttentionInTransformerLayer(transformer_params, - rel_pos_emb_dim, - atten_type=ATTEN_TRANSFORMER_XL): - """Uses transformer-xl attention for self attention of a transformer layer. - - Args: - transformer_params: A mt_attention_layer.TransformerLayer.Params() object. - rel_pos_emb_dim: (int) Relative positional embedding dim to be set. - atten_type: (string) Attention type. Supported: - - 'transformer_xl': mt_attention_layer.MultiHeadedAttentionXL - - 'rpe': mt_attention_layer.MultiHeadedAttentionRPE - - Returns: - A mt_attention_layer.TransformerLayer.Params() object with relative pos emb. - """ - if not issubclass(transformer_params.cls, TransformerLayer): - raise ValueError('Unsupported input transformer layer: %s' % - transformer_params.cls) - - if atten_type not in (ATTEN_TRANSFORMER_XL, ATTEN_RPE): - raise ValueError('Relative attention type: %s unsupported' % atten_type) - - # Gets multiheaded attention tpl from self attention config in transformer. - trans_params_copy = transformer_params.Copy() - if trans_params_copy.tr_self_atten_tpl is None: - trans_params_copy.tr_self_atten_tpl = trans_params_copy.tr_atten_tpl.Copy() - atten_tpl = trans_params_copy.tr_self_atten_tpl.atten_tpl - - # If already using relative attention class. - if atten_tpl.cls in (MultiHeadedAttentionRPE, MultiHeadedAttentionXL, - LocalSelfAttentionXL): - atten_tpl.rel_pos_emb_dim = rel_pos_emb_dim - return trans_params_copy - - if atten_type == ATTEN_TRANSFORMER_XL: - if atten_tpl.cls == MultiHeadedAttention: - rel_atten_tpl = MultiHeadedAttentionXL.Params() - elif atten_tpl.cls == LocalSelfAttention: - rel_atten_tpl = (LocalSelfAttentionXL.Params()) - else: - raise ValueError('Unsupported attention: %s' % atten_tpl.cls) - elif atten_type == ATTEN_RPE: - rel_atten_tpl = MultiHeadedAttentionRPE.Params() - - rel_atten_tpl = hyperparams.CopyFieldsTo(atten_tpl, rel_atten_tpl) - rel_atten_tpl.rel_pos_emb_dim = rel_pos_emb_dim - - trans_params_copy.tr_self_atten_tpl.atten_tpl = rel_atten_tpl - return trans_params_copy - - -def ClearRelativeAttentionInTransformerLayer(transformer_params): - """Removes relative position attention in the transformer layer. - - Args: - transformer_params: A mt_attention_layer.TransformerLayer param. - - Returns: - A mt_attention_layer.TransformerLayer param without relative attention. - """ - if not issubclass(transformer_params.cls, TransformerLayer): - raise ValueError('Unsupported input transformer layer: %s' % - transformer_params.cls) - trans_params_copy = transformer_params.Copy() - if trans_params_copy.tr_self_atten_tpl is None: - trans_params_copy.tr_self_atten_tpl = trans_params_copy.tr_atten_tpl.Copy() - attention_tpl = trans_params_copy.tr_self_atten_tpl.atten_tpl - if attention_tpl.cls == MultiHeadedAttentionXL: - new_attention_tpl = MultiHeadedAttention.Params() - elif attention_tpl.cls == (LocalSelfAttentionXL): - new_attention_tpl = LocalSelfAttention.Params() - else: - raise ValueError('Unsupported attention params: %s' % attention_tpl.cls) - - new_attention_tpl = hyperparams.CopyFieldsTo( - attention_tpl, new_attention_tpl, skip=['rel_pos_emb_dim', 'skip_term_b']) - trans_params_copy.tr_self_atten_tpl.atten_tpl = new_attention_tpl - return trans_params_copy - - -class TransformerDecoderLayer(TransformerLayer): - """Transformer decoder layer with multiheaded attention.""" - - @classmethod - def Params(cls): - p = super().Params() - p.has_aux_atten = True - p.mask_self_atten = True - return p - - -class MultiSourceTransformerDecoderLayer(MultiSourceTransformerLayer): - """Multi-source transformer decoder layer with multiheaded attention.""" - - @classmethod - def Params(cls): - p = super().Params() - p.has_aux_atten = True - p.mask_self_atten = True - return p - - -class StackedTransformerLayers(base_layer.BaseLayer): - """A stack of Batch-Major Transformer layers.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('has_aux_atten', False, - 'If set, introduces a second attention layer') - p.Define('mask_self_atten', False, 'If True, use masked self-attention.') - p.Define('num_layers', 0, 'Num of layers in this stack.') - p.Define('mdl_dim', 0, 'Model dimension in Transformer layers.') - p.Define('hidden_dim', 0, - 'The hidden layer dimension in Transformer layers.') - p.Define('num_atten_heads', 0, 'Num of attention heads.') - p.Define('dropout_prob', 0.0, - 'Apply dropout at this prob at various places.') - p.Define('add_unnormalized_input', True, - 'If set, uses unnormalized input in the residual add.') - p.Define('transformer_layer_params_tpl', TransformerLayer.Params(), - 'A template of TransformerLayer.params.') - p.Define('final_layer_norm', False, - 'If true, apply layer normalization to the final output.') - p.Define('packed_input', False, - 'If True, each training example may pack multiple sequences.') - p.Define('use_fused_layernorm', False, 'Whether to use fused layernorm.') - p.Define( - 'splits', None, 'None or a list of layer indices. If None, all layers ' - 'are placed on the same and only one partition. Else, len(splits) is ' - 'the number of partitions the stack is sliced into. layer_i is placed ' - 'on the kth partition (0-based) where split[k] < i <= split[k+1].') - return p - - def __init__(self, params): - if not params.splits: - params.splits = [params.num_layers - 1] - else: - assert all(x <= params.num_layers - 1 for x in params.splits) - # Assert p.splits is strictly monotonically increasing. - assert sorted(list(set(params.splits))) == params.splits - super().__init__(params) - p = self.params - - assert p.num_layers > 0 - assert p.mdl_dim > 0 - assert p.hidden_dim > 0 - assert p.num_atten_heads > 0 - assert 0.0 <= p.dropout_prob < 1.0 - - def _LayerParams(ii): - """Construct ii-th layer params.""" - p_ii = p.transformer_layer_params_tpl.Copy() - p_ii.name = 'layer_%d' % ii - p_ii.has_aux_atten = p.has_aux_atten - p_ii.mask_self_atten = p.mask_self_atten - p_ii.input_dim = p.mdl_dim - p_ii.output_dim = p.mdl_dim - p_ii.packed_input = p.packed_input - p_ii.tr_atten_tpl.num_heads = p.num_atten_heads - p_ii.tr_atten_tpl.atten_dropout_prob = p.dropout_prob - p_ii.tr_atten_tpl.residual_dropout_prob = p.dropout_prob - p_ii.tr_atten_tpl.add_unnormalized_input = p.add_unnormalized_input - p_ii.tr_fflayer_tpl.hidden_dim = p.hidden_dim - p_ii.tr_fflayer_tpl.residual_dropout_prob = p.dropout_prob - p_ii.tr_fflayer_tpl.relu_dropout_prob = p.dropout_prob - return p_ii - - layer_params = [_LayerParams(ii) for ii in range(p.num_layers)] - - self.CreateChildren('x_layers', layer_params) - - if p.final_layer_norm: - final_ln_p = layers.LayerNorm.Params().Set( - input_dim=p.mdl_dim, use_fused_layernorm=p.use_fused_layernorm) - self.CreateChild('final_ln', final_ln_p) - - @classmethod - def GetSplitForLayer(cls, buckets, layer_index): - assert layer_index <= buckets[-1], ( - f'layer_index:{layer_index} > buckets[-1]:{buckets[-1]}') - # Return index of the smallest element greater than or equal to layer_index - return bisect.bisect_left(buckets, layer_index) - - def FProp(self, - theta, - query_vec, - paddings, - aux_vec=None, - aux_paddings=None, - segment_mask=None, - aux_segment_mask=None): - """Stacked Transformer layer. - - Args: - theta: A `NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [batch, target_time, dim]. - paddings: [batch, target_time]. - aux_vec: [batch, source_time, dim]. - aux_paddings: [batch, source_time]. - segment_mask: [batch, 1, target_time, target_time] - aux_segment_mask: [batch, 1, target_time, source_time] - - Returns: - (context, paddings), where the context vector has shape [batch, - target_time, dim]. - """ - p = self.params - x_out = query_vec - cluster = self.cluster - - with tf.name_scope(p.name): - for i in range(p.num_layers): - x_in = x_out - with tf.device( - cluster.WorkerDeviceInModelSplit( - self.GetSplitForLayer(self.params.splits, i))): - x_out, _ = self.x_layers[i].FProp(theta.x_layers[i], x_in, paddings, - aux_vec, aux_paddings, segment_mask, - aux_segment_mask) - if p.final_layer_norm: - # Place on the last device. - with tf.device( - cluster.WorkerDeviceInModelSplit( - self.GetSplitForLayer(self.params.splits, p.num_layers - 1))): - x_out = self.final_ln.FProp(theta.final_ln, x_out) - return x_out, paddings - - def InitStates(self, theta, *args, **kwargs): - return py_utils.NestedMap(x_layers=[ - layer.InitStates(layer_theta, *args, **kwargs) - for layer, layer_theta in zip(self.x_layers, theta.x_layers) - ]) - - def ExtendStep(self, - theta, - query_vec, - aux_vec, - aux_paddings, - cached_states, - time_step, - use_short_seq_opt=False): - """Transformer decoder layer, extend one step in autoregressive decoding. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [target_batch, 1, dim]. - aux_vec: [source_batch, source_time, dim] - aux_paddings: [source_batch, source_time] - cached_states: A `.NestedMap` object containing tensors which are the - results of previous attentions, used for fast decoding. - cached_states.x_layers is a list corresponding to self.x_layers, where - each element is a NestedMap with attention keys and values: - "key" - [target_time, target_batch, num_heads, dim_per_head]. - "value" - [target_time, target_batch, num_heads, dim_per_head]. - time_step: A scalar, the current decode step, 0-based. - use_short_seq_opt: A bool, whether using short sequence optimization. - - Returns: - cur_output: The last decoder layer output of shape [target_batch, 1, dim]. - updated_states: A `.NestedMap` object containing the updated states. - updated_states.x_layers is a list corresponding to self.x_layers, where - each element is a NestedMap with attention keys and values: - "key" - [target_time, target_batch, num_heads, dim_per_head]. - "value" - [target_time, target_batch, num_heads, dim_per_head]. - """ - p = self.params - with tf.name_scope(p.name): - updated_states = py_utils.NestedMap(x_layers=[]) - decoder_input = query_vec - for layer, layer_theta, layer_states in zip(self.x_layers, theta.x_layers, - cached_states.x_layers): - decoder_output, updated_layer_states = layer.ExtendStep( - layer_theta, decoder_input, aux_vec, aux_paddings, layer_states, - time_step, use_short_seq_opt) - updated_states.x_layers.append(updated_layer_states) - decoder_input = decoder_output - return decoder_output, updated_states - - -class TransformerFeedForwardLayerWithTaskId( - layers_with_attention.TransformerFeedForwardLayer): - """TransformerFeedForwardLayer with optional task_id input args.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('use_task_ids', False, - 'If set, introduces a second attention layer') - return p - - def FProp(self, theta, inputs, paddings, task_id=None): - """Feed-forward, residual and layer-norm. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: [batch, time, dim]. - paddings: [batch, time] - task_id: optional task_id with shape [batch] - - Returns: - tensor of the same shape with inputs - """ - p = self.params - if p.use_task_ids: - if task_id is None: - raise ValueError('Must pass task_id if use_task_ids.') - inputs_normalized = self.layer_norm.FProp(theta.layer_norm, inputs) - if hasattr(self, 'res_proj_layer'): - inputs = self.res_proj_layer.FProp(theta.res_proj_layer, inputs) - expanded_paddings = tf.expand_dims(paddings, -1) - fflayer_args = [inputs_normalized, expanded_paddings] - fflayer_args += [task_id] if p.use_task_ids else [] - h = inputs + self.residual_dropout.FProp( - theta.residual_dropout, self.fflayer.FProp(theta.fflayer, - *fflayer_args)) - return h - - -# TODO(ankurbpn,huangyp): Remove this layer. -class GPipeTransformerLayer(TransformerLayer): - """GPipe compatible transformer layer. - - DEPRECATED: This layer and its use in GPipeTransformerStack is - deprecated. Consider using the new GPipeBatchMajorTransformerStack instead. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.tr_fflayer_tpl = TransformerFeedForwardLayerWithTaskId.Params() - return p - - def FProp(self, - theta, - source_vecs, - source_paddings, - target_vecs, - target_paddings, - source_segment_id, - target_segment_id, - transparent_acc, - transparent_acc_helper, - source_task_id=None, - target_task_id=None): - p = self.params - with tf.name_scope(p.name): - if p.has_aux_atten: # Decoder FProp - seg_mask = SegmentMask(target_segment_id, target_segment_id) - aux_seg_mask = SegmentMask(target_segment_id, source_segment_id) - atten_vec, _ = self.self_atten.FProp( - theta.self_atten, - target_vecs, - None, - target_paddings, - segment_mask=seg_mask) - atten_vec, _ = self.cross_atten.FProp( - theta.cross_atten, - atten_vec, - source_vecs, - source_paddings, - segment_mask=aux_seg_mask) - atten_vec = self.fflayer.FProp(theta.fflayer, atten_vec, - target_paddings, target_task_id) - atten_vec.set_shape(target_vecs.shape) - return (source_vecs, source_paddings, atten_vec, target_paddings, - source_segment_id, target_segment_id, transparent_acc, - transparent_acc_helper, source_task_id, target_task_id) - # Encoder FProp - seg_mask = SegmentMask(source_segment_id, source_segment_id) - atten_vec, _ = self.self_atten.FProp( - theta.self_atten, - source_vecs, - None, - source_paddings, - segment_mask=seg_mask) - atten_vec = self.fflayer.FProp(theta.fflayer, atten_vec, source_paddings, - source_task_id) - atten_vec.set_shape(source_vecs.shape) - - return (atten_vec, source_paddings, target_vecs, target_paddings, - source_segment_id, target_segment_id, transparent_acc, - transparent_acc_helper, source_task_id, target_task_id) - - @classmethod - def FPropMeta(cls, p, inputs, *args): - py_utils.CheckShapes((inputs,)) - flops_per_element = 5 - source_batch, src_time, dim = inputs - flops = flops_per_element * src_time * src_time * source_batch * dim - args = args if isinstance(args, tuple) else (args,) - return py_utils.NestedMap(flops=flops, out_shapes=(inputs,) + args) - - @classmethod - def SetupDeterministicDropout(cls, params): - """Replaced dropout layers in transformer with deterministic ones.""" - params.tr_atten_tpl.dropout_tpl = ( - layers.DeterministicDropoutLayer.Params()) - params.tr_atten_tpl.atten_tpl.dropout_tpl = ( - layers.DeterministicDropoutLayer.Params()) - params.tr_fflayer_tpl.residual_dropout_tpl = ( - layers.DeterministicDropoutLayer.Params()) - params.tr_fflayer_tpl.fflayer_tpl.dropout = ( - layers.DeterministicDropoutLayer.Params()) - return params - - def ExtendStep(self, - theta, - query_vec, - aux_vec, - aux_paddings, - cached_states, - time_step, - task_id=None, - use_short_seq_opt=False): - """Transformer decoder layer, extend one step in autoregressive decoding. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [target_batch, 1, dim]. - aux_vec: [source_batch, source_time, dim] - aux_paddings: [source_batch, source_time] - cached_states: A `.NestedMap` object containing tensors which are the - results of previous attentions, used for fast decoding. key - - [target_time, target_batch, num_heads, dim_per_head]. value - - [target_time, target_batch, num_heads, dim_per_head]. - time_step: A scalar, the current decode step, 0-based. - task_id: [batch_size]: the input task_id meta information. - use_short_seq_opt: A bool, whether using short sequence optimization. - - Returns: - cur_output: [target_batch, 1, dim] - updated_states: A `.NestedMap` object containing the updated states. - key - [target_time, target_batch, num_heads, dim_per_head]. - value - [target_time, target_batch, num_heads, dim_per_head]. - """ - target_batch, _, dim = py_utils.GetShape(query_vec, 3) - source_batch = py_utils.GetShape(aux_vec)[0] - - # First the self-attention layer. - atten_vec, updated_states = self.self_atten.ExtendStep( - theta.self_atten, query_vec, cached_states, time_step, - use_short_seq_opt) - - # Next the cross-attention layer. - atten_vec = tf.reshape(atten_vec, [source_batch, -1, dim]) - atten_vec, _ = self.cross_atten.FProp(theta.cross_atten, atten_vec, aux_vec, - aux_paddings) - atten_vec = tf.reshape(atten_vec, [target_batch, 1, -1]) - - # Finally the feed-forward layer. - cur_output = self.fflayer.FProp( - theta.fflayer, atten_vec, - tf.zeros([target_batch, 1], dtype=atten_vec.dtype), task_id) - return cur_output, updated_states - - -class GPipeBatchMajorTransformerLayer(TransformerLayer): - """GPipe compatible batch majortransformer layer. - - To be used with the new GPipeBatchMajorStack. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('ln_tpl', layers.LayerNorm.Params(), - 'Layer norm default params. No layernorm if set to None.') - p.Define('output_layer_norm', False, - 'Whether to layer normalize the output of the layer.') - return p - - def __init__(self, params): - # Initialize output layer norm. - super().__init__(params) - p = self.params - if p.output_layer_norm: - params = p.ln_tpl.Copy() - params.name = 'output_ln' - params.input_dim = p.input_dim - self.CreateChild('layer_norm', params) - - def FProp(self, theta, source_vecs, source_paddings, target_vecs, - target_paddings, encoder_self_atten_segment_mask, - decoder_self_atten_segment_mask, decoder_cross_atten_segment_mask): - p = self.params - with tf.name_scope(p.name): - if p.has_aux_atten: # Decoder FProp - sa_mask, ca_mask = None, None - if p.packed_input: - # This computation doesn't behave nicely when outside - # recurrent.Recurrent resulting in nans for splits > 1 - min_val = GetDtypeMin(decoder_self_atten_segment_mask.dtype) - # Operator overloading with * produces type-errors when running on - # borg with splits > 1. - sa_mask = tf.math.multiply(min_val, decoder_self_atten_segment_mask) - ca_mask = tf.math.multiply(min_val, decoder_cross_atten_segment_mask) - atten_vec, _ = self.self_atten.FProp( - theta.self_atten, - target_vecs, - None, - target_paddings, - segment_mask=sa_mask) - atten_vec, _ = self.cross_atten.FProp( - theta.cross_atten, - atten_vec, - source_vecs, - source_paddings, - segment_mask=ca_mask) - atten_vec = self.fflayer.FProp(theta.fflayer, atten_vec, - target_paddings) - atten_vec.set_shape(target_vecs.shape) - if p.output_layer_norm: - atten_vec = self.layer_norm.FProp(theta.layer_norm, atten_vec) - return (source_vecs, source_paddings, atten_vec, target_paddings, - encoder_self_atten_segment_mask, - decoder_self_atten_segment_mask, - decoder_cross_atten_segment_mask) - - # Encoder FProp - sa_mask = None - if p.packed_input: - min_val = GetDtypeMin(encoder_self_atten_segment_mask.dtype) - sa_mask = tf.math.multiply(min_val, encoder_self_atten_segment_mask) - atten_vec, _ = self.self_atten.FProp( - theta.self_atten, - source_vecs, - None, - source_paddings, - segment_mask=sa_mask) - atten_vec = self.fflayer.FProp(theta.fflayer, atten_vec, source_paddings) - atten_vec.set_shape(source_vecs.shape) - if p.output_layer_norm: - atten_vec = self.layer_norm.FProp(theta.layer_norm, atten_vec) - - return (atten_vec, source_paddings, target_vecs, target_paddings, - encoder_self_atten_segment_mask, decoder_self_atten_segment_mask, - decoder_cross_atten_segment_mask) - - @classmethod - def FPropMeta(cls, p, inputs, *args): - py_utils.CheckShapes((inputs,)) - flops_per_element = 5 - source_batch, src_time, dim = inputs - flops = flops_per_element * src_time * src_time * source_batch * dim - args = args if isinstance(args, tuple) else (args,) - return py_utils.NestedMap(flops=flops, out_shapes=(inputs,) + args) - - @classmethod - def SetupDeterministicDropout(cls, params): - """Replaced dropout layers in transformer with deterministic ones.""" - params.tr_atten_tpl.dropout_tpl = ( - layers.DeterministicDropoutLayer.Params()) - params.tr_atten_tpl.atten_tpl.dropout_tpl = ( - layers.DeterministicDropoutLayer.Params()) - params.tr_fflayer_tpl.residual_dropout_tpl = ( - layers.DeterministicDropoutLayer.Params()) - params.tr_fflayer_tpl.fflayer_tpl.dropout = ( - layers.DeterministicDropoutLayer.Params()) - return params - - def ExtendStep(self, - theta, - query_vec, - aux_vec, - aux_paddings, - cached_states, - time_step, - use_short_seq_opt=False): - """Transformer decoder layer, extend one step in autoregressive decoding. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [target_batch, 1, dim]. - aux_vec: [source_batch, source_time, dim] - aux_paddings: [source_batch, source_time] - cached_states: A `.NestedMap` object containing tensors which are the - results of previous attentions, used for fast decoding. key - - [target_time, target_batch, num_heads, dim_per_head]. value - - [target_time, target_batch, num_heads, dim_per_head]. - time_step: A scalar, the current decode step, 0-based. - use_short_seq_opt: A bool, whether using short sequence optimization. - - Returns: - cur_output: [target_batch, 1, dim] - updated_states: A `.NestedMap` object containing the updated states. - key - [target_time, target_batch, num_heads, dim_per_head]. - value - [target_time, target_batch, num_heads, dim_per_head]. - """ - target_batch, _, dim = py_utils.GetShape(query_vec, 3) - source_batch = py_utils.GetShape(aux_vec)[0] - - # First the self-attention layer. - atten_vec, updated_states = self.self_atten.ExtendStep( - theta.self_atten, query_vec, cached_states, time_step, - use_short_seq_opt) - - # Next the cross-attention layer. - atten_vec = tf.reshape(atten_vec, [source_batch, -1, dim]) - atten_vec, _ = self.cross_atten.FProp(theta.cross_atten, atten_vec, aux_vec, - aux_paddings) - atten_vec = tf.reshape(atten_vec, [target_batch, 1, -1]) - - # Finally the feed-forward layer. - cur_output = self.fflayer.FProp( - theta.fflayer, atten_vec, - tf.zeros([target_batch, 1], dtype=atten_vec.dtype)) - - if self.params.output_layer_norm: - cur_output = self.layer_norm.FProp(theta.layer_norm, cur_output) - return cur_output, updated_states - - -# pyformat: disable -class Builder(builder.Base): - """Builder for self-attention layers.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('model_dim', 4, 'Model dim of this layer.') - p.Define('num_heads', 1, 'Number of heads in the atten layer.') - p.Define('ff_hidden_dim', 4, 'Hidden dim of the feedforward layer') - p.Define('attention_hidden_dim', None, - 'Hidden dim of the attention layer.') - p.Define('residual_dropout_prob', 0, - 'Dropout prob to the output of each sub-layer before it is added ' - 'to the sub-layer input.') - p.Define('ff_activation_fn', tf.nn.relu, - 'Activation function in Feedforward layer.') - p.Define('ff_residual_weight', 1.0, 'Weight given to F(x) in the residual ' - 'connection: y = x + ff_residual_weight * F(x), in Feedforward ' - 'layer.') - p.Define('relu_dropout_prob', 0, - 'Probability at which we apply dropout to the hidden layer of ' - 'feed-forward network.') - p.Define('atten_dropout_prob', 0, - 'Probability at which we apply dropout to the attention layer') - p.Define('selfatten_add_unnormalized_input', True, - 'Whether to use unnormalized input in the residual add.') - p.Define('selfatten_enable_value_proj', True, - 'Whether value v is pre-projected before self attention or not.') - p.Define('conv_activation', 'RELU', - 'Activation function for convolution layer in Builder.') - p.Define('num_splits', 1, - 'Number of model parallelism splits.') - p.Define('num_micro_batches', 1, - 'Number of spatial partition along the batch dimension. ' - 'When num_micro_batches > 1, the effective batch size of the ' - 'intermediate activation is batch_size // num_micro_batches.' - 'This allows models to try larger batch size which might improve ' - 'model quality') - p.Define('glu_with_tanh', False, - 'If the Gated Linear Unit should apply tanh on the activation ' - 'input.') - p.Define('packed_input', False, - 'Whether to support packed input') - p.Define('enable_per_dim_scale', True, - 'Whether using per_dim_scale or scaling by a constant factor.') - p.Define('use_fused_layernorm', False, 'Whether to use fused layernorm.') - p.Define('use_bias', True, 'Whether to use bias for projection layer.') - p.Define('norm_layer_tpl', None, - 'If specified, the normalization layer template.') - p.Define( - 'enable_scaling_code_motion', False, 'Move scalings from the side ' - 'of T^2 to the side of T for better performance. This may result ' - 'in model quality drops when using bf16 for some models due to ' - 'different XLA fusion decisions.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.num_splits > 1 or p.num_micro_batches > 1: - assert p.deterministic_dropout - - def _Dropout(self, name, drop_prob): - """Returns a DropoutLayer Params.""" - return super()._Dropout(name, keep_prob=1.0 - drop_prob) - - def _Add(self, name, residual_weight=1.0): - return self._Fn(name, fn=lambda x, y: x + residual_weight * y, - fn_out=lambda x, y: x) - - def _ExpandDims(self, name): - return self._Fn(name, - fn=lambda x: tf.expand_dims(x, 2), - fn_out=lambda x: tshape.Shape(x[0:2] + [1] + x[2:]), - fn_flops=lambda x: 1) - - def _Squeeze(self, name): - return self._Fn(name, - fn=lambda x: tf.squeeze(x, 2), - fn_out=lambda x: tshape.Shape(x[0:2] + x[3:]), - fn_flops=lambda x: 1) - - def _Glu(self, name): - def _GLUFn(inputs): - gated_inputs, act_inputs = tf.split(inputs, 2, axis=-1) - return act_inputs * tf.sigmoid(gated_inputs) - - def _GatedTanhFn(inputs): - gated_inputs, act_inputs = tf.split(inputs, 2, axis=-1) - return tf.tanh(act_inputs) * tf.sigmoid(gated_inputs) - - fn = _GatedTanhFn if self.params.glu_with_tanh else _GLUFn - - return self._Fn(name, - fn=fn, - fn_out=lambda x: tshape.Shape(x[:-1] + [x[-1] / 2]), - fn_flops=lambda x: 15 * x.size) - - def _Pad(self, name): - return self._Fn( - name, - fn=lambda x, p: py_utils.ApplyPadding(tf.expand_dims(p, -1), x), - fn_out=lambda x, p: x, - fn_flops=lambda x, p: 2 * max(x.size, p.size)) - - def _MultiHeadedAtten(self, name, num_heads=None): - """Returns a MultiHeadedAttention params.""" - p = self.params - if num_heads is None: - num_heads = p.num_heads - atten_p = MultiHeadedAttention.Params().Set( - name=name, - input_dim=p.model_dim, - hidden_dim=p.attention_hidden_dim or p.model_dim, - num_heads=num_heads, - atten_dropout_prob=p.atten_dropout_prob, - enable_value_proj=p.selfatten_enable_value_proj, - enable_per_dim_scale=p.enable_per_dim_scale, - packed_input=p.packed_input, - fprop_dtype=p.fprop_dtype, - use_bias=p.use_bias, - enable_scaling_code_motion=p.enable_scaling_code_motion, - ) - if p.deterministic_dropout: - atten_p.dropout_tpl = layers.DeterministicDropoutLayer.Params() - return atten_p - - def Feedforward(self, name, is_causal=False, ff_hidden_dim=None): - del is_causal - p = self.params - if ff_hidden_dim is None: - ff_hidden_dim = p.ff_hidden_dim - sub_list = [ - ('i.vec->after_feedforward', - self._Seq( - 'feedforward', - self._LN('ln', p.model_dim, - use_fused_layernorm=p.use_fused_layernorm), - self._Linear('linear01', p.model_dim, ff_hidden_dim), - self._Bias('bias01', ff_hidden_dim), - self._Activation('act', p.ff_activation_fn), - self._Dropout('relu_dropout', p.relu_dropout_prob), - self._Linear('linear02', ff_hidden_dim, p.model_dim), - self._Bias('bias02', p.model_dim), - self._Dropout('dropout', p.residual_dropout_prob))), - ('i.vec,after_feedforward->added', self._Add('add', p.ff_residual_weight)), - ('added,i.paddings->o.vec', self._Pad('pad')), - ('i.paddings->o.paddings', self._Id('id')), - ] - - if p.packed_input: - sub_list.append(('i.segment_mask->o.segment_mask', - self._Id('segment_mask'))) - - return self._Graph( - name, - ['i'], # input NestedMap with {vec, paddings, segment_mask} - ['o'], # output NestedMap with {vec, paddings, segment_mask} - *sub_list) - - def _MaybeSplit(self, name, blocks): - p = self.params - if p.num_splits == 1 and p.num_micro_batches == 1: - return None - - num_layers = len(blocks) - assert num_layers >= p.num_splits - layers_per_split = (num_layers - 1) // p.num_splits + 1 - cells = [] - while blocks: - head, blocks = blocks[:layers_per_split], blocks[layers_per_split:] - cells.append(self._Seq('cell_{}'.format(len(cells)), *head)) - assert len(cells) == p.num_splits - - return gpipe.PipeliningLayer.Params().Set( - name=name, - cell_tpl=cells, - nested_map_fprop=True, - num_micro_batches=p.num_micro_batches) - - def _DepthwiseConv2D(self, name, filter_size, is_causal=False): - """A depthwise convolution block for lightweight conv.""" - p = self.params - conv_builder_params = conv_layers.Builder.Params() - if p.norm_layer_tpl: - conv_builder_params.norm_layer_tpl = p.norm_layer_tpl - conv_builder = conv_builder_params.Instantiate() - return conv_builder.DepthwiseConv2D( - name=name, - in_dim=p.model_dim, - depth_multiplier=1, - filter_shape=[filter_size, 1], - stride=(1, 1), - dilation=(1, 1), - activation=p.conv_activation, - is_causal=is_causal) - - def _NormalizedDepthwiseConv2D(self, name, kernel_size, is_causal=False): - """A depthwise convolution block for lightweight conv.""" - p = self.params - conv_builder_params = conv_layers.Builder.Params() - conv_builder = conv_builder_params.Instantiate() - return conv_builder.NormalizedDepthwiseConv2D( - name=name, - kernel_size=kernel_size, - num_heads=p.num_heads, - in_dim=p.model_dim, - dropconnect_prob=p.atten_dropout_prob, - deterministic_dropout=p.deterministic_dropout, - is_causal=is_causal) - - def LConv(self, - name, - kernel_size, - is_causal=False, - convolution_fn=None): - """[DEPRECATED] A lightweight convolution block as described in. - - Use conv_layer_builder.LConv() instead. - - https://arxiv.org/abs/1901.10430 - Corresponding PyTorch Implementation (L587): - https://github.com/pytorch/fairseq/blob/v0.6.2/fairseq/models/lightconv.py - - - This block can be used as an alternative to self-attention block. - - Args: - name: name of the params - kernel_size: kernel size used in the conv layer. - is_causal: is causal padding or not. - convolution_fn: Convolution to apply, default _NormalizedDepthwiseConv2D. - - Returns: - A LightWeightConvLayerBlock layer params. - """ - p = self.params - if convolution_fn is None: - convolution_fn = getattr(self, '_NormalizedDepthwiseConv2D') - - sub_list = [ - ('i.vec->pre_conv', - self._Seq( - 'pre_conv', - self._LN('ln', p.model_dim, - use_fused_layernorm=p.use_fused_layernorm), - self._Linear('linear', p.model_dim, p.model_dim * 2), - self._Bias('bias', p.model_dim * 2), - self._Glu('glu'), - self._ExpandDims('expand'))), - ('pre_conv,i.paddings->post_conv,o.paddings', - convolution_fn('conv', kernel_size, is_causal)), - ('post_conv->after_dropout', - self._Seq( - 'post_conv', - self._Squeeze('squeeze'), - self._Linear('linear', p.model_dim, p.model_dim), - self._Bias('bias', p.model_dim), - self._Dropout('dropout', p.residual_dropout_prob))), - ('i.vec,after_dropout->o.vec', self._Add('add')), - ] - if p.packed_input: - sub_list.append(('i.segment_mask->o.segment_mask', self._Id('segment_mask'))) - - return self._Graph( - name, - ['i'], # input NestedMap with {vec, paddings, segment_mask} - ['o'], # output NestedMap with {vec, paddings, segment_mask} - *sub_list - ) - - def LconvBlock(self, name, kernel_size, is_causal, - convolution_fn): - """A lightweight conv block followed by a feedforward one.""" - return self._Seq( - name, - self.LConv( - name='lconv', - kernel_size=kernel_size, - is_causal=is_causal, - convolution_fn=convolution_fn), - self.Feedforward('ff', is_causal)) - - def Seq(self, name, *subs): - """Returns a stack of sequential layers.""" - return self._Seq(name, *subs) - - def LConvStack(self, name, kernel_sizes, is_causal=False): - """Returns a stack of LConv layers with kernel size in kernel_sizes.""" - blocks = [] - for i, kernel_size in enumerate(kernel_sizes): - blocks.append( - self.LconvBlock( - name='block_{}'.format(i), - kernel_size=kernel_size, - is_causal=is_causal, - convolution_fn=None)) - return self._MaybeSplit(name, blocks) or self._Seq(name, *blocks) - - def _Stride(self, name, stride, first_n=None): - """Strides the input sequence. - - Args: - name: name of this layer. - stride: To use every k-th token, set the stride to k. When stride == 0, - only returns the first token of the input. When stride == 1, returns - every token in the input. - first_n: only considers the first N tokens for the output. We use - [:first_n:stride] to select the output tokens. If first_n is None, this - flag is a no-op. If stride is positive, the output sequence length is - "(first_n-1) // stride + 1". If stride is 0, first_n has to be None or - 1. first_n can't be 0. If first_n <= stride, only the first token is - used. - - Returns: - A layer params that does stride. - """ - assert first_n is None or first_n > 0 - if stride == 0: - assert first_n is None or first_n == 1 - return self._Fn( - name=name, - fn=lambda x: tf.expand_dims(x[:, 0], 1), - fn_out=lambda x: tshape.Shape(x[0:1] + [1] + x[2:]), - fn_flops=lambda x: 1) - - if first_n: - # out_seq_len is 1 if first_n is 1 ~ stride and is 2 if it's stride+1 ~ - # 2*stride... - out_seq_len = (first_n - 1) // stride + 1 - return self._Fn( - name=name, - fn=lambda x: x[:, :first_n:stride], - fn_out=lambda x: tshape.Shape(x[0:1] + [out_seq_len] + x[2:]), - fn_flops=lambda x: 1) - - if stride == 1: - return self._Id(name) - - return self._Fn( - name=name, - fn=lambda x: x[:, ::stride], - fn_out=lambda x: tshape.Shape(x[0:1] + x[1] // stride + x[2:]), - fn_flops=lambda x: 1) - - def _StridedAttention(self, name, stride=1, first_n=None, num_heads=None): - """Computes self attention with optional stride. - - Args: - name: name of this layer. - stride: If omitted, the default is 1: use every token in the query. To use - every k-th token, set the stride to k. When set to 0, only use the first - token of the query. - first_n: only considers the first N tokens for the output. We use - [:first_n:stride] to select the output tokens. If first_n is None, this - flag is a no-op. If stride is positive, the output sequence length is - "(first_n-1) // stride + 1". If stride is 0, first_n has to be None or - 1. first_n can't be 0. If first_n <= stride, only the first token is - used. - num_heads: the number of heads. - - Returns: - A self attention layer params. - """ - p = self.params - input_to_add = ('i.vec' - if p.selfatten_add_unnormalized_input else 'after_ln') - - attention_inputs = 'strided_query,after_ln,after_ln,i.paddings' - if p.packed_input: - attention_inputs += ',i.segment_mask' - - if num_heads is None: - num_heads = p.num_heads - - sub_list = [ - ('i.vec->after_ln', - self._LN('LN', p.model_dim, - use_fused_layernorm=p.use_fused_layernorm)), - ('after_ln->strided_query', - self._Stride('query_after_stride', stride, first_n)), - ('{}->after_att,prob'.format(attention_inputs), - self._MultiHeadedAtten('atten', num_heads)), - ('after_att->after_dropout', - self._Dropout('dropout', p.residual_dropout_prob)), - ('{}->strided_input'.format(input_to_add), - self._Stride('before_add', stride, first_n)), - ('strided_input,after_dropout->o.vec', - self._Add('add')), - ('i.paddings->o.paddings', - self._Stride('padding_after_Stride', stride, first_n)), - ] - if p.packed_input: - sub_list.append(('i.segment_mask->o.segment_mask', self._Id('segment_mask'))) - - return self._Graph( - name, - ['i'], # input NestedMap with {vec, paddings, segment_mask} - ['o'], # output NestedMap with {vec, paddings, segment_mask} - *sub_list) - - def TransformerEncoderLayer(self, name, stride=1, first_n=None, - ff_hidden_dim=None, num_heads=None): - """(inputs, paddings) -> (encoded, paddings). - - Args: - name: the string name of the encoder layer params. - stride: To use every k-th token, set the stride to k. When stride == 0, - only returns the first token of the input. When stride == 1, returns - every token in the input. - first_n: only considers the first N tokens for the output. We use - [:first_n:stride] to select the output tokens. If first_n is None, this - flag is a no-op. If stride is positive, the output sequence length is - "(first_n-1) // stride + 1". If stride is 0, first_n has to be None or - 1. first_n can't be 0. If first_n <= stride, only the first token is - used. - ff_hidden_dim: The feed forward layer's hidden dimension. If specified, - this will override p.ff_hidden_dim. - num_heads: The number of heads for the multi-head attention module. If - specified, this will override p.num_heads. - - Returns: - A transformer encoder layer params that supports optional stride. - """ - p = self.params - if ff_hidden_dim is None: - ff_hidden_dim = p.ff_hidden_dim - if num_heads is None: - num_heads = p.num_heads - return self._Seq(name, self._Seq( - 'block', - self._StridedAttention('self_atten', stride=stride, - first_n=first_n, num_heads=num_heads), - self.Feedforward('ff', ff_hidden_dim=ff_hidden_dim))) - - def Stack(self, name, blocks): - """Returns a stack of sequential layers.""" - return self._MaybeSplit(name, blocks) or self._Seq(name, *blocks) - - def TransformerEncoderStack(self, name, num_layers=1): - """Returns a stack of num_layers self-attention layers.""" - blocks = [ - self.TransformerEncoderLayer(name='iter_{:0>3d}'.format(d)) - for d in range(num_layers) - ] - return self.Stack(name, blocks) -# pyformat: enable - - -class LmBuilder(Builder): - """Langange model builder with causal padding.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('xla_num_partitions', None, 'Number of SPMD partitions.') - p.Define('dtype', tf.float32, 'Datatype to use.') - return p - - def _ShardedVar(self, name, weights, split_dim): - return moe_layers.ShardedVarLayer.Params().Set( - name=name, - weights=weights, - split_dimension=split_dim, - fprop_dtype=self.params.fprop_dtype, - num_devices=self.params.xla_num_partitions) - - def _LinearWeight(self, name, input_dim, output_dim, split_dim): - return self._ShardedVar( - name=name, - weights=[('w', - py_utils.WeightParams( - shape=[input_dim, output_dim], - init=py_utils.WeightInit.Uniform((3. / input_dim)**0.5), - dtype=self.params.dtype))], - split_dim=split_dim) - - def _Linear(self, name, input_dim, output_dim, split_dim=0): - return self._Graph( - name, - ['inputs'], - ['outputs'], - ('->w', self._LinearWeight('w', input_dim, output_dim, split_dim)), - ('inputs,w->outputs', - self._Fn( - 'linear', - fn=lambda inputs, w: tf.einsum('BLI,IO->BLO', inputs, w))), - ) - - def _BiasWeight(self, name, dim): - return self._ShardedVar( - name=name, - weights=[('b', - py_utils.WeightParams( - shape=[dim], - init=py_utils.WeightInit.Constant(0.0), - dtype=self.params.dtype))], - split_dim=0) - - def _Bias(self, name, dim): - return self._Graph( - name, - ['inputs'], - ['outputs'], - ('->b', self._BiasWeight('b', dim)), - ('inputs,b->outputs', self._Fn('bias', - fn=lambda inputs, b: inputs + b)), - ) - - def Feedforward(self, name): - p = self.params - - ff_list = [ - self._LN('ln', p.model_dim, use_fused_layernorm=p.use_fused_layernorm), - self._Linear('linear01', p.model_dim, p.ff_hidden_dim, split_dim=1) - ] - if p.use_bias: - ff_list.append(self._Bias('bias01', p.ff_hidden_dim)) - ff_list += [ - self._Activation('act', p.ff_activation_fn), - self._Dropout('relu_dropout', p.relu_dropout_prob), - self._Linear('linear02', p.ff_hidden_dim, p.model_dim, split_dim=0) - ] - if p.use_bias: - ff_list.append(self._Bias('bias02', p.model_dim)) - ff_list.append(self._Dropout('dropout', p.residual_dropout_prob)) - - sub_list = [ - ('i.vec->after_feedforward', self._Seq('feedforward', *ff_list)), - ('i.vec,after_feedforward->added', self._Add('add', - p.ff_residual_weight)), - ('added,i.paddings->o.vec', self._Pad('pad')), - ('i.paddings->o.paddings', self._Id('id')), - ] - - return self._Graph( - name, - ['i'], # input NestedMap with {vec, paddings} - ['o'], # output NestedMap with {vec, paddings} - *sub_list) - - def _Attention(self, name, is_causal=True): - """Computes self attention with optional stride. - - Args: - name: name of this layer. - is_causal: If true, add cause per_step padding to the attention layer. - - Returns: - A self attention layer params. - """ - p = self.params - tr_atten_p = TransformerAttentionLayer.Params().Set( - name='transformer_atten', - input_dim=p.model_dim, - hidden_dim=p.attention_hidden_dim or p.model_dim, - is_masked=is_causal, - num_heads=p.num_heads, - residual_dropout_prob=p.residual_dropout_prob, - atten_dropout_prob=p.atten_dropout_prob, - fprop_dtype=p.fprop_dtype, - add_unnormalized_input=p.selfatten_add_unnormalized_input, - ) - tr_atten_p.atten_tpl.use_bias = p.use_bias - tr_atten_p.atten_tpl.enable_value_proj = p.selfatten_enable_value_proj - tr_atten_p.atten_tpl.enable_per_dim_scale = p.enable_per_dim_scale - tr_atten_p.atten_tpl.xla_num_partitions = p.xla_num_partitions - if p.deterministic_dropout: - tr_atten_p.dropout_tpl = layers.DeterministicDropoutLayer.Params() - tr_atten_p.atten_p.dropout_tpl = layers.DeterministicDropoutLayer.Params() - - return self._Graph( - name, - ['i'], # input NestedMap with {vec, paddings} - ['o'], # output NestedMap with {vec, paddings} - ('i.vec,i.vec,i.paddings->o.vec,unused_prob', tr_atten_p), - ('i.paddings->o.paddings', self._Id('id'))) - - def TransformerEncoderLayer(self, name, is_causal=True): - """(inputs, paddings) -> (encoded, paddings). - - Args: - name: the string name of the encoder layer params. - is_causal: If true, add cause per_step padding to the attention layer. - - Returns: - A transformer encoder layer params that supports optional stride. - """ - # Hack to be compatible with ckpt generated by self._rep - return self._Seq( - name, - self._Seq('block', self._Attention('self_atten', is_causal=is_causal), - self.Feedforward('ff'))) diff --git a/lingvo/core/batch_major_attention_test.py b/lingvo/core/batch_major_attention_test.py deleted file mode 100644 index 91a96df7..00000000 --- a/lingvo/core/batch_major_attention_test.py +++ /dev/null @@ -1,2796 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for batch_major_attention.""" - -from absl.testing import parameterized -from lingvo import compat as tf -from lingvo.core import attention as tm_attention -from lingvo.core import attention_util -from lingvo.core import base_layer -from lingvo.core import batch_major_attention as attention -from lingvo.core import hyperparams -from lingvo.core import py_utils -from lingvo.core import test_utils -import numpy as np - - -class MultiHeadSelfAttentionTest(test_utils.TestCase, parameterized.TestCase): - """Test attention models.""" - - def _AttentionInputs(self, input_dim=4, dtype=tf.float32): - np.random.seed(6348575) - batch_size = 6 - seq_len = 6 - input_vecs_p = [ - np.random.rand(seq_len, input_dim) for _ in range(batch_size) - ] - input_vecs = tf.stack([tf.constant(x, dtype=dtype) for x in input_vecs_p]) - # pyformat: disable - input_padding_p = [[0, 0, 1, 1, 0, 0], [1, 0, 0, 0, 1, 0], - [0, 0, 1, 0, 1, 0], [0, 0, 1, 1, 0, 0], - [1, 0, 0, 0, 1, 0], [0, 0, 1, 0, 1, 0]] - # pyformat: enable - input_padding = tf.constant(input_padding_p, dtype=dtype) - - return input_vecs, input_padding, input_vecs_p, input_padding_p - - def testDotProductAttention(self): - (input_vecs, input_padding, input_vecs_p, - input_padding_p) = self._AttentionInputs() - p = attention.MultiHeadedAttention.Params().Set( - name='self_atten', - input_dim=4, - hidden_dim=4, - enable_scaling_code_motion=True) - l = p.Instantiate() - - probs, probs_sum = l.AttenProbs( - l.theta, - tf.expand_dims(input_vecs, 2), - tf.expand_dims(input_vecs, 2), - input_padding, - segment_mask=None) - - with self.session(use_gpu=False) as sess: - tf.global_variables_initializer().run() - prob_out = sess.run(tf.squeeze(probs / probs_sum)) - - # Use numpy to perform the same computation to generate expected results. - input_vecs_p = np.array(input_vecs_p) - target_vecs_p = np.transpose(input_vecs_p, (0, 2, 1)) - expected_logit = np.matmul(input_vecs_p, target_vecs_p) - expected_logit = np.transpose(expected_logit, (0, 2, 1)) - elexp = np.exp(expected_logit) - input_padding_p = np.array(input_padding_p) - input_padding_p = np.expand_dims(input_padding_p, axis=1) - input_padding_p = np.tile(input_padding_p, (1, 6, 1)) - elexp *= (1 - input_padding_p) - expected_prob_out = elexp / np.expand_dims(np.sum(elexp, axis=-1), axis=-1) - expected_prob_out = np.reshape(expected_prob_out, (6, 6, 6)) - self.assertAllClose(expected_prob_out, prob_out) - - def testMultiHeadedAttentionDotProduct(self): - # input_batch:6, seq_len:6. Test n = 2 case. - with self.session(use_gpu=True) as sess: - input_vecs, input_padding, _, _ = self._AttentionInputs() - p = attention.MultiHeadedAttention.Params().Set( - name='self_atten', num_heads=2, input_dim=4, hidden_dim=4) - - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - - l = p.Instantiate() - tf.global_variables_initializer().run() - ctx_vec, _ = l.FProp( - l.theta, - input_vecs, - input_vecs, - input_vecs, - input_padding, - segment_mask=None) - context_vec_out = sess.run(ctx_vec) - context_vec_out = np.reshape(context_vec_out, (6, 24)) - self.assertAllClose( - [27.417763, 31.783672, 19.99568, 23.907103, 21.078259, 28.429199], - np.sum(context_vec_out, axis=1)) - - def testMultiHeadedAttentionDotProductSegmentMask(self): - # input_batch:6, seq_len:6. Test n = 2 case. - with self.session(use_gpu=True) as sess: - input_vecs, input_padding, _, _ = self._AttentionInputs() - p = attention.MultiHeadedAttention.Params().Set( - name='self_atten', - num_heads=2, - input_dim=4, - hidden_dim=4, - packed_input=True) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - - segment_id = tf.zeros([6, 6]) - segment_mask = attention.SegmentMask(segment_id, segment_id) - padding = tf.tile(tf.reshape(input_padding, [6, 1, 1, 6]), [1, 1, 6, 1]) - padding_mask = padding * segment_mask.dtype.max * tf.constant( - -0.7, dtype=segment_mask.dtype) - segment_mask += padding_mask - - l = p.Instantiate() - tf.global_variables_initializer().run() - ctx_vec, _ = l.FProp( - l.theta, - input_vecs, - input_vecs, - input_vecs, - input_padding, - segment_mask=segment_mask) - context_vec_out = sess.run(ctx_vec) - context_vec_out = np.reshape(context_vec_out, (6, 24)) - self.assertAllClose( - [27.417763, 31.783672, 19.99568, 23.907103, 21.078259, 28.429199], - np.sum(context_vec_out, axis=1)) - - -class MultiHeadedAttentionXLOracle: - """Oracle layer used for computing ground truths for MultiHeadedAttention. - - Written in a non-vectorized way. - """ - - def __init__(self, u, v, pos_proj, sinusoid_emb): - """Constructor. - - Args: - u: A numpy ndarray of shape [N, H] - v: A numpy ndarray of shape [N, H] - pos_proj: A numpy ndarray of shape [embed_dim, N, H] - sinusoid_emb: A numpy ndarray of shape [seqlen, emb_dim]. - """ - assert u.shape == v.shape - assert u.shape == pos_proj.shape[1:] - assert sinusoid_emb.shape[-1] == pos_proj.shape[0] - # [N, H] - self._u = u - # [N, H] - self._v = v - # [?, N, H] - self._pos_proj = pos_proj - - self._num_heads = u.shape[0] - self._atten_dim = u.shape[-1] - self._hidden_dim = u.shape[0] * u.shape[-1] - self._sinusoid_emb = sinusoid_emb - - def _GetPositionEnc(self, tgt_t, src_t, head, seqlen): - """Gets positional encoding. - - Args: - tgt_t: A Python int, time step of target seq. - src_t: A Python int, time step of source seq. - head: A Python int, num of heads of the attention. - seqlen: A Python int, sequence length of target/source seq. - - Returns: - A numpy array of shape [head, emb_dim // head]. - """ - # [emb_dim] - sinusoid_enc = self._sinusoid_emb[tgt_t - src_t + seqlen - 1] - return np.einsum('DNH,D->NH', self._pos_proj, sinusoid_enc)[head] - - def AttenProbs(self, key, query, paddings, per_step_padding): - """Computes attention probs in a non vectorized way. - - Args: - key: A numpy ndarray of shape [batch, seqlen, heads, dim]. - query: A numpy ndarray of the same shape as `key`. - paddings: A numpy ndarray of shape [batch, seqlen]. - per_step_padding: A numpy ndarray of shape [batch, seqlen, seqlen]. - - Returns: - A numpy ndarray of shape [batch, query_seqlen, key_seqlen] - """ - - assert query.ndim == 4 - assert paddings.ndim == 2 - assert key.shape == query.shape - - batch, seqlen = query.shape[:2] - tgtlen, srclen = seqlen, seqlen - assert query.shape[2] == self._num_heads - assert query.shape[3] == self._atten_dim - assert paddings.shape == query.shape[:2] - - logits = np.zeros((batch, self._num_heads, tgtlen, srclen)) - probs = np.zeros((batch, self._num_heads, tgtlen, srclen)) - - def Normalize(vec): - expx = np.exp(vec) - expxsum = np.sum(expx, axis=-1) - return expx / expxsum - - # [b, tgtlen, srclen] - paddings = np.broadcast_to( - np.reshape(paddings, (batch, 1, seqlen)), (batch, seqlen, seqlen)) - for b in range(batch): - for h in range(self._num_heads): - for i in range(tgtlen): - for j in range(srclen): - pos_enc = self._GetPositionEnc(i, j, h, seqlen) - logits[b][h][i][j] = ( - np.dot(query[b][i][h], key[b][j][h]) + - np.dot(query[b][i][h], pos_enc) + - np.dot(self._u[h], key[b][j][h]) + np.dot(self._v[h], pos_enc)) - - total_padding = paddings[b][i] + per_step_padding[b][i] - logits[b][h][i] = np.where(total_padding > 0, - np.finfo(np.float32).max * (-0.7), - logits[b][h][i]) - probs[b][h][i] = Normalize(logits[b][h][i]) - return probs - - -def _AttentionInputs(input_dim=4, dtype=tf.float32, is_causal=True): - np.random.seed(6348575) - batch_size = 6 - seq_len = 6 - query_vec_p = [np.random.rand(seq_len, input_dim) for _ in range(batch_size)] - query_vec_p = np.array(query_vec_p).astype(dtype.as_numpy_dtype) - query_vec = tf.convert_to_tensor(query_vec_p) - - memory_vec_p = [np.random.rand(seq_len, input_dim) for _ in range(batch_size)] - memory_vec_p = np.array(memory_vec_p).astype(dtype.as_numpy_dtype) - memory_vec = tf.convert_to_tensor(memory_vec_p) - # pyformat: disable - paddings_p = np.array( - [[0, 0, 1, 1, 1, 1], [0, 1, 1, 1, 1, 1], - [0, 0, 0, 0, 1, 1], [0, 0, 1, 1, 1, 1], - [0, 0, 0, 1, 1, 1], [0, 0, 0, 0, 0, 1]]).astype(dtype.as_numpy_dtype) - paddings = tf.convert_to_tensor(paddings_p) - # causal padding. - if is_causal: - per_step_padding_p = [ - [0, 1, 1, 1, 1, 1], [0, 0, 1, 1, 1, 1], - [0, 0, 0, 1, 1, 1], [0, 0, 0, 0, 1, 1], - [0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 0]] - else: - per_step_padding_p = np.zeros((seq_len, seq_len)) - per_step_padding_p = [per_step_padding_p for _ in range(batch_size)] - per_step_padding_p = np.array(per_step_padding_p).astype(dtype.as_numpy_dtype) - per_step_padding = tf.convert_to_tensor(per_step_padding_p) - - # pyformat: enable - return (query_vec, memory_vec, paddings, per_step_padding, query_vec_p, - memory_vec_p, paddings_p, per_step_padding_p) - - -class MultiHeadedAttentionTest(test_utils.TestCase, parameterized.TestCase): - """Test dot-product multiheaded attention.""" - - def _AttentionExtendStepInputs(self, - input_dim=4, - num_heads=2, - dtype=tf.float32): - np.random.seed(6348575) - batch_size = 6 - seq_len = 6 - query_vec_p = [np.random.rand(1, input_dim) for _ in range(batch_size)] - query_vec = tf.stack([tf.constant(x, dtype=dtype) for x in query_vec_p]) - # pyformat: disable - per_step_padding_p = [[0, 1, 1, 1, 1, 1]] - per_step_padding_p = [per_step_padding_p for _ in range(batch_size)] - # pyformat: enable - per_step_padding = tf.stack( - [tf.constant(x, dtype=dtype) for x in per_step_padding_p]) - source_vecs = tf.constant( - np.random.normal( - 0.1, 0.5, [seq_len, batch_size, num_heads, input_dim // num_heads]), - dtype=dtype) - source_ctxs = tf.constant( - np.random.normal( - 0.1, 0.5, [seq_len, batch_size, num_heads, input_dim // num_heads]), - dtype=dtype) - cached_states = py_utils.NestedMap(key=source_vecs, value=source_ctxs) - return query_vec, cached_states, per_step_padding - - def testAttenProbs(self): - (query_vec, key_vec, paddings, per_step_padding, query_vec_p, key_vec_p, - paddings_p, per_step_padding_p) = _AttentionInputs() - p = attention.MultiHeadedAttention.Params().Set( - name='atten', - input_dim=4, - hidden_dim=4, - enable_scaling_code_motion=True) - l = p.Instantiate() - probs, probs_sum = l.AttenProbs( - l.theta, - tf.expand_dims(query_vec, 2), - tf.expand_dims(key_vec, 2), - paddings, - segment_mask=None, - per_step_padding=per_step_padding) - - with self.session(use_gpu=False) as sess: - tf.global_variables_initializer().run() - prob_out = sess.run(tf.squeeze(probs / probs_sum)) - - # Use numpy to perform the same computation to generate expected results. - query_vec_p = np.array(query_vec_p) - key_vec_p = np.array(key_vec_p) - key_vec_p = np.transpose(key_vec_p, (0, 2, 1)) - expected_logit = np.matmul(query_vec_p, key_vec_p) - paddings_p = np.array(paddings_p) - paddings_p = np.expand_dims(paddings_p, axis=1) - paddings_p = np.tile(paddings_p, (1, 6, 1)) - per_step_padding_p = np.array(per_step_padding_p) - paddings_p = 1.0 * np.logical_or(paddings_p, per_step_padding_p) - elexp = np.exp(expected_logit) - elexp *= (1.0 - paddings_p) - elexp += 1e-9 - expected_prob_out = elexp / np.expand_dims(np.sum(elexp, axis=-1), axis=-1) - expected_prob_out = np.reshape(expected_prob_out, (6, 6, 6)) - self.assertAllClose(expected_prob_out, prob_out) - - def testFPropCrossAttention(self): - # input_batch:6, seq_len:6. Test n = 2 case. - with self.session(use_gpu=True) as sess: - query_vec, memory_vec, paddings, per_step_padding, _, _, _, _ = ( - _AttentionInputs()) - p = attention.MultiHeadedAttention.Params().Set( - name='cross_atten', num_heads=2, input_dim=4, hidden_dim=4) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - tf.global_variables_initializer().run() - ctx_vec, _ = l.FProp( - l.theta, - query_vec, - memory_vec, - memory_vec, - paddings, - segment_mask=None, - per_step_padding=per_step_padding) - context_vec_out = sess.run(ctx_vec) - context_vec_out = np.reshape(context_vec_out, (6, 24)) - self.assertAllClose( - [24.624561, 27.805634, 23.358835, 11.085404, 27.165989, 23.750813], - np.sum(context_vec_out, axis=1)) - - @parameterized.named_parameters( - { - 'testcase_name': '_short_seq', - 'use_short_seq_opt': True, - }, { - 'testcase_name': '_long_seq', - 'use_short_seq_opt': False, - }) - def testExtendStepSelfAttention(self, use_short_seq_opt): - # input_batch:6, seq_len:6, query_len: 1. Test n = 2 case. - with self.session(use_gpu=True) as sess: - query_vec, cached_states, per_step_padding = self._AttentionExtendStepInputs( - ) - p = attention.MultiHeadedAttention.Params().Set( - name='atten', num_heads=2, input_dim=4, hidden_dim=4) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - tf.global_variables_initializer().run() - ctx_vec, updated_states = l.ExtendStep(l.theta, query_vec, cached_states, - None, None, per_step_padding, 0, - use_short_seq_opt) - context_vec_out = sess.run(ctx_vec) - new_source_vecs = sess.run(updated_states.key) - context_vec_out = np.reshape(context_vec_out, (6, 4)) - self.assertAllClose( - [5.381485, 5.384035, 4.493689, 3.544395, 3.424472, 3.311054], - np.sum(context_vec_out, axis=1)) - new_source_vecs = np.reshape(new_source_vecs, (6, 24)) - self.assertAllClose( - [4.116683, 1.340482, 1.065773, 1.035415, 4.928454, 3.161165], - np.sum(new_source_vecs, axis=1)) - - @parameterized.named_parameters({ - 'testcase_name': '_long_seq', - 'use_short_seq_opt': False, - }) - def testExtendStepAsyncTimeStepSelfAttention(self, use_short_seq_opt): - # input_batch:6, seq_len:6, query_len: 1. Test n = 2 case. - with self.session(use_gpu=True) as sess: - query_vec, cached_states, per_step_padding = self._AttentionExtendStepInputs( - ) - p = attention.MultiHeadedAttention.Params().Set( - name='atten', num_heads=2, input_dim=4, hidden_dim=4) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - - allzero_time_step = tf.constant([0] * 6) - time_step = tf.constant([0, 1, 2, 3, 4, 5]) - l = p.Instantiate() - tf.global_variables_initializer().run() - ctx_vec, updated_states = l.ExtendStep(l.theta, query_vec, cached_states, - None, None, per_step_padding, 0, - use_short_seq_opt) - ctx_vec_async, updated_states_async = l.ExtendStep( - l.theta, query_vec, cached_states, None, None, per_step_padding, - allzero_time_step, use_short_seq_opt) - - context_vec_out = sess.run(ctx_vec) - new_source_vecs = sess.run(updated_states.key) - context_vec_out_async = sess.run(ctx_vec_async) - new_source_vecs_async = sess.run(updated_states_async.key) - - self.assertAllClose( - np.sum(context_vec_out, axis=1), - np.sum(context_vec_out_async, axis=1)) - self.assertAllClose( - np.sum(new_source_vecs, axis=1), - np.sum(new_source_vecs_async, axis=1)) - - ctx_vec_async, updated_states_async = l.ExtendStep( - l.theta, query_vec, cached_states, None, None, per_step_padding, - time_step, use_short_seq_opt) - _, updated_states_step1 = l.ExtendStep(l.theta, query_vec, cached_states, - None, None, per_step_padding, 1, - use_short_seq_opt) - - context_vec_out_async = sess.run(ctx_vec_async) - new_source_vecs_async = sess.run(updated_states_async.key) - - new_source_vecs_async_step1 = sess.run(updated_states_step1.key) - - context_vec_out_async = np.reshape(context_vec_out_async, (6, 4)) - self.assertAllClose( - [5.381485, -1.943824, 2.214111, 0.840045, -0.939259, 0.752783], - np.sum(context_vec_out_async, axis=1)) - # Updated status are the same at step 0. - self.assertAllClose(new_source_vecs_async[0][0], new_source_vecs[0][0]) - self.assertAllClose(new_source_vecs_async[1][1], - new_source_vecs_async_step1[1][1]) - - @parameterized.named_parameters({ - 'testcase_name': '_long_seq', - 'use_short_seq_opt': False, - }) - def testMultipleExtendStepAsyncTimeStepSelfAttention(self, use_short_seq_opt): - # input_batch:6, seq_len:6, query_len: 1. Test n = 2 case. - num_heads, input_dim, hidden_dim, batch, seqlen = 2, 4, 4, 6, 6 - with self.session(use_gpu=True): - tf.random.set_seed(12345) - (query_vec, _, paddings, _, _, _, _, _) = _AttentionInputs() - p = attention.MultiHeadedAttention.Params().Set( - name='atten', - num_heads=num_heads, - input_dim=input_dim, - hidden_dim=hidden_dim) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - - tf.global_variables_initializer().run() - - # Verify ExtendStep() via compare N ExtendStep() with one FProp() call on - # a seq with length N. - per_step_padding = 1 - tf.linalg.band_part( - tf.ones((seqlen, seqlen)), -1, 0) - per_step_padding = tf.stack([per_step_padding] * batch) - dims_per_head = hidden_dim // num_heads - - def _ResetCachedStates(): - cached_source_vecs = tf.constant( - np.random.normal(0.1, 0.5, - [seqlen, batch, num_heads, dims_per_head]), - dtype=tf.float32) - cached_source_ctxs = tf.constant( - np.random.normal(0.1, 0.5, - [seqlen, batch, num_heads, dims_per_head]), - dtype=tf.float32) - cached_states = py_utils.NestedMap( - key=cached_source_vecs, value=cached_source_ctxs) - return cached_states - - encoded_all = [] - cached_states = _ResetCachedStates() - for i in range(seqlen): - per_step_paddings = 1. - tf.cast( - tf.sequence_mask([i + 1] * batch, seqlen), tf.float32) - per_step_paddings = tf.expand_dims(per_step_paddings, 1) - encoded, cached_states = l.ExtendStep(l.theta, query_vec[:, i:i + 1, :], - cached_states, paddings, None, - per_step_paddings, i) - # [batch, 1, dims_per_head] - encoded_all.append(encoded) - - encoded_all_async = [] - cached_states = _ResetCachedStates() - for i in range(seqlen): - # Sample 1 to batch -1 time step are synchoronized: 1 -> Seqlen - # Sample batch, the time step are [0, 0, 0, 1, .., Seqlen-2] - index = i - 3 if i > 2 else 0 - new_query_vec = tf.concat([ - query_vec[:(batch - 1), i:i + 1, :], query_vec[(batch - 1):, - index:index + 1, :] - ], - axis=0) - time_step = tf.constant([i] * (batch - 1) + [index], dtype=tf.int32) - per_step_paddings = 1. - tf.cast( - tf.sequence_mask([i + 1] * - (batch - 1) + [index + 1], seqlen), tf.float32) - per_step_paddings = tf.expand_dims(per_step_paddings, 1) - encoded, cached_states = l.ExtendStep(l.theta, new_query_vec, - cached_states, paddings, None, - per_step_paddings, time_step) - # [batch, 1, dims_per_head] - encoded_all_async.append(encoded) - # [batch, T, dims_per_head] - actual_ctx_vec = tf.concat(encoded_all, axis=1) - actual_ctx_vec_async = tf.concat(encoded_all_async, axis=1) - - self.assertAllClose(actual_ctx_vec_async.eval()[:-1], - actual_ctx_vec.eval()[:-1]) - # Sample batch move 3 step slower than the synchronized version. - self.assertAllClose(actual_ctx_vec_async.eval()[-1][3:], - actual_ctx_vec.eval()[-1][:3]) - - -class MultiSourceMultiHeadedAttentionTest(MultiHeadedAttentionTest): - - def testAttenProbs(self): - (query_vec, key_vec, paddings, per_step_padding, query_vec_p, key_vec_p, - paddings_p, per_step_padding_p) = _AttentionInputs() - - # Two-source attention. - mha_params = attention.MultiHeadedAttention.Params().Set( - name='atten', - input_dim=4, - hidden_dim=4, - enable_scaling_code_motion=True) - atten_merger_p = tm_attention.MergerLayer.Params().Set( - params_init=py_utils.WeightInit.Uniform(0.04), - merger_op='concat', # concatenate attention - pre_proj_input_dims=[4, 4], - pre_proj_output_dims=[4, 4]) - params = attention.MultiSourceAttention.Params().Set( - name='two_source_atten', - input_dim=4, - hidden_dim=4, - source_atten_tpls=[('src_1', mha_params), - ('src_2', mha_params.Copy().Set(name='atten2'))], - primary_source_key='src_1', - atten_merger_tpl=atten_merger_p) - l = params.Instantiate() - - probs, probs_sum = l.AttenProbs( - l.theta, - tf.expand_dims(query_vec, 2), - py_utils.NestedMap({ - 'src_1': tf.expand_dims(key_vec, 2), - 'src_2': tf.expand_dims(key_vec, 2) - }), - py_utils.NestedMap({ - 'src_1': paddings, - 'src_2': paddings - }), - segment_mask=None, - per_step_padding=per_step_padding) - - with self.session(use_gpu=False) as sess: - tf.global_variables_initializer().run() - prob_out = sess.run(tf.squeeze(probs / probs_sum)) - - # Use numpy to perform the same computation to generate expected results. - query_vec_p = np.array(query_vec_p) - key_vec_p = np.array(key_vec_p) - key_vec_p = np.transpose(key_vec_p, (0, 2, 1)) - expected_logit = np.matmul(query_vec_p, key_vec_p) - paddings_p = np.array(paddings_p) - paddings_p = np.expand_dims(paddings_p, axis=1) - paddings_p = np.tile(paddings_p, (1, 6, 1)) - per_step_padding_p = np.array(per_step_padding_p) - paddings_p = 1.0 * np.logical_or(paddings_p, per_step_padding_p) - elexp = np.exp(expected_logit) - elexp *= (1.0 - paddings_p) - elexp += 1e-9 - expected_prob_out = elexp / np.expand_dims(np.sum(elexp, axis=-1), axis=-1) - expected_prob_out = np.reshape(expected_prob_out, (6, 6, 6)) - self.assertAllClose(expected_prob_out, prob_out) - - def testFPropCrossAttention(self): - # input_batch:6, seq_len:6. Test n = 2 case. - with self.session(use_gpu=True) as sess: - query_vec, memory_vec, paddings, per_step_padding, _, _, _, _ = ( - _AttentionInputs()) - mha_params = attention.MultiHeadedAttention.Params().Set( - name='cross_atten', num_heads=2, input_dim=4, hidden_dim=4) - mha_params.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - atten_merger_p = tm_attention.MergerLayer.Params().Set( - params_init=py_utils.WeightInit.Uniform(0.04), - merger_op='concat', # concatenate attention - pre_proj_input_dims=[4, 4], - pre_proj_output_dims=[4, 4]) - # Two-source attention. - p = attention.MultiSourceAttention.Params().Set( - name='two_source_atten', - input_dim=4, - hidden_dim=4, - source_atten_tpls=[('src_1', mha_params), - ('src_2', mha_params.Copy().Set(name='atten2'))], - primary_source_key='src_1', - atten_merger_tpl=atten_merger_p) - l = p.Instantiate() - - tf.global_variables_initializer().run() - ctx_vec, _ = l.FProp( - l.theta, - query_vec, - py_utils.NestedMap({ - 'src_1': memory_vec, - 'src_2': memory_vec - }), - py_utils.NestedMap({ - 'src_1': memory_vec, - 'src_2': memory_vec - }), - py_utils.NestedMap({ - 'src_1': paddings, - 'src_2': paddings - }), - segment_mask=None, - per_step_padding=per_step_padding) - context_vec_out = sess.run(ctx_vec) - context_vec_out = np.reshape(context_vec_out, (12, 24)) - self.assertAllClose([ - 5.6162043, 5.0109887, 6.0565553, 6.0565553, 4.5718207, 5.253615, - 2.0541124, 2.490314, 6.049119, 5.5567484, 4.409875, 5.8939424 - ], np.sum(context_vec_out, axis=1)) - - -class MultiHeadedAttentionXLTest(test_utils.TestCase, parameterized.TestCase): - """Test dot-product multiheaded attention.""" - - def _AttentionExtendStepInputs(self, - input_dim, - batch_size, - seq_len, - dtype=tf.float32): - np.random.seed(6348575) - query_vec_p = [ - np.random.rand(seq_len, input_dim) for _ in range(batch_size) - ] - query_vec = tf.stack([tf.constant(x, dtype=dtype) for x in query_vec_p]) - paddings_p = [[0] * seq_len] * batch_size - paddings = tf.constant(paddings_p, dtype=dtype) - return query_vec, paddings - - @parameterized.named_parameters(('OneHead', 1), ('OneHeadCausal', 1, True), - ('MultiHead', 2), - ('MultiHeadCausal', 2, True)) - def testAttenProbs(self, num_heads, is_causal=False): - batch, slen = 6, 6 - atten_dim = 4 - input_dim = num_heads * atten_dim - (input_vecs, _, input_padding, per_step_padding, input_vecs_p, _, - input_padding_p, per_step_padding_p) = _AttentionInputs( - input_dim=input_dim, is_causal=is_causal) - p = attention.MultiHeadedAttentionXL.Params().Set( - name='self_atten', - input_dim=input_dim, - num_heads=num_heads, - hidden_dim=input_dim, - rel_pos_emb_dim=input_dim, - enable_scaling_code_motion=True) - - l = p.Instantiate() - query = tf.reshape(input_vecs, (batch, slen, num_heads, atten_dim)) - probs, probs_sum = l.AttenProbs( - l.theta, - query, - query, - input_padding, - segment_mask=None, - per_step_padding=per_step_padding) - - # [1, 2 * slen - 1] - positions = np.expand_dims(np.arange(-(slen - 1), slen), 0) - sinusoid_emb = l.pos_emb.FPropWithPosition(l.theta.pos_emb, - tf.convert_to_tensor(positions)) - # [ 2 * slen - 1, emb_dim=input_dim] - sinusoid_emb = tf.squeeze(sinusoid_emb, 0) - - with self.session(use_gpu=False) as sess: - tf.global_variables_initializer().run() - u, v, pos_proj = sess.run([l.vars.u, l.vars.v, l.pos_proj.vars.w]) - actual_probs = sess.run(probs / probs_sum) - sinusoid_emb_p = sess.run(sinusoid_emb) - - # Compute ground truth with oracle class. - - # Use numpy to perform the same computation to generate expected results. - # [B, tgt_t, H] - input_vecs_p = np.array(input_vecs_p) - # [B, tgt_t, N, H] - input_vecs_p = np.reshape(input_vecs_p, (batch, slen, num_heads, atten_dim)) - input_padding_p = np.array(input_padding_p) - oracle = MultiHeadedAttentionXLOracle(u, v, pos_proj, sinusoid_emb_p) - expected_probs = oracle.AttenProbs(input_vecs_p, input_vecs_p, - input_padding_p, per_step_padding_p) - self.assertAllClose(expected_probs, actual_probs) - - def testFPropSelfAttention(self): - # input_batch:6, seq_len:6. Test n = 2 case. - with self.session(use_gpu=True) as sess: - query_vec, _, paddings, _, _, _, _, _ = _AttentionInputs() - num_heads, input_dim, hidden_dim = 2, 4, 4 - p = attention.MultiHeadedAttentionXL.Params().Set( - name='self_atten', - num_heads=num_heads, - input_dim=input_dim, - hidden_dim=hidden_dim, - rel_pos_emb_dim=num_heads * hidden_dim) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - - l = p.Instantiate() - ctx_vec, _ = l.FPropDefaultTheta( - query_vec, query_vec, query_vec, paddings, segment_mask=None) - - tf.global_variables_initializer().run() - context_vec_out = sess.run(ctx_vec) - context_vec_out = np.reshape(context_vec_out, (6, 24)) - self.assertAllClose( - [32.33513, 28.584404, 20.54517, 23.407812, 18.616188, 24.212755], - np.sum(context_vec_out, axis=1)) - - def testExtendStepAsyncTimeStepSelfAttention(self): - num_heads, input_dim, hidden_dim, batch, seqlen = 2, 4, 4, 6, 6 - emb_dim = 4 - with self.session(use_gpu=True): - tf.random.set_seed(12345) - query_vec, paddings = self._AttentionExtendStepInputs( - input_dim, batch, seqlen) - p = attention.MultiHeadedAttentionXL.Params().Set( - name='atten', - num_heads=num_heads, - input_dim=input_dim, - hidden_dim=hidden_dim, - rel_pos_emb_dim=emb_dim, - random_seed=0) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - - tf.global_variables_initializer().run() - - # Verify ExtendStep() via compare N ExtendStep() with one FProp() call on - # a seq with length N. - per_step_padding = 1 - tf.linalg.band_part( - tf.ones((seqlen, seqlen)), -1, 0) - per_step_padding = tf.stack([per_step_padding] * batch) - dims_per_head = hidden_dim // num_heads - - def _ResetCachedStates(): - cached_source_vecs = tf.constant( - np.random.normal(0.1, 0.5, - [seqlen, batch, num_heads, dims_per_head]), - dtype=tf.float32) - cached_source_ctxs = tf.constant( - np.random.normal(0.1, 0.5, - [seqlen, batch, num_heads, dims_per_head]), - dtype=tf.float32) - cached_states = py_utils.NestedMap( - key=cached_source_vecs, value=cached_source_ctxs) - return cached_states - - encoded_all = [] - cached_states = _ResetCachedStates() - for i in range(seqlen): - per_step_paddings = 1. - tf.cast( - tf.sequence_mask([i + 1] * batch, seqlen), tf.float32) - per_step_paddings = tf.expand_dims(per_step_paddings, 1) - encoded, cached_states = l.ExtendStep(l.theta, query_vec[:, i:i + 1, :], - cached_states, paddings, None, - per_step_paddings, i) - # [batch, 1, dims_per_head] - encoded_all.append(encoded) - - encoded_all_async = [] - cached_states = _ResetCachedStates() - for i in range(seqlen): - # Sample 1 to batch -1 time step are synchoronized: 1 -> Seqlen - # Sample batch, the time step are [0, 0, 0, 1, .., Seqlen-2] - index = i - 3 if i > 2 else 0 - new_query_vec = tf.concat([ - query_vec[:(batch - 1), i:i + 1, :], query_vec[(batch - 1):, - index:index + 1, :] - ], - axis=0) - time_step = tf.constant([i] * (batch - 1) + [index], dtype=tf.int32) - per_step_paddings = 1. - tf.cast( - tf.sequence_mask([i + 1] * - (batch - 1) + [index + 1], seqlen), tf.float32) - per_step_paddings = tf.expand_dims(per_step_paddings, 1) - encoded, cached_states = l.ExtendStep(l.theta, new_query_vec, - cached_states, paddings, None, - per_step_paddings, time_step) - # [batch, 1, dims_per_head] - encoded_all_async.append(encoded) - # [batch, T, dims_per_head] - actual_ctx_vec = tf.concat(encoded_all, axis=1) - actual_ctx_vec_async = tf.concat(encoded_all_async, axis=1) - - self.assertAllClose(actual_ctx_vec_async.eval()[:-1], - actual_ctx_vec.eval()[:-1]) - # Sample batch move 3 step slower than the synchronized version. - self.assertAllClose(actual_ctx_vec_async.eval()[-1][3:], - actual_ctx_vec.eval()[-1][:3]) - - def testExtendStepSelfAttention(self): - num_heads, input_dim, hidden_dim, batch, seqlen = 2, 4, 4, 6, 6 - emb_dim = 4 - with self.session(use_gpu=True): - tf.random.set_seed(12345) - query_vec, paddings = self._AttentionExtendStepInputs( - input_dim, batch, seqlen) - p = attention.MultiHeadedAttentionXL.Params().Set( - name='atten', - num_heads=num_heads, - input_dim=input_dim, - hidden_dim=hidden_dim, - rel_pos_emb_dim=emb_dim, - random_seed=0) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - tf.global_variables_initializer().run() - - # Verify ExtendStep() via compare N ExtendStep() with one FProp() call on - # a seq with length N. - per_step_padding = 1 - tf.linalg.band_part( - tf.ones((seqlen, seqlen)), -1, 0) - per_step_padding = tf.stack([per_step_padding] * batch) - expected_ctx_vec, _ = l.FPropDefaultTheta( - query_vec, - query_vec, - query_vec, - paddings, - segment_mask=None, - per_step_padding=per_step_padding) - dims_per_head = hidden_dim // num_heads - cached_source_vecs = tf.constant( - np.random.normal(0.1, 0.5, [seqlen, batch, num_heads, dims_per_head]), - dtype=tf.float32) - cached_source_ctxs = tf.constant( - np.random.normal(0.1, 0.5, [seqlen, batch, num_heads, dims_per_head]), - dtype=tf.float32) - cached_states = py_utils.NestedMap( - key=cached_source_vecs, value=cached_source_ctxs) - - encoded_all = [] - for i in range(seqlen): - per_step_paddings = 1. - tf.cast( - tf.sequence_mask([i + 1] * batch, seqlen), tf.float32) - per_step_paddings = tf.expand_dims(per_step_paddings, 1) - encoded, cached_states = l.ExtendStep(l.theta, query_vec[:, i:i + 1, :], - cached_states, paddings, None, - per_step_paddings, i) - # [batch, 1, dims_per_head] - encoded_all.append(encoded) - # [batch, T, dims_per_head] - actual_ctx_vec = tf.concat(encoded_all, axis=1) - self.assertAllClose(expected_ctx_vec.eval(), actual_ctx_vec.eval()) - - -class MultiHeadedAttentionRPEOracle: - """Computes ground truths for MultiHeadedfAttentionRPE. - - Written in a non-vectorized way. - """ - - def __init__(self, num_heads, key_embs, value_embs): - """Constructor. - - Args: - num_heads: A Python int. - key_embs: A numpy array of shape [2 * radius + 1, hidden_dim] - value_embs: A numpy array of shape [2 * radius + 1, hidden_dim] - """ - assert key_embs.shape == value_embs.shape - self._num_heads = num_heads - self._hidden_dim = key_embs.shape[-1] - self._atten_dim = self._hidden_dim // self._num_heads - assert self._atten_dim * self._num_heads == self._hidden_dim - - self._key_embs = np.reshape( - key_embs, [key_embs.shape[0], self._num_heads, self._atten_dim]) - self._value_embs = np.reshape( - value_embs, [value_embs.shape[0], self._num_heads, self._atten_dim]) - self._radius = key_embs.shape[0] // 2 - - def _GetEmb(self, tgt_t, src_t, head, emb_wt): - radius = self._radius - distance = np.clip(src_t - tgt_t, -radius, radius) - return emb_wt[distance][head] - - def GetKeyEmb(self, tgt_t, src_t, head): - return self._GetEmb(tgt_t, src_t, head, self._key_embs) - - def GetValueEmb(self, tgt_t, src_t, head): - return self._GetEmb(tgt_t, src_t, head, self._value_embs) - - def AttenProbs(self, key, query, paddings): - assert query.ndim == 4 - assert paddings.ndim == 2 - assert key.shape == query.shape - - batch, seqlen = query.shape[:2] - tgtlen, srclen = seqlen, seqlen - assert query.shape[2] == self._num_heads - assert query.shape[3] == self._atten_dim - assert paddings.shape == query.shape[:2] - - # [B, N, T, T] - logits = np.zeros((batch, self._num_heads, tgtlen, srclen)) - # [B, N, T, T] - probs = np.zeros((batch, self._num_heads, tgtlen, srclen)) - - paddings = np.broadcast_to( - np.reshape(paddings, (batch, 1, 1, seqlen)), - (batch, self._num_heads, seqlen, seqlen)) - - def Normalize(vec): - expx = np.exp(vec) - expxsum = np.sum(expx, axis=-1) - return expx / expxsum - - for b in range(batch): - for h in range(self._num_heads): - for i in range(tgtlen): - for j in range(srclen): - logits[b][h][i][j] = np.dot(query[b][i][h], - key[b][j][h] + self.GetKeyEmb(i, j, h)) - logits[b][h][i] = np.where(paddings[b][h][i] > 0, - np.finfo(np.float32).max * (-0.7), - logits[b][h][i]) - probs[b][h][i] = Normalize(logits[b][h][i]) - return probs - - def AttenContext(self, probs, values): - assert probs.ndim == 4 - assert values.ndim == 4 - - assert probs.shape[0] == values.shape[0] # batch - assert probs.shape[1] == values.shape[2] # head - assert probs.shape[2] == values.shape[1] # tgtlen - assert probs.shape[3] == probs.shape[2] # slen - assert values.shape[-1] == self._atten_dim - - batch, _, tgtlen, srclen = probs.shape - # [B, N, T, H] - ctx = np.zeros((batch, self._num_heads, tgtlen, self._atten_dim)) - for b in range(batch): - for h in range(self._num_heads): - for i in range(tgtlen): - for j in range(srclen): - ctx[b][h][i] += probs[b][h][i][j] * ( - values[b][j][h] + self.GetValueEmb(i, j, h)) - # [B, T, N, H] - return np.transpose(ctx, (0, 2, 1, 3)) - - -class MultiHeadedAttentionRPETest(test_utils.TestCase, parameterized.TestCase): - - @parameterized.named_parameters(('OneHead', 1), ('MultiHead', 2)) - def testAttenProbs(self, num_heads): - batch, slen = 6, 6 - atten_dim = 4 - radius = 3 - input_dim = num_heads * atten_dim - (input_vecs, _, input_padding, _, input_vecs_p, _, input_padding_p, - _) = _AttentionInputs(input_dim=input_dim) - p = attention.MultiHeadedAttentionRPE.Params().Set( - name='self_atten', - input_dim=input_dim, - num_heads=num_heads, - hidden_dim=input_dim, - rel_pos_radius=radius, - enable_scaling_code_motion=True) - - l = p.Instantiate() - query = tf.reshape(input_vecs, (batch, slen, num_heads, atten_dim)) - probs, probs_sum = l.AttenProbs( - l.theta, query, query, input_padding, segment_mask=None) - - with self.session(use_gpu=False) as sess: - tf.global_variables_initializer().run() - # [radius * 2 + 1, hidden_dim], [B, tgt_t, src_t] - key_emb, value_emb, actual_probs = sess.run( - [l.key_emb.vars.w, l.value_emb.vars.w, probs / probs_sum]) - - oracle = MultiHeadedAttentionRPEOracle(num_heads, key_emb, value_emb) - - # Use numpy to perform the same computation to generate expected results. - # [B, tgt_t, N, H] - input_vecs_p = np.reshape(input_vecs_p, (batch, slen, num_heads, atten_dim)) - expected_probs = oracle.AttenProbs(input_vecs_p, input_vecs_p, - input_padding_p) - self.assertAllClose(expected_probs, actual_probs) - - @parameterized.named_parameters(('OneHead', 1), ('MultiHead', 2)) - def testAttenContext(self, num_heads): - batch, slen = 6, 6 - atten_dim = 4 - radius = 3 - input_dim = num_heads * atten_dim - (input_vecs, _, _, _, input_vecs_p, _, _, - _) = _AttentionInputs(input_dim=input_dim) - p = attention.MultiHeadedAttentionRPE.Params().Set( - name='self_atten', - input_dim=input_dim, - num_heads=num_heads, - hidden_dim=input_dim, - rel_pos_radius=radius) - - l = p.Instantiate() - probs = np.random.rand(batch, num_heads, slen, slen).astype(np.float32) - probs = np.exp(probs) / np.sum(np.exp(probs), axis=-1, keepdims=True) - ctx = l._AttenContext( - l.theta, tf.convert_to_tensor(probs), - tf.reshape(input_vecs, (batch, slen, num_heads, atten_dim))) - - with self.session(use_gpu=False) as sess: - tf.global_variables_initializer().run() - key_emb, value_emb, actual_ctx = sess.run( - [l.key_emb.vars.w, l.value_emb.vars.w, ctx]) - - oracle = MultiHeadedAttentionRPEOracle(num_heads, key_emb, value_emb) - - # [B, tgt_t, N, H] - input_vecs_p = np.reshape(input_vecs_p, (batch, slen, num_heads, atten_dim)) - expected_ctx = oracle.AttenContext(probs, input_vecs_p) - self.assertAllClose(expected_ctx, actual_ctx) - - @parameterized.named_parameters(('OneHead', 1), ('MultiHead', 2)) - def testAttenLogitsOneStep(self, num_heads): - batch, slen = 6, 6 - atten_dim = 4 - radius = 3 - input_dim = num_heads * atten_dim - (input_vecs, _, _, _, _, _, _, _) = _AttentionInputs( - input_dim=input_dim, is_causal=True) - p = attention.MultiHeadedAttentionRPE.Params().Set( - name='self_atten', - input_dim=input_dim, - num_heads=num_heads, - hidden_dim=input_dim, - rel_pos_radius=radius) - - l = p.Instantiate() - # [B, T, N, H] - query = tf.reshape(input_vecs, (batch, slen, num_heads, atten_dim)) - - # Causal self attention. - # [B, N, T, S] - logits = l._AttenLogits( - l.theta, - query, - query, - ) - - one_step_logits = [] - # [S=T, B, N, H] - key = tf.transpose(query, [1, 0, 2, 3]) - for i in range(slen): - local_logits = l._AttenLogitsOneStep(l.theta, query[:, i, :, :], key, i) - one_step_logits.append(local_logits) - # [T, S, B, N] - stacked_logits = tf.stack(one_step_logits) - stacked_logits = tf.transpose(stacked_logits, [2, 3, 0, 1]) - - with self.session(use_gpu=False) as sess: - tf.global_variables_initializer().run() - expected_logits, actual_logits = sess.run([logits, stacked_logits]) - self.assertAllClose(expected_logits, actual_logits) - - @parameterized.named_parameters(('OneHead', 1), ('MultiHead', 2)) - def testAttenContextsOneStep(self, num_heads): - batch, slen = 6, 6 - atten_dim = 4 - radius = 3 - input_dim = num_heads * atten_dim - (input_vecs, _, _, per_step_padding, _, _, _, _) = _AttentionInputs( - input_dim=input_dim, is_causal=True) - p = attention.MultiHeadedAttentionRPE.Params().Set( - name='self_atten', - input_dim=input_dim, - num_heads=num_heads, - hidden_dim=input_dim, - rel_pos_radius=radius) - - l = p.Instantiate() - # [B, N, T, S=T] - # Make causal attention probs. - probs = np.random.rand(batch, num_heads, slen, slen).astype(np.float32) - per_step_padding = 1 - np.tril(np.ones((slen, slen))).astype(np.float32) - probs *= per_step_padding - # Normalize - probs = np.exp(probs) / np.sum(np.exp(probs), axis=-1, keepdims=True) - - # Causal self attention. - # [B, N, T, S] - ctx = l._AttenContext( - l.theta, tf.convert_to_tensor(probs), - tf.reshape(input_vecs, (batch, slen, num_heads, atten_dim))) - - one_step_ctx = [] - # [B, T, N, H] -> [S=T, B, N, H] - value = tf.reshape(input_vecs, (batch, slen, num_heads, atten_dim)) - value = tf.transpose(value, [1, 0, 2, 3]) - for i in range(slen): - # [B, N, S] - local_prob = probs[:, :, i, :] - # [S, B, N] - local_prob = tf.transpose(local_prob, [2, 0, 1]) - # [B, N, H] - local_ctx = l._AttenContextOneStep(l.theta, local_prob, value, i) - one_step_ctx.append(local_ctx) - # [T, B, N, H] - stacked_ctx = tf.stack(one_step_ctx) - stacked_ctx = tf.transpose(stacked_ctx, [1, 0, 2, 3]) - - with self.session(use_gpu=False) as sess: - tf.global_variables_initializer().run() - expected_ctx, actual_ctx = sess.run([ctx, stacked_ctx]) - self.assertAllClose(expected_ctx, actual_ctx) - - -class LocalSelfAttentionTest(test_utils.TestCase, parameterized.TestCase): - """Test local causual self attention.""" - - def _LocalCasualPadding(self, b, t, l, r): - padding = np.ones((b, t, t)) - for i in range(t): - padding[:, i, max(0, i - l + 1):i + r + 1] = 0 - return tf.constant(padding, dtype=tf.float32) - - @parameterized.named_parameters( - { - 'testcase_name': 'block_size_unspecified', - 'block_size': None, - 'left_context': 4, - 'right_context': 1 - }, { - 'testcase_name': 'left_context_only', - 'block_size': 3, - 'left_context': 4, - 'right_context': 0, - }, { - 'testcase_name': 'block_longer_than_sequence', - 'block_size': 10, - 'left_context': 7, - 'right_context': 0, - }, { - 'testcase_name': 'pos_emb_left_context_only', - 'block_size': 3, - 'left_context': 4, - 'right_context': 0, - 'pos_emb_dim': 8, - }, { - 'testcase_name': 'pos_emb_left_and_right_context', - 'block_size': 3, - 'left_context': 4, - 'right_context': 2, - 'pos_emb_dim': 8, - }, { - 'testcase_name': 'lite_pos_emb_left_and_right_context', - 'block_size': 3, - 'left_context': 4, - 'right_context': 2, - 'pos_emb_dim': 8, - 'skip_term_b': True, - }) - def testFPropAgainstReference(self, - block_size, - left_context, - right_context, - pos_emb_dim=0, - num_heads=2, - input_dim=4, - hidden_dim=4, - skip_term_b=False, - use_additional_per_step_padding=False): - tf.reset_default_graph() - with self.session(use_gpu=True) as sess: - query_vec, _, paddings, _, _, _, _, _ = _AttentionInputs(input_dim) - if use_additional_per_step_padding: - # Generate a random binary mask of shape [N, T, S]. - additional_per_step_padding_val = np.random.random_integers( - low=0, high=1, size=(6, 6, 6)) - additional_per_step_padding = tf.constant( - additional_per_step_padding_val, tf.float32) - else: - additional_per_step_padding = None - - # Use the reference implementation + local casual padding to verify - # correctness. - if pos_emb_dim == 0: - p_cls = attention.LocalSelfAttention - expected_p_cls = attention.MultiHeadedAttention - else: - p_cls = attention.LocalSelfAttentionXL - expected_p_cls = attention.MultiHeadedAttentionXL - p = p_cls.Params().Set( - name='self_atten', - num_heads=num_heads, - input_dim=input_dim, - hidden_dim=hidden_dim, - block_size=block_size, - left_context=left_context, - right_context=right_context) - expected_p = expected_p_cls.Params().Set( - name='expected_self_atten', - num_heads=num_heads, - input_dim=input_dim, - hidden_dim=hidden_dim) - if pos_emb_dim != 0: - p.rel_pos_emb_dim = pos_emb_dim - expected_p.rel_pos_emb_dim = pos_emb_dim - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - expected_p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - - l = p.Instantiate() - expected_l = expected_p.Instantiate() - - tf.global_variables_initializer().run() - ctx_vec, _ = l.FProp( - l.theta, - query_vec, - query_vec, - query_vec, - paddings, - segment_mask=None, - per_step_padding=additional_per_step_padding) - context_vec_out = sess.run(ctx_vec) - per_step_padding = self._LocalCasualPadding(6, 6, left_context, - right_context) - if additional_per_step_padding is not None: - per_step_padding += additional_per_step_padding - expected_ctx_vec, _ = expected_l.FProp(expected_l.theta, query_vec, - query_vec, query_vec, paddings, - None, per_step_padding) - expected_context_vec_out = sess.run(expected_ctx_vec) - - # Don't compare if the query position is padded, or if all key positions - # are padded. - paddings_val = sess.run(paddings) - per_step_padding_val = sess.run(per_step_padding) - per_step_padding_val += paddings_val[:, :, np.newaxis] - per_step_padding_val += paddings_val[:, np.newaxis, :] - - dont_compare = np.sum( - per_step_padding_val > 0, axis=-1) == per_step_padding_val.shape[-1] - expected_context_vec_out *= (1 - dont_compare)[..., np.newaxis] - context_vec_out *= (1 - dont_compare)[..., np.newaxis] - self.assertAllClose(context_vec_out, expected_context_vec_out) - - def testFPropWithDropout(self): - with self.session(use_gpu=True) as sess: - query_vec, _, paddings, _, _, _, _, _ = _AttentionInputs(input_dim=4) - p = attention.LocalSelfAttention.Params().Set( - name='self_atten', - num_heads=2, - input_dim=4, - hidden_dim=4, - block_size=2, - left_context=2, - right_context=0, - atten_dropout_prob=0.3, - ) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - tf.global_variables_initializer().run() - ctx_vec, _ = l.FProp( - l.theta, query_vec, query_vec, query_vec, paddings, segment_mask=None) - ctx_vec_val = sess.run(ctx_vec) - print(ctx_vec_val) - - def _AttentionExtendStepInputs(self, - batch_size=6, - input_dim=4, - num_heads=2, - dtype=tf.float32): - np.random.seed(6348575) - seq_len = 6 - query_vec_p = [np.random.rand(1, input_dim) for _ in range(batch_size)] - query_vec = tf.stack([tf.constant(x, dtype=dtype) for x in query_vec_p]) - source_vecs = tf.constant( - np.random.normal( - 0.1, 0.5, [seq_len, batch_size, num_heads, input_dim // num_heads]), - dtype=dtype) - source_ctxs = tf.constant( - np.random.normal( - 0.1, 0.5, [seq_len, batch_size, num_heads, input_dim // num_heads]), - dtype=dtype) - cached_states = py_utils.NestedMap(key=source_vecs, value=source_ctxs) - return query_vec, cached_states - - def testExtendStepSelfAttention(self): - # input_batch:6, seq_len:6, query_len: 1. Test n = 2 case. - batch_size = 6 - input_dim = 4 - num_heads = 2 - with self.session(use_gpu=True) as sess: - query_vec, cached_states = ( - self._AttentionExtendStepInputs( - batch_size=batch_size, input_dim=input_dim, num_heads=num_heads)) - p = attention.LocalSelfAttention.Params().Set( - name='self_atten', - num_heads=num_heads, - input_dim=input_dim, - hidden_dim=4, - block_size=2, - left_context=2, - right_context=0, - atten_dropout_prob=0.3, - ) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - tf.global_variables_initializer().run() - ctx_vec, updated_states = l.ExtendStep( - l.theta, - query_vec, - cached_states, - paddings=None, - segment_mask=None, - per_step_padding=None, - time_step=3, - use_short_seq_opt=False) - context_vec_out = sess.run(ctx_vec) - new_source_vecs = sess.run(updated_states.key) - context_vec_out = np.reshape(context_vec_out, (6, 4)) - - tf.logging.info(np.array_repr(np.sum(context_vec_out, axis=1))) - self.assertAllClose( - [3.303124, 3.90266, 2.971359, 2.486641, 3.109267, 1.54773], - np.sum(context_vec_out, axis=1)) - new_source_vecs = np.reshape(new_source_vecs, (6, 24)) - tf.logging.info(np.array_repr(np.sum(new_source_vecs, axis=1))) - self.assertAllClose( - [5.135725, 1.340482, 1.065773, 4.116683, 4.928454, 3.161165], - np.sum(new_source_vecs, axis=1)) - - def testStreamingExtendStep(self): - batch_size = 1 - input_dim = 4 - hidden_dim = 4 - num_heads = 2 - seq_length = 5 - left_context = 2 - with self.session(use_gpu=True): - # Prepares inputs. - tf.random.set_seed(12345) - np.random.seed(123456789) - inputs = tf.constant( - np.random.normal(0.1, 0.5, [batch_size, seq_length, input_dim]), - dtype=tf.float32) - paddings = tf.zeros(shape=[batch_size, seq_length], dtype=tf.float32) - - # Prepares the model. - p = attention.LocalSelfAttention.Params().Set( - name='local_self_atten', - num_heads=num_heads, - input_dim=input_dim, - hidden_dim=hidden_dim, - left_context=left_context, - right_context=0, - ) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - model = p.Instantiate() - self.evaluate(tf.global_variables_initializer()) - - outputs, _ = model.FProp(model.theta, inputs, inputs, inputs, paddings) - outputs_shape = py_utils.GetShape(outputs) - self.assertListEqual([batch_size, seq_length, input_dim], outputs_shape) - - # Streaming inference - state = model.zero_state(batch_size) - for i in range(0, seq_length): - time_step = tf.constant([i], dtype=tf.int32) - i_input = inputs[:, i:(i + 1), :] # [B, 1, D] - i_output, state = model.StreamingExtendStep(i_input, state, time_step) - i_output_shape = py_utils.GetShape(i_output) - self.assertListEqual([batch_size, 1, input_dim], i_output_shape) - - i_expected_output = outputs[:, i:(i + 1), :] - v, expected_v = self.evaluate([i_output, i_expected_output]) - self.assertAllClose(v, expected_v) - - -class LocalSelfAttentionXLTest(test_utils.TestCase, parameterized.TestCase): - """Test local causual self attention with relative pos embedding.""" - - def testStreamingExtendStep(self): - batch_size = 1 - input_dim = 4 - hidden_dim = 4 - num_heads = 2 - seq_length = 5 - left_context = 2 - rel_pos_emb_dim = 2 - with self.session(use_gpu=True): - # Prepares inputs. - tf.random.set_seed(12345) - np.random.seed(123456789) - inputs = tf.constant( - np.random.normal(0.1, 0.5, [batch_size, seq_length, input_dim]), - dtype=tf.float32) - paddings = tf.zeros(shape=[batch_size, seq_length], dtype=tf.float32) - - # Prepares the model. - p = attention.LocalSelfAttentionXL.Params().Set( - name='local_self_atten_xl', - rel_pos_emb_dim=rel_pos_emb_dim, - num_heads=num_heads, - input_dim=input_dim, - hidden_dim=hidden_dim, - left_context=left_context, - right_context=0, - ) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - model = p.Instantiate() - self.evaluate(tf.global_variables_initializer()) - - outputs, _ = model.FProp(model.theta, inputs, inputs, inputs, paddings) - outputs_shape = py_utils.GetShape(outputs) - self.assertListEqual([batch_size, seq_length, input_dim], outputs_shape) - - # Streaming inference - state = model.zero_state(batch_size) - for i in range(0, seq_length): - time_step = tf.constant([i], dtype=tf.int32) - i_input = inputs[:, i:(i + 1), :] # [B, 1, D] - i_output, state = model.StreamingExtendStep(i_input, state, time_step) - i_output_shape = py_utils.GetShape(i_output) - self.assertListEqual([batch_size, 1, input_dim], i_output_shape) - - i_expected_output = outputs[:, i:(i + 1), :] - v, expected_v = self.evaluate([i_output, i_expected_output]) - self.assertAllClose(v, expected_v) - - -class RoutingAttentionTest(test_utils.TestCase, parameterized.TestCase): - """Tests for RoutingAttention.""" - - def testDotAttenSlow(self): - batch_size = 7 - source_length = 6 - target_length = 4 - num_heads = 2 - dim_per_head = 5 - num_clusters = 3 - attention_window = 4 - q = np.random.rand(batch_size, target_length, num_heads, - dim_per_head).astype(np.float32) - k = np.random.rand(batch_size, source_length, num_heads, - dim_per_head).astype(np.float32) - v = np.random.rand(batch_size, source_length, num_heads, - dim_per_head).astype(np.float32) - query_paddings = np.zeros([batch_size, target_length], dtype=np.float32) - key_paddings = np.zeros([batch_size, source_length], dtype=np.float32) - p = attention.RoutingAttention.Params().Set( - name='routing_atten', - input_dim=1, - hidden_dim=num_heads * dim_per_head, - num_heads=num_heads, - num_clusters=num_clusters, - attention_window=attention_window, - fast_path=False) - atten = p.Instantiate() - with self.session() as sess: - tf.global_variables_initializer().run() - encoded, probs = sess.run( - atten._DotAtten( - atten.theta, q, k, v, key_paddings, - query_paddings=query_paddings)) - self.assertEqual(encoded.shape, - (batch_size, target_length, num_heads, dim_per_head)) - self.assertEqual(probs.shape, - (batch_size, target_length, num_heads, source_length)) - # attention weights sum to 1. - self.assertAllClose( - np.sum(probs, axis=-1), - np.ones([batch_size, target_length, num_heads])) - - def testDotAttenFast(self): - batch_size = 6 - source_length = 8 - target_length = 7 - num_heads = 3 - dim_per_head = 5 - num_clusters = 2 - attention_window = source_length - q = np.random.rand(batch_size, target_length, num_heads, - dim_per_head).astype(np.float32) - k = np.random.rand(batch_size, source_length, num_heads, - dim_per_head).astype(np.float32) - v = np.random.rand(batch_size, source_length, num_heads, - dim_per_head).astype(np.float32) - - q_paddings = np.zeros([batch_size, target_length], dtype=np.float32) - k_paddings = np.zeros([batch_size, source_length], dtype=np.float32) - p = attention.RoutingAttention.Params().Set( - name='routing_atten', - input_dim=1, - hidden_dim=num_heads * dim_per_head, - num_heads=num_heads, - num_clusters=num_clusters, - attention_window=attention_window, - query_group_size_factor=1.5, # each group has 6 queries: 8 / 2 * 1.5. - fast_path=True) - atten = p.Instantiate() - # increase group size to 7. - atten2 = p.Copy().Set( - name='increase_group_size_routing_atten', - query_group_size_factor=1.75).Instantiate() - p = attention.MultiHeadedAttention.Params().Set( - name='full_atten', - input_dim=1, - hidden_dim=num_heads * dim_per_head, - num_heads=num_heads) - full_atten = p.Instantiate() - with self.session() as sess: - tf.global_variables_initializer().run() - encoded, probs = sess.run( - atten._DotAtten( - atten.theta, q, k, v, k_paddings, query_paddings=q_paddings)) - self.assertEqual(encoded.shape, - (batch_size, target_length, num_heads, dim_per_head)) - self.assertEqual(probs.shape, - (batch_size, target_length, num_heads, source_length)) - _, probs2 = sess.run( - atten2._DotAtten( - atten2.theta, q, k, v, k_paddings, query_paddings=q_paddings)) - # In order to match the full attention, we apply layer norm first. - q_ln = attention_util.KMeansClusteringForAtten.LayerNorm(q) - k_ln = attention_util.KMeansClusteringForAtten.LayerNorm(k) - full_encoded_t, full_probs_t = full_atten._DotAtten( - full_atten.theta, q_ln, k_ln, v, k_paddings, None) - full_probs_t = tf.transpose(full_probs_t, [0, 2, 1, 3]) - full_probs, full_encoded = sess.run([full_probs_t, full_encoded_t]) - - # When we increase p.query_group_size_factor, the number of left out queries - # decreases. - self.assertLess(np.sum(probs), np.sum(probs2)) - for batch_idx in range(batch_size): - for time_idx in range(target_length): - for head_idx in range(num_heads): - sub_probs = probs[batch_idx, time_idx, head_idx, :] - sub_encoded = encoded[batch_idx, time_idx, head_idx, :] - # encoded output is either 0 or matching full attention output - # for each query position. - if np.allclose(sub_probs, np.zeros_like(sub_probs)): - self.assertAllClose(sub_encoded, np.zeros_like(sub_encoded)) - continue - self.assertAllClose(sub_probs, full_probs[batch_idx, time_idx, - head_idx, :]) - self.assertAllClose(sub_encoded, full_encoded[batch_idx, time_idx, - head_idx, :]) - - @parameterized.parameters((False, 0), (False, 1), (False, 2), (True, 0), - (True, 1), (True, 2)) - def testDotAttenFull(self, fast_path, num_padded): - batch_size = 2 - source_length = 5 - target_length = 6 - num_heads = 2 - dim_per_head = 5 - # fast_path=True with multiple clusters might leave out some queries. - # For the purpose of this test we only use a single cluster. - num_clusters = 1 if fast_path else 3 - attention_window = source_length - q = tf.random.normal( - shape=[batch_size, target_length, num_heads, dim_per_head]) - k = tf.random.normal( - shape=[batch_size, source_length, num_heads, dim_per_head]) - v = tf.random.normal( - shape=[batch_size, source_length, num_heads, dim_per_head]) - - q_paddings = np.zeros([batch_size, target_length], dtype=np.float32) - k_paddings = np.zeros([batch_size, source_length], dtype=np.float32) - if num_padded: - # randomly pad elements. - for i in range(batch_size): - zero_index = np.random.choice(source_length, num_padded, False) - for j in zero_index: - k_paddings[i, j] = 1. - p = attention.RoutingAttention.Params().Set( - name='routing_atten', - input_dim=1, - hidden_dim=num_heads * dim_per_head, - num_heads=num_heads, - num_clusters=num_clusters, - attention_window=attention_window, - query_group_size_factor=1.0, - fast_path=fast_path) - atten = p.Instantiate() - p = attention.MultiHeadedAttention.Params().Set( - name='full_atten', - input_dim=1, - hidden_dim=num_heads * dim_per_head, - num_heads=num_heads) - full_atten = p.Instantiate() - with self.session() as sess: - tf.global_variables_initializer().run() - encoded_t, probs_t = atten._DotAtten( - atten.theta, q, k, v, k_paddings, query_paddings=q_paddings) - gradients_t = tf.gradients(encoded_t, [q, k, v]) - # In order to match the full attention, we apply layer norm first. - q_ln = attention_util.KMeansClusteringForAtten.LayerNorm(q) - k_ln = attention_util.KMeansClusteringForAtten.LayerNorm(k) - full_encoded_t, full_probs_t = full_atten._DotAtten( - full_atten.theta, q_ln, k_ln, v, k_paddings, None) - full_probs_t = tf.transpose(full_probs_t, [0, 2, 1, 3]) - full_gradients_t = tf.gradients(full_encoded_t, [q, k, v]) - (encoded, probs, full_encoded, full_probs, gradients, - full_gradients) = sess.run([ - encoded_t, probs_t, full_encoded_t, full_probs_t, gradients_t, - full_gradients_t - ]) - self.assertAllClose(probs, full_probs) - self.assertAllClose(encoded, full_encoded) - # The 3 gradients (dq, dk, dv) should also match - self.assertAllClose(gradients, full_gradients) - - @parameterized.parameters(False, True) - def testDotAttenCausalMasking(self, fast_path): - batch_size = 3 - seq_length = 12 - num_heads = 2 - dim_per_head = 4 - num_clusters = 1 if fast_path else 3 - attention_window = seq_length - q = np.random.rand(batch_size, seq_length, num_heads, - dim_per_head).astype(np.float32) - k = np.random.rand(batch_size, seq_length, num_heads, - dim_per_head).astype(np.float32) - v = np.random.rand(batch_size, seq_length, num_heads, - dim_per_head).astype(np.float32) - - q_paddings = np.zeros([batch_size, seq_length], dtype=np.float32) - k_paddings = np.zeros([batch_size, seq_length], dtype=np.float32) - p = attention.RoutingAttention.Params().Set( - name='routing_atten', - input_dim=1, - hidden_dim=num_heads * dim_per_head, - num_heads=num_heads, - num_clusters=num_clusters, - attention_window=attention_window, - causal_masking=True, - query_group_size_factor=1.0, - fast_path=fast_path) - atten = p.Instantiate() - p = attention.MultiHeadedAttention.Params().Set( - name='full_atten', - input_dim=1, - hidden_dim=num_heads * dim_per_head, - num_heads=num_heads) - full_atten = p.Instantiate() - with self.session() as sess: - tf.global_variables_initializer().run() - encoded, probs = sess.run( - atten._DotAtten( - atten.theta, q, k, v, k_paddings, query_paddings=q_paddings)) - # In order to match the full attention, we apply layer norm first. - q_ln = attention_util.KMeansClusteringForAtten.LayerNorm(q) - k_ln = attention_util.KMeansClusteringForAtten.LayerNorm(k) - # Manually apply causal padding to full attention. - per_step_padding = tf.tile( - tf.expand_dims( - attention.CausalPadding(seq_length, dtype=q_ln.dtype), 0), - [batch_size, 1, 1]) - full_encoded, full_probs = full_atten._DotAtten( - full_atten.theta, - q_ln, - k_ln, - v, - k_paddings, - segment_mask=None, - per_step_padding=per_step_padding) - full_probs = tf.transpose(full_probs, [0, 2, 1, 3]) - self.assertAllClose(probs, full_probs.eval()) - self.assertAllClose(encoded, full_encoded.eval()) - - # Verify that the first token only attends to position 0. - first_token_probs = probs[:, 0, :, :] - expected = np.zeros_like(first_token_probs) - expected[:, :, 0] = 1. - self.assertAllClose(first_token_probs, expected) - - @parameterized.parameters(False, True) - def testSelfAtten(self, fast_path): - batch_size = 4 - target_length = 8 - num_heads = 4 - dim_per_head = 5 - num_clusters = 3 - attention_window = 6 - q = tf.random.normal( - shape=[batch_size, target_length, num_heads, dim_per_head]) - v = tf.random.normal( - shape=[batch_size, target_length, num_heads, dim_per_head]) - q_copy = tf.identity(q) - - paddings = np.zeros([batch_size, target_length], dtype=np.float32) - p = attention.RoutingAttention.Params().Set( - name='routing_atten', - input_dim=1, - hidden_dim=num_heads * dim_per_head, - num_heads=num_heads, - num_clusters=num_clusters, - attention_window=attention_window, - query_group_size_factor=1.0, - fast_path=fast_path) - atten = p.Instantiate() - with self.session() as sess: - tf.global_variables_initializer().run() - # self attention path - encoded_self_t, probs_self_t = atten._DotAtten( - atten.theta, q, q, v, paddings, query_paddings=paddings) - # computed as cross attention - encoded_t, probs_t = atten._DotAtten( - atten.theta, q, q_copy, v, paddings, query_paddings=paddings) - encoded, probs, encoded_self, probs_self = sess.run( - [encoded_t, probs_t, encoded_self_t, probs_self_t]) - self.assertAllClose(probs, probs_self) - self.assertAllClose(encoded, encoded_self) - - def testExtendStep(self): - batch_size = 8 - target_length = 10 - num_heads = 4 - dim_per_head = 5 - num_clusters = 6 - attention_window = target_length - input_dim = 7 - q = np.random.rand(batch_size, target_length, input_dim).astype(np.float32) - paddings = np.zeros([batch_size, target_length], dtype=np.float32) - p = attention.RoutingAttention.Params().Set( - name='routing_atten', - input_dim=input_dim, - hidden_dim=num_heads * dim_per_head, - num_heads=num_heads, - num_clusters=num_clusters, - attention_window=attention_window, - causal_masking=True, - fast_path=False) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - atten = p.Instantiate() - # We ensure that the encoded attention result is the same between FProp() - # and sequential calls to ExtendStep(). - with self.session() as sess: - # self attention path via ExtendStep - encoded_all = [] - states = atten.InitStates(atten.theta, batch_size, target_length) - self.assertEqual(states.key.shape, - (target_length, batch_size, num_heads, dim_per_head)) - self.assertEqual(states.value.shape, - (target_length, batch_size, num_heads, dim_per_head)) - self.assertEqual(states.key_dists.shape, - (target_length, batch_size, num_heads, num_clusters)) - for i in range(target_length): - encoded, states = atten.ExtendStep(atten.theta, q[:, i:i + 1, :], - states, paddings, i) - self.assertEqual(encoded.shape, (batch_size, 1, input_dim)) - encoded_all.append(encoded) - encoded_extend_t = tf.concat(encoded_all, axis=1) - - # self attention path via FProp - encoded_fprop_t, _ = atten.FProp(atten.theta, q, q, q, paddings) - self.assertEqual(encoded_fprop_t.shape, - (batch_size, target_length, input_dim)) - - tf.global_variables_initializer().run() - encoded_extend, encoded_fprop = sess.run( - [encoded_extend_t, encoded_fprop_t]) - self.assertAllClose(encoded_extend, encoded_fprop) - - -class TransformerLayerTest(test_utils.TestCase, parameterized.TestCase): - """Test Transformer decoder layers.""" - - def _TransformerAttentionLayerInputs(self, input_dim=4, dtype=tf.float32): - np.random.seed(6348575) - query_vec = tf.transpose( - tf.stack([ - tf.constant(np.random.rand(2, input_dim), dtype=dtype) - for _ in range(5) - ]), [1, 0, 2]) - paddings = tf.constant([[0, 0, 1, 1, 0], [1, 0, 0, 0, 1]], dtype=dtype) - aux_vec = tf.transpose( - tf.stack([ - tf.constant(np.random.rand(2, input_dim), dtype=dtype) - for _ in range(7) - ]), [1, 0, 2]) - aux_paddings = tf.constant([[0, 1, 0, 1, 0, 1, 0], [1, 0, 1, 0, 1, 0, 1]], - dtype=dtype) - return query_vec, paddings, aux_vec, aux_paddings - - def testTransformerAttentionLayerFPropMaskedSelfAttention(self): - with self.session(use_gpu=True) as sess: - query_vec, paddings, _, _ = self._TransformerAttentionLayerInputs() - - p = attention.TransformerAttentionLayer.Params().Set( - name='transformer_masked_self_atten', - input_dim=4, - is_masked=True, - num_heads=2) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - ctx_vec, _ = l.FProp(l.theta, query_vec, None, paddings) - - tf.global_variables_initializer().run() - actual_ctx = sess.run(ctx_vec) - actual_ctx = np.reshape(actual_ctx, (10, 4)) - tf.logging.info(np.array_repr(actual_ctx)) - expected_ctx = [7.777687, 5.219166, 6.305151, 4.817311] - self.assertAllClose(expected_ctx, np.sum(actual_ctx, axis=0)) - - def testAttentionLayerFPropMaskedSelfAttentionPaddingOverride(self): - with self.session(use_gpu=True) as sess: - query_vec, paddings, _, _ = self._TransformerAttentionLayerInputs() - - p = attention.TransformerAttentionLayer.Params().Set( - name='transformer_masked_self_atten', - input_dim=4, - is_masked=True, - num_heads=2) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - triangle_padding = 1.0 - tf.linalg.band_part( - tf.ones([5, 5], dtype=query_vec.dtype), -1, 0) - per_step_padding_override = tf.tile( - tf.expand_dims(triangle_padding, 0), [2, 1, 1]) - - ctx_vec1, _ = l.FProp(l.theta, query_vec, None, paddings, - per_step_padding_override) - expected_ctx1, _ = l.FProp(l.theta, query_vec, None, paddings) - per_step_padding_override = tf.zeros([2, 5, 5]) - ctx_vec2, _ = l.FProp(l.theta, query_vec, None, paddings, - per_step_padding_override) - - tf.global_variables_initializer().run() - actual_ctx1, actual_ctx2, actual_expected_ctx1 = sess.run( - [ctx_vec1, ctx_vec2, expected_ctx1]) - tf.logging.info(np.array_repr(actual_ctx1)) - tf.logging.info(np.array_repr(actual_ctx2)) - expected_ctx2 = [7.9491496, 5.2976646, 6.5383415, 5.0169916] - self.assertAllClose(actual_expected_ctx1, ctx_vec1) - self.assertAllClose(expected_ctx2, - np.sum(np.reshape(actual_ctx2, (10, 4)), axis=0)) - - def testTransformerAttentionLayerFPropCrossAttention(self): - with self.session(use_gpu=True) as sess: - (query_vec, _, aux_vec, - aux_paddings) = self._TransformerAttentionLayerInputs() - p = attention.TransformerAttentionLayer.Params().Set( - name='transformer_cross_atten', - input_dim=4, - is_masked=False, - num_heads=2) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - ctx_vec, _ = l.FProp(l.theta, query_vec, aux_vec, aux_paddings) - - tf.global_variables_initializer().run() - actual_ctx = sess.run(ctx_vec) - actual_ctx = np.reshape(actual_ctx, (10, 4)) - tf.logging.info(np.array_repr(actual_ctx)) - expected_ctx = [19.345360, 15.057412, 13.744134, 13.387347] - self.assertAllClose(expected_ctx, np.sum(actual_ctx, axis=0)) - - def testMultiSourceTransformerAttentionLayerFPropCrossAttention(self): - with self.session(use_gpu=True) as sess: - (query_vec, _, aux_vec, - aux_paddings) = self._TransformerAttentionLayerInputs() - p = attention.TransformerMultiSourceAttentionLayer.Params().Set( - name='transformer_multi_source_cross_atten', - input_dim=4, - is_masked=False, - num_heads=2, - num_source=2) - p.multi_source_atten.atten_merger_tpl = ( - tm_attention.MergerLayer.Params().Set(merger_op='sum')) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - ctx_vec, _ = l.FProp( - l.theta, query_vec, - py_utils.NestedMap({ - 'source_0': aux_vec, - 'source_1': aux_vec - }), - py_utils.NestedMap({ - 'source_0': aux_paddings, - 'source_1': aux_paddings - })) - - tf.global_variables_initializer().run() - actual_ctx = sess.run(ctx_vec) - actual_ctx = np.reshape(actual_ctx, (10, 4)) - tf.logging.info(np.array_repr(actual_ctx)) - expected_ctx = [32.4878, 25.145725, 21.534966, 22.007454] - self.assertAllClose(expected_ctx, np.sum(actual_ctx, axis=0)) - - @parameterized.named_parameters( - { - 'testcase_name': '_short_seq', - 'use_short_seq_opt': True, - }, { - 'testcase_name': '_long_seq', - 'use_short_seq_opt': False, - }) - def testTransformerAttentionLayerExtendStep(self, use_short_seq_opt): - with self.session(use_gpu=True) as sess: - query_vec, _, _, _ = self._TransformerAttentionLayerInputs() - paddings = tf.zeros([2, 5]) - cached_key = tf.constant( - np.random.normal(0.1, 0.5, [5, 2, 2, 2]), dtype=tf.float32) - cached_value = tf.constant( - np.random.normal(0.1, 0.5, [5, 2, 2, 2]), dtype=tf.float32) - prefix_states = py_utils.NestedMap(key=cached_key, value=cached_value) - - p = attention.TransformerAttentionLayer.Params().Set( - name='transformer_atten', input_dim=4, is_masked=True, num_heads=2) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - - ctx_vec1, _ = l.FProp(l.theta, query_vec, None, paddings) - - ctx_vec2 = [] - for i in range(5): - ctx_vec, prefix_states = l.ExtendStep( - l.theta, tf.expand_dims(query_vec[:, i, :], 1), prefix_states, i, - use_short_seq_opt) - ctx_vec2.append(tf.squeeze(ctx_vec, 1)) - ctx_vec2 = tf.transpose(tf.stack(ctx_vec2), [1, 0, 2]) - - tf.global_variables_initializer().run() - ctx1, ctx2 = sess.run([ctx_vec1, ctx_vec2]) - self.assertAllClose(ctx1, ctx2) - - def testTransformerAttentionLayerNoLayernorm(self): - """Verify if Transformer attention allows no layernorm in FProp and Extend.""" - with self.session(use_gpu=True) as sess: - query_vec, _, _, _ = self._TransformerAttentionLayerInputs() - paddings = tf.zeros([2, 5]) - cached_key = tf.constant( - np.random.normal(0.1, 0.5, [5, 2, 2, 2]), dtype=tf.float32) - cached_value = tf.constant( - np.random.normal(0.1, 0.5, [5, 2, 2, 2]), dtype=tf.float32) - prefix_states = py_utils.NestedMap(key=cached_key, value=cached_value) - - p = attention.TransformerAttentionLayer.Params().Set( - name='transformer_atten', - input_dim=4, - is_masked=True, - num_heads=2, - ln_tpl=None) # Set ln_tpl to None. - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - - ctx_vec1, _ = l.FProp(l.theta, query_vec, None, paddings) - - ctx_vec2 = [] - for i in range(5): - ctx_vec, prefix_states = l.ExtendStep( - l.theta, tf.expand_dims(query_vec[:, i, :], 1), prefix_states, i, - False) - ctx_vec2.append(tf.squeeze(ctx_vec, 1)) - ctx_vec2 = tf.transpose(tf.stack(ctx_vec2), [1, 0, 2]) - - tf.global_variables_initializer().run() - ctx1, ctx2 = sess.run([ctx_vec1, ctx_vec2]) - self.assertAllClose(ctx1, ctx2) - - def _ConstructTransformerDecoderLayer(self, use_relative_atten=False): - p = attention.TransformerDecoderLayer.Params() - p.name = 'transformer_decoder_layer' - p.input_dim = 4 - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_heads = 2 - if use_relative_atten: - p = attention.UseRelativeAttentionInTransformerLayer(p, 4) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - return attention.TransformerDecoderLayer(p) - - def testTransformerLayerCommonParams(self): - with self.session(use_gpu=True) as sess: - input_dim, fflayer_hidden_dim, num_heads = 4, 7, 2 - (query_vec, _, aux_vec, - aux_paddings) = self._TransformerAttentionLayerInputs( - input_dim=input_dim) - query_vec = tf.tile(query_vec, [1, 1, 1]) - paddings = tf.zeros([2, 5]) - p = attention.TransformerLayer.CommonParams( - input_dim=input_dim, - atten_num_heads=num_heads, - fflayer_hidden_dim=fflayer_hidden_dim) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - ctx_vec, _ = l.FProp(l.theta, query_vec, paddings, aux_vec, aux_paddings) - - tf.global_variables_initializer().run() - actual_ctx = sess.run(ctx_vec) - actual_ctx = np.reshape(actual_ctx, (10, 4)) - tf.logging.info(np.array_repr(actual_ctx)) - expected_ctx = [ - 4.7839108, 4.5303655, 5.5551023, 5.0657663, 5.0493064, 3.2142467, - 2.820018, 5.659971, 4.3814187, 2.60475 - ] - self.assertAllClose(expected_ctx, np.sum(actual_ctx, axis=1)) - - @parameterized.named_parameters(('SingleBatch', 1), ('DoubleBatch', 2)) - def testTransformerLayerFPropWithCrossAttention(self, multiplier): - with self.session(use_gpu=True) as sess: - (query_vec, _, aux_vec, - aux_paddings) = self._TransformerAttentionLayerInputs() - query_vec = tf.tile(query_vec, [multiplier, 1, 1]) - paddings = tf.zeros([2 * multiplier, 5]) - p = attention.TransformerLayer.Params() - p.name = 'transformer_layer' - p.input_dim = 4 - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_heads = 2 - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - ctx_vec, _ = l.FProp(l.theta, query_vec, paddings, aux_vec, aux_paddings) - - tf.global_variables_initializer().run() - actual_ctx = sess.run(ctx_vec) - actual_ctx = np.reshape(actual_ctx, (10 * multiplier, 4)) - tf.logging.info(np.array_repr(actual_ctx)) - expected_ctx = [ - 4.7839108, 4.5303655, 5.5551023, 5.065767, 5.0493064, 3.2142467, - 2.8200178, 5.659971, 4.3814187, 2.60475 - ] * multiplier - self.assertAllClose(expected_ctx, np.sum(actual_ctx, axis=1)) - - @parameterized.named_parameters(('SingleBatch', 1), ('DoubleBatch', 2)) - def testMultiSourceTransformerLayerFPropWithCrossAttention(self, multiplier): - with self.session(use_gpu=True) as sess: - (query_vec, _, aux_vec, - aux_paddings) = self._TransformerAttentionLayerInputs() - query_vec = tf.tile(query_vec, [multiplier, 1, 1]) - paddings = tf.zeros([2 * multiplier, 5]) - p = attention.TransformerLayer.Params() - p.name = 'transformer_layer' - p.input_dim = 4 - p.tr_fflayer_tpl.hidden_dim = 7 - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - # multi-source cross attention - p.tr_atten_tpl = ( - attention.TransformerMultiSourceAttentionLayer.Params().Set( - num_source=2, primary_source_index=0, num_heads=2)) - p.tr_self_atten_tpl = attention.TransformerAttentionLayer.Params().Set( - input_dim=4, num_heads=2) - l = p.Instantiate() - ctx_vec, _ = l.FProp( - l.theta, query_vec, paddings, - py_utils.NestedMap({ - 'source_0': aux_vec, - 'source_1': aux_vec - }), - py_utils.NestedMap({ - 'source_0': aux_paddings, - 'source_1': aux_paddings - })) - - tf.global_variables_initializer().run() - actual_ctx = sess.run(ctx_vec) - actual_ctx = np.reshape(actual_ctx, (10 * multiplier, 4)) - tf.logging.info(np.array_repr(actual_ctx)) - expected_ctx = [ - 4.7839108, 4.5303655, 5.5551023, 5.0657663, 5.0493064, 3.2142467, - 2.820018, 5.659971, 4.3814187, 2.60475 - ] * multiplier - self.assertAllClose(expected_ctx, np.sum(actual_ctx, axis=1)) - - @parameterized.named_parameters(('Base', False), ('RelativeAtten', True)) - def testTransformerDecoderLayerConstruction(self, use_relative_atten): - _ = self._ConstructTransformerDecoderLayer( - use_relative_atten=use_relative_atten) - - def testTransformerDecoderLayerFProp(self): - with self.session(use_gpu=True) as sess: - (query_vec, paddings, aux_vec, - aux_paddings) = self._TransformerAttentionLayerInputs() - l = self._ConstructTransformerDecoderLayer() - - layer_output, _ = l.FProp(l.theta, query_vec, paddings, aux_vec, - aux_paddings) - - tf.global_variables_initializer().run() - actual_layer_output = sess.run(layer_output) - actual_layer_output = np.reshape(actual_layer_output, (10, 4)) - tf.logging.info(np.array_repr(actual_layer_output)) - expected_layer_output = [16.939590, 24.121685, 19.975197, 15.924350] - self.assertAllClose(expected_layer_output, - np.sum(actual_layer_output, axis=0)) - - def _ConstructTransformerEncoderLayerStack(self): - p = attention.StackedTransformerLayers.Params() - p.name = 'encoder_layers' - p.has_aux_atten = False - p.mask_self_atten = False - p.num_layers = 2 - p.mdl_dim = 4 - p.hidden_dim = 8 - p.num_atten_heads = 2 - p.dropout_prob = 0.2 - p.params_init = py_utils.WeightInit.Xavier() - p.random_seed = 12345 - return p.Instantiate() - - def _ConstructTransformerDecoderLayerStack(self, dropout_prob=0.2): - p = attention.StackedTransformerLayers.Params() - p.name = 'decoder_layers' - p.has_aux_atten = True - p.mask_self_atten = True - p.num_layers = 2 - p.mdl_dim = 4 - p.hidden_dim = 8 - p.num_atten_heads = 2 - p.dropout_prob = dropout_prob - p.params_init = py_utils.WeightInit.Xavier() - p.random_seed = 12345 - return p.Instantiate() - - def testStackedTransformerGetSplitForLayer(self): - cls = attention.StackedTransformerLayers - - buckets = [2, 4, 5, 6, 9, 11, 15] - ys = [cls.GetSplitForLayer(buckets, i) for i in range(16)] - self.assertEqual(0, ys[0]) - self.assertEqual(0, ys[1]) - self.assertEqual(0, ys[2]) - self.assertEqual(1, ys[3]) - - self.assertEqual(1, ys[4]) - self.assertEqual(2, ys[5]) - self.assertEqual(3, ys[6]) - self.assertEqual(4, ys[7]) - - self.assertEqual(4, ys[8]) - self.assertEqual(4, ys[9]) - self.assertEqual(5, ys[10]) - self.assertEqual(5, ys[11]) - - self.assertEqual(6, ys[12]) - self.assertEqual(6, ys[13]) - self.assertEqual(6, ys[14]) - self.assertEqual(6, ys[15]) - - def testTransformerEncoderLayerStackFProp(self): - with self.session(use_gpu=True) as sess: - (query_vec, paddings, _, _) = self._TransformerAttentionLayerInputs() - l = self._ConstructTransformerEncoderLayerStack() - layer_output, _ = l.FProp(l.theta, query_vec=query_vec, paddings=paddings) - tf.global_variables_initializer().run() - actual_layer_output = sess.run(layer_output) - actual_layer_output = np.reshape(actual_layer_output, (10, 4)) - tf.logging.info(np.array_repr(actual_layer_output)) - expected_layer_output = [6.178955, -11.376661, 7.032681, -1.532627] - self.assertAllClose(expected_layer_output, - np.sum(actual_layer_output, axis=0)) - - def testTransformerDecoderLayerStackFProp(self): - with self.session(use_gpu=True) as sess: - (query_vec, paddings, aux_vec, - aux_paddings) = self._TransformerAttentionLayerInputs() - l = self._ConstructTransformerDecoderLayerStack() - layer_output, _ = l.FProp( - l.theta, - query_vec=query_vec, - paddings=paddings, - aux_vec=aux_vec, - aux_paddings=aux_paddings) - tf.global_variables_initializer().run() - actual_layer_output = sess.run(layer_output) - actual_layer_output = np.reshape(actual_layer_output, (10, 4)) - tf.logging.info(np.array_repr(actual_layer_output)) - expected_layer_output = [9.926413, -4.491376, 27.051598, 2.112684] - self.assertAllClose(expected_layer_output, - np.sum(actual_layer_output, axis=0)) - - @parameterized.named_parameters( - { - 'testcase_name': '_short_seq', - 'use_short_seq_opt': True, - }, { - 'testcase_name': '_long_seq', - 'use_short_seq_opt': False, - }) - def testTransformerDecoderLayerStackExtendStep(self, use_short_seq_opt): - - def _Rnd(seed): - return tf.random.normal([5, 2, 2, 2], seed=seed) - - graph = tf.Graph() - with graph.as_default(): - tf.random.set_seed(123456) - query_vec, _, aux_vec, aux_paddings = ( - self._TransformerAttentionLayerInputs()) - paddings = tf.zeros([2, 5]) - layer_prefix_states_1 = py_utils.NestedMap(key=_Rnd(1), value=_Rnd(2)) - layer_prefix_states_2 = py_utils.NestedMap(key=_Rnd(3), value=_Rnd(4)) - prefix_states = py_utils.NestedMap( - x_layers=[layer_prefix_states_1, layer_prefix_states_2]) - - l = self._ConstructTransformerDecoderLayerStack(dropout_prob=0.) - - layer_output1, _ = l.FProp(l.theta, query_vec, paddings, aux_vec, - aux_paddings) - - layer_output2 = [] - for i in range(5): - layer_output, prefix_states = l.ExtendStep( - l.theta, tf.expand_dims(query_vec[:, i, :], 1), aux_vec, - aux_paddings, prefix_states, i, use_short_seq_opt) - layer_output2.append(tf.squeeze(layer_output, 1)) - layer_output2 = tf.transpose(tf.stack(layer_output2), [1, 0, 2]) - - with self.session(graph=graph, use_gpu=True) as sess: - tf.global_variables_initializer().run() - actual_layer_output1, actual_layer_output2 = sess.run( - [layer_output1, layer_output2]) - - self.assertAllClose(actual_layer_output1, actual_layer_output2) - - @parameterized.named_parameters( - { - 'testcase_name': '_short_seq', - 'use_short_seq_opt': True, - }, { - 'testcase_name': '_long_seq', - 'use_short_seq_opt': False, - }) - def testTransformerDecoderLayerExtendStep(self, use_short_seq_opt): - with self.session(use_gpu=True) as sess: - (query_vec, _, aux_vec, - aux_paddings) = self._TransformerAttentionLayerInputs() - paddings = tf.zeros([2, 5]) - cached_key = tf.constant( - np.random.normal(0.1, 0.5, [5, 2, 2, 2]), dtype=tf.float32) - cached_value = tf.constant( - np.random.normal(0.1, 0.5, [5, 2, 2, 2]), dtype=tf.float32) - prefix_states = py_utils.NestedMap(key=cached_key, value=cached_value) - - l = self._ConstructTransformerDecoderLayer() - - layer_output1, _ = l.FProp(l.theta, query_vec, paddings, aux_vec, - aux_paddings) - - layer_output2 = [] - for i in range(5): - layer_output, prefix_states = l.ExtendStep( - l.theta, tf.expand_dims(query_vec[:, i, :], 1), aux_vec, - aux_paddings, prefix_states, i, use_short_seq_opt) - layer_output2.append(tf.squeeze(layer_output, 1)) - layer_output2 = tf.transpose(tf.stack(layer_output2), [1, 0, 2]) - - tf.global_variables_initializer().run() - actual_layer_output1, actual_layer_output2 = sess.run( - [layer_output1, layer_output2]) - self.assertAllClose(actual_layer_output1, actual_layer_output2) - - def _ConstructMultiSourceTransformerDecoderLayer(self, - use_relative_atten=False): - p = attention.MultiSourceTransformerDecoderLayer.Params().Set(num_source=2) - p.name = 'multi_source_transformer_decoder_layer' - p.input_dim = 4 - p.tr_fflayer_tpl.hidden_dim = 7 - # multi-source cross attention - p.tr_atten_tpl = ( - attention.TransformerMultiSourceAttentionLayer.Params().Set( - num_source=2, primary_source_index=0, num_heads=2)) - p.tr_self_atten_tpl = attention.TransformerAttentionLayer.Params().Set( - input_dim=4, num_heads=2) - p.tr_atten_tpl.multi_source_atten.atten_merger_tpl = ( - tm_attention.MergerLayer.Params().Set(merger_op='sum')) - if use_relative_atten: - p = attention.UseRelativeAttentionInTransformerLayer(p, 4) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - return attention.MultiSourceTransformerDecoderLayer(p) - - @parameterized.named_parameters( - { - 'testcase_name': '_short_seq', - 'use_short_seq_opt': True, - }, { - 'testcase_name': '_long_seq', - 'use_short_seq_opt': False, - }) - def testMultiSourceTransformerDecoderLayerExtendStep(self, use_short_seq_opt): - with self.session(use_gpu=True) as sess: - (query_vec, _, aux_vec, - aux_paddings) = self._TransformerAttentionLayerInputs() - paddings = tf.zeros([2, 5]) - cached_key = tf.constant( - np.random.normal(0.1, 0.5, [5, 2, 2, 2]), dtype=tf.float32) - cached_value = tf.constant( - np.random.normal(0.1, 0.5, [5, 2, 2, 2]), dtype=tf.float32) - prefix_states = py_utils.NestedMap(key=cached_key, value=cached_value) - - l = self._ConstructMultiSourceTransformerDecoderLayer() - - ms_aux_vec = py_utils.NestedMap({ - 'source_0': aux_vec, - 'source_1': aux_vec - }) - ms_aux_paddings = py_utils.NestedMap({ - 'source_0': aux_paddings, - 'source_1': aux_paddings - }) - layer_output1, _ = l.FProp(l.theta, query_vec, paddings, ms_aux_vec, - ms_aux_paddings) - - layer_output2 = [] - for i in range(5): - layer_output, prefix_states = l.ExtendStep( - l.theta, tf.expand_dims(query_vec[:, i, :], 1), ms_aux_vec, - ms_aux_paddings, prefix_states, i, use_short_seq_opt) - layer_output2.append(tf.squeeze(layer_output, 1)) - layer_output2 = tf.transpose(tf.stack(layer_output2), [1, 0, 2]) - - tf.global_variables_initializer().run() - actual_layer_output1, actual_layer_output2 = sess.run( - [layer_output1, layer_output2]) - self.assertAllClose(actual_layer_output1, actual_layer_output2) - - def testGPipeTransformerLayerConstruction(self): - p = attention.GPipeTransformerLayer.Params() - p.name = 'gpipe_transformer_layer' - p.input_dim = 4 - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_heads = 2 - p.tr_atten_tpl.residual_dropout_prob = 0.1 - p.cls.SetupDeterministicDropout(p) - layer = p.Instantiate() - self.assertEqual(0.1, layer.params.tr_atten_tpl.residual_dropout_prob) - - -class GPipeBatchMajorTransformerLayerTest(test_utils.TestCase, - parameterized.TestCase): - """Test GPipeBatchMajorTransformer layers.""" - - def _ConstructGPipeBatchMajorTransformerLayer(self, - decoder=False, - packed=True, - dropout=0.1): - p = attention.GPipeBatchMajorTransformerLayer.Params() - p.name = 'gpipe_transformer_layer' - p.input_dim = 4 - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_heads = 2 - p.tr_atten_tpl.residual_dropout_prob = dropout - p.packed_input = packed - if decoder: - p.has_aux_atten = True - p.mask_self_atten = True - p.cls.SetupDeterministicDropout(p) - layer = p.Instantiate() - return p, layer - - def _GPipeBatchMajorTransformerLayerInputs(self, - input_dim=4, - dtype=tf.float32): - np.random.seed(6348575) - target_vec = tf.transpose( - tf.stack([ - tf.constant(np.random.rand(2, input_dim), dtype=dtype) - for _ in range(5) - ]), [1, 0, 2]) - target_paddings = tf.constant([[0, 0, 0, 0, 1], [0, 0, 0, 0, 0]], - dtype=dtype) - aux_vec = tf.transpose( - tf.stack([ - tf.constant(np.random.rand(2, input_dim), dtype=dtype) - for _ in range(7) - ]), [1, 0, 2]) - aux_paddings = tf.constant([[0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 1]], - dtype=dtype) - aux_segment_ids = tf.constant( - [[0, 0, 0, 1, 1, 1, 1], [0, 0, 0, 0, 1, 1, 1]], dtype=dtype) - target_segment_ids = tf.constant([[0, 0, 0, 1, 1], [0, 0, 1, 1, 1]], - dtype=dtype) - target_sa_mask = attention.SegmentMask( - target_segment_ids, target_segment_ids, apply_dtype_min=False) - aux_sa_mask = attention.SegmentMask( - aux_segment_ids, aux_segment_ids, apply_dtype_min=False) - ca_mask = attention.SegmentMask( - target_segment_ids, aux_segment_ids, apply_dtype_min=False) - causal_padding = tf.expand_dims( - tf.tile( - tf.expand_dims(attention.CausalPadding(5, dtype=dtype), 0), - [2, 1, 1]), 1) - target_sa_mask = tf.math.maximum(causal_padding, target_sa_mask) - return (target_vec, target_paddings, target_sa_mask, aux_vec, aux_paddings, - aux_sa_mask, ca_mask) - - def testGPipeBatchMajorTransformerEncoderLayerConstruction(self): - _, layer = self._ConstructGPipeBatchMajorTransformerLayer() - self.assertEqual(0.1, layer.params.tr_atten_tpl.residual_dropout_prob) - - def testGPipeBatchMajorTransformerDecoderLayerConstruction(self): - _, layer = self._ConstructGPipeBatchMajorTransformerLayer(decoder=True) - self.assertEqual(0.1, layer.params.tr_atten_tpl.residual_dropout_prob) - - def testGPipeBatchMajorTransformerEncoderLayerFProp(self): - with self.session(use_gpu=True) as sess: - (_, _, _, aux_vec, aux_paddings, aux_sa_mask, - _) = self._GPipeBatchMajorTransformerLayerInputs() - _, l = self._ConstructGPipeBatchMajorTransformerLayer() - - layer_output = l.FProp(l.theta, aux_vec, aux_paddings, None, None, - aux_sa_mask, None, None)[0] - - tf.global_variables_initializer().run() - actual_layer_output = sess.run(layer_output) - actual_layer_output = np.reshape(actual_layer_output, (14, 4)) - tf.logging.info(np.array_repr(actual_layer_output)) - expected_layer_output = [7.616176, 8.611565, -0.932456, -4.5797] - self.assertAllClose(expected_layer_output, - np.sum(actual_layer_output, axis=0)) - - def testGPipeBatchMajorTransformerDecoderLayerFProp(self): - with self.session(use_gpu=True) as sess: - (target_vec, target_paddings, target_sa_mask, aux_vec, aux_paddings, - aux_sa_mask, ca_mask) = self._GPipeBatchMajorTransformerLayerInputs() - _, l = self._ConstructGPipeBatchMajorTransformerLayer(decoder=True) - - layer_output = l.FProp(l.theta, aux_vec, aux_paddings, target_vec, - target_paddings, aux_sa_mask, target_sa_mask, - ca_mask)[2] - - tf.global_variables_initializer().run() - actual_layer_output = sess.run(layer_output) - actual_layer_output = np.reshape(actual_layer_output, (10, 4)) - tf.logging.info(np.array_repr(actual_layer_output)) - expected_layer_output = [2.721037, 5.228053, 2.27512, 6.92945] - self.assertAllClose(expected_layer_output, - np.sum(actual_layer_output, axis=0)) - - def testGPipeBatchMajorTransformerDecoderLayerExtendStep(self): - with self.session(use_gpu=True) as sess: - (target_vec, _, _, aux_vec, aux_paddings, _, - _) = self._GPipeBatchMajorTransformerLayerInputs() - target_paddings = tf.zeros([2, 5]) - cached_key = tf.constant( - np.random.normal(0.1, 0.5, [5, 2, 2, 2]), dtype=tf.float32) - cached_value = tf.constant( - np.random.normal(0.1, 0.5, [5, 2, 2, 2]), dtype=tf.float32) - prefix_states = py_utils.NestedMap(key=cached_key, value=cached_value) - _, l = self._ConstructGPipeBatchMajorTransformerLayer( - decoder=True, packed=False, dropout=0.0) - - layer_output1 = l.FProp(l.theta, aux_vec, aux_paddings, target_vec, - target_paddings, None, None, None)[2] - - layer_output2 = [] - for i in range(5): - layer_output, prefix_states = l.ExtendStep( - l.theta, tf.expand_dims(target_vec[:, i, :], 1), aux_vec, - aux_paddings, prefix_states, i) - layer_output2.append(tf.squeeze(layer_output, 1)) - layer_output2 = tf.transpose(tf.stack(layer_output2), [1, 0, 2]) - - tf.global_variables_initializer().run() - actual_layer_output1, actual_layer_output2 = sess.run( - [layer_output1, layer_output2]) - self.assertAllClose(actual_layer_output1, actual_layer_output2) - - -class BuilderTest(test_utils.TestCase, parameterized.TestCase): - - def _testGraph(self, glu_with_tanh=False, dtype=tf.float32): - tf.random.set_seed(398847392) - np.random.seed(12345) - atten_builder = attention.Builder.Params().Set( - model_dim=4, num_heads=2, ff_hidden_dim=16, glu_with_tanh=glu_with_tanh) - params = atten_builder.Instantiate().LConvStack( - name='lightconv', kernel_sizes=[3, 3]) - params.dtype = dtype - params.random_seed = 0 - params.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = params.Instantiate() - l_in = tf.constant(np.random.rand(2, 3, 4), dtype=dtype) - l_padding = tf.zeros([2, 3], dtype=dtype) - l_out = l.FPropDefaultTheta( - py_utils.NestedMap(vec=l_in, paddings=l_padding)) - return l_out.vec - - @parameterized.parameters((False, 38.163662), (True, 35.88797)) - def testFprop(self, glu_with_tanh, expected_result): - with self.session(use_gpu=False, graph=tf.Graph()) as sess: - l_out = self._testGraph(glu_with_tanh) - l_out = tf.reduce_sum(l_out) - tf.global_variables_initializer().run() - l_out_eval = sess.run(l_out) - self.assertAllClose(expected_result, l_out_eval) - - def testBProp(self): - with self.session(use_gpu=True) as sess: - output = self._testGraph(dtype=tf.float64) - loss = tf.reduce_sum(output) - all_vars = tf.trainable_variables() - grads = tf.gradients(loss, all_vars) - tf.global_variables_initializer().run() - sym_grads = [sg.eval() for sg in grads] - num_grads = [ - test_utils.ComputeNumericGradient(sess, loss, v) for v in all_vars - ] - for ng, sg in zip(num_grads, sym_grads): - self.assertAllClose(ng, sg, rtol=5e-02, atol=5e-02) - - @parameterized.named_parameters( - { - 'testcase_name': '_baseline', - 'strides': [1, 1], - }, { - 'testcase_name': '_stride_2', - 'strides': [2, 1], - }, { - 'testcase_name': '_first_token', - 'strides': [2, 0], - }) - def testTransformerStackWithStride(self, strides): - with self.session(use_gpu=False) as sess: - bs = 2 - sl = 10 - d = 16 - tf.random.set_seed(12345) - atten_builder = attention.Builder.Params().Set( - model_dim=d, num_heads=2, ff_hidden_dim=5).Instantiate() - layers = [] - accumulate_stride = 1 - for layer_i, stride in enumerate(strides): - accumulate_stride *= stride - layers.append( - atten_builder.TransformerEncoderLayer( - name='atten_{}'.format(layer_i), stride=stride)) - p = atten_builder.Seq('model', *layers) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - input_embs = tf.constant( - np.random.random(size=[bs, sl, d]), dtype=np.float) - paddings = tf.zeros([bs, sl]) - l_out = l.FPropDefaultTheta( - py_utils.NestedMap(vec=input_embs, paddings=paddings)) - enc_out = l_out.vec - tf.global_variables_initializer().run() - actual_enc_out = sess.run(enc_out) - seq_len = sl // accumulate_stride if accumulate_stride != 0 else 1 - self.assertAllEqual([bs, seq_len, d], actual_enc_out.shape) - - @parameterized.named_parameters( - { - 'testcase_name': '_baseline', - 'strides': [(1, 6), (1, 3), 3], - }, { - 'testcase_name': '_stride_2', - 'strides': [(2, 4), (1, None), 2], - }, { - 'testcase_name': '_first_token', - 'strides': [(2, 5), (0, None), 1], - }) - def testTransformerStackWithStrideAndOutLength(self, strides): - with self.session(use_gpu=False) as sess: - bs = 2 - sl = 10 - d = 16 - tf.random.set_seed(12345) - atten_builder = attention.Builder.Params().Set( - model_dim=d, num_heads=2, ff_hidden_dim=5).Instantiate() - layers = [] - out_seq_len = strides.pop() - for layer_i, (stride, first_n) in enumerate(strides): - layers.append( - atten_builder.TransformerEncoderLayer( - name='atten_{}'.format(layer_i), stride=stride, - first_n=first_n)) - p = atten_builder.Seq('model', *layers) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - input_embs = tf.constant( - np.random.random(size=[bs, sl, d]), dtype=np.float) - paddings = tf.zeros([bs, sl]) - l_out = l.FPropDefaultTheta( - py_utils.NestedMap(vec=input_embs, paddings=paddings)) - enc_out = l_out.vec - tf.global_variables_initializer().run() - actual_enc_out = sess.run(enc_out) - self.assertAllEqual([bs, out_seq_len, d], actual_enc_out.shape) - - def testEncoderLayerWithPerLayerParam(self): - with self.session(use_gpu=False) as sess: - bs = 2 - sl = 10 - d = 16 - tf.random.set_seed(398847392) - np.random.seed(12345) - heads = [1, 2, 4] - ff_dims = [16, 32, 16] - atten_builder = attention.Builder.Params().Set( - model_dim=16, num_heads=heads, ff_hidden_dim=ff_dims).Instantiate() - layers = [] - for layer_i, (head, ff_dim) in enumerate(zip(heads, ff_dims)): - layers.append( - atten_builder.TransformerEncoderLayer( - name='atten_{}'.format(layer_i), - ff_hidden_dim=ff_dim, - num_heads=head, - stride=1 if layer_i < 2 else 0)) - p = atten_builder.Seq('model', *layers) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - input_embs = tf.constant( - np.random.random(size=[bs, sl, d]), dtype=np.float) - paddings = tf.zeros([bs, sl]) - l_out = l.FPropDefaultTheta( - py_utils.NestedMap(vec=input_embs, paddings=paddings)) - out = tf.reduce_sum(l_out.vec) - tf.global_variables_initializer().run() - actual_out = sess.run(out) - self.assertAllClose(actual_out, 17.40516) - - -class LmBuilderTest(test_utils.TestCase): - - def _testGraph(self, dtype=tf.float32): - tf.random.set_seed(398847392) - np.random.seed(12345) - atten_builder = attention.LmBuilder.Params().Set( - model_dim=4, num_heads=2, ff_hidden_dim=16, dtype=dtype) - params = atten_builder.Instantiate().TransformerEncoderStack( - name='xformer', num_layers=2) - params.dtype = dtype - params.random_seed = 0 - params.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = params.Instantiate() - l_in = tf.constant(np.random.rand(2, 3, 4), dtype=dtype) - l_padding = tf.zeros([2, 3], dtype=dtype) - l_out = l.FPropDefaultTheta( - py_utils.NestedMap(vec=l_in, paddings=l_padding)) - return l_out.vec - - def testFprop(self): - with self.session(use_gpu=False, graph=tf.Graph()) as sess: - l_out = self._testGraph() - l_out = tf.reduce_sum(l_out) - tf.global_variables_initializer().run() - l_out_eval = sess.run(l_out) - self.assertAllClose(36.04808, l_out_eval) - - def testBProp(self): - with self.session(use_gpu=True) as sess: - output = self._testGraph(dtype=tf.float64) - loss = tf.reduce_sum(output) - all_vars = tf.trainable_variables() - grads = tf.gradients(loss, all_vars) - tf.global_variables_initializer().run() - sym_grads = [sg.eval() for sg in grads] - num_grads = [ - test_utils.ComputeNumericGradient(sess, loss, v) for v in all_vars - ] - for ng, sg in zip(num_grads, sym_grads): - self.assertAllClose(ng, sg, rtol=5e-02, atol=5e-02) - - -def _CreateDummyParams(field_names): - p = hyperparams.Params() - for name in field_names: - p.Define(name, None, 'Dummy') - return p - - -class DummyDecoderRNNT(base_layer.BaseLayer): - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'dummy_decoder_rnnt' - p.Define('emb', _CreateDummyParams(['vocab_size']), 'Dummy emb.') - p.Define('target_seq_len', 20, 'Dummy target seq len.') - p.Define('num_classes', None, 'Dummy num classes.') - return p - - @classmethod - def UpdateTargetVocabSize(cls, p, vocab_size, wpm_model=None): - p.emb.vocab_size = vocab_size - p.num_classes = vocab_size - return p - - -class RelativeAttentionHelperTest(test_utils.TestCase, parameterized.TestCase): - - @parameterized.named_parameters( - ('MultiHeadedAttentionXL', attention.MultiHeadedAttentionXL, - attention.MultiHeadedAttention), - ('LocalSelfAttentionXL', attention.LocalSelfAttentionXL, - attention.LocalSelfAttention)) - def testClearRelativeAttentionInTransformerLayer(self, atten_cls, - expected_atten_cls): - """Tests scenarios in clear relative attention in transformer layer.""" - trans_p = attention.TransformerLayer.Params() - # set attention params in transformer layer. - input_dim = 4 - rel_pos_emb_dim = 4 - # Set rel_pos_emb_dim in attention params. - trans_p.tr_atten_tpl.atten_tpl = ( - atten_cls.Params().Set( - input_dim=input_dim, rel_pos_emb_dim=rel_pos_emb_dim)) - new_trans_p = attention.ClearRelativeAttentionInTransformerLayer(trans_p) - tr_atten_tpl = new_trans_p.tr_self_atten_tpl.atten_tpl - self.assertEqual(tr_atten_tpl.cls, expected_atten_cls) - self.assertEqual(tr_atten_tpl.input_dim, input_dim) - - def testClearRelativeAttentionTransformerLayerNotSupportedError(self): - transformer_params = DummyDecoderRNNT.Params() - with self.assertRaises(ValueError): - _ = attention.ClearRelativeAttentionInTransformerLayer(transformer_params) - - def testClearRelativeAttentionAttentionParamsNotSupportedError(self): - trans_p = attention.TransformerLayer.Params() - # MultiHeadedAttention is not supported in ClearRelativeAttention. - attention_params = attention.MultiHeadedAttention.Params() - trans_p.tr_atten_tpl.atten_tpl = attention_params - with self.assertRaises(ValueError): - _ = attention.ClearRelativeAttentionInTransformerLayer(trans_p) - - @parameterized.named_parameters( - ('AttentionParamsNotSupported', _CreateDummyParams( - ['name', 'cls']), attention.ATTEN_TRANSFORMER_XL), - ('AttentionTypeNotSupported', attention.MultiHeadedAttention.Params(), - 'unsupported_atten_type')) - def testUseRelativeAttentionInTransformerLayerValueError( - self, attention_params, attention_type): - """Tests unsupported Use Relative Attention cases.""" - transformer_param = attention.TransformerLayer.Params() - transformer_param.tr_atten_tpl.atten_tpl = attention_params - rel_pos_emb_dim = 4 - with self.assertRaises(ValueError): - _ = attention.UseRelativeAttentionInTransformerLayer( - transformer_param, rel_pos_emb_dim, atten_type=attention_type) - - def testUseRelativeAttentionInTransformerLayerNotSupportedError(self): - """Tests unsupported input transformer params in Use Relative Attention.""" - transformer_params = DummyDecoderRNNT.Params() - with self.assertRaises(ValueError): - _ = attention.UseRelativeAttentionInTransformerLayer( - transformer_params, 4, atten_type=attention.ATTEN_TRANSFORMER_XL) - - @parameterized.named_parameters( - ('MultiHeadedAttention', attention.MultiHeadedAttention, - attention.MultiHeadedAttentionXL, attention.ATTEN_TRANSFORMER_XL), - ('LocalSelfAttention', attention.LocalSelfAttention, - attention.LocalSelfAttentionXL, attention.ATTEN_TRANSFORMER_XL), - ('MultiHeadedAttentionRPE', attention.MultiHeadedAttention, - attention.MultiHeadedAttentionRPE, attention.ATTEN_RPE)) - def testUseRelativeAttentionInTransformerLayer(self, atten_cls, - expected_atten_cls, - atten_type): - """Tests different scenarios in Use Relative Attention.""" - trans_p = attention.TransformerLayer.Params() - # set attenion params in transformer layer. - input_dim = 4 - trans_p.tr_atten_tpl.atten_tpl = atten_cls.Params().Set(input_dim=input_dim) - rel_pos_emb_dim = 4 - new_trans_p = attention.UseRelativeAttentionInTransformerLayer( - trans_p, rel_pos_emb_dim, atten_type=atten_type) - tr_atten_tpl = new_trans_p.tr_self_atten_tpl.atten_tpl - self.assertEqual(tr_atten_tpl.cls, expected_atten_cls) - self.assertEqual(tr_atten_tpl.rel_pos_emb_dim, rel_pos_emb_dim) - self.assertEqual(tr_atten_tpl.input_dim, input_dim) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/batch_utils.py b/lingvo/core/batch_utils.py deleted file mode 100644 index cd01bcd9..00000000 --- a/lingvo/core/batch_utils.py +++ /dev/null @@ -1,61 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Utilities to calculate batch sizes.""" - -from lingvo.core import cluster_factory -from lingvo.core import py_utils - - -def scale_infeed_to_global(infeed_batch_size, use_per_host_infeed): - """Obtains a global batch size from an infeed batch size and cluster configs. - - Args: - infeed_batch_size: int: Per-infeed batch size. - use_per_host_infeed: bool: Whether to use an individual infeed for each - host. - - Returns: - int: Global batch size. - """ - cluster = cluster_factory.Current() - if use_per_host_infeed and cluster.num_tpu_hosts > 0: - if not py_utils.use_tpu(): - raise ValueError('Scaling to TPU hosts without TPUs. {}'.format( - cluster.num_tpu_hosts)) - return infeed_batch_size * cluster.num_tpu_hosts - else: - return infeed_batch_size - - -def scale_split_to_infeed(split_batch_size, use_per_host_infeed): - """Obtains an infeed batch size from a split batch size and cluster configs. - - Args: - split_batch_size: int: Per-split batch size. - use_per_host_infeed: bool: Whether to use an individual infeed for each - host. - - Returns: - int: Per-infeed batch size. - """ - cluster = cluster_factory.Current() - global_batch_size = split_batch_size * cluster.num_splits_per_client - # If use_per_host_infeed, each input op is only responsible - # for generating a subset of the whole batch. - if use_per_host_infeed and cluster.num_tpu_hosts > 0: - return global_batch_size // cluster.num_tpu_hosts - else: - return global_batch_size diff --git a/lingvo/core/batch_utils_test.py b/lingvo/core/batch_utils_test.py deleted file mode 100644 index b39e0587..00000000 --- a/lingvo/core/batch_utils_test.py +++ /dev/null @@ -1,101 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lingvo.core.batch_utils.""" - -import itertools - -from absl.testing import flagsaver -from absl.testing import parameterized -from lingvo import compat as tf -from lingvo.core import batch_utils -from lingvo.core import cluster_factory - - -class BatchUtilsTest(tf.test.TestCase, parameterized.TestCase): - - @parameterized.parameters({'use_per_host_infeed': False}, - {'use_per_host_infeed': True}) - def testScaleInfeedToGlobalCPU(self, use_per_host_infeed): - with cluster_factory.ForTestingWorker(cpus=128): - self.assertEqual( - batch_utils.scale_infeed_to_global(1024, use_per_host_infeed), 1024) - - @parameterized.parameters({'use_per_host_infeed': False}, - {'use_per_host_infeed': True}) - def testScaleInfeedToGlobalGPU(self, use_per_host_infeed): - with cluster_factory.ForTestingWorker(gpus=128): - self.assertEqual( - batch_utils.scale_infeed_to_global(1024, use_per_host_infeed), 1024) - - @parameterized.parameters( - itertools.product( - (False, True), # use_per_host_infeed - (1, 4))) # num_tpu_hosts - def testScaleInfeedToGlobalTPU(self, use_per_host_infeed, num_tpu_hosts): - with flagsaver.flagsaver(xla_device='tpu', enable_asserts=False): - with cluster_factory.ForTestingWorker( - tpus=128, num_tpu_hosts=num_tpu_hosts): - num_infeeds = num_tpu_hosts if use_per_host_infeed else 1 - self.assertEqual( - batch_utils.scale_infeed_to_global(1024, use_per_host_infeed), - 1024 * num_infeeds) - - @parameterized.parameters( - itertools.product( - (False, True), # use_per_host_infeed - (1, 8))) # split_size - def testScaleSplitToInfeedCPU(self, use_per_host_infeed, split_size): - with cluster_factory.ForTestingWorker( - cpus=128, split_size=split_size) as cluster: - num_splits = 128 // split_size - self.assertEqual(cluster.num_splits_per_client, num_splits) - self.assertEqual( - batch_utils.scale_split_to_infeed(1024, use_per_host_infeed), - 1024 * num_splits) - - @parameterized.parameters( - itertools.product( - (False, True), # use_per_host_infeed - (1, 8))) # split_size - def testScaleSplitToInfeedGPU(self, use_per_host_infeed, split_size): - with cluster_factory.ForTestingWorker( - gpus=128, split_size=split_size) as cluster: - num_splits = 128 // split_size - self.assertEqual(cluster.num_splits_per_client, num_splits) - self.assertEqual( - batch_utils.scale_split_to_infeed(1024, use_per_host_infeed), - 1024 * num_splits) - - @parameterized.parameters( - itertools.product( - (False, True), # use_per_host_infeed - (1, 8), # split_size - (1, 4))) # num_tpu_hosts - def testScaleSplitToInfeedTPU(self, use_per_host_infeed, split_size, - num_tpu_hosts): - with cluster_factory.ForTestingWorker( - tpus=128, split_size=split_size, - num_tpu_hosts=num_tpu_hosts) as cluster: - num_splits = 128 // split_size - num_infeeds = num_tpu_hosts if use_per_host_infeed else 1 - self.assertEqual(cluster.num_splits_per_client, num_splits) - self.assertEqual( - batch_utils.scale_split_to_infeed(1024, use_per_host_infeed), - 1024 * num_splits // num_infeeds) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/beam_search_helper.py b/lingvo/core/beam_search_helper.py deleted file mode 100644 index 0f1d6cde..00000000 --- a/lingvo/core/beam_search_helper.py +++ /dev/null @@ -1,795 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Helper class for implementing a beam search decoder. - -Individual models just need to provide a few callback functions. -""" - -import collections -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import ops -from lingvo.core import py_utils - -from tensorflow.python.ops import inplace_ops - -# TODO(yonghui): -# 1) Change the tensor shape [max_decoder_time_steps, batch_size * -# num_hyps_per_beam] to [max_decoder_time_steps, num_hyps_per_beam, -# batch_size] to avoid confusing and mis-interpretation of the results. - -# Defines a namedtuple to store the results of BeamSearchDecode. It contains -# the following entries: -# done_hyps: A string Tensor of shape -# [max_decoder_time_steps, batch_size * num_hyps_per_beam] which can be -# either an empty string, or a serialized Hypothesis proto. The non-empty -# hyps in done_hyps are terminated hypotheses. The 'h'-th hyp for sample -# 'b' at time step 't' can be found at done_hyps[t, batch_size * h + b]. -# topk_hyps: A string Tensor of shape [batch_size, num_hyps_per_beam]. -# topk_hyps[b, h] is the h-th hypothesis for the sample 'b' in the -# batch, which can either be an empty string or a serialized Hypothesis -# proto. -# topk_ids: Int32 Tensor of shape [batch_size * num_hyps_per_beam, -# target_seq_len] which contains the IDs of the targets in each of the -# hypotheses in the beam for the samples in the batch. For sample -# 'b' in the batch, the h-th hypothesis for this sample can be found at -# position [b * num_hyps_per_beam + h, :]. -# topk_lens: Int32 Tensor of shape [batch_size * num_hyps_per_beam] which -# indicates the length (>=0) of each of the hypotheses. -# topk_scores: Float32 Tensor of shape [batch_size * num_hyps_per_beam] -# containing the scores (negative log probabilities) of each of the -# hypotheses in the beam. -# topk_decoded: A string Tensor of shape [batch_size * num_hyps_per_beam] which -# contains the decoded target strings in each of the hypotheses in the -# beam for the samples in the batch. The 'h'-th hyp for sample 'b' can -# be found at topk_decoded[b * num_hyps_per_beam + h] -BeamSearchDecodeOutput = collections.namedtuple( - 'BeamSearchDecodeOutput', - [ - 'done_hyps', 'topk_hyps', 'topk_ids', 'topk_lens', 'topk_scores', - 'topk_decoded', 'other_states' - ], -) -# Make the last attribute default to None. -BeamSearchDecodeOutput.__new__.__defaults__ = (None,) - - -# Keys in fusion state that can be two dimensional, with the batch element in -# the second dimension, requiring special treatment in hypothesis reordering. -POSSIBLY_TIME_MAJOR_STATE_KEYS = [ - 'misc_states.fusion_states.lm_states.prev_ids', - 'misc_states.fusion_states.lm_states.prev_paddings', - 'fusion_states.lm_states.prev_ids', - 'fusion_states.lm_states.prev_paddings', -] - - -class BeamSearchHelper(base_layer.BaseLayer): - """Helper class for performing beam search. - - The user of this helper class needs to implement three callbacks. - - This callback is called once only at the beginning of beam search: - - .. code-block:: none - - def InitBeamSearchState(theta, encoder_outputs, num_hyps_per_beam): - Args: - theta: A NestedMap object containing weights' values of this layer and - its children layers. - encoder_outputs: A NestedMap computed by encoder. - num_hyps_per_beam: An int, number hyps to keep for source sentence. - - Returns: - A tuple (initial_results, states): - - - initial_results: a `.NestedMap` of initial results. It must contain - the 'atten_probs' and 'log_probs' tensors. Optionally it may - contain 'step_ids'. - - - log_probs: The initial log probs for each of the tokens in the - target vocab of shape [num_hyps_per_beam * src_batch, vocab_size]. - src_batch "b" and hyp_per_beam "h" is represented at index - ``(h * src_batch + b)``. - - atten_probs: The initial attention probs, of shape - [num_hyps_per_beam * src_batch, src_len]. src_batch "b" and - hyp_per_beam "h" is represented at index ``(h * src_batch + b)``. - - step_ids: Optional. The initial ids of shape [num_hyps_per_beam * - src_batch, 1] for which to start the beam search. src_batch "b" - hyp_per_beam "h" is represented at index ``(h * src_batch + b)``. - If not specified, defaults to a tensor filled with target_sos_id. - - - states: a `.NestedMap` of tensors representing states that the - client would like to keep track of for each hyp. - - This callback is called once every decoding time step before beam_search_step - is called: - - .. code-block:: none - - def PreBeamSearchStepCallback(theta, - encoder_outputs, - step_ids, - in_states, - num_hyps_per_beam): - Args: - theta: A NestedMap object containing weights' values of this layer and - its children layers. - encoder_outputs: A NestedMap computed by encoder. - step_ids: A tensor of shape [num_hyps_per_beam * src_batch, 1]. - in_states: A `.NestedMap` of tensors representing states that the - clients would like to keep track of for each of the active hyps. - - Returns: - A tuple (results, out_states): - - - results: A `.NestedMap` of beam search results. It should contain - the 'atten_probs' and 'log_probs' tensors at the minimal. - Optionally it may contain 'is_last_chunk' if it is decoding a - neural transducer model. - - - atten_probs: The updated attention probs, of shape - [num_hyps_per_beam * src_batch, src_len]. src_batch "b" and - hyp_per_beam "h" is represented at index ``(h * src_batch + b)``. - - log_probs: Log prob for each of the tokens in the target vocab. - This is of shape [num_hyps_per_beam * src_batch, vocab_size]. - src_batch "b" and hyp_per_beam "h" is represented at index - ``(h * src_batch + b)``. - - is_last_chunk: Whether each of the hyp is at the end of a chunk. - If non-empty, it has shape [num_hyps_per_beam * src_batch, 1]. - - - out_states: A `.NestedMap`. The updated states. This 'out_states' - should be of the exact same structure as 'in_states' - - This callback is called once every decoding time step after beam_search_step - is called: - - .. code-block:: none - - def PostBeamSearchStepCallback(theta, - encoder_outputs, - new_step_ids, - other_states): - Args: - theta: A NestedMap object containing weights' values of this layer and - its children layers. - encoder_outputs: A NestedMap computed by encoder. - new_step_ids: Token ids for the next beam search step. - other_states: A `.NestedMap`. - - Returns: - final_states, A `.NestedMap`. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_hyps_per_beam', 8, - 'Num of hyps to keep per beam during decoding.') - p.Define( - 'target_seq_length_ratio', 1.0, - 'Ratio of the average target sequence length over the average ' - 'source sequence length.') - p.Define('length_normalization', 0.0, - 'Beam search length normalization ratio.') - p.Define('coverage_penalty', 0.0, 'Beam search coverage penalty.') - p.Define( - 'valid_eos_max_logit_delta', 5.0, - 'During beam search, allow to terminate a hyp only if its ' - 'logit is no more than than this value away from the logit of the ' - 'best candidate.') - p.Define( - 'beam_size', 3.0, - 'The maximum difference between best hyp and the worst in a beam.' - ' This allows to prune our search when none of the active hyp is' - ' close enough to the current best.') - p.Define('target_sos_id', 1, 'Id of the start of sentence token.') - p.Define('target_eos_id', 2, 'Id of the end of sentence token.') - p.Define( - 'target_eoc_id', -1, - 'Id of the end of chunk token. Used by neural transducer only.' - ' Set this id to a non-negative value only for NT.') - p.Define( - 'target_seq_len', 0, 'Maximum allowed target seq length. Note ' - 'that decoding terminates if an end of sentence token ' - 'is not emitted after target_seq_len decode steps.') - p.Define( - 'merge_paths', False, 'If true, hyps which are identical when ' - 'epsilons are removed will be combined into a single hyp. The ' - 'probability for that combined hyp will be the sum of the ' - 'probabilities of the component hyps. This can only be applied ' - 'for epsilon-emitting models (RNN-T and NT).') - p.Define( - 'allow_empty_terminated_hyp', True, 'Whether it is okay to consider a ' - 'hyp that consists only of epsilons as terminated. By default this ' - 'is true, as an utterance may consist of silence. It should be set ' - 'to false when EMBR training epsilon-emitting models (e.g., RNN-T), ' - 'which are prone to emit all-epsilon hyps even in the presence of ' - 'speech. Note that a hyp that terminates in EOS is not considered ' - 'empty, so this flag has no effect for non-epsilon-emitting models.') - p.Define( - 'ensure_full_beam', False, 'If True, we will not terminate the search ' - 'until both of these conditions are satisfied: we have found ' - 'num_hyps_per_beam terminated hyps AND no active hyps have a score ' - 'within beam_size of the best terminated hyp. If False, only the ' - 'second condition must be satisfied. Note that in either case, we can ' - 'also terminate if we have run for target_seq_len steps. Generally ' - 'this should be False unless beam search is being run as part of ' - 'minimum word error rate training.') - p.Define( - 'force_eos_in_last_step', False, - 'For all active hyps that are still on the beam after target_seq_len ' - 'steps, return partial hyps with EOS set as the last token.') - p.Define( - 'batch_major_state', True, 'If True, we use batch as the major ' - 'dimension of the hyp states. Otherwise, timing becomes the major ' - 'dimension, and the gathers are performed along the second-to-major ' - 'dimension.') - p.Define( - 'batch_major_compute', False, 'If True, the target batch dimension ' - 'is organized as num_beams by num_hyps_per_beam during the ' - 'ExtendStep computation and the cache is stored following this order. ' - 'So the topk indices into the cache for ReOrderHyps needs to be ' - 'reordered before usage. Otherwise, the indices will be directly used ' - 'without extra transformation.') - p.Define( - 'short_seq_limit', 0, - 'An integer, the sequence length limit for using early stop ' - 'method in attention layer (batch-major implementation). The sequence ' - 'is always treated as the default long sequence for decoding when the ' - 'limit is set to 0. For typical mt transformer config ' - '(batch 16, sequence length 150), the break even point is around 40 ' - 'on TPU V3, and 50 on TPU V2. This may slightly change for ' - 'different batch size and sequence length, which requires more ' - 'experiments to set the value.') - p.Define( - 'local_eos_threshold', -100.0, - 'During beam search, allow to terminate a hyp if the local score ' - 'for is greater than local_eos_threshold.') - p.name = 'beam_search' - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self._model_uses_eoc_id = p.target_eoc_id >= 0 - - def _BeamSearchStep(self, theta, encoder_outputs, cur_step, step_ids, - core_bs_states, other_states, num_hyps_per_beam, - pre_beam_search_step_callback, - post_beam_search_step_callback): - """Extend beam search hyps for one step. - - | num_beams = Number of source sequences to be decoded. - | num_hyps_per_beam = Number of hyps to keep per source sequence. - | num_hyps = num_beams * num_hyps_per_beam - | src_seq_len = Number of time steps in the source sequence. - | src_batch = Number of examples in the source sequence. - | tgt_seq_len = Maximum allowed time steps in the target sequence. - | tgt_batch = num_hyps_per_beam * src_batch - - Args: - theta: A `.NestedMap` object containing weights' values of the decoder - layer and its children layers. - encoder_outputs: A `.NestedMap` containing encoder outputs to be passed to - the callbacks. - cur_step: A scalar int tensor, the current time step, 0-based. - step_ids: An int tensor of shape [num_hyps, 1]. The input ids to the - current search step. - core_bs_states: A tuple of core beam search states. This list is - maintained by this helper class. - other_states: A `.NestedMap` of other beam search states. This - `.NestedMap` is managed and updated by the client. It is expected that - each of its member tensors are of rank >= 1. t[i, ...] is the state of - the i-th hyp at the beginning of this search step. - num_hyps_per_beam: Num of hyps to keep per beam. - pre_beam_search_step_callback: The `PreBeamSearchStepCallback` callback. - See class header comments for more details. - post_beam_search_step_callback: The `PostBeamSearchStepCallback` callback. - See class header comments for more details. - - Returns: - A tuple of following elements for the next beam search step, - (next step, all_done, step_ids, core_bs_states, other_states) - """ - p = self.params - - bs_results, other_states = pre_beam_search_step_callback( - theta, encoder_outputs, step_ids, other_states, num_hyps_per_beam) - - (best_scores, cumulative_scores, in_scores, in_hyps, in_prev_hyps, - in_done_hyps, in_atten_probs) = core_bs_states - - (out_best_scores, out_cumulative_scores, out_scores, out_hyps, - out_prev_hyps, out_done_hyps, out_atten_probs, - all_done) = ops.beam_search_step( - tf.cast(bs_results.log_probs, dtype=p.dtype), - tf.cast(bs_results.atten_probs, dtype=p.dtype), - best_scores, - cumulative_scores, - in_scores, - in_hyps, - in_prev_hyps, - in_done_hyps, - in_atten_probs, - bs_results.is_last_chunk if self._model_uses_eoc_id else [], - cur_step, - eoc_id=p.target_eoc_id, - eos_id=p.target_eos_id, - beam_size=p.beam_size, - num_hyps_per_beam=num_hyps_per_beam, - valid_eos_max_logit_delta=p.valid_eos_max_logit_delta, - merge_paths=p.merge_paths, - allow_empty_terminated_hyp=p.allow_empty_terminated_hyp, - ensure_full_beam=p.ensure_full_beam, - force_eos_in_last_step=p.force_eos_in_last_step, - local_eos_threshold=p.local_eos_threshold) - - new_step_ids = tf.reshape(out_hyps[cur_step, :], tf.shape(step_ids)) - new_step_ids.set_shape(step_ids.get_shape()) - - old_hyp_ids = tf.reshape( - tf.slice(out_prev_hyps, begin=[cur_step, 0], size=[1, -1]), [-1]) - - if p.batch_major_compute: - # Transformed the indices into the key/value cache for fast decoding - # (prefix_states in other_states) due to the num_hyps dimension of - # cache is computed as num_beams by num_hyps_per_beam, which is different - # from the old_hyp_ids assumption (num_hyps_per_beam by num_beams). - # Both transpose and recomputation are required to correct the indices. - num_beams = tf.shape(best_scores)[0] - old_hyp_ids_in_cache_order = tf.reshape( - tf.transpose(tf.reshape(old_hyp_ids, [num_hyps_per_beam, -1])), [-1]) - old_hyp_ids_in_cache_order = ( - (old_hyp_ids_in_cache_order % num_beams) * num_hyps_per_beam + - old_hyp_ids_in_cache_order // num_beams) - - new_bs_states = (out_best_scores, out_cumulative_scores, out_scores, - out_hyps, out_prev_hyps, out_done_hyps, out_atten_probs) - - def ReOrderHyps(key, x_in): - """Reorders x_in based on prev hyp ids.""" - if (isinstance(x_in, tf.Tensor) and x_in.shape.ndims and - x_in.shape.ndims > 0): - if x_in.shape.ndims > 2 and not p.batch_major_state: - # Use corrected indices only here for batch major compute as key/value - # caches are the states being affected. - correct_old_hyp_ids = ( - old_hyp_ids_in_cache_order - if p.batch_major_compute else old_hyp_ids) - x_out = tf.gather(x_in, correct_old_hyp_ids, axis=1) - elif key in POSSIBLY_TIME_MAJOR_STATE_KEYS: - x_out = tf.gather(x_in, old_hyp_ids, axis=-1) - else: - x_out = tf.gather(x_in, old_hyp_ids) - x_out.set_shape(x_in.get_shape()) - return x_out - else: - return x_in - - new_other_states = other_states.TransformWithKey(ReOrderHyps) - - final_other_states = post_beam_search_step_callback(theta, encoder_outputs, - new_step_ids, - new_other_states) - - return (cur_step + 1, all_done, new_step_ids, new_bs_states, - final_other_states) - - def BeamSearchDecode(self, - theta, - encoder_outputs, - num_hyps_per_beam_override=0, - init_beam_search_state=None, - pre_beam_search_step_callback=None, - post_beam_search_step_callback=None, - max_steps=None): - """Performs beam-search based decoding. - - Args: - theta: A NestedMap object containing weights' values of the decoder layer - and its children layers. - encoder_outputs: A NestedMap containing encoder outputs to be passed to - the callbacks. Mostly opaque to BeamSearchHelper, except that it should - contain either a 'seq_lengths' field of shape [source_batch_size] or - a 'paddings' field of shape [source_max_lengths, source_batch_size]. - num_hyps_per_beam_override: If set to a value <= 0, this parameter is - ignored. If set to a value > 0, then this value will be used to override - `p.num_hyps_per_beam`. - init_beam_search_state: The `InitBeamSearchState` callback. Please refer - to the class header comments for more details. - pre_beam_search_step_callback: The `PreBeamSearchStepCallback` callback. - Please refer to the class header comments for more details. - post_beam_search_step_callback: The `PostBeamSearchStepCallback` callback. - Please refer to the class header comments for more details. - max_steps: maximum beam search steps. If None, use - self.params.target_seq_len. - - Returns: - A `BeamSearchDecodeOutput`. - """ - p = self.params - num_hyps_per_beam = p.num_hyps_per_beam - if num_hyps_per_beam_override > 0: - num_hyps_per_beam = num_hyps_per_beam_override - if max_steps is None: - max_steps = p.target_seq_len - - initial_results, other_states = init_beam_search_state( - theta, encoder_outputs, num_hyps_per_beam) - - num_hyps = tf.shape(initial_results.log_probs)[0] - num_beams = num_hyps // num_hyps_per_beam - - if 'step_ids' in initial_results: - # [num_hyps, 1] - step_ids = tf.ensure_shape(initial_results.step_ids, [None, 1]) - else: - step_ids = tf.fill([num_hyps, 1], - tf.constant(p.target_sos_id, dtype=tf.int32)) - - min_score = -1e36 - best_scores = (tf.zeros(shape=[num_beams], dtype=p.dtype) + min_score) - cumulative_scores = tf.zeros(shape=[num_hyps], dtype=p.dtype) - in_scores = tf.zeros([max_steps, num_hyps], dtype=p.dtype) - in_hyps = tf.zeros([max_steps, num_hyps], dtype=tf.int32) - in_prev_hyps = tf.zeros([max_steps, num_hyps], dtype=tf.int32) - in_done_hyps = tf.zeros([max_steps, num_hyps], dtype=tf.string) - bs_atten_probs = tf.zeros( - [max_steps, num_hyps, - tf.shape(initial_results.atten_probs)[1]], - dtype=p.dtype) - cur_step = tf.constant(0, dtype=tf.int32) - all_done = tf.constant(False, dtype=tf.bool) - core_bs_states = (best_scores, cumulative_scores, in_scores, in_hyps, - in_prev_hyps, in_done_hyps, bs_atten_probs) - - def LoopContinue(cur_step, all_done, unused_step_ids, unused_core_bs_states, - unused_other_states_list): - return tf.math.logical_and(cur_step < max_steps, - tf.math.logical_not(all_done)) - - def LoopBody(cur_step, unused_all_done, step_ids, core_bs_states, - other_states_list): - (cur_step, all_done, new_step_ids, new_bs_states, - new_other_states) = self._BeamSearchStep( - theta, encoder_outputs, cur_step, step_ids, core_bs_states, - other_states.Pack(other_states_list), num_hyps_per_beam, - pre_beam_search_step_callback, post_beam_search_step_callback) - return (cur_step, all_done, new_step_ids, new_bs_states, - new_other_states.Flatten()) - - flat_other_states = other_states.Flatten() - _, _, _, final_bs_states, flat_final_other_states = tf.while_loop( - LoopContinue, - LoopBody, - loop_vars=(cur_step, all_done, step_ids, core_bs_states, - flat_other_states), - parallel_iterations=10, - back_prop=False, - swap_memory=False, - shape_invariants=(tf.TensorShape(cur_step.get_shape()), - tf.TensorShape(all_done.get_shape()), - tf.TensorShape(step_ids.get_shape()), - _GetShapes(core_bs_states), - _GetShapes(flat_other_states, none_shapes=True))) - # [target_seq_len, num_beams * num_hyps_per_beam]. - final_done_hyps = final_bs_states[5] - final_other_states = other_states.Pack(flat_final_other_states) - - # Assume that `paddings` has shape [source_max_lengths, source_batch_size] - # by default, and compute `encoded_seq_lengths` accordingly. This can be - # overridden by directly passing `seq_lengths` in the `encoder_outputs` - # NestedMap. - encoded_seq_lengths = getattr(encoder_outputs, 'seq_lengths', None) - if encoded_seq_lengths is None: - source_paddings = encoder_outputs.padding - if isinstance(source_paddings, py_utils.NestedMap): - encoded_seq_lengths = tf.cast( - tf.round( - tf.reduce_sum(1.0 - tf.transpose(source_paddings.Flatten()[0]), - 1)), tf.int32) - else: - encoded_seq_lengths = tf.cast( - tf.round(tf.reduce_sum(1.0 - tf.transpose(source_paddings), 1)), - tf.int32) - - # [num_beams, num_hyps_per_beam]. - topk_hyps = ops.top_k_terminated_hyps( - final_done_hyps, - encoded_seq_lengths, - k=num_hyps_per_beam, - num_hyps_per_beam=num_hyps_per_beam, - length_normalization=p.length_normalization, - coverage_penalty=p.coverage_penalty, - target_seq_length_ratio=p.target_seq_length_ratio, - eoc_id=p.target_eoc_id, - merge_paths=p.merge_paths) - # [num_beams * num_hyps_per_beam, ...]. - max_seq_length = 0 if isinstance(max_steps, tf.Tensor) else max_steps - topk_ids, topk_lens, topk_scores = ops.unpack_hyp( - tf.reshape(topk_hyps, [-1]), max_seq_length=max_seq_length) - # [num_beams, num_hyps_per_beam]. - topk_scores = tf.reshape(topk_scores, tf.shape(topk_hyps)) - - return BeamSearchDecodeOutput(final_done_hyps, topk_hyps, topk_ids, - topk_lens, topk_scores, None, - final_other_states) - - -def _GetShapes(tensors, none_shapes=False): - """Util for getting nested structure of shapes from structure of tensors. - - Args: - tensors: Structure of Tensors to get shapes for. - none_shapes: Returns None shapes if true. - - Returns: - The same structure as tensors but of corresponding `TensorShape` objects. - """ - shapes = [] - for t in tf.nest.flatten(tensors): - shape = t.get_shape() if isinstance(t, tf.Tensor) else None - if none_shapes: - if shape: - shapes.append(tf.TensorShape([None] * len(shape))) - else: - shapes.append(tf.TensorShape(None)) - else: - shapes.append(tf.TensorShape(shape)) - - return type(tensors)(tf.nest.pack_sequence_as(tensors, shapes)) - - -def MergeBeamSearchOutputs(max_hyps_per_beam, beam_search_outputs): - """Merges beam search hyps from multiple decoders. - - Args: - max_hyps_per_beam: the number of top hyps in the merged results. Must be - less than or equal to total number of input hyps. - beam_search_outputs: a list of BeamSearchDecodeOutput objects. Must share - the same source_batch and max sequence length. - - Returns: - A BeamSearchDecodeOutput object containing max_hyps_per_beam hypotheses per - beam. - """ - source_batch = tf.shape(beam_search_outputs[0].topk_hyps)[0] - value_dict = {} - for output in beam_search_outputs: - hyps_per_beam = py_utils.with_dependencies([ - py_utils.assert_equal(source_batch, - tf.shape(output.topk_hyps)[0]), - ], - tf.shape(output.topk_hyps)[1]) - for k, v in output._asdict().items(): - if v is None: - continue - if k == 'done_hyps': - v = tf.transpose(v) - if k not in value_dict: - value_dict[k] = [] - value_dict[k].append(tf.reshape(v, [source_batch, hyps_per_beam, -1])) - - # Concatenate the tensors along the 'num_hyps_per_beam' dimension. - concatenated = {} - for k, values in value_dict.items(): - if len(values) != len(beam_search_outputs): - raise ValueError('Incomplete values for %s: %s' % - (k, beam_search_outputs)) - concatenated[k] = tf.concat(values, axis=1) - - scores = concatenated['topk_scores'] - scores = tf.where( - tf.equal(concatenated['topk_lens'], 0), tf.fill(tf.shape(scores), -1e6), - scores) - scores = tf.squeeze(scores, -1) - - # Select top max_hyps_per_beam indices per beam. - _, top_indices = tf.nn.top_k(scores, max_hyps_per_beam) - batch_ids = tf.tile( - tf.expand_dims(tf.range(source_batch), -1), [1, max_hyps_per_beam]) - # [source_batch, max_hyps_per_beam, 2] - gather_indices = tf.stack([batch_ids, top_indices], axis=-1) - - # Gather the merged top hyps according to 'gather_indices'. - top = beam_search_outputs[0]._asdict() - total_hyps = source_batch * max_hyps_per_beam - for k, v in concatenated.items(): - v = tf.gather_nd(v, gather_indices) - if k == 'done_hyps': - v = tf.transpose(tf.reshape(v, [total_hyps, -1])) - elif k == 'topk_hyps': - v = tf.reshape(v, [source_batch, max_hyps_per_beam]) - elif k == 'topk_ids': - v = tf.reshape(v, [total_hyps, -1]) - elif k in ('topk_lens', 'topk_scores', 'topk_decoded'): - v = tf.reshape(v, [total_hyps]) - else: - raise ValueError('Unexpected field: %s' % k) - top[k] = v - return BeamSearchDecodeOutput(**top) - - -class GreedySearchHelper(base_layer.BaseLayer): - """Helper class for performing greedy decoding. - - The user of this helper class needs to implement three callbacks just as in a - beam search decoder. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('target_sos_id', 1, 'Id of the start of sentence token.') - p.Define('target_eos_id', 2, 'Id of the end of sentence token.') - p.Define( - 'target_seq_len', 0, 'Maximum allowed target seq length. Note ' - 'that decoding terminates if an end of sentence token ' - 'is not emitted after target_seq_len decode steps.') - p.name = 'greedy_search' - return p - - def _GreedySearchStep(self, theta, encoder_outputs, cur_step, step_ids, - hyp_ids, hyp_lens, done_hyps, other_states, - pre_beam_search_step_callback, - post_beam_search_step_callback): - """Extend greedy search hyps for one step. - - Args: - theta: A `.NestedMap` object containing weights' values of the decoder - layer and its children layers. - encoder_outputs: A `.NestedMap` containing encoder outputs to be passed to - the callbacks. - cur_step: A scalar int tensor, the current time step, 0-based. - step_ids: An int tensor of shape [num_hyps, 1]. The input ids to the - current search step. - hyp_ids: An int tensor of shape [num_hyps, tgt_seq_len]. - hyp_lens: Valid length of all the hyps. Tokens after eos ids are not - counted. - done_hyps: Whether or not a hyp has finished. - other_states: A `.NestedMap` of other beam search states. This - `.NestedMap` is managed and updated by the client. It is expected that - each of its member tensors are of rank >= 1. t[i, ...] is the state of - the i-th hyp at the beginning of this search step. - pre_beam_search_step_callback: The `PreBeamSearchStepCallback` callback. - See class header comments for more details. - post_beam_search_step_callback: The `PostBeamSearchStepCallback` callback. - See class header comments for more details. - - Returns: - A tuple of following elements for the next greedy search step, - (next step, new_step_ids, hyp_ids, hyp_lens, done_hyps, other_states) - """ - p = self.params - # Increment hyp_lens by 1 if the hyp is not finished yet. - hyp_lens = hyp_lens + (1 - tf.cast(done_hyps, tf.int32)) - - bs_results, new_other_states = pre_beam_search_step_callback( - theta, encoder_outputs, step_ids, other_states, 1) # num_hyps_per_beam - new_step_ids = tf.math.argmax(bs_results.log_probs, 1) - new_step_ids = tf.cast(new_step_ids, tf.int32) - new_step_ids = tf.reshape(new_step_ids, tf.shape(step_ids)) - final_other_states = post_beam_search_step_callback(theta, encoder_outputs, - new_step_ids, - new_other_states) - - # Stash new_step_ids into the right slot. - new_step_ids_1d = tf.reshape(new_step_ids, [-1]) - hyp_ids = inplace_ops.alias_inplace_update(hyp_ids, cur_step, - new_step_ids_1d) - # Update done_hyps if the current step_ids is the end of sequence token. - done_hyps = tf.math.logical_or(done_hyps, - tf.equal(new_step_ids_1d, p.target_eos_id)) - - return (cur_step + 1, new_step_ids, hyp_ids, hyp_lens, done_hyps, - final_other_states) - - def GreedySearchDecode(self, - theta, - encoder_outputs, - init_beam_search_state=None, - pre_beam_search_step_callback=None, - post_beam_search_step_callback=None, - max_steps=None): - """Performs greedy-search based decoding. - - Args: - theta: A NestedMap object containing weights' values of the decoder layer - and its children layers. - encoder_outputs: A NestedMap containing encoder outputs to be passed to - the callbacks. - init_beam_search_state: The `InitBeamSearchState` callback. Please refer - to the class header comments for more details. - pre_beam_search_step_callback: The `PreBeamSearchStepCallback` callback. - Please refer to the class header comments for more details. - post_beam_search_step_callback: The `PostBeamSearchStepCallback` callback. - Please refer to the class header comments for more details. - max_steps: maximum beam search steps. If None, use - self.params.target_seq_len. - - Returns: - A tuple (hyp_ids, hyp_lens, done_hyps). Note that num_hyps is same as - src_batch_size. - - - hyp_ids: [num_hyps, max_step]. Hyps end with token if the - token is encountered during search. - - hyp_lens: [num_hyps]. - - done_hyps: [num_hyps], whether or not an eos is encountered. - """ - p = self.params - if max_steps is None: - max_steps = p.target_seq_len - - initial_results, other_states = init_beam_search_state( - theta, - encoder_outputs, - 1 # num_hyps_per_beam - ) - - num_hyps = tf.shape(initial_results.log_probs)[0] - - if 'step_ids' in initial_results: - # [num_hyps, 1] - step_ids = tf.ensure_shape(initial_results.step_ids, [None, 1]) - else: - step_ids = tf.fill([num_hyps, 1], - tf.constant(p.target_sos_id, dtype=tf.int32)) - - cur_step = tf.constant(0, dtype=tf.int32) - done_hyps = inplace_ops.empty(shape=[num_hyps], dtype=tf.bool, init=True, - name='done_hyps') - hyp_lens = inplace_ops.empty(shape=[num_hyps], dtype=tf.int32, init=True, - name='hyp_lens') - hyp_ids = inplace_ops.empty( - shape=[max_steps, num_hyps], dtype=tf.int32, init=True, - name='hyp_ids') - - def LoopContinue(cur_step, unused_step_ids, unused_hyp_ids, unused_hyp_lens, - done_hyps, unused_other_states_list): - return tf.math.logical_and(cur_step < max_steps, - tf.math.logical_not(tf.reduce_all(done_hyps))) - - def LoopBody(cur_step, step_ids, hyp_ids, hyp_lens, done_hyps, - other_states_list): - (cur_step, new_step_ids, hyp_ids, hyp_lens, done_hyps, - new_other_states) = self._GreedySearchStep( - theta, encoder_outputs, cur_step, - step_ids, hyp_ids, hyp_lens, done_hyps, - other_states.Pack(other_states_list), pre_beam_search_step_callback, - post_beam_search_step_callback) - return (cur_step, new_step_ids, hyp_ids, hyp_lens, done_hyps, - new_other_states.Flatten()) - - flat_other_states = other_states.Flatten() - _, _, final_hyp_ids, final_hyp_lens, final_done_hyps, _ = tf.while_loop( - LoopContinue, - LoopBody, - loop_vars=(cur_step, step_ids, hyp_ids, hyp_lens, done_hyps, - flat_other_states), - parallel_iterations=10, - back_prop=False, - swap_memory=False, - shape_invariants=(tf.TensorShape(cur_step.get_shape()), - tf.TensorShape(step_ids.get_shape()), - tf.TensorShape(hyp_ids.get_shape()), - tf.TensorShape(hyp_lens.get_shape()), - tf.TensorShape(done_hyps.get_shape()), - _GetShapes(flat_other_states, none_shapes=True))) - - # transpose hyp_ids so it matches BeamSearchDecode's output - final_hyp_ids = tf.transpose(final_hyp_ids) - return final_hyp_ids, final_hyp_lens, final_done_hyps diff --git a/lingvo/core/beam_search_helper_test.py b/lingvo/core/beam_search_helper_test.py deleted file mode 100644 index bd39eada..00000000 --- a/lingvo/core/beam_search_helper_test.py +++ /dev/null @@ -1,308 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for beam_search_helper.""" - -import lingvo.compat as tf -from lingvo.core import beam_search_helper -from lingvo.core import py_utils -from lingvo.core import test_utils -import numpy as np - - -def GetBeamSearchHelperResults(sess, num_hyps_per_beam, pass_seq_lengths=False): - np.random.seed(9384758) - tf.random.set_seed(8274758) - vocab_size = 12 - src_len = 5 - tgt_len = 7 - src_batch_size = 2 - tgt_batch_size = src_batch_size * num_hyps_per_beam - p = beam_search_helper.BeamSearchHelper.Params().Set( - name='bsh', target_seq_len=tgt_len) - bs_helper = p.Instantiate() - - def InitBeamSearchState(unused_theta, unused_encoder_outputs, - unused_num_hyps_per_beam): - atten_probs = tf.constant( - np.random.normal(size=(tgt_batch_size, src_len)), dtype=tf.float32) - return (py_utils.NestedMap({ - 'log_probs': tf.zeros([tgt_batch_size, vocab_size]), - 'atten_probs': atten_probs, - }), py_utils.NestedMap({'atten_probs': atten_probs})) - - def PreBeamSearchStepCallback(unused_theta, unused_encoder_outputs, - unused_step_ids, states, - unused_num_hyps_per_beam): - atten_probs = tf.identity(states.atten_probs) - logits = tf.random.normal([tgt_batch_size, vocab_size], seed=8273747) - return (py_utils.NestedMap({ - 'atten_probs': atten_probs, - 'log_probs': logits - }), states) - - def PostBeamSearchStepCallback(unused_theta, unused_encoder_outputs, - unused_new_step_ids, states): - return states - - src_enc = tf.random.normal([src_len, src_batch_size, 8], seed=982774838) - src_enc_padding = tf.constant( - [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 1.0], [1.0, 1.0]], - dtype=tf.float32) - encoder_outputs = py_utils.NestedMap(encoded=src_enc, padding=src_enc_padding) - if pass_seq_lengths: - encoder_outputs['seq_lengths'] = tf.constant([4, 3], dtype=tf.int32) - - theta = py_utils.NestedMap() - decoder_output = bs_helper.BeamSearchDecode(theta, encoder_outputs, - num_hyps_per_beam, - InitBeamSearchState, - PreBeamSearchStepCallback, - PostBeamSearchStepCallback) - - topk_ids, topk_lens, topk_scores = sess.run([ - decoder_output.topk_ids, decoder_output.topk_lens, - decoder_output.topk_scores - ]) - return topk_ids, topk_lens, topk_scores - - -class BeamSearchHelperTest(test_utils.TestCase): - - # TODO(yonghui): Add more thorough tests. - def testBeamSearchHelper(self): - with self.session(use_gpu=False) as sess: - topk_ids, topk_lens, topk_scores = GetBeamSearchHelperResults( - sess, num_hyps_per_beam=3) - print(np.array_repr(topk_ids)) - print(np.array_repr(topk_lens)) - print(np.array_repr(topk_scores)) - expected_topk_ids = [[4, 3, 4, 3, 2, 0, 0], [4, 3, 11, 2, 0, 0, 0], - [4, 3, 6, 2, 0, 0, 0], [6, 0, 4, 6, 6, 11, 2], - [6, 0, 4, 6, 1, 2, 0], [6, 0, 4, 6, 6, 2, 0]] - expected_topk_lens = [5, 4, 4, 7, 6, 6] - expected_topk_scores = [[8.27340603, 6.26949024, 5.59490776], - [9.74691486, 8.46679497, 7.14809656]] - self.assertEqual(expected_topk_ids, topk_ids.tolist()) - self.assertEqual(expected_topk_lens, topk_lens.tolist()) - self.assertAllClose(expected_topk_scores, topk_scores) - - def testBeamSearchHelperHypsOne(self): - with self.session(use_gpu=False) as sess: - topk_ids, topk_lens, topk_scores = GetBeamSearchHelperResults( - sess, num_hyps_per_beam=1) - print(np.array_repr(topk_ids)) - print(np.array_repr(topk_lens)) - print(np.array_repr(topk_scores)) - expected_topk_ids = [[9, 2, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0]] - expected_topk_lens = [2, 0] - expected_topk_scores = [[3.778749], [0.0]] - self.assertEqual(expected_topk_ids, topk_ids.tolist()) - self.assertEqual(expected_topk_lens, topk_lens.tolist()) - self.assertAllClose(expected_topk_scores, topk_scores) - - def testBeamSearchHelperWithSeqLengths(self): - with self.session(use_gpu=False) as sess: - topk_ids, topk_lens, topk_scores = GetBeamSearchHelperResults( - sess, num_hyps_per_beam=3, pass_seq_lengths=True) - print(np.array_repr(topk_ids)) - print(np.array_repr(topk_lens)) - print(np.array_repr(topk_scores)) - expected_topk_ids = [[4, 3, 4, 3, 2, 0, 0], [4, 3, 11, 2, 0, 0, 0], - [4, 3, 6, 2, 0, 0, 0], [6, 0, 4, 6, 6, 11, 2], - [6, 0, 4, 6, 1, 2, 0], [6, 0, 4, 6, 6, 2, 0]] - expected_topk_lens = [5, 4, 4, 7, 6, 6] - expected_topk_scores = [[8.27340603, 6.26949024, 5.59490776], - [9.74691486, 8.46679497, 7.14809656]] - self.assertEqual(expected_topk_ids, topk_ids.tolist()) - self.assertEqual(expected_topk_lens, topk_lens.tolist()) - self.assertAllClose(expected_topk_scores, topk_scores) - - def testCustomStepIds(self): - with self.session(use_gpu=False): - np.random.seed(9384758) - tf.random.set_seed(8274758) - vocab_size = 12 - src_len = 5 - tgt_len = 7 - num_hyps_per_beam = 3 - src_batch_size = 2 - tgt_batch_size = src_batch_size * num_hyps_per_beam - p = beam_search_helper.BeamSearchHelper.Params().Set( - name='bsh', target_seq_len=tgt_len) - bs_helper = p.Instantiate() - - def InitBeamSearchState(unused_theta, unused_encoder_outputs, - unused_num_hyps_per_beam): - atten_probs = tf.constant( - np.random.normal(size=(tgt_batch_size, src_len)), dtype=tf.float32) - return (py_utils.NestedMap({ - 'log_probs': tf.zeros([tgt_batch_size, vocab_size]), - 'atten_probs': atten_probs, - 'step_ids': tf.zeros([tgt_batch_size, 1], dtype=tf.int32) - }), py_utils.NestedMap({'atten_probs': atten_probs})) - - def PreBeamSearchStepCallback(unused_theta, unused_encoder_outputs, - unused_step_ids, states, - unused_num_hyps_per_beam): - atten_probs = tf.identity(states.atten_probs) - logits = tf.random.normal([tgt_batch_size, vocab_size], seed=8273747) - return (py_utils.NestedMap({ - 'atten_probs': atten_probs, - 'log_probs': logits - }), states) - - def PostBeamSearchStepCallback(unused_theta, unused_encoder_outputs, - unused_new_step_ids, states): - return states - - src_enc = tf.random.normal([src_len, src_batch_size, 8], seed=982774838) - src_enc_padding = tf.constant( - [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 1.0], [1.0, 1.0]], - dtype=tf.float32) - encoder_outputs = py_utils.NestedMap( - encoded=src_enc, padding=src_enc_padding) - - theta = py_utils.NestedMap() - decoder_output = bs_helper.BeamSearchDecode(theta, encoder_outputs, - num_hyps_per_beam, - InitBeamSearchState, - PreBeamSearchStepCallback, - PostBeamSearchStepCallback) - - topk_ids, topk_lens, topk_scores = self.evaluate([ - decoder_output.topk_ids, decoder_output.topk_lens, - decoder_output.topk_scores - ]) - print(np.array_repr(topk_ids)) - print(np.array_repr(topk_lens)) - print(np.array_repr(topk_scores)) - expected_topk_ids = [[4, 3, 4, 3, 2, 0, 0], [4, 3, 11, 2, 0, 0, 0], - [4, 3, 6, 2, 0, 0, 0], [6, 0, 4, 6, 6, 11, 2], - [6, 0, 4, 6, 1, 2, 0], [6, 0, 4, 6, 6, 2, 0]] - expected_topk_lens = [5, 4, 4, 7, 6, 6] - expected_topk_scores = [[8.27340603, 6.26949024, 5.59490776], - [9.74691486, 8.46679497, 7.14809656]] - self.assertEqual(expected_topk_ids, topk_ids.tolist()) - self.assertEqual(expected_topk_lens, topk_lens.tolist()) - self.assertAllClose(expected_topk_scores, topk_scores) - - -class MergeBeamSearchOutputsTest(test_utils.TestCase): - - def testMergeBeamSearchOutputs(self): - with self.session(): - topk_scores_1 = [[1., 3., 5.], [-2., -1., 0.]] - topk_ids_1 = [[[10, 11, 12], [30, 31, 32], [50, 51, 52]], - [[20, 21, 22], [10, 11, 12], [0, 0, 0]]] - topk_lens_1 = [[3, 3, 2], [3, 3, 0]] - topk_hyps_1 = [['one', 'three', 'five'], ['minus two', 'minus one', '']] - topk_1 = beam_search_helper.BeamSearchDecodeOutput( - None, tf.constant(topk_hyps_1), - tf.reshape(tf.constant(topk_ids_1), [6, -1]), - tf.reshape(tf.constant(topk_lens_1), [-1]), - tf.reshape(tf.constant(topk_scores_1), [-1]), None, None) - - topk_scores_2 = [[2., 4.], [-3., 0.]] - topk_ids_2 = [[[20, 21, 22], [40, 41, 42]], [[30, 31, 33], [0, 0, 0]]] - topk_lens_2 = [[3, 2], [3, 0]] - topk_hyps_2 = [['two', 'four'], ['minus three', '']] - topk_2 = beam_search_helper.BeamSearchDecodeOutput( - None, tf.constant(topk_hyps_2), - tf.reshape(tf.constant(topk_ids_2), [4, -1]), - tf.reshape(tf.constant(topk_lens_2), [-1]), - tf.reshape(tf.constant(topk_scores_2), [-1]), None, None) - - topk = beam_search_helper.MergeBeamSearchOutputs(3, [topk_1, topk_2]) - self.assertIsNone(topk.done_hyps) - self.assertIsNone(topk.topk_decoded) - self.assertAllEqual([5., 4., 3., -1., -2., -3.], topk.topk_scores.eval()) - self.assertAllEqual([2, 2, 3, 3, 3, 3], topk.topk_lens.eval()) - self.assertAllEqual([[50, 51, 52], [40, 41, 42], [30, 31, 32], - [10, 11, 12], [20, 21, 22], [30, 31, 33]], - topk.topk_ids.eval()) - self.assertAllEqual([[b'five', b'four', b'three'], - [b'minus one', b'minus two', b'minus three']], - topk.topk_hyps.eval()) - - -class GreedySearchHelperTest(test_utils.TestCase): - - def testGreedySearchHelper(self): - with self.session(use_gpu=False): - np.random.seed(9384758) - tf.random.set_seed(8274758) - vocab_size = 12 - src_len = 5 - tgt_len = 7 - src_batch_size = 2 - tgt_batch_size = src_batch_size - p = beam_search_helper.GreedySearchHelper.Params().Set( - name='gsh', target_seq_len=tgt_len) - gs_helper = p.Instantiate() - - def InitGreedySearchState(unused_theta, unused_encoder_outputs, - unused_num_hyps_per_beam): - atten_probs = tf.constant( - np.random.normal(size=(tgt_batch_size, src_len)), dtype=tf.float32) - return (py_utils.NestedMap({ - 'log_probs': tf.zeros([tgt_batch_size, vocab_size]), - 'atten_probs': atten_probs, - }), py_utils.NestedMap({'atten_probs': atten_probs})) - - def PreGreedySearchStepCallback(unused_theta, unused_encoder_outputs, - unused_step_ids, states, - unused_num_hyps_per_beam): - atten_probs = tf.identity(states.atten_probs) - logits = tf.random.normal([tgt_batch_size, vocab_size], seed=8273747) - return (py_utils.NestedMap({ - 'atten_probs': atten_probs, - 'log_probs': logits - }), states) - - def PostGreedySearchStepCallback(unused_theta, unused_encoder_outputs, - unused_new_step_ids, states): - return states - - src_enc = tf.random.normal([src_len, src_batch_size, 8], seed=982774838) - src_enc_padding = tf.constant( - [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 1.0], [1.0, 1.0]], - dtype=tf.float32) - encoder_outputs = py_utils.NestedMap( - encoded=src_enc, padding=src_enc_padding) - - theta = py_utils.NestedMap() - (final_hyp_ids, final_hyp_lens, - final_done_hyps) = gs_helper.GreedySearchDecode( - theta, encoder_outputs, InitGreedySearchState, - PreGreedySearchStepCallback, PostGreedySearchStepCallback) - - (final_hyp_ids, final_hyp_lens, final_done_hyps) = self.evaluate( - [final_hyp_ids, final_hyp_lens, final_done_hyps]) - - print(np.array_repr(final_hyp_ids)) - print(np.array_repr(final_hyp_lens)) - print(np.array_repr(final_done_hyps)) - - expected_hyp_ids = [[2, 2, 6, 7, 1, 9, 4], [3, 9, 3, 9, 6, 5, 10]] - expected_hyp_lens = [1, 7] - expected_done_hyps = [True, False] - self.assertEqual(expected_hyp_ids, final_hyp_ids.tolist()) - self.assertEqual(expected_hyp_lens, final_hyp_lens.tolist()) - self.assertEqual(expected_done_hyps, final_done_hyps.tolist()) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/bfloat16_variables.py b/lingvo/core/bfloat16_variables.py deleted file mode 100644 index 7c916345..00000000 --- a/lingvo/core/bfloat16_variables.py +++ /dev/null @@ -1,93 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Various methods for bfloat16 training & inference. - -Bfloat16VariableSaveable: Saveable that restores variable into bfloat16 type. - -Usage: - - Given a checkpoint_path with a variable of type tf.float32, this particular - saveable allows restore them as tf.bfloat16. This is specifically useful for - inference. - - Say: checkpoint_path contains a variable "var" with dtype tf.float32:: - - variable_name = "var" - original_dtype = tf.float32 - - bfloat16_var = tf.Variable( - 0.0, name=variable_name, dtype=tf.bfloat16, use_resource=True) - - saveable = bfloat16_variables.Bfloat16VariableSaveable( - bfloat16_var, original_dtype, slice_spec, variable_name) - saver = tf.train.Saver( - {variable_name: saveable}, restore_sequentially=True) - saver.restore(sess, checkpoint_path) - - # bfloat16_var is now loaded from the checkpoint. -""" -import lingvo.compat as tf -from tensorflow.python.training import saver - - -class Bfloat16VariableSaveable(saver.BaseSaverBuilder.SaveableObject): - """Saveable that loads Variables as bfloat16.""" - - def __init__(self, var, orig_dtype, slice_spec, name): - # TODO(rohananil): Investigate if we can avoid using a callable, instead - # change the saveable api to make use of dtype passed in. - def _make_callable_var(): - return var - - spec = saver.BaseSaverBuilder.SaveSpec( - _make_callable_var, - slice_spec, - name, - dtype=orig_dtype, - device=var.device) - super().__init__(var, [spec], name) - - def restore(self, restored_tensors, restored_shapes): - restored_tensor = restored_tensors[0] - if restored_shapes is not None: - restored_tensor = tf.reshape(restored_tensor, restored_shapes[0]) - return tf.assign( - self.op, - tf.cast(restored_tensor, tf.bfloat16), - validate_shape=restored_shapes is None and - self.op.get_shape().is_fully_defined()) - - -def get_saver_spec_for_variables_with_bf16_overrides(variables_to_restore): - """Returns a dictionary containing overrides to load variables as bf16. - - Args: - variables_to_restore: A mapping from variable to name (on checkpoint) to the - Variable object. - - Returns: - A saver dictionary which can be used to load from checkpoints. - """ - saver_dict = {} - for var_name, v in variables_to_restore.items(): - if v.dtype == tf.bfloat16: - # TODO(rohananil): Add support for PartitionedVariables if there is - # demand. - savable = Bfloat16VariableSaveable(v, tf.float32, '', var_name) - saver_dict[var_name] = savable - else: - saver_dict[var_name] = v - return saver_dict diff --git a/lingvo/core/bfloat16_variables_test.py b/lingvo/core/bfloat16_variables_test.py deleted file mode 100644 index adfeac41..00000000 --- a/lingvo/core/bfloat16_variables_test.py +++ /dev/null @@ -1,57 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for bfloat16_variables.""" - -import os -import lingvo.compat as tf -from lingvo.core import bfloat16_variables -from lingvo.core import test_utils - - -class Bfloat16VariablesTest(test_utils.TestCase): - - def testBfloat16Reload(self): - checkpoint_path = os.path.join(self.get_temp_dir(), "bfloat16_restore") - - # Create a resource variable of type tf.float32 and save them to disk. - g_for_save_graph = tf.Graph() - fl = 0.99 - with self.session(graph=g_for_save_graph) as sess: - v0 = tf.Variable(fl, name="v0", dtype=tf.float32, use_resource=True) - self.evaluate(tf.global_variables_initializer()) - self.assertAlmostEqual(fl, v0.eval()) - - saver = tf.train.Saver({ - "v0": v0, - }, restore_sequentially=True) - val = saver.save(sess, checkpoint_path) - self.assertEqual(checkpoint_path, val) - - # Restore the variable as bfloat16. - g_for_restore_graph = tf.Graph() - with self.session(graph=g_for_restore_graph) as sess: - v0 = tf.Variable(0.0, name="v0", dtype=tf.bfloat16, use_resource=True) - self.evaluate(tf.global_variables_initializer()) - self.assertAlmostEqual(0.0, v0.eval()) - saveable = bfloat16_variables.Bfloat16VariableSaveable( - v0, tf.float32, "", "v0") - saver = tf.train.Saver({"v0": saveable}, restore_sequentially=True) - saver.restore(sess, checkpoint_path) - self.assertAlmostEqual(fl, v0.eval()) - - -if __name__ == "__main__": - tf.test.main() diff --git a/lingvo/core/bn_layers.py b/lingvo/core/bn_layers.py deleted file mode 100644 index ebea9e62..00000000 --- a/lingvo/core/bn_layers.py +++ /dev/null @@ -1,823 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================= -"""Batch normalization layers.""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils -from lingvo.core import summary_utils - -from tensorflow.python.ops import nn # pylint:disable=g-direct-tensorflow-import -from tensorflow.python.tpu import tpu_function # pylint:disable=g-direct-tensorflow-import - -_BN_FLOPS_PER_ELEMENT = 10 - - -# TODO(rpang): move AddingAccumulator to a separate library. -class AddingAccumulator(base_layer.Accumulator): - """Accumulator for the sufficient statistics.""" - - def __init__(self, shape, dtype): - super().__init__() - self.dtype = dtype - self.shape = shape - - def DefaultValue(self): - """Returns the default value of the accumulator.""" - return tf.zeros(self.shape, dtype=self.dtype) - - def Update(self, value): - """Adds value to the accumulator.""" - self.SetValue(self.GetValue() + tf.cast(value, self.dtype)) - - -def ComputeMomentsWithPadding(inputs, - padding, - reduce_over_dims, - cumulative_axis=None, - enable_cross_replica_sum_on_tpu=False, - keepdims=False): - """Computes mean and variance over the valid data points in inputs.""" - mask = 1.0 - padding - inputs = py_utils.with_dependencies([ - py_utils.assert_equal(tf.rank(inputs), tf.rank(mask)), - py_utils.assert_greater_equal(mask, tf.zeros_like(mask)), - ], inputs) - sum_v = tf.reduce_sum( - inputs * tf.cast(mask, inputs.dtype), reduce_over_dims, keepdims=keepdims) - count_v = tf.reduce_sum(mask, reduce_over_dims, keepdims=keepdims) - - if cumulative_axis is not None: - sum_v = tf.math.cumsum(sum_v, axis=cumulative_axis) - count_v = tf.math.cumsum(count_v, axis=cumulative_axis) - # Input shape is guaranteed to be a multiple of mask shape because the - # inputs * mask op above was successfully broadcasted. - input_size_on_reduced_dims = tf.reduce_prod( - tf.gather(tf.shape(inputs), reduce_over_dims)) - mask_size_on_reduced_dims = tf.reduce_prod( - tf.gather(tf.shape(mask), reduce_over_dims)) - mask_multiplier = tf.math.truediv(input_size_on_reduced_dims, - mask_size_on_reduced_dims) - count_v *= tf.cast(mask_multiplier, count_v.dtype) - if py_utils.use_tpu() and enable_cross_replica_sum_on_tpu: - sum_v = tf.tpu.cross_replica_sum(sum_v) - count_v = tf.tpu.cross_replica_sum(count_v) - - count_v = tf.maximum(count_v, 1.0) - mean = sum_v / count_v - sum_vv = tf.reduce_sum( - (inputs - mean) * (inputs - mean) * mask, - reduce_over_dims, - keepdims=keepdims) - if cumulative_axis is not None: - sum_vv = tf.math.cumsum(sum_vv, axis=cumulative_axis) - - if py_utils.use_tpu() and enable_cross_replica_sum_on_tpu: - sum_vv = tf.tpu.cross_replica_sum(sum_vv) - - variance = py_utils.with_dependencies([ - py_utils.assert_greater_equal(sum_vv, tf.zeros_like(sum_vv)), - ], sum_vv / count_v) - return mean, variance - - -class BatchNormLayer(base_layer.BaseLayer): - """Batch normalization layer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('dim', 0, 'Depth of the input/output.') - p.Define( - 'decay', 0.999, - 'Decay in updating the mean and variance moving average used in' - ' batch normalization.') - p.Define( - 'enable_cross_replica_sum_on_tpu', True, - 'If true, calls cross_replica_sum to the aggregate moving averages' - ' across all replicas.') - p.Define( - 'use_moving_avg_in_training', False, - 'If True, use global moving avg (mean, variance) during training' - ' to avoid mismatch between train and eval, which then' - ' essentially acts as an adaptive normalization step.') - p.Define( - 'freeze_bn_stats', False, - 'If True, uses moving avg (mean, variance) during both training and ' - 'inference. It behaves like force_eval but the gamma/beta are still ' - 'trained when do_eval is False. The moving mean/var can be set by ' - 'loading pretrained checkpoints. A use case is training detectors ' - 'based on an pretrained checkpoint while BN stats are frozen.') - p.Define( - 'gamma_zero_init', False, - 'If True, initialize gamma to zeros according to the technique ' - 'introduced in the tech report: https://arxiv.org/abs/1706.02677') - p.Define( - 'gamma_one_init', False, - 'If True, explicitly initialize gamma to one without invoking ' - 'theta_fn.') - # TODO(rpang): remove this hparam, as it is replaced - # by p.train.ema_decay_moving_vars. - p.Define( - 'add_stats_to_moving_average_variables', None, - 'If True, adds (mean, variance) to the MOVING_AVERAGE_VARIABLES ' - 'collection to be compatible with ema_decay. ' - 'Recommendation: set to True for new models, and to False to maintain ' - 'checkpoint compatibility.') - p.Define('set_padded_output_to_zero', True, - 'If True, sets the padded outputs to zero.') - p.Define( - 'use_fused_batch_norm_for_eval', False, - 'If True, uses tf.compat.v1.nn.fused_batch_norm instead of ' - 'tf.nn.batch_normalization during eval. The fused version may be more ' - 'efficient but it has more restrictions on the expected input shapes.' - 'The input tensor has to be rank 4, where the first dimension ' - 'corresponds to the batch, and the last dimension corresponds to the ' - 'features to normalize over. This usually corresponds to NHWC with ' - 'image inputs. Note that fused_batch_norm wants to track its own ' - 'mean and variance during training, so we are unable to use it ' - 'for training since we want to have a custom mean and variance to ' - 'support padding.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self._epsilon = 0.001 - self._decay = p.decay - - def _GetWeightShape(self): - return [self.params.dim] - - def _CreateLayerVariables(self): - p = self.params - - beta_pc = py_utils.WeightParams( - shape=self._GetWeightShape(), - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - - gamma_pc = py_utils.WeightParams( - shape=self._GetWeightShape(), - init=py_utils.WeightInit.Constant(1.0) - if p.gamma_one_init else py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - - if not p.use_moving_avg_in_training: - self.CreateVariable('beta', beta_pc) - if p.gamma_zero_init or p.gamma_one_init: - # initialization to BN gamma - self.CreateVariable('gamma', gamma_pc) - else: - # Note, The real gamma to use is 1 + gamma. - self.CreateVariable('gamma', gamma_pc, lambda x: 1.0 + x) - - # Two statistics. - moving_collections = ['moving_vars', self.__class__.__name__ + '_vars'] - if p.add_stats_to_moving_average_variables: - moving_collections += [tf.GraphKeys.MOVING_AVERAGE_VARIABLES] - elif p.add_stats_to_moving_average_variables is None: - # TODO(rpang): force all models to set this param explicitly. - tf.logging.warning( - 'BatchNormLayer.add_stats_to_moving_average_variables should be ' - 'set to True for new models, and to False explicitly for ' - 'checkpoint compatibility.') - # Add to the MOVING_AVERAGE_VARIABLES collection so that they are returned - # by tf.moving_average_variables() and included in EMA variables if - # ema_decay is enabled. - mva = py_utils.WeightParams( - shape=[p.dim], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=moving_collections) - self.CreateVariable( - 'moving_mean', - mva, - trainable=False, - aggregation=tf.VariableAggregation.MEAN) - - mvv = py_utils.WeightParams( - shape=[p.dim], - init=py_utils.WeightInit.Constant(1.0), - dtype=p.dtype, - collections=moving_collections) - self.CreateVariable( - 'moving_variance', - mvv, - trainable=False, - aggregation=tf.VariableAggregation.MEAN) - - @property - def epsilon(self): - return self._epsilon - - def _GetDefaultPaddings(self, inputs): - """Gets the default paddings for an input.""" - return tf.zeros( - tf.concat([tf.shape(inputs)[:-1], [1]], 0), dtype=inputs.dtype) - - def _GetBetaGamma(self, theta, inputs, **kwargs): - del inputs - del kwargs - p = self.params - if p.use_moving_avg_in_training: - beta = 0.0 - gamma = 1.0 - else: - beta = theta.beta - gamma = theta.gamma - return beta, gamma - - def GetCurrentMoments(self, theta): - """Gets the current computed moments, which should be applied at eval. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - Returns: - Tuple of (mean, variance, beta, gamma). - """ - p = self.params - if p.use_moving_avg_in_training: - return self.vars.moving_mean, self.vars.moving_variance, 0.0, 1.0 - else: - return (self.vars.moving_mean, self.vars.moving_variance, theta.beta, - theta.gamma) - - def ComputeAndUpdateMoments(self, theta, inputs, paddings=None, **kwargs): - """Computes moments and updates state. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: The inputs tensor. Shaped [..., dim]. - paddings: The paddings tensor. Shaped [..., 1], with the same rank as the - input tensor. - **kwargs: Additional inputs. - - Returns: - Tuple of (mean, variance, beta, gamma). - """ - p = self.params - if paddings is None: - paddings = self._GetDefaultPaddings(inputs) - inputs = py_utils.with_dependencies([ - py_utils.assert_shape_match([tf.shape(paddings)[-1]], [1]), - ], inputs) - with tf.name_scope(p.name): - if self.do_eval or p.freeze_bn_stats: - # The mean and variance used for normalization. - norm_mean, norm_variance = (self.vars.moving_mean, - self.vars.moving_variance) - else: - rank = tf.rank(paddings) - reduce_over_dims = tf.range(0, rank - 1) - mean, variance = ComputeMomentsWithPadding( - inputs, paddings, reduce_over_dims, None, - p.enable_cross_replica_sum_on_tpu) - - py_utils.UpdateBatchNormVars(self.vars.moving_mean, mean, self._decay) - py_utils.UpdateBatchNormVars(self.vars.moving_variance, variance, - self._decay) - # Add some summaries for visualization. - summary_utils.histogram('%s_mean' % p.name, tf.cast(mean, tf.float32)) - summary_utils.histogram('%s_variance' % p.name, - tf.cast(variance, tf.float32)) - summary_utils.histogram('%s_moving_mean' % p.name, - tf.cast(self.vars.moving_mean, tf.float32)) - summary_utils.histogram('%s_moving_variance' % p.name, - tf.cast(self.vars.moving_variance, tf.float32)) - summary_utils.histogram( - '%s_mean_diff' % p.name, - tf.cast( - tf.cast(mean, self.vars.moving_mean.dtype.base_dtype) - - self.vars.moving_mean, tf.float32)) - summary_utils.histogram( - '%s_variance_diff' % p.name, - tf.cast( - tf.cast(variance, self.vars.moving_variance.dtype.base_dtype) - - self.vars.moving_variance, tf.float32)) - if p.use_moving_avg_in_training: - # Use the global statistics for normalization. - # Control dependencies on mean and variance make sure - # moving_mean and variance will be updated for every training step. - norm_mean = py_utils.with_dependencies([mean], self.vars.moving_mean) - norm_variance = py_utils.with_dependencies([variance], - self.vars.moving_variance) - else: - # Use the batch statistics for normalization. - norm_mean = mean - norm_variance = variance - - norm_mean = py_utils.CheckNumerics( - norm_mean, 'mean of %s failed numeric check' % p.name) - norm_variance = py_utils.CheckNumerics( - norm_variance, 'variance of %s failed numeric check' % p.name) - - beta, gamma = self._GetBetaGamma(theta, inputs, **kwargs) - return norm_mean, norm_variance, beta, gamma - - def _ComputeBN(self, inputs, paddings, gamma, beta, norm_mean, norm_variance): - p = self.params - with tf.control_dependencies([ - py_utils.assert_greater_equal(norm_variance, - tf.zeros_like(norm_variance)), - py_utils.assert_shape_match([tf.shape(inputs)[-1]], - tf.shape(norm_mean)), - py_utils.assert_shape_match([tf.shape(inputs)[-1]], - tf.shape(norm_variance)), - ]): - if p.use_fused_batch_norm_for_eval and (self.do_eval or - p.freeze_bn_stats): - bn_output, _, _ = nn.fused_batch_norm( - inputs, - gamma, - beta, - norm_mean, - norm_variance, - self._epsilon, - is_training=False) - else: - bn_output = tf.nn.batch_normalization(inputs, norm_mean, norm_variance, - beta, gamma, self._epsilon) - if p.set_padded_output_to_zero: - bn_output *= 1.0 - paddings - return bn_output - - def _MaybeExpandPaddings(self, inputs, paddings): - # rank difference is at most one. - rank_diff = tf.rank(inputs) - tf.rank(paddings) - paddings = py_utils.with_dependencies([ - py_utils.assert_less_equal(rank_diff, 1), - py_utils.assert_greater_equal(rank_diff, 0) - ], paddings) - - # Pads [1] to the end of paddings. - paddings = tf.reshape( - paddings, - tf.concat( - [tf.shape(paddings), tf.tile([1], [rank_diff])], axis=0)) - return paddings - - def FProp(self, theta, inputs, paddings=None): - """Apply batch normalization. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: The inputs tensor. Shaped [..., dim]. - paddings: The paddings tensor. Shaped [..., 1] or [...], the rank is - either the same as inputs or tf.rank(inputs) - 1. - - Returns: - Output after applying batch normalization, with the same shape as - 'inputs'. - """ - p = self.params - if paddings is None: - paddings = self._GetDefaultPaddings(inputs) - - # shape [..., 1] - paddings = self._MaybeExpandPaddings(inputs, paddings) - - with tf.name_scope(p.name): - norm_mean, norm_variance, beta, gamma = self.ComputeAndUpdateMoments( - theta, inputs, paddings) - - return self._ComputeBN(inputs, paddings, gamma, beta, norm_mean, - norm_variance) - - @classmethod - def FPropMeta(cls, p, inputs, padding=None): - py_utils.CheckShapes((inputs,)) - return py_utils.NestedMap( - flops=inputs.num_elements() * _BN_FLOPS_PER_ELEMENT, - out_shapes=(inputs,)) - - -class CategoricalBN(BatchNormLayer): - """Implements a categorical BN which is akin to ... - - https://arxiv.org/pdf/1809.11096.pdf - - Specifically, the moving stats are category-agnostic, while {beta, gamma} are - category-aware. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('class_emb_dim', None, 'Dim of input class embedding.') - - p.use_moving_avg_in_training = False - p.use_fused_batch_norm_for_eval = False - p.add_stats_to_moving_average_variables = True - return p - - def __init__(self, params): - assert params.name - assert not params.use_moving_avg_in_training - assert not params.use_fused_batch_norm_for_eval - assert params.add_stats_to_moving_average_variables - super().__init__(params) - - def _GetWeightShape(self): - return [self.params.class_emb_dim, self.params.dim] - - def _GetBetaGamma(self, theta, inputs, **kwargs): - assert 'class_emb' in kwargs - class_emb = kwargs['class_emb'] - - # class_emb is a one-hot vector of shape [batch, class_emb_dim=num_classes]. - class_ids = tf.math.argmax(class_emb, axis=-1, output_type=tf.int32) - # [batch, dim] - # Not using matmul/einsum to avoid potential precision problem on TPU with - # sparse inputs. - beta = tf.gather(theta.beta, class_ids) - gamma = tf.gather(theta.gamma, class_ids) - - # Extend to [batch, 1, ... 1, dim] - batch = py_utils.GetShape(inputs)[0] - to_shape = tf.concat( - [[batch], - tf.ones([py_utils.GetRank(inputs) - 2], tf.int32), [self.params.dim]], - axis=0) - beta = tf.reshape(beta, to_shape) - gamma = tf.reshape(gamma, to_shape) - return beta, gamma - - def FProp(self, theta, inputs, paddings, class_emb): - """Apply batch normalization. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: The inputs tensor. Shaped [batch, ..., dim]. - paddings: The paddings tensor. Shaped [batch, ..., 1], with the same rank - as the input tensor. - class_emb: The conditioning inputs, Shaped [batch, emb_dim]. - - Returns: - Output after applying batch normalization, with the same shape as - 'inputs'. - """ - p = self.params - batch = py_utils.GetShape(inputs)[0] - class_emb = py_utils.HasShape(class_emb, [batch, p.class_emb_dim]) - if not py_utils.use_tpu(): - class_emb = py_utils.with_dependencies([ - py_utils.assert_less_equal( - tf.cast(class_emb, tf.int32), 1, name='one_hot_assert1'), - py_utils.assert_greater_equal( - tf.cast(class_emb, tf.int32), 0, name='one_hot_assert2'), - py_utils.assert_equal( - tf.ones([batch], tf.int32), - tf.cast(tf.reduce_sum(class_emb, -1), tf.int32), - name='one_hot_assert3'), - ], class_emb) - - with tf.name_scope(p.name): - norm_mean, norm_variance, beta, gamma = self.ComputeAndUpdateMoments( - theta, inputs, paddings=paddings, class_emb=class_emb) - return self._ComputeBN(inputs, paddings, gamma, beta, norm_mean, - norm_variance) - - -class BatchNormLayerNoPadding(base_layer.BaseLayer): - """Batchnorm layer without padding.""" - - @classmethod - def Params(cls): - """Parameters for BatchNormLayerNoPadding.""" - p = super().Params() - p.Define('dim', 0, 'Depth of the input/output.') - p.Define( - 'decay', 0.997, - 'Decay in updating the mean and variance moving average used in' - ' batch normalization.') - p.Define('epsilon', 0.001, - 'Small float added to variance to avoid dividing by zero.') - p.Define( - 'bn_group_size', 1, - 'The number of shards participating in normalization when distributed' - ' batchnorm is used. Only used for TPU.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name, 'Name of BatchNormLayerNoPadding is not set.' - p.fprop_dtype = None - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - - # Skip L-P regularization for these variables. - collections = [ - self.__class__.__name__ + '_vars', py_utils.SKIP_LP_REGULARIZATION - ] - pc = py_utils.WeightParams( - shape=[p.dim], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=collections) - - self.CreateVariable('beta', pc) - # Note, The real gamma to use is 1 + gamma. - self.CreateVariable('gamma', pc, lambda x: 1.0 + x) - - moving_collections = [ - 'moving_vars', tf.GraphKeys.MOVING_AVERAGE_VARIABLES, - self.__class__.__name__ + '_vars' - ] - mva = py_utils.WeightParams( - shape=[p.dim], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=moving_collections) - # Two statistics computed from sufficient stats. - self.CreateVariable('moving_mean', mva, trainable=False) - mvv = py_utils.WeightParams( - shape=[p.dim], - init=py_utils.WeightInit.Constant(1.0), - dtype=p.dtype, - collections=moving_collections) - self.CreateVariable('moving_variance', mvv, trainable=False) - - # Accumulate bn sufficient stats over micro-batches. - dim = self.vars.beta.shape[0] - self.RegisterAccumulator('counts', AddingAccumulator([], p.dtype)) - self.RegisterAccumulator('mean_ss', AddingAccumulator([dim], p.dtype)) - self.RegisterAccumulator('variance_ss', AddingAccumulator([dim], p.dtype)) - - def PostTrainingStepUpdate(self, global_step): - """Updates moving_mean, moving_variance after each training step.""" - p = self.params - # Get sufficient stats that accumulates over microbatches. - counts = self.accumulators.counts.GetValue() - mean_ss = self.accumulators.mean_ss.GetValue() - variance_ss = self.accumulators.variance_ss.GetValue() - # Compute batch mean and batch variance from sufficient stats - mean, variance = tf.nn.normalize_moments(counts, mean_ss, variance_ss, None) - decay = tf.convert_to_tensor(1.0 - p.decay, p.dtype) - # Update moving_mean, moving_variance from batch mean and batch variance. - with tf.name_scope(p.name) as scope: - with tf.ops.colocate_with(self.vars.moving_mean): - mean_update = tf.assign_sub( - self.vars.moving_mean, - tf.where( - tf.greater(counts, 0.5), - (self.vars.moving_mean - tf.cast(mean, p.dtype)) * decay, - tf.zeros_like(self.vars.moving_mean)), - name='moving_mean_update') - with tf.ops.colocate_with(self.vars.moving_variance): - var_update = tf.assign_sub( - self.vars.moving_variance, - tf.where( - tf.greater(counts, 0.5), - (self.vars.moving_variance - tf.cast(variance, p.dtype)) * - decay, tf.zeros_like(self.vars.moving_variance)), - name='moving_variance_update') - py_utils.CheckNumerics( - self.vars.moving_mean, - 'moving mean of {} failed numeric check'.format(scope)) - py_utils.CheckNumerics( - self.vars.moving_variance, - 'moving variance of {} failed numeric check'.format(scope)) - self.accumulators.counts.Reset() - self.accumulators.mean_ss.Reset() - self.accumulators.variance_ss.Reset() - return tf.group(mean_update, var_update) - - def _Moments(self, inputs, group_size): - """Computes mean and variance over N,H,W dimensions in inputs.""" - counts, mean_ss, variance_ss, _, = tf.nn.sufficient_statistics( - inputs, axes=[0, 1, 2], keepdims=False) - self.accumulators.counts.Update(counts) - self.accumulators.mean_ss.Update(mean_ss) - self.accumulators.variance_ss.Update(variance_ss) - # Distributed batch norm that computes sufficient statistics from group_size - # replicas. This is useful when batch_size_per_replica is too small to - # compute reliable sufficient statistics. - if py_utils.use_tpu() and group_size > 1: - group_assignment = None - num_shards = tpu_function.get_tpu_context().number_of_shards - if num_shards is not None: - if num_shards < group_size: - raise ValueError('TPU shards={} less than bn_gropu_size={}.'.format( - num_shards, group_size)) - if num_shards % group_size: - raise ValueError( - 'TPU shards={} not divisible by bn_group_size={}.'.format( - num_shards, group_size)) - num_groups = num_shards // group_size - group_assignment = [] - for g in range(num_groups): - replica_ids = [g * group_size + i for i in range(group_size)] - group_assignment.append(replica_ids) - counts *= group_size - mean_ss = tf.tpu.cross_replica_sum(mean_ss, group_assignment) - variance_ss = tf.tpu.cross_replica_sum(variance_ss, group_assignment) - # At each micro-step, batch_mean and batch_variance are computed - # to normalize inputs. But they are not used to update moving_mean and - # moving_variance variables until the last micro batch. - mean, variance = tf.nn.normalize_moments(counts, mean_ss, variance_ss, None) - return mean, variance - - def FProp(self, theta, inputs): - """Applies batch normalization. - - Using the implementation in github.com/ - tensorflow/tpu/blob/master/models/official/amoeba_net/network_utils.py#L550 - - Args: - theta: A nested map object containing weights' values of this layer and - its children layers. - inputs: The inputs tensor. Shaped [..., dim]. - - Returns: - Output after applying batch normalization, with the same shape as - 'inputs'. - """ - p = self.params - inputs_dtype = inputs.dtype - inputs = tf.cast(inputs, p.dtype) - inputs = py_utils.with_dependencies([ - py_utils.assert_shape_match([tf.shape(inputs)[-1]], tf.shape( - theta.beta)) - ], inputs) - with tf.name_scope(p.name) as scope: - if self.do_eval: - outputs = tf.nn.batch_normalization(inputs, theta.moving_mean, - theta.moving_variance, - theta.beta, theta.gamma, p.epsilon) - else: - mean, variance = self._Moments(inputs, p.bn_group_size) - mean = py_utils.CheckNumerics( - mean, 'mean of {} failed numeric check'.format(scope)) - variance = py_utils.CheckNumerics( - variance, 'variance of {} failed numeric check'.format(scope)) - outputs = tf.nn.batch_normalization(inputs, mean, variance, theta.beta, - theta.gamma, p.epsilon) - outputs.set_shape(inputs.get_shape()) - return tf.cast(outputs, inputs_dtype) - - @classmethod - def FPropMeta(cls, p, inputs): - """Returns metadata about the `FProp` computation for this layer.""" - py_utils.CheckShapes((inputs,)) - return py_utils.NestedMap( - flops=inputs.num_elements() * _BN_FLOPS_PER_ELEMENT, - out_shapes=(inputs,)) - - -class GroupNormLayer(base_layer.BaseLayer): - """Group normalization layer(https://arxiv.org/abs/1803.08494).""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('dim', 0, 'Depth of the input/output.') - p.Define('num_groups', 32, 'Number of groups for GroupNorm.') - p.Define('min_group_size', 1, 'Minimum group size for GroupNorm') - p.Define('cumulative', False, 'If true, only normalize by current and ' - 'previous time steps.') - p.Define('input_rank', 4, 'Rank of input. Only 3(BTD) and 4(NHWC) are ' - 'supported.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.num_groups > 0 - assert p.min_group_size > 0 - if p.dim >= p.num_groups: - assert p.dim % p.num_groups == 0, ('p.dim({0}) is not dividable by ' - 'p.num_groups({1})').format( - p.dim, p.num_groups) - self._epsilon = 0.001 - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - assert p.input_rank == 3 or p.input_rank == 4 - - collections = [ - self.__class__.__name__ + '_vars', py_utils.SKIP_LP_REGULARIZATION - ] - - shape = [1, 1, 1, p.dim] if p.input_rank == 4 else [1, 1, p.dim] - pc = py_utils.WeightParams( - shape=shape, - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=collections) - - self.CreateVariable('beta', pc) - # Note, The real gamma to use is 1 + gamma. - self.CreateVariable('gamma', pc, lambda x: 1.0 + x) - - def FProp(self, theta, inputs, paddings=None): - """Apply group normalization. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - inputs: The inputs tensor with shape [batch_size, height, width, channel]. - paddings: The paddings tensor with shape [batch_size, height]. Intended to - be used for sequence processing where `height` is `time`. - - Returns: - A single tensor as the output after applying group normalization, with - the same shape as 'inputs'. Or a output, output_paddings pair if input - paddings is not None. - """ - p = self.params - inputs = py_utils.with_dependencies( - [py_utils.assert_greater_equal(py_utils.GetRank(inputs), p.input_rank)], - inputs) - - min_group_size = min(p.min_group_size, p.dim) - group_size = max(p.dim // p.num_groups, min_group_size) - num_groups = p.dim // group_size - - input_shape = py_utils.GetShape(inputs) - with tf.name_scope(p.name): - x = tf.reshape(inputs, input_shape[:-1] + [num_groups, group_size]) - expanded_rank = p.input_rank + 1 - all_dims = list(range(expanded_rank)) - if paddings is None: - # Skip d0, d[-2] - axes = all_dims[1:-2] + all_dims[-1:] - counts, means_ss, variance_ss, _, = tf.nn.sufficient_statistics( - x, axes=axes, keepdims=True) - norm_mean, norm_variance = tf.nn.normalize_moments( - counts, means_ss, variance_ss, None) - else: - expanded_paddings = tf.reshape( - paddings, input_shape[:2] + [1] * (expanded_rank - 2)) - # skip the batching and group dim - if p.cumulative: - # Skip d0, d1 and d[-2] - reduce_over_dims = all_dims[2:-2] + all_dims[-1:] - norm_mean, norm_variance = ComputeMomentsWithPadding( - x, - expanded_paddings, - reduce_over_dims=reduce_over_dims, - cumulative_axis=1, - keepdims=True) - else: - # Skip d0, d[-2] - reduce_over_dims = all_dims[1:-2] + all_dims[-1:] - norm_mean, norm_variance = ComputeMomentsWithPadding( - x, expanded_paddings, reduce_over_dims, keepdims=True) - - norm_mean = py_utils.CheckNumerics( - norm_mean, 'mean of %s failed numeric check' % p.name) - norm_variance = py_utils.CheckNumerics( - norm_variance, 'variance of %s failed numeric check' % p.name) - - beta = theta.beta - gamma = theta.gamma - n = input_shape[0] - t = input_shape[1] if p.cumulative else 1 - norm_shape = [n, t, 1, num_groups, 1 - ] if p.input_rank == 4 else [n, t, num_groups, 1] - with tf.control_dependencies([ - py_utils.assert_greater_equal(norm_variance, - tf.cast(0., norm_variance.dtype)), - py_utils.assert_shape_match(norm_shape, tf.shape(norm_mean)), - py_utils.assert_shape_match(norm_shape, tf.shape(norm_variance)), - ]): - x = (x - norm_mean) / tf.sqrt(norm_variance + self._epsilon) - x = tf.reshape(x, input_shape) - gn_output = x * gamma + beta - gn_output = tf.reshape(gn_output, input_shape) - if paddings is None: - return gn_output - else: - return gn_output, paddings - - @classmethod - def FPropMeta(cls, p, inputs): - py_utils.CheckShapes((inputs,)) - flops_per_element = 10 # Approximately 10 flops per element. - return py_utils.NestedMap( - flops=inputs.num_elements() * flops_per_element, out_shapes=(inputs,)) diff --git a/lingvo/core/build_data.py b/lingvo/core/build_data.py deleted file mode 100644 index 138ae4c1..00000000 --- a/lingvo/core/build_data.py +++ /dev/null @@ -1,24 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Build data.""" - -from lingvo.core import hyperparams - - -def BuildData(): - """Returns a hyperparam recording build information of this py binary.""" - p = hyperparams.Params() - return p diff --git a/lingvo/core/builder.py b/lingvo/core/builder.py deleted file mode 100644 index 605bc9d6..00000000 --- a/lingvo/core/builder.py +++ /dev/null @@ -1,314 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""A library to build composite layers. - -WARNING: - The builder pattern is still experimental and we need to gain experience - on when to use and when not to use. - Please discuss w/ teammates before using it to build complicated - layers. -""" - -import functools -from lingvo import compat as tf -from lingvo.core import builder_layers -from lingvo.core import hyperparams -from lingvo.core import layers -from lingvo.core import py_utils -from lingvo.core import tshape - - -class Base: - """Model builder with commonly used layers. - - A method in a builder class constructs a layer param. FProp of a layer - constructed by a builder takes a tuple of tf.Tensor (one or more) and returns - a tuple of tf.Tensor (one or more). Even though certain layers support FProp - argument being None (e.g., Conv2DLayer), builder should not depend on such a - support. - - The constructed layer is often a composition of multiple sub-layers connected - in certain patterns. We expect to have a few methods to facilitate building - these patterns. For example, _Seq() helps to build a sequential layer that - calls its sub-layer one after another. - - TODO(zhifengc): Adds a more concrete example. - """ - - @classmethod - def Params(cls): - """The params of this layer.""" - p = hyperparams.InstantiableParams(cls) - p.Define('deterministic_dropout', False, - 'Used deterministic dropout or not.') - p.Define( - 'fprop_dtype', None, - 'Activations datatype to use. To enable bfloat16 activations for ' - 'layers built using model builder, set fprop_dtype to ' - 'tf.bfloat16, which will be propagated to layers that support ' - 'bfloat16 activations. Default is None, which will use float32 ' - 'activations.') - return p - - @property - def params(self): - """Returns the params upon which this layer is built.""" - return self._params - - def __init__(self, params): - # Sub-classes should put some options common to many layers in __init__. - self._params = params.Copy() - - ###################################################################### - # Layers to compose multiple layers. - # - # Sub-classes are discouraged to override these composition method. - ###################################################################### - def _Rep(self, name, repeat, *subs): - r"""Connects sub-layers sequentially and repeat multiple times. - - E.g., _Rep('foo', 2, sa, sb, sc) constructs a layer with 6 layers - sequentially connected: [sa1, sb1, sc1, sa2, sb2, sc2]. sa1 and sa2 have - the same structure as the given sa, but sa1 and sa2 do not share the same - weight. - - Args: - name: The layer name. - repeat: Repeat \*subs this many times in the compose layer. - *subs: A list of sub-layers. - - Returns: - The param for the composed layer. - """ - iterations = [] - for i in range(repeat): - iterations.append(self._Seq('iter_%03d' % i, *[p.Copy() for p in subs])) - return self._Seq(name, *iterations) - - def _Seq(self, name, *subs): - """Connects sub-layers sequentially.""" - return builder_layers.SequentialLayer.Params().Set( - name=name, sub=list(subs)) - - def _Graph(self, name, input_endpoints, output_endpoints, - *signature_sub_param_list): - """Connects sub-layers into a data flow graph.""" - return builder_layers.GraphLayer.Params().Set( - name=name, - input_endpoints=input_endpoints, - output_endpoints=output_endpoints, - sub=list(signature_sub_param_list)) - - def _Id(self, name): - """Identity. (t_1, ..., t_n) -> (t1, ..., t_n).""" - return self._Seq(name) - - def _Arg(self, name, index): - """Picks index-th element. (t_1, ..., t_n) -> (t_{index},).""" - return builder_layers.ArgIndexLayer.Params().Set(name=name, idx=[index]) - - def _Par(self, name, *subs): - """y = (f1, f2, ..., fn)(x). - - We feed the input tuple to all sub-layers and concatenates their output - tuples into one tuple. - - Args: - name: The layer name. - *subs: A list of sub-layers. - - Returns: - The param for the composed layer. - """ - - def ConcatTuples(tuples): - # tuples is a list of tuples. - return tuple(functools.reduce(lambda x, y: x + list(y), tuples, [])) - - def ConcatMeta(tuples): - return py_utils.NestedMap( - flops=0, - out_shapes=tuple( - functools.reduce(lambda x, y: x + list(y), tuples, []))) - - return builder_layers.ParallelLayer.Params().Set( - name=name, sub=list(subs), merge=ConcatTuples, merge_meta=ConcatMeta) - - def _Fn(self, name, fn, fn_out=None, fn_flops=None): - """y = fn(x). - - Applies a fn: tuple(Tensor) -> a single Tensor or tuple(Tensor) to the input - tuple. Typically, fn is a very simple python function. If fn is rather - complicated, we advice to implement the logic as a sub-class of BaseLayer. - - Args: - name: The layer name. - fn: A lambda tuple(Tensor) -> tuple(Tensor). - fn_out: A lambda tuple(tshape.Shape) -> output tuple(tshape.Shape) - fn_flops: A lambda tuple(tshape.Shape) -> estimated flops of fn. - If None, we assume flops == sum of elements in the inputs. - - Returns: - The param for the composed layer. - """ - - def FnMeta(*shapes): - """A lambda tuple(tshape.Shape) -> NestedMap{flops, out_shapes}.""" - if fn_out: - out_shapes = fn_out(*shapes) - if isinstance(out_shapes, tshape.Shape): - out_shapes = (out_shapes,) - else: - out_shapes = shapes - if fn_flops: - flops = fn_flops(*shapes) - else: - flops = sum([s.size for s in shapes]) - return py_utils.NestedMap(flops=flops, out_shapes=out_shapes) - - return builder_layers.FnLayer.Params().Set(name=name, fn=fn, fn_meta=FnMeta) - - def _Save(self, name): - """Returns a layer from which the activation and gradient can be accessed.""" - return layers.FetchLayer.Params().Set(name=name) - - def _AddFetches(self, name, body, fetches): - """Fetches saved activations in the body sub-layer. - - E.g.: - _AddFetches('foo', _Seq( 'stack', _Layer('layer1', ...), - _Save('layer1_out', ...), _Layer('layer2', ...), _Save('layer2_out', ...), - _Output('output', ...)), ['layer1_out', 'layer2_out']) - - The layer returns the stack's final output together with intermediate - activations from layer1_out and layer2_out. - - Args: - name: This layer's name. - body: The sub-layer. - fetches: A list of fetch names inside the sub-layer body. - - Returns: - A layer whose outputs correspond to the activations of fetch points - in the sub-layer body. [input1, input2, ..., inputN, fetch1, ..., fetchM]. - """ - return builder_layers.BranchLayer.Params().Set( - name=name, body=body, fetches=fetches) - - def _Rematerialize(self, name, body): - """Forces rematerialization on FProp of the body layer.""" - return builder_layers.RematerializationLayer.Params().Set( - name=name, body=body) - - def _BatchParallel(self, name, sub): - """Splits the batch and compute the forward pass on multiple devices. - - Args: - name: This layer's name. - sub: The sub-layer. - - Returns: - A BatchParallel layer which splits the batch and computes the forward pass - on multiple devices. - """ - return builder_layers.BatchParallelLayer.Params().Set(name=name, sub=sub) - - def _PrintShape(self, name): - """Print FProp input shape information.""" - return builder_layers.PrintShapeLayer.Params().Set(name=name) - - ########################################################################### - # Basic nn layers. - # - # The following method returns a layer param, whose FProp takes a single - # Tensor and returns a single Tensor. - # - # These methods are designed to have minimal knobs. Sub-classes which needs to - # be flexible can override these methods with different options. E.g., a - # sub-class builder can override _BN() to tune the decay option. - ########################################################################### - def _BN(self, name, dims): - """Batch norm.""" - return layers.BatchNormLayer.Params().Set(name=name, dim=dims, decay=0.99) - - def _LN(self, name, dims, use_fused_layernorm=False): - """Layer norm.""" - return layers.LayerNorm.Params().Set( - name=name, - input_dim=dims, - use_fused_layernorm=use_fused_layernorm, - fprop_dtype=self.params.fprop_dtype) - - def _Dropout(self, name, keep_prob, noise_shape_broadcast_dims=None): - """Returns a DropoutLayer Params.""" - if self.params.deterministic_dropout: - return layers.DeterministicDropoutLayer.Params().Set( - name=name, - keep_prob=keep_prob, - noise_shape_broadcast_dims=noise_shape_broadcast_dims) - return layers.DropoutLayer.Params().Set( - name=name, - keep_prob=keep_prob, - noise_shape_broadcast_dims=noise_shape_broadcast_dims, - fprop_dtype=self.params.fprop_dtype) - - def _Linear(self, name, idims, odims): - """Linear layer. y = matmul([..., idims], [idims, odims]).""" - return builder_layers.LinearLayer.Params().Set( - name=name, - input_dims=idims, - output_dims=odims, - fprop_dtype=self.params.fprop_dtype) - - def _Bias(self, name, dims): - """Bias layer. The bias is added to the last dimension of the input.""" - return builder_layers.BiasLayer.Params().Set( - name=name, dims=dims, fprop_dtype=self.params.fprop_dtype) - - def _Activation(self, name, fn=tf.nn.relu): - """Activation layer.""" - return self._Fn(name, fn) - - def _FC(self, name, idims, odims): - """Feed-forward fully connected. y = relu(matmul(x, w) + b).""" - # pyformat: disable - return self._Seq( - name, - self._Linear('linear', idims, odims), - self._Bias('bias', odims), - self._Activation('act')) - - def _MLP(self, name, dims): - """Multiple layers of feed-forward fully connected. - - Args: - name: The layer name. - dims: A list of int. i-th layer has dims[i] as its input dimension, and - dims[i+1] as its output dimensions. - - Returns: - The param for the composed layer. - """ - l = [] - for n, (i, o) in enumerate(zip(dims[:-1], dims[1:])): - l += [self._FC('l%03d' % n, i, o)] - return self._Seq(name, *l) - - def _Conv2D(self, name, filter_shape, filter_stride): - """Conv2D layer.""" - return layers.Conv2DLayerNoPadding.Params().Set( - name=name, filter_shape=filter_shape, filter_stride=filter_stride, - fprop_dtype=self.params.fprop_dtype) diff --git a/lingvo/core/builder_layers.py b/lingvo/core/builder_layers.py deleted file mode 100644 index 7dbbb123..00000000 --- a/lingvo/core/builder_layers.py +++ /dev/null @@ -1,1259 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Abstractions for composing layers.""" - -import re -from lingvo import compat as tf -from lingvo.core import base_layer -from lingvo.core import computation_cost -from lingvo.core import py_utils -from lingvo.core import recurrent -from lingvo.core import summary_utils -from lingvo.core import symbolic -from lingvo.core import tshape - - -class FirstNLayer(base_layer.BaseLayer): - """Returns the first n args.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('n', 0, 'The number of args to return.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.n > 0 - - def FProp(self, theta, *args): - """Return the first n args.""" - p = self.params - assert len(args) >= p.n - return tuple(args[:p.n]) if p.n > 1 else args[0] - - @classmethod - def FPropMeta(cls, p, *args): - py_utils.CheckShapes(args) - if p.n > 1: - out_shapes = args[:p.n] - else: - out_shapes = (args[0],) - return py_utils.NestedMap(flops=0, out_shapes=out_shapes) - - -class ArgIndexLayer(base_layer.BaseLayer): - """Select args with a list of indices.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('idx', [], 'The indices of args to return.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.idx - - def FProp(self, theta, *args): - """Return the indexed args.""" - p = self.params - assert p.idx - for i in p.idx: - assert 0 <= i <= len(args) - r = [args[i] for i in p.idx] - return tuple(r) if len(r) > 1 else r[0] - - -def _ToTuple(x): - return x if isinstance(x, tuple) else (x,) - - -def _MaybeStackExtraTheta(theta, all_vars, repeat): - var_set = set([key for key, _ in all_vars.FlattenItems()]) - values = [] - for key, value in theta.FlattenItems(): - if key not in var_set and value is not None: - # Replicate non-variable theta by p.repeat times. - value = tf.stack([value] * repeat) - values.append(value) - return theta.Pack(values) - - -class RepeatLayer(base_layer.BaseLayer): - """A layer which repeats itself sequentially using lingvo Recurrent.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('body', None, 'The param for the main network layer.') - p.Define('repeat', 1, - 'Repeat layers specified in \'body\' this many times.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.repeat > 0 - self.CreateChild('body', p.body) - - def _CreateChildrenVariables(self): - with tf.variable_scope(self.params.name): - with py_utils.VariableShapePrefixContext(self.params.repeat): - self.body.InstantiateVariables() - super()._CreateChildrenVariables() - - def FProp(self, theta, *args): - p = self.params - # Collects all variable key and values into sets. - theta_stack = _MaybeStackExtraTheta(theta.body, self.body.vars, p.repeat) - - def _ArgsToState(arg_list): - """Returns a NestedMap from a list of FProp args.""" - state = py_utils.NestedMap() - # Maintains a mapping from arg_idx to tensor. states cannot contains - # None tensors. - for idx in range(len(args)): - if arg_list[idx] is not None: - state['_s{}'.format(idx)] = arg_list[idx] - return state - - def _StateToArgs(state): - """Returns a list of FProp args from a NestedMap.""" - arg_list = [] - for idx in range(len(args)): - attr = '_s{}'.format(idx) - arg_list.append(state[attr] if attr in state else None) - if arg_list[-1] is not None: - arg_list[-1].set_shape(args[idx].shape) - return arg_list - - def _CellFn(unused_theta, state0, theta_i): - """Recurrent cell function wrapper of body.FProp.""" - # Retrieves fprop arguments from state and sets shapes. - frop_inputs = _StateToArgs(state0) - - # Sets shapes for theta_i as well. - for dst, src in zip(theta_i.Flatten(), theta_stack.Flatten()): - if src is not None: - dst.set_shape(tf.TensorShape(src.shape.as_list()[1:])) - - # Runs the actual body.FProp - frop_outputs = self.body.FProp(theta_i, *frop_inputs) - frop_outputs = _ToTuple(frop_outputs) - assert len(frop_outputs) == len(frop_inputs) - - # Passes fprop outputs to the next layer through state. - state1 = _ArgsToState(frop_outputs) - return state1, py_utils.NestedMap() - - with tf.name_scope(p.name): - # Add FProp arg list to state0. - state0 = _ArgsToState(args) - # Runs body.FProp k times using Recurrent where k = dim 0 of var_nmap. - _, state1 = recurrent.Recurrent( - theta=py_utils.NestedMap(), - state0=state0, - inputs=theta_stack, # Pass cell_fn theta through inputs. - cell_fn=_CellFn) - - # Retrieves fprop outputs from state1 and sets shapes. - output_tensors = _StateToArgs(state1) - return output_tensors[0] if len(args) == 1 else tuple(output_tensors) - - @classmethod - def FPropMeta(cls, p, *args): - py_utils.CheckShapes(args) - meta = p.body.cls.FPropMeta(p.body, *args) - py_utils.CheckShapes(meta.out_shapes) - total = meta.flops * p.repeat - return py_utils.NestedMap(flops=total, out_shapes=args) - - -class SoftCondLayer(base_layer.BaseLayer): - r"""A wrapper layer implements soft conditional computation. - - This layer computes - - output = p.body.FProp( \sum_i w_i theta_i, \*inputs) - - where the theta passed to p.body is the weighted average over p.num_experts - copies of theta.body, and w_i is the example-dependent coefficient for the - i-th expert (theta_i). - - Reference: - Soft Conditional Computation, B. Yang, G. Bender, Q.V. Le, J. Ngiam - https://arxiv.org/abs/1904.04971 - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_tasks', None, 'The Params for the main network layer.') - p.Define('body', None, 'The Params for the main network layer.') - p.Define('num_experts', None, 'Number of experts.') - p.Define( - 'cond_dim', None, - 'This layer maintains a weight matrix of shape [cond_dim, num_experts] ' - 'to map from inputs to the expert dimension.') - p.Define('nonzeros_mean', False, 'Whether to only use nonzero values for ' - 'calculating the mean.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.num_experts - assert p.cond_dim - - self.CreateChild('body', p.body) - - def _CreateChildrenVariables(self): - with tf.variable_scope(self.params.name): - # Prepends p.num_experts to the tensor shape of every variable created - # by p.body. - with py_utils.VariableShapePrefixContext(self.params.num_experts): - self.body.InstantiateVariables() - super()._CreateChildrenVariables() - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - # Create Variables for task weight mapping. - collections = [ - self.__class__.__name__ + '_vars', - ] - w_p = py_utils.WeightParams( - shape=[p.cond_dim, p.num_experts], - init=p.params_init, # TODO(huangyp): try zero init instead. - dtype=p.dtype, - collections=collections) - self.CreateVariable('w', w_p) - - def _GetExpertDist(self, theta, inputs, *args): - """Get the task id from inputs tensors.""" - # TODO(huangyp): support the more general case when batch size is not 1. - # Input shape can be either [batch, length, dim] or [length, batch, dim] - reshaped_inputs = tf.reshape(inputs, [-1, self.params.cond_dim]) - if self.params.nonzeros_mean: - per_example_emb = tf.reduce_sum(reshaped_inputs, 0) - nonzeros = tf.cast( - tf.math.count_nonzero(reshaped_inputs, 0), dtype=tf.float32) - per_example_emb /= (nonzeros + 1e-10) - else: - per_example_emb = tf.reduce_mean(reshaped_inputs, 0) - expert_dist = tf.nn.sigmoid(tf.einsum('i,ij->j', per_example_emb, theta.w)) - return expert_dist - - def FProp(self, theta, inputs, *args): - p = self.params - with tf.name_scope(p.name) as scope: - expert_dist = self._GetExpertDist(theta, inputs, *args) - if not self.do_eval: - summary_utils.histogram('soft_cond_{}'.format(scope), expert_dist) - - # Excludes non-variable extra_theta like global_step. - var_set = set([key for key, _ in self.body.vars.FlattenItems()]) - values = [] - for key, value in theta.body.FlattenItems(): - if key in var_set and value is not None: - # Weighted average for all variables created in the body layer. - value = tf.einsum('i,i...->...', expert_dist, value) - values.append(value) - weighted_theta = theta.body.Pack(values) - return self.body.FProp(weighted_theta, inputs, *args) - - -class ParallelRepeatLayer(RepeatLayer): - """A layer that connects identical sublayers in parallel. - - This layer consists of p.repeat copies of sub-layers, each of which - shares the same structure but with different weights. Conceptually it - computes: - - output= [f(theta[i], input[i]) for i in range(p.repeat)] - - where f is the fprop function of the sublayer, theta[i] and input[i] are - the weights and inputs of the i-th sublayer. - """ - - def _InferOutShapes(self, args): - input_shapes = [ - None if arg is None else tshape.Shape(arg.get_shape().as_list()[1:]) - for arg in args - ] - out_shapes = self.body.FPropMeta(self.body.params, *input_shapes).out_shapes - return [None if s is None else s.ToTensorShape() for s in out_shapes] - - def FProp(self, theta, *args): - """Runs p.repeat copies of self.body.FProp independently. - - Args: - theta: Layer model parameters. The shape of each variable in theta is - always [p.repeat, ...]. And the i-th slice theta[i] becomes theta of the - i-th copy of self.body. - *args: Input arguments. The shape of each tensor in args is always - [p.repeat, ....]. And the list [arg[i] for arg in args] becomes inputs - to the i-th copy of self.body.FProp. - - Returns: - The accumulated output_tensors. Each tensor t in the return has the shape - [p.repeat, ....] and the tuple (t[i] for i in output_tensors) is the - return tuple of the i-th self.body.FProp. - """ - p = self.params - for arg in args: - if arg is not None: - arg = py_utils.HasShape(arg, [p.repeat], ndims=1) - - theta_stack = _MaybeStackExtraTheta(theta.body, self.body.vars, p.repeat) - inputs = py_utils.NestedMap(theta=theta_stack, args=list(args)) - # Infer out_shapes from FPropMeta. - out_shapes = self._InferOutShapes(args) - - def _CellFn(unused_theta, unused_state0, inputs): - """Recurrent cell function wrapper of body.FProp.""" - # Sets shapes for both theta and inputs to self.body.FProp. - for dst, src in zip(inputs.args + inputs.theta.Flatten(), - list(args) + theta_stack.Flatten()): - if src is not None: - dst.set_shape(tf.TensorShape(src.shape.as_list()[1:])) - - # Runs the actual body.FProp - fprop_outputs = self.body.FProp(inputs.theta, *inputs.args) - fprop_outputs = _ToTuple(fprop_outputs) - assert len(fprop_outputs) == len(out_shapes) - # Passes fprop outputs to the next layer through state. - state1 = py_utils.NestedMap(outputs=list(fprop_outputs)) - return state1, py_utils.NestedMap() - - with tf.name_scope(p.name): - # Initiate state0 with inferred output shapes. - state0 = py_utils.NestedMap( - outputs=[tf.zeros(shape, args[0].dtype) for shape in out_shapes]) - # Runs body.FProp p.repeat times using Recurrent. - acc_states, _ = recurrent.Recurrent( - theta=py_utils.NestedMap(), - state0=state0, - inputs=inputs, - cell_fn=_CellFn) - - # Retrieves fprop outputs from state1 and sets shapes. - output_tensors = tuple(acc_states.outputs) - for out_idx in range(len(output_tensors)): - output_tensors[out_idx].set_shape( - tf.TensorShape([p.repeat] + out_shapes[out_idx].as_list())) - - return output_tensors[0] if len(args) == 1 else tuple(output_tensors) - - @classmethod - def FPropMeta(cls, p, *args): - py_utils.CheckShapes(args) - input_shapes = [ - None if arg is None else tshape.Shape(arg.get_shape().as_list()[1:]) - for arg in args - ] - meta = p.body.cls.FPropMeta(p.body, *input_shapes) - py_utils.CheckShapes(meta.out_shapes) - total = meta.flops * p.repeat - out_shapes = [ - None if s is None else tshape.Shape([p.repeat] + s[:]) - for s in meta.out_shapes - ] - return py_utils.NestedMap(flops=total, out_shapes=tuple(out_shapes)) - - -class SequentialLayer(base_layer.BaseLayer): - """A layer which connects a few layers in a sequence.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('sub', [], 'A list of layers\' params.') - p.Define('repeat', 1, 'Repeat layers specified in \'sub\' ' - 'this many times.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - if p.repeat <= 1: - self._seq = [] - for sub in p.sub: - self.CreateChild(sub.name, sub) - self._seq.append((sub.name, self.children[sub.name])) - else: - # We create 'repeat' number of sub layers. Each sub layer is a - # sequential layer specified by 'sub'. This allows us to name each - # repetition with a unique name. - children = [] - for i in range(p.repeat): - children.append(p.Copy().Set(name='%03d' % i, repeat=1)) - self.CreateChildren('rep', children) - - def FProp(self, theta, *args): - p = self.params - with tf.name_scope(p.name): - tf.logging.vlog(1, 'layer %s', self.params.name) - if p.repeat <= 1: - for (name, ch) in self._seq: - th = theta[name] - args = _ToTuple(args) - tf.logging.vlog(1, 'SequentialLayer: call %s %s %d %s', - ch.params.name, ch, len(args), str(args)) - args = ch.FProp(th, *args) - else: - for (ch, th) in zip(self.rep, theta.rep): - args = _ToTuple(args) - tf.logging.vlog(1, ' call %s %s %d %s', ch.params.name, ch, - len(args), str(args)) - args = ch.FProp(th, *args) - args = _ToTuple(args) - return args[0] if len(args) == 1 else args - - @classmethod - def FPropMeta(cls, p, *args): - py_utils.CheckShapes(args) - total = 0 - for _ in range(p.repeat): - for sub in p.sub: - tf.logging.vlog(1, ' seq abs fprop %s %s %d %s', sub.name, sub.cls, - len(args), str(args)) - meta = sub.cls.FPropMeta(sub, *args) - py_utils.CheckShapes(meta.out_shapes) - total += meta.flops - args = meta.out_shapes - return py_utils.NestedMap(flops=total, out_shapes=args) - - -class UnarySequentialLayer(base_layer.BaseLayer): - """A layer which connects a few layers in a sequence. - - Each layer FProp must take a single input arg (besides theta) and its return - value will be used as the input for the next layer or as the final output - if it's the last layer. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('sub', [], 'A list of layers\' params.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - self._seq = [] - for sub in p.sub: - self.CreateChild(sub.name, sub) - self._seq.append((sub.name, self.children[sub.name])) - - def FProp(self, theta, x): - tf.logging.vlog(1, 'layer %s', self.params.name) - with tf.name_scope(self.params.name): - for (name, ch) in self._seq: - th = theta[name] - tf.logging.vlog(1, ' call %s %s %s', ch.params.name, ch, x) - x = ch.FProp(th, x) - return x - - @classmethod - def FPropMeta(cls, p, x): - total = 0 - for sub in p.sub: - tf.logging.vlog(1, ' seq abs fprop %s %s %s', sub.name, sub.cls, x) - meta = sub.cls.FPropMeta(sub, x) - total += meta.flops - x = meta.out_shapes - return py_utils.NestedMap(flops=total, out_shapes=x) - - -class GraphTensors: - """A collection of named tensors (or NestedMaps of tensors).""" - - def __init__(self): - self._named_tensors = py_utils.NestedMap() - - def StoreTensor(self, path, tensor): - """Add tensor 't' to 'named_tensors' at 'path'. - - A path may be a name or a path into a NestedMap. For instance, - StoreTensor('a.b.c', [1]), is equivalent to this: - {'a', {'b': {'c': [1]}}. - - NestedMaps will be created if they do not already exist, or modified if they - do exist. However, tensors cannot be overwritten. - - Args: - path: A path input a NestedMap. - tensor: The item to store (may be a NestedMap or a tensor). - """ - names = path.strip().split('.') - named_tensors = self._named_tensors - while len(names) > 1: - n = names.pop(0) - assert isinstance(named_tensors, py_utils.NestedMap), named_tensors - if n not in named_tensors: - named_tensors[n] = py_utils.NestedMap() - named_tensors = named_tensors[n] - n = names.pop(0) - if n in named_tensors: - raise ValueError('A tensor named "%s" (%s) already exists.' % (n, path)) - named_tensors[n] = tensor - - def GetTensor(self, path): - """Returns the tensor at 'path' in 'named_tensors'. - - Path may be a NestedMap key or a path through a series of NestedMaps. - For instance, a path of 'a.b.c' could be used to retrieve [1] from - this structure: - {'a': {'b': {'c': [1]}}} - - Args: - path: A path through a series of NestedMaps. - """ - names = path.strip().split('.') - named_tensors = self._named_tensors - while names: - assert isinstance(named_tensors, py_utils.NestedMap), named_tensors - n = names.pop(0) - assert n in named_tensors, '%s not found in %s' % (n, named_tensors) - named_tensors = named_tensors[n] - return named_tensors - - -class GraphSignature: - """Represents the input/output signature of a GraphLayer. - - A signature is of the form: - input->output - or, optionally just an input: - input - - The output part may be: - out1 (a tensor name) - out1.a.b (a path into a NestedMap) - out1,myvar,out2.x (or a sequence of these things) - - The input part may be: - in1 (a tensor name) - in1.b (a path into a NestedMap) - [in1,myvar.c] (a list) - (a=in1,b=in2) (a NestedMap, using the syntax of the NestedMap constructor) - in1,[a,b] (or a sequence of these things) - - In BNF:: - - key := [A-Za-z_][A-Za-z0-9_]* - path := key | path '.' key - path_seq := | path | path_seq ',' path - item := path | list | map - list := '[' item_seq ']' - item_seq := | item | item_seq ',' item - map := '(' map_pair_seq ')' - map_pair := key '=' item - map_pair_seq := | map_pair | map_pair_seq ',' map_pair - input := item_seq - output := path_seq - """ - - def __init__(self, signature): - self.signature = signature - self.symbols = set(['[', ']', '=', '(', ')', ',']) - parts = self.signature.split('->') - self._input_signature = parts[0] - self._output_signature = None - if len(parts) > 1: - self._output_signature = parts[1] - - self._tokens = self._TokenizeInputs(self._input_signature) - self._outputs = self._ParseOutputs(self._output_signature) - self._inputs = self._ParseInputs() - - def __str__(self): - return '(%s->%s)' % (self._inputs, self.outputs) - - @property - def inputs(self): - return self._inputs - - @property - def outputs(self): - return self._outputs - - def _ParseOutputs(self, outputs): - """Splits the output spec into comma-delimited parts. - - Args: - outputs: A string containing the output specification, like, 'a,b'. - - Returns: - The parsed representation, e.g. ['a', 'b'] - """ - if outputs is None: - return [] - o_tensors = [x.strip() for x in outputs.split(',')] - for x in o_tensors: - assert x - return o_tensors - - def _TokenizeInputs(self, inputs): - """Splits the input signature into tokens (not data structures). - - Args: - inputs: A string containing the input speficiation, like "[a,b],c" - - Returns: - The tokenized representation, like ['[', 'a', ',', 'b', ']', ',', 'c'] - """ - start = -1 - tokens = [] - for j in range(len(inputs)): - # Each letter can either be a symbol or part of a variable/path. - if inputs[j] in self.symbols: - if start >= 0: - # Output the last variable/path. - tokens.append(inputs[start:j].strip()) - start = -1 - tokens.append(inputs[j]) - elif start < 0: - start = j - if start >= 0: - tokens.append(inputs[start:].strip()) - id_regex = re.compile(r'[_a-zA-Z][_a-zA-Z0-9]*(\.[_a-zA-Z][_a-zA-Z0-9]*)*$') - for token in tokens: - if not (token in self.symbols or id_regex.match(token)): - raise ValueError( - 'token(%s) is not a symbol(%r) or matched by id_regex.' % - (token, self.symbols)) - # Wrapping the tokens in list brackets allows us to parse this using - # _ConsumeList. - return ['['] + tokens + [']'] - - def _ConsumePath(self): - """Return the path found at the current token position and increment. - - Returns: - The path at the current token position. - """ - if self._i >= len(self._tokens): - raise ValueError('Ran out of tokens while looking for a path/key') - if self._tokens[self._i] in self.symbols: - raise ValueError('Found a symbol %s while looking for a path/key' % - (self._tokens[self._i])) - self._i += 1 - return self._tokens[self._i - 1] - - def _ConsumeKey(self): - """Return the key found at the current token position and increment. - - Returns: - The key at the current token position. - """ - token = self._ConsumePath() - assert '.' not in token, token - return token - - def _ConsumeSymbol(self, symbol): - """Verify that the current token is this symbol, and increment. - - Args: - symbol: The symbol that we expect at the current token position. - """ - assert symbol in self.symbols - if self._i >= len(self._tokens): - raise ValueError('Ran out of tokens while looking for a %s' % symbol) - if not self._MaybeConsumeSymbol(symbol): - raise ValueError('Found a symbol %s while looking for %s' % - (self._tokens[self._i], symbol)) - - def _MaybeConsumeSymbol(self, symbol): - """Attempt to consume the given symbol. - - The symbol must be a member of the symbol set: []()=, - - Args: - symbol: The symbol that we expect at the current token position. - - Returns: - True if the symbol was consumed, False otherwise. - """ - assert symbol in self.symbols - if self._i < len(self._tokens) and self._tokens[self._i] == symbol: - self._i += 1 - return True - return False - - def _ConsumeItem(self): - """Return whatever is at the current token position and increment. - - Returns: - A list, NestedMap, or path/key. - """ - if self._i >= len(self._tokens): - raise ValueError( - 'Ran out of tokens while looking for a variable, list or NestedMap.') - if self._tokens[self._i] == '(': - return self._ConsumeMap() - if self._tokens[self._i] == '[': - return self._ConsumeList() - return self._ConsumePath() - - def _ConsumeMap(self): - """Return the NestedMap that starts at the current position, and increment. - - Returns: - The NestedMap that starts at the current token position. - """ - if self._i >= len(self._tokens): - raise ValueError('Ran out of tokens while looking for a NestedMap.') - if self._tokens[self._i] != '(': - raise ValueError('Expected ( at token position %d' % self._i) - self._i += 1 - if self._MaybeConsumeSymbol(')'): - # Empty NestedMaps are allowed. - return py_utils.NestedMap() - result = py_utils.NestedMap() - while self._i < len(self._tokens): - name = self._ConsumeKey() - self._ConsumeSymbol('=') - result[name] = self._ConsumeItem() - if self._MaybeConsumeSymbol(')'): - return result - self._ConsumeSymbol(',') - raise ValueError('Ran out of tokens while looking for end of NestedMap.') - - def _ConsumeList(self): - """Return the list that starts at the current position, and increment. - - Returns: - The list that starts at the current token position. - """ - if self._i >= len(self._tokens): - raise ValueError('Ran out of tokens while looking for a list.') - if self._tokens[self._i] != '[': - raise ValueError('Expected [ at token position %d' % self._i) - self._i += 1 - if self._MaybeConsumeSymbol(']'): - # Empty lists are allowed. - return [] - result = [] - while self._i < len(self._tokens): - result.append(self._ConsumeItem()) - if self._MaybeConsumeSymbol(']'): - return result - self._ConsumeSymbol(',') - raise ValueError('Ran out of tokens while looking for end of list.') - - def _ParseInputs(self): - """Parse the inputs signature string. - - Returns: - dict: The parsed inputs structure, like ``{'a': ['b', 'c']}``. - """ - self._i = 0 - # The tokenization process adds fake '[' and ']' tokens so we can - # parse them as a list. - return self._ConsumeList() - - -class GraphLayer(base_layer.BaseLayer): - r"""A layer that connects a few layers in a simple data flow graph. - - Params.sub specifies a list of (signature, layer param) pairs for all the - sub-layers. 'Signature' of a layer specifies the signature of the - corresponding layer's FProp function. It also names its input and output - tensors. A 'signature' is of the form 'x,b->c', where tensors on the left - side of '->' specifies the input to the layer and tensors to the right - specifies the output from the layer. Note, the input to a layer has to be - produced before, either as input to the GraphLayer or as produced by some - previous layer. The output of a layer must be uniquely named, i.e. they can't - reuse names assigned previous layer output or the input to this GraphLayer. - - The full grammar of the signature is described in the GraphSignature class - definition above. - - Example - input: ['a', 'b'] - 'a->c', Fn(lambda x : tf.nn.relu(x)) - 'c,b->d', Fn(lambda x, y : x + y) - - The above example computes relu(a) + b - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_endpoints', [], 'Names of the input tensors.') - p.Define('output_endpoints', [], 'Names of the output tensors.') - # TODO(yonghui): Define a NamedTuple for this pair. - p.Define('sub', [], 'A list of (signature, layer params) pairs.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.input_endpoints - self._seq = [] - for i, (signature, sub) in enumerate(p.sub): - assert signature - sig = GraphSignature(signature) - assert sig.outputs, '{}'.format(signature) - name = sub.name - if not name: - name = '%s_%02d' % (sig.outputs[0], i) - sub.name = name - self.CreateChild(name, sub) - self._seq.append((name, sig, self.children[name])) - - def FProp(self, theta, *args): - p = self.params - - graph_tensors = self._fprop = GraphTensors() - with tf.name_scope(p.name): - if len(p.input_endpoints) != len(args): - raise ValueError( - 'Wrong number of inputs for {}: required={}, provided={}'.format( - p.name, len(p.input_endpoints), len(args))) - for n, t in zip(p.input_endpoints, args): - if isinstance(t, py_utils.NestedMap): - assert all(isinstance(x, tf.Tensor) for x in t.Flatten()), t - else: - assert isinstance(t, tf.Tensor) - graph_tensors.StoreTensor(n, t) - - ch_out = None - for i, (name, sig, ch) in enumerate(self._seq): - th = theta[name] - template = py_utils.NestedMap(inputs=sig.inputs) - packed = template.Transform(graph_tensors.GetTensor) - input_args = packed.inputs - tf.logging.vlog(1, 'signature: %s', p.sub[i][0]) - tf.logging.vlog(1, 'GraphLayer: call %s %s %d %s', ch.params.name, ch, - len(input_args), str(input_args)) - ch_out = ch.FProp(th, *input_args) - if len(sig.outputs) == 1: - ch_out = (ch_out,) - assert len(sig.outputs) == len(ch_out) - for n, t in zip(sig.outputs, ch_out): - graph_tensors.StoreTensor(n, t) - - layer_out = tuple(graph_tensors.GetTensor(x) for x in p.output_endpoints) - if len(layer_out) == 1: - layer_out = layer_out[0] - - return layer_out - - @classmethod - def FPropMeta(cls, p, *args): - py_utils.CheckShapes(args) - total = 0 - - graph_tensors = GraphTensors() - assert len(p.input_endpoints) == len(args) - for n, t in zip(p.input_endpoints, args): - graph_tensors.StoreTensor(n, t) - - ch_out = None - for signature, sub in p.sub: - sig = GraphSignature(signature) - template = py_utils.NestedMap(inputs=sig.inputs) - packed = template.Transform(graph_tensors.GetTensor) - input_args = packed.inputs - - meta = sub.cls.FPropMeta(sub, *input_args) - total += meta.flops - ch_out = meta.out_shapes - assert len(ch_out) == len(sig.outputs) - for n, t in zip(sig.outputs, ch_out): - graph_tensors.StoreTensor(n, t) - - layer_out = tuple(graph_tensors.GetTensor(x) for x in p.output_endpoints) - return py_utils.NestedMap(flops=total, out_shapes=layer_out) - - -class ParallelLayer(base_layer.BaseLayer): - """A layer which connects a few layers in a parallel.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'sub', [], 'A list of layers\' params. Each layer\'s ' - 'FProp must return one Tensor or a tuple of Tensors. ' - 'Their return values then can be merged according to the ' - 'merge method. ') - p.Define( - 'merge', None, 'Method to combine sub-layers\' outputs.' - 'It must be a callable list(tuple(tf.Tensor)) -> tuple(tf.Tensor).') - p.Define( - 'merge_meta', None, 'Callable to compute the meta of merge(). It ' - 'takes a list of tuples of TensorShape, and returns a NestedMap with ' - 'flops and out_shapes, etc.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - self._seq = [] - for sub in p.sub: - self.CreateChild(sub.name, sub) - self._seq.append((sub.name, self.children[sub.name])) - - def FProp(self, theta, *args): - p = self.params - - with tf.name_scope(p.name): - # Computes sub layers in parallel. - outputs = [] - for (name, ch) in self._seq: - th = theta[name] - out = ch.FProp(th, *args) - if isinstance(out, (list, tuple)): - outputs.append(tuple(out)) - else: - outputs.append((out,)) - rets = p.merge(outputs) - return rets if len(rets) > 1 else rets[0] - - @classmethod - def FPropMeta(cls, p, *args): - py_utils.CheckShapes(args) - total = 0 - outputs = [] - for sub in p.sub: - tf.logging.vlog(1, ' par abs fprop %s %s %d %s', sub.name, sub.cls, - len(args), str(args)) - meta = sub.cls.FPropMeta(sub, *args) - py_utils.CheckShapes(meta.out_shapes) - meta.VLog( - 1, ' par abs fprop {} {} {} {}'.format(sub.name, sub.cls, len(args), - str(args))) - total += meta.flops - outputs.append(meta.out_shapes) - - meta = p.merge_meta(outputs) - py_utils.CheckShapes(meta.out_shapes) - meta.flops += total - return meta - - -class MapLayer(base_layer.BaseLayer): - """A layer applies a lambda on every argument.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('fn', None, 'A callable tensor->tensor.') - p.Define('fn_meta', None, 'A callable shape->(flops, shape).') - p.Define('kwargs', {}, 'Keyword arguments to fn.') - return p - - def FProp(self, theta, *args): - r"""Applies lambda(x, \*kwargs) for every non-None arg.""" - del theta - p = self.params - with tf.name_scope(p.name): - ret = [None if x is None else p.fn(x, **p.kwargs) for x in args] - return tuple(ret) if len(ret) > 1 else ret[0] - - @classmethod - def FPropMeta(cls, p, *args): - flops, rets = 0, [] - for x in args: - if x is None: - rets.append(None) - else: - cost, shape = p.fn_meta(x) - py_utils.CheckShapes((shape,)) - flops += cost - rets.append(shape) - return py_utils.NestedMap(flops=flops, out_shapes=tuple(rets)) - - -class LinearLayer(base_layer.BaseLayer): - """Linear layer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dims', 0, 'Depth of the input.') - p.Define('output_dims', 0, 'Depth of the output.') - return p - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - self.CreateVariable( - 'w', - py_utils.WeightParams( - shape=[p.input_dims, p.output_dims], - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars'])) - - def FProp(self, theta, inputs): - """Apply projection to inputs. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - inputs: The inputs tensor. Shaped [..., input_dims]. - - Returns: - Projected inputs. - """ - p = self.params - with tf.name_scope(p.name): - computation_cost.Add( - self, 'flops', - tf.reduce_prod(tf.cast(tf.shape(inputs)[:-1], tf.int64)) * - tf.cast(symbolic.ToTensor(p.input_dims * p.output_dims), tf.int64) * - 2) - return py_utils.ProjectLastDim(inputs, theta.w, p.input_dims, - p.output_dims) - - @classmethod - def FPropMeta(cls, p, inputs): - py_utils.CheckShapes((inputs,)) - assert p.input_dims == inputs[-1] - # c_{ij} += x_{ik} * y_{kj} are considered 2 flops. - return py_utils.NestedMap( - flops=inputs.size * p.output_dims * 2, - out_shapes=(inputs[:-1] + [p.output_dims],)) - - -class BiasLayer(base_layer.BaseLayer): - """Bias layer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('dims', 0, 'Depth of the input.') - return p - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - self.CreateVariable( - 'b', - py_utils.WeightParams( - shape=[p.dims], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars'])) - - def FProp(self, theta, inputs): - """Adds bias to inputs. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - inputs: The inputs tensor. Shaped [..., dims]. - - Returns: - Inputs plus bias. - """ - with tf.name_scope(self.params.name): - return inputs + theta.b - - @classmethod - def FPropMeta(cls, p, inputs): - py_utils.CheckShapes((inputs,)) - assert inputs[-1] == p.dims - return py_utils.NestedMap(flops=inputs.size, out_shapes=(inputs,)) - - -class BranchLayer(base_layer.BaseLayer): - """A layer to help constructing a network structure with multiple outputs.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('body', None, 'The param for the main network layer.') - p.Define( - 'fetches', [], 'Fetch points within the body layer. Each fetch ' - 'layers\' activation is appended to the output of body.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - self.CreateChild('body', p.body) - - def FProp(self, theta, *args): - p = self.params - with tf.name_scope(p.name): - args = _ToTuple(self.body.FProp(theta.body, *args)) - for fetch in p.fetches: - args += (self.body.GetDescendant(fetch).activation,) - return args - - -class BatchParallelLayer(base_layer.BaseLayer): - """A layer splits the batch and compute the FProp on multiple devices.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('sub', None, 'A layer param.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - self.CreateChild('sub', p.sub) - - def FProp(self, theta, *args): - """FProp through multiple devices in the split. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - *args: A tuple of Tensors (one or more). Every tensor's first dimension is - the same (the batch dimension). - - Returns: - The sub layer's output. - """ - p = self.params - with tf.name_scope(p.name): - assert all(isinstance(x, tf.Tensor) for x in args) - cluster = self.cluster - num = cluster.num_devices_per_split - if num == 1: - return self.sub.FProp(theta.sub, *args) - inps = py_utils.SplitRecursively(list(args), num, axis=0) - outs = [] - for i, xs in enumerate(inps): - device = cluster.WorkerDeviceInModelSplit(i) - tf.logging.info('%d on device %s', i, device) - with tf.device(device): - ys = self.sub.FProp(theta.sub, *xs) - if isinstance(ys, tuple): - outs += [list(ys)] - else: - outs += [ys] # ys is a single tensor - ret = py_utils.ConcatRecursively(outs, axis=0) - if isinstance(ret, list): - return tuple(ret) - else: - return ret # ys is a single tensor - - -class FnLayer(base_layer.BaseLayer): - """A layer applies a function on a tuple of tensors.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('fn', None, 'A lambda tuple(Tensor) -> tuple(Tensor) ' - 'or a single Tensor.') - p.Define( - 'fn_meta', None, 'Callable to compute the meta of fn(). It ' - 'takes a tuple of TensorShape, and returns a NestedMap with ' - 'flops and out_shapes, etc.') - return p - - def FProp(self, theta, *args): - r"""Applies a function (p.fn) on args. - - Args: - theta: Unused. - *args: A tuple of Tensors (one or more). - - Returns: - fn(\*args). - """ - with tf.name_scope(self.params.name): - return self.params.fn(*args) - - @classmethod - def FPropMeta(cls, p, *args): - py_utils.CheckShapes(args) - meta = p.fn_meta(*args) - py_utils.CheckShapes(meta.out_shapes) - return meta - - -class RematerializationLayer(base_layer.BaseLayer): - """A wrapper layer with rematerialization.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('body', None, - 'The main layer whose FProp will be wrapped by RematerializeFn.') - return p - - def __init__(self, params): - super().__init__(params) - self.CreateChild('body', self.params.body) - - def FProp(self, theta, *xs): - input_list = theta.body.Flatten() - theta_len = len(input_list) - input_list += list(xs) - input_len = len(input_list) - - def Fn(*args): - body_theta = theta.body.Pack(args[:theta_len]) - return self.body.FProp(body_theta, *args[theta_len:input_len]) - - return py_utils.RematerializeFn(Fn, *input_list) - - @classmethod - def FPropMeta(cls, p, *args): - py_utils.CheckShapes(args) - return p.body.cls.FPropMeta(p.body, *args) - - -class PrintShapeLayer(base_layer.BaseLayer): - """A layer prints debug information.""" - - def FProp(self, theta, *args): - p = self.params - with tf.name_scope(p.name) as name_scope: - for i, arg in enumerate(args): - if not isinstance(arg, tf.Tensor): - tf.logging.info( - 'FProp non-Tensor input in {}: arg_{} arg = {}'.format( - name_scope, i, arg)) - else: - tf.logging.info( - 'FProp inputs in {}: arg_{} shape = {} dtype = {}'.format( - name_scope, i, arg.shape, arg.dtype.name)) - if len(args) == 1: - return args[0] - else: - return args - - @classmethod - def FPropMeta(cls, p, *args): - py_utils.CheckShapes(args) - return py_utils.NestedMap(flops=0, out_shapes=args) diff --git a/lingvo/core/builder_layers_test.py b/lingvo/core/builder_layers_test.py deleted file mode 100644 index 8697c55a..00000000 --- a/lingvo/core/builder_layers_test.py +++ /dev/null @@ -1,554 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for builder_layers.""" - -from lingvo import compat as tf -from lingvo.core import builder_layers as layers -from lingvo.core import layers as lingvo_layers -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.core import tshape -import numpy as np - - -class BuilderLayerTest(test_utils.TestCase): - - def testFirstNLayerFProp(self): - with self.session(): - params = layers.FirstNLayer.Params() - params.name = 'fn' - params.n = 2 - - fn_layer = layers.FirstNLayer(params) - a = tf.constant(1) - b = tf.constant(2) - c = tf.constant(3) - - fn_out = self.evaluate(fn_layer.FPropDefaultTheta(a, b, c)) - - self.assertEqual((1, 2), fn_out) - - def testArgIndexLayerFProp(self): - with self.session(): - params = layers.ArgIndexLayer.Params().Set(name='argidx', idx=[1, 3]) - argidx_layer = layers.ArgIndexLayer(params) - args = [tf.constant(i) for i in range(5)] - - argidx_out = self.evaluate(argidx_layer.FPropDefaultTheta(*args)) - self.assertEqual((1, 3), argidx_out) - - def testSequentialLayer(self): - g = tf.Graph() - with g.as_default(), self.SetEval(True): - tf.random.set_seed(24332) - p = layers.SequentialLayer.Params().Set( - name='seq', - repeat=2, - sub=[ - lingvo_layers.FCLayer.Params().Set( - name='foo', input_dim=32, output_dim=8), - lingvo_layers.FCLayer.Params().Set( - name='bar', input_dim=8, output_dim=8), - lingvo_layers.FCLayer.Params().Set( - name='baz', input_dim=8, output_dim=32), - lingvo_layers.DropoutLayer.Params().Set( - name='dropout', keep_prob=0.5) - ]) - l = p.Instantiate() - x = tf.random.normal(shape=[2, 32]) - y = l.FPropDefaultTheta(x) - l.vars.Transform(lambda x: x.shape).VLog(0, 'vars: ') - - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - x_val, y_val, w = self.evaluate([x, y, l.vars]) - - act = x_val - # relu(act \dot w + b) - for i in range(2): - act = np.maximum(0, np.dot(act, w.rep[i].foo.w) + w.rep[i].foo.b) - act = np.maximum(0, np.dot(act, w.rep[i].bar.w) + w.rep[i].bar.b) - act = np.maximum(0, np.dot(act, w.rep[i].baz.w) + w.rep[i].baz.b) - self.assertAllClose(act, y_val) - - def testEmptySequentialLayer(self): - g = tf.Graph() - with g.as_default(): - tf.random.set_seed(24332) - p = layers.SequentialLayer.Params().Set(name='seq') - l = p.Instantiate() - x = tf.random.normal(shape=[2, 32]) - y = l.FPropDefaultTheta(x) - self.assertIsInstance(y, tf.Tensor) - - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - x_val, y_val = self.evaluate([x, y]) - self.assertAllEqual(x_val, y_val) - - def testEmptySequentialLayerFPropMeta(self): - g = tf.Graph() - with g.as_default(): - p = layers.SequentialLayer.Params().Set(name='seq') - l = p.Instantiate() - x = py_utils.NestedMap(val=tf.random.normal(shape=[2, 32])) - y = l.FPropDefaultTheta(x) - self.assertIsInstance(y.val, tf.Tensor) - y_shape = l.FPropMeta( - p, py_utils.Transform(lambda t: tshape.Shape(t.shape), - x)).out_shapes[0] - self.assertEqual(y.val.shape.as_list(), - y_shape.val.ToTensorShape().as_list()) - - def testUnarySequentialLayer(self): - g = tf.Graph() - with g.as_default(), self.SetEval(True): - tf.random.set_seed(24332) - p = layers.UnarySequentialLayer.Params().Set( - name='seq', - sub=[ - lingvo_layers.FCLayer.Params().Set( - name='foo', input_dim=32, output_dim=8), - lingvo_layers.FCLayer.Params().Set( - name='bar', input_dim=8, output_dim=8), - lingvo_layers.FCLayer.Params().Set( - name='baz', input_dim=8, output_dim=32), - lingvo_layers.DropoutLayer.Params().Set( - name='dropout', keep_prob=0.5) - ]) - l = p.Instantiate() - x = tf.random.normal(shape=[2, 32]) - y = l.FPropDefaultTheta(x) - l.vars.Transform(lambda x: x.shape).VLog(0, 'vars: ') - - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - x_val, y_val, w = self.evaluate([x, y, l.vars]) - - act = x_val - # relu(act \dot w + b) - act = np.maximum(0, np.dot(act, w.foo.w) + w.foo.b) - act = np.maximum(0, np.dot(act, w.bar.w) + w.bar.b) - act = np.maximum(0, np.dot(act, w.baz.w) + w.baz.b) - self.assertAllClose(act, y_val) - - def testParallelLayer(self): - g = tf.Graph() - with g.as_default(), self.SetEval(True): - tf.random.set_seed(24332) - p = layers.ParallelLayer.Params().Set( - name='test', - merge=lambda xs: tuple([tf.add_n(x) for x in zip(*xs)]), - sub=[ - lingvo_layers.FCLayer.Params().Set( - name='foo', input_dim=32, output_dim=4), - lingvo_layers.FCLayer.Params().Set( - name='bar', input_dim=32, output_dim=4), - layers.SequentialLayer.Params().Set( - name='seq', - sub=[ - lingvo_layers.FCLayer.Params().Set( - name='baz', input_dim=32, output_dim=4), - lingvo_layers.DropoutLayer.Params().Set( - name='dropout', keep_prob=0.5) - ]) - ]) - l = p.Instantiate() - x = tf.random.normal(shape=[2, 32]) - y = l.FPropDefaultTheta(x) - - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - x_val, y_val, w = self.evaluate([x, y, l.vars]) - - out = [] - act = x_val - # relu(act \dot w + b) - out += [np.maximum(0, np.matmul(act, w.foo.w) + w.foo.b)] - self.assertEqual(out[-1].shape, (2, 4)) - out += [np.maximum(0, np.matmul(act, w.bar.w) + w.bar.b)] - self.assertEqual(out[-1].shape, (2, 4)) - out += [np.maximum(0, np.matmul(act, w.seq.baz.w) + w.seq.baz.b)] - self.assertEqual(out[-1].shape, (2, 4)) - - np_result = out[0] - for v in out[1:]: - np_result = np.add(np_result, v) - self.assertAllClose(np_result, y_val) - - def testParallelMatmulLayer(self): - g = tf.Graph() - with g.as_default(): - tf.random.set_seed(24332) - - def MergeFn(xs): - result = [] - for x in zip(*xs): - val = x[0] - for v in x[1:]: - val = tf.matmul(val, v) - result.append(val) - return tuple(result) - - p = layers.ParallelLayer.Params().Set( - name='parallel', - merge=MergeFn, - sub=[ - lingvo_layers.FCLayer.Params().Set( - name='foo', input_dim=32, output_dim=4), - lingvo_layers.FCLayer.Params().Set( - name='bar', input_dim=32, output_dim=4), - lingvo_layers.FCLayer.Params().Set( - name='baz', input_dim=32, output_dim=4) - ]) - l = p.Instantiate() - x = tf.random.normal(shape=[2, 4, 32]) - y = l.FPropDefaultTheta(x) - - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - x_val, y_val, w = self.evaluate([x, y, l.vars]) - - out = [] - act = x_val - # relu(act \dot w + b) - out += [np.maximum(0, np.matmul(act, w.foo.w) + w.foo.b)] - self.assertEqual(out[-1].shape, (2, 4, 4)) - out += [np.maximum(0, np.matmul(act, w.bar.w) + w.bar.b)] - self.assertEqual(out[-1].shape, (2, 4, 4)) - out += [np.maximum(0, np.matmul(act, w.baz.w) + w.baz.b)] - self.assertEqual(out[-1].shape, (2, 4, 4)) - - np_result = out[0] - for v in out[1:]: - np_result = np.matmul(np_result, v) - self.assertAllClose(np_result, y_val, atol=1e-5, rtol=1e-5) - - def testParalellMultiOutputsLayer(self): - g = tf.Graph() - with g.as_default(): - tf.random.set_seed(24332) - - def Merge(xs): - rets = [] - for x in zip(*xs): - if x[0] is None: - rets.append(None) - else: - rets.append(tf.add_n(list(x))) - return tuple(rets) - - p = layers.ParallelLayer.Params().Set( - name='parallel', - merge=Merge, - sub=[ - lingvo_layers.ConvLayer.Params().Set( - name='p%d' % i, - filter_shape=(3, 3, 3, 5), - filter_stride=(1, 1), - batch_norm=False) for i in range(3) - ]) - l = p.Instantiate() - x = tf.zeros(shape=[2, 32, 32, 3]) - y0, y1 = l.FPropDefaultTheta(x) - y_sum = tf.reduce_sum(y0) - # Ensures the 2nd return value (None) are handled properly. - self.assertEqual(None, y1) - - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - y_sum_val = self.evaluate(y_sum) - - self.assertEqual(y_sum_val, 0.) - - def testMapLayer(self): - g = tf.Graph() - with g.as_default(): - tf.random.set_seed(24332) - p = layers.MapLayer.Params().Set( - name='map', fn=tf.reduce_max, kwargs={'axis': 1}) - l = p.Instantiate() - x0, x1 = [tf.random.normal(shape=[2, 3, 5])] * 2 - y0, y1 = l.FPropDefaultTheta(x0, x1) - - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - vx0, vx1, vy0, vy1 = self.evaluate([x0, x1, y0, y1]) - - self.assertAllClose(np.max(vx0, 1), vy0) - self.assertAllClose(np.max(vx1, 1), vy1) - - def testLinearLayer(self): - g = tf.Graph() - with g.as_default(): - tf.random.set_seed(24332) - p = layers.LinearLayer.Params().Set( - name='test', input_dims=10, output_dims=5) - l = p.Instantiate() - xs = [] - ys = [] - for shape in ([2, 10], [2, 3, 10], [2, 3, 5, 10], [2, 3, 5, 7, 10]): - x = tf.random.normal(shape=shape) - y = l.FPropDefaultTheta(x) - xs += [x] - ys += [y] - - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - xs_val, ys_val, w_val = self.evaluate([xs, ys, l.vars]) - - self.assertEqual(w_val.w.shape, (10, 5)) - for (xv, yv) in zip(xs_val, ys_val): - self.assertAllClose(np.matmul(xv, w_val.w), yv) - - def testBiasLayer(self): - g = tf.Graph() - with g.as_default(): - tf.random.set_seed(24332) - p = layers.BiasLayer.Params().Set(name='test', dims=10) - l = p.Instantiate() - x = tf.random.normal(shape=[2, 10]) - y = l.FPropDefaultTheta(x) - - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - x_val, y_val, w_val = self.evaluate([x, y, l.vars]) - - self.assertEqual(w_val.b.shape, (10,)) - self.assertAllClose(x_val + w_val.b, y_val) - - def testGraphTensors(self): - graph_tensors = layers.GraphTensors() - graph_tensors.StoreTensor('t', - py_utils.NestedMap(a=py_utils.NestedMap(b='c'))) - self.assertEqual('c', graph_tensors.GetTensor('t.a.b')) - - def testSignatureParsing(self): - sig = layers.GraphSignature('a,b->c') - self.assertEqual(['a', 'b'], sig.inputs) - self.assertEqual(['c'], sig.outputs) - - sig = layers.GraphSignature('[a,b],d->c') - self.assertEqual([['a', 'b'], 'd'], sig.inputs) - self.assertEqual(['c'], sig.outputs) - - # also test nested structures, like nested lists and dicts. - sig = layers.GraphSignature('(x=a,y=b)->c') - self.assertEqual([{'x': 'a', 'y': 'b'}], sig.inputs) - self.assertEqual(['c'], sig.outputs) - - # Make sure that empty lists and dicts work. - sig = layers.GraphSignature('(x=[]),()->d') - self.assertEqual([{'x': []}, {}], sig.inputs) - - sig = layers.GraphSignature('(x=a,y=[f,(z=g.h)]),[d,e]->j') - self.assertEqual([{ - 'x': 'a', - 'y': ['f', { - 'z': 'g.h' - }] - }, ['d', 'e']], sig.inputs) - self.assertEqual(['j'], sig.outputs) - - def testGraphLayer(self): - g = tf.Graph() - with g.as_default(), self.SetEval(True): - tf.random.set_seed(24332) - - def _FnMeta(*shapes): - return py_utils.NestedMap(flops=1, out_shapes=shapes) - - p = layers.GraphLayer.Params().Set( - name='graph', - input_endpoints=['x'], - output_endpoints=['y'], - sub=[ - ('x.a->y.c', - layers.FnLayer.Params().Set(fn=lambda x: 2 * x, - fn_meta=_FnMeta)), - ('x.b->y.d', layers.FnLayer.Params().Set( - name='bar', fn=lambda x: x + 2, fn_meta=_FnMeta)), - ('y.c,y.d->y.e, y.f', layers.FnLayer.Params().Set( - name='baz', fn=lambda x, y: (x + y, x - y), fn_meta=_FnMeta)), - ]) - l = p.Instantiate() - x = py_utils.NestedMap(a=tf.constant(1.0), b=tf.constant(2.0)) - y = l.FProp(l.theta, x) - y_shape = l.FPropMeta( - p, py_utils.Transform(lambda t: tshape.Shape(t.shape), - x)).out_shapes[0] - self.assertDictEqual( - py_utils.Transform(lambda t: t.shape.as_list(), y), - py_utils.Transform(lambda t: t.ToTensorShape().as_list(), y_shape)) - - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - y_val = self.evaluate(y) - print(y_val) - self.assertEqual(py_utils.NestedMap(c=2.0, d=4.0, e=6.0, f=-2.0), y_val) - - def testSoftCondLayer(self): - num_experts = 100 - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(24332) - p = layers.SoftCondLayer.Params().Set( - name='soft_cond', - cond_dim=2, - num_experts=num_experts, - body=lingvo_layers.FCLayer.Params().Set(input_dim=2, output_dim=2)) - l = p.Instantiate() - x = tf.random.normal(shape=[1, 2, 2]) - y = l.FPropDefaultTheta(x) - self.evaluate(tf.global_variables_initializer()) - x_val, y_val, vars_val = self.evaluate([x, y, l.vars]) - - p_nz = layers.SoftCondLayer.Params().Set( - name='soft_cond_nonzeros', - cond_dim=2, - num_experts=num_experts, - nonzeros_mean=True, - body=lingvo_layers.FCLayer.Params().Set(input_dim=2, output_dim=2)) - l_nz = p_nz.Instantiate() - x_nz = tf.random.normal(shape=[1, 2, 2]) - y_nz = l_nz.FPropDefaultTheta(x_nz) - self.evaluate(tf.global_variables_initializer()) - x_nz_val, y_nz_val, vars_nz_val = self.evaluate([x_nz, y_nz, l_nz.vars]) - - np_val = x_val[0] - np_nz_val = x_nz_val[0] - taks_weight = np.exp(-1.0 * np.dot(np.mean(np_val, 0), vars_val.w)) - taks_weight = 1.0 / (1.0 + taks_weight) - nzs = np.count_nonzero(np_nz_val, 0).astype('float32') + 1e-10 - taks_weight_nz = np.exp(-1.0 * - np.dot(np.sum(np_nz_val, 0) / nzs, vars_nz_val.w)) - taks_weight_nz = 1.0 / (1.0 + taks_weight_nz) - weighted_weight = np.einsum('i,ijk->jk', taks_weight, vars_val.body.w) - weighted_weight_nz = np.einsum('i,ijk->jk', taks_weight_nz, - vars_nz_val.body.w) - weighted_bias = np.einsum('i,ij->j', taks_weight, vars_val.body.b) - weighted_bias_nz = np.einsum('i,ij->j', taks_weight_nz, vars_nz_val.body.b) - np_val_out = np.maximum(0, np.dot(np_val, weighted_weight) + weighted_bias) - np_val_out_nz = np.maximum( - 0, - np.dot(np_nz_val, weighted_weight_nz) + weighted_bias_nz) - self.assertAllClose(np_val_out, y_val[0]) - self.assertAllClose(np_val_out_nz, y_nz_val[0]) - - def testRepeatLayer(self): - repeat = 100 - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(24332) - p = layers.RepeatLayer.Params().Set( - name='recurrent', - repeat=repeat, - body=lingvo_layers.FCLayer.Params().Set(input_dim=2, output_dim=2)) - l = p.Instantiate() - x = tf.random.normal(shape=[2, 2]) - y = l.FPropDefaultTheta(x) - self.evaluate(tf.global_variables_initializer()) - x_val, y_val, w = self.evaluate([x, y, l.vars]) - - np_val = x_val - - # relu(act \dot w + b) - for i in range(repeat): - np_val = np.maximum(0, np.dot(np_val, w.body.w[i]) + w.body.b[i]) - self.assertAllClose(np_val, y_val) - - def testParallelRepeatLayerLayer(self): - repeat = 100 - body_p = layers.SequentialLayer.Params().Set( - name='body', - sub=[ - layers.LinearLayer.Params().Set( - name='ln1', input_dims=2, output_dims=4), - layers.FnLayer.Params().Set( - name='relu', - fn=tf.nn.relu, - fn_meta=lambda x: py_utils.NestedMap(flops=1, out_shapes=(x,))), - layers.LinearLayer.Params().Set( - name='ln2', input_dims=4, output_dims=2) - ]) - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(24332) - p = layers.ParallelRepeatLayer.Params().Set( - name='moe', repeat=repeat, body=body_p) - l = p.Instantiate() - x = tf.random.normal(shape=[repeat, 2, 2]) - y = l.FPropDefaultTheta(x) - self.evaluate(tf.global_variables_initializer()) - x_val, y_val, w = self.evaluate([x, y, l.vars]) - - np_val = [] - - for i in range(repeat): - # relu(act \dot w_1) \dot w_2 - np_val.append( - np.dot( - np.maximum(0, np.dot(x_val[i], w.body.ln1.w[i])), - w.body.ln2.w[i])) - np_val = np.stack(np_val) - self.assertAllClose(np_val, y_val) - - def testRematerializationLayer(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(24332) - - def MulSumFnMeta(x): - return py_utils.NestedMap(flops=2, out_shapes=(x,)) - - def AddFnMeta(x, y): - del y - return py_utils.NestedMap(flops=2, out_shapes=(x,)) - - p = layers.GraphLayer.Params().Set( - name='graph', - input_endpoints=['a', 'b'], - output_endpoints=['e'], - sub=[ - ('a->c', layers.FnLayer.Params().Set( - fn=lambda x: 2 * x, fn_meta=MulSumFnMeta)), - ('b->d', layers.FnLayer.Params().Set( - name='bar', fn=lambda x: x + 2, fn_meta=MulSumFnMeta)), - ('c,d->e', layers.FnLayer.Params().Set( - name='baz', fn=lambda x, y: x + y, fn_meta=AddFnMeta)), - ]) - p = layers.RematerializationLayer.Params().Set(name='remat', body=p) - l = p.Instantiate() - x = tf.constant(1.0) - y = tf.constant(2.0) - z = l.FProp(l.theta, x, y) - self.evaluate(tf.global_variables_initializer()) - z_val = self.evaluate(z) - print(z_val) - self.assertAllClose(6.0, z_val) - - def testPrintShapeLayer(self): - g = tf.Graph() - with g.as_default(): - p = layers.PrintShapeLayer.Params().Set(name='test') - l = p.Instantiate() - x = tf.constant(1.0) - y = l.FPropDefaultTheta(x) - - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - x_val, y_val = self.evaluate([x, y]) - self.assertEqual(x_val, y_val) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/builder_test.py b/lingvo/core/builder_test.py deleted file mode 100644 index 9dbeec32..00000000 --- a/lingvo/core/builder_test.py +++ /dev/null @@ -1,336 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for the Builder pattern.""" - -import inspect -from lingvo import compat as tf -from lingvo.core import builder -from lingvo.core import cluster_factory -from lingvo.core import layers -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.core import tshape -import numpy as np -import sympy - - -class BuilderTest(test_utils.TestCase): - - def _Expect(self, expected_cost, p, *inputs): - meta = p.cls.FPropMeta(p, *(tshape.Shape(s) for s in inputs)) - self.assertEqual(meta.flops, expected_cost) - - g = tf.Graph() - with g.as_default(): - l = p.Instantiate() - xs = [tf.random.normal(shape=s) for s in inputs] - ys = l.FPropDefaultTheta(*xs) - - with self.session(graph=g) as sess: - sess.run(tf.global_variables_initializer()) - _ = sess.run(ys) - - def testCostEstimate(self): - b = builder.Base.Params() - b = b.Instantiate() - # pyformat: disable - self._Expect(2400, b._Linear('l', 10, 20), [2, 3, 10]) - self._Expect(60, b._Bias('b', 10), [2, 3, 10]) - self._Expect(150, b._Activation('a'), [3, 5, 10]) - self._Expect(600, b._BN('b', 10), [2, 3, 10]) - self._Expect( - (2**21) * 9 * 2, - b._Conv2D('c', (3, 3, 8, 16), (1, 1)), - [16, 32, 32, 8]) - self._Expect( - (2**19) * 9 * 2, - b._Conv2D('c', (3, 3, 8, 16), (2, 2)), - [16, 32, 32, 8]) - self._Expect( - (2**20) * 9 * 2, - b._Conv2D('c', (3, 3, 8, 16), (1, 2)), - [16, 32, 32, 8]) - self._Expect( - 24 * 10 * 10 * 2 * 7, - b._Rep( - 'rep', - 7, - b._Linear('l', 10, 10)), - [8, 3, 10]) - self._Expect( - (2**21) * 9 * 2 + (2**14) * 16 * 32 * 2 + (2**14) * 32, - b._Seq( - 'seq', - b._Conv2D('c', (3, 3, 8, 16), (1, 1)), - b._Linear('l', 16, 32), - b._Activation('a')), - [16, 32, 32, 8]) - self._Expect( - (((2**19) * 9 * 2 + (2**12) * 16 * 32 * 2 + (2**12) * 32) * 2), - b._Par( - 'p', - b._Seq( - 'b0', - b._Conv2D('c', (3, 3, 8, 16), (2, 2)), - b._Linear('l', 16, 32), - b._Activation('a')), - b._Seq( - 'b1', - b._Conv2D('c', (3, 3, 8, 16), (2, 2)), - b._Linear('l', 16, 32), - b._Activation('a'))), - [16, 32, 32, 8]) - # pyformat: enable - - def testFetch(self): - # Construct a layer w/ two parallel branches. We want to demonstrate that - # we can fetch intermediate values from two branches independently. - # pyformat: disable - b = builder.Base.Params() - b = b.Instantiate() - p = b._Par( - 'p', - b._Seq( - 'b0', - b._Conv2D('c', (3, 3, 8, 16), (2, 2)), - b._Linear('l', 16, 32), - b._Save('fetch'), - b._Activation('a')), - b._Seq( - 'b1', - b._Conv2D('c', (3, 3, 8, 16), (2, 2)), - b._Linear('l', 16, 32), - b._Activation('a'), - b._Save('fetch'))) - p = b._AddFetches('mh', p, ['b0.fetch', 'b1.fetch']) - # pyformat: enable - g = tf.Graph() - with g.as_default(): - l = p.Instantiate() - x = tf.random.normal(shape=[17, 64, 64, 8]) - outs = l.FPropDefaultTheta(x) - x, y, b0, b1 = outs - - with self.session(graph=g) as sess: - sess.run(tf.global_variables_initializer()) - v0, v1, u, v = sess.run([b0, b1, x, y]) - - self.assertAllClose(np.maximum(0, v0), u) - self.assertAllClose(v1, v) - - def testFetchGrad(self): - # Tests we can fetch backprop gradients. - # pyformat: disable - b = builder.Base.Params() - b = b.Instantiate() - p = b._Seq( - 'seq', - b._Linear('l', 16, 32), - b._Bias('b', 32), - b._Save('fetch'), - b._Activation('a')) - # pyformat: enable - - g = tf.Graph() - with g.as_default(): - l = p.Instantiate() - x = tf.random.normal(shape=[4, 16]) - y = l.FPropDefaultTheta(x) - loss = tf.reduce_sum(tf.square(y)) - _ = tf.gradients(ys=loss, xs=x) - - act, dact = l.fetch.activation, l.fetch.gradient - - with self.session(graph=g) as sess: - sess.run(tf.global_variables_initializer()) - act_v, dact_v = sess.run([act, dact]) - - # The last layer two layers is sum(square(relu(act))). - # So the dact is simply 2*relu(act). - self.assertAllClose(2 * np.maximum(0, act_v), dact_v) - - def testBatchParallel(self): - # pyformat: disable - b = builder.Base.Params() - b = b.Instantiate() - p = b._BatchParallel( - 'bp', - b._Seq( - 'main', - b._Linear('l', 8, 4), - b._PrintShape('debug'), - b._Bias('b', 4))) - # pyformat: enable - g = tf.Graph() - with g.as_default(): - l = p.Instantiate() - x = tf.random.normal(shape=[4, 8]) - with cluster_factory.ForTestingWorker(cpus=4, split_size=1): - y1 = l.FPropDefaultTheta(x) - with cluster_factory.ForTestingWorker(cpus=4, split_size=2): - y2 = l.FPropDefaultTheta(x) - with cluster_factory.ForTestingWorker(cpus=4, split_size=4): - y4 = l.FPropDefaultTheta(x) - - cfg = tf.config_pb2.ConfigProto() - cfg.device_count['CPU'] = 4 - with self.session(config=cfg, graph=g) as sess: - sess.run(tf.global_variables_initializer()) - v1, v2, v4 = sess.run([y1, y2, y4]) - - self.assertAllClose(v1, v2) - self.assertAllClose(v1, v4) - - def testFn(self): - b = builder.Base.Params() - b = b.Instantiate() - - p = b._Fn('fn', lambda x, y: x + y, fn_out=lambda x, y: x) - - meta = p.cls.FPropMeta(p, tshape.Shape([4, 6]), tshape.Shape([4, 6])) - self.assertEqual(meta.flops, 48) - self.assertEqual(meta.out_shapes[0].ToTensorShape().as_list(), [4, 6]) - - g = tf.Graph() - with g.as_default(): - l = p.Instantiate() - x = tf.random.normal(shape=[4, 8]) - y = tf.random.normal(shape=[4, 1]) - z = l.FPropDefaultTheta(x, y) - - with self.session(graph=g) as sess: - sess.run(tf.global_variables_initializer()) - v = sess.run([x, y, z]) - - self.assertAllClose(v[0] + v[1], v[2]) - - def testRematerialize(self): - # Test the dropout consistency between fprop and bprop. - b = builder.Base.Params() - b = b.Instantiate() - start_block = layers.DeterministicDropoutLayer.Params().Set( - name='start_dropout', keep_prob=0.7) - # Build 4 dropout layers, each wrapped by RematerializeFn. - num_blocks = 4 - blocks = [] - blocks_per_cell = 2 - for i in range(num_blocks): - blocks.append(layers.DeterministicDropoutLayer.Params().Set( - name='dropout_{}'.format(i), keep_prob=0.7)) - cells = [] - while blocks: - heads, blocks = blocks[:blocks_per_cell], blocks[blocks_per_cell:] - cell_name = 'cell_{}'.format(len(cells)) - cells.append( - b._Rematerialize(name=cell_name, body=b._Seq(cell_name, *heads))) - with self.session(use_gpu=False, graph=tf.Graph()) as sess: - tf.random.set_seed(12345) - p = b._Seq('test', start_block, *cells) - mdl = p.Instantiate() - # y = mdl.Frop(x * w) - # Fake input - x = tf.ones([4, 5]) - # Construct weights. - w = tf.get_variable( - 'w', shape=[4, 5], initializer=tf.constant_initializer([[1] * 5] * 4)) - y = mdl.FPropDefaultTheta(x * w) - # Construct loss function such that gradients = final activation. - # dy/dw = y = mdl.Frop(x * w) when w is 1. - loss = tf.reduce_sum(y) - grads = py_utils.ComputeGradients(loss, py_utils.NestedMap(w=w)) - tf.global_variables_initializer().run() - y_val, grads_val = sess.run([y, grads.Transform(tuple)]) - grads_val = grads_val['w'][1] - self.assertAllClose(y_val, grads_val) - self.assertEqual(py_utils.GetStepSeed().eval(), 1553244033) - - def testFnDefaultMeta(self): - b = builder.Base.Params() - b = b.Instantiate() - - def Foo(x, y): - return x * x, y * 2 - - p = b._Fn('fn', Foo) - - meta = p.cls.FPropMeta(p, tshape.Shape([4, 6]), tshape.Shape([3, 3])) - self.assertEqual(meta.flops, 33) - self.assertEqual(meta.out_shapes[0].ToTensorShape().as_list(), [4, 6]) - self.assertEqual(meta.out_shapes[1].ToTensorShape().as_list(), [3, 3]) - - g = tf.Graph() - with g.as_default(): - l = p.Instantiate() - x = tf.random.normal(shape=[4, 8]) - y = tf.random.normal(shape=[3, 3]) - z0, z1 = l.FPropDefaultTheta(x, y) - - with self.session(graph=g) as sess: - sess.run(tf.global_variables_initializer()) - vx, vy, vz0, vz1 = sess.run([x, y, z0, z1]) - - self.assertAllClose(vx * vx, vz0) - self.assertAllClose(vy * 2, vz1) - - def testSymbolicDims(self): - p = builder.Base.Params() - b = p.Instantiate() - - f1 = tshape.Shape(['kh', 'kw', 'idims', 'odims']) - kh, kw, idims, odims = f1 - f2 = tshape.Shape([kh, kw, odims, odims]) - p = b._Seq('test', b._Conv2D('conv', f1, (2, 2)), - b._Conv2D('conv', f2, (2, 2)), b._Bias('bias', odims)) - - inp = tshape.Shape(['b', 'h', 'w', idims]) - b, h, w, _ = inp - meta = p.cls.FPropMeta(p, inp) - print('flops = ', meta.flops) - out = meta.out_shapes[0] - print('outputs = ', out) - - # sympy.lambdify can help us to do faster numerical evaluation. - # Might be useful to build a "cost" model given a builder layer. - f = sympy.lambdify([b, h, w, kh, kw, idims, odims], meta.flops, 'numpy') - print('f.source = ', inspect.getsource(f)) - self.assertEqual(f(8, 224, 224, 3, 3, 8, 32), 925646848) - self.assertEqual(f(8, 224, 224, 5, 5, 8, 32), 2569814016) - - def testDoEval(self): - p = builder.Base.Params().Instantiate()._Dropout('dropout', 0.5) - - g = tf.Graph() - with g.as_default(): - l = p.Instantiate() - x = tf.random.normal(shape=[16, 16]) - # FProp three times each with different do_eval mode. - with cluster_factory.SetEval(mode=None): - a = l.FPropDefaultTheta(x) - with cluster_factory.SetEval(mode=False): - b = l.FPropDefaultTheta(x) - with cluster_factory.SetEval(mode=True): - c = l.FPropDefaultTheta(x) - - with self.session(graph=g) as sess: - x, a, b, c = sess.run([x, a, b, c]) - - self.assertGreater(np.linalg.norm(x - a), 0) - self.assertGreater(np.linalg.norm(x - b), 0) - self.assertAllEqual(x, c) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/checkpointer.py b/lingvo/core/checkpointer.py deleted file mode 100644 index 6d904ea5..00000000 --- a/lingvo/core/checkpointer.py +++ /dev/null @@ -1,264 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Checkpointing utilities for save/restore.""" - -import os -import time - -import lingvo.compat as tf -from lingvo.core import cluster_factory -from lingvo.core import py_utils -import six - - -class Checkpointer: - """Checkpointing utility class. - - Needs to be created within a graph context. - """ - - def __init__(self, - train_dir, - model, - init_op=None, - train_params=None, - save_only=False): - """Initialize Checkpointer. - - Args: - train_dir: Training directory for saving checkpoints. - model: A BaseModel instance or None. - init_op: The initialize variables op. If unset, it will call - tf.global_variables_initializer(). - train_params: If specified, use these training params instead of those in - the `model`. - save_only: This checkpointer is only intended for saving checkpoints. - """ - self._train_dir = train_dir - self._save_only = save_only - if init_op: - self._init_op = init_op - else: - self._init_op = tf.global_variables_initializer() - - self._save_path = os.path.join(self._train_dir, 'ckpt') - - if train_params: - self._train_params = train_params - self._model = None - else: - assert model - self._train_params = model.params.train - self._model = model - - if not self._save_only: - self._params = model.params - self._model_tasks = model.tasks - self._model = model - - self._next_checkpoint_seconds = 0 - self._save_interval_seconds = self._train_params.save_interval_seconds - self._saver = self._GetSaver() - - self._uninitialized_vars = tf.report_uninitialized_variables( - tf.global_variables()) - - def _GetSaver(self): - """Returns a saver.""" - do_eval = cluster_factory.Current().do_eval - if not self._save_only and self._model.ema and do_eval: - tf.logging.info('Using EMA for evaluation.') - return tf.train.Saver( - self._model.ema.variables_to_restore(self._model.variables_for_ema)) - return tf.train.Saver( - sharded=True, - max_to_keep=self._train_params.save_max_to_keep, - keep_checkpoint_every_n_hours=( - self._train_params.save_keep_checkpoint_every_n_hours), - pad_step_number=True, # %08d - write_version=tf.train.SaverDef.V2) - - def RestoreFromPath(self, sess, checkpoint_path): - """Load the checkpoint from specified path.""" - assert not self._save_only - tf.logging.info('Load from checkpoint %s.', checkpoint_path) - self._saver.restore(sess, checkpoint_path) - tf.logging.info('Load checkpoint done.') - # Successfully restored from checkpoint. - uninitialized_var_names = self._GetUninitializedVarNames(sess) - assert not uninitialized_var_names, uninitialized_var_names - - def MaybeSave(self, sess, gsteps): - """If it's time to save, save the checkpoint. - - Args: - sess: tf.Session. - gsteps: Current global step. - Returns: - Whether a checkpoint was saved. - """ - now = time.time() - if now >= self._next_checkpoint_seconds: - self.Save(sess, gsteps) - self._next_checkpoint_seconds = now + self._save_interval_seconds - return True - return False - - def Save(self, sess, gsteps): - """Save the checkpoint. - - Args: - sess: tf.Session. - gsteps: Current global step. - """ - tf.logging.info('Save checkpoint') - path = self._saver.save(sess, self._save_path, gsteps) - tf.logging.info('Save checkpoint done: %s', path) - - def _RestoreFromLatestCheckpoint(self, sess): - assert not self._save_only - path = tf.train.latest_checkpoint(self._train_dir) - if path: - self.RestoreFromPath(sess, path) - return True - return False - - def _GetUninitializedVarNames(self, sess): - uninitialized_var_names = sorted(list(sess.run(self._uninitialized_vars))) - # uninitialized_var_names is a list of strings without ":0" suffix. - # tf.report_uninitialized_variables returns binary strings. - assert all(isinstance(s, bytes) for s in uninitialized_var_names) - return uninitialized_var_names - - def Restore(self, sess, force_reinitialize=False): - """Restore from latest checkpoint if available, or initialize.""" - # Try and restore from the latest checkpoint. - if self._RestoreFromLatestCheckpoint(sess): - # Successfully restored from checkpoint. - uninitialized_var_names = self._GetUninitializedVarNames(sess) - assert not uninitialized_var_names, uninitialized_var_names - return - - # Otherwise we need to initialize. - uninitialized_var_names = self._GetUninitializedVarNames(sess) - tf.logging.info('Uninitialized var list: %s', uninitialized_var_names) - if not force_reinitialize: - # There should only be uninitialized variables if all variables are - # uninitialized - with the exception of global_step due to - # RestoreGlobalStepIfNeeded in the _LoopEnqueue of TrainerTpu. - all_var_names = [ - six.ensure_binary(v.name[:-2]) for v in tf.global_variables() - ] - already_initialized_vars = ( - set(all_var_names) - set(uninitialized_var_names)) - already_initialized_vars.discard(b'global_step') - assert not already_initialized_vars, ('Already initialized vars: %s' % - sorted(already_initialized_vars)) - - # At this point all variables are uninitialized, so it is safe to run a - # global initializer. - sess.run(self._init_op) - tf.logging.info('Initialized all vars.') - - # TODO(b/160786085): Move this logic into Overriding vars logic itself, - # which requires refactoring things out of py_utils to avoid circular deps. - def _ResolveCkptPath(ckpt_rules): - return {GetSpecificCheckpoint(k): v for k, v in ckpt_rules.items()} - - # Restore specific variables based on init_from_checkpoint_rules. - for task in self._model.tasks: - tp = task.params.train - if tp.init_from_checkpoint_rules: - rules = _ResolveCkptPath(tp.init_from_checkpoint_rules) - tf.logging.info('OverrideVarsFromCheckpoints %s', rules) - py_utils.OverrideVarsFromCheckpoints(sess, tf.global_variables(), rules) - - if self._params.train.init_from_checkpoint_rules: - tp = self._params.train - rules = _ResolveCkptPath(tp.init_from_checkpoint_rules) - tf.logging.info('OverrideVarsFromCheckpoints %s', rules) - py_utils.OverrideVarsFromCheckpoints(sess, tf.global_variables(), rules) - - def RestoreIfNeeded(self, sess): - """If vars are not initialized, restore from checkpoint.""" - assert not self._save_only - uninitialized_var_names = self._GetUninitializedVarNames(sess) - if not uninitialized_var_names: - # All variables are already initialized. - return - - self.Restore(sess) - - def RestoreGlobalStepIfNeeded(self, sess): - """If global step is not initialized, load it from the checkpoint. - - Args: - sess: tf.Session. - """ - assert not self._save_only - uninitialized_vars = self._GetUninitializedVarNames(sess) - if six.ensure_binary('global_step') not in uninitialized_vars: - return - - with sess.graph.as_default(): - gstep = py_utils.GetGlobalStep() - - path = tf.train.latest_checkpoint(self._train_dir) - if path: - reader = tf.train.NewCheckpointReader(path) - value = reader.get_tensor('global_step') - tf.logging.info('Restoring global step: %s', value) - sess.run(gstep.assign(value)) - else: - tf.logging.info('Initializing global step') - sess.run(gstep.initializer) - - -def GetSpecificCheckpoint(load_checkpoint_from): - """Returns a specific checkpoint given `load_checkpoint_from`. - - When load_checkpoint_from is a directory, we find the latest - checkpoint in the directory and use that as the checkpoint - to evaluate. - - When load_checkpoint_from is a specific checkpoint, we - validate the path and return it. - - Args: - load_checkpoint_from: If not None, specifies the directory or specific - checkpoint to load. If a directory, the latest checkpoint in the - directory will be used. - """ - if not load_checkpoint_from: - return None - - # If load_checkpoint_from is a directory, return the latest - # checkpoint in the directory. - if tf.io.gfile.isdir(load_checkpoint_from): - return tf.train.latest_checkpoint(load_checkpoint_from) - - # We assume that load_checkpoint_from is a specific checkpoint to - # evaluate since it is not a directory. - # - # Check validity of eval path by looking for the index file. - if tf.io.gfile.exists(load_checkpoint_from + '.index'): - return load_checkpoint_from - - # Fail if we see an unexpected load_checkpoint_from. - # - # This might happen if load_checkpoint_from refers to a checkpoint - # but the index file cannot be found. - raise ValueError('Invalid load_checkpoint_from: %s' % load_checkpoint_from) diff --git a/lingvo/core/checkpointer_test.py b/lingvo/core/checkpointer_test.py deleted file mode 100644 index 11d65f5a..00000000 --- a/lingvo/core/checkpointer_test.py +++ /dev/null @@ -1,249 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for checkpointer.""" - -import os -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import base_model -from lingvo.core import checkpointer -from lingvo.core import py_utils -from lingvo.core import test_utils - - -class LinearModel(base_model.BaseTask): - """A basic linear model.""" - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'linear_model' - return p - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - w = py_utils.WeightParams( - shape=[3], - init=py_utils.WeightInit.Gaussian(scale=1.0, seed=123456), - dtype=p.dtype) - b = py_utils.WeightParams( - shape=[], - init=py_utils.WeightInit.Gaussian(scale=1.0, seed=234567), - dtype=p.dtype) - self.CreateVariable('w', w) - self.CreateVariable('b', b) - - -class CheckpointerTest(test_utils.TestCase): - - def testSaveRestore(self): - train_dir = os.path.join(self.get_temp_dir(), 'testSaveRestore') - os.mkdir(train_dir) - p = base_model.SingleTaskModel.Params(LinearModel.Params()) - p.input = base_input_generator.BaseInputGenerator.Params() - - final_global_step = 10 - expected_w = [0.38615, 2.975221, -0.852826] - initial_b = 1.418741 - final_b = 1234 - - with self.session(graph=tf.Graph()) as sess: - model = p.Instantiate() - self.evaluate(tf.global_variables_initializer()) - w, b = self.evaluate([model.GetTask().vars.w, model.GetTask().vars.b]) - self.assertAllClose(expected_w, w) - self.assertAlmostEqual(initial_b, b, places=5) - - saver = checkpointer.Checkpointer(train_dir, model) - self.evaluate( - tf.assign(py_utils.GetOrCreateGlobalStepVar(), final_global_step)) - self.evaluate(tf.assign(model.GetTask().vars.b, final_b)) - saver.Save(sess, model.global_step) - - w, b = self.evaluate([model.GetTask().vars.w, model.GetTask().vars.b]) - self.assertAllClose(expected_w, w) - self.assertEqual(final_b, b) - - self.assertTrue( - os.path.isfile( - os.path.join(train_dir, 'ckpt-%08d.index' % final_global_step))) - - with self.session(graph=tf.Graph()) as sess: - model = p.Instantiate() - saver = checkpointer.Checkpointer(train_dir, model) - saver.RestoreIfNeeded(sess) - - w, b, global_step = self.evaluate( - [model.GetTask().vars.w, - model.GetTask().vars.b, model.global_step]) - self.assertAllClose(expected_w, w) - self.assertEqual(final_b, b) - self.assertEqual(final_global_step, global_step) - - # Restore from checkpoint will always work, even though vars are already - # initialized. - saver.Restore(sess) - - def testRestoreWithGlobalStepAlreadyInitialized(self): - train_dir = os.path.join(self.get_temp_dir(), - 'testRestoreWithGlobalStepAlreadyInitialized') - os.mkdir(train_dir) - p = base_model.SingleTaskModel.Params(LinearModel.Params()) - p.input = base_input_generator.BaseInputGenerator.Params() - - with self.session(graph=tf.Graph()) as sess: - global_step = tf.compat.v1.train.get_or_create_global_step() - self.evaluate(tf.global_variables_initializer()) - - model = p.Instantiate() - saver = checkpointer.Checkpointer(train_dir, model) - - with self.assertRaises(tf.errors.FailedPreconditionError): - self.evaluate([model.GetTask().vars.w, model.GetTask().vars.b]) - - saver.RestoreIfNeeded(sess) - w, b, global_step = self.evaluate( - [model.GetTask().vars.w, - model.GetTask().vars.b, model.global_step]) - self.assertAllClose([0.38615, 2.975221, -0.852826], w) - self.assertAlmostEqual(1.418741, b, places=5) - self.assertEqual(0, global_step) - - self.assertFalse( - os.path.isfile(os.path.join(train_dir, 'ckpt-00000000.index'))) - - def testRestore(self): - train_dir = os.path.join(self.get_temp_dir(), 'testRestore') - os.mkdir(train_dir) - p = base_model.SingleTaskModel.Params(LinearModel.Params()) - p.input = base_input_generator.BaseInputGenerator.Params() - - with self.session(graph=tf.Graph()) as sess: - model = p.Instantiate() - saver = checkpointer.Checkpointer(train_dir, model) - - with self.assertRaises(tf.errors.FailedPreconditionError): - self.evaluate([model.GetTask().vars.w, model.GetTask().vars.b]) - - saver.RestoreIfNeeded(sess) - w, b, global_step = self.evaluate( - [model.GetTask().vars.w, - model.GetTask().vars.b, model.global_step]) - self.assertAllClose([0.38615, 2.975221, -0.852826], w) - self.assertAlmostEqual(1.418741, b, places=5) - self.assertEqual(0, global_step) - - with self.assertRaises(AssertionError): - # When initializing from scratch, variables are expected to not already - # be initialized. - saver.Restore(sess) - - # Unless force_reinitialize is used. - saver.Restore(sess, force_reinitialize=True) - - def testRestoreWithoutCheckpointInitializesVars(self): - train_dir = os.path.join(self.get_temp_dir(), - 'testRestoreWithoutCheckpointInitializesVars') - os.mkdir(train_dir) - p = base_model.SingleTaskModel.Params(LinearModel.Params()) - p.input = base_input_generator.BaseInputGenerator.Params() - - with self.session(graph=tf.Graph()) as sess: - model = p.Instantiate() - saver = checkpointer.Checkpointer(train_dir, model) - - with self.assertRaises(tf.errors.FailedPreconditionError): - self.evaluate([model.GetTask().vars.w, model.GetTask().vars.b]) - - saver.RestoreIfNeeded(sess) - w, b, global_step = self.evaluate( - [model.GetTask().vars.w, - model.GetTask().vars.b, model.global_step]) - self.assertAllClose([0.38615, 2.975221, -0.852826], w) - self.assertAlmostEqual(1.418741, b, places=5) - self.assertEqual(0, global_step) - - self.assertFalse( - os.path.isfile(os.path.join(train_dir, 'ckpt-00000000.index'))) - - def testInitRulesDirectory(self): - train_dir = os.path.join(self.get_temp_dir(), 'testInitRulesDirectory') - os.mkdir(train_dir) - p = base_model.SingleTaskModel.Params(LinearModel.Params()) - p.input = base_input_generator.BaseInputGenerator.Params() - b1 = 1234 - g1 = 10 - b2 = 12345 - g2 = 100 - - with self.session(graph=tf.Graph()) as sess: - model = p.Instantiate() - self.evaluate(tf.global_variables_initializer()) - saver = checkpointer.Checkpointer(train_dir, model) - self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), g1)) - self.evaluate(tf.assign(model.GetTask().vars.b, b1)) - saver.Save(sess, model.global_step) - self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), g2)) - self.evaluate(tf.assign(model.GetTask().vars.b, b2)) - saver.Save(sess, model.global_step) - - train_dir_2 = os.path.join(self.get_temp_dir(), 'testInitRulesDirectory_2') - - # Set init_checkpoint_rules to only restore b from a specific ckpt - # the first one, not the latest one. - rules = [('(.*)', '%s')] - spec_dir = os.path.join(train_dir, 'ckpt-00000010') - p.train.init_from_checkpoint_rules = {spec_dir: (rules, [])} - with self.session(graph=tf.Graph()) as sess: - model = p.Instantiate() - saver = checkpointer.Checkpointer(train_dir_2, model) - saver.RestoreIfNeeded(sess) - new_b = self.evaluate(model.GetTask().vars.b) - self.assertEqual(b1, new_b) - - # Set init_checkpoint_rules to restore all from the latest checkpoint - # by specifying just the original train directory, not a specific - # checkpoint. - rules = [('(.*)', '%s')] - p.train.init_from_checkpoint_rules = {train_dir: (rules, [])} - with self.session(graph=tf.Graph()) as sess: - model = p.Instantiate() - saver = checkpointer.Checkpointer(train_dir_2, model) - saver.RestoreIfNeeded(sess) - new_b = self.evaluate(model.GetTask().vars.b) - self.assertEqual(b2, new_b) - - def testSaveOnly(self): - train_dir = os.path.join(self.get_temp_dir(), 'testSaveOnly') - os.mkdir(train_dir) - p = base_model.SingleTaskModel.Params(LinearModel.Params()) - p.input = base_input_generator.BaseInputGenerator.Params() - - with self.session(graph=tf.Graph()) as sess: - model = p.Instantiate() - self.evaluate(tf.global_variables_initializer()) - saver = checkpointer.Checkpointer(train_dir, model, save_only=True) - saver.Save(sess, model.global_step) - with self.assertRaises(AssertionError): - saver.RestoreIfNeeded(sess) - - self.assertTrue( - os.path.isfile(os.path.join(train_dir, 'ckpt-00000000.index'))) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/cluster.py b/lingvo/core/cluster.py deleted file mode 100644 index 4718a2ac..00000000 --- a/lingvo/core/cluster.py +++ /dev/null @@ -1,549 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Specification of a training cluster.""" - -import heapq -import lingvo.compat as tf -from lingvo.core import hyperparams -from lingvo.core import py_utils -import numpy as np - -FLAGS = tf.flags.FLAGS - -_CLUSTER_STACK = py_utils.ThreadLocalStack() - - -class _Cluster: - """The whole training cluster from a single task's point of view.""" - - @classmethod - def _JobSpec(cls, replicas): - """Construct a job spec param with the given number of replicas.""" - p = hyperparams.Params() - # By default, we use /job:localhost so that most of tests can just - # work out of the box. trainer.py will then set job names accordingly. - # Is this being redefined? - p.Define('name', '/job:localhost', - 'TensorFlow job spec, e.g., /job:trainer, /job:ps') - p.Define('replicas', replicas, 'The number of tasks of a job.') - p.Define( - 'targets', '', 'The target network address(es) to which we can ' - 'create tf sessions. E.g., a single ip:port, or a list of ' - 'comma-separated grpc://ip:port, etc.') - p.Define('cpus_per_replica', 1, 'The number of CPU devices to use per ' - 'replica.') - p.Define('gpus_per_replica', 0, 'The number of GPU devices to use per ' - 'replica.') - p.Define( - 'devices_per_split', 1, 'Devices of a replica are grouped into ' - 'splits. Each split contains these many devices. One split is a ' - 'group of devices on which the computation nodes of a graph is ' - 'placed upon.E.g., one can place the forward lstm on device 0 of ' - 'a split and place the backward lstm on device 1. etc.') - p.Define('tpus_per_replica', 0, - 'The number of tpu cores to use per replica.') - p.Define('num_tpu_hosts', 0, 'The number of tpu hosts.') - return p - - @classmethod - def Params(cls): - """Defaults parameters for a cluster.""" - p = hyperparams.InstantiableParams(cls) - p.Define( - 'mode', 'async', 'A string noting the overall training method. ' - 'Valid values: sync, async.') - p.Define( - 'job', 'trainer', 'The role of this job in the training cluster. ' - 'E.g., trainer_client, trainer, controller, etc.') - p.Define('task', 0, 'This process is the task-th task in the job.') - p.Define('logdir', '', 'The log directory.') - - # Helpful - # How the cluster is composed. - # - # A typical training cluster has a few jobs (controller, worker, ps, etc). - # One can potentially place computation on any device of these jobs. - # Here, we specify how each job is configured. E.g., number of GPUs each - # task is equipped with, the number of replicas, etc. - # - # Note that trainer client may dispatch operations on just a - # smaller subset of jobs. For example, the controller only places - # computations onto the controller and ps devices; while evaler - # only places computations on the evaler devices. - # - # cluster.job refers to the role of a client process performs. It - # can be 'controller', 'trainer', 'trainer_client', 'evaler' and - # 'decoder', etc. Often, a client can be the same process as one - # of the compute devices (e.g., controller). Sometimes, they can - # be a separate processes. E.g., trainer_client is a separate - # standalone process. It places computations on the worker and - # ps devices, while itself does not host any. - p.Define('controller', cls._JobSpec(1), 'The controller job.') - p.Define('worker', cls._JobSpec(1), 'The worker job.') - p.Define('ps', cls._JobSpec(1), 'The ps job.') - p.Define('input', cls._JobSpec(0), 'The input job.') - p.Define('evaler', cls._JobSpec(0), 'The evaler job.') - p.Define('decoder', cls._JobSpec(0), 'The decoder job.') - - # A few 'global' knobs. - p.Define( - 'add_summary', None, 'Whether to add summaries. If None, ' - 'decides based on the job type.') - p.Define('do_eval', None, 'Whether to do eval.') - p.Define('split_id', 0, 'Split id for the model.') - p.Define('immediately_create_variables', True, - 'Whether to create variables immediately.') - return p - - @classmethod - def _MakeDeviceString(cls, job_name, task_id, device_name, device_id): - del cls - return '%s/replica:0/task:%d/device:%s:%d' % (job_name, task_id, - device_name, device_id) - - @classmethod - def ListDevices(cls, job_spec): - """Lists devices in the job. - - Args: - job_spec: A param object specifying a job in a training cluster. - - Returns: - Returns a 2D np string array. ret[i, j] is the i-th replica's j-th - devices. - """ - if job_spec.tpus_per_replica: - tpus = job_spec.tpus_per_replica - # Unfortunately, decoder replicas is 0. I need to fix this. - ret = np.empty((job_spec.replicas, tpus), np.object) - for i in range(job_spec.replicas): - for j in range(tpus): - ret[i, j] = cls._MakeDeviceString(job_spec.name, i, 'TPU', j) - elif not job_spec.gpus_per_replica: - cpus = job_spec.cpus_per_replica - ret = np.empty((job_spec.replicas, cpus), np.object) - for i in range(job_spec.replicas): - for j in range(cpus): - ret[i, j] = cls._MakeDeviceString(job_spec.name, i, 'CPU', j) - else: - ret = np.empty((job_spec.replicas, job_spec.gpus_per_replica), np.object) - for i in range(job_spec.replicas): - for j in range(job_spec.gpus_per_replica): - ret[i, j] = cls._MakeDeviceString(job_spec.name, i, 'GPU', j) - return ret - - def __enter__(self): - _CLUSTER_STACK.stack.append(self) - return self - - def __exit__(self, type_arg, value_arg, traceback_arg): - assert _CLUSTER_STACK.stack - assert _CLUSTER_STACK.stack[-1] is self - _CLUSTER_STACK.stack.pop() - - @staticmethod - def Top(): - return _CLUSTER_STACK.stack[-1] if _CLUSTER_STACK.stack else None - - def __init__(self, params): - self._params = params.Copy() - p = self.params - - print("GALVEZ:params=", str(params)) - print("GALVEZ:task_id=", params.task) - import sys; sys.stdout.flush() - - # A set of invariants about the setup of the cluster. - # - # NOTE. Two job specs can be identical. E.g., if p.worker.name is - # the same as p.ps.name, that means ps is colocated with worker. - assert p.ps.replicas >= 0 - assert p.ps.gpus_per_replica >= 0 - if p.mode == 'async' and p.job == 'controller': - # There is only 1 controller. - assert p.controller.replicas == 1 - assert p.task == 0 - elif p.mode == 'async' and p.job == 'trainer': - assert p.worker.replicas >= 1 - assert p.worker.gpus_per_replica >= 0 - assert p.worker.devices_per_split >= 1 - # In async mode, trainers colocate with workers. - assert 0 <= p.task and p.task < p.worker.replicas - if p.ps.replicas == 0: - # There is no ps. We are doing single-replica training. - assert p.worker.replicas == 1 - elif p.mode == 'async' and p.job == 'evaler': - assert 0 <= p.task and p.task < p.evaler.replicas - elif p.mode == 'async' and p.job == 'decoder': - assert 0 <= p.task and p.task < p.decoder.replicas - elif p.mode == 'sync' and p.job == 'controller': - # There is only 1 controller. - assert p.controller.replicas == 1 - assert p.task == 0 - elif p.mode == 'sync' and p.job == 'trainer_client': - assert p.worker.replicas >= 1 - assert p.worker.gpus_per_replica >= 0 - assert p.worker.devices_per_split >= 1 - elif p.mode == 'sync' and p.job == 'evaler': - assert 0 <= p.task and p.task < p.evaler.replicas - elif p.mode == 'sync' and p.job == 'decoder': - assert 0 <= p.task and p.task < p.decoder.replicas - elif p.mode == 'sync' and p.job == 'executor_tpu': - assert p.worker.replicas >= 1 - elif p.mode == 'sync' and p.job == 'tpu_decoder': - # from IPython import embed; embed() - # assert p.decoder.replicas == 1 - pass - else: - assert False, (p.mode, p.job) - - if p.job == 'controller': - self._job_spec = p.controller - elif p.job in ('trainer', 'worker', 'trainer_client'): - self._job_spec = p.worker - elif p.job == 'evaler': - self._job_spec = p.evaler - elif p.job == 'decoder': - self._job_spec = p.decoder - elif p.job == 'executor_tpu': - self._job_spec = p.worker - elif p.job == 'tpu_decoder': - # Not quite sure of this is right - self._job_spec = p.decoder - else: - assert False, (p.mode, p.job) - - @property - def params(self): - return self._params - - @property - def mode(self): - return self.params.mode - - @property - def job(self): - return self.params.job - - @property - def logdir(self): - return self.params.logdir - - @property - def task(self): - return self.params.task - - @property - def job_spec(self): - return self._job_spec - - @property - def asynchronous(self): - """Returns True if configured for asynchronous training.""" - return self.params.mode == 'async' - - @property - def synchronous(self): - """Returns True if configured for synchronous training.""" - return self.params.mode == 'sync' - - @property - def num_replicas(self): - return self._job_spec.replicas - - @property - def tpus_per_replica(self): - return self._job_spec.tpus_per_replica - - @property - def num_tpu_hosts(self): - return self._job_spec.num_tpu_hosts - - @property - def num_devices_per_replica(self): - return (self._job_spec.gpus_per_replica or - self._job_spec.tpus_per_replica or self._job_spec.cpus_per_replica) - - @property - def total_worker_devices(self): - """Return the total number of discrete worker devices in the cluster.""" - worker_spec = self.params.worker - devices_per_replica = ( - worker_spec.gpus_per_replica or worker_spec.tpus_per_replica or - self._job_spec.cpus_per_replica) - num_replicas = worker_spec.replicas - return devices_per_replica * num_replicas - - @property - def num_devices_per_split(self): - """Return number of accelerators to use per split.""" - return self._job_spec.devices_per_split - - @property - def num_splits_per_replica(self): - # Note that a split must be within a replica. - assert self.num_devices_per_replica % self.num_devices_per_split == 0 - return int(self.num_devices_per_replica / self.num_devices_per_split) - - @property - def num_splits_per_client(self): - """The number of splits visible by one trainer client.""" - if self.synchronous and self.job == 'trainer_client': - # One client drives all the workers. - return self.num_splits_per_replica * self.num_replicas - elif self.synchronous and self.job == 'executor_tpu': - # One client drives all the workers. - return self.num_splits_per_replica * self.num_replicas - else: - # One client colocates with one worker and drives the worker only. - return self.num_splits_per_replica - - @property - def available_devices(self): - """Returns all compute devices available in a 2D array. - - Returns: - A 2D array (python list of python lists) of strings. ret[i, j] - is the j-th visible device on i-th visible replica. - """ - if self._job_spec.tpus_per_replica: - ret = np.empty((1, self.num_devices_per_split), np.object) - for i in range(self.num_devices_per_split): - ret[0, i] = tf.tpu.core(i) - return ret - - if self.job == 'trainer' and self.asynchronous: - # In async mode, each trainer task can only use its own devices. - return self.ListDevices(self._job_spec)[self.task:(self.task + 1), :] - - if self.job == 'trainer_client' and self.synchronous: - # In sync mode, trainer_client can use every device. - return self.ListDevices(self._job_spec) - - if self.job == 'executor_tpu' and self.synchronous: - # executor_tpu can use every device. - return self.ListDevices(self._job_spec) - - if self.job in ('controller', 'evaler', 'decoder', 'tpu_decoder'): - # Our current policy is that each controller/evaler/decoder task - # only uses 1 replica. - - # Okay, so what is replica? Am I stuck using just one TPU for - # decoding then? No, I don't think so. - return self.ListDevices(self._job_spec)[self.task:(self.task + 1), :] - - assert False, (self.job, self.mode) - - @property - def input_device(self): - """Returns the tensorflow device name to place input op on.""" - p = self.params - if self.synchronous and p.input.replicas > 0: - # Uses a separate job for input processing. - assert p.input.replicas == 1 - return self.ListDevices(p.input)[0, 0] - else: - return '' - - def PlaceInput(self, input_params): - """Applies a placement policy on the given input generator params. - - By default, the policy is to place the input generator onto the input - device. Subclass can override PlaceInput method to implement more advanced - placement policy. - - Args: - input_params: An input generator params. - - Returns: - An input params which places the input generator on the input device. - """ - - class _UseInputDevice(input_params.cls): - """Places the input generator on the input device.""" - - def __init__(self, params): - with tf.device(self.cluster.input_device): - super().__init__(params) - - def SplitInputBatch(self, num_splits): - with tf.device(self.cluster.input_device): - return super().SplitInputBatch(num_splits) - - return input_params.Copy().Set(cls=_UseInputDevice) - - @property - def input_targets(self): - """Returns a list of network addresses of the input job.""" - p = self.params.input - if not p.targets: - return [] - targets = p.targets.split(',') - assert p.replicas == len(targets), '{} vs. {}'.format(p.replicas, targets) - return targets - - def WorkerDeviceInModelSplit(self, device_index): - """Returns the device to use for 'device_index' for the current model split. - - Args: - device_index: An int, the device index within 'model_split'. - - Returns: - A string. The device to place ops onto. - - Raises: - ValueError: if split_id of cluster is incorrectly set. - """ - devices = self.available_devices.reshape([-1]).tolist() - if not devices: - return '' - else: - split_id = self.params.split_id - if split_id < 0 or split_id >= self.num_splits_per_client: - raise ValueError('split_id (%d) not in [0, %d)' % - (split_id, self.num_splits_per_client)) - devices_per_split = self.num_devices_per_split - return devices[devices_per_split * split_id + - device_index % devices_per_split] - - def GetPlacer(self, strategy=None): - """Returns a device function for placing ops within the cluster. - - Args: - strategy: A string. Identifier for a placement strategy. By default, we - use a least loaded policy to place variables. - - Returns: - Returns a device function can be used in tf.device(). - - Raises: - ValueError: when strategy is not supported. - """ - if self.job == 'evaler': # or self.job == 'decoder': - # Currently, we only support evaler/decoder uses 1 accelerator. - return self.ListDevices(self.job_spec)[self.task, 0] - elif strategy is None: - return _LeastLoadedPlacer(self).DeviceFunction - raise ValueError('Unsupported placement policy: ', strategy) - - @property - def add_summary(self): - p = self.params - if p.add_summary is None: - return self.job in ['controller', 'decoder'] - else: - return p.add_summary - - @property - def do_eval(self): - return self.params.do_eval - - @property - def immediately_create_variables(self): - return self.params.immediately_create_variables - - @property - def worker_cluster_def(self): - """Returns a tf.train.ClusterDef representing the worker cluster.""" - p = self.params.worker - - if not p.targets: - return None - - job = p.name.replace('/job:', '', 1) - workers = [addr.replace('grpc://', '', 1) for addr in p.targets.split(',')] - - return tf.train.ClusterSpec({job: workers}).as_cluster_def() - - -# Ops that must be placed on the 'ps' devices. -_VAR_OPS = ['Variable', 'VariableV2', 'AutoReloadVariable', 'VarHandleOp'] - - -class VarPlacer: - """Placer which places variables across a set of devices. - - VarPlacer places non-variable ops on the worker device. - """ - - def __init__(self, cluster): - self._cluster = cluster - self._devices = cluster.ListDevices(cluster.job_spec) - - def _AssignVar(self, _): - raise ValueError('Unimplemented') - - def DeviceFunction(self, op): - """Choose a device for 'op'. - - Args: - op: an Operation. - - Returns: - The device to use for the Operation. - """ - # Op has already assigned to a device explicitly. Don't change it. - if op.device: - return op.device - - # Place vars according our policy. - if op.type in _VAR_OPS: - return self._AssignVar(op) - - # The default policy is to place the op on the 1st device visible - # to this task. - assert self._devices is not None, ('Unexpected job: %s' % self._cluster.job) - # You know, I wonder if this is exactly the cuase of the problem Anjali and I encountered. This DeivceFunction may have misplaced a the AsciiToId operation on TPU... - # print("GALVEZ:", self._devices) - # print("GALVEZ:", len(self._devices)) - # import sys; sys.stdout.flush() - task = self._cluster.params.task - assert 0 <= task and task < len(self._devices) - return self._devices[task, 0] - - -class _LeastLoadedPlacer(VarPlacer): - """Placer which places a variable on the least loaded var device. - - We use total byte sizes of variables placed on a device to indicate - the device's load. - - """ - - def __init__(self, cluster): - super().__init__(cluster) - # A min heap of (size, device) - var_devices = cluster.ListDevices(cluster.params.ps).flatten().tolist() - tf.logging.info('_LeastLoadedPlacer : %s', var_devices) - self._var_space_pq = [(0, d) for d in var_devices] - - def _AssignVar(self, var_op): - size = var_op.get_attr('dtype').size - shape = tf.TensorShape(var_op.get_attr('shape')) - assert self._var_space_pq, ('No ps devices to use.') - allocated, device = heapq.heappop(self._var_space_pq) - if shape.num_elements() is None: - assert var_op.name.endswith( - 'wb/var'), 'Unexpected name pattern: %s' % var_op.name - # CuDNN RNN vars shape aren't known statically, decide to make a constant - # estimate to avoid introducing more complexities. - allocated += 10 * 1024**2 * size - else: - allocated += shape.num_elements() * size - heapq.heappush(self._var_space_pq, (allocated, device)) - tf.logging.info('Place variable %s on %s %d', var_op.name, device, - allocated) - return device diff --git a/lingvo/core/cluster_factory.py b/lingvo/core/cluster_factory.py deleted file mode 100644 index 97be958b..00000000 --- a/lingvo/core/cluster_factory.py +++ /dev/null @@ -1,129 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Cluster factory.""" - -from lingvo.core import cluster - -Cluster = cluster._Cluster # pylint: disable=protected-access - - -def SetCluster(cls): - """Sets Cluster implementation.""" - global Cluster # pylint: disable=invalid-name - Cluster = cls - - -def Current(): - """Returns the current cluster specification. - - E.g.:: - - with Cluster(cluster_params) as foo: - model = p.Instantiate() - model.FProp() # FProp can access foo through cluster_factory.Current(). - """ - current = Cluster.Top() - if not current: - current = Cluster(Cluster.Params()) - return current - - -def ForTestingWorker(mode=None, - job=None, - gpus=None, - split_size=None, - tpus=None, - add_summary=None, - cpus=None, - do_eval=None, - num_tpu_hosts=None): - """Returns a Cluster for unittesting with a worker.""" - p = Current().params.Copy() - if mode is not None: - p.mode = mode - if job is not None: - p.job = job - if gpus is not None: - p.worker.gpus_per_replica = gpus - if tpus is not None: - p.worker.tpus_per_replica = tpus - p.worker.num_tpu_hosts = 1 - if cpus is not None: - p.worker.cpus_per_replica = cpus - if split_size is not None: - p.worker.devices_per_split = split_size - if add_summary is not None: - p.add_summary = add_summary - if do_eval is not None: - p.do_eval = do_eval - if num_tpu_hosts is not None: - p.worker.num_tpu_hosts = num_tpu_hosts - return p.Instantiate() - - -def SetEval(mode): - """Returns a cluster with do_eval option turned on/off. - - E.g.:: - - def FProp(...): - with SetEval(mode=True): - # Turns off dropout, noise, etc. - y = self.foo.FProp(..., x) - z = self.bar.FProp(..., y) - # Returns to previous state (e.g., training). - y = self.foo.FProp(..., x) - z = self.foo.FProp(..., y) - - Args: - mode: True, False or None. - - Returns: - A new Cluster instance. - """ - return Current().params.Copy().Set(do_eval=mode).Instantiate() - - -def SetImmediatelyInstantiateVariables(mode): - """Returns a cluster with immediately_create_variables option turned on/off. - - Args: - mode: True, False or None. - - Returns: - A new Cluster instance. - """ - return Current().params.Copy().Set( - immediately_create_variables=mode).Instantiate() - - -def SetModelSplit(split_id): - """Returns the current cluster with the model split id set. - - E.g.:: - - def FProp(...): - with cluster_factory.SetModelSplit(1) as c: - with tf.device(c.WorkerDeviceInModelSplit(0)): - ... - - Args: - split_id: Integer split id for the model. - - Returns: - A new Cluster instance. - """ - return Current().params.Copy().Set(split_id=split_id).Instantiate() diff --git a/lingvo/core/cluster_test.py b/lingvo/core/cluster_test.py deleted file mode 100644 index 7530d1a7..00000000 --- a/lingvo/core/cluster_test.py +++ /dev/null @@ -1,440 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for cluster.""" - -import lingvo.compat as tf -from lingvo.core import cluster_factory -from lingvo.core import test_utils -import numpy as np - - -class ClusterTest(test_utils.TestCase): - - def testDefaultParams(self): - p = cluster_factory.Cluster.Params() - c = cluster_factory.Cluster(p) - self.assertFalse(c.add_summary) - g = tf.Graph() - vs = [] - with g.as_default(): - with tf.device(c.GetPlacer()): - for i in range(10): - vs.append(tf.get_variable('x%d' % i, (10, 10, 10))) - sum_all = tf.add_n(vs) - for v in vs: - self.assertEqual( - v.device, - c._MakeDeviceString( - job_name='/job:localhost', - task_id=0, - device_name='CPU', - device_id=0)) - self.assertEqual( - sum_all.device, - c._MakeDeviceString( - job_name='/job:localhost', - task_id=0, - device_name='CPU', - device_id=0)) - - def testDefaultParamsWithDynamicShape(self): - p = cluster_factory.Cluster.Params() - c = cluster_factory.Cluster(p) - g = tf.Graph() - vs = [] - with g.as_default(): - with tf.device(c.GetPlacer()): - for i in range(10): - dyn_shape = tf.constant([2], dtype=tf.int32) - dyn_shape = tf.placeholder_with_default(dyn_shape, shape=[None]) - v = tf.get_variable( - 'x%d_wb/var' % i, - initializer=tf.random.uniform(dyn_shape, dtype=tf.float64), - validate_shape=False) - vs.append(v) - sum_all = tf.add_n(vs) - for v in vs: - self.assertEqual( - v.device, - c._MakeDeviceString( - job_name='/job:localhost', - task_id=0, - device_name='CPU', - device_id=0)) - self.assertEqual( - sum_all.device, - c._MakeDeviceString( - job_name='/job:localhost', - task_id=0, - device_name='CPU', - device_id=0)) - - def testNoPS(self): - p = cluster_factory.Cluster.Params() - p.worker.name = '/job:trainer' - p.worker.replicas = 1 - p.ps.name = '/job:trainer' - p.ps.replicas = 1 - c = cluster_factory.Cluster(p) - g = tf.Graph() - vs = [] - with g.as_default(): - with tf.device(c.GetPlacer()): - for i in range(10): - vs.append(tf.get_variable('x%d' % i, (10, 10, 10))) - sum_all = tf.add_n(vs) - for v in vs: - self.assertEqual( - v.device, - c._MakeDeviceString( - job_name='/job:trainer', - task_id=0, - device_name='CPU', - device_id=0)) - self.assertEqual( - sum_all.device, - c._MakeDeviceString( - job_name='/job:trainer', task_id=0, device_name='CPU', device_id=0)) - - def testNoPSWithGPUs(self): - p = cluster_factory.Cluster.Params() - p.worker.name = '/job:trainer' - p.worker.replicas = 1 - p.worker.gpus_per_replica = 4 - p.ps.name = '/job:trainer' - p.ps.replicas = 1 - p.ps.gpus_per_replica = 4 - - c = cluster_factory.Cluster(p) - g = tf.Graph() - vs = [] - with g.as_default(): - with tf.device(c.GetPlacer()): - for i in range(10): - vs.append(tf.get_variable('x%d' % i, (10, 10, 10))) - sum_all = tf.add_n(vs) - for i, v in enumerate(vs): - self.assertEqual( - v.device, - c._MakeDeviceString( - job_name='/job:trainer', - task_id=0, - device_name='GPU', - device_id=i % 4)) - self.assertEqual( - sum_all.device, - c._MakeDeviceString( - job_name='/job:trainer', - task_id=0, - device_name='GPU', - device_id=0)) - - def testPS(self): - p = cluster_factory.Cluster.Params() - p.worker.name = '/job:trainer' - p.worker.replicas = 1 - p.ps.name = '/job:ps' - p.ps.replicas = 4 - c = cluster_factory.Cluster(p) - g = tf.Graph() - vs = [] - with g.as_default(): - with tf.device(c.GetPlacer()): - for i in range(10): - vs.append(tf.get_variable('x%d' % i, (10, 10, 10))) - sum_all = tf.add_n(vs) - for i, v in enumerate(vs): - self.assertEqual( - v.device, - c._MakeDeviceString( - job_name='/job:ps', task_id=i % 4, device_name='CPU', - device_id=0)) - self.assertEqual( - sum_all.device, - c._MakeDeviceString( - job_name='/job:trainer', task_id=0, device_name='CPU', device_id=0)) - - def testPSWithGPUs(self): - p = cluster_factory.Cluster.Params() - p.worker.name = '/job:trainer' - p.worker.replicas = 1 - p.ps.name = '/job:ps' - p.ps.replicas = 4 - p.ps.gpus_per_replica = 2 - c = cluster_factory.Cluster(p) - g = tf.Graph() - vs = [] - with g.as_default(): - with tf.device(c.GetPlacer()): - for i in range(10): - vs.append(tf.get_variable('x%d' % i, (10, 10, 10))) - sum_all = tf.add_n(vs) - for i, v in enumerate(vs): - self.assertEqual( - v.device, - c._MakeDeviceString( - job_name='/job:ps', - task_id=(i / 2) % 4, - device_name='GPU', - device_id=i % 2)) - self.assertEqual( - sum_all.device, - c._MakeDeviceString( - job_name='/job:trainer', task_id=0, device_name='CPU', device_id=0)) - - def testPSRandomSize(self): - p = cluster_factory.Cluster.Params() - p.worker.name = '/job:trainer' - p.ps.name = '/job:ps' - p.ps.replicas = 10 - c = cluster_factory.Cluster(p) - g = tf.Graph() - vs = [] - np.random.seed(301) - with g.as_default(): - with tf.device(c.GetPlacer()): - # Creates 200 variables with different sizes. - for i in range(200): - if i % 13: - size = np.random.randint(10000) - elif i % 7: - size = np.random.randint(100) - else: - size = np.random.randint(10) - vs.append(tf.get_variable('x%d' % i, shape=(size))) - sum_all = tf.add_n([tf.reduce_sum(x) for x in vs]) - # Computes the total size of variables placed on each device. - total_size = {} # device name -> size - for v in vs: - size = tf.TensorShape(v.op.get_attr('shape')).num_elements() - if v.device in total_size: - total_size[v.device] += size - else: - total_size[v.device] = size - for (device, allocated) in zip( - sorted(total_size), - [91701, 91361, 90346, 88738, 87240, 89265, 91944, 92472, 88051, 95053]): - self.assertEqual(total_size[device], allocated) - self.assertEqual( - sum_all.device, - c._MakeDeviceString( - job_name='/job:trainer', task_id=0, device_name='CPU', device_id=0)) - - def testDeviceListOneRepliaCpu(self): - p = cluster_factory.Cluster.Params() - p.mode = 'async' - p.job = 'trainer' - p.worker.cpus_per_replica = 2 - c = cluster_factory.Cluster(p) - cpu_devices = c.available_devices - expected_cpu_devices = [[ - c._MakeDeviceString( - job_name='/job:localhost', - task_id=0, - device_name='CPU', - device_id=0), - c._MakeDeviceString( - job_name='/job:localhost', - task_id=0, - device_name='CPU', - device_id=1), - ]] - print(expected_cpu_devices) - self.assertAllEqual(cpu_devices, expected_cpu_devices) - - def testDeviceListOneReplicaGpu(self): - p = cluster_factory.Cluster.Params() - p.mode = 'async' - p.job = 'trainer' - p.worker.gpus_per_replica = 2 - c = cluster_factory.Cluster(p) - gpu_devices = c.available_devices - expected_gpu_devices = [[ - c._MakeDeviceString( - job_name='/job:localhost', - task_id=0, - device_name='GPU', - device_id=0), - c._MakeDeviceString( - job_name='/job:localhost', - task_id=0, - device_name='GPU', - device_id=1), - ]] - self.assertAllEqual(gpu_devices, expected_gpu_devices) - - def testDeviceListMultiReplicaNoSyncSgd(self): - p = cluster_factory.Cluster.Params() - p.mode = 'async' - p.job = 'trainer' - p.task = 1 - p.worker.replicas = 2 - p.worker.gpus_per_replica = 2 - c = cluster_factory.Cluster(p) - gpu_devices = c.available_devices - expected_gpu_devices = [[ - c._MakeDeviceString( - job_name='/job:localhost', - task_id=1, - device_name='GPU', - device_id=0), - c._MakeDeviceString( - job_name='/job:localhost', - task_id=1, - device_name='GPU', - device_id=1), - ]] - self.assertAllEqual(gpu_devices, expected_gpu_devices) - - # Compute the total number of worker devices for a multi - # replica setup. - self.assertEqual(4, c.total_worker_devices) - - # Even when the job is different, we still look at the worker - # information. - p.job = 'controller' - p.task = 0 - c = cluster_factory.Cluster(p) - self.assertEqual(4, c.total_worker_devices) - - def testDeviceListMultiReplicaSyncSgd(self): - p = cluster_factory.Cluster.Params() - p.mode = 'sync' - p.job = 'trainer_client' - p.worker.name = '/job:localhost' - p.worker.replicas = 2 - p.worker.gpus_per_replica = 2 - c = cluster_factory.Cluster(p) - gpu_devices = c.available_devices - expected_gpu_devices = [[ - c._MakeDeviceString( - job_name='/job:localhost', - task_id=0, - device_name='GPU', - device_id=0), - c._MakeDeviceString( - job_name='/job:localhost', - task_id=0, - device_name='GPU', - device_id=1), - ], [ - c._MakeDeviceString( - job_name='/job:localhost', - task_id=1, - device_name='GPU', - device_id=0), - c._MakeDeviceString( - job_name='/job:localhost', - task_id=1, - device_name='GPU', - device_id=1), - ]] - self.assertAllEqual(gpu_devices, expected_gpu_devices) - - def testInputDevice(self): - p = cluster_factory.Cluster.Params() - p.mode = 'sync' - p.job = 'decoder' - p.decoder.replicas = 1 - p.task = 0 - p.input.name = '/job:input' - p.input.replicas = 1 - c = cluster_factory.Cluster(p) - input_device = c.input_device - expected_device = c._MakeDeviceString( - job_name='/job:input', task_id=0, device_name='CPU', device_id=0) - self.assertEqual(input_device, expected_device) - - def testModelSplit(self): - p = cluster_factory.Cluster.Params() - with p.Instantiate() as c: - with cluster_factory.SetModelSplit(2) as c1: - self.assertEqual(c1.params.split_id, 2) - with cluster_factory.SetModelSplit(3) as c2: - self.assertEqual(c2.params.split_id, 3) - self.assertEqual(c.params.split_id, 0) - - def testInputTargets(self): - p = cluster_factory.Cluster.Params() - p.input.name = '/job:input' - p.input.replicas = 2 - p.input.targets = '10.100.1.1:10001,10.100.1.2:10002' - c = cluster_factory.Cluster(p) - self.assertEqual(c.input_targets, ['10.100.1.1:10001', '10.100.1.2:10002']) - - def testWorkerDeviceInModelSplitSync(self): - p = cluster_factory.Cluster.Params() - p.mode = 'sync' - p.job = 'trainer_client' - p.worker.name = '/job:trainer' - p.worker.replicas = 4 - p.worker.gpus_per_replica = 4 - p.worker.devices_per_split = 2 - with cluster_factory.Cluster(p): - with cluster_factory.SetModelSplit(1) as c: - d = c.WorkerDeviceInModelSplit(1) - expected_device = c._MakeDeviceString( - job_name='/job:trainer', task_id=0, device_name='GPU', device_id=3) - self.assertEqual(expected_device, d) - - def testWorkerDeviceInModelSplit(self): - p = cluster_factory.Cluster.Params() - p.mode = 'async' - p.job = 'trainer' - p.task = 3 - p.worker.name = '/job:trainer' - p.worker.replicas = 4 - p.worker.gpus_per_replica = 4 - p.worker.devices_per_split = 2 - with cluster_factory.Cluster(p): - with cluster_factory.SetModelSplit(1) as c: - d = c.WorkerDeviceInModelSplit(1) - expected_device = c._MakeDeviceString( - job_name='/job:trainer', task_id=3, device_name='GPU', device_id=3) - self.assertEqual(expected_device, d) - - def testWorkerClusterDef(self): - p = cluster_factory.Cluster.Params() - - p.worker.name = '/job:trainer_client' - p.worker.targets = ','.join([ - 'grpc://localhost:8470', - 'grpc://localhost:8471', - 'grpc://localhost:8472', - 'grpc://localhost:8473', - ]) - - cluster_def = tf.train.ClusterSpec({ - 'trainer_client': [ - 'localhost:8470', 'localhost:8471', 'localhost:8472', - 'localhost:8473' - ] - }).as_cluster_def() - - cluster = p.Instantiate() - self.assertEqual(cluster.worker_cluster_def, cluster_def) - - def testWorkerClusterDefWithoutTargets(self): - p = cluster_factory.Cluster.Params() - - p.worker.name = '/job:trainer_client' - - cluster = p.Instantiate() - self.assertIsNone(cluster.worker_cluster_def) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/computation_cost.py b/lingvo/core/computation_cost.py deleted file mode 100644 index d1b54482..00000000 --- a/lingvo/core/computation_cost.py +++ /dev/null @@ -1,73 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Utilities to estimate computation costs of layers.""" - -from lingvo import compat as tf -from lingvo.core import bn_layers # for AddingAccumulator - -COST_METRICS = { - 'flops': tf.int64, -} - - -def Prepare(layer): - """Registers cost accumulators for layer and each of its child layers. - - This function should be called before FProp(). - - Args: - layer: the root layer. - """ - - def _Traverse(layer): - """Adds accumulators to layer and its descendant layers.""" - if isinstance(layer, (list, tuple)): - for layer_i in layer: - _Traverse(layer_i) - return - with tf.name_scope(layer.params.name): - for cost_metric_name in COST_METRICS: - dtype = COST_METRICS[cost_metric_name] - layer.RegisterAccumulator( - cost_metric_name, - bn_layers.AddingAccumulator(shape=[], dtype=dtype)) - for _, child in sorted(layer.children.items()): - _Traverse(child) - - _Traverse(layer) - - -def _HasAccumulator(layer, cost_metric_name): - assert cost_metric_name in COST_METRICS - return cost_metric_name in layer.accumulators - - -def _GetAccumulator(layer, cost_metric_name): - assert cost_metric_name in COST_METRICS - if cost_metric_name not in layer.accumulators: - raise ValueError('Prepare was not called for %s' % layer.path) - return layer.accumulators[cost_metric_name] - - -def Add(layer, cost_metric_name, cost): - if not _HasAccumulator(layer, cost_metric_name): - return - _GetAccumulator(layer, cost_metric_name).Update(cost) - - -def Get(layer, cost_metric_name): - """Returns the aggregated cost metric.""" - return _GetAccumulator(layer, cost_metric_name).GetValue() diff --git a/lingvo/core/conformer_layer.py b/lingvo/core/conformer_layer.py deleted file mode 100644 index 5de9b17c..00000000 --- a/lingvo/core/conformer_layer.py +++ /dev/null @@ -1,357 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Conformer layers as in https://arxiv.org/abs/2005.08100.""" - -from lingvo import compat as tf - -from lingvo.core import base_layer -from lingvo.core import batch_major_attention as attention_lib -from lingvo.core import bn_layers -from lingvo.core import conv_layers_with_time_padding -from lingvo.core import layers -from lingvo.core import layers_with_attention - - -class LConvLayer(base_layer.BaseLayer): - r"""Lightweight conv layer. - - architecture:: - - input - / \ - | ln(.) # input_dim - | fflayer(.) # 2 * input_dim - | | - | glu(.) # input_dim - | depthwise_conv_1d(.) - | norm(.) - | act(.) - | | - | fflayer(.) - | dropout(.) - \ / - + - | - output - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', None, 'Input and (in fact,) output dimension.') - p.Define('kernel_size', None, 'Kernel size of 1d deptwise conv.') - p.Define('conv_activation', 'SWISH', 'Activation after normalization.') - p.Define( - 'glu_activation', 'NONE', - 'Activation in GLU. Check lingvo.core.layers._ACTIVATIONS for ' - 'other options.') - p.Define('dropout_prob', 0., 'Dropout probability.') - - p.Define('ln_tpl', layers.LayerNorm.Params(), 'Input layer norm template.') - p.Define('linear_start_tpl', layers.FCLayer.Params(), 'Linear start layer.') - p.Define('depthwise_conv_tpl', - conv_layers_with_time_padding.DepthwiseConv2DLayer.Params(), - 'Depthwise conv template.') - p.Define('conv_norm_layer_tpl', bn_layers.BatchNormLayer.Params(), - 'Normalization layer after conv.') - p.Define('linear_end_tpl', layers.FCLayer.Params(), 'Linear end layer.') - p.Define('dropout_tpl', layers.DropoutLayer.Params(), - 'Residual dropout layer.') - - p.linear_start_tpl.Set(activation='NONE', has_bias=True) - p.linear_end_tpl.Set(activation='NONE', has_bias=True) - return p - - @classmethod - def CommonParams(cls, - input_dim=None, - kernel_size=None, - conv_activation='SWISH', - dropout_prob=0.): - p = cls.Params().Set( - input_dim=input_dim, - kernel_size=kernel_size, - conv_activation=conv_activation, - dropout_prob=dropout_prob) - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - ln_p = p.ln_tpl.Copy().Set(name='ln', input_dim=p.input_dim) - self.CreateChild('ln', ln_p) - - linear_start_p = p.linear_start_tpl.Copy().Set( - name='linear_start', input_dim=p.input_dim, output_dim=2 * p.input_dim) - linear_end_p = p.linear_end_tpl.Copy().Set( - name='linear_end', input_dim=p.input_dim, output_dim=p.input_dim) - self.CreateChild('linear_start', linear_start_p) - self.CreateChild('linear_end', linear_end_p) - - norm_p = p.conv_norm_layer_tpl.Copy().Set( - name='norm_layer', dim=p.input_dim) - self.CreateChild('norm', norm_p) - - # 1d depthwise conv with channel_mulitplier = 1 - depthwise_conv_p = p.depthwise_conv_tpl.Copy().Set( - name='depthwise_conv', - filter_shape=(p.kernel_size, 1, p.input_dim, 1), - filter_stride=(1, 1)) - self.CreateChild('depthwise_conv1d', depthwise_conv_p) - - dropout_p = p.dropout_tpl.Copy().Set( - name='dropout', keep_prob=1. - p.dropout_prob) - self.CreateChild('dropout', dropout_p) - - def _GLU(self, inputs): - p = self.params - gated_inputs, act_inputs = tf.split(inputs, 2, axis=-1) - return self._ApplyActivation(act_inputs, - p.glu_activation) * tf.sigmoid(gated_inputs) - - def _ApplyActivation(self, inputs, act_name): - if act_name == 'NONE': - return inputs - return layers._ACTIVATIONS[act_name](inputs) # pylint:disable=protected-access - - def FProp(self, theta, inputs, paddings): - """Builds FProp graph. - - Args: - theta: A NestedMap of Tensors, see base class. - inputs: A Tensor of shape [batch, seqlen, dim0]. - paddings: A Tensor of shape [batch, seqlen]. - - Returns: - output: A Tensor of shape [batch, seqlen, dim0]. - out_paddings: A Tensor of shape [batch, seqlen]. - """ - - p = self.params - with tf.name_scope(p.name): - unnormalized_inputs = inputs - - inputs = self.ln.FProp(theta.ln, inputs) - inputs = self.linear_start.FProp(theta.linear_start, inputs) - - inputs = self._GLU(inputs) - - # [b, t, d] --> [b, t, 1, d] - inputs = tf.expand_dims(inputs, 2) - inputs, paddings = self.depthwise_conv1d.FProp(theta.depthwise_conv1d, - inputs, paddings) - inputs = tf.squeeze(inputs, 2) - - inputs = self.norm.FProp(theta.norm, inputs) - inputs = self._ApplyActivation(inputs, p.conv_activation) - - inputs = self.linear_end.FProp(theta.linear_end, inputs) - inputs = self.dropout.FProp(theta.dropout, inputs) - - output = inputs + unnormalized_inputs - return output, paddings - - -class ConformerLayer(base_layer.BaseLayer): - """Conformer layer as in https://arxiv.org/abs/2005.08100. - - Canonical version (with default params.) - x = x + 1/2 * FFN(x) - x = x + MHSA(x) - x = x + Lconv(x) - x = x + 1/2 * FFN(x) - y = ln(x) - - Optionally one can change the order of MHSA and conv. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', None, 'Input dimension.') - # atten layer - p.Define('atten_num_heads', None, - 'Num of heads in multi-head self-attention.') - p.Define( - 'atten_left_context', None, 'Local self attention left context.' - 'If None, infinite left context.') - p.Define( - 'atten_right_context', None, 'Local self attention right context.' - 'If None, infinite right context.') - p.Define('use_relative_atten', True, 'If using relative attention.') - p.Define( - 'relative_pos_emb_dim', None, - 'If use_relative_atten, sets the relative pos embedding dim.' - 'Default is the same as input_dim.') - p.Define('layer_order', 'mhsa_before_conv', - 'Only mhsa_before_conv or conv_before_mhsa are supported.') - - # lconv layer - p.Define('kernel_size', None, 'Kernel size of 1d lightweight conv.') - - # fflayer - p.Define('fflayer_hidden_dim', None, - 'Hidden dim of the fflayers (start and end).') - p.Define('fflayer_activation', 'SWISH', 'fflayer activation.') - p.Define('fflayer_residual_weight', 0.5, 'fflayer residual weight.') - p.Define('dropout_prob', None, 'Signature dropout prob of inner componets.') - - # tpl - p.Define('fflayer_start_tpl', - layers_with_attention.TransformerFeedForwardLayer.Params(), - 'Layer params for Feed forward layer at the beginning.') - p.Define('trans_atten_tpl', - attention_lib.TransformerAttentionLayer.Params(), - 'Self attention layer params.') - p.Define('lconv_tpl', LConvLayer.Params(), 'Convolution module params.') - p.Define('fflayer_end_tpl', - layers_with_attention.TransformerFeedForwardLayer.Params(), - 'Layer params for Feed forward layer at the end.') - p.Define('final_ln_tpl', layers.LayerNorm.Params(), 'Final layer norm.') - return p - - @classmethod - def CommonParams(cls, - input_dim=None, - atten_num_heads=None, - atten_local_context=None, - atten_left_context=None, - atten_right_context=None, - use_relative_atten=True, - kernel_size=None, - fflayer_hidden_dim=None, - fflayer_activation='SWISH', - layer_order='mhsa_before_conv', - dropout_prob=0.): - assert all([input_dim, atten_num_heads, kernel_size, fflayer_hidden_dim]) - - if atten_local_context: - assert not any([atten_left_context, atten_right_context]), ( - 'atten_local_context and atten_{left,right}_context can not be set' - 'at the same time.') - atten_left_context = atten_local_context + 1 # including self position. - atten_right_context = atten_local_context - - p = cls.Params().Set( - input_dim=input_dim, - atten_num_heads=atten_num_heads, - atten_left_context=atten_left_context, - atten_right_context=atten_right_context, - use_relative_atten=use_relative_atten, - fflayer_hidden_dim=fflayer_hidden_dim, - fflayer_activation=fflayer_activation, - kernel_size=kernel_size, - layer_order=layer_order, - dropout_prob=0.) - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.layer_order in ['mhsa_before_conv', 'conv_before_mhsa'] - - fflayer_start_p = p.fflayer_start_tpl.Copy().Set( - input_dim=p.input_dim, - hidden_dim=p.fflayer_hidden_dim, - activation='SWISH', - residual_weight=p.fflayer_residual_weight, - residual_dropout_prob=p.dropout_prob, - relu_dropout_prob=p.dropout_prob) - self.CreateChild('fflayer_start', fflayer_start_p) - - fflayer_end_p = p.fflayer_end_tpl.Copy().Set( - input_dim=p.input_dim, - hidden_dim=p.fflayer_hidden_dim, - activation='SWISH', - residual_weight=p.fflayer_residual_weight, - residual_dropout_prob=p.dropout_prob, - relu_dropout_prob=p.dropout_prob) - self.CreateChild('fflayer_end', fflayer_end_p) - - trans_atten_p = p.trans_atten_tpl.Copy().Set( - input_dim=p.input_dim, - num_heads=p.atten_num_heads, - atten_dropout_prob=p.dropout_prob, - residual_dropout_prob=p.dropout_prob) - self._ConfigSelfAttenParams(trans_atten_p) - self.CreateChild('trans_atten', trans_atten_p) - - lconv_p = p.lconv_tpl.cls.CommonParams( - input_dim=p.input_dim, kernel_size=p.kernel_size) - self.CreateChild('lconv', lconv_p) - - ln_p = p.final_ln_tpl.Copy().Set(name='final_ln', input_dim=p.input_dim) - self.CreateChild('final_ln', ln_p) - - def _ConfigSelfAttenParams(self, trans_atten_p): - p = self.params - if not p.relative_pos_emb_dim: - p.relative_pos_emb_dim = p.input_dim - if not any([p.atten_left_context, p.atten_right_context]): - atten_type = 'global' if not p.use_relative_atten else 'global_relative' - elif p.atten_left_context is None and p.atten_right_context == 0: - assert not p.use_relative_atten, ( - 'Relative attention isn\'t supported for causal attention.') - atten_type = 'causal' - else: - atten_type = 'local_relative' if p.use_relative_atten else 'local' - - if atten_type == 'causal': - trans_atten_p.is_masked = True - elif atten_type == 'global_relative': - trans_atten_p.atten_tpl = ( - attention_lib.MultiHeadedAttentionXL.Params().Set( - rel_pos_emb_dim=p.relative_pos_emb_dim)) - elif atten_type == 'local_relative': - trans_atten_p.atten_tpl = attention_lib.LocalSelfAttentionXL.Params().Set( - left_context=p.atten_left_context, - right_context=p.atten_right_context, - rel_pos_emb_dim=p.relative_pos_emb_dim) - elif atten_type == 'local': - trans_atten_p.atten_tpl = attention_lib.LocalSelfAttention.Params().Set( - left_context=p.atten_left_context, - right_context=p.atten_right_context) - # No op for 'global' atten - - def _SelfAtten(self, theta, inputs, paddings): - inputs, _ = self.trans_atten.FProp( - theta.trans_atten, - query_vec=inputs, - source_vecs=None, - paddings=paddings) - return inputs, paddings - - def _LConv(self, theta, inputs, paddings): - inputs, paddings = self.lconv.FProp(theta.lconv, inputs, paddings) - return inputs, paddings - - def FProp(self, theta, inputs, paddings): - p = self.params - - with tf.name_scope(p.name): - inputs = self.fflayer_start.FProp(theta.fflayer_start, inputs, paddings) - if p.layer_order == 'mhsa_before_conv': - inputs, paddings = self._SelfAtten(theta, inputs, paddings) - inputs, paddings = self._LConv(theta, inputs, paddings) - else: - assert p.layer_order == 'conv_before_mhsa' - inputs, paddings = self._LConv(theta, inputs, paddings) - inputs, paddings = self._SelfAtten(theta, inputs, paddings) - inputs = self.fflayer_end.FProp(theta.fflayer_end, inputs, paddings) - - inputs = self.final_ln.FProp(theta.final_ln, inputs) - return inputs, paddings diff --git a/lingvo/core/conformer_layer_test.py b/lingvo/core/conformer_layer_test.py deleted file mode 100644 index 64287c16..00000000 --- a/lingvo/core/conformer_layer_test.py +++ /dev/null @@ -1,75 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for conformer layers as in https://arxiv.org/abs/2005.08100.""" -# Lint as: PY3 -from absl.testing import parameterized - -from lingvo import compat as tf -from lingvo.core import conformer_layer -from lingvo.core import test_utils - - -class LConvLayerTest(test_utils.TestCase, parameterized.TestCase): - - def testBasic(self): - batch, seqlen, dim = 2, 16, 4 - inputs = tf.zeros([batch, seqlen, dim]) - paddings = tf.zeros([batch, seqlen]) - - p = conformer_layer.LConvLayer.CommonParams(input_dim=dim, kernel_size=3) - p.name = 'lconv_layer' - l = p.Instantiate() - outputs = l.FPropDefaultTheta(inputs, paddings) - - with self.session() as sess: - tf.global_variables_initializer().run() - out_vals = sess.run(outputs) - print([x.shape for x in out_vals]) - - -class ConformerLayerTest(test_utils.TestCase, parameterized.TestCase): - - @parameterized.named_parameters( - ('Base',), - ('Reordered', 'conv_before_mhsa'), - ) - def testBasic(self, layer_order='mhsa_before_conv'): - batch, seqlen, dim, heads = 2, 32, 4, 2 - context = 2 - - inputs = tf.zeros([batch, seqlen, dim]) - paddings = tf.zeros([batch, seqlen]) - - p = conformer_layer.ConformerLayer.CommonParams( - input_dim=dim, - atten_num_heads=heads, - atten_left_context=context + 1, - atten_right_context=context, - kernel_size=3, - fflayer_hidden_dim=4 * dim) - p.name = 'conformer_layer' - p.layer_order = layer_order - l = p.Instantiate() - outputs = l.FPropDefaultTheta(inputs, paddings) - - with self.session() as sess: - tf.global_variables_initializer().run() - out_vals = sess.run(outputs) - print([x.shape for x in out_vals]) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/constants.py b/lingvo/core/constants.py deleted file mode 100644 index c5d6ee77..00000000 --- a/lingvo/core/constants.py +++ /dev/null @@ -1,19 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Constants used by lingvo.""" - - -REFERENCE_ANNOTATION = 'arxiv.org/abs/1902.08295' diff --git a/lingvo/core/conv_layers_builder.py b/lingvo/core/conv_layers_builder.py deleted file mode 100644 index c7a6d3e5..00000000 --- a/lingvo/core/conv_layers_builder.py +++ /dev/null @@ -1,530 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""2D conv layers that are expected to be used with sequence inputs.""" -import math -from absl import flags - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import builder -from lingvo.core import builder_layers -from lingvo.core import conv_layers_with_time_padding -from lingvo.core import py_utils -from lingvo.core import tshape - -FLAGS = flags.FLAGS - -Conv2DLayerWithPadding = conv_layers_with_time_padding.Conv2DLayerWithPadding -CausalConv2DLayerWithPadding = conv_layers_with_time_padding.CausalConv2DLayerWithPadding -DepthwiseConv2DLayer = conv_layers_with_time_padding.DepthwiseConv2DLayer -CausalDepthwiseConv2DLayer = conv_layers_with_time_padding.CausalDepthwiseConv2DLayer -ConvBatchNormLayer = conv_layers_with_time_padding.ConvBatchNormLayer -ConvCategoricalBN = conv_layers_with_time_padding.ConvCategoricalBN -ActivationLayer = conv_layers_with_time_padding.ActivationLayer -PaddingLayer = conv_layers_with_time_padding.PaddingLayer -NormalizedDepthwiseConv2DLayer = conv_layers_with_time_padding.NormalizedDepthwiseConv2DLayer -CausalNormalizedDepthwiseConv2DLayer = conv_layers_with_time_padding.CausalNormalizedDepthwiseConv2DLayer -GlobalPoolingLayer = conv_layers_with_time_padding.GlobalPoolingLayer - - -class BiasLayer(builder_layers.BiasLayer): - - def FProp(self, theta, inputs, paddings): - bias_added = super().FProp(theta, inputs) - return bias_added, paddings - - -class CausalPoolingLayer(base_layer.BaseLayer): - """Pooling layer with causal dependency on the time axis.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('pooling_type', 'AVG', 'Pooling type: MAX|AVG') - p.Define( - 'left_context', None, 'Number of frames to the left in the pooling' - 'window (including the current frame). A special value "-1" means ' - 'using all left frames') - return p - - def FProp(self, theta, inputs, paddings): - """Applies causal pooling to inputs. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: The inputs tensor. It is expected to be of shape [batch, time, - frequency, channel]. The time dimension corresponds to the height - dimension as in images and the frequency dimension corresponds to the - width dimension as in images. - paddings: The paddings tensor. It is expected to be of shape [batch, - time]. - - Returns: - outputs, out_paddings pair. - - outputs: has the same shape as inputs. - - out_paddings: has the same tshape as paddings. - """ - - p = self.params - if p.left_context == -1: - if p.pooling_type == 'AVG': - cumulative_sum = tf.math.cumsum(inputs, axis=1) - cumulative_count = 1.0 + tf.range( - py_utils.GetShape(inputs)[1], dtype=p.dtype) - cumulative_mean = cumulative_sum / cumulative_count[tf.newaxis, :, - tf.newaxis, - tf.newaxis] - cumulative_mean *= 1.0 - paddings[..., tf.newaxis, tf.newaxis] - return cumulative_mean, paddings - else: - raise NotImplementedError('Cumulative max pooling not implemented.') - - window_size = p.left_context - left_pad_size = window_size - 1 - large_negative = p.dtype.max * tf.constant(-0.7, dtype=p.dtype) - # For max pooling, use a large negative padding value such that the max - # element is almost always from a non-padding position. - pad_value = 0 if p.pooling_type == 'AVG' else large_negative - inputs = tf.pad( - inputs, [[0, 0], [left_pad_size, 0], [0, 0], [0, 0]], - constant_values=pad_value) - - out_feature = tf.nn.pool( - inputs, - window_shape=(window_size, 1), - pooling_type=p.pooling_type, - padding='VALID') - - if p.pooling_type == 'AVG': - # Count the fraction of non-padding elements inside each pooling window. - max_seq_len = py_utils.GetShape(paddings)[1] - num_non_padded_elements = tf.range(1, 1 + max_seq_len, dtype=p.dtype) - num_non_padded_elements = tf.minimum(num_non_padded_elements, - tf.cast(window_size, p.dtype)) - non_padded_ratio = num_non_padded_elements / tf.cast(window_size, p.dtype) - # Divide by non-padding ratios to eliminate the effect of padded zeros. - out_feature *= tf.math.reciprocal_no_nan(non_padded_ratio[tf.newaxis, :, - tf.newaxis, - tf.newaxis]) - out_feature *= 1.0 - paddings[..., tf.newaxis, tf.newaxis] - return out_feature, paddings - - -class Builder(builder.Base): - """Builder patterns for commonly used conv layers.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('norm_layer_tpl', - ConvBatchNormLayer.Params().Set(decay=0.999), - 'If specified, the normalization layer template.') - p.Define('weight_norm', False, 'Add weight norm for kernel weights or not.') - p.Define( - 'v2_padding', False, 'Prefer setting to True. The default ' - 'implementation is incorrect for strided convolutions.') - - return p - - def _BiasNoPadding(self, name, dims): - return super()._Bias(name, dims) - - def _Bias(self, name, dims): - """Bias layer. The bias is added to the last dimension of the input.""" - return BiasLayer.Params().Set(name=name, dims=dims) - - def _Norm(self, name, dims): - return self.params.norm_layer_tpl.Copy().Set(name=name, dim=dims) - - def _NormOrBias(self, name, dims): - if self.params.norm_layer_tpl: - return self._Norm(name, dims) - else: - return self._Bias(name, dims) - - def _MaybeNorm(self, name, dims): - if self.params.norm_layer_tpl: - return self._Norm(name, dims) - else: - return self._Id(name) - - def _Activation(self, name, activation): - return ActivationLayer.Params().Set(name=name, activation=activation) - - def _Padding(self, name): - return PaddingLayer.Params().Set(name=name) - - def _RawConv2D(self, - name, - in_dim, - out_dim, - filter_shape, - stride, - dilation, - is_causal): - if is_causal: - conv_cls = CausalConv2DLayerWithPadding - else: - conv_cls = Conv2DLayerWithPadding - return conv_cls.Params().Set( - name=name, - filter_shape=filter_shape + [in_dim, out_dim], - filter_stride=stride, - dilation_rate=dilation, - weight_norm=self.params.weight_norm, - v2_padding=self.params.v2_padding) - - def _RawDepthwiseConv2D(self, - name, - in_dim, - depth_multiplier, - filter_shape, - stride, - dilation, - is_causal): - if is_causal: - conv_cls = CausalDepthwiseConv2DLayer - else: - conv_cls = DepthwiseConv2DLayer - return conv_cls.Params().Set( - name=name, - filter_shape=filter_shape + [in_dim, depth_multiplier], - filter_stride=stride, - dilation_rate=dilation, - weight_norm=self.params.weight_norm, - v2_padding=self.params.v2_padding) - - def _GlobalPooling(self, name, pooling_type): - return GlobalPoolingLayer.Params().Set(name=name, pooling_type=pooling_type) - - def Conv2D(self, - name, - in_dim, - out_dim, - filter_shape, - stride=None, - dilation=None, - activation='RELU', - conv_last=False, - is_causal=False): - if stride is None: - stride = [1, 1] - if dilation is None: - dilation = [1, 1] - if conv_last: - layers_in_sequence = [ - self._MaybeNorm('bn', in_dim), - self._Activation('act', activation), - self._RawConv2D('conv_2d', in_dim, out_dim, filter_shape, stride, - dilation, is_causal), - self._Bias('bias', out_dim), - self._Padding('pad') - ] - else: - layers_in_sequence = [ - self._RawConv2D('conv_2d', in_dim, out_dim, filter_shape, stride, - dilation, is_causal), - self._NormOrBias('bn_or_bias', out_dim), - self._Activation('act', activation), - self._Padding('pad') - ] - return self._Seq(name, *layers_in_sequence) - - def DepthwiseConv2D(self, - name, - in_dim, - depth_multiplier, - filter_shape, - stride=None, - dilation=None, - activation='RELU', - conv_last=False, - is_causal=False): - if stride is None: - stride = [1, 1] - if dilation is None: - dilation = [1, 1] - if conv_last: - layers_in_sequence = [ - self._MaybeNorm('bn', in_dim), - self._Activation('act', activation), - self._RawDepthwiseConv2D('conv_2d', in_dim, depth_multiplier, - filter_shape, stride, dilation, is_causal), - self._Bias('bias', in_dim * depth_multiplier), - self._Padding('pad') - ] - else: - layers_in_sequence = [ - self._RawDepthwiseConv2D('conv_2d', in_dim, depth_multiplier, - filter_shape, stride, dilation, is_causal), - self._NormOrBias('bn_or_bias', in_dim * depth_multiplier), - self._Activation('act', activation), - self._Padding('pad') - ] - return self._Seq(name, *layers_in_sequence) - - def SeparableConv2D(self, - name, - in_dim, - out_dim, - depth_multiplier, - filter_shape, - stride=None, - dilation=None, - activation='RELU', - conv_last=False, - is_causal=False): - if stride is None: - stride = [1, 1] - if dilation is None: - dilation = [1, 1] - if conv_last: - layers_in_sequence = [ - self._MaybeNorm('bn', in_dim), - self._Activation('act', activation), - self._RawDepthwiseConv2D('conv_2d', in_dim, depth_multiplier, - filter_shape, stride, dilation, is_causal), - # No need to add a padding layer here as subsequent conv layer always - # properly zeros out padded nodes. - self._RawConv2D( - 'conv_1x1', - in_dim * depth_multiplier, - out_dim, - filter_shape=[1, 1], - stride=[1, 1], - dilation=[1, 1], - is_causal=False), - self._Bias('bias', out_dim), - self._Padding('pad') - ] - else: - layers_in_sequence = [ - self._RawDepthwiseConv2D('conv_2d', in_dim, depth_multiplier, - filter_shape, stride, dilation, is_causal), - # No need to add a padding layer here as subsequent conv layer always - # properly zeros out padded nodes. - self._RawConv2D( - 'conv_1x1', - in_dim * depth_multiplier, - out_dim, - filter_shape=[1, 1], - stride=[1, 1], - dilation=[1, 1], - is_causal=False), - self._NormOrBias('bn_or_bias', out_dim), - self._Activation('act', activation), - self._Padding('pad') - ] - return self._Seq(name, *layers_in_sequence) - - def NormalizedDepthwiseConv2D(self, - name, - kernel_size, - num_heads, - in_dim, - dropconnect_prob=0, - deterministic_dropout=False, - is_causal=False): - if is_causal: - conv_cls = CausalNormalizedDepthwiseConv2DLayer - else: - conv_cls = NormalizedDepthwiseConv2DLayer - return conv_cls.Params().Set( - name=name, - filter_shape=[kernel_size, 1, num_heads, 1], - weight_tiling_factor=in_dim // num_heads, - deterministic_dropout=deterministic_dropout, - params_init=py_utils.WeightInit.TruncatedGaussian( - scale=math.sqrt(2.6 / kernel_size)), # Fan-out initialization. - dropconnect_prob=dropconnect_prob, - v2_padding=self.params.v2_padding) - - def _Add(self, name, residual_weight=1.0): - return self._Fn( - name, fn=lambda x, y: x + residual_weight * y, fn_out=lambda x, y: x) - - def _ExpandDims(self, name): - return self._Fn( - name, - fn=lambda x: tf.expand_dims(x, 2), - fn_out=lambda x: tshape.Shape(x[0:2] + [1] + x[2:]), - fn_flops=lambda x: 1) - - def _Squeeze(self, name): - return self._Fn( - name, - fn=lambda x: tf.squeeze(x, 2), - fn_out=lambda x: tshape.Shape(x[0:2] + x[3:]), - fn_flops=lambda x: 1) - - def _Glu(self, name, glu_with_tanh): - - def _GLUFn(inputs): - gated_inputs, act_inputs = tf.split(inputs, 2, axis=-1) - return act_inputs * tf.sigmoid(gated_inputs) - - def _GatedTanhFn(inputs): - gated_inputs, act_inputs = tf.split(inputs, 2, axis=-1) - return tf.tanh(act_inputs) * tf.sigmoid(gated_inputs) - - fn = _GatedTanhFn if glu_with_tanh else _GLUFn - - return self._Fn( - name, - fn=fn, - fn_out=lambda x: tshape.Shape(x[:-1] + [x[-1] / 2]), - fn_flops=lambda x: 15 * x.size) - - def _LConvCommon(self, - name, - input_dim, - kernel_size, - activation='RELU', - is_causal=False, - glu_with_tanh=False, - residual_dropout_prob=0): - # pyformat: disable - return py_utils.NestedMap( - pre_conv=('i.vec->pre_conv', - self._Seq( - 'pre_conv', - self._LN('ln', input_dim), - self._Linear('linear', input_dim, input_dim * 2), - self._BiasNoPadding('bias', input_dim * 2), - self._Glu('glu', glu_with_tanh), - self._ExpandDims('expand'))), - post_conv=('post_conv->after_dropout', - self._Seq( - 'post_conv', - self._Squeeze('squeeze'), - self._Linear('linear', input_dim, input_dim), - self._BiasNoPadding('bias', input_dim), - self._Dropout( - 'dropout', keep_prob=1 - residual_dropout_prob))), - residual_add=('i.vec,after_dropout->o.vec', self._Add('add')) - ) - # pyformat: enable - - def LConv(self, - name, - input_dim, - kernel_size, - activation='RELU', - is_causal=False, - glu_with_tanh=False, - residual_dropout_prob=0): - """A lightweight convolution block as described in ... - - https://arxiv.org/abs/1901.10430 - - Reference PyTorch Implementation (L587): - https://github.com/pytorch/fairseq/blob/v0.6.2/fairseq/models/lightconv.py - - Args: - name: name of the params - input_dim: Input dimension. - kernel_size: kernel size used in the conv layer. - activation: A string, activation function used by the inner conv block. - is_causal: is causal padding or not. - glu_with_tanh: if the Gated Linear Unit should apply tanh on the - activation input. - residual_dropout_prob: Residual dropout prob. - - Returns: - A GraphLayer params with a FProp() function of signature - f(inputs, paddings) -> outputs, out_paddings - """ - sub_nmap = self._LConvCommon( - name, - input_dim, - kernel_size, - activation=activation, - is_causal=is_causal, - residual_dropout_prob=residual_dropout_prob) - conv_graph = ('pre_conv,i.paddings->post_conv,o.paddings', - self.DepthwiseConv2D( - name, - in_dim=input_dim, - depth_multiplier=1, - filter_shape=[kernel_size, 1], - activation=activation, - is_causal=is_causal)) - sub_list = [ - sub_nmap.pre_conv, conv_graph, sub_nmap.post_conv, sub_nmap.residual_add - ] - - return self._Graph( - name, - ['i'], # input NestedMap with {vec, paddings} - ['o'], # output NestedMap with {vec, paddings} - *sub_list) - - def NormalizedLConv(self, - name, - input_dim, - kernel_size, - num_heads, - activation='RELU', - is_causal=False, - glu_with_tanh=False, - residual_dropout_prob=0, - dropconnect_prob=0): - """A lightweight convolution block as described in ... - - https://arxiv.org/abs/2004.11886 - - Args: - name: name of the params - input_dim: Input dimension. - kernel_size: kernel size used in the conv layer. - num_heads: Num of heads. - activation: A string, activation function used by the inner conv block. - is_causal: is causal padding or not. - glu_with_tanh: if the Gated Linear Unit should apply tanh on the - activation input. - residual_dropout_prob: Residual dropout prob. - dropconnect_prob: attention dropout prob. - - Returns: - A GraphLayer params with a FProp() function of signature - f(inputs, paddings) -> outputs, out_paddings - """ - sub_nmap = self._LConvCommon( - name, - input_dim, - kernel_size, - activation=activation, - is_causal=is_causal, - residual_dropout_prob=residual_dropout_prob) - conv_graph = ('pre_conv,i.paddings->post_conv,o.paddings', - self.NormalizedDepthwiseConv2D( - name, - kernel_size=kernel_size, - num_heads=num_heads, - in_dim=input_dim, - dropconnect_prob=dropconnect_prob, - deterministic_dropout=self.params.deterministic_dropout, - is_causal=is_causal)) - sub_list = [ - sub_nmap.pre_conv, conv_graph, sub_nmap.post_conv, sub_nmap.residual_add - ] - - return self._Graph( - name, - ['i'], # input NestedMap with {vec, paddings} - ['o'], # output NestedMap with {vec, paddings} - *sub_list) diff --git a/lingvo/core/conv_layers_builder_test.py b/lingvo/core/conv_layers_builder_test.py deleted file mode 100644 index f71a1a2a..00000000 --- a/lingvo/core/conv_layers_builder_test.py +++ /dev/null @@ -1,616 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for conv layers builder.""" - -from absl.testing import parameterized -from lingvo import compat as tf -from lingvo.core import bn_layers -from lingvo.core import conv_layers_builder -from lingvo.core import conv_layers_with_time_padding -from lingvo.core import layers -from lingvo.core import test_utils -import numpy as np - - -class ConvPaddedLayersTest(test_utils.TestCase): - - def _ConvTestHelper(self, dilation, stride, activation, batch_norm, - weight_norm, in_dim, out_dim, filter_shape, conv_last, - causal_conv): - with self.session(use_gpu=True) as sess: - p1 = layers.Conv2DLayer.Params().Set( - name='conv_2d01', - filter_shape=filter_shape + [in_dim, out_dim], - filter_stride=stride, - dilation_rate=dilation, - activation=activation, - batch_norm=batch_norm, - weight_norm=weight_norm, - bias=not batch_norm, - conv_last=conv_last, - causal_convolution=causal_conv) - builder_params = conv_layers_builder.Builder.Params().Set( - weight_norm=weight_norm) - if batch_norm: - norm_p = conv_layers_with_time_padding.ConvBatchNormLayer.Params().Set( - decay=0.999) - builder_params.norm_layer_tpl = norm_p - else: - builder_params.norm_layer_tpl = None - p2 = builder_params.Instantiate().Conv2D( - 'conv_2d02', - in_dim, - out_dim, - filter_shape, - stride=stride, - dilation=dilation, - activation=activation, - conv_last=conv_last, - is_causal=causal_conv) - - l1 = p1.Instantiate() - l2 = p2.Instantiate() - - conv_in = tf.constant(np.random.normal(size=[4, 5, 6, 3]), tf.float32) - conv_pad = np.full([4, 5], 0.0) - conv_pad[2, 3] = 1.0 - conv_pad[2, 4] = 1.0 - conv_pad = tf.constant(conv_pad, tf.float32) - l1_theta = l1.theta.Transform(tf.identity) - l2_theta = l2.theta.Transform(tf.identity) - conv_out1, out1_padding = l1.FProp(l1_theta, conv_in, conv_pad) - conv_out2, out2_padding = l2.FProp(l2_theta, conv_in, conv_pad) - - tf.logging.info(l1_theta) - tf.logging.info(l2_theta) - l1_num_vars = l1_theta.Flatten() - l2_num_var2 = l2_theta.Flatten() - if len(l1_num_vars) != len(l2_num_var2): - tf.logging.info( - 'Mismatched number of vars: l1: %d vars, l2: %d vars', - len(l1_num_vars), len(l2_num_var2)) - - w1 = l1_theta.w - w2 = l2_theta.conv_2d.w - # b1 = l1_theta.b - # b2 = l2_theta.bn_or_bias.b - - tf.global_variables_initializer().run() - v1, p1 = sess.run([conv_out1, out1_padding]) - w1_v = sess.run(w1) - v2, p2 = sess.run([conv_out2, out2_padding], feed_dict={w2: w1_v}) - - self.assertAllClose(v1, v2) - self.assertAllClose(p1, p2) - - def testConvBasic(self): - dilation = [1, 1] - stride = [2, 3] - activation = 'NONE' - batch_norm = False - weight_norm = False - in_dim = 3 - out_dim = 3 - filter_shape = [2, 2] - conv_last = False - causal_conv = False - self._ConvTestHelper(dilation, stride, activation, batch_norm, weight_norm, - in_dim, out_dim, filter_shape, conv_last, causal_conv) - - def testConvBnWnTanh(self): - dilation = [1, 1] - stride = [2, 3] - activation = 'TANH' - batch_norm = True - weight_norm = True - in_dim = 3 - out_dim = 3 - filter_shape = [2, 2] - conv_last = False - causal_conv = False - self._ConvTestHelper(dilation, stride, activation, batch_norm, weight_norm, - in_dim, out_dim, filter_shape, conv_last, causal_conv) - - def testConvGn(self): - dilation = [1, 1] - stride = [2, 3] - activation = 'TANH' - in_dim = 3 - out_dim = 4 - filter_shape = [2, 2] - conv_last = False - causal_conv = False - - with self.session(use_gpu=True) as sess: - builder_params = conv_layers_builder.Builder.Params().Set( - weight_norm=True) - builder_params.norm_layer_tpl = bn_layers.GroupNormLayer.Params().Set( - num_groups=2) - p = builder_params.Instantiate().Conv2D( - 'conv_2d02', - in_dim, - out_dim, - filter_shape, - stride=stride, - dilation=dilation, - activation=activation, - conv_last=conv_last, - is_causal=causal_conv) - - l = p.Instantiate() - - conv_in = tf.constant(np.random.normal(size=[4, 5, 6, 3]), tf.float32) - conv_pad = np.full([4, 5], 0.0) - conv_pad[2, 3] = 1.0 - conv_pad[2, 4] = 1.0 - conv_pad = tf.constant(conv_pad, tf.float32) - conv_out, _ = l.FProp(l.theta, conv_in, conv_pad) - tf.global_variables_initializer().run() - v = sess.run(tf.reduce_sum(conv_out, 0)) - - expected_out = [[[-0.35070014, -1.7821487, 0.8349923, 1.1709788], - [-0.18872532, 0.9702145, 0.5534694, -1.1386856]], - [[0.34970748, -0.5403709, -0.9809327, -2.0930214], - [0.54232424, 1.1565661, 1.0349312, 1.3458138]], - [[0, 0, 0, 0], [0, 0, 0, 0]]] - - self.assertAllClose(v, expected_out) - - def testConvLastWnTanh(self): - dilation = [1, 1] - stride = [2, 3] - activation = 'TANH' - batch_norm = False - weight_norm = True - in_dim = 3 - out_dim = 3 - filter_shape = [2, 2] - conv_last = True - causal_conv = False - self._ConvTestHelper(dilation, stride, activation, batch_norm, weight_norm, - in_dim, out_dim, filter_shape, conv_last, causal_conv) - - def testConvLastCausal(self): - dilation = [1, 1] - stride = [2, 3] - activation = 'TANH' - batch_norm = True - weight_norm = True - in_dim = 3 - out_dim = 3 - filter_shape = [2, 1] - conv_last = True - causal_conv = True - self._ConvTestHelper(dilation, stride, activation, batch_norm, weight_norm, - in_dim, out_dim, filter_shape, conv_last, causal_conv) - - def _DepthwiseConvTestHelper(self, dilation, stride, activation, batch_norm, - weight_norm, in_dim, depth_multiplier, - filter_shape, conv_last, causal_conv): - with self.session(use_gpu=True) as sess: - p1 = layers.DepthwiseConv2DLayer.Params().Set( - name='conv_2d01', - filter_shape=filter_shape + [in_dim, depth_multiplier], - filter_stride=stride, - dilation_rate=dilation, - activation=activation, - batch_norm=batch_norm, - weight_norm=weight_norm, - bias=not batch_norm, - conv_last=conv_last, - causal_convolution=causal_conv) - builder_params = conv_layers_builder.Builder.Params().Set( - weight_norm=weight_norm) - if batch_norm: - norm_p = conv_layers_with_time_padding.ConvBatchNormLayer.Params().Set( - decay=0.999) - builder_params.norm_layer_tpl = norm_p - else: - builder_params.norm_layer_tpl = None - - p2 = builder_params.Instantiate().DepthwiseConv2D( - 'conv_2d02', - in_dim, - depth_multiplier, - filter_shape, - stride=stride, - activation=activation, - dilation=dilation, - conv_last=conv_last, - is_causal=causal_conv) - - l1 = p1.Instantiate() - l2 = p2.Instantiate() - - conv_in = tf.constant(np.random.normal(size=[4, 5, 6, 3]), tf.float32) - conv_pad = np.full([4, 5], 0.0) - conv_pad[2, 3] = 1.0 - conv_pad[2, 4] = 1.0 - conv_pad = tf.constant(conv_pad, tf.float32) - l1_theta = l1.theta.Transform(tf.identity) - l2_theta = l2.theta.Transform(tf.identity) - conv_out1, out1_padding = l1.FProp(l1_theta, conv_in, conv_pad) - conv_out2, out2_padding = l2.FProp(l2_theta, conv_in, conv_pad) - - tf.logging.info(l1_theta) - tf.logging.info(l2_theta) - l1_num_vars = l1_theta.Flatten() - l2_num_var2 = l2_theta.Flatten() - if len(l1_num_vars) != len(l2_num_var2): - tf.logging.info( - 'Mismatched number of vars: l1: %d vars, l2: %d vars', - len(l1_num_vars), len(l2_num_var2)) - - w1 = l1_theta.w - w2 = l2_theta.conv_2d.w - # b1 = l1_theta.b - # b2 = l2_theta.bn_or_bias.b - - tf.global_variables_initializer().run() - v1, p1 = sess.run([conv_out1, out1_padding]) - w1_v = sess.run([w1])[0] - v2, p2 = sess.run([conv_out2, out2_padding], feed_dict={w2: w1_v}) - - self.assertAllClose(v1, v2) - self.assertAllClose(p1, p2) - - def testDepthConvBasic(self): - dilation = [1, 1] - stride = [2, 2] - activation = 'NONE' - batch_norm = False - weight_norm = False - in_dim = 3 - depth_multiplier = 2 - filter_shape = [2, 2] - conv_last = False - causal_conv = False - self._DepthwiseConvTestHelper(dilation, stride, activation, batch_norm, - weight_norm, in_dim, depth_multiplier, - filter_shape, conv_last, causal_conv) - - def testDepthConvBnWnTanh(self): - dilation = [1, 1] - stride = [2, 2] - activation = 'TANH' - batch_norm = True - weight_norm = True - in_dim = 3 - depth_multiplier = 3 - filter_shape = [2, 2] - conv_last = False - causal_conv = False - self._DepthwiseConvTestHelper(dilation, stride, activation, batch_norm, - weight_norm, in_dim, depth_multiplier, - filter_shape, conv_last, causal_conv) - - def testDepthConvGn(self): - dilation = [1, 1] - stride = [2, 2] - activation = 'TANH' - in_dim = 4 - depth_multiplier = 1 - filter_shape = [2, 2] - conv_last = False - causal_conv = False - - with self.session(use_gpu=True) as sess: - builder_params = conv_layers_builder.Builder.Params().Set( - weight_norm=True) - builder_params.norm_layer_tpl = bn_layers.GroupNormLayer.Params().Set( - num_groups=2) - p = builder_params.Instantiate().DepthwiseConv2D( - 'conv_2d02', - in_dim, - depth_multiplier, - filter_shape, - stride=stride, - activation=activation, - dilation=dilation, - conv_last=conv_last, - is_causal=causal_conv) - l = p.Instantiate() - - conv_in = tf.constant(np.random.normal(size=[4, 5, 6, 4]), tf.float32) - conv_pad = np.full([4, 5], 0.0) - conv_pad[2, 3] = 1.0 - conv_pad[2, 4] = 1.0 - conv_pad = tf.constant(conv_pad, tf.float32) - conv_out, _ = l.FProp(l.theta, conv_in, conv_pad) - tf.global_variables_initializer().run() - v = sess.run(tf.reduce_sum(conv_out, 0)) - - expected_out = [[[-0.77095497, 0.30285388, -0.05714864, 1.0386012], - [0.74034333, 0.04982221, -0.41769135, -2.9531932], - [-0.2647084, -0.1936804, 0.6598473, 0.42537105]], - [[1.3095646, -0.85996866, 2.2734299, -1.8457952], - [-0.9542263, -0.14199251, 0.51472515, 0.91931283], - [0.47267163, 1.4824618, 0.4548889, 0.93488806]], - [[0., 0., 0., 0.], [0., 0., 0., 0.], [0., 0., 0., 0.]]] - - self.assertAllClose(expected_out, v) - - def testDepthConvLastWnTanh(self): - dilation = [1, 1] - stride = [2, 2] - activation = 'TANH' - batch_norm = False - weight_norm = True - in_dim = 3 - depth_multiplier = 3 - filter_shape = [2, 2] - conv_last = True - causal_conv = False - self._DepthwiseConvTestHelper(dilation, stride, activation, batch_norm, - weight_norm, in_dim, depth_multiplier, - filter_shape, conv_last, causal_conv) - - def testDepthConvLastCausal(self): - dilation = [1, 1] - stride = [2, 2] - activation = 'TANH' - batch_norm = True - weight_norm = True - in_dim = 3 - depth_multiplier = 3 - filter_shape = [2, 1] - conv_last = True - causal_conv = True - self._DepthwiseConvTestHelper(dilation, stride, activation, batch_norm, - weight_norm, in_dim, depth_multiplier, - filter_shape, conv_last, causal_conv) - - def _SeparableConvTestHelper(self, dilation, stride, activation, batch_norm, - weight_norm, in_dim, depth_multiplier, out_dim, - filter_shape, conv_last, causal_conv, - assert_equality=True): - with self.session(use_gpu=True) as sess: - p1 = layers.SeparableConv2DLayer.Params().Set( - name='conv_2d01', - filter_shape=filter_shape + [in_dim, out_dim], - depth_multiplier=depth_multiplier, - filter_stride=stride, - dilation_rate=dilation, - activation=activation, - batch_norm=batch_norm, - weight_norm=weight_norm, - bias=not batch_norm, - conv_last=conv_last, - causal_convolution=causal_conv) - builder_params = conv_layers_builder.Builder.Params().Set( - weight_norm=weight_norm) - if batch_norm: - norm_p = conv_layers_with_time_padding.ConvBatchNormLayer.Params().Set( - decay=0.999) - builder_params.norm_layer_tpl = norm_p - else: - builder_params.norm_layer_tpl = None - p2 = builder_params.Instantiate().SeparableConv2D( - 'conv_2d02', - in_dim, - out_dim, - depth_multiplier, - filter_shape, - stride=stride, - activation=activation, - dilation=dilation, - conv_last=conv_last, - is_causal=causal_conv) - - l1 = p1.Instantiate() - l2 = p2.Instantiate() - - conv_in = tf.constant(np.random.normal(size=[4, 5, 6, 3]), tf.float32) - conv_pad = np.full([4, 5], 0.0) - conv_pad[2, 3] = 1.0 - conv_pad[2, 4] = 1.0 - conv_pad = tf.constant(conv_pad, tf.float32) - l1_theta = l1.theta.Transform(tf.identity) - l2_theta = l2.theta.Transform(tf.identity) - conv_out1, out1_padding = l1.FProp(l1_theta, conv_in, conv_pad) - conv_out2, out2_padding = l2.FProp(l2_theta, conv_in, conv_pad) - - tf.logging.info(l1_theta) - tf.logging.info(l2_theta) - l1_num_vars = l1_theta.Flatten() - l2_num_var2 = l2_theta.Flatten() - if len(l1_num_vars) != len(l2_num_var2): - tf.logging.info( - 'Mismatched number of vars: l1: %d vars, l2: %d vars', - len(l1_num_vars), len(l2_num_var2)) - - pointwise_conv_w1 = l1_theta.w - depth_conv_w1 = l1_theta.depthwise_conv.w - pointwise_conv_w2 = l2_theta.conv_1x1.w - depth_conv_w2 = l2_theta.conv_2d.w - # b1 = l1_theta.b - # b2 = l2_theta.bn_or_bias.b - tf.global_variables_initializer().run() - v1, p1 = sess.run([conv_out1, out1_padding]) - p_w1_v, d_w1_v = sess.run([pointwise_conv_w1, depth_conv_w1]) - v2, p2 = sess.run([conv_out2, out2_padding], - feed_dict={ - pointwise_conv_w2: p_w1_v, - depth_conv_w2: d_w1_v - }) - - if assert_equality: - self.assertAllClose(v1, v2) - self.assertAllClose(p1, p2) - - def testSeparableConv2DLayerBasic(self): - dilation = [1, 1] - stride = [2, 2] - activation = 'NONE' - batch_norm = False - weight_norm = False - in_dim = 3 - depth_multiplier = 3 - out_dim = 2 - filter_shape = [2, 2] - conv_last = False - causal_conv = False - self._SeparableConvTestHelper(dilation, stride, activation, batch_norm, - weight_norm, in_dim, depth_multiplier, - out_dim, filter_shape, conv_last, causal_conv) - - def testSeparableConvWnWnTanh(self): - dilation = [1, 1] - stride = [2, 2] - activation = 'TANH' - batch_norm = False - weight_norm = True - in_dim = 3 - depth_multiplier = 3 - out_dim = 2 - filter_shape = [2, 1] - conv_last = False - causal_conv = True - self._SeparableConvTestHelper(dilation, stride, activation, batch_norm, - weight_norm, in_dim, depth_multiplier, - out_dim, filter_shape, conv_last, causal_conv) - - def testSeparableConvLastBnWnTanh(self): - dilation = [1, 1] - stride = [2, 2] - activation = 'TANH' - batch_norm = True - weight_norm = True - in_dim = 3 - depth_multiplier = 3 - out_dim = 2 - filter_shape = [2, 1] - conv_last = True - causal_conv = True - # New implementation is not equivallent to the old. - self._SeparableConvTestHelper(dilation, stride, activation, batch_norm, - weight_norm, in_dim, depth_multiplier, - out_dim, filter_shape, conv_last, causal_conv, - assert_equality=False) - - def testSeparableConvGn(self): - dilation = [1, 1] - stride = [2, 2] - activation = 'TANH' - in_dim = 4 - depth_multiplier = 1 - out_dim = 2 - filter_shape = [2, 1] - conv_last = True - causal_conv = True - - with self.session(use_gpu=True) as sess: - builder_params = conv_layers_builder.Builder.Params().Set( - weight_norm=True) - builder_params.norm_layer_tpl = bn_layers.GroupNormLayer.Params().Set( - num_groups=2) - p = builder_params.Instantiate().SeparableConv2D( - 'conv_2d02', - in_dim, - out_dim, - depth_multiplier, - filter_shape, - stride=stride, - activation=activation, - dilation=dilation, - conv_last=conv_last, - is_causal=causal_conv) - - l = p.Instantiate() - - conv_in = tf.constant(np.random.normal(size=[4, 5, 6, 4]), tf.float32) - conv_pad = np.full([4, 5], 0.0) - conv_pad[2, 3] = 1.0 - conv_pad[2, 4] = 1.0 - conv_pad = tf.constant(conv_pad, tf.float32) - conv_out, _ = l.FProp(l.theta, conv_in, conv_pad) - tf.global_variables_initializer().run() - v = sess.run(tf.reduce_sum(conv_out, 0)) - - expected_out = [[[0.00963847, -0.04019006], [0.36265337, -0.06592329], - [0.65582913, -0.1533944]], - [[0.7512939, -0.7282307], [0.96100605, -1.9509676], - [0.4639647, 0.2485837]], [[0., 0.], [0., 0.], [0., 0.]]] - - self.assertAllClose(expected_out, v) - - -class CausalPoolingLayerTest(test_utils.TestCase, parameterized.TestCase): - """Tests for CausalPoolingLayer.""" - - @parameterized.named_parameters( - { - 'testcase_name': 'max_pooling', - 'pooling_type': 'MAX', - 'left_context': 2, - 'inputs': np.array([-2, 0, 2, 4, 0, 0]), - 'input_paddings': np.array([0, 0, 0, 0, 1, 1]), - 'expected_output': np.array([-2, 0, 2, 4, 0, 0]), - 'expected_output_padding': np.array([0, 0, 0, 0, 1, 1]), - }, { - 'testcase_name': 'avg_pooling', - 'pooling_type': 'AVG', - 'left_context': 2, - 'inputs': np.array([-2, 0, 2, 4, 0, 0]), - 'input_paddings': np.array([0, 0, 0, 0, 1, 1]), - 'expected_output': np.array([-2, -1, 1, 3, 0, 0]), - 'expected_output_padding': np.array([0, 0, 0, 0, 1, 1]), - }, { - 'testcase_name': 'max_pooling_large_window', - 'pooling_type': 'MAX', - 'left_context': 10, - 'inputs': np.array([-2, 0, 2, 4, 0, 0]), - 'input_paddings': np.array([0, 0, 0, 0, 1, 1]), - 'expected_output': np.array([-2, 0, 2, 4, 0, 0]), - 'expected_output_padding': np.array([0, 0, 0, 0, 1, 1]), - }, { - 'testcase_name': 'avg_pooling_large_window', - 'pooling_type': 'AVG', - 'left_context': 10, - 'inputs': np.array([-2, 0, 2, 4, 0, 0]), - 'input_paddings': np.array([0, 0, 0, 0, 1, 1]), - 'expected_output': np.array([-2, -1, 0, 1, 0, 0]), - 'expected_output_padding': np.array([0, 0, 0, 0, 1, 1]), - }, { - 'testcase_name': 'avg_pooling_infinte_window', - 'pooling_type': 'AVG', - 'left_context': -1, - 'inputs': np.array([-2, 0, 2, 4, 0, 0]), - 'input_paddings': np.array([0, 0, 0, 0, 1, 1]), - 'expected_output': np.array([-2, -1, 0, 1, 0, 0]), - 'expected_output_padding': np.array([0, 0, 0, 0, 1, 1]), - }) - def testSimpleCase(self, pooling_type, left_context, inputs, input_paddings, - expected_output, expected_output_padding): - inputs = inputs[np.newaxis, :, np.newaxis, np.newaxis] - input_paddings = input_paddings[np.newaxis, :] - param = conv_layers_builder.CausalPoolingLayer.Params().Set( - name='test_layer', pooling_type=pooling_type, left_context=left_context) - pooling_layer = param.Instantiate() - with self.session(use_gpu=True) as sess: - inputs = tf.convert_to_tensor(inputs, dtype=tf.float32) - input_paddings = tf.convert_to_tensor(input_paddings, dtype=tf.float32) - output, output_paddings = pooling_layer.FPropDefaultTheta( - inputs, input_paddings) - tf.global_variables_initializer().run() - output_val, output_paddings_val = sess.run([output, output_paddings]) - - self.assertAllClose(expected_output, output_val.flatten()) - self.assertAllEqual(expected_output_padding, output_paddings_val.flatten()) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/conv_layers_with_time_padding.py b/lingvo/core/conv_layers_with_time_padding.py deleted file mode 100644 index 0992e9df..00000000 --- a/lingvo/core/conv_layers_with_time_padding.py +++ /dev/null @@ -1,800 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Common conv layers. - -WARNING: Strided convolutions are buggy. Consider using v2_padding=True. -""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import bn_layers -from lingvo.core import py_utils -from lingvo.core import symbolic -from lingvo.core import tshape - - -# Yummy! -def ComputeConvOutputShape(in_shape, - t_stride, - f_stride, - outc=None, - padding='SAME'): - """Computes output shape for convolution and pooling layers. - - If `in_shape` is a dynamic shape, the output will be Tensors, while if - `in_shape` is a list of ints then the output will also be a list of ints. - - Args: - in_shape: A length 4 Tensor or list representing the input shape. - t_stride: The stride along the time dimension. - f_stride: The stride along the frequency dimension. - outc: The expected output channel. If None, will use the input channel. - padding: 'SAME' or 'VALID'. - - Returns: - The expected output shape. - """ - # In the order of batch, time, frequency, channel - n = in_shape[0] - t = in_shape[1] - f = in_shape[2] - c = in_shape[3] - # Last two dimensions has to be specified. - assert f is not None and c is not None - if padding == 'VALID': - if t: - t -= t_stride - 1 - f -= f_stride - 1 - ot = t - if ot is not None: - ot = (ot + t_stride - 1) // t_stride - of = (f + f_stride - 1) // f_stride - if outc is None: - outc = c - return [n, ot, of, outc] - - -def ComputeConvOutputPadding(paddings, - window, - stride, - padding_algorithm='SAME', - v2_padding=False): - """Computes paddings for convolution and pooling output. - - WARNING: This implementation is buggy prefer using ComputeConvOutputPaddingV2. - - out_padding[i] == 1 iff any in_padding corresponding to that output is 1. - - Args: - paddings: The paddings tensor. It is expected to be of shape [batch, time]. - window: The size of the windows. - stride: The time-stride between adjacent windows. - padding_algorithm: 'SAME' or 'VALID'. - v2_padding: Prefer setting to True. The default implementation is buggy for - strided convolutions. - - Returns: - out_padding, The new padding tensor of size [batch, ceil(time / stride)]. - """ - if v2_padding: - return _ComputeConvOutputPaddingV2(paddings, window, stride, - padding_algorithm) - - if stride == 1: - return paddings - - # Pad so input_length divides stride. - input_length = py_utils.GetShape(paddings)[1] - pad_len = (input_length + stride - 1) // stride * stride - input_length - paddings = tf.pad(paddings, [[0, 0], [0, pad_len]], constant_values=1.0) - out_padding = tf.nn.pool( - tf.expand_dims(paddings, -1), - [window], - 'MAX', - padding=padding_algorithm, - strides=[stride], - ) - return tf.squeeze(out_padding, -1) - - -def _ComputeConvOutputPaddingV2(paddings, - window, - stride, - padding_algorithm='SAME'): - """Computes paddings for convolution and pooling output. - - - If padding_algorithm='SAME': out_padding[i] == 0 if the in_padding - corresponding to that output is 0. This prevents the output from shrinking - unnecessarily when striding. - - If padding algorithm='VALID': out_padding[i] == 1 iff any in_padding - corresponding to that output is 1. - - Args: - paddings: The paddings tensor. It is expected to be of shape [batch, time]. - window: The size of the windows. - stride: The time-stride between adjacent windows. - padding_algorithm: 'SAME' or 'VALID'. - - Returns: - out_padding, The new padding tensor of size [batch, ceil(time / stride)]. - """ - if stride == 1 and padding_algorithm == 'SAME': - return paddings - - paddings, slice_len = _PadForLengthCompatibleStridesV2( - paddings, stride, padding_algorithm, 1.0) - - expanded_paddings = tf.expand_dims(paddings, -1) - - if padding_algorithm == 'SAME': - # Using a strided conv1d of size 1x1 we find all non-padded positions for - # the specified stride. - out_paddings = tf.nn.conv1d( - expanded_paddings, - filters=tf.ones([1, 1, 1], paddings.dtype), - stride=stride, - padding='SAME', - name='padding_conv') - elif padding_algorithm == 'VALID': - out_paddings = tf.nn.pool( - expanded_paddings, [window], - 'MAX', - padding=padding_algorithm, - strides=[stride]) - out_paddings = tf.squeeze(out_paddings, -1) - if stride > 1: - slice_end = py_utils.GetShape(out_paddings)[1] - slice_len - out_paddings = out_paddings[:, :slice_end] - return out_paddings - - -def _PadForLengthCompatibleStridesV2(tensor, stride, padding_algorithm, - constant_values): - """Pads tensor to make strided convolutions start in the first position. - - Tensorflow strided convolutions and Lingvo paddings are incompatible. - Strided convolutions always end at the last index of the length dimension. - Therefore, the output of a Lingvo padded tensor depends on the length - dimension. Here we remove this dependency by pre-padding the tensor so that - the first convolution starts in the first position. - - Args: - tensor: The tensor to prepare for convolution. [batch, time, ...]. - stride: The stride in the length dimension. - padding_algorithm: 'SAME' or 'VALID'. - constant_values: Value to pad 0. for data tensor and 1.0 for padding tensor. - - Returns: - A tuple (tensor, padded_length) where tensor is the potentionally padded - tensor and padded_length is the number paddings. - """ - if padding_algorithm == 'VALID': - return tensor, 0 - - input_length = py_utils.GetShape(tensor)[1] - pad_len = ((input_length // stride) + 1) * stride - 1 - input_length - if pad_len == 0: - return tensor, 0 - tensor = py_utils.PadSequenceDimension(tensor, input_length + pad_len, - constant_values) - return tensor, pad_len - - -class BaseConv2DLayerWithPadding(base_layer.BaseLayer): - """Abstract base class for 2D convolution layers. - - WARNING: Strided convolutions are buggy. Prefer using v2_padding=True. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'filter_shape', (0, 0, 0, 0), - 'Filter shape. Must be a sequence of length 4. Elements are in' - ' the order of height (time), width (frequency), in_channel,' - ' out_channel. For causal convolution, filter_shape[0]' - ' is the actual number of trained weights in the time dimension' - ' of the kernel.') - p.Define( - 'filter_stride', (1, 1), - 'Filter stride to use. Must be a pair of ints. The first int' - ' specifies the stride on the time dimension. The second int' - ' specifies the stride on the frequency dimension.') - p.Define( - 'dilation_rate', (1, 1), - 'If > 1, dilation rate for atrous convolution. ' - 'Must be a pair of ints. ' - 'The first int specifies the dilation rate on the time dimension. ' - 'The second int specifies the dilation rate on the frequency ' - 'dimension. ' - 'If any value of dilation_rate is > 1, then all values of strides ' - 'must be 1.') - p.Define( - 'weight_norm', False, - 'If true, apply weight normalization to weights as proposed by' - ' Salimans and Kingma, 2016: https://arxiv.org/abs/1602.07868') - p.Define( - 'partial_conv', False, 'If true, rescale positions near sequence' - 'boundaries as proposed in https://arxiv.org/abs/1811.11718') - p.Define( - 'v2_padding', False, 'Prefer setting to True. The default ' - 'implementation is incorrect for strided convolutions.') - - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert len(p.filter_shape) == 4 - assert len(p.filter_stride) == 2 - assert all(x > 0 for x in p.filter_stride) - assert len(p.dilation_rate) == 2 - assert all(x > 0 for x in p.dilation_rate) - # Dilation and stride can't be combined. - if any(x > 1 for x in p.dilation_rate): - assert all(x == 1 for x in p.filter_stride) - - @classmethod - def OutputChannels(cls, p): - """The number of output channels for this conv layer.""" - raise NotImplementedError() - - @property - def output_channels(self): - return self.OutputChannels(self.params) - - @property - def input_channels(self): - """The number of input channels for this conv layer.""" - return self.params.filter_shape[2] - - @property - def filter_stride(self): - return self.params.filter_stride - - def OutShape(self, in_shape): - """Compute the output shape given the input shape.""" - p = self.params - return ComputeConvOutputShape(in_shape, p.filter_stride[0], - p.filter_stride[1], self.output_channels) - - def FProp(self, theta, inputs, paddings): - """Apply convolution to inputs. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: The inputs tensor. It is expected to be of shape [batch, time, - frequency, channel]. The time dimension corresponds to the height - dimension as in images and the frequency dimension corresponds to the - width dimension as in images. - paddings: The paddings tensor, expected to be of shape [batch, time]. - - Returns: - outputs, out_paddings pair. - """ - p = self.params - with tf.name_scope(p.name): - inputs = py_utils.with_dependencies([ - py_utils.assert_shape_match(tf.shape(paddings), [-1, -1]), - py_utils.assert_shape_match( - tf.shape(inputs), - tf.concat([ - tf.shape(paddings), - [-1, symbolic.ToStatic(self.input_channels)] - ], 0)) - ], inputs) - - def _ApplyPadding(tensor_in, padding_in): - padding_expanded = tf.expand_dims(tf.expand_dims(padding_in, -1), -1) - return tensor_in * (1.0 - padding_expanded) - - # Zeroing out padded inputs. - inputs = _ApplyPadding(inputs, paddings) - - # Apply conv on 'inputs'. - if p.v2_padding: - padded_inputs, slice_len = _PadForLengthCompatibleStridesV2( - inputs, p.filter_stride[0], 'SAME', 0.) - out = self._ApplyConv(theta, padded_inputs) - if p.filter_stride[0] > 1: - slice_end = py_utils.GetShape(out)[1] - slice_len - out = out[:, :slice_end, :, :] - else: - out = self._ApplyConv(theta, inputs) - - if p.partial_conv: - out = self._RescaleBoundary(out, paddings) - # NOTE: this may be slightly inaccurate when p.dilation_rate[0] > 1. - # But there's likely no real problems. Trying to set it gives an error: - # pooling with SAME padding is not implemented for dilation_rate > 1. - # implementation. Consider updating it to be the actual shape. - if p.v2_padding: - conv_padding = _ComputeConvOutputPaddingV2( - paddings, window=p.filter_shape[0], stride=p.filter_stride[0]) - else: - conv_padding = ComputeConvOutputPadding( - paddings, window=p.filter_stride[0], stride=p.filter_stride[0]) - - # Assuming padded nodes will be properly zero-ed out if necessary by - # sub-sequent layers. - # out = _ApplyPadding(out, conv_padding) - out = py_utils.HasShape( - out, symbolic.ToStatic(self.OutShape(tf.shape(inputs)))) - return out, conv_padding - - def _RescaleBoundary(self, out, in_paddings): - # Rescale every output position by: - # (# input positions) / (# non-padding input positions) - # where (# input posisions) = filter_size. - p = self.params - in_mask = 1.0 - in_paddings - - # Compute the left and right implicity padding size used in 'SAME' mode. - filter_t = p.filter_shape[0] - effective_filter_size = (filter_t - 1) * p.dilation_rate[0] + 1 - left_pad_size = (effective_filter_size - 1) // 2 - right_pad_size = effective_filter_size // 2 - - # Compute the rescaling factor. - # This expanded tensor has 1 on all valid positions, 0 on all padded ones, - # which include both explicit padding provided by 'in_padding', and implicit - # padding on boundaries. - in_mask_padded = tf.pad(in_mask, [[0, 0], [left_pad_size, right_pad_size]]) - # (# non-padding input positions) / (# input positions) - factor_inverse = tf.nn.pool( - in_mask_padded[:, :, tf.newaxis], - window_shape=(filter_t,), - pooling_type='AVG', - strides=(p.filter_stride[0],), - padding='VALID', - dilations=(p.dilation_rate[0],)) - - factor = tf.math.reciprocal_no_nan(factor_inverse) - return out * factor[..., tf.newaxis] - - def _ApplyConv(self, theta, conv_input): - return self._EvaluateConvKernel(theta, conv_input) - - def _EvaluateConvKernel(self, theta, conv_input): - """Evaluate the convolution kernel on input 'conv_input'.""" - raise NotImplementedError - - -class Conv2DLayerWithPadding(BaseConv2DLayerWithPadding): - """Conv2D layer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('bias', False, 'Whether or not to apply a bias before activation.') - return p - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - w_pc = py_utils.WeightParams( - shape=p.filter_shape, - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('w', w_pc) - if p.weight_norm: - self.CreateVariable( - 'g', - py_utils.WeightParams( - shape=[p.filter_shape[-1]], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars'])) - if p.bias: - # NOTE(jiahuiyu): bias is subject to LP regularization in this version. - self.CreateVariable( - 'b', - py_utils.WeightParams( - shape=[self.output_channels], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars'])) - - @classmethod - def OutputChannels(cls, p): - """The number of output channels for this conv layer.""" - return p.filter_shape[-1] - - def _GetWeight(self, theta): - p = self.params - if p.weight_norm: - # Normalize along the last dim (standard conv). - filter_w = tf.nn.l2_normalize(theta.w, [0, 1, 2]) * tf.reshape( - (theta.g + 1.0), [1, 1, 1, p.filter_shape[-1]]) - else: - filter_w = theta.w - return filter_w - - def _ApplyConv(self, theta, conv_input): - out = self._EvaluateConvKernel(theta, conv_input) - p = self.params - if p.bias: - out = tf.nn.bias_add(out, theta.b) - return out - - def _EvaluateConvKernel(self, theta, inputs): - """Apply convolution to inputs.""" - p = self.params - filter_w = self._GetWeight(theta) - return tf.nn.convolution( - inputs, - filter_w, - strides=p.filter_stride, - dilations=p.dilation_rate, - data_format='NHWC', - padding='SAME') - - -class CausalConv2DLayerWithPadding(Conv2DLayerWithPadding): - """2D conv layer with causal dependency on the time axis.""" - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.filter_shape[1] == 1, 'Only 1d causal convolution is supported.' - - def _EvaluateConvKernel(self, theta, inputs): - """Apply convolution to inputs.""" - p = self.params - assert p.filter_shape[1] == 1, 'Only 1D causal convolutions supported.' - # Use VALID padding and shift the inputs to the right to ensure that the - # first output only depends on the first input and so on. The output is - # the same size as the input, as if the convolution used SAME padding. - padding_algorithm = 'VALID' - # The effective spatial filter width for dilated convolutions is - # (kernel_width - 1) * dilation_rate + 1 as according to - # https://www.tensorflow.org/api_docs/python/tf/nn/convolution. - causal_pad_size = (p.filter_shape[0] - 1) * p.dilation_rate[0] - inputs = tf.pad(inputs, [[0, 0], [causal_pad_size, 0], [0, 0], [0, 0]]) - - filter_w = self._GetWeight(theta) - return tf.nn.convolution( - inputs, - filter_w, - strides=p.filter_stride, - dilations=p.dilation_rate, - data_format='NHWC', - padding=padding_algorithm) - - -class DepthwiseConv2DLayer(BaseConv2DLayerWithPadding): - """Depthwise conv 2D layer. - - paper: https://arxiv.org/abs/1610.02357 - """ - - @classmethod - def Params(cls): - p = super().Params() - # Redefine 'filter_shape' since the semantic of shape elements is different - # from regular Conv2D. - p.Delete('filter_shape') - p.Define( - 'filter_shape', (0, 0, 0, 0), - 'Filter shape. Must be a sequence of length 4. Elements are in' - ' the order of height (time), width (frequency), in_channel,' - ' channel_multipliers. ') - p.Define('bias', False, 'Whether or not to apply a bias before activation.') - return p - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - w_pc = py_utils.WeightParams( - shape=p.filter_shape, - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - - self.CreateVariable('w', w_pc) - if p.weight_norm: - self.CreateVariable( - 'g', - py_utils.WeightParams( - shape=[p.filter_shape[2], p.filter_shape[3]], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars'])) - if p.bias: - # NOTE(jiahuiyu): bias is subject to LP regularization in this version. - self.CreateVariable( - 'b', - py_utils.WeightParams( - shape=[self.output_channels], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars'])) - - @classmethod - def OutputChannels(cls, p): - """The number of output channels for this conv layer.""" - # Depthwise convolution filter shape is: - # [..., in_channels, channel_multiplier]. - return p.filter_shape[2] * p.filter_shape[3] - - def _GetWeight(self, theta): - p = self.params - if p.weight_norm: - # Normalize along the last two dims. - filter_w = tf.nn.l2_normalize(theta.w, [0, 1]) * tf.reshape( - (theta.g + 1.0), [1, 1, p.filter_shape[2], p.filter_shape[3]]) - else: - filter_w = theta.w - return filter_w - - def _ApplyConv(self, theta, conv_input): - out = self._EvaluateConvKernel(theta, conv_input) - p = self.params - if p.bias: - out = tf.nn.bias_add(out, theta.b) - return out - - def _EvaluateConvKernel(self, theta, inputs): - """Apply convolution to inputs.""" - p = self.params - filter_w = self._GetWeight(theta) - return tf.nn.depthwise_conv2d( - inputs, - filter_w, - strides=[1, p.filter_stride[0], p.filter_stride[1], 1], - dilations=p.dilation_rate, - data_format='NHWC', - padding='SAME') - - -class CausalDepthwiseConv2DLayer(DepthwiseConv2DLayer): - """Depthwise conv layer with causal dependency on the time axis.""" - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.filter_shape[1] == 1, 'Only 1d causal convolution is supported.' - - def _EvaluateConvKernel(self, theta, inputs): - """Apply convolution to inputs.""" - p = self.params - assert p.filter_shape[1] == 1, 'Only 1D causal convolutions supported.' - # Use VALID padding and shift the inputs to the right to ensure that the - # first output only depends on the first input and so on. The output is - # the same size as the input, as if the convolution used SAME padding. - padding_algorithm = 'VALID' - # The effective spatial filter width for dilated convolutions is - # (kernel_width - 1) * dilation_rate + 1 as according to - # https://www.tensorflow.org/api_docs/python/tf/nn/convolution. - causal_pad_size = (p.filter_shape[0] - 1) * p.dilation_rate[0] - inputs = tf.pad(inputs, [[0, 0], [causal_pad_size, 0], [0, 0], [0, 0]]) - filter_w = self._GetWeight(theta) - return tf.nn.depthwise_conv2d( - inputs, - filter_w, - strides=[1, p.filter_stride[0], p.filter_stride[1], 1], - dilations=p.dilation_rate, - data_format='NHWC', - padding=padding_algorithm) - - -class NormalizedDepthwiseConv2DLayer(DepthwiseConv2DLayer): - """DepthwiseConv2DLayer where weights are normalized over the time dim. - - https://arxiv.org/abs/1901.10430 - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('dropconnect_prob', 0.0, - 'Prob at which DropConnect regularization is performed.') - p.Define('deterministic_dropout', False, 'Use determnisitc dropout or not.') - p.Define('temperature', 1.0, - 'Temperature for the softmax normalization of the weights.') - p.Define('weight_tiling_factor', 1, - 'Number of times weights are tiled over the input channels.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.filter_shape[1] == 1, 'Only 1d convolution is supported.' - assert p.temperature > 0.0, 'Absolute zero temperature is not possible.' - - @classmethod - def OutputChannels(cls, p): - """The number of output channels for this conv layer.""" - # Depthwise convolution filter shape is: - # [kernel_size, 1, in_channels, channel_multiplier]. - return p.filter_shape[2] * p.filter_shape[3] * p.weight_tiling_factor - - @property - def input_channels(self): - """The number of output channels for this conv layer.""" - p = self.params - return p.filter_shape[2] * p.weight_tiling_factor - - def _GetWeight(self, theta): - p = self.params - filter_w = theta.w - - # First normalize filter_w over the temporal dimension here. - filter_w = tf.nn.softmax(filter_w / p.temperature, axis=0) - - # Add dropconnect on the weights for regularization. - if p.dropconnect_prob > 0.0 and not self.do_eval: - if p.deterministic_dropout: - filter_w = py_utils.DeterministicDropout( - filter_w, 1.0 - p.dropconnect_prob, - py_utils.GenerateStepSeedPair(p, theta.global_step)) - else: - filter_w = tf.nn.dropout( - filter_w, rate=p.dropconnect_prob, seed=p.random_seed) - - # Tie the parameters of every subsequent number of weight_tiling_factor - # channels. - filter_w = tf.tile(filter_w, [1, 1, p.weight_tiling_factor, 1]) - return filter_w - - @classmethod - def FPropMeta(cls, p, inputs, paddings): - py_utils.CheckShapes((inputs, paddings)) - b, t, f, _ = inputs - assert f == 1 - oc = p.filter_shape[2] * p.filter_shape[3] * p.weight_tiling_factor - outputs = tshape.Shape([b, t, f, oc]) - flops = b * t * f * p.filter_shape[0] * oc * 5 - return py_utils.NestedMap(flops=flops, out_shapes=(outputs, paddings)) - - -class CausalNormalizedDepthwiseConv2DLayer(NormalizedDepthwiseConv2DLayer): - """Depthwise conv layer with causal dependency on the time axis.""" - - def _EvaluateConvKernel(self, theta, inputs): - """Apply convolution to inputs.""" - # Same as CausalDepthwiseConv2DLayer. - p = self.params - assert p.filter_shape[1] == 1, 'Only 1D causal convolutions supported.' - padding_algorithm = 'VALID' - causal_pad_size = (p.filter_shape[0] - 1) * p.dilation_rate[0] - inputs = tf.pad(inputs, [[0, 0], [causal_pad_size, 0], [0, 0], [0, 0]]) - filter_w = self._GetWeight(theta) - return tf.nn.depthwise_conv2d( - inputs, - filter_w, - strides=[1, p.filter_stride[0], p.filter_stride[1], 1], - dilations=p.dilation_rate, - data_format='NHWC', - padding=padding_algorithm) - - -class ConvBatchNormLayer(bn_layers.BatchNormLayer): - """A wrapper around regular BatchNormLayer that pass around the ... - - paddings layers. - """ - - def FProp(self, theta, inputs, paddings): - paddings_expanded = tf.expand_dims(tf.expand_dims(paddings, -1), -1) - bned = super().FProp(theta, inputs, paddings_expanded) - return bned, paddings - - -class ConvCategoricalBN(bn_layers.CategoricalBN): - """A wrapper around regular CategoricalBN that pass around the ... - - paddings layers. - """ - - def FProp(self, theta, inputs, paddings, class_emb): - paddings_expanded = tf.expand_dims(tf.expand_dims(paddings, -1), -1) - bned = super().FProp(theta, inputs, paddings_expanded, class_emb) - return bned, paddings - - -# Supported activation functions. -_ACTIVATIONS = { - 'RELU': tf.nn.relu, - 'RELU6': tf.nn.relu6, - 'SIGMOID': tf.sigmoid, - 'TANH': tf.tanh, - 'SWISH': tf.nn.swish, - 'NONE': tf.identity, -} - - -class ActivationLayer(base_layer.BaseLayer): - """Applies activation function to the inputs.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('activation', 'RELU', 'The activation function to apply') - return p - - def FProp(self, theta, inputs, paddings): - p = self.params - out = _ACTIVATIONS[p.activation](inputs) - return out, paddings - - -class PaddingLayer(base_layer.BaseLayer): - """Zeros out padded positions.""" - - def FProp(self, theta, inputs, paddings): - paddings_expanded = tf.expand_dims(tf.expand_dims(paddings, -1), -1) - return inputs * (1.0 - paddings_expanded), paddings - - -class GlobalPoolingLayer(base_layer.BaseLayer): - """Padding aware global pooling.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('pooling_type', 'MAX', 'Pooling type: MAX|AVG') - return p - - def FProp(self, theta, inputs, paddings): - """Apply global spatial pooling to inputs. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: The inputs tensor. It is expected to be of shape [batch, time, - frequency, channel]. The time dimension corresponds to the height - dimension as in images and the frequency dimension corresponds to the - width dimension as in images. - paddings: The paddings tensor. It is expected to be of shape [batch, - time]. Defaults to None, which means there no paddings. - - Returns: - outputs, out_paddings pair. - - outputs: has shape [batch, 1, 1, channel]. - - out_paddings: None or has shape [batch, 1]. - """ - p = self.params - assert p.pooling_type in ['MAX', 'AVG'], p.pooling_type - b, t, f = py_utils.GetShape(inputs, ndims=3) - - if paddings is not None: - paddings = py_utils.HasShape(paddings, [b, t]) - - if paddings is not None: - mask = 1.0 - paddings[..., tf.newaxis, tf.newaxis] - else: - mask = tf.ones([b, t, 1, 1], p.dtype) - if p.pooling_type == 'AVG': - global_sum = tf.reduce_sum(inputs * mask, axis=[1, 2], keepdims=True) - f = tf.cast(tf.convert_to_tensor(f), p.dtype) - count = f * tf.reduce_sum(mask, axis=[1, 2], keepdims=True) - out_feature = global_sum / tf.maximum(1.0, count) - elif p.pooling_type == 'MAX': - large_negative = ( - tf.ones_like(inputs) * p.dtype.max * tf.constant(-0.7, dtype=p.dtype)) - padded_inputs = tf.where_v2(mask > 0.0, inputs, large_negative) - out_feature = tf.reduce_max(padded_inputs, axis=[1, 2], keepdims=True) - if paddings is None: - out_paddings = None - else: - out_paddings = tf.reduce_min(paddings, axis=1, keepdims=True) - out_feature *= 1.0 - out_paddings[..., tf.newaxis, tf.newaxis] - return out_feature, out_paddings diff --git a/lingvo/core/conv_layers_with_time_padding_test.py b/lingvo/core/conv_layers_with_time_padding_test.py deleted file mode 100644 index 70d384b4..00000000 --- a/lingvo/core/conv_layers_with_time_padding_test.py +++ /dev/null @@ -1,723 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lingvo.core.conv_layers.""" - -from absl.testing import parameterized -import lingvo.compat as tf -from lingvo.core import conv_layers_with_time_padding as conv_layers -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.core import tshape -import numpy as np - - -class ConvLayerTest(parameterized.TestCase, test_utils.TestCase): - """Tests conv layers. - - Note that there are multiple subclasses of BaseConv2DLayer and most cases - are tested via the concrete Conv2DLayer. Other tests are done against - other subclasses to cover key differences. - """ - - def testConv2DLayerConstruction(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = conv_layers.Conv2DLayerWithPadding.Params() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 32] - params.filter_stride = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - _ = params.Instantiate() - conv_vars = tf.get_collection('Conv2DLayerWithPadding_vars') - conv_var_names = [x.name for x in conv_vars] - expected_var_names = ['conv/w/var:0'] - self.assertEqual(expected_var_names, conv_var_names) - - def testConv2DLayerWithPaddingOutputChannels(self): - with self.session(): - params = conv_layers.Conv2DLayerWithPadding.Params() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 32] - actual_output_channels = params.cls.OutputChannels(params) - self.assertEqual(32, actual_output_channels) - - @parameterized.parameters(False, True) - def testConv2DLayerOutShape(self, v2_padding): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = conv_layers.Conv2DLayerWithPadding.Params() - params.v2_padding = v2_padding - params.name = 'conv' - params.filter_shape = [3, 3, 3, 32] - params.filter_stride = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - conv_layer = params.Instantiate() - in_shape = [None, None, 10, 3] - out_shape = conv_layer.OutShape(in_shape) - self.assertEqual(out_shape, [None, None, 5, 32]) - in_shape = [None, 20, 10, 3] - out_shape = conv_layer.OutShape(in_shape) - self.assertEqual(out_shape, [None, 10, 5, 32]) - - # ComputeConvOutputPadding is broken for strided convolutions. Below we mark - # cases that are in correct with "Bug". testComputeConvOutputPaddingV2 below - # has the same test cases but without the bugs. - @parameterized.parameters( - ([0, 0, 0, 1], [0, 0, 0, 1], 1, 'SAME'), - ([0, 0, 0, 0], [0, 0], 2, 'SAME'), - ([0, 0, 0, 1], [0, 1], 2, 'SAME'), - ([0, 0, 1, 1], [1, 1], 2, 'SAME'), # Bug - ([0, 0, 0, 0, 0], [0, 0, 1], 2, 'SAME'), # Bug - ([0, 0, 0, 0, 1], [0, 1, 1], 2, 'SAME'), # Bug - ([0, 0, 0, 1, 1], [0, 1, 1], 2, 'SAME'), # Bug - ([0, 0, 1, 1, 1], [1, 1, 1], 2, 'SAME'), # Bug - ([0, 0, 0, 0, 0, 0], [0, 0, 0], 2, 'SAME'), - ([0, 0, 0, 0, 0, 1], [0, 0, 1], 2, 'SAME'), - ([0, 0, 0, 0, 1, 1], [0, 1, 1], 2, 'SAME'), # Bug - ([0, 0, 0, 1, 1, 1], [0, 1, 1], 2, 'SAME'), - ([0, 0, 1, 1, 1, 1], [1, 1, 1], 2, 'SAME'), - ([0, 0, 0, 0], [0, 0, 0, 0], 1, 'VALID'), # Bug - ([0, 0, 0, 1], [0, 0, 0, 1], 1, 'VALID'), # Bug - ([0, 0, 0, 0], [0], 2, 'VALID'), - ([0, 0, 0, 1], [0], 2, 'VALID'), - ([0, 0, 1, 1], [1], 2, 'VALID'), - ([0, 0, 0, 0, 0], [0, 0], 2, 'VALID'), - ([0, 0, 0, 0, 1], [0, 1], 2, 'VALID'), - ([0, 0, 0, 1, 1], [0, 1], 2, 'VALID'), - ([0, 0, 1, 1, 1], [1, 1], 2, 'VALID'), - ([0, 0, 0, 0, 0, 0], [0, 0], 2, 'VALID'), - ([0, 0, 0, 0, 0, 1], [0, 0], 2, 'VALID'), - ([0, 0, 0, 0, 1, 1], [0, 1], 2, 'VALID'), - ([0, 0, 0, 1, 1, 1], [0, 1], 2, 'VALID'), - ([0, 0, 1, 1, 1, 1], [1, 1], 2, 'VALID'), - ) - def testComputeConvOutputPadding(self, padding, expected_padding, stride, - padding_algorithm): - """Tests padding behavior. There are multiple bugs in the implementation.""" - padding = tf.constant([padding], tf.float32) - expected_padding = tf.constant([expected_padding], tf.float32) - with self.session(use_gpu=True): - conv_padding = conv_layers.ComputeConvOutputPadding( - padding, window=3, stride=stride, padding_algorithm=padding_algorithm) - self.evaluate(tf.global_variables_initializer()) - conv_padding = py_utils.Debug(conv_padding) - conv_padding = self.evaluate(conv_padding) - tf.logging.info('expected_padding {expected_padding}') - self.assertAllClose(expected_padding, conv_padding) - - @parameterized.parameters( - ([0, 0, 0, 1], [0, 0, 0, 1], 1, 'SAME'), - ([0, 0, 0, 0], [0, 0], 2, 'SAME'), - ([0, 0, 0, 1], [0, 0], 2, 'SAME'), - ([0, 0, 1, 1], [0, 1], 2, 'SAME'), - ([0, 0, 0, 0, 0], [0, 0, 0], 2, 'SAME'), - ([0, 0, 0, 0, 1], [0, 0, 1], 2, 'SAME'), - ([0, 0, 0, 1, 1], [0, 0, 1], 2, 'SAME'), - ([0, 0, 1, 1, 1], [0, 1, 1], 2, 'SAME'), - ([0, 0, 0, 0, 0, 0], [0, 0, 0], 2, 'SAME'), - ([0, 0, 0, 0, 0, 1], [0, 0, 0], 2, 'SAME'), - ([0, 0, 0, 0, 1, 1], [0, 0, 1], 2, 'SAME'), - ([0, 0, 0, 1, 1, 1], [0, 0, 1], 2, 'SAME'), - ([0, 0, 1, 1, 1, 1], [0, 1, 1], 2, 'SAME'), - ([0, 0, 0, 0], [0, 0], 1, 'VALID'), - ([0, 0, 0, 1], [0, 1], 1, 'VALID'), - ([0, 0, 0, 0], [0], 2, 'VALID'), - ([0, 0, 0, 1], [0], 2, 'VALID'), - ([0, 0, 1, 1], [1], 2, 'VALID'), - ([0, 0, 0, 0, 0], [0, 0], 2, 'VALID'), - ([0, 0, 0, 0, 1], [0, 1], 2, 'VALID'), - ([0, 0, 0, 1, 1], [0, 1], 2, 'VALID'), - ([0, 0, 1, 1, 1], [1, 1], 2, 'VALID'), - ([0, 0, 0, 0, 0, 0], [0, 0], 2, 'VALID'), - ([0, 0, 0, 0, 0, 1], [0, 0], 2, 'VALID'), - ([0, 0, 0, 0, 1, 1], [0, 1], 2, 'VALID'), - ([0, 0, 0, 1, 1, 1], [0, 1], 2, 'VALID'), - ([0, 0, 1, 1, 1, 1], [1, 1], 2, 'VALID'), - ) - def testComputeConvOutputPaddingV2(self, padding, expected_padding, stride, - padding_algorithm): - """Test Convolution padding computation.""" - padding = tf.constant([padding], tf.float32) - expected_padding = tf.constant([expected_padding], tf.float32) - with self.session(use_gpu=True): - conv_padding = conv_layers._ComputeConvOutputPaddingV2( - padding, window=3, stride=stride, padding_algorithm=padding_algorithm) - self.evaluate(tf.global_variables_initializer()) - conv_padding = py_utils.Debug(conv_padding) - conv_padding = self.evaluate(conv_padding) - tf.logging.info('expected_padding {expected_padding}') - self.assertAllClose(expected_padding, conv_padding) - - @parameterized.parameters(5, 6) - def testConv2DLayerStridedWithPaddingFProp(self, seq_len): - """Check strided convs get the same values for different length dim.""" - # TODO(isaace): THIS TEST SHOWS THAT THERE IS A BUG IN THE CODE. - with self.session(use_gpu=True): - batch_size = 3 - expected_seq_len = 3 - - params = conv_layers.Conv2DLayerWithPadding.Params() - params.weight_norm = False - params.filter_stride = [2, 2] - params.name = 'conv' - params.filter_shape = [3, 3, 1, 1] - params.params_init = py_utils.WeightInit.Constant(1.0) - conv_layer = params.Instantiate() - - # Set up the padding for the sequence length. (starting at 5). - in_padding = tf.constant([ - [0, 0, 0, 0, 0], - [0, 0, 0, 0, 1], - [0, 0, 0, 1, 1], - ], tf.float32) - in_padding = tf.pad( - in_padding, [[0, 0], [0, seq_len - 5]], constant_values=1.0) - - inputs = 1.0 + tf.tile( - tf.reshape(tf.range(seq_len, dtype=tf.float32), [1, seq_len, 1, 1]), - [batch_size, 1, 3, 1]) - inputs = py_utils.ApplyPadding( - tf.reshape(in_padding, [batch_size, seq_len, 1, 1]), inputs) - - inputs = py_utils.Debug(inputs) - - output, out_padding = conv_layer.FPropDefaultTheta(inputs, in_padding) - - output = py_utils.Debug(output) - out_padding = py_utils.Debug(out_padding) - - self.evaluate(tf.global_variables_initializer()) - output, out_padding = self.evaluate([output, out_padding]) - - self.assertEqual((batch_size, expected_seq_len, 2, 1), output.shape) - self.assertAllClose([ - [0, 0, 1], - [0, 0, 1], - [0, 1, 1], - ], out_padding) - - # This here shows a bug in the implementation; the output should be the - # same. Also there are bugs with the output not having the correct - # padding. - if seq_len == 5: - self.assertAllClose([ - [[[6], [6]], [[18], [18]], [[18], [18]]], - [[[6], [6]], [[18], [18]], [[8], [8]]], - [[[6], [6]], [[10], [10]], [[0], [0]]], - ], output) - elif seq_len == 6: - self.assertAllClose([ - [[[12], [12]], [[24], [24]], [[10], [10]]], - [[[12], [12]], [[14], [14]], [[0], [0]]], - [[[12], [12]], [[6], [6]], [[0], [0]]], - ], output) - else: - raise ValueError('Test does not handle length {seq_len}') - - @parameterized.parameters(5, 6) - def testConv2DLayerStridedWithPaddingFPropV2(self, seq_len): - """Check strided convs get the same values for different seq_len.""" - with self.session(use_gpu=True): - batch_size = 3 - expected_seq_len = 3 - - params = conv_layers.Conv2DLayerWithPadding.Params() - params.v2_padding = True - params.weight_norm = False - params.filter_stride = [2, 2] - params.name = 'conv' - params.filter_shape = [3, 3, 1, 1] - params.params_init = py_utils.WeightInit.Constant(1.0) - conv_layer = params.Instantiate() - - # Set up the padding for the sequence length. (starting at 5). - in_padding = tf.constant([ - [0, 0, 0, 0, 0], - [0, 0, 0, 0, 1], - [0, 0, 0, 1, 1], - ], tf.float32) - in_padding = tf.pad( - in_padding, [[0, 0], [0, seq_len - 5]], constant_values=1.0) - - inputs = 1.0 + tf.tile( - tf.reshape(tf.range(seq_len, dtype=tf.float32), [1, seq_len, 1, 1]), - [batch_size, 1, 3, 1]) - inputs = py_utils.ApplyPadding( - tf.reshape(in_padding, [batch_size, seq_len, 1, 1]), inputs) - - inputs = py_utils.Debug(inputs) - - output, out_padding = conv_layer.FPropDefaultTheta(inputs, in_padding) - - output = py_utils.Debug(output) - out_padding = py_utils.Debug(out_padding) - - self.evaluate(tf.global_variables_initializer()) - output, out_padding = self.evaluate([output, out_padding]) - - self.assertEqual((batch_size, expected_seq_len, 2, 1), output.shape) - self.assertAllClose([ - [0, 0, 0], - [0, 0, 1], - [0, 0, 1], - ], out_padding) - - # Explanation of some computations (0s are padded) - # 6 = (1*0 + 1*0 + 1*0) + (1*0 + 1*1 + 1*1) + (1*0 + 1*2 + 1*2) - # 18 = (1*1 + 1*1 + 1*0) + (1*3 + 1*3 + 1*0) + (1*5 + 1*5 + 1*0) - self.assertAllClose( - [ - [[[6], [6]], [[18], [18]], [[18], [18]]], - [[[6], [6]], [[18], [18]], [[8], [8]]], # NOTE: Not padded. - [[[6], [6]], [[10], [10]], [[0], [0]]], - ], - output) - - def testConv2DLayerWithPaddingFPropRandom(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - - params = conv_layers.Conv2DLayerWithPadding.Params() - params.weight_norm = True - params.filter_stride = [2, 2] - params.name = 'conv' - params.filter_shape = [3, 3, 3, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - conv_layer = params.Instantiate() - in_padding1 = tf.zeros([2, 4], dtype=tf.float32) - inputs1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 4, 3]), dtype=tf.float32) - output, _ = conv_layer.FPropDefaultTheta(inputs1, in_padding1) - out_sum = tf.reduce_sum(output) - out_sum_squared = tf.reduce_sum(output * output) - self.evaluate(tf.global_variables_initializer()) - v1, v2 = self.evaluate([out_sum, out_sum_squared]) - tf.logging.info('actual = %f, %f', v1, v2) - self.assertAllClose([-0.293671, 4.198602], [v1, v2]) - - @parameterized.parameters(5, 6) - def testCausalConv2DLayerStridedWithPaddingFProp(self, seq_len): - """Check strided convs get the same values for different length dim.""" - # TODO(isaace): THIS TEST SHOWS THAT THERE IS A BUG WITH PADDING - with self.session(use_gpu=True): - batch_size = 5 - expected_seq_len = 3 - - params = conv_layers.CausalConv2DLayerWithPadding.Params() - params.weight_norm = False - params.filter_stride = [2, 2] - params.name = 'conv' - params.filter_shape = [3, 1, 1, 1] - params.params_init = py_utils.WeightInit.Constant(1.0) - conv_layer = params.Instantiate() - - # Set up the padding for the sequence length. (starting at 5). - in_padding = tf.constant([ - [0, 0, 0, 0, 0], - [0, 0, 0, 0, 1], - [0, 0, 0, 1, 1], - [0, 0, 1, 1, 1], - [0, 1, 1, 1, 1], - ], tf.float32) - in_padding = tf.pad( - in_padding, [[0, 0], [0, seq_len - 5]], constant_values=1.0) - - inputs = 1.0 + tf.tile( - tf.reshape(tf.range(seq_len, dtype=tf.float32), [1, seq_len, 1, 1]), - [batch_size, 1, 3, 1]) - inputs = py_utils.ApplyPadding( - tf.reshape(in_padding, [batch_size, seq_len, 1, 1]), inputs) - - inputs = py_utils.Debug(inputs) - - output, out_padding = conv_layer.FPropDefaultTheta(inputs, in_padding) - - output = py_utils.Debug(output) - out_padding = py_utils.Debug(out_padding) - - self.evaluate(tf.global_variables_initializer()) - output, out_padding = self.evaluate([output, out_padding]) - - self.assertEqual((batch_size, expected_seq_len, 2, 1), output.shape) - self.assertAllClose([ - [0, 0, 1], - [0, 0, 1], - [0, 1, 1], - [0, 1, 1], - [1, 1, 1], - ], out_padding) - - # NOTE: There is a bug in the output not being padded correctly. - self.assertAllClose([ - [[[1], [1]], [[6], [6]], [[12], [12]]], - [[[1], [1]], [[6], [6]], [[7], [7]]], - [[[1], [1]], [[6], [6]], [[3], [3]]], - [[[1], [1]], [[3], [3]], [[0], [0]]], - [[[1], [1]], [[1], [1]], [[0], [0]]], - ], output) - - @parameterized.parameters(5, 6) - def testCausalConv2DLayerStridedWithPaddingFPropV2(self, seq_len): - """Check strided convs get the same values for different length dim.""" - with self.session(use_gpu=True): - batch_size = 5 - expected_seq_len = 3 - - params = conv_layers.CausalConv2DLayerWithPadding.Params() - params.v2_padding = True - params.weight_norm = False - params.filter_stride = [2, 2] - params.name = 'conv' - params.filter_shape = [3, 1, 1, 1] - params.params_init = py_utils.WeightInit.Constant(1.0) - conv_layer = params.Instantiate() - - # Set up the padding for the sequence length. (starting at 5). - in_padding = tf.constant([ - [0, 0, 0, 0, 0], - [0, 0, 0, 0, 1], - [0, 0, 0, 1, 1], - [0, 0, 1, 1, 1], - [0, 1, 1, 1, 1], - ], tf.float32) - in_padding = tf.pad( - in_padding, [[0, 0], [0, seq_len - 5]], constant_values=1.0) - - inputs = 1.0 + tf.tile( - tf.reshape(tf.range(seq_len, dtype=tf.float32), [1, seq_len, 1, 1]), - [batch_size, 1, 3, 1]) - inputs = py_utils.ApplyPadding( - tf.reshape(in_padding, [batch_size, seq_len, 1, 1]), inputs) - - inputs = py_utils.Debug(inputs) - - output, out_padding = conv_layer.FPropDefaultTheta(inputs, in_padding) - - output = py_utils.Debug(output) - out_padding = py_utils.Debug(out_padding) - - self.evaluate(tf.global_variables_initializer()) - output, out_padding = self.evaluate([output, out_padding]) - - self.assertEqual((batch_size, expected_seq_len, 2, 1), output.shape) - self.assertAllClose([ - [0, 0, 0], - [0, 0, 1], - [0, 0, 1], - [0, 1, 1], - [0, 1, 1], - ], out_padding) - - self.assertAllClose( - [ - [[[1], [1]], [[6], [6]], [[12], [12]]], - [[[1], [1]], [[6], [6]], [[7], [7]]], - [[[1], [1]], [[6], [6]], [[3], [3]]], # NOTE: not padded. - [[[1], [1]], [[3], [3]], [[0], [0]]], - [[[1], [1]], [[1], [1]], [[0], [0]]], - ], - output) - - def testCausalConv2DLayerWithPaddingFPropRandom(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - - params = (conv_layers.CausalConv2DLayerWithPadding.Params()) - params.weight_norm = True - params.filter_stride = [2, 2] - params.name = 'conv' - params.filter_shape = [2, 1, 3, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - conv_layer = params.Instantiate() - in_padding1 = tf.zeros([2, 4], dtype=tf.float32) - inputs1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 4, 3]), dtype=tf.float32) - output, _ = conv_layer.FPropDefaultTheta(inputs1, in_padding1) - self.evaluate(tf.global_variables_initializer()) - out_sum = tf.reduce_sum(output) - out_sum_squared = tf.reduce_sum(output * output) - self.evaluate(tf.global_variables_initializer()) - v1, v2 = self.evaluate([out_sum, out_sum_squared]) - tf.logging.info('actual = %f, %f', v1, v2) - self.assertAllClose([-3.584711, 3.324082], [v1, v2]) - - def testDepthwiseConv2DLayerOutputChannels(self): - with self.session(): - params = conv_layers.DepthwiseConv2DLayer.Params() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 2] - params.bias = True - actual_output_channels = params.cls.OutputChannels(params) - self.assertEqual(6, actual_output_channels) - - def testDepthwiseConv2DLayerFProp(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - - params = conv_layers.DepthwiseConv2DLayer.Params() - params.weight_norm = True - params.filter_stride = [2, 2] - params.name = 'conv' - params.filter_shape = [3, 3, 3, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - conv_layer = params.Instantiate() - in_padding1 = tf.zeros([2, 4], dtype=tf.float32) - inputs1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 4, 3]), dtype=tf.float32) - output, _ = conv_layer.FPropDefaultTheta(inputs1, in_padding1) - self.evaluate(tf.global_variables_initializer()) - out_sum = tf.reduce_sum(output) - out_sum_squared = tf.reduce_sum(output * output) - self.evaluate(tf.global_variables_initializer()) - v1, v2 = self.evaluate([out_sum, out_sum_squared]) - tf.logging.info('actual = %f, %f', v1, v2) - self.assertAllClose([-1.455162, 6.813269], [v1, v2]) - - def testCausalDepthwiseConv2DLayer(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - - params = conv_layers.CausalDepthwiseConv2DLayer.Params() - params.weight_norm = True - params.filter_stride = [2, 2] - params.name = 'conv' - params.filter_shape = [2, 1, 3, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - conv_layer = params.Instantiate() - in_padding1 = tf.zeros([2, 4], dtype=tf.float32) - inputs1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 4, 3]), dtype=tf.float32) - output, _ = conv_layer.FPropDefaultTheta(inputs1, in_padding1) - self.evaluate(tf.global_variables_initializer()) - self.evaluate(tf.global_variables_initializer()) - out_sum = tf.reduce_sum(output) - out_sum_squared = tf.reduce_sum(output * output) - self.evaluate(tf.global_variables_initializer()) - v1, v2 = self.evaluate([out_sum, out_sum_squared]) - tf.logging.info('actual = %f, %f', v1, v2) - self.assertAllClose([-2.031689, 7.911201], [v1, v2]) - - def testActivationLayer(self): - with self.session(use_gpu=True): - p = conv_layers.ActivationLayer.Params() - p.name = 'act' - l = p.Instantiate() - inputs = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 4, 3]), dtype=tf.float32) - in_padding = tf.zeros([2, 4], dtype=tf.float32) - out, out_padding = l.FProp(l.theta, inputs, in_padding) - self.evaluate(tf.global_variables_initializer()) - v1, v2 = self.evaluate([out, out_padding]) - print(v1, v2) - - def _testNormalizedDepthwiseConv2DHelper(self, - is_causal=False, - dropconnect_prob=0): - if is_causal: - conv_cls = (conv_layers.CausalNormalizedDepthwiseConv2DLayer) - else: - conv_cls = conv_layers.NormalizedDepthwiseConv2DLayer - tf.random.set_seed(398847392) - np.random.seed(12345) - params = conv_cls.Params().Set( - name='conv', - weight_tiling_factor=2, - filter_shape=[3, 1, 2, 1], - dropconnect_prob=dropconnect_prob, - deterministic_dropout=True) - conv_layer = params.Instantiate() - in_padding = tf.zeros([2, 4], dtype=tf.float32) - inputs = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 1, 4]), dtype=tf.float32) - output, _ = conv_layer.FPropDefaultTheta(inputs, in_padding) - return output - - def testNormalizedDepthwiseConv2DLayerOutputChannels(self): - with self.session(): - params = (conv_layers.NormalizedDepthwiseConv2DLayer.Params()) - params.name = 'conv' - params.filter_shape = [3, 1, 2, 1] - params.weight_tiling_factor = 2 - actual_output_channels = params.cls.OutputChannels(params) - self.assertEqual(4, actual_output_channels) - - def testNormalizedDepthwiseConv2DLayerFPropMeta(self): - params = (conv_layers.NormalizedDepthwiseConv2DLayer.Params()) - params.name = 'conv' - params.filter_shape = [3, 1, 2, 1] - params.weight_tiling_factor = 2 - batch, time, frequency, in_channel = 2, 4, 1, 4 - output_channels = 4 - inputs_shape = tshape.Shape([batch, time, frequency, in_channel]) - paddings_shape = tshape.Shape([batch, time]) - with self.session(): - out = params.cls.FPropMeta(params, inputs_shape, paddings_shape) - expected_flops = batch * time * frequency * params.filter_shape[ - 0] * output_channels * 5 - self.assertEqual(expected_flops, out.flops) - out_shapes = out.out_shapes - self.assertEqual(out_shapes[0].ToTensorShape().as_list(), - [batch, time, frequency, output_channels]) - self.assertEqual(out_shapes[1].ToTensorShape().as_list(), [batch, time]) - - def testNormalizedDepthwiseConv2DLayerFProp(self): - expected_output = [[0.91136134, 1.25781929, 1.76708317, 0.9021343], - [0.52296412, 0.7703352, 0.65711987, 0.23177178]] - with self.session(use_gpu=True): - output = self._testNormalizedDepthwiseConv2DHelper() - output_sum = tf.squeeze(tf.reduce_sum(output, -1)) - self.evaluate(tf.global_variables_initializer()) - output_sum_val = self.evaluate(output_sum) - self.assertAllClose(expected_output, output_sum_val) - - def testCausalNormalizedDepthwiseConv2DLayerFProp(self): - expected_output = [[0.00819603, 0.91136134, 1.25781929, 1.76708317], - [-0.07673456, 0.52296412, 0.7703352, 0.65711987]] - with self.session(use_gpu=True): - output = self._testNormalizedDepthwiseConv2DHelper(is_causal=True) - output_sum = tf.squeeze(tf.reduce_sum(output, -1)) - self.evaluate(tf.global_variables_initializer()) - output_sum_val = self.evaluate(output_sum) - self.assertAllClose(expected_output, output_sum_val) - - def testNormalizedDepthwiseConv2DLayerBackProp(self): - with self.session(use_gpu=True) as sess: - output = self._testNormalizedDepthwiseConv2DHelper(dropconnect_prob=0.1) - loss = tf.reduce_sum(output) - all_vars = tf.trainable_variables() - grads = tf.gradients(loss, all_vars) - self.evaluate(tf.global_variables_initializer()) - sym_grads = [sg.eval() for sg in grads] - num_grads = [ - test_utils.ComputeNumericGradient(sess, loss, v) for v in all_vars - ] - for sg, ng in zip(sym_grads, num_grads): - self.assertAllClose(sg, ng, rtol=1e-02, atol=1e-02) - - def testCausualNormalizedDepthwiseConv2DLayerBackProp(self): - with self.session(use_gpu=True) as sess: - output = self._testNormalizedDepthwiseConv2DHelper( - is_causal=True, dropconnect_prob=0.1) - loss = tf.reduce_sum(output) - all_vars = tf.trainable_variables() - grads = tf.gradients(loss, all_vars) - self.evaluate(tf.global_variables_initializer()) - sym_grads = [sg.eval() for sg in grads] - num_grads = [ - test_utils.ComputeNumericGradient(sess, loss, v) for v in all_vars - ] - for sg, ng in zip(sym_grads, num_grads): - self.assertAllClose(sg, ng, rtol=1e-02, atol=1e-02) - - -class GlobalPoolingLayerTest(test_utils.TestCase): - """Tests for GlobalPoolingLayer.""" - - def _testHelper(self, - pooling_type, - inputs, - input_paddings, - expected_output, - expected_output_padding, - feed_dict=None): - param = conv_layers.GlobalPoolingLayer.Params().Set( - name='test_layer', pooling_type=pooling_type) - pooling_layer = param.Instantiate() - with self.session(use_gpu=True) as sess: - inputs = tf.convert_to_tensor(inputs, dtype=tf.float32) - input_paddings = None if input_paddings is None else tf.convert_to_tensor( - input_paddings, dtype=tf.float32) - output, output_paddings = pooling_layer.FPropDefaultTheta( - inputs, input_paddings) - self.evaluate(tf.global_variables_initializer()) - if input_paddings is None: - self.assertIsNone(output_paddings) - output_val = sess.run(output, feed_dict=feed_dict) - else: - output_val, output_paddings_val = sess.run([output, output_paddings], - feed_dict=feed_dict) - - self.assertAllClose(expected_output, output_val) - if input_paddings is not None: - self.assertAllEqual(expected_output_padding, output_paddings_val) - - def testPooling(self): - inputs = np.random.random([3, 5, 2, 4]) - 0.5 - expected_avg_output = np.mean(inputs, axis=(1, 2), keepdims=True) - expected_max_output = np.amax(inputs, axis=(1, 2), keepdims=True) - self._testHelper('AVG', inputs, None, expected_avg_output, None) - self._testHelper('MAX', inputs, None, expected_max_output, None) - - def testPoolingWithPadding(self): - inputs = np.random.random([4, 3, 2, 4]) - 0.5 - paddings = np.array([[0, 0, 0], [0, 0, 1], [0, 1, 1], [1, 1, 1]]) - expected_paddings = np.array([[0], [0], [0], [1]]) - expected_avg_output = np.array([ - np.mean(inputs[0][:3], axis=(0, 1), keepdims=True), - np.mean(inputs[1][:2], axis=(0, 1), keepdims=True), - np.mean(inputs[2][:1], axis=(0, 1), keepdims=True), - np.zeros((1, 1, 4)) - ]) - expected_max_output = np.array([ - np.amax(inputs[0][:3], axis=(0, 1), keepdims=True), - np.amax(inputs[1][:2], axis=(0, 1), keepdims=True), - np.amax(inputs[2][:1], axis=(0, 1), keepdims=True), - np.zeros((1, 1, 4)) - ]) - - self._testHelper('AVG', inputs, paddings, expected_avg_output, - expected_paddings) - self._testHelper('MAX', inputs, paddings, expected_max_output, - expected_paddings) - - def testPoolingWithUnknowShapeInput(self): - """Tests GlobalPooling layer with unknown shape tensor.""" - - def remove_shape(tensor): - shape = tf.placeholder(tf.int32, name='removed_shape') - return tf.reshape(tensor, shape) - - g = tf.Graph() - with g.as_default(), tf.Session(graph=g) as _: - tf.random.set_seed(24332) - input_shape = [3, 5, 2, 4] - inputs = np.random.random(input_shape) - 0.5 - expected_avg_output = np.mean(inputs, axis=(1, 2), keepdims=True) - input_tensor = tf.convert_to_tensor(inputs, dtype=tf.float32) - # initial shape is [3, 5, 2, 4] - self.assertEqual(py_utils.GetShape(input_tensor), input_shape) - # remove shape using a tf Defun and verify dynamic tensor shape. - input_tensor = remove_shape(input_tensor) - self.assertIsInstance(py_utils.GetShape(input_tensor), tf.Tensor) - self.assertIsNone(input_tensor.shape.rank) - self._testHelper( - 'AVG', - input_tensor, - None, - expected_avg_output, - None, - feed_dict={'removed_shape:0': input_shape}) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/datasource.py b/lingvo/core/datasource.py deleted file mode 100644 index 49710e97..00000000 --- a/lingvo/core/datasource.py +++ /dev/null @@ -1,409 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""DataSources describe how files should be used to provide data.""" - -import os - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils - - -class DataSource(base_layer.BaseLayer): - """A base class for file based Data Sources.""" - - @classmethod - def Params(cls): - return super().Params().Set(name='datasource') - - def __init__(self, params): - super().__init__(params) - self.SetVariableFree() - - def BuildDataSource(self, data_source_from_file_pattern_fn): - """Builds a data source. - - Subclasses implement this. - - Args: - data_source_from_file_pattern_fn: a function that takes file_pattern and - input_source_weights as arguments and returns an input batch from a - string file_pattern. - - Returns: - A NestedMap containing - - - data: (Required) a tuple of tf.Tensor or `.NestedMap` of tf.Tensor same - as ``BaseInputGeneratorFromFiles._DataSourceFromFilePattern( - file_pattern, input_source_weights=None)`` - - source_selected: (Optional) a tensor of size - [batch_size, number of datasources] - - selected_bprop: (Optional) a tensor of size [number of data sources] - - bprop_variable_filters: (Optional) containing a list of bprop_variable - filters for each source. - """ - raise NotImplementedError() - - -class SimpleDataSource(DataSource): - """A simple file based data source.""" - - @classmethod - def Params(cls): - p = super().Params() - # TODO(b/139345706): move filetype prefix (eg tfrecord:) into its own param - # and clean up existing usages. - p.Define( - 'file_pattern', '', 'A single file pattern string which can ' - 'contain a single file pattern, or a comma separated list of patterns.' - 'Samples from each file with unspecified likelihood, though in practice' - ' this will be roughly equal per file. To explicitly ' - 'describe the mixture weights of different file patterns use ' - 'WithinBatchMixingDataSource or CrossBatchMixingDataSource') - p.Define('file_type', '', 'A file type, such as `tfrecord`.') - - return p - - def BuildDataSource(self, data_source_from_file_pattern_fn): - """Builds a simple, unweighted Data Source. - - Args: - data_source_from_file_pattern_fn: a function that takes file_pattern as an - argument and returns an input batch. - - Returns: - A NestedMap containing `data`, which is a tuple of tf.Tensor or - `.NestedMap` of tf.Tensor. - """ - p = self.params - if not isinstance(p.file_pattern, str): - raise ValueError('SimpleDataSource expects p.file_pattern to be a string.' - ' To use multiple files use a comma separated string, ' - 'e.g. \', \'.join(list_of_file_patterns)') - - if p.file_type: - file_pattern = '{}:{}'.format(p.file_type, p.file_pattern) - else: - file_pattern = p.file_pattern - - ret = py_utils.NestedMap() - ret.data = data_source_from_file_pattern_fn(file_pattern) - ret.bprop_variable_filters = [''] - return ret - - -class ChainingDataSource(DataSource): - """A data source that reads each file_pattern in sequence.""" - - @classmethod - def Params(cls): - p = super().Params() - # TODO(b/139345706): This can probably be a list of DataSource params - # instead of a list of file_patterns to be more generic. - p.Define( - 'file_patterns', [], 'A list of file pattern strings which are read ' - 'from in sequence. Commas cannot be used in individual file_patterns.') - return p - - def BuildDataSource(self, data_source_from_file_pattern_fn): - """Builds a Chaining Data Source. - - Args: - data_source_from_file_pattern_fn: a function that takes file_pattern as an - argument and returns an input batch. - - Returns: - A NestedMap containing `data`, which is a tuple of tf.Tensor or - `.NestedMap` of tf.Tensor. - - Raises: - ValueError: If unknown token type. - """ - p = self.params - if not isinstance(p.file_patterns, list): - raise ValueError('Expected a list, got %s' % p.file_patterns) - if not all(isinstance(x, str) for x in p.file_patterns): - # Chaining doesn't work with weights or backprop filters, i.e. when - # file_pattern param contains a list of - # tuples. - raise ValueError('Expected a list of strings, got %s' % p.file_patterns) - - for file_pattern in p.file_patterns: - if ',' in file_pattern: - raise ValueError(('Can not use commas in file_pattern when chaining ' - 'is used. file_pattern: %s') % file_pattern) - ret = py_utils.NestedMap() - ret.data = data_source_from_file_pattern_fn(','.join(p.file_patterns)) - ret.bprop_variable_filters = [''] * len(p.file_patterns) - return ret - - -class WithinBatchMixingDataSource(DataSource): - """Mixes records from different sources into the same batch.""" - - @classmethod - def Params(cls): - p = super().Params() - # TODO(b/139345706): This can probably be a list of DataSource params - # instead of a list of file_patterns to be more generic. - p.Define( - 'file_patterns', [], 'A list of file pattern strings which are read ' - 'from in sequence. Commas cannot be used in individual file_patterns. ') - p.Define('weights', [], 'A list of weights for each file pattern') - return p - - def BuildDataSource(self, data_source_from_file_pattern_fn): - """Read and return input batch from p.file_patterns list weighted by p.weights. - - Examples in the batch will be mixed together from different file_pattern - source proportionally to the weights. - - Args: - data_source_from_file_pattern_fn: a function that takes file_pattern and - input_source_weights as arguments and returns an input batch from a - string file_pattern. - - Returns: - A NestedMap containing: data: a tuple of tf.Tensor or `.NestedMap` of - tf.Tensor - - Raises: - ValueError: If unknown token type. - """ - p = self.params - if not isinstance(p.file_patterns, list): - raise ValueError('Expected a list, got %s' % p.file_patterns) - if not isinstance(p.weights, list): - raise ValueError('Expected a list, got %s' % p.weights) - if len(p.file_patterns) != len(p.weights): - raise ValueError( - 'Expected p.file_patterns and p.weights to be the same length. ' - 'Found %d file_patterns, and %d weights' % - (len(p.file_patterns), len(p.weights))) - # TODO(rosenberg) confirm that weights are numeric - if not all(isinstance(x, str) for x in p.file_patterns): - raise ValueError('Expected all elements of p.file_patterns to be strings') - - file_patterns = p.file_patterns - weights = p.weights - for file_pattern in file_patterns: - if ',' in file_pattern: - raise ValueError( - ('Can not use commas in file_pattern when within-batch ' - 'mixing is used. file_pattern: %s') % file_pattern) - ret = py_utils.NestedMap() - ret.data = data_source_from_file_pattern_fn( - ','.join(file_patterns), input_source_weights=weights) - ret.bprop_variable_filters = [''] * len(file_patterns) - return ret - - -class CrossBatchMixingDataSource(DataSource): - """Mixes batches from different sources, each batch from only one source.""" - - @classmethod - def Params(cls): - p = super().Params() - # TODO(b/139345706): This can probably be a list of DataSource params - # instead of a list of file_patterns to be more generic. - p.Define( - 'file_patterns', [], 'A list of file pattern strings which are read ' - 'from in sequence. Commas cannot be used in individual file_patterns. ') - p.Define('weights', [], 'A list of weights for each file pattern') - p.Define( - 'bprop_variable_filters', [], 'An optional list of ' - 'bprop_variariable_filters for each file_pattern. If not empty, ' - 'expected to have the same length as file_pattern and weights') - return p - - def BuildDataSource(self, data_source_from_file_pattern_fn): - """Read and return input batch from a p.file_pattern list. - - `p.file_patterns` is a list of file patterns, `p.weights` contains - weights for each file pattern. If provided `p.bprop_variable_filters` - includes a bprop_variable_filter for each file pattern. - - Args: - data_source_from_file_pattern_fn: a function that takes file_pattern as an - argument and returns an input batch. - - Returns: - A NestedMap containing: - data: a tuple of tf.Tensor or `.NestedMap` of tf.Tensor - source_selected: a tensor of size [batch_size, number of data sources] - selected_bprop: a tensor of size [number of data sources] - bprop_variable_filters: containing a list of bprop_variable filters for - each source - - Raises: - ValueError: If unknown token type. - """ - p = self.params - - def _MakeDataSourceFromFilePatternFunc(data_source_from_file_pattern_fn, - file_pattern): - # It's important to invoke self._DataSourceFromFilePattern() inside the - # lambda to make sure that the record is drawn from data source - # only if it will be used. Weights are handled by MixByWeight, not the - # data_source_from_file_pattern_fn. - return lambda: data_source_from_file_pattern_fn(file_pattern) - - if len(p.weights) != len(p.file_patterns): - raise ValueError( - 'Expected p.file_patterns and p.weights to be the same length. ' - 'Found %d file_patterns, and %d weights' % - (len(p.file_patterns), len(p.weights))) - if not all(isinstance(x, str) for x in p.file_patterns): - raise ValueError('Expected all elements of p.file_patterns to be strings') - - # TODO(rosenberg) replace this with functools.partial - inputs = [ - _MakeDataSourceFromFilePatternFunc(data_source_from_file_pattern_fn, - file_pattern) - for file_pattern in p.file_patterns - ] - weights = p.weights - if not p.bprop_variable_filters: - bprop_variable_filters = [''] * len(inputs) - else: - bprop_variable_filters = p.bprop_variable_filters - - data_source, selected_bprop = py_utils.MixByWeight( - inputs, weights, seed=p.random_seed) - # TODO(neerajgaur): Remove _bprop_onehot and change code that uses it to - # use source_selected from input_batch. - batch_size = py_utils.GetShape(tf.nest.flatten(data_source)[0])[0] - ret = py_utils.NestedMap() - ret.data = data_source - ret.bprop_variable_filters = bprop_variable_filters - ret.selected_bprop = selected_bprop - ret.source_selected = tf.tile( - tf.expand_dims(selected_bprop, 0), [batch_size, 1]) - return ret - - -class CurriculumDataSource(DataSource): - """A data source that reads different DataSources in stages. - - Supports multiple stages of training, where each stage yields batches for - based on global step boundaries. The contents per stage are defined by nested - DataSources. - - Boundaries are defined by the training global step. - - Currently bprop_variable_filters are not_supported. - # TODO(rosenberg) support bprop_variable_filter within CurriculumDataSource - The issue here is that by conditioning on tf.Variable global_step, the - bprop_variable_filter from the selected DataSource is itself a tf.Variable. - Other bprop_variable_filters are python strings. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'datasource_params', [], 'A list of DataSource Params which define ' - 'the DataSource curriculum.') - p.Define( - 'boundaries', [], 'A list of global step thresholds determining when ' - 'to move from one training stage to another.') - p.Define( - 'bprop_variable_filters', [''], 'A list of bprop_variable_filters to ' - 'apply during training. NOTE: these are constant across all stages.' - 'Changing variable filters per stage is not supported.') - return p - - def BuildDataSource(self, data_source_from_file_pattern_fn): - """Read and return input batch. - - Args: - data_source_from_file_pattern_fn: a function to read and return input - batch from a string file_pattern - - Returns: - A NestedMap containing: - data: a tuple of tf.Tensor or `.NestedMap` of tf.Tensor - - Raises: - ValueError: inconsistent sizes between boundaries and datasource_params, - specification of unsupported datasources, or out of order boundaries. - """ - p = self.params - - if len(p.datasource_params) != len(p.boundaries) + 1: - raise ValueError( - 'Expected p.datasource_params to have one more entry than ' - 'p.boundaries. Found %d datasource_params, and %d boundaries' % - (len(p.datasource_params), len(p.boundaries))) - - for ds_p in p.datasource_params: - if 'bprop_variable_filters' in ds_p: - if any(filter for filter in ds_p.bprop_variable_filters): - raise ValueError('CurriculumDataSource does not support distinct ' - 'bprop_variable_filters per stage.') - - for idx in range(len(p.boundaries) - 1): - if p.boundaries[idx] > p.boundaries[idx + 1]: - raise ValueError('Expected p.boundaries to monotonically increase, but ' - 'found %d > %d at position %d' % - (p.boundaries[idx], p.boundaries[idx + 1], idx)) - - global_step = py_utils.GetGlobalStep() - datasources = [ds_p.Instantiate() for ds_p in p.datasource_params] - - def GetDatasourceFn(idx): - - def DatasourceFn(): - datasource = datasources[idx].BuildDataSource( - data_source_from_file_pattern_fn) - datasource.pop('bprop_variable_filters', None) - return datasource - - return DatasourceFn - - cases = [] - for idx in range(len(p.boundaries)): - cases.append( - (tf.less(global_step, - tf.constant(p.boundaries[idx], - dtype=global_step.dtype)), GetDatasourceFn(idx))) - - ret = tf.case(cases, default=GetDatasourceFn(-1)) - ret.bprop_variable_filters = p.bprop_variable_filters - return ret - - -class PrefixedDataSource(SimpleDataSource): - """Prepends path prefix to file patterns.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'file_pattern_prefix', '', - 'Prefix to add to file_pattern, eg. a base directory that contains ' - 'dataset files.') - - return p - - def __init__(self, params): - super().__init__(params) - - p = self.params - - patterns = p.file_pattern.split(',') - p.file_pattern = ','.join( - os.path.join(p.file_pattern_prefix, pattern) for pattern in patterns) diff --git a/lingvo/core/datasource_test.py b/lingvo/core/datasource_test.py deleted file mode 100644 index 1aa2337f..00000000 --- a/lingvo/core/datasource_test.py +++ /dev/null @@ -1,296 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lingvo.core.datasource.""" - -import lingvo.compat as tf - -from lingvo.core import datasource -from lingvo.core import py_utils -from lingvo.core import test_utils - - -def _MockDataSourceFromFilePattern(file_pattern, input_source_weights=None): - """Read and return input batch from a string file_pattern. - - Args: - file_pattern: A string file pattern. - input_source_weights: A list of float input source weights to control input - example mix in the batch. The records will be sampled from inputs - proportionally to these weights. Defaults to None which should be treated - as an empty list. - - Returns: - file_pattern. This is the file_pattern that will be sent to InputOp along - with input_source_weights and other parameters to generate a batch. - """ - del input_source_weights # Unused. - return tf.constant([file_pattern]) - - -class DatasourceTest(test_utils.TestCase): - - def testSimpleDataSourceSucceedsWithStringInput(self): - ds_params = datasource.SimpleDataSource.Params().Set( - file_pattern='path_to_file') - ds = ds_params.Instantiate() - ret = ds.BuildDataSource(_MockDataSourceFromFilePattern) - with tf.Session(): - ret.data = self.evaluate([ret.data]) - - self.assertCountEqual( - sorted(ret.keys()), ['bprop_variable_filters', 'data']) - self.assertAllEqual(ret.data, [[b'path_to_file']]) - - def testSimpleDataSourceSucceedsWithFileType(self): - ds_params = datasource.SimpleDataSource.Params().Set( - file_pattern='pattern1,pattern2', file_type='tfrecord') - ds = ds_params.Instantiate() - ret = ds.BuildDataSource(_MockDataSourceFromFilePattern) - with tf.Session(): - ret.data = self.evaluate([ret.data]) - - self.assertAllEqual(ret.data, [[b'tfrecord:pattern1,pattern2']]) - - def testSimpleDataSourceFailsWithListInput(self): - files = ['file1', 'file2'] - ds_params = datasource.SimpleDataSource.Params().Set(file_pattern=files) - - ds = ds_params.Instantiate() - with self.assertRaises(ValueError): - ds.BuildDataSource(_MockDataSourceFromFilePattern) - - def testChainingDataSourceSucceedsWithListInput(self): - files = ['path_to_file1', 'path_to_file2'] - ds_params = datasource.ChainingDataSource.Params().Set(file_patterns=files) - ds = ds_params.Instantiate() - ret = ds.BuildDataSource(_MockDataSourceFromFilePattern) - - with tf.Session(): - ret.data = self.evaluate([ret.data]) - - self.assertCountEqual( - sorted(ret.keys()), ['bprop_variable_filters', 'data']) - self.assertAllEqual(ret.data, [[b'path_to_file1,path_to_file2']]) - self.assertCountEqual(ret.bprop_variable_filters, [''] * len(files)) - - def testChainingDataSourceFailsWithWeightedTupleListInput(self): - files = [('file1', 1.0), ('file2', 2.0)] - ds_params = datasource.ChainingDataSource.Params().Set(file_patterns=files) - ds = ds_params.Instantiate() - - with self.assertRaises(ValueError): - ds.BuildDataSource(_MockDataSourceFromFilePattern) - - def testWithinBatchMixingDataSourceSucceedsWithListFilesAndWeights(self): - files = ['path_to_file1', 'path_to_file2'] - weights = [1, 4] - ds_params = datasource.WithinBatchMixingDataSource.Params().Set( - file_patterns=files, weights=weights) - ds = ds_params.Instantiate() - ret = ds.BuildDataSource(_MockDataSourceFromFilePattern) - - with tf.Session(): - ret.data = self.evaluate([ret.data]) - - self.assertCountEqual( - sorted(ret.keys()), ['bprop_variable_filters', 'data']) - self.assertAllEqual(ret.data, [[b'path_to_file1,path_to_file2']]) - self.assertCountEqual(ret.bprop_variable_filters, [''] * len(files)) - - # TODO(b/139345706) should fail when the p.file_pattern behavior is deprecated - def testWithinBatchMixingDataSourceFailsWithListTuplesFiles(self): - # This legacy p.file_pattern behavior is only supported through - # base_input_generator. - files = [('file1', 1.0), ('file2', 2.0)] - ds_params = datasource.WithinBatchMixingDataSource.Params().Set( - file_patterns=files) - ds = ds_params.Instantiate() - - with self.assertRaises(ValueError): - ds.BuildDataSource(_MockDataSourceFromFilePattern) - - def testCrossBatchMixingDataSourceSucceedsWithListFilesAndWeights(self): - files = ['path_to_file', 'path_to_file'] - weights = [1, 4] - ds_params = datasource.CrossBatchMixingDataSource.Params().Set( - file_patterns=files, weights=weights) - ds = ds_params.Instantiate() - ret = ds.BuildDataSource(_MockDataSourceFromFilePattern) - - with tf.Session(): - ret.data = self.evaluate([ret.data]) - - self.assertCountEqual( - sorted(ret.keys()), - ['bprop_variable_filters', 'data', 'selected_bprop', 'source_selected']) - # CrossBatchMixing operates on the python side of the tf op so a single - # element will be returned by _MockDataSourceFromFilePattern - self.assertAllEqual(ret.data, [[b'path_to_file']]) - self.assertCountEqual(ret.bprop_variable_filters, [''] * len(files)) - self.assertAllEqual(ret.selected_bprop.shape, [2]) - self.assertAllEqual(ret.source_selected.shape, [1, 2]) - - def testCrossBatchMixingDataSourceFailsWithListTuplesFiles(self): - # This legacy p.file_pattern behavior is only supported through - # base_input_generator. - files = [('file1', 1.0), ('file2', 2.0)] - ds_params = datasource.CrossBatchMixingDataSource.Params().Set( - file_patterns=files) - ds = ds_params.Instantiate() - - with self.assertRaises(ValueError): - ds.BuildDataSource(_MockDataSourceFromFilePattern) - - def testCurriculumDataSourceSucceedsWithSimpleDataSource(self): - sources = [ - datasource.SimpleDataSource.Params().Set(file_pattern='file1'), - datasource.SimpleDataSource.Params().Set(file_pattern='file2'), - ] - ds_params = datasource.CurriculumDataSource.Params().Set( - datasource_params=sources, boundaries=[5]) - ds = ds_params.Instantiate() - - ret = ds.BuildDataSource(_MockDataSourceFromFilePattern) - with tf.Session(): - self.evaluate(tf.global_variables_initializer()) - ret.data = self.evaluate([ret.data]) - - self.assertCountEqual( - sorted(ret.keys()), ['bprop_variable_filters', 'data']) - self.assertAllEqual(ret.data, [[b'file1']]) - self.assertCountEqual(ret.bprop_variable_filters, ['']) - - def testCurriculumDataSourceTransitionsCorrectlyWithSimpleDataSource(self): - sources = [ - datasource.SimpleDataSource.Params().Set(file_pattern='file1'), - datasource.SimpleDataSource.Params().Set(file_pattern='file2'), - ] - boundary = 5 - ds_params = datasource.CurriculumDataSource.Params().Set( - datasource_params=sources, boundaries=[boundary]) - ds = ds_params.Instantiate() - - ret = ds.BuildDataSource(_MockDataSourceFromFilePattern) - with tf.Session(): - # Advance the global step to the next curriculum stage - global_step = py_utils.GetOrCreateGlobalStepVar() - self.evaluate(tf.global_variables_initializer()) - set_global_step = tf.assign(global_step, boundary, name='advance_step') - self.evaluate(set_global_step) - - ret.data = self.evaluate([ret.data]) - - self.assertCountEqual( - sorted(ret.keys()), ['bprop_variable_filters', 'data']) - self.assertAllEqual(ret.data, [[b'file2']]) - self.assertCountEqual(ret.bprop_variable_filters, ['']) - - def testCurriculumDataSourceTransitionsCorrectlyWithMixingDataSource(self): - sources = [ - datasource.WithinBatchMixingDataSource.Params().Set( - file_patterns=['file1', 'file2'], weights=[1, 5]), - datasource.WithinBatchMixingDataSource.Params().Set( - file_patterns=['file3', 'file4'], weights=[2, 3]) - ] - boundary = 5 - ds_params = datasource.CurriculumDataSource.Params().Set( - datasource_params=sources, boundaries=[boundary]) - ds = ds_params.Instantiate() - - ret = ds.BuildDataSource(_MockDataSourceFromFilePattern) - with tf.Session(): - # Advance the global step to the next curriculum stage - global_step = py_utils.GetOrCreateGlobalStepVar() - self.evaluate(tf.global_variables_initializer()) - set_global_step = tf.assign(global_step, boundary, name='advance_step') - self.evaluate(set_global_step) - - ret.data = self.evaluate([ret.data]) - - self.assertCountEqual( - sorted(ret.keys()), ['bprop_variable_filters', 'data']) - self.assertAllEqual(ret.data, [[b'file3,file4']]) - self.assertCountEqual(ret.bprop_variable_filters, ['']) - - def testCurriculumDataSourceFailsWithBadBoundaries(self): - sources = [ - datasource.WithinBatchMixingDataSource.Params().Set( - file_patterns=['file1', 'file2'], weights=[1, 5]), - datasource.WithinBatchMixingDataSource.Params().Set( - file_patterns=['file3', 'file4'], weights=[2, 3]) - ] - ds_params = datasource.CurriculumDataSource.Params().Set( - datasource_params=sources, boundaries=[10, 5]) - ds = ds_params.Instantiate() - with self.assertRaises(ValueError): - ds.BuildDataSource(_MockDataSourceFromFilePattern) - - def testPrefixDataSourceSucceedsWithDirectory(self): - ds_params = datasource.PrefixedDataSource.Params().Set( - file_pattern='filename-*.tfrecord', - file_type=None, - file_pattern_prefix='/dir/') - ds = ds_params.Instantiate() - ret = ds.BuildDataSource(_MockDataSourceFromFilePattern) - - with tf.Session(): - ret.data = self.evaluate([ret.data]) - - self.assertAllEqual(ret.data, [[b'/dir/filename-*.tfrecord']]) - - def testPrefixDataSourceSucceedsWithMultiplePatterns(self): - ds_params = datasource.PrefixedDataSource.Params().Set( - file_pattern='filename-*.tfrecord,other/file/pattern/*', - file_type=None, - file_pattern_prefix='/dir/') - ds = ds_params.Instantiate() - ret = ds.BuildDataSource(_MockDataSourceFromFilePattern) - - with tf.Session(): - ret.data = self.evaluate([ret.data]) - - self.assertAllEqual( - ret.data, [[b'/dir/filename-*.tfrecord,/dir/other/file/pattern/*']]) - - def testPrefixDataSourceSucceedsWithGcsBucket(self): - ds_params = datasource.PrefixedDataSource.Params().Set( - file_pattern='filename-*.tfrecord', - file_type=None, - file_pattern_prefix='gs://bucket/dir') - ds = ds_params.Instantiate() - ret = ds.BuildDataSource(_MockDataSourceFromFilePattern) - - with tf.Session(): - ret.data = self.evaluate([ret.data]) - - self.assertAllEqual(ret.data, [[b'gs://bucket/dir/filename-*.tfrecord']]) - - def testPrefixDataSourceSucceedsWithFileType(self): - ds_params = datasource.PrefixedDataSource.Params().Set( - file_pattern='filename-*.tfrecord', - file_type='tfrecord', - file_pattern_prefix='dir') - ds = ds_params.Instantiate() - ret = ds.BuildDataSource(_MockDataSourceFromFilePattern) - - with tf.Session(): - ret.data = self.evaluate([ret.data]) - - self.assertAllEqual(ret.data, [[b'tfrecord:dir/filename-*.tfrecord']]) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/decoder_lib.py b/lingvo/core/decoder_lib.py deleted file mode 100644 index 9b813b05..00000000 --- a/lingvo/core/decoder_lib.py +++ /dev/null @@ -1,25 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Helpers for the decoding phase of jobs.""" - -import pickle -import lingvo.compat as tf - - -def WriteKeyValuePairs(filename, key_value_pairs): - """Writes `key_value_pairs` to `filename`.""" - with tf.io.gfile.GFile(filename, mode='wb') as f: - pickle.dump(key_value_pairs, f, protocol=pickle.HIGHEST_PROTOCOL) diff --git a/lingvo/core/distillation_task.py b/lingvo/core/distillation_task.py deleted file mode 100644 index 8f0fb759..00000000 --- a/lingvo/core/distillation_task.py +++ /dev/null @@ -1,170 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Base class for tasks that implement knowledge distillation.""" -from lingvo import compat as tf -from lingvo.core import base_model -from lingvo.core import py_utils -from lingvo.core import schedule - - -class DistillationTask(base_model.BaseTask): - """A task to distill knowledge from a teacher task to a student task. - - The training parameters (e.g., learning rate) are determined only by - `DistillationTask.params.train`. Teacher and student task's training and eval - parameters must be set to None. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('teacher', None, 'The teacher task params.') - p.Define('student', None, 'The student task params.') - p.Define( - 'distillation_loss_weight', - # Only uses distillation loss by default. - schedule.ConstantOne.Params(), - 'A schedule of distillation loss weight. ' - 'The weight determines the fraction of total loss contributed by ' - 'distillation loss, while the rest loss will be computed against ' - 'the ground truth. ' - 'A weight of 0 means to only use ground-truth and ignore teacher ' - 'predictions, while a weight 1 means to only use teacher ' - 'predictions and ignore ground truth. ' - 'The weight is specified as a schedule to allow it to change ' - 'during training.') - p.Define( - 'teacher_target_type', 'truth', 'The target type for the teacher. ' - 'Choices are: ' - ' "truth": using the ground-truth target labels ' - ' "beam": using the 1-best hypothesis from the beam search.') - p.Define( - 'beam_search_temperature', 1.0, 'The temperature to scale the' - 'log-prob of each beam search hypothesis. This is used in ' - 'training only') - p.Define( - 'train_teacher', False, 'Adds the teacher\'s loss (w.r.t the ground ' - 'truth labels) to the overall ground truth loss. This can be used for ' - 'instance when the teacher is trained in parallel to the student.') - return p - - def __init__(self, params): - assert issubclass(params.cls, DistillationTask) - super().__init__(params) - - p = self.params - # While student does not need its own input generator for training, it - # needs an input generator for inference graphs. - p.student.input = p.input - # Teacher also might need an input generator, eg. for waveform_processor. - p.teacher.input = p.input - for child in ('teacher', 'student'): - child_p = getattr(p, child) - assert issubclass(child_p.cls, base_model.BaseTask) - assert child_p.train is None - assert child_p.eval is None - # In theory it's ok for teacher to be a DistillationTask. In practice - # it probably won't happen. - assert not issubclass(child_p.cls, DistillationTask) - child_p.name = child - self.CreateChild(child, child_p) - self.CreateChild('distillation_loss_weight', p.distillation_loss_weight) - - def ComputePredictions(self, theta, input_batch): - p = self.params - with tf.name_scope(p.name): - if p.teacher_target_type == 'truth': - teacher_predictions = self.teacher.ComputePredictions( - theta.teacher, input_batch) - student_predictions = self.student.ComputePredictions( - theta.student, input_batch) - return py_utils.NestedMap( - teacher=teacher_predictions, student=student_predictions) - elif p.teacher_target_type == 'beam': - (teacher_predictions, teacher_input_batch, - teacher_beam_prob) = self.teacher.ComputeBeamPredictions( - theta.teacher, input_batch, p.beam_search_temperature) - # We use 'teacher_input_batch' instead of 'input_batch' for 'student' - # because the training of student network uses target transcripts for - # the "teacher forcing" mode and here the target transcripts should come - # from the teacher's beam search. - student_predictions = self.student.ComputePredictions( - theta.student, teacher_input_batch) - return py_utils.NestedMap( - teacher=teacher_predictions, - student=student_predictions, - teacher_beam_prob=teacher_beam_prob) - else: - raise ValueError('teacher target type not defined properly: %s' % - self.p.teacher_target_type) - - def ComputeLoss(self, theta, predictions, input_batch): - p = self.params - per_example = {} - with tf.name_scope('groundtruth_loss'): - student_groundtruth_loss, student_groundtruth_per_example = ( - self.student.ComputeLoss(theta.student, predictions.student, - input_batch)) - groundtruth_loss = student_groundtruth_loss - groundtruth_loss['student_groundtruth_loss'] = ( - student_groundtruth_loss['loss']) - per_example.update(student_groundtruth_per_example) - - if p.train_teacher: - teacher_groundtruth_loss, _ = self.teacher.ComputeLoss( - theta.teacher, predictions.teacher, input_batch) - groundtruth_loss['teacher_groundtruth_loss'] = ( - teacher_groundtruth_loss['loss']) - # The new loss is the wighted sum of the teacher and student losses. - groundtruth_loss['loss'] = py_utils.WeightedAvg(*zip( - teacher_groundtruth_loss['loss'], student_groundtruth_loss['loss'])) - - with tf.name_scope('distillation_loss'): - distillation_loss, distill_per_example = self.ComputeDistillationLoss( - theta, predictions, input_batch) - distillation_loss['distillation_loss'] = distillation_loss['loss'] - per_example.update(distill_per_example) - - distillation_loss_weight = self.distillation_loss_weight.FProp( - theta.distillation_loss_weight, self.global_step) - metrics = py_utils.CombineMetrics([ - (groundtruth_loss, 1 - distillation_loss_weight), - (distillation_loss, distillation_loss_weight), - ]) - return metrics, per_example - - def ComputeDistillationLoss(self, theta, predictions, input_batch): - raise NotImplementedError('Abstract method') - - def BProp(self): - p = self.params - if p.train_teacher: - return super().BProp() - else: - # Only bprop on student variables. - self._BPropForVariables(self.student.vars) - - def Decode(self, input_batch): - return self.student.Decode(input_batch) - - def Inference(self): - return self.student.Inference() - - def CreateDecoderMetrics(self): - return self.student.CreateDecoderMetrics() - - def PostProcessDecodeOut(self, dec_out_dict, dec_metrics_dict): - return self.student.PostProcessDecodeOut(dec_out_dict, dec_metrics_dict) diff --git a/lingvo/core/distributed_shampoo.py b/lingvo/core/distributed_shampoo.py deleted file mode 100644 index e9a6da02..00000000 --- a/lingvo/core/distributed_shampoo.py +++ /dev/null @@ -1,668 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Implementation for distributed Shampoo optimizer.""" - -import functools -import lingvo.compat as tf -from lingvo.core import matrix_functions -from lingvo.core import ops as x_ops -import numpy as np - -# pylint: disable=g-direct-tensorflow-import -from tensorflow.python.framework import ops -from tensorflow.python.ops import array_ops -from tensorflow.python.ops import math_ops -from tensorflow.python.ops import state_ops -from tensorflow.python.training import optimizer -# pylint: enable=g-direct-tensorflow-import - - -class PartitionConfig: - """Config for tensor partitioning.""" - - def __init__(self, max_dim_size, partition_size): - """Initialize the `PartitionConfig`. - - Args: - max_dim_size: Partitions dimensions with size greater than this value. - partition_size: Size of each partition - """ - self.max_dim_size = max_dim_size - self.partition_size = partition_size - - -class PartitionMetadata: - """Metadata for partitioning.""" - - def __init__(self, split_sizes_per_dim, num_splits_per_dim): - """Initialize the `PartitionMetadata`. - - Args: - split_sizes_per_dim: Split sizes per dimemsion. - num_splits_per_dim: Number of splits per dimension ( inferred from - split_sizes_per_dim). - """ - self.split_sizes_per_dim = split_sizes_per_dim - self.num_splits_per_dim = num_splits_per_dim - - -class TensorPartitioner: - """Shards Tensor's across its axis. - - In cases of TPUs, these partitions are zero cost, and does not involve data - movement. - """ - - @classmethod - def partition_metadata(cls, tensor, partition_info): - """Returns metadata required for partitioning and reforming tensors. - - Args: - tensor: Tensor to partition. - partition_info: Partitioning info. - - Returns: - split_sizes_per_dim and num_splits_per_dim. - """ - shape = tensor.get_shape() - # Split if dim is greater than max_dim. - axis_to_shard = [s > partition_info.max_dim_size for s in shape] - split_sizes_per_dim = [] - # Compute the number of splits, and the sizes of the splits for - # each dimension - for sharded, dim in zip(axis_to_shard, shape): - dim = int(dim) - split_sizes_per_dim.append([dim]) - if sharded: - split_sizes = [] - num_shards = dim // partition_info.partition_size - if num_shards > 0: - split_sizes = [partition_info.partition_size] * num_shards - last_shard_size = dim % partition_info.partition_size - if last_shard_size > 0: - split_sizes.append(last_shard_size) - else: - split_sizes.append(dim) - split_sizes_per_dim[-1] = split_sizes - num_splits_per_dim = [len(v) for v in split_sizes_per_dim] - return PartitionMetadata(split_sizes_per_dim, num_splits_per_dim) - - @classmethod - def partition_tensor(cls, tensor, partition_info): - """Returns partitioned tensors.""" - metadata = (TensorPartitioner.partition_metadata(tensor, partition_info)) - # Split from last to first axis. - partitioned_tensors = [tensor] - rank = len(metadata.num_splits_per_dim) - for raxis, (num_splits, sizes) in enumerate( - zip( - reversed(metadata.num_splits_per_dim), - reversed(metadata.split_sizes_per_dim))): - if num_splits > 1: - tmp_partitioned_tensors = [] - for item in partitioned_tensors: - tmp_partitioned_tensors += tf.split( - item, sizes, axis=rank - raxis - 1) - partitioned_tensors = tmp_partitioned_tensors - return partitioned_tensors - - @classmethod - def reform_tensor(cls, partitioned_tensors, num_splits_per_dim): - """Returns a tensor concatenated from the given partitions.""" - # Concatenates tensors across all dimension. Assumes the `partitions` tensor - # was created by partition_tensor. - for axis, num_splits in enumerate(num_splits_per_dim): - if num_splits > 1: - tmp_partitioned_tensors = [] - num_concat = len(partitioned_tensors) // num_splits - for i in range(num_concat): - tensors_to_concat = ( - partitioned_tensors[i * num_splits:(i + 1) * num_splits]) - tmp_partitioned_tensors.append( - tf.concat(tensors_to_concat, axis=axis)) - partitioned_tensors = tmp_partitioned_tensors - return partitioned_tensors[0] - - -class DistributedShampoo(optimizer.Optimizer): - """Approximates full-matrix AdaGrad per layer. - - Approximates full-matrix AdaGrad with kronecker-products of two statistics - matrices based on only the first-order gradients of the layer. - - "Second-order optimization made practical.", 2019 - Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, Yoram Singer. - """ - - def __init__(self, - learning_rate, - momentum=0.0, - initial_accumulator_value=0.0, - start_preconditioning_steps=1000, - statistics_computation_frequency=1, - matrix_epsilon=1e-6, - synchronous_preconditioning=False, - second_moment_averaging=1.0, - fallback_to_diagonal_dim=4096, - max_any_dim=6656, - block_size=4096, - block_partition_threshold_size=1000000, - global_step=None, - exponent_multiplier=1.0, - name="DistributedShampoo"): - """Construct a DistributedShampoo optimizer. - - Args: - learning_rate: A `Tensor` or a floating point value. The learning rate. - momentum: A `Tensor` or a floating point value. Momentum is not applied to - sparse updates. - initial_accumulator_value: A floating point value. - start_preconditioning_steps: A int32 value which indicates when to start - preconditioning. - statistics_computation_frequency: A int32 step value which indicates how - often to compute statistics for preconditioning. - matrix_epsilon: An epsilon regularizer to make the matrices positive - definite. - synchronous_preconditioning: Whether to run preconditioning synchronously. - second_moment_averaging: 1.0 means sum of gradients squares, while less - than 1.0 switches to RMSProp style exponential moving averages of the - second moments. - fallback_to_diagonal_dim: Fallback to diagonal version of AFMA if the any - of the dimension is larger than fallback_to_diagonal_dim. - max_any_dim: If maximum value for any dimension is greater than this value - we skip preconditioning and fall back to the diagonal. - block_size: Dimension of the partitioned tensors. - block_partition_threshold_size: Partitions diemnsions beyond this size. - global_step: Global step for training. - exponent_multiplier: A multiplier 'e` for the exponent for the inverse - calculation. e * -1/(2*rank). Only applies when calculating inverses - through svd. - name: Optional name prefix for the operations created when applying - gradients. - """ - super().__init__(False, name) - self._learning_rate = learning_rate - self._momentum = momentum - self._initial_accumulator_value = initial_accumulator_value - self._start_preconditioning_steps = start_preconditioning_steps - self._matrix_epsilon = matrix_epsilon - self._synchronous_preconditioning = synchronous_preconditioning - self._second_moment_averaging = second_moment_averaging - self._fallback_to_diagonal_dim = fallback_to_diagonal_dim - self._max_any_dim = max_any_dim - self._block_size = block_size - # NOTE: On XLA - int64 is not handled properly. - if global_step is not None: - self._global_step = tf.cast(tf.identity(global_step), tf.int32) - else: - self._global_step = tf.cast( - tf.identity(tf.train.get_or_create_global_step()), tf.int32) - self._run_nondiagonal_update = tf.greater_equal( - self._global_step, self._start_preconditioning_steps) - start_steps_f = tf.cast(self._start_preconditioning_steps, tf.float32) - global_step_f = tf.cast(self._global_step, tf.float32) - self._run_nondiagonal_update_warmup = tf.minimum( - 1.0, tf.maximum((global_step_f - start_steps_f) / start_steps_f, 0.0)) - # Computes statistics every K steps. - self._statistics_computation_frequency = statistics_computation_frequency - self._run_statistics_computation = tf.equal( - tf.math.floormod(self._global_step, - self._statistics_computation_frequency), 0) - # All vars that are preconditioned. - self._all_vars_for_preconditioning = [] - self._exponent_multiplier = exponent_multiplier - self._partition_info = PartitionConfig(block_partition_threshold_size, - block_size) - self._partitioner_metadata = {} - - def _fallback_to_diagonal_for_shape(self, shape): - """Returns whether we should fallback to the diagonal update given shape.""" - # We fallback to diagonal for the following usecases: - # - # (a) Rank <= 1 tensors - # (b) if any dim of Tensor is > max_any_dim. - # (c) if all dims are 1 or are greater than fallback_to_diagonal_dim - # - if len(shape) <= 1: - return True - if any([d > self._max_any_dim for d in shape]): - return True - if all([d == 1 for d in shape]): - return True - return False - - def _preconditioner_available_for_dims(self, shape): - """Returns indicator vector if preconditioner exists for each axis.""" - # If any of the dims < fallback_to_diagonal_dim and not 1, we run a - # a preconditioner for that particular dimension. - return [d <= self._fallback_to_diagonal_dim and d != 1 for d in shape] - - def _preconditioner_indices(self, shape): - """Returns indices of the available preconditioner.""" - preconditioners_available_for_dims = ( - self._preconditioner_available_for_dims(shape)) - indices = [] - index = 0 - for is_avail_for_dim_i in preconditioners_available_for_dims: - indices.append(index) - if is_avail_for_dim_i: - index += 1 - return indices - - def _make_named_slot(self, var, val, slot_name): - _ = self._get_or_make_slot(var, val, slot_name, - self._name + "_" + slot_name) - - def make_named_zeros_slot(self, var, slot_name): - self._zeros_slot(var, slot_name, self._name + "_" + slot_name) - - def _generalized_inverse_pth_root(self, input_t, exponent, epsilon=1e-12): - input_t_f64 = tf.cast(input_t, tf.float64) - s, u, v = tf.linalg.svd( - input_t_f64 + - tf.eye(tf.shape(input_t_f64)[0], dtype=tf.float64) * epsilon, - full_matrices=True) - inv_s = tf.reshape( - tf.pow(tf.maximum(s, epsilon), tf.cast(exponent, tf.float64)), [1, -1]) - val = tf.matmul(u * inv_s, v, adjoint_b=True) - return tf.cast(val, tf.float32), tf.reduce_max(tf.abs(u - v)) - - def _specialized_inverse_pth_root(self, input_t, exponent, epsilon=1e-12): - input_t_f64 = tf.cast(input_t, tf.float64) - val, error = matrix_functions.inlined_matrix_inverse_pth_root( - input_t_f64, - tf.shape(input_t_f64)[0], - exponent, - iter_count=40, - ridge_epsilon=epsilon) - return tf.cast(val, tf.float32), error - - def _inverse_pth_root_graph(self, epsilon): - graph = tf.Graph() - with graph.as_default(): - exponent_t = tf.reshape( - tf.placeholder(dtype=tf.float32, name="exponent", shape=None), []) - # Apply exponent multiplier. - exponent_t = exponent_t * self._exponent_multiplier - input_t = tf.placeholder(dtype=tf.float32, name="input", shape=None) - # For p = 2, 4 or 8, we use the iterative Newton-Schur method for - # computing the inverse-pth root. - either_p_2_4_8 = tf.math.logical_or( - tf.math.logical_or( - tf.equal(-1.0 / exponent_t, 2), tf.equal(-1.0 / exponent_t, 4)), - tf.equal(-1.0 / exponent_t, 8)) - # 4096 is the larger dimension SVD is tractable for. - greater_than_4096 = tf.greater(tf.shape(input_t)[0], 4096) - run_specialized_iterative_method = tf.math.logical_and( - greater_than_4096, either_p_2_4_8) - specialized_fn = functools.partial(self._specialized_inverse_pth_root, - input_t, exponent_t, epsilon) - generalized_fn = functools.partial(self._generalized_inverse_pth_root, - input_t, exponent_t, epsilon) - output, diff = tf.cond(run_specialized_iterative_method, specialized_fn, - generalized_fn) - - tf.identity(output, "output") - tf.identity(tf.cast(diff, tf.float32), "diff") - return graph.as_graph_def().SerializeToString() - - def _create_slots(self, var_list): - self._preconditioner_compute_graphdef = self._inverse_pth_root_graph( - epsilon=self._matrix_epsilon) - for v in var_list: - self._make_named_slot(v, - tf.ones_like(v) * self._initial_accumulator_value, - "accumulator") - - if self._momentum > 0.0: - self.make_named_zeros_slot(v, "momentum") - shape = np.array(v.get_shape()) - self._partitioner_metadata[v] = TensorPartitioner.partition_metadata( - v, self._partition_info) - partitioned_v = TensorPartitioner.partition_tensor( - v, self._partition_info) - if not self._fallback_to_diagonal_for_shape(shape): - self._all_vars_for_preconditioning.append(v) - if self._momentum > 0.0: - self.make_named_zeros_slot(v, "precond_grad_momentum") - num_partitions = len(partitioned_v) - for pt_idx, pt_v in enumerate(partitioned_v): - pt_v_shape = pt_v.get_shape() - preconditioner_exists_for_dim = ( - self._preconditioner_available_for_dims(pt_v_shape)) - for i, d in enumerate(pt_v_shape): - if preconditioner_exists_for_dim[i]: - mat_stat_init = array_ops.zeros([d, d], dtype=pt_v.dtype) - self._make_named_slot( - v, mat_stat_init, - self._statistics_key_for_partition_and_dim( - i, pt_idx, num_partitions)) - self._make_named_slot( - v, mat_stat_init, - self._preconditioner_key_for_partition_and_dim( - i, pt_idx, num_partitions)) - - def _prepare(self): - learning_rate = self._call_if_callable(self._learning_rate) - self._learning_rate_tensor = ops.convert_to_tensor( - learning_rate, name="learning_rate") - momentum = self._call_if_callable(self._momentum) - self._momentum_tensor = ops.convert_to_tensor(momentum, name="momentum") - - def invoke_async_preconditioner_computation(self, global_step_int32): - """Invokes SVD preconditioner and graph runs on the CPU.""" - keys_stats_and_rank = [] - for var in self._all_vars_for_preconditioning: - shape = var.get_shape() - if not self._fallback_to_diagonal_for_shape(shape): - partitioned_v = TensorPartitioner.partition_tensor( - var, self._partition_info) - num_partitions = len(partitioned_v) - for pt_idx, pt_v in enumerate(partitioned_v): - pt_v_shape = pt_v.get_shape() - preconditioner_exists_for_dim = ( - self._preconditioner_available_for_dims(pt_v_shape)) - for i in range(len(pt_v_shape)): - if preconditioner_exists_for_dim[i]: - rank = sum(preconditioner_exists_for_dim) - key = self._key_for_var(var, i, pt_idx) - stat = self.get_slot( - var, - self._statistics_key_for_partition_and_dim( - i, pt_idx, num_partitions)) - keys_stats_and_rank.append((key, stat, rank)) - - if not keys_stats_and_rank: - return tf.no_op() - keys, stats, ranks = zip(*keys_stats_and_rank) - - return x_ops.compute_preconditioners( - stats, [-1.0 / (2.0 * r) for r in ranks], - global_step_int32, - keys=keys, - sync=self._synchronous_preconditioning, - preconditioner_compute_graphdef=self._preconditioner_compute_graphdef) - - def assign_preconditioner_to_host_vars(self): - """Assign/Grab latest copy of preconditioners.""" - keys_shapes_and_preconditioner_vars = [] - assign_ops = [] - for var in self._all_vars_for_preconditioning: - shape = var.get_shape() - if not self._fallback_to_diagonal_for_shape(shape): - partitioned_v = TensorPartitioner.partition_tensor( - var, self._partition_info) - num_partitions = len(partitioned_v) - for pt_idx, pt in enumerate(partitioned_v): - pt_shape = pt.get_shape() - preconditioner_exists_for_dim = ( - self._preconditioner_available_for_dims(pt_shape)) - var_rank = len(pt_shape) - for i in range(var_rank): - if preconditioner_exists_for_dim[i]: - key = self._key_for_var(var, i, pt_idx) - preconditioner = self.get_slot( - var, - self._preconditioner_key_for_partition_and_dim( - i, pt_idx, num_partitions)) - keys_shapes_and_preconditioner_vars.append( - (key, tf.shape(preconditioner), preconditioner)) - - if not keys_shapes_and_preconditioner_vars: - return tf.no_op() - - keys, shapes, preconditioner_vars = zip( - *keys_shapes_and_preconditioner_vars) - - preconditioner_vals, successes = x_ops.get_preconditioners( - shapes, - keys=keys, - preconditioner_compute_graphdef=( - self._preconditioner_compute_graphdef)) - - for preconditioner_var, preconditioner_val, success in zip( - preconditioner_vars, preconditioner_vals, successes): - success_mult = tf.cast(success, preconditioner.dtype) - assign_ops.append( - state_ops.assign(preconditioner_var, - (1.0 - success_mult) * preconditioner_var + - success_mult * preconditioner_val)) - return tf.group(*assign_ops) - - def _statistics_key_for_partition_and_dim(self, dim_index, partition_index, - num_partitions): - if num_partitions == 1: - return "mat_statistics_" + str(dim_index) - else: - return str(partition_index) + "_mat_statistics_" + str(dim_index) - - def _preconditioner_key_for_partition_and_dim(self, dim_index, - partition_index, - num_partitions): - if num_partitions == 1: - return "mat_preconditioner_" + str(dim_index) - else: - return str(partition_index) + "_mat_preconditioner_" + str(dim_index) - - def _key_for_var(self, var, dim_index, partition_index): - return "P_" + str(partition_index) + "_D_" + str(dim_index) + "_" + var.name - - def _updated_statistics(self, var, partitioned_grads): - """Returns updated Shampoo statistics L_t, R_t, etc. - - Args: - var: tf.Variable associated with the gradient. - partitioned_grads: Partitioned gradient tensor. - - Returns: - A list of updated statistics matrices. - """ - precond_statistics_update = [] - num_partitions = len(partitioned_grads) - mat_stats = [] - mat_grads = [] - mat_dims = [] - for pt_idx, pt_grad in enumerate(partitioned_grads): - pt_shape = pt_grad.get_shape() - preconditioner_exists_for_dim = ( - self._preconditioner_available_for_dims(pt_shape)) - rank = len(pt_shape) - # Calculates the preconditioner statistics for each tensor. - for i in range(rank): - if preconditioner_exists_for_dim[i]: - mat_stats.append( - self.get_slot( - var, - self._statistics_key_for_partition_and_dim( - i, pt_idx, num_partitions))) - mat_grads.append(pt_grad) - mat_dims.append(i) - - # axes is the list of indices to reduce - everything but - # the current i. - def _update_statistics(dim, stat_var, grad): - """Update preconditioner statistics.""" - with tf.name_scope("GradientStatistics"): - var_rank = len(grad.get_shape()) - axes = list(range(dim)) + list(range(dim + 1, var_rank)) - new_stat = math_ops.tensordot(grad, grad, axes=(axes, axes)) - if self._second_moment_averaging == 1.0: - updated_stat = state_ops.assign_add(stat_var, new_stat) - else: - updated_stat = state_ops.assign_add( - stat_var, (self._second_moment_averaging - 1.0) * stat_var + - (1.0 - self._second_moment_averaging) * new_stat) - return updated_stat - - if self._statistics_computation_frequency <= 1: - for mat_stat, mat_grad, dim in zip(mat_stats, mat_grads, mat_dims): - precond_statistics_update.append( - _update_statistics(dim, mat_stat, mat_grad)) - else: - - # NOTE: We rewrite tf.cond() as a while loop to avoid certain overheads - # in XLA from buffer allocation. - def _loop_body(mat_stats, mat_grads, mat_dims, unused_perform_step): - precond_statistics_update_ops = [] - for mat_stat, mat_grad, dim in zip(mat_stats, mat_grads, mat_dims): - precond_statistics_update_ops.append( - _update_statistics(dim, mat_stat, mat_grad)) - with tf.control_dependencies(precond_statistics_update_ops): - return tf.constant(False) - - loop_body_fn = functools.partial(_loop_body, mat_stats, mat_grads, - mat_dims) - precond_statistics_update.append( - tf.while_loop(lambda perform_step: perform_step, loop_body_fn, - [self._run_statistics_computation])) - - return precond_statistics_update - - def _compute_preconditioned_raw_grad(self, var, partitioned_grads): - """Returns preconditioned gradient. - - Args: - var: tf.Variable associated with the gradient. - partitioned_grads: Partitioned gradient tensor. - - Returns: - A preconditioned gradient tensor. - """ - - partitioned_preconditioned_grads = [] - num_partitions = len(partitioned_grads) - for pt_idx, pt_grad in enumerate(partitioned_grads): - pt_shape = pt_grad.get_shape() - rank = len(pt_shape) - preconditioner_exists_for_dim = ( - self._preconditioner_available_for_dims(pt_shape)) - preconditioner_indices = self._preconditioner_indices(pt_shape) - mat_preconditioner_list = [] - for i in range(rank): - if preconditioner_exists_for_dim[i]: - mat_preconditioner_list.append( - self.get_slot( - var, - self._preconditioner_key_for_partition_and_dim( - i, pt_idx, num_partitions))) - precond_grad = pt_grad - if rank == 2 and all(preconditioner_exists_for_dim): - # Fast path for speedup. - precond_grad = tf.matmul( - tf.matmul(mat_preconditioner_list[0], precond_grad), - mat_preconditioner_list[1]) - else: - for i in range(rank): - if preconditioner_exists_for_dim[i]: - precond_grad = tf.tensordot( - precond_grad, - mat_preconditioner_list[preconditioner_indices[i]], - axes=([0], [0])) - else: - # if preconditioner is not available we transpose it to - # permute the axis for the next preconditioner. - precond_grad = tf.transpose( - precond_grad, perm=list(range(1, rank)) + [0]) - partitioned_preconditioned_grads.append(precond_grad) - return TensorPartitioner.reform_tensor( - partitioned_preconditioned_grads, - self._partitioner_metadata[var].num_splits_per_dim) - - def _preconditioned_update(self, var, partitioned_grads, - diagonal_grad_update): - """Computes the matrix preconditioned update. - - Args: - var: Variable for which we are computing the preconditioned gradient. - partitioned_grads: Partitioned gradients. - diagonal_grad_update: Update as given by diagonal adagrad. - - Returns: - scaled preconditioned gradient. - """ - - def _l2_norm(v): - return tf.sqrt(tf.reduce_sum(tf.square(v))) - - precond_grad = self._compute_preconditioned_raw_grad(var, partitioned_grads) - if self._momentum > 0.0: - gbar = self.get_slot(var, "precond_grad_momentum") - matrix_preconditioned_grad = state_ops.assign( - gbar, gbar * self._momentum_tensor + precond_grad * - (1.0 - self._momentum_tensor)) - else: - matrix_preconditioned_grad = precond_grad - - # We use the direction from Shampoo while using the step size scale from - # diagonal AdaGrad. - precond_l2_norm = _l2_norm(matrix_preconditioned_grad) - diagonal_l2_norm = _l2_norm(diagonal_grad_update) - multiplier = tf.where( - tf.greater(precond_l2_norm, 0.0), - tf.maximum(diagonal_l2_norm, 1e-30) / - (tf.maximum(precond_l2_norm, 1e-30)), 1.0) - return matrix_preconditioned_grad * multiplier - - def _apply_dense(self, grad, var): - # Calculates the preconditioner statistics for each tensor. - partitioned_grads = TensorPartitioner.partition_tensor( - grad, self._partition_info) - shape = var.get_shape() - fallback_to_diagonal = self._fallback_to_diagonal_for_shape(shape) - - precond_statistics_update = [] - if not fallback_to_diagonal: - precond_statistics_update = self._updated_statistics( - var, partitioned_grads) - - accumulator = self.get_slot(var, "accumulator") - accumulator_updated = state_ops.assign_add(accumulator, grad * grad) - accumulator_inv_sqrt = math_ops.rsqrt(accumulator_updated + 1e-30) - if self._momentum > 0.0: - scaled_g = (1.0 - self._momentum_tensor) * (grad * accumulator_inv_sqrt) - gbar = self.get_slot(var, "momentum") - gbar_updated = state_ops.assign_add( - gbar, - gbar * (self._momentum_tensor - 1.0) + scaled_g) - else: - gbar_updated = (grad * accumulator_inv_sqrt) - - if not fallback_to_diagonal: - # Update the preconditioner statistics followed by computing the - # preconditioned gradient. - with ops.control_dependencies(precond_statistics_update): - s = tf.cast(self._run_nondiagonal_update, tf.float32) - preconditioned_grad = self._preconditioned_update( - var, partitioned_grads, gbar_updated) - # slowly adapt from diagonal to preconditioned gradient. - w = self._run_nondiagonal_update_warmup - warmup_update = s * self._learning_rate_tensor * ( - w * preconditioned_grad + (1.0 - w) * gbar_updated) - fallback_update = (1 - s) * (self._learning_rate_tensor * gbar_updated) - return state_ops.assign_sub(var, warmup_update + fallback_update) - else: - return state_ops.assign_sub(var, - self._learning_rate_tensor * gbar_updated) - - def _resource_apply_dense(self, grad, var): - return self._apply_dense(grad, var) - - # Sparse gradients are not handled currently and is part of future work. - def _resource_apply_sparse(self, grad_values, var, grad_indices): - return tf.no_op() - - def _apply_sparse(self, grad, var): - return tf.no_op() diff --git a/lingvo/core/distributed_shampoo_test.py b/lingvo/core/distributed_shampoo_test.py deleted file mode 100644 index 56fc2c90..00000000 --- a/lingvo/core/distributed_shampoo_test.py +++ /dev/null @@ -1,414 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lingvo.distributed_shampoo.""" - -from lingvo.core import distributed_shampoo -from lingvo.core import test_utils - -import numpy as np -import tensorflow.compat.v1 as tf - - -class DistributedShampooTest(test_utils.TestCase): - """A test that demonstrates the use of distributed matrix preconditioner.""" - - def testShampooWithMatrixShapedTensors(self): - # Parameter matrix of size [4,2] would result in L_{t}, and R_{t} of - # sizes [4, 4] and [2, 2] - size = [4, 2] - init_var_np = np.zeros(size) - # Initialize gradient as random tensor. - grad_np = np.random.rand(size[0], size[1]) - - with tf.Session(): - global_step = tf.Variable(0, dtype=tf.int64) - var = tf.Variable(init_var_np, dtype=tf.float32) - grad = tf.constant(grad_np, dtype=tf.float32) - - opt = distributed_shampoo.DistributedShampoo( - learning_rate=1.0, - momentum=0.0, - start_preconditioning_steps=0, - synchronous_preconditioning=True, - global_step=global_step) - - # Run a single step of gradient update. - update = opt.apply_gradients(zip([grad], [var]), global_step=global_step) - - # Preconditioner computation and assignments to variables. - compute_preconditioner_op = opt.invoke_async_preconditioner_computation( - tf.cast(global_step, tf.int32)) - assign_preconditioners_to_vars_op = ( - opt.assign_preconditioner_to_host_vars()) - - self.evaluate(tf.global_variables_initializer()) - tf.tables_initializer().run() - - init_val = self.evaluate(var) - self.assertAllCloseAccordingToType(init_var_np, init_val) - - def np_power(mat_g, alpha, matrix_epsilon=1e-6): - """Computes mat_g^alpha for a square symmetric matrix mat_g.""" - mat_for_svd = mat_g + np.eye(mat_g.shape[0]) * matrix_epsilon - mat_u, diag_d, mat_v = np.linalg.svd(mat_for_svd, full_matrices=True) - diag_d = np.power(np.maximum(diag_d, matrix_epsilon), alpha) - return np.dot(mat_u, np.dot(np.diag(diag_d), mat_v)) - - def norm(val): - return np.sqrt(np.sum(np.square(val))) - - # Run a step of preconditioner update. - update.run() - - mat_g1 = np.dot(grad_np, grad_np.transpose()) - expected_mat_g1 = self.evaluate(opt.get_slot(var, 'mat_statistics_0')) - self.assertAllCloseAccordingToType(mat_g1, expected_mat_g1, atol=1e-1) - - mat_g2 = np.dot(grad_np.transpose(), grad_np) - expected_mat_g2 = self.evaluate(opt.get_slot(var, 'mat_statistics_1')) - self.assertAllCloseAccordingToType(mat_g2, expected_mat_g2, atol=1e-1) - - compute_preconditioner_op.run() - assign_preconditioners_to_vars_op.run() - - mat_left = np_power(mat_g1, -0.25) - expected_mat_left = self.evaluate( - opt.get_slot(var, 'mat_preconditioner_0')) - self.assertAllCloseAccordingToType(mat_left, expected_mat_left, atol=1e-1) - - mat_right = np_power(mat_g2, -0.25) - expected_mat_right = self.evaluate( - opt.get_slot(var, 'mat_preconditioner_1')) - self.assertAllCloseAccordingToType( - mat_right, expected_mat_right, atol=1e-1) - - # As the preconditioners are initialized to all zero. We don't make - # any update. - var_step_0_val = self.evaluate(var) - self.assertAllCloseAccordingToType(init_var_np, var_step_0_val, atol=1e-1) - - # Run another step of training. - update.run() - var_step_1_val = self.evaluate(var) - - # New update has the scale of the second diagonal adagrad update. - adagrad_update = grad_np / np.sqrt(2 * np.square(grad_np)) - preconditioned_grad_update = np.dot(np.dot(mat_left, grad_np), mat_right) - - # With normalization by diagonal enabled. - var_step_1_np = init_var_np - preconditioned_grad_update * norm( - adagrad_update) / norm(preconditioned_grad_update) - self.assertAllCloseAccordingToType( - var_step_1_np, var_step_1_val, atol=1e-1) - - # Compute new preconditioners. - compute_preconditioner_op.run() - assign_preconditioners_to_vars_op.run() - - # Gradients are summed over time. - mat_g1 += np.dot(grad_np, grad_np.transpose()) - mat_left = np_power(mat_g1, -0.25) - expected_mat_left = self.evaluate( - opt.get_slot(var, 'mat_preconditioner_0')) - self.assertAllCloseAccordingToType(mat_left, expected_mat_left, atol=1e-1) - - mat_g2 += np.dot(grad_np.transpose(), grad_np) - mat_right = np_power(mat_g2, -0.25) - expected_mat_right = self.evaluate( - opt.get_slot(var, 'mat_preconditioner_1')) - self.assertAllCloseAccordingToType( - mat_right, expected_mat_right, atol=1e-1) - - def testShampooWithMatrixShapedTensorsRightOnlyPreconditioner(self): - # Parameter matrix of size [4,2] would result in L_{t}, and R_{t} of - # sizes [4, 4] and [2, 2]. Since max_any_dim is set to 3, it would skip - # L_{t} and only use R_{t}. The exponent in the inverse used to compute - # the preconditioner becomes -1/2. - size = [4, 2] - init_var_np = np.zeros(size) - # Initialize gradient as random tensor. - grad_np = np.random.rand(size[0], size[1]) - - with tf.Session(): - global_step = tf.Variable(0, dtype=tf.int64) - var = tf.Variable(init_var_np, dtype=tf.float32) - grad = tf.constant(grad_np, dtype=tf.float32) - - opt = distributed_shampoo.DistributedShampoo( - learning_rate=1.0, - momentum=0.0, - fallback_to_diagonal_dim=3, - start_preconditioning_steps=0, - synchronous_preconditioning=True, - global_step=global_step) - - # Run a single step of gradient update. - update = opt.apply_gradients(zip([grad], [var]), global_step=global_step) - - # Preconditioner computation and assignments to variables. - compute_preconditioner_op = opt.invoke_async_preconditioner_computation( - tf.cast(global_step, tf.int32)) - assign_preconditioners_to_vars_op = ( - opt.assign_preconditioner_to_host_vars()) - - self.evaluate(tf.global_variables_initializer()) - tf.tables_initializer().run() - - init_val = self.evaluate(var) - self.assertAllCloseAccordingToType(init_var_np, init_val) - - def np_power(mat_g, alpha, matrix_epsilon=1e-6): - """Computes mat_g^alpha for a square symmetric matrix mat_g.""" - mat_for_svd = mat_g + np.eye(mat_g.shape[0]) * matrix_epsilon - mat_u, diag_d, mat_v = np.linalg.svd(mat_for_svd, full_matrices=True) - diag_d = np.power(np.maximum(diag_d, matrix_epsilon), alpha) - return np.dot(mat_u, np.dot(np.diag(diag_d), mat_v)) - - def norm(val): - return np.sqrt(np.sum(np.square(val))) - - # Run a step of preconditioner update. - update.run() - - mat_g2 = np.dot(grad_np.transpose(), grad_np) - expected_mat_g2 = self.evaluate(opt.get_slot(var, 'mat_statistics_1')) - self.assertAllCloseAccordingToType(mat_g2, expected_mat_g2, atol=1e-1) - - compute_preconditioner_op.run() - assign_preconditioners_to_vars_op.run() - - mat_right = np_power(mat_g2, -0.5) - expected_mat_right = self.evaluate( - opt.get_slot(var, 'mat_preconditioner_1')) - self.assertAllCloseAccordingToType( - mat_right, expected_mat_right, atol=1e-1) - - # As the preconditioners are initialized to all zero. We don't make - # any update. - var_step_0_val = self.evaluate(var) - self.assertAllCloseAccordingToType(init_var_np, var_step_0_val, atol=1e-1) - - # Run another step of training. - update.run() - var_step_1_val = self.evaluate(var) - - # New update has the scale of the second diagonal adagrad update. - adagrad_update = grad_np / np.sqrt(2 * np.square(grad_np)) - preconditioned_grad_update = np.matmul(grad_np, mat_right) - - # With normalization by diagonal enabled. - var_step_1_np = init_var_np - preconditioned_grad_update * norm( - adagrad_update) / norm(preconditioned_grad_update) - - self.assertAllCloseAccordingToType( - var_step_1_np, var_step_1_val, atol=1e-1) - - # Compute new preconditioners. - compute_preconditioner_op.run() - assign_preconditioners_to_vars_op.run() - - # Gradients are summed over time. - - mat_g2 += np.dot(grad_np.transpose(), grad_np) - mat_right = np_power(mat_g2, -0.5) - expected_mat_right = self.evaluate( - opt.get_slot(var, 'mat_preconditioner_1')) - self.assertAllCloseAccordingToType( - mat_right, expected_mat_right, atol=1e-1) - - def testShampooWithMatrixShapedTensorsWithBlocks(self): - # Parameter matrix of size [4,2] would result in 4 L_{t}, and R_{t} of - # sizes [2, 2] and [2, 2]. - size = [4, 2] - init_var_np = np.zeros(size) - # Initialize gradient as random tensor. - grad_np = np.random.rand(size[0], size[1]) - - with tf.Session(): - global_step = tf.Variable(0, dtype=tf.int64) - var = tf.Variable(init_var_np, dtype=tf.float32) - grad = tf.constant(grad_np, dtype=tf.float32) - - opt = distributed_shampoo.DistributedShampoo( - learning_rate=1.0, - momentum=0.0, - block_partition_threshold_size=3, - block_size=2, - start_preconditioning_steps=0, - synchronous_preconditioning=True, - global_step=global_step) - - # Run a single step of gradient update. - update = opt.apply_gradients(zip([grad], [var]), global_step=global_step) - - # Preconditioner computation and assignments to variables. - compute_preconditioner_op = opt.invoke_async_preconditioner_computation( - tf.cast(global_step, tf.int32)) - assign_preconditioners_to_vars_op = ( - opt.assign_preconditioner_to_host_vars()) - - self.evaluate(tf.global_variables_initializer()) - tf.tables_initializer().run() - - init_val = self.evaluate(var) - self.assertAllCloseAccordingToType(init_var_np, init_val) - - def np_power(mat_g, alpha, matrix_epsilon=1e-6): - """Computes mat_g^alpha for a square symmetric matrix mat_g.""" - mat_for_svd = mat_g + np.eye(mat_g.shape[0]) * matrix_epsilon - mat_u, diag_d, mat_v = np.linalg.svd(mat_for_svd, full_matrices=True) - diag_d = np.power(np.maximum(diag_d, matrix_epsilon), alpha) - return np.dot(mat_u, np.dot(np.diag(diag_d), mat_v)) - - def norm(val): - return np.sqrt(np.sum(np.square(val))) - - # Run a step of preconditioner update. - update.run() - - block_0_grad_np = grad_np[:2, :2] - block_1_grad_np = grad_np[2:4, :2] - - block_0_mat_g1 = np.dot(block_0_grad_np, block_0_grad_np.transpose()) - expected_block_0_mat_g1 = self.evaluate( - opt.get_slot(var, '0_mat_statistics_0')) - - self.assertAllCloseAccordingToType( - block_0_mat_g1, expected_block_0_mat_g1, atol=1e-1) - - block_0_mat_g2 = np.dot(block_0_grad_np.transpose(), block_0_grad_np) - expected_block_0_mat_g2 = self.evaluate( - opt.get_slot(var, '0_mat_statistics_1')) - self.assertAllCloseAccordingToType( - block_0_mat_g2, expected_block_0_mat_g2, atol=1e-1) - - block_1_mat_g1 = np.dot(block_1_grad_np, block_1_grad_np.transpose()) - expected_block_1_mat_g1 = self.evaluate( - opt.get_slot(var, '1_mat_statistics_0')) - self.assertAllCloseAccordingToType( - block_1_mat_g1, expected_block_1_mat_g1, atol=1e-1) - - block_1_mat_g2 = np.dot(block_1_grad_np.transpose(), block_1_grad_np) - expected_block_1_mat_g2 = self.evaluate( - opt.get_slot(var, '1_mat_statistics_1')) - self.assertAllCloseAccordingToType( - block_1_mat_g2, expected_block_1_mat_g2, atol=1e-1) - - compute_preconditioner_op.run() - assign_preconditioners_to_vars_op.run() - - block_0_mat_left = np_power(block_0_mat_g1, -0.25) - expected_block_0_mat_left = self.evaluate( - opt.get_slot(var, '0_mat_preconditioner_0')) - self.assertAllCloseAccordingToType( - block_0_mat_left, expected_block_0_mat_left, atol=1e-1) - - block_0_mat_right = np_power(block_0_mat_g2, -0.25) - expected_block_0_mat_right = self.evaluate( - opt.get_slot(var, '0_mat_preconditioner_1')) - self.assertAllCloseAccordingToType( - block_0_mat_right, expected_block_0_mat_right, atol=1e-1) - - block_1_mat_left = np_power(block_1_mat_g1, -0.25) - expected_block_1_mat_left = self.evaluate( - opt.get_slot(var, '1_mat_preconditioner_0')) - self.assertAllCloseAccordingToType( - block_1_mat_left, expected_block_1_mat_left, atol=1e-1) - - block_1_mat_right = np_power(block_1_mat_g2, -0.25) - expected_block_1_mat_right = self.evaluate( - opt.get_slot(var, '1_mat_preconditioner_1')) - self.assertAllCloseAccordingToType( - block_1_mat_right, expected_block_1_mat_right, atol=1e-1) - - # As the preconditioners are initialized to all zero. We don't make - # any update. - var_step_0_val = self.evaluate(var) - self.assertAllCloseAccordingToType(init_var_np, var_step_0_val, atol=1e-1) - - # Run another step of training. - update.run() - var_step_1_val = self.evaluate(var) - - # New update has the scale of the second diagonal adagrad update. - adagrad_update = grad_np / np.sqrt(2 * np.square(grad_np)) - - block_0_update = np.dot( - np.dot(block_0_mat_left, block_0_grad_np), block_0_mat_right) - block_1_update = np.dot( - np.dot(block_1_mat_left, block_1_grad_np), block_1_mat_right) - preconditioned_grad_update = np.concatenate( - (block_0_update, block_1_update), axis=0) - # With normalization by diagonal enabled. - var_step_1_np = init_var_np - preconditioned_grad_update * norm( - adagrad_update) / norm(preconditioned_grad_update) - self.assertAllCloseAccordingToType( - var_step_1_np, var_step_1_val, atol=1e-1) - - # Compute new preconditioners. - compute_preconditioner_op.run() - assign_preconditioners_to_vars_op.run() - - # Gradients are summed over time. - block_0_mat_g1 += np.dot(block_0_grad_np, block_0_grad_np.transpose()) - block_0_mat_left = np_power(block_0_mat_g1, -0.25) - expected_block_0_mat_left = self.evaluate( - opt.get_slot(var, '0_mat_preconditioner_0')) - self.assertAllCloseAccordingToType( - block_0_mat_left, expected_block_0_mat_left, atol=1e-1) - - block_0_mat_g2 += np.dot(block_0_grad_np.transpose(), block_0_grad_np) - block_0_mat_right = np_power(block_0_mat_g2, -0.25) - expected_block_0_mat_right = self.evaluate( - opt.get_slot(var, '0_mat_preconditioner_1')) - self.assertAllCloseAccordingToType( - block_0_mat_right, expected_block_0_mat_right, atol=1e-1) - - block_1_mat_g1 += np.dot(block_1_grad_np, block_1_grad_np.transpose()) - block_1_mat_left = np_power(block_1_mat_g1, -0.25) - expected_block_1_mat_left = self.evaluate( - opt.get_slot(var, '1_mat_preconditioner_0')) - self.assertAllCloseAccordingToType( - block_1_mat_left, expected_block_1_mat_left, atol=1e-1) - - block_1_mat_g2 += np.dot(block_1_grad_np.transpose(), block_1_grad_np) - block_1_mat_right = np_power(block_1_mat_g2, -0.25) - expected_block_1_mat_right = self.evaluate( - opt.get_slot(var, '1_mat_preconditioner_1')) - self.assertAllCloseAccordingToType( - block_1_mat_right, expected_block_1_mat_right, atol=1e-1) - - -class TensorPartitionerTest(test_utils.TestCase): - """Tensor partitioner tests.""" - - def testTensorPartitioner(self): - with tf.Session(): - w1 = tf.get_variable('w1', [255, 255], tf.float32) - self.evaluate(tf.global_variables_initializer()) - partition_info = distributed_shampoo.PartitionConfig(200, 128) - grad = tf.constant(w1.eval()) - metadata = distributed_shampoo.TensorPartitioner.partition_metadata( - w1, partition_info) - partitioned_grad = distributed_shampoo.TensorPartitioner.partition_tensor( - w1, partition_info) - reformed_grad = distributed_shampoo.TensorPartitioner.reform_tensor( - partitioned_grad, metadata.num_splits_per_dim) - self.assertAllCloseAccordingToType(reformed_grad, grad) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/early_stop.py b/lingvo/core/early_stop.py deleted file mode 100644 index a7baa517..00000000 --- a/lingvo/core/early_stop.py +++ /dev/null @@ -1,206 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Early stopping based on dev-set performance.""" - -import os -import lingvo.compat as tf -from lingvo.core import hyperparams -from lingvo.core import ops - - -class MetricHistory: - """Record given metric versus global_step history to a file.""" - - @staticmethod - def SetLogdirInMetricHistories(params, logdir): - """Set the logdir member in all MetricHistory.params objects in params. - - Args: - params: global model params. - logdir: root dir for current run. - - Needs to be called by trainer once the logdir is known, but before - MetricHistory objects are constructed. - """ - for _, p in params.IterParams(): - if isinstance(p, hyperparams.Params): - try: - p.Get('name') - except AttributeError: - pass - else: - if p.name == 'MetricHistory': - p.logdir = logdir - MetricHistory.SetLogdirInMetricHistories(p, logdir) - - # Global map from jobname + metric key to class. - _metric_histories_map = {} - - @staticmethod - def _Key(jobname, metric): - """Generate a key for _metric_histories_map.""" - return jobname + '.' + metric - - @classmethod - def Params(cls): - p = hyperparams.Params() - p.Define('name', 'MetricHistory', 'Used by SetLogdirInMetricHistories.') - p.Define('jobname', 'eval_dev', 'Job and dataset to which metric applies.') - p.Define('metric', 'log_pplx', 'Metric to record.') - p.Define( - 'minimize', True, - 'If True, training minimizes the metric. If False, training ' - 'maximizes the metric.') - p.Define('logdir', '', 'Root dir for BF logs.') - p.Define( - 'tfevent_file', False, 'If True, read the metric from ' - 'events.out.tfevents.* files in the job dir instead of ' - 'maintaining a history file.') - p.Define('local_filesystem', False, - 'Logdir is on local filesystem (needed for unit test).') - return p - - def __init__(self, params): - self.params = params.Copy() - if params.tfevent_file: - self._hist_file = os.path.join(params.logdir, params.jobname, - 'events.out.tfevents*') - else: - fname = params.metric + '.history.txt' - self._hist_file = os.path.join(params.logdir, params.jobname, fname) - self._metric_histories_map[self._Key(params.jobname, params.metric)] = self - self._minimize = params.minimize - self._metric = params.metric - self._tfevent_file = params.tfevent_file - - @property - def hist_file(self): - return self._hist_file - - @property - def minimize(self): - return self._minimize - - @property - def metric(self): - return self._metric - - @property - def tfevent_file(self): - return self._tfevent_file - - @classmethod - def ConditionalAppend(cls, jobname, metric, global_step, value): - """Updates history file iff we are recording given metric and jobname.""" - key = cls._Key(jobname, metric) - if key in cls._metric_histories_map: - if cls._metric_histories_map[key].tfevent_file: - return False - cls._metric_histories_map[key].Append(global_step, value) - return True - else: - return False - - def Append(self, global_step, value): - """Updates history file with given record.""" - fname = self._hist_file - if not self.params.local_filesystem: - fname += '%r=3.2:sl=8M' - with tf.io.gfile.GFile(fname, 'a') as f: - f.write('%d %f\n' % (global_step, value)) - - -class EarlyStop: - """Early stopping based on dev-set performance. - - Factors out the steps needed to perform early stopping in the trainer when a - selected metric hasn't improved for a given number of steps. If the window - param is 0 this is guaranteed to be a no-op. - """ - - @classmethod - def Params(cls): - p = hyperparams.Params() - p.Define('name', 'EarlyStop', '') - p.Define('metric_history', MetricHistory.Params(), 'Metric history params.') - p.Define( - 'tolerance', 0.0, 'Minimum significant difference in metric; ' - 'useful if progress is asymptotic.') - p.Define('window', 0, 'Maximum number of steps between best and current.') - p.Define('verbose', True, 'Log early-stop checks.') - p.Define('min_steps', 0, 'Minimum number of steps before stopping.') - return p - - def __init__(self, params): - self.params = params.Copy() - if self.params.window: - self._metric_history = MetricHistory(self.params.metric_history) - else: - self._metric_history = None - self._node = None - self._best_step = 0 - self._last_step = 0 - - @property - def metric_history(self): - return self._metric_history - - @property - def best_step(self): - return self._best_step - - @property - def last_step(self): - return self._last_step - - def FProp(self, theta): - """Creates an op to determine the best step from the metric history file. - - Args: - theta: Not currently used. - Returns: - The created op. - - This uses BestStepOp rather than reading the file directly from python in - order to ensure compatibility with DevBasedSchedule for learning-rate decay. - It is natural to use dev-based decay and early stopping together, for - example decaying when dev-set perplexity hasn't improved for n steps, and - stopping when it hasn't improved for 3n steps. - """ - del theta # not used - if self.params.window: - self._node = ops.best_step(self.metric_history.hist_file, - self.params.tolerance, - self.metric_history.minimize, - self.metric_history.metric) - else: - self._node = None - return self._node - - def Stop(self, session): - """Returns true if stop criterion is met.""" - if self.params.window and self._node is not None: - self._best_step, self._last_step = session.run(self._node) - s = ( - self._last_step - self._best_step > self.params.window and - self._last_step >= self.params.min_steps) - if self.params.verbose: - tf.logging.info( - 'early stop check: best_step=%d, last_step=%d, stop=%d', - self._best_step, self._last_step, s) - return s - else: - return False diff --git a/lingvo/core/early_stop_test.py b/lingvo/core/early_stop_test.py deleted file mode 100644 index f6c800e9..00000000 --- a/lingvo/core/early_stop_test.py +++ /dev/null @@ -1,211 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for early_stop.""" - -import os -import lingvo.compat as tf -from lingvo.core import early_stop -from lingvo.core import hyperparams -from lingvo.core import test_helper -from lingvo.core import test_utils - - -class MetricHistoryTest(test_utils.TestCase): - - def setUp(self): - super().setUp() - early_stop.MetricHistory._metric_histories_map = {} - - def testSetLogdirInMetricHistories(self): - p = hyperparams.Params() - p.Define('name', 'testparams', 'test params') - p.Define('logdir', None, 'dummy logdir') - p.Define('mh1', early_stop.MetricHistory.Params(), 'history1') - p2 = hyperparams.Params() - p2.Define('mh2', early_stop.MetricHistory.Params(), 'history2') - p.Define('subparams', p2, 'subparams') - - early_stop.MetricHistory.SetLogdirInMetricHistories(p, 'dir') - - self.assertEqual(p.mh1.logdir, 'dir') - self.assertEqual(p.subparams.mh2.logdir, 'dir') - self.assertNotEqual(p.logdir, 'dir') - - def testMetricHistoriesMapUniqueness(self): - # pylint: disable=unused-variable - p = early_stop.MetricHistory.Params() - mh1 = early_stop.MetricHistory(p.Set(jobname='job1', metric='m1')) - mh2 = early_stop.MetricHistory(p.Set(jobname='job2', metric='m2')) - mh3 = early_stop.MetricHistory(p.Set(jobname='job1', metric='m1')) - - m = early_stop.MetricHistory._metric_histories_map - self.assertEqual(len(m), 2) - self.assertEqual(m[early_stop.MetricHistory._Key('job1', 'm1')], mh3) - self.assertEqual(m[early_stop.MetricHistory._Key('job2', 'm2')], mh2) - - def testMetricHistoriesFiles(self): - logdir = tf.test.get_temp_dir() - tf.io.gfile.mkdir(os.path.join(logdir, 'job1')) - tf.io.gfile.mkdir(os.path.join(logdir, 'job2')) - - p = early_stop.MetricHistory.Params().Set(logdir=logdir) - mh1 = early_stop.MetricHistory( - p.Set(jobname='job1', metric='m1', local_filesystem=True)) - mh2 = early_stop.MetricHistory( - p.Set(jobname='job2', metric='m2', local_filesystem=True)) - - early_stop.MetricHistory.ConditionalAppend('job1', 'm1', 1, 10.0) - early_stop.MetricHistory.ConditionalAppend('job1', 'm2', 1, 10.0) - early_stop.MetricHistory.ConditionalAppend('job2', 'm2', 1, 10.0) - early_stop.MetricHistory.ConditionalAppend('job1', 'm1', 2, 5.0) - - self.assertTrue(tf.io.gfile.exists(mh1.hist_file)) - self.assertTrue(tf.io.gfile.exists(mh2.hist_file)) - with tf.io.gfile.GFile(mh1.hist_file) as f: - lines = f.readlines() - self.assertEqual(len(lines), 2) - self.assertEqual(lines[0].rstrip(), '1 10.000000') - self.assertEqual(lines[1].rstrip(), '2 5.000000') - with tf.io.gfile.GFile(mh2.hist_file) as f: - lines = f.readlines() - self.assertEqual(len(lines), 1) - self.assertEqual(lines[0].rstrip(), '1 10.000000') - - -class EarlyStopTest(test_utils.TestCase): - - def setUp(self): - super().setUp() - early_stop.MetricHistory._metric_histories_map = {} - - def testEarlyStopDefaultIsNoOp(self): - p = early_stop.EarlyStop.Params() - es = early_stop.EarlyStop(p) - es.FProp(None) - mh = early_stop.MetricHistory - a = mh.ConditionalAppend(es.params.metric_history.jobname, - es.params.metric_history.metric, 1, 10.0) - s = es.Stop(None) - - self.assertFalse(a) - self.assertFalse(s) - self.assertIsNone(es._node) - self.assertEqual(len(early_stop.MetricHistory._metric_histories_map), 0) - - def testEarlyStopping(self): - logdir = tf.test.get_temp_dir() - tf.io.gfile.mkdir(os.path.join(logdir, 'eval_dev')) - - p = early_stop.EarlyStop.Params() - p.window = 2 - p.tolerance = 1.0 - p.metric_history.local_filesystem = True - early_stop.MetricHistory.SetLogdirInMetricHistories(p, logdir) - - es = early_stop.EarlyStop(p) - es.FProp(None) - with self.session() as sess: - jobname = es.metric_history.params.jobname - metric = es.metric_history.params.metric - mh = early_stop.MetricHistory - - mh.ConditionalAppend(jobname, metric, 1, 10.0) - self.assertFalse(es.Stop(sess)) - self.assertEqual(es.best_step, 1) - self.assertEqual(es.last_step, 1) - - mh.ConditionalAppend(jobname, metric, 2, 5.0) - self.assertFalse(es.Stop(sess)) - self.assertEqual(es.best_step, 2) - self.assertEqual(es.last_step, 2) - - mh.ConditionalAppend(jobname, metric, 3, 4.0) - self.assertFalse(es.Stop(sess)) - self.assertEqual(es.best_step, 2) - self.assertEqual(es.last_step, 3) - - mh.ConditionalAppend(jobname, metric, 5, 4.0) - self.assertTrue(es.Stop(sess)) - self.assertEqual(es.best_step, 2) - self.assertEqual(es.last_step, 5) - - def testEarlyStoppingAscendingMetric(self): - logdir = tf.test.get_temp_dir() - tf.io.gfile.mkdir(os.path.join(logdir, 'decoder_dev')) - - p = early_stop.EarlyStop.Params() - p.window = 2 - p.tolerance = 1.0 - p.metric_history.local_filesystem = True - p.metric_history.minimize = False - p.metric_history.jobname = 'decoder_dev' - p.metric_history.metric = 'canonical_bleu' - early_stop.MetricHistory.SetLogdirInMetricHistories(p, logdir) - - es = early_stop.EarlyStop(p) - es.FProp(None) - with self.session() as sess: - jobname = es.metric_history.params.jobname - metric = es.metric_history.params.metric - mh = early_stop.MetricHistory - - mh.ConditionalAppend(jobname, metric, 1, 0.0) - self.assertFalse(es.Stop(sess)) - self.assertEqual(es.best_step, 1) - self.assertEqual(es.last_step, 1) - - mh.ConditionalAppend(jobname, metric, 2, 1.0) - self.assertFalse(es.Stop(sess)) - self.assertEqual(es.best_step, 1) - self.assertEqual(es.last_step, 2) - - mh.ConditionalAppend(jobname, metric, 3, 2.5) - self.assertFalse(es.Stop(sess)) - self.assertEqual(es.best_step, 3) - self.assertEqual(es.last_step, 3) - - mh.ConditionalAppend(jobname, metric, 5, 2.0) - self.assertFalse(es.Stop(sess)) - self.assertEqual(es.best_step, 3) - self.assertEqual(es.last_step, 5) - - mh.ConditionalAppend(jobname, metric, 6, 1.0) - self.assertTrue(es.Stop(sess)) - self.assertEqual(es.best_step, 3) - self.assertEqual(es.last_step, 6) - - def testEarlyStoppingAscendingTfEvents(self): - logdir = test_helper.test_src_dir_path('core/ops') - p = early_stop.EarlyStop.Params() - p.window = 1000 - p.tolerance = 0.0 - p.metric_history.local_filesystem = True - p.metric_history.minimize = False - p.metric_history.jobname = 'testdata' - p.metric_history.metric = 'bleu/dev' - p.metric_history.tfevent_file = True - early_stop.MetricHistory.SetLogdirInMetricHistories(p, logdir) - - es = early_stop.EarlyStop(p) - es.FProp(None) - with self.session() as sess: - self.assertTrue(es.Stop(sess)) - self.assertEqual(es.best_step, 102600) - self.assertEqual(es.last_step, 185200) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/egdd.py b/lingvo/core/egdd.py deleted file mode 100644 index c8f0ef04..00000000 --- a/lingvo/core/egdd.py +++ /dev/null @@ -1,181 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Exponentiated Gradient Delta-Delta optimizer.""" - -# pylint: disable=g-direct-tensorflow-import -from tensorflow.python.framework import ops -from tensorflow.python.ops import array_ops -from tensorflow.python.ops import clip_ops -from tensorflow.python.ops import control_flow_ops -from tensorflow.python.ops import linalg_ops -from tensorflow.python.ops import math_ops -from tensorflow.python.ops import state_ops -from tensorflow.python.training import optimizer -# pylint: enable=g-direct-tensorflow-import - - -class EGDD(optimizer.Optimizer): - """A version of GD Momentum with adaptive gain and learning rate. - - Exponentiated Gradient Delta-delta optimizer starts with a local gain of 1.0 - for every weight and a lr_scale of 1.0 for all weights. The EGDD update rule - applies: - - momentum <- mu * momentum + learning_rate * gain * grad - var <- var - lr_scale * momentum - - The gain as well as the lr_scale are updated using the unnormalized - exponentiated gradient algorithm [KW97]. - - Reference: TBA - - [KW97] Kivinen, J., & Warmuth, M. K. Exponentiated gradient versus gradient - descent for linear predictors. Information and Computation, 1997. - """ - - def __init__(self, - learning_rate, - momentum, - beta=0.9, - gain_learning_rate=0.01, - scale_learning_rate=0.001, - initial_gain=1.0, - min_gain=1e-2, - max_gain=1e2, - initial_scale=1.0, - min_scale=1e-1, - max_scale=1e1, - use_directions=True, - use_signs=True, - name="EGDD"): - """Construct a new EG-DD optimizer. - - Args: - learning_rate: A `Tensor` or a floating point value. The learning rate. - momentum: A `Tensor` or a floating point value. - beta: `float` decay rate of the gradient EMA. - gain_learning_rate: `float` gain learning rate. - scale_learning_rate: `float` scale learning rate. - initial_gain: `float` initial gain. - min_gain: `float` minimum gain. - max_gain: `float` maximum gain, - initial_scale: `float` initial scale. - min_scale: `float` minimum learning rate scale. - max_scale: `float` maximum learning rate scale. - use_directions: `bool` whether to use directions only for scale updates. - use_signs: `bool` whether to use the signs for updating gains. - name: Optional name prefix for the operations created when applying - gradients. - - Raises: - ValueError: If the `initial_accumulator_value` is invalid. - - """ - super().__init__(False, name) - self._learning_rate = learning_rate - self._momentum = momentum - self._beta = beta - self._gain_learning_rate = gain_learning_rate - self._scale_learning_rate = scale_learning_rate - self._initial_gain = initial_gain - self._min_gain = min_gain - self._max_gain = max_gain - self._initial_scale = initial_scale - self._min_scale = min_scale - self._max_scale = max_scale - self._use_directions = use_directions - self._use_signs = use_signs - - def _create_slots(self, var_list): - for v in var_list: - self._zeros_slot(v, "momentum", self._name) - self._zeros_slot(v, "gbar", self._name) - g_tensor = ops.convert_to_tensor(v) - gain_init = self._initial_gain * array_ops.ones_like(g_tensor) - _ = self._get_or_make_slot(v, self._initial_scale * array_ops.ones((1)), - "lr_scale", self._name) - _ = self._get_or_make_slot(v, gain_init, "gain", self._name) - _ = self._get_or_make_slot(v, array_ops.zeros((1)), "counter", self._name) - - def _prepare(self): - learning_rate = self._call_if_callable(self._learning_rate) - self._learning_rate_tensor = ops.convert_to_tensor( - learning_rate, name="learning_rate") - momentum = self._call_if_callable(self._momentum) - self._momentum_tensor = ops.convert_to_tensor(momentum, name="momentum") - - def _apply_dense(self, grad, var): - lr_scale = self.get_slot(var, "lr_scale") - momentum = self.get_slot(var, "momentum") - gbar = self.get_slot(var, "gbar") - gain = self.get_slot(var, "gain") - counter = self.get_slot(var, "counter") - counter_updated = state_ops.assign(counter, counter + 1) - - # lr_scale update uses normalized grad and momentum to be independent of dim - normalized_grad = grad / (linalg_ops.norm(grad) + 1e-10) - normalized_momentum = momentum / (linalg_ops.norm(momentum) + 1e-10) - # Apply EG updates on lr_scale: - # grad_lr_scale = -inner_product(current_grad, old_momentum) - # lr_scale <- lr_scale * exp(-scale_learning_rate * grad_lr_scale) - lr_scale_unnormalized_updated = clip_ops.clip_by_value( - lr_scale * math_ops.exp( - self._scale_learning_rate * math_ops.reduce_sum(grad * momentum)), - self._min_scale, self._max_scale) - lr_scale_normalized_updated = clip_ops.clip_by_value( - lr_scale * math_ops.exp(self._scale_learning_rate * math_ops.reduce_sum( - normalized_grad * normalized_momentum)), self._min_scale, - self._max_scale) - lr_scale_updated = state_ops.assign( - lr_scale, - array_ops.where(self._use_directions, lr_scale_normalized_updated, - lr_scale_unnormalized_updated)) - # remove the bias of zero initialization in gbar - corrected_gbar = gbar / ( - 1.0 - self._beta**math_ops.maximum(counter_updated - 1, 1)) - # Apply EG updates on gain: - # grad_gain = - current_grad * old_gbar - # gain <- gain * exp(-gain_learning_rate * grad_gain) - gain_unnormalized_updated = clip_ops.clip_by_value( - gain * math_ops.exp(self._gain_learning_rate * grad * corrected_gbar), - self._min_gain, self._max_gain) - # Normalized update uses sign(grad) * sign(gbar) as a proxy for grad_gain. - gain_normalized_updated = clip_ops.clip_by_value( - gain * math_ops.exp(self._gain_learning_rate * math_ops.sign(grad) * - math_ops.sign(gbar)), self._min_gain, - self._max_gain) - gain_updated = state_ops.assign( - gain, - array_ops.where(self._use_signs, gain_normalized_updated, - gain_unnormalized_updated)) - scaled_g = self._learning_rate_tensor * gain_updated * grad - with ops.control_dependencies([lr_scale_updated, scaled_g]): - momentum_updated = state_ops.assign( - momentum, self._momentum_tensor * momentum + scaled_g) - gbar_updated = state_ops.assign( - gbar, self._beta * gbar + (1.0 - self._beta) * grad) - with ops.control_dependencies([gbar_updated]): - return state_ops.assign_sub(var, lr_scale_updated * momentum_updated) - - def _resource_apply_dense(self, grad, var): - return self._apply_dense(grad, var) - - # Sparse gradients are not handled currently and is part of future work. - def _resource_apply_sparse(self, grad_values, var, grad_indices): - return control_flow_ops.no_op() - - def _apply_sparse(self, grad, var): - return control_flow_ops.no_op() diff --git a/lingvo/core/egdd_test.py b/lingvo/core/egdd_test.py deleted file mode 100644 index e537fdd3..00000000 --- a/lingvo/core/egdd_test.py +++ /dev/null @@ -1,114 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for EG-DD optimizer.""" -from lingvo import compat as tf -from lingvo.core import egdd -from lingvo.core import test_utils - - -class EGDD(test_utils.TestCase): - - def testDenseLayer(self): - """EG-DD update.""" - - with self.cached_session() as sess: - var = tf.Variable([0.5, 1.0]) - grad = tf.placeholder(tf.float32, shape=[2]) - opt = egdd.EGDD( - learning_rate=0.1, - momentum=0.9, - beta=0.1, - gain_learning_rate=1e-2, - scale_learning_rate=1e-3, - use_signs=False) - - step = opt.apply_gradients([(grad, var)]) - tf.global_variables_initializer().run() - - pre_var = sess.run(var) - pre_momentum = sess.run(opt.get_slot(var, 'momentum')) - pre_gain = sess.run(opt.get_slot(var, 'gain')) - pre_lr_scale = sess.run(opt.get_slot(var, 'lr_scale')) - self.assertAllClose([0.5, 1.0], pre_var) - self.assertAllClose([0.0, 0.0], pre_momentum) - self.assertAllClose([1.0, 1.0], pre_gain) - self.assertAllClose([1.0], pre_lr_scale) - sess.run(step, feed_dict={grad: [0.1, -0.5]}) - pre_var = sess.run(var) - pre_momentum = sess.run(opt.get_slot(var, 'momentum')) - pre_gain = sess.run(opt.get_slot(var, 'gain')) - pre_lr_scale = sess.run(opt.get_slot(var, 'lr_scale')) - self.assertAllClose([0.49, 1.05], pre_var) - self.assertAllClose([0.01, -0.05], pre_momentum) - self.assertAllClose([1, 1], pre_gain) - self.assertAllClose([1.0], pre_lr_scale) - sess.run(step, feed_dict={grad: [-1.0, -1.5]}) - pre_var = sess.run(var) - pre_momentum = sess.run(opt.get_slot(var, 'momentum')) - pre_gain = sess.run(opt.get_slot(var, 'gain')) - pre_lr_scale = sess.run(opt.get_slot(var, 'lr_scale')) - self.assertAllClose([0.5810, 1.2463], pre_var, atol=1e-4) - self.assertAllClose([-0.0909, -0.1961], pre_momentum, atol=1e-4) - self.assertAllClose([0.9990, 1.0075], pre_gain, atol=1e-4) - self.assertAllClose([1.0007], pre_lr_scale, atol=1e-4) - - def testDenseLayerSigns(self): - """EG-DD update.""" - - with self.cached_session() as sess: - var = tf.Variable([0.5, 1.0]) - grad = tf.placeholder(tf.float32, shape=[2]) - opt = egdd.EGDD( - learning_rate=0.1, - momentum=0.9, - beta=0.1, - gain_learning_rate=1e-2, - scale_learning_rate=1e-3, - use_signs=True) - - step = opt.apply_gradients([(grad, var)]) - tf.global_variables_initializer().run() - - pre_var = sess.run(var) - pre_momentum = sess.run(opt.get_slot(var, 'momentum')) - pre_gain = sess.run(opt.get_slot(var, 'gain')) - pre_lr_scale = sess.run(opt.get_slot(var, 'lr_scale')) - self.assertAllClose([0.5, 1.0], pre_var) - self.assertAllClose([0.0, 0.0], pre_momentum) - self.assertAllClose([1.0, 1.0], pre_gain) - self.assertAllClose([1.0], pre_lr_scale) - sess.run(step, feed_dict={grad: [0.1, -0.5]}) - pre_var = sess.run(var) - pre_momentum = sess.run(opt.get_slot(var, 'momentum')) - pre_gain = sess.run(opt.get_slot(var, 'gain')) - pre_lr_scale = sess.run(opt.get_slot(var, 'lr_scale')) - self.assertAllClose([0.49, 1.05], pre_var) - self.assertAllClose([0.01, -0.05], pre_momentum) - self.assertAllClose([1, 1], pre_gain) - self.assertAllClose([1.0], pre_lr_scale) - sess.run(step, feed_dict={grad: [-1.0, -1.5]}) - pre_var = sess.run(var) - pre_momentum = sess.run(opt.get_slot(var, 'momentum')) - pre_gain = sess.run(opt.get_slot(var, 'gain')) - pre_lr_scale = sess.run(opt.get_slot(var, 'lr_scale')) - self.assertAllClose([0.5801, 1.2466], pre_var, atol=1e-4) - self.assertAllClose([-0.0900, -0.1965], pre_momentum, atol=1e-4) - self.assertAllClose([0.9900, 1.0101], pre_gain, atol=1e-4) - self.assertAllClose([1.0007], pre_lr_scale, atol=1e-4) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/ema_test.py b/lingvo/core/ema_test.py deleted file mode 100644 index 61161657..00000000 --- a/lingvo/core/ema_test.py +++ /dev/null @@ -1,142 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for ExponentialMovingAverage support in Lingvo.""" - -import os -import lingvo.compat as tf -from lingvo.core import base_decoder -from lingvo.core import base_input_generator -from lingvo.core import base_model -from lingvo.core import checkpointer -from lingvo.core import layers -from lingvo.core import py_utils -from lingvo.core import test_utils -import numpy as np - - -class TestTask(base_model.BaseTask): - """A task with a single 'encoder' child layer.""" - - def __init__(self, params): - super().__init__(params) - p = self.params - self.CreateChild('encoder', p.encoder) - - -class EmaTest(test_utils.TestCase): - - @classmethod - def TestParams(cls, encoder_params): - p = TestTask.Params() - p.name = 'base_mdl' - p.input = base_input_generator.BaseSequenceInputGenerator.Params() - p.encoder = encoder_params - p.decoder = base_decoder.BaseDecoder.Params() - return p - - def testBatchNormLayer(self): - p = base_model.SingleTaskModel.Params() - p.task = self.TestParams(layers.BatchNormLayer.Params().Set(dim=1)) - p.task.train.ema_decay = 0.9 - p.task.train.ema_decay_moving_vars = True - model = p.Instantiate() - self.assertIsNotNone(model.ema) - task = model._task - task._train_op = tf.no_op() - task.ApplyExponentialMovingAverage(model.ema) - - layer = task.encoder - self.assertLen(layer.vars, 4) - for var in layer.vars.Flatten(): - self.assertIsNotNone(model.ema.average(var), msg=var.name) - beta = layer.vars.beta - mean = layer.vars.moving_mean - - global_step = 100 - beta_1 = np.asarray([.2]) - mean_1 = np.asarray([.03]) - beta_1_ema = beta_1 * .1 - mean_1_ema = mean_1 * .1 - with self.session() as sess: - # Test EMA values. - self.evaluate(tf.global_variables_initializer()) - self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), global_step)) - self.evaluate(tf.assign(beta, beta_1)) - self.evaluate(tf.assign(mean, mean_1)) - self.evaluate(task._post_train_ops) - - self.assertAllClose([beta_1, beta_1_ema, mean_1, mean_1_ema], - self.evaluate([ - beta, - model.ema.average(beta), mean, - model.ema.average(mean) - ])) - - # Test checkpointer. - train_dir = os.path.join(self.get_temp_dir(), 'testSaveRestore') - os.mkdir(train_dir) - saver = checkpointer.Checkpointer(train_dir, model) - saver.Save(sess, model.global_step) - - self.assertTrue( - os.path.isfile( - os.path.join(train_dir, 'ckpt-%08d.index' % global_step))) - - # Restore from ckpt in training mode. - with self.session(graph=tf.Graph()) as sess: - model = p.Instantiate() - self.assertIsNotNone(model.ema) - task = model._task - task._train_op = tf.no_op() - task.ApplyExponentialMovingAverage(model.ema) - layer = task.encoder - for var in layer.vars.Flatten(): - self.assertIsNotNone(model.ema.average(var), msg=var.name) - beta = layer.vars.beta - mean = layer.vars.moving_mean - - saver = checkpointer.Checkpointer(train_dir, model) - saver.RestoreIfNeeded(sess) - - self.assertAllClose([beta_1, beta_1_ema, mean_1, mean_1_ema], - self.evaluate([ - beta, - model.ema.average(beta), mean, - model.ema.average(mean) - ])) - - # Restore from ckpt in eval mode. - with self.session(graph=tf.Graph()) as sess, self.SetEval(True): - model = p.Instantiate() - self.assertIsNotNone(model.ema) - task = model._task - # task._train_op = tf.no_op() - # task.ApplyExponentialMovingAverage(model.ema) - layer = task.encoder - # for var in layer.vars.Flatten(): - # self.assertIsNotNone(model.ema.average(var), msg=var.name) - beta = layer.vars.beta - mean = layer.vars.moving_mean - - saver = checkpointer.Checkpointer(train_dir, model) - saver.RestoreIfNeeded(sess) - - # Both beta and mean should use the EMA value. - self.assertAllClose([beta_1_ema, mean_1_ema], self.evaluate([beta, mean])) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/generic_input.py b/lingvo/core/generic_input.py deleted file mode 100644 index be6693cc..00000000 --- a/lingvo/core/generic_input.py +++ /dev/null @@ -1,170 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Generic input.""" - -import lingvo.compat as tf -from lingvo.core import ops -from lingvo.core import py_utils -from tensorflow.python.util import tf_inspect # pylint: disable=g-direct-tensorflow-import - - -def GenericInput(processor, **kwargs): - """Builds a generic input pipeline. - - Example usage:: - - def ParseRecord(record): - # Given a tf.string record, return a (NestedMap, bucketing key) pair. - feature_map = ... - features = tf.io.parse_single_example(record, feature_map) - # Each example is represented by a NestedMap of tensors (without a - # batch dimension). - example = py_utils.NestedMap(field1=..., field2=...) - # bucketing_key is a scalar convertible to tf.int32. - # Use 1 if all examples are of the same size. - bucketing_key = 1 - return example, bucketing_key - - input_batch, bucket_keys = GenericInput(ParseRecord, file_pattern=..., ...) - # input_batch is a NestedMap of tensors, where dim 0 of each tensor - # represents the batch dimension. - input_batch.field1 = ... - - ParseRecord can also take both 'source_id' and 'record' as inputs (the arg - names must be exactly 'source_id' and 'record'): - - def ParseRecord(source_id, record): - # Given a tf.int32 source_id and a tf.string record, return a (NestedMap, - # bucketing key) pair. - example = py_utils.NestedMap(source_id=source_id, ...) - ... - return example, bucketing_key - - input_batch, bucket_keys = GenericInput(ParseRecord, file_pattern=..., ...) - - Args: - processor: a function that takes either a tf.string record or a - (source_id: tf.int32, record: tf.string) pair as input and returns a tuple - (output, bucketing_key). `output` must be a NestedMap or a list of tensors - representing an example. `bucketing_key` must be a scalar convertible to - a tf.int32 tensor that represents the bucketing key (e.g., sequence - length for sequence inputs). If `bucketing_key` is a negative number, - the record is dropped. - **kwargs: additional keyword args for x_ops.generic_input. - - Returns: - A tuple of (outputs, bucket_keys): - - - outputs: a NestedMap or a list of tensors, similar to `processor`'s - return, except every tensor will have an additional dimension 0 that - represents the batch dimension. - - bucket_keys: a tf.int32 vector. - """ - output_tmpl = py_utils.NestedMap() - - @tf.function(autograph=False) - def _FlatOutputProcessor(source_id, record): - """Returns a flattened list of 'processor(inputs)'.""" - processor_spec = tf_inspect.getargspec(processor) - tf.logging.debug('GenericInput.processor.argspec=%s', processor_spec) - processor_args = set(processor_spec.args) - set(['self']) - if len(processor_args) == 1: - output, bucketing_key = processor(record) - elif processor_args == set(['source_id', 'record']): - output, bucketing_key = processor(source_id=source_id, record=record) - else: - raise ValueError( - 'GenericInput: processor should take either a single arg ' - 'or two args named as "source_id" and "record". ' - 'Actual: %s' % processor_args) - if isinstance(output, list): - assert output - assert all(isinstance(x, tf.Tensor) for x in output), '{}'.format(output) - else: - assert isinstance(output, py_utils.NestedMap), '{}'.format(output) - assert output - assert all( - isinstance(x, tf.Tensor) for x in output.Flatten()), '{}'.format( - output.DebugString()) - bucketing_key = tf.cast(bucketing_key, tf.int32) - tf.logging.debug('Processor outputs=%s bucketing_key=%s', output, - bucketing_key) - output_tmpl.out_values = output - flat_output_tmpl = output_tmpl.Flatten() - tf.logging.debug('Processor flat outputs=%s', flat_output_tmpl) - tf.logging.debug('extra_inputs=%s extra_args=%s extra_vars=%s', - py_utils.GetExtraInputs(), py_utils.GetExtraArgs(), - py_utils.GetExtraVars()) - assert not py_utils.GetExtraArgs(), ( - 'fns {} is not pure: extra_args={}'.format(processor, - py_utils.GetExtraArgs())) - return flat_output_tmpl + [bucketing_key] - - proc_fn = _FlatOutputProcessor.get_concrete_function( - tf.TensorSpec([], tf.int32), tf.TensorSpec([], tf.string)) - - out_types = [ - tf.DType(a.type) for a in proc_fn.function_def.signature.output_arg - ] - assert out_types[-1] == tf.int32, ('%s is not expected.' % out_types[-1]) - flat_outputs, bucket_keys = ops.gen_x_ops.generic_input( - processor=proc_fn, out_types=out_types[:-1], **kwargs) - tf.logging.debug('x_ops.generic_input flat_outputs=%s', flat_outputs) - # Pack flat_outputs to outputs. - outputs = output_tmpl.Pack(flat_outputs).out_values - tf.logging.debug('x_ops.generic_input outputs=%s', outputs) - return outputs, bucket_keys - - -def ReplicatedGenericInput(processor, num_replicas, replica_device_fn, - **kwargs): - """Builds a replicated input pipeline. - - This is similar to GenericInput, except that the input processing can be - distributed across devices and then concatenated at the current device. - - Args: - processor: see comments for GenericInput. - num_replicas: the number of input processing replicas. Usually set to number - of infeed hosts. - replica_device_fn: a int -> string function that takes the replica index in - range [0, num_replicas) and returns a TF device string, e.g., - lambda i: '/task:{}/device:CPU:0'.format(i) - **kwargs: additional keyword args for x_ops.generic_input. - - Returns: - A tuple of (outputs, bucket_keys): - - - outputs: a NestedMap or a list of tensors, similar to `processor`'s - return, except every tensor will have an additional dimension 0 that - represents the batch dimension. The batch size will be - (num_replicas * bucket_batch_limit[...]), i.e., - kwargs['bucket_batch_limit'] specifies the per-replica batch size. - - bucket_keys: a tf.int32 vector. - """ - if num_replicas > 1 and 'bucket_batch_limit' in kwargs: - assert all(b == max(kwargs['bucket_batch_limit']) - for b in kwargs['bucket_batch_limit']) - replica_outputs = [] - for replica_i in range(num_replicas): - replica_device = replica_device_fn(replica_i) - with tf.device(replica_device): - replica_outputs.append(GenericInput(processor, **kwargs)) - output_nmaps, output_bucket_keys = zip(*replica_outputs) - concat_nmap = tf.nest.map_structure(lambda *t: tf.concat(t, axis=0), - *output_nmaps) - concat_bucket_keys = tf.concat(output_bucket_keys, axis=0) - return concat_nmap, concat_bucket_keys diff --git a/lingvo/core/generic_input_test.py b/lingvo/core/generic_input_test.py deleted file mode 100644 index 5c8535c3..00000000 --- a/lingvo/core/generic_input_test.py +++ /dev/null @@ -1,300 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for generic_input_op.""" - -import collections -import os -import pickle -from absl.testing import parameterized -from lingvo import compat as tf -from lingvo.core import generic_input -from lingvo.core import py_utils -from lingvo.core import test_utils -import numpy as np - - -def get_test_input(path, bucket_batch_limit=8, **kwargs): - return generic_input.GenericInput( - file_pattern='tfrecord:' + path, - file_random_seed=0, - file_buffer_size=32, - file_parallelism=4, - bucket_batch_limit=[bucket_batch_limit], - **kwargs) - - -def run_basic_graph(use_nested_map, - bucket_fn=lambda x: 1, - bucket_batch_limit=8): - # Generate a test file w/ 100 records. - tmp = os.path.join(tf.test.get_temp_dir(), 'basic') - with tf.python_io.TFRecordWriter(tmp) as w: - for i in range(100): - w.write(('%08d' % i).encode('utf-8')) - - # A simple string parsing routine. Just convert a string to a - # number. - def str_to_num(s): - return np.array(float(s), dtype=np.float32) - - # A record processor written in TF graph. - def _process(source_id, record): - num, = tf.py_func(str_to_num, [record], [tf.float32]) - num = tf.stack([num, tf.square(num)]) - if use_nested_map: - return py_utils.NestedMap( - source_id=source_id, record=record, num=num), bucket_fn(num) - else: - return [source_id, record, num], bucket_fn(num) - - # Samples random records from the data files and processes them - # to generate batches. - inputs, _ = get_test_input( - tmp, - bucket_batch_limit=bucket_batch_limit, - bucket_upper_bound=[1], - processor=_process) - if use_nested_map: - return inputs - else: - src_ids, strs, vals = inputs - return py_utils.NestedMap(source_id=src_ids, record=strs, num=vals) - - -class GenericInputOpTest(test_utils.TestCase, parameterized.TestCase): - - @parameterized.named_parameters(('OutputList', False, 8), - ('OutputNestedMap', True, 8), - ('OutputNestedMap_Batch1', True, 1)) - def testBasic(self, use_nested_map, bucket_batch_limit): - input_batch = run_basic_graph( - use_nested_map=use_nested_map, bucket_batch_limit=bucket_batch_limit) - with self.session(): - record_seen = set() - for i in range(100): - ans_input_batch = self.evaluate(input_batch) - for s in ans_input_batch.record: - record_seen.add(s) - self.assertEqual(ans_input_batch.source_id.shape, (bucket_batch_limit,)) - self.assertEqual(ans_input_batch.record.shape, (bucket_batch_limit,)) - self.assertEqual(ans_input_batch.num.shape, (bucket_batch_limit, 2)) - ans_vals = ans_input_batch.num - self.assertAllEqual(np.square(ans_vals[:, 0]), ans_vals[:, 1]) - for i in range(100): - self.assertIn(('%08d' % i).encode('utf-8'), record_seen) - - def testPadding(self): - # Generate a test file w/ 50 records of different lengths. - tmp = os.path.join(tf.test.get_temp_dir(), 'basic') - with tf.python_io.TFRecordWriter(tmp) as w: - for n in range(1, 50): - w.write(pickle.dumps(np.full([n, 3, 3], n, np.int32))) - - g = tf.Graph() - with g.as_default(): - # A record processor written in TF graph. - def _process(record): - num = tf.py_func(pickle.loads, [record], tf.int32) - bucket_key = tf.shape(num)[0] - return [num, tf.transpose(num, [1, 0, 2])], bucket_key - - # Samples random records from the data files and processes them - # to generate batches. - (vals_t, transposed_vals_t), _ = get_test_input( - tmp, - bucket_upper_bound=[10], - processor=_process, - dynamic_padding_dimensions=[0, 1], - dynamic_padding_constants=[0] * 2) - - with self.session(graph=g): - for _ in range(10): - vals, transposed_vals = self.evaluate([vals_t, transposed_vals_t]) - print(vals, np.transpose(transposed_vals, [0, 2, 1, 3])) - self.assertEqual(vals.shape[0], 8) - self.assertEqual(vals.shape[2], 3) - self.assertEqual(vals.shape[3], 3) - largest = np.amax(vals) - self.assertLessEqual(largest, 10) - self.assertEqual(vals.shape[1], largest) - for j in range(8): - n = vals[j, 0, 0, 0] - self.assertTrue(np.all(vals[j, :n] == n)) - self.assertTrue(np.all(vals[j, n:] == 0)) - self.assertAllEqual(vals, np.transpose(transposed_vals, [0, 2, 1, 3])) - - def testDropRecordIfNegativeBucketKey(self): - - def bucket_fn(num): - # Drops record if num[0] is odd. - return tf.cond( - tf.equal(tf.math.floormod(num[0], 2), 0), lambda: 1, - lambda: -tf.cast(num[0], tf.int32)) - - input_batch = run_basic_graph(use_nested_map=False, bucket_fn=bucket_fn) - - with self.session(): - record_seen = set() - for i in range(100): - ans_input_batch = self.evaluate(input_batch) - for s in ans_input_batch.record: - record_seen.add(s) - for i in range(100): - if i % 2 == 0: - self.assertIn(('%08d' % i).encode('utf-8'), record_seen) - else: - self.assertNotIn(('%08d' % i).encode('utf-8'), record_seen) - - def testWithinBatchMixing(self): - # Generate couple files. - def generate_test_data(tag, cnt): - tmp = os.path.join(tf.test.get_temp_dir(), tag) - with tf.python_io.TFRecordWriter(tmp) as w: - for i in range(cnt): - w.write(('%s:%08d' % (tag, i)).encode('utf-8')) - return tmp - - path1 = generate_test_data('input1', 100) - path2 = generate_test_data('input2', 200) - path3 = generate_test_data('input3', 10) - - g = tf.Graph() - with g.as_default(): - # A record processor written in TF graph. - def _process(source_id, record): - return py_utils.NestedMap(source_id=source_id, record=record), 1 - - # Samples random records from the data files and processes them - # to generate batches. - input_batch, buckets = generic_input.GenericInput( - file_pattern=','.join( - ['tfrecord:' + path1, 'tfrecord:' + path2, 'tfrecord:' + path3]), - input_source_weights=[0.2, 0.3, 0.5], - file_random_seed=0, - file_buffer_size=32, - file_parallelism=4, - bucket_batch_limit=[8], - bucket_upper_bound=[1], - processor=_process) - - with self.session(graph=g): - source_id_count = collections.defaultdict(int) - tags_count = collections.defaultdict(int) - total_count = 10000 - for _ in range(total_count): - ans_input_batch, ans_buckets = self.evaluate([input_batch, buckets]) - for s in ans_input_batch.source_id: - source_id_count[s] += 1 - for s in ans_input_batch.record: - tags_count[s.split(b':')[0]] += 1 - self.assertEqual(ans_input_batch.source_id.shape, (8,)) - self.assertEqual(ans_input_batch.record.shape, (8,)) - self.assertAllEqual(ans_buckets, [1] * 8) - self.assertEqual(sum(source_id_count.values()), total_count * 8) - self.assertEqual(sum(tags_count.values()), total_count * 8) - num_records = 8. * total_count - self.assertAlmostEqual( - tags_count[b'input1'] / num_records, 0.2, delta=0.01) - self.assertAlmostEqual( - tags_count[b'input2'] / num_records, 0.3, delta=0.01) - self.assertAlmostEqual( - tags_count[b'input3'] / num_records, 0.5, delta=0.01) - self.assertAlmostEqual(source_id_count[0] / num_records, 0.2, delta=0.01) - self.assertAlmostEqual(source_id_count[1] / num_records, 0.3, delta=0.01) - self.assertAlmostEqual(source_id_count[2] / num_records, 0.5, delta=0.01) - - def testBoolDType(self): - tmp = os.path.join(tf.test.get_temp_dir(), 'bool') - with tf.python_io.TFRecordWriter(tmp) as w: - for i in range(50): - w.write(pickle.dumps(True if i % 2 == 0 else False)) - - g = tf.Graph() - with g.as_default(): - # A record processor written in TF graph. - def _process(record): - bucket_key = 1 - num, = tf.py_func(pickle.loads, [record], [tf.bool]) - return [num], bucket_key - - # Samples random records from the data files and processes them - # to generate batches. - inputs, _ = get_test_input( - tmp, bucket_upper_bound=[1], processor=_process) - - with self.session(graph=g): - for _ in range(10): - inputs_vals = self.evaluate(inputs)[0] - self.assertEqual(inputs_vals.dtype, bool) - - def testExtraArgs(self): - - def _parse_record(record): - del record - example = py_utils.NestedMap(t=tf.convert_to_tensor(0)) - bucketing_key = 1 - return example, bucketing_key - - def _parse_record_stateful(record): - del record - extra = tf.Variable(0) - example = py_utils.NestedMap(t=extra.value()) - bucketing_key = 1 - return example, bucketing_key - - generic_input.GenericInput( - _parse_record, - file_pattern='', - bucket_upper_bound=[1], - bucket_batch_limit=[1]) - - with self.assertRaisesRegex(AssertionError, 'is not pure: extra_args='): - generic_input.GenericInput( - _parse_record_stateful, - file_pattern='', - bucket_upper_bound=[1], - bucket_batch_limit=[1]) - - def testTfData(self): - """Checks that GenericInput can be invoked from a tf.data.Dataset. - - TODO(b/162015923): Fix generic_input op to work with tf.data. - """ - - def _input_batch(): - return run_basic_graph(use_nested_map=True) - - # Trick to create dataset from tensor coming from custom op. - dummy_dataset = tf.data.Dataset.from_tensors(0).repeat() - dataset = dummy_dataset.map(lambda _: _input_batch()) - - with self.session(use_gpu=False) as sess: - it = tf.compat.v1.data.make_initializable_iterator(dataset) - with self.assertRaises(tf.errors.NotFoundError): - # Gives an error that the user-provided function is undefined. - sess.run(it.initializer) - - -class GenericInputOpBenchmark(tf.test.Benchmark): - - def benchmark_basic(self): - input_batch = run_basic_graph(use_nested_map=True) - with tf.Session() as sess: - print(self.run_op_benchmark(sess, input_batch, min_iters=10)) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/gpipe.py b/lingvo/core/gpipe.py deleted file mode 100644 index 57e54996..00000000 --- a/lingvo/core/gpipe.py +++ /dev/null @@ -1,607 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""A recurrent model which enables pipelining model parallelism. - -Reference: -'GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism' -https://arxiv.org/abs/1811.06965 - -Example implementation of Transformer Language model: -tasks/lm/layers.GPipeTransformerLm - -Sample params for the one billion words task: -tasks/lm/params/one_billion_wds.OneBWdsGPipeTransformer. - -More examples in machine translation, image classifications and others -will be included. -""" - -import contextlib -import copy - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import builder_layers -from lingvo.core import py_utils -from lingvo.core import recurrent -from lingvo.core import tshape - - -_MICRO_BATCH_STATE_NAME = 'micro_batch_state' -_OVERWRITE_GLOBAL_STEP_COLLECTION = 'lingvo__OVERWRITE_GLOBAL_STEP_COLLECTION' - - -def GetOverWriteGlobalStep(graph=None): - graph = graph or tf.get_default_graph() - mb_tensors = graph.get_collection_ref(_OVERWRITE_GLOBAL_STEP_COLLECTION) - if len(mb_tensors) == 1: - mb_tensor = mb_tensors[0] - else: - mb_tensor = py_utils.GetGlobalStep() - return mb_tensor - - -def SetOverWriteGlobalStep(tensor, graph=None): - graph = graph or tf.get_default_graph() - mb_tensors = graph.get_collection_ref(_OVERWRITE_GLOBAL_STEP_COLLECTION) - if len(mb_tensors) == 1: - mb_tensors[0] = tensor - else: - graph.add_to_collection(_OVERWRITE_GLOBAL_STEP_COLLECTION, tensor) - - -def GenerateStepSeedPair(p, unused_global_step=None, op_seed=None): - """Override py_utils.GenerateStepSeedPair to use GetOverWriteGlobalStep.""" - seed_dtype = tf.int32 if py_utils.use_tpu() else tf.int64 - if p.is_inference and p.random_seed is None: - # Unlike tf.random*, stateless random ops are completely determined by the - # passed-in seeds. This means at inference time the same inputs will produce - # the same outputs, even if the model is supposed to have randomness such as - # dropout during inference. We inject additional randomness only during - # inference if the graph is exported with random_seed=None as a workaround. - return tf.random.uniform([2], maxval=seed_dtype.max, dtype=seed_dtype) - - with tf.name_scope('op_seed') as scope: - global_step = tf.cast(GetOverWriteGlobalStep(), seed_dtype) - step_seed = tf.cast(py_utils.GenerateSeedFromName(scope), seed_dtype) - seeds = tf.stack([global_step, step_seed]) - - if p.random_seed is not None: - seeds += p.random_seed - if op_seed is not None: - seeds += op_seed - return seeds - - -@contextlib.contextmanager -def CellFnFPropOpReplacementWrapper(): - """Hacks to replace certain unwanted tensorflow ops.""" - # Hack to replace GenerateStepSeedPair since global_step is not available - # in temp graph created by optional.while. - saved_get_op_seed = py_utils.GenerateStepSeedPair - py_utils.GenerateStepSeedPair = GenerateStepSeedPair - - yield - - py_utils.GenerateStepSeedPair = saved_get_op_seed - - -def _ToTuple(x): - if isinstance(x, list): - return tuple(x) - return x if isinstance(x, tuple) else (x,) - - -class FeatureExtractionLayer(base_layer.BaseLayer): - """A layer that extrac features from a sequence of layers. - - FeatureExtractionLayer is a layer which connects a few layers in a sequence. - It is also capable of fetching and forwarding activation endpoints. - # TODO(huangyp): Make it a sublayer of builder_layers.SequentialLayer - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('variable_name_prefix', '', - 'Prefix for variable names in sub layers') - p.Define('sub', [], 'A list of layers\' params.') - p.Define('num_act_inputs', 0, 'Number of activation inputs.') - p.Define('num_act_outputs', 0, 'Number of activation outputs.') - p.Define('act_fetch_layers', [], - 'Names of fetch layers that cached extra activations') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.num_act_inputs >= 0 - assert p.num_act_outputs >= 0 - p.act_fetch_layers = p.act_fetch_layers or [] - assert p.num_act_outputs == p.num_act_inputs + len(p.act_fetch_layers) - self._seq = [] - for sub in p.sub: - assert sub.name - sub.name = p.variable_name_prefix + sub.name - self.CreateChild(sub.name, sub) - self._seq.append((sub.name, self.children[sub.name])) - - def FProp(self, theta, *args): - p = self.params - assert len(args) > p.num_act_inputs - out_args = args[:-p.num_act_inputs] if p.num_act_inputs > 0 else args - extra_args = args[-p.num_act_inputs:] if p.num_act_inputs > 0 else () - for (name, ch) in self._seq: - th = theta[name] - out_args = _ToTuple(out_args) - out_args = ch.FProp(th, *out_args) - # Append fetched activations to fprop outputs. - for fetch_layer in p.act_fetch_layers: - assert fetch_layer in self.children - activation = self.children[fetch_layer].activation - if isinstance(activation, (tuple, list)): - activation = activation[0] - extra_args += (activation,) - if extra_args: - out_args = _ToTuple(out_args) + extra_args - return out_args - - @classmethod - def FPropMeta(cls, p, *args): - assert len(args) > p.num_act_inputs - seq_args = args[:-p.num_act_inputs] if p.num_act_inputs > 0 else args - extra_args = args[-p.num_act_inputs:] if p.num_act_inputs > 0 else () - total = 0 - act_fetch_metas = {} - for sub in p.sub: - meta = sub.cls.FPropMeta(sub, *seq_args) - if sub.name in p.act_fetch_layers: - act_fetch_metas[sub.name] = meta.out_shapes[0] - total += meta.flops - seq_args = meta.out_shapes - for fetch_layer in p.act_fetch_layers: - extra_args += (act_fetch_metas[fetch_layer],) - return py_utils.NestedMap(flops=total, out_shapes=seq_args + extra_args) - - -def PartitionSequentialLayers(params, num_partitions, *shapes): - r"""Partition a layer composed of sequential layers. - - This routine strives to partition layers so that each partition costs roughly - the same flops given the input shapes. - - Args: - params: A layer param or a list of layer param. - num_partitions: The desired number of partitions. - *shapes: A tuple of tshape.Shape representing input tensors to the first - layer. - - Returns: - A list of FeatureExtractionLayer params. - """ - - # Recursively concatenate SequentialLayer into a list. - def FlattenSeq(p): - if isinstance(p, list): - return p - if p.cls not in [builder_layers.SequentialLayer, FeatureExtractionLayer]: - return [p.Copy()] - subs = [] - for _ in range(p.repeat): - for s in p.sub: - subs += FlattenSeq(s) - return subs - - subs = FlattenSeq(params) - - assert len(shapes) == 1 - tf.logging.info('num_partitions: {} input_shape: {}'.format( - num_partitions, shapes[0])) - - # Computes the estimate cost for each sub layer. - total, histo, output_shapes = 0, [], [] - for i, s in enumerate(subs): - s.name = 'cell_%03d' % i - meta = s.cls.FPropMeta(s, *shapes) - total += meta.flops - histo.append(total) - output_shapes.append(meta.out_shapes) - shapes = meta.out_shapes - tf.logging.vlog(1, 'len %d histogram = %s', len(subs), histo) - - # Computes the normalized cumulative histogram of the layer's cost. - histo_pct = [float(x / total) for x in histo] - tf.logging.vlog(1, 'cost pct = %s', histo_pct) - - # i-th sub layer is put into partition j, where j is roughly i-th cumulative - # histogram times num_partitions. - parts = [[] for _ in range(num_partitions)] - parts_cost = [0] * num_partitions - pre_hist_cost = 0 - for i, s in enumerate(subs): - j = min(int(histo_pct[i] * num_partitions), num_partitions - 1) - # The boundary at parts[j] where j > 0 - if j > 0 and not parts[j]: - parts_cost[j - 1] = histo_pct[i - 1] - pre_hist_cost - pre_hist_cost = histo_pct[i - 1] - parts[j].append(s) - - parts_cost[num_partitions - 1] = 1.0 - pre_hist_cost - seqs = [] - for i, pa in enumerate(parts): - tf.logging.info('Partition %d #subs %d #cost %.3f', i, len(pa), - parts_cost[i]) - - seqs.append(FeatureExtractionLayer.Params().Set(name='d%d' % i, sub=pa)) - return seqs - - -class SeqLayer(base_layer.BaseLayer): - """Round-robin every children cells in cell_tpl among worker devices.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('before_tpl', [], - 'Config for the CNN layers that runs before pipelining.') - p.Define('cell_tpl', [], 'A list of FeatureExtractionLayer layers.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self._before_layers = [] - self._cells = [] - for l in p.before_tpl: - self.CreateChild(l.name, l) - self._before_layers.append((l.name, self.children[l.name])) - for l in p.cell_tpl: - self.CreateChild(l.name, l) - self._cells.append((l.name, self.children[l.name])) - - def _CreateChildrenVariables(self): - p = self.params - - num_cells = len(p.cell_tpl) - before_tpl_device = '' - cell_devices = [''] * num_cells - if py_utils.use_tpu(): - cluster = self.cluster - before_tpl_device = cluster.WorkerDeviceInModelSplit(0) - cell_devices = [ - cluster.WorkerDeviceInModelSplit(i) for i in range(num_cells) - ] - - for unused_name, l in self._before_layers: - with tf.device(before_tpl_device): - l.InstantiateVariables() - - for i, (unused_name, l) in enumerate(self._cells): - with tf.device(cell_devices[i]): - l.InstantiateVariables() - - super()._CreateChildrenVariables() - - def FProp(self, theta, *args): - """Round-robin every children cells in cell_tpl among worker devices. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - *args: Input args - - Returns: - A list contains one tensor of [batch_size, feature_height, feature_width, - channel]. - """ - num_layers = len(self.params.cell_tpl) - cluster = self.cluster - - for (name, l) in self._before_layers: - l_theta = theta[name] - args = _ToTuple(args) - args = l.FProp(l_theta, *args) - for i in range(num_layers): - with tf.device(cluster.WorkerDeviceInModelSplit(i)): - cell_name, cell = self._cells[i] - args = _ToTuple(args) - args = cell.FProp(theta[cell_name], *args) - - return args - - -class PipeliningLayer(SeqLayer): - """Pipelining a sequence of layers on multiple devices.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_micro_batches', 1, 'Number of micro batches.') - p.Define('micro_batch_size', None, 'Size of a micro batch.') - p.Define('batch_dim', 0, 'The batch dimension.') - p.Define('state_dtype', None, 'Externally specify dtype for states.') - p.Define( - 'nested_map_fprop', False, 'Whether arguments and returns of ' - 'cell fprop functions are nested maps') - return p - - def _CalculateOutputShapes(self, input_shapes): - """Calcuate the output shape of intermediate layers. - - Given the FPropMeta function in each FeatureExtractionLayer, calcuates - the shapes of outputs of that layer. This is used to recover the shape - information in StackedRecurrent. - - Args: - input_shapes: NestedMap or tuple of input TensorShapes. - - Returns: - Return a list of K + 1 NestedMaps or lists of tShape where K is - the number of partitions. - """ - p = self.params - shapes = [] - - # Converts TensorShape to tshape.Shape. - def _ToTShape(x): - if x is None: - return None - return tshape.Shape(x.as_list()) - - shapes = py_utils.Transform(_ToTShape, input_shapes) - shapes = _ToTuple(shapes) - - state_shapes = [] - for (_, cell) in self._before_layers: - shapes = cell.FPropMeta(cell.params, *shapes).out_shapes - - state_shapes.append(shapes[0] if p.nested_map_fprop else shapes) - - for (_, cell) in self._cells: - shapes = cell.FPropMeta(cell.params, *shapes).out_shapes - state_shapes.append(shapes[0] if p.nested_map_fprop else shapes) - - return state_shapes - - def _get_state_dtype(self, *args): - if self.params.state_dtype: - return self.params.state_dtype - if self.params.nested_map_fprop: - inputs = args[0].Filter(lambda x: x is not None) - return py_utils.Flatten(inputs)[0].dtype - return args[0].dtype - - def _get_input_shapes(self, *args): - p = self.params - if p.nested_map_fprop: - assert len(args) == 1 - assert isinstance(args[0], py_utils.NestedMap) - input_tensors = py_utils.Flatten(args[0]) - else: - input_tensors = _ToTuple(args) - # Get batch size from the first tensor which is not None. - mini_batch_size = None - for input_tensor in input_tensors: - if input_tensor is not None: - mini_batch_size = input_tensor.get_shape().as_list()[p.batch_dim] - assert mini_batch_size is not None - micro_batch_size = p.micro_batch_size - if not micro_batch_size: - if p.num_micro_batches > mini_batch_size: - p.num_micro_batches = mini_batch_size - micro_batch_size = mini_batch_size // p.num_micro_batches - if mini_batch_size is not None: - if micro_batch_size * p.num_micro_batches != mini_batch_size: - raise ValueError('micro_batch_size * num_micro_batches != batch_size.') - - input_shapes = () - for input_tensor in input_tensors: - if input_tensor is not None: - input_shape = input_tensor.get_shape().as_list() - input_shape[p.batch_dim] = micro_batch_size - input_shapes += (tf.TensorShape(input_shape),) - else: - input_shapes += (None,) - - if p.nested_map_fprop: - input_shapes = py_utils.Pack(args[0], input_shapes) - return input_shapes - - def FProp(self, theta, *args): - """Run multiple cells in different devices in a pipelining manner. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - *args: Non-keyworded variable length argument list of input tensors. - - Returns: - A list of output tensors - """ - # TODO(huangyp): handle optional None inputs. - p = self.params - if self.do_eval and self.cluster.num_devices_per_split == 1: - outputs = copy.copy(args) - for (name, l) in self._before_layers + self._cells: - outputs = _ToTuple(outputs) - outputs = l.FProp(theta[name], *outputs) - return outputs - - num_cells = len(p.cell_tpl) - cluster = self.cluster - - # Compute shapes of input and output tensors. - input_shapes = self._get_input_shapes(*args) - state_dtype = self._get_state_dtype(*args) - state_shapes = self._CalculateOutputShapes(input_shapes) - tf.logging.info('state_shapes={}'.format(state_shapes)) - - def GetCellFn(i): - """Get the ith feature extraction layer.""" - - def CellFn(theta, state0, inputs): - """A cell fn is exectued inside of StackedRecurrent.""" - del state0 - - def _FPropInputSetShape(name, t_shape): - if t_shape is None: - return None - inputs[name].set_shape(t_shape.ToTensorShape().as_list()) - return inputs[name] - - if p.nested_map_fprop: - # pylint: disable=protected-access - fprop_inputs = state_shapes[i]._RecursiveMap(_FPropInputSetShape) - # pylint: enable=protected-access - else: - fprop_inputs = [] - for input_idx, input_shape in enumerate(state_shapes[i]): - name = 's{}'.format(input_idx) - fprop_inputs.append(_FPropInputSetShape(name, input_shape)) - - with py_utils.RemoveAssertContext(remove=True): - with CellFnFPropOpReplacementWrapper(): - tf.logging.info('cell {} input {}'.format(i, fprop_inputs)) - mb_tensor = inputs[_MICRO_BATCH_STATE_NAME] - SetOverWriteGlobalStep(mb_tensor) - _, cell = self._cells[i] - fprop_inputs = _ToTuple(fprop_inputs) - outputs = cell.FProp(theta, *fprop_inputs) - - if p.nested_map_fprop: - assert py_utils.IsCompatible(outputs, state_shapes[i + 1]) - state1 = outputs.Filter(lambda x: x is not None) - else: - state1 = py_utils.NestedMap() - outputs = _ToTuple(outputs) - assert len(outputs) == len(state_shapes[i + 1]) - for output_idx in range(len(outputs)): - if outputs[output_idx] is not None: - name = 's{}'.format(output_idx) - state1[name] = outputs[output_idx] - state1[_MICRO_BATCH_STATE_NAME] = mb_tensor - return state1, py_utils.NestedMap() - - return CellFn - - cell_fns = [] - accumulator_layers = [] - thetas = [] - init_states = [] - devices = [] - for cell_idx in range(num_cells): - cell_name, cell = self._cells[cell_idx] - accumulator_layers.append(cell) - cell_fns.append(GetCellFn(cell_idx)) - thetas.append(theta[cell_name]) - - def _TfZeros(t_shape): - if t_shape is None: - return None - return tf.zeros(t_shape.ToTensorShape().as_list(), dtype=state_dtype) - - if p.nested_map_fprop: - init_state = py_utils.Transform(_TfZeros, state_shapes[cell_idx + 1]) - init_state = init_state.Filter(lambda x: x is not None) - else: - init_state = py_utils.NestedMap() - for output_idx, state in enumerate(state_shapes[cell_idx + 1]): - state = _TfZeros(state) - if state is not None: - name = 's{}'.format(output_idx) - init_state[name] = state - init_state[_MICRO_BATCH_STATE_NAME] = tf.cast(0, dtype=state_dtype) - init_states.append(init_state) - - devices.append(cluster.WorkerDeviceInModelSplit(cell_idx)) - - cell_grads = [None] * num_cells - cell_outs = [lambda x: x] * num_cells - cell_out_grads = [lambda x: x] * num_cells - - with tf.device(devices[0]): - previous = _ToTuple(args) - for (name, l) in self._before_layers: - previous = l.FProp(theta[name], *previous) - previous = _ToTuple(previous) - - def _StackAndSplit(x): - # Split tensors into microbatches. - if x is None: - return None - return tf.stack(tf.split(x, p.num_micro_batches, axis=p.batch_dim)) - - if p.nested_map_fprop: - inputs = py_utils.Transform(_StackAndSplit, previous[0]) - inputs = inputs.Filter(lambda x: x is not None) - else: - inputs = py_utils.NestedMap() - for output_idx, output_tensor in enumerate(previous): - output_tensor = _StackAndSplit(output_tensor) - if output_tensor is not None: - name = 's{}'.format(output_idx) - inputs[name] = output_tensor - gs_tensor = py_utils.GetGlobalStep() - inputs[_MICRO_BATCH_STATE_NAME] = tf.stack([ - tf.cast(gs_tensor * p.num_micro_batches + t, dtype=state_dtype) - for t in range(p.num_micro_batches) - ]) - tf.logging.info('pipeline input = {}'.format(inputs)) - output_state, _ = recurrent.StackedRecurrent( - devices=devices, - cell_fns=cell_fns, - cell_grads=cell_grads, - cell_outs=cell_outs, - cell_out_grads=cell_out_grads, - thetas=thetas, - init_states=init_states, - inputs=inputs, - accumulator_layers=accumulator_layers, - unused_acc_state=True) - - with tf.device(devices[-1]): - - def _ReshapeRetVal(name, t_shape): - """Restore shape for tensors in microbatches.""" - if t_shape is None: - return None - output_tensor = output_state[name] - if p.batch_dim != 0: - perm = list(range(1, p.batch_dim + 1)) + [0] - perm += list(range(p.batch_dim + 1, t_shape.rank + 1)) - output_tensor = tf.transpose(output_tensor, perm=perm) - output_shape = t_shape.ToTensorShape().as_list() - output_shape[p.batch_dim] *= p.num_micro_batches - output_tensor = tf.reshape(output_tensor, output_shape) - return output_tensor - - # Construct the final return values from output_state. - if p.nested_map_fprop: - # pylint: disable=protected-access - output_tensors = state_shapes[-1]._RecursiveMap(_ReshapeRetVal) - # pylint: enable=protected-access - else: - output_tensors = [] - for output_idx, state_shape in enumerate(state_shapes[-1]): - output_name = 's{}'.format(output_idx) - output_tensor = _ReshapeRetVal(output_name, state_shape) - output_tensors.append(output_tensor) - if len(output_tensors) == 1: - output_tensors = output_tensors[0] - else: - output_tensors = tuple(output_tensors) - tf.logging.info('pipeline output = {}'.format(output_tensors)) - return output_tensors diff --git a/lingvo/core/gpipe_test.py b/lingvo/core/gpipe_test.py deleted file mode 100644 index 4fd82416..00000000 --- a/lingvo/core/gpipe_test.py +++ /dev/null @@ -1,241 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lingvo gpipe.""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.core import tshape -from lingvo.core.gpipe import FeatureExtractionLayer -from lingvo.core.gpipe import PartitionSequentialLayers -from lingvo.core.gpipe import PipeliningLayer -from lingvo.core.layers import Conv2DLayerNoPadding -from lingvo.core.layers import FetchLayer - - -class _TimestepAccumulator(base_layer.Accumulator): - """Simple accumulator for counting timesteps in the pipeline.""" - - def DefaultValue(self): - return tf.convert_to_tensor(0.0) - - def Increment(self): - self.SetValue(self.GetValue() + 1.0) - - -class _SimpyLayer(base_layer.BaseLayer): - """Simpy Layer with accumulator that counts time step.""" - - def __init__(self, params): - super().__init__(params) - conv = Conv2DLayerNoPadding.Params().Set( - name='conv', - filter_shape=(3, 3, 1, 1), - filter_stride=(1, 1), - params_init=py_utils.WeightInit.Constant(0.1)) - self.CreateChild('conv', conv) - self.RegisterAccumulator('ts_count', _TimestepAccumulator()) - - def FProp(self, theta, inputs): - p = self.params - with tf.name_scope(p.name): - inputs = self.conv.FProp(theta.conv, inputs) - self.accumulators.ts_count.Increment() - return tf.nn.relu(inputs) - - @classmethod - def FPropMeta(cls, p, inputs): - py_utils.CheckShapes((inputs,)) - return py_utils.NestedMap(flops=1, out_shapes=(inputs,)) - - -class _SimpyLayerWithNestedMapInput(_SimpyLayer): - - def FProp(self, theta, inputs): - p = self.params - with tf.name_scope(p.name): - vec = self.conv.FProp(theta.conv, inputs.vec) - return py_utils.NestedMap(vec=vec, paddings=inputs.paddings) - - @classmethod - def FPropMeta(cls, p, inputs): - py_utils.CheckShapes( - tuple(inputs.Filter(lambda x: x is not None).Flatten())) - return py_utils.NestedMap(flops=1, out_shapes=(inputs,)) - - -def _Partition(params, num_splits, *shapes): - seqs = PartitionSequentialLayers(params, num_splits, *shapes) - return [ - FeatureExtractionLayer.Params().Set(name='d%d' % i, sub=seqs[i].sub) - for i in range(len(seqs)) - ] - - -def _BuildDummyPipelineCnn(num_splits=4, - num_micro_batches=8, - micro_batch_size=None): - """Construct a dummy layer that consist of 16 3x3 conv layers. - - In addition, each conv layer increments a count every time step. - - Args: - num_splits: number of cells for pipeline cnn - num_micro_batches: number of time steps. - micro_batch_size: Size of a micro batch. - - Returns: - A PipeliningLayer layer. - """ - assert num_splits in [1, 2, 4, 8, 16] - num_layers = 16 - layers = [] - for i in range(num_layers): - layers.append(_SimpyLayer.Params().Set(name='layer_{}'.format(i))) - - if num_splits == 1: - p = FeatureExtractionLayer.Params().Set(name='seq', sub=layers) - else: - cell_tpl = [] - layers_per_split = num_layers // num_splits - num_act_outputs = 0 - num_act_inputs = 0 - act_fetch_layers = None - for split in range(num_splits): - sub = layers[split * layers_per_split:(split + 1) * layers_per_split] - if split == 0: - sub.append(FetchLayer.Params().Set(name='fetch')) - num_act_outputs = 1 - act_fetch_layers = ['fetch'] - else: - num_act_inputs = 1 - act_fetch_layers = [] - split_layer = FeatureExtractionLayer.Params().Set( - name='split_{}'.format(split), - sub=sub, - act_fetch_layers=act_fetch_layers, - num_act_inputs=num_act_inputs, - num_act_outputs=num_act_outputs) - cell_tpl.append(split_layer) - p = PipeliningLayer.Params().Set( - name='pipeline', - num_micro_batches=num_micro_batches, - micro_batch_size=micro_batch_size, - cell_tpl=cell_tpl, - before_tpl=[]) - layer = p.Instantiate() - return layer - - -class DummyPipelineCnnTest(test_utils.TestCase): - - def _verify_timestep_counts(self, - num_splits, - auto_partition=False, - micro_batch_size=None): - num_micro_batches = 8 - batch_size = 16 - with self.session(graph=tf.Graph()): - tf.random.set_seed(1245) - inputs = tf.random.uniform([batch_size, 8, 8, 1], seed=12345) - if auto_partition: - layers = [ - _SimpyLayer.Params().Set(name='layer_{}'.format(i)) - for i in range(16) - ] - net = PipeliningLayer.Params().Set( - name='pipeline', - num_micro_batches=num_micro_batches, - cell_tpl=_Partition(layers, num_splits, - tshape.Shape([batch_size, 8, 8, - 1]))).Instantiate() - else: - net = _BuildDummyPipelineCnn( - num_splits=num_splits, - micro_batch_size=micro_batch_size, - num_micro_batches=num_micro_batches) - endpoints = net.FPropDefaultTheta(inputs) - if isinstance(endpoints, (list, tuple)): - logits, aux_logits = endpoints - else: - logits = endpoints - aux_logits = None - loss = tf.reduce_mean(logits) - grads = tf.gradients(loss, tf.trainable_variables()) - grad_norm = tf.sqrt(py_utils.SumSquared(grads)) - ts = net.GetAccumulatorValues().Flatten() - - self.evaluate(tf.global_variables_initializer()) - grad_norm_val, ts_vals = self.evaluate([grad_norm, ts]) - test_utils.CompareToGoldenSingleFloat(self, 0.268087, grad_norm_val) - # Accumulator values should be equal to number of time steps in pipeline. - for ts_val in list(ts_vals): - expected_ts = num_micro_batches if num_splits > 1 else 1 - self.assertEqual(ts_val, expected_ts) - if aux_logits is not None: - aux_logit_tensor = self.evaluate(aux_logits) - self.assertEqual(aux_logit_tensor.shape, (batch_size, 8, 8, 1)) - - def testDummyPipelineCnnOneSplit(self): - self._verify_timestep_counts(num_splits=1) - - def testDummyPipelineCnnTwoSplits(self): - self._verify_timestep_counts(num_splits=2) - - def testDummyPipelineCnnTwoSplitsMicroBatchSize1(self): - with self.assertRaises(ValueError): - self._verify_timestep_counts(num_splits=2, micro_batch_size=1) - - def testDummyPipelineCnnTwoSplitsMicroBatchSize2(self): - self._verify_timestep_counts(num_splits=2, micro_batch_size=2) - - def testDummyPipelineCnnFourSplits(self): - self._verify_timestep_counts(num_splits=4) - - def testDummyPipelineCnnAutoPartitionTwoSplits(self): - self._verify_timestep_counts(num_splits=2, auto_partition=True) - - def testDummyPipelineCnnAutoPartitionFourSplits(self): - self._verify_timestep_counts(num_splits=4, auto_partition=True) - - def testDummyPipelineCnnNestedMapInput(self): - batch_size = 16 - num_layers = 4 - cells = [] - with self.session(graph=tf.Graph()): - for i in range(num_layers): - cells.append(_SimpyLayerWithNestedMapInput.Params().Set( - name='layer_{}'.format(i))) - p = PipeliningLayer.Params().Set( - name='pipeline', - num_micro_batches=8, - micro_batch_size=2, - nested_map_fprop=True, - cell_tpl=cells, - before_tpl=[]) - layer = p.Instantiate() - tf.random.set_seed(1245) - inputs = tf.random.uniform([batch_size, 8, 8, 1], seed=12345) - outputs = layer.FPropDefaultTheta( - py_utils.NestedMap(vec=inputs, paddings=None)) - self.evaluate(tf.global_variables_initializer()) - self.evaluate(outputs.vec) - self.assertEqual(outputs.vec.shape, (batch_size, 8, 8, 1)) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/gradient_combiner.py b/lingvo/core/gradient_combiner.py deleted file mode 100644 index ff01d2cf..00000000 --- a/lingvo/core/gradient_combiner.py +++ /dev/null @@ -1,45 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Layers to combine gradients computed from multiple losses. - -Multi-task learning can sometimes benefit from more sophisticated gradient -combination algorithms than simple linear aggregation, for example - -* Gradient surgery: https://arxiv.org/abs/2001.06782 -* GradNorm: https://arxiv.org/abs/1711.02257 -""" - -from lingvo.core import base_layer - - -class GradientCombiner(base_layer.BaseLayer): - """An abstract class to combine gradients (from multiple loss functions).""" - - def Combine(self, vmap, losses_and_gradients): - """Combines gradients on the variables. - - Args: - vmap: a NestedMap containing the variables. - losses_and_gradients: a Dict[str, loss_and_grads], where each key - represents the loss name used to compute the gradients and each value is - a NestedMap with the following entries, 'loss_metric', a (loss, weight) - pair representing the loss; 'grads', a NestedMap containing the gradient - tensors for variables, with the identical structure as 'vmap'. - - Returns: - A NestedMap of combined gradients, with an identical structure as 'vmap'. - """ - raise NotImplementedError(type(self)) diff --git a/lingvo/core/hyperparams.proto b/lingvo/core/hyperparams.proto deleted file mode 100644 index ca86019d..00000000 --- a/lingvo/core/hyperparams.proto +++ /dev/null @@ -1,58 +0,0 @@ -syntax = "proto2"; - -package tensorflow.lingvo; - -// This is a top-level entry, typically each entry corresponds to a p.Define() -// statement in the Python code. The key is the parameter name. It is also used -// for when the parameter itself is a dictionary. -message Hyperparam { - map items = 1; -} - -// This is to serialize a list or tuple of parameters. For example, some -// parameters are a list of integers. The distinction between list and tuple is -// by the field name in the HyperparamValue below. -message HyperparamRepeated { - repeated HyperparamValue items = 1; -} - -// Once the repetition is handled (with HyperparamList et al. above), the -// parameter can either be a recursively set to another set of params via -// p.Define, or a raw type (such as a string, float, ...). -message HyperparamValue { - oneof kind { - // The NoneType is represented by not setting any of the values below. - Hyperparam param_val = 1; - HyperparamRepeated list_val = 2; - HyperparamRepeated tuple_val = 3; - Hyperparam dict_val = 4; - string type_val = 5; - string dtype_val = 6; - string string_val = 7; - bool bool_val = 8; - int64 int_val = 9; - float float_val = 10; - ProtoVal proto_val = 11; - EnumVal enum_val = 12; - NamedTupleVal named_tuple_val = 13; - } -} - -// A serialized proto. The message includes both the proto message name and the -// proto serialized value. -message ProtoVal { - optional string type = 1; - optional bytes val = 2; -} - -// An enum. -message EnumVal { - optional string type = 1; - optional string name = 2; -} - -// A namedtuple. -message NamedTupleVal { - optional string type = 1; - repeated HyperparamValue items = 2; -} diff --git a/lingvo/core/hyperparams.py b/lingvo/core/hyperparams.py deleted file mode 100644 index fc41b43d..00000000 --- a/lingvo/core/hyperparams.py +++ /dev/null @@ -1,891 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Defines Params base class, used for defining class/function parameters.""" - -import ast -import copy -import enum -import importlib -import inspect -import re -import sys - -import dataclasses -import lingvo.compat as tf -from lingvo.core import hyperparams_pb2 -from lingvo.core import symbolic - -from google.protobuf import message -from google.protobuf import text_format - - -def _QuoteString(s): - """Quotes a string with appropriate quotes and escaping. - - This performs lite escaping by choosing enclosing quotation marks that would - escape the least (either single or double quotes) and escaping those quotes - and the backslash. Note that this does not escape newlines. If the string - contains embedded newlines, they will be output verbatim. - - Args: - s: String to quote. - Returns: - Quotes string (possibly multiline). - """ - single_quote_count = s.count('\'') - double_quote_count = s.count('"') - quote_delim = '\'' if single_quote_count <= double_quote_count else '"' - # Apply escaping to the chosen quote character and the backslash. - encoded = re.sub(r'([%s\\])' % quote_delim, r'\\\1', s) - return quote_delim + encoded + quote_delim - - -def _UnquoteString(quoted): - if quoted and quoted[0] in ['"', '\'']: - # Note that only the limited set of escaping produced by _QuoteString is - # supported. - contents = quoted.strip(quoted[0]) - return re.sub(r"""\\([\\'"])""", r'\1', contents) - else: - # Just return literal text. - return quoted - - -def _EndsWithTerminalQuote(s, quote_char): - """Returns whether a string ends with a valid terminal quote.""" - endm = re.search(r'(\\*)%s$' % quote_char, s) - if not endm: - return False - backslashes = endm.group(1) - if len(backslashes) % 2 == 0: - # Even number of backslashes preceding the quote means the quote is - # not escaped. - return True - else: - # Terminal quote is escaped. - return False - - -def _IsNamedTuple(x): - """Returns whether an object is an instance of a collections.namedtuple. - - Examples:: - - _IsNamedTuple((42, 'hi')) ==> False - Foo = collections.namedtuple('Foo', ['a', 'b']) - _IsNamedTuple(Foo(a=42, b='hi')) ==> True - - Args: - x: The object to check. - """ - return isinstance(x, tuple) and hasattr(x, '_fields') - - -class _SortedDict(dict): - """A dict with a __repr__ that is always sorted by key.""" - - def __repr__(self): - return '{' + ', '.join( - '%r: %r' % item for item in sorted(self.items())) + '}' - - -class _Param: - """Stores data for a single parameter.""" - - def __init__(self, name, default_value, description): - self._name = name - self._value = default_value - self._description = description - - def __eq__(self, other): - # pylint: disable=protected-access - return self._name == other._name and self._value == other._value - - # Deep copy the value only if it is supported. - def __deepcopy__(self, memo): - if isinstance(self._value, (tf.Tensor, symbolic.Symbol)): - # In case self._value is a tensor/symbol, let's just make a reference. - value = self._value - else: - value = copy.deepcopy(self._value, memo) - p = _Param(self._name, value, self._description) - # Q(yonghui): Is this the right use of memo. - memo[id(self)] = p - return p - - def ToString(self, nested_depth): - """Prints the parameter as a string.""" - - def GetRepr(val): - """Get the representation of `val`.""" - if isinstance(val, Params): - return _SortedDict({k: GetRepr(v) for k, v in val.IterParams()}) - if isinstance(val, dict): - return _SortedDict({k: GetRepr(v) for k, v in val.items()}) - if isinstance(val, (list, tuple)) and not _IsNamedTuple(val): - # NB: this constructor signature works for tuples, but not namedtuples. - return type(val)([GetRepr(v) for v in val]) - # NOTE(markmurphy): I introduced Repr() because it's impossible (afaik) to - # overwrite the __str__ or __repr__ method of a types.FunctionType object. - if hasattr(val, 'Repr'): - return val.Repr() - return val - - nested_indent = ' ' * nested_depth - if isinstance(self._value, Params): - # pylint: disable=protected-access - value_str = self._value._ToString(nested_depth) - elif isinstance(self._value, str): - return '%s%s: "%s"' % (nested_indent, self._name, self._value) - else: - value_str = str(GetRepr(self._value)) - return '%s%s: %s' % (nested_indent, self._name, value_str) - - def Set(self, value): - # Note that we don't make a copy of Params objects. - # TODO(sadovsky): Maybe add safeguard to ensure that Params object is not - # owned by other Params objects. - self._value = value - - def Get(self): - return self._value - - -def CopyFieldsTo(from_p, to_p, skip=None): - """Copy fields from one Params to another, with optional skipped params. - - Preserves `type(to_p.Instantiate())`. Use `from_p.Copy()` instead if requiring - a deep copy of `from_p`, without updating `to_p`. - - Args: - from_p: Source params to copy from. - to_p: Destination params to copy to. - skip: If not None, a list of strings of param names to skip. Automatically - skips InstantiableParams' 'cls' parameter. - - Returns: - The updated to_p. - """ - skip = skip or [] - skip.append('cls') - for n, p in from_p.IterParams(): - if n in skip: - continue - if isinstance(p, Params): - to_p.Set(**{n: p.Copy()}) - else: - to_p.Set(**{n: p}) - return to_p - - -class Params: - """Stores data for a set of parameters. - - Provides attribute-based API, e.g. "params.foo = 5". - Uses internal {'name': _Param} dict for storing parameter data. - """ - - def __init__(self): - self.__dict__['_immutable'] = False - self._params = {} # name => _Param - - def __setattr__(self, name, value): - if self._immutable: - raise TypeError('This Params instance is immutable.') - if name == '_params' or name == '_immutable': - self.__dict__[name] = value - else: - try: - self._params[name].Set(value) - except KeyError: - raise AttributeError(self._KeyErrorString(name)) - - def __getattr__(self, name): - if name == '_params' or name == '_immutable': - return self.__dict__[name] - try: - return self._params[name].Get() - except KeyError: - # cPickle expects __getattr__ to raise AttributeError, not KeyError. - raise AttributeError(self._KeyErrorString(name)) - - def __dir__(self): - return sorted(self._params.keys()) - - def __contains__(self, name): - return name in self._params - - def __len__(self): - return len(self._params) - - # Note: This gets called by _Param.__eq__() on nested Params objects. - def __eq__(self, other): - return isinstance(other, Params) and self._params == other._params # pylint: disable=protected-access - - def __ne__(self, other): - return not self == other - - def __str__(self): - return self._ToString(0) - - def _ToString(self, nested_depth): - # Note: We use iteritems() below so as to sort by name. - sorted_param_strs = [ - v.ToString(nested_depth + 1) for (_, v) in sorted(self._params.items()) - ] - nested_indent = ' ' * nested_depth - return '{\n%s\n%s}' % ('\n'.join(sorted_param_strs), nested_indent) - - # Override __deepcopy__ so that copy.deepcopy(self._params) properly - # deep-copies nested Params objects. - # TODO(sadovsky): Is it okay not to touch memo? - def __deepcopy__(self, unused_memo): - return self.Copy() - - def _SimilarKeys(self, name): - """Return a list of params keys that are similar to name.""" - - def _Overlaps(name, key): - """The fraction of 3-char substrings in that appear in key.""" - matches = 0 - trials = 0 - for i in range(len(name) - 3): - trials += 1 - if name[i:i + 3] in key: - matches += 1 - if trials: - return float(matches) / trials - return 0 - - if '_params' in self.__dict__: - return [key for key in self._params if _Overlaps(name, key) > 0.5] - return [] - - def _KeyErrorString(self, name): - similar = self._SimilarKeys(name) - if similar: - return name + ' (did you mean: [%s])' % (','.join(sorted(similar))) - return name - - def Copy(self): - """Creates a deep copy of self.""" - return self._CopyTo(type(self)()) - - def _CopyTo(self, res): - # pylint: disable=protected-access - res._params = copy.deepcopy(self._params) - res._immutable = self._immutable - # pylint: enable=protected-access - return res - - # TODO(sadovsky): - # - Maybe let users specify whether this parameter is allowed to have - # value=None, and if not, assert on Get(), like required proto field. - # - Maybe enforce that value is one of - # {number, string, bool, list, dict, Params}. - def Define(self, name, default_value, description): - """Defines a parameter. - - Args: - name: The parameter name. Must only contain lowercase letters, numbers, - and underscores. Must start with lowercase letter. - default_value: Default value for this parameter. May be None. - description: String description of this parameter. - - Raises: - AttributeError: If parameter 'name' is already defined. - """ - if self._immutable: - raise TypeError('This Params instance is immutable.') - assert name is not None and isinstance( - name, str) and (re.match('^[a-z][a-z0-9_]*$', name) is not None) - if name in self._params: - raise AttributeError('Parameter %s is already defined' % name) - self._params[name] = _Param(name, default_value, description) - - def Freeze(self): - """Marks this Params as immutable.""" - self._immutable = True - - def IsImmutable(self): - """Return whether this Params is immutable.""" - return self._immutable - - def _GetNested(self, name): - """Returns nested param by its name.""" - parts = name.split('.') - curr = self - for i, part in enumerate(parts[:-1]): - # Get the value (nested Params object) associated with name 'part'. - try: - is_list = re.match(r'^(.+)\[(.+)\]$', part) - if is_list: - part = is_list.group(1) - list_index = int(is_list.group(2)) - # pylint: disable=protected-access - curr = curr._params[part].Get() - if is_list: - curr = curr[list_index] - except KeyError: - raise AttributeError('.'.join(parts[:i + 1])) - assert isinstance(curr, Params), ( - 'Cannot introspect %s for %s' % (type(curr), '.'.join(parts[:i + 1]))) - return curr, parts[-1] - - def Set(self, **kwargs): - """Sets multiple parameters. - - Dots in names indicate navigation into nested Params objects. We do not - allow navigation into lists or dicts, and may ban these types altogether in - favor of string representations. - - Args: - **kwargs: Name-value pairs to set. - - Returns: - self - """ - if self._immutable: - raise TypeError('This Params instance is immutable: %s' % self) - for name, value in kwargs.items(): - # Get nested param. - param, key = self._GetNested(name) - # Update the value associated with key. - try: - # pylint: disable=protected-access - param._params[key].Set(value) - except KeyError: - raise AttributeError(self._KeyErrorString(name)) - return self - - def Get(self, name): - """Get parameter. - - Dots in names indicate navigation into nested Params objects. We do not - allow navigation into lists or dicts, and may ban these types altogether in - favor of string representations. - - Args: - name: (str) Name. - - Returns: - value. - - Raises: - AttributeError: if parameter is not found - """ - param, key = self._GetNested(name) - # Get the value associated with key. - try: - # pylint: disable=protected-access - return param._params[key].Get() - except KeyError: - raise AttributeError(self._KeyErrorString(name)) - - def Delete(self, *args): - """Deletes multiple parameters. - - Dots in names indicate navigation into nested Params objects. We do not - allow navigation into lists or dicts, and may ban these types altogether in - favor of string representations. - - Args: - *args: List of names. - - Returns: - self - """ - if self._immutable: - raise TypeError('This Params instance is immutable.') - for name in args: - # Get nested param. - param, key = self._GetNested(name) - # Delete the key. - try: - # pylint: disable=protected-access - del param._params[key] - except KeyError: - raise AttributeError(self._KeyErrorString(name)) - return self - - def IterParams(self): - """Pythonic dict-like iteration.""" - for name, param in self._params.items(): - yield (name, param.Get()) - - def ToProto(self): - """Writes to a Hyperparams proto. - - Serializes the Hyperparams into a proto that can be then written to disk or - sent over the network. Note that serialization is not guaranteed to be - unique or stable (this is a feature of protos themselves, not this code), so - using it for fingerprinting for example may not be appropriate. Refer to the - ToText() method for a serialization approach that Lingvo controls. - - Returns: - The serialized params as a Hyperparams proto. - """ - - def _ToParamValue(val): - """Serializes to HyperparamValue proto.""" - param_pb = hyperparams_pb2.HyperparamValue() - if isinstance(val, Params): - param_pb.param_val.CopyFrom(_ToParam(val)) - elif isinstance(val, list) or isinstance(val, range): - # The range function is serialized by explicitely calling it. - param_pb.list_val.items.extend([_ToParamValue(v) for v in val]) - elif dataclasses.is_dataclass(val) or _IsNamedTuple(val): - val_cls = type(val) - vals = val.__dict__.values() if dataclasses.is_dataclass( - val) else val._asdict().values() - param_pb.named_tuple_val.type = inspect.getmodule( - val_cls).__name__ + '/' + val_cls.__name__ - param_pb.named_tuple_val.items.extend([_ToParamValue(v) for v in vals]) - elif isinstance(val, tuple): - param_pb.tuple_val.items.extend([_ToParamValue(v) for v in val]) - elif isinstance(val, dict): - param_pb.dict_val.SetInParent() - for k, v in val.items(): - param_pb.dict_val.items[k].CopyFrom(_ToParamValue(v)) - elif isinstance(val, type): - param_pb.type_val = inspect.getmodule(val).__name__ + '/' + val.__name__ - elif isinstance(val, tf.DType): - param_pb.dtype_val = val.name - elif isinstance(val, str): - param_pb.string_val = val - elif isinstance(val, bool): - param_pb.bool_val = val - elif isinstance(val, int): - param_pb.int_val = val - elif isinstance(val, float): - param_pb.float_val = val - elif isinstance(val, enum.Enum): - enum_cls = type(val) - param_pb.enum_val.type = inspect.getmodule( - enum_cls).__name__ + '/' + enum_cls.__name__ - param_pb.enum_val.name = val.name - elif isinstance(val, message.Message): - proto_cls = type(val) - param_pb.proto_val.type = inspect.getmodule( - proto_cls).__name__ + '/' + proto_cls.__name__ - param_pb.proto_val.val = val.SerializeToString() - elif val is None: - # We represent a NoneType by the absence of any of the oneof. - pass - else: - raise AttributeError('Unsupported type: %s' % type(val)) - return param_pb - - def _ToParam(val): - """Serializes to Hyperparam proto.""" - - param_pb = hyperparams_pb2.Hyperparam() - for k, v in val.IterParams(): - param_pb.items[k].CopyFrom(_ToParamValue(v)) - return param_pb - - return _ToParam(self) - - # TODO(tonybruguier): Move to module-level function (cls is never used). - @classmethod - def FromProto(cls, param_pb): - """Reads from a Hyperparams proto.""" - - def _LoadClass(module_and_class_name): - tokens = module_and_class_name.split('/') - assert len(tokens) == 2, module_and_class_name - return getattr(importlib.import_module(tokens[0]), tokens[1]) - - def _FromParamValue(param_pb): - """Deserializes HyperparamValue proto.""" - - which_oneof = param_pb.WhichOneof('kind') - if which_oneof == 'param_val': - return _FromParam(param_pb.param_val) - elif which_oneof == 'list_val': - return [_FromParamValue(val) for val in param_pb.list_val.items] - elif which_oneof == 'named_tuple_val': - named_tuple_cls = _LoadClass(param_pb.named_tuple_val.type) - if not dataclasses.is_dataclass(named_tuple_cls) and not issubclass( - named_tuple_cls, tuple): - return None - return named_tuple_cls( - *[_FromParamValue(val) for val in param_pb.named_tuple_val.items]) - elif which_oneof == 'tuple_val': - return tuple([_FromParamValue(val) for val in param_pb.tuple_val.items]) - elif which_oneof == 'dict_val': - dict_val = dict() - for k in param_pb.dict_val.items: - dict_val[k] = _FromParamValue(param_pb.dict_val.items[k]) - return dict_val - elif which_oneof == 'type_val': - tokens = param_pb.type_val.split('/') - assert len(tokens) == 2 - return getattr(importlib.import_module(tokens[0]), tokens[1]) - elif which_oneof == 'dtype_val': - return tf.as_dtype(param_pb.dtype_val) - elif which_oneof == 'string_val': - return param_pb.string_val - elif which_oneof == 'int_val': - return param_pb.int_val - elif which_oneof == 'float_val': - return param_pb.float_val - elif which_oneof == 'bool_val': - return param_pb.bool_val - elif which_oneof == 'enum_val': - enum_cls = _LoadClass(param_pb.enum_val.type) - if not issubclass(enum_cls, enum.Enum): - return None - return enum_cls[param_pb.enum_val.name] - elif which_oneof == 'proto_val': - proto_cls = _LoadClass(param_pb.proto_val.type) - if not issubclass(proto_cls, message.Message): - return None - proto_msg = proto_cls() - proto_msg.ParseFromString(param_pb.proto_val.val) - return proto_msg - else: - # If nothing is set, it's the None type. - return None - - def _FromParam(param_pb): - """Deserializes Hyperparam proto.""" - - params = InstantiableParams() if 'cls' in param_pb.items else Params() - for k in param_pb.items: - val = _FromParamValue(param_pb.items[k]) - if k == 'cls': - params.Set(**{k: val}) - else: - params.Define(k, val, '') - return params - - return _FromParam(param_pb) - - def ToText(self, include_types=False): - """Encodes params into a simple text format. - - Each param is represented as a single line in the output. The param - name and value is separated by a ":". The nest param name is - separated by ".". For values of non-trivial types (types other than - int, float, bool, str, and a few, etc.), we just print out the name - of its type. - - Note that strings are enclosed in appropriate single or double quotes - (whichever would involve the least escaping) and will have some characters - backslash escaped. String properties can span multiple lines. - - Args: - include_types: Should we return types of the values. If True, the types - dict will be returned as a second val in a return tuple - - Returns: - The encoded text or (encoded text, types dict) if include_types is True. - """ - kv = {} - types = {} - - def GetRepr(val): - """Get the representation of `val`.""" - if isinstance(val, Params): - return _SortedDict({k: GetRepr(v) for k, v in val.IterParams()}) - if isinstance(val, dict): - return _SortedDict({k: GetRepr(v) for k, v in val.items()}) - if dataclasses.is_dataclass(val): - return _SortedDict({k: GetRepr(v) for k, v in val.__dict__.items()}) - if _IsNamedTuple(val): - return _SortedDict({k: GetRepr(v) for k, v in val._asdict().items()}) - if isinstance(val, (list, tuple)): - return type(val)([GetRepr(v) for v in val]) - if isinstance(val, (int, float, bool, str, enum.Enum)): - return val - if isinstance(val, tf.DType): - return val.name - if isinstance(val, message.Message): - proto_str = text_format.MessageToString(val, as_one_line=True) - return 'proto/%s/%s/%s' % (inspect.getmodule(val).__name__, - type(val).__name__, proto_str) - if isinstance(val, type): - return 'type/' + inspect.getmodule(val).__name__ + '/' + val.__name__ - return type(val).__name__ - - def Traverse(p, prefix, kv): - """Traverses 'p' and inserts key-value pairs to 'kv'.""" - if isinstance(p, Params): - for key, val in p.IterParams(): - Traverse(val, prefix + '.' + key, kv) - elif (isinstance(p, (list, tuple)) and - all(isinstance(x, Params) for x in p)): - for i, val in enumerate(p): - Traverse(val, '%s[%d]' % (prefix, i), kv) - elif isinstance(p, str): - kv[prefix] = _QuoteString(p) - types[prefix[1:]] = 'str' - else: - kv[prefix] = str(GetRepr(p)) - types[prefix[1:]] = type(p).__name__ - - Traverse(self, '', kv) - ret = '' - for (k, v) in sorted(kv.items()): - ret += k[1:] + ' : ' + v + '\n' - - return (ret, types) if include_types else ret - - def FromText(self, text, type_overrides=None): - """Merges params specified in 'text' into 'params'. - - 'text' follows the simple text format as produced by ToText. - For a param specified in both 'params' and 'text', overwrites the value in - 'params' according to 'text'. Params specified in 'text' but not in 'params' - are ignored. - - Args: - text: A text representation of params. - type_overrides: Overrides for the types of the params. - Raises: - AttributeError: text contains invalid parameter key - ValueError: text contains invalid parameter value, or the format is - wrong. - """ - if self._immutable: - raise TypeError('This Params instance is immutable.') - kv = {} - type_overrides = type_overrides or {} - string_continue = None # None or (key, quote, value) - for line in text.split('\n'): - # Continuing a multi-line string. - if string_continue: - value_stripped = line.rstrip() - if not _EndsWithTerminalQuote(value_stripped, string_continue[1]): - # String continues - string_continue = (string_continue[0], string_continue[1], - string_continue[2] + '\n' + line) - continue - # String terminates. - kv[string_continue[0]] = string_continue[2] + '\n' + value_stripped - string_continue = None - continue - - # Regular line. - line = line.strip() - if not line or line[0] == '#': - # empty line or comment - continue - pair = line.split(':', 1) - if len(pair) == 2: - key = pair[0].strip() - value = pair[1].lstrip() - value_stripped = value.rstrip() - # Detect single vs multi-line string start. - if value and value[0] in ['"', '\'']: - quote_char = value[0] - if not _EndsWithTerminalQuote(value[1:], quote_char): - # Multi-line string. - string_continue = (key, quote_char, value) - continue - kv[key] = value_stripped - else: - raise ValueError('Line {} is not in : format'.format(line)) - - def _ValueFromText(key, old_val, val): - """Returns the new param value from its text representation.""" - val_type = type(old_val).__name__ - if isinstance(old_val, str): - val_type = 'str' - if key in type_overrides: - val_type = type_overrides[key] - # Converts val (a string) to a best-guessed typed value. - if val_type == 'bool': - return val and (val != 'False') and (val != 'false') - elif val_type == 'int': - return int(val) - elif val_type == 'float': - return float(val) - elif val_type == 'DType': - return tf.as_dtype(val) - elif dataclasses.is_dataclass(old_val) or _IsNamedTuple(old_val): - # Maps field name to new value (or its string repr, if non-POD). - name_to_new_value = ast.literal_eval(val) - contents = {} - items = old_val.__dict__.items() if dataclasses.is_dataclass( - old_val) else old_val._asdict().items() - for k, old_field_value in items: - new_field_value = name_to_new_value[k] - # Recurse to parse any non-POD contents not converted by - # literal_eval(). - if isinstance(new_field_value, str): - contents[k] = _ValueFromText(k, old_field_value, new_field_value) - else: - contents[k] = new_field_value - return type(old_val)(**contents) - elif val_type in ['list', 'tuple']: - return ast.literal_eval(val) - elif val_type == 'dict': - return ast.literal_eval(val) if val != 'dict' else {} - elif val_type == 'str': - val = _UnquoteString(val) - if val.startswith('[') and val.endswith(']'): - # We may have stored a list as a string, try converting to a list. - # In case of ValueError - use the string as is. - try: - return ast.literal_eval(val) - except ValueError: - pass - return val - elif isinstance(old_val, enum.Enum): - cls, _, name = val.rpartition('.') - if val_type != cls: - raise ValueError('Expected enum of class %s but got %s' % - (val_type, cls)) - return type(old_val)[name] - elif (isinstance(old_val, type) or isinstance(old_val, message.Message) or - old_val is None): - if val == 'NoneType': - return None - elif old_val is None and val in ('False', 'false'): - return False - elif old_val is None and val in ('True', 'true'): - return True - else: - try: - val_type, pkg, cls = val.split('/', 2) - if val_type == 'type': - return getattr(sys.modules[pkg], cls) - elif val_type == 'proto': - cls, proto_str = cls.split('/', 1) - proto_cls = getattr(sys.modules[pkg], cls) - if not issubclass(proto_cls, message.Message): - raise ValueError('%s is not a proto class.' % proto_cls) - return text_format.Parse(proto_str, proto_cls()) - except ValueError as e: - raise ValueError('Error processing %r : %r with %r' % (key, val, e)) - else: - raise ValueError('Failed to read a parameter: %r : %r' % (key, val)) - - for key, val in kv.items(): - old_val = self.Get(key) - new_val = _ValueFromText(key, old_val, val) - self.Set(**{key: new_val}) - - def ToTextWithTypes(self): - """Same as ToText but encodes both params and their types.""" - text, types = self.ToText(include_types=True) - text += '\n\n' - for (k, v) in sorted(types.items()): - text += k + ' : ' + v + '\n' - return text - - def FromTextWithTypes(self, text): - """Same as FromText but expects to have types encoded in the text.""" - text, types_str = text.split('\n\n') - types = {} - for row in types_str.split('\n'): - if not row: - continue - k, v = row.split(':') - types[k.strip()] = v.strip() - self.FromText(text, type_overrides=types) - - def TextDiff(self, other): - """Return the differences between this object and another as a string. - - Args: - other: The other Params object. - - Returns: - A string of differences. - """ - - def TextDiffHelper(a, b, spaces): - """Return the differences between a and b as a string.""" - a_keys = set([key for key, _ in a.IterParams()]) - b_keys = set([key for key, _ in b.IterParams()]) - all_keys = a_keys.union(b_keys) - diff = '' - for key in sorted(all_keys): - if key in a_keys and key not in b_keys: - diff += '>' + spaces + key + ': ' + str(a.Get(key)) + '\n' - elif key in b_keys and key not in a_keys: - diff += '<' + spaces + key + ': ' + str(b.Get(key)) + '\n' - elif a.Get(key) != b.Get(key): - if isinstance(a.Get(key), Params): - diff += '?' + spaces + key + ':\n' - diff += TextDiffHelper(a.Get(key), b.Get(key), spaces + ' ') - else: - diff += '>' + spaces + key + ': ' + str(a.Get(key)) + '\n' - diff += '<' + spaces + key + ': ' + str(b.Get(key)) + '\n' - return diff - - return TextDiffHelper(self, other, spaces=' ') - - -class InstantiableParams(Params): - """Params which can be instantiated. - - When using InstantiableParams, callers must provide a class which supports - initialization using a Params instance. - - This covers a common use case of Params to hold a configuration for a given - class. - """ - - def __init__(self, cls=None): - super().__init__() - self.Define('cls', cls, 'Cls that this param object is associated with.') - - def Instantiate(self, **args): - """Instantiate an instance that this Params is configured for. - - Example: - params = InstantiableParams(cls=MyObject) - params.Define('weight', 0.2, 'Training weight.') - params.weight = 0.9 - obj = params.Instantiate() - - It's common for classes to have a classmethod called Params that returns - a pre-made InstantiableParams, like this: - - params = MyObject.Params() - params.weight = 0.9 - obj = params.Instantiate() - - By convention, anything that parameterizes the behavior of your class - should be stored in this Params object. However, your class may also use - shared state objects which aren't really parameters, like a shared lock. - These can be passed as extra arguments to Instantiate. - - Example: - lock = threading.Lock() - params = MyObject.Params() - obj_a = params.Instantiate(lock=lock) - obj_b = params.Instantiate(lock=lock) - - Args: - **args: Additional keyword arguments to pass to the constructor in - addition to this Params object. - - Returns: - A constructed object where type(object) == cls. - """ - assert self.cls is not None - - # The class initializer is expected to support initialization using Params. - return self.cls(self, **args) - - def Copy(self): - """See base class.""" - return self._CopyTo(type(self)(self.cls)) diff --git a/lingvo/core/hyperparams_test.py b/lingvo/core/hyperparams_test.py deleted file mode 100644 index 6eae84fa..00000000 --- a/lingvo/core/hyperparams_test.py +++ /dev/null @@ -1,613 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for hyperparams.""" - -import collections -import enum - -import dataclasses -import lingvo.compat as tf -from lingvo.core import hyperparams as _params -from lingvo.core import hyperparams_pb2 -from lingvo.core import symbolic -from lingvo.core import test_utils - -FLAGS = tf.flags.FLAGS - - -class TestClass1: - """This class is used in ParamsTest.testToText as a value of a variable.""" - pass - - -class TestClass2: - """This class is used in ParamsTest.testToText as a value of a variable.""" - pass - - -class TestEnum(enum.Enum): - """Test enum class.""" - A = 1 - B = 2 - - -@dataclasses.dataclass -class TestDataClass: - """Test dataclasses.dataclass.""" - a: str - b: tf.DType - - -class TestNamedTuple(collections.namedtuple('TestNamedTuple', ['a', 'b'])): - """Test namedtuple class.""" - pass - - -class InstantiableClass: - """Used for testing InstantiableParams.""" - - def __init__(self, params, other=None): - self.params = params - self.other = other - - -class ParamsTest(test_utils.TestCase): - - def testEquals(self): - # pylint: disable=g-generic-assert - params1 = _params.Params() - params2 = _params.Params() - self.assertTrue(params1 == params2) - params1.Define('first', 'firstvalue', '') - self.assertFalse(params1 == params2) - params2.Define('first', 'firstvalue', '') - self.assertTrue(params1 == params2) - some_object = object() - other_object = object() - params1.Define('second', some_object, '') - params2.Define('second', other_object, '') - self.assertFalse(params1 == params2) - params2.second = some_object - self.assertTrue(params1 == params2) - params1.Define('third', _params.Params(), '') - params2.Define('third', _params.Params(), '') - self.assertTrue(params1 == params2) - params1.third.Define('fourth', 'x', '') - params2.third.Define('fourth', 'y', '') - self.assertFalse(params1 == params2) - params2.third.fourth = 'x' - self.assertTrue(params1 == params2) - # Comparing params to non-param instances. - self.assertFalse(params1 == 3) - self.assertFalse(3 == params1) - # pylint: enable=g-generic-assert - - def testDeepCopy(self): - inner = _params.Params() - inner.Define('alpha', 2, '') - inner.Define('tensor', tf.constant(0), '') - inner.Define('symbol', symbolic.Symbol('symbol'), '') - outer = _params.Params() - outer.Define('beta', 1, '') - outer.Define('inner', inner, '') - outer_copy = outer.Copy() - self.assertIsNot(outer, outer_copy) - self.assertEqual(outer, outer_copy) - self.assertIsNot(outer.inner, outer_copy.inner) - self.assertEqual(outer.inner, outer_copy.inner) - self.assertEqual(outer.inner.alpha, outer_copy.inner.alpha) - self.assertIs(outer.inner.tensor, outer_copy.inner.tensor) - self.assertIs(outer.inner.symbol, outer_copy.inner.symbol) - - def testCopyFieldsTo(self): - source = _params.Params() - dest = _params.Params() - source.Define('a', 'a', '') - source.Define('b', 'b', '') - source.Define('c', 'c', '') - dest.Define('a', '', '') - _params.CopyFieldsTo(source, dest, skip=['b', 'c']) - self.assertEqual(source.a, dest.a) - self.assertNotIn('b', dest) - self.assertNotIn('c', dest) - - def testCopyFieldsToDoesNotCopyClass(self): - source = _params.InstantiableParams(cls=_params.Params) - dest = _params.InstantiableParams(cls=_params.InstantiableParams) - _params.CopyFieldsTo(source, dest) - self.assertEqual(dest.cls, _params.InstantiableParams) - - def testDefineExisting(self): - p = _params.Params() - p.Define('foo', 1, '') - self.assertRaisesRegex(AttributeError, 'already defined', - lambda: p.Define('foo', 1, '')) - - def testLegalParamNames(self): - p = _params.Params() - self.assertRaises(AssertionError, lambda: p.Define(None, 1, '')) - self.assertRaises(AssertionError, lambda: p.Define('', 1, '')) - self.assertRaises(AssertionError, lambda: p.Define('_foo', 1, '')) - self.assertRaises(AssertionError, lambda: p.Define('Foo', 1, '')) - self.assertRaises(AssertionError, lambda: p.Define('1foo', 1, '')) - self.assertRaises(AssertionError, lambda: p.Define('foo$', 1, '')) - p.Define('foo_bar', 1, '') - p.Define('foo9', 1, '') - - def testSetAndGet(self): - p = _params.Params() - self.assertRaisesRegex(AttributeError, 'foo', lambda: p.Set(foo=4)) - # We use setattr() because lambda cannot contain explicit assignment. - self.assertRaisesRegex(AttributeError, 'foo', lambda: setattr(p, 'foo', 4)) - p.Define('foo', 1, '') - self.assertEqual(p.foo, 1) - self.assertEqual(p.Get('foo'), 1) - self.assertIn('foo', p) - self.assertNotIn('bar', p) - p.Set(foo=2) - self.assertEqual(p.foo, 2) - self.assertEqual(p.Get('foo'), 2) - p.foo = 3 - self.assertEqual(p.foo, 3) - self.assertEqual(p.Get('foo'), 3) - p.Delete('foo') - self.assertNotIn('foo', p) - self.assertNotIn('bar', p) - self.assertRaisesRegex(AttributeError, 'foo', lambda: p.foo) - self.assertRaisesRegex(AttributeError, 'foo', p.Get, 'foo') - - def testSetAndGetNestedParam(self): - innermost = _params.Params() - innermost.Define('delta', 22, '') - innermost.Define('zeta', 5, '') - - inner = _params.Params() - inner.Define('alpha', 2, '') - inner.Define('innermost', innermost, '') - - outer = _params.Params() - outer.Define('beta', 1, '') - outer.Define('inner', inner, '') - outer.Define('d', dict(foo='bar'), '') - - self.assertEqual(inner.alpha, 2) - self.assertEqual(outer.beta, 1) - self.assertEqual(outer.d['foo'], 'bar') - self.assertEqual(outer.inner.alpha, 2) - self.assertEqual(outer.inner.innermost.delta, 22) - self.assertEqual(outer.inner.innermost.zeta, 5) - - self.assertEqual(inner.Get('alpha'), 2) - self.assertEqual(outer.Get('beta'), 1) - self.assertEqual(outer.Get('d')['foo'], 'bar') - self.assertEqual(outer.Get('inner.alpha'), 2) - self.assertEqual(outer.Get('inner.innermost.delta'), 22) - self.assertEqual(outer.Get('inner.innermost.zeta'), 5) - - outer.Set(**{'inner.alpha': 3}) - outer.Set(d=dict(foo='baq')) - outer.Delete('beta') - outer.Delete('inner.innermost.zeta') - - self.assertEqual(inner.alpha, 3) - self.assertRaisesRegex(AttributeError, 'beta', lambda: outer.beta) - self.assertEqual(outer.d['foo'], 'baq') - self.assertEqual(outer.inner.alpha, 3) - self.assertEqual(outer.inner.innermost.delta, 22) - self.assertRaisesRegex(AttributeError, 'zeta', - lambda: outer.inner.innermost.zeta) - - self.assertEqual(inner.Get('alpha'), 3) - self.assertRaisesRegex(AttributeError, 'beta', outer.Get, 'beta') - self.assertEqual(outer.Get('d')['foo'], 'baq') - self.assertEqual(outer.Get('inner.alpha'), 3) - self.assertEqual(outer.Get('inner.innermost.delta'), 22) - self.assertRaisesRegex(AttributeError, 'inner.innermost.zeta', outer.Get, - 'inner.innermost.zeta') - - # NOTE(igushev): Finding nested Param object is shared between Get, Set and - # Delete, so we test only Set. - self.assertRaisesRegex(AttributeError, r'inner\.gamma', - lambda: outer.Set(**{'inner.gamma': 5})) - self.assertRaisesRegex(AttributeError, r'inner\.innermost\.bad', - lambda: outer.Set(**{'inner.innermost.bad': 5})) - self.assertRaisesRegex(AssertionError, '^Cannot introspect', - lambda: outer.Set(**{'d.foo': 'baz'})) - - def testFreeze(self): - p = _params.Params() - self.assertRaises(AssertionError, lambda: p.Define('_immutable', 1, '')) - self.assertRaisesRegex(AttributeError, 'foo', lambda: p.Set(foo=4)) - # We use setattr() because lambda cannot contain explicit assignment. - self.assertRaisesRegex(AttributeError, 'foo', lambda: setattr(p, 'foo', 4)) - p.Define('foo', 1, '') - p.Define('nested', p.Copy(), '') - self.assertEqual(p.foo, 1) - self.assertEqual(p.Get('foo'), 1) - self.assertEqual(p.nested.foo, 1) - p.Freeze() - - self.assertRaises(TypeError, lambda: p.Set(foo=2)) - self.assertEqual(p.Get('foo'), 1) - self.assertRaises(TypeError, lambda: setattr(p, 'foo', 3)) - self.assertEqual(p.foo, 1) - self.assertRaises(TypeError, lambda: p.Delete('foo')) - self.assertEqual(p.foo, 1) - self.assertRaises(TypeError, lambda: p.Define('bar', 1, '')) - self.assertRaisesRegex(AttributeError, 'bar', p.Get, 'bar') - - p.nested.foo = 2 - self.assertEqual(p.foo, 1) - self.assertEqual(p.nested.foo, 2) - - self.assertRaises(TypeError, lambda: setattr(p, '_immutable', False)) - - # Copies are still immutable. - q = p.Copy() - self.assertRaises(TypeError, lambda: q.Set(foo=2)) - - def testToString(self): - outer = _params.Params() - outer.Define('foo', 1, '') - inner = _params.Params() - inner.Define('bar', 2, '') - outer.Define('inner', inner, '') - outer.Define('list', [1, inner, 2], '') - outer.Define('dict', {'a': 1, 'b': inner}, '') - outer.Define('enum', TestEnum.B, '') - self.assertEqual( - '\n' + str(outer), """ -{ - dict: {'a': 1, 'b': {'bar': 2}} - enum: TestEnum.B - foo: 1 - inner: { - bar: 2 - } - list: [1, {'bar': 2}, 2] -}""") - - def testIterParams(self): - keys, values = ['a', 'b', 'c', 'd', 'e'], [True, None, 'zippidy', 78.5, 5] - p = _params.Params() - for k, v in zip(keys, values): - p.Define(k, v, 'description of %s' % k) - - k_set, v_set = set(keys), set(values) - number_of_params = 0 - for k, v in p.IterParams(): - self.assertIn(k, k_set) - self.assertIn(v, v_set) - number_of_params += 1 - self.assertEqual(number_of_params, len(keys)) - - def testToText(self): - outer = _params.Params() - outer.Define('foo', 1, '') - inner = _params.Params() - inner.Define('bar', 2.71, '') - inner.Define('baz', 'hello', '') - outer.Define('inner', inner, '') - outer.Define('tau', False, '') - outer.Define('dtype', tf.float32, '') - outer.Define('dtype2', tf.int32, '') - outer.Define('seqlen', [10, inner, 30], '') - outer.Define('tuple', (1, None), '') - outer.Define('list_of_params', [inner.Copy()], '') - outer.Define('class', TestClass1, '') - outer.Define('plain_dict', {'a': 10}, '') - outer.Define('complex_dict', {'a': 10, 'b': inner}, '') - outer.Define('complex_dict_escape', {'a': 'abc"\'\ndef'}, '') - outer.Define('some_class', complex(0, 1), '') - outer.Define('optional_bool', None, '') - outer.Define('enum', TestEnum.B, '') - outer.Define('dataclass', TestDataClass(a=[42], b=tf.float32), '') - outer.Define('namedtuple', TestNamedTuple([42], tf.float32), '') - outer.Define('namedtuple2', tf.io.FixedLenSequenceFeature([42], tf.float32), - '') - # Arbitrarily use HyperparameterValue as some example proto. - outer.Define('proto', hyperparams_pb2.HyperparamValue(int_val=42), '') - - self.assertEqual( - '\n' + outer.ToText(), r""" -class : type/__main__/TestClass1 -complex_dict : {'a': 10, 'b': {'bar': 2.71, 'baz': 'hello'}} -complex_dict_escape : {'a': 'abc"\'\ndef'} -dataclass : {'a': [42], 'b': 'float32'} -dtype : float32 -dtype2 : int32 -enum : TestEnum.B -foo : 1 -inner.bar : 2.71 -inner.baz : 'hello' -list_of_params[0].bar : 2.71 -list_of_params[0].baz : 'hello' -namedtuple : {'a': [42], 'b': 'float32'} -namedtuple2 : {'allow_missing': False, 'default_value': 'NoneType', 'dtype': 'float32', 'shape': [42]} -optional_bool : NoneType -plain_dict : {'a': 10} -proto : proto/lingvo.core.hyperparams_pb2/HyperparamValue/int_val: 42 -seqlen : [10, {'bar': 2.71, 'baz': 'hello'}, 30] -some_class : complex -tau : False -tuple : (1, 'NoneType') -""") - - outer.FromText(""" - dataclass : {'a': 27, 'b': 'int32'} - dtype2 : float32 - inner.baz : 'world' - # foo : 123 - optional_bool : true - list_of_params[0].bar : 2.72 - seqlen : [1, 2.0, '3', [4]] - plain_dict : {'x': 0.3} - class : type/__main__/TestClass2 - tau : true - tuple : (2, 3) - enum : TestEnum.A - # Note dtypes and other non-POD are represented as strings. - namedtuple : {'a': 27, 'b': 'int32'} - namedtuple2 : {'allow_missing': True, 'default_value': 'NoneType', 'dtype': 'int32', 'shape': [43]} - proto : proto/lingvo.core.hyperparams_pb2/HyperparamValue/string_val: "a/b" - """) - - # Note that the 'hello' has turned into 'world'! - self.assertEqual( - '\n' + outer.ToText(), r""" -class : type/__main__/TestClass2 -complex_dict : {'a': 10, 'b': {'bar': 2.71, 'baz': 'world'}} -complex_dict_escape : {'a': 'abc"\'\ndef'} -dataclass : {'a': 27, 'b': 'int32'} -dtype : float32 -dtype2 : float32 -enum : TestEnum.A -foo : 1 -inner.bar : 2.71 -inner.baz : 'world' -list_of_params[0].bar : 2.72 -list_of_params[0].baz : 'hello' -namedtuple : {'a': 27, 'b': 'int32'} -namedtuple2 : {'allow_missing': True, 'default_value': 'NoneType', 'dtype': 'int32', 'shape': [43]} -optional_bool : True -plain_dict : {'x': 0.3} -proto : proto/lingvo.core.hyperparams_pb2/HyperparamValue/string_val: "a/b" -seqlen : [1, 2.0, '3', [4]] -some_class : complex -tau : True -tuple : (2, 3) -""") - self.assertEqual(outer.dataclass.b, tf.int32) - self.assertEqual(outer.namedtuple.b, tf.int32) - self.assertEqual(outer.namedtuple2.dtype, tf.int32) - self.assertIsNone(outer.namedtuple2.default_value, tf.int32) - - def testToFromProto(self): - outer = _params.Params() - outer.Define('integer_val', 1, '') - outer.Define('cls_type', type(int), '') - inner = _params.Params() - inner.Define('float_val', 2.71, '') - inner.Define('string_val', 'rosalie et adrien', '') - inner.Define('bool_val', True, '') - inner.Define('list_of_tuples_of_dicts', [({'string_key': 1729})], '') - inner.Define('range', range(1, 3), '') - outer.Define('inner', inner, '') - outer.Define('empty_list', [], '') - outer.Define('empty_tuple', (), '') - outer.Define('empty_dict', {}, '') - outer.Define('enum', TestEnum.B, '') - outer.Define('proto', hyperparams_pb2.HyperparamValue(int_val=42), '') - outer.Define('dataclass', TestDataClass(a=[42], b=tf.float32), '') - outer.Define('namedtuple', tf.io.FixedLenSequenceFeature([42], tf.float32), - '') - - rebuilt_outer = _params.InstantiableParams.FromProto(outer.ToProto()) - - self.assertNotIn('cls', rebuilt_outer) - self.assertEqual(outer.integer_val, rebuilt_outer.integer_val) - self.assertEqual(outer.cls_type, rebuilt_outer.cls_type) - self.assertNear(outer.inner.float_val, rebuilt_outer.inner.float_val, 1e-6) - self.assertEqual(outer.inner.string_val, rebuilt_outer.inner.string_val) - self.assertEqual(outer.inner.bool_val, rebuilt_outer.inner.bool_val) - self.assertEqual(outer.inner.list_of_tuples_of_dicts, - rebuilt_outer.inner.list_of_tuples_of_dicts) - self.assertEqual([1, 2], rebuilt_outer.inner.range) # Rebuilt as list. - self.assertEqual(outer.empty_list, rebuilt_outer.empty_list) - self.assertEqual(outer.empty_tuple, rebuilt_outer.empty_tuple) - self.assertEqual(outer.empty_dict, rebuilt_outer.empty_dict) - self.assertEqual(outer.enum, rebuilt_outer.enum) - self.assertEqual(outer.proto, rebuilt_outer.proto) - self.assertEqual(outer.dataclass, rebuilt_outer.dataclass) - self.assertEqual(outer.namedtuple, rebuilt_outer.namedtuple) - - def testStringEscaping(self): - p = _params.Params() - p.Define('bs_end_quote', 'Single\\', '') - p.Define('embedded_newlines', 'Split\nAcross\nLines', '') - p.Define('empty', '', '') - p.Define('empty_first_line', '\nNext', '') - p.Define('end_escape_quote', '""Split\'\nLine', '') - p.Define('escaping_single', 'In "quotes"', '') - p.Define('escaping_double', 'In \\\'quotes\'', '') - - # Make sure it escapes properly. - text_value = p.ToText() - self.assertEqual( - '\n' + text_value, r""" -bs_end_quote : 'Single\\' -embedded_newlines : 'Split -Across -Lines' -empty : '' -empty_first_line : ' -Next' -end_escape_quote : '""Split\' -Line' -escaping_double : "In \\'quotes'" -escaping_single : 'In "quotes"' -""") - - # Reset the values and make sure that reading back in parses. - p.bs_end_quote = '' - p.embedded_newlines = '' - p.empty_first_line = '' - p.end_escape_quote = '' - p.escaping_double = '' - p.escaping_single = '' - - p.FromText(text_value) - self.assertEqual(p.bs_end_quote, 'Single\\') - self.assertEqual(p.embedded_newlines, 'Split\nAcross\nLines') - self.assertEqual(p.empty_first_line, '\nNext') - self.assertEqual(p.end_escape_quote, '""Split\'\nLine') - self.assertEqual(p.escaping_single, 'In "quotes"') - self.assertEqual(p.escaping_double, 'In \\\'quotes\'') - - def testFromToText(self): - p = _params.Params() - p.Define('activation', 'RELU', 'Can be a string or a list of strings.') - np = p.Copy() - p.Set(activation=['RELU', 'NONE']) - np.FromText(p.ToText()) - self.assertEqual(np.activation, ['RELU', 'NONE']) - - def testSimilarKeys(self): - p = _params.Params() - p.Define('activation', 'RELU', 'Can be a string or a list of strings.') - p.Define('activations', 'RELU', 'Many activations.') - p.Define('cheesecake', None, 'dessert') - p.Define('tofu', None, 'not dessert') - - def set_param(): - p.actuvation = 1 - - self.assertRaisesWithLiteralMatch( - AttributeError, 'actuvation (did you mean: [activation,activations])', - set_param) - - def testFromToTextTypes(self): - p = _params.Params() - p.Define('scale', None, 'A float scale but default is None.') - np1 = p.Copy() - np2 = p.Copy() - p.Set(scale=1.0) - self.assertRaises(ValueError, lambda: np1.FromText(p.ToText())) - text, types = p.ToText(include_types=True) - self.assertEqual(types['scale'], 'float') - np2.FromText(text, type_overrides=types) - self.assertEqual(np2.scale, 1.0) - - def testFromTextBadFormat(self): - p = _params.Params() - p.Define('scale', 1.0, 'A float parameter.') - np1 = p.Copy() - self.assertRaises(ValueError, lambda: np1.FromText('scale=2.0')) - np2 = p.Copy() - np2.FromText('scale:2.0') - self.assertEqual(np2.scale, 2.0) - - def testTypeOverride(self): - p = _params.Params() - p.Define('scale', '1', 'A str that will be overriden by float.') - np1 = p.Copy() - np2 = p.Copy() - p.Set(scale=2.1) - - # Check that type is erased if types are not used (old behavior). - no_types_text = p.ToText() - np1.FromText(no_types_text) - self.assertEqual(np1.scale, '2.1') - - # Check that type is not erased if we use type_overrides. - types_text, types = p.ToText(include_types=True) - self.assertEqual(types['scale'], 'float') - np2.FromText(types_text, type_overrides=types) - self.assertEqual(np2.scale, 2.1) - - def testDeterministicSerialize(self): - p = _params.Params() - p.Define('a', 42, '') - p.Define('b', None, '') - p.Define('c', 'C', '') - p.Define('d', None, '') - pnest = _params.Params() - pnest.Define('x', 'X', '') - p.Define('e', pnest, '') - p.Define('f', [pnest.Copy().Set(x=2)], '') - pclean = p.Copy() - - p.a = 43 - p.d = [1, 2, 3] - p.e.x = 7 - base_serialized = p.ToTextWithTypes() - for _ in range(10): - serialized = p.ToTextWithTypes() - serialized_copy = p.Copy().ToTextWithTypes() - self.assertEqual(serialized, base_serialized) - self.assertEqual(serialized_copy, base_serialized) - for x in [serialized, serialized_copy]: - deserialized = pclean.Copy() - deserialized.FromTextWithTypes(x) - self.assertEqual(p, deserialized) - - def testDiff(self): - a = _params.Params() - d_inner = _params.Params() - d_inner.Define('hey', 'hi', '') - a.Define('a', 42, '') - a.Define('c', 'C', '') - a.Define('d', d_inner, '') - b = a.Copy() - - # Everything is the same so we don't expect diffs. - self.assertEqual(a.TextDiff(b), '') - - a.a = 43 - self.assertEqual(a.TextDiff(b), '> a: 43\n' '< a: 42\n') - - b.d.hey = 'hello' - self.assertEqual( - a.TextDiff(b), '> a: 43\n' - '< a: 42\n' - '? d:\n' - '> hey: hi\n' - '< hey: hello\n') - - def testInstantiate(self): - a = _params.InstantiableParams(InstantiableClass) - a.Define('new_param', None, 'A meaningless param.') - a.new_param = 'hi' - - obj = a.Instantiate() - self.assertIsInstance(obj, InstantiableClass) - self.assertEqual(obj.params.new_param, 'hi') - - def testInstantiateWithParams(self): - a = _params.InstantiableParams(InstantiableClass) - a.Define('new_param', None, 'A meaningless param.') - a.new_param = 'hi' - - # Same as the previous test, but InstantiableClass should also get - # other=15 passed as a keyword argument to the constructor. - obj = a.Instantiate(other=15) - self.assertIsInstance(obj, InstantiableClass) - self.assertEqual(obj.params.new_param, 'hi') - self.assertEqual(obj.other, 15) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/inference_graph.proto b/lingvo/core/inference_graph.proto deleted file mode 100644 index 0cb45881..00000000 --- a/lingvo/core/inference_graph.proto +++ /dev/null @@ -1,125 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -syntax = "proto2"; - -package tensorflow.lingvo; - -import "google/protobuf/any.proto"; -import "tensorflow/core/framework/graph.proto"; -import "tensorflow/core/framework/types.proto"; -import "tensorflow/core/protobuf/meta_graph.proto"; -import "tensorflow/core/protobuf/saver.proto"; - -// Represents a frozen graph constructed at training time which can be used with -// the Predictor class in predictor.py to perform inference. -message InferenceGraph { - // The graph definition, which can be imported using tf.import_graph_def. - // Note that in some situations, the actual graph def is saved separately - // and may be absent here. - optional tensorflow.GraphDef graph_def = 1; - - // The saver definition, which is used to load checkpoints correctly. Can be - // imported using tf.Saver(saver_def=_). - // Note that if the graph is frozen (all vars converted to constants) and - // can no longer import a checkpoint, this may be absent. - optional tensorflow.SaverDef saver_def = 2; - - // Defines discrete subgraphs of the overall inference graph. - // This is populated if the overall inference graph is generated for piecewise - // evaluation within a larger pipeline. - message Subgraph { - // Subgraph level meta data. - optional SubgraphMeta meta = 1; - - // Maps logical feed/fetch name to tensor name with the GraphDef. - // - // RNN states should be named the same in both the feed and fetch lists. - // Optionally if multiple related RNN states are present they can be of the - // form some_state_name:0, some_state_name:1, ... to indicate to tooling and - // readers that the states are grouped. - map feeds = 2; - map fetches = 3; - - // Separate feed/fetch metadata. - map feeds_meta = 4; - map fetches_meta = 5; - } - map subgraphs = 5; - - // Serialized dump of the model hyperparameters used to generate this - // instance. - optional string hyperparameters = 7; - - // Optional metadata about a subgraph. - message SubgraphMeta { - // Private extension data between producer and driver code. - repeated google.protobuf.Any extensions = 1; - } - - // Optional metadata about a feed or fetch. - // Some of this is redundant with data that can be derived from the GraphDef - // but is useful to have explicitly. - // - // NEXT TAG: 14 - message FeedFetchMeta { - // Private extension data between producer and driver code. - repeated google.protobuf.Any extensions = 1; - - // Datatype of the feed/fetch. - optional tensorflow.DataType data_type = 2; - - // Shape of the feed/fetch. If a dimension is not fixed, it is -1. - // Omitted/empty if shape not specified. - repeated int32 shape = 3 [packed = true]; - - // Char-per-dimension layout of a feed/fetch. - // This is by convention between the producer and driver or just for - // documentation. It is not intended for driving behavior but can be used - // for checks. - // In general, the following is recommended: - // t=time, b=batch, d=depth, f=feature, c=channel, 1=single - // Examples: - // 'tbd': time-batch-depth - // 'btd': batch-time-depth - // 'btfc': batch-time-feature-channel - optional string layout = 4; - - // Axis along which the feed/fetch can be coalesced at runtime. - // For example, a feed having the shape [time, batch, depth] will have a - // dispatch_stride_axis of 0. - optional int32 dispatch_stride_axis = 13; - - // If the feed/fetch is quantized, this is the number of bits in the - // quantized representation. If the ranges are non-const, then the - // *_tensor variants will be populated with the tensor name (which - // will be guaranteed to be a reference within the overall subgraph). - optional int32 quantized_num_bits = 8; - oneof quantized_min_oneof { - double quantized_min_const = 9; - string quantized_min_tensor = 10; - } - oneof quantized_max_oneof { - double quantized_max_const = 11; - string quantized_max_tensor = 12; - } - - reserved 5, 6, 7; // deprecated fields. - } - - reserved 3, 4, 6; // deprecated fields. - - repeated tensorflow.AssetFileDef asset_file_def = 10; -} diff --git a/lingvo/core/inference_graph_exporter.py b/lingvo/core/inference_graph_exporter.py deleted file mode 100644 index 3294a614..00000000 --- a/lingvo/core/inference_graph_exporter.py +++ /dev/null @@ -1,535 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Utility for exporting an InferenceGraph proto from model params.""" - -import collections -import contextlib -import re -import lingvo.compat as tf -from lingvo.core import base_model -from lingvo.core import bfloat16_variables -from lingvo.core import inference_graph_pb2 -from lingvo.core import py_utils -import six - -from google.protobuf import text_format - -FLAGS = tf.flags.FLAGS - -# InferenceDeviceOptions contains options to configure inference on the device. -# device: Device to infer on. -# retain_device_placement: If true, the specified device in the generated -# inference graph nodes will be retained. Otherwise, the specified device -# will be cleared, so that the runtime can choose automatically. -# var_options: Options on handling variables. For TPUs, variables can be -# either placed on device through 'ON_DEVICE' option, or treated as -# constants with AS_CONSTANTS. -# gen_init_op: Whether to serialize initialization ops for the device. For TPUs, -# servers can be initialized globally once, in which case this should be -# turned off to avoid tripping initialization checks. -# dtype_override: Whether to override the dtype to use for activations and -# weights in the model. Options supported are None or tf.bfloat16. -InferenceDeviceOptions = collections.namedtuple('InferenceDeviceOptions', [ - 'device', 'retain_device_placement', 'var_options', 'gen_init_op', - 'dtype_override' -]) - -_CONST_GUARANTEE = None - - -@contextlib.contextmanager -def NoConstGuaranteeScope(): - """Disallow const gauranteeing variable with-in scope.""" - global _CONST_GUARANTEE - var_scope = tf.get_variable_scope() - old_caching_device = var_scope.caching_device - old_val = _CONST_GUARANTEE - var_scope.set_caching_device(None) - _CONST_GUARANTEE = False - yield - _CONST_GUARANTEE = old_val - var_scope.set_caching_device(old_caching_device) - - -# Marks variable as constants for compilation -def MaybeGuaranteeConstGetter(getter, name, *args, **kwargs): - global _CONST_GUARANTEE - if _CONST_GUARANTEE: - with tf.control_dependencies(None): - return tf.guarantee_const( - getter(name, *args, **kwargs), name=name + '/GuaranteeConst') - else: - return getter(name, *args, **kwargs) - - -@contextlib.contextmanager -def ConstGuaranteeScope(): - """Treats all variables under this scope as constants.""" - global _CONST_GUARANTEE - var_scope = tf.get_variable_scope() - old_custom_getter = var_scope.custom_getter - old_caching_device = var_scope.caching_device - old_val = _CONST_GUARANTEE - var_scope.set_custom_getter(MaybeGuaranteeConstGetter) - var_scope.set_caching_device(lambda op: op.device) - _CONST_GUARANTEE = True - yield - _CONST_GUARANTEE = old_val - var_scope.set_custom_getter(old_custom_getter) - var_scope.set_caching_device(old_caching_device) - - -@contextlib.contextmanager -def _DummyScope(): - yield None - - -def _GetVarName(v): - return v.name[:-len(':0')] - - -def _MakeVariableDictionary(variables): - """Returns a dictionary with name -> tf.Variable() mapping.""" - vars_dict = {} - for v in variables: - vars_dict[_GetVarName(v)] = v - return vars_dict - - -def IsTpu(device_options): - return device_options.device == 'tpu' - - -def ShouldForceBfloat16ForWeightsAndActivations(device_options): - return device_options.dtype_override == tf.bfloat16 - - -def ConvertSubgraphDictToProto(subgraphs_dict): - """Converts dict of subgraphs/feeds/fetches to InferenceGraph. - - Args: - subgraphs_dict: Dict of (fetches, feeds) where each fetches/feeds is a - NestedMap. - - Returns: - Equivalent InferenceGraph. - """ - # Build the output inference graph. - inference_graph_proto = inference_graph_pb2.InferenceGraph() - for subgraph_name, tensors in subgraphs_dict.items(): - fetches = tensors[0] - feeds = tensors[1] - - # Rewrite fetches and feeds to map to their tensor name instead of - # Tensor instance. - named_fetches = {k: v.name for k, v in fetches.items() if v is not None} - named_feeds = {k: v.name for k, v in feeds.items() if v is not None} - - # Export as subgraph. - inference_graph_proto.subgraphs[subgraph_name].fetches.update(named_fetches) - inference_graph_proto.subgraphs[subgraph_name].feeds.update(named_feeds) - return inference_graph_proto - - -def GetOutputOpNames(graph, - inference_graph_proto, - subgraphs=None, - preserve_colocation_nodes=True, - preserve_saver_restore_nodes=False, - preserve_extra_ops=None): - """Gets output op names from an inference graph. - - Args: - graph: The tf graph. - inference_graph_proto: an InferenceGraph proto. - subgraphs: an optional list of subgraph names. If provided, only output ops - from these subgraphs are preserved. Otherwise, all subgraphs are included. - preserve_colocation_nodes: a Python bool, default to True. Preserves nodes - colocating with the closure of output ops in the returned array. - preserve_saver_restore_nodes: a Python bool, default to True. Preserves - nodes for restoring according to inference_graph_proto.saver_def. - preserve_extra_ops: an optional list of extra op names to preserve as long - as they present in the graph. - - Returns: - Array of tf op names that should be preserved in the graph. - """ - output_op_names = set() - - def _GetOpName(tensor_or_op_name): - """Returns the op name of the given node name.""" - # Tensor names have format :. Some inference - # graphs put tensors and others put ops in the feeds/fetches (depends - # on how it is used). We differentiate here. We still do the lookup in - # the graph to sanity check (versus relying on the text manipulation). - # If this logic ever breaks, TensorFlow will raise a ValueError with - # a description of the syntax of each. - if re.search(r':[0-9]+$', tensor_or_op_name): - # Tensor-name. - t = graph.get_tensor_by_name(tensor_or_op_name) - return t.op.name - else: - op = graph.get_operation_by_name(tensor_or_op_name) - return op.name - - for subgraph_name, subgraph in inference_graph_proto.subgraphs.items(): - if subgraphs and subgraph_name not in subgraphs: - tf.logging.info('Skip subgraph %s.', subgraph_name) - continue - # Sometimes feeds aren't connected to any outputs but keep them in the graph - # anyways to avoid errors. - for tensor_or_op_name in ( - list(subgraph.feeds.values()) + list(subgraph.fetches.values())): - output_op_names.add(_GetOpName(tensor_or_op_name)) - - if preserve_saver_restore_nodes: - # Only nodes for restoring is preserved. saver_def.save_tensor_name is - # skipped because it's only used for saving. - saver_def = inference_graph_proto.saver_def - output_op_names.add(_GetOpName(saver_def.filename_tensor_name)) - output_op_names.add(_GetOpName(saver_def.restore_op_name)) - - if not preserve_colocation_nodes and not preserve_extra_ops: - return sorted(list(output_op_names)) - - # We also need to preserve any nodes that are used for colocation. - # E.g., a node may have this attr: - # attr { - # key: "_class" - # value { - # list { - # s: "loc:@inference/embedding_lookup/Read/ReadVariableOp" - # } - # } - # } - # - # In this case, we need to make sure the node - # inference/embedding_lookup/Read/ReadVariableOp is not pruned. - # - # TODO(zhifengc): It's possible that it's better to fix in - # tf.graph_util.extract_sub_graph. - graph_def = tf.graph_util.extract_sub_graph(graph.as_graph_def(), - list(output_op_names)) - reachable_vars = [node.name for node in graph_def.node] - - for node in graph.get_operations(): - if preserve_extra_ops and node.name in preserve_extra_ops: - output_op_names.add(node.name) - elif preserve_colocation_nodes and '_class' in node.node_def.attr: - for loc in node.node_def.attr['_class'].list.s: - loc = six.ensure_text(loc, 'utf-8') - if loc.startswith('loc:@'): - loc_name = loc[5:] - if loc_name not in reachable_vars: - # Skip nodes that cannot be reached from the pruned graph. - continue - output_op_names.add(node.name) - - return sorted(list(output_op_names)) - - -def _ParamExists(param_obj, param_name): - """Tests whether param_name is contained in param_obj.""" - if not param_obj: - return - for k, _ in param_obj.IterParams(): - if k == param_name: - return True - return False - - -def _FreezeGraphFromCheckpoint(graph, saver, checkpoint, output_op_names): - """Freezes a graph from a checkpoint. - - Args: - graph: tf.Graph. - saver: The tf.Saver to use for restoration. - checkpoint: The checkpoint to restore. - output_op_names: Names of output ops. - - Returns: - Resulting tf.GraphDef. - """ - sess = tf.Session(graph=graph, config=py_utils.SessionConfig()) - saver.restore(sess, checkpoint) - return tf.graph_util.convert_variables_to_constants( - sess, graph.as_graph_def(), output_op_names) - - -def _FreezeDefaults(graph, output_op_names): - """Default initializes a graph and freezes it. - - Args: - graph: tf.Graph. - output_op_names: Names of output ops. - - Returns: - Resulting tf.GraphDef. - """ - with tf.Session(graph=graph, config=py_utils.SessionConfig()) as sess: - sess.run(graph.get_operation_by_name('init_all_variables')) - return tf.graph_util.convert_variables_to_constants(sess, - graph.as_graph_def(), - output_op_names) - - -class InferenceGraphExporter: - """Class for exporting inference graphs.""" - - @classmethod - def Export(cls, - model_cfg, - model_task_name=None, - device_options=InferenceDeviceOptions( - device='', - retain_device_placement=False, - var_options=None, - gen_init_op=True, - dtype_override=None), - freeze_checkpoint=None, - freeze_defaults=False, - export_path=None, - subgraph_filter=None, - random_seed=None, - disable_packed_input=True): - """Exports a InferenceGraph proto with piecewise subgraphs. - - Sets FLAGS.enable_asserts to False unless user explicitly sets it to True. - - Args: - model_cfg: a Params instance as returned by - model_registry.GetParams(modelname, 'Test') or model_params.Model(). - model_task_name: The task to generate an inference graph for. Should be - None for single-task models. - device_options: Device options for the accelerator used for serving. - freeze_checkpoint: The checkpoint to load. Loads and freezes the model if - given. - freeze_defaults: Default initializes the graph and freeze. Useful for - early testing of downstream tools without having a checkpoint. - export_path: If not None, write the inference graph in ASCII to this path. - subgraph_filter: A list of subgraph names. If not None or empty, export - only this list of inference subgraphs. - random_seed: Fixes the random seed in the exported inference graph. - disable_packed_input: Disable packed input for inference writing purposes. - - Returns: - InferenceGraph proto. - - Raises: - ValueError: if the model does not support the listed subgraphs. - """ - assert issubclass(model_cfg.cls, base_model.BaseModel) - - # Disable assertions unless user explicitly enables it. - if FLAGS['enable_asserts'].using_default_value: - FLAGS.enable_asserts = False - - # TODO(laurenzo): Work out how much we need to specify here in terms of - # cluster configuration. - cls._SetClusterParams(model_cfg.cluster, device_options) - - # Configure the model. - model_cfg.random_seed = random_seed - model_cfg.is_inference = True - - if disable_packed_input: - - def _DisablePackedInput(task): - if (_ParamExists(task, 'encoder') and - _ParamExists(task.encoder, 'packed_input')): - task.encoder.packed_input = False - if (_ParamExists(task, 'decoder') and - _ParamExists(task.decoder, 'packed_input')): - task.decoder.packed_input = False - - if issubclass(model_cfg.cls, base_model.MultiTaskModel): - for _, task_param in model_cfg.task_params.IterParams(): - _DisablePackedInput(task_param) - else: - _DisablePackedInput(model_cfg.task) - - tf.logging.info('Model %s params:', model_cfg.name) - for line in model_cfg.ToText().split('\n'): - tf.logging.info('%s', line) - - # Instantiate the graph. - graph = tf.Graph() - with graph.as_default(): - tf.random.set_seed(random_seed) - cluster = model_cfg.cluster.Instantiate() - device = cluster.GetPlacer() - tpu_const_scope = _DummyScope() - if (IsTpu(device_options) and - device_options.var_options == 'AS_CONSTANTS'): - # Do not specify devices for variables if we are marking them as - # constants. - device = '' - tpu_const_scope = ConstGuaranteeScope() - - with cluster, tf.device(device), tpu_const_scope: - - bfloat16_override = ShouldForceBfloat16ForWeightsAndActivations( - device_options) - - if bfloat16_override: - py_utils.UpdateDtype(model_cfg, tf.bfloat16) - py_utils.UpdateFpropDtype(model_cfg, tf.bfloat16) - - # Hard-code TPU-related flags prior to instantiating model. - old_enable_asserts = FLAGS.enable_asserts - old_xla_device = FLAGS.xla_device - if IsTpu(device_options): - FLAGS.enable_asserts = False - FLAGS.xla_device = 'tpu' - - try: - mdl = model_cfg.Instantiate() - task = mdl.GetTask(model_task_name) - - variables_to_restore = ( - _MakeVariableDictionary(tf.global_variables()) if not mdl.ema else - mdl.ema.variables_to_restore(mdl.variables_for_ema)) - - if bfloat16_override: - saver_var_spec = ( - bfloat16_variables - .get_saver_spec_for_variables_with_bf16_overrides( - variables_to_restore)) - else: - saver_var_spec = variables_to_restore - - saver = tf.train.Saver(saver_var_spec) - tf.variables_initializer( - tf.global_variables(), name='init_all_variables') - if IsTpu(device_options) and device_options.gen_init_op: - tf.group(tf.tpu.initialize_system(), name='tpu_init_op') - - inference_graph_proto = inference_graph_pb2.InferenceGraph() - subgraphs_proto = task.Inference() - if isinstance(subgraphs_proto, dict): - subgraphs_proto = ConvertSubgraphDictToProto(subgraphs_proto) - for name, subgraph in subgraphs_proto.subgraphs.items(): - if not subgraph_filter or name in subgraph_filter: - inference_graph_proto.subgraphs[name].CopyFrom(subgraph) - - # Yes, graph collections are bad, however this seems to be the - # easiest way to get this assets registered from - # TextFileInitializer. - assets_collection = tf.compat.v1.get_collection( - tf.compat.v1.GraphKeys.ASSET_FILEPATHS) - for asset in assets_collection: - if asset.op.type == 'Const' and asset.op.get_attr( - 'dtype') == tf.dtypes.string: - constant_value = asset.op.get_attr('value') - if constant_value.string_val: - tf.logging.info('Found asset file_path: %s', - constant_value.string_val[0]) - asset_file_def = inference_graph_proto.asset_file_def.add() - asset_file_def.tensor_info.name = asset.name - asset_file_def.filename = constant_value.string_val[0] - - # Add a table init op and global variable init op to the graph. - # Tables can be declared anywhere in the graph, so this op has to be - # added last. - tf.tables_initializer(name='init_all_tables') - finally: - # Reset TPU-related flags after model instantiation. - FLAGS.enable_asserts = old_enable_asserts - FLAGS.xla_device = old_xla_device - - tf.logging.info('Graph contains ops: %r', - [op.name for op in graph.get_operations()]) - - inference_graph_proto.saver_def.CopyFrom(saver.as_saver_def()) - - # Freezing. - if freeze_defaults or freeze_checkpoint: - output_op_names = GetOutputOpNames( - graph, inference_graph_proto, preserve_colocation_nodes=False) - if cls._DeviceSupportsFreezing(device_options): - raise ValueError('freeze_checkpoint cannot be used with device ' + - device_options.device) - if freeze_checkpoint: - tf.logging.info('Freezing graph from checkpoint: %s', - freeze_checkpoint) - graph_def = _FreezeGraphFromCheckpoint(graph, saver, freeze_checkpoint, - output_op_names) - elif freeze_defaults: - tf.logging.info('Default initializing graph and freezing.') - graph_def = _FreezeDefaults(graph, output_op_names) - else: - output_op_names = GetOutputOpNames(graph, inference_graph_proto) - - # Prune the graph to just the parts we need. - # To support restoring, we have to not prune out the restore node. - output_op_names.append('init_all_tables') - output_op_names.append('init_all_variables') - output_op_names.append('save/control_dependency') - output_op_names.append('save/restore_all') - if IsTpu(device_options) and device_options.gen_init_op: - output_op_names.append('tpu_init_op') - graph_def = graph.as_graph_def() - tf.logging.info('Pruning graph to output ops: %r', output_op_names) - graph_def = tf.graph_util.extract_sub_graph(graph_def, output_op_names) - - if not device_options.retain_device_placement: - # Clear the device so that the runtime can choose. - tf.logging.info('Clearing device placement for: %s', - device_options.device) - for node in graph_def.node: - node.ClearField('device') - for function in graph_def.library.function: - for node_def in function.node_def: - node_def.ClearField('device') - - inference_graph_proto.graph_def.CopyFrom(graph_def) - - if export_path: - with tf.io.gfile.GFile(export_path, 'w') as f: - f.write(text_format.MessageToString(inference_graph_proto)) - return inference_graph_proto - - @classmethod - def _SetClusterParams(cls, cluster_params, device_options): - """Sets cluster params. - - Args: - cluster_params: Model().cluster config. - device_options: InferenceDeviceOptions. - """ - - def Update(p): - """Update cluster params `p`.""" - p.name = '/job:localhost' - p.replicas = 1 - p.tpus_per_replica = 1 if IsTpu(device_options) else 0 - p.gpus_per_replica = 0 - p.devices_per_split = 1 - - cluster_params.mode = 'sync' - cluster_params.job = 'decoder' - cluster_params.add_summary = False - cluster_params.do_eval = True - Update(cluster_params.controller) - Update(cluster_params.worker) - Update(cluster_params.ps) - Update(cluster_params.evaler) - Update(cluster_params.decoder) - Update(cluster_params.input) - - @classmethod - def _DeviceSupportsFreezing(cls, device_options): - return IsTpu(device_options) diff --git a/lingvo/core/inference_graph_exporter_test.py b/lingvo/core/inference_graph_exporter_test.py deleted file mode 100644 index 35dc2a81..00000000 --- a/lingvo/core/inference_graph_exporter_test.py +++ /dev/null @@ -1,453 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for inference_graph_exporter.""" - -from lingvo import model_registry -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import base_model -from lingvo.core import base_model_params -from lingvo.core import inference_graph_exporter -from lingvo.core import inference_graph_pb2 -from lingvo.core import predictor -from lingvo.core import py_utils -from lingvo.core import test_utils - - -class DummyLegacyModel(base_model.BaseTask): - - def Inference(self): - if py_utils.use_tpu(): - raise NotImplementedError('TPU is not supported.') - with tf.name_scope('inference'): - feed1 = tf.placeholder(name='feed1_node', dtype=tf.float32, shape=[1]) - fetch1 = tf.identity(feed1, name='fetch1_node') - return { - 'default': ( - py_utils.NestedMap({ - 'fetch1': fetch1, - 'fetch_op': fetch1.op, # Tests that ops are supported. - }), - py_utils.NestedMap({ - 'feed1': feed1, - })), - 'unused': (py_utils.NestedMap({}), py_utils.NestedMap({})), - } - - -@model_registry.RegisterSingleTaskModel -class DummyLegacyModelParams(base_model_params.SingleTaskModelParams): - - @classmethod - def Test(cls): - p = base_input_generator.BaseSequenceInputGenerator.Params() - p.name = 'input' - return p - - @classmethod - def Task(cls): - p = DummyLegacyModel.Params() - p.name = 'testing' - return p - - -class DummyModel(base_model.BaseTask): - - def Inference(self): - with tf.name_scope('inference'): - feed1 = tf.placeholder(name='feed1_node', dtype=tf.float32, shape=[1]) - fetch1 = tf.identity(feed1, name='fetch1_node') - inference_graph = inference_graph_pb2.InferenceGraph() - subgraph = inference_graph.subgraphs['default'] - subgraph.feeds['feed1'] = feed1.name - subgraph.fetches['fetch1'] = fetch1.name - # Tests that ops are supported. - subgraph.fetches['fetch_op'] = fetch1.op.name - return inference_graph - - -@model_registry.RegisterSingleTaskModel -class DummyModelParams(base_model_params.SingleTaskModelParams): - - @classmethod - def Test(cls): - p = base_input_generator.BaseSequenceInputGenerator.Params() - p.name = 'input' - return p - - @classmethod - def Task(cls): - p = DummyModel.Params() - p.name = 'testing' - return p - - -class InferenceGraphExporterTest(test_utils.TestCase): - - def testExportModelParamsWithSubgraphDict(self): - params = model_registry.GetParams('test.DummyLegacyModelParams', 'Test') - inference_graph = inference_graph_exporter.InferenceGraphExporter.Export( - params, subgraph_filter=['default']) - - # Should populate subgraphs. - self.assertIn('default', inference_graph.subgraphs) - self.assertNotIn('unused', inference_graph.subgraphs) - subgraph = inference_graph.subgraphs['default'] - self.assertIn('feed1', subgraph.feeds) - self.assertIn('fetch1', subgraph.fetches) - - self.assertEqual(subgraph.feeds['feed1'], 'inference/feed1_node:0') - self.assertEqual(subgraph.fetches['fetch1'], 'inference/fetch1_node:0') - self.assertEqual(subgraph.fetches['fetch_op'], 'inference/fetch1_node') - - def testExportModelParamsWithInferenceGraph(self): - params = model_registry.GetParams('test.DummyModelParams', 'Test') - inference_graph = inference_graph_exporter.InferenceGraphExporter.Export( - params) - - # Should populate subgraphs. - self.assertIn('default', inference_graph.subgraphs) - subgraph = inference_graph.subgraphs['default'] - self.assertIn('feed1', subgraph.feeds) - self.assertIn('fetch1', subgraph.fetches) - - self.assertEqual(subgraph.feeds['feed1'], 'inference/feed1_node:0') - self.assertEqual(subgraph.fetches['fetch1'], 'inference/fetch1_node:0') - self.assertEqual(subgraph.fetches['fetch_op'], 'inference/fetch1_node') - - def testExportModelDoesNotAffectFlagsOnException(self): - initial_flags = {k: tf.flags.FLAGS[k].value for k in tf.flags.FLAGS} - params = model_registry.GetParams('test.DummyLegacyModelParams', 'Test') - with self.assertRaises(NotImplementedError): - inference_graph_exporter.InferenceGraphExporter.Export( - params, - device_options=inference_graph_exporter.InferenceDeviceOptions( - device='tpu', - retain_device_placement=False, - var_options=None, - gen_init_op=True, - dtype_override=None)) - self.assertDictEqual(initial_flags, - {k: tf.flags.FLAGS[k].value for k in tf.flags.FLAGS}) - - -class NoConstGuaranteeScopeTest(test_utils.TestCase): - - def testNoConsting(self): - with inference_graph_exporter.ConstGuaranteeScope(): - wp = py_utils.WeightParams( - shape=[1], - init=py_utils.WeightInit.Constant(0.0), - dtype=tf.float32, - collections=['v']) - v = py_utils.CreateVariable('v', wp) - self.assertEqual(tf.Tensor, type(v)) - with inference_graph_exporter.NoConstGuaranteeScope(): - v = py_utils.CreateVariable('v', wp, reuse=True) - self.assertIsInstance(v, tf.Variable) - - -class LinearModel(base_model.BaseTask): - """A basic linear model.""" - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'linear_model' - return p - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - w = py_utils.WeightParams( - shape=[3], - init=py_utils.WeightInit.Gaussian(scale=1.0, seed=123456), - dtype=p.dtype) - b = py_utils.WeightParams( - shape=[], - init=py_utils.WeightInit.Gaussian(scale=1.0, seed=234567), - dtype=p.dtype) - self.CreateVariable('w', w) - self.CreateVariable('b', b) - - def Inference(self): - """Computes y = w^T x + b. Returns y and x, as outputs and inputs.""" - # Add a dummy file def to the collection - filename = tf.convert_to_tensor( - 'dummy.txt', tf.dtypes.string, name='asset_filepath') - tf.compat.v1.add_to_collection(tf.compat.v1.GraphKeys.ASSET_FILEPATHS, - filename) - - with tf.name_scope('inference'): - x = tf.placeholder(dtype=tf.float32, name='input') - r = tf.random.stateless_uniform([3], - seed=py_utils.GenerateStepSeedPair( - self.params, self.theta.global_step)) - y = tf.reduce_sum((self.vars.w + r) * x) + self.vars.b - return {'default': ({'output': y}, {'input': x})} - - -class LinearModelTpu(LinearModel): - """A basic linear model that runs inference on the TPU.""" - - def Inference(self): - """Computes y = w^T x + b. Returns y and x, as outputs and inputs.""" - with tf.name_scope('inference'): - x = tf.placeholder(dtype=tf.bfloat16, name='input') - - def InferenceFn(x): - return tf.reduce_sum(self.vars.w * x) + self.vars.b - - y = tf.tpu.rewrite(InferenceFn, [x]) - return {'tpu': ({'output': y[0]}, {'input': x})} - - -@model_registry.RegisterSingleTaskModel -class LinearModelParams(base_model_params.SingleTaskModelParams): - - @classmethod - def Test(cls): - p = base_input_generator.BaseSequenceInputGenerator.Params() - p.name = 'input' - return p - - @classmethod - def Task(cls): - p = LinearModel.Params() - p.name = 'testing' - return p - - -@model_registry.RegisterSingleTaskModel -class LinearModelTpuParams(base_model_params.SingleTaskModelParams): - - @classmethod - def Test(cls): - p = base_input_generator.BaseSequenceInputGenerator.Params() - p.name = 'input' - return p - - @classmethod - def Task(cls): - p = LinearModelTpu.Params() - p.name = 'testing' - return p - - -@model_registry.RegisterSingleTaskModel -class LinearModelTpuParamsWithEma(base_model_params.SingleTaskModelParams): - - @classmethod - def Test(cls): - p = base_input_generator.BaseSequenceInputGenerator.Params() - p.name = 'input' - return p - - @classmethod - def Task(cls): - p = LinearModelTpu.Params() - p.name = 'testing' - p.train.ema_decay = 0.99 - return p - - -class InferenceGraphExporterLinearModelTest(test_utils.TestCase): - - def testExport(self): - """Test basic export.""" - params = model_registry.GetParams('test.LinearModelParams', 'Test') - inference_graph = inference_graph_exporter.InferenceGraphExporter.Export( - params, subgraph_filter=['default']) - self.assertIn('default', inference_graph.subgraphs) - self.assertEqual(1, len(inference_graph.asset_file_def)) - - def testExportFreezeDefault(self): - """Test exporting frozen graph.""" - params = model_registry.GetParams('test.LinearModelParams', 'Test') - inference_graph = inference_graph_exporter.InferenceGraphExporter.Export( - params, freeze_defaults=True, subgraph_filter=['default']) - self.assertIn('default', inference_graph.subgraphs) - # Test graphs are well-formed and importable. - with tf.Graph().as_default(): - tf.import_graph_def(inference_graph.graph_def) - - def testTpuBfloat16OverrideExport(self): - """Test that we can export with tf.bfloat16 dtype.""" - params = model_registry.GetParams('test.LinearModelTpuParams', 'Test') - inference_graph = inference_graph_exporter.InferenceGraphExporter.Export( - params, - subgraph_filter=['tpu'], - device_options=inference_graph_exporter.InferenceDeviceOptions( - device='tpu', - retain_device_placement=True, - var_options='ON_DEVICE', - gen_init_op=True, - dtype_override=tf.bfloat16)) - self.assertIn('tpu', inference_graph.subgraphs) - - def testTpuBfloat16OverrideExportWithEma(self): - """Test that we can export with tf.bfloat16 dtype.""" - params = model_registry.GetParams('test.LinearModelTpuParamsWithEma', - 'Test') - inference_graph = inference_graph_exporter.InferenceGraphExporter.Export( - params, - subgraph_filter=['tpu'], - device_options=inference_graph_exporter.InferenceDeviceOptions( - device='tpu', - retain_device_placement=True, - var_options='ON_DEVICE', - gen_init_op=True, - dtype_override=tf.bfloat16)) - self.assertIn('tpu', inference_graph.subgraphs) - - def testExportWithRandomSeeds(self): - """Test the effect of setting random seeds on export.""" - params = model_registry.GetParams('test.LinearModelParams', 'Test') - # Default -- use random_seed = None. - inference_graph = inference_graph_exporter.InferenceGraphExporter.Export( - params, subgraph_filter=['default']) - pred = predictor.Predictor(inference_graph) - [no_op_seed_1] = pred.Run(['output'], input=3) - [no_op_seed_2] = pred.Run(['output'], input=3) - self.assertNotEqual(no_op_seed_1, no_op_seed_2) - pred = predictor.Predictor(inference_graph) - [no_op_seed_3] = pred.Run(['output'], input=3) - self.assertNotEqual(no_op_seed_1, no_op_seed_3) - - # Use a fixed random_seed. - inference_graph = inference_graph_exporter.InferenceGraphExporter.Export( - params, subgraph_filter=['default'], random_seed=1234) - pred = predictor.Predictor(inference_graph) - [fixed_op_seed_1] = pred.Run(['output'], input=3) - [fixed_op_seed_2] = pred.Run(['output'], input=3) - self.assertEqual(fixed_op_seed_1, fixed_op_seed_2) - pred = predictor.Predictor(inference_graph) - [fixed_op_seed_3] = pred.Run(['output'], input=3) - self.assertEqual(fixed_op_seed_1, fixed_op_seed_3) - - # A different seed gives different results. - inference_graph = inference_graph_exporter.InferenceGraphExporter.Export( - params, subgraph_filter=['default'], random_seed=1235) - pred = predictor.Predictor(inference_graph) - [fixed_op_seed_4] = pred.Run(['output'], input=3) - self.assertNotEqual(fixed_op_seed_1, fixed_op_seed_4) - - -class GetOutputNamesTest(test_utils.TestCase): - - def _TestGraph(self): - params = model_registry.GetParams('test.LinearModelParams', 'Test') - inference_graph = inference_graph_exporter.InferenceGraphExporter.Export( - params) - graph = tf.Graph() - with graph.as_default(): - tf.import_graph_def(inference_graph.graph_def, name='') - return graph, inference_graph - - def testDefault(self): - graph, inference_graph = self._TestGraph() - output_op_names = inference_graph_exporter.GetOutputOpNames( - graph, inference_graph) - self.assertEqual(output_op_names, [ - # pyformat: disable - 'inference/add_2', - 'inference/input', - 'testing/b/var', - 'testing/b/var/Initializer/random_normal', - 'testing/b/var/Initializer/random_normal/RandomStandardNormal', - 'testing/b/var/Initializer/random_normal/mean', - 'testing/b/var/Initializer/random_normal/mul', - 'testing/b/var/Initializer/random_normal/shape', - 'testing/b/var/Initializer/random_normal/stddev', - 'testing/w/var', - 'testing/w/var/Initializer/random_normal', - 'testing/w/var/Initializer/random_normal/RandomStandardNormal', - 'testing/w/var/Initializer/random_normal/mean', - 'testing/w/var/Initializer/random_normal/mul', - 'testing/w/var/Initializer/random_normal/shape', - 'testing/w/var/Initializer/random_normal/stddev', - # pyformat: enable - ]) - - def testNoPreserveColocationNodes(self): - graph, inference_graph = self._TestGraph() - output_op_names = inference_graph_exporter.GetOutputOpNames( - graph, inference_graph, preserve_colocation_nodes=False) - self.assertEqual(output_op_names, [ - # pyformat: disable - 'inference/add_2', - 'inference/input', - # pyformat: enable - ]) - - def testPreserveSaverRestoreNodes(self): - graph, inference_graph = self._TestGraph() - output_op_names = inference_graph_exporter.GetOutputOpNames( - graph, - inference_graph, - preserve_colocation_nodes=False, - preserve_saver_restore_nodes=True) - self.assertEqual(output_op_names, [ - # pyformat: disable - 'inference/add_2', - 'inference/input', - 'save/Const', - 'save/restore_all', - # pyformat: enable - ]) - - def testPreserveExtraOps(self): - graph, inference_graph = self._TestGraph() - output_op_names = inference_graph_exporter.GetOutputOpNames( - graph, - inference_graph, - preserve_colocation_nodes=False, - preserve_extra_ops=[ - 'init_all_tables', 'init_all_variables', 'tpu_init_op' - ]) - self.assertEqual(output_op_names, [ - # pyformat: disable - 'inference/add_2', - 'inference/input', - 'init_all_tables', - 'init_all_variables', - # pyformat: enable - ]) - - def testPreserveSaverNodesAndExtraOps(self): - graph, inference_graph = self._TestGraph() - output_op_names = inference_graph_exporter.GetOutputOpNames( - graph, - inference_graph, - preserve_colocation_nodes=False, - preserve_saver_restore_nodes=True, - preserve_extra_ops=[ - 'init_all_tables', 'init_all_variables', 'tpu_init_op' - ]) - self.assertEqual(output_op_names, [ - # pyformat: disable - 'inference/add_2', - 'inference/input', - 'init_all_tables', - 'init_all_variables', - 'save/Const', - 'save/restore_all', - # pyformat: enable - ]) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/input_generator_helper.py b/lingvo/core/input_generator_helper.py deleted file mode 100644 index 253232fa..00000000 --- a/lingvo/core/input_generator_helper.py +++ /dev/null @@ -1,122 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Helper functions for model.""" - -import lingvo.compat as tf -from lingvo.core import py_utils - - -def ComputeSplits(batch_size, num_splits): - """Creates a tensor of size num_splits of number of values per split. - - Assigns each split floor(batch_size/num_splits) and round-robins - the remainder (if any) to each split. - - Example:: - - batch_size: [5] - num_splits: 3 - returns: [2, 2, 1] - - Args: - batch_size: tensor of rank 0, size of tensor to be split - num_splits: number of splits to split tensor into - Returns: - tensor of length num_splits containing sizes of each split - """ - values = tf.tile( - tf.div([batch_size], num_splits), - tf.constant( - [num_splits], dtype=tf.int32)) - mods = tf.tile(tf.constant([1]), tf.math.floormod([batch_size], num_splits)) - zeros = tf.tile(tf.constant([0]), - tf.subtract(tf.shape(values), tf.shape(mods))) - mods = tf.concat([mods, zeros], 0) - ret = tf.add(values, mods) - # for some reason TF erases shape information if num_splits is 1 - if num_splits == 1: - ret.set_shape([1]) - return ret - - -def SplitTensors(xs, num_splits): - """Splits tensors in `xs` evenly into num_splits along the 1st dimenion. - - Args: - xs: A tuple of tensors. Each tensor's 1st dimension is the same size. - num_splits: A python integer. - - Returns: - A tuple of lists of tensors, num elements in the tuple = len(xs). - - i-th element in each list corresponds to i-th split of each tensor in xs - along the first dimension of each tensor. - """ - # assert first dim of all tensors in xs is equal - batch_dims = [tf.shape(x)[0] for x in xs] - all_batch_dims = tf.stack(batch_dims) - - all_batch_dims = py_utils.with_dependencies([ - py_utils.assert_equal( - all_batch_dims, - tf.shape(xs[0])[0], - message='first dim of tensors in xs must match'), - py_utils.assert_greater_equal( - tf.shape(xs[0])[0], - num_splits, - message='first dim of tensors in xs must be greater than num_splits') - ], all_batch_dims) - - splits = ComputeSplits(tf.shape(xs[0])[0], num_splits) - # add the above assertion into the compute graph - splits = py_utils.with_dependencies([all_batch_dims], splits) - split_xs = [tf.split(axis=0, num_or_size_splits=splits, value=x) for x in xs] - - return split_xs - - -def SplitDictOfTensors(t_dict, num_splits): - """Splits tensors in `t_dict` evenly into `num_splits` along the 1st dimenion. - - Args: - t_dict: A dictionary of tensors. Each tensor's 1st dimension is the same - size. - num_splits: A python integer. - - Returns: - A list of dictionaries of tensors, num elements in the list = num_splits - - i-th dictionary in the list corresponds to i-th split of each tensor - along the first dimension of each tensor for each key in the original dict. - """ - keys = [] - values = [] - for k, v in sorted(t_dict.items()): - keys.append(k) - values.append(v) - - splits = SplitTensors(tuple(values), num_splits) - - assert all(len(lst) == len(splits[0]) for lst in splits) - - ret_list = [] - for s in range(num_splits): - d = {} - for k in range(len(splits)): - d[keys[k]] = splits[k][s] - ret_list.append(d) - - return ret_list diff --git a/lingvo/core/input_generator_helper_test.py b/lingvo/core/input_generator_helper_test.py deleted file mode 100644 index 2c4d3412..00000000 --- a/lingvo/core/input_generator_helper_test.py +++ /dev/null @@ -1,226 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for input_generator_helper.""" - -import lingvo.compat as tf -from lingvo.core import input_generator_helper -from lingvo.core import test_utils -import numpy as np - - -class InputGeneratorHelperTest(test_utils.TestCase): - - def testComputeSplitsLessThanNumSplits(self): - with self.session(use_gpu=False): - batch_size = tf.constant(2, dtype=tf.int32) - num_splits = 4 - splits = input_generator_helper.ComputeSplits(batch_size, num_splits) - expected = [1, 1, 0, 0] - - actual = self.evaluate(splits) - self.assertAllEqual(actual, expected) - - def testComputeSplitsEven(self): - with self.session(use_gpu=False): - batch_size = tf.constant(32, dtype=tf.int32) - num_splits = 4 - splits = input_generator_helper.ComputeSplits(batch_size, num_splits) - expected = [8, 8, 8, 8] - - actual = self.evaluate(splits) - self.assertAllEqual(actual, expected) - - def testComputeSplitsUnevenOne(self): - with self.session(use_gpu=False): - batch_size = tf.constant(31, dtype=tf.int32) - num_splits = 4 - splits = input_generator_helper.ComputeSplits(batch_size, num_splits) - expected = [8, 8, 8, 7] - - actual = self.evaluate(splits) - self.assertAllEqual(actual, expected) - - def testComputeSplitsUnevenTwo(self): - with self.session(use_gpu=False): - batch_size = tf.constant(30, dtype=tf.int32) - num_splits = 4 - splits = input_generator_helper.ComputeSplits(batch_size, num_splits) - expected = [8, 8, 7, 7] - - actual = self.evaluate(splits) - self.assertAllEqual(actual, expected) - - def testComputeSplitsUnevenThree(self): - with self.session(use_gpu=False): - batch_size = tf.constant(29, dtype=tf.int32) - num_splits = 4 - splits = input_generator_helper.ComputeSplits(batch_size, num_splits) - expected = [8, 7, 7, 7] - - actual = self.evaluate(splits) - self.assertAllEqual(actual, expected) - - def _assertTupleOfListsEqual(self, actual, expected): - self.assertEqual(len(actual), len(expected)) - for i in range(len(actual)): - self.assertEqual(len(actual[i]), len(expected[i])) - for j in range(len(actual[i])): - self.assertAllEqual(actual[i][j], expected[i][j]) - - def _assertListOfDictsEqual(self, actual, expected): - self.assertEqual(len(actual), len(expected)) - for i in range(len(actual)): - self.assertSetEqual(set(actual[i].keys()), set(expected[i].keys())) - for k in actual[i].keys(): - self.assertAllEqual(actual[i][k], expected[i][k]) - - def testSplitTensorsLessThanNumSplits(self): - t1 = tf.constant([[1, 2, 3, 4]]) - t2 = tf.constant([[5, 6, 7, 8]]) - t3 = tf.constant([[13, 14, 15, 16]]) - - tensor_tuple = (t1, t2, t3) - num_splits = 2 - splits = input_generator_helper.SplitTensors(tensor_tuple, num_splits) - - with self.session(use_gpu=False): - with self.assertRaisesRegex( - tf.errors.InvalidArgumentError, - 'first dim of tensors in xs must be greater ' - 'than num_splits'): - self.evaluate(splits) - - def testSplitTensorsOne(self): - with self.session(use_gpu=False): - t1 = tf.constant([[1, 2, 3, 4], [4, 5, 6, 7]]) - t2 = tf.constant([[5, 6, 7, 8], [9, 10, 11, 12]]) - t3 = tf.constant([[13, 14, 15, 16], [14, 15, 16, 17]]) - - tensor_tuple = (t1, t2, t3) - num_splits = 1 - splits = input_generator_helper.SplitTensors(tensor_tuple, num_splits) - expected = ([np.array([[1, 2, 3, 4], [4, 5, 6, 7]])], - [np.array([[5, 6, 7, 8], [9, 10, 11, 12]])], - [np.array([[13, 14, 15, 16], [14, 15, 16, 17]])]) - - actual = self.evaluate(splits) - self._assertTupleOfListsEqual(actual, expected) - - def testSplitTensorsEven(self): - with self.session(use_gpu=False): - t1 = tf.constant([[1, 2, 3, 4], [4, 5, 6, 7]]) - t2 = tf.constant([[5, 6, 7, 8], [9, 10, 11, 12]]) - t3 = tf.constant([[13, 14, 15, 16], [14, 15, 16, 17]]) - - tensor_tuple = (t1, t2, t3) - num_splits = 2 - splits = input_generator_helper.SplitTensors(tensor_tuple, num_splits) - expected = ([np.array([[1, 2, 3, 4]]), np.array([[4, 5, 6, 7]])], - [np.array([[5, 6, 7, 8]]), np.array([[9, 10, 11, 12]])], - [np.array([[13, 14, 15, 16]]), np.array([[14, 15, 16, 17]])]) - - actual = self.evaluate(splits) - self._assertTupleOfListsEqual(actual, expected) - - def testSplitTensorsUneven(self): - with self.session(use_gpu=False): - t1 = tf.constant([[1], [4], [8]]) - t2 = tf.constant([[5], [9], [10]]) - t3 = tf.constant([[13], [14], [11]]) - - tensor_tuple = (t1, t2, t3) - num_splits = 2 - splits = input_generator_helper.SplitTensors(tensor_tuple, num_splits) - expected = ([np.array([[1], [4]]), np.array([[8]])], - [np.array([[5], [9]]), np.array([[10]])], - [np.array([[13], [14]]), np.array([[11]])]) - - actual = self.evaluate(splits) - self._assertTupleOfListsEqual(actual, expected) - - def testSplitTensorsAssert(self): - t1 = tf.constant([[1], [7], [8]]) - t2 = tf.constant([[5], [9], [10]]) - t3 = tf.constant([[13], [14]]) - - tensor_tuple = (t1, t2, t3) - num_splits = 2 - - with self.assertRaisesRegex( - ValueError, r'can\'t split axis of size 2 into pieces of size \[2,1\]'): - _ = input_generator_helper.SplitTensors(tensor_tuple, num_splits) - - def testSplitDictOfTensorsEven(self): - with self.session(use_gpu=False): - t1 = tf.constant([[1], [4], [8], [9]]) - t2 = tf.constant([[5], [9], [10], [12]]) - t3 = tf.constant([[13], [14], [11], [15]]) - - tensor_dict = {'a': t1, 'b': t2, 'c': t3} - num_splits = 2 - splits = input_generator_helper.SplitDictOfTensors(tensor_dict, - num_splits) - expected = [{ - 'a': np.array([[1], [4]]), - 'b': np.array([[5], [9]]), - 'c': np.array([[13], [14]]) - }, { - 'a': np.array([[8], [9]]), - 'b': np.array([[10], [12]]), - 'c': np.array([[11], [15]]) - }] - - actual = self.evaluate(splits) - self._assertListOfDictsEqual(actual, expected) - - def testSplitDictOfTensorsUneven(self): - with self.session(use_gpu=False): - t1 = tf.constant([[1], [4], [8]]) - t2 = tf.constant([[5], [9], [10]]) - t3 = tf.constant([[13], [14], [11]]) - - tensor_dict = {'a': t1, 'b': t2, 'c': t3} - num_splits = 2 - splits = input_generator_helper.SplitDictOfTensors(tensor_dict, - num_splits) - expected = [{ - 'a': np.array([[1], [4]]), - 'b': np.array([[5], [9]]), - 'c': np.array([[13], [14]]) - }, { - 'a': np.array([[8]]), - 'b': np.array([[10]]), - 'c': np.array([[11]]) - }] - - actual = self.evaluate(splits) - self._assertListOfDictsEqual(actual, expected) - - def testSplitDictOfTensorsAssert(self): - t1 = tf.constant([[1], [7], [8]]) - t2 = tf.constant([[5], [9], [10]]) - t3 = tf.constant([[13], [14]]) - - tensor_dict = {'a': t1, 'b': t2, 'c': t3} - num_splits = 2 - - with self.assertRaisesRegex( - ValueError, 'can\'t split axis of size 2 into pieces of size \[2,1\]'): - splits = input_generator_helper.SplitDictOfTensors(tensor_dict, - num_splits) - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/insertion.py b/lingvo/core/insertion.py deleted file mode 100644 index e029ec34..00000000 --- a/lingvo/core/insertion.py +++ /dev/null @@ -1,328 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Insertion-based Framework. - -References: - KERMIT: https://arxiv.org/pdf/1906.01604.pdf - Insertion Transformer: https://arxiv.org/pdf/1902.03249.pdf -""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils - - -def SequenceTrimLastToken(x, x_paddings): - """Trims the last token off of sequence `x`, and set trimmed elements to 0. - - Args: - x: A sequence of tokens of shape [batch_size, x_len_max]. - x_paddings: The paddings of `x`. - - Returns: - A tuple. - - The new sequence, Tensor of shape [batch_size, x_len_max]. - - The new paddings, Tensor of shape [batch_size, x_len_max]. - """ - x_len = tf.reduce_sum(1 - x_paddings, 1) - x_len_max = py_utils.GetShape(x)[1] - x_trimmed_len = tf.maximum(x_len - 1, 0) - x_trimmed_paddings = tf.sequence_mask(x_trimmed_len, x_len_max, - x_paddings.dtype) - x_trimmed = x * tf.cast(x_trimmed_paddings, x.dtype) - return x_trimmed, 1 - x_trimmed_paddings - - -def SequenceAppendToken(x, x_paddings, token, extend=False): - """Appends to sequence `x`. - - Args: - x: A sequence of tokens of shape [batch_size, x_len_max]. - x_paddings: The paddings of `x`. - token: The token to append (of type integer). - extend: Whether to extend `x` along the length dimension, this must be true - for any sequence length in `x` that is `x_len_max` or else an invalid - sequence will be emitted. - - Returns: - A tuple. - - The new sequence, Tensor of shape [batch_size, x_len_max]. - - The new paddings, Tensor of shape [batch_size, x_len_max]. - """ - batch_size = py_utils.GetShape(x)[0] - x_len = tf.cast(tf.round(tf.reduce_sum(1 - x_paddings, 1)), tf.int32) - if extend: - x = tf.pad(x, [[0, 0], [0, 1]]) - # Mask all invalid entries of `x` to 0. - x *= tf.sequence_mask(x_len, py_utils.GetShape(x)[1], x.dtype) - # Append the based on `x_len`. - x += tf.scatter_nd( - tf.stack([tf.range(batch_size), x_len], axis=1), - tf.cast(tf.fill([batch_size], token), x.dtype), py_utils.GetShape(x)) - x_paddings = 1 - tf.sequence_mask(x_len + 1, - py_utils.GetShape(x)[1], x_paddings.dtype) - return x, x_paddings - - -def SequenceConcat(x, x_paddings, y, y_paddings, pad=0): - """Concats sequence `x` with sequence `y`. - - This function is length aware (based off the paddings). - - Args: - x: A sequence of tokens of shape [batch_size, x_len_max]. - x_paddings: The paddings of `x`. - y: A sequence of tokens of shape [batch_size, y_len_max]. - y_paddings: The paddings of `y`. - pad: The token to fill the concatenated sequence (of type integer). - - Returns: - A tuple. - - Concatenation of `x` and `y` of shape - [batch_size, x_len_max + y_len_max]. - - Paddings of the concatenation of shape - [batch_size, x_len_max + y_len_max]. - """ - # Get the length (w/ eos). - x_len = tf.cast(tf.round(tf.reduce_sum(1 - x_paddings, 1)), tf.int32) - y_len = tf.cast(tf.round(tf.reduce_sum(1 - y_paddings, 1)), tf.int32) - - batch_size = py_utils.GetShape(x)[0] - y_len_max = py_utils.GetShape(y)[1] - - # Pad `x` with necessary . - x = tf.concat([x, tf.fill(py_utils.GetShape(y), pad)], 1) - # Replace all with 0. - x = tf.where(tf.not_equal(x, pad), x, tf.fill(py_utils.GetShape(x), 0)) - - # Compute the write indices of `y` in `xy`. - indices = tf.stack([ - tf.tile(tf.expand_dims(tf.range(batch_size), 1), [1, y_len_max]), - (tf.tile(tf.expand_dims(tf.range(y_len_max), 0), [batch_size, 1]) + - tf.expand_dims(x_len, 1)), - ], 2) - - xy = x + tf.scatter_nd(indices, y, py_utils.GetShape(x)) - - # We need to remap all to `pad`. - xy = tf.where( - tf.less( - tf.expand_dims(tf.range(py_utils.GetShape(xy)[1]), 0), - tf.expand_dims(x_len + y_len, 1)), xy, - tf.fill(py_utils.GetShape(xy), pad)) - xy_paddings = 1 - tf.sequence_mask(x_len + y_len, - py_utils.GetShape(xy)[1], x_paddings.dtype) - return xy, xy_paddings - - -class SymbolInsertionLayer(base_layer.BaseLayer): - """Insertion-based framework for symbols. - - This constructs the sampled rollin (observed) canvas, as well as the targets - for an insertion-based model. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'rollin_policy', 'oracle', - 'Rollin policy, should be {oracle, uniform}. Rollin policy is the ' - 'sampling policy from which we draw the canvas. ' - '`oracle` means same as `oracle_policy`.') - p.Define( - 'oracle_policy', 'uniform', - 'Oracle policy, should be one of {uniform}. Oracle policy is the ' - 'target policy from which we select our targets and train our ' - 'models.') - return p - - def __init__(self, params): - super().__init__(params) - - def FProp(self, - theta, - x, - x_paddings=None, - eos_id=1, - force_sample_last_token=True): - """Applies SymbolInsertionLayer. - - We take in a `x`, which represents the groundtruth sequence (i.e., English - sequence). We return a sampled rollin (observed) canvas (i.e., random subset - of the English sequence), as well as the target (indices) for an - insertion-based model (i.e., the targets given the random observed subset). - - Args: - theta: Ignored, this can be None. - x: The symbol ids of shape `[batch_size, time_dim]`. - x_paddings: The paddings (1 or 0) of shape `[batch_size, time_dim]` where - 0 is valid and 1 is invalid. - eos_id: The token id to represent end-of-slot. - force_sample_last_token: Set True to force sample the last token of `x`. - - Returns: - A `NestedMap`. - - canvas: The canvas (based off of the `rollin_policy`) of shape - [batch_size, c_dim]. Note that, `c_dim` <= `time_dim` but need not be - equal. - - canvas_indices: The canvas indices (into `x`). - - canvas_paddings: The paddings of `canvas_indices`. - - target_indices: The target indices of shape [num_targets, 3]. - `num_targets` is the number of total targets in the entire batch. - [:, 0] captures the batch, [:, 1] captures the slot, and [:, 2] - captures the token. Each row [batch, slot, vocab] represents the - indices of the target -- i.e., the batch, slot and vocab combination - of the target. Typical usage of these indices is to tf.gather_nd - the log-probs (from the softmax layer). - - target_weights: The target weights. - - Raises: - ValueError: If invalid params. - """ - p = self.params - - batch_size = py_utils.GetShape(x)[0] - time_dim = py_utils.GetShape(x)[1] - - if x_paddings is None: - x_paddings = tf.zeros([batch_size, time_dim], tf.float32) - - oracle_policy = p.oracle_policy - rollin_policy = ( - oracle_policy if p.rollin_policy == 'oracle' else p.rollin_policy) - - if rollin_policy != 'uniform': - raise ValueError('Unknown or unsupported rollin policy: %s' % - rollin_policy) - if oracle_policy != 'uniform': - raise ValueError('Unknown or unsupported oracle policy: %s' % - oracle_policy) - - x_len = tf.cast(tf.round(tf.reduce_sum(1 - x_paddings, 1)), tf.int32) - - # Compute the desired length per example in the batch. - ratio = tf.random.uniform([batch_size], 0.0, 1.0, seed=p.random_seed) - if force_sample_last_token: - c_len = tf.minimum( - tf.cast(ratio * tf.cast(x_len, tf.float32), tf.int32), x_len - 1) + 1 - else: - c_len = tf.minimum( - tf.cast(ratio * tf.cast(x_len + 1, tf.float32), tf.int32), x_len) - # Compute the maximum length across the batch. - c_len_max = tf.reduce_max(c_len) - - # Grab subset of random valid indices per example. - z_logits = tf.cast( - tf.expand_dims(tf.range(time_dim), 0) >= tf.expand_dims(x_len, 1), - tf.float32) * -1e9 - if force_sample_last_token: - # Force sample the last token -- i.e., as indexed by `x_len - 1`. We can - # accomplish this by add +LARGE_NUMBER to the logits. - z_logits += tf.cast( - tf.equal( - tf.expand_dims(tf.range(time_dim), 0), tf.expand_dims( - x_len - 1, 1)), tf.float32) * 1e9 - # Gumbel-max trick to sample (we only sample valid positions per sample in - # the batch). - z = -tf.math.log(-tf.math.log( - tf.random.uniform([batch_size, time_dim], seed=p.random_seed))) - unused_c_values, c_indices = tf.nn.top_k(z_logits + z, time_dim) - - # Trim everything > c_len_max. - c_indices = c_indices[:, :c_len_max] - - # Invalidate any indices >= c_len, we use the last index as the default - # invalid index. - c_indices = tf.where( - tf.expand_dims(tf.range(c_len_max), 0) < tf.expand_dims(c_len, 1), - c_indices, tf.fill(py_utils.GetShape(c_indices), time_dim - 1)) - - # Materialize the canvas. - c_indices = tf.sort(c_indices) - c = tf.gather_nd( - x, - tf.stack([ - tf.reshape( - tf.tile( - tf.expand_dims(tf.range(batch_size), 1), [1, c_len_max]), - [-1]), - tf.reshape(c_indices, [-1]) - ], 1)) - c = tf.reshape(c, [batch_size, c_len_max]) - - # Compute the paddings. - c_paddings = 1 - tf.sequence_mask(c_len, c_len_max, dtype=x_paddings.dtype) - c *= tf.cast(1 - c_paddings, tf.int32) - - indices = tf.concat([ - tf.reshape( - tf.tile(tf.expand_dims(tf.range(batch_size), 1), [1, c_len_max]), - [batch_size * c_len_max, 1]), - tf.reshape(c_indices, [batch_size * c_len_max, 1]) - ], 1) - x_token_is_observed = tf.scatter_nd( - indices, tf.ones([batch_size * c_len_max], tf.int32), - py_utils.GetShape(x)) - # `x_segments` captures which slot each `x` belongs to (both observed and - # tokens that need to be observed). - x_segments = tf.cumsum(x_token_is_observed, 1, exclusive=True) - - x_token_is_observed = tf.cast(x_token_is_observed, tf.bool) - prev_x_token_is_observed = tf.pad( - x_token_is_observed[:, :-1], [[0, 0], [1, 0]], constant_values=True) - x_token_is_observed = tf.reshape(x_token_is_observed, [-1]) - prev_x_token_is_observed = tf.reshape(prev_x_token_is_observed, [-1]) - x_is_valid = tf.cast(1 - x_paddings, tf.bool) - x_is_valid = tf.reshape(x_is_valid, [-1]) - - # Remap all the observed to , note some of these need a zero weight - # (or else there would be and valid token in the same slot). - target_indices = tf.cast(tf.reshape(x, [-1, 1]), tf.int32) - target_indices = tf.where( - x_token_is_observed, tf.fill(py_utils.GetShape(target_indices), eos_id), - target_indices) - - # TODO(williamchan): We give uniform 1.0 weight, however, math suggests - # we may want to weigh this term by the original sequence length. - target_weights = tf.ones_like(target_indices, tf.float32) - - # We need to set all the weights for which actually have valid tokens - # in the slot to zero. - target_weights = tf.where(x_token_is_observed & ~prev_x_token_is_observed, - tf.zeros_like(target_weights), target_weights) - - # TODO(williamchan): Consider dropping the entries w/ weight zero. - - # Add the batch and slot indices. - target_indices = tf.concat([ - tf.reshape( - tf.tile(tf.expand_dims(tf.range(batch_size), 1), [1, time_dim]), - [batch_size * time_dim, 1]), - tf.reshape(x_segments, [-1, 1]), target_indices - ], 1) - - # Select only the valid indices. The selected valid ones include slots w/ - # . - target_indices = target_indices[x_is_valid] - target_weights = target_weights[x_is_valid] - - return py_utils.NestedMap( - canvas=c, - canvas_indices=c_indices, - canvas_paddings=c_paddings, - target_indices=target_indices, - target_weights=target_weights) diff --git a/lingvo/core/insertion_test.py b/lingvo/core/insertion_test.py deleted file mode 100644 index a2caedc8..00000000 --- a/lingvo/core/insertion_test.py +++ /dev/null @@ -1,346 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Insertion Framework tests.""" - -import lingvo.compat as tf -from lingvo.core import insertion -from lingvo.core import test_utils -import numpy as np - - -class SequenceTest(test_utils.TestCase): - - def testSequenceTrimLastToken(self): - x = np.asarray([[1, 2, 3, 4], [1, 2, 3, 4], [1, 2, 3, 4], [1, 2, 3, 4]], - np.int32) - x_paddings = np.asarray( - [[0, 0, 0, 0], [0, 0, 0, 1], [1, 1, 1, 1], [0, 1, 1, 1]], np.float32) - - with self.session(): - x_trimmed, x_trimmed_paddings = insertion.SequenceTrimLastToken( - tf.convert_to_tensor(x), tf.convert_to_tensor(x_paddings)) - - x_trimmed, x_trimmed_paddings = self.evaluate( - [x_trimmed, x_trimmed_paddings]) - - # `x_trimmed_gold` is the same as `x` w/ last token removed. - # `x_trimmed_paddings_gold` is the corresponding paddings. - x_trimmed_gold = np.asarray( - [[1, 2, 3, 0], [1, 2, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]], np.int32) - x_trimmed_paddings_gold = np.asarray( - [[0, 0, 0, 1], [0, 0, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1]], np.float32) - - self.assertAllEqual(x_trimmed, x_trimmed_gold) - self.assertAllEqual(x_trimmed_paddings, x_trimmed_paddings_gold) - - def testSequenceAppendToken(self): - x = np.asarray([[1, 2, 3, 0], [1, 2, 3, 4], [0, 0, 0, 0], [1, 0, 0, 0]], - np.int32) - x_paddings = np.asarray( - [[0, 0, 0, 1], [0, 0, 0, 1], [1, 1, 1, 1], [0, 1, 1, 1]], np.float32) - - with self.session(): - x_appended, x_appended_paddings = insertion.SequenceAppendToken( - tf.convert_to_tensor(x), tf.convert_to_tensor(x_paddings), 10) - - x_appended, x_appended_paddings = self.evaluate([ - tf.convert_to_tensor(x_appended), - tf.convert_to_tensor(x_appended_paddings) - ]) - - # `x_appended_gold` is the same as `x` w/ token `10` appended. - # `x_appended_paddings_gold` is the corresponding paddings. - x_appended_gold = np.asarray( - [[1, 2, 3, 10], [1, 2, 3, 10], [10, 0, 0, 0], [1, 10, 0, 0]], - np.int32) - x_appended_paddings_gold = np.asarray( - [[0, 0, 0, 0], [0, 0, 0, 0], [0, 1, 1, 1], [0, 0, 1, 1]], np.float32) - - self.assertAllEqual(x_appended, x_appended_gold) - self.assertAllEqual(x_appended_paddings, x_appended_paddings_gold) - - def testSequenceAppendTokenExtend(self): - x = np.asarray([[1, 2, 3, 0], [1, 2, 3, 4], [0, 0, 0, 0], [1, 0, 0, 0]], - np.int32) - x_paddings = np.asarray( - [[0, 0, 0, 1], [0, 0, 0, 0], [1, 1, 1, 1], [0, 1, 1, 1]], np.int32) - - with self.session(): - x_appended, x_appended_paddings = insertion.SequenceAppendToken( - tf.convert_to_tensor(x), tf.convert_to_tensor(x_paddings), 10, True) - - x_appended, x_appended_paddings = self.evaluate( - [x_appended, x_appended_paddings]) - - # `x_appended_gold` is the same as `x` w/ token `10` appended, we also - # test for the condition of extend=True which requires +1 dim in the - # time dimension. - # `x_appended_paddings_gold` is the corresponding paddings. - x_appended_gold = np.asarray([[1, 2, 3, 10, 0], [1, 2, 3, 4, 10], - [10, 0, 0, 0, 0], [1, 10, 0, 0, 0]], - np.int32) - x_appended_paddings_gold = np.asarray( - [[0, 0, 0, 0, 1], [0, 0, 0, 0, 0], [0, 1, 1, 1, 1], [0, 0, 1, 1, 1]], - np.int32) - - self.assertAllEqual(x_appended, x_appended_gold) - self.assertAllEqual(x_appended_paddings, x_appended_paddings_gold) - - def testSequenceConcat(self): - x = np.asarray([[1, 2, 3, 0], [1, 2, 3, 4], [0, 0, 0, 0], [1, 0, 0, 0]], - np.int32) - x_paddings = np.asarray( - [[0, 0, 0, 1], [0, 0, 0, 0], [1, 1, 1, 1], [0, 1, 1, 1]], np.float32) - - y = np.asarray( - [[10, 20, 30, 0], [10, 20, 30, 40], [0, 0, 0, 0], [10, 0, 0, 0]], - np.int32) - y_paddings = np.asarray( - [[0, 0, 0, 1], [0, 0, 0, 0], [1, 1, 1, 1], [0, 1, 1, 1]], np.float32) - - with self.session(): - xy, xy_paddings = insertion.SequenceConcat( - tf.convert_to_tensor(x), tf.convert_to_tensor(x_paddings), - tf.convert_to_tensor(y), tf.convert_to_tensor(y_paddings), 999) - - xy, xy_paddings = self.evaluate( - [tf.convert_to_tensor(xy), - tf.convert_to_tensor(xy_paddings)]) - - # `xy_gold` is `x` and `y` concatenated. - # `xy_paddings_gold` is the corresponding paddings. - xy_gold = np.asarray( - [[1, 2, 3, 10, 20, 30, 999, 999], [1, 2, 3, 4, 10, 20, 30, 40], - [999, 999, 999, 999, 999, 999, 999, 999], - [1, 10, 999, 999, 999, 999, 999, 999]], np.int32) - xy_paddings_gold = np.asarray( - [[0, 0, 0, 0, 0, 0, 1, 1], [0, 0, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1], [0, 0, 1, 1, 1, 1, 1, 1]], np.float32) - - self.assertAllEqual(xy, xy_gold) - self.assertAllEqual(xy_paddings, xy_paddings_gold) - - -class SymbolInsertionLayerTest(test_utils.TestCase): - - def testGetValidCanvasUnderUniformRollinPolicy(self): - with self.session(use_gpu=True): - params = insertion.SymbolInsertionLayer.Params() - params.name = 'insertion' - params.rollin_policy = 'oracle' - params.oracle_policy = 'uniform' - - insertion_layer = insertion.SymbolInsertionLayer(params) - - batch_size = 4 - time_dim = 10 - - inputs = tf.tile(tf.expand_dims(tf.range(time_dim), 0), [batch_size, 1]) - spec = insertion_layer.FProp(None, inputs, force_sample_last_token=False) - - canvas, canvas_indices, canvas_paddings = self.evaluate( - [spec.canvas, spec.canvas_indices, spec.canvas_paddings]) - - for b in range(batch_size): - length = np.sum(1 - canvas_paddings[b, :]).astype(np.int32) - self.assertAllEqual(canvas[b, :length], canvas_indices[b, :length]) - # Test the invalid slots. - self.assertAllEqual(canvas[b, length:], - [0] * (canvas.shape[1] - length)) - self.assertAllEqual(canvas_indices[b, length:], - [time_dim - 1] * (canvas.shape[1] - length)) - - def testMaxCanvasSizeUnderUniformRollinPolicy(self): - """Tests for valid canvas size.""" - with self.session(use_gpu=True): - params = insertion.SymbolInsertionLayer.Params() - params.name = 'insertion' - params.rollin_policy = 'oracle' - params.oracle_policy = 'uniform' - - insertion_layer = insertion.SymbolInsertionLayer(params) - - batch_size = 4 - time_dim = 10 - - inputs = tf.tile(tf.expand_dims(tf.range(time_dim), 0), [batch_size, 1]) - inputs_len = tf.random.uniform([batch_size], 0, time_dim, tf.int32) - paddings = 1 - tf.sequence_mask(inputs_len, time_dim, tf.int32) - spec = insertion_layer.FProp( - None, inputs, paddings, force_sample_last_token=False) - - canvas_with_max_length = False - for _ in range(1000): - canvas_max_len, canvas, canvas_paddings = self.evaluate( - [inputs_len, spec.canvas, spec.canvas_paddings]) - - for b in range(batch_size): - max_len = canvas_max_len[b] - length = np.sum(1 - canvas_paddings[b, :]).astype(np.int32) - canvas_with_max_length |= length == max_len - self.assertLessEqual(length, max_len) - # Invalid entries of canvas should be 0. - self.assertAllEqual(canvas[b, length:], - [0] * (canvas.shape[1] - length)) - - # With high probability, there should be at least one canvas that is - # of the same size as the maximum canvas size. - self.assertEqual(canvas_with_max_length, True) - - def testContiguousCanvasUnderUniformRollinPolicy(self): - """Tests for valid canvas size.""" - with self.session(use_gpu=True): - params = insertion.SymbolInsertionLayer.Params() - params.name = 'insertion' - params.rollin_policy = 'oracle' - params.oracle_policy = 'uniform' - - insertion_layer = insertion.SymbolInsertionLayer(params) - - batch_size = 4 - time_dim = 10 - - inputs = tf.tile( - tf.expand_dims(tf.range(time_dim), 0) + 100, [batch_size, 1]) - inputs_len = tf.random.uniform([batch_size], 0, time_dim, tf.int32) - paddings = 1 - tf.sequence_mask(inputs_len, time_dim, tf.int32) - spec = insertion_layer.FProp( - None, inputs, paddings, force_sample_last_token=False) - - for _ in range(1000): - canvas, canvas_paddings = self.evaluate( - [spec.canvas, spec.canvas_paddings]) - - for b in range(batch_size): - length = np.sum(1 - canvas_paddings[b, :]).astype(np.int32) - # Check for valid part of the canvas and padding. - for l in range(length): - self.assertEqual(canvas_paddings[b, l], 0) - self.assertNotEqual(canvas[b, l], 0) - # Check for invalid part of the canvas and padding. - for l in range(length, canvas.shape[1]): - self.assertEqual(canvas_paddings[b, l], 1) - self.assertEqual(canvas[b, l], 0) - - def testGetValidCanvasAndTargetsUnderUniformOraclePolicyWithoutForcedSample( - self): - """Tests for canvas+targets under uniform (rollin+oracle) policy.""" - with self.session(use_gpu=True): - params = insertion.SymbolInsertionLayer.Params() - params.name = 'insertion' - params.rollin_policy = 'oracle' - params.oracle_policy = 'uniform' - params.random_seed = 12345 - - insertion_layer = params.Instantiate() - - x = np.asarray( - [[10, 11, 12, 13, 14, 15, 16], [10, 11, 12, 13, 14, 15, 16], - [10, 0, 0, 0, 0, 0, 0], [10, 11, 12, 13, 14, 15, 0]], np.int32) - x_paddings = np.asarray([[0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], - [0, 1, 1, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 1]], - np.float32) - - spec = insertion_layer.FProp( - None, - tf.convert_to_tensor(x), - tf.convert_to_tensor(x_paddings), - force_sample_last_token=False) - - (canvas, canvas_paddings, target_indices, - target_weights) = self.evaluate([ - spec.canvas, spec.canvas_paddings, spec.target_indices, - spec.target_weights - ]) - - canvas_gold = np.asarray([[10, 12, 13, 15, 16], [14, 0, 0, 0, 0], - [10, 0, 0, 0, 0], [11, 12, 13, 0, 0]], np.int32) - canvas_paddings_gold = np.asarray( - [[0., 0., 0., 0., 0.], [0., 1., 1., 1., 1.], [0., 1., 1., 1., 1.], - [0., 0., 0., 1., 1.]], np.float32) - target_indices_gold = np.asarray( - [[0, 0, 1], [0, 1, 11], [0, 1, 1], [0, 2, 1], [0, 3, 14], [0, 3, 1], - [0, 4, 1], [1, 0, 10], [1, 0, 11], [1, 0, 12], [1, 0, 13], [1, 0, 1], - [1, 1, 15], [1, 1, 1], [2, 0, 1], [3, 0, 10], [3, 0, 1], [3, 1, 1], - [3, 2, 1], [3, 3, 14], [3, 3, 15]], np.int32) - target_weights_gold = np.asarray([1, 1, 0, 1, 1, 0, 1] + - [1, 1, 1, 1, 0, 1, 0] + [1] + - [1, 0, 1, 1, 1, 1], np.float32) - target_weights_gold = np.reshape(target_weights_gold, - [target_weights_gold.shape[0], 1]) - - self.assertAllEqual(canvas, canvas_gold) - self.assertAllEqual(canvas_paddings, canvas_paddings_gold) - self.assertAllEqual(target_indices, target_indices_gold) - self.assertAllEqual(target_weights, target_weights_gold) - - def testGetValidCanvasAndTargetsUnderUniformOraclePolicyForcedSample(self): - """Tests for canvas+targets under uniform (rollin+oracle) policy.""" - with self.session(use_gpu=True): - params = insertion.SymbolInsertionLayer.Params() - params.name = 'insertion' - params.rollin_policy = 'oracle' - params.oracle_policy = 'uniform' - params.random_seed = 12345 - - insertion_layer = insertion.SymbolInsertionLayer(params) - - x = np.asarray( - [[10, 11, 12, 13, 14, 15, 16, 1], [10, 11, 12, 13, 14, 15, 16, 1], - [10, 1, 0, 0, 0, 0, 0, 0], [10, 11, 12, 13, 14, 15, 1, 0]], np.int32) - x_paddings = np.asarray( - [[0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 1, 1, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 0, 1]], np.float32) - - spec = insertion_layer.FProp(None, tf.convert_to_tensor(x), - tf.convert_to_tensor(x_paddings)) - - (canvas, canvas_indices, canvas_paddings, target_indices, - target_weights) = self.evaluate([ - spec.canvas, spec.canvas_indices, spec.canvas_paddings, - spec.target_indices, spec.target_weights - ]) - - canvas_gold = np.asarray([[10, 12, 13, 15, 16, 1], [13, 1, 0, 0, 0, 0], - [10, 1, 0, 0, 0, 0], [10, 12, 14, 1, 0, 0]], - np.int32) - canvas_indices_gold = np.asarray([[0, 2, 3, 5, 6, 7], [3, 7, 7, 7, 7, 7], - [0, 1, 7, 7, 7, 7], [0, 2, 4, 6, 7, 7]], - np.int32) - canvas_paddings_gold = np.asarray( - [[0., 0., 0., 0., 0., 0.], [0., 0., 1., 1., 1., 1.], - [0., 0., 1., 1., 1., 1.], [0., 0., 0., 0., 1., 1.]], np.float32) - target_indices_gold = np.asarray( - [[0, 0, 1], [0, 1, 11], [0, 1, 1], [0, 2, 1], [0, 3, 14], [0, 3, 1], - [0, 4, 1], [0, 5, 1], [1, 0, 10], [1, 0, 11], [1, 0, 12], [1, 0, 1], - [1, 1, 14], [1, 1, 15], [1, 1, 16], [1, 1, 1], [2, 0, 1], [2, 1, 1], - [3, 0, 1], [3, 1, 11], [3, 1, 1], [3, 2, 13], [3, 2, 1], [3, 3, 15], - [3, 3, 1]], np.int32) - target_weights_gold = np.asarray([1, 1, 0, 1, 1, 0, 1, 1] + - [1, 1, 1, 0, 1, 1, 1, 0] + [1, 1] + - [1, 1, 0, 1, 0, 1, 0], np.float32) - target_weights_gold = np.reshape(target_weights_gold, - [target_weights_gold.shape[0], 1]) - - self.assertAllEqual(canvas, canvas_gold) - self.assertAllEqual(canvas_indices, canvas_indices_gold) - self.assertAllEqual(canvas_paddings, canvas_paddings_gold) - self.assertAllEqual(target_indices, target_indices_gold) - self.assertAllEqual(target_weights, target_weights_gold) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/inspect_utils.py b/lingvo/core/inspect_utils.py deleted file mode 100644 index 11d1da32..00000000 --- a/lingvo/core/inspect_utils.py +++ /dev/null @@ -1,174 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Utilities to bind function signatures to params.""" - -import inspect - -# Groups of parameter kinds. -DEFINABLE_PARAMETER_KINDS = (inspect.Parameter.POSITIONAL_OR_KEYWORD, - inspect.Parameter.KEYWORD_ONLY) -IGNORABLE_PARAMETER_KINDS = (inspect.Parameter.VAR_POSITIONAL, - inspect.Parameter.VAR_KEYWORD) - - -def _IsDefinableParameter(parameter): - """Checks if the parameter can be defined in `Params`. - - Args: - parameter: inspect.Parameter to be checked. - - Returns: - True if the `parameter`'s kind is either `POSITIONAL_OR_KEYWORD` or - `KEYWORD_ONLY` which are definable in `Params`, False if it is either - `VAR_POSITIONAL` or `VAR_KEYWORD` which are ignorable. - - Raises: - ValueError: The `parameter` has another kind which are possibly not - supported, e.g., `POSITIONAL_ONLY` parameters. - """ - if parameter.kind in DEFINABLE_PARAMETER_KINDS: - return True - elif parameter.kind in IGNORABLE_PARAMETER_KINDS: - return False - else: - raise ValueError('Unsupported parameter signature `%s` with kind `%s`.' % - (parameter.name, parameter.kind)) - - -def _ExtractParameters(func, ignore, bound): - """Extracts parameters of func which can be defined in Params. - - Args: - func: A callable to be analysed. - ignore: A collection of parameter names in `func` to be ignored. - bound: Whether the `func` is used as a bound function (an object method or a - class method) or not. If True, the first parameter of the `func` will be - ignored. - - Returns: - A generator of `inspect.Parameter` representing definable parameters. - """ - ignore = set(ignore if ignore is not None else ()) - - # Obtains parameter signatures. - parameters = tuple(inspect.signature(func).parameters.values()) - # Ignores the bound parameter: typically `self` or `cls`. - parameters = parameters[(1 if bound else 0):] - # Filters unnecessary parameters. - parameters = filter(_IsDefinableParameter, parameters) - parameters = (p for p in parameters if p.name not in ignore) - - return parameters - - -def DefineParams(func, params, ignore=None, bound=False): - """Defines params for each parameter of given callable. - - This allows you to define the parameters necessary to call a callable without - having to type the Define statements yourself. - Default values for the function parameters will be copied into the params - object as well. - - To use this function for analysing a class instantiation, users usually can - pass the class type as the `func`. If it does not work correctly, pass the - `__init__` method of the class with `bound=True` instead. - - Args: - func: A callable to be analysed. Parameters of this function will be defined - in `params`. This function expects that `func` maintains the explicit - signature of its parameters. Implicit parameters that are stored in - `*args` or `**kwargs` could not be analysed correctly. - params: A `Params` object to be updated. New parameters will be defined - here. - ignore: A collection of parameter names in `func` to be ignored from - defining corresponding entries in `params`. - bound: Whether `func` will be used as a bound function (an object method or - a class method) or not. If True, the first parameter of `func` (typically - `self` or `cls`) will be ignored. - """ - for p in _ExtractParameters(func, ignore, bound): - default = p.default - if default is inspect.Parameter.empty: - # TODO(oday): If Params supported required fields, change this behavior to - # set the "required" flag. - default = None - - params.Define(p.name, default, 'Function parameter.') - - -def _MakeArgs(func, params, **kwargs): - """Generates an argument list to call func. - - Args: - func: A callable to be called. - params: A Params object containing arguments for `func`. - **kwargs: Argument/value pairs that should override params. - - Returns: - A dict containing function parameters to be used as `**kwargs` of `func`. - """ - out_kwargs = {} - - # Here we set bound=False so the `func` is expected to be already a bound - # function. - for p in _ExtractParameters(func, ignore=None, bound=False): - key = p.name - - # We will collect only args defined in at least either `kwargs` or `params`. - # Args in `func`'s signature but in neither both will be skipped. - - if key in kwargs: - # Anything in kwargs overrides parameters. - out_kwargs[key] = kwargs[key] - elif key in params: - value = params.Get(key) - # If the value in params is the same as the function default, we do not - # set the arg so that we will let the function signature fill in this - # parameter by itself. - if value != p.default: - out_kwargs[key] = value - - return out_kwargs - - -def CallWithParams(func, params, **kwargs): - """Call a function or method with a Params object. - - Args: - func: A callable to be called. - params: A Params object containing parameters of `func`. - **kwargs: Argument/value pairs that should override `params`. - - Returns: - The return values from func. - """ - return func(**_MakeArgs(func, params, **kwargs)) - - -# TODO(oday): Remove this function and replace it with CallWithParams when the -# bug on the initializer of Keras layers has been resolved. -def ConstructWithParams(cls, params, **kwargs): - """Constructs a class object with a Params object. - - Args: - cls: A class type to be constructed. - params: A Params object containing parameters of `cls.__init__`. - **kwargs: Argument/value pairs that should override `params`. - - Returns: - The constructed object. - """ - return cls(**_MakeArgs(cls.__init__, params, **kwargs)) diff --git a/lingvo/core/inspect_utils_test.py b/lingvo/core/inspect_utils_test.py deleted file mode 100644 index 77a70541..00000000 --- a/lingvo/core/inspect_utils_test.py +++ /dev/null @@ -1,184 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for inspect_utils.""" - -import lingvo.compat as tf -from lingvo.core import hyperparams -from lingvo.core import inspect_utils -from lingvo.core import test_utils - - -class InspectUtilsTest(test_utils.TestCase): - - def testBareFunction(self): - - def my_function(a, b): - return a + 1, b + 2 - - params = hyperparams.Params() - inspect_utils.DefineParams(my_function, params) - self.assertIn('a', params) - self.assertIn('b', params) - self.assertIsNone(params.a) - self.assertIsNone(params.b) - - params.a = 5 - params.b = 6 - a1, b1 = inspect_utils.CallWithParams(my_function, params) - self.assertEqual(a1, 5 + 1) - self.assertEqual(b1, 6 + 2) - - def testFunctionWithDefaults(self): - - def my_function(a, b=3): - return a + 1, b + 2 - - params = hyperparams.Params() - inspect_utils.DefineParams(my_function, params) - self.assertIn('a', params) - self.assertIn('b', params) - self.assertIsNone(params.a) - self.assertEqual(params.b, 3) - - params.a = 6 - a1, b1 = inspect_utils.CallWithParams(my_function, params) - self.assertEqual(a1, 6 + 1) - self.assertEqual(b1, 3 + 2) - - def testFunctionWithIgnore(self): - - def my_function(a, b=3, c=4): - return a + 1, b + 2, c + 3 - - params = hyperparams.Params() - inspect_utils.DefineParams(my_function, params, ignore=['c']) - self.assertIn('a', params) - self.assertIn('b', params) - self.assertNotIn('c', params) - self.assertIsNone(params.a) - self.assertEqual(params.b, 3) - - params.a = 6 - a1, b1, c1 = inspect_utils.CallWithParams(my_function, params, c=9) - self.assertEqual(a1, 6 + 1) - self.assertEqual(b1, 3 + 2) - self.assertEqual(c1, 9 + 3) - - def testFunctionWithOverrides(self): - - def my_function(a, b=3): - return a + 1, b + 2 - - params = hyperparams.Params() - inspect_utils.DefineParams(my_function, params) - self.assertIn('a', params) - self.assertIn('b', params) - self.assertIsNone(params.a) - self.assertEqual(params.b, 3) - - params.a = 6 - a1, b1 = inspect_utils.CallWithParams(my_function, params, a=7) - self.assertEqual(a1, 7 + 1) - self.assertEqual(b1, 3 + 2) - - def testFunctionWithVarArgs(self): - - def my_function(a, *args, b=3, **kwargs): - del args - del kwargs - return a + 1, b + 2 - - params = hyperparams.Params() - inspect_utils.DefineParams(my_function, params) - self.assertIn('a', params) - self.assertNotIn('args', params) - self.assertIn('b', params) - self.assertNotIn('kwargs', params) - self.assertIsNone(params.a) - self.assertEqual(params.b, 3) - - params.a = 6 - a1, b1 = inspect_utils.CallWithParams(my_function, params) - self.assertEqual(a1, 6 + 1) - self.assertEqual(b1, 3 + 2) - - def testClassInit(self): - - class MyClass: - - def __init__(self, a, b=3): - self.a = a - self.b = b - - params = hyperparams.Params() - inspect_utils.DefineParams(MyClass, params) - self.assertIn('a', params) - self.assertIn('b', params) - self.assertIsNone(params.a) - self.assertEqual(params.b, 3) - - params.a = 9 - params.b = 5 - obj = inspect_utils.CallWithParams(MyClass, params) - self.assertEqual(obj.a, 9) - self.assertEqual(obj.b, 5) - - # TODO(oday): Remove this test when the bug on Keras has been resolved. - def testClassInit2(self): - - class MyClass: - - def __init__(self, a, b=3): - self.a = a - self.b = b - - params = hyperparams.Params() - inspect_utils.DefineParams(MyClass.__init__, params, bound=True) - self.assertNotIn('self', params) - self.assertIn('a', params) - self.assertIn('b', params) - self.assertIsNone(params.a) - self.assertEqual(params.b, 3) - - params.a = 9 - params.b = 5 - obj = inspect_utils.ConstructWithParams(MyClass, params) - self.assertEqual(obj.a, 9) - self.assertEqual(obj.b, 5) - - def testMethod(self): - - class MyClass: - - def __init__(self): - self._s = 'a/b' - - def split(self, sep): - return self._s.split(sep) - - params = hyperparams.Params() - inspect_utils.DefineParams(MyClass.split, params, bound=True) - self.assertNotIn('self', params) - self.assertIn('sep', params) - self.assertIsNone(params.sep) - - params.sep = '/' - parts = inspect_utils.CallWithParams(MyClass().split, params) - self.assertEqual(['a', 'b'], parts) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/layers.py b/lingvo/core/layers.py deleted file mode 100644 index 78fbbe41..00000000 --- a/lingvo/core/layers.py +++ /dev/null @@ -1,4978 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Common layers.""" - -import copy -import math -import numbers -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import bn_layers -from lingvo.core import builder_layers -from lingvo.core import computation_cost -from lingvo.core import conv_layers_with_time_padding -from lingvo.core import pruning_utils -from lingvo.core import py_utils -from lingvo.core import quant_utils -from lingvo.core import recurrent -from lingvo.core import schedule -from lingvo.core import summary_utils -from lingvo.core import symbolic -from lingvo.core import tshape -import numpy as np -import sympy - -# pylint:disable=g-direct-tensorflow-import -from tensorflow.python.ops import inplace_ops -# pylint:enable=g-direct-tensorflow-import - - -class DeconvLayer(base_layer.BaseLayer): - """Deconv (transposed conv2d) layer. - - DeconvLayer is different from ConvTransposeLayer in that - DeconvLayer does not support padding and biasing. Hence, - it's simpler and more basic than ConvTransposeLayer. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'filter_shape', (0, 0, 0, 0), - 'Filter shape. Must be a sequence of length 4. Elements are in' - ' the order of height, width, out_channel, in_channel.') - p.Define( - 'filter_stride', (0, 0), - 'Filter stride to use. Must be a pair of ints. The first int' - ' specifies the stride on the height dimension. The second int' - ' specifies the stride on the width dimension.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert len(p.filter_shape) == 4 - assert len(p.filter_stride) == 2 - assert all(x > 0 for x in p.filter_shape) - assert all(x > 0 for x in p.filter_stride) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - w_pc = py_utils.WeightParams( - shape=p.filter_shape, - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('w', w_pc) - - def OutShape(self, in_shape): - """Compute the output shape given the input shape.""" - p = self.params - t_stride = p.filter_stride[0] - f_stride = p.filter_stride[1] - return tf.stack([ - in_shape[0], in_shape[1] * t_stride, in_shape[2] * f_stride, - p.filter_shape[2] - ]) - - def _ApplyConv(self, theta, inputs): - p = self.params - w = theta.w - strides = [1, p.filter_stride[0], p.filter_stride[1], 1] - # TODO(miachen): remove casting once tf.nn.conv2d supports tf.float64. - assert inputs.dtype == w.dtype - dtype = inputs.dtype - if dtype != tf.float32: - inputs = tf.cast(inputs, tf.float32) - w = tf.cast(w, tf.float32) - # TODO(zhifengc): Try some better way to do Deconv. Search for - # "resize-convolution". - out = tf.nn.conv2d_transpose( - inputs, - w, - output_shape=self.OutShape(tf.shape(inputs)), - strides=strides, - padding='SAME') - if dtype != tf.float32: - out = tf.cast(out, dtype) - return py_utils.HasShape(out, [-1, -1, -1, p.filter_shape[2]]) - - def FProp(self, theta, inputs): - """Apply deconvolution to inputs. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - inputs: The inputs tensor. It is expected to be of shape [batch, height, - width, channel]. - - Returns: - outputs. outputs is expected to have shape [batch, height * height_stride, - width * width_stride, out_channel]. - """ - p = self.params - inputs = py_utils.HasShape(inputs, [-1, -1, -1, p.filter_shape[3]]) - return self._ApplyConv(theta, inputs) - - -def Gelu(input_tensor): - """Gaussian Error Linear Unit. - - This is a smoother version of the RELU. - Original paper: https://arxiv.org/abs/1606.08415 - - Args: - input_tensor: float Tensor to perform activation. - - Returns: - `input_tensor` with the GELU activation applied. - """ - cdf = 0.5 * (1.0 + tf.math.erf( - input_tensor / tf.cast(tf.sqrt(2.0), input_tensor.dtype))) - return input_tensor * cdf - - -# Supported activation functions. -_ACTIVATIONS = { - 'RELU': tf.nn.relu, - 'RELU6': tf.nn.relu6, - 'SIGMOID': tf.sigmoid, - 'TANH': tf.tanh, - 'GELU': Gelu, - 'SWISH': tf.nn.swish, - 'SOFTPLUS': tf.nn.softplus, -} - -_ACTIVATIONS_FLOPS = { - 'NONE': 0, - 'RELU': 1, - 'RELU6': 1, - # 1 / (1 + exp(-x)) - 'SIGMOID': 4, # neg, exp, add, div - # (exp(2*x) - 1) / (exp(2*x) - 1) - 'TANH': 7, # mul, exp, sub, mul, exp, add, div - # Gelu is tough, let's assume it is approximated as x * sigmoid(1.702 * x). - 'GELU': 6, # mul, sigmoid, mul - # x * sigmoid(x) - 'SWISH': 5, # sigmoid, mul - # ln(1+exp(x)) - 'SOFTPLUS': 3, # exp, add, ln -} - -# A subset of activation functions are supported by TFLite as fused activation -# functions with a preceding matmul or conv. If this is the case, then they -# require special treatment for quantization. -_TFLITE_FUSED_ACTIVATION_NAMES = ( - 'RELU', - 'RELU6', -) - -LOG_SCALE_CLAMP_BOUND = 20.0 - - -class IdentityLayer(base_layer.BaseLayer): - """Identity layer, adds name and propagates its input.""" - - def FProp(self, theta, inputs, *args): - """Identity mapping. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: The input tensor or the input NestedMap. - *args: Arguments to be ignored. - - Returns: - Tensor with the same shape and type of inputs. - """ - p = self.params - with tf.name_scope(p.name): - return tf.nest.map_structure(tf.identity, inputs) - - @classmethod - def FPropMeta(cls, p, inputs, *args): - py_utils.CheckShapes((inputs,)) - return py_utils.NestedMap(flops=0, out_shapes=(inputs,)) - - -# TODO(yonghui/jonathanasdf): Remove the forwarded links. -_ComputeConvOutputShape = conv_layers_with_time_padding.ComputeConvOutputShape -_ComputeConvOutputPadding = ( - conv_layers_with_time_padding.ComputeConvOutputPadding) -BatchNormLayer = bn_layers.BatchNormLayer -BatchNormLayerNoPadding = bn_layers.BatchNormLayerNoPadding -AddingAccumulator = bn_layers.AddingAccumulator - - -class BaseConv2DLayer(quant_utils.QuantizableLayer): - """Base class for 2D convolution layers. - - Has support for optional batch-normalization, activation and sequence - padding. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'filter_shape', (0, 0, 0, 0), - 'Filter shape. Must be a sequence of length 4. Elements are in' - ' the order of height (time), width (frequency), in_channel,' - ' out_channel. When causal_convolution is True, filter_shape[1]' - ' is the actual number of trained weights in the time dimension' - ' of the kernel.') - p.Define( - 'filter_stride', (0, 0), - 'Filter stride to use. Must be a pair of ints. The first int' - ' specifies the stride on the time dimension. The second int' - ' specifies the stride on the frequency dimension.') - p.Define( - 'dilation_rate', (1, 1), - 'If > 1, dilation rate for atrous convolution. ' - 'Must be a pair of ints. ' - 'The first int specifies the dilation rate on the time dimension. ' - 'The second int specifies the dilation rate on the frequency ' - 'dimension. ' - 'If any value of dilation_rate is > 1, then all values of strides ' - 'must be 1.') - p.Define( - 'activation', 'RELU', - 'Activation function to use. Options are RELU, RELU6, SIGMOID, ' - 'TANH, NONE.') - p.Define('bias', False, 'Whether or not to apply a bias before activation.') - p.Define('batch_norm', True, 'Whether or not to apply batch norm.') - p.Define( - 'bn_decay', 0.999, - 'Decay in updating the mean and variance moving average used in' - ' batch normalization.') - p.Define( - 'bn_fold_weights', None, - 'Fold the batch norm parameters into the convolution weights at ' - 'eval/inference time as per https://arxiv.org/pdf/1712.05877.pdf. ' - 'Requires that batch_norm be True and is incompatible with some other ' - 'parameters (conv_last=True).') - p.Define( - 'causal_convolution', False, - 'If true, conv layer output only depends on time steps in' - ' the past.') - p.Define( - 'conv_last', False, - 'If true, apply the convolution transformation as the last step, ' - 'i.e., first apply batch normalization on the input, followed ' - 'by activation, and finally the convolution. ' - 'Otherwise, apply convolution first, followed by batch ' - 'normalization and activation. Not compatible with bn_fold_weights ' - 'or quantization.') - p.Define( - 'weight_norm', False, - 'If true, apply weight normalization to weights as proposed by' - ' Salimans and Kingma, 2016: https://arxiv.org/abs/1602.07868') - p.Define( - 'disable_activation_quantization', False, - 'Disables the quantization tracking/clamping for the output ' - 'activation. This is most often used in conjunction with a concat ' - 'layer which needs to have a merged set of statistics.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert len(p.filter_shape) == 4 - assert len(p.filter_stride) == 2 - assert len(p.dilation_rate) == 2 - assert all(x > 0 for x in p.filter_stride) - assert all(x > 0 for x in p.dilation_rate) - if any(x > 1 for x in p.dilation_rate): - assert all(x == 1 for x in p.filter_stride) - # Bias is not needed with batch_norm=True. - if p.batch_norm: - assert not p.bias - assert (p.activation == 'NONE' or p.activation in _ACTIVATIONS) - - if p.batch_norm: - # batch normalization dimension is number of input channels - # (filter_shape[2]) if we apply batch_norm on input and convolution - # in the end, number of output channels otherwise. - bn_dim = p.filter_shape[2] if p.conv_last else self.output_channels - bn_params = BatchNormLayer.Params().Set( - dim=bn_dim, decay=p.bn_decay, name=p.name, params_init=p.params_init) - self.CreateChild('bn', bn_params) - - if self._is_bn_folded: - assert p.batch_norm, 'bn_fold_weights requires batch_norm = True' - assert not p.conv_last, 'bn_fold_weights requires conv_last = False' - - # TODO(yonghui): implement the variational noise logic. - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - w_pc = py_utils.WeightParams( - shape=p.filter_shape, - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('w', w_pc) - if p.bias: - self.CreateVariable( - 'b', - py_utils.WeightParams( - shape=[self.output_channels], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars'])) - if p.weight_norm: - self.CreateVariable( - 'g', - py_utils.WeightParams( - shape=self.filter_output_shape, - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars'])) - - if not p.disable_activation_quantization: - self.TrackQTensor('activation') - if (p.activation not in _TFLITE_FUSED_ACTIVATION_NAMES and - p.activation != 'NONE'): - self.TrackQTensor('pre_activation') - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - if self.params.batch_norm: - self.bn.InstantiateVariables() - super()._CreateChildrenVariables() - - @property - def output_channels(self): - """The number of output channels for this conv layer.""" - # Normal convolution filter shape is [..., out_channels]. - p = self.params - return p.filter_shape[-1] - - @property - def filter_output_shape(self): - """Final dims of the filter corresponding to the output channels. - - Returns: - A one (standard conv) or two (depthwise conv) element shape representing - the final dimensions of the filter weights that are output channel - specific for this layer. This shape is needed for any arithmetic that - needs to convert between a linear list of filter weights and the - arrangement in the actual filter. - """ - # Standard convolution has all output channels in the last dim. - p = self.params - return [p.filter_shape[-1]] - - @property - def _is_bn_folded(self): - """Whether batchnorm folded weights are effectively enabled.""" - p = self.params - if not p.batch_norm: - return False - return (p.bn_fold_weights or - (p.bn_fold_weights is None and p.qdomain.default is not None)) - - def _EvaluateConvKernel(self, inputs, filter_w, strides, dilation_rate, - padding_algorithm, data_format): - """Evaluates the lower level convolution kernel. - - Args: - inputs: As to tf.nn.convolution. - filter_w: As to tf.nn.depthwise_conv2d. - strides: As to tf.nn.convolution. - dilation_rate: As to tf.nn.convolution. - padding_algorithm: As to tf.nn.convolution (padding argument). - data_format: As to tf.nn.convolution. - - Returns: - Convolution kernel output. - """ - raise NotImplementedError() - - @classmethod - def OutputShape(cls, params, in_shape): - return _ComputeConvOutputShape(in_shape, params.filter_stride[0], - params.filter_stride[1], - params.filter_shape[-1]) - - def OutShape(self, in_shape): - """Compute the output shape given the input shape.""" - p = self.params - return _ComputeConvOutputShape(in_shape, p.filter_stride[0], - p.filter_stride[1], self.output_channels) - - def _GetWeights(self, - theta, - convolution_lambda, - folded_bn_padding, - cast_dtype=None): - """Gets a dictionary of weights and biases for the convolution. - - This is necessary for some operating modes where the weights are fused - with batch normalization differently for training vs eval. - - Args: - theta: A `.NestedMap` object containing underlying weights values of this - layer and its children layers. - convolution_lambda: Lambda which takes the convolution weights and runs - the convolution. - folded_bn_padding: Padding to apply to folded batch normalization moment - computation (or None for no padding). - cast_dtype: If not None, cast weights to the given dtype. - - Returns: - Tuple of (filter, biases). - """ - p = self.params - - # Original weights. - filter_w = theta.w - filter_output_shape = self.filter_output_shape - # TODO(miachen): remove casting once tf.nn.conv2d supports tf.float64. - if cast_dtype: - filter_w = tf.cast(filter_w, tf.float32) - if p.weight_norm: - if len(filter_output_shape) == 1: - # Normalize along the last dim (standard conv). - filter_w = tf.nn.l2_normalize(filter_w, [0, 1, 2]) * tf.reshape( - (theta.g + 1.0), [1, 1, 1, p.filter_shape[-1]]) - elif len(filter_output_shape) == 2: - # Normalize along the last two dimensions (depthwise conv). - filter_w = tf.nn.l2_normalize(filter_w, [0, 1]) * tf.reshape( - (theta.g + 1.0), [1, 1] + filter_output_shape) - else: - assert False, 'Unsupported weight norm filter shape' - - # Original bias. - if p.bias: - b = theta.b - else: - b = tf.zeros([symbolic.ToStatic(self.output_channels)], - dtype=filter_w.dtype) - - # Pass-through if weights are not folded with batch normalization. - if not self._is_bn_folded: - return filter_w, b - - # If batch norm is fused with weights, then compute the weights as from - # figure C.8 of https://arxiv.org/pdf/1712.05877.pdf for training and - # figure C.6 for eval. - if self.do_eval: - # Gets current moments without updating. - mean, variance, beta, gamma = self.bn.GetCurrentMoments(theta.bn) - else: - # Updates moments based on a trial run of the convolution. - raw_conv_output = convolution_lambda(filter_w) - mean, variance, beta, gamma = self.bn.ComputeAndUpdateMoments( - theta.bn, raw_conv_output, folded_bn_padding) - - # Fold weights and bias. Note that this layer's bias is not used (not - # applicable for batch norm case). - sigma_recip = tf.math.rsqrt(variance + self.bn.epsilon) - scale_correction = gamma * sigma_recip - # Normal conv will have all weights in the last dim - # ([_, _, _, output_channels]), which matches the 1D layout from - # batch norm. Depthwise uses the last two dims so reshape - # ([_, _, in_c, c_multiplier]). - scale_correction = tf.reshape(scale_correction, filter_output_shape) - filter_w = filter_w * scale_correction - b = (beta - (gamma * mean * sigma_recip)) - return filter_w, b - - def _ApplyConv(self, theta, inputs, folded_bn_padding=None): - p = self.params - strides = [p.filter_stride[0], p.filter_stride[1]] - dtype = inputs.dtype - cast_dtype = None - if dtype != tf.float32: - cast_dtype = tf.float32 - inputs = tf.cast(inputs, cast_dtype) - - padding_algorithm = 'SAME' - if p.causal_convolution: - # Causal convolution is only applied in time (height) dimension. - # Use VALID padding and shift the inputs to the right to ensure that the - # first output only depends on the first input and so on. The output is - # the same size as the input, as if the convolution used SAME padding. - padding_algorithm = 'VALID' - # The effective spatial filter size for dilated convolutions is - # (kernel - 1) * dilation_rate + 1 as according to - # https://www.tensorflow.org/api_docs/python/tf/nn/convolution. - causal_pad_size = (p.filter_shape[0] - 1) * p.dilation_rate[0] - - # Apply padding in width dimension to mimic SAME padding. - # Using the similar logic as above to produce the same number of output - # as if SAME padding is used. - width_pad_size = (p.filter_shape[1] - 1) * p.dilation_rate[1] - - # The amount of padding on the left is tricky. If stride > 1, total - # padding required for SAME padding would be: - # pad = ceil(input_size / stride - 1) * stride + eff_kernel - input_size - # where eff_kernel = (kernel - 1) * dilation_rate + 1 - # TensorFlow also pads more on the right / bottom side if total padding - # required is an odd number, so pad_left = pad // 2 - # Therefore pad_left could depend on input size, which might be dynamic. - # Here we only handle two special cases where 1) stride = 1, then - # pad_left = (eff_kernel - 1) // 2 - # and 2) kernel = 1, then - # pad_left = 0 - if p.filter_stride[1] > 1 and p.filter_shape[1] > 1: - raise ValueError('Causal convolution only supports width stride = 1 ' - 'or filter width = 1.') - width_pad_left = max(0, width_pad_size - 1) // 2 - width_pad_right = width_pad_size - width_pad_left - inputs = tf.pad(inputs, [[0, 0], [causal_pad_size, 0], - [width_pad_left, width_pad_right], [0, 0]]) - - # Lambda for computing the actual convolution. - def ComputeRawConvolution(filter_w): - return self._EvaluateConvKernel( - inputs, - filter_w=filter_w, - strides=strides, - dilation_rate=p.dilation_rate, - data_format='NHWC', - padding_algorithm=padding_algorithm) - - filter_w, b = self._GetWeights( - theta, ComputeRawConvolution, folded_bn_padding, cast_dtype=cast_dtype) - - # TODO(miachen): remove casting once tf.nn.conv2d supports tf.float64. - assert inputs.dtype == filter_w.dtype - - filter_w = self.QWeight(filter_w) - out = ComputeRawConvolution(filter_w) - - # Note that we always apply the bias (which may be zero) because some - # normalization mechanisms do implicitly produce a bias. - b = tf.cast(b, tf.float32) - out = tf.nn.bias_add(out, b) - - if dtype != tf.float32: - out = tf.cast(out, dtype) - return out - - def FProp(self, theta, inputs, paddings=None): - """Apply convolution to inputs. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: The inputs tensor. It is expected to be of shape [batch, time, - frequency, channel]. The time dimension corresponds to the height - dimension as in images and the frequency dimension corresponds to the - width dimension as in images. - paddings: The paddings tensor. If None, the inputs have no paddings in the - sense of sequence training (e.g., in CNN models). Otherwise, it is - expected to be of shape [batch, time]. - - Returns: - outputs, out_paddings pair. - """ - p = self.params - if paddings is None: - inputs = py_utils.with_dependencies([ - py_utils.assert_shape_match( - tf.shape(inputs), [-1, -1, -1, p.filter_shape[2]]) - ], inputs) - else: - inputs = py_utils.with_dependencies([ - py_utils.assert_shape_match(tf.shape(paddings), [-1, -1]), - py_utils.assert_shape_match( - tf.shape(inputs), - tf.concat([tf.shape(paddings), [-1, p.filter_shape[2]]], 0)) - ], inputs) - # Zeroing out padded inputs. - qpadding = self.QRPadding( - tf.expand_dims(tf.expand_dims(paddings, -1), -1)) - # Select based padding is required for quantized inference but is - # causing regressions on other platforms. TODO: Remove use_select - # attribute when root-caused/resolved. - inputs = py_utils.ApplyPadding( - qpadding, - inputs, - use_select=p.is_inference and p.qdomain.default is not None) - - with tf.name_scope(p.name): - input_shape = tf.shape(inputs) - - if paddings is None: - conv_padding = None - else: - # NOTE: this may be slightly inaccurate when p.dilation_rate[0] > 1. - # But there's likely no real problems. Trying to set it gives an error: - # pooling with SAME padding is not implemented for dilation_rate > 1. - # NOTE: window=p.filter_stride[0] means output i will be padded if any - # input in the stride between the two conv centers are padded. - conv_padding = _ComputeConvOutputPadding( - paddings, window=p.filter_stride[0], stride=p.filter_stride[0]) - - if p.conv_last: - out = self._ComputeConvLast(theta, inputs, paddings, conv_padding) - else: - out = self._Compute(theta, inputs, paddings, conv_padding) - - # Lastly zeroing out padded states. - if conv_padding is not None: - qpadding = self.QRPadding( - tf.expand_dims(tf.expand_dims(conv_padding, -1), -1)) - # Select based padding is required for quantized inference but is - # causing regressions on other platforms. TODO: Remove use_select - # attribute when root-caused/resolved. - out = py_utils.ApplyPadding( - qpadding, - out, - use_select=p.is_inference and p.qdomain.default is not None) - - out = py_utils.HasShape( - out, symbolic.ToStatic(BaseConv2DLayer.OutShape(self, input_shape))) - return out, conv_padding - - def _Compute(self, theta, inputs, paddings, conv_padding): - """Computes the forward prop (conv, bn, act).""" - p = self.params - - bn_padding = conv_padding - if bn_padding is None: - bn_padding_expanded = None - else: - batch_time = tf.shape(bn_padding) - batch_time_any_any = tf.concat([batch_time, [-1, -1]], 0) - bn_padding_expanded = tf.reshape(bn_padding, - tf.concat([batch_time, [1, 1]], 0)) - - out = self._ApplyConv(theta, inputs, bn_padding_expanded) - if bn_padding is not None: - out = py_utils.with_dependencies([ - py_utils.assert_shape_match(batch_time, [-1, -1]), - py_utils.assert_shape_match(tf.shape(out), batch_time_any_any) - ], out) - - # Only apply batch norm if it was not folded into the weights. - if p.batch_norm and not p.bn_fold_weights: - out = self.bn.FProp(theta.bn, out, bn_padding_expanded) - - # Apply activation. - if p.activation != 'NONE': - if p.activation not in _TFLITE_FUSED_ACTIVATION_NAMES: - out = self.QTensor('pre_activation', out) - out = _ACTIVATIONS[p.activation](out) - if not p.disable_activation_quantization: - out = self.QTensor('activation', out) - - return out - - def _ComputeConvLast(self, theta, inputs, paddings, conv_padding): - """Computes the forward prop in conv_last mode (bn, act, conv).""" - p = self.params - out = inputs - out_padding = paddings - - if p.batch_norm: - if out_padding is None: - out_padding_expanded = None - else: - batch_time = tf.shape(out_padding) - batch_time_any_any = tf.concat([batch_time, [-1, -1]], 0) - out = py_utils.with_dependencies([ - py_utils.assert_shape_match(batch_time, [-1, -1]), - py_utils.assert_shape_match(tf.shape(out), batch_time_any_any) - ], out) - out_padding_expanded = tf.reshape(out_padding, - tf.concat([batch_time, [1, 1]], 0)) - out = self.bn.FProp(theta.bn, out, out_padding_expanded) - - if p.activation != 'NONE': - out = _ACTIVATIONS[p.activation](out) - - out = self._ApplyConv(theta, out) - - return out - - -class Conv2DLayer(BaseConv2DLayer): - """Convolution layer, with optional batch-normalization and activation.""" - - def _EvaluateConvKernel(self, inputs, filter_w, strides, dilation_rate, - padding_algorithm, data_format): - p = self.params - return tf.nn.convolution( - inputs, - filter_w, - strides=strides, - dilations=p.dilation_rate, - data_format='NHWC', - padding=padding_algorithm) - - -class ConvNN2DLayer(BaseConv2DLayer): - """Convolution layer, based on tf.nn.conv2d instead of tf.nn.convolution. - - tf.nn.convolution is using a different implementation on atrous convolutions, - by wrapping the actual convolution with space_to_batch and batch_to_space. - This implementation is not supported in tflite conversion, hence we need - a different layer for using atrous convolutions. - """ - - def _EvaluateConvKernel(self, inputs, filter_w, strides, dilation_rate, - padding_algorithm, data_format): - p = self.params - return tf.nn.conv2d( - inputs, - filter_w, - strides=strides, - dilations=p.dilation_rate, - data_format='NHWC', - padding='SAME') - - -# Alias of Conv2DLayer (for compatibility with historical uses). -ConvLayer = Conv2DLayer - - -class DepthwiseConv2DLayer(BaseConv2DLayer): - """Depthwise conv 2D layer. - - paper: https://arxiv.org/abs/1610.02357 - """ - - @classmethod - def Params(cls): - p = super().Params() - # Redefine 'filter_shape' since the semantic of shape elements is different - # from regular Conv2D. - p.Delete('filter_shape') - p.Define( - 'filter_shape', (0, 0, 0, 0), - 'Filter shape. Must be a sequence of length 4. Elements are in' - ' the order of height (time), width (frequency), in_channel,' - ' channel_multipliers. ') - return p - - @property - def output_channels(self): - """The number of output channels for this conv layer.""" - p = self.params - # Depthwise convolution filter shape is: - # [..., in_channels, channel_multiplier]. - return p.filter_shape[-2] * p.filter_shape[-1] - - @property - def filter_output_shape(self): - """Final dims of the filter corresponding to the output channels.""" - # Depthwise convolution uses the final two dims for output channels. - p = self.params - _, _, in_c, c_mul = p.filter_shape - return [in_c, c_mul] - - def _EvaluateConvKernel(self, inputs, filter_w, strides, dilation_rate, - padding_algorithm, data_format): - p = self.params - return tf.nn.depthwise_conv2d( - inputs, - filter=filter_w, - strides=[1, strides[0], strides[1], 1], - dilations=p.dilation_rate, - data_format='NHWC', - padding=padding_algorithm) - - -class SeparableConv2DLayer(Conv2DLayer): - """Separable 2D convolution. - - This class aggregates a DepthwiseConv2DLayer that feeds in to the point - wise convolution defined by this layer. Since the point wise convolution - controls the output, this class is defined in terms of that and delegates - to a depthwise sub-layer. - - The `filter_shape` parameter is rewritten on initialization from the form: - (h, w, cin, cout) - To: - Depthwise filter: (h, w, cin, p.depth_multiplier) - Pointwise filter (on this instance): (1, 1, cin * p.depth_multiplier, cout) - - This way, the layer is configured as if it were a normal 2D convolution - but is internally reconfigured to be separable. - - paper: https://arxiv.org/abs/1610.02357 - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'depth_multiplier', 1, - 'Number of depthwise convolution output channels per input channel. ' - 'The total number of depthwise convolution output channels will be.' - 'equal to in_channel * depth_multiplier.') - p.Define('depthwise_tpl', - DepthwiseConv2DLayer.Params().Set(activation='NONE'), - 'Template for the depthwise conv sub-layer.') - return p - - def __init__(self, params): - # Rewrite the filter. - params = params.Copy() - h, w, cin, cout = params.filter_shape - params.filter_shape = (1, 1, cin * params.depth_multiplier, cout) - depthwise_filter_shape = (h, w, cin, params.depth_multiplier) - - # Dilation rate and stride go to the depthwise layer and reset ours. - depthwise_filter_stride = params.filter_stride - depthwise_dilation_rate = params.dilation_rate - params.filter_stride = (1, 1) - params.dilation_rate = (1, 1) - - super().__init__(params) - p = self.params - del params - - # Create the depthwise sub-layer. - depthwise_params = p.depthwise_tpl.Copy().Set( - filter_shape=depthwise_filter_shape, - filter_stride=depthwise_filter_stride, - dilation_rate=depthwise_dilation_rate, - causal_convolution=p.causal_convolution, - weight_norm=p.weight_norm, - batch_norm=p.batch_norm, - bn_decay=p.bn_decay, - bn_fold_weights=p.bn_fold_weights) - depthwise_params.qdomain.default = p.qdomain.default - self.CreateChild('depthwise_conv', depthwise_params) - - def FProp(self, theta, inputs, paddings=None): - inputs, paddings = self.depthwise_conv.FProp(theta.depthwise_conv, inputs, - paddings) - return super().FProp(theta, inputs, paddings) - - def OutShape(self, in_shape): - """Compute the output shape given the input shape.""" - in_shape = self.depthwise_conv.OutShape(in_shape) - return super().OutShape(in_shape) - - -class ProjectionLayer(quant_utils.QuantizableLayer): - """Projection layer, with batch normalization and relu activation.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', 0, 'Depth of the input.') - p.Define('output_dim', 0, 'Depth of the output.') - p.Define( - 'activation', 'RELU', - 'Activation function to use. Options are RELU, RELU6, SIGMOID, ' - 'TANH, NONE.') - p.Define('batch_norm', None, 'Whether or not to apply batch norm.') - p.Define('has_bias', False, - 'Whether or not to introduce the bias params to the layer.') - p.Define('bias_init', 0.0, 'Initial value for the bias') - p.Define( - 'affine_last', False, - 'If true, apply the affine transformation as the last step, i.e., ' - 'first apply batch normalization on the input, followed ' - 'by activation, and finally the affine transformation. ' - 'Otherwise, apply affine transformation first, followed by batch ' - 'normalization and activation.') - p.Define( - 'weight_norm', False, - 'If true, apply weight normalization to weights as proposed by' - ' Salimans and Kingma, 2016: https://arxiv.org/abs/1602.07868') - p.Define( - 'bn_fold_weights', None, - 'Fold the batch norm parameters into the convolution weights at ' - 'eval/inference time as per https://arxiv.org/pdf/1712.05877.pdf. ' - 'Defaults to None which means that it will be disabled by default ' - 'and enabled when quantized training is enabled. Not compatible with ' - 'affine_last=True') - p.Define('bn_params', - BatchNormLayer.Params().Set(decay=0.999), - 'Default params for batch norm layer.') - p.Define('apply_pruning', False, - 'Whether to prune the weights while training') - p.Define( - 'use_einsum', True, 'Whether to use tf.einsum for optimizing ' - 'computations. When this is set to False, this causes an increase in ' - 'TPU memory usage (b/158336491). When this is set to True, it might ' - ' cause problems with model quantization for on device inference ' - '(b/146421936)') - p.Define( - 'use_blocked_matmul', False, 'Whether to use blocked matrix ' - 'multiplications. This allows for weight updates to be paralellized' - ' across the cores for Shampoo optimizer.') - p.Define('block_dim', 1024, 'Dimension of the block') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert symbolic.EvalExpr(symbolic.STATIC_VALUES, p.input_dim) > 0 - assert symbolic.EvalExpr(symbolic.STATIC_VALUES, p.output_dim) > 0 - assert p.activation == 'NONE' or p.activation in _ACTIVATIONS - if p.batch_norm is None: - raise RuntimeError( - 'ProjectionLayer.batch_norm not set explicitly for %s' % self.path) - if p.batch_norm and p.has_bias: - tf.logging.warning( - 'Projection layer enables both batch_norm and has_bias. ' - 'This is generally redundant/wasteful and may introduce ' - 'accuracy problems in some inference scenarios.') - if self._is_bn_folded: - assert not p.use_blocked_matmul, ( - 'bn_fold_weights requires use_blocked_matmul = False') - assert not p.affine_last, ( - 'Folded batchnorm is not compatible with affine_last') - - if p.use_einsum: - assert not p.use_blocked_matmul, ( - 'use_einsum requires use_blocked_matmul = False') - - if p.batch_norm: - bn_params = p.bn_params.Copy() - bn_params.name = p.name - bn_params.dim = p.input_dim if p.affine_last else p.output_dim - - self.CreateChild('bn', bn_params) - # TODO(yonghui): implement the variational noise logic. - - def _GetBlockedMatMulInputOutputMultipliers(self): - """Get number of input and output blocks.""" - p = self.params - # Number of input and output blocks. - w_im = p.input_dim // p.block_dim - w_om = p.output_dim // p.block_dim - # Add padding if input_dim / output_dim is not divisible by block_dim. - if p.input_dim % p.block_dim != 0: - w_im += 1 - if p.output_dim % p.block_dim != 0: - w_om += 1 - return w_im, w_om - - def _GetBlockedWeightMatrix(self, w): - """Returns a 3D weight matrix for blocked matmul.""" - p = self.params - # w is 3D Tensor of shape [i * o, block_dim, block_dim] such that - # i * block_dim = num_inputs (modulo padding). - # j * block_dim = num_outputs - # - # To efficiently apply forward prop, we transpose and reshape w into - # shape [i * block_dim, o, block_dim] - w_im, w_om = self._GetBlockedMatMulInputOutputMultipliers() - block_dim = p.block_dim - w_4d = tf.reshape(w, [w_im, w_om, block_dim, block_dim]) - # Transpose to [i, block_dim, o, block_dim]. - w_4d_t = tf.transpose(w_4d, [0, 2, 1, 3]) - w = tf.reshape(w_4d_t, [w_im * block_dim, w_om, block_dim]) - # Slice out padding from the weight matrix. - if p.input_dim % p.block_dim != 0: - w = tf.slice(w, [0, 0, 0], [p.input_dim, w_om, block_dim]) - return w - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - if p.use_blocked_matmul: - w_im, w_om = self._GetBlockedMatMulInputOutputMultipliers() - w_pc = py_utils.WeightParams( - shape=[w_im * w_om, p.block_dim, p.block_dim], - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - else: - w_pc = py_utils.WeightParams( - shape=[p.input_dim, p.output_dim], - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - - if p.apply_pruning: - mask_w_pc = py_utils.WeightParams(w_pc.shape, - py_utils.WeightInit.Constant(1.0), - p.dtype) - threshold_w_pc = py_utils.WeightParams([], - py_utils.WeightInit.Constant(0.0), - tf.float32) - if p.has_bias: - b_pc = py_utils.WeightParams( - shape=[p.output_dim], - init=py_utils.WeightInit.Constant(scale=p.bias_init), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - if p.weight_norm: - g_pc = py_utils.WeightParams( - shape=[p.output_dim], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - - weights_var_name = 'w' - if p.apply_pruning: - mask_var_name = 'mask' - threshold_var_name = 'threshold' - self.CreateVariable( - mask_var_name, mask_w_pc, theta_fn=None, trainable=False) - self.CreateVariable( - threshold_var_name, threshold_w_pc, theta_fn=None, trainable=False) - - def MaskWeightFn(weight): - return tf.multiply( - self.AddGlobalVN(weight), getattr(self.vars, mask_var_name), - 'masked_w') - - self.CreateVariable(weights_var_name, w_pc, theta_fn=MaskWeightFn) - pruning_utils.AddToPruningCollections( - getattr(self.vars, weights_var_name), getattr(self.vars, - mask_var_name), - getattr(self.vars, threshold_var_name)) - else: - self.CreateVariable(weights_var_name, w_pc) - - if p.has_bias: - self.CreateVariable('b', b_pc) - if p.weight_norm: - self.CreateVariable('g', g_pc) - - # Determine quantization needs based on whether fusing activation - # or not. - self._pre_activation_qt_name = None - self._output_qt_name = ('activation' - if p.activation != 'NONE' else 'affine_matmul') - if (p.activation != 'NONE' and - p.activation not in _TFLITE_FUSED_ACTIVATION_NAMES): - # Not a fused activation function. - # Need a qtensor to track the pre-activation tensor. The name is - # compatible with older checkpoints. - self._pre_activation_qt_name = 'affine_matmul' - self.TrackQTensor(self._output_qt_name) - if self._pre_activation_qt_name: - self.TrackQTensor(self._pre_activation_qt_name) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - if self.params.batch_norm: - self.bn.InstantiateVariables() - super()._CreateChildrenVariables() - - @classmethod - def NumOutputNodes(cls, p): - return p.output_dim - - @property - def output_qt_name(self): - """Name of QTensor used for the output value. - - Useful for grabbing the quantization of the output. - - Returns: - String name of output qtensor. - """ - return self._output_qt_name - - def FProp(self, theta, inputs, paddings=None): - """Apply projection to inputs. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: The inputs tensor. Shaped [..., input_dim]. - paddings: The paddings tensor. Shaped [..., 1], where all but the last - dimension match. - - Returns: - Output after applying projection, and optionally batch normalization and - relu non-linearity. - """ - p = self.params - with tf.name_scope(p.name): - if paddings is None: - paddings = tf.zeros( - tf.concat([py_utils.GetShape(inputs)[:-1], [1]], axis=0), - dtype=inputs.dtype) - w, b = self._GetWeights(theta, inputs, paddings) - w = self.QWeight(w) - - if p.affine_last: - # Reversed computation. Does not handle folding. - out = inputs - if p.batch_norm: - out = self.bn.FProp(theta.bn, out, paddings) - if p.activation != 'NONE': - if not p.is_inference: - out = py_utils.CheckNumerics(out) - out = _ACTIVATIONS[p.activation](out) - out = self._ApplyProjectionKernel(w, b, out, with_activation=False) - else: - # Normal ordered projection. - if self._is_bn_folded or not p.batch_norm: - # Everything folded together. This is the only variant that supports - # quantization. - out = self._ApplyProjectionKernel(w, b, inputs, quant=True) - else: - # Projection kernel(no activation fn) -> BN -> Activation fn. - out = self._ApplyProjectionKernel(w, b, inputs, with_activation=False) - if p.batch_norm: - out = self.bn.FProp(theta.bn, out, paddings) - if p.activation != 'NONE': - if not p.is_inference: - out = py_utils.CheckNumerics(out) - out = _ACTIVATIONS[p.activation](out) - return py_utils.ApplyPadding(self.QRPadding(paddings), out) - - @property - def _is_bn_folded(self): - """Whether batchnorm folded weights are effectively enabled.""" - p = self.params - if not p.batch_norm: - return False - return (p.bn_fold_weights or - (p.bn_fold_weights is None and p.qdomain.default is not None)) - - def _GetWeights(self, theta, inputs, paddings): - """Gets the weights for the computation. - - Weights will always have weight_norm applied and may have batch_norm - folded if enabled. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: Inputs (needed for batchnorm folding). - paddings: Paddings (needed for batchnorm folding). - - Returns: - Tuple of (w, b) to use for the forward pass. b may be None if bias is - disabled. - """ - p = self.params - w = theta.w - b = theta.b if p.has_bias else None - if p.use_blocked_matmul: - w = self._GetBlockedWeightMatrix(w) - if p.weight_norm: - w = tf.nn.l2_normalize(w, 0) - else: - if p.weight_norm: - w = tf.reshape((theta.g + 1.0) * tf.nn.l2_normalize(w, [0]), - py_utils.ToStaticShape([p.input_dim, p.output_dim])) - - if not self._is_bn_folded: - return w, b - - # If batch norm is fused with weights, then compute the weights as from - # figure C.8 of https://arxiv.org/pdf/1712.05877.pdf for training and - # figure C.6 for eval. - if self.do_eval: - # Gets current moments without updating. - mean, variance, beta, gamma = self.bn.GetCurrentMoments(theta.bn) - else: - # Updates moments based on a trial run of the kernel (without activation - # function). - raw_output = self._ApplyProjectionKernel( - w, b, inputs, with_activation=False) - mean, variance, beta, gamma = self.bn.ComputeAndUpdateMoments( - theta.bn, raw_output, paddings) - - # Fold weights and bias. - sigma_recip = tf.math.rsqrt(variance + self.bn.epsilon) - scale_correction = gamma * sigma_recip - w = w * scale_correction - b = beta - (gamma * mean * sigma_recip) - return w, b - - def _ApplyProjectionKernel(self, - w, - b, - inputs, - with_activation=True, - quant=False, - bn=False): - """Applies matmul/bias/activation in one step. - - Note that it is important that these three ops be computed in this way as - downstream inference engines (esp. for quantized inference) can recognize - and fuse them. For floating point, this is an optimization, but for - quantization, it is required. - - Args: - w: Weight matrix. - b: Bias vector (or None). - inputs: FProp inputs. - with_activation: Whether to also compute the activation function. - quant: Whether to apply quantization. - bn: Apply batchnorm. - - Returns: - Output tensor reshaped. - """ - p = self.params - - if not p.use_blocked_matmul: - if p.use_einsum: - out = py_utils.ProjectLastDim(inputs, w, p.input_dim, p.output_dim) - else: - out = py_utils.Matmul( - tf.reshape(inputs, py_utils.ToStaticShape([-1, p.input_dim])), w) - else: - x = tf.reshape(inputs, py_utils.ToStaticShape([-1, p.input_dim])) - out = tf.einsum('bn,nmk->bmk', x, w) - # Create an output layer [b, num_outputs]. - bsz = py_utils.GetShape(out)[0] - out = tf.reshape(out, [bsz, -1]) - if p.output_dim % p.block_dim != 0: - out_shape = [bsz, p.output_dim] - out = tf.slice(out, [0, 0], out_shape) - - if b is not None: - out += b # NOTE: Bias on matmul is never quantized. - return self._ApplyActivationFunction(out, inputs, with_activation, quant) - - def _ApplyActivationFunction(self, - out, - inputs, - with_activation=True, - quant=False): - """Applies the activation function in one step. - - Args: - out: The result of applying the weight matrix (and bias) to the inputs. - inputs: FProp inputs. - with_activation: Whether to also compute the activation function. - quant: Whether to apply quantization. - - Returns: - Output tensor reshaped. - """ - p = self.params - if with_activation and p.activation != 'NONE': - if self._pre_activation_qt_name: - # Track quantization for unfused activation function. - out = self.QTensor(self._pre_activation_qt_name, out) - if not p.is_inference: - out = py_utils.CheckNumerics(out) - out = _ACTIVATIONS[p.activation](out) - if quant: - out = self.QTensor(self._output_qt_name, out) - if not p.use_einsum: - out = tf.reshape( - out, - tf.concat([ - py_utils.GetShape(inputs)[:-1], - py_utils.ToStaticShape([p.output_dim]) - ], - axis=0)) - return out - - @classmethod - def FPropMeta(cls, p, inputs, paddings=None): - py_utils.CheckShapes((inputs,)) - assert inputs[-1] == p.input_dim - flops = 0 - in_dim = inputs[-1] - other_dims = inputs.num_elements() / in_dim - # matmuls. - flops += other_dims * p.input_dim * p.output_dim * 2 - # activations. - flops += other_dims * p.output_dim * _ACTIVATIONS_FLOPS[p.activation] - if p.has_bias: - flops += p.output_dim - out_shape = tshape.Shape(inputs[:-1] + [p.output_dim]) - if p.batch_norm: - bn_meta = p.bn_params.cls.FPropMeta( - p.bn_params.Copy().Set(dim=p.output_dim), out_shape) - flops += bn_meta.flops - if p.weight_norm: - # l2 normalize + element-wise multiply. - flops += 2 * p.input_dim + 2 * p.input_dim * p.output_dim + 2 - - return py_utils.NestedMap(flops=flops, out_shapes=(out_shape,)) - - -class FCLayer(ProjectionLayer): - """Fully-connected layer (matmul + bias + optional activation).""" - - @classmethod - def Params(cls): - p = super().Params() - p.batch_norm = False - p.has_bias = True - return p - - -class FeedForwardNet(quant_utils.QuantizableLayer): - """A simple multiple layer feedforward network. - - This class represents a stack of fully connected feedforward network. Each - layer in the network can be configured for whether or not to have batch-norm - applied to its output, its activation function, whether or not to apply - dropout to post-activation output. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', 0, 'Depth of the input to the network.') - p.Define('hidden_layer_dims', [], 'Depth of the hidden layer outputs.') - p.Define( - 'projection', ProjectionLayer.Params(), - 'Projection layer params. A single parameter that will be shared by' - 'all layers.') - p.Define( - 'dropout', DropoutLayer.Params(), - 'Dropout layer params. Can be a single params or a tuple/list of params' - ' having the same length as the number of layers.') - p.Define( - 'batch_norm', False, - 'Whether or not to apply BN to hidden layer output. ' - 'This can be a single bool or a tuple/list of bools having the' - ' same length as the number of layers.') - p.Define( - 'activation', 'RELU', - 'The activation function to use. Can be a single string, or a' - ' tuple/list of strings having the same length as the number' - ' of layers.') - p.Define( - 'has_bias', None, 'Whether or not to use bias for projection layers.' - 'This can be a None, single bool or a tuple/list of bools having the ' - 'same length as the number of layers. If None, the has_bias is set to ' - 'True whenever batch_norm is False for each projection layer.') - p.Define( - 'weight_norm', False, - 'Whether or not to apply weight normalization to weights. This can be ' - 'a single bool or a tuple/list of bools having the same length as the ' - 'number of layers.') - p.Define('skip_connections', None, 'Must be None.') - p.Define( - 'bn_fold_weights', None, 'Force folding the batch normalization ' - 'weights in the projection layer.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert symbolic.ToStatic(p.input_dim) > 0 - assert all(symbolic.ToStatic(x) > 0 for x in p.hidden_layer_dims) - - assert p.skip_connections is None - batch_norm = p.batch_norm - num_layers = len(p.hidden_layer_dims) - if isinstance(batch_norm, (list, tuple)): - assert len(batch_norm) == num_layers - else: - batch_norm = [batch_norm] * num_layers - weight_norm = p.weight_norm - if isinstance(weight_norm, (list, tuple)): - assert len(weight_norm) == num_layers - else: - weight_norm = [weight_norm] * num_layers - - activation = p.activation - if isinstance(activation, str): - activation = [activation] * num_layers - else: - assert len(activation) == num_layers - has_bias = p.has_bias - if isinstance(has_bias, (list, tuple)): - assert len(has_bias) == num_layers - else: - has_bias = [has_bias] * num_layers - # Set has_bias to (not batch_norm) if None. - for i in range(num_layers): - if has_bias[i] is None: - has_bias[i] = (not batch_norm[i]) - params_dropout_layers = p.dropout - if isinstance(params_dropout_layers, (list, tuple)): - assert len(params_dropout_layers) == num_layers - else: - params_dropout_layers = [params_dropout_layers] * num_layers - - # Residual connections work better in the form of: - # y = x + Affine(Activation(BatchNorm(x))) - params_fc_layers = [] - in_dim = p.input_dim - for i in range(num_layers): - out_dim = p.hidden_layer_dims[i] - proj_out_dim = out_dim - name = '%s_%d' % (p.name, i) - params_i = p.projection.Copy().Set( - batch_norm=batch_norm[i], - weight_norm=weight_norm[i], - has_bias=has_bias[i], - activation=activation[i], - input_dim=in_dim, - output_dim=proj_out_dim, - bn_fold_weights=p.bn_fold_weights, - name=name) - params_fc_layers.append(params_i) - in_dim = out_dim - - if p.qdomain.default is not None: - params_i.qdomain.default = p.qdomain.default.Copy() - - self.CreateChildren('fc', params_fc_layers) - self.CreateChildren('dropout', params_dropout_layers) - - @property - def output_dim(self): - """Returns output dimension of the FeedForwardNet.""" - return self.params.hidden_layer_dims[-1] - - def FProp(self, theta, inputs, paddings=None): - p = self.params - num_layers = len(self.fc) - - in_dim, layer_in = p.input_dim, inputs - for i in range(num_layers): - layer_in = py_utils.with_dependencies([ - py_utils.assert_shape_match([tf.shape(layer_in)[-1]], - [symbolic.ToStatic(in_dim)]) - ], layer_in) - out_dim = p.hidden_layer_dims[i] - layer_out = self.fc[i].FProp(theta.fc[i], layer_in, paddings) - layer_out = self.dropout[i].FProp(theta.dropout[i], layer_out) - layer_in = layer_out - in_dim = out_dim - return layer_in - - @classmethod - def FPropMeta(cls, p, inputs, paddings=None): - py_utils.CheckShapes((inputs,)) - assert inputs[-1] == p.input_dim - flops = 0 - with tf.Graph().as_default(): # throw-away graph. - instance = p.Instantiate() - for fc in instance.fc: - proj_params = fc.params - proj_shape = tshape.Shape(inputs[:-1] + [proj_params.input_dim]) - proj_meta = proj_params.cls.FPropMeta(proj_params, proj_shape) - flops += proj_meta.flops - out_shape = tshape.Shape(inputs[:-1] + [p.hidden_layer_dims[-1]]) - return py_utils.NestedMap(flops=flops, out_shapes=(out_shape,)) - - -class StackingOverTime(base_layer.BaseLayer): - """Stacking applied along the time axis. - - At each time step of an input sequence, elements are stacked over the - window of ('left_context' + 1 + 'right_context') steps around the current - time step. Zeros will be padded to the left or right of the sequence for - elements around the boundaries. Finally the stacked outputs are emitted - once every 'stride' steps. - - E.g. if an input sequence is: [4], [1], [9], [3], [5], [2], [8] - left_context = 1, right_context = 1, stride = 3, - then the output sequence would be: [0, 4, 1], [9, 3, 5], [2, 8, 0] - - Note that this layer only performs tensor transformation, so there are no - learnable parameters. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('left_context', 0, - 'Number of time steps to stack on the left to the central step.') - p.Define('right_context', 0, - 'Number of time steps to stack on the right to the central step.') - p.Define('stride', 1, 'The stride for emitting the stacked output.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.left_context >= 0 - assert p.right_context >= 0 - assert p.stride >= 1 - - @property - def window_size(self): - """Returns the stacking window size. - - The output dimension will be window_size * the input dimension. - - Returns: - Window size. - """ - p = self.params - return p.left_context + p.right_context + 1 - - def _ApplyStack(self, inputs, pad_value=0.0): - """The core function to apply the stacking to inputs. - - Args: - inputs: [batch, time, depth]. - pad_value: the padding value for left/right context. - - Returns: - [batch, ceil(time / stride), depth * stacking_window_length] tensor. - """ - p = self.params - if p.left_context == 0 and p.right_context == 0: - out = inputs - else: - inputs_max_len = py_utils.GetShape(inputs, 3)[1] - # Add zero paddings to the left and right of the input sequence. - inputs = tf.pad( - inputs, [[0, 0], [p.left_context, p.right_context], [0, 0]], - constant_values=pad_value) - - # Make window_size() copies of the padded sequence with the original - # sequence length, where each copy is offset by 1 time step. - pieces = [] - for i in range(self.window_size): - pieces.append(inputs[:, i:i + inputs_max_len]) - # Apply stacking. - out = tf.concat(pieces, 2) - - # Apply striding. - out = out[:, ::p.stride] - return out - - def FProp(self, inputs, paddings=None): - """Apply the stacking to inputs along the time axis. - - Args: - inputs: The inputs tensor. It is expected to be of shape [batch, time, - feature]. - paddings: The paddings tensor. It is expected to be of shape [batch, time, - 1], where all but the last dimension match inputs. Each value is 0 or 1 - indicating whether a time step of a sequence is padded in the inputs to - reach the max length in the batch. - - Returns: - (outputs, out_paddings) pair. - outputs is of shape [batch, ceil(time / stride), feature * stacking]. - out_paddings is of shape [batch, ceil(time / stride), 1]. out_paddings - will be 0 if any of the corresponding input padding is 0. - """ - if paddings is None: - paddings = tf.zeros( - tf.concat([py_utils.GetShape(inputs)[:-1], [1]], 0), - dtype=inputs.dtype) - inputs = py_utils.with_dependencies( - [ - # Checks the inputs shape has 3 dimensions. - py_utils.assert_shape_match(tf.shape(inputs), [-1, -1, -1]), - # Checks the paddings shape has 3 dimensions, and the last one is 1. - py_utils.assert_shape_match(tf.shape(paddings), [-1, -1, 1]), - # Checks the first two dimensions of inputs and paddings match. - py_utils.assert_shape_match( - tf.shape(inputs)[:-1], - tf.shape(paddings)[:-1]) - ], - inputs) - p = self.params - with tf.name_scope(p.name): - outputs = self._ApplyStack(inputs) - - # Stack the padding values with the same context and stride parameters. - # Then take the minimum padding values within each stacking window, since - # an output time step becomes a padded one only if all of the underlying - # stacked steps are padded ones. - out_paddings = self._ApplyStack(paddings, pad_value=1) - out_paddings = tf.reduce_min(out_paddings, axis=2, keepdims=True) - - return outputs, out_paddings - - def Unstack(self, stacked): - """Inverts stacking over time. - - Given 'stacked' outputs from this StackingOverTime layer, - - stacked, _ = this_layer.FProp(inputs), - - this method attempts to reconstruct the original 'inputs'. - - If stride > window_size, the original input cannot be recovered, and a - ValueError is raised. - - Otherwise, if right_context + 1 >= stride, this method returns a Tensor that - is identical to 'inputs' but potentially longer due to paddings. - - If right_context + 1 < stride, this method returns a Tensor that may be up - to ```stride - right_context - 1``` frames shorter than the original input, - but identical in the frames that are returned. e.g.:: - - left_context = 2, right_context = 1, stride = 4 - input sequence: 1 2 3 4 5 6 7 8 - after padding: 0 0 1 2 3 4 5 6 7 8 0 - windows: - [0 0 (1) 2] 3 4 5 6 7 8 0 - 0 0 1 2 [3 4 (5) 6] 7 8 0 - stacked: - [[0 0 1 2], [3 4 5 6]] - unstacked: - [1 2 3 4 5 6], which is 4 - 1 - 1 = 2 (stride - right_context - 1) - frames shorter than the original input. - - `Unstack()` can be used to project the outputs of downstream layers back to - the shape of the original unstacked inputs. For example:: - - inputs = ... # [batch, length, input_dim] - # [batch, ceil(length / stride), rnn_dim] - rnn_out = rnn.FProp(stacking.FProp(inputs)[0]) - # [batch, length, rnn_dim] - back_projected_rnn_out = py_utils.PadOrTrimTo( - stacking.Unstack(tf.tile(rnn_out, [1, 1, stacking.window_size])), - py_utils.GetShape(inputs)) - - Note this method does not take or return a separate padding tensor. The - caller is responsible for knowing which of outputs are padding (e.g. based - on the padding of the original FProp inputs). - - Args: - stacked: Tensor of shape [batch, time, window_size * feature_dim], assumed - to be the output of `FProp`. - - Returns: - The reconstructed input Tensor, with shape - [batch, (frames - 1) * stride + right_context + 1, feature_dim]. - - Raises: - ValueError: if stride > window_size. - """ - p = self.params - if p.stride > self.window_size: - raise ValueError( - "Can't invert StackingOverTime with stride (%d) > window_size (%d)" % - (p.stride, self.window_size)) - - # Reshape to allow indexing individual frames within each stacked window. - batch_size, stacked_length, _ = py_utils.GetShape(stacked, 3) - stacked = tf.reshape(stacked, - [batch_size, stacked_length, self.window_size, -1]) - - # Compute the index of the window and frame in 'stacked' where each frame of - # the original input is located, and extract them with tf.gather_nd. - # First compute for all except the last window, since these elements have - # the potential of being looked up from the next window. - input_indices = tf.range(0, (stacked_length - 1) * p.stride) - mod = input_indices % p.stride - in_next_window = tf.cast(tf.greater(mod, p.right_context), tf.int32) - window_index = input_indices // p.stride + in_next_window - frame_index = p.left_context + mod - p.stride * in_next_window - # Now handle the last window explicitly and concatenate onto the existing - # window_index/frame_index tensors. - last_window_length = p.right_context + 1 - window_index = tf.concat( - [window_index, - tf.fill([last_window_length], stacked_length - 1)], - axis=0) - frame_index = tf.concat( - [frame_index, p.left_context + tf.range(last_window_length)], axis=0) - # Stack the indices for tf.gather_nd. - window_and_frame_indices = tf.stack([window_index, frame_index], axis=1) - window_and_frame_indices = tf.tile( - tf.expand_dims(window_and_frame_indices, 0), [batch_size, 1, 1]) - return tf.gather_nd(stacked, window_and_frame_indices, batch_dims=1) - - -class PoolingLayer(quant_utils.QuantizableLayer): - """Pooling layer, by default performs max-pooling. - - Quantization notes: Unlike the common pattern, the pooling layer inputs - and output must be quantized to the same range, so it tracks both (vs - just the output). The preceding layer must have its output quantization - disabled. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'window_shape', (0, 0), - 'Window shape. Must be a pair of ints. Elements are in' - ' the order of height (time), width (frequency).') - p.Define( - 'window_stride', (0, 0), - 'Window stride to use. Must be a pair of ints. The first int' - ' specifies the stride on the time dimension. The second int' - ' specifies the stride on the frequency dimension.') - p.Define('pooling_type', 'MAX', 'Pooling type: MAX|AVG') - p.Define( - 'padding_algorithm', 'SAME', - 'Padding algorithm. See the "returns" section of ' - '`tf.nn.convolution` for details. ' - 'Roughly, VALID = NO_PADDING and SAME (default) = PAD INPUT') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert len(p.window_shape) == 2 - assert len(p.window_stride) == 2 - assert all([x > 0 for x in p.window_shape]) - assert all([x > 0 for x in p.window_stride]) - assert p.pooling_type in ['MAX', 'AVG'] - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - self.TrackQTensor('output') - - def OutShape(self, in_shape): - """Compute the output shape given the input shape.""" - p = self.params - return _ComputeConvOutputShape( - in_shape, - p.window_stride[0], - p.window_stride[1], - padding=p.padding_algorithm) - - def FProp(self, theta, inputs, paddings=None): - """Apply pooling to inputs. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: The inputs tensor. It is expected to be of shape [batch, time, - frequency, channel]. The time dimension corresponds to the height - dimension as in images and the frequency dimension corresponds to the - width dimension as in images. - paddings: The paddings tensor. It is expected to be of shape [batch, - time]. Defaults to None, which means there no paddings. - - Returns: - outputs, out_paddings pair. - """ - p = self.params - stride = p.window_stride - window = p.window_shape - if paddings is not None: - inputs = py_utils.with_dependencies([ - py_utils.assert_shape_match(tf.shape(paddings), [-1, -1]), - py_utils.assert_shape_match(tf.shape(inputs)[:2], tf.shape(paddings)) - ], inputs) - with tf.name_scope(p.name): - if paddings is not None: - out_padding = _ComputeConvOutputPadding(paddings, window[0], stride[0], - p.padding_algorithm) - if p.pooling_type == 'MAX': - # Fill dtype.min in padded positions. - min_value = tf.ones_like(inputs) * p.dtype.min - inputs = py_utils.ApplyPadding(paddings[..., tf.newaxis, tf.newaxis], - inputs, min_value) - else: - out_padding = None - inputs = self.QTensor('output', inputs) - - out = tf.nn.pool( - inputs, - window, - p.pooling_type, - strides=stride, - padding=p.padding_algorithm, - data_format='NHWC', - ) - if paddings is not None and p.pooling_type == 'AVG': - # Count the fraction of non-padding elements inside each pooling window. - in_mask = 1.0 - paddings - non_padding_ratio = tf.nn.pool( - in_mask[:, :, tf.newaxis], - window_shape=(p.window_shape[0],), - pooling_type='AVG', - strides=(p.window_stride[0],), - padding=p.padding_algorithm) - # Divide by non-padding ratios to eliminate the effect of padded values. - out *= tf.math.reciprocal_no_nan(non_padding_ratio)[..., tf.newaxis] - - out = self.QTensor('output', out) - if out_padding is not None: - out *= tf.expand_dims(tf.expand_dims(1.0 - out_padding, -1), -1) - return out, out_padding - - -class BlurPoolLayer(base_layer.BaseLayer): - """BlurPool from https://arxiv.org/pdf/1904.11486.pdf. - - This layer blurs the input with a fixed filter and performs subsampling - afterwards. Only supports 2x1 or 2x2 spatial reduction. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('blur_filter', 'B5', 'One of [R2, T3, B5]; the fixed blur filter.') - p.Define('subsample_type', '1D', 'Choose between [1D, 2D] subsampling.') - p.Define('input_channels', None, 'Number of input channels.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.blur_filter in ['R2', 'T3', 'B5'] - assert p.subsample_type in ['1D', '2D'] - assert p.input_channels - - filter_dict = { - 'B5': np.array([1, 4, 6, 4, 1], dtype=np.float32), - 'T3': np.array([1, 2, 1], dtype=np.float32), - 'R2': np.array([1, 1], dtype=np.float32) - } - base_filter = filter_dict[p.blur_filter] - - if p.subsample_type == '2D': - base_filter = base_filter[:, np.newaxis] * base_filter[np.newaxis, :] - else: - base_filter = base_filter[:, np.newaxis] - base_filter /= base_filter.sum() - - self._blur_filter = np.tile(base_filter[..., np.newaxis, np.newaxis], - (1, 1, p.input_channels, 1)) - conv_params = DepthwiseConv2DLayer.Params().Set( - activation='NONE', - batch_norm=False, - filter_stride=(1, 1), - filter_shape=self._blur_filter.shape) - - self.CreateChild('blur_conv', conv_params) - - def FProp(self, theta, inputs, paddings=None): - """Apply blur pooling. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: The inputs tensor. It is expected to be of shape [batch, time, - frequency, channel]. The time dimension corresponds to the height - dimension as in images and the frequency dimension corresponds to the - width dimension as in images. - paddings: The paddings tensor. It is expected to be of shape [batch, - time]. Defaults to None, which means there no paddings. - - Returns: - outputs, out_paddings pair. - """ - p = self.params - if paddings is not None: - inputs = py_utils.with_dependencies([ - py_utils.assert_shape_match(tf.shape(paddings), [-1, -1]), - py_utils.assert_shape_match(tf.shape(inputs)[:2], tf.shape(paddings)) - ], inputs) - # blur - theta_cp = copy.copy(theta.blur_conv) - theta_cp.w = tf.convert_to_tensor(self._blur_filter, dtype=p.dtype) - out, out_padding = self.blur_conv.FProp(theta_cp, inputs, paddings) - - # b/142399320 - # Use stride in blur conv for subsampling once non-square stride gets - # supported. - if p.subsample_type == '2D': - out = out[:, ::2, ::2, :] - else: - out = out[:, ::2, :, :] - - if out_padding is not None: - out_padding = _ComputeConvOutputPadding( - out_padding, window=2, stride=2, padding_algorithm='SAME') - out *= (1.0 - out_padding)[..., tf.newaxis, tf.newaxis] - else: - out_padding = None - - return out, out_padding - - -class SingleShardEmbeddingLayer(base_layer.BaseLayer): - """Embedding layer that is not sharded. - - This embedding layer is expected to be replicated over all compute devices - (e.g. tpu cores). It is intended to support small to medium embedding tables - (< 50k) only. - - This is intended to be a unification of EmbeddingLayer and - SimpleEmbeddingLayer (and cleanup of both). It is targeting the most common - use-case we have in speech/nmt/tts/deeprank. Currently we often first - configure a model using EmbeddingLayer, and then call ChangeToSimpleEmbedding - to switch to SimpleEmbedding where we lose some configuration (e.g. - scale_by_sqrt_dim). - - TODO(lingvo): Implement the matmul option which should be more efficient for - small vocabs (e.g. < 1k vocab). - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('vocab_size', 0, 'Num tokens in vocab.') - p.Define('embedding_dim', 0, 'Depth of the output.') - p.Define( - 'scale_sqrt_depth', False, 'If set True, activations are scaled' - ' with sqrt(embedding_dim) in EmbLookup.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.vocab_size > 0 - assert p.embedding_dim > 0 - assert p.name - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - w_pc = py_utils.WeightParams( - shape=[p.vocab_size, p.embedding_dim], - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('emb_var', w_pc) - - def EmbLookupDefaultTheta(self, ids): - return self.EmbLookup(self.theta, ids) - - def EmbLookup(self, theta, ids): - """Looks up embedding vectors for ids. - - Args: - theta: Named tuple with the weight matrix for the embedding. - ids: A rank-N int32 tensor. - - Returns: - A rank-(N+1) params.dtype tensor. - embs[indices, :] is the embedding vector for ids[indices]. - """ - p = self.params - ids = tf.convert_to_tensor(ids) - ids = py_utils.with_dependencies([ - py_utils.assert_between( - ids, 0, p.vocab_size, name='vocab_id_validation') - ], ids) - embs = tf.nn.embedding_lookup(theta.emb_var, tf.reshape(ids, [-1])) - if p.scale_sqrt_depth: - embs *= p.embedding_dim**0.5 - if p.vn.global_vn or p.vn.per_step_vn: - embs = py_utils.AddGlobalVN(p, embs) - out_shape = tf.concat([tf.shape(ids), [p.embedding_dim]], 0) - return tf.reshape(embs, out_shape) - - def FProp(self, theta, ids): - return self.EmbLookup(theta, ids) - - -class EmbeddingLayer(base_layer.BaseLayer): - """Embedding layer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('vocab_size', 0, 'Depth of the input.') - p.Define('embedding_dim', 0, 'Depth of the output.') - p.Define('max_num_shards', 0, 'Num param shards.') - p.Define('on_ps', True, 'True if to perform the embedding lookup on ps.') - p.Define( - 'scale_sqrt_depth', False, 'If set True, activations are scaled' - ' with sqrt(embedding_dim) in EmbLookup.') - return p - - # Min number of params per shard. - MIN_PARAMS_PER_SHARD = 1024 * 256 - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.vocab_size > 0 - assert p.embedding_dim > 0 - assert p.max_num_shards > 0 - assert p.name - - total_size = p.vocab_size * p.embedding_dim - self._actual_shards = min( - p.max_num_shards, - int(math.ceil(float(total_size) / self.MIN_PARAMS_PER_SHARD))) - self._ids_per_shard = int( - math.ceil(float(p.vocab_size) / self._actual_shards)) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - w_pc = py_utils.WeightParams( - shape=[self._ids_per_shard, p.embedding_dim], - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - - # EmbeddingLayer handles vars/theta differently from other layers - # because when embedding shards are placed on ps, it's more - # efficiently to do embedding lookups on ps and sends the result - # back to the worker. - emb_vars = [] - emb_shards = [] - for i in range(self._actual_shards): - var_name = 'var_%d' % i - self.CreateVariable(var_name, w_pc) - emb_vars.append(self.vars[var_name]) - # NOTE: self.theta[var_name] has transformations such as variational noise - # applied via theta_fn in self.CreateVariable. For embedding layer we - # apply variational noise explicitly in EmbLookup, so we do not use - # self.theta[var_name] here. - v = self.vars[var_name] - if not p.on_ps: - v = tf.identity(v) - if p.fprop_dtype is not None and p.fprop_dtype != p.dtype: - v = tf.cast(v, p.fprop_dtype) - emb_shards.append(v) - # Remove from _private_vars / _private_thetas to be added later as wm. - del self._private_vars[var_name] - del self._private_theta[var_name] - self._private_vars['wm'] = emb_vars - self._private_theta['wm'] = emb_shards - - def EmbLookupDefaultTheta(self, ids): - return self.EmbLookup(self.theta, ids) - - def EmbLookup(self, theta, ids): - """Looks up embedding vectors for ids. - - Args: - theta: Named tuple with the weight matrix for the embedding. - ids: A rank-N int32 tensor. - - Returns: - A rank-(N+1) params.dtype tensor. - embs[indices, :] is the embedding vector for ids[indices]. - """ - p = self.params - ids = tf.convert_to_tensor(ids) - ids = py_utils.with_dependencies([ - py_utils.assert_between( - ids, 0, p.vocab_size, name='vocab_id_validation') - ], ids) - embs = tf.nn.embedding_lookup(theta.wm, tf.reshape(ids, [-1])) - if p.scale_sqrt_depth: - embs *= p.embedding_dim**0.5 - if p.vn.global_vn or p.vn.per_step_vn: - embs = py_utils.AddGlobalVN(p, embs) - out_shape = tf.concat([tf.shape(ids), [p.embedding_dim]], 0) - return tf.reshape(embs, out_shape) - - -class SimpleEmbeddingLayer(quant_utils.QuantizableLayer): - """An embedding layer that is simple to compile (by XLA and Toco). - - The params use_matmul and use_gather control how the lookup is performed. - If neither is True, then a loop is used to compute the embedding. - - This layer is "simple" in comparison to 'EmbeddingLayer' in that it does - not shard the embeddings. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('vocab_size', 0, - 'Depth of the input. I.e., the number of classes.') - p.Define('embedding_dim', 0, 'Depth of the output.') - p.Define( - 'use_matmul', False, 'If True, use a matmul to implement ' - 'the embedding lookup. Depending on vocab_size and #ids, ' - 'e.g., when vocab_size is small, use_matmul can be more ' - 'efficient. On the other hand, use_matmul creates a 0/1 ' - 'sparse matrix and hence may use more memory than the ' - 'final output.') - p.Define( - 'fprop_mode', None, 'Sets the mode used for computing the fprop ' - '(different inference engines have different capabilities and this ' - 'accomodates them). Can be "loop", "matmul" or "gather". If None, ' - 'defaults to "matmul" if use_matmul or "loop" if false.') - p.Define( - 'use_3d_weight_tensor', False, 'If True, and use_matmul is False,' - 'in TPU compatibility mode, we reshape the normal 2D weight' - 'tensor to [num_rows, embed_dim] to be ' - '[num_rows, embed_dim // 128, 128].') - p.Define('apply_pruning', False, - 'Whether to prune the weights while training') - p.Define( - 'scale_sqrt_depth', False, 'If set True, activations are scaled' - ' with sqrt(embedding_dim) in EmbLookup.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.vocab_size > 0 - assert symbolic.ToStatic(p.embedding_dim) > 0 - - valid_fprop_modes = ['loop', 'matmul', 'gather'] - self._fprop_mode = p.fprop_mode - if not self._fprop_mode: - self._fprop_mode = 'matmul' if p.use_matmul else 'gather' - assert self._fprop_mode in valid_fprop_modes, ( - 'fprop_mode must be one of %r' % valid_fprop_modes) - - def _FpropImpl(self, embs, ids_vec): - """The embedding lookup implementation.""" - p = self.params - emb_shape_suf, weight_shape = self._GetWeightShape() - - def EmbBprop(xs, ys, dys): - """Embedding backprop. - - Effectively, it computes: - num = size of xs.ids_vec - dembs = zeros_like(xs.embs) - for i in range(num): - dembs[xs.ids_vec[i], :] += dys[i, :] - return dembs, zeros_like(xs.ids_vec) - - Args: - xs: A NestedMap containing: - - embs: The embedding matrix. Unused in the backprop. - - ids_vec: A vector of int32 embedding ids. - ys: Required by py_utils._DefineDefun, not used here. - dys: A matrix of size (size of xs.ids_vec, embedding dims). - - Returns: - A NestedMap containing: - - - embs: A matrix of the same shape of xs.embs. Gradients for xs.embs. - - ids_vec: Zeros. Same shape as xs.ids_vec. - """ - del ys - num = tf.shape(xs.ids_vec)[0] - dembs = inplace_ops.empty(weight_shape, py_utils.FPropDtype(p), init=True) - if len(weight_shape) != 2: - dys_shape = tf.shape(dys) - dys = tf.reshape(dys, [dys_shape[0]] + emb_shape_suf) - - def EmbBpropLoop(i, state): - # row_id = state.ids_vec[i] - row_id = tf.gather(state.ids_vec, i) - # row = state.drets[i] - row = tf.reshape(tf.gather(state.drets, i), [1] + emb_shape_suf) - # state.dembs[row_id] = row - state.dembs = inplace_ops.alias_inplace_add(state.dembs, [row_id], row) - return state - - dembs = py_utils.ForLoop( - body=EmbBpropLoop, - start=0, - limit=num, - delta=1, - loop_state=py_utils.NestedMap( - ids_vec=xs.ids_vec, drets=dys, dembs=dembs)).dembs - - if p.scale_sqrt_depth: - dembs *= p.embedding_dim**0.5 - - return py_utils.NestedMap(embs=dembs, ids_vec=tf.zeros_like(ids_vec)) - - def EmbFprop(xs): - """Embedding forward prop. - - Effectively, it computes: - num = size of xs.ids_vec - rets = zeros([num, embedding dim]) - for i in range(num): - rets[i, :] = xs.embs[xs.ids_vec[i], :] - return rets - - Args: - xs: A NestedMap containing: - - embs: The embedding matrix. - - ids_vec: A vector of int32 embedding ids. - - Returns: - The result of embedding lookups. A matrix of shape - [num ids in xs.ids_vec, embedding dims]. - """ - num = tf.shape(xs.ids_vec)[0] - rets = inplace_ops.empty([num] + emb_shape_suf, py_utils.FPropDtype(p)) - - def EmbFpropLoop(i, state): - # row_id = state.ids_vec[i] - row_id = tf.gather(state.ids_vec, i) - # row = state.embs[row_id] - row = tf.reshape(tf.gather(state.embs, row_id), [1] + emb_shape_suf) - # state.rets[i] = row - state.rets = inplace_ops.alias_inplace_update(state.rets, [i], row) - return state - - rets = py_utils.ForLoop( - body=EmbFpropLoop, - start=0, - limit=num, - delta=1, - loop_state=py_utils.NestedMap( - embs=xs.embs, ids_vec=xs.ids_vec, rets=rets)).rets - if len(weight_shape) > 2: - rets = tf.reshape(rets, [num, symbolic.ToStatic(p.embedding_dim)]) - return rets - - def EmbMatmul(xs): - """Lookups embedding vectors by doing Matmul with one-hot vector.""" - # lhs[i, j] is True iff xs.ids_vec[i] == j. - lhs = tf.equal( - tf.expand_dims(xs.ids_vec, 1), - tf.range(p.vocab_size, dtype=xs.ids_vec.dtype)) - return tf.matmul(tf.cast(lhs, xs.embs.dtype), xs.embs) - - def EmbGather(xs): - """Lookups embedding vectors.""" - return tf.nn.embedding_lookup(xs.embs, xs.ids_vec) - - xs = py_utils.NestedMap(embs=embs, ids_vec=ids_vec) - if self._fprop_mode == 'matmul': - return py_utils.CallDefun(EmbMatmul, xs) - elif self._fprop_mode == 'loop': - return py_utils.CallDefun( - EmbFprop, xs, bak=EmbBprop, bak_as_function=True) - elif self._fprop_mode == 'gather': - return EmbGather(xs) - - def _GetWeightShape(self): - p = self.params - if py_utils.tpu_compat() and self._fprop_mode != 'matmul': - if p.use_3d_weight_tensor: - assert symbolic.ToStatic(p.embedding_dim) % 128 == 0 - emb_shape_suf = [symbolic.ToStatic(p.embedding_dim) // 128, 128] - else: - emb_shape_suf = [symbolic.ToStatic(p.embedding_dim)] - else: - emb_shape_suf = [symbolic.ToStatic(p.embedding_dim)] - weight_shape = [p.vocab_size] + emb_shape_suf - return emb_shape_suf, weight_shape - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - _, weight_shape = self._GetWeightShape() - - # Define weights - pc = py_utils.WeightParams( - shape=weight_shape, - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - - if p.apply_pruning: - mask_pc = py_utils.WeightParams(pc.shape, - py_utils.WeightInit.Constant(1.0), - p.dtype) - threshold_pc = py_utils.WeightParams([], - py_utils.WeightInit.Constant(0.0), - tf.float32) - self.CreateVariable('mask', mask_pc, theta_fn=None, trainable=False) - self.CreateVariable( - 'threshold', threshold_pc, theta_fn=None, trainable=False) - - def MaskWeightFn(weight): - return tf.multiply( - self.AddGlobalVN(weight), self.vars.mask, 'masked_weights') - - self.CreateVariable('wm', pc, theta_fn=MaskWeightFn) - pruning_utils.AddToPruningCollections(self.vars.wm, self.vars.mask, - self.vars.threshold) - else: - self.CreateVariable('wm', pc) - - def EmbLookupDefaultTheta(self, ids): - """Lookups embedding vectors for ids.""" - return self.FProp(self.theta, ids) - - def EmbLookup(self, theta, ids): - return self.FProp(theta, ids) - - def EmbLookupDefaultThetaOnCpu(self, ids): - """A faster path for CPU inference than the default gather.""" - p = self.params - embs = tf.nn.embedding_lookup(self.theta.wm, tf.reshape(ids, [-1])) - out_shape = tf.concat([tf.shape(ids), [symbolic.ToStatic(p.embedding_dim)]], - 0) - if p.scale_sqrt_depth: - embs *= p.embedding_dim**0.5 - return tf.reshape(embs, out_shape) - - def _FlatFProp(self, theta, ids): - """Lookups embedding vectors for ids. - - Args: - theta: Named tuple collection of weights for the layer. - ids: A rank-N int32 tensor. - - Returns: - A tuple of the flattened inputs to the embedding lookup, and a tensor that - is ready to be reshaped into the final shape in FProp. - """ - if not isinstance(ids, tf.Tensor): - tf.logging.warning('ids should be a tf.Tensor!') - ids = tf.convert_to_tensor(ids, tf.int32) - elif ids.dtype != tf.int32: - tf.logging.warning('ids should be tf.int32, but is %s!', ids.dtype) - ids = tf.cast(ids, tf.int32) - p = self.params - ids = py_utils.with_dependencies([ - py_utils.assert_between( - ids, 0, p.vocab_size, name='vocab_id_validation') - ], ids) - flat_ids = tf.reshape(ids, [-1]) - embs_result = self._FpropImpl(self.QWeight(theta.wm), flat_ids) - if p.vn.global_vn or p.vn.per_step_vn: - emb_noise = p.vn.scale * tf.random.normal( - tf.shape(embs_result), - stddev=1.0, - dtype=embs_result.dtype, - seed=p.random_seed) - embs_result += emb_noise - - if p.scale_sqrt_depth: - embs_result *= p.embedding_dim**0.5 - return flat_ids, embs_result - - def FProp(self, theta, ids): - """Lookups embedding vectors for ids. - - Args: - theta: Named tuple collection of weights for the layer. - ids: A rank-N int32 tensor. - - Returns: - A rank-(N+1) params.dtype tensor. - embs[indices, :] is the embedding vector for ids[indices]. - """ - _, embs_result = self._FlatFProp(theta, ids) - out_shape = tf.concat( - [tf.shape(ids), [symbolic.ToStatic(self.params.embedding_dim)]], 0) - return tf.reshape(embs_result, out_shape) - - -class OneHotEmbeddingLayer(base_layer.BaseLayer): - """Generates one-hot embeddings with uncertainties.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('vocab_size', 0, - 'Depth of the input. I.e., the number of classes.') - p.Define('embedding_dim', 0, 'Depth of the output.') - p.Define('uncertainty', 0.0, 'Uncertainty of the correct ID.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.vocab_size > 1 - assert p.embedding_dim == p.vocab_size - - def EmbLookupDefaultTheta(self, ids): - """Lookups embedding vectors for ids.""" - return self.FProp(self.theta, ids) - - def EmbLookup(self, theta, ids): - return self.FProp(theta, ids) - - def FProp(self, theta, ids): - """Lookups embedding vectors for ids. - - Args: - theta: Named tuple collection of weights for the layer. - ids: A rank-N int32 tensor. - - Returns: - A rank-(N+1) params.dtype tensor. - embs[indices, :] is the embedding vector for ids[indices]. - """ - del theta - p = self.params - ids = py_utils.with_dependencies([ - py_utils.assert_between( - ids, 0, p.vocab_size, name='vocab_id_validation') - ], ids) - low_confidence = p.uncertainty / tf.cast(p.vocab_size - 1, tf.float32) - high_confidence = 1.0 - p.uncertainty - embs_result = tf.one_hot( - ids, - depth=p.vocab_size, - on_value=high_confidence, - off_value=low_confidence) - if p.fprop_dtype is not None: - embs_result = tf.cast(embs_result, p.fprop_dtype) - return embs_result - - -class PositionalEmbeddingLayer(base_layer.BaseLayer): - """Generates sinusoidals with respect to the position in time and dimension. - - Implements the positional embedding layer from 'Attention is All You Need', - the Transformer Network. - - Code and comments are adapted from tensor2tensor/layers/common_attention.py - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'min_timescale', 1, 'Start of the geometric index.' - 'Determines the periodicity of the added signal.') - p.Define( - 'max_timescale', 10000, 'End of the geometric index. ' - 'Determines the frequency of the added signal.') - p.Define('embedding_dim', 0, 'Dimension of the embedding to be generated.') - p.Define( - 'trainable_scaling', False, - 'Introduces a trainable scaling parameter (a scalar) that' - ' multiplies the positional embedding in FProp.') - p.Define('trainable_scaling_init', 1.0, - 'Initial value of the scaling parameter.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.min_timescale - assert p.max_timescale - assert p.embedding_dim % 2 == 0 - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - if p.trainable_scaling: - pc = py_utils.WeightParams( - shape=[1], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('scale', pc) - - def _PosEmbeddingsFromPositions(self, theta, position): - """Generates the positional embeddings given the position tensor. - - Factors out the common code from FProp and FPropWithPosition. Returns - positional embeddings corresponding to the input position tensor. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - position: Position tensor of dtype float and shape [bs, seq_length] to - generate positional embeddings. - - Returns: - a Tensor of shape [bs, seq_length, embedding_dim]. - """ - p = self.params - seq_length = tf.shape(position)[1] - num_timescales = p.embedding_dim // 2 - log_timescale_increment = ( - math.log(float(p.max_timescale) / float(p.min_timescale)) / tf.maximum( - tf.cast(1.0, py_utils.FPropDtype(p)), - tf.cast(num_timescales, py_utils.FPropDtype(p)) - 1)) - - inv_timescales = p.min_timescale * tf.exp( - tf.cast(tf.range(num_timescales), py_utils.FPropDtype(p)) * - -log_timescale_increment) - - scaled_time = tf.expand_dims(position, 2) * tf.reshape( - inv_timescales, [1, 1, -1]) - - signal = tf.concat([tf.sin(scaled_time), tf.cos(scaled_time)], axis=2) - signal = tf.pad( - signal, [[0, 0], [0, 0], [0, tf.math.floormod(p.embedding_dim, -1)]]) - signal = tf.reshape(signal, [-1, seq_length, p.embedding_dim]) - if p.trainable_scaling: - signal *= (p.trainable_scaling_init + theta.scale) - return signal - - def FProp(self, theta, seq_length): - """Generates a Tensor of sinusoids with different frequencies. - - Each channel (dimension) of the generated positionanl embedding Tensor - corresponds to a sinusoid of different frequency and phase. - - This allows attention to learn to use absolute and relative positions. - Timing signals should be added to some precursors of both the query and the - memory inputs to attention. - - The use of relative position is possible because sin(x+y) and cos(x+y) can - be experessed in terms of y, sin(x) and cos(x). - - In particular, we use a geometric sequence of timescales starting with - min_timescale and ending with max_timescale. The number of different - timescales is equal to channels (dimension) / 2. For each timescale, we - generate the two sinusoidal signals sin(timestep/timescale) and - cos(timestep/timescale). All of these sinusoids are concatenated in - the channels dimension. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - seq_length: Sequence length of the embeddings to be generated - - Returns: - a Tensor of shape [seq_length, embedding_dim]. - """ - p = self.params - position = tf.reshape( - tf.cast(tf.range(seq_length), py_utils.FPropDtype(p)), [1, seq_length]) - pos_emb = self._PosEmbeddingsFromPositions(theta, position) - return tf.reshape(pos_emb, [seq_length, -1]) - - def FPropWithPosition(self, theta, position_tensor): - """Generates a Tensor of sinusoids with different frequencies. - - Uses the provided position tensor to generate positional embeddings. Refer - to FProp description for details of sinusoidal positional embeddings. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - position_tensor: Position tensor of shape [bs, seq_length] to generate - positional embeddings. - - Returns: - a Tensor of shape [bs, seq_length, embedding_dim]. - """ - position = tf.cast(position_tensor, py_utils.FPropDtype(self.params)) - return self._PosEmbeddingsFromPositions(theta, position) - - -class RelativePositionalEmbeddingLayer(base_layer.BaseLayer): - """Relative positional embedding. - - Section 3.2 of https://arxiv.org/pdf/1803.02155.pdf - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'radius', None, - 'Radius of the relative window size. Distance are clipped to ' - '[-radius, radius].') - p.Define('dim', None, 'Dimension of embedding.') - return p - - def __init__(self, params): - super().__init__(params) - params = self.params - if not isinstance(params.radius, numbers.Integral) or params.radius <= 0: - raise ValueError('params.radius must be a positive int, but is %s' % - params.radius) - if not isinstance(params.dim, numbers.Integral) or params.dim <= 0: - raise ValueError('params.dim must be a positive int, but is %s' % - params.radius) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - pc = py_utils.WeightParams( - shape=[2 * self.params.radius + 1, self.params.dim], - init=py_utils.WeightInit.Constant(0.0), - dtype=self.params.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('w', pc) - - def FProp(self, theta, relative_distance): - """Computes relative positional embedding. - - Args: - theta: A NestedMap of Tensors of layer weights. - relative_distance: A Tensor. - - Returns: - A Tensor of shape relative_distance.shape + [params.dim] - """ - params = self.params - clipped_indices = tf.clip_by_value(relative_distance, -params.radius, - params.radius) - # Right-shift indices to make them all non-negative. - calibrated_indices = clipped_indices + params.radius - return tf.gather_nd(theta.w, tf.expand_dims(calibrated_indices, -1)) - - -class SinusoidalPositionalEmbeddingLayer(base_layer.BaseLayer): - """Generates sinusoidals with respect to the position in time and dimension. - - Implements the a variant of the positional embedding layer from 'Attention is - All You Need', the Transformer Network that doesn't require tuning of the - max_timescale/min_timescale. See this blog post and Ron's colab. - https://kazemnejad.com/blog/transformer_architecture_positional_encoding - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('embedding_dim', 0, 'Dimension of the embedding to be generated.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.embedding_dim % 2 != 0: - raise ValueError('embedding_dim needs to be even.') - - def FProp(self, theta, seq_length): - """Generates a Tensor of sinusoids with different frequencies. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - seq_length: Sequence length of the embeddings to be generated - - Returns: - a Tensor of shape [seq_length, embedding_dim]. - """ - p = self.params - positions = tf.cast(tf.range(seq_length), py_utils.FPropDtype(p)) - num_timescales = p.embedding_dim // 2 - freq = tf.range( - 1, num_timescales + 1, - dtype=py_utils.FPropDtype(p)) * (2 * math.pi / seq_length) - scaled_pos = tf.matmul(positions[:, tf.newaxis], freq[tf.newaxis, :]) - sincos = tf.concat([tf.sin(scaled_pos), tf.cos(scaled_pos)], axis=-1) - return tf.reshape(sincos, [seq_length, -1]) - - -class SoftmaxLayer(quant_utils.QuantizableLayer): - """Base class for softmax layers.""" - - @classmethod - def Params(cls): - """Params for SoftmaxLayer.""" - p = super().Params() - p.Define('input_dim', 0, 'Dimension of the input.') - p.Define('num_classes', 0, 'Total number of target classes.') - p.Define( - 'logits_abs_max', None, 'If not None, logits are clipped to be within' - ' [-logits_abs_max, logits_abs_max]. This can be a scalar' - ' or a scalar tensor. Applies back pressure at training time; ignored' - ' for inference.') - p.Define( - 'chunk_size', 0, 'If non-zero, computes the per example ' - 'xent by small chunks along the batch dimension.') - return p - - def Logits(self, **unused): - """Returns the logits computed before the softmax.""" - raise NotImplementedError('GetLogits is not implemented.') - - def XentLossFromLogits(self, **unused): - """Returns the Xent loss from pre-computed logits.""" - raise NotImplementedError('XentLossFromLogits is not implemented.') - - def XentLoss(self, *args, **kwargs): - """Computes cross entropy.""" - return self.FProp(self.theta, *args, **kwargs) - - def _FProp2D(self, - theta, - inputs, - class_weights, - class_ids=None, - class_probabilities=None): - """Specialized FProp for matrix inputs.""" - raise NotImplementedError( - 'Subclasses of SoftmaxLayer must implement _FProp2D') - - def FProp(self, - theta, - inputs, - class_weights, - class_ids=None, - class_probabilities=None): - """Computes logit, cross entropy etc. - - This function can both work with class_ids, or probability distributions - over classes. Exactly one of class_ids or class_probabilities must be - provided. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: a list of a single tensor, or a single tensor with the shape [..., - input_dim]. - class_weights: a tensor with shape [...] containing the weights for each - target word. - class_ids: a tensor with shape [..., 1] of int32 dtype containing the - target class labels. - class_probabilities: a tensor with shape [..., num_classes] of float - values indicating class-membership probabilities. - - Returns: - A `.NestedMap` containing the following fields - - - logits: with shape [..., num_classes]. Unnormalized softmax's logits. - - per_example_argmax: with shape [...]. argmax of i-th example. - - per_example_xent: with shape [...]. Cross entropy between i-th example's - prediction and its label. - - per_example_weight: with shape [...]. class_weights casted to - this layer's dtype. - - total_xent: A scalar. The sum of per_example_weight * per_example_xent. - - total_weight: A scalar. The sum of per_example_weight. - - avg_xent: A scalar. total_loss / total_weight. - """ - p = self.params - - # Consolidate list/single value into a list. - if not isinstance(inputs, list): - inputs = [inputs] - - # If inputs are matrices already, delegate to _FProp2D. - if inputs[0].shape.ndims == 2: - return self._FProp2D(theta, inputs, class_weights, class_ids, - class_probabilities) - - # Remembers the original shape[1:-1]. - shape_mid = tf.shape(inputs[0])[1:-1] - - # Reshape inputs to matrices, labels to vectors, etc. - inputs = [ - tf.reshape(x, py_utils.ToStaticShape([-1, p.input_dim])) for x in inputs - ] - class_weights = tf.reshape(class_weights, [-1]) - if class_ids is not None: - class_ids = tf.reshape(class_ids, [-1, 1]) - if class_probabilities is not None: - class_probabilities = tf.reshape(class_probabilities, [-1, p.num_classes]) - - # Delegates to _FProp2D. - xent_loss = self._FProp2D(theta, inputs, class_weights, class_ids, - class_probabilities) - - # Reshapes xent_loss fields according to the inputs' shape. - xent_loss.logits = tf.reshape( - xent_loss.logits, tf.concat([[-1], shape_mid, [p.num_classes]], axis=0)) - per_example_shape = tf.concat([[-1], shape_mid], axis=0) - xent_loss.per_example_argmax = tf.reshape(xent_loss.per_example_argmax, - per_example_shape) - xent_loss.per_example_xent = tf.reshape(xent_loss.per_example_xent, - per_example_shape) - xent_loss.per_example_weight = tf.reshape(xent_loss.per_example_weight, - per_example_shape) - return xent_loss - - -class SimpleFullSoftmax(SoftmaxLayer): - """A somewhat simple softmax layer.""" - - @classmethod - def Params(cls): - """Params for SimpleFullSoftmax.""" - p = super().Params() - p.Define( - 'num_sampled', 0, 'Number of samples to use for the sampled soft-max. ' - 'Default value of 0 means no sampling is done; if set to > 0 then ' - 'training will use sampled soft-max when both chunk_size == 0 and ' - 'FProp is called with class_probabilities=None.') - p.Define( - 'num_shards', 1, - 'Number of shards to split params into. num_shards should' - ' divide num_classes.') - p.Define('apply_pruning', False, - 'Whether to prune the weights while training') - p.Define( - 'use_num_classes_major_weight', False, - 'Whether to use num_classes as major dimension for weight params. ' - 'This shows performance benefit especially when sharing embedding ' - 'and softmax. By removing the transpose before gather, it allows ' - 'better XLA fusions and optimizations.') - - p.Define( - 'use_bias', True, 'Whether or not to use a bias variable.' - 'Not using bias is not compatible with sampled softmax ' - '(num_sampled > 0).') - return p - - def __init__(self, params): - """Constructs a SimpleFullSoftmax layer.""" - super().__init__(params) - p = self.params - assert p.name - # We shard params across the class dimension. - assert p.num_classes % p.num_shards == 0 - if not p.use_bias: - assert p.num_sampled == 0, 'Sampled softmax requires bias.' - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - - num_classes_per_shard = p.num_classes // p.num_shards - # When using sampled soft-max we'd rather work with weights of - # shape=[num_classes_per_shard, p.input_dim] to avoid an expensive transpose - # op before computing the sampled_softmax_loss. - self._transpose_weight_params = False - weights_shard_shape = [p.input_dim, num_classes_per_shard] - if p.num_sampled or p.use_num_classes_major_weight: - self._transpose_weight_params = True - weights_shard_shape = [num_classes_per_shard, p.input_dim] - - pc = py_utils.WeightParams( - shape=weights_shard_shape, - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - - if p.apply_pruning: - mask_pc = py_utils.WeightParams(pc.shape, - py_utils.WeightInit.Constant(1.0), - p.dtype) - threshold_pc = py_utils.WeightParams([], - py_utils.WeightInit.Constant(0.0), - tf.float32) - - for i in range(p.num_shards): - weights_var_name = 'weight_%d' % i - if p.apply_pruning: - mask_var_name = 'mask_%d' % i - threshold_var_name = 'threshold_%d' % i - self.CreateVariable( - mask_var_name, mask_pc, theta_fn=None, trainable=False) - self.CreateVariable( - threshold_var_name, threshold_pc, theta_fn=None, trainable=False) - - def MaskWeightFn(weight): - return tf.multiply( - self.AddGlobalVN(weight), getattr(self.vars, mask_var_name), - 'masked_weights') - - self.CreateVariable(weights_var_name, pc, theta_fn=MaskWeightFn) - pruning_utils.AddToPruningCollections( - getattr(self.vars, weights_var_name), - getattr(self.vars, mask_var_name), - getattr(self.vars, threshold_var_name)) - - else: - self.CreateVariable(weights_var_name, pc, self.AddGlobalVN) - - pc = py_utils.WeightParams( - shape=[num_classes_per_shard], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - if p.use_bias: - for i in range(p.num_shards): - self.CreateVariable('bias_%d' % i, pc, self.AddGlobalVN) - - self.TrackQTensor('inputs', 'logits') - - def _GetInputs(self, inputs): - if isinstance(inputs, list): - assert len(inputs) == 1 - return inputs[0] - return inputs - - def _ConcatWeights(self, theta): - p = self.params - # Add per-step noise if configured so. - concat_axis = 1 - if self._transpose_weight_params: - concat_axis = 0 - weights = [ - self.QWeight(theta['weight_%d' % i]) for i in range(p.num_shards) - ] - new_theta = theta.copy() - if p.use_bias: - biases = [self.QWeight(theta['bias_%d' % i]) for i in range(p.num_shards)] - new_theta.bias = py_utils.AddPerStepVN(p, tf.concat(biases, axis=0)) - new_theta.wm = py_utils.AddPerStepVN(p, - tf.concat(weights, axis=concat_axis)) - return new_theta - - def _LogitsUsingConcatenatedWeightsHelper(self, theta, inputs): - p = self.params - inputs = self.QTensor('inputs', inputs) - wm = self.QWeight(theta.wm) - - if p.use_bias: - bias = self.QWeight(theta.bias) - - # x * w + b - # Note that theta.wm and theta.bias are transformed to concated/clipped - # by caller. - logits = tf.nn.bias_add( - py_utils.Matmul( - inputs, wm, transpose_b=self._transpose_weight_params), bias) - else: - logits = py_utils.Matmul( - inputs, wm, transpose_b=self._transpose_weight_params) - - # Clip logits by range. - # Note that this is generally not used in conjunction with quantization and - # shouldn't be needed at inference time as the quantized matmul above will - # take care of clipping naturally based on the data type and qparams. - abs_max = p.logits_abs_max - if abs_max is not None and not p.is_inference: - abs_min = -abs_max # pylint: disable=invalid-unary-operand-type - logits = py_utils.clip_by_value(logits, abs_min, abs_max) - return logits - - def _LogitsUsingConcatenatedWeights(self, theta, inputs): - logits = self._LogitsUsingConcatenatedWeightsHelper(theta, inputs) - return self.QTensor('logits', logits) - - def SimpleLogits(self, theta, inputs): - """Returns the simple logits computed before the softmax. - - Compared to the Logits function, this one has only weights, no bias for the - linear projection. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: A tensor with the shape [N, input_dim]. - - Returns: - logits: [N, num_classes] - """ - inputs = self.QTensor('inputs', inputs) - theta = self._ConcatWeights(theta) - wm = self.QWeight(theta.wm) - logits = py_utils.Matmul( - inputs, wm, transpose_b=self._transpose_weight_params) - - return self.QTensor('logits', logits) - - def Logits(self, theta, inputs): - """Returns the logits computed before the softmax. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: a list of a single tensor, or a single tensor with the shape [N, - input_dim]. - - Returns: - logits [batch, num_classes] - """ - return self._LogitsUsingConcatenatedWeights( - self._ConcatWeights(theta), self._GetInputs(inputs)) - - def _XentLossByChunk(self, theta, activation, class_ids): - """Computes per-example xent loss between activation and class_ids.""" - p = self.params - - # We reshape activation from a matrix to a 3-D tensor (a sequence - # of matrices), where the 2nd dimenion is p.chunk_size. Because - # the batch dimenion may not be multiple of p.chunk_size, we pad - # zeros. - activation = py_utils.HasRank(activation, 2) - batch, input_dim = tf.unstack(tf.shape(activation)) - dim0, dim1 = (batch + p.chunk_size - 1) // p.chunk_size, p.chunk_size - pad = dim0 * dim1 - batch - padded_activation = tf.concat( - [activation, - tf.zeros([pad, input_dim], dtype=activation.dtype)], - axis=0) - class_ids = py_utils.HasShape(class_ids, [batch, 1]) - padded_class_ids = tf.concat( - [class_ids, tf.zeros([pad, 1], dtype=class_ids.dtype)], axis=0) - - if py_utils.use_tpu(): - id_dtype = tf.int32 - else: - id_dtype = tf.int64 - padded_class_ids = tf.cast(padded_class_ids, id_dtype) - - # For each chunk, we compute logits of padded_activation[i, :, :], - # and its xent loss with padded_class_ids[i, :]. - def ChunkFn(theta, state0, inputs): - del state0 - activation, class_ids = inputs.activation, inputs.class_ids - logits = self._LogitsUsingConcatenatedWeights(theta, activation) - xent = tf.nn.sparse_softmax_cross_entropy_with_logits( - logits=logits, labels=class_ids) - amax = tf.stop_gradient(py_utils.ArgMax(logits)) - return py_utils.NestedMap(xent=xent, amax=amax), py_utils.NestedMap() - - acc, _ = recurrent.Recurrent( - theta=self._ConcatWeights(theta), - state0=py_utils.NestedMap( - xent=tf.zeros([p.chunk_size], dtype=p.dtype), - amax=tf.zeros([p.chunk_size], dtype=id_dtype)), - inputs=py_utils.NestedMap( - activation=tf.reshape(padded_activation, [dim0, dim1, input_dim]), - class_ids=tf.reshape(padded_class_ids, [dim0, dim1])), - cell_fn=ChunkFn) - - # acc.xent has the shape [dim0, dim1]. acc.xent[i, :] are - # per-example xent loss for examples in the i-th chunk. We - # reshape acc.xent to a vector and slice the first 'batch' values. - def GetBatch(x): - return tf.reshape(x, [-1])[:batch] - - return GetBatch(acc.xent), GetBatch(acc.amax) - - def _FProp2D(self, - theta, - inputs, - class_weights, - class_ids=None, - class_probabilities=None): - """Computes xent loss and log-prob logit.""" - p = self.params - inputs = self._GetInputs(inputs) - logits = self.Logits(theta, inputs) - if class_probabilities is not None: - per_example_xent, per_example_argmax = self.XentLossFromLogits( - theta, logits, class_weights, class_ids, class_probabilities) - elif p.chunk_size: - class_ids = py_utils.HasShape(class_ids, [-1, 1]) - per_example_xent, per_example_argmax = self._XentLossByChunk( - theta, inputs, class_ids) - elif p.num_sampled == 0 or self.do_eval: - per_example_xent, per_example_argmax = self.XentLossFromLogits( - theta, logits, class_weights, class_ids, class_probabilities) - else: # Use sampled soft-max in training mode with p.num_sampled set. - assert p.num_sampled > 0 - assert p.use_bias - tf.logging.vlog( - 0, 'Using sampled_softmax_loss(..., num_sampled=%d, ' - 'num_classes=%d) in SimpleFullSoftmax::_FProp2D', p.num_sampled, - p.num_classes) - per_example_xent = tf.nn.sampled_softmax_loss( - weights=[theta['weight_%d' % i] for i in range(p.num_shards)], - biases=tf.concat([theta['bias_%d' % i] for i in range(p.num_shards)], - axis=0), - labels=tf.reshape(class_ids, [-1, 1]), - inputs=self._GetInputs(inputs), - num_sampled=p.num_sampled, - num_classes=p.num_classes, - seed=p.random_seed) - # Avoid computing logits; per_example_argmax is going to be always right. - per_example_argmax = tf.identity(class_ids) - - label_weights = tf.reshape( - tf.cast(class_weights, py_utils.FPropDtype(p)), [-1]) - total_xent = tf.reduce_sum(per_example_xent * label_weights) - total_weights = tf.reduce_sum(label_weights) - return py_utils.NestedMap( - logits=logits, - log_probs=tf.nn.log_softmax(logits), - per_example_argmax=per_example_argmax, - per_example_xent=per_example_xent, - per_example_weight=label_weights, - total_xent=total_xent, - total_weight=total_weights, - avg_xent=total_xent / total_weights) - - def XentLossFromLogits(self, - theta, - logits, - class_weights, - class_ids=None, - class_probabilities=None): - """Computes cross-entropy, argmax etc. from logits.""" - p = self.params - assert logits is not None - if class_probabilities is not None: - per_example_xent = tf.nn.softmax_cross_entropy_with_logits( - labels=class_probabilities, logits=logits) - per_example_argmax = py_utils.ArgMax(logits) - elif p.num_sampled == 0 or self.do_eval: - assert class_ids is not None - tf.logging.vlog( - 0, 'Using sparse_softmax_cross_entropy_with_logits() in ' - 'SimpleFullSoftmax::_FProp2D logits_shape=%r', - py_utils.GetShape(logits)) - per_example_xent = tf.nn.sparse_softmax_cross_entropy_with_logits( - labels=tf.reshape(class_ids, [-1]), logits=logits) - per_example_argmax = py_utils.ArgMax(logits) - else: - raise ValueError( - 'This set of arguments is not supported for XentLossFromLogits.') - return per_example_xent, per_example_argmax - - -class SharedSoftmaxLayer(SimpleFullSoftmax): - """Shared softmax layer for decoder embedding/softmax matrix.""" - - @classmethod - def Params(cls): - """Params for SharedSoftmaxLayer.""" - p = super().Params() - p.Define( - 'scale_sqrt_depth', False, 'If set True, activations are scaled' - ' with sqrt(input_dim) in EmbLookup.') - p.Define( - 'embedding_dim', 0, 'Set to be compatible with embedding layer, ' - ' and it is equivalent to input_dim') - p.Define( - 'vocab_size', 0, 'Set to be compatible with embedding layer, and ' - 'it is equivalent to num_classes') - return p - - def EmbLookup(self, theta, ids): - p = self.params - ids = py_utils.with_dependencies([ - py_utils.assert_between( - ids, - 0, - p.num_classes, - summarize=100000, - message='{}:class_id_validation'.format(p.cls)) - ], ids) - - wm = self._ConcatWeights(theta).wm - if not self._transpose_weight_params: - wm = tf.transpose(wm) - embs_result = tf.gather(wm, ids) - - if p.scale_sqrt_depth: - embs_result *= p.input_dim**0.5 - - return embs_result - - -class SingleShardFullSoftmax(SoftmaxLayer): - """Full softmax layer.""" - - def __init__(self, params): - """Constructs a SingleShardFullSoftmax layer.""" - super().__init__(params) - p = self.params - assert p.name - linear_p = builder_layers.LinearLayer.Params().Set( - name='linear', input_dims=p.input_dim, output_dims=p.num_classes) - self.CreateChild('linear', linear_p) - bias_p = builder_layers.BiasLayer.Params().Set( - name='bias', dims=p.num_classes) - self.CreateChild('bias', bias_p) - - def Logits(self, theta, inputs): - """Returns the logits computed before the softmax. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: A single tensor with shape [..., input_dim]. - - Returns: - logits [..., num_classes] - """ - p = self.params - if isinstance(inputs, (list, tuple)): - assert len(inputs) == 1 - inputs = inputs[0] - after_proj = self.linear.FProp(theta.linear, inputs) - logits = self.bias.FProp(theta.bias, after_proj) - # Clip logits by range. - # Note that this is generally not used in conjunction with quantization and - # shouldn't be needed at inference time as the quantized matmul above will - # take care of clipping naturally based on the data type and qparams. - abs_max = p.logits_abs_max - if abs_max is not None and not p.is_inference: - abs_min = -abs_max # pylint: disable=invalid-unary-operand-type - logits = py_utils.clip_by_value(logits, abs_min, abs_max) - return logits - - def XentLossFromLogits(self, - theta, - logits, - class_ids=None, - class_probabilities=None): - """Computes cross-entropy, argmax etc. from logits.""" - assert logits is not None - if class_probabilities is not None: - per_example_xent = tf.nn.softmax_cross_entropy_with_logits( - labels=class_probabilities, logits=logits) - per_example_argmax = tf.stop_gradient(py_utils.ArgMax(logits)) - else: - assert class_ids is not None - per_example_xent = tf.nn.sparse_softmax_cross_entropy_with_logits( - labels=class_ids, logits=logits) - per_example_argmax = tf.stop_gradient(py_utils.ArgMax(logits)) - return per_example_xent, per_example_argmax - - def XentLossByChunk(self, theta, activation, class_ids, class_probabilities): - """Computes per-example xent loss.""" - p = self.params - - act_orig_shape = tf.shape(activation) - batch_size = act_orig_shape[0] - chunk_size = p.chunk_size - num_chunks = batch_size // chunk_size - - num_chunks = py_utils.with_dependencies([ - py_utils.assert_equal( - 0, - tf.math.floormod(batch_size, chunk_size), - summarize=2, - message='assert_equal') - ], num_chunks) - - def ReshapeX(x): - if x is None: - return None - x_shape = tf.shape(x) - new_shape = tf.concat([[num_chunks, chunk_size], x_shape[1:]], 0) - return tf.reshape(x, new_shape) - - activation = ReshapeX(activation) - class_ids = ReshapeX(class_ids) - class_probabilities = ReshapeX(class_probabilities) - - # For each chunk, we compute logits of activation[i, :, :], - # and its xent loss with class_ids[i, :]. - def ChunkFn(theta, state0, inputs): - del state0 - activation = inputs.activation - class_ids = inputs.get('class_ids', None) - class_probabilities = inputs.get('class_probabilities', None) - logits = self.Logits(theta, activation) - per_example_xent, per_example_argmax = self.XentLossFromLogits( - theta, logits, class_ids, class_probabilities) - return py_utils.NestedMap( - xent=per_example_xent, amax=per_example_argmax), py_utils.NestedMap() - - inputs_nmap = py_utils.NestedMap(activation=activation) - if class_ids is not None: - inputs_nmap.class_ids = class_ids - if class_probabilities is not None: - inputs_nmap.class_probabilities = class_probabilities - - xent_state0 = tf.zeros(tf.shape(activation)[1:-1], dtype=p.dtype) - argmax_out_dtype = tf.int32 if py_utils.use_tpu() else tf.int64 - amax_state0 = tf.zeros(tf.shape(activation)[1:-1], dtype=argmax_out_dtype) - - acc, _ = recurrent.Recurrent( - theta=theta, - state0=py_utils.NestedMap(xent=xent_state0, amax=amax_state0), - inputs=inputs_nmap, - cell_fn=ChunkFn) - - # acc.xent has the shape [dim0, dim1]. acc.xent[i, :] are - # per-example xent loss for examples in the i-th chunk. We - # reshape acc.xent to a vector and slice the first 'batch' values. - def GetBatch(x): - return tf.reshape(x, act_orig_shape[:-1]) - - return GetBatch(acc.xent), GetBatch(acc.amax) - - def FProp(self, - theta, - inputs, - class_weights, - class_ids=None, - class_probabilities=None): - """Computes logits, cross entropy etc. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: a single tensor with shape [..., input_dim]. - class_weights: a tensor with shape [..., 1] containing the weights for - each target word. - class_ids: a tensor with shape [..., 1] of int32 dtype containing the - target class labels. - class_probabilities: a tensor with shape [..., num_classes] of float - values indicating class-membership probabilities. - - Returns: - A `.NestedMap` containing the following fields - - - logits: with shape [..., num_classes]. Unnormalized softmax's logits. - - per_example_argmax: with shape [...]. argmax of i-th example. - - per_example_xent: with shape [...]. Cross entropy between i-th example's - prediction and its label. - - per_example_weight: with shape [...]. class_weights casted to - this layer's dtype. - - total_xent: A scalar. The sum of per_example_weight * per_example_xent. - - total_weight: A scalar. The sum of per_example_weight. - - avg_xent: A scalar. total_loss / total_weight. - """ - p = self.params - if isinstance(inputs, (list, tuple)): - assert len(inputs) == 1 - inputs = inputs[0] - - inputs_shape = tf.shape(inputs) - ids_shape = tf.concat([inputs_shape[:-1], [1]], 0) - probs_shape = tf.concat([inputs_shape[:-1], [p.num_classes]], 0) - - class_weights = py_utils.HasShape(class_weights, ids_shape) - class_weights = tf.squeeze(class_weights, -1) - if class_ids is not None: - class_ids = py_utils.HasShape(class_ids, ids_shape) - class_ids = tf.squeeze(class_ids, -1) - if class_probabilities is not None: - class_probabilities = py_utils.HasShape(class_probabilities, probs_shape) - - if (not self.do_eval) and (p.chunk_size > 0): - # Chunking. - logits = None - log_probs = None - per_example_xent, per_example_argmax = self.XentLossByChunk( - theta, inputs, class_ids, class_probabilities) - else: - logits = self.Logits(theta, inputs) - log_probs = tf.nn.log_softmax(logits) - per_example_xent, per_example_argmax = self.XentLossFromLogits( - theta, logits, class_ids, class_probabilities) - - label_weights = tf.cast(class_weights, py_utils.FPropDtype(p)) - total_xent = tf.reduce_sum(per_example_xent * label_weights) - total_weights = tf.reduce_sum(label_weights) - output_nmap = py_utils.NestedMap( - per_example_argmax=per_example_argmax, - per_example_xent=per_example_xent, - per_example_weight=label_weights, - total_xent=total_xent, - total_weight=total_weights, - avg_xent=total_xent / (total_weights + 1e-6)) - if logits is not None: - output_nmap.logits = logits - output_nmap.log_probs = log_probs - return output_nmap - - -class SingleShardSharedEmbeddingSoftmax(SingleShardFullSoftmax): - """A shared softmax/embedding layer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('vocab_size', 0, 'Num tokens in vocab.') - p.Define('embedding_dim', 0, 'Depth of the output.') - p.Define( - 'scale_sqrt_depth', False, 'If set True, activations are scaled' - ' with sqrt(embedding_dim) in EmbLookup.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.vocab_size == p.num_classes - assert p.embedding_dim == p.input_dim - - def EmbLookupDefaultTheta(self, ids): - return self.EmbLookup(self.theta, ids) - - def EmbLookup(self, theta, ids): - """Looks up embedding vectors for ids. - - Args: - theta: Named tuple with the weight matrix for the embedding. - ids: A rank-N int32 tensor. - - Returns: - A rank-(N+1) params.dtype tensor. - embs[indices, :] is the embedding vector for ids[indices]. - """ - p = self.params - ids = tf.convert_to_tensor(ids) - ids = py_utils.with_dependencies([ - py_utils.assert_between( - ids, 0, p.vocab_size, name='vocab_id_validation') - ], ids) - # TODO(yonghui): Get rid of this extra copy (tf.transpose). - emb_vars = tf.transpose(theta.linear.w) - embs = tf.nn.embedding_lookup(emb_vars, tf.reshape(ids, [-1])) - if p.scale_sqrt_depth: - embs *= p.embedding_dim**0.5 - if p.vn.global_vn or p.vn.per_step_vn: - embs = py_utils.AddGlobalVN(p, embs) - out_shape = tf.concat([tf.shape(ids), [p.embedding_dim]], 0) - return tf.reshape(embs, out_shape) - - -class ConvSoftmax(quant_utils.QuantizableLayer): - """A softmax implementation based on 1x1 convolution. - - On TPU this is much more memory efficient than MatMul after reshaping logits - to a matrix. - """ - - @classmethod - def Params(cls): - """Params for SoftmaxLayer.""" - p = super().Params() - p.Define('input_dim', 0, 'Dimension of the input.') - p.Define('hidden_dim', 0, 'Dimension of the hidden layer.') - p.Define('num_classes', 0, 'Total number of target classes.') - return p - - def _CreateLayerVariables(self): - """Constructs a SimpleFullSoftmax layer.""" - super()._CreateLayerVariables() - p = self.params - if p.hidden_dim: - w_proj_pc = py_utils.WeightParams( - shape=(1, p.input_dim, p.hidden_dim), - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('w_proj', w_proj_pc) - w_pc = py_utils.WeightParams( - shape=(1, p.hidden_dim or p.input_dim, p.num_classes), - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('w', w_pc) - self.CreateVariable( - 'b', - py_utils.WeightParams( - shape=[p.num_classes], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars'])) - - def Logits(self, theta, inputs): - p = self.params - with tf.name_scope(p.name): - if inputs.shape.ndims == 2: - # [batch, time, depth] - x = inputs[:, tf.newaxis, :] - else: - x = py_utils.HasShape(inputs, [-1, -1, -1]) - if p.hidden_dim: - x = tf.nn.conv1d(x, theta.w_proj, 1, 'VALID') - logits = tf.nn.bias_add(tf.nn.conv1d(x, theta.w, 1, 'VALID'), theta.b) - if inputs.shape.ndims == 2: - return logits[:, 0, :] - else: - return logits - - -class DropoutLayer(base_layer.BaseLayer): - """Apply dropout during trainig.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('keep_prob', 1.0, 'Keep probability.') - # noise_shape is unknown when building layer params. - p.Define( - 'noise_shape', None, 'A 1-D `Tensor` of type `int32`, representing' - ' the shape for randomly generated keep/drop flags.') - p.Define( - 'noise_shape_broadcast_dims', None, - 'A list of dimension where the noise shape is broadcasted. For ' - 'example, noise_shape = [n, h, w, 1] when ' - 'noise_shape_broadcast_dims=[-1] ') - # We typically want to replace dropout by expectation during eval. - # However, in certain cases E(f(x)) != f(E(x)), and replacing dropout by its - # expectation during eval leads to worse quality. - p.Define('dropout_at_eval', False, - 'Whether or not to also perform dropout at eval time.') - return p - - def _Dropout(self, theta, inputs, noise_shape): - return tf.nn.dropout( - inputs, - rate=1 - self.params.keep_prob, - noise_shape=noise_shape, - seed=self.params.random_seed) - - @classmethod - def NumOutputNodes(cls, p): - # The layer does element-wise processing thus is input-shape agnostic. - return - - def FProp(self, theta, inputs): - """Apply dropout to inputs. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: The inputs tensor. - - Returns: - inputs with dropout applied at training time. - """ - p = self.params - if not self.do_eval or p.dropout_at_eval: - if isinstance(p.keep_prob, numbers.Real) and p.keep_prob == 1.0: - return inputs - if p.noise_shape_broadcast_dims: - noise_shape = p.noise_shape or py_utils.GetShape(inputs) - for dim in p.noise_shape_broadcast_dims: - if dim >= len(noise_shape): - raise ValueError('Invalid broadcasted dim {}'.format(dim)) - noise_shape[dim] = 1 - else: - noise_shape = p.noise_shape - ret = self._Dropout(theta, inputs, noise_shape) - ret.set_shape(inputs.get_shape()) - return ret - else: - return inputs - - @classmethod - def FPropMeta(cls, p, inputs, *args): - py_utils.CheckShapes((inputs,)) - flops_per_element = 10 # Approximately 10 flops per element. - return py_utils.NestedMap( - flops=inputs.num_elements() * flops_per_element, out_shapes=(inputs,)) - - -class DeterministicDropoutLayer(DropoutLayer): - """Apply dropout during trainig.""" - - def _Dropout(self, theta, inputs, noise_shape): - return py_utils.DeterministicDropout( - inputs, - keep_prob=self.params.keep_prob, - seeds=py_utils.GenerateStepSeedPair(self.params, theta.global_step), - noise_shape=noise_shape) - - -class LayerNorm(base_layer.BaseLayer): - """Layer normalization. - - Implements layer normalization: - https://arxiv.org/abs/1607.06450 - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', 0, 'Depth of the input to the network.') - p.Define('epsilon', 1e-6, 'Tiny value to guard rsqrt.') - p.Define('use_fused_layernorm', False, 'Whether to use fused layernorm.') - p.Define( - 'direct_scale', False, 'Whether to apply scale directly ' - 'without a +1.0. Var is initialized to 1.0 instead. This makes ' - 'the layer weight-compatible with the implementation in ' - 'contrib.layers.') - - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.input_dim > 0 - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - pc = py_utils.WeightParams( - shape=[p.input_dim], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars'] + - [py_utils.SKIP_LP_REGULARIZATION]) - self.CreateVariable('bias', pc) - - if p.direct_scale: - scale_pc = py_utils.WeightParams( - shape=[p.input_dim], - init=py_utils.WeightInit.Constant(1.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars'] + - [py_utils.SKIP_LP_REGULARIZATION]) - else: - scale_pc = pc - self.CreateVariable('scale', scale_pc) - - def _GetScaleAndBias(self, theta): - return theta.scale, theta.bias - - def FProp(self, theta, inputs): - """Applies normalization over the last dimension (layer). - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: A tensor of shape [..., hidden_dim]. - - Returns: - tensor of the same shape with inputs - """ - p = self.params - inputs = py_utils.with_dependencies( - [py_utils.assert_equal(tf.shape(inputs)[-1], p.input_dim)], inputs) - - cur_scale, cur_bias = self._GetScaleAndBias(theta) - - if p.direct_scale: - scale = cur_scale - else: - scale = 1.0 + cur_scale - - if p.use_fused_layernorm: - counts, means_ss, variance_ss, _, = tf.nn.sufficient_statistics( - inputs, axes=[-1], keepdims=True) - mean, variance = tf.nn.normalize_moments(counts, means_ss, variance_ss, - None) - inputs_norm = (inputs - mean) * tf.math.rsqrt(variance + p.epsilon) - return inputs_norm * scale + cur_bias - - def Normalize(xs): - """Normalize `xs.x` w/ `xs.scale` and `xs.bias` gain/shift.""" - x_shape = py_utils.GetShape(xs.x) - inner_dim = x_shape[-1] - x_reshaped = tf.reshape(xs.x, [-1, inner_dim]) - mean = tf.reduce_mean(x_reshaped, axis=[1], keepdims=True) - variance = tf.reduce_mean( - tf.square(x_reshaped - mean), axis=[1], keepdims=True) - x_norm = (x_reshaped - mean) * tf.math.rsqrt(variance + p.epsilon) - x_norm = tf.reshape(x_norm, x_shape) - return x_norm * xs.scale + xs.bias - - return py_utils.CallDefun( - Normalize, py_utils.NestedMap(x=inputs, scale=scale, bias=cur_bias)) - - @classmethod - def NumOutputNodes(cls, p): - return p.input_dim - - @classmethod - def FPropMeta(cls, p, inputs): - py_utils.CheckShapes((inputs,)) - return py_utils.NestedMap( - flops=inputs.num_elements() * 10, out_shapes=(inputs,)) - - -class CategoricalLayerNorm(LayerNorm): - """Categorical layer normalization. - - Allow dynamic switch of normalization params based on given class_index. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_classes', 1, - 'Number of privatized copies of layer norm params.') - return p - - def _BiasVarName(self, i): - return 'bias_' + str(i) - - def _ScaleVarName(self, i): - return 'scale_' + str(i) - - def _CreateLayerVariables(self): - # Skip LayerNorm's _CreateLayerVariables() as bias and scale variables will - # be created in this function. - super(LayerNorm, self)._CreateLayerVariables() # pylint: disable=bad-super-call - p = self.params - pc = py_utils.WeightParams( - shape=[self.params.input_dim], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars'] + - [py_utils.SKIP_LP_REGULARIZATION]) - for i in range(p.num_classes): - self.CreateVariable(self._BiasVarName(i), pc) - self.CreateVariable(self._ScaleVarName(i), pc) - - def __init__(self, params): - super().__init__(params) - p = self.params - assert isinstance(p.num_classes, int) - assert p.num_classes > 0 - self.AddExtraTheta('class_index', tf.constant(0, dtype=tf.int32)) - - def _GetScaleAndBias(self, theta): - p = self.params - with tf.control_dependencies( - [py_utils.assert_between(theta.class_index, 0, p.num_classes)]): - biases = [theta[self._BiasVarName(i)] for i in range(p.num_classes)] - cur_bias = tf.gather(biases, theta.class_index) - scales = [theta[self._ScaleVarName(i)] for i in range(p.num_classes)] - cur_scale = tf.gather(scales, theta.class_index) - return cur_scale, cur_bias - - -class ConvSetLayer(quant_utils.QuantizableLayer): - """Set of Convolutions with different filter sizes in a single layer. - - Applies a set of convolutions with different filter shapes to the inputs and - returns the concatenated outputs. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('cnn_tpl', - ConvLayer.Params().Set(filter_stride=(1, 1)), - 'Conv layer template for the set of conv layers.') - p.Define( - 'filter_shapes', [(0, 0, 0, 0)], - 'Must be a list of sequences of 4. Elements are in order of height' - ' (time), width (frequency), in_channel, out_channel') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - - filter_set = set() - input_shape = None - # Asserting kernel sizes are different and input sizes are the same. - for filter_shape in p.filter_shapes: - key = '%d_%d' % (filter_shape[0], filter_shape[1]) - assert key not in filter_set - filter_set.add(key) - if input_shape is None: - input_shape = filter_shape[2] - assert input_shape == filter_shape[2] - - params_conv_set = [] - for filter_shape in p.filter_shapes: - conv_p = p.cnn_tpl.Copy() - conv_p.name = '%d_%d' % (filter_shape[0], filter_shape[1]) - # Important: combined quantization will be done pre-concat versus - # by each layer on its output. Otherwise, inherit quantization params - # from this layer. - if p.qdomain.default is not None: - conv_p.qdomain.default = p.qdomain.default.Copy() - conv_p.disable_activation_quantization = True - conv_p.filter_shape = filter_shape - params_conv_set.append(conv_p) - self.CreateChildren('conv_set', params_conv_set) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - # The same QTensor is used for all inputs to the concat. - self.TrackQTensor('activation') - - def FProp(self, theta, inputs, paddings): - """Apply all convolution sets to inputs and concatenate outputs. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: The inputs tensor. It is expected to be of shape [batch, time, - frequency, channel]. The time dimension corresponds to the height - dimension as in images and the frequency dimension corresponds to the - width dimension as in images. - paddings: The paddings tensor. It is expected to be of shape [batch, - time]. - - Returns: - A tuple (out, output_paddings). - - - out: output tensor. Expected to be of shape [batch, time_mod, - frequency_mod, out_channel_1 + out_channel_2 ...] where time_mod and - frequency_mod depend on the conv layer strides and out_channel_i is - the output channel size of the i-th conv layer in the set. - - output_paddings: Modified paddings generated within `ConvLayer.FProp`. - Expected to be of the shape [batch, time_mod]. - """ - p = self.params - inputs = py_utils.with_dependencies([ - py_utils.assert_shape_match(tf.shape(paddings), [-1, -1]), - py_utils.assert_shape_match( - tf.shape(inputs), - tf.concat([tf.shape(paddings), [-1, p.filter_shapes[0][2]]], 0)) - ], inputs) - - conv_outputs = [] - output_paddings = None - # output_padding should be same for all filters for the same stride. - for i, conv_i in enumerate(self.conv_set): - conv_i_output, conv_i_padding = conv_i.FProp(theta.conv_set[i], inputs, - paddings) - if output_paddings is None: - output_paddings = conv_i_padding - conv_outputs.append(conv_i_output) - - # Track for quantization. - conv_outputs = [self.QTensor('activation', t) for t in conv_outputs] - - out = tf.concat(conv_outputs, -1) - return out, output_paddings - - -class LocalizedLabelSmoother(base_layer.BaseLayer): - """Smooths labels given as class ids. - - Implements the smoothing from https://arxiv.org/abs/1612.02695. Instead of - 1-hot class ids the model is trained to predict a distribution over classes - that includes the correct class label and with a small probability the labels - of tokens that appear nearby in time in the ground truth. This typically acts - as a strong regularizer. - - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_classes', 0, 'Number of classes') - p.Define( - 'offsets', [], 'Offset (over time) for smoothing. At time T the ' - 'smoothed target is class[T] + sum_i weights[i]*class[T+offset[i]]') - p.Define('weights', [], 'Weight of the smoothing at corresponding offset') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.num_classes > 0 - assert len(p.offsets) == len(p.weights) - assert p.name - - def FProp(self, theta, target_paddings, target_labels, target_ids): - """Convert class_ids to 1hot and smooth by neighborhood. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - target_paddings: float32 matrix [bs, seq_len] - target_labels: int32 matrix [bs, seq_len]. This stores the target label - output at each decoder step as generated by the speech input generator - input_batch.tgt.labels - target_ids: int32 matrix [bs, seq_len]. This stores the target_id that is - fed to the decoder, as generated by the speech input generator - input_batch.tgt.ids - - Returns: - A tensor [bs, seq_len, num_classes] denoting a smoothed distribution over - num_classes. - """ - del target_ids # Unused. - p = self.params - class_probabilities = tf.one_hot( - target_labels, p.num_classes, dtype=py_utils.FPropDtype(p)) - - # Start list keeping the scaled class-probabilities at different offsets. - output_distributions = [class_probabilities] - seq_len = tf.shape(class_probabilities)[1] - # If offsets < 0 we force a future output_act to be like a past token. - # If offsets > 0 we force a past output_act to be like a future token. - min_offset = np.min(p.offsets + [0]) - max_offset = np.max(p.offsets + [0]) - class_probabilities = tf.pad(class_probabilities, - [[0, 0], [-min_offset, max_offset], [0, 0]]) - # Shift the weights to the left by one location - we don't make the - # EOS more probable. - class_weights = tf.pad(1.0 - target_paddings[:, 1:], - [[0, 0], [-min_offset, max_offset + 1]]) - class_weights = tf.expand_dims(class_weights, 2) - - for offset, weight in zip(p.offsets, p.weights): - offset_in_padded = offset - min_offset - output_distributions.append( - class_probabilities[:, offset_in_padded:offset_in_padded + seq_len, :] - * class_weights[:, offset_in_padded:offset_in_padded + seq_len, :] * - weight) - output_distributions = tf.add_n(output_distributions) - output_distributions /= tf.reduce_sum( - output_distributions, axis=-1, keepdims=True) - return output_distributions - - -class UniformLabelSmoother(base_layer.BaseLayer): - """Smooths labels given as class ids and confidence. - - Implements the smoothing from https://arxiv.org/abs/1512.00567. Correct class - label confidence is dropped by eps and all the other classes are increased - by eps/num_classes. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_classes', 0, 'Number of classes') - p.Define('uncertainty', 0.1, 'Uncertainty of correct label, eps.') - p.Define( - 'uncertainty_larger', 0.1, - 'Apply a larger uncertainty to specific tokens, as specified ' - 'by token_from_target_ids.') - p.Define('token_id_uncertainty_larger', None, 'Id of token from target_ids ' - 'to apply uncertainty_larger to.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.num_classes > 0 - assert 0.0 <= p.uncertainty < 1.0 - assert p.token_id_uncertainty_larger is None or ( - p.token_id_uncertainty_larger >= 0) - assert p.name - - def FProp(self, theta, target_paddings, target_labels, target_ids): - """Convert target_labels to 1hot and smooth uniformly. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - target_paddings: float32 matrix [bs, seq_len] - target_labels: int32 matrix [bs, seq_len]. This stores the target label - output at each decoder step as generated by the speech input generator - input_batch.tgt.labels - target_ids: int32 matrix [bs, seq_len]. This stores the target_id that is - fed to the decoder, as generated by the speech input generator - input_batch.tgt.ids - - Returns: - A tensor of float32 [bs, seq_len, num_classes] denoting a smoothed - distribution over num_classes. - """ - del target_paddings # Unused by FProp. - p = self.params - - low_confidence = p.uncertainty / tf.cast(p.num_classes - 1, tf.float32) - high_confidence = (1.0 - p.uncertainty) - - smooth_targets = tf.one_hot( - tf.cast(target_labels, tf.int32), - depth=p.num_classes, - on_value=high_confidence, - off_value=low_confidence) - if p.token_id_uncertainty_larger is not None: - assert target_ids is not None - low_confidence_larger = p.uncertainty_larger / tf.cast( - p.num_classes - 1, tf.float32) - high_confidence_larger = (1.0 - p.uncertainty_larger) - smooth_targets_larger = tf.one_hot( - tf.cast(target_labels, tf.int32), - depth=p.num_classes, - on_value=high_confidence_larger, - off_value=low_confidence_larger) - should_smooth_larger = tf.tile( - tf.expand_dims( - tf.equal(target_ids, p.token_id_uncertainty_larger), -1), - multiples=[1, 1, p.num_classes]) - smooth_targets = tf.where(should_smooth_larger, smooth_targets_larger, - smooth_targets) - return smooth_targets - - -class HighwaySkipLayer(base_layer.BaseLayer): - """A highway skip layer. - - This class represents a highway skip layer, which takes multiple - inputs (from different layers of the network) and gates them. - This returns C(x)x + T(x)h, initially biasing C to be open. - For some discussion about initialization please see: - Section 2.2 in [Srivastava, 2015]: https://arxiv.org/pdf/1505.00387v2.pdf - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', 0, 'Dimension of the input to the network.') - p.Define( - 'batch_norm', False, - 'Whether or not to apply BN to the highway skip layer output. ' - 'Note this is only a single bool.') - p.Define('carry_bias_init', 1.0, 'carry gates bias initialization') - p.Define('couple_carry_transform_gates', False, - 'Boolean on whether to couple the transform and carry gates.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - carry_gate_params = ProjectionLayer.Params().Set( - batch_norm=p.batch_norm, - has_bias=True, - activation='SIGMOID', - input_dim=p.input_dim, - output_dim=p.input_dim, - bias_init=p.carry_bias_init, - name='%s_carry_gate' % p.name) - self.CreateChild('carry_gate', carry_gate_params) - - if not p.couple_carry_transform_gates: - transform_gate_params = ProjectionLayer.Params().Set( - batch_norm=p.batch_norm, - has_bias=True, - activation='SIGMOID', - input_dim=p.input_dim, - output_dim=p.input_dim, - bias_init=-p.carry_bias_init, - name='%s_transform_gate' % p.name) - self.CreateChild('transform_gate', transform_gate_params) - - def FProp(self, theta, x, transformed_x, paddings=None): - """Fprop for Highway Skip layer. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - x: feature at the lower layer. - transformed_x: transformation of x at a higher layer. - paddings: padding applied to the features. - - Returns: - layer_out - activations after forward propagation. - """ - p = self.params - assert self.carry_gate is not None - carry = self.carry_gate.FProp(theta.carry_gate, x, paddings) - if p.couple_carry_transform_gates: - transform = 1 - carry - else: - assert self.transform_gate is not None - transform = self.transform_gate.FProp(theta.transform_gate, x, paddings) - layer_out = x * carry + transformed_x * transform - return layer_out - - -class GatingLayer(base_layer.BaseLayer): - """A gating layer. - - This class represents a gating layer, which takes 2 inputs of the same shape - and gates them. - - The output is: carry * x + (1 - carry) * y where, carry is given by - sigmoid(x @ w_1 + y @ w_2 + bias). - - This is different from the HighwaySkipLayer above in that carry is also a - function of y (named transformed_x in HighwaySkipLayer). - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', 0, 'Dimension of the input to the network.') - p.Define('has_bias', False, 'Whether carry has a bias term.') - p.Define('carry_bias_init', 0.0, 'carry gates bias initialization') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - carry_gate_params = ProjectionLayer.Params().Set( - batch_norm=False, - has_bias=p.has_bias, - activation='SIGMOID', - input_dim=p.input_dim * 2, - output_dim=p.input_dim, - bias_init=p.carry_bias_init, - name='carry') - self.CreateChild('carry_gate', carry_gate_params) - - def FProp(self, theta, x, y, paddings=None): - """Fprop for the gating layer. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - x: An input feature, the last dimension must match p.input_dim. - y: Another input feature. Must have the same shape as 'x'. - paddings: padding applied to the features. When x and y have shape [..., - input_dim], 'paddings', when specified, must have shaped [..., 1], where - all but the last dimension match. - - Returns: - layer_out - activations after forward propagation. Same shape as x and y. - """ - y = py_utils.with_dependencies( - [py_utils.assert_shape_match(tf.shape(x), tf.shape(y))], y) - carry = self.carry_gate.FProp(theta.carry_gate, tf.concat([x, y], axis=-1), - paddings) - layer_out = x * carry + y * (1 - carry) - return layer_out - - -class GradNormTracker(base_layer.BaseLayer): - """A helper class to keep track of gradient norm stats.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('decay', 0.995, - 'Decay in updating the moving avgs in grad norm stats') - p.Define('grad_norm_lower_cap', 1e-2, 'The minimal gradient norm value.') - p.Define( - 'clip_threshold', 4.0, - 'Distance threshold at which gradients are clipped to 0.0.' - ' Distance is measured in the number of standard deviations a' - ' given gradient norm is from the mean gradient norm. The' - ' default value of 4.0 means we are throwing away roughly' - ' 0.15% of steps.') - p.Define( - 'grad_norm_clip_cap_min', 0.0, - 'We stop clipping if grad norm is already smaller than this' - ' value.') - return p - - def __init__(self, params): - super().__init__(params) - self._decay = params.decay - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - - pc = py_utils.WeightParams( - shape=[], - init=py_utils.WeightInit.Constant(0.0), - dtype=tf.float32, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('log_mean', pc, trainable=False) - self.CreateVariable('log_mean_squared', pc, trainable=False) - self.CreateVariable('total_weight', pc, trainable=False) - self.CreateVariable('total_rejections', pc, trainable=False) - - def FProp(self, theta, grad_norm, has_nan=None): - """Update gradient norm moving avgs, and returns whether or not ... - - to clip gradients to 0.0. If the current batch has NaN grads, does not - update the moving avgs and forces to clip the gradients to 0.0. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - grad_norm: A float scalar tensor. - has_nan: A boolean scalar tensor to indicate if the current batch has nan. - - Returns: - A scalar float tensor with value of either 1.0 or 0.0. The value of 0.0 - means the gradient norm is excessively large or contains NaN, and the step - should be aborted completely. - """ - p = self.params - with tf.name_scope(p.name): - grad_norm = tf.maximum(grad_norm, p.grad_norm_lower_cap) - - # Exponentially decayed moving avg of log(grad_norm) mean. - mean = theta.log_mean / tf.maximum(theta.total_weight, 1e-6) - # Exponentially decayed moving avg of log(grad_norm) variance. - var = ((theta.log_mean_squared / tf.maximum(theta.total_weight, 1e-6)) - - mean * mean) - std = tf.sqrt(tf.maximum(var, 1e-6)) - - summary_utils.scalar('log_grad_norm_mean', mean) - summary_utils.scalar('log_grad_norm_std', std) - summary_utils.scalar('clip_ratio_threshold', - tf.exp(std * p.clip_threshold)) - summary_utils.scalar('clip_threshold', - tf.exp(mean + std * p.clip_threshold) - 1.0) - summary_utils.scalar('total_rejections', theta.total_rejections) - - log_grad_norm = tf.math.log(grad_norm + 1.0) - log_grad_norm_cap = tf.cast(mean + std * p.clip_threshold, tf.float32) - log_grad_norm_cap_min = tf.math.log(p.grad_norm_clip_cap_min + 1.0) - log_grad_norm_cap = tf.maximum(log_grad_norm_cap, log_grad_norm_cap_min) - - def UpdateExpMovingAvg(ref_var, val, ignore): - if ignore is not None: - delta = tf.where(ignore, tf.zeros([]), - (1.0 - p.decay) * (val - ref_var)) - else: - delta = (1.0 - p.decay) * (val - ref_var) - return tf.assign_add(ref_var, delta) - - # We trigger when total_weight is at least half of max weight or the - # current batch contains NaNs. - trigger = tf.math.logical_and(log_grad_norm > log_grad_norm_cap, - theta.total_weight > 0.75) - if has_nan is not None: - trigger = tf.math.logical_or(trigger, has_nan) - - log_grad_norm_capped = tf.minimum(log_grad_norm, log_grad_norm_cap) - - update_moving_avg = tf.group( - UpdateExpMovingAvg(self.vars.log_mean, log_grad_norm_capped, has_nan), - UpdateExpMovingAvg(self.vars.log_mean_squared, - log_grad_norm_capped * log_grad_norm_capped, - has_nan), - UpdateExpMovingAvg(self.vars.total_weight, tf.constant(1.0), has_nan), - tf.assign_add(self.vars.total_rejections, - tf.cast(trigger, tf.float32))) - - return py_utils.with_dependencies([update_moving_avg], - 1.0 - tf.cast(trigger, tf.float32)) - - -class WeightedSumLayer(base_layer.BaseLayer): - """Returns the weighted sum of a list of input tensors.""" - - @classmethod - def Params(cls): - """Params for this MergerLayer class.""" - p = super().Params() - p.Define('num_sources', 0, 'Number of input sources to combine.') - p.Define('weighted_merger_dropout_prob', 0.1, - 'Applies dropout to the weights.') - p.Define( - 'weighted_merger_softmax', True, 'If set, applies a softmax ' - 'layer on top of the weights for normalization.') - p.Define('global_weight_scale', 1.0, 'A global scale put on weights.') - p.Define('minimal_prob', 0.0, 'The minimal weight for each component.') - p.Define('add_weight_summaries', False, 'If set, creates summaries for the ' - 'sum weights.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if not p.name: - raise ValueError('Layer must have a specified name!') - - assert p.num_sources > 0, ('Must specify num_sources > 0.') - - if p.weighted_merger_dropout_prob > 0.0: - dropout_tpl = DropoutLayer.Params() - dropout_tpl.keep_prob = (1.0 - p.weighted_merger_dropout_prob) - self.CreateChild('weighted_merger_dropout', dropout_tpl) - else: - self.CreateChild('weighted_merger_dropout', IdentityLayer.Params()) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - params_init = py_utils.WeightInit.Constant(0.0) - # Weights to be learned. - pw = py_utils.WeightParams( - shape=[p.num_sources], - init=params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('sum_weight', pw) - - def FProp(self, theta, inputs): - """Combines the list of input tensors into a single tensor. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: A list of tensors of shape [time, batch, hidden_dim] - - Returns: - A tensor of the same shape with input tensors. - """ - p = self.params - n_sources = len(inputs) - - if n_sources == 1: - return inputs[0] - - # Weighted sum of all sources, all dims must match. - # For weighted_sum, assume input is a list of rank 3 tensors - inputs = tf.stack(inputs) - inputs = py_utils.HasRank(inputs, 4) - - # The constant factor is just meant to support the non-normalized scenario. - # If softmax is applied, this factor will cancel out. - w = theta.sum_weight * p.global_weight_scale + (1 / p.num_sources) - w = self.weighted_merger_dropout.FProp(theta.weighted_merger_dropout, w) - - if p.weighted_merger_softmax: - residual_weights = p.minimal_prob * p.num_sources - assert residual_weights >= 0.0 - assert residual_weights < 1.0 - w = tf.nn.softmax(w, axis=0) * (1.0 - residual_weights) + p.minimal_prob - - if p.add_weight_summaries: - for i in range(p.num_sources): - summary_utils.scalar(p.name + 'weight_%d' % i, w[i]) - w = tf.reshape(w, [p.num_sources, 1, 1, 1]) - output = tf.reduce_sum(inputs * w, axis=0) - - return output - - -class GatedAverageLayer(base_layer.BaseLayer): - """Gated combination of n input vectors. - - Given n inputs, x_1 ... x_n. First learns a gate g in a single layer. - Returns g_1 * x_1 + ... g_n * x_n. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_nodes', 0, 'Number of nodes in each input vector.') - p.Define('num_inputs', 0, 'Number of input vectors to combine.') - return p - - def __init__(self, params): - """Initializes GatedAverageLayer.""" - super().__init__(params) - p = self.params - - assert p.num_nodes > 0, 'Number of dimensions should be greater than 0.' - assert p.num_inputs > 0, 'Number of inputs should be greater than 0.' - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - in_size = p.num_inputs * p.num_nodes - - # Weight matrix for scalar gates - gm_pc = py_utils.WeightParams( - shape=[in_size, p.num_inputs], - init=p.params_init, - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('gm', gm_pc) - - def FProp(self, theta, inputs): - """Gates, then merges a list of n input vectors. - - Args: - theta: gm (gate matrix) - inputs: List of inputs, each of shape [..., num_nodes] - - Returns: - a gated output vector [..., num_nodes] - """ - p = self.params - assert len(inputs) == p.num_inputs, 'Number of inputs should match params.' - - for i, inp in enumerate(inputs): - inputs[i] = py_utils.with_dependencies([ - py_utils.assert_shape_match([tf.shape(inp)[-1]], [p.num_nodes]), - py_utils.assert_shape_match(tf.shape(inp), tf.shape(inputs[0])), - ], inp) - - input_shape = tf.shape(inputs[0]) - - reshaped_inputs = [tf.reshape(inp, [-1, p.num_nodes]) for inp in inputs] - concat_inputs = tf.concat(reshaped_inputs, axis=1) - - xmg = tf.nn.softmax(py_utils.Matmul(concat_inputs, theta.gm)) - xmg = tf.expand_dims(xmg, 2) - inputs = tf.reshape(concat_inputs, [-1, p.num_inputs, p.num_nodes]) - gated_sum = tf.reduce_sum(xmg * inputs, axis=1) - - return tf.reshape(gated_sum, input_shape) - - -class LHUCLayer(base_layer.BaseLayer): - """`Learning Hidden Unit Contribution (LHUC)` layer. - - This paper proposes to use LHUC layer for NMT adaptation: - http://aclweb.org/anthology/N18-2080 - - During base model training, LHUC layer is fixed to 1.0 (no-op in - multiplication). During adaptation, only LHUC layer is trained, and all other - parameters in the model are frozen. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', 0, 'Dimension of the input and output.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.input_dim > 0 - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - pc = py_utils.WeightParams( - shape=[p.input_dim], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('w', pc) - - def FProp(self, theta, inp): - """Add learnt gate for adaptation.""" - out = 2.0 * tf.sigmoid(theta.w) * inp - return out - - -class ResidualAdapterLayer(base_layer.BaseLayer): - """Residual Adapter layer for NLP tasks. - - This paper proposes using residual adapters for fine-tuning new tasks on BERT. - https://arxiv.org/pdf/1902.00751.pdf - - During adaptation, residual adapter layers can be added to a pre-trained - model and trained, while all other parameters are frozen. - In terms of operations, the layer is identical to a vanilla Transformer - feedforward layer. Separate implementation is meant to distinguish function. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', 0, 'Dimension of the input to the adapter.') - p.Define('bottleneck_dim', 0, 'Dimension of the feedforward inner layer.') - p.Define('ln_tpl', LayerNorm.Params(), 'Layer norm default params.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - - bottleneck_params = FeedForwardNet.Params().Set( - name='bottleneck', - activation=['RELU', 'NONE'], - input_dim=p.input_dim, - hidden_layer_dims=[p.bottleneck_dim, p.input_dim]) - self.CreateChild('bottleneck', bottleneck_params) - - params = p.ln_tpl.Copy() - params.name = 'adapter_ln' - params.input_dim = p.input_dim - self.CreateChild('layer_norm', params) - - def FProp(self, theta, x, paddings=None): - """Fprop for Residual Adapter. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - x: [..., input_dim]. - paddings: padding applied to the features. - - Returns: - layer_out - [..., input_dim]. - """ - normalized_x = self.layer_norm.FProp(theta.layer_norm, x) - bottleneck_x = self.bottleneck.FProp(theta.bottleneck, normalized_x, - paddings) - return x + bottleneck_x - - -def Conv2DFlops(inputs, filter_shape, stride, padding): - """Returns number of float operations (mult/adds) for a Conv2D op. - - Args: - inputs: the input shape. Must have four elements. - filter_shape: the convolution filter shape. Must have four elements. - stride: the strides along height and width, respectively. - padding: 'SAME' or 'VALID'. - - Returns: - Number of multiplications and additions. - """ - b, h, w = inputs[0], inputs[1], inputs[2] - fh, fw, ic, oc = filter_shape - sh, sw = stride - - def _CeilDiv(x, y): - return tf.math.floordiv(x + y - 1, y) - - if padding == 'SAME': - oh = _CeilDiv(h, sh) - ow = _CeilDiv(w, sw) - else: - assert padding == 'VALID' - oh = _CeilDiv(h - fh + 1, sh) - ow = _CeilDiv(w - fw + 1, sw) - # Mul/add counts as 2 flops. - return (tf.cast(b * oh * ow, tf.int64) * - tf.cast(fh * fw * ic * oc, tf.int64) * 2) - - -class Conv2DLayerNoPadding(base_layer.BaseLayer): - """2-D Convolution layer w/o padding. - - TODO(laurenzo): Dedup in favor of SeparableConv2DLayer where possible. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'filter_shape', (0, 0, 0, 0), - 'Filter shape. Must be a sequence of length 4. Elements are in' - ' the order of height (time), width (frequency), in_channel,' - ' out_channel. ') - p.Define( - 'filter_stride', (0, 0), - 'Filter stride to use. Must be a pair of ints. The first int' - ' specifies the stride on the height dimension. The second int' - ' specifies the stride on the width dimension.') - p.Define( - 'dilations', (1, 1), ' An optional list of ints. Defaults to [1, 1]. ' - '1-D tensor of length 2. The dilation factor for each dimension ' - 'of input. If set to k > 1, there will be k-1 skipped cells ' - 'between each filter element on that dimension.') - p.Define('padding', 'SAME', 'SAME|VALID') - - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.padding in ['SAME', 'VALID'] - assert len(p.filter_shape) == 4 - assert len(p.filter_stride) == 2 - assert len(p.dilations) == 2 - assert all(x > 0 for x in p.filter_stride) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - w_pc = py_utils.WeightParams( - shape=p.filter_shape, - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('w', w_pc) - - def FProp(self, theta, x): - """Apply convolution to inputs. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - x: The inputs tensor. It is expected to be of shape [batch, height, width, - channel]. - - Returns: - Convolution output. - """ - p = self.params - with tf.name_scope(p.name): - computation_cost.Add( - self, 'flops', - Conv2DFlops( - tf.shape(x), - filter_shape=symbolic.EvalExpr(symbolic.TENSOR_VALUES, - p.filter_shape), - stride=p.filter_stride, - padding=p.padding)) - return tf.nn.conv2d( - input=x, - filters=theta.w, - strides=[1, p.filter_stride[0], p.filter_stride[1], 1], - padding=p.padding, - dilations=[1, p.dilations[0], p.dilations[1], 1], - data_format='NHWC') - - @classmethod - def FPropMeta(cls, p, inputs): - py_utils.CheckShapes((inputs,)) - b, h, w, c = inputs - fh, fw, ic, oc = p.filter_shape - assert ic == c - sh, sw = p.filter_stride - if p.padding == 'SAME': - oh = sympy.ceiling(h / sh) - ow = sympy.ceiling(w / sw) - else: - oh = sympy.ceiling((h - fh + 1) / sh) - ow = sympy.ceiling((w - fw + 1) / sw) - flops = b * oh * ow * fh * fw * ic * oc * 2 # mul/add counts as 2 flop. - outputs = tshape.Shape([b, oh, ow, oc]) - return py_utils.NestedMap(flops=flops, out_shapes=(outputs,)) - - -class FetchLayer(base_layer.BaseLayer): - """A layer facilitating fetching activations and their gradients.""" - - def __init__(self, params): - super().__init__(params) - assert self.params.name - self._activations = None - self._gradients = None - - @classmethod - def FPropMeta(cls, params, *args): - return py_utils.NestedMap(flops=0, out_shapes=args) - - def _ReturnSingleValueOrList(self, lst): - assert lst is not None - assert isinstance(lst, list) - return lst if len(lst) > 1 else lst[0] - - @property - def activation(self): - return self._ReturnSingleValueOrList(self._activations) - - @property - def gradient(self): - return self._ReturnSingleValueOrList(self._gradients) - - def FProp(self, theta, *args): - del theta - num = len(args) - self._activations = [None] * num - self._gradients = [None] * num - - for i, v in enumerate(args): - - def FetchBak(xs, ys, dys, index=i): - del xs, ys - self._gradients[index] = dys - return dys - - def FetchFwd(x): - return x - - self._activations[i] = py_utils.CallDefun(FetchFwd, v, bak=FetchBak) - - return tuple(self._activations) if num > 1 else self._activations[0] - - -class GluLayer(base_layer.BaseLayer): - """Gated Linear Unit. - - See https://arxiv.org/abs/1612.08083 for more details. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', 0, 'Dimension of the layer input.') - p.Define('output_dim', 0, 'Dimension of the layer output.') - p.Define('ln_tpl', LayerNorm.Params(), 'Layer norm default params.') - p.Define('dense_tpl', FCLayer.Params().Set(), 'Fully connected layer.') - p.Define( - 'activation', 'RELU', - 'Non-linearity applied after the dense layer in the value branch.') - p.Define('dropout_tpl', DropoutLayer.Params(), 'Dropout applied to output.') - p.Define('apply_residual', True, 'Whether or not to add inputs to outputs.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.input_dim - - if p.output_dim: - output_dim = p.output_dim - else: - output_dim = p.input_dim - - if p.apply_residual: - assert output_dim == p.input_dim - - # Initialize value feed-forward layer. - params = p.dense_tpl.Copy() - params.name = 'value_layer' - params.input_dim = p.input_dim - params.activation = p.activation - params.output_dim = output_dim - self.CreateChild('value_layer', params) - - # Initialize gate feed-forward layer. - params = p.dense_tpl.Copy() - params.name = 'gate_layer' - params.input_dim = p.input_dim - params.activation = 'SIGMOID' - params.output_dim = output_dim - self.CreateChild('gate_layer', params) - - # Initialize layer norm. - params = p.ln_tpl.Copy() - params.name = 'layer_norm' - params.input_dim = p.input_dim - self.CreateChild('layer_norm', params) - - # Initialize dropout. - dropout_tpl = p.dropout_tpl.Copy() - self.CreateChild('dropout', dropout_tpl) - - def FProp(self, theta, inputs, paddings): - inputs_normalized = self.layer_norm.FProp(theta.layer_norm, inputs) - values = self.value_layer.FProp(theta.value_layer, inputs_normalized, - tf.expand_dims(paddings, -1)) - gates = self.gate_layer.FProp(theta.gate_layer, inputs_normalized, - tf.expand_dims(paddings, -1)) - glu_output = values * gates - glu_output = self.dropout.FProp(theta.dropout, glu_output) - if self.params.apply_residual: - return inputs + glu_output - return glu_output - - -class MultitaskAdapterLayer(base_layer.BaseLayer): - """Residual adapter layer for multilingual models. - - Residual adapters can be used to fine-tune a single model to multiple - domains, tasks, or languages: https://arxiv.org/pdf/1902.00751.pdf - - Each adapter consists of a "down" projection to a smaller dimension followed - by an "up" projection, the result of which is added back to the input - activation. The projection weights and biases are task-specific. - - Whereas ResidualAdapterLayer learns and applies the parameters for a single - task, this layer learns and applies the parameters for multiple tasks so that - we have a single model serving the different tasks. The parameters can be - trained for all tasks at the same time, or in one-off per-task training jobs. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_tasks', 0, 'Number of tasks.') - p.Define('input_dim', 0, 'Dimension of the input to the adapter.') - p.Define('bottleneck_dim', 0, 'Dimension of the bottleneck.') - p.Define('layer_norm_tpl', LayerNorm.Params(), 'Layer norm default params.') - p.Define( - 'projection_params_init', None, - 'Weight initialization for up and down projections. Only used for ' - 'weights, not biases. If None, uses default weight init, which is ' - 'typically Xavier with scale of 1.0.') - p.Define( - 'data_format', 'TBC', 'String(enum) specifying the input and output ' - 'data format for this layer. Supported formats: ' - '"TBC": [time, batch, input_dim] and "BTC": [batch, time, input_dim].') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - # Data format is either 'TBC' (time-major) or 'BTC' (batch-major). - assert p.data_format in ('TBC', 'BTC') - base_emb_params = EmbeddingLayer.Params().Set( - vocab_size=p.num_tasks, max_num_shards=1) - down_proj_w_params = base_emb_params.Copy() - down_proj_w_params.Set( - embedding_dim=p.input_dim * p.bottleneck_dim, name='down_proj_w') - if p.projection_params_init: - down_proj_w_params.params_init = p.projection_params_init - down_proj_b_params = base_emb_params.Copy() - down_proj_b_params.Set(embedding_dim=p.bottleneck_dim, name='down_proj_b') - up_proj_w_params = base_emb_params.Copy() - up_proj_w_params.Set( - embedding_dim=p.bottleneck_dim * p.input_dim, name='up_proj_w') - if p.projection_params_init: - up_proj_w_params.params_init = p.projection_params_init - up_proj_b_params = base_emb_params.Copy() - up_proj_b_params.Set(embedding_dim=p.input_dim, name='up_proj_b') - - self.CreateChild('down_proj_w', down_proj_w_params) - self.CreateChild('down_proj_b', down_proj_b_params) - self.CreateChild('up_proj_w', up_proj_w_params) - self.CreateChild('up_proj_b', up_proj_b_params) - params = p.layer_norm_tpl.Copy() - params.name = 'adapter_ln' - params.input_dim = p.input_dim - self.CreateChild('layer_norm', params) - - def FProp(self, theta, inputs, tasks): - """Fprop for multitask adapter. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - inputs: A tensor containing the activations from the previous layer. For - 'TBC', the shape is [time, batch, input_dim] and for 'BTC', it's [batch, - time, input_dim]. - tasks: An int32 tensor containing the task ID for each input. If 'tasks' - is of rank 2, we assume it to be of shape [time, batch] if 'BTC' and - [batch, time] if 'TBC', indicating a different task for each timestep. - In this case we look up adapter params for each timestep. If 'tasks' is - of rank 1, we assume it to be of shape [batch], indicating a single task - for all timesteps of a sequence. This latter setup uses substantially - less memory and is generally preferred. - - Returns: - A tensor containing the adapted activations with shape - [time, batch, input_dim] for 'TBC' and [batch, time, input_dim] for 'BTC'. - """ - p = self.params - inputs_shape = tf.shape(inputs) - per_timestep_task = (tasks.shape.ndims == 2) - batch_index = 1 if p.data_format == 'TBC' else 0 - time_index = 1 - batch_index - inputs = py_utils.with_dependencies( - [ - # Checks that inputs has 3 dimensions, last is hidden dim. - py_utils.assert_shape_match(inputs_shape, [-1, -1, p.input_dim]), - # Checks that inputs and tasks have same batch dimension. - py_utils.assert_shape_match([inputs_shape[batch_index]], [ - tf.shape(tasks)[batch_index] - if per_timestep_task else tf.shape(tasks)[0] - ]) - ], - inputs) - - # To support different task for each timetstep, flatten inputs and - # tasks. Below, 'batch' now refers to flattened batch size, time * batch. - if per_timestep_task: - tasks = py_utils.with_dependencies( - [ - # Checks that inputs and tasks have same time dimension. - py_utils.assert_shape_match(inputs_shape[:1], - tf.shape(tasks)[:1]) - ], - tasks) - tasks = tf.reshape(tasks, [-1]) - if p.data_format == 'TBC': - inputs = tf.reshape(inputs, [1, -1, p.input_dim]) - else: - inputs = tf.reshape(inputs, [-1, 1, p.input_dim]) - - # Lookup all weights and biases - # [batch] -> [batch, hidden * k] -> [batch, hidden, k] - down_weights = tf.reshape( - self.down_proj_w.EmbLookup(theta.down_proj_w, tasks), - [-1, p.input_dim, p.bottleneck_dim]) - # [batch] -> [batch, k] -> [1, batch, k] if 'TBC' else [batch, 1, k] - down_biases = tf.expand_dims( - self.down_proj_b.EmbLookup(theta.down_proj_b, tasks), time_index) - # [batch] -> [batch, k * hidden] -> [batch, k, hidden] - up_weights = tf.reshape( - self.up_proj_w.EmbLookup(theta.up_proj_w, tasks), - [-1, p.bottleneck_dim, p.input_dim]) - # [batch] -> [batch, h] -> [1, batch, h] if 'TBC' else [batch, 1, h] - up_biases = tf.expand_dims( - self.up_proj_b.EmbLookup(theta.up_proj_b, tasks), time_index) - - # Layer norm -> down-projection -> non-linearity -> up-projection - norm_inputs = self.layer_norm.FProp(theta.layer_norm, inputs) - # If per_timestep_task, t = 1, b = time * batch. - # Otherwise, t = time, b = batch. - if p.data_format == 'TBC': - down_projected = tf.einsum('tbh,bhk->tbk', norm_inputs, down_weights) - else: - down_projected = tf.einsum('bth,bhk->btk', norm_inputs, down_weights) - down_projected += down_biases - down_projected = tf.nn.relu(down_projected) - if p.data_format == 'TBC': - up_projected = tf.einsum('tbk,bkh->tbh', down_projected, up_weights) - else: - up_projected = tf.einsum('btk,bkh->bth', down_projected, up_weights) - up_projected += up_biases - output = inputs + up_projected - - # Unflatten output: - # for 'TBC': [1, time * batch, hidden] -> [time, batch, hidden] - # for 'BTC': [1, batch * time, hidden] -> [batch, time, hidden] - if per_timestep_task: - output = tf.reshape(output, inputs_shape) - return output - - -class CCTGatingNetwork(quant_utils.QuantizableLayer): - """A gating network that is continous for training and discrete for eval. - - Based on the gating network from https://arxiv.org/abs/2002.07106. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', 0, 'Depth of the input to the network.') - p.Define('hidden_layer_dim', 0, 'Depth of the hidden layer outputs.') - p.Define('num_outputs', 0, 'Number of scalar gate outputs.') - p.Define('noise_std', 1.0, 'Standard deviation for gating noise.') - p.Define('noise_warmup_steps', 1.0, 'Steps to full noise.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - params = schedule.PolynomialSchedule.Params() - params.start = (0, 0.0) - params.limit = (p.noise_warmup_steps, p.noise_std) - self.CreateChild('noise_std', params) - - params = FeedForwardNet.Params() - params.name = 'gating_layer' - params.input_dim = p.input_dim - params.activation = ['RELU', 'NONE'] - params.hidden_layer_dims = [p.hidden_layer_dim, p.num_outputs] - self.CreateChild('gatingfflayer', params) - - def FProp(self, theta, inputs, paddings=None): - p = self.params - p_c = self.gatingfflayer.FProp(theta.gatingfflayer, inputs, paddings) - if self.do_eval: - ones = tf.ones(tf.shape(p_c), py_utils.FPropDtype(p)) - zeros = tf.zeros(tf.shape(p_c), py_utils.FPropDtype(p)) - p_c = tf.where( - tf.greater_equal(p_c, tf.constant(0.0, dtype=py_utils.FPropDtype(p))), - ones, zeros) - else: - noise_std = self.noise_std.FProp(theta.noise_std, theta.global_step) - noise = py_utils.DeterministicVN( - p, - py_utils.GenerateStepSeedPair(p, theta.global_step), - tf.shape(p_c), - std=noise_std) - p_c = tf.nn.sigmoid(p_c + noise) - return p_c - - @classmethod - def FPropMeta(cls, p, inputs, paddings=None): - py_utils.CheckShapes((inputs,)) - assert inputs[-1] == p.input_dim - flops = 0 - in_dim = inputs[-1] - other_dims = inputs.num_elements() / in_dim - flops = 5 * other_dims * in_dim * p.hidden_layer_dim - flops = 5 * other_dims * p.num_outputs * p.hidden_layer_dim - out_shape = tshape.Shape(inputs[:-1] + [symbolic.ToStatic(p.num_outputs)]) - return py_utils.NestedMap(flops=flops, out_shapes=(out_shape,)) - - -class CondScaleShiftFFNLayer(base_layer.BaseLayer): - """Feature Modulation layer. - - https://distill.pub/2018/feature-wise-transformations/ - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', 0, 'Depth of the input.') - p.Define('output_dim', 0, 'Depth of the output.') - p.Define('ffn', FeedForwardNet.Params(), 'Projection layer params') - p.Define('scale_fn', 'NONE', - 'The activation function to use for scale output') - p.Define('shift_fn', 'NONE', - 'The activation function to use for shift output') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - - output_dim = p.output_dim * 2 # 1st split for shift, 2nd split for scale - params_ffn = p.ffn.Copy().Set( - input_dim=p.input_dim, name='{}_ffn'.format(p.name)) - params_fcout = FCLayer.Params().Copy().Set( - input_dim=params_ffn.hidden_layer_dims[-1], - output_dim=output_dim, - activation='NONE', - name='{}_fcout'.format(p.name)) - self.CreateChild('ffn', params_ffn) - self.CreateChild('fcout', params_fcout) - - def FProp(self, theta, inputs, paddings=None): - """Calculate scale shift and modify input. - - Args: - theta: params. - inputs: The input tensor. Shaped [..., input_dim]. - paddings: The input padding tensors. - - Returns: - Output after calculating shift and scale (2 tensors). - Shaped [..., output_dim]. - """ - p = self.params - - ffn_output = self.ffn.FProp(theta.ffn, inputs, paddings) - fcout_output = self.fcout.FProp(theta.fcout, ffn_output, paddings) - scale_output, shift_output = tf.split( - fcout_output, num_or_size_splits=2, axis=-1) - - def OpWrapper(name, tensor): - """Wrapper for retrieve tf operations.""" - if name in _ACTIVATIONS: - op = _ACTIVATIONS[name] - else: - if name == 'EXP': - op = tf.exp - elif name == 'NONE': - op = tf.identity - else: - raise ValueError() - return op(tensor) - - scale_output = OpWrapper(p.scale_fn, scale_output) - shift_output = OpWrapper(p.shift_fn, shift_output) - return scale_output, shift_output diff --git a/lingvo/core/layers_test.py b/lingvo/core/layers_test.py deleted file mode 100644 index be2b33d5..00000000 --- a/lingvo/core/layers_test.py +++ /dev/null @@ -1,5338 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for layers.""" - -import math - -from absl.testing import parameterized -import lingvo.compat as tf -from lingvo.core import bn_layers -from lingvo.core import cluster_factory -from lingvo.core import gpipe -from lingvo.core import layers -from lingvo.core import py_utils -from lingvo.core import quant_utils -from lingvo.core import symbolic -from lingvo.core import test_utils -from lingvo.core import tshape -import numpy as np - - -class ActivationsTest(test_utils.TestCase): - - def testGeluActivation(self): - with self.session(use_gpu=True): - inputs = tf.constant( - np.linspace(-10.0, 10.0, num=21, dtype='float32'), dtype=tf.float32) - grads_gelu = tf.gradients(layers.Gelu(inputs), inputs) - grads_relu = tf.gradients(tf.nn.relu(inputs), inputs) - - self.assertEqual(0.0, - layers.Gelu(tf.constant(-10.0, dtype='float32')).eval()) - self.assertEqual(0.0, - layers.Gelu(tf.constant(0.0, dtype='float32')).eval()) - self.assertEqual(10.0, - layers.Gelu(tf.constant(10.0, dtype='float32')).eval()) - actual_grads_gelu = grads_gelu[0].eval() - actual_grads_relu = grads_relu[0].eval() - - self.assertAllClose(actual_grads_gelu[-5:], actual_grads_relu[-5:]) - self.assertAllClose(actual_grads_gelu[:5], actual_grads_relu[:5]) - - # pyformat: disable - # pylint: disable=bad-whitespace - expected_grads_gelu = [ - -7.69459925e-22, -9.25176121e-18, -4.04182472e-14, -6.39430453e-11, - -3.64552299e-08, -7.13557529e-06, -5.03641320e-04, -1.19456425e-02, - -8.52318183e-02, -8.33154917e-02, 5.00000000e-01, 1.08331549e+00, - 1.08523178e+00, 1.01194561e+00, 1.00050366e+00, 1.00000715e+00, - 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, - 1.00000000e+00] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_grads_gelu, actual_grads_gelu) - - -class BatchNormLayerTest(test_utils.TestCase, parameterized.TestCase): - - def testBatchNormLayerConstruction(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.BatchNormLayer.Params() - params.name = 'bn' - params.dim = 2 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - params.add_stats_to_moving_average_variables = True - layers.BatchNormLayer(params) - bn_vars = tf.get_collection('BatchNormLayer_vars') - bn_var_names = [x.name for x in bn_vars] - expected_var_names = [ - 'bn/beta/var:0', 'bn/gamma/var:0', 'bn/moving_mean/var:0', - 'bn/moving_variance/var:0' - ] - self.assertEqual(expected_var_names, bn_var_names) - self.assertEqual(['bn/moving_mean/var:0', 'bn/moving_variance/var:0'], - [x.name for x in tf.moving_average_variables()]) - - def testBatchNormLayerMoments(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - - in_padding1 = tf.zeros([2, 2, 8, 1], dtype=tf.float32) - bn_in1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 2, 8, 2]), dtype=tf.float32) - mean1, var1 = bn_layers.ComputeMomentsWithPadding( - bn_in1, in_padding1, reduce_over_dims=[0, 1, 2]) - mean2, var2 = tf.nn.moments(bn_in1, [0, 1, 2]) - - in_padding2 = tf.ones([2, 2, 8, 1], dtype=tf.float32) - bn_in2 = tf.constant( - np.random.normal(-0.3, 1.0, [2, 2, 8, 2]), dtype=tf.float32) - in_padding3 = tf.concat([in_padding1, in_padding2], 1) - bn_in3 = tf.concat([bn_in1, bn_in2], 1) - mean3, var3 = bn_layers.ComputeMomentsWithPadding( - bn_in3, in_padding3, reduce_over_dims=[0, 1, 2]) - mean4, var4 = tf.nn.moments(bn_in3, [0, 1, 2]) - - mean_diff = tf.reduce_sum(tf.square(mean3 - mean4)) - var_diff = tf.reduce_sum(tf.square(var3 - var4)) - - self.evaluate(tf.global_variables_initializer()) - - self.assertAllClose(mean2.eval(), mean1.eval()) - self.assertAllClose(var2.eval(), var1.eval()) - self.assertAllClose(mean3.eval(), mean1.eval()) - self.assertAllClose(var3.eval(), var1.eval()) - # Since tf.nn.moments() doesn't support padding, it is expected to produce - # different results than our own implementation (of moments). - self.assertAllClose(0.095987, mean_diff.eval()) - self.assertAllClose(0.364456, var_diff.eval()) - - def testBatchNormLayerFProp(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.BatchNormLayer.Params() - params.name = 'bn' - params.dim = 3 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - bn_layer = layers.BatchNormLayer(params) - in_padding1 = tf.zeros([2, 8, 1], dtype=tf.float32) - bn_in1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 8, 3]), dtype=tf.float32) - - bn_out = bn_layer.FPropDefaultTheta(bn_in1, in_padding1) - sig1 = tf.reduce_sum(bn_out) - sig2 = tf.reduce_sum(bn_out * bn_out) - self.evaluate(tf.global_variables_initializer()) - self.assertAllClose(0.0, sig1.eval(), atol=1e-5) - self.assertAllClose(47.8371887, sig2.eval()) - - def testBatchNormLayerFPropUseGlobalStatsForTraining(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.BatchNormLayer.Params() - params.name = 'bn' - params.dim = 3 - params.use_moving_avg_in_training = True - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - bn_layer = layers.BatchNormLayer(params) - in_padding1 = tf.zeros([2, 8, 1], dtype=tf.float32) - bn_in1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 8, 3]), dtype=tf.float32) - - bn_out = bn_layer.FPropDefaultTheta(bn_in1, in_padding1) - sig1 = tf.reduce_sum(bn_out) - sig2 = tf.reduce_sum(bn_out * bn_out) - self.evaluate(tf.global_variables_initializer()) - self.assertAllClose(2.6593573, sig1.eval(), atol=1e-5) - self.assertAllClose(15.464208, sig2.eval()) - - def testBatchNormLayerFPropWithUpdateUseGlobalStatsForTraining(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.BatchNormLayer.Params() - params.name = 'bn' - params.dim = 3 - params.use_moving_avg_in_training = True - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - bn_layer = layers.BatchNormLayer(params) - in_padding1 = tf.zeros([2, 8, 1], dtype=tf.float32) - bn_in1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 8, 3]), dtype=tf.float32) - - bn_out = bn_layer.FPropDefaultTheta(bn_in1, in_padding1) - sig1 = tf.reduce_sum(bn_out) - sig2 = tf.reduce_sum(bn_out * bn_out) - - # get updates which should be invoked during training step - # but we call them here, so that UpdateBatchNormVars is tested too - bn_update_dict = py_utils._get_batch_norm_updates_dict() - bn_update_list = list(bn_update_dict.keys()) - - self.evaluate(tf.global_variables_initializer()) - self.evaluate(bn_update_list) - self.assertAllClose(2.6575434, sig1.eval(), atol=1e-5) - self.assertAllClose(15.473802, sig2.eval()) - - def testBatchNormLayerMomentsForConv(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - - in_padding1 = tf.zeros([2, 8, 1, 1], dtype=tf.float32) - bn_in1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 8, 4, 3]), dtype=tf.float32) - mean1, var1 = bn_layers.ComputeMomentsWithPadding( - bn_in1, in_padding1, reduce_over_dims=[0, 1, 2]) - mean2, var2 = tf.nn.moments(bn_in1, [0, 1, 2]) - - in_padding2 = tf.ones([2, 8, 1, 1], dtype=tf.float32) - bn_in2 = tf.constant( - np.random.normal(-0.3, 1.0, [2, 8, 4, 3]), dtype=tf.float32) - in_padding3 = tf.concat([in_padding1, in_padding2], 1) - bn_in3 = tf.concat([bn_in1, bn_in2], 1) - mean3, var3 = bn_layers.ComputeMomentsWithPadding( - bn_in3, in_padding3, reduce_over_dims=[0, 1, 2]) - mean4, var4 = tf.nn.moments(bn_in3, [0, 1, 2]) - - mean_diff = tf.reduce_sum(tf.square(mean3 - mean4)) - var_diff = tf.reduce_sum(tf.square(var3 - var4)) - - self.evaluate(tf.global_variables_initializer()) - - self.assertAllClose(mean2.eval(), mean1.eval()) - self.assertAllClose(var2.eval(), var1.eval()) - self.assertAllClose(mean3.eval(), mean1.eval()) - self.assertAllClose(var3.eval(), var1.eval()) - self.assertAllClose(0.1726295, mean_diff.eval()) - self.assertAllClose(0.5592572093009949, var_diff.eval()) - - def testBatchNormLayerFPropForConv(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.BatchNormLayer.Params() - params.name = 'bn_conv' - params.dim = 32 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - bn_layer = layers.BatchNormLayer(params) - in_padding1 = tf.zeros([2, 8, 1, 1], dtype=tf.float32) - bn_in1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 8, 4, 32]), dtype=tf.float32) - - bn_out = bn_layer.FPropDefaultTheta(bn_in1, in_padding1) - sig1 = tf.reduce_sum(bn_out) - sig2 = tf.reduce_sum(bn_out * bn_out) - self.evaluate(tf.global_variables_initializer()) - self.assertAllClose(0.0, sig1.eval(), atol=1e-4) - self.assertAllClose(2039.398681, sig2.eval()) - - @parameterized.named_parameters( - ('FuseEvalNoFreeze', True, True, False), - ('NoFuseEvalNoFreeze', False, True, False), - ('FuseTrainingFreeze', True, False, True), - ('NoFuseTrainingFreeze', False, False, True), - ) - def testBatchNormLayerFPropForConvWithFusedEvalWithFreezeBNStats( - self, use_fused_batch_norm_for_eval, do_eval, freeze_bn_stats): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.BatchNormLayer.Params() - params.name = 'bn_conv' - params.dim = 32 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - params.use_fused_batch_norm_for_eval = use_fused_batch_norm_for_eval - params.freeze_bn_stats = freeze_bn_stats - with cluster_factory.ForTestingWorker( - mode='sync', job='trainer_client', do_eval=True): - bn_layer = params.Instantiate() - bn_layer._epsilon = 0.0 # Enables a lower tolerance in the test check. - in_padding1 = tf.zeros([2, 4, 1, 1], dtype=tf.float32) - np_in1 = np.random.normal(0.1, 0.5, [2, 4, 1, 32]) - bn_in1 = tf.constant(np_in1, dtype=tf.float32) - bn_out = bn_layer.FPropDefaultTheta(bn_in1, in_padding1) - self.evaluate(tf.global_variables_initializer()) - # Moving mean and variance are set to defaults, we set gamma and beta - # through assignment such that the outputs are inputs * 2 + 1. - moving_mean_init = np.zeros(bn_layer.vars.moving_mean.shape.as_list()) - moving_variance_init = np.ones( - bn_layer.vars.moving_variance.shape.as_list()) - self.evaluate([ - tf.assign(bn_layer.vars.gamma, - np.ones(bn_layer.vars.gamma.shape.as_list())), - tf.assign(bn_layer.vars.beta, - np.ones(bn_layer.vars.beta.shape.as_list())), - tf.assign(bn_layer.vars.moving_mean, moving_mean_init), - tf.assign(bn_layer.vars.moving_variance, moving_variance_init), - ]) - self.assertAllClose( - np_in1 * 2. + 1., bn_out.eval(), atol=1e-5, rtol=1e-5) - # check moving stats modified or not - moving_mean = self.evaluate(bn_layer.vars.moving_mean) - moving_variance = self.evaluate(bn_layer.vars.moving_variance) - if do_eval or freeze_bn_stats: - self.assertAllClose(moving_mean, moving_mean_init) - self.assertAllClose(moving_variance, moving_variance_init) - - -class CategoricalBNTest(test_utils.TestCase, parameterized.TestCase): - - def testConstruction(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = bn_layers.CategoricalBN.Params() - params.name = 'bn' - params.dim = 2 - params.class_emb_dim = 4 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - bn_layers.CategoricalBN(params) - bn_vars = tf.get_collection('CategoricalBN_vars') - bn_var_names = [x.name for x in bn_vars] - expected_var_names = [ - 'bn/beta/var:0', 'bn/gamma/var:0', 'bn/moving_mean/var:0', - 'bn/moving_variance/var:0' - ] - self.assertEqual(expected_var_names, bn_var_names) - self.assertEqual(['bn/moving_mean/var:0', 'bn/moving_variance/var:0'], - [x.name for x in tf.moving_average_variables()]) - - def testFPropSameClass(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = bn_layers.CategoricalBN.Params() - params.name = 'bn' - params.dim = 3 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - params.class_emb_dim = 4 - - bn_layer = bn_layers.CategoricalBN(params) - in_padding1 = tf.zeros([2, 8, 1], dtype=tf.float32) - bn_in1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 8, 3]), dtype=tf.float32) - domain_in1 = tf.one_hot([0, 0], - depth=params.class_emb_dim, - dtype=tf.float32) - - bn_out = bn_layer.FPropDefaultTheta(bn_in1, in_padding1, domain_in1) - sig1 = tf.reduce_sum(bn_out) - sig2 = tf.reduce_sum(bn_out * bn_out) - self.evaluate(tf.global_variables_initializer()) - self.assertAllClose(0.0, sig1.eval(), atol=1e-5) - self.assertAllClose(47.8371887, sig2.eval()) - - def testFPropDifferenClasses(self): - with self.session(use_gpu=True) as sess: - tf.random.set_seed(398847392) - np.random.seed(12345) - params = bn_layers.CategoricalBN.Params() - params.name = 'bn' - params.dim = 3 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - params.class_emb_dim = 4 - - bn_layer = bn_layers.CategoricalBN(params) - in_padding1 = tf.zeros([4, 8, 1], dtype=tf.float32) - bn_in1 = tf.constant( - np.random.normal(0.1, 0.5, [4, 8, 3]), dtype=tf.float32) - domain_in1 = tf.one_hot([0, 1, 1, 2], - depth=params.class_emb_dim, - dtype=tf.float32) - - bn_out = bn_layer.FPropDefaultTheta(bn_in1, in_padding1, domain_in1) - sig1 = tf.reduce_sum(bn_out) - sig2 = tf.reduce_sum(bn_out * bn_out) - self.evaluate(tf.global_variables_initializer()) - - sig1_v, sig2_v = sess.run([sig1, sig2]) - self.assertAllClose(0.0, sig1_v, atol=1e-5) - self.assertAllClose(95.6266, sig2_v) - - -class GroupNormLayerTest(test_utils.TestCase, parameterized.TestCase): - - def testGroupNormLayerConstruction(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = bn_layers.GroupNormLayer.Params() - params.name = 'gn' - params.dim = 2 - params.num_groups = 2 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - bn_layers.GroupNormLayer(params) - gn_vars = tf.get_collection('GroupNormLayer_vars') - gn_var_names = [x.name for x in gn_vars] - expected_var_names = ['gn/beta/var:0', 'gn/gamma/var:0'] - self.assertEqual(expected_var_names, gn_var_names) - - def testGroupNormLayerFProp(self): - with self.session(use_gpu=True): - params = bn_layers.GroupNormLayer.Params() - params.name = 'gn' - params.dim = 4 - params.num_groups = 2 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - gn_in = tf.reshape(np.arange(32, dtype=np.float32), [2, 2, 2, 4]) - - gn_layer = bn_layers.GroupNormLayer(params) - gn_out = gn_layer.FPropDefaultTheta(gn_in) - - tf.global_variables_initializer().run() - base_block = np.array([[[-1.44440889, -1.22219217], - [-0.55554187, -0.33332515]], - [[0.33332515, 0.55554187], - [1.22219217, 1.44440889]]]) - expected_out = np.array([ - np.concatenate((base_block, base_block), -1), - np.concatenate((base_block, base_block), -1) - ]) - self.assertAllClose(expected_out, gn_out.eval(), atol=1e-5) - - def testGroupNormLayerFPropWithPaddings(self): - with self.session(use_gpu=True): - params = bn_layers.GroupNormLayer.Params() - params.name = 'gn' - params.dim = 4 - params.num_groups = 2 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - gn_in = tf.reshape(np.arange(32, dtype=np.float32), [2, 2, 2, 4]) - paddings = tf.convert_to_tensor([[0, 0], [0, 1]], dtype=tf.float32) - - gn_layer = bn_layers.GroupNormLayer(params) - gn_out, paddings_out = gn_layer.FPropDefaultTheta(gn_in, paddings) - - tf.global_variables_initializer().run() - base_block1 = np.array([[[-1.44440889, -1.22219217], - [-0.55554187, -0.33332515]], - [[0.33332515, 0.55554187], - [1.22219217, 1.44440889]]]) - - base_block2 = np.array([[[-1.2125355, -0.7275213], [0.7275213, - 1.2125355]], - [[2.6675782, 3.1525922], [4.607635, 5.092649]]]) - - expected_out = np.array([ - np.concatenate((base_block1, base_block1), -1), - np.concatenate((base_block2, base_block2), -1) - ]) - print(gn_out.eval()) - self.assertAllClose(expected_out, gn_out.eval(), atol=1e-5) - self.assertAllEqual(paddings.eval(), paddings_out.eval()) - - def testGroupNormLayerFPropWithPaddings3DInput(self): - with self.session(use_gpu=True): - params = bn_layers.GroupNormLayer.Params() - params.name = 'gn' - params.dim = 4 - params.num_groups = 2 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - gn_in = np.reshape(np.arange(16, dtype=np.float32), [2, 2, 1, 4]) - paddings = np.array([[0, 0], [0, 1]], dtype=np.float32) - - gn_layer = bn_layers.GroupNormLayer(params) - gn_out, paddings_out = gn_layer.FPropDefaultTheta( - tf.convert_to_tensor(gn_in), tf.convert_to_tensor(paddings)) - - params_3d = params.Copy().Set(input_rank=3, name='gn_3d') - gn_layer_3d = bn_layers.GroupNormLayer(params_3d) - gn_out_3d, paddings_out_3d = gn_layer_3d.FPropDefaultTheta( - tf.convert_to_tensor(gn_in.reshape([2, 2, 4])), - tf.convert_to_tensor(paddings)) - - tf.global_variables_initializer().run() - print(gn_out.eval()) - # Tests against reference. - self.assertAllClose( - gn_out_3d.eval(), gn_out.eval().reshape([2, 2, 4]), atol=1e-5) - self.assertAllEqual(paddings_out_3d.eval(), paddings_out.eval()) - - @parameterized.named_parameters(('4D',), ('3D', 3)) - def testGroupNormLayerFPropCumulativeMode(self, input_rank=4): - with self.session(use_gpu=True): - params = bn_layers.GroupNormLayer.Params() - params.name = 'gn' - params.dim = 2 - params.num_groups = 2 - params.cumulative = True - params.input_rank = input_rank - # gn_in[0]: [[0, 1], [2, 3], [4, 5], [6, 7]] - # gn_in[1]: [[8, 9], [10, 11], [12, 13], [14, 15]] - input_shape = [2, 4, 1, 2] if input_rank == 4 else [2, 4, 2] - gn_in = tf.reshape(np.arange(16, dtype=np.float32), input_shape) - paddings = tf.zeros([2, 4], tf.float32) - gn_layer = bn_layers.GroupNormLayer(params) - gn_out, _ = gn_layer.FPropDefaultTheta(gn_in, paddings) - - tf.global_variables_initializer().run() - base_block = np.array([[0., 0.], [1.4128014, 1.4128014], - [1.5487288, 1.5487288], [1.6033384, 1.6033384]]) - - expected_out = np.stack([base_block, base_block], - axis=0).reshape(input_shape) - self.assertAllClose(expected_out, gn_out.eval(), atol=1e-5) - - -class ConvLayerTest(test_utils.TestCase): - """Tests conv layers. - - Note that there are multiple subclasses of BaseConv2DLayer and most cases - are tested via the concrete Conv2DLayer. Other tests are done against - other subclasses to cover key differences. - """ - - def testConv2DLayerConstruction(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.Conv2DLayer.Params() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 32] - params.filter_stride = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - layers.Conv2DLayer(params) - conv_vars = tf.get_collection('Conv2DLayer_vars') - conv_var_names = [x.name for x in conv_vars] - expected_var_names = ['conv/w/var:0'] - self.assertEqual(expected_var_names, conv_var_names) - bn_vars = tf.get_collection('BatchNormLayer_vars') - bn_var_names = [x.name for x in bn_vars] - expected_var_names = [ - 'conv/beta/var:0', 'conv/gamma/var:0', 'conv/moving_mean/var:0', - 'conv/moving_variance/var:0' - ] - self.assertEqual(expected_var_names, bn_var_names) - - def testDepthwiseConv2DLayerConstruction(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.DepthwiseConv2DLayer.Params() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 32] - params.filter_stride = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - layers.DepthwiseConv2DLayer(params) - conv_vars = tf.get_collection('DepthwiseConv2DLayer_vars') - conv_var_names = [x.name for x in conv_vars] - expected_var_names = ['conv/w/var:0'] - self.assertEqual(expected_var_names, conv_var_names) - bn_vars = tf.get_collection('BatchNormLayer_vars') - bn_var_names = [x.name for x in bn_vars] - expected_var_names = [ - 'conv/beta/var:0', 'conv/gamma/var:0', 'conv/moving_mean/var:0', - 'conv/moving_variance/var:0' - ] - self.assertEqual(expected_var_names, bn_var_names) - - def testDepthwiseConv2DLayerModuleInterface(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.DepthwiseConv2DLayer.Params() - params.name = 'conv1' - params.filter_shape = [3, 3, 3, 32] - params.filter_stride = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - conv1 = layers.DepthwiseConv2DLayer(params) - params.name = 'conv2' - conv2 = layers.DepthwiseConv2DLayer(params) - - def ModuleName(m): - return m.name - - conv1_variables = [v.name for v in conv1.variables] - conv1_submodules = [ModuleName(v) for v in conv1.submodules] - conv2_variables = [v.name for v in conv2.variables] - conv2_submodules = [ModuleName(v) for v in conv2.submodules] - expected_conv1_vars = [ - 'global_step:0', 'conv1/w/var:0', 'conv1/moving_mean/var:0', - 'conv1/moving_variance/var:0', 'conv1/beta/var:0', 'conv1/gamma/var:0' - ] - expected_conv2_vars = [ - 'global_step:0', 'conv2/w/var:0', 'conv2/moving_mean/var:0', - 'conv2/moving_variance/var:0', 'conv2/beta/var:0', 'conv2/gamma/var:0' - ] - expected_conv1_modules = ['bbf_BatchNormLayer_conv1'] - expected_conv2_modules = ['bbf_BatchNormLayer_conv2'] - self.assertCountEqual(expected_conv1_vars, conv1_variables) - self.assertCountEqual(expected_conv2_vars, conv2_variables) - self.assertCountEqual(expected_conv1_modules, conv1_submodules) - self.assertCountEqual(expected_conv2_modules, conv2_submodules) - - def testSeparableConv2DLayerConstruction(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.SeparableConv2DLayer.Params() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 32] - params.filter_stride = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - params.Instantiate() - # Vars for the outer conv layer. - conv_vars = tf.get_collection('SeparableConv2DLayer_vars') - conv_var_names = [x.name for x in conv_vars] - expected_var_names = ['conv/w/var:0'] - self.assertSetEqual(set(expected_var_names), set(conv_var_names)) - # Vars for the inner depthwise layer. - conv_vars = tf.get_collection('DepthwiseConv2DLayer_vars') - conv_var_names = [x.name for x in conv_vars] - expected_var_names = ['conv/depthwise_conv/w/var:0'] - self.assertSetEqual(set(expected_var_names), set(conv_var_names)) - bn_vars = tf.get_collection('BatchNormLayer_vars') - bn_var_names = [x.name for x in bn_vars] - expected_var_names = [ - # Outer conv batchnorm. - 'conv/beta/var:0', - 'conv/gamma/var:0', - 'conv/moving_mean/var:0', - 'conv/moving_variance/var:0', - # Inner depthwise batchnorm. - 'conv/depthwise_conv/beta/var:0', - 'conv/depthwise_conv/gamma/var:0', - 'conv/depthwise_conv/moving_mean/var:0', - 'conv/depthwise_conv/moving_variance/var:0', - ] - self.assertSetEqual(set(expected_var_names), set(bn_var_names)) - - def testConv2DLayerWithBiasConstruction(self): - """Tests Conv2DLayer with only bias and without batch normalization.""" - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.Conv2DLayer.Params() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 32] - params.filter_stride = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - params.bias = True - params.batch_norm = False - layers.Conv2DLayer(params) - conv_vars = tf.get_collection('Conv2DLayer_vars') - conv_var_names = [x.name for x in conv_vars] - # Has both 'w' and 'b'. - expected_var_names = ['conv/w/var:0', 'conv/b/var:0'] - self.assertEqual(expected_var_names, conv_var_names) - # No BatchNorm variables. - bn_vars = tf.get_collection('BatchNormLayer_vars') - bn_var_names = [x.name for x in bn_vars] - expected_var_names = [] - self.assertEqual(expected_var_names, bn_var_names) - - def testDepthwiseConv2DLayerWithBiasConstruction(self): - """Tests DepthwiseConv2D with only bias and without batch normalization.""" - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.DepthwiseConv2DLayer.Params() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 32] - params.filter_stride = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - params.bias = True - params.batch_norm = False - layers.DepthwiseConv2DLayer(params) - conv_vars = tf.get_collection('DepthwiseConv2DLayer_vars') - conv_var_names = [x.name for x in conv_vars] - # Has both 'w' and 'b'. - expected_var_names = ['conv/w/var:0', 'conv/b/var:0'] - self.assertEqual(expected_var_names, conv_var_names) - # No BatchNorm variables. - bn_vars = tf.get_collection('BatchNormLayer_vars') - bn_var_names = [x.name for x in bn_vars] - expected_var_names = [] - self.assertEqual(expected_var_names, bn_var_names) - - def testConv2DLayerOutShape(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.Conv2DLayer.Params() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 32] - params.filter_stride = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - conv_layer = layers.Conv2DLayer(params) - in_shape = [None, None, 10, 3] - out_shape = conv_layer.OutShape(in_shape) - self.assertEqual(out_shape, [None, None, 5, 32]) - in_shape = [None, 20, 10, 3] - out_shape = conv_layer.OutShape(in_shape) - self.assertEqual(out_shape, [None, 10, 5, 32]) - - def testDepthwiseConv2DLayerOutShape(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.DepthwiseConv2DLayer.Params() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 32] - params.filter_stride = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - conv_layer = layers.DepthwiseConv2DLayer(params) - in_shape = [None, None, 10, 3] - out_shape = conv_layer.OutShape(in_shape) - self.assertEqual(out_shape, [None, None, 5, 96]) - in_shape = [None, 20, 10, 3] - out_shape = conv_layer.OutShape(in_shape) - self.assertEqual(out_shape, [None, 10, 5, 96]) - - def testSeparableConv2DLayerOutShape(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.SeparableConv2DLayer.Params() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 32] - params.filter_stride = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - conv_layer = params.Instantiate() - in_shape = [None, None, 10, 3] - out_shape = conv_layer.OutShape(in_shape) - self.assertEqual(out_shape, [None, None, 5, 32]) - in_shape = [None, 20, 10, 3] - out_shape = conv_layer.OutShape(in_shape) - self.assertEqual(out_shape, [None, 10, 5, 32]) - - def testConv2DLayerWithDilationOutShape(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.Conv2DLayer.Params() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 32] - params.filter_stride = [1, 1] - params.dilation_rate = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - conv_layer = layers.Conv2DLayer(params) - # dilation_rate does not change output shape. - in_shape = [None, None, 10, 3] - out_shape = conv_layer.OutShape(in_shape) - self.assertEqual(out_shape, [None, None, 10, 32]) - in_shape = [None, 20, 10, 3] - out_shape = conv_layer.OutShape(in_shape) - self.assertEqual(out_shape, [None, 20, 10, 32]) - - def testDepthwiseConv2DLayerWithDilationOutShape(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.DepthwiseConv2DLayer.Params() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 32] - params.filter_stride = [1, 1] - params.dilation_rate = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - conv_layer = layers.DepthwiseConv2DLayer(params) - # dilation_rate does not change output shape. - in_shape = [None, None, 10, 3] - out_shape = conv_layer.OutShape(in_shape) - self.assertEqual(out_shape, [None, None, 10, 96]) - in_shape = [None, 20, 10, 3] - out_shape = conv_layer.OutShape(in_shape) - self.assertEqual(out_shape, [None, 20, 10, 96]) - - def testSeparableConv2DLayerWithDilationOutShape(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.SeparableConv2DLayer.Params() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 32] - params.filter_stride = [1, 1] - params.dilation_rate = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - conv_layer = params.Instantiate() - # dilation_rate does not change output shape. - in_shape = [None, None, 10, 3] - out_shape = conv_layer.OutShape(in_shape) - self.assertEqual(out_shape, [None, None, 10, 32]) - in_shape = [None, 20, 10, 3] - out_shape = conv_layer.OutShape(in_shape) - self.assertEqual(out_shape, [None, 20, 10, 32]) - - def testConvPoolComputeOutPadding(self): - with self.session(use_gpu=True): - in_padding = tf.constant( - [[0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0], - [0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0]], - dtype=tf.float32) - out_padding = layers._ComputeConvOutputPadding(in_padding, 2, 2) - expected_out_padding = [[1, 1, 0, 0, 0, 1, 1, 0], - [1, 1, 0, 0, 0, 1, 1, 0]] - - self.evaluate(tf.global_variables_initializer()) - self.assertAllClose(expected_out_padding, out_padding.eval().tolist()) - - def testConvPoolComputeOutPaddingUnevenStride(self): - with self.session(use_gpu=True): - in_padding = tf.constant([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1], - [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1]], - dtype=tf.float32) - out_padding = layers._ComputeConvOutputPadding(in_padding, 3, 3) - expected_out_padding = [[0, 0, 0, 0, 1], [0, 0, 0, 1, 1], [0, 0, 1, 1, 1]] - - self.evaluate(tf.global_variables_initializer()) - self.assertAllClose(expected_out_padding, out_padding.eval().tolist()) - - def _checkConvLayerShapes(self, - input_shape, - filter_shape, - filter_stride, - dilation_rate=None, - depth_multiplier=None, - params_builder=layers.Conv2DLayer.Params): - g = tf.Graph() - with g.as_default(): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = params_builder() - params.name = 'conv' - params.filter_shape = filter_shape - params.filter_stride = filter_stride - if dilation_rate: - params.dilation_rate = dilation_rate - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - if depth_multiplier is not None: - params.depth_multiplier = depth_multiplier - conv_layer = params.Instantiate() - - inp = tf.random.uniform(input_shape) - inp_pad = tf.floor(0.5 + tf.random.uniform(input_shape[:2])) - out, out_pad = conv_layer.FPropDefaultTheta(inp, inp_pad) - - with self.session(use_gpu=True, graph=g): - self.evaluate(tf.global_variables_initializer()) - out, out_pad = self.evaluate([out, out_pad]) - print(out.shape, out_pad.shape) - # We expect conv_layer.OutShape can compute the actual output shape. - self.assertAllEqual(out.shape, conv_layer.OutShape(inp.shape.as_list())) - # We expect out_pad.shape matches the 1st 2 dimensions of out. - self.assertAllEqual(out.shape[:2], out_pad.shape) - - def testConv2DLayerOutputShapes(self): - self._checkConvLayerShapes([2, 4, 4, 3], [3, 3, 3, 32], [1, 1]) - self._checkConvLayerShapes([2, 4, 4, 3], [3, 3, 3, 32], [2, 2]) - self._checkConvLayerShapes([2, 10, 4, 3], [3, 3, 3, 32], [3, 3]) - - self._checkConvLayerShapes([2, 10, 4, 3], [3, 3, 3, 32], [1, 1], - dilation_rate=[2, 2]) - self._checkConvLayerShapes([2, 10, 4, 3], [3, 3, 3, 32], [1, 1], - dilation_rate=[3, 3]) - - def testDepthwiseConv2DLayerOutputShapes(self): - self._checkConvLayerShapes( - [2, 4, 4, 3], [3, 3, 3, 32], [1, 1], - params_builder=layers.DepthwiseConv2DLayer.Params) - self._checkConvLayerShapes( - [2, 4, 4, 3], [3, 3, 3, 32], [2, 2], - params_builder=layers.DepthwiseConv2DLayer.Params) - self._checkConvLayerShapes( - [2, 10, 4, 3], [3, 3, 3, 32], [3, 3], - params_builder=layers.DepthwiseConv2DLayer.Params) - - self._checkConvLayerShapes( - [2, 10, 4, 3], [3, 3, 3, 32], [1, 1], - dilation_rate=[2, 2], - params_builder=layers.DepthwiseConv2DLayer.Params) - self._checkConvLayerShapes( - [2, 10, 4, 3], [3, 3, 3, 32], [1, 1], - dilation_rate=[3, 3], - params_builder=layers.DepthwiseConv2DLayer.Params) - - def testSeparableConv2DLayerOutputShapes(self): - self._checkConvLayerShapes( - [2, 4, 4, 3], [3, 3, 3, 32], [1, 1], - params_builder=layers.SeparableConv2DLayer.Params) - self._checkConvLayerShapes( - [2, 4, 4, 3], [3, 3, 3, 32], [2, 2], - params_builder=layers.SeparableConv2DLayer.Params) - self._checkConvLayerShapes( - [2, 10, 4, 3], [3, 3, 3, 32], [3, 3], - params_builder=layers.SeparableConv2DLayer.Params) - # Dilations. - self._checkConvLayerShapes( - [2, 10, 4, 3], [3, 3, 3, 32], [1, 1], - dilation_rate=[2, 2], - params_builder=layers.SeparableConv2DLayer.Params) - self._checkConvLayerShapes( - [2, 10, 4, 3], [3, 3, 3, 32], [1, 1], - dilation_rate=[3, 3], - params_builder=layers.SeparableConv2DLayer.Params) - # Depth multiplier. - self._checkConvLayerShapes( - [2, 4, 4, 3], [3, 3, 3, 32], [1, 1], - params_builder=layers.SeparableConv2DLayer.Params, - depth_multiplier=2) - self._checkConvLayerShapes( - [2, 4, 4, 3], [3, 3, 3, 32], [2, 2], - params_builder=layers.SeparableConv2DLayer.Params, - depth_multiplier=6) - self._checkConvLayerShapes( - [2, 10, 4, 3], [3, 3, 3, 32], [3, 3], - params_builder=layers.SeparableConv2DLayer.Params, - depth_multiplier=12) - - def _evalConvLayerFProp(self, - params_builder=layers.Conv2DLayer.Params, - batch_norm=True, - weight_norm=False, - bias=False, - activation='RELU', - conv_last=False, - strides=(2, 2), - dilation_rate=(1, 1), - bn_fold_weights=False, - is_eval=False, - quantized=False): - self._ClearCachedSession() - tf.reset_default_graph() - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = params_builder() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 2] - params.filter_stride = strides - params.dilation_rate = dilation_rate - params.params_init = py_utils.WeightInit.Gaussian(0.1) - params.conv_last = conv_last - params.batch_norm = batch_norm - params.bn_fold_weights = bn_fold_weights - params.weight_norm = weight_norm - params.bias = bias - params.activation = activation - - if quantized: - params.qdomain.default = quant_utils.PassiveAsymQDomain.Params() - - conv_layer = params.Instantiate() - in_padding1 = tf.zeros([2, 4], dtype=tf.float32) - inputs1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 4, 3]), dtype=tf.float32) - - output1, _ = conv_layer.FPropDefaultTheta(inputs1, in_padding1) - output2, _ = conv_layer.FPropDefaultTheta(inputs1) - self.evaluate(tf.global_variables_initializer()) - v1, v2 = self.evaluate([output1, output2]) - self.assertAllClose(v1, v2) - return v1 - - def testConv2DLayerFProp(self): - # pyformat: disable - # pylint: disable=bad-whitespace - expected_output1 = [ - [[[ 0.36669245, 0.91488785], - [ 0.07532132, 0. ]], - [[ 0.34952009, 0. ], - [ 1.91783941, 0. ]]], - [[[ 0.28304493, 0. ], - [ 0. , 0. ]], - [[ 0. , 0.86575812], - [ 0. , 1.60203481]]]] - # pyformat: enable - # pylint: enable=bad-whitespace - actual = self._evalConvLayerFProp() - print('actual = ', np.array_repr(actual)) - self.assertAllClose(expected_output1, actual) - - def testDepthwiseConv2DLayerFProp(self): - # pyformat: disable - # pylint: disable=bad-whitespace - expected_output1 = [ - [[[ 0.93514717, 0.35602099, 0. , 0.51261222, 0. , - 1.4310323 ], - [ 0. , 0. , 0.49176404, 0. , 1.01494753, - 0.51337928]], - [[ 0.62087697, 0.34572476, 0. , 0.19352221, 0.47142431, - 0. ], - [ 0.81119895, 1.00890303, 0.90471351, 0. , 1.22736526, - 0. ]]], - [[[ 0. , 0. , 0.48927376, 0. , 0.74019426, - 0. ], - [ 0. , 0. , 1.49952257, 0. , 0. , - 0. ]], - [[ 0.29156703, 0. , 0. , 1.14509106, 0. , - 0.74238932], - [ 0.91312039, 1.39783907, 0. , 1.47650909, 0. , - 0.37969294]]]] - # pyformat: enable - # pylint: enable=bad-whitespace - actual = self._evalConvLayerFProp( - params_builder=layers.DepthwiseConv2DLayer.Params) - print('actual = ', np.array_repr(actual)) - self.assertAllClose(expected_output1, actual) - - def testSeparableConv2DLayerFProp(self): - # pyformat: disable - # pylint: disable=bad-whitespace - expected_output1 =[ - [[[ 0.39866772, 0. ], - [ 1.36471784, 0. ]], - [[ 0. , 0. ], - [ 0. , 0. ]]], - [[[ 1.15356529, 0.1036691 ], - [ 0.12865055, 0.61244327]], - [[ 0.03609803, 1.81620765], - [ 0. , 0.23052886]]]] - # pyformat: enable - # pylint: enable=bad-whitespace - actual = self._evalConvLayerFProp( - params_builder=layers.SeparableConv2DLayer.Params) - print('actual = ', np.array_repr(actual)) - self.assertAllClose(expected_output1, actual) - - def testConv2DLayerWithDilationFProp(self): - # pyformat: disable - # pylint: disable=bad-whitespace - expected_output1 = [ - [[[ 0. , 0.48857123], - [ 1.07320869, 0. ], - [ 0. , 0.1550007 ], - [ 0. , 1.59097648]], - [[ 0. , 0. ], - [ 0.20024362, 0. ], - [ 0. , 0.64265913], - [ 1.52903616, 0. ]], - [[ 0.099805 , 0. ], - [ 0. , 0.61720949], - [ 1.31608474, 0. ], - [ 0. , 0. ]], - [[ 0.0175612 , 0. ], - [ 0. , 0.17234094], - [ 0.21719536, 0. ], - [ 1.68514931, 0. ]]], - [[[ 1.45240796, 0. ], - [ 0. , 0. ], - [ 0.72675145, 1.971596 ], - [ 0. , 0.01062769]], - [[ 0. , 1.70299017], - [ 1.36936104, 1.29897082], - [ 1.40132439, 1.74345171], - [ 0.02585058, 0.29061913]], - [[ 0. , 0. ], - [ 0.32962656, 0.05025356], - [ 0. , 0. ], - [ 0. , 0. ]], - [[ 0.97244394, 0. ], - [ 0.23401484, 0.5722279 ], - [ 0. , 0.40940297], - [ 0. , 0.52711827]]]] - # pyformat: enable - # pylint: enable=bad-whitespace - actual = self._evalConvLayerFProp(strides=[1, 1], dilation_rate=[2, 2]) - print('testConvLayerWithDilationFProp actual = ', np.array_repr(actual)) - self.assertAllClose(expected_output1, actual, atol=1e-5) - - def testSeparableConv2DLayerWithDilationFProp(self): - # pyformat: disable - # pylint: disable=bad-whitespace - expected_output1 = [ - [[[ 0.21535617, 0.86965537], - [ 2.11499524, 1.2463783 ], - [ 0. , 0.39275286], - [ 0. , 0. ]], - [[ 1.12706482, 1.37450278], - [ 0. , 0. ], - [ 0. , 0. ], - [ 1.2390101 , 0.22932449]], - [[ 0. , 0. ], - [ 0.15051894, 1.32616639], - [ 0. , 0. ], - [ 0.72912866, 0.47753802]], - [[ 0.91655868, 0. ], - [ 0.88526261, 0.26690534], - [ 0. , 0.26084688], - [ 0.42923039, 0. ]]], - [[[ 0.82440329, 0. ], - [ 0.49015623, 0.52662987], - [ 0. , 0. ], - [ 0.35344127, 0. ]], - [[ 0. , 0. ], - [ 0. , 0. ], - [ 0.43848675, 0. ], - [ 0. , 1.21124518]], - [[ 1.1026746 , 1.39578998], - [ 0. , 0. ], - [ 0.34652925, 0. ], - [ 0. , 1.26868236]], - [[ 0.91519427, 0.09030763], - [ 0. , 0.59271163], - [ 0. , 0.54207176], - [ 0. , 0. ]]]] - # pyformat: enable - # pylint: enable=bad-whitespace - actual = self._evalConvLayerFProp( - strides=[1, 1], - dilation_rate=[2, 2], - params_builder=layers.SeparableConv2DLayer.Params) - print('testConvLayerWithDilationFProp actual = ', np.array_repr(actual)) - self.assertAllClose(expected_output1, actual) - - def testConv2DLayerConvFirstVsLastFProp(self): - """Compare results of conv first vs. last.""" - # ... with batch_norm and activation disabled. - self.assertAllClose( - self._evalConvLayerFProp( - batch_norm=False, activation='NONE', conv_last=False), - self._evalConvLayerFProp( - batch_norm=False, activation='NONE', conv_last=True)) - - def testConv2DLayerFPropConvLast(self): - # pyformat: disable - # pylint: disable=bad-whitespace - expected_output1 = [ - [[[ 0.22165056, 0.20731729], - [ 0.09577402, -0.15359652]], - [[ 0.07151584, 0.03027298], - [ 0.05370769, 0.0143405 ]]], - [[[-0.08854639, 0.06143938], - [-0.37708873, 0.00889082]], - [[-0.58154356, 0.30798748], - [-0.37575331, 0.54729235]]]] - # pyformat: enable - # pylint: enable=bad-whitespace - actual = self._evalConvLayerFProp(conv_last=True) - print(['ConvLast actual = ', np.array_repr(actual)]) - self.assertAllClose(expected_output1, actual) - - def testConv2DLayerConvWithBias(self): - """Compare results with bias vs. with neither batch_norm nor bias.""" - # Results should match since bias is initialized to be 0. - self.assertAllClose( - self._evalConvLayerFProp(batch_norm=False, bias=False), - self._evalConvLayerFProp(batch_norm=False, bias=True)) - - def testConv2DLayerWeightNormFProp(self): - # pyformat: disable - # pylint: disable=bad-whitespace - expected_output = [ - [[[ 0.37172362, 0.92405349], - [ 0.07635488, 0.]], - [[ 0.35431579, 0.], - [ 1.94415355, 0.]]], - [[[ 0.28692839, 0.], - [ 0. , 0.]], - [[ 0. , 0.87443149], - [ 0. , 1.61808443]]]] - # pyformat: enable - # pylint: enable=bad-whitespace - actual = self._evalConvLayerFProp(weight_norm=True) - print('actual1 = ', np.array_repr(actual)) - self.assertAllClose(expected_output, actual) - - def testDepthwiseConv2DLayerWeightNormFProp(self): - # pyformat: disable - # pylint: disable=bad-whitespace - expected_output = [ - [[[ 0.97023201, 0.37429881, 0. , 0.53157473, 0. , - 1.60764372], - [ 0. , 0. , 0.50401598, 0. , 1.07683432, - 0.57673818]], - [[ 0.644171 , 0.36347377, 0. , 0.20068097, 0.50016963, - 0. ], - [ 0.8416335 , 1.06069875, 0.92725372, 0. , 1.30220449, - 0. ]]], - [[[ 0. , 0. , 0.50146359, 0. , 0.78532791, - 0. ], - [ 0. , 0. , 1.53688192, 0. , 0. , - 0. ]], - [[ 0.302506 , 0. , 0. , 1.18745029, 0. , - 0.83401161], - [ 0.94737887, 1.46960247, 0. , 1.53112805, 0. , - 0.42655289]]]] - # pyformat: enable - # pylint: enable=bad-whitespace - actual = self._evalConvLayerFProp( - weight_norm=True, params_builder=layers.DepthwiseConv2DLayer.Params) - print('actual1 = ', np.array_repr(actual)) - self.assertAllClose(expected_output, actual) - - def testSeparableConv2DLayerWeightNormFProp(self): - # pyformat: disable - # pylint: disable=bad-whitespace - expected_output = [ - [[[ 0.41837293, 0. ], - [ 1.39592457, 0. ]], - [[ 0. , 0. ], - [ 0. , 0. ]]], - [[[ 1.20513153, 0.11938372], - [ 0.1284119 , 0.6927582 ]], - [[ 0.0227453 , 2.05591369], - [ 0. , 0.26530063]]]] - # pyformat: enable - # pylint: enable=bad-whitespace - actual = self._evalConvLayerFProp( - weight_norm=True, params_builder=layers.SeparableConv2DLayer.Params) - print('actual1 = ', np.array_repr(actual)) - self.assertAllClose(expected_output, actual) - - def testConv2DLayerFoldedBatchNormFProp(self): - actual_unfolded = self._evalConvLayerFProp( - batch_norm=True, bn_fold_weights=False) - actual_folded = self._evalConvLayerFProp( - batch_norm=True, bn_fold_weights=True) - print('testConvLayerFoldedBatchNormFProp folded = ', - np.array_repr(actual_folded)) - print('testConvLayerFoldedBatchNormFProp unfolded = ', - np.array_repr(actual_unfolded)) - self.assertAllClose(actual_folded, actual_unfolded) - - def testDepthwiseConv2DLayerFoldedBatchNormFProp(self): - actual_unfolded = self._evalConvLayerFProp( - batch_norm=True, - bn_fold_weights=False, - params_builder=layers.DepthwiseConv2DLayer.Params) - actual_folded = self._evalConvLayerFProp( - batch_norm=True, - bn_fold_weights=True, - params_builder=layers.DepthwiseConv2DLayer.Params) - print('testDepthwiseConvLayerFoldedBatchNormFProp folded = ', - np.array_repr(actual_folded)) - print('testDepthwiseConvLayerFoldedBatchNormFProp unfolded = ', - np.array_repr(actual_unfolded)) - self.assertAllClose(actual_folded, actual_unfolded) - - def testSeparableConv2DLayerFoldedBatchNormFProp(self): - actual_unfolded = self._evalConvLayerFProp( - batch_norm=True, - bn_fold_weights=False, - params_builder=layers.SeparableConv2DLayer.Params) - actual_folded = self._evalConvLayerFProp( - batch_norm=True, - bn_fold_weights=True, - params_builder=layers.SeparableConv2DLayer.Params) - print('testSeparableConvLayerFoldedBatchNormFProp folded = ', - np.array_repr(actual_folded)) - print('testSeparableConvLayerFoldedBatchNormFProp unfolded = ', - np.array_repr(actual_unfolded)) - self.assertAllClose(actual_folded, actual_unfolded) - - def testConvLayerFoldedBatchNormFPropEval(self): - actual_unfolded = self._evalConvLayerFProp( - batch_norm=True, bn_fold_weights=False, is_eval=True) - actual_folded = self._evalConvLayerFProp( - batch_norm=True, bn_fold_weights=True, is_eval=True) - print('testConvLayerFoldedBatchNormFPropEval folded = ', - np.array_repr(actual_folded)) - print('testConvLayerFoldedBatchNormFPropEval unfolded = ', - np.array_repr(actual_unfolded)) - self.assertAllClose(actual_folded, actual_unfolded) - - def testConv2DLayerNoPadding(self): - g = tf.Graph() - with g.as_default(): - tf.random.set_seed(24332) - p = layers.Conv2DLayerNoPadding.Params().Set( - name='test', filter_shape=(3, 3, 3, 5), filter_stride=(2, 2)) - l = p.Instantiate() - x = tf.random.normal(shape=[17, 64, 64, 3]) - y = l.FPropDefaultTheta(x) - - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - y_val = self.evaluate(y) - - self.assertEqual(y_val.shape, (17, 32, 32, 5)) - - def testConvLayerFoldedBatchNormFPropQuantized(self): - # pyformat: disable - # pylint: disable=bad-whitespace - expected_output = [ - [[[ 0.36997819, 0.91361964], - [ 0.07550576, 0. ]], - - [[ 0.35487702, 0. ], - [ 1.92539668, 0. ]]], - [[[ 0.27937129, 0. ], - [ 0. , 0. ]], - - [[ 0. , 0.86831617], - [ 0. , 1.59317136]]]] - # pyformat: enable - # pylint: enable=bad-whitespace - - actual_folded = self._evalConvLayerFProp( - batch_norm=True, bn_fold_weights=True, quantized=True) - print('testConvLayerFoldedBatchNormFPropQuantized folded = ', - np.array_repr(actual_folded)) - self.assertAllClose(actual_folded, expected_output) - - def testCausalConvLayerFProp(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.ConvLayer.Params() - params.name = 'conv' - params.filter_shape = [2, 1, 3, 2] - params.filter_stride = [1, 1] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - params.causal_convolution = True - params.activation = 'NONE' - params.batch_norm = False - - conv_layer = layers.ConvLayer(params) - in_padding1 = tf.zeros([2, 4], dtype=tf.float32) - inputs1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 3, 3]), dtype=tf.float32) - # Change the input for the last two steps. - inputs2 = tf.concat([inputs1[:, :2, :, :], inputs1[:, 2:, :, :] + 0.5], 1) - - output1, _ = conv_layer.FPropDefaultTheta(inputs1, in_padding1) - output2, _ = conv_layer.FPropDefaultTheta(inputs2, in_padding1) - self.evaluate(tf.global_variables_initializer()) - v1, v2 = self.evaluate([output1, output2]) - tf.logging.info('CausalConv output: %s', np.array_repr(v1)) - # pylint: disable=bad-whitespace,bad-continuation,line-too-long - self.assertAllClose(v1, [ - [[[-0.01093466, 0.00369835], - [ 0.03474921, 0.01418608], - [ 0.01887876, -0.00763734]], - [[-0.06922598, -0.04526342], - [-0.02428233, 0.02042499], - [-0.04504267, -0.01260209]], - [[-0.14253227, -0.11353028], - [-0.09067881, 0.03742362], - [ 0.01281691, 0.00644186]], - [[-0.06524619, -0.0555004 ], - [-0.18850081, -0.05325979], - [ 0.04960757, 0.05512709]]], - [[[-0.01077277, 0.03013588], - [ 0.00325067, -0.0223705 ], - [-0.00895232, 0.03310337]], - [[ 0.03113075, -0.02388876], - [ 0.03238059, 0.00590346], - [ 0.12839797, -0.02194144]], - [[-0.09115655, -0.06798521], - [-0.09801255, -0.01440183], - [-0.04321899, 0.00340509]], - [[-0.089603 , -0.07257183], - [-0.04469771, -0.0389927 ], - [-0.01747611, 0.00903451]]] - ]) # pyformat: disable - # pylint: enable=bad-whitespace,bad-continuation,line-too-long - self.assertAllClose(v1[:, :2, :, :], v2[:, :2, :, :]) - with self.assertRaises(AssertionError): - self.assertAllClose(v1[:, 2:, :, :], v2[:, 2:, :, :]) - - def testCausalConv2DLayerFProp(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.ConvLayer.Params() - params.name = 'causal_conv' - params.filter_shape = [2, 2, 3, 2] - params.filter_stride = [1, 1] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - params.causal_convolution = True - params.activation = 'NONE' - params.batch_norm = False - - conv_layer = layers.ConvLayer(params) - in_padding1 = tf.zeros([2, 4], dtype=tf.float32) - inputs1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 3, 3]), dtype=tf.float32) - - output1, _ = conv_layer.FPropDefaultTheta(inputs1, in_padding1) - self.evaluate(tf.global_variables_initializer()) - v1 = self.evaluate(output1) - tf.logging.info('CausalConv output: %s', np.array_repr(v1)) - # pylint: disable=bad-whitespace,bad-continuation,line-too-long - self.assertAllClose(v1, [ - [[[-0.065196 , -0.0597635 ], - [ 0.02871699, -0.02915794], - [-0.00529849, -0.02677475]], - [[-0.0227601 , 0.06118587], - [ 0.25884673, -0.13917476], - [ 0.03899311, -0.06894699]], - [[-0.28780231, -0.12121122], - [ 0.2447218 , 0.09553684], - [-0.07054863, 0.12110104]], - [[ 0.17036264, -0.00258163], - [ 0.28644818, -0.02746056], - [ 0.06173857, -0.11599959]]], - [[[ 0.1468567 , 0.12725323], - [-0.00131077, -0.03644447], - [ 0.0266833 , 0.01140832]], - [[-0.23816 , -0.07873908], - [-0.07348203, 0.25653225], - [-0.21931274, -0.0569509 ]], - [[-0.06972647, -0.03123237], - [ 0.07432974, -0.03340006], - [ 0.10474236, 0.00807726]], - [[ 0.07581483, 0.25381109], - [ 0.07091375, -0.14229891], - [ 0.05247882, -0.08783717]]] - ]) # pyformat: disable - # pylint: enable=bad-whitespace,bad-continuation,line-too-long - - def testCausalConv2DEqualsConv2DWithPadding(self): - # Causal conv is equivalent to regular conv with zero pre-padding. - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.ConvLayer.Params() - params.name = 'causal_conv' - params.filter_shape = [2, 2, 3, 2] - params.filter_stride = [1, 1] - params.params_init = py_utils.WeightInit.Gaussian(0.1, seed=12345) - - params.causal_convolution = True - params.activation = 'NONE' - params.batch_norm = False - causal_conv_layer = layers.ConvLayer(params) - - normal_conv_params = params.Copy() - normal_conv_params.name = 'conv' - normal_conv_params.causal_convolution = False - normal_conv_layer = layers.ConvLayer(normal_conv_params) - inputs1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 3, 3]), dtype=tf.float32) - # Causal conv with kernel height (time) = 2 requires prepadding size 1. - inputs1_pad = tf.concat([tf.zeros([2, 1, 3, 3]), inputs1], axis=1) - - output_causal, _ = causal_conv_layer.FPropDefaultTheta(inputs1) - output_normal, _ = normal_conv_layer.FPropDefaultTheta(inputs1_pad) - self.evaluate(tf.global_variables_initializer()) - v_causal, v_normal = self.evaluate([output_causal, output_normal]) - # Normal conv would produce an extra timestep due to SAME padding at the - # end. - self.assertAllClose(v_causal, v_normal[:, :-1]) - - def testCausalConv2DEqualsConv2DWithKernelHeightOne(self): - # When kernel height (time) = 1, causal convolution regresses to normal - # convolution with SAME padding. - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.ConvLayer.Params() - params.name = 'causal_conv' - params.filter_shape = [1, 2, 3, 2] - params.filter_stride = [1, 1] - params.params_init = py_utils.WeightInit.Gaussian(0.1, seed=12345) - - params.causal_convolution = True - params.activation = 'NONE' - params.batch_norm = False - causal_conv_layer = layers.ConvLayer(params) - - normal_conv_params = params.Copy() - normal_conv_params.name = 'conv' - normal_conv_params.causal_convolution = False - normal_conv_layer = layers.ConvLayer(normal_conv_params) - inputs1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 3, 3]), dtype=tf.float32) - - output_causal, _ = causal_conv_layer.FPropDefaultTheta(inputs1) - output_normal, _ = normal_conv_layer.FPropDefaultTheta(inputs1) - self.evaluate(tf.global_variables_initializer()) - v_causal, v_normal = self.evaluate([output_causal, output_normal]) - - self.assertAllClose(v_causal, v_normal) - - def testConvLayerBackProp(self): - with self.session(use_gpu=True) as sess: - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.ConvLayer.Params() - params.name = 'conv' - params.filter_shape = [3, 3, 3, 2] - params.filter_stride = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - conv_layer = layers.ConvLayer(params) - in_padding1 = tf.zeros([2, 4], dtype=tf.float32) - inputs1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 4, 3]), dtype=tf.float32) - output1, _ = conv_layer.FPropDefaultTheta(inputs1, in_padding1) - loss = tf.reduce_sum(output1) - - all_vars = tf.trainable_variables() - self.assertEqual(3, len(all_vars)) - - grads = tf.gradients(loss, all_vars) - self.evaluate(tf.global_variables_initializer()) - sym_grads = [sg.eval() for sg in grads] - num_grads = [ - test_utils.ComputeNumericGradient(sess, loss, v) for v in all_vars - ] - - for sg, ng in zip(sym_grads, num_grads): - self.assertAllClose(sg, ng, rtol=1e-02, atol=1e-02) - - def testConvLayerFPropTanh(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.ConvLayer.Params() - params.activation = 'TANH' - params.name = 'conv' - params.filter_shape = [3, 3, 3, 2] - params.filter_stride = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - conv_layer = layers.ConvLayer(params) - in_padding1 = tf.zeros([2, 4], dtype=tf.float32) - inputs1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 4, 3]), dtype=tf.float32) - - output1, _ = conv_layer.FPropDefaultTheta(inputs1, in_padding1) - self.evaluate(tf.global_variables_initializer()) - - # pyformat: disable - # pylint: disable=bad-whitespace - expected_output1 = [ - [[[ 0.35109526, 0.72346997], - [ 0.0751792 , -0.84315312]], - [[ 0.33594984, -0.18976833], - [ 0.95773894, -0.28015777]]], - [[[ 0.27572086, -0.26577294], - [-0.38503852, -0.88501388]], - [[-0.92332661, 0.69921255], - [-0.75103623, 0.9219743 ]]]] - # pyformat: enable - # pylint: enable=bad-whitespace - actual = output1.eval() - print(['actual = ', actual]) - self.assertAllClose(expected_output1, actual) - - def testConvSetLayerConstruction(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.ConvSetLayer.Params() - params.name = 'conv_set' - params.filter_shapes = [[3, 3, 3, 32], [8, 5, 3, 64]] - params.cnn_tpl.filter_stride = [2, 2] - params.cnn_tpl.params_init = py_utils.WeightInit.Gaussian(0.1) - - layers.ConvSetLayer(params) - - def _evalConvSetLayerFProp(self, - batch_norm=True, - bn_fold_weights=False, - weight_norm=False, - bias=False, - activation='RELU', - conv_last=False, - strides=(2, 2), - dilation_rate=(1, 1), - quantized=False, - dump_graphdef=False): - self._ClearCachedSession() - tf.reset_default_graph() - with self.session(use_gpu=True) as sess: - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.ConvSetLayer.Params() - params.name = 'conv_set' - params.filter_shapes = [[2, 2, 6, 1], [3, 5, 6, 3]] - params.cnn_tpl.filter_stride = strides - params.cnn_tpl.dilation_rate = dilation_rate - params.cnn_tpl.params_init = py_utils.WeightInit.Gaussian(0.1) - params.cnn_tpl.conv_last = conv_last - params.cnn_tpl.batch_norm = batch_norm - params.cnn_tpl.bn_fold_weights = bn_fold_weights - params.cnn_tpl.weight_norm = weight_norm - params.cnn_tpl.bias = bias - params.cnn_tpl.activation = activation - - if quantized: - params.qdomain.default = quant_utils.PassiveAsymQDomain.Params() - - conv_set_layer = layers.ConvSetLayer(params) - in_padding1 = tf.zeros([2, 4], dtype=tf.float32) - inputs1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 4, 6]), dtype=tf.float32) - - output1, _ = conv_set_layer.FPropDefaultTheta(inputs1, in_padding1) - self.evaluate(tf.global_variables_initializer()) - - if dump_graphdef: - print('ConvSet GraphDef:', sess.graph.as_graph_def()) - assert False, 'Disable "dump_graphdef" before submit' - - return output1.eval() - - def testConvSetLayerFProp(self): - # pyformat: disable - # pylint: disable=bad-whitespace,bad-continuation - expected_output1 = [ - [[[ 1.04307961, 0. , 1.27613628, 0. ], - [ 0. , 0. , 0. , 1.21081829 ]], - [[ 0. , 0.18475296, 0. , 0. ], - [ 1.34087086 , 2.2726357 , 0. , 0. ]]], - [[[ 0. , 0.25231963, 0. , 0. ], - [ 1.13677704 , 0. , 0.996117 , 1.836285 ]], - [[ 0. , 0. , 1.04101253, 0. ], - [ 0.12628449 , 0.37599814, 0.3134549 , 0.51208746 ]]] - ] - # pyformat: enable - # pylint: enable=bad-whitespace,bad-continuation - actual = self._evalConvSetLayerFProp() - print(['actual = ', np.array_repr(actual)]) - self.assertAllClose(expected_output1, actual) - - def testConvSetLayerFPropQuantized(self): - # pyformat: disable - # pylint: disable=bad-whitespace,bad-continuation - expected_output1 = [ - [[[ 1.04016984, 0. , 1.28103447, 0. ], - [ 0. , 0. , 0. , 1.20986581]], - [[ 0. , 0.18681753, 0. , 0. ], - [ 1.35328221, 2.26849842, 0. , 0. ]]], - [[[ 0. , 0.24909003, 0. , 0. ], - [ 1.14100266, 0. , 0.98746401, 1.83259094]], - [[ 0. , 0. , 1.04084051, 0. ], - [ 0.12736773, 0.38253111, 0.32025862, 0.5159722 ]]]] - # pyformat: enable - # pylint: enable=bad-whitespace,bad-continuation - actual = self._evalConvSetLayerFProp(bn_fold_weights=True, quantized=True) - # Note that we don't have many ways to verify in a unit test that the - # quant nodes were added properly; however, if their placement changes, - # it will very likely perturb the golden values above. If digging deeper, - # add 'dump_graphdef=True' to the above call and inspect the graphdef: - # There should be one layer of fake_quant* nodes before the ConcatV2. - print('actual = ', np.array_repr(actual)) - self.assertAllClose(expected_output1, actual) - - # TODO(yonghui): more test for convolution layer - - -class PoolingLayerTest(test_utils.TestCase, parameterized.TestCase): - - # TODO(lingvo): fix 'VALID' padding in pooling. - @parameterized.named_parameters( - { - 'testcase_name': 'max_pooling_same_padding', - 'pooling_type': 'MAX', - 'padding_algorithm': 'SAME', - 'window_shape': (3, 1), - 'window_stride': (1, 1), - 'inputs': np.array([-2, -10, -3, -4, 0, 0]), - 'input_paddings': np.array([0, 0, 0, 0, 1, 1]), - 'expected_output': np.array([-2, -2, -3, -3, 0, 0]), - 'expected_output_padding': np.array([0, 0, 0, 0, 1, 1]), - 'expected_output_without_padding': np.array([-2, -2, -3, 0, 0, 0]), - }, { - 'testcase_name': 'avg_pooling_same_padding', - 'pooling_type': 'AVG', - 'padding_algorithm': 'SAME', - 'window_shape': (3, 1), - 'window_stride': (1, 1), - 'inputs': np.array([-2, 0, 2, 4, 0, 0]), - 'input_paddings': np.array([0, 0, 0, 0, 1, 1]), - 'expected_output': np.array([-1, 0, 2, 3, 0, 0]), - 'expected_output_padding': np.array([0, 0, 0, 0, 1, 1]), - 'expected_output_without_padding': np.array([-1, 0, 2, 2, 4 / 3, 0]), - }) - def testSimpleCases(self, inputs, input_paddings, pooling_type, window_shape, - window_stride, padding_algorithm, expected_output, - expected_output_padding, expected_output_without_padding): - inputs = inputs[np.newaxis, :, np.newaxis, np.newaxis] - input_paddings = input_paddings[np.newaxis, :] - param = layers.PoolingLayer.Params().Set( - name='test_layer', - pooling_type=pooling_type, - window_shape=window_shape, - window_stride=window_stride, - padding_algorithm=padding_algorithm) - pooling_layer = param.Instantiate() - with self.session(use_gpu=True) as sess: - inputs = tf.convert_to_tensor(inputs, dtype=tf.float32) - input_paddings = tf.convert_to_tensor(input_paddings, dtype=tf.float32) - output, output_paddings = pooling_layer.FPropDefaultTheta( - inputs, input_paddings) - output_without_padding, _ = pooling_layer.FPropDefaultTheta(inputs) - tf.global_variables_initializer().run() - output_val, output_paddings_val, output_without_padding_val = sess.run( - [output, output_paddings, output_without_padding]) - - self.assertAllClose(expected_output, output_val.flatten()) - self.assertAllEqual(expected_output_padding, output_paddings_val.flatten()) - self.assertAllClose(expected_output_without_padding, - output_without_padding_val.flatten()) - - def testPoolLayerFProp(self): - with self.session(use_gpu=True): - params = layers.PoolingLayer.Params() - params.name = 'pool' - params.window_shape = [3, 3] - params.window_stride = [1, 2] - - pool_layer = layers.PoolingLayer(params) - in_padding1 = tf.zeros([2, 4], dtype=tf.float32) - inputs1 = tf.constant( - np.arange(96, dtype='float32').reshape([2, 4, 4, 3]), - dtype=tf.float32) - - output1, _ = pool_layer.FPropDefaultTheta(inputs1, in_padding1) - self.evaluate(tf.global_variables_initializer()) - print([np.array_repr(output1.eval())]) - # pyformat: disable - expected_output1 = [ - [[[18., 19., 20.], - [21., 22., 23.]], - [[30., 31., 32.], - [33., 34., 35.]], - [[42., 43., 44.], - [45., 46., 47.]], - [[42., 43., 44.], - [45., 46., 47.]]], - [[[66., 67., 68.], - [69., 70., 71.]], - [[78., 79., 80.], - [81., 82., 83.]], - [[90., 91., 92.], - [93., 94., 95.]], - [[90., 91., 92.], - [93., 94., 95.]]]] - # pyformat: enable - self.assertAllClose(expected_output1, output1.eval()) - - def testPoolLayerMoreShapes(self): - with self.session(use_gpu=True): - for window_shape, window_stride in [ - [[3, 3], [1, 2]], - [[2, 2], [1, 2]], - [[3, 4], [1, 3]], - ]: - params = layers.PoolingLayer.Params() - params.name = 'pool' - params.window_shape = window_shape - params.window_stride = window_stride - - pool_layer = layers.PoolingLayer(params) - in_padding1 = tf.zeros([2, 4], dtype=tf.float32) - inputs1 = tf.constant( - np.arange(96, dtype='float32').reshape([2, 4, 4, 3]), - dtype=tf.float32) - - output1, _ = pool_layer.FPropDefaultTheta(inputs1, in_padding1) - - output2 = tf.nn.max_pool(inputs1, [1] + params.window_shape + [1], - [1] + params.window_stride + [1], 'SAME') - - predicted_out_shape = pool_layer.OutShape(inputs1.shape.as_list()) - - self.evaluate(tf.global_variables_initializer()) - output1_v = output1.eval() - self.assertAllClose(output2.eval(), output1_v) - self.assertAllClose(predicted_out_shape, output1_v.shape) - - -class BlurPoolLayerTest(test_utils.TestCase): - - def _testBlurPool(self, subsample_type, blur_filter, expected_output): - with self.session(use_gpu=True): - p = layers.BlurPoolLayer.Params().Set( - name='blur_pool', - input_channels=3, - subsample_type=subsample_type, - blur_filter=blur_filter) - - layer = p.Instantiate() - in_padding1 = tf.convert_to_tensor([[0, 0, 0, 1], [0, 0, 1, 1]], - dtype=tf.float32) - expected_out_padding = [[0, 1], [0, 1]] - inputs1 = tf.constant( - np.arange(24, dtype='float32').reshape([2, 4, 1, 3]), - dtype=tf.float32) - - output1, out_padding1 = layer.FPropDefaultTheta(inputs1, in_padding1) - self.evaluate(tf.global_variables_initializer()) - print([np.array_repr(output1.eval())]) - self.assertAllClose(expected_output, output1.eval()) - self.assertAllClose(expected_out_padding, out_padding1.eval()) - - def testBlurPool1D(self): - expected_output = np.array([[[[1.125, 1.8125, 2.5]], [[0, 0, 0]]], - [[[8.25, 8.875, 9.5]], [[0, 0, 0]]]], - dtype=np.float32) - self._testBlurPool('1D', 'B5', expected_output) - - def testBlurPool2D(self): - expected_output = np.array([[[[0.421875, 0.6796875, 0.9375]], [[0, 0, 0]]], - [[[3.09375, 3.328125, 3.5625]], [[0, 0, 0]]]], - dtype=np.float32) - self._testBlurPool('2D', 'B5', expected_output) - - -class ProjectionLayerTest(test_utils.TestCase, parameterized.TestCase): - - def testProjectionLayerConstruction(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.ProjectionLayer.Params() - params.name = 'proj' - params.input_dim = 2 - params.output_dim = 3 - params.batch_norm = True - params.params_init = py_utils.WeightInit.Gaussian(0.1) - layers.ProjectionLayer(params) - proj_vars = tf.get_collection('ProjectionLayer_vars') - proj_var_names = [x.name for x in proj_vars] - self.assertEqual(['proj/w/var:0'], proj_var_names) - bn_vars = tf.get_collection('BatchNormLayer_vars') - bn_var_names = [x.name for x in bn_vars] - expected_var_names = [ - 'proj/beta/var:0', 'proj/gamma/var:0', 'proj/moving_mean/var:0', - 'proj/moving_variance/var:0' - ] - self.assertEqual(expected_var_names, bn_var_names) - - def _evalProjectionLayer(self, - reshape_to_2d=False, - batch_norm=True, - weight_norm=False, - activation='RELU', - affine_last=False, - input_dim=3, - output_dim=2, - quantized=False, - has_bias=False, - bn_fold_weights=None, - expect_bn_fold_weights=None, - is_eval=False, - layer_callback=None, - bn_decay=0.999, - bn_use_moving_avg_in_training=False, - use_einsum=True, - block_dim=0): - self._ClearCachedSession() - tf.reset_default_graph() - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.ProjectionLayer.Params() - params.name = 'proj' - params.input_dim = input_dim - params.output_dim = output_dim - params.has_bias = has_bias - if has_bias: - params.bias_init = 5.0 - params.activation = activation - params.batch_norm = batch_norm - params.weight_norm = weight_norm - params.affine_last = affine_last - params.params_init = py_utils.WeightInit.Gaussian(0.1) - params.bn_fold_weights = bn_fold_weights - params.bn_params.decay = bn_decay - params.bn_params.use_moving_avg_in_training = bn_use_moving_avg_in_training - params.use_einsum = use_einsum - params.block_dim = block_dim - params.use_blocked_matmul = True if block_dim > 0 else False - - if quantized: - cc_schedule = quant_utils.FakeQuantizationSchedule.Params().Set( - clip_end_step=1, quant_start_step=1) - qdomain_default = quant_utils.SymmetricScheduledClipQDomain.Params( - ).Set(cc_schedule=cc_schedule.Copy()) - params.qdomain.default = qdomain_default.Copy() - - in_padding = tf.zeros([2, 4, 1], dtype=tf.float32) - inputs = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, input_dim]), dtype=tf.float32) - if reshape_to_2d: - in_padding = tf.reshape(in_padding, [-1, 1]) - inputs = tf.reshape(inputs, [-1, input_dim]) - - proj_layer = layers.ProjectionLayer(params) - if layer_callback: - layer_callback(proj_layer) - if expect_bn_fold_weights is not None: - self.assertEqual(expect_bn_fold_weights, proj_layer._is_bn_folded) - - output = proj_layer.FPropDefaultTheta(inputs, in_padding) - self.evaluate(tf.global_variables_initializer()) - if quantized: - # Put it in the fully quantized range. - self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), 5)) - return output.eval() - - def testProjectionLayerFProp(self): - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [[ 0. , 0.33779466], - [ 0.4527415 , 0.99911398], - [ 0.44320837, 0. ], - [ 0. , 0.04557215]], - [[ 0.69273949, 0. ], - [ 0.30908319, 0. ], - [ 0. , 0. ], - [ 0. , 1.54578114]]] - # pyformat: enable - # pylint: enable=bad-whitespace - for reshape_to_2d in (False, True): - actual = self._evalProjectionLayer( - reshape_to_2d=reshape_to_2d, expect_bn_fold_weights=False) - if reshape_to_2d: - expected_output = np.reshape(np.array(expected_output), (-1, 2)) - tf.logging.info('expected = %s', expected_output) - tf.logging.info('actual = %s', np.array_repr(actual)) - self.assertAllClose(expected_output, actual) - - def testProjectionLayerFPropWithBias(self): - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [[ 4.98987579, 5.03493643], - [ 5.01192808, 5.0917592 ], - [ 5.01156807, 4.99741936], - [ 4.96849394, 5.00982761]], - [[ 5.02098131, 4.98014927], - [ 5.00650883, 4.87676954], - [ 4.98995209, 4.91770315], - [ 4.95948696, 5.138731 ]]] - # pyformat: enable - # pylint: enable=bad-whitespace - # Tested without batch_norm because batch_norm will mostly cancel out the - # affect of bias. - actual = self._evalProjectionLayer( - has_bias=True, - batch_norm=False, - expect_bn_fold_weights=False, - activation='RELU6') - tf.logging.info('expected = %s', expected_output) - tf.logging.info('actual = %s', np.array_repr(actual)) - self.assertAllClose(expected_output, actual) - - def testProjectionLayerExplicitFolding(self): - unfolded = self._evalProjectionLayer( - bn_fold_weights=False, expect_bn_fold_weights=False) - folded = self._evalProjectionLayer( - bn_fold_weights=True, expect_bn_fold_weights=True) - tf.logging.info('unfolded = %s', np.array_repr(unfolded)) - tf.logging.info('folded = %s', np.array_repr(folded)) - self.assertAllClose(folded, unfolded) - - def testProjectionLayerExplicitFoldingEval(self): - unfolded = self._evalProjectionLayer( - bn_fold_weights=False, expect_bn_fold_weights=False, is_eval=True) - folded = self._evalProjectionLayer( - bn_fold_weights=True, expect_bn_fold_weights=True, is_eval=True) - tf.logging.info('unfolded = %s', np.array_repr(unfolded)) - tf.logging.info('folded = %s', np.array_repr(folded)) - self.assertAllClose(folded, unfolded) - - def testProjectionLayerExplicitFoldingNoBatchNorm(self): - unfolded = self._evalProjectionLayer( - batch_norm=False, bn_fold_weights=False, expect_bn_fold_weights=False) - # Note that weight folding will report as disabled because batch norm is - # disabled. - folded = self._evalProjectionLayer( - batch_norm=False, bn_fold_weights=True, expect_bn_fold_weights=False) - tf.logging.info('unfolded = %s', np.array_repr(unfolded)) - tf.logging.info('folded = %s', np.array_repr(folded)) - self.assertAllClose(folded, unfolded) - - def testProjectionLayerExplicitFoldingWithWeightNorm(self): - unfolded = self._evalProjectionLayer( - weight_norm=True, bn_fold_weights=False, expect_bn_fold_weights=False) - folded = self._evalProjectionLayer( - weight_norm=True, bn_fold_weights=True, expect_bn_fold_weights=True) - tf.logging.info('unfolded = %s', np.array_repr(unfolded)) - tf.logging.info('folded = %s', np.array_repr(folded)) - self.assertAllClose(folded, unfolded) - - def testProjectionLayerWeightNorm(self): - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [[ 0. , 0.36285588], - [ 0.82909501, 1.07323885], - [ 0.81163716, 0. ], - [ 0. , 0.04895319]], - [[ 1.26859784, 0. ], - [ 0.56601691, 0. ], - [ 0. , 0. ], - [ 0. , 1.66046333]]] - # pyformat: enable - # pylint: enable=bad-whitespace - for reshape_to_2d in (False, True): - actual = self._evalProjectionLayer( - reshape_to_2d=reshape_to_2d, weight_norm=True) - if reshape_to_2d: - expected_output = np.reshape(np.array(expected_output), (-1, 2)) - tf.logging.info('expected = %s', expected_output) - tf.logging.info('actual = %s', np.array_repr(actual)) - self.assertAllClose(expected_output, actual) - - def testProjectionLayerAffineFirstVsLastFProp(self): - """Compare results of affine first vs. last.""" - # ... with batch_norm and activation disabled. - self.assertAllClose( - self._evalProjectionLayer( - batch_norm=False, activation='NONE', affine_last=False), - self._evalProjectionLayer( - batch_norm=False, activation='NONE', affine_last=True)) - - def testProjectionLayerAffineLastFProp(self): - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output1 = [ - [[ 0. , 0. ], - [ 0.03410175, 0.04741348], - [ 0.02665393, -0.02072855], - [-0.01116518, -0.06280501]], - [[ 0.04615254, -0.03589247], - [-0.00376316, -0.0464084 ], - [-0.01111402, -0.13706152], - [-0.02596203, 0.16340451]]] - # pyformat: enable - # pylint: enable=bad-whitespace - actual = self._evalProjectionLayer(affine_last=True) - print(['actual = ', np.array_repr(actual)]) - self.assertAllClose(expected_output1, actual) - - def testProjectionLayerBackProp(self): - with self.session(use_gpu=True) as sess: - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.ProjectionLayer.Params() - params.name = 'proj' - params.dtype = tf.float64 - params.input_dim = 3 - params.output_dim = 2 - params.batch_norm = True - params.params_init = py_utils.WeightInit.Gaussian(0.01) - - proj_layer = layers.ProjectionLayer(params) - in_padding1 = tf.zeros([2, 4, 1], dtype=tf.float64) - inputs1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 3]), dtype=tf.float64) - output1 = proj_layer.FPropDefaultTheta(inputs1, in_padding1) - loss = tf.reduce_sum(output1) - - all_vars = tf.trainable_variables() - self.assertLen(all_vars, 3) - - grads = tf.gradients(loss, all_vars) - self.evaluate(tf.global_variables_initializer()) - sym_grads = [sg.eval() for sg in grads] - num_grads = [ - test_utils.ComputeNumericGradient(sess, loss, v, 1e-6) - for v in all_vars - ] - - for sg, ng in zip(sym_grads, num_grads): - self.assertAllClose(sg, ng, rtol=1e-06, atol=1e-06) - - def testProjectionLayerFPropQuantizedWithUnfusedActivation(self): - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [[-0.1328125, 0.3125 ], - [ 0.421875 , 0.734375 ], - [ 0.421875 , -0.109375 ], - [-0.6015625, 0.0078125]], - [[ 0.6015625, -0.3046875], - [ 0.3046875, -0.7578125], - [-0.125 , -0.7578125], - [-0.734375 , 0.7578125]]] - # pyformat: enable - # pylint: enable=bad-whitespace - def CheckLayer(proj_layer): - # Should not error because this qtensor is defined. - proj_layer.QTensor('activation', tf.convert_to_tensor(0.)) - # The intermediate tensor should be defined. - proj_layer.QTensor('affine_matmul', tf.convert_to_tensor(0.)) - - # When quantization enabled, batchnorm folding should auto enable. - # TANH is unfused. - actual = self._evalProjectionLayer( - activation='TANH', - quantized=True, - expect_bn_fold_weights=True, - layer_callback=CheckLayer) - tf.logging.info('expected = %s', expected_output) - tf.logging.info('actual = %s', np.array_repr(actual)) - self.assertAllClose(expected_output, actual) - - def testProjectionLayerFPropQuantizedWithFusedActivation(self): - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [[ 0. , 0.3203125], - [ 0.453125 , 0.9375 ], - [ 0.4453125, 0. ], - [ 0. , 0.0078125]], - [[ 0.6953125, 0. ], - [ 0.3125 , 0. ], - [ 0. , 0. ], - [ 0. , 0.9921875]]] - # pyformat: enable - # pylint: enable=bad-whitespace - def CheckLayer(proj_layer): - # Should not error because this qtensor is defined. - proj_layer.QTensor('activation', tf.convert_to_tensor(0.)) - with self.assertRaises(AssertionError): - # The intermediate tensor should *not* be quantized. - proj_layer.QTensor('affine_matmul', tf.convert_to_tensor(0.)) - - # When quantization enabled, batchnorm folding should auto enable. - # RELU6 is fused. - actual = self._evalProjectionLayer( - activation='RELU6', - quantized=True, - expect_bn_fold_weights=True, - layer_callback=CheckLayer) - tf.logging.info('expected = %s', expected_output) - tf.logging.info('actual = %s', np.array_repr(actual)) - self.assertAllClose(expected_output, actual) - - def testProjectionLayerFPropQuantizedOnlyMatmul(self): - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [[-0.0078125, 0.0390625], - [ 0.0078125, 0.09375 ], - [ 0.0078125, 0. ], - [-0.03125 , 0.015625 ]], - [[ 0.015625 , -0.015625 ], - [ 0.0078125, -0.125 ], - [-0.0078125, -0.078125 ], - [-0.0390625, 0.1484375]]] - # pyformat: enable - # pylint: enable=bad-whitespace - def CheckLayer(proj_layer): - # Should not error because this qtensor is defined. - proj_layer.QTensor('affine_matmul', tf.convert_to_tensor(0.)) - - actual = self._evalProjectionLayer( - activation='NONE', - quantized=True, - batch_norm=False, - expect_bn_fold_weights=False, - layer_callback=CheckLayer) - tf.logging.info('expected = %s', expected_output) - tf.logging.info('actual = %s', np.array_repr(actual)) - self.assertAllClose(expected_output, actual) - - def testProjectionLayerFPropQuantizedOnlyMatmulBias(self): - # pylint: disable=bad-whitespace - # pyformat: disable - # Saturated because of the out of range bias. - expected_output = [[[0.9921875, 0.9921875], [0.9921875, 0.9921875], - [0.9921875, 0.9921875], [0.9921875, 0.9921875]], - [[0.9921875, 0.9921875], [0.9921875, 0.9921875], - [0.9921875, 0.9921875], [0.9921875, 0.9921875]]] - - # pyformat: enable - # pylint: enable=bad-whitespace - def CheckLayer(proj_layer): - # Should not error because this qtensor is defined. - proj_layer.QTensor('affine_matmul', tf.convert_to_tensor(0.)) - - actual = self._evalProjectionLayer( - activation='NONE', - quantized=True, - has_bias=True, - batch_norm=False, - expect_bn_fold_weights=False, - layer_callback=CheckLayer) - tf.logging.info('expected = %s', expected_output) - tf.logging.info('actual = %s', np.array_repr(actual)) - self.assertAllClose(expected_output, actual) - - def testFCLayerConstruction(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.FCLayer.Params() - params.name = 'fc' - params.input_dim = 2 - params.output_dim = 3 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - layers.FCLayer(params) - proj_vars = tf.get_collection('FCLayer_vars') - proj_var_names = [x.name for x in proj_vars] - expected_var_names = ['fc/w/var:0', 'fc/b/var:0'] - self.assertEqual(expected_var_names, proj_var_names) - - def testFCLayerFProp(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.FCLayer.Params() - params.name = 'fc' - params.input_dim = 3 - params.output_dim = 2 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - proj_layer = layers.FCLayer(params) - inputs = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 3]), dtype=tf.float32) - - output = proj_layer.FPropDefaultTheta(inputs) - self.evaluate(tf.global_variables_initializer()) - - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [[ 0. , 0.04883499], - [ 0.17094055, 0. ], - [ 0.09287541, 0. ], - [ 0. , 0.19471419]], - [[ 0.15290432, 0. ], - [ 0. , 0. ], - [ 0. , 0.10548697], - [ 0. , 0.22610095]]] - # pyformat: enable - # pylint: enable=bad-whitespace - actual = output.eval() - print(['actual = ', np.array_repr(actual)]) - self.assertAllClose(expected_output, actual) - - def testFCLayerBackProp(self): - with self.session(use_gpu=True) as sess: - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.FCLayer.Params() - params.name = 'fc' - params.dtype = tf.float64 - params.input_dim = 3 - params.output_dim = 2 - params.params_init = py_utils.WeightInit.Gaussian(0.01) - - proj_layer = layers.FCLayer(params) - inputs = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 3]), dtype=tf.float64) - output = proj_layer.FPropDefaultTheta(inputs) - loss = tf.reduce_sum(output) - - all_vars = tf.trainable_variables() - self.assertLen(all_vars, 2) - - grads = tf.gradients(loss, all_vars) - self.evaluate(tf.global_variables_initializer()) - sym_grads = [sg.eval() for sg in grads] - num_grads = [ - test_utils.ComputeNumericGradient(sess, loss, v, 1e-6) - for v in all_vars - ] - - for sg, ng in zip(sym_grads, num_grads): - self.assertAllClose(sg, ng, rtol=1e-06, atol=1e-06) - - def testProjectionLayerFPropUsingMovingAvgInTraining(self): - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [[[0. , 0.03491905], - [0.01192194, 0.09171353], - [0.01156251, 0. ], - [0. , 0.00982281]], - [[0.02097072, 0. ], - [0.00650552, 0. ], - [0. , 0. ], - [0. , 0.13866161]]] - # pyformat: enable - # pylint: enable=bad-whitespace - for reshape_to_2d in (False, True): - actual = self._evalProjectionLayer( - reshape_to_2d=reshape_to_2d, - expect_bn_fold_weights=False, - bn_use_moving_avg_in_training=True) - if reshape_to_2d: - expected_output = np.reshape(np.array(expected_output), (-1, 2)) - tf.logging.info('expected = %s', expected_output) - tf.logging.info('actual = %s', np.array_repr(actual)) - self.assertAllClose(expected_output, actual) - - def testProjectionLayerFPropEinsum(self): - output_with_einsum = self._evalProjectionLayer(use_einsum=True) - output_without_einsum = self._evalProjectionLayer(use_einsum=False) - self.assertAllClose(output_with_einsum, output_without_einsum) - - def testProjectionLayerFPropBlockMatmul(self): - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [[[0., 0., 0., 0.], - [0., 0.03153732, 0.01891183, 0.02154643], - [0.01373154, 0.09479743, 0., 0.03639744], - [0.05870617, 0.19162716, 0.05567084, 0.15362406]], - - [[0.06591958, 0.2321615, 0., 0.14735883], - [0.1003774, 0.33637568, 0., 0.22276597], - [0.02362791, 0.10728893, 0.01922233, 0.07299631], - [0., 0., 0., 0. ]]] - # pyformat: enable - # pylint: enable=bad-whitespace - output_with_block_matmul = self._evalProjectionLayer( - input_dim=4, - output_dim=4, - batch_norm=False, - use_einsum=False, - block_dim=2) - tf.logging.info(output_with_block_matmul) - self.assertAllClose(output_with_block_matmul, expected_output) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [[[0., 0., 0., 0., 0.], - [0., 0., 0.06118044, 0., 0.02968791], - [0., 0., 0.09687695, 0., 0.], - [0.10228965, 0., 0.01826946, 0.0219113, 0.16076824]], - - [[0.03506518, 0.27432495, 0.25932777, 0., 0.], - [0., 0.00882578, 0.06655132, 0., 0.], - [0., 0., 0., 0.10196716, 0.], - [0., 0., 0.08253407, 0., 0.]]] - # pyformat: enable - # pylint: enable=bad-whitespace - # Test case for odd input and output dimensions. - output_with_block_matmul = self._evalProjectionLayer( - input_dim=5, - output_dim=5, - batch_norm=False, - use_einsum=False, - block_dim=2) - tf.logging.info(output_with_block_matmul) - self.assertAllClose(output_with_block_matmul, expected_output) - - @parameterized.named_parameters( - { - 'testcase_name': 'RELU', - 'activation': 'RELU', - 'input_dims': [2, 4, 3], - 'expected_extra_flops': 0, - 'expected_per_fn_flops': 1 - }, - { - 'testcase_name': 'SIGMOID', - 'activation': 'SIGMOID', - 'input_dims': [2048, 10], - 'expected_extra_flops': 0, - 'expected_per_fn_flops': 4 - }, - { - 'testcase_name': 'BatchNorm', - 'activation': 'RELU', - 'input_dims': [2048, 10], - 'output_dim': 8, - 'expected_extra_flops': 2048 * 8 * 10, # 10 flops per element. - 'expected_per_fn_flops': 1, - 'batch_norm': True - }, - { - 'testcase_name': 'WeightNorm', - 'activation': 'RELU', - 'input_dims': [2048, 10], - 'output_dim': 100, - 'expected_extra_flops': 2 * 10 + 2 * 10 * 100 + 2, - 'expected_per_fn_flops': 1, - 'weight_norm': True - }, - { - 'testcase_name': 'Bias', - 'activation': 'RELU', - 'input_dims': [2048, 10], - 'output_dim': 100, - 'expected_extra_flops': 100, - 'expected_per_fn_flops': 1, - 'has_bias': True - }) - # Extra flops are for bias, batch norm, weight norm, etc. - def testProjectionLayerMeta(self, - input_dims, - expected_per_fn_flops, - expected_extra_flops, - activation='RELU', - batch_norm=False, - weight_norm=False, - has_bias=False, - output_dim=2): - with self.session(use_gpu=True): - params = layers.ProjectionLayer.Params() - params.name = 'fc' - params.input_dim = input_dims[-1] - params.output_dim = output_dim - params.params_init = py_utils.WeightInit.Gaussian(0.1) - params.activation = activation - params.batch_norm = batch_norm - params.weight_norm = weight_norm - params.has_bias = has_bias - - meta = params.cls.FPropMeta(params, tshape.Shape(input_dims)) - self.assertEqual( - meta.flops, - expected_extra_flops + (np.prod(input_dims[:-1]) * - (2 * params.input_dim * params.output_dim + - params.output_dim * expected_per_fn_flops))) - self.assertEqual(meta.out_shapes[0].ToTensorShape().as_list(), - input_dims[:-1] + [params.output_dim]) - - -class StackingOverTimeLayerTest(test_utils.TestCase): - - def testStackingOverTimeFProp(self): - with self.session(use_gpu=True): - params = layers.StackingOverTime.Params() - params.name = 'stackingOverTime' - params.left_context = 2 - params.right_context = 0 - params.stride = 2 - - stacker = layers.StackingOverTime(params) - self.assertEqual(stacker.window_size, 3) - - inputs = tf.constant([[[1, 1], [2, 2], [3, 3], [4, 4], [5, 5], [6, 6]], - [[7, 7], [8, 8], [0, 0], [0, 0], [0, 0], [0, 0]]], - dtype=tf.float32) - paddings = tf.constant( - [[[0], [0], [0], [0], [0], [0]], [[0], [0], [1], [1], [1], [1]]], - dtype=tf.float32) - - outputs, output_paddings = stacker.FProp(inputs, paddings) - self.evaluate(tf.global_variables_initializer()) - print([np.array_repr(outputs.eval())]) - - expected_outputs = [ - [[0, 0, 0, 0, 1, 1], [1, 1, 2, 2, 3, 3], [3, 3, 4, 4, 5, 5]], - [[0, 0, 0, 0, 7, 7], [7, 7, 8, 8, 0, 0], [0, 0, 0, 0, 0, 0]], - ] - self.assertAllClose(expected_outputs, outputs.eval()) - - expected_output_paddings = [[[0], [0], [0]], [[0], [0], [1]]] - self.assertAllClose(expected_output_paddings, output_paddings.eval()) - - def testStackingOverTimeFProp2(self): - with self.session(use_gpu=True): - params = layers.StackingOverTime.Params() - params.name = 'stackingOverTime' - params.left_context = 0 - params.right_context = 1 - params.stride = 2 - - stacker = layers.StackingOverTime(params) - self.assertEqual(stacker.window_size, 2) - - inputs = tf.random.normal([2, 21, 16], seed=78123) - paddings = 1.0 - tf.sequence_mask([9, 14], 21, tf.float32) - paddings = tf.expand_dims(paddings, -1) - - outputs, output_paddings = stacker.FProp(inputs, paddings) - self.evaluate(tf.global_variables_initializer()) - - inputs_v, outputs_v, paddings_v = self.evaluate( - [inputs, outputs, output_paddings]) - - # length - self.assertAllEqual([5, 7], np.sum(1.0 - paddings_v, (1, 2))) - # input and output sums are equal - self.assertAllClose(np.sum(inputs_v, (1, 2)), np.sum(outputs_v, (1, 2))) - - def testStackingOverTimeIdentityFProp(self): - with self.session(use_gpu=True): - params = layers.StackingOverTime.Params() - params.name = 'stackingOverTime' - params.left_context = 0 - params.right_context = 0 - params.stride = 1 - - stacker = layers.StackingOverTime(params) - self.assertEqual(stacker.window_size, 1) - inputs = tf.constant([[[1], [2], [3], [4], [5]]], dtype=tf.float32) - paddings = tf.zeros([1, 5, 1], dtype=tf.float32) - - outputs, output_paddings = stacker.FProp(inputs, paddings) - self.evaluate(tf.global_variables_initializer()) - print([np.array_repr(outputs.eval())]) - expected_outputs = [[[1], [2], [3], [4], [5]]] - self.assertAllClose(expected_outputs, outputs.eval()) - expected_output_paddings = [[[0], [0], [0], [0], [0]]] - self.assertAllClose(expected_output_paddings, output_paddings.eval()) - - def _testUnstack(self, inputs, **kwargs): - params = layers.StackingOverTime.Params().Set( - name='stackingOverTime', **kwargs) - with self.session(use_gpu=True): - stacker = params.Instantiate() - stacked, _ = stacker.FProp(inputs) - unstacked = stacker.Unstack(stacked) - inputs, stacked, unstacked = self.evaluate([inputs, stacked, unstacked]) - - batch, input_length, depth = inputs.shape - stacked_length = stacked.shape[1] - stride = stacker.params.stride - right_context = stacker.params.right_context - - self.assertAllEqual( - unstacked.shape, - [batch, (stacked_length - 1) * stride + right_context + 1, depth]) - if right_context + 1 >= stride: - self.assertGreaterEqual(unstacked.shape[1], input_length) - self.assertAllClose(inputs, unstacked[:, :input_length]) - else: - self.assertLessEqual(unstacked.shape[1], input_length) - # The final up to stride - right_context - 1 values are missing. - self.assertLessEqual(input_length - unstacked.shape[1], - stride - right_context - 1) - self.assertAllClose(inputs[:, :unstacked.shape[1]], unstacked) - - def testStackingOverTimeUnstack(self): - batch_size = 2 - length = 7 - depth = 3 - inputs = tf.reshape( - tf.cast(tf.range(batch_size * length * depth), tf.float32), - [batch_size, length, depth]) - self._testUnstack(inputs, left_context=2, stride=1) - with self.assertRaises(ValueError): - self._testUnstack(inputs, stride=2) - self._testUnstack(inputs, stride=2, right_context=3) - self._testUnstack(inputs, left_context=2, stride=3) - self._testUnstack(inputs, stride=4, right_context=3) - self._testUnstack(inputs, stride=4, left_context=1, right_context=2) - - -class SingleShardEmbeddingLayerTest(test_utils.TestCase): - - def testSingleShardEmbeddingLayer(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - params = layers.SingleShardEmbeddingLayer.Params() - params.name = 'emb' - params.dtype = tf.float32 - params.vocab_size = 80000 - params.embedding_dim = 128 - params.params_init = py_utils.WeightInit.Gaussian(0.01) - params.vn.global_vn = False - params.vn.per_step_vn = False - emb_layer = params.Instantiate() - ids = tf.constant([[89], [100]]) - embs = emb_layer.EmbLookupDefaultTheta(ids) - embs_sum = tf.reduce_sum(embs) - self.evaluate(tf.global_variables_initializer()) - test_utils.CompareToGoldenSingleFloat(self, 0.126485, embs_sum.eval()) - - def testCheckedIds(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - params = layers.SingleShardEmbeddingLayer.Params() - params.name = 'emb' - params.dtype = tf.float32 - params.vocab_size = 16 - params.embedding_dim = 128 - params.params_init = py_utils.WeightInit.Gaussian(0.01) - params.vn.global_vn = False - params.vn.per_step_vn = False - emb_layer = params.Instantiate() - - neg_ids = tf.constant([[-1]]) - oov_ids = tf.constant([[params.vocab_size]]) - self.evaluate(tf.global_variables_initializer()) - - with self.assertRaises(tf.errors.InvalidArgumentError): - neg_embs = emb_layer.EmbLookupDefaultTheta(neg_ids) - neg_embs.eval() - with self.assertRaises(tf.errors.InvalidArgumentError): - oov_embs = emb_layer.EmbLookupDefaultTheta(oov_ids) - oov_embs.eval() - - def testEmbeddingLayerScaling(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - params = layers.SingleShardEmbeddingLayer.Params() - params.name = 'emb' - params.dtype = tf.float32 - params.vocab_size = 80000 - params.embedding_dim = 128 - params.params_init = py_utils.WeightInit.Gaussian(0.01) - params.vn.global_vn = False - params.vn.per_step_vn = False - params.scale_sqrt_depth = True - emb_layer = params.Instantiate() - ids = tf.constant([[89], [100]]) - embs = emb_layer.EmbLookupDefaultTheta(ids) - embs_sum = tf.reduce_sum(embs) - self.evaluate(tf.global_variables_initializer()) - self.assertAllClose(0.126485 * params.embedding_dim**0.5, - self.evaluate(embs_sum)) - - def testEmbeddingLayerWithVN(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - params = layers.SimpleEmbeddingLayer.Params() - params.name = 'emb' - params.dtype = tf.float32 - params.vocab_size = 80000 - params.embedding_dim = 128 - params.params_init = py_utils.WeightInit.Gaussian(0.01, seed=398847392) - params.vn.global_vn = True - params.vn.per_step_vn = False - params.vn.scale = 0.5 - params.vn.seed = 398847392 - params.random_seed = 12345 - emb_layer = params.Instantiate() - self.assertEqual(len(emb_layer.vars.Flatten()), 1) - ids = tf.constant([[89], [100]]) - embs = emb_layer.EmbLookupDefaultTheta(ids) - embs_sum = tf.reduce_sum(embs) - self.evaluate(tf.global_variables_initializer()) - test_utils.CompareToGoldenSingleFloat(self, 1.56561, embs_sum.eval()) - - -class EmbeddingLayerTest(test_utils.TestCase): - - def testEmbeddingLayer(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - params = layers.EmbeddingLayer.Params() - params.name = 'emb' - params.dtype = tf.float32 - params.vocab_size = 80000 - params.embedding_dim = 128 - params.max_num_shards = 4 - params.params_init = py_utils.WeightInit.Gaussian(0.01) - params.vn.global_vn = False - params.vn.per_step_vn = False - emb_layer = layers.EmbeddingLayer(params) - ids = tf.constant([[89], [100]]) - embs = emb_layer.EmbLookupDefaultTheta(ids) - embs_sum = tf.reduce_sum(embs) - self.evaluate(tf.global_variables_initializer()) - test_utils.CompareToGoldenSingleFloat(self, 0.234941, embs_sum.eval()) - - def testCheckedIds(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - params = layers.EmbeddingLayer.Params() - params.name = 'emb' - params.dtype = tf.float32 - params.vocab_size = 16 - params.embedding_dim = 128 - params.max_num_shards = 4 - params.params_init = py_utils.WeightInit.Gaussian(0.01) - params.vn.global_vn = False - params.vn.per_step_vn = False - emb_layer = layers.EmbeddingLayer(params) - - neg_ids = tf.constant([[-1]]) - oov_ids = tf.constant([[params.vocab_size]]) - self.evaluate(tf.global_variables_initializer()) - - with self.assertRaises(tf.errors.InvalidArgumentError): - neg_embs = emb_layer.EmbLookupDefaultTheta(neg_ids) - neg_embs.eval() - with self.assertRaises(tf.errors.InvalidArgumentError): - oov_embs = emb_layer.EmbLookupDefaultTheta(oov_ids) - oov_embs.eval() - - def testEmbeddingLayerScaling(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - params = layers.EmbeddingLayer.Params() - params.name = 'emb' - params.dtype = tf.float32 - params.vocab_size = 80000 - params.embedding_dim = 128 - params.max_num_shards = 4 - params.params_init = py_utils.WeightInit.Gaussian(0.01) - params.vn.global_vn = False - params.vn.per_step_vn = False - params.scale_sqrt_depth = True - emb_layer = layers.EmbeddingLayer(params) - ids = tf.constant([[89], [100]]) - embs = emb_layer.EmbLookupDefaultTheta(ids) - embs_sum = tf.reduce_sum(embs) - self.evaluate(tf.global_variables_initializer()) - self.assertAllClose(0.23494134843349457 * params.embedding_dim**0.5, - self.evaluate(embs_sum)) - - def testEmbeddingLayerWithVN(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - params = layers.EmbeddingLayer.Params() - params.name = 'emb' - params.dtype = tf.float32 - params.vocab_size = 80000 - params.embedding_dim = 128 - params.max_num_shards = 4 - params.params_init = py_utils.WeightInit.Gaussian(0.01, seed=398847392) - params.vn.global_vn = True - params.vn.per_step_vn = False - params.vn.scale = 0.5 - params.vn.seed = 398847392 - emb_layer = layers.EmbeddingLayer(params) - self.assertEqual(len(emb_layer.vars.Flatten()), 4) - ids = tf.constant([[89], [100]]) - embs = emb_layer.EmbLookupDefaultTheta(ids) - embs_sum = tf.reduce_sum(embs) - self.evaluate(tf.global_variables_initializer()) - test_utils.CompareToGoldenSingleFloat(self, -6.807296, embs_sum.eval()) - - def _testSimpleEmbeddingLayer(self, - use_matmul, - use_3d_weight_tensor, - fprop_mode, - scale_sqrt_depth=False): - g = tf.Graph() - with g.as_default(): - tf.random.set_seed(398847392) - params = layers.SimpleEmbeddingLayer.Params() - params.name = 'emb' - params.dtype = tf.float32 - params.vocab_size = 8000 - params.embedding_dim = 128 - params.use_matmul = use_matmul - params.fprop_mode = fprop_mode - params.use_3d_weight_tensor = use_3d_weight_tensor - params.scale_sqrt_depth = scale_sqrt_depth - params.params_init = py_utils.WeightInit.Gaussian(0.01) - params.vn.global_vn = False - params.vn.per_step_vn = False - - emb_layer = layers.SimpleEmbeddingLayer(params) - expected_fprop_mode = fprop_mode - if expected_fprop_mode is None: - expected_fprop_mode = 'matmul' if use_matmul else 'gather' - self.assertEqual(emb_layer._fprop_mode, expected_fprop_mode) - - emb_matrix = emb_layer.vars.wm - ids = tf.constant([[89], [100]]) - outputs = emb_layer.EmbLookupDefaultTheta(ids) - fast_outputs = emb_layer.EmbLookupDefaultThetaOnCpu(ids) - - with self.session(use_gpu=True, graph=g): - self.evaluate(tf.global_variables_initializer()) - emb_matrix_val, ids_val, outputs_val, fast_outputs_val = self.evaluate( - [emb_matrix, ids, outputs, fast_outputs]) - if scale_sqrt_depth: - emb_matrix_val *= params.embedding_dim**0.5 - - self.assertEqual(emb_matrix_val.shape, (8000, 128)) - self.assertEqual(ids_val.shape, (2, 1)) - - self.assertEqual(outputs_val.shape, (2, 1, 128)) - self.assertAllClose(emb_matrix_val[89, :], outputs_val[0, 0, :]) - self.assertAllClose(emb_matrix_val[100, :], outputs_val[1, 0, :]) - - self.assertEqual(fast_outputs_val.shape, (2, 1, 128)) - self.assertAllClose(emb_matrix_val[89, :], fast_outputs_val[0, 0, :]) - self.assertAllClose(emb_matrix_val[100, :], fast_outputs_val[1, 0, :]) - - def testSimpleEmbeddingLayerForLoop(self): - self._testSimpleEmbeddingLayer(False, True, None) - - def testSimpleEmbeddingLayerForLoop2D(self): - self._testSimpleEmbeddingLayer(False, False, None) - - def testSimpleEmbeddingLayerMatmul(self): - self._testSimpleEmbeddingLayer(True, False, None) - - def testSimpleEmbeddingLayerGather(self): - self._testSimpleEmbeddingLayer(False, False, 'gather') - - def testSimpleEmbeddingLayerScaling(self): - self._testSimpleEmbeddingLayer(True, False, None, True) - - def testSimpleEmbeddingLayerMasked(self): - g = tf.Graph() - with g.as_default(): - tf.random.set_seed(398847392) - params = layers.SimpleEmbeddingLayer.Params() - params.name = 'emd' - params.dtype = tf.float32 - params.vocab_size = 10 - params.embedding_dim = 5 - params.fprop_mode = 'gather' - params.use_3d_weight_tensor = False - params.params_init = py_utils.WeightInit.Gaussian(0.01) - params.vn.global_vn = False - params.vn.per_step_vn = False - params.apply_pruning = True - - emb_layer = layers.SimpleEmbeddingLayer(params) - emb_matrix = emb_layer.vars.wm - ids = tf.constant([[1], [2]]) - outputs = emb_layer.EmbLookupDefaultTheta(ids) - - self.assertIn('wm', emb_layer.vars.wm.name) - self.assertIn('mask', emb_layer.vars.mask.name) - self.assertIn('threshold', emb_layer.vars.threshold.name) - - self.assertEqual(emb_layer.theta.wm.get_shape(), tf.TensorShape([10, 5])) - self.assertEqual(emb_layer.theta.mask.get_shape(), tf.TensorShape([10, - 5])) - self.assertEqual(emb_layer.theta.threshold.get_shape(), - tf.TensorShape([])) - - embedding_var_count = 1 - wts = tf.get_collection('SimpleEmbeddingLayer_vars') - self.assertEqual(embedding_var_count, len(wts)) - - embedding_mask_count = 1 - masks = tf.get_collection('masks') - self.assertEqual(embedding_mask_count, len(masks)) - - emebdding_threshold_count = 1 - threshold = tf.get_collection('thresholds') - self.assertEqual(emebdding_threshold_count, len(threshold)) - - with self.session(use_gpu=False, graph=g): - self.evaluate(tf.global_variables_initializer()) - emb_matrix_val, _, outputs_val = self.evaluate([emb_matrix, ids, outputs]) - - self.assertAllClose(emb_matrix_val[1:3], outputs_val[:, 0, :]) - - def _testSimpleEmbeddingLayerGrad(self, - use_matmul, - use_3d_weight_tensor, - scale_sqrt_depth=False): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - params = layers.SimpleEmbeddingLayer.Params() - params.name = 'emb' - params.dtype = tf.float32 - params.vocab_size = 8000 - params.embedding_dim = 128 - params.use_matmul = use_matmul - params.use_3d_weight_tensor = use_3d_weight_tensor - params.scale_sqrt_depth = scale_sqrt_depth - params.params_init = py_utils.WeightInit.Gaussian(0.01) - params.vn.global_vn = False - params.vn.per_step_vn = False - emb_layer = layers.SimpleEmbeddingLayer(params) - ids = tf.constant([89, 100, 89, 89]) - embs = emb_layer.EmbLookupDefaultTheta(ids) * tf.constant([[0.1], [0.2], - [0.3], [0.4]]) - embs_sum = tf.reduce_sum(embs) - emb_weight = emb_layer.vars.wm - emb_grad, = tf.gradients(ys=[embs_sum], xs=[emb_weight]) - self.evaluate(tf.global_variables_initializer()) - emb_grad_val = self.evaluate(emb_grad) - - if not use_matmul: - # tf.embedding_lookup's gradient is a sparse representation. - # For testing, we convert it to a dense representation. - o_grad_matrix = np.zeros((8000, 128)) - for i in range(emb_grad_val.indices.shape[0]): - o_grad_matrix[emb_grad_val.indices[i], :] += emb_grad_val.values[i, :] - emb_grad_val = o_grad_matrix - - expected_emb_grad = np.zeros(shape=(8000, 128)) - expected_emb_grad[89, :] = 0.8 - expected_emb_grad[100, :] = 0.2 - if scale_sqrt_depth: - expected_emb_grad *= params.embedding_dim**0.5 - self.assertAllClose(expected_emb_grad, emb_grad_val) - - def testSimpleEmbeddingLayerGradForLoop(self): - self._testSimpleEmbeddingLayerGrad(False, True) - - def testSimpleEmbeddingLayerGradForLoop2D(self): - self._testSimpleEmbeddingLayerGrad(False, False) - - def testSimpleEmbeddingLayerGradMatmul(self): - self._testSimpleEmbeddingLayerGrad(True, False) - - def testSimpleEmbeddingLayerGradScaling(self): - self._testSimpleEmbeddingLayerGrad(True, False, True) - - def testCompareEmbeddingLayers(self): - classes = 8000 - dims = 128 - g = tf.Graph() - with g.as_default(): - ids = tf.placeholder(tf.int32) - - def CreateSimple(): - tf.random.set_seed(398847392) - p = layers.SimpleEmbeddingLayer.Params() - p.name = 'emb' - p.dtype = tf.float32 - p.vocab_size = classes - p.embedding_dim = dims - p.params_init = py_utils.WeightInit.Gaussian(0.01) - p.vn.global_vn = False - p.vn.per_step_vn = False - return layers.SimpleEmbeddingLayer(p) - - simple = CreateSimple() - simple_outs = simple.EmbLookupDefaultTheta(ids) - simple_grad = tf.gradients(simple_outs, simple.vars.wm)[0] - - def CreateOriginal(): - tf.random.set_seed(398847392) - p = layers.EmbeddingLayer.Params() - p.name = 'emb' - p.dtype = tf.float32 - p.vocab_size = classes - p.embedding_dim = dims - p.max_num_shards = 1 - p.params_init = py_utils.WeightInit.Gaussian(0.01) - p.vn.global_vn = False - p.vn.per_step_vn = False - return layers.EmbeddingLayer(p) - - original = CreateOriginal() - weight = tf.identity(simple.vars.wm) - theta = py_utils.NestedMap() - theta.wm = [weight] - original_outs = original.EmbLookup(theta, ids) - original_grad = tf.gradients(original_outs, weight)[0] - - ids_val = np.random.randint(0, high=classes, size=(4000,)) - with self.session(graph=g) as sess: - self.evaluate(tf.global_variables_initializer()) - s_outs, s_grad, o_outs, o_grad = sess.run( - [simple_outs, simple_grad, original_outs, original_grad], - feed_dict={ids: ids_val}) - self.assertAllClose(s_outs, o_outs) - self.assertAllClose(s_grad, o_grad) - - def testPositionalEmbeddingLayer(self): - with self.session(use_gpu=False): - p = layers.PositionalEmbeddingLayer.Params() - p.name = 'position_emb' - p.min_timescale = 1 - p.max_timescale = 7 - p.embedding_dim = 4 - seq_length = 11 - - pos_emb_layer = layers.PositionalEmbeddingLayer(p) - position_embs = pos_emb_layer.FPropDefaultTheta(seq_length) - actual_position_embs, = self.evaluate([position_embs]) - - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [ 0. , 0. , 1. , 1. ], - [ 0.84147096, 0.14237173, 0.54030228, 0.98981327], - [ 0.90929741, 0.28184283, -0.41614676, 0.95946062], - [ 0.14112 , 0.4155719 , -0.9899925 , 0.90956032], - [-0.7568025 , 0.54083425, -0.65364361, 0.84112918], - [-0.95892417, 0.65507787, 0.28366217, 0.75556135], - [-0.27941549, 0.75597537, 0.96017027, 0.65460002], - [ 0.65698659, 0.84147096, 0.7539022 , 0.54030228], - [ 0.98935831, 0.90982294, -0.14550003, 0.41499668], - [ 0.41211855, 0.9596386 , -0.91113025, 0.28123617], - [-0.54402113, 0.98990309, -0.83907151, 0.14174587]] - # pyformat: enable - # pylint: enable=bad-whitespace - print('expected_position_embs:', expected_output) - print('actual_position_embs:', actual_position_embs) - self.assertAllClose(actual_position_embs, expected_output) - - def testPositionalEmbeddingLayerWithPosition(self): - with self.session(use_gpu=False): - p = layers.PositionalEmbeddingLayer.Params() - p.name = 'position_emb' - p.min_timescale = 1 - p.max_timescale = 7 - p.embedding_dim = 4 - pos_tensor = tf.constant( - np.asarray([[0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3], - [0, 1, 2, 0, 1, 2, 3, 4, 0, 1, 0]]), - dtype=tf.int32) - - pos_emb_layer = layers.PositionalEmbeddingLayer(p) - position_embs = pos_emb_layer.FPropWithPosition(pos_emb_layer.theta, - pos_tensor) - actual_position_embs, = self.evaluate([position_embs]) - - # pylint: disable=bad-whitespace,bad-continuation - # pyformat: disable - expected_output = [ - [[ 0. , 0. , 1. , 1. ], - [ 0.84147096, 0.14237173, 0.54030228, 0.98981327], - [ 0.90929741, 0.28184283, -0.41614676, 0.95946062], - [ 0.14112 , 0.4155719 , -0.9899925 , 0.90956032], - [-0.7568025 , 0.54083425, -0.65364361, 0.84112918], - [-0.95892417, 0.65507787, 0.28366217, 0.75556135], - [-0.27941549, 0.75597537, 0.96017027, 0.65460002], - [ 0. , 0. , 1. , 1. ], - [ 0.84147096, 0.14237173, 0.54030228, 0.98981327], - [ 0.90929741, 0.28184283, -0.41614676, 0.95946062], - [ 0.14112 , 0.4155719 , -0.9899925 , 0.90956032]], - [[ 0. , 0. , 1. , 1. ], - [ 0.84147096, 0.14237173, 0.54030228, 0.98981327], - [ 0.90929741, 0.28184283, -0.41614676, 0.95946062], - [ 0. , 0. , 1. , 1. ], - [ 0.84147096, 0.14237173, 0.54030228, 0.98981327], - [ 0.90929741, 0.28184283, -0.41614676, 0.95946062], - [ 0.14112 , 0.4155719 , -0.9899925 , 0.90956032], - [-0.7568025 , 0.54083425, -0.65364361, 0.84112918], - [ 0. , 0. , 1. , 1. ], - [ 0.84147096, 0.14237173, 0.54030228, 0.98981327], - [ 0. , 0. , 1. , 1. ]] - ] - # pyformat: enable - # pylint: enable=bad-whitespace,bad-continuation - print('expected_position_embs:', expected_output) - print('actual_position_embs:', actual_position_embs) - self.assertAllClose(actual_position_embs, expected_output) - - def testPositionalEmbeddingLayerWithScaling(self): - with self.session(use_gpu=False): - p = layers.PositionalEmbeddingLayer.Params() - p.name = 'position_emb' - p.min_timescale = 1 - p.max_timescale = 7 - p.embedding_dim = 4 - p.trainable_scaling = True - p.trainable_scaling_init = 1.0 / np.sqrt(p.embedding_dim) - seq_length = 11 - - pos_emb_layer = layers.PositionalEmbeddingLayer(p) - position_embs = pos_emb_layer.FPropDefaultTheta(seq_length) - self.evaluate(tf.global_variables_initializer()) - actual_position_embs, = self.evaluate([position_embs]) - - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [ 0. , 0. , 1. , 1. ], - [ 0.84147096, 0.14237173, 0.54030228, 0.98981327], - [ 0.90929741, 0.28184283, -0.41614676, 0.95946062], - [ 0.14112 , 0.4155719 , -0.9899925 , 0.90956032], - [-0.7568025 , 0.54083425, -0.65364361, 0.84112918], - [-0.95892417, 0.65507787, 0.28366217, 0.75556135], - [-0.27941549, 0.75597537, 0.96017027, 0.65460002], - [ 0.65698659, 0.84147096, 0.7539022 , 0.54030228], - [ 0.98935831, 0.90982294, -0.14550003, 0.41499668], - [ 0.41211855, 0.9596386 , -0.91113025, 0.28123617], - [-0.54402113, 0.98990309, -0.83907151, 0.14174587]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_output / np.sqrt(p.embedding_dim), - actual_position_embs) - - def testRelativePositionalEmbeddingLayer(self): - with self.session(use_gpu=False): - radius = 3 - p = layers.RelativePositionalEmbeddingLayer.Params().Set( - name='rel_position_emb', radius=radius, dim=4) - layer = p.Instantiate() - indices = np.array([-5, -2, 0, 1, 4], dtype=np.int32) - pos_emb = layer.FPropDefaultTheta(tf.convert_to_tensor(indices)) - - self.evaluate(tf.global_variables_initializer()) - actual_pos_emb, full_emb = self.evaluate([pos_emb, layer.vars.w]) - - clipped_indices = np.vectorize(lambda x: max(-radius, min(radius, x)))( - indices) + radius - expected_output = np.take_along_axis(full_emb, - np.expand_dims(clipped_indices, -1), - 0) - print('expected_position_embs:', expected_output) - print('actual_position_embs:', actual_pos_emb) - self.assertAllClose(actual_pos_emb, expected_output) - - def testSinusoidalPositionalEmbeddingLayer(self): - with self.session(use_gpu=False): - p = layers.SinusoidalPositionalEmbeddingLayer.Params() - p.name = 'position_emb' - p.embedding_dim = 2 - seq_length = 4 - - pos_emb_layer = layers.SinusoidalPositionalEmbeddingLayer(p) - position_embs = pos_emb_layer.FPropDefaultTheta(seq_length) - actual_position_embs, = self.evaluate([position_embs]) - expected_output = [[math.sin(p / 2 * math.pi), - math.cos(p / 2 * math.pi)] for p in range(4)] - self.assertAllClose(actual_position_embs, expected_output) - - def testOneHotEmbeddingLayer(self): - with self.session(use_gpu=True): - params = layers.OneHotEmbeddingLayer.Params() - params.name = 'emb' - params.dtype = tf.float32 - params.vocab_size = 4 - params.embedding_dim = 4 - emb_layer = layers.OneHotEmbeddingLayer(params) - ids = tf.constant([[0], [2]]) - embs = emb_layer.EmbLookupDefaultTheta(ids) - self.evaluate(tf.global_variables_initializer()) - expected_output = [[[1., 0., 0., 0.]], [[0., 0., 1., 0.]]] - self.assertAllClose(expected_output, embs.eval()) - - def testOneHotEmbeddingLayerWithUncertainty(self): - with self.session(use_gpu=True): - params = layers.OneHotEmbeddingLayer.Params() - params.name = 'emb' - params.dtype = tf.float32 - params.vocab_size = 4 - params.embedding_dim = 4 - params.uncertainty = 0.3 - emb_layer = layers.OneHotEmbeddingLayer(params) - ids = tf.constant([[0], [2]]) - embs = emb_layer.EmbLookupDefaultTheta(ids) - self.evaluate(tf.global_variables_initializer()) - expected_output = [[[0.7, 0.1, 0.1, 0.1]], [[0.1, 0.1, 0.7, 0.1]]] - self.assertAllClose(expected_output, embs.eval()) - - -class SoftmaxLayerTest(test_utils.TestCase): - - def _RunSimpleFullSoftmax(self, - num_shards=1, - chunk_size=0, - inputs=None, - class_ids=None, - class_weights=None, - class_probabilities=None, - num_samples=0, - default_qdomain=None, - training_step=-1, - seed=None, - dtype=tf.float32, - fprop_dtype=None, - apply_pruning=False, - use_bias=True): - if fprop_dtype is None: - fprop_dtype = dtype - with self.session(use_gpu=True, graph=tf.Graph()): - if seed is not None: - tf.random.set_seed(seed) - if class_ids is None: - class_ids = tf.constant([[1], [5], [10]], dtype=tf.int32) - else: - class_ids = tf.constant(class_ids) - if class_weights is None: - class_weights = tf.constant([1.0, 0.4, 0.8], dtype=fprop_dtype) - else: - class_weights = tf.constant(class_weights) - np.random.seed(12345) - if inputs is None: - inputs = [tf.constant(np.random.rand(3, 10), dtype=fprop_dtype)] - else: - inputs = [tf.constant(inputs, dtype=fprop_dtype)] - - params = layers.SimpleFullSoftmax.Params() - params.dtype = dtype - params.fprop_dtype = fprop_dtype - params.name = 'softmax' - params.input_dim = 10 - params.num_classes = 32 - params.num_shards = num_shards - params.chunk_size = chunk_size - params.apply_pruning = apply_pruning - params.params_init = py_utils.WeightInit.Gaussian(0.5, 123456) - params.random_seed = 12345678 - params.use_bias = use_bias - - if default_qdomain is not None: - params.qdomain.default = default_qdomain - - if num_samples > 0: - # Turn on sampled soft-max; the asserts need to hold for it to be used. - params.num_sampled = num_samples - assert class_probabilities is None - assert chunk_size == 0 - - params.vn.global_vn = False - softmax = layers.SimpleFullSoftmax(params) - xent_loss = softmax.FProp( - softmax.theta, - inputs, - class_weights=class_weights, - class_ids=class_ids, - class_probabilities=class_probabilities) - - all_vars = tf.get_collection('SimpleFullSoftmax_vars') - expected_var_names = [] - for i in range(num_shards): - expected_var_names.append(u'softmax/weight_%d/var:0' % i) - if use_bias: - expected_var_names.append(u'softmax/bias_%d/var:0' % i) - - all_var_names = [v.name for v in all_vars] - self.assertCountEqual(expected_var_names, all_var_names) - - self.evaluate(tf.global_variables_initializer()) - if training_step >= 0: - self.evaluate( - tf.assign(py_utils.GetOrCreateGlobalStepVar(), training_step)) - return self.evaluate(xent_loss) - - def testSimpleFullSoftmaxMasked(self): - num_shards = 2 - apply_pruning = True - params = layers.SimpleFullSoftmax.Params() - params.name = 'softmax' - params.dtype = tf.float32 - params.input_dim = 10 - params.num_classes = 32 - params.fprop_dtype = tf.float32 - params.num_shards = num_shards - params.apply_pruning = apply_pruning - params.random_seed = 12345678 - softmax_layer = layers.SimpleFullSoftmax(params) - - self.assertIn('weight_0', softmax_layer.vars.weight_0.name) - self.assertIn('weight_1', softmax_layer.vars.weight_1.name) - self.assertIn('mask_0', softmax_layer.vars.mask_0.name) - self.assertIn('mask_1', softmax_layer.vars.mask_1.name) - self.assertIn('threshold_0', softmax_layer.vars.threshold_0.name) - self.assertIn('threshold_1', softmax_layer.vars.threshold_1.name) - - self.assertEqual(softmax_layer.theta.weight_0.get_shape(), - tf.TensorShape([10, 16])) - self.assertEqual(softmax_layer.theta.weight_1.get_shape(), - tf.TensorShape([10, 16])) - self.assertEqual(softmax_layer.theta.mask_0.get_shape(), - tf.TensorShape([10, 16])) - self.assertEqual(softmax_layer.theta.mask_1.get_shape(), - tf.TensorShape([10, 16])) - self.assertEqual(softmax_layer.theta.threshold_0.get_shape(), - tf.TensorShape([])) - self.assertEqual(softmax_layer.theta.threshold_0.get_shape(), - tf.TensorShape([])) - - softmax_var_count = 4 # 2 each for weights and biases (we have 2 shards) - wts = tf.get_collection('SimpleFullSoftmax_vars') - self.assertEqual(softmax_var_count, len(wts)) - - softmax_mask_count = 2 - masks = tf.get_collection('masks') - self.assertEqual(softmax_mask_count, len(masks)) - - softmax_threshold_count = 2 - threshold = tf.get_collection('thresholds') - self.assertEqual(softmax_threshold_count, len(threshold)) - - # Sampled and Masked - xent_loss = self._RunSimpleFullSoftmax( - num_samples=32, seed=12345, apply_pruning=True) - loss = xent_loss.total_xent - log_perplexity = xent_loss.avg_xent - self.assertNear(loss, 8.681571, 1e-5) - self.assertNear(log_perplexity, 3.946169, 1e-5) - - # Sharded and Masked - xent_loss = self._RunSimpleFullSoftmax(num_shards=2, apply_pruning=True) - loss = xent_loss.total_xent - log_perplexity = xent_loss.avg_xent - self.assertNear(loss, 6.14888, 1e-5) - self.assertNear(log_perplexity, 2.79495, 1e-5) - - # Non_2D and Masked - xent_loss = self._RunSimpleFullSoftmax( - inputs=np.random.rand(4, 3, 10), - class_weights=np.ones((4, 3)), - class_ids=np.random.randint(32, size=(4, 3)), - apply_pruning=True) - self.assertEqual(xent_loss.logits.shape, (4, 3, 32)) - self.assertEqual(xent_loss.per_example_xent.shape, (4, 3)) - self.assertEqual(xent_loss.per_example_weight.shape, (4, 3)) - - xent_loss = self._RunSimpleFullSoftmax( - inputs=np.random.rand(4, 3, 10), - class_weights=np.ones((4, 3)), - class_probabilities=np.random.uniform(size=(4, 3, 32)), - apply_pruning=True) - self.assertEqual(xent_loss.logits.shape, (4, 3, 32)) - self.assertEqual(xent_loss.per_example_xent.shape, (4, 3)) - self.assertEqual(xent_loss.per_example_weight.shape, (4, 3)) - - # Chunked and Masked - for chunk_size in (0, 1, 2, 3, 4, 5): - print('chunk_size = ', chunk_size) - xent_output = self._RunSimpleFullSoftmax( - chunk_size=chunk_size, apply_pruning=True) - loss = xent_output.total_xent - log_perplexity = xent_output.avg_xent - print('xent_output ', xent_output) - print('xent_output.per_example_argmax.dtype ', - xent_output.per_example_argmax.dtype) - self.assertAllClose(loss, 6.22425) - self.assertAllClose(log_perplexity, 2.82920) - self.assertAllEqual(xent_output.per_example_argmax, - np.argmax(xent_output.logits, axis=1)) - - def testSimpleFullSoftmax_Sampled(self): - xent_loss = self._RunSimpleFullSoftmax(num_samples=32, seed=12345) - loss = xent_loss.total_xent - log_perplexity = xent_loss.avg_xent - self.assertNear(loss, 8.681571, 1e-5) - self.assertNear(log_perplexity, 3.946169, 1e-5) - - def testSimpleFullSoftmax_NoBias(self): - xent_loss = self._RunSimpleFullSoftmax(seed=12345, use_bias=False) - loss = xent_loss.total_xent - log_perplexity = xent_loss.avg_xent - err = 1e-5 - self.assertNear(loss, 12.476410, err=err) - self.assertNear(log_perplexity, 5.671095, err=err) - - def testSimpleFullSoftmax_SampledAndSharded(self): - xent_loss = self._RunSimpleFullSoftmax( - num_shards=4, num_samples=32, seed=12345) - loss = xent_loss.total_xent - log_perplexity = xent_loss.avg_xent - self.assertNear(loss, 8.510439, 1e-5) - self.assertNear(log_perplexity, 3.868381, 1e-5) - - def testSimpleFullSoftmax_Non2D(self): - xent_loss = self._RunSimpleFullSoftmax( - inputs=np.random.rand(4, 3, 10), - class_weights=np.ones((4, 3)), - class_ids=np.random.randint(32, size=(4, 3))) - self.assertEqual(xent_loss.logits.shape, (4, 3, 32)) - self.assertEqual(xent_loss.per_example_xent.shape, (4, 3)) - self.assertEqual(xent_loss.per_example_weight.shape, (4, 3)) - - xent_loss = self._RunSimpleFullSoftmax( - inputs=np.random.rand(4, 3, 10), - class_weights=np.ones((4, 3)), - class_probabilities=np.random.uniform(size=(4, 3, 32))) - self.assertEqual(xent_loss.logits.shape, (4, 3, 32)) - self.assertEqual(xent_loss.per_example_xent.shape, (4, 3)) - self.assertEqual(xent_loss.per_example_weight.shape, (4, 3)) - - def _testSimpleFullSoftmax_Basic_Helper(self, dtype, fprop_dtype): - xent_loss = self._RunSimpleFullSoftmax(dtype=dtype, fprop_dtype=fprop_dtype) - loss = xent_loss.total_xent - log_perplexity = xent_loss.avg_xent - print(['loss', loss]) - print(['log_perplexity', log_perplexity]) - err = 1e-5 - if fprop_dtype == tf.float16 or fprop_dtype == tf.bfloat16: - err = 1e-2 - self.assertNear(loss, 6.22425, err=err) - self.assertNear(log_perplexity, 2.8292, err=err) - self.assertAllEqual(xent_loss.per_example_argmax, - np.argmax(xent_loss.logits, axis=1)) - - def testSimpleFullSoftmax_Basic_Float32(self): - self._testSimpleFullSoftmax_Basic_Helper( - dtype=tf.float32, fprop_dtype=tf.float32) - - def testSimpleFullSoftmax_Basic_Float32Float16(self): - self._testSimpleFullSoftmax_Basic_Helper( - dtype=tf.float32, fprop_dtype=tf.float16) - - def testSimpleFullSoftmax_Sharded(self): - xent_loss = self._RunSimpleFullSoftmax(2) - loss = xent_loss.total_xent - log_perplexity = xent_loss.avg_xent - print(['loss', loss]) - print(['log_perplexity', log_perplexity]) - self.assertNear(loss, 6.14888, 1e-5) - self.assertNear(log_perplexity, 2.79495, 1e-5) - - def testSimpleFullSoftmax_Chunked(self): - for chunk_size in (0, 1, 2, 3, 4, 5): - print('chunk_size = ', chunk_size) - xent_output = self._RunSimpleFullSoftmax(chunk_size=chunk_size) - loss = xent_output.total_xent - log_perplexity = xent_output.avg_xent - print('xent_output ', xent_output) - print('xent_output.per_example_argmax.dtype ', - xent_output.per_example_argmax.dtype) - self.assertAllClose(loss, 6.22425) - self.assertAllClose(log_perplexity, 2.82920) - self.assertAllEqual(xent_output.per_example_argmax, - np.argmax(xent_output.logits, axis=1)) - - def testSimpleFullSoftmax_Basic_Distributions(self): - with self.session(use_gpu=False): - class_ids = tf.constant([1, 5, 10], dtype=tf.int32) - class_weights = tf.constant([1.0, 0.4, 0.8], dtype=tf.float32) - np.random.seed(12345) - inputs = [tf.constant(np.random.rand(3, 10), dtype=tf.float32)] - - params = layers.SimpleFullSoftmax.Params() - params.name = 'softmax' - params.input_dim = 10 - params.num_classes = 32 - params.params_init = py_utils.WeightInit.Gaussian(0.5, 123456) - params.vn.global_vn = False - softmax = layers.SimpleFullSoftmax(params) - xent_loss = softmax.XentLoss( - inputs, - class_weights=class_weights, - class_probabilities=tf.one_hot(class_ids, params.num_classes)) - self.evaluate(tf.global_variables_initializer()) - loss = self.evaluate(xent_loss.total_xent) - log_perplexity = self.evaluate(xent_loss.avg_xent) - print(['loss', loss]) - print(['log_perplexity', log_perplexity]) - self.assertNear(loss, 6.22425, 1e-5) - self.assertNear(log_perplexity, 2.8292, 1e-5) - - def testSimpleFullSoftmax_GlobalVN(self): - with self.session(use_gpu=False): - class_ids = tf.constant([1, 5, 10], dtype=tf.int32) - class_weights = tf.constant([1.0, 0.4, 0.8], dtype=tf.float32) - np.random.seed(12345) - inputs = [tf.constant(np.random.rand(3, 10), dtype=tf.float32)] - - params = layers.SimpleFullSoftmax.Params() - params.name = 'softmax' - params.input_dim = 10 - params.num_classes = 32 - params.params_init = py_utils.WeightInit.Gaussian(0.5, 123456) - params.vn.global_vn = True - params.vn.seed = 23456 - params.vn.scale = 1.0 - softmax = layers.SimpleFullSoftmax(params) - xent_loss = softmax.XentLoss( - inputs, class_weights=class_weights, class_ids=class_ids) - self.evaluate(tf.global_variables_initializer()) - loss = self.evaluate(xent_loss.total_xent) - log_perplexity = self.evaluate(xent_loss.avg_xent) - print(['testSimpleFullSoftmax_GlobalVN loss', loss]) - print(['testSimpleFullSoftmax_GlobalVN log_perplexity', log_perplexity]) - self.assertNear(loss, 19.9612, 1e-4) - self.assertNear(log_perplexity, 3.46426, 1e-4) - - def testSimpleFullSoftmax_PerStepVN(self): - with self.session(use_gpu=False): - class_ids = tf.constant([1, 5, 10], dtype=tf.int32) - class_weights = tf.constant([1.0, 0.4, 0.8], dtype=tf.float32) - np.random.seed(12345) - inputs = [tf.constant(np.random.rand(3, 10), dtype=tf.float32)] - - params = layers.SimpleFullSoftmax.Params() - params.name = 'softmax' - params.input_dim = 10 - params.num_classes = 32 - params.params_init = py_utils.WeightInit.Gaussian(0.5, 123456) - params.vn.global_vn = False - params.vn.per_step_vn = True - params.vn.seed = 23456 - params.vn.scale = 1.0 - softmax = layers.SimpleFullSoftmax(params) - xent_loss = softmax.XentLoss( - inputs, class_weights=class_weights, class_ids=class_ids) - self.evaluate(tf.global_variables_initializer()) - loss = self.evaluate(xent_loss.total_xent) - log_perplexity = self.evaluate(xent_loss.avg_xent) - print(['testShardedFullSoftmax_PerStepVN loss', loss]) - print(['testShardedFullSoftmax_PerStepVN log_perplexity', log_perplexity]) - self.assertNear(loss, 19.9612, 1e-4) - self.assertNear(log_perplexity, 3.46426, 1e-4) - - def testSimpleFullSoftmax_FakeQuantized(self): - default_qdomain = quant_utils.SymmetricScheduledClipQDomain.Params() - default_qdomain.cc_schedule = quant_utils.FakeQuantizationSchedule.Params( - ).Set( - clip_start_step=0, clip_end_step=2, quant_start_step=2) - xent_loss = self._RunSimpleFullSoftmax( - default_qdomain=default_qdomain, training_step=5) - loss = xent_loss.total_xent - log_perplexity = xent_loss.avg_xent - print(['loss', loss]) - print(['log_perplexity', log_perplexity]) - self.assertNear(loss, 6.285590, 1e-5) - self.assertNear(log_perplexity, 2.857086, 1e-5) - - def _RunSimpleFullSoftmaxGradientChecker(self, batch_size, num_classes, - chunk_size, num_shards): - for (dtype, use_gpu, tolerance) in [(tf.float32, True, 1e-2), - (tf.float64, False, 1e-6)]: - tf.logging.info('dtype %s tolerance %g', dtype, tolerance) - with self.session(use_gpu=use_gpu, graph=tf.Graph()) as sess: - input_dim = 10 - np.random.seed(12345) - class_ids = tf.constant( - np.random.randint(num_classes, size=(batch_size, 1)), - dtype=tf.int32) - class_weights = tf.constant(np.random.rand(batch_size), dtype=dtype) - inputs = [ - tf.constant(np.random.rand(batch_size, input_dim), dtype=dtype) - ] - - params = layers.SimpleFullSoftmax.Params() - params.name = 'softmax' - params.dtype = dtype - params.input_dim = input_dim - params.num_classes = num_classes - params.num_shards = num_shards - params.chunk_size = chunk_size - params.params_init = py_utils.WeightInit.Gaussian(0.5, 123456) - params.vn.global_vn = False - softmax = layers.SimpleFullSoftmax(params) - xent_loss = softmax.XentLoss( - inputs, class_weights=class_weights, class_ids=class_ids) - softmax_vars = softmax.vars.Flatten() - # Now add the backward graph. - grads = tf.gradients(xent_loss.total_xent, softmax_vars) - - self.evaluate(tf.global_variables_initializer()) - assert len(softmax_vars) == len(grads) - for x, grad_x in zip(softmax_vars, grads): - grad_symbolic = self.evaluate(grad_x) - grad_numeric = test_utils.ComputeNumericGradient( - sess, xent_loss.total_xent, x) - self.assertAllClose( - grad_symbolic, grad_numeric, atol=tolerance, rtol=tolerance) - - def testSimpleFullSoftmaxGradientChecker(self): - self._RunSimpleFullSoftmaxGradientChecker(3, 4, 0, 1) - self._RunSimpleFullSoftmaxGradientChecker(3, 4, 0, 2) - self._RunSimpleFullSoftmaxGradientChecker(3, 4, 2, 2) - self._RunSimpleFullSoftmaxGradientChecker(3, 4, 5, 2) - - def testSimpleFullSoftmax_SymbolicShape(self): - with self.session(use_gpu=False): - class_ids = tf.constant([1, 5, 10], dtype=tf.int32) - class_weights = tf.constant([1.0, 0.4, 0.8], dtype=tf.float32) - np.random.seed(12345) - inputs = [tf.constant(np.random.rand(3, 10), dtype=tf.float32)] - - # Use a symbol to represent the input dim. - input_dim = symbolic.Symbol('input_dim') - params = layers.SimpleFullSoftmax.Params() - params.name = 'softmax' - params.input_dim = input_dim - params.num_classes = 32 - with symbolic.SymbolToValueMap(symbolic.STATIC_VALUES, {input_dim: 10}): - softmax = layers.SimpleFullSoftmax(params) - xent_loss = softmax.XentLoss( - inputs, class_weights=class_weights, class_ids=class_ids) - self.evaluate(tf.global_variables_initializer()) - self.evaluate(xent_loss.total_xent) - - -class SingleShardSoftmaxLayerTest(test_utils.TestCase): - - def _RunSimpleFullSoftmax(self, - inputs=None, - class_ids=None, - class_weights=None, - class_probabilities=None, - chunk_size=0, - dtype=tf.float32, - fprop_dtype=None): - if fprop_dtype is None: - fprop_dtype = dtype - with self.session(use_gpu=True, graph=tf.Graph()): - inputs = tf.constant(inputs, dtype=fprop_dtype) - if class_ids is not None: - class_ids = tf.constant(class_ids, dtype=tf.int32) - if class_weights is not None: - class_weights = tf.constant(class_weights, dtype=dtype) - if class_probabilities is not None: - class_probabilities = tf.constant(class_probabilities, dtype=dtype) - - params = layers.SingleShardFullSoftmax.Params() - params.dtype = dtype - params.fprop_dtype = fprop_dtype - params.name = 'softmax' - params.input_dim = 10 - params.num_classes = 32 - params.chunk_size = chunk_size - params.params_init = py_utils.WeightInit.Gaussian(0.5, 123456) - params.random_seed = 12345678 - - params.vn.global_vn = False - softmax = params.Instantiate() - xent_loss = softmax.FProp( - softmax.theta, - inputs, - class_weights=class_weights, - class_ids=class_ids, - class_probabilities=class_probabilities) - - self.evaluate(tf.global_variables_initializer()) - return self.evaluate(xent_loss) - - def testSimpleFullSoftmax_Non2D_ClassId(self): - np.random.seed(1234578) - xent_loss = self._RunSimpleFullSoftmax( - inputs=np.random.rand(4, 3, 10), - class_weights=np.ones((4, 3, 1)), - class_ids=np.random.randint(32, size=(4, 3, 1)), - chunk_size=2) - self.assertEqual(xent_loss.per_example_xent.shape, (4, 3)) - self.assertEqual(xent_loss.per_example_weight.shape, (4, 3)) - - def testSimpleFullSoftmax_Non2D_ClassProb(self): - np.random.seed(12345) - xent_loss = self._RunSimpleFullSoftmax( - inputs=np.random.rand(4, 3, 10), - class_weights=np.ones((4, 3, 1)), - class_probabilities=np.random.randint(32, size=(4, 3, 32)), - chunk_size=1) - self.assertEqual(xent_loss.per_example_xent.shape, (4, 3)) - self.assertEqual(xent_loss.per_example_weight.shape, (4, 3)) - - def _testSimpleFullSoftmax_Basic_Helper(self, dtype, fprop_dtype): - np.random.seed(12345) - class_ids = [[1], [5], [10]] - class_weights = [[1.0], [0.4], [0.8]] - inputs = np.random.rand(3, 10) - xent_loss = self._RunSimpleFullSoftmax( - inputs=inputs, - class_weights=class_weights, - class_ids=class_ids, - dtype=dtype, - fprop_dtype=fprop_dtype) - loss = xent_loss.total_xent - log_perplexity = xent_loss.avg_xent - print(['loss', loss]) - print(['log_perplexity', log_perplexity]) - err = 1e-5 - if fprop_dtype == tf.float16 or fprop_dtype == tf.bfloat16: - err = 1e-2 - self.assertNear(loss, 6.22425, err=err) - self.assertNear(log_perplexity, 2.8292, err=err) - self.assertAllEqual(xent_loss.per_example_argmax, - np.argmax(xent_loss.logits, axis=1)) - - def testSimpleFullSoftmax_Basic_Float32(self): - self._testSimpleFullSoftmax_Basic_Helper( - dtype=tf.float32, fprop_dtype=tf.float32) - - def testSimpleFullSoftmax_Basic_Float32Float16(self): - self._testSimpleFullSoftmax_Basic_Helper( - dtype=tf.float32, fprop_dtype=tf.float16) - - def testSimpleFullSoftmax_Chunked(self): - np.random.seed(12345) - class_ids = [[1], [5], [10]] - class_weights = [[1.0], [0.4], [0.8]] - inputs = np.random.rand(3, 10) - per_example_xent = None - per_example_argmax = None - for chunk_size in (0, 1, 3): - xent_output = self._RunSimpleFullSoftmax( - inputs=inputs, - class_weights=class_weights, - class_ids=class_ids, - chunk_size=chunk_size) - loss = xent_output.total_xent - log_perplexity = xent_output.avg_xent - print('xent_output ', xent_output) - print('xent_output.per_example_argmax.dtype ', - xent_output.per_example_argmax.dtype) - self.assertAllClose(loss, 6.22425) - self.assertAllClose(log_perplexity, 2.82920) - if per_example_xent is None: - per_example_xent = xent_output.per_example_xent - per_example_argmax = xent_output.per_example_argmax - else: - self.assertAllClose(per_example_xent, xent_output.per_example_xent) - self.assertAllClose(per_example_argmax, xent_output.per_example_argmax) - - def _RunSimpleFullSoftmaxGradientChecker(self, batch_size, num_classes, - chunk_size): - for (dtype, use_gpu, tolerance) in [(tf.float32, True, 1e-2), - (tf.float64, False, 1e-6)]: - tf.logging.info('dtype %s tolerance %g', dtype, tolerance) - with self.session(use_gpu=use_gpu, graph=tf.Graph()) as sess: - input_dim = 10 - np.random.seed(12345) - class_ids = tf.constant( - np.random.randint(num_classes, size=(batch_size, 1)), - dtype=tf.int32) - class_weights = tf.constant(np.random.rand(batch_size, 1), dtype=dtype) - inputs = tf.constant(np.random.rand(batch_size, input_dim), dtype=dtype) - - params = layers.SingleShardFullSoftmax.Params() - params.name = 'softmax' - params.dtype = dtype - params.input_dim = input_dim - params.num_classes = num_classes - params.chunk_size = chunk_size - params.params_init = py_utils.WeightInit.Gaussian(0.5, 123456) - params.vn.global_vn = False - softmax = params.Instantiate() - xent_loss = softmax.FProp( - softmax.theta, - inputs, - class_weights=class_weights, - class_ids=class_ids) - softmax_vars = softmax.vars.Flatten() - # Now add the backward graph. - grads = tf.gradients(xent_loss.total_xent, softmax_vars) - - self.evaluate(tf.global_variables_initializer()) - assert len(softmax_vars) == len(grads) - for x, grad_x in zip(softmax_vars, grads): - grad_symbolic = self.evaluate(grad_x) - grad_numeric = test_utils.ComputeNumericGradient( - sess, xent_loss.total_xent, x) - self.assertAllClose( - grad_symbolic, grad_numeric, atol=tolerance, rtol=tolerance) - - def testSimpleFullSoftmaxGradientChecker(self): - self._RunSimpleFullSoftmaxGradientChecker(3, 4, 0) - self._RunSimpleFullSoftmaxGradientChecker(3, 4, 1) - self._RunSimpleFullSoftmaxGradientChecker(3, 4, 3) - - -class SoftmaxLayerLogitsTest(test_utils.TestCase): - """Testing SoftmaxLayer.Logits().""" - - def _Logits(self, params, batch_size=2, seq_length=None): - with self.session(use_gpu=True, graph=tf.Graph()): - np.random.seed(12345) - tf.random.set_seed(1234) - - params.name = 'softmax' - if not params.input_dim: - params.input_dim = 3 - if not params.num_classes: - params.num_classes = 4 - params.params_init = py_utils.WeightInit.Gaussian(0.5, 123456) - softmax = params.Instantiate() - - input_dim = params.input_dim - if seq_length: - inputs = np.random.rand(batch_size, seq_length, input_dim) - else: - inputs = np.random.rand(batch_size, input_dim) - inputs = tf.constant(inputs, dtype=py_utils.FPropDtype(params)) - logits = softmax.Logits(softmax.theta, inputs) - - if seq_length: - logits = py_utils.HasShape(logits, - [batch_size, seq_length, params.num_classes]) - else: - logits = py_utils.HasShape(logits, [batch_size, params.num_classes]) - self.evaluate(tf.global_variables_initializer()) - return logits.eval() - - def testConvSoftmaxLogits(self): - params = layers.ConvSoftmax.Params() - self.assertAllClose([[0.52536774, -0.17598523, 0.38314393, -0.36068222], - [0.75792629, -0.18001975, 0.42298675, -0.35423514]], - self._Logits(params)) - - def testSimpleFullSoftmax(self): - params = layers.SimpleFullSoftmax.Params() - self.assertAllClose([[0.52536774, -0.17598523, 0.38314393, -0.36068222], - [0.75792629, -0.18001975, 0.42298675, -0.35423514]], - self._Logits(params)) - - def testConvSoftmaxLogitsWith3DInputs(self): - params = layers.ConvSoftmax.Params() - logits = self._Logits(params, seq_length=5) - self.assertAllClose(6.9934864, np.sum(logits)) - - -class SharedSoftmaxLayerTest(SoftmaxLayerTest): - - def _testSharedSoftmaxLayerEmbLookup(self, scale_sqrt_depth=False): - g = tf.Graph() - with g.as_default(): - tf.random.set_seed(398847392) - params = layers.SharedSoftmaxLayer.Params() - params.dtype = tf.float32 - params.fprop_dtype = None - params.name = 'shared_layer' - params.input_dim = 128 - params.num_classes = 8000 - params.num_shards = 1 - params.chunk_size = 0 - params.apply_pruning = False - params.params_init = py_utils.WeightInit.Gaussian(0.5, 123456) - params.scale_sqrt_depth = scale_sqrt_depth - params.random_seed = 12345678 - - emb_layer = layers.SharedSoftmaxLayer(params) - - emb_matrix = tf.einsum('ji', emb_layer._ConcatWeights(emb_layer.theta).wm) - ids = tf.constant([[89], [100]]) - outputs = emb_layer.EmbLookup(emb_layer.theta, ids) - - with self.session(use_gpu=True, graph=g): - self.evaluate(tf.global_variables_initializer()) - emb_matrix_val, ids_val, outputs_val = self.evaluate( - [emb_matrix, ids, outputs]) - self.assertEqual(emb_matrix_val.shape, (8000, 128)) - self.assertEqual(ids_val.shape, (2, 1)) - self.assertEqual(outputs_val.shape, (2, 1, 128)) - if scale_sqrt_depth: - emb_matrix_val *= params.input_dim**0.5 - self.assertAllClose(emb_matrix_val[89, :], outputs_val[0, 0, :]) - self.assertAllClose(emb_matrix_val[100, :], outputs_val[1, 0, :]) - - def testSharedSoftmaxLayerEmbLookup(self): - self._testSharedSoftmaxLayerEmbLookup() - - def testSharedSoftmaxLayerEmbLookupScaling(self): - self._testSharedSoftmaxLayerEmbLookup(True) - - -class FeedForwardNetTest(test_utils.TestCase): - - def testFeedForwardNetConstruction(self): - with self.session(use_gpu=False): - p = layers.FeedForwardNet.Params().Set( - name='ffn', - input_dim=10, - hidden_layer_dims=[20, 30], - batch_norm=True, - activation='TANH', - params_init=py_utils.WeightInit.Uniform(1.0)) - p.dropout.keep_prob = 0.5 - proj_l = p.Instantiate() - a = tf.constant(1.0, shape=[20, 10]) - proj_l.FPropDefaultTheta(a) - # check output_dim equals last hidden layer dim. - self.assertEqual(p.hidden_layer_dims[-1], proj_l.output_dim) - - p = layers.FeedForwardNet.Params().Set( - name='ffn2', - input_dim=10, - hidden_layer_dims=[20, 30], - batch_norm=True, - activation='TANH', - params_init=py_utils.WeightInit.Uniform(1.0)) - p.dropout = [ - layers.DropoutLayer.Params().Set(keep_prob=0.5), - layers.DropoutLayer.Params().Set(keep_prob=0.9) - ] - proj_l = p.Instantiate() - a = tf.constant(1.0, shape=[20, 10]) - proj_l.FPropDefaultTheta(a) - - p = layers.FeedForwardNet.Params().Set( - name='ffn3', - input_dim=10, - hidden_layer_dims=[20, 30], - batch_norm=[True, False], - activation=['TANH', 'RELU'], - params_init=py_utils.WeightInit.Uniform(1.0)) - p.dropout = [ - layers.DropoutLayer.Params().Set(keep_prob=0.5), - layers.DropoutLayer.Params().Set(keep_prob=0.9) - ] - proj_l = p.Instantiate() - a = tf.constant(1.0, shape=[20, 10]) - proj_l.FPropDefaultTheta(a) - - def testFeedForwardNet(self): - with self.session(use_gpu=False): - tf.random.set_seed(398847392) - np.random.seed(12345) - p = layers.FeedForwardNet.Params().Set( - name='ffn', - input_dim=10, - hidden_layer_dims=[20, 30], - batch_norm=False, - activation=['RELU', 'NONE']) - params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=837465638) - p.params_init = params_init - feedforward_net = p.Instantiate() - - p1 = layers.ProjectionLayer.Params().Set( - name='p1', - input_dim=10, - output_dim=20, - activation='RELU', - batch_norm=False) - p1.params_init = params_init - p1_l = p1.Instantiate() - - p2 = layers.ProjectionLayer.Params().Set( - name='p2', - input_dim=20, - output_dim=30, - activation='NONE', - batch_norm=False) - p2.params_init = params_init - p2_l = p2.Instantiate() - - a = tf.constant(np.random.rand(5, 10), dtype=tf.float32) - out1 = feedforward_net.FPropDefaultTheta(a) - - out2 = p2_l.FPropDefaultTheta(p1_l.FPropDefaultTheta(a)) - - self.evaluate(tf.global_variables_initializer()) - out1_v, out2_v = self.evaluate([out1, out2]) - self.assertAllClose(out1_v, out2_v) - - def testFeedForwardNetQuantized(self): - with self.session(use_gpu=False): - tf.random.set_seed(398847392) - np.random.seed(12345) - - cc_schedule = quant_utils.FakeQuantizationSchedule.Params().Set( - clip_start_step=1, - clip_end_step=2, - quant_start_step=2, - start_cap=8.0, - end_cap=2.0) - proj_qdomain = quant_utils.SymmetricScheduledClipQDomain.Params().Set( - cc_schedule=cc_schedule) - - p = layers.FeedForwardNet.Params().Set( - name='ffn', - input_dim=10, - hidden_layer_dims=[20, 30], - batch_norm=False, - activation=['RELU', 'NONE']) - p.qdomain.default = proj_qdomain.Copy() - params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=837465638) - p.params_init = params_init - feedforward_net = p.Instantiate() - - p1 = layers.ProjectionLayer.Params().Set( - name='p1', - input_dim=10, - output_dim=20, - activation='RELU', - batch_norm=False) - p1.qdomain.default = proj_qdomain.Copy() - p1.params_init = params_init - p1_l = p1.Instantiate() - - p2 = layers.ProjectionLayer.Params().Set( - name='p2', - input_dim=20, - output_dim=30, - activation='NONE', - batch_norm=False) - p2.params_init = params_init - p2.qdomain.default = proj_qdomain.Copy() - p2_l = p2.Instantiate() - - a = tf.constant(np.random.rand(5, 10), dtype=tf.float32) - out1 = feedforward_net.FPropDefaultTheta(a) - out2 = p2_l.FPropDefaultTheta(p1_l.FPropDefaultTheta(a)) - - self.evaluate(tf.global_variables_initializer()) - - self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), 5)) - out1_v, out2_v = self.evaluate([out1, out2]) - self.assertAllClose(out1_v, out2_v) - - def testFeedForwardNetBnFolded(self): - with self.session(use_gpu=False): - tf.random.set_seed(398847392) - np.random.seed(12345) - p = layers.FeedForwardNet.Params().Set( - name='ffn', - input_dim=10, - hidden_layer_dims=[20, 30], - batch_norm=True, - bn_fold_weights=True, - activation=['RELU', 'NONE']) - params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=837465638) - p.params_init = params_init - feedforward_net = p.Instantiate() - - p1 = layers.ProjectionLayer.Params().Set( - name='p1', - input_dim=10, - output_dim=20, - activation='RELU', - batch_norm=True, - bn_fold_weights=True) - p1.params_init = params_init - p1_l = p1.Instantiate() - - p2 = layers.ProjectionLayer.Params().Set( - name='p2', - input_dim=20, - output_dim=30, - activation='NONE', - batch_norm=True, - bn_fold_weights=True) - p2.params_init = params_init - p2_l = p2.Instantiate() - - a = tf.constant(np.random.rand(5, 10), dtype=tf.float32) - out1 = feedforward_net.FPropDefaultTheta(a) - - out2 = p2_l.FPropDefaultTheta(p1_l.FPropDefaultTheta(a)) - - self.evaluate(tf.global_variables_initializer()) - out1_v, out2_v = self.evaluate([out1, out2]) - self.assertAllClose(out1_v, out2_v) - - def testFeedForwardNetSmokeTest(self): - with self.session(use_gpu=False): - tf.random.set_seed(398847392) - np.random.seed(12345) - p = layers.FeedForwardNet.Params().Set( - name='ffn', - input_dim=10, - hidden_layer_dims=[20, 30], - activation=['RELU', 'NONE']) - params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=837465638) - p.params_init = params_init - feedforward_net = p.Instantiate() - a = tf.constant(np.random.rand(5, 10), dtype=tf.float32) - out = tf.reduce_sum(feedforward_net.FPropDefaultTheta(a)) - out_abs = tf.reduce_sum(tf.abs(feedforward_net.FPropDefaultTheta(a))) - - self.evaluate(tf.global_variables_initializer()) - # pyformat: disable - test_utils.CompareToGoldenSingleFloat(self, 8.190775, out.eval(), atol=1e-5) # pylint: disable=line-too-long - # pyformat: enable - test_utils.CompareToGoldenSingleFloat(self, 36.773586, out_abs.eval()) - - def testDropoutLayerTrain(self): - with self.session(use_gpu=True): - tf.random.set_seed(3980847392) - p = layers.DropoutLayer.Params() - p.keep_prob = 0.5 - p.random_seed = 1234 - p.name = 'dropout' - - dl = p.Instantiate() - - x = tf.random.normal([10, 10, 10, 3]) - xd = dl.FPropDefaultTheta(x) - x, xd = self.evaluate([x, xd]) - self.assertGreater((xd == 0).mean(), 0.3) - self.assertLess((xd == 0).mean(), 0.7) - self.assertAllClose(xd[xd != 0], x[xd != 0] / p.keep_prob) - - def testDropoutLayerEval(self): - with self.session(use_gpu=True), self.SetEval(True): - tf.random.set_seed(3980847392) - p = layers.DropoutLayer.Params() - p.keep_prob = 0.5 - p.random_seed = 1234 - p.name = 'dropout' - dl = p.Instantiate() - - x = tf.random.normal([10, 10, 10, 3]) - xd = dl.FPropDefaultTheta(x) - - x, xd = self.evaluate([x, xd]) - - self.assertAllEqual(xd, x) - - def testDeterministicSerialize(self): - p = layers.FeedForwardNet.Params().Set( - input_dim=4, - projection=layers.ProjectionLayer.Params().Set( - has_bias=True, - params_init=py_utils.WeightInit.KaimingUniformFanInRelu()), - activation='TANH', - hidden_layer_dims=[5, 5, 1], - batch_norm=True, - weight_norm=False) - base_serialized = p.ToTextWithTypes() - for _ in range(10): - serialized = p.ToTextWithTypes() - serialized_copy = p.Copy().ToTextWithTypes() - self.assertEqual(serialized, base_serialized) - self.assertEqual(serialized_copy, base_serialized) - for x in [serialized, serialized_copy]: - deserialized = layers.FeedForwardNet.Params() - deserialized.FromTextWithTypes(x) - self.assertEqual(p, deserialized) - - def testFeedForwardNetMeta(self): - p = layers.FeedForwardNet.Params().Set( - name='ffn', - input_dim=10, - hidden_layer_dims=[20, 30], - activation=['RELU', 'NONE']) - meta = p.cls.FPropMeta(p, tshape.Shape([5, 10])) - self.assertEqual( - meta.flops, - # Last layer has no activation fns but need to add bias. - 5 * 2 * (10 * 20 + 20 * 30) + 5 * 20 + (20 + 30)) - self.assertEqual(meta.out_shapes[0].ToTensorShape().as_list(), - [5, p.hidden_layer_dims[-1]]) - - -class AddingAccumulatorTest(test_utils.TestCase): - """Test for AddingAccumulator.""" - - def testAddingAccumulator(self): - with self.session(): - layer_p = layers.IdentityLayer.Params() - layer_p.name = 'test' - layer = layer_p.Instantiate() - - layer.RegisterAccumulator('acc1', layers.AddingAccumulator([], - tf.float32)) - - # Initial value. - self.assertEqual(0.0, layer.accumulators.acc1.GetValue().eval()) - - # Update/merge. - layer.accumulators.acc1.Update(1.0) - layer.accumulators.acc1.Update(1.0) - self.assertEqual(2.0, layer.accumulators.acc1.GetValue().eval()) - - # Reset. - layer.accumulators.Transform(lambda acc: acc.Reset()) - self.assertEqual(0.0, layer.accumulators.acc1.GetValue().eval()) - - -class BatchNormLayerNoPaddingTest(test_utils.TestCase, parameterized.TestCase): - - def testBatchNormLayerNoPaddingConstruction(self): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.BatchNormLayerNoPadding.Params() - params.name = 'bn' - params.dim = 2 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - layers.BatchNormLayerNoPadding(params) - bn_vars = tf.get_collection('BatchNormLayerNoPadding_vars') - bn_var_names = [x.name for x in bn_vars] - expected_var_names = [ - 'bn/beta/var:0', 'bn/gamma/var:0', 'bn/moving_mean/var:0', - 'bn/moving_variance/var:0' - ] - self.assertEqual(expected_var_names, bn_var_names) - - @parameterized.named_parameters({ - 'testcase_name': '_eval', - 'is_eval': True, - }, { - 'testcase_name': '_train', - 'is_eval': False, - }) - def testBatchNormLayerNoPaddingFProp(self, is_eval): - with self.session(use_gpu=True), self.SetEval(is_eval): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.BatchNormLayerNoPadding.Params() - params.name = 'bn' - params.dim = 3 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - bn_layer = layers.BatchNormLayerNoPadding(params) - bn_in1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 8, 3]), dtype=tf.float32) - - bn_out = bn_layer.FPropDefaultTheta(bn_in1) - sig1 = tf.reduce_sum(bn_out) - sig2 = tf.reduce_sum(bn_out * bn_out) - expected_sig1 = 2.6593573 if is_eval else 0 - expected_sig2 = 15.4642076 if is_eval else 47.850193 - - self.evaluate(tf.global_variables_initializer()) - self.assertAllClose(expected_sig1, sig1.eval(), atol=1e-5) - self.assertAllClose(expected_sig2, sig2.eval(), atol=1e-5) - - def testBatchNormLayerNoPaddingFPropUseGlobalStatsForTraining(self): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.BatchNormLayerNoPadding.Params() - params.name = 'bn' - params.dim = 3 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - bn_layer = layers.BatchNormLayerNoPadding(params) - bn_in1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 8, 3]), dtype=tf.float32) - - bn_out = bn_layer.FPropDefaultTheta(bn_in1) - sig1 = tf.reduce_sum(bn_out) - sig2 = tf.reduce_sum(bn_out * bn_out) - with self.session(use_gpu=True): - self.evaluate(tf.global_variables_initializer()) - self.assertAllClose(1.19209289551e-06, sig1.eval(), atol=1e-5) - self.assertAllClose(47.8501930237, sig2.eval(), atol=1e-5) - - def testBatchNormLayerNoPaddingPostTrainingStepUpdate(self): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.BatchNormLayerNoPadding.Params() - params.name = 'bn' - params.dim = 2 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - bn_layer = layers.BatchNormLayerNoPadding(params) - bn_layer.accumulators.counts.Update(0.0) - bn_layer.accumulators.mean_ss.Update([1.0, 1.0]) - bn_layer.accumulators.variance_ss.Update([5.0, 5.0]) - bn_updates = bn_layer.PostTrainingStepUpdate(tf.constant(100)) - - with self.session(use_gpu=True): - self.evaluate(tf.global_variables_initializer()) - self.evaluate(bn_updates) - moving_mean = self.evaluate(bn_layer.vars.moving_mean) - moving_std = self.evaluate(bn_layer.vars.moving_variance) - self.assertAllClose([0.0, 0.0], moving_mean) - self.assertAllClose([1.0, 1.0], moving_std) - - def testBatchNormLayerNoPaddingFPropForConv(self): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.BatchNormLayerNoPadding.Params() - params.name = 'bn_conv' - params.dim = 32 - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - bn_layer = layers.BatchNormLayerNoPadding(params) - bn_in1 = tf.constant( - np.random.normal(0.1, 0.5, [2, 8, 4, 32]), dtype=tf.float32) - - bn_out = bn_layer.FPropDefaultTheta(bn_in1) - sig1 = tf.reduce_sum(bn_out) - sig2 = tf.reduce_sum(bn_out * bn_out) - with self.session(use_gpu=True): - self.evaluate(tf.global_variables_initializer()) - self.assertAllClose(0.0, sig1.eval(), atol=1e-4) - self.assertAllClose(2039.398681, sig2.eval()) - - def _BuildDummyStackedBNLayer(self, splits): - num_micro_batches = 8 - if splits == 0: - endpoint = layers.BatchNormLayerNoPadding.Params().Set( - decay=0.997, name='bn', dim=1) - else: - cell_tpl = [] - for split in range(splits): - nets_to_split = [ - layers.BatchNormLayerNoPadding.Params().Set( - decay=0.997, name='bn_{}'.format(split), dim=1), - ] - split_layer = gpipe.FeatureExtractionLayer.Params().Set( - name='split_{}'.format(split), sub=nets_to_split) - cell_tpl.append(split_layer) - endpoint = gpipe.PipeliningLayer.Params().Set( - name='pipeline', - num_micro_batches=num_micro_batches, - cell_tpl=cell_tpl, - before_tpl=[]) - layer = endpoint.Instantiate() - return layer - - @parameterized.named_parameters({ - 'testcase_name': '_baseline', - 'splits': 0, - }, { - 'testcase_name': '_two_splits', - 'splits': 2, - }, { - 'testcase_name': '_four_splits', - 'splits': 4, - }) - def testBatchNormLayerNoPaddingAccumulators(self, splits): - batch_size = 1024 - with self.session(graph=tf.Graph()): - # Construct a network where loss = w * x + b - inputs = tf.concat([ - tf.ones([batch_size // 2, 1, 1, 1]), - tf.zeros([batch_size // 2, 1, 1, 1]) - ], - axis=0) - net = self._BuildDummyStackedBNLayer(splits) - logits = net.FPropDefaultTheta(inputs) - loss = tf.reduce_mean(logits) - grads = tf.gradients(loss, tf.trainable_variables()) - # Check the accumulator values - counts = [] - means = [] - variances = [] - for i in range(splits): - l = net.children['split_{}'.format(i)].children['bn_{}'.format(i)] - counts.append(l.accumulators.counts.GetValue()) - means.append(l.accumulators.mean_ss.GetValue()) - variances.append(l.accumulators.variance_ss.GetValue()) - if splits == 0: - counts.append(net.accumulators.counts.GetValue()) - means.append(net.accumulators.mean_ss.GetValue()) - variances.append(net.accumulators.variance_ss.GetValue()) - post_training_step_updates = net.PostTrainingStepUpdate( - net.theta.global_step) - - self.evaluate(tf.global_variables_initializer()) - _, count_vals, mean_vals, var_vals = self.evaluate( - [grads, counts, means, variances]) - - self.assertSameElements(count_vals, {batch_size}) - - self.assertEqual(batch_size // 2, mean_vals[0]) - if len(mean_vals) > 1: - self.assertSameElements(mean_vals[1:], {0}) - - self.assertEqual(batch_size // 2, var_vals[0]) - if len(var_vals) > 1: - self.assertSameElements(var_vals[1:], {0}) - self.evaluate(post_training_step_updates) - moving_vars = self.evaluate(tf.get_collection('moving_vars')) - self.assertEqual(0.0015, moving_vars[0]) - self.assertNear(0.997750, moving_vars[1], err=1.0e-6) - - -class LayerNormTest(test_utils.TestCase): - - def testLayerNormFProp(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - p = layers.LayerNorm.Params() - p.name = 'ln' - p.input_dim = 3 - layer_norm = layers.LayerNorm(p) - npy_input = np.random.normal(1.0, 0.5, - [2, 4, 4, p.input_dim]).astype('float32') - inputs = tf.constant(npy_input, dtype=tf.float32) - output = layer_norm.FPropDefaultTheta(inputs) - - self.evaluate(tf.global_variables_initializer()) - sym_output = self.evaluate(output) - - # Mean should be zero and variance should be close to one. - self.assertNear(0.0, sym_output.sum(), 1e-5) - self.assertNear(1.0, np.var(sym_output), 1e-4) - - # Compare with numpy. - mean = npy_input.mean(-1, keepdims=True) - variance = np.mean(np.square(npy_input - mean), -1, keepdims=True) - npy_output = (npy_input - mean) / np.sqrt(variance + p.epsilon) - self.assertAllClose(sym_output, npy_output) - - def testLayerNormFPropDirectScale(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - p = layers.LayerNorm.Params() - p.name = 'ln' - p.input_dim = 3 - p.direct_scale = True - layer_norm = layers.LayerNorm(p) - npy_input = np.random.normal(1.0, 0.5, - [2, 4, 4, p.input_dim]).astype('float32') - inputs = tf.constant(npy_input, dtype=tf.float32) - output = layer_norm.FPropDefaultTheta(inputs) - - self.evaluate(tf.global_variables_initializer()) - sym_output = self.evaluate(output) - - # Mean should be zero and variance should be close to one. - self.assertNear(0.0, sym_output.sum(), 1e-5) - self.assertNear(1.0, np.var(sym_output), 1e-4) - - # Compare with numpy. - mean = npy_input.mean(-1, keepdims=True) - variance = np.mean(np.square(npy_input - mean), -1, keepdims=True) - npy_output = (npy_input - mean) / np.sqrt(variance + p.epsilon) - self.assertAllClose(sym_output, npy_output) - - def testLayerNormBProp(self): - with self.session(use_gpu=True) as sess: - tf.random.set_seed(398847392) - np.random.seed(12345) - p = layers.LayerNorm.Params() - p.name = 'ln' - p.input_dim = 3 - layer_norm = layers.LayerNorm(p) - - inputs = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 4, p.input_dim]), dtype=tf.float32) - output = layer_norm.FPropDefaultTheta(inputs) - loss = tf.reduce_sum(output) - - all_vars = tf.trainable_variables() - self.assertEqual(2, len(all_vars)) - - grads = tf.gradients(loss, all_vars) - self.evaluate(tf.global_variables_initializer()) - sym_grads = [sg.eval() for sg in grads] - num_grads = [ - test_utils.ComputeNumericGradient(sess, loss, v) for v in all_vars - ] - - for sg, ng in zip(sym_grads, num_grads): - self.assertAllClose(sg, ng, rtol=1e-02, atol=1e-02) - - -class CategoricalLayerNormTest(test_utils.TestCase): - - def testCategoricalLayerNormFProp(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - p = layers.CategoricalLayerNorm.Params() - p.name = 'cat_ln' - p.input_dim = 3 - p.num_classes = 2 - layer_norm = layers.CategoricalLayerNorm(p) - npy_input = np.random.normal(1.0, 0.5, - [2, 4, 4, p.input_dim]).astype('float32') - - inputs = tf.constant(npy_input, dtype=tf.float32) - output = layer_norm.FPropDefaultTheta(inputs) - - self.evaluate(tf.global_variables_initializer()) - # Set different bias and scale for different copy of ln params - self.evaluate(tf.assign(layer_norm.vars.scale_0, [0.0] * 3)) - self.evaluate(tf.assign(layer_norm.vars.scale_1, [1.0] * 3)) - self.evaluate(tf.assign(layer_norm.vars.bias_0, [0.0] * 3)) - self.evaluate(tf.assign(layer_norm.vars.bias_1, [1.0] * 3)) - - output = layer_norm.FPropDefaultTheta(inputs) - sym_output_c1 = self.evaluate(output) - - # Redefine output to use the value of new theta - theta = layer_norm.theta - theta.class_index = tf.constant(1, dtype=tf.int32) - output = layer_norm.FProp(theta, inputs) - sym_output_c2 = self.evaluate(output) - - # Mean should be zero and variance should be close to one. - self.assertNotAllClose(sym_output_c1, sym_output_c2) - self.assertNear(0.0, sym_output_c1.mean(), 1e-5) - self.assertNear(1.0, np.var(sym_output_c1), 1e-4) - # Mean should be 1 and variance should be close to 2^2 - self.assertNear(1.0, sym_output_c2.mean(), 1e-5) - self.assertNear(4.0, np.var(sym_output_c2), 1e-4) - - # Compare with numpy. - mean = npy_input.mean(-1, keepdims=True) - variance = np.mean(np.square(npy_input - mean), -1, keepdims=True) - npy_output = (npy_input - mean) / np.sqrt(variance + p.epsilon) - self.assertAllClose(sym_output_c1, npy_output) - - variance = np.mean(np.square(npy_input - mean), -1, keepdims=True) - npy_output = 2 * (npy_input - mean) / np.sqrt(variance + p.epsilon) + 1 - self.assertAllClose(sym_output_c2, npy_output) - - def testCategoricalLayerNormBProp(self): - with self.session(use_gpu=True) as sess: - tf.random.set_seed(398847392) - np.random.seed(12345) - p = layers.CategoricalLayerNorm.Params() - p.name = 'cat_ln' - p.input_dim = 3 - p.num_classes = 2 - layer_norm = layers.CategoricalLayerNorm(p) - - inputs = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 4, p.input_dim]), dtype=tf.float32) - output = layer_norm.FPropDefaultTheta(inputs) - loss = tf.reduce_sum(output) - - all_vars = tf.trainable_variables() - self.assertEqual(4, len(all_vars)) - grads = tf.gradients(loss, all_vars) - self.evaluate(tf.global_variables_initializer()) - print('grads = {}'.format(grads)) - sym_grads = [self.evaluate(sg) for sg in grads] - num_grads = [ - test_utils.ComputeNumericGradient(sess, loss, v) for v in all_vars - ] - - for sg, ng in zip(sym_grads, num_grads): - self.assertAllClose(sg, ng, rtol=1e-02, atol=1e-02) - - -class DeterministicDropoutTest(test_utils.TestCase, parameterized.TestCase): - - def testDeterministicDropoutLayer(self): - params = layers.DeterministicDropoutLayer.Params().Set(keep_prob=0.7) - params.name = 'drop' - dropout = layers.DeterministicDropoutLayer(params) - - x = tf.ones([4, 6], dtype=tf.float32) - x_expected = np.array([ - [1, 0, 0, 0, 1, 1], - [1, 1, 1, 1, 1, 1], - [1, 0, 0, 1, 1, 0], - [1, 0, 0, 1, 1, 1], - ]) / 0.7 - - with self.session(): - tf.assign(py_utils.GetOrCreateGlobalStepVar(), 1234).eval() - py_utils.ResetStepSeed(seed=5678) - x_val = dropout.FPropDefaultTheta(x).eval() - self.assertAllClose(x_expected, x_val) - self.assertEqual(5679, py_utils.GetStepSeed().eval()) - - # Different step seed gives different result. - x_val = dropout.FPropDefaultTheta(x).eval() - self.assertNotAllClose(x_expected, x_val) - - # Different global step gives different result - tf.assign(py_utils.GetOrCreateGlobalStepVar(), 1235).eval() - py_utils.ResetStepSeed(seed=5678) - x_val = dropout.FPropDefaultTheta(x).eval() - self.assertNotAllClose(x_expected, x_val) - - # The same seeds in the same session is consistent. - tf.assign(py_utils.GetOrCreateGlobalStepVar(), 1234).eval() - py_utils.ResetStepSeed(seed=5678) - x_val = dropout.FPropDefaultTheta(x).eval() - self.assertAllClose(x_expected, x_val) - - # The same seeds in a different session is consistent. - with self.session(): - tf.assign(py_utils.GetOrCreateGlobalStepVar(), 1234).eval() - py_utils.ResetStepSeed(seed=5678) - x_val = dropout.FPropDefaultTheta(x).eval() - self.assertAllClose(x_expected, x_val) - - def testNoiseShapeBroadcastDims(self): - params = layers.DeterministicDropoutLayer.Params().Set( - keep_prob=0.7, noise_shape_broadcast_dims=[-1]) - params.name = 'drop' - dropout = layers.DeterministicDropoutLayer(params) - - x = tf.ones([4, 6]) - x_expected = np.array([ - [1, 1, 1, 1, 1, 1], - [0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 0], - ]) / 0.7 - - with self.session(): - tf.assign(py_utils.GetOrCreateGlobalStepVar(), 1234).eval() - self.assertEqual(1234, dropout.theta.global_step.eval()) - py_utils.ResetStepSeed(seed=5678) - x_val = dropout.FPropDefaultTheta(x).eval() - self.assertEqual(5679, py_utils.GetStepSeed().eval()) - self.assertAllClose(x_expected, x_val) - - @parameterized.named_parameters( - { - 'testcase_name': 'baseline', - 'splits': 1, - 'num_micro_batches': 1 - }, - { - 'testcase_name': 'OneSplitTwoMicroBatches', - 'splits': 1, - 'num_micro_batches': 2 - }, - { - 'testcase_name': 'TwoSplitsOneMicroBatch', - 'splits': 2, - 'num_micro_batches': 1 - }, - { - 'testcase_name': 'TwoSplitsTwoMicroBatches', - 'splits': 2, - 'num_micro_batches': 2 - }, - ) - def testDropoutInRecurrent(self, splits=1, num_micro_batches=1): - """Test to verify the drop mask used in fprop and bprop is identical.""" - assert splits in [1, 2, 4] - with self.session(): - tf.random.set_seed(12345) - num_layers = 4 - # Build a model with 4 dropout layers. - blocks = [] - for l in range(num_layers): - blocks.append(layers.DeterministicDropoutLayer.Params().Set( - name='dropout_{}'.format(l), keep_prob=0.7)) - # Divide the model into splits partitions. - cell_tpl = [] - blocks_per_split = num_layers // splits - for i in range(splits): - sub = blocks[i * blocks_per_split:(i + 1) * blocks_per_split] - cell_tpl.append(gpipe.FeatureExtractionLayer.Params().Set( - name='cell_{}'.format(i), sub=sub)) - # Parallelize partitions using pipeline. - p = gpipe.PipeliningLayer.Params().Set( - name='pipeline', - num_micro_batches=num_micro_batches, - cell_tpl=cell_tpl) - # Fake input - x = tf.ones([2, 3]) - # Construct weights. - w = tf.get_variable( - 'w', shape=[2, 3], initializer=tf.constant_initializer([[1] * 3] * 2)) - mdl = p.Instantiate() - y = mdl.FPropDefaultTheta(x * w) - # Construct loss function such that gradients = final activation. - loss = tf.reduce_sum(y) - grads = py_utils.ComputeGradients(loss, py_utils.NestedMap(w=w)) - self.evaluate(tf.global_variables_initializer()) - y_val = self.evaluate(y) - grads_val = self.evaluate(grads.w.grad) - self.assertAllClose(y_val, grads_val) - - -class GradNormTrackerTest(test_utils.TestCase): - - def testGradNormTracker(self): - with self.session(use_gpu=False) as sess: - tf.random.set_seed(398847392) - np.random.seed(12345) - p = layers.GradNormTracker.Params().Set( - name='grad_norm_tracker', clip_threshold=3.0) - grad_norm_tracker = p.Instantiate() - grad_norm = tf.placeholder(tf.float32) - grad_norm_clip = grad_norm_tracker.FPropDefaultTheta(grad_norm) - - self.evaluate(tf.global_variables_initializer()) - - random_normal = np.exp(np.random.normal(5.0, 1.0, size=10000)) - # We are expected to reject 16% of the outliers. - outliers = np.exp(np.random.normal(7.0, 1.0, size=100)) - total_rejections = 0 - for i in range(100): - for j in range(100): - sess.run([grad_norm_clip], {grad_norm: random_normal[i * 100 + j]}) - clip = sess.run([grad_norm_clip], {grad_norm: outliers[i]})[0] - if clip == 0.0: - total_rejections += 1 - # Q(yonghui): Why is total_rejections not deterministic? - print('total_rejections', total_rejections) - self.assertGreater(total_rejections, 5) - - def testGradNormTrackerClipCapMin(self): - with self.session(use_gpu=False) as sess: - tf.random.set_seed(398847392) - np.random.seed(12345) - p = layers.GradNormTracker.Params().Set( - name='grad_norm_tracker', - clip_threshold=3.0, - grad_norm_clip_cap_min=math.exp(10.0)) - grad_norm_tracker = p.Instantiate() - grad_norm = tf.placeholder(tf.float32) - grad_norm_clip = grad_norm_tracker.FPropDefaultTheta(grad_norm) - - self.evaluate(tf.global_variables_initializer()) - - random_normal = np.exp(np.random.normal(5.0, 1.0, size=10000)) - # We expect no outliers being rejected due to the grad_norm_clip_cap_min. - outliers = np.exp(np.random.normal(7.0, 1.0, size=100)) - total_rejections = 0 - for i in range(100): - for j in range(100): - sess.run([grad_norm_clip], {grad_norm: random_normal[i * 100 + j]}) - clip = sess.run([grad_norm_clip], {grad_norm: outliers[i]})[0] - if clip == 0.0: - total_rejections += 1 - print('total_rejections', total_rejections) - self.assertEqual(total_rejections, 0) - - def testGradNormTrackerHasNan(self): - with self.session(use_gpu=False) as sess: - tf.random.set_seed(398847392) - np.random.seed(12345) - p = layers.GradNormTracker.Params().Set( - name='grad_norm_tracker', clip_threshold=3.0) - grad_norm_tracker = p.Instantiate() - grad_norm = tf.placeholder(tf.float32) - has_nan = tf.cast(tf.ones([]), dtype=tf.bool) - grad_norm_clip = grad_norm_tracker.FPropDefaultTheta(grad_norm, has_nan) - - self.evaluate(tf.global_variables_initializer()) - - random_normal = np.exp(np.random.normal(5.0, 1.0, size=10000)) - outliers = np.exp(np.random.normal(7.0, 1.0, size=100)) - total_rejections = 0 - for i in range(100): - for j in range(100): - sess.run([grad_norm_clip], {grad_norm: random_normal[i * 100 + j]}) - clip = sess.run([grad_norm_clip], {grad_norm: outliers[i]})[0] - if clip == 0.0: - total_rejections += 1 - self.assertEqual(total_rejections, 100) - - -class HighwaySkipLayerTest(test_utils.TestCase): - - def testHighwaySkipLayerConstruction(self): - with self.session(use_gpu=False): - p = layers.HighwaySkipLayer.Params().Set( - name='gffn', - input_dim=10, - carry_bias_init=1.0, - couple_carry_transform_gates=True, - batch_norm=False, - params_init=py_utils.WeightInit.Uniform(1.0)) - proj_l = p.Instantiate() - a = tf.constant(1.0, shape=[20, 10]) - b = tf.constant(-2.0, shape=[20, 10]) - proj_l.FPropDefaultTheta(a, b) - - def testHighwaySkipLayerCarryGate(self): - with self.session(use_gpu=False): - tf.random.set_seed(398847392) - np.random.seed(12345) - p = layers.HighwaySkipLayer.Params().Set( - name='gffn', - input_dim=10, - carry_bias_init=1000.0, - couple_carry_transform_gates=True, - batch_norm=False, - params_init=py_utils.WeightInit.Uniform(1.0)) - proj_l = p.Instantiate() - a = tf.constant(1.0, shape=[20, 10]) - b = tf.constant(-2.0, shape=[20, 10]) - out = proj_l.FPropDefaultTheta(a, b) - self.evaluate(tf.global_variables_initializer()) - a, out = self.evaluate([a, out]) - self.assertAllClose(a, out) - - -class GatingLayerTest(test_utils.TestCase): - - def testGatingLayerConstruction(self): - with self.session(use_gpu=False): - p = layers.GatingLayer.Params().Set( - name='gating', - input_dim=10, - carry_bias_init=1.0, - params_init=py_utils.WeightInit.Uniform(1.0)) - gate = p.Instantiate() - a = tf.constant(1.0, shape=[20, 10]) - b = tf.constant(-2.0, shape=[20, 10]) - gate.FPropDefaultTheta(a, b) - - def testGatingLayerFProp(self): - with self.session(use_gpu=True): - p = layers.GatingLayer.Params().Set( - name='gate', input_dim=6, has_bias=False) - gate = p.Instantiate() - a = tf.constant(np.random.uniform(size=[10, 6]), dtype=tf.float32) - b = tf.constant(np.random.uniform(size=[10, 6]), dtype=tf.float32) - out = gate.FPropDefaultTheta(a, b) - self.evaluate(tf.global_variables_initializer()) - actual_out, w = self.evaluate([out, gate.theta.carry_gate.w]) - self.assertAllEqual(np.shape(w), [12, 6]) - w = np.matmul(a.eval(), w[:6, :]) + np.matmul(b.eval(), w[6:, :]) - sigmoid_w = 1 / (1 + np.exp(-w)) - expected_out = a * sigmoid_w + b * (1 - sigmoid_w) - self.assertAllClose(actual_out, expected_out) - - def testGatingLayerFPropSaturated(self): - with self.session(use_gpu=True): - p = layers.GatingLayer.Params().Set( - name='gate', input_dim=6, has_bias=True, carry_bias_init=100) - gate = p.Instantiate() - a = tf.constant(np.random.uniform(size=[10, 6]), dtype=tf.float32) - b = tf.constant(np.random.uniform(size=[10, 6]), dtype=tf.float32) - out = gate.FPropDefaultTheta(a, b) - self.evaluate(tf.global_variables_initializer()) - # High initial bias, causing the carry gate to saturate and the - # output will be very close to a. - self.assertAllClose(out.eval(), a.eval()) - - -class UniformLabelSmootherTest(test_utils.TestCase): - - def testUniformLabelSmoother(self): - with self.session(use_gpu=False): - params = layers.UniformLabelSmoother.Params() - params.name = 'uls' - params.num_classes = 5 - params.uncertainty = 0.1 - - smooth_layer = layers.UniformLabelSmoother(params) - target_labels = tf.constant([[0, 1, 2, 3, 3, 3, 4]], dtype=tf.int32) - target_ids = tf.constant([[0, 0, 1, 2, 3, 3, 3]], dtype=tf.int32) - target_paddings = tf.zeros(tf.shape(target_ids)) - output = smooth_layer.FPropDefaultTheta(target_paddings, target_labels, - target_ids) - self.evaluate(tf.global_variables_initializer()) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [[ - [0.89999998, 0.025 , 0.025 , 0.025 , 0.025 ], - [0.025 , 0.89999998, 0.025 , 0.025 , 0.025 ], - [0.025 , 0.025 , 0.89999998, 0.025 , 0.025 ], - [0.025 , 0.025 , 0.025 , 0.89999998, 0.025 ], - [0.025 , 0.025 , 0.025 , 0.89999998, 0.025 ], - [0.025 , 0.025 , 0.025 , 0.89999998, 0.025 ], - [0.025 , 0.025 , 0.025 , 0.025 , 0.89999998] - ]] - # pyformat: enable - # pylint: enable=bad-whitespace - output_v = output.eval() - self.assertAllClose(expected_output, output_v, atol=1e-2, rtol=1e-2) - self.assertAllClose(np.ones(output_v.shape[:-1]), output_v.sum(-1)) - - def testUniformLabelSmootherLargerToken(self): - with self.session(use_gpu=False): - params = layers.UniformLabelSmoother.Params() - params.name = 'uls' - params.num_classes = 5 - params.uncertainty = 0.1 - params.uncertainty_larger = 0.2 - params.token_id_uncertainty_larger = 4 - - smooth_layer = layers.UniformLabelSmoother(params) - target_labels = tf.constant([[0, 1, 2, 3, 3, 3, 3]], dtype=tf.int32) - target_ids = tf.constant([[0, 0, 1, 2, 4, 4, 4]], dtype=tf.int32) - target_paddings = tf.zeros(tf.shape(target_ids)) - output = smooth_layer.FPropDefaultTheta(target_paddings, target_labels, - target_ids) - self.evaluate(tf.global_variables_initializer()) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [[ - [0.89999998, 0.025 , 0.025 , 0.025 , 0.025 ], - [0.025 , 0.89999998, 0.025 , 0.025 , 0.025 ], - [0.025 , 0.025 , 0.89999998, 0.025 , 0.025 ], - [0.025 , 0.025 , 0.025 , 0.89999998, 0.025 ], - [0.05 , 0.05 , 0.05 , 0.80000001, 0.05 ], - [0.05 , 0.05 , 0.05 , 0.80000001, 0.05 ], - [0.05 , 0.05 , 0.05 , 0.80000001, 0.05 ] - ]] - # pyformat: enable - # pylint: enable=bad-whitespace - output_v = output.eval() - self.assertAllClose(expected_output, output_v, atol=1e-2, rtol=1e-2) - self.assertAllClose(np.ones(output_v.shape[:-1]), output_v.sum(-1)) - - -class WeightedSumLayerTest(test_utils.TestCase): - - def testWeightedSumLayer(self): - with self.session(use_gpu=True): - np.random.seed(505837249) - depth = 4 - batch = 2 - n_sources = 3 - ctxs = [[[1.0, 2.0, 3.0, 4.0], [2.0, 3.0, 4.0, 5.0]], - [[3.0, 4.0, 5.0, 6.0], [6.0, 7.0, 8.0, 9.0]], - [[4.0, 5.0, 6.0, 7.0], [7.0, 8.0, 1.0, 2.0]]] - p = layers.WeightedSumLayer.Params() - p.name = 'transparent_layer' - p.num_sources = n_sources - p.random_seed = 505837249 - merger = p.Instantiate() - - ctxs = [tf.expand_dims(i, 2) for i in ctxs] - ctx = tf.squeeze(merger.FProp(merger.theta, ctxs), 2) - self.evaluate(tf.global_variables_initializer()) - actual_ctx = self.evaluate(ctx) - - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [[ 2.66666675, 3.66666675, 4.66666698, 5.66666698], - [ 5.0, 6.0, 4.33333349, 5.33333349]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertEqual(actual_ctx.shape, (batch, depth)) - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - - def testWeightedSumLayerGlobalWeightAndMinimalProb(self): - with self.session(use_gpu=True): - np.random.seed(505837249) - depth = 4 - batch = 2 - n_sources = 3 - ctxs = [[[1.0, 2.0, 3.0, 4.0], [2.0, 3.0, 4.0, 5.0]], - [[3.0, 4.0, 5.0, 6.0], [6.0, 7.0, 8.0, 9.0]], - [[4.0, 5.0, 6.0, 7.0], [7.0, 8.0, 1.0, 2.0]]] - p = layers.WeightedSumLayer.Params() - p.name = 'transparent_layer' - p.num_sources = n_sources - p.random_seed = 505837249 - p.minimal_prob = 0.01 - p.global_weight_scale = 10.0 - merger = p.Instantiate() - - ctxs = [tf.expand_dims(i, 2) for i in ctxs] - ctx = tf.squeeze(merger.FProp(merger.theta, ctxs), 2) - self.evaluate(tf.global_variables_initializer()) - actual_ctx = self.evaluate(ctx) - - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [[ 2.66666675, 3.66666675, 4.66666698, 5.66666698], - [ 5.0, 6.0, 4.33333349, 5.33333349]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertEqual(actual_ctx.shape, (batch, depth)) - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - - -class DeconvLayerTest(test_utils.TestCase): - - def testDeconvLayerFProp(self): - with self.session(use_gpu=True): - tf.random.set_seed(398847392) - np.random.seed(12345) - params = layers.DeconvLayer.Params() - params.name = 'deconv' - params.filter_shape = [3, 3, 2, 8] - params.filter_stride = [2, 2] - params.params_init = py_utils.WeightInit.Gaussian(0.1) - - conv_layer = params.Instantiate() - inputs = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 4, 8]), dtype=tf.float32) - - out = conv_layer.FPropDefaultTheta(inputs) - out_shape = conv_layer.OutShape(tf.shape(inputs)) - - self.evaluate(tf.global_variables_initializer()) - out_v, shape_v = self.evaluate([out, out_shape]) - self.assertAllEqual(shape_v, [2, 8, 8, 2]) - self.assertAllEqual(out_v.shape, shape_v) - - summary = np.sum(np.square(out_v), axis=(1, 2, 3)) - tf.logging.info('testDeconvLaye rFProp actual = %s', - np.array_repr(summary)) - self.assertAllClose([4.77159977, 5.47860432], summary) - - -class GatedAverageLayerTest(test_utils.TestCase): - - def testGatedAverageLayer(self): - with self.session(use_gpu=True): - np.random.seed(505837249) - depth = 4 - batch = 2 - num_inputs = 3 - - inp_1 = np.asarray([[0.0, 0.0, 0.0, 0.0], [-1.0, -1.0, 1.0, 1.0]], - dtype=np.float32) - inp_2 = np.asarray([[1.0, 1.0, 1.0, 1.0], [-1.0, -1.0, 1.0, 1.0]], - dtype=np.float32) - inp_3 = np.asarray([[-1.0, -1.0, -1.0, -1.0], [-1.0, -1.0, 1.0, 1.0]], - dtype=np.float32) - p = layers.GatedAverageLayer.Params() - p.name = 'gated_avg_layer' - p.num_inputs = num_inputs - p.num_nodes = depth - p.random_seed = 505837249 - g_avg = p.Instantiate() - - avg = g_avg.FProp(g_avg.theta, [inp_1, inp_2, inp_3]) - self.evaluate(tf.global_variables_initializer()) - actual_avg = self.evaluate(avg) - - expected_avg = [[0.092766, 0.092766, 0.092766, 0.092766], - [-1., -1., 1., 1.]] - self.assertEqual(actual_avg.shape, (batch, depth)) - self.assertAllClose(expected_avg, actual_avg, rtol=1e-05, atol=1e-05) - - -class LHUCLayerTest(test_utils.TestCase): - - def testLHUCLayer(self): - with self.session(use_gpu=True): - np.random.seed(505837249) - depth = 4 - batch = 2 - - inp = np.asarray([[1.0, 1.0, 1.0, 1.0], [-1.0, -1.0, -1.0, -1.0]], - dtype=np.float32) - p = layers.LHUCLayer.Params() - p.name = 'lhuc_layer' - p.input_dim = depth - p.random_seed = 505837249 - lhuc = p.Instantiate() - - lhuc = lhuc.FProp(lhuc.theta, inp) - self.evaluate(tf.global_variables_initializer()) - actual_avg = self.evaluate(lhuc) - - # pylint: disable=bad-whitespace - # pyformat: disable - expected_avg = [[1.0, 1.0, 1.0, 1.0], [-1.0, -1.0, -1.0, -1.0]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertEqual(actual_avg.shape, (batch, depth)) - self.assertAllClose(expected_avg, actual_avg, rtol=1e-05, atol=1e-05) - - -class ResidualAdapterLayerTest(test_utils.TestCase): - - def testResidualAdapterLayer(self): - with self.session(use_gpu=True): - np.random.seed(505837249) - depth = 4 - batch = 2 - - inp = np.asarray([[1.0, 1.0, 1.0, 1.0], [-1.0, -1.0, -1.0, -1.0]], - dtype=np.float32) - p = layers.ResidualAdapterLayer.Params() - p.name = 'resadap_layer' - p.input_dim = depth - p.bottleneck_dim = 2 - p.random_seed = 505837249 - resadap = p.Instantiate() - - resadap = resadap.FProp(resadap.theta, inp) - self.evaluate(tf.global_variables_initializer()) - actual_avg = self.evaluate(resadap) - - # pylint: disable=bad-whitespace - # pyformat: disable - expected_avg = [[1.0, 1.0, 1.0, 1.0], [-1.0, -1.0, -1.0, -1.0]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertEqual(actual_avg.shape, (batch, depth)) - self.assertAllClose(expected_avg, actual_avg, rtol=1e-05, atol=1e-05) - - -class GluLayerTest(test_utils.TestCase): - - def testGlu(self): - with self.session(use_gpu=True): - tf.random.set_seed(3980847392) - inputs = tf.random.normal([5, 2, 3], seed=948387483) - paddings = tf.zeros([5, 2]) - p = layers.GluLayer.Params() - p.name = 'glu_layers' - p.input_dim = 3 - glu_layer = layers.GluLayer(p) - - h = glu_layer.FPropDefaultTheta(inputs, paddings) - self.evaluate(tf.global_variables_initializer()) - actual_layer_output = self.evaluate(h) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [[ -1.84272185e-01, -3.82728219e-01, 8.69752645e-01], - [ 4.42533880e-01, 1.51665461e+00, 3.26201534e+00]], - [[ -7.06624031e-01, -6.52632236e-01, 1.22156203e+00], - [ 1.66484845e+00, 5.98078966e-01, 1.14039946e+00]], - [[ 3.26439053e-01, 2.47359693e-01, -1.14889514e+00], - [ 7.71084905e-01, 1.07083774e+00, 1.74589559e-01]], - [[ 5.70576251e-01, 7.95466423e-01, -4.07778949e-01], - [ -8.71581078e-01, -5.38501918e-01, -2.50373930e-01]], - [[ -3.88817638e-01, 5.84501982e-01, -6.60797715e-01], - [ -1.34579837e+00, -2.18637614e-03, 1.55258143e+00]]] - # pyformat: enable - # pylint: enable=bad-whitespace - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testGluWithoutResidual(self): - with self.session(use_gpu=True): - tf.random.set_seed(3980847392) - inputs = tf.random.normal([5, 2, 3], seed=948387483) - paddings = tf.zeros([5, 2]) - p = layers.GluLayer.Params() - p.name = 'glu_layers' - p.input_dim = 3 - p.output_dim = 4 - p.apply_residual = False - glu_layer = layers.GluLayer(p) - - h = glu_layer.FPropDefaultTheta(inputs, paddings) - self.evaluate(tf.global_variables_initializer()) - actual_layer_output = self.evaluate(h) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [[ 0.2498899 , 0. , 0.62683833, 0. ], - [ 0.34115699, 0. , 0.38020864, 0. ]], - [[ 0.3014423 , 0. , 0.59274423, 0. ], - [ 0. , 0.35897657, 0.2908403 , 0.03678071]], - [[ 0. , 0.78786391, 0. , 0.38839644], - [ 0. , 0.44012907, 0. , 0.41553062]], - [[ 0. , 0.61838603, 0. , 0.41521466], - [ 0.34117079, 0. , 0.0372162 , 0. ]], - [[ 0. , 0. , 0. , 0.28136203], - [ 0.34413674, 0. , 0.30943182, 0. ]]] - # pyformat: enable - # pylint: enable=bad-whitespace - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - -class MultitaskAdapterLayerTest(test_utils.TestCase, parameterized.TestCase): - - def _MultitaskAdapterParams(self, data_format='TBC'): - return layers.MultitaskAdapterLayer.Params().Set( - name='multi_adapter', - input_dim=4, - bottleneck_dim=2, - num_tasks=3, - data_format=data_format, - random_seed=505837249) - - @parameterized.parameters('TBC', 'BTC') - def testSingleStepFProp(self, data_format): - with self.session(use_gpu=True): - np.random.seed(1234567) - # Inputs are of shape [1, batch, input_dim] (single time step) - # Batch elements 0, 2, and 3 are identical, but 0 and 2 have the same - # task ID where as 3 has a different task ID. - inputs = tf.constant([[[0.5, 0.3, -0.2, 0.0], [0.0, 0.7, -1.0, 2.0], - [0.5, 0.3, -0.2, 0.0], [0.5, 0.3, -0.2, 0.0]]], - dtype=tf.float32) - tasks = tf.constant([1, 0, 1, 0], dtype=tf.int32) - if data_format == 'BTC': - inputs = tf.transpose(inputs, [1, 0, 2]) - p = self._MultitaskAdapterParams(data_format) - adapter = p.Instantiate() - output = adapter.FProp(adapter.theta, inputs, tasks) - self.evaluate(tf.global_variables_initializer()) - actual = self.evaluate(output) - if data_format == 'BTC': - actual = tf.transpose(actual, [1, 0, 2]) - tf.logging.info('testSingleStepFProp actual=%r' % actual) - expected = [[[1.1579462, 1.2241995, -0.6177901, 0.23089096], - [0.05022227, 0.9056754, -0.3771479, 2.1245508], - [1.1579462, 1.2241995, -0.6177901, 0.23089096], - [0.55389655, 0.5207228, 0.46842042, 0.13366304]]] - self.assertEqual(actual.shape, (1, 4, 4)) - # Batch elements 0 and 2 are equal because they had the same input - # and the same task ID. - self.assertAllClose(actual[0][0], actual[0][2], rtol=1e-05, atol=1e-05) - self.assertAllClose(expected, actual, rtol=1e-05, atol=1e-05) - - @parameterized.parameters('TBC', 'BTC') - def testMultiStepFProp(self, data_format): - with self.session(use_gpu=True): - np.random.seed(1234567) - # Inputs are same as above but of shape [time, batch, input_dim] - inputs = tf.constant([[[0.5, 0.3, -0.2, 0.0], [0.0, 0.7, -1.0, 2.0]], - [[0.5, 0.3, -0.2, 0.0], [0.5, 0.3, -0.2, 0.0]]], - dtype=tf.float32) - # tasks is of shape [batch] indicating one task for each sequence. - tasks = tf.constant([1, 0], dtype=tf.int32) - if data_format == 'BTC': - inputs = tf.transpose(inputs, [1, 0, 2]) - p = self._MultitaskAdapterParams(data_format) - adapter = p.Instantiate() - output = adapter.FProp(adapter.theta, inputs, tasks) - self.evaluate(tf.global_variables_initializer()) - actual = self.evaluate(output) - if data_format == 'BTC': - actual = tf.transpose(actual, [1, 0, 2]) - tf.logging.info('testMultiStepFProp actual=%r' % actual) - # Output is same as above but with shape same as input. - expected = [[[1.1579462, 1.2241995, -0.6177901, 0.23089096], - [0.05022227, 0.9056754, -0.3771479, 2.1245508]], - [[1.1579462, 1.2241995, -0.6177901, 0.23089096], - [0.55389655, 0.5207228, 0.46842045, 0.13366304]]] - self.assertEqual(actual.shape, (2, 2, 4)) - self.assertAllClose(expected, actual, rtol=1e-05, atol=1e-05) - - @parameterized.parameters('TBC', 'BTC') - def testSpecifyTaskPerTimestepFProp(self, data_format): - with self.session(use_gpu=True): - np.random.seed(1234567) - inputs = tf.constant([[[0.5, 0.3, -0.2, 0.0], [0.0, 0.7, -1.0, 2.0]], - [[0.5, 0.3, -0.2, 0.0], [0.5, 0.3, -0.2, 0.0]]], - dtype=tf.float32) - # tasks are same as above but of shape [time, batch] indicating that - # we should look up adapter params per timestep. In this example we - # still have the task ID consistent across timesteps in order to - # replicate the previous test's output. - tasks = tf.constant([[1, 0], [1, 0]], dtype=tf.int32) - if data_format == 'BTC': - inputs = tf.transpose(inputs, [1, 0, 2]) - tasks = tf.transpose(tasks, [1, 0]) - p = self._MultitaskAdapterParams(data_format) - adapter = p.Instantiate() - output = adapter.FProp(adapter.theta, inputs, tasks) - self.evaluate(tf.global_variables_initializer()) - actual = self.evaluate(output) - if data_format == 'BTC': - actual = tf.transpose(actual, [1, 0, 2]) - tf.logging.info('testSpecifyTaskPerTimestepFProp actual=%r' % actual) - # Output is same as above. - expected = [[[1.1579462, 1.2241995, -0.6177901, 0.23089096], - [0.05022227, 0.9056754, -0.3771479, 2.1245508]], - [[1.1579462, 1.2241995, -0.6177901, 0.23089096], - [0.55389655, 0.5207228, 0.46842042, 0.13366304]]] - self.assertEqual(actual.shape, (2, 2, 4)) - self.assertAllClose(expected, actual, rtol=1e-05, atol=1e-05) - - @parameterized.parameters('TBC', 'BTC') - def testDifferentTaskPerTimestepFProp(self, data_format): - with self.session(use_gpu=True): - np.random.seed(1234567) - inputs = tf.constant([[[0.5, 0.3, -0.2, 0.0], [0.0, 0.7, -1.0, 2.0]], - [[0.5, 0.3, -0.2, 0.0], [0.5, 0.3, -0.2, 0.0]]], - dtype=tf.float32) - # tasks are again of shape [time, batch] but with different tasks - # for each timestep. - tasks = tf.constant([[1, 0], [2, 1]], dtype=tf.int32) - if data_format == 'BTC': - inputs = tf.transpose(inputs, [1, 0, 2]) - tasks = tf.transpose(tasks, [1, 0]) - p = self._MultitaskAdapterParams(data_format) - adapter = p.Instantiate() - output = adapter.FProp(adapter.theta, inputs, tasks) - self.evaluate(tf.global_variables_initializer()) - actual = self.evaluate(output) - if data_format == 'BTC': - actual = tf.transpose(actual, [1, 0, 2]) - tf.logging.info('testDifferentTaskPerTimestepFProp actual=%r' % actual) - expected = [[[1.1579462, 1.2241995, -0.6177901, 0.23089096], - [0.05022227, 0.9056754, -0.3771479, 2.1245508]], - [[0.6961179, 0.06690431, -0.08757646, 0.40129724], - [1.1579462, 1.2241995, -0.6177901, 0.23089096]]] - self.assertEqual(actual.shape, (2, 2, 4)) - self.assertAllClose(expected, actual, rtol=1e-05, atol=1e-05) - - def testGradientChecker(self): - with self.session(use_gpu=True): - np.random.seed(1234567) - inputs = tf.constant([[[0.5, 0.3, -0.2, 0.0], [0.0, 0.7, -1.0, 2.0]], - [[0.5, 0.3, -0.2, 0.0], [0.5, 0.3, -0.2, 0.0]]], - dtype=tf.float32) - tasks = tf.constant([1, 0], dtype=tf.int32) - p = self._MultitaskAdapterParams() - adapter = p.Instantiate() - output = adapter.FProp(adapter.theta, inputs, tasks) - loss = tf.reduce_sum(output) - all_vars = tf.trainable_variables() - grads = tf.gradients(loss, all_vars) - self.evaluate(tf.global_variables_initializer()) - - def DenseGrad(var, grad): - if isinstance(grad, tf.Tensor): - return grad - elif isinstance(grad, tf.IndexedSlices): - return tf.math.unsorted_segment_sum(grad.values, grad.indices, - tf.shape(var)[0]) - - dense_grads = [DenseGrad(x, y) for (x, y) in zip(all_vars, grads)] - dense_grad_sums = [tf.reduce_sum(g) for g in dense_grads] - grad_vs = self.evaluate(dense_grad_sums) - self.assertAllClose([ - -5.364418e-07, 3.405262e+00, 1.252710e+01, 1.600000e+01, 1.335246e+00, - 2.513876e-01 - ], - grad_vs, - rtol=1e-05, - atol=1e-05) - - -class CCTGatingNetworkTest(test_utils.TestCase): - - def testCCTGatingNetworkConstruction(self): - with self.session(use_gpu=False): - p = layers.CCTGatingNetwork.Params().Set( - name='cct_gating', - input_dim=10, - hidden_layer_dim=20, - num_outputs=3, - noise_std=5.0, - noise_warmup_steps=300) - cct_l = p.Instantiate() - a = tf.constant(1.0, shape=[20, 10]) - cct_l.FPropDefaultTheta(a) - - def testCCTGatingNetworkTraining(self): - with self.session(use_gpu=False): - tf.random.set_seed(398847392) - np.random.seed(12345) - p = layers.CCTGatingNetwork.Params().Set( - name='cct_gating', - input_dim=10, - hidden_layer_dim=20, - num_outputs=3, - noise_std=5.0, - noise_warmup_steps=300) - params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=837465638) - p.params_init = params_init - p.is_inference = False - cct_net = p.Instantiate() - - a = tf.constant(np.random.rand(3, 10), dtype=tf.float32) - out = cct_net.FPropDefaultTheta(a) - - self.evaluate(tf.global_variables_initializer()) - out_v = self.evaluate([out]) - self.assertAllClose( - out_v, - [[[0.412904, 0.520129, 0.694699], [0.395485, 0.47316, 0.632451], - [0.404144, 0.502593, 0.644338]]]) - - def testCCTGatingNetworkInference(self): - with self.session(use_gpu=False), self.SetEval(True): - tf.random.set_seed(398847392) - np.random.seed(12345) - p = layers.CCTGatingNetwork.Params().Set( - name='cct_gating', - input_dim=10, - hidden_layer_dim=20, - num_outputs=3, - noise_std=5.0, - noise_warmup_steps=300) - params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=837465638) - p.params_init = params_init - cct_net = p.Instantiate() - - a = tf.constant(np.random.rand(3, 10), dtype=tf.float32) - out = cct_net.FPropDefaultTheta(a) - - self.evaluate(tf.global_variables_initializer()) - out_v = self.evaluate([out]) - self.assertAllClose(out_v, [[[0., 1., 1.], [0., 0., 1.], [0., 1., 1.]]]) - - -class CondScaleShiftFFNLayerTest(test_utils.TestCase): - - def testCondScaleShiftFFNLayerConstruction(self): - with self.session(use_gpu=False): - params = layers.CondScaleShiftFFNLayer.Params().Set( - name='ss_ffn', - input_dim=10, - output_dim=7, - scale_fn='NONE', - shift_fn='NONE') - params.ffn.hidden_layer_dims = [5, 5] - layer_ss = params.Instantiate() - time_c, batch_c, in_dim_c = 15, 2, 10 - a = tf.constant(1.0, shape=[time_c, batch_c, in_dim_c]) - layer_ss.FPropDefaultTheta(a) - - def testCondScaleShiftFFNLayerFprop(self): - with self.session(use_gpu=False): - time_c, batch_c, in_dim_c, out_dim_c = 15, 2, 10, 7 - params = layers.CondScaleShiftFFNLayer.Params().Set( - name='ss_ffn', - input_dim=in_dim_c, - output_dim=out_dim_c, - scale_fn='NONE', - shift_fn='NONE') - params.ffn.hidden_layer_dims = [5, 5] - layer_ss = params.Instantiate() - - a = tf.constant(1.0, shape=[time_c, batch_c, in_dim_c]) - scale_out, shift_out = layer_ss.FPropDefaultTheta(a) - self.evaluate(tf.global_variables_initializer()) - scale_out, shift_out = self.evaluate([scale_out, shift_out]) - self.assertEqual(scale_out.shape, (time_c, batch_c, out_dim_c)) - self.assertEqual(shift_out.shape, (time_c, batch_c, out_dim_c)) - - -class IdentityLayerTest(test_utils.TestCase): - - def testIdentityLayerNestedMap(self): - with self.session(use_gpu=False): - p = layers.IdentityLayer.Params().Set(name='Nested') - layer = p.Instantiate() - a = tf.constant(1.0, shape=[20, 10]) - b = tf.constant(-2.0, shape=[20, 10]) - inputs = py_utils.NestedMap(a=a, b=b) - outputs = layer.FPropDefaultTheta(inputs) - self.assertAllEqual(inputs.a.eval(), outputs.a.eval()) - self.assertAllEqual(inputs.b.eval(), outputs.b.eval()) - a_copy = layer.FPropDefaultTheta(a) - self.assertAllEqual(a.eval(), a_copy.eval()) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/layers_with_attention.py b/lingvo/core/layers_with_attention.py deleted file mode 100644 index c6c54ea3..00000000 --- a/lingvo/core/layers_with_attention.py +++ /dev/null @@ -1,2205 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Lingvo layers that depend on attention layers but are not recurrent.""" - -import lingvo.compat as tf -from lingvo.core import attention -from lingvo.core import base_layer -from lingvo.core import layers -from lingvo.core import py_utils -from lingvo.core import symbolic - - -class TransformerAttentionLayer(base_layer.BaseLayer): - """Multi-headed attention, add and norm used by 'Attention Is All You Need'. - - This class implements the first sub-layer of Transformer Layer. Input is - first processed using a multi-headed (self) attention. Output of the - attention layer is combined with the residual connection. And the finally, - output is normalized using Layer Normalization. - - Layer can be used in five scenarios: - - 1. Multi-Headed Self-Attention, where attention keys (source vectors), - attention values (context vectors) and queries come from the same previous - layer output, `query_vec`. This is the general use case for encoder - Transformer Layers. - 2. Masked Multi-Headed Self-Attention, where attention keys, attention values - and queries all come from the same previous layer output, but rightward - activations are masked to prevent information flow from future. This is the - use case for decoder self-attention Transformer Layers. Can be activated by - setting `is_masked` flag of this layer. - 3. Multi-Headed Attention, where attention keys and attention values - `source_vecs`, are coming from a different source (output of the encoder) - and queries `query_vec`, coming from the previous layer outputs (decoder). - This corresponds to the standard attention mechanism, decoder attending the - encoder outputs. - 4. Multi-Headed Attention, where attention values `context_vecs` are coming - from a different source than queries and keys, e.g. for positional - attention, where keys and queries are positional encodings and values are - decoder states. - 5. Masked Multi-Headed Self-Attention, where attention keys, attention values - and queries all come from the same previous layer output, but the - activations for the current position are masked to reduce the impact of - high self-similarity. This is the use case for non-autoregressive decoder - self-attention Transformer Layers. Can be activated by setting `is_masked` - flag of this layer and setting `mask_type="eye"`. - 6. Masked Multi-Headed Self-Attention, where attention keys, attention values - and queries all come from the same previous layer output, but: - . rightward activations are masked to prevent information flow from future. - . leftward activations are also masked to prevent information flow from - past tokens that are beyond the N-gram context [K-N+1, K-1] when predicting - the target token in position K. This is the use case for decoder - self-attention Transformer Layers in N-gram mode. Can be activated by - setting `is_masked` flag of this layer, and setting both - `mask_type="ngram"` and `mask_ngram_order=N-1` to use as context only the - previous N-1 tokens (as expected for an N-gram model); for details and - experimental results see https://arxiv.org/abs/2001.04589. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('source_dim', 0, 'Dimension of the transformer block input.') - p.Define('context_dim', 0, 'Dimension of the attention contexts.') - p.Define('atten_hidden_dim', 0, 'Dimension of the attention hidden dim.') - p.Define('num_attention_heads', 8, 'Number of attention heads.') - p.Define('is_masked', False, 'If set, uses masked MultiHeadedAttention.') - p.Define( - 'mask_ngram_order', 0, 'N-gram order, relevant only when' - '`mask_type` is set to "ngram".') - p.Define( - 'mask_type', 'future', 'Type of attention mask if `is_masked` is' - 'set. Either "future" for masking out attention to future' - 'positions or "eye" for masking out the token itself, or "ngram" for' - 'bounding the left context to the previous N-1 tokens, where N is set' - 'by `mask_ngram_order`.') - p.Define('ln_tpl', layers.LayerNorm.Params(), 'Layer norm default params.') - p.Define( - 'atten_tpl', - attention.MultiHeadedAttention.Params().Set( - use_source_vec_as_attention_value=False, enable_ctx_post_proj=True), - 'Multi-Headed Dot-Attention default params.') - p.Define( - 'atten_dropout_prob', 0.0, - 'Probability at which we apply dropout to the attention probs. ' - 'This practically drops memory values at random positions.') - p.Define( - 'residual_dropout_prob', 0.0, - 'Probability at which we apply dropout to the residual layers, ' - 'such that, residual(x, f(x)) = (x + dropout(f(x))).') - p.Define( - 'residual_dropout_tpl', layers.DropoutLayer.Params(), - 'Residual dropout params template. keep_prop will be reset to ' - '(1.0 - residual_dropout_prob).') - p.Define('packed_input', False, - 'If True, each training example may pack multiple sequences.') - p.Define('add_unnormalized_input', False, 'If set, uses unnormalized input ' - 'in the residual add.') - p.Define( - 'residual_function', None, 'When None (the default), use simple ' - 'sum for the residual connection (output = x + f(x)). For example, can ' - 'use layers.HighwaySkipLayer.Params() or layers.GatingLayer.Params() ' - 'for gated residual add, where output is instead ' - 'residual_function.FProp(x, f(x)).') - p.Define( - 'pre_layer_norm', True, 'When True, layer norm is used before attention' - 'module, otherwise used after attention module which is consistent with' - 'Vaswani et al\'s paper') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.source_dim - - if not p.atten_hidden_dim: - p.atten_hidden_dim = p.source_dim - - if not p.context_dim: - p.context_dim = p.source_dim - - if p.is_masked: - assert p.mask_type in ['future', 'eye', 'ngram'] - - params = self._InitAttention(p.atten_tpl) - self.CreateChild('atten', params) - - # Initialize attention layer norm - params = p.ln_tpl.Copy() - params.name = 'atten_ln' - params.input_dim = p.source_dim - self.CreateChild('layer_norm', params) - - dropout_tpl = p.residual_dropout_tpl.Copy() - dropout_tpl.keep_prob = (1.0 - p.residual_dropout_prob) - self.CreateChild('residual_dropout', dropout_tpl) - - if p.residual_function is not None: - params = p.residual_function.Copy() - params.input_dim = p.atten_hidden_dim - self.CreateChild('residual_function', params) - - def _InitAttention(self, atten_tpl): - p = self.params - # Initialize multi-headed attention - params = atten_tpl.Copy() - params.name = 'multihead_atten' - params.source_dim = p.source_dim - params.query_dim = p.source_dim - params.hidden_dim = p.atten_hidden_dim - params.context_dim = p.context_dim - params.ctx_post_proj_dim = p.source_dim - params.num_attention_heads = p.num_attention_heads - params.atten_dropout_prob = p.atten_dropout_prob - params.packed_input = p.packed_input - return params - - def _GetSourceLength(self, source_paddings): - return py_utils.GetShape(source_paddings)[0] - - def FProp(self, - theta, - query_vec, - source_paddings, - source_vecs=None, - query_segment_id=None, - source_segment_id=None, - context_vecs=None, - **kwargs): - """Transformer attention, residual and normalization layer. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [target_time, target_batch, dim] - source_paddings: [source_time, source_batch] - source_vecs: [source_time, source_batch, dim]. - query_segment_id: [target_time, target_batch] - source_segment_id: [source_time, source_batch] - context_vecs: [source_time, target_batch, dim] - **kwargs: Can be optional params for the attention layer, eg. attention - projection index tensor. - - Returns: - (output, atten_probs). output is of shape [target_time, target_batch, - context_dim], atten_probs is of shape [target_time, target_batch, - source_time]. - """ - p = self.params - unnormalized_query_vec = query_vec - if p.pre_layer_norm: - query_vec = self.layer_norm.FProp(theta.layer_norm, query_vec) - - if source_vecs is None: # For self-attention: keys = queries. - source_vecs = query_vec - source_segment_id = query_segment_id - - if context_vecs is None: # Inter/self-attention: keys = values/contexts. - context_vecs = source_vecs - - target_time, target_bs, query_dim = py_utils.GetShape(query_vec, 3) - if p.is_masked: - assert source_vecs is not None - query_vec = py_utils.with_dependencies([ - py_utils.assert_shape_match( - tf.shape(source_vecs), tf.shape(query_vec)) - ], query_vec) - # Prepares mask for self-attention - # Padding is complemented, so time indexes that we want to mask out - # receive padding weight 1.0. - if p.mask_type == 'future': - padding = py_utils.CausalSelfAttenPadding( - target_time, dtype=py_utils.FPropDtype(p)) - elif p.mask_type == 'eye': - padding = tf.eye(target_time, target_time, dtype=py_utils.FPropDtype(p)) - elif p.mask_type == 'ngram': # Maybe apply N-gram mask. - assert p.mask_ngram_order - padding = 1.0 - tf.linalg.band_part( - tf.ones([target_time, target_time], dtype=py_utils.FPropDtype(p)), - tf.minimum(p.mask_ngram_order - 1, target_time - 1), 0) - - # [time, batch, time] - causal_padding = tf.tile(tf.expand_dims(padding, 1), [1, target_bs, 1]) - - causal_padding = tf.reshape(causal_padding, [-1, target_time]) - else: - causal_padding = None - - # Projects keys and values. - packed_src = self.atten.PackSource( - theta=theta.atten, - source_vecs=source_vecs, # keys - source_contexts=context_vecs, # values - source_padding=source_paddings, - source_segment_id=source_segment_id) - - if query_segment_id is not None: - query_segment_id = tf.reshape(query_segment_id, [-1]) - - ctx_vec, atten_prob, _ = self.atten.ComputeContextVectorWithSource( - theta=theta.atten, - packed_src=packed_src, - query_vec=tf.reshape(query_vec, [-1, query_dim]), - per_step_source_padding=causal_padding, - query_segment_id=query_segment_id, - **kwargs) - - ctx_vec = self.residual_dropout.FProp(theta.residual_dropout, ctx_vec) - input_to_add = ( - unnormalized_query_vec if p.add_unnormalized_input else query_vec) - input_after_sublayer = tf.reshape( - ctx_vec, - [ - target_time, - target_bs, - -1 # Either projected or not. - ]) - if p.residual_function is None: - h = input_to_add + input_after_sublayer - else: - h = self.residual_function.FProp(theta.residual_function, input_to_add, - input_after_sublayer) - if not p.pre_layer_norm: - h = self.layer_norm.FProp(theta.layer_norm, h) - atten_prob = tf.reshape( - atten_prob, - [target_time, target_bs, - self._GetSourceLength(source_paddings)]) - return h, atten_prob - - def _FinishExtendStep(self, - theta, - query_vec, - unnormalized_query_vec, - extended_packed_src, - t=None): - """Finish extending prefix by one more time step. - - Isolating this function from ExtendStep allows generalizing self-attention - to causal attention on other inputs. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [target_batch, dim] - unnormalized_query_vec: [target_batch, dim] - extended_packed_src: A `.NestedMap` object containing source_vecs, - source_contexts, source_paddings, and source_segment_ids - t: a scalar, the current time step, 0-based. - - Returns: - A triplet (cur_output, atten_prob, new_state) where cur_output is a tensor - representing the output from the current state, and new_state is the new - state `.NestedMap`. - """ - p = self.params - - # Compute per_step_source_padding. Padding is complemented, so time indexes - # that we want to mask out receive padding weight 1.0. - query_batch_size = py_utils.GetShape(query_vec)[0] - source_seq_len = py_utils.GetShape(extended_packed_src.source_vecs)[0] - zero_padding = tf.fill([source_seq_len], - tf.constant(0.0, dtype=query_vec.dtype)) - ones_padding = tf.ones_like(zero_padding, dtype=query_vec.dtype) - if t is not None: - per_step_source_padding = tf.where( - tf.less(tf.range(source_seq_len), tf.fill([source_seq_len], t + 1)), - zero_padding, ones_padding) - per_step_source_padding = tf.tile( - tf.expand_dims(per_step_source_padding, axis=0), - [query_batch_size, 1]) - # Maybe apply N-gram masking. - # TODO(ciprianchelba): As pointed out by miachen, to get the expected - # speed-up we should go with per_step_source_padding=None here, and - # everytime we update the prefix_states, we not only extend one step, but - # also only keep the prefix_states for the most recent N steps instead of - # the prefix states all the way from step 0. - elif p.is_masked and p.mask_type == 'ngram': - assert p.mask_ngram_order - idx = tf.maximum(0, source_seq_len - p.mask_ngram_order) - per_step_source_padding = tf.where( - tf.less(tf.range(source_seq_len), tf.fill([source_seq_len], idx)), - ones_padding, zero_padding) - per_step_source_padding = tf.tile( - tf.expand_dims(per_step_source_padding, axis=0), - [query_batch_size, 1]) - else: - per_step_source_padding = None - - ctx_vec, atten_prob, _ = self.atten.ComputeContextVectorWithCachedSource( - theta.atten, - extended_packed_src, - query_vec, - per_step_source_padding=per_step_source_padding) - - ctx_vec = self.residual_dropout.FProp(theta.residual_dropout, ctx_vec) - input_to_add = ( - unnormalized_query_vec if p.add_unnormalized_input else query_vec) - input_after_sublayer = tf.reshape(ctx_vec, py_utils.GetShape(query_vec)) - if p.residual_function is None: - h = input_to_add + input_after_sublayer - else: - h = self.residual_function.FProp(theta.residual_function, input_to_add, - input_after_sublayer) - - if not p.pre_layer_norm: - h = self.layer_norm.FProp(theta.layer_norm, h) - - new_states = py_utils.NestedMap( - key=extended_packed_src.source_vecs, - value=extended_packed_src.source_contexts) - return h, atten_prob, new_states - - def ExtendStep(self, theta, query_vec, prefix_state, t=None): - """Extend prefix by one more time step. - - This function is expected to be called during fast decoding of the - Transformer model. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [target_batch, dim] - prefix_state: dict, containing tensors which are the results of previous - attentions, used for fast decoding. - t: a scalar, the current time step, 0-based. - - Returns: - A triplet (cur_output, atten_prob, new_state) where cur_output is a tensor - representing the output from the current state, and new_state is the new - state `.NestedMap`. - """ - p = self.params - assert p.is_masked # Must be causal attention. - unnormalized_query_vec = query_vec - if p.pre_layer_norm: - query_vec = self.layer_norm.FProp(theta.layer_norm, query_vec) - - cached_packed_src = py_utils.NestedMap( - source_vecs=prefix_state.key, - source_contexts=prefix_state.value, - source_padding=None, - source_segment_id=None) - extended_packed_src = self.atten.ExtendSourcePacked(theta.atten, query_vec, - query_vec, None, None, - cached_packed_src, t) - return self._FinishExtendStep(theta, query_vec, unnormalized_query_vec, - extended_packed_src, t) - - -class TransformerMultiSourceAttentionLayer(TransformerAttentionLayer): - """Multi-source multi-headed attention. - - Only supports scenarios 3 and 4 in the base class. Now the two scenarios are: - - 3. Multi-source multi-Headed Attention, where attention keys and attention - values `source_vecs`, are different encodings and queries `query_vec`, - coming from the previous layer outputs (decoder). In addition, - attention keys and values are NestedMaps containing encodings of different - sources. This corresponds to a multi-source decoder-to-encoder attention - mechanism, i.e., decoder attends to encoder outputs and other sources. - 4. Similar to 3 but attention values `context_vecs` are coming from a - different source than queries and keys. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_source', 0, 'Number of sources to attend to.') - p.Define( - 'primary_source_index', 0, 'Index of the primary source whose ' - 'attention probs will be returned.') - p.Define('multi_source_atten', attention.MultiSourceAttention.Params(), - 'Multi-source attention params.') - # Only used for case 3 and 4. - p.is_masked = False - return p - - def _InitAttention(self, atten_tpl): - p = self.params - source_atten_tpls = [] - # Set up each source attention. - for i in range(p.num_source): - src_key = 'source_%d' % i - src_atten = atten_tpl.Copy() - src_atten = super()._InitAttention(src_atten) - src_atten.name = 'multihead_atten_%s' % src_key - source_atten_tpls.append((src_key, src_atten)) - - # Initialize multi-source attention. - msa = p.multi_source_atten.Copy() - msa.name = 'multi_source_atten' - msa.source_dim = p.source_dim - msa.query_dim = p.source_dim - msa.source_atten_tpls = source_atten_tpls - msa.primary_source_key = 'source_%d' % p.primary_source_index - return msa - - def _GetSourceLength(self, source_paddings): - return py_utils.GetShape( - source_paddings['source_%d' % self.params.primary_source_index])[0] - - -class TransformerFeedForwardLayer(base_layer.BaseLayer): - """Feed-forward, add and norm layer used by 'Attention Is All You Need'. - - This class implements the second sub-layer of Transformer Layer. First, - input passes through a feed-forward neural network with one hidden layer and - then projected back to the original input dimension to apply residual. Output - of the layer, is then normalized using Layer Normalization. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', 0, 'Dimension of the layer input.') - p.Define('output_dim', 0, 'Dimension of the layer output.') - p.Define('hidden_dim', 0, 'Dimension of the hidden layer.') - p.Define('ln_tpl', layers.LayerNorm.Params(), 'Layer norm default params') - p.Define('activation', 'RELU', 'Non-linearity.') - p.Define( - 'residual_weight', 1., 'Weight applied on residual connection.' - 'Final output is residual_weight * residual_fn(x) + x.' - 'Only effective when add_skip_connection is True.') - p.Define('fflayer_tpl', - layers.FeedForwardNet.Params().Set(activation=['RELU', 'NONE']), - 'Feed forward layer default params') - p.Define( - 'res_proj_tpl', - layers.ProjectionLayer.Params().Set(batch_norm=True), - 'Residual projection default params, used when input_dim != ' - 'output_dim.') - p.Define( - 'residual_dropout_prob', 0.0, - 'Probability at which we apply dropout to the residual layers, ' - 'such that, residual(x, y) = (x + dropout(y)).') - p.Define( - 'residual_dropout_tpl', layers.DropoutLayer.Params(), - 'Residual dropout params template. keep_prop will be reset to ' - '(1.0 - residual_dropout_prob).') - p.Define( - 'relu_dropout_prob', 0.0, - 'Probability at which we apply dropout to the hidden layer ' - 'of feed-forward network.') - p.Define('add_skip_connection', True, - 'If True, add skip_connection from input to output.') - p.Define('pre_layer_norm', True, 'Pre or post layer norm') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.input_dim - assert symbolic.ToStatic(p.hidden_dim) > 0 - - # Initialize feed-forward layer - params = p.fflayer_tpl.Copy() - params.name = 'fflayer' - params.input_dim = p.input_dim - params.activation = [p.activation, 'NONE'] - if p.output_dim == 0: - params.hidden_layer_dims = [p.hidden_dim, p.input_dim] - else: - params.hidden_layer_dims = [p.hidden_dim, p.output_dim] - - if p.output_dim != p.input_dim: - pj = p.res_proj_tpl.Copy() - pj.name = 'res_proj' - pj.input_dim = p.input_dim - pj.output_dim = p.output_dim - pj.activation = 'NONE' - self.CreateChild('res_proj_layer', pj) - - params.dropout = [ - params.dropout.cls.Params().Set(keep_prob=1.0 - p.relu_dropout_prob), - params.dropout.cls.Params().Set(keep_prob=1.0) - ] - self.CreateChild('fflayer', params) - - # Initialize feed-forward layer norm - params = p.ln_tpl.Copy() - params.name = 'fflayer_ln' - params.input_dim = p.input_dim - self.CreateChild('layer_norm', params) - - dropout_tpl = p.residual_dropout_tpl.Copy() - dropout_tpl.keep_prob = (1.0 - p.residual_dropout_prob) - self.CreateChild('residual_dropout', dropout_tpl) - - @property - def output_dim(self): - """Returns output dimension of the transformer layer.""" - return self.fflayer.output_dim - - @classmethod - def NumOutputNodes(cls, p): - return p.output_dim if p.output_dim else p.input_dim - - def FProp(self, theta, inputs, paddings): - """Feed-forward, residual and layer-norm. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: [time, batch, dim]. - paddings: [time, batch] - - Returns: - tensor of the same shape with inputs - """ - if self.params.pre_layer_norm: - inputs_normalized = self.layer_norm.FProp(theta.layer_norm, inputs) - else: - inputs_normalized = inputs - if hasattr(self, 'res_proj_layer'): - inputs = self.res_proj_layer.FProp(theta.res_proj_layer, inputs) - h = self.residual_dropout.FProp( - theta.residual_dropout, - self.fflayer.FProp(theta.fflayer, inputs_normalized, - tf.expand_dims(paddings, -1))) - if self.params.add_skip_connection: - h = inputs + h * self.params.residual_weight - if not self.params.pre_layer_norm: - h = self.layer_norm.FProp(theta.layer_norm, h) - return h - - -class TransformerLayer(base_layer.BaseLayer): - """Transformer Layer proposed by 'Attention Is All You Need'. - - Applies self-attention followed by a feed forward network and - layer normalization. Uses residual connections between each consecutive - layer. In particular, adds residuals from layer input and attention output - and from attention output (feed-forward input) to feed-forward output. - - Implements the transformer block in 'Attention is All You Need': - https://arxiv.org/abs/1706.03762. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('source_dim', 0, 'Dimension of the transformer block input.') - p.Define('output_dim', 0, 'Dimension of the transformer block output.') - p.Define('tr_atten_tpl', - TransformerAttentionLayer.Params().Set(num_attention_heads=8), - 'Transformer Attention Layer params.') - p.Define('tr_post_ln_tpl', None, - '(Optional) Layer norm at end of transformer layer.') - p.Define('tr_fflayer_tpl', - TransformerFeedForwardLayer.Params().Set(hidden_dim=2048), - 'Transformer Feed-Forward Layer params.') - p.Define( - 'has_aux_atten', False, - 'If set, introduces a second attention layer, which attends to' - ' the auxiliary source contexts.') - p.Define('tr_aux_atten_tpl', None, 'Transformer Attention Layer params.') - p.Define('mask_self_atten', False, 'If True, use masked self-attention.') - p.Define('packed_input', False, - 'If True, each training example may pack multiple sequences.') - p.Define( - 'is_decoder', False, '(Deprecated) ' - 'If true, forces both has_aux_atten and mask_self_atten to true.') - p.Define( - 'num_aux_atten_post_proj', 1, 'Number of post projections for aux ' - 'attention. This is usually used in multi-task setting, in which ' - 'each task uses one dedicated projection layer.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.source_dim - - if p.is_decoder: - tf.logging.warning('TransformerLayer.is_decoder is deprecated.') - p.has_aux_atten = True - p.mask_self_atten = True - - # Initialize multi-headed self-attention - params = p.tr_atten_tpl.Copy() - params.name = 'multihead_self_atten' - params.source_dim = p.source_dim - params.packed_input = p.packed_input - params.is_masked = p.mask_self_atten - self.CreateChild('self_atten', params) - - if p.has_aux_atten: - # Initialize masked-multi-headed attention - params = ( - p.tr_atten_tpl.Copy() - if p.tr_aux_atten_tpl is None else p.tr_aux_atten_tpl.Copy()) - params.name = 'multihead_atten' - params.source_dim = p.source_dim - params.packed_input = p.packed_input - if hasattr(params.atten_tpl, 'num_post_proj'): - params.atten_tpl.num_post_proj = p.num_aux_atten_post_proj - self.CreateChild('atten', params) - - # Initialize feed-forward layer - params = p.tr_fflayer_tpl.Copy() - params.name = 'tr_fflayer' - params.input_dim = p.source_dim - params.output_dim = p.output_dim - self.CreateChild('fflayer', params) - - # Initialize output layer norm - if p.tr_post_ln_tpl: - params = p.tr_post_ln_tpl.Copy() - params.name = 'tr_post_layer_norm' - params.input_dim = p.source_dim - self.CreateChild('layer_norm', params) - - @property - def output_dim(self): - """Returns output dimension of the transformer layer.""" - # output_dim is equal to p.source_dim when p.output_dim is zero. - return self.fflayer.output_dim - - @classmethod - def NumOutputNodes(cls, p): - return p.output_dim if p.output_dim else p.source_dim - - def FProp(self, - theta, - source_vecs, - source_paddings, - aux_vecs=None, - aux_paddings=None, - source_segment_id=None, - aux_segment_id=None, - **kwargs): - """Transformer Layer. - - Transformer layer has the naming scheme as follows: `source_vecs` and - `source_paddings` are all assumed to be coming from the activations of the - layer below. When `TransformerLayer` is used in the Encoder (default - behavior of this layer) `source_*` tensors correspond to the outputs of - previous encoder layer. Further, keys, values and queries are all - forked from `source_vecs`. When TransformerLayer is used in the Decoder - (has_aux_atten=True), `source_*` tensors correspond to the outputs of - previous decoder layer and used as the queries. - - For the cases when `TransformerLayer` is used in the decoder - (has_aux_atten=True) `aux_*` tensors have to be provided. Auxiliary inputs, - `aux_*` tensors, are then correspond to the top-most layer encoder outputs - and used by the second `TransformerAttentionLayer` as keys and values. - - Regardless of the encoder or decoder, queries are always assumed to be - coming from the activations of layer below, in particular `source_vecs`. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - source_vecs: [source_time, source_batch, dim]. - source_paddings: [source_time, source_batch] - aux_vecs: [aux_time, aux_batch, dim] - aux_paddings: [aux_time, aux_batch] - source_segment_id: [source_time, source_batch] - aux_segment_id: [aux_time, aux_batch] - **kwargs: Can be optional params for the attention layer, eg. attention - projection index tensor. - - Returns: - The attention context vector, [source_time, source_batch, dim]. - - The attention probability vector, [source_time, source_batch, source_time] - if has_aux_atten is False, otherwise [source_time, source_batch, - aux_time]. - """ - p = self.params - if p.packed_input: - assert source_segment_id is not None, ('Need to specify segment id for ' - 'packed input.') - - with tf.name_scope('self_atten'): - atten_vec, atten_prob = self.self_atten.FProp( - theta.self_atten, - source_vecs, - source_paddings, - query_segment_id=source_segment_id) - - if p.has_aux_atten: - assert aux_vecs is not None - assert aux_paddings is not None - with tf.name_scope('aux_atten'): - atten_vec, atten_prob = self.atten.FProp(theta.atten, atten_vec, - aux_paddings, aux_vecs, - source_segment_id, - aux_segment_id, **kwargs) - - with tf.name_scope('fflayer'): - h = self.fflayer.FProp(theta.fflayer, atten_vec, source_paddings) - if p.tr_post_ln_tpl: - with tf.name_scope('layer_norm'): - h = self.layer_norm.FProp(theta.layer_norm, h) - return h, atten_prob - - def ExtendStep(self, - theta, - source_vecs, - prefix_states, - aux_vecs=None, - aux_paddings=None, - t=None, - **kwargs): - """Transformer Layer, extend one step in decoding. - - This function is expected to be called during fast decoding of Transformer - models. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - source_vecs: [source_batch, dim]. - prefix_states: dict, containing tensors which are the results of previous - attentions, used for fast decoding. - aux_vecs: [aux_time, aux_batch, dim] - aux_paddings: [aux_time, aux_batch] - t: a scalar, the current time step, 0-based. - **kwargs: Can be optional params for the attention layer, eg. attention - projection index tensor. - - Returns: - The attention context vector, [target_batch, source_dim] - - The attention probability vector, [source_time, target_batch] - - Updated prefix states - """ - p = self.params - - if p.has_aux_atten: - assert aux_vecs is not None - assert aux_paddings is not None - - batch_size = py_utils.GetShape(source_vecs)[0] - - # First the self-attention layer. - atten_vec, atten_prob, new_states = self.self_atten.ExtendStep( - theta.self_atten, source_vecs, prefix_states, t) - - atten_vec = tf.expand_dims(atten_vec, axis=0) - # Next the source attention layer. - if p.has_aux_atten: - atten_vec, atten_prob = self.atten.FProp(theta.atten, atten_vec, - aux_paddings, aux_vecs, **kwargs) - - # Finally, the feedforward layer. - h = self.fflayer.FProp( - theta.fflayer, atten_vec, - tf.zeros([1, batch_size], dtype=py_utils.FPropDtype(p))) - if p.tr_post_ln_tpl: - h = self.layer_norm.FProp(theta.layer_norm, h) - h = tf.squeeze(h, 0) - return h, atten_prob, new_states - - -class EvolvedTransformerEncoderBranchedConvsLayer(base_layer.BaseLayer): - """Evolved Transformer encoder branched convolutions layer. - - This constructs the branched convolution portion of the Evolved Transformer - encoder described in https://arxiv.org/abs/1901.11117 . - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('ln_tpl', layers.LayerNorm.Params(), 'Layer norm default params') - p.Define('input_dim', 0, 'Dimension of the layer input.') - p.Define('activation', 'RELU', - 'Activation applied after the left and right branches.') - p.Define('dropout_tpl', layers.DropoutLayer.Params(), - 'Dropout applied to each layer output.') - p.Define('dense_tpl', layers.FCLayer.Params(), - 'Fully connected "dense" layer.') - p.Define('conv_tpl', layers.Conv2DLayer.Params(), - 'Standard convolution layer.') - p.Define('separable_conv_tpl', layers.SeparableConv2DLayer.Params(), - 'Separable convolution layer.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.input_dim - - # Initialize first layer norm. - params = p.ln_tpl.Copy() - params.name = 'first_layer_norm' - params.input_dim = p.input_dim - self.CreateChild('first_layer_norm', params) - - # Initialize second layer norm. - params = p.ln_tpl.Copy() - params.name = 'second_layer_norm' - params.input_dim = p.input_dim * 4 - self.CreateChild('second_layer_norm', params) - - # Initialize dense layer. - params = p.dense_tpl.Copy() - params.name = 'dense_layer' - params.input_dim = p.input_dim - params.activation = p.activation - params.output_dim = p.input_dim * 4 - self.CreateChild('dense_layer', params) - - # Initialize standard conv. - params = p.conv_tpl.Copy() - params.name = 'conv_layer' - params.bias = True - params.batch_norm = False - params.activation = p.activation - params.filter_stride = (1, 1) - params.filter_shape = (3, 1, p.input_dim, int(p.input_dim / 2)) - self.CreateChild('conv_layer', params) - - # Initialize separable conv. - params = p.separable_conv_tpl.Copy() - params.name = 'separable_conv_layer' - params.bias = True - params.batch_norm = False - params.activation = 'NONE' - params.filter_stride = (1, 1) - params.filter_shape = (9, 1, int(p.input_dim * 4), p.input_dim) - self.CreateChild('separable_conv_layer', params) - - # Initialize dropout. - dropout_tpl = p.dropout_tpl.Copy() - self.CreateChild('dropout', dropout_tpl) - - def FProp(self, theta, inputs, paddings): - inputs_normalized = self.first_layer_norm.FProp(theta.first_layer_norm, - inputs) - - left_branch = self.dense_layer.FProp(theta.dense_layer, inputs_normalized, - tf.expand_dims(paddings, -1)) - left_branch = self.dropout.FProp(theta.dropout, left_branch) - # Newly computed padding is discarded. - right_branch = self.conv_layer.FProp( - theta.conv_layer, tf.expand_dims(inputs_normalized, axis=2), - paddings)[0] - right_branch = tf.squeeze(right_branch, axis=2) - right_branch = self.dropout.FProp(theta.dropout, right_branch) - right_branch = tf.pad( - right_branch, - [[0, 0], [0, 0], - [0, tf.shape(left_branch)[-1] - tf.shape(right_branch)[-1]]], - constant_values=0) - - hidden_state = left_branch + right_branch - - hidden_state = self.second_layer_norm.FProp(theta.second_layer_norm, - hidden_state) - # Newly computed padding is discarded. - hidden_state = self.separable_conv_layer.FProp( - theta.separable_conv_layer, tf.expand_dims(hidden_state, axis=2), - paddings)[0] - hidden_state = tf.squeeze(hidden_state, axis=2) - hidden_state = tf.pad( - hidden_state, [[0, 0], [0, 0], - [0, tf.shape(inputs)[-1] - tf.shape(hidden_state)[-1]]], - constant_values=0) - hidden_state = self.dropout.FProp(theta.dropout, hidden_state) - hidden_state += inputs - - return hidden_state - - -class EvolvedTransformerDecoderBranchedConvsLayer(base_layer.BaseLayer): - """Evolved Transformer decoder branched convolutions layer. - - This constructs the branched convolution portion of the Evolved Transformer - decoder described in https://arxiv.org/abs/1901.11117 . - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('ln_tpl', layers.LayerNorm.Params(), 'Layer norm default params') - p.Define('input_dim', 0, 'Dimension of the layer input.') - p.Define('activation', 'RELU', - 'Activation applied to the left convolution branch output.') - p.Define('dropout_tpl', layers.DropoutLayer.Params(), - 'Dropout applied to each layer output.') - p.Define('separable_conv_tpl', - layers.SeparableConv2DLayer.Params().Set(causal_convolution=True), - 'Separable convolution layer.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.input_dim - - # Initialize first layer norm. - params = p.ln_tpl.Copy() - params.name = 'first_layer_norm' - params.input_dim = p.input_dim - self.CreateChild('first_layer_norm', params) - - # Initialize second layer norm. - params = p.ln_tpl.Copy() - params.name = 'second_layer_norm' - params.input_dim = p.input_dim * 2 - self.CreateChild('second_layer_norm', params) - - # Initialize separable conv. - params = p.separable_conv_tpl.Copy() - params.name = 'separable_conv_11x1_layer' - params.bias = True - params.batch_norm = False - params.activation = p.activation - params.filter_stride = (1, 1) - params.filter_shape = (11, 1, p.input_dim, int(p.input_dim * 2)) - self.CreateChild('separable_conv_11x1_layer', params) - - # Initialize first separable conv. - params = p.separable_conv_tpl.Copy() - params.name = 'separable_conv_7x1_layer' - params.bias = True - params.batch_norm = False - params.activation = 'NONE' - params.filter_stride = (1, 1) - params.filter_shape = (7, 1, p.input_dim, int(p.input_dim / 2)) - self.CreateChild('separable_conv_7x1_layer', params) - - # Initialize second separable conv. - params = p.separable_conv_tpl.Copy() - params.name = 'separable_conv_7x1_layer_2' - params.bias = True - params.batch_norm = False - params.activation = 'NONE' - params.filter_stride = (1, 1) - params.filter_shape = (7, 1, int(p.input_dim * 2), p.input_dim) - self.CreateChild('separable_conv_7x1_layer_2', params) - - # Initialize dropout. - dropout_tpl = p.dropout_tpl.Copy() - self.CreateChild('dropout', dropout_tpl) - - def FProp(self, theta, inputs, paddings): - inputs_normalized = self.first_layer_norm.FProp(theta.first_layer_norm, - inputs) - - left_branch = self.separable_conv_11x1_layer.FProp( - theta.separable_conv_11x1_layer, - tf.expand_dims(inputs_normalized, axis=2), paddings)[0] - left_branch = self.dropout.FProp(theta.dropout, left_branch) - - right_branch = self.separable_conv_7x1_layer.FProp( - theta.separable_conv_7x1_layer, tf.expand_dims( - inputs_normalized, axis=2), paddings)[0] - right_branch = self.dropout.FProp(theta.dropout, right_branch) - right_branch = tf.pad( - right_branch, - [[0, 0], [0, 0], [0, 0], - [0, tf.shape(left_branch)[-1] - tf.shape(right_branch)[-1]]], - constant_values=0) - - hidden_state = left_branch + right_branch - hidden_state = self.second_layer_norm.FProp(theta.second_layer_norm, - hidden_state) - - hidden_state = self.separable_conv_7x1_layer_2.FProp( - theta.separable_conv_7x1_layer_2, hidden_state, paddings)[0] - hidden_state = self.dropout.FProp(theta.dropout, hidden_state) - - hidden_state = tf.squeeze(hidden_state, axis=2) - return hidden_state + inputs - - -class EvolvedTransformerBaseLayer(base_layer.BaseLayer): - """Base layer for the Evolved Transformer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('source_dim', 0, 'Dimension of the transformer block input.') - p.Define( - 'has_aux_atten', False, - 'If set, introduces a second attention layer, which attends to' - ' the auxiliary source contexts.') - p.Define('packed_input', False, - 'If True, each training example may pack multiple sequences.') - return p - - -class EvolvedTransformerEncoderLayer(EvolvedTransformerBaseLayer): - """Evolved Transformer encoder layer. - - An Evolved Transformer encoder layer as described in - https://arxiv.org/abs/1901.11117 . - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('glu_tpl', layers.GluLayer.Params(), 'Glu layer.') - p.Define('branched_convs_tpl', - EvolvedTransformerEncoderBranchedConvsLayer.Params(), - 'Evolved Transformer branched convolutional layers.') - p.Define('transformer_tpl', TransformerLayer.Params(), 'Transformer layer.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.source_dim - # Auxiliary attention not supported. - if p.has_aux_atten: - raise ValueError('Auxiliary attention not supported.') - - # Initialize Glu layer. - params = p.glu_tpl.Copy() - params.name = 'glu_layer' - params.input_dim = p.source_dim - self.CreateChild('glu_layer', params) - - # Initialize branched convolutions layer. - params = p.branched_convs_tpl.Copy() - params.name = 'branched_convs_layer' - params.input_dim = p.source_dim - self.CreateChild('branched_convs_layer', params) - - # Initialize branched convolutional layers. - params = p.transformer_tpl.Copy() - params.name = 'transformer_layer' - params.source_dim = p.source_dim - params.output_dim = p.source_dim - params.tr_fflayer_tpl.hidden_dim = 4 * p.source_dim - # Decoder functionality is not supported so disable auxiliary attention. - params.has_aux_atten = False - params.tr_aux_atten_tpl = None - params.mask_self_atten = False - params.is_decoder = False - params.packed_input = p.packed_input - self.CreateChild('transformer_layer', params) - - def FProp(self, - theta, - source_vecs, - source_paddings, - aux_vecs=None, - aux_paddings=None, - source_segment_id=None, - aux_segment_id=None): - hidden_state = self.glu_layer.FProp(theta.glu_layer, source_vecs, - source_paddings) - - hidden_state = tf.transpose(hidden_state, [1, 0, 2]) - source_paddings = tf.transpose(source_paddings, [1, 0]) - hidden_state = self.branched_convs_layer.FProp( - theta.branched_convs_layer, hidden_state, source_paddings) - hidden_state = tf.transpose(hidden_state, [1, 0, 2]) - source_paddings = tf.transpose(source_paddings, [1, 0]) - - hidden_state, atten_prob = self.transformer_layer.FProp( - theta.transformer_layer, hidden_state, source_paddings, aux_vecs, - aux_paddings, source_segment_id, aux_segment_id) - - return hidden_state, atten_prob - - -class EvolvedTransformerDecoderLayer(EvolvedTransformerBaseLayer): - """Evolved Transformer decoder layer. - - An Evolved Transformer decoder layer as described in - https://arxiv.org/abs/1901.11117 . - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('tr_atten_tpl', - TransformerAttentionLayer.Params().Set(num_attention_heads=8), - 'Transformer attention layer params.') - p.Define('tr_double_heads_atten_tpl', - TransformerAttentionLayer.Params().Set(num_attention_heads=16), - 'Transformer double heads attention layer params.') - p.Define('branched_convs_tpl', - EvolvedTransformerDecoderBranchedConvsLayer.Params(), - 'Evolved Transformer branched convolutional layers.') - p.Define('transformer_tpl', TransformerLayer.Params(), 'Transformer layer.') - p.Define('tr_aux_atten_tpl', None, 'Transformer Attention Layer params.') - p.Define('mask_self_atten', False, 'If True, use masked self-attention.') - p.has_aux_atten = True - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.source_dim - - # Initialize multi-headed self-attention. - params = p.tr_double_heads_atten_tpl.Copy() - params.name = 'self_atten_double_heads' - params.source_dim = p.source_dim - params.is_masked = p.mask_self_atten - # Packed input is not supported. - params.packed_input = p.packed_input - self.CreateChild('self_atten_double_heads', params) - - if p.has_aux_atten: - # Initialize masked-multi-headed encoder attention. - params = ( - p.tr_aux_atten_tpl.Copy() - if p.tr_aux_atten_tpl is not None else p.tr_atten_tpl.Copy()) - params.name = 'attend_to_encoder' - params.source_dim = p.source_dim - # Packed input is not supported. - params.packed_input = p.packed_input - self.CreateChild('attend_to_encoder', params) - - # Initialize branched convolutional layers. - params = p.branched_convs_tpl.Copy() - params.name = 'branched_convs' - params.input_dim = p.source_dim - self.CreateChild('branched_convs', params) - - # Initialize transformer layer. - params = p.transformer_tpl.Copy() - params.name = 'transformer_layer' - params.source_dim = p.source_dim - params.output_dim = p.source_dim - params.tr_fflayer_tpl.hidden_dim = 4 * p.source_dim - params.tr_aux_atten_tpl = p.tr_aux_atten_tpl - params.has_aux_atten = p.has_aux_atten - params.mask_self_atten = p.mask_self_atten - params.tr_fflayer_tpl.activation = 'SWISH' - # Packed input is not supported. - params.packed_input = p.packed_input - self.CreateChild('transformer_layer', params) - - def FProp(self, - theta, - source_vecs, - source_paddings, - aux_vecs=None, - aux_paddings=None, - source_segment_id=None, - aux_segment_id=None): - p = self.params - - if p.has_aux_atten: - assert aux_vecs is not None - assert aux_paddings is not None - - with tf.name_scope('self_atten_double_heads'): - left_branch, _ = self.self_atten_double_heads.FProp( - theta.self_atten_double_heads, - source_vecs, - source_paddings, - query_segment_id=source_segment_id) - - if p.has_aux_atten: - with tf.name_scope('attend_to_encoder'): - right_branch, _ = self.attend_to_encoder.FProp( - theta.attend_to_encoder, source_vecs, aux_paddings, aux_vecs, - source_segment_id, aux_segment_id) - - hidden_state = left_branch + right_branch + source_vecs - else: - hidden_state = left_branch + source_vecs - - hidden_state = tf.transpose(hidden_state, [1, 0, 2]) - source_paddings = tf.transpose(source_paddings, [1, 0]) - hidden_state = self.branched_convs.FProp(theta.branched_convs, hidden_state, - source_paddings) - hidden_state = tf.transpose(hidden_state, [1, 0, 2]) - source_paddings = tf.transpose(source_paddings, [1, 0]) - - hidden_state, atten_prob = self.transformer_layer.FProp( - theta.transformer_layer, hidden_state, source_paddings, aux_vecs, - aux_paddings, source_segment_id, aux_segment_id) - - return hidden_state, atten_prob - - def ExtendStep(self, - theta, - source_vecs, - prefix_states, - aux_vecs=None, - aux_paddings=None, - t=None): - """Evolved Transformer decoder layer, extended one step in decoding. - - This function is expected to be called during fast decoding of Evolved - Transformer models. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - source_vecs: [source_batch, dim]. - prefix_states: dict, containing tensors which are the results of previous - attentions, used for fast decoding. - aux_vecs: [aux_time, aux_batch, dim] - aux_paddings: [aux_time, aux_batch] - t: a scalar, the current time step, 0-based. - - Returns: - The attention context vector, [target_batch, source_dim]. - - The attention probability vector, [source_time, target_batch]. - - Updated prefix states. - """ - p = self.params - - if p.has_aux_atten: - assert aux_vecs is not None - assert aux_paddings is not None - - inputs = tf.expand_dims(source_vecs, axis=0) - new_states = prefix_states - - double_head_attention_states = prefix_states.double_head_attention_states - # First the self-attention layer. - (left_branch, _, - double_head_attention_states) = self.self_atten_double_heads.ExtendStep( - theta.self_atten_double_heads, source_vecs, - double_head_attention_states, t) - new_states.double_head_attention_states = double_head_attention_states - left_branch = tf.expand_dims(left_branch, axis=0) - - hidden_state = left_branch + inputs - - # Next the source attention layer. - if p.has_aux_atten: - hidden_state += self.attend_to_encoder.FProp( - theta.attend_to_encoder, inputs, aux_paddings, aux_vecs)[0] - - branched_convs_input = prefix_states.branched_convs_input - branched_convs_input = tf.concat([branched_convs_input, hidden_state], - axis=0) - new_states.branched_convs_input = branched_convs_input - # The receptive field of the branched convs is 17 and so we do not need - # to consider inputs that come before that to compute the final position. - # TODO(davidso): Create an ExtendStep method for branched_convs to make this - # more efficient. - inputs_length = tf.minimum(tf.shape(branched_convs_input)[0], 17) - branched_convs_input = branched_convs_input[-inputs_length:, :, :] - branched_convs_input = tf.transpose(branched_convs_input, [1, 0, 2]) - hidden_state = self.branched_convs.FProp(theta.branched_convs, - branched_convs_input, None) - hidden_state = tf.transpose(hidden_state, [1, 0, 2]) - - transformer_layer_input = tf.squeeze(hidden_state[-1, :, :]) - transformer_layer_states = prefix_states.transformer_layer_states - (hidden_state, atten_prob, - transformer_layer_states) = self.transformer_layer.ExtendStep( - theta.transformer_layer, - transformer_layer_input, - transformer_layer_states, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings, - t=t) - - new_states.transformer_layer_states = transformer_layer_states - - return hidden_state, atten_prob, new_states - - -class StyleLayer(base_layer.BaseLayer): - """A layer that performs weighted style emb lookup.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', 0, 'Dimension of the input.') - p.Define('output_dim', 0, 'Dimension of the output.') - p.Define('num_styles', 0, 'Num of styles.') - p.Define('num_heads', 4, 'Number of attention heads.') - p.Define( - 'enable_ctx_post_proj', True, - 'If True, computed context is post projected into' - ' ctx_post_proj_dim.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.num_styles > 0 - assert p.input_dim > 0 - assert p.output_dim > 0 - - atten_p = attention.MultiHeadedAttention.Params().Set( - source_dim=p.output_dim, - context_dim=p.output_dim, - hidden_dim=p.output_dim, - query_dim=p.input_dim, - ctx_post_proj_dim=p.output_dim, - num_attention_heads=p.num_heads, - use_source_vec_as_attention_value=False, - enable_ctx_post_proj=p.enable_ctx_post_proj) - self.CreateChild('atten', atten_p) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - # The styles table. - w_shape = [p.num_styles, 1, p.output_dim] - w_init = py_utils.WeightInit.Gaussian(scale=1.0, seed=p.random_seed) - w_pc = py_utils.WeightParams( - shape=w_shape, - init=w_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('styles_w', w_pc) - - def EmbLookup(self, theta, ids): - """Looks up style embedding vectors for ids only for test purpose. - - Args: - theta: Named tuple with the weight matrix for the embedding. - ids: A rank-N int32 tensor. - - Returns: - embs, A rank-(N+1) params.dtype tensor. - embs[indices, :] is the embedding vector for ids[indices]. - """ - p = self.params - # TODO(ngyuzh): call this function for virsualize big discrete table, - # e.g. num_styles > 2^10. - embs = tf.nn.embedding_lookup(theta.styles_w, tf.reshape(ids, [-1])) - out_shape = tf.concat([tf.shape(ids), [p.output_dim]], 0) - return tf.reshape(tf.nn.tanh(embs), out_shape) - - def StyleEmbFromProbs(self, theta, inp): - """Look up style embedding based on feedin probabilities. - - Args: - theta: params for this layer and its sub-layers. - inp: attention probabilities of shape [batch_size, num_styles]. - - Returns: - style_emb - weighted combined style embedding based on inp. - """ - p = self.params - b_size = tf.shape(inp)[0] - styles_w = tf.tile(tf.nn.tanh(theta.styles_w), [1, b_size, 1]) - styles_paddings = tf.zeros([p.num_styles, b_size], - dtype=py_utils.FPropDtype(p)) - atten_probs = tf.tile(tf.expand_dims(inp, 1), [1, p.num_heads, 1]) - atten_probs = tf.reshape(atten_probs, [-1, p.num_styles]) - packed_src = self.atten.InitForSourcePacked(theta.atten, styles_w, styles_w, - styles_paddings) - style_emb, _ = self.atten.ComputeContextVectorWithAttenProbs( - theta.atten, packed_src.source_contexts, atten_probs) - return style_emb - - def FProp(self, theta, inp): - """Look up style embedding.""" - - p = self.params - b_size = tf.shape(inp)[0] - styles_w = tf.tile(tf.nn.tanh(theta.styles_w), [1, b_size, 1]) - styles_paddings = tf.zeros([p.num_styles, b_size], - dtype=py_utils.FPropDtype(p)) - packed_src = self.atten.InitForSourcePacked(theta.atten, styles_w, styles_w, - styles_paddings) - style_emb, probs, _ = self.atten.ComputeContextVectorWithSource( - theta.atten, packed_src, inp) - # TODO(yonghui): Extract and return the attention probabilities. - return style_emb, probs - - -class TransformerLayerWithMultitaskAdapters(TransformerLayer): - """Transformer Layer with multitask residual adapters. - - Applies transformer layer, followed by multitask adapters. Requires an - additional input specifying the task_id for each input. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('adapter_tpl', layers.MultitaskAdapterLayer.Params(), - 'Template to use for multitask adapters.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - params = p.adapter_tpl.Copy() - params.name = 'adapters' - self.CreateChild('adapters', params) - - def FProp(self, - theta, - source_vecs, - source_paddings, - aux_vecs=None, - aux_paddings=None, - source_segment_id=None, - aux_segment_id=None, - source_task_id=None): - """Transformer Layer with multitask adapters. - - First applies the standard transformer layer. Then applies adapter layers. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - source_vecs: [source_time, source_batch, dim]. - source_paddings: [source_time, source_batch] - aux_vecs: [aux_time, aux_batch, dim] - aux_paddings: [aux_time, aux_batch] - source_segment_id: [source_time, source_batch] - aux_segment_id: [aux_time, aux_batch] - source_task_id: [source_time, source_batch] - - Returns: - The attention context vector, [source_time, source_batch, dim]. - - The attention probability vector, [source_time, source_batch, source_time] - if has_aux_atten is False, otherwise [source_time, source_batch, - aux_time]. - """ - p = self.params - hidden, atten_prob = super().FProp(theta, source_vecs, source_paddings, - aux_vecs, aux_paddings, - source_segment_id, aux_segment_id) - # Assumes the same task_id for the entire sequence during eval or when - # not using packed_input. - if not p.packed_input and not self.do_eval: - source_task_id = source_task_id[0, :] - hidden = self.adapters.FProp(theta.adapters, hidden, source_task_id) - return hidden, atten_prob - - def ExtendStep(self, - theta, - source_vecs, - prefix_states, - aux_vecs=None, - aux_paddings=None, - timestep=None, - source_task_id=None): - """Transformer Layer with adapters, extend one step in decoding. - - Applies TransformerLayer.ExtendStep, then applies adapters. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - source_vecs: [source_batch, dim]. - prefix_states: dict, containing tensors which are the results of previous - attentions, used for fast decoding. - aux_vecs: [aux_time, aux_batch, dim] - aux_paddings: [aux_time, aux_batch] - timestep: a scalar, the current time step, 0-based. - source_task_id: [source_batch] - - Returns: - The attention context vector, [target_batch, source_dim] - - The attention probability vector, [source_time, target_batch] - - Updated prefix states - """ - p = self.params - - if p.has_aux_atten: - assert aux_vecs is not None - assert aux_paddings is not None - - batch_size = tf.shape(source_vecs)[0] - - # First the self-attention layer. - atten_vec, atten_prob, new_states = self.self_atten.ExtendStep( - theta.self_atten, source_vecs, prefix_states, timestep) - - atten_vec = tf.expand_dims(atten_vec, axis=0) - # Next the source attention layer. - if p.has_aux_atten: - atten_vec, atten_prob = self.atten.FProp(theta.atten, atten_vec, - aux_paddings, aux_vecs) - - # Finally, the feedforward layer. - hidden = self.fflayer.FProp( - theta.fflayer, atten_vec, - tf.zeros([1, batch_size], dtype=py_utils.FPropDtype(p))) - - # Now adapter layers. - hidden = self.adapters.FProp(theta.adapters, hidden, source_task_id) - - hidden = tf.squeeze(hidden, 0) - return hidden, atten_prob, new_states - - -# TODO(ankurbpn): Implementation is slightly different from the original. -# In the original implementation the KV projection outputs were explicitly -# zeroed out by the gating networks. Here we control the inputs instead. -# Verify if this still works as well as the original implementation. -class CCTAttentionLayer(base_layer.BaseLayer): - """Multi-headed attention, add and norm used by 'Attention Is All You Need'. - - Supports CCT attention gating as in the paper here: - https://arxiv.org/abs/2002.07106 - """ - - @classmethod - def Params(cls): - p = super().Params() - - # Transformer Attention params. - p.Define('source_dim', 0, 'Dimension of the transformer block input.') - p.Define('context_dim', 0, 'Dimension of the attention contexts.') - p.Define('atten_hidden_dim', 0, 'Dimension of the attention hidden dim.') - p.Define('num_attention_heads', 8, 'Number of attention heads.') - p.Define('is_masked', False, 'If set, uses masked MultiHeadedAttention.') - p.Define( - 'mask_type', 'future', 'Type of attention mask if `is_masked` is' - 'set. Either "future" for masking out attention to future' - 'positions or "eye" for masking out the token itself.') - p.Define('ln_tpl', layers.LayerNorm.Params(), 'Layer norm default params') - p.Define( - 'atten_tpl', - attention.MultiHeadedAttention.Params().Set( - use_source_vec_as_attention_value=False, enable_ctx_post_proj=True), - 'Multi-Headed Dot-Attention default params') - p.Define( - 'atten_dropout_prob', 0.0, - 'Probability at which we apply dropout to the attention probs. ' - 'This practically drops memory values at random positions.') - p.Define( - 'residual_dropout_prob', 0.0, - 'Probability at which we apply dropout to the residual layers, ' - 'such that, residual(x, y) = (x + dropout(y)).') - p.Define( - 'residual_dropout_tpl', layers.DropoutLayer.Params(), - 'Residual dropout params template. keep_prop will be reset to ' - '(1.0 - residual_dropout_prob).') - p.Define('packed_input', False, - 'If True, each training example may pack multiple sequences.') - p.Define('add_unnormalized_input', False, 'If set, uses unnormalized input ' - 'in the residual add.') - - # CCT params. - p.Define('gating_tpl', layers.CCTGatingNetwork.Params(), '') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.source_dim - - if not p.atten_hidden_dim: - p.atten_hidden_dim = p.source_dim - - if not p.context_dim: - p.context_dim = p.source_dim - - if p.is_masked: - assert p.mask_type in ['future', 'eye'] - - # Initialize multi-headed attention - params = p.atten_tpl.Copy() - params.name = 'multihead_atten' - params.source_dim = p.source_dim - params.query_dim = p.source_dim - params.hidden_dim = p.atten_hidden_dim - params.context_dim = p.context_dim - params.ctx_post_proj_dim = p.source_dim - params.num_attention_heads = p.num_attention_heads - params.atten_dropout_prob = p.atten_dropout_prob - params.packed_input = p.packed_input - self.CreateChild('atten', params) - - dropout_tpl = p.residual_dropout_tpl.Copy() - dropout_tpl.keep_prob = (1.0 - p.residual_dropout_prob) - self.CreateChild('residual_dropout', dropout_tpl) - - # Initialize attention layer norm - params = p.ln_tpl.Copy() - params.name = 'atten_ln' - params.input_dim = p.source_dim - self.CreateChild('layer_norm', params) - - # CCT specific operations. - ff_gating = p.gating_tpl.Copy() - ff_gating.input_dim = p.source_dim - ff_gating.num_outputs = 1 - ff_gating.name = 'query_gating_net' - self.CreateChild('query_gating', ff_gating) - - ff_gating = p.gating_tpl.Copy() - ff_gating.input_dim = p.source_dim - ff_gating.num_outputs = 1 - ff_gating.name = 'kv_gating_net' - self.CreateChild('kv_gating', ff_gating) - - # Initialize source_vec layer norm - params = p.ln_tpl.Copy() - params.name = 'source_ln' - params.input_dim = p.source_dim - self.CreateChild('source_layer_norm', params) - - # Initialize ctx_vec layer norm - params = p.ln_tpl.Copy() - params.name = 'ctx_ln' - params.input_dim = p.source_dim - self.CreateChild('ctx_layer_norm', params) - - def FProp(self, - theta, - query_vec, - source_paddings, - source_vecs=None, - query_segment_id=None, - source_segment_id=None, - **kwargs): - """CCT attention, residual and normalization layer. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [target_time, target_batch, dim] - source_paddings: [source_time, source_batch] - source_vecs: [source_time, source_batch, dim]. - query_segment_id: [target_time, target_batch] - source_segment_id: [source_time, source_batch] - **kwargs: Can be optional params for the attention layer, eg. attention - projection index tensor. - - Returns: - (output, atten_probs). output is of shape [target_time, target_batch, - context_dim], atten_probs is of shape [target_time, target_batch, - source_time]. - """ - p = self.params - unnormalized_query_vec = query_vec - query_vec = self.layer_norm.FProp(theta.layer_norm, query_vec) - - if source_vecs is None: # For self-attention: keys = queries. - source_vecs = query_vec - source_segment_id = query_segment_id - else: - source_vecs = self.source_layer_norm.FProp(theta.source_layer_norm, - source_vecs) - - # Gating the query computation. - query_p_c = self.query_gating.FProp(theta.query_gating, query_vec) - source_p_c = self.kv_gating.FProp(theta.kv_gating, source_vecs) - source_vecs *= source_p_c # Gate the source vectors. - - if p.is_masked: - assert source_vecs is not None - query_vec = py_utils.with_dependencies([ - py_utils.assert_shape_match( - tf.shape(source_vecs), tf.shape(query_vec)) - ], query_vec) - # Prepares mask for self-attention - # [time, time] - target_time = tf.shape(query_vec)[0] - target_bs = tf.shape(query_vec)[1] - - if p.mask_type == 'future': - padding = 1.0 - tf.linalg.band_part( - tf.ones([target_time, target_time], dtype=py_utils.FPropDtype(p)), - -1, 0) - elif p.mask_type == 'eye': - padding = tf.eye(target_time, target_time, dtype=py_utils.FPropDtype(p)) - - # [time, batch, time] - causal_padding = tf.tile(tf.expand_dims(padding, 1), [1, target_bs, 1]) - - causal_padding = tf.reshape(causal_padding, [-1, target_time]) - else: - causal_padding = None - - query_dim = tf.shape(query_vec)[-1] - - # Projects keys and values. - packed_src = self.atten.PackSource( - theta=theta.atten, - source_vecs=source_vecs, # keys - source_contexts=source_vecs, # values - source_padding=source_paddings, - source_segment_id=source_segment_id) - - if query_segment_id is not None: - query_segment_id = tf.reshape(query_segment_id, [-1]) - - ctx_vec, atten_prob, _ = self.atten.ComputeContextVectorWithSource( - theta=theta.atten, - packed_src=packed_src, - query_vec=tf.reshape(query_vec, [-1, query_dim]), - per_step_source_padding=causal_padding, - query_segment_id=query_segment_id, - **kwargs) - - # Gating operations - ctx_vec = query_p_c * tf.reshape( - self.ctx_layer_norm.FProp(theta.ctx_layer_norm, ctx_vec), - tf.shape(query_vec)) - - ctx_vec = self.residual_dropout.FProp(theta.residual_dropout, ctx_vec) - input_to_add = ( - unnormalized_query_vec if p.add_unnormalized_input else query_vec) - h = input_to_add + ctx_vec - atten_prob = tf.reshape(atten_prob, [ - tf.shape(query_vec)[0], - tf.shape(query_vec)[1], - tf.shape(source_vecs)[0] - ]) - return h, atten_prob, query_p_c, source_p_c - - def _FinishExtendStep(self, - theta, - query_vec, - unnormalized_query_vec, - extended_packed_src, - t=None): - """Finish extending prefix by one more time step. - - Isolating this function from ExtendStep allows generalizing self-attention - to causal attention on other inputs. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [target_batch, dim] - unnormalized_query_vec: [target_batch, dim] - extended_packed_src: A `.NestedMap` object containing source_vecs, - source_contexts, source_paddings, and source_segment_ids - t: a scalar, the current time step, 0-based. - - Returns: - A triplet (cur_output, atten_prob, new_state) where cur_output is a tensor - representing the output from the current state, and new_state is the new - state `.NestedMap`. - """ - p = self.params - # Gating operations - query_p_c = self.query_gating.FProp(theta.query_gating, query_vec) - - if t is not None: - source_seq_len = tf.shape(extended_packed_src.source_vecs)[0] - zero_padding = tf.fill([source_seq_len], - tf.constant(0.0, dtype=query_vec.dtype)) - per_step_source_padding = tf.where( - tf.less(tf.range(source_seq_len), tf.fill([source_seq_len], t + 1)), - zero_padding, tf.ones_like(zero_padding, dtype=query_vec.dtype)) - query_batch_size = tf.shape(query_vec)[0] - per_step_source_padding = tf.tile( - tf.expand_dims(per_step_source_padding, axis=0), - [query_batch_size, 1]) - else: - per_step_source_padding = None - ctx_vec, atten_prob, _ = self.atten.ComputeContextVectorWithCachedSource( - theta.atten, - extended_packed_src, - query_vec, - per_step_source_padding=per_step_source_padding) - - # Gating operations - ctx_vec = self.ctx_layer_norm.FProp(theta.ctx_layer_norm, ctx_vec) - ctx_vec = query_p_c * tf.reshape(ctx_vec, tf.shape(query_vec)) - ctx_vec = self.residual_dropout.FProp(theta.residual_dropout, ctx_vec) - input_to_add = ( - unnormalized_query_vec if p.add_unnormalized_input else query_vec) - h = input_to_add + ctx_vec - - new_states = py_utils.NestedMap( - key=extended_packed_src.source_vecs, - value=extended_packed_src.source_contexts) - return h, atten_prob, new_states - - def ExtendStep(self, theta, query_vec, prefix_state, t=None): - """Extend prefix by one more time step. - - This function is expected to be called during fast decoding of the - Transformer model. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - query_vec: [target_batch, dim] - prefix_state: dict, containing tensors which are the results of previous - attentions, used for fast decoding. - t: a scalar, the current time step, 0-based. - - Returns: - A triplet (cur_output, atten_prob, new_state) where cur_output is a tensor - representing the output from the current state, and new_state is the new - state `.NestedMap`. - """ - p = self.params - assert p.is_masked # Must be causal attention. - - # Gating operations - unnormalized_query_vec = query_vec - query_vec = self.layer_norm.FProp(theta.layer_norm, query_vec) - source_p_c = self.kv_gating.FProp(theta.kv_gating, query_vec) - source_vec = source_p_c * query_vec - - cached_packed_src = py_utils.NestedMap( - source_vecs=prefix_state.key, - source_contexts=prefix_state.value, - source_padding=None, - source_segment_id=None) - extended_packed_src = self.atten.ExtendSourcePacked(theta.atten, source_vec, - source_vec, None, None, - cached_packed_src, t) - return self._FinishExtendStep(theta, query_vec, unnormalized_query_vec, - extended_packed_src, t) - - -class CCTFeedForwardLayer(base_layer.BaseLayer): - """Transformer FF layer with CCT gating. - - https://arxiv.org/abs/2002.07106 - - Differences from standard Transformer FF layer: - 1. Each feedforward layer is divided into num_blocks smaller layers (divided - along the hidden dimension). - 2. Each block has its separate input layer norm. - 3. Each block has its separate output layer norm. - 4. Outputs from each block are gated with CCTGatingNetwork output - which is - between 0 and 1 for training and either 0 or 1 during inference. - """ - - @classmethod - def Params(cls): - p = super().Params() - # Transformer Feedforward params. - p.Define('input_dim', 0, 'Dimension of the layer input.') - p.Define('output_dim', 0, 'Dimension of the layer output.') # Deprecated. - p.Define('hidden_dim', 0, 'Dimension of the hidden layer.') - p.Define('ln_tpl', layers.LayerNorm.Params(), 'Layer norm default params') - p.Define('activation', 'RELU', 'Non-linearity.') - p.Define('fflayer_tpl', - layers.FeedForwardNet.Params().Set(activation=['RELU', 'NONE']), - 'Feed forward layer default params') - p.Define( - 'res_proj_tpl', layers.ProjectionLayer.Params(), - 'Residual projection default params, used when input_dim != ' - 'output_dim.') - p.Define( - 'residual_dropout_prob', 0.0, - 'Probability at which we apply dropout to the residual layers, ' - 'such that, residual(x, y) = (x + dropout(y)).') - p.Define( - 'residual_dropout_tpl', layers.DropoutLayer.Params(), - 'Residual dropout params template. keep_prop will be reset to ' - '(1.0 - residual_dropout_prob).') - p.Define( - 'relu_dropout_prob', 0.0, - 'Probability at which we apply dropout to the hidden layer ' - 'of feed-forward network.') - - # Expert params. - p.Define('num_blocks', 1, 'Number of separately gated ff blocks.') - p.Define('gating_tpl', layers.CCTGatingNetwork.Params(), 'gating template.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - assert p.input_dim - assert p.hidden_dim - assert not p.output_dim, 'output_dim should not be set.' - - # Initialize feed-forward layer - params = p.fflayer_tpl.Copy() - params.name = 'fflayer' - params.input_dim = p.input_dim - params.activation = [p.activation, 'NONE'] - if p.output_dim == 0: - params.hidden_layer_dims = [p.hidden_dim, p.input_dim] - else: - params.hidden_layer_dims = [p.hidden_dim, p.output_dim] - - params.dropout = [ - params.dropout.cls.Params().Set(keep_prob=1.0 - p.relu_dropout_prob), - params.dropout.cls.Params().Set(keep_prob=1.0) - ] - - ffs = [] - ln_params = [] - out_layer_norm = [] # Required for stabilizing CCT. - for i in range(p.num_blocks): - ff_p = params.Copy() - ff_p.name += '_%d' % i - ffs.append(ff_p) - - ln_p = p.ln_tpl.Copy() - ln_p.name = 'fflayer_ln_%d' % i - ln_p.input_dim = p.input_dim - ln_params.append(ln_p) - - ln_p = p.ln_tpl.Copy() - ln_p.name = 'fflayer_ln_out_%d' % i - ln_p.input_dim = p.input_dim - out_layer_norm.append(ln_p) - self.CreateChildren('fflayers', ffs) - self.CreateChildren('layer_norm', ln_params) - self.CreateChildren('out_layer_norm', out_layer_norm) - - # Note: Set gating noise and warmup in parent layer. - ff_gating = p.gating_tpl.Copy() - ff_gating.input_dim = p.input_dim - ff_gating.num_outputs = p.num_blocks - ff_gating.name = 'gating_net' - self.CreateChild('ff_gating', ff_gating) - - dropout_tpl = p.residual_dropout_tpl.Copy() - dropout_tpl.keep_prob = (1.0 - p.residual_dropout_prob) - self.CreateChild('residual_dropout', dropout_tpl) - - def FProp(self, theta, inputs, paddings): - """Feed-forward, layer-norm, residual, gating and layer-norm. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: [time, batch, dim]. - paddings: [time, batch] - - Returns: - tensor of the same shape with inputs - """ - p = self.params - ff_outputs = [] - for i in range(p.num_blocks): - inputs_normalized = self.layer_norm[i].FProp(theta.layer_norm[i], inputs) - ff_output = self.fflayers[i].FProp( - theta.fflayers[i], - inputs_normalized, - paddings=tf.expand_dims(paddings, -1)) - ff_output = self.out_layer_norm[i].FProp(theta.out_layer_norm[i], - ff_output) - ff_outputs.append(ff_output) - p_c = self.ff_gating.FProp(theta.ff_gating, inputs_normalized) - out = inputs + self.residual_dropout.FProp( - theta.residual_dropout, - tf.reduce_sum( - tf.expand_dims(p_c, -1) * tf.stack(ff_outputs, -2), axis=-2)) - return out, p_c - - -class TransformerWithContextLayer(base_layer.BaseLayer): - """A transformer layer with 3 attention layers. - - The same as layers_with_attention.TransformerLayer, but with an - additional attention layer to attend to a third transformer stack - representing context. - - self-attention => context attention (newly added as tertiary_atten) => - encoder attention (named aux_atten in TransformerLayer). - - The weights are *not* shared between these three attention layers. - - See https://arxiv.org/pdf/1810.03581.pdf - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('source_dim', 0, 'Dimension of the transformer block input.') - p.Define('output_dim', 0, 'Dimension of the transformer block output.') - p.Define( - 'tr_atten_tpl', - TransformerAttentionLayer.Params().Set(num_attention_heads=8), - 'Transformer Attention Layer params. The same template is applied ' - 'to all three attention layers.') - p.Define( - 'tr_tertiary_atten_tpl', None, - 'Transformer Attention Layer params for the tertiary attention. ' - 'When None, copies tr_atten_tpl above.') - p.Define('tr_fflayer_tpl', - TransformerFeedForwardLayer.Params().Set(hidden_dim=2048), - 'Transformer Feed-Forward Layer params.') - p.Define('packed_input', False, - 'If True, each training example may pack multiple sequences.') - # Required params used by the decoder. - p.Define('has_aux_atten', True, 'Must be True.') - p.Define('mask_self_atten', True, 'Must be True.') - # removed: p.num_aux_atten_post_proj, p.tr_post_ln_tpl - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.has_aux_atten - assert p.mask_self_atten - if not p.source_dim: - raise ValueError('p.source_dim not set') - - # Initialize multi-headed self-attention - params = p.tr_atten_tpl.Copy() - params.name = 'multihead_self_atten' - params.source_dim = p.source_dim - params.packed_input = p.packed_input - params.is_masked = True - self.CreateChild('self_atten', params) - - # Initialize tertiary attention. - # If p.tr_tertiary_atten_tpl is None, we fall back to using - # p.tr_tertiary_atten_tpl. - params = p.tr_tertiary_atten_tpl or p.tr_atten_tpl.Copy() - params.name = 'tertiary_multihead_atten' - params.source_dim = p.source_dim - params.packed_input = p.packed_input - self.CreateChild('tertiary_atten', params) - - # Initialize multi-headed encoder attention - params = p.tr_atten_tpl.Copy() - params.name = 'multihead_atten' - params.source_dim = p.source_dim - params.packed_input = p.packed_input - self.CreateChild('atten', params) - - # Initialize feed-forward layer - params = p.tr_fflayer_tpl.Copy() - params.name = 'tr_fflayer' - params.input_dim = p.source_dim - params.output_dim = p.output_dim - self.CreateChild('fflayer', params) - - def FProp(self, - theta, - source_vecs, - source_paddings, - aux_vecs, - aux_paddings, - tertiary_vecs, - tertiary_paddings, - source_segment_id=None, - aux_segment_id=None, - tertiary_segment_id=None, - **kwargs): - """Transformer Layer. - - Please see docstring of TransformerAttentionLayer.FProp. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - source_vecs: [source_time, source_batch, dim]. - source_paddings: [source_time, source_batch] - aux_vecs: [aux_time, aux_batch, dim] - aux_paddings: [aux_time, aux_batch] - tertiary_vecs: [tertiary_time, tertiary_batch, dim] - tertiary_paddings: [tertiary_time, tertiary_batch] - source_segment_id: [source_time, source_batch] - aux_segment_id: [aux_time, aux_batch] - tertiary_segment_id: [tertiary_time, tertiary_batch] - **kwargs: Can be optional params for the attention layer, eg. attention - projection index tensor. - - Returns: - The attention context vector, [source_time, source_batch, dim]. - - The attention probability vector, [source_time, source_batch, aux_time]. - """ - p = self.params - if p.packed_input: - assert source_segment_id is not None, ('Need to specify segment id for ' - 'packed input.') - assert aux_segment_id is not None, ('Need to specify segment id for ' - 'packed input.') - assert tertiary_segment_id is not None, ('Need to specify segment id for ' - 'packed input.') - - atten_vec, atten_prob = self.self_atten.FProp( - theta.self_atten, - source_vecs, - source_paddings, - query_segment_id=source_segment_id) - atten_vec, atten_prob = self.tertiary_atten.FProp( - theta.tertiary_atten, atten_vec, tertiary_paddings, tertiary_vecs, - source_segment_id, tertiary_segment_id) - atten_vec, atten_prob = self.atten.FProp(theta.atten, atten_vec, - aux_paddings, aux_vecs, - source_segment_id, aux_segment_id, - **kwargs) - - h = self.fflayer.FProp(theta.fflayer, atten_vec, source_paddings) - return h, atten_prob - - def ExtendStep(self, - theta, - source_vecs, - prefix_states, - aux_vecs, - aux_paddings, - tertiary_vecs, - tertiary_paddings, - t=None, - **kwargs): - """Transformer Layer, extend one step in decoding. - - Please see docstring of TransformerAttentionLayer.ExtendStep. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - source_vecs: [source_batch, dim]. - prefix_states: dict, containing tensors which are the results of previous - attentions, used for fast decoding. - aux_vecs: [aux_time, aux_batch, dim] - aux_paddings: [aux_time, aux_batch] tertiary_vecs=None, - tertiary_paddings=None, - tertiary_vecs: [tertiary_time, tertiary_batch, dim] - tertiary_paddings: [tertiary_time, tertiary_batch] - t: a scalar, the current time step, 0-based. - **kwargs: Can be optional params for the attention layer, eg. attention - projection index tensor. - - Returns: - The attention context vector, [target_batch, source_dim] - - The attention probability vector from the encoder attention layer (the - last attention layer) only, [source_time, target_batch]. - TODO(zhouwk): Return also the attention prob from the tertiary attention. - - Updated prefix states - """ - p = self.params - - batch_size = py_utils.GetShape(source_vecs)[0] - - # First the self-attention layer. - atten_vec, _, new_states = self.self_atten.ExtendStep( - theta.self_atten, source_vecs, prefix_states, t) - - # Next the context attention (tertiary_atten) layer. - atten_vec = tf.expand_dims(atten_vec, axis=0) - atten_vec, _ = self.tertiary_atten.FProp(theta.tertiary_atten, atten_vec, - tertiary_paddings, tertiary_vecs) - - # Next the source attention (aux_atten) layer. - atten_vec, atten_prob = self.atten.FProp(theta.atten, atten_vec, - aux_paddings, aux_vecs, **kwargs) - - # Finally, the feedforward layer. - h = self.fflayer.FProp( - theta.fflayer, atten_vec, - tf.zeros([1, batch_size], dtype=py_utils.FPropDtype(p))) - h = tf.squeeze(h, 0) - return h, atten_prob, new_states diff --git a/lingvo/core/layers_with_attention_test.py b/lingvo/core/layers_with_attention_test.py deleted file mode 100644 index db408070..00000000 --- a/lingvo/core/layers_with_attention_test.py +++ /dev/null @@ -1,2735 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for layers_with_attention.""" - -import lingvo.compat as tf -from lingvo.core import layers -from lingvo.core import layers_with_attention -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.core.test_utils import CompareToGoldenSingleFloat -import numpy as np - - -class LayersWithAttentionTest(test_utils.TestCase): - - def testTransformerFeedForwardLayerConstruction(self): - p = layers_with_attention.TransformerFeedForwardLayer.Params() - p.name = 'transformer_fflayer_1' - p.input_dim = 3 - p.hidden_dim = 7 - transformer_fflayer = layers_with_attention.TransformerFeedForwardLayer(p) - self.assertEqual(0, p.output_dim) - # output_dim = p.input_dim when p.output_dim is zero. - self.assertEqual(p.input_dim, transformer_fflayer.output_dim) - - # output_dim equals p.output_dim when p.output_dim is non zero. - p.output_dim = 10 - p.name = 'transformer_fflayer_2' - transformer_fflayer = p.Instantiate() - self.assertEqual(p.output_dim, transformer_fflayer.output_dim) - - def testTransformerFeedForwardLayer(self): - with self.session(use_gpu=True): - tf.random.set_seed(3980847392) - inputs = tf.random.normal([5, 2, 3], seed=948387483) - paddings = tf.zeros([5, 2]) - p = layers_with_attention.TransformerFeedForwardLayer.Params() - p.name = 'transformer_fflayer' - p.input_dim = 3 - p.hidden_dim = 7 - transformer_fflayer = layers_with_attention.TransformerFeedForwardLayer(p) - - h = transformer_fflayer.FPropDefaultTheta(inputs, paddings) - self.evaluate(tf.global_variables_initializer()) - actual_layer_output = self.evaluate(h) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [[-0.88366592, -0.05049637, 0.01003706], - [-0.10550675, 1.68050027, 2.29110384]], - [[-1.30083609, -0.40521634, 0.1911681 ], - [ 1.2597878 , 1.45850968, 1.58734488]], - [[ 0.10373873, -0.2716777 , 0.2314173 ], - [ 0.46293864, -0.06359965, 1.20189023]], - [[ 0.3673597 , -0.1691664 , 0.78656065], - [-1.51081395, -0.70281881, -0.9093715 ]], - [[-1.04800868, -0.70610946, -0.35321558], - [-1.92480004, 0.08361804, 0.62713993]]] - # pyformat: enable - # pylint: enable=bad-whitespace - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testTransformerFeedForwardLayerSpecOutDim(self): - with self.session(use_gpu=True): - tf.random.set_seed(3980847392) - inputs = tf.random.normal([5, 2, 3], seed=948387483) - paddings = tf.zeros([5, 2]) - p = layers_with_attention.TransformerFeedForwardLayer.Params() - p.name = 'transformer_fflayer' - p.input_dim = 3 - p.output_dim = 5 - p.hidden_dim = 7 - transformer_fflayer = layers_with_attention.TransformerFeedForwardLayer(p) - - h = transformer_fflayer.FPropDefaultTheta(inputs, paddings) - self.evaluate(tf.global_variables_initializer()) - actual_layer_output = self.evaluate(h) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [[ 1.42697251, 0.79269135, -0.85500956, -0.8122285 , -1.56555367], - [-1.7876718 , 0.26025945, -3.18244219, 1.34756351, 0.25739765]], - [[ 1.27962363, 0.88677615, -1.23556185, -1.06855559, -1.27293301], - [ 0.89336467, 2.46229172, 0.11302143, 1.19385004, -2.37805009]], - [[ 2.80146003, -0.66912627, 1.50160134, -2.30645609, -1.18872762], - [ 1.61967182, -0.51639485, 0.24441491, -1.0871532 , -0.95539457]], - [[ 2.03333473, -0.78205228, 0.71245927, -1.63276744, -0.91654319], - [ 1.54542768, -0.30343491, 0.10666496, -1.67965126, -0.15671858]], - [[ 1.60873222, -1.88402128, 0.79040933, -1.97199082, 0.4778356 ], - [-0.13516766, -0.42583361, -1.86275542, -1.09650302, 0.83263111]]] - # pyformat: enable - # pylint: enable=bad-whitespace - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def _testTransformerAttentionLayerInputs(self, - depth=3, - context_depth=3, - dtype=tf.float32): - np.random.seed(505837249) - source_vecs = tf.stack( - [tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(5)]) - source_padding = tf.transpose( - tf.constant([[0, 0, 1, 1, 0], [1, 0, 0, 0, 1]], dtype=dtype)) - aux_source_vecs = tf.stack( - [tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(7)]) - aux_source_paddings = tf.transpose( - tf.constant([[0, 1, 0, 1, 0, 1, 0], [1, 0, 1, 0, 1, 0, 1]], - dtype=dtype)) - context_vecs = tf.stack([ - tf.constant(np.random.rand(2, context_depth), dtype=dtype) - for _ in range(7) - ]) - return (source_vecs, source_padding, aux_source_vecs, aux_source_paddings, - context_vecs) - - def testTransformerAttentionLayerCase1(self): - with self.session(use_gpu=True): - depth = 4 - p = layers_with_attention.TransformerAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = False - p.num_attention_heads = 2 - transformer_atten = layers_with_attention.TransformerAttentionLayer(p) - - (source_vecs, source_padding, _, _, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - - ctx, probs = transformer_atten.FPropDefaultTheta(source_vecs, - source_padding) - self.evaluate(tf.global_variables_initializer()) - actual_ctx, actual_probs = self.evaluate([ctx, probs]) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [[-1.47126436, 1.46579707, 0.39105844, -0.88563323], - [-1.29514003, -1.08241224, 1.49894714, 2.5935874 ]], - [[-0.00313053, 1.17399275, -1.28071034, -1.6311729 ], - [-0.77028418, -0.18855178, -0.75814998, 2.19872856]], - [[ 1.72851753, -0.40323859, -1.19053328, -1.39761829], - [-1.72141743, -0.78715289, 1.28404212, 2.78338313]], - [[-0.8881942 , 0.33776048, 1.28791749, -0.45082122], - [ 1.4362365 , 0.46009994, -1.45436597, -1.90602148]], - [[-0.51681399, -0.70075679, -0.48352116, 1.93754733], - [-1.44486678, 0.81801879, -1.03079689, 1.86697066]]] - expected_probs = [ - [[ 0.21387868, 0.22080734, 0. , 0. , 0.56531399], - [ 0. , 0.30584112, 0.24723588, 0.44692296, 0. ]], - [[ 0.25358215, 0.50932312, 0. , 0. , 0.23709476], - [ 0. , 0.56834149, 0.2632803 , 0.16837817, 0. ]], - [[ 0.38519409, 0.55454361, 0. , 0. , 0.06026226], - [ 0. , 0.33708778, 0.21976741, 0.4431448 , 0. ]], - [[ 0.27139962, 0.12790371, 0. , 0. , 0.60069668], - [ 0. , 0.31849149, 0.28174096, 0.39976761, 0. ]], - [[ 0.16272782, 0.15781289, 0. , 0. , 0.67945927], - [ 0. , 0.55003977, 0.26049581, 0.18946445, 0. ]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_probs, actual_probs, rtol=1e-05, atol=1e-05) - - def testTransformerAttentionLayerCase1GatedResidualConnection(self): - with self.session(use_gpu=True): - depth = 4 - p = layers_with_attention.TransformerAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = False - p.num_attention_heads = 2 - p.add_unnormalized_input = True - p.residual_function = layers.HighwaySkipLayer.Params().Set( - carry_bias_init=100, couple_carry_transform_gates=True) - transformer_atten = layers_with_attention.TransformerAttentionLayer(p) - - (source_vecs, source_padding, _, _, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - - ctx, probs = transformer_atten.FPropDefaultTheta(source_vecs, - source_padding) - self.evaluate(tf.global_variables_initializer()) - actual_ctx, _, actual_source_vecs = self.evaluate( - [ctx, probs, source_vecs]) - # Due to the high bias, the gated residual connection is saturated and - # returns the original (unnormalized) input. - self.assertAllClose(actual_source_vecs, actual_ctx, rtol=1e-4, atol=1e-4) - - def testTransformerAttentionLayerCase2(self): - with self.session(use_gpu=True): - depth = 4 - p = layers_with_attention.TransformerAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = True - p.num_attention_heads = 2 - transformer_atten = layers_with_attention.TransformerAttentionLayer(p) - - (source_vecs, source_padding, _, _, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - ctx, probs = transformer_atten.FPropDefaultTheta(source_vecs, - source_padding) - self.evaluate(tf.global_variables_initializer()) - actual_ctx, actual_probs = self.evaluate([ctx, probs]) - tf.logging.info(np.array_repr(actual_ctx)) - tf.logging.info(np.array_repr(actual_probs)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [[-0.14429152, 1.15510106, 1.11930299, -1.19245839], - [-0.69580591, -0.47006619, 0.82592297, 0.69593251]], - [[ 0.24164687, 0.53328454, -1.02119482, -1.49412084], - [-0.82601064, 0.024203 , -1.11880171, 1.80784416]], - [[ 1.7644347 , -0.53346401, -1.1461122 , -1.42797422], - [-0.95326459, 0.39580142, 0.39262164, 0.67513674]], - [[-0.28252155, -0.95237327, 2.08757687, -0.21231559], - [ 1.4362365 , 0.46009994, -1.45436597, -1.90602148]], - [[-0.51681399, -0.70075679, -0.48352116, 1.93754733], - [-1.44486678, 0.81801879, -1.03079689, 1.86697066]]] - expected_probs = [ - [[ 1. , 0. , 0. , 0. , 0. ], - [ 0.2 , 0.2 , 0.2 , 0.2 , 0.2 ]], - [[ 0.3966811 , 0.60331887, 0. , 0. , 0. ], - [ 0. , 1. , 0. , 0. , 0. ]], - [[ 0.41050252, 0.58949745, 0. , 0. , 0. ], - [ 0. , 0.5245893 , 0.4754107 , 0. , 0. ]], - [[ 0.58882225, 0.41117775, 0. , 0. , 0. ], - [ 0. , 0.31849149, 0.28174096, 0.39976761, 0. ]], - [[ 0.16272782, 0.15781289, 0. , 0. , 0.67945927], - [ 0. , 0.55003977, 0.26049581, 0.18946445, 0. ]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_ctx, actual_ctx) - self.assertAllClose(expected_probs, actual_probs) - - def testTransformerAttentionLayerDeterministicDropout(self): - with self.session(use_gpu=True): - depth = 4 - p = layers_with_attention.TransformerAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = False - p.num_attention_heads = 2 - - p.residual_dropout_tpl = layers.DeterministicDropoutLayer.Params() - p.residual_dropout_prob = 0.1 - - transformer_atten = layers_with_attention.TransformerAttentionLayer(p) - - (source_vecs, source_padding, _, _, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - - ctx, probs = transformer_atten.FProp(transformer_atten.theta, source_vecs, - source_padding) - - self.evaluate(tf.global_variables_initializer()) - actual_ctx, actual_probs = self.evaluate([ctx, probs]) - - # pylint: disable=bad-whitespace - # pyformat: disable - print(np.array_repr(actual_ctx)) - expected_ctx = np.array([ - [[-1.45762944, 1.5337404 , 0.34037334, -0.97208667], - [-1.35992002, -1.06530988, 1.53705895, 2.79370689]], - [[ 0.00657134, 1.12030125, -1.32564592, -1.73569465], - [-0.80793667, -0.10877949, -0.80295694, 2.25494242]], - [[ 1.76956046, -0.50777751, -1.19745886, -1.46751583], - [-1.79178905, -0.77374339, 1.31586027, 2.98173356]], - [[-0.85498607, -0.37413225, 1.25707364, -0.50043333], - [ 1.62276983, 0.50820369, -1.52967572, -2.02076197]], - [[-0.66754031, -0.68657839, -0.51643699, 1.96581018], - [-1.4816376 , 0.89419198, -0.57226259, 1.90177512]] - ], dtype=np.float32) - - print(np.array_repr(actual_probs)) - expected_probs = np.array([ - [[ 0.21387868, 0.22080734, 0. , 0. , 0.56531399], - [ 0. , 0.30584112, 0.24723588, 0.44692296, 0. ]], - [[ 0.25358215, 0.50932312, 0. , 0. , 0.23709476], - [ 0. , 0.56834149, 0.2632803 , 0.16837817, 0. ]], - [[ 0.38519409, 0.55454361, 0. , 0. , 0.06026226], - [ 0. , 0.33708778, 0.21976741, 0.4431448 , 0. ]], - [[ 0.27139962, 0.12790371, 0. , 0. , 0.60069668], - [ 0. , 0.31849149, 0.28174096, 0.39976761, 0. ]], - [[ 0.16272782, 0.15781289, 0. , 0. , 0.67945927], - [ 0. , 0.55003977, 0.26049581, 0.18946445, 0. ]] - ], dtype=np.float32) - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_probs, actual_probs, rtol=1e-05, atol=1e-05) - - def testTransformerAttentionLayerStepByStep(self): - with self.session(use_gpu=True): - depth = 4 - p = layers_with_attention.TransformerAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = True - p.num_attention_heads = 2 - x_atten = layers_with_attention.TransformerAttentionLayer(p) - - (source_vecs, _, _, _, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - source_padding = tf.zeros([5, 2]) - - ctx1, probs1 = x_atten.FPropDefaultTheta(source_vecs, source_padding) - ctx2 = [] - probs2 = [] - cached_source_vecs = tf.zeros([0, 2, 4]) - cached_source_contexts = tf.zeros([0, 2, 4]) - prefix_states = py_utils.NestedMap( - key=cached_source_vecs, value=cached_source_contexts) - for i in range(5): - ctx, probs, prefix_states = x_atten.ExtendStep(x_atten.theta, - source_vecs[i, :, :], - prefix_states) - probs_pad = tf.zeros([2, 5 - i - 1]) - padded_probs = tf.concat([probs, probs_pad], 1) - ctx2.append(ctx) - probs2.append(padded_probs) - - ctx2 = tf.stack(ctx2) - probs2 = tf.stack(probs2) - - self.evaluate(tf.global_variables_initializer()) - ctx1_v, probs1_v, ctx2_v, probs2_v = self.evaluate( - [ctx1, probs1, ctx2, probs2]) - tf.logging.info(np.array_repr(ctx1_v)) - tf.logging.info(np.array_repr(probs1_v)) - tf.logging.info(np.array_repr(ctx2_v)) - tf.logging.info(np.array_repr(probs2_v)) - self.assertAllClose(ctx1_v, ctx2_v) - self.assertAllClose(probs1_v, probs2_v) - - def testTransformerAttentionLayerGatedResidualConnectionStepByStep(self): - with self.session(use_gpu=True): - depth = 4 - p = layers_with_attention.TransformerAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = True - p.num_attention_heads = 2 - p.residual_function = layers.HighwaySkipLayer.Params().Set( - couple_carry_transform_gates=True) - x_atten = layers_with_attention.TransformerAttentionLayer(p) - - (source_vecs, _, _, _, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - source_padding = tf.zeros([5, 2]) - - ctx1, probs1 = x_atten.FPropDefaultTheta(source_vecs, source_padding) - ctx2 = [] - probs2 = [] - cached_source_vecs = tf.zeros([0, 2, 4]) - cached_source_contexts = tf.zeros([0, 2, 4]) - prefix_states = py_utils.NestedMap( - key=cached_source_vecs, value=cached_source_contexts) - for i in range(5): - ctx, probs, prefix_states = x_atten.ExtendStep(x_atten.theta, - source_vecs[i, :, :], - prefix_states) - probs_pad = tf.zeros([2, 5 - i - 1]) - padded_probs = tf.concat([probs, probs_pad], 1) - ctx2.append(ctx) - probs2.append(padded_probs) - - ctx2 = tf.stack(ctx2) - probs2 = tf.stack(probs2) - - self.evaluate(tf.global_variables_initializer()) - ctx1_v, probs1_v, ctx2_v, probs2_v = self.evaluate( - [ctx1, probs1, ctx2, probs2]) - self.assertAllClose(ctx1_v, ctx2_v) - self.assertAllClose(probs1_v, probs2_v) - - def testTransformerAttentionLayerCase3(self): - with self.session(use_gpu=True): - depth = 4 - p = layers_with_attention.TransformerAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = False - p.num_attention_heads = 2 - transformer_atten = layers_with_attention.TransformerAttentionLayer(p) - - (query_vec, _, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - - ctx, probs = transformer_atten.FPropDefaultTheta(query_vec, aux_paddings, - aux_vecs) - self.evaluate(tf.global_variables_initializer()) - actual_ctx, actual_probs = self.evaluate([ctx, probs]) - tf.logging.info(np.array_repr(actual_ctx)) - tf.logging.info(np.array_repr(actual_probs)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [[-1.42420077, 1.19024372, 1.35146523, 0.85896158], - [-0.44974625, -1.00108492, 1.63387251, 1.678146 ]], - [[ 0.1134335 , 1.97617495, -0.35918081, 0.26396495], - [-0.19688171, -0.71197301, 0.0659425 , 2.5417304 ]], - [[ 1.58169425, 0.81259179, -0.58948535, 0.20254248], - [-0.84438968, -0.65845209, 1.45584249, 1.87587976]], - [[-1.01532316, -0.05166581, 2.07901478, 0.97540361], - [ 2.08563352, 0.34328598, -0.23240227, -0.19035631]], - [[-0.53881919, -0.60117185, 0.29170275, 2.6474514 ], - [-0.88318163, 0.37149727, -0.16098523, 2.3810885 ]]] - expected_probs = [ - [[ 0.32392544, 0., 0.27218491, 0., 0.19574419, 0., 0.20814547], - [ 0., 0.273045 , 0., 0.43572819, 0., 0.2912268 , 0.]], - [[ 0.24094662, 0., 0.23919827, 0., 0.26563686, 0., 0.25421822], - [ 0., 0.21680018, 0., 0.33962148, 0.,0.44357836 , 0.]], - [[ 0.20083594, 0., 0.20683075, 0., 0.28931937, 0., 0.30301392], - [ 0., 0.24710922, 0., 0.453915 , 0.,0.29897571 , 0.]], - [[ 0.32845193, 0., 0.26491433, 0., 0.18304622, 0., 0.22358747], - [ 0., 0.39426237, 0., 0.19774443, 0.,0.4079932 , 0.]], - [[ 0.23542665, 0., 0.27910906, 0., 0.30036426, 0., 0.18510005], - [ 0., 0.20147586, 0., 0.37759233, 0., 0.42093182, 0.]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_probs, actual_probs, rtol=1e-05, atol=1e-05) - - def _testTransformerAttentionLayerInputsMultiAuxSource( - self, aux_source_list, depth=3, context_depth=3, dtype=tf.float32): - (source_vecs, source_padding, _, _, _) = ( - self._testTransformerAttentionLayerInputs(depth, context_depth, dtype)) - np.random.seed(505837249) - aux_source_vecs = py_utils.NestedMap() - for aux_src_key in aux_source_list: - aux_source_vecs[aux_src_key] = tf.stack([ - tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(7) - ]) - aux_source_paddings = py_utils.NestedMap({ - aux_src_key: tf.transpose( - tf.constant([[0, 1, 0, 1, 0, 1, 0], [1, 0, 1, 0, 1, 0, 1]], - dtype=dtype)) for aux_src_key in aux_source_list - }) - context_vecs = py_utils.NestedMap() - for aux_src_key in aux_source_list: - context_vecs[aux_src_key] = tf.stack([ - tf.constant(np.random.rand(2, context_depth), dtype=dtype) - for _ in range(7) - ]) - return (source_vecs, source_padding, aux_source_vecs, aux_source_paddings, - context_vecs) - - def testTransformerAttentionLayerCase3MultiSource(self): - with self.session(use_gpu=True) as sess: - depth = 4 - p = layers_with_attention.TransformerMultiSourceAttentionLayer.Params() - p.name = 'transformer_atten_multisource' - p.source_dim = depth - p.is_masked = False - p.num_attention_heads = 2 - p.num_source = 2 - transformer_atten = ( - layers_with_attention.TransformerMultiSourceAttentionLayer(p)) - - (query_vec, _, aux_vecs, aux_paddings, _) = ( - self._testTransformerAttentionLayerInputsMultiAuxSource( - ['source_0', 'source_1'], depth=depth)) - - ctx, probs = transformer_atten.FPropDefaultTheta(query_vec, aux_paddings, - aux_vecs) - tf.global_variables_initializer().run() - actual_ctx, actual_probs = sess.run([ctx, probs]) - tf.logging.info(np.array_repr(actual_ctx)) - tf.logging.info(np.array_repr(actual_probs)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [[-1.9893163 , 0.8076348 , -0.33805895, -0.20369706], - [-1.4164762 , -1.0597495 , -0.3834126 , 0.3456189 ]], - [[-0.32503036, 1.4952568 , -1.9324137 , -0.77024114], - [-0.9230547 , -0.89096445, -1.7928462 , 1.0901089 ]], - [[ 1.2240632 , 0.26689315, -2.0940783 , -0.9101793 ], - [-1.805772 , -0.74725944, -0.5485071 , 0.5403221 ]], - [[-1.5880606 , -0.43595213, 0.3818947 , -0.15712431], - [ 0.968494 , 0.19423638, -2.308594 , -1.4253062 ]], - [[-0.8178122 , -1.1570994 , -1.1993079 , 1.4127911 ], - [-1.7231476 , 0.17116357, -2.0703826 , 0.96320933]]] - expected_probs = [ - [[0.16679956, 0., 0.2122806 , 0., 0.23512313, 0., 0.38579667], - [0., 0.28562695, 0., 0.3442661 , 0., 0.370107 , 0.]], - [[0.28629708, 0., 0.18837643, 0., 0.2644571 , 0., 0.26086944], - [0., 0.5590873 , 0., 0.22519027, 0., 0.21572247, 0.]], - [[0.3374045 , 0., 0.21468817, 0., 0.25822428, 0., 0.18968314], - [0., 0.2896077 , 0., 0.34381902, 0., 0.36657327, 0.]], - [[0.14310986, 0., 0.2507791 , 0., 0.22308563, 0., 0.3830254 ], - [0., 0.43070328, 0., 0.2930708 , 0., 0.27622598, 0.]], - [[0.30523974, 0., 0.30610216, 0., 0.2248916 , 0., 0.1637665 ], - [0., 0.49082592, 0., 0.26013914, 0., 0.24903494, 0.]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_probs, actual_probs, rtol=1e-05, atol=1e-05) - - def testTransformerAttentionLayerCase3MultiSourceMatchSingle(self): - with self.session(use_gpu=True) as sess: - # Prepare inputs. - depth = 4 - (query_vec, _, aux_vecs, aux_paddings, _) = ( - self._testTransformerAttentionLayerInputsMultiAuxSource( - ['source_0', 'source_1'], depth=depth)) - - # Create two source inputs but use single-source attention - p = layers_with_attention.TransformerMultiSourceAttentionLayer.Params() - p.random_seed = 123 - p.name = 'transformer_atten_multisource_single' - p.source_dim = depth - p.is_masked = False - p.num_attention_heads = 2 - p.num_source = 1 - msa = layers_with_attention.TransformerMultiSourceAttentionLayer(p) - msa_ctx, msa_probs = ( - msa.FPropDefaultTheta(query_vec, aux_paddings, aux_vecs)) - - # Original single source attention layer. - p = layers_with_attention.TransformerAttentionLayer.Params() - p.random_seed = 123 - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = False - p.num_attention_heads = 2 - ssa = layers_with_attention.TransformerAttentionLayer(p) - ssa_ctx, ssa_probs = ssa.FPropDefaultTheta(query_vec, - aux_paddings['source_0'], - aux_vecs['source_0']) - - # Compare two context vectors and probabilities. - tf.global_variables_initializer().run() - actual_msa_ctx, actual_msa_probs, actual_ssa_ctx, actual_ssa_probs = ( - sess.run([msa_ctx, msa_probs, ssa_ctx, ssa_probs])) - - # pylint: disable=bad-whitespace - # pyformat: disable - self.assertAllClose(actual_msa_ctx, actual_ssa_ctx, - rtol=1e-05, atol=1e-05) - self.assertAllClose(actual_msa_probs, actual_ssa_probs, - rtol=1e-05, atol=1e-05) - - def testTransformerAttentionLayerSourceContext(self): - # Equivalent: Passing no context vecs and source vecs as context vecs. - with self.session(use_gpu=True): - depth = 4 - p = layers_with_attention.TransformerAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = False - p.num_attention_heads = 2 - transformer_atten = layers_with_attention.TransformerAttentionLayer(p) - - (query_vec, _, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputs( - depth=depth, context_depth=depth) - - ctx1, probs1 = transformer_atten.FPropDefaultTheta( - query_vec=query_vec, - source_paddings=aux_paddings, - source_vecs=aux_vecs, - context_vecs=aux_vecs) - - ctx2, probs2 = transformer_atten.FPropDefaultTheta( - query_vec=query_vec, - source_paddings=aux_paddings, - source_vecs=aux_vecs) - - self.evaluate(tf.global_variables_initializer()) - actual_ctx1, actual_probs1, actual_ctx2, actual_probs2 = self.evaluate( - [ctx1, probs1, ctx2, probs2]) - self.assertAllEqual(actual_ctx1, actual_ctx2) - self.assertAllEqual(actual_probs1, actual_probs2) - - def testTransformerAttentionLayerCase4a(self): - # Distinct key and value vectors of the same size. - with self.session(use_gpu=True): - depth = 4 - p = layers_with_attention.TransformerAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = False - p.num_attention_heads = 2 - transformer_atten = layers_with_attention.TransformerAttentionLayer(p) - - (query_vec, _, aux_vecs, aux_paddings, - context_vecs) = self._testTransformerAttentionLayerInputs( - depth=depth, context_depth=depth) - - ctx, probs = transformer_atten.FPropDefaultTheta( - query_vec=query_vec, - source_paddings=aux_paddings, - source_vecs=aux_vecs, - context_vecs=context_vecs) - - self.evaluate(tf.global_variables_initializer()) - actual_ctx, actual_probs = self.evaluate([ctx, probs]) - tf.logging.info(np.array_repr(actual_ctx)) - tf.logging.info(np.array_repr(actual_probs)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [[-1.20854747, 1.25685954, 1.39818001, 0.558267 ], - [-0.39904317, -0.85738903, 1.45404375, 1.16389585]], - [[ 0.27544549, 1.93070388, -0.24477535, 0.12131107], - [-0.07007086, -0.53334039, -0.01144788, 2.03883505]], - [[ 1.72718525, 0.73558617, -0.45405889, 0.1063388 ], - [-0.76255953, -0.52610761, 1.30195093, 1.3571732 ]], - [[-0.79346895, 0.03049853, 2.11432981, 0.64747918], - [ 1.86823332, 0.3250314 , -0.50979781, -0.40038702]], - [[-0.30053592, -0.53348505, 0.41098642, 2.43903708], - [-0.75298154, 0.50427407, -0.23542863, 1.89634883]]] - expected_probs = [ - [[ 0.32392544, 0., 0.27218491, 0., 0.19574417, 0., 0.20814548], - [ 0., 0.273045 , 0., 0.43572825, 0., 0.2912268 , 0.]], - [[ 0.24094665, 0., 0.23919825, 0., 0.26563686, 0., 0.25421822], - [ 0., 0.21680018, 0., 0.33962148, 0., 0.44357836, 0.]], - [[ 0.20083596, 0., 0.20683077, 0., 0.28931937, 0., 0.30301392], - [ 0., 0.24710923, 0., 0.45391506, 0., 0.29897574, 0.]], - [[ 0.32845187, 0., 0.26491439, 0., 0.18304622, 0., 0.22358751], - [ 0., 0.39426237, 0., 0.1977444 , 0., 0.4079932 , 0.]], - [[ 0.23542665, 0., 0.27910906, 0., 0.30036426, 0., 0.18510005], - [ 0., 0.20147583, 0., 0.37759233, 0., 0.42093182, 0.]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_probs, actual_probs, rtol=1e-05, atol=1e-05) - - def testTransformerAttentionLayerCase4aMultiSource(self): - # Distinct key and value vectors of the same size. - with self.session(use_gpu=True) as sess: - depth = 4 - p = layers_with_attention.TransformerMultiSourceAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = False - p.num_attention_heads = 2 - p.num_source = 2 - transformer_atten = ( - layers_with_attention.TransformerMultiSourceAttentionLayer(p)) - - (query_vec, _, aux_vecs, aux_paddings, - context_vecs) = self._testTransformerAttentionLayerInputsMultiAuxSource( - ['source_0', 'source_1'], depth=depth, context_depth=depth) - - ctx, probs = transformer_atten.FPropDefaultTheta( - query_vec=query_vec, - source_paddings=aux_paddings, - source_vecs=aux_vecs, - context_vecs=context_vecs) - - tf.global_variables_initializer().run() - actual_ctx, actual_probs = sess.run([ctx, probs]) - tf.logging.info(np.array_repr(actual_ctx)) - tf.logging.info(np.array_repr(actual_probs)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [[-2.263544 , -0.6288333 , 0.56436384, 0.01389617], - [-1.2714428 , -2.6551175 , 1.2088637 , 0.48963785]], - [[-0.7530552 , 0.2863059 , -1.0583341 , -0.62887365], - [-0.96861804, -2.3108015 , -0.32213187, 1.4070555 ]], - [[ 0.6888912 , -0.83782226, -1.3349627 , -0.69250315], - [-1.646423 , -2.3046758 , 1.0617565 , 0.6768545 ]], - [[-1.8710074 , -1.9080507 , 1.2318314 , 0.14334393], - [ 0.92007947, -1.775676 , -1.1390316 , -0.9541185 ]], - [[-1.375605 , -2.3637016 , -0.5955716 , 1.8448071 ], - [-1.6682272 , -1.2519215 , -0.5330956 , 1.2296966 ]]] - expected_probs = [ - [[0.22346233, 0., 0.27624047, 0., 0.18855348, 0., 0.31174374], - [0., 0.17387941, 0., 0.4642802 , 0., 0.36184043, 0.]], - [[0.23724607, 0., 0.24033949, 0., 0.3725937 , 0., 0.14982074], - [0., 0.15892553, 0., 0.4639521 , 0., 0.37712237, 0.]], - [[0.25570837, 0., 0.21216837, 0., 0.40378904, 0., 0.12833425], - [0., 0.16656096, 0., 0.47455215, 0., 0.3588869 , 0.]], - [[0.22077632, 0., 0.27379048, 0., 0.14691363, 0., 0.35851952], - [0., 0.5620029 , 0., 0.21104112, 0., 0.22695602, 0.]], - [[0.20673111, 0., 0.22832122, 0., 0.12665181, 0., 0.43829578], - [0., 0.17881572, 0., 0.45228398, 0., 0.36890027, 0.]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_probs, actual_probs, rtol=1e-05, atol=1e-05) - - def testTransformerAttentionLayerCase4b(self): - # Distinct key and value vectors of different sizes. - with self.session(use_gpu=True): - depth = 4 - context_depth = 3 - p = layers_with_attention.TransformerAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = False - print(p) - p.num_attention_heads = 2 - p.atten_tpl.enable_ctx_pre_proj = True # Project values first. - p.context_dim = context_depth - transformer_atten = layers_with_attention.TransformerAttentionLayer(p) - - (query_vec, _, aux_vecs, aux_paddings, - context_vecs) = self._testTransformerAttentionLayerInputs( - depth=depth, context_depth=context_depth) - - ctx, probs = transformer_atten.FPropDefaultTheta( - query_vec=query_vec, - source_paddings=aux_paddings, - source_vecs=aux_vecs, - context_vecs=context_vecs) - - self.evaluate(tf.global_variables_initializer()) - actual_ctx, actual_probs = self.evaluate([ctx, probs]) - tf.logging.info(np.array_repr(actual_ctx)) - tf.logging.info(np.array_repr(actual_probs)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [[-1.78694427, 0.47923172, 0.89032698, 0.05556235], - [-0.91133636, -2.05677342, 1.30821121, 1.17388368]], - [[-0.24106422, 1.27436733, -0.84274787, -0.58437365], - [-0.58214164, -1.7144506 , -0.21780583, 2.03152227]], - [[ 1.22925639, 0.15926462, -1.10279834, -0.69442266], - [-1.2955091 , -1.72805309, 1.15411568, 1.39945638]], - [[-1.38178754, -0.7436831 , 1.60785818, 0.16023314], - [ 1.5662415 , -0.77094424, -0.63392496, -0.6477108 ]], - [[-0.83664525, -1.20021605, -0.15795891, 1.81301379], - [-1.27991939, -0.67706013, -0.42443359, 1.92405224]]] - # Probabilities are unaffected by change of value vectors. - expected_probs = [ - [[ 0.32392544, 0., 0.27218491, 0., 0.19574417, 0., 0.20814548], - [ 0., 0.273045 , 0., 0.43572825, 0., 0.2912268 , 0.]], - [[ 0.24094665, 0., 0.23919825, 0., 0.26563686, 0., 0.25421822], - [ 0., 0.21680018, 0., 0.33962148, 0., 0.44357836, 0.]], - [[ 0.20083596, 0., 0.20683077, 0., 0.28931937, 0., 0.30301392], - [ 0., 0.24710923, 0., 0.45391506, 0., 0.29897574, 0.]], - [[ 0.32845187, 0., 0.26491439, 0., 0.18304622, 0., 0.22358751], - [ 0., 0.39426237, 0., 0.1977444 , 0., 0.4079932 , 0.]], - [[ 0.23542665, 0., 0.27910906, 0., 0.30036426, 0., 0.18510005], - [ 0., 0.20147583, 0., 0.37759233, 0., 0.42093182, 0.]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_probs, actual_probs, rtol=1e-05, atol=1e-05) - - def testTransformerAttentionLayerCase4bMultiSource(self): - # Distinct key and value vectors of different sizes. - with self.session(use_gpu=True) as sess: - depth = 4 - context_depth = 3 - p = layers_with_attention.TransformerMultiSourceAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = False - print(p) - p.num_attention_heads = 2 - p.atten_tpl.enable_ctx_pre_proj = True # Project values first. - p.context_dim = context_depth - p.num_source = 2 - transformer_atten = ( - layers_with_attention.TransformerMultiSourceAttentionLayer(p)) - - (query_vec, _, aux_vecs, aux_paddings, - context_vecs) = self._testTransformerAttentionLayerInputsMultiAuxSource( - ['source_0', 'source_1'], depth=depth, context_depth=context_depth) - - ctx, probs = transformer_atten.FPropDefaultTheta( - query_vec=query_vec, - source_paddings=aux_paddings, - source_vecs=aux_vecs, - context_vecs=context_vecs) - - tf.global_variables_initializer().run() - actual_ctx, actual_probs = sess.run([ctx, probs]) - tf.logging.info(np.array_repr(actual_ctx)) - tf.logging.info(np.array_repr(actual_probs)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [[-0.52144265, 1.7370229 , 0.09479183, 1.3142197 ], - [ 0.48182625, -0.41524518, 0.2950616 , 2.3245158 ]], - [[ 1.0139368 , 2.6589985 , -1.528513 , 0.6880791 ], - [ 0.7810391 , -0.05419022, -1.227257 , 3.2472034 ]], - [[ 2.4781933 , 1.5413835 , -1.7759092 , 0.6057711 ], - [ 0.11952043, -0.07813096, 0.12346762, 2.5386043 ]], - [[-0.12219751, 0.46310303, 0.7768879 , 1.4295386 ], - [ 2.8404353 , 0.901297 , -1.5073049 , 0.60736287]], - [[ 0.37801886, -0.05114734, -1.003877 , 3.0894797 ], - [ 0.10942292, 0.975695 , -1.4856565 , 3.1215234 ]]] - # Probabilities are unaffected by change of value vectors. - expected_probs = [ - [[0.22346234, 0., 0.27624047, 0., 0.18855348, 0., 0.31174374], - [0., 0.17387941, 0., 0.4642802 , 0., 0.36184043, 0.]], - [[0.23724607, 0., 0.24033949, 0., 0.3725937 , 0., 0.14982076], - [0., 0.15892553, 0., 0.4639521 , 0., 0.3771224 , 0.]], - [[0.2557084 , 0., 0.21216837, 0., 0.403789 , 0., 0.12833424], - [0., 0.16656098, 0., 0.47455215, 0., 0.3588869 , 0.]], - [[0.22077632, 0., 0.27379048, 0., 0.14691365, 0., 0.35851952], - [0., 0.5620028 , 0., 0.21104114, 0., 0.22695604, 0.]], - [[0.20673111, 0., 0.22832122, 0., 0.12665181, 0., 0.43829578], - [0., 0.17881574, 0., 0.45228398, 0., 0.36890027, 0.]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_probs, actual_probs, rtol=1e-05, atol=1e-05) - - def testTransformerAttentionLayerCase5(self): - with self.session(use_gpu=True): - depth = 4 - p = layers_with_attention.TransformerAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = True - p.mask_type = 'eye' - p.num_attention_heads = 2 - transformer_atten = layers_with_attention.TransformerAttentionLayer(p) - - (source_vecs, source_padding, _, _, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - ctx, probs = transformer_atten.FPropDefaultTheta(source_vecs, - source_padding) - self.evaluate(tf.global_variables_initializer()) - actual_ctx, actual_probs = self.evaluate([ctx, probs]) - tf.logging.info(np.array_repr(actual_ctx)) - tf.logging.info(np.array_repr(actual_probs)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [[-1.89149332, 1.18417633, 0.09695292, -0.83397102], - [-1.29514003, -1.08241224, 1.49894726, 2.59358764]], - [[ 0.79232693, 2.47633171, -0.90657401, -1.5221628 ], - [-0.14457735, 0.09040731, -0.12422991, 2.13300467]], - [[ 1.72851753, -0.40323859, -1.19053328, -1.39761829], - [-2.15129089, -1.16594994, 1.1004864 , 3.07194686]], - [[-0.88819426, 0.3377606 , 1.28791749, -0.45082125], - [1.97874951, 1.50414598, -1.15547466, -1.18697572]], - [[ 0.10235745, -1.51675844, 0.13308235, 1.26194644], - [-1.44486666, 0.81801897, -1.03079677, 1.86697078]]] - expected_probs = [ - [[ 0. , 0.33807203, 0. , 0. , 0.661928 ], - [ 0. , 0.30584112, 0.24723586, 0.44692296, 0. ]], - [[ 0.63300228, 0. , 0. , 0. , 0.36699772], - [ 0. , 0. , 0.70683479, 0.29316518, 0. ]], - [[ 0.38519406, 0.55454367, 0. , 0. , 0.06026225], - [ 0. , 0.51602799, 0. , 0.48397198, 0. ]], - [[ 0.27139962, 0.12790368, 0. , 0. , 0.60069668], - [ 0. , 0.46712866, 0.53287131, 0. , 0. ]], - [[ 0.55518425, 0.4448157 , 0. , 0. , 0. ], - [ 0. , 0.55003977, 0.26049584, 0.18946445, 0. ]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_ctx, actual_ctx) - self.assertAllClose(expected_probs, actual_probs) - - def testTransformerAttentionLayerCase6(self): - with self.session(use_gpu=True): - depth = 4 - p = layers_with_attention.TransformerAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = True - p.mask_type = 'ngram' - p.mask_ngram_order = 3 - p.num_attention_heads = 2 - transformer_atten = layers_with_attention.TransformerAttentionLayer(p) - - (source_vecs, source_padding, _, _, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - ctx, probs = transformer_atten.FPropDefaultTheta(source_vecs, - source_padding) - self.evaluate(tf.global_variables_initializer()) - actual_ctx, actual_probs = self.evaluate([ctx, probs]) - tf.logging.info(np.array_repr(actual_ctx)) - tf.logging.info('actual_probs=%r', np.array_repr(actual_probs)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [[-0.14429152, 1.155101, 1.119303, -1.1924583], - [-0.6958059, -0.47006613, 0.8259231, 0.6959326]], - [[0.24164662, 0.5332843, -1.0211949, -1.4941208], - [-0.8260106, 0.024203, -1.1188016, 1.807844]], - [[1.7644346, -0.533464, -1.1461123, -1.4279743], - [-0.95326424, 0.39580172, 0.39262217, 0.6751373]], - [[-1.3441969, -2.3305228, 1.7523124, 0.15416345], - [1.4362367, 0.46009994, -1.4543657, -1.9060212]], - [[-0.8291472, 0.21259767, -0.9077787, 1.6243731], - [-1.0709695, 0.74920934, -0.5950014, 1.5919089]]] - expected_probs = [ - [[1. , 0. , 0. , 0. , 0. ], - [0.2 , 0.2 , 0.2 , 0.2 , 0.2 ]], - [[0.3966811 , 0.6033189 , 0. , 0. , 0. ], - [0. , 1. , 0. , 0. , 0. ]], - [[0.41050246, 0.5894975 , 0. , 0. , 0. ], - [0. , 0.5245893 , 0.4754107 , 0. , 0. ]], - [[0. , 1. , 0. , 0. , 0. ], - [0. , 0.31849146, 0.28174093, 0.39976764, 0. ]], - [[0. , 0. , 0. , 0. , 1. ], - [0. , 0. , 0.5881755 , 0.41182452, 0. ]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_ctx, actual_ctx) - self.assertAllClose(expected_probs, actual_probs) - - def testTransformerLayerConstruction(self): - p = layers_with_attention.TransformerLayer.Params() - p.name = 'transformer_1' - p.source_dim = 4 - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_attention_heads = 2 - p.has_aux_atten = True - p.mask_self_atten = True - layer = layers_with_attention.TransformerLayer(p) - # output_dim is equal to source_dim when p.output_dim == 0 - self.assertEqual(0, p.output_dim) - self.assertEqual(p.source_dim, layer.output_dim) - # output_dim corresponds to p.output_dim when it is non-zero. - p.output_dim = 6 - p.name = 'transformer_2' - layer = p.Instantiate() - self.assertEqual(p.output_dim, layer.output_dim) - - def testTransformerLayerFProp(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.TransformerLayer.Params() - p.name = 'transformer' - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_attention_heads = 2 - transformer = layers_with_attention.TransformerLayer(p) - - (source_vecs, source_padding, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - - h, probs = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings) - - self.evaluate(tf.global_variables_initializer()) - actual_layer_output, actual_prob_output = self.evaluate([h, probs]) - tf.logging.info(np.array_repr(actual_layer_output)) - tf.logging.info(np.array_repr(actual_prob_output)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_layer_output = [ - [[ 0.68134278, 0.74287307, 0.04602078, 1.99463582], - [ 0.20382279, -1.50973201, 1.33421206, 0.53317755]], - [[ 2.46715426, 2.84406185, -0.60359633, 0.51742059], - [ 1.06444919, -1.45264888, -0.06196141, 0.35242724]], - [[ 2.3442452 , -0.56243378, -1.1149826 , 0.50276589], - [ 1.04868603, -1.68515253, 0.3093726 , -0.19512933]], - [[-0.11517292, -1.21290886, 1.31996512, 1.14821553], - [ 3.14395714, -1.07060659, 0.27842081, -1.81273639]], - [[ 1.39219522, -0.81882864, -0.32732445, 1.36851478], - [-0.79119539, -0.28148842, 0.29963702, 1.37034667]]] - expected_prob_output = [ - [[ 0.21795762, 0., 0.26612395, 0., 0.31251648, 0., 0.20340192], - [ 0., 0.2677784 , 0., 0.32895881, 0., 0.40326279, 0.]], - [[ 0.25721505, 0., 0.24116731, 0., 0.25138181, 0., 0.2502358 ], - [ 0., 0.25691482, 0., 0.31076014, 0., 0.43232504, 0.]], - [[ 0.24550268, 0., 0.25128055, 0., 0.25109866, 0., 0.25211811], - [ 0., 0.26769161, 0., 0.32481128, 0., 0.40749705, 0.]], - [[ 0.22675318, 0., 0.26633731, 0., 0.28919035, 0., 0.21771915], - [ 0., 0.35955882, 0., 0.36869824, 0., 0.271743 , 0.]], - [[ 0.21504655, 0., 0.26958644, 0., 0.30847484, 0., 0.20689213], - [ 0., 0.29516917, 0., 0.29359812, 0., 0.41123265, 0.]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_layer_output, actual_layer_output) - self.assertAllClose(expected_prob_output, actual_prob_output) - - def testMultiAuxSourceTransformerLayerFProp(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.TransformerLayer.Params() - p.name = 'transformer' - p.source_dim = depth - p.has_aux_atten = True - p.tr_aux_atten_tpl = ( - layers_with_attention.TransformerMultiSourceAttentionLayer.Params() - .Set( - source_dim=p.source_dim, - num_source=2, - primary_source_index=0, - num_attention_heads=4)) - p.mask_self_atten = True - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_attention_heads = 2 - transformer = layers_with_attention.TransformerLayer(p) - - (source_vecs, source_padding, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputsMultiAuxSource( - ['source_0', 'source_1'], depth=depth) - - h, probs = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings) - - self.evaluate(tf.global_variables_initializer()) - actual_layer_output, actual_prob_output = self.evaluate([h, probs]) - tf.logging.info(np.array_repr(actual_layer_output)) - tf.logging.info(np.array_repr(actual_prob_output)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_layer_output = [ - [[-0.06297368, 0.75025094, -0.18167767, 2.27935 ], - [-0.22771487, -1.9459789 , 0.758848 , 1.2273839 ]], - [[ 1.6866916 , 2.9894042 , -1.2287276 , 0.8018402 ], - [ 0.656631 , -1.2074132 , -0.41612232, 1.4099871 ]], - [[ 1.6463919 , -0.493517 , -1.3494966 , 0.6977608 ], - [ 0.49527422, -1.5192728 , -0.1677584 , 0.781141 ]], - [[-0.86701846, -1.2044021 , 1.0710557 , 1.4103888 ], - [ 3.0039275 , -0.98788637, -0.48796502, -0.90612394]], - [[ 0.6298464 , -0.33676302, -0.22484902, 1.8341833 ], - [-1.2259507 , -0.716857 , -0.1336647 , 1.9020087 ]]] - expected_prob_output = [ - [[0.23055646, 0., 0.270754 , 0., 0.20824522, 0., 0.2904443 ], - [0., 0.34072176, 0., 0.34083408, 0., 0.31844413, 0.]], - [[0.25588194, 0., 0.21465777, 0., 0.26527345, 0., 0.26418683], - [0., 0.31694067, 0., 0.35715103, 0., 0.32590824, 0.]], - [[0.24147315, 0., 0.22742277, 0., 0.2734162 , 0., 0.25768787], - [0., 0.33686832, 0., 0.34380934, 0., 0.31932235, 0.]], - [[0.22445586, 0., 0.29794338, 0., 0.20764738, 0., 0.26995337], - [0., 0.3731808 , 0., 0.29736063, 0., 0.32945853, 0.]], - [[0.2221506 , 0., 0.2830769 , 0., 0.21007922, 0., 0.2846933 ], - [0., 0.3024338 , 0., 0.36399618, 0., 0.33357003, 0.]]] - # # pyformat: enable - # # pylint: enable=bad-whitespace - self.assertAllClose(expected_layer_output, actual_layer_output) - self.assertAllClose(expected_prob_output, actual_prob_output) - - def testMultiAuxSourceTransformerLayerFPropMatchSingle(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - # Multi-source transformer layer - p = layers_with_attention.TransformerLayer.Params().Set( - name='multi_source_trans', random_seed=123) - p.tr_atten_tpl.num_attention_heads = 4 - p.source_dim = depth - p.has_aux_atten = True - p.tr_aux_atten_tpl = ( - layers_with_attention.TransformerMultiSourceAttentionLayer.Params() - .Set( - source_dim=p.source_dim, - num_source=1, - primary_source_index=0, - num_attention_heads=4)) - p.mask_self_atten = True - p.tr_fflayer_tpl.hidden_dim = 7 - msa_trans = layers_with_attention.TransformerLayer(p) - - (source_vecs, source_padding, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputsMultiAuxSource( - ['source_0', 'source_1'], depth=depth) - - msa_h, msa_probs = msa_trans.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings) - - # Original single-source transformer decoder. - p = layers_with_attention.TransformerLayer.Params().Set( - name='single_source_trans', random_seed=123) - p.tr_atten_tpl.num_attention_heads = 4 - p.tr_atten_tpl.random_seed = 123 - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_fflayer_tpl.hidden_dim = 7 - ssa_trans = layers_with_attention.TransformerLayer(p) - ssa_h, ssa_probs = ssa_trans.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs['source_0'], - aux_paddings=aux_paddings['source_0']) - - self.evaluate(tf.global_variables_initializer()) - msa_layer_output, msa_prob_output, ssa_layer_output, ssa_prob_output = ( - self.evaluate([msa_h, msa_probs, ssa_h, ssa_probs])) - - self.assertAllClose( - msa_layer_output, ssa_layer_output, rtol=1e-05, atol=1e-05) - self.assertAllClose( - msa_prob_output, ssa_prob_output, rtol=1e-05, atol=1e-05) - - def testTransformerLayerOutputLayerNormFProp(self): - """Test post-layernorm Fprop.""" - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.TransformerLayer.Params() - p.name = 'transformer' - p.source_dim = depth - p.has_aux_atten = True - p.tr_post_ln_tpl = layers.LayerNorm.Params() - p.mask_self_atten = True - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_attention_heads = 2 - transformer = layers_with_attention.TransformerLayer(p) - - (source_vecs, source_padding, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - - h, probs = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings) - - self.evaluate(tf.global_variables_initializer()) - actual_layer_output, actual_prob_output = self.evaluate([h, probs]) - tf.logging.info(np.array_repr(actual_layer_output)) - tf.logging.info(np.array_repr(actual_prob_output)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_layer_output = [ - [[-0.2617511, -0.17463534, -1.1612566, 1.5976431], - [ 0.06115358, -1.5903126, 1.1505843, 0.37857458]], - [[ 0.821784, 1.0885929, -1.351966, -0.5584109], - [ 1.1864979, -1.5562507, -0.04089222, 0.41064504]], - [[ 1.5548539, -0.6477773, -1.0664893, 0.15941268], - [ 1.1784918, -1.5536082, 0.43964866, -0.06453241]], - [[-0.38961875, -1.4583365, 1.0075824, 0.84037286], - [ 1.5903242, -0.6370207, 0.07592358, -1.0292271]], - [[ 0.99643826, -1.232215, -0.73679215, 0.972569], - [-1.1702524, -0.5360445, 0.18702725, 1.5192697]]] - expected_prob_output = [ - [[ 0.21795762, 0., 0.26612395, 0., 0.31251648, 0., 0.20340192], - [ 0., 0.2677784 , 0., 0.32895881, 0., 0.40326279, 0.]], - [[ 0.25721505, 0., 0.24116731, 0., 0.25138181, 0., 0.2502358 ], - [ 0., 0.25691482, 0., 0.31076014, 0., 0.43232504, 0.]], - [[ 0.24550268, 0., 0.25128055, 0., 0.25109866, 0., 0.25211811], - [ 0., 0.26769161, 0., 0.32481128, 0., 0.40749705, 0.]], - [[ 0.22675318, 0., 0.26633731, 0., 0.28919035, 0., 0.21771915], - [ 0., 0.35955882, 0., 0.36869824, 0., 0.271743 , 0.]], - [[ 0.21504655, 0., 0.26958644, 0., 0.30847484, 0., 0.20689213], - [ 0., 0.29516917, 0., 0.29359812, 0., 0.41123265, 0.]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_layer_output, actual_layer_output) - self.assertAllClose(expected_prob_output, actual_prob_output) - - def testTransformerLayerFPropMultiPostProj(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.TransformerLayer.Params() - p.name = 'transformer' - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_attention_heads = 2 - p.num_aux_atten_post_proj = 2 - transformer = layers_with_attention.TransformerLayer(p) - - (source_vecs, source_padding, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - - # Duplicate atten_idx n=2 times. - atten_idx = tf.constant([0, 1, 1, 0, 1] * 2, dtype=tf.int32) - h, probs = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings, - atten_idx=atten_idx) - - self.evaluate(tf.global_variables_initializer()) - actual_layer_output, actual_prob_output = self.evaluate([h, probs]) - tf.logging.info(np.array_repr(actual_layer_output)) - tf.logging.info(np.array_repr(actual_prob_output)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_layer_output = [ - [[-0.77411413, 0.86493313, 0.08914688, 1.4910977 ], - [-1.0093606 , -1.7337079 , 1.2784883 , 0.49974248]], - [[ 1.0396315 , 2.902943 , -1.1812847 , 0.19860795], - [-0.37676954, -0.79837584, 0.6419263 , 0.45496815]], - [[ 1.0858665 , -0.6838142 , -1.2464247 , 0.14764154], - [-0.45331526, -1.0229169 , 1.0660815 , -0.06151289]], - [[-1.3433903 , -1.3154784 , 1.1818855 , 0.790216 ], - [ 1.8400799 , -1.5192697 , 0.05896807, -1.94113 ]], - [[-0.11429042, -0.24730963, 0.06099784, 1.0156208 ], - [-1.9910344 , -0.5176018 , 0.2490384 , 1.3254449 ]]] - expected_prob_output = [ - [[ 0.21795762, 0., 0.26612395, 0., 0.31251648, 0., 0.20340192], - [ 0., 0.2677784 , 0., 0.32895881, 0., 0.40326279, 0.]], - [[ 0.25721505, 0., 0.24116731, 0., 0.25138181, 0., 0.2502358 ], - [ 0., 0.25691482, 0., 0.31076014, 0., 0.43232504, 0.]], - [[ 0.24550268, 0., 0.25128055, 0., 0.25109866, 0., 0.25211811], - [ 0., 0.26769161, 0., 0.32481128, 0., 0.40749705, 0.]], - [[ 0.22675318, 0., 0.26633731, 0., 0.28919035, 0., 0.21771915], - [ 0., 0.35955882, 0., 0.36869824, 0., 0.271743 , 0.]], - [[ 0.21504655, 0., 0.26958644, 0., 0.30847484, 0., 0.20689213], - [ 0., 0.29516917, 0., 0.29359812, 0., 0.41123265, 0.]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_layer_output, actual_layer_output) - self.assertAllClose(expected_prob_output, actual_prob_output) - - def testTransformerLayerWithInputPackingFProp(self): - with self.session(use_gpu=True): - with tf.variable_scope('transformer_packed_test', reuse=tf.AUTO_REUSE): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.TransformerLayer.Params() - p.name = 'transformer' - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_attention_heads = 2 - packed_params = p.Copy() - transformer = layers_with_attention.TransformerLayer(p) - packed_params.packed_input = True - transformer_packed = layers_with_attention.TransformerLayer( - packed_params) - - dtype = tf.float32 - source_vecs = tf.stack([ - tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(5) - ]) - source_padding = tf.transpose( - tf.constant([[0, 0, 0, 0, 1], [0, 0, 0, 0, 0]], dtype=dtype)) - aux_vecs = tf.stack([ - tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(7) - ]) - aux_paddings = tf.transpose( - tf.constant([[0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1, 1]], - dtype=dtype)) - - source_vecs_packed = tf.reshape(source_vecs, [-1, 1, depth]) - aux_vecs_packed = tf.reshape(aux_vecs, [-1, 1, depth]) - source_padding_packed = tf.reshape(source_padding, [-1, 1]) - aux_padding_packed = tf.reshape(aux_paddings, [-1, 1]) - source_segment_id = tf.transpose( - tf.constant([[0, 1, 0, 1, 0, 1, 0, 1, 0, 1]], dtype=tf.float32)) - aux_segment_id = tf.transpose( - tf.constant([[0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1]], - dtype=tf.float32)) - - h, _ = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings, - source_segment_id=None, - aux_segment_id=None) - - h_packed, _ = transformer_packed.FPropDefaultTheta( - source_vecs_packed, - source_padding_packed, - aux_vecs=aux_vecs_packed, - aux_paddings=aux_padding_packed, - source_segment_id=source_segment_id, - aux_segment_id=aux_segment_id) - h_packed = tf.reshape(h_packed, tf.shape(h)) - - self.evaluate(tf.global_variables_initializer()) - actual_layer, p_layer = self.evaluate([h, h_packed]) - self.assertAllClose(actual_layer, p_layer) - - def testTransformerLayerExtendStep(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.TransformerLayer.Params() - p.name = 'transformer' - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_atten_tpl.num_attention_heads = 2 - transformer = layers_with_attention.TransformerLayer(p) - - (source_vecs, _, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - source_padding = tf.zeros([5, 2]) - - h1, probs1 = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings) - - h2 = [] - probs2 = [] - cached_source_vecs = tf.zeros([0, 2, 4]) - cached_source_contexts = tf.zeros([0, 2, 4]) - prefix_states = py_utils.NestedMap( - key=cached_source_vecs, value=cached_source_contexts) - for i in range(5): - h, probs, prefix_states = transformer.ExtendStep( - transformer.theta, source_vecs[i, :, :], prefix_states, aux_vecs, - aux_paddings) - h2.append(h) - probs2.append(probs) - - h2 = tf.stack(h2) - probs2 = tf.concat(probs2, 0) - - self.evaluate(tf.global_variables_initializer()) - h1_v, probs1_v, h2_v, probs2_v = self.evaluate([h1, probs1, h2, probs2]) - self.assertAllClose(h1_v, h2_v) - self.assertAllClose(probs1_v, probs2_v) - - def testMultiAuxSourceTransformerLayerExtendStep(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.TransformerLayer.Params() - p.name = 'transformer' - p.source_dim = depth - p.has_aux_atten = True - p.tr_aux_atten_tpl = ( - layers_with_attention.TransformerMultiSourceAttentionLayer.Params() - .Set( - source_dim=p.source_dim, - num_source=2, - primary_source_index=0, - num_attention_heads=4)) - p.mask_self_atten = True - p.tr_atten_tpl.num_attention_heads = 2 - transformer = layers_with_attention.TransformerLayer(p) - - (source_vecs, _, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputsMultiAuxSource( - ['source_0', 'source_1'], depth=depth) - source_padding = tf.zeros([5, 2]) - - h1, probs1 = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings) - - h2 = [] - probs2 = [] - cached_source_vecs = tf.zeros([0, 2, 4]) - cached_source_contexts = tf.zeros([0, 2, 4]) - prefix_states = py_utils.NestedMap( - key=cached_source_vecs, value=cached_source_contexts) - for i in range(5): - h, probs, prefix_states = transformer.ExtendStep( - transformer.theta, source_vecs[i, :, :], prefix_states, aux_vecs, - aux_paddings) - h2.append(h) - probs2.append(probs) - - h2 = tf.stack(h2) - probs2 = tf.concat(probs2, 0) - - self.evaluate(tf.global_variables_initializer()) - h1_v, probs1_v, h2_v, probs2_v = self.evaluate([h1, probs1, h2, probs2]) - self.assertAllClose(h1_v, h2_v) - self.assertAllClose(probs1_v, probs2_v) - - def testMultiAuxSourceTransformerLayerExtendStepMatchSingle(self): - with self.session(use_gpu=True): - # Prepare inputs - np.random.seed(6348575) - depth = 4 - (source_vecs, _, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputsMultiAuxSource( - ['source_0', 'source_1'], depth=depth) - - # Multi-source transformer layer - p = layers_with_attention.TransformerLayer.Params().Set( - name='multi_source_trans', random_seed=123) - p.tr_atten_tpl.num_attention_heads = 4 - p.source_dim = depth - p.has_aux_atten = True - p.tr_aux_atten_tpl = ( - layers_with_attention.TransformerMultiSourceAttentionLayer.Params() - .Set( - source_dim=p.source_dim, - num_source=1, - primary_source_index=0, - num_attention_heads=4)) - p.mask_self_atten = True - p.tr_fflayer_tpl.hidden_dim = 7 - msa_trans = layers_with_attention.TransformerLayer(p) - - h_msa = [] - probs_msa = [] - cached_source_vecs = tf.zeros([0, 2, 4]) - cached_source_contexts = tf.zeros([0, 2, 4]) - prefix_states = py_utils.NestedMap( - key=cached_source_vecs, value=cached_source_contexts) - for i in range(5): - h, probs, prefix_states = msa_trans.ExtendStep(msa_trans.theta, - source_vecs[i, :, :], - prefix_states, aux_vecs, - aux_paddings) - h_msa.append(h) - probs_msa.append(probs) - h_msa = tf.stack(h_msa) - probs_msa = tf.concat(probs_msa, 0) - - # Original single-source transformer decoder. - p = layers_with_attention.TransformerLayer.Params().Set( - name='single_source_trans', random_seed=123) - p.tr_atten_tpl.num_attention_heads = 4 - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_fflayer_tpl.hidden_dim = 7 - ssa_trans = layers_with_attention.TransformerLayer(p) - - h_ssa = [] - probs_ssa = [] - cached_source_vecs = tf.zeros([0, 2, 4]) - cached_source_contexts = tf.zeros([0, 2, 4]) - prefix_states = py_utils.NestedMap( - key=cached_source_vecs, value=cached_source_contexts) - for i in range(5): - h, probs, prefix_states = ssa_trans.ExtendStep(ssa_trans.theta, - source_vecs[i, :, :], - prefix_states, - aux_vecs['source_0'], - aux_paddings['source_0']) - h_ssa.append(h) - probs_ssa.append(probs) - h_ssa = tf.stack(h_ssa) - probs_ssa = tf.concat(probs_ssa, 0) - - self.evaluate(tf.global_variables_initializer()) - h_msa_v, h_ssa_v, probs_msa_v, probs_ssa_v = self.evaluate( - [h_msa, h_ssa, probs_msa, probs_ssa]) - tf.logging.info(np.array_repr(h_msa_v)) - tf.logging.info(np.array_repr(h_ssa_v)) - self.assertAllClose(h_msa_v, h_ssa_v) - self.assertAllClose(probs_msa_v, probs_ssa_v) - - def testTransformerLayerWithNgramMaskExtendStep(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.TransformerLayer.Params() - p.name = 'transformer' - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_atten_tpl.num_attention_heads = 2 - # Turn on N-gram masking in the TransformerLayer. - # Before doing so though copy the self-attention params to avoid - # the auxilliary attention being masked as well. - p.tr_aux_atten_tpl = p.tr_atten_tpl.Copy() - p.tr_atten_tpl.is_masked = True - p.tr_atten_tpl.mask_ngram_order = 3 - p.tr_atten_tpl.mask_type = 'ngram' - transformer = layers_with_attention.TransformerLayer(p) - - (source_vecs, _, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - source_padding = tf.zeros([5, 2]) - - h1, probs1 = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings) - - h2 = [] - probs2 = [] - cached_source_vecs = tf.zeros([0, 2, 4]) - cached_source_contexts = tf.zeros([0, 2, 4]) - prefix_states = py_utils.NestedMap( - key=cached_source_vecs, value=cached_source_contexts) - for i in range(5): - h, probs, prefix_states = transformer.ExtendStep( - transformer.theta, source_vecs[i, :, :], prefix_states, aux_vecs, - aux_paddings) - h2.append(h) - probs2.append(probs) - - h2 = tf.stack(h2) - probs2 = tf.concat(probs2, 0) - - self.evaluate(tf.global_variables_initializer()) - h1_v, probs1_v, h2_v, probs2_v = self.evaluate([h1, probs1, h2, probs2]) - self.assertAllClose(h1_v, h2_v) - self.assertAllClose(probs1_v, probs2_v) - - def testTransformerLayerWithPostLayernormExtendStep(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.TransformerLayer.Params() - p.name = 'transformer' - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_atten_tpl.num_attention_heads = 2 - p.tr_post_ln_tpl = layers.LayerNorm.Params() - transformer = layers_with_attention.TransformerLayer(p) - - (source_vecs, _, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - source_padding = tf.zeros([5, 2]) - - h1, probs1 = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings) - - h2 = [] - probs2 = [] - cached_source_vecs = tf.zeros([0, 2, 4]) - cached_source_contexts = tf.zeros([0, 2, 4]) - prefix_states = py_utils.NestedMap( - key=cached_source_vecs, value=cached_source_contexts) - for i in range(5): - h, probs, prefix_states = transformer.ExtendStep( - transformer.theta, source_vecs[i, :, :], prefix_states, aux_vecs, - aux_paddings) - h2.append(h) - probs2.append(probs) - - h2 = tf.stack(h2) - probs2 = tf.concat(probs2, 0) - - self.evaluate(tf.global_variables_initializer()) - h1_v, probs1_v, h2_v, probs2_v = self.evaluate([h1, probs1, h2, probs2]) - self.assertAllClose(h1_v, h2_v) - self.assertAllClose(probs1_v, probs2_v) - - def testEvolvedTransformerEncoderBranchedConvsLayer(self): - layer = layers_with_attention.EvolvedTransformerEncoderBranchedConvsLayer - with self.session(use_gpu=True): - tf.random.set_seed(3980847392) - inputs = tf.random.normal([5, 2, 3], seed=948387483) - paddings = tf.zeros([5, 2]) - p = layer.Params() - p.name = 'et_encoder_branched_convs' - p.input_dim = 3 - et_branched_convs = layer(p) - - h = et_branched_convs.FPropDefaultTheta(inputs, paddings) - self.evaluate(tf.global_variables_initializer()) - actual_layer_output = self.evaluate(h) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [[-0.13232423, -0.46060669, 0.72598207], - [ 0.6725747 , 1.58664441, 2.64087844]], - [[-0.21702465, -0.68267912, 1.20886588], - [ 1.69793618, 0.53306532, 1.02958691]], - [[-0.46037287, -0.42950529, -1.68443251], - [ 0.21459752, 0.42246291, -0.01271994]], - [[-0.23293658, 0.15300342, -0.83518255], - [-0.48914853, -0.44239512, -0.2328119 ]], - [[-0.57934833, 0.24165238, -1.05392623], - [-0.8292231 , 0.06175411, 1.28672981]]] - # pyformat: enable - # pylint: enable=bad-whitespace - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testEvolvedTransformerDecoderBranchedConvsLayer(self): - layer = layers_with_attention.EvolvedTransformerDecoderBranchedConvsLayer - with self.session(use_gpu=True): - tf.random.set_seed(3980847392) - inputs = tf.random.normal([5, 2, 3], seed=948387483) - paddings = tf.zeros([5, 2]) - p = layer.Params() - p.name = 'et_decoder_branched_convs' - p.input_dim = 3 - et_branched_convs = layer(p) - - h = et_branched_convs.FPropDefaultTheta(inputs, paddings) - self.evaluate(tf.global_variables_initializer()) - actual_layer_output = self.evaluate(h) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [[-0.31987068, -0.65715098, 0.90350437], - [ 0.00773269, 1.07779562, 4.11094666]], - [[-0.84862059, -0.93186408, 1.16371167], - [ 1.31467259, 0.03560367, 2.36822462]], - [[ 0.02183507, -0.0799394 , -1.68870354], - [ 0.77921551, 1.30145741, -0.86353606]], - [[ 0.31672907, 0.50000876, -0.93973017], - [-0.54707348, 0.19211179, -1.45307386]], - [[-0.46405494, 0.65833056, -1.09345317], - [-1.17221224, -0.08027397, 0.84021652]]] - # pyformat: enable - # pylint: enable=bad-whitespace - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testEvolvedTransformerEncoderLayerConstruction(self): - p = layers_with_attention.EvolvedTransformerEncoderLayer.Params() - p.name = 'evolved_transformer_encoder' - p.source_dim = 4 - p.transformer_tpl.tr_fflayer_tpl.hidden_dim = 7 - p.transformer_tpl.tr_atten_tpl.num_attention_heads = 2 - _ = layers_with_attention.EvolvedTransformerEncoderLayer(p) - - def testEvolvedTransformerEncoderLayerFProp(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.EvolvedTransformerEncoderLayer.Params() - p.name = 'evolved_transformer_encoder' - p.source_dim = depth - p.transformer_tpl.tr_atten_tpl.num_attention_heads = 2 - transformer = layers_with_attention.EvolvedTransformerEncoderLayer(p) - - (source_vecs, source_padding, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - - h, probs = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings) - - self.evaluate(tf.global_variables_initializer()) - actual_layer_output, actual_prob_output = self.evaluate([h, probs]) - tf.logging.info(np.array_repr(actual_layer_output)) - tf.logging.info(np.array_repr(actual_prob_output)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_layer_output = [ - [[-1.69196141e+00, -5.03859818e-01, 2.42652583e+00, -1.47606134e+00], - [-1.27175665e+00, -1.81504273e+00, 7.16031432e-01, 1.40093648e+00]], - [[-1.83027089e-02, 8.73535872e-04, 2.04444170e+00, -2.74493122e+00], - [-6.90074801e-01, -2.06015229e-01, -1.21154499e+00, 1.17104244e+00]], - [[1.84662449e+00, 6.33037746e-01, -2.02951849e-01, -1.70747042e+00], - [-9.46833491e-01, -3.77074480e-01, -8.77807617e-01, 1.56822240e+00]], - [[-7.63426960e-01, -3.30413729e-01, 2.38541245e+00, -8.12596083e-01], - [1.91653550e+00, -1.47251439e+00, -2.19180465e+00, 5.32990336e-01]], - [[-1.46879780e+00, 5.38376629e-01, 1.50257730e+00, -8.52106392e-01], - [-1.55632758e+00, -3.48120153e-01, -9.21136498e-01, 2.02043033e+00]]] - expected_prob_output = [ - [[ 0.27934468, 0.28112975, 0., 0., 0.43952557], - [ 0., 0.24881637, 0.25068569, 0.50049794, 0. ]], - [[ 0.32433772, 0.33424711, 0., 0., 0.34141517], - [ 0., 0.33490175, 0.29024804, 0.37485027, 0. ]], - [[ 0.38673952, 0.33638299, 0., 0., 0.27687752], - [ 0., 0.30134204, 0.25906932, 0.43958867, 0. ]], - [[ 0.30161232, 0.33053303, 0., 0., 0.36785468], - [ 0., 0.36827549, 0.39080781, 0.24091662, 0. ]], - [[ 0.21496946, 0.22309673, 0., 0., 0.56193388], - [ 0., 0.27062884, 0.23510428, 0.4942669, 0. ]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_layer_output, actual_layer_output) - self.assertAllClose(expected_prob_output, actual_prob_output) - - def testEvolvedTransformerDecoderLayerConstruction(self): - p = layers_with_attention.EvolvedTransformerDecoderLayer.Params() - p.name = 'evolved_transformer_decoder' - p.source_dim = 16 - p.transformer_tpl.tr_atten_tpl.num_attention_heads = 2 - p.has_aux_atten = True - p.mask_self_atten = True - _ = layers_with_attention.EvolvedTransformerDecoderLayer(p) - - def testEvolvedTransformerDecoderLayerFProp(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.EvolvedTransformerDecoderLayer.Params() - p.name = 'evolved_transformer_decoder' - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_double_heads_atten_tpl.num_attention_heads = 2 - p.tr_atten_tpl.num_attention_heads = 2 - p.transformer_tpl.tr_atten_tpl.num_attention_heads = 2 - transformer = layers_with_attention.EvolvedTransformerDecoderLayer(p) - - (source_vecs, source_padding, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - - h, probs = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings) - - self.evaluate(tf.global_variables_initializer()) - actual_layer_output, actual_prob_output = self.evaluate([h, probs]) - tf.logging.info(np.array_repr(actual_layer_output)) - tf.logging.info(np.array_repr(actual_prob_output)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_layer_output =[ - [[-2.15844011, 0.54941475, 1.01636434, 0.13751738], - [-1.31499887, -0.9501676, 0.874282, 0.58270419]], - [[-0.49268177, 2.71167898, -0.78087997, 0.43936318], - [-1.11428595, -1.38933206, 0.34404463, 0.43363893]], - [[ 0.57303172, 0.42080224, -0.50416583, -1.36097562], - [-1.26460135, -1.21081781, 0.9377467, 0.03642488]], - [[-1.52767372, -0.93615997, 1.33185053, 0.24640131], - [ 0.16062447, 2.39912128, 0.1896024, -0.70986807]], - [[-1.27725732, -1.51283062, 0.26704332, 0.65503371], - [-1.64287043, -0.30310085, -0.36987182, 1.57325172]]] - expected_prob_output = [ - [[0.28604817, 0., 0.24327257, 0., 0.26117378, 0., 0.20950545], - [0., 0.26639479, 0., 0.38120365, 0., 0.35240155, 0.]], - [[0.24309734, 0., 0.24040565, 0., 0.22922358, 0., 0.2872735], - [0., 0.27082229, 0., 0.36431897, 0., 0.36485875, 0.]], - [[0.25640261, 0., 0.25117433, 0., 0.25067171, 0., 0.24175137], - [0., 0.27037328, 0., 0.38163245, 0., 0.34799421, 0.]], - [[0.27474535, 0., 0.25523224, 0., 0.27800021, 0., 0.19202216], - [0., 0.34553668, 0., 0.35240823, 0., 0.30205506, 0.]], - [[0.24020916, 0., 0.25431803, 0., 0.26219654, 0., 0.24327625], - [0., 0.30723149, 0., 0.32563132, 0., 0.36713719, 0.]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_layer_output, actual_layer_output) - self.assertAllClose(expected_prob_output, actual_prob_output) - - def testEvolvedTransformerDecoderLayerExtendStep(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.EvolvedTransformerDecoderLayer.Params() - p.name = 'evolved_transformer_decoder' - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_double_heads_atten_tpl.num_attention_heads = 2 - p.tr_atten_tpl.num_attention_heads = 2 - p.transformer_tpl.tr_atten_tpl.num_attention_heads = 2 - et_decoder = layers_with_attention.EvolvedTransformerDecoderLayer(p) - - (source_vecs, _, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - source_padding = tf.zeros([5, 2]) - - h1, probs1 = et_decoder.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings) - - h2 = [] - probs2 = [] - - double_head_attention_states = py_utils.NestedMap( - key=tf.zeros([0, 2, 4]), value=tf.zeros([0, 2, 4])) - transformer_layer_states = py_utils.NestedMap( - key=tf.zeros([0, 2, 4]), value=tf.zeros([0, 2, 4])) - branched_convs_input = tf.zeros([0, 2, 4]) - - prefix_states = py_utils.NestedMap( - double_head_attention_states=double_head_attention_states, - transformer_layer_states=transformer_layer_states, - branched_convs_input=branched_convs_input) - - for i in range(5): - h, probs, prefix_states = et_decoder.ExtendStep(et_decoder.theta, - source_vecs[i, :, :], - prefix_states, aux_vecs, - aux_paddings) - h2.append(h) - probs2.append(probs) - - h2 = tf.stack(h2) - probs2 = tf.concat(probs2, 0) - - self.evaluate(tf.global_variables_initializer()) - h1_v, probs1_v, h2_v, probs2_v = self.evaluate([h1, probs1, h2, probs2]) - self.assertAllClose(h1_v, h2_v) - self.assertAllClose(probs1_v, probs2_v) - - def testStyleLayer(self): - with self.session(use_gpu=False): - p = layers_with_attention.StyleLayer.Params().Set( - name='style_layer', - input_dim=10, - output_dim=8, - num_styles=16, - random_seed=28384) - - tf.random.set_seed(8372749040) - np.random.seed(12345) - sl = p.Instantiate() - features = tf.random.normal([2, 10], seed=28384) - latent, atten_probs = sl.FPropDefaultTheta(features) - self.evaluate(tf.global_variables_initializer()) - latent_v, atten_probs_v = self.evaluate([latent, atten_probs]) - CompareToGoldenSingleFloat(self, -1.208686, np.sum(latent_v)) - CompareToGoldenSingleFloat(self, 2.0, np.sum(atten_probs_v)) - - def testStyleLayerWithFeedinAttenProbs(self): - with self.session(use_gpu=False): - p = layers_with_attention.StyleLayer.Params().Set( - name='style_layer', - input_dim=10, - output_dim=8, - num_styles=16, - num_heads=4, - enable_ctx_post_proj=False, - random_seed=28384) - - tf.random.set_seed(8372749040) - np.random.seed(12345) - sl = p.Instantiate() - atten_probs = tf.constant([[1.0] + [0.0] * 15] * 2, dtype=tf.float32) - ids = tf.constant([0, 0], dtype=tf.int32) - latent_from_probs = sl.StyleEmbFromProbs(sl.theta, atten_probs) - latent_from_lookup = sl.EmbLookup(sl.theta, ids) - self.evaluate(tf.global_variables_initializer()) - latent_p, latent_l = self.evaluate( - [latent_from_probs, latent_from_lookup]) - self.assertAllClose(latent_p, latent_l) - - def testStyleLayer02(self): - with self.session(use_gpu=False): - p = layers_with_attention.StyleLayer.Params().Set( - name='style_layer', - input_dim=10, - output_dim=8, - num_styles=16, - random_seed=72738) - tf.random.set_seed(8372749040) - np.random.seed(12345) - sl = p.Instantiate() - features = tf.random.normal([2, 10]) - features = tf.concat([features, features], 0) - latent, _ = sl.FPropDefaultTheta(features) - self.evaluate(tf.global_variables_initializer()) - latent_v = self.evaluate(latent) - # Makes sure identical input results in identical style output. - self.assertAllClose(latent_v[:2], latent_v[2:]) - - def _testTransformerMultitaskLayerInputs(self, depth=3, dtype=tf.float32): - np.random.seed(505837249) - source_vecs = tf.stack( - [tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(5)]) - source_padding = tf.transpose( - tf.constant([[0, 0, 1, 1, 0], [1, 0, 0, 0, 1]], dtype=dtype)) - aux_source_vecs = tf.stack( - [tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(7)]) - aux_source_paddings = tf.transpose( - tf.constant([[0, 1, 0, 1, 0, 1, 0], [1, 0, 1, 0, 1, 0, 1]], - dtype=dtype)) - source_task_id = tf.constant([[2, 3]], dtype=tf.int32) - return (source_vecs, source_padding, aux_source_vecs, aux_source_paddings, - source_task_id) - - def testTransformerLayerWithMultitaskAdaptersConstruction(self): - p = layers_with_attention.TransformerLayerWithMultitaskAdapters.Params() - p.name = 'transformer_with_adapters' - p.source_dim = 4 - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_attention_heads = 2 - p.has_aux_atten = True - p.mask_self_atten = True - p.adapter_tpl.input_dim = 4 - p.adapter_tpl.num_tasks = 4 - p.adapter_tpl.bottleneck_dim = 2 - _ = layers_with_attention.TransformerLayerWithMultitaskAdapters(p) - - def testTransformerLayerWithMultitaskAdaptersFProp(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.TransformerLayerWithMultitaskAdapters.Params() - p.name = 'transformer' - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_attention_heads = 2 - p.adapter_tpl.input_dim = 4 - p.adapter_tpl.num_tasks = 4 - p.adapter_tpl.bottleneck_dim = 2 - transformer = layers_with_attention.TransformerLayerWithMultitaskAdapters( - p) - - (source_vecs, source_padding, aux_vecs, aux_paddings, - source_task_id) = self._testTransformerMultitaskLayerInputs(depth=depth) - - h, probs = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings, - source_task_id=source_task_id) - - self.evaluate(tf.global_variables_initializer()) - actual_layer_output, actual_prob_output = self.evaluate([h, probs]) - tf.logging.info(np.array_repr(actual_layer_output)) - tf.logging.info(np.array_repr(actual_prob_output)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_layer_output = [ - [[ 0.02441728, 0.26923186, 0.68582684, 1.1531992 ], - [ 0.69027936, -1.94770098, 2.00558615, 0.17057157]], - [[ 1.81022859, 2.37042093, 0.03620988, -0.32401592], - [ 1.66707945, -1.95131969, 0.64937419, 0.05853128]], - [[ 1.53475547, -0.60239077, -0.05797344, -0.48760295], - [ 1.53514266, -2.1231215 , 0.98074663, -0.5577352 ]], - [[-1.32504404, -1.28702664, 2.597996 , 0.24809647], - [ 3.7842629 , -1.46549737, 0.91363102, -2.37071466]], - [[ 0.52196532, -0.73371518, 0.86030912, 0.33838278], - [ 0.01923725, -0.8887378 , 1.08245265, 1.19935369]] - ] - expected_prob_output = [ - [[ 0.21795765, 0, 0.26612395, 0, 0.31251645, 0, 0.20340192], - [ 0, 0.2677784 , 0, 0.32895881, 0, 0.40326279, 0]], - [[ 0.25721508, 0, 0.24116732, 0, 0.25138181, 0, 0.2502358 ], - [ 0, 0.25691482, 0, 0.31076014, 0, 0.43232504, 0]], - [[ 0.24550268, 0, 0.25128055, 0, 0.25109866, 0, 0.25211811], - [ 0, 0.26769164, 0, 0.32481131, 0, 0.40749705, 0]], - [[ 0.22675318, 0, 0.26633731, 0, 0.28919035, 0, 0.21771917], - [ 0, 0.35955882, 0, 0.36869821, 0, 0.271743 , 0]], - [[ 0.21504655, 0, 0.26958644, 0, 0.30847484, 0, 0.20689213], - [ 0, 0.29516917, 0, 0.29359812, 0, 0.41123268, 0]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_layer_output, actual_layer_output) - self.assertAllClose(expected_prob_output, actual_prob_output) - - def testTransformerLayerWithMultitaskAdaptersWithInputPackingFProp(self): - with self.session(use_gpu=True): - with tf.variable_scope('transformer_packed_test', reuse=tf.AUTO_REUSE): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.TransformerLayerWithMultitaskAdapters.Params() - p.name = 'transformer_with_adapters' - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_attention_heads = 2 - p.adapter_tpl.input_dim = 4 - p.adapter_tpl.num_tasks = 4 - p.adapter_tpl.bottleneck_dim = 2 - packed_params = p.Copy() - transformer = layers_with_attention.TransformerLayerWithMultitaskAdapters( - p) - packed_params.packed_input = True - transformer_packed = layers_with_attention.TransformerLayerWithMultitaskAdapters( - packed_params) - - dtype = tf.float32 - source_vecs = tf.stack([ - tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(5) - ]) - source_padding = tf.transpose( - tf.constant([[0, 0, 0, 0, 1], [0, 0, 0, 0, 0]], dtype=dtype)) - aux_vecs = tf.stack([ - tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(7) - ]) - aux_paddings = tf.transpose( - tf.constant([[0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1, 1]], - dtype=dtype)) - source_task_id = tf.constant([[2, 3]], dtype=tf.int32) - - source_vecs_packed = tf.reshape(source_vecs, [-1, 1, depth]) - aux_vecs_packed = tf.reshape(aux_vecs, [-1, 1, depth]) - source_padding_packed = tf.reshape(source_padding, [-1, 1]) - aux_padding_packed = tf.reshape(aux_paddings, [-1, 1]) - source_task_id_packed = tf.transpose( - tf.constant([[2, 3, 2, 3, 2, 3, 2, 3, 2, 3]], dtype=tf.int32)) - source_segment_id = tf.transpose( - tf.constant([[0, 1, 0, 1, 0, 1, 0, 1, 0, 1]], dtype=tf.float32)) - aux_segment_id = tf.transpose( - tf.constant([[0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1]], - dtype=tf.float32)) - - h, _ = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings, - source_segment_id=None, - aux_segment_id=None, - source_task_id=source_task_id) - - h_packed, _ = transformer_packed.FPropDefaultTheta( - source_vecs_packed, - source_padding_packed, - aux_vecs=aux_vecs_packed, - aux_paddings=aux_padding_packed, - source_segment_id=source_segment_id, - aux_segment_id=aux_segment_id, - source_task_id=source_task_id_packed) - h_packed = tf.reshape(h_packed, tf.shape(h)) - - self.evaluate(tf.global_variables_initializer()) - actual_layer, p_layer = self.evaluate([h, h_packed]) - self.assertAllClose(actual_layer, p_layer) - - def testTransformerLayerWithMultitaskAdaptersExtendStep(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.TransformerLayerWithMultitaskAdapters.Params() - p.name = 'transformer' - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_atten_tpl.num_attention_heads = 2 - p.adapter_tpl.input_dim = 4 - p.adapter_tpl.num_tasks = 4 - p.adapter_tpl.bottleneck_dim = 2 - transformer = layers_with_attention.TransformerLayerWithMultitaskAdapters( - p) - - (source_vecs, _, aux_vecs, aux_paddings, - source_task_id) = self._testTransformerMultitaskLayerInputs(depth=depth) - source_padding = tf.zeros([5, 2]) - - h1, probs1 = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings, - source_task_id=source_task_id) - - h2 = [] - probs2 = [] - cached_source_vecs = tf.zeros([0, 2, 4]) - cached_source_contexts = tf.zeros([0, 2, 4]) - prefix_states = py_utils.NestedMap( - key=cached_source_vecs, value=cached_source_contexts) - for i in range(5): - h, probs, prefix_states = transformer.ExtendStep( - transformer.theta, - source_vecs[i, :, :], - prefix_states, - aux_vecs, - aux_paddings, - source_task_id=source_task_id[0, :]) - h2.append(h) - probs2.append(probs) - - h2 = tf.stack(h2) - probs2 = tf.concat(probs2, 0) - - self.evaluate(tf.global_variables_initializer()) - h1_v, probs1_v, h2_v, probs2_v = self.evaluate([h1, probs1, h2, probs2]) - self.assertAllClose(h1_v, h2_v) - self.assertAllClose(probs1_v, probs2_v) - - def testCCTFeedForwardLayerConstruction(self): - p = layers_with_attention.CCTFeedForwardLayer.Params() - p.name = 'cct_fflayer_1' - p.input_dim = 3 - p.hidden_dim = 7 - p.num_blocks = 2 - p.gating_tpl.hidden_layer_dim = 2 - p.gating_tpl.noise_std = 5.0 - p.gating_tpl.noise_warmup_steps = 100 - _ = layers_with_attention.CCTFeedForwardLayer(p) - - def testCCTFeedForwardLayerTraining(self): - with self.session(use_gpu=True): - tf.random.set_seed(3980847392) - inputs = tf.random.normal([5, 2, 3], seed=948387483) - paddings = tf.zeros([5, 2]) - p = layers_with_attention.CCTFeedForwardLayer.Params() - p.name = 'transformer_fflayer' - p.input_dim = 3 - p.hidden_dim = 7 - p.num_blocks = 2 - p.gating_tpl.hidden_layer_dim = 2 - p.gating_tpl.noise_std = 5.0 - p.gating_tpl.noise_warmup_steps = 100 - cct_fflayer = layers_with_attention.CCTFeedForwardLayer(p) - - h, p_c = cct_fflayer.FPropDefaultTheta(inputs, paddings) - self.evaluate(tf.global_variables_initializer()) - actual_layer_output, p_c_val = self.evaluate([h, p_c]) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [[ 0.49714983, -1.1684668 , 0.4889576 ], - [ 1.7869478 , 1.4456576 , 1.4123362 ]], - [[ 0.10564739, -1.5359519 , 0.67742175], - [ 1.6211604 , 0.583192 , 1.056936 ]], - [[-0.01121134, -0.78554434, -0.84111285], - [ 0.45078042, 0.63005054, 0.08024757]], - [[ 0.162924 , 0.14500974, -0.32797086], - [ 0.41885388, -0.5852693 , -1.7245001 ]], - [[-0.6601118 , 0.30835745, -0.48543385], - [-0.04813027, -0.04633661, -0.21723843]]] - expected_p_c = [ - [[0.5607947 , 0.49624035], - [0.72082597, 0.50216115]], - [[0.6352798 , 0.49843985], - [0.5 , 0.5 ]], - [[0.5 , 0.5 ], - [0.5 , 0.5 ]], - [[0.5 , 0.5 ], - [0.7562946 , 0.50510687]], - [[0.62267053, 0.50738835], - [0.73273706, 0.5029184 ]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(actual_layer_output, expected_output) - self.assertAllClose(p_c_val, expected_p_c) - - def testCCTFeedForwardLayerInference(self): - with self.session(use_gpu=True), self.SetEval(True): - tf.random.set_seed(3980847392) - inputs = tf.random.normal([5, 2, 3], seed=948387483) - paddings = tf.zeros([5, 2]) - p = layers_with_attention.CCTFeedForwardLayer.Params() - p.name = 'transformer_fflayer' - p.input_dim = 3 - p.hidden_dim = 7 - p.num_blocks = 2 - p.gating_tpl.hidden_layer_dim = 2 - p.gating_tpl.noise_std = 5.0 - p.gating_tpl.noise_warmup_steps = 100 - cct_fflayer = layers_with_attention.CCTFeedForwardLayer(p) - - h, p_c = cct_fflayer.FPropDefaultTheta(inputs, paddings) - self.evaluate(tf.global_variables_initializer()) - actual_layer_output, p_c_val = self.evaluate([h, p_c]) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_output = [ - [[ 1.1921753 , -0.78980637, -0.58472836], - [ 2.5051842 , 1.6491661 , 0.49059153]], - [[ 0.6877271 , -1.1452659 , -0.29534382], - [ 1.5774723 , 0.6462606 , 1.0375552 ]], - [[ 0.12175584, -1.2262938 , -0.5333306 ], - [ 0.4632102 , 0.7119628 , -0.01409443]], - [[ 0.16090955, 0.06721614, -0.24816278], - [ 0.9799552 , -0.2861529 , -2.5847178 ]], - [[-0.48719 , 0.18763718, -0.53763545], - [ 0.5886377 , 0.21293162, -1.1132748 ]] - ] - expected_p_c = [ - [[1., 0.], - [1., 1.]], - [[1., 0.], - [1., 1.]], - [[1., 1.], - [1., 1.]], - [[1., 1.], - [1., 1.]], - [[1., 1.], - [1., 1.]] - ] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(actual_layer_output, expected_output) - self.assertAllClose(p_c_val, expected_p_c) - - def testTransformerWithContextLayerConstruction(self): - p = layers_with_attention.TransformerWithContextLayer.Params() - p.name = 'transformer_1' - p.source_dim = 4 - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_attention_heads = 2 - layer = p.Instantiate() - # output_dim is equal to source_dim when p.output_dim == 0 - self.assertEqual(0, p.output_dim) - self.assertEqual(p.source_dim, layer.fflayer.output_dim) - - def testTransformerWithContextLayerFProp(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.TransformerWithContextLayer.Params() - p.name = 'transformer' - p.source_dim = depth - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_attention_heads = 2 - transformer = p.Instantiate() - - (source_vecs, source_padding, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputs(depth) - - h, probs = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings, - tertiary_vecs=aux_vecs, - tertiary_paddings=aux_paddings) - - self.evaluate(tf.global_variables_initializer()) - actual_layer_output, actual_prob_output = self.evaluate([h, probs]) - tf.logging.info(np.array_repr(actual_layer_output)) - tf.logging.info(np.array_repr(actual_prob_output)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_layer_output = [ - [[ 0.55129296, -0.7571765 , 0.281192 , 0.8710322 ], - [ 0.5072957 , -1.3714458 , 1.5689826 , -0.0971924 ]], - [[ 2.2560897 , 2.7890472 , 0.016873 , -0.5172725 ], - [ 1.4128124 , -2.0595124 , 0.37241971, -0.6075135 ]], - [[ 2.57011 , -0.8678784 , -0.33203793, -0.18508816], - [ 1.3549538 , -2.0990794 , 0.62103236, -0.9975941 ]], - [[ 0.15144205, -1.1681134 , 1.7113727 , 0.4682465 ], - [ 2.9454587 , -1.4413761 , 0.5215157 , -2.1541023 ]], - [[ 1.5092299 , -1.7608491 , 0.21144068, 0.22785848], - [-0.766488 , -0.487573 , 1.0574573 , 0.81118184]]] - expected_prob_output = [ - [[0.223735 , 0. , 0.26685917, 0. , 0.2968173 , - 0. , 0.2125885 ], - [0. , 0.28585374, 0. , 0.35088098, 0. , - 0.36326528, 0. ]], - [[0.2703818 , 0. , 0.23092957, 0. , 0.2249705 , - 0. , 0.27371815], - [0. , 0.26997963, 0. , 0.33745134, 0. , - 0.39256904, 0. ]], - [[0.25208434, 0. , 0.24830116, 0. , 0.23168065, - 0. , 0.26793382], - [0. , 0.2847324 , 0. , 0.3477454 , 0. , - 0.36752218, 0. ]], - [[0.23778549, 0. , 0.26169604, 0. , 0.26542395, - 0. , 0.23509452], - [0. , 0.3603859 , 0. , 0.37519425, 0. , - 0.26441985, 0. ]], - [[0.22522289, 0. , 0.26782405, 0. , 0.28599125, - 0. , 0.22096181], - [0. , 0.29979968, 0. , 0.31155068, 0. , - 0.38864967, 0. ]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_layer_output, actual_layer_output) - self.assertAllClose(expected_prob_output, actual_prob_output) - - def testTransformerWithContextLayerPackedInputFProp(self): - with self.session(use_gpu=True): - with tf.variable_scope('transformer_packed_test', reuse=tf.AUTO_REUSE): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.TransformerLayer.Params() - p.name = 'transformer' - p.source_dim = depth - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_attention_heads = 2 - transformer = p.Instantiate() - packed_params = p.Copy() - packed_params.packed_input = True - transformer_packed = packed_params.Instantiate() - - dtype = tf.float32 - source_vecs = tf.stack([ - tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(5) - ]) - source_padding = tf.transpose( - tf.constant([[0, 0, 0, 0, 1], [0, 0, 0, 0, 0]], dtype=dtype)) - aux_vecs = tf.stack([ - tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(7) - ]) - tertiary_vecs = tf.stack([ - tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(7) - ]) - aux_paddings = tf.transpose( - tf.constant([[0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1, 1]], - dtype=dtype)) - - source_vecs_packed = tf.reshape(source_vecs, [-1, 1, depth]) - aux_vecs_packed = tf.reshape(aux_vecs, [-1, 1, depth]) - tertiary_vecs_packed = tf.reshape(tertiary_vecs, [-1, 1, depth]) - source_padding_packed = tf.reshape(source_padding, [-1, 1]) - aux_padding_packed = tf.reshape(aux_paddings, [-1, 1]) - source_segment_id = tf.transpose( - tf.constant([[0, 1, 0, 1, 0, 1, 0, 1, 0, 1]], dtype=tf.float32)) - aux_segment_id = tf.transpose( - tf.constant([[0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1]], - dtype=tf.float32)) - - h, _ = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings, - tertiary_vecs=tertiary_vecs, - tertiary_paddings=aux_paddings) - - h_packed, _ = transformer_packed.FPropDefaultTheta( - source_vecs_packed, - source_padding_packed, - aux_vecs=aux_vecs_packed, - aux_paddings=aux_padding_packed, - source_segment_id=source_segment_id, - aux_segment_id=aux_segment_id, - tertiary_vecs=tertiary_vecs_packed, - tertiary_paddings=aux_padding_packed, - tertiary_segment_id=aux_segment_id) - h_packed = tf.reshape(h_packed, tf.shape(h)) - - self.evaluate(tf.global_variables_initializer()) - actual_layer, p_layer = self.evaluate([h, h_packed]) - self.assertAllClose(actual_layer, p_layer) - - def testTransformerWithContextLayerExtendStep(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = layers_with_attention.TransformerWithContextLayer.Params() - p.name = 'transformer' - p.source_dim = depth - p.tr_atten_tpl.num_attention_heads = 2 - transformer = p.Instantiate() - - (source_vecs, source_padding, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputs(depth) - source_padding = tf.zeros([5, 2]) - - h1, probs1 = transformer.FPropDefaultTheta( - source_vecs, - source_padding, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings, - tertiary_vecs=aux_vecs, - tertiary_paddings=aux_paddings) - - h2 = [] - probs2 = [] - cached_source_vecs = tf.zeros([0, 2, 4]) - cached_source_contexts = tf.zeros([0, 2, 4]) - prefix_states = py_utils.NestedMap( - key=cached_source_vecs, value=cached_source_contexts) - for i in range(5): - h, probs, prefix_states = transformer.ExtendStep( - transformer.theta, - source_vecs[i, :, :], - prefix_states, - aux_vecs, - aux_paddings, - tertiary_vecs=aux_vecs, - tertiary_paddings=aux_paddings) - h2.append(h) - probs2.append(probs) - - h2 = tf.stack(h2) - probs2 = tf.concat(probs2, 0) - - self.evaluate(tf.global_variables_initializer()) - h1_v, probs1_v, h2_v, probs2_v = self.evaluate([h1, probs1, h2, probs2]) - self.assertAllClose(h1_v, h2_v) - self.assertAllClose(probs1_v, probs2_v) - - def testCCTAttentionLayerSelfAttentionTraining(self): - with self.session(use_gpu=True) as sess: - depth = 4 - p = layers_with_attention.CCTAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = True - p.num_attention_heads = 2 - p.gating_tpl.hidden_layer_dim = 2 - p.gating_tpl.noise_std = 5.0 - p.gating_tpl.noise_warmup_steps = 100 - transformer_atten = layers_with_attention.CCTAttentionLayer(p) - - (source_vecs, source_padding, _, _, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - - ctx, probs, qpc, spc = transformer_atten.FPropDefaultTheta( - source_vecs, source_padding) - tf.global_variables_initializer().run() - actual_ctx, actual_probs, actual_qpc, actual_spc = sess.run( - [ctx, probs, qpc, spc]) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [[-0.9170906 , 0.89127994, 0.8682031 , -0.8423924 ], - [-1.2874005 , -0.76474655, 0.5771928 , 1.4749541 ]], - [[ 0.34465155, 0.74996084, -0.48622286, -0.6083897 ], - [-0.7486481 , -0.07628638, -0.99187833, 1.8168143 ]], - [[ 1.6986014 , -0.44173932, -0.7130059 , -0.5438557 ], - [-1.3927674 , -0.09861529, 0.3361559 , 1.1552272 ]], - [[-0.5439662 , -1.0707575 , 1.8813989 , -0.26667514], - [ 1.1484473 , 0.9964316 , -1.2344118 , -0.91046673]], - [[-0.06898946, -1.5815425 , -0.45298773, 2.1035194 ], - [-1.7475295 , 0.27231437, -0.8034381 , 2.2786536 ]]] - expected_probs = [ - [[1. , 0. , 0. , 0. , 0. ], - [0.2 , 0.2 , 0.2 , 0.2 , 0.2 ]], - [[0.4238176 , 0.57618237, 0. , 0. , 0. ], - [0. , 1. , 0. , 0. , 0. ]], - [[0.34105754, 0.65894246, 0. , 0. , 0. ], - [0. , 0.55719167, 0.44280833, 0. , 0. ]], - [[0.6528083 , 0.34719166, 0. , 0. , 0. ], - [0. , 0.32477915, 0.36445653, 0.31076428, 0. ]], - [[0.28325003, 0.21873125, 0. , 0. , 0.49801874], - [0. , 0.43867606, 0.2793855 , 0.28193837, 0. ]]] - expected_qpc = [ - [[0.5 ], - [0.5818492 ]], - [[0.5411409 ], - [0.55023897]], - [[0.56948507], - [0.5499979 ]], - [[0.5166038 ], - [0.58645904]], - [[0.54155153], - [0.5 ]]] - expected_spc = [ - [[0.21472901], - [0.06997871]], - [[0.53207266], - [0.39812705]], - [[0.5217048 ], - [0.07829338]], - [[0.06743541], - [0.5 ]], - [[0.32987863], - [0.5442441 ]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_probs, actual_probs, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_qpc, actual_qpc, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_spc, actual_spc, rtol=1e-05, atol=1e-05) - - def testCCTAttentionLayerSelfAttentionEval(self): - with self.session(use_gpu=True) as sess, self.SetEval(True): - depth = 4 - p = layers_with_attention.CCTAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = True - p.num_attention_heads = 2 - p.gating_tpl.hidden_layer_dim = 2 - p.gating_tpl.noise_std = 5.0 - p.gating_tpl.noise_warmup_steps = 100 - transformer_atten = layers_with_attention.CCTAttentionLayer(p) - - (source_vecs, source_padding, _, _, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - - ctx, probs, qpc, spc = transformer_atten.FPropDefaultTheta( - source_vecs, source_padding) - tf.global_variables_initializer().run() - actual_ctx, actual_probs, actual_qpc, actual_spc = sess.run( - [ctx, probs, qpc, spc]) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [[-1.5939784e+00, 8.5430717e-01, 8.4722424e-01, -1.0755297e-01], - [-1.6199683e+00, -1.9144357e+00, 1.0950426e+00, 2.4393613e+00]], - [[ 2.0492536e-01, -5.0217152e-02, -1.5521961e-01, 5.1122904e-04], - [-4.3141130e-01, -9.0650195e-01, -3.5488802e-01, 1.6928028e+00]], - [[ 1.7034934e+00, -1.1774492e+00, -4.2603785e-01, -1.0000569e-01], - [-1.0880733e+00, -9.0783793e-01, 9.9768031e-01, 9.9823117e-01]], - [[-1.1584746e+00, -2.0163212e+00, 2.3776212e+00, 7.9717481e-01], - [ 1.3303024e+00, -1.4763023e+00, 2.6441175e-01, -1.1841190e-01]], - [[-3.0323851e-01, -2.5461116e+00, 5.0698155e-01, 2.3423686e+00], - [-2.0771229e+00, -8.0027932e-01, -7.4258000e-02, 2.9516606e+00]]] - expected_probs = [ - [[1. , 0. , 0. , 0. , 0. ], - [0.2 , 0.2 , 0.2 , 0.2 , 0.2 ]], - [[0.35538384, 0.6446162 , 0. , 0. , 0. ], - [0. , 1. , 0. , 0. , 0. ]], - [[0.18125553, 0.8187444 , 0. , 0. , 0. ], - [0. , 0.5 , 0.5 , 0. , 0. ]], - [[0.7752405 , 0.22475953, 0. , 0. , 0. ], - [0. , 0.36166608, 0.36166608, 0.27666792, 0. ]], - [[0.40603536, 0.18792923, 0. , 0. , 0.40603536], - [0. , 0.32476988, 0.32476988, 0.35046023, 0. ]]] - expected_qpc = [ - [[1.], - [1.]], - [[1.], - [1.]], - [[1.], - [1.]], - [[1.], - [1.]], - [[1.], - [1.]]] - expected_spc = [ - [[0.], - [0.]], - [[1.], - [0.]], - [[1.], - [0.]], - [[0.], - [1.]], - [[0.], - [1.]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_probs, actual_probs, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_qpc, actual_qpc, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_spc, actual_spc, rtol=1e-05, atol=1e-05) - - def testCCTAttentionLayerStepByStep(self): - with self.session(use_gpu=True) as sess, self.SetEval(True): - depth = 4 - p = layers_with_attention.CCTAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = True - p.num_attention_heads = 2 - p.gating_tpl.hidden_layer_dim = 2 - p.gating_tpl.noise_std = 5.0 - p.gating_tpl.noise_warmup_steps = 100 - x_atten = layers_with_attention.CCTAttentionLayer(p) - - (source_vecs, _, _, _, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - source_padding = tf.zeros([5, 2]) - - ctx1, probs1, _, _ = x_atten.FPropDefaultTheta(source_vecs, - source_padding) - ctx2 = [] - probs2 = [] - cached_source_vecs = tf.zeros([0, 2, 4]) - cached_source_contexts = tf.zeros([0, 2, 4]) - prefix_states = py_utils.NestedMap( - key=cached_source_vecs, value=cached_source_contexts) - for i in range(5): - ctx, probs, prefix_states = x_atten.ExtendStep(x_atten.theta, - source_vecs[i, :, :], - prefix_states) - probs_pad = tf.zeros([2, 5 - i - 1]) - padded_probs = tf.concat([probs, probs_pad], 1) - ctx2.append(ctx) - probs2.append(padded_probs) - - ctx2 = tf.stack(ctx2) - probs2 = tf.stack(probs2) - - tf.global_variables_initializer().run() - ctx1_v, probs1_v, ctx2_v, probs2_v = sess.run( - [ctx1, probs1, ctx2, probs2]) - self.assertAllClose(ctx1_v, ctx2_v) - self.assertAllClose(probs1_v, probs2_v) - - def testCCTAttentionLayerCrossAttenTraining(self): - with self.session(use_gpu=True) as sess: - depth = 4 - p = layers_with_attention.CCTAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = False - p.num_attention_heads = 2 - p.gating_tpl.hidden_layer_dim = 2 - p.gating_tpl.noise_std = 5.0 - p.gating_tpl.noise_warmup_steps = 100 - transformer_atten = layers_with_attention.CCTAttentionLayer(p) - - (query_vec, _, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - - ctx, probs, qpc, spc = transformer_atten.FPropDefaultTheta( - query_vec, aux_paddings, aux_vecs) - tf.global_variables_initializer().run() - actual_ctx, actual_probs, actual_qpc, actual_spc = sess.run( - [ctx, probs, qpc, spc]) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [[-1.9043474 , 1.6999874 , 0.4292767 , -0.22491673], - [-0.84242177, -0.50577486, 0.29762083, 1.0505756 ]], - [[-0.33607534, 2.5800223 , -1.3375163 , -0.90643084], - [-0.4973639 , -0.17019022, -1.1589761 , 1.8265318 ]], - [[ 1.1859869 , 1.5021455 , -1.6327672 , -1.0553647 ], - [-1.2359238 , -0.22244841, 0.19330817, 1.2650642 ]], - [[-1.5131142 , 0.49699292, 1.129034 , -0.11291274], - [ 2.1162672 , 0.6308829 , -1.0373113 , -1.7098385 ]], - [[-0.9935959 , 0.07386243, -0.6836246 , 1.6033579 ], - [-1.0807116 , 0.85268646, -1.2622242 , 1.4902495 ]]] - expected_probs = [ - [[0.24303743, 0. , 0.30685946, 0. , 0.25564623, - 0. , 0.1944569 ], - [0. , 0.28801104, 0. , 0.34431183, 0. , - 0.36767715, 0. ]], - [[0.2644446 , 0. , 0.23458862, 0. , 0.23393473, - 0. , 0.26703206], - [0. , 0.22837642, 0. , 0.2820819 , 0. , - 0.4895417 , 0. ]], - [[0.2599384 , 0. , 0.19412258, 0. , 0.21307275, - 0. , 0.33286628], - [0. , 0.27514488, 0. , 0.35259444, 0. , - 0.3722607 , 0. ]], - [[0.24153353, 0. , 0.3045342 , 0. , 0.2569951 , - 0. , 0.19693717], - [0. , 0.36325702, 0. , 0.26765382, 0. , - 0.36908916, 0. ]], - [[0.21663833, 0. , 0.28198314, 0. , 0.29308724, - 0. , 0.20829134], - [0. , 0.2337277 , 0. , 0.319759 , 0. , - 0.44651327, 0. ]]] - expected_qpc = [ - [[0.5 ], - [0.5818492 ]], - [[0.541141 ], - [0.55023897]], - [[0.56948507], - [0.5499979 ]], - [[0.5166038 ], - [0.58645904]], - [[0.54155153], - [0.5 ]]] - expected_spc = [ - [[0.09838167], - [0.5 ]], - [[0.51203823], - [0.22011107]], - [[0.27349436], - [0.5230051 ]], - [[0.5 ], - [0.0911701 ]], - [[0.2730832 ], - [0.5 ]], - [[0.54982626], - [0.44889307]], - [[0.10193098], - [0.11123485]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_probs, actual_probs, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_qpc, actual_qpc, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_spc, actual_spc, rtol=1e-05, atol=1e-05) - - def testCCTAttentionLayerCrossAttenEval(self): - with self.session(use_gpu=True) as sess, self.SetEval(True): - depth = 4 - p = layers_with_attention.CCTAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = False - p.num_attention_heads = 2 - p.gating_tpl.hidden_layer_dim = 2 - p.gating_tpl.noise_std = 5.0 - p.gating_tpl.noise_warmup_steps = 100 - transformer_atten = layers_with_attention.CCTAttentionLayer(p) - - (query_vec, _, aux_vecs, aux_paddings, - _) = self._testTransformerAttentionLayerInputs(depth=depth) - - ctx, probs, qpc, spc = transformer_atten.FPropDefaultTheta( - query_vec, aux_paddings, aux_vecs) - tf.global_variables_initializer().run() - actual_ctx, actual_probs, actual_qpc, actual_spc = sess.run( - [ctx, probs, qpc, spc]) - tf.logging.info(np.array_repr(actual_ctx)) - tf.logging.info(np.array_repr(actual_probs)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [[-1.5939784 , 0.8543072 , 0.84722424, -0.10755297], - [-0.7121205 , -1.2363338 , 1.1559415 , 0.7925127 ]], - [[-0.09044743, 1.6572162 , -0.87628996, -0.69047904], - [-0.4314113 , -0.90650195, -0.35488802, 1.6928028 ]], - [[ 1.3591317 , 0.5376119 , -1.1282029 , -0.7685402 ], - [-1.0880733 , -0.9078379 , 0.9976803 , 0.9982312 ]], - [[-1.1870676 , -0.37413225, 1.5655125 , -0.00431258], - [ 1.62277 , 0.02716666, -0.7765793 , -0.87335706]], - [[-0.6675403 , -0.8283625 , -0.18727894, 1.6831816 ], - [-1.113929 , 0.13246097, -0.57226247, 1.5537308 ]]] - expected_probs = [ - [[0.25 , 0. , 0.25 , 0. , 0.25 , - 0. , 0.25 ], - [0. , 0.33333334, 0. , 0.33333334, 0. , - 0.33333334, 0. ]], - [[0.25 , 0. , 0.25 , 0. , 0.25 , - 0. , 0.25 ], - [0. , 0.33333334, 0. , 0.33333334, 0. , - 0.33333334, 0. ]], - [[0.25 , 0. , 0.25 , 0. , 0.25 , - 0. , 0.25 ], - [0. , 0.33333334, 0. , 0.33333334, 0. , - 0.33333334, 0. ]], - [[0.25 , 0. , 0.25 , 0. , 0.25 , - 0. , 0.25 ], - [0. , 0.33333334, 0. , 0.33333334, 0. , - 0.33333334, 0. ]], - [[0.25 , 0. , 0.25 , 0. , 0.25 , - 0. , 0.25 ], - [0. , 0.33333334, 0. , 0.33333334, 0. , - 0.33333334, 0. ]]] - expected_qpc = [ - [[1.], - [1.]], - [[1.], - [1.]], - [[1.], - [1.]], - [[1.], - [1.]], - [[1.], - [1.]]] - expected_spc = [ - [[0.], - [1.]], - [[1.], - [0.]], - [[0.], - [1.]], - [[1.], - [0.]], - [[0.], - [1.]], - [[1.], - [0.]], - [[0.], - [0.]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_ctx, actual_ctx, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_probs, actual_probs, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_qpc, actual_qpc, rtol=1e-05, atol=1e-05) - self.assertAllClose(expected_spc, actual_spc, rtol=1e-05, atol=1e-05) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/layers_with_gpipe.py b/lingvo/core/layers_with_gpipe.py deleted file mode 100644 index 66bcb8a1..00000000 --- a/lingvo/core/layers_with_gpipe.py +++ /dev/null @@ -1,1432 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Lingvo layers that depend on layers and gpipe.""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import batch_major_attention -from lingvo.core import layers -from lingvo.core import layers_with_attention -from lingvo.core import py_utils -from lingvo.core import tshape -from lingvo.core.gpipe import FeatureExtractionLayer -from lingvo.core.gpipe import PipeliningLayer - - -def _common_gpipe_transformer_params(p): - """Add GPipe params to layer.""" - p.Define( - 'is_transparent', False, - 'If set, encoder outputs a list of layer outputs while decoder ' - 'expects a list of source input vectors.') - p.Define('transparent_merger_tpl', None, - 'Creates weights for transparent combination.') - p.Define( - 'final_enc_layer', False, - 'True for final encoder layer. To be used for final transparent merger.') - p.Define( - 'normalize_output', False, - 'If set, encoder outputs a list of layer outputs while decoder ' - 'expects a list of source input vectors.') - p.Define('ln_tpl', layers.LayerNorm.Params(), 'Layer norm default params') - return p - - -def _common_gpipe_transformer_init(layer): - """Initialize a GPipe layer.""" - p = layer.params - - if p.normalize_output: - params = p.ln_tpl.Copy() - params.name = 'encoder_ln' - params.input_dim = p.source_dim - layer.CreateChild('layer_norm', params) - - if p.is_transparent and p.transparent_merger_tpl is not None: - transparent_param = p.transparent_merger_tpl.Copy() - transparent_param.name = 'transparent_0' - layer.CreateChild('transparent_merger', transparent_param) - assert p.name - - -def _common_gpipe_transformer_encoder_fprop( - layer, layer_class, theta, source_vecs, source_paddings, target_vecs, - target_paddings, source_segment_id, target_segment_id, transparent_acc, - transparent_acc_helper, source_task_id, target_task_id): - """GPipe encoder FProp.""" - p = layer.params - if source_task_id is not None or target_task_id is not None: - h, _ = super(layer_class, layer).FProp( - theta, - source_vecs, - source_paddings, - source_segment_id=source_segment_id, - source_task_id=source_task_id, - target_task_id=target_task_id) - else: - h, _ = super(layer_class, layer).FProp( - theta, - source_vecs, - source_paddings, - source_segment_id=source_segment_id) - h.set_shape(source_vecs.shape) - if p.is_transparent: - if p.transparent_merger_tpl is not None: - transparent_acc_helper = layer.transparent_merger.FProp( - theta.transparent_merger) - transparent_acc = tf.zeros_like(source_vecs) - transparent_acc = transparent_acc + transparent_acc_helper[0] * source_vecs - if p.final_enc_layer: - h = transparent_acc + h * transparent_acc_helper[-1] - transparent_acc = None - transparent_acc_helper = None - else: - transparent_acc_helper = transparent_acc_helper[1:] - if p.normalize_output: - h = layer.layer_norm.FProp(theta.layer_norm, h) - - if source_task_id is not None or target_task_id is not None: - return (h, source_paddings, target_vecs, target_paddings, source_segment_id, - target_segment_id, transparent_acc, transparent_acc_helper, - source_task_id, target_task_id) - else: - return (h, source_paddings, target_vecs, target_paddings, source_segment_id, - target_segment_id, transparent_acc, transparent_acc_helper) - - -def _common_gpipe_transformer_decoder_fprop( - layer, layer_class, theta, source_vecs, source_paddings, target_vecs, - target_paddings, source_segment_id, target_segment_id, transparent_acc, - transparent_acc_helper, source_task_id, target_task_id): - """GPipe decoder FProp.""" - assert target_vecs is not None - assert target_paddings is not None - if source_task_id is not None or target_task_id is not None: - h, _ = super(layer_class, layer).FProp( - theta, - target_vecs, - target_paddings, - aux_vecs=source_vecs, - aux_paddings=source_paddings, - source_segment_id=target_segment_id, - aux_segment_id=source_segment_id, - source_task_id=source_task_id, - target_task_id=target_task_id) - else: - h, _ = super(layer_class, layer).FProp( - theta, - target_vecs, - target_paddings, - aux_vecs=source_vecs, - aux_paddings=source_paddings, - source_segment_id=target_segment_id, - aux_segment_id=source_segment_id) - h.set_shape(target_vecs.shape) - - if source_task_id is not None or target_task_id is not None: - return (source_vecs, source_paddings, h, target_paddings, source_segment_id, - target_segment_id, transparent_acc, transparent_acc_helper, - source_task_id, target_task_id) - else: - return (source_vecs, source_paddings, h, target_paddings, source_segment_id, - target_segment_id, transparent_acc, transparent_acc_helper) - - -def _common_gpipe_transformer_fprop_meta(p, inputs, *args): - """GPipe FPropMeta function.""" - # TODO(huangyp): return accurate estimate of flops. - py_utils.CheckShapes((inputs,)) - flops_per_element = 5 - src_time, source_batch, dim = inputs - flops = flops_per_element * src_time * src_time * source_batch * dim - args = args if isinstance(args, tuple) else (args,) - if not p.has_aux_atten and p.is_transparent: # Transparent Encoder FPropMeta - if p.transparent_merger_tpl is not None: - args = args[:5] + (inputs, - tshape.Shape([p.transparent_merger_tpl.num_sources])) - args = args[:6] + (tshape.Shape([args[6][0] - 1]),) - if p.final_enc_layer: - args = args[:5] + (None, None) - return py_utils.NestedMap(flops=flops, out_shapes=(inputs,) + args) - - -class GPipeTransformerLayer(layers_with_attention.TransformerLayer): - """GPipe compatible transformer layer.""" - - @classmethod - def Params(cls): - """Configs for TransformerStack.""" - p = super().Params() - return _common_gpipe_transformer_params(p) - - def __init__(self, params): - super().__init__(params) - _common_gpipe_transformer_init(self) - - @classmethod - def SetupDeterministicDropout(cls, params): - """Replaced dropout layers in transformer with deterministic ones.""" - params.tr_atten_tpl.residual_dropout_tpl = ( - layers.DeterministicDropoutLayer.Params()) - params.tr_atten_tpl.atten_tpl.atten_dropout_deterministic = True - params.tr_atten_tpl.atten_tpl.inner_atten_params \ - .atten_dropout_deterministic = True - params.tr_fflayer_tpl.residual_dropout_tpl = ( - layers.DeterministicDropoutLayer.Params()) - params.tr_fflayer_tpl.fflayer_tpl.dropout = ( - layers.DeterministicDropoutLayer.Params()) - return params - - def FProp(self, - theta, - source_vecs, - source_paddings, - target_vecs, - target_paddings, - source_segment_id, - target_segment_id, - transparent_acc, - transparent_acc_helper, - source_task_id=None, - target_task_id=None): - p = self.params - with tf.name_scope(p.name): - if p.has_aux_atten: # Decoder FProp - return _common_gpipe_transformer_decoder_fprop( - self, GPipeTransformerLayer, theta, source_vecs, source_paddings, - target_vecs, target_paddings, source_segment_id, target_segment_id, - transparent_acc, transparent_acc_helper, source_task_id, - target_task_id) - else: # Encoder FProp - return _common_gpipe_transformer_encoder_fprop( - self, GPipeTransformerLayer, theta, source_vecs, source_paddings, - target_vecs, target_paddings, source_segment_id, target_segment_id, - transparent_acc, transparent_acc_helper, source_task_id, - target_task_id) - - @classmethod - def FPropMeta(cls, p, inputs, *args): - return _common_gpipe_transformer_fprop_meta(p, inputs, *args) - - -class GPipeEvolvedTransformerEncoderLayer( - layers_with_attention.EvolvedTransformerEncoderLayer): - """GPipe-compatible Evolved Transformer encoder layer.""" - - @classmethod - def Params(cls): - p = super().Params() - return _common_gpipe_transformer_params(p) - - def __init__(self, params): - super().__init__(params) - _common_gpipe_transformer_init(self) - - def FProp(self, - theta, - source_vecs, - source_paddings, - target_vecs, - target_paddings, - source_segment_id, - target_segment_id, - transparent_acc, - transparent_acc_helper, - source_task_id=None, - target_task_id=None): - with tf.name_scope(self.params.name): - return _common_gpipe_transformer_encoder_fprop( - self, GPipeEvolvedTransformerEncoderLayer, theta, source_vecs, - source_paddings, target_vecs, target_paddings, source_segment_id, - target_segment_id, None, None, source_task_id, target_task_id) - - @classmethod - def FPropMeta(cls, p, inputs, *args): - return _common_gpipe_transformer_fprop_meta(p, inputs, *args) - - @classmethod - def _AttentionSetupDeterministicDropout(cls, tr_atten_tpl): - tr_atten_tpl.residual_dropout_tpl = ( - layers.DeterministicDropoutLayer.Params()) - tr_atten_tpl.atten_tpl.atten_dropout_deterministic = True - tr_atten_tpl.atten_tpl.inner_atten_params.atten_dropout_deterministic = True - - @classmethod - def _TransformerSetupDeterministicDropout(cls, transformer_tpl): - cls._AttentionSetupDeterministicDropout(transformer_tpl.tr_atten_tpl) - transformer_tpl.tr_fflayer_tpl.residual_dropout_tpl = ( - layers.DeterministicDropoutLayer.Params()) - transformer_tpl.tr_fflayer_tpl.fflayer_tpl.dropout = ( - layers.DeterministicDropoutLayer.Params()) - - @classmethod - def SetupDeterministicDropout(cls, params): - """Replaces dropout layers in ET with deterministic ones.""" - cls._TransformerSetupDeterministicDropout(params.transformer_tpl) - params.branched_convs_tpl.dropout_tpl = \ - layers.DeterministicDropoutLayer.Params() - if hasattr(params, 'glu_tpl'): - params.glu_tpl.dropout_tpl = layers.DeterministicDropoutLayer.Params() - if hasattr(params, 'tr_atten_tpl'): - cls._AttentionSetupDeterministicDropout(params.tr_atten_tpl) - if hasattr(params, 'tr_double_heads_atten_tpl'): - cls._AttentionSetupDeterministicDropout(params.tr_double_heads_atten_tpl) - return params - - -class GPipeEvolvedTransformerDecoderLayer( - layers_with_attention.EvolvedTransformerDecoderLayer): - """GPipe-compatible Evolved Transformer decoder layer.""" - - @classmethod - def Params(cls): - p = super().Params() - return _common_gpipe_transformer_params(p) - - def __init__(self, params): - super().__init__(params) - _common_gpipe_transformer_init(self) - - def FProp(self, - theta, - source_vecs, - source_paddings, - target_vecs, - target_paddings, - source_segment_id, - target_segment_id, - transparent_acc, - transparent_acc_helper, - source_task_id=None, - target_task_id=None): - with tf.name_scope(self.params.name): - return _common_gpipe_transformer_decoder_fprop( - self, GPipeEvolvedTransformerDecoderLayer, theta, source_vecs, - source_paddings, target_vecs, target_paddings, source_segment_id, - target_segment_id, transparent_acc, transparent_acc_helper, - source_task_id, target_task_id) - - @classmethod - def FPropMeta(cls, p, inputs, *args): - return _common_gpipe_transformer_fprop_meta(p, inputs, *args) - - @classmethod - def _AttentionSetupDeterministicDropout(cls, tr_atten_tpl): - tr_atten_tpl.residual_dropout_tpl = ( - layers.DeterministicDropoutLayer.Params()) - tr_atten_tpl.atten_tpl.atten_dropout_deterministic = True - tr_atten_tpl.atten_tpl.inner_atten_params.atten_dropout_deterministic = True - - @classmethod - def _TransformerSetupDeterministicDropout(cls, transformer_tpl): - cls._AttentionSetupDeterministicDropout(transformer_tpl.tr_atten_tpl) - transformer_tpl.tr_fflayer_tpl.residual_dropout_tpl = ( - layers.DeterministicDropoutLayer.Params()) - transformer_tpl.tr_fflayer_tpl.fflayer_tpl.dropout = ( - layers.DeterministicDropoutLayer.Params()) - - @classmethod - def SetupDeterministicDropout(cls, params): - """Replaces dropout layers in ET with deterministic ones.""" - cls._TransformerSetupDeterministicDropout(params.transformer_tpl) - params.branched_convs_tpl.dropout_tpl = \ - layers.DeterministicDropoutLayer.Params() - if hasattr(params, 'glu_tpl'): - params.glu_tpl.dropout_tpl = layers.DeterministicDropoutLayer.Params() - if hasattr(params, 'tr_atten_tpl'): - cls._AttentionSetupDeterministicDropout(params.tr_atten_tpl) - if hasattr(params, 'tr_double_heads_atten_tpl'): - cls._AttentionSetupDeterministicDropout(params.tr_double_heads_atten_tpl) - return params - - -class GPipeTransformerSoftmaxLayer(layers.SimpleFullSoftmax): - """GPipe compatible softmax layer for transformers for computing logits.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('inputs_from_decoder', False, - 'Bool, whether inputs to this layer come from decoder or not.') - return p - - def FProp(self, - theta, - source_vecs, - source_paddings, - target_vecs, - target_paddings, - source_segment_id, - target_segment_id, - transparent_acc, - transparent_acc_helper, - source_task_id=None, - target_task_id=None): - del source_task_id - del target_task_id - p = self.params - if p.inputs_from_decoder: - transformer_output = target_vecs - else: - transformer_output = source_vecs - dim1, dim2 = tf.shape(transformer_output)[0], tf.shape( - transformer_output)[1] - softmax_input = tf.reshape(transformer_output, [-1, p.input_dim]) - output_shape = [dim1, dim2, p.num_classes] - return tf.reshape(super().Logits(theta, [softmax_input]), output_shape) - - @classmethod - def FPropMeta(cls, p, inputs, *args): - dim1, dim2 = args[1][:2] if p.inputs_from_decoder else inputs[:2] - logits = tshape.Shape([dim1, dim2, p.num_classes]) - return py_utils.NestedMap(flops=100, out_shapes=(logits,)) - - -class GPipeTransformerEmbeddingLayer(base_layer.BaseLayer): - """GPipe compatible embeddings for transformers.""" - - @classmethod - def Params(cls): - """Configs of Embedding layers for TransformerStack.""" - p = super().Params() - # Note: we use the same configs for src and tgt embeddings right now. - p.Define('token_emb', layers.SimpleEmbeddingLayer.Params(), - 'The embedding layer params.') - p.Define('position_emb', layers.PositionalEmbeddingLayer.Params(), - 'Position embedding layer params.') - p.Define('input_dropout_prob', 0.0, 'Prob at which we do input dropout.') - p.Define( - 'dropout_tpl', layers.DropoutLayer.Params(), - 'Replace with deterministic dropout for splits > 1 ' - 'or microbatches > 1.') - p.Define('add_tgt_embedding_layer', False, - 'Set True if layer embeds tgt instead of src.') - p.Define('packed_input', False, 'Set True to support packed inputs.') - p.Define( - 'is_transparent', False, - 'If set, encoder outputs a list of layer outputs while decoder ' - 'expects a list of source input vectors.') - p.Define('max_seq_len', 300, 'Max. seq len for decoding.') - p.Define('target_vocab_size', 0, 'Target vocab size, if different.') - - # Supporting task embeddings as additional input. - p.Define('dec_task_emb', None, - 'Adds task embeddings to every decoder timestep.') - p.Define('enc_task_emb', None, - 'Adds task embeddings to every encoder timestep.') - p.Define('batch_dim', 1, 'The batch dimension.') - p.Define('ret_task_ids', False, - 'Includes src_task_id and tgt_id in the fprop returns') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - p.token_emb.name = 'src_token_emb' - p.position_emb.name = 'src_position_emb' - self.CreateChild('src_token_emb', p.token_emb) - self.CreateChild('src_pos_emb', p.position_emb) - if p.enc_task_emb: - self.CreateChild('src_task_emb', p.enc_task_emb) - - p.dropout_tpl.keep_prob = (1.0 - p.input_dropout_prob) - p.dropout_tpl.name = 'src_dropout' - self.CreateChild('src_dropout', p.dropout_tpl) - - if p.add_tgt_embedding_layer: - params = p.token_emb.Copy() - if p.target_vocab_size: - params.vocab_size = p.target_vocab_size - params.name = 'tgt_token_emb' - self.CreateChild('tgt_token_emb', params) - params = p.position_emb.Copy() - params.name = 'tgt_position_emb' - self.CreateChild('tgt_pos_emb', params) - if p.dec_task_emb: - self.CreateChild('tgt_task_emb', p.dec_task_emb) - - params = p.dropout_tpl.Copy() - params.keep_prob = (1.0 - p.input_dropout_prob) - params.name = 'tgt_dropout' - self.CreateChild('tgt_dropout', params) - - def GetEmbeddings(self, emb_theta, emb, pos_emb_theta, pos_emb, dropout_theta, - dropout, input_ids, input_segment_pos, task_emb_theta, - task_emb, task_ids): - p = self.params - time_dim = 0 if p.batch_dim else 1 - seq_len = tf.shape(input_ids)[time_dim] - input_embs = emb.EmbLookup(emb_theta, input_ids) - if p.packed_input: # Packed inputs. - pos_embs = pos_emb.FPropWithPosition(pos_emb_theta, input_segment_pos) - else: - pos_embs = tf.expand_dims( - pos_emb.FProp(pos_emb_theta, seq_len), p.batch_dim) - - input_embs += pos_embs - if task_emb: - input_embs += task_emb.EmbLookup(task_emb_theta, task_ids) - input_embs = dropout.FProp(dropout_theta, input_embs) - return input_embs - - # To be used for decoding. - def GetEncoderEmbeddingsDefaultTheta(self, input_ids, task_ids=None): - p = self.params - time_dim = 0 if p.batch_dim else 1 - seq_len = tf.shape(input_ids)[time_dim] - input_embs = self.src_token_emb.EmbLookup(self.theta.src_token_emb, - input_ids) - pos_embs = tf.expand_dims( - self.src_pos_emb.FProp(self.theta.src_pos_emb, seq_len), p.batch_dim) - input_embs += pos_embs - if task_ids is not None and p.enc_task_emb: - input_embs += self.src_task_emb.EmbLookup(self.theta.src_task_emb, - task_ids) - input_embs = self.src_dropout.FProp(self.theta.src_dropout, input_embs) - return input_embs - - # To be used for decoding. - def GetDecoderEmbeddingsDefaultTheta(self, input_ids, task_ids=None, t=None): - p = self.params - input_embs = self.tgt_token_emb.EmbLookup(self.theta.tgt_token_emb, - input_ids) - if t is None: - time_dim = 0 if p.batch_dim else 1 - seq_len = tf.shape(input_ids)[time_dim] - pos_embs = tf.expand_dims( - self.tgt_pos_emb.FProp(self.theta.tgt_pos_emb, seq_len), p.batch_dim) - else: # Support decoding. - pos_embs = tf.slice( - self.tgt_pos_emb.FProp(self.theta.tgt_pos_emb, p.max_seq_len), [t, 0], - [1, p.token_emb.embedding_dim]) - input_embs += pos_embs - if task_ids is not None and p.dec_task_emb: - input_embs += self.tgt_task_emb.EmbLookup(self.theta.tgt_task_emb, - task_ids) - input_embs = self.tgt_dropout.FProp(self.theta.tgt_dropout, input_embs) - return input_embs - - def FProp(self, theta, source_id, source_paddings, target_id, target_paddings, - source_segment_id, target_segment_id, source_segment_pos, - target_segment_pos, source_task_id, target_task_id): - p = self.params - with tf.name_scope(p.name): - src_task_emb, src_task_emb_theta = None, None - if p.enc_task_emb: - src_task_emb, src_task_emb_theta = self.src_task_emb, theta.src_task_emb - source_vecs = self.GetEmbeddings(theta.src_token_emb, self.src_token_emb, - theta.src_pos_emb, self.src_pos_emb, - theta.src_dropout, self.src_dropout, - source_id, source_segment_pos, - src_task_emb_theta, src_task_emb, - source_task_id) - target_vecs = None - if p.add_tgt_embedding_layer: - tgt_task_emb, tgt_task_emb_theta = None, None - if p.enc_task_emb: - tgt_task_emb, tgt_task_emb_theta = (self.tgt_task_emb, - theta.tgt_task_emb) - target_vecs = self.GetEmbeddings(theta.tgt_token_emb, - self.tgt_token_emb, theta.tgt_pos_emb, - self.tgt_pos_emb, theta.tgt_dropout, - self.tgt_dropout, target_id, - target_segment_pos, tgt_task_emb_theta, - tgt_task_emb, target_task_id) - rets = (source_vecs, source_paddings, target_vecs, target_paddings, - source_segment_id, target_segment_id, None, None) - rets += (source_task_id, target_task_id) if p.ret_task_ids else () - return rets - - @classmethod - def FPropMeta(cls, p, inputs, *args): - # TODO(ankurbpn): return accurate estimate of flops. - py_utils.CheckShapes((inputs,)) - flops_per_element = 2 # Is this correct? - vocab = p.token_emb.vocab_size - dim = p.token_emb.embedding_dim - src_dim_0, src_dim_1 = inputs - flops = flops_per_element * src_dim_0 * src_dim_1 * dim * vocab - args = args if isinstance(args, tuple) else (args,) - new_inputs = tshape.Shape([src_dim_0, src_dim_1, dim]) - new_args = list(args) - if p.add_tgt_embedding_layer: - tgt_dim_0, tgt_dim_1 = args[1] - new_args[1] = tshape.Shape([tgt_dim_0, tgt_dim_1, dim]) - if p.ret_task_ids: - new_args = new_args[:5] + [None, None] + new_args[7:] - else: - new_args = new_args[:5] + [None, None] - new_args = tuple(new_args) - return py_utils.NestedMap(flops=flops, out_shapes=(new_inputs,) + new_args) - - -# TODO(ankurbpn,huangyp): Deprecate support for batch major layers here. -class GPipeTransformerStack(PipeliningLayer): - """Stacked self- multi-head attention and fully connected layers. - - With optional layer normalization applied to the final output. - - See 'Attention Is All You Need' https://arxiv.org/abs/1706.03762 - for details. - - The use of this stack for batch major transformer is deprecated. - Use GPipeBatchMajorTransformerStack instead. - """ - - @classmethod - def Params(cls): - """Configs for TransformerStack.""" - p = super().Params() - - # GPipe Related - p.Define( - 'splits', 1, - 'Number of splits, or list of integers specifying the ending index for ' - 'each split in ascending order. Last index should be num_layers.') - - # Transformer related - p.Define('model_dim', 1024, 'Characteristic depth (dimension).') - p.Define('num_encoder_layers', 0, 'Number of transformer encoder layers.') - p.Define('num_decoder_layers', 0, 'Number of transformer encoder layers.') - p.Define('use_pipelined_embeddings', True, 'Deprecated.') - p.Define('emb_tpl', GPipeTransformerEmbeddingLayer.Params(), - 'Prepare embeddings for Transformer input.') - p.Define('softmax_tpl', GPipeTransformerSoftmaxLayer.Params(), - 'Optional softmax layer to compute the logits.') - p.Define('label_smoothing', None, 'Label smoothing Params.') - p.Define('encoder_tpl', GPipeTransformerLayer.Params(), - 'TransformerLayer Encoder params tpl.') - p.Define('decoder_tpl', GPipeTransformerLayer.Params(), - 'TransformerLayer Decoder params tpl.') - p.Define('transparent_merger_dropout_prob', 0.1, - 'Dropout probability in WeightedSumLayer') - p.Define( - 'is_transparent', False, - 'If set, encoder outputs a merger of embeddings and ' - 'layer outputs.') - p.Define('transparent_merger_tpl', DeterministicWeightsLayer.Params(), - 'Creates weights for transparent combination.') - p.Define('packed_input', False, - 'If True, assumes multiple training samples per input.') - p.Define('normalize_encoder', False, - 'If True, layer-normalizes final encoder layer output.') - p.encoder_tpl.has_aux_atten = False - p.decoder_tpl.has_aux_atten = True - p.decoder_tpl.mask_self_atten = True - p.batch_dim = 1 - return p - - def __init__(self, params): - p = params.Copy() - num_layers = p.num_encoder_layers + p.num_decoder_layers - - if isinstance(p.splits, (list, tuple)): - assert p.splits[-1] == num_layers - for i, j in zip(p.splits[:-1], p.splits[1:]): - assert i <= j, 'Splits must be in increasing order.' - else: - num_splits = p.splits - layers_per_split = (num_layers - 1) // num_splits + 1 - p.splits = [] - for i in range(num_splits): - p.splits.append((i + 1) * layers_per_split) - p.splits[-1] = num_layers - - transformers = [] - - if p.is_transparent: - p.transparent_merger_tpl.num_sources = p.num_encoder_layers + 1 - p.transparent_merger_tpl.dropout_tpl.keep_prob = ( - 1 - p.transparent_merger_dropout_prob) - - # Encoder Embedding layer. - if len(p.splits) > 1 or p.num_micro_batches > 1: - p.emb_tpl.dropout_tpl = layers.DeterministicDropoutLayer.Params() - p.emb_tpl.packed_input = p.packed_input - p.emb_tpl.is_transparent = p.is_transparent - p.emb_tpl.add_tgt_embedding_layer = (p.num_decoder_layers > 0) - p.emb_tpl.name = 'emb' - p.emb_tpl.batch_dim = p.batch_dim - transformers.append(p.emb_tpl) - if p.softmax_tpl: - p.softmax_tpl.name = 'softmax' - p.softmax_tpl.inputs_from_decoder = p.num_decoder_layers > 0 - # Encoder layers. - for i in range(p.num_encoder_layers): - params = p.encoder_tpl.Copy() - params.name = 'encoder_%d' % i - if p.is_transparent: - params.is_transparent = p.is_transparent - params.final_enc_layer = (i == (p.num_encoder_layers - 1)) - if p.normalize_encoder and (i == (p.num_encoder_layers - 1)): - params.normalize_output = p.normalize_encoder - params.final_enc_layer = (i == (p.num_encoder_layers - 1)) - if p.packed_input: - params.packed_input = p.packed_input - # Use DeterministicDropoutLayer when used in temp graphs. - if len(p.splits) > 1 or p.num_micro_batches > 1: - params = params.cls.SetupDeterministicDropout(params) - assert not params.has_aux_atten - if p.is_transparent and i == 0: - params.transparent_merger_tpl = p.transparent_merger_tpl.Copy() - transformers.append(params) - - # Decoder layers. - for i in range(p.num_decoder_layers): - params = p.decoder_tpl.Copy() - params.name = 'decoder_%d' % i - params.mask_self_atten = True - if p.packed_input: - params.packed_input = p.packed_input - if len(p.splits) > 1 or p.num_micro_batches > 1: - params = params.cls.SetupDeterministicDropout(params) - assert params.has_aux_atten - transformers.append(params) - cells = [] - cell_start = 0 - # To account for embedding layers in the pipeline. - offset = 1 - for split, cell_end in enumerate(p.splits): - # Layer 0 (embeddings) is always in split 0. - sub = transformers[cell_start:(cell_end + offset)] - if split == len(p.splits) - 1 and p.softmax_tpl: - sub.append(p.softmax_tpl) - cell = FeatureExtractionLayer.Params().Set( - name='cell_{}'.format(split), sub=sub) - cells.append(cell) - cell_start = cell_end + offset - p.cell_tpl = cells - super().__init__(p) - - if p.label_smoothing: - self.CreateChild('smoother', p.label_smoothing) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - if self.params.label_smoothing: - self.smoother.InstantiateVariables() - super()._CreateChildrenVariables() - - def Logits(self, theta, inputs): - num_splits = len(self.params.splits) - softmax = self.children['cell_{}'.format(num_splits - 1)].softmax - softmax_theta = theta['cell_{}'.format(num_splits - 1)].softmax - return softmax.Logits(softmax_theta, inputs) - - def GetEncoders(self): - encoders = [] - p = self.params - cell_start = 0 - for split, cell_end in enumerate(p.splits): - for encoder_id in range(cell_start, cell_end): - if encoder_id >= p.num_encoder_layers: - break - encoder_l = self.children['cell_{}'.format(split)].children[ - 'encoder_{}'.format(encoder_id)] - encoders.append(encoder_l) - cell_start = cell_end - return encoders - - def GetDecoders(self): - decoders = [] - p = self.params - cell_start = 0 - for split, cell_end in enumerate(p.splits): - for layer_id in range(cell_start, cell_end): - decoder_id = layer_id - p.num_encoder_layers - if decoder_id < 0: - continue - decoder_l = self.children['cell_{}'.format(split)].children[ - 'decoder_{}'.format(decoder_id)] - decoders.append(decoder_l) - cell_start = cell_end - assert len(decoders) == p.num_decoder_layers - return decoders - - def EncoderEmbedFPropDefaultTheta(self, source_id, source_task_id=None): - emb = self.children['cell_0'].children['emb'] - return emb.GetEncoderEmbeddingsDefaultTheta(source_id, source_task_id) - - def DecoderEmbedFPropDefaultTheta(self, tgt_id, tgt_task_id=None, t=None): - emb = self.children['cell_0'].children['emb'] - return emb.GetDecoderEmbeddingsDefaultTheta(tgt_id, tgt_task_id, t) - - def EncoderFPropDefaultTheta(self, - source_vecs, - source_paddings, - source_segment_id=None, - source_task_id=None, - target_task_id=None): - p = self.params - transparent_acc = None - transparent_weights = None - for encoder_l in self.GetEncoders(): - if source_task_id is not None or target_task_id is not None: - encoder_outs = encoder_l.FProp( - encoder_l.theta, - source_vecs, - source_paddings, - None, - None, - source_segment_id, - None, - transparent_acc, - transparent_weights, - source_task_id=source_task_id, - target_task_id=target_task_id) - else: - encoder_outs = encoder_l.FProp(encoder_l.theta, source_vecs, - source_paddings, None, None, - source_segment_id, None, transparent_acc, - transparent_weights) - source_vecs = encoder_outs[0] - if p.is_transparent and len(encoder_outs) == 8: - transparent_acc = encoder_outs[6] - transparent_weights = encoder_outs[7] - return source_vecs - - def FProp(self, - theta, - source_input, - source_paddings, - target_input=None, - target_paddings=None, - source_segment_id=None, - target_segment_id=None, - labels=None, - label_weights=None, - source_segment_pos=None, - target_segment_pos=None, - source_task_id=None, - target_task_id=None): - """Transforms source sequence of Tensors with Transformers layers. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - source_input: A sequence of ints indicating source input ids of [time, - batch] shape or [batch, time] if batch_dim is 0. - source_paddings: A sequence of 0s and 1s indicating input paddings of - [time, batch] shape or [batch, time] if batch_dim is 0. - target_input: A sequence of ints indicating target input ids of [time, - batch] shape or [batch, time] if batch_dim is 0. - target_paddings: [target_time, target_batch] or [target_batch, - target_time] if batch_dim is 0. - source_segment_id: A sequence of ints indicating source segment ids of - [time, batch] shape or [batch, time] if batch_dim is 0. - target_segment_id: A sequence of ints indicating target segment ids of - [time, batch] shape or [batch, time] if batch_dim is 0. - labels: A sequence of ints indicating label ids of [time, batch] shape, or - [batch, time] if batch_dim is 0. - label_weights: A sequence of floats indicates label weights of [time, - batch] shape, or [batch, time] if batch_dim is 0. - source_segment_pos: A sequence of ints indicating source position ids of - [time, batch] shape, or [batch, time] if batch_dim is 0. - target_segment_pos: A sequence of ints indicating target position ids of - [time, batch] shape, or [batch, time] if batch_dim is 0. - source_task_id: A sequence of ints indicating source task ids of [time, - batch] shape, or [batch, time] if batch_dim is 0. - target_task_id: A sequence of ints indicating target task ids of [time, - batch] shape, or [batch, time] if batch_dim is 0. - - Returns: - transformer_output with shape [time, batch, dim] or [batch, time, dim] - if batch_dim is 0. - """ - p = self.params - if p.num_decoder_layers > 0: - assert target_input is not None - assert target_paddings is not None - if p.packed_input: - assert source_segment_id is not None, ( - 'Need to specify src_segment_id if packed input is supported.') - assert source_segment_pos is not None, ( - 'Need to specify src_segment_pos for packed input and embeddings.') - - logits = super().FProp(theta, source_input, source_paddings, target_input, - target_paddings, source_segment_id, - target_segment_id, source_segment_pos, - target_segment_pos, source_task_id, target_task_id) - if not p.softmax_tpl: - return logits - label_weights = tf.reshape(label_weights, [-1]) - target_probs = None - if p.label_smoothing: - if p.batch_dim: # Time-major - target_probs = tf.transpose( - self.smoother.FProp( - theta.smoother, - tf.transpose(target_paddings), - tf.transpose(labels), - target_ids=None), [1, 0, 2]) - else: - target_probs = self.smoother.FProp( - theta.smoother, target_paddings, labels, target_ids=None) - target_probs = tf.reshape(target_probs, [-1, p.softmax_tpl.num_classes]) - reshaped_logits = tf.reshape(logits, [-1, p.softmax_tpl.num_classes]) - tgt_labels = tf.reshape(labels, [-1]) - num_splits = len(p.splits) - softmax = self.children['cell_{}'.format(num_splits - 1)].softmax - softmax_theta = theta['cell_{}'.format(num_splits - 1)].softmax - per_example_xent, _ = softmax.XentLossFromLogits( - softmax_theta, - reshaped_logits, - class_weights=tf.reshape(label_weights, [-1]), - class_ids=tgt_labels, - class_probabilities=target_probs) - xent_shape = tf.shape(logits)[:2] - per_example_xent = tf.reshape(per_example_xent, xent_shape) - return per_example_xent, logits - - -class GPipeEvolvedTransformerStack(GPipeTransformerStack): - """Evolved Transformer stack for GPipe. - - With optional layer normalization applied to the final output. - - See 'Evolved Transformer' for more details: - https://arxiv.org/abs/1901.11117 . - """ - - @classmethod - def Params(cls): - """Configs for EvolvedTransformerStack.""" - p = super().Params() - p.encoder_tpl = GPipeEvolvedTransformerEncoderLayer.Params() - p.decoder_tpl = GPipeEvolvedTransformerDecoderLayer.Params() - return p - - -class DeterministicWeightsLayer(base_layer.BaseLayer): - """WeightedSumLayer with deterministic dropout.""" - - @classmethod - def Params(cls): - """Params for this MergerLayer class.""" - p = super().Params() - p.Define('num_sources', 0, 'Number of input sources to combine.') - p.Define('weighted_merger_dropout_prob', 0.0, - 'Applies dropout to the weights.') - p.Define( - 'weighted_merger_softmax', True, 'If set, applies a softmax ' - 'layer on top of the weights for normalization.') - p.Define('global_weight_scale', 1.0, 'A global scale put on weights.') - p.Define('minimal_prob', 0.0, 'The minimal weight for each component.') - p.Define('dropout_tpl', layers.DeterministicDropoutLayer.Params(), - 'Dropout layer') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if not p.name: - raise ValueError('Layer must have a specified name!') - - assert p.num_sources > 0, ('Must specify num_sources > 0.') - - p.dropout_tpl.name = 'dropout' - self.CreateChild('weighted_merger_dropout', p.dropout_tpl) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - params_init = py_utils.WeightInit.Constant(0.0) - # Weights to be learned. - pw = py_utils.WeightParams( - shape=[p.num_sources], - init=params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('sum_weight', pw) - - def FProp(self, theta): - """Combines the list of input tensors into a single tensor. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - - Returns: - A tensor of weights with dropout applied with shape [num_sources]. - """ - p = self.params - - # The constant factor is just meant to support the non-normalized scenario. - # If softmax is applied, this factor will cancel out. - w = theta.sum_weight * p.global_weight_scale + (1 / p.num_sources) - w = tf.reshape(w, [p.num_sources]) - w = self.weighted_merger_dropout.FProp(theta.weighted_merger_dropout, w) - if p.weighted_merger_softmax: - residual_weights = p.minimal_prob * p.num_sources - assert residual_weights >= 0.0 - assert residual_weights < 1.0 - w = tf.nn.softmax(w, axis=0) * (1.0 - residual_weights) + p.minimal_prob - return w - - -class GPipeBatchMajorTransformerSoftmaxLayer(layers.SimpleFullSoftmax): - """GPipe compatible softmax layer for transformers for computing logits. - - FProp interface is different for batch major stack, using segment_masks - instead of segment_ids. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('inputs_from_decoder', False, - 'Bool, whether inputs to this layer come from decoder or not.') - return p - - def FProp(self, theta, source_vecs, source_paddings, target_vecs, - target_paddings, encoder_self_atten_segment_mask, - decoder_self_atten_segment_mask, decoder_cross_atten_segment_mask): - p = self.params - if p.inputs_from_decoder: - transformer_output = target_vecs - else: - transformer_output = source_vecs - dim1, dim2 = tf.shape(transformer_output)[0], tf.shape( - transformer_output)[1] - softmax_input = tf.reshape(transformer_output, [-1, p.input_dim]) - output_shape = [dim1, dim2, p.num_classes] - return tf.reshape(super().Logits(theta, [softmax_input]), output_shape) - - @classmethod - def FPropMeta(cls, p, inputs, *args): - dim1, dim2 = args[1][:2] if p.inputs_from_decoder else inputs[:2] - logits = tshape.Shape([dim1, dim2, p.num_classes]) - return py_utils.NestedMap(flops=100, out_shapes=(logits,)) - - -class GPipeBatchMajorTransformerEmbeddingLayer(base_layer.BaseLayer): - """GPipe compatible embeddings for transformers.""" - - @classmethod - def Params(cls): - """Configs of Embedding layers for TransformerStack.""" - p = super().Params() - # Note: we use the same configs for src and tgt embeddings right now. - p.Define('token_emb', - layers.SimpleEmbeddingLayer.Params().Set(scale_sqrt_depth=True), - 'The embedding layer params.') - p.Define('position_emb', layers.PositionalEmbeddingLayer.Params(), - 'Position embedding layer params.') - p.Define('input_dropout_prob', 0.0, 'Prob at which we do input dropout.') - p.Define( - 'dropout_tpl', layers.DropoutLayer.Params(), - 'Replace with deterministic dropout for splits > 1 ' - 'or microbatches > 1.') - p.Define('add_tgt_embedding_layer', True, - 'Set True if layer embeds tgt instead of src.') - p.Define('packed_input', True, 'Set True to support packed inputs.') - p.Define('max_seq_len', 300, 'Max. seq len for decoding.') - p.Define('target_vocab_size', 0, 'Target vocab size, if different.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - p.token_emb.name = 'src_token_emb' - p.position_emb.name = 'src_position_emb' - self.CreateChild('src_token_emb', p.token_emb) - self.CreateChild('src_pos_emb', p.position_emb) - - p.dropout_tpl.keep_prob = (1.0 - p.input_dropout_prob) - p.dropout_tpl.name = 'src_dropout' - self.CreateChild('src_dropout', p.dropout_tpl) - - if p.add_tgt_embedding_layer: - params = p.token_emb.Copy() - if p.target_vocab_size: - params.vocab_size = p.target_vocab_size - params.name = 'tgt_token_emb' - self.CreateChild('tgt_token_emb', params) - params = p.position_emb.Copy() - params.name = 'tgt_position_emb' - self.CreateChild('tgt_pos_emb', params) - - params = p.dropout_tpl.Copy() - params.keep_prob = (1.0 - p.input_dropout_prob) - params.name = 'tgt_dropout' - self.CreateChild('tgt_dropout', params) - - def GetEmbeddings(self, emb_theta, emb, pos_emb_theta, pos_emb, dropout_theta, - dropout, input_ids, input_segment_pos): - p = self.params - time_dim = 1 - seq_len = tf.shape(input_ids)[time_dim] - input_embs = emb.EmbLookup(emb_theta, input_ids) - if p.packed_input: # Packed inputs. - pos_embs = pos_emb.FPropWithPosition(pos_emb_theta, input_segment_pos) - else: - pos_embs = tf.expand_dims(pos_emb.FProp(pos_emb_theta, seq_len), 0) - - input_embs += pos_embs - input_embs = dropout.FProp(dropout_theta, input_embs) - return input_embs - - # To be used for decoding. - def GetEncoderEmbeddingsDefaultTheta(self, input_ids): - seq_len = tf.shape(input_ids)[1] - input_embs = self.src_token_emb.EmbLookup(self.theta.src_token_emb, - input_ids) - pos_embs = tf.expand_dims( - self.src_pos_emb.FProp(self.theta.src_pos_emb, seq_len), 0) - input_embs += pos_embs - input_embs = self.src_dropout.FProp(self.theta.src_dropout, input_embs) - return input_embs - - # To be used for decoding. - def GetDecoderEmbeddingsDefaultTheta(self, input_ids, t): - seq_len = tf.shape(input_ids)[1] - input_embs = self.tgt_token_emb.EmbLookup(self.theta.tgt_token_emb, - input_ids) - # [target_batch, 1, dim] - # t should be shaped as [target_batch, 1]. - if t is None: - pos_embs = tf.expand_dims( - self.tgt_pos_emb.FProp(self.theta.tgt_pos_emb, seq_len), 0) - else: - pos_embs = self.tgt_pos_emb.FPropWithPosition(self.theta.tgt_pos_emb, t) - input_embs += pos_embs - input_embs = self.tgt_dropout.FProp(self.theta.tgt_dropout, input_embs) - return input_embs - - def FProp(self, theta, source_id, source_paddings, target_id, target_paddings, - encoder_self_atten_segment_mask, decoder_self_atten_segment_mask, - decoder_cross_atten_segment_mask, source_segment_pos, - target_segment_pos): - p = self.params - with tf.name_scope(p.name): - source_vecs = self.GetEmbeddings(theta.src_token_emb, self.src_token_emb, - theta.src_pos_emb, self.src_pos_emb, - theta.src_dropout, self.src_dropout, - source_id, source_segment_pos) - target_vecs = None - if p.add_tgt_embedding_layer: - target_vecs = self.GetEmbeddings(theta.tgt_token_emb, - self.tgt_token_emb, theta.tgt_pos_emb, - self.tgt_pos_emb, theta.tgt_dropout, - self.tgt_dropout, target_id, - target_segment_pos) - rets = (source_vecs, source_paddings, target_vecs, target_paddings, - encoder_self_atten_segment_mask, decoder_self_atten_segment_mask, - decoder_cross_atten_segment_mask) - return rets - - @classmethod - def FPropMeta(cls, p, inputs, *args): - # TODO(ankurbpn): return accurate estimate of flops. - py_utils.CheckShapes((inputs,)) - flops_per_element = 2 # Is this correct? - vocab = p.token_emb.vocab_size - dim = p.token_emb.embedding_dim - src_dim_0, src_dim_1 = inputs - flops = flops_per_element * src_dim_0 * src_dim_1 * dim * vocab - args = args if isinstance(args, tuple) else (args,) - new_inputs = tshape.Shape([src_dim_0, src_dim_1, dim]) - new_args = list(args) - if p.add_tgt_embedding_layer: - tgt_dim_0, tgt_dim_1 = args[1] - new_args[1] = tshape.Shape([tgt_dim_0, tgt_dim_1, dim]) - new_args = new_args[:6] - new_args = tuple(new_args) - return py_utils.NestedMap(flops=flops, out_shapes=(new_inputs,) + new_args) - - -class GPipeBatchMajorTransformerStack(PipeliningLayer): - """Stacked self- multi-head attention and fully connected layers. - - With optional layer normalization applied to the final output. - - See 'Attention Is All You Need' https://arxiv.org/abs/1706.03762 - for details. - - Implements a gipe stack for the batch major transformer variant. - """ - - @classmethod - def Params(cls): - """Configs for TransformerStack.""" - p = super().Params() - - # GPipe Related - p.Define( - 'splits', 1, - 'Number of splits, or list of integers specifying the ending index for ' - 'each split in ascending order. Last index should be num_layers.') - - # Transformer related - p.Define('model_dim', 1024, 'Characteristic depth (dimension).') - p.Define('num_encoder_layers', 0, 'Number of transformer encoder layers.') - p.Define('num_decoder_layers', 0, 'Number of transformer encoder layers.') - p.Define('emb_tpl', GPipeBatchMajorTransformerEmbeddingLayer.Params(), - 'Prepare embeddings for Transformer input.') - p.Define('softmax_tpl', GPipeBatchMajorTransformerSoftmaxLayer.Params(), - 'Optional softmax layer to compute the logits.') - p.Define('label_smoothing', None, 'Label smoothing Params.') - p.Define('encoder_tpl', - batch_major_attention.GPipeBatchMajorTransformerLayer.Params(), - 'TransformerLayer Encoder params tpl.') - p.Define('decoder_tpl', - batch_major_attention.GPipeBatchMajorTransformerLayer.Params(), - 'TransformerLayer Decoder params tpl.') - p.Define('packed_input', True, - 'If True, assumes multiple training samples per input.') - p.encoder_tpl.has_aux_atten = False - p.decoder_tpl.has_aux_atten = True - p.decoder_tpl.mask_self_atten = True - p.batch_dim = 0 - return p - - def __init__(self, params): - p = params.Copy() - num_layers = p.num_encoder_layers + p.num_decoder_layers - - if isinstance(p.splits, (list, tuple)): - assert p.splits[-1] == num_layers - for i, j in zip(p.splits[:-1], p.splits[1:]): - assert i <= j, 'Splits must be in increasing order.' - else: - num_splits = p.splits - layers_per_split = (num_layers - 1) // num_splits + 1 - p.splits = [] - for i in range(num_splits): - p.splits.append((i + 1) * layers_per_split) - p.splits[-1] = num_layers - - p.state_dtype = p.dtype - if p.fprop_dtype: - p.state_dtype = p.fprop_dtype - - transformers = [] - - # Encoder Embedding layer. - if len(p.splits) > 1 or p.num_micro_batches > 1: - p.emb_tpl.dropout_tpl = layers.DeterministicDropoutLayer.Params() - p.emb_tpl.packed_input = p.packed_input - p.emb_tpl.add_tgt_embedding_layer = (p.num_decoder_layers > 0) - p.emb_tpl.name = 'emb' - transformers.append(p.emb_tpl) - if p.softmax_tpl: - p.softmax_tpl.name = 'softmax' - p.softmax_tpl.inputs_from_decoder = p.num_decoder_layers > 0 - # Encoder layers. - for i in range(p.num_encoder_layers): - params = p.encoder_tpl.Copy() - params.name = 'encoder_%d' % i - if i == (p.num_encoder_layers - 1): - params.output_layer_norm = True - params.packed_input = p.packed_input - # Use DeterministicDropoutLayer when used in temp graphs. - if len(p.splits) > 1 or p.num_micro_batches > 1: - params = params.cls.SetupDeterministicDropout(params) - assert not params.has_aux_atten - transformers.append(params) - - # Decoder layers. - for i in range(p.num_decoder_layers): - params = p.decoder_tpl.Copy() - params.name = 'decoder_%d' % i - params.mask_self_atten = True - if i == (p.num_decoder_layers - 1): - params.output_layer_norm = True - params.packed_input = p.packed_input - if len(p.splits) > 1 or p.num_micro_batches > 1: - params = params.cls.SetupDeterministicDropout(params) - assert params.has_aux_atten - transformers.append(params) - cells = [] - cell_start = 0 - # To account for embedding layers in the pipeline. - offset = 1 - for split, cell_end in enumerate(p.splits): - # Layer 0 (embeddings) is always in split 0. - sub = transformers[cell_start:(cell_end + offset)] - if split == len(p.splits) - 1 and p.softmax_tpl: - sub.append(p.softmax_tpl) - cell = FeatureExtractionLayer.Params().Set( - name='cell_{}'.format(split), sub=sub) - cells.append(cell) - cell_start = cell_end + offset - p.cell_tpl = cells - super().__init__(p) - - if p.label_smoothing: - self.CreateChild('smoother', p.label_smoothing) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - if self.params.label_smoothing: - self.smoother.InstantiateVariables() - super()._CreateChildrenVariables() - - def Logits(self, theta, inputs): - num_splits = len(self.params.splits) - softmax = self.children['cell_{}'.format(num_splits - 1)].softmax - softmax_theta = theta['cell_{}'.format(num_splits - 1)].softmax - return softmax.Logits(softmax_theta, inputs) - - def GetEncoders(self): - encoders = [] - p = self.params - cell_start = 0 - for split, cell_end in enumerate(p.splits): - for encoder_id in range(cell_start, cell_end): - if encoder_id >= p.num_encoder_layers: - break - encoder_l = self.children['cell_{}'.format(split)].children[ - 'encoder_{}'.format(encoder_id)] - encoders.append(encoder_l) - cell_start = cell_end - return encoders - - def GetDecoders(self): - decoders = [] - p = self.params - cell_start = 0 - for split, cell_end in enumerate(p.splits): - for layer_id in range(cell_start, cell_end): - decoder_id = layer_id - p.num_encoder_layers - if decoder_id < 0: - continue - decoder_l = self.children['cell_{}'.format(split)].children[ - 'decoder_{}'.format(decoder_id)] - decoders.append(decoder_l) - cell_start = cell_end - assert len(decoders) == p.num_decoder_layers - return decoders - - def EncoderEmbedFPropDefaultTheta(self, source_id): - emb = self.children['cell_0'].children['emb'] - return emb.GetEncoderEmbeddingsDefaultTheta(source_id) - - def DecoderEmbedFPropDefaultTheta(self, tgt_id, t=None): - emb = self.children['cell_0'].children['emb'] - return emb.GetDecoderEmbeddingsDefaultTheta(tgt_id, t) - - def EncoderFPropDefaultTheta(self, - source_vecs, - source_paddings, - encoder_self_atten_segment_mask=None): - for encoder_l in self.GetEncoders(): - encoder_outs = encoder_l.FProp(encoder_l.theta, source_vecs, - source_paddings, None, None, - encoder_self_atten_segment_mask, None, - None) - source_vecs = encoder_outs[0] - return source_vecs - - def FProp(self, - theta, - source_input, - source_paddings, - target_input=None, - target_paddings=None, - source_segment_id=None, - target_segment_id=None, - labels=None, - label_weights=None, - source_segment_pos=None, - target_segment_pos=None): - """Transforms source sequence of Tensors with Transformers layers. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - source_input: A sequence of ints indicating source input ids of [batch, - time]. - source_paddings: A sequence of 0s and 1s indicating input paddings of - [batch, time]. - target_input: A sequence of ints indicating target input ids of [batch, - time]. - target_paddings: [target_batch, target_time]. - source_segment_id: A sequence of ints indicating source segment ids of - [batch, time]. - target_segment_id: A sequence of ints indicating target segment ids of - [batch, time]. - labels: A sequence of ints indicating label ids of [batch, time]. - label_weights: A sequence of floats indicates label weights of [batch, - time]. - source_segment_pos: A sequence of ints indicating source position ids of - [batch, time]. - target_segment_pos: A sequence of ints indicating target position ids of - [batch, time]. - - Returns: - transformer_output with shape [batch, time, dim]. - """ - p = self.params - if p.num_decoder_layers > 0: - assert target_input is not None - assert target_paddings is not None - target_time = tf.shape(target_input)[1] - batch = tf.shape(target_input)[0] - encoder_self_atten_segment_mask = None - decoder_self_atten_segment_mask = None - decoder_cross_atten_segment_mask = None - - # Prepare segment masks from segment ids. - if p.packed_input: - dtype = py_utils.FPropDtype(p) - assert source_segment_id is not None, ( - 'Need to specify src_segment_id if packed input is supported.') - assert source_segment_pos is not None, ( - 'Need to specify src_segment_pos for packed input and embeddings.') - encoder_self_atten_segment_mask = batch_major_attention.SegmentMask( - source_segment_id, source_segment_id, dtype, False) - if target_segment_id is not None: - decoder_self_atten_segment_mask = batch_major_attention.SegmentMask( - target_segment_id, target_segment_id, dtype, False) - causal_padding = tf.expand_dims( - tf.tile( - tf.expand_dims( - batch_major_attention.CausalPadding( - target_time, dtype=dtype), 0), [batch, 1, 1]), 1) - decoder_self_atten_segment_mask = tf.math.maximum( - causal_padding, decoder_self_atten_segment_mask) - decoder_cross_atten_segment_mask = batch_major_attention.SegmentMask( - target_segment_id, source_segment_id, dtype, False) - - # FProp through the gpipe pipeline. - logits = super().FProp(theta, source_input, source_paddings, target_input, - target_paddings, encoder_self_atten_segment_mask, - decoder_self_atten_segment_mask, - decoder_cross_atten_segment_mask, source_segment_pos, - target_segment_pos) - label_weights = tf.reshape(label_weights, [-1]) - target_probs = None - if p.label_smoothing: - target_probs = self.smoother.FProp( - theta.smoother, target_paddings, labels, target_ids=None) - target_probs = tf.reshape(target_probs, [-1, p.softmax_tpl.num_classes]) - reshaped_logits = tf.reshape(logits, [-1, p.softmax_tpl.num_classes]) - tgt_labels = tf.reshape(labels, [-1]) - num_splits = len(p.splits) - softmax = self.children['cell_{}'.format(num_splits - 1)].softmax - softmax_theta = theta['cell_{}'.format(num_splits - 1)].softmax - per_example_xent, _ = softmax.XentLossFromLogits( - softmax_theta, - reshaped_logits, - class_weights=tf.reshape(label_weights, [-1]), - class_ids=tgt_labels, - class_probabilities=target_probs) - xent_shape = tf.shape(logits)[:2] - per_example_xent = tf.reshape(per_example_xent, xent_shape) - return per_example_xent, logits diff --git a/lingvo/core/layers_with_gpipe_test.py b/lingvo/core/layers_with_gpipe_test.py deleted file mode 100644 index afe194b7..00000000 --- a/lingvo/core/layers_with_gpipe_test.py +++ /dev/null @@ -1,990 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for layers_with_gpipe.""" - -from absl.testing import parameterized -import lingvo.compat as tf -from lingvo.core import layers_with_gpipe -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.core.layers_with_gpipe import GPipeEvolvedTransformerDecoderLayer -from lingvo.core.layers_with_gpipe import GPipeEvolvedTransformerEncoderLayer -from lingvo.core.layers_with_gpipe import GPipeEvolvedTransformerStack -from lingvo.core.layers_with_gpipe import GPipeTransformerLayer -from lingvo.core.layers_with_gpipe import GPipeTransformerStack -import numpy as np - - -class GPipeTransformerLayersTest(test_utils.TestCase): - - def _testInputs(self, depth=3, dtype=tf.float32): - np.random.seed(505837249) - source_vecs = tf.stack( - [tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(5)]) - source_padding = tf.constant([[0, 0, 1, 1, 0], [1, 0, 0, 0, 1]], - dtype=dtype) - aux_source_vecs = tf.stack( - [tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(7)]) - aux_source_paddings = tf.constant( - [[0, 1, 0, 1, 0, 1, 0], [1, 0, 1, 0, 1, 0, 1]], dtype=dtype) - source_padding = tf.transpose(source_padding) - aux_source_paddings = tf.transpose(aux_source_paddings) - - input_task_arr = np.array([[0] * depth, [0] * depth]) - tgt_task_arr = np.array([[0] * depth] * 3) - input_tasks = tf.constant(input_task_arr.tolist(), dtype=tf.int32) - tgt_tasks = tf.constant(tgt_task_arr.tolist(), dtype=tf.int32) - return (source_vecs, source_padding, aux_source_vecs, aux_source_paddings, - input_tasks, tgt_tasks) - - def testGPipeSoftmaxLayerInputfromEncoder(self, use_task_ids=False): - with self.session(use_gpu=True): - depth = 4 - np.random.seed(6348575) - p = GPipeTransformerLayer.Params() - p.name = 'transformer' - p.source_dim = depth - p.has_aux_atten = False - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_attention_heads = 2 - transformer = p.Instantiate() - softmax = layers_with_gpipe.GPipeTransformerSoftmaxLayer.Params() - softmax.name = 'softmax' - softmax.inputs_from_decoder = False - softmax.num_classes = 2 - softmax.input_dim = depth - softmax = softmax.Instantiate() - (source_vecs, _, _, _, _, _) = self._testInputs(depth=depth) - source_padding = tf.zeros([5, 2]) - softmax_inputs = transformer.FPropDefaultTheta(source_vecs, - source_padding, None, None, - None, None, None, None) - softmax_outputs = softmax.FPropDefaultTheta(*softmax_inputs) - self.assertEqual([5, 2, 2], softmax_outputs.shape) - - def testGPipeSoftmaxLayerInputfromDecoder(self): - with self.session(use_gpu=True): - depth = 4 - np.random.seed(6348575) - p = GPipeTransformerLayer.Params() - p.name = 'transformer' - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_attention_heads = 2 - transformer = p.Instantiate() - softmax = layers_with_gpipe.GPipeTransformerSoftmaxLayer.Params() - softmax.name = 'softmax' - softmax.inputs_from_decoder = True - softmax.num_classes = 2 - softmax.input_dim = depth - softmax = softmax.Instantiate() - (source_vecs, _, aux_vecs, aux_paddings, _, - _) = self._testInputs(depth=depth) - source_padding = tf.zeros([5, 2]) - softmax_inputs = transformer.FPropDefaultTheta(aux_vecs, aux_paddings, - source_vecs, - source_padding, None, None, - None, None) - softmax_outputs = softmax.FPropDefaultTheta(*softmax_inputs) - self.assertEqual([5, 2, 2], softmax_outputs.shape) - - def testTransformerLayerExtendStep(self): - with self.session(use_gpu=True): - depth = 4 - np.random.seed(6348575) - p = GPipeTransformerLayer.Params() - p.name = 'transformer' - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_attention_heads = 2 - transformer = GPipeTransformerLayer(p) - - (source_vecs, _, aux_vecs, aux_paddings, input_tasks, - tgt_tasks) = self._testInputs(depth=depth) - source_padding = tf.zeros([5, 2]) - - output1 = transformer.FPropDefaultTheta(aux_vecs, aux_paddings, - source_vecs, source_padding, None, - None, input_tasks, tgt_tasks) - h1 = output1[2] - out_src_task, out_tgt_task = output1[-2], output1[-1] - - h2 = [] - cached_source_vecs = tf.zeros([0, 2, 4]) - cached_source_contexts = tf.zeros([0, 2, 4]) - prefix_states = py_utils.NestedMap( - key=cached_source_vecs, value=cached_source_contexts) - for i in range(5): - h, _, prefix_states = transformer.ExtendStep(transformer.theta, - source_vecs[i, :, :], - prefix_states, aux_vecs, - aux_paddings) - h2.append(h) - - h2 = tf.stack(h2) - - self.evaluate(tf.global_variables_initializer()) - h1_v, h2_v = self.evaluate([h1, h2]) - self.assertAllClose(h1_v, h2_v) - self.assertAllClose(out_src_task, input_tasks) - self.assertAllClose(out_tgt_task, tgt_tasks) - self.assertAllClose(h1_v[2][1], - [1.10429943, -1.64884555, 0.15726769, -0.00250494]) - - def testEvolvedTransformerEncoderLayerConstruction(self): - p = GPipeEvolvedTransformerEncoderLayer.Params() - p.name = 'gpipe_evolved_transformer_encoder' - p.source_dim = 4 - p.transformer_tpl.tr_fflayer_tpl.hidden_dim = 7 - p.transformer_tpl.tr_atten_tpl.num_attention_heads = 2 - _ = GPipeEvolvedTransformerEncoderLayer(p) - - def testEvolvedTransformerEncoderLayerFProp(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = GPipeEvolvedTransformerEncoderLayer.Params() - p.name = 'gpipe_evolved_transformer_encoder' - p.source_dim = depth - p.transformer_tpl.tr_fflayer_tpl.hidden_dim = 7 - p.transformer_tpl.tr_atten_tpl.num_attention_heads = 2 - transformer = GPipeEvolvedTransformerEncoderLayer(p) - - (source_vecs, source_padding, _, _, _, _) = self._testInputs(depth=depth) - - output = transformer.FPropDefaultTheta(source_vecs, source_padding, None, - None, None, None, None, None, None, - None) - h = output[0] - - self.evaluate(tf.global_variables_initializer()) - actual_layer_output = self.evaluate([h])[0] - tf.logging.info(np.array_repr(actual_layer_output)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_layer_output = [ - [[-2.05546284, -1.15067506, -0.00898855, 0.26481438], - [-0.8181392, -1.40835416, 0.47274107, 0.86176264]], - [[-1.43251371, 0.69686228, -0.70026731, -0.47239268], - [-0.39946821, -0.27037358, -0.22701442, 1.33816898]], - [[ 0.89412129, -1.07294774, -0.86541933, -0.21121001], - [-1.55683649, -1.14919782, 0.95287859, 0.11334917]], - [[-1.11910486, -1.01226425, 0.68622279, 0.00536875], - [ 2.33264184, 1.45991778, -0.71802276, -1.77120328]], - [[ 0.52163047, -1.90511549, -0.56069887, 1.1521647 ], - [-1.49890876, 0.20399603, -0.78263998, 1.30187178]]] - - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_layer_output, actual_layer_output) - - def testEvolvedTransformerDecoderLayerConstruction(self): - p = GPipeEvolvedTransformerDecoderLayer.Params() - p.name = 'gpipe_evolved_transformer_decoder' - p.source_dim = 16 - p.transformer_tpl.tr_atten_tpl.num_attention_heads = 2 - p.has_aux_atten = True - p.mask_self_atten = True - _ = GPipeEvolvedTransformerDecoderLayer(p) - - def testEvolvedTransformerDecoderLayerFProp(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = GPipeEvolvedTransformerDecoderLayer.Params() - p.name = 'gpipe_evolved_transformer_decoder' - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_double_heads_atten_tpl.num_attention_heads = 2 - p.tr_atten_tpl.num_attention_heads = 2 - p.transformer_tpl.tr_atten_tpl.num_attention_heads = 2 - transformer = GPipeEvolvedTransformerDecoderLayer(p) - - (source_vecs, source_padding, aux_vecs, aux_paddings, _, - _) = self._testInputs(depth=depth) - - output = transformer.FPropDefaultTheta(aux_vecs, aux_paddings, - source_vecs, source_padding, None, - None, None, None, None, None) - h = output[0] - - self.evaluate(tf.global_variables_initializer()) - actual_layer_output = self.evaluate([h])[0] - tf.logging.info(np.array_repr(actual_layer_output)) - # pylint: disable=bad-whitespace - # pyformat: disable - expected_layer_output = [ - [[ 0.5904724 , 0.05267439, 0.89581013, 0.63010913], - [ 0.79584485, 0.07670615, 0.40381077, 0.26504567]], - [[ 0.35448784, 0.28477612, 0.05394353, 0.06531866], - [ 0.44413447, 0.81940264, 0.98786688, 0.35846332]], - [[ 0.66811442, 0.07942203, 0.56781054, 0.83598584], - [ 0.45858502, 0.44949403, 0.06522893, 0.10947803]], - [[ 0.58166796, 0.94657594, 0.17643142, 0.02062288], - [ 0.40596515, 0.01996579, 0.93727112, 0.97478259]], - [[ 0.34873158, 0.0095871 , 0.34063059, 0.64620447], - [ 0.70584863, 0.69263214, 0.38247514, 0.28985959]], - [[ 0.66496903, 0.20383522, 0.35497066, 0.66646087], - [ 0.0787568 , 0.26172587, 0.23034802, 0.88751978]], - [[ 0.68153989, 0.81061888, 0.90142977, 0.87612331], - [ 0.15129775, 0.56084079, 0.87029755, 0.37908044]]] - # pyformat: enable - # pylint: enable=bad-whitespace - self.assertAllClose(expected_layer_output, actual_layer_output) - - def testEvolvedTransformerDecoderLayerExtendStep(self): - with self.session(use_gpu=True): - np.random.seed(6348575) - depth = 4 - p = GPipeEvolvedTransformerDecoderLayer.Params() - p.name = 'gpipe_evolved_transformer_decoder' - p.source_dim = depth - p.has_aux_atten = True - p.mask_self_atten = True - p.tr_double_heads_atten_tpl.num_attention_heads = 2 - p.tr_atten_tpl.num_attention_heads = 2 - p.transformer_tpl.tr_atten_tpl.num_attention_heads = 2 - et_decoder = GPipeEvolvedTransformerDecoderLayer(p) - - (source_vecs, _, aux_vecs, aux_paddings, input_tasks, - tgt_tasks) = self._testInputs(depth=depth) - source_padding = tf.zeros([5, 2]) - - output1 = et_decoder.FPropDefaultTheta(aux_vecs, aux_paddings, - source_vecs, source_padding, None, - None, input_tasks, tgt_tasks) - h1 = output1[2] - out_src_task, out_tgt_task = output1[-2], output1[-1] - h2 = [] - - double_head_attention_states = py_utils.NestedMap( - key=tf.zeros([0, 2, 4]), value=tf.zeros([0, 2, 4])) - transformer_layer_states = py_utils.NestedMap( - key=tf.zeros([0, 2, 4]), value=tf.zeros([0, 2, 4])) - branched_convs_input = tf.zeros([0, 2, 4]) - - prefix_states = py_utils.NestedMap( - double_head_attention_states=double_head_attention_states, - transformer_layer_states=transformer_layer_states, - branched_convs_input=branched_convs_input) - - for i in range(5): - h, _, prefix_states = et_decoder.ExtendStep(et_decoder.theta, - source_vecs[i, :, :], - prefix_states, aux_vecs, - aux_paddings) - h2.append(h) - - h2 = tf.stack(h2) - - self.evaluate(tf.global_variables_initializer()) - h1_v, h2_v = self.evaluate([h1, h2]) - self.assertAllClose(h1_v, h2_v) - self.assertAllClose(out_src_task, input_tasks) - self.assertAllClose(out_tgt_task, tgt_tasks) - - -def _AddClassesToTestParams(base_parameters_set, class_parameters_set): - output_parameters = [] - for class_parameters in class_parameters_set: - for base_parameters in base_parameters_set: - testcase_name = ( - base_parameters['testcase_name'] + class_parameters['testcase_name']) - new_parameters = base_parameters.copy() - new_parameters.update(class_parameters) - new_parameters['testcase_name'] = testcase_name - output_parameters.append(new_parameters) - return output_parameters - - -def _TransformerParamsWithEmbeddings(num_decoder_layers=0, - num_encoder_layers=4, - splits=1, - num_micro_batches=1, - has_softmax=False, - use_task_ids=False): - model_dim = 4 - params = GPipeTransformerStack.Params() - params.name = 'transformer' - params.model_dim = model_dim - params.num_decoder_layers = num_decoder_layers - params.decoder_tpl.source_dim = model_dim - params.decoder_tpl.tr_atten_tpl.num_attention_heads = 1 - params.decoder_tpl.tr_fflayer_tpl.hidden_dim = model_dim - params.num_encoder_layers = num_encoder_layers - params.encoder_tpl.source_dim = model_dim - params.encoder_tpl.tr_atten_tpl.num_attention_heads = 1 - params.encoder_tpl.tr_fflayer_tpl.hidden_dim = model_dim - params.num_micro_batches = num_micro_batches - params.state_dtype = tf.float32 - if has_softmax: - params.softmax_tpl.input_dim = model_dim - params.softmax_tpl.num_classes = 2 - else: - params.softmax_tpl = None - - emb_params = params.emb_tpl - # Default config for the token embedding. - emb_params.token_emb.use_matmul = True - emb_params.token_emb.use_3d_weight_tensor = False - emb_params.token_emb.vocab_size = 10 - emb_params.token_emb.embedding_dim = model_dim - - # Default config for the position embedding. - emb_params.position_emb.embedding_dim = model_dim - emb_params.position_emb.trainable_scaling = False - - # Task embeddings. - if use_task_ids: - emb_params.enc_task_emb = emb_params.token_emb.Copy() - emb_params.dec_task_emb = emb_params.token_emb.Copy() - params.splits = splits - params.random_seed = 0 - return params - - -def _EvolvedTransformerParamsWithEmbeddings(num_decoder_layers=0, - num_encoder_layers=4, - splits=1, - num_micro_batches=1, - has_softmax=False, - use_task_ids=False): - model_dim = 4 - params = GPipeEvolvedTransformerStack.Params() - params.name = 'evolved_transformer' - params.model_dim = model_dim - params.num_decoder_layers = num_decoder_layers - params.decoder_tpl.source_dim = model_dim - params.decoder_tpl.tr_atten_tpl.num_attention_heads = 1 - params.decoder_tpl.tr_double_heads_atten_tpl.num_attention_heads = 1 - params.decoder_tpl.transformer_tpl.tr_atten_tpl.num_attention_heads = 1 - params.num_encoder_layers = num_encoder_layers - params.encoder_tpl.source_dim = model_dim - params.encoder_tpl.transformer_tpl.tr_atten_tpl.num_attention_heads = 1 - params.num_micro_batches = num_micro_batches - params.state_dtype = tf.float32 - if has_softmax: - params.softmax_tpl.input_dim = model_dim - params.softmax_tpl.num_classes = 2 - else: - params.softmax_tpl = None - - emb_params = params.emb_tpl - # Default config for the token embedding. - emb_params.token_emb.use_matmul = True - emb_params.token_emb.use_3d_weight_tensor = False - emb_params.token_emb.vocab_size = 10 - emb_params.token_emb.embedding_dim = model_dim - - # Default config for the position embedding. - emb_params.position_emb.embedding_dim = model_dim - emb_params.position_emb.trainable_scaling = False - - # Task embeddings. - if use_task_ids: - emb_params.enc_task_emb = emb_params.token_emb.Copy() - emb_params.dec_task_emb = emb_params.token_emb.Copy() - params.splits = splits - params.random_seed = 0 - return params - - -def _TransformerRandomInputs(batch): - input_arr = np.array([ - [[0, 1]] * batch, - [[1, -1]] * batch, - ]) - paddings_arr = np.array([[0] * batch] * 2) - tgt_input_arr = np.array([ - [[1, 2]] * batch, - [[1, -1]] * batch, - [[2, 1]] * batch, - ]) - tgt_paddings_arr = np.array([[0] * batch] * 3) - inputs = tf.constant(input_arr.tolist(), dtype=tf.float32) - paddings = tf.constant(paddings_arr.tolist(), dtype=tf.float32) - tgt_inputs = tf.constant(tgt_input_arr.tolist(), dtype=tf.float32) - tgt_paddings = tf.constant(tgt_paddings_arr.tolist(), dtype=tf.float32) - return inputs, paddings, tgt_inputs, tgt_paddings - - -def _TransformerRandomInputsIds(batch): - input_arr = np.array([[6] * batch, [4] * batch]) - paddings_arr = np.array([[0] * batch] * 2) - input_task_arr = np.array([[0] * batch, [0] * batch]) - tgt_input_arr = np.array([[3] * batch, [7] * batch, [9] * batch]) - tgt_paddings_arr = np.array([[0] * batch] * 3) - tgt_task_arr = np.array([[0] * batch] * 3) - inputs = tf.constant(input_arr.tolist(), dtype=tf.int32) - paddings = tf.constant(paddings_arr.tolist(), dtype=tf.float32) - input_tasks = tf.constant(input_task_arr.tolist(), dtype=tf.int32) - tgt_inputs = tf.constant(tgt_input_arr.tolist(), dtype=tf.int32) - tgt_paddings = tf.constant(tgt_paddings_arr.tolist(), dtype=tf.float32) - tgt_tasks = tf.constant(tgt_task_arr.tolist(), dtype=tf.int32) - return inputs, paddings, tgt_inputs, tgt_paddings, input_tasks, tgt_tasks - - -def _TransformerRandomInputsVecs(batch): - input_arr = np.array([[[0, 1, 1, -1]] * batch, [[0, 2, 7, 1]] * batch]) - paddings_arr = np.array([[0] * batch] * 2) - tgt_input_arr = np.array([[[1, 2, 0, 1]] * batch, [[1, -1, 1, 0]] * batch, - [[2, 1, 2, 1]] * batch]) - tgt_paddings_arr = np.array([[0] * batch] * 3) - inputs = tf.constant(input_arr.tolist(), dtype=tf.float32) - paddings = tf.constant(paddings_arr.tolist(), dtype=tf.float32) - tgt_inputs = tf.constant(tgt_input_arr.tolist(), dtype=tf.float32) - tgt_paddings = tf.constant(tgt_paddings_arr.tolist(), dtype=tf.float32) - return inputs, paddings, tgt_inputs, tgt_paddings - - -def _EvolvedTransformerRandomInputs(batch): - padding_length = 10 - input_padding = [[[0, 0]] * batch] * padding_length - input_arr = np.array([[[0, 1]] * batch] + input_padding + [[[1, -1]] * batch]) - padding_indexes = [[1] * batch] * padding_length - paddings_arr = np.array([[0] * batch] + padding_indexes + [[0] * batch]) - tgt_input_arr = np.array([[[1, 2]] * batch] + input_padding + - [[[1, -1]] * batch] + input_padding + - [[[2, 1]] * batch]) - tgt_paddings_arr = np.array([[0] * batch] + padding_indexes + [[0] * batch] + - padding_indexes + [[0] * batch]) - inputs = tf.constant(input_arr.tolist(), dtype=tf.float32) - paddings = tf.constant(paddings_arr.tolist(), dtype=tf.float32) - tgt_inputs = tf.constant(tgt_input_arr.tolist(), dtype=tf.float32) - tgt_paddings = tf.constant(tgt_paddings_arr.tolist(), dtype=tf.float32) - return inputs, paddings, tgt_inputs, tgt_paddings - - -def _EvolvedTransformerRandomInputsIds(batch): - padding_length = 10 - input_padding = [[0] * batch] * padding_length - input_arr = np.array([[6] * batch] + input_padding + [[4] * batch]) - input_task_arr = np.array([[0] * batch] + input_padding + [[0] * batch]) - padding_indexes = [[1] * batch] * padding_length - paddings_arr = np.array([[0] * batch] + padding_indexes + [[0] * batch]) - tgt_input_arr = np.array([[3] * batch] + input_padding + [[7] * batch] + - input_padding + [[9] * batch]) - tgt_task_arr = np.array([[0] * batch] + input_padding + [[0] * batch] + - input_padding + [[0] * batch]) - tgt_paddings_arr = np.array([[0] * batch] + padding_indexes + [[0] * batch] + - padding_indexes + [[0] * batch]) - inputs = tf.constant(input_arr.tolist(), dtype=tf.int32) - paddings = tf.constant(paddings_arr.tolist(), dtype=tf.float32) - input_tasks = tf.constant(input_task_arr.tolist(), dtype=tf.int32) - tgt_inputs = tf.constant(tgt_input_arr.tolist(), dtype=tf.int32) - tgt_paddings = tf.constant(tgt_paddings_arr.tolist(), dtype=tf.float32) - tgt_tasks = tf.constant(tgt_task_arr.tolist(), dtype=tf.int32) - return inputs, paddings, tgt_inputs, tgt_paddings, input_tasks, tgt_tasks - - -class GPipeTransformerStackTest(test_utils.TestCase, - parameterized.TestCase): # was tf.test.TestCase - """Tests for GPipeTransformerStack layer.""" - - @parameterized.named_parameters({ - 'testcase_name': '_one_split', - 'splits': 1 - }, { - 'testcase_name': '_two_splits', - 'splits': 2 - }) - def testGPipeTransformerBatchMajorConstruction(self, splits=1): - batch = 4 - tf.flags.FLAGS.tpu_compatible = True - with self.session(): - with tf.variable_scope('transformer_test', reuse=tf.AUTO_REUSE): - params = _TransformerParamsWithEmbeddings( - splits=splits, num_decoder_layers=4, has_softmax=True) - params.batch_dim = 0 - params.emb_tpl.batch_dim = 0 - xformer = params.Instantiate() - input_ids, id_paddings, tgt_inputs, tgt_paddings, _, _ = ( - _TransformerRandomInputsIds(batch=batch)) - input_ids = tf.transpose(input_ids) - id_paddings = tf.transpose(id_paddings) - tgt_inputs = tf.transpose(tgt_inputs) - tgt_paddings = tf.transpose(tgt_paddings) - labels = tf.ones([batch, tgt_inputs.shape.as_list()[1]], dtype=tf.int32) - label_weights = tf.ones([batch, tgt_inputs.shape.as_list()[1]]) - tf.random.set_seed(1234) - self.evaluate(tf.global_variables_initializer()) - xent, logits = xformer.FProp(xformer.theta, input_ids, id_paddings, - tgt_inputs, tgt_paddings, None, None, - labels, label_weights, None, None, None, - None) - xent_out, logits_out = self.evaluate([xent, logits]) - print('xent_out={}'.format(xent_out)) - print('logits_out={}'.format(logits_out)) - - @parameterized.named_parameters({ - 'testcase_name': '_one_split', - 'splits': 1 - }, { - 'testcase_name': '_two_splits', - 'splits': 2 - }) - def testGPipeTransformerFPropPackedInputWithEmbeddings(self, splits=1): - batch = 4 - tf.flags.FLAGS.tpu_compatible = True - with self.session(): - with tf.variable_scope('transformer_test', reuse=tf.AUTO_REUSE): - params = _TransformerParamsWithEmbeddings( - splits=splits, num_decoder_layers=2) - params.dtype = tf.float32 - params.fprop_dtype = tf.float32 - packed_params = params.Copy() - packed_params.packed_input = True - xformer = GPipeTransformerStack(params) - packed_xformer = GPipeTransformerStack(packed_params) - # Prepare inputs - inputs, paddings, tgt_inputs, tgt_paddings, _, _ = _TransformerRandomInputsIds( - batch) - packed_inputs = tf.reshape(inputs, [-1, 1]) - packed_tgt_inputs = tf.reshape(tgt_inputs, [-1, 1]) - packed_paddings = tf.reshape(paddings, [-1, 1]) - packed_tg_paddings = tf.reshape(tgt_paddings, [-1, 1]) - segment_ids = tf.transpose( - tf.constant([[0, 1, 2, 3, 0, 1, 2, 3]], dtype=tf.float32)) - tgt_segment_id = tf.transpose( - tf.constant([[0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3]], - dtype=tf.float32)) - segment_segment_pos = tf.transpose( - tf.constant([[0, 0, 0, 0, 1, 1, 1, 1]], dtype=tf.int32)) - tgt_segment_segment_pos = tf.transpose( - tf.constant([[0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2]], dtype=tf.int32)) - - output = xformer.FProp(xformer.theta, inputs, paddings, tgt_inputs, - tgt_paddings)[2] - packed_output = packed_xformer.FProp( - packed_xformer.theta, packed_inputs, packed_paddings, - packed_tgt_inputs, packed_tg_paddings, segment_ids, tgt_segment_id, - None, None, segment_segment_pos, tgt_segment_segment_pos)[2] - packed_output = tf.reshape(packed_output, output.shape) - - self.evaluate(tf.global_variables_initializer()) - output, packed_output = self.evaluate([output, packed_output]) - self.assertAllClose(output, packed_output, rtol=1e-05, atol=1e-05) - - @parameterized.named_parameters( - { - 'testcase_name': '_split1', - 'splits': 1, - 'num_micro_batches': 1 - }, { - 'testcase_name': '_split1_nmb2', - 'splits': 1, - 'num_micro_batches': 2 - }, { - 'testcase_name': '_split2_nmb2', - 'splits': 2, - 'num_micro_batches': 2 - }, { - 'testcase_name': '_split_manual_nmb2', - 'splits': [3, 4], - 'num_micro_batches': 2 - }) - def testGPipeTransformerStackTrainTransparentFPropWithEmbeddings( - self, splits=1, num_micro_batches=1): - # time = 2, - batch = 4 - with self.session(): - params = _TransformerParamsWithEmbeddings( - splits=splits, - num_micro_batches=num_micro_batches, - num_decoder_layers=3, - num_encoder_layers=1) - params.is_transparent = True - params.transparent_merger_dropout_prob = 0.0 - xformer = GPipeTransformerStack(params) - - input_ids, id_paddings, tgt_inputs, tgt_paddings, _, _ = _TransformerRandomInputsIds( - batch=batch) - inputs, paddings, _, _ = _TransformerRandomInputsVecs(batch=batch) - tf.random.set_seed(1234) - self.evaluate(tf.global_variables_initializer()) - enc_outputs = xformer.EncoderFPropDefaultTheta(inputs, paddings) - dec_output = xformer.FProp(xformer.theta, input_ids, id_paddings, - tgt_inputs, tgt_paddings)[2] - enc_out_1 = self.evaluate(enc_outputs) - dec_out = self.evaluate(dec_output) - self.assertAllClose( - [[[0.017581, 0.802863, 0.975554, -1.164572]] * batch, - [[-0.549953, 1.196884, 4.910457, -0.102137]] * batch], enc_out_1) - self.assertAllClose( - [[[-1.122128, 1.111972, 4.642949, -2.14831]] * batch, - [[-1.336919, 1.182709, 4.785938, -2.039246]] * batch, - [[-1.335168, 1.297679, 4.720459, -2.111006]] * batch], dec_out) - - # pylint: disable=bad-continuation - # pyformat: disable - @parameterized.named_parameters( - _AddClassesToTestParams(({ - 'testcase_name': '_split1', - 'splits': 1, - 'num_micro_batches': 1 - }, { - 'testcase_name': '_split1_nmb2', - 'splits': 1, - 'num_micro_batches': 2 - }, { - 'testcase_name': '_split2_nmb1', - 'splits': 2, - 'num_micro_batches': 1 - }, { - 'testcase_name': '_split2_nmb2', - 'splits': 2, - 'num_micro_batches': 2 - }), ({ - 'testcase_name': - '_transformer', - 'params_fn': - _TransformerParamsWithEmbeddings, - 'stack_cls': - GPipeTransformerStack, - 'inputs_fn': - _TransformerRandomInputsIds, - 'expected_output': - [[[-2.306972, 2.07353, 5.542715, -2.421023]] * 4, - [[-2.366269, 2.0730767, 5.573641, -2.371158]] * 4, - [[-2.379758, 2.092285, 5.570339, -2.370932]] * 4] - }, { - 'testcase_name': - '_evolved_transformer', - 'params_fn': - _EvolvedTransformerParamsWithEmbeddings, - 'stack_cls': - GPipeEvolvedTransformerStack, - 'inputs_fn': - _EvolvedTransformerRandomInputsIds, - 'expected_output': [ - [[-0.563482, 0.873324, 4.807824, -2.21867]] * 4, - [[-0.69335, 0.275365, 3.85347, -1.25291]] * 4, - [[-0.691535, 0.513666, 3.818152, -1.448952]] * 4, - [[-0.906161, 0.207199, 3.865673, -1.015405]] * 4, - [[-0.779552, 0.199017, 3.864474, -1.115817]] * 4, - [[-0.733586, 0.229975, 3.860421, -1.181348]] * 4, - [[-0.712061, 0.24913, 3.857376, -1.215363]] * 4, - [[-0.697319, 0.267139, 3.854558, -1.242636]] * 4, - [[-0.667346, 0.455872, 3.825749, -1.421671]] * 4, - [[-0.933211, 0.304563, 3.863014, -1.08345]] * 4, - [[-0.804276, 0.187855, 3.865573, -1.084767]] * 4, - [[-0.481617, 0.657398, 4.772114, -2.071473]] * 4, - [[-0.658759, 0.130042, 3.853722, -1.142575]] * 4, - [[-0.641755, 0.148885, 3.851666, -1.173375]] * 4, - [[-0.623731, 0.205668, 3.847772, -1.23985]] * 4, - [[-0.827914, 0.091931, 3.861872, -0.967819]] * 4, - [[-0.74599, 0.066758, 3.858107, -1.010512]] * 4, - [[-0.69355, 0.099376, 3.856509, -1.085727]] * 4, - [[-0.664674, 0.124304, 3.854321, -1.132531]] * 4, - [[-0.646149, 0.143505, 3.852249, -1.164976]] * 4, - [[-0.630899, 0.169327, 3.849945, -1.200752]] * 4, - [[-0.83285, 0.199918, 3.866308, -1.072462]] * 4, - [[-0.440338, 0.562651, 4.749378, -2.003749]] * 4 - ]}))) - # pyformat: enable - # pylint: enable=bad-continuation - def testGPipeTransformerDecoderStackFPropWithEmbeddings( - self, - params_fn, - expected_output, - inputs_fn, - stack_cls, - splits=1, - num_micro_batches=1): - batch = 4 - tf.flags.FLAGS.tpu_compatible = True - with self.session(): - params = params_fn( - num_decoder_layers=4, - num_encoder_layers=0, - splits=splits, - num_micro_batches=num_micro_batches) - params.dtype = tf.float32 - xformer = stack_cls(params) - - inputs, paddings, tgt_inputs, tgt_paddings, _, _ = inputs_fn(batch) - - output = xformer.FProp(xformer.theta, inputs, paddings, tgt_inputs, - tgt_paddings)[2] - - self.evaluate(tf.global_variables_initializer()) - output_val = self.evaluate(output) - self.assertAllCloseAccordingToType( - expected_output, output_val, rtol=1e-05, atol=1e-05) - - @parameterized.named_parameters( - _AddClassesToTestParams(({ - 'testcase_name': '_split1', - 'splits': 1, - 'num_micro_batches': 1 - }, { - 'testcase_name': '_split1_nmb2', - 'splits': 1, - 'num_micro_batches': 2 - }, { - 'testcase_name': '_split2_nmb1', - 'splits': 2, - 'num_micro_batches': 1 - }, { - 'testcase_name': '_split2_nmb2', - 'splits': 2, - 'num_micro_batches': 2 - }), ({ - 'testcase_name': '_transformer', - 'params_fn': _TransformerParamsWithEmbeddings, - 'stack_cls': GPipeTransformerStack, - 'inputs_fn': _TransformerRandomInputsIds - }, { - 'testcase_name': '_evolved_transformer', - 'params_fn': _EvolvedTransformerParamsWithEmbeddings, - 'stack_cls': GPipeEvolvedTransformerStack, - 'inputs_fn': _EvolvedTransformerRandomInputsIds - }))) - def testGPipeTransformerLmModel(self, - params_fn, - stack_cls, - inputs_fn, - splits=1, - num_micro_batches=1): - batch = 4 - tf.flags.FLAGS.tpu_compatible = True - with self.session(): - with tf.variable_scope('transformer_test', reuse=tf.AUTO_REUSE): - params = params_fn( - splits=splits, - num_micro_batches=num_micro_batches, - num_decoder_layers=0, - has_softmax=True) - params.state_dtype = tf.float32 - xformer = stack_cls(params) - - input_ids, id_paddings, _, _, _, _ = inputs_fn(batch=batch) - labels = tf.ones([input_ids.shape.as_list()[0], batch], dtype=tf.int32) - label_weights = tf.ones([input_ids.shape.as_list()[0], batch]) - tf.random.set_seed(1234) - self.evaluate(tf.global_variables_initializer()) - xent, logits = xformer.FProp(xformer.theta, input_ids, id_paddings, None, - None, None, None, labels, label_weights) - xent_out, logits_out = self.evaluate([xent, logits]) - print('xent_out={}'.format(xent_out)) - print('logits_out={}'.format(logits_out)) - - @parameterized.named_parameters( - _AddClassesToTestParams(({ - 'testcase_name': '_split1', - 'splits': 1, - 'num_micro_batches': 1 - }, { - 'testcase_name': '_split1_nmb2', - 'splits': 1, - 'num_micro_batches': 2 - }, { - 'testcase_name': '_split2_nmb1', - 'splits': 2, - 'num_micro_batches': 1 - }, { - 'testcase_name': '_split2_nmb2', - 'splits': 2, - 'num_micro_batches': 2 - }, { - 'testcase_name': '_split1_task_embs', - 'use_task_embs': True - }), ({ - 'testcase_name': '_transformer', - 'params_fn': _TransformerParamsWithEmbeddings, - 'stack_cls': GPipeTransformerStack, - 'inputs_fn': _TransformerRandomInputsIds - }, { - 'testcase_name': '_evolved_transformer', - 'params_fn': _EvolvedTransformerParamsWithEmbeddings, - 'stack_cls': GPipeEvolvedTransformerStack, - 'inputs_fn': _EvolvedTransformerRandomInputsIds - }))) - def testGPipeTransformerMtModel(self, - params_fn, - stack_cls, - inputs_fn, - splits=1, - num_micro_batches=1, - use_task_embs=False): - batch = 4 - tf.flags.FLAGS.tpu_compatible = True - with self.session(): - with tf.variable_scope('transformer_test', reuse=tf.AUTO_REUSE): - params = params_fn( - splits=splits, - num_micro_batches=num_micro_batches, - num_decoder_layers=2, - has_softmax=True, - use_task_ids=use_task_embs) - params.state_dtype = tf.float32 - xformer = stack_cls(params) - - input_ids, id_paddings, tgt_inputs, tgt_paddings, input_task_ids, tgt_task_ids = ( - inputs_fn(batch=batch)) - labels = tf.ones([tgt_inputs.shape.as_list()[0], batch], dtype=tf.int32) - label_weights = tf.ones([tgt_inputs.shape.as_list()[0], batch]) - tf.random.set_seed(1234) - self.evaluate(tf.global_variables_initializer()) - xent, logits = xformer.FProp(xformer.theta, input_ids, id_paddings, - tgt_inputs, tgt_paddings, None, None, labels, - label_weights, None, None, input_task_ids, - tgt_task_ids) - xent_out, logits_out = self.evaluate([xent, logits]) - print('xent_out={}'.format(xent_out)) - print('logits_out={}'.format(logits_out)) - - -def _BatchMajorTransformerParams(splits=1, - num_micro_batches=1, - packed_input=True): - model_dim = 4 - params = layers_with_gpipe.GPipeBatchMajorTransformerStack.Params() - params.name = 'transformer' - params.model_dim = model_dim - params.packed_input = packed_input - params.num_decoder_layers = 4 - params.decoder_tpl.input_dim = model_dim - params.decoder_tpl.tr_atten_tpl.num_heads = 1 - params.decoder_tpl.tr_fflayer_tpl.hidden_dim = model_dim - params.decoder_tpl.mask_self_atten = True - params.decoder_tpl.has_aux_atten = True - params.num_encoder_layers = 2 - params.encoder_tpl.input_dim = model_dim - params.encoder_tpl.tr_atten_tpl.num_heads = 1 - params.encoder_tpl.tr_fflayer_tpl.hidden_dim = model_dim - params.state_dtype = tf.float32 - params.softmax_tpl.input_dim = model_dim - params.softmax_tpl.num_classes = 2 - - emb_params = params.emb_tpl - # Default config for the token embedding. - emb_params.token_emb.use_matmul = True - emb_params.token_emb.use_3d_weight_tensor = False - emb_params.token_emb.vocab_size = 10 - emb_params.token_emb.embedding_dim = model_dim - - # Default config for the position embedding. - emb_params.position_emb.embedding_dim = model_dim - emb_params.position_emb.trainable_scaling = False - params.splits = splits - params.num_micro_batches = num_micro_batches - params.random_seed = 0 - return params - - -def _BatchMajorTransformerRandomInputs(batch): - input_arr = np.array([[6, 4]] * batch) - paddings_arr = np.array([[0] * 2] * batch) - input_seg_arr = np.array([[0, 1]] * batch) - input_pos_arr = np.array([[0, 0]] * batch) - tgt_input_arr = np.array([[3, 7, 9]] * batch) - tgt_paddings_arr = np.array([[0] * 3] * batch) - tgt_seg_arr = np.array([[0, 1, 1]] * batch) - tgt_pos_arr = np.array([[0, 0, 1]] * batch) - inputs = tf.constant(input_arr.tolist(), dtype=tf.int32) - paddings = tf.constant(paddings_arr.tolist(), dtype=tf.float32) - input_seg = tf.constant(input_seg_arr.tolist(), dtype=tf.int32) - input_pos = tf.constant(input_pos_arr.tolist(), dtype=tf.int32) - tgt_inputs = tf.constant(tgt_input_arr.tolist(), dtype=tf.int32) - tgt_paddings = tf.constant(tgt_paddings_arr.tolist(), dtype=tf.float32) - tgt_seg = tf.constant(tgt_seg_arr.tolist(), dtype=tf.int32) - tgt_pos = tf.constant(tgt_pos_arr.tolist(), dtype=tf.int32) - return (inputs, paddings, tgt_inputs, tgt_paddings, input_seg, tgt_seg, - input_pos, tgt_pos) - - -class GPipeBatchMajorTransformerStackTest(test_utils.TestCase, - parameterized.TestCase - ): # was tf.test.TestCase - """Tests for GPipeBatchMajorTransformerStack layer.""" - - @parameterized.named_parameters( - { - 'testcase_name': '_one_split_one_mb_packed', - 'splits': 1, - 'num_micro_batches': 1, - 'packed_input': True - }, { - 'testcase_name': '_two_splits_one_mb_packed', - 'splits': 2, - 'num_micro_batches': 1, - 'packed_input': True - }, { - 'testcase_name': '_two_splits_two_mb_packed', - 'splits': 2, - 'num_micro_batches': 2, - 'packed_input': True - }, { - 'testcase_name': '_one_split_two_mb_packed', - 'splits': 1, - 'num_micro_batches': 2, - 'packed_input': True - }, { - 'testcase_name': '_one_split_one_mb_unpacked', - 'splits': 1, - 'num_micro_batches': 1, - 'packed_input': False - }, { - 'testcase_name': '_two_splits_one_mb_unpacked', - 'splits': 2, - 'num_micro_batches': 1, - 'packed_input': False - }, { - 'testcase_name': '_two_splits_two_mb_unpacked', - 'splits': 2, - 'num_micro_batches': 2, - 'packed_input': False - }, { - 'testcase_name': '_one_split_two_mb_unpacked', - 'splits': 1, - 'num_micro_batches': 2, - 'packed_input': False - }) - def testGPipeBatchMajorTransformerFProp(self, - splits=1, - num_micro_batches=1, - packed_input=True): - batch = 4 - tf.flags.FLAGS.tpu_compatible = True - with self.session(): - with tf.variable_scope('transformer_test', reuse=tf.AUTO_REUSE): - params = _BatchMajorTransformerParams(splits, num_micro_batches, - packed_input) - xformer = params.Instantiate() - (input_ids, id_paddings, tgt_inputs, tgt_paddings, input_seg, tgt_seg, - input_pos, tgt_pos) = ( - _BatchMajorTransformerRandomInputs(batch=batch)) - labels = tf.ones([batch, tgt_inputs.shape.as_list()[1]], dtype=tf.int32) - label_weights = tf.ones([batch, tgt_inputs.shape.as_list()[1]]) - tf.random.set_seed(1234) - self.evaluate(tf.global_variables_initializer()) - xent, logits = xformer.FProp(xformer.theta, input_ids, id_paddings, - tgt_inputs, tgt_paddings, input_seg, - tgt_seg, labels, label_weights, input_pos, - tgt_pos) - xent_out, logits_out = self.evaluate([xent, logits]) - print('xent_out={}'.format(xent_out)) - print('logits_out={}'.format(logits_out)) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/learner.py b/lingvo/core/learner.py deleted file mode 100644 index ffa47fba..00000000 --- a/lingvo/core/learner.py +++ /dev/null @@ -1,454 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""A learner optimizes a subset of variables according to a loss. - -It consists of a learning rate schedule, an optimizer, and gradient clipping -mechanisms. A BaseTask can have multiple learners, each optimizing a (usually -disjoint) subset of variables. -""" - -import re -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import optimizer -from lingvo.core import py_utils -from lingvo.core import schedule -from lingvo.core import summary_utils - - -class Learner(base_layer.BaseLayer): - """A training program layer. - - The layer takes a loss tensor as input and outputs a trainer op. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'l2_regularizer_weight', None, - 'If not None, L2 regularization to apply to the weights. ' - 'Otherwise, disable L2 regularization.') - p.Define( - 'loss_name', None, 'Name(s) of the loss(es) this learner to optimize. ' - 'If not set, use learner name directly. ' - 'If given as a list, the gradients will be combined via a ' - 'GradientCombiner created from p.gradient_combiner, which must be ' - 'specified as well.') - p.Define( - 'gradient_combiner', None, - 'Params of a gradient_combiner.GradientCombiner used to combine ' - 'gradients from multiple losses.') - p.Define( - 'l1_regularizer_weight', None, - 'If not None, L1 regularization to apply to the weights. ' - 'Otherwise, disable L1 regularization.') - p.Define('learning_rate', 0.0, 'learning rate to use.') - p.Define( - 'clip_gradient_norm_to_value', 0.0, - 'Clip gradient by global norm to this value. This is similar to ' - 'the bahaviour of tf.clip_by_global_norm, if you are looking for ' - 'tf.clip_by_norm refer to clip_gradient_single_norm_to_value. Note ' - 'these are mutually exclusive.') - p.Define( - 'clip_gradient_single_norm_to_value', 0.0, - 'Clip gradient by single tensor norm to this value. This is ' - 'similar to the bahaviour of tf.clip_by_norm. Note this is mutually ' - 'exlusive to using clip_gradient_norm_to_value.') - p.Define('grad_norm_to_clip_to_zero', 0.0, - 'Clip gradient to 0 if its norm exceeds this value.') - p.Define('grad_norm_tracker', None, 'Params for GradNormTracker.') - p.Define('optimizer', optimizer.Adam.Params(), 'Params for the optimizer.') - p.Define('lr_schedule', schedule.ContinuousSchedule.Params(), - 'Learning rate decay schedule.') - p.Define( - 'bprop_variable_filter', None, - 'If set, only backprop variables whose names partially match ' - 'this regexp (re.search).') - p.Define( - 'bprop_variable_exclusion', None, - 'If set, do not backprop variables whose names partially match ' - 'this regexp (re.search).') - p.Define( - 'grad_aggregation_method', tf.AggregationMethod.EXPERIMENTAL_TREE, - 'Specifies the method used to combine gradient terms. Accepted ' - 'values are constants defined in the class AggregationMethod.') - p.Define( - 'gate_gradients', False, - 'If True, add a tuple around the gradients returned for an ' - 'operations. This avoids some race conditions.') - p.Define('colocate_gradients_with_ops', True, - 'If True, try colocating gradients with the corresponding op.') - p.Define( - 'skip_zero_gradients', None, - 'If set, skips aggregating zero gradients while computing gradients.' - 'This helps in case where some weights may not be used in forward ' - 'computation, e.g., sparsely activated networks or switchable layers ' - 'in neural architectural search. ' - 'Possible values are: ' - 'None: do not skip zero gradients; ' - '"variable": skip if the entire variable gradients are almost zero; ' - '"weight": skip if the individual weight gradients are almost zero.') - p.Define('scale_gradients', True, - 'Whether to apply gradients adjustment and scaling.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - self._var_grads = None - self._eval_metrics = {} - if p.grad_norm_tracker: - self.CreateChild('grad_norm_tracker', p.grad_norm_tracker) - self.CreateChild('lr_schedule', p.lr_schedule) - self.CreateChild('optimizer', p.optimizer) - if isinstance(p.loss_name, (list, tuple)): - assert p.gradient_combiner - self.CreateChild('gradient_combiner', p.gradient_combiner) - else: - assert p.gradient_combiner is None - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - if self.params.grad_norm_tracker: - self.grad_norm_tracker.InstantiateVariables() - self.lr_schedule.InstantiateVariables() - self.optimizer.InstantiateVariables() - super()._CreateChildrenVariables() - - def GetVarGrads(self): - return self._var_grads - - def GetTrainableVariables(self, vmap): - p = self.params - pos = re.compile( - p.bprop_variable_filter) if p.bprop_variable_filter else None - neg = re.compile( - p.bprop_variable_exclusion) if p.bprop_variable_exclusion else None - - def VariableFilter(v): - """Returns True if variable v should be optimized by this learner.""" - if not v.trainable: - return False - - if pos and not pos.search(v.name): - tf.logging.info('%s: disabled by bprop_variable_filter: %s', p.name, - v.name) - return False - if neg and neg.search(v.name): - tf.logging.info('%s: disabled by bprop_variable_exclusion: %s', p.name, - v.name) - return False - return True - - return vmap.Filter(VariableFilter) - - def ApplyPostTrainingLoop(self, global_step): - """Applies any computation to run after each tpu trainining loop. - - Args: - global_step: Global step variable. - - Returns: - Ops to run after training loop ends. - """ - return self.optimizer.ApplyPostTrainingLoop(global_step) - - def LearningRate(self, step): - p = self.params - lrs = self.lr_schedule.Value(step) - lrs.set_shape([]) - self._AddEvalMetric('lr_schedule', lrs, tf.constant(1.0)) - return p.learning_rate * lrs - - def Apply(self, metrics, vmap, gradient_mask=None, gradient_adjuster=None): - """Computes updates on 'vmap' to optimize 'loss'. - - TODO(rpang): explore merging gradient_mask and gradient_adjuster. - - Args: - metrics: A Dict[str, (value, weight)], from which loss can be extracted - according to p.loss_name. - vmap: A `.NestedMap` object containing variables to optimize. - gradient_mask: if not None, a dict mapping variable names to a 0/1 scalar. - gradient_adjuster: if not None, a function that mutates a given var_grads. - - Returns: - (losses, op, eval_metrics), where - - losses is a list of scalar tensors; - - op is a tf.Operation to update variables; - - eval_metrics is a Dict[str, (value, weight)], where each value/weight - is a scalar tensor. - """ - # We apply gradients outside the name_scope to maintain backwards - # compatibility on variables created by self.optimizer.Apply(). - losses, var_grads = self._ComputeLossesAndGradients(metrics, vmap) - - var_grads, stats = self.AdjustGradients( - var_grads, - gradient_mask=gradient_mask, - gradient_adjuster=gradient_adjuster) - self._var_grads = var_grads - - assert self.theta.global_step is not None, self.theta - lr = self.LearningRate(self.theta.global_step) - - var_update_op = self.optimizer.Apply(lr, var_grads) - return losses, var_update_op, stats - - def _CustomComputeGradientsFn(self): - """Returns the compute_gradients_fn to use for py_utils.ComputeGradients.""" - return None # use the default function - - def _ComputeLossesAndGradients(self, metrics, vmap): - p = self.params - vmap = self.GetTrainableVariables(vmap) - - for v in vmap.Flatten(): - tf.logging.info('%s: bprop variable: %s', p.name, v.name) - - def LossAndGradients(metric_name): - """Returns (loss, var_grads) computed from metrics[metric_name].""" - metric = metrics.get(metric_name, None) - if metric is None: - raise ValueError('Loss %s not found in metrics %s' % - (metric_name, list(metrics.keys()))) - # TODO(b/154785713): pass (loss, loss_weight) to ComputeGradients(). - loss = metric[0] - return metric, self.optimizer.ComputeGradients( - loss, - vmap, - p.grad_aggregation_method, - p.colocate_gradients_with_ops, - p.gate_gradients, - compute_gradients_fn=self._CustomComputeGradientsFn(), - skip_zero_gradients=p.skip_zero_gradients, - skip_none_gradients=False) - - loss_name = p.loss_name or p.name - losses = [] - if isinstance(loss_name, (list, tuple)): - losses_and_grads = {} - variables = None - for metric_name in loss_name: - loss_metric, var_grads = LossAndGradients(metric_name) - losses_and_grads[metric_name] = py_utils.NestedMap( - loss_metric=loss_metric, - grads=tf.nest.map_structure(lambda vg: vg.grad, var_grads)) - current_vars = tf.nest.map_structure(lambda vg: vg.var, var_grads) - if variables is None: - variables = current_vars - else: - tf.nest.assert_same_structure(variables, current_vars) - losses.append(loss_metric[0]) - grads = self.gradient_combiner.Combine(variables, losses_and_grads) - var_grads = tf.nest.map_structure( - lambda v, g: py_utils.VarGrad(var=v, grad=g), variables, grads) - else: - loss_metric, var_grads = LossAndGradients(loss_name) - losses.append(loss_metric[0]) - - return losses, py_utils.SkipNoneGradients(var_grads) - - def AdjustGradients(self, - var_grads, - gradient_mask=None, - gradient_adjuster=None): - """Adjusts gradients according to learner params. - - Args: - var_grads: a `.NestedMap` whose values are (var, grad) pairs. - gradient_mask: if not None, a dict mapping variable names to a 0/1 scalar. - gradient_adjuster: if not None, a function that mutates a given var_grads. - - Returns: - (var_grads, eval_metrics), where var_grads is a `.NestedMap` whose values - (var, grad) pairs representing adjusted gradients. - """ - p = self.params - # L2 regularizer. - if p.l2_regularizer_weight is not None: - l2_loss, var_grads = py_utils.AdjustGradientsWithLpLoss( - var_grads, p.l2_regularizer_weight, p=2.0) - self._AddEvalMetric('l2_loss', l2_loss, tf.constant(1.0)) - - # L1 regularizer. - if p.l1_regularizer_weight is not None: - l1_loss, var_grads = py_utils.AdjustGradientsWithLpLoss( - var_grads, p.l1_regularizer_weight, p=1.0) - self._AddEvalMetric('l1_loss', l1_loss, tf.constant(1.0)) - - # Mask gradients only if the mask is set. - if gradient_mask: - var_grads = py_utils.MaskGradients(var_grads, gradient_mask) - - # Scale gradients, e.g., gradient clipping. - if p.scale_gradients: - scaled_vars = self.ScaleGradients( - var_grads, gradient_adjuster=gradient_adjuster) - var_grads = scaled_vars.final_var_grads - - # Histogram summary. - summary_utils.CollectVarHistogram(var_grads) - return var_grads, self._eval_metrics - - def _GetGlobalGradScale(self, all_grad_norm, has_nan_or_inf): - """Returns a scaling factor for all gradients according to their norm. - - In case there are NaN or Inf values the function will return 0.0. - - Args: - all_grad_norm: A scalar represeting the total norm of all vars. - has_nan_or_inf: A scalar of 0 or 1, indicating whether there is any NaN or - Inf in input gradients. - - Returns: - The gradient scale. 0 if gradient updates should be skipped for the step. - """ - p = self.params - # Computes gradient's scale. - grad_scale = tf.constant(1.0) - if p.clip_gradient_norm_to_value: - # If all_grad_norm > p.clip_gradient_norm_to_value, scales - # all_grads so that the norm is 1.0. - grad_scale = tf.minimum(1.0, - p.clip_gradient_norm_to_value / all_grad_norm) - - if p.grad_norm_to_clip_to_zero: - # If all_grad_norm > p.grad_norm_to_clip_to_zero, treats - # grad_scale as 0. This way, we ignore this step. - grad_scale *= tf.cast(all_grad_norm < p.grad_norm_to_clip_to_zero, - p.dtype) - - if p.grad_norm_tracker: - grad_scale *= self.grad_norm_tracker.FPropDefaultTheta( - all_grad_norm, has_nan_or_inf) - - # Force grad_scale to be 0 if there is any NaN or Inf in gradients. - grad_scale = tf.where(has_nan_or_inf, 0.0, grad_scale) - - return grad_scale - - def ScaleGradients(self, var_grads, gradient_adjuster=None): - """Scales gradients according to training params. - - Args: - var_grads: a `.NestedMap` whose values are (var, grad) pairs. - gradient_adjuster: if not None, a function that mutates a given var_grads. - - Returns: - A `.NestedMap` containing - - - final_var_grads: a `.NestedMap` whose values are (var, grad) pairs, - where gradients have already been scaled. - - grad_scale: the gradient scale. 0 if gradient updates should be skipped - for the step. (Optional, only returned in case global norm clipping is - used.) - """ - p = self.params - - # Computes gradients' norm and adds their summaries. Note that all_grad_norm - # may be nan, which may cause grad_scale to be nan. - for name, vg in var_grads.FlattenItems(): - summary_utils.AddNormSummary( - py_utils.SanitizeScopeKey(name) + '/' + p.name, vg) - flatten = py_utils.Flatten(var_grads) - all_grad_norm = tf.sqrt(py_utils.SumSquared([g for (_, g) in flatten])) - all_var_norm = tf.sqrt(py_utils.SumSquared([v for (v, _) in flatten])) - grad_norm_is_nan_or_inf = tf.math.logical_or( - tf.math.is_nan(all_grad_norm), tf.math.is_inf(all_grad_norm)) - - # Optional gradient adjustment. Note that this happens after computing - # all_grad_norm. - if gradient_adjuster is not None: - tf.logging.info('gradient_adjuster=%s', gradient_adjuster) - var_grads = gradient_adjuster(var_grads) - - # Handles NaN/Inf gradients. - has_nan_or_inf = py_utils.HasNanOrInfGradient(var_grads) - # Grad norm can still be inf even if none of the individual grad is inf. - has_nan_or_inf = tf.math.logical_or(has_nan_or_inf, grad_norm_is_nan_or_inf) - self._AddEvalMetric('has_nan_or_inf', has_nan_or_inf, tf.constant(1.0)) - - return_values = py_utils.NestedMap() - if p.clip_gradient_single_norm_to_value: - # Currently using both types of clipping simultaneously is unsupported. - if p.clip_gradient_norm_to_value: - raise ValueError('Cannot use clip_gradient_single_norm_to_value=%f and ' - 'clip_gradient_norm_to_value=%f.' % - (p.clip_gradient_single_norm_to_value, - p.clip_gradient_norm_to_value)) - final_var_grads = py_utils.ApplyGradNormClipping( - var_grads, p.clip_gradient_single_norm_to_value) - - else: - grad_scale = self._GetGlobalGradScale(all_grad_norm, has_nan_or_inf) - self._AddEvalMetric('grad_norm/all', all_grad_norm, tf.constant(1.0)) - self._AddEvalMetric('var_norm/all', all_var_norm, tf.constant(1.0)) - self._AddEvalMetric('grad_scale_all', grad_scale, tf.constant(1.0)) - final_var_grads = py_utils.ApplyGradMultiplier(var_grads, grad_scale) - return_values.grad_scale = grad_scale - - return_values.final_var_grads = final_var_grads - return return_values - - def _AddEvalMetric(self, key, value, weight): - self._eval_metrics[key] = (value, weight) - -_LEGACY_LEARNER_PARAMS = [ - 'bprop_variable_filter', - 'bprop_variable_exclusion', - 'clip_gradient_norm_to_value', - 'clip_gradient_single_norm_to_value', - 'colocate_gradients_with_ops', - 'gate_gradients', - 'scale_gradients', - 'grad_aggregation_method', - 'grad_norm_to_clip_to_zero', - 'grad_norm_tracker', - 'l1_regularizer_weight', - 'l2_regularizer_weight', - 'learning_rate', - 'lr_schedule', - 'optimizer', -] - - -def ExtractLearnerFromLegacyParams(tp, cls=Learner): - """Extracts legacy learner params from 'tp' to a Learner params. - - Args: - tp: BaseTask training params (p.train). Its legacy params will be cleared to - be None after the conversion. - cls: Learner class where we set the params. - - Returns: - A params for Learner. - """ - lp = cls.Params() - lp.name = 'loss' - for k, v in tp.IterParams(): - if k not in _LEGACY_LEARNER_PARAMS: - tf.logging.info( - 'Ignoring legacy param %s=%s for optimization program', k, v) - continue - setattr(lp, k, v) - setattr(tp, k, None) - for line in lp.ToText().split('\n'): - tf.logging.info('Learner params: %s', line) - return lp diff --git a/lingvo/core/learner_test.py b/lingvo/core/learner_test.py deleted file mode 100644 index 70d2629f..00000000 --- a/lingvo/core/learner_test.py +++ /dev/null @@ -1,154 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lingvo.core.learner.""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import gradient_combiner -from lingvo.core import learner -from lingvo.core import optimizer -from lingvo.core import py_utils -from lingvo.core import test_utils - - -class TestLayer(base_layer.BaseLayer): - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - pc = py_utils.WeightParams( - shape=[], - init=py_utils.WeightInit.Constant(0), - dtype=self.params.dtype, - collections=self._VariableCollections()) - self.CreateVariable('hello', pc) - self.CreateVariable('world', pc) - self.CreateVariable('moon', pc) - self.CreateVariable('mars', pc, trainable=False) - - def Loss(self, theta): - return self.MainLoss(theta) + self.AuxLoss(theta) - - def MainLoss(self, theta): - return theta.hello - - def AuxLoss(self, theta): - return -2 * theta.world - - -class GradientSum(gradient_combiner.GradientCombiner): - - def Combine(self, vmap, losses_and_grads): - """Computes the sum of gradients on the variables.""" - - def GradSum(v, *gs): - tf.logging.info('GradSum: %s: %s', v, gs) - if all(g is None for g in gs): - return None - return tf.add_n([g for g in gs if g is not None]) - - grads = [l_and_g.grads for l_and_g in losses_and_grads.values()] - tf.logging.info('grads: %s', grads) - return tf.nest.map_structure(GradSum, vmap, *grads) - - -class LearnerTest(test_utils.TestCase): - - def testBasic(self): - learner_p = learner.Learner.Params().Set( - name='learner', learning_rate=.1, optimizer=optimizer.SGD.Params()) - var_grads, updated_vars, _ = self._testLearner(learner_p) - tf.logging.info('var_grads=%s, updated_vars=%s', var_grads, updated_vars) - self.assertAllClose(var_grads, {'hello': (0., 1.), 'world': (0., -2.)}) - self.assertAllClose(updated_vars, { - 'hello': -0.1, - 'world': 0.2, - 'moon': 0., - 'mars': 0. - }) - - def testMultiLoss(self): - learner_p = learner.Learner.Params().Set( - name='learner', learning_rate=.1, optimizer=optimizer.SGD.Params()) - learner_p.loss_name = ('main_loss', 'aux_loss') - learner_p.gradient_combiner = GradientSum.Params() - var_grads, updated_vars, _ = self._testLearner(learner_p) - self.assertAllClose(var_grads, {'hello': (0., 1.), 'world': (0., -2.)}) - self.assertAllClose(updated_vars, { - 'hello': -0.1, - 'world': 0.2, - 'moon': 0., - 'mars': 0. - }) - - def testBPropVariableFilter(self): - learner_p = learner.Learner.Params().Set( - name='learner', - learning_rate=.1, - optimizer=optimizer.SGD.Params(), - bprop_variable_filter='ello') - var_grads, updated_vars, eval_metrics = self._testLearner(learner_p) - # Only 'hello' is updated. - self.assertAllClose(var_grads, {'hello': (0., 1.)}) - self.assertAllClose(updated_vars, { - 'hello': -0.1, - 'world': 0., - 'moon': 0., - 'mars': 0. - }) - self.assertIn('grad_scale_all', eval_metrics) - - def testBPropVariableExclusion(self): - learner_p = learner.Learner.Params().Set( - name='learner', - learning_rate=.1, - optimizer=optimizer.SGD.Params(), - bprop_variable_filter='o', - bprop_variable_exclusion='ello') - var_grads, updated_vars, _ = self._testLearner(learner_p) - # Only 'world' is updated. - self.assertAllClose(var_grads, {'world': (0., -2.)}) - self.assertAllClose(updated_vars, { - 'hello': 0., - 'world': 0.2, - 'moon': 0., - 'mars': 0. - }) - - def _testLearner(self, learner_p): - tf.train.get_or_create_global_step() # needed for lr_schedule - lrnr = learner_p.Instantiate() - layer = TestLayer.Params().Set(name='test').Instantiate() - if isinstance(learner_p.loss_name, (list, tuple)): - main_loss = layer.MainLoss(layer.theta) - aux_loss = layer.AuxLoss(layer.theta) - metrics = {'main_loss': (main_loss, 1.), 'aux_loss': (aux_loss, 1.)} - expected_losses = [main_loss, aux_loss] - else: - loss = layer.Loss(layer.theta) - metrics = {learner_p.name: (loss, 1.)} - expected_losses = [loss] - losses, update_op, eval_metrics = lrnr.Apply(metrics, layer.vars) - self.assertAllEqual(losses, expected_losses) - with self.session(): - self.evaluate(tf.global_variables_initializer()) - var_grads = self.evaluate(lrnr.GetVarGrads().Transform(tuple)) - update_op.run() - updated_vars = self.evaluate(layer.vars) - return var_grads, updated_vars, eval_metrics - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/lstm_frnn_layer.py b/lingvo/core/lstm_frnn_layer.py deleted file mode 100644 index 28a7cfd9..00000000 --- a/lingvo/core/lstm_frnn_layer.py +++ /dev/null @@ -1,253 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Functional LSTM RNN layer with LSTM cells fused into FRNN to run input ... - -projections in parallel. -""" - -from lingvo import compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils -from lingvo.core import recurrent -from lingvo.core import rnn_cell -from lingvo.core import rnn_layers - - -class LSTMCellExt: - """Extends LSTM-based cell classes with extra methods for parallelizing ... - - input projections across steps. - """ - - def ProjectInputSequence(self, theta, inputs): - """Applies input projection for the entire sequence. - - Args: - theta: a NestedMap of layer weights. Notably, it's expected to contain - separate weight tensors for input and hidden state projections, for - performance reasons, under the key 'wm_i' (input) and 'wm_h' - inputs: A NestedMap with the following fields: - - act: A list of Tensors of shape [seqlen, batch, input_dim]. - - Returns: - A Tensor of shape [seqlen, batch, 4 * hidden_dim]. - """ - assert isinstance(inputs.act, list) - if len(inputs.act) > 1: - x = tf.concat(inputs.act, -1) - else: - x = inputs.act[0] - # [T, B, 4 * H] - proj_inputs = tf.einsum('TBD,DH->TBH', x, theta.wm_i) - return proj_inputs - - def _MixWithProjectedInput(self, theta, state0, inputs): - """Computes _Mix() with inputs already projected. - - Args: - theta: a NestedMap of layer weights. Notably, it's expected to contain - separate weight tensors for input and hidden state projections, for - performance reasons, under the key 'wm_i' (input) and 'wm_h' - state0: A NestedMap with the same structure as return value of - `self.zero_state()`. - inputs: A Tensor of shape [batch, 4 * hidden_dim]. - - Returns: - A Tensor of the same shape as `inputs`. - """ - proj_m = tf.matmul(state0.m, theta.wm_h) - return inputs + proj_m - - def FPropWithProjectedInput(self, theta, state0, inputs): - """FProp with inputs already projected. - - This method is for parallelizing the input projection across time steps to - accelerate training. - - The following are equivalent: - - >>> inputs = - >>> paddings = tf.zeros([T, B]) - >>> theta = cell.theta - >>> state = cell.zero_state(theta, B) - - # a. Use FProp(). - >>> for i in range(T): - ... state, _ = cell.FProp(theta, inputs[i, :, :], paddings, state) - - # b. Use FPropWithProjectedInput(). - >>> proj_inputs = cell.ProjectInputSequence(theta, inputs) - >>> for i in range(T): - ... state, _ = cell.FPropWithProjectedInputs( - ... theta, proj_inputs[i, :, :], paddings, state) - - Args: - theta: a NestedMap of layer weights. Notably, it's expected to contain - separate weight tensors for input and hidden state projections, for - performance reasons, under the key 'wm_i' (input) and 'wm_h' (hidden - state). - state0: A NestedMap with the same structure as return value of - `self.zero_state()`. - inputs: A NestedMap with the following fields: - - proj_inputs: A single Tensors of shape [batch, 4 * hidden_dim]. - - padding: A Tensor of shape [batch, 1]. - - reset_mask: A Tensor of shape [batch, 1]. - - Returns: - state1: A NestedMap of the same structure as `state0`. - extras: Intermediate results to facilitate backprop. A NestedMap. - """ - if self.params.reset_cell_state: - state0_modified = self._ResetState(state0.DeepCopy(), inputs) - else: - state0_modified = state0 - xmw = self._MixWithProjectedInput(theta, state0_modified, - inputs.proj_inputs) - gates_input = inputs.copy() - gates_input.act = [inputs.proj_inputs] - state1 = self._Gates(xmw, theta, state0_modified, gates_input) - return state1, py_utils.NestedMap() - - -class LSTMCellSimpleExt(rnn_cell.LSTMCellSimple, LSTMCellExt): - """Extends LSTMCellSimple with extra methods for parallelizing ... - - input projections across steps. - """ - pass - - -class LayerNormalizedLSTMCellSimpleExt(rnn_cell.LayerNormalizedLSTMCellSimple, - LSTMCellExt): - """Extends LayerNormalizedLSTMCellSimple with extra methods for ... - - parallelizing input projections across steps. - """ - pass - - -class LayerNormalizedLSTMCellLeanExt(rnn_cell.LayerNormalizedLSTMCellLean, - LSTMCellExt): - """Extends LayerNormalizedLSTMCellLean with extra methods for parallelizing ... - - input projections across steps. - """ - pass - - -class LstmFRNN(base_layer.BaseLayer): - """A FRNN for LSTMCellSimple or LayerNormalizedLSTMCellLean cell. - - It exploits the parallelism in input projection across time steps, and is in - general faster than the combination of LayerNormalizedLSTMCellLean and FRNN. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('packed_input', False, 'To reset states for packed inputs.') - p.Define( - 'cell', None, - 'Configs for the RNN cell. Supported classes are LSTMCellSimpleExt, ' - 'LayerNormalizedLSTMCellLeanExt.') - p.Define('reverse', False, - 'Whether or not to unroll the sequence in reversed order.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.cell.cls not in (LSTMCellSimpleExt, LayerNormalizedLSTMCellSimpleExt, - LayerNormalizedLSTMCellLeanExt): - raise ValueError( - 'Only LSTMCellSimpleExt, LayerNormalizedLSTMCellSimpleExt and ' - 'LayerNormalizedLSTMCellLeanExt are supported, got {}.'.format( - p.cell.cls.__name__)) - self.CreateChild('cell', p.cell) - - def zero_state(self, theta, batch_size): - return self.cell.zero_state(theta.cell, batch_size) - - def FProp(self, theta, inputs, paddings, state0=None, segment_id=None): - """Computes LSTM forward pass. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: A single tensor or a tuple of tensors with cardinality equal to - rnn_cell.inputs_arity. For every input tensor, the first dimension is - assumed to be time, second dimension batch, and third dimension depth. - paddings: A tensor. First dim is time, second dim is batch, and third dim - is expected to be 1. - state0: If not None, the initial rnn state in a `.NestedMap`. Defaults to - the cell's zero-state. - segment_id: A tensor to support packed inputs. First dim is time, second - dim is batch, and third dim is expected to be 1. - - Returns: - A tensor of [time, batch, dims]. - The final recurrent state. - """ - p = self.params - rcell = self.cell - assert isinstance(rcell, (rnn_cell.RNNCell)) - - if not isinstance(inputs, (list, tuple)): - inputs = [inputs] - - # Slicing wm to wm_{i,h} outside the loop to get 20% speedup over regular - # LSTM baseline. - # Keeping slicing within the loop gives only < 3% speedup. - cell_theta = theta.cell.copy() - num_input_nodes = p.cell.num_input_nodes - cell_theta['wm_i'] = cell_theta.wm[:num_input_nodes, :] - cell_theta['wm_h'] = cell_theta.wm[num_input_nodes:, :] - tf.logging.vlog(1, 'cell_theta: %r', cell_theta) - if p.packed_input: - assert segment_id is not None - reset_mask = rnn_layers.GeneratePackedInputResetMask( - segment_id, is_reverse=False) - reset_mask = py_utils.HasShape(reset_mask, tf.shape(paddings)) - else: - reset_mask = tf.zeros_like(paddings) - - if p.reverse: - inputs = [tf.reverse(x, [0]) for x in inputs] - paddings = tf.reverse(paddings, [0]) - reset_mask = tf.reverse(reset_mask, [0]) - - if not state0: - batch_size = py_utils.GetShape(paddings)[1] - state0 = rcell.zero_state(cell_theta, batch_size) - - # [T, B, H] - proj_inputs = rcell.ProjectInputSequence(cell_theta, - py_utils.NestedMap(act=inputs)) - proj_inputs = py_utils.NestedMap( - proj_inputs=proj_inputs, padding=paddings, reset_mask=reset_mask) - - acc_state, final_state = recurrent.Recurrent( - theta=cell_theta, - state0=state0, - inputs=proj_inputs, - cell_fn=rcell.FPropWithProjectedInput, - cell_type=rcell.layer_type, - accumulator_layer=self, - allow_implicit_capture=p.allow_implicit_capture) - - act = rcell.GetOutput(acc_state) - if p.reverse: - act = tf.reverse(act, [0]) - return act, final_state diff --git a/lingvo/core/lstm_frnn_layer_test.py b/lingvo/core/lstm_frnn_layer_test.py deleted file mode 100644 index b7ab9fc6..00000000 --- a/lingvo/core/lstm_frnn_layer_test.py +++ /dev/null @@ -1,231 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for lingvo.core.lstm_frnn_layer.""" - -from absl.testing import parameterized - -from lingvo import compat as tf -from lingvo.core import lstm_frnn_layer -from lingvo.core import py_utils -from lingvo.core import rnn_cell -from lingvo.core import rnn_layers -from lingvo.core import test_utils - -import numpy as np - -_INIT_RANDOM_SEED = 2020 -_RANDOM_SEED = 2019 - - -class LSTMCellExtTest(test_utils.TestCase, parameterized.TestCase): - - def _GetParams(self, num_hidden_nodes=None): - params = lstm_frnn_layer.LayerNormalizedLSTMCellLeanExt.Params() - params.name = 'lstm' - params.output_nonlinearity = True - params.params_init = py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED) - params.vn.global_vn = False - params.vn.per_step_vn = False - params.num_input_nodes = 2 - params.num_output_nodes = 2 - if num_hidden_nodes: - params.num_hidden_nodes = num_hidden_nodes - params.zo_prob = 0.0 - params.random_seed = _RANDOM_SEED - return params - - def testLayerNormalizedLSTMCellLeanExt(self): - cell_p = self._GetParams() - - seqlen, batch, input_dim = 4, 2, 2 - inputs = tf.convert_to_tensor( - np.random.rand(seqlen, batch, input_dim).astype(np.float32)) - input_lens = np.random.randint(1, seqlen + 1, size=batch) - paddings = 1. - tf.sequence_mask( - input_lens, maxlen=seqlen, dtype=tf.float32) - paddings = tf.transpose(paddings) - reset_mask = tf.zeros((seqlen, batch), tf.float32) - m0 = tf.convert_to_tensor( - np.random.rand(batch, input_dim).astype(np.float32)) - c0 = tf.convert_to_tensor( - np.random.rand(batch, input_dim).astype(np.float32)) - state0 = py_utils.NestedMap(m=m0, c=c0) - - with self.session(): - cell = cell_p.Instantiate() - self.evaluate(tf.global_variables_initializer()) - - # The canonical path - state = state0 - for i in range(seqlen): - state, _ = cell.FPropDefaultTheta( - state, - py_utils.NestedMap( - act=[inputs[i, :, :]], - padding=paddings[i, :, tf.newaxis], - reset_mask=reset_mask[i, :, tf.newaxis])) - expected_state = self.evaluate(state) - - # Taking input projection outside of the loop. - cell_theta = cell.theta.copy() - cell_theta.wm_i = cell_theta.wm[:cell.params.num_input_nodes, :] - cell_theta.wm_h = cell_theta.wm[cell.params.num_input_nodes:, :] - proj_inputs = cell.ProjectInputSequence(cell_theta, - py_utils.NestedMap(act=[inputs])) - state = state0 - for i in range(seqlen): - state, _ = cell.FPropWithProjectedInput( - cell_theta, state, - py_utils.NestedMap( - proj_inputs=proj_inputs[i, :, :], - padding=paddings[i, :, tf.newaxis], - reset_mask=reset_mask[i, :, tf.newaxis])) - actual_state = self.evaluate(state) - - tf.logging.info('expected_state:{}'.format(expected_state)) - tf.logging.info('actual_state:{}'.format(actual_state)) - self.assertAllClose(expected_state.m, actual_state.m) - self.assertAllClose(expected_state.c, actual_state.c) - - -class LstmFRNNTest(test_utils.TestCase, parameterized.TestCase): - - def _SetCellParams(self, cell_p, num_hidden_nodes=None): - params = cell_p - params.name = cell_p.cls.__name__ - params.output_nonlinearity = True - params.params_init = py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED) - params.vn.global_vn = False - params.vn.per_step_vn = False - params.num_input_nodes = 7 - params.num_output_nodes = 9 - if num_hidden_nodes: - params.num_hidden_nodes = num_hidden_nodes - params.zo_prob = 0.0 - params.random_seed = _RANDOM_SEED - - def _GetTestInputs(self, packed_input): - seqlen, batch, input_dim, output_dim = 4, 5, 7, 9 - inputs = tf.convert_to_tensor( - np.random.rand(seqlen, batch, input_dim).astype(np.float32)) - input_lens = np.random.randint(1, seqlen + 1, size=batch) - padding = 1. - tf.sequence_mask(input_lens, maxlen=seqlen, dtype=tf.float32) - padding = tf.transpose(padding)[:, :, tf.newaxis] - segment_id = None - if packed_input: - segment_id = tf.convert_to_tensor( - np.random.randint(0, seqlen, (seqlen, batch, 1), np.int32)) - - m = tf.convert_to_tensor( - np.random.rand(batch, output_dim).astype(np.float32)) - c = tf.convert_to_tensor( - np.random.rand(batch, output_dim).astype(np.float32)) - return inputs, padding, m, c, segment_id - - def _testHelper(self, base_frnn_p, frnn_p, packed_input=False): - inputs, padding, m0, c0, segment_id = self._GetTestInputs(packed_input) - base_frnn = base_frnn_p.Instantiate() - frnn = frnn_p.Instantiate() - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - - state0 = py_utils.NestedMap(m=m0, c=c0) - act, state = base_frnn.FPropDefaultTheta( - inputs, padding, state0=state0, segment_id=segment_id) - # Compute grads - loss = -tf.math.log( - tf.sigmoid((tf.reduce_sum(tf.math.square(act)) + - tf.reduce_sum(state.m * state.c * state.c)))) - grads = tf.gradients(loss, base_frnn.vars.Flatten()) - - expected_act, expected_state, expected_grads = self.evaluate( - [act, state, grads]) - - act, state = frnn.FPropDefaultTheta( - inputs, padding, state0=state0, segment_id=segment_id) - # Compute grads - loss = -tf.math.log( - tf.sigmoid((tf.reduce_sum(tf.math.square(act)) + - tf.reduce_sum(state.m * state.c * state.c)))) - grads = tf.gradients(loss, frnn.vars.Flatten()) - - actual_act, actual_state, actual_grads = self.evaluate( - [act, state, grads]) - - tf.logging.info('expected_act:{}'.format(expected_act)) - tf.logging.info('actual_act:{}'.format(actual_act)) - - tf.logging.info('expected_state:{}'.format(expected_state)) - tf.logging.info('actual_state:{}'.format(actual_state)) - - tf.logging.info('expected_grads:{}'.format(expected_grads)) - tf.logging.info('actual_grads:{}'.format(actual_grads)) - - self.assertAllClose(expected_act, actual_act) - self.assertAllClose(expected_state.m, actual_state.m) - self.assertAllClose(expected_state.c, actual_state.c) - for (vname, _), expected, actual in zip(frnn.vars.FlattenItems(), - expected_grads, actual_grads): - self.assertAllClose(expected, actual, msg=vname) - - @parameterized.named_parameters( - ('HasBias', True), - ('NoBias', False), - ('HasBiasPackedInputs', True, True), - ('NoBiasPackedInputs', False, True), - ) - def testLSTMCellSimple(self, enable_lstm_bias, packed_inputs=False): - base_cell_p = rnn_cell.LSTMCellSimple.Params() - base_cell_p.enable_lstm_bias = enable_lstm_bias - self._SetCellParams(base_cell_p) - base_frnn_p = rnn_layers.FRNN.Params().Set( - name='base_frnn', cell=base_cell_p) - - cell_p = lstm_frnn_layer.LSTMCellSimpleExt.Params() - cell_p.enable_lstm_bias = enable_lstm_bias - self._SetCellParams(cell_p) - frnn_p = lstm_frnn_layer.LstmFRNN.Params().Set(name='frnn', cell=cell_p) - self._testHelper(base_frnn_p, frnn_p, packed_inputs) - - def testLayerNormalizedLSTMCellSimple(self): - base_cell_p = rnn_cell.LayerNormalizedLSTMCellSimple.Params() - base_cell_p.enable_lstm_bias = True - self._SetCellParams(base_cell_p) - base_frnn_p = rnn_layers.FRNN.Params().Set( - name='base_frnn', cell=base_cell_p) - - cell_p = lstm_frnn_layer.LayerNormalizedLSTMCellSimpleExt.Params() - cell_p.enable_lstm_bias = True - self._SetCellParams(cell_p) - frnn_p = lstm_frnn_layer.LstmFRNN.Params().Set(name='frnn', cell=cell_p) - self._testHelper(base_frnn_p, frnn_p) - - def testLayerNormalizedLSTMCellLean(self): - base_cell_p = rnn_cell.LayerNormalizedLSTMCellLean.Params() - base_cell_p.enable_lstm_bias = True - self._SetCellParams(base_cell_p) - base_frnn_p = rnn_layers.FRNN.Params().Set( - name='base_frnn', cell=base_cell_p) - - cell_p = lstm_frnn_layer.LayerNormalizedLSTMCellLeanExt.Params() - cell_p.enable_lstm_bias = True - self._SetCellParams(cell_p) - frnn_p = lstm_frnn_layer.LstmFRNN.Params().Set(name='frnn', cell=cell_p) - self._testHelper(base_frnn_p, frnn_p) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/matrix_functions.py b/lingvo/core/matrix_functions.py deleted file mode 100644 index f2f9f5c8..00000000 --- a/lingvo/core/matrix_functions.py +++ /dev/null @@ -1,175 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Matrix functions contains iterative methods for M^p.""" - -import functools -import lingvo.compat as tf - - -def matrix_square_root(mat_a, mat_a_size, iter_count=100, ridge_epsilon=1e-4): - """Iterative method to get matrix square root. - - Stable iterations for the matrix square root, Nicholas J. Higham - - Page 231, Eq 2.6b - http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.6.8799&rep=rep1&type=pdf - - Args: - mat_a: the symmetric PSD matrix whose matrix square root be computed - mat_a_size: size of mat_a. - iter_count: Maximum number of iterations. - ridge_epsilon: Ridge epsilon added to make the matrix positive definite. - - Returns: - mat_a^0.5 - """ - - def _iter_condition(i, unused_mat_y, unused_old_mat_y, unused_mat_z, - unused_old_mat_z, err, old_err): - """This method require that we check for divergence every step.""" - return tf.math.logical_and(i < iter_count, err < old_err) - - def _iter_body(i, mat_y, unused_old_mat_y, mat_z, unused_old_mat_z, err, - unused_old_err): - """Iterative method to compute the square root of matrix.""" - current_iterate = 0.5 * (3.0 * identity - tf.matmul(mat_z, mat_y)) - current_mat_y = tf.matmul(mat_y, current_iterate) - current_mat_z = tf.matmul(current_iterate, mat_z) - # Compute the error in approximation. - mat_sqrt_a = current_mat_y * tf.sqrt(norm) - mat_a_approx = tf.matmul(mat_sqrt_a, mat_sqrt_a) - residual = mat_a - mat_a_approx - current_err = tf.sqrt(tf.reduce_sum(residual * residual)) / norm - return i + 1, current_mat_y, mat_y, current_mat_z, mat_z, current_err, err - - identity = tf.eye(tf.cast(mat_a_size, tf.int32)) - mat_a = mat_a + ridge_epsilon * identity - norm = tf.sqrt(tf.reduce_sum(mat_a * mat_a)) - mat_init_y = mat_a / norm - mat_init_z = identity - init_err = norm - - _, _, prev_mat_y, _, _, _, _ = tf.while_loop(_iter_condition, _iter_body, [ - 0, mat_init_y, mat_init_y, mat_init_z, mat_init_z, init_err, - init_err + 1.0 - ]) - return prev_mat_y * tf.sqrt(norm) - - -def inlined_matrix_inverse_pth_root(mat_g, - mat_g_size, - alpha, - iter_count=100, - error_tolerance=1e-6, - ridge_epsilon=1e-6): - """Computes mat_g^alpha, where alpha = -1/p, p is one of 2, 4, or 8. - - We use an iterative Schur-Newton method from equation 3.2 on page 9 of: - - A Schur-Newton Method for the Matrix p-th Root and its Inverse - by Chun-Hua Guo and Nicholas J. Higham - SIAM Journal on Matrix Analysis and Applications, - 2006, Vol. 28, No. 3 : pp. 788-804 - https://pdfs.semanticscholar.org/0abe/7f77433cf5908bfe2b79aa91af881da83858.pdf - - Args: - mat_g: the symmetric PSD matrix whose power it to be computed - mat_g_size: size of mat_g. - alpha: exponent, must be -1/p for p a positive integer. - iter_count: Maximum number of iterations. - error_tolerance: Error indicator, useful for early termination. - ridge_epsilon: Ridge epsilon added to make the matrix positive definite. - - Returns: - mat_g^alpha - """ - alpha = tf.cast(alpha, tf.float64) - neg_alpha = -1.0 * alpha - exponent = 1.0 / neg_alpha - identity = tf.eye(tf.cast(mat_g_size, tf.int32), dtype=tf.float64) - - def _unrolled_mat_pow_2(mat_m): - """Computes mat_m^2.""" - return tf.matmul(mat_m, mat_m) - - def _unrolled_mat_pow_4(mat_m): - """Computes mat_m^4.""" - mat_pow_2 = _unrolled_mat_pow_2(mat_m) - return tf.matmul(mat_pow_2, mat_pow_2) - - def _unrolled_mat_pow_8(mat_m): - """Computes mat_m^4.""" - mat_pow_4 = _unrolled_mat_pow_4(mat_m) - return tf.matmul(mat_pow_4, mat_pow_4) - - def mat_power(mat_m, p): - """Computes mat_m^p, for p == 2 or 4 or 8. - - Args: - mat_m: a square matrix - p: a positive integer - - Returns: - mat_m^p - """ - log2_p = tf.math.log(p) / tf.math.log(tf.constant(2.0, dtype=p.dtype)) - return tf.switch_case( - tf.cast(tf.math.round(log2_p), tf.int32), { - 0: functools.partial(_unrolled_mat_pow_2, mat_m), - 1: functools.partial(_unrolled_mat_pow_4, mat_m), - 2: functools.partial(_unrolled_mat_pow_8, mat_m), - }) - - def _iter_condition(i, unused_mat_m, unused_mat_h, unused_old_mat_h, error, - run_step): - return tf.math.logical_and( - i < iter_count, tf.math.logical_or(error > error_tolerance, run_step)) - - def _iter_body(i, mat_m, mat_h, unused_old_mat_h, error, unused_run_step): - mat_m_i = (1 - alpha) * identity + alpha * mat_m - new_mat_m = tf.matmul(mat_power(mat_m_i, exponent), mat_m) - new_mat_h = tf.matmul(mat_h, mat_m_i) - new_error = tf.reduce_max(tf.abs(new_mat_m - identity)) - return (i + 1, new_mat_m, new_mat_h, mat_h, new_error, - new_error < error * 1.2) - - if mat_g_size == 1: - mat_h = tf.pow(mat_g + ridge_epsilon, alpha) - else: - damped_mat_g = mat_g + ridge_epsilon * identity - z = (1 - 1 / alpha) / (2 * tf.norm(damped_mat_g)) - # The best value for z is - # (1 - 1/alpha) * (c_max^{-alpha} - c_min^{-alpha}) / - # (c_max^{1-alpha} - c_min^{1-alpha}) - # where c_max and c_min are the largest and smallest singular values of - # damped_mat_g. - # The above estimate assumes that c_max > c_min * 2^p. (p = -1/alpha) - # Can replace above line by the one below, but it is less accurate, - # hence needs more iterations to converge. - # z = (1 - 1/alpha) / tf.trace(damped_mat_g) - # If we want the method to always converge, use z = 1 / norm(damped_mat_g) - # or z = 1 / tf.trace(damped_mat_g), but these can result in many - # extra iterations. - new_mat_m_0 = damped_mat_g * z - new_error = tf.reduce_max(tf.abs(new_mat_m_0 - identity)) - new_mat_h_0 = identity * tf.pow(z, neg_alpha) - _, mat_m, mat_h, old_mat_h, error, convergence = tf.while_loop( - _iter_condition, _iter_body, - [0, new_mat_m_0, new_mat_h_0, new_mat_h_0, new_error, True]) - error = tf.reduce_max(tf.abs(mat_m - identity)) - is_converged = tf.cast(convergence, old_mat_h.dtype) - resultant_mat_h = is_converged * mat_h + (1 - is_converged) * old_mat_h - return resultant_mat_h, error diff --git a/lingvo/core/matrix_functions_test.py b/lingvo/core/matrix_functions_test.py deleted file mode 100644 index 7d078ab3..00000000 --- a/lingvo/core/matrix_functions_test.py +++ /dev/null @@ -1,61 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Functional tests for Matrix functions.""" - -import lingvo.compat as tf -from lingvo.core import matrix_functions - -import numpy as np - -TOLERANCE = 1e-3 - - -def np_power(mat_g, alpha): - """Computes mat_g^alpha for a square symmetric matrix mat_g.""" - - mat_u, diag_d, mat_v = np.linalg.svd(mat_g) - diag_d = np.power(diag_d, alpha) - return np.dot(np.dot(mat_u, np.diag(diag_d)), mat_v) - - -class MatrixFunctionTests(tf.test.TestCase): - - def testMatrixSquareRootFunction(self): - """Tests for matrix square roots.""" - - size = 20 - mat_a = np.random.rand(size, size) - mat = np.dot(mat_a, mat_a.T) - expected_mat_root = np_power(mat, 0.5) - mat_root = matrix_functions.matrix_square_root(mat, size) - self.assertAllCloseAccordingToType( - expected_mat_root, mat_root, atol=TOLERANCE, rtol=TOLERANCE) - - def testMatrixInversePthRootFunction(self): - """Tests for matrix inverse pth roots.""" - - size = 4 - mat_a = np.random.rand(size, size) - mat = np.dot(mat_a, mat_a.T) - expected_mat_root = np_power(mat, -0.125) - mat_root, _ = matrix_functions.inlined_matrix_inverse_pth_root( - mat, size, -0.125, ridge_epsilon=0.0) - self.assertAllCloseAccordingToType( - expected_mat_root, mat_root, atol=TOLERANCE, rtol=TOLERANCE) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/metrics.py b/lingvo/core/metrics.py deleted file mode 100644 index 32e46d7d..00000000 --- a/lingvo/core/metrics.py +++ /dev/null @@ -1,443 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Helper classes for computing performance metrics.""" - -import lingvo.compat as tf -from lingvo.core import hyperparams -from lingvo.core import plot -from lingvo.core import py_utils -from lingvo.core import scorers -import numpy as np -try: - import sklearn.metrics # pylint: disable=g-import-not-at-top - HAS_SKLEARN = True -except ImportError: - HAS_SKLEARN = False -try: - import scipy.stats # pylint: disable=g-import-not-at-top - HAS_SCIPY_STATS = True -except ImportError: - HAS_SCIPY_STATS = False - - -def CreateScalarSummary(name, simple_value): - return tf.Summary( - value=[tf.Summary.Value(tag=name, simple_value=simple_value)]) - - -class BaseMetric: - """Base class for aggregating statistics to compute a performance metric.""" - - def Update(self, *args, **kwargs): - """Updates this metric (e.g. accumulates statistics) from the arguments.""" - pass - - @property - def value(self): - """Current value of this metric.""" - return None - - def Summary(self, name): - """Converts the current state of this metric to a `tf.Summary`. - - Args: - name: A string to use as the summary value tag. - - Returns: - A `tf.Summary` proto. - """ - return CreateScalarSummary(name, self.value) - - -class ConfigurableMetric(BaseMetric): - """A Metric class with configurable params.""" - - @classmethod - def Params(cls): - p = hyperparams.InstantiableParams(cls) - return p - - def __init__(self, params): - self.params = params - - -class AverageMetric(BaseMetric): - """Class to compute a weighted (arithmetic) average value metric.""" - - def __init__(self): - self._total_value = 0.0 - self._total_weight = 0.0 - - def Update(self, value, weight=1.0): - if weight < 0.0: - raise ValueError('weight must be non-negative. Got: %f' % weight) - self._total_value += value * weight - self._total_weight += weight - - # We may want both a getter and a setter method for total_value and - # total_weight, respectively. - def GetTotalValue(self): - return self._total_value - - def SetTotalValue(self, val): - self._total_value = val - - total_value = property(GetTotalValue, SetTotalValue) - - def GetTotalWeight(self): - return self._total_weight - - def SetTotalWeight(self, val): - self._total_weight = val - - total_weight = property(GetTotalWeight, SetTotalWeight) - - @property - def value(self): - return (self._total_value / - self._total_weight if self._total_weight > 0 else 0) - - -class F1Metric(BaseMetric): - """Class to compute F1 metrics.""" - - def __init__(self): - self._true_pos = 0.0 - self._false_pos = 0.0 - self._false_neg = 0.0 - - def UpdateTruePositive(self, count=1.0): - self._true_pos += count - - def UpdateFalsePositive(self, count=1.0): - self._false_pos += count - - def UpdateFalseNegative(self, count=1.0): - self._false_neg += count - - @property - def value(self): - if (self._true_pos + self._false_pos) > 0: - precision = self._true_pos / (self._true_pos + self._false_pos) - else: - precision = 0.0 - if (self._true_pos + self._false_neg) > 0: - recall = self._true_pos / (self._true_pos + self._false_neg) - else: - recall = 0.0 - if (precision + recall) > 0: - return 2.0 * precision * recall / (precision + recall) - else: - return 0.0 - - -class CorpusBleuMetric(BaseMetric): - """Metric class to compute the corpus-level BLEU score.""" - - def __init__(self, **kwargs): - self._scorer = scorers.BleuScorer(**kwargs) - - def Update(self, ref_str, hyp_str): - self._scorer.AddSentence(ref_str, hyp_str) - - @property - def unsegmenter(self): - return self._scorer.unsegmenter - - @property - def value(self): - return self._scorer.ComputeOverallScore() - - -class TpuEvalMetrics: - """Manages computation of metrics during TPU execution. - - TPU execution runs a training loop on device. To get eval metrics out of this, - metric values and weights must be carried through the loop. This requires - passing initial values to the loop setup, updated the values during the loop, - and doing a final aggregation after the loop. This class wraps the metrics - dictionary so that the needed ops can be built at the right time as the - training loop is built. - - Note that because the model is not constructed until the loop body function is - called, the initial values must be known statically. This is done currently by - hardcoding a limit on the number of metrics and casting each metric and value - count to float32, regardless of the number of actual metrics the model - produced. - - Note that this implementation computes the metrics over all replicas, for the - last step of the loop only (could be changed to average over all loop steps - instead). - """ - - def __init__(self): - self._metrics = None - self._max_metrics = 256 - - # Loop-carried values alternate value and weight; all values are scalars. - self._initial_values = (2 * - self._max_metrics) * [tf.constant(0, tf.float32)] - - def SetMetrics(self, metric_dict, step_args): - """Sets the metrics to evaluate and the per-step output tensors. - - Args: - metric_dict: dict of (name -> (tensor of values, tensor of weights)) - step_args: the tensors being passed to the training loop body. These share - the same structure of alternating value and weight scalars as the - initial values and the output of this function. - - Returns: - The tensors to return from the training loop body. For entries that are - for metrics in self._metrics, returns the value computed within the loop - (the step_args value passed in); for all others, the value will never be - used at the end and so the step_args value is passed through (which has - the effect of passing the initial values through every iteration of the - loop). - """ - num_metrics = len(metric_dict) - assert num_metrics <= self._max_metrics, ('Increase _max_metrics to >= %d' % - num_metrics) - self._metrics = metric_dict - - # self._metrics contains a map of (metric_value, - # metric_weight). We convert it into [metric_value * - # metric_weight, metric_weight] to make it easier to aggregate - # metric values across steps and TPU replicas. - ret = [] - for _, (value, weight) in sorted(self._metrics.items()): - assert value.shape.is_fully_defined(), ('%s' % value) - assert weight.shape.is_fully_defined(), ('%s' % weight) - weight = tf.cast(weight, tf.float32) - value = tf.cast(value, tf.float32) * weight - ret += [value, weight] - # Each metric has two tensors: value and weight. - assert len(ret) == 2 * num_metrics - ret += list(step_args)[len(ret):] - return ret - - @property - def initial_values(self): - """Returns the initial loop values.""" - return self._initial_values - - @property - def metrics(self): - return self._metrics - - def _Zip(self, values): - assert isinstance(values, list) - return list(zip(values[::2], values[1::2])) - - def FinalizeMetrics(self, loop_result): - """Compute final average of the metrics, given loop_result tensors. - - To be called outside the training loop body , but still in the scope of - tpu.batch_parallel. - - Args: - loop_result: Result of the training loop. - - Returns: - The tensors of the final avg values and total weights. - """ - # Each metric has two tensors in the loop carrying result. - metrics = loop_result[:2 * len(self._metrics.items())] - # Aggregate across tpu replicas. - metrics = [tf.tpu.cross_replica_sum(x) for x in metrics] - ret = [] - for (value, weight) in self._Zip(metrics): - value, weight = py_utils.WeightedAvg( - tf.math.divide_no_nan(value, weight), weight) - ret += [value, weight] - return ret - - def PackMetricsValues(self, values): - """Packs numpy values into a dict of metrics.""" - for k, v in zip(sorted(self._metrics.keys()), self._Zip(values)): - self._metrics[k] = v - - -class AUCMetric(BaseMetric): - """Class to compute the AUC score for binary classification.""" - - def __init__(self, mode='roc', samples=-1): - """Constructor of the class. - - Args: - mode: Possible values: 'roc' or 'pr'. - samples: The number of sample points to compute the AUC. If -1, include - all points seen thus far. - - Raises: - ImportError: If user has not installed sklearn, raise an ImportError. - """ - if not HAS_SKLEARN: - raise ImportError('AUCMetric depends on sklearn.') - self._mode = mode - self._samples = samples - self._label = [] - self._prob = [] - self._weight = [] - if self._mode == 'roc': - self._curve_fn = sklearn.metrics.roc_curve - self._score_fn = sklearn.metrics.roc_auc_score - self._plot_labels = ['False Positive Rate', 'True Positive Rate'] - elif self._mode == 'pr': - self._curve_fn = sklearn.metrics.precision_recall_curve - self._score_fn = sklearn.metrics.average_precision_score - self._plot_labels = ['Recall', 'Precision'] - else: - raise ValueError('mode in AUCMetric must be one of "roc" or "pr".') - - def Update(self, label, prob, weight=None): - """Updates the metrics. - - Args: - label: An array to specify the groundtruth binary labels. Values must be - either 0 or 1. - prob: An array to specify the prediction probabilities. Values must be - within [0, 1.0]. - weight: An array to specify the sample weight for the auc computation. - """ - self._label += label - self._prob += prob - if weight: - self._weight += weight - else: - self._weight += [1 for _ in range(len(label))] - - if self._samples > 0: - self._label = self._label[-self._samples:] - self._prob = self._prob[-self._samples:] - self._weight = self._weight[-self._samples:] - - @property - def value(self): - return self._score_fn(self._label, self._prob, sample_weight=self._weight) - - def Summary(self, name): - - def _Setter(fig, axes): - # 20 ticks betweein 0 and 1. - ticks = np.arange(0, 1.05, 0.05) - axes.grid(b=True) - axes.set_xlabel(self._plot_labels[0]) - axes.set_xticks(ticks) - axes.set_ylabel(self._plot_labels[1]) - axes.set_yticks(ticks) - fig.tight_layout() - - xs, ys, _ = self._curve_fn( - self._label, self._prob, sample_weight=self._weight) - if self._mode == 'pr': - # Swap because sklearn returns <'precision', 'recall'>. - xs, ys = ys, xs - ret = plot.Curve(name=name, figsize=(12, 12), xs=xs, ys=ys, setter=_Setter) - ret.value.add(tag=name, simple_value=self.value) - return ret - - -class CorrelationMetric(BaseMetric): - """Class to compute correlation.""" - - def __init__(self, mode='pearson', samples=-1): - """Constructor of the class. - - Args: - mode: Possible values: 'pearson', 'spearman', 'kendalltau'. - samples: The number of sample points to compute the correlation. If -1, - include all points seen thus far. - - Raises: - ImportError: If user has not installed scipy.stats, raise an ImportError. - """ - if not HAS_SCIPY_STATS: - raise ImportError('CorrelationMetric depends on scipy.stats.') - - assert mode in ['pearson', 'spearman', 'kendalltau'] - self._mode = mode - self._samples = samples - self._target = [] - self._pred = [] - - def Update(self, target, pred): - """Updates the metrics. - - Args: - target: An array to specify the groundtruth float target. - pred: An array to specify the prediction. - """ - self._target += target - self._pred += pred - - if self._samples > 0: - self._target = self._target[-self._samples:] - self._pred = self._pred[-self._samples:] - - @property - def value(self): - # only use the correlation, p-value is ignored. - if self._mode == 'pearson': - return scipy.stats.pearsonr(self._target, self._pred)[0] - elif self._mode == 'spearman': - return scipy.stats.spearmanr(self._target, self._pred)[0] - else: - return scipy.stats.kendalltau(self._target, self._pred)[0] - - -class SamplingMetric(ConfigurableMetric): - """Sampling metric base class. - - Subclasses must implement _CreateSummary(); sampling will be handled - by this base class. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_samples', 8, 'The number of samples to store uniformly.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self._sampler = py_utils.UniformSampler(num_samples=p.num_samples) - self._summary = None - - @property - def samples(self): - """Returns an iterable of sampled decoded outputs to compute Summaries.""" - return self._sampler.samples - - def Update(self, decoded_outputs): - """Samples the input decoded_outputs NestedMap. - - Args: - decoded_outputs: A `.NestedMap`. - """ - self._sampler.Add(decoded_outputs) - # Invalidate cache. - self._summary = None - - def Summary(self, name): - if self._summary is None: - self._summary = self._CreateSummary(name) - return self._summary - - def _CreateSummary(self, name): - """Returns a tf.Summary for this metric.""" - raise NotImplementedError() diff --git a/lingvo/core/metrics_test.py b/lingvo/core/metrics_test.py deleted file mode 100644 index 7bf1ff74..00000000 --- a/lingvo/core/metrics_test.py +++ /dev/null @@ -1,114 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for metrics.""" - -import lingvo.compat as tf -from lingvo.core import metrics -from lingvo.core import py_utils -from lingvo.core import test_utils -import numpy as np - - -class MetricsTest(test_utils.TestCase): - - def testAverageMetric(self): - m = metrics.AverageMetric() - m.Update(1.0) - m.Update(2.0, 10.0) - - self.assertEqual(1.0 + 2.0*10.0, m.total_value) - expected_average = (1.0 + 2.0*10.0) / (1.0 + 10.0) - self.assertEqual(expected_average, m.value) - - name = 'metric_name' - self.assertEqual( - tf.Summary(value=[tf.Summary.Value(tag=name, - simple_value=expected_average)]), - m.Summary(name)) - - # Calling m.Summary() does not reset statistics. - m.Update(1.0) - self.assertEqual(1.0 + 2.0*10.0 + 1.0, m.total_value) - - def testF1Metric(self): - m = metrics.F1Metric() - m.UpdateTruePositive(count=2.0) - m.UpdateFalsePositive() - m.UpdateFalseNegative() - - precision = 2.0 / 3.0 - recall = 2.0 / 3.0 - expected_f1 = 2 * precision * recall / (precision + recall) - self.assertAlmostEqual(expected_f1, m.value) - - name = 'my_f1_metric' - self.assertEqual( - tf.Summary(value=[tf.Summary.Value(tag=name, - simple_value=expected_f1)]), - m.Summary(name)) - - def testCorpusBleuMetric(self): - m = metrics.CorpusBleuMetric() - m.Update('a b c d', 'a b c d') - m.Update('a b c', 'a b c') - - self.assertEqual(1.0, m.value) - - name = 'corpus_bleu' - self.assertEqual( - tf.Summary(value=[tf.Summary.Value(tag=name, simple_value=1.0)]), - m.Summary(name)) - - def testCorrelationMetric(self): - m = metrics.CorrelationMetric() - m.Update([1.0, 2.0, 3.0], [0.1, 0.2, 0.3]) - m.Update([1.0, 2.0, 3.0], [0.1, 0.2, 0.3]) - self.assertEqual(1.0, m.value) - - def testSamplingMetric(self): - - class TestSamplingMetric(metrics.SamplingMetric): - - def _CreateSummary(self, name): - ret = tf.Summary() - for sample in self.samples: - value = sample.value - ret.value.add(tag=name, simple_value=value) - return ret - - np.random.seed(1337) - p = TestSamplingMetric.Params() - p.num_samples = 2 - m = p.Instantiate() - m.Update(py_utils.NestedMap(value=1)) - summary = m.Summary('test') - self.assertEqual(1, summary.value[0].simple_value) - - # Add four more updates. - m.Update(py_utils.NestedMap(value=2)) - m.Update(py_utils.NestedMap(value=3)) - m.Update(py_utils.NestedMap(value=4)) - m.Update(py_utils.NestedMap(value=5)) - summary = m.Summary('test') - # Reservoir sampling will sample values 5 and 3 to remain with the current - # seed. - self.assertEqual(2, len(summary.value)) - self.assertEqual(5, summary.value[0].simple_value) - self.assertEqual(3, summary.value[1].simple_value) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/ml_perf_bleu_metric.py b/lingvo/core/ml_perf_bleu_metric.py deleted file mode 100644 index 4e5f2a9d..00000000 --- a/lingvo/core/ml_perf_bleu_metric.py +++ /dev/null @@ -1,228 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""The MLPerf reference implementation of BLEU.""" - -import collections -import math -import re -import sys -import unicodedata - -import lingvo.compat as tf -from lingvo.core import metrics -import numpy as np -import six - - -def is_unicode(s): - if isinstance(s, str): - return True - return False - - -def to_unicode(s, ignore_errors=False): - if is_unicode(s): - return s - error_mode = "ignore" if ignore_errors else "strict" - return s.decode("utf-8", errors=error_mode) - - -def native_to_unicode(s): - if is_unicode(s): - return s - try: - return to_unicode(s) - except UnicodeDecodeError: - res = to_unicode(s, ignore_errors=True) - return res - - -def _get_ngrams(segment, max_order): - """Extracts all n-grams up to a given maximum order from an input segment. - - Args: - segment: text segment from which n-grams will be extracted. - max_order: maximum length in tokens of the n-grams returned by this methods. - - Returns: - The Counter containing all n-grams up to max_order in segment - with a count of how many times each n-gram occurred. - """ - ngram_counts = collections.Counter() - for order in range(1, max_order + 1): - for i in range(0, len(segment) - order + 1): - ngram = tuple(segment[i:i + order]) - ngram_counts[ngram] += 1 - return ngram_counts - - -def compute_bleu(reference_corpus, - translation_corpus, - max_order=4, - use_bp=True): - """Computes BLEU score of translated segments against one or more references. - - Args: - reference_corpus: list of references for each translation. Each reference - should be tokenized into a list of tokens. - translation_corpus: list of translations to score. Each translation should - be tokenized into a list of tokens. - max_order: Maximum n-gram order to use when computing BLEU score. - use_bp: boolean, whether to apply brevity penalty. - - Returns: - BLEU score. - """ - reference_length = 0 - translation_length = 0 - bp = 1.0 - geo_mean = 0 - - matches_by_order = [0] * max_order - possible_matches_by_order = [0] * max_order - precisions = [] - - for (references, translations) in zip(reference_corpus, translation_corpus): - reference_length += len(references) - translation_length += len(translations) - ref_ngram_counts = _get_ngrams(references, max_order) - translation_ngram_counts = _get_ngrams(translations, max_order) - - overlap = dict((ngram, min(count, translation_ngram_counts[ngram])) - for ngram, count in ref_ngram_counts.items()) - - for ngram in overlap: - matches_by_order[len(ngram) - 1] += overlap[ngram] - for ngram in translation_ngram_counts: - possible_matches_by_order[len(ngram) - - 1] += translation_ngram_counts[ngram] - precisions = [0] * max_order - smooth = 1.0 - for i in range(0, max_order): - if possible_matches_by_order[i] > 0: - precisions[i] = matches_by_order[i] / possible_matches_by_order[i] - if matches_by_order[i] > 0: - precisions[i] = matches_by_order[i] / possible_matches_by_order[i] - else: - smooth *= 2 - precisions[i] = 1.0 / (smooth * possible_matches_by_order[i]) - else: - precisions[i] = 0.0 - - if max(precisions) > 0: - p_log_sum = sum(math.log(p) for p in precisions if p) - geo_mean = math.exp(p_log_sum / max_order) - - if use_bp: - if not reference_length: - bp = 1.0 - else: - ratio = translation_length / reference_length - if ratio <= 0.0: - bp = 0.0 - elif ratio >= 1.0: - bp = 1.0 - else: - bp = math.exp(1 - 1. / ratio) - bleu = geo_mean * bp - return np.float32(bleu) - - -def bleu_score(predictions, labels, **unused_kwargs): - """BLEU score computation between labels and predictions. - - An approximate BLEU scoring method since we do not glue word pieces or - decode the ids and tokenize the output. By default, we use ngram order of 4 - and use brevity penalty. Also, this does not have beam search. - - Args: - predictions: tensor, model predictions - labels: tensor, gold output. - - Returns: - bleu: int, approx bleu score - """ - outputs = tf.cast(tf.argmax(predictions, axis=-1), dtype=tf.int32) - # Convert the outputs and labels to a [batch_size, input_length] tensor. - outputs = tf.squeeze(outputs, axis=[-1, -2]) - labels = tf.squeeze(labels, axis=[-1, -2]) - - bleu = tf.py_func(compute_bleu, (labels, outputs), tf.float32) - return bleu, tf.constant(1.0) - - -class UnicodeRegex: - """Ad-hoc hack to recognize all punctuation and symbols.""" - - def __init__(self): - punctuation = self.property_chars("P") - self.nondigit_punct_re = re.compile(r"([^\d])([" + punctuation + r"])") - self.punct_nondigit_re = re.compile(r"([" + punctuation + r"])([^\d])") - self.symbol_re = re.compile("([" + self.property_chars("S") + "])") - - def property_chars(self, prefix): - return "".join( - six.unichr(x) - for x in range(sys.maxunicode) - if unicodedata.category(six.unichr(x)).startswith(prefix)) - - -uregex = UnicodeRegex() - - -def bleu_tokenize(string): - """Tokenize a string following the official BLEU implementation. - - Args: - string: the input string - - Returns: - a list of tokens - """ - string = uregex.nondigit_punct_re.sub(r"\1 \2 ", string) - string = uregex.punct_nondigit_re.sub(r" \1 \2", string) - string = uregex.symbol_re.sub(r" \1 ", string) - return string.split() - - -def bleu_wrapper(ref_lines, hyp_lines, case_sensitive=False): - """Compute BLEU for two files (reference and hypothesis translation).""" - assert len(ref_lines) == len(hyp_lines), ("{} != {}".format( - len(ref_lines), len(hyp_lines))) - ref_lines = [native_to_unicode(x) for x in ref_lines] - hyp_lines = [native_to_unicode(x) for x in hyp_lines] - if not case_sensitive: - ref_lines = [x.lower() for x in ref_lines] - hyp_lines = [x.lower() for x in hyp_lines] - ref_tokens = [bleu_tokenize(x) for x in ref_lines] - hyp_tokens = [bleu_tokenize(x) for x in hyp_lines] - return compute_bleu(ref_tokens, hyp_tokens) - - -class MlPerfBleuMetric(metrics.BaseMetric): - """Use the MLPerf reference impelmentation.""" - - def __init__(self, **kwargs): - self._ref_lines = [] - self._hyp_lines = [] - - def Update(self, ref_str, hyp_str): - self._ref_lines.append(ref_str) - self._hyp_lines.append(hyp_str) - - @property - def value(self): - return bleu_wrapper(self._ref_lines, self._hyp_lines) diff --git a/lingvo/core/ml_perf_bleu_metric_test.py b/lingvo/core/ml_perf_bleu_metric_test.py deleted file mode 100644 index 1714e127..00000000 --- a/lingvo/core/ml_perf_bleu_metric_test.py +++ /dev/null @@ -1,33 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for metrics.""" - -import lingvo.compat as tf -from lingvo.core import ml_perf_bleu_metric -from lingvo.core import test_utils - - -class MlPerfMetricsTest(test_utils.TestCase): - - def testMlPerfBleuMetric(self): - m = ml_perf_bleu_metric.MlPerfBleuMetric() - m.Update(u"a b a z", u"a b a c") - m.Update(u"y f g d k l m", u"e f \u2028 d") - self.assertAllClose(0.2638, m.value, atol=1e-03) - - -if __name__ == "__main__": - tf.test.main() diff --git a/lingvo/core/ml_perf_log.py b/lingvo/core/ml_perf_log.py deleted file mode 100644 index d499f5b8..00000000 --- a/lingvo/core/ml_perf_log.py +++ /dev/null @@ -1,80 +0,0 @@ -# Lint as: python3 - -# Copyright 2018 MLBenchmark Group. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Convenience function for logging compliance tags to stdout.""" - -import inspect -import json -import logging -import os -import re -import sys -import time - -PATTERN = re.compile('[a-zA-Z0-9]+') - -LOG_FILE = os.getenv('COMPLIANCE_FILE') -# create logger with 'spam_application' -LOGGER = logging.getLogger('mlperf_compliance') -LOGGER.setLevel(logging.DEBUG) - -_STREAM_HANDLER = logging.StreamHandler(stream=sys.stdout) -_STREAM_HANDLER.setLevel(logging.INFO) -LOGGER.addHandler(_STREAM_HANDLER) - -if LOG_FILE: - _FILE_HANDLER = logging.FileHandler(LOG_FILE) - _FILE_HANDLER.setLevel(logging.DEBUG) - LOGGER.addHandler(_FILE_HANDLER) -else: - _STREAM_HANDLER.setLevel(logging.DEBUG) - - -def get_caller(stack_index=2, root_dir=None): - caller = inspect.getframeinfo(inspect.stack()[stack_index][0]) - - # Trim the filenames for readability. - filename = caller.filename - if root_dir is not None: - filename = re.sub('^' + root_dir + '/', '', filename) - return (filename, caller.lineno) - - -# :::MLL 1556733699.71 run_start: {"value": null, -# "metadata": {"lineno": 77, "file": main.py}} -LOG_TEMPLATE = ':::MLL {:.3f} {}: {{"value": {}, "metadata": {}}}' - - -def mlperf_format(key, value, now, stack_offset=0, metadata=None): - """Format a message for MLPerf.""" - if metadata is None: - metadata = {} - - if 'lineno' not in metadata: - filename, lineno = get_caller(2 + stack_offset, root_dir=None) - metadata['lineno'] = lineno - metadata['file'] = filename - - msg = LOG_TEMPLATE.format(now, key, json.dumps(value), json.dumps(metadata)) - return msg - - -def mlperf_print(key, value, stack_offset=0, metadata=None): - now = time.time() - LOGGER.info( - mlperf_format( - key, value, now, stack_offset=stack_offset + 1, metadata=metadata)) - return now diff --git a/lingvo/core/ml_perf_log_test.py b/lingvo/core/ml_perf_log_test.py deleted file mode 100644 index 3a4d8d32..00000000 --- a/lingvo/core/ml_perf_log_test.py +++ /dev/null @@ -1,39 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lingvo.core.tshape.""" - -import json -import lingvo.compat as tf -from lingvo.core import ml_perf_log as mlp_log - - -class TestMLPerfLog: - """Test mlperf log.""" - - def testFormat(self): - msg = mlp_log.mlperf_format('foo_key', {'whiz': 'bang'}) - parts = msg.split() - assert parts[0] == ':::MLL' - assert float(parts[1]) > 10 - assert parts[2] == 'foo_key:' - j = json.loads(' '.join(parts[3:])) - assert j['value'] == {'whiz': 'bang'} - assert j['metadata']['lineno'] == 21 - assert 'test_mlp_log' in j['metadata']['file'] - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/ml_perf_tokenizer.py b/lingvo/core/ml_perf_tokenizer.py deleted file mode 100644 index fe764ade..00000000 --- a/lingvo/core/ml_perf_tokenizer.py +++ /dev/null @@ -1,34 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tokenizer for use for the MLPerf transformer benchmark.""" - -from lingvo.core import ops -from lingvo.core import tokenizers - - -class MlPerfTokenizer(tokenizers.BaseTokenizer): - """Id->String only for MLPerf decoding.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define("vocab_filepath", None, "Specifies a filepath to the vocab.") - return p - - def IdsToStrings(self, ids, lens): - p = self.params - return ops.ml_perf_subword_id_to_string( - ids, lens, vocab_filepath=p.vocab_filepath) diff --git a/lingvo/core/model_helper.py b/lingvo/core/model_helper.py deleted file mode 100644 index 018aec15..00000000 --- a/lingvo/core/model_helper.py +++ /dev/null @@ -1,79 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Some utilities for configuring models.""" - -import lingvo.compat as tf -from lingvo.core import rnn_layers - -FLAGS = tf.flags.FLAGS - - -def CreateUnidirectionalRNNParams(layer_params, cell_params): - """Creates parameters for uni-directional RNN layer. - - Based on `layer_params.unidi_rnn_type`. - - Args: - layer_params: parameters for the layers, has to contain unidi_rnn_type. - cell_params: parameters for the RNN cell. - Returns: - Parameters for uni-directional RNN layer. - Raises: - ValueError: if `unidi_rnn_type` is invalid. - """ - p = layer_params - assert hasattr(p, - 'unidi_rnn_type'), 'layer params must contain unidi_rnn_type' - unidi_rnn_type = p.unidi_rnn_type - if unidi_rnn_type == 'func': - unidi_cls = rnn_layers.FRNN - elif unidi_rnn_type == 'quasi_ifo' or unidi_rnn_type == 'sru': - unidi_cls = rnn_layers.FRNN - else: - raise ValueError('Invalid unidi_rnn_type: %s', unidi_rnn_type) - params = unidi_cls.Params() - params.cell = cell_params - return params - - -def CreateBidirectionalRNNParams(layer_params, forward_cell_params, - backward_cell_params): - """Creates parameters for bi-directional RNN layer. - - Based on `layer_params.bidi_rnn_type`. - - Args: - layer_params: parameters for the layers, has to contain bidi_rnn_type. - forward_cell_params: parameters for the forward RNN cell. - backward_cell_params: parameters for the backward RNN cell. - Returns: - Parameters for bi-directional RNN layer. - Raises: - ValueError: if `bidi_rnn_type` is invalid. - """ - p = layer_params - assert hasattr(p, 'bidi_rnn_type'), 'layer params must contain bidi_rnn_type' - bidi_rnn_type = p.bidi_rnn_type - if bidi_rnn_type == 'func': - bidi_cls = rnn_layers.BidirectionalFRNN - elif bidi_rnn_type == 'quasi_ifo' or bidi_rnn_type == 'sru': - bidi_cls = rnn_layers.BidirectionalFRNNQuasi - else: - raise ValueError('Invalid bidi_rnn_type: %s', bidi_rnn_type) - params = bidi_cls.Params() - params.fwd = forward_cell_params - params.bak = backward_cell_params - return params diff --git a/lingvo/core/moe_layers.py b/lingvo/core/moe_layers.py deleted file mode 100644 index 559a9d72..00000000 --- a/lingvo/core/moe_layers.py +++ /dev/null @@ -1,1373 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Layers and utilities that facilitate building MOE models.""" - -from lingvo import compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils -from lingvo.core import tpu_summary - -# pylint: disable=g-direct-tensorflow-import -from tensorflow.compiler.xla.experimental.xla_sharding import xla_sharding -# pylint: enable=g-direct-tensorflow-import - - -def Split(x, - split_dimension, - num_devices, - use_sharding_op=True, - input_shape=None): - """Wrapper for xla_sharding.split. - - Args: - x: Tensor to annotate. - split_dimension: xla_sharding.split arg. - num_devices: xla_sharding.split arg. - use_sharding_op: If true, adds a sharding op to set the sharding: - tensor = gen_xla_ops.xla_sharding(tensor) - See http://cs/search/?q=XlaSharding+file:xla_ops.cc - - hyouklee@: use_sharding_op=False - "It adds the sharding attribute to the op itself. The outcome is that, - that information could be lost by TF graph transformations. Also, - directly attaching the sharding annotation to the op caused some - compilation failures in the past (due to incompatible shardings), so the - plan is to make use_sharding_op to be the default." - - "The only case I would set it to False today is when annotating weights. - Weight annotation does some special handling, so there may be some - changes needed in that logic if we add separate sharding op." - input_shape: The shape of the original tensor. - - Returns: - Tensor conditionally annotated with sharding. - """ - if not py_utils.use_tpu() or num_devices is None or not num_devices > 1: - return x - return xla_sharding.split( - x, - split_dimension, - num_devices, - input_shape=input_shape, - use_sharding_op=use_sharding_op, - ) - - -class VarLayer(base_layer.BaseLayer): - """Container for variables.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('weights', None, '[(name, WeightParams)..] list.') - p.name = p.name or 'w' - return p - - def __init__(self, params): - super().__init__(params) - for k, v in self.params.weights: - vp = v.Copy() - if vp.init is None: - vp.init = self.params.params_init - self.CreateVariable(k, vp) - - def FProp(self, theta, *args, **kwargs): - - def MaybeCastToFPropDtype(x): - if x is None or not x.dtype.is_floating or x.dtype == self._params.fprop_dtype: - return x - if self._params.fprop_dtype is None: - return x - return tf.cast(x, self._params.fprop_dtype) - - # TODO(lepikhin): MoEBuilder.Embedding can not use '->emb' rule without - # returning single element of list of one element below. - retval = [MaybeCastToFPropDtype(theta[k]) for k, _ in self.params.weights] - return retval[0] if len(retval) == 1 else retval - - -class ShardedVarLayer(VarLayer): - """Container for variables whose values shared across different devices.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('split_dimension', 0, 'Dimension to split') - p.Define('num_devices', None, - 'The number of cores to split weights and computation over.') - p.Define('cast_to_fprop_dtype', True, - 'Whether to cast variables to fprop_dtype') - return p - - def FProp(self, theta, *args, **kwargs): - p = self.params - - # TODO(huangyp, lepikhin): Maybe cast to fprop dtype as well. - def MaybeWeightSplitAndCastToFPropDtype(k): - x = self.vars[k].read_value() - if x is None: - return None - x = Split(x, p.split_dimension, p.num_devices, use_sharding_op=True) - if (p.cast_to_fprop_dtype and x.dtype.is_floating and - x.dtype != p.fprop_dtype and p.fprop_dtype): - x = tf.cast(x, p.fprop_dtype) - return x - - retval = [MaybeWeightSplitAndCastToFPropDtype(k) for k, _ in p.weights] - return retval[0] if len(retval) == 1 else retval - - -class StateLayer(base_layer.BaseLayer): - """Container for recurrent state for incremental decoding. - - It has two operation modes. - - During training, it does nothing. - It expects that FProp(x, t) is called with t=None, and returns x unchanged. - - During decoding, it expects: - - t: an int32 scalar and - x: a tensor of shape `[batch, 1, ...]`. - - It updates state `x_full[:, t, :] <- x[:, 0, :]` and returns x_full. - The shape of x_full is then `[batch, time, ...]`. - - The state is stored as theta.state attribute. - - To construct initial state, call InitState classmethod on the root layer. - InitState() will traverse root layer children recursively, will initialize - internal state for each StateLayer instance, and will return a nested - tuple of states. - - For incremental iteration the static methods work as follows:: - - dec = builder.DecoderLayerStack(...).Instantiate() - state0 = StateLayer.InitState(dec, shape=[tgt_batch, max_len]) - theta0 = StateLayer.UpdateTheta(dec, dec.theta, state0, t=0) - # (FProp in nested StateLayer now has access to 'state0' and 't') - dec.FProp(theta0, ...) - # FProp will modify theta0 in-place - state1 = state0.copy() - state1 = StateLayer.UpdateState(dec, theta0, state1) - - (see `moe_builder_test.py` for a more complete example.) - """ - _use_flat_beam_search = False - - @classmethod - def Params(cls): - p = super().Params() - p.Define('shape', [None, None], 'batch, time, etc...') - return p - - def NewState(self, shape): - """Returns initial state. - - Args: - shape: [batch, time] for beam_search_tpu_helper or [batch, beam, time] for - flat_beam_search. - - Returns: - zero-initialized state tensor with shape [batch, time, ...] for - beam_search_tpu_helper or [time, batch, beam, ...] for flat_beam_search. - - Raises: - ValueError: the length of shape is not 2 or 3. - """ - fprop_dtype = py_utils.FPropDtype(self.params) - - if len(shape) == 2: - # For use with beam_search_tpu_helper batch_major_compute=1 - p = self.params - shape = tuple(shape) + tuple(p.shape[2:]) - state = tf.zeros(shape, fprop_dtype) - return state - elif len(shape) == 3: - # For use with flat_beam_search - self._use_flat_beam_search = True - p = self.params - batch, beam, max_steps = shape - state = tf.Empty( - [max_steps, batch, beam] + p.shape[2:], fprop_dtype, init=True) - return state - else: - raise ValueError('bad shape: %r' % shape) - - def FProp(self, theta, x): - p = self.params - if not hasattr(theta, 't'): - return x - t = theta.t - if t is None: - return x - assert hasattr(theta, 'state') - state = theta.state - - tf.logging.info('p.name=%r', p.name) - tf.logging.info('state=%r', state) - tf.logging.info('x=%r', x) - tf.logging.info('t=%r', t) - - with tf.name_scope(p.name): - if not self._use_flat_beam_search: - # For tpu_beam_search_helper - z = tf.one_hot(t, tf.shape(state)[1]) - z = tf.expand_dims(z, 0) - while len(z.shape) < len(x.shape): - z = tf.expand_dims(z, -1) - y = state = (1 - z) * state + z * x - if self._use_flat_beam_search: - state_slice_size = int(state.shape[2]) - update_slice_size = int(x.shape[1]) - if update_slice_size == state_slice_size: - state = tf.InplaceUpdate(state, t, tf.cast(x, state.dtype)) - else: - # With prefix decoding the first call to decoder can have - # sequence length (N * beam_size) with N > 1. - # In this special case state tensor update is implemented as multiple - # InplaceUpdate ops each for a slice [batch_size, beam_size]. - div = int(update_slice_size / state_slice_size) - assert update_slice_size == state_slice_size * div, ( - update_slice_size, state_slice_size) - for i, x_i in enumerate(tf.split(x, div, 1)): - state = tf.InplaceUpdate(state, t + i, tf.cast(x_i, state.dtype)) - tf.logging.info('state*=%r', state) - # [T,B,L,...] - y = tf.cast(state, x.dtype) - # [T, B, L, ...] -> [B, T, L, ...] - perm = list(range(len(y.shape))) - perm[:2] = [1, 0] - y = tf.transpose(y, perm) - # [B, T, L, ...] -> [B, T*L, ...] - y_shape = list(y.shape) - y_shape[1:3] = [int(y_shape[1]) * int(y_shape[2])] - y = tf.reshape(y, y_shape) - theta.state = state - - tf.logging.info('y=%r', y) - return y - - @classmethod - def InitState(cls, layer, shape): - """Returns new state with leading shape=[batch, time].""" - - def Rec(layer): # pylint: disable=missing-docstring - state = None - if isinstance(layer, StateLayer): - assert not layer.children - state = layer.NewState(shape) - return state - for c_name, c in layer.children.items(): - c_state = Rec(c) - if c_state is not None: - if state is None: - state = py_utils.NestedMap() - state[c_name] = c_state - return state - - state = Rec(layer) - assert state is not None - return state - - @classmethod - def UpdateTheta(cls, layer, theta, state, t=None): - """Returns theta with state.""" - - def Rec(layer, theta, state): # pylint: disable=missing-docstring - if isinstance(layer, StateLayer): - theta.state = state - theta.t = t - return - for c_name, c in layer.children.items(): - if c_name in state: - Rec(c, theta[c_name], state[c_name]) - - Rec(layer, theta, state) - return theta - - @classmethod - def UpdateState(cls, layer, theta, state): - """Returns updated state from theta.""" - - def Rec(layer, theta, state): # pylint: disable=missing-docstring - for c_name, c in layer.children.items(): - if isinstance(c, StateLayer): - state[c_name] = theta[c_name].state - elif c_name in state: - Rec(c, theta[c_name], state[c_name]) - - Rec(layer, theta, state) - return state - - -class OverrideLayer(base_layer.BaseLayer): - """Allows to override arbitrary tensors in the graph. - - If key is not set in the global context, FProp does nothing. - Otherwise it returns value associated to 'key'. - - To override a tensor during my_layer.FProp: - - OverrideLayer.Set(key, value) - out_with_override = my_layer.FProp(...) - OverrideLayer.Clear() - """ - - _OVERRIDE = {} - - @classmethod - def Params(cls): - p = super().Params() - p.Define('key', None, 'Context key') - return p - - def FProp(self, theta, x): - p = self.params - if p.key and p.key in self._OVERRIDE: - return self._OVERRIDE[p.key] - else: - return x - - @classmethod - def Set(cls, k, v): - cls._OVERRIDE[k] = v - - @classmethod - def Clear(cls): - cls._OVERRIDE.clear() - - -class SharedEmbeddingSoftmaxLayer(base_layer.BaseLayer): - """Shared weights for embemdding lookup and softmax.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('vocab_size', 0, 'Num tokens in vocab.') - p.Define('max_len', 0, 'Num of token in the sequence.') - p.Define('embedding_dim', 0, 'Depth of the output.') - p.Define('z_loss_coef', 1e-4, 'Label smoothing.') - p.Define('num_devices', 1, 'Number of devices for sharding.') - p.Define('logits_abs_max', None, 'Logits clipping.') - p.Define('label_smoothing', 0.1, 'Label smoothing.') - p.Define( - 'use_tgt_labels_size_as_loss_denominator', True, - 'False to use total number of non-padding tokens instead of ' - 'fixed tgt_labels tensor size.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - emb_p = py_utils.WeightParams( - init=py_utils.WeightInit.Gaussian(), - shape=[p.vocab_size, p.embedding_dim]) - pos_emb_p = py_utils.WeightParams( - init=py_utils.WeightInit.Gaussian(), shape=[p.max_len, p.embedding_dim]) - self.CreateVariable('embedding', emb_p) - self.CreateVariable('pos_emb', pos_emb_p) - - def _MaybeSplit(self, x): - if True or self.params.num_devices <= 1: - return x - return Split(x, 0, self.params.num_devices) - - def FProp(self, theta, ids, segment_pos): - p = self.params - fprop_dtype = py_utils.FPropDtype(p) - - ids = self._MaybeSplit(ids) - segment_pos = self._MaybeSplit(segment_pos) - - one_hot_ids = tf.one_hot(ids, p.vocab_size, dtype=fprop_dtype) - one_hot_ids = self._MaybeSplit(one_hot_ids) - - one_hot_pos = tf.one_hot(segment_pos, p.max_len, dtype=fprop_dtype) - one_hot_pos = self._MaybeSplit(one_hot_pos) - - token_emb = tf.einsum('VH,BLV->BLH', theta.embedding, one_hot_ids) - token_emb = self._MaybeSplit(token_emb) - - pos_emb = tf.einsum('VH,BLV->BLH', theta.pos_emb, one_hot_pos) - pos_emb = self._MaybeSplit(pos_emb) - return self._MaybeSplit(token_emb + pos_emb) - - def ComputeLoss(self, theta, activation, labels, segment_ids): - p = self.params - activation = self._MaybeSplit( - self._MaybeSplit(activation) * (p.embedding_dim**-0.5)) - softmax_weights = theta.embedding - if activation.dtype != softmax_weights.dtype: - softmax_weights = tf.cast(softmax_weights, activation.dtype) - logits = self._MaybeSplit( - tf.einsum('BLM,VM->BLV', activation, softmax_weights)) - if p.logits_abs_max is not None: - logits = self._MaybeSplit( - py_utils.clip_by_value(logits, -p.logits_abs_max, p.logits_abs_max)) - - off_value = p.label_smoothing / p.vocab_size - on_value = 1.0 - p.label_smoothing + off_value - soft_targets = self._MaybeSplit( - tf.one_hot( - labels, p.vocab_size, on_value=on_value, off_value=off_value)) - xent = self._MaybeSplit( - tf.nn.softmax_cross_entropy_with_logits( - labels=tf.one_hot(labels, p.vocab_size), logits=logits)) - loss = self._MaybeSplit( - tf.nn.softmax_cross_entropy_with_logits( - labels=soft_targets, logits=logits)) - soft_targets_xent = loss - - if p.z_loss_coef > 0.0: - log_z = tf.math.reduce_logsumexp(logits, -1) - z_loss_inc = p.z_loss_coef * tf.math.square(log_z) - loss += z_loss_inc - - non_padding = self._MaybeSplit( - tf.cast(tf.not_equal(segment_ids, 0), py_utils.FPropDtype(p))) - - per_token_loss = self._MaybeSplit(loss * non_padding) - if p.z_loss_coef > 0.0: - per_token_z_loss_inc = self._MaybeSplit(z_loss_inc * non_padding) - - if p.use_tgt_labels_size_as_loss_denominator: - # E.g. loss is going to be tiny if inputs are not packed and only a - # fraction of tgt_labels are non-padding. - loss_denom = tf.reduce_sum(tf.ones_like(non_padding)) - per_example_loss_denom = tf.reduce_sum(tf.ones_like(non_padding), 1) - else: - loss_denom = tf.reduce_sum(non_padding) - per_example_loss_denom = tf.reduce_sum(non_padding, 1) - avg_loss = tf.reduce_sum(per_token_loss) / loss_denom - avg_z_loss_inc = (tf.reduce_sum(per_token_z_loss_inc) / - loss_denom) if p.z_loss_coef > 0.0 else 0.0 - - soft_targets_xent = ( - tf.reduce_sum(self._MaybeSplit(soft_targets_xent * non_padding)) / - tf.reduce_sum(non_padding)) - - # TODO(lepikhin): consider returning - # {'loss': (unnormalized per_token_loss, tf.reduce_sum(non_padding))} - per_example_loss = { - 'loss': tf.reduce_sum(per_token_loss, 1) / per_example_loss_denom - } - return { - 'mean_xent': (tf.reduce_sum(self._MaybeSplit(xent * non_padding)) / - tf.reduce_sum(non_padding), tf.reduce_sum(non_padding)), - 'soft_targets_xent': (soft_targets_xent, tf.reduce_sum(non_padding)), - 'weight': (tf.reduce_sum(non_padding), 1.0), - 'loss': (avg_loss, 1.0), - 'avg_z_loss_inc': (avg_z_loss_inc, 1.0), - }, per_example_loss - - -def Top2GatingOnLogits(inputs, - paddings, - logits, - num_devices, - experts_dim, - expert_capacity_dim, - fprop_dtype, - use_xla_sharding=True, - second_expert_policy='all', - second_expert_threshold=0.0, - legacy_mtf_behavior=True, - capacity_factor=None): - """Computes Top-2 gating for Mixture-of-Experts. - - There are two expected usages of this function: - - 1. used with xla_sharding. In this case, 'inputs' corresponds to a sharded - tensor across multiple tpu cores. The operations within this function are - automatically sharded/replicated across tpu cores. - 2. used within ML-Pathways. In this case, 'inputs' is always local to one tpu - core. All computations below are carried out on one tpu core only. This - function tries to dispatch examples across tpu cores in such a way that - each expert is assigned no more than 'expert_capacity_dim' number of - examples. - - Below ` indicates common way of splitting along mesh dimension. - - Dimensions cheat sheet: - - G: group_dim - S: group_size_dim - E: number of experts - C: capacity per expert - M: model_dim (same as input_dim, same as output_dim) - B: original batch_dim - L: original sequence_length_dim - - Note that for local_dispatch original batch BLM is reshaped into GSM, each - group `g = 0...G-1` is being dispatched independently. - - Args: - inputs: G`SM Tensor. - paddings: G`S Tensor. - logits: G`SE Tensor. - num_devices: number of MoE devices for local dispatch - experts_dim: number of experts. - expert_capacity_dim: number of examples per minibatch(group) per expert. - Each example is typically a vector of size input_dim, representing - embedded token or an element of Transformer layer output. - fprop_dtype: activations datatype to use. - use_xla_sharding: bool, True if this function is used for the xla_sharding - case. - second_expert_policy: 'all', 'sampling' or 'random'. - - - 'all': we greedily pick the 2nd expert. - - 'sampling': we sample the 2nd expert from the softmax. - - 'random': we optionally 'random'-ize dispatch to second-best expert - proportional to (weight / second_expert_threshold). - - second_expert_threshold: threshold for probability normalization for - second_expert_policy == 'random'. - legacy_mtf_behavior: bool, True if to match legacy mtf behavior exactly. - capacity_factor: if set, increases expert_capacity_dim to at least - (group_size * capacity_factor) / experts_dim - where `group_size` is the size of G dimension of `inputs`. If the - value of expert_capacity_dim is already big enough no change is made. - - TODO(lepikhin): get rid of the legacy_mtf_behavior flag. - - Returns: - A tuple (aux_loss, combine_tensor, dispatch_tensor). - - - aux_loss: auxiliary loss, for equalizing the expert assignment ratios. - - combine_tensor: G`SEC Tensor for combining expert outputs. - - dispatch_tensor: G`SEC Tensor, scattering/dispatching inputs to - experts. - """ - del inputs # inputs is currently not used. - raw_gates = tf.nn.softmax(logits) # along E dim - - if capacity_factor is not None: - # Determine expert capacity automatically depedning on the input size. - group_size_dim = int(logits.shape[1]) - auto_expert_capacity = int((group_size_dim * capacity_factor) / experts_dim) - if expert_capacity_dim < auto_expert_capacity: - expert_capacity_dim = auto_expert_capacity - # Round up to a multiple of 4 to avoid possible padding. - while expert_capacity_dim % 4: - expert_capacity_dim += 1 - tf.logging.info( - 'Setting expert_capacity_dim=%r (capacity_factor=%r ' - 'group_size_dim=%r experts_dim=%r name_scope=%r)', - expert_capacity_dim, capacity_factor, group_size_dim, experts_dim, - tf.get_default_graph().get_name_scope()) - tpu_summary.scalar('expert_capacity', expert_capacity_dim) - - # top first and second gate value and expert index for each input - # - # GSK Tensors, K=2 - def _MaybeSplit(x): - if use_xla_sharding: - return Split(x, 0, num_devices) - else: - return x - - def _CreateOverCapacityRatioSummary(mask, position_in_expert, capacity, name): - with tf.name_scope('over_capacity'): - over_capacity = tf.reduce_sum( - tf.cast( - tf.greater_equal(mask * position_in_expert, capacity), - mask.dtype)) - over_capacity_ratio = over_capacity / tf.maximum( - tf.constant(1.0, dtype=mask.dtype), tf.reduce_sum(mask)) - py_utils.AddTpuSummaryTensor(name, over_capacity_ratio) - tpu_summary.scalar(name, over_capacity_ratio, while_loop_reduce='mean') - - # As pointed out by zhifengc@ this method needs to be refactored. lepikhin@ - # and krikun@ will: - # - expand moe_spmd_test to compare Adafactor updates, slots on TPU - # including 2x2 with sharding - # - # - add more tests for policy="random" - # - # - add single step test for full size WMT model on CPU - # - # and then break this function into modules. - # - # GS - index_1 = tf.math.argmax(raw_gates, axis=-1, output_type=tf.int32) - index_1 = _MaybeSplit(index_1) - tpu_summary.tensor('index_1', index_1) - - # GSE - mask_1 = tf.one_hot(index_1, experts_dim, dtype=fprop_dtype) - mask_1 = _MaybeSplit(mask_1) - density_1_proxy = raw_gates - - importance = tf.ones_like(mask_1[:, :, 0]) - - if paddings is not None: - importance = 1.0 - paddings - mask_1 *= tf.expand_dims(importance, -1) - density_1_proxy *= tf.expand_dims(importance, -1) - - gate_1 = tf.einsum('GSE,GSE->GS', raw_gates, mask_1) - gates_without_top_1 = raw_gates * (1.0 - mask_1) - - if second_expert_policy == 'sampling': - # We directly sample the 2nd expert index from the softmax over of the 2nd - # expert by getting rid of the 1st expert already selected above. To do so, - # we set a very negative value to the logit corresponding to the 1st expert. - # Then we sample from the softmax (categorical) distribution using the - # Gumbel max trick. - noise = _MaybeSplit(tf.random.uniform(logits.shape, dtype=logits.dtype)) - # Generates standard Gumbel(0, 1) noise, GSE Tensors - noise = -tf.math.log(-tf.math.log(noise)) - very_negative_logits = _MaybeSplit( - (tf.ones_like(logits) * logits.dtype.max * - tf.constant(-0.7, dtype=logits.dtype))) - # Gets rid of the first expert by setting its logit to be very negative - updated_logits = _MaybeSplit( - tf.where(mask_1 > 0.0, very_negative_logits, logits)) - # Adds the Gumbel noise to the updated logits - noised_logits = _MaybeSplit(updated_logits + noise) - # Picks the index of the largest noised logit as the 2nd expert. This is - # equivalent to sampling from the softmax over the 2nd experts. - index_2 = tf.math.argmax(noised_logits, axis=-1, output_type=tf.int32) - else: - index_2 = tf.math.argmax(gates_without_top_1, axis=-1, output_type=tf.int32) - - index_2 = _MaybeSplit(index_2) - mask_2 = tf.one_hot(index_2, experts_dim, dtype=fprop_dtype) - mask_2 = _MaybeSplit(mask_2) - if paddings is not None: - mask_2 *= tf.expand_dims(importance, -1) - gate_2 = tf.einsum('GSE,GSE->GS', gates_without_top_1, mask_2) - - if legacy_mtf_behavior: - # cl/298510175 moved this branch for gate_{1,2} denom calculation here. - # - # For policy=random, it's better to nomalize gate_{1,2} before taking - # capacity into account and before potentially dropping second expert. - # - # According to mean_xent (http://short/_NzbZ5rINr5): - # MoE_512_102xen_PolicyAll_298510175 - # MoE_512_102xen_PolicyRandom_298510175 - # - # vs pre-cl/298510175 - # MoE_512_102xen_PolicyRandom - # MoE_512_102xen_PolicyAll - # - # it substantially improves policy=random with threshold=0.5 which - # historically was better than policy="all" - # - # Also confirmed this by decoding - # nmt_train/m4/data/es_en/test.txt - # nmt_train/m4/data/ru_en/test.txt - # nmt_train/m4/data/zh_en/test.txt - # and improving BLEU - # - # moe_decode.MoE_512_102xen_PolicyRandom_298510175-160000.batch1024.beam4.c_dim4.ln0.8.rkv.mteval102 - # 0.421443 - # 0.327102 - # 0.315693 - # vs - # moe_decode.feb18_non_fig_snapshot_2626_MoE_512_102xen_PolicyRandom-190000.batch1024.beam4.c_dim4.ln0.8.rkv.mteval102 - # 0.399232 - # 0.310606 - # 0.288229 - # - # Additional comparison, see mean_xent http://short/_YHccOhQtdu with - # legacy_mtf_behavior=False models - # 3 - MoE_512_102xen_PolicyAll_LegacyFalse - # 6 - MoE_512_102xen_PolicyRandom_LegacyFalse - # shows that policy="random" gets worse with legacy_mtf_behavior=False, and - # is similar to pre-cl/298510175 - # 4 - MoE_512_102xen_PolicyRandom - # - # gate_1 can become 0 due to Expert being out of capacity. - # - # gate_2 can become 0 due to - # second_expert_policy == 'random' - # or "out of capacity" scenario. - # - # Here we renormalize regardless of cases above. - denom = gate_1 + gate_2 + 1e-9 - gate_1 /= denom - gate_2 /= denom - - # We reshape the mask as [X*S, E], and compute cumulative sums of - # assignment indicators for each expert index e \in 0..E-1 independently. - # First occurrence of assignment indicator is excluded, see exclusive=True - # flag below. - position_in_expert_1 = tf.cumsum(mask_1, exclusive=True, axis=1) - - # GS Tensor - capacity = tf.cast(expert_capacity_dim, dtype=position_in_expert_1.dtype) - - # GE Tensor (reducing S out of GSE tensor mask_1) - # density_1[:, e] represents assignment ratio (num assigned / total) to - # expert e as top_1 expert without taking capacity into account. - if legacy_mtf_behavior: - density_denom = 1.0 - else: - density_denom = tf.reduce_mean( - importance, axis=(1))[:, tf.newaxis] + 1e-6 - density_1 = tf.reduce_mean(mask_1, axis=(1)) / density_denom - # density_1_proxy[:, e] represents mean of raw_gates for expert e, including - # those of examples not assigned to e with top_k. - density_1_proxy = tf.reduce_mean(density_1_proxy, axis=1) / density_denom - - with tf.name_scope('aux_loss'): - # The MoE paper (https://arxiv.org/pdf/1701.06538.pdf) uses an aux loss of - # reduce_mean(density_1_proxy * density_1_proxy). Here we replace one of - # the density_1_proxy with the discrete density_1 following - # mesh_tensorflow/transformer/moe.py?rcl=283569345. - aux_loss = tf.reduce_mean(density_1_proxy * density_1) # element-wise - aux_loss *= experts_dim * experts_dim # const coefficient - - # Add the over capacity ratio for expert 1 - _CreateOverCapacityRatioSummary(mask_1, position_in_expert_1, capacity, - 'over_capacity_1_ratio') - - mask_1 *= tf.cast(tf.less(position_in_expert_1, capacity), dtype=mask_1.dtype) - position_in_expert_1 = tf.einsum('GSE,GSE->GS', position_in_expert_1, mask_1) - - # How many examples in this sequence go to this expert - mask_1_count = tf.einsum('GSE->GE', mask_1) - # [batch, group] - mostly ones, but zeros where something didn't fit - mask_1_flat = tf.einsum('GSE->GS', mask_1) - - if second_expert_policy == 'all' or second_expert_policy == 'sampling': - pass - elif second_expert_policy == 'random': - # gate_2 is between 0 and 1, reminder: - # - # raw_gates = tf.nn.softmax(logits) - # index_1 = tf.math.argmax(raw_gates, axis=-1, output_type=tf.int32) - # mask_1 = tf.one_hot(index_1, experts_dim, dtype=fprop_dtype) - # gate_1 = tf.einsum('GSE,GSE->GS', raw_gates, mask_1) - # - # E.g. if gate_2 exceeds second_expert_threshold, then we definitely - # dispatch to second-best expert. Otherwise we dispatch with probability - # proportional to (gate_2 / threshold). - # - sampled_2 = tf.less( - _MaybeSplit(tf.random.uniform(gate_2.shape, dtype=gate_2.dtype)), - (gate_2 / max(second_expert_threshold, 1e-9))) - gate_2 *= tf.cast(sampled_2, gate_2.dtype) - mask_2 *= tf.cast(tf.expand_dims(sampled_2, -1), mask_2.dtype) - else: - raise ValueError(second_expert_policy) - - position_in_expert_2 = tf.cumsum( - mask_2, exclusive=True, axis=1) + tf.expand_dims(mask_1_count, 1) - - # Add the over capacity ratio for expert 2 - _CreateOverCapacityRatioSummary(mask_2, position_in_expert_2, capacity, - 'over_capacity_2_ratio') - - mask_2 *= tf.cast(tf.less(position_in_expert_2, capacity), mask_2.dtype) - position_in_expert_2 = tf.einsum('GSE,GSE->GS', position_in_expert_2, mask_2) - mask_2_flat = tf.reduce_sum(mask_2, axis=-1) - - # Equivalent non-einsum implementation: - # - # position_in_expert_2 *= mask_2 - # position_in_expert_2 = tf.reduce_sum( - # position_in_expert_2, axis=-1, name='position_in_expert_2') - - gate_1 *= mask_1_flat - gate_2 *= mask_2_flat - - if not legacy_mtf_behavior: - denom = gate_1 + gate_2 - # To avoid divide by 0. - denom = tf.where(denom > 0, denom, tf.ones_like(denom)) - gate_1 /= denom - gate_2 /= denom - - # GSC Tensor - b = tf.one_hot( - tf.cast(position_in_expert_1, dtype=tf.int32), - expert_capacity_dim, - dtype=fprop_dtype, - name='one_hot_b_0') - # GSE Tensor - a = tf.expand_dims(gate_1 * mask_1_flat, -1) * tf.one_hot( - index_1, experts_dim, dtype=fprop_dtype) - # GSEC Tensor - first_part_of_combine_tensor = tf.einsum( - 'GSE,GSC->GSEC', a, b, name='first_part_of_combine_tensor') - - # GSC Tensor - b = tf.one_hot( - tf.cast(position_in_expert_2, dtype=tf.int32), - expert_capacity_dim, - dtype=fprop_dtype, - name='one_hot_b_1') - # GSE Tensor - a = tf.expand_dims(gate_2 * mask_2_flat, -1) * tf.one_hot( - index_2, experts_dim, dtype=fprop_dtype) - second_part_of_combine_tensor = tf.einsum( - 'GSE,GSC->GSEC', a, b, name='second_part_of_combine_tensor') - - # GSEC Tensor - combine_tensor = ( - first_part_of_combine_tensor + second_part_of_combine_tensor) - combine_tensor = _MaybeSplit(combine_tensor) - - # GSEC Tensor - dispatch_tensor = tf.cast(tf.cast(combine_tensor, tf.bool), fprop_dtype) - dispatch_tensor = _MaybeSplit(dispatch_tensor) - - # TODO(yonghui): compute and return per-group aux_loss. - return aux_loss, combine_tensor, dispatch_tensor - - -def Top2Gating(w, - inputs, - paddings, - num_devices, - experts_dim, - expert_capacity_dim, - local_dispatch, - fprop_dtype, - use_xla_sharding=True, - second_expert_policy='all', - second_expert_threshold=0.0, - legacy_mtf_behavior=True, - capacity_factor=None): - """Computes Top-2 gating for Mixture-of-Experts. - - See Top2GatingOnLogits for more details. - - Note that for local_dispatch original batch BLM is reshaped into GSM, each - group `g = 0...G-1` is being dispatched independently. - - Args: - w: gating weights for each experts. - inputs: G`SM Tensor. - paddings: G`S Tensor. - num_devices: number of MoE devices for local dispatch - experts_dim: number of experts. - expert_capacity_dim: number of examples per minibatch(group) per expert. - Each example is typically a vector of size input_dim, representing - embedded token or an element of Transformer layer output. - local_dispatch: whether dispatch is local to the group (G dim) - fprop_dtype: activations datatype to use. - use_xla_sharding: bool, True if this function is used for the xla_sharding - case. - second_expert_policy: 'all' or 'random', we optionally 'random'-ize dispatch - to second-best expert proportional to (weight / second_expert_threshold). - second_expert_threshold: threshold for probability normalization for - second_expert_policy == 'random'. - legacy_mtf_behavior: True for legacy behavior with no re-normalization of - expert assignment weights if we go over capacity or randomly decide to not - dispatch to second expert. - capacity_factor: if set, increases expert_capacity_dim to at least - `(group_size * capacity_factor) / experts_dim` - where `group_size` is the size of G dimension of `inputs`. If the - value of expert_capacity_dim is already big enough no change is made. - - Returns: - A tuple (dispatch_tensor, combine_tensor, aux_loss). - - - dispatch_tensor: G`SEC Tensor, scattering/dispatching inputs to - experts. - - combine_tensor: G`SEC Tensor. - combining expert outputs. - - aux_loss: auxiliary loss, equalizing the expert assignment ratios. - """ - orig_inputs = inputs - if not local_dispatch: - inputs = tf.reshape(inputs, [1, inputs.shape[0] * inputs.shape[1], -1]) - if paddings is not None: - paddings = tf.reshape(paddings, [1, -1]) - - logits = tf.einsum('GSM,ME->GSE', inputs, w) - - top1_expert_per_example = tf.math.argmax(logits, -1) - - tpu_summary.tensor('top1_expert', top1_expert_per_example) - - aux_loss, combine_tensor, dispatch_tensor = Top2GatingOnLogits( - inputs, paddings, logits, num_devices, experts_dim, expert_capacity_dim, - fprop_dtype, use_xla_sharding, second_expert_policy, - second_expert_threshold, legacy_mtf_behavior, capacity_factor) - - if not local_dispatch: - dispatch_tensor = tf.reshape( - dispatch_tensor, orig_inputs.shape[:2] + dispatch_tensor.shape[2:]) - combine_tensor = tf.reshape( - combine_tensor, orig_inputs.shape[:2] + combine_tensor.shape[2:]) - - return py_utils.NestedMap( - combine_tensor=combine_tensor, - dispatch_tensor=dispatch_tensor, - aux_loss=aux_loss) - - -def FeedForwardNetworksApplyGating(gating, - inputs, - reshaped_inputs, - wi_split, - wo_split, - num_devices, - num_groups, - bi_split=None, - bo_split=None, - dropout_rate=0.0): - """Apply top_2 gating to feedforward networks. - - Args: - gating: returns from Top2Gating consisting of: dispatch_tensor, G`SEC - Tensor, scattering/dispatching inputs to experts. combine_tensor, G`SEC - Tensor, combining expert outputs. aux_loss. auxiliary loss, equalizing the - expert assignment ratios - inputs: G`SM Tensor. - reshaped_inputs: [G*S/E, M] Tensor. - wi_split: First projection weights [E, M, H] of the feedforward networks. - wo_split: Last projection weights [E, H, M] of the feedforward networks. - num_devices: number of devices. - num_groups: number of groups (generally matches to or proportional to - num_devices). - bi_split: First projection bias [E, 1, H] of the feedforward networks. - bo_split: Last projection bias [E, 1, M] of the feedforward networks. - dropout_rate: Dropout rate. - - Returns: - outputs: G`SM Tensor. - aux_loss: scalar auxilliar loss. - """ - # dispatch_tensor G`SEC - expert_inputs = tf.einsum('GSEC,GSM->EGCM', - Split(gating.dispatch_tensor, 0, num_devices), - Split(reshaped_inputs, 0, num_devices)) - expert_inputs = Split(expert_inputs, 0, num_devices) - - M = reshaped_inputs.shape[-1] # pylint: disable=invalid-name - E = expert_inputs.shape[0] # pylint: disable=invalid-name - - # combine_tensor: G`SEC - # pylint: disable=invalid-name - G = gating.combine_tensor.shape[0] - assert num_groups == tf.compat.dimension_value(G) - C = gating.combine_tensor.shape[-1] # pylint: disable=invalid-name - A = G * C - # pylint: enable=invalid-name - - # Reshaping EGCM => EAM where A = G*C, e.g. - # - # with E=512, G=1024 - # - # (512, 1024, 4, 1024) => (512, 4096, 1024) - expert_inputs = tf.reshape(expert_inputs, [expert_inputs.shape[0], A, M]) - expert_inputs = Split(expert_inputs, 0, num_devices) - - h = tf.einsum('EAM,EMH->EAH', expert_inputs, wi_split) - h = Split(h, 0, num_devices) - if bi_split is not None: - h += Split(bi_split, 0, num_devices) - h = Split(h, 0, num_devices) - - h = tf.nn.relu(h) - - if dropout_rate: - # we generally do not use stateless dropout in MoE since it introduces - # large uint32 tensor broadcast (per dehao@ study) - h = tf.nn.dropout(h, dropout_rate) - - expert_outputs = tf.einsum('EAH,EHM->EAM', Split(h, 0, num_devices), wo_split) - if bo_split is not None: - expert_outputs = Split(expert_outputs, 0, num_devices) - expert_outputs += Split(bo_split, 0, num_devices) - expert_outputs = Split(expert_outputs, 0, num_devices) - expert_outputs = tf.reshape(expert_outputs, [E, G, C, M]) - - # same as tf.transpose - expert_outputs = tf.einsum('EGCM->GECM', expert_outputs) - - expert_outputs = Split(expert_outputs, 0, num_devices) - - # split by G dim - combined_outputs = tf.einsum('GSEC,GECM->GSM', - Split(gating.combine_tensor, 0, num_devices), - expert_outputs) - - outputs = Split(tf.reshape(combined_outputs, inputs.shape), 0, num_devices) - aux_loss = gating.aux_loss - return outputs, aux_loss - - -def GatherK(selected_pos, values, k, num_devices=1): - """Gather up to k elements from given tensors at selected pos under SPMD. - - Example:: - - # Input - k = 3 - - selected_pos = [ - [0, 0, 1, 1], - [0, 1, 1, 0], - [0, 0, 0, 0], - [1, 1, 1, 0], - [1, 1, 1, 1], # topk(k=3) largest indices are selected in this row. - ] - - value_2d = [ - [1, 3, 5, 7], - [9, 11, 13, 15], - [17, 19, 21, 23], - [25, 27, 29, 31], - [33, 35, 37, 39], - ] - - # Output: - output = [ - [0, 5, 7], - [0, 11, 13], - [0, 0, 0], - [25, 27, 29], - [35, 37, 39], - ] - - # Output padding: - output_padding = [ - [1, 0, 0], - [1, 0, 0], - [1, 1, 1], - [0, 0, 0], - [0, 0, 0], - ] - - Args: - selected_pos: a 0/1 2D tf.int32 tensor of shape [batch, time]. - values: a list of tensors, the rank of each is at least rank=2. [batch, - time, ...]. - k: a scalar tf.int32 tensor or a Python int. On TPU, k must be a - compile-time constant. - num_devices: number of TPU devices used in xla_sharding SPMD. - - Returns: - A tuple (output, padding). - - - output: a list of tensors of shape [batch, k, ...]. - - padding: a 2D 0/1 tensor of shape [batch, k], '1's are padded locations. - """ - global_batch, seq_len = py_utils.GetShape(selected_pos, 2) - if num_devices: - device_batch = global_batch // num_devices - else: - device_batch = global_batch - - for i in range(len(values)): - # Assert the first 2 dim of values[i] is [global_batch, seq_len] - values[i] = py_utils.HasShape(values[i], [global_batch, seq_len], 2) - # indices are 1-based for now, to distinguish between padding and selected - # locations. - indices = 1 + tf.range(tf.shape(values[0])[1], dtype=tf.int32) - # [1, seq_len] - indices = tf.expand_dims(indices, axis=0) - - # if 0, the position is not selected. - # [1, seq_len] * [global_batch, seq_len] => [global_batch, t] - # -- topk --> [global_batch, k] - topk_indices, _ = tf.math.top_k( - indices * tf.cast(selected_pos, indices.dtype), k) - - # [global_batch, k], sorted in ascending order. - indices = tf.reverse(topk_indices, [-1]) - # [global_batch, k], padded positions are '1's. - padding = tf.cast(tf.equal(indices, 0), values[0].dtype) - padding = Split(padding, 0, num_devices) - - # [global_batch, k], zero_based_indices - mp_idx = tf.maximum(0, indices - 1) - mp_idx = Split(mp_idx, 0, num_devices) - - # [device_batch, k] - if num_devices > 1 and py_utils.use_tpu(): - mp_idx = xla_sharding.auto_to_manual_spmd_partition( - mp_idx, xla_sharding.get_op_sharding(mp_idx.op)) - # [device_batch, k, 1] - mp_idx = tf.expand_dims(mp_idx, -1) - - # [device_batch] - batch_ids = tf.range(device_batch, dtype=tf.int32) - # [device_batch, 1, 1] - batch_ids = tf.reshape(batch_ids, [device_batch, 1, 1]) - # [device_batch, k, 1] - batch_ids = tf.broadcast_to(batch_ids, [device_batch, k, 1]) - - # [device_batch, k, 2] - final_indices = tf.concat([batch_ids, mp_idx], axis=-1) - - output = [] - for v in values: - # Begin manually partition gather. - v = Split(v, 0, num_devices) - v_shape = v.shape.as_list() - if num_devices > 1 and py_utils.use_tpu(): - op_sharding = xla_sharding.get_op_sharding(v.op) - v = xla_sharding.auto_to_manual_spmd_partition(v, op_sharding) - # Returns [global_batch, k, ...] - v_out = tf.gather_nd(v, final_indices) - - if num_devices > 1 and py_utils.use_tpu(): - v_shape[1] = k - v_out = xla_sharding.manual_to_auto_spmd_partition( - v_out, op_sharding, full_shape=tf.TensorShape(v_shape)) - output.append(v_out) - - return output, padding - - -def GetSentenceEmbeddings(inputs, segment_id): - """Returns the average sentence embedding to gate by. - - Example:: - - inputs: - segment_id: - - Args: - inputs: G`SM Tensor. - segment_id: G`S Tensor. - - Returns: - sentence_embeddings: GSM Tensor that is an average of the input embeddings - per segment. - """ - reshaped_inputs = tf.reshape(inputs, [-1, inputs.shape[-1]]) - - # We set num_segments to a large value so that shape is known at compile time. - max_segments = py_utils.GetShape(reshaped_inputs)[0] - # We change the padding to be max_segments - 1 instead of 0 because - # tf.math.unsorted_segment_mean because it only accepts values between 1 and - # max_segments. - modified_segment_id = tf.cast( - segment_id + max_segments * tf.cast( - tf.equal(segment_id, 0), dtype=tf.dtypes.as_dtype(segment_id.dtype)) - - 1, - dtype=tf.int32) - reshaped_segment_id = tf.reshape(modified_segment_id, [-1]) - - # Takes the mean of all segments, w/ 0s for the padding. - params = tf.concat([ - tf.math.unsorted_segment_mean(reshaped_inputs, reshaped_segment_id, - max_segments)[:-1], - tf.zeros([1, reshaped_inputs.shape[-1]], dtype=reshaped_inputs.dtype) - ], - axis=0) - raw_sentence_embeddings = tf.gather(params, modified_segment_id) - - # sentence_embedding: - sentence_embeddings = tf.reshape(raw_sentence_embeddings, inputs.shape) - - return sentence_embeddings - - -def SentenceTop2Gating(w, - inputs, - paddings, - segment_id, - num_devices, - experts_dim, - expert_capacity_dim, - local_dispatch, - fprop_dtype, - use_xla_sharding=True, - second_expert_policy='all', - second_expert_threshold=0.0, - legacy_mtf_behavior=True, - embedding_type='sentence', - capacity_factor=None): - """Computes Top-2 sentence gating for Mixture-of-Experts. - - Instead of using the each token, this function uses embedding_type to return a - sentence-wise embedding to create dispatch and combine tensors that gate - the entire sentence. - - See SentenceTop2GatingOnLogits for more details. - - Note that for local_dispatch original batch BLM is reshaped into GSM, each - group `g = 0...G-1` is being dispatched independently. - - Args: - w: gating weights for each experts. - inputs: G`SM Tensor. - paddings: G`S Tensor. - segment_id: G`SM Tensor used for differentiating different sentences in an - input example. - num_devices: number of MoE devices for local dispatch - experts_dim: number of experts. - expert_capacity_dim: number of examples per minibatch(group) per expert. - Each example is typically a vector of size input_dim, representing - embedded token or an element of Transformer layer output. - local_dispatch: whether dispatch is local to the group (G dim) - fprop_dtype: activations datatype to use. - use_xla_sharding: bool, True if this function is used for the xla_sharding - case. - second_expert_policy: 'all' or 'random', we optionally 'random'-ize dispatch - to second-best expert proportional to (weight / second_expert_threshold). - second_expert_threshold: threshold for probability normalization for - second_expert_policy == 'random'. - legacy_mtf_behavior: True for legacy behavior with no re-normalization of - expert assignment weights if we go over capacity or randomly decide to not - dispatch to second expert. - embedding_type: 'sentence' by default. Options: 'sentence'. Setting this - option calls GetSentenceEmbeddings. - capacity_factor: if set, increases expert_capacity_dim to at least - (group_size * capacity_factor) / experts_dim where `group_size` is the - size of G dimension of `inputs`. If the value of expert_capacity_dim is - already big enough no change is made. - - Returns: - A tuple (dispatch_tensor, combine_tensor, aux_loss). - - - dispatch_tensor: G`SEC Tensor, scattering/dispatching inputs to - experts. - - combine_tensor: G`SEC Tensor. - combining expert outputs. - - aux_loss: auxiliary loss, equalizing the expert assignment ratios. - """ - assert embedding_type in ['sentence'] - orig_inputs = inputs - - if not local_dispatch: - inputs = tf.reshape(inputs, [1, inputs.shape[0] * inputs.shape[1], -1]) - if paddings is not None: - paddings = tf.reshape(paddings, [1, -1]) - - if embedding_type == 'sentence': - sentence_embeddings = GetSentenceEmbeddings(inputs, segment_id) - - logits = tf.einsum('GSM,ME->GSE', sentence_embeddings, w) - aux_loss, combine_tensor, dispatch_tensor = Top2GatingOnLogits( - sentence_embeddings, paddings, logits, num_devices, experts_dim, - expert_capacity_dim, fprop_dtype, use_xla_sharding, second_expert_policy, - second_expert_threshold, legacy_mtf_behavior, capacity_factor) - - if not local_dispatch: - dispatch_tensor = tf.reshape( - dispatch_tensor, orig_inputs.shape[:2] + dispatch_tensor.shape[2:]) - combine_tensor = tf.reshape( - combine_tensor, orig_inputs.shape[:2] + combine_tensor.shape[2:]) - - return py_utils.NestedMap( - combine_tensor=combine_tensor, - dispatch_tensor=dispatch_tensor, - aux_loss=aux_loss) - - -def TaskTop2Gating(w, - inputs, - paddings, - task_embeddings, - num_devices, - experts_dim, - expert_capacity_dim, - local_dispatch, - fprop_dtype, - use_xla_sharding=True, - second_expert_policy='all', - second_expert_threshold=0.0, - legacy_mtf_behavior=True): - """Computes Top-2 sentence gating for Mixture-of-Experts. - - Instead of using the each token, this function uses embedding_type to return a - sentence-wise embedding to create dispatch and combine tensors that gate - the entire sentence. - - See SentenceTop2GatingOnLogits for more details. - - Note that for local_dispatch original batch BLM is reshaped into GSM, each - group `g = 0...G-1` is being dispatched independently. - - Args: - w: gating weights for each experts. - inputs: G`SM Tensor. - paddings: G`S Tensor. - task_embeddings: G`SM Tensor. - num_devices: number of MoE devices for local dispatch - experts_dim: number of experts. - expert_capacity_dim: number of examples per minibatch(group) per expert. - Each example is typically a vector of size input_dim, representing - embedded token or an element of Transformer layer output. - local_dispatch: whether dispatch is local to the group (G dim) - fprop_dtype: activations datatype to use. - use_xla_sharding: bool, True if this function is used for the xla_sharding - case. - second_expert_policy: 'all' or 'random', we optionally 'random'-ize dispatch - to second-best expert proportional to (weight / second_expert_threshold). - second_expert_threshold: threshold for probability normalization for - second_expert_policy == 'random'. - legacy_mtf_behavior: True for legacy behavior with no re-normalization of - expert assignment weights if we go over capacity or randomly decide to not - dispatch to second expert. - - Returns: - A tuple (dispatch_tensor, combine_tensor, aux_loss): - - - dispatch_tensor: G`SEC Tensor, scattering/dispatching inputs to - experts. - - combine_tensor: G`SEC Tensor. - combining expert outputs. - - aux_loss: auxiliary loss, equalizing the expert assignment ratios. - """ - orig_inputs = inputs - if not local_dispatch: - inputs = tf.reshape(inputs, [1, inputs.shape[0] * inputs.shape[1], -1]) - task_embeddings = tf.reshape( - task_embeddings, - [1, task_embeddings.shape[0] * task_embeddings.shape[1], -1]) - if paddings is not None: - paddings = tf.reshape(paddings, [1, -1]) - - logits = tf.einsum('GSM,ME->GSE', task_embeddings, w) - aux_loss, combine_tensor, dispatch_tensor = Top2GatingOnLogits( - task_embeddings, paddings, logits, num_devices, experts_dim, - expert_capacity_dim, fprop_dtype, use_xla_sharding, second_expert_policy, - second_expert_threshold, legacy_mtf_behavior) - - if not local_dispatch: - dispatch_tensor = tf.reshape( - dispatch_tensor, orig_inputs.shape[:2] + dispatch_tensor.shape[2:]) - combine_tensor = tf.reshape( - combine_tensor, orig_inputs.shape[:2] + combine_tensor.shape[2:]) - - return py_utils.NestedMap( - combine_tensor=combine_tensor, - dispatch_tensor=dispatch_tensor, - aux_loss=aux_loss) diff --git a/lingvo/core/multitask_model.py b/lingvo/core/multitask_model.py deleted file mode 100644 index ab4876d9..00000000 --- a/lingvo/core/multitask_model.py +++ /dev/null @@ -1,127 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Multitask models.""" - -from lingvo import compat as tf -from lingvo.core import base_model -from lingvo.core import py_utils - - -class SharedEncoderModel(base_model.MultiTaskModel): - """Multitask model that shares encoder between tasks.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('encoder_to_share', None, - 'The task name whose encoder should be shared.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.encoder_to_share in self.task_names - - # Assign the encoder from p.encoder_to_share task to all other tasks. - encoder = self.GetTask(p.encoder_to_share).encoder - for name in self.task_names: - if name != p.encoder_to_share: - task = self.GetTask(name) - assert 'encoder' not in task.children - task.AddChild('encoder', encoder) - - def _CreateChildrenVariables(self): - # Ensure p.encoder_to_share is created first. - task_name = self.params.encoder_to_share - with tf.name_scope(self.params.name): - if self.params.task_name_var_scope: - with tf.variable_scope(task_name): - self.GetTask(task_name).InstantiateVariables() - else: - self.GetTask(task_name).InstantiateVariables() - super()._CreateChildrenVariables() - - -class SharedEncoderDecoderModel(base_model.MultiTaskModel): - """Multitask model that shares both encoder and decoder between tasks.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('encoder_to_share', None, - 'The task name whose encoder should be shared.') - p.Define('decoder_to_share', None, - 'The task name whose decoder should be shared.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.encoder_to_share in self.task_names - assert p.decoder_to_share in self.task_names - - # Assign the encoder from p.encoder_to_share task to all other tasks, and - # assign the decoder from p.decoder_to_share task to all other tasks. - encoder = self.GetTask(p.encoder_to_share).encoder - decoder = self.GetTask(p.decoder_to_share).decoder - for name in self.task_names: - - if name != p.encoder_to_share: - task = self.GetTask(name) - assert 'encoder' not in task.children - task.AddChild('encoder', encoder) - - if name != p.decoder_to_share: - task = self.GetTask(name) - assert 'decoder' not in task.children - task.AddChild('decoder', decoder) - - -class RegExSharedVariableModel(base_model.MultiTaskModel): - """Multitask models that share variables across different tasks. - - Note, do NOT use this model unless you know exactly what you are trying to do - and you have verified that it indeed achieves what you would have expected. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'variable_renaming_rules', None, - 'A list/tuple of variable renaming rules. Each element in the' - ' list is a pair of strings. The first element is a regex' - ' expression while the second element is a python format string.') - return p - - def __init__(self, params): - # Enable variable sharing. - with py_utils.OpportunisticVariableReuseScope(): - with py_utils.VariableRenameScope(params.variable_renaming_rules): - super().__init__(params) - - def InstantiateVariables(self): - # Enable variable sharing. - with py_utils.OpportunisticVariableReuseScope(): - with py_utils.VariableRenameScope(self.params.variable_renaming_rules): - super().InstantiateVariables() - - def ConstructFPropBPropGraph(self): - # We need to override this since constructing the BPropGraph - # creates slot variables. - with py_utils.OpportunisticVariableReuseScope(): - with py_utils.VariableRenameScope(self.params.variable_renaming_rules): - super().ConstructFPropBPropGraph() diff --git a/lingvo/core/multitask_model_test.py b/lingvo/core/multitask_model_test.py deleted file mode 100644 index 3279f360..00000000 --- a/lingvo/core/multitask_model_test.py +++ /dev/null @@ -1,138 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for multitask_model.""" - -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import base_layer -from lingvo.core import base_model -from lingvo.core import base_model_params -from lingvo.core import hyperparams -from lingvo.core import multitask_model -from lingvo.core import py_utils -from lingvo.core import test_utils - - -class MultiTaskModelTest(test_utils.TestCase): - - class _TestTask(base_model.BaseTask): - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'test_task' - p.encoder = base_layer.BaseLayer.Params() - p.encoder.name = 'enc' - p.decoder = base_layer.BaseLayer.Params() - p.decoder.name = 'dec' - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.encoder: - self.CreateChild('encoder', p.encoder) - if p.decoder: - self.CreateChild('decoder', p.decoder) - - def testSharedEncoderModel(self): - p = multitask_model.SharedEncoderModel.Params() - p.name = 'test' - p.encoder_to_share = 'p0' - - p0 = MultiTaskModelTest._TestTask.Params() - p1 = MultiTaskModelTest._TestTask.Params() - p1.encoder = None - - p.input = base_model_params.MultiTaskModelParams().Train() - p.input.Define('p0', base_input_generator.BaseInputGenerator.Params(), '') - p.input.Define('p1', base_input_generator.BaseInputGenerator.Params(), '') - p.task_params = hyperparams.Params() - p.task_params.Define('p0', p0, '') - p.task_params.Define('p1', p1, '') - p.task_probs = hyperparams.Params() - p.task_probs.Define('p0', 0.5, '') - p.task_probs.Define('p1', 0.5, '') - - model = p.Instantiate() - self.assertEqual(model.p0.encoder, model.p1.encoder) - - def testSharedEncoderDecoderModel(self): - p = multitask_model.SharedEncoderDecoderModel.Params() - p.name = 'test' - p.encoder_to_share = 'p0' - p.decoder_to_share = 'p0' - - p0 = MultiTaskModelTest._TestTask.Params() - p1 = MultiTaskModelTest._TestTask.Params() - p1.encoder = None - p1.decoder = None - - p.input = base_model_params.MultiTaskModelParams().Train() - p.input.Define('p0', base_input_generator.BaseInputGenerator.Params(), '') - p.input.Define('p1', base_input_generator.BaseInputGenerator.Params(), '') - p.task_params = hyperparams.Params() - p.task_params.Define('p0', p0, '') - p.task_params.Define('p1', p1, '') - p.task_probs = hyperparams.Params() - p.task_probs.Define('p0', 0.5, '') - p.task_probs.Define('p1', 0.5, '') - - model = p.Instantiate() - self.assertEqual(model.p0.encoder, model.p1.encoder) - self.assertEqual(model.p0.decoder, model.p1.decoder) - - class _TestTaskWithVars(base_model.BaseTask): - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'test_task' - return p - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - pc = py_utils.WeightParams(shape=[10, 10], dtype=tf.float32) - self.CreateVariable('weight', pc) - - def testRegExSharedVariableModel(self): - p = multitask_model.RegExSharedVariableModel.Params() - p.name = 'test' - p.variable_renaming_rules = [('p./(.*)', 'shared/%s')] - - p0 = MultiTaskModelTest._TestTaskWithVars.Params() - p1 = MultiTaskModelTest._TestTaskWithVars.Params() - - p.input = base_model_params.MultiTaskModelParams().Train() - p.input.Define('p0', base_input_generator.BaseInputGenerator.Params(), '') - p.input.Define('p1', base_input_generator.BaseInputGenerator.Params(), '') - - p.task_params = hyperparams.Params() - p.task_params.Define('p0', p0, '') - p.task_params.Define('p1', p1, '') - p.task_probs = hyperparams.Params() - - p.task_probs.Define('p0', 0.5, '') - p.task_probs.Define('p1', 0.5, '') - - model = p.Instantiate() - all_vars = model.vars - self.assertEqual('shared/test_task/weight/var:0', all_vars.p0.weight.name) - self.assertEqual('shared/test_task/weight/var:0', all_vars.p1.weight.name) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/ops/BUILD b/lingvo/core/ops/BUILD deleted file mode 100644 index 960d3e76..00000000 --- a/lingvo/core/ops/BUILD +++ /dev/null @@ -1,518 +0,0 @@ -load( - "//lingvo:lingvo.bzl", - "custom_kernel_library", - "gen_op_cclib", - "gen_op_pylib", - "lingvo_cc_library", - "lingvo_cc_test", - "lingvo_cc_test_library", - "lingvo_proto_cc", - "lingvo_proto_py", -) - -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -cc_library( - name = "x_ops_helper", - hdrs = ["x_ops_helper.h"], -) - -# Op definitions to be used from python. -gen_op_cclib( - name = "x_ops", - srcs = ["x_ops.cc"], - deps = [ - ":x_ops_helper", - # Implicit tensorflow C++ proto dependency. - ], -) - -gen_op_pylib( - name = "py_x_ops", - srcs = ["__init__.py"], - cc_lib_name = "x_ops", - kernel_deps = [ - ":op_kernels", - ], - py_deps = [ - "//lingvo:compat", - ], - visibility = ["//visibility:private"], -) - -py_library( - name = "ops", - srcs = ["__init__.py"], - srcs_version = "PY3", - deps = [ - ":py_x_ops", # buildcleaner: keep - "//lingvo:compat", - ], -) - -# Op kernel deps. -cc_library( - name = "op_kernels", - deps = [ - ":assert_kernels", - ":beam_search_step_op_kernels", - ":best_step_op_kernels", - ":functional_ops_kernels", - ":generic_input_op_kernels", - ":mass_op", - ":ml_perf_subword_op", - ":pack_ops", - ":preconditioner_op_kernels", - ":random_ops_kernels", - ":static_map_op", - ":tokenizer_ops_kernels", - ], -) - -lingvo_cc_library( - name = "rope", - hdrs = ["rope.h"], - deps = [ - # Implicit rope dependency. - ], -) - -lingvo_cc_library( - name = "ascii_tokenizer", - srcs = ["ascii_tokenizer.cc"], - hdrs = ["ascii_tokenizer.h"], -) - -lingvo_cc_library( - name = "simple_vocab", - srcs = ["simple_vocab.cc"], - hdrs = ["simple_vocab.h"], -) - -custom_kernel_library( - name = "ml_perf_subword_op", - srcs = ["ml_perf_subword_op.cc"], - hdrs = ["ml_perf_subword_op.h"], - op_def_lib = [":x_ops"], - deps = [ - "@icu//:common", - ], -) - -py_test( - name = "simple_vocab_test", - srcs = ["simple_vocab_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ops", - "//lingvo/core:test_utils", - # Implicit tensorflow dependency. - ], -) - -lingvo_cc_library( - name = "record", - srcs = [ - "chain_record_yielder.cc", - "record_batcher.cc", - "record_debug.cc", - "record_yielder.cc", - "sequential_record_yielder.cc", - "weighted_mix_record_yielder.cc", - ], - hdrs = [ - "chain_record_yielder.h", - "record_batcher.h", - "record_yielder.h", - "sequential_record_yielder.h", - "weighted_mix_record_yielder.h", - ], - deps = [ - ":rope", - ":versioned_file_set_proto_cc", - # Implicit absl.synchronization dependency. - ], -) - -lingvo_cc_test( - name = "record_yielder_test", - srcs = ["record_yielder_test.cc"], - deps = [ - ":input_common", - ":record", - ":yielder_test_helper", - ], -) - -lingvo_cc_test( - name = "weighted_mix_record_yielder_test", - srcs = ["weighted_mix_record_yielder_test.cc"], - deps = [ - ":input_common", - ":record", - ":yielder_test_helper", - ], -) - -lingvo_cc_test( - name = "chain_record_yielder_test", - srcs = ["chain_record_yielder_test.cc"], - deps = [ - ":input_common", - ":record", - ":yielder_test_helper", - ], -) - -lingvo_cc_test( - name = "record_batcher_test", - srcs = ["record_batcher_test.cc"], - deps = [ - ":input_common", - ":record", - ], -) - -lingvo_cc_library( - name = "input_common", - srcs = ["input_common.cc"], - hdrs = ["input_common.h"], - deps = [ - ":record", - ], -) - -lingvo_cc_test_library( - name = "yielder_test_helper", - srcs = [ - "yielder_test_helper.cc", - ], - hdrs = [ - "yielder_test_helper.h", - ], - deps = [ - ":record", - ], -) - -lingvo_cc_library( - name = "text_packing", - srcs = ["text_packing.cc"], - hdrs = ["text_packing.h"], -) - -lingvo_cc_test( - name = "text_packing_test", - srcs = ["text_packing_test.cc"], - deps = [ - ":text_packing", - ], -) - -###################### Op kernel implementations. - -custom_kernel_library( - name = "static_map_op", - srcs = ["static_map_op.cc"], - op_def_lib = [":x_ops"], -) - -py_test( - name = "static_map_op_test", - srcs = ["static_map_op_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ops", - "//lingvo:compat", - "//lingvo/core:test_utils", - ], -) - -custom_kernel_library( - name = "assert_kernels", - srcs = ["assert_kernels.cc"], - op_def_lib = [":x_ops"], -) - -py_test( - name = "assert_ops_test", - srcs = ["assert_ops_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ops", - "//lingvo/core:test_utils", - # Implicit tensorflow dependency. - ], -) - -custom_kernel_library( - name = "beam_search_step_op_kernels", - srcs = ["beam_search_step_op_kernels.cc"], - hdrs = ["beam_search_step_op_kernels.h"], - op_def_lib = [":x_ops"], - deps = [ - ":hyps_proto_cc", - ":simple_vocab", - ], -) - -lingvo_cc_test( - name = "beam_search_step_op_top_k_test", - srcs = ["beam_search_step_op_top_k_test.cc"], - deps = [ - ":beam_search_step_op_kernels", - ], -) - -py_test( - name = "beam_search_step_op_test", - srcs = ["beam_search_step_op_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":hyps_py_pb2", - ":ops", - # Implicit python proto dependency. - "//lingvo:compat", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - # Implicit tensorflow dependency. - ], -) - -custom_kernel_library( - name = "best_step_op_kernels", - srcs = ["best_step_op_kernels.cc"], - op_def_lib = [":x_ops"], - deps = [ - # Implicit tensorflow C++ proto dependency. - ], -) - -py_test( - name = "best_step_op_test", - srcs = ["best_step_op_test.py"], - data = [ - "//lingvo/core/ops/testdata:best_step_testdata", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ops", - "//lingvo:compat", - "//lingvo/core:test_helper", - "//lingvo/core:test_utils", - # Implicit tensorflow dependency. - ], -) - -custom_kernel_library( - name = "functional_ops_kernels", - srcs = ["functional_ops_kernels.cc"], - op_def_lib = [":x_ops"], - deps = [ - # Implicit absl.synchronization dependency. - ], -) - -py_test( - name = "functional_ops_test", - srcs = ["functional_ops_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ops", - "//lingvo:compat", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - # Implicit tensorflow dependency. - # Implicit tensorflow python framework_for_generated_wrappers dependency. - ], -) - -custom_kernel_library( - name = "generic_input_op_kernels", - srcs = ["generic_input_op_kernels.cc"], - op_def_lib = [":x_ops"], - deps = [":input_common"], -) - -custom_kernel_library( - name = "random_ops_kernels", - srcs = ["random_ops_kernels.cc"], - op_def_lib = [":x_ops"], - deps = [ - # Implicit absl.synchronization dependency. - ], -) - -py_test( - name = "random_ops_test", - srcs = ["random_ops_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ops", - "//lingvo:compat", - "//lingvo/core:test_utils", - # Implicit tensorflow dependency. - ], -) - -lingvo_cc_library( - name = "preconditioner_captain", - srcs = ["preconditioner_captain.cc"], - hdrs = ["preconditioner_captain.h"], - deps = [ - # Implicit tensorflow core_cpu dependency. - # Implicit tensorflow C++ proto dependency. - ], -) - -custom_kernel_library( - name = "preconditioner_op_kernels", - srcs = ["preconditioner_op_kernels.cc"], - op_def_lib = [":x_ops"], - deps = [ - ":preconditioner_captain", - ], -) - -py_test( - name = "preconditioner_op_kernels_test", - srcs = ["preconditioner_op_kernels_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ops", - "//lingvo/core:test_helper", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - # Implicit tensorflow dependency. - ], -) - -lingvo_cc_library( - name = "tokenizer_op_headers", - hdrs = ["tokenizer_op_headers.h"], -) - -custom_kernel_library( - name = "tokenizer_ops_kernels", - srcs = ["tokenizer_ops_kernels.cc"], - hdrs = ["tokenizer_op_headers.h"], - op_def_lib = [":x_ops"], - deps = [ - ":ascii_tokenizer", - ":simple_vocab", - ], -) - -py_test( - name = "tokenizer_ops_test", - srcs = ["tokenizer_ops_test.py"], - data = [ - "//lingvo/core/ops/testdata:bpe_codes_vocab", - "//lingvo/core/ops/testdata:bpe_words_vocab", - "//lingvo/core/ops/testdata:mlperf_vocab", - "//lingvo/core/ops/testdata:test_ngrams", - "//lingvo/core/ops/testdata:test_vocab", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ops", - "//lingvo/core:test_helper", - "//lingvo/core:test_utils", - # Implicit tensorflow dependency. - ], -) - -custom_kernel_library( - name = "pack_ops", - srcs = ["pack_ops.cc"], - op_def_lib = [":x_ops"], - deps = [ - ":text_packing", - # Implicit absl.container.flat_hash_map dependency. - # Implicit absl.strings dependency. - # Implicit absl.synchronization dependency. - ], -) - -py_test( - name = "pack_ops_test", - srcs = ["pack_ops_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ops", - "//lingvo:compat", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - # Implicit tensorflow dependency. - ], -) - -custom_kernel_library( - name = "mass_op", - srcs = ["mass_op.cc"], - op_def_lib = [":x_ops"], -) - -py_test( - name = "mass_op_test", - srcs = ["mass_op_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ops", - "//lingvo:compat", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - # Implicit tensorflow dependency. - ], -) - -lingvo_proto_cc( - name = "hyps_proto", - src = "hyps.proto", -) - -lingvo_proto_py( - name = "hyps_py_pb2", - src = "hyps.proto", - deps = [":hyps_proto"], -) - -lingvo_proto_cc( - name = "record_proto", - src = "record.proto", - deps = [ - # Implicit tensorflow proto dependency. - ], -) - -lingvo_proto_cc( - name = "versioned_file_set_proto", - src = "versioned_file_set.proto", -) - -lingvo_proto_py( - name = "versioned_file_set_py_pb2", - src = "versioned_file_set.proto", - deps = [":versioned_file_set_proto"], -) - -lingvo_proto_py( - name = "record_py_pb2", - src = "record.proto", - deps = [":record_proto"], -) diff --git a/lingvo/core/ops/__init__.py b/lingvo/core/ops/__init__.py deleted file mode 100644 index e7ca7006..00000000 --- a/lingvo/core/ops/__init__.py +++ /dev/null @@ -1,67 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Operations.""" - -from lingvo import compat as tf - -# Try static linking: -try: - from lingvo.core.ops import gen_x_ops # pylint: disable=g-import-not-at-top -except ImportError: - gen_x_ops = tf.load_op_library( - tf.resource_loader.get_path_to_datafile('x_ops.so')) - -# Set gen_x_ops function module so sphinx generates documentation. -for v in gen_x_ops.__dict__.values(): - try: - v.__module__ = 'lingvo.core.ops' - except: # pylint: disable=bare-except - pass - -assert_shape_match = gen_x_ops.assert_shape_match -assert_same_dim0 = gen_x_ops.assert_same_dim0 -random_permutation_sequence = gen_x_ops.random_permutation_sequence - -best_step = gen_x_ops.best_step - -beam_search_step = gen_x_ops.beam_search_step -top_k_terminated_hyps = gen_x_ops.top_k_terminated_hyps -unpack_hyp = gen_x_ops.unpack_hyp -hyps_from_beam_search_outs = gen_x_ops.hyps_from_beam_search_outs - -cached_call = gen_x_ops.cached_call - -vocab_token_to_id = gen_x_ops.vocab_token_to_id -vocab_id_to_token = gen_x_ops.vocab_id_to_token -token_in_vocab = gen_x_ops.token_in_vocab -ascii_to_token_id = gen_x_ops.ascii_to_token_id -str_to_vocab_tokens = gen_x_ops.str_to_vocab_tokens -id_to_ascii = gen_x_ops.id_to_ascii -ml_perf_subword_id_to_string = gen_x_ops.ml_perf_subword_id_to_string -ngram_id_to_token = gen_x_ops.ngram_id_to_token -bpe_ids_to_words = gen_x_ops.bpe_ids_to_words -bpe_words_to_ids = gen_x_ops.bpe_words_to_ids - -static_map_string_int = gen_x_ops.static_map_string_int -static_map_int_string = gen_x_ops.static_map_int_string -static_map_int_int = gen_x_ops.static_map_int_int - -get_preconditioners = gen_x_ops.get_preconditioners -compute_preconditioners = gen_x_ops.compute_preconditioners - -pack_sequences = gen_x_ops.pack_sequences -apply_packing = gen_x_ops.apply_packing -mass = gen_x_ops.mass diff --git a/lingvo/core/ops/ascii_tokenizer.cc b/lingvo/core/ops/ascii_tokenizer.cc deleted file mode 100644 index 661ee878..00000000 --- a/lingvo/core/ops/ascii_tokenizer.cc +++ /dev/null @@ -1,180 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/ascii_tokenizer.h" - -#include -#include -#include - -#include "tensorflow/core/lib/strings/str_util.h" -#include "tensorflow/core/platform/logging.h" - -namespace tensorflow { -namespace lingvo { -namespace { - -const string& FindOrDie(const std::unordered_map& m, int32 k) { - const auto it = m.find(k); - CHECK(it != m.end()); - return it->second; -} - -const int32 kUnkId = 0; -const int32 kSOSId = 1; -const int32 kEOSId = 2; -const int32 kEOWId = 3; -const int32 kNoiseId = 4; -const int32 kEpsilonId = 73; -const int32 kTextOnlyId = 74; -const int32 kSORWId = 75; // indicator for start of rare word -const int32 kMaxTokenId = 75; - -struct CharTokenizer { - std::unordered_map token_to_id; - std::unordered_map id_to_token; - - string epsilon_token; - string unk_token; - string noise_token; - string eos_token; - string sos_token; - string text_only_token; - string sorw_token; - - string IdToToken(int32 id) const { - const auto it = id_to_token.find(id); - if (it != id_to_token.end()) return it->second; - return unk_token; - } - - int32 TokenToId(const string& tok) const { - const auto it = token_to_id.find(tok); - if (it != token_to_id.end()) return it->second; - return kUnkId; - } -}; - -const CharTokenizer* CreateTokenizer() { - CharTokenizer* ct = new CharTokenizer(); - ct->id_to_token = {{0, ""}, {1, ""}, {2, ""}, - {3, " "}, {4, ""}, {5, "a"}, - {6, "b"}, {7, "c"}, {8, "d"}, - {9, "e"}, {10, "f"}, {11, "g"}, - {12, "h"}, {13, "i"}, {14, "j"}, - {15, "k"}, {16, "l"}, {17, "m"}, - {18, "n"}, {19, "o"}, {20, "p"}, - {21, "q"}, {22, "r"}, {23, "s"}, - {24, "t"}, {25, "u"}, {26, "v"}, - {27, "w"}, {28, "x"}, {29, "y"}, - {30, "z"}, {31, "."}, {32, "\'"}, - {33, "-"}, {34, ":"}, {35, "!"}, - {36, "~"}, {37, "`"}, {38, ";"}, - {39, "0"}, {40, "1"}, {41, "2"}, - {42, "3"}, {43, "4"}, {44, "5"}, - {45, "6"}, {46, "7"}, {47, "8"}, - {48, "9"}, {49, "\""}, {50, "#"}, - {51, "$"}, {52, "%"}, {53, "&"}, - {54, "("}, {55, ")"}, {56, "*"}, - {57, "+"}, {58, ","}, {59, "/"}, - {60, "<"}, {61, "="}, {62, ">"}, - {63, "?"}, {64, "@"}, {65, "["}, - {66, "\\"}, {67, "]"}, {68, "^"}, - {69, "_"}, {70, "{"}, {71, "|"}, - // 73 - {72, "}"}, {73, ""}, {74, ""}, - {75, ""}}; - // What is a "block" for an RNN-T? It is presumably what is emitted - // by the RNN-T model to signal that it is time to switch from - // prediction to encoder. - - // kEpsilonWord: end-of-block for neural transducer. - for (const std::pair& p : ct->id_to_token) { - CHECK_LE(p.first, kMaxTokenId); - CHECK(ct->token_to_id.insert({p.second, p.first}).second); - } - ct->unk_token = FindOrDie(ct->id_to_token, kUnkId); - ct->noise_token = FindOrDie(ct->id_to_token, kNoiseId); - ct->epsilon_token = FindOrDie(ct->id_to_token, kEpsilonId); - ct->sos_token = FindOrDie(ct->id_to_token, kSOSId); - ct->eos_token = FindOrDie(ct->id_to_token, kEOSId); - ct->sorw_token = FindOrDie(ct->id_to_token, kSORWId); - ct->text_only_token = FindOrDie(ct->id_to_token, kTextOnlyId); - return ct; -} - -const CharTokenizer* GetTokenizer() { - static const CharTokenizer* tokenizer = CreateTokenizer(); - return tokenizer; -} - -} // namespace - -string AsciiTokenizer::ConvertString(const string& transcript) { - string result = transcript; - std::transform(result.begin(), result.end(), result.begin(), ::tolower); - return result; -} - -int32 AsciiTokenizer::NumTokens() { return kMaxTokenId + 1; } - -std::vector AsciiTokenizer::StringToIds(const string& label) { - const CharTokenizer* tokenizer = GetTokenizer(); - const string converted = ConvertString(label); - const StringPiece converted_view(converted); - - std::vector ids; - const std::vector> special_token_ids{ - {tokenizer->unk_token, kUnkId}, - {tokenizer->noise_token, kNoiseId}, - {tokenizer->sos_token, kSOSId}, - {tokenizer->eos_token, kEOSId}, - {tokenizer->epsilon_token, kEpsilonId}, - {tokenizer->text_only_token, kTextOnlyId}, - {tokenizer->sorw_token, kSORWId}, - }; - - for (int i = 0; i < converted.size(); ++i) { - bool is_special_token = false; - for (const auto& token_id : special_token_ids) { - if (str_util::StartsWith(converted_view.substr(i), token_id.first)) { - ids.push_back(token_id.second); - i += token_id.first.size() - 1; - is_special_token = true; - break; - } - } - if (!is_special_token) { - ids.push_back(tokenizer->TokenToId(string(1, converted[i]))); - } - } - return ids; -} - -std::vector AsciiTokenizer::IdToStrings(const std::vector& ids) { - const CharTokenizer* tokenizer = GetTokenizer(); - std::vector out_strings(ids.size()); - for (int i = 0; i < ids.size(); ++i) { - out_strings[i] = tokenizer->IdToToken(ids[i]); - } - return out_strings; -} - -string AsciiTokenizer::JoinLabels(const std::vector& labels) { - return str_util::Join(labels, ""); -} - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/ascii_tokenizer.h b/lingvo/core/ops/ascii_tokenizer.h deleted file mode 100644 index 1f19be75..00000000 --- a/lingvo/core/ops/ascii_tokenizer.h +++ /dev/null @@ -1,49 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#ifndef LINGVO_CORE_OPS_ASCII_TOKENIZER_H_ -#define LINGVO_CORE_OPS_ASCII_TOKENIZER_H_ - -#include -#include - -#include "tensorflow/core/platform/types.h" - -namespace tensorflow { -namespace lingvo { - -// A static simple tokenizer that maps a small vocabulary of character -// tokens for (lower case) letters, digits, and punctuation symbols. -class AsciiTokenizer { - public: - // Convert string to a set of graphemes. - static string ConvertString(const string& transcript); - - // Returns the number of tokens. - static int32 NumTokens(); - - // Splits 'label' into tokens and returns their token ids. - static std::vector StringToIds(const string& label); - - // Convert 'ids' back into tokens. - static std::vector IdToStrings(const std::vector& ids); - - // Joins the token labels into a string. - static string JoinLabels(const std::vector& labels); -}; - -} // namespace lingvo -} // namespace tensorflow -#endif // LINGVO_CORE_OPS_ASCII_TOKENIZER_H_ diff --git a/lingvo/core/ops/assert_kernels.cc b/lingvo/core/ops/assert_kernels.cc deleted file mode 100644 index aca98e30..00000000 --- a/lingvo/core/ops/assert_kernels.cc +++ /dev/null @@ -1,113 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" - -namespace tensorflow { -namespace lingvo { -namespace { - -static const int kUnknown = -1; - -class AssertShapeMatchOp : public OpKernel { - public: - explicit AssertShapeMatchOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, ctx->GetAttr("msg", &msg_)); - } - - ~AssertShapeMatchOp() override {} - - void Compute(OpKernelContext* ctx) override { - const Tensor& x = ctx->input(0); - const Tensor& y = ctx->input(1); - OP_REQUIRES(ctx, TensorShapeUtils::IsVector(x.shape()), - errors::InvalidArgument("x must be a vector.")); - OP_REQUIRES(ctx, TensorShapeUtils::IsVector(y.shape()), - errors::InvalidArgument("y must be a vector.")); - bool match = true; - if (x.NumElements() != y.NumElements()) { - match = false; - } else { - auto Tx = x.flat(); - auto Ty = y.flat(); - for (int i = 0; i < x.NumElements(); ++i) { - if ((Tx(i) != kUnknown) && (Ty(i) != kUnknown) && (Tx(i) != Ty(i))) { - match = false; - } - } - } - OP_REQUIRES(ctx, match, - errors::InvalidArgument(msg_, " mismatch shape: x=[", - x.SummarizeValue(10), "] y=[", - y.SummarizeValue(10), "]")); - } - - private: - string msg_; -}; -REGISTER_KERNEL_BUILDER(Name("AssertShapeMatch").Device(DEVICE_CPU), - AssertShapeMatchOp); - -#if GOOGLE_CUDA -REGISTER_KERNEL_BUILDER( - Name("AssertShapeMatch").Device(DEVICE_GPU).HostMemory("x").HostMemory("y"), - AssertShapeMatchOp); -#endif - -class AssertSameDim0Op : public OpKernel { - public: - explicit AssertSameDim0Op(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, ctx->GetAttr("msg", &msg_)); - } - - ~AssertSameDim0Op() override {} - - void Compute(OpKernelContext* ctx) override { - if (ctx->num_inputs() == 0) { - // A no-op for empty list of inputs. - return; - } - const auto& x = ctx->input(0); - OP_REQUIRES(ctx, !TensorShapeUtils::IsScalar(x.shape()), - errors::InvalidArgument(msg_, " 0-th input is a scalar.")); - const auto dim0 = x.dim_size(0); - for (int i = 1; i < ctx->num_inputs(); ++i) { - const auto& y = ctx->input(i); - OP_REQUIRES( - ctx, !TensorShapeUtils::IsScalar(y.shape()), - errors::InvalidArgument(msg_, " ", i, "-th input is a scalar.")); - OP_REQUIRES(ctx, dim0 == y.dim_size(0), - errors::InvalidArgument( - msg_, " ", i, "-th input has a different dim0: ", dim0, - " ", y.dim_size(0))); - } - } - - private: - string msg_; -}; -REGISTER_KERNEL_BUILDER(Name("AssertSameDim0").Device(DEVICE_CPU), - AssertSameDim0Op); - -#if GOOGLE_CUDA -REGISTER_KERNEL_BUILDER(Name("AssertSameDim0").Device(DEVICE_GPU), - AssertSameDim0Op); -#endif - -} // namespace -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/assert_ops_test.py b/lingvo/core/ops/assert_ops_test.py deleted file mode 100644 index c8626f56..00000000 --- a/lingvo/core/ops/assert_ops_test.py +++ /dev/null @@ -1,66 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for assert_ops.""" - -from lingvo import compat as tf -from lingvo.core import ops -from lingvo.core import test_utils - -# Disable lint error: The name of ops.assert_shape_match triggers the -# lint which doesn't apply in this case. -# pylint:disable=g-error-prone-assert-raises - - -class AssertOpsTest(test_utils.TestCase): - - def testBasic(self): - with self.session(): - ops.assert_shape_match([10, 20, 30, 40], [-1, -1, -1, -1]).run() - ops.assert_shape_match([10, 20, 30, 40], [-1, 20, -1, -1]).run() - ops.assert_shape_match([10, 20, 30, 40], [-1, 20, -1, 40]).run() - with self.assertRaisesRegex(tf.errors.InvalidArgumentError, - "Yo mismatch"): - ops.assert_shape_match([10, 20, 30, 40], [10, 20, 40], "Yo").run() - with self.assertRaisesRegex(tf.errors.InvalidArgumentError, - "Foo mismatch"): - ops.assert_shape_match([10, 20, 30, 40], [8, 20, -1, 40], "Foo").run() - with self.assertRaisesRegex(tf.errors.InvalidArgumentError, "mismatch"): - ops.assert_shape_match([10, 20, 30, 40], [10, 20, 30, 44]).run() - - def testSameBatchSize(self): - - def t(*args): - return tf.zeros(shape=args, dtype=tf.float32) - - with self.session(): - ops.assert_same_dim0([t(2, 3, 4), t(2, 3, 4)]).run() - ops.assert_same_dim0([t(2, 3), t(2, 8)]).run() - ops.assert_same_dim0([t(2, 3)]).run() - ops.assert_same_dim0([t(2, 3)] * 100).run() - with self.assertRaisesRegex(tf.errors.InvalidArgumentError, "a scalar"): - ops.assert_same_dim0([t(), t(2, 3)]).run() - with self.assertRaisesRegex(tf.errors.InvalidArgumentError, "a scalar"): - ops.assert_same_dim0([t(2, 3), t()]).run() - with self.assertRaisesRegex(tf.errors.InvalidArgumentError, - "different dim0"): - ops.assert_same_dim0([t(2, 3), t(3, 2)]).run() - with self.assertRaisesRegex(tf.errors.InvalidArgumentError, - "different dim0"): - ops.assert_same_dim0([t(2, 3)] * 10 + [t(3, 2)]).run() - - -if __name__ == "__main__": - tf.test.main() diff --git a/lingvo/core/ops/beam_search_step_op_kernels.cc b/lingvo/core/ops/beam_search_step_op_kernels.cc deleted file mode 100644 index 11253859..00000000 --- a/lingvo/core/ops/beam_search_step_op_kernels.cc +++ /dev/null @@ -1,1071 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/beam_search_step_op_kernels.h" - -#include - -#include "lingvo/core/ops/hyps.pb.h" -#include "lingvo/core/ops/simple_vocab.h" -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" -#include "tensorflow/core/platform/env.h" -#include "tensorflow/core/util/work_sharder.h" - -namespace tensorflow { -namespace lingvo { - -namespace { -constexpr int kNumWorkers = 8; -} // namespace - -namespace debug { -static string IdsToStr(const google::protobuf::RepeatedField& ids) { - return debug::IdsToStr(std::vector(ids.begin(), ids.end())); -} -} // namespace debug - -bool IdsMatchUpToIndex(const std::vector& cur_hyp_ids, - const std::vector& other_hyp_ids, const int index) { - CHECK_LE(index, cur_hyp_ids.size()); - CHECK_LE(index, other_hyp_ids.size()); - return std::equal(cur_hyp_ids.begin(), cur_hyp_ids.begin() + index, - other_hyp_ids.begin()); -} - -bool IsDuplicateHyp(const Hyp& cur_hyp, const Hyp& other_hyp, - const int epsilon_id) { - const std::vector& cur_hyp_ids = cur_hyp.prev_ids; - const std::vector& other_hyp_ids = other_hyp.prev_ids; - // Note word_id refers to id of current label, which could be grapheme, - // phoneneme, wordpiece, ectc. - if (cur_hyp.word_id == other_hyp.word_id) { - // If the cur step is the same (epsilon or otherwise), just need to compare - // prev ids which already has epsilons stripped. - return ( - cur_hyp_ids.size() == other_hyp_ids.size() && - IdsMatchUpToIndex(cur_hyp_ids, other_hyp_ids, other_hyp_ids.size())); - } else if (cur_hyp.word_id == epsilon_id) { - // If exactly one of the hyps has a cur step of epsilon, then need to - // compare that hyp's final prev id to other hyp's current step id, - // then compare the rest of the prev ids. - return ( - cur_hyp_ids.size() == other_hyp_ids.size() + 1 && - cur_hyp_ids[cur_hyp_ids.size() - 1] == other_hyp.word_id && - IdsMatchUpToIndex(cur_hyp_ids, other_hyp_ids, other_hyp_ids.size())); - } else if (other_hyp.word_id == epsilon_id) { - return (other_hyp_ids.size() == cur_hyp_ids.size() + 1 && - other_hyp_ids[other_hyp_ids.size() - 1] == cur_hyp.word_id && - IdsMatchUpToIndex(other_hyp_ids, cur_hyp_ids, cur_hyp_ids.size())); - } else { - // If the cur step is not the same for the two hyps and neither is an - // epsilon then this cannot be a duplicate hyp. - return false; - } -} - -float LogSumExp(float a, float b) { - const float m = std::max(a, b); - return m + std::log(std::exp(a - m) + std::exp(b - m)); -} - -#ifdef __AVX__ -// AVX version all_less_than. -bool all_less_than(const float* p, float threshold) { - __m256 kth_logp = _mm256_set1_ps(threshold); - __m256 logp_vec = _mm256_loadu_ps(p); - // Skip this 8 elements if all of them are worst than 'kth_logp'. - // 'OQ' in '_CMP_LE_OQ' means comparison against NaN fails - // quietly (no crash). - __m256 mask = _mm256_cmp_ps(logp_vec, kth_logp, _CMP_LT_OQ); - return _mm256_movemask_ps(mask) == 0xFF; -} -#endif - -// Given the current partial hypothesis in 'hyps' for all beams in a batch and -// the predicted next step scores 'scores', return the best scored 'k+m' -// hypotheses where the first 'k' hypotheses are used for search in the next -// step while the remaining 'm' hypotheses are kept as alternatives to increase -// diversity. 'eos_id' is the end of beam id of the target language. -// -// eos_in_topk is filled with true/false to indicate whether or not the eos -// symbol is among the topk candidate for a hyp. -void ComputeTopKPlusM(const std::vector& hyps, const Tensor& scores, - const int32 k, const int32 m, const int32 eos_id, - const int32 eoc_id, const int32 num_beams, - const float valid_eos_max_logit_delta, - const float local_eos_threshold, bool is_first_step, - bool is_last_decoder_step, const Tensor& is_last_chunk, - bool merge_paths, bool allow_empty_terminated_hyp, - // Note that this is functionally a bool, however - // vector is not safe to parallel write into - // since it's underlying storage is at the byte-level. - std::vector* eos_in_topk, std::vector* top_k, - std::vector* extra_m, std::vector* eos_hyps, - std::vector* terminal_syms) { - VLOG(1) << "Topk clear, num_beams: " << num_beams; - CHECK_EQ(hyps.size(), num_beams * k); - CHECK_GE(m, 0); - CHECK(eos_in_topk && top_k && extra_m && eos_hyps && terminal_syms); - CHECK_EQ(hyps.size(), scores.dim_size(0)); - CHECK_LT(eos_id, scores.dim_size(1)); - int hyps_size = hyps.size(); - eos_in_topk->clear(); - top_k->clear(); - extra_m->clear(); - eos_in_topk->resize(hyps_size); - eos_hyps->resize(hyps_size); - terminal_syms->resize(hyps_size); - static thread::ThreadPool* workers = - new thread::ThreadPool(Env::Default(), "topk", kNumWorkers); - const int num_ids = scores.dim_size(1); - const auto scores_matrix = scores.matrix(); - const int epsilon_id_for_path_merging = merge_paths ? eoc_id : -1; - std::vector< - TopK> - merged_topk_vec(num_beams, TopK( - m + k, epsilon_id_for_path_merging)); - // Each mutex is used to protect corresponding merged_topk_vec. - std::vector mu_vec(num_beams); - // The thread sharding is along the hyps_size. - Shard( - kNumWorkers, workers, hyps_size, num_ids, [&](int64 start, int64 limit) { - for (int32 hyp_id = start; hyp_id < limit; ++hyp_id) { - if (is_first_step && hyp_id >= num_beams) { - // For first step, we only consider the first hyp of each beam, as - // otherwise we will be continuing k identical hyps along the way. - continue; - } - // +1 to make sure that at least top-k hypotheses survive even with - // the special treatment for eos. +2 if we are also using eoc. - const int topk_size = eoc_id >= 0 ? k + 2 : k + 1; - TopK - topk(topk_size, epsilon_id_for_path_merging, eos_id, - is_last_decoder_step); - float bottom_of_topk = -INFINITY; - int32 id = 0; - const float current_global_score = hyps[hyp_id].global_score; - // TODO(xbing): Try AVX512 if it is supported by machine. -#ifdef __AVX__ - const int STRIDE = - sizeof(__m256) / - sizeof(std::result_of::type); - // We read STRIDE float values at a single iteration and compare - // them with this k-th best value. STRIDE - 1 not to read outside - // the row. - for (; id + STRIDE - 1 < num_ids; id += STRIDE) { - if (!all_less_than(&scores_matrix(hyp_id, id), - bottom_of_topk - current_global_score)) { - for (int i = 0; i < STRIDE; ++i) { - const float score = scores_matrix(hyp_id, id + i); - const float global_score = - current_global_score + score; - if (global_score >= bottom_of_topk) { - bottom_of_topk = - topk.Add({hyps[hyp_id].beam_id, hyp_id, id + i, score, - global_score, hyps[hyp_id].prev_ids}); - } - } - } - } - // Non-AVX code below handles the remaining elements. -#endif - for (; id != num_ids; ++id) { - const float score = scores_matrix(hyp_id, id); - const float global_score = current_global_score + score; - if (global_score >= bottom_of_topk) { - bottom_of_topk = - topk.Add({hyps[hyp_id].beam_id, hyp_id, id, score, - global_score, hyps[hyp_id].prev_ids}); - } - } - - auto entries = topk.Get(); - CHECK(!entries.empty()) << "No entries in TopK. This typically " << - "happens if your model is producing NaNs in the output."; - std::sort(entries.begin(), entries.end(), HigherScore()); - const float eos_score_threshold = - entries[0].global_score - valid_eos_max_logit_delta; - VLOG(3) << "Best_score=" << entries[0].global_score - << " eos_score_threshold=" << eos_score_threshold; - { - const int beam_id = hyps[hyp_id].beam_id; - mutex_lock l(mu_vec[beam_id]); - for (const auto& e : entries) { - if (e.word_id == eos_id) { - VLOG(3) << "EOS hyp score=" << e.global_score - << " toks=" << debug::IdsToStr(e.prev_ids); - // We move terminated hyps off of the beam. - if (is_last_decoder_step || - (e.global_score > eos_score_threshold && - e.local_score > local_eos_threshold)) { - (*eos_in_topk)[hyp_id] = true; - (*eos_hyps)[hyp_id] = e; - (*terminal_syms)[hyp_id] = eos_id; - } - } else if (eoc_id >= 0 && is_last_chunk.vec()(hyp_id) && - e.word_id == eoc_id) { - VLOG(3) << "last chunk hyp score=" << e.global_score - << " toks=" << debug::IdsToStr(e.prev_ids); - // At the last chunk and output . We terminate the - // hypothesis, even though was not predicted, and - // indicate that the final symbol for the hypothesis is - // , not . - if (e.global_score > eos_score_threshold && - // Only allow an empty hyp (all s) to be - // considered terminated, if explicitly permitted. - // 'prev_ids' contains only non-epsilons. - (allow_empty_terminated_hyp || !e.prev_ids.empty())) { - (*eos_in_topk)[hyp_id] = true; - (*eos_hyps)[hyp_id] = e; - (*terminal_syms)[hyp_id] = eoc_id; - } - } else { - merged_topk_vec[beam_id].Add(e); - } - } - } - } - }); - - const int hyps_per_beam = k; - top_k->resize(hyps_per_beam * num_beams); - for (int i = 0; i < num_beams; ++i) { - auto ith_topk = merged_topk_vec[i].Get(); - std::sort(ith_topk.begin(), ith_topk.end(), HigherScore()); - const int num_hyps = - std::min(static_cast(ith_topk.size()), hyps_per_beam); - for (int j = 0; j < num_hyps; ++j) { - (*top_k)[j * num_beams + i] = ith_topk[j]; - } - for (int j = hyps_per_beam; j < ith_topk.size(); ++j) { - extra_m->push_back(ith_topk[j]); - } - } - VLOG(1) << "Topk done"; -} - -class BeamSearchStepOp : public OpKernel { - public: - explicit BeamSearchStepOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, ctx->GetAttr("eos_id", &eos_id_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("eoc_id", &eoc_id_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("beam_size", &beam_size_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("num_hyps_per_beam", &num_hyps_per_beam_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("valid_eos_max_logit_delta", - &valid_eos_max_logit_delta_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("local_eos_threshold", - &local_eos_threshold_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("merge_paths", &merge_paths_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("allow_empty_terminated_hyp", - &allow_empty_terminated_hyp_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("ensure_full_beam", &ensure_full_beam_)); - OP_REQUIRES_OK( - ctx, ctx->GetAttr("force_eos_in_last_step", &force_eos_in_last_step_)); - - CHECK_GE(eos_id_, 0); - CHECK_GT(beam_size_, 0.0); - CHECK_GT(num_hyps_per_beam_, 0); - } - - private: - Tensor* ForwardOrCopyInputToOutput(OpKernelContext* ctx, int input_idx, - int output_idx) { - Tensor* output = nullptr; - const Tensor& input = ctx->input(input_idx); - CHECK(ctx->forward_input_or_allocate_output({input_idx}, output_idx, - input.shape(), &output) - .ok()); - if (!output->SharesBufferWith(input)) { - // Copy the input data if we were unable to forward the underlying buffer. - if (DataTypeCanUseMemcpy(input.dtype())) { - if (input.NumElements() > 0) { - StringPiece input_data = input.tensor_data(); - StringPiece output_data = output->tensor_data(); - memcpy(const_cast(output_data.data()), input_data.data(), - input_data.size()); - } - } else if (input.dtype() == DT_STRING) { - output->flat() = input.flat(); - } - } - return output; - } - - string AssembleDoneHyp(const Hyp& hyp, const int32 terminal_sym, - const TTypes::Matrix& t_out_prev_hyps, - const TTypes::Matrix& t_out_hyps, - const TTypes::Matrix& t_out_scores, - const TTypes::Tensor t_out_atten_probs, - const Tensor& atten_probs, int t) const { - std::vector hyp_ids(t); - int hyp_id = hyp.hyp_id; - for (int i = t - 1; i >= 0; --i) { - hyp_ids[i] = hyp_id; - hyp_id = t_out_prev_hyps(i, hyp_id); - } - Hypothesis hypothesis; - hypothesis.set_beam_id(hyp.beam_id); - // Add one to account for t-th step (terminal sym). - const float average_step_score = hyp.global_score / (t + 1); - for (int i = 0; i < t; i++) { - const int hyp_id = hyp_ids[i]; - hypothesis.add_ids(t_out_hyps(i, hyp_id)); - // If this is a model with epsilons (RNN-T or NT), then this hyp - // may represent many possible paths that have been merged. The - // recorded per-step scores only are valid for one of these paths, so - // they are not meaningful for the merged path. For the merged path - // we simpy take an average per-step score. - const float score_this_step = - (merge_paths_ ? average_step_score : t_out_scores(i, hyp_id)); - hypothesis.add_scores(score_this_step); - auto* att_vec = hypothesis.add_atten_vecs(); - for (int j = 0; j < atten_probs.dim_size(1); ++j) { - att_vec->add_prob(t_out_atten_probs(i, hyp_id, j)); - } - } - // Now add the terminal symbol. - hypothesis.add_ids(terminal_sym); - // As above, use the average per-step score for RNN-T and NT. - const float score_this_step = - merge_paths_ ? average_step_score : hyp.local_score; - hypothesis.add_scores(score_this_step); - auto* att_vec = hypothesis.add_atten_vecs(); - auto t_atten_probs = atten_probs.matrix(); - for (int j = 0; j < atten_probs.dim_size(1); ++j) { - att_vec->add_prob(t_atten_probs(hyp.hyp_id, j)); - } - return hypothesis.SerializeAsString(); - } - - public: - void Compute(OpKernelContext* ctx) override { - const Tensor& scores = ctx->input(0); - const Tensor& atten_probs = ctx->input(1); - const Tensor& best_scores = ctx->input(2); - const Tensor& cumulative_scores = ctx->input(3); - const Tensor& in_scores = ctx->input(4); - const Tensor& in_hyps = ctx->input(5); - const Tensor& in_prev_hyps = ctx->input(6); - const Tensor& in_done_hyps = ctx->input(7); - const Tensor& in_atten_probs = ctx->input(8); - const Tensor& is_last_chunk = ctx->input(9); - const Tensor& cur_step = ctx->input(10); - - OP_REQUIRES( - ctx, scores.dims() == 2, - errors::InvalidArgument( - "Failed tensor shape sanity check. scores.dims() == 2. Got ", - scores.dims())); - OP_REQUIRES( - ctx, atten_probs.dims() == 2, - errors::InvalidArgument( - "Failed tensor shape sanity check. atten_probs.dims() == 2. Got ", - atten_probs.dims())); - OP_REQUIRES( - ctx, best_scores.dims() == 1, - errors::InvalidArgument( - "Failed tensor shape sanity check. best_scores.dims() == 1. Got ", - best_scores.dims())); - OP_REQUIRES(ctx, cumulative_scores.dims() == 1, - errors::InvalidArgument("Failed tensor shape sanity check. " - "cumulative_scores.dims() == 1. Got ", - cumulative_scores.dims())); - OP_REQUIRES( - ctx, in_scores.dims() == 2, - errors::InvalidArgument( - "Failed tensor shape sanity check. in_scores.dims() == 2. Got ", - in_scores.dims())); - OP_REQUIRES(ctx, in_hyps.dims() == 2, - errors::InvalidArgument("Failed tensor shape sanity check. " - "in_hyps.dims() == 2. Got ", - in_hyps.dims())); - - OP_REQUIRES( - ctx, in_prev_hyps.dims() == 2, - errors::InvalidArgument( - "Failed tensor shape sanity check. in_prev_hyps.dims() == 2. Got ", - in_prev_hyps.dims())); - OP_REQUIRES( - ctx, in_done_hyps.dims() == 2, - errors::InvalidArgument( - "Failed tensor shape sanity check. in_done_hyps.dims() == 2. Got ", - in_done_hyps.dims())); - OP_REQUIRES(ctx, in_atten_probs.dims() == 3, - errors::InvalidArgument("Failed tensor shape sanity check. " - "in_atten_probs.dims() == 3. Got ", - in_atten_probs.dims())); - OP_REQUIRES( - ctx, cur_step.dims() == 0, - errors::InvalidArgument( - "Failed tensor shape sanity check. cur_step.dims() == 0. Got ", - cur_step.dims())); - OP_REQUIRES(ctx, scores.dim_size(0) == atten_probs.dim_size(0), - errors::InvalidArgument("Failed tensor shape sanity check. " - "scores.dim_size(0) == " - "atten_probs.dim_size(0). Got ", - scores.dim_size(0), " and ", - atten_probs.dim_size(0))); - OP_REQUIRES(ctx, scores.dim_size(0) == cumulative_scores.dim_size(0), - errors::InvalidArgument("Failed tensor shape sanity check. " - "scores.dim_size(0) == " - "cumulative_scores.dim_size(0). Got ", - scores.dim_size(0), " and ", - cumulative_scores.dim_size(0))); - OP_REQUIRES(ctx, scores.dim_size(0) % best_scores.dim_size(0) == 0, - errors::InvalidArgument("Failed tensor shape sanity check. " - "scores.dim_size(0) % " - "best_scores.dim_size(0) == 0. Got ", - scores.dim_size(0), " and ", - best_scores.dim_size(0))); - OP_REQUIRES( - ctx, scores.dim_size(0) / best_scores.dim_size(0) == num_hyps_per_beam_, - errors::InvalidArgument( - "Failed tensor shape sanity check. " - "scores.dim_size(0) / best_scores.dim_size(0) " - "== num_hyps_per_beam_. Got ", - scores.dim_size(0), " and ", best_scores.dim_size(0), - " where num_hyps_per_beam_ = ", num_hyps_per_beam_)); - OP_REQUIRES(ctx, scores.dim_size(0) == in_hyps.dim_size(1), - errors::InvalidArgument( - "Failed tensor shape sanity check. " - "scores.dim_size(0) == in_hyps.dim_size(1). Got ", - scores.dim_size(0), " and ", in_hyps.dim_size(1))); - OP_REQUIRES(ctx, in_hyps.dim_size(0) == in_scores.dim_size(0), - errors::InvalidArgument("Failed tensor shape sanity check. " - "in_hyps.dim_size(0) == " - "in_scores.dim_size(0). Got ", - in_hyps.dim_size(0), " and ", - in_scores.dim_size(0))); - OP_REQUIRES(ctx, in_hyps.dim_size(0) == in_prev_hyps.dim_size(0), - errors::InvalidArgument("Failed tensor shape sanity check. " - "in_hyps.dim_size(0) == " - "in_prev_hyps.dim_size(0). Got ", - in_hyps.dim_size(0), " and ", - in_prev_hyps.dim_size(0))); - OP_REQUIRES(ctx, in_hyps.dim_size(0) == in_done_hyps.dim_size(0), - errors::InvalidArgument("Failed tensor shape sanity check. " - "in_hyps.dim_size(0) == " - "in_done_hyps.dim_size(0). Got ", - in_hyps.dim_size(0), " and ", - in_done_hyps.dim_size(0))); - OP_REQUIRES(ctx, in_hyps.dim_size(0) == in_atten_probs.dim_size(0), - errors::InvalidArgument("Failed tensor shape sanity check. " - "in_hyps.dim_size(0) == " - "in_atten_probs.dim_size(0). Got ", - in_hyps.dim_size(0), " and ", - in_atten_probs.dim_size(0))); - OP_REQUIRES(ctx, in_hyps.dim_size(1) == in_scores.dim_size(1), - errors::InvalidArgument("Failed tensor shape sanity check. " - "in_hyps.dim_size(1) == " - "in_scores.dim_size(1). Got ", - in_hyps.dim_size(1), " and ", - in_scores.dim_size(1))); - OP_REQUIRES(ctx, in_hyps.dim_size(1) == in_prev_hyps.dim_size(1), - errors::InvalidArgument("Failed tensor shape sanity check. " - "in_hyps.dim_size(1) == " - "in_prev_hyps.dim_size(1). Got ", - in_hyps.dim_size(1), " and ", - in_prev_hyps.dim_size(1))); - OP_REQUIRES(ctx, in_hyps.dim_size(1) == in_done_hyps.dim_size(1), - errors::InvalidArgument("Failed tensor shape sanity check. " - "in_hyps.dim_size(1) == " - "in_done_hyps.dim_size(1). Got ", - in_hyps.dim_size(1), " and ", - in_done_hyps.dim_size(1))); - OP_REQUIRES(ctx, in_hyps.dim_size(1) == in_atten_probs.dim_size(1), - errors::InvalidArgument("Failed tensor shape sanity check. " - "in_hyps.dim_size(1) == " - "in_atten_probs.dim_size(1). Got ", - in_hyps.dim_size(1), " and ", - in_atten_probs.dim_size(1))); - OP_REQUIRES( - ctx, atten_probs.dim_size(1) == in_atten_probs.dim_size(2), - errors::InvalidArgument( - "Failed tensor shape sanity check. " - "atten_probs.dim_size(1) == in_atten_probs.dim_size(2). Got ", - atten_probs.dim_size(1), " and ", in_atten_probs.dim_size(2))); - - if (merge_paths_) { - OP_REQUIRES( - ctx, eoc_id_ >= 0, - errors::InvalidArgument( - "Paths can only be merged for an epsilon-emitting model (RNN-T " - "or NT). Epsilon id must be non-negative, but got: ", - eoc_id_)); - } - int num_beams = best_scores.dim_size(0); - int num_hyps = cumulative_scores.dim_size(0); - CHECK_EQ(num_hyps_per_beam_, num_hyps / num_beams); - const int t = cur_step.scalar()(); - CHECK_LT(t, in_hyps.dim_size(0)); - - VLOG(2) << "BeamSearchStepOp(" << num_hyps_per_beam_ << ") step=" << t; - auto t_cumulative_scores = cumulative_scores.vec(); - std::vector hyps(num_hyps); - for (int i = 0; i < num_hyps; ++i) { - hyps[i].beam_id = i % num_beams; - hyps[i].hyp_id = i; - hyps[i].global_score = t_cumulative_scores(i); - // Determines the sequence of prev ids that this hypothesis represents. - std::vector hyp_id_at_step(t); - int hyp_id = i; - for (int j = t - 1; j >= 0; --j) { - hyp_id_at_step[j] = hyp_id; - hyp_id = in_prev_hyps.matrix()(j, hyp_id); - } - for (int j = 0; j < t; ++j) { - const int prev_id = in_hyps.matrix()(j, hyp_id_at_step[j]); - if (prev_id != eoc_id_) { - hyps[i].prev_ids.push_back(prev_id); - } - } - VLOG(3) << "Step " << t << " hyp " << i - << " score=" << hyps[i].global_score - << " toks=" << debug::IdsToStr(hyps[i].prev_ids); - } - std::vector top_k_hyps; - std::vector extra_m_hyps; - std::vector eos_hyps; - std::vector eos_in_topk; - std::vector terminal_syms; - const bool is_last_decoder_step = - (t == (in_hyps.dim_size(0) - 1)) && force_eos_in_last_step_; - ComputeTopKPlusM(hyps, scores, /*k=*/num_hyps_per_beam_, /*m=*/0, - /*eos_id=*/eos_id_, /*eoc_id=*/eoc_id_, num_beams, - valid_eos_max_logit_delta_, local_eos_threshold_, - /*is_first_step=*/t == 0, is_last_decoder_step, - is_last_chunk, merge_paths_, allow_empty_terminated_hyp_, - &eos_in_topk, &top_k_hyps, &extra_m_hyps, &eos_hyps, - &terminal_syms); - - Tensor* out_best_scores = NULL; - Tensor* out_cumulative_scores = NULL; - OP_REQUIRES_OK( - ctx, ctx->allocate_output(0, best_scores.shape(), &out_best_scores)); - OP_REQUIRES_OK(ctx, ctx->allocate_output(1, cumulative_scores.shape(), - &out_cumulative_scores)); - Tensor* out_scores = ForwardOrCopyInputToOutput(ctx, 4, 2); - Tensor* out_hyps = ForwardOrCopyInputToOutput(ctx, 5, 3); - Tensor* out_prev_hyps = ForwardOrCopyInputToOutput(ctx, 6, 4); - Tensor* out_done_hyps = ForwardOrCopyInputToOutput(ctx, 7, 5); - Tensor* out_atten_probs = ForwardOrCopyInputToOutput(ctx, 8, 6); - Tensor* all_done; - OP_REQUIRES_OK(ctx, ctx->allocate_output(7, TensorShape({}), &all_done)); - - auto t_out_best_scores = out_best_scores->vec(); - auto t_out_cumulative_scores = out_cumulative_scores->vec(); - auto t_out_scores = out_scores->matrix(); - auto t_out_hyps = out_hyps->matrix(); - auto t_out_prev_hyps = out_prev_hyps->matrix(); - auto t_out_done_hyps = out_done_hyps->matrix(); - auto t_out_atten_probs = out_atten_probs->tensor(); - auto t_all_done = all_done->scalar(); - - // To initialize the two vectors. - t_out_best_scores = best_scores.vec(); - t_out_cumulative_scores = cumulative_scores.vec(); - - // Fill in all the output tensors. - for (int i = 0; i < num_hyps; ++i) { - const Hyp& hyp = top_k_hyps[i]; - t_out_scores(t, i) = hyp.local_score; - t_out_cumulative_scores(i) = hyp.global_score; - t_out_hyps(t, i) = hyp.word_id; - t_out_prev_hyps(t, i) = hyp.hyp_id; - t_out_atten_probs.chip(t, 0).chip(i, 0) = - atten_probs.matrix().chip(hyp.hyp_id, 0); - if (eos_in_topk[i]) { - // We have a good terminated hyp. - const int beam_id = eos_hyps[i].beam_id; - CHECK_EQ(beam_id, i % num_beams); - VLOG(2) << "Top EOS hyp @step " << t - << " score=" << eos_hyps[i].global_score - << " toks=" << debug::IdsToStr(eos_hyps[i].prev_ids); - // Update the best scores. - if (eos_hyps[i].global_score > t_out_best_scores(beam_id)) { - t_out_best_scores(beam_id) = eos_hyps[i].global_score; - } - string done_hyp = AssembleDoneHyp( - eos_hyps[i], terminal_syms[i], t_out_prev_hyps, t_out_hyps, - t_out_scores, t_out_atten_probs, atten_probs, t); - t_out_done_hyps(t, i) = done_hyp; - } - } - - // Now check for all_done - t_all_done() = true; - if (ensure_full_beam_) { - // First check how many EOS hyps we have. If we have fewer than - // num_hyps_per_beam for any beam, we are NOT done. - for (int beam_id = 0; beam_id < num_beams; ++beam_id) { - int num_done_hyps = 0; - for (int index_in_beam = 0; index_in_beam < num_hyps_per_beam_; - ++index_in_beam) { - for (int time_step = 0; time_step < t; ++time_step) { - int index = beam_id * num_hyps_per_beam_ + index_in_beam;; - if (!t_out_done_hyps(time_step, index).empty()) { - ++num_done_hyps; - } - } - } - if (num_done_hyps < num_hyps_per_beam_) { - t_all_done() = false; - break; - } - } - if (t_all_done() == false) return; - } - // Now check for hyp quality. If for any beam we still have hyps within - // 'beam_size' of best score, we are NOT done. - for (int i = 0; i < num_hyps; ++i) { - const Hyp& hyp = top_k_hyps[i]; - const int beam_id = hyp.beam_id; - CHECK_EQ(beam_id, i % num_beams); - VLOG(3) << "Hyp score=" << hyp.global_score - << " beam best=" << t_out_best_scores(beam_id) - << " beam size=" << beam_size_; - if (hyp.global_score > t_out_best_scores(beam_id) - beam_size_) { - t_all_done() = false; - break; - } - } - } - - private: - int eos_id_ = 0; - int eoc_id_ = -1; - float beam_size_ = 0.0; - int num_hyps_per_beam_ = 0; - float valid_eos_max_logit_delta_ = 0.0; - float local_eos_threshold_ = 0.0; - bool merge_paths_ = false; - bool allow_empty_terminated_hyp_ = true; - bool ensure_full_beam_ = false; - bool force_eos_in_last_step_ = false; -}; - -REGISTER_KERNEL_BUILDER(Name("BeamSearchStep").Device(DEVICE_CPU), - BeamSearchStepOp); - -class TopKTerminatedHypsOp : public OpKernel { - public: - explicit TopKTerminatedHypsOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, ctx->GetAttr("k", &k_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("num_hyps_per_beam", &num_hyps_per_beam_)); - OP_REQUIRES_OK( - ctx, ctx->GetAttr("length_normalization", &length_normalization_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("coverage_penalty", &coverage_penalty_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("target_seq_length_ratio", - &target_seq_length_ratio_)); - // TODO(anjuli): Remove eoc_id_ which is no longer used. - OP_REQUIRES_OK(ctx, ctx->GetAttr("eoc_id", &eoc_id_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("merge_paths", &merge_paths_)); - CHECK_GE(length_normalization_, 0.0); - CHECK_GE(target_seq_length_ratio_, 0.0); - CHECK_GE(coverage_penalty_, 0); - CHECK_GT(num_hyps_per_beam_, 0); - CHECK_GT(k_, 0); - } - - void ComputeTopK(const Tensor& in_done_hyps, - const std::vector src_seq_lengths, const int32 k, - const int32 num_beams, Tensor* topk_hyps) { - VLOG(1) << "Topk clear, num_beams: " << num_beams; - int hyps_size = in_done_hyps.dim_size(1); - int num_steps = in_done_hyps.dim_size(0); - static thread::ThreadPool* workers = new thread::ThreadPool( - Env::Default(), "topk_terminated_hyps", kNumWorkers); - // No Insert struct is provided, so we use DefaultInsert, which inserts - // without deduping. No deduping is necessary here because we dedupe - // partial hyps at each step of beam search. - std::vector> - topk_vec(num_beams, - TopK( - k, /* unused epsilon id */ -1)); - // Each mutex is used to protect corresponding topk_vec. - std::vector mu_vec(num_beams); - auto t_done_hyps = in_done_hyps.matrix(); - // The thread sharding is along hyps_size. - Shard(kNumWorkers, workers, hyps_size, 1000 * num_steps, - [&](int64 start, int64 limit) { - Hypothesis hypothesis; - for (int32 hyp_id = start; hyp_id < limit; ++hyp_id) { - // The topk for this beam. - TopK* - topk = &topk_vec[hyp_id % num_beams]; - for (int32 step_id = 0; step_id < num_steps; ++step_id) { - const string& str_hyps = t_done_hyps(step_id, hyp_id); - if (!str_hyps.empty()) { - hypothesis.ParseFromString(str_hyps); - if (hypothesis.has_beam_id()) { - // This hypothesis is a real terminated hyps. - int src_size = src_seq_lengths[hypothesis.beam_id()]; - float normalized_score = - NormalizedScore(hypothesis, src_size); - hypothesis.set_normalized_score(normalized_score); - VLOG(2) << "Add to terminated top-k " - << " score=" << hypothesis.normalized_score() - << " toks=" << debug::IdsToStr(hypothesis.ids()); - // TODO(xbing): avoid acquiring a mutex for each record. - mutex_lock l(mu_vec[hyp_id % num_beams]); - topk->Add(hypothesis); - } - } - } - } - }); - - auto t_topk_hyps = topk_hyps->matrix(); - for (int i = 0; i < num_beams; ++i) { - auto ith_topk = topk_vec[i].Get(); - CHECK_LE(ith_topk.size(), k); - std::sort(ith_topk.begin(), ith_topk.end(), BetterTerminatedHyp()); - for (int j = 0; j < ith_topk.size(); ++j) { - t_topk_hyps(i, j) = ith_topk[j].SerializeAsString(); - VLOG(2) << "TopK(" << i << ", " << j << ") = " - << debug::IdsToStr(ith_topk[j].ids()); - } - } - } - - float NormalizedScore(const Hypothesis& hypothesis, - const int src_size) const { - int length = hypothesis.atten_vecs_size(); - Tensor cumulative_atten_prob(DT_FLOAT, {src_size}); - auto cumulative_atten_prob_vec = cumulative_atten_prob.vec(); - cumulative_atten_prob_vec.setZero(); - for (int step = 0; step < hypothesis.atten_vecs_size(); ++step) { - const int hyp_prob_size = hypothesis.atten_vecs(step).prob_size(); - for (int src_id = 0; src_id < src_size; ++src_id) { - if (src_id < hyp_prob_size) { - cumulative_atten_prob_vec(src_id) += - hypothesis.atten_vecs(step).prob(src_id); - } else { - // This can happen e.g. for RNNT model. Here we simply assume - // atten_prob for those source positions are 0.0 - VLOG(5) << "Missing atten_prob for source position " << src_id - << ". Total available positions are " << hyp_prob_size << "."; - cumulative_atten_prob_vec(src_id) += 0.0; - } - } - } - // Coverage is capped at 0.5 so that so long as a word is - // reasonably covered, it is not penalized anymore. - Tensor penalty(DT_FLOAT, {}); - penalty.scalar() = - (cumulative_atten_prob_vec / target_seq_length_ratio_) - .cwiseMax(0.001f) - .cwiseMin(0.5f) - .log() - .sum().eval(); - const float coverage_penalty = penalty.scalar()(); - const float length_norm = std::pow(length + 5.0, length_normalization_) / - std::pow(5.0, length_normalization_); - - float global_score = 0.0; - for (const auto& score : hypothesis.scores()) { - global_score += score; - } - const float normalized_score = - global_score / length_norm + - (target_seq_length_ratio_ * coverage_penalty_ * coverage_penalty); - return normalized_score; - } - - void Compute(OpKernelContext* ctx) override { - const Tensor& in_done_hyps = ctx->input(0); - // Some sanity check on the tensor shapes. - OP_REQUIRES(ctx, in_done_hyps.dim_size(1) % num_hyps_per_beam_ == 0, - errors::InvalidArgument("Failed tensor shape sanity check.")); - - int num_beams = in_done_hyps.dim_size(1) / num_hyps_per_beam_; - - const Tensor& in_src_seq_lens = ctx->input(1); - OP_REQUIRES( - ctx, in_src_seq_lens.dim_size(0) == num_beams, - errors::InvalidArgument( - "src_seq_lengths should be a 1-d Tensor of length num_beams. Got ", - in_src_seq_lens.dim_size(0), " vs ", num_beams)); - std::vector src_seq_lengths(num_beams); - for (int i = 0; i < num_beams; ++i) { - src_seq_lengths[i] = in_src_seq_lens.flat()(i); - } - // Set the output tensors. - Tensor* out_topk_hyps = nullptr; - OP_REQUIRES_OK(ctx, ctx->allocate_output(0, TensorShape{num_beams, k_}, - &out_topk_hyps)); - ComputeTopK(in_done_hyps, src_seq_lengths, k_, num_beams, out_topk_hyps); - VLOG(1) << "TopKTerminatedHypsOp(" << num_hyps_per_beam_ << ") done"; - } - - private: - int32 num_hyps_per_beam_; - float length_normalization_; - float coverage_penalty_; - float target_seq_length_ratio_; - int32 k_; - int32 eoc_id_; - bool merge_paths_ = false; -}; - -REGISTER_KERNEL_BUILDER(Name("TopKTerminatedHyps").Device(DEVICE_CPU), - TopKTerminatedHypsOp); - -class UnpackHypOp : public OpKernel { - public: - explicit UnpackHypOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, ctx->GetAttr("max_seq_length", &max_seq_length_)); - CHECK_GE(max_seq_length_, 0); - } - - void Compute(OpKernelContext* ctx) override { - const Tensor& in_hyps = ctx->input(0); - const auto& t_in_hyps = in_hyps.flat(); - const int batch_size = t_in_hyps.size(); - std::vector hyps(batch_size); - for (int i = 0; i < batch_size; ++i) { - // TODO(yonghui): parallelize this loop. - const tstring& t_in_hyps_i = t_in_hyps(i); - if (!t_in_hyps(i).empty()) { - hyps[i].ParseFromArray(t_in_hyps_i.data(), t_in_hyps_i.size()); - } - } - int max_seq_length = max_seq_length_; - if (max_seq_length <= 0) { - // Derive max_seq_length from input hyps. - for (int i = 0; i < batch_size; ++i) { - if (hyps[i].ids_size() > max_seq_length) { - max_seq_length = hyps[i].ids_size(); - } - } - } - Tensor* out_ids; - OP_REQUIRES_OK( - ctx, ctx->allocate_output(0, TensorShape({batch_size, max_seq_length}), - &out_ids)); - Tensor* out_seq_lens; - OP_REQUIRES_OK( - ctx, ctx->allocate_output(1, TensorShape({batch_size}), &out_seq_lens)); - Tensor* out_scores; - OP_REQUIRES_OK( - ctx, ctx->allocate_output(2, TensorShape({batch_size}), &out_scores)); - auto t_out_ids = out_ids->matrix(); - auto t_out_seq_lens = out_seq_lens->vec(); - auto t_out_scores = out_scores->vec(); - t_out_ids.setZero(); - t_out_seq_lens.setZero(); - t_out_scores.setZero(); - for (int i = 0; i < batch_size; ++i) { - const Hypothesis& hyp = hyps[i]; - // TODO(yonghui): parallelize this loop. - if (hyp.ids_size() > 0) { - for (int j = 0; j < hyp.ids_size() && j < max_seq_length; ++j) { - t_out_ids(i, j) = hyp.ids(j); - } - t_out_seq_lens(i) = std::min(hyp.ids_size(), max_seq_length); - t_out_scores(i) = hyp.normalized_score(); - } - } - } - - private: - int32 max_seq_length_ = 0; -}; - -REGISTER_KERNEL_BUILDER(Name("UnpackHyp").Device(DEVICE_CPU), UnpackHypOp); - -template -class HypsFromBeamSearchOuts : public OpKernel { - public: - explicit HypsFromBeamSearchOuts(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, ctx->GetAttr("eos_id", &eos_id_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("num_hyps_per_beam", &num_hyps_per_beam_)); - } - - void Compute(OpKernelContext* ctx) override { - const Tensor& hyps = ctx->input(0); - const Tensor& prev_hyps = ctx->input(1); - const Tensor& done_hyps = ctx->input(2); - const Tensor& scores = ctx->input(3); - const Tensor& atten_probs = ctx->input(4); - const Tensor& eos_scores = ctx->input(5); - const Tensor& eos_atten_probs = ctx->input(6); - - OP_REQUIRES(ctx, hyps.dims() == 2, - errors::InvalidArgument( - "Failed tensor shape sanity check. hyps.dims() == 2. Got ", - hyps.dims())); - OP_REQUIRES( - ctx, prev_hyps.dims() == 2, - errors::InvalidArgument( - "Failed tensor shape sanity check. prev_hyps.dims() == 2. Got ", - prev_hyps.dims())); - OP_REQUIRES( - ctx, done_hyps.dims() == 2, - errors::InvalidArgument( - "Failed tensor shape sanity check. done_hyps.dims() == 2. Got ", - done_hyps.dims())); - OP_REQUIRES( - ctx, scores.dims() == 2, - errors::InvalidArgument( - "Failed tensor shape sanity check. scores.dims() == 2. Got ", - scores.dims())); - OP_REQUIRES( - ctx, atten_probs.dims() == 3, - errors::InvalidArgument( - "Failed tensor shape sanity check. atten_probs.dims() == 2. Got ", - atten_probs.dims())); - - OP_REQUIRES(ctx, atten_probs.dim_size(1) == scores.dim_size(1), - errors::InvalidArgument( - "Failed tensor shape sanity check. " - "atten_probs.dim_size(1) == scores.dim_size(1). Got ", - atten_probs.dim_size(1), " and ", scores.dim_size(1))); - - OP_REQUIRES(ctx, hyps.IsSameSize(prev_hyps), - errors::InvalidArgument( - "Failed tensor shape sanity check. " - "hyps and prev_hyps should have the same shape. Got ", - hyps.shape().DebugString(), " and ", - prev_hyps.shape().DebugString())); - - OP_REQUIRES(ctx, hyps.IsSameSize(done_hyps), - errors::InvalidArgument( - "Failed tensor shape sanity check. " - "hyps and done_hyps should have the same shape. Got ", - hyps.shape().DebugString(), " and ", - done_hyps.shape().DebugString())); - - OP_REQUIRES( - ctx, hyps.IsSameSize(scores), - errors::InvalidArgument( - "Failed tensor shape sanity check. " - "hyps and scores should have the same shape. Got ", - hyps.shape().DebugString(), " and ", scores.shape().DebugString())); - - OP_REQUIRES(ctx, hyps.IsSameSize(done_hyps), - errors::InvalidArgument( - "Failed tensor shape sanity check. " - "hyps and done_hyps should have the same shape. Got ", - hyps.shape().DebugString(), " and ", - done_hyps.shape().DebugString())); - - OP_REQUIRES(ctx, hyps.IsSameSize(eos_scores), - errors::InvalidArgument( - "Failed tensor shape sanity check. " - "hyps and eos_scores should have the same shape. Got ", - hyps.shape().DebugString(), " and ", - eos_scores.shape().DebugString())); - OP_REQUIRES( - ctx, atten_probs.IsSameSize(eos_atten_probs), - errors::InvalidArgument( - "Failed tensor shape sanity check. " - "atten_probs and eos_atten_probs should have the same shape. Got ", - atten_probs.shape().DebugString(), " and ", - eos_atten_probs.shape().DebugString())); - - auto t_hyps = hyps.matrix(); - auto t_prev_hyps = prev_hyps.matrix(); - auto t_done_hyps = done_hyps.matrix(); - auto t_scores = scores.matrix(); - auto t_atten_probs = atten_probs.tensor(); - auto t_eos_scores = eos_scores.matrix(); - auto t_eos_atten_probs = eos_atten_probs.tensor(); - const int seq_length = hyps.dim_size(0); - const int num_hyps = hyps.dim_size(1); - - Tensor* out_hyps; - OP_REQUIRES_OK(ctx, ctx->allocate_output(0, hyps.shape(), &out_hyps)); - auto out_hyps_t = out_hyps->matrix(); - - // Use the same thread pool as topk operator. - static thread::ThreadPool* workers = - new thread::ThreadPool(Env::Default(), "topk", kNumWorkers); - - Shard( - kNumWorkers, workers, num_hyps, seq_length * seq_length, - [&](int64 start, int64 end) { - std::vector hyp_token_ids; - std::vector hyp_local_scores; - std::vector hyp_ids; - Hypothesis terminated_hyps; - const int num_beams = (num_hyps / num_hyps_per_beam_); - for (int i = 0; i < seq_length; ++i) { - for (int j = start; j < end; ++j) { - // If the hyp is terminated, then assemble the output proto. - if (t_done_hyps(i, j)) { - // Reuse the buffers. - hyp_token_ids.clear(); - hyp_local_scores.clear(); - hyp_ids.clear(); - terminated_hyps.Clear(); - - // Walk through the token id matrix, and assemble id, score, and - // prev_hyp_id for each step. - hyp_token_ids.push_back(eos_id_); - hyp_local_scores.push_back(t_eos_scores(i, j)); - int prev_hyp_id = j; - hyp_ids.push_back(prev_hyp_id); - for (int k = i - 1; k >= 0; --k) { - hyp_token_ids.push_back(t_hyps(k, prev_hyp_id)); - hyp_local_scores.push_back(t_scores(k, prev_hyp_id)); - prev_hyp_id = t_prev_hyps(k, prev_hyp_id); - hyp_ids.push_back(prev_hyp_id); - } - - // Assemble terminated hyp. - terminated_hyps.set_beam_id(j % num_beams); - for (int l = hyp_local_scores.size() - 1; l >= 0; --l) { - terminated_hyps.add_scores(float(hyp_local_scores[l])); - terminated_hyps.add_ids(hyp_token_ids[l]); - const int cur_step = hyp_local_scores.size() - 1 - l; - auto* att_vec = terminated_hyps.add_atten_vecs(); - for (int d = 0; d < atten_probs.dim_size(2); ++d) { - if (l == 0) { - att_vec->add_prob( - float(t_eos_atten_probs(cur_step, hyp_ids[l], d))); - } else { - att_vec->add_prob( - float(t_atten_probs(cur_step, hyp_ids[l], d))); - } - } - } - out_hyps_t(i, j) = terminated_hyps.SerializeAsString(); - } - } - } - }); - } - - private: - int32 eos_id_ = 0; - int32 num_hyps_per_beam_ = 0; -}; - -REGISTER_KERNEL_BUILDER(Name("HypsFromBeamSearchOuts") - .Device(DEVICE_CPU) - .TypeConstraint("T"), - HypsFromBeamSearchOuts); -REGISTER_KERNEL_BUILDER(Name("HypsFromBeamSearchOuts") - .Device(DEVICE_CPU) - .TypeConstraint("T"), - HypsFromBeamSearchOuts); - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/beam_search_step_op_kernels.h b/lingvo/core/ops/beam_search_step_op_kernels.h deleted file mode 100644 index 446d5dd6..00000000 --- a/lingvo/core/ops/beam_search_step_op_kernels.h +++ /dev/null @@ -1,244 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#ifndef LINGVO_CORE_OPS_BEAM_SEARCH_STEP_OP_KERNELS_H_ -#define LINGVO_CORE_OPS_BEAM_SEARCH_STEP_OP_KERNELS_H_ - -#include // std::sort -#include // std::greater -#include - -#include "lingvo/core/ops/hyps.pb.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/lib/strings/strcat.h" - -namespace tensorflow { -namespace lingvo { - -// Simple tuple for book keeping during beam pruning. -struct Hyp { - int32 beam_id; // The beam that this hyp belongs to. - int32 hyp_id; // The hypothesis id. - int32 word_id; // The id for the predicted next word. - float local_score; // Local score from the current step. - float global_score; // Cumulative score till the current step. - std::vector prev_ids; // The (non-epsilon) token ids up to this step. - - string DebugString() const { - return strings::StrCat(beam_id, " ", hyp_id, " ", word_id, " ", local_score, - " ", global_score); - } -}; - -struct HigherScore { - bool operator()(const Hyp& x, const Hyp& y) const { - // We only compare hyps belonging to the same beams. - CHECK_EQ(x.beam_id, y.beam_id); - if (x.global_score > y.global_score) return true; - if (x.global_score < y.global_score) return false; - if (x.word_id < y.word_id) return true; - if (x.word_id > y.word_id) return false; - return x.hyp_id < y.hyp_id; - } -}; - -// Similar to HigherScore, but optionally return Hyps with eos_id in it -// as the one with the higher score. -struct HigherScoreWithEos { - int eos_id_; - // Determines whether or not to prioritize Hyps with eos_id. If False, - // this will behave exactly like HigherScore. - bool include_eos_; - - HigherScoreWithEos(int eos_id, bool include_eos) - : eos_id_(eos_id), include_eos_(include_eos) {} - - bool operator()(const Hyp& x, const Hyp& y) const { - // We only compare hyps belonging to the same beams. - CHECK_EQ(x.beam_id, y.beam_id); - // Note that we revert to HigherScore's behavior when _both_ paths contain - // eos_id_ as the word_id. - if (!(x.word_id == eos_id_ && y.word_id == eos_id_)) { - if (x.word_id == eos_id_ && include_eos_) return true; - if (y.word_id == eos_id_ && include_eos_) return false; - } - // The following behavior is the same as that of HigherScore's. - if (x.global_score > y.global_score) return true; - if (x.global_score < y.global_score) return false; - if (x.word_id < y.word_id) return true; - if (x.word_id > y.word_id) return false; - return x.hyp_id < y.hyp_id; - } -}; - -struct BetterTerminatedHyp { - bool operator()(const Hypothesis& x, const Hypothesis& y) const { - // We only compare hyps belonging to the same beams. - CHECK_EQ(x.beam_id(), y.beam_id()); - if (x.normalized_score() > y.normalized_score()) return true; - if (x.normalized_score() < y.normalized_score()) return false; - return x.ids_size() < y.ids_size(); - } -}; - -struct ExtractGlobalScore { - float operator()(const Hyp& x) const { return x.global_score; } -}; - -struct ExtractNormalizedScore { - float operator()(const Hypothesis& x) const { return x.normalized_score(); } -}; - -template -struct Id { - const T& operator()(const T& t) const { return t; } -}; - -template -struct DefaultInsert { - explicit DefaultInsert(int unused_epsilon_id) {} - void operator()(const T& t, std::vector* items) const { - items->push_back(t); - } -}; - -// Returns true if 'cur_hyp' ad 'other_hyp' represent the same label sequence -// when epsilons are ignored, and false otherwise. -bool IsDuplicateHyp(const Hyp& cur_hyp, const Hyp& other_hyp, - const int epsilon_id); - -float LogSumExp(float a, float b); - -// An insertion operator that first checks whether 'hyp' is a duplicate of any -// hyp already in 'items'. If so, these two hyps are merged. -// This check is only performed if we are using a model that emits epsilons -// (NT or RNN-T). For models that do not emit epsilons (ie epsilon_id < 0) -// 'hyp' is always added to 'items', identical to DefaultInsert. -struct InsertHypWithEpsilonDedupe { - explicit InsertHypWithEpsilonDedupe(int _epsilon_id) - : epsilon_id(_epsilon_id), better_hyp() {} - void operator()(const Hyp& hyp, std::vector* items) const { - if (epsilon_id < 0) { - items->push_back(hyp); - return; - } - for (int i = 0; i < items->size(); ++i) { - const Hyp& old_hyp = (*items)[i]; - if (IsDuplicateHyp(hyp, old_hyp, epsilon_id)) { - Hyp combined_hyp = better_hyp(hyp, old_hyp) ? hyp : old_hyp; - combined_hyp.global_score = - LogSumExp(hyp.global_score, old_hyp.global_score); - (*items)[i] = combined_hyp; - return; - } - } - items->push_back(hyp); - } - int epsilon_id; - const HigherScore better_hyp; -}; - - -// A helper class keeps track of top K highest ranked elements added. -// Comp(x, y) returns true iff x is ranked higher than y. -// Epsilon id should be set to -1 for models which do not use epsilon (e.g. LAS -// and any non-speech model), or to the id used for epsilon (end of chunk) -// for epsilon emitting models (RNN-T, NT). -// -// E.g., -// TopK topk> (100, /* epsilon id */ -1); -// topk.Add(100); -// topk.Add(-100); -// ... -// result = topk.Get(); -// // results contains the smallest 100 int added to topk. -template , typename Extract = Id, - typename Insert = DefaultInsert> -class TopK { - public: - explicit TopK(int k, int epsilon_id) - : k_(k), comp_(), extract_(), insert_(epsilon_id), selected_(false) {} - // eos_id and inlclude_eos flag will be passed on to the Comparator. - explicit TopK(int k, int epsilon_id, int eos_id, bool include_eos) - : k_(k), - comp_(Comp(eos_id, include_eos)), - extract_(), - insert_(epsilon_id), - selected_(false) {} - - using U = typename std::result_of::type; - - // Return an element that is less than or equal to the least element - // of the top k. - U Add(const T& e) { - if (!selected_ || comp_(e, items_[k_ - 1])) { - insert_(e, &items_); - if (items_.size() >= 2 * k_) Shrink(); - } - if (!selected_) return std::numeric_limits::lowest(); - return extract_(items_[k_ - 1]); - } - - const std::vector& Get() { - if (items_.size() > k_) Shrink(); - return items_; - } - - void Clear() { - selected_ = false; - items_.clear(); - } - - private: - const int k_; - const Comp comp_; - const Extract extract_; - const Insert insert_; - bool selected_; // Becomes true if k-th top element so far is known. - std::vector items_; - - void Shrink() { - // Pivot is the k-th element, i.e., items_[k_-1]. - std::nth_element(items_.begin(), items_.begin() + k_ - 1, items_.end(), - comp_); - items_.resize(k_); - selected_ = true; - } -}; - -// Exposed for benchmarking purposes. -// Given the current partial hypothesis in 'hyps' for all beams in a batch and -// the predicted next step scores 'scores', return the best scored 'k+m' -// hypotheses where the first 'k' hypotheses are used for search in the next -// step while the remaining 'm' hypotheses are kept as alternatives to increase -// diversity. 'eos_id' is the end of beam id of the target language. -// -// eos_in_topk is filled with true/false to indicate whether or not the eos -// symbol is among the topk candidate for a hyp. -void ComputeTopKPlusM(const std::vector& hyps, const Tensor& scores, - const int32 k, const int32 m, const int32 eos_id, - const int32 eoc_id, const int32 num_beams, - const float valid_eos_max_logit_delta, - const float local_eos_threshold, bool is_first_step, - bool is_last_decoder_step, const Tensor& is_last_chunk, - bool merge_paths, bool allow_empty_terminated_hyp, - std::vector* eos_in_topk, std::vector* top_k, - std::vector* extra_m, std::vector* eos_hyps, - std::vector* terminal_symbol); - -} // namespace lingvo -} // namespace tensorflow - -#endif // LINGVO_CORE_OPS_BEAM_SEARCH_STEP_OP_KERNELS_H_ diff --git a/lingvo/core/ops/beam_search_step_op_test.py b/lingvo/core/ops/beam_search_step_op_test.py deleted file mode 100644 index 8d2bbf1f..00000000 --- a/lingvo/core/ops/beam_search_step_op_test.py +++ /dev/null @@ -1,655 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for beam_search_op.""" - -from lingvo import compat as tf -from lingvo.core import ops -from lingvo.core import test_utils -from lingvo.core.ops import hyps_pb2 -import numpy as np - -from google.protobuf import text_format - -_MIN_SCORE = -1e36 - - -class BeamSearchOpTest(test_utils.TestCase): - - def setUp(self): - super().setUp() - np.random.seed(12345) - tf.random.set_seed(398849988) - - def _runBeamSearchOpHelper(self, - b_size, - num_beams, - seq_len, - init_best_score, - probs, - init_atten_probs, - atten_probs, - beam_size=3.0, - ensure_full_beam=False, - force_eos_in_last_step=False, - local_eos_threshold=-100.0): - eos_id = 2 - num_hyps_per_beam = b_size / num_beams - - best_scores = tf.zeros([num_beams]) - cumulative_scores = tf.zeros([b_size]) - scores = tf.zeros([seq_len, b_size]) - hyps = tf.zeros([seq_len, b_size], dtype=tf.int32) - prev_hyps = tf.zeros([seq_len, b_size], dtype=tf.int32) - done_hyps = tf.as_string(tf.zeros([seq_len, b_size], dtype=tf.int32)) - best_scores += init_best_score - - for i, prob in enumerate(probs): - (best_scores, cumulative_scores, scores, hyps, prev_hyps, done_hyps, - atten_probs, done) = ops.beam_search_step( - prob, - init_atten_probs, - best_scores, - cumulative_scores, - scores, - hyps, - prev_hyps, - done_hyps, - atten_probs, [], - i, - eos_id=eos_id, - beam_size=beam_size, - ensure_full_beam=ensure_full_beam, - num_hyps_per_beam=num_hyps_per_beam, - valid_eos_max_logit_delta=0.1, - force_eos_in_last_step=force_eos_in_last_step, - local_eos_threshold=local_eos_threshold) - - with self.session(use_gpu=False): - (best_scores, cumulative_scores, scores, hyps, prev_hyps, done_hyps, - atten_probs, done, scores, atten_probs) = self.evaluate([ - best_scores, cumulative_scores, scores, hyps, prev_hyps, done_hyps, - atten_probs, done, scores, atten_probs - ]) - - return (best_scores, cumulative_scores, scores, hyps, prev_hyps, done_hyps, - atten_probs, done, scores, atten_probs) - - def _testBeamSearchOpHelper(self, - b_size, - num_beams, - seq_len, - init_best_score, - probs, - init_atten_probs, - atten_probs, - best_scores_expected, - cum_scores_expected, - scores_expected, - hyps_expected, - prev_hyps_expected, - atten_probs_expected, - force_eos_in_last_step=False, - local_eos_threshold=-100.0): - - (best_scores, cumulative_scores, scores, hyps, prev_hyps, done_hyps, - atten_probs, done, scores, atten_probs) = self._runBeamSearchOpHelper( - b_size, - num_beams, - seq_len, - init_best_score, - probs, - init_atten_probs, - atten_probs, - force_eos_in_last_step=force_eos_in_last_step, - local_eos_threshold=local_eos_threshold) - - tf.logging.info(np.array_repr(best_scores)) - tf.logging.info(np.array_repr(cumulative_scores)) - tf.logging.info(np.array_repr(scores)) - tf.logging.info(np.array_repr(hyps)) - tf.logging.info(np.array_repr(prev_hyps)) - tf.logging.info(np.array_repr(done_hyps)) - tf.logging.info(np.array_repr(atten_probs)) - tf.logging.info(np.array_repr(done)) - tf.logging.info(np.array_repr(scores)) - tf.logging.info(np.array_repr(atten_probs)) - - self.assertAllClose(best_scores_expected, best_scores) - self.assertAllClose(cum_scores_expected, cumulative_scores) - self.assertAllClose(scores_expected, scores) - self.assertAllClose(hyps_expected, hyps) - self.assertAllClose(prev_hyps_expected, prev_hyps) - self.assertAllClose(atten_probs_expected, atten_probs) - self.assertEqual(False, done) - - return done_hyps - - def testBeamSearchOp(self): - b_size = 8 - num_beams = 2 - seq_len = 6 - num_classes = 5 - - best_scores_expected = [1.769434, 1.640316] - cum_scores_expected = [ - 1.823942, 1.609159, 1.610366, 1.454234, 1.348811, 1.3167, 1.346274, - 1.045735 - ] - scores_expected = [ - [ - 0.86230338, 0.84442794, 0.45372832, 0.38127339, 0.42067075, - 0.25818801, 0.38612545, 0.18693292 - ], - [ - 0.96163845, 0.76473117, 0.74806261, 0.60980642, 0.9281404, - 0.47227204, 0.89254606, 0.20130682 - ], - [0., 0., 0., 0., 0., 0., 0., 0.], - [0., 0., 0., 0., 0., 0., 0., 0.], - [0., 0., 0., 0., 0., 0., 0., 0.], - [0., 0., 0., 0., 0., 0., 0., 0.], - ] - hyps_expected = [[1, 0, 0, 3, 4, 1, 3, 4], [1, 4, 4, 1, 1, 3, 1, 0], - [0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0]] - prev_hyps_expected = [[0, 1, 0, 1, 0, 1, 0, 1], [0, 1, 0, 1, 4, 1, 2, 1], - [0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0]] - - hyp_str_expected = """ - beam_id: 1 - ids: 1 - ids: 2 - scores: 0.25818801 - scores: 0.65319967 - atten_vecs { - prob: 0.38612545 - prob: 0.42067075 - prob: 0.84442794 - } - atten_vecs { - prob: 0.45298624 - prob: 0.53518069 - prob: 0.57700801 - } - """ - atten_probs_expected = [ - [ - [0.45372832, 0.86230338, 0.65504861], - [0.38612545, 0.42067075, 0.84442794], - [0.45372832, 0.86230338, 0.65504861], - [0.38612545, 0.42067075, 0.84442794], - [0.45372832, 0.86230338, 0.65504861], - [0.38612545, 0.42067075, 0.84442794], - [0.45372832, 0.86230338, 0.65504861], - [0.38612545, 0.42067075, 0.84442794], - ], - [ - [0.45372832, 0.86230338, 0.65504861], - [0.38612545, 0.42067075, 0.84442794], - [0.45372832, 0.86230338, 0.65504861], - [0.38612545, 0.42067075, 0.84442794], - [0.0532794, 0.53777719, 0.07609642], - [0.38612545, 0.42067075, 0.84442794], - [0.25818801, 0.03645897, 0.38127339], - [0.38612545, 0.42067075, 0.84442794], - ], - [[0., 0., 0.], [0., 0., 0.], [0., 0., 0.], [0., 0., 0.], [0., 0., 0.], - [0., 0., 0.], [0., 0., 0.], [0., 0., 0.]], - [[0., 0., 0.], [0., 0., 0.], [0., 0., 0.], [0., 0., 0.], [0., 0., 0.], - [0., 0., 0.], [0., 0., 0.], [0., 0., 0.]], - [[0., 0., 0.], [0., 0., 0.], [0., 0., 0.], [0., 0., 0.], [0., 0., 0.], - [0., 0., 0.], [0., 0., 0.], [0., 0., 0.]], - [[0., 0., 0.], [0., 0., 0.], [0., 0., 0.], [0., 0., 0.], [0., 0., 0.], - [0., 0., 0.], [0., 0., 0.], [0., 0., 0.]], - ] - - scores = [ - tf.random.uniform([b_size, num_classes], seed=12345), - tf.random.uniform([b_size, num_classes], seed=12346), - ] - init_atten_probs = tf.random.uniform([b_size, 3], seed=12345) - atten_probs = tf.zeros([seq_len, b_size, 3]) - done_hyps = self._testBeamSearchOpHelper( - b_size, num_beams, seq_len, 0., scores, init_atten_probs, atten_probs, - best_scores_expected, cum_scores_expected, scores_expected, - hyps_expected, prev_hyps_expected, atten_probs_expected) - - self._SameHyp(hyp_str_expected, done_hyps[1, 5]) - - # The following 3 tests, test each step of this decoding tree. - # Test that beam search finds the most probable sequence. - # These probabilities represent the following search - # - # G0 (0) - # / \ - # / \ - # / \ - # / \ - # 0(0.6) 1(0.4) - # / \ / \ - # / \ / \ - # / \ / \ - # 0(0.55) 1(0.45) 0(0.05) 1(0.95) - # - # and these decoding probabilities - # 000 - 0.6 * 0.55 = 0.33 - # 001 - 0.6 * 0.45 = 0.27 - # 010 - 0.4 * 0.05 = 0.02 - # 011 - 0.4 * 0.95 = 0.38 - # - # Greedy would decode 000 since the first 0 is the most probable, but beam - # should decode 011 since it's the highest probability then followed by 000. - def _test_single_step_small_vocab_3(self): - b_size = 2 - num_beams = 1 - seq_len = 3 - - probs = [np.log([[0.6, 0.4, 0.0000001], [0.6, 0.4, 0.0000001]])] - done_hyps = self._testBeamSearchOpHelper( - b_size, - num_beams, - seq_len, - _MIN_SCORE, - probs, - init_atten_probs=tf.zeros([b_size, 0]), - atten_probs=np.zeros([seq_len, b_size, 0]), - best_scores_expected=[_MIN_SCORE], - cum_scores_expected=np.log([0.6, 0.4]), - scores_expected=[np.log([0.6, 0.4]), [0, 0], [0, 0]], - hyps_expected=[[0, 1], [0, 0], [0, 0]], - prev_hyps_expected=[[0, 0], [0, 0], [0, 0]], - atten_probs_expected=np.zeros([seq_len, b_size, 0])) - - np.testing.assert_array_equal([['0', '0'], ['0', '0'], ['0', '0']], - done_hyps) - - def test_two_steps_small_vocab_3(self): - b_size = 2 - num_beams = 1 - seq_len = 3 - - probs = [ - np.log([[0.6, 0.4, 0.0000001], [0.6, 0.4, 0.0000001]]), - np.log([[0.55, 0.45, 0.0000001], [0.05, 0.95, 0.0000001]]), - ] - done_hyps = self._testBeamSearchOpHelper( - b_size, - num_beams, - seq_len, - _MIN_SCORE, - probs, - init_atten_probs=tf.zeros([b_size, 0]), - atten_probs=np.zeros([seq_len, b_size, 0]), - best_scores_expected=[_MIN_SCORE], - # Note, probabilites are swapped due to beams being swapped. - cum_scores_expected=np.log([0.4 * 0.95, 0.6 * 0.55]), - scores_expected=[np.log([0.6, 0.4]), - np.log([0.95, 0.55]), [0, 0]], - hyps_expected=[[0, 1], [1, 0], [0, 0]], - prev_hyps_expected=[[0, 0], [1, 0], [0, 0]], - atten_probs_expected=np.zeros([seq_len, b_size, 0])) - - np.testing.assert_array_equal([[b'0', b'0'], [b'0', b'0'], [b'0', b'0']], - done_hyps) - - def test_three_steps_eos(self): - b_size = 2 - num_beams = 1 - seq_len = 3 - - probs = [ - np.log([[0.6, 0.4, 0.0000001], [0.6, 0.4, 0.0000001]]), - np.log([[0.55, 0.45, 0.0000001], [0.05, 0.95, 0.0000001]]), - # Finish the beams with EOS - np.log([[0.05, 0.05, 0.9], [0.05, 0.05, 0.9]]), - ] - - done_hyps = self._testBeamSearchOpHelper( - b_size, - num_beams, - seq_len, - _MIN_SCORE, - probs, - init_atten_probs=tf.zeros([b_size, 0]), - atten_probs=np.zeros([seq_len, b_size, 0]), - best_scores_expected=np.log([0.4 * 0.95 * 0.9]), - cum_scores_expected=np.log([0.4 * 0.95 * 0.05, 0.4 * 0.95 * 0.05]), - scores_expected=[ - np.log([0.6, 0.4]), - np.log([0.95, 0.55]), - np.log([0.05, 0.05]) - ], - hyps_expected=[[0, 1], [1, 0], [0, 1]], - prev_hyps_expected=[[0, 0], [1, 0], [0, 0]], - atten_probs_expected=np.zeros([seq_len, b_size, 0])) - - expected_for_beam_0 = """ - beam_id: 0 - ids: 1 - ids: 1 - ids: 2 - scores: -0.916290700436 # = log 0.4 - scores: -0.0512933060527 # = log 0.95 - scores: -0.105360545218 # = log 0.9 - atten_vecs { - } - atten_vecs { - } - atten_vecs { - } - """ - - expected_for_beam_1 = """ - beam_id: 0 - ids: 0 - ids: 0 - ids: 2 - scores: -0.510825574398 # = log 0.6 - scores: -0.597836971283 # = log 0.55 - scores: -0.105360545218 # = log 0.9 - atten_vecs { - } - atten_vecs { - } - atten_vecs { - } - """ - - self._SameHyp(expected_for_beam_0, done_hyps[2, 0]) - self._SameHyp(expected_for_beam_1, done_hyps[2, 1]) - - def test_three_steps_force_eos(self): - b_size = 2 - num_beams = 1 - seq_len = 3 - - probs = [ - np.log([[0.6, 0.4, 0.0000001], [0.6, 0.4, 0.0000001]]), - np.log([[0.55, 0.45, 0.0000001], [0.05, 0.95, 0.0000001]]), - # EOS probability is still very low, so unless it is forced it will - # not be in the beam. - np.log([[0.45, 0.44, 0.01], [0.5, 0.5, 0.01]]), - ] - - # Set expected values - cum_scores_expected = np.log([0.4 * 0.95 * 0.45, 0.4 * 0.95 * 0.44]) - scores_expected = [ - np.log([0.6, 0.4]), - np.log([0.95, 0.55]), - np.log([0.45, 0.44]) - ] - hyps_expected = [[0, 1], [1, 0], [0, 1]] - prev_hyps_expected = [[0, 0], [1, 0], [0, 0]] - - # If force EOS is false, the we get empty hyps after beam search. - done_hyps = self._testBeamSearchOpHelper( - b_size, - num_beams, - seq_len, - _MIN_SCORE, - probs, - init_atten_probs=tf.zeros([b_size, 0]), - atten_probs=np.zeros([seq_len, b_size, 0]), - best_scores_expected=[_MIN_SCORE], - cum_scores_expected=cum_scores_expected, - scores_expected=scores_expected, - hyps_expected=hyps_expected, - prev_hyps_expected=prev_hyps_expected, - atten_probs_expected=np.zeros([seq_len, b_size, 0]), - force_eos_in_last_step=False) - np.testing.assert_array_equal([[b'0', b'0'], [b'0', b'0'], [b'0', b'0']], - done_hyps) - - # If force eos is true, we get valid results as in test_three_step_eos, - # but with lower probabilities (because of lower eos probs). - done_hyps = self._testBeamSearchOpHelper( - b_size, - num_beams, - seq_len, - _MIN_SCORE, - probs, - init_atten_probs=tf.zeros([b_size, 0]), - atten_probs=np.zeros([seq_len, b_size, 0]), - best_scores_expected=np.log([0.4 * 0.95 * 0.01]), - cum_scores_expected=cum_scores_expected, - scores_expected=scores_expected, - hyps_expected=hyps_expected, - prev_hyps_expected=prev_hyps_expected, - atten_probs_expected=np.zeros([seq_len, b_size, 0]), - force_eos_in_last_step=True) - - expected_for_beam_0 = """ - beam_id: 0 - ids: 1 - ids: 1 - ids: 2 - scores: -0.916290700436 # = log 0.4 - scores: -0.0512933060527 # = log 0.95 - scores: -4.605170185988 # = log 0.01 - atten_vecs { - } - atten_vecs { - } - atten_vecs { - } - """ - - expected_for_beam_1 = """ - beam_id: 0 - ids: 0 - ids: 0 - ids: 2 - scores: -0.510825574398 # = log 0.6 - scores: -0.597836971283 # = log 0.55 - scores: -4.605170185988 # = log 0.01 - atten_vecs { - } - atten_vecs { - } - atten_vecs { - } - """ - - self._SameHyp(expected_for_beam_0, done_hyps[2, 0]) - self._SameHyp(expected_for_beam_1, done_hyps[2, 1]) - - def _testBeamSearchStoppingHelper(self, - beam_size, - ensure_full_beam, - local_eos_threshold=-100): - b_size = 2 - num_beams = 1 - seq_len = 3 - probs = [ - # Only finish one beam with EOS. - np.log([[0.05, 0.05, 0.9], [0.05, 0.9, 0.05]]), - ] - - results = self._runBeamSearchOpHelper( - b_size, - num_beams, - seq_len, - _MIN_SCORE, - probs, - init_atten_probs=tf.zeros([b_size, 0]), - atten_probs=np.zeros([seq_len, b_size, 0]), - beam_size=beam_size, - ensure_full_beam=ensure_full_beam, - local_eos_threshold=local_eos_threshold) - all_done = results[7] - return all_done - - def test_beam_size_large(self): - # With default beam size, we are not yet all done, because we still have an - # active hyp within 3.0 of best done hyp. - all_done = self._testBeamSearchStoppingHelper(3.0, False) - self.assertEqual(False, all_done) - - def test_beam_size_small(self): - # With small beam size, we are all done, because the active hyp is not - # within such a narrow margin of best done hyp. - all_done = self._testBeamSearchStoppingHelper(0.1, False) - self.assertEqual(True, all_done) - - def test_ensure_full_beam(self): - # With small beam size and ensure_full_beam, we are _not_ yet done, - # because we require to have two done hyps before stopping, regardless of - # beam size. - all_done = self._testBeamSearchStoppingHelper(0.1, True) - self.assertEqual(False, all_done) - - def test_small_eos_threshold(self): - # With a small eos_threshold, we are done because the active hyp produced, - # , independent of small beam size. - all_done = self._testBeamSearchStoppingHelper(0.1, False, -100.0) - self.assertTrue(all_done) - - def test_large_eos_threshold(self): - # With larger eos_threshold, we are _not_ yet done, because we do not hit - # criteria we we require to have two done hyps before stopping, - # regardless of beam size. - all_done = self._testBeamSearchStoppingHelper(0.1, False, 0.01) - self.assertFalse(all_done) - - def _SameHyp(self, expected_hyp_str, real_serialized_hyp): - hyp1 = hyps_pb2.Hypothesis() - text_format.Parse(expected_hyp_str, hyp1) - hyp2 = hyps_pb2.Hypothesis() - hyp2.ParseFromString(real_serialized_hyp) - - self.assertEqual(hyp1.beam_id, hyp2.beam_id) - self.assertEqual(hyp1.ids, hyp2.ids) - self.assertNear(hyp1.normalized_score, hyp2.normalized_score, 1e-6) - self.assertAllClose(hyp1.scores, hyp2.scores) - self.assertEqual(len(hyp1.atten_vecs), len(hyp2.atten_vecs)) - for av1, av2 in zip(hyp1.atten_vecs, hyp2.atten_vecs): - self.assertAllClose(av1.prob, av2.prob) - - def testTopKTerminatedHypsOp(self): - with self.session(use_gpu=False): - b_size = 8 - num_beams = 2 - num_hyps_per_beam = b_size / num_beams - seq_len = 6 - scores = tf.random.uniform([b_size, 5], seed=12345) - atten_probs = tf.random.uniform([b_size, 3], seed=12345) - src_seq_lengths = [3, 3] - best_scores = tf.zeros([num_beams]) - cumulative_scores = tf.zeros([b_size]) - in_scores = tf.zeros([seq_len, b_size]) - in_hyps = tf.zeros([seq_len, b_size], dtype=tf.int32) - in_prev_hyps = tf.zeros([seq_len, b_size], dtype=tf.int32) - in_done_hyps = tf.as_string(tf.zeros([seq_len, b_size], dtype=tf.int32)) - in_atten_probs = tf.zeros([seq_len, b_size, 3]) - - (out_best_scores_0, out_cumulative_scores_0, out_scores_0, out_hyps_0, - out_prev_hyps_0, out_done_hyps_0, out_atten_probs_0, - _) = ops.beam_search_step( - scores, - atten_probs, - best_scores, - cumulative_scores, - in_scores, - in_hyps, - in_prev_hyps, - in_done_hyps, - in_atten_probs, [], - 0, - eos_id=2, - beam_size=3.0, - num_hyps_per_beam=num_hyps_per_beam) - - outputs = ops.beam_search_step( - scores, - atten_probs, - out_best_scores_0, - out_cumulative_scores_0, - out_scores_0, - out_hyps_0, - out_prev_hyps_0, - out_done_hyps_0, - out_atten_probs_0, [], - 1, - eos_id=2, - beam_size=3.0, - num_hyps_per_beam=num_hyps_per_beam) - - # Get the topk terminated hyps. - in_done_hyps = outputs[5] - topk_hyps = ops.top_k_terminated_hyps( - in_done_hyps, - src_seq_lengths, - k=2, - num_hyps_per_beam=num_hyps_per_beam, - length_normalization=0.2, - coverage_penalty=0.2, - target_seq_length_ratio=1.0) - seq_ids, seq_lens, seq_scores = ops.unpack_hyp( - tf.reshape(topk_hyps, [-1]), max_seq_length=5) - - k1, k2, k3, k4 = self.evaluate([topk_hyps, seq_ids, seq_lens, seq_scores]) - print(np.array_repr(k1)) - assert k1.size == 4 - - expected_top1_for_beam_0 = """ - beam_id: 0 - ids: 1 - ids: 2 - scores: 0.86230338 - scores: 0.65504861 - atten_vecs { - prob: 0.45372832 - prob: 0.86230338 - prob: 0.65504861 - } - atten_vecs { - prob: 0.45372832 - prob: 0.86230338 - prob: 0.65504861 - } - normalized_score: 1.002714 - """ - expected_top2_for_beam_1 = """ - beam_id: 1 - ids: 3 - ids: 2 - scores: 0.38127339 - scores: 0.57700801 - atten_vecs { - prob: 0.38612545 - prob: 0.42067075 - prob: 0.84442794 - } - atten_vecs { - prob: 0.18693292 - prob: 0.17821217 - prob: 0.66380036 - } - normalized_score: 0.480028 - """ - self._SameHyp(expected_top1_for_beam_0, k1[0, 0]) - self._SameHyp(expected_top2_for_beam_1, k1[1, 1]) - - self.assertAllClose( - k2, - [[1, 2, 0, 0, 0], [4, 2, 0, 0, 0], [4, 2, 0, 0, 0], [3, 2, 0, 0, 0]]) - self.assertAllClose(k3, [2, 2, 2, 2]) - self.assertAllClose(k4, [1.002714, 0.684296, 0.522484, 0.480028]) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/ops/beam_search_step_op_top_k_test.cc b/lingvo/core/ops/beam_search_step_op_top_k_test.cc deleted file mode 100644 index e4b906cc..00000000 --- a/lingvo/core/ops/beam_search_step_op_top_k_test.cc +++ /dev/null @@ -1,172 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include -#include "lingvo/core/ops/beam_search_step_op_kernels.h" - -namespace tensorflow { -namespace lingvo { -namespace { - -using ::testing::Eq; -using ::testing::FloatEq; -using ::testing::FloatNear; -using ::testing::SizeIs; - -template -void Populate(T* top_k) { - // Add 3 distinct hyps. - float bottom_of_topk; - // Hyp struct consists of: - // beam_id, hyp_id, word_id, local_score, global_score, prev_ids - bottom_of_topk = top_k->Add({0, 2, 3, -0.4, -2.0, {1, 2, 3}}); - EXPECT_THAT(bottom_of_topk, FloatEq(std::numeric_limits::lowest())); - bottom_of_topk = top_k->Add({0, 1, 8, -0.7, -1.3, {1, 7, 2}}); - EXPECT_THAT(bottom_of_topk, FloatEq(std::numeric_limits::lowest())); - bottom_of_topk = top_k->Add({0, 6, 2, -0.1, -1.7, {1, 9, 5}}); - // No resize yet, since we haven't gotten to k * 2 = 4 elements yet. - EXPECT_THAT(bottom_of_topk, FloatEq(std::numeric_limits::lowest())); - bottom_of_topk = top_k->Add({0, 4, 3, -0.5, -2.5, {1, 2, 4}}); - // After fourth element we resize down to two best elements. - EXPECT_THAT(bottom_of_topk, FloatNear(-1.7, 0.001)); - - const auto& hyps = top_k->Get(); - EXPECT_THAT(hyps, SizeIs(2)); - EXPECT_THAT(hyps[0].hyp_id, Eq(1)); - EXPECT_THAT(hyps[0].global_score, FloatNear(-1.3, 0.001)); - EXPECT_THAT(hyps[1].hyp_id, Eq(6)); - EXPECT_THAT(hyps[1].global_score, FloatNear(-1.7, 0.001)); - - // Add a dupe. - top_k->Add({0, 5, 8, -0.7, -1.5, {1, 7, 2}}); -} - -template -void PopulateWithEpsilons(T* top_k) { - // Add 3 distinct hyps. - float bottom_of_topk; - bottom_of_topk = top_k->Add({0, 2, 3, -0.4, -2.0, {1, 2, 3}}); - EXPECT_THAT(bottom_of_topk, FloatEq(std::numeric_limits::lowest())); - bottom_of_topk = top_k->Add({0, 1, 8, -0.7, -1.3, {1, 7, 2}}); - EXPECT_THAT(bottom_of_topk, FloatEq(std::numeric_limits::lowest())); - bottom_of_topk = top_k->Add({0, 6, 2, -0.1, -1.7, {1, 3, 4, 9, 5}}); - // No resize yet, since we haven't gotten to k * 2 = 4 elements yet. - EXPECT_THAT(bottom_of_topk, FloatEq(std::numeric_limits::lowest())); - bottom_of_topk = top_k->Add({0, 4, 3, -0.5, -2.5, {1, 2, 4, 3}}); - // After fourth element we resize down to two best elements. - EXPECT_THAT(bottom_of_topk, FloatNear(-1.7, 0.001)); - - const auto& hyps = top_k->Get(); - EXPECT_THAT(hyps, SizeIs(2)); - EXPECT_THAT(hyps[0].hyp_id, Eq(1)); - EXPECT_THAT(hyps[0].global_score, FloatNear(-1.3, 0.001)); - EXPECT_THAT(hyps[1].hyp_id, Eq(6)); - EXPECT_THAT(hyps[1].global_score, FloatNear(-1.7, 0.001)); - - // Add a dupe. Doing the dedupe requires comparing last prev id of this hyp - // with cur id of candidate hyp since cur id for this hyp is epsilon. - const int epsilon_id = 0; - bottom_of_topk = top_k->Add({0, 5, epsilon_id, -0.7, -1.5, {1, 7, 2, 8}}); -} - -bool IsDupe(const Hyp& hyp1, const Hyp& hyp2) { - if (hyp1.word_id != hyp2.word_id) { - return false; - } - if (hyp1.prev_ids.size() != hyp2.prev_ids.size()) { - return false; - } - return std::equal(hyp1.prev_ids.begin(), hyp1.prev_ids.end(), - hyp2.prev_ids.begin()); -} - -// Tests that when we use the default Insert, there is NO deduping. -TEST(TopKTest, TestInsertDefault) { - const int k = 2; - TopK top_k(k, /* epsilon id */ -1); - Populate>(&top_k); - // Check contents: The two items in the TopK are duplicates. - const auto& new_hyps = top_k.Get(); - EXPECT_THAT(new_hyps, SizeIs(2)); - EXPECT_THAT(new_hyps[0].hyp_id, Eq(1)); - EXPECT_THAT(new_hyps[1].hyp_id, Eq(5)); - EXPECT_TRUE(IsDupe(new_hyps[0], new_hyps[1])); -} - -// Tests that when we use InsertHypWithEpsilonDedupe but set the epsilon id -// to be less than zero, there is NO deduping. -TEST(TopKTest, TestInsertNoDedupe) { - const int k = 2; - TopK top_k( - k, /* epsilon_id */ -1); - Populate< - TopK>( - &top_k); - // Check contents: The two items in the TopK are duplicates. - const auto& new_hyps = top_k.Get(); - EXPECT_THAT(new_hyps, SizeIs(2)); - EXPECT_THAT(new_hyps[0].hyp_id, Eq(1)); - EXPECT_THAT(new_hyps[1].hyp_id, Eq(5)); - EXPECT_TRUE(IsDupe(new_hyps[0], new_hyps[1])); -} - -// Tests that when we use InsertHypWithEpsilonDedupe and set the epsilon id -// to be greater than or equal to zero, there IS deduping. -TEST(TopKTest, TestInsertWithDedupe) { - const int k = 2; - const int epsilon_id = 0; - TopK top_k( - k, epsilon_id); - Populate< - TopK>( - &top_k); - // Check contents: The two items in the TopK are not duplicates. The dupe - // has been merged into one of them. - const auto& new_hyps = top_k.Get(); - EXPECT_THAT(new_hyps, SizeIs(2)); - EXPECT_THAT(new_hyps[0].hyp_id, Eq(1)); - // Merging has happened so combined global score is: - // log(exp(-1.5) + exp(-1.3)) - EXPECT_THAT(new_hyps[0].global_score, FloatNear(-0.70186, 0.001)); - EXPECT_THAT(new_hyps[1].hyp_id, Eq(6)); - EXPECT_THAT(new_hyps[1].global_score, FloatNear(-1.7, 0.001)); - EXPECT_TRUE(!IsDupe(new_hyps[0], new_hyps[1])); -} - -// Tests that the deduping ignores epsilons. -TEST(TopKTest, TestInsertWithDedupeEpsilon) { - const int k = 2; - const int epsilon_id = 0; - TopK top_k( - k, epsilon_id); - PopulateWithEpsilons< - TopK>( - &top_k); - // Check contents: The two items in the TopK are not duplicates. The dupe - // has been merged into one of them. - const auto& new_hyps = top_k.Get(); - EXPECT_THAT(new_hyps, SizeIs(2)); - EXPECT_THAT(new_hyps[0].hyp_id, Eq(1)); - // Merging has happened so combined global score is: - // log(exp(-1.5) + exp(-1.3)) - EXPECT_THAT(new_hyps[0].global_score, FloatNear(-0.70186, 0.001)); - EXPECT_THAT(new_hyps[1].hyp_id, Eq(6)); - EXPECT_THAT(new_hyps[1].global_score, FloatNear(-1.7, 0.001)); - EXPECT_TRUE(!IsDupe(new_hyps[0], new_hyps[1])); -} - -} // namespace -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/best_step_op_kernels.cc b/lingvo/core/ops/best_step_op_kernels.cc deleted file mode 100644 index c5e38fec..00000000 --- a/lingvo/core/ops/best_step_op_kernels.cc +++ /dev/null @@ -1,169 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/summary.pb.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" -#include "tensorflow/core/lib/io/buffered_inputstream.h" -#include "tensorflow/core/lib/io/random_inputstream.h" -#include "tensorflow/core/lib/io/record_reader.h" -#include "tensorflow/core/lib/strings/numbers.h" -#include "tensorflow/core/lib/strings/str_util.h" -#include "tensorflow/core/platform/env.h" -#include "tensorflow/core/util/event.pb.h" - -namespace tensorflow { -namespace lingvo { -using ::tensorflow::io::RecordReader; -namespace { - -// Reads a text file containing 'step value' records, and finds the step that -// corresponds to the lowest-value record, within a given tolerance. -class BestStepOp : public OpKernel { - public: - explicit BestStepOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, ctx->GetAttr("hist_file", &hist_file_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("tol", &tol_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("minimize", &minimize_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("metric", &metric_)); - CHECK_GE(tol_, 0.0); - } - - void ExtractValueFromOneTfEvent(OpKernelContext* ctx, const string& filename, - std::map* step_value, - const string& metric, bool minimize) { - const Status status = ctx->env()->FileExists(filename); - if (status.ok()) { - ::std::unique_ptr file; - OP_REQUIRES_OK(ctx, ctx->env()->NewRandomAccessFile(filename, &file)); - ::std::unique_ptr reader(new RecordReader(file.get())); - - uint64 offset = 0; - tstring raw_proto; - while (reader->ReadRecord(&offset, &raw_proto).ok()) { - Event event; - CHECK(::tensorflow::ParseProtoUnlimited(&event, raw_proto.data(), - raw_proto.size())); - if (event.what_case() != Event::WhatCase::kSummary) { - continue; - } - if (event.has_summary()) { - for (const auto& value : event.summary().value()) { - // Look for the tag that matches the metric. - if (value.tag() == metric) { - if (minimize) { - step_value->insert( - std::pair(event.step(), value.simple_value())); - } else { - step_value->insert( - std::pair(event.step(), -value.simple_value())); - } - break; - } - } - } - } - } else { - LOG(WARNING) << "tf events file '" << filename << "' doesn't exist."; - } - } - - void ExtractValueFromTfEvents(OpKernelContext* ctx, const string& filename, - std::map* step_value) { - std::vector tf_events; - const Status status = ctx->env()->GetMatchingPaths(filename, &tf_events); - if (!tf_events.empty()) { - for (const auto& fname : tf_events) { - // Loop through all found tf events files. - ExtractValueFromOneTfEvent(ctx, fname, step_value, metric_, minimize_); - } - } else { - LOG(WARNING) << "Couldn't find tf events files that match pattern: '" - << filename; - } - } - - void ExtractValueFromTxt(OpKernelContext* ctx, const string& filename, - std::map* step_value) { - const Status status = ctx->env()->FileExists(filename); - if (status.ok()) { - std::unique_ptr file; - OP_REQUIRES_OK(ctx, ctx->env()->NewRandomAccessFile(filename, &file)); - std::unique_ptr input_stream( - new io::RandomAccessInputStream(file.get())); - io::BufferedInputStream in(input_stream.get(), 4 << 10); - string line; - while (true) { - const Status s = in.ReadLine(&line); - if (errors::IsOutOfRange(s)) break; - TF_CHECK_OK(s); - std::vector split_line = str_util::Split(line, ' '); - CHECK_EQ(split_line.size(), 2); - - int x; - CHECK(strings::safe_strto32(split_line[0], &x)); - float y; - CHECK(strings::safe_strtof(split_line[1], &y)); - - if (minimize_) { - step_value->insert(std::pair(x, y)); - } else { // Negate the value if it's the larger the better. - step_value->insert(std::pair(x, -y)); - } - } - } else { - LOG(WARNING) << "hist_file '" << &filename << "' doesn't exist."; - } - } - - void Compute(OpKernelContext* ctx) override { - int64 best_step = 0, last_step = 0; - float best_val = 0.0; - std::map step_value; - if (hist_file_.find("events.out.tfevents") != std::string::npos) { - // History file are tf events. - ExtractValueFromTfEvents(ctx, hist_file_, &step_value); - } else { // History file is a txt. - ExtractValueFromTxt(ctx, hist_file_, &step_value); - } - std::map::iterator itr; - for (itr = step_value.begin(); itr != step_value.end(); ++itr) { - last_step = itr->first; - const float val = itr->second; - if (best_step == 0 || val + tol_ < best_val) { - best_step = last_step; - best_val = val; - } - } - - Tensor* res; - OP_REQUIRES_OK(ctx, ctx->allocate_output(0, TensorShape({2}), &res)); - res->vec()(0) = best_step; - res->vec()(1) = last_step; - } - - private: - string hist_file_; - string metric_; - float tol_ = 0.0; - bool minimize_ = true; -}; - -REGISTER_KERNEL_BUILDER(Name("BestStep").Device(DEVICE_CPU), BestStepOp); - -} // namespace -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/best_step_op_test.py b/lingvo/core/ops/best_step_op_test.py deleted file mode 100644 index 09df4383..00000000 --- a/lingvo/core/ops/best_step_op_test.py +++ /dev/null @@ -1,85 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for best_step_op.""" - -from lingvo import compat as tf -from lingvo.core import ops -from lingvo.core import test_helper -from lingvo.core import test_utils - -FLAGS = tf.flags.FLAGS - - -class BestStepOp(test_utils.TestCase): - - def _HistFile(self): - return test_helper.test_src_dir_path('core/ops/testdata/history.txt') - - def _BleuFile(self): - return test_helper.test_src_dir_path('core/ops/testdata/history_bleu.txt') - - def _TfEventFile(self): - return test_helper.test_src_dir_path( - 'core/ops/testdata/events.out.tfevents.test') - - def testTol0(self): - g = tf.Graph() - with g.as_default(): - output = ops.best_step(self._HistFile()) - with self.session(graph=g): - best_step, last_step = self.evaluate(output) - self.assertEqual(best_step, 42122) - self.assertEqual(last_step, 42792) - - def testTolNon0(self): - g = tf.Graph() - with g.as_default(): - output = ops.best_step(self._HistFile(), 0.1) - with self.session(graph=g): - best_step, last_step = self.evaluate(output) - self.assertEqual(best_step, 37553) - self.assertEqual(last_step, 42792) - - def testNoFile(self): - g = tf.Graph() - with g.as_default(): - output = ops.best_step('') - with self.session(graph=g): - best_step, last_step = self.evaluate(output) - self.assertEqual(best_step, 0) - self.assertEqual(last_step, 0) - - def testAscendingValTol0(self): - g = tf.Graph() - with g.as_default(): - output = ops.best_step(self._BleuFile(), 0.0, False) - with self.session(graph=g): - best_step, last_step = self.evaluate(output) - self.assertEqual(best_step, 41500) - self.assertEqual(last_step, 46800) - - def testTfEventAscendingValTol0(self): - g = tf.Graph() - with g.as_default(): - output = ops.best_step(self._TfEventFile(), 0.0, False, 'bleu/dev') - with self.session(graph=g): - best_step, last_step = self.evaluate(output) - self.assertEqual(best_step, 102600) - self.assertEqual(last_step, 185200) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/ops/chain_record_yielder.cc b/lingvo/core/ops/chain_record_yielder.cc deleted file mode 100644 index 77922962..00000000 --- a/lingvo/core/ops/chain_record_yielder.cc +++ /dev/null @@ -1,77 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/chain_record_yielder.h" - -#include - -namespace tensorflow { -namespace lingvo { - -ChainRecordYielder::ChainRecordYielder( - const std::vector& yielder_options) - : current_yielder_idx_(0), - current_yielder_(nullptr), - yielder_options_(yielder_options) { - if (yielder_options.empty()) { - LOG(FATAL) << "There should be at least one set of options provided."; - } - current_yielder_ = BasicRecordYielder::New(yielder_options_.front()); -} - -ChainRecordYielder* ChainRecordYielder::New( - const std::vector& yielder_options) { - return new ChainRecordYielder(yielder_options); -} - -ChainRecordYielder::~ChainRecordYielder() {} - -void ChainRecordYielder::Close() { - { - absl::MutexLock l(&mu_); - if (current_yielder_) { - current_yielder_->Close(); - } - } - LOG(INFO) << this << "Chain record yielder exit"; - delete this; -} - -Status ChainRecordYielder::Yield(Record* record) { - absl::MutexLock l(&mu_); - - if (current_yielder_->current_epoch() > 1) { - int new_idx = (current_yielder_idx_ + 1) % yielder_options_.size(); - if (current_yielder_) { - current_yielder_->Close(); - } - current_yielder_idx_ = new_idx; - current_yielder_ = BasicRecordYielder::New(yielder_options_.at(new_idx)); - } - while (true) { - // Retry indefinitely until we get an Ok status from the specific yielder. - // This will stall the training if there is any unrecoverable error with - // the child yielder. - Status s = current_yielder_->Yield(record); - if (!s.ok()) { - LOG(WARNING) << s; - continue; - } - return s; - } -} - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/chain_record_yielder.h b/lingvo/core/ops/chain_record_yielder.h deleted file mode 100644 index b3223a95..00000000 --- a/lingvo/core/ops/chain_record_yielder.h +++ /dev/null @@ -1,74 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ -#ifndef LINGVO_CORE_OPS_CHAIN_RECORD_YIELDER_H_ -#define LINGVO_CORE_OPS_CHAIN_RECORD_YIELDER_H_ - -#include "lingvo/core/ops/record_yielder.h" - -namespace tensorflow { -namespace lingvo { - -// ChainRecordYielder is a special RecordYielder that consecutively yields -// examples based on a list of yielder options consuming those options in order. -// -// Usage example: -// BasicRecordYielder::Options opts1; -// opts1.file_pattern = ; -// opts1.seed = 301; -// opts1.bufsize = 1000000; // A randomized buffer with 1M records. -// opts1.parallelism = 8; // Use 8 iterators. -// BasicRecordYielder::Options opts2; -// opts2.file_pattern = ; -// opts2.seed = 301; -// opts2.bufsize = 1000000; // A randomized buffer with 1M records. -// opts2.parallelism = 8; // Use 8 iterators. -// // Yield all records defined by opts1, then all records defined by opts2. -// ChainRecordYielder* yielder = ChainRecordYielder::New({opts1, opts2}); -// Record record; -// while (true) { -// yielder->Yield(&record); -// // process record. -// } -// yielder->Close(); -// -// ChainRecordYielder can be accessed by multiple threads concurrently. -class ChainRecordYielder : public RecordYielder { - public: - ~ChainRecordYielder() override; - void Close() override; - Status Yield(Record* record) override; - - // Creates new ChainRecordYielder that will be creating child yielders using - // options provided. Caller is responsible for closing the ChainRecordYielder - // returned by this function. Caller should not delete the yielder as it will - // be handled internally. - static ChainRecordYielder* New( - const std::vector& yielder_options); - - protected: - ChainRecordYielder( - const std::vector& yielder_options); - - private: - mutable absl::Mutex mu_; - int current_yielder_idx_ ABSL_GUARDED_BY(mu_); - BasicRecordYielder* current_yielder_ ABSL_GUARDED_BY(mu_); - - std::vector yielder_options_; -}; - -} // namespace lingvo -} // namespace tensorflow -#endif // LINGVO_CORE_OPS_CHAIN_RECORD_YIELDER_H_ diff --git a/lingvo/core/ops/chain_record_yielder_test.cc b/lingvo/core/ops/chain_record_yielder_test.cc deleted file mode 100644 index eee88191..00000000 --- a/lingvo/core/ops/chain_record_yielder_test.cc +++ /dev/null @@ -1,95 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/chain_record_yielder.h" - -#include - -#include -#include -#include "lingvo/core/ops/input_common.h" -#include "lingvo/core/ops/record_yielder.h" -#include "lingvo/core/ops/yielder_test_helper.h" -#include "tensorflow/core/lib/core/stringpiece.h" -#include "tensorflow/core/lib/io/path.h" -#include "tensorflow/core/lib/strings/stringprintf.h" -#include "tensorflow/core/platform/env.h" - -namespace tensorflow { -namespace lingvo { - -TEST(RecordYielderTest, ChainBasicTest) { - const int N = 10; - const int M = 1000; - GeneratePlainTextTestData("chain_yielder1", N, M); - GeneratePlainTextTestData("chain_yielder2", N, M); - - BasicRecordYielder::Options opts1; - opts1.file_pattern = - strings::StrCat("text:", io::JoinPath("/tmp", - "chain_yielder1.*")); - opts1.seed = 301; - opts1.bufsize = 2000; - opts1.parallelism = 1; - - BasicRecordYielder::Options opts2; - opts2.file_pattern = - strings::StrCat("text:", io::JoinPath("/tmp", - "chain_yielder2.*")); - opts2.seed = 301; - opts2.bufsize = 2000; - opts2.parallelism = 1; - ChainRecordYielder* yielder = ChainRecordYielder::New({opts1, opts2}); - - std::vector vals; - Record record; - record.source_id = kDefaultSourceId; - // Consume first yielder entirely. - for (int i = 0; i < N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - VLOG(1) << i << " " << record.value; - vals.emplace_back(string(record.value)); - } - ASSERT_NEAR(ComputeInputSourceDistribution(vals)["chain_yielder1"], - 1.0, 0.001); - vals.clear(); - // Consume second yielder entirely. - for (int i = 0; i < N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - VLOG(1) << i << " " << record.value; - vals.emplace_back(string(record.value)); - } - ASSERT_NEAR(ComputeInputSourceDistribution(vals)["chain_yielder2"], - 1.0, 0.001); - vals.clear(); - // Consume first yielder again. - for (int i = 0; i < N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - VLOG(1) << i << " " << record.value; - vals.emplace_back(string(record.value)); - } - ASSERT_NEAR(ComputeInputSourceDistribution(vals)["chain_yielder1"], - 1.0, 0.001); - - yielder->Close(); -} - -TEST(RecordYielderDeathTest, ChainNoYielders) { - ASSERT_DEATH(ChainRecordYielder::New({}), - "There should be at least one set of options provided"); -} - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/functional_ops_kernels.cc b/lingvo/core/ops/functional_ops_kernels.cc deleted file mode 100644 index bdb4e8ff..00000000 --- a/lingvo/core/ops/functional_ops_kernels.cc +++ /dev/null @@ -1,127 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include - -#include "absl/synchronization/mutex.h" -#include "tensorflow/core/common_runtime/function.h" -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" -#include "tensorflow/core/platform/thread_annotations.h" - -namespace tensorflow { -namespace lingvo { -namespace { - -typedef FunctionLibraryRuntime::Handle FHandle; - -void SetRunOptions(OpKernelContext* ctx, FunctionLibraryRuntime::Options* opts, - bool always_collect_stats) { - opts->rendezvous = ctx->rendezvous(); - opts->cancellation_manager = ctx->cancellation_manager(); - if (always_collect_stats) { - opts->stats_collector = ctx->stats_collector(); - } - opts->runner = ctx->runner(); -} - -class CachedCallOp : public AsyncOpKernel { - public: - explicit CachedCallOp(OpKernelConstruction* ctx) - : AsyncOpKernel(ctx), not_initing_(this, &ME::NotIniting) { - flib_ = ctx->function_library(); - OP_REQUIRES(ctx, flib_ != nullptr, errors::Internal("No function library")); - const NameAttrList* func; - OP_REQUIRES_OK(ctx, ctx->GetAttr("f", &func)); - OP_REQUIRES_OK(ctx, flib_->Instantiate(func->name(), - AttrSlice(&func->attr()), &handle_)); - } - - ~CachedCallOp() override {} - - void ComputeAsync(OpKernelContext* ctx, DoneCallback done) override { - mu_.Lock(); - - while (true) { - // First call. - if (state_ == UNINIT) { - break; - } - - // Has called f and cached the result. - if (state_ == INITED) { - ctx->SetStatus(status_); - for (int i = 0; i < rets_.size(); ++i) { - ctx->set_output(i, rets_[i]); - } - mu_.Unlock(); - done(); - return; - } - - // Another call is being executed. - mu_.Await(not_initing_); - } - - state_ = INITING; - mu_.Unlock(); - - // Call f once and cache the result. - SetRunOptions(ctx, &opts_, true /* always_collect_stats */); - flib_->Run(opts_, handle_, args_, &rets_, - // Done callback - [this, ctx, done](Status s) { - ctx->SetStatus(s); - for (int i = 0; i < rets_.size(); ++i) { - ctx->set_output(i, rets_[i]); - } - done(); - - absl::MutexLock l(&mu_); - status_ = s; - state_ = INITED; - }); - } - - private: - typedef CachedCallOp ME; - - bool NotIniting() const ABSL_SHARED_LOCKS_REQUIRED(mu_) { - return state_ != INITING; - } - - FunctionLibraryRuntime* flib_ = nullptr; - FunctionLibraryRuntime::Options opts_; - FHandle handle_; - - absl::Mutex mu_; - absl::Condition not_initing_; - enum State { - UNINIT, - INITING, - INITED, - }; - State state_ = UNINIT; - Status status_; - std::vector args_; - std::vector rets_; -}; - -REGISTER_KERNEL_BUILDER(Name("CachedCall").Device(DEVICE_CPU), CachedCallOp); - -} // namespace -} // end namespace lingvo -} // end namespace tensorflow diff --git a/lingvo/core/ops/functional_ops_test.py b/lingvo/core/ops/functional_ops_test.py deleted file mode 100644 index 23449ac1..00000000 --- a/lingvo/core/ops/functional_ops_test.py +++ /dev/null @@ -1,55 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for functional ops.""" - -from lingvo import compat as tf -from lingvo.core import ops -from lingvo.core import test_utils -import numpy as np - - -class FunctionalOpsTest(test_utils.TestCase): - - def testCachedCall(self): - # A generator returns different values for each invocation. - def Gen(): - for i in range(1, 1000): - yield np.array([[0, i], [i, 0]]).astype(np.float32), np.array( - [[i, 0], [0, -i]]).astype(np.float32) - - it = Gen() - - # Wraps gen() in a defun. - @tf.function(autograph=False) - def MyFn(): - return tf.py_func(lambda: next(it), [], [tf.float32, tf.float32]) - - # A graph calls MyFn via CachedCall. - g = tf.Graph() - with g.as_default(): - fn = MyFn.get_concrete_function() - fn.add_to_graph() - u, v = ops.cached_call(fn, [tf.float32, tf.float32]) - - with self.session(graph=g): - for _ in range(10): - x, y = self.evaluate([u, v]) - self.assertAllEqual(x, [[0, 1], [1, 0]]) - self.assertAllEqual(y, [[1, 0], [0, -1]]) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/ops/generic_input_op_kernels.cc b/lingvo/core/ops/generic_input_op_kernels.cc deleted file mode 100644 index 7143b90f..00000000 --- a/lingvo/core/ops/generic_input_op_kernels.cc +++ /dev/null @@ -1,317 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include - -#include "tensorflow/core/lib/strings/strcat.h" -#include "lingvo/core/ops/input_common.h" -#include "tensorflow/core/common_runtime/function.h" -#include "tensorflow/core/framework/function.h" -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" -#include "tensorflow/core/framework/types.h" -#include "tensorflow/core/lib/core/threadpool.h" -#include "tensorflow/core/platform/env.h" -#include "tensorflow/core/util/work_sharder.h" - -namespace tensorflow { -namespace lingvo { -namespace { - -typedef std::function Closure; -typedef std::function Runner; - -// ThreadLocalRunner::PerThread() is a thread local object which owns a thread -// pool with one thread. That thread is configured to disable as much -// TensorFlow runtime parallelism as we can. -// -// NOTE: Maybe a cpu-local object will work better, and the thread in -// ThreadLocalRunner can be affined to one cpu. -class ThreadLocalRunner { - public: - static ThreadLocalRunner& PerThread() { - thread_local ThreadLocalRunner tl_runner; - return tl_runner; - } - - ThreadLocalRunner() : pool_(Env::Default(), "single", 1) { - runner_ = [this](Closure c) { pool_.Schedule(Wrapper(c)); }; - } - - Runner* runner() { return &runner_; } - - private: - thread::ThreadPool pool_; - Runner runner_; - - class Wrapper : Closure { - public: - explicit Wrapper(Closure c) : c_(std::move(c)) {} - - void operator()() const { - ScopedPerThreadMaxParallelism scope(1); - c_(); - } - - private: - Closure c_; - }; -}; - -class GenericInputProcessor : public RecordProcessor { - public: - explicit GenericInputProcessor(OpKernelConstruction* ctx) { - auto flib = ctx->function_library(); - OP_REQUIRES(ctx, flib != nullptr, errors::Internal("No function library")); - OP_REQUIRES_OK(ctx, flib->Clone(&fld_, &pflr_, &flib_)); - const NameAttrList* func; - OP_REQUIRES_OK(ctx, ctx->GetAttr("processor", &func)); - OP_REQUIRES_OK(ctx, flib_->Instantiate(func->name(), - AttrSlice(&func->attr()), &handle_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("num_threads", &num_merger_threads_)); - num_merger_threads_ = std::max(4, num_merger_threads_ / 4); // An estimate. - merger_ = new thread::ThreadPool( - Env::Default(), ThreadOptions(), "generic_input_merger", - num_merger_threads_, /* low_latency_hint */ false); - merger_runner_ = [this](Closure c) { merger_->Schedule(c); }; - OP_REQUIRES_OK(ctx, ctx->GetAttr("dynamic_padding_dimensions", - &dynamic_padding_dimensions_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("dynamic_padding_constants", - &dynamic_padding_constants_)); - } - - ~GenericInputProcessor() { delete merger_; } - - Status Process(const Record& record, int64* bucket_key, - TensorVec* sample) override { - // We expect that this input processor is used in conjunction with - // RecordBatcher, which uses multiple threads to call this input - // processor's Process(). Therefore, there is not much need for - // processing each individual record using multiple threads - // (tf_compute). - FunctionLibraryRuntime::Options opts; - // Create a step container that uses resource manager to cleanup state - // after the step is complete. - ScopedStepContainer step_container( - step_id_counter_.fetch_add(1), - [this](const string& name) { - auto status = flib_->device()->resource_manager()->Cleanup(name); - if (!status.ok()) { - LOG(ERROR) << "Error cleaning up resources:" << status; - } - }, - "GenericInputProcessor"); - opts.step_container = &step_container; - opts.runner = ThreadLocalRunner::PerThread().runner(); - - // Generates pair as the resulting Tensors. - TensorVec args(2); - args[0] = Tensor(DT_INT32, {}); - args[0].scalar()() = record.source_id; - args[1] = Tensor(DT_STRING, {}); - args[1].scalar()().append(std::string(record.value)); - *bucket_key = 1; - sample->clear(); - Status status; - Notification done; - flib_->Run(opts, handle_, args, sample, [&](const Status& s) { - status = s; - done.Notify(); - }); - done.WaitForNotification(); - TF_RETURN_IF_ERROR(status); - if (sample->size() < 2) { - LOG(FATAL) - << "Generic input processor must return at least 2 tensors. but got " - << sample->size(); - } - const auto& bucket_key_tensor = (*sample)[sample->size() - 1]; - if (bucket_key_tensor.dtype() != DT_INT32 || - !TensorShapeUtils::IsScalar(bucket_key_tensor.shape())) { - LOG(FATAL) << "Bucket key tensor is not an int32 scalar: " - << DataTypeString(bucket_key_tensor.dtype()); - } - *bucket_key = bucket_key_tensor.scalar()(); - if (*bucket_key < 0) { - return tensorflow::errors::Cancelled( - strings::StrCat("Batch has negative bucket key: ", *bucket_key)); - } - sample->pop_back(); - return Status::OK(); - } - - Status Merge(int64 bucket_size, const std::vector& samples, - TensorVec* batch) override { - CHECK(!samples.empty()); - const auto num_samples = samples.size(); - const auto num_outs = samples[0].size(); - - std::vector padded_samples(samples.begin(), samples.end()); - if (!dynamic_padding_dimensions_.empty()) { - CHECK(dynamic_padding_dimensions_.size() == num_outs); - CHECK(dynamic_padding_constants_.size() == num_outs); - - for (int j = 0; j < num_outs; ++j) { - const int pad_dim = dynamic_padding_dimensions_[j]; - if (pad_dim == -1) { - continue; - } - const int pad_value = dynamic_padding_constants_[j]; - - int64 max_length = 0; - for (int i = 0; i < samples.size(); ++i) { - max_length = std::max(max_length, samples[i][j].dim_size(pad_dim)); - } - - for (int i = 0; i < samples.size(); ++i) { - const auto& src = samples[i][j]; - if (src.dims() > 0 && src.dim_size(pad_dim) < max_length) { - DataType dtype = src.dtype(); - TensorShape dst_shape(src.shape()); - dst_shape.set_dim(pad_dim, max_length); - Tensor dst(dtype, dst_shape); - switch (dtype) { -#define CASE(T) \ - case DataTypeToEnum::value: \ - dst.flat().setConstant(pad_value); \ - if (src.NumElements() > 0) { \ - auto src_t = src.flat_inner_outer_dims(pad_dim - 1); \ - auto dst_t = dst.flat_inner_outer_dims(pad_dim - 1); \ - typedef Eigen::DSizes DSizes; \ - dst_t.slice(DSizes(), DSizes(src_t.dimensions())) = src_t; \ - } \ - break - - CASE(float); - CASE(int32); - CASE(int64); -#undef CASE - default: - LOG(FATAL) << "Unexpected " << DataTypeString(dtype); - } - std::swap(padded_samples[i][j], dst); - } - } - } - } - - // Validate that samples can be merged: samples[:][i] has the same - // type and shape. - for (int i = 1; i < padded_samples.size(); ++i) { - if (padded_samples[i].size() != num_outs) { - LOG(FATAL) << "Samples have different sizes: " << samples[i].size() - << " vs. " << num_outs; - } - for (int j = 0; j < num_outs; ++j) { - if (padded_samples[i][j].dtype() != padded_samples[0][j].dtype()) { - LOG(FATAL) << "Mismatch data types of samples (" << i << "/" << j - << "): " << samples[i][j].dtype() << " vs. " - << samples[0][j].dtype(); - } - if (padded_samples[i][j].shape() != padded_samples[0][j].shape()) { - LOG(FATAL) << "Mismatch shape of samples (" << i << "/" << j - << "): " << samples[i][j].shape().DebugString() << " vs. " - << samples[0][j].shape().DebugString(); - } - } - } - - batch->clear(); - for (int i = 0; i < num_outs; ++i) { - const Tensor& src = padded_samples[0][i]; - DataType dtype = src.dtype(); - switch (dtype) { - case DT_FLOAT: - case DT_UINT8: - case DT_INT32: - case DT_INT64: - case DT_STRING: - case DT_BFLOAT16: - case DT_COMPLEX64: - case DT_COMPLEX128: - case DT_BOOL: - break; - default: - LOG(FATAL) << DataTypeString(dtype) << " is not supported."; - } - TensorShape shape = src.shape(); - shape.InsertDim(0, num_samples); - // The merged tensor is 1-rank higher and its 1st dimension - // is the num_samples. - if (num_samples == 1) { - // Avoid memcpy if there is just one sample. - Tensor reshaped(dtype); - CHECK(reshaped.CopyFrom(src, shape)); - batch->push_back(reshaped); - } else { - batch->push_back(Tensor(dtype, shape)); - } - } - // If there is just one sample, 'batch' already has the copy. - if (num_samples == 1) return Status::OK(); - - Sharder::Do(num_samples /* total */, 1000 /* cost_per_unit */, - [&](int64 start, int64 limit) { - for (int i = 0; i < num_outs; ++i) { - DataType dtype = padded_samples[0][i].dtype(); - Tensor* merged = &(*batch)[i]; - for (int j = start; j < limit; ++j) { - switch (dtype) { -#define CASE(T) \ - case DataTypeToEnum::value: \ - merged->flat_outer_dims().chip<0>(j) = padded_samples[j][i].flat(); \ - break - CASE(float); - CASE(int32); - CASE(int64); - CASE(tstring); - CASE(uint8); - CASE(bfloat16); - CASE(complex64); - CASE(complex128); - CASE(bool); -#undef CASE - default: - LOG(FATAL) << "Unexpected " << DataTypeString(dtype); - } - } - } - }, - merger_runner_, 1 + num_merger_threads_); - return Status::OK(); - } - - private: - std::unique_ptr fld_; - std::unique_ptr pflr_; - FunctionLibraryRuntime* flib_ = nullptr; // Not owned. - FunctionLibraryRuntime::Handle handle_; - std::atomic_int_fast64_t step_id_counter_; - - int num_merger_threads_ = -1; - thread::ThreadPool* merger_ = nullptr; - Runner merger_runner_; - std::vector dynamic_padding_dimensions_; - std::vector dynamic_padding_constants_; - - TF_DISALLOW_COPY_AND_ASSIGN(GenericInputProcessor); -}; - -REGISTER_KERNEL_BUILDER(Name("GenericInput").Device(DEVICE_CPU), - InputOp); -} // namespace -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/hyps.proto b/lingvo/core/ops/hyps.proto deleted file mode 100644 index d34da4f9..00000000 --- a/lingvo/core/ops/hyps.proto +++ /dev/null @@ -1,31 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -syntax = "proto2"; - -package tensorflow.lingvo; - -message Hypothesis { - // The beam to which this hyp belongs. - optional int32 beam_id = 1; - repeated int32 ids = 2; - repeated float scores = 3; - - message AttenVec { - repeated float prob = 1; - } - repeated AttenVec atten_vecs = 4; - optional float normalized_score = 5; -} diff --git a/lingvo/core/ops/input_common.cc b/lingvo/core/ops/input_common.cc deleted file mode 100644 index 9b99a1ec..00000000 --- a/lingvo/core/ops/input_common.cc +++ /dev/null @@ -1,95 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/input_common.h" - -#include "lingvo/core/ops/sequential_record_yielder.h" -#include "lingvo/core/ops/chain_record_yielder.h" -#include "lingvo/core/ops/weighted_mix_record_yielder.h" - -namespace tensorflow { -namespace lingvo { - -RecordYielder* ConstructYielder(const string& file_pattern, - const std::vector& input_source_weights, - int64 file_random_seed, int64 file_buffer_size, - int64 file_buffer_size_in_seconds, - int64 file_parallelism, - bool require_sequential_order, - int64 repeat_count, bool use_chaining) { - std::vector file_patterns; - if (input_source_weights.empty()) { - LOG(INFO) << "Input source weights are empty, fall back to legacy " - << "behavior."; - file_patterns.push_back(file_pattern); - } else { - file_patterns = str_util::Split(file_pattern, ','); - CHECK_EQ(file_patterns.size(), input_source_weights.size()) - << "There should be exactly one " - << "input_source_weight per coma-separated value " - << "in file_pattern."; - } - if (require_sequential_order) { - CHECK_EQ(file_patterns.size(), 1) - << "require_sequential_order does not support record mixing or " - << "chaining."; - return SequentialRecordYielder::New(file_patterns.front(), repeat_count); - } else { - CHECK_EQ(repeat_count, -1) << "Repeat count must not be set unless " - "require_sequential_order is true."; - } - std::vector yielder_options; - - for (int i = 0; i < file_patterns.size(); ++i) { - BasicRecordYielder::Options yopts; - yopts.file_pattern = file_patterns[i]; - if (file_random_seed == 0) { - yopts.seed = 0; // Let the yielder pick a random seed. - } else { - yopts.seed = - (file_random_seed + i) % (std::numeric_limits::max() - 1); - if (yopts.seed == 0) { - // Add 1 to avoid 0. - ++yopts.seed; - } - } - yopts.bufsize = file_buffer_size; - yopts.bufsize_in_seconds = file_buffer_size_in_seconds; - yopts.parallelism = file_parallelism; - yopts.source_id = i; - yielder_options.push_back(yopts); - } - - RecordYielder* yielder = nullptr; - if (yielder_options.size() == 1) { - yielder = BasicRecordYielder::New(yielder_options.front()); - } else if (use_chaining) { - yielder = ChainRecordYielder::New(yielder_options); - } else { - std::vector yielders; - yielders.reserve(yielder_options.size()); - for (const auto& yopts : yielder_options) { - yielders.push_back(BasicRecordYielder::New(yopts)); - } - yielder = WeightedMixRecordYielder::New( - file_random_seed, - yielders, - input_source_weights); - } - return yielder; -} - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/input_common.h b/lingvo/core/ops/input_common.h deleted file mode 100644 index cd16ecf9..00000000 --- a/lingvo/core/ops/input_common.h +++ /dev/null @@ -1,116 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#ifndef LINGVO_CORE_OPS_INPUT_COMMON_H_ -#define LINGVO_CORE_OPS_INPUT_COMMON_H_ - -#include - -#include "lingvo/core/ops/record_batcher.h" -#include "lingvo/core/ops/record_yielder.h" -#include "tensorflow/core/framework/op_kernel.h" - -namespace tensorflow { -namespace lingvo { - -// Constructs single Yielder for a given file pattern or mixes multiple yielders -// with weights. -RecordYielder* ConstructYielder(const string& file_pattern, - const std::vector& input_source_weights, - int64 file_random_seed, int64 file_buffer_size, - int64 file_buffer_size_in_seconds, - int64 file_parallelism, - bool require_sequential_order, - int64 repeat_count, bool use_chaining); - -// Base class for op kernels that emit training examples. -template -class InputOp : public OpKernel { - static_assert( - std::is_base_of::value, - "InputOp requires a RecordProcessor subclass as the template arg."); - - public: - explicit InputOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - typedef std::vector Int64Vec; -#define GETATTR(TYPE, FIELD) \ - TYPE FIELD; \ - OP_REQUIRES_OK(ctx, ctx->GetAttr(#FIELD, &FIELD)); - - GETATTR(string, file_pattern); - GETATTR(std::vector, input_source_weights); - GETATTR(int64, file_random_seed); - GETATTR(int64, file_buffer_size); - GETATTR(int64, file_buffer_size_in_seconds); - GETATTR(int64, file_parallelism); - GETATTR(Int64Vec, bucket_upper_bound); - GETATTR(Int64Vec, bucket_batch_limit); - GETATTR(int64, bucket_adjust_every_n); - GETATTR(int64, flush_every_n); - GETATTR(int64, num_threads); - GETATTR(bool, require_sequential_order); - GETATTR(int64, repeat_count); - GETATTR(bool, use_chaining); -#undef GETATTR - OP_REQUIRES( - ctx, - std::is_sorted(bucket_upper_bound.begin(), bucket_upper_bound.end()), - errors::InvalidArgument("Bucket_upper_bound is not sorted")); - if (require_sequential_order) { - num_threads = 1; - } - LOG(INFO) << "Create RecordProcessor"; - processor_ = new RecordProcessorClass(ctx); - RecordYielder* yielder = CHECK_NOTNULL(ConstructYielder( - file_pattern, input_source_weights, file_random_seed, file_buffer_size, - file_buffer_size_in_seconds, file_parallelism, require_sequential_order, - repeat_count, use_chaining)); - LOG(INFO) << "Create batcher"; - RecordBatcher::Options bopts; - bopts.bucket_upper_bound = bucket_upper_bound; - bopts.bucket_batch_limit = bucket_batch_limit; - bopts.bucket_adjust_every_n = bucket_adjust_every_n; - bopts.flush_every_n = flush_every_n; - bopts.num_threads = num_threads; - batcher_ = new RecordBatcher(bopts, yielder, processor_); - } - - ~InputOp() override { delete batcher_; } - - void Compute(OpKernelContext* ctx) override { - int64 bucket_id; - TensorVec batch; - OP_REQUIRES_OK(ctx, batcher_->GetNext(ctx, &bucket_id, &batch)); - VLOG(1) << "Produce a batch from bucket : " << bucket_id; - OP_REQUIRES(ctx, static_cast(batch.size()) == ctx->num_outputs(), - errors::Internal("Unexpected batch: ", batch.size())); - for (int i = 0; i < batch.size(); ++i) { - ctx->set_output(i, batch[i]); - } - } - - protected: - // Not owned. - RecordProcessorClass* processor_ = nullptr; - - private: - // Owned. - RecordBatcher* batcher_ = nullptr; -}; - -} // namespace lingvo -} // namespace tensorflow - -#endif // LINGVO_CORE_OPS_INPUT_COMMON_H_ diff --git a/lingvo/core/ops/mass_op.cc b/lingvo/core/ops/mass_op.cc deleted file mode 100644 index c246c962..00000000 --- a/lingvo/core/ops/mass_op.cc +++ /dev/null @@ -1,256 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include -#include -#include - -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" -#include "tensorflow/core/lib/core/errors.h" - -namespace tensorflow { -namespace lingvo { -namespace { - -static const int kBOS = 1; -static const int kEOS = 2; - -class MassOp : public OpKernel { - public: - explicit MassOp(OpKernelConstruction* ctx); - void Compute(OpKernelContext* ctx) override; - - private: - float mask_ratio_; - int mask_minlen_; - int mask_id_; - int span_len_; - float random_start_prob_; - float keep_prob_; - float rand_prob_; - float mask_prob_; - bool mask_target_; - int vocab_size_; - int first_unreserved_id_; - - std::mt19937 rng_; - - // Populates a vector of length seq_len with 1's in positions where - // the corresponding token will be masked, and 0's elsewhere. - void GenerateMask(std::vector* mask); - void ValidateInput(OpKernelContext* ctx, const Tensor& ids, - const Tensor& weights, const Tensor& actual_seq_len); - - template - void CopyTensorToMutableOutput(const TensorShape& shape, const string& name, - const Tensor& source, OpKernelContext* ctx, - Tensor** out) { - OP_REQUIRES_OK(ctx, ctx->allocate_output(name, shape, out)); - (*out)->flat() = source.flat(); - } -}; - -MassOp::MassOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, ctx->GetAttr("mask_ratio", &mask_ratio_)); - CHECK_GT(mask_ratio_, 0); - CHECK_LT(mask_ratio_, 1); - OP_REQUIRES_OK(ctx, ctx->GetAttr("mask_minlen", &mask_minlen_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("mask_id", &mask_id_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("span_len", &span_len_)); - CHECK_GT(span_len_, 0); - OP_REQUIRES_OK(ctx, ctx->GetAttr("random_start_prob", &random_start_prob_)); - CHECK_GE(random_start_prob_, 0); - CHECK_LE(random_start_prob_, 1); - OP_REQUIRES_OK(ctx, ctx->GetAttr("keep_prob", &keep_prob_)); - CHECK_GE(keep_prob_, 0); - CHECK_LE(keep_prob_, 1); - OP_REQUIRES_OK(ctx, ctx->GetAttr("rand_prob", &rand_prob_)); - CHECK_GE(rand_prob_, 0); - CHECK_LE(rand_prob_, 1); - OP_REQUIRES_OK(ctx, ctx->GetAttr("mask_prob", &mask_prob_)); - CHECK_GE(mask_prob_, 0); - CHECK_LE(mask_prob_, 1); - CHECK_EQ(keep_prob_ + rand_prob_ + mask_prob_, 1); - OP_REQUIRES_OK(ctx, ctx->GetAttr("mask_target", &mask_target_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("vocab_size", &vocab_size_)); - OP_REQUIRES_OK(ctx, - ctx->GetAttr("first_unreserved_id", &first_unreserved_id_)); - - rng_.seed(7743); // seed the random generator -} - -void MassOp::ValidateInput(OpKernelContext* ctx, const Tensor& ids, - const Tensor& weights, - const Tensor& actual_seq_len) { - // Verify shapes and sizes - OP_REQUIRES(ctx, TensorShapeUtils::IsMatrix(ids.shape()), - errors::InvalidArgument("ids must be matrix, but got ", - ids.shape().DebugString())); - OP_REQUIRES(ctx, TensorShapeUtils::IsMatrix(weights.shape()), - errors::InvalidArgument("weights must be matrix, but got ", - weights.shape().DebugString())); - OP_REQUIRES(ctx, TensorShapeUtils::IsVector(actual_seq_len.shape()), - errors::InvalidArgument("actual_seq_len must be vector, but got ", - actual_seq_len.shape().DebugString())); - OP_REQUIRES(ctx, ids.dim_size(0) > 0, - errors::InvalidArgument("batch size must be > 0")); - OP_REQUIRES(ctx, ids.dim_size(1) > 0, - errors::InvalidArgument("max seq length must be > 0")); - OP_REQUIRES(ctx, ids.dim_size(0) == weights.dim_size(0), - errors::InvalidArgument("inconsistent batch size")); - OP_REQUIRES(ctx, ids.dim_size(0) == actual_seq_len.dim_size(0), - errors::InvalidArgument("inconsistent batch size")); - OP_REQUIRES(ctx, ids.dim_size(1) == weights.dim_size(1), - errors::InvalidArgument("inconsistent seq length")); -} - -void MassOp::GenerateMask(std::vector* mask) { - int seq_len = mask->size(); - int mask_len = seq_len * mask_ratio_; - - // skip short sentences - if (mask_len < mask_minlen_) return; - - int num_segments = std::ceil(static_cast(mask_len) / span_len_); - int non_mask_len = seq_len - mask_len; - // segments = [span_len, span_len, ..., span_len] - std::vector segments(num_segments + non_mask_len, span_len_); - // if there is a remainder, last segment will be a different length - if (mask_len % span_len_ > 0) { - // now segments are [span_len, span_len, ..., remaining_mask_len] - segments.back() = mask_len % span_len_; - } - // non-masked segments of length 1 are represented by 0 - // now segments are [0, 0, ..., span_len, span_len, ... remaining_mask_len] - std::fill(segments.begin(), segments.begin() + non_mask_len, 0); - std::uniform_real_distribution<> realdistr(0.0, 1.0); - float px = realdistr(rng_); - float fixed_start_prob = 1 - random_start_prob_; - if (px >= random_start_prob_ + fixed_start_prob / 2) { - // place a segment at position 0, shuffle the rest - std::swap(segments.front(), segments.back()); - if (num_segments > 1) { - std::shuffle(segments.begin() + 1, segments.end(), rng_); - } - } else if (px >= random_start_prob_) { - // keep a segment at final position, shuffle the rest - if (num_segments > 1) { - std::shuffle(segments.begin(), segments.end() - 1, rng_); - } - } else { - // shuffle all segments - std::shuffle(segments.begin(), segments.end(), rng_); - } - int idx = 0; - for (int n : segments) { - if (n == 0) { - // Each 0 in segments represents a non-masked token; i.e. mask will be 0 - (*mask)[idx++] = 0; - } else { - // Each n in segments represents a masked segment of length n tokens - for (int i = 0; i < n; i++) { - (*mask)[idx++] = 1; - } - } - } -} - -void MassOp::Compute(OpKernelContext* ctx) { - // get inputs - const Tensor& ids = ctx->input(0); - const Tensor& weights = ctx->input(1); - const Tensor& actual_seq_len = ctx->input(2); - auto Tactual_seq_len = actual_seq_len.vec(); - ValidateInput(ctx, ids, weights, actual_seq_len); - OP_REQUIRES_OK(ctx, ctx->status()); - - int batch_size = ids.dim_size(0); - int max_seq_len = ids.dim_size(1); - - // create outputs - Tensor* src_ids; - Tensor* tgt_ids; - Tensor* tgt_weights; - Tensor* tgt_labels; - CopyTensorToMutableOutput(TensorShape({batch_size, max_seq_len}), - "src_ids", ids, ctx, &src_ids); - CopyTensorToMutableOutput(TensorShape({batch_size, max_seq_len}), - "tgt_ids", ids, ctx, &tgt_ids); - CopyTensorToMutableOutput(TensorShape({batch_size, max_seq_len}), - "tgt_weights", weights, ctx, &tgt_weights); - CopyTensorToMutableOutput(TensorShape({batch_size, max_seq_len}), - "tgt_labels", ids, ctx, &tgt_labels); - auto Tsrc_ids = src_ids->matrix(); - auto Ttgt_ids = tgt_ids->matrix(); - auto Ttgt_weights = tgt_weights->matrix(); - - // for each example, mask the source and target to implement MASS. - for (int i = 0; i < batch_size; i++) { - int seq_len = Tactual_seq_len(i); - if (seq_len <= 0) { - LOG(WARNING) << "Skipping zero-length sequence"; - continue; - } - if (seq_len > max_seq_len) seq_len = max_seq_len; - std::vector mask(seq_len, 0); - GenerateMask(&mask); - - // Right shift tgt and drop EOS - for (int j = max_seq_len - 1; j > 0; j--) { - if (Ttgt_ids(i, j - 1) == kEOS) { - Ttgt_ids(i, j) = 0; - Ttgt_weights(i, j) = 0.0; - } else { - Ttgt_ids(i, j) = Ttgt_ids(i, j - 1); - Ttgt_weights(i, j) = Ttgt_weights(i, j - 1); - } - } - Ttgt_ids(i, 0) = kBOS; - Ttgt_weights(i, 0) = 1.0; - - std::uniform_int_distribution<> vocabdistr(first_unreserved_id_, - vocab_size_); - - // mask source and target tokens. - if (std::accumulate(mask.begin(), mask.end(), 0) > 0) { - for (int j = 0; j < mask.size(); j++) { - if (mask[j] == 1) { - std::uniform_real_distribution<> realdistr(0.0, 1.0); - float tx = realdistr(rng_); - if (tx >= mask_prob_ + rand_prob_) { - // Keep token - } else if (tx >= mask_prob_) { - Tsrc_ids(i, j) = vocabdistr(rng_); // replace with random token - } else { - Tsrc_ids(i, j) = mask_id_; // replace with mask id - } - } else if (mask_target_) { - // either source is masked or target is masked. - Ttgt_ids(i, j) = mask_id_; - // where target is masked, target weights are zero - Ttgt_weights(i, j) = 0; - } - } - } - } -} - -REGISTER_KERNEL_BUILDER(Name("Mass").Device(DEVICE_CPU), MassOp); - -} // namespace -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/mass_op_test.py b/lingvo/core/ops/mass_op_test.py deleted file mode 100644 index 712b6855..00000000 --- a/lingvo/core/ops/mass_op_test.py +++ /dev/null @@ -1,929 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for mass_op.""" - -import collections -from lingvo import compat as tf -from lingvo.core import ops -from lingvo.core import test_utils -import numpy as np - -FLAGS = tf.flags.FLAGS - -BOS = 1 -EOS = 2 - -MassOutput = collections.namedtuple( - 'MassOutput', ['src_ids', 'tgt_ids', 'tgt_labels', 'tgt_weights']) - -# Set default to empty. -MassOutput.__new__.__defaults__ = (None,) * len(MassOutput._fields) - - -def FindResultFromList(result, expected_results): - """Find the given result from a list of expected results. - - Args: - result: A MassOutput tuple, from running ops.mass(). - expected_results: A list of MassOutput. The test asserts `result` is equal - to at least one result from `expected_results`. - - Returns: - The index of first match found, or None for not found. - - We use this when the specific output from ops.mass() is not stable across - different platforms. Specifically, the implementation currently uses - std::shuffle(), which have different implementations between libc++ and - stdlibc++. - """ - for idx, expected in enumerate(expected_results): - match = True - for attr in MassOutput._fields: - if not np.array_equal(getattr(result, attr), getattr(expected, attr)): - match = False - break - if match: - return idx - - tf.logging.error('Found unexpected output from op.mass that fails to match' - ' any expected result.') - for attr in MassOutput._fields: - tf.logging.info('%s = %s', attr, np.array_repr(getattr(result, attr))) - return None - - -class MassOpTest(test_utils.TestCase): - - def testFixedStart(self): - ids = np.array( - [[4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32) - weights = np.array( - [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32) - actual_seq_len = np.array([14, 10, 14, 10], dtype=np.int32) - - g = tf.Graph() - with g.as_default(): - (src_ids, tgt_ids, tgt_labels, tgt_weights) = ops.mass( - ids, - weights, - actual_seq_len, - mask_id=3, - mask_ratio=0.5, - mask_minlen=1, - span_len=8, - random_start_prob=0, - keep_prob=0, - rand_prob=0, - mask_prob=1, - mask_target=True, - vocab_size=9) - - with self.session(graph=g) as sess: - (src_ids, tgt_ids, tgt_labels, - tgt_weights) = sess.run([src_ids, tgt_ids, tgt_labels, tgt_weights]) - self.assertAllEqual( - src_ids, - np.array([[ - 3, 3, 3, 3, 3, 3, 3, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, - 0 - ], [ - 3, 3, 3, 3, 3, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [4, 5, 6, 7, 8, 9, 10, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0 - ], [4, 5, 6, 7, 8, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32)) - self.assertAllEqual( - tgt_ids, - np.array([ - [BOS, 4, 5, 6, 7, 8, 9, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0], - [BOS, 4, 5, 6, 7, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [ - 3, 3, 3, 3, 3, 3, 3, 10, 11, 12, 13, 14, 15, 16, 0, 0, 0, 0, - 0, 0 - ], - [3, 3, 3, 3, 3, 8, 9, 10, 11, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] - ], - dtype=np.int32)) - self.assertAllEqual( - tgt_labels, - np.array([[ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, - 0, 0 - ], [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, - 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0 - ]], - dtype=np.int32)) - self.assertAllEqual( - tgt_weights, - np.array( - [[1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32)) - - def testRandomStart(self): - ids = np.array( - [[4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32) - weights = np.array( - [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32) - actual_seq_len = np.array([14, 10, 14, 10], dtype=np.int32) - - g = tf.Graph() - with g.as_default(): - (src_ids, tgt_ids, tgt_labels, tgt_weights) = ops.mass( - ids, - weights, - actual_seq_len, - mask_id=3, - mask_ratio=0.5, - mask_minlen=1, - span_len=100000, - random_start_prob=1, - keep_prob=0, - rand_prob=0, - mask_prob=1, - mask_target=True, - vocab_size=9) - - with self.session(graph=g) as sess: - (src_ids, tgt_ids, tgt_labels, tgt_weights) = sess.run([ - src_ids, - tgt_ids, - tgt_labels, - tgt_weights, - ]) - result = MassOutput(src_ids, tgt_ids, tgt_labels, tgt_weights) - expected_output1 = MassOutput( - np.array([[ - 4, 5, 6, 7, 8, 9, 10, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0 - ], [ - 4, 3, 3, 3, 3, 3, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [ - 4, 5, 6, 7, 3, 3, 3, 3, 3, 3, 3, 15, 16, EOS, 0, 0, 0, 0, 0, 0 - ], [4, 5, 3, 3, 3, 3, 3, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] - ], - dtype=np.int32), - np.array([[ - 3, 3, 3, 3, 3, 3, 3, 10, 11, 12, 13, 14, 15, 16, 0, 0, 0, 0, 0, - 0 - ], [ - 3, 4, 5, 6, 7, 8, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [3, 3, 3, 3, 7, 8, 9, 10, 11, 12, 13, 3, 3, 3, 0, 0, 0, 0, 0, 0 - ], [3, 3, 5, 6, 7, 8, 9, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32), - np.array([[ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, - 0, 0 - ], [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, - 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0 - ]], - dtype=np.int32), - np.array( - [[0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32)) - - expected_output2 = MassOutput( - np.array([[ - 3, 3, 3, 3, 3, 3, 3, 11, 12, 13, 14, 15, 16, 2, 0, 0, 0, 0, 0, 0 - ], [ - 3, 3, 3, 3, 3, 9, 10, 11, 12, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [ - 4, 5, 6, 7, 3, 3, 3, 3, 3, 3, 3, 15, 16, 2, 0, 0, 0, 0, 0, 0 - ], [4, 5, 3, 3, 3, 3, 3, 11, 12, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32), - np.array([[ - 1, 4, 5, 6, 7, 8, 9, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0 - ], [ - 1, 4, 5, 6, 7, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [3, 3, 3, 3, 7, 8, 9, 10, 11, 12, 13, 3, 3, 3, 0, 0, 0, 0, 0, 0 - ], [3, 3, 5, 6, 7, 8, 9, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32), - np.array([[ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 2, 0, 0, 0, 0, 0, - 0 - ], [4, 5, 6, 7, 8, 9, 10, 11, 12, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 2, 0, 0, - 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 2, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0 - ]], - dtype=np.int32), - np.array( - [[1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32)) - - match_idx = FindResultFromList(result, - [expected_output1, expected_output2]) - self.assertIsNotNone(match_idx, - '{} is not a valid result'.format(result)) - - def testSegmented(self): - ids = np.array( - [[4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32) - weights = np.array( - [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32) - actual_seq_len = np.array([14, 10, 14, 10], dtype=np.int32) - - g = tf.Graph() - with g.as_default(): - (src_ids, tgt_ids, tgt_labels, tgt_weights) = ops.mass( - ids, - weights, - actual_seq_len, - mask_id=3, - mask_ratio=0.5, - mask_minlen=1, - span_len=3, - keep_prob=0, - rand_prob=0, - mask_prob=1, - mask_target=True, - vocab_size=9) - - with self.session(graph=g) as sess: - (src_ids, tgt_ids, tgt_labels, tgt_weights) = sess.run([ - src_ids, - tgt_ids, - tgt_labels, - tgt_weights, - ]) - - result = MassOutput(src_ids, tgt_ids, tgt_labels, tgt_weights) - expected_output1 = MassOutput( - np.array([[ - 4, 3, 3, 3, 3, 3, 3, 11, 12, 13, 14, 15, 16, 3, 0, 0, 0, 0, 0, 0 - ], [ - 4, 3, 3, 7, 8, 9, 10, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [4, 5, 6, 7, 8, 3, 3, 3, 12, 3, 14, 3, 3, 3, 0, 0, 0, 0, 0, 0], - [ - 3, 3, 6, 3, 3, 3, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0 - ]], - dtype=np.int32), - np.array([[ - 3, 4, 5, 6, 7, 8, 9, 3, 3, 3, 3, 3, 3, 16, 0, 0, 0, 0, 0, 0 - ], [ - 3, 4, 5, 3, 3, 3, 3, 10, 11, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [ - 3, 3, 3, 3, 3, 8, 9, 10, 3, 12, 3, 14, 15, 16, 0, 0, 0, 0, 0, 0 - ], [BOS, 4, 3, 6, 7, 8, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32), - np.array([[ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, - 0, 0 - ], [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, - 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0 - ]], - dtype=np.int32), - np.array( - [[0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0], - [0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32)) - - expected_output2 = MassOutput( - np.array([[ - 4, 5, 6, 3, 3, 3, 10, 3, 3, 3, 14, 15, 16, 3, 0, 0, 0, 0, 0, 0 - ], [ - 3, 3, 6, 7, 8, 9, 10, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [4, 3, 3, 3, 8, 9, 10, 11, 12, 3, 3, 3, 16, 3, 0, 0, 0, 0, 0, 0 - ], [4, 5, 3, 3, 8, 9, 3, 3, 3, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32), - np.array([[ - 3, 3, 3, 6, 7, 8, 3, 10, 11, 12, 3, 3, 3, 16, 0, 0, 0, 0, 0, 0 - ], [ - 1, 4, 3, 3, 3, 3, 3, 10, 11, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [ - 3, 4, 5, 6, 3, 3, 3, 3, 3, 12, 13, 14, 3, 16, 0, 0, 0, 0, 0, 0 - ], [3, 3, 5, 6, 3, 3, 9, 10, 11, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32), - np.array([[ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 2, 0, 0, 0, 0, 0, - 0 - ], [4, 5, 6, 7, 8, 9, 10, 11, 12, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 2, 0, 0, - 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 2, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0 - ]], - dtype=np.int32), - np.array([[ - 0., 0., 0., 1., 1., 1., 0., 1., 1., 1., 0., 0., 0., 1., 0., 0., - 0., 0., 0., 0. - ], - [ - 1., 1., 0., 0., 0., 0., 0., 1., 1., 1., 0., 0., 0., - 0., 0., 0., 0., 0., 0., 0. - ], - [ - 0., 1., 1., 1., 0., 0., 0., 0., 0., 1., 1., 1., 0., - 1., 0., 0., 0., 0., 0., 0. - ], - [ - 0., 0., 1., 1., 0., 0., 1., 1., 1., 0., 0., 0., 0., - 0., 0., 0., 0., 0., 0., 0. - ]], - dtype=np.float32)) - match_idx = FindResultFromList(result, - [expected_output1, expected_output2]) - self.assertIsNotNone(match_idx, - '{} is not a valid result'.format(result)) - - def testNoMaskTarget(self): - ids = np.array( - [[4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32) - weights = np.array( - [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32) - actual_seq_len = np.array([14, 10, 14, 10], dtype=np.int32) - - g = tf.Graph() - with g.as_default(): - (src_ids, tgt_ids, tgt_labels, tgt_weights) = ops.mass( - ids, - weights, - actual_seq_len, - mask_id=3, - mask_ratio=0.5, - mask_minlen=1, - span_len=100000, - random_start_prob=0, - keep_prob=0, - rand_prob=0, - mask_prob=1, - mask_target=False, - vocab_size=9) - - with self.session(graph=g) as sess: - (src_ids, tgt_ids, tgt_labels, tgt_weights) = sess.run([ - src_ids, - tgt_ids, - tgt_labels, - tgt_weights, - ]) - - self.assertAllEqual( - src_ids, - np.array([[ - 3, 3, 3, 3, 3, 3, 3, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, - 0 - ], [ - 3, 3, 3, 3, 3, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [4, 5, 6, 7, 8, 9, 10, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0 - ], [4, 5, 6, 7, 8, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32)) - self.assertAllEqual( - tgt_ids, - np.array([[ - 1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 0, 0, 0, 0, 0, - 0 - ], [ - BOS, 4, 5, 6, 7, 8, 9, 10, 11, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], - [ - 1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 0, 0, - 0, 0, 0, 0 - ], - [ - 1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0 - ]], - dtype=np.int32)) - self.assertAllEqual( - tgt_labels, - np.array([[ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, - 0, 0 - ], [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, - 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0 - ]], - dtype=np.int32)) - self.assertAllEqual( - tgt_weights, - np.array( - [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32)) - - def testKeepOrRandMaskedTokens(self): - ids = np.array( - [[4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32) - weights = np.array( - [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32) - actual_seq_len = np.array([14, 10, 14, 10], dtype=np.int32) - - g = tf.Graph() - with g.as_default(): - (src_ids, tgt_ids, tgt_labels, tgt_weights) = ops.mass( - ids, - weights, - actual_seq_len, - mask_id=3, - mask_ratio=0.5, - mask_minlen=1, - span_len=100000, - random_start_prob=0, - keep_prob=0.5, - rand_prob=0.5, - mask_prob=0, - mask_target=True, - vocab_size=9) - - with self.session(graph=g) as sess: - (src_ids, tgt_ids, tgt_labels, tgt_weights) = sess.run([ - src_ids, - tgt_ids, - tgt_labels, - tgt_weights, - ]) - - result = MassOutput(src_ids, tgt_ids, tgt_labels, tgt_weights) - expected_output1 = MassOutput( - np.array([[ - 4, 7, 6, 9, 5, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, - 0, 0 - ], [ - 4, 5, 6, 7, 8, 8, 9, 8, 12, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [ - 4, 5, 6, 7, 8, 9, 10, 7, 12, 5, 7, 15, 16, EOS, 0, 0, 0, 0, 0, 0 - ], [4, 5, 6, 7, 8, 9, 6, 11, 5, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32), - np.array([ - [BOS, 4, 5, 6, 7, 8, 9, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0], - [3, 3, 3, 3, 3, 8, 9, 10, 11, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [ - 3, 3, 3, 3, 3, 3, 3, 10, 11, 12, 13, 14, 15, 16, 0, 0, 0, 0, - 0, 0 - ], - [3, 3, 3, 3, 3, 8, 9, 10, 11, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] - ], - dtype=np.int32), - np.array([[ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, - 0, 0 - ], [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, - 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0 - ]], - dtype=np.int32), - np.array( - [[1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32)) - - expected_output2 = MassOutput( - np.array([[ - 4, 8, 6, 7, 6, 9, 10, 11, 12, 13, 14, 15, 16, 2, 0, 0, 0, 0, 0, - 0 - ], [4, 5, 6, 7, 8, 4, 9, 4, 12, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [ - 4, 5, 6, 7, 8, 9, 10, 5, 7, 6, 4, 15, 16, 2, 0, 0, 0, 0, 0, 0 - ], [ - 4, 5, 6, 7, 8, 9, 6, 11, 12, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ]], - dtype=np.int32), - np.array([ - [1, 4, 5, 6, 7, 8, 9, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0], - [3, 3, 3, 3, 3, 8, 9, 10, 11, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [ - 3, 3, 3, 3, 3, 3, 3, 10, 11, 12, 13, 14, 15, 16, 0, 0, 0, 0, - 0, 0 - ], - [3, 3, 3, 3, 3, 8, 9, 10, 11, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] - ], - dtype=np.int32), - np.array([[ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 2, 0, 0, 0, 0, 0, - 0 - ], [4, 5, 6, 7, 8, 9, 10, 11, 12, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 2, 0, 0, - 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 2, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0 - ]], - dtype=np.int32), - np.array( - [[1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32)) - - match_idx = FindResultFromList(result, - [expected_output1, expected_output2]) - self.assertIsNotNone(match_idx, - '{} is not a valid result'.format(result)) - - def testKeepMaskedTokens(self): - ids = np.array( - [[4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32) - weights = np.array( - [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32) - actual_seq_len = np.array([14, 10, 14, 10], dtype=np.int32) - - g = tf.Graph() - with g.as_default(): - (src_ids, tgt_ids, tgt_labels, tgt_weights) = ops.mass( - ids, - weights, - actual_seq_len, - mask_id=3, - mask_ratio=0.5, - mask_minlen=1, - span_len=3, - keep_prob=1, - rand_prob=0, - mask_prob=0, - mask_target=True, - vocab_size=9) - - with self.session(graph=g) as sess: - (src_ids, tgt_ids, tgt_labels, tgt_weights) = sess.run([ - src_ids, - tgt_ids, - tgt_labels, - tgt_weights, - ]) - - result = MassOutput(src_ids, tgt_ids, tgt_labels, tgt_weights) - expected_output1 = MassOutput( - np.array([[ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, - 0, 0 - ], [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, - 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0 - ]], - dtype=np.int32), - np.array([[ - 3, 4, 5, 6, 7, 8, 9, 3, 3, 3, 3, 3, 3, 16, 0, 0, 0, 0, 0, 0 - ], [ - 3, 4, 5, 3, 3, 3, 3, 10, 11, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [ - 3, 3, 3, 3, 3, 8, 9, 10, 3, 12, 3, 14, 15, 16, 0, 0, 0, 0, 0, 0 - ], [BOS, 4, 3, 6, 7, 8, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32), - np.array([[ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, - 0, 0 - ], [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, - 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0 - ]], - dtype=np.int32), - np.array( - [[0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0], - [0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32)) - - expected_output2 = MassOutput( - np.array([[ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 2, 0, 0, 0, 0, 0, - 0 - ], [4, 5, 6, 7, 8, 9, 10, 11, 12, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 2, 0, 0, - 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 2, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0 - ]], - dtype=np.int32), - np.array([[ - 3, 3, 3, 6, 7, 8, 3, 10, 11, 12, 3, 3, 3, 16, 0, 0, 0, 0, 0, 0 - ], [ - 1, 4, 3, 3, 3, 3, 3, 10, 11, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [ - 3, 4, 5, 6, 3, 3, 3, 3, 3, 12, 13, 14, 3, 16, 0, 0, 0, 0, 0, 0 - ], [3, 3, 5, 6, 3, 3, 9, 10, 11, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32), - np.array([[ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 2, 0, 0, 0, 0, 0, - 0 - ], [4, 5, 6, 7, 8, 9, 10, 11, 12, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 2, 0, 0, - 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 2, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0 - ]], - dtype=np.int32), - np.array( - [[0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0], - [0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32)) - - match_idx = FindResultFromList(result, - [expected_output1, expected_output2]) - self.assertIsNotNone(match_idx, - '{} is not a valid result'.format(result)) - - def testSpanLen1(self): - ids = np.array( - [[4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, 0, 0], - [4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32) - weights = np.array( - [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32) - actual_seq_len = np.array([14, 10, 14, 10], dtype=np.int32) - - g = tf.Graph() - with g.as_default(): - (src_ids, tgt_ids, tgt_labels, tgt_weights) = ops.mass( - ids, - weights, - actual_seq_len, - mask_id=3, - mask_ratio=0.5, - mask_minlen=1, - span_len=1, - keep_prob=0, - rand_prob=0, - mask_prob=1, - mask_target=True, - vocab_size=9) - - with self.session(graph=g) as sess: - (src_ids, tgt_ids, tgt_labels, tgt_weights) = sess.run([ - src_ids, - tgt_ids, - tgt_labels, - tgt_weights, - ]) - - result = MassOutput(src_ids, tgt_ids, tgt_labels, tgt_weights) - expected_output1 = MassOutput( - np.array([ - [4, 3, 3, 7, 8, 9, 3, 3, 3, 13, 14, 3, 16, 3, 0, 0, 0, 0, 0, 0], - [3, 3, 6, 3, 3, 9, 10, 11, 12, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [4, 5, 3, 3, 8, 9, 10, 3, 3, 13, 3, 15, 3, 3, 0, 0, 0, 0, 0, 0], - [3, 5, 6, 3, 8, 3, 10, 3, 12, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] - ], - dtype=np.int32), - np.array([[ - 3, 4, 5, 3, 3, 3, 9, 10, 11, 3, 3, 14, 3, 16, 0, 0, 0, 0, 0, 0 - ], [ - 1, 4, 3, 6, 7, 3, 3, 3, 3, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [ - 3, 3, 5, 6, 3, 3, 3, 10, 11, 3, 13, 3, 15, 16, 0, 0, 0, 0, 0, 0 - ], [BOS, 3, 3, 6, 3, 8, 3, 10, 3, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] - ], - dtype=np.int32), - np.array([[ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, 0, 0, 0, - 0, 0 - ], [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, EOS, 0, - 0, 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, EOS, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0 - ]], - dtype=np.int32), - np.array( - [[0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0], - [1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0], - [1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32)) - - expected_output2 = MassOutput( - np.array([[ - 4, 5, 6, 3, 3, 3, 10, 3, 12, 13, 3, 3, 16, 3, 0, 0, 0, 0, 0, 0 - ], [ - 3, 5, 3, 7, 3, 9, 3, 3, 12, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [ - 3, 5, 3, 3, 8, 9, 3, 3, 12, 3, 3, 15, 16, 2, 0, 0, 0, 0, 0, 0 - ], [3, 3, 3, 7, 8, 9, 3, 11, 12, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.int32), - np.array([[ - 3, 3, 3, 6, 7, 8, 3, 10, 3, 3, 13, 14, 3, 16, 0, 0, 0, 0, 0, 0 - ], [1, 3, 5, 3, 7, 3, 9, 10, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [ - 1, 3, 5, 6, 3, 3, 9, 10, 3, 12, 13, 3, 3, 3, 0, 0, 0, 0, 0, 0 - ], [1, 4, 5, 3, 3, 3, 9, 3, 3, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] - ], - dtype=np.int32), - np.array([[ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 2, 0, 0, 0, 0, 0, - 0 - ], [4, 5, 6, 7, 8, 9, 10, 11, 12, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 2, 0, 0, - 0, 0, 0, 0 - ], - [ - 4, 5, 6, 7, 8, 9, 10, 11, 12, 2, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0 - ]], - dtype=np.int32), - np.array( - [[0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0], - [1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - dtype=np.float32)) - - match_idx = FindResultFromList(result, - [expected_output1, expected_output2]) - self.assertIsNotNone(match_idx, - '{} is not a valid result'.format(result)) - - def testZeroLengthSeq(self): - ids = np.array([[0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], - dtype=np.int32) - weights = np.array([[0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], - dtype=np.float32) - actual_seq_len = np.array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=np.int32) - - g = tf.Graph() - with g.as_default(): - (src_ids, tgt_ids, tgt_labels, tgt_weights) = ops.mass( - ids, - weights, - actual_seq_len, - mask_id=3, - mask_ratio=0.5, - mask_minlen=0, - span_len=8, - random_start_prob=0, - keep_prob=0, - rand_prob=0, - mask_prob=1, - mask_target=True, - vocab_size=9) - - with self.session(graph=g) as sess: - (src_ids, tgt_ids, tgt_labels, - tgt_weights) = sess.run([src_ids, tgt_ids, tgt_labels, tgt_weights]) - - self.assertAllEqual( - src_ids, - np.array([[0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], - dtype=np.int32)) - self.assertAllEqual( - tgt_ids, - np.array([[0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], - dtype=np.int32)) - self.assertAllEqual( - tgt_labels, - np.array([[0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], - dtype=np.int32)) - self.assertAllEqual( - tgt_weights, - np.array([[0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], - dtype=np.float32)) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/ops/ml_perf_subword_op.cc b/lingvo/core/ops/ml_perf_subword_op.cc deleted file mode 100644 index 645c2d01..00000000 --- a/lingvo/core/ops/ml_perf_subword_op.cc +++ /dev/null @@ -1,139 +0,0 @@ -/* Copyright 2020 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/ml_perf_subword_op.h" - -#include "unicode/uchar.h" -#include "unicode/utf8.h" -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/lib/strings/str_util.h" -#include "tensorflow/core/platform/env.h" -#include "tensorflow/core/platform/logging.h" - -namespace tensorflow { -namespace lingvo { - -Status MlPerfSubword::Load(const string& vocab_glob) { - std::vector vocab_filenames; - TF_CHECK_OK(Env::Default()->GetMatchingPaths(vocab_glob, &vocab_filenames)) - << "Unable to match vocab pattern: " << vocab_glob; - CHECK_EQ(vocab_filenames.size(), 1) - << "Did not match exactly one file with pattern: " << vocab_glob; - const string& vocab_filename = vocab_filenames[0]; - - string content; - TF_RETURN_IF_ERROR( - ReadFileToString(Env::Default(), vocab_filename, &content)); - - return LoadLines(str_util::Split(content, '\n')); -} - -Status MlPerfSubword::LoadLines(const std::vector& lines) { - for (StringPiece line : lines) { - if (line.empty()) continue; - // Strip surrounding single quotes. - auto len = line.size(); - CHECK_GT(line.size(), 2); - auto subtoken = string(line.substr(1, len - 2)); - id_to_token_.push_back(subtoken); - } - return Status::OK(); -} - -// This is a direct port of the tokenizer decode method in the MLPerf -// reference implementation for Translate/Transformer. -void MlPerfSubword::Decode(const std::vector& ids, string* out) { - std::vector subtokens_raw(ids.size()); - for (const auto& id : ids) { - subtokens_raw.emplace_back(id_to_token_[id]); - } - string inter = absl::StrJoin(subtokens_raw, ""); - - std::vector subtokens = absl::StrSplit(inter, '_'); - - std::vector token_is_alnum; - for (const auto& token : subtokens) { - int token_end = 0; - UChar32 c; - U8_NEXT(token, token_end, token.length(), c); - token_is_alnum.push_back(u_isalnum(c)); - } - std::vector ret; - - for (int i = 0; i < subtokens.size(); ++i) { - const auto& token = subtokens[i]; - if (i > 0 && token_is_alnum[i - 1] & token_is_alnum[i]) { - ret.push_back(" "); - } - ret.push_back(token); - } - *out = absl::StrJoin(ret, ""); -} - -class MlPerfSubwordIdToStringOp : public OpKernel { - public: - explicit MlPerfSubwordIdToStringOp(OpKernelConstruction* ctx) - : OpKernel(ctx) { - string vocab_filepath; - OP_REQUIRES_OK(ctx, ctx->GetAttr("vocab_filepath", &vocab_filepath)); - OP_REQUIRES_OK(ctx, vocab_.Load(vocab_filepath)); - } - - void Compute(OpKernelContext* ctx) override { - const Tensor* token_ids; - const Tensor* seq_lengths; - OP_REQUIRES_OK(ctx, ctx->input("token_ids", &token_ids)); - OP_REQUIRES_OK(ctx, ctx->input("seq_lengths", &seq_lengths)); - OP_REQUIRES(ctx, TensorShapeUtils::IsMatrix(token_ids->shape()), - errors::InvalidArgument("token_ids must be a matrix, but get ", - token_ids->shape().DebugString())); - OP_REQUIRES( - ctx, TensorShapeUtils::IsVector(seq_lengths->shape()), - errors::InvalidArgument("seq_lengths must be a vector, but get ", - seq_lengths->shape().DebugString())); - - const int batch = seq_lengths->NumElements(); - OP_REQUIRES( - ctx, batch == token_ids->dim_size(0), - errors::InvalidArgument("batch size has to match between token_ids and " - "seq_lengths")); - Tensor* out; - OP_REQUIRES_OK(ctx, ctx->allocate_output(0, TensorShape({batch}), &out)); - const auto& t_ids = token_ids->matrix(); - const auto& t_seq_lens = seq_lengths->vec(); - auto t_out = out->template vec(); - - for (int i = 0; i < batch; ++i) { - const int len_i = std::max(0, t_seq_lens(i)); - std::vector ids_i(len_i); - for (int j = 0; j < len_i; ++j) { - ids_i[j] = t_ids(i, j); - } - string decode_output; - vocab_.Decode(ids_i, &decode_output); - t_out(i) = decode_output; - } - } - - private: - MlPerfSubword vocab_; -}; - -REGISTER_KERNEL_BUILDER(Name("MlPerfSubwordIdToString").Device(DEVICE_CPU), - MlPerfSubwordIdToStringOp); - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/ml_perf_subword_op.h b/lingvo/core/ops/ml_perf_subword_op.h deleted file mode 100644 index d8b31deb..00000000 --- a/lingvo/core/ops/ml_perf_subword_op.h +++ /dev/null @@ -1,46 +0,0 @@ -/* Copyright 2020 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#ifndef THIRD_PARTY_PY_LINGVO_CORE_OPS_ML_PERF_SUBWORD_OP_H_ -#define THIRD_PARTY_PY_LINGVO_CORE_OPS_ML_PERF_SUBWORD_OP_H_ - -#include -#include - -#include "tensorflow/core/platform/env.h" -#include "tensorflow/core/platform/macros.h" - -namespace tensorflow { -namespace lingvo { - -// Subword vocabulary class. -class MlPerfSubword { - public: - MlPerfSubword() {} - ~MlPerfSubword() {} - - Status Load(const string& vocab_glob); - Status LoadLines(const std::vector& lines); - - void Decode(const std::vector& ids, string* out); - - private: - std::vector id_to_token_; -}; - -} // namespace lingvo -} // namespace tensorflow - -#endif // THIRD_PARTY_PY_LINGVO_CORE_OPS_MLPERF_SUBWORD_OP_H diff --git a/lingvo/core/ops/pack_ops.cc b/lingvo/core/ops/pack_ops.cc deleted file mode 100644 index e2f2c5ec..00000000 --- a/lingvo/core/ops/pack_ops.cc +++ /dev/null @@ -1,502 +0,0 @@ -/* Copyright 2020 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include -#include -#include - -#include "absl/container/flat_hash_map.h" -#include "absl/strings/str_join.h" -#include "absl/strings/string_view.h" -#include "absl/synchronization/mutex.h" -#include "lingvo/core/ops/text_packing.h" -#include "tensorflow/core/framework/bounds_check.h" -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/register_types.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" -#include "tensorflow/core/lib/core/errors.h" -#include "tensorflow/core/lib/strings/str_util.h" -#include "tensorflow/core/platform/env.h" -#include "tensorflow/core/platform/threadpool.h" - -namespace tensorflow { -namespace lingvo { - -namespace { -struct PackSequencesOutputs { - Tensor* src_segment_ids = nullptr; - Tensor* src_segment_pos = nullptr; - Tensor* src_indices_in_input = nullptr; - Tensor* tgt_segment_ids = nullptr; - Tensor* tgt_segment_pos = nullptr; - Tensor* tgt_indices_in_input = nullptr; -}; - -// The record of packing one input sequence, i.e. a (src, tgt) pair. -struct PackRecord { - int index_in_input; - TextPacking::PackingIndex packing; -}; -} // namespace - -// An op that outputs a packing pattern based on actual sequence lengths. -class PackSequencesOp : public OpKernel { - public: - explicit PackSequencesOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - int64 seed; - OP_REQUIRES_OK(ctx, ctx->GetAttr("seed", &seed)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("packed_batch_size", &packed_batch_size_)); - OP_REQUIRES_OK(ctx, - ctx->GetAttr("packed_src_seq_len", &packed_src_seq_len_)); - OP_REQUIRES_OK(ctx, - ctx->GetAttr("packed_tgt_seq_len", &packed_tgt_seq_len_)); - if (seed == 0) { - // If seed is unspecified, use completely random seed. - std::random_device device("/dev/urandom"); - seed = std::mt19937_64(device())(); - } - rnd_.seed(seed); - } - - ~PackSequencesOp() override {} - - void Compute(OpKernelContext* ctx) override { - ValidateInputs(ctx); - if (!ctx->status().ok()) { - return; - } - - const auto input_num = ctx->input(0).vec().size(); - std::vector pack_records; - pack_records.reserve(input_num); - const int output_num = PackEntireInputs(ctx, &pack_records); - - bool dropping_inputs; - absl::flat_hash_map new_indices; - PackSequencesOutputs outputs; - if (packed_batch_size_ == 0) { - AllocateOutputs(ctx, &outputs, output_num); - dropping_inputs = false; - } else { - AllocateOutputs(ctx, &outputs, packed_batch_size_); - dropping_inputs = DropPackedRows(ctx, output_num, &new_indices); - } - WriteOutputs(ctx, pack_records, dropping_inputs, &new_indices, &outputs); - } - - private: - // Validates the shapes and types of inputs. - void ValidateInputs(OpKernelContext* ctx); - - // Allocates (and zero initializes) all outputs. - void AllocateOutputs(OpKernelContext* ctx, PackSequencesOutputs* outputs, - const int32 packed_batch_size); - - // Pack entire inputs. Returns the number of rows needed to pack all of input - // sequences. Also outputs the packing records. - int PackEntireInputs(OpKernelContext* ctx, - std::vector* pack_records); - - // Returns whether we need to drop packed rows, and if so, returns a mapping - // from the original output row index to its new index after dropping. - bool DropPackedRows(OpKernelContext* ctx, int num_rows, - absl::flat_hash_map* new_indices); - - // Writes the packing results to the output tensors. - void WriteOutputs(OpKernelContext* ctx, - const std::vector& pack_records, - bool dropping_inputs, - const absl::flat_hash_map* new_indices, - PackSequencesOutputs* outputs); - - int packed_batch_size_; - int packed_src_seq_len_; - int packed_tgt_seq_len_; - mutable absl::Mutex mu_; - // Used for randomizing the dropping of input rows when needed. - std::mt19937 rnd_ ABSL_GUARDED_BY(mu_); - - TF_DISALLOW_COPY_AND_ASSIGN(PackSequencesOp); -}; - -void PackSequencesOp::ValidateInputs(OpKernelContext* ctx) { - const Tensor& src_actual_seq_len = ctx->input(0); - OP_REQUIRES(ctx, - TensorShapeUtils::IsVector(src_actual_seq_len.shape()) && - (src_actual_seq_len.dtype() == DataType::DT_INT32), - errors::InvalidArgument( - "src_actual_seq_len must be a vector of int32, got: ", - src_actual_seq_len.DebugString())); - const Tensor& tgt_actual_seq_len = ctx->input(1); - OP_REQUIRES(ctx, - TensorShapeUtils::IsVector(tgt_actual_seq_len.shape()) && - (tgt_actual_seq_len.dtype() == DataType::DT_INT32), - errors::InvalidArgument( - "tgt_actual_seq_len must be a vector of int32, got: ", - tgt_actual_seq_len.DebugString())); - OP_REQUIRES(ctx, src_actual_seq_len.shape() == tgt_actual_seq_len.shape(), - errors::InvalidArgument( - "src_actual_seq_len must be the same shape as " - "tgt_actual_seq_len, got: src shape ", - src_actual_seq_len.shape().DebugString(), " vs. tgt shape ", - tgt_actual_seq_len.shape().DebugString())); -} - -void PackSequencesOp::AllocateOutputs(OpKernelContext* ctx, - PackSequencesOutputs* outputs, - const int32 packed_batch_size) { - TensorShape packed_src_shape({packed_batch_size, packed_src_seq_len_}); - TensorShape packed_tgt_shape({packed_batch_size, packed_tgt_seq_len_}); - - int output_id = 0; - OP_REQUIRES_OK(ctx, ctx->allocate_output(output_id++, packed_src_shape, - &outputs->src_segment_ids)); - OP_REQUIRES_OK(ctx, ctx->allocate_output(output_id++, packed_src_shape, - &outputs->src_segment_pos)); - OP_REQUIRES_OK(ctx, ctx->allocate_output(output_id++, packed_src_shape, - &outputs->src_indices_in_input)); - outputs->src_segment_ids->matrix().setZero(); - outputs->src_segment_pos->matrix().setZero(); - outputs->src_indices_in_input->matrix().setZero(); - - OP_REQUIRES_OK(ctx, ctx->allocate_output(output_id++, packed_tgt_shape, - &outputs->tgt_segment_ids)); - OP_REQUIRES_OK(ctx, ctx->allocate_output(output_id++, packed_tgt_shape, - &outputs->tgt_segment_pos)); - OP_REQUIRES_OK(ctx, ctx->allocate_output(output_id++, packed_tgt_shape, - &outputs->tgt_indices_in_input)); - outputs->tgt_segment_ids->matrix().setZero(); - outputs->tgt_segment_pos->matrix().setZero(); - outputs->tgt_indices_in_input->matrix().setZero(); -} - -int PackSequencesOp::PackEntireInputs(OpKernelContext* ctx, - std::vector* pack_records) { - const auto& src_actual_seq_len = ctx->input(0).vec(); - const auto& tgt_actual_seq_len = ctx->input(1).vec(); - const auto input_num = src_actual_seq_len.size(); - - // We ask for a sufficiently large output batch size to pack all input - // sequences in its entirety. We drop input sequences if needed afterward, - // We also ensure that the first `packed_batch_size` rows are never empty - // by packing into them first. - TextPacking packing(/*columns=*/2, /*batch=*/input_num, - {packed_src_seq_len_, packed_tgt_seq_len_}, - /*align=*/1, - /*pack=*/true, /*spread_first_n=*/packed_batch_size_); - int max_output_batch_idx = 0; - for (int i = 0; i < input_num; ++i) { - int src_seq_len = src_actual_seq_len(i); - int tgt_seq_len = tgt_actual_seq_len(i); - if (src_seq_len < 1 || tgt_seq_len < 1) { - LOG_EVERY_N(WARNING, 500) - << "Input sequence with lengths (src=" << src_seq_len - << ", tgt=" << tgt_seq_len - << ") dropped as only positive actual lengths are allowed."; - continue; - } - PackRecord p; - p.index_in_input = i; - if (!packing.Add({src_seq_len, tgt_seq_len}, &p.packing)) { - // This only happens when the actual sequence length of the input is - // larger than the packed sequence length specified on the output. - LOG_EVERY_N(WARNING, 500) - << "Input sequence with lengths (src=" << src_seq_len - << ", tgt=" << tgt_seq_len - << ") dropped as its lengths exceed the output sequence length."; - continue; - } - max_output_batch_idx = std::max(p.packing.batch, max_output_batch_idx); - pack_records->push_back(p); - } - return max_output_batch_idx + 1; -} - -bool PackSequencesOp::DropPackedRows( - OpKernelContext* ctx, int num_rows, - absl::flat_hash_map* new_indices) { - if (num_rows <= packed_batch_size_) { - return false; - } - - // Simple reservoir sampling to pick `packed_batch_size` items out of - // `num_rows`: - // https://en.wikipedia.org/wiki/Reservoir_sampling#Simple_algorithm - std::vector indices_kept(packed_batch_size_); - for (int i = 1; i < packed_batch_size_; ++i) { - indices_kept[i] = i; - } - for (int i = packed_batch_size_; i < num_rows; ++i) { - std::uniform_int_distribution<> distribution(0, i); - int j; // Uniformly picked on [0, i]. - { - absl::MutexLock l(&mu_); - j = distribution(rnd_); - } - if (j < packed_batch_size_) { - indices_kept[j] = i; - } - } - for (int i = 0; i < packed_batch_size_; ++i) { - new_indices->insert({indices_kept[i], i}); - } - return true; -} - -void PackSequencesOp::WriteOutputs( - OpKernelContext* ctx, const std::vector& pack_records, - bool dropping_inputs, const absl::flat_hash_map* new_indices, - PackSequencesOutputs* outputs) { - const auto& src_actual_seq_len = ctx->input(0).vec(); - const auto& tgt_actual_seq_len = ctx->input(1).vec(); - - auto src_segment_ids = outputs->src_segment_ids->matrix(); - auto src_segment_pos = outputs->src_segment_pos->matrix(); - auto src_indices_in_input = outputs->src_indices_in_input->matrix(); - auto tgt_segment_ids = outputs->tgt_segment_ids->matrix(); - auto tgt_segment_pos = outputs->tgt_segment_pos->matrix(); - auto tgt_indices_in_input = outputs->tgt_indices_in_input->matrix(); - - for (const auto& p : pack_records) { - int output_idx = p.packing.batch; - if (dropping_inputs) { - auto new_idx_iter = new_indices->find(output_idx); - if (new_idx_iter == new_indices->end()) { - // This row is being dropped. - continue; - } - output_idx = new_idx_iter->second; - } - const int src_seq_idx = p.packing.time[0]; - for (int i = 0; i < src_actual_seq_len(p.index_in_input); ++i) { - src_segment_ids(output_idx, src_seq_idx + i) = p.packing.seq; - src_segment_pos(output_idx, src_seq_idx + i) = i; - src_indices_in_input(output_idx, src_seq_idx + i) = p.index_in_input; - } - const int tgt_seq_idx = p.packing.time[1]; - for (int i = 0; i < tgt_actual_seq_len(p.index_in_input); ++i) { - tgt_segment_ids(output_idx, tgt_seq_idx + i) = p.packing.seq; - tgt_segment_pos(output_idx, tgt_seq_idx + i) = i; - tgt_indices_in_input(output_idx, tgt_seq_idx + i) = p.index_in_input; - } - } -} - -REGISTER_KERNEL_BUILDER(Name("PackSequences").Device(DEVICE_CPU), - PackSequencesOp); - -// An op that applies a packing pattern on an input data. -template -class ApplyPackingOp : public OpKernel { - public: - explicit ApplyPackingOp(OpKernelConstruction* ctx) : OpKernel(ctx) {} - - ~ApplyPackingOp() override {} - - void Compute(OpKernelContext* ctx) override { - ValidateInputs(ctx); - if (!ctx->status().ok()) { - return; - } - Tensor* output = nullptr; - // Allocates output and initializes with padding. - const Tensor& segment_ids = ctx->input(2); - OP_REQUIRES_OK(ctx, ctx->allocate_output(0, segment_ids.shape(), &output)); - const T padding = ctx->input(1).scalar()(); - output->matrix().setConstant(padding); - Apply(ctx, output); - } - - private: - // Validates the shapes and types of inputs. - void ValidateInputs(OpKernelContext* ctx) { - const Tensor& input = ctx->input(0); - OP_REQUIRES( - ctx, TensorShapeUtils::IsMatrix(input.shape()), - errors::InvalidArgument("input must be a matrix, got input shape: ", - input.shape().DebugString())); - - const Tensor& padding = ctx->input(1); - OP_REQUIRES( - ctx, TensorShapeUtils::IsScalar(padding.shape()), - errors::InvalidArgument("padding must be a scalar, got padding shape: ", - padding.shape().DebugString())); - - const Tensor& segment_ids = ctx->input(2); - const Tensor& indices_in_input = ctx->input(3); - OP_REQUIRES( - ctx, - segment_ids.shape() == indices_in_input.shape() && - TensorShapeUtils::IsMatrix(segment_ids.shape()), - errors::InvalidArgument("segment_ids and indices_in_input must be " - "matrices of the same shape, got: ", - segment_ids.shape().DebugString(), " vs. ", - indices_in_input.shape().DebugString())); - } - - void Apply(OpKernelContext* ctx, Tensor* output) { - const auto& input = ctx->input(0).matrix(); - const auto input_rows = ctx->input(0).dim_size(0); - const auto input_columns = ctx->input(0).dim_size(1); - const auto& segment_ids = ctx->input(2).matrix(); - const auto& indices_in_input = ctx->input(3).matrix(); - auto output_matrix = output->matrix(); - const int64 rows = output->dim_size(0); - const int64 columns = output->dim_size(1); - // The CPU cost per each row is linear in the number of elements in that row - // (`columns`). We need to read segment_ids to discern each segment, and - // copy from input to output, plus a few extra cycles (e.g. checking - // bounds). The constant is guesstimated to be 4. - const int64 cost_per_unit = columns << 2; - ctx->device()->tensorflow_cpu_worker_threads()->workers->ParallelFor( - rows, cost_per_unit, [&](int64 begin, int64 end) { - for (int i = begin; i < end; ++i) { - for (int j = 0; j < columns; ++j) { - if (segment_ids(i, j) <= 0) { - // We do not skip the rest of the row because we cannot assume 0 - // segment ids only occur at end of row. For example, it might - // occur due to alignment requirements during packing. - continue; - } - const int start = j; - for (; j + 1 < columns && - segment_ids(i, j) == segment_ids(i, j + 1); - ++j) { - } - // [start, j] have the same, positive segment_ids. - const int actual_seq_len = j - start + 1; - // At output position (i, start), we need to copy over - // `actual_seq_len` T values from the input. - const int index_in_input = indices_in_input(i, start); - OP_REQUIRES( - ctx, - tensorflow::FastBoundsCheck(index_in_input, input_rows) && - tensorflow::FastBoundsCheck(actual_seq_len, - input_columns + 1), - errors::InvalidArgument( - "out of bound found packing at (", i, ", ", start, - ") for input index ", index_in_input, " with length ", - actual_seq_len, " where input shape is ", - ctx->input(0).shape().DebugString())); - std::memcpy(&output_matrix(i, start), &input(index_in_input, 0), - actual_seq_len * sizeof(T)); - } - } - }); - } - - TF_DISALLOW_COPY_AND_ASSIGN(ApplyPackingOp); -}; - -template <> -class ApplyPackingOp<::tensorflow::tstring> : public OpKernel { - public: - explicit ApplyPackingOp(OpKernelConstruction* ctx) : OpKernel(ctx) {} - - ~ApplyPackingOp() override {} - - void Compute(OpKernelContext* ctx) override { - ValidateInputs(ctx); - if (!ctx->status().ok()) { - return; - } - Tensor* output = nullptr; - TensorShape output_shape({ctx->input(3).shape().dim_size(0)}); - OP_REQUIRES_OK(ctx, ctx->allocate_output(0, output_shape, &output)); - Apply(ctx, output); - } - - private: - // Validates the shapes and types of inputs. - void ValidateInputs(OpKernelContext* ctx) { - const Tensor& input = ctx->input(0); - OP_REQUIRES( - ctx, TensorShapeUtils::IsVector(input.shape()), - errors::InvalidArgument("input must be a vector, got input shape: ", - input.shape().DebugString())); - - const Tensor& padding = ctx->input(1); - OP_REQUIRES( - ctx, TensorShapeUtils::IsScalar(padding.shape()), - errors::InvalidArgument("padding must be a scalar, got padding shape: ", - padding.shape().DebugString())); - - const Tensor& segment_ids = ctx->input(2); - const Tensor& indices_in_input = ctx->input(3); - OP_REQUIRES( - ctx, - segment_ids.shape() == indices_in_input.shape() && - TensorShapeUtils::IsMatrix(segment_ids.shape()), - errors::InvalidArgument("segment_ids and indices_in_input must be " - "matrices of the same shape, got: ", - segment_ids.shape().DebugString(), " vs. ", - indices_in_input.shape().DebugString())); - } - - void Apply(OpKernelContext* ctx, Tensor* output) { - const auto& input = ctx->input(0).vec<::tensorflow::tstring>(); - const auto num_input_rows = ctx->input(0).dim_size(0); - const auto& segment_ids = ctx->input(2).matrix(); - const auto& indices_in_input = ctx->input(3).matrix(); - auto output_vec = output->vec<::tensorflow::tstring>(); - const auto& sep = ctx->input(1).scalar<::tensorflow::tstring>()(); - for (int i = 0; i < output->dim_size(0); ++i) { - // input_rows condenses row i of indices_in_input, e.g. from - // [0, 0, 0, 3, 3, 4, 4, 4, 4, 0, 0] to [0, 3, 4]. - std::vector input_rows; - for (int j = 0; j < ctx->input(3).dim_size(1); ++j) { - auto row = indices_in_input(i, j); - if (segment_ids(i, j) && - (input_rows.empty() || input_rows.back() != row)) { - OP_REQUIRES(ctx, tensorflow::FastBoundsCheck(row, num_input_rows), - errors::InvalidArgument( - "out of bound found packing at (", i, ", ", j, - ") for input index ", row, " where input shape is ", - ctx->input(0).shape().DebugString())); - input_rows.push_back(row); - } - } - std::vector strs; - strs.reserve(input_rows.size()); - for (auto row : input_rows) { - strs.push_back(input(row)); - } - // Output on row i is the joined input strings for that row. - output_vec(i) = absl::StrJoin(strs, sep); - } - } - - TF_DISALLOW_COPY_AND_ASSIGN(ApplyPackingOp); -}; - -#define REGISTER(TYPE) \ - REGISTER_KERNEL_BUILDER( \ - Name("ApplyPacking").Device(DEVICE_CPU).TypeConstraint("T"), \ - ApplyPackingOp); - -TF_CALL_float(REGISTER); -TF_CALL_double(REGISTER); -TF_CALL_int32(REGISTER); -TF_CALL_int64(REGISTER); -TF_CALL_uint32(REGISTER); -TF_CALL_uint64(REGISTER); -TF_CALL_bool(REGISTER); -TF_CALL_string(REGISTER); - -#undef REGISTER - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/pack_ops_test.py b/lingvo/core/ops/pack_ops_test.py deleted file mode 100644 index c3c374a4..00000000 --- a/lingvo/core/ops/pack_ops_test.py +++ /dev/null @@ -1,379 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for pack_ops.""" - -import collections - -from lingvo import compat as tf -from lingvo.core import ops -from lingvo.core import test_utils -import numpy as np - -PackSequenceTestCase = collections.namedtuple('PackSequenceTestCase', [ - 'src_actual_seq_len', 'tgt_actual_seq_len', 'packed_batch_size', - 'packed_src_seq_len', 'packed_tgt_seq_len', 'src_segment_ids', - 'src_segment_pos', 'src_indices_in_input', 'tgt_segment_ids', - 'tgt_segment_pos', 'tgt_indices_in_input' -]) - -# Set default to empty. -PackSequenceTestCase.__new__.__defaults__ = (None,) * len( - PackSequenceTestCase._fields) - -ApplyPackingTestCase = collections.namedtuple( - 'ApplyPackingTestCase', - ['input', 'padding', 'segment_ids', 'indices_in_input', 'output']) - -# Set default to empty. -ApplyPackingTestCase.__new__.__defaults__ = (None,) * len( - ApplyPackingTestCase._fields) - - -def FindResultFromList(result, test_cases): - """Whether a result is among a list of possible results. - - Args: - result: A ops.PackSequences object that is the result of running - ops.pack_sequences(). - test_cases: A list of PackSequenceTestCase. - - Returns: - The index of first match found, or None for not found. - """ - for idx, test_case in enumerate(test_cases): - match = True - for attr in [ - 'src_segment_ids', 'src_segment_pos', 'src_indices_in_input', - 'tgt_segment_ids', 'tgt_segment_pos', 'tgt_indices_in_input' - ]: - if not np.array_equal(getattr(result, attr), getattr(test_case, attr)): - match = False - break - if not match: - continue - return idx - return None - - -class PackSequencesOpTest(test_utils.TestCase): - - def testPackSequences(self): - test_cases = { - 'Basic': - PackSequenceTestCase([1, 2, 1], [1, 2, 1], 2, 5, 5, - [[1, 2, 0, 0, 0], [1, 1, 0, 0, 0]], - [[0, 0, 0, 0, 0], [0, 1, 0, 0, 0]], - [[0, 2, 0, 0, 0], [1, 1, 0, 0, 0]], - [[1, 2, 0, 0, 0], [1, 1, 0, 0, 0]], - [[0, 0, 0, 0, 0], [0, 1, 0, 0, 0]], - [[0, 2, 0, 0, 0], [1, 1, 0, 0, 0]]), - 'SpreadFirstN': - PackSequenceTestCase([3, 1, 2], [4, 2, 1], 2, 5, 5, - [[1, 1, 1, 2, 2], [1, 0, 0, 0, 0]], - [[0, 1, 2, 0, 1], [0, 0, 0, 0, 0]], - [[0, 0, 0, 2, 2], [1, 0, 0, 0, 0]], - [[1, 1, 1, 1, 2], [1, 1, 0, 0, 0]], - [[0, 1, 2, 3, 0], [0, 1, 0, 0, 0]], - [[0, 0, 0, 0, 2], [1, 1, 0, 0, 0]]), - 'DifferentSrcTgtLengths': - PackSequenceTestCase([3, 2, 1], [4, 1, 5], 2, 4, 6, - [[1, 1, 1, 0], [1, 1, 2, 0]], - [[0, 1, 2, 0], [0, 1, 0, 0]], - [[0, 0, 0, 0], [1, 1, 2, 0]], - [[1, 1, 1, 1, 0, 0], [1, 2, 2, 2, 2, 2]], - [[0, 1, 2, 3, 0, 0], [0, 0, 1, 2, 3, 4]], - [[0, 0, 0, 0, 0, 0], [1, 2, 2, 2, 2, 2]]), - 'Padding': - PackSequenceTestCase([1], [2], 3, 3, 3, - [[1, 0, 0], [0, 0, 0], [0, 0, 0]], - [[0, 0, 0], [0, 0, 0], [0, 0, 0]], - [[0, 0, 0], [0, 0, 0], [0, 0, 0]], - [[1, 1, 0], [0, 0, 0], [0, 0, 0]], - [[0, 1, 0], [0, 0, 0], [0, 0, 0]], - [[0, 0, 0], [0, 0, 0], [0, 0, 0]]), - 'DroppingInputsTooLong': - PackSequenceTestCase([1, 3, 1], [4, 1, 2], 2, 2, 3, - [[1, 0], [0, 0]], [[0, 0], [0, 0]], - [[2, 0], [0, 0]], [[1, 1, 0], [0, 0, 0]], - [[0, 1, 0], [0, 0, 0]], - [[2, 2, 0], [0, 0, 0]]), - 'DroppingNonPositiveLengths': - PackSequenceTestCase([1, 0, 1], [0, 1, 3], 2, 2, 3, - [[1, 0], [0, 0]], [[0, 0], [0, 0]], - [[2, 0], [0, 0]], [[1, 1, 1], [0, 0, 0]], - [[0, 1, 2], [0, 0, 0]], - [[2, 2, 2], [0, 0, 0]]), - 'PackedBatchSize0': - PackSequenceTestCase( - [3, 1, 2, 0, 1, 6, 2, 3, 4, 1, 1], - [4, 2, 1, 1, 0, 2, 6, 1, 1, 4, 3], 0, 5, 5, - [[1, 1, 1, 2, 2], [1, 2, 2, 2, 0], [1, 1, 1, 1, 2], - [1, 0, 0, 0, 0]], [[0, 1, 2, 0, 1], [0, 0, 1, 2, 0], - [0, 1, 2, 3, 0], [0, 0, 0, 0, 0]], - [[0, 0, 0, 2, 2], [1, 7, 7, 7, 0], [8, 8, 8, 8, 9], - [10, 0, 0, 0, 0]], [[1, 1, 1, 1, 2], [1, 1, 2, 0, 0], - [1, 2, 2, 2, 2], [1, 1, 1, 0, 0]], - [[0, 1, 2, 3, 0], [0, 1, 0, 0, 0], [0, 0, 1, 2, 3], - [0, 1, 2, 0, 0]], [[0, 0, 0, 0, 2], [1, 1, 7, 0, 0], - [8, 9, 9, 9, 9], [10, 10, 10, 0, 0]]) - } - for name, test in test_cases.items(): - with self.session() as sess: - r = sess.run( - ops.pack_sequences( - tf.constant(test.src_actual_seq_len, tf.int32), - tf.constant(test.tgt_actual_seq_len, - tf.int32), test.packed_batch_size, - test.packed_src_seq_len, test.packed_tgt_seq_len)) - self.assertEqual(6, len(r), name) - self.assertAllEqual(r[0], test.src_segment_ids, name) - self.assertAllEqual(r[1], test.src_segment_pos, name) - self.assertAllEqual(r[2], test.src_indices_in_input, name) - self.assertAllEqual(r[3], test.tgt_segment_ids, name) - self.assertAllEqual(r[4], test.tgt_segment_pos, name) - self.assertAllEqual(r[5], test.tgt_indices_in_input, name) - - def testPackSequencesErrors(self): - test_cases = { - 'actual_seq_len must be the same shape': - PackSequenceTestCase([1, 1, 1], [1, 1], 2, 2, 2), - 'actual_seq_len must be a vector': - PackSequenceTestCase([[1], [1]], [[1], [1]], 2, 2, 2) - } - for name, test in test_cases.items(): - with self.assertRaisesRegex(tf.errors.InvalidArgumentError, name): - with self.session() as sess: - sess.run( - ops.pack_sequences(test.src_actual_seq_len, - test.tgt_actual_seq_len, - test.packed_batch_size, - test.packed_src_seq_len, - test.packed_tgt_seq_len)) - - # seq_len must be a scalar. - test = PackSequenceTestCase([1, 1], [1, 1], 2, [2, 2], 2) - with self.assertRaisesRegex(TypeError, 'Expected int'): - with self.session() as sess: - sess.run( - ops.pack_sequences(test.src_actual_seq_len, test.tgt_actual_seq_len, - test.packed_batch_size, test.packed_src_seq_len, - test.packed_tgt_seq_len)) - - def testDroppingInputsFixedSeed(self): - # Packing 3 rows into 2, where we need to drop one row. - inputs = [[2, 1, 2], [1, 2, 1], 2, 2, 2] - test_cases = [ - # (seed, test_case) - ( - 45, - # dropping the last row - PackSequenceTestCase(*( - inputs + - [[[1, 1], [1, 0]], [[0, 1], [0, 0]], [[0, 0], [1, 0]], - [[1, 0], [1, 1]], [[0, 0], [0, 1]], [[0, 0], [1, 1]]]))), - ( - (1 << 37) + 1, - # dropping the second row - PackSequenceTestCase(*( - inputs + - [[[1, 1], [1, 1]], [[0, 1], [0, 1]], [[0, 0], [2, 2]], - [[1, 0], [1, 0]], [[0, 0], [0, 0]], [[0, 0], [2, 0]]]))), - ] - for seed, test in test_cases: - with self.session() as sess: - r = sess.run( - ops.pack_sequences( - tf.constant(test.src_actual_seq_len, tf.int32), - tf.constant(test.tgt_actual_seq_len, tf.int32), - test.packed_batch_size, - test.packed_src_seq_len, - test.packed_tgt_seq_len, - seed=seed)) - name = 'test case with seed {}'.format(seed) - self.assertEqual(6, len(r), name) - self.assertAllEqual(r[0], test.src_segment_ids, name) - self.assertAllEqual(r[1], test.src_segment_pos, name) - self.assertAllEqual(r[2], test.src_indices_in_input, name) - self.assertAllEqual(r[3], test.tgt_segment_ids, name) - self.assertAllEqual(r[4], test.tgt_segment_pos, name) - self.assertAllEqual(r[5], test.tgt_indices_in_input, name) - - def testDroppingInputsNonDeterministic(self): - # Packing 3 rows into 1, where we need to drop two rows. - inputs = [[2, 1, 2], [1, 2, 1], 1, 2, 2] - test_cases = [ - # keeping only the first row - PackSequenceTestCase(*( - inputs + - [[[1, 1]], [[0, 1]], [[0, 0]], [[1, 0]], [[0, 0]], [[0, 0]]])), - # keeping only the second row - PackSequenceTestCase(*( - inputs + - [[[1, 0]], [[0, 0]], [[1, 0]], [[1, 1]], [[0, 1]], [[1, 1]]])), - # keeping only the last row - PackSequenceTestCase(*( - inputs + - [[[1, 1]], [[0, 1]], [[2, 2]], [[1, 0]], [[0, 0]], [[2, 0]]])), - ] - counts = [0] * 3 - with self.session() as sess: - test = test_cases[0] - for _ in range(100): - r = sess.run( - ops.pack_sequences( - tf.constant(test.src_actual_seq_len, tf.int32), - tf.constant(test.tgt_actual_seq_len, - tf.int32), test.packed_batch_size, - test.packed_src_seq_len, test.packed_tgt_seq_len)) - match_idx = FindResultFromList(r, test_cases) - self.assertIsNotNone(match_idx, '{} is not a valid result'.format(r)) - counts[match_idx] += 1 - # We test that all possible outcomes occur sufficiently often to ensure that - # dropping is not biased. - # The probability of this test failing due to chance is less than 1 in a - # million runs, as scipy.stats.binom.cdf(10, 100, 0.3333) ~= 5e-8 - for idx, count in enumerate(counts): - self.assertGreater( - count, 10, - 'test case {} does not occur sufficiently often: {}'.format( - idx, counts)) - - -class ApplyPackingOpTest(test_utils.TestCase): - - def testApplyPacking(self): - test_cases = { - 'Basic': - ApplyPackingTestCase([[0, 1], [2, 3]], 0, [[1, 1], [1, 1]], - [[1, 1], [0, 0]], [[2, 3], [0, 1]]), - 'Padding': - ApplyPackingTestCase([[0, 1], [2, 3]], -1, [[1, 1, 0], [1, 1, 0]], - [[1, 1, 0], [0, 0, 0]], - [[2, 3, -1], [0, 1, -1]]), - 'Tiny': - ApplyPackingTestCase([[0, 1], [2, 3]], 0, [[1], [1]], [[0], [1]], - [[0], [2]]), - '5x2 input to 2x4': - ApplyPackingTestCase( - np.reshape(np.arange(10), - [5, 2]), -1, [[1, 1, 2, 2], [0, 1, 0, 2]], - [[2, 2, 3, 3], [0, 4, 0, 0]], [[4, 5, 6, 7], [-1, 8, -1, 0]]), - '6x4 input to 3x4': - ApplyPackingTestCase( - np.reshape(np.arange(24), [6, 4]), -1, - [[0, 0, 0, 1], [1, 1, 1, 1], [1, 2, 3, 4]], - [[0, 0, 0, 0], [1, 1, 1, 1], [2, 3, 4, 5]], - [[-1, -1, -1, 0], [4, 5, 6, 7], [8, 12, 16, 20]]), - '6x4 input to 3x5': - ApplyPackingTestCase( - np.reshape(np.arange(24), [6, 4]), -1, - [[0, 0, 0, 0, 0], [0, 1, 1, 0, 0], [0, 1, 1, 0, 2]], - [[0, 0, 0, 0, 0], [0, 1, 1, 0, 0], [0, 0, 0, 0, 3]], - [[-1, -1, -1, -1, -1], [-1, 4, 5, -1, -1], [-1, 0, 1, -1, 12]]), - '100x4 input to 3x5': - ApplyPackingTestCase( - np.reshape(np.arange(400), [100, 4]), -1, - [[1, 1, 1, 2, 2], [0, 1, 1, 1, 1], [2, 2, 0, 0, 3]], - [[99, 99, 99, 1, 1], [0, 50, 50, 50, 50], [90, 90, 0, 0, 3]], - [[396, 397, 398, 4, 5], [-1, 200, 201, 202, 203], - [360, 361, -1, -1, 12]]), - } - for name, test in test_cases.items(): - with self.session(): - output = ops.apply_packing( - tf.constant(test.input, tf.int32), - tf.constant(test.padding, tf.int32), - tf.constant(test.segment_ids, tf.int32), - tf.constant(test.indices_in_input, tf.int32)).eval() - self.assertAllEqual(output, test.output, name) - - def testApplyPackingTypes(self): - test = ApplyPackingTestCase([[0, 1], [2, 3]], 99, [[1, 1, 0], [1, 1, 0]], - [[1, 1, 0], [0, 0, 0]], - [[2, 3, 99], [0, 1, 99]]) - for dtype in [ - tf.int32, tf.int64, tf.float32, tf.float64, tf.uint32, tf.uint64 - ]: - with self.session(): - output = ops.apply_packing( - tf.constant(test.input, dtype), tf.constant(test.padding, dtype), - tf.constant(test.segment_ids, tf.int32), - tf.constant(test.indices_in_input, tf.int32)).eval() - expected = tf.constant(test.output, dtype).eval() - self.assertAllEqual(output, expected, dtype) - - def testApplyPackingStrings(self): - test_cases = { - 'Basic': - ApplyPackingTestCase(['a', 'b'], ',', [[1, 1]], [[1, 0]], [b'b,a']), - 'Repeated': - ApplyPackingTestCase(['a', 'b'], ',', [[1, 1, 1]], [[1, 1, 1]], - [b'b']), - 'Separator': - ApplyPackingTestCase(['a', 'b', 'c', 'd'], '=', [[1, 1, 1, 0]], - [[1, 0, 3, 2]], [b'b=a=d']), - 'MultiRows': - ApplyPackingTestCase(['a', 'b', 'c', 'd'], ';', - [[1, 1, 1, 0], [0, 1, 1, 1]], - [[2, 2, 1, 0], [2, 0, 1, 1]], - [b'c;b', b'a;b']), - 'SingleString': - ApplyPackingTestCase(['a', 'b', 'c', 'd'], ',', - [[0, 0, 1], [0, 1, 0]], [[0, 1, 2], [0, 1, 2]], - [b'c', b'b']), - 'EmptyRow': - ApplyPackingTestCase(['a', 'b', 'c', 'd'], ',', - [[0, 0, 0], [1, 1, 1]], [[0, 1, 2], [0, 0, 2]], - [b'', b'a,c']), - } - for name, test in test_cases.items(): - with self.session(): - output = ops.apply_packing( - tf.constant(test.input, tf.string), - tf.constant(test.padding, tf.string), - tf.constant(test.segment_ids, tf.int32), - tf.constant(test.indices_in_input, tf.int32)).eval() - self.assertAllEqual(output, test.output, name) - - def testApplyPackingErrors(self): - test_cases = { - 'out of bound': - ApplyPackingTestCase([[0, 1], [2, 3]], 0, [[1, 1, 1], [1, 0, 0]], - [[1, 1, 1], [0, 0, 0]]), - 'out of bound ': - ApplyPackingTestCase([[0, 1], [2, 3]], 0, [[1, 1], [1, 0]], - [[0, 0], [2, 0]]), - 'out of bound.': - ApplyPackingTestCase(['a', 'b'], ',', [[1, 2]], [[1, 2]]), - 'segment_ids and indices_in_input must be matrices': - ApplyPackingTestCase([[0, 1], [2, 3]], 0, [1, 1], [0, 0]), - 'segment_ids and indices_in_input must be matrices of the same shape': - ApplyPackingTestCase([[0, 1], [2, 3]], 0, [[1, 1], [1, 0]], - [[0, 0], [0, 0], [0, 0]]), - 'input must be a matrix': - ApplyPackingTestCase([0, 1], 0, [[1]], [[0]]), - 'padding must be a scalar': - ApplyPackingTestCase([[0, 1], [2, 3]], [-1], [[1]], [[0]]), - } - for name, test in test_cases.items(): - with self.assertRaisesRegex(tf.errors.InvalidArgumentError, name): - with self.session(): - ops.apply_packing(test.input, test.padding, test.segment_ids, - test.indices_in_input).eval() - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/ops/preconditioner_captain.cc b/lingvo/core/ops/preconditioner_captain.cc deleted file mode 100644 index cfdfc589..00000000 --- a/lingvo/core/ops/preconditioner_captain.cc +++ /dev/null @@ -1,133 +0,0 @@ -/* Copyright 2020 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ -#include "lingvo/core/ops/preconditioner_captain.h" - -#include "tensorflow/core/framework/graph.pb.h" - -namespace tensorflow { -namespace lingvo { -namespace { - -tensorflow::Session* CreateSessionForPreconditioning( - const PreconditionerCaptainOptions& options) { - tensorflow::SessionOptions session_options; - session_options.target = ""; - session_options.config.set_log_device_placement(true); - auto* session = NewSession(session_options); - GraphDef gdef; - CHECK(!options.preconditioner_compute_graphdef.empty()); - gdef.ParseFromString(options.preconditioner_compute_graphdef); - TF_CHECK_OK(session->Create(gdef)); - return session; -} - -const float kEpsilon = 5e-2; - -} // namespace - -PreconditionerCaptain::PreconditionerCaptain( - const PreconditionerCaptainOptions& options) - : options_(options) { - workers_ = absl::make_unique( - tensorflow::Env::Default(), "preconditioners-threads", - options.num_compute_threads); - // NOTE: For distributing svd across all host machines, you may add many - // sessions against other CPU hosts. - sessions_.emplace_back(CreateSessionForPreconditioning(options)); -} - -PreconditionerCaptain::~PreconditionerCaptain() {} - -Tensor PreconditionerCaptain::GetPreconditioner(const std::string& key, - bool* ok) { - mutex_lock l(mu_); - if (preconditioners_.find(key) == preconditioners_.end()) { - *ok = false; - return {}; - } - *ok = true; - return preconditioners_[key]; -} - -void PreconditionerCaptain::InsertGradientStatistics(const std::string& key, - Tensor statistics, - Tensor exponent, - int global_step, - bool sync) { - const int session_to_use = std::hash{}(key) % sessions_.size(); - bool should_calculate_preconditioner = true; - { - mutex_lock l(mu_); - if (gradient_statistics_.find(key) != gradient_statistics_.end()) { - if (gradient_statistics_[key].global_step == global_step) { - should_calculate_preconditioner = false; - } - } - if (should_calculate_preconditioner) { - gradient_statistics_[key] = {global_step, statistics}; - } - } - if (should_calculate_preconditioner) { - auto run_preconditioner = [this, key, session_to_use, global_step, - statistics, exponent] { - { - mutex_lock l(mu_); - ++active_preconditioners_; - } - - Status status; - std::vector outputs; - std::vector> inputs; - inputs.push_back(std::make_pair("input", statistics)); - inputs.push_back(std::make_pair("exponent", exponent)); - do { - outputs.clear(); - LOG(INFO) << "START: inverse pth root for " << key << " @ " - << global_step << " " << statistics.shape().DebugString(); - status = sessions_[session_to_use]->Run(inputs, {"output", "diff"}, {}, - &outputs); - LOG(INFO) << "DONE: Inverse pth root for " << key << " @ " - << global_step << " " << statistics.shape().DebugString(); - LOG(INFO) << status.error_message(); - } while (!status.ok()); - - // Certain matrices cause SVD to have less precision with its calculation - // of inverse pth root. We handle that case by ignoring preconditioners - // for those updates. - if (outputs[1].scalar()() < kEpsilon) { - mutex_lock l(mu_); - preconditioners_[key] = outputs[0]; - LOG(INFO) << "For " << key << " @ " << global_step - << " with diff (u-v):" << outputs[1].scalar()(); - } else { - LOG(INFO) << "Skipping preconditioner update for " << key << " @ " - << global_step - << " with diff (u-v):" << outputs[1].scalar()(); - } - { - mutex_lock l(mu_); - --active_preconditioners_; - } - }; - if (!sync) { - workers_->Schedule(run_preconditioner); - } else { - run_preconditioner(); - } - } -} - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/preconditioner_captain.h b/lingvo/core/ops/preconditioner_captain.h deleted file mode 100644 index 990f5adb..00000000 --- a/lingvo/core/ops/preconditioner_captain.h +++ /dev/null @@ -1,81 +0,0 @@ -/* Copyright 2020 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ -#ifndef LINGVO_CORE_OPS_PRECONDITIONER_CAPTAIN_H_ -#define LINGVO_CORE_OPS_PRECONDITIONER_CAPTAIN_H_ - -#include -#include -#include - -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/platform/threadpool.h" -#include "tensorflow/core/public/session.h" - -namespace tensorflow { -namespace lingvo { - -// PreconditionerCaptain service options. -struct PreconditionerCaptainOptions { - // Number of threads to run; - int32 num_compute_threads = 64; - // Graph that computes the inverse pth root. - string preconditioner_compute_graphdef; -}; - -struct StatisticsValue { - // Global step for entry for the statistics. - int global_step; - // Value of the tensor statistics. - Tensor value; -}; - -class PreconditionerCaptain { - public: - explicit PreconditionerCaptain(const PreconditionerCaptainOptions& options); - ~PreconditionerCaptain(); - - // Disable copy (and move) semantics. - PreconditionerCaptain(const PreconditionerCaptain&) = delete; - PreconditionerCaptain& operator=(const PreconditionerCaptain&) = delete; - - // Insert gradient statistics. - void InsertGradientStatistics(const std::string& key, Tensor statistics, - Tensor exponent, int global_step, bool sync); - - // Get preconditioner with status. - Tensor GetPreconditioner(const std::string& key, bool* ok); - - private: - // Options for the captain. - const PreconditionerCaptainOptions options_; - - // Options to throttle preconditioning. - int32 active_preconditioners_ = 0; - - // Executor used to serve the requests and compute preconditioners. - std::unique_ptr workers_; - // Mutex protecting the statistics, and preconditioners. - mutex mu_; - // A map name to preconditioners - std::unordered_map gradient_statistics_; - std::unordered_map preconditioners_; - // Internal sessions used to compute preconditioners. - std::vector> sessions_; -}; - -} // namespace lingvo -} // namespace tensorflow - -#endif // LINGVO_CORE_OPS_PRECONDITIONER_CAPTAIN_H_ diff --git a/lingvo/core/ops/preconditioner_op_kernels.cc b/lingvo/core/ops/preconditioner_op_kernels.cc deleted file mode 100644 index f275e71e..00000000 --- a/lingvo/core/ops/preconditioner_op_kernels.cc +++ /dev/null @@ -1,152 +0,0 @@ -/* Copyright 2020 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ -#include -#include -#include - -#include "lingvo/core/ops/preconditioner_captain.h" -#include "tensorflow/core/framework/kernel_def_builder.h" -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/types.h" -#include "tensorflow/core/lib/core/errors.h" -#include "tensorflow/core/lib/core/status.h" -#include "tensorflow/core/platform/logging.h" - - -const int kShampooComputeThreads = 64; - -namespace tensorflow { -namespace lingvo { -namespace { - -void MakePreconditionerCaptainOptionsFromContext( - OpKernelConstruction* context, PreconditionerCaptainOptions* options) { - OP_REQUIRES_OK(context, - context->GetAttr("preconditioner_compute_graphdef", - &options->preconditioner_compute_graphdef)); - options->num_compute_threads = kShampooComputeThreads; -} - -} // namespace - -PreconditionerCaptain* global_preconditioner_captain = nullptr; - -void PreconditionerCaptainServiceInit( - const PreconditionerCaptainOptions& options) { - global_preconditioner_captain = new PreconditionerCaptain(options); -} - -PreconditionerCaptain* get_or_create_preconditioner_captain( - const PreconditionerCaptainOptions& options) { - static std::once_flag global_preconditioner_captain_init_once; - std::call_once(global_preconditioner_captain_init_once, - &PreconditionerCaptainServiceInit, options); - return global_preconditioner_captain; -} - -class GetPreconditioners : public OpKernel { - public: - explicit GetPreconditioners(OpKernelConstruction* context) - : OpKernel(context) { - OP_REQUIRES_OK(context, context->GetAttr("keys", &keys_)); - MakePreconditionerCaptainOptionsFromContext(context, &options_); - } - - void Compute(OpKernelContext* context) override { - auto* preconditioner_captain = - get_or_create_preconditioner_captain(options_); - std::vector statuses(keys_.size()); - OpInputList shapes; - OP_REQUIRES_OK(context, context->input_list("shapes", &shapes)); - - OpOutputList output_list; - OP_REQUIRES_OK(context, context->output_list("outputs", &output_list)); - - OpOutputList status_list; - OP_REQUIRES_OK(context, context->output_list("statuses", &status_list)); - - for (int i = 0; i < keys_.size(); ++i) { - bool ok; - Tensor output = preconditioner_captain->GetPreconditioner(keys_[i], &ok); - if (ok) { - output_list.set(i, output); - } else { - auto shape_t = shapes[i].flat(); - TensorShape shape; - TF_CHECK_OK(TensorShapeUtils::MakeShape(shape_t.data(), shape_t.size(), - &shape)); - Tensor zero_output(DT_FLOAT, shape); - zero_output.flat().setZero(); - output_list.set(i, zero_output); - } - Tensor status_t(DT_BOOL, TensorShape({})); - status_t.scalar()() = ok; - status_list.set(i, status_t); - } - } - - private: - // Options for preconditioner. - PreconditionerCaptainOptions options_; - // Keys for the tensors. - std::vector keys_; -}; - -REGISTER_KERNEL_BUILDER(Name("GetPreconditioners").Device(DEVICE_CPU), - GetPreconditioners); - -class ComputePreconditionersOp : public OpKernel { - public: - explicit ComputePreconditionersOp(OpKernelConstruction* context) - : OpKernel(context) { - OP_REQUIRES_OK(context, context->GetAttr("keys", &keys_)); - MakePreconditionerCaptainOptionsFromContext(context, &options_); - OP_REQUIRES_OK(context, context->GetAttr("sync", &sync_)); - } - - void Compute(OpKernelContext* context) override { - auto* preconditioner_captain = - get_or_create_preconditioner_captain(options_); - OpInputList inputs; - OP_REQUIRES_OK(context, context->input_list("inputs", &inputs)); - OpInputList exponents; - OP_REQUIRES_OK(context, context->input_list("exponents", &exponents)); - const Tensor* global_step_t; - OP_REQUIRES_OK(context, context->input("global_step", &global_step_t)); - const int global_step = global_step_t->scalar()(); - - for (int i = 0; i < inputs.size(); ++i) { - Tensor statistics = inputs[i]; - Tensor exponent = exponents[i]; - preconditioner_captain->InsertGradientStatistics( - keys_[i], statistics, exponent, global_step, sync_); - } - } - - private: - // Options for preconditioner. - PreconditionerCaptainOptions options_; - // Keys for the tensors. - std::vector keys_; - // Whether to run preconditioner synchronously. - bool sync_ = false; -}; - -REGISTER_KERNEL_BUILDER(Name("ComputePreconditioners").Device(DEVICE_CPU), - ComputePreconditionersOp); - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/preconditioner_op_kernels_test.py b/lingvo/core/ops/preconditioner_op_kernels_test.py deleted file mode 100644 index 9c75f7a7..00000000 --- a/lingvo/core/ops/preconditioner_op_kernels_test.py +++ /dev/null @@ -1,88 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for preconditioner driver.""" - -import lingvo.compat as tf -from lingvo.core import ops -import numpy as np - - -class PreconditionerTest(tf.test.TestCase): - - def inverse_pth_root(self, input_t, exponent, epsilon=1e-12): - input_t_f64 = tf.cast(input_t, tf.float64) - s, u, v = tf.linalg.svd( - input_t_f64 + - tf.eye(tf.shape(input_t_f64)[0], dtype=tf.float64) * epsilon, - full_matrices=True) - val = tf.matmul( - tf.matmul( - u, - tf.linalg.tensor_diag( - tf.pow(tf.maximum(s, epsilon), tf.cast(exponent, tf.float64)))), - tf.transpose(v)) - return tf.cast(val, tf.float32), tf.reduce_max(tf.abs(u - v)) - - def inverse_pth_root_graph(self, epsilon=1e-12): - graph = tf.Graph() - with graph.as_default(): - exponent_t = tf.placeholder(dtype=tf.float32, name='exponent', shape=None) - input_t = tf.placeholder(dtype=tf.float32, name='input', shape=None) - output, diff = self.inverse_pth_root(input_t, exponent_t, epsilon) - tf.identity(output, 'output') - tf.identity(tf.cast(diff, tf.float32), 'diff') - return graph.as_graph_def().SerializeToString() - - def testPreconditioning(self): - preconditioner_compute_graphdef = self.inverse_pth_root_graph() - with tf.Session(): - global_step = tf.train.get_or_create_global_step() - self.evaluate(tf.global_variables_initializer()) - rand_input_1_t = np.random.rand(4, 4) - rand_input_2_t = np.random.rand(4, 4) - exponents = [-0.25, -0.25] - symmetric_input_1_t = np.dot(rand_input_1_t, rand_input_1_t.transpose()) - symmetric_input_2_t = np.dot(rand_input_2_t, rand_input_2_t.transpose()) - outputs, statuses = ops.get_preconditioners( - [tf.shape(symmetric_input_1_t), - tf.shape(symmetric_input_2_t)], - keys=['a', 'b'], - preconditioner_compute_graphdef=preconditioner_compute_graphdef) - self.assertFalse(any(self.evaluate(statuses))) - preconditioner = ops.compute_preconditioners( - [symmetric_input_1_t, symmetric_input_2_t], - exponents, - tf.cast(global_step, tf.int32), - keys=['a', 'b'], - sync=True, - preconditioner_compute_graphdef=preconditioner_compute_graphdef) - self.assertAllClose(outputs[0].eval(), np.zeros((4, 4)), atol=1e-4) - self.assertAllClose(outputs[1].eval(), np.zeros((4, 4)), atol=1e-4) - preconditioner.run() - self.assertTrue(any(self.evaluate(statuses))) - expected_output_1_t = self.inverse_pth_root(symmetric_input_1_t, - exponents[0]) - expected_output_2_t = self.inverse_pth_root(symmetric_input_2_t, - exponents[1]) - outputs_np = self.evaluate(outputs) - self.assertAllClose( - outputs_np[0], expected_output_1_t[0].eval(), atol=1e-1) - self.assertAllClose( - outputs_np[1], expected_output_2_t[0].eval(), atol=1e-1) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/ops/random_ops_kernels.cc b/lingvo/core/ops/random_ops_kernels.cc deleted file mode 100644 index 15e981f8..00000000 --- a/lingvo/core/ops/random_ops_kernels.cc +++ /dev/null @@ -1,96 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include - -#include "absl/synchronization/mutex.h" -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" - -namespace tensorflow { -namespace lingvo { -namespace { - -class RandomPermutationSequenceOp : public OpKernel { - public: - explicit RandomPermutationSequenceOp(OpKernelConstruction* ctx) - : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, ctx->GetAttr("num", &num_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("batch", &batch_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("repeat", &repeat_)); - int64 seed; - OP_REQUIRES_OK(ctx, ctx->GetAttr("seed", &seed)); - if (seed == 0) { - std::random_device device("/dev/urandom"); - seed = std::mt19937_64(device())(); - } - rnd_.seed(seed); - Fill(); - } - - void Compute(OpKernelContext* ctx) override { - absl::MutexLock l(&mu_); - OP_REQUIRES(ctx, !ids_.empty() || repeat_, - errors::OutOfRange("Epoch ended.")); - if (ids_.empty()) Fill(); - - int start = 0; - int n = std::min(batch_, ids_.size()); - - Tensor* out; - const int out_size = repeat_ ? batch_ : n; - OP_REQUIRES_OK(ctx, ctx->allocate_output(0, TensorShape({out_size}), &out)); - for (int i = 0; i < n; ++i) { - out->flat()(i) = ids_[ids_.size() - 1 - i]; - } - ids_.resize(ids_.size() - n); - start += n; - - while (repeat_ && ids_.empty()) { - Fill(); - n = std::min(batch_ - start, ids_.size()); - for (int i = 0; i < n; ++i) { - out->flat()(start + i) = ids_[ids_.size() - 1 - i]; - } - ids_.resize(ids_.size() - n); - start += n; - } - } - - private: - int32 num_; - int32 batch_; - bool repeat_; - - absl::Mutex mu_; - std::mt19937 rnd_; - std::vector ids_; - - void Fill() { - CHECK(ids_.empty()); - ids_.resize(num_); - for (int i = 0; i < num_; ++i) ids_[i] = i; - for (int i = num_ - 1; i > 0; --i) { - const int32 pos = rnd_() % i; - std::swap(ids_[i], ids_[pos]); - } - } -}; -REGISTER_KERNEL_BUILDER(Name("RandomPermutationSequence").Device(DEVICE_CPU), - RandomPermutationSequenceOp); -} // namespace -} // end namespace lingvo -} // end namespace tensorflow diff --git a/lingvo/core/ops/random_ops_test.py b/lingvo/core/ops/random_ops_test.py deleted file mode 100644 index 0ca00fe5..00000000 --- a/lingvo/core/ops/random_ops_test.py +++ /dev/null @@ -1,62 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for random_ops.""" - -from lingvo import compat as tf -from lingvo.core import ops -from lingvo.core import test_utils - -FLAGS = tf.flags.FLAGS - - -class RandomOpsTest(test_utils.TestCase): - - def testRandomPermutationSequenceRepeat(self): - with self.session(): - out = ops.random_permutation_sequence(num=20, batch=7, repeat=True) - - remaining = list(range(20)) - for _ in range(10): - # Each epoch takes exactly 3 steps. - vals = self.evaluate(out).tolist() + self.evaluate( - out).tolist() + self.evaluate(out).tolist() - self.assertEqual(len(vals), 21) - - # Contains all the remaining values from previous epoch. - for x in remaining: - vals.remove(x) # Raises exception if x is not in vals. - - # Remaining items have no duplicates. - self.assertEqual(len(vals), len(set(vals))) - - remaining = list(set(range(20)) - set(vals)) - - def testRandomPermutationSequenceNoRepeat(self): - with self.session(): - out = ops.random_permutation_sequence(num=20, batch=7, repeat=False) - - # Each epoch takes exactly 3 steps. - vals = self.evaluate(out).tolist() + self.evaluate( - out).tolist() + self.evaluate(out).tolist() - self.assertEqual(list(range(20)), sorted(vals)) - - # repeat=False. We should see OutOfRange error. - with self.assertRaises(tf.errors.OutOfRangeError): - self.evaluate(out) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/ops/record.proto b/lingvo/core/ops/record.proto deleted file mode 100644 index 3d6e7ad3..00000000 --- a/lingvo/core/ops/record.proto +++ /dev/null @@ -1,10 +0,0 @@ -syntax = "proto2"; - -package tensorflow.lingvo; - -import "tensorflow/core/framework/tensor.proto"; - -// A message Record is just a few named tensors. -message Record { - map fields = 1; -} diff --git a/lingvo/core/ops/record_batcher.cc b/lingvo/core/ops/record_batcher.cc deleted file mode 100644 index 376692be..00000000 --- a/lingvo/core/ops/record_batcher.cc +++ /dev/null @@ -1,485 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -// Implementation notes: -// -// * RecordBatcher uses two threadpools. processor_thread_ and -// merger_thread_. -// -// * Processor threads -// -// * call the yielder to get a Record and process it into a TensorVec. -// -// * Processed TensorVec are put into buckets according to the -// bucket key returned by processor->Process(). -// -// * When one bucket is full (according to bucket_batch_limit), all -// TensorVec accumulated in that bucket is handed off to -// to_flush_. -// -// * If to_flush_ is non-empty, the processor thread blocks. -// -// * The only merger thread pulls off TensorVecs from to_flush_ and -// calls processor->Merge() to merge samples into a single batch. It -// then hands the merged batch into curr_. If there is already one -// unconsumed batch, the merger thread blocks. -// -// NOTE: merger_thread_ itself is single-threaded. We expect that if -// processor->Merge() becomes bottleneck (memory copy bounded), we -// can change the implementation of processor->Merge() to leverage -// multi-threaded merge. -// -// -// Peak memory usage estimate given the above algorithm: -// yielder buffer memory // RecordYielder.file_buffer_size -// + all TensorVecs in buckets // Sum of RecordBatcher.bucket_batch_limit -// + to_flush_ // The batch to be merged. -// + to_flush // The batch being merged. -// + merged // The batch being merged. -// + curr_ // The batch to be consumed. -// -// If we assume each sample is roughly M bytes (in its string format -// or tensor format), each output batch has the batch size B, we can -// estimate the peak memory usage of one RecordYielder+RecordBatcher -// is roughly -// -// M * (file_buffer_size + sum(bucket_batch_limit) * 2 + B * 2) - -#include "lingvo/core/ops/record_batcher.h" - -#include -#include - -#include "absl/synchronization/mutex.h" -#include "tensorflow/core/lib/core/errors.h" -#include "tensorflow/core/lib/strings/str_util.h" -#include "tensorflow/core/lib/strings/strcat.h" -#include "tensorflow/core/platform/env.h" -#include "tensorflow/core/platform/errors.h" - -namespace tensorflow { -namespace lingvo { - -RecordBatcher::RecordBatcher(const Options& opts, RecordYielder* yielder, - RecordProcessor* processor) - : opts_(opts), - yielder_(yielder), - processor_(processor), - processor_thread_(new thread::ThreadPool( - Env::Default(), ThreadOptions(), "record_batcher_processor", - opts_.num_threads, /* low_latency_hint */ false)), - merger_thread_(new thread::ThreadPool(Env::Default(), ThreadOptions(), - "record_batcher_merger", 1, - /* low_latency_hint */ false)), - curr_empty_(this, &ME::CurrEmpty), - curr_non_empty_(this, &ME::CurrNonEmpty), - to_flush_empty_(this, &ME::ToFlushEmpty), - to_flush_non_empty_(this, &ME::ToFlushNonEmpty), - bucket_upper_bound_(opts_.bucket_upper_bound) { - CHECK_EQ(opts_.bucket_upper_bound.size(), opts_.bucket_batch_limit.size()); - buckets_.resize(opts_.bucket_upper_bound.size()); - length_histogram_.resize(opts_.bucket_upper_bound.back() + 1, 0); - start_time_ = std::time(nullptr); - { - absl::MutexLock l(&mu_); - last_log_update_time_ = start_time_; - } - - merger_thread_->Schedule([this]() { - MergerLoop(); - absl::MutexLock l(&mu_); - merger_loop_done_ = true; - }); -} - -RecordBatcher::~RecordBatcher() { - { - absl::MutexLock l(&mu_); - stop_ = true; - } - delete processor_thread_; - delete merger_thread_; - yielder_->Close(); - delete processor_; -} - -Status RecordBatcher::EnsureInitialized(OpKernelContext* ctx) { - if (is_initialized_) { - return Status::OK(); - } - TF_RETURN_IF_ERROR(processor_->Initialize(ctx)); - for (int i = 0; i < opts_.num_threads; i++) { - processor_thread_->Schedule([this]() { - ProcessorLoop(); - absl::MutexLock l(&mu_); - processor_loop_done_count_++; - }); - } - is_initialized_ = true; - LOG(INFO) << "batcher initialized"; - return Status::OK(); -} - -Status RecordBatcher::GetNext(OpKernelContext* ctx, int64* bucket, - TensorVec* batch) { - absl::MutexLock l(&mu_); - TF_RETURN_IF_ERROR(EnsureInitialized(ctx)); - // Wait for either curr to be non-empty, or for the merger thread to be - // complete. - WaitForCurrNonEmpty(); - - // If the buffer is still empty, it must be because the merger loop is done - // due to an EoF. - if (curr_.empty()) { - CHECK(merger_loop_done_); - return stop_status_; - } - - *bucket = curr_bucket_; - curr_bucket_ = -1; - using std::swap; - swap(*(batch), curr_); - curr_.clear(); - return Status::OK(); -} - -void RecordBatcher::IncrementHistogram(int64 bucket) { - if (bucket > bucket_upper_bound_.back()) return; - length_histogram_[bucket]++; -} - -void RecordBatcher::AdjustBuckets() { - // The length histogram is too big to compute with quickly. - // We distill it down to a histogram with bins of equal cost (equal area). - int64 ideal_cost = 0; - for (int i = 0; i < length_histogram_.size(); i++) { - ideal_cost += length_histogram_[i] * i; - } - if (ideal_cost == 0) { - return; - } - - // The histogram is pairs of (upper_length_bound, items_in_bucket). - std::vector> compact_histogram; - int64 cost_so_far = 0; - int64 count_in_bucket = 0; - int bucket_index = 0; - const int kCompactBuckets = 500; - for (int i = 0; i < length_histogram_.size(); i++) { - cost_so_far += length_histogram_[i] * i; - count_in_bucket += length_histogram_[i]; - int64 cost_target = ideal_cost * ((bucket_index + 1.0) / kCompactBuckets); - if (cost_so_far >= cost_target && - compact_histogram.size() != kCompactBuckets) { - compact_histogram.push_back(std::make_pair(i, count_in_bucket)); - count_in_bucket = 0; - bucket_index++; - } - } - // Make sure that the last bucket ends at the user-specified upper bound. - compact_histogram.push_back( - std::make_pair(bucket_upper_bound_.back(), count_in_bucket)); - - // Clear the fine-grained histogram to prepare for the next cycle. - length_histogram_.assign(opts_.bucket_upper_bound.back() + 1, 0); - - const int num_lengths = compact_histogram.size(); - const int num_buckets = bucket_upper_bound_.size(); - - // Now we shrink the compact histogram even further using dynamic programming. - // c[i, j, k]: the cumulative cost for the first i histogram groups, if the - // next bucketing point is at index j (j > i), and there are k more bucketing - // points available to use for the remaining elements. - Tensor cost(DT_INT64, TensorShape({num_lengths, num_lengths, num_buckets})); - auto c = cost.tensor(); - - // s(i, j) is the total cost of computing the items in histogram bucket i - // when padding them out to the size of bucket j. - auto s = [&compact_histogram](int i, int j) { - const int64 bucket_j_width = compact_histogram[j].first; - const int64 bucket_i_count = compact_histogram[i].second; - return bucket_i_count * bucket_j_width; - }; - - for (int i = 0; i < num_lengths; i++) { - for (int j = i + 1; j < num_lengths; j++) { - // If there are no buckets left and j is the next bucket, the - // cost is \sum_i 0) { - c(i, j, 0) += c(i - 1, j, 0); - } - // When we have some buckets to use, we can choose to insert bucket - // boundaries to reduce computation. - for (int k = 1; k < num_buckets; k++) { - if (i > 0) { - // When we choose to put a bucket boundary here at position i, we - // can compute these new items with minimal padding [s(i, i)]. - int64 cost_choose = c(i - 1, i, k - 1) + s(i, i); - // If we don't put a bucket boundary here, we have to wait until - // position j, which means extra padding. [s(i, j)]. - int64 cost_not_choose = c(i - 1, j, k) + s(i, j); - c(i, j, k) = std::min(cost_choose, cost_not_choose); - } else { - c(i, j, k) = s(i, j); - } - } - } - } - - std::vector buckets; - buckets.push_back(num_lengths - 1); - const int64 best_cost = c(num_lengths - 2, num_lengths - 1, num_buckets - 1); - int64 remaining_cost = best_cost; - for (int i = num_lengths - 2; i > 0; i--) { - int buckets_left = num_buckets - buckets.size(); - if (buckets_left <= 0) break; - int prev_bucket = buckets.back(); - int64 cost_choose = c(i - 1, i, buckets_left - 1) + s(i, i); - int64 cost_not_choose = - c(i - 1, prev_bucket, buckets_left) + s(i, prev_bucket); - if (remaining_cost == cost_choose) { - buckets.push_back(i); - remaining_cost -= s(i, i); - } else if (remaining_cost == cost_not_choose) { - remaining_cost -= s(i, prev_bucket); - } else { - // This didn't make sense; keep the buckets as they are. - LOG(WARNING) << "AdjustBuckets: backtrace failed."; - return; - } - } - - std::reverse(buckets.begin(), buckets.end()); - - // We keep the same maximum value that the user entered, but all other - // boundaries are updated. - std::vector bucket_strings; - for (int i = 0; i < buckets.size() - 1; i++) { - bucket_upper_bound_[i] = compact_histogram[buckets[i]].first; - bucket_strings.push_back(strings::StrCat(bucket_upper_bound_[i])); - } - bucket_strings.push_back(strings::StrCat(bucket_upper_bound_.back())); - - // Compute the amount of padding waste from choosing this bucket assignment. - LOG(INFO) << "Buckets: [" << str_util::Join(bucket_strings, ", ") << "] " - << "Waste: " - << (best_cost - static_cast(ideal_cost)) / best_cost; -} - -void RecordBatcher::FlushAllBuckets() { - for (int i = 0; i < buckets_.size(); ++i) { - if (!buckets_[i].empty()) { - CHECK_LE(static_cast(buckets_[i].size()), - opts_.bucket_batch_limit[i]); - to_flush_.push_back({i, std::move(buckets_[i])}); - buckets_[i].clear(); - } - } -} - -void RecordBatcher::ProcessorLoop() { - // Multiply next_status_update_duration_seconds_ by 2 every update. - const int64 status_update_duration_multiplier = 2; - std::vector out_of_range_buckets; - while (true) { - { - absl::MutexLock l(&mu_); - if (stop_) { - return; - } - } - - // Get the next record. - Record record; - record.source_id = kDefaultSourceId; - Status s = yielder_->Yield(&record); - // If yielder returns OutOfRange, set - // the out status appropriately and return. - if (errors::IsOutOfRange(s)) { - absl::MutexLock l(&mu_); - stop_status_ = s; - stop_ = true; - return; - } - - if (!s.ok()) { - LOG(WARNING) << s; - continue; - } - - // Parse the record. - int64 bucket; - TensorVec sample; - s = processor_->Process(record, &bucket, &sample); - if (!s.ok()) { - // Print error message. Some example processors use CANCELLED for data - // that are filtered out. Print only first 10 such errors. - if (errors::IsCancelled(s)) { - { - absl::MutexLock l(&mu_); - ++total_records_skipped_; - - static int log_counter = 0; - if (log_counter < 10) { - log_counter++; - LOG(WARNING) << s; - } - } - } else if (errors::IsNotFound(s) || errors::IsPermissionDenied(s)) { - // Terminates program if an unregistered custom op is used by - // the processor, or any access permission denied error. - // - // Consider setting *out_status with s and returning, instead - // of killing program? - LOG(FATAL) << s; - } else { - LOG(WARNING) << s; - } - continue; - } - - absl::MutexLock l(&mu_); - - if (opts_.bucket_adjust_every_n > 0) { - const int64 records_processed = - total_records_yielded_ + total_records_skipped_; - if (records_processed % opts_.bucket_adjust_every_n == 0 && - total_records_yielded_ > 0) { - AdjustBuckets(); - } - IncrementHistogram(bucket); - } - - // Figure out which bucket it belongs to. - auto iter = std::lower_bound(bucket_upper_bound_.begin(), - bucket_upper_bound_.end(), bucket); - - if (iter == bucket_upper_bound_.end()) { - VLOG(1) << "Skip. bucket out-of-range " << bucket; - if (out_of_range_buckets.size() < 10) { - out_of_range_buckets.push_back(bucket); - } - ++total_records_skipped_; - } else { - // Figure out which buckets we should return to the consumer. - // A bucket (id-th) is full. - const int id = iter - bucket_upper_bound_.begin(); - const int64 batch_limit = opts_.bucket_batch_limit[id]; - if (buckets_[id].size() + 1 == batch_limit) { - WaitForToFlushEmpty(); - if (stop_) { - return; - } - } - // Invariant is either we don't need to flush this bucket after adding a - // new element to it, or to_flush_ is empty and we can flush this bucket. - CHECK(buckets_[id].size() + 1 < batch_limit || to_flush_.empty()); - buckets_[id].push_back({bucket, std::move(sample)}); - if (buckets_[id].size() == batch_limit) { - to_flush_.push_back({id, std::move(buckets_[id])}); - buckets_[id].clear(); - } - CHECK_LT(buckets_[id].size(), batch_limit); // invariant. - - ++records_yielded_; - ++total_records_yielded_; - - if (opts_.flush_every_n > 0 && records_yielded_ >= opts_.flush_every_n) { - FlushAllBuckets(); - records_yielded_ = 0; - } - } - - std::time_t current_time = std::time(nullptr); - if (current_time - last_log_update_time_ > - next_status_update_duration_seconds_) { - LOG(INFO) << current_time - start_time_ - << " total seconds passed. Total records yielded: " - << total_records_yielded_ - << ". Total records skipped: " << total_records_skipped_; - for (auto bucket : out_of_range_buckets) { - LOG(INFO) << "Out-of-range sample: " << bucket; - } - CHECK_GT(total_records_yielded_, 0) - << "No records were yielded. Most likely the input is misconfigured!"; - out_of_range_buckets.clear(); - last_log_update_time_ = current_time; - next_status_update_duration_seconds_ *= status_update_duration_multiplier; - } - } -} - -void RecordBatcher::MergerLoop() { - FlushList to_flush; - std::vector samples; - TensorVec merged; - bool continue_loop = true; - while (continue_loop) { - { - absl::MutexLock l(&mu_); - WaitForToFlushNonEmpty(); - if (stop_ && stop_status_.ok()) { - // The object is being destroyed, just exit. - return; - } else if (ProcessorsDone()) { - // The yielder hit EOF and all processors are done. - // Flush all buckets, and then signal to exit the merger - // loop once all items are flushed to curr_. - FlushAllBuckets(); - continue_loop = false; - } - - to_flush = std::move(to_flush_); - to_flush_.clear(); - } - - // Now, flush out batches we accumulated. Typically, to_flush has only 1 - // batch unless flush_every_n is > 0. - for (auto& p : to_flush) { - const int64 id = p.first; - const int32 num = p.second.size(); - Tensor bucket_keys(DT_INT32, {num}); - auto t_bucket_keys = bucket_keys.flat(); - for (int i = 0; i < num; ++i) { - auto processed = p.second[i]; - t_bucket_keys(i) = processed.bucket_key; - samples.push_back(std::move(processed.sample)); - } - merged.clear(); - Status s = processor_->Merge(bucket_upper_bound_[id], samples, &merged); - samples.clear(); - if (!s.ok()) { - LOG(WARNING) << "Failed to create a batch: " << s; - } else { - merged.push_back(bucket_keys); - absl::MutexLock l(&mu_); - WaitForCurrEmpty(); - - // If stopped due to destructor, just exit, since there should be no - // further calls to GetNext(). - if (stop_ && stop_status_.ok()) { - return; - } - curr_bucket_ = id; - curr_ = std::move(merged); - } - } - to_flush.clear(); - } -} - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/record_batcher.h b/lingvo/core/ops/record_batcher.h deleted file mode 100644 index a8d30727..00000000 --- a/lingvo/core/ops/record_batcher.h +++ /dev/null @@ -1,205 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#ifndef LINGVO_CORE_OPS_RECORD_BATCHER_H_ -#define LINGVO_CORE_OPS_RECORD_BATCHER_H_ - -#include -#include - -#include "absl/synchronization/mutex.h" -#include "lingvo/core/ops/record_yielder.h" -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/lib/core/errors.h" -#include "tensorflow/core/lib/core/status.h" -#include "tensorflow/core/lib/core/threadpool.h" -#include "tensorflow/core/platform/macros.h" -#include "tensorflow/core/platform/thread_annotations.h" - -namespace tensorflow { -namespace lingvo { - -// We use a vector of Tensor to represent a training example or a -// batch of merged training examples. In the latter case, by -// convention, the 1st dimension of every Tensor represents the batch -// dimension. -typedef std::vector TensorVec; - -// An interface for processing records yielded by RecordYielder into a -// training example (single one or batched). -class RecordProcessor { - public: - virtual ~RecordProcessor() {} - - // Parses 'record' (typically a protocol buffer) and fills - // in 'sample' a vector of Tensor representing a training example. - // - // 'bucket_key' is an auxilary annotation extracted from the record - // used by RecordBatcher to bucketize training examples. Typically, - // 'bucket_key' is just a length measure about the record. - virtual Status Process(const Record& record, int64* bucket_key, - TensorVec* sample) = 0; - - // Gives a list of training 'samples' returned by Process() and bucketed into - // a bucket with size `bucket_size`, merges them into a single 'batch'. - virtual Status Merge(int64 bucket_size, const std::vector& samples, - TensorVec* batch) = 0; - - // Initializes the RecordProcessor. This should be invoked only once before - // calling 'Process', since some processors expect some initialization with an - // OpKernelContext before running. - virtual Status Initialize(OpKernelContext* ctx) { return Status::OK(); } -}; - -// RecordBatcher takes a RecordYielder, batches the record yielded and -// converts them into training examples. -class RecordBatcher { - public: - // Uses 'yielder' to produce records and 'processor' to produce - // training example batches. - // - // The bucketing scheme is specified by 'bucket_upper_bound' and - // 'bucket_batch_limit'. Records with bucket_key within - // (bucket_upper_bound[i-1], bucket_upper_bound[i]] - // is put into i-th bucket and as soon as i-th bucket contains - // more than bucket_batch_limit[i] samples, RecordBatcher yields - // one training batch. - struct Options { - // REQUIRES: bucket_upper_bound.size() == bucket_batch_limit.size() - std::vector bucket_upper_bound; - std::vector bucket_batch_limit; - - // If non-zero, optimize bucket_upper_bound values (except the last one) - // every n records based on input lengths. - int64 bucket_adjust_every_n = 0; - - // If non-zero, flushes all batches buffered so far every these - // many records are yielded. - int64 flush_every_n = 0; - - // Number of threads to use for record batcher, each thread - // fills separate batches based on bucket limits. - int64 num_threads = 1; - }; - RecordBatcher(const Options& opts, RecordYielder* yielder, - RecordProcessor* processor); - - virtual ~RecordBatcher(); - - // Returns the a training batch in 'batch' and the batch comes out - // from 'bucket_id'-th bucket. - // Processor threads will be launched on the first call to this function. - Status GetNext(OpKernelContext* ctx, int64* bucket_id, TensorVec* batch); - - private: - // Initializes the batcher and launches the processor threads, if not done. - // This should be invoked on the first call of 'GetNext', since some - // processors expect some initialization with an OpKernelContext before - // running. - Status EnsureInitialized(OpKernelContext* ctx); - - typedef RecordBatcher ME; - struct Processed { - int64 bucket_key; - TensorVec sample; - }; - typedef std::vector Batch; - - // FlushList is a list of bucket id and one batch for that bucket. - typedef std::vector> FlushList; - - // Owned. - Options opts_; - RecordYielder* yielder_ = nullptr; - RecordProcessor* processor_ = nullptr; - thread::ThreadPool* processor_thread_ = nullptr; - thread::ThreadPool* merger_thread_ = nullptr; - absl::Mutex mu_; - int64 curr_bucket_ ABSL_GUARDED_BY(mu_) = -1; - TensorVec curr_ ABSL_GUARDED_BY(mu_); - - // True if either the yielder hits EOF or the destructor triggers. - bool stop_ ABSL_GUARDED_BY(mu_) = false; - - // Status is not OK when a yielder hits an EOF. - Status stop_status_ ABSL_GUARDED_BY(mu_); - - // True when the merger thread is finished. - bool merger_loop_done_ ABSL_GUARDED_BY(mu_) = false; - - bool is_initialized_ = false; - absl::Condition curr_empty_; - absl::Condition curr_non_empty_; - int64 records_yielded_ ABSL_GUARDED_BY(mu_) = 0; - int64 total_records_yielded_ ABSL_GUARDED_BY(mu_) = 0; - int64 total_records_skipped_ ABSL_GUARDED_BY(mu_) = 0; - std::vector buckets_ ABSL_GUARDED_BY(mu_); - int64 processor_loop_done_count_ ABSL_GUARDED_BY(mu_) = 0; - FlushList to_flush_ ABSL_GUARDED_BY(mu_); - absl::Condition to_flush_empty_; - absl::Condition to_flush_non_empty_; - std::time_t start_time_; // Not necessary to guard. - std::time_t last_log_update_time_ ABSL_GUARDED_BY(mu_); - int64 next_status_update_duration_seconds_ ABSL_GUARDED_BY(mu_) = 60; - - std::vector length_histogram_; - std::vector bucket_upper_bound_; - - // Conditions. - bool CurrEmpty() const ABSL_SHARED_LOCKS_REQUIRED(mu_) { - return ((stop_ && stop_status_.ok()) || // The object is being destroyed - curr_.empty()); // We can push work onto curr_. - } - - bool CurrNonEmpty() const ABSL_SHARED_LOCKS_REQUIRED(mu_) { - return (!curr_.empty() || // There is data to deliver to GetNext(). - merger_loop_done_); // There merger loop is done (no more data). - } - - bool ToFlushEmpty() const ABSL_SHARED_LOCKS_REQUIRED(mu_) { - return stop_ || to_flush_.empty(); - } - - bool ToFlushNonEmpty() const ABSL_SHARED_LOCKS_REQUIRED(mu_) { - return ((stop_ && stop_status_.ok()) || // The object is being destroyed. - !to_flush_.empty() || // There is work to flush. - ProcessorsDone()); // All processor threads have exited. - } - - bool ProcessorsDone() const ABSL_SHARED_LOCKS_REQUIRED(mu_) { - return processor_loop_done_count_ == opts_.num_threads; - } - - void ProcessorLoop(); - void MergerLoop(); - - void AdjustBuckets() ABSL_EXCLUSIVE_LOCKS_REQUIRED(mu_); - void FlushAllBuckets() ABSL_EXCLUSIVE_LOCKS_REQUIRED(mu_); - void IncrementHistogram(int64 bucket) ABSL_EXCLUSIVE_LOCKS_REQUIRED(mu_); - - // For performance debugging. - void WaitForCurrEmpty() ABSL_EXCLUSIVE_LOCKS_REQUIRED(mu_); - void WaitForCurrNonEmpty() ABSL_EXCLUSIVE_LOCKS_REQUIRED(mu_); - void WaitForToFlushEmpty() ABSL_EXCLUSIVE_LOCKS_REQUIRED(mu_); - void WaitForToFlushNonEmpty() ABSL_EXCLUSIVE_LOCKS_REQUIRED(mu_); - - TF_DISALLOW_COPY_AND_ASSIGN(RecordBatcher); -}; - -} // namespace lingvo -} // namespace tensorflow - -#endif // LINGVO_CORE_OPS_RECORD_BATCHER_H_ diff --git a/lingvo/core/ops/record_batcher_test.cc b/lingvo/core/ops/record_batcher_test.cc deleted file mode 100644 index 4620fcf6..00000000 --- a/lingvo/core/ops/record_batcher_test.cc +++ /dev/null @@ -1,316 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/record_batcher.h" - -#include -#include "lingvo/core/ops/input_common.h" -#include "lingvo/core/ops/sequential_record_yielder.h" -#include "tensorflow/core/lib/core/errors.h" -#include "tensorflow/core/lib/io/path.h" -#include "tensorflow/core/lib/io/record_writer.h" -#include "tensorflow/core/lib/strings/stringprintf.h" - -namespace tensorflow { -namespace lingvo { - -void GenerateTestData(const string& filename, int n, bool random_value) { - std::unique_ptr file; - TF_CHECK_OK(Env::Default()->NewWritableFile(filename, &file)); - io::RecordWriter writer(file.get()); - for (int i = 0; i < n; ++i) { - if (random_value) { - const string val(1 + (i % 100), 'x'); // Length of [1 .. 100] - TF_CHECK_OK(writer.WriteRecord(val)); - } else { - const string val = strings::Printf("%010d", i); - TF_CHECK_OK(writer.WriteRecord(val)); - } - } -} - -class TestRP : public RecordProcessor { - public: - TestRP() {} - - ~TestRP() override {} - - Status Process(const Record& record, int64* bucket_key, - TensorVec* sample) override { - const string val = string(record.value); - *bucket_key = val.size(); - Tensor t(DT_STRING, {}); - t.scalar()().append(std::string(record.value)); - Tensor ids(DT_STRING, {1}); - auto lab = ids.flat(); - lab(0) = absl::StrCat(record.source_id); - sample->clear(); - sample->push_back(std::move(t)); - sample->push_back(std::move(ids)); - return Status::OK(); - } - - Status Merge(int64 bucket_size, const std::vector& samples, - TensorVec* batch) override { - const int64 n = samples.size(); - Tensor t(DT_STRING, {n}); - Tensor source_ids(DT_STRING, {n}); - for (int i = 0; i < samples.size(); ++i) { - t.flat()(i) = samples[i][0].scalar()(); - source_ids.flat()(i) = samples[i][1].scalar()(); - } - batch->clear(); - batch->push_back(std::move(t)); - batch->push_back(std::move(source_ids)); - return Status::OK(); - } -}; - -TEST(RecordBatcher, Basic) { - const string filename = io::JoinPath("/tmp", "basic"); - GenerateTestData(filename, 1000, true /* random_value */); - - BasicRecordYielder::Options yopts; - yopts.file_pattern = strings::StrCat("tfrecord:", filename); - yopts.seed = 301; - yopts.bufsize = 10; - yopts.parallelism = 1; - - RecordBatcher::Options bopts; - bopts.bucket_upper_bound = {20, 50, 90, 95}; - bopts.bucket_batch_limit = {8, 4, 2, 1}; - - RecordBatcher batcher(bopts, BasicRecordYielder::New(yopts), new TestRP()); - int64 bucket_id; - TensorVec batch; - for (int i = 0; i < 1000; ++i) { - TF_CHECK_OK(batcher.GetNext(/*ctx=*/nullptr, &bucket_id, &batch)); - ASSERT_LE(0, bucket_id); - ASSERT_LT(bucket_id, bopts.bucket_upper_bound.size()); - const Tensor& t = batch[0]; - ASSERT_EQ(t.dims(), 1); - ASSERT_LE(t.dim_size(0), bopts.bucket_batch_limit[bucket_id]); - const Tensor& source_ids = batch[1]; - ASSERT_EQ(source_ids.dims(), 1); - ASSERT_LE(source_ids.dim_size(0), bopts.bucket_batch_limit[bucket_id]); - int maxlen = 0; - for (int j = 0; j < t.dim_size(0); ++j) { - auto len = t.vec()(j).size(); - EXPECT_LE(len, bopts.bucket_upper_bound[bucket_id]); - if (bucket_id != 0) { - EXPECT_LT(bopts.bucket_upper_bound[bucket_id - 1], len); - } - maxlen = std::max(maxlen, len); - ASSERT_EQ(source_ids.vec()(j), "0"); - } - VLOG(1) << bucket_id << " " << t.dim_size(0) << " " << maxlen; - } -} - -TEST(RecordBatcher, BasicMultiThread) { - const string filename = io::JoinPath("/tmp", "basic"); - GenerateTestData(filename, 1000, true /* random_value */); - - BasicRecordYielder::Options yopts; - yopts.file_pattern = strings::StrCat("tfrecord:", filename); - yopts.seed = 301; - yopts.bufsize = 10; - yopts.parallelism = 1; - - RecordBatcher::Options bopts; - bopts.bucket_upper_bound = {20, 50, 90, 95}; - bopts.bucket_batch_limit = {8, 4, 2, 1}; - bopts.num_threads = 4; - - RecordBatcher batcher(bopts, BasicRecordYielder::New(yopts), new TestRP()); - int64 bucket_id; - TensorVec batch; - for (int i = 0; i < 1000; ++i) { - TF_CHECK_OK(batcher.GetNext(/*ctx=*/nullptr, &bucket_id, &batch)); - ASSERT_LE(0, bucket_id); - ASSERT_LT(bucket_id, bopts.bucket_upper_bound.size()); - const Tensor& t = batch[0]; - ASSERT_EQ(t.dims(), 1); - ASSERT_LE(t.dim_size(0), bopts.bucket_batch_limit[bucket_id]); - int maxlen = 0; - for (int j = 0; j < t.dim_size(0); ++j) { - auto len = t.vec()(j).size(); - EXPECT_LE(len, bopts.bucket_upper_bound[bucket_id]); - if (bucket_id != 0) { - EXPECT_LT(bopts.bucket_upper_bound[bucket_id - 1], len); - } - maxlen = std::max(maxlen, len); - } - VLOG(1) << bucket_id << " " << t.dim_size(0) << " " << maxlen; - } -} - -TEST(RecordBatcher, LearnBuckets) { - const string filename = io::JoinPath("/tmp", "basic"); - GenerateTestData(filename, 1000, true /* random_value */); - - BasicRecordYielder::Options yopts; - yopts.file_pattern = strings::StrCat("tfrecord:", filename); - yopts.seed = 301; - yopts.bufsize = 10; - yopts.parallelism = 1; - - RecordBatcher::Options bopts; - bopts.bucket_upper_bound = {100, 100, 100, 100}; - bopts.bucket_batch_limit = {8, 8, 8, 8}; - bopts.bucket_adjust_every_n = 550; - - RecordBatcher batcher(bopts, BasicRecordYielder::New(yopts), new TestRP()); - int64 bucket_id; - TensorVec batch; - - // For the first 500 batches we just make sure the batches are the right - // size. - for (int i = 0; i < 500; ++i) { - TF_CHECK_OK(batcher.GetNext(/*ctx=*/nullptr, &bucket_id, &batch)); - ASSERT_LE(0, bucket_id); - const Tensor& t = batch[0]; - ASSERT_EQ(8, t.dim_size(0)); - } - - // For the next 1000 batches we measure the max length distribution. - std::vector maxlens; - std::vector batches; - maxlens.resize(4, 0); - batches.resize(4, 0); - for (int i = 0; i < 1000; ++i) { - TF_CHECK_OK(batcher.GetNext(/*ctx=*/nullptr, &bucket_id, &batch)); - const Tensor& t = batch[0]; - int maxlen = 0; - for (int j = 0; j < t.dim_size(0); ++j) { - int len = t.vec()(j).size(); - maxlen = std::max(maxlen, len); - } - maxlens[bucket_id] += maxlen; - batches[bucket_id]++; - } - - // The data has a uniform distribution of [1 .. 100]. So we expect - // bucket boundaries around 25, 50, 75, 100, and roughly equal numbers of - // batches of each ID. - EXPECT_NEAR(250, batches[0], 25); - EXPECT_NEAR(250, batches[1], 25); - EXPECT_NEAR(250, batches[2], 25); - EXPECT_NEAR(250, batches[3], 25); - - EXPECT_NEAR(25, maxlens[0] / batches[0], 5); - EXPECT_NEAR(50, maxlens[1] / batches[1], 5); - EXPECT_NEAR(75, maxlens[2] / batches[2], 5); - EXPECT_NEAR(100, maxlens[3] / batches[3], 5); -} - -TEST(RecordBatcher, FullEpoch) { - const int N = 1000; - const string filename = - io::JoinPath("/tmp", "full_epoch"); - GenerateTestData(filename, N, false /* random_value */); - - BasicRecordYielder::Options yopts; - yopts.file_pattern = strings::StrCat("tfrecord:", filename); - yopts.seed = 301; - yopts.bufsize = 10; - yopts.parallelism = 1; - - RecordBatcher::Options bopts; - bopts.bucket_upper_bound = {20, 50, 90, 120}; - bopts.bucket_batch_limit = {8, 4, 2, 1}; - bopts.flush_every_n = N; // Same number of records in the data file. - - RecordBatcher batcher(bopts, BasicRecordYielder::New(yopts), new TestRP()); - int64 bucket_id; - TensorVec batch; - std::vector records; - while (records.size() < N) { - TF_CHECK_OK(batcher.GetNext(/*ctx=*/nullptr, &bucket_id, &batch)); - const Tensor& t = batch[0]; - for (int j = 0; j < t.dim_size(0); ++j) { - records.push_back(t.vec()(j)); - } - } - ASSERT_EQ(N, records.size()); - // We expect to see exactly non-duplicated N records. - std::sort(records.begin(), records.end()); - for (int i = 0; i < N; ++i) { - EXPECT_EQ(strings::Printf("%010d", i), records[i]); - } -} - -TEST(RecordBatcher, CaptureYielderStatus) { - const int N = 50; - const string filename = io::JoinPath("/tmp", "full_epoch"); - GenerateTestData(filename, N, false /* random_value */); - - RecordBatcher::Options bopts; - bopts.bucket_upper_bound = {1000000000}; - bopts.bucket_batch_limit = {1}; - bopts.num_threads = 1; - - const string file_pattern = strings::StrCat("tfrecord:", filename); - const int num_epochs = 2; - RecordBatcher batcher(bopts, - SequentialRecordYielder::New(file_pattern, num_epochs), - new TestRP()); - int64 bucket_id; - TensorVec batch; - std::vector records; - // Fetch N * num_epochs worth of data, which should all be there. - // Note that when there are multiple streams, we need repeat it - // (batch.size() - 1) times. - while (records.size() < num_epochs * N * (batch.size() - 1)) { - TF_CHECK_OK(batcher.GetNext(/*ctx=*/nullptr, &bucket_id, &batch)); - for (int i = 0; i < batch.size(); ++i) { - const Tensor& t = batch[i]; - if (t.dtype() == DT_STRING) { - for (int j = 0; j < t.dim_size(0); ++j) { - records.push_back(t.vec()(j)); - } - } - } - } - - // With a sequential record yielder, the next call will exhaust the - // repeat count of the iterator, and so we should expect that no more - // data can be yielded. - Status s = batcher.GetNext(/*ctx=*/nullptr, &bucket_id, &batch); - ASSERT_TRUE(errors::IsOutOfRange(s)); -} - -TEST(RecordBatcher, SequentialEoFImmediately) { - const string filename = io::JoinPath("/tmp", "full_epoch"); - // Generate no data. - GenerateTestData(filename, 0, false /* random_value */); - - RecordBatcher::Options bopts; - bopts.bucket_upper_bound = {1000000000}; - bopts.bucket_batch_limit = {1}; - bopts.num_threads = 1; - const string file_pattern = strings::StrCat("tfrecord:", filename); - const int num_epochs = 1; - RecordBatcher batcher(bopts, - SequentialRecordYielder::New(file_pattern, num_epochs), - new TestRP()); - int64 bucket_id; - TensorVec batch; - Status s = batcher.GetNext(/*ctx=*/nullptr, &bucket_id, &batch); - ASSERT_TRUE(errors::IsOutOfRange(s)); -} - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/record_debug.cc b/lingvo/core/ops/record_debug.cc deleted file mode 100644 index e42592ad..00000000 --- a/lingvo/core/ops/record_debug.cc +++ /dev/null @@ -1,76 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/record_batcher.h" -#include "lingvo/core/ops/record_yielder.h" -#include "tensorflow/core/platform/env.h" - -namespace tensorflow { -namespace lingvo { - -void BasicRecordYielder::WaitForBufEnough() { - if (!BufEnough()) { - auto start = Env::Default()->NowMicros(); - mu_.Await(buf_enough_); - VLOG(1) << "Wait for buf containing enough records: " - << (Env::Default()->NowMicros() - start) * 1e-6 - << " Hint: Check network condition (e.g., are files in the same " - << "data center) and/or increase file_parallelism."; - } -} - -void RecordBatcher::WaitForCurrEmpty() { - if (!CurrEmpty()) { - auto start = Env::Default()->NowMicros(); - mu_.Await(curr_empty_); - VLOG(2) - << "Wait for curr empty: " - << (Env::Default()->NowMicros() - start) * 1e-6 - << " Hint: Processing is not fast enough to consume example batches."; - } -} - -void RecordBatcher::WaitForCurrNonEmpty() { - if (!CurrNonEmpty()) { - auto start = Env::Default()->NowMicros(); - mu_.Await(curr_non_empty_); - VLOG(1) << "Wait for curr non empty: " - << (Env::Default()->NowMicros() - start) * 1e-6 - << " Hint: Consider improving Merge() method."; - } -} - -void RecordBatcher::WaitForToFlushEmpty() { - if (!ToFlushEmpty()) { - auto start = Env::Default()->NowMicros(); - mu_.Await(to_flush_empty_); - VLOG(3) << "Wait for to_flush empty: " - << (Env::Default()->NowMicros() - start) * 1e-6 - << " Hint: Expected to be the common case."; - } -} - -void RecordBatcher::WaitForToFlushNonEmpty() { - if (!ToFlushNonEmpty()) { - auto start = Env::Default()->NowMicros(); - mu_.Await(to_flush_non_empty_); - VLOG(1) << "Wait for to_flush non empty: " - << (Env::Default()->NowMicros() - start) * 1e-6 - << " Hint: Increase num_batcher_thread."; - } -} - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/record_yielder.cc b/lingvo/core/ops/record_yielder.cc deleted file mode 100644 index e30e1803..00000000 --- a/lingvo/core/ops/record_yielder.cc +++ /dev/null @@ -1,584 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/record_yielder.h" - -#include // NOLINT(build/c++11) -#include -#include -#include // NOLINT(build/c++11) -#include - -#include "absl/synchronization/mutex.h" -#include "lingvo/core/ops/versioned_file_set.pb.h" -#include "tensorflow/core/lib/core/status.h" -#include "tensorflow/core/lib/hash/hash.h" -#include "tensorflow/core/lib/io/buffered_inputstream.h" -#include "tensorflow/core/lib/io/compression.h" -#include "tensorflow/core/lib/io/path.h" -#include "tensorflow/core/lib/io/random_inputstream.h" -#include "tensorflow/core/lib/io/record_reader.h" -#include "tensorflow/core/lib/strings/numbers.h" -#include "tensorflow/core/lib/strings/str_util.h" -#include "tensorflow/core/lib/strings/strcat.h" -#include "tensorflow/core/lib/strings/stringprintf.h" -#include "tensorflow/core/platform/env.h" - -namespace tensorflow { -namespace lingvo { - -namespace { - -// Number of records to batch for a single call to Add. -const int kRecordsPerAdd = 16; - -struct Factory { - absl::Mutex mu; - std::unordered_map creators; - std::unordered_map - pattern_parsers; -}; - -Factory* GetFactory() { - static Factory* factory = new Factory; - return factory; -} - -// A sharded file pattern looks like /path/name@100, which is expanded to -// a glob pattern /path/name-?????-of-00100 by this function. The number of -// shards shouldn't exceed 5 digits. A wildcard is also support for the shards -// number: /path/name@* is expanded to /path/name-?????-of-?????. -Status MaybeExpandShardedFilePattern(const string& file_pattern, - string* expanded) { - const auto pos = file_pattern.find('@'); - if (pos == string::npos) { // not a sharded file pattern - *expanded = file_pattern; - return Status::OK(); - } - - const string prefix = file_pattern.substr(0, pos); - const string suffix = file_pattern.substr(pos + 1); - - uint32 num_shards = 0; - if (suffix == "*") { - *expanded = strings::StrCat(prefix, "-\?\?\?\?\?-of-\?\?\?\?\?"); - return Status::OK(); - } else if (strings::safe_strtou32(suffix, &num_shards)) { - if (num_shards > 99999) { - return errors::InvalidArgument(strings::StrCat( - "The number of shards should not exceed 5 digits: ", num_shards)); - } - *expanded = - strings::Printf("%s-\?\?\?\?\?-of-%05d", prefix.c_str(), num_shards); - return Status::OK(); - } else { - return errors::InvalidArgument( - strings::StrCat("Invalid sharded file pattern: ", file_pattern)); - } -} - -// ParallelFilePatterns look like this -// /a-*-of-10;/b-*-of-10,/c-*-of-10;/d-*-of-10 -// Each "," separated pattern is a parallel file pattern. -// In a given parallel file pattern, all the ";" separated file patterns -// should have the same number of shards. This method creates ";" separated -// filenames aligning the shards. e,g. /a-01-of-10;/b-01-of-10 -// /a-02-of-10;/b-02-of-10 -// /c-01-of-10;/d-01-of-10 -// /c-02-of-10;/d-02-of-10 -Status MatchParallelFilePattern(const string& parallel_file_pattern, - std::vector* filenames) { - std::vector parallel_filenames; - for (const auto& file_pattern : str_util::Split(parallel_file_pattern, ';')) { - string expanded_file_pattern; - TF_RETURN_IF_ERROR( - MaybeExpandShardedFilePattern(file_pattern, &expanded_file_pattern)); - std::vector filenames_per_pattern; - TF_RETURN_IF_ERROR(Env::Default()->GetMatchingPaths( - expanded_file_pattern, &filenames_per_pattern)); - if (parallel_filenames.empty()) { - parallel_filenames.swap(filenames_per_pattern); - continue; - } - if (parallel_filenames.size() != filenames_per_pattern.size()) { - return Status(tensorflow::error::INVALID_ARGUMENT, - "All file patterns in the parallel file pattern do not " - "have the same number of elements."); - } - for (auto it1 = parallel_filenames.begin(), - it2 = filenames_per_pattern.begin(); - it1 != parallel_filenames.end(); ++it1, ++it2) { - strings::StrAppend(&(*it1), ";", *it2); - } - } - filenames->insert(filenames->end(), - std::make_move_iterator(parallel_filenames.begin()), - std::make_move_iterator(parallel_filenames.end())); - - return Status::OK(); -} - -// Input to CkptFilePattern should look like /ckpt -// This pattern parser will read the ckpt file, decode it into a -// tensorflow::CheckpointState proto and use the latest model_checkpoint_path -// as the actual file-pattern. -// model_checkpoint_path should be a legitimate file-pattern, for eg. -// /a@* or /a or /a-?????-of-????? etc. -Status GetFilePatternsFromCkptFile(const string& fileset_file, - std::vector* filenames) { - string fileset_str; - VersionedFileSet fileset; - TF_RETURN_IF_ERROR( - ReadFileToString(Env::Default(), fileset_file, &fileset_str)); - - CHECK(google::protobuf::TextFormat::ParseFromString(fileset_str, &fileset)); - - string prefix = std::string(io::Dirname(fileset_file)); - for (const auto& file_pattern : fileset.current().file_pattern()) { - TF_RETURN_IF_ERROR(MatchParallelFilePattern( - io::JoinPath(prefix, file_pattern), filenames)); - } - return Status::OK(); -} - -} // end namespace - -bool RecordIterator::Register(const string& type_name, FactoryMethod method) { - return RegisterWithPatternParser(type_name, std::move(method), - RecordIterator::PatternParserMethod()); -} - -bool RecordIterator::RegisterWithPatternParser( - const string& type_name, FactoryMethod method, - PatternParserMethod parser_method) { - Factory* factory = GetFactory(); - absl::MutexLock l(&factory->mu); - bool ret = factory->creators.insert({type_name, std::move(method)}).second; - CHECK(ret) << "Possibly duplicated registration: " << type_name; - if (parser_method) { - factory->pattern_parsers.insert({type_name, std::move(parser_method)}); - } - return ret; -} - -RecordIterator* RecordIterator::New(const string& type_name, - const string& filename) { - Factory* factory = GetFactory(); - RecordIterator::FactoryMethod method; - { - absl::MutexLock l(&factory->mu); - const auto iter = factory->creators.find(type_name); - CHECK(iter != factory->creators.end()) - << "Unable to create RecordIterator for format \"" << type_name << "\""; - method = iter->second; - } - return method(filename); -} - -// Returns the prefix in a file pattern, or an empty string if not exist. -// Example: "tfrecord:data_dir/data.tfrecord" => "tfrecord" -string RecordIterator::GetFilePatternPrefix(const string& file_pattern) { - const auto prefix_end = file_pattern.find(':'); - if (prefix_end == string::npos) { - return ""; - } - // The prefix should not contain '/'. If so, it's likely part of the path. - const auto first_slash = file_pattern.find('/'); - if (first_slash != string::npos && first_slash < prefix_end) { - return ""; - } - return file_pattern.substr(0, prefix_end); -} - -string RecordIterator::StripPrefixFromFilePattern(string* file_pattern) { - const string prefix = RecordIterator::GetFilePatternPrefix(*file_pattern); - if (!prefix.empty()) file_pattern->erase(0, prefix.size() + 1); - return prefix; -} - -Status RecordIterator::ParsePattern(const string& type_name, - const string& file_pattern_list, - std::vector* filenames) { - Factory* factory = GetFactory(); - RecordIterator::PatternParserMethod parser_method; - { - absl::MutexLock l(&factory->mu); - const auto iter = factory->pattern_parsers.find(type_name); - if (iter != factory->pattern_parsers.end()) { - parser_method = iter->second; - } - } - if (parser_method) { - return parser_method(file_pattern_list, filenames); - } - for (const auto& file_pattern : str_util::Split(file_pattern_list, ',')) { - std::vector files_per_glob; - TF_RETURN_IF_ERROR( - MatchParallelFilePattern(file_pattern, &files_per_glob)); - filenames->insert(filenames->end(), files_per_glob.begin(), - files_per_glob.end()); - } - return Status::OK(); -} - -RandomAccessFile* OpenOrDie(const string& filename) { - std::unique_ptr file; - TF_CHECK_OK(Env::Default()->NewRandomAccessFile(filename, &file)); - return file.release(); -} - -class PlainTextIterator : public RecordIterator { - public: - PlainTextIterator(const string& filename) - : file_(OpenOrDie(filename)), - stream_(file_.get()), - buf_(&stream_, 2 << 20) {} - - bool Next(string* key, Rope* value) override { - Status s = buf_.ReadLine(&line_); - if (errors::IsOutOfRange(s)) return false; - TF_CHECK_OK(s); - *key = strings::Printf("%08lld", static_cast(num_++)); - *value = line_; - return true; - } - - private: - std::unique_ptr file_; - io::RandomAccessInputStream stream_; - io::BufferedInputStream buf_; - int64 num_ = 0; - string line_; -}; - -class TFRecordIterator : public RecordIterator { - public: - TFRecordIterator(const string& filename, const string& compression_type) - : file_(OpenOrDie(filename)), - reader_(file_.get(), ReaderOptions(compression_type)) {} - - bool Next(string* key, Rope* value) override { - Status s = reader_.ReadRecord(&record_); - if (errors::IsOutOfRange(s)) return false; - *key = strings::Printf("%08lld", static_cast(num_++)); - *value = Rope(record_); - return true; - } - - private: - std::unique_ptr file_; - io::SequentialRecordReader reader_; - int64 num_ = 0; - tstring record_; - - io::RecordReaderOptions ReaderOptions(const string& compression_type) { - auto opts = - io::RecordReaderOptions::CreateRecordReaderOptions(compression_type); - opts.buffer_size = 2LL << 20; // 2MB. - return opts; - } -}; - -// An iterator generates [0 .. max_). -class IotaIterator : public RecordIterator { - public: - IotaIterator(const string& filename) { - if (filename.empty() || !strings::safe_strto64(filename, &max_)) { - max_ = kint64max; - } - } - - bool Next(string* key, Rope* value) override { - if (num_ >= max_) { - return false; - } - *key = strings::Printf("%010lld", static_cast(num_++)); - *value = *key; - return true; - } - - private: - int64 max_ = kint64max; - int64 num_ = 0; -}; - -namespace { - -bool register_text_iterator = RecordIterator::Register( - "text", - [](const string& filename) { return new PlainTextIterator(filename); }); - -// Iterator for plain text files that checks indirect "ckpt" file for latest -// pointer to data files. -// For example, if filename is .../checkpoint, the pattern parser will open the -// checkpoint file and read file-patterns that it points to. If empty, it will -// return error and die. -bool register_indirect_text_iterator = - RecordIterator::RegisterWithPatternParser( - "text_indirect", - [](const string& filename) { return new PlainTextIterator(filename); }, - [](const string& pattern, std::vector* outs) { - TF_RETURN_IF_ERROR(GetFilePatternsFromCkptFile(pattern, outs)); - return Status::OK(); - }); - -bool register_tf_record_iterator = - RecordIterator::Register("tfrecord", [](const string& filename) { - return new TFRecordIterator(filename, io::compression::kNone); - }); - -bool register_tf_record_gzip_iterator = - RecordIterator::Register("tfrecord_gzip", [](const string& filename) { - return new TFRecordIterator(filename, io::compression::kGzip); - }); - -bool register_iota_iterator = RecordIterator::RegisterWithPatternParser( - "iota", [](const string& filename) { return new IotaIterator(filename); }, - [](const string& pattern, std::vector* outs) { - // The pattern is just a stringified number. - *outs = {pattern}; - return Status::OK(); - }); - -} // namespace - -RecordYielder::~RecordYielder() {} - -BasicRecordYielder* BasicRecordYielder::New(Options opts) { - auto yielder = new BasicRecordYielder(opts); - yielder->Start(); - return yielder; -} - -BasicRecordYielder::BasicRecordYielder(const Options& opts) - : opts_(opts), - thread_(new thread::ThreadPool(Env::Default(), ThreadOptions(), - "record_yielder", 2 + opts.parallelism, - /* low_latency_hint */ false)), - epoch_(1), - rnd_(opts.seed), - yields_(0), - buf_empty_(this, &ME::BufEmpty), - buf_not_full_(this, &ME::BufNotFull), - buf_enough_(this, &ME::BufEnough) { - LOG(INFO) << this << " Record yielder start"; - if (opts_.seed == 0) { - LOG(INFO) << "Randomly seed RecordYielder."; - rnd_.seed(std::random_device{}()); - } - file_type_ = RecordIterator::StripPrefixFromFilePattern(&opts_.file_pattern); - if (opts_.bufsize_in_seconds > 0) { - bufsize_ = kRecordsPerAdd * opts_.parallelism; - } else { - bufsize_ = opts_.bufsize; - } -} - -BasicRecordYielder::BasicRecordYielder() - : buf_empty_(this, &ME::BufEmpty), - buf_not_full_(this, &ME::BufNotFull), - buf_enough_(this, &ME::BufEnough) {} // USED ONLY FOR TESTS. -BasicRecordYielder::~BasicRecordYielder() {} - -void BasicRecordYielder::Start() { - thread_->Schedule([this]() { MainLoop(); }); -} - -void BasicRecordYielder::Close() { - { - absl::MutexLock l(&mu_); - stop_ = true; - } - main_loop_done_.WaitForNotification(); - delete thread_; - thread_ = nullptr; - LOG(INFO) << this << "Basic record yielder exit"; - delete this; -} - -Status BasicRecordYielder::Yield(Record* record) { - absl::MutexLock l(&mu_); - WaitForBufEnough(); - ++yields_; - - if (status_.ok()) { - CHECK(!stop_ && !buf_.empty()); - ExtractValue(&record->value); - if (epoch_end_ && buf_.empty()) { - ++epoch_; - } - record->source_id = static_cast(opts_.source_id); - ++num_records_yielded_in_epoch_; - } - return status_; -} - -bool BasicRecordYielder::ShouldFinish(const Status& s) { - absl::MutexLock l(&mu_); - status_.Update(s); - return stop_ || !status_.ok(); -} - -void BasicRecordYielder::AdjustBufferSizeLoop() { - if (opts_.bufsize_in_seconds == 0) { - // Nothing to do for a fixed buffer size. - return; - } - - // Wake up each second and adjust the buffer size. - while (true) { - { - absl::MutexLock l(&mu_); - - // Quit if requested. - if (stop_) break; - - // Smoothed bufsize_ estimate based on current yields_ requests. - // With this set of parameters, the contribution of the current buffer - // size decays as follows: - // 10 seconds -> 90% (.99^10) - // 1 minute -> 50% (.99^60) - // 5 minutes -> 5% (.99^300) - bufsize_ = 0.99 * static_cast(bufsize_) + - 0.01 * yields_ * opts_.bufsize_in_seconds; - - // Make sure the buffer is large enough to hold one batch of Add records - // per thread. - bufsize_ = std::max(opts_.parallelism * kRecordsPerAdd, bufsize_); - - // Make sure the buffer is not larger than the bufsize parameter in the - // options. - if (opts_.bufsize > 0) { - bufsize_ = std::min(opts_.bufsize, bufsize_); - } - VLOG(1) << "Yields:" << yields_ << " Bufsize:" << bufsize_ - << " Pattern:" << opts_.file_pattern; - - // Reset yields_ to zero to count another second of requests. - yields_ = 0; - } - std::this_thread::sleep_for(std::chrono::seconds(1)); - } -} - -void BasicRecordYielder::MainLoop() { - absl::Notification adjust_done; - thread_->Schedule([this, &adjust_done]() { - AdjustBufferSizeLoop(); - adjust_done.Notify(); - }); - - while (true) { - num_records_yielded_in_epoch_ = 0; - LOG(INFO) << "Epoch " << current_epoch() << " " << opts_.file_pattern; - - // Finds all files. - std::vector filenames; - Status s = RecordIterator::ParsePattern(file_type_, opts_.file_pattern, - &filenames); - if (ShouldFinish(s)) break; - - if (filenames.empty()) { - LOG(FATAL) << "Found no files at " << opts_.file_pattern; - } - - int shuffle_seed = opts_.seed; - if (opts_.seed == 0) { - absl::MutexLock l(&mu_); - shuffle_seed = rnd_(); - } - - // Shuffles these files according to the epoch # and random seed. - std::mt19937_64 shuffle_rnd(Hash64Combine(current_epoch(), shuffle_seed)); - std::shuffle(filenames.begin(), filenames.end(), shuffle_rnd); - - // Shards files and use one thread to go through each shard. - const int N = opts_.parallelism; - std::vector shards(N); - for (int i = 0; i < N; ++i) { - Shard* shard = &shards[i]; - shard->index = i; - for (int j = i; j < filenames.size(); j += N) { - shard->filenames.push_back(filenames[j]); - } - thread_->Schedule([this, shard]() { ShardLoop(shard); }); - } - for (int i = 0; i < N; ++i) { - shards[i].done.WaitForNotification(); - s.Update(shards[i].status); - } - if (ShouldFinish(s)) break; - - // Do not start the next epoch until all buffered records are consumed. - { - absl::MutexLock l(&mu_); - epoch_end_ = true; - mu_.Await(buf_empty_); - epoch_end_ = false; - } - - LOG(INFO) << "Epoch " << current_epoch() << ": total records " - << num_records_yielded_in_epoch_; - } - - adjust_done.WaitForNotification(); - main_loop_done_.Notify(); -} - -bool BasicRecordYielder::Add(std::vector* values) { - absl::MutexLock l(&mu_); - mu_.Await(buf_not_full_); - while (BufNotFull() && !values->empty()) { - // Adds values->back(). Swaps its position with another random - // element. - auto index = rnd_() % (buf_.size() + 1); - if (index == buf_.size()) { - buf_.push_back(std::move(values->back())); - } else { - buf_.push_back(std::move(buf_[index])); - buf_[index] = std::move(values->back()); - } - values->pop_back(); - } - return stop_; -} - - -void BasicRecordYielder::ShardLoop(Shard* shard) { - std::vector values; - for (const string& filename : shard->filenames) { - if (ShouldFinish(Status::OK())) break; - VLOG(1) << "Shard " << shard->index << " " << filename; - std::unique_ptr iter( - RecordIterator::New(file_type_, filename)); - string key; - Rope val; - while (iter->Next(&key, &val)) { - values.emplace_back(val); - if (values.size() >= kRecordsPerAdd && Add(&values)) { - shard->status = errors::Aborted("stopped"); - break; - } - } - } - // Adds the remaining values of this shard to buf_. - while (!values.empty()) { - Add(&values); - } - shard->done.Notify(); -} - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/record_yielder.h b/lingvo/core/ops/record_yielder.h deleted file mode 100644 index 06e120f9..00000000 --- a/lingvo/core/ops/record_yielder.h +++ /dev/null @@ -1,311 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#ifndef LINGVO_CORE_OPS_RECORD_YIELDER_H_ -#define LINGVO_CORE_OPS_RECORD_YIELDER_H_ - -#include -#include -#include -#include -#include -#include -#include - -#include "absl/synchronization/mutex.h" -#include "absl/synchronization/notification.h" -#include "lingvo/core/ops/rope.h" -#include "tensorflow/core/lib/core/errors.h" -#include "tensorflow/core/lib/core/status.h" -#include "tensorflow/core/lib/core/threadpool.h" -#include "tensorflow/core/platform/macros.h" -#include "tensorflow/core/platform/thread_annotations.h" - -namespace tensorflow { -namespace lingvo { - -// TODO(oday): Separate some constants and Record to other file. -// TODO(oday): Change "source_id" to another appropriate name, because this is -// one of the typically used symbols in some tasks (e.g., NMTExample has such -// field for source tokens). - -constexpr int kDefaultSourceId = 0; - -// A data structure representing a record with its context information. -// TODO(oday): The typename may sound too simple to represent itself. Consider -// to revise it if it may become really problematic. -struct Record { - // Byte sequence representing the record value. - Rope value; - - // ID of the source of the record, typically an index of the given file - // pattern list. This field is used to determine where the record comes from, - // and is useful to specify the source when reading from multiple input - // sources. - int source_id; -}; - -// An interface to iterate sequentially a set of record (Rope). -class RecordIterator { - public: - virtual ~RecordIterator() {} - - // Get the next record. If EOF, returns false. Otherwise returns true and - // fills in 'key' and 'value'. - virtual bool Next(string* key, Rope* value) = 0; - - // Resets this iterator, if possible. Sub-classes should return true on a - // successful reset. If 'randomize_location' is true, then the iterator is - // moved to point to a random record (if supported). - virtual bool ResetIterator(bool randomize_location) { - return false; - } - - // Register a method to create a RecordIterator for the 'type_name'. - typedef std::function FactoryMethod; - static bool Register(const string& type_name, FactoryMethod method); - - // As above, but also register a custom method for parsing file_pattern - // strings into lists of shards. - typedef std::function*)> - PatternParserMethod; - static bool RegisterWithPatternParser(const string& type_name, - FactoryMethod method, - PatternParserMethod parser_method); - - // Returns a record iterator for 'filename' of 'type_name'. - static RecordIterator* New(const string& type_name, const string& filename); - - // Returns the prefix in a file pattern, or an empty string if not exist. - // Example: "tfrecord:data_dir/data.tfrecord" => "tfrecord" - static string GetFilePatternPrefix(const string& file_pattern); - - // Similar to the function above, but in addition to returning the prefix also - // modifies the file_pattern by stripping away the prefix portion. - static string StripPrefixFromFilePattern(string* file_pattern); - - // Parse a file pattern into a list of matching files. - static Status ParsePattern(const string& type_name, - const string& file_pattern_list, - std::vector* filenames); -}; - -// RecordYielder defines an interface that should be used for producing value -// records from files in a random order. Most users should use -// BasicRecordYielder and BasicRecordYielder::New (see example below). -// -// RecordYielder guarantees that the order in which records are yielded are -// highly randomized. -// -// Usage example: -// BasicRecordYielder::Options opts; -// opts.file_pattern = ; -// opts.seed = 301; -// opts.bufsize = 1000000; // A randomized buffer with 1M records. -// opts.parallelism = 8; // Use 8 iterators to iterate through all files. -// RecordYielder* yielder = BasicRecordYielder::New(opts); -// Record record; -// while (true) { -// yielder->Yield(&record); -// // process record. -// } -// yielder->Close(); -// -// RecordYielder can be accessed by multiple threads concurrently. -class RecordYielder { - public: - virtual ~RecordYielder(); - - // Yields one Record. - // To read from multiple input sources and keep track of the source id, create - // a WeightedMixRecordYielder and create a BasicRecordYielder for each - // source. Each BasicRecordYielder can assign some fields in 'record' to - // indicate some characteristics of the data source. - virtual Status Yield(Record* record) = 0; - - // Stop this yielder and then delete it. - virtual void Close() = 0; -}; - -// BasicRecordYielder is a RecordYielder that implements a main loop and makes -// it possible to write a custom RecordYielder by only defining a shard loop. -// Most of the record yielders should inherit from this class. -// -// BasicRecordYielder guarantees that: -// 1) all records are yielded within every epoch; -// 2) each record is yielded only once within every epoch; -// 3) the order in which records are yielded are highly randomized. -// 4) the peak memory usage is roughly avg record size * -// (opts.bufsize + opts.parellelism * 16). -class BasicRecordYielder : public RecordYielder { - public: - struct Options { - // The set of files to yield records from. file_pattern follows: - // [:], where must have an - // associated factory method through registration via New(). - // - // TODO(zhifengc): Document it better the current support format. - string file_pattern; - - // Random seed. It determines how data files are shuffled. - int64 seed = 0; - - // Randomization buffer keeps these many records. - int64 bufsize = 1; - - // If non-zero, attempt to keep this many seconds of records in the - // randomization buffer. The buffer size will never exceed bufsize. - int64 bufsize_in_seconds = 0; - - // Uses this many concurrent iterators to iterate through files. - int32 parallelism = 1; - - // Source id to be supplied with yield. - int32 source_id = 0; - }; - - // Returns a record yielder according to 'opts'. A caller is responsible for - // calling Close when this yielder is no longer required. A caller shouldn't - // delete the yielder. - static BasicRecordYielder* New(Options opts); - - // Yields one 'record' from which the value was read. - Status Yield(Record* record) override; - - // Stop this yielder and then delete it. - void Close() override; - - // Returns the current epoch number. Epoch number starts from 1 and reflects - // the epoch number of the record returned by the next Yield() call. - virtual int64 current_epoch() const { - // TODO(tilarids): Use ReaderMutexLock here. - absl::MutexLock l(&mu_); - return epoch_; - } - - // Returns the current buffer size. - int64 bufsize() const { - absl::MutexLock l(&mu_); - return bufsize_; - } - - protected: - explicit BasicRecordYielder(const Options& opts); - explicit BasicRecordYielder(); // USED ONLY FOR TESTS. - - ~BasicRecordYielder() override; - - // Subclass should implement ShardLoop which processes all records - // in the 'shard'. - struct Shard { - int index; // Shard index. - std::vector filenames; // File names given to this shard. - absl::Notification done; // Notified when this shard is done. - Status status; // Shard status. - }; - void ShardLoop(Shard* shard); - - // Returns true iff 's' indicates the yielder should stop. - bool ShouldFinish(const Status& s); - - // Adds 'values' into the random shuffling buffer buf_. - bool Add(std::vector* values); - - private: - typedef BasicRecordYielder ME; - - Options opts_; - string file_type_; - - // Background threads. Owned. - thread::ThreadPool* thread_; - - mutable absl::Mutex mu_; - - // Epoch number. - int64 epoch_ ABSL_GUARDED_BY(mu_); - - // Turned to true when the yielder is deleted. - bool stop_ ABSL_GUARDED_BY(mu_) = false; - Status status_ ABSL_GUARDED_BY(mu_); - - // PRG used for randomization. - std::mt19937_64 rnd_ ABSL_GUARDED_BY(mu_); - - // Randomization buffer. - std::vector buf_ ABSL_GUARDED_BY(mu_); - - // True iff we are draining an epoch. - bool epoch_end_ ABSL_GUARDED_BY(mu_) = false; - - int64 num_records_yielded_in_epoch_ = 0; - - // Dynamically adjusted buffer size. - double bufsize_ ABSL_GUARDED_BY(mu_); - - // Number of Yield calls in the current adjustment interval. - int64 yields_ ABSL_GUARDED_BY(mu_); - - // Trigger when the main loop has exited. - absl::Notification main_loop_done_; - - // Conditions. - absl::Condition buf_empty_; - bool BufEmpty() const ABSL_SHARED_LOCKS_REQUIRED(mu_) { - return stop_ || buf_.empty(); - } - - absl::Condition buf_not_full_; - bool BufNotFull() const ABSL_SHARED_LOCKS_REQUIRED(mu_) { - return stop_ || static_cast(buf_.size()) < bufsize_; - } - - absl::Condition buf_enough_; - bool BufEnough() const ABSL_SHARED_LOCKS_REQUIRED(mu_) { - // NOTE: Unless we are finishing an epoch, we want to make sure - // the buf_ contains enough randomized elements before yielding any. - return stop_ || !status_.ok() || (epoch_end_ && !buf_.empty()) || - (!epoch_end_ && static_cast(buf_.size()) >= - std::max(1, bufsize_ / 2)); - } - - void ExtractValue(Rope* value) ABSL_EXCLUSIVE_LOCKS_REQUIRED(mu_) { - if (opts_.seed == 0) { - // Randomize at the consumer side as well. - const auto index = rnd_() % buf_.size(); - *value = std::move(buf_[index]); - if (index != buf_.size() - 1) { - buf_[index] = std::move(buf_.back()); - } - } else { - *value = std::move(buf_.back()); - } - buf_.pop_back(); - } - - void Start(); - void MainLoop(); - void AdjustBufferSizeLoop(); - - // For performance debugging. - void WaitForBufEnough() ABSL_EXCLUSIVE_LOCKS_REQUIRED(mu_); - - TF_DISALLOW_COPY_AND_ASSIGN(BasicRecordYielder); -}; - -} // namespace lingvo -} // namespace tensorflow - -#endif // LINGVO_CORE_OPS_RECORD_YIELDER_H_ diff --git a/lingvo/core/ops/record_yielder_test.cc b/lingvo/core/ops/record_yielder_test.cc deleted file mode 100644 index b258f44a..00000000 --- a/lingvo/core/ops/record_yielder_test.cc +++ /dev/null @@ -1,538 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/record_yielder.h" - -#include // NOLINT(build/c++11) - -#include -#include "lingvo/core/ops/input_common.h" -#include "lingvo/core/ops/sequential_record_yielder.h" -#include "lingvo/core/ops/yielder_test_helper.h" -#include "tensorflow/core/lib/core/errors.h" -#include "tensorflow/core/lib/core/stringpiece.h" -#include "tensorflow/core/lib/io/compression.h" -#include "tensorflow/core/lib/io/path.h" -#include "tensorflow/core/lib/io/record_writer.h" -#include "tensorflow/core/lib/strings/strcat.h" -#include "tensorflow/core/lib/strings/stringprintf.h" -#include "tensorflow/core/platform/env.h" - -namespace tensorflow { -namespace lingvo { - -TEST(RecordYielderTest, PlainTextYielderBasicTest) { - const int N = 10; - const int M = 1000; - GeneratePlainTextTestData("basic", N, M); - BasicRecordYielder::Options opts; - opts.file_pattern = - strings::StrCat("text:", io::JoinPath("/tmp", "basic.*")); - opts.seed = 301; - opts.bufsize = 2000; - opts.parallelism = 1; - - BasicRecordYielder* yielder = BasicRecordYielder::New(opts); - std::vector vals; - Record record; - record.source_id = kDefaultSourceId; - for (int i = 0; i < N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - VLOG(1) << i << " " << record.value; - vals.emplace_back(string(record.value)); - } - std::sort(vals.begin(), vals.end()); - auto new_end = std::unique(vals.begin(), vals.end()); - - // One epoch should have no duplicates. - ASSERT_EQ(new_end, vals.end()); - - // Iterates another two epochs. - for (int i = 0; i < 2 * N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - } - - // Iterates another 34 epochs. - for (int i = 0; i < 2 * 17 * N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - } - - // End of the 37th epoch | start of the 38th epoch. - EXPECT_TRUE(yielder->current_epoch() == 37 || yielder->current_epoch() == 38); - yielder->Close(); -} - -TEST(SequentialRecordYielderTest, SequentialRecordYielderBasicTest) { - const int N = 10; - const int M = 1000; - GeneratePlainTextTestData("basic", N, M); - const string& file_pattern = - strings::StrCat("text:", io::JoinPath("/tmp", "basic.*")); - - SequentialRecordYielder* yielder = - SequentialRecordYielder::New(file_pattern, -1); - Record record; - record.source_id = kDefaultSourceId; - for (int i = 0; i < N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - ASSERT_EQ(string(record.value), strings::Printf("basic:%010d", i)); - } - - // Iterate another epoch. - for (int i = 0; i < N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - ASSERT_EQ(string(record.value), strings::Printf("basic:%010d", i)); - } - - yielder->Close(); -} - -TEST(SequentialRecordYielderTest, SequentialRecordYielderRepeatCount) { - const int N = 10; - const int M = 1000; - GeneratePlainTextTestData("basic", N, M); - const string& file_pattern = - strings::StrCat("text:", io::JoinPath("/tmp", "basic.*")); - - // Yield two epochs. - SequentialRecordYielder* yielder = - SequentialRecordYielder::New(file_pattern, 2); - Record record; - record.source_id = kDefaultSourceId; - for (int i = 0; i < N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - ASSERT_EQ(string(record.value), strings::Printf("basic:%010d", i)); - } - - // Iterate another epoch. - for (int i = 0; i < N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - ASSERT_EQ(string(record.value), strings::Printf("basic:%010d", i)); - } - - // Trying to yield one more element should throw an out of range error. - Status s = yielder->Yield(&record); - ASSERT_TRUE(errors::IsOutOfRange(s)) << s; - - yielder->Close(); -} - -void GenerateTfRecordTestData(const string& prefix, int n, int m, - const string& compression_type) { - for (int i = 0; i < n; ++i) { - std::unique_ptr file; - TF_CHECK_OK(Env::Default()->NewWritableFile( - io::JoinPath("/tmp", strings::StrCat(prefix, ".", i)), - &file)); - io::RecordWriter writer( - file.get(), - io::RecordWriterOptions::CreateRecordWriterOptions(compression_type)); - for (int j = 0; j < m; ++j) { - TF_CHECK_OK(writer.WriteRecord(strings::Printf("%010d", m * i + j))); - } - } -} - -void GenerateShardedTfRecordTestData(const string& prefix, int n, int m) { - for (int i = 0; i < n; ++i) { - std::unique_ptr file; - const string filename = - strings::Printf("%s-%05d-of-%05d", prefix.c_str(), i, n); - TF_CHECK_OK(Env::Default()->NewWritableFile( - io::JoinPath("/tmp", filename), &file)); - io::RecordWriter writer(file.get()); - for (int j = 0; j < m; ++j) { - TF_CHECK_OK(writer.WriteRecord(strings::Printf("%010d", m * i + j))); - } - } -} - -typedef testing::TestWithParam TfRecordYielderTest; - -string PrefixFromCompressionType(const string& compression_type) { - if (compression_type == io::compression::kGzip) { - return "tfrecord_gzip:"; - } else if (compression_type != io::compression::kNone) { - LOG(ERROR) << "Unknown compression type, using no compression"; - } - return "tfrecord:"; -} - -TEST_P(TfRecordYielderTest, TfRecordYielderBasicTest) { - const int N = 10; - const int M = 1000; - GenerateTfRecordTestData("basic", N, M, GetParam()); - BasicRecordYielder::Options opts; - opts.file_pattern = - strings::StrCat(PrefixFromCompressionType(GetParam()), - io::JoinPath("/tmp", "basic.*")); - opts.seed = 301; - opts.bufsize = 2000; - opts.parallelism = 1; - - BasicRecordYielder* yielder = BasicRecordYielder::New(opts); - std::vector vals; - Record record; - record.source_id = kDefaultSourceId; - for (int i = 0; i < N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - VLOG(1) << i << " " << record.value; - vals.emplace_back(string(record.value)); - } - std::sort(vals.begin(), vals.end()); - auto new_end = std::unique(vals.begin(), vals.end()); - - // One epoch should have no duplicates. - ASSERT_EQ(new_end, vals.end()); - - // Iterates another two epochs. - for (int i = 0; i < 2 * N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - } - - // Iterates another 34 epochs. - for (int i = 0; i < 2 * 17 * N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - } - - // End of the 37th epoch | start of the 38th epoch. - EXPECT_TRUE(yielder->current_epoch() == 37 || yielder->current_epoch() == 38); - yielder->Close(); -} - -int NumMatches(const std::vector& vals1, - const std::vector& vals2) { - CHECK_EQ(vals1.size(), vals2.size()); - int num_matches = 0; - for (int i = 0; i < vals1.size(); ++i) { - num_matches += vals1[i] == vals2[i]; - } - return num_matches; -} - -TEST_P(TfRecordYielderTest, ShufflesShard) { - const int M = 32; - GenerateTfRecordTestData("oneshard", 1 /* num_shards */, M, GetParam()); - - BasicRecordYielder::Options opts; - opts.file_pattern = - strings::StrCat(PrefixFromCompressionType(GetParam()), - io::JoinPath("/tmp", "oneshard.0")); - opts.bufsize = M; - opts.parallelism = 1; - - // Subsequent epochs should be yielded in different orders. - std::vector epoch1, epoch2; - { - opts.seed = 301; - auto yielder = BasicRecordYielder::New(opts); - for (int i = 0; i < M; ++i) { - Record record; - record.source_id = kDefaultSourceId; - TF_CHECK_OK(yielder->Yield(&record)); - epoch1.push_back(record.value); - } - for (int i = 0; i < M; ++i) { - Record record; - record.source_id = kDefaultSourceId; - TF_CHECK_OK(yielder->Yield(&record)); - epoch2.push_back(record.value); - } - yielder->Close(); - EXPECT_LT(NumMatches(epoch1, epoch2), M); - } - - // Ordering should change if the seed changes. - std::vector epoch1_different_seed; - { - opts.seed = 103; - auto yielder = BasicRecordYielder::New(opts); - for (int i = 0; i < M; ++i) { - Record record; - record.source_id = kDefaultSourceId; - TF_CHECK_OK(yielder->Yield(&record)); - epoch1_different_seed.push_back(record.value); - } - yielder->Close(); - EXPECT_LT(NumMatches(epoch1, epoch1_different_seed), M); - } -} - -TEST(RecordYielderDeathTest, Error) { - EXPECT_DEATH([](){ - BasicRecordYielder::Options opts; - opts.file_pattern = strings::StrCat( - "tfrecord:", io::JoinPath("/tmp", "nothing.*")); - auto yielder = BasicRecordYielder::New(opts); - Record record; - record.source_id = kDefaultSourceId; - auto unused = yielder->Yield(&record); - }(), "Found no files at .*nothing"); -} - -TEST_P(TfRecordYielderTest, MatchFilesFromMultiplePatterns) { - const int N = 2; - const int M = 32; - GenerateTfRecordTestData("twoshard", N /* num_shards */, - M /* record per shard */, GetParam()); - BasicRecordYielder::Options opts; - const string path0 = io::JoinPath("/tmp", "twoshard.0"); - const string path1 = io::JoinPath("/tmp", "twoshard.1"); - opts.file_pattern = - strings::StrCat(PrefixFromCompressionType(GetParam()), path0, ",", path1); - opts.bufsize = M; - opts.parallelism = 1; - std::vector epoch; - auto yielder = BasicRecordYielder::New(opts); - Record record; - record.source_id = kDefaultSourceId; - for (int i = 0; i < N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - epoch.emplace_back(string(record.value)); - } - auto new_end = std::unique(epoch.begin(), epoch.end()); - // If we iterated through both shards (rather than 1 shard twice), there - // should be no duplicates, and we should be at the end of the first epoch. - EXPECT_EQ(new_end, epoch.end()); - // End of the 1st epoch | start of the 2nd epoch. - EXPECT_TRUE(yielder->current_epoch() == 1 || yielder->current_epoch() == 2); - TF_CHECK_OK(yielder->Yield(&record)); - // End of the 2st epoch | start of the 3nd epoch. - EXPECT_TRUE(yielder->current_epoch() == 2 || yielder->current_epoch() == 3); - yielder->Close(); -} - -INSTANTIATE_TEST_CASE_P(All, TfRecordYielderTest, - testing::Values(io::compression::kNone, - io::compression::kGzip)); - -TEST(RecordYielder, MatchShardedFilePattern) { - const int num_shards = 16; - const int records_per_shard = 8; - GenerateShardedTfRecordTestData("sharded_data", num_shards, - records_per_shard); - - BasicRecordYielder::Options opts; - opts.file_pattern = strings::StrCat( - "tfrecord:", io::JoinPath("/tmp", "sharded_data@16")); - opts.bufsize = records_per_shard; - opts.parallelism = 1; - std::vector epoch; - auto yielder = BasicRecordYielder::New(opts); - Record record; - record.source_id = kDefaultSourceId; - for (int i = 0; i < num_shards * records_per_shard; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - epoch.emplace_back(string(record.value)); - } - auto new_end = std::unique(epoch.begin(), epoch.end()); - // If we iterated through all shards (rather than 1 shard twice), there - // should be no duplicates, and we should be at the end of the first epoch. - EXPECT_EQ(new_end, epoch.end()); - // End of the 1st epoch | start of the 2nd epoch. - EXPECT_TRUE(yielder->current_epoch() == 1 || yielder->current_epoch() == 2); - TF_CHECK_OK(yielder->Yield(&record)); - // End of the 2st epoch | start of the 3nd epoch. - EXPECT_TRUE(yielder->current_epoch() == 2 || yielder->current_epoch() == 3); - yielder->Close(); -} - -TEST(RecordYielder, MatchWildcardShardedFilePattern) { - const int num_shards = 9; - const int records_per_shard = 8; - GenerateShardedTfRecordTestData("sharded_data2", num_shards, - records_per_shard); - - BasicRecordYielder::Options opts; - opts.file_pattern = strings::StrCat( - "tfrecord:", io::JoinPath("/tmp", "sharded_data2@*")); - opts.bufsize = records_per_shard; - opts.parallelism = 1; - std::vector epoch; - auto yielder = BasicRecordYielder::New(opts); - Record record; - record.source_id = kDefaultSourceId; - - for (int i = 0; i < num_shards * records_per_shard; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - epoch.emplace_back(string(record.value)); - } - auto new_end = std::unique(epoch.begin(), epoch.end()); - // If we iterated through all shards (rather than 1 shard twice), there - // should be no duplicates, and we should be at the end of the first epoch. - EXPECT_EQ(new_end, epoch.end()); - // End of the 1st epoch | start of the 2nd epoch. - EXPECT_TRUE(yielder->current_epoch() == 1 || yielder->current_epoch() == 2); - TF_CHECK_OK(yielder->Yield(&record)); - // End of the 2st epoch | start of the 3nd epoch. - EXPECT_TRUE(yielder->current_epoch() == 2 || yielder->current_epoch() == 3); - yielder->Close(); -} - -TEST(RecordYielder, MatchIndirectFilePattern) { - const int records_per_shard = 100; - GenerateCheckpointPlainTextTestData("checkpoint", records_per_shard); - - BasicRecordYielder::Options opts; - opts.file_pattern = strings::StrCat( - "text_indirect:", io::JoinPath("/tmp", "checkpoint")); - opts.bufsize = records_per_shard; - opts.parallelism = 1; - std::vector epoch; - auto yielder = BasicRecordYielder::New(opts); - Record record; - record.source_id = kDefaultSourceId; - - // Iterate over all but 1 record in the entire data file. - for (int i = 0; i < records_per_shard - 1; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - epoch.emplace_back(string(record.value)); - } - // Update checkpoint file to point to new data and iterate over final file. - UpdateCheckpointPlainTextTestData("checkpoint", records_per_shard); - TF_CHECK_OK(yielder->Yield(&record)); - epoch.emplace_back(string(record.value)); - auto new_end = std::unique(epoch.begin(), epoch.end()); - // If we iterated through only the first version of the file, there - // should be no duplicates, and we should be at the end of the first epoch. - EXPECT_EQ(new_end, epoch.end()); - // End of the 1st epoch | start of the 2nd epoch. - EXPECT_TRUE(yielder->current_epoch() == 1 || yielder->current_epoch() == 2); - - // Now we should be iterating over the new file. - TF_CHECK_OK(yielder->Yield(&record)); - epoch.emplace_back(string(record.value)); - new_end = std::unique(epoch.begin(), epoch.end()); - EXPECT_EQ(new_end, epoch.end()); - yielder->Close(); -} - -namespace { - -class FakeIterator : public RecordIterator { - public: - FakeIterator(const string& pattern) : RecordIterator(), pattern_(pattern) {} - bool Next(string* key, Rope* value) { - if (pattern_.empty()) return false; - *key = pattern_; - *value = pattern_; - pattern_ = ""; - return true; - } - - private: - std::string pattern_; -}; - -bool register_fake_iterator = RecordIterator::RegisterWithPatternParser( - "fakeiter", [](const string& pattern) { return new FakeIterator(pattern); }, - [](const string& file_pattern, std::vector* shards) { - shards->push_back(file_pattern); - return Status::OK(); - }); - -} // namespace - -TEST(RecordYielder, RegisterFakeIterator) { - ASSERT_TRUE(register_fake_iterator); - BasicRecordYielder::Options options; - options.file_pattern = "fakeiter:hello1"; - BasicRecordYielder* yielder = BasicRecordYielder::New(options); - EXPECT_TRUE(yielder != nullptr); - Record record; - record.source_id = kDefaultSourceId; - EXPECT_TRUE(yielder->Yield(&record).ok()); - EXPECT_EQ("hello1", record.value); - EXPECT_TRUE(yielder->Yield(&record).ok()); - EXPECT_EQ("hello1", record.value); - yielder->Close(); -} - -TEST(RecordYielder, Iota) { - BasicRecordYielder::Options opts; - opts.file_pattern = "iota:100"; - opts.bufsize = 16; - opts.parallelism = 1; - BasicRecordYielder* yielder = BasicRecordYielder::New(opts); - std::vector vals; - Record record; - record.source_id = kDefaultSourceId; - for (int i = 0; i < 100; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - VLOG(1) << i << " " << record.value; - vals.emplace_back(string(record.value)); - } - std::sort(vals.begin(), vals.end()); - auto new_end = std::unique(vals.begin(), vals.end()); - EXPECT_EQ(new_end, vals.end()); - yielder->Close(); -} - -TEST(RecordYielder, AdjustStaysLow) { - BasicRecordYielder::Options opts; - opts.file_pattern = "iota:100"; - opts.bufsize = 100; - opts.bufsize_in_seconds = 1; - opts.parallelism = 1; - BasicRecordYielder* yielder = BasicRecordYielder::New(opts); - - // We didn't read anything for 5 seconds, so we expect that the buffer size - // has stayed at the initial value of 16. - std::this_thread::sleep_for(std::chrono::seconds(5)); - EXPECT_EQ(16, yielder->bufsize()); - yielder->Close(); -} - -TEST(RecordYielder, AdjustUp) { - BasicRecordYielder::Options opts; - opts.file_pattern = "iota:100"; - opts.bufsize = 100; - opts.bufsize_in_seconds = 1; - opts.parallelism = 1; - BasicRecordYielder* yielder = BasicRecordYielder::New(opts); - - // We are reading 1000 records/second, so we expect that the buffer size - // will be much higher than the initial value of 16. - for (int i = 0; i < 50; i++) { - for (int j = 0; j < 100; j++) { - Record record; - record.source_id = kDefaultSourceId; - LOG(INFO) << "yield " << j; - TF_CHECK_OK(yielder->Yield(&record)); - } - std::this_thread::sleep_for(std::chrono::milliseconds(100)); - } - EXPECT_LT(30, yielder->bufsize()); - EXPECT_GE(100, yielder->bufsize()); - yielder->Close(); -} - -TEST(RecordIterator, GetFilePatternPrefix) { - EXPECT_EQ("", RecordIterator::GetFilePatternPrefix("/foo/bar/*")); - EXPECT_EQ("baz", RecordIterator::GetFilePatternPrefix("baz:/foo/bar/*")); - - string file_pattern; - - file_pattern = "/foo/bar/*"; - EXPECT_EQ("", RecordIterator::StripPrefixFromFilePattern(&file_pattern)); - EXPECT_EQ("/foo/bar/*", file_pattern); - - file_pattern = "baz:/foo/bar/*"; - EXPECT_EQ("baz", RecordIterator::StripPrefixFromFilePattern(&file_pattern)); - EXPECT_EQ("/foo/bar/*", file_pattern); -} - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/rope.h b/lingvo/core/ops/rope.h deleted file mode 100644 index ae82a72c..00000000 --- a/lingvo/core/ops/rope.h +++ /dev/null @@ -1,40 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#ifndef LINGVO_CORE_OPS_ROPE_H_ -#define LINGVO_CORE_OPS_ROPE_H_ - -#include - -namespace tensorflow { -namespace lingvo { - -class Rope : public std::string { - public: - Rope() = default; - Rope(const std::string& s) : std::string(s) {} - Rope(std::string&& s) : std::string(std::move(s)) {} - - void AppendTo(std::string* dst) const { - dst->append(*this); - } - - std::string ToString() const { return *this; } -}; - -} // namespace lingvo -} // namespace tensorflow - -#endif // LINGVO_CORE_OPS_ROPE_H_ diff --git a/lingvo/core/ops/sequential_record_yielder.cc b/lingvo/core/ops/sequential_record_yielder.cc deleted file mode 100644 index 758c3fb2..00000000 --- a/lingvo/core/ops/sequential_record_yielder.cc +++ /dev/null @@ -1,89 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/sequential_record_yielder.h" - -#include - -#include "lingvo/core/ops/record_yielder.h" -#include "tensorflow/core/lib/core/errors.h" - -namespace tensorflow { -namespace lingvo { - -namespace { -constexpr int kInfinite = -1; -} // namespace - -SequentialRecordYielder::SequentialRecordYielder(const string& file_pattern, - const int64_t repeat_count) - : file_type_(RecordIterator::GetFilePatternPrefix(file_pattern)), - repeat_count_(repeat_count) { - LOG(INFO) << this << "Sequential record yielder start"; - string mutable_file_pattern(file_pattern); - if (!file_type_.empty()) { - mutable_file_pattern.erase(0, file_type_.size() + 1); - } - // Finds all files. - TF_CHECK_OK(RecordIterator::ParsePattern(file_type_, mutable_file_pattern, - &filenames_)); - std::sort(filenames_.begin(), filenames_.end()); - if (filenames_.empty()) { - LOG(FATAL) << "Found no files at " << file_pattern; - } - - CHECK(repeat_count == kInfinite || repeat_count > 0) - << "Repeat count must either be -1 (infinite) or a positive integer."; - - record_iterator_ = std::unique_ptr( - RecordIterator::New(file_type_, filenames_[0])); -} - -SequentialRecordYielder* SequentialRecordYielder::New( - const string& file_pattern, const int64_t repeat_count) { - return new SequentialRecordYielder(file_pattern, repeat_count); -} - -SequentialRecordYielder::~SequentialRecordYielder() {} - -void SequentialRecordYielder::Close() { - LOG(INFO) << this << "Sequential record yielder exit"; - delete this; -} - -Status SequentialRecordYielder::Yield(Record* record) { - string key; - if (record_iterator_->Next(&key, &record->value)) { - return Status::OK(); - } - - // No more records from current iterator, advance to next iterator. - cur_file_index_ = (cur_file_index_ + 1) % filenames_.size(); - if (cur_file_index_ == 0) { - ++num_repeats_; - LOG(INFO) << "SequentialRecordYielder finished " << num_repeats_ - << " repeats."; - if (repeat_count_ != kInfinite && num_repeats_ == repeat_count_) { - return errors::OutOfRange("SequentialRecordYielder reached ", - repeat_count_, " repeats."); - } - } - record_iterator_ = std::unique_ptr( - RecordIterator::New(file_type_, filenames_[cur_file_index_])); - return Yield(record); -} - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/sequential_record_yielder.h b/lingvo/core/ops/sequential_record_yielder.h deleted file mode 100644 index 7c1a8960..00000000 --- a/lingvo/core/ops/sequential_record_yielder.h +++ /dev/null @@ -1,55 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ -#ifndef LINGVO_CORE_OPS_SEQUENTIAL_RECORD_YIELDER_H_ -#define LINGVO_CORE_OPS_SEQUENTIAL_RECORD_YIELDER_H_ - -#include "lingvo/core/ops/record_yielder.h" - -namespace tensorflow { -namespace lingvo { - -// SequentialRecordYielder processes records in order. -class SequentialRecordYielder : public RecordYielder { - public: - ~SequentialRecordYielder() override; - void Close() override; - Status Yield(Record* record) override; - - // Returns a sequential record yielder. The caller is responsible for calling - // Close when this yielder is no longer required. The caller shouldn't delete - // the yielder. - static SequentialRecordYielder* New(const string& file_pattern, - const int64_t repeat_count); - - protected: - explicit SequentialRecordYielder(const string& file_pattern, - const int64_t repeat_count); - - private: - const string file_type_; - // Target number of repetitions of the dataset. -1 means to repeat - // forever. - const int64_t repeat_count_; - - std::vector filenames_; - int cur_file_index_ = 0; - std::unique_ptr record_iterator_; - // Current number of repetitions of the dataset. - int64_t num_repeats_ = 0; -}; - -} // namespace lingvo -} // namespace tensorflow -#endif // LINGVO_CORE_OPS_SEQUENTIAL_RECORD_YIELDER_H_ diff --git a/lingvo/core/ops/simple_vocab.cc b/lingvo/core/ops/simple_vocab.cc deleted file mode 100644 index 79eb678d..00000000 --- a/lingvo/core/ops/simple_vocab.cc +++ /dev/null @@ -1,258 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/simple_vocab.h" - -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/lib/strings/str_util.h" -#include "tensorflow/core/platform/env.h" -#include "tensorflow/core/platform/logging.h" - -namespace { - -constexpr char kSosToken[] = ""; -constexpr char kEosToken[] = ""; -constexpr char kUnkToken[] = ""; -constexpr char kSowToken[] = ""; -constexpr char kEowToken[] = ""; -constexpr char kSosTokenUpper[] = ""; -constexpr char kEosTokenUpper[] = ""; -constexpr char kUnkTokenUpper[] = ""; - -} // end namespace - -namespace tensorflow { -namespace lingvo { - -namespace debug { - -static Vocab* vocab = nullptr; - -void SetUpVocab(const string& vocab_filename) { - if (vocab == nullptr) { - vocab = new Vocab(); - TF_CHECK_OK(vocab->Load(vocab_filename)); - } -} - -string IdsToStr(const std::vector& ids) { - if (vocab != nullptr) { - const std::vector toks = vocab->IdsToTokens(ids); - return str_util::Join(toks, " "); - } else { - return str_util::Join(ids, " "); - } -} -} // namespace debug - -Status Vocab::Load(const string& vocab_glob, bool load_token_ids) { - std::vector vocab_filenames; - TF_RETURN_IF_ERROR( - Env::Default()->GetMatchingPaths(vocab_glob, &vocab_filenames)); - if (vocab_filenames.size() != 1) { - return errors::InvalidArgument( - "Did not match exactly one file with pattern: ", vocab_glob); - } - const string& vocab_filename = vocab_filenames[0]; - - debug::SetUpVocab(vocab_filename); - - string content; - TF_RETURN_IF_ERROR( - ReadFileToString(Env::Default(), vocab_filename, &content)); - - return Load(str_util::Split(content, '\n'), load_token_ids); -} - -Status Vocab::Load(const std::vector& lines, bool load_token_ids) { - id_to_token_.clear(); - token_to_id_.clear(); - int32 next_id = 0; - for (StringPiece line : lines) { - if (line.empty()) continue; - const std::vector parts = str_util::Split(line, '\t'); - CHECK_GE(parts.size(), 1); - const string tok = parts[0]; - if (!load_token_ids) { - token_to_id_[tok] = next_id; - id_to_token_[next_id] = tok; - next_id++; - } else { - CHECK_GE(parts.size(), 2); - const int32 id = std::stoi(parts[1]); - token_to_id_[tok] = id; - id_to_token_[id] = tok; - } - VLOG(2) << "Vocab " << token_to_id_[tok] << " " << tok; - } - use_upper_token_symbols_ = false; - std::vector expected_tokens = {kSosToken, kEosToken, kUnkToken}; - std::vector unexpected_tokens = {kSosTokenUpper, kEosTokenUpper, - kUnkTokenUpper}; - if (token_to_id_.find(sos_token()) == token_to_id_.end()) { - use_upper_token_symbols_ = true; - expected_tokens.swap(unexpected_tokens); - } - sos_id_ = token_to_id_[sos_token()]; - - for (const auto& token : expected_tokens) { - if (token_to_id_.find(token) == token_to_id_.end()) { - return errors::InvalidArgument(token, " is not found in the vocab."); - } - } - for (const auto& token : unexpected_tokens) { - if (token_to_id_.find(token) != token_to_id_.end()) { - return errors::InvalidArgument("Invalid token ", token, - " is found in the vocab."); - } - } - unk_id_ = -1; - sos_id_ = TokenToId(sos_token()); - eos_id_ = TokenToId(eos_token()); - sow_id_ = TokenToId(sow_token()); - eow_id_ = TokenToId(eow_token()); - unk_id_ = TokenToId(unk_token()); - return Status::OK(); -} - -const char* Vocab::sos_token() const { - return use_upper_token_symbols_ ? kSosTokenUpper : kSosToken; -} - -const char* Vocab::eos_token() const { - return use_upper_token_symbols_ ? kEosTokenUpper : kEosToken; -} - -const char* Vocab::unk_token() const { - return use_upper_token_symbols_ ? kUnkTokenUpper : kUnkToken; -} - -const char* Vocab::sow_token() const { return kSowToken; } - -const char* Vocab::eow_token() const { return kEowToken; } - -namespace { - -class VocabTokenToIdOp : public OpKernel { - public: - explicit VocabTokenToIdOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - std::vector vocab; - OP_REQUIRES_OK(ctx, ctx->GetAttr("vocab", &vocab)); - bool load_token_ids_from_vocab; - OP_REQUIRES_OK(ctx, ctx->GetAttr("load_token_ids_from_vocab", - &load_token_ids_from_vocab)); - OP_REQUIRES_OK(ctx, vocab_.Load(vocab, load_token_ids_from_vocab)); - } - - void Compute(OpKernelContext* ctx) override { - const Tensor* token; - OP_REQUIRES_OK(ctx, ctx->input("token", &token)); - Tensor* id; - OP_REQUIRES_OK(ctx, ctx->allocate_output("id", token->shape(), &id)); - if (token->dims() == 0) { - id->scalar()() = vocab_.TokenToId(token->scalar()()); - } else { - OP_REQUIRES( - ctx, token->dims() == 1, - errors::InvalidArgument("Input must be a scalar or 1D tensor.")); - for (int i = 0; i < token->dim_size(0); i++) { - id->vec()(i) = vocab_.TokenToId(token->vec()(i)); - } - } - } - - private: - Vocab vocab_; -}; - -REGISTER_KERNEL_BUILDER(Name("VocabTokenToId").Device(DEVICE_CPU), - VocabTokenToIdOp); - -class VocabIdToTokenOp : public OpKernel { - public: - explicit VocabIdToTokenOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - std::vector vocab; - OP_REQUIRES_OK(ctx, ctx->GetAttr("vocab", &vocab)); - bool load_token_ids_from_vocab; - OP_REQUIRES_OK(ctx, ctx->GetAttr("load_token_ids_from_vocab", - &load_token_ids_from_vocab)); - OP_REQUIRES_OK(ctx, vocab_.Load(vocab, load_token_ids_from_vocab)); - } - - void Compute(OpKernelContext* ctx) override { - const Tensor* id; - OP_REQUIRES_OK(ctx, ctx->input("id", &id)); - Tensor* token; - OP_REQUIRES_OK(ctx, ctx->allocate_output("token", id->shape(), &token)); - if (id->dims() == 0) { - token->scalar()() = vocab_.IdToToken(id->scalar()()); - } else { - OP_REQUIRES( - ctx, id->dims() == 1, - errors::InvalidArgument("Input must be a scalar or 1D tensor.")); - for (int i = 0; i < id->dim_size(0); i++) { - token->vec()(i) = vocab_.IdToToken(id->vec()(i)); - } - } - } - - private: - Vocab vocab_; -}; - -REGISTER_KERNEL_BUILDER(Name("VocabIdToToken").Device(DEVICE_CPU), - VocabIdToTokenOp); - -class TokenInVocabOp : public OpKernel { - public: - explicit TokenInVocabOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - std::vector vocab; - OP_REQUIRES_OK(ctx, ctx->GetAttr("vocab", &vocab)); - bool load_token_ids_from_vocab; - OP_REQUIRES_OK(ctx, ctx->GetAttr("load_token_ids_from_vocab", - &load_token_ids_from_vocab)); - OP_REQUIRES_OK(ctx, vocab_.Load(vocab, load_token_ids_from_vocab)); - } - - void Compute(OpKernelContext* ctx) override { - const Tensor* token; - OP_REQUIRES_OK(ctx, ctx->input("token", &token)); - Tensor* result; - OP_REQUIRES_OK(ctx, - ctx->allocate_output("result", token->shape(), &result)); - if (token->dims() == 0) { - result->scalar()() = vocab_.InVocab(token->scalar()()); - } else { - OP_REQUIRES( - ctx, token->dims() == 1, - errors::InvalidArgument("Input must be a scalar or 1D tensor.")); - for (int i = 0; i < token->dim_size(0); i++) { - result->vec()(i) = vocab_.InVocab(token->vec()(i)); - } - } - } - - private: - Vocab vocab_; -}; - -REGISTER_KERNEL_BUILDER(Name("TokenInVocab").Device(DEVICE_CPU), - TokenInVocabOp); - -} // namespace - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/simple_vocab.h b/lingvo/core/ops/simple_vocab.h deleted file mode 100644 index db599f90..00000000 --- a/lingvo/core/ops/simple_vocab.h +++ /dev/null @@ -1,129 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#ifndef LINGVO_CORE_OPS_SIMPLE_VOCAB_H_ -#define LINGVO_CORE_OPS_SIMPLE_VOCAB_H_ -// TODO(zhifengc): Add comments for this class. - -#include -#include -#include - -#include "tensorflow/core/lib/strings/str_util.h" -#include "tensorflow/core/platform/env.h" -#include "tensorflow/core/platform/macros.h" - -namespace tensorflow { -namespace lingvo { - -class Vocab { - public: - Vocab() {} - - ~Vocab() {} - - Status Load(const string& vocab_filename, bool load_token_ids = false); - Status Load(const std::vector& lines, bool load_token_ids = false); - - int32 sos_id() const { return sos_id_; } - int32 eos_id() const { return eos_id_; } - int32 unk_id() const { return unk_id_; } - int32 sow_id() const { return sow_id_; } - int32 eow_id() const { return eow_id_; } - - const char* sos_token() const; - const char* eos_token() const; - const char* unk_token() const; - const char* sow_token() const; - const char* eow_token() const; - - bool InVocab(const string& tok) const { return token_to_id_.count(tok) > 0; } - - int32 TokenToId(const string& tok) const { - auto it = token_to_id_.find(tok); - if (it != token_to_id_.end()) return it->second; - return unk_id_; - } - - int GetVocabSize() const { return id_to_token_.size(); } - - void GreedyMatchStringToTokenId(StringPiece text, int32* token_id, - int* token_size) const { - // This finds the longest prefix of the "text" in the given list of tokens - // and returns the ID of the found token (if nothing found, unk_id_ is - // returned) and the length of the found token through input argument - // pointers. - *token_id = unk_id_; - *token_size = 1; // For , the input is of length 1 char, but output is - // (length of 5). - for (const auto kv : id_to_token_) { - if (str_util::StartsWith(text, kv.second)) { - // Find the longest matching token. - if (*token_id == unk_id_ || *token_size < kv.second.size()) { - *token_id = kv.first; - *token_size = kv.second.size(); - } - } - } - } - - std::vector TokensToIds(const std::vector& toks) const { - std::vector ids; - ids.reserve(toks.size()); - for (const string& tok : toks) { - ids.push_back(TokenToId(tok)); - } - return ids; - } - - const string IdToToken(const int32 id) const { - const auto it = id_to_token_.find(id); - if (it != id_to_token_.end()) { - return it->second; - } else { - return unk_token(); - } - } - - std::vector IdsToTokens(const std::vector& ids) const { - std::vector toks; - toks.reserve(ids.size()); - for (const int32 id : ids) { - toks.push_back(IdToToken(id)); - } - return toks; - } - - private: - int32 sos_id_ = -1; - int32 eos_id_ = -1; - int32 unk_id_ = -1; - int32 sow_id_ = -1; - int32 eow_id_ = -1; - bool use_upper_token_symbols_ = false; - std::unordered_map id_to_token_; - std::unordered_map token_to_id_; - - TF_DISALLOW_COPY_AND_ASSIGN(Vocab); -}; - -namespace debug { -string IdsToStr(const std::vector& ids); -} // namespace debug - -} // namespace lingvo -} // namespace tensorflow - -#endif // LINGVO_CORE_OPS_SIMPLE_VOCAB_H_ diff --git a/lingvo/core/ops/simple_vocab_test.py b/lingvo/core/ops/simple_vocab_test.py deleted file mode 100644 index 2a14e8cb..00000000 --- a/lingvo/core/ops/simple_vocab_test.py +++ /dev/null @@ -1,163 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for simple_vocab.""" - -from lingvo import compat as tf -from lingvo.core import ops -from lingvo.core import test_utils - - -class VocabOpsTest(test_utils.TestCase): - - def testVocabTokenToId(self): - with self.session(use_gpu=False): - vocab = [ - '', - '', - '', - '', - 'a', - 'b c d e', - 'øut', - 'über', - '♣', - '愤青', - '←', - ] - self.assertEqual(0, ops.vocab_token_to_id('', vocab=vocab).eval()) - self.assertEqual(4, ops.vocab_token_to_id('a', vocab=vocab).eval()) - self.assertAllEqual([5, 8], - ops.vocab_token_to_id(['b c d e', '♣'], - vocab=vocab).eval()) - self.assertEqual(2, ops.vocab_token_to_id('unknown', vocab=vocab).eval()) - - def testVocabTokenToIdLoadId(self): - with self.session(use_gpu=False): - vocab = [ - ' 3', - ' 5', - ' 7', - ' 9', - 'a 2', - 'b c d e 4', - 'øut 8', - 'über 10', - '♣ -1', - '愤青 -3', - '← -5', - ] - self.assertEqual( - 3, - ops.vocab_token_to_id( - '', vocab=vocab, load_token_ids_from_vocab=True).eval()) - self.assertEqual( - 2, - ops.vocab_token_to_id( - 'a', vocab=vocab, load_token_ids_from_vocab=True).eval()) - self.assertAllEqual([4, -1], - ops.vocab_token_to_id( - ['b c d e', '♣'], - vocab=vocab, - load_token_ids_from_vocab=True).eval()) - self.assertEqual( - 7, - ops.vocab_token_to_id( - 'unknown', vocab=vocab, load_token_ids_from_vocab=True).eval()) - - def testVocabIdToToken(self): - with self.session(use_gpu=False): - vocab = [ - '', - '', - '', - '', - 'a', - 'b c d e', - 'øut', - 'über', - '♣', - '愤青', - '←', - ] - self.assertEqual(b'', ops.vocab_id_to_token(0, vocab=vocab).eval()) - self.assertEqual(b'a', ops.vocab_id_to_token(4, vocab=vocab).eval()) - self.assertAllEqual([b'b c d e', u'♣'.encode('utf-8')], - ops.vocab_id_to_token([5, 8], vocab=vocab).eval()) - self.assertEqual(b'', ops.vocab_id_to_token(2, vocab=vocab).eval()) - self.assertEqual(b'', ops.vocab_id_to_token(-1, vocab=vocab).eval()) - self.assertEqual(b'', ops.vocab_id_to_token(11, vocab=vocab).eval()) - - def testVocabIdToTokenLoadId(self): - with self.session(use_gpu=False): - vocab = [ - ' 3', - ' 5', - ' 7', - ' 9', - 'a 2', - 'b c d e 4', - 'øut 8', - 'über 10', - '♣ -1', - '愤青 -3', - '← -5', - ] - self.assertEqual( - b'', - ops.vocab_id_to_token(3, vocab=vocab, - load_token_ids_from_vocab=True).eval()) - self.assertEqual( - b'a', - ops.vocab_id_to_token(2, vocab=vocab, - load_token_ids_from_vocab=True).eval()) - self.assertAllEqual([b'b c d e', u'♣'.encode('utf-8')], - ops.vocab_id_to_token( - [4, -1], - vocab=vocab, - load_token_ids_from_vocab=True).eval()) - self.assertEqual( - b'', - ops.vocab_id_to_token(7, vocab=vocab, - load_token_ids_from_vocab=True).eval()) - self.assertEqual( - b'', - ops.vocab_id_to_token(0, vocab=vocab, - load_token_ids_from_vocab=True).eval()) - - def testTokenInVocab(self): - with self.session(use_gpu=False): - vocab = [ - '', - '', - '', - '', - 'a', - 'b c d e', - 'øut', - 'über', - '♣', - '愤青', - '←', - ] - self.assertTrue(ops.token_in_vocab('a', vocab=vocab).eval()) - self.assertTrue(ops.token_in_vocab('', vocab=vocab).eval()) - self.assertTrue( - ops.token_in_vocab(['b c d e', '♣'], vocab=vocab).eval().all()) - self.assertFalse(ops.token_in_vocab('unknown', vocab=vocab).eval()) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/ops/static_map_op.cc b/lingvo/core/ops/static_map_op.cc deleted file mode 100644 index cf3abe6f..00000000 --- a/lingvo/core/ops/static_map_op.cc +++ /dev/null @@ -1,125 +0,0 @@ -/* Copyright 2019 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ -#include -#include -#include - -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" -#include "tensorflow/core/lib/gtl/flatmap.h" -#include "tensorflow/core/lib/strings/str_util.h" -#include "tensorflow/core/platform/logging.h" -#include "tensorflow/core/util/work_sharder.h" - -namespace tensorflow { -namespace lingvo { -namespace { - -template -void Iota(std::vector* vec) { - std::iota(vec->begin(), vec->end(), T()); -} - -template <> -void Iota(std::vector* vec) { - // Do nothing. -} - -template -class StaticMapOp : public OpKernel { - public: - explicit StaticMapOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - std::vector keys; - OP_REQUIRES_OK(ctx, ctx->GetAttr("keys", &keys)); - std::vector vals; - OP_REQUIRES_OK(ctx, ctx->GetAttr("vals", &vals)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("unk", &unk_)); - - if (keys.empty() && !vals.empty()) { - keys.resize(vals.size()); - Iota(&keys); - } - - if (!keys.empty() && vals.empty()) { - vals.resize(keys.size()); - Iota(&vals); - } - - OP_REQUIRES(ctx, keys.size() == vals.size(), - errors::InvalidArgument("keys and vals are different sizes: ", - keys.size(), " / ", vals.size())); - - for (int i = 0; i < keys.size(); ++i) { - OP_REQUIRES(ctx, dict_.insert({keys[i], vals[i]}).second, - errors::InvalidArgument("keys have duplicates: ", keys[i])); - } - } - - void Compute(OpKernelContext* ctx) override { - const Tensor& x = ctx->input(0); - Tensor* y; - OP_REQUIRES_OK(ctx, ctx->allocate_output(0, x.shape(), &y)); - - auto tx = x.flat(); - auto ty = y->flat(); - int total = tx.size(); - auto workers = ctx->device()->tensorflow_cpu_worker_threads(); - Shard(workers->num_threads, workers->workers, total, 250, - [this, &tx, &ty](int64 start, int64 limit) { - for (int64 i = start; i < limit; ++i) { - auto it = dict_.find(tx(i)); - if (it != dict_.end()) { - ty(i) = it->second; - } else { - ty(i) = unk_; - } - } - }); - } - - protected: - tensorflow::gtl::FlatMap dict_; - V unk_; -}; - -REGISTER_KERNEL_BUILDER(Name("StaticMapStringInt").Device(DEVICE_CPU), - StaticMapOp); -REGISTER_KERNEL_BUILDER(Name("StaticMapIntString").Device(DEVICE_CPU), - StaticMapOp); -REGISTER_KERNEL_BUILDER(Name("StaticMapIntInt").Device(DEVICE_CPU), - StaticMapOp); - -#if GOOGLE_CUDA -REGISTER_KERNEL_BUILDER(Name("StaticMapStringInt") - .Device(DEVICE_GPU) - .HostMemory("x") - .HostMemory("y"), - StaticMapOp); - -REGISTER_KERNEL_BUILDER(Name("StaticMapIntString") - .Device(DEVICE_GPU) - .HostMemory("x") - .HostMemory("y"), - StaticMapOp); - -REGISTER_KERNEL_BUILDER( - Name("StaticMapIntInt").Device(DEVICE_GPU).HostMemory("x").HostMemory("y"), - StaticMapOp); -#endif - -} // namespace -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/static_map_op_test.py b/lingvo/core/ops/static_map_op_test.py deleted file mode 100644 index 6fbd7c04..00000000 --- a/lingvo/core/ops/static_map_op_test.py +++ /dev/null @@ -1,60 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for static_map_op.""" - -from lingvo import compat as tf -from lingvo.core import ops -from lingvo.core import test_utils - -FLAGS = tf.flags.FLAGS - - -class StaticMapOpsTest(test_utils.TestCase): - - def testStaticMap(self): - with self.session(): - self.assertAllEqual([[-1, -2, -3], [-1, -5, -6]], - ops.static_map_int_int( - x=[[1, 2, 3], [8, 5, 6]], - keys=[1, 2, 3, 4, 5, 6], - vals=[-1, -2, -3, -4, -5, -6]).eval()) - self.assertAllEqual([[1, 3, 5], [7, 9, 11]], - ops.static_map_string_int( - x=[['a', 'b', 'c'], ['d', 'e', 'f']], - keys=['d', 'e', 'f', 'a', 'b', 'c'], - vals=[7, 9, 11, 1, 3, 5]).eval()) - self.assertAllEqual([[3, 4, 5], [0, 1, 2]], - ops.static_map_string_int( - x=[['a', 'b', 'c'], ['d', 'e', 'f']], - keys=['d', 'e', 'f', 'a', 'b', 'c']).eval()) - self.assertAllEqual([[2, -1, -1], [0, -1, 1]], - ops.static_map_string_int( - x=[['a', 'b', 'c'], ['d', 'e', 'f']], - keys=['d', 'f', 'a']).eval()) - - # Error cases. - with self.assertRaisesRegex(tf.errors.InvalidArgumentError, 'sizes'): - ops.static_map_string_int( - x=[['a', 'b', 'c'], ['d', 'e', 'f']], - keys=['d', 'f', 'a'], - vals=[1, 2]).eval() - with self.assertRaisesRegex(tf.errors.InvalidArgumentError, 'duplicates'): - ops.static_map_string_int( - x=[['a', 'b', 'c'], ['d', 'e', 'f']], keys=['d', 'f', 'd']).eval() - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/ops/testdata/BUILD b/lingvo/core/ops/testdata/BUILD deleted file mode 100644 index b66733b7..00000000 --- a/lingvo/core/ops/testdata/BUILD +++ /dev/null @@ -1,44 +0,0 @@ -package(default_visibility = ["//visibility:public"]) - -licenses(["notice"]) # Apache 2.0 - -filegroup( - name = "best_step_testdata", - data = [ - "events.out.tfevents.test", - "history.txt", - "history_bleu.txt", - ], -) - -filegroup( - name = "test_vocab", - data = ["test_vocab.txt"], -) - -filegroup( - name = "test_ngrams", - data = ["test_ngrams.txt"], -) - -filegroup( - name = "bpe_words_vocab", - data = ["bpe_words.vocab"], -) - -filegroup( - name = "bpe_codes_vocab", - data = ["bpe_codes.vocab"], -) - -# Data from "Findings of the 2014 Workshop on Statistical Machine Translation" -# http://www.aclweb.org/anthology/W14-3302 -filegroup( - name = "wmt", - data = glob(["wmt/*"]), -) - -filegroup( - name = "mlperf_vocab", - data = ["mlperf.ende.subwords.vocab"], -) diff --git a/lingvo/core/ops/testdata/bpe_codes.vocab b/lingvo/core/ops/testdata/bpe_codes.vocab deleted file mode 100644 index 8599b2fe..00000000 --- a/lingvo/core/ops/testdata/bpe_codes.vocab +++ /dev/null @@ -1,67 +0,0 @@ - - - - -E@@ 44 -O@@ 35 -N@@ 32 -A@@ 31 -S@@ 31 -I@@ 28 -Y 26 -P@@ 26 -L@@ 23 -S 22 -E 21 -H@@ 20 -TH@@ 18 -T 18 -N 17 -B@@ 17 -U@@ 17 -R@@ 15 -F@@ 15 -W@@ 14 -ED 13 -ER@@ 13 -THE 13 -G@@ 13 -M@@ 12 -ER 12 -V@@ 11 -TOM 11 -HE 11 -OU@@ 10 -D 10 -AT 10 -WAS 10 -IN@@ 10 -HI@@ 9 -Y@@ 9 -AD 9 -C@@ 9 -D@@ 9 -T@@ 9 -TO 8 -'@@ 8 -LA@@ 8 -O 8 -AND 8 -ON@@ 8 -K@@ 7 -WA@@ 7 -A 7 -G 7 -AN@@ 5 -L 5 -R 5 -U 4 -K 3 -F 3 -W 3 -H 3 -TO@@ 2 -Q@@ 1 -X@@ 1 -I 1 -M 1 diff --git a/lingvo/core/ops/testdata/bpe_words.vocab b/lingvo/core/ops/testdata/bpe_words.vocab deleted file mode 100644 index 6baa4758..00000000 --- a/lingvo/core/ops/testdata/bpe_words.vocab +++ /dev/null @@ -1,143 +0,0 @@ -A 52 -ACT 7,41,17 -AFTER 7,22,43,29 -ALL 7,12,55 -ALONE 7,12,49,14 -ALTHOUGH 7,12,16,33,27,61 -AND 48 -ANSWERED 54,8,23,25,24 -ANY 54,10 -AROUND 7,21,33,6,34 -ASKED 7,8,50,24 -AT 35 -AWAY 7,51,10 -BABY 19,7,19,10 -BAD 19,40 -BADLY 19,7,42,12,10 -BARNEY 19,7,21,6,4,10 -BEAT 19,4,35 -BEEN 19,4,4,18 -BEGUN 19,4,27,20,18 -BOWSER'S 19,5,23,8,25,45,13 -BOY 19,5,10 -BUN 19,20,18 -BUT 19,20,17 -CAUGHT 41,7,20,27,15,17 -CAUTIOUS 41,7,20,43,9,33,13 -CRYING 41,21,39,37,53 -DAY 42,7,10 -DID 42,9,34 -DIED 42,9,24 -DO 42,47 -DOWN 42,5,23,18 -EAT 4,35 -END 4,6,34 -EVER 4,30,29 -EXCEPT 4,64,41,4,11,17 -FALLEN 22,7,12,12,4,18 -FARMER 22,7,21,28,29 -FATHER 22,7,16,29 -FATHER'S 22,7,16,25,45,13 -FELL 22,4,12,55 -FEW 22,4,60 -FOR 22,5,56 -FOUND 22,33,6,34 -GIVE 27,9,30,14 -GO 27,47 -GOOD 27,5,5,34 -GUIDANCE 27,20,9,42,54,41,14 -HAD 15,40 -HE 32 -HEAD 15,4,40 -HILL 38,12,55 -HIM 38,66 -HIS 38,13 -HONEST 15,49,4,8,17 -HOUSE 15,33,8,14 -HUNGRY 15,20,6,27,21,10 -HUT 15,20,17 -I 65 -IF 9,59 -IN 9,18 -INTO 37,44 -IT 9,17 -KNOW 50,6,5,60 -LAST 46,8,17 -LEARNED 12,4,7,21,6,24 -LITTLE 12,9,43,43,12,14 -LIVED 12,9,30,24 -MAN'S 28,54,45,13 -MANY 28,54,10 -MAY 28,7,10 -ME 28,14 -MISSED 28,9,8,8,24 -MORNING 28,5,21,6,37,53 -MOTHER 28,5,16,29 -MUST 28,20,8,17 -MY 28,10 -NAMED 6,7,28,24 -NEITHER 6,4,9,16,29 -NEVER 6,4,30,29 -NO 6,47 -NONE 6,49,14 -NOT 6,5,17 -OF 5,59 -OLD 5,12,34 -ONE 49,14 -OUT 33,17 -OVER 5,30,29 -PENNIES 11,4,6,6,9,4,13 -PENNY 11,4,6,6,10 -PIG 11,9,53 -PIPER'S 11,9,11,25,45,13 -PIPES 11,9,11,4,13 -PLAY 11,46,10 -PLAYED 11,46,39,24 -PLAYING 11,46,39,37,53 -POUCH 11,33,41,61 -QUEER 63,20,4,29 -RAN 21,7,18 -REPLIED 21,4,11,12,9,24 -SHAKING 8,15,7,50,37,53 -SHALL 8,15,7,12,55 -SINCE 8,37,41,14 -SLY 8,12,10 -SO 8,47 -SON 8,5,18 -SOON 8,5,5,18 -STEALING 8,43,4,7,12,37,53 -STREET 8,43,21,4,4,17 -SUCH 8,20,41,61 -SUPPOSE 8,20,11,11,5,8,14 -TAKE 43,7,50,14 -THAT 16,35 -THE 26 -THEIR 16,4,9,56 -THERE 16,25,14 -THEY 16,4,10 -THINGS 16,37,27,13 -TO 44 -TOM 31 -TOM'S 62,28,45,13 -TOOK 62,5,58 -UNDER 20,6,42,29 -UNLESS 20,6,12,4,8,13 -VERY 30,25,10 -VILLAGE 30,9,12,46,27,14 -WALKED 51,12,50,24 -WANT 51,6,17 -WANTED 51,6,43,24 -WAS 36 -WAY 51,10 -WAYS 51,39,13 -WE 23,14 -WEEK 23,4,4,58 -WENT 23,4,6,17 -WERE 23,25,14 -WHAT 23,15,35 -WHEN 23,15,4,18 -WHISPERED 23,38,8,11,25,24 -WILL 23,9,12,55 -WORK 23,5,21,58 -YOU 39,5,57 -YOUNG 39,33,6,53 diff --git a/lingvo/core/ops/testdata/events.out.tfevents.test b/lingvo/core/ops/testdata/events.out.tfevents.test deleted file mode 100644 index a59f35f4..00000000 Binary files a/lingvo/core/ops/testdata/events.out.tfevents.test and /dev/null differ diff --git a/lingvo/core/ops/testdata/history.txt b/lingvo/core/ops/testdata/history.txt deleted file mode 100644 index 42fa9dbe..00000000 --- a/lingvo/core/ops/testdata/history.txt +++ /dev/null @@ -1,131 +0,0 @@ -0 10.373487 -298 7.033164 -625 6.470803 -950 6.105780 -1279 5.820248 -1608 5.530968 -1935 5.308480 -2263 5.057889 -2590 4.815093 -2919 4.659936 -3247 4.439473 -3570 4.281693 -3897 4.187109 -4223 4.042700 -4549 3.953553 -4875 3.898013 -5201 3.795292 -5527 3.737963 -5854 3.658121 -6186 3.592632 -6512 3.559976 -6840 3.488496 -7167 3.438186 -7496 3.396586 -7826 3.353678 -8149 3.312054 -8477 3.272446 -8814 3.260937 -9141 3.212824 -9472 3.180816 -9805 3.159665 -10136 3.120418 -10470 3.113019 -10797 3.063556 -11125 3.042007 -11453 3.039486 -11780 3.001927 -12106 2.983079 -12442 2.956867 -12768 2.934884 -13096 2.926650 -13420 2.908608 -13748 2.884295 -14074 2.875808 -14402 2.846520 -14729 2.831213 -15066 2.819663 -15394 2.805727 -15783 2.788498 -16119 2.775941 -16446 2.772502 -16771 2.752264 -17095 2.743278 -17435 2.723992 -17759 2.712796 -18088 2.703130 -18416 2.692253 -18746 2.680207 -19076 2.669302 -19409 2.656425 -19761 2.644988 -20091 2.640521 -20438 2.626620 -20765 2.619319 -21094 2.603612 -21422 2.603181 -21751 2.594443 -22082 2.580626 -22408 2.573240 -22738 2.568288 -23067 2.563837 -23517 2.544863 -23850 2.542182 -24178 2.542404 -24503 2.525611 -24830 2.521632 -25156 2.515293 -25482 2.500205 -25810 2.505846 -26138 2.489248 -26462 2.489169 -26789 2.480322 -27116 2.474624 -27443 2.461814 -27768 2.454810 -28097 2.459774 -28420 2.448776 -28749 2.439800 -29072 2.432903 -29402 2.430021 -29726 2.430029 -30049 2.423343 -30376 2.414850 -30706 2.406802 -31028 2.407033 -31353 2.397416 -31690 2.387606 -32019 2.388219 -32345 2.385543 -32673 2.379700 -33007 2.379280 -33334 2.375431 -33663 2.370022 -33989 2.360705 -34314 2.362377 -34637 2.363039 -34963 2.357430 -35290 2.348929 -35615 2.344244 -35939 2.337197 -36241 2.337495 -36566 2.331192 -36890 2.333216 -37228 2.324359 -37553 2.322670 -37878 2.323006 -38207 2.321044 -38531 2.321342 -38851 2.316578 -39177 2.315294 -39504 2.316487 -39835 2.308898 -40163 2.302155 -40488 2.302747 -40814 2.297515 -41144 2.304822 -41469 2.299697 -41796 2.289855 -42122 2.287741 -42464 2.288147 -42792 2.288710 diff --git a/lingvo/core/ops/testdata/history_bleu.txt b/lingvo/core/ops/testdata/history_bleu.txt deleted file mode 100644 index 66d3ff41..00000000 --- a/lingvo/core/ops/testdata/history_bleu.txt +++ /dev/null @@ -1,97 +0,0 @@ -0 0.000000 -1400 0.087791 -1800 0.151373 -2300 0.190187 -2800 0.213798 -3300 0.231403 -3800 0.239085 -4300 0.246855 -4700 0.254193 -5200 0.259330 -5700 0.264080 -6200 0.266159 -6700 0.267900 -7000 0.274283 -7400 0.276814 -7900 0.279553 -8400 0.277692 -8900 0.279560 -9300 0.280146 -9800 0.285728 -10300 0.285033 -10800 0.286540 -11300 0.289871 -11800 0.290158 -12200 0.290732 -12700 0.168033 -13200 0.290840 -13700 0.296553 -14200 0.292199 -14600 0.294382 -15100 0.295986 -15600 0.294707 -16100 0.299057 -16600 0.294427 -17000 0.294130 -17500 0.295190 -18000 0.296744 -18500 0.297920 -19000 0.299808 -19400 0.300091 -19900 0.301278 -20400 0.298620 -20900 0.303506 -21300 0.301692 -21800 0.303308 -22300 0.303862 -22800 0.305339 -23300 0.306642 -23800 0.301688 -24200 0.306026 -24700 0.304330 -25200 0.308000 -25700 0.307455 -26100 0.308091 -26600 0.308430 -27100 0.308874 -27600 0.308740 -28100 0.259248 -28600 0.311853 -29000 0.309061 -29500 0.309771 -30000 0.310924 -30500 0.311255 -31000 0.310881 -31400 0.310177 -31900 0.313925 -32400 0.309031 -32900 0.310402 -33400 0.311731 -33900 0.312070 -34300 0.312387 -34800 0.312391 -35300 0.309882 -35800 0.312969 -36300 0.312694 -36700 0.315303 -37200 0.314271 -37700 0.313711 -38200 0.284366 -38600 0.313390 -39100 0.312131 -39600 0.315337 -40100 0.312599 -40600 0.315118 -41000 0.313815 -41500 0.316021 -42000 0.313759 -42500 0.314820 -43000 0.311660 -43500 0.314023 -43900 0.312915 -44400 0.312980 -44900 0.315555 -45400 0.312320 -45900 0.313274 -46300 0.312491 -46800 0.314273 diff --git a/lingvo/core/ops/testdata/mlperf.ende.subwords.vocab b/lingvo/core/ops/testdata/mlperf.ende.subwords.vocab deleted file mode 100644 index 68dcf208..00000000 --- a/lingvo/core/ops/testdata/mlperf.ende.subwords.vocab +++ /dev/null @@ -1,33708 +0,0 @@ -'_' -'_' -', _' -'._' -'the_' -'_' -'in_' -'of_' -'and_' -'to_' -'die_' -'der_' -'und_' -'a_' -'s_' -'-_' -'is_' -'that_' -'zu_' -'for_' -'den_' -'von_' -'n_' -'on_' -'ist_' -'an_' -'für_' -'. _' -'be_' -'The_' -'with_' -'en_' -'es_' -'are_' -'das_' -'as_' -'e_' -'des_' -'auf_' -'mit_' -'it_' -'eine_' -'dass_' -'nicht_' -'I_' -'im_' -'not_' -'have_' -'by_' -'this_' -' (_' -' – _' -'sich_' -'was_' -'ein_' -'werden_' -'Die_' -'will_' -'from_' -'we_' -'dem_' -'’_' -'t_' -': _' -'at_' -'or_' -'Sie_' -'which_' -'has_' -'er_' -'als_' -'auch_' -'you_' -'wir_' -'r_' -'In_' -'um_' -'sind_' -'wird_' -') _' -'so_' -'can_' -'sie_' -'ing_' -'all_' -''_' -' - _' -'einer_' -'hat_' -'wie_' -'also_' -'their_' -'European_' -'haben_' -'d_' -'would_' -'ed_' -'oder_' -'its_' -'more_' -'über_' -'but_' -'?_' -'einen_' -'ich_' -'y_' -'zur_' -'our_' -'they_' -'aus_' -'bei_' -'Das_' -'one_' -'been_' -'; _' -'nur_' -'Union_' -'should_' -'It_' -'EU_' -'einem_' -'/_' -'nach_' -'durch_' -'This_' -'können_' -'diese_' -'ung_' -'other_' -'zum_' -'noch_' -'only_' -'there_' -' , _' -'do_' -'am_' -'de_' -'countries_' -'1_' -'kann_' -'dieser_' -'war_' -'than_' -'We_' -'new_' -'o_' -'your_' -'Europe_' -'Der_' -'must_' -'Mr_' -'no_' -'vor_' -'were_' -'2_' -'like_' -'wenn_' -'man_' -'US_' -'Ich_' -'wurde_' -'- _' -'about_' -' "_' -'us_' -'President_' -'m_' -'time_' -'Es_' -'these_' -'if_' -'aber_' -'te_' -'sein_' -'who_' -'up_' -'very_' -'Hotel_' -'world_' -' ._' -'uns_' -'Commission_' -'when_' -'such_' -'A_' -'But_' -'Wir_' -'people_' -'müssen_' -' “_' -'into_' -'ten_' -'ng_' -'China_' -'out_' -'3_' -'mehr_' -'ihre_' -'his_' -'5_' -'now_' -'most_' -'some_' -'what_' -'sehr_' -'Kommission_' -'many_' -'!_' -'i_' -')._' -'l_' -'he_' -'any_' -'% _' -'had_' -' „_' -'States_' -'them_' -',_' -'eines_' -'4_' -'well_' -'Herr_' -'), _' -'" _' -'economic_' -'diesem_' -'need_' -'unter_' -'years_' -'political_' -'between_' -'ly_' -'zwischen_' -'first_' -'hotel_' -'alle_' -'even_' -'policy_' -'make_' -'bis_' -'two_' -'muss_' -'could_' -'over_' -'anderen_' -'use_' -'Parliament_' -'keine_' -'my_' -'work_' -'may_' -'way_' -'important_' -'Council_' -'gegen_' -'report_' -'Präsident_' -'0_' -'system_' -'Europäischen_' -'Europa_' -'gibt_' -'because_' -'If_' -'those_' -'just_' -'support_' -'vom_' -'seine_' -'sowie_' -'k_' -'country_' -'year_' -'much_' -'Wenn_' -'dieses_' -'after_' -'government_' -'Member_' -'al_' -'made_' -' _' -'ungen_' -'able_' -'take_' -'h_' -'möchte_' -'market_' -'being_' -'immer_' -'“ _' -'” _' -'public_' -'own_' -'long_' -'' _' -'Welt_' -'dies_' -'sondern_' -'Zeit_' -'Menschen_' -'Jahren_' -'international_' -'(_' -'where_' -'right_' -'good_' -'financial_' -'how_' -'ihrer_' -'da_' -'diesen_' -'USA_' -'wurden_' -'andere_' -'For_' -':_' -'g_' -'As_' -'Diese_' -'Jahr_' -'both_' -'information_' -'against_' -'Länder_' -'part_' -'same_' -'last_' -'Bericht_' -'unsere_' -'global_' -'dann_' -'z_' -'through_' -'würde_' -'re_' -'6_' -'then_' -'sollte_' -'There_' -'jedoch_' -'hier_' -'high_' -'does_' -'end_' -'damit_' -'Im_' -'10_' -'seiner_' -'heute_' -'S_' -'United_' -'Und_' -'under_' -'social_' -'too_' -'ion_' -'7_' -'national_' -'order_' -'growth_' -'8_' -'example_' -'still_' -'see_' -'me_' -'le_' -'neue_' -'ation_' -'ohne_' -'free_' -'europäischen_' -'Parlament_' -'Land_' -'number_' -'That_' -'Mitgliedstaaten_' -'rights_' -'place_' -'könnte_' -'development_' -'area_' -'And_' -'within_' -'power_' -'her_' -'course_' -'room_' -'point_' -'fact_' -'before_' -'bereits_' -'used_' -'Frage_' -'’ _' -'neuen_' -'while_' -'denen_' -'far_' -'possible_' -'Entwicklung_' -'Ein_' -'20_' -'selbst_' -'wieder_' -'economy_' -' [[_' -'want_' -'la_' -'future_' -'sollten_' -'You_' -'zwei_' -'dazu_' -'Europäische_' -'say_' -'Regierung_' -'great_' -'already_' -' | _' -'without_' -'c_' -'set_' -'Aber_' -'less_' -'9_' -'large_' -'während_' -'human_' -'here_' -'! _' -'weil_' -'today_' -'jetzt_' -'ihren_' -'30_' -'ihr_' -'So_' -'view_' -'se_' -'machen_' -'wäre_' -'therefore_' -'cannot_' -'believe_' -'problem_' -'liegt_' -'wo_' -'since_' -'go_' -'Lage_' -' '_' -'u_' -'crisis_' -'three_' -'state_' -'per_' -'find_' -'few_' -'down_' -'America_' -'et_' -'Ländern_' -'viele_' -'services_' -'000_' -'..._' -'st_' -'process_' -'issue_' -'help_' -'Unternehmen_' -'trade_' -'including_' -'available_' -'15_' -'level_' -'case_' -'Maßnahmen_' -'viel_' -'know_' -'geht_' -'einige_' -'Eine_' -'means_' -'darauf_' -'denn_' -'dafür_' -'00_' -'next_' -'mich_' -'different_' -'Jahre_' -'seit_' -'city_' -'change_' -'areas_' -'rs_' -'real_' -'get_' -'problems_' -'When_' -'Staaten_' -'health_' -'el_' -'Politik_' -'E_' -'C_' -'   _' -'making_' -'best_' -'Mit_' -'Ihnen_' -'however_' -'clear_' -'better_' -'allem_' -'politischen_' -'lassen_' -'finden_' -'x_' -'why_' -'habe_' -'gut_' -'Frau_' -'ers_' -'Dies_' -'B_' -'ne_' -'D_' -'energy_' -'during_' -'become_' -'allen_' -'They_' -'service_' -'access_' -'waren_' -'ganz_' -'Japan_' -'2009_' -'unserer_' -'etwas_' -'daß_' -'Committee_' -'current_' -'wollen_' -'question_' -'doch_' -'stellen_' -'politische_' -'com_' -'back_' -'To_' -'yang_' -'particular_' -'small_' -'ob_' -'did_' -'day_' -'Rat_' -'think_' -'These_' -'Israel_' -'based_' -'bar_' -'ve_' -'interest_' -'debate_' -'common_' -'beim_' -'Doch_' -'Commissioner_' -'letzten_' -'each_' -'again_' -'bietet_' -'Germany_' -'At_' -'security_' -'nun_' -'measures_' -'business_' -'Zimmer_' -'situation_' -'schon_' -'put_' -'offers_' -'p_' -'markets_' -'Wie_' -'ts_' -'tion_' -'taken_' -'seinen_' -'military_' -'major_' -'Ihre_' -'ge_' -'come_' -'another_' -'Bereich_' -'Arbeit_' -'stay_' -'Iran_' -'rate_' -'provide_' -'Ende_' -'tun_' -'result_' -'rather_' -'law_' -'continue_' -'citizens_' -'ce_' -'50_' -'said_' -'action_' -'page_' -'might_' -'House_' -'term_' -'American_' -'2008_' -'worden_' -'recent_' -'given_' -'Unterstützung_' -'Internet_' -'Euro_' -'whether_' -'ies_' -'every_' -'Probleme_' -'|_' -'ment_' -'give_' -'einfach_' -'2000_' -'steht_' -'rooms_' -'governments_' -'ersten_' -'debt_' -'called_' -'Teil_' -'Auch_' -'besteht_' -'Problem_' -'sten_' -'proposal_' -'needs_' -'gen_' -'budget_' -'Bank_' -'12_' -'private_' -'ns_' -'Herrn_' -'weniger_' -'lich_' -'institutions_' -'full_' -'erhalten_' -'On_' -'New_' -'Fall_' -'11_' -'non_' -'P_' -'vote_' -'terms_' -'os_' -'issues_' -'f_' -'data_' -'away_' -'Nach_' -'sehen_' -'little_' -'line_' -'least_' -'further_' -'around_' -'always_' -'Ziel_' -'Sicherheit_' -'Als_' -'que_' -'million_' -'großen_' -'du_' -'ch_' -'France_' -'ter_' -'open_' -'eigenen_' -'Russia_' -'Rolle_' -'systems_' -'life_' -'insbesondere_' -'hatte_' -'group_' -'geben_' -'close_' -'World_' -'sagen_' -'hope_' -'With_' -'Art_' -'All_' -'zwar_' -'using_' -'quality_' -'policies_' -'main_' -'legal_' -'ern_' -'deren_' -'ons_' -'off_' -'located_' -'ab_' -'System_' -'Bürger_' -'money_' -'mir_' -'kommen_' -'bin_' -'age_' -'What_' -'Namen_' -'risk_' -'investment_' -'foreign_' -'drei_' -'ble_' -'allerdings_' -'Zusammenarbeit_' -'Wirtschaft_' -'He_' -'Bedeutung_' -'2005_' -'role_' -'position_' -'große_' -']] _' -'Was_' -'Informationen_' -'würden_' -'women_' -'möglich_' -'form_' -'certain_' -'T_' -'2006_' -'etwa_' -'einmal_' -'done_' -'German_' -'weiter_' -'sector_' -'agreement_' -'2007_' -'rules_' -'increase_' -'ihnen_' -'dan_' -'companies_' -'basis_' -'b_' -'unseren_' -'particularly_' -'members_' -'local_' -'führen_' -'di_' -'create_' -'century_' -'öffentlichen_' -'pro_' -'old_' -'necessary_' -'erreichen_' -'Er_' -'Deutschland_' -'Bei_' -'.”_' -'start_' -'second_' -'once_' -'davon_' -'ago_' -'O_' -'yet_' -'together_' -'ry_' -'going_' -'soll_' -'others_' -'könnten_' -'dort_' -'staff_' -'\u_' -'Für_' -' & _' -'stehen_' -'needed_' -'location_' -'ensure_' -'dabei_' -'conditions_' -'Millionen_' -'Community_' -'Auf_' -'2001_' -'weitere_' -'though_' -'rn_' -'often_' -'key_' -'ins_' -'alles_' -'short_' -'price_' -'meisten_' -'him_' -'fully_' -'especially_' -'September_' -'Fragen_' -'rates_' -'control_' -'central_' -'Weise_' -'Weg_' -'One_' -'low_' -'kein_' -'itself_' -'greater_' -'gegenüber_' -'capital_' -'Rahmen_' -'nt_' -'modern_' -'decision_' -'aller_' -' _' -'sogar_' -'cooperation_' -'Recht_' -'However_' -'innerhalb_' -'costs_' -'Vorschlag_' -'reform_' -'led_' -'food_' -'Zukunft_' -'Wachstum_' -'Stadt_' -'Indeed_' -'2004_' -' $_' -'wissen_' -'third_' -'region_' -'account_' -' ..._' -'leaders_' -'among_' -'Seite_' -'M_' -'Beispiel_' -'whole_' -'tax_' -'protection_' -'present_' -'ic_' -'early_' -'aid_' -'Um_' -'Europas_' -'100_' -'vielen_' -'schen_' -'name_' -'meine_' -'ischen_' -'income_' -'following_' -'excellent_' -'darüber_' -'children_' -'banks_' -'25_' -'welche_' -'several_' -'reason_' -'nen_' -'having_' -'UN_' -'Seiten_' -'weit_' -'progress_' -'centre_' -'Chinese_' -'wirklich_' -'really_' -'möchten_' -'macht_' -'kommt_' -'ja_' -'europäische_' -'value_' -'resources_' -'environment_' -'bringen_' -'bieten_' -'Region_' -'Möglichkeit_' -'16_' -'water_' -'something_' -'left_' -'internationalen_' -'home_' -'democratic_' -'Zusammenhang_' -'Grund_' -'Frauen_' -'2003_' -'sowohl_' -'include_' -'democracy_' -'daher_' -'weiterhin_' -'stellt_' -'states_' -'production_' -'kan_' -'fast_' -'Bush_' -'wirtschaftlichen_' -'single_' -'shall_' -'nden_' -'longer_' -'efforts_' -'Mrs_' -'India_' -'working_' -'until_' -'ty_' -'poor_' -'matter_' -'land_' -'force_' -'chen_' -'French_' -'society_' -'indem_' -'2002_' -'unterstützen_' -'regard_' -'offer_' -'nd_' -'nationalen_' -'list_' -'likely_' -'halten_' -'No_' -'Group_' -'Central_' -'period_' -'never_' -'meiner_' -'gentlemen_' -'entfernt_' -'difficult_' -'beiden_' -'Kinder_' -'18_' -'un_' -'prices_' -'look_' -'fiscal_' -'besonders_' -'ar_' -'approach_' -'Parlaments_' -'Mittel_' -'views_' -'verschiedenen_' -'standards_' -'results_' -'respect_' -'resolution_' -'research_' -'les_' -'industry_' -'developing_' -'cost_' -'L_' -'Iraq_' -'International_' -'G_' -'Dollar_' -' % _' -'wichtig_' -'special_' -'member_' -'hand_' -'created_' -'U_' -'N_' -'Investitionen_' -'40_' -'wirtschaftliche_' -'show_' -'nehmen_' -'management_' -'interests_' -'enough_' -'breakfast_' -'State_' -'24_' -'wenig_' -'proposals_' -'parties_' -'nichts_' -'ihm_' -'experience_' -'etc_' -'almost_' -'Vereinigten_' -'R_' -'Obama_' -'Leben_' -' ' -'transport_' -'taking_' -'remain_' -'programme_' -'play_' -'near_' -'general_' -'family_' -'erste_' -'aufgrund_' -'address_' -'Ziele_' -'Fraktion_' -'Daten_' -'side_' -'ren_' -'products_' -'men_' -'ihn_' -'history_' -'hinaus_' -'easy_' -'billion_' -'Service_' -'Richtlinie_' -'Krise_' -'Grundlage_' -'thus_' -'things_' -'site_' -'sei_' -'questions_' -'ous_' -'nuclear_' -'known_' -'ierung_' -'feel_' -'call_' -'building_' -'Thema_' -'Russland_' -'Regierungen_' -'Kollegen_' -'towards_' -'seinem_' -'ner_' -'iert_' -'ieren_' -'hin_' -'dessen_' -'befindet_' -'Verfügung_' -'Landes_' -'An_' -'wish_' -'technology_' -'rt_' -'proposed_' -'natürlich_' -'keinen_' -'demand_' -'darin_' -'Lösung_' -'Kommissar_' -'F_' -'14_' -'workers_' -'themselves_' -'solution_' -'projects_' -'pay_' -'months_' -'ity_' -'Programm_' -'Macht_' -'version_' -'regional_' -'program_' -'past_' -'ness_' -'lichen_' -'ische_' -'hard_' -'ever_' -'eren_' -'ask_' -'adopted_' -'Gemeinschaft_' -'East_' -'By_' -'60_' -'strategy_' -'points_' -'personal_' -'lead_' -'ive_' -'ihrem_' -'ia_' -'higher_' -'gute_' -'directive_' -'cultural_' -'beispielsweise_' -'agree_' -'Man_' -'welcome_' -'ted_' -'software_' -'sicher_' -'serious_' -'sche_' -'liche_' -'levels_' -'gilt_' -'gehen_' -'found_' -'either_' -'effective_' -'education_' -'above_' -'Situation_' -'After_' -'19_' -'vielleicht_' -'various_' -'specific_' -'schnell_' -'schaffen_' -'related_' -'freedom_' -'deal_' -'che_' -'besser_' -'bedeutet_' -'South_' -'Institutionen_' -'21_' -'rapporteur_' -'ls_' -'enjoy_' -'economies_' -'ebenfalls_' -'direkt_' -'bleiben_' -'big_' -'authorities_' -'allow_' -'November_' -'Meinung_' -'Even_' -'Ansicht_' -'. _' -'à_' -'user_' -'seen_' -'remains_' -'reforms_' -'ra_' -'provided_' -'plan_' -'opportunity_' -'ladies_' -'keit_' -'impact_' -'groups_' -'framework_' -'ens_' -'comes_' -'Schutz_' -'th_' -'strong_' -'simply_' -'significant_' -'quite_' -'let_' -'leading_' -'language_' -'em_' -'concerns_' -'climate_' -'behalf_' -'X_' -'Restaurant_' -'Kosten_' -'Dieser_' -'Da_' -'Africa_' -'town_' -'ting_' -'party_' -'pages_' -'makes_' -'globalen_' -'due_' -'Our_' -'Hilfe_' -'Demokratie_' -'2010_' -'upon_' -'seems_' -'relations_' -'peace_' -'online_' -'oil_' -'forward_' -'effect_' -'W_' -'Treaty_' -'Of_' -'Menschenrechte_' -'Form_' -'zurück_' -'true_' -'total_' -'subject_' -'safety_' -'later_' -'jeder_' -'growing_' -'face_' -'appropriate_' -'amendments_' -'West_' -'Minister_' -'Geschichte_' -'Alle_' -';_' -'young_' -'top_' -'solche_' -'she_' -'recently_' -'kind_' -'internationale_' -'individual_' -'ig_' -'euro_' -'environmental_' -'ebenso_' -'daran_' -'concerned_' -'Ukraine_' -'Strategie_' -'Madam_' -'Gesellschaft_' -'GDP_' -'Dieses_' -'w_' -'spending_' -'share_' -'positive_' -'opinion_' -'ma_' -'light_' -'führt_' -'eurozone_' -'ert_' -'community_' -'care_' -'attention_' -'Some_' -'Rates_' -'Milliarden_' -'EUR_' -'sure_' -'success_' -'son_' -'sollen_' -'sen_' -'outside_' -'km_' -'improve_' -'huge_' -'half_' -'genau_' -'funds_' -'four_' -'content_' -'changes_' -'addition_' -'Western_' -'é_' -'words_' -'traditional_' -'text_' -'station_' -'setzen_' -'rise_' -'nächsten_' -'nutzen_' -'nothing_' -'ions_' -'idea_' -'gab_' -'fundamental_' -'bekannt_' -'V_' -'T' -'Krieg_' -'Italy_' -'Beziehungen_' -'stand_' -'legislation_' -'ging_' -'field_' -'currently_' -'art_' -'Turkey_' -'Paris_' -'Mitglieder_' -'London_' -'Ihr_' -'Hotels_' -'Geld_' -'Frankreich_' -'Bevölkerung_' -'v_' -'times_' -'thank_' -'sense_' -'restaurant_' -'required_' -'population_' -'person_' -'house_' -'held_' -'heart_' -'gemacht_' -'fall_' -'developed_' -'deutlich_' -'bzw_' -'amerikanischen_' -'Während_' -'Members_' -'British_' -'"_' -' . _' -'ur_' -'minutes_' -'living_' -'inflation_' -'ie_' -'forces_' -'facilities_' -'arbeiten_' -'actually_' -'Zugang_' -'UK_' -'Moreover_' -'M' -'Dienstleistungen_' -'Bedingungen_' -'Auswirkungen_' -'Aus_' -'"._' -'sea_' -'run_' -'product_' -'principle_' -'nämlich_' -'model_' -'majority_' -'ll_' -'jobs_' -'hätte_' -'hours_' -'gesagt_' -'erreicht_' -'date_' -'certainly_' -'alten_' -'across_' -'statt_' -'sch_' -'reduce_' -'potential_' -'los_' -'indeed_' -'hatten_' -'game_' -'exchange_' -'employment_' -'einigen_' -'deshalb_' -'days_' -'cases_' -'benefits_' -'Windows_' -'Schritt_' -'National_' -'Markt_' -'Greece_' -'Bezug_' -'1999_' -')_' -'wichtige_' -'whose_' -'via_' -'unterstützt_' -'unserem_' -'tes_' -'stability_' -'similar_' -'range_' -'keyword_' -'igen_' -'directly_' -'consider_' -'company_' -'beautiful_' -'along_' -'air_' -'Umsetzung_' -'Interesse_' -'Erfolg_' -'Entscheidung_' -'Asia_' -',” _' -'si_' -'self_' -'regulation_' -'pool_' -'parts_' -'natural_' -'media_' -'gerade_' -'five_' -'enden_' -'elections_' -'ca_' -'application_' -'anti_' -'TV_' -'Office_' -'Liste_' -'Ihrer_' -'Haus_' -'Berlin_' -'understand_' -'lot_' -'lediglich_' -'job_' -'ende_' -'emerging_' -'War_' -'Tatsache_' -'Personen_' -'April_' -' : _' -'size_' -'prevent_' -'opportunities_' -'nde_' -'na_' -'mean_' -'lange_' -'involved_' -'cy_' -'conflict_' -'co_' -'capacity_' -'bring_' -'bleibt_' -'bed_' -'achieve_' -'While_' -'Tag_' -'Nähe_' -'January_' -'Ergebnis_' -'Ebene_' -'Debatte_' -'&_' -'wide_' -'thing_' -'sprechen_' -'rule_' -'regions_' -'project_' -'meeting_' -'handelt_' -'eher_' -'complete_' -'body_' -'Deshalb_' -'13_' -'violence_' -'verwendet_' -'verhindern_' -'table_' -'ss_' -'response_' -'politik_' -'je_' -'heißt_' -'ger_' -'focus_' -'dürfen_' -'competition_' -'clearly_' -'check_' -'car_' -'ben_' -'ant_' -'add_' -'Verantwortung_' -'Präsidenten_' -'La_' -'Jahres_' -'Interessen_' -'Government_' -'Afghanistan_' -'17_' -' ( _' -'unser_' -'scheint_' -'move_' -'lässt_' -'jeden_' -'increasingly_' -'image_' -'großer_' -'gleichzeitig_' -'gemeinsame_' -'features_' -'existing_' -'everything_' -'event_' -'ent_' -'ds_' -'computer_' -'clean_' -'civil_' -'bereit_' -'amount_' -'administration_' -'Park_' -'Irak_' -'Gewalt_' -'ways_' -'ste_' -'sozialen_' -'provides_' -'popular_' -'obwohl_' -'nie_' -'negotiations_' -'nature_' -'fight_' -'direct_' -'del_' -'culture_' -'center_' -'came_' -'Z' -'Kampf_' -'II_' -'G' -'23_' -'verfügt_' -'training_' -'six_' -'road_' -'night_' -'network_' -'live_' -'internal_' -'instead_' -'file_' -'decisions_' -'betrifft_' -'balance_' -'ary_' -'Türkei_' -'North_' -'Hier_' -'Bereichen_' -'Banken_' -'Aussprache_' -' -_' -'ya_' -'space_' -'meet_' -'keep_' -'extremely_' -'effects_' -'ck_' -'chinesischen_' -'below_' -'activities_' -'Zu_' -'San_' -'Partei_' -'Möglichkeiten_' -'I' -'Einsatz_' -'BIP_' -'Allerdings_' -'", _' -' ''_' -'weiß_' -'weiteren_' -'week_' -'wahrscheinlich_' -'values_' -'unternehmen_' -'simple_' -'return_' -'rest_' -'perhaps_' -'notwendig_' -'net_' -'infrastructure_' -'increased_' -'included_' -'il_' -'contains_' -'commitment_' -'besten_' -'Staat_' -'Spain_' -'Richtung_' -'Ort_' -'Booking_' -'   – _' -'vergangenen_' -'turn_' -'try_' -'tragen_' -'toward_' -'took_' -'tatsächlich_' -'step_' -'reviews_' -'responsible_' -'poverty_' -'negara_' -'möglicherweise_' -'late_' -'importance_' -'ideal_' -'hätten_' -'hohen_' -'former_' -'favour_' -'essential_' -'doing_' -'design_' -'customers_' -'currency_' -'] _' -'Software_' -'80_' -'“, _' -'zusammen_' -'ziehen_' -'wären_' -'spielen_' -'soziale_' -'performance_' -'oft_' -'moment_' -'lack_' -'kleinen_' -'klar_' -'fünf_' -'erst_' -'derzeit_' -'dar_' -'brauchen_' -'befinden_' -'beach_' -'ally_' -'Wert_' -'Party_' -'Großbritannien_' -'Grenzen_' -'Chinas_' -' ‘_' -'wichtigen_' -'weltweit_' -'wegen_' -'responsibility_' -'require_' -'reasons_' -'ngen_' -'negative_' -'liegen_' -'integration_' -'ings_' -'ian_' -'hen_' -'größten_' -'geführt_' -'external_' -'develop_' -'credit_' -'bank_' -'York_' -'Vor_' -'Today_' -'Preis_' -'Außerdem_' -'-, _' -'ta_' -'successful_' -'red_' -'protect_' -'president_' -'places_' -'largest_' -'implementation_' -'heit_' -'friendly_' -'double_' -'decades_' -'darf_' -'challenges_' -'Von_' -'Tatsächlich_' -'S' -'O' -'My_' -'Aufgabe_' -'Am_' -'ure_' -'sub_' -'stark_' -'soon_' -'rich_' -'pressure_' -'option_' -'neu_' -'jedes_' -'ien_' -'glaube_' -'events_' -'established_' -'despite_' -'comfortable_' -'cause_' -'built_' -'board_' -'benefit_' -'although_' -'Wahl_' -'Verfahren_' -'Regionen_' -'May_' -'Mal_' -'K_' -'Ihren_' -'Enterprise_' -'Britain_' -'Behörden_' -'Ausschuss_' -'Amerika_' -'ändern_' -'verschiedene_' -'takes_' -'strategic_' -'steps_' -'status_' -'len_' -'labor_' -'guarantee_' -'gehört_' -'einzelnen_' -'designed_' -'ces_' -'card_' -'cal_' -'behind_' -'agenda_' -'Website_' -'Verbindung_' -'Russian_' -'Rechte_' -'Presidency_' -'Initiative_' -'F' -'Court_' -'City_' -'Abstimmung_' -'�_' -'wichtigsten_' -'walk_' -'video_' -'type_' -'terrorism_' -'stop_' -'standard_' -'schließlich_' -'risks_' -'regime_' -'post_' -'nya_' -'nor_' -'internet_' -'gehören_' -'ermöglichen_' -'bisher_' -'beginning_' -'became_' -'Zum_' -'Verfassung_' -'Uhr_' -'Spanien_' -'Platz_' -'First_' -'Fed_' -'D' -'Bemühungen_' -'Armut_' -'. - (_' -'zeit_' -'zeigen_' -'wohl_' -'visit_' -'verbunden_' -'threat_' -'thought_' -'receive_' -'reach_' -'probably_' -'practice_' -'official_' -'nice_' -'mal_' -'lower_' -'looking_' -'ler_' -'gemeinsam_' -'gar_' -'findet_' -'fen_' -'complex_' -'committee_' -'closed_' -'angesichts_' -'ability_' -'Zahl_' -'Yet_' -'Vorschläge_' -'Version_' -'Verhandlungen_' -'Politiker_' -'More_' -'Mehrheit_' -'How_' -'Führung_' -'Eurozone_' -'Ergebnisse_' -'Bar_' -'B' -'Ansatz_' -'70_' -'200_' -'ze_' -'wobei_' -'study_' -'started_' -'rund_' -'reality_' -'purpose_' -'programs_' -'plans_' -'music_' -'monetary_' -'limited_' -'konnte_' -'ke_' -'isch_' -'highly_' -'guests_' -'falls_' -'enable_' -'confidence_' -'bad_' -'according_' -'accept_' -'V' -'Reformen_' -'Prozess_' -'Nationen_' -'NATO_' -'Kunden_' -'K' -'Indien_' -'Handel_' -'From_' -'Druck_' -'Dabei_' -'Antwort_' -'..." _' -' / _' -'vier_' -'sustainable_' -'style_' -'shown_' -'raise_' -'previous_' -'matters_' -'lives_' -'ken_' -'industrial_' -'helfen_' -'creating_' -'context_' -'consumers_' -'consequences_' -'con_' -'basic_' -'answer_' -'Prozent_' -'Only_' -'June_' -'English_' -'Development_' -', “_' -'zehn_' -'werde_' -'unique_' -'ton_' -'setting_' -'seines_' -'presented_' -'ors_' -'lost_' -'konnten_' -'knowledge_' -'ihres_' -'gegeben_' -'gebracht_' -'gas_' -'erforderlich_' -'effort_' -'creation_' -'cht_' -'choose_' -'caused_' -'categories_' -'bus_' -'beyond_' -'asked_' -'active_' -'Wort_' -'Seit_' -'Punkt_' -'Now_' -'Gruppe_' -'Entscheidungen_' -'Berichterstatter_' -'Artikel_' -'Arab_' -'öffentliche_' -'zen_' -'zeigt_' -'ties_' -'seem_' -'saying_' -'politicians_' -'partner_' -'note_' -'nahe_' -'latest_' -'ks_' -'hohe_' -'guten_' -'gewährleisten_' -'gesamten_' -'finance_' -'failure_' -'evidence_' -'entwickeln_' -'enthält_' -'darum_' -'dadurch_' -'challenge_' -'alone_' -'act_' -'Spiel_' -'Putin_' -'P' -'Hinblick_' -'General_' -'Gelegenheit_' -'Gefahr_' -'Gebiet_' -'Förderung_' -'Europeans_' -'Darüber_' -'Dank_' -'Damit_' -'Beginn_' -'Barcelona_' -'August_' -'Abkommen_' -'Öffentlichkeit_' -'verwenden_' -'unemployment_' -'treatment_' -'source_' -'sound_' -'sometimes_' -'solutions_' -'quickly_' -'programmes_' -'please_' -'objective_' -'lines_' -'larger_' -'ker_' -'guest_' -'damage_' -'build_' -'aware_' -'average_' -'aktuellen_' -'agricultural_' -'achieved_' -'University_' -'St_' -'Schlusselwortern_' -'Regeln_' -'Produkte_' -'Middle_' -'March_' -'H' -'Datei_' -'CD_' -'Bildung_' -'500_' -'1990_' -' " _' -'“._' -'äußerst_' -'zone_' -'ves_' -'v' -'throughout_' -'t' -'ssen_' -'request_' -'politics_' -'movement_' -'mentioned_' -'leben_' -'jede_' -'independent_' -'gleichen_' -'gleich_' -'ganzen_' -'fragen_' -'fest_' -'fair_' -'failed_' -'ermöglicht_' -'equal_' -'enlargement_' -'distribution_' -'direction_' -'ding_' -'danken_' -'coming_' -'choice_' -'cally_' -'Terrorismus_' -'Palestinian_' -'Minuten_' -'IMF_' -'Herren_' -'Funktion_' -'Anfang_' -'Abgeordneten_' -'zed_' -'völlig_' -'verstehen_' -'test_' -'supported_' -'shows_' -'setzt_' -'recht_' -'procedure_' -'principles_' -'lt_' -'lose_' -'ini_' -'includes_' -'ht_' -'hold_' -'gestellt_' -'gemeinsamen_' -'final_' -'fear_' -'e' -'domestic_' -'deficit_' -'consumer_' -'cher_' -'charge_' -'book_' -'base_' -'anything_' -'akan_' -'advanced_' -'X' -'W' -'Since_' -'Ressourcen_' -'Notwendigkeit_' -'Natürlich_' -'Kraft_' -'Korea_' -'Kontrolle_' -'Israeli_' -'Hand_' -'Fortschritte_' -'Erweiterung_' -'Debian_' -'Ausdruck_' -'Aufmerksamkeit_' -'übernachten_' -'x' -'web_' -'verfügen_' -'submitted_' -'speed_' -'reached_' -'produce_' -'perfect_' -'objectives_' -'mind_' -'ments_' -'initiative_' -'i' -'hoffe_' -'ground_' -'goods_' -'giving_' -'famous_' -'fallen_' -'entwickelt_' -'don_' -'considered_' -'class_' -'ck' -'cities_' -'bekommen_' -'additional_' -'accommodation_' -'Y_' -'Wann_' -'Viele_' -'Tage_' -'Security_' -'Rights_' -'Many_' -'Lisbon_' -'Folgen_' -'Federal_' -'E' -'Damen_' -'Blick_' -'Bild_' -'Bekämpfung_' -'Ausgaben_' -'Anwendung_' -'Angesichts_' -'Americans_' -'90_' -'45_' -'27_' -'22_' -'%._' -' % _' -'zudem_' -'wrong_' -'worked_' -'weder_' -'ut_' -'untuk_' -'tell_' -'später_' -'speak_' -'situated_' -'richtig_' -'restaurants_' -'res_' -'produced_' -'p' -'news_' -'m' -'ige_' -'häufig_' -'größte_' -'globale_' -'est_' -'enthalten_' -'emissions_' -'decided_' -'death_' -'completely_' -'brought_' -'au_' -'annual_' -'added_' -'Veränderungen_' -'Umwelt_' -'Services_' -'Schaffung_' -'Reihe_' -'Reform_' -'Instead_' -'Here_' -'Gesamt' -'Fund_' -'Finally_' -'Einfluss_' -'Durch_' -'December_' -'Dazu_' -'1791_' -'Änderungsantrag_' -'zahlen_' -'weapons_' -'voted_' -'technologies_' -'target_' -'secure_' -'requirements_' -'partners_' -'package_' -'options_' -'massive_' -'ism_' -'increasing_' -'goal_' -'files_' -'extent_' -'erung_' -'erster_' -'eigene_' -'contact_' -'consumption_' -'ber_' -'allows_' -'aim_' -'agreements_' -'Zentrum_' -'Text_' -'Schließlich_' -'Qualität_' -'Mitglied_' -'L' -'Kosovo_' -'Its_' -'Frieden_' -'During_' -'Chance_' -'300_' -'zweite_' -'won_' -'trotz_' -'tions_' -'technical_' -'students_' -'send_' -'prepared_' -'original_' -'mobile_' -'mail_' -'item_' -'function_' -'front_' -'f' -'extra_' -'entire_' -'election_' -'eben_' -'dialogue_' -'critical_' -'changed_' -'ang_' -'allowed_' -'Wettbewerb_' -'Verwendung_' -'So' -'Nutzung_' -'Nations_' -'Märkte_' -'Kultur_' -'Jahrhundert_' -'Italien_' -'Gästebewertungen_' -'Furthermore_' -'Erklärung_' -'Daher_' -'DE_' -'Beitrag_' -': „_' -'28_' -'1781_' -'zweiten_' -'wenige_' -'website_' -'wealth_' -'voll_' -'versuchen_' -'team_' -'supply_' -'stärker_' -'sorgen_' -'solidarity_' -'scale_' -'ring_' -'providing_' -'players_' -'paid_' -'opposition_' -'ling_' -'lang_' -'kam_' -'influence_' -'ier_' -'geworden_' -'genug_' -'gain_' -'ft_' -'forms_' -'follow_' -'erte_' -'erklären_' -'einschließlich_' -'distance_' -'concern_' -'concept_' -'carried_' -'campaign_' -'borders_' -'began_' -'ate_' -'aspects_' -'allein_' -'[_' -'WTO_' -'Server_' -'Programme_' -'Meer_' -'July_' -'Forschung_' -'Fehler_' -'Familie_' -'Ausschusses_' -'Although_' -'African_' -'? _' -': ' -'story_' -'stage_' -'server_' -'officials_' -'office_' -'offered_' -'nis_' -'legen_' -'leave_' -'jene_' -'insgesamt_' -'immigration_' -'hinter_' -'genannten_' -'fördern_' -'ful_' -'erwartet_' -'erwarten_' -'erneut_' -'doubt_' -'digital_' -'dari_' -'concerning_' -'bitte_' -'bevor_' -'apartment_' -'anderer_' -'Verbraucher_' -'Unsere_' -'Portugal_' -'Person_' -'Pakistan_' -'Organisation_' -'Opfer_' -'Ko' -'Idee_' -'H_' -'Griechenland_' -'Gesundheit_' -'EN_' -'DVD_' -'C' -' [_' -' ) _' -'zumindest_' -'z' -'writing_' -'worldwide_' -'verbessern_' -'uses_' -'users_' -'treffen_' -'tidak_' -'sed_' -'search_' -'save_' -'reports_' -'quiet_' -'professional_' -'privaten_' -'parking_' -'month_' -'map_' -'kosten_' -'jedem_' -'historical_' -'head_' -'gt_' -'generation_' -'funding_' -'einzige_' -'disease_' -'d' -'construction_' -'connection_' -'committed_' -'code_' -'child_' -'airport_' -'Werte_' -'Wasser_' -'Vergangenheit_' -'Unter_' -'Themen_' -'Stunden_' -'Prime_' -'Obwohl_' -'Most_' -'Mediterranean_' -'Linux_' -'Le_' -'Italian_' -'Information_' -'Herausforderung_' -'Flughafen_' -'Dialog_' -'Anti' -'Afrika_' -'übernehmen_' -'Änderungsanträge_' -'ying_' -'usually_' -'tic_' -'seek_' -'practical_' -'nimmt_' -'mus_' -'ms_' -'morning_' -'meinen_' -'material_' -'links_' -'kleine_' -'ja' -'implemented_' -'hoch_' -'helpful_' -'glauben_' -'getan_' -'geschaffen_' -'fishing_' -'erklärt_' -'effectively_' -'dollar_' -'deutschen_' -'demokratischen_' -'demands_' -'decline_' -'communication_' -'ch' -'benötigen_' -'applied_' -'angenommen_' -'amerikanische_' -'alternative_' -'Zeitpunkt_' -'Wahlen_' -'Tat_' -'Stelle_' -'Room_' -'Risiken_' -'People_' -'Parteien_' -'Lösungen_' -'Let_' -'Industrie_' -'Ihrem_' -'Hamas_' -'Fällen_' -'Frühstück_' -'Erstens_' -'Einkommen_' -'Dinge_' -'Dezember_' -'Center_' -'Austria_' -'Affairs_' -': "_' -'   ._' -'   . _' -'zunehmend_' -'zentrale_' -'works_' -'warum_' -'wants_' -'wanted_' -'vermeiden_' -'ver_' -'statement_' -'served_' -'series_' -'safe_' -'relationship_' -'provisions_' -'police_' -'neuer_' -'neben_' -'nce_' -'leadership_' -'leader_' -'join_' -'illegal_' -'gewesen_' -'ess_' -'eigentlich_' -'cs_' -'cken_' -'businesses_' -'border_' -'avoid_' -'authority_' -'applications_' -'appears_' -'agreed_' -'actions_' -'Zudem_' -'Schwierigkeiten_' -'Republic_' -'Präsidentin_' -'Please_' -'Juni_' -'J' -'IT_' -'Höhe_' -'Heute_' -'Greek_' -'Google_' -'George_' -'Egypt_' -'Economic_' -'Convention_' -'Amerikas_' -'2013_' -'1998_' -'160_' -'.  _' -' "..._' -'tra' -'structural_' -'star_' -'stable_' -'speech_' -'somit_' -'solchen_' -'schützen_' -'regards_' -'received_' -'read_' -'property_' -'powerful_' -'politischer_' -'path_' -'overall_' -'nearly_' -'n' -'method_' -'meinem_' -'lle_' -'legislative_' -'ine_' -'igkeit_' -'ideas_' -'getting_' -'folgt_' -'everyone_' -'establish_' -'ell_' -'drive_' -'cut_' -'competitive_' -'compared_' -'chinesische_' -'bit_' -'beitragen_' -'bare_' -'b' -'ations_' -'Zeiten_' -'Tagen_' -'Such_' -'Station_' -'Sozial' -'R' -'Position_' -'Nachfrage_' -'Management_' -'Latin_' -'Kingdom_' -'Integration_' -'Herzen_' -'Globalisierung_' -'Financial_' -'Club_' -'Bestimmungen_' -'Aktivitäten_' -'31_' -'3' -''' _' -'с' -'Änderungen_' -'zahlreiche_' -'word_' -'variety_' -'union_' -'trading_' -'talk_' -'serve_' -'rising_' -'requires_' -'reducing_' -'reduced_' -'mehrere_' -'leicht_' -'jüngsten_' -'joint_' -'instruments_' -'immediately_' -'ierte_' -'hinsichtlich_' -'geschlossen_' -'folgen_' -'erhöhen_' -'ere_' -'equipped_' -'elsewhere_' -'efficient_' -'durchgeführt_' -'discussion_' -'difference_' -'developments_' -'comprehensive_' -'bringt_' -'bewusst_' -'beide_' -'attacks_' -'anders_' -'Vertrauen_' -'Revolution_' -'Plan_' -'PC_' -'Konferenz_' -'Japanese_' -'Ireland_' -'Great_' -'Centre_' -'CO2_' -'Bitte_' -'Anzahl_' -'.' -'wichtiger_' -'weltweiten_' -'types_' -'train_' -'tools_' -'thousands_' -'suggest_' -'stock_' -'sectors_' -'school_' -'sagte_' -'representatives_' -'reichen_' -'promote_' -'productivity_' -'priority_' -'possibility_' -'park_' -'nationale_' -'mit' -'message_' -'medical_' -'las_' -'instrument_' -'initiatives_' -'ierten_' -'ial_' -'genommen_' -'ga_' -'frei_' -'farmers_' -'expected_' -'elements_' -'elected_' -'easily_' -'degree_' -'deficits_' -'chance_' -'bestehen_' -'ausgestattet_' -'attack_' -'ated_' -'affected_' -'Woche_' -'Web_' -'Vertrag_' -'Tu' -'Syria_' -'Stabilität_' -'Pro' -'Preise_' -'Policy_' -'Nicht_' -'Neu' -'März_' -'Microsoft_' -'Markt' -'Mail_' -'Lissabon_' -'Land' -'Jo' -'His_' -'Global_' -'Finanz' -'Energie_' -'Design_' -'Constitution_' -'Brazil_' -'Besuch_' -'Bereiche_' -'Bad_' -'A' -'. ' -'whom_' -'ums_' -'ue_' -'tät_' -'turned_' -'relativ_' -'refugees_' -'reduction_' -'played_' -'para_' -'nächste_' -'ning_' -'ni_' -'middle_' -'mein_' -'letzte_' -'leider_' -'kaum_' -'k' -'ismus_' -'institutional_' -'forced_' -'expect_' -'erfolgreich_' -'enter_' -'diejenigen_' -'crucial_' -'commercial_' -'circumstances_' -'carry_' -'becoming_' -'bald_' -'aufgenommen_' -'activity_' -'Why_' -'Vereinten_' -'Verbesserung_' -'Technologie_' -'Te' -'Systems_' -'Standards_' -'Site_' -'Personal_' -'Osten_' -'Oktober_' -'October_' -'Not_' -'Küche_' -'Just_' -'Infrastruktur_' -'High_' -'Guest_' -'Grand_' -'Freiheit_' -'Free_' -'Finanzierung_' -'Directive_' -'CA' -'Auffassung_' -'According_' -'35_' -'29_' -'1980_' -'., _' -'* _' -')' -'® _' -'y' -'went_' -'travel_' -'ten' -'task_' -'sieht_' -'ship_' -'review_' -'religious_' -'relevant_' -'record_' -'procedures_' -'precisely_' -'pleased_' -'paar_' -'minute_' -'minister_' -'mention_' -'maintain_' -'leisten_' -'jeweiligen_' -'island_' -'investors_' -'improving_' -'hour_' -'hotels_' -'h' -'größere_' -'gesamte_' -'gekommen_' -'firms_' -'ence_' -'dringend_' -'dangerous_' -'conference_' -'colleagues_' -'c' -'break_' -'betrachtet_' -'bereich_' -'apply_' -'ance_' -'akzeptieren_' -'Währung_' -'Waffen_' -'Umgebung_' -'Trade_' -'Therefore_' -'Star_' -'Sicht_' -'N' -'IWF_' -'England_' -'Einführung_' -'Do_' -'Conference_' -'Co' -'Auswahl_' -'Asien_' -'Arbeitnehmer_' -'4' -', ' -'). _' -'zusätzliche_' -'ze' -'written_' -'white_' -'weise_' -'walking_' -'unbedingt_' -'trust_' -'tor_' -'tabled_' -'sts_' -'sse_' -'sign_' -'schaft_' -'sa_' -'s' -'round_' -'reserves_' -'regulations_' -'raised_' -'presence_' -'ped_' -'organisation_' -'neither_' -'namely_' -'mag_' -'länger_' -'ku' -'justice_' -'holiday_' -'historischen_' -'hands_' -'gives_' -'genießen_' -'ganze_' -'feature_' -'facing_' -'equipment_' -'draw_' -'documents_' -'denke_' -'deine_' -'boost_' -'banking_' -'attempt_' -'atmosphere_' -'assistance_' -'aimed_' -'agriculture_' -'advantage_' -'Verordnung_' -'Transparenz_' -'Tagesordnung_' -'Spanish_' -'Sorge_' -'Social_' -'Secondly_' -'Sea_' -'Rooms_' -'Robert_' -'Restaurants_' -'Nur_' -'Mai_' -'Linie_' -'Gründen_' -'Erfahrung_' -'Den_' -'Code_' -'Asian_' -'Also_' -'2' -'.)_' -'--_' -'weeks_' -'voting_' -'votes_' -'unterschiedlichen_' -'trying_' -'stets_' -'sm_' -'shift_' -'section_' -'sechs_' -'recovery_' -'programm_' -'pro' -'press_' -'pre_' -'phone_' -'ory_' -'oben_' -'networks_' -'ned_' -'nations_' -'nation_' -'modified_' -'merely_' -'membership_' -'meines_' -'lösen_' -'ley_' -'largely_' -'keiten_' -'kannst_' -'implement_' -'historic_' -'happen_' -'gs_' -'grounds_' -'goes_' -'gesetzt_' -'friends_' -'fort' -'floor_' -'eten_' -'establishment_' -'erkennen_' -'erfordert_' -'efficiency_' -'draft_' -'der' -'daily_' -'conclusion_' -'ches_' -'changing_' -'carbon_' -'buy_' -'burden_' -'bathroom_' -'assessment_' -'Wettbewerbsfähigkeit_' -'Schulden_' -'Rezession_' -'Regierungs' -'Raum_' -'Punkte_' -'Per' -'Museum_' -'Monaten_' -'Methode_' -'Jahrhunderts_' -'Islam_' -'Inter' -'Innovation_' -'Human_' -'Gruppen_' -'Groß' -'Europäer_' -'Diskussion_' -'Both_' -'Bedrohung_' -'Arbeitslosigkeit_' -'Al_' -'Airport_' -'8' -'2012_' -'Änderung_' -'zuletzt_' -'ys_' -'unten_' -'sta' -'sing_' -'ses_' -'scientific_' -'schwierig_' -'running_' -'rten_' -'regarding_' -'plus_' -'plant_' -'participation_' -'output_' -'normal_' -'nbsp_' -'mission_' -'lag_' -'ko' -'j_' -'ität_' -'innovation_' -'innen_' -'improved_' -'impossible_' -'hält_' -'hol' -'haus_' -'gezeigt_' -'ges_' -'gen' -'financing_' -'fe_' -'express_' -'export_' -'entsprechende_' -'ei_' -'deep_' -'decade_' -'contribution_' -'considerable_' -'competitiveness_' -'bodies_' -'bilden_' -'begin_' -'außerhalb_' -'Zweitens_' -'Zweifel_' -'Wirtschaftswachstum_' -'Wer_' -'Warum_' -'Vorteile_' -'Unterkategorien_' -'She_' -'Resort_' -'Republik_' -'Ra' -'Projekt_' -'Produktion_' -'Partner_' -'No' -'Mitte_' -'Lo' -'Investoren_' -'Forum_' -'Erfahrungen_' -'Energie' -'Einigung_' -'Du' -'Article_' -'Angebot_' -'... _' -'überzeugt_' -'ß_' -'zing_' -'za' -'vision_' -'versucht_' -'up' -'treten_' -'transparent_' -'told_' -'ti' -'spend_' -'speaking_' -'sites_' -'shopping_' -'sh_' -'screen_' -'says_' -'refer_' -'reading_' -'rd_' -'raum_' -'post' -'policymakers_' -'outcome_' -'operations_' -'operation_' -'opening_' -'ol' -'nach' -'multi_' -'mass_' -'manufacturing_' -'lies_' -'king_' -'ir_' -'intended_' -'insurance_' -'hu' -'hin' -'highest_' -'happened_' -'handeln_' -'gewinnen_' -'film_' -'families_' -'exports_' -'erfüllen_' -'ellen_' -'easier_' -'document_' -'derartige_' -'defense_' -'darstellt_' -'darstellen_' -'controls_' -'congratulate_' -'compromise_' -'clients_' -'braucht_' -'betrachten_' -'bestimmten_' -'bar' -'au' -'appear_' -'ans_' -'ale_' -'addressed_' -'Westen_' -'Welt' -'Wasser' -'Vielfalt_' -'Technologien_' -'Su' -'Street_' -'Spa_' -'RE' -'Public_' -'Privat' -'Poland_' -'Online_' -'Musik_' -'Kommissarin_' -'Kolleginnen_' -'John_' -'Inflation_' -'Handels' -'Folge_' -'Erholung_' -'ECB_' -'Da' -'Costa_' -'Because_' -'A5_' -'26_' -'2011_' -'überhaupt_' -'überall_' -'ßen_' -'zunächst_' -'ya' -'wählen_' -'wing_' -'wider_' -'waste_' -'vital_' -'victims_' -'useful_' -'urban_' -'theory_' -'structure_' -'ster_' -'staatlichen_' -'schwer_' -'saw_' -'sales_' -'relation_' -'rapidly_' -'profitieren_' -'primary_' -'presidency_' -'pre' -'powers_' -'planning_' -'offen_' -'numerous_' -'neues_' -'muß_' -'moral_' -'mainly_' -'lagen_' -'kurz_' -'ka_' -'investments_' -'inter' -'innovative_' -'heard_' -'gern_' -'generally_' -'gelegen_' -'fund_' -'freien_' -'finally_' -'establishing_' -'entspricht_' -'entscheiden_' -'eiten_' -'eindeutig_' -'details_' -'desire_' -'dengan_' -'core_' -'calls_' -'bestimmte_' -'bessere_' -'analysis_' -'amendment_' -'alt_' -'aktuelle_' -']], _' -'Zentralbank_' -'Vorschriften_' -'Volkswirtschaften_' -'Unser_' -'Tri' -'Time_' -'Team_' -'Strand_' -'Stimme_' -'Sinne_' -'Sicherheits' -'Sektor_' -'See' -'Schritte_' -'Reserve_' -'Re' -'Maße_' -'Juli_' -'Installation_' -'Herausforderungen_' -'Haushalts' -'Euro' -'En' -'Despite_' -'Dateien_' -'Congress_' -'Bundes' -'Brexit_' -'Brasilien_' -'Berichte_' -'Benutzer_' -'Arbeitsplätze_' -'Anteil_' -'Annahme_' -'Angelegenheiten_' -'Amsterdam_' -'Amendment_' -'worth_' -'worse_' -'vorgeschlagen_' -'transfer_' -'tool_' -'territory_' -'taxes_' -'steigen_' -'stated_' -'spirit_' -'spa' -'sent_' -'richtige_' -'release_' -'reference_' -'ping_' -'phase_' -'paper_' -'o' -'numbers_' -'moderne_' -'met_' -'menu_' -'me' -'linked_' -'limits_' -'learn_' -'interesting_' -'interested_' -'icht_' -'hoch' -'helped_' -'halte_' -'gesehen_' -'genuine_' -'gefunden_' -'g' -'freie_' -'fordern_' -'fixed_' -'fine_' -'fail_' -'extension_' -'examples_' -'erzielt_' -'enormous_' -'endlich_' -'encourage_' -'else_' -'eit_' -'dly_' -'district_' -'criteria_' -'continued_' -'consensus_' -'candidate_' -'buffet_' -'britischen_' -'books_' -'bonds_' -'bestimmt_' -'becomes_' -'baren_' -'apartments_' -'animals_' -'adopt_' -'accepted_' -'Worten_' -'Weltwirtschaft_' -'Wege_' -'Vor' -'Vertreter_' -'Urlaub_' -'Um' -'Turkish_' -'Ste' -'Status_' -'Selbst_' -'Sache_' -'Red_' -'RI' -'Nice_' -'NEW_' -'Muslim_' -'Meine_' -'Lebens' -'Last_' -'Konzept_' -'Januar_' -'Golf_' -'Gaza_' -'Einige_' -'Behandlung_' -'Alternative_' -'Agenda_' -'400_' -'0' -', „_' -'“_' -'у' -'Über' -'wo' -'willing_' -'weisen_' -'warm_' -'verbundenen_' -'understanding_' -'ul_' -'tried_' -'traffic_' -'tomorrow_' -'to' -'ter' -'television_' -'targets_' -'suggests_' -'sufficient_' -'stärken_' -'stronger_' -'spread_' -'signed_' -'shared_' -'separate_' -'seeking_' -'scope_' -'ro' -'respond_' -'released_' -'regionale_' -'ready_' -'putting_' -'published_' -'pass_' -'owing_' -'org_' -'modernen_' -'mittel_' -'minimum_' -'managed_' -'lu' -'log_' -'lernen_' -'kommenden_' -'kennen_' -'integrated_' -'improvement_' -'immediate_' -'identity_' -'hear_' -'green_' -'governance_' -'got_' -'games_' -'flight_' -'fellow_' -'exactly_' -'evening_' -'europäischer_' -'el' -'ehemaligen_' -'earlier_' -'difficulties_' -'damals_' -'dagegen_' -'cross_' -'crime_' -'comfort_' -'character_' -'camera_' -'box_' -'bezüglich_' -'beste_' -'behavior_' -'aten_' -'approved_' -'anti' -'acht_' -'abge' -'Zustimmung_' -'Za' -'Vielleicht_' -'USS_' -'Tra' -'Sterne_' -'Solidarität_' -'Sinn_' -'Risiko_' -'Regime_' -'Rechts' -'Prodi_' -'Nachbarn_' -'Monat_' -'Modell_' -'Ku' -'J_' -'In' -'III_' -'Her' -'Health_' -'Eindruck_' -'EZB_' -'Do' -'Clinton_' -'Business_' -'Bis_' -'Bilder_' -'Bau' -'Barack_' -'Au' -'+_' -'о' -'ät_' -'Überwachung_' -'www_' -'wirtschaftlicher_' -'wert_' -'welfare_' -'voters_' -'vo' -'verringern_' -'verpflichtet_' -'van_' -'unge' -'une_' -'unable_' -'umgesetzt_' -'ultimately_' -'summer_' -'street_' -'specifically_' -'sort_' -'sicherzustellen_' -'ser_' -'revolution_' -'resolve_' -'rer_' -'reflect_' -'quote_' -'ps_' -'protected_' -'port_' -'planet_' -'placed_' -'pada_' -'otherwise_' -'ones_' -'offering_' -'morgen_' -'millions_' -'mer' -'measure_' -'machine_' -'licher_' -'letztlich_' -'konzentrieren_' -'je' -'j' -'io_' -'host_' -'holding_' -'größeren_' -'greatest_' -'gleiche_' -'gelangen_' -'ga' -'führte_' -'faces_' -'expressed_' -'expensive_' -'era_' -'entweder_' -'entsprechend_' -'ensuring_' -'durch' -'download_' -'divided_' -'discussions_' -'discussed_' -'described_' -'dennoch_' -'deliver_' -'continues_' -'continent_' -'conclude_' -'comments_' -'color_' -'click_' -'broad_' -'bestand_' -'berg_' -'begrüße_' -'beginnen_' -'bedarf_' -'außerdem_' -'aus' -'arbeitet_' -'anderem_' -'alte_' -'accession_' -'abzu' -'Vorteil_' -'Video_' -'Versuch_' -'Trotz_' -'Tre' -'Standard_' -'Saudi_' -'Polen_' -'Pe' -'Or' -'Open_' -'Nachdem_' -'NA' -'Mo' -'Mit' -'Michael_' -'James_' -'Ist_' -'Haltung_' -'Gäste_' -'Gegenteil_' -'Entschließung_' -'Ent' -'El_' -'Bürgern_' -'Ben' -'Beitritt_' -'Arbeits' -'Anstieg_' -'75_' -'33_' -'32_' -'30' -'.._' -'%, _' -'” – _' -'ı' -'Ä' -'­_' -'window_' -'widely_' -'west_' -'wer_' -'vollständig_' -'veröffentlicht_' -'ve' -'ufen_' -'tte_' -'tradition_' -'thereby_' -'tan_' -'spent_' -'southern_' -'sources_' -'skills_' -'sanctions_' -'rural_' -'root_' -'reception_' -'profit_' -'priorities_' -'player_' -'partly_' -'oc' -'obligations_' -'nachdem_' -'militärische_' -'mar' -'läuft_' -'loss_' -'lo_' -'llen_' -'liberal_' -'ite_' -'it' -'industrie_' -'individuals_' -'höhere_' -'himself_' -'heutigen_' -'granted_' -'gi' -'format_' -'firm_' -'ff_' -'fand_' -'fahren_' -'expectations_' -'exclusive_' -'erlaubt_' -'entsprechenden_' -'eln_' -'einiger_' -'dürfte_' -'doesn_' -'detailed_' -'denken_' -'default_' -'cuts_' -'cover_' -'communities_' -'claim_' -'britische_' -'außer_' -'associated_' -'article_' -'ahead_' -'actual_' -'absolutely_' -'Wohn' -'Uni' -'UNO_' -'Trump_' -'Teilen_' -'Systeme_' -'Strategien_' -'Square_' -'Secretary_' -'Schulden' -'Regel_' -'Präsidentschaft_' -'Po' -'Männer_' -'Ma' -'Leute_' -'Kindern_' -'Kapital' -'Justice_' -'Ja' -'Islamic_' -'Homepage_' -'Geschäfts' -'Ger' -'Gebäude_' -'Frankfurt_' -'Firmen_' -'Erachtens_' -'Einrichtungen_' -'Ebenso_' -'Di' -'Christian_' -'Breakfast_' -'Bio' -'Ausland_' -'Argentina_' -'Ad' -'2014_' -'" (_' -' -, _' -'—_' -'ó' -'ä' -'Öl' -'wieder' -'whatever_' -'vorhanden_' -'ures_' -'unver' -'uner' -'ul' -'tt_' -'treated_' -'sun_' -'suchen_' -'stra' -'someone_' -'so' -'smaller_' -'slow_' -'sides_' -'seven_' -'ro_' -'represents_' -'relating_' -'regionalen_' -'rapid_' -'r' -'pour_' -'permanent_' -'pe_' -'payments_' -'parliamentary_' -'oren_' -'operating_' -'ons' -'nie' -'ne' -'möglichkeiten_' -'monitoring_' -'miteinander_' -'mark_' -'loans_' -'lo' -'listed_' -'link_' -'limit_' -'leads_' -'languages_' -'land' -'kt_' -'ju' -'ji' -'ization_' -'iten_' -'inzwischen_' -'introduced_' -'ians_' -'hören_' -'höher_' -'geschehen_' -'gel' -'garden_' -'funktioniert_' -'fuel_' -'französischen_' -'ff' -'expansion_' -'enti' -'discuss_' -'cutting_' -'corporate_' -'contemporary_' -'connected_' -'combination_' -'causes_' -'benutzt_' -'begann_' -'bear_' -'battle_' -'bars_' -'auszu' -'ating_' -'as' -'ana' -'Zur_' -'Zinsen_' -'Währungs' -'Volk_' -'U' -'Thus_' -'Their_' -'Standpunkt_' -'Se' -'Realität_' -'Prioritäten_' -'Nutzen_' -'Netherlands_' -'Natur_' -'MySQL_' -'Mer' -'Leistungen_' -'Krankheiten_' -'Klima' -'Klicken_' -'Insel_' -'Hoffnung_' -'Gre' -'God_' -'Gesellschaften_' -'Gegensatz_' -'Film_' -'Fe' -'Falle_' -'Eastern_' -'Dr_' -'Denn_' -'Democrats_' -'Car' -'Bewegung_' -'Best_' -'Augen_' -'Atmosphäre_' -'Abschluss_' -'2015_' -'. (_' -', "_' -'„_' -'über' -'äu' -'  _' -'wrote_' -'write_' -'wodurch_' -'win_' -'wesentlich_' -'weg_' -'verlieren_' -'verfahren_' -'va' -'unto_' -'unless_' -'ungefähr_' -'unacceptable_' -'umfassende_' -'turning_' -'trägt_' -'truth_' -'transparency_' -'transition_' -'traditionellen_' -'thinking_' -'thanks_' -'terrorist_' -'technological_' -'talks_' -'swimming_' -'suffer_' -'strategies_' -'stimmen_' -'stellte_' -'starke_' -'st' -'schließen_' -'russischen_' -'resulting_' -'represent_' -'relate_' -'regulatory_' -'ran_' -'punkt_' -'presidential_' -'picture_' -'oni' -'ok_' -'offiziellen_' -'offensichtlich_' -'ns' -'nis' -'nachhaltige_' -'models_' -'migration_' -'mid_' -'meaning_' -'maßnahmen_' -'materials_' -'lebih_' -'laut_' -'laid_' -'komplett_' -'ing' -'industries_' -'ical_' -'ha' -'großes_' -'gleich' -'gezwungen_' -'getroffen_' -'geo' -'functions_' -'followed_' -'folgenden_' -'figures_' -'faced_' -'fa' -'ey_' -'extensive_' -'eu' -'erhöht_' -'equally_' -'enz' -'entstehen_' -'ele' -'einander_' -'directives_' -'determined_' -'ded_' -'debates_' -'deaths_' -'daten_' -'contain_' -'closer_' -'cheap_' -'che' -'besondere_' -'berücksichtigt_' -'behandelt_' -'auf' -'arms_' -'arbeit_' -'announced_' -'Wochen_' -'Vergleich_' -'Unabhängigkeit_' -'Un' -'Umständen_' -'Two_' -'Stimmen_' -'Steuer' -'Sta' -'Spieler_' -'Shi' -'Second_' -'SA' -'Royal_' -'Pri' -'Op' -'Old_' -'Ohne_' -'Nun_' -'Nachfolgend_' -'Mitteilung_' -'Me' -'Lu' -'Le' -'Kritik_' -'Ke' -'Is_' -'Instrument_' -'ID_' -'Home_' -'Hinsicht_' -'Haupt' -'Gra' -'Gold_' -'Go' -'Given_' -'Gesetz_' -'Forschungs' -'Engagement_' -'Einrichtung_' -'EADS_' -'Dennoch_' -'Bildungs' -'Beschäftigung_' -'Beschreibung_' -'Apartments_' -'Amt_' -'Alliance_' -'Air_' -'AIDS_' -'9' -'5' -'1997_' -'1996_' -'1995_' -'150_' -'01_' -'и' -'е' -'á_' -'Über_' -'zimmer_' -'wollte_' -'weltweite_' -'vertreten_' -'verloren_' -'unseres_' -'unlikely_' -'track_' -'tischen_' -'the' -'supporting_' -'suffering_' -'sub' -'stress_' -'strengthen_' -'starting_' -'stands_' -'standing_' -'signal_' -'selbstverständlich_' -'sea' -'saving_' -'rt' -'ries_' -'restrictions_' -'radical_' -'proper_' -'politisch_' -'piece_' -'physical_' -'persönlichen_' -'perspective_' -'per' -'olitik_' -'older_' -'moving_' -'mis' -'min_' -'medium_' -'manage_' -'maintained_' -'laws_' -'keits' -'ked_' -'kaufen_' -'jährlich_' -'ischer_' -'introduction_' -'introduce_' -'inside_' -'independence_' -'increases_' -'imports_' -'ik_' -'humanitarian_' -'housing_' -'historische_' -'guidelines_' -'gs' -'gold_' -'gerecht_' -'gave_' -'gar' -'fo' -'flexible_' -'fire_' -'fields_' -'falsch_' -'expression_' -'exist_' -'except_' -'eventually_' -'euch_' -'erten_' -'ep' -'entry_' -'employees_' -'emphasis_' -'eingeführt_' -'ee_' -'duty_' -'dir_' -'dia' -'delegation_' -'criminal_' -'collapse_' -'coffee_' -'claims_' -'chi' -'chaft_' -'cat' -'carefully_' -'car' -'bottom_' -'bestehenden_' -'begrüßen_' -'barkeit_' -'ausge' -'armed_' -'anyone_' -'angeht_' -'ah_' -'ad_' -'Wissenschaft_' -'Wein' -'Verb' -'Ungleichheit_' -'Teile_' -'THE_' -'Stärkung_' -'Staats' -'Staates_' -'Sehr_' -'Sehenswürdigkeiten_' -'Rotary_' -'Ro' -'Reaktion_' -'Produktions' -'Ph' -'Ne' -'Name_' -'Na' -'NI_' -'Mi' -'La' -'Kunst_' -'Kompromiss_' -'Ka' -'Israelis_' -'Irish_' -'Initiativen_' -'IS' -'Hintergrund_' -'Forderung_' -'Ereignisse_' -'Copenhagen_' -'Chi' -'Can_' -'Bur' -'Binnenmarkt_' -'Beispiele_' -'Bau_' -'Basis_' -'Barroso_' -'Bar' -'Aufbau_' -'Aspekt_' -'Anfrage_' -'36_' -'14' -''._' -' = _' -' ... _' -'ö' -'ée_' -'zusätzlich_' -'zation_' -'welches_' -'weiterer_' -'victory_' -'vergessen_' -'ver' -'unbe' -'ub' -'truly_' -'teilweise_' -'tar' -'suffered_' -'struggle_' -'ski_' -'shops_' -'seriously_' -'selected_' -'riesigen_' -'resort_' -'remember_' -'pursue_' -'purchase_' -'playing_' -'phrase_' -'ourselves_' -'ort_' -'oral_' -'on' -'no' -'möglichen_' -'ming_' -'mehreren_' -'lt' -'looks_' -'lichkeit_' -'leistungen_' -'ld_' -'ld' -'launch_' -'laufen_' -'lan' -'laden_' -'kürzlich_' -'kle' -'jo' -'ium_' -'ish_' -'ire_' -'intervention_' -'implementing_' -'he' -'hauptsächlich_' -'happy_' -'grundlegende_' -'geändert_' -'gerne_' -'fähigkeit_' -'fresh_' -'flexibility_' -'fish_' -'erinnern_' -'erhält_' -'equivalent_' -'enterprises_' -'ene_' -'email_' -'dynamic_' -'diplomatic_' -'declaration_' -'database_' -'counter_' -'contrast_' -'conflicts_' -'completed_' -'combat_' -'collective_' -'calling_' -'ber' -'benutzen_' -'automatisch_' -'asylum_' -'asset_' -'anstatt_' -'animal_' -'angezeigt_' -'ach_' -'Zi' -'Y' -'Volks' -'Vo' -'Vereinbarung_' -'Verbindungen_' -'Unter' -'Soviet_' -'Sorgen_' -'Ri' -'Projekte_' -'Pro_' -'Private_' -'Post' -'Pi' -'Organisationen_' -'Mitarbeiter_' -'Krieges_' -'Korruption_' -'Investitions' -'Institute_' -'Informations' -'IP_' -'He' -'Haushalt_' -'Gesetze_' -'Front_' -'Foundation_' -'Fortschritt_' -'Fort' -'February_' -'Familien' -'Entwicklungsländern_' -'Dra' -'Computer_' -'Ca' -'CAMBRIDGE_' -'Bre' -'Board_' -'Bo' -'Beziehung_' -'Aufgrund_' -'Another_' -'48_' -'13' -' /_' -'”, _' -'т' -'р' -'és_' -'är' -'Ökonomen_' -'zufolge_' -'zahlreichen_' -'warming_' -'wa' -'verlangen_' -'ven_' -'under' -'tz_' -'tu' -'translation_' -'tests_' -'terrace_' -'tasks_' -'ständig_' -'stations_' -'starken_' -'staatliche_' -'spacious_' -'sofort_' -'sin' -'siehe_' -'shower_' -'selection_' -'seemed_' -'science_' -'rn' -'ri_' -'responsibilities_' -'relax_' -'relatively_' -'prosperity_' -'promoting_' -'por_' -'platz_' -'partnership_' -'parliament_' -'opened_' -'ongoing_' -'obvious_' -'nf' -'nennen_' -'methods_' -'meetings_' -'mechanism_' -'les' -'langen_' -'labour_' -'ise_' -'internationaler_' -'installation_' -'ina' -'ill_' -'ierungs' -'ier' -'ide_' -'ice_' -'houses_' -'ha_' -'gutes_' -'größer_' -'goals_' -'gemäß_' -'gegenwärtig_' -'französische_' -'faster_' -'erzielen_' -'ergreifen_' -'erfolgen_' -'entwickelten_' -'entirely_' -'entered_' -'eingesetzt_' -'economists_' -'du' -'driving_' -'dollars_' -'display_' -'defined_' -'darunter_' -'danger_' -'danach_' -'dalam_' -'crimes_' -'corruption_' -'contract_' -'constitution_' -'charged_' -'cer' -'cancer_' -'bu' -'bre' -'bly_' -'biggest_' -'beruht_' -'benötigt_' -'believed_' -'beds_' -'ausschließlich_' -'assets_' -'ans' -'agen_' -'advance_' -'administrative_' -'ade' -'achten_' -'accordance_' -'a' -']] [[_' -'\\_' -'Zunächst_' -'Your_' -'Würde_' -'Wissen_' -'Waren_' -'Vietnam_' -'Verpflichtungen_' -'Verpflichtung_' -'Verhältnis_' -'Verfassungs' -'Unterschied_' -'Unfortunately_' -'TA' -'Syrien_' -'Straße_' -'San' -'SI' -'SE' -'Ru' -'Q_' -'Pre' -'Pacific_' -'Neben_' -'Mor' -'Monetary_' -'Miss' -'Mexiko_' -'Mexico_' -'Men' -'Medien_' -'Mal' -'Live_' -'Landwirtschaft_' -'Königreich_' -'Kultur' -'Kopf_' -'Je' -'Irland_' -'Internationalen_' -'Hotel' -'Hong_' -'Hoch' -'Hause_' -'Han' -'HIV_' -'HA' -'Geld' -'Formen_' -'Fahr' -'Every_' -'Einklang_' -'EC_' -'Dar' -'DI' -'Click_' -'Cha' -'Ce' -'Bus' -'Bra' -'Bi' -'Atom' -'Arten_' -'Angriff_' -'Abend_' -'95_' -'64_' -'1989_' -'', _' -'”._' -'ст' -'а' -'ür' -'ß' -'zuvor_' -'wi' -'weg' -'wages_' -'w' -'verfolgen_' -'umzusetzen_' -'trend_' -'tre' -'tly_' -'ti_' -'tel_' -'teilen_' -'summit_' -'significantly_' -'sets_' -'sektor_' -'scha' -'sagt_' -'sa' -'rr' -'rin_' -'reservation_' -'reported_' -'rely_' -'rejected_' -'recognize_' -'rechts' -'rasch_' -'qua' -'prime_' -'pri' -'plants_' -'pictures_' -'persons_' -'peaceful_' -'par' -'ou_' -'ou' -'opposite_' -'op' -'obviously_' -'nu' -'north_' -'ni' -'ng' -'nes_' -'nder_' -'nationaler_' -'nar' -'named_' -'moved_' -'mm_' -'mer_' -'manchmal_' -'machte_' -'ll' -'lessons_' -'learning_' -'krise_' -'ki_' -'initial_' -'igung_' -'iger_' -'ied_' -'hn_' -'helping_' -'hei' -'guaranteed_' -'gesprochen_' -'gender_' -'genannt_' -'gelten_' -'geleistet_' -'formal_' -'fisheries_' -'finanziellen_' -'finanzielle_' -'figure_' -'fat' -'extended_' -'extend_' -'explain_' -'experts_' -'enen_' -'dy_' -'durchaus_' -'drug_' -'dra' -'do' -'diseases_' -'deutsche_' -'cuisine_' -'courses_' -'couple_' -'cor' -'contrary_' -'constitutional_' -'commitments_' -'charges_' -'cast_' -'capable_' -'candidates_' -'bound_' -'beachten_' -'ban_' -'balanced_' -'außerordentlich_' -'argue_' -'appeal_' -'anzu' -'ang' -'allowing_' -'alliance_' -'allgemeine_' -'ages_' -'absolute_' -'abhängig_' -'Wähler_' -'Wegen_' -'Verhalten_' -'Umwelt' -'Transport_' -'Tradition_' -'Städte_' -'Stadt' -'Sol' -'Si' -'Schäden_' -'Schule_' -'SS_' -'Rezeption_' -'Report_' -'Q' -'Perhaps_' -'Paul_' -'Pa' -'PHP_' -'Niveau_' -'Ni' -'Nahen_' -'NE' -'Mu' -'Mitgliedschaft_' -'Militär' -'Merkel_' -'Mat' -'Located_' -'Lin' -'Leistung_' -'Las_' -'LI' -'Kong_' -'Klein' -'Kern' -'Jetzt_' -'Instrumente_' -'Hälfte_' -'Generation_' -'Gegen' -'Flug' -'Finanzkrise_' -'Far' -'Familien_' -'Erde_' -'Du_' -'Daten' -'Chancen_' -'Cameron_' -'Berichts_' -'BMW_' -'Auto' -'Ausweitung_' -'Ausbildung_' -'Aufenthalt_' -'Anstrengungen_' -'Anforderungen_' -'Am' -'Altstadt_' -'AR' -'25' -'15' -'11' -' — _' -' –, _' -'ы' -'é' -'è' -'Öl_' -'Ägypten_' -'yourself_' -'wine_' -'wiederum_' -'wenigen_' -'welt' -'weight_' -'vulnerable_' -'voice_' -'verlassen_' -'verantwortlich_' -'vast_' -'urgent_' -'ual_' -'te' -'tan' -'tahun_' -'supports_' -'studies_' -'structures_' -'ss' -'species_' -'south_' -'solve_' -'smoking_' -'sitting_' -'sion_' -'sierung_' -'ships_' -'sharing_' -'severe_' -'session_' -'select_' -'seien_' -'season_' -'schlagen_' -'remove_' -'relative_' -'recommend_' -'recession_' -'reaching_' -'race_' -'pu' -'provision_' -'proved_' -'prospects_' -'promotion_' -'promise_' -'practices_' -'positions_' -'photos_' -'photo_' -'pension_' -'owned_' -'out' -'organisations_' -'nisse_' -'names_' -'nahme_' -'mutual_' -'mountain_' -'minority_' -'micro' -'memory_' -'love_' -'ln_' -'lesen_' -'langfristige_' -'la' -'kitchen_' -'ker' -'kar' -'ir' -'improvements_' -'images_' -'hy' -'hundreds_' -'honourable_' -'gre' -'garantiert_' -'führenden_' -'fällt_' -'funktionieren_' -'frühen_' -'founded_' -'fighting_' -'felt_' -'eye_' -'exists_' -'exercise_' -'ethnic_' -'essentially_' -'equality_' -'entschieden_' -'entscheidender_' -'enk' -'elegant_' -'einzigen_' -'einge' -'dritten_' -'dinner_' -'defend_' -'defence_' -'currencies_' -'criticism_' -'crises_' -'compatible_' -'closely_' -'cha' -'budgetary_' -'bt_' -'ble' -'berücksichtigen_' -'automatically_' -'austerity_' -'arrangements_' -'arabischen_' -'anderes_' -'and' -'amp' -'ag_' -'Wunsch_' -'Worte_' -'West' -'Washington_' -'Wahl' -'Wachstums' -'Voraussetzungen_' -'Vielzahl_' -'Verwaltungs' -'Untersuchung_' -'Treffen_' -'Times_' -'Teil' -'Süd' -'Sweden_' -'Steuern_' -'Stand_' -'Sprache_' -'Sohn_' -'Serbia_' -'See_' -'Schlüssel' -'Safety_' -'Rome_' -'Regulation_' -'Rechtsvorschriften_' -'Phase_' -'PT_' -'Ost' -'NT' -'Monate_' -'Mindest' -'MO' -'Liberalisierung_' -'Krisen' -'Kontakt_' -'Kind_' -'Kar' -'Ju' -'Jean_' -'Iraqi_' -'Inseln_' -'Größe_' -'Grund' -'Grenze_' -'Gleichzeitig_' -'Gi' -'Gesundheits' -'Friedens' -'Erhöhung_' -'El' -'Ei' -'Durchführung_' -'Drittens_' -'De_' -'David_' -'Dann_' -'Buch_' -'Bel' -'Bank' -'Bal' -'Bahn_' -'Aspekte_' -'Anwendungen_' -'Anspruch_' -'Angst_' -'Anerkennung_' -'Al' -'Agreement_' -'65_' -'39_' -'з' -'überprüfen_' -'ón_' -'í' -'á' -'   . – _' -'za_' -'wor' -'wirksam_' -'weak_' -'war' -'wagen_' -'vieler_' -'verstanden_' -'usw_' -'us' -'universal_' -'unabhängig_' -'uf' -'tz' -'turns_' -'tung_' -'tourist_' -'touch_' -'tive_' -'tis' -'testing_' -'surrounding_' -'sugar_' -'subsidies_' -'stre' -'sti' -'ssi' -'sonst_' -'soft_' -'societies_' -'serves_' -'ser' -'schreiben_' -'schneller_' -'scheme_' -'russische_' -'run' -'rit' -'richten_' -'representative_' -'remained_' -'reich_' -'recognise_' -'rechts_' -'ra' -'prüfen_' -'productive_' -'pra' -'po' -'pick_' -'peoples_' -'pen' -'payment_' -'participate_' -'parliaments_' -'parents_' -'over' -'origin_' -'organizations_' -'onen_' -'ogen_' -'occur_' -'musste_' -'mu' -'motion_' -'mode_' -'mo' -'maximum_' -'manner_' -'mandate_' -'mal' -'lä' -'losses_' -'lokalen_' -'lity_' -'lesson_' -'legt_' -'leaving_' -'le' -'kurzem_' -'killed_' -'kept_' -'jener_' -'ized_' -'ionen_' -'institution_' -'ig' -'ici' -'ia' -'hostels_' -'hit_' -'gung_' -'griechischen_' -'gla' -'gestimmt_' -'gegenwärtigen_' -'focused_' -'father_' -'factors_' -'extreme_' -'expense_' -'expenditure_' -'exit_' -'enorme_' -'emergency_' -'element_' -'einfache_' -'ehen_' -'east_' -'drugs_' -'discovered_' -'differences_' -'destruction_' -'demokratische_' -'dank_' -'da' -'cu' -'coordination_' -'consideration_' -'confirmed_' -'command_' -'choices_' -'ce' -'buildings_' -'brings_' -'bility_' -'bi' -'bestätigen_' -'belief_' -'bath_' -'außen_' -'ausreichend_' -'ausdrücklich_' -'aufzu' -'aufnehmen_' -'attractive_' -'arrival_' -'ard_' -'apart_' -'aims_' -'ably_' -'abe_' -'Ziel' -'Zeitraum_' -'Weltbank_' -'War' -'Vom_' -'Verwaltung_' -'Venezuela_' -'Var' -'Umfang_' -'Tour_' -'Tor' -'To' -'Thank_' -'Stu' -'Stil_' -'Stellung_' -'Sat' -'Santa_' -'Sanktionen_' -'Sand' -'Saddam_' -'SQL_' -'Rück' -'Russlands_' -'Roman_' -'Roma_' -'Richtlinien_' -'Rede_' -'Rechts_' -'Quelle_' -'Prozesses_' -'Pen' -'Patienten_' -'Other_' -'Option_' -'Mas' -'Mar' -'Luft' -'Leit' -'Leistungs' -'Lebanon_' -'Kyoto_' -'Krankheit_' -'Konflikt_' -'Klimawandel_' -'Kinder' -'Kan' -'Jeder_' -'Hel' -'Hauptstadt_' -'Ha' -'Gründe_' -'Green_' -'Gespräche_' -'Gemeinsamen_' -'Gebieten_' -'Führer_' -'Fähigkeit_' -'Funktionen_' -'Fonds_' -'Find_' -'Faktoren_' -'FR_' -'Emissionen_' -'Ein' -'Each_' -'Dadurch_' -'Charakter_' -'Brüssel_' -'Berichterstatterin_' -'Beide_' -'Begriff_' -'Beach_' -'BA' -'Auf' -'Administration_' -'80' -'23' -'21' -'1' -'.“ _' -' (' -'ären_' -'äre_' -'äge_' -'Übereinstimmung_' -'zählen_' -'zuge' -'yesterday_' -'wirtschaftlich_' -'wichtigste_' -'westlichen_' -'weiteres_' -'wachsende_' -'vous_' -'verstärken_' -'verfolgt_' -'van' -'unter' -'uni' -'ungs' -'und' -'umfassenden_' -'um' -'tzen_' -'trotzdem_' -'tor' -'talking_' -'systeme_' -'surrounded_' -'supposed_' -'super' -'succeed_' -'substances_' -'su' -'stru' -'spielt_' -'sovereignty_' -'sor' -'sit' -'sell_' -'seitens_' -'schools_' -'ru' -'rte' -'rs' -'rly_' -'rental_' -'remote_' -'reicht_' -'referred_' -'records_' -'recognition_' -'radio_' -'quick_' -'properly_' -'producers_' -'processes_' -'prevention_' -'pp' -'patients_' -'pan' -'packages_' -'pa' -'ot_' -'osi' -'ordinary_' -'müssten_' -'myself_' -'mittlerweile_' -'miss' -'militärischen_' -'mas' -'markt_' -'malaria_' -'losing_' -'looked_' -'ler' -'len' -'legitimate_' -'langsam_' -'l' -'kulturelle_' -'kt' -'konkrete_' -'keinerlei_' -'keiner_' -'intellectual_' -'informiert_' -'informed_' -'imposed_' -'impose_' -'import_' -'immigrants_' -'imagine_' -'households_' -'her' -'har' -'hand' -'han_' -'guide_' -'gruppe_' -'grow_' -'golf_' -'ght_' -'geschützt_' -'geb' -'gap_' -'ft' -'früher_' -'freiheit_' -'flows_' -'flat_' -'fit_' -'fell_' -'explanation_' -'ex_' -'ets_' -'est' -'eri' -'ended_' -'ek_' -'eight_' -'economics_' -'dunia_' -'division_' -'discover_' -'devices_' -'device_' -'detail_' -'derzeitigen_' -'depends_' -'del' -'definition_' -'deeply_' -'cycle_' -'cri' -'covered_' -'consultation_' -'conducted_' -'concluded_' -'compensation_' -'colleague_' -'coal_' -'cies_' -'cars_' -'bringing_' -'born_' -'bor' -'bon' -'blocks_' -'block_' -'bildet_' -'beziehen_' -'bezeichnet_' -'bestimmen_' -'beschlossen_' -'bemüht_' -'beigetragen_' -'beaches_' -'ban' -'ball_' -'back' -'ba' -'ausgesetzt_' -'attempts_' -'ati' -'at' -'assume_' -'asking_' -'arguments_' -'appeared_' -'andererseits_' -'an' -'allgemeinen_' -'allgemein_' -'al' -'ai' -'ahmen_' -'agency_' -'Wohlstand_' -'Will_' -'Widerstand_' -'Villa_' -'Very_' -'VO' -'Trek_' -'Ton' -'Tod_' -'Test' -'Ta' -'Streit' -'Straßen' -'Standard' -'Sprachen_' -'Speicher' -'Skype_' -'Sieg_' -'Sa' -'Rückgang_' -'Risiko' -'Regelung_' -'Real' -'Que' -'Pu' -'Produkt' -'Problemen_' -'Praxis_' -'Partnerschaft_' -'Ordnung_' -'OS_' -'OR_' -'Not' -'Nobel_' -'Nevertheless_' -'Media_' -'Mann_' -'Macht' -'MI' -'Leider_' -'Lei' -'Lebens_' -'Kriterien_' -'Kommunikation_' -'Kombination_' -'Karte_' -'Inhalt_' -'Industrie' -'Identität_' -'IT' -'Hä' -'Hostel_' -'Handels_' -'Geschäftsordnung_' -'Geldpolitik_' -'Geb' -'Fra' -'Foto_' -'Foreign_' -'Forderungen_' -'Februar_' -'Fax_' -'Experten_' -'Entwurf_' -'Entwicklungs' -'End' -'Ed' -'ER' -'Download_' -'Direkt' -'Dimension_' -'DE' -'Control_' -'Bos' -'Balkan_' -'Austrian_' -'Aussicht_' -'Aufgaben_' -'Arm' -'Analyse_' -'Allgemeinen_' -'Ale' -'Ala' -'Absch' -'AN' -'192' -'™ _' -'– _' -'м' -'zurückge' -'zero_' -'work' -'weit' -'vorge' -'virtually_' -'village_' -'ur' -'unterschiedliche_' -'unternommen_' -'unmittelbar_' -'tut_' -'ts' -'trifft_' -'trans' -'title_' -'temporary_' -'telephone_' -'substantial_' -'stance_' -'square_' -'sprach_' -'sports_' -'spa_' -'sovereign_' -'sized_' -'sieben_' -'sicherlich_' -'sha' -'sensitive_' -'senior_' -'schönen_' -'sage_' -'returns_' -'represented_' -'relaxing_' -'registered_' -'reflects_' -'referendum_' -'reden_' -'rag' -'quantitative_' -'profits_' -'producing_' -'print_' -'pi' -'perfectly_' -'pan_' -'overcome_' -'ord' -'or' -'onto_' -'olo' -'nts_' -'ngs_' -'newly_' -'nan' -'mussten_' -'multi' -'mor' -'ministers_' -'meist_' -'match_' -'marketing_' -'macroeconomic_' -'länder_' -'lten_' -'lovely_' -'lim' -'launched_' -'kraft_' -'klare_' -'kla' -'ki' -'keeping_' -'itu' -'isierung_' -'ise' -'ip' -'instance_' -'install_' -'inequality_' -'il' -'identify_' -'ian' -'hot_' -'ho' -'hinweisen_' -'heiten_' -'head' -'hardly_' -'groß_' -'globalization_' -'gli' -'gewählt_' -'gewisse_' -'gestalten_' -'ged_' -'ge' -'furniture_' -'formed_' -'forget_' -'flow_' -'fel' -'federal_' -'farming_' -'et' -'erstellt_' -'ernst_' -'erle' -'ergeben_' -'erfahren_' -'entschlossen_' -'enabling_' -'emphasise_' -'elle_' -'ek' -'ehr' -'edi' -'ear' -'distributed_' -'disputes_' -'destroyed_' -'deserves_' -'demanding_' -'decide_' -'dealing_' -'crew_' -'contribute_' -'continuing_' -'concrete_' -'comment_' -'combined_' -'combating_' -'cohesion_' -'cards_' -'button_' -'bul' -'broader_' -'briefly_' -'boom_' -'blood_' -'bezahlen_' -'bewegen_' -'bee' -'background_' -'auto' -'ausgaben_' -'aufge' -'atau_' -'argument_' -'ara_' -'ar' -'angeboten_' -'ancient_' -'ana_' -'am' -'aktiv_' -'afternoon_' -'ae' -'ada_' -'ad' -'ach' -'accounts_' -'accompanied_' -'accessible_' -'Zusammenbruch_' -'Zentralbanken_' -'Ze' -'Wo' -'Wirkung_' -'Verlust_' -'Unternehmens' -'Texte_' -'TI' -'Studie_' -'Sprach' -'Sport_' -'Spar' -'Sonder' -'Selbst' -'Sein_' -'School_' -'Schluss_' -'Schaden_' -'Runde_' -'Reform' -'Priorität_' -'Politik' -'Over_' -'Nord' -'Nach' -'Musik' -'Menschenrechts' -'Menge_' -'Madrid_' -'MA' -'Li' -'Law_' -'Lateinamerika_' -'Kredit' -'Kon' -'Justiz_' -'IN' -'Hinweis_' -'Hill_' -'Grundrechte_' -'Grunde_' -'Grad_' -'Good_' -'Gerichtshof_' -'Gemeinschafts' -'Enjoy_' -'Earth_' -'EL' -'Deutsche_' -'Dass_' -'Dan' -'DA' -'Cre' -'Con' -'Cho' -'Charta_' -'Cap' -'CO' -'Bus_' -'Budget_' -'Book_' -'Bon' -'Beihilfen_' -'Bay_' -'BR' -'Ausnahme_' -'Armee_' -'Apple_' -'Antrag_' -'Anreize_' -'Akteure_' -'Airbus_' -'AG_' -'@_' -': “_' -'44_' -'42_' -'1945_' -'194' -'190' -'188' -'12' -' €_' -'“' -'де' -'д' -'übernommen_' -'ör' -'zurück' -'zufrieden_' -'zeit' -'ysteme_' -'yo' -'wünschen_' -'wirtschaft_' -'wind_' -'wie' -'werk_' -'vorschlagen_' -'vorher_' -'vorgelegt_' -'vor' -'verändert_' -'verdient_' -'verbessert_' -'usual_' -'usi' -'una_' -'umfasst_' -'ue' -'täglich_' -'tting_' -'tten_' -'transfers_' -'ther_' -'th' -'tera' -'technische_' -'survive_' -'super_' -'sun' -'suite_' -'stone_' -'statements_' -'spring_' -'sold_' -'sobald_' -'sit_' -'secret_' -'seats_' -'schwe' -'runs_' -'roads_' -'rein_' -'regardless_' -'refugee_' -'recognized_' -'rch' -'rat_' -'rat' -'railway_' -'rage_' -'purposes_' -'protecting_' -'promised_' -'processing_' -'primarily_' -'precise_' -'politisches_' -'platform_' -'permitted_' -'paragraph_' -'organization_' -'offizielle_' -'occasions_' -'ob' -'nötig_' -'nten_' -'nte_' -'nta' -'notwendigen_' -'normalerweise_' -'nord' -'nit' -'niemand_' -'nge_' -'nearby_' -'ndo_' -'naturally_' -'na' -'museums_' -'mostly_' -'mini_' -'mini' -'metro_' -'metres_' -'menschlichen_' -'mechanisms_' -'luxurious_' -'liquidity_' -'leisure_' -'learned_' -'lay_' -'latter_' -'lage_' -'kh' -'journalists_' -'itt_' -'issued_' -'involve_' -'initially_' -'incentives_' -'ina_' -'impressive_' -'implications_' -'ik' -'id_' -'höheren_' -'höchsten_' -'ht' -'household_' -'hostel_' -'hoher_' -'hn' -'hip_' -'hinzu' -'heits' -'guter_' -'gun' -'gu' -'go' -'gewährleistet_' -'gewa' -'gespielt_' -'gelungen_' -'gel_' -'gegründet_' -'gegenwärtige_' -'gefallen_' -'garantieren_' -'gained_' -'führung_' -'fun_' -'forum_' -'fordert_' -'for' -'finding_' -'finanziert_' -'fied_' -'feststellen_' -'festgelegt_' -'feed_' -'fantastic_' -'existence_' -'exclusively_' -'excessive_' -'erwiesen_' -'erleben_' -'erklärte_' -'ering_' -'erfolgreichen_' -'erfolg' -'engineering_' -'endorse_' -'end' -'electronic_' -'electricity_' -'einzusetzen_' -'einsetzen_' -'eingerichtet_' -'eingehen_' -'effectiveness_' -'dé' -'dri' -'diversity_' -'disaster_' -'determine_' -'danke_' -'correct_' -'convenient_' -'communications_' -'coast_' -'club_' -'cho' -'chief_' -'chen' -'centuries_' -'cation_' -'category_' -'bur' -'brachte_' -'booking_' -'bla' -'bitten_' -'besseren_' -'bedroom_' -'availability_' -'aufzunehmen_' -'ationen_' -'army_' -'ari' -'appreciate_' -'apa' -'ante_' -'anbieten_' -'ama' -'ai_' -'agencies_' -'af' -'addressing_' -'ace' -'aba' -'ab' -']]_' -'Zweiten_' -'Zeit' -'Z_' -'YORK_' -'Within_' -'Willen_' -'Wieder' -'White_' -'Wahrheit_' -'WI' -'Verteidigung_' -'Vereinbarungen_' -'Verbrechen_' -'Val' -'Ur' -'Unterschiede_' -'Trans' -'Those_' -'Test_' -'Tau' -'Taiwan_' -'Tages_' -'TNG_' -'Städten_' -'Studien_' -'Spiele_' -'Son' -'Sommer_' -'Sind_' -'Sin' -'She' -'Sha' -'Seine_' -'Schul' -'Sarkozy_' -'SP' -'Regimes_' -'RA' -'Prä' -'Problems_' -'Prinzipien_' -'Premierminister_' -'Preis' -'Plaza_' -'Pin' -'Perspektive_' -'Page_' -'PS' -'Opposition_' -'ON' -'Nor_' -'Nein_' -'Neben' -'Muslims_' -'Multi' -'Meiner_' -'Meanwhile_' -'Location_' -'Libanon_' -'Lassen_' -'Lang' -'Kriegs' -'Kredite_' -'Kor' -'Kontroll' -'Konsens_' -'Klimaanlage_' -'Kirk_' -'King_' -'Kenntnis_' -'Jedes_' -'JavaScript_' -'Jahrzehnten_' -'Jahres' -'Island_' -'Iranian_' -'Ideen_' -'IC' -'Holz' -'Gefahren_' -'Finanzierungs' -'Fest' -'Facilities_' -'FI' -'Executive_' -'Erwartungen_' -'Erd' -'Entwicklungsländer_' -'Einzel' -'Einwohner_' -'Eigen' -'Doha_' -'Des' -'Depression_' -'Data_' -'Cuba_' -'Chile_' -'Bä' -'Bri' -'Ber' -'Bas' -'Ban' -'BE' -'AP' -'AM' -'> _' -'800_' -'52_' -'1991_' -'10' -'05_' -'.” _' -'."_' -') ' -' € _' -' > _' -'қа' -'я' -'к' -'в' -'б' -'č_' -'übertragen_' -'überrascht_' -'Österreich_' -'zugleich_' -'zo' -'zer' -'zentralen_' -'yr' -'winter_' -'widespread_' -'wesentlichen_' -'welcher_' -'weiter' -'wars_' -'warning_' -'wage_' -'volle_' -'vic' -'verwende_' -'verurteilt_' -'vehicles_' -'vehicle_' -'uss_' -'update_' -'unit_' -'unde' -'u' -'tä' -'troops_' -'tro' -'traditionelle_' -'tourism_' -'tori' -'tisch_' -'tend_' -'temp' -'technischen_' -'taste_' -'tal' -'swa' -'sustainability_' -'surplus_' -'sur' -'strengthening_' -'store_' -'sterben_' -'sport_' -'spoke_' -'spectrum_' -'sought_' -'solcher_' -'smus_' -'sme' -'situations_' -'sing' -'signs_' -'sel' -'schwierigen_' -'sar' -'sam' -'ré' -'rung_' -'rum' -'rti' -'rio' -'rin' -'rf' -'returned_' -'resource_' -'resolved_' -'replaced_' -'rei' -'regular_' -'raten_' -'push_' -'ption_' -'propose_' -'proc' -'prior_' -'preis' -'pleasure_' -'planned_' -'pie' -'phi' -'persönlich_' -'personally_' -'ought_' -'ora' -'opposed_' -'opinions_' -'ome' -'ok' -'offenen_' -'of' -'oe' -'nte' -'niemals_' -'ngs' -'nen' -'müsste_' -'möglichst_' -'mö' -'motor' -'mont' -'mon' -'mix_' -'mental_' -'meant_' -'mat' -'ma' -'luxury_' -'lp' -'li' -'letzter_' -'law' -'künftige_' -'kulturellen_' -'kostenlos_' -'ko_' -'knows_' -'kleiner_' -'kamen_' -'jeweils_' -'jam' -'intensive_' -'im' -'ile_' -'ie' -'ichen_' -'hrt_' -'hof_' -'hel' -'gruppen_' -'grundlegenden_' -'greenhouse_' -'greatly_' -'gra' -'gone_' -'glo' -'gewissen_' -'gan' -'führende_' -'fruit_' -'freue_' -'fi' -'fairly_' -'eyes_' -'essen_' -'erscheinen_' -'ersch' -'erg' -'erforderlichen_' -'era' -'enormen_' -'eng_' -'enabled_' -'elite_' -'electoral_' -'durchzuführen_' -'drei' -'dishes_' -'dining_' -'dienen_' -'derart_' -'depend_' -'definiert_' -'ct_' -'convinced_' -'condition_' -'cing_' -'chl' -'cher' -'cas' -'ca' -'burg_' -'budgets_' -'break' -'black_' -'bie' -'beschäftigt_' -'beobachten_' -'bela' -'bekämpfen_' -'basiert_' -'ay_' -'author_' -'ange' -'ambitious_' -'ale' -'ain_' -'afrikanischen_' -'aff' -'adoption_' -'acts_' -'acceptable_' -'absence_' -'abgeschlossen_' -'[[_' -'Zug' -'Ziffer_' -'Zer' -'Zentral' -'Wirklichkeit_' -'Win' -'Weitere_' -'Wandel_' -'WLAN_' -'Völker_' -'Vorbereitung_' -'Volkes_' -'Vienna_' -'Verkehrs' -'Verkehr_' -'Us' -'Umfeld_' -'Tele' -'Teilnahme_' -'Tar' -'Swedish_' -'Summit_' -'Suche_' -'Strukturen_' -'Straf' -'Stellen_' -'Stadtzentrum_' -'Sport' -'Spitze_' -'Spiel' -'Special_' -'Sollte_' -'Sel' -'Science_' -'Sche' -'Samsung_' -'Saint_' -'Saa' -'Rou' -'Rest' -'Research_' -'Rahmen' -'Protokoll_' -'Pay' -'Pan' -'Paket_' -'Optionen_' -'ON_' -'Nutzer_' -'Mon' -'Mitteln_' -'Menschen' -'Maß_' -'Martin_' -'Mac' -'MEPs_' -'Los_' -'LONDON_' -'Körper_' -'Kräfte_' -'Konvent_' -'Kommunikations' -'Kil' -'Kandidaten_' -'Java_' -'Japans_' -'Gu' -'GmbH_' -'General' -'Gefühl_' -'Gedanken_' -'Gebiete_' -'Gas' -'Garten_' -'Fähigkeiten_' -'For' -'Firstly_' -'Fach' -'Export' -'Ex' -'Erklärungen_' -'Eisenbahn' -'ES' -'EA' -'Druck' -'Dor' -'Der' -'DS' -'Czech_' -'Ci' -'Chris_' -'Chinesen_' -'Check_' -'Che' -'CIA_' -'CH' -'Bor' -'Beweis_' -'Beschäftigungs' -'Bei' -'Ball' -'Bade' -'BI' -'Atlantic_' -'Argentinien_' -'Apartment_' -'Angelegenheit_' -'Amerikaner_' -'Aktien' -'Ag' -'Absicht_' -'Ab' -'AC' -'?' -'47_' -'16' -'06_' -'03_' -'00' -'...' -',' -'!' -'і' -'п' -'ни' -'н' -'л' -'ück' -'übrigen_' -'öl' -'öffentlich_' -'ía_' -'än' -'Überprüfung_' -'Ü' -'zweifellos_' -'zentrum_' -'zahl' -'ystem_' -'xi' -'wirken_' -'welt_' -'wall_' -'waiting_' -'wait_' -'wachsenden_' -'wa_' -'vorgesehenen_' -'vorgeschlagenen_' -'visitors_' -'vir' -'verstärkt_' -'verleihen_' -'urs_' -'unmöglich_' -'unity_' -'unfortunately_' -'ug' -'uer' -'uel' -'typical_' -'tt' -'tit' -'tionen_' -'threats_' -'threatened_' -'tet_' -'teile_' -'sustained_' -'sur_' -'suitable_' -'submit_' -'strongly_' -'strength_' -'stimulus_' -'steuer' -'ster' -'steigern_' -'stattfinden_' -'sozial' -'son' -'sky_' -'ski' -'sinnvoll_' -'sight_' -'settlement_' -'sen' -'selbst' -'seite_' -'sei' -'seat_' -'schriftlich_' -'schlecht_' -'scher_' -'scheinen_' -'rou' -'rk' -'ride_' -'rge' -'reply_' -'rep' -'rend' -'remaining_' -'relief_' -'reliable_' -'regimes_' -'reb' -'reagieren_' -'raising_' -'quarter_' -'prove_' -'prospect_' -'proposing_' -'prison_' -'praktisch_' -'possibly_' -'plenty_' -'pattern_' -'outdoor_' -'operate_' -'ola' -'obtain_' -'nsi' -'nommen_' -'nom' -'nk_' -'neighbors_' -'necessarily_' -'nda' -'mögen_' -'mp' -'movements_' -'monitor_' -'mm' -'meter_' -'med_' -'managing_' -'maintenance_' -'lor' -'loo' -'lokale_' -'lly_' -'liked_' -'lier' -'lie_' -'leu' -'length_' -'legitimacy_' -'lad' -'kti' -'kostet_' -'konzentriert_' -'ite' -'islands_' -'invest_' -'interface_' -'ini' -'ingly_' -'immt_' -'ih' -'iel' -'identified_' -'hundred_' -'hr' -'hoffen_' -'hing_' -'highlight_' -'hi' -'heißen_' -'hearing_' -'harus_' -'han' -'halt_' -'globalisation_' -'gerichtet_' -'geraten_' -'generated_' -'gelegt_' -'gehalten_' -'freundlich_' -'frei' -'foot_' -'fonds_' -'festgestellt_' -'false_' -'falling_' -'experienced_' -'estate_' -'eröffnet_' -'erwähnt_' -'erstellen_' -'erfolgt_' -'ered_' -'entscheidenden_' -'entscheidende_' -'enten_' -'ent' -'enhance_' -'ene' -'ending_' -'enables_' -'eg' -'edited_' -'ed' -'ect' -'drop_' -'drink_' -'dramatic_' -'didn_' -'demonstrated_' -'delivered_' -'delay_' -'dal' -'customer_' -'ction_' -'credibility_' -'counter' -'controlled_' -'contracts_' -'considering_' -'confirm_' -'conclusions_' -'client_' -'cit' -'church_' -'chosen_' -'chain_' -'browser_' -'bo' -'bin' -'bezahlt_' -'betroffen_' -'besuchen_' -'besonderen_' -'besitzt_' -'beschränkt_' -'bequem_' -'behalten_' -'begonnen_' -'bedeuten_' -'bas' -'bahwa_' -'aut_' -'ausgesprochen_' -'audio_' -'ative_' -'association_' -'argued_' -'approximately_' -'applies_' -'applicable_' -'ano' -'alt' -'allies_' -'alb' -'akzeptiert_' -'aka' -'aircraft_' -'afford_' -'advantages_' -'adults_' -'adjustment_' -'absch' -']]._' -'Zahlen_' -'Yu' -'Währungsunion_' -'Women_' -'Without_' -'Wien_' -'Webseite_' -'Web' -'Vorsitzenden_' -'Vorschlägen_' -'Voraussetzung_' -'Verhandlungs' -'Verg' -'Verbreitung_' -'Verbraucher' -'Unternehmens_' -'USD_' -'USB_' -'UR' -'Tro' -'Then_' -'Thailand_' -'Termin' -'Ter' -'Technik_' -'Tal' -'Tages' -'TE' -'Symbol_' -'Sub' -'Start_' -'Sitzung_' -'Sim' -'Schutz' -'Schau' -'Reise_' -'Rechnung_' -'Ratsvorsitz_' -'Pool_' -'Player_' -'Plan' -'Partei' -'Par' -'Palästinenser_' -'Palace_' -'Pact_' -'PPE_' -'Ob' -'OECD_' -'Nr_' -'Note_' -'Nos_' -'Nigeria_' -'Nacht' -'Märkten_' -'Myanmar_' -'Munich_' -'Mittelpunkt_' -'Mitgliedern_' -'Mission_' -'Ministerpräsident_' -'Minderheiten_' -'Micro' -'Market_' -'Mag' -'Len' -'Laufe_' -'Labour_' -'Kunst' -'Kooperation_' -'Kontinent_' -'Kim_' -'Israels_' -'Indian_' -'Hussein_' -'Hostelsclub_' -'Handeln_' -'Hamburg_' -'Hal' -'Gründung_' -'Glaubwürdigkeit_' -'Gesch' -'Geh' -'Frühstücksbuffet_' -'Frei' -'Fi' -'Fernsehen_' -'Fal' -'Faktor_' -'Express_' -'Environment_' -'Entwicklungen_' -'Energy_' -'Eltern_' -'Due_' -'Dienste_' -'Die' -'Dep' -'Dem_' -'Datenbank_' -'Cyprus_' -'Cor' -'Company_' -'Cold_' -'Charter_' -'Charles_' -'Char' -'CI' -'CAN' -'Buch' -'Brussels_' -'Blog_' -'Bild' -'Bewertung_' -'Belarus_' -'Before_' -'Bedenken_' -'Back' -'Audio_' -'Association_' -'Arbeitskräfte_' -'Alt' -'Aktionen_' -'Adresse_' -'Abgeordnete_' -'AT' -'3D_' -'37_' -'24' -'20' -'193' -'191' -'09_' -'03' -'/ _' -'/' -') - _' -'() _' -'"' -' -- _' -'” (_' -'н_' -'е_' -'überzeugen_' -'Öffnung_' -'zusätzlichen_' -'zi' -'yn' -'yes_' -'ye_' -'wächst_' -'wood_' -'wonderful_' -'wishes_' -'wireless_' -'weist_' -'weather_' -'wave_' -'wahr' -'vorzu' -'vorgesehen_' -'visible_' -'vis_' -'verändern_' -'verteidigen_' -'vert' -'verlangt_' -'verhindert_' -'ven' -'uren_' -'una' -'uk' -'tzung_' -'tur' -'trouble_' -'trip' -'trillion_' -'treaty_' -'totally_' -'top' -'tele' -'teil_' -'tas_' -'symbol_' -'swi' -'sum_' -'successfully_' -'streets_' -'strategischen_' -'strategische_' -'ste' -'stag' -'sso' -'spricht_' -'spot_' -'somewhat_' -'solely_' -'sodass_' -'sichern_' -'scho' -'schlägt_' -'schi' -'sch' -'sbe' -'sauna_' -'sale_' -'rta' -'roughly_' -'rischen_' -'rg_' -'restore_' -'residential_' -'rescue_' -'removed_' -'religion_' -'regularly_' -'reflection_' -'ree' -'rechte_' -'reaction_' -'rare_' -'qualität_' -'preparation_' -'preise_' -'populist_' -'pollution_' -'pli' -'pleasant_' -'pit' -'pin' -'personnel_' -'perform_' -'patent_' -'passiert_' -'passed_' -'outstanding_' -'ott' -'orm' -'op_' -'ology_' -'olin' -'ol_' -'od' -'noise_' -'nke' -'nights_' -'nicht' -'ndi' -'muslimischen_' -'mpe' -'motor_' -'momentan_' -'ml' -'mindestens_' -'meaningfully_' -'mba' -'markt' -'manche_' -'maintaining_' -'lla' -'lived_' -'linie_' -'liches_' -'liberalisation_' -'let' -'lar' -'lan_' -'kur' -'kontrollieren_' -'kom' -'kin' -'ken' -'karte_' -'jenen_' -'jemand_' -'jegliche_' -'japanische_' -'ita' -'iso' -'installed_' -'ins' -'ingen_' -'inform_' -'inevitable_' -'index_' -'inden_' -'incomes_' -'imbalances_' -'ien' -'ida' -'hä' -'humanity_' -'http_' -'hoo' -'hingegen_' -'hi_' -'heavy_' -'hardware_' -'guarantees_' -'grammatically_' -'gradually_' -'gr' -'gezogen_' -'gestärkt_' -'genügend_' -'genutzt_' -'generations_' -'gene' -'gem' -'gefordert_' -'gan_' -'fördert_' -'free' -'frage_' -'forest_' -'flo' -'firstly_' -'fest' -'fer' -'fen' -'female_' -'fehlt_' -'fees_' -'fears_' -'fe' -'ez_' -'explained_' -'excess_' -'exception_' -'ex' -'estimated_' -'es' -'erreichbar_' -'erinnert_' -'erf' -'er' -'entstanden_' -'entlang_' -'entitled_' -'enst' -'eni' -'ender_' -'ende' -'ences_' -'en' -'eliminate_' -'ektor_' -'einerseits_' -'ei' -'egen_' -'echte_' -'dro' -'driver_' -'driven_' -'door_' -'discussing_' -'discrimination_' -'dis' -'direkten_' -'diplomacy_' -'dimension_' -'dent' -'deiner_' -'declared_' -'daraus_' -'dapat_' -'dans_' -'cru' -'cou' -'copy_' -'convergence_' -'conventional_' -'contextually_' -'containing_' -'contained_' -'cons' -'connections_' -'conce' -'computers_' -'col' -'co' -'classic_' -'ckt_' -'civilian_' -'chten_' -'charming_' -'channels_' -'cash_' -'capitalism_' -'cad' -'blue_' -'bislang_' -'bildung_' -'bewältigen_' -'betreffen_' -'beschleunigen_' -'ben' -'bel' -'bei' -'beginnt_' -'bef' -'barriers_' -'bal' -'baby_' -'ba_' -'ausgezeichneten_' -'aufzubauen_' -'aufgezahlt_' -'attitude_' -'assure_' -'asi_' -'arbeits' -'appointed_' -'amounts_' -'alter_' -'alle' -'album_' -'ala' -'ag' -'affects_' -'adi' -'acknowledge_' -'achieving_' -'acce' -'abuse_' -'Zustand_' -'Zeichen_' -'Ya' -'Wohl' -'Wird_' -'Wiki' -'Wettbewerbs' -'Wa' -'Volkswirtschaft_' -'Veränderung_' -'Verträge_' -'Verf' -'Verbot_' -'Verbesserungen_' -'Up' -'Universität_' -'Under_' -'Tun' -'Ti' -'The' -'Süden_' -'Sy' -'Sur' -'Stärke_' -'Straßen_' -'Stellungnahme_' -'Stability_' -'Staats_' -'Spe' -'Slo' -'Ski' -'Sitz_' -'Schulen_' -'SM' -'SA_' -'Rund' -'Rom' -'Regional_' -'Regierungschefs_' -'Ratschlag_' -'Quellen_' -'Puerto_' -'Programms_' -'Prinzip_' -'Pra' -'Port' -'Para' -'Paper_' -'Palestine_' -'PL' -'Os' -'Orten_' -'Open' -'OK_' -'Norden_' -'Non_' -'Netz_' -'Nacht_' -'NO' -'Mont' -'Ministers_' -'Mel' -'Mehr_' -'May' -'Man' -'Malaysia_' -'Lord_' -'Likewise_' -'Like_' -'Lebensmittel' -'LE' -'LA' -'Konsum_' -'Komfort_' -'Klima_' -'KDE_' -'Jahrzehnt_' -'Islands_' -'Innen' -'Herstellung_' -'Hersteller_' -'Herrschaft_' -'Grün' -'Gro' -'Gesetzgebung_' -'Gerichts' -'Geist_' -'Gegner_' -'Ge' -'Gast' -'Gang_' -'GE' -'Fotos_' -'Foto' -'Flüchtlinge_' -'Finland_' -'Finde_' -'Falls_' -'Est' -'Einwanderung_' -'Einstellung_' -'EC' -'Does_' -'Dienstleistungs' -'Dia' -'Demokraten_' -'Deb' -'Datei' -'DO' -'DC_' -'Class_' -'Canada_' -'C5_' -'Botschaft_' -'Beim_' -'BO' -'Avi' -'Auto_' -'Aufstieg_' -'As' -'Arbeitsmarkt_' -'Arbeiter_' -'Amendments_' -'Alter_' -'Ai' -'Agency_' -'Adobe_' -'Action_' -'Abschließend_' -'AU' -'://_' -'700_' -'49_' -'250_' -'1979_' -'1970_' -'196' -'186' -'08_' -'07_' -'.“_' -'-/_' -'," _' -'): _' -') (_' -'(' -'' ' -''' -' ….. _' -' )._' -'š' -'überlassen_' -'ös' -'öffnen_' -'ès_' -'ätzen_' -'äng' -'ähnlich_' -'Übersetzung_' -'Übersch' -'Ökonomie_' -'Ö' -'zy' -'zustimmen_' -'zusammenge' -'zusammen' -'zugrunde_' -'zuerst_' -'zeichen_' -'ys' -'yp' -'yer_' -'ws_' -'worst_' -'wonder_' -'wollten_' -'windows_' -'willkommen_' -'wesentliche_' -'warten_' -'wal' -'wahren_' -'vors' -'visited_' -'vis' -'versions_' -'verschärft_' -'verehrte_' -'vere' -'va_' -'ux_' -'ute' -'ut' -'ust' -'use' -'ursprünglichen_' -'ursprünglich_' -'uan_' -'twenty_' -'tti' -'tter_' -'travel' -'towns_' -'topic_' -'time' -'tersebut_' -'teachers_' -'taxpayers_' -'tag_' -'ta' -'sz' -'substance_' -'stun' -'stocks_' -'stem_' -'staying_' -'stayed_' -'statistics_' -'stadt_' -'sse' -'speakers_' -'spe' -'soil_' -'sla' -'sis_' -'sicherheit_' -'shed_' -'shares_' -'serviert_' -'sensible_' -'seg' -'see' -'schä' -'savings_' -'saved_' -'satellite_' -'sal' -'ry' -'rund' -'rst' -'rre' -'rot' -'roll_' -'ris' -'richtlinie_' -'richtigen_' -'revolutionary_' -'retten_' -'retirement_' -'reserve_' -'remind_' -'religiösen_' -'reject_' -'ref' -'reasonable_' -'ram' -'que' -'prä' -'prepare_' -'praktische_' -'pr' -'ports_' -'porta' -'politically_' -'phenomenon_' -'pem' -'pass' -'parallel_' -'par_' -'pal' -'oy' -'operational_' -'oma' -'offices_' -'offene_' -'obstacles_' -'näher_' -'nz' -'ntr' -'nto' -'nt' -'notwendige_' -'noted_' -'nochmals_' -'nc' -'nahm_' -'nachhaltig_' -'my' -'mut' -'mother_' -'mixed_' -'mittel' -'mic' -'messen_' -'marks_' -'mand' -'manager_' -'machines_' -'ländlichen_' -'ländern_' -'ls' -'listen_' -'lin_' -'ließen_' -'li_' -'lei' -'lea' -'lawyers_' -'langfristig_' -'lam' -'kurze_' -'kov' -'kor' -'knew_' -'kinds_' -'kel' -'jungen_' -'jpg_' -'joined_' -'items_' -'interessiert_' -'intention_' -'input_' -'innovations_' -'indicated_' -'inde' -'inc' -'in' -'ili' -'igt_' -'ified_' -'ideological_' -'ich' -'ible_' -'höchste_' -'hren_' -'hole_' -'hm' -'hilft_' -'hier' -'hervor_' -'heraus_' -'heil' -'heavily_' -'ham_' -'gäbe_' -'gro' -'gets_' -'gesellschaft_' -'ges' -'gera' -'ger' -'genauso_' -'früheren_' -'frag' -'ford_' -'flights_' -'fla' -'finde_' -'ffer' -'fertig_' -'failures_' -'expand_' -'ese' -'escape_' -'ern' -'erheblich_' -'erb' -'epi' -'entwickelte_' -'ents_' -'ente_' -'enhanced_' -'engagement_' -'engage_' -'encouraging_' -'emi' -'emerged_' -'ema' -'eite_' -'einzu' -'einzigartigen_' -'einfachen_' -'eigener_' -'ef' -'echten_' -'eat_' -'drawn_' -'down' -'disc' -'died_' -'democracies_' -'dee' -'court_' -'corner_' -'convert_' -'contributions_' -'consequence_' -'conditioning_' -'components_' -'collection_' -'cold_' -'coalition_' -'ci_' -'ci' -'careful_' -'cal' -'cable_' -'bri' -'brand_' -'betreiben_' -'besitzen_' -'berg' -'beinahe_' -'behaupten_' -'behandeln_' -'begrenzt_' -'bathrooms_' -'attractions_' -'ator_' -'ates_' -'applying_' -'angesprochen_' -'angen_' -'amenities_' -'alen_' -'ah' -'affairs_' -'advertising_' -'adequate_' -'abroad_' -'Zugriff_' -'Zug_' -'Zu' -'Yes_' -'Währungsfonds_' -'Wor' -'Wirtschaftspolitik_' -'Who_' -'Where_' -'Weltkrieg_' -'Wall_' -'Videos_' -'Verh' -'Ve' -'Valencia_' -'Ursachen_' -'UN' -'Truppen_' -'Todes' -'Tiere_' -'Though_' -'Tests_' -'Terroristen_' -'Telefon' -'Tab' -'TER' -'System' -'Stimm' -'Stein' -'Stan' -'Sony_' -'Sho' -'Set_' -'Scho' -'Ruhe_' -'Rest_' -'Reise' -'Regulierung_' -'Regelungen_' -'Ratspräsidentschaft_' -'Rats' -'RD' -'Prüfung_' -'Produkt_' -'Polizei_' -'Polish_' -'Place_' -'Pla' -'Peter_' -'Palestinians_' -'Pala' -'Or_' -'Ol' -'OS' -'Nas' -'My' -'Moscow_' -'Moo' -'Moment_' -'Mittelmeer_' -'Meeres' -'Medien' -'Mass' -'Marktes_' -'Mad' -'MS_' -'MP' -'Licht_' -'Kurz_' -'Kr' -'Konsum' -'Konsequenzen_' -'Konjunktur' -'Komponenten_' -'Kilometer_' -'Ker' -'Ken' -'Keine_' -'Katastrophe_' -'Kap' -'Joseph_' -'Institution_' -'Insbesondere_' -'Inn_' -'Immobilien' -'Ihres_' -'IA' -'Ho' -'Having_' -'Haushalte_' -'Hauptbahnhof_' -'Gut' -'Golden_' -'Glück_' -'Gleichgewicht_' -'Gipfel_' -'Gew' -'Gemeinden_' -'Gegenstand_' -'GNU_' -'Fälle_' -'Fu' -'Freunde_' -'Freuen_' -'Fla' -'Fisch' -'Firma_' -'Fer' -'Fenster_' -'FA' -'Einheit_' -'Einhaltung_' -'ES_' -'Dutch_' -'Deutschlands_' -'Details_' -'Definition_' -'Day_' -'Dam' -'Cu' -'Common_' -'Comm' -'College_' -'Cam' -'CE_' -'CE' -'CD' -'Burg' -'Bulgaria_' -'Boden_' -'Bla' -'Bis' -'Betriebs' -'Besides_' -'Beschluss_' -'Berücksichtigung_' -'Berufs' -'Bedarf_' -'Bars_' -'Banken' -'Ba' -'Auge_' -'Auftrag_' -'Aufst' -'Aufnahme_' -'Are_' -'Arbeiten_' -'Ar' -'Ant' -'Anpassung_' -'Angela_' -'Angaben_' -'Anbetracht_' -'Amtszeit_' -'Ambiente_' -'All' -'AS' -'A3' -':' -'85_' -'60' -'34_' -'26' -'1993_' -'1992_' -'02' -'-' -') – _' -'({{_' -'  _' -'’._' -'у_' -'ж' -'во' -'ün' -'übertr' -'überaus_' -'ña_' -'ätze_' -'änder_' -'älteren_' -'Übergang_' -'Ärzte_' -'´_' -' – _' -'zuständig_' -'zin' -'ziemlich_' -'zieht_' -'zer_' -'zar' -'ystems_' -'wissenschaftlichen_' -'whereas_' -'western_' -'wel' -'wei' -'wachsen_' -'vorstellen_' -'vorlegen_' -'voller_' -'ving_' -'vieles_' -'vi' -'vern' -'verfügbar_' -'verbindet_' -'urge_' -'unterzeichnet_' -'unt' -'une' -'undoubtedly_' -'uncertainty_' -'unab' -'umzu' -'ult' -'uh' -'ug_' -'uf_' -'uen_' -'tätig_' -'twice_' -'tri' -'trends_' -'tours_' -'tough_' -'tle' -'tische_' -'tin' -'thy_' -'terrorists_' -'telah_' -'tea_' -'tackle_' -'syn' -'swe' -'surface_' -'subsequent_' -'su_' -'storage_' -'stimmt_' -'stic_' -'stein_' -'spaces_' -'soweit_' -'solid_' -'sm' -'sion' -'sh' -'se' -'schweren_' -'sche' -'san' -'rück' -'rte_' -'route_' -'robust_' -'rm' -'rkt_' -'rip' -'ringen_' -'rig' -'revenue_' -'reu' -'respected_' -'rent_' -'renewable_' -'rem' -'regeln_' -'reduzieren_' -'raw_' -'ratio_' -'rap' -'ral_' -'qualified_' -'puts_' -'pun' -'pt_' -'providers_' -'promises_' -'profile_' -'produktion_' -'pred' -'potentially_' -'possibilities_' -'pose_' -'pointed_' -'piel_' -'pic' -'ph' -'periods_' -'percentage_' -'percent_' -'passengers_' -'passenger_' -'para' -'oz' -'ow_' -'our' -'organised_' -'ordered_' -'ona' -'occurred_' -'obligation_' -'object_' -'nötigen_' -'nst' -'nsch' -'novel_' -'none_' -'nk' -'nine_' -'nic' -'ney_' -'newspaper_' -'neu' -'nder' -'nces_' -'ms' -'mountains_' -'mn' -'miles_' -'mes' -'mere_' -'menschliche_' -'medicines_' -'meals_' -'marquis_' -'lug' -'los' -'logical_' -'locations_' -'lobby_' -'ller_' -'lit' -'leaves_' -'layer_' -'lac' -'ky_' -'krieg_' -'klein_' -'kit' -'ize_' -'ists_' -'iron_' -'integriert_' -'informieren_' -'ine' -'incentive_' -'ij' -'ignored_' -'ifi' -'if' -'ics_' -'hs_' -'hot' -'hoped_' -'herrscht_' -'heim' -'happens_' -'handle_' -'haften_' -'grown_' -'griechische_' -'gri' -'grateful_' -'grad' -'gg' -'geöffnet_' -'gestaltet_' -'gering_' -'genannte_' -'gegen' -'gedr' -'gardens_' -'gang_' -'fying_' -'fu' -'fri' -'fourth_' -'foundation_' -'finanzieren_' -'financed_' -'feeling_' -'facto_' -'exciting_' -'everywhere_' -'eur_' -'eta' -'esta' -'erte' -'erscheint_' -'ero' -'erm' -'erlauben_' -'erk' -'erfüllt_' -'erfordern_' -'entw' -'enr' -'engine_' -'engaged_' -'enb' -'ely_' -'ell' -'eli' -'ein' -'eht_' -'eh' -'educational_' -'eco' -'eau_' -'eastern_' -'earth_' -'earned_' -'ea' -'don' -'distinguish_' -'difficulty_' -'dient_' -'di' -'describes_' -'deflation_' -'dedicated_' -'de' -'dates_' -'creative_' -'cre' -'constantly_' -'consistent_' -'clubs_' -'cleaning_' -'cla' -'ción_' -'citizen_' -'chal' -'centres_' -'causing_' -'castle_' -'cas_' -'bs' -'breath' -'boot_' -'boot' -'blind_' -'bild_' -'bilateral_' -'betroffenen_' -'betonen_' -'beschäftigen_' -'beschränken_' -'besch' -'berühmten_' -'beraten_' -'behörde_' -'begins_' -'beendet_' -'bedeutenden_' -'bea' -'bau_' -'backed_' -'awareness_' -'ausschuss_' -'art' -'ari_' -'architecture_' -'anzunehmen_' -'ani' -'angemessene_' -'angegebenen_' -'anc' -'anb' -'amongst_' -'alp' -'aks' -'air' -'affordable_' -'affect_' -'admit_' -'ade_' -'adalah_' -'accepting_' -']]' -'Zwischen_' -'Zweck_' -'Zwar_' -'Zusammenhalt_' -'Zivil' -'Zinssätze_' -'Zimmern_' -'Zei' -'Year_' -'Wirtschafts' -'William_' -'Wesentlichen_' -'Werk_' -'Weiß' -'Vol' -'Vision_' -'Vice_' -'Verd' -'Vel' -'Va' -'VI' -'Trotzdem_' -'Titel_' -'Thanks_' -'Terror' -'Terrasse_' -'Technology_' -'TO' -'Swiss_' -'Support_' -'Suite_' -'Such' -'Steuer_' -'Statt_' -'Staff_' -'Souveränität_' -'Sogar_' -'Signal_' -'Sierra_' -'Serie_' -'Sektoren_' -'Schwellenländer_' -'Schiff_' -'Sc' -'Sar' -'Resolution_' -'Referendum_' -'Ram' -'Rad' -'Profi' -'Produkten_' -'Poker_' -'Past' -'Pap' -'Pakete_' -'PRO' -'PE' -'Organe_' -'Oder_' -'Ob_' -'Nicht' -'News_' -'Mädchen_' -'Mubarak_' -'Mou' -'Module_' -'Mitgliedsstaaten_' -'Met' -'Menü_' -'Menschheit_' -'Maria_' -'Mac_' -'Life_' -'Lern' -'Küsten' -'Kur' -'Kra' -'Koordinierung_' -'Konvents_' -'Konflikte_' -'Kommunismus_' -'Komm' -'Kluft_' -'Kla' -'Kapital_' -'Kampagne_' -'Ji' -'Jerusalem_' -'Jede_' -'Is' -'IM' -'Haus' -'Hand' -'Haftung_' -'Großteil_' -'Gri' -'Grenz' -'Gewinne_' -'Geschäfte_' -'Georgia_' -'Garden_' -'GA' -'Fuß_' -'Form' -'Fischerei' -'Film' -'Ferner_' -'Federation_' -'Fair' -'Entscheidungsträger_' -'Entscheidungs' -'Ents' -'Effizienz_' -'EN' -'Dynamik_' -'Distribution_' -'Din' -'Digital_' -'Desk_' -'Delegation_' -'Del' -'Darum_' -'Cro' -'Colo' -'Climate_' -'Chairman_' -'Cat' -'Card_' -'Captain_' -'Call' -'CF' -'CC' -'Bug' -'Blue_' -'Blair_' -'Black_' -'Bewältigung_' -'Bevölkerungs' -'Beschlüsse_' -'Berg' -'Bereitschaft_' -'Bemerkungen_' -'Bemerkung_' -'Beiträge_' -'Beh' -'Baltic_' -'Balkans_' -'Außenpolitik_' -'Authority_' -'Arabia_' -'Any_' -'Anst' -'Anleger_' -'Anlagen_' -'Agrar' -'Agentur_' -'AI' -'; ' -'98_' -'97_' -'79' -'63_' -'59_' -'55_' -'41_' -'38_' -'237' -'2020_' -'182' -'.  _' -'...._' -'''_' -'''' -' !_' -'‘_' -'ра' -'но' -'ме' -'ма' -'ла' -'й' -'übers' -'ú' -'ände_' -'zugunsten_' -'zierung_' -'zieren_' -'zeug' -'yt' -'ye' -'yar' -'wunder' -'wu' -'woman_' -'wirksame_' -'wir' -'wife_' -'wiederholen_' -'weitgehend_' -'waters_' -'vorliegenden_' -'vorbei_' -'volume_' -'vollständige_' -'vollen_' -'violent_' -'vin' -'verursacht_' -'versetzt_' -'verpflichten_' -'verk' -'veri' -'verabschiedet_' -'variable_' -'unterstütze_' -'untergraben_' -'ungsp' -'underway_' -'understood_' -'unbea' -'ums' -'umgeben_' -'ular_' -'uk_' -'tte' -'trip_' -'tions' -'tief_' -'terror_' -'tennis_' -'tea' -'tat' -'stärkere_' -'stä' -'stylish_' -'stu' -'stellung_' -'speichern_' -'sou' -'solches_' -'smooth_' -'sleep_' -'showed_' -'shape_' -'securities_' -'secular_' -'schöne_' -'schu' -'schrieb_' -'schnelle_' -'schie' -'sake_' -'sah_' -'ruling_' -'ruhig_' -'rl' -'rich' -'ric' -'rg' -'resulted_' -'restructuring_' -'rest' -'ress_' -'respects_' -'respective_' -'resistance_' -'requirement_' -'renowned_' -'regionaler_' -'regarded_' -'reflected_' -'reduziert_' -'red' -'recorded_' -'reconstruction_' -'recommendations_' -'realen_' -'rc' -'raph' -'rant' -'ranging_' -'rain' -'rah' -'purchases_' -'ps' -'präsentiert_' -'proportion_' -'profound_' -'produktiv' -'produces_' -'previously_' -'preis_' -'practically_' -'positiv_' -'pol' -'plenary_' -'ple' -'plays_' -'piel' -'persistent_' -'peri' -'pea' -'pays_' -'password_' -'passieren_' -'partnerships_' -'palästinensischen_' -'ow' -'overlooking_' -'ous' -'ote' -'orts_' -'organized_' -'organisiert_' -'organisationen_' -'organis' -'ordnung_' -'opponents_' -'ond' -'odu' -'observers_' -'oa' -'nz_' -'ny_' -'nse' -'notice_' -'notably_' -'nnen_' -'nn' -'nken_' -'niveau_' -'niedrigen_' -'new' -'ner' -'neighbouring_' -'nch' -'nb' -'nab' -'mpo' -'mount_' -'modest_' -'mod' -'mil' -'mi' -'mend' -'marked_' -'mani' -'male_' -'ly' -'lounge_' -'lots_' -'logic_' -'lle' -'lität_' -'lin' -'liefern_' -'leg' -'langfristigen_' -'landscape_' -'lands_' -'konkreten_' -'komp' -'kin_' -'keineswegs_' -'ka' -'jährigen_' -'justified_' -'ji_' -'japanischen_' -'isn_' -'ions' -'involvement_' -'introducing_' -'installiert_' -'ino_' -'informal_' -'ines_' -'illi' -'id' -'ichts' -'iche_' -'ica' -'hör' -'hr_' -'home' -'hl_' -'hl' -'hilfreich_' -'herzlich_' -'hed_' -'harm_' -'happening_' -'größerer_' -'großem_' -'groß' -'graphics_' -'gh' -'gestiegen_' -'gestern_' -'geschieht_' -'geringere' -'gemein' -'gele' -'geeignet_' -'gains_' -'furnished_' -'freuen_' -'frequently_' -'fort_' -'folgende_' -'floors_' -'finanzierung_' -'fill_' -'fewer_' -'feld_' -'fate_' -'fashion_' -'fare_' -'faith_' -'failing_' -'fach' -'explore_' -'existieren_' -'exi' -'evaluation_' -'euros_' -'esti' -'erstmals_' -'erst' -'ers' -'error_' -'erhielt_' -'ergriffen_' -'entsprechen_' -'enterprise_' -'enf' -'enc' -'ements_' -'einzig_' -'einzelne_' -'einheitlichen_' -'edge_' -'economically_' -'ebene_' -'eas' -'durchführen_' -'druck_' -'drivers_' -'dor_' -'dnung_' -'diverse_' -'dite' -'diskutieren_' -'disco' -'discipline_' -'directed_' -'dig' -'dien' -'destination_' -'designs_' -'demonstrate_' -'demokratischer_' -'define_' -'decisive_' -'deals_' -'dead_' -'dea' -'dam' -'cti' -'ct' -'creates_' -'cosy_' -'cop' -'contributing_' -'constitute_' -'conse' -'commodity_' -'com' -'colours_' -'collaboration_' -'clo' -'cin' -'chtlich_' -'certainty_' -'cameras_' -'bs_' -'broken_' -'brief_' -'blu' -'beteiligt_' -'bet' -'besorgt_' -'bers' -'bericht_' -'bemühen_' -'bell' -'bekommt_' -'beinhaltet_' -'behaviour_' -'beha' -'begun_' -'begr' -'begegnen_' -'bedrooms_' -'bed' -'baru_' -'ay' -'avoided_' -'ausgezeichnete_' -'ausger' -'ausgehen_' -'ausgegeben_' -'aufmerksam_' -'assi' -'aspect_' -'aside_' -'asiatischen_' -'arrested_' -'array_' -'aro' -'arkt' -'are' -'arc' -'ara' -'apparent_' -'ap_' -'ant' -'ank' -'angel' -'amo' -'ament' -'ambi' -'altung_' -'ail' -'agi' -'aggressive_' -'adapt_' -'abkommen_' -'Zuständigkeit_' -'Zusammen' -'Zivilgesellschaft_' -'Zins' -'Zeitung_' -'Wikitravel_' -'Whi' -'Whether_' -'Welche_' -'Wald' -'Wachstums_' -'Vorsch' -'Vorbe' -'View_' -'Vier' -'Verständnis_' -'Vater_' -'Van_' -'Update_' -'Untersuchungen_' -'Universitäten_' -'Ums' -'Transport' -'Tat' -'TH' -'Switzerland_' -'Super' -'Stunde_' -'Studio_' -'Strukturreformen_' -'Strukturfonds_' -'Struktur_' -'Strom' -'Strategy_' -'Strasbourg_' -'Steigerung_' -'Spezialitäten_' -'Sowjetunion_' -'Similarly_' -'Siege' -'Show_' -'Set' -'Sep' -'Selbstverständlich_' -'Sec' -'Sea' -'Schweden_' -'Schlussfolgerungen_' -'Schicksal_' -'Schi' -'Schatten_' -'Sau' -'SO' -'Rückkehr_' -'Ruh' -'Romania_' -'Rob' -'Road_' -'River_' -'Richard_' -'Renten' -'Ren' -'Religion_' -'Regierungskonferenz_' -'Regi' -'Red' -'Raum' -'Projekt' -'Premier_' -'Point_' -'PSE_' -'PL_' -'Nu' -'Nonetheless_' -'Nie' -'Nero_' -'Nazi_' -'National' -'Nachrichten_' -'Nachricht_' -'Monopol' -'Mol' -'Mitgliedstaat_' -'Minute_' -'Mikro' -'Methoden_' -'Mein_' -'Maßnahme_' -'Massen' -'ME' -'MB_' -'League_' -'Laut_' -'LabVIEW_' -'Küste_' -'Künstler_' -'König_' -'Kä' -'Kulturen_' -'Krisen_' -'Kit' -'Kirche_' -'Kategorie_' -'Kas' -'Karten_' -'Kampf' -'Jones_' -'Je_' -'Ins' -'Innenstadt_' -'Indiens_' -'Index_' -'IR' -'IL' -'IG' -'IF' -'Has' -'Har' -'Hai' -'HTML_' -'Grundsatz_' -'Glauben_' -'Gerechtigkeit_' -'Gegens' -'Gas_' -'GS' -'GI' -'Fun' -'Früh' -'Format_' -'Fo' -'Flugzeuge_' -'Finnish_' -'Festlegung_' -'Fat' -'Errichtung_' -'Erm' -'Entw' -'Einbeziehung_' -'Ehe' -'Effekt_' -'Education_' -'ET' -'EP' -'Dritte_' -'Donald_' -'Diskussionen_' -'Diskriminierung_' -'Disk' -'Dis' -'Demokratien_' -'Clo' -'Chávez_' -'Cas' -'Cal' -'Cab' -'Budgets_' -'Budapest_' -'Box_' -'Block' -'Bill_' -'Betr' -'Besucher_' -'Besch' -'Belgium_' -'Beijing_' -'Be' -'Bahnhof_' -'B5_' -'Außen' -'Ausnahme' -'Arbeitgeber_' -'Anlass_' -'Anders_' -'An' -'Alm' -'Alles_' -'Alexander_' -'Abe_' -'Abe' -'AB' -'78_' -'7' -'28' -'2030_' -'1990er_' -'189' -'183' -'… _' -'• _' -'я_' -'ы_' -'ч' -'те' -'п_' -'не' -'ž' -'üsse_' -'üge_' -'üche_' -'überwinden_' -'überprüft_' -'ön' -'ó_' -'ét' -'ég' -'Ökonomien_' -'· _' -' ' -'{{_' -'zählt_' -'zurückzu' -'zunehmende_' -'wünsche_' -'wre' -'wohnen_' -'wohn' -'wohl' -'widerspr' -'wider' -'werte_' -'wer' -'wenden_' -'welcoming_' -'weiterge' -'web' -'water' -'wart' -'ware_' -'vorgenommen_' -'voraus_' -'virus_' -'verspricht_' -'verkaufen_' -'vent' -'veni' -'vel' -'valid_' -'ution' -'uss' -'usa' -'upgrade_' -'unlike_' -'university_' -'universities_' -'ungsm' -'umfangreiche_' -'ultra' -'ud' -'uch_' -'twin_' -'twe' -'tude_' -'tru' -'treat_' -'tobacco_' -'threshold_' -'tha' -'ters_' -'teilnehmen_' -'tech_' -'teaching_' -'targeted_' -'tap' -'tabl' -'supporters_' -'supplies_' -'sul' -'suites_' -'suggested_' -'successes_' -'studio_' -'string_' -'stories_' -'stor' -'stops_' -'sser' -'spüren_' -'spezifische_' -'spezielle_' -'solange_' -'sofern_' -'sma' -'sichtig' -'showing_' -'shortage_' -'shock_' -'sheet_' -'sharp_' -'sexual_' -'setzte_' -'serving_' -'senken_' -'selten_' -'seeks_' -'schön_' -'schnellen_' -'sb' -'sala' -'rä' -'ruhigen_' -'rob' -'rib' -'resolutions_' -'requested_' -'representation_' -'reporting_' -'replace_' -'repeatedly_' -'repeated_' -'repeat_' -'remarks_' -'reli' -'rein' -'regulated_' -'regret_' -'registration_' -'refused_' -'recommended_' -'rechtlichen_' -'recall_' -'rd' -'rb' -'rational_' -'ration_' -'rapporteurs_' -'ran' -'quo_' -'q_' -'q' -'proposes_' -'produziert_' -'privileged_' -'privacy_' -'premi' -'prefer_' -'ppe' -'populations_' -'pon' -'plu' -'phon' -'peak_' -'patterns_' -'parks_' -'oth' -'ose_' -'oro' -'orders_' -'opt_' -'ong_' -'oli' -'olen_' -'oh' -'offiziell_' -'ock' -'occasion_' -'nunmehr_' -'notion_' -'noti' -'not' -'nnte' -'nds_' -'natürlichen_' -'natur' -'nan_' -'namen_' -'mü' -'multiple_' -'multilateral_' -'mpa' -'movie_' -'moralische_' -'mon_' -'moments_' -'mo_' -'minimal_' -'mine_' -'meters_' -'merk' -'medi' -'meal_' -'mbi' -'mber' -'mas_' -'maritime_' -'mann_' -'man' -'magnificent_' -'lte_' -'losigkeit_' -'literature_' -'lis' -'lebt_' -'lebens' -'lat' -'kriege' -'kop' -'konfrontiert_' -'klicken_' -'killing_' -'key' -'kernel_' -'kern_' -'kation_' -'junge_' -'joining_' -'jahr_' -'itte' -'italienischen_' -'istischen_' -'islamischen_' -'is' -'ious_' -'ior' -'involving_' -'invested_' -'inva' -'intentions_' -'intelligence_' -'insufficient_' -'ino' -'inis' -'inf' -'indischen_' -'indi' -'impr' -'implies_' -'ils_' -'igkeiten_' -'ight_' -'idi' -'ica_' -'hängt_' -'humans_' -'hoping_' -'homes_' -'holds_' -'hit' -'hinter' -'hingewiesen_' -'hero' -'hergestellt_' -'helps_' -'hat' -'handel_' -'halbe' -'größeres_' -'grant_' -'grand' -'gor' -'gle' -'glass_' -'gewähren_' -'gesetz' -'gesellschaften_' -'gese' -'geplant_' -'geltenden_' -'gefährdet_' -'gefährden_' -'gebunden_' -'gal' -'fühlen_' -'fy_' -'fte_' -'fs_' -'forthcoming_' -'foot' -'follows_' -'fleet_' -'fitness_' -'fication_' -'fall' -'factor_' -'extraordinary_' -'extra' -'exclusion_' -'eva' -'etz' -'ete_' -'estr' -'esse' -'esi' -'ese_' -'erweisen_' -'erlangen_' -'erheblichen_' -'erhebliche_' -'equity_' -'entscheidend_' -'entgegen_' -'endo' -'emerge_' -'elt_' -'ela' -'eist' -'eintr' -'einfacher_' -'eilung_' -'eil' -'ehe' -'effektiv_' -'eer' -'eda' -'ea_' -'dus' -'dt_' -'drawing_' -'dos_' -'dominant_' -'dl' -'dit' -'disk_' -'direkte_' -'devoted_' -'det' -'derer_' -'dependent_' -'departure_' -'dep' -'den' -'delicious_' -'dealt_' -'cur' -'ctive_' -'corresponding_' -'copyright_' -'converted_' -'contents_' -'conta' -'constant_' -'considerably_' -'conservative_' -'concentrate_' -'con' -'composed_' -'component_' -'comply_' -'comp' -'colour' -'class' -'claimed_' -'cker_' -'cker' -'cht' -'chemical_' -'chel' -'charm' -'caught_' -'capabilities_' -'can' -'calendar_' -'bä' -'bun' -'buchen_' -'bubble_' -'bridge_' -'breiten_' -'breite_' -'boo' -'boa' -'bo_' -'blieb_' -'birth_' -'bild' -'betreffenden_' -'bestehende_' -'beseitigen_' -'beschließen_' -'bekannten_' -'bek' -'beige' -'befassen_' -'beauty_' -'beantworten_' -'bat' -'basieren_' -'badly_' -'autumn_' -'autonomy_' -'auswählen_' -'auswirken_' -'ausländische_' -'ause' -'aufrechtzuerhalten_' -'atz_' -'atz' -'attempted_' -'ath' -'articles_' -'arten_' -'arrived_' -'armen_' -'arise_' -'arian_' -'arge' -'anzuzeigen_' -'anywhere_' -'anf' -'ami' -'amerikanischer_' -'alo' -'alcohol_' -'ak_' -'agr' -'adopting_' -'acting_' -'aci' -'abwe' -']' -'Zusammens' -'Zo' -'Yo' -'Währungen_' -'Would_' -'Work_' -'Wladimir_' -'Winter_' -'Wild' -'Wikicars_' -'Wi_' -'Wel' -'Vorstellung_' -'Vorausschau_' -'Vielmehr_' -'Vielen_' -'Video' -'Verteidigungs' -'Versuche_' -'Vermi' -'Verm' -'Verlauf_' -'Verfolgung_' -'Verein' -'Valley_' -'VE_' -'Use_' -'Uns_' -'Umgang_' -'Treaties_' -'Tochter_' -'Tip_' -'Tim' -'Tier' -'Ticket_' -'Texas_' -'Syn' -'Sudan_' -'Stä' -'Stattdessen_' -'Start' -'Spannungen_' -'Sor' -'Sonnen' -'Sid' -'Shanghai_' -'Several_' -'Serbien_' -'Sei' -'Schweiz_' -'Schuld' -'Sauna_' -'SD' -'Roman' -'Republicans_' -'Rec' -'Punkten_' -'Presse' -'Preisen_' -'Pop' -'Politikern_' -'Pläne_' -'Planung_' -'PP' -'PI' -'Otherwise_' -'Opera_' -'Once_' -'Ober' -'Noch_' -'Niemand_' -'Natur' -'Nat' -'Mur' -'Mos' -'Morgen_' -'Mobile_' -'Migranten_' -'Messe' -'Mensch_' -'Mechanismen_' -'Marketing_' -'Mark_' -'Mario_' -'Mari' -'Manhattan_' -'Mangel_' -'Mala' -'Luc' -'Links_' -'Libya_' -'Lea' -'LateRooms_' -'Ladies_' -'Lab' -'LL' -'LE_' -'Können_' -'Kurz' -'Kurs_' -'Kunden' -'Kuba_' -'Konf' -'Klasse_' -'Keynes_' -'Key' -'Kauf_' -'Kat' -'Kal' -'KO' -'Journalisten_' -'Ja_' -'Interventionen_' -'Internationale_' -'Insgesamt_' -'Ing' -'Infrastruktur' -'Info' -'IS_' -'ISO_' -'Hu' -'Helsinki_' -'Hauses_' -'Handelss' -'Ham' -'Haft' -'Guests_' -'Großen_' -'Grab' -'Ges' -'Gentoo_' -'Generationen_' -'Gelder_' -'Gef' -'Garantie_' -'Ga' -'GB_' -'Führungs' -'Funds_' -'Frank_' -'Formular_' -'Florida_' -'Fischerei_' -'Fi_' -'Feld_' -'Farb' -'Europa' -'Empire_' -'Empfehlungen_' -'Emissions' -'Em' -'Elemente_' -'Element_' -'Einst' -'Eg' -'Dorf_' -'Disc' -'Deutschen_' -'Des_' -'Demo' -'Declaration_' -'Darin_' -'DER_' -'Cra' -'Constitutional_' -'Chu' -'CON' -'Bud' -'Branchen_' -'Bl' -'Bestandteil_' -'Bes' -'Beitritts' -'Beispielsweise_' -'Band' -'Ave' -'Ava' -'Australia_' -'Ausführungen_' -'Ausführung_' -'Arc' -'Arbeitslosen' -'Anschluss_' -'Ann' -'Anlage_' -'Anlage' -'Andernfalls_' -'Alli' -'Alle' -'Aktion_' -'Airlines_' -'Afrikas_' -'Abhängigkeit_' -'ASEAN_' -'AL' -'72_' -'600_' -'6' -'50' -'22' -'200' -'1994_' -'1971_' -'1967_' -'187' -'184' -'120_' -'1000_' -'04_' -'01' -'. „_' -''' (_' -'$_' -'’, _' -'қ' -'да' -'ł' -'ütte' -'üh' -'übersch' -'überdenken_' -'üb' -'öffentlicher_' -'äten_' -'änge_' -'ähnliche_' -'ão_' -'» _' -'  ' -'}} **{{_' -'zweit' -'zwei' -'zunehmenden_' -'zun' -'zugänglich_' -'zuges' -'zione_' -'zentren_' -'youth_' -'yma' -'yi' -'yard_' -'wonach_' -'wis' -'wirkt_' -'wirkliche_' -'whilst_' -'whi' -'wesen_' -'we' -'wan' -'wake_' -'voran' -'voluntary_' -'vollkommen_' -'vier' -'vielmehr_' -'verteilt_' -'verringert_' -'vermutlich_' -'vermitteln_' -'ury_' -'urgently_' -'urf_' -'unte' -'unsch' -'uno' -'united_' -'ungss' -'ungs_' -'undermine_' -'unange' -'umwelt' -'umso_' -'ultimate_' -'uct' -'uchen_' -'uche_' -'tzt_' -'typically_' -'ture_' -'tritt_' -'trial_' -'trag' -'tik_' -'tier' -'ticket_' -'territories_' -'tern' -'tely_' -'teilt_' -'tall' -'tal_' -'tag' -'süd' -'sé' -'sätze_' -'surprised_' -'surely_' -'sure' -'stützen_' -'ständigen_' -'struck_' -'strike_' -'stri' -'strategie_' -'sto' -'stie' -'stell' -'steel_' -'starts_' -'stammen_' -'stal' -'staatlicher_' -'staaten_' -'sst_' -'spoken_' -'specified_' -'sp' -'sozialer_' -'sons_' -'some' -'soldiers_' -'sna' -'sn' -'ska' -'shop_' -'sho' -'shadow_' -'senden_' -'scu' -'sci' -'schr' -'schla' -'schemes_' -'schaden_' -'sauber_' -'san_' -'rweise_' -'rop' -'rooted_' -'ron' -'rle' -'ris_' -'rightly_' -'rie_' -'ria' -'revealed_' -'reta' -'ret' -'reputation_' -'repair_' -'renoviert_' -'ren' -'reg' -'rede' -'recover_' -'recommendation_' -'rechten_' -'reas' -'reactions_' -'rea' -'rating_' -'radi' -'qui_' -'qui' -'question' -'quest_' -'quel' -'qu' -'punkte_' -'psycho' -'prose' -'prop' -'promoted_' -'proceedings_' -'pris' -'presents_' -'posed_' -'poorest_' -'poly' -'play' -'pho' -'pf' -'perubahan_' -'pert' -'permanently_' -'path' -'participants_' -'pace_' -'ously_' -'os' -'orte' -'organ' -'optional_' -'optimal_' -'oppose_' -'opi' -'oo' -'ona_' -'off' -'ode' -'occupation_' -'obtained_' -'objects_' -'ntu' -'nti' -'nswerte' -'normalen_' -'niedriger_' -'nger' -'neun_' -'netz_' -'nent_' -'ndes_' -'narrow_' -'nachhaltigen_' -'nachge' -'müsse_' -'mächtig' -'mä' -'myth' -'modell_' -'mitten_' -'mittels_' -'mistake_' -'missing_' -'minorities_' -'mid' -'messages_' -'mess' -'mes_' -'menye' -'menu' -'menjadi_' -'mela' -'mein' -'mea' -'manufacturers_' -'manchen_' -'makers_' -'lung_' -'lunch_' -'loved_' -'logy_' -'lm' -'lling_' -'lla_' -'lk' -'lists_' -'linguistic_' -'ling' -'light' -'ließ_' -'lieb' -'lev' -'lets_' -'lem' -'leistung_' -'lee' -'lde' -'lau' -'lasting_' -'last' -'lake_' -'kümmern_' -'kämpfen_' -'kä' -'kurzfristig_' -'kurzer_' -'ksi' -'kosten' -'kontrolliert_' -'kon' -'komme_' -'km' -'klär' -'kita_' -'ket' -'ke' -'kan' -'justify_' -'jemals_' -'jederzeit_' -'iter' -'irre' -'ique_' -'interessen_' -'intend_' -'instability_' -'ink' -'indicate_' -'inder' -'inadequate_' -'impression_' -'impa' -'imagin' -'ika' -'ignore_' -'ign' -'iges_' -'iesen_' -'ide' -'ichten_' -'icher' -'ib' -'holidays_' -'hohem_' -'hinder' -'hielt_' -'heritage_' -'herauszu' -'hence_' -'hau' -'handling_' -'haft_' -'gy_' -'gy' -'gui' -'granting_' -'gn' -'gh_' -'gewiss_' -'getragen_' -'geschrieben_' -'geringe_' -'genetic_' -'gelöst_' -'gefa' -'geboren_' -'gallery_' -'gaben_' -'förder' -'functioning_' -'ften_' -'friend_' -'freundliche_' -'franc' -'fossil_' -'fore' -'fon' -'focusing_' -'flying_' -'fly_' -'flug' -'flu' -'filled_' -'fic' -'favor_' -'farm_' -'familiar_' -'exploitation_' -'expert_' -'exceptional_' -'ewa' -'evi' -'eure_' -'etzen_' -'essi' -'erlassen_' -'erkennt_' -'erkannt_' -'eric' -'eo' -'entsteht_' -'entf' -'enge' -'enforcement_' -'ends_' -'empfangen_' -'els_' -'eit' -'eins_' -'einh' -'eingegangen_' -'eingebracht_' -'eignet_' -'eigenes_' -'egal_' -'een_' -'educated_' -'ede' -'economist_' -'echen_' -'duties_' -'dul' -'dte' -'dritte_' -'doctors_' -'doctor_' -'displayed_' -'disasters_' -'ding' -'dialog_' -'des' -'derzeitige_' -'dens' -'demo' -'demi' -'delighted_' -'defeat_' -'decorated_' -'debian_' -'deadline_' -'dd' -'davor_' -'dat' -'dar' -'dahin_' -'cted_' -'creditors_' -'covering_' -'countless_' -'correctly_' -'coordinated_' -'cooperate_' -'compact_' -'codes_' -'cle' -'clause_' -'classes_' -'chef_' -'checking_' -'channel_' -'chances_' -'centr' -'centers_' -'cast' -'cart' -'career_' -'cam' -'buses_' -'bug_' -'bright_' -'bond_' -'bol' -'bit' -'bisa_' -'bis' -'bill_' -'bilateralen_' -'bi_' -'bewirken_' -'bewahren_' -'besar_' -'beruhen_' -'bert' -'bel_' -'behoben_' -'behauptet_' -'beg' -'beantwortet_' -'bauen_' -'basi' -'bahn' -'aver' -'ausländischen_' -'ausgedrückt_' -'aufgeb' -'auff' -'aub' -'ate' -'assen_' -'assa' -'artists_' -'artistic_' -'appropriations_' -'annehmen_' -'angewiesen_' -'anerkannt_' -'anda' -'amp_' -'ambition_' -'alts' -'aktive_' -'ak' -'agen' -'aged_' -'ado_' -'adjustments_' -'actor_' -'achen_' -'acc' -'Zwei' -'Zuge_' -'Zen' -'Zahlungen_' -'Wit' -'Werke_' -'Wal' -'WE' -'Vorstellungen_' -'Vorsitz_' -'Vors' -'Vorlage_' -'Vorgehen_' -'Viertel_' -'Vertretern_' -'Verteidigungspolitik_' -'Verst' -'Using_' -'Unterkunft_' -'Ukrainian_' -'Tunisia_' -'Tru' -'Traum_' -'Transfer' -'Traditionen_' -'Town_' -'Tonnen_' -'Tok' -'Teilnehmer_' -'Tausende_' -'Taliban_' -'TO_' -'Sunday_' -'Straßburg_' -'Stra' -'Stimmung_' -'Steuerzahler_' -'Star' -'Staatsanleihen_' -'Soldaten_' -'Ska' -'Simbabwe_' -'Schätzungen_' -'Schnell' -'Schlüssel_' -'Schlag' -'Schie' -'Satz_' -'SU' -'SG' -'Russians_' -'Rot' -'Roll' -'Rock_' -'Regional' -'Ree' -'Rand_' -'Radio_' -'Qualitäts' -'Professor_' -'Power_' -'Portuguese_' -'Politiken_' -'Playa_' -'Planeten_' -'Pl' -'Pis' -'Philosophie_' -'Pf' -'Personal' -'Performance_' -'Pas' -'Partnership_' -'Partnerschaften_' -'PA' -'Orte_' -'Orient' -'Ok' -'Offenheit_' -'Notes_' -'Northern_' -'Normen_' -'Nizza_' -'Nation_' -'NGOs_' -'NG' -'NCC_' -'Mot' -'Mobil' -'Mittelmeer' -'Mitglieds' -'Missbrauch_' -'Minutes_' -'Mess' -'Menschenhandel_' -'Mei' -'Mau' -'Mark' -'Mallorca_' -'Mach' -'Maastricht_' -'Lö' -'Luft_' -'Ltd_' -'Lounge_' -'Lockerung_' -'Lizenz' -'Lim' -'Liberal' -'Legitimität_' -'Lee_' -'Landwirte_' -'Lake_' -'Lag' -'LO' -'Kritiker_' -'Korean_' -'Klimawandels_' -'Kir' -'Ki' -'Kara' -'Jugend' -'Jahrzehnte_' -'Invasion_' -'Internet' -'Indonesien_' -'Indonesia_' -'Indem_' -'Imp' -'Image_' -'Il_' -'Höchst' -'Hostels_' -'Hor' -'Hit' -'Hin' -'Hilfs' -'Hezbollah_' -'Have_' -'Hall_' -'Hafen_' -'Gua' -'Gil' -'Gewicht_' -'Geschwindigkeit_' -'Germans_' -'Gerichte_' -'Gericht_' -'Gegend_' -'Gard' -'Ganz_' -'Gal' -'GH' -'Friendly_' -'Freude_' -'Freiheiten_' -'Fre' -'Franc' -'Ford_' -'Flüchtlings' -'Fluss_' -'Fischer_' -'Finanzmärkte_' -'Fin' -'Fil' -'Festival_' -'Ferienwohnung_' -'Feld' -'FO' -'Erstellung_' -'Entschließungsantrag_' -'Ele' -'Einsch' -'Einkommens' -'Einer_' -'Eigenschaften_' -'EL_' -'Dun' -'Drogen_' -'Dreh' -'Dokument_' -'Di_' -'Description_' -'Democracy_' -'Dec' -'Dauer_' -'DS9_' -'Cons' -'Conf' -'Com' -'Cit' -'Church_' -'Christ_' -'Chamber_' -'Canon_' -'Camp' -'Cambridge_' -'CS' -'Bürgerinnen_' -'Bürger' -'Börse_' -'Bulgarien_' -'Briten_' -'Blut' -'Blo' -'Ble' -'Betrieb_' -'Beteiligung_' -'Besitz' -'Bereitstellung_' -'Beratung_' -'Begleit' -'Bedürfnisse_' -'Bed' -'Be_' -'Bauern_' -'BS' -'BB' -'Außenminister_' -'Aussch' -'Ausl' -'Assad_' -'Armen_' -'Ari' -'Arch' -'Anzeichen_' -'Anti_' -'Annehmlichkeiten_' -'Anl' -'Anhänger_' -'Angebote_' -'Andere_' -'Ana' -'Amb' -'Ak' -'Air' -'Af' -'Acc' -'Abwe' -'Abr' -'Abd' -'Ab_' -'AV' -'92_' -'68_' -'51_' -'43_' -'40' -'32' -'31' -'19th_' -'02_' -'" ' -' -' -' ,,_' -' ), _' -'ң_' -'та' -'й_' -'и_' -'ды_' -'ε' -'übernimmt_' -'ökonomischen_' -'éa' -'ätig' -'ändische' -'ägen_' -'án_' -'ßt_' -'Überzeugung_' -'Überwachungs' -'Übers' -'Überleben_' -'Übergangs' -'Überblick_' -'zusammenarbeiten_' -'zug_' -'zuf' -'zub' -'zerstört_' -'zerstören_' -'zel' -'zeiten_' -'zahl_' -'younger_' -'wr' -'worrying_' -'win' -'wiederholt_' -'wichtiges_' -'welchen_' -'weiten_' -'watch_' -'wasser' -'warn' -'wahr_' -'wahl_' -'vorauss' -'voices_' -'vit' -'viertel_' -'viable_' -'verwaltung_' -'verr' -'verkehr_' -'verhalten_' -'verboten_' -'vely_' -'var' -'ute_' -'urn' -'untersuchen_' -'unterscheiden_' -'unta' -'unc' -'unan' -'unabhängigen_' -'umge' -'umb' -'uit' -'uar' -'uan' -'türkischen_' -'tsche' -'tsch' -'tro_' -'trib' -'treiben_' -'trees_' -'transportation_' -'transitional_' -'transferred_' -'tranquil' -'tragic_' -'tracks_' -'tra_' -'tour_' -'topics_' -'tone_' -'ton' -'tlich_' -'tion' -'ting' -'till_' -'tig' -'tifi' -'tie' -'tickets_' -'tic' -'tes' -'territorial_' -'terra' -'teils_' -'tehen_' -'taxi_' -'tables_' -'systemen_' -'systematic_' -'sy' -'switch_' -'sustain_' -'surf' -'supplementary_' -'summe' -'suited_' -'sucht_' -'subs' -'strict_' -'stream_' -'strange_' -'stood_' -'sting_' -'stimme_' -'steigenden_' -'stei' -'stars_' -'stammt_' -'stad' -'staat_' -'spät_' -'spiel_' -'spezifischen_' -'socio' -'sli' -'sis' -'significance_' -'sicheren_' -'sicher' -'shocks_' -'shel' -'settlements_' -'seri' -'selben_' -'sehe_' -'seh' -'seconds_' -'schutz_' -'schrittweise_' -'schme' -'schm' -'satz_' -'sation_' -'sand' -'safer_' -'sad' -'sab' -'rv' -'rut' -'rum_' -'row_' -'roots_' -'rme' -'risk' -'rie' -'rian_' -'ri' -'rhetoric_' -'residence_' -'rese' -'reproduction_' -'rene' -'remo' -'remarkable_' -'refers_' -'reductions_' -'recognised_' -'rechtliche_' -'rec' -'realistic_' -'realer_' -'react_' -'re' -'rbe' -'rau' -'rati' -'rail_' -'ragen_' -'racism_' -'quent' -'purchasing_' -'pur' -'prozess_' -'provider_' -'proven_' -'progressive_' -'professor_' -'prisoners_' -'pride_' -'predicted_' -'praktischen_' -'ppi' -'potenzielle_' -'pledge' -'plan' -'pipeline_' -'pioneer' -'pil' -'photograph' -'phones_' -'persönliche_' -'personenbezogene' -'perfekt_' -'pen_' -'painful_' -'pain_' -'overn' -'ov_' -'ost' -'ort' -'originally_' -'organiz' -'ore' -'optimale_' -'ommen_' -'omi' -'om_' -'oll' -'og' -'oftmals_' -'offen' -'odi' -'nü' -'nter' -'notes_' -'northern_' -'nj' -'nischen_' -'nische_' -'niedrig_' -'nieder' -'ngi' -'negativen_' -'nci' -'nche' -'natürliche' -'mögliche_' -'märkten_' -'murder_' -'mou' -'motiv' -'mona' -'mol' -'modify_' -'moderate' -'mode' -'mittleren_' -'minor_' -'minds_' -'min' -'migrants_' -'mi_' -'mete' -'mereka_' -'memper' -'mel' -'medicine_' -'meat_' -'measured_' -'mail' -'magic_' -'lös' -'lum' -'llo' -'lled_' -'lig' -'lift_' -'lieber_' -'lid' -'library_' -'lenken_' -'leidet_' -'leich' -'legte_' -'legally_' -'leak' -'lbe' -'lav' -'lateral' -'langem_' -'lang' -'lai' -'kürze' -'künftigen_' -'könne_' -'kö' -'ktor' -'korrekt_' -'klaren_' -'kill_' -'kesehatan_' -'kas' -'kana' -'jüngste_' -'jedenfalls_' -'iz' -'iv' -'ition_' -'itali' -'isation_' -'ional_' -'involves_' -'invite_' -'investigation_' -'intervene_' -'inten' -'institutionellen_' -'inl' -'incredible_' -'incon' -'immense_' -'illegalen_' -'illegale_' -'illa' -'ility_' -'iel_' -'ideale_' -'hyper' -'hundert_' -'hopes_' -'hood_' -'holders_' -'histori' -'hinzufügen_' -'hinge' -'hierbei_' -'hes' -'hervorragende_' -'heraus' -'hera' -'hee' -'heads_' -'handels' -'halt' -'hal' -'hab' -'günstig_' -'gültige' -'gä' -'gte_' -'größtenteils_' -'grundsätzlich_' -'gramm' -'governing_' -'gos' -'gleichermaßen_' -'gkeit_' -'gi_' -'gesunde' -'gesch' -'genuinely_' -'geni' -'gelingt_' -'gefährliche_' -'gee' -'gebiet_' -'gate' -'fällen_' -'fur' -'fundamental' -'fuels_' -'fro' -'freely_' -'fre' -'fortsetzen_' -'forschung_' -'forests_' -'football_' -'folge' -'flee' -'flash_' -'flag_' -'fire' -'finish_' -'fin' -'figur' -'ffe' -'festgelegten_' -'fanden_' -'fam' -'fails_' -'fahrt_' -'expenses_' -'expanded_' -'exp' -'examine_' -'exa' -'eti' -'esch' -'erzeugt_' -'erwähnen_' -'erreichte_' -'erne' -'erl' -'erfolgreiche_' -'entre' -'enthalt_' -'enth' -'entertainment_' -'entering_' -'ensch' -'ens' -'ener_' -'encouraged_' -'ena' -'employed_' -'emergence_' -'ement_' -'embrace_' -'em' -'eller_' -'elektronischen_' -'electric_' -'eld_' -'eite' -'einzuführen_' -'einverstanden_' -'eintreten_' -'einr' -'einl' -'einheitliche_' -'eingerichtete_' -'ego' -'egen' -'efe' -'ec' -'easing_' -'dw' -'durchzusetzen_' -'dung_' -'dun' -'ds' -'drängen_' -'dream_' -'dot' -'dor' -'doors_' -'distinction_' -'dispute_' -'disposal_' -'diskutiert_' -'dise' -'directory_' -'director_' -'dim' -'dieselbe_' -'dictatorship_' -'determination_' -'deswegen_' -'destroy_' -'dependence_' -'departments_' -'department_' -'demonstrations_' -'definitely_' -'dating_' -'dark_' -'cz' -'cut' -'critique_' -'credits_' -'crat' -'cra' -'count_' -'corporations_' -'conventions_' -'convenience_' -'contributed_' -'contextual_' -'constructive_' -'considerations_' -'conduct_' -'cond' -'compete_' -'coherent_' -'codecision_' -'chtig' -'chte' -'chau' -'characterized_' -'cel' -'cc' -'catch_' -'carrying_' -'busy_' -'bre_' -'bought_' -'bot_' -'boards_' -'black' -'bio' -'binding_' -'bigger_' -'bezieht_' -'beträgt_' -'bestätigt_' -'bes_' -'bere' -'bera' -'bene' -'bele' -'beings_' -'begründet_' -'beglückwünschen_' -'begl' -'beeinflussen_' -'bedroh' -'bedeutende_' -'barer_' -'barely_' -'band_' -'backing_' -'az' -'aw' -'authoritarian_' -'ausb' -'auftreten_' -'aufgeführt_' -'atte' -'att_' -'att' -'ato_' -'asi' -'arte' -'aria' -'argumentieren_' -'apparently_' -'api' -'ap' -'ants_' -'ano_' -'anhaltenden_' -'angebracht_' -'ands_' -'anderswo_' -'alternatives_' -'alongside_' -'ali' -'aktuell_' -'ahm' -'aga' -'adverse_' -'adds_' -'adaptation_' -'acknowledged_' -'ack' -'accused_' -'acceptance_' -'Zwecke_' -'Zusätzlich_' -'Zusatz' -'Zimmerservice_' -'Zielen_' -'Zerstörung_' -'You' -'Wirksamkeit_' -'Wir' -'Werten_' -'Werkzeuge_' -'Wellness_' -'Welcome_' -'Weiter' -'We' -'Wander' -'Wall' -'Wahrscheinlichkeit_' -'WHO_' -'Vorbild_' -'Virus_' -'Vir' -'Vic' -'Verz' -'Versprechen_' -'Verringerung_' -'Vermögenswerte_' -'Verkauf_' -'Verhaltens' -'Vereinigte_' -'Vene' -'Urteil_' -'Umge' -'UE' -'Tätigkeit_' -'Typ_' -'True_' -'Tool_' -'Tibet_' -'Three_' -'Ten' -'Temple_' -'Tee' -'Teams_' -'Südkorea_' -'Sus' -'Sul' -'Structural_' -'Stre' -'Still_' -'Stahl' -'Spo' -'Spielen_' -'Spezial' -'Spa' -'Sound_' -'Sky' -'Situationen_' -'Sit' -'Sir' -'Siehe_' -'Sicherheitspolitik_' -'Service' -'Section_' -'Scotland_' -'Schwäche_' -'Schwerpunkt_' -'Schreiben_' -'Schon_' -'Schli' -'Schle' -'Schiff' -'Scheitern_' -'Schalt' -'Sach' -'SE_' -'Rä' -'Ruf_' -'Rub' -'Rom_' -'Ret' -'Republican_' -'Rechtsstaatlichkeit_' -'Rechten_' -'Rechnungs' -'Rat' -'Rand' -'RO' -'RM' -'Qui' -'Prozess' -'Projekts_' -'Procedure_' -'Presse_' -'Potenzial_' -'Pos' -'Pol' -'Pflicht_' -'Passwort_' -'Parlaments' -'Pakt_' -'PRI' -'PR' -'PO' -'PD' -'Out_' -'Orl' -'Organization_' -'Offen' -'Off' -'Od' -'Nichtraucherzimmer_' -'Netzwerk_' -'Net' -'NI' -'Modern_' -'Mittel' -'Mir_' -'Mini' -'Minderheit_' -'Min' -'Mil' -'Mechanismus_' -'Mc' -'Maschinen_' -'Martin' -'Mar_' -'Mandat_' -'Manchmal_' -'Mak' -'Mai' -'MPEG_' -'MP3_' -'Löhne_' -'Long_' -'Lie' -'Leitung_' -'Lehr' -'Later_' -'Lac' -'LIN' -'Kru' -'Konvention_' -'Konto_' -'Kons' -'Kongress_' -'Kompromiss' -'Kompetenzen_' -'Komp' -'Kernel_' -'Kauf' -'Kai' -'KORE_' -'KE' -'KA' -'Jung' -'Jak' -'Jackson_' -'Jack' -'Internal_' -'Instruments_' -'Innovationen_' -'Inf' -'IP' -'ION_' -'ING_' -'ID' -'Hinzu' -'Herz' -'Henry_' -'Heil' -'Harvard_' -'HI' -'Gästen_' -'Gott_' -'Gor' -'Gleich' -'Glas' -'Gipfeltreffen_' -'Gh' -'Gewerkschaften_' -'Gener' -'Gem' -'Gel' -'Gehminuten_' -'Gebrauch_' -'Gar' -'Fuß' -'Fur' -'Friedens_' -'Fremd' -'Free' -'Frankreichs_' -'Frank' -'Framework_' -'Flughäfen_' -'Flash_' -'Firefox_' -'Feier' -'Fehl' -'Fast_' -'Farben_' -'Fahrzeuge_' -'Fa' -'Exp' -'Esta' -'Er' -'Energies' -'Elite_' -'Einwanderer_' -'Einstellungen_' -'Einnahmen_' -'Einb' -'Egyptian_' -'Edition_' -'EX' -'ER_' -'EM' -'EIB_' -'EI' -'EG_' -'ED' -'Dy' -'Dublin_' -'Drogen' -'Drive' -'Dritten_' -'Drittel_' -'Don_' -'Deswegen_' -'Department_' -'Denmark_' -'Dek' -'Debatten_' -'Cur' -'Cruz_' -'Country_' -'Cou' -'Colombia_' -'Col' -'Client_' -'Chef_' -'Chal' -'Ch' -'Card' -'Can' -'California_' -'Burma_' -'Bu' -'Brok_' -'Bro' -'Br' -'Beweise_' -'Beurteilung_' -'Bett' -'Berichten_' -'Beobachter_' -'Befürworter_' -'Befugnisse_' -'Beachtung_' -'Bea' -'Based_' -'BU' -'BC_' -'Ausz' -'Ausstattung_' -'Ausst' -'Ausr' -'Aus' -'At' -'Asyl' -'Aspekten_' -'Architektur_' -'Arafat_' -'Ansichten_' -'Ans' -'Anonymous_' -'Anliegen_' -'Ange' -'Andererseits_' -'Allein_' -'Act' -'Ac' -'Abschnitt_' -'Absatz_' -'Abf' -'AD' -'66_' -'62_' -'54_' -'46_' -'35' -'29' -'1985_' -'1980s_' -'180_' -'17' -'.) _' -'. - _' -'. &#_' -''', _' -' ?_' -' ; _' -' +_' -' + _' -' "' -'…"..._' -'” ' -'’' -'ül' -'üg' -'ü' -'év' -'éc' -'äußern_' -'äts' -'ät' -'ähl' -'äh' -'Übrigen_' -'Übertragung_' -'Übernachtung_' -'Übereinkommen_' -'Überdies_' -'Ära_' -'zweitens_' -'zuzu' -'zusch' -'zung_' -'zulassen_' -'zug' -'zu' -'ziel_' -'zie' -'zeigte_' -'zb' -'zahlungen_' -'yu' -'you' -'ym' -'yield_' -'yc' -'yan' -'würdigen_' -'wü' -'worthy_' -'wonder' -'wise_' -'willingness_' -'werke_' -'werfen_' -'welcomes_' -'weis' -'wards_' -'ward_' -'waffen_' -'vulnerability_' -'vorhandenen_' -'vollständigen_' -'viv' -'vil' -'vielfältige' -'vid' -'vessels_' -'veröffentlichte_' -'verä' -'verst' -'versorgung_' -'versichern_' -'versch' -'verlor_' -'verkauft_' -'verg' -'verbreitet_' -'venture_' -'ved_' -'vas' -'vacation_' -'uung_' -'uta' -'unterstützte_' -'unterge' -'unsustainable_' -'unions_' -'unin' -'ungsf' -'ung' -'unf' -'unentgeltlich_' -'underlying_' -'underground_' -'umfassen_' -'uis' -'ui' -'ugs' -'uga' -'ues_' -'ude' -'uch' -'tätigen_' -'täglichen_' -'tw' -'ture' -'trick' -'tras' -'trained_' -'traditions_' -'trace' -'tr' -'tolerance_' -'tip' -'thumbnail_' -'throw_' -'texts_' -'teuer_' -'tested_' -'termin' -'tene' -'tendency_' -'techniques_' -'tec' -'tau' -'tari' -'tab' -'sämtliche_' -'sympathy_' -'sver' -'survival_' -'surveillance_' -'suppliers_' -'supervision_' -'superior_' -'succ' -'subsequently_' -'strategi' -'straightforward_' -'str' -'stored_' -'steigt_' -'starten_' -'stand' -'stan_' -'stake_' -'stabiliz' -'stab' -'ssu' -'ssion_' -'sr' -'späten_' -'spl' -'spiegelt_' -'spiegel' -'speziell_' -'sole_' -'ske' -'situ' -'silence_' -'sil' -'side' -'sicherstellen_' -'shot_' -'sf' -'settings_' -'seperti_' -'sens' -'sending_' -'selling_' -'sekarang_' -'sek' -'screens_' -'score_' -'scientists_' -'schätzen_' -'schwer' -'schutz' -'schlimmsten_' -'schlechten_' -'schaften_' -'sama_' -'sai' -'rücken_' -'rü' -'rze' -'ruktur' -'rp' -'rose_' -'ront' -'rken_' -'rium_' -'rif' -'rier' -'rid' -'rh' -'rgi' -'revenues_' -'reve' -'restricted_' -'responses_' -'respect' -'reserved_' -'rer' -'render' -'religiöse_' -'rel' -'reich' -'regulate_' -'regionen_' -'reforme' -'recruit' -'receiving_' -'rece' -'reagiert_' -'read' -'rder' -'rans' -'rai' -'rage' -'queries_' -'qualifi' -'qi' -'pue' -'publicly_' -'proximity_' -'province_' -'promising_' -'prominent_' -'professionals_' -'producer_' -'problem' -'prinzip_' -'preserve_' -'preferred_' -'port' -'politi' -'poli' -'planen_' -'pillar_' -'performed_' -'perceived_' -'pe' -'paying_' -'past' -'parameters_' -'pala' -'pack' -'pa_' -'ov' -'orn' -'orit' -'ores_' -'order' -'ora_' -'opti' -'onne' -'onia_' -'ong' -'omm' -'oleh_' -'oke' -'ogen' -'og_' -'officers_' -'offens' -'offenbar_' -'occupied_' -'observed_' -'observations_' -'obliged_' -'nützlich_' -'nutz' -'nung_' -'nowadays_' -'novel' -'noticed_' -'nominal_' -'nne' -'nn_' -'nh' -'ngan_' -'net' -'neo' -'negotiate_' -'ndung_' -'ncy_' -'nati' -'nant_' -'nal' -'nachfrage_' -'märkte_' -'mächtigen_' -'má' -'mus' -'mot' -'mono' -'moderate_' -'mobility_' -'mmer_' -'minder' -'mier' -'met' -'merc' -'mente_' -'menc' -'mena' -'men' -'mema' -'melt' -'mehr' -'medieval_' -'maßen_' -'max_' -'mate' -'mam' -'machten_' -'lösung_' -'läge_' -'lut' -'lon_' -'logi' -'load_' -'llte_' -'listings_' -'lip' -'life' -'lic' -'leichter_' -'leib' -'legacy_' -'leb' -'lautet_' -'laufenden_' -'langer_' -'lah' -'lag' -'künftig_' -'kü' -'ktion_' -'kritisiert_' -'kri' -'kostenlose_' -'kontextuellen_' -'kontextuell_' -'konst' -'konse' -'komplexe_' -'kommunistischen_' -'komm' -'kne' -'kli' -'kleineren_' -'klarer_' -'kette_' -'kes_' -'kers_' -'kennt_' -'kem' -'kel_' -'keen_' -'kat' -'jährige_' -'jara' -'jar' -'ivi' -'ively_' -'iti' -'israelischen_' -'isi' -'isen_' -'iri' -'iranische_' -'ira' -'investiert_' -'interpretation_' -'internationales_' -'intends_' -'intelligent_' -'instructions_' -'instantly_' -'inspired_' -'inspection_' -'initiated_' -'inhabitants_' -'inflows_' -'indicators_' -'inclusion_' -'importantly_' -'ille' -'ill' -'ification_' -'ideology_' -'ic' -'hältnis' -'hydro' -'hor' -'honour_' -'honest_' -'hoff' -'hire_' -'hinzufugen_' -'hilfe_' -'hidden_' -'hes_' -'hervorragenden_' -'hervorheben_' -'hervorgehoben_' -'herunterladen_' -'herunter' -'heat_' -'healthy_' -'hd' -'hast_' -'hart_' -'harmful_' -'harder_' -'hamm' -'halte' -'hack' -'guided_' -'gua' -'gründen_' -'gru' -'grati' -'grammatisch_' -'governmental_' -'gne_' -'gis' -'gier' -'ghe' -'gewählten_' -'gewonnen_' -'gewicht' -'get' -'gesto' -'gest' -'geschw' -'gerü' -'geprüft_' -'gep' -'gens' -'generous_' -'generell_' -'gend' -'gelang_' -'gefördert_' -'gefragt_' -'gebildet_' -'gear' -'gat' -'gaining_' -'ful' -'fueled_' -'friedlichen_' -'frequency_' -'fran' -'frame_' -'fra' -'foundations_' -'foster_' -'fordere_' -'forcing_' -'fli' -'fle' -'flaw' -'flags_' -'fix_' -'firmly_' -'finanz' -'films_' -'fie' -'festival_' -'ferner_' -'fee_' -'fans_' -'falsche_' -'facts_' -'facilitate_' -'extremists_' -'expressing_' -'explo' -'experiences_' -'expenditures_' -'expe' -'expanding_' -'executive_' -'ew' -'evident_' -'everybody_' -'estimates_' -'ess' -'erzeugen_' -'ery_' -'erwä' -'erweitern_' -'ersetzen_' -'erkunden_' -'erhöhte' -'erholen_' -'erforderliche_' -'erd' -'erarbeitet_' -'eo_' -'environmentally_' -'entst' -'entdeckst_' -'ensures_' -'ensi' -'enn' -'enjoying_' -'enjoyed_' -'ening_' -'englische' -'enger_' -'enge_' -'enemy_' -'endet_' -'endes_' -'emotional_' -'eme' -'embark' -'eingel' -'eingehalten_' -'ege' -'ega' -'effektive_' -'echt' -'ebe' -'eb' -'eate' -'earn_' -'ean' -'durchs' -'dungen_' -'dru' -'drives_' -'dramatically_' -'dol' -'dm' -'disa' -'dil' -'dignity_' -'digit' -'die' -'df' -'desktop_' -'desk_' -'designer_' -'deserve_' -'description_' -'describe_' -'ders_' -'dern_' -'derl' -'derartigen_' -'depth_' -'depending_' -'denk' -'dem' -'def' -'das' -'dak' -'customs_' -'cua' -'creat' -'courts_' -'count' -'corrupt_' -'cope_' -'conveniently_' -'consult' -'constraints_' -'connect_' -'confronted_' -'confrontation_' -'confront_' -'concentration_' -'complicated_' -'comparison_' -'communism_' -'commit_' -'combine_' -'collect_' -'closing_' -'clarity_' -'civilians_' -'cit_' -'circle_' -'chtung_' -'chne' -'chant' -'chan' -'challenging_' -'cepti' -'cell_' -'cand' -'camp_' -'bö' -'broadcast' -'brechen_' -'brauch' -'bot' -'bon_' -'blog_' -'blind' -'billig' -'bevölkerung_' -'betrieben_' -'bes' -'bert_' -'berge' -'berat' -'belegt_' -'beitr' -'beibehalten_' -'behe' -'beenden_' -'bau' -'band' -'ball' -'bail' -'bai' -'bags_' -'bagi_' -'bag' -'bad' -'backs_' -'awa' -'authors_' -'aut' -'ausschusses_' -'aussch' -'ausgeschlossen_' -'ausgeführt_' -'ausführliche' -'ausführ' -'aufgegeben_' -'aufgefordert_' -'asso' -'ass_' -'arti' -'arrest_' -'aris' -'arg' -'arbitrary_' -'appearance_' -'app' -'antr' -'annt' -'anlagen_' -'anischen_' -'angs_' -'angewandt_' -'angest' -'anger_' -'angepasst_' -'anerkennen_' -'anerkannte' -'ander' -'andel' -'all' -'alism_' -'alisierung_' -'akt' -'aki' -'ake' -'aine' -'aft_' -'advances_' -'adv' -'achtet_' -'accurate_' -'abzusch' -'abst' -'abi' -'aben_' -'abandon_' -'Zucker' -'Zone_' -'Zimmer' -'Zimbabwe_' -'Ye' -'Wr' -'Wort' -'Wissenschaft' -'Widerstands' -'Wi' -'Werbung_' -'Wende' -'Weg' -'Wechselkurs_' -'Webseiten_' -'Watson_' -'Water' -'Ware_' -'Wahr' -'Waffen' -'WASHINGTON_' -'WAR' -'Vorgehensweise_' -'Voll' -'Vi' -'Vert' -'Verluste_' -'Verl' -'Verge' -'Veranstaltungen_' -'Veranstaltung_' -'Vera' -'Ver' -'VER' -'User_' -'Up_' -'Umweltschutz_' -'US' -'UM' -'Transa' -'Trag' -'Tourismus_' -'Todesstrafe_' -'Thu' -'Tha' -'Textil' -'Temperaturen_' -'Tas' -'Tarifa_' -'Tan' -'TE_' -'Sä' -'Subventionen_' -'Str' -'Stor' -'Stockholm_' -'Stell' -'Stau' -'Stat' -'Sports_' -'Spielraum_' -'Spi' -'Space_' -'Sound' -'Sol_' -'Sof' -'Sign' -'Siemens_' -'Sicherheitsrat_' -'Shar' -'Serb' -'Ser' -'Senkung_' -'Sen' -'Sektors_' -'Schwer' -'Schwellenländern_' -'Schwei' -'Schw' -'Schu' -'Schlusselwort_' -'Schlusselphrase_' -'Schloss_' -'Rücks' -'Russ' -'Rumänien_' -'Rules_' -'Rubrik_' -'Rose' -'Right_' -'Ric' -'Rela' -'Rei' -'Ref' -'Rather_' -'RT' -'Provence_' -'Project_' -'Prize_' -'Praktiken_' -'Positionen_' -'Porto_' -'Pod' -'Plus_' -'Plat' -'Phänomen_' -'Pers' -'Pen_' -'Peace_' -'Park' -'Papa' -'Pana' -'Palma_' -'Palm' -'Pal' -'Pad' -'PC' -'PARIS_' -'Original' -'On' -'OP' -'OF_' -'Null' -'Notenbank_' -'Nordkorea_' -'Nord_' -'Niederlage_' -'Netto' -'Nationalismus_' -'Nar' -'Mutter_' -'Mut_' -'Muslime_' -'Morocco_' -'Monitor' -'Modelle_' -'Milosevic_' -'Mid' -'Metro_' -'Meter_' -'Medizin_' -'Material_' -'Marina_' -'Mani' -'Manager_' -'Manage' -'Mana' -'Malaria_' -'Mah' -'Magi' -'MM' -'Ly' -'Logik_' -'Lit' -'Listings_' -'Link_' -'Light' -'Liefer' -'Lib' -'Let' -'Lernen_' -'Leitlinien_' -'Lehren_' -'Laun' -'Lastly_' -'Laeken_' -'Kürze_' -'Kö' -'Kopenhagen_' -'Komplex' -'Kle' -'Kie' -'Kata' -'Kapazitäten_' -'Kandidat' -'Kanada_' -'Jun' -'Jer' -'Iran' -'Irak' -'Interessen' -'Install' -'Industrieländern_' -'Indi' -'Import' -'Immobilien_' -'IBM_' -'IB' -'Hunger_' -'Hunde' -'Hongkong_' -'Hol' -'Hitler_' -'Histori' -'Hei' -'Haushaltsdefizit' -'Haupts' -'Halb' -'Had' -'HD' -'Güter_' -'Gulf_' -'Guide_' -'Grünbuch_' -'Große_' -'Großbritanniens_' -'Growth_' -'Griechenlands_' -'Governments_' -'Golf' -'Gleichstellung_' -'Gla' -'Get_' -'Gestaltung_' -'Gesicht_' -'Genu' -'Gene' -'Geber' -'Games_' -'GP' -'Funktions' -'Fro' -'Friday_' -'Fri' -'Freunden_' -'Freizügigkeit_' -'Fraktionen_' -'Fortunately_' -'Force_' -'Food_' -'Fol' -'Florence_' -'Fle' -'Fischler_' -'Fisch_' -'Finnland_' -'Finanzsystem_' -'Finanzs' -'Finanzen_' -'Fernseh' -'FT' -'FS' -'FE' -'Export_' -'Existenz_' -'Exchange_' -'Eta' -'Este' -'Erwachsene_' -'Ep' -'Entwicklungsp' -'Einfluss' -'Eigentum_' -'ED_' -'Dä' -'Durchsetzung_' -'Dort_' -'Dokumente_' -'Dim' -'Dienst_' -'Det' -'Design' -'Democratic_' -'Def' -'Dea' -'De' -'Davos_' -'Darfur_' -'Daniel_' -'Cy' -'Creati' -'Content_' -'Consumer_' -'Comp' -'Communist_' -'Clubs_' -'Cla' -'Civil_' -'Chan' -'Casino_' -'Café_' -'CT' -'CA_' -'Bürokratie_' -'Bun' -'Bucht_' -'Bol' -'Black' -'Billionen_' -'Bez' -'Bewohner_' -'Betten_' -'Berlusconi_' -'Berg_' -'Belgien_' -'Bele' -'Beifall_' -'Beha' -'Bedürfnissen_' -'Bearbeitung_' -'Bay' -'Baum' -'Base_' -'Band_' -'Bahn' -'Az' -'Außen_' -'Autonomie_' -'Automobil' -'Ausmaß_' -'Ausgangspunkt_' -'Aufforderung_' -'Ass' -'Argument_' -'Antworten_' -'Ansätze_' -'Anleihen_' -'Angriffe_' -'Angl' -'Andr' -'And' -'Ami' -'Alters' -'Alta' -'Ali' -'Album_' -'Aktien_' -'Ah' -'Agrarpolitik_' -'Age_' -'Ablehnung_' -'Ablauf_' -'AS_' -'AC_' -'? ' -'96_' -'88_' -'86_' -'67_' -'61_' -'57_' -'53' -'33' -'27' -'236' -'1990s_' -'1960_' -'185' -'04' -'001' -',/_' -', ‘_' -'+ _' -'%-_' -'%' -'#_' -'! ' -' ("_' -'�' -'“ – _' -'–_' -'ш' -'по' -'м_' -'ли' -'ко' -'ка' -'ер' -'ва' -'ар' -'üstung_' -'ührung_' -'übrigens_' -'überwiegend_' -'überw' -'übernahm_' -'österreichischen_' -'öffentlich' -'ô' -'í_' -'ë' -'è_' -'ç' -'ätz' -'ärmsten_' -'änden_' -'ält_' -'ähr' -'äd' -'Übernahme_' -'°_' -' –, _' -'}}) _' -'| _' -'{_' -'zö' -'zust' -'zukommen_' -'zivil' -'zins' -'ziert_' -'zi_' -'zes_' -'zentral_' -'zen' -'zahlt_' -'yields_' -'yea_' -'xo' -'xe' -'xa' -'wort_' -'worried_' -'worker_' -'witness_' -'wissenschaftliche_' -'winning_' -'will' -'wild_' -'widmen_' -'whereby_' -'wh' -'weshalb_' -'werken_' -'welchem_' -'weite_' -'weigh' -'weapon_' -'weaker_' -'was' -'warme' -'wander' -'walt' -'walls_' -'wahlen_' -'vs' -'vorzunehmen_' -'vorsieht_' -'vorschlag_' -'vorn' -'vore' -'voran_' -'vollst' -'voll' -'volatility_' -'villages_' -'via' -'vez_' -'verz' -'versuch' -'verse' -'verschiedener_' -'verschaffen_' -'verme' -'verliert_' -'verletzt_' -'verhe' -'verh' -'vergeben_' -'verge' -'verfügbaren_' -'verd' -'verbr' -'verbinden_' -'vegeta' -'vari' -'valuable_' -'vak' -'uto' -'uti' -'urs' -'uri' -'urf' -'updated_' -'unzureichend_' -'unw' -'unusual_' -'unterwegs_' -'untern' -'unprecedented_' -'unpa' -'ungew' -'ungen' -'undi' -'unders' -'undergo' -'unden_' -'unau' -'un' -'umr' -'ump' -'uli' -'ukan_' -'ui_' -'uge' -'ucht_' -'ucht' -'ually_' -'ua_' -'ua' -'u0027s_' -'türkische_' -'tät' -'tänd' -'tube_' -'tter' -'très_' -'träge' -'triumph' -'treaties_' -'travellers_' -'transformation_' -'trafficking_' -'tow' -'tors_' -'tons_' -'toll_' -'tol' -'tl' -'tionally_' -'tional_' -'tim' -'til' -'thumb_' -'ths_' -'thousand_' -'thirds_' -'thir' -'thi' -'theo' -'tensions_' -'tens_' -'tellung_' -'teams_' -'taxation_' -'tax' -'tas' -'tana' -'sus_' -'surroundings_' -'surprising_' -'suddenly_' -'substantially_' -'stü' -'studied_' -'student_' -'stoppen_' -'stoff' -'stimulate_' -'stimmung' -'stil' -'stic' -'stelle_' -'steigende_' -'stehenden_' -'steer' -'sted_' -'stattfindet_' -'stages_' -'stabilize_' -'sta_' -'sste' -'ssa' -'spra' -'spor' -'spiel' -'spi' -'spect' -'specifications_' -'speaker_' -'spart' -'sparen_' -'spar' -'spanischen_' -'sow' -'sounds_' -'sorgt_' -'som' -'sol' -'slowdown_' -'slopes_' -'slight_' -'slave' -'skin_' -'sixt' -'simultaneously_' -'sieren_' -'sichere_' -'si' -'shrink' -'sharply_' -'sex' -'ses' -'servers_' -'sent' -'semi' -'segment' -'sec' -'sd' -'schönsten_' -'schwieriger_' -'schwerer_' -'schwach_' -'scht_' -'schri' -'schlu' -'schlimm' -'schlicht_' -'schl' -'schau' -'scene_' -'sav' -'saubere_' -'sat' -'sant' -'sand_' -'rz' -'rufen_' -'rse' -'rsch' -'rov' -'routes_' -'ros' -'rom' -'roll' -'roles_' -'rma' -'rku' -'reverse_' -'retain_' -'resse' -'res' -'requiring_' -'requests_' -'rent' -'rena' -'rek' -'reinforcing_' -'reine_' -'reiben_' -'regulat' -'regelung_' -'refuse_' -'referring_' -'rechtzeitig_' -'reali' -'rasche' -'rar' -'racing_' -'quin' -'quil' -'quantities_' -'qualitative_' -'pä' -'put' -'pursued_' -'punkt' -'proud_' -'protest' -'prote' -'profession' -'private' -'prim' -'preventing_' -'prevented_' -'prevailing_' -'pressures_' -'presenting_' -'predict_' -'ppl' -'ppen_' -'pou' -'posts_' -'positiven_' -'pools_' -'polnischen_' -'polls_' -'pm_' -'plötzlich_' -'ple_' -'platforms_' -'planung_' -'plant' -'pick' -'pflicht' -'pet_' -'pers' -'permit_' -'periphery_' -'pens' -'pel_' -'patio_' -'passing_' -'pas_' -'pas' -'partition_' -'part' -'palästinensische_' -'pakete_' -'pag' -'oß_' -'owners_' -'overseas_' -'oto' -'ossen_' -'ose' -'ories_' -'orientierte' -'oriented_' -'org' -'orc' -'ops_' -'operated_' -'opera' -'ony_' -'ono' -'ole' -'ois' -'oi' -'oga' -'officially_' -'od_' -'nos_' -'normale_' -'non' -'nobody_' -'nnung_' -'nni' -'nin' -'nig' -'nem' -'nell' -'neighbourhood_' -'negotiation_' -'nau' -'native_' -'nai' -'nahmen_' -'m²_' -'museum_' -'moderner_' -'mobilis' -'mmi' -'mme_' -'missions_' -'missile_' -'miss_' -'minu' -'milk_' -'militärischer_' -'merit' -'menja' -'mengen' -'mene' -'medizinischen_' -'mediat' -'meaningful_' -'mb' -'mati' -'massiven_' -'maschine_' -'mart' -'mar_' -'manu' -'mano' -'mals_' -'m2_' -'lässig' -'lur' -'lter_' -'lter' -'lst' -'lowest_' -'low' -'losen_' -'loi' -'logische' -'llt_' -'llers_' -'llen' -'linear_' -'lie' -'lichkeiten_' -'leng' -'leiden_' -'legend' -'lament' -'lalu_' -'lak' -'lacks_' -'lab' -'kund' -'kun' -'kredit' -'konzipiert_' -'kontra' -'konf' -'kommerziellen_' -'klima' -'kek' -'kehren_' -'kata' -'kat_' -'kandidat' -'juga_' -'judicial_' -'journey_' -'jointly_' -'jer' -'jahr' -'itu_' -'ith' -'isten_' -'ising_' -'isiert_' -'isch' -'isa' -'irgend' -'ione' -'investing_' -'interven' -'internen_' -'interessante_' -'institut' -'inso' -'insist_' -'inos_' -'inn' -'ining_' -'incidents_' -'inci' -'impos' -'ime_' -'iklim_' -'ika_' -'igra' -'iger' -'iew' -'ierungen_' -'iegen_' -'ieben_' -'hur' -'hung_' -'human' -'htig' -'hte' -'hro' -'host' -'hom' -'hochwertige' -'hochge' -'hlt_' -'hinaus' -'hill_' -'heu' -'herstell' -'hen' -'harten_' -'harmonisation_' -'happiness_' -'hall_' -'hair_' -'hai' -'guard_' -'gt' -'gründe' -'grew_' -'grenzen_' -'greifen_' -'gratulieren_' -'gran' -'grab' -'gon' -'gol' -'glaub' -'gk' -'geäußert_' -'gewinn' -'gewachsen_' -'getötet_' -'gestr' -'gestartet_' -'gesellschaftliche_' -'geschäft' -'geräumige' -'gerät_' -'gers_' -'geringer_' -'gepr' -'genes_' -'generate_' -'gemeinschaftlichen_' -'gemeinsamer_' -'geli' -'gelassen_' -'gek' -'geistigen_' -'gegangen_' -'gefährlich_' -'gefolgt_' -'gefe' -'gebühr' -'geboten_' -'gd' -'gari' -'gam' -'führer_' -'führe' -'fte' -'frische' -'fried' -'fragment' -'forums_' -'forth_' -'formation_' -'forge_' -'folder_' -'fläche_' -'fiskal' -'fine' -'finds_' -'finances_' -'fft_' -'ffic' -'ffi' -'fei' -'feet_' -'featuring_' -'fault_' -'fan' -'ey' -'expos' -'exporting_' -'exhibit' -'execution_' -'exe' -'excuse_' -'exceptions_' -'exc' -'exact_' -'everyday_' -'eun' -'eth' -'eten' -'essen' -'esca' -'erzielte_' -'ersi' -'ersetzt_' -'erse' -'err' -'erode' -'ernst' -'erlebt_' -'erhoben_' -'erh' -'ergänzen_' -'erer_' -'erende' -'erei' -'ere' -'enze' -'entr' -'entdecken_' -'ense' -'enha' -'energie_' -'enemies_' -'endgültigen_' -'empfängt_' -'empfehlen_' -'emp' -'emo' -'emission' -'embedded_' -'elli' -'elegante_' -'electrical_' -'ekonomi_' -'einzusch' -'einstellen_' -'einst_' -'einrichtungen_' -'einkommen_' -'eingeschränkt_' -'eingesch' -'eindeutige_' -'egte' -'effizienter_' -'ees_' -'eba' -'dürften_' -'dynamism_' -'durchge' -'dur' -'dramatisch_' -'dr' -'downturn_' -'dorthin_' -'domin' -'dle' -'dism' -'discovery_' -'disagree_' -'disabled_' -'direkt' -'dige' -'differ' -'dieselben_' -'dienste_' -'dic' -'deutliche_' -'deuten_' -'desto_' -'denkt_' -'deni' -'demanded_' -'delivery_' -'defining_' -'defi' -'deckt_' -'deck' -'decent_' -'deb' -'dau' -'darstell' -'cycli' -'cyber' -'curs' -'cro' -'credit' -'credible_' -'covers_' -'costly_' -'controlling_' -'contract' -'consolidation_' -'consists_' -'conscious' -'conc' -'como_' -'committees_' -'comme' -'colors_' -'cm_' -'clock_' -'cler' -'cken' -'cke_' -'cia_' -'chts' -'chs' -'chri' -'ched_' -'characters_' -'chaos_' -'champion' -'chair_' -'cet' -'ceme' -'cara_' -'capita_' -'calm_' -'büro' -'bus' -'bury_' -'bungen_' -'broadly_' -'brit' -'bric' -'breed' -'bracht_' -'boundaries_' -'boosting_' -'boat_' -'blow' -'bing_' -'binde' -'big' -'bha' -'bezahl' -'bez' -'bewiesen_' -'beweisen_' -'betont_' -'best' -'besonderer_' -'benutzer' -'benefited_' -'believes_' -'bekannte_' -'bekam_' -'behindert_' -'bedroht_' -'beding' -'bedeutsame' -'be' -'batt' -'bah' -'award_' -'avi' -'avec_' -'ava' -'aux_' -'auszuüben_' -'aussehen_' -'ausgewogene' -'ausgew' -'ausgeb' -'auseinander_' -'aufw' -'auftr' -'aufb' -'attraktive_' -'attract_' -'attended_' -'atr' -'asymmetri' -'ast' -'aspirations_' -'ash' -'ase_' -'artificial_' -'arn' -'arme_' -'arm' -'argues_' -'ard' -'archive_' -'arat' -'appl' -'apo' -'anzuwenden_' -'anzupassen_' -'anwenden_' -'answers_' -'ansehen_' -'anonym' -'anis' -'angesehen_' -'angemessen_' -'angeb' -'andes_' -'amt_' -'ame_' -'allzu_' -'allerg' -'alität_' -'alis' -'ales_' -'ald' -'aku' -'aj' -'airline_' -'ahn' -'ahme' -'aha' -'agieren_' -'afrika_' -'affecting_' -'aer' -'advised_' -'advise' -'adult_' -'ada' -'actors_' -'acted_' -'accounting_' -'accelerating_' -'aca' -'abgesehen_' -'abgelehnt_' -'abg' -'abend' -'aan_' -']], [[_' -'] ' -'Zust' -'Zus' -'Zur' -'Zugleich_' -'Youth_' -'Yas' -'XP_' -'XI' -'Wäscheservice_' -'Wärme' -'Wä' -'Wunder' -'Wissenschaftler_' -'Wirtschaftsp' -'Wirtschaftsf' -'Wirtschafts_' -'Wireless_' -'Wiederherstellung_' -'Wie' -'Widerspruch_' -'Wichtigkeit_' -'Wettbewerbs_' -'Wesen_' -'Werbe' -'Wer' -'Wen' -'Weit' -'Wein_' -'Wei' -'Wea' -'WC_' -'Vorschlags_' -'Vork' -'Vorhaben_' -'Vorf' -'Vollbeschäftigung_' -'Vertrags' -'Verteilung_' -'Versorgung_' -'Verletzung_' -'Verle' -'Verla' -'Verk' -'VI_' -'Urlaubs' -'Unternehmer_' -'Unterkünfte_' -'Unt' -'Typ' -'Twitter_' -'Trend_' -'Tot' -'Top' -'Thursday_' -'Third_' -'Thai_' -'Telefon_' -'Take_' -'Tagung_' -'Systemen_' -'Sun' -'Summe_' -'Suites_' -'Sturz_' -'Studenten_' -'Strom_' -'Strand' -'Stoffe_' -'Stabilitäts_' -'Spring_' -'Spitzen' -'Solange_' -'Smoking_' -'Situated_' -'Sie' -'Sicherheits_' -'Sicher' -'Shuttle_' -'Should_' -'Shop_' -'Sevilla_' -'Seba' -'Schrift' -'Schmerz' -'Schlag_' -'Schl' -'Schei' -'Salz' -'Salo' -'Sala' -'Sai' -'Russen_' -'Ross' -'Rit' -'Ris' -'Rhetorik_' -'Rettungs' -'Rese' -'Reg' -'Real_' -'Reagan_' -'Ratifizierung_' -'Rap' -'Radio' -'RER_' -'Provinz_' -'Programm' -'Prognosen_' -'Produzenten_' -'Produktivität_' -'Press_' -'Presidents_' -'Post_' -'Pont' -'Poll' -'Pole' -'Poker' -'Plenum_' -'Piazza_' -'Patten_' -'Patri' -'Pat' -'Papier' -'Panorama' -'Pale' -'Pak' -'Out' -'Oh' -'OC' -'Nummer_' -'Nuklear' -'None_' -'Nielson_' -'Nichtraucherzonen_' -'Next_' -'Netzwerk' -'Net_' -'Nei' -'Natura_' -'NS' -'NC' -'München_' -'Mul' -'Much_' -'Moskau' -'Mona' -'Moment' -'Modernisierung_' -'Modell' -'Mio_' -'Mini_' -'Millennium_' -'Mill' -'Migration_' -'Mic' -'Meinungs' -'Med' -'Mart' -'Marktwirtschaft_' -'Marg' -'Mao_' -'Mail' -'MD' -'Lü' -'Lärm' -'Lobby' -'Liu_' -'Libyen_' -'Les_' -'Lehrer_' -'Legal_' -'Lebensmittel_' -'Lau' -'Lap' -'Language_' -'Labor_' -'LU' -'LS_' -'LG' -'Kriminalität_' -'Kriege_' -'Kremlin_' -'Kontrollen_' -'Kontext_' -'Kont' -'Konkurrenz_' -'Kollegin_' -'Klaus_' -'Kl' -'Key_' -'Kern_' -'Kenntnisse_' -'Kart' -'Karriere_' -'Kann_' -'KI' -'Jü' -'Justiz' -'Jur' -'Juan_' -'Jordanien_' -'Jordan_' -'Jahrestag' -'Investition_' -'Institut_' -'Ingenieur' -'Industry_' -'Industrieländer_' -'Inc_' -'Ill' -'Id' -'IK' -'Hände_' -'Hypotheken' -'Human' -'Hour_' -'Holiday_' -'Hohe_' -'Hir' -'Hind' -'Herzlich' -'Heraus' -'Haut' -'Hau' -'Hart' -'Harmonisierung_' -'Haar' -'HO' -'HD_' -'Güter' -'Guan' -'Grundsätze_' -'Grundlagen_' -'Gross' -'Governance_' -'Gottes_' -'Gold' -'Global' -'Gestatten_' -'Geschlechter' -'Gen' -'Gemeinschaften_' -'Gefängnis_' -'Gebi' -'Gay_' -'Gates_' -'Gat' -'Gast_' -'Gara' -'GN' -'Führungsrolle_' -'Fü' -'Freizeit' -'Fr' -'Fou' -'Flu' -'Fire' -'Ferienwohnungen_' -'Ferien' -'Feinde_' -'Farm' -'Family_' -'Fallout_' -'FL' -'FC_' -'Excellent_' -'Everything_' -'Europol_' -'Etwa' -'Et' -'Essen_' -'Esp' -'Erwerbs' -'Erne' -'Erk' -'Erfolge_' -'Engine_' -'Energien_' -'Empf' -'Einschätzung_' -'Einschränkung_' -'Eingabe' -'Effektivität_' -'Economy_' -'Ebola_' -'Eb' -'EP_' -'Doppel' -'Dom' -'Dol' -'District_' -'Disp' -'Diplomatie_' -'Deutsch_' -'Deposit_' -'Denken_' -'Defence_' -'Deep_' -'Danach_' -'DL' -'DD' -'DA_' -'Culture_' -'Cru' -'Corb' -'Consensus_' -'Commissioners_' -'Come_' -'Code' -'Coa' -'Churchill_' -'Children_' -'Chat' -'Chancellor_' -'Catal' -'Castle_' -'Cast' -'Cand' -'CP' -'CM' -'Büro' -'Böse' -'Börsen' -'Butt' -'Bru' -'Brand' -'Boy' -'Bord_' -'Boot_' -'Bolkestein_' -'Blin' -'Binnen' -'Bilanz' -'Big_' -'Bie' -'Bibliothek_' -'Bewegungen_' -'Bew' -'Beu' -'Betracht_' -'Beteiligten_' -'Bet' -'Best' -'Besondere' -'Besitz_' -'Benjamin_' -'Bene' -'Ben_' -'Begründung_' -'Beendigung_' -'Bee' -'Bedrohungen_' -'Bedien' -'Beamten_' -'Basel_' -'Barón_' -'Badezimmer_' -'Back_' -'BRICS_' -'Avenue_' -'Automati' -'Ausgangs' -'Ausgabe_' -'Ausflüge_' -'Augen' -'Aufzug_' -'Aufnahme' -'Ast' -'Assembly_' -'Arte' -'Argumente_' -'Arabs_' -'Aqua' -'Ap' -'Anwendungs' -'Anträge_' -'Allianz_' -'Alex' -'Aktivität_' -'Aktionäre_' -'Act_' -'Absichten_' -'Abschaffung_' -'Abg' -'Abendessen_' -'AGE' -'AAA_' -'= _' -'900_' -'90' -'56_' -'2050_' -'1986_' -'1950_' -'195' -'181' -'176' -'.&_' -'. “_' -'+' -'))._' -'", "_' -' {{_' -' ,_' -' &_' -' $ _' -'”' -'“-_' -'“ (_' -'א' -'ө' -'ұ' -'с_' -'ры' -'ны' -'нд' -'в_' -'še' -'üssen_' -'üssel' -'ühren_' -'überschuss_' -'überlegen_' -'überf' -'öm' -'én' -'él' -'äus' -'äufe' -'äst' -'äss' -'ändert_' -'ämpf' -'äl' -'Überlegungen_' -'Überl' -'Überg' -'Är' -'Änderungs' -'zzi' -'zwingen_' -'zweiter_' -'zweifel' -'zutiefst_' -'zuständigen_' -'zus' -'zurückzuführen_' -'zurückf' -'zunehmen_' -'zul' -'zugute_' -'zue' -'zt_' -'zog_' -'zige' -'ziele_' -'zial' -'zentraler_' -'würdig_' -'worry_' -'wood' -'withdrawal_' -'wirklichen_' -'wherever_' -'whenever_' -'wheel_' -'wettbewerbsfähige' -'westliche_' -'wesens_' -'werde' -'wenigsten_' -'wealthy_' -'waves_' -'warrant' -'wann_' -'wand' -'wai' -'wahrscheinliche' -'wachstum_' -'wachs' -'vorzulegen_' -'vorw' -'vorn_' -'vorgeschlagene_' -'vorbe' -'volatile_' -'vist' -'visa_' -'virtue_' -'virtual_' -'villa_' -'videos_' -'victim_' -'vest' -'verwiesen_' -'verwei' -'verwe' -'verwandelt_' -'verwandeln_' -'verwa' -'vertr' -'versteht_' -'versetzen_' -'versa' -'vermittelt_' -'verkehrs_' -'verifi' -'verf' -'verbieten_' -'verbesserte_' -'verband_' -'vera' -'vene' -'varia' -'val_' -'val' -'utz_' -'uted_' -'uste' -'ust_' -'usage_' -'ure' -'uranium_' -'ura' -'upaya_' -'unterstützten_' -'unterst' -'unterschiedlich_' -'unterbreitet_' -'unstable_' -'uns' -'unre' -'unn' -'universe_' -'ungsbe' -'ungsa' -'ungeachtet_' -'unemployed_' -'undenen_' -'unclear_' -'uncertain_' -'unce' -'unabhängige_' -'umgekehrt_' -'umgehen_' -'umen_' -'ull' -'ulat' -'ula_' -'uj' -'uck' -'uchs_' -'uc' -'ubi' -'tätigkeit_' -'tyr' -'typisch_' -'twentieth_' -'tut' -'turno' -'tun' -'tum_' -'tua' -'tta' -'tsunami_' -'träger_' -'tragedy_' -'traf_' -'toxi' -'tot' -'tom' -'tn' -'tliche' -'tigen_' -'tige' -'tien' -'tiefer_' -'tie_' -'threaten_' -'thin' -'thes' -'thermal_' -'there' -'theme_' -'thek' -'theater_' -'tf' -'terrible_' -'terr' -'teri' -'terhadap_' -'tent' -'temperature_' -'tells_' -'teiln' -'teil' -'teh' -'technologischen_' -'techno' -'teacher_' -'tatsächlichen_' -'taten_' -'taste' -'südlichen_' -'sus' -'surprise_' -'supp' -'suggestions_' -'suggestion_' -'sufficiently_' -'succeeded_' -'suc' -'stärksten_' -'stunning_' -'strukturelle_' -'strictly_' -'strengthened_' -'streit' -'stores_' -'stopp' -'stimulat' -'sth' -'steiger' -'steady_' -'station' -'startet_' -'star' -'ssp' -'ssl' -'ssel' -'ssch' -'split_' -'spielte_' -'specialities_' -'sowjetischen_' -'sonn' -'songs_' -'solch_' -'soa' -'smo' -'slow' -'sle' -'sko' -'sk' -'sitze' -'sische' -'sinken_' -'sin_' -'shipping_' -'shing_' -'sher' -'sheer_' -'shareholders_' -'sey_' -'setz' -'sett' -'servi' -'sene' -'sem' -'sely_' -'seits_' -'seit' -'seemingly_' -'seeing_' -'sed' -'sebagai_' -'seas' -'script_' -'schwere_' -'schwachen_' -'schuld' -'schlimmer_' -'schiff_' -'schien_' -'scher' -'sau' -'sarge' -'sanit' -'saf' -'sache' -'räume_' -'rus' -'ruins_' -'ruck' -'rub' -'rts_' -'rtet_' -'roof_' -'rolle_' -'rock_' -'roc' -'rnen_' -'rmo' -'rkte' -'river_' -'riv' -'ring' -'rik_' -'rien' -'rieg' -'restored_' -'responsi' -'reso' -'resist_' -'resign' -'researchers_' -'repression_' -'renewed_' -'relevante' -'reiten_' -'reite' -'reif' -'reichsten_' -'rei_' -'referen' -'redu' -'reco' -'reck' -'realise_' -'real' -'rchi' -'rauch' -'rated_' -'rami' -'radical' -'rab' -'quit' -'quer' -'quelle_' -'pushing_' -'pus' -'pursuing_' -'pure_' -'pto' -'präg' -'provo' -'propaganda_' -'projekte_' -'prohibited_' -'programming_' -'produzieren_' -'proceed_' -'privatis' -'printed_' -'principal_' -'prevail_' -'prev' -'preserv' -'pres' -'preference_' -'preferable_' -'prefer' -'precious_' -'prec' -'popula' -'poorer_' -'politician_' -'pola' -'plä' -'plug_' -'plat' -'plane_' -'pla' -'pilot_' -'pig' -'pieces_' -'physi' -'phy' -'perm' -'performing_' -'penalty_' -'pemerintah_' -'ped' -'peacefully_' -'parteien_' -'parlamentarischen_' -'param' -'pane' -'paket_' -'pak' -'pain' -'overwhelming_' -'overview_' -'overs' -'overr' -'ours_' -'ots_' -'oten_' -'ost_' -'oss' -'osa' -'orthodox' -'orn_' -'orge' -'organic_' -'ore_' -'ordnungsgemäß_' -'ordnete' -'oran' -'ont' -'oned_' -'one' -'onder' -'omis' -'om' -'oliti' -'oldest_' -'ohn' -'ohl_' -'ograph' -'officer_' -'occurs_' -'obe' -'nächster_' -'nze' -'num' -'ntly_' -'nth' -'nso' -'nr' -'nowhere_' -'normally_' -'nm' -'nissen_' -'nga' -'neig' -'nehme_' -'neck' -'ndel' -'nd' -'nationalism_' -'national' -'nat' -'nahe' -'nah' -'nachzu' -'nachh' -'nachdrücklich_' -'multimedia_' -'mul' -'mouth_' -'mounting_' -'mos' -'mood_' -'monopoly_' -'modification_' -'modes_' -'mk' -'mittelalterliche' -'mite' -'mist' -'missbrauch' -'mild' -'mie' -'mehrs' -'mega' -'mee' -'medizinische' -'measur' -'mble_' -'maß' -'maybe_' -'maximi' -'mant' -'manage' -'mana' -'mala' -'mailing_' -'mage' -'mach' -'lö' -'lé' -'lusi' -'loyalty_' -'loyal' -'lowering_' -'lou' -'loose_' -'log' -'locker' -'loan_' -'llig' -'literally_' -'linken_' -'line' -'like' -'liefert_' -'lick' -'lichsten_' -'lich' -'liation_' -'lgen' -'letter_' -'leiten_' -'legitime' -'lays_' -'laste' -'lass' -'lant' -'lance_' -'labelling_' -'kurs' -'ktur' -'ktive' -'kte_' -'ks' -'kritische_' -'konsum' -'konstruktive_' -'komplette' -'kommuni' -'kol' -'kno' -'kni' -'knapp_' -'kleinere_' -'klar' -'kit_' -'kilo' -'kehrt_' -'keb' -'kau' -'jä' -'jum' -'jo_' -'jetzigen_' -'jet' -'ject' -'jan_' -'jah' -'izing_' -'its' -'iter_' -'iss' -'israelische_' -'isolation_' -'ishing_' -'ish' -'ised_' -'ironi' -'iro_' -'irischen_' -'invi' -'investieren_' -'intr' -'internasional_' -'interm' -'intensi' -'intense_' -'installieren_' -'insta' -'innocent_' -'initiat' -'ingung_' -'infolge_' -'info_' -'inflation' -'infected_' -'iner' -'indoor_' -'individuellen_' -'individuelle_' -'individuali' -'indig' -'indicates_' -'ind_' -'incorporated_' -'inad' -'implied_' -'imperative_' -'imper' -'imo' -'imme' -'iller' -'ile' -'iko' -'igen' -'ically_' -'höchst_' -'hy_' -'hs' -'hren' -'hou' -'holes_' -'hob' -'hne' -'hly_' -'hinnehmen_' -'hide_' -'hide' -'heutige_' -'hervorgebracht_' -'herum' -'herself_' -'herausge' -'hem' -'hegemony_' -'headquarters_' -'harte' -'harsh_' -'halben_' -'hal_' -'günstige_' -'gues' -'grünen_' -'grund' -'grosse_' -'gross_' -'grand_' -'graf' -'grade_' -'governed_' -'golden_' -'gnen_' -'gm' -'gio' -'gil' -'ggl' -'gewählte_' -'getr' -'gespeichert_' -'gespa' -'geräte_' -'geringen_' -'geregelt_' -'genießt_' -'geniessen_' -'geneti' -'generi' -'geholfen_' -'geführten_' -'gefährlichen_' -'geeigneten_' -'gedacht_' -'gebi' -'gas' -'ganze' -'fünf' -'führten_' -'füg' -'fü' -'fähr' -'funded_' -'functional_' -'fs' -'freundliches_' -'freige' -'freier_' -'fraud_' -'fragt_' -'formuliert_' -'format' -'forderte_' -'focuses_' -'flicht' -'flexib' -'fits_' -'finished_' -'finanzieller_' -'fina' -'fin_' -'fil' -'fifth_' -'festzustellen_' -'festen_' -'fertig' -'ferr' -'fern_' -'feelings_' -'fan_' -'fak' -'fait' -'fahr' -'factory_' -'fac' -'extremen_' -'extrem_' -'explicitly_' -'experiment_' -'existed_' -'exce' -'ewi' -'ever' -'eut' -'europäisches_' -'etzung_' -'ett' -'etic_' -'ethnischen_' -'ethical_' -'essa' -'esc' -'erwerben_' -'erste' -'ernsthafte_' -'ernsthaft_' -'erleichtern_' -'erhältlich_' -'erhielten_' -'erfreut_' -'erbe' -'erba' -'erati' -'episode_' -'eous_' -'enve' -'entwurf_' -'enthaltenen_' -'entfernen_' -'entdeckt_' -'entar' -'enta' -'engen_' -'empty_' -'employers_' -'empire_' -'emb' -'elte' -'elo' -'eliminating_' -'elf' -'elect' -'ekt' -'einziges_' -'einschl' -'einladende' -'eingestellt_' -'eing' -'eine' -'eind' -'eig' -'eib' -'ehemalige_' -'egung_' -'egt_' -'egi' -'effektiven_' -'edition_' -'ech' -'ebo' -'ease_' -'dynami' -'dscha' -'dry_' -'druck' -'drew_' -'downtown_' -'doubts_' -'donor_' -'domain_' -'disp' -'disorder_' -'disk' -'discharge_' -'disastrous_' -'direct' -'diplomats_' -'dine' -'dienst_' -'dian' -'diam' -'diagnose' -'detaillierte' -'desirable_' -'desi' -'deri' -'depart' -'deny_' -'denselben_' -'denied_' -'demonstrates_' -'demnächst_' -'deli' -'delegate' -'dein' -'dei_' -'degradation_' -'defin' -'deemed_' -'decrease_' -'deco' -'declining_' -'debts_' -'debating_' -'daz' -'dauert_' -'dauerhafte_' -'dasselbe_' -'dangers_' -'dance_' -'damaged_' -'cyclical_' -'cuti' -'ctions_' -'crowd' -'crack' -'countryside_' -'cos' -'cook' -'conte' -'constructed_' -'connect' -'confusion_' -'confirmation_' -'configuration_' -'confident_' -'condemned_' -'comprises_' -'compliance_' -'complement' -'commande' -'coloni' -'collaborat' -'coincide' -'codi' -'coc' -'clip' -'clarify_' -'cks_' -'cio_' -'cio' -'cian' -'cia' -'choosing_' -'chet_' -'cheaper_' -'char' -'centrally_' -'cells_' -'celebrate_' -'ced_' -'catalog' -'capitalist_' -'cancelled_' -'campaigns_' -'by' -'buying_' -'brutale' -'brutal_' -'bru' -'bro' -'breit' -'breakdown_' -'brain_' -'bra' -'boy' -'bou' -'bold_' -'bod' -'boasts_' -'blo' -'blic' -'bles_' -'blank' -'bitter' -'bisherigen_' -'bir' -'biodiversity_' -'bill' -'bic' -'bewegt_' -'betreu' -'betre' -'besuchte_' -'beste' -'beschä' -'beschreibt_' -'beri' -'berechtig' -'berechnet_' -'bemerkenswert_' -'beliebte' -'beizutragen_' -'begrüßt_' -'begab_' -'beeinträchtigt_' -'bedürf' -'bedingten_' -'bedienen_' -'bedauer' -'beda' -'bec' -'bearing_' -'beantragen_' -'battery_' -'bankers_' -'bahn_' -'bab' -'aß' -'az_' -'awarded_' -'aute' -'ausste' -'ausgestattete' -'ausgesch' -'ausgerichtet_' -'ausgelöst_' -'aufweist_' -'aufs' -'aufn' -'aufh' -'aufgerufen_' -'aufgeh' -'aufbauen_' -'audience_' -'attraction_' -'attend_' -'atten' -'attempting_' -'ator' -'ath_' -'atas_' -'assurance_' -'assuming_' -'assumed_' -'assist_' -'assigned_' -'asiatische_' -'ars_' -'arrogant_' -'arro' -'arin' -'argumentiert_' -'approve_' -'approaches_' -'appreciated_' -'appr' -'appe' -'anzeigen_' -'anwe' -'anste' -'anscheinend_' -'annimmt_' -'anne' -'anlage_' -'anhand_' -'angefangen_' -'angef' -'ane_' -'ane' -'amm' -'amend_' -'ama_' -'allocated_' -'allgemeinem_' -'ality_' -'alist' -'alismus_' -'aktivieren_' -'airports_' -'ahl' -'aggregate_' -'ager' -'agents_' -'age' -'ado' -'adequately_' -'aden_' -'addresses_' -'addict' -'activists_' -'achievements_' -'achievement_' -'accus' -'accelerate_' -'abzielt_' -'abt' -'abstimmen_' -'absolut_' -'abrupt' -'abr' -'abo' -'abhängen_' -'abgew' -'abgestimmt_' -'abgest' -'abf' -'abd' -'abandoned_' -'aa' -'Zuständigkeiten_' -'Zugeständnisse_' -'Zugangs' -'Zoll' -'Zeitalter_' -'Zeichen' -'Yugoslavia_' -'Yan' -'Word_' -'Wo_' -'Westens_' -'Wes' -'Weltk' -'Weil_' -'Websites_' -'Watch_' -'Wart' -'Waren' -'Warcraft_' -'Vorz' -'Vorsitzende_' -'Volksgesundheit_' -'Voi' -'Vista_' -'Veröffentlichung_' -'Versionen_' -'Verschl' -'Versammlung_' -'Verkehrsmittel_' -'Verfügbarkeit_' -'Verfahrens' -'Verbündeten_' -'Vac' -'Until_' -'Unterzeichnung_' -'Unterstütz' -'Unterhaltungs' -'Ungarn_' -'Unf' -'Umweltfragen_' -'Umf' -'UT' -'UNG_' -'UB' -'Tö' -'Turn' -'Turk' -'Turb' -'Tschetschenien_' -'Ts' -'Trends_' -'Tourism_' -'Tour' -'Torre' -'Todesfälle_' -'Thomas_' -'Theorie_' -'Tempo_' -'TR' -'TB_' -'Syrian_' -'Swi' -'Swa' -'Störungen_' -'Studium_' -'Struktur' -'Strei' -'Stream' -'Storage_' -'Starfleet_' -'Standort_' -'Spur' -'Spieler' -'Speisen_' -'Sowohl_' -'Southern_' -'Sonne_' -'Sommer' -'Sollten_' -'Smith_' -'Singapore_' -'Sicherheitsa' -'Shin' -'Serbi' -'Select_' -'Sekunde_' -'Scott_' -'Sco' -'Schüler_' -'Schwarz' -'Schulz_' -'Schr' -'Schne' -'Schm' -'Schlafzimmer_' -'Schlaf' -'Sad' -'Sachen_' -'SPA' -'SER' -'Rü' -'Roosevelt_' -'Ronald_' -'Roh' -'Richter_' -'Rhein' -'Rettung_' -'Respekt_' -'Resi' -'Reservierung_' -'Rent' -'Renaissance_' -'Rena' -'Remo' -'Reduzierung_' -'Reaktionen_' -'Ratspräsident_' -'Railway_' -'Rahmens_' -'Rahmenbedingungen_' -'Raf' -'Rab' -'ROM_' -'RG' -'Quick' -'Question_' -'Queen_' -'Qua' -'Qaeda_' -'Pul' -'Prozesse_' -'Protocol_' -'Proteste_' -'Projekten_' -'Privatisierung_' -'Prague_' -'Pr' -'Port_' -'Por' -'Play' -'Pierre_' -'Pha' -'Pflanzen' -'Pfa' -'Peking_' -'Partnern_' -'Paradi' -'Panel_' -'Pack_' -'Pac' -'PS_' -'Over' -'Orts' -'Organ' -'Opti' -'Oper' -'Olympic_' -'Oc' -'OU' -'OT' -'OD' -'Nä' -'Nutz' -'Nor' -'Nokia_' -'Nikon_' -'Niederlanden_' -'Niederlande_' -'Nicolas_' -'Nichts' -'New' -'Never_' -'Napoleon_' -'Nachhaltigkeit_' -'NT_' -'NICHT_' -'ND' -'Mö' -'Muster_' -'Mugabe_' -'Mozilla_' -'Motto_' -'Motor' -'Motiv' -'Moral' -'Monte' -'Mond' -'Modus_' -'Mode_' -'Mitt' -'Milliarde_' -'Mili' -'Michel' -'Metro' -'Merk' -'Ment' -'Mengen_' -'Medic' -'Media' -'Maßstab_' -'Marx_' -'Mars' -'Marokko_' -'Marine_' -'Marc' -'Luxembourg_' -'Love' -'Lou' -'Log' -'List_' -'Liebe_' -'Library_' -'Less' -'Lese' -'Leicht' -'Lehre_' -'Late' -'Lange' -'Lama_' -'Lam' -'LA_' -'Ky' -'Kreuz' -'Kreditkarte_' -'Krediten_' -'Krebs' -'Kranken' -'Kosten' -'Konzept' -'Konvergenz_' -'Kontakt' -'Kongo_' -'Konflikten_' -'Kommissionspräsident_' -'Kommissions' -'Kol' -'Kohlen' -'Koalition_' -'Klar' -'Kei' -'Katastrophen_' -'Katalog_' -'Karls' -'Kamera_' -'Kalten_' -'Jewish_' -'Jesus_' -'Jere' -'Jen' -'Jan_' -'Jacques_' -'Jac' -'Isa' -'Irr' -'Ironi' -'Intervention_' -'Internetzugang_' -'Intergovernmental_' -'Intel_' -'Installer_' -'Innerhalb_' -'Inha' -'Inflationsrate_' -'Immo' -'Images_' -'Ima' -'Identifi' -'Ibiza_' -'IE' -'IC_' -'Hü' -'Hy' -'Hungary_' -'Hum' -'Hub' -'Hot' -'Host' -'Hon' -'Hollande_' -'Hilfen_' -'Hi' -'Herb' -'Heizung_' -'Hea' -'Hauptver' -'Hass' -'Hardware_' -'Harbour_' -'Handlungs' -'HT' -'HS' -'Gö' -'Gäste' -'Gun' -'Griff_' -'Graf' -'Gläubiger_' -'Gläubige' -'Glo' -'Gewiss' -'Geschäft_' -'Geräte_' -'Gepäckraum_' -'Georgien_' -'Geo' -'Gender_' -'Gegenstände_' -'Gefolge_' -'Garantien_' -'Gan' -'Galileo_' -'Freund_' -'Freund' -'Freiheits' -'Frauen' -'Franzosen_' -'Francisco_' -'Forscher_' -'Flugzeug' -'Flor' -'Flo' -'Fleisch_' -'Five_' -'File_' -'Fett' -'Festplatte_' -'Fern' -'Feed' -'Fas' -'Farbe_' -'Fang' -'Fahrrad' -'Extra' -'Exporte_' -'Explorer_' -'Ever' -'Euros_' -'Erz' -'Erst_' -'Erst' -'Erreichung_' -'Erinnerung_' -'Ergebnissen_' -'Englisch_' -'Energiequellen_' -'Energieeffizienz_' -'Enc' -'Emi' -'Elevator_' -'Einsatz' -'Eines_' -'Effekte_' -'Eco' -'EU' -'ECH' -'EB' -'Dubai_' -'Dro' -'Drittländern_' -'Dos' -'Domin' -'Display_' -'Director_' -'Dir' -'Dingen_' -'Dilemma_' -'Diesel' -'Diagnose' -'Desktop_' -'Den' -'Dem' -'Defizit' -'Dau' -'Darstellung_' -'Danish_' -'DAT' -'Custom' -'Crespo_' -'Create_' -'Count' -'Corporation_' -'Consider_' -'Computers' -'Computer' -'Clearly_' -'Cle' -'Christmas_' -'Christine_' -'Chin' -'Child_' -'Cher' -'Chen' -'Chaos_' -'Change_' -'Chancengleichheit_' -'Cent' -'Celsius_' -'Cardassian' -'Cannes_' -'Cala_' -'Cai' -'CU' -'CR' -'COM_' -'CC_' -'CB' -'Bü' -'Brust' -'Brief_' -'Brennstoffe_' -'Brennstoff' -'Branche_' -'Brad' -'Botschaften_' -'Borg_' -'Boo' -'Blue' -'Bilanz_' -'Bil' -'Bier' -'Bezahlung_' -'Betreiber' -'Bestimm' -'Beste' -'Besorgnis_' -'Beschäftigten_' -'Beschr' -'Bern' -'Bericht' -'Bereits_' -'Beobachtung' -'Behörde_' -'Bef' -'Balkon_' -'Bai' -'Bag' -'BM' -'Ayatollah_' -'Ay' -'Autor_' -'Austr' -'Auslands' -'Ausf' -'Ausbruch_' -'Ausbeutung_' -'Aufklärung_' -'Aufhebung_' -'Aufenthalts' -'Att' -'Atomwaffen_' -'Ash' -'Arzneimittel' -'Arti' -'Area_' -'Architekt' -'Arbeitsplätzen_' -'Arbeitsmarkt' -'Ara' -'Apo' -'Anspr' -'Ansehen_' -'Anschläge_' -'Anschl' -'Annan_' -'Ank' -'Anfragen_' -'Andre' -'Anbindung_' -'Among_' -'Ameri' -'Alpen_' -'Alp' -'Allow_' -'Allah_' -'Albert_' -'Albani' -'Aktions' -'Adria' -'Abu_' -'Abteilung_' -'About_' -'Abk' -'Abb' -'AVI_' -'AT_' -'AM_' -'AF' -'ACP_' -'?”_' -'== _' -'="_' -': '_' -'99_' -'93_' -'77_' -'76_' -'68' -'65' -'64' -'58_' -'53_' -'51' -'47' -'350_' -'1st_' -'1956_' -'1930_' -'150' -'.&#_' -'. )' -'. "_' -'--' -'- (_' -', '_' -'); _' -'( _' -'% ' -'!!_' -' – ' -' »_' -' «_' -' - ' -' * _' -' %._' -' %' -'€' -'ә' -'ғ' -'ір' -'э' -'ты' -'ки' -'жа' -'ен' -'ге' -'г' -'ал' -'а_' -'α_' -'α' -'ğ' -'č' -'ültig' -'ückte' -'ücher_' -'überz' -'überwachen_' -'überst' -'übermäßige' -'überge' -'überd' -'ø' -'öße' -'ötig' -'ösen_' -'ökonomische_' -'öh' -'öf' -'ín' -'éta' -'ér' -'ça_' -'å_' -'å' -'äußerster_' -'ärkte' -'är_' -'änglich' -'änger' -'änd' -'ällig' -'äft' -'ächtig' -'án' -'ßig' -'ße_' -'Äußerungen_' -'Änderungsanträgen_' -'®' -'« _' -'«' -'}} ' -'zügig' -'züge' -'zz' -'zusammenbr' -'zurzeit_' -'zum' -'zukünftige_' -'zuh' -'zogen_' -'zl' -'zier' -'zia' -'zh' -'zerstör' -'zers' -'zeichne' -'zei' -'zehn' -'zan' -'ypt' -'yl_' -'yacht' -'xp' -'wünscht_' -'wünschenswert_' -'wäch' -'wusste' -'wur' -'worte' -'word' -'wohin_' -'woch' -'wn_' -'wn' -'with' -'wit' -'wisdom_' -'wirk' -'wines_' -'wettbewerbs' -'wett' -'wes' -'wertung_' -'wellness_' -'wellbeing_' -'weißen_' -'weiterzu' -'weitem_' -'watching_' -'watch' -'wasn_' -'warf' -'wahre_' -'wahl' -'wachsende' -'vorschl' -'vornehmen_' -'vorliegt_' -'vorgeschrieben' -'voranzutreiben_' -'voneinander_' -'volunteers_' -'vollz' -'vol' -'virtuelle' -'vin_' -'vielf' -'vie' -'vi_' -'veto_' -'veränderte' -'verweisen_' -'vertrete' -'verständlich_' -'versi' -'verschwinden_' -'verschle' -'verschieden_' -'verschieben_' -'verschie' -'vermieden_' -'vermi' -'verlängern_' -'verlust' -'verletz' -'verlager' -'verkehrs' -'verha' -'verglichen_' -'vergleich' -'verfassungs' -'verdienen_' -'verdi' -'verbindlich_' -'verbesserten_' -'verantwort' -'veran' -'variations_' -'uve' -'usse' -'usel' -'usch' -'urm' -'url' -'uring_' -'urgency_' -'ups_' -'upcoming_' -'uous_' -'unzählige_' -'unweit_' -'unwahrscheinlich_' -'untersucht_' -'unterm' -'unpro' -'unmittelbaren_' -'unmi' -'unknown_' -'univers' -'unhe' -'ungssystem' -'ungsre' -'ungan_' -'unft' -'unfair_' -'unen' -'uneingeschränkt_' -'understands_' -'underscore' -'unbr' -'unanimously_' -'ume' -'ulen' -'ule' -'uer_' -'ud_' -'tür' -'té' -'tungen_' -'tum' -'tub' -'ttel' -'tron' -'tremendous_' -'treatments_' -'traten_' -'transmission_' -'transit' -'transformed_' -'transc' -'transatlantischen_' -'transactions_' -'transaction_' -'trains_' -'trad' -'tou' -'total' -'tory_' -'tolle' -'todo' -'tod' -'tive' -'tischer_' -'tire' -'tir' -'tieren_' -'tiefe_' -'tib' -'throw' -'thor' -'tho' -'tge' -'teuer' -'terminal_' -'teria' -'tens' -'tende' -'tema' -'technische' -'techni' -'tc' -'tasty_' -'tar_' -'tankers_' -'talent_' -'tak' -'tai' -'tage_' -'tage' -'systematically_' -'symbolic_' -'sym' -'sy_' -'suspension_' -'surpluses_' -'surg' -'supplement' -'suff' -'sudah_' -'subway_' -'substan' -'stück_' -'stö' -'stuff_' -'stuf' -'studie' -'struktur_' -'stro' -'strikte' -'strikes_' -'stretch' -'streng_' -'straight_' -'straf' -'stoffe_' -'stle' -'stieg_' -'stick_' -'stet_' -'stes_' -'steam_' -'steadily_' -'stea' -'statt' -'statisti' -'stamm' -'stakes_' -'stabile_' -'ssen' -'sprachliche' -'sprachen_' -'spell' -'speed' -'speeches_' -'spectacular_' -'specify_' -'sound' -'soul_' -'sorgfältige' -'sorgfältig_' -'sophisticated_' -'solved_' -'sogenannten_' -'sofa_' -'smi' -'smart_' -'slowly_' -'sive_' -'sinnvolle' -'simpli' -'sim' -'sili' -'signing_' -'signature_' -'sierungs' -'sierte' -'sieg' -'sie' -'sic' -'shut_' -'shortcomings_' -'ship' -'shifts_' -'shee' -'sge' -'sexuelle_' -'settled_' -'sema' -'secondly_' -'seba' -'scrutin' -'screening_' -'scrap' -'schwächere' -'schwäch' -'schwedischen_' -'scholar' -'schließt_' -'sches_' -'schauen_' -'sca' -'saubere' -'satisfied_' -'sak' -'safeguard_' -'rückg' -'rö' -'räge_' -'rui' -'ruh' -'ruch_' -'rter' -'roo' -'romantic_' -'roman' -'roa' -'rnähr' -'rna' -'rm_' -'rka' -'ritten_' -'risiko_' -'rige_' -'richtet_' -'ria_' -'rhe' -'reward_' -'revive' -'revers' -'reveal_' -'returning_' -'retro' -'retr' -'restoration_' -'ress' -'responded_' -'residents_' -'reside' -'reproduc' -'repr' -'repli' -'repa' -'renov' -'renminbi_' -'removing_' -'remind' -'remark' -'relocat' -'reinforce_' -'reiche_' -'register_' -'regierung_' -'regelmäßig_' -'refusal_' -'redistribution_' -'recon' -'recht' -'rechnen_' -'receives_' -'reben_' -'realistische' -'realisier' -'realised_' -'reader_' -'rator' -'ratifiziert_' -'ratification_' -'rapide_' -'ranks_' -'rank' -'rall' -'rahmen' -'ract' -'quoten_' -'quisit' -'quir' -'quie' -'ques_' -'quanti' -'qualifizierte_' -'qualifications_' -'pushed_' -'pup' -'präsentier' -'prozesses_' -'prove' -'protektionistische' -'prosperous_' -'proof_' -'prol' -'projekt_' -'programmen_' -'professionelle' -'prob' -'prize_' -'privilege_' -'printing_' -'preventive_' -'prevail' -'prestigious_' -'preserved_' -'presentation_' -'prescri' -'premature_' -'pragmatic_' -'potenziellen_' -'poss' -'poses_' -'pollut' -'pole' -'polar_' -'po_' -'plural' -'pill' -'pier' -'philosophy_' -'phas' -'pflege' -'pfl' -'pfel' -'pfe_' -'petit' -'pes_' -'persönliche' -'persuade_' -'persone' -'persist' -'perpet' -'permi' -'periode_' -'pere' -'perat' -'pensions_' -'penda' -'pemba' -'pel' -'peacekeeping_' -'patient_' -'passes_' -'partitions_' -'parlamentarische_' -'parity_' -'paren' -'papers_' -'panel_' -'pana' -'painting_' -'own' -'owe_' -'overw' -'oversight_' -'overe' -'outs_' -'outs' -'outl' -'oti' -'ote_' -'oste' -'osen_' -'osc' -'ori' -'operates_' -'opens_' -'openly_' -'opa' -'onym' -'onat' -'onal_' -'ome_' -'ologie_' -'ologi' -'oil' -'ofi' -'offensichtliche' -'odie' -'ock_' -'ochen_' -'och_' -'occupation' -'oca' -'obst' -'observation_' -'obli' -'ober' -'nzi' -'nutrition_' -'ntw' -'nsu' -'nous_' -'nost' -'nos' -'nomin' -'noi' -'nnt_' -'nner' -'nne_' -'nlage' -'nko' -'niu' -'nity_' -'nio' -'nik_' -'nightlife_' -'nien_' -'niedrigere' -'niedrige_' -'ngst' -'ngn' -'nglich_' -'ngel' -'neuro' -'neuesten_' -'nett' -'ness' -'nesian' -'ners_' -'neglig' -'neglect_' -'negativ_' -'necessity_' -'ndete' -'nden' -'naval_' -'namens_' -'nah_' -'nada_' -'männ' -'mutige' -'musik_' -'multilateralen_' -'mpin' -'mpf' -'mp_' -'movies_' -'mov' -'mounted_' -'mortgage_' -'monument' -'moment' -'modules_' -'moderni' -'modern' -'mixture_' -'mitge' -'missi' -'mism' -'misc' -'mir' -'minist' -'mining_' -'minimi' -'mina' -'militärisch' -'militant' -'meta' -'mengu' -'mengak' -'menga' -'membu' -'med' -'mechanismen_' -'measurement_' -'matche' -'master' -'massiv_' -'massage_' -'marine_' -'marginal_' -'mapp' -'mali' -'makroökonomische' -'mak' -'mai' -'mah' -'magne' -'magazine_' -'ländliche_' -'lve' -'lv' -'lungs' -'lue' -'luc' -'lua' -'lu_' -'loses_' -'logo_' -'loc' -'lobb' -'lize' -'liza' -'litik' -'liti' -'literar' -'listening_' -'liste' -'list' -'limiting_' -'liebe_' -'licherweise_' -'license_' -'lib' -'lf' -'leute_' -'letting_' -'lending_' -'lem_' -'leitung_' -'leitete_' -'leitet_' -'leis' -'legislat' -'leere' -'lect' -'laute' -'lati' -'latein' -'lager_' -'künstlich' -'käm' -'kta' -'kritisch' -'kris' -'kre' -'kopieren_' -'kontrollierte' -'kontrolle_' -'konkret_' -'konflikt' -'komplette_' -'kommunistische_' -'kolle' -'kenne_' -'kebijakan_' -'karten_' -'kampf' -'kam' -'kali' -'kale' -'kai' -'jüngere' -'jährliche_' -'judgment_' -'judge_' -'judge' -'jegliche' -'jed' -'jas' -'itze' -'itution_' -'ities_' -'istisch' -'ister' -'issen_' -'irt' -'irgendwann_' -'iranischen_' -'ira_' -'ip_' -'iot' -'io' -'invade' -'interpreti' -'interprete' -'interim_' -'interessanten_' -'intellektuelle' -'integrierten_' -'integrieren_' -'integral' -'institutionelle_' -'installer_' -'insch' -'inner_' -'inj' -'inie' -'inglich' -'ingen' -'infringement_' -'informati' -'influential_' -'ineffective_' -'industrielle_' -'induce' -'incredibly_' -'inan' -'imposing_' -'immer' -'ily_' -'illo' -'ildung_' -'ihrerseits_' -'ignoriert_' -'ignorieren_' -'ift' -'iete' -'iere_' -'iere' -'ielen_' -'iehen_' -'iegel' -'idor' -'identical_' -'idealer_' -'icul' -'ichte' -'ices_' -'ican' -'ias_' -'hü' -'hôtel_' -'hunting_' -'hunderte' -'hrung_' -'hre' -'hosted_' -'hospitality_' -'hone' -'holen_' -'hol_' -'hochwertige_' -'hne_' -'hme' -'hm_' -'hle' -'historisch_' -'his' -'hinzugefügt_' -'hint' -'hil' -'hike' -'hierzu_' -'hic' -'heutzutage_' -'hersteller_' -'heri' -'here' -'herausragende' -'height_' -'hearts_' -'health' -'heading_' -'hbo' -'haushalt' -'has' -'harg' -'hard' -'handled_' -'handed_' -'ham' -'hall' -'halb' -'hafte_' -'had' -'habit' -'gutem_' -'gten_' -'greift_' -'grasp_' -'good' -'gni' -'gn_' -'globale' -'gleicher_' -'git' -'gewünschten_' -'gespe' -'geru' -'gericht_' -'gerei' -'gerechter' -'gerechten_' -'geprägt_' -'gepflegt' -'geographical_' -'gent' -'gena' -'gemäßigte' -'gelo' -'geklärt_' -'geist' -'gehandelt_' -'gehabt_' -'gegens' -'geblieben_' -'gebeten_' -'geber' -'gation_' -'ganis' -'gangen_' -'fäll' -'fä' -'fut' -'fus' -'fung_' -'friedliche_' -'freundlicher_' -'freu' -'freez' -'fossile' -'formula_' -'formats_' -'formally_' -'forever_' -'fores' -'foods_' -'fond' -'fließen_' -'fl' -'finnische' -'findings_' -'fig' -'fiel_' -'fi_' -'ffs' -'ffen_' -'fet' -'fern' -'fer_' -'fel_' -'feind' -'fehler' -'federa' -'favourite_' -'favour' -'fassen_' -'fascinating_' -'fantas' -'fals' -'fairen_' -'faire_' -'fahrzeuge_' -'facility_' -'eßen_' -'extremism_' -'externen_' -'externe_' -'express' -'explicit_' -'expertise_' -'experimental_' -'exhibition_' -'executi' -'executed_' -'excluded_' -'exclude' -'ewe' -'evol' -'evo' -'eventuelle' -'eve' -'etzt_' -'eter_' -'ete' -'este_' -'esan' -'erweiterten_' -'erwe' -'ervi' -'ertrag' -'erstklassige' -'erson' -'erschienen' -'errichtete' -'errichten_' -'erra' -'erp' -'ernste_' -'erneuer' -'ermutigen_' -'erla' -'erka' -'erin' -'erheben_' -'ergibt_' -'erfü' -'erarbeiten_' -'entscheide' -'entl' -'ensu' -'enm' -'eng' -'ened_' -'enco' -'emphasis' -'emm' -'els' -'ellt_' -'elit' -'eliminated_' -'elf_' -'einse' -'einsch' -'eins' -'einmalige_' -'einig' -'eingereicht_' -'eingeg' -'eingeb' -'einf' -'einbr' -'einbezogen_' -'eigentlichen_' -'eigentliche_' -'eich' -'egel' -'effektiver_' -'ee' -'ecological_' -'echn' -'eche' -'ebu' -'eat' -'earn' -'durchd' -'dunkle' -'duc' -'drittens_' -'dre' -'draw' -'drag_' -'dos' -'donat' -'dominate_' -'dlich' -'dle_' -'disturb' -'distribut' -'distr' -'diss' -'disrupti' -'disi' -'discretion_' -'disappear' -'dip' -'dina' -'digitalen_' -'digitale_' -'dig_' -'differently_' -'dier' -'dich_' -'dge_' -'dge' -'devastating_' -'dete' -'destabili' -'desp' -'design' -'dero' -'derjenigen_' -'derive_' -'dera' -'deposit_' -'deport' -'deployment_' -'deployed_' -'denjenigen_' -'demonstrators_' -'demographic_' -'delle_' -'delicate_' -'defending_' -'defect' -'defeat' -'declined_' -'declare_' -'declar' -'decken_' -'dad_' -'dac' -'cus' -'cultures_' -'cts_' -'crystal' -'cry_' -'courage_' -'coordinat' -'convey' -'convert' -'conv' -'controller_' -'contaminated_' -'consistently_' -'cong' -'conf_' -'comput' -'compulsory_' -'complaints_' -'competent_' -'competences_' -'compatibility_' -'command' -'column' -'colo' -'collected_' -'clothing_' -'clim' -'clicking_' -'clarification_' -'citizenship_' -'cil' -'cigarette' -'cien' -'cial_' -'chung_' -'chte_' -'chs_' -'chronische' -'chro' -'christliche' -'chlich' -'chlag' -'chir' -'chic' -'chemische_' -'chee' -'chart' -'chairman_' -'chafts' -'ces' -'cent_' -'cen' -'cement_' -'cea' -'carrier_' -'cap' -'camp' -'cali' -'burned_' -'bung_' -'buch' -'bt' -'brother_' -'broker' -'breiter_' -'breast_' -'bran' -'brachten_' -'boy_' -'boom' -'booked_' -'blow_' -'blis' -'blick_' -'bless' -'blame_' -'bili' -'bike_' -'bien_' -'bewährte' -'beweist_' -'bewaffneten_' -'bevorzugte' -'betragen_' -'betrachte_' -'bet_' -'beständig' -'bestre' -'beschrieben_' -'beschleunigt' -'besagt_' -'berl' -'berichtet_' -'bereitgestellt_' -'bereich' -'berechtigt_' -'beneficial_' -'benefi' -'benannt' -'benachrichtigt_' -'bemerkt_' -'beliebtes' -'bekämpfung_' -'behörden_' -'befürwortet_' -'befreien_' -'befehl' -'befasst_' -'beeinflusst_' -'bedingungen_' -'bede' -'bedanken_' -'beck_' -'beberapa_' -'bbi' -'baut_' -'bathing_' -'bath' -'basically_' -'banyak_' -'bankr' -'bahan_' -'azi' -'aya_' -'automatic_' -'auto_' -'authorise' -'auszusch' -'auswärtige_' -'ausschl' -'ausreichende_' -'ausreichen_' -'ausgi' -'ausgel' -'aufweisen_' -'aufste' -'aufrecht_' -'aufges' -'auffordern_' -'auber' -'attribute' -'attending_' -'attacking_' -'attached_' -'attach' -'ats_' -'atis' -'atic_' -'aster' -'aste' -'assessments_' -'arts_' -'armo' -'ark' -'aries_' -'ards_' -'archa' -'arbeiter_' -'approval_' -'approaching_' -'appointment_' -'ape' -'anzus' -'anzug' -'anxiety_' -'anu' -'anticipated_' -'anschließend_' -'anschl' -'ansa' -'anr' -'annually_' -'annten_' -'anle' -'anhalten' -'angka' -'anger' -'angenehme' -'anfangen_' -'ando' -'andern' -'ande' -'ances_' -'analyze' -'analyst' -'amtierende' -'ame' -'ambitions_' -'amazing_' -'albeit_' -'alarm' -'ala_' -'akzept' -'aktionen_' -'airlines_' -'ahren_' -'ahr' -'ahl_' -'agte' -'aging_' -'affi' -'advocate_' -'advice_' -'adidas_' -'adhere' -'adapti' -'adapted_' -'adapt' -'acy_' -'actively_' -'act' -'acr' -'acquired_' -'acknowledg' -'achts' -'accura' -'accountable_' -'accountability_' -'accomplished_' -'accidents_' -'accident_' -'academic_' -'ac' -'abzulehnen_' -'absolute' -'ablehnen_' -'abh' -'abges' -'abe' -'abba' -'abb' -']] | _' -'Zwischenzeit_' -'Zwei_' -'Zustellbetten_' -'Zusammenfassung_' -'Zusagen_' -'Zugriff' -'Zuge' -'Zimmerbeschreibung_' -'Zertifikat' -'Zeitungen_' -'Zeitplan_' -'Zauber' -'Zahlreiche_' -'Young_' -'Xi' -'XML_' -'Wur' -'Wu' -'Wohnungs' -'Wissenschaftlern_' -'Wirtschaftss' -'Wirtschaftsr' -'Wirtschaftsm' -'Wirtschaftskrise_' -'Wirtschaftsa' -'Wirtschaftlich' -'Winter' -'Wilhelm_' -'Wiederg' -'Wichtig_' -'Who' -'Werte' -'Weltraum' -'Wellness' -'Well_' -'Weiteren_' -'Wechselkurs' -'Wachstumspakt' -'WO' -'WM' -'Voyager_' -'Votum_' -'Vorsorge' -'Vorgänger' -'Vordergrund_' -'Vorbereitungen_' -'Visionen_' -'Vin' -'Village_' -'Vil' -'Viertens_' -'Vid' -'Verwe' -'Vertretung_' -'Versorgungs' -'Versicherungen_' -'Versi' -'Verr' -'Verordnungen_' -'Vern' -'Veri' -'Verheugen_' -'Vergnügen_' -'Vereinigung_' -'Verbrauch_' -'Verantwortungs' -'Ven' -'VP_' -'Ut' -'Untersuchungs' -'Unterricht' -'Unterdrückung_' -'Unsere' -'Unruhen_' -'Unr' -'Unm' -'Unlike_' -'Ungl' -'Una' -'Umstrukturierung_' -'UNESCO_' -'Trä' -'Tower_' -'Touristen_' -'Tourismus' -'Tos' -'Tools_' -'Tom' -'Together_' -'Tisch_' -'Tickets_' -'Throughout_' -'Through_' -'Thro' -'Thi' -'Theater_' -'Terror_' -'Tempora' -'Tel_' -'Tech_' -'Tea' -'Taxi' -'Taten_' -'Tap' -'Tao' -'Tak' -'Tag' -'Tabelle_' -'TU' -'TC' -'Sü' -'Szen' -'Swoboda_' -'Sun_' -'Suche' -'Subject_' -'Stü' -'Sturm_' -'Stri' -'Streben_' -'Stiftung_' -'Steuerung_' -'Sternen' -'Staatsverschuldung_' -'Staatsschulden_' -'Staatss' -'Spiels' -'Spezie' -'Speci' -'Spani' -'Spaltung_' -'Sp' -'Sometimes_' -'Solid' -'Solche_' -'Solarium_' -'Solar' -'Soci' -'Sm' -'Slow' -'Six_' -'Sisko_' -'Single_' -'Singapur_' -'Similar_' -'Signale_' -'Shops_' -'Server' -'Series_' -'Sende' -'Senate_' -'Senat' -'Semi' -'Scien' -'Schöne' -'Schweizer_' -'Schre' -'Schlu' -'Scha' -'Say' -'Saturday_' -'SanDisk_' -'Sam' -'Salzburg_' -'ST_' -'STO' -'STE' -'ST' -'Rüstungs' -'Rule_' -'Rohstoff' -'Rock' -'Robert' -'River' -'Rica_' -'Republikaner_' -'Repo' -'Rental_' -'Renn' -'Rem' -'Rekord' -'Registrierung_' -'Regions_' -'Reduc' -'Rede' -'Recent_' -'Rea' -'RS_' -'RI_' -'RC_' -'Quo' -'Quest' -'QE_' -'Putins_' -'Publik' -'Prävention_' -'Program_' -'Political_' -'Polit' -'Poi' -'Plu' -'Platz' -'Picard_' -'Photo' -'Philip' -'Phasen_' -'Pflege_' -'Petr' -'Pet' -'Pes' -'Personen' -'Perl' -'Pent' -'Peer_' -'Patt' -'Patent' -'Patch' -'Parlament' -'Parks_' -'Parag' -'Palästina_' -'Palacio_' -'Paar_' -'PXI_' -'PM' -'PH' -'PDF_' -'Ozean' -'Ot' -'Oslo_' -'Original_' -'Ordner_' -'Ora' -'Operation_' -'Online' -'Ombudsman_' -'Olympus_' -'Og' -'Offizier' -'Obamas_' -'ODER_' -'Nue' -'Nova' -'Normal' -'Nob' -'Nik' -'Niger' -'Nichts_' -'Neue_' -'Netzwerke' -'Neigung' -'Nehmen_' -'Nan' -'Nahrungsmittel_' -'Nachrichten' -'Nachfolger_' -'Nachf' -'NP' -'NH' -'NF' -'Mü' -'Möglich' -'Männern_' -'Mund' -'Multi_' -'Mountain' -'Motion_' -'Mord_' -'Monats_' -'Moldova_' -'Moham' -'Modul_' -'Mode' -'Mod' -'Mittag' -'Mitgliedsländer_' -'Mitentscheidung' -'Michel_' -'Messen' -'Messe_' -'Mes' -'Mem' -'Meinungsumfragen_' -'Medi' -'Mea' -'McC' -'Mazedonien_' -'Maus' -'Materialien_' -'Marco_' -'Map' -'Manu' -'Manche_' -'Male' -'Make_' -'Maha' -'MH' -'ME_' -'MED' -'Lösungs' -'Länge_' -'Län' -'Luxus_' -'Lula_' -'Lui' -'Luftverschmutzung_' -'Luftraum' -'Low_' -'Louis_' -'Look_' -'Logo_' -'Loc' -'Lobby_' -'Lis' -'Linke_' -'Lif' -'Leuten_' -'Lesung_' -'Leone_' -'Lektion_' -'Leiden' -'Leid_' -'Leader' -'Lauf' -'Last' -'Lands' -'Lah' -'LT' -'LD' -'Körper' -'König' -'Kurd' -'Kreis' -'Kre' -'Kontinents_' -'Kontin' -'Konsumenten_' -'Konst' -'Konflikts_' -'Konferenz' -'Kom' -'Kolonial' -'Koll' -'Kohle' -'Know_' -'Know' -'Klin' -'Keynesian_' -'Kel' -'Kein_' -'Kau' -'Kath' -'Kapitalismus_' -'Kanal' -'Kam' -'Kaffee' -'Kabel_' -'KON' -'Juli' -'Jugendlichen_' -'Jud' -'Jews_' -'Jakob' -'Ist' -'Islamist_' -'Ir' -'Investment_' -'Intera' -'Instanz_' -'Instan' -'Installations' -'Inhalte_' -'Inflations' -'Industriep' -'Impl' -'Impf' -'Immunität_' -'Imagin' -'Illusion' -'Il' -'IV_' -'ISIS_' -'IN_' -'ICEcat_' -'IA_' -'Höhen' -'Häufig' -'Häfen_' -'Hyatt_' -'Hop' -'Hollywood_' -'Holland_' -'Hohen_' -'Hisbollah_' -'Hinweise_' -'Hinter' -'Hindernis_' -'Hil' -'High' -'Herz_' -'Herrsch' -'Herbst_' -'Her_' -'Hen' -'Heinrich_' -'Hat_' -'Hard' -'Hannover_' -'HI_' -'Guinea_' -'Guatemala_' -'Grupp' -'Greens_' -'Greeks_' -'Glücklicherweise_' -'Gibt_' -'Gewinner_' -'Gewinn_' -'Gewalt' -'Gesetzes' -'Gerichten_' -'Generalsekretär_' -'Genau_' -'Gemeinw' -'Gemeinschaftsp' -'Gemein' -'Geiste' -'Gegenwärtig_' -'Gegenwart_' -'Gee' -'Gaz' -'Gaulle_' -'Ganze_' -'Gall' -'G20_' -'Förder' -'Föderation_' -'Fußball' -'Funktionsweise_' -'Fundamental_' -'Frie' -'Freundschaft_' -'Freitag_' -'Freihandelsabkommen_' -'Freihandels' -'Franklin_' -'Format' -'Folgendes_' -'Folge' -'Flug_' -'Flexibilität_' -'Fis' -'Find' -'Finance_' -'Fehler' -'Fee' -'Fau' -'Fass' -'Fam' -'Fakten_' -'Fahrer_' -'Face' -'FM' -'FD' -'Explosi' -'Experience_' -'Evans_' -'Eva' -'Eu' -'Ethi' -'Estonia_' -'Esc' -'Erwärmung_' -'Erste' -'Erscheinung_' -'Erl' -'Erkenntnis_' -'Erfind' -'Erdgas' -'Erbe_' -'Entschlossenheit_' -'Entr' -'Entlastung_' -'Ens' -'End_' -'Empfänger_' -'Emp' -'Elf' -'Electronic' -'Einzelnen_' -'Einw' -'Einladung_' -'Einigkeit_' -'Eigentümer' -'Ehre_' -'Ehr' -'Ecuador_' -'Ebenen_' -'EWG_' -'ENT_' -'Dé' -'Dusche_' -'Durch' -'Dur' -'Drei_' -'Dre' -'Donnerstag_' -'Dona' -'Don' -'Disabled_' -'Differenzen_' -'Differenz_' -'Diesen_' -'Diag' -'Demonstranten_' -'Demokrati' -'Deflation_' -'Defense_' -'Deck' -'Davi' -'Darau' -'DP' -'Cup_' -'Crew_' -'Countries_' -'Cooperation_' -'Converter_' -'Continental_' -'Cont' -'Congo_' -'Conce' -'Compa' -'Come' -'Colon' -'Citizens_' -'Cin' -'Christ' -'Chr' -'Chicago_' -'Chemi' -'Cer' -'Cau' -'Carolin' -'Carlo_' -'Car_' -'CEO_' -'CDs_' -'Bühne_' -'Bushs_' -'Bull' -'Bulgarian_' -'Buchung_' -'Brut' -'Browser_' -'Brit' -'Boutique_' -'Bot' -'Boh' -'Binnenmarktes_' -'Bin' -'Beweg' -'Betrug_' -'Betroffenen_' -'Besteuerung_' -'Beschwerden_' -'Besatzung_' -'Bernanke_' -'Berliner_' -'Belo' -'Bekanntlich_' -'Beit' -'Beförderung_' -'Beamte_' -'Basic_' -'Bara' -'BL' -'Azer' -'Autor' -'Australien_' -'Austausch_' -'Ausgaben' -'Ausb' -'Augenmerk_' -'Auftr' -'Aufs' -'Aufr' -'Assa' -'Asiens_' -'Asi' -'Archer_' -'Arbeitss' -'Arbeitskräften_' -'Arabischen_' -'Arabien_' -'Arabi' -'Araber_' -'Appe' -'Anstatt_' -'Anna' -'Anh' -'Angola_' -'Ander' -'Amts' -'Amm' -'American' -'Alternativen_' -'Alpe' -'Akzeptanz_' -'Aktionsplan_' -'Akte' -'Airways_' -'Addi' -'Achtung_' -'Account' -'Abz' -'Abst' -'Abs' -'Able' -'Abgesehen_' -'Abfall' -'Abenteuer_' -'Abbau_' -'AND_' -'AG' -'AB_' -'A6_' -'>' -'==' -'87_' -'83_' -'76' -'74' -'69_' -'66' -'44' -'41' -'38' -'360_' -'1988_' -'1984_' -'1980er_' -'198' -'1978_' -'1975_' -'1970er_' -'1940_' -'1920' -'178' -'002' -') ._' -'%_' -'") _' -' “ _' -' » _' -' ``_' -' = [[_' -' --> _' -'…' -'”)' -'ү' -'ын' -'ыл' -'ті' -'ти' -'са' -'рі' -'ро' -'ре' -'ос' -'ле' -'ке' -'ес' -'да_' -'бол' -'ба' -'ай' -'Б' -'ι' -'üs' -'ürd' -'ündig' -'üllung' -'ühr' -'üge' -'ückt_' -'übt_' -'üblichen_' -'überzeugende' -'überwacht_' -'überschreiten_' -'überleben_' -'überh' -'überb' -'öss' -'öse_' -'ór' -'ña' -'ê' -'éri' -'äußere' -'äuft_' -'ätzung_' -'ätte' -'ärk' -'älteste_' -'älter' -'ält' -'äisch' -'ähnlichen_' -'ächte' -'äc' -'ä_' -'â' -'ár' -'ße' -'Übersetzungs' -'Überschuss_' -'Übereinkunft_' -'° _' -'}} _' -'{' -'zza_' -'zwölf_' -'zweite' -'zusätzliche' -'zustellen_' -'zustande_' -'zurückz' -'zurecht' -'zukünftig' -'zufrieden' -'zte' -'zones_' -'zon' -'zis' -'zip_' -'zio' -'zil' -'zig_' -'zierungs' -'zielen_' -'ziel' -'zeug_' -'zersch' -'zent' -'zeitge' -'zauber' -'yo_' -'yn_' -'xy' -'wäh' -'wunderschönen_' -'writ' -'worauf_' -'wooden_' -'wm' -'witnessing_' -'witnessed_' -'withdraw_' -'wissens' -'wirtschaftspolitische' -'wirtschaftliches_' -'wirt' -'winner_' -'wind' -'width_' -'widerspiegelt_' -'wi_' -'whol' -'westlich_' -'wertvollen_' -'werten_' -'wert' -'weltweite' -'welle' -'welcomed_' -'weiß' -'weich' -'wede' -'wed_' -'wechseln_' -'weakened_' -'weak' -'wd' -'wartet_' -'warned_' -'wan_' -'wage' -'wachstums_' -'vr' -'vorschlägt_' -'vorschläge_' -'vorsch' -'vorr' -'vorherige_' -'vorgegeben' -'vorgeb' -'vorg' -'vora' -'volks' -'void' -'vity_' -'visits_' -'visitor_' -'violations_' -'violation_' -'ville_' -'viewed_' -'view' -'vierte' -'vet' -'veröffentlichten_' -'verwu' -'verwirklichen_' -'verwendeten_' -'verträge' -'vertraut_' -'vertrauen_' -'vertrag_' -'verteilen_' -'verstr' -'versammlung_' -'verp' -'vernichte' -'vermögen_' -'verlä' -'verliehen_' -'verheerende' -'vergi' -'vereint_' -'vereinfacht' -'verde' -'verbreiteten_' -'verbreite' -'verbrauche' -'verbrauch' -'verbl' -'verbindung' -'verarbeitung' -'verantwortungsvolle' -'verabschieden_' -'vention' -'veness_' -'vegetables_' -'vat' -'vary_' -'variant' -'valley_' -'uß_' -'uß' -'utzte' -'utiliz' -'utilit' -'uten_' -'usu' -'uropa_' -'uro' -'urg_' -'urce' -'uran' -'ural' -'upt' -'upgrading_' -'upgraded_' -'updates_' -'unvermeidlich_' -'unv' -'unum' -'unterliegt_' -'unsp' -'unmittelbare_' -'unm' -'unli' -'unl' -'unktion' -'unkt' -'unkonventionelle' -'unklar_' -'unk' -'universelle' -'unif' -'ungsw' -'ungsr' -'ungsprogramm' -'ungsmaßnahmen_' -'ungsk' -'ungleiche' -'unfa' -'unein' -'undertaking_' -'undertaken_' -'understandable_' -'underp' -'undermining_' -'uncon' -'unch' -'unbekannte' -'umu' -'umi' -'umfassende' -'umer' -'umbu' -'ult_' -'uldig' -'uku' -'uin' -'uh_' -'uft_' -'uen' -'uble_' -'uber' -'uat' -'uali' -'ual' -'tzten_' -'typ' -'tv' -'turm_' -'tune' -'tu_' -'tst' -'trum' -'truck_' -'trocken' -'triggered_' -'trigger_' -'tric' -'treffen' -'treat' -'trea' -'traum' -'transportier' -'transnational_' -'translated_' -'translate_' -'transform_' -'tragische' -'traditionell_' -'traditionally_' -'tract' -'tour' -'tos_' -'tm' -'tlin' -'tipp' -'tili' -'tighte' -'tight' -'thriv' -'threatening_' -'thread_' -'thought' -'thick' -'therapie' -'theoretische' -'themes_' -'thanking_' -'testen_' -'teru' -'terk' -'terie' -'tere' -'tends_' -'tenden' -'tell' -'tel' -'teilzunehmen_' -'technisch_' -'technik_' -'teach_' -'tausende_' -'taus' -'tatsächliche_' -'tation_' -'tat_' -'tant' -'tank_' -'tangible_' -'tande' -'tand_' -'tam' -'taatliche' -'szen' -'symptom' -'sympath' -'switched_' -'swee' -'suspicion_' -'surveys_' -'superb_' -'summ' -'suit_' -'suicide_' -'sui' -'subsid' -'stärkt_' -'städte' -'stunde_' -'strukturen_' -'strongest_' -'stressed_' -'strengths_' -'strecke' -'streben_' -'stischen_' -'stige' -'stete' -'stern_' -'steri' -'stereo' -'stepp' -'stens_' -'stellten_' -'stel' -'stehende_' -'state' -'starb' -'stands' -'stair' -'stagnation_' -'stabilen_' -'sst' -'ssions' -'ssion' -'squa' -'squ' -'späte' -'spre' -'spo' -'spite_' -'spezifisch' -'speziellen_' -'spezi' -'specific' -'specialize' -'spec' -'spare_' -'soziales_' -'south' -'somewhere_' -'sociali' -'sno' -'smart' -'sman' -'slu' -'slogan' -'slo' -'slightly_' -'sleeping_' -'sku' -'situationen_' -'sions_' -'sinkende' -'simulation_' -'simi' -'signat' -'sicherer_' -'shu' -'shortly_' -'shoe' -'shelter' -'shell_' -'shame' -'shake' -'sg' -'sexuellen_' -'sex_' -'sevent' -'sessions_' -'separate' -'sep' -'sentence_' -'sende' -'sel_' -'sektors_' -'sees_' -'sections_' -'secondary_' -'secara_' -'script' -'schützt_' -'schwächen_' -'schwi' -'schwarz' -'schnelles_' -'schloss_' -'schließe_' -'schlich' -'schlechte_' -'schle' -'schic' -'schem_' -'schein' -'schei' -'schalte' -'schaftliche' -'schafft_' -'scare' -'sc' -'sba' -'sat_' -'santa_' -'sang' -'sammeln_' -'samkeit' -'salt_' -'sailing_' -'sagten_' -'safely_' -'safeguard' -'saat_' -'räng' -'räger' -'räft' -'rup' -'rud' -'rtung_' -'rteil' -'rounds_' -'ros_' -'rop_' -'rogramm' -'rod' -'robot' -'robe' -'road' -'rne' -'rke' -'riskier' -'riots_' -'rim' -'right' -'rig_' -'richte' -'rian' -'rez' -'reveals_' -'retail' -'resum' -'restrict_' -'restraint_' -'resso' -'ression' -'respektieren_' -'resiste' -'resilien' -'reservations_' -'republic' -'representing_' -'renovated_' -'reno' -'remuneration_' -'remit' -'reminiscent_' -'reminded_' -'reliability_' -'relative' -'relationships_' -'relat' -'rejection_' -'reit' -'reisen_' -'reine' -'reihe' -'reifen_' -'reichlich' -'reha' -'regen' -'regelungen_' -'rega' -'reformist' -'reformi' -'reflecting_' -'refle' -'referendums_' -'recourse_' -'reconciliation_' -'reconcile' -'reckung' -'recipient' -'rechung' -'reche' -'recal' -'rebel' -'reasonably_' -'reap' -'ream' -'realistische_' -'reale_' -'readily_' -'rbeit_' -'ratione' -'rational' -'ras_' -'ras' -'rangi' -'raises_' -'radar_' -'rachte' -'rach' -'race' -'rac' -'rable_' -'quotas_' -'quip' -'quest' -'py_' -'py' -'pursuit_' -'punish' -'pun_' -'pull' -'pte' -'psychische' -'prüf' -'präsident' -'protest_' -'protects_' -'pros' -'prophe' -'properties_' -'produkte_' -'produkt' -'produk' -'proceed' -'prisons_' -'printer_' -'prevents_' -'pretty_' -'presidents_' -'preparing_' -'premium_' -'prejudice_' -'predictable_' -'prechen_' -'preach' -'power' -'poten' -'pot_' -'possession_' -'posi' -'populär' -'popularity_' -'pop' -'polo' -'poll' -'polic' -'poker_' -'pointing_' -'pocket' -'poc' -'plätze_' -'pläne_' -'pity_' -'pis' -'pipe' -'pine' -'pilot' -'pi_' -'physici' -'pharmaceutical_' -'phan' -'pha' -'ph_' -'pflanz' -'personen_' -'personali' -'permission_' -'perception_' -'perc' -'pent' -'pene' -'pemimpin_' -'peer_' -'pedia_' -'pear' -'pazi' -'patr' -'pat_' -'participated_' -'partei_' -'park' -'panoramic_' -'panisch' -'panic_' -'ows_' -'owner_' -'owed_' -'ove' -'outset_' -'outline_' -'outcomes_' -'outbreak_' -'ound' -'otr' -'otel_' -'ota' -'ossene' -'osition' -'orth' -'orm_' -'orium_' -'orientiert_' -'organisieren_' -'ordnungsgemäße' -'ordert_' -'ord_' -'orbit' -'oppress' -'opp' -'operator_' -'opera_' -'ont_' -'onisch' -'omp' -'omin' -'olt' -'ological_' -'olge' -'ole_' -'oku' -'oin' -'oid_' -'oi_' -'ohner' -'ogi' -'offset_' -'odo' -'oci' -'obtaining_' -'obi' -'oat' -'nützliche_' -'näh' -'näch' -'nza' -'nutzt_' -'nutzbar_' -'nummer_' -'nuklearen_' -'nuestr' -'ntsch' -'nting' -'ntin' -'notification_' -'nor' -'nomi' -'noisy_' -'nna_' -'nka' -'nitte' -'nisses_' -'ning' -'niederländischen_' -'ngly_' -'ngka' -'nger_' -'nge' -'newspapers_' -'newsletter_' -'neue' -'nera' -'neo_' -'nel_' -'nein' -'neighbor' -'neigen_' -'nehm' -'negotiating_' -'negotiated_' -'nego' -'neg' -'nec' -'ndige' -'nders' -'nded_' -'ndan' -'nca' -'nba' -'navi' -'nationalist_' -'nary_' -'nal_' -'nacht' -'nachfolgenden_' -'nac' -'müt' -'mündlichen_' -'müh' -'möge_' -'mäßige' -'mäßig_' -'myster' -'mysql' -'music' -'muscle' -'mung' -'multinational_' -'multinational' -'muda' -'mt' -'mpt' -'mouse_' -'mos_' -'moon_' -'moo' -'monopol' -'monitored_' -'moni' -'modula' -'modernisier' -'mmt_' -'mmen' -'mma' -'mix' -'mitzu' -'mitteln_' -'mitteilen_' -'mitte' -'mitglieder_' -'misu' -'mischen_' -'mirror' -'ming' -'mineral_' -'mind' -'mill' -'migra' -'mien' -'midst_' -'micro_' -'methoden_' -'mesis' -'ment' -'mening' -'menge' -'memba' -'melalui_' -'mel_' -'meister' -'meistens_' -'mei' -'mehrfach_' -'mehrerer_' -'media' -'mechanismus_' -'meantime_' -'mbur' -'mayor_' -'mature_' -'master_' -'masse' -'maschinen_' -'masa_' -'market' -'marken' -'march' -'maps_' -'manufacturer_' -'manipulation_' -'manifest' -'mangelnde_' -'mangel' -'mandatory_' -'mall' -'malig' -'maler' -'mais_' -'maintain' -'mainstream_' -'magni' -'mac' -'lüge' -'lü' -'löst_' -'längst_' -'läng' -'ländische' -'länd' -'lz' -'lw' -'luxuriöse_' -'lust' -'lungen_' -'luen' -'luar_' -'ltung_' -'loud_' -'lone' -'lon' -'lokal_' -'loka' -'logisti' -'logie_' -'logg' -'locked_' -'lock_' -'lob' -'loaded_' -'ln' -'lli_' -'lively_' -'lis_' -'liquid_' -'liquid' -'likelihood_' -'lik' -'liegenden_' -'liefer' -'liederung_' -'licht_' -'libr' -'liberalization_' -'liability_' -'lfe' -'leverage_' -'level' -'lette_' -'lend_' -'leichter' -'leicht' -'legitimier' -'lec' -'lebenslange' -'lds_' -'lava' -'laud' -'lateralism' -'lasse' -'lase' -'lar_' -'langjährigen_' -'langfristige' -'lamp' -'label_' -'kürz' -'künstlerische' -'kühne' -'kör' -'kuli' -'ktions' -'kräftig' -'krä' -'kritischen_' -'kraft' -'kr' -'kos' -'korrigieren_' -'konzept' -'kontrollen_' -'kontro' -'kontinuierliche' -'kont' -'konsolidier' -'konservativen_' -'konkurr' -'konkreter_' -'kong' -'kondi' -'komplizierte' -'kommerzielle_' -'kommen' -'komfortablen_' -'komfortable_' -'kombiniert_' -'kollektiven_' -'knowing_' -'klingt_' -'klassischen_' -'klassische_' -'klasse_' -'kki' -'kis_' -'kirche_' -'kins_' -'kingdom_' -'kilometers_' -'kie' -'kg_' -'keys_' -'keu' -'kes' -'kere' -'kenn' -'kemajuan_' -'keinesfalls_' -'keinem_' -'keeps_' -'kb' -'kate' -'kapazität' -'kap' -'kalte' -'jüng' -'jährlichen_' -'just' -'jurisdiction' -'junger_' -'junct' -'jug' -'judiciary_' -'judges_' -'ju_' -'jekt' -'jak' -'jahres' -'ié' -'izier' -'ix_' -'ives_' -'iven_' -'itten_' -'ito_' -'ition' -'istung' -'ista' -'iss_' -'isla' -'isierte' -'isches_' -'irtschaft' -'irakischen_' -'ionary_' -'ional' -'ion' -'investitionen_' -'investigate_' -'invention' -'inv' -'inu' -'interp' -'interna' -'intern' -'interinstitutional_' -'interact' -'intensiven_' -'integrity_' -'integrierte_' -'integrate_' -'insurgents_' -'insure' -'insu' -'insti' -'installi' -'inspections_' -'insecurity_' -'innovat' -'inmitten_' -'inkl' -'inhalt' -'inhal' -'inhaftiert' -'inge' -'influenced_' -'inen_' -'inefficient_' -'industri' -'indispensable_' -'indirekt_' -'indebted' -'ind' -'incident_' -'inch_' -'inacti' -'improv' -'imported_' -'immune_' -'immo' -'immens' -'imm' -'imi' -'ilt' -'illusion' -'ille_' -'ilig' -'ilfe_' -'ild_' -'ild' -'ila' -'iki' -'ikan_' -'ii' -'iha' -'igung' -'igu' -'ignoring_' -'igi' -'iff' -'iet' -'iest_' -'ies' -'ierter_' -'ierende' -'ieht_' -'ieb' -'ids' -'ideologi' -'icon_' -'ico' -'ickl' -'ichtung_' -'ichtung' -'ibt_' -'iati' -'iPod_' -'iPhone_' -'höhere' -'hängige' -'händler_' -'hypo' -'hybrid_' -'hunger_' -'hung' -'humanitäre_' -'hum' -'hub' -'hu_' -'hrte_' -'hospital_' -'horror' -'hop_' -'honor' -'hon' -'holdings_' -'ho_' -'hnung' -'hlen_' -'hle_' -'hir' -'hinweg_' -'hinein_' -'hind' -'hiking_' -'hielten_' -'hidup_' -'herzustellen_' -'herunter_' -'herrliche' -'herr' -'herbei' -'herb' -'heran' -'henden_' -'hell_' -'heli' -'heiz' -'heim_' -'heed_' -'heating_' -'heart' -'hazardous_' -'harmoni' -'happi' -'handful_' -'hamper' -'hak_' -'gänger' -'guard' -'gründlich_' -'größter_' -'grundlegender_' -'grundlegend_' -'grouping' -'grobe' -'grip' -'grenzüberschreitenden_' -'grenzt_' -'grenze_' -'greifende' -'gratis_' -'graphi' -'graph_' -'got' -'gna' -'gly' -'glu' -'globally_' -'globaler_' -'glieder' -'gliche' -'glaubt_' -'glad_' -'gipfel' -'gingen_' -'ging' -'gien' -'ghter_' -'ghte' -'ggi' -'gezielte' -'gewährt_' -'gewo' -'gewisser' -'gewidmet_' -'gewer' -'gew' -'gesundheitliche' -'gestü' -'gestattet_' -'gestatten_' -'gesorgt_' -'gesetzgeb' -'geschwächt_' -'geschlossene' -'geschafft_' -'gesamt' -'gentl' -'genetische' -'generating_' -'genera' -'genauer_' -'gemütliche_' -'gemi' -'gemeinschaft_' -'geln_' -'gelegentlich_' -'gelder_' -'gelangt_' -'geko' -'gekennzeichnet_' -'gegenseitige' -'gefährlicher_' -'gefr' -'gebühren_' -'gebra' -'gebaut_' -'gases_' -'gard' -'garage_' -'gangs_' -'gabe_' -'fünfzig_' -'fühl' -'fällig_' -'fähig_' -'futures_' -'funktion_' -'funk' -'functionality_' -'frühere_' -'früh_' -'freilich_' -'freedoms_' -'frau' -'fraction_' -'fr' -'founding_' -'fought_' -'fossilen_' -'forming_' -'formen_' -'formation' -'form' -'forgotten_' -'forge' -'foreigners_' -'followers_' -'folg' -'fol' -'flood_' -'fizier' -'fixes_' -'fitt' -'fit' -'firm' -'finali' -'film' -'file' -'fh' -'fg' -'fetch' -'festzulegen_' -'fests' -'fenster_' -'felder_' -'feier' -'fehlen_' -'feels_' -'feedback_' -'fax_' -'favourable_' -'faszinierend' -'fas' -'farms_' -'far' -'familiengeführte' -'famili' -'falschen_' -'faktor' -'eye' -'extr' -'extending_' -'expulsion_' -'export' -'exploit_' -'expla' -'experiment' -'existiert_' -'exhaust' -'exemption_' -'exacerbate' -'evolution_' -'eventuell_' -'evan' -'evaluate_' -'euren_' -'eu_' -'etu' -'eto' -'etch' -'eta_' -'esten_' -'este' -'ession_' -'espa' -'esk' -'esen_' -'esen' -'erö' -'erzeug' -'erz' -'erwi' -'erweitert_' -'ert' -'ersucht_' -'ersuchen_' -'erstreckt_' -'erstre' -'erschw' -'erregend' -'erre' -'ernannt_' -'ermutigt_' -'erleb' -'erlangt_' -'erklärung_' -'erkennbar' -'erke' -'eria' -'ergänzt_' -'erge' -'erg_' -'erfolgte_' -'erfa' -'erbringen_' -'equi' -'equ' -'enw' -'entwicklung_' -'entstand_' -'entsprechende' -'entspannen_' -'entsch' -'ently_' -'entlich' -'entlassen_' -'entgegenzu' -'entertain' -'ente' -'ental' -'enor' -'eno' -'enme' -'enjoys_' -'enjoyable_' -'enheit_' -'engi' -'enforce_' -'endgültig_' -'endeavour' -'enda' -'encourages_' -'enburg_' -'employ' -'emie_' -'ement' -'embr' -'embo' -'ember' -'embargo_' -'ella_' -'ella' -'elites_' -'eligible_' -'eless' -'elektrische' -'electronic' -'elb' -'eise_' -'einzelner_' -'eint' -'einnehmen_' -'einiges_' -'eingetr' -'eingebe' -'eile' -'eight' -'eidung' -'eide' -'ehnen_' -'eg_' -'efizit' -'effizienten_' -'een' -'eel' -'edu' -'edl' -'eden_' -'ecken_' -'ecke' -'eben' -'earnings_' -'ean_' -'eager_' -'dys' -'dy' -'dt' -'dropped_' -'droht_' -'dringende' -'dream' -'drau' -'drafted_' -'downward_' -'dow' -'doubled_' -'donors_' -'donations_' -'dokument' -'doctrine_' -'dit_' -'districts_' -'distortion' -'distant_' -'disruption_' -'displays_' -'dismantl' -'dish' -'disen' -'discret' -'discredit' -'diplomatische_' -'din_' -'din' -'differentia' -'diesbezüglichen_' -'did' -'dicht_' -'dial' -'dhi' -'dha' -'dez_' -'deux_' -'deutet_' -'deteriorati' -'detected_' -'desired_' -'descent_' -'desa' -'derselben_' -'derivative' -'deregulation_' -'deprived_' -'deno' -'dene' -'dema' -'delivers_' -'delivering_' -'defende' -'deepening_' -'deckung' -'dec' -'debtor' -'debatte' -'deadl' -'daten' -'dateien_' -'databases_' -'dara' -'dan' -'damp' -'damages_' -'daf' -'cy' -'cult' -'cul' -'criticized_' -'criminals_' -'creature_' -'cow' -'counterparts_' -'counterfeit' -'council_' -'cost' -'coole' -'cool_' -'conversion_' -'convention_' -'contradict' -'contin' -'conten' -'construct_' -'constrain' -'consiste' -'considers_' -'conservation_' -'consequently_' -'confused_' -'confront' -'config' -'confi' -'conduc' -'condemn_' -'condemn' -'concludes_' -'conci' -'comprehen' -'compound' -'compli' -'complexity_' -'completion_' -'complain' -'compile' -'competing_' -'competence_' -'compari' -'compare_' -'compar' -'compa' -'communist_' -'communicate_' -'communi' -'commonly_' -'comment' -'combines_' -'college_' -'coherence_' -'cog' -'coa' -'closest_' -'cliff' -'cleane' -'clau' -'classical_' -'clara_' -'claiming_' -'cks' -'ckel' -'cked_' -'citi' -'circum' -'ciones_' -'cier' -'cide' -'chä' -'chsen' -'chose_' -'chnungen_' -'chil' -'chief' -'chest' -'chemicals_' -'checks_' -'chauen' -'chaften_' -'ceremon' -'cer_' -'cease_' -'caution_' -'catastrophe_' -'casinos_' -'cari' -'care' -'capt' -'capitals_' -'capitali' -'cano' -'cac' -'cabin' -'bürgerliche' -'butt' -'bust' -'burn_' -'bureaucrats_' -'bull' -'broke_' -'brochure' -'brilliant_' -'bread_' -'branche_' -'bour' -'borrowing_' -'bombing' -'bombe' -'bolster' -'blocked_' -'block' -'bl' -'biot' -'billions_' -'bezug_' -'bewusste' -'bewegung_' -'bewa' -'bevorzug' -'beu' -'betriebe_' -'betrieb_' -'betrachtete' -'betra' -'besucht_' -'bestä' -'bestimmter_' -'besticht_' -'beso' -'beschaff' -'berufen_' -'berp' -'bereiten_' -'bereiche_' -'berei' -'bemerkenswerte' -'beliebige' -'belie' -'bekannter' -'beizu' -'beis' -'behold_' -'beh' -'begleiten_' -'beginne' -'begehen_' -'began' -'befr' -'beeinträchtigen_' -'bedeutendsten_' -'beauftragt' -'bears_' -'beanspruch' -'bble' -'bay_' -'basierte_' -'bargain' -'baltischen_' -'balcony_' -'balancing_' -'bak' -'bach' -'ax_' -'avo' -'außer' -'außenpolitische' -'automo' -'automatische' -'auszur' -'auszuführen_' -'auswe' -'auslösen_' -'ausgewählt_' -'ausgenutzt_' -'auft' -'aufrechterhalten_' -'aufl' -'aufkommen_' -'aufget' -'aufgebaut_' -'auen_' -'audit' -'attracti' -'ats' -'ata' -'astr' -'assignment' -'asser' -'asia' -'ase' -'ars' -'arrive_' -'arre' -'arme' -'arit' -'arises_' -'arische' -'aren_' -'architekt' -'architect' -'arche' -'arb' -'appro' -'appar' -'apologi' -'anzuh' -'anz' -'anyway_' -'any' -'anxi' -'anwesend_' -'antwortete_' -'antworten_' -'anten_' -'anteil_' -'ante' -'anstreben_' -'ansteigen_' -'ansprechen_' -'anschließen_' -'ansch' -'annu' -'announce_' -'anno' -'anme' -'anlässlich_' -'anl' -'ani_' -'angenehm_' -'angemessenen_' -'angelegt_' -'angegriffen_' -'angebot_' -'angebliche' -'angan_' -'anes' -'anen' -'anderweitig_' -'anden_' -'andauern' -'anbelangt_' -'analyse_' -'amplif' -'amme' -'ami_' -'amended_' -'amen_' -'amb' -'alter' -'alte' -'allocat' -'allo' -'alliances_' -'alleged_' -'alleg' -'alla' -'alit' -'alike_' -'alig' -'alie' -'alia' -'alg' -'ain' -'agog' -'agent_' -'aft' -'afrikanische' -'affirm' -'advertisement' -'adventur' -'admit' -'activat' -'aco' -'achung_' -'accr' -'accord_' -'accompanying_' -'accommodati' -'ac_' -'abw' -'absurd_' -'abse' -'abschließen' -'abs' -'able' -'abl' -'abhängt_' -'abhängig' -'abgele' -'abgehalten_' -'abgeben_' -'aat' -'`_' -']]''' -'] ._' -'Zypern_' -'Zwangs' -'Zunahme_' -'Zulassung_' -'Zukunfts' -'Zivilisten_' -'Zie' -'Zh' -'Zentrala' -'Zeita' -'Zealand_' -'Zahlungs' -'Zahlung_' -'YO' -'Xi_' -'XL' -'Würden_' -'Wünsche_' -'Wäre' -'Wurzel' -'Wri' -'Works' -'Wood' -'Wolf_' -'Wol' -'Wohl_' -'Wis' -'Wirtschaftsl' -'Wind' -'Willi' -'Wies' -'Wiederaufbau_' -'Wider' -'WiFi_' -'Wetter_' -'Wett' -'Werk' -'Werden_' -'Weltb' -'Weiterhin_' -'Weis' -'Wechselkurse_' -'Wechsel_' -'Wars' -'Wand' -'Walt' -'Wai' -'Wah' -'Wach' -'WA' -'Vorw' -'Vorteil' -'Vorr' -'Vorfeld_' -'Volle' -'Vogel' -'Vit' -'Visual_' -'Visit' -'Ville' -'Villa' -'Viel' -'Vet' -'Verzögerungen_' -'Verzeichnis_' -'Verwa' -'Vertrauens' -'Versicherungs' -'Versand' -'Versa' -'Vermögens' -'Vermittlungs' -'Verlängerung_' -'Vergleich' -'Vergessen' -'Verfasser' -'Vereinfachung_' -'Verda' -'Veranstaltungsräume_' -'Vegas_' -'Varia' -'Van' -'Vale' -'VAT_' -'VA' -'Users_' -'Ursprung_' -'Urheber' -'Unterschied' -'Untere' -'Unsicherheit_' -'Uns' -'Universum' -'Unge' -'Unbe' -'Unabhängig_' -'Un_' -'Umstände_' -'Uganda' -'Ub' -'UL' -'UC' -'Tür_' -'Tätigkeiten_' -'Twin' -'Tul' -'Tuesday_' -'Treiber_' -'Treib' -'Travel_' -'Translat' -'Track' -'Total_' -'Tor_' -'Top_' -'Tochter' -'Tob' -'Tir' -'Tibe' -'Theater' -'Text' -'Teufel' -'Territori' -'Terminal_' -'Techno' -'Tarif' -'Tanz' -'TT' -'TS' -'TEC' -'Südafrika_' -'Szenario_' -'Synchron' -'Suppo' -'Suiten_' -'Stütz' -'Studie' -'Strände_' -'Strahlung_' -'Straftat' -'Store_' -'Steuersystem' -'Statistik_' -'Stand' -'Stalin_' -'Stagnation_' -'Spri' -'Sprech' -'Spre' -'Spl' -'Spiegel' -'Sozialversicherung' -'Sozialdemokrati' -'Source_' -'Sou' -'Sonnens' -'Solo' -'Solana_' -'Software' -'Soft_' -'Socialist_' -'Sno' -'Sn' -'Small_' -'Sla' -'Sky_' -'SilverFast_' -'Sigma' -'Sicherung_' -'Sicherheitsfragen_' -'Sh' -'Seve' -'Settings_' -'Sem' -'Sek' -'Seen_' -'Seele_' -'Script' -'Schwan' -'Schulungs' -'Schuld_' -'Schuh' -'Schriftsteller_' -'Schriften_' -'Schreib' -'Schluss' -'Schiffs' -'Schic' -'Schach' -'Satellite' -'Santiago_' -'Sanc' -'Sammlung_' -'Sale' -'SW' -'SV_' -'STA' -'SS' -'SP_' -'SLR_' -'SL' -'SH' -'SD_' -'SCO' -'SCH' -'Rö' -'Routine' -'Route' -'Round_' -'Roten_' -'Rosa' -'Rod' -'Richtlinien' -'Rich' -'Rezept' -'Revolution' -'Revision_' -'Reserven_' -'Res' -'Repr' -'Renminbi_' -'Religio' -'Relax_' -'Reisende' -'Reisen_' -'Reinigung_' -'Reichen_' -'Regulierungs' -'Registr' -'Redner_' -'Record' -'Rechtsg' -'Rechtsetzung' -'Ratsvorsitzes_' -'Rate' -'Randlage_' -'Rande_' -'Ramblas_' -'Rahmenprogramm' -'RSS_' -'RGB_' -'RA_' -'Quer' -'Quark' -'Pun' -'Prüf' -'Präsidenten' -'Präsenz_' -'Prozent' -'Protest' -'Protection_' -'Pros' -'Promo' -'Progress' -'Programmen_' -'Prof_' -'Product_' -'Process_' -'Prob' -'Priv' -'Princes' -'Prince' -'Premi' -'Prad' -'Power' -'Pou' -'Portal' -'Polizei' -'Poli' -'Plätze_' -'Pil' -'Pie' -'Physi' -'Photos' -'Phone' -'Philosoph' -'Philadelphia_' -'Petersburg_' -'Peru_' -'Pensions' -'Pel' -'Paul' -'Partners' -'Parliamentary_' -'Parlamentswahlen_' -'Parkplätze_' -'Parameter_' -'Papier_' -'Pakistani_' -'Ox' -'Outs' -'Others_' -'Ost_' -'Os_' -'Ori' -'Options' -'Optimismus_' -'Olive' -'Oli' -'Offensi' -'Objektiv' -'Oberfläche_' -'OLAF_' -'OL' -'OG' -'Nü' -'Nuevo_' -'Nothing_' -'Notenbanken_' -'Norm' -'Nordic_' -'Nin' -'Nieder' -'News' -'Nepal' -'Nelson_' -'Neg' -'Near_' -'Nap' -'Nahrungsmittel' -'Nachdruck_' -'NS_' -'NRO_' -'NOT' -'NE_' -'NB' -'Müll' -'Mä' -'Mut' -'Music_' -'Mus' -'Monte_' -'Mitter' -'Mittelschicht_' -'Mitarbeitern_' -'Ministeri' -'Mine' -'Milo' -'Mille' -'Militar' -'Milan_' -'Migrations' -'Miet' -'Mexik' -'Metall' -'Meta' -'Meinungsäußerung_' -'Meines_' -'Meilen' -'Mehrere_' -'McCa' -'Maz' -'Mauer_' -'Master_' -'Marken' -'Marke_' -'Mang' -'Malta_' -'MC' -'M5' -'Lösch' -'Luxemburg_' -'Lux' -'Lunch' -'Luggage_' -'Lud' -'Lot' -'Lor' -'Lohn_' -'Lohn' -'Lock' -'Loca' -'Lizenz_' -'Literatur_' -'Linien_' -'Line_' -'Lind' -'Liikanen_' -'Lich' -'Liberalen_' -'Leu' -'Letzte' -'Leonardo_' -'Leistungsbilanz' -'Leica_' -'Leh' -'Lega' -'Lebensst' -'Lebensqualität_' -'Learn_' -'Las' -'Lar' -'Landes' -'Lan' -'Lama' -'Lai' -'Lager_' -'Laden_' -'Kürz' -'Königs' -'Kämpfe_' -'Kurse' -'Kub' -'Kräften_' -'Kroatien_' -'Kro' -'Kreditgeber_' -'Krankenh' -'Kraftstoff' -'Korr' -'Kopie' -'Konservativen_' -'Kompon' -'Komplettpreise_' -'Kollege_' -'Kohäsions' -'Kohlenstoff' -'Kohle_' -'Knoten_' -'Kn' -'Klon' -'Kleid' -'Kis' -'Kinders' -'Ket' -'Kenya_' -'Kennzeichnung' -'Kathedrale_' -'Kaffee_' -'Jose' -'Jon' -'Jobs_' -'Jin' -'Jh' -'Jas' -'Jam' -'Jagd' -'Jacuzzi_' -'JA' -'Italia' -'Inzwischen_' -'Investi' -'Inve' -'Intelligen' -'Intellektuelle' -'Integrität_' -'Inte' -'Instabilität_' -'Insp' -'Insofern_' -'Innere' -'Inland' -'Impfung' -'Imperial' -'Imm' -'Ice' -'INI_' -'ICA' -'Höhepunkt_' -'Händen_' -'Hyper' -'Hurri' -'Hungers' -'How' -'Hos' -'Homo' -'Holocaust_' -'Holl' -'Hold' -'Hoffnungen_' -'Hof' -'Hindernisse_' -'Hilton_' -'Hillary_' -'Hierzu_' -'Hier' -'Hes' -'Heran' -'Hence_' -'Help_' -'Helm' -'Heimatl' -'Heb' -'Heads_' -'Haushalts_' -'Hauptb' -'Hast' -'Hans_' -'Hall' -'HP_' -'Gü' -'Göteborg_' -'Gute_' -'Gui' -'Greenspan_' -'Greater_' -'Grad' -'Gr' -'Gordon_' -'God' -'Goals_' -'Gn' -'Glaubens' -'Gipfels_' -'Gewährleistung_' -'Gesellschafts' -'Gesan' -'Gerät_' -'Gere' -'Gerade_' -'Gepäck' -'Geno' -'Genf' -'Gelegenheiten_' -'Gelds' -'Gegen_' -'Gefähr' -'Gebühr_' -'Game' -'Gam' -'Gad' -'Gabriel' -'Gab' -'GR' -'Fünfte' -'Fünf' -'Further_' -'Fuji' -'Fris' -'Friedensprozess_' -'Freib' -'Freedom_' -'Four_' -'Fotokopiereinrichtungen_' -'Fond' -'Fokus_' -'Flughafen' -'Fluggesellschaften_' -'Flexi' -'Fleisch' -'Fitness_' -'Fischereipolitik_' -'Finanzsystems_' -'Finanzminister_' -'Finanzau' -'Filter' -'Fertig' -'Fernseher_' -'Fehlen_' -'Features_' -'Fabriken_' -'FR' -'FDI_' -'FA_' -'FARC_' -'Extrem' -'Expo' -'Exi' -'Excel_' -'Evolution_' -'Events_' -'Event_' -'Europäisches_' -'Eth' -'Ess' -'Erö' -'Erweiterungs' -'Ersch' -'Erre' -'Eropa_' -'Ern' -'Ermittlung' -'Erkrankung' -'Ericsson_' -'Erfüllung_' -'Erfolgs' -'Erdöl' -'Episode' -'Entwicklungsst' -'Entwicklungsa' -'Enth' -'Enron_' -'Enl' -'Enk' -'Eng' -'Elektronik' -'Elektro' -'Eisen' -'Einzelh' -'Einkaufss' -'Einhei' -'Einh' -'Eing' -'Einfuhr' -'Einf' -'Einerseits_' -'Eine' -'Eigenschaft_' -'Eigenheim' -'Edin' -'Eden_' -'Eck' -'Easy_' -'ESS' -'ENT' -'Dörfer' -'Dutzend_' -'Drittstaaten_' -'Drei' -'Draghi_' -'Dow_' -'Dou' -'Dominion_' -'Dominica' -'Doc' -'Diktator' -'Digi' -'Dienstes_' -'Dienste' -'Did' -'Dick' -'Dez' -'Dev' -'Dest' -'Denk' -'Demand_' -'Defi' -'Days_' -'Damals_' -'Dach' -'DR_' -'DR' -'DPJ_' -'DB' -'DAM' -'Crystal_' -'Croatia_' -'Cri' -'Court' -'Could_' -'Corr' -'Conver' -'Contact_' -'Consult' -'Consequently_' -'Conc' -'Comple' -'Commerce_' -'Commander_' -'Coll' -'Coffee_' -'Chirac_' -'Chip' -'Chechnya_' -'Chat_' -'Charlie_' -'Champs_' -'Chai' -'Centr' -'Cell' -'Cath' -'Carr' -'Carmen_' -'Cari' -'CL' -'Büro_' -'Büchern_' -'Bäume_' -'By' -'Business' -'Bundesrepublik_' -'Bundeskanzlerin_' -'Building_' -'Buck' -'Brüder_' -'Bruttoinlandsprodukt' -'Brown_' -'Bridge_' -'Brandenburg' -'Bour' -'Borde' -'Bonus_' -'Bonn_' -'Bomb' -'Bog' -'Bob' -'Blut_' -'Blockade_' -'Bilde' -'Bibliotheken_' -'Bezeichnung_' -'Bewe' -'Bevor_' -'Between_' -'Betriebssystem_' -'Betrachtung_' -'Besuche' -'Bestätigung_' -'Bestände_' -'Bestellung_' -'Bestand' -'Besser' -'Beruf' -'Berechnung_' -'Bere' -'Benutzern' -'Benach' -'Bem' -'Bell_' -'Beli' -'Belgian_' -'Bekenntnis_' -'Bekannt' -'Beg' -'Befürchtungen_' -'Befür' -'Beau' -'Bearbeit' -'Bath_' -'Baj' -'Bahrain_' -'Bac' -'BSE_' -'BERLIN_' -'BD' -'B6_' -'Award' -'Automobilindustrie_' -'Auswahl' -'Austritt_' -'Australian_' -'Austin_' -'Ausstellung_' -'Aussichten_' -'Ausschluss_' -'Aussagen_' -'Aussage_' -'Ausrüstung_' -'Ausbr' -'Ausblick_' -'Ausbildungs' -'Ausarbeitung_' -'Aufwertung_' -'Aufl' -'Aufg' -'Aufb' -'Atlantik' -'Ate' -'Astro' -'Arzneimittel_' -'Arth' -'Arr' -'Arms' -'Armenia_' -'Armeni' -'Arme' -'Archiv' -'Arbeitsplatz_' -'Arbeitnehmern_' -'Arbeitnehmer' -'Applikation' -'Appl' -'Appart' -'Appar' -'Apart_' -'Apa' -'Any' -'Anschlag_' -'Ansch' -'Anruf' -'Anreiz_' -'Anr' -'Anpassungs' -'Annäherung_' -'Anne' -'Anmeld' -'Angabe_' -'Anf' -'Anderson_' -'Anbieter_' -'Analysen_' -'Anal' -'Amerikanern_' -'Alon' -'Algeria_' -'Alge' -'Alf' -'Alc' -'Albu' -'Aktienm' -'Aero' -'Admi' -'Acro' -'Acqu' -'Acht_' -'Access' -'Abänderung' -'Abw' -'Absicherung_' -'Abschwung' -'Absa' -'Abkommens_' -'Abbas_' -'ASE' -'AL_' -'ALDE_' -'AKVIS_' -'89_' -'82_' -'240_' -'215' -'1982_' -'1973_' -'1948_' -'1900_' -'18th_' -'177' -'175' -'1701_' -'160' -'128_' -'/-_' -'. ({{_' -'*_' -'**' -')]' -''']]' -'%) _' -'">' -' �_' -' – _' -' ==' -' "._' -' ", _' -'€ _' -'’ – _' -'– ' -'ע' -'өз' -'і_' -'ю' -'х_' -'то' -'т_' -'ри' -'про' -'ор' -'на' -'мо' -'ль' -'лы' -'к_' -'ин' -'ем' -'ве' -'ο_' -'μ' -'ž_' -'ši' -'Č' -'ützen' -'üte' -'ügen_' -'üchter' -'üch' -'übrig_' -'üblicherweise_' -'überzeugend_' -'überwältigende' -'übertrag' -'überraschen' -'übermitteln_' -'überla' -'übergeben_' -'überarbeitet' -'ør' -'öt' -'ört_' -'örper' -'örigkeit_' -'ören_' -'örder' -'öpf' -'ökonomisch' -'ökologischen_' -'ökologische_' -'ök' -'ño' -'és' -'éo' -'ées_' -'ça' -'äßig_' -'änk' -'ängt_' -'änen_' -'ändig' -'äm' -'ältig' -'älle_' -'ährung_' -'ähnliches_' -'ächtnis' -'ál' -'Übertr' -'Übel' -'Äthiopien_' -'Ägypte' -'» | _' -'»' -'}: _' -'|''_' -'zügig_' -'zähl' -'zynisch' -'zyklus_' -'zwing' -'zweckmäßig' -'zwanzig_' -'zw' -'zuzus' -'zuzug' -'zuwe' -'zutage_' -'zunichte_' -'zukünftigen_' -'zuha' -'zugeg' -'zugeben_' -'zona_' -'zo_' -'zn' -'zle' -'zk' -'zivile_' -'zim' -'zierte' -'zid' -'zessi' -'zens' -'zeitlich' -'zahlreicher_' -'ystemen_' -'yss' -'yle' -'yh' -'yen_' -'yed_' -'ycl' -'xio' -'xen' -'würdig' -'wür' -'wählt' -'wä' -'wunderbare' -'wun' -'worm' -'womöglich_' -'wol' -'wiss' -'wished_' -'wische' -'wirkungen_' -'wirksamer_' -'winzige' -'winds_' -'winding_' -'willig_' -'wiederholte' -'widerspiegel' -'widen' -'wid' -'wichtig' -'wich' -'wheel' -'wesentlicher_' -'werte' -'werk' -'wenigstens_' -'wen_' -'weithin_' -'weil' -'weigert' -'wechselt' -'wechsel_' -'websites_' -'wear_' -'wear' -'watched_' -'warrant_' -'warnen_' -'ware' -'wandeln_' -'wandel' -'waktu_' -'wak' -'wahrgenommen_' -'wag' -'vu' -'vre' -'vot' -'vorzus' -'vorteil' -'vorsichtig' -'vorsehen_' -'vorläufige' -'vorliegende_' -'vorgez' -'vorgestellt_' -'vorgehen_' -'voraussichtlich_' -'volunteer_' -'volle' -'vo_' -'vital' -'visual_' -'visiting_' -'vice_' -'veterinar' -'vessel_' -'verzeichnet' -'verzeichnen_' -'verwunde' -'verwi' -'verw' -'verursachten_' -'verursachte_' -'vertrieb' -'vertrau' -'verteidig' -'verstärkte_' -'versich' -'versehen' -'verschw' -'verschli' -'verschlechtert' -'verschl' -'verpflichtend' -'vermute' -'vermis' -'verlängert_' -'verlorene' -'verle' -'verlangsamt_' -'verkündet_' -'verknüpft_' -'verhäng' -'verhältnis_' -'vergangen' -'verfüg' -'verbundene_' -'verbringen_' -'verbreitete_' -'verbot' -'verbi' -'verbe' -'verbal' -'verb' -'verantwortlichen_' -'verans' -'verab' -'vede' -'vate' -'varied_' -'valued_' -'vall' -'validate' -'vacuum_' -'vaccination_' -'uver' -'uu' -'utz' -'uts_' -'utm' -'usted_' -'usst' -'ussi' -'usr' -'usion' -'user' -'ursprüngliche_' -'uropäischen_' -'urh' -'urges_' -'urge' -'upload_' -'upholding_' -'uo' -'unzulä' -'unwilling_' -'unwi' -'unverzüglich_' -'unus' -'untr' -'unterz' -'unterstreicht_' -'unterschr' -'unterscheidet_' -'unterrichte' -'unternehme' -'unterliegen_' -'unterhalt' -'untereinander_' -'unterbrochen' -'unterb' -'unsu' -'unsicheren_' -'unschuldige' -'unsa' -'unres' -'unr' -'unnecessary_' -'unle' -'unjust' -'universali' -'uniti' -'union' -'unified_' -'ungst' -'ungspro' -'ungsge' -'ungsfrei' -'ungl' -'unequ' -'underw' -'underli' -'unbest' -'umgehend_' -'umgeb' -'ulation' -'ulan' -'uhig' -'uhe' -'ugh' -'ufung_' -'ufte' -'udi' -'uche' -'uc_' -'tück' -'töten_' -'tö' -'té_' -'tätige_' -'tägliche_' -'tze' -'typischer' -'typischen_' -'typische_' -'tures_' -'tungs' -'tts' -'ttl' -'tse' -'trü' -'träum' -'truktur' -'troubled_' -'troll' -'tril' -'tribunal' -'trenn' -'trauma' -'trate' -'trap_' -'transpos' -'transmitted_' -'trail' -'trade' -'toxic_' -'tower_' -'towels_' -'tot_' -'tonn' -'toffe' -'tnis' -'tliche_' -'tles' -'tiv_' -'tisa' -'tionary_' -'tionali' -'tiny_' -'timely_' -'tile' -'tide_' -'tics_' -'tically_' -'tical' -'tial_' -'tial' -'thro' -'thou' -'thirty_' -'thinks_' -'therapy_' -'thee_' -'tern_' -'termasuk_' -'terb' -'tep' -'tension_' -'tener' -'tena' -'tempora' -'tempo_' -'tem' -'telling_' -'telecommunications_' -'teilig' -'technologi' -'td' -'tche' -'tch_' -'taucht' -'tati' -'tapi_' -'tang' -'tane' -'tain_' -'tain' -'tailored_' -'tail' -'tags' -'tac' -'tab_' -'szi' -'systematische_' -'system' -'sys_' -'synthesi' -'synchroniz' -'symboli' -'symbol' -'swor' -'sweise' -'svoll' -'sverfahren_' -'sv' -'suspect' -'survey_' -'surprisingly_' -'surge_' -'suprem' -'suppressi' -'supplier_' -'supplied_' -'supermarkets_' -'suns' -'sunnitische' -'sumber_' -'suche_' -'successor_' -'substanzielle' -'subst' -'subsidiz' -'subsidiar' -'subjects_' -'stürzen_' -'stärkeren_' -'stärke' -'stupid_' -'struggle' -'strophen' -'strive_' -'strik' -'stric' -'strengere' -'streng' -'strebt_' -'stream' -'strategisch' -'strat' -'strand' -'strahl' -'strafrechtlich' -'ston' -'stolz_' -'stoff_' -'stin' -'stim' -'steuern_' -'sten' -'stellend_' -'steile' -'steckt_' -'stecken_' -'stec' -'stays_' -'statute_' -'starv' -'starters_' -'starkes_' -'standen_' -'stande' -'standardis' -'stakeholder' -'stabili' -'staats' -'staatlich_' -'staate' -'ssta' -'ssin' -'sses_' -'sserung' -'ssene_' -'ssend' -'ssed_' -'ssal' -'srecht' -'squeeze_' -'sque' -'sprin' -'spreche_' -'spreading_' -'spoil' -'spirit' -'sphere_' -'spends_' -'spell_' -'specialist_' -'special' -'speaks_' -'spatial' -'spann' -'sos' -'sorti' -'sorry_' -'sore' -'sooner_' -'sont_' -'sonder' -'sollt' -'solide_' -'solid' -'sold' -'solar_' -'socially_' -'socialism_' -'smokers_' -'smit' -'slos' -'slim' -'slide_' -'slee' -'slan' -'skri' -'skr' -'skiing_' -'skie' -'sket' -'skepticism_' -'skat' -'sir_' -'sir' -'sip' -'sins' -'simul' -'silver_' -'sika' -'sik' -'signals_' -'sights_' -'sig' -'siert_' -'sier' -'sichtbar_' -'sible_' -'sian_' -'sia_' -'sia' -'shutt' -'show' -'shortc' -'shores_' -'shaping_' -'shaped_' -'shal' -'sgr' -'seu' -'setzung_' -'set' -'separation_' -'sentiment_' -'sensi' -'sensation' -'sends_' -'sement' -'seltene' -'seller' -'sell' -'seiner' -'sebuah_' -'sebe' -'sco_' -'sco' -'scienti' -'schüre' -'schö' -'schwerwiegende' -'schweigen_' -'schwank' -'schung_' -'schulden' -'schuh' -'schreib' -'schottische' -'schnitt' -'schnellere' -'schneide' -'schmerzhaft' -'schlechteste' -'schlechter_' -'schlag_' -'schickt' -'schicken_' -'schenken_' -'scheinlich' -'scheinbar_' -'schafts' -'scarc' -'scan' -'sberei' -'satu_' -'satisfactor' -'sani' -'sanct' -'sample_' -'sal_' -'saja_' -'sacrifice_' -'rät' -'ränk' -'rzt_' -'rungen_' -'ruled_' -'ruhige_' -'ruct' -'rth' -'rro' -'royal' -'ron_' -'roi' -'rodukt' -'rocket' -'roblem' -'rla' -'rk_' -'rist' -'risik' -'risen_' -'rine' -'rily_' -'rigoro' -'rigid' -'ried' -'riding_' -'rid_' -'rice_' -'rgen_' -'rf_' -'revolutionäre' -'reviv' -'revis' -'rever' -'resur' -'resultierende' -'restrictive_' -'restr' -'respekt' -'resorts_' -'resolving_' -'reserven_' -'reserv' -'republik' -'repressive_' -'reporte' -'reor' -'renz' -'rende' -'relies_' -'relaxed_' -'relaxation_' -'rej' -'reitung' -'reiterate_' -'reiches_' -'reichende' -'reibung' -'regulierung' -'registrierte' -'registriert_' -'regiert_' -'regelmäßige' -'refusing_' -'recurr' -'recover' -'recording' -'recipe_' -'reci' -'rechtliche' -'rechtfertigen_' -'rechnung' -'rech' -'recap' -'reasoning_' -'realized_' -'reaches_' -'rdl' -'rdin' -'rche' -'raus' -'rats_' -'ratified_' -'rase' -'rasant' -'rarely_' -'rapi' -'ramp' -'ramm' -'rale' -'rak' -'rail' -'raft_' -'raff' -'radiation_' -'rad_' -'query_' -'quellen_' -'quarte' -'quantitativen_' -'purely_' -'puls' -'pull_' -'pulat' -'publish_' -'pts_' -'pta' -'pré' -'proxy_' -'provinc' -'proto' -'protests_' -'protecti' -'protagonist' -'propriet' -'proportional_' -'propo' -'prope' -'pron' -'promptly_' -'proliferation_' -'projected_' -'profit' -'profi' -'proclaimed_' -'problemlos_' -'prinzipie' -'prin' -'primäre' -'pricing_' -'pric' -'pretext_' -'prediction' -'precedent_' -'precautionary_' -'preca' -'prachigen_' -'ppo' -'pping_' -'potenziell_' -'potato' -'pot' -'postponed_' -'positioned_' -'portray' -'porte' -'por' -'poorly_' -'politischem_' -'polariz' -'pod' -'plum' -'plot' -'plo' -'plai' -'placing_' -'pita' -'pit_' -'piscin' -'pin_' -'pilla' -'pielen_' -'piele_' -'pie_' -'pian' -'photograph_' -'phe' -'pfer' -'pfen_' -'pets_' -'petition_' -'pers_' -'permits_' -'peripheral_' -'penu' -'peni' -'pende' -'pek' -'pedestrian_' -'patron' -'paths_' -'patent' -'patch_' -'passport_' -'passive_' -'passiv' -'passionate' -'partnerschaftliche' -'participating_' -'parl' -'parent_' -'paraly' -'paradi' -'pap' -'pang' -'pad_' -'packaging_' -'owe' -'overha' -'outsourc' -'outlined_' -'outlet' -'outf' -'outd' -'ount' -'ounce' -'oul_' -'oui' -'oud_' -'otis' -'osten_' -'osk' -'orti' -'ork' -'origins_' -'orient' -'ori_' -'organisms_' -'orf' -'ordne' -'ordentliche' -'orat' -'orang_' -'oran_' -'optimize' -'optimism_' -'optimier' -'optical_' -'oppo' -'opoulo' -'opol' -'opo' -'operators_' -'openness_' -'oon_' -'ood_' -'ontr' -'onist' -'onic' -'ones' -'onds_' -'ond_' -'ommene' -'ologis' -'oli_' -'oka' -'oing_' -'oid' -'ogramm' -'ogn' -'ogg' -'ofs_' -'offene' -'odell' -'occup' -'observer_' -'observe_' -'oberste_' -'oberfläch' -'obere' -'oasis_' -'nütz' -'nördlichen_' -'nächstes_' -'nä' -'nwe' -'nver' -'nur' -'null' -'ntion_' -'ntie' -'nted_' -'nstein_' -'nsh' -'nsa' -'np' -'notorious' -'nostalgi' -'normali' -'nominat' -'noc' -'nkt_' -'nineteenth_' -'nifi' -'niederge' -'nian' -'neueste_' -'netze_' -'nest' -'nese_' -'nern_' -'nente' -'neiden' -'neglect' -'neben' -'ndu' -'nds' -'ndr' -'ndo' -'ndl' -'ndet_' -'nde' -'ncia' -'nationales_' -'nas' -'nant' -'nano' -'nami' -'nahezu_' -'nad' -'nacht_' -'nachhaltiger' -'mé' -'mäßigen_' -'mäß' -'mächte_' -'muy_' -'muti' -'muster' -'murdered_' -'murder' -'mund' -'mt_' -'mst' -'mse' -'mpi' -'moves_' -'motive' -'mortgages_' -'morph' -'morat' -'moralischen_' -'monatlich' -'molecul' -'modernization_' -'modernis' -'modernes_' -'mobile' -'mmern_' -'mium_' -'mittlere' -'mistakes_' -'misleading_' -'miracle_' -'ministeri' -'minded_' -'mili' -'mie_' -'mh' -'mexikanische' -'metr' -'method' -'metal_' -'messe' -'menya' -'menta' -'menschen' -'meme' -'melden_' -'md' -'mba_' -'maximize_' -'maximal' -'mau' -'matic_' -'mathematics_' -'materielle' -'massa' -'mask_' -'marginali' -'margin_' -'marble_' -'mann' -'manipulier' -'mangelnde' -'manc' -'managers_' -'malt' -'maker_' -'maj' -'magi' -'maga' -'mad' -'macro_' -'mache' -'läßt_' -'läufig' -'lze' -'lying_' -'lush_' -'luck' -'ltungs' -'lous_' -'loss' -'lokaler_' -'lohnt_' -'logistics_' -'llusion' -'llung_' -'llschaft' -'lli' -'ller' -'llel' -'live' -'listened_' -'lism' -'lisi' -'lische' -'lio' -'link' -'lings_' -'liness_' -'lighting_' -'lifetime_' -'ließ' -'lien' -'licensing_' -'liberat' -'liberalisi' -'liberali' -'liberalen_' -'liberale' -'liat' -'lian' -'liabilities_' -'letztendlich_' -'lens_' -'lengthy_' -'lender' -'leistungs' -'leh' -'left' -'lebte' -'leas' -'lding_' -'ldet_' -'lba' -'lb' -'laureate' -'laundry_' -'laufende' -'lauf_' -'latz' -'laser_' -'laptop_' -'landwirtschaftlichen_' -'landschaft_' -'lana' -'lakes_' -'lah_' -'lage' -'ladi' -'lacking_' -'könnt_' -'käme_' -'kw' -'kurzfristigen_' -'kurzfristige_' -'kurzen_' -'kurse_' -'kup' -'kunden' -'kultur_' -'kula' -'kua' -'ktivi' -'kten_' -'kreuz' -'kret' -'kreise' -'krei' -'kratisch' -'krank_' -'kow' -'kostenfreien_' -'kostenfreiem_' -'korre' -'koreanische' -'kontinentale' -'kontaktieren_' -'konstitutionelle' -'konservative' -'komplizierten_' -'komplexen_' -'kompet' -'kompakt' -'komfortabel_' -'kolonial' -'koll' -'kok_' -'knappe' -'kluge' -'klinische' -'klich' -'kleines_' -'klassische' -'klassi' -'klapp' -'kl' -'kka' -'kische' -'kinder' -'kilometre' -'kid' -'kf' -'kett' -'kerja_' -'kehr_' -'kehr' -'kamera' -'kal' -'jö' -'justification_' -'juristische' -'jung' -'jun' -'journalist_' -'journal' -'jou' -'join' -'jeu' -'jeni' -'jeglicher_' -'jan' -'izi' -'iza' -'ix' -'ivat' -'iva' -'iv_' -'ium' -'ité_' -'itz' -'itts' -'itis' -'ita_' -'istic_' -'isti' -'iste' -'issuing_' -'issi' -'ison_' -'isolated_' -'isolat' -'islamistische' -'islamische_' -'isi_' -'irrev' -'irr' -'irgendeiner_' -'ires_' -'irc' -'iplin' -'ios_' -'ioni' -'iona' -'inviting_' -'investasi_' -'invasion_' -'intuitive' -'intra' -'intolera' -'interview_' -'interventions_' -'interv' -'interrupt' -'internationally_' -'interior_' -'interg' -'intensive' -'intensiv_' -'intelligente_' -'inte' -'intak' -'inta' -'institutionalis' -'instances_' -'inst' -'inspiri' -'inspectors_' -'insofern_' -'insist' -'inse' -'inquir' -'innehat' -'inne' -'injection' -'iniert' -'inhibit' -'inhaltlich' -'infra' -'inflict' -'infi' -'infectious_' -'infe' -'inex' -'inevitably_' -'iness_' -'ineffiziente' -'industriellen_' -'industrialis' -'indu' -'indische_' -'indirect_' -'indication_' -'incur' -'incorrect' -'incorporate_' -'inclusive_' -'incl' -'incapable_' -'inca' -'inat' -'inander_' -'inability_' -'impl' -'imperial' -'impedi' -'impede' -'impair' -'immigrant_' -'imba' -'ilte' -'illnesses_' -'ilde' -'iken_' -'ignor' -'igli' -'ight' -'ifor' -'ifel' -'ieß' -'iev' -'iess' -'ient' -'ieg_' -'ieden' -'ied' -'ido_' -'idio' -'identification_' -'identifi' -'ideale' -'ick' -'icial' -'ichtig' -'icht' -'icate' -'ibility_' -'ibi' -'iba' -'hübsch' -'höf' -'höchste' -'häus' -'häufiger_' -'hängen_' -'hw' -'hus' -'hurr' -'hunt' -'hts_' -'htm' -'hten_' -'hrung' -'hospitals_' -'hos' -'hors' -'hop' -'homo' -'homeland_' -'holt' -'hohes_' -'hochrangige' -'hnte' -'hmig' -'hmen_' -'hija' -'highway_' -'hierfür_' -'herz' -'hervorrufen_' -'hervorragende' -'herkömmlichen_' -'herausf' -'hende_' -'hemm' -'heb' -'heav' -'heated_' -'headlines_' -'header' -'headed_' -'hb' -'hav' -'hauses_' -'hause' -'haus' -'hate_' -'hast' -'harmonise_' -'harm' -'harde' -'harass' -'hanya_' -'handlungen_' -'halb_' -'haft' -'haf' -'habt_' -'günstige' -'gängig' -'gym' -'gura' -'gul' -'guitar' -'guide' -'guidance_' -'guest' -'guardian_' -'gste' -'grüne' -'gründet_' -'größerem_' -'größe' -'grösste' -'großartige_' -'grou' -'grie' -'gravierende' -'grausam' -'gog' -'god' -'gnet' -'gma' -'glücklich_' -'globalis' -'glichen_' -'girls_' -'gion' -'gin_' -'gin' -'gie_' -'giant_' -'geäußerten_' -'gewöhnlich' -'gewi' -'gewec' -'gewe' -'gewalt' -'getau' -'gesti' -'gesteckt_' -'gestaltete' -'geson' -'gesenkt_' -'gesellschaftlichen_' -'geschäft_' -'geschwindigkeit_' -'geschm' -'gerufen_' -'geriet' -'geri' -'gerechtfertigt_' -'gerechte_' -'geprägten_' -'geopolitische' -'geopolitical_' -'genom' -'gener' -'genen_' -'gende' -'gence_' -'gemessen_' -'gemeins' -'geldpolitische' -'geld' -'gehe_' -'geha' -'gegr' -'gegner' -'gegenseitige_' -'gefü' -'gefo' -'gefangen_' -'gef' -'geeignete_' -'geehrt' -'gedenkt_' -'gebiete_' -'geber_' -'gebe' -'gea' -'gather_' -'gastro' -'gasse_' -'gaps_' -'gall' -'gage' -'füllen_' -'förderung_' -'fähigen_' -'fungier' -'fung' -'fundamentally_' -'func' -'ftig' -'fter_' -'fst' -'früh' -'fruit' -'fronts_' -'frontier' -'fristig' -'frisch_' -'freut_' -'freundlichen_' -'frequent' -'fragile_' -'founder_' -'foun' -'fortzu' -'fortune_' -'fortgeschrittene' -'formulierte_' -'formali' -'formale' -'forg' -'foresee' -'forecast_' -'forder' -'forci' -'font' -'folgte' -'fold_' -'fn' -'flü' -'flugzeuge' -'flowing_' -'flower' -'flexibler' -'flexible' -'fled' -'flavo' -'flag' -'fizi' -'fix' -'fisch' -'finger' -'fiction_' -'fica' -'ffl' -'ffene' -'ffekt' -'feststellt' -'feste_' -'feiern_' -'fehlenden_' -'fec' -'featured_' -'feasible_' -'fea' -'favorite_' -'favored_' -'fau' -'fasc' -'farbe' -'fantastische' -'fangen_' -'faire' -'fahrts' -'fahrer' -'fahren' -'facilitating_' -'facilitat' -'fache_' -'fache' -'fabric_' -'fab' -'exzellenten_' -'extensi' -'exquisite_' -'exposure_' -'exploration_' -'exploiting_' -'experiencing_' -'exklusiv' -'existierende' -'exert_' -'exercise' -'exempl' -'exclu' -'exchange' -'excellence_' -'even' -'euro' -'euer' -'ette_' -'eting' -'eth_' -'eternal_' -'etabliert' -'etablieren_' -'esu' -'esto' -'estig' -'esta_' -'esses_' -'eso' -'erwecken_' -'erupt' -'erungen_' -'erta' -'erstens_' -'erstatte' -'erschl' -'errors_' -'errichtet_' -'erreich' -'erpr' -'ernen' -'ermä' -'erma' -'erläutern_' -'erleichter' -'erklär' -'erit' -'eris' -'eries_' -'erholsame' -'ergi' -'erge_' -'ergab' -'erfe' -'ereich_' -'erbe_' -'erbaut_' -'erbar' -'eran' -'equip' -'equat' -'epidemic_' -'epidemi' -'envi' -'entziehen_' -'entworfen_' -'enttäuscht' -'entspr' -'entschied_' -'entscheidung' -'entity_' -'entgegens' -'entgegenge' -'entfernten_' -'entfallen_' -'entail' -'ensured_' -'enorm_' -'enli' -'enh' -'engineer' -'energisch' -'ener' -'endanger' -'enact' -'employee_' -'empfäng' -'empfunden' -'empf' -'emotions_' -'emble' -'ellung_' -'elles_' -'electr' -'ekte' -'ej' -'eitig_' -'eir' -'einzust' -'einzuräumen_' -'einzurichten_' -'einzul' -'einzuf' -'einseitig' -'einräumen_' -'eingerichteten_' -'eingeh' -'einführen_' -'einfü' -'einbezieh' -'einbar' -'einb' -'eilt_' -'eilen_' -'eigne' -'eigenständige' -'eigens' -'eigenem_' -'eid' -'eichn' -'eho' -'ehl' -'eful_' -'efi' -'effizientere' -'effiziente_' -'effizient_' -'effizien' -'effekt_' -'editor' -'ede_' -'ecu' -'eck' -'echo' -'ebung_' -'ebnen_' -'dü' -'durchsetzen_' -'durchschnittliche_' -'durchschnittlich_' -'durchgesetzt_' -'duration_' -'duk' -'dst' -'dry' -'drucken' -'dron' -'drinking_' -'drift' -'dreh' -'drastisch_' -'drastic_' -'drasti' -'drag' -'dou' -'dorf_' -'dop' -'dog_' -'document' -'divisions_' -'dividing_' -'divide_' -'divi' -'diversen_' -'div' -'dition' -'distress_' -'distort' -'dist' -'dispose' -'dismissed_' -'disg' -'discre' -'discourse_' -'discourage' -'dische' -'disappointed_' -'disabilit' -'diper' -'dioxide_' -'dik' -'dih' -'digung' -'digkeit' -'diesbezügliche_' -'dib' -'diak' -'dg' -'devote_' -'devi' -'deutlichen_' -'detriment_' -'deten' -'det_' -'destru' -'despair_' -'designation_' -'desde_' -'derived_' -'dere' -'derartiger_' -'depressed_' -'deposits_' -'deploy' -'dense' -'denounce' -'demonstrieren_' -'demokratisch_' -'demograph' -'democrat' -'dementsprechend_' -'dell_' -'delightful_' -'delete_' -'degrees_' -'ded' -'decor' -'declines_' -'debe' -'debated_' -'dde' -'dba' -'dauerhafte' -'dare' -'dad' -'dacht' -'cycles_' -'curtail' -'cultivat' -'culminat' -'cue' -'ctur' -'ction' -'crush' -'crude_' -'critici' -'cript' -'crash_' -'cras' -'craft_' -'craft' -'cr' -'cozy_' -'couple' -'coup_' -'cotton_' -'cosmetic' -'correspond_' -'corps' -'copie' -'convince_' -'convicted_' -'convict' -'conversation_' -'controversial_' -'contagion_' -'contacts_' -'consult_' -'constitutes_' -'constituenc' -'constitu' -'consist' -'conquer_' -'connecting_' -'coni' -'confo' -'confined_' -'configure_' -'confe' -'conf' -'conciliation_' -'concentrated_' -'compre' -'compr' -'compliment_' -'complaint_' -'complain_' -'comparative_' -'common' -'commodities_' -'commission' -'commercial' -'comer' -'come' -'colonial_' -'collectively_' -'collections_' -'cola' -'cock' -'coat' -'coastal_' -'clu' -'closure_' -'clinical_' -'clin' -'cli' -'ckig' -'cket_' -'cke' -'cis' -'cinema' -'chtigen_' -'chsel' -'child' -'checke' -'chas' -'charisma' -'charakter_' -'characteristics_' -'characteristic_' -'change' -'chair' -'cere' -'censorship_' -'ced' -'cce' -'cca' -'cave' -'cautious_' -'cau' -'casual' -'casino_' -'carries_' -'capture_' -'captur' -'capability_' -'cap_' -'cant' -'cans_' -'cana' -'came' -'calculat' -'cafe' -'caci' -'bürokratische' -'bösartige' -'byl' -'but' -'burning_' -'burn' -'bureaucracy_' -'bum' -'bullet' -'builds_' -'buffer' -'brü' -'browsing_' -'brothers_' -'brake' -'boxe' -'bottle' -'borrowers_' -'borne_' -'bora' -'bookings_' -'bombard' -'boli' -'boil' -'bn' -'bloße_' -'bloß' -'blocking_' -'blockiert_' -'bloc' -'bliche' -'blase_' -'blam' -'birds_' -'billig_' -'bilität' -'bid_' -'bias' -'bia' -'bezi' -'bezeichnen_' -'bewert' -'beverages_' -'beunruhig' -'bett_' -'bett' -'betrü' -'beträchtliche' -'betrieb' -'betreiber' -'bete' -'bestrafen_' -'bestehender_' -'bestanden_' -'besitze' -'beside' -'besetzte' -'beset' -'beschw' -'beschreiben_' -'beschrei' -'berühmte_' -'beru' -'bereitstellen_' -'bereitet_' -'benötigten_' -'benötigte_' -'benen' -'benchmark' -'benachteiligt' -'benachbarten_' -'bemerken_' -'belong' -'beliefs_' -'beke' -'bekannteste' -'beit' -'beides_' -'behinder' -'begünstig' -'begriffen_' -'begrenzten_' -'begleitet_' -'begeben_' -'bege' -'begangen_' -'beförder' -'befriedigen_' -'befri' -'befrei' -'befa' -'bedeutete_' -'bedding_' -'bedauerlich_' -'bearbeiten_' -'bb' -'bat_' -'bases_' -'base' -'barri' -'banner' -'banned_' -'bang_' -'bailout_' -'bai_' -'ays_' -'aya' -'aw_' -'avoiding_' -'aviation_' -'avel' -'außergewöhnlichen_' -'automati' -'author' -'auszuw' -'auszul' -'auszuarbeiten_' -'ausstatt' -'aussi' -'ausser' -'auss' -'ausreichende' -'ausr' -'ausp' -'ausn' -'ausgest' -'ausges' -'ausgehend_' -'ausgedehnt' -'ausfallen_' -'ausbr' -'aum_' -'ault_' -'auli' -'aufzuh' -'aufrufen_' -'aufri' -'aufregende' -'aufgez' -'aufgel' -'aufgegriffen_' -'aufgef' -'auferleg' -'aufbe' -'auer_' -'audit_' -'aucht' -'ature_' -'aturan_' -'attr' -'atta' -'atra' -'atori' -'atm' -'atla' -'ativ' -'ata_' -'asy' -'astu' -'ast_' -'assumption_' -'assum' -'assessing_' -'assess_' -'asserti' -'assembly_' -'assembl' -'asse_' -'aspir' -'asks_' -'asa' -'artige' -'artif' -'arsen' -'arrangement_' -'arranged_' -'arm_' -'arität' -'arist' -'arising_' -'arde' -'archives_' -'arbeitende' -'arabische_' -'aqua' -'apt' -'approv' -'appoint' -'apart' -'anzust' -'anzukurbeln_' -'anzugehen_' -'anzuerkennen_' -'anzi' -'antis' -'antiqu' -'antic' -'anta' -'anstieg' -'anstehende' -'anspruchsvolle' -'ansi' -'anny' -'announcement' -'anni' -'anna' -'ann' -'ankomm' -'anische_' -'angry_' -'angre' -'angestrebte' -'angesprochene' -'anges' -'angene' -'angelegte' -'angekündigte' -'angekündigt_' -'angehör' -'angehen_' -'angegeben_' -'angebot' -'angeblich_' -'anfä' -'anen_' -'ands' -'anat' -'analysier' -'analyses_' -'amin' -'amer' -'ambiance_' -'aman' -'alu' -'altogether_' -'alternat' -'allu' -'alls' -'algo' -'aler_' -'alem' -'akzeptabel_' -'aktivist' -'aktiviert_' -'aktiven_' -'aktiv' -'aktion' -'ais' -'aim' -'ahlung_' -'ahlen_' -'ahe' -'ags_' -'agon' -'aggressive' -'aggre' -'after' -'afraid_' -'afi' -'afford' -'afflict' -'advocating_' -'advocates_' -'adventure_' -'adu' -'ads_' -'admitted_' -'administrat' -'adjust' -'adj' -'add' -'acute_' -'actress_' -'acquisition_' -'acle' -'acke' -'aches_' -'ache_' -'ace_' -'accumulati' -'accounted_' -'accessories_' -'accesse' -'abzust' -'abzule' -'abuses_' -'abstain_' -'abstain' -'absor' -'abschl' -'aboard_' -'ablen' -'ablauf' -'abkommens_' -'abgez' -'abgegeben_' -'abandoning_' -'Zwischen' -'Zwi' -'Zweig' -'Zut' -'Zuschauer' -'Zuf' -'Zucker_' -'Zit' -'Zell' -'Zeitschrift_' -'Zeitr' -'Zeilen' -'Zehn_' -'Zar' -'Yuk' -'Yen_' -'Yemen_' -'Yam' -'Xa' -'XVI' -'XLS' -'Wüsten' -'Wür' -'Wälder_' -'Wy' -'Worf_' -'Word' -'Woods_' -'Wissens_' -'Wissens' -'Wirtschaftswachstums_' -'Winters' -'Winds' -'Will' -'Wiener_' -'Widersprüche_' -'Wide' -'Whenever_' -'Wettbewerbe' -'Wertpapiere_' -'Wertpapier' -'Werkzeug_' -'Werkst' -'Werde_' -'Wenige' -'Weltwirtschafts' -'Wellnessbereich_' -'Weiterentwicklung_' -'Weihnachten_' -'Weich' -'Wed' -'Weber_' -'Wave' -'Watt' -'Wasch' -'Warnung' -'Wandels_' -'Wan' -'Wahrnehmung_' -'Wahlkampf' -'Wag' -'Wachstumss' -'WP' -'Völkerrecht' -'Vs_' -'Vorredner' -'Vorre' -'Vorrang_' -'Vorherrschaft_' -'Voraus_' -'Voraus' -'Volume' -'Vitorino_' -'Visu' -'Visa_' -'Vis' -'Vinc' -'Victoria_' -'Via_' -'Verwirklichung_' -'Vertrieb' -'Vertreter' -'Vertrags_' -'Vertrages_' -'Verteidigungsminister' -'Vermögen_' -'Verletz' -'Verlagerung_' -'Verkehrsnetz' -'Verkehrsa' -'Verkaufs' -'Verhältnis' -'Vereinig' -'Verbraucherschutz' -'Verbrauchern_' -'Verantwortlichkeit' -'Vario' -'VIC' -'VE' -'VD' -'Ura' -'Updates_' -'Unterscheidung_' -'Unters' -'Unternehmer' -'Unterfangen_' -'Unst' -'Universal_' -'Unionsbürger' -'Unfälle' -'Underground_' -'Unde' -'Umst' -'Umsatz_' -'Umbr' -'Ultimate' -'Ul' -'Uf' -'USE_' -'UP' -'UNM' -'Türen_' -'Tät' -'Twe' -'Turni' -'Turm' -'Turi' -'Tunnel' -'Tud' -'Tsi' -'Tschech' -'Truppe' -'Troi' -'Tric' -'Tradi' -'Tr' -'Toyota_' -'Ton_' -'Tomo' -'Tom_' -'Toleranz_' -'Tode_' -'Tod' -'Thor' -'Thom' -'Thirdly_' -'Thinking_' -'Theor' -'Theatre_' -'Thal' -'Th' -'Terrace_' -'Terra' -'Tenn' -'Tendenz_' -'Ten_' -'Temp' -'Tell' -'Tehran_' -'Technologie' -'Tay' -'Tausend' -'Tatsachen_' -'Task_' -'Take' -'Table' -'Tabak' -'TP' -'TOS_' -'TION' -'Süde' -'Südamerika' -'Säule' -'Sydney_' -'Superma' -'Sum' -'Sud' -'Subve' -'Substanz' -'Subsidiarität_' -'Stück_' -'Stuttgart_' -'Stufe_' -'Studierende' -'Student' -'Stress_' -'Stock_' -'Sto' -'Stil' -'Stig' -'Stift' -'Sti' -'Steuererhöhungen_' -'Stereo' -'Steigen' -'Stay' -'Statut' -'Statistiken_' -'Station' -'Starts' -'Standort' -'Stamm' -'Stal' -'Stabilitäts' -'Staatsb' -'Staatsa' -'Sri_' -'Sponsor' -'Spenden' -'Spekulation' -'Speed_' -'Spaziergang_' -'Sozials' -'Sozialpartner' -'Souvenir' -'Sonic_' -'Songs' -'Somit_' -'Solutions_' -'Sobald_' -'Slowakei_' -'Slideshows_' -'Sk' -'Sina' -'Simpl' -'Silver_' -'Silv' -'Sil' -'Siedlungen_' -'Sichtweise_' -'Sich' -'Shopping_' -'Sharon_' -'Sex_' -'Seuche' -'Session_' -'Serikat_' -'Seri' -'Sensor' -'Selbstver' -'Selbstbe' -'Sekunden_' -'Sekt' -'Seitens' -'Segel' -'Seg' -'Schü' -'Schönheit_' -'Schä' -'Schwerpunkt' -'Schulb' -'Schra' -'Schmidt_' -'Schlacht_' -'Schiffe_' -'Schichten_' -'Schengen_' -'Schauspieler_' -'Scandinavia' -'Save_' -'Sav' -'Sat_' -'Sanierung_' -'Samu' -'Samstag_' -'Same' -'Saharan_' -'Sah' -'Sag' -'Safe_' -'Sac' -'Sab' -'Saatgut_' -'SOL' -'SC_' -'Rückzug_' -'Rücken_' -'Roth' -'Rollen' -'Ring' -'Rig' -'Ries' -'Richtig' -'Rice_' -'Ria' -'Review_' -'Reu' -'Result' -'Ressourcen' -'Residenz_' -'Residence_' -'Reparatur' -'Rennen_' -'Renditen_' -'Rek' -'Reit' -'Reinh' -'Reihenfolge_' -'Reife' -'Reichtum_' -'Reichs' -'Reich' -'Regulierungsbehörden_' -'Regen' -'Reformp' -'Refle' -'Referen' -'Redebeitr' -'Recovery_' -'Rechtsvorschrift_' -'Rechtsgrundlage_' -'Rechnungshof' -'Rechner_' -'Rechn' -'Recently_' -'Read_' -'Read' -'Raumfahrt' -'Rauch' -'Ras' -'Rang' -'Radisson_' -'RS' -'RP' -'REACH_' -'RC' -'Quin' -'Quart' -'Qi' -'Pä' -'Pyr' -'Putsch' -'Ps' -'Präsidentschafts' -'Provid' -'Protokolls_' -'Prost' -'Promi' -'Produktp' -'Produktivitäts' -'Prinz' -'Print' -'Primär' -'Prima' -'Price_' -'Pres' -'Prag_' -'Posten_' -'Portfolio' -'Populis' -'Polizist' -'Polizeia' -'Poettering_' -'Poe' -'Plugin' -'PlayStation_' -'Plattform_' -'Pir' -'Pipe' -'Philippines_' -'Phil' -'Pfe' -'Persönlichkeiten_' -'Persian_' -'Pec' -'Pazifik' -'Passag' -'Partition' -'Part_' -'Part' -'Parlamente_' -'Parking_' -'Palästinensern_' -'Paketen_' -'Paa' -'PNR_' -'PCs_' -'PA_' -'PAR' -'Otto' -'Osteuropa_' -'Ostasien_' -'Oscar_' -'Ort' -'Oro' -'Orange_' -'Oppositions' -'Operationen_' -'Olympischen_' -'Olympi' -'Office' -'Ocean_' -'Obs' -'Obl' -'Oberflächen' -'OSZE_' -'OM' -'Nähr' -'Nous_' -'Nixon_' -'Nicaragua_' -'Nic' -'Neus' -'Netz' -'Netanyahu_' -'Nes' -'Nenn' -'Navigation_' -'Nau' -'Natural_' -'Nationalen_' -'Namens' -'Nahrung_' -'Nag' -'Nad' -'Nachweis_' -'Nachk' -'NU' -'NPT_' -'NN_' -'NL_' -'NL' -'NET_' -'NAFTA_' -'Mühl' -'Mächte' -'Must' -'Motors' -'Motor_' -'Moro' -'Morgan_' -'Morg' -'Monti_' -'Mont_' -'Mone' -'Monday_' -'Moderni' -'Mittelmeerraum_' -'Mittela' -'Mitgliedsländern_' -'Mis' -'Minister' -'Mind' -'Migu' -'Mexican_' -'Meth' -'Mercosur_' -'Menschenrechten_' -'Meldung' -'Mehrheit' -'Meeting_' -'Medikament' -'Mayo' -'Maximum_' -'Materi' -'Masse_' -'Maschine_' -'Marktk' -'Marken_' -'Marine' -'Marin' -'Mandrake' -'Mandela_' -'Mandel' -'Mand' -'Manchester_' -'Main' -'Maes' -'MU' -'MT' -'MIT_' -'MIL' -'Lyon_' -'Lun' -'Luftverkehr_' -'Los' -'Londoner_' -'Liv' -'Little_' -'Lithuania_' -'Liquiditäts' -'Linz_' -'Linken_' -'Line' -'Limit_' -'Lig' -'Licht' -'Libert' -'Liberia_' -'Liberal_' -'Level_' -'Lev' -'Les' -'Leiter_' -'Leib' -'Legislat' -'Legi' -'Lebensmittelsicherheit_' -'Lebensmitteln_' -'Leb' -'Lay' -'Lauf_' -'Large_' -'Lanzarote_' -'Lane_' -'Landschaft_' -'Lad' -'Labora' -'Labor' -'LL_' -'LCD_' -'LAN_' -'Kurs' -'Kura' -'Kuch' -'Kreis_' -'Kredit_' -'Kosm' -'Kopf' -'Kooperations' -'Konzert' -'Konzentration_' -'Kontakte_' -'Konsultation_' -'Konsolidierung' -'Konse' -'Kongress' -'Konflikt' -'Konfiguration' -'Kompl' -'Kommen' -'Kommando_' -'Kommando' -'Komit' -'Kohä' -'Kofi_' -'Koch' -'Kob' -'Knopf' -'Klä' -'Klick' -'Klarheit_' -'Klang' -'Kirchen' -'Kirch' -'Ki_' -'Khamenei_' -'Kernkraft' -'Kennedy_' -'Kay' -'Kasse' -'Kartell' -'Karibik_' -'Kapitel_' -'Kapitalm' -'Kapazität_' -'Kanten_' -'Kampa' -'Kaiser_' -'KT_' -'KT' -'KP' -'KOM_' -'Jungen_' -'Jugoslawien_' -'Jugendliche_' -'Jugend_' -'Journal_' -'José_' -'Joint_' -'Joe' -'Jia' -'Jeff' -'Jedoch_' -'Jede' -'Jardin_' -'Jane' -'Jahrzehnts_' -'Jahr' -'Isle' -'Islamist' -'Investmentbank' -'Investitionsbank_' -'Interpret' -'Internetseite' -'Interinstitution' -'Integrationsp' -'Instrument' -'Instituts_' -'Institution' -'Insi' -'Innovat' -'Innenhof_' -'Innen_' -'Ingenieure_' -'Informationsschalter_' -'Infolgedessen_' -'Indians_' -'Index' -'Independen' -'Impulse_' -'Importe' -'Immigration_' -'Ig' -'Ideal_' -'Ib' -'IST' -'ION' -'IDE' -'ICC_' -'Händler_' -'Hyde_' -'Hung' -'Hotelsafe_' -'Horn' -'Horde_' -'Hochschule' -'Hochschul' -'History_' -'Hintergr' -'Him' -'Heu' -'Herkunft_' -'Herausgeber_' -'Heim' -'Heat' -'Haushaltsp' -'Haushaltsl' -'Hat' -'Harbor_' -'Handt' -'Handlungen_' -'Handelspartner_' -'Handelsb' -'Handb' -'Halle_' -'Halle' -'Halbinsel_' -'Haiti_' -'Hag' -'Hafen' -'Hab' -'HN' -'HE' -'Gy' -'Gul_' -'Grünen_' -'Gründer_' -'Grundst' -'Grill' -'Graz' -'Gras' -'Gran' -'Grafik' -'Governor_' -'Gouverneur' -'Go_' -'Gleichw' -'Gleichheit_' -'Gleiche' -'Gift_' -'Gewissen_' -'Gett' -'Get' -'Gesundheitsp' -'Gesprächen_' -'Gespräch' -'Gesetzgeb' -'Geschäftsle' -'Geschäftsb' -'Geschäfts_' -'Gerhard_' -'Gent' -'Geni' -'Geneva_' -'Genehmigung_' -'Gene_' -'Gemüse' -'Gemeinschaftsm' -'Gemeinde_' -'Geltung' -'Geis' -'Geheimdienst' -'Gegenzug_' -'Gefühle_' -'Gefä' -'Gefangenen_' -'Geduld_' -'Gebäuden_' -'Gebäude' -'Geburt_' -'Gazastreifen_' -'Garten' -'Garni' -'Gare_' -'GNOME_' -'GM_' -'GEN' -'GC' -'GAP_' -'G8_' -'Führungskräfte' -'Födera' -'Future_' -'Futtermittel' -'Fusion' -'Furcht_' -'Funktion' -'Fuku' -'Fuerte' -'Frühling_' -'Frühjahr_' -'Friedrich' -'Friedman_' -'Friedensnobelpreis' -'Fremden' -'Freie_' -'Fran' -'Frageb' -'Fracht' -'Former_' -'Forge' -'Foot' -'Fon' -'Following_' -'Flüchtlingen_' -'Flächen_' -'Fläche' -'Flugzeug_' -'Fluggäste_' -'Flugg' -'Flotte' -'Florenz_' -'Fli' -'Fisher' -'Fine_' -'Finanzr' -'Finanzinstitute_' -'Finanzielle_' -'Finanz_' -'Files_' -'Fift' -'Few_' -'Fests' -'Festplatten_' -'Festland' -'Ferien_' -'Fels' -'Felder' -'Feind' -'Fei' -'Fea' -'Fav' -'Fasc' -'Fantas' -'Fall' -'Fahrzeug_' -'Fachw' -'FU' -'FPGA_' -'FP' -'FOR_' -'FF_' -'Extremisten_' -'External_' -'Ex_' -'Evo' -'Ev' -'Euros' -'Euch_' -'Eti' -'Etage_' -'Erwerb_' -'Erw' -'Ersten_' -'Ersatz_' -'Ero' -'Erneuerung_' -'Eri' -'Erforder' -'Erdoğan_' -'Erdbeben' -'Erbe' -'Entwicklungsb' -'Entschädigung' -'Entschuldigung' -'Entschließungen_' -'Enhance' -'Engl' -'Energieb' -'Energiea' -'Endes_' -'Employment_' -'Empfang_' -'Electric' -'Eisb' -'Eis' -'Einzig' -'Einver' -'Eintritt' -'Einsparungen_' -'Einschränkungen_' -'Eins' -'Einkaufs' -'Einheiten_' -'Eingreif' -'Einfa' -'Eigentumsrechte' -'Eigenkapital_' -'Eiffel_' -'Eif' -'Ef' -'Economi' -'Ecke_' -'Ec' -'Dänemark_' -'Dut' -'Durban_' -'Drug' -'Drohungen_' -'Dritt' -'Dringlichkeit_' -'Dresden_' -'Drama' -'Download' -'Double_' -'Doll' -'Dokument' -'Document' -'Division_' -'Dist' -'Diskurs' -'Disco' -'Direktinvestitionen_' -'Diplom' -'Dies' -'Dialog' -'Dha' -'Devisen' -'Denkens_' -'Deng' -'Demokratischen_' -'Deli' -'Delhi_' -'Deg' -'Defizite_' -'Decision_' -'Datum_' -'Datenschutz' -'Dat' -'Danke_' -'Dan_' -'Dali' -'DT' -'DNA_' -'Cz' -'Currently_' -'Curren' -'Cubase_' -'Cs_' -'Crown' -'Cross_' -'Crisis_' -'Criminal_' -'Cove' -'Cost' -'Corporate_' -'Corn' -'Cori' -'Copyright_' -'Convenient' -'Contra' -'Continu' -'Connect' -'Competiti' -'Columbia_' -'Color_' -'Colla' -'Cocktail' -'Client' -'Clear' -'Claudi' -'Clar' -'Civi' -'Choose_' -'Chemie' -'Chef' -'Check' -'Charakter' -'Channel_' -'Chame' -'Certain' -'Catholic_' -'Cathedral_' -'Castel' -'Cash_' -'Case_' -'Casa_' -'Casa' -'Carolyn_' -'Carne' -'Cara' -'Capital_' -'Cance' -'Cala' -'Cafés_' -'CS_' -'CSS_' -'COM' -'COD_' -'CNS_' -'CN' -'CHI' -'CAS' -'Burk' -'Bundesregierung_' -'Bui' -'Buche' -'Brutto' -'Brun' -'Bruch' -'Brothers_' -'Brot' -'Broadway_' -'Bring' -'Brid' -'Brea' -'Brazilian_' -'Bou' -'Boris_' -'Bombe' -'Bolivien_' -'Blume' -'Blizzard_' -'Blitz' -'Bisc' -'Bir' -'Biokraftstoffe' -'Bildungss' -'Bib' -'Bh' -'Bezug' -'Bezirk' -'Bevölkerungen_' -'Betriebe' -'Betre' -'Betrag_' -'Bestrebungen_' -'Beste_' -'Besonders_' -'Beseitigung_' -'Beschränkungen_' -'Bergen_' -'Berechtigung' -'Berater_' -'Berat' -'Benzin' -'Benutzer' -'Benutz' -'Bemühen_' -'Belle' -'Bell' -'Beleidigung' -'Beitrittsverhandlungen_' -'Behinderte' -'Behauptung_' -'Begriffe_' -'Begriff' -'Begleiter' -'Begin' -'Befehl' -'Bedauerlicherweise_' -'Bed_' -'Beam' -'Bavaria_' -'Baust' -'Battle' -'Basi' -'Bashir_' -'Bart' -'Barrier' -'Barnier_' -'Barcode_' -'Barcelon' -'Barc' -'Barbara_' -'Banglades' -'Bang' -'Ban_' -'Balkan' -'Baker' -'Bahnh' -'BT' -'BES' -'BA_' -'Außenministeri' -'Autorit' -'Autonom' -'Ausw' -'Ausse' -'Ausmaße' -'Auslös' -'August' -'Augenblick_' -'Auftritt' -'Auftrags' -'Aufsichtsrat' -'Aufsichts' -'Aufschwung_' -'Aufruf' -'Aufpreis_' -'Auflösung_' -'Audio' -'Ath' -'Astrium_' -'Asp' -'Argumentation_' -'Arbeitsweise_' -'Arbeitsk' -'Arbeitsbedingungen_' -'Arbeits_' -'App' -'Anwe' -'Antonio_' -'Anton' -'Anth' -'Anreisedatum_' -'Ano' -'Anhang_' -'Angriffs' -'Angestellte' -'Angehörige' -'Ang' -'Andrew_' -'Andreas' -'Anb' -'Alten' -'Alt_' -'Alpha_' -'Alltag_' -'Allgemein' -'Allen_' -'Alco' -'Alan_' -'Akteur_' -'Akku' -'Aix_' -'Ahn' -'Ahmadinejad_' -'Agr' -'Again_' -'Afrikan' -'Affi' -'Aff' -'Adv' -'Admiral_' -'Adi' -'Add_' -'Add' -'Activ' -'Achse_' -'Academy_' -'Abstand_' -'Abend' -'Abbildung' -'ATT' -'ASPs_' -'API' -'AN_' -'AKP_' -'A4' -'=' -'94' -'91_' -'89' -'86' -'77' -'75' -'74_' -'73' -'61' -'58' -'55' -'500' -'48' -'400' -'34' -'225' -'20th_' -'1976_' -'1970s_' -'197' -'1962_' -'1933_' -'1929_' -'1914_' -'18' -'169' -'168' -'163' -'154' -'152' -'111_' -'108' -'0s_' -'05' -'006' -'.. _' -'.-_' -', [_' -'*' -'), "_' -'))_' -') , _' -'''' ' -'&#_' -'"-_' -'")' -' :' -' // _' -' ...' -' ,' -' *' -' (* _' -' ''[[_' -' #' -' _' -'„' -'“ ' -'— _' -'ь_' -'ше' -'ци' -'х' -'ф' -'тт' -'сти' -'от' -'он' -'ол' -'о_' -'за' -'же' -'ді' -'ды' -'ан' -'С' -'И' -'τ' -'ša' -'Š' -'ław_' -'če' -'ć_' -'ütz' -'üt' -'üstung' -'üste_' -'üste' -'ürzung' -'ürzt' -'ünst' -'ünder' -'ünde' -'üllen_' -'ührt_' -'ühre' -'ügliche' -'üchte_' -'ücht' -'übl' -'überwunden_' -'überwiegen' -'übertrieben_' -'übersteig' -'überraschend_' -'übern' -'übermäßig_' -'übermittelt_' -'überlegt_' -'übergreifende' -'überga' -'überein_' -'üben_' -'ún_' -'úl' -'ösung_' -'östlich' -'österreichische' -'öpfen_' -'öne' -'öhne' -'öhn' -'ögen' -'öfe_' -'öder' -'ño_' -'ère_' -'çais' -'äuter' -'äumt_' -'ätzlich' -'äte_' -'äte' -'ässer' -'äse' -'ärts' -'ärs' -'ärme' -'äni' -'ängen' -'ändigkeit_' -'ändig_' -'änderungen_' -'änderung_' -'äme' -'älteste' -'ältere_' -'äle' -'ähnlicher_' -'ähnel' -'ähigkeit_' -'ägyptische_' -'äglich' -'ägige' -'ât' -'ßte' -'ßnahmen' -'Überw' -'Übersetzungen_' -'Überschw' -'Überraschung' -'Überprüf' -'Überlegen' -'Überein' -'Üb' -'Ölpreise_' -'Ökosystem' -'Ökonom' -'Öffentliche_' -'Öff' -'Ängste_' -'Än' -'Ähnlich' -'©' -'    ' -'}}) {{_' -'}}' -'}' -'zögern_' -'zzo_' -'zze' -'zy_' -'zwischenstaatlichen_' -'zwecke' -'zwangsläufig_' -'zuwei' -'zuversichtlich_' -'zuverlässig' -'zuteil_' -'zut' -'zuständige_' -'zuste' -'zusammensetz' -'zusammengebr' -'zurückzukehren_' -'zurückl' -'zurückgreifen_' -'zurückgeh' -'zurückgeg' -'zur' -'zuläss' -'zuk' -'zugestimmt_' -'zugefü' -'zt' -'zqu' -'zoom_' -'zna' -'zitier' -'zini' -'zin_' -'zig' -'zielle' -'ziell_' -'ziehungen_' -'ziehung' -'zeugen_' -'zeuge' -'zentrums_' -'zellen' -'zeitwei' -'zeitung' -'zeichnet_' -'zehnte' -'zed' -'zahlung_' -'zahler' -'zad' -'yw' -'yto' -'ypto' -'yl' -'ybo' -'xw' -'xt' -'xit' -'xin' -'xenophob' -'xe_' -'xamp' -'würdigkeit_' -'wünschte' -'wöhnlich' -'wäsche_' -'wärtig' -'wunderschöne_' -'wound' -'worsen_' -'wors' -'workforce_' -'wora' -'woody_' -'wollend' -'woll' -'wohlhabende' -'wle' -'wl' -'wishing_' -'wirtschafts' -'wirkungs' -'wirksamere' -'wirksamen_' -'wirksam' -'wing' -'willi' -'wil' -'wig' -'wiederherzustellen_' -'widerst' -'widening_' -'wick_' -'whit' -'westliche' -'weste' -'west' -'wertvoll' -'werb' -'wendet' -'wem' -'weiße' -'weitreichende' -'weitergehen_' -'weig' -'weifel' -'wehr' -'week' -'wedding' -'weakness_' -'weaken_' -'wea' -'wast' -'washing_' -'wary_' -'warnings_' -'wandel_' -'walke' -'wald_' -'wahrnehmen_' -'waffen' -'wad' -'wab' -'völker' -'vé' -'vä' -'vy_' -'vri' -'vou' -'vorübergehend_' -'vorzubereiten_' -'vorstellung' -'vorste' -'vorsitzende' -'vorschriften_' -'vorliegen_' -'vorkommen_' -'vork' -'vorige' -'vorhers' -'vorher' -'vord' -'vorbereitet' -'vorbereiten_' -'vorbei' -'vorange' -'vons' -'volu' -'voltage_' -'vollem_' -'volcan' -'vocational_' -'voc' -'visuali' -'visor' -'vision' -'visib' -'vine' -'vigor' -'vielerlei_' -'vici' -'vibrant_' -'veränderten_' -'verzögert' -'verzweifelt' -'verzichte' -'verweist_' -'verweigern_' -'verwalten_' -'veru' -'vertreter_' -'vertreib' -'vertraue' -'vertrags_' -'vertraglich' -'verteilung_' -'verstärkten_' -'verständ' -'verstoßen_' -'verstorben' -'versprochen_' -'versprechen' -'verspr' -'versicherung' -'verschärfen_' -'verschwunden_' -'verschwende' -'verschuld' -'verscho' -'verschmutz' -'verschi' -'verringerte' -'vernünftigen_' -'vernünftige_' -'vernünftig_' -'vermeid' -'vermehrt' -'verma' -'verm' -'verleite' -'verleg' -'verlaufen_' -'verla' -'verkörpert_' -'verkn' -'verhandeln_' -'vergrößern_' -'vergr' -'vergleichen_' -'vergeb' -'verfolgten_' -'verfasst' -'verfallen_' -'vereinen_' -'vereinbarung' -'vereinbart_' -'vereinbar' -'verei' -'verehrte' -'verda' -'verbot_' -'verbleibenden_' -'verbindlichen_' -'verarbeitet' -'verarbeiten' -'verantwortungs' -'veranstaltungen_' -'veranlassen_' -'veraltet' -'verabschiedeten_' -'venue' -'ventu' -'vehement' -'vea' -'variables_' -'valu' -'validity_' -'valid' -'vai' -'vague_' -'vage' -'vaccine' -'vaca' -'vac' -'uzz' -'uum' -'utze' -'utung' -'utuh' -'utu' -'utter_' -'utter' -'utt' -'utilis' -'utig' -'utan' -'ustan' -'ussion' -'usschuss_' -'uso' -'usly_' -'usher' -'ush' -'usan' -'urteile' -'urte' -'urt_' -'urso' -'urlaub_' -'uris' -'urie' -'ured_' -'uras_' -'uptc' -'upper_' -'uppe' -'upl' -'uph' -'unüber' -'unwind_' -'unvorher' -'unverzichtbar_' -'unvers' -'unvergessliche' -'unterziehen_' -'unterschiedlicher_' -'unterl' -'unterhält_' -'untere' -'unterbreite' -'unsicher' -'unsi' -'unse' -'unrec' -'unrealisti' -'unpopul' -'unp' -'unnötige' -'unke' -'units_' -'unit' -'unilateral_' -'uniform_' -'unification_' -'ungswe' -'ungsv' -'ungsl' -'ungsgr' -'ungsan' -'ungewöhnliche' -'ungene_' -'ungeb' -'unga' -'unfähig_' -'unfo' -'unerlässlich_' -'undin' -'undertakings_' -'undertake_' -'underpinn' -'undermined_' -'underline_' -'unda' -'unconventional_' -'uncom' -'unchanged_' -'unberührt' -'unanimity_' -'unal' -'unabhängiges_' -'umweltfreundlich' -'umsetzen_' -'umg' -'umfeld' -'umfassender_' -'umfassend_' -'umfangreicher' -'umfang_' -'umd' -'umben' -'ultur' -'ultimative' -'uls' -'ule_' -'uldung' -'ular' -'uing_' -'ugh_' -'ufhin_' -'ufer' -'ued_' -'uder' -'uda' -'uci' -'uation_' -'tü' -'töt' -'tödliche_' -'täusch' -'tändig' -'tzte_' -'typen_' -'two' -'twist' -'twelve_' -'twar' -'turn' -'turbulen' -'tums' -'tumor' -'tue' -'tuc' -'tual_' -'ttung' -'tto' -'ttler_' -'ttet_' -'tsu' -'träglich' -'trust' -'trump' -'trugen_' -'tropical_' -'triple' -'tries_' -'trial' -'trend' -'treibende' -'treib' -'trei' -'tre_' -'travelling_' -'traveller_' -'trav' -'traurige' -'trategie_' -'transported_' -'transporte' -'transport' -'transparenter' -'transpar' -'transnationale' -'translations_' -'translat' -'transform' -'transferring_' -'trans_' -'train' -'tragi' -'trafe' -'traditione' -'traders_' -'trademark_' -'traded_' -'tracking_' -'toute' -'tout_' -'tourists_' -'touristi' -'toughe' -'touching_' -'touched_' -'totalitäre' -'torture_' -'toren_' -'tooth' -'toleran' -'toc' -'tne' -'tle_' -'tland' -'tk' -'tious' -'tionier' -'tine_' -'timetable_' -'tigt_' -'tigkeit' -'tiert' -'tiefe' -'tiat' -'thrill' -'thre' -'thr' -'therm' -'therapeuti' -'theorie_' -'theme' -'theit_' -'theatre_' -'theatr' -'thal' -'texte' -'teure' -'teu' -'tete' -'tet' -'testi' -'ters' -'terrasse_' -'terp' -'tenure_' -'tentang_' -'tenen_' -'tender' -'tendenziell_' -'temptation_' -'temples_' -'tem_' -'tellt' -'tellen_' -'telefoni' -'tein' -'teilnehmenden_' -'teilgenommen_' -'teigerung' -'teg' -'teen' -'tee' -'technologische_' -'technike' -'tched_' -'tausche' -'tariff_' -'tante' -'tanta' -'tanding' -'talked_' -'talis' -'talent' -'tad' -'tackl' -'sämtliche' -'sze' -'systemische' -'synt' -'synchronisier' -'switching_' -'swer' -'swell' -'sw' -'svor' -'survive' -'surro' -'surrender' -'supranational_' -'supposedly_' -'supervisor' -'superf' -'sung' -'summari' -'suits_' -'sue' -'sudden_' -'sud' -'successive_' -'subversi' -'subscribe' -'submi' -'sua' -'stützt_' -'stür' -'stät' -'ständige_' -'ständige' -'styles_' -'studierte_' -'stud' -'strukturellen_' -'structured_' -'strom_' -'stroll_' -'stricte' -'stress' -'strengen_' -'strafe_' -'stoßen_' -'stom' -'stol' -'stoffen_' -'stock' -'stl' -'stitch' -'stirbt_' -'stilvolle' -'still' -'steuerung_' -'steuerlichen_' -'stetig' -'sters_' -'steril' -'stenz' -'stehe' -'statu' -'stattgefunden_' -'stattfindenden_' -'statistische' -'statistical_' -'stationen_' -'stat' -'start' -'starker_' -'stando' -'standardiz' -'stan' -'staltung_' -'stagnieren' -'stagnat' -'stagn' -'stabiler_' -'stabil_' -'staatlich' -'ssysteme' -'ssungs' -'sstsein' -'ssing_' -'ssige' -'ssan' -'ssa_' -'sreg' -'sre' -'sq' -'späteren_' -'spyware_' -'spur_' -'spur' -'sprü' -'sprache_' -'spr' -'sporting_' -'spontane' -'sponsor' -'spiral_' -'spin' -'sphäre' -'spen' -'spektakulär' -'speedi' -'speculat' -'spars' -'sozialistische' -'souveränen_' -'sorte' -'soph' -'sonstige_' -'solving_' -'solide' -'sole' -'sola' -'sogenannte_' -'soft' -'soeben_' -'socialist_' -'soc' -'sob' -'soared_' -'snowb' -'snow_' -'snack' -'smooth' -'sly_' -'slower_' -'slot' -'slos_' -'slippe' -'slides_' -'slic' -'sl' -'sky' -'skill_' -'skill' -'skand' -'ska_' -'sk_' -'siz' -'sive' -'siv' -'sitzung' -'sition' -'sinnlos' -'sinn' -'singles_' -'sine' -'simplifi' -'simple' -'signifikant' -'sightseeing_' -'sighted_' -'sien' -'sid_' -'sicht_' -'sicht' -'sichert_' -'shoulder_' -'shou' -'shortages_' -'short' -'shooting_' -'shoot' -'shocked_' -'shock' -'shim' -'shifted_' -'shi' -'shar' -'sew' -'severely_' -'sever' -'settle' -'seti' -'servant' -'serta_' -'sers_' -'seren' -'serbische' -'sequenc' -'seperate' -'senti' -'sensors_' -'sensor_' -'senk' -'selecting_' -'sele' -'selber_' -'seja' -'seize_' -'seekers_' -'securing_' -'secured_' -'sect' -'secretar' -'seating_' -'seasons_' -'scre' -'scra' -'score' -'schöner_' -'schätzungsweise_' -'schätzt_' -'schädliche' -'schwierige_' -'schwache' -'schw' -'schrä' -'schreckliche' -'schreck' -'schnell' -'schn' -'schmutz' -'scheitern_' -'scheduled_' -'schedule_' -'scharfe' -'schaftung_' -'schaftl' -'schaff' -'sceptic' -'scenes_' -'scenario_' -'scape' -'scann' -'scandal_' -'scan_' -'sber' -'sati' -'samt_' -'samples_' -'sam_' -'salaries_' -'saison' -'sahen_' -'sade' -'saddle' -'sacrific' -'sack' -'sach' -'röme' -'räumen_' -'räum' -'räsident' -'räfte_' -'räch' -'rá' -'rust_' -'runde_' -'rulers_' -'ruk' -'ruin' -'ruhiger_' -'ruhe' -'ructi' -'rtuni' -'rtu' -'rtain' -'rows_' -'row' -'route' -'rot_' -'rosy_' -'ropriate' -'romantische' -'rom_' -'rojekt' -'rohstoff' -'roduc' -'robuste' -'rmen_' -'rlich_' -'rlich' -'rland' -'rke_' -'rivers_' -'rival_' -'rival' -'rity_' -'rism' -'riskante' -'risch' -'ringt' -'rimier' -'ril' -'rike' -'rika' -'rift_' -'ries' -'riert_' -'rieren_' -'rien_' -'rider' -'richtig' -'richtete_' -'richt_' -'riches' -'rial' -'rhythm' -'rfen_' -'rfe' -'revised_' -'review' -'reuni' -'rette' -'retre' -'retired_' -'retire' -'reti' -'restriktive' -'restricti' -'restoring_' -'restaur' -'respon' -'resid' -'resc' -'repu' -'repräsentieren_' -'reproductive_' -'reprodu' -'repri' -'replacement_' -'replace' -'repetiti' -'repeal' -'repar' -'repai' -'renovierte_' -'renn' -'reng' -'renamed_' -'removal_' -'rement' -'remember' -'reme' -'rema' -'relying_' -'reluctant_' -'rejects_' -'rejecting_' -'reise_' -'reis' -'reint' -'reinsta' -'reinforced_' -'reicht' -'reicher_' -'regulieren' -'regulators_' -'regrettabl' -'registrieren_' -'regierung' -'regieren' -'refund' -'refuge_' -'refrain' -'refor' -'redirect' -'redefin' -'recycle' -'recreational_' -'recoveri' -'reconst' -'recognizes_' -'recognis' -'rechtmäßig' -'rechtl' -'rebalancing_' -'reassure_' -'realm_' -'realize_' -'realist' -'realis' -'readers_' -'reactor' -'rdi' -'rde' -'rce' -'rca' -'rben_' -'rbeiten_' -'rba' -'ray_' -'raw' -'rativ' -'rategie_' -'rate' -'rar_' -'ranking' -'rane' -'random_' -'rand_' -'rand' -'rance' -'ralis' -'ral' -'raft' -'radioa' -'radikalen_' -'radikale_' -'rade' -'rada' -'quota' -'quot_' -'quet_' -'quet' -'quem' -'quarters_' -'quantity_' -'qualitativ_' -'qualit' -'qualifizierten_' -'qualifiziert' -'quaint' -'pursu' -'puede' -'publik' -'publici' -'public' -'pub_' -'ptu' -'ptions_' -'psychiatris' -'pse' -'prüfung_' -'präsent' -'provin' -'prototype' -'protectionism_' -'prospe' -'proportiona' -'prompt' -'promote' -'prom' -'prolonged_' -'projection' -'prohibiti' -'programm' -'profitiert_' -'profitable_' -'profitability_' -'professionellen_' -'profess' -'prof' -'produzierende' -'produkti' -'produ' -'problematisch_' -'problematic_' -'privileg' -'prioriti' -'print' -'pries' -'pretend_' -'presu' -'prest' -'pressing_' -'pressed_' -'preside' -'present' -'prerequisite_' -'prer' -'preparatory_' -'preparations_' -'prematurely_' -'preliminary_' -'preiswerte' -'preferences_' -'predict' -'preci' -'prech' -'precarious_' -'praxis' -'praktizieren' -'praktikable' -'prakti' -'praise_' -'pragmatische' -'practic' -'pph' -'pped_' -'ppe_' -'pp_' -'powered_' -'pour' -'postpone' -'posted_' -'possess' -'positiv' -'portugiesische' -'portrait' -'portion_' -'porn' -'populistischen_' -'pok' -'pois' -'poet' -'plötzliche' -'plate_' -'plastic' -'plas' -'planes_' -'plain_' -'plag' -'placem' -'plac' -'pixels_' -'pir' -'pio' -'pilgrim' -'pid' -'pia' -'physicians_' -'philosophi' -'phenomena_' -'pf_' -'pez' -'peu' -'petani_' -'pet' -'pest' -'perver' -'pertan' -'personnes_' -'personality_' -'permissi' -'perlu_' -'perl' -'perish' -'perf' -'perba' -'pera' -'penge' -'penetration_' -'penduduk_' -'pendidikan_' -'pendi' -'penalties_' -'pembe' -'pei' -'peg' -'pec' -'pd' -'pav' -'patents_' -'pate' -'pat' -'paste' -'passt_' -'passion_' -'partic' -'parti' -'pari' -'parameter_' -'pand' -'pal_' -'pair' -'painter' -'paint_' -'pact' -'packen_' -'pac' -'oßen' -'oxida' -'oxi' -'ox' -'ovi' -'overt' -'overlooks_' -'overlooked_' -'overlook_' -'overflow_' -'overco' -'outweigh' -'outlook_' -'outermost_' -'otic_' -'oster' -'orsch' -'ors' -'orische' -'oris' -'orin' -'organize_' -'organisierte' -'orene' -'ordnungsp' -'ordnet_' -'orderly_' -'orde' -'oration_' -'opu' -'optionen_' -'optimiert_' -'oppos' -'opia' -'oph' -'opfer' -'operier' -'operativen_' -'oot_' -'ook_' -'onsk' -'onna' -'omy_' -'omo' -'ologischen_' -'ollst' -'olla' -'olis' -'olic' -'oler_' -'olat' -'oh_' -'offs_' -'offre' -'offizieller_' -'offenbar' -'odds_' -'oda' -'och' -'ocean_' -'oce' -'obsess' -'obs' -'oblem_' -'objecti' -'obersten_' -'oberst' -'oberh' -'obat' -'oad' -'nützliche' -'nötige_' -'nördlich' -'nö' -'nähere' -'nya' -'nw' -'nuts_' -'nut' -'ntwicklung_' -'ntl' -'ntif' -'nta_' -'nswert_' -'nste' -'nsp' -'nschl' -'nsche' -'nre' -'nqu' -'npr' -'nov' -'notable_' -'north' -'norms_' -'norm_' -'nommene' -'nomen' -'nod' -'nnen' -'nme' -'nly_' -'nle' -'nkt' -'nji' -'niños_' -'niv' -'nisch_' -'nießer' -'niert' -'nico' -'nick' -'ngun' -'ngt_' -'ngg' -'nges_' -'ngeb' -'nfalls_' -'newer_' -'nevertheless_' -'neutrali' -'neutral_' -'neuartige' -'netzwerk' -'nes' -'neoliberal' -'nennt_' -'neighborhood_' -'nehmbar_' -'negotiators_' -'near' -'ndt' -'ndre' -'ndlung' -'ndere' -'ncier' -'nchi' -'navigati' -'navigate_' -'nav' -'nationals_' -'nationalists_' -'nata' -'nast' -'nas_' -'narrative' -'nament' -'nam' -'nachweis' -'nachteilig' -'nachkommen_' -'nachgewiesen' -'nachdenken_' -'nable_' -'mündliche_' -'mäßigkeit_' -'mäler' -'mysti' -'mutual' -'musik' -'musical_' -'multipl' -'multilaterale' -'multic' -'mption_' -'mpfe' -'moti' -'monst' -'monet' -'momentum_' -'module_' -'modu' -'modernem_' -'modell' -'mobiliz' -'mmungen_' -'mml' -'mmer' -'mmen_' -'mmel' -'mliche' -'mle' -'mittelfristig' -'mitigat' -'mitget' -'mission' -'missed_' -'misg' -'mische' -'mins' -'ministry_' -'minimize_' -'mines_' -'miner' -'mik' -'mig' -'mia_' -'mia' -'mg' -'mers_' -'mern_' -'merkwürdig' -'mercury_' -'merchan' -'merat' -'mera' -'meny' -'menti' -'menschlicher_' -'mengha' -'meng' -'memu' -'memo' -'memi' -'meldungen_' -'meister_' -'meist' -'meint' -'mehrmals_' -'meets_' -'medication' -'mechanisch' -'mean' -'mbl' -'maß_' -'matik' -'materiellen_' -'mate_' -'masih_' -'masalah_' -'marriage_' -'marri' -'marktor' -'markieren_' -'marke' -'marit' -'marina_' -'mare' -'manufactured_' -'manual_' -'mankind_' -'manipulati' -'manipulated_' -'manifest_' -'manden_' -'mammals_' -'mainst' -'magneti' -'machine' -'löschen_' -'läufe' -'längerfristige' -'längere_' -'lz_' -'lys' -'luxu' -'lun' -'luggage_' -'luft' -'ludi' -'lud' -'ltig' -'lth' -'lte' -'lovers_' -'lof' -'locken' -'lock' -'locally_' -'loca' -'loads_' -'llung' -'llin' -'llige' -'llia' -'lizi' -'litera' -'lisiert_' -'lion_' -'liner_' -'lineare' -'linder' -'lige' -'lifestyle_' -'lif' -'lieferte' -'lieben_' -'lichste_' -'licence_' -'libysche' -'liberty_' -'liaison_' -'lh' -'lg' -'letzte' -'letz' -'letters_' -'lessly_' -'lernt' -'lerat' -'lep' -'leistungsfähig' -'leistet_' -'leiden' -'leichte_' -'lehnt' -'legitim' -'legislator' -'lebend' -'lear' -'league_' -'lder_' -'lde_' -'laying_' -'laya' -'laus' -'launch' -'lauf' -'lation_' -'las' -'landm' -'landing_' -'lain' -'laime' -'labeled_' -'label' -'künft' -'köstliche' -'könig' -'ky' -'kwa' -'kungen_' -'kultur' -'ktr' -'ktivitäten_' -'ktionen_' -'kter' -'kst' -'ksch' -'kreativ' -'krati' -'krat' -'krank' -'kra' -'kostspielige' -'kostenloses_' -'kostenlose' -'kostengünstig' -'kostenfreie' -'korrekte_' -'koordinierte' -'koordinieren_' -'konzer' -'konvertier' -'konvention' -'konte' -'konstruktiv' -'konsequente_' -'konsequent_' -'kons' -'kono' -'konferenz_' -'komplizierter_' -'kompatib' -'kombinier' -'kollektive' -'kohlenstoffarme' -'kod' -'knüpf' -'know' -'klu' -'kleinste' -'klargestellt_' -'klare' -'kis' -'kirch' -'kir' -'kill' -'kidnapp' -'ketten_' -'kese' -'kepe' -'kepada' -'kens' -'kende' -'kemu' -'keln' -'kei' -'kea' -'kaya' -'kategori' -'kare' -'kapital_' -'kant' -'kanis' -'kand' -'kanadische' -'kamer_' -'kali_' -'kala' -'kah_' -'jut' -'junt' -'jud' -'jos' -'joga' -'jin' -'jeweilige_' -'jeti' -'jes' -'jeopardi' -'jenseits_' -'jenige_' -'jenem_' -'jat' -'jalan' -'jahre' -'jadi' -'jack_' -'ière' -'ivo' -'ivit' -'ius_' -'itäten_' -'ität' -'itting_' -'itter' -'itra' -'itor' -'itive' -'itier' -'itet_' -'ites_' -'italienische_' -'isung_' -'ister_' -'isten' -'ist' -'isoliert_' -'isk' -'isieren_' -'ished_' -'irrational' -'irgendeinem_' -'irgendeine' -'ipati' -'iones_' -'invoke' -'invites_' -'invited_' -'invitation_' -'investor_' -'investigat' -'intri' -'intra_' -'intl' -'intimate_' -'interpret_' -'interne' -'intermediar' -'interiors_' -'interinstitutionelle' -'interfere_' -'interessante' -'interessant_' -'interess' -'interaktive' -'inter_' -'intent_' -'intensivier' -'intelligente' -'insula' -'instrumente' -'instructi' -'institutionen_' -'institute_' -'instan' -'installations_' -'instabile' -'inspiration' -'insisting_' -'insisted_' -'insig' -'insert_' -'insect' -'inoffiziell' -'innovativen_' -'inneren_' -'innere_' -'inner' -'inklusive_' -'injured_' -'inige' -'inhuman' -'inherent_' -'inhaber_' -'inh' -'ings' -'ingredient' -'ingl' -'inger_' -'inga' -'infrastruktur' -'informellen_' -'influences_' -'infla' -'infer' -'ined' -'industrializ' -'indlich' -'individuell_' -'individually_' -'indis' -'indirectly_' -'indigenous_' -'indifferen' -'indicating_' -'indicat' -'inconsisten' -'incompeten' -'incom' -'inal' -'inakzeptabel_' -'importiert' -'impo' -'implement' -'impetus_' -'immunity_' -'imen_' -'imate' -'imaging_' -'ima_' -'ima' -'ilung' -'ilm' -'illustr' -'illiberal' -'illegale' -'illegal' -'ilität' -'ilita' -'ilie' -'ili_' -'ilen_' -'ileg' -'ilat' -'ilan' -'ikation' -'ige' -'iga' -'ifft_' -'iffen_' -'ifen_' -'ießen_' -'ieß_' -'ierungsa' -'iens_' -'ielt_' -'iele' -'iegs' -'iede' -'iebene' -'ieb_' -'idle' -'idier' -'ider_' -'ider' -'identit' -'identisch' -'identifiziert_' -'ideally_' -'idad_' -'icon' -'icken_' -'ick_' -'ichtigt' -'iche' -'ibut' -'ibu' -'iat' -'iali' -'höhe' -'höh' -'höchstens_' -'hö' -'hän' -'häl' -'hypocrisy_' -'hydrocarbon' -'hungry_' -'humo' -'humanitären_' -'hts' -'hrs' -'house' -'hostile_' -'hostage_' -'horse_' -'horn' -'hore_' -'hopeful' -'honor_' -'homosexual' -'hofft' -'hof' -'hoc_' -'hnen' -'hman' -'hkan_' -'historically_' -'hip' -'hing' -'hinein' -'hindern_' -'hinder_' -'hinausgeh' -'hilfsbereit_' -'hilfs' -'hilfen_' -'hila' -'highlight' -'hierarch' -'hful' -'heti' -'hetero' -'hervorr' -'hervor' -'hers' -'herrschaft_' -'herein_' -'herd' -'herbeizuführen_' -'herausstellen_' -'heran_' -'herab' -'hent' -'hens' -'hend' -'hemi' -'heme' -'heirate' -'hedge' -'hed' -'hebt_' -'healing_' -'hea' -'hazard' -'haushaltspolitische' -'haupt' -'haul_' -'hath_' -'hase_' -'hars' -'harmonisiert' -'harmonis' -'hari_' -'harbour_' -'har_' -'hang' -'handl' -'handicap' -'hag' -'hace' -'haber_' -'habe' -'güter_' -'gänge' -'gw' -'guy' -'gur' -'gungs' -'gsa' -'gründ' -'grundsätzliche' -'grundlegend' -'großzügige_' -'groundwater_' -'grossen_' -'grosse' -'griffen_' -'griff_' -'grenzübergreifende' -'grenz' -'green' -'greed' -'grave_' -'grat' -'grass_' -'graphic_' -'graph' -'gram_' -'grafische' -'graduate' -'grade' -'glä' -'glich' -'gleichberechtigt' -'glaubten_' -'glas_' -'gische_' -'girl' -'giga' -'gifts_' -'gian' -'ghan' -'ggen' -'geza' -'gewor' -'gewohn' -'gewissem_' -'gewinnt_' -'getä' -'getro' -'getre' -'getestet_' -'geta' -'gesunken_' -'gesund' -'gestützt_' -'gestellten_' -'geste' -'gespräche_' -'gesp' -'geschätzte' -'geschäfte_' -'geschä' -'geschenkt_' -'geschah_' -'gescha' -'gesamte' -'geräten_' -'gerisch' -'gerichtete_' -'gerech' -'gere' -'geplanten_' -'gepa' -'geordnet' -'geographische' -'gent_' -'genocide_' -'genk' -'genia' -'generieren_' -'generator' -'genehmigt' -'gene_' -'gende_' -'genaue_' -'gemütliche' -'gemeldet' -'gemeinsames_' -'gemachte' -'geltend_' -'gelenk' -'geleitete' -'gelegene_' -'geleg' -'gelb' -'gekü' -'gekämpft_' -'geiz' -'gei' -'geho' -'geheim' -'gefällt_' -'gefährlich' -'gefä' -'gefasst_' -'geeignet' -'geehrte_' -'gedie' -'gede' -'ged' -'gebraucht_' -'gebot' -'gebor' -'gebildete' -'gebene' -'gb' -'gau_' -'garr' -'garette' -'garantierte' -'gant' -'gang' -'gaming_' -'galt_' -'galax' -'gag' -'fürchten_' -'fünfte' -'fühlte' -'fähige' -'fähig' -'furthermore_' -'furt' -'furnishings_' -'funktionierende' -'funktionen_' -'fund' -'fun' -'fulfilling_' -'fuer' -'ftl' -'frustrated_' -'fru' -'frischen_' -'friend' -'friedlich_' -'freundschaft' -'freundlich' -'fres' -'freight_' -'freies_' -'französischer_' -'frankly_' -'fotografi' -'foster' -'fortgeführt_' -'formulier' -'formulate' -'formul' -'formel' -'foreg' -'forecasts_' -'foo' -'folk' -'foc' -'fläch' -'fluss_' -'flourish_' -'flour' -'flotte' -'flos' -'florier' -'flora_' -'flop' -'flood' -'fliegen_' -'fleisch' -'flas' -'fishermen_' -'fischen_' -'fis' -'firma' -'firewall' -'fires_' -'finest_' -'find' -'financially_' -'filter_' -'filter' -'fika' -'fighters_' -'fifteen_' -'fierce' -'fielen_' -'ffnung' -'ffende' -'ffel' -'ffe_' -'festlegen_' -'festivals_' -'festi' -'festges' -'festgehalten_' -'fes' -'fertigte' -'fertige' -'fere' -'fens' -'feine' -'feindlichen_' -'fehlge' -'fed' -'fast' -'fassung' -'fass' -'fashion' -'fascis' -'fare' -'farb' -'faktisch' -'fairer_' -'fad' -'factories_' -'ezieh' -'ez' -'exzellente' -'extremist' -'extraordinarily_' -'extracti' -'extern' -'extends_' -'exposed_' -'exporter_' -'exploring_' -'explains_' -'expire' -'experiments_' -'expedi' -'expectation_' -'existent' -'execute_' -'exe_' -'excite' -'excessively_' -'excel' -'examined_' -'evil_' -'eventual_' -'eve_' -'evade' -'europaweite' -'europaweit_' -'europa' -'europ' -'eur' -'etzungs' -'ette' -'ett_' -'etr' -'etisch' -'eties_' -'etie' -'etho' -'ethnische_' -'ethischen_' -'etan' -'estimat' -'ester_' -'espo' -'esh' -'esco' -'escalati' -'erörtert_' -'erörtern_' -'eröffnete' -'erzähl' -'erzeugte' -'erwähnte_' -'erwies_' -'erweiterte_' -'erweise_' -'erwartete' -'erw' -'erve' -'erto' -'ertig' -'erstes_' -'erstaunliche' -'erstaun' -'erstattung' -'erspar' -'erschien_' -'erschaffen_' -'eros' -'erobe' -'ernsthaften_' -'erni' -'erneute_' -'erneuerbaren_' -'erneuerbare_' -'ermu' -'erledigen_' -'erlan' -'erj' -'erinner' -'erier' -'erhöhungen_' -'erhobene' -'erhalte' -'ergänzende' -'ergen' -'ergebnisse' -'ergeb' -'erfreu' -'erfolgreicher_' -'erfahrene' -'erfahr' -'ereigne' -'erect' -'erbr' -'eras' -'erarbeitete' -'equal' -'episodes_' -'env' -'entspannten_' -'entsende' -'entschuldig' -'entries_' -'entrant' -'entrance_' -'entm' -'entlich_' -'entities_' -'entirety_' -'enthusiasm_' -'entha' -'entgehen_' -'entgegen' -'enswerte_' -'ension' -'ensh' -'enriche' -'enp' -'enorm' -'enl' -'englis' -'engineers_' -'engaging_' -'engagieren_' -'enforced_' -'energi_' -'energi' -'enduring_' -'endung' -'endorsed_' -'endi' -'endete' -'endemi' -'ency' -'encryption_' -'encouragement_' -'encountered_' -'encompassing_' -'encompass' -'ena_' -'empower' -'empl' -'empiri' -'empha' -'empfind' -'empfiehlt_' -'empfehle' -'empe' -'emotional' -'emer' -'embryon' -'embass' -'embarrass' -'eman' -'eln' -'ellungs' -'ellte' -'elle' -'elin' -'elig' -'elevated_' -'elevat' -'elemente' -'elem' -'eleganten_' -'elegan' -'elan' -'ektiv' -'ektion' -'eks' -'eke' -'eitige' -'eitet_' -'eisung' -'eisen' -'eis_' -'einzutreten_' -'einzuh' -'einzugehen_' -'einzub' -'einzigartige_' -'einzigartig' -'einz' -'einw' -'eintritt_' -'einstimmig_' -'einsti' -'einstellungen_' -'einste' -'einsa' -'einrichten_' -'einkommens' -'einhergehen' -'einher' -'einheimischen_' -'eingest' -'eingesp' -'eingeschl' -'eingeräumt_' -'eingeleitet_' -'eingeladen_' -'eingehe' -'einfl' -'eindruck' -'eindeutige' -'einander' -'eih' -'eigen_' -'eigen' -'eift' -'eien_' -'eichnet_' -'ehrung' -'ehrgeizige_' -'ehrgeizige' -'ehn' -'ehmen_' -'ehens' -'ehe_' -'egg_' -'efor' -'eff' -'ef_' -'editing_' -'edit_' -'eding' -'eder' -'ection_' -'eci' -'echse' -'ebenen_' -'dämm' -'dynamische' -'dyna' -'dwindl' -'dust_' -'durchschnitt' -'dumping_' -'dumm' -'dum' -'dua_' -'drohende' -'droh' -'dritter_' -'dring' -'dres' -'dramatische_' -'dox' -'downs_' -'downloading_' -'downloade' -'dose_' -'dore' -'doppelt_' -'doppel' -'dong' -'dominiert_' -'dominated_' -'domains_' -'dom' -'dogs_' -'dog' -'documentation_' -'dn' -'dk' -'diving_' -'diversi' -'divergen' -'disziplin' -'distributions_' -'distract' -'distinct_' -'disso' -'disre' -'dispo' -'displa' -'dispe' -'dispa' -'diskriminierung' -'discriminat' -'discovering_' -'discount_' -'disclose_' -'disciplin' -'disappointing_' -'disagreement' -'disadvantage' -'dire_' -'diplomatische' -'dip_' -'dioxid' -'dio' -'diminish' -'dimensions_' -'dili' -'dilemma_' -'dikti' -'digt_' -'dif' -'diet_' -'dies' -'dienste' -'dick' -'dichte' -'diagnosti' -'dez' -'devo' -'devis' -'developer' -'devaluation' -'deutscher_' -'deutsch' -'deut' -'determining_' -'determin' -'deter_' -'deter' -'detection_' -'detai' -'destinations_' -'designated_' -'deserv' -'ders' -'derogations_' -'derartig' -'depression_' -'depreciation_' -'depreciat' -'depre' -'depos' -'dependency_' -'depan_' -'denomination' -'dende' -'demonstration_' -'demonstr' -'delte' -'deliberate' -'delegati' -'delays_' -'deko' -'dek' -'deinem_' -'deflation' -'definitive' -'definitions_' -'definierte' -'definieren_' -'defer' -'defensive_' -'defa' -'deeper_' -'deepen' -'decreas' -'decoupling_' -'deciding_' -'decides_' -'debattiert' -'deal' -'dauernd' -'date' -'dargelegt' -'dankbar_' -'dana_' -'damaligen_' -'damalige_' -'damaging_' -'dahin' -'cynic' -'customize' -'custom_' -'curr' -'cure' -'cula' -'ctable_' -'crown' -'crossing' -'cron' -'criticise' -'criterion_' -'criminali' -'cove' -'coupon' -'coun' -'couldn_' -'cott' -'corresponds_' -'correlation' -'corrections_' -'correction_' -'correct' -'corr' -'corpora' -'cornerstone' -'corn' -'cooperative_' -'cooked_' -'coo' -'convers' -'contribu' -'contradiction' -'contra' -'continuously_' -'continuous_' -'continents_' -'contest_' -'contemp' -'cont' -'consume' -'consultations_' -'constructi' -'constitution' -'conso' -'consent_' -'connectivity_' -'conjunction_' -'congress_' -'congratulations_' -'confort_' -'conform' -'conflict' -'configured_' -'confess' -'conductors_' -'conditione' -'concessions_' -'concert_' -'concert' -'composit' -'composer' -'complied_' -'complementary_' -'complement_' -'comple' -'complacen' -'compensat' -'compelling_' -'compe' -'comparable_' -'communa' -'commu' -'commend' -'comf' -'comb' -'colorful_' -'color' -'colony_' -'collecti' -'collect' -'collapsed_' -'coi' -'coe' -'cob' -'coastline' -'coach_' -'cluster_' -'clus' -'cloth' -'clos' -'climb_' -'clearer_' -'clean' -'clamp' -'cking_' -'cited_' -'cisi' -'circles_' -'circ' -'cipa' -'cion_' -'cil_' -'cic' -'chw' -'chuld' -'chu' -'chtung' -'chtigt' -'chter_' -'chtbar' -'chste' -'chst' -'chor' -'choi' -'chk' -'chips_' -'ching' -'chi_' -'chei' -'cheat' -'chat_' -'chat' -'chase' -'charit' -'charakteristisch' -'chapter' -'challenge' -'chairs_' -'cerca' -'centrali' -'center' -'cens' -'celebration_' -'cease' -'cci' -'categori' -'cated_' -'cate_' -'catalyst_' -'carri' -'card' -'capa' -'cancell' -'camps_' -'cals_' -'call' -'calculation' -'calculated_' -'caf' -'cabl' -'bürge' -'busi' -'bureaucratic_' -'bureaucra' -'burdens_' -'bundle' -'bukan' -'bud' -'buck' -'buchs' -'brücke' -'bräu' -'broaden' -'broad' -'bridge' -'brid' -'breites_' -'breakthrough_' -'breaking_' -'breach_' -'brat' -'brand' -'branche' -'brai' -'brac' -'boutique_' -'bout' -'borrow_' -'bone_' -'bond' -'bombs_' -'bold' -'boh' -'board' -'blä' -'blutigen_' -'blutig' -'blue' -'bloods' -'blo_' -'blier' -'bliebe' -'blicken' -'blick' -'blen_' -'bkommen_' -'bj' -'bits_' -'biss' -'bish' -'bip' -'biologischen_' -'biologis' -'biolog' -'bination' -'billigen_' -'bilie' -'bile' -'bilaterale' -'bila' -'bicycle' -'bezogene' -'beziehungsweise_' -'bezeichnete' -'bewä' -'bewussten_' -'bewirkt_' -'bewerten_' -'beweg' -'bew' -'bevorstehende' -'bev' -'beurteilen_' -'beträchtliche_' -'beträ' -'betrug_' -'betreffend_' -'betray' -'besuchten_' -'bestätig' -'bestraft_' -'bestmögliche' -'bestimmungen_' -'bester_' -'bestehe' -'besseres_' -'besonderem_' -'besitz_' -'besiege' -'besichtig' -'beseitig' -'bese' -'beschädigt_' -'beschränkung' -'beschränkte' -'beschl' -'berühr' -'berühmteste' -'berüc' -'beruh' -'berufliche_' -'berufliche' -'bern_' -'berkembang_' -'berk' -'berichterstatt' -'berichten_' -'bericht' -'bereitzustellen_' -'bereite' -'berda' -'berb' -'bequeme' -'beobacht' -'bent' -'bemerkenswerte_' -'beme' -'bem' -'belonging' -'believ' -'beliebige_' -'belgischen_' -'beleb' -'belastet_' -'belast' -'bekämpft' -'bekräftigt_' -'bekräftig' -'bekanntlich_' -'beiträgt_' -'beitreten_' -'beiten_' -'beit_' -'beispiellose' -'beispielhaft' -'bein' -'behält_' -'behindertenfreundliche_' -'behi' -'beherrsch' -'beher' -'behandlung' -'begrenzte_' -'begreifen_' -'begeistert' -'befolgt_' -'befand' -'beer_' -'beeindruckende' -'beein' -'bedien' -'bedenke' -'bedenk' -'beautifully_' -'beating' -'beat_' -'beamte' -'beam_' -'beam' -'beachtet_' -'beabsichtigt_' -'bd' -'baute' -'baue' -'basket' -'basierende' -'barometer' -'bares_' -'barem_' -'bao_' -'bankruptcy_' -'banken_' -'bands_' -'ballot_' -'balances_' -'baik_' -'backward_' -'backd' -'awful_' -'awaken' -'await_' -'aven' -'avant' -'autoritären_' -'autor' -'automated_' -'autob' -'authorize' -'authentic' -'ausüben_' -'auszugleichen_' -'auszud' -'auszubauen_' -'ausspr' -'ausse' -'ausschließ' -'ausmachen_' -'auslös' -'ausgezeichnet_' -'ausgez' -'ausgewählten_' -'ausgewogenen_' -'ausgestellt' -'ausgeprägte' -'ausgelegt_' -'ausgehende' -'ausgeh' -'ausgebildete' -'ausg' -'ausführlich_' -'ausf' -'ausd' -'ausbe' -'augment' -'aufzuz' -'aufwa' -'auftragten_' -'aufstrebende' -'aufle' -'aufhören_' -'aufgestellt_' -'aufen_' -'aufeinander' -'aue' -'audiovisual_' -'auchen_' -'atur' -'attraktiv_' -'attitudes_' -'attent' -'attacked_' -'ato' -'atively_' -'ations' -'ational_' -'ation' -'atie' -'atemberaubende' -'asur' -'astronom' -'astonish' -'asti' -'aster_' -'assung_' -'assiste' -'assist' -'asses' -'ass' -'asil' -'ash_' -'asch' -'asc' -'aru' -'artner' -'articulate' -'arte_' -'arrives_' -'arri' -'arkt_' -'ark_' -'aring_' -'ardo' -'architectural_' -'arbe' -'approximat' -'appreciation_' -'applicant_' -'appease' -'appear' -'appeal' -'appalling_' -'app_' -'aper' -'anzusehen_' -'anze' -'anza' -'antwort' -'antra' -'antly_' -'antin' -'antik' -'antibiotics_' -'antara_' -'answered_' -'anstr' -'anstelle_' -'ansta' -'anspruch' -'anspr' -'anor' -'announc' -'anniversary_' -'annex' -'ankurbel' -'ankung' -'animation' -'anima' -'ania' -'anhe' -'anhaltende_' -'anh' -'angu' -'angle_' -'angez' -'angewendet_' -'angewa' -'angeschlossen_' -'angeh' -'angeführt' -'angebotenen_' -'anfällig_' -'aneinander_' -'anderung' -'andern_' -'andards_' -'anbieter' -'anbet' -'analog' -'ams' -'amount' -'amerikanische' -'aly' -'aluminum_' -'altige' -'alti' -'alor' -'allocation_' -'allgemeiner' -'allge' -'aller' -'allen' -'alkohol' -'aliz' -'alische' -'algorithms_' -'ald_' -'alas' -'alarming' -'alarmier' -'aktuelle' -'akibat' -'akh' -'aket' -'aken_' -'ais_' -'ains' -'aining_' -'ahrt_' -'agu' -'ags' -'agree' -'agne' -'agn' -'aggressi' -'aggravat' -'agentur' -'afts' -'aften_' -'affiliate' -'affe' -'advoca' -'advisor' -'advisers_' -'advanc' -'adr' -'adore' -'ador' -'admi' -'adjusted_' -'additi' -'adding_' -'acu' -'activate_' -'acquire_' -'acion' -'acies_' -'achse' -'achievable_' -'achi' -'accumulate' -'accordingly_' -'accord' -'accomplish_' -'accompany_' -'accom' -'accidental' -'access' -'abzuwe' -'abz' -'abweichen' -'abus' -'abstract' -'absti' -'abschließend_' -'abra' -'abolish' -'ables_' -'ablehn' -'abit' -'abgeschnitten_' -'abger' -'abgeb' -'aber' -']]), ' -'].' -'Zö' -'Zweit' -'Zwangsv' -'Zuwanderer' -'Zusätzlich' -'Zurich_' -'Zun' -'Zuerst_' -'Zube' -'Zor' -'Zoom' -'Zoo_' -'Zionis' -'Zielsetzung' -'Ziels' -'Zentrali' -'Zel' -'Zehn' -'Yugoslav_' -'YouTube_' -'Yi' -'Yel' -'Yacht' -'Xinjiang_' -'Xin' -'Xe' -'Wörter_' -'Wört' -'Wählern_' -'Worse_' -'Working_' -'Worker' -'Wonder' -'Wollen_' -'Woll' -'Wochenende_' -'Wirtschaftsw' -'Wirtschaftse' -'Wirkung' -'Willkommen_' -'Williams_' -'Wild_' -'Wil' -'Wiederbelebung_' -'Whatever_' -'Wette_' -'Westjordanland_' -'Werke' -'Wen_' -'Weltmarkt' -'Weltkriegs_' -'Weltgesundheitsorganisation_' -'Wellen' -'Welle_' -'Weißbuch_' -'Weiterver' -'Wehr' -'Week' -'Wasserstoff' -'Wassers' -'Want_' -'Wales_' -'Wahrscheinlich' -'Waf' -'WWII_' -'WE_' -'Vulkan' -'Vr' -'Vorwand_' -'Vortr' -'Vorsicht_' -'Vormittag_' -'Vorm' -'Vorhan' -'Vorg' -'Vorb' -'Vollst' -'Voice_' -'Vita' -'Visa' -'Vila' -'Vig' -'Vietnam' -'Vie' -'Victor' -'Veto' -'Verwundbarkeit' -'Verwend' -'Verv' -'Verurteilung_' -'Vertriebs' -'Vertreibung' -'Vertiefung_' -'Versuchen_' -'Versu' -'Verstöße' -'Verstärkung_' -'Verständ' -'Verschärf' -'Verschuldung_' -'Verschmutzung_' -'Versch' -'Versagen_' -'Vermögenswerten_' -'Vermittlung_' -'Verlierern_' -'Verknüpfung' -'Verkehrsanbindung_' -'Veridian_' -'Verhältnisse_' -'Verhinderung_' -'Verhaftung_' -'Verfechter_' -'Verfassungsvertrag_' -'Verdienst' -'Verbrennung' -'Verbreche' -'Verbindungs' -'Verarbeitung' -'Verabschiedung_' -'Vend' -'Vat' -'Vas' -'Vall' -'Vali' -'Valent' -'VIP_' -'VII' -'VC_' -'Uruguay_' -'Ursprungs' -'Ursache_' -'Urb' -'Unzufriedenheit_' -'Unz' -'Unw' -'Unver' -'Untu' -'Unternehmenss' -'Untergang' -'Unterg' -'Unterb' -'Unless_' -'Ungleichgewichte_' -'Ungeachtet_' -'Unemployment_' -'Unein' -'Unabhängig' -'Umweltbe' -'Umweltausschuss_' -'Umwelta' -'Umsch' -'Ultra_' -'Ult' -'Ukrain' -'Uhr' -'Ufer_' -'Ud' -'UNI' -'UNHCR_' -'UND_' -'UA' -'Tür' -'Täuschung' -'Ty' -'Tube' -'Tschechische' -'Tsa' -'Trü' -'Trump' -'Truc' -'Trop' -'Trock' -'Trip' -'Trinkwasser' -'Trin' -'Tribu' -'Trial' -'Trennung_' -'Traum' -'Transparen' -'Tran' -'Tram' -'Trainings' -'Training_' -'Train_' -'Trail' -'Touristen' -'Toulouse_' -'Touch' -'Toten_' -'Tol' -'Titel' -'Titan' -'Tit' -'Tip' -'Tin' -'Tierschutz' -'Tiefe' -'Tie' -'Tibetan' -'Tia' -'Thyssen' -'Threa' -'Thr' -'Tho' -'Therma' -'Thema' -'Textes_' -'Termine_' -'Tennis_' -'Templates_' -'Telekommunikations' -'Techniken_' -'Tauche' -'Tastatur' -'Tank' -'Tang' -'Tamp' -'Tam' -'Tai' -'Tahrir_' -'Tabellen_' -'TTIP_' -'TS_' -'TRA' -'TG' -'TFT' -'Sünden_' -'Südwest' -'Sá' -'Synth' -'Symptom' -'Symp' -'Symbol' -'Swe' -'Sustainable_' -'Surf' -'Supp' -'Superior_' -'Super_' -'Sunni_' -'Subsidiarit' -'Stö' -'Sty' -'Student_' -'Stro' -'Stress' -'Strecke_' -'Strategi' -'Strat' -'Strassen_' -'Stran' -'Strafgericht' -'Strafe' -'Stol' -'Stoffen_' -'Stipendi' -'Still' -'Stick' -'Stich' -'Steve_' -'Steuersenkungen_' -'Stern_' -'Sterb' -'Steph' -'Step' -'Stellvertreter_' -'Stellungnahmen_' -'Stellenwert_' -'Steinberg_' -'Steel' -'Statisti' -'Stadtteil_' -'Stadium_' -'Stable_' -'Stabilisierung' -'Staatsanw' -'Staat' -'Später_' -'Spyware_' -'Spy' -'Spr' -'Sporta' -'Spion' -'Speak' -'Spannung' -'Spanische' -'Spaniens_' -'South' -'Sonntag_' -'Song' -'Sonders' -'Sonderbe' -'Somet' -'Solution_' -'Solidarity_' -'Sofi' -'Socialists_' -'Smart' -'Slovakia' -'Sli' -'Skript' -'Ski_' -'Skepti' -'Ske' -'Sixt' -'Sitzung' -'Sis' -'Singh' -'Simply_' -'Silva_' -'Silicon' -'Signal' -'Sig' -'Sicherheitsst' -'Sicherheitsrates_' -'Sicherheitskräfte_' -'Sich_' -'Show' -'Short_' -'Sheraton_' -'Sham' -'Shaf' -'Shadow' -'Sev' -'Setz' -'Sensibili' -'Sender_' -'Seminar_' -'Select' -'Segment' -'Securit' -'Screen' -'Scottish_' -'Schätz' -'Schäd' -'Schwung_' -'Schwimm' -'Schwelle' -'Schwed' -'Schwarzen_' -'Schutzmaßnahmen_' -'Schutze' -'Schur' -'Schuldner_' -'Schuldenerlass_' -'Schottland_' -'Schnittstelle_' -'Schn' -'Schmu' -'Schmit' -'Schmi' -'Schlussfolgerung_' -'Schließung_' -'Schim' -'Schiffen_' -'Schar' -'Schan' -'Schall' -'Scar' -'Scanne' -'Scal' -'Saudis_' -'Satelliten_' -'Sant_' -'Sample' -'Sammel' -'Sal' -'Saison' -'Saf' -'Sacr' -'Sachver' -'Sachs_' -'SY' -'SV' -'SSL' -'SSE' -'SR' -'SN' -'SK' -'SITE_' -'SIM' -'SDR' -'SARS_' -'SAP_' -'Rückz' -'Rückschlag_' -'Rückg' -'Röm' -'Räume' -'Russische' -'Rus' -'Rumsfeld_' -'Rov' -'Route_' -'Rotarian' -'Rota' -'Romulan' -'Romans_' -'Romano' -'Rol' -'Rog' -'Roc' -'Road' -'Rival' -'Risk' -'Rindfleisch' -'Rin' -'Rif' -'Ride' -'Richtungen_' -'Rh' -'Revolutionary_' -'Revol' -'Resultate_' -'Restaura' -'Resources_' -'Resol' -'Reso' -'Repu' -'Representative' -'Rentner' -'Renov' -'Reli' -'Release' -'Relati' -'Rein' -'Reichweite_' -'Reich_' -'Regulierungen_' -'Regulat' -'Regionalpolitik_' -'Regierungsvertreter' -'Regierungsführung_' -'Regierungse' -'Regent' -'Regens' -'Regel' -'Rega' -'Referenz' -'Redner' -'Recon' -'Rechtsp' -'Rechtsakt' -'Rechtfertigung_' -'Recht' -'Rechenschaft_' -'Rebellen' -'Realis' -'Reakt' -'Rav' -'Raus' -'Ratspräsident' -'Rassismus_' -'Rasse' -'Range' -'Ramada' -'Rak' -'Rai' -'Rag' -'Radi' -'Rabatt' -'RU' -'ROM' -'RL' -'Quit' -'Quartet' -'Quartal' -'Quality_' -'Qualifi' -'Quadra' -'QU' -'Py' -'Purvis_' -'Pur' -'Puffer' -'Präventi' -'Prämi' -'Prozessor_' -'Provokation' -'Provi' -'Protest_' -'Protektionismus_' -'Prophet' -'Prope' -'Prog' -'Profite_' -'Production' -'Produ' -'Probe' -'Privatsphäre_' -'Privathaus' -'Price' -'Preventi' -'Prestige_' -'Prese' -'Preiss' -'Portug' -'Porti' -'Popular' -'Poor_' -'Poly' -'Politis' -'Polic' -'Points_' -'Plug_' -'Ple' -'Platte' -'Platform_' -'Plasma' -'Pix' -'Pitt' -'Pira' -'Picture' -'PiS_' -'Photo_' -'Pflege' -'Pferde' -'Pfei' -'Petitions_' -'Petitions' -'Petition' -'Pesti' -'Peru' -'Persönlichkeit' -'Peripherie_' -'Period' -'Pere' -'Pentax_' -'Pensionen_' -'Penis' -'Peninsula_' -'Pear' -'Pav' -'Pauls' -'Patente_' -'Passi' -'Passagiere_' -'Partition_' -'Partie' -'Parlamentsabgeordnete' -'Pare' -'Panzer' -'Panasonic_' -'Palästin' -'Palais_' -'Paket' -'Pai' -'Pack' -'PV_' -'PU' -'PT' -'POS' -'PM_' -'PE_' -'Oz' -'Outdoor_' -'Organismen_' -'Optimierung' -'Omniture_' -'Om' -'Okt' -'Ohren_' -'Oft' -'Offs' -'Official_' -'Occ' -'Obst' -'Obr' -'Objekte_' -'Objekt_' -'Objekt' -'Obers' -'OR' -'ONE_' -'OE' -'Nächte' -'Ny' -'Nuclear_' -'Novo' -'Nov' -'Nots' -'Noti' -'Noten' -'Norwege' -'Norway_' -'Norw' -'Normalerweise_' -'Norde' -'Nordamerika' -'Nikotin' -'Nieders' -'Nicholas_' -'Ng' -'Neuigkeiten_' -'Neug' -'Neue' -'Neube' -'Neub' -'Neuan' -'Netze' -'Network' -'Nephi_' -'Need_' -'Need' -'Nazis_' -'Navy_' -'Navig' -'Natursch' -'Native_' -'Nationalstaaten_' -'Nationalist' -'Nationale_' -'Namun_' -'Nai' -'Nahverkehr' -'Nachteil_' -'Nachbarschafts' -'Münzen_' -'Möchte' -'Mé' -'Mär' -'Mängel_' -'Muss' -'Musk' -'Musiker_' -'Musi' -'Muni' -'Mun' -'Mum' -'Movi' -'Movement_' -'Mosc' -'Montp' -'Montenegro_' -'Montage' -'Monaco_' -'Modells_' -'Mobili' -'Mits' -'Mitgliedstaat' -'Mitgefühl_' -'Mischung_' -'Ministr' -'Minimum_' -'Minderheiten' -'Million' -'Millennium' -'Militär_' -'Miles_' -'Mig' -'Messung_' -'Merkmale' -'Mercur' -'Menschenrechtsverletzungen_' -'Meinungsverschiedenheiten_' -'Mehrwert_' -'Mehr' -'Megapixel' -'Meg' -'Meer' -'Medit' -'Medikamenten_' -'Medikamente_' -'Medicine' -'Maur' -'Matth' -'Matte' -'Mathematik_' -'Mathe' -'Massenvernichtungswaffen_' -'Massen_' -'Massagen_' -'Massage' -'Massachusetts_' -'Maschinen' -'Mary' -'Marshall_' -'Marra' -'Maritime_' -'Marie' -'Margaret_' -'Marco' -'Marbella_' -'Mao' -'Mant' -'Mano' -'Mannschaft_' -'Mann' -'Mandat' -'Malay' -'Mahm' -'Magne' -'Magn' -'Maf' -'Madr' -'Made' -'Maci' -'Machthaber_' -'Machi' -'Macedonia_' -'Maca' -'MID' -'MDGs_' -'MAR' -'MADRID_' -'Luftfahrt' -'Luca' -'Lub' -'Lore' -'Looking_' -'Loo' -'Lond' -'Lodge_' -'Lob' -'Ll' -'Lith' -'Lita' -'List' -'Liquidität_' -'Lip' -'Lion' -'Linu' -'Limited_' -'Lieferungen_' -'Lieferant' -'Lieblings' -'Liebe' -'Lichte_' -'License' -'Letztere_' -'Letter' -'Lek' -'Leistungsfähigkeit_' -'Leipzig_' -'Legitimation_' -'Legislative_' -'Lef' -'Led' -'Lebensbedingungen_' -'Learning_' -'Laur' -'Laufwe' -'Laser_' -'Laos_' -'Lanka_' -'Langstrecken' -'Landschaft' -'Lande_' -'Lance' -'Lamanites_' -'Lack' -'LS' -'LP' -'LOS_' -'LICH' -'LES' -'LDP_' -'Kürzungen_' -'Künstler' -'Könnte_' -'Kurzu' -'Kurve' -'Kun' -'Kumari_' -'Krugman_' -'Krone' -'Krist' -'Krim' -'Kriegsverbreche' -'Kreditk' -'Kreativität_' -'Krat' -'Krank' -'Kraftfahr' -'Korrekt' -'Koordination_' -'Kooperationsabkommen' -'Konzern' -'Konzepte_' -'Konve' -'Kontroverse' -'Kontrast_' -'Konstruktion' -'Konkur' -'Kondition' -'Kompromisse_' -'Kompe' -'Kommunistische' -'Kommissare_' -'Kommentare_' -'Kode' -'Kne' -'Klimasch' -'Klassen_' -'Klassen' -'Klagen' -'Kissinger' -'Kind' -'Kha' -'Kenne' -'Keep_' -'Kaz' -'Katzen' -'Kategorien_' -'Kaste' -'Kasach' -'Karten' -'Karr' -'Karl_' -'Kare' -'Kapitalst' -'Kapell' -'Kanzle' -'Kandidatenländer' -'Kamera' -'Kambodscha_' -'Kalt' -'Kalifornien' -'Kalif' -'Kad' -'Kabine' -'KB_' -'Jörg_' -'Jup' -'Juncker_' -'Julian_' -'Jugendherberge_' -'Jos' -'Johannes_' -'Jiang_' -'Jel' -'Jazz' -'Jarzembowski_' -'Jame' -'JPEG_' -'JP' -'Iss' -'Isolation_' -'Islamischen_' -'Iron' -'Iri' -'Ira' -'Ion' -'Inva' -'Intr' -'Interview' -'Interv' -'Intern' -'Interi' -'Interess' -'Inten' -'Integrat' -'Inspektor' -'Insel' -'Inn' -'Inkrafttreten_' -'Ink' -'Ini' -'Informations_' -'Industrien' -'Indikatoren_' -'Indikat' -'Indian' -'Inde' -'Inc' -'Implementi' -'Imperi' -'Ih' -'If' -'Identitäten_' -'Iber' -'IX' -'ISS' -'INS' -'IND' -'INC' -'Hürde' -'Höh' -'Häusern_' -'Häuser_' -'Hut_' -'Hut' -'Hus' -'Humanit' -'Hug' -'Hu_' -'Hoste' -'Hospi' -'Honor' -'Home' -'Hoheit' -'Hohe' -'Hochzeit' -'Hob' -'Hirsch' -'Hip' -'Hinterl' -'Hinsichtlich_' -'Himm' -'Highlight' -'Hierbei_' -'Herzego' -'Herunter' -'Herstell' -'Herkunftsl' -'Herberge_' -'Hell' -'Held' -'Hektar_' -'Heimat_' -'Heid' -'Hegemonie_' -'Heer' -'Heating_' -'Head_' -'Head' -'Haza' -'Haw' -'Havel_' -'Haushaltsk' -'Haushalten_' -'Hauptziel' -'Hauptg' -'Harry_' -'Hardliner' -'Handy_' -'Hands' -'Handelsd' -'Halbjahr_' -'Haft_' -'Had_' -'Habr' -'Gut_' -'Gus' -'Gul' -'Guid' -'Guer' -'Guantánamo_' -'Gründungs' -'Grö' -'Grä' -'Grundwerte_' -'Grundv' -'Grundge' -'Gru' -'Green' -'Grap' -'Grant_' -'Grand' -'Granada_' -'Gran_' -'Gou' -'Got' -'Goe' -'Glückw' -'Globalis' -'Globale_' -'Glaube_' -'Glacier_' -'Gl' -'Giu' -'Gitarren' -'Gir' -'Ghana_' -'Gewässer' -'Gewi' -'Gewebe' -'Getreide' -'Gesundheitswesen_' -'Gesundheitsschutz' -'Gestern_' -'Gestalt_' -'Gesta' -'Gesichtspunkt' -'Gesetzen' -'Geschäftsreise' -'Geschäftsführ' -'Geschmack' -'Geschichts' -'Geschenk_' -'Gescheh' -'Geräten_' -'Georgian_' -'Gemeinschaftsrecht' -'Gemeinsame_' -'Gemeinsam_' -'Gegebenheiten_' -'Gefangene' -'Gefa' -'Gedanke' -'Ged' -'Gebühren_' -'Gebietskörperschaften_' -'Gaul' -'Gau' -'Gastl' -'Garc' -'Ganzes_' -'Gandhi_' -'Galic' -'Galax' -'Gai' -'GT' -'GMO_' -'GM' -'GIMP_' -'GF' -'GBP_' -'Fürsten' -'Fürs' -'Führungsp' -'Führungen_' -'Führer' -'Fäl' -'Fäh' -'Fuss_' -'Funktionieren_' -'Funktionalität_' -'Full_' -'Fue' -'Früher' -'Frucht' -'Fronte' -'Frist_' -'Friedh' -'Freilassung_' -'FreeBSD_' -'Frattini_' -'Frassoni_' -'Franz_' -'Franz' -'Fox_' -'Fourth_' -'Fotograf' -'Fortschritts' -'Forst' -'Formulierung_' -'Forme' -'Forex_' -'Folter_' -'Folgende' -'Flüge_' -'Flü' -'Flugver' -'Flucht' -'Flach' -'Fl' -'Fitnessraum_' -'Fitnesscenter_' -'Fitness' -'Fiskalp' -'Fischf' -'Finn' -'Finanzt' -'Finanzministeri' -'Finanzma' -'Finanzinstitut' -'Finanziellen_' -'Finanzi' -'Finanzhilfe' -'Finanzdienstleistungen_' -'Filip' -'Figu' -'Field' -'Fie' -'Feu' -'Ferrer' -'Ferr' -'Fernsehs' -'Fen' -'Feli' -'Feinds' -'Fehlern_' -'Fehle' -'Faz' -'Fax' -'Fanati' -'Fan' -'Fail' -'Fahnen' -'Fah' -'FOR' -'FI_' -'FC' -'FBI_' -'Extremismus_' -'Exporte' -'Expansion' -'Exce' -'Exa' -'Everyone_' -'Eurosta' -'Europäer' -'Europe' -'Europarat' -'Eure' -'Eto' -'Ethiopia_' -'Es' -'Erzeug' -'Erwägung_' -'Erwe' -'Ersparnisse_' -'Ernährungs' -'Erlös' -'Erlebnis_' -'Erla' -'Erkenntnisse_' -'Erkenn' -'Erinnerungen_' -'Erika_' -'Eric' -'Erhalt_' -'Erfordernissen_' -'Erfa' -'Erf' -'Ereignissen_' -'Ereignis_' -'Erb' -'Era' -'Equally_' -'Equal' -'Epidemi' -'Entwicklungszusammenarbeit_' -'Entwicklungshilfe_' -'Entspannen_' -'Entschl' -'Entscheidungsfindung_' -'Entscheide' -'Entlassung' -'Entfernung_' -'Entdeckung_' -'Entdecke' -'Engine' -'Engel' -'Energieversorgung_' -'Energieverbrauch' -'Energietr' -'Empfehlung_' -'Emm' -'Emb' -'Eman' -'Elys' -'Eliten_' -'Eli' -'Element' -'Elektrizität' -'Elect' -'Elb' -'Einwohnern_' -'Eintrag_' -'Einstei' -'Einnahme' -'Einla' -'Einig' -'Einheitswährung_' -'Eingriff' -'Eingang_' -'Eindr' -'Eight' -'Eigentums' -'Ehren' -'Effe' -'Edward' -'Economists_' -'Ecol' -'Echt' -'Eben' -'East' -'Earl' -'EV' -'EQ' -'EOS_' -'ENE' -'EME' -'EFSF_' -'EF' -'EEC_' -'EE' -'EA_' -'Durchschnitt_' -'Durchs' -'Duke' -'Duc' -'Dua' -'Dry_' -'Drogenh' -'Drac' -'Dosi' -'Domain' -'Dollars_' -'Dokumenten_' -'Dokt' -'Dod' -'Doctor_' -'Divi' -'Disziplin' -'Distri' -'Disney_' -'Discover' -'Direktor_' -'Direkt_' -'Diamant' -'Devisenwechsel_' -'Device' -'Develope' -'Dess' -'Demonstrationen_' -'Demonstration_' -'Dei' -'Defizit_' -'Definitionen_' -'Deep' -'Deco' -'Deborah_' -'Deal_' -'Davon_' -'Dav' -'Datenbank' -'Darstell' -'Dark_' -'Danube_' -'Dalai_' -'Daily' -'Dafür_' -'Dach_' -'DU' -'DSLR_' -'DF' -'DEL' -'Cyp' -'Cus' -'Cub' -'Crow' -'Cristi' -'Cris' -'Cr' -'Coven' -'Course' -'Coup' -'Cort' -'Corp' -'Cookies_' -'Cookie_' -'Coo' -'Control' -'Conte' -'Contain' -'Const' -'Conservative_' -'Configur' -'Cond' -'Compo' -'Compli' -'Communities_' -'Communis' -'Commonwealth_' -'Commons_' -'Commerc' -'Collection_' -'Co_' -'Cluster' -'Cli' -'Claude_' -'Citi' -'Christi' -'Chief_' -'Chic' -'Chian' -'Chest' -'Chelsea_' -'Charme_' -'Charlotte' -'Champions' -'Catherine_' -'Catalunya_' -'Cart' -'Carol_' -'Cap_' -'Cana' -'Campi' -'Camp_' -'Came' -'Cale' -'Cadiz_' -'CV' -'CK' -'CF_' -'CER' -'C6_' -'Bürgerrecht' -'Bügelservice_' -'Byrne' -'Button_' -'Burning_' -'Bureau_' -'Bul' -'Built_' -'Buffe' -'Buen' -'Budget' -'Buddhist_' -'Bucharest_' -'Brüder' -'Brücken' -'Brücke_' -'Brook' -'Broad' -'Bristol_' -'Brief' -'Brasiliens_' -'Brand_' -'Bow' -'Boutique' -'Borr' -'Born' -'Boote' -'Boom_' -'Bolivia_' -'Boliv' -'Bod' -'Blase' -'Blanch' -'Blan' -'Bit' -'Binnenmarkts_' -'Bind' -'Bildschirm' -'Bic' -'Bibl' -'Bewusstsein_' -'Beton' -'Besuchen_' -'Bestec' -'Bestands' -'Besichtigung' -'Besi' -'Beschl' -'Besatzungs' -'Berna' -'Berichterstatters_' -'Berge' -'Bergbau' -'Berei' -'Beratungs' -'Benalmadena_' -'Beleg' -'Belange_' -'Bek' -'Beitrittsländer_' -'Behinderung' -'Begr' -'Befreiung' -'Befehl_' -'Beck' -'Beantwortung_' -'Bauern' -'Batterie' -'Bat' -'Bass' -'Barre' -'Barr' -'Baron' -'Barba' -'Bankr' -'Bala' -'Bak' -'Bagdad_' -'Baby_' -'Baby' -'BP' -'BEA' -'BBC_' -'Außens' -'Außenhandel' -'Autovermietung_' -'Autos_' -'Autorität_' -'Autok' -'Auth' -'Ausstellung' -'Ausser' -'Ausschüsse_' -'Auss' -'Ausrichtung_' -'Ausländer' -'Auslegung' -'Ausge' -'Ausgang_' -'Ausg' -'Ausfuhr' -'Auseinandersetzungen_' -'Auseinandersetzung_' -'Ausdruck' -'Auschecken_' -'Ausbreitung_' -'Ausbau' -'Auktion' -'Aufwand_' -'Aufw' -'Aufschub_' -'Aufrechterhaltung_' -'Auflagen_' -'Aufla' -'Auditor' -'Attrakti' -'Athens_' -'Assozi' -'Asc' -'Arts_' -'Artikels_' -'Arra' -'Aro' -'Arn' -'Argentiniens_' -'Arg' -'Archive_' -'Arbeitszeit' -'Arbeitsplatz' -'Arbeitsmärkte' -'Arbeiter' -'Aqu' -'Application' -'Applause_' -'Appartement_' -'Apollo_' -'Aparthotel' -'Apartamentos_' -'Apache_' -'Anwende' -'Anweisungen_' -'Antw' -'Antrieb' -'Ansta' -'Anschuldigungen_' -'Anschluss' -'Anrei' -'Anleitung_' -'Ankündigung_' -'Ankara_' -'Animat' -'Anhä' -'Angreifer' -'Angeli' -'Anfä' -'Anfa' -'Andy_' -'Analyst' -'Amat' -'Alvaro_' -'Alternative' -'Alte' -'Almost_' -'Allgemeine' -'Alkohol_' -'Ali_' -'Alexanderplatz_' -'Alba' -'Aktiv' -'Akt_' -'Ahmed' -'Ahmadinedschad_' -'Agriculture_' -'Agent' -'Against_' -'Afrikanischen_' -'Afghan_' -'Advi' -'Advent' -'Adress' -'Adam_' -'Ach' -'Aca' -'Abwärts' -'Abstände' -'Absp' -'Abschreckung_' -'Abschnitt' -'Abschluss' -'Abre' -'Abraham_' -'Above_' -'Abn' -'Abhol' -'AX' -'ATION' -'ANY_' -'ALL_' -'AK' -'AE' -'ABAP_' -'AA_' -'A350_' -'A1_' -'; • _' -';' -'93' -'92' -'91' -'84_' -'84' -'825' -'81_' -'78' -'73_' -'71_' -'70' -'57' -'450' -'42' -'320' -'3000_' -'2nd_' -'226' -'204' -'2018_' -'2017_' -'2016_' -'1981_' -'1974_' -'1969_' -'1958_' -'1957_' -'1955_' -'1951_' -'1930er_' -'1918_' -'179' -'158' -'140_' -'13th_' -'125' -'110_' -'10th_' -'104' -'103_' -'100' -'0ern_' -'011' -'010' -'// _' -'.  ' -'.: _' -'...) _' -'.)._' -'. – ' -'.   _' -'. .' -'. ) _' -'. " _' -', ..._' -')|_' -'): «' -'() , _' -'%), _' -'": _' -'")._' -'!”' -'!!!!' -' „ _' -' –&' -' –' -' « _' -'  ' -' ..' -' ($_' -' ''' -'™-_' -'€_' -'…_' -'”) _' -'“) _' -'‘' -'ا' -'י' -'ң' -'қ_' -'ін' -'ында' -'ші' -'ть_' -'сын' -'со' -'р_' -'пр' -'пар' -'ных_' -'на_' -'кономи' -'ка_' -'ит' -'ел' -'гі' -'го_' -'га' -'бе' -'ас' -'Т' -'К' -'Г' -'ρ' -'ο' -'Ž' -'ż' -'ška_' -'ý_' -'üße' -'üß' -'ütung' -'ütlich' -'ütige' -'üstet_' -'ürze' -'ürt' -'ürge' -'ürfe' -'ürdige' -'üpf' -'üng' -'ündete' -'ünde_' -'üllt_' -'ührten_' -'ührende' -'ühmt' -'üf' -'ücken_' -'üblich_' -'überzogen' -'übersetz' -'überschü' -'überschreitende' -'überra' -'übernahme' -'überleb' -'überholt' -'übergehen' -'überflü' -'übereinstimmen_' -'übereinkommen_' -'ößt_' -'ött' -'östlichen_' -'öster' -'öst_' -'öse' -'örtlichen_' -'örtliche' -'örte' -'örig' -'ör_' -'ökologisch' -'öhnlich_' -'öhe' -'öglichkeiten_' -'öge' -'öffnete' -'öffnet_' -'öffentliches_' -'öd' -'öcke' -'ôte_' -'ò' -'ï' -'î' -'être_' -'ête' -'ém' -'èr' -'ège_' -'äßig' -'äußerte_' -'äußert_' -'äuser' -'äume_' -'äum' -'ässig' -'ärz' -'ärt' -'ärmeren_' -'ärer_' -'ändler' -'ändiger_' -'änderte' -'ämte' -'ällt_' -'äle_' -'ähle' -'ähig' -'ägyptischen_' -'ägt_' -'äger' -'äg' -'ächtige' -'äche' -'ás' -'ßlich' -'ßer_' -'ßer' -'ßb' -'Überwindung_' -'Übersetzer' -'Überna' -'Überlebens' -'Übergriffe' -'Überflu' -'Übereinkommens_' -'Überarbeitung_' -'Österreich' -'Öls' -'Ölpreis' -'Öle' -'Ökolo' -'Äußer' -'Äquivalen' -'Ähnliche' -'Ã_' -'·      ' -'²_' -'®_' -'®, _' -' – ' -' %, _' -'}}) ==' -'}})' -'}{_' -'}, _' -'|' -'zünd' -'zzle' -'zykli' -'zwischenstaatliche' -'zweitgrößte_' -'zweimal_' -'zweige' -'zweier_' -'zweie' -'zwec' -'zwang' -'zuwider' -'zuvorkommende' -'zuverlässiger_' -'zuv' -'zutreffend' -'zusammenzuf' -'zusammenzuarbeiten_' -'zusammentre' -'zusammenhä' -'zusammenf' -'zurückzuh' -'zurückkehren_' -'zurückk' -'zurückbl' -'zunahm' -'zulässig_' -'zula' -'zukomm' -'zugr' -'zugewiesen' -'zugeschnitten' -'zugesa' -'zugelassen_' -'zugehen_' -'zufügen_' -'zufällig' -'zubereitet_' -'zte_' -'zst' -'zoo_' -'zonen_' -'zol' -'zitä' -'zitiere_' -'zit' -'zipation_' -'zines' -'zien' -'zheimer_' -'zerr' -'zero' -'zerbrech' -'zentri' -'zentral' -'zend' -'zeitweilige' -'zeitl' -'zeitiger_' -'zeitig' -'zeita' -'zeilen_' -'zeil' -'zeigten_' -'zeige' -'zeichnung_' -'zar_' -'yte' -'yment' -'yla' -'yie' -'yg' -'yev' -'yell' -'yea' -'ydr' -'yak' -'xts_' -'xon_' -'xn' -'xist' -'xim' -'xes_' -'würdige_' -'wört' -'wöchentlich' -'wärt' -'wäl' -'währung' -'wski_' -'wron' -'writer_' -'wrap' -'wozu_' -'wovon_' -'wounded_' -'worthwhile_' -'worries_' -'workshop' -'workplace' -'womit_' -'wofür_' -'wling' -'withstand_' -'withdrawn_' -'withdraw' -'wissenschaftliche' -'wissenschaftl' -'wirkte' -'wins_' -'winners_' -'winkel' -'willkürlich' -'willen_' -'wilde' -'wiese' -'wiederher' -'wiederge' -'widmet' -'widersetz' -'wick' -'whatsoever_' -'wettbewerbsfähig_' -'wettbewerb_' -'wertig' -'werkzeug' -'wende_' -'wend' -'wen' -'welding_' -'welders_' -'weites' -'weiterf' -'weiterentwickel' -'weilen_' -'weiche_' -'wege_' -'weekend' -'weck' -'wechsels' -'wechs' -'weakening_' -'wc' -'way' -'wav' -'watt' -'watershed_' -'waterfalls_' -'wasted_' -'warrior' -'ward' -'wanting_' -'wang' -'wandte' -'wahrha' -'wach' -'völligen_' -'völ' -'vá' -'vulnerabilities_' -'vul' -'votre_' -'vorzugehen_' -'vorzei' -'vorz' -'vorsorge_' -'vorrangige' -'vorne_' -'vorle' -'vorlag' -'vorhin_' -'vorhersehbar' -'vorherrschende' -'vorherigen_' -'vorherge' -'vorhandene_' -'vorhaben' -'vorgetragen_' -'vorgesehene_' -'vorgeleg' -'vorgebracht_' -'vorb' -'voraus' -'vorantreib' -'vorangehen' -'vorangegangenen_' -'voranbringen_' -'voor_' -'von' -'volunt' -'volumes_' -'vollziehen_' -'vollendet' -'voli' -'vole_' -'voic' -'vivid' -'vität_' -'vita' -'visit' -'viru' -'viou' -'violate_' -'violate' -'viol' -'vio' -'villa' -'vigorously_' -'vigilant' -'viewing_' -'viewer_' -'viet' -'vierz' -'vielversprechend' -'viels' -'vielfach' -'vidue' -'vide' -'vib' -'viat' -'vete' -'vet_' -'verzögern_' -'verzerr' -'verzeichnis_' -'verzauber' -'verwöhn' -'verwirr' -'verwir' -'verwendete_' -'verweigert' -'verweh' -'verwand' -'verwaltung' -'verwaltet_' -'verurteilen_' -'verursachen_' -'vertritt_' -'vertretene' -'vertretende' -'vertiefen_' -'vertie' -'verteil' -'verteidigt_' -'versäume' -'versuchten_' -'versu' -'verstümmel' -'verstehe_' -'versteckt' -'verstecken_' -'versta' -'versp' -'versorgen_' -'versorg' -'versicherung_' -'verschwenderische' -'verschre' -'verschr' -'versatil' -'versammel' -'versagt_' -'vers_' -'verpflichtung' -'verordn' -'vernünftige' -'vernetz' -'vernehm' -'vernachlässigt' -'vernachlässigen_' -'vermindert' -'verlässliche' -'verließ' -'verlie' -'verletzen' -'verleih_' -'verlangsam' -'verlang' -'verlagerung' -'verkünde' -'verkl' -'verkehr' -'verkaufte' -'verhältnisse' -'verhält' -'verheirat' -'verhandl' -'verhandelt' -'verhaftet_' -'vergift' -'verfügbare' -'verfolgte_' -'verfe' -'verfa' -'vereinte' -'vereinig' -'vereinfachen_' -'vereinbarten_' -'vereinbarte' -'verein' -'verdreifach' -'verdoppelt_' -'verdoppel' -'verdan' -'verbu' -'verbrenn' -'verbrachte' -'verborgen' -'verbleiben' -'verban' -'veranstaltet' -'veranlasst_' -'verankert_' -'verabscheu' -'venture' -'vent_' -'vendor' -'veil' -'vec' -'vd' -'vastly_' -'variier' -'variat' -'var_' -'valve' -'vale' -'vald' -'ußen' -'ux' -'uv' -'utsbe' -'utr' -'utin' -'uth' -'utenant' -'ustausch_' -'usgaben_' -'urz' -'uru' -'urt' -'ursa' -'urging_' -'urg' -'urch' -'urbaniz' -'urban' -'urate' -'ura_' -'upte' -'uphold' -'upheaval' -'upa' -'uon' -'unzähligen_' -'unzwe' -'unzureichende' -'unz' -'unwillingness_' -'unweigerlich' -'unwanted_' -'unverä' -'unu' -'unthink' -'unterzeichnete' -'unterzeichn' -'unterworfen_' -'unterteilt_' -'untersuch' -'unterstrichen_' -'unterstreiche' -'unterschätzt' -'unterschiedlich' -'unterschied' -'untersch' -'unterminieren_' -'untergräbt_' -'unteren_' -'unterbrechen_' -'unsc' -'unresolved_' -'unrea' -'unqu' -'unpredictab' -'unmittelbarer_' -'unmen' -'unlängst_' -'unkomp' -'unko' -'unite_' -'unis' -'unilateralism_' -'unica' -'uni_' -'unha' -'ungü' -'ungsze' -'ungsverfahren' -'ungso' -'ungser' -'unglück' -'unglaublich' -'ungerecht' -'ungenügend' -'ungehe' -'unfr' -'unfortunate_' -'unforgettable_' -'unfolding_' -'unfair' -'unexp' -'uneven' -'unes_' -'unerwünscht' -'unerwartete' -'undurch' -'undung' -'undi_' -'underestimate' -'underc' -'undeniabl' -'unden' -'unconditional' -'uncomfortable_' -'uncle' -'unbegrenzt' -'unaufhaltsam' -'unat' -'unam' -'umstr' -'umst' -'umsetz' -'umm' -'umin' -'umh' -'umgest' -'umgeh' -'umgebung' -'umfassendere_' -'umfangreichen_' -'umfangreich' -'umfa' -'umf' -'ume_' -'umbrella' -'uma_' -'ultra_' -'uls_' -'ulos' -'ulierung' -'uldn_' -'ulde' -'ularly_' -'ukr' -'uko' -'uit_' -'ugu' -'ugi' -'ugge' -'ufs' -'ufl' -'uff' -'uerung_' -'uern_' -'uellen_' -'uelle_' -'udo' -'udia' -'ubt_' -'uba' -'ub_' -'uay' -'uate' -'uas' -'tüt' -'tümer' -'tödlich' -'täts' -'tätigkeit' -'täter' -'tär' -'tzl' -'tzende' -'tze_' -'tyrant' -'typis' -'twin' -'tutt' -'tutor' -'tus_' -'turm' -'tur_' -'tuous' -'tunnel' -'tuna_' -'tuition' -'tuber' -'tti_' -'ttert' -'tters' -'tsp' -'tsi' -'tscheni' -'tsa' -'trö' -'trä' -'trusted_' -'truppen_' -'trupp' -'trug_' -'truc' -'trouble' -'trou' -'trot' -'tropis' -'trophi' -'trivial' -'trink' -'trim' -'trigger' -'trieben_' -'trieb' -'tribute_' -'treu_' -'tres_' -'tres' -'tree_' -'treasure_' -'treas' -'traße_' -'travelers_' -'travail_' -'traue' -'trau' -'trat_' -'trat' -'trapp' -'transparente_' -'transp' -'transmitt' -'translator' -'transformer_' -'transformati' -'transatlantic_' -'tran' -'tram_' -'trali' -'trail_' -'trai' -'tragbar' -'traditioneller_' -'trades_' -'trac' -'tournament_' -'tourismus' -'touri' -'totalit' -'tos' -'tops_' -'topbonus_' -'too' -'toni' -'tone' -'toma' -'tolerate' -'token_' -'toilet_' -'toilet' -'tment' -'tma' -'tkan_' -'titles_' -'titi' -'tist' -'tisch' -'tis_' -'tings_' -'timme' -'timing_' -'timi' -'tilt' -'till' -'tiker' -'tightening_' -'tig_' -'tiere' -'tier_' -'tiefen_' -'tief' -'tied_' -'tid' -'tick' -'tici' -'tiate' -'tian_' -'thwart' -'thun' -'throne_' -'thresholds_' -'threatens_' -'thoroughly_' -'thon_' -'theori' -'theoretical_' -'tgut' -'tformen_' -'textiles_' -'textile_' -'text' -'teurer' -'test' -'terte_' -'tert' -'terroristische_' -'terroris' -'territoriale_' -'territoriale' -'terminology_' -'terminat' -'term' -'terli' -'teren' -'terd' -'terblichkeit' -'terba' -'tent_' -'tendi' -'tempt' -'template' -'temperature' -'temperatur' -'tels' -'teles' -'telephones_' -'tekn' -'teilung_' -'teilnehmer' -'technologie_' -'technologie' -'technically_' -'tching_' -'tbe' -'tbar' -'tav' -'tausende' -'taught_' -'tatte' -'tator' -'tation' -'tate_' -'tasa' -'tariffs_' -'targeting_' -'tape_' -'tandard' -'tand' -'tamb' -'tally_' -'tali' -'takti' -'takeover_' -'tains_' -'tail_' -'tahu' -'taha' -'tactic' -'taat' -'taa' -'südlich_' -'sü' -'säum' -'säu' -'sätz' -'säkulare' -'säch' -'szei' -'systemi' -'systematische' -'syrischen_' -'syrische_' -'synd' -'symbolische' -'swing_' -'swin' -'sweet_' -'sweeping_' -'sust' -'suspended_' -'suspend' -'survivors_' -'surviving_' -'surv' -'surgery_' -'surfing_' -'supr' -'suppress_' -'support' -'superpower_' -'superiority_' -'sunny_' -'sung_' -'sums_' -'sug' -'suffice' -'suchte' -'succeed' -'subtr' -'subti' -'substitute' -'subsistence_' -'subsidiaries_' -'subscription' -'subo' -'subjecti' -'stücke' -'ständnis' -'ständi' -'stände_' -'städtischen_' -'styl' -'sty_' -'stumbl' -'stuhl_' -'studying_' -'studios_' -'studiere' -'stub' -'ströme_' -'strukturierte' -'stroke' -'stritt' -'strip' -'stringen' -'striking_' -'strenge_' -'streiche' -'streaming_' -'straße_' -'strain_' -'strain' -'strafrechtliche_' -'storm_' -'stopped_' -'stones_' -'stole' -'stm' -'stliche' -'stische_' -'stipulate' -'stimm' -'stills' -'stig' -'stian' -'stere' -'stems_' -'stelle' -'stee' -'stattfand_' -'stattdessen_' -'statis' -'stating_' -'starte' -'starship' -'starring_' -'starr' -'standar' -'stam' -'stall_' -'staged_' -'staff' -'stabilisi' -'ssystem_' -'ssung_' -'ssten_' -'sslich' -'ssige_' -'ssert' -'ssenen_' -'ssad' -'srat' -'square' -'spürbar' -'spü' -'spy' -'spu' -'sprung' -'sprozess' -'sprogramm' -'spro' -'spritz' -'sprech' -'spreads' -'sprachig' -'spots_' -'spotlight_' -'spora_' -'spoo' -'sponsor_' -'spolitik_' -'spokes' -'splendid_' -'spitz' -'spiritual' -'spin_' -'spill' -'spielte' -'spiele_' -'spher' -'sph' -'spezialisierte' -'spezialisiert_' -'sperr' -'sper' -'spektrum_' -'spekt' -'speicher' -'speculati' -'spectac' -'specif' -'specially_' -'specialist' -'specialis' -'spark_' -'spannende' -'spanische' -'spalte' -'spac' -'sozioökonomische' -'soz' -'sovi' -'souverän' -'souls_' -'sorgte_' -'sorgan' -'sonstigen_' -'song_' -'sonabl' -'solusi_' -'solo_' -'soll' -'soi' -'sofortige' -'soci' -'soaring_' -'soap' -'sness_' -'sneak' -'smug' -'smoke' -'smitte' -'smell_' -'smallest_' -'small' -'slowenisch' -'slope_' -'slin' -'slide' -'slau' -'sland_' -'skontroll' -'skew' -'skeptisch' -'siu' -'sitzt_' -'sistem_' -'sinkt_' -'sincere_' -'simplify_' -'simo' -'similari' -'simila' -'silent' -'signifikante_' -'signi' -'signalisier' -'signal' -'sige' -'sies_' -'sierende' -'sieb' -'sider' -'sidelin' -'sid' -'sichtlich_' -'sicherge' -'sicherere' -'sica_' -'sibl' -'sibi' -'shri' -'showers_' -'shot' -'shortfall_' -'shorter_' -'shore_' -'shof' -'shir' -'shifting_' -'shie' -'shi_' -'she' -'shatter' -'sharpe' -'sham' -'shadow' -'shade' -'sges' -'sger' -'sgebiet' -'sfähig' -'sfr' -'sfe' -'sfa' -'sexual' -'setzten_' -'setze_' -'setup_' -'settle_' -'servici' -'serv' -'seria' -'sequence_' -'sequen' -'sepe' -'separated_' -'sensitivity_' -'sensibl' -'senh' -'sender_' -'sena' -'seminar' -'semi_' -'selu' -'selige' -'self' -'selecti' -'sela' -'sekitar' -'seitig' -'sein' -'sehingga_' -'sehbare' -'seeds_' -'secu' -'sectarian_' -'secrets_' -'secrecy_' -'sechs' -'sece' -'season' -'searche' -'sdi' -'sdat' -'sda' -'scrib' -'scourge_' -'schönes_' -'schädlich_' -'schwing' -'schwierig' -'schwerwiegende_' -'schweizer' -'schwarzen_' -'schwache_' -'schule' -'schuldig_' -'schte_' -'schs' -'schrittweise' -'schriftliche_' -'schriftliche' -'schon' -'scholarship' -'schnellstmöglich' -'schnellstens_' -'schlä' -'schlosse' -'schlichte' -'schlechte' -'schlage' -'schizophreni' -'schirm' -'schienen_' -'schenk' -'scheine' -'scheiden' -'schadet' -'scenery_' -'scenarios_' -'scen' -'scatter' -'scar' -'scal' -'sburg_' -'saver' -'saudi' -'satisfy' -'satisfaction_' -'sar_' -'sangat_' -'sanf' -'sandy_' -'sammlung' -'sames_' -'salv' -'salon' -'salmon' -'sall' -'sail' -'sah' -'sacred_' -'sabotage' -'saa' -'rüst' -'rüh' -'rüf' -'rückt_' -'rücke_' -'rüch' -'römische' -'rés' -'ränken_' -'räglich' -'räder' -'rwart' -'rva' -'rust' -'russi' -'rush' -'rupulous' -'rupt' -'rumo' -'ruft_' -'rufe' -'rue' -'ruch' -'ru_' -'rting_' -'rtig' -'rthe' -'rter_' -'rsion' -'rse_' -'rren_' -'rozess_' -'rox' -'routine' -'rous_' -'rounde' -'rote_' -'rotat' -'rose' -'ror_' -'rooftop' -'rome_' -'roman_' -'roller' -'roh' -'rogue' -'rogramm_' -'rocks_' -'robie' -'rnt_' -'rmt' -'rmina' -'rmat' -'rman_' -'rlin' -'rkste_' -'riös' -'riu' -'ritte' -'ritt_' -'ritis' -'riter' -'risi' -'rises_' -'rische_' -'ringung_' -'ringe' -'rima' -'rigkeit_' -'rigen_' -'rifft_' -'riffen' -'riff' -'riesiger_' -'riesige_' -'rieben' -'ridge' -'richtungen_' -'richtung' -'richer_' -'ribut' -'riad' -'rgen' -'rfs' -'rfer' -'rey_' -'reward' -'revolution' -'revolt' -'revo' -'revision_' -'revision' -'reva' -'rev' -'rett' -'retreat_' -'retic' -'reth' -'reten_' -'rete' -'retains_' -'retained_' -'ret_' -'result' -'rests_' -'restru' -'reste' -'responds_' -'responding_' -'respi' -'respecting_' -'resolu' -'resistant_' -'resident_' -'resi' -'reservier' -'resentment' -'resemble' -'repudiat' -'republikanischen_' -'repräsentati' -'repro' -'representa' -'repositor' -'replacing_' -'repercussions_' -'repea' -'repay_' -'rente' -'renewal_' -'renc' -'remedy_' -'religiös' -'relian' -'relevan' -'releases_' -'rela' -'reitungs' -'reist' -'reise' -'reinve' -'reinforces_' -'reif_' -'reie' -'reicher' -'reichend_' -'reibungslosen_' -'reh' -'reguläre' -'regulier' -'registr' -'regio' -'regi' -'regel' -'regain_' -'refuses_' -'refurbish' -'refu' -'refrigerat' -'refresh' -'reformiert' -'refine' -'reduzierte' -'reduce' -'redo' -'redis' -'redet' -'recyc' -'rect' -'recreat' -'recommend' -'recom' -'reckt_' -'reciproca' -'rechnet' -'rechen_' -'recepti' -'receiver' -'receipt' -'rebuilt_' -'rebuild' -'rebell' -'reba' -'realm' -'reakti' -'reagierte' -'reaffirm' -'readin' -'reacted_' -'react' -'rds_' -'rdnung' -'rdinate_' -'rde_' -'rda' -'rbu' -'rbi' -'rbeit' -'raums_' -'rations_' -'rationalis' -'ratings_' -'ratify_' -'ratifizieren_' -'rassis' -'raschen_' -'rape_' -'rangig' -'rally_' -'rakete' -'raid_' -'raf' -'rado_' -'radikal' -'rad' -'raci' -'quote' -'quicker_' -'quero' -'quenz' -'quee' -'quasi_' -'quart' -'quar' -'qualities_' -'qualify_' -'pé' -'puri' -'purchased_' -'punishment_' -'punishe' -'pundits_' -'punct' -'puff' -'publishe' -'publications_' -'publication_' -'publi' -'pub' -'ption' -'ptic' -'pti' -'pter_' -'psychologi' -'psychi' -'pson_' -'prüfe' -'präzise_' -'präventive' -'pruden' -'prozessor' -'prozess' -'proz' -'proyecto' -'proximité_' -'provoke' -'provisional_' -'protocol' -'protestier' -'protectionist_' -'prostitution_' -'prosper_' -'prosecutor' -'prosecute' -'propag' -'proofed_' -'prone_' -'prompt_' -'promo' -'promi' -'projekten_' -'proj' -'progress' -'programmier' -'programme' -'prognos' -'profile' -'profession_' -'produz' -'produktiver' -'produktiven_' -'prochen' -'processor_' -'processed_' -'problematische' -'probe' -'probability_' -'prizes_' -'privatization' -'privatiz' -'privat' -'principal' -'preview_' -'presumably_' -'pressu' -'presse' -'presenta' -'prep' -'premises_' -'premier_' -'prejud' -'preisen_' -'predecessor_' -'predator' -'precon' -'precis' -'preceding_' -'preceded_' -'praktischer_' -'practise' -'prachige' -'prach' -'ppt_' -'ppne' -'pper_' -'pparat' -'potenzial_' -'potenti' -'posting_' -'poster' -'possesse' -'positiver_' -'positively_' -'positionier' -'posit' -'portugiesischen_' -'portfolio_' -'pornographi' -'porat' -'populist' -'popu' -'pop_' -'pons' -'poni' -'pone' -'polizeilichen_' -'polio_' -'pointer' -'poet_' -'pneum' -'pluralist_' -'plug' -'pling_' -'plikat' -'plica' -'ples' -'pleas' -'playground_' -'plausible_' -'plaus' -'platzier' -'platz' -'platte' -'plates_' -'planet' -'pl' -'pix' -'piso' -'pirate' -'pira' -'pieler' -'phä' -'physische' -'physics_' -'photographer' -'philosophers_' -'philo' -'pher' -'phen' -'phases_' -'pharma' -'pg' -'pflicht_' -'pfer_' -'pfei' -'pfa' -'pezifische' -'petro' -'pest_' -'pessimist' -'pes' -'perusahaan_' -'perta' -'pert_' -'persuaded_' -'perspectiv' -'personal' -'persona' -'perso' -'persiste' -'persecut' -'perse' -'permanente' -'perja' -'periodi' -'performa' -'perform' -'perfekten_' -'perfekte_' -'perceptions_' -'peny' -'pensioner' -'pengu' -'peng' -'penal' -'pektive' -'pedi' -'peculiar' -'pect' -'pc' -'paßt_' -'payer_' -'paya' -'pay' -'pause_' -'patron_' -'patriot' -'patriarch' -'patience_' -'pati' -'patenti' -'patch' -'patan_' -'passp' -'passenden_' -'passende' -'passen_' -'passage_' -'passa' -'pasa' -'partition' -'partisan' -'particul' -'participa' -'partially_' -'partial_' -'partei' -'parte' -'parque' -'parlam' -'paris' -'parc' -'parasit' -'paran' -'parag' -'paradoxical' -'parado' -'paradise_' -'panne' -'panels_' -'panc' -'palästinensische' -'palm' -'pale' -'palace_' -'pai' -'pack_' -'pable_' -'overwhelmingly_' -'overthrow' -'overs_' -'overlap' -'overl' -'overd' -'ova_' -'ova' -'outright_' -'outrageous_' -'outr' -'outer_' -'outbreaks_' -'oup' -'ough' -'otten_' -'osten' -'ostasiatische' -'osphär' -'osph' -'oso' -'osh' -'orz' -'orum_' -'orten' -'orre' -'ormi' -'origine' -'originated_' -'original' -'orientieren_' -'orientation_' -'orie' -'oria' -'organisierten_' -'organise_' -'organisch' -'ordinat' -'orden' -'orch' -'orb' -'orate' -'oral' -'optimistic_' -'optimist' -'optimis' -'optimalen_' -'optim' -'opfern' -'operative_' -'operationen_' -'open' -'ope_' -'oor' -'oo_' -'onwards_' -'onste' -'onomi' -'onment' -'oniert' -'oner' -'onc' -'onale' -'onal' -'ommt_' -'ommen' -'oming_' -'omen_' -'oman' -'olv' -'olungs' -'ols' -'olli' -'olle' -'oll_' -'oliz' -'olive_' -'oliga' -'okan_' -'oire' -'oint_' -'ohnehin_' -'ohl' -'ogic' -'ogenheit_' -'ogati' -'offsho' -'offenkundig_' -'offe_' -'offe' -'ocken_' -'ocht' -'oceans_' -'occurring_' -'occurrence_' -'occupying_' -'occupies_' -'occasion' -'obsole' -'observ' -'obscure' -'oblige' -'obligator' -'objektive' -'objektiv' -'objection_' -'obgleich_' -'obesity_' -'obacht' -'oba' -'oard' -'nützig' -'nöte' -'nössische' -'näher' -'nzende' -'nym' -'nvi' -'nven' -'nutzung' -'nutri' -'nurse' -'nungsl' -'nungs' -'nukl' -'nuan' -'nty_' -'ntrat' -'ntische' -'ntal' -'nstig' -'nst_' -'nsic_' -'npo' -'nour' -'notwendiger' -'notw' -'notified_' -'notebook' -'normen_' -'normale' -'nop' -'nons' -'nonetheless_' -'nominal' -'noble_' -'nob' -'nnial' -'nlich' -'nli' -'nland' -'nl' -'nitt_' -'nistr' -'niss' -'nischer_' -'nik' -'night' -'nieren' -'niedrigste' -'niederzu' -'nichts' -'nia_' -'nho' -'nheit' -'nhaft' -'ngliche' -'ngle' -'ngl' -'ngkin' -'ngh' -'ngens' -'ngen' -'ngeh' -'nfr' -'neutr' -'neuerliche' -'neueren_' -'neuem_' -'nette_' -'nets_' -'nerv' -'nent' -'nenn' -'nel' -'neighbours_' -'neighbor_' -'neid' -'nei_' -'negeri_' -'negativ' -'need' -'ndten_' -'ndliche_' -'ndli' -'nding_' -'nderung_' -'ndern' -'ndene' -'ndelt' -'ndbar_' -'nbild' -'nberg_' -'nbe' -'nationality_' -'nationalistische' -'nationalis' -'nannten_' -'name' -'naive_' -'nahegeleg' -'nage' -'nachzudenken_' -'nachlassen_' -'nachl' -'nachfo' -'münd' -'möglichkeit_' -'möglich' -'même_' -'mé_' -'männliche' -'mw' -'mußt' -'mutmaßliche' -'mutati' -'muslimische_' -'muslimisch' -'municipality_' -'muni' -'multiplie' -'multinationalen_' -'multilingual' -'mulat' -'mud' -'muc' -'mps' -'mple' -'mpho' -'mp3' -'mour' -'motorways_' -'motivierte' -'motivieren_' -'motivated_' -'mother' -'mosqu' -'mosa' -'mort' -'moreover_' -'moralisch_' -'moralis' -'moral' -'monsters_' -'monopolies_' -'mond_' -'monate' -'mois' -'mog' -'modifications_' -'modif' -'moderati' -'modelle_' -'modele' -'modalit' -'modal' -'mobilisiert_' -'mobil' -'mmu' -'mmission' -'mming_' -'mitzuteilen_' -'mittlere_' -'mittl' -'mitt' -'mithilfe_' -'mitgliede' -'mitb' -'mismanage' -'miser' -'mino' -'minimalis' -'minati' -'minat' -'mimic_' -'millenni' -'militia' -'militarily_' -'militari' -'milia' -'milestone' -'mildern_' -'migrator' -'miete' -'mfa' -'metry_' -'metropolitan' -'metropolis' -'meti' -'metaphor' -'metal' -'merupakan_' -'mering' -'merikas_' -'merika' -'merger_' -'merge_' -'menun' -'mention' -'mentally_' -'mentali' -'menschlich' -'meno' -'meni' -'mengg' -'mendo' -'mendapat' -'memori' -'memiliki_' -'membe' -'melo' -'meld' -'melak' -'meinte_' -'mehrheitlich' -'mediterran' -'medien' -'mbr' -'mbo' -'maßgebliche' -'maßgeblich_' -'mayors_' -'may' -'maturit' -'matt' -'matri' -'matisch' -'matis' -'mathematical_' -'mathemati' -'material' -'matching_' -'masyarakat_' -'massacre' -'mass' -'marvel' -'mars' -'markiert_' -'marki' -'mari' -'manufa' -'mans' -'manifestation' -'mane' -'mandate' -'mancher_' -'managements' -'macht' -'lüsse' -'lösungen_' -'lärung' -'lärm' -'längeren_' -'läh' -'lädt_' -'läche' -'lve_' -'luss' -'lui' -'ltungen_' -'lts' -'lton' -'ltet' -'ltes' -'ltere' -'ltene' -'lpin' -'love' -'lov' -'loser_' -'lop' -'loop' -'looming_' -'longe' -'lohnen' -'logisch_' -'login' -'logen_' -'lod' -'locals_' -'loading_' -'load' -'lls_' -'lligte' -'lles_' -'llect' -'llb' -'lj' -'liziert_' -'livel' -'litä' -'litt' -'lita' -'lit_' -'listing_' -'lish' -'lio_' -'linux' -'linking_' -'linke' -'lining_' -'linien_' -'lini' -'linguist' -'lingeri' -'linge_' -'linen_' -'limitations_' -'limit' -'likes_' -'liga' -'lifts_' -'lifting_' -'liest_' -'liegende_' -'lieferungen_' -'lief' -'lied' -'liebe' -'licht' -'lichem_' -'licenses_' -'licens' -'lic_' -'libert' -'liberalism' -'liable_' -'lia_' -'lia' -'lge_' -'lga' -'lfi' -'leverag' -'leva' -'leut' -'letztere' -'letzt' -'lete' -'lest_' -'lesse' -'lers_' -'lern_' -'leo' -'lent_' -'lende' -'lend' -'lement' -'lek' -'leite' -'leistungsstarke' -'leiste' -'leidenden_' -'leid' -'leichten_' -'lehnte_' -'legung_' -'legu' -'legislative' -'legale' -'lega' -'lecht' -'lebendige' -'lebenden_' -'lean' -'lden' -'lc' -'layo' -'layers_' -'laut' -'launder' -'laugh_' -'laub' -'latte' -'lations' -'lateinamerikanischen_' -'lated_' -'lares_' -'lapse' -'langwierige' -'langsamer' -'langsame' -'landwirtschaftliche' -'landscape' -'landmark_' -'landes_' -'landes' -'landed_' -'lamat' -'lain_' -'laim_' -'lager' -'laboratory_' -'küste_' -'künften_' -'kündig' -'kümmert_' -'kühle' -'körperlich' -'käufe' -'kämpfung_' -'kämpft' -'kus_' -'kurzlebig' -'kurdische' -'kunst_' -'kung_' -'kundig' -'kulturelle' -'kul' -'ktive_' -'ktie' -'ksh' -'kse' -'kräfte_' -'kritisieren_' -'kriti' -'kriminelle' -'krie' -'kreis_' -'krebs_' -'kreative_' -'krankheit' -'kraftwerke_' -'kota' -'kosm' -'korrigiert_' -'korr' -'kopp' -'koordin' -'kooperieren' -'konzipier' -'konzentrierte_' -'konzentr' -'konv' -'kontinu' -'kontakt' -'konsum_' -'konsultier' -'konfigur' -'kompromi' -'komponente' -'komplexer' -'komple' -'kommiss' -'kommene_' -'kommende_' -'kommand' -'komitmen' -'komfort' -'kolo' -'kohärent' -'kna' -'klüg' -'klini' -'kling' -'klick_' -'klein' -'klause' -'kland_' -'kk' -'kipun_' -'king' -'kids_' -'kick' -'keyboards_' -'keyboard_' -'keti' -'kep' -'kennzeichne' -'kenntnis' -'kenne' -'kelo' -'kell' -'kela' -'keitsp' -'kehrten_' -'kehrte_' -'keh' -'keeper' -'ked' -'kd' -'kauft' -'kati' -'katastrophen' -'katastrophalen_' -'katastroph' -'kass' -'kart' -'kapitalistischen_' -'kapitalistisch' -'kapitali' -'kapit' -'kanäle_' -'kanzler_' -'kannte' -'kanische_' -'kane' -'kampagnen_' -'kammer' -'kad' -'kabel' -'kab' -'kW_' -'jüngst_' -'jüdisch' -'jähriger_' -'jähr' -'juta_' -'justizielle' -'justifiable_' -'junior' -'jump_' -'juice' -'judgments_' -'juan' -'jp' -'joy_' -'joy' -'jours_' -'journe' -'journal_' -'jour_' -'jor_' -'jon' -'jm' -'jk_' -'jk' -'jen' -'jemandem_' -'jected_' -'jaz' -'jang' -'jail_' -'jahrzehntelang' -'jahrelange' -'jad' -'jacuzzi_' -'iß' -'izer_' -'iw' -'iver' -'itäre' -'itä' -'itz_' -'ituation' -'itted_' -'itiv' -'itischen_' -'itions' -'itin' -'ithm' -'itglied' -'ited_' -'itas_' -'istische_' -'istans_' -'istan_' -'issen' -'isse_' -'isse' -'issa' -'israel' -'ism' -'ision' -'isier' -'irs' -'irrig' -'irresponsibl' -'irresp' -'irrel' -'irregular' -'iro' -'irku' -'irische_' -'irgendwo_' -'irgendwie_' -'irgendwelche' -'ipt' -'ipp' -'ipl' -'iose' -'ios' -'ionss' -'ionist' -'iologi' -'inz' -'inward_' -'invoice' -'investition' -'investigations_' -'invest' -'inventor' -'invented_' -'invent' -'introduces_' -'inti' -'inters' -'interoperability_' -'interne_' -'internally_' -'interfere' -'interessierte' -'interdi' -'interconnect' -'interactive_' -'intensiver_' -'intensify' -'intellectuals_' -'intell' -'integrative' -'integrati' -'integr' -'int_' -'int' -'insurgen' -'insul' -'instruct' -'institutionelle' -'inste' -'inspirierende' -'inspire' -'inspe' -'insolvenc' -'insight_' -'inser' -'insel' -'inputs_' -'innu' -'innovativer' -'inners' -'innenpolitische_' -'innenpolitisch' -'inne_' -'inm' -'inland_' -'inko' -'injustice_' -'initi' -'iniste' -'inherit' -'inherently_' -'ingt_' -'informi' -'informelle' -'informationen_' -'informa' -'influen' -'infl' -'infizier' -'inferior_' -'infection_' -'infecti' -'inexpe' -'inertia' -'inequalities_' -'inen' -'ineff' -'indung' -'indices_' -'indications_' -'indexe' -'indefinite' -'inda' -'incremental' -'incorporati' -'incor' -'incomplete_' -'incompa' -'incline' -'inciden' -'inbe' -'inau' -'ination_' -'inapp' -'inanzierung' -'inan_' -'inali' -'inacc' -'imstande_' -'impul' -'impu' -'imprisoned_' -'impressi' -'impressed_' -'importi' -'importe' -'implying_' -'implizi' -'impli' -'implant' -'impf' -'imperialist' -'imperial_' -'imperfect_' -'impe' -'immun' -'imminent_' -'immigkeit' -'immerhin_' -'imma' -'imeter' -'imer_' -'iment' -'iman' -'imagination' -'image' -'illustrate' -'illig' -'ilis' -'ilian' -'ilet' -'iles' -'iler' -'ildet_' -'ila_' -'iki_' -'ikan' -'ihood' -'igte_' -'igte' -'igste' -'igert_' -'igenen_' -'igende_' -'igend' -'igat' -'igan' -'ify_' -'ifica' -'ific' -'iffe' -'ifer' -'ießungs' -'ieu' -'iertes_' -'ierte' -'ierli' -'ierenden_' -'ientierte' -'ient_' -'iell' -'iehungs' -'iegende' -'iegen' -'ieferung_' -'iefe' -'idylli' -'idos' -'ido' -'ideologische' -'identifizieren_' -'identi' -'iden_' -'iden' -'icu' -'ickt_' -'ickel' -'ichtliche' -'ichteten_' -'ichtete' -'icherte_' -'icherheit_' -'icherheit' -'ibus' -'ibly_' -'iben' -'ibel' -'ibe' -'iate' -'iar' -'iani' -'iana_' -'iam' -'ially_' -'iale' -'iad' -'höheres_' -'höchstwahrscheinlich_' -'häuser_' -'häufigste' -'härter' -'hysteri' -'hypothes' -'husband_' -'hurt_' -'hunde' -'humiliati' -'humanis' -'humane' -'huk' -'hub_' -'hter' -'hrte' -'hrop' -'hp' -'hov' -'hotele' -'hostilit' -'hostages_' -'hospi' -'horses_' -'horri' -'hormon' -'horizontal' -'horizon' -'honour' -'hones' -'holis' -'holder_' -'hochgradig_' -'hochentwickelte' -'hnya_' -'hns' -'hma' -'hlung_' -'hls' -'hlen' -'hitt' -'historischer_' -'hist' -'hion' -'hinweg' -'hinterla' -'hina' -'hilfe' -'highlights_' -'high' -'hig' -'hieß' -'hie' -'hid' -'hibit' -'hib' -'het_' -'hesitate_' -'hesita' -'herzlichen_' -'hervorgeh' -'hervorge' -'herstellen_' -'herrschende_' -'herrschen_' -'herrsche' -'herrlichen_' -'heroic_' -'hergestell' -'hered' -'heranzu' -'hene_' -'helpful' -'helme' -'hellen' -'helle_' -'helle' -'helicopter_' -'heitlich' -'heilige' -'heikle' -'hegemon' -'heftig' -'heel' -'hectare' -'heblich' -'heat' -'hear' -'headwa' -'haven_' -'hasn' -'harvest' -'harn' -'harmonisch' -'hap' -'hanging_' -'handlung' -'handelte_' -'handelbar' -'haltung_' -'haltig' -'haltestelle' -'halle_' -'half' -'hake_' -'hairdryer' -'hafter_' -'hafte' -'hadi' -'habitacion' -'habita' -'habet' -'güt' -'günstigste' -'günstigen_' -'gültig' -'gé' -'gäste' -'gänzlich_' -'gut' -'gust' -'guise' -'guilty_' -'guiding_' -'gue_' -'guaranteeing_' -'größtmögliche' -'grundlage_' -'großzügige' -'großartigen_' -'grow' -'groundwork_' -'grin' -'grim' -'grid_' -'gres' -'greife' -'greet' -'gray_' -'grave' -'grau' -'gratul' -'grass' -'grants_' -'grande' -'gram' -'grain_' -'graduated_' -'gradual_' -'grac' -'governor_' -'gove' -'gott' -'goldene' -'goa' -'gno' -'gnizing_' -'gnisse' -'gne' -'gnant_' -'gmatis' -'glück' -'glori' -'globe_' -'globalisierte' -'global' -'glied' -'gler' -'gleichg' -'gle_' -'glaubwürdige' -'glanc' -'gl' -'gische' -'gins' -'ginat' -'giganti' -'gift_' -'gift' -'ghts_' -'gha' -'ggf_' -'gger' -'gged' -'gf' -'geänderte' -'gezielt_' -'geze' -'gez' -'gewünschte_' -'gewü' -'gewöhnt_' -'gewährte' -'gewä' -'gewohnt_' -'gewisses_' -'gewerbliche' -'geweiht_' -'gewarnt_' -'gewann_' -'gewandt' -'gewalttätig' -'gewaltsam' -'gewaltigen_' -'gewaltige' -'getreten_' -'getrennt_' -'geteilte' -'geteilt_' -'gesunde_' -'gesucht_' -'gestric' -'gesteuert' -'gestell' -'gestalt' -'gesichert_' -'gesetzlichen_' -'gesetzliche_' -'gesetzlich_' -'geschütz' -'geschu' -'geschränkt' -'geschri' -'geschoss' -'geschn' -'geschmackvoll_' -'geschla' -'geschickt' -'geschichte' -'geschic' -'geschi' -'gescheiterten_' -'gescheitert' -'gerä' -'gert' -'germ' -'geringst' -'geringfügig_' -'geringe' -'gerichteten_' -'gerichte' -'gericht' -'geplante' -'geometri' -'geologi' -'geographic' -'geografisch' -'genügt_' -'gentur' -'generosity_' -'generali' -'genauen_' -'gemischte' -'gemeinschaftliche_' -'geme' -'gelöscht_' -'gelä' -'gelobt_' -'gelingen_' -'gelin' -'geliebt' -'geleistete' -'gelei' -'gelegenen_' -'gelangte' -'geladen_' -'gela' -'gekauft_' -'geka' -'gehörte' -'gehöre' -'gehend' -'geheimnis' -'gehe' -'geh' -'gegl' -'gegenübersteht_' -'gegebenenfalls_' -'gegebene' -'geführte' -'gefühl_' -'gefüg' -'gefälscht' -'gefährdete' -'gefundenen_' -'gefl' -'gefahren_' -'geeinigt' -'geehrter_' -'geda' -'gebun' -'gebro' -'gebnis' -'gebilligt_' -'gebieten_' -'geben' -'gay_' -'gathering' -'gathered_' -'gastronomy_' -'garten_' -'garde_' -'gant_' -'game' -'gamb' -'galow' -'galleries_' -'gale' -'gab' -'fürs' -'fürchtet' -'fünfzehn' -'führend_' -'führ' -'fühle' -'fügt_' -'fügen_' -'förm' -'fälsch' -'fusion_' -'funktions' -'funktional' -'funktion' -'fundamentalism_' -'fulfilled_' -'fulfill_' -'fulfil_' -'fug' -'fueling_' -'fting_' -'ftige' -'frühe' -'frustration' -'fruitful_' -'fruchtbare' -'front' -'froh_' -'frighten' -'frequen' -'fremden' -'freiz' -'freiwillige' -'frameworks_' -'frames_' -'fram' -'fraglich' -'foto' -'fosil_' -'fortunately_' -'fortunate_' -'fortschrittliche' -'fortschritt' -'fortgesetzte' -'fortgesetzt_' -'forsche' -'fors' -'formulation' -'formier' -'formelle_' -'forme' -'forgive' -'foreseeable_' -'foremost_' -'forecast' -'forbidden_' -'foodstuffs_' -'folgender' -'folgend' -'folde' -'flüsse_' -'flüge' -'flächen_' -'flus' -'fluctuations_' -'fluc' -'flow' -'fließ' -'fliehen_' -'flich' -'flex' -'flat' -'flam' -'fk' -'fizierten_' -'fixing_' -'fist' -'fishe' -'fische' -'firmen_' -'finishing_' -'finishes_' -'fing' -'finanzierten_' -'finanzier' -'finanziell_' -'filme' -'fill' -'fies_' -'fid' -'fici' -'fib' -'feuer' -'fett' -'festh' -'festge' -'fester_' -'fertiliz' -'fert' -'fers' -'fernseh' -'ferenz' -'fera' -'feminist' -'felde' -'feindliche_' -'fehlerhafte' -'fehlende_' -'fehl' -'feed' -'fear' -'faz' -'favori' -'favorable_' -'faun' -'fathers_' -'fastest_' -'fashioned_' -'farm' -'fang_' -'fana' -'familien' -'falt' -'fallend' -'fair' -'fahrten_' -'facult' -'faction_' -'faction' -'facet' -'fabricat' -'fabri' -'fa_' -'ezi' -'ezei' -'eze' -'extract_' -'extinction_' -'exte' -'expressions_' -'exportiert' -'exportieren' -'exporters_' -'exported_' -'exponenti' -'explosion' -'exploited_' -'explod' -'explizit' -'explanations_' -'expecting_' -'expectancy_' -'expansive' -'expans' -'expandier' -'exot' -'exklusive_' -'existen' -'exert' -'exer' -'excuse' -'exceeds_' -'exceeded_' -'exceed_' -'examining_' -'examination' -'ewicht' -'evolved_' -'evolve_' -'eviden' -'evaluated_' -'eva_' -'ev_' -'eut_' -'eus' -'europä' -'etung_' -'ets' -'etri' -'etliche' -'ethni' -'ethisch' -'ethics_' -'etet' -'estan' -'establishes_' -'establ' -'esst' -'essourcen_' -'essors_' -'espe' -'esp' -'esisch' -'esie' -'esh_' -'esar' -'esa_' -'esa' -'erzählt_' -'erzwingen_' -'erzw' -'erzig' -'erzielten_' -'erzieh' -'erzi' -'erzeugung_' -'erworben' -'erwirt' -'erweckt_' -'erwach' -'eruf' -'eru' -'ertr' -'erti' -'erteilt_' -'erteilen_' -'ersönlichkeit' -'ersto' -'erstmalig' -'ersorgung_' -'erschütter' -'erschweren_' -'erri' -'erreg' -'ero_' -'ernähr' -'ernsten_' -'erneuten_' -'ermöglichte' -'ermutigend' -'ermordet' -'ermittelt' -'ermitt' -'ermaßen_' -'erläutert' -'erlich' -'erkrank' -'erklärten_' -'erische' -'erin_' -'erhol' -'erho' -'erhebt_' -'erhaltene' -'ergreif' -'erfuhr' -'erforsch' -'erfasst_' -'erfassen_' -'eres_' -'erer' -'erenz' -'erend_' -'eren' -'ereit' -'ereignis' -'erbracht_' -'erate' -'erals' -'eradicati' -'erac' -'eption' -'epl' -'eou' -'envisage' -'environments_' -'entzi' -'entwickelnde' -'entwick' -'entum' -'entsp' -'entschä' -'entschl' -'ents' -'entrepreneurs_' -'entrepreneurial_' -'entrepr' -'entra' -'ento' -'entn' -'entla' -'entko' -'entitlement' -'ention' -'entia' -'enthusiastic' -'enthusiast' -'enthielt_' -'entg' -'entfern' -'entfalten_' -'enter' -'enteil' -'entdeckte' -'entailed_' -'entag' -'ensw' -'enrichment_' -'enpr' -'enormer_' -'ennen_' -'enne' -'enna_' -'enlarged_' -'enke' -'enische' -'enie' -'enhancing_' -'enhancement' -'engst' -'engines_' -'engere_' -'engagiert_' -'engag' -'energies' -'enerati' -'endors' -'endlos' -'endig' -'endg' -'encr' -'encounter' -'enchant' -'enberg_' -'enau' -'enan' -'enade' -'emulat' -'empör' -'empt' -'emphasized_' -'emphasises_' -'empfohlen_' -'empfinden' -'empfan' -'emitte' -'emit_' -'emis' -'eminent' -'emic' -'emergen' -'ementa' -'embraced_' -'embodie' -'emancipation_' -'elz' -'elu' -'elte_' -'elpr' -'elnde' -'elm' -'ellit' -'ellig' -'elimin' -'elektronische_' -'elektroni' -'eleg' -'electorate_' -'elderly_' -'ela_' -'ektors_' -'eiz' -'eiv' -'eise' -'eis' -'einzubringen_' -'einziger_' -'einzigartiger_' -'einzelstaatliche_' -'einzelstaatliche' -'einsetzt_' -'einschließ' -'einnimmt_' -'einn' -'einm' -'einleite' -'einlage' -'einka' -'einig_' -'einheitlich' -'einheit_' -'einheimische_' -'einhalten_' -'eingreif' -'eingesetzte' -'eingereichten_' -'einger' -'eingehende' -'eingefü' -'eingebrachte' -'einführt' -'einfließen' -'einfache' -'eindr' -'eilnehmer' -'eilig' -'eigt' -'eifen_' -'eichnungen_' -'eichnete' -'eichheit' -'eichen_' -'eiche' -'eibungen_' -'ehrliche' -'ehren' -'ehmend' -'ehm' -'ehemaliger_' -'egu' -'egr' -'egio' -'efficiently_' -'effi' -'effektive' -'eein' -'eed' -'educat' -'edoni' -'edly_' -'editorial_' -'editi' -'edingungen_' -'ediate_' -'ectiv' -'ect_' -'ecosystem_' -'economic' -'ecommerce_' -'eckung' -'echter_' -'echnologi' -'echni' -'ece' -'ec_' -'ebi' -'earthquake_' -'eare' -'eagle' -'ead' -'eab' -'eB' -'düstere_' -'dürft' -'dünne' -'dün' -'dó' -'dí' -'dè' -'dänische' -'dämpf' -'dynamisch_' -'dyn' -'dying_' -'durchgeführten_' -'durchführ' -'durable_' -'dura' -'duplicate' -'dule' -'dubio' -'dual_' -'dsch' -'drückt' -'drücke' -'drängt_' -'drink' -'dringender_' -'drige' -'dress_' -'dreim' -'dreie' -'drea' -'dran_' -'dramatische' -'drama_' -'drai' -'drafts' -'dozens_' -'dozen_' -'downs' -'downfall_' -'douche' -'doubl' -'dort' -'doppelten_' -'door' -'doom' -'dominieren_' -'dominance_' -'dom_' -'dokument_' -'doct' -'dock' -'doc' -'dne' -'dlin' -'divide' -'divert' -'diversification_' -'diversif' -'diver' -'dity' -'distributi' -'distinguish' -'distinctive_' -'distin' -'distanzier' -'distan' -'dissi' -'disqualif' -'disproportionate' -'disposition' -'displaced_' -'dispens' -'disparities_' -'disorders_' -'disor' -'diskriminier' -'diskreditier' -'disintegrati' -'discover' -'discontent' -'discl' -'discard' -'disar' -'disappoint' -'disappeared_' -'disadvantage_' -'disabl' -'dirty_' -'diri' -'direkter_' -'directors_' -'dir' -'diplomati' -'diplom' -'dingung' -'dinar' -'dimensionale' -'dimension' -'dilakukan_' -'diktat' -'dika' -'digte' -'diffus' -'differentiated_' -'differ_' -'diesjährigen_' -'diesel_' -'diesbezüglich_' -'diente_' -'dienstleistung' -'dictator_' -'dictator' -'dicta' -'dich' -'dice' -'dicat' -'dibandingkan_' -'dial_' -'diagnostizier' -'diagnosis_' -'dezentral' -'devise_' -'develop' -'devastat' -'deva' -'dev_' -'deutschs' -'deutliche' -'deutig' -'deterrence_' -'determina' -'detect_' -'detaine' -'destroying_' -'destotrotz_' -'destiny_' -'desti' -'destabilisieren' -'dest' -'desse' -'desperate_' -'desk' -'designi' -'designe' -'desert_' -'desert' -'descript' -'describing_' -'desc' -'derte' -'derse' -'derin' -'deput' -'depri' -'depressive' -'deplor' -'depict' -'depe' -'dent_' -'denkbar' -'demselben_' -'demonstrati' -'demoli' -'demokratische' -'demografische' -'democratiz' -'democratically_' -'demis' -'demics_' -'demi_' -'demagogue' -'dell' -'delight' -'delic' -'deleveraging_' -'deleted_' -'delet' -'deleg' -'delay' -'dei' -'deg' -'defizite' -'definite_' -'definierten_' -'deficien' -'defenses_' -'dedicati' -'decree' -'decoration_' -'decisively_' -'decen' -'dece' -'deca' -'debu' -'debat' -'deba' -'dealers' -'deaktiviert' -'deadlock_' -'daya_' -'day' -'davon' -'daughter_' -'dauern_' -'dauerhaft_' -'dauer_' -'datei_' -'dat_' -'darl' -'dark' -'darge' -'dank' -'dangerously_' -'dane' -'dance' -'dairy_' -'dai' -'dage' -'dae' -'dachte_' -'custom' -'curi' -'cup_' -'culos' -'culi' -'culati' -'cui' -'cture_' -'ctor_' -'cto' -'cteri' -'cruel_' -'crop_' -'crop' -'critics_' -'criticise_' -'cription' -'cred' -'creativity_' -'crea' -'cram' -'crafted_' -'covert' -'coverage_' -'courte' -'court' -'courag' -'counting_' -'counterpart' -'cosmo' -'corrupt' -'correspond' -'coordinate_' -'cool' -'cooking_' -'cookies_' -'convinc' -'conviction_' -'convey_' -'controvers' -'contro' -'contributes_' -'contraction' -'continuity_' -'continual' -'continental_' -'conti' -'contempt_' -'contemplate' -'container' -'consum' -'consultant' -'consolidate' -'console' -'consig' -'conservat' -'conscien' -'congratulat' -'confirms_' -'confine' -'conducti' -'condo' -'conditi' -'conceptual' -'concepts_' -'concentrating_' -'conceive' -'conceivable_' -'compromises_' -'comprise_' -'composition_' -'compos' -'complimentary_' -'competitors_' -'compet' -'communicati' -'commission_' -'commi' -'commerce' -'commentator' -'comitology_' -'combination' -'combin' -'colleg' -'collateral_' -'collaps' -'colla' -'coins_' -'cof' -'coexist' -'codec' -'cod_' -'cock_' -'clou' -'cloning_' -'cliente' -'clear' -'classification_' -'classif' -'clash_' -'clarifi' -'ckung_' -'ckten_' -'ckr' -'ckne' -'ckier' -'civiliz' -'civilisation' -'civ' -'circulation_' -'circula' -'circuit_' -'cip' -'ciona' -'ciation' -'ciar' -'chwor' -'chwi' -'chwe' -'chut' -'chuss' -'chur' -'chunk' -'chuh' -'chtliche' -'chtet_' -'chtern' -'chslung' -'chrono' -'chronis' -'chronic_' -'christ' -'chriften_' -'chreib' -'chooses_' -'choc' -'chnik' -'chni' -'chn' -'chme' -'chläge_' -'chli' -'chke' -'chisc' -'chip_' -'chinesischer_' -'chine' -'china_' -'chier' -'chicken_' -'chess' -'cherung' -'chere' -'chemis' -'chem_' -'charta_' -'characteris' -'chara' -'chanis' -'champions_' -'chambres_' -'chamber' -'cham' -'chain' -'chaff' -'ceu' -'cet_' -'certification' -'certificates_' -'certificate_' -'cents_' -'centrist_' -'centraliz' -'cent' -'cen_' -'cema' -'celebrati' -'celebrat' -'cele' -'ceiling_' -'cattle_' -'cations_' -'cater' -'catching_' -'catas' -'carte_' -'caro' -'cardi' -'cape' -'capacities_' -'cane_' -'campaign' -'calme_' -'calibration' -'cache_' -'bürger_' -'bünd' -'bücher' -'bü' -'byte_' -'byp' -'buyers_' -'bustl' -'burst' -'burgh' -'burg' -'bure' -'bundes' -'build' -'bug' -'buf' -'buen' -'bten' -'bte_' -'bsc' -'brut' -'brus' -'bruch' -'browse' -'brow' -'britischer_' -'bringen' -'brig' -'brew' -'brethren_' -'breiteren_' -'brechen' -'brav' -'brasilianische_' -'bras' -'brands_' -'bourne_' -'boundary_' -'boss' -'bos' -'borrowed_' -'bonus_' -'bomb' -'bodi' -'boden_' -'bnis' -'blü' -'blur' -'blood' -'blogs_' -'blogg' -'blockier' -'blin' -'bled_' -'blaue' -'blatt_' -'blasen' -'bisc' -'birthday_' -'birth' -'bird_' -'biom' -'biological_' -'biog' -'biofuel' -'bindung_' -'binary_' -'bilitati' -'bilis' -'bildete' -'bilder_' -'bilanz_' -'bil_' -'bike' -'biete_' -'bid' -'bibliot' -'bias_' -'bia_' -'bezwe' -'bezog' -'bezieh' -'bewunder' -'bewohne' -'bewirt' -'bewi' -'bewerbe' -'bewege' -'bewe' -'bewahr' -'bewaffnet' -'beur' -'beunruhigende' -'betriebs' -'betriebene' -'betr' -'betonte' -'beteiligten_' -'beteiligen_' -'beta' -'bestände_' -'bestr' -'besto' -'bestimmtes_' -'besti' -'bestens_' -'bestellt' -'bestellen_' -'besser' -'besorgniserregende' -'besie' -'besetzt_' -'beschriebenen_' -'beschrieb_' -'beschr' -'beschlossene' -'beschloss_' -'bescheiden' -'besche' -'besa' -'berühren_' -'beruhte_' -'beruhigen' -'berke' -'berichtete_' -'berh' -'bereits' -'bereichen_' -'berater' -'berada_' -'beobachtet_' -'benign_' -'bende' -'benar_' -'bemüh' -'belt_' -'belt' -'belongs_' -'belohn' -'belo' -'beln_' -'beliebt_' -'belegen_' -'beleben' -'bekunde' -'bekomm' -'beklagen_' -'bekl' -'beinhalten_' -'beider_' -'beherrscht_' -'behave' -'behauptete_' -'beharrt' -'begrüß' -'begrenzter' -'begrenzen_' -'beginn' -'begi' -'begegnet' -'befürworten_' -'befürworte_' -'befürchten_' -'befürchte' -'befu' -'befo' -'befindlichen_' -'befindliche' -'beeindruckt' -'beeindruckenden_' -'bedingt_' -'bedeutsam' -'bedeutender' -'bedeutend_' -'bedauere_' -'bedank' -'beauf' -'bearbeitung' -'bearbeitet' -'bear' -'beant' -'beans' -'beachtliche' -'beachte' -'beabsichtig' -'bbe' -'battles_' -'baths_' -'basierte' -'basel' -'baru' -'barre' -'barr' -'baro' -'barbe' -'barbari' -'bant' -'bans_' -'banker' -'bani' -'bana' -'bakar_' -'baj' -'baggage_' -'baden_' -'bacteria' -'backup_' -'backpack' -'backlash_' -'až_' -'año' -'axis_' -'ax' -'awards_' -'await' -'awa_' -'avoid' -'außergewöhnliche_' -'außerge' -'außenst' -'autoritäre' -'automatisier' -'automatischen_' -'authorita' -'authorisation_' -'authentic_' -'ausüb' -'auszusp' -'auszugeben_' -'auswirkt_' -'ausweiten' -'ausweis' -'auswa' -'austausch' -'aust' -'aussp' -'aussichten_' -'aussa' -'ausrüstung' -'ausrichte' -'ausreicht_' -'auslöste' -'ausländischer_' -'ausle' -'ausla' -'ausgewählte_' -'ausgewogen_' -'ausgewertet_' -'ausgetragen_' -'ausgerichteten_' -'ausgenommen_' -'ausgeglichene' -'ausgegli' -'ausgefü' -'ausgebe' -'ausgearbeitet' -'ausgab' -'ausfäll' -'auseinandersetz' -'ausdrückliche' -'ausdrück' -'ausar' -'aur' -'augen' -'aufzut' -'aufzust' -'aufzub' -'aufwer' -'auftrag' -'aufsch' -'aufs_' -'aufr' -'aufnahme_' -'aufkommende' -'aufk' -'aufhalten' -'aufgreif' -'aufgeworfen' -'aufgewe' -'aufgetr' -'aufgeben_' -'aufeinander_' -'auern_' -'audi' -'auction' -'auch' -'auben' -'atu' -'attraktiver_' -'attraktiv' -'attracted_' -'attacker' -'atrocities_' -'aton' -'atom' -'atmospher' -'ative' -'atische' -'atisch_' -'ationsa' -'athlet' -'ater_' -'aten' -'atel' -'atastrophe' -'assured_' -'assur' -'assu' -'associate' -'assert_' -'asse' -'assault' -'assassination' -'asis' -'asin' -'asce' -'artung' -'artist' -'artis' -'artikel_' -'artig' -'arriving_' -'arrange' -'arra' -'arom' -'armies_' -'arl' -'arke' -'aris_' -'arien_' -'arians_' -'argument' -'arguabl' -'arf_' -'arena_' -'ardin' -'archäologische' -'archiv' -'architektonische' -'archipelago' -'archi' -'arbeitung_' -'arbeitslos_' -'arbeitete' -'arabisch' -'appropriati' -'appropriate' -'appreciat' -'applicants_' -'applaud' -'appelliere_' -'apli' -'ao_' -'anzub' -'anzeig' -'anybody_' -'anwendungen_' -'antwortlich' -'antrags_' -'antivirus_' -'antim' -'antiken_' -'anticipate_' -'antibioti' -'anter' -'antag' -'anstrebt' -'ansieht_' -'anschließende' -'anschaue' -'anpassen_' -'annähernd_' -'anktionen_' -'ankl' -'anken_' -'ank_' -'anj' -'angriffe' -'angig' -'angga' -'angetrieben' -'angesp' -'angese' -'angesch' -'angesammelt' -'angenomme' -'angenehme_' -'angemessener' -'angeme' -'angehoben_' -'anga' -'anfü' -'anfängt_' -'anforderung' -'anesische' -'anerkenn' -'aner_' -'andin' -'andi' -'andeu' -'anchor' -'ance' -'analyti' -'analysing_' -'analy' -'analog_' -'anak_' -'ample_' -'amid' -'amending_' -'ambiguous_' -'ambient' -'ambience_' -'amba' -'altr' -'altet_' -'alternative' -'altern' -'alm' -'allo_' -'allmählich' -'alljährlich' -'alli' -'alleine_' -'allegedly_' -'alk' -'alive_' -'alität' -'alition' -'alisieren_' -'alin' -'align_' -'align' -'alienation_' -'alien_' -'alan_' -'aktualisiert' -'aktu' -'aktivitäten_' -'aktive' -'aktion_' -'akti' -'airspace_' -'aire' -'aikan' -'aid' -'ahlung' -'agung_' -'agt_' -'agrees_' -'agierende' -'aggebe' -'agendas_' -'agein' -'agar' -'aftermath_' -'afor' -'afghanischen_' -'afghanische_' -'affen_' -'affair_' -'aerospace_' -'advertise' -'adversaries_' -'adver' -'admissi' -'admira' -'administrator_' -'administrativ' -'administer' -'admin_' -'adjusting_' -'adic' -'ader' -'adan' -'activis' -'activ' -'acquis_' -'acquiring_' -'acqui' -'achtung' -'achte' -'acht' -'ached_' -'accueil_' -'accomodati' -'accommodate_' -'acco' -'accessibility_' -'accede' -'abzuw' -'abzus' -'abzielen_' -'abwechs' -'abundan' -'abstra' -'absorbi' -'absorb_' -'absol' -'absi' -'absent_' -'abschrecken' -'abort' -'abolition_' -'abolished_' -'abnehmen' -'abilität' -'abilities_' -'abili' -'abide_' -'abhalten_' -'abgeschl' -'abgescha' -'abgesch' -'aben' -'abduction' -'aban' -'aat_' -']]. _' -']] ' -'[ _' -'[' -'Zürich_' -'Zü' -'Zyp' -'Zyklus_' -'Zyklen_' -'Zwe' -'Zw' -'Zuwa' -'Zuw' -'Zuverlässigkeit' -'Zutaten_' -'Zuständ' -'Zusch' -'Zusammenschluss_' -'Zusammenhänge' -'Zusammen_' -'Zurückh' -'Zula' -'Zuhöre' -'Zuhause_' -'Zugeh' -'Zufriedenheit_' -'Zuflu' -'Zivilisation_' -'Zinssatz' -'Zin' -'Zimmerman' -'Zielvorgabe' -'Zeug' -'Zentren_' -'Zensur' -'Zem' -'Zellen' -'Zeitungs' -'Zeits' -'Zeitlinie_' -'Zeitge' -'Zahlungsv' -'Zahlungsaus' -'Zah' -'Yor' -'Yale_' -'YA' -'XWB_' -'XT' -'XII' -'XF' -'X1' -'Wünsch' -'Wäldern_' -'Wähler' -'Wähl' -'Wut' -'Wunde' -'Works_' -'Wolfs' -'Wolfgang_' -'Wolfensohn_' -'Wohnungen_' -'Wohnung_' -'Wohnb' -'Wohlstands_' -'Wohlergehen' -'Wohlbefinden_' -'Wochen' -'Wirtschaftsn' -'Wirtschaftsg' -'Wirtschaftsbe' -'Wirts' -'Wirks' -'Wire' -'Wirbelst' -'Wins' -'Wing' -'Winde' -'Wind_' -'Willens' -'Wille_' -'Wilders_' -'Wiedervereinigung' -'Wiederholung_' -'Wiederh' -'Wiederaufnahme_' -'Wieder_' -'Widget' -'Wichtiger' -'Wic' -'Whirlpool_' -'Whereas_' -'Whe' -'Wetter' -'Wettbewerbsvor' -'Wettbewerbsver' -'Wettbewerbspolitik_' -'Westeuropa_' -'Wesentliche' -'Wertschöpfung' -'Werts' -'Wert' -'Weltr' -'Weltor' -'Weltme' -'Weltkultur' -'Weltkrieges_' -'Welthandels' -'Weltg' -'Weltbevölkerung_' -'Weiße_' -'Weish' -'Weine_' -'Weile' -'Weigerung_' -'Wednesday_' -'Wechsel' -'Webserver' -'Way' -'Wasserkraft' -'Washing' -'Wanderung' -'Wallström_' -'Waldbrände' -'Wald_' -'Wake' -'Wahler' -'Wahlbe' -'Wagen_' -'Waffenstillstand_' -'Wachstumsraten_' -'Wachstumsrate_' -'WT' -'WS_' -'WM_' -'WEI' -'WAV_' -'Völkern_' -'Völkermord' -'Vö' -'Vé' -'Vä' -'Vá' -'Vulcan_' -'Vul' -'Vot' -'Vos' -'Vorstands' -'Vorsta' -'Vorst' -'Vorsitzende' -'Vorliebe' -'Vorla' -'Vorhersage' -'Vorgang' -'Vorgabe' -'Vorfall_' -'Vorder' -'Volvo_' -'Vollm' -'Volkswagen_' -'Volkspartei_' -'Volksabstimmung_' -'Vladimir_' -'Vizepräsident_' -'Vize' -'Vitamin' -'Virginia' -'Viol' -'Vik' -'Viewer_' -'View' -'Viertel' -'Vielf' -'Via' -'Verzögerung_' -'Verzweiflung' -'Verzug' -'Verwer' -'Verweise' -'Verw' -'Verträgen_' -'Vertrauens_' -'Vertragsver' -'Verteil' -'Verteidiger' -'Versäum' -'Verste' -'Versp' -'Verschwörung' -'Verschwi' -'Verschwendung_' -'Verschuld' -'Verschlechterung' -'Verschiedene_' -'Verschieb' -'Vers' -'Verpa' -'Verp' -'Vero' -'Vernichtung_' -'Verne' -'Vermeidung_' -'Verlaufe_' -'Verlangsamung_' -'Verkäufe' -'Verkehrst' -'Verkehrss' -'Verhandlung_' -'Vergleichss' -'Vergew' -'Verfassungsentwurf_' -'Verfall' -'Vere' -'Verbü' -'Verbr' -'Verbe' -'Verantwortlichen_' -'Veranstaltungs' -'Veran' -'Venus_' -'Vent' -'Venezia' -'Vec' -'Vaters_' -'Variablen_' -'VS' -'VIN' -'VA_' -'Uti' -'Urteile' -'Ursprünge_' -'Urlaube' -'Uribe_' -'Urheberrechts' -'Urbanis' -'Upgrade_' -'Unterwa' -'Untert' -'Unterschrift' -'Untersch' -'Unternehmensf' -'Unterl' -'Unterhaltung' -'Unterha' -'Unterbrechung' -'Untera' -'Unsinn_' -'Unrecht_' -'Unix_' -'Universitäts' -'Universal' -'Unit_' -'Unglücklicherweise_' -'Unglück_' -'Ungleichgewicht' -'Ungere' -'Une' -'Umweltver' -'Umweltschutz' -'Umwelts' -'Umweltpr' -'Umweltpolitik_' -'Umverteilung' -'Umstellung_' -'Umfragen_' -'UV_' -'URL' -'UNICEF_' -'Türke' -'Tyrol' -'Tyran' -'Tyr' -'Type_' -'Tusk' -'Turnier_' -'Tunis' -'Tunes_' -'Tsunami_' -'Tschad_' -'Träger_' -'Truste' -'Trojan' -'Trichet_' -'Trend' -'Tren' -'Treibhausgasemissionen_' -'Treibhausgas' -'Treasury_' -'Travel' -'Trau' -'Trap' -'Transit_' -'Transform' -'Transaktionen_' -'Trans_' -'Trainer_' -'Tow' -'Tours_' -'Tourist' -'Tool' -'Too' -'Tole' -'Tisch' -'Timor' -'Timo' -'Til' -'Tiger' -'Tiere' -'Tiera' -'Tic' -'Think' -'Thing' -'Therme' -'Therap' -'Ther' -'Theorien_' -'Theodore_' -'Thatcher_' -'Texte' -'Terrorismusbekämpfung_' -'Terroranschl' -'Territorium_' -'Terr' -'Terms_' -'Tendenzen_' -'Temperatur_' -'Temperatur' -'Televis' -'Telekommunikation_' -'Teilung' -'Teilnehmer' -'Technologies_' -'Technis' -'Technical_' -'Tea_' -'Taylor_' -'Tax_' -'Tax' -'Tatatabot_' -'Taste_' -'Tasche' -'Target_' -'Tale' -'Tagung' -'TZ' -'TW' -'TU_' -'TRO' -'TPP_' -'TEN' -'TA_' -'T2' -'Südtirol_' -'Südoste' -'Südostasien' -'Süd_' -'Säug' -'Sän' -'São_' -'Sz' -'Syriza_' -'Syndrom' -'Swim' -'Sustain' -'Surve' -'Surely_' -'Suprem' -'Sup' -'Sunnis_' -'Sunn' -'Summers_' -'Summer_' -'Summen_' -'Summ' -'Suf' -'Suda' -'Substantiv' -'Subsi' -'Subscri' -'Subs' -'Subm' -'Subjekt' -'Stücke_' -'Störun' -'Stärken_' -'Stuf' -'Studi' -'Ström' -'Stric' -'Strau' -'Strategic_' -'Strasse' -'Strafv' -'Stornierung' -'Stop' -'Stone' -'Stock' -'Stimmen' -'Stillstand_' -'Stie' -'Steven' -'Steuerh' -'Steuerer' -'Steuereinnahmen_' -'Stern' -'Stehende_' -'Stefan_' -'Stea' -'Statute_' -'Standpunkten_' -'Standpunkte_' -'Stammzellen' -'Stahl_' -'Stadtk' -'Stadi' -'Stabilitätspakt_' -'Staatsp' -'Staatsbürgerschaft_' -'Sprung' -'Sprin' -'Split' -'Spitzenpo' -'Spit' -'Spirit' -'Spezifikation' -'Spezialist' -'Spend' -'Spen' -'Spektrum' -'Speise' -'Speicherka' -'Spaß_' -'Spazier' -'Sparpolitik_' -'Sparmaßnahmen_' -'Sparen' -'Spam' -'Spalte' -'Sozialp' -'Sozialisten_' -'Sozialismus_' -'Sozialdemokraten_' -'Sowjet' -'Sonnenunterg' -'Sonderg' -'Sommers' -'Somalia_' -'Soft' -'Society_' -'Snowboard' -'Slu' -'Slovenia_' -'Sle' -'Skl' -'Skandal' -'Sitzungsperiode_' -'Sitz' -'Simon_' -'Silber' -'Siena_' -'Siedl' -'Sieben' -'Side' -'Sicherheitsbe' -'Sicherheit' -'Shut' -'Shows' -'Shop' -'Shinzo' -'Shell_' -'Shel' -'Sharia_' -'Sex' -'Seven_' -'Ses' -'Serv' -'Senkaku_' -'Senior' -'Senegal_' -'Sendung_' -'Semit' -'Self' -'Selbstz' -'Selbstvertrauen_' -'Selbstmord' -'Seiten' -'Seit' -'Sehen' -'Seeverkehr' -'Sechste' -'Sech' -'Seattle_' -'Seas' -'Screening' -'Schüt' -'Schönheit' -'Schön' -'Schö' -'Schwächung' -'Schwächen_' -'Schwinde' -'Schwimmbad_' -'Schwie' -'Schwester_' -'Schwellenmärkte' -'Schweigen_' -'Schwe' -'Schwarze' -'Schwachstelle' -'Schutzge' -'Schuss' -'Schuman' -'Schulter' -'Schuldenlast' -'Schuhputzmaschine_' -'Schröder' -'Schritten_' -'Schrei' -'School' -'Schock_' -'Schock' -'Schnitt' -'Schloss' -'Schlo' -'Schlecht' -'Schlacht' -'Schla' -'Schil' -'Schiene_' -'Schied' -'Schein' -'Schauspiele' -'Schatten' -'Schaff' -'Schaf' -'Schadens' -'Schad' -'Saš' -'Savo' -'Savi' -'Sauberkeit_' -'Sauber' -'Sard' -'Saraj' -'Sarah_' -'Santo' -'Sandstr' -'Sand_' -'Samo' -'Sammlungen_' -'Samb' -'Salva' -'Sali' -'Sak' -'Saint' -'Sahara_' -'Sachverhalte' -'Saal_' -'SSI' -'SPE' -'SOEs_' -'SMEs_' -'SL_' -'SIS_' -'SING' -'SDL_' -'SB' -'Rücküberweisung' -'Rücktritt_' -'Rückst' -'Rückhalt_' -'Rückführung_' -'Ryan_' -'Ry' -'Rural_' -'Run_' -'Run' -'Rum' -'Ruine' -'Ruhes' -'Rue' -'Roy' -'Row' -'Rotterdam_' -'Rotarier' -'Rost' -'Ros' -'Root' -'Roo' -'Ron_' -'Ron' -'Rohstoffpreise' -'Rohstoffe_' -'Robot' -'Robin' -'Risikobe' -'Rio' -'Rim' -'Richter' -'Rib' -'Rhythm' -'Rho' -'Rhin' -'Reze' -'Revi' -'Reve' -'Respons' -'Residen' -'Reservierung' -'Reservation' -'Republikan' -'Repräsentanten_' -'Repräsentant' -'Repression' -'Renzi_' -'Renten_' -'Rente_' -'Religions' -'Relevanz_' -'Relation' -'Reisez' -'Reisever' -'Reisetipp_' -'Reis' -'Reha' -'Register' -'Regierungspo' -'Regarding_' -'Refu' -'Reden_' -'Reco' -'Rechtssysteme' -'Rechtssystem' -'Rechtsst' -'Rechtssicherheit_' -'Rechtsr' -'Rechtschreib' -'Rechtsbe' -'Rechtsausschuss' -'Rechtsanw' -'Recherche' -'Rechenschaftspflicht_' -'Recep' -'Rebellion_' -'Realit' -'Realisierung_' -'Reading' -'Reac' -'Raums_' -'Raume' -'Rauchen_' -'Ration' -'Rating_' -'Ratifi' -'Rathaus' -'Rapid_' -'Rangliste' -'Ral' -'Raketen_' -'Rail_' -'Raci' -'RT_' -'RK' -'RANT' -'RAM_' -'RAC' -'Quoten_' -'Quellcode_' -'Quar' -'Quanti' -'Quant' -'Qualifikationen_' -'Qual' -'Quadrat' -'Qu' -'Qatar_' -'Qaddafi_' -'Qa' -'QUI' -'Pv' -'Push' -'Pump' -'Pull' -'Pub' -'Psycho' -'Psych' -'Präzisi' -'Präsidentschaftswahlen_' -'Präsentation' -'Prämien_' -'Präf' -'Provinz' -'Provider_' -'Prototyp' -'Proto' -'Proteine_' -'Protein' -'Protect' -'Propo' -'Propaganda_' -'Promenade_' -'Programmier' -'Profit_' -'Profil_' -'Professional_' -'Produkts' -'Produktionsst' -'Produktionspr' -'Product' -'Proc' -'Problemati' -'Privileg_' -'Privats' -'Privatis' -'Prior_' -'Priester' -'Prev' -'Prepa' -'Premierminister' -'Preisstabilität_' -'Preises_' -'Prakti' -'Prag' -'PowerP' -'Potter' -'Potsdam' -'Poten' -'Postgre' -'Postdienst' -'Posse' -'Position' -'Portugies' -'Portale' -'Porta' -'Population_' -'Pon' -'Politicians_' -'Pola' -'Plo' -'Plenar' -'Play_' -'Plata_' -'Plast' -'Planet_' -'Pizza' -'Pino' -'Pilote' -'Pilot' -'Pier' -'Pick' -'Picc' -'Pic' -'Photocopying_' -'Phoeni' -'Philippi' -'Pharma' -'Pflanz' -'Pfl' -'Pfeiler_' -'Pfad_' -'Peters' -'Perspective_' -'Persi' -'Perm' -'Perfe' -'Pension_' -'Pennsylvania_' -'Pend' -'Pemb' -'Pedro' -'Peak' -'Pax_' -'Paus' -'Pati' -'Patent_' -'Passw' -'Passport_' -'Passei' -'Passe' -'Pass_' -'Partner' -'Partic' -'Parti' -'Parm' -'Parlamentarier_' -'Parkplatz_' -'Parke' -'Paris' -'Parc' -'Paramet' -'Parallelen_' -'Parallel_' -'Paradox' -'Paolo_' -'Pani' -'Pand' -'Pan_' -'Palästinensischen_' -'Pakets_' -'Paint' -'Packag' -'PROGR' -'POL' -'PN' -'PLAYER' -'PIC' -'PAS' -'Oxford_' -'Ow' -'Outlook_' -'Ostse' -'Oster' -'Osborne_' -'Ortho' -'Ortega' -'Organisat' -'Organen_' -'Ordn' -'Order' -'Orden_' -'Optimi' -'Opportuni' -'Opfern_' -'Operation' -'Omni' -'Offi' -'Oe' -'Obersten_' -'Oberste_' -'Oberh' -'Obergrenze_' -'ORT' -'OM_' -'OFI' -'Nöt' -'Nö' -'Nutzungsbedingungen_' -'Nutze' -'Num' -'Now' -'Nove' -'Notwendig' -'Notfall' -'North' -'Normali' -'Nordi' -'Nomin' -'Nomad_' -'Nobody_' -'Nis' -'Nina_' -'Niko' -'Night_' -'Night' -'Niederla' -'Niedergang_' -'Nico' -'Nick' -'Nichte' -'Newsletter_' -'Nevada_' -'Nev' -'Neustart' -'Neuschwanstein_' -'Neuf' -'Neues_' -'Neuen' -'Neuausrichtung_' -'Nerv' -'Ner' -'Neo' -'Neighbo' -'Nego' -'Neb' -'Neapel_' -'Nazi' -'Naturwissenschaft' -'Naturpark' -'Naturk' -'Natural' -'Nationalst' -'Nationalpark_' -'Nationalpar' -'Namib' -'Nam' -'Nak' -'Nahrungs' -'Nahost_' -'Nahe' -'Nachw' -'Nachmittag_' -'Nachhaltige' -'Nachbarschaftspolitik_' -'Nachbarschaft_' -'Nachbarländer' -'Nachbar' -'NY' -'NIC' -'NG_' -'NGO_' -'NEC' -'ND_' -'NDE' -'Mütter_' -'Mün' -'Mönch' -'Möglicherweise_' -'Mythos_' -'MySpace_' -'Mutter' -'Muslim' -'Musical' -'Museums_' -'Museen_' -'Muse' -'Murdoch_' -'Mull' -'Mozart' -'Moz' -'Movielearn_' -'Movie_' -'Mosle' -'Moses_' -'Mosambik_' -'Mord' -'Mora' -'Montreal_' -'Monterrey_' -'Montai' -'Montag' -'Monster_' -'Mons' -'Molda' -'Modi_' -'Moderne_' -'Moder' -'Model' -'Moda' -'Mobiltelefon_' -'Mobilität_' -'Mittleren_' -'Mittelwe' -'Mitteleurop' -'Mitteil' -'Mitleid' -'Mitbe' -'Mitarbeiter' -'Mist' -'Missverständnis' -'Misstrauen_' -'Mission' -'Missb' -'Mira' -'Mir' -'Mins' -'Ministerrat_' -'Ministerpräsidenten_' -'Min_' -'Millions' -'Millia' -'Militära' -'Milchprodukt' -'Milch' -'Mikrof' -'Mike_' -'Metropoli' -'Metalle' -'Metal' -'Messa' -'Mercedes_' -'Menü' -'Menschenrechtsko' -'Mena' -'Memory_' -'Memori' -'Meli' -'Meister' -'Mein' -'Mehrzahl_' -'Mehrwertsteuer' -'Meeresf' -'Meere' -'Medina' -'Mechani' -'McK' -'McG' -'Maßstäbe' -'Maxi' -'Max_' -'Mauri' -'Mauer' -'Massenm' -'Massaker' -'Marí' -'Marta_' -'Mars_' -'Marr' -'Marqu' -'Marktzug' -'Marktt' -'Market' -'Mare' -'Mara' -'Malware_' -'Mali_' -'Maje' -'Mainstream' -'Main_' -'Mailand_' -'Mahlzeiten_' -'Magst_' -'Magazin_' -'Magazin' -'Machtver' -'Machts' -'MX' -'MT_' -'MS' -'MP4_' -'MOV_' -'MOS' -'MIN' -'MIDI_' -'MG' -'MENT' -'MB' -'MAT' -'MAN_' -'MAN' -'M4' -'Lüg' -'Lücke_' -'Lä' -'Luz' -'Lup' -'Luk' -'Lufthansa_' -'Ludwig_' -'Lucas_' -'Loyali' -'Louvre' -'Lohns' -'Logi' -'Lloyd' -'Livi' -'Liverpool_' -'Liter' -'Listen_' -'Lima_' -'Liese' -'Lieferu' -'Lied' -'Libye' -'Libyan' -'Leser_' -'Lesen_' -'Leo' -'Leno' -'Leiche' -'Lehrer' -'Lehman_' -'Legend' -'Leg' -'Lebensstandard_' -'Lebense' -'Lebanese_' -'Leave_' -'Laz' -'Lava' -'Laufzeit_' -'Laufen' -'Latvia' -'Latin' -'Lass' -'Larry_' -'Lannoye_' -'Langzeit' -'Landwirtschafts' -'Landw' -'Lampe' -'Lamaniten_' -'Lager' -'Lagen' -'Lady' -'Label_' -'Lab_' -'LR' -'LC_' -'Küsten_' -'Künst' -'Kün' -'Kühlschra' -'Kü' -'Körperschaft' -'Königreichs_' -'Köln' -'Käufer' -'Käuf' -'Kyr' -'Kyi' -'Kuwait_' -'Kurze' -'Kuro' -'Kurdish_' -'Kunde_' -'Kulissen_' -'Kuli' -'Kriterium_' -'Kriminelle' -'Krieg' -'Krem' -'Kreditvergabe_' -'Kreditv' -'Kreditnehmer' -'Kreditkarten' -'Kreditb' -'Kreditaufnahme_' -'Krebs_' -'Kreaturen' -'Kreat' -'Kraftwerk' -'Kow' -'Kosov' -'Korruptions' -'Kori' -'Koreans_' -'Koran' -'Kora' -'Kopiere' -'Kopfs' -'Koordin' -'Konzerne' -'Konvertier' -'Konvers' -'Kontextmenü' -'Konte' -'Konsultationen_' -'Konserv' -'Konferenzr' -'Konferenze' -'Kompetenz_' -'Kommuni' -'Kommissionsvorschlag_' -'Kommissionsmitglied' -'Kommissar' -'Kommentar_' -'Kolumbien_' -'Kolonie' -'Kollekti' -'Koizumi_' -'Kohlendioxid_' -'Koh' -'Kofinanzierung' -'Knowledge_' -'Kno' -'Kni' -'Klu' -'Klingon' -'Kleine' -'Klein_' -'Klausel' -'Klassifi' -'Klage_' -'Kinnock_' -'King' -'Kindes' -'Kinderar' -'Kin' -'Khomeini_' -'Kho' -'Khan' -'Khal' -'Keys' -'Kernel' -'Kenn' -'Kaukasus_' -'Katastrophenschutz' -'Katalo' -'Kaschmir' -'Karzai_' -'Kapitalmärkte_' -'Kapitalflu' -'Kanäle_' -'Kana' -'Kammer' -'Kamin' -'Kama' -'Kalk' -'Kaliningrad' -'Kale' -'Kade' -'Kaczyński_' -'Kabel' -'KW' -'KU' -'KMU_' -'KLM_' -'KING' -'KEI' -'KB' -'Jury' -'Juris' -'Jugendlicher_' -'Jugendherberg' -'Juden_' -'Journalist' -'Jord' -'Jong_' -'Jonas' -'Johnson' -'Johanne' -'Jog' -'Joan' -'Jim_' -'Jet' -'Jes' -'Jem_' -'Javi' -'Jaro' -'Jar' -'Japaner_' -'Jan' -'Jahrtausend' -'Jahrhunderte' -'Jahresz' -'Jag' -'Jacob_' -'JRE_' -'JE' -'Ivan' -'Iv' -'Ite' -'Italiens_' -'Italien' -'Issu' -'Iraker_' -'Investor' -'Internetverbindung_' -'Interneta' -'International' -'Intensität' -'Int' -'Insur' -'Insti' -'Inse' -'Innovation' -'Innenpoliti' -'Initiati' -'Inhalts' -'Inhaber_' -'Infrastructure_' -'Infos_' -'Informationss' -'Informationsa' -'Informati' -'Inflationsr' -'Infineon' -'Infektions' -'Infekt' -'Infe' -'Industriesta' -'Industrielle' -'Industrial_' -'Industri' -'Indoor_' -'Indo' -'Indiz' -'Individual' -'Indic' -'Increased_' -'Inco' -'Inclu' -'Inci' -'Improvi' -'Impres' -'Impfstoffe_' -'Immerhin_' -'Immer_' -'Ign' -'Ideologie_' -'Ideolog' -'Ideally_' -'Ideale' -'Ideal' -'Icon' -'Ibn_' -'IR_' -'INE' -'INCLUD' -'ILA' -'IFA' -'IES_' -'IEN' -'ICT' -'Hügel_' -'Hör' -'Höl' -'Höf' -'Hôtel_' -'Händler' -'Hypo' -'Hyg' -'Hydr' -'Hybri' -'Hv' -'Hungarian_' -'Hunderttausende' -'Hunderte' -'Hund' -'Hul' -'Hubschrauber_' -'Hua' -'House' -'Hotelzimmer_' -'Hot_' -'Horizonte' -'Horizont' -'Hond' -'Hom' -'Holy_' -'Hoff' -'Hochwasser' -'Hochgeschwindigkeits' -'His' -'Hinr' -'Hindus_' -'Himmel_' -'Hilfsmittel_' -'Hilfe' -'Highway' -'Hig' -'Heut' -'Het_' -'Herzog' -'Herv' -'Herrscher_' -'Herman' -'Herangehensweise_' -'Heilige' -'Hed' -'Hebr' -'Header' -'Hay' -'Hava' -'Haustür' -'Haushaltspolitik_' -'Haushaltsplan' -'Haushaltsmittel' -'Haushaltskon' -'Haushaltsausschusses_' -'Haushaltsaus' -'Hauptt' -'Hauptstr' -'Hauptp' -'Hauptau' -'Hatoyama_' -'Hass_' -'Harris_' -'Harr' -'Harm' -'Hariri' -'Hara' -'Happ' -'Hanse' -'Handl' -'Handelsa' -'Hande' -'Hamm' -'Halte' -'Hack' -'Haben' -'Haag' -'HR' -'HOT' -'HER' -'HEN' -'HC_' -'HAVEN_' -'H1' -'Gültigkeit_' -'Gül' -'Göttin' -'Gän' -'Gutes_' -'Gunsten_' -'Gues' -'Guardi' -'Guard' -'Größen_' -'Größ' -'Grundzüge' -'Grundwasser' -'Grundsätzlich' -'Grundsätzen_' -'Grundsatze' -'Grunds' -'Grundrechte' -'Grundl' -'Großka' -'Große' -'Grou' -'Griechen' -'Grey' -'Grenzwert' -'Gremien_' -'Gregori' -'Grego' -'Gramm' -'Graci' -'Gourmet_' -'Gothic_' -'Goth' -'Gos' -'Gore_' -'Good' -'Golfplätze' -'Golds' -'Gob' -'Glück' -'Gloucester' -'Glied' -'Glen' -'Gleichg' -'Gleichbehandlung_' -'Gle' -'Girl' -'Gio' -'Gibraltar_' -'Gia' -'Gewinne' -'Gewin' -'Gewerkschaft' -'Gewerbe' -'Gewalttaten_' -'Getränke' -'Getränk_' -'Gesundheitsz' -'Gesundheitsv' -'Gesundheitssystem' -'Gesundheitsm' -'Geste' -'Gesicht' -'Gesetz' -'Geschäftsv' -'Geschäftsbereich' -'Geschäften_' -'Geschw' -'Geschlecht' -'Geschirr' -'Geschichten_' -'Gesamth' -'Gesamtbe' -'Geräte' -'Germani' -'Germ' -'Gerichtsh' -'Georgi' -'George' -'Geogra' -'Genießen_' -'Generalsekret' -'Genauigkeit' -'Genau' -'Gen_' -'Gemäß' -'Gemeinschaftsin' -'Gelände_' -'Gele' -'Geldes_' -'Gei' -'Gehälter' -'Gehirn_' -'Gehirn' -'Geheimnis' -'Gehei' -'Gehalt_' -'Gegenden_' -'Gefüh' -'Gefängnisse' -'Gefahren' -'Gedächtnis_' -'Gedicht' -'Geburtstag' -'Geburten' -'Gebot' -'Gate_' -'Gastge' -'Garden' -'Garan' -'Ganz' -'Gamm' -'Gaming_' -'Gallery_' -'Galerie_' -'Gale' -'Gala' -'GW_' -'GUI' -'GUE_' -'GP_' -'GO_' -'GMT_' -'GMOs_' -'GL' -'GE_' -'Fülle' -'Fä' -'Fut' -'Fundament' -'Frühstücks' -'Frustration_' -'Frontier' -'Frist' -'Freud' -'Frequenz' -'Fremdenverkehr_' -'Fremdenfeindlichkeit_' -'Freizeita' -'Freilich' -'Freigabe_' -'Freedoms_' -'Fred_' -'Fred' -'François_' -'Franco' -'Fragment' -'Fragestunde_' -'Frage' -'Founde' -'Fotografie_' -'Fortschr' -'Fort_' -'Forschungss' -'Forschungsergebnisse_' -'Forschungsa' -'Formular' -'Formel_' -'Forest_' -'Fore' -'Football_' -'Fonta' -'Folglich_' -'Focus' -'Flut_' -'Flusse' -'Flus' -'Flie' -'Flemi' -'Flasche' -'Flam' -'Flagg' -'Fixed_' -'Fit' -'Fischereiabkommen_' -'Fische_' -'Firmware_' -'Firm' -'Firew' -'Finger_' -'Finanzwesen_' -'Finanzst' -'Finanzp' -'Finanzmittel_' -'Finanzmi' -'Finanzinstrument' -'Finanzb' -'Fina' -'Filme_' -'Filme' -'Fig' -'Fic' -'Feuerwe' -'Feststellung' -'Festiv' -'Feste' -'Fertigkeit' -'Ferna' -'Ferienhäuser_' -'Fed' -'Fatah_' -'Fans_' -'Familienzimmer_' -'Falle' -'Faktum_' -'Fahrzeug' -'Fahrt_' -'Fahrplan_' -'Fahren_' -'Fact' -'Faci' -'Fachk' -'Fabrik' -'Fab' -'FRE' -'FAQ' -'Extreme_' -'Extras_' -'Exten' -'Exporteure' -'Experts_' -'Experte' -'Experimente' -'Exper' -'Exo' -'Exist' -'Exhibit' -'Exekutiv' -'Except' -'Ew' -'Evidence_' -'Everest_' -'Even' -'Evangeli' -'Eurojust_' -'Eurocopter_' -'Euroc' -'Eurobonds_' -'EuroM' -'Eurasi' -'Eur' -'Euph' -'Establishment_' -'Essential_' -'Esse' -'Especially_' -'Erziehung_' -'Erzeugung' -'Erzeugnisse_' -'Erwähnung_' -'Erwägungen_' -'Erweiterungen_' -'Erwachsenen_' -'Erwach' -'Erträge_' -'Ertr' -'Erstelle' -'Erste_' -'Erstau' -'Ersparnissen_' -'Erscheinungsbild_' -'Ersatz' -'Err' -'Ernähr' -'Ernst_' -'Erleichterung_' -'Erleb' -'Erheb' -'Erha' -'Erh' -'Ergänz' -'Erg' -'Erfolgsgeschichte' -'Erfolg' -'Erbr' -'Eras' -'Equip' -'Equ' -'Entwicklungsziele' -'Entwicklungs_' -'Entstehung' -'Entspann' -'Entscheidungsprozess' -'Entführung' -'Entf' -'Enter' -'Enr' -'Englischen_' -'Englische' -'Energietechnologie' -'Energier' -'Energiepolitik_' -'Endp' -'Endl' -'Ende' -'Employ' -'Empfang' -'Emotion' -'Emirate' -'Embr' -'Email_' -'Elysées_' -'Ell' -'Elizabeth_' -'Elis' -'Elend_' -'Elemente' -'Eleganz_' -'Elefanten' -'Elde' -'Eisenbahnver' -'Einzelpersonen_' -'Einwi' -'Einwanderungspolitik_' -'Einwanderungs' -'Eint' -'Einsp' -'Einse' -'Einrei' -'Einmischung' -'Einmarsch' -'Einm' -'Einkommens_' -'Einkaufszentr' -'Einflüsse' -'Einfach_' -'Eindämmung' -'Einblick_' -'Einbindung_' -'Eid' -'Editor_' -'Edit_' -'Echtzeit_' -'Early_' -'EUR' -'ESM' -'ENI' -'EMAS_' -'Düsseldorf_' -'Dür' -'Dü' -'Dynast' -'Dynamik' -'Dutzende_' -'Dus' -'Durchschnitts' -'Durchführ' -'Durchb' -'Duomo_' -'Duff_' -'Dub' -'Ds_' -'Drücke' -'Drum' -'Drucker' -'Droh' -'Drittl' -'Dritte' -'Dringlichkeits' -'Dri' -'Dream' -'Drago' -'Dr' -'Doyle_' -'Downloads' -'Down' -'Dornik_' -'Dorn_' -'Dorf' -'Doppelzimmer_' -'Doo' -'Dominikan' -'Dolomit' -'Dolmetsch' -'Dokumentation_' -'Divers' -'DivX_' -'Div' -'Distributoren_' -'Disku' -'Diskriminierung' -'Directory_' -'Director' -'Directi' -'Diplomaten_' -'Diktatur_' -'Different_' -'Dienststelle' -'Dienstleistungssektor' -'Dienstleist' -'Diensta' -'Dienst' -'Dictionary_' -'Dichte' -'Dich' -'Dialogs_' -'Diabetes_' -'Deutsche' -'Deut' -'Detail_' -'Designs' -'Designer_' -'Desi' -'Deregul' -'Derartige_' -'Denkweise' -'Denis' -'Demokratisierung_' -'Demogra' -'Demagog' -'Delu' -'Delta_' -'Delo' -'Delegation' -'Dele' -'Dela' -'Deine' -'Deckmantel_' -'Death_' -'Daw' -'Datenschutz_' -'Datenbl' -'Date' -'Dasselbe_' -'Daseins' -'Das' -'Darwi' -'Darlehen' -'Darauf_' -'Dara' -'Dampfb' -'Damas' -'Dalma' -'Dai' -'Dafürhalten_' -'DVDs_' -'DK' -'DJ_' -'DI_' -'DIC' -'DES' -'DEN_' -'DC' -'DAX_' -'Cycl' -'Cyber' -'Curt' -'Cul' -'Cott' -'Cord' -'Copy' -'Cop' -'Cooper' -'Cool' -'Controller_' -'Conti' -'Constant' -'Conservati' -'Congress' -'Confedera' -'Condo' -'Conditions_' -'Conci' -'Concern' -'Computern_' -'Compr' -'Compani' -'Communication' -'Commen' -'Comi' -'Comfort' -'Combi' -'Colomb' -'Collect' -'Cohe' -'Coelho_' -'Coch' -'Cob' -'Clu' -'Close_' -'Clip' -'Clif' -'Cleverl' -'Cleaning_' -'Clean_' -'Classic' -'Clas' -'Clark' -'Circle_' -'Circ' -'Cind' -'Chrom' -'Christus_' -'Christopher_' -'Christo' -'Christie_' -'Christiani' -'Christen_' -'Christdemokraten_' -'Chr_' -'Chev' -'Cherno' -'Chechen' -'Charle' -'Chapel_' -'Chap' -'Champ' -'Ces' -'Center' -'Cav' -'Caucas' -'Castil' -'Cassi' -'Casio' -'Casino' -'Cash' -'Carrie_' -'Carp' -'Carlo' -'Carl_' -'Caribbean_' -'Care' -'Cardi' -'Capt' -'Canc' -'Canari' -'Canal' -'Canadian_' -'Campingpl' -'Camera' -'Cambodia_' -'Calendar_' -'Cairo' -'Caesar' -'COS' -'CONT' -'CHF_' -'CHE' -'CGI_' -'CET_' -'Bürgerkrieg_' -'Bürgerkrieg' -'Bürgerbe' -'Bündnis_' -'Bünd' -'Bücher_' -'Böge_' -'Buy' -'Busse' -'Busc' -'Bundestag_' -'Bundesstaaten_' -'Bundesp' -'Bum' -'Buddh' -'Buchungs' -'Buchst' -'Buchführung' -'Bucher_' -'Bry' -'Brunnen_' -'Bruc' -'Brow' -'Brooklyn_' -'Bronze' -'Broc' -'Britis' -'Brigade' -'Brian_' -'Brew' -'Bretton_' -'Bret' -'Brenner' -'Bremen_' -'Breitband' -'Brei' -'Bree' -'Brav' -'Braun' -'Branc' -'Box' -'Boul' -'Botschafter' -'Boston' -'Bosnien_' -'Bosnia_' -'Boots' -'Boot' -'Bond' -'Bombardier' -'Boeing' -'Boe' -'Bodensch' -'Bode' -'Boar' -'Blog' -'Blingee_' -'Blick' -'Blaž_' -'Birma' -'Biot' -'Biog' -'Binnenm' -'Bindungen_' -'Bin_' -'Billig' -'Billi' -'Bildern_' -'Bhutan_' -'Bezirks' -'Bey' -'Bewä' -'Bewu' -'Bewertungs' -'Bewert' -'Bewerber' -'Better_' -'Betrü' -'Beträge' -'Betreuer' -'Betracht' -'Besuchern_' -'Bestra' -'Besten' -'Bestell' -'Besonderheiten_' -'Besetzung_' -'Beschwerde_' -'Beschlussfassung' -'Beschlusse' -'Beschleunigung' -'Beschaff' -'Berufsbildung' -'Berufe' -'Bert' -'Berl' -'Berichterstattung' -'Berb' -'Beobachtungsst' -'Benz_' -'Belohnung_' -'Beliebt' -'Belgrade_' -'Belastung_' -'Belast' -'Belarus' -'Beitrittsl' -'Beitrittskandidaten_' -'Beis' -'Being_' -'Beine' -'Bein' -'Beihilfe' -'Behau' -'Behandlungs' -'Begrenzung_' -'Bege' -'Beförderungs' -'Befu' -'Befr' -'Bedürf' -'Bedienung_' -'Bede' -'Bec' -'Beauf' -'Beatri' -'Beachten_' -'Bayer' -'Bavarian_' -'Baute' -'Baustein' -'Baum_' -'Batt' -'Bath' -'Basket' -'Basis' -'Base' -'Barry_' -'Barro' -'Baro' -'Bargeld' -'Barbe' -'Bann' -'Banker' -'Bankensystem' -'Bande' -'Banc' -'Ballo' -'Bald_' -'Bajor_' -'Bahnstation_' -'Bah' -'Baghdad' -'Baden_' -'Bachelo' -'Bach_' -'Babys' -'BOJ_' -'BN' -'BJ' -'BF' -'BERKELEY_' -'Azu' -'Ax' -'Aw' -'Avatar' -'Außenbe' -'Autoren_' -'Autonomiebehörde_' -'Autobahn_' -'Ausübung' -'Auswärtige' -'Ausweg_' -'Auswe' -'Austausch' -'Aust' -'Ausspr' -'Ausschü' -'Ausschuß_' -'Ausscheiden' -'Ausscha' -'Auskunft' -'Ausgrenzung_' -'Ausgew' -'Ausgehend_' -'Ausgabe' -'Ausflu' -'Ausd' -'Aurora_' -'Aug' -'Aufträge_' -'Auft' -'Aufstände' -'Aufstand_' -'Aufse' -'Aufge' -'Auffü' -'Auffassungen_' -'Attraktionen_' -'Attent' -'Atta' -'Atlantis' -'Atla' -'Asylsuchende' -'Asylant' -'Assist' -'Asset' -'Asians_' -'Arzt' -'Aru' -'Articles_' -'Arsenal_' -'Ars' -'Arou' -'Armenian_' -'Ark' -'Argumen' -'Argentine' -'Argen' -'Arena' -'Are' -'Arbeitszeit_' -'Arbeitsver' -'Arbeitsrecht' -'Arbeitspro' -'Arbeitsp' -'Arbeitsgruppe_' -'Arbeitsg' -'Arbeitsbe' -'Arbeit' -'Apr' -'Appro' -'Anzeigen' -'Anzeige' -'Anz' -'Anyone_' -'Anwesenheit_' -'Anwendungsbereich_' -'Anwender_' -'Anwa' -'Antrags' -'Anteils' -'Anteile_' -'Ansä' -'Ansprüchen_' -'Ansprech' -'Anson' -'Anschrift' -'Anschließend_' -'Ansatzes_' -'Anreise' -'Anregungen_' -'Anpassungen_' -'Annex' -'Annahmen_' -'Anmerkungen_' -'Anmerkung_' -'Anleihe' -'Ankunft_' -'Anku' -'Anklage_' -'Ani' -'Anhörung_' -'Angr' -'Angeles_' -'Angel' -'Angebots' -'Angeb' -'Ane' -'Andria_' -'Andorra' -'Anden' -'Andalusien' -'Andalusia' -'Anda' -'Anbau_' -'Anato' -'Analys' -'Amu' -'Amo' -'Ambitionen_' -'Amazon_' -'Ama' -'Aly' -'Alum' -'Altern' -'Alta_' -'Alr' -'Alps_' -'Alpi' -'Alltags' -'Alkohol' -'Algor' -'Algerie' -'Algar' -'Alegr' -'Alber' -'Albanian' -'Alb' -'Aktuell' -'Aktualisierung_' -'Aktivisten_' -'Aktionär' -'Aktionsprogramm_' -'Akk' -'Aki' -'Aka' -'Airp' -'Airconditioning_' -'Aid_' -'Agrars' -'Agenturen_' -'Agent_' -'Afri' -'Advanced_' -'Administrat' -'Ade' -'Addis_' -'Ada' -'Active_' -'Acid' -'Acht' -'Achsen' -'Accord' -'Abzug_' -'Abwicklung_' -'Abweichung' -'Abtreibung' -'Abtei' -'Abstimmungs' -'Absti' -'Absolvent' -'Abso' -'Absen' -'Abschl' -'Abl' -'Abhängig' -'Abh' -'Abgeordnete' -'Abfälle_' -'Abdullah' -'Abbe' -'Aa' -'ATM' -'AR_' -'AP_' -'AO' -'AMS' -'AMR_' -'ALE_' -'AD_' -'ADE' -'ACI' -'ACCE' -'ABS' -'A2' -'=_' -';&' -'90er_' -'83' -'750' -'70er_' -'681' -'67' -'63' -'5th_' -'520' -'52' -'4th_' -'45' -'43' -'3G_' -'3G' -'39' -'370' -'37' -'270_' -'220_' -'21s' -'202' -'201' -'199' -'1972_' -'1961_' -'1960s_' -'1960' -'1959_' -'1950er_' -'1946_' -'1939_' -'1936_' -'1907_' -'171' -'170' -'16th_' -'145_' -'142' -'127_' -'124_' -'121_' -'117' -'116' -'105_' -'102' -'101' -'0er_' -'07' -'020' -'007' -'/+_' -'/ ' -'......' -'....' -'.'_' -'->' -',..._' -',- ' -',,_' -', (_' -', $_' -'++' -'* ' -'):_' -'), ' -')) (' -'))' -'() ._' -'': _' -'')' -'''.' -''''_' -'%\\' -'$ ' -'">- _' -'"...' -'". _' -'" ._' -'" -' -'!! _' -'!! !' -' …' -' ”' -' ’_' -' ­' -' £_' -' [...]' -' = {_' -' = ' -' ;' -' -> _' -' ***' -' ). _' -' (“_' -' (.' -' ('' -' &#_' -' !!' -'−' -'ي' -'ט' -'ג' -'ь' -'щ' -'ц' -'σ' -'ş' -'œ' -'ě' -'ę' -'ā' -'õ' -'ñ' -'¿' -'º' -'~' -'$' -'™' -'†' -'–' -'ن' -'ل' -'ف' -'ر' -'ר' -'נ' -'Ж' -'Д' -'υ' -'ν' -'λ' -'ś' -'ń' -'ù' -'ì' -'Ñ' -'É' -'Ã' -'Á' -'§' -'–' -'&' -'ー' -'‚' -'م' -'ק' -'ד' -'Я' -'П' -'О' -'Л' -'Е' -'А' -'π' -'κ' -'θ' -'β' -'ū' -'Ś' -'ō' -'ć' -'æ' -'Ê' -'Â' -'¼' -'·' -'¶' -'´' -'¥' -'`' -'@' -'#' -'' -'년' -'語' -'简' -'本' -'日' -'文' -'年' -'中' -'•' -'ṳ' -'ศ' -'พ' -'ा' -'र' -'ى' -'ه' -'ص' -'ت' -'ب' -'פ' -'ס' -'ן' -'ו' -'ֿ' -'В' -'ω' -'χ' -'δ' -'Ω' -'̤' -'ư' -'ů' -'ř' -'ľ' -'ė' -'ĕ' -'ą' -'û' -'À' -'½' -'¹' -'­' -'¤' -'¡' -'’' -'\' -'：' -'' -'fi' -'黵' -'黃' -'鰀' -'鋘' -'鋓' -'遝' -'蒸' -'致' -'美' -'网' -'紙' -'熨' -'斗' -'応' -'女' -'味' -'友' -'信' -'介' -'丨' -'一' -'ャ' -'バ' -'チ' -'ジ' -'カ' -'ん' -'ら' -'め' -'●' -'▼' -'→' -'※' -'ớ' -'ọ' -'ị' -'ẽ' -'ẻ' -'ấ' -'ी' -'ि' -'य' -'ब' -'त' -'छ' -'आ' -'ِ' -'ك' -'غ' -'ع' -'د' -'ج' -'إ' -'،' -'צ' -'ל' -'ה' -'Қ' -'Ғ' -'Э' -'Ш' -'Ц' -'Х' -'Р' -'М' -'φ' -'ζ' -'γ' -'Χ' -'Τ' -'Ι' -'Ε' -'̯' -'̆' -'ː' -'ˈ' -'ɾ' -'ɛ' -'ɐ' -'ſ' -'ű' -'ŭ' -'ő' -'Ő' -'ŏ' -'ň' -'İ' -'ī' -'đ' -'Đ' -'ă' -'ý' -'ã' -'à' -'Ô' -'Ó' -'È' -'Å' -'¾' -'µ' -'³' -'°' -'¬' -'¢' -'' -'™' -'—' -'“' -'' -'^' -'—' -'²' -'£' -'<' diff --git a/lingvo/core/ops/testdata/test_ngrams.txt b/lingvo/core/ops/testdata/test_ngrams.txt deleted file mode 100644 index dc14e9aa..00000000 --- a/lingvo/core/ops/testdata/test_ngrams.txt +++ /dev/null @@ -1,255 +0,0 @@ - 0 - 0 - 0 -{ 32402 -m 32010 -, 31793 -? 31481 -" 31002 -s 30592 -r 28832 -) 28316 -n 27667 -b 26399 -t 26268 -p 25947 -w 25748 -a 24249 -l 23717 -u 21727 -: 21703 -f 20237 -} 20153 -c 18509 -. 16781 -o 15746 -y 14335 -~ 14253 -_ 13262 -q 12548 -> 11036 - 10859 -g 9676 -h 9605 -i 9595 -! 9097 -d 8964 -* 8786 -x 8786 -< 8146 -z 5961 -; 5605 -k 5411 -j 3322 -& 3026 -( 2351 -- 2061 -v 483 -/ 293 -e 200 -` 139 -ed t 32707 -ag 32644 -su 32457 -t th 32396 -es 32072 -un 32012 -os 32003 -gh 31975 -si 31641 -mi 31435 -n t 31226 -op 31038 -le 30862 -wi 30709 -ex 30688 -an 30443 -ter 30325 -mo 30234 -re 29997 -ci 29863 -ie 29788 -to 29761 -id 29697 -rt 29497 -s o 29452 -e a 29440 -de 29423 -one 28763 -ai 28386 -n th 28352 -all 28078 -her 27964 -we 27827 -ee 27687 -ow 27617 -or 27517 -e i 27344 -ss 26988 -ce 26785 -th 26722 -nc 26552 -are 26482 -ve 26415 -eri 26381 -that 26104 -e o 25761 -d a 25713 -am 25334 -ise 25140 -ei 25004 -fo 25004 -ha 24996 -ca 24870 -res 24718 -t t 24698 -ev 24464 -il 24420 -wo 24382 -mp 23652 -ate 23227 -ho 23105 -ine 23024 -ted 23007 -ere 22940 -sh 22918 -ati 22902 -el 22814 -for 22811 -no 22645 -at 22400 -wh 22372 -een 22348 -st 22331 -ct 22329 -he 22229 -la 22150 -e c 22020 -pr 21855 -ot 21724 -ri 21347 -ur 21142 -er 21127 -ive 20825 -so 20732 -in 20702 -ill 20601 -est 20545 -ia 20531 -of 20308 -ts 20244 -ul 20231 -s a 19938 -ly 19877 -en 19810 -con 19726 -bu 19446 -ng 19282 -wa 19158 -ld 18999 -se 18943 -lo 18521 -com 18474 -bl 18425 -ig 18400 -io 18293 -ar 18079 -ic 17858 -ad 17331 -the 17207 -tio 16749 -ne 16721 -vi 16660 -ge 16491 -nt 16147 -ear 16143 -per 16007 -hat 15856 -ra 15502 -ba 15135 -li 14949 -ht 14835 -ion 14689 -al 14674 -ut 14585 -ver 14335 -ov 14333 -ers 13880 -'s 13667 -men 13639 -n a 13603 -ir 13250 -ro 13227 -igh 12875 -ti 12840 -he s 12509 -pl 12297 -iv 12209 -pro 12015 -em 11860 -bo 11836 -us 11771 -e f 11594 -ent 11453 -di 11395 -eve 10976 -d th 10750 -pa 10471 -and 10461 -on 10017 -it 9914 -d t 9875 -qu 9754 -atio 9701 -om 9676 -ll 9662 -te 9538 -ch 9059 -ni 8679 -do 8500 -me 8462 -f t 8088 -co 7770 -rs 7570 -et 7550 -s t 7394 -ma 7284 -ke 6972 -be 6912 -ol 6465 -int 6429 -r t 6249 -hi 6107 -ed 6033 -is 6019 -tha 5714 -nd 5685 -as 5631 -fi 5479 -e th 4577 -s i 4439 -mm 4358 -pe 3914 -t a 3849 -po 3563 -e b 3506 -tion 3371 -na 3247 -e s 2648 -s s 2547 -ta 2341 -ec 2297 -ty 2202 -oi 1988 -ay 1372 -ea 1217 -ght 1002 -s th 924 -sa 818 -ou 794 -e t 792 -ing 646 -tr 578 -ns 353 -e p 124 -ac 71 diff --git a/lingvo/core/ops/testdata/test_vocab.txt b/lingvo/core/ops/testdata/test_vocab.txt deleted file mode 100644 index 6e760b3e..00000000 --- a/lingvo/core/ops/testdata/test_vocab.txt +++ /dev/null @@ -1,14 +0,0 @@ - 0 - 1 - 2 - 3 -the 4 -a 5 -b 6 -c 7 -d 8 -e 9 -øut 10 -über 11 -♣ 12 -愤青 13 diff --git a/lingvo/core/ops/testdata/wmt/sm18.txt b/lingvo/core/ops/testdata/wmt/sm18.txt deleted file mode 100644 index 373b8909..00000000 --- a/lingvo/core/ops/testdata/wmt/sm18.txt +++ /dev/null @@ -1,3003 +0,0 @@ - $ de secours pour la santé Plan de sauvetage de 325 M $ destiné au système de santé de la Tasmanie -" La colline de n' est pas stratégique " , a déclaré le gouverneur du Nord-Kivu , Julien , aux AFP . " Celle de n' est pas très stratégique " , a indiqué a l' AFP le gouverneur du Nord-Kivu , Julien . -" Le soutien de l' opposition a été pris en charge par le Front islamique du Turkestan oriental ( ) basé en Asie centrale et occidentale " , a déclaré le chef des services de sécurité chinois à une chaîne de télévision de Hong Kong , selon une vidéo diffusée en ligne jeudi soir . " Le soutien en coulisses à l' attentat est venu du Mouvement islamique du Turkestan oriental basé en Asie centrale et occidentale " , a déclaré le patron des organes de sécurité chinois à une chaîne de télévision de Hong Kong , selon une vidéo mise en ligne jeudi soir sur un site internet . -" n' a pas été accessible pendant plusieurs heures ce soir en raison d' une erreur interne survenue lors d' une mise à jour programmée " , a déclaré l' agence d' espionnage dans une déclaration envoyée par courriel . « est resté inaccessible pendant quelques heures ce soir en raison d' une erreur interne qui a eu lieu pendant une mise à jour prévue » , a indiqué l' agence d' espionnage dans une déclaration par email . -" Un biométrique empêche la situation dans laquelle une au nom d' une autre " affirme Cyrille d' Accenture . " Une biométrique évite le phénomène du copain qui pointe pour un autre " , justifie Cyrille , d' Accenture . -" Un événement marquant a eu lieu aujourd ' hui : la construction a commencé dans la section bulgare du gazoduc South Stream , le projet le plus important et le plus économique de l' Europe " , a déclaré le président de Gazprom , M. Vincent , dans une déclaration jeudi . « Un évènement historique a eu lieu aujourd' hui : La construction a commencé dans la partie bulgare du gazoduc South Stream , le projet à grande échelle le plus important d' Europe » , a expliqué Alexei Miller , le directeur général de Gazprom , dans une déclaration jeudi . -" Un grand nombre de militants craignent que la règle allemande n' encourage les parents à prendre des décisions rapides et à donner à l' enfant un statut " indéterminé " , " a-t-elle déclaré " . « De nombreux activistes s' inquiètent que la loi allemande encourage les parents à prendre des décisions rapides et donnent à leur enfant un sexe " indéterminé " » , a-t-elle déclaré . -" Les adultes devraient pouvoir prendre leurs propres décisions sur le genre légal " , a-t-elle dit . « Les adultes devraient pouvoir prendre leurs propres décisions à propos d' un genre légal » , a-t-elle ajouté . -" Alex Turner a été diagnostiqué et n' est malheureusement pas capable de faire " . « Alex Turner souffre d' une et regrette de ne pouvoir se produire en concert . » -" Tous les soldats blessés reçoivent les soins appropriés en vue du retour à la vie civile , et aucun n' a été libéré avant qu' il ne soit prêt " , a-t-il déclaré . Tous les soldats blessés reçoivent les soins appropriés en vue de leur retour à la vie civile et aucun d' entre eux n' est libéré avant d' être prêt , a-t-il affirmé . -" Une enquête épidémiologique a été entreprise pour identifier et traiter tous les individus susceptibles d' entrer en contact avec le chaton entre le 30 et le 28 octobre inclusivement " , a déclaré les ministres de la Santé et de l' Agriculture . " Une enquête épidémiologique a été engagée afin d' identifier et prendre en charge les personnes qui auraient pu entrer en contact avec ce chaton entre le 8 octobre et le 28 octobre inclus , " précise les ministères de la santé et de l' agriculture . -Et parce que si ce qui se passe en Irak n' est pas traité , il se , de même que ce qui se passe en Syrie . Et aussi parce que si ce qui se passe en Irak n' est pas réglé , cela va s' étendre , et ce qui se passe en Syrie va s' étendre aussi . -" Et le courrier exprimant le soutien que j' ai reçu depuis hier n' est pas toujours à partir de la banlieue " , dit-il avec le plus grand plaisir . " Et les mails de soutien que je reçois depuis hier n' émanent pas tous de banlieue " se . -" Un autre groupe de les a trouvé dans le piège creux , mais en bonne santé et a reçu un message à l' une des équipes de sauvetage travaillant dans le réseau des tunnels " , a expliqué le journal local , Le . " C' est un autre groupe de qui , après les avoir trouvés épuisés mais en bonne santé dans la cavité , a transmis l' information à l' une des équipes de secours engagées dans le réseau " explique Le . -" Au milieu du XXe siècle , on l' appelait " urgence psychiatrique " , dit M. . « Au milieu de XXe siècle , on appelait cela une urgence psychiatrique » , a indiqué . -" Parce que l' incinération est un développement récent " , dit , présidente de la Société et auteur du livre the Death . " Parce que la crémation est un fait nouveau " , répond , présidente de la Société de et auteur du livre la mort . -" Boeing propose un vol de longue distance dans des sièges plus restreints que les régionaux " , a déclaré John Leahy , Chief Operating Officer Customers d' Airbus . « Boeing propose des vols long-courriers dans des sièges plus étroits que dans les avions turbopropulseurs régionaux » , a indiqué John Leahy , directeur commercial d' Airbus . -" Le fait est que , à mesure que les coûts augmentent , les personnes qui se trouvent dans la tranche inférieure ( à la fin de l' échelle des revenus ) seront en mesure de tirer profit de leur revenu " , a déclaré Song , économiste à , une banque malaisienne . « Le bilan est qu' avec la hausse des coûts , les personnes au milieu ou au bas [ de l' échelle des salaires ] cherchent à compléter leurs revenus comme ils peuvent » , déclare Song , économiste chez , une banque malaisienne . -" Mais cette démarche pourrait inciter certains investisseurs étrangers à acheter des biens à Londres ou aux propriétaires actuels qui hésitent à vendre " , a-t-il ajouté . « Mais ce changement pourrait rendre les investisseurs étrangers réticents à l' idée d' acheter des propriétés dans Londres ou les propriétaires actuels réticents à l' idée de vendre » , a-t-il ajouté . -" À la fin de l' année financière , nous aurons sept nouveaux pharmaciens et trois du congé de maternité . D' ici la fin de l' année financière , nous aurons sept nouveaux pharmaciens et trois retours de maternité . -" À l' heure actuelle , la CASA n' a pas de réglementation spécifique régissant l' utilisation des appareils électroniques dans les avions " , a-t-il déclaré . « La CASA ne dispose pas de réglementations spécifiques régissant l' utilisation des appareils électroniques en avion » , a-t-elle déclaré . -" Les campagnes ne dépensent pas le montant de l' argent dans les sondages et les groupes de discussion , à moins qu' ils ne s' intéressent sérieusement à quelque chose " , a déclaré Martin sur . « Les campagnes ne consacrent pas beaucoup d' argent aux sondages et aux groupes de discussion à moins qu' elles n' envisagent sérieusement de prendre des mesures » , a déclaré Martin sur . -" Les téléphones cellulaires , c' est vraiment une question , non seulement parce que cela pourrait créer des interférences avec les appareils de communication , mais nous savons , d' après la FCC , qu' elle pourrait interférer avec les pylônes de téléphone lorsqu' ils sont dans l' air " , dit M. . « Les téléphones portables sont véritablement un problème , non seulement parce qu' ils pourraient éventuellement créer des interférences avec les instruments de navigation , mais parce que nous savons , d' après la FCC , qu' ils pourraient perturber les de téléphonie mobile s' ils sont utilisés à bord » , a déclaré . -" Les inquiétudes à propos de Big Brother et de ces types de problèmes étaient un problème majeur " , a déclaré Khan , qui dirige la gestion stratégique et des performances au sein du département des Transports du Nevada . « Les inquiétudes au sujet de Big Brother et ce genre de choses sont un gros problème » , a déclaré Khan , qui dirige la gestion stratégique de la performance au département des Transports du Nevada . -" Les clients des biens immobiliers qu' ils possèdent déjà , et les biens personnels ne font pas augmenter la dette des ménages " , affirme-t-elle . « Les clients des biens qu' ils possèdent déjà , et le fait de leurs biens personnels n' augmente pas l' endettement des ménages » , dit-elle . -" Selon l' état du plan , nous pourrions approuver l' utilisation à grande échelle des appareils électroniques plus tôt " , a déclaré la FAA dans une déclaration . « En fonction du plan , nous pourrions approuver une utilisation élargie des appareils électroniques très prochainement » , a indiqué la FAA dans une déclaration . -" Malgré sa perte dans la tentative d' acquisition de l' aux enchères , Google a violé et continue d' enfreindre " , a déclaré la commission . « Bien qu' ayant échoué dans sa tentative d' acquérir les brevets en cause au cours des enchères , Google a violé et continue à violer lesdits brevets » , ont indiqué les conclusions du procès . -" Les empreintes digitales sont stockées dans un serveur central ; l' identité des individus est stockée dans une deuxième base de données . " Les empreintes digitales sont stockées sur un premier serveur ; les identités des personnes sont dans une seconde base : la corrélation entre les deux numéros est cryptée et stockée dans un boîtier hautement sécurisé , qui se bloque si on tente de le déplacer " , énumère Philippe Robin , de Thales . -Les " doubles " ont fait l' éloge de Gates , en dépit de leur " relation personnelle étroite " , avant de décider en fin de compte de leur déplacement , alors que les données fournies par Craig le billet au " ne améliorerait pas sensiblement les chances d' Obama " . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; . « Double Down » indique que a mené la charge pour remplacer Biden , malgré leurs « liens personnels étroits » , avant de finalement se décider contre le changement lorsque les chiffres ont montré qu' avoir Clinton sur la liste présidentielle « n' améliorait pas nettement les chances de réussite d' Obama » . -" L' Egypte aujourd' hui est un État civil , constitutionnel , national et moderne " , a déclaré Mladic , avec un bras bleu et une cravate verte , a dit aux juges de l' atelier de où il avait prêté serment . « L' Égypte est aujourd' hui un État civil , national , constitutionnel et moderne » , a déclaré Mohamed Morsi , en costume bleu et cravate rouge , s' adressant aux juges présents dans la salle ornée de boiseries où il a prêté serment . -" Dix-huit pouces pour la largeur des sièges seraient très élevés pour les passagers , mais la réalité , d' un point de vue de l' objectif opérationnel d' Airbus , est le résultat de la menace de 777 " , a déclaré le spécialiste de l' intérieur , Mary Kirby , fondateur et directeur du " Girl Network " . « Une largeur de siège de 18 pouces serait parfaite pour les passagers , mais la réalité est que , du point de vue commercial , la proposition d' Airbus est motivée par la menace du 777 » , a déclaré l' experte en intérieurs de cabines Mary Kirby , fondatrice et rédactrice en chef de Girl Network . -" Même si la réorganisation de l' a une justification fondamentale , nous ne pouvons y arriver sans un plan acceptable de transfert de patients à d' autres hôpitaux " . Même si la restructuration de l' a sa légitimité sur le fond , nous ne pouvons pas l' admettre sans un schéma acceptable de report vers les autres hôpitaux . -" Tout le monde est " , a déclaré Andrea , l' un des participants . Tout le monde est , raconte une des participantes , Andrea . -" Tout le monde a un temps fou " , ajoute . On s' amuse comme des fous , ajoute . -" Chacun sera bientôt identifiable , partout et à tout moment " , a déclaré David , directeur de la protection des consommateurs de pour le Centre ( Center for Democracy and Technology ) de Washington . " Tout le monde pourra bientôt être identifié , n' importe où et n' importe quand " , s' Justin , responsable de la vie privée des consommateurs au sein du CDT ( Center for Democracy and Technology ) , à Washington . -" Tout a été mis en place afin que le puisse être déployé comme prévu à la fin 2013 " , explique . " Tout est mis en œuvre pour que le puisse faire son déploiement prévu en fin d' année 2013 " , explique cependant . -" Les finances sont et nous espérons voir le 50e anniversaire de l' , un jalon que nous en sept ans " , a déclaré l' administrateur . " Les finances sont à flot et nous espérons durer jusqu' au 50e anniversaire de l' , une étape que nous dans sept ans " , a commenté l' administrateur . -" Cinq personnes qui étaient en contact avec le chaton ont déjà été identifiées " et ont reçu un traitement préventif . " Cinq personnes ayant été en contact avec le chaton ont déjà été identifiées " et ont reçu un traitement préventif . -" Pendant une dizaine d' années , ils ont fait des efforts pour sauver les familles du mauvais esprit de la vie en attendant une heure et demie , sans rien faire , immédiatement après leur remise des cendres " , dit , . " Depuis une dizaine d' années , ils s' efforcent d' éviter aux familles la violence d' une attente , sans rien faire , pendant une heure et demie , immédiatement suivie de la remise des cendres " , observe M. . -" Par exemple , ceux qui travaillent avec du ciment ont des blessures à leurs doigts qui rendent leurs empreintes illisibles " , explique Luc Philippe , directeur technique de l' identification à Thales Communications & Security . " Par exemple , certaines personnes qui travaillent le béton ont les doigts très abîmés et leurs empreintes sont illisibles " , constate Philippe Robin , directeur technique du domaine identitaire chez Thales Communications & Security . -" Pour moi , cela n' indique pas un changement de tendance " , a-t-il noté . Pour moi , ce n' est pas l' indication d' un changement de tendance , a-t-il observé . -" Pendant plusieurs mois , j' ai demandé s' il y avait une façon de rester , et ils ont répondu " non , ajoute-t-il . J' ai demandé pendant des mois s' il y avait un moyen pour que je puisse rester , et ils ont dit non . -" Pour la grande majorité des personnes qui ont une assurance santé , vous pouvez le garder , dit-il dans un discours à Boston . « La grande majorité de personnes détenant une assurance santé qui leur convient peuvent la garder » , a-t-il déclaré lors d' un discours à Boston . -" Pour nous , le nombre de salariés est toujours précaire , car il y a un milieu de travail où beaucoup de jeunes femmes sont employées , c' est pourquoi on peut toujours compter sur trois personnes en congé de maternité quand les choses vont bien " , a-t-elle ajouté . Pour nous [ le nombre d' effectifs ] est toujours précaire , car on a un milieu qui est jeune et on emploie beaucoup de jeunes femmes , donc avec les grossesses , on roule presque toujours à trois congés de maternité quand ça va bien , a-t-elle ajouté . -" La France a été exposée à la rage depuis 2001 , et la mère ou la mère a été importée d' un autre pays où elle est encore présente " , dit le communiqué . " La France étant indemne de rage depuis 2001 , ce chaton ou la mère ont été importés d' un autre pays , non indemne " , selon le communiqué . -" L' Allemagne doit se réveiller " , a déclaré Oliver , président de , qui représente les petites et moyennes entreprises informatiques allemandes . « L' Allemagne doit se réveiller » , déclare Oliver , président de , qui représente les PME allemandes du secteur des TI . -" Il est un héros sans aucun doute " , dit O' Malley . « C' est un héros , il n' y a pas de doute » , a déclaré Katie O' Malley . -" Il a un gros conflit d' intérêts en ce qui concerne le congé parental rémunéré parce que si l' une de ces filles tombe enceintes , il aura un intérêt direct s' il a ou non un congé " , a déclaré M. Palmer . « Il aura un important conflit d' intérêt quand il s' agira de payer le congé parental car si l' une de ses filles est enceinte , il aura un intérêt direct dans le fait qu' elle obtienne ou non un congé » , a déclaré M. Palmer . -" Il ne se pas " , a dit Mme . « Il ne s' est pas rendu » , a ajouté Mme . -" Il est mort dimanche matin , en regardant les arbres et en prenant la célèbre 21e forme du chi , avec ses musiciens jouant l' air " , a écrit Laurie Anderson . " Il est mort dimanche matin , alors qu' il regardait les arbres et effectuait le célèbre 21e mouvement de chi , avec ses mains de musicien qui l' air " , écrit Laurie Anderson . -" Il ne veut pas de démocratie multipartite , il ne veut pas d' élections transparentes , il ne veut pas de la paix parce qu' il ne veut pas quitter la présidence " , a déclaré M. . « Il ne veut pas d' une démocratie multipartite , il ne veut pas d' élections transparentes , il ne veut pas la paix car il ne veut pas quitter la présidence » , a déclaré M. . -" Il n' a toujours pas pu voir un avocat et aucun volontaire n' a pu obtenir une copie des dossiers . Jusqu' ici aucun avocat n' a pu le rencontrer et aucun volontaire n' a pu prendre une copie des documents de l' affaire . -" Il doit être placé devant le peloton d' entraînement " , a déclaré Mme au . « Il devrait être traîné devant le peloton d' exécution » , a déclaré Mme à l' AAP . -" Nous sommes donc en train d' élaborer des projets pour lesquels ils peuvent s' identifier à tous " , explique le président de Vision , . " Donc , on crée des projets de façon à ce qu' ils s' identifient à tout ça " , explique la présidente de Vision diversité , . -" Son corps a été enterré dans une tombe commune dans le cimetière juif de Berlin Mitte en 1945 " , a-t-il confirmé dans le journal populaire quotidien , fondant sa déposition sur les archives . " Son corps a été enterré en 1945 dans une fosse commune du cimetière juif de " , a-t-il assuré au quotidien à grand tirage en s' appuyant sur des documents d' archives . -" Son côté humain est venu " , se . " Son côté humain est ressorti " , . -" Sa relation avec le succès est la nôtre et nous nous sommes : ce qui est un mal de beauté et un candidat qui n' est pas au bon endroit au bon moment , sans être honnête , mais avec une logique d' impunité " , , qui , avec son frère Joel , a reçu plusieurs Oscars et Cannes Awards , a dit à la presse . " Sa relation avec le succès est torturée , c' est cela qui nous intéressait : un mélange de malchance , un gars qui n' est pas là au bon moment , qui n' est pas mais intègre , avec un comportement " , déclarait à la presse , couronné avec son frère Joel de plusieurs Oscars et prix à Cannes . -" Ses papiers de service et sa photo , entre autres , ont été retrouvés dans la poche gauche " , a-t-il poursuivi . " Dans la poche intérieure gauche se trouvaient notamment ses états de service avec une photo " , poursuit-il . -" Cependant , certains États ont refusé d' envoyer ces mêmes cartes à des conjoints de même sexe dans les établissements de la Garde nationale qu' ils ont mis en place dans cette région , les accusant , eux aussi , d' enfreindre la loi fédérale et le principe d' égalité . " Mais certains États refusent de délivrer ces cartes aux conjoints de même sexe dans les installations de la Garde nationale " implantées sur leur territoire , a-t-il dénoncé , reprochant à ces États de violer la loi fédérale et le principe d' égalité . -" Cependant , nous continuerons d' accepter les situations d' urgence " , a-t-il souligné , insistant sur la présence permanente des véhicules de service médical d' urgence sur le site afin de transporter des cas critiques . " Mais nous continuerons à accueillir des patients en urgence " , , faisant valoir la présence permanente de véhicules du sur place pour transférer les cas graves . -" J' espère que le paquet d' aide du gouvernement fédéral contribuera grandement à faire passer le système de santé publique de l' État à la liste critique " , a déclaré M. . « J' espère que le plan d' aide fédéral contribuera grandement à retirer le système de santé public de Tasmanie de la liste noire » , a déclaré M. . -" Je ne dis pas " , Gerard de Villiers a dit dans sa défense : " Je ne fais que des conjectures basées sur des pays que je connais bien et , à l' occasion , certaines de mes régressions sont vraies " . Je ne suis pas , se défendait Gérard de Villiers , je fais simplement des hypothèses à partir de pays que je connais bien et , de temps en temps , certaines de mes hypothèses se réalisent . -" Je ne suis pas sûr quand il dort " , raconte Katie O' Malley , un étudiant en médecine de deuxième classe de l' Université de à Philadelphie et bénévole du groupe . « Je ne sais pas quand il dort » , a déclaré Katie O' Malley , une étudiante en médecine en deuxième année de l' université à Philadelphie et bénévole au sein du groupe du Dr . -" Je suis passé d' un château à un autre " . Bernard l' a dit ironiquement , il est familier non seulement avec les habitudes des six présidents qu' il a , mais aussi avec ceux de ses épouses et de certains de ses illustres hôtes . " Je suis passé d' un château l' autre ! " , Bernard , qui connaît non seulement les des six présidents pour lesquels il a , mais aussi de leurs épouses et de certains de leurs illustres invités . -" Je suis frustré et comme quelqu' un , dit-elle . « Je suis aussi frustrée et irritée que n' importe qui » , a-t-elle ajouté . -" Je suis d' avis qu' il existe une base suffisante pour une enquête officielle " , a-t-il déclaré . « Je considère qu' il existe une base suffisante » pour ouvrir une enquête officielle , a-t-il déclaré . -" Je suis heureux que les États membres aient accepté de lancer et de prendre le contrôle d' un processus visant à établir des objectifs universels de développement durable - " , a-t-il dit . « Je suis heureux que les États membres aient accepté de lancer et de s' approprier le processus visant à fixer des objectifs de développement durable universels ( ) » , a-t-il ajouté . -" Je suis si heureuse lorsque je fais ce travail " , dit . « Je suis si heureux quand je fais ce travail » , a ajouté M. . -" J' ai choisi une carte de la France comme logo " , ajoute-t-il . J' ai choisi une carte de France comme logo , justifie t-il . -" Je trouve merveilleux qu' en Amérique , le mythe et le folklore aient déjà une place dans l' éducation " , a-t-il dit . « Je trouve formidable qu' en Amérique , les mythes et le folklore fassent déjà partie de l' éducation » , a-t-il dit . -" Je suit mes dossiers ( sur l' Afghanistan , la Syrie , etc. ) sans tarder " , a ajouté Marc de Villiers . " Je suis en permanence mes dossiers ( Afghanistan , Syrie , ... ) avant de partir " , ajoutait Gérard de Villiers . -" Je suis sorti et je suis allée enlever les gros blocs qui le couvraient . Je suis débarqué et je suis allé enlever les grosses briques qui le recouvraient . -" J' ai porté un fusil noir SS à deux fois par semaine pendant trois ans " , affirme Paul Dutton , expliquant qu' il souffrait de " troubles mentaux " . " Je porte un noir SS deux fois par semaine depuis trois ans chez " , a affirmé Paul Dutton , expliquant souffrir de " problèmes mentaux " . -" J' espère que le président nous écoutera , et nous l' invitons à rejeter ce projet de loi et à le renvoyer aux députés " , a-t-il dit . « J' espère que le président nous écoutera , et nous lui demandons de rejeter ce projet de loi et de le renvoyer devant les parlementaires » , a-t-il ajouté . -" Je savais que je n' étais pas une fille " , a-t-il dit . « Je savais que je n' étais pas une fille » , a-t-il déclaré sur -" Je sais que les valeurs de l' égalité que je défends dans mon manifeste sont liées à la gauche , mais si les gens de la droite sont prêts à me soutenir , je les aussi " , . Je sais bien que les valeurs d' égalité que je prône dans mon programme sont rattachées à la gauche mais si les gens de droite sont prêts à me soutenir , je les aussi , poursuit-il . -" Je n' ai jamais prétendu être un auteur de grande littérature " , a expliqué Gérard de Villiers . " Je n' ai jamais eu la prétention d' être un auteur littéraire " , expliquait Gérard de Villiers . -" Je salue le courage et le courage dont vous avez fait preuve dans le cadre de la mission la plus ardue jamais entreprise par cette organisation " , a déclaré le directeur général de la sécurité , Ahmet , dans les commentaires de l' OIAC . « Je salue la force morale et le courage dont vous avez tous fait preuve dans le cadre de la mission la plus difficile jamais entreprise par cette organisation » , a déclaré le directeur général de l' organisation Ahmet , dans un commentaire publié par l' OIAC . -" Je passe beaucoup de temps à rassurer les parents -- en général , cela ne témoigne pas d' une progression rapide vers la puberté " , dit . « Je passe beaucoup de temps à rassurer les parents – en général , cela n' indique pas une évolution rapide vers la pleine puberté » , a déclaré . -" Je me considère comme un conteur qui écrit pour divertir les gens , ne lui envoie pas de message . Je me considère comme un conteur qui écrit pour distraire des gens à qui je n' envoie pas de message . -" Je lui ai dit que je n' étais pas prêt " , a-t-il déclaré lors d' une entrevue avec La Presse canadienne mercredi . " Je leur ai dit que je n' étais pas prêt " , a-t-il affirmé lors d' une entrevue avec La Presse Canadienne mercredi . -" Je veux que nous notre " , a-t-il dit . « Je veux que nous nous sur notre talent » , a-t-il déclaré . -" J' ai pu porter le costume que j' avais prévu l' année dernière " . Moi , j' ai pu enfin mettre le costume que j' avais prévu l' an dernier . -" Je dans ma voiture quand j' ai vu le mur , créant un énorme nuage de poussière " , a déclaré Sylvain Jean , qui vit près du site . Je en voiture et j' ai vu le mur s' écrouler avec un gros nuage de fumée , a dit Sylvain Jean , qui demeure à proximité du chantier . -" Je quand j' ai vu une femme qui semblait très perturbée . Je faisais la queue quand j' ai vu une femme qui semblait bouleversée . -" J' ai eu l' occasion de discuter d' une foule de choses , et c' est l' une d' elles " , a dit M. au journal . « J' ai dit que j' avais examiné toute une série de questions , et c' était l' une d' elles » , a déclaré au journal . -" Je dirais même plus loin " , a ajouté le directeur général , Hélène , avec un sourire . " Je dirais même au-delà " , a ajouté la directrice générale , Hélène , en souriant . -" Si un client dépense trente secondes de moins par rapport à la biométrie , c' est beau " , dit le directeur d' un grand magasin . " Si , grâce à la biométrie , un client reste en caisse trente secondes de moins , c' est appréciable " , explique le directeur d' une grande surface . -" Si l' accent était mis sur les soins palliatifs au Canada , l' appel à l' euthanasie deviendrait minime " , a-t-elle ajouté . Si on mettait l' accent sur les soins palliatifs au Canada , l' approche de l' euthanasie deviendrait minime . -" Si tous les malades avaient accès à des soins efficaces pour soulager leur souffrance , en plus de pouvoir demeurer chez eux , ils auraient très peu de chances de terminer leur vie " , a déclaré un porte-parole de l' Association canadienne des soins palliatifs , Maryse . Si tous les malades avaient accès à des soins efficaces pour apaiser leurs souffrances , en plus de pouvoir rester à la maison , très peu d' entre eux voudraient mettre fin à leurs jours , selon une porte-parole de l' Association canadienne des soins palliatifs , Maryse . -" Si d' autres supporters veulent se rendre à Paris par d' autres moyens , la police française a indiqué que des mesures ont été prises pour permettre l' arrestation et la prise en charge de personnes " , a ajouté Mme Marina Verbeke . " Si d' autres supporters avaient une intention de se rendre à Paris par d' autres moyens , la préfecture de police française a fait savoir que des mesures ont été prises permettant de procéder à des arrestations voire des gardes à vue " , a encore annoncé Marie Verbeke . -" Si le ministre ne prend pas un lundi , nous prendrons une approche différente " , Christophe , médecin d' urgence et membre de la HSE . Si la ministre ne prend pas de mesure d' ici lundi , nous allons changer de , prévient Christophe , médecin et membre de la CGT Santé . -" Si le véhicule est percuté à l' arrière , l' appuie-tête ne peut pas protéger adéquatement les occupants , augmentant le risque de blessures " , selon l' avis affiché sur le site Web de l' Administration nationale de la sécurité routière . « Si le véhicule est percuté par l' arrière , il se peut que l' appuie-tête ne protège pas correctement les passagers , augmentant ainsi le risque de blessures » , selon la déclaration affichée sur le site de la National Highway Traffic Safety Administration ( NHTSA ) . -" Dans le domaine de l' aérospatiale , les résultats étaient conformes à notre orientation , mais la faible prise de commandes et les conditions générales du marché étaient décevantes " , a déclaré M. Beaudoin . « En Aéronautique , les résultats étaient conformes à nos prévisions , mais le faible nombre de commandes et les conditions générales du marché ont été décevants » , a déclaré M. Beaudoin . -" Dans certains cas , je reconnais , comme le président , que certaines de ces activités se sont et que nous ne garantissons pas que cela sera possible à l' avenir " , a déclaré John Kerry lors d' une conférence à Londres où il a pris part à Washington via un lien vidéo le jeudi 31 octobre . " Dans certains cas , je vous le concède , comme l' a fait le président , certaines de ces actions sont allées trop loin et nous allons nous assurer que cela n' arrive plus à l' avenir " , a déclaré John Kerry lors d' une conférence à Londres à laquelle il participait depuis Washington jeudi 31 octobre par liaison vidéo . -" Dans le passé , les cibles étaient les missiles transférés au Hezbollah " , a-t-il ajouté . " Par le passé , les cibles ont été des missiles transférés au Hezbollah " , s' est-il contenté d' ajouter . -" Dans ce cas , j' ai infligé des dommages à et à l' ensemble de l' industrie des médias d' information et de sa capacité à gagner la confiance du public " , a déclaré la chaîne de télévision publique . « Dans cette affaire , j' ai nui à ainsi qu' à l' industrie des médias et à sa capacité à gagner la confiance du public » , a-t-il déclaré au réseau télévisé public CCTV . -" Inside Davis " par les frères , le dernier Grand Prix du Festival de Cannes , est un concours de comédie historique qui s' est tenu dans le village de en 1840 et qui se fonde sur la musique folklorique , qui s' est fait juste avant l' arrivée de Bob Dylan . " Inside Davis " des frères , Grand prix du dernier Festival de Cannes , est un film nostalgique et drôle sur le Greenwich village de 1961 et la musique folk qui commençait tout juste à y résonner , avant l' arrivée de Bob Dylan . -" Inside Davis " raconte l' histoire d' une semaine de tribulations pour un chanteur folk qui n' a pas réussi une pause et est en colère contre le monde entier . " Inside Davis " raconte sur une semaine les tribulations d' un chanteur de folk qui n' arrive pas à percer et se avec la terre entière . -" with and est la deuxième solution pour les enfants " , souligne Ken Albert , vice-président des produits numériques au groupe Disney / ABC Television Group , dans une citation dans le communiqué de presse . " avec les smartphones et les tablettes est une seconde nature pour les enfants aujourd' hui " , a commenté Albert Cheng , vice-président chargé des produits numériques chez Disney / ABC Television Group , cité dans le communiqué . -" C' est toujours le chef d' État ou son partenaire ou épouse qui choisit le menu " , . " C' est toujours le chef de l' Etat ou sa compagne ou épouse qui choisit les menus " , . -" C' est manifestement dû à l' avertissement du profit de Nissan " , a déclaré un représentant commercial d' un courtier de Paris . " C' est clairement le profit warning de Nissan " , dit un responsable du trading d' un courtier parisien . -" Ce n' est pas un procès " , a déclaré un militant des Frères musulmans . Ce n' est pas un procès souligne ce militant des Frères musulmans . -" Ce n' est pas facile " , dit . « Ce n' est pas facile » , ajoute M. . -" Il est évident que c' est un moment inquiétant , mais nous avons hâte de la ramener chez nous en toute sécurité au plus vite " , dit . « Nous avons évidemment vécu un moment , mais nous espérons la ramener à la maison en toute sécurité dès que possible » , a déclaré . -" Il est de la plus haute importance que les intérêts des citoyens européens en matière de sécurité pour les Européens soient maintenus avec nous dans le cadre de nos efforts mutuels en matière de lutte antiterroriste " , a déclaré M. Kerry , président et président de la section des Affaires étrangères du Sénat sur les affaires européennes , dans une interview de Washington . « Il est vraiment important pour la protection de la sécurité nationale aux États-Unis que les Européens restent à nos côtés dans l' engagement mutuel dans la lutte contre le terrorisme » , a déclaré Murphy , un démocrate servant son premier mandat , et président de la sous-commission des affaires européennes du Sénat américain , lors d' un entretien depuis Washington . -" Il s' agit de deux grands hommes ( et ) main main " , dit Joseph , professeur à l' Open University et expert en Mozambique . « Ce sont deux grands hommes ( et ) qui n' arrêtent pas de se disputer » , a expliqué Joseph , un conférencier de l' université ouverte et spécialiste du Mozambique . -" Il amène les enfants à prendre la décision et retire l' anxiété qui motive les parents parce qu' ils ne sentent pas qu' ils font ce qu' il faut " , dit-elle . « Cela met les enfants au cœur de la décision et atténue le sentiment d' anxiété qui motive les parents parce qu' ils ont peur de ne pas faire ce qu' il faut » , a-t-elle déclaré . -" C' est un endroit effrayant et c' est une question à défendre : qu' est-ce qui empêche le parlement de simplement l' indépendance de la justice ? " , a déclaré le projet de loi inconstitutionnel . « C' est un pays effrayant et il est légitime de se poser la question suivante : qu' est-ce qui empêchera le Parlement de balayer l' indépendance du système judiciaire demain ? » , a déclaré le journal en contestant le projet de loi et demandant à ce qu' il soit déclaré inconstitutionnel . -" Il est toujours préférable d' examiner , de consulter et de trouver des solutions à ces problèmes " , dit Mme . " C' est toujours mieux de s' en parler , de se concerter et de trouver des solutions " , estime Mme . -" Il est de plus en plus improbable que la stratégie du gouvernement permette de garantir suffisamment de devises étrangères pour répondre à ses engagements financiers extérieurs accrus " , explique l' agence américaine . " Il est de moins en moins probable que la stratégie du gouvernement permette de garantir suffisamment de devises étrangères pour faire face à ses besoins de financement extérieurs élevés " , explique l' agence américaine . -" Il ne s' agit pas de quelque chose que nous pourrions faire " , a déclaré Hasan , directeur exécutif du de la Chine du sud des gouvernements , qui a l' intention de l' État de commencer à franchir les parcourus par tous les Nord-Américains d' ici 2025 . « Ce n' est pas comme si nous avions le choix » , a déclaré Hasan , directeur général de la Southern California Association of Governments , qui prévoit que l' État commence à enregistrer les miles parcourus par chaque automobiliste californien d' ici 2025 . -" Ce n' est que des partis extrémistes que je ne vais pas parler " . Il n' y a qu' aux partis extrémistes que je parle pas . -" Il peut s' agir d' une conversation " , a déclaré . « Cela peut vouloir dire avoir une conversation plus tôt que prévu » , a expliqué . -" Il est vraiment temps de donner de la souplesse aux compagnies aériennes et de leur permettre de faire les choses qu' ils croient devoir faire pour réussir " , a expliqué Andrew , de Boeing . « Il s' agit vraiment d' apporter de la souplesse aux compagnies aériennes et de leur permettre de faire ce qu' elles pensent devoir faire pour mener à bien leurs activités » , a déclaré Kent , en charge de la satisfaction passagers chez Boeing . -" Il varie d' un quart à l' autre face à une concurrence très vive " . Ça varie d' un trimestre à l' autre avec une concurrence très vive . -" C' était un choc " , se souvient d' une femme dont la fille a commencé sa période de dix ans . « Ça a été un choc » , se rappelle une femme dont la fille a eu ses premières règles à 10 ans . -" Il sera intéressant de voir si le nouveau pape proposera des cardinaux du reste du monde pour rétablir l' équilibre " , a-t-il déclaré . « Il sera intéressant de voir si le nouveau pape nommera des cardinaux du reste du monde pour restaurer l' équilibre » , a-t-il ajouté . -" La Nouvelle Star a fait un grand bond en avant en termes de population , avec 1,3 million de téléspectateurs , ce qui représente une part de 5,6 % du lectorat . La Nouvelle Star a réussi un retour en force , côté audiences , avec 1,3 millions de téléspectateurs derrière leur poste de télé , soit 5,6 % de part d' audience . -" La Royale " - la marine française - vient de confirmer que le navire sera " indisponible jusqu' à la mi-novembre " afin de " laisser le temps nécessaire à l' exécution des mesures correctives nécessaires " sur le réacteur . La Royale vient en effet de nous confirmer " l' indisponibilité " du navire jusqu' à " la mi-novembre " , " le temps de prendre les mesures correctrices qui s' imposent " sur le réacteur . -" L' an dernier , il y avait beaucoup plus de gens et il n' y avait plus de pharmaciens disponibles pour les remplacer , ce qui était plus difficile " . La dernière année , on en a eu beaucoup plus et on n' a pas de pharmaciens qui sont disponibles pour faire des remplacements , donc c' était plus difficile . -" Les agents de la force publique sont sur le terrain " , a déclaré l' aéroport sur son compte Twitter , alors que des images télévisées montraient que les gens étaient évacués dans des ambulances . " Les forces de l' ordre sont sur place " , précise l' aéroport sur son compte Twitter , alors que les images des télévisions montraient des personnes évacuées dans des ambulances . -" Le Nouvel Observateur " interviewé Jean-Louis , ancien journaliste senior qui a pris sa retraite en 2008 et est président de l' association " du Monde " . " Le Nouvel Observateur " a questionné Jean-Louis , 62 ans , ex grand reporter , retraité depuis 2008 , président de l' association " du monde " . -" à eux-mêmes , les politiciens le pays et nous à la chasse et à la cueillette " , a-t-il écrit . « à eux-mêmes , les hommes politiques mettraient le pays en faillite et nous à la chasse et à la cueillette . » -" Je tiens à dire directement à ces Américains : vous méritez mieux " , a déclaré lors d' un témoignage devant la commission de l' énergie et du commerce de Washington . « Permettez-moi de m' adresser directement à ces Américains : vous méritez mieux » , a déclaré Kathleen lors de son témoignage devant la commission de l' énergie et du commerce à Washington . -" Lou et moi avons passé beaucoup de temps ici ces dernières années . Lou et moi avons passé beaucoup de temps ici ces dernières années . -" Les déclarations sur les " étrangers qui arrivent à la queue " et les emplois " " sont quelque peu à un moment où les syndicats ont demandé un marché du travail protégé , a déclaré aujourd ' hui à l' Institut australien d' Inde . « Les déclarations sur les " étrangers retournant au bout de la file " et " des emplois pour les Australiens d' abord " sont un retour désagréable à l' époque où les syndicats demandaient un marché du travail protégé » , a-t-elle dit à l' Australia India Institute aujourd' hui . -" Mme Chirac était un peu plus impliquée et dans la cuisine " , au point où elle a interdit aux femmes d' entrer dans les cuisines du Palais de l' Elysée ! " Mme Chirac était un peu plus impliquée , elle venait directement en cuisine " , au point qu' elle avait interdit les femmes dans les cuisines de l' Elysée ! -" L' argent a présenté le cas comme un cas de transition réussie , mais ce n' est pas le cas " , a déclaré M. . « Money a présenté cela comme un cas de transition réussi , mais ce n' était pas le cas » , a déclaré . -" La plupart de ces sont apparus à et sont allés plus loin , au point où ils sont trop grands pour jouer un club " , a déclaré Hirsch . « La plupart des têtes d' affiche ont émergé au , et ont connu un grand succès , jusqu' à ce qu' ils soient devenus trop célèbres pour jouer dans un club » , a déclaré Hirsch . -" Les automobilistes seraient incités à participer en raison de la valeur des avantages qu' ils offrent " , a déclaré un document de planification de la ville . « Les automobilistes seraient incités à participer en raison des avantages que le système leur offre » , a indiqué un document sur l' aménagement urbain . -" Le corps de Muller a été retrouvé en août 1945 par un commando dans une fosse temporaire près de l' ancien ministère de l' aviation du Reich " , dit M. . " Dès août 1945 , le corps de Müller a été retrouvé dans une tombe provisoire près de l' ancien ministère de l' Aviation du Reich par un commando " , selon M. . -" Mes premiers cours de musique m' ont été donnés par mon grand-père qui m' a appris à jouer la guitare " , a-t-il ajouté . " Mes premières leçons de musique m' ont été données par mon grand-père qui m' a appris à jouer de la guitare " , avait-il ajouté . -" Mon mari et moi , je viens d' acheter une maison , de sorte que tout mon argent est bloqué " , dit Mme . « Mon mari et moi venons d' acheter une maison donc tout mon argent est parti dedans » , déclare Mme . -" Ma principale proposition est de créer un ministère du Multiculturalisme " , dit-il . Ma proposition n ° 1 est de créer un ministère du multiculturalisme , indique t-il . -" La gestion éditoriale de New Express a été désordonnée " , a déclaré l' organisme de réglementation dans une déclaration . « La rédaction de New Express était » , a dit le régulateur dans une déclaration . -" Non , M. a répondu . « Non » , a répondu M. . -" Personne ne se soucie de et de , ils veulent juste la paix , ils veulent un accès libre aux routes " , a-t-il dit . « Personne ne se soucie de la ni du , ils veulent juste retrouver la paix , ils veulent avoir un accès libre aux routes » , a-t-il ajouté . -" Personne ne saura jamais " , écrivait le journaliste de la nation , décrivant les médias kényans comme une importante source de contrepoids dans la vie publique . « Personne ne saura jamais » , a écrit , journaliste à The Nation , en décrivant les médias kenyans comme une source de contrôle clé dans la vie publique . -" Personne ne nous dit : " Je veux mourir , me donner une injection " . On ne nous demande jamais : " je veux mourir , une injection " . -" Ne tenant pas compte du fait que les clients de la région de Bruxelles n' aient pas d' argent dans ce domaine en vue de la mise en place d' un de lumière aux couleurs de la région bruxelloise , Brigitte poursuit ses efforts pour transformer la capitale flamande " , a déclaré une dame de De dans un communiqué de presse . " Non content de dépenser l' argent des dans des opérations de des poteaux de signalisation aux couleurs de la Région bruxelloise , Brigitte continue ses opérations de de la Capitale " , s' insurge M. De dans un communiqué . -" Maintenant , M. Francis a choisi ces huit cardinaux pour l' aider " , a déclaré M. . « Désormais , François a choisi ces huit cardinaux pour qu' ils l' aident » , a expliqué . -" Le 4 novembre , les patients ne seront plus amenés par la brigade des pompiers , ce qui signifie qu' ils n' arriveront plus que par leurs propres moyens " , confirme le Professeur Jean-Yves , chef de la direction médicale du nouvel . " Le 4 novembre , il n' y aura plus de patients amenés par les pompiers , uniquement des personnes venant par leur propre moyen " , confirme le professeur Jean-Yves , le responsable médical du nouvel . -" Une fois qu' une compagnie aérienne vérifiera la tolérance de sa flotte , elle peut permettre aux passagers d' utiliser des appareils électroniques légers et portatifs comme des comprimés , des et des toutes les altitudes " , a déclaré la FAA . « Dès qu' une compagnie aérienne a vérifié la tolérance de sa flotte , elle peut autoriser les passagers à utiliser des appareils électroniques portables légers , comme des tablettes , des lecteurs de livres électroniques et des smartphones à toutes les altitudes » , a déclaré la FAA . -" Une fois que les seins commencent à se développer , il faut au moins deux ou trois ans avant les premières règles " , dit , auteur du " Early in Girls : The Essential Guide to This Common Problem " . « Une fois que la poitrine a commencé à se développer , il faut au moins deux ou trois ans avant que les premières règles apparaissent » , a indiqué , également l' auteur de « Early in Girls : The Essential Guide to with This Common Problem » . -" Une fois la nouvelle politique mise en oeuvre de façon sécuritaire - et nous allons travailler étroitement avec le transporteur pour le faire - nous aurons une chance " , a déclaré M. dans une déclaration . « Une fois la nouvelle politique mise en œuvre en toute sécurité – et nous allons travailler en étroite collaboration avec les transporteurs pour cela – ce sera profitable à tout le monde » , a expliqué dans une déclaration . -" Notre politique actuelle est que les appareils électroniques ne peuvent pas être utilisés au décollage et à l' atterrissage et nous n' avons pas de plan immédiat pour le changer " , a-t-il dit . « Conformément à notre politique actuelle , les appareils électroniques ne peuvent pas être utilisés pendant le décollage et l' atterrissage , et nous n' avons pas l' intention dans l' immédiat de changer cela » , a-t-elle déclaré . -" Notre enquête en cours sur cette question se poursuit et nous coopérons pleinement avec la CAF et les autres organismes de réglementation " , a déclaré la banque il y a deux semaines . « L' enquête en cours sur cette affaire se poursuit et nous coopérons pleinement avec la FCA et les autres régulateurs » , a déclaré un représentant de la banque il y a deux semaines . -" Notre vision est de revenir aux négociations , mais avec beaucoup de sérieux " , a déclaré M. . « Nous voulons reprendre les négociations , mais avec le plus grand sérieux » , a indiqué M. . -" Le comportement parental est assez stressant avec des limitations supplémentaires , surtout si tu ne connais pas le sexe de ton enfant " , dit-elle sur le site Web des parents . « Le rôle de parent est suffisamment stressant sans restrictions supplémentaires , en particulier si vous ne connaissez pas le sexe de votre enfant » , a-t-elle déclaré sur le site destiné aux jeunes parents . -" Les gens sont heureux . Les gens sont contents . -" Les gens pensent que les responsables de l' avenir du pays sont le gouvernement et le président , et il devrait être le responsable de trouver des solutions aux problèmes " , dit-il . « Les gens pensent que les responsables de l' avenir du pays sont le gouvernement et le président , et que c' est à ce dernier de trouver des solutions aux problèmes » , a-t-il expliqué . -" Les gens seront plus disposés à le faire si vous ne suivez pas leur rythme et que vous ne suivez pas leur route " , indique Ryan Morrison , directeur général de . « Les gens seront plus enclins à faire ça si vous n' enregistrez pas leur vitesse et ne les pas » , a déclaré Ryan Morrison , directeur général de True . -" Le plan d' acheter Goodyear commence par zéro , a déclaré le CEO de Titan ... Pour le PDG de Titan , " le projet d' achat de Goodyear Amiens démarre avec zéro employé " -" Le Pape a décidé de faire part de sa décision de convoquer les de février à afin de faciliter la planification d' autres réunions impliquant la participation de de différentes parties du monde " , a déclaré . « Le pape François a décidé de communiquer sa décision de convoquer le de février à l' avance afin de faciliter la planification des autres réunions nécessitant la participation des cardinaux de différentes parties du monde » , a déclaré . -" Peut-être " , répondit Martin . « Peut-être » , a répondu Martin . -" Le traitement préventif pour la rage humaine administré après le contact avec l' animal , bien que les symptômes soient très efficaces , est indiqué dans le communiqué de presse . " Chez l' homme , le traitement préventif de la rage humaine , administré après le contact avec l' animal porteur , mais avant l' apparition des symptômes , est très efficace " , précise le communiqué . -" probablement entre 120 et 150 millions , si vous incluez tous les pays " , a-t-il déclaré en mars dernier . " Sans doute entre 120 et 150 millions tous pays confondus " , en mars dernier . -" La rage est une maladie mortelle si elle n' est pas traitée rapidement " , a rappelé les ministères , et elle peut être transmise au cours de la seconde quinzaine avant même les premiers symptômes de la maladie . " La rage est une maladie mortelle si elle n' est pas traitée à temps " , rappellent les ministères et elle est transmissible pendant près de 15 jours avant l' apparition des premiers symptômes de la maladie . -" Des médecins d' urgence de haut rang resteront sur place " , lui aussi . Des médecins seniors resteront sur place , également . -" Partager ce qu' est la culture québécoise , par exemple , le sentiment d' appartenance , le soutien aux personnes qui vivent des changements que nous vivons dans l' organisme étudiant " , . " Sur la transmission , par exemple , de ce qu' est la culture québécoise , le sentiment d' appartenance , le soutien au personnel dans les transformations qu' on vit , au sein de la clientèle étudiante " , . -" La signature de l' accord serait bénéfique pour les entreprises à long terme , mais elle pourrait avoir des conséquences négatives à court terme en réponse à la réaction de la Russie " , a expliqué M. Fernandes et , à ce sujet , que Moscou pourrait introduire des " restrictions commerciales " . " l' accord serait positif pour le commerce à long terme mais il pourrait y avoir des conséquences négatives à court et moyen termes liées à la réaction de la Russie " , a expliqué S & P , qui craint des " restrictions commerciales " de la part de Moscou . -" Comme la viande de cheval n' est pas mentionnée dans la liste des ingrédients , elle n' aurait pas dû être présente dans le produit " , a expliqué l' agence britannique . " Comme la viande de cheval n' est pas mentionnée dans la liste des ingrédients , elle n' aurait pas dû être présente dans le produit " , a expliqué l' agence britannique . -" Les petites et moyennes entreprises manquent souvent d' expérience , de personnel et de ressources financières pour protéger les secrets d' entreprise contre l' accès non autorisé " , a souligné le dans un rapport . « Les PME manquent souvent de l' expérience , du personnel et des ressources financières pour protéger efficacement leurs secrets industriels contre un accès non autorisé » , avertit le dans un rapport . -" Certaines personnes ont des conditions qui nécessitent une intervention chirurgicale , mais la plupart des enfants ne le font pas " , dit-il . « Certaines personnes souffrent de maladies extrêmement graves qui nécessitent un acte chirurgical , mais ce n' est pas le cas de la plupart des enfants » , a-t-il ajouté . -" Certaines femmes dans mes livres sont des objets sexuels , d' autres sont de belles , intelligentes femmes courageuses . Certaines femmes sont des objets sexuels dans mes livres mais d' autres sont des femmes belles , intelligentes et courageuses . -" Il ne tient pas compte des publications de pirates " , a-t-il ajouté , soulignant un tas de livres sur la table basse , extraits des artefacts de bronze et d' ivoire provenant des 130 pays qu' il avait étudiés pour les réglages de ses romans . " Sans compter les éditions pirate " , , désignant une pile de livres sur la table basse entre les ou les rapportés des 130 pays pour y situer ses romans . -" Ce n' est pas le cas " , a déclaré M. , président de la société allemande de logiciels SAP . « Ce n' est pas la bonne façon de faire » , indique , président de la société de logiciels d' entreprise allemande , SAP . -" Cela fait partie de l' enquête " , dit Jacqueline , porte-parole de l' ASFC . Ça fait partie de l' enquête , a indiqué Jacqueline , porte-parole de l' ASFC . -" Cet appui se poursuivra pendant les six prochains mois " , explique le directeur régional des relations de travail dans les hôtels Delta , Felix . " Ce va continuer pour les six prochains mois " , explique le directeur régional des relations de travail des Hôtels Delta , Félix . -" Le Groupe et le Groupe ont annoncé aujourd' hui l' expiration de la période d' examen de la prétendue fusion du Groupe et , par le biais de la Antitrust Act de 1976 , telle que modifiée , les deux groupes annoncés dans un communiqué de presse . " Group et Groupe ont annoncé aujourd' hui l' expiration du délai d' examen de la fusion précédemment annoncée de Groupe et , prévu par le Antitrust Act of 1976 , tel qu' amendé " , annoncent les deux groupes dans un communiqué . -" Le semble proche de sa fin " , a annoncé un expert dans les affaires congolaises , Jason Stearns , sur son blog Congo . " Le semble toucher à sa fin " , prédit un expert des affaires congolaises , Jason Stearns , sur son blog Congo . -" Les devaient revenir à environ 5 heures " , a déclaré Thierry , consultant technique auprès de l' organisation qui a organisé les opérations de sauvetage . " Les auraient dû ressortir aux alentours de 5h du matin " , a dit Thierry , conseiller technique au secours en charge de l' organisation des secours . -" La Commission a proposé un service de médiation et j' ai été intéressée " , a déclaré Mme . " Il y a un service de médiation qui est proposé par la commission et j' étais intéressée " , explique Mme . -" La Fédération allemande des journalistes recommande aux journalistes d' éviter d' utiliser Google et Yahoo Search et les services de messagerie jusqu' à nouvel avis " , a déclaré dans un communiqué de presse . " La Fédération allemande des journalistes recommande aux journalistes d' éviter jusqu' à nouvel ordre d' utiliser les moteurs de recherche et les messageries de Google et Yahoo " , dit-elle dans un communiqué . -" Le gouvernement a présenté un plan de sauvetage d' urgence qui nous permettra de relever les défis uniques auxquels est confronté l' État " , a déclaré Mme aujourd ' hui . « Le gouvernement a proposé un plan de sauvetage d' urgence qui , selon nous , règlera les difficultés uniques auxquelles l' État est confronté » , a déclaré Mme aujourd' hui . -" Le service d' urgence d' doit fermer le plus tôt possible et , pour nous , le 4 novembre " , Loïc , président du comité médical ( CCS ) au qui soutient le projet de gestion , dit sans détour . " Le service d' accueil des urgences de l' doit fermer dans le plus bref délai , et pour nous , c' est le 4 novembre " , assure sans détour Loïc , le président de la commission médicale d' établissement ( CME ) de l' , qui soutient le projet de la direction . -" Les progrès dans le sens de la critique , l' augmentation du nombre de caméras de surveillance et le nombre considérable de photos diffusées sur Flickr , Facebook ou me font craindre le pire : une surveillance accrue " , a déclaré Andrew Patrick du Commissariat à la protection de la vie privée du Canada . " Les progrès de la reconnaissance faciale , la multiplication des caméras de surveillance et l' énorme quantité de photos disponibles sur Facebook , Flickr ou , me font craindre le pire : une surveillance généralisée " , prédit Andrew Patrick , du Commissariat à la protection de la vie privée au Canada . -" Les alarmes ont décollé une fois l' autre parce qu' il y a eu des renforts pour s' occuper de l' immeuble , parce que les bâtiments ici sont proches " , a déclaré Christian , porte-parole du service de lutte contre le feu du Québec . " Alors les alarmes se sont succédé , parce que ça prenait du renfort pour attaquer le bâtiment , parce que c' est des bâtiments collés ici " , a expliqué France , porte-parole des pompiers de Québec . -" Le processus d' embarquement était le plus agréable que j' avais observé dans ma carrière de transporteur aérien " , a-t-il dit . « Je n' avais jamais vu une procédure d' embarquement aussi fluide de toute ma carrière professionnelle » , . -" La ville est incontestablement dans l' univers de l' humour de tous les travaux de Larry " , a déclaré Steinberg . « La ville est définitivement dans l' ADN comique de tout le travail de Larry » , a déclaré Steinberg . -" Le convoi d' autobus sera escorté par la police d' un ancien poste frontière à au stade du PSG " , a déclaré le porte-parole . " Le convoi de bus sera escorté par la police depuis un ancien poste frontière de jusqu' au stade du PSG " , a transmis la porte-parole . -" La réaction rapide de l' agent de police et son aptitude à désamorcer la situation étaient absolument cruciaux pour mettre fin immédiatement à cet incident " , a déclaré le caporal Jamie Paquet Paquet dans un communiqué de presse . " Les réactions rapides des policiers impliqués et leur capacité de désamorcer la situation immédiatement ont été essentielles à la conclusion sécuritaire de cet incident " , a déclaré le caporal Bert Paquet par voie de communiqué . -" Les combats n' ont pas cessé depuis le matin et se sont poursuivis malgré l' obscurité " , un résident de , selon lequel une jeune fille a été blessée par des tirs au cours de la matinée , a indiqué à l' AFP par téléphone au , près de , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , . " Ça n' a pas cessé depuis ce matin , les combats continuent malgré la nuit " , a indiqué à l' AFP un habitant de , joint par téléphone vers 12h30 , et selon qui une fillette a été blessée par balle dans la matinée . -" Les combats ont légèrement diminué , ce qui semble indiquer que les Forces armées de la République démocratique du Congo ( FARDC ) ont repoussé les rebelles un peu " , a-t-il ajouté . " L' intensité des affrontements a diminué un peu " , a-t-il ajouté , " il semble que les FARDC Forces armées de la RDC ont repoussé un peu les rebelles " . -" La première action n' a eu lieu qu' à trois institutions , et lorsque la décision a été rendue publique , plusieurs personnes m' ont dit : " Nous avons été saisis d' une façon ou d' une autre , a-t-il ajouté , puis-je donner mon nom ? " , a déclaré l' avocat Alain Arsenault , qui défend les victimes dans les deux actions . Le premier recours ne visait que trois établissements , et quand on a annoncé le règlement , beaucoup de gens nous ont dit : " moi , j' étais à tel endroit , est-ce que je peux m' inscrire ? " , raconte l' avocat Alain Arsenault , qui défend les victimes dans les deux recours . -" La taxe sur le gaz n' est tout simplement pas viable " , a déclaré M. Lee , expert en politique des transports à l' Université du Minnesota . « La taxe sur l' essence n' est tout simplement pas tenable » , a déclaré Lee , un spécialiste des politiques de transports à l' université du Minnesota . -" Le groupe qui se cache derrière les événements était le Mouvement islamique du Turkestan oriental " , a-t-il ajouté , ses mots ont été relayés par l' agence Xinhua . " Le groupe qui se tenait en coulisses était le mouvement islamique du Turkestan oriental " , a-t-il ajouté , ses propos étant repris par l' agence Chine nouvelle . -" La loi donne aux parents un peu d' espace pour ne pas se précipiter pour prendre des décisions " , a déclaré . « La loi laisse une certaine latitude aux parents pour qu' ils n' aient pas à prendre de décisions hâtives » , a déclaré . -" La question est devenue exagérée " , pense Joël , directeur de bureau . L' affaire a pris des proportions exagérées , estime Joël , le directeur de l' office . -" La mémoire des victimes est dans la pire des situations " , a déclaré dans le document . " On foule grossièrement du pied la mémoire des victimes " , s' est-il dans le journal . -" La fusion est également subordonnée à l' obtention d' autres autorisations réglementaires et à l' approbation des deux actionnaires du groupe " , ajoute-t-il . " La fusion est également conditionnée à l' obtention d' autres autorisations réglementaires et à l' approbation des actionnaires des deux groupes " , . -" Le ministre nous a menti ou a menti " , a déclaré , médecin d' urgence . " La ministre nous a menti ou on lui a menti " , déplore le médecin Gérald . -" Le seul problème qu' ils rencontrent est celui de la pauvreté " , a déclaré . « Le seul problème qu' ils ont , c' est la pauvreté » , a expliqué le Dr . -" Le problème est le temps et l' argent . C' est un problème de temps et d' argent . -" L' accusation indique que M. , qui est actuellement le rédacteur en chef de l' , n' est pas celui qui est à l' extérieur des maisons populaires en attente de prendre les choses , il est l' homme qui a l' intention de raconter l' histoire pour voir ce qu' il dira " , a déclaré M. . « L' accusation suggère que M. , qui est désormais le rédacteur en chef de , n' est pas le genre d' homme à rester devant la maison des gens en espérant les surprendre , c' est le genre d' homme à aimer présenter l' histoire aux gens pour voir ce qu' ils en disent » , a déclaré M. . -" Le besoin réel est d' avoir différentes stratégies éducatives " , . " Des stratégies pédagogiques différentes , c' est ça le véritable besoin " , . -" La raison pour laquelle Boeing est là , c' est qu' ils plus de sièges afin de rendre leur avion plus compétitif à nos produits " , a déclaré Kevin , chef du confort des passagers de l' Airbus européen . « Boeing fait ça pour pouvoir plus de sièges et rendre ses avions plus compétitifs par rapports à nos produits » , a déclaré Kevin , directeur de Confort Passager chez l' avionneur européen Airbus . -" Nous sommes ici parce que le gouvernement du pays d' origine a sauvé 430 millions de dollars de son système de santé " , a-t-il déclaré à la télévision ABC . « La raison pour laquelle nous sommes ici est que le gouvernement travailliste de l' État a retiré 430 M $ de son système de santé » , a-t-il déclaré sur ABC TV . -" La recherche effectuée par les journalistes est tout aussi confidentielle que les détails de leurs sources et la nature de leur communication avec eux " , a ajouté Michael , président du syndicat , qui compte trente-huit membres . " Les recherches effectuées par les journalistes sont aussi confidentielles que les coordonnées de leurs sources et la nature de leurs communications avec elles " , a ajouté Michael , président du syndicat qui revendique 38.000 adhérents . -" La même chose se produira avec les unités de maternité de niveau 3 . Cela va se passer comme pour les maternités de niveau 3 . -" L' appareil d' essai n' est pas resté au sol plus longtemps que prévu " , a déclaré Beaudoin lors d' une conférence téléphonique , ajoutant que les essais sur le terrain et les mises à jour du programme étaient prévus pendant le temps d' immobilisation de l' avion . « L' avion d' essai n' est pas resté au sol plus longtemps que prévu » , a déclaré M. Beaudoin lors d' une conférence téléphonique , ajoutant que les essais au sol et les mises à jour logicielles étaient prévus pendant le temps d' arrêt de l' avion . -" Le procès du président Morsi est un faux procès " , selon un défenseur respecté de Morsi , " il doit faire preuve de , et non l' inverse " . Le procès du président Morsi est un faux procès s' insurge une , c' est lui qui est censé faire juger , pas le contraire . -" Les deux victimes ont été et traitées pendant un certain temps au centre , avant d' être remises au CHU de Liège , par hélicoptère et par ambulance pour la grand-mère et son petit-fils , respectivement " , a déclaré le maire José lors de la visite du site . " Les deux victimes ont été et longuement conditionnées sur place avant d' être emmenées au CHU de Liège , elle en hélicoptère , lui en ambulance " , a précisé le bourgmestre José , descendu sur les lieux . -" L' incident terroriste violent qui a eu lieu à Pékin a été organisé et prémédité " , a déclaré sur la chaîne de télévision Hong Kong Television Phoenix . " Le violent incident terroriste qui est survenu à Pékin a été organisé et prémédité " , a dit à la chaîne Phoenix TV . -" Il y a plusieurs questions à répondre ou à confirmer " , a-t-il dit . « Plusieurs questions méritent encore une réponse ou une confirmation » , a-t-il indiqué . -" Il y a toujours eu des rats dans les villes . Des rats , il y en a toujours eu dans les villes . -" Il n' y a pas grand espoir que des réformes impopulaires aient lieu dans une année préélectorale " , a déclaré l' expert lors d' une interview de l' AFP . Les élections présidentielles sont prévues pour 2015 . " Il y a peu d' espoir que des réformes impopulaires soient lancées dans une année préélectorale " , les élections présidentielles étant prévues en 2015 , relève l' expert , interrogé à l' AFP . -" Il n' est pas nécessaire d' entreprendre une chirurgie radicale lorsque vous avez besoin de prendre une aspirine " , a déclaré Randy , directeur de la politique et des affaires publiques de la Commission . « Il n' est pas nécessaire de recourir à une chirurgie radicale lorsqu ’ on a tout simplement besoin d ’ aspirine » , a déclaré Randy , le directeur de la législation et des affaires publiques auprès de la commission . -" Il y a eu une fusillade " , a déclaré un porte-parole de la US Transportation Security Administration ( TSA ) auprès de l' AFP . " Il y a eu une fusillade " , a déclaré à l' AFP un porte-parole de l' agence américaine de sécurité des transports ( TSA ) . -" Il n' y avait pas de au Cameroun " , dit Georges . « Il n' y avait pas de au Cameroun » , a déclaré Georges . -" Il y aura beaucoup de larmes cette année " , a déclaré Mme . « Il y aura beaucoup de pleurs cette année » , a dit Mme . -" Ces discussions doivent produire un nouveau type de cadre de référence pour la définition de la nouvelle politique culturelle au Canada concernant la télévision " , a déclaré le président-directeur général de . " Il devra émerger de ces discussions un genre de cadre de référence nouveau pour définir notre nouvelle politique culturelle canadienne en ce qui a trait à la télévision " , a dit le de . -" Ces investissements répondent aux idées que les médecins de première ligne ont dit être les meilleurs moyens de faire face au système de santé de la Montérégie " , a déclaré Mme . « Ces investissements répondent à l ’ idée selon laquelle , d ’ après les cliniciens de première ligne , il s ’ agit du meilleur moyen de soigner le système de santé de Tasmanie » , a déclaré Mme . -" Ils pensent tout à fait que leur fils a été assassiné " , a déclaré Benjamin , avocat représentant Kenneth et Johnson . « Ils sont absolument certains que leur fils a été assassiné » , a déclaré Benjamin , l' avocat représentant Kenneth et Johnson , sur -" Ils battent des tambours toute la nuit pour nous et continuer leur travail " , a déclaré M. . « Ils frappent sur des tambours toute la nuit pour nous tenir afin que nous puissions continuer à travailler » , a expliqué M. . -" Ils sont en train d' abandonner et de détruire des unités d' urgence qui ont été réformées il y a moins de cinq ans " , explique-t-il . " Ils sont en train de vider et d' des urgences qui ont été rénovées il y a moins de cinq ans " , estime-t-il . -" Ils croient que cela va à l' encontre de la logique , des lois de la physique et du sens commun " , a déclaré M. . « Ils pensent que cela défie toute logique , les lois de la physique ainsi que le sens commun » , a ajouté . -" Il s' agit de personnes riches qui ont besoin d' emprunter à court terme pour des entreprises commerciales ou des investissements , ou des petites entreprises qui ont besoin de connaître leurs besoins en liquidités " , dit Mme . « Ce peut être des personnes aisées qui ont besoin d' emprunter à court terme pour des activités commerciales ou des investissements , ou des petites entreprises qui ont des besoins de trésorerie pour passer un cap difficile » , ajoute Mlle . -" Ils ont envoyé leur fils à l' école avec un et ils l' ont retourné dans un sac corporel " , a-t-il dit . « Ils ont envoyé leur fils à l' école avec un et on le leur a rendu dans un sac » , a-t-il ajouté . -" Ils vont savoir s' ils commencent à fumer à nouveau , ils ne peuvent pas se faire plaisir à cause du vaccin de la nicotine , et ils peuvent les aider à attraper l' habitude " , dit-il . « Ils sauront que s' ils recommencent à fumer , ils n' y prendront aucun plaisir en raison du vaccin , et cela pourra les aider à perdre leur mauvaise habitude » , a-t-il ajouté . -" Ils se préoccupent du risque de grossesse ou même de la façon dont ils manipulent l' hygiène " , dit . « Ils craignent le risque de grossesse ou même la façon dont ils vont gérer les questions d' hygiène » , a déclaré . -" Les choses vont beaucoup mieux aujourd ' hui " , insiste le pharmacien . Maintenant , ça va beaucoup mieux , a souligné la . -" Les choses bougent progressivement " , déclare Nicolas , porte-parole de l' ARS , pour qui " il n' y aura pas de changement quant au service offert " viendra le 4 novembre . " Les choses se font progressivement " , répond Nicolas , porte-parole de l' ARS pour qui , le 4 novembre , " il n' y aura pas de changement en termes de service rendu . " -" Il s' agit d' un sujet depuis de nombreuses années et n' a pas fondamentalement changé depuis le débat actuel " , déclare Kurt , chef de la direction de BASF Chemical . « C' est un sujet récurrent depuis de nombreuses années et rien n' a fondamentalement changé dans le cadre de la discussion actuelle » , déclare Kurt , directeur général de la société productrice de produits chimiques BASF . -" Ce n' est pas seulement une taxe dans un trou noir " , a déclaré Adrian Moore , vice-président de la politique à la Raison . « Il ne s' agit pas simplement d' une taxe s' dans un grand trou noir » , a déclaré Adrian Moore , vice-président de la politique à la fondation Reason . -" Cette loi est draconienne et draconienne et nous l' interdit " , a déclaré Cyrus , directeur de la direction du groupe Capital - , l' une des radios et des journaux les plus indépendants du Kenya . « Cette loi est draconienne et très punitive , et nous la refusons » , a expliqué Cyrus , directeur général de Capital Group – propriétaire de , l' une des stations de radio et sites Internet d' actualité indépendants les plus respectés au Kenya . -" Cette mission était essentielle en raison du risque que le tunnel terroriste soit utilisé pour des attaques contre des civils israéliens " , a déclaré le porte-parole de l' armée , Peter Lerner . " Cette mission était impérative en raison du risque d' utilisation du tunnel terroriste pour des attaques contre des civils israéliens " , a indiqué par ailleurs le porte-parole de l' armée Peter Lerner . -" Ce matin , nous sommes revenus à la frontière pour rentrer dans nos champs , mais les militaires nous ont dit de retourner . " L' AFP a été dénoncée par , entourée de ses neuf enfants , dont un bébé sur son dos . " Ce matin nous avons pour aller dans nos champs , mais les militaires nous ont dit de nous replier " , a indiqué à l' AFP , entourée de ses neuf enfants , dont un nourrisson sur le dos . -" Cette lenteur des essais de vol - bien que conforme au calendrier interne de Bombardier - renforce notre vision selon laquelle le sera porté à Q1 / 15 " , a déclaré M. . « Cette lenteur des essais en vol – bien qu' elle soit apparemment conforme au calendrier interne de Bombardier – renforce notre opinion selon laquelle l' entrée en service sera repoussée au premier trimestre de 2015 » , a déclaré M. . -" Cela changera mon avenir avec ma famille " , a dit l' homme . « Cela va changer mon avenir et celui de ma famille » , a déclaré l' homme . -" Cela nous permettra de développer des interfaces plus conviviales et un plus grand nombre d' options " , a expliqué le président et chef de la direction de , Louis , lors d' une entrevue . " Cela nous permettra de mettre en valeur des interfaces plus conviviales ainsi que des options plus nombreuses " , a expliqué le président et chef de la direction de , Louis , en entrevue . -" Il s' agit de l' option la plus logique à long terme " , a-t-il dit . « Cela s' avère être l' alternative la plus logique à long terme » , a-t-il dit . -" Aujourd ' hui , le peuple égyptien a jeté les bases d' une nouvelle vie , d' une liberté absolue et d' une démocratie véritable " , a déclaré Ted , un ancien artisan des Frères musulmans , un groupe fondamentaliste qui a occupé la majeure partie des 84 dernières années depuis sa création comme une organisation illégale par les gouvernements successifs . « Aujourd' hui , le peuple égyptien a posé les fondements d' une nouvelle vie – la liberté absolue , une véritable démocratie et la stabilité » , a expliqué M. Morsi , un ingénieur de 60 ans formé aux États-Unis , issu des Frères musulmans , un groupe fondamentaliste considéré pendant plus de 84 ans depuis sa formation comme une organisation illégale et durement réprimé par les gouvernements successifs . -" Aujourd ' hui , nous menons des recherches sur la reconnaissance de la réalité et de la résolution en utilisant des capteurs acoustiques , mais nous avons besoin d' un environnement silencieux " , dit , directeur de la recherche en sécurité chez Intel Labs . " Aujourd' hui , nous menons des recherches sur la reconnaissance de la voix ou de la façon de marcher à l' aide de capteurs sonores , mais il faut un environnement silencieux " , indique , directeur de la recherche sur la sécurité aux Intel Labs . -" La question a été débattue lors de ces conversations téléphoniques à propos de l' élaboration de la politique de location commerciale " , a déclaré le commissaire adjoint Anthony . « Le sujet débattu au cours de ces conversations téléphoniques était-il l' élaboration de la politique relative aux baux commerciaux ? » , a demandé le Commissaire adjoint Anthony à M. . -" Nous sommes heureux que la FAA reconnaisse qu' une expérience de voyage agréable n' est pas incompatible avec la sûreté et la sécurité " , a déclaré Roger Chabot , directeur général de l' Association des transporteurs du Canada . « Nous sommes heureux que la FAA reconnaisse qu' une expérience passager agréable n' est pas incompatible avec la sécurité » , a déclaré Roger Dow , Président et directeur de l' U.S. Travel Association . -" Nous avons pu dans des interventions chirurgicales où nous le sang ou des outils pour le Dr . " , remarque O' Malley . « On a pu participer à des opérations pour aider à éponger le sang ou faire passer les instruments au Dr » , a expliqué Mlle O' Malley . -" Nous sommes confiants que le trader a agi seul et que la question a été posée " , a déclaré le Crédit Suisse . « Nous sommes convaincus que le trader a agi seul et que l' affaire a été maîtrisée » , a indiqué un représentant du Crédit Suisse . -" Nous rassemblons des images de caméras de surveillance et les avec des photographies de célébrités qui sont disponibles sur Internet " , explique Dany , directeur du développement des entreprises pour les solutions d' identification biométrique à NEC France . " Nous les images des caméras de surveillance et les comparons aux photos des célébrités qui circulent librement sur Internet " , explique Dany , chargé , chez NEC France , du développement commercial pour les solutions d' identification biométrique . -" Nous ne demandons pas au monde de nous soutenir et de nous soutenir , mais nous avons le droit de demander le monde parce que nous en avons fait partie " , a déclaré le à Washington ce jeudi . " Nous ne disons pas au monde d' être à nos côtés et de nous soutenir , nous avons le droit de le demander au monde parce que nous en faisons partie " , a déclaré ce jeudi à Washington . -" Nous recevons 500 personnes à chaque voyage " , a déclaré . « Nous recevons 500 personnes à chaque visite » , a déclaré M. . -" Nous espérons que ce type de stratégie de vaccination aide actuellement les millions de fumeurs qui ont essayé d' arrêter , tout en mettant à l' épreuve toutes les méthodes disponibles sur le marché , mais qui trouvent leur dépendance à la nicotine suffisamment pour résister à ces approches actuelles " , a déclaré M. Cornell . « Nous espérons vraiment que ce genre de stratégie vaccinale pourra aider des millions de fumeurs qui ont essayé d' arrêter , épuisant toutes les méthodes aujourd' hui disponibles sur le marché , mais découvrent que leur addiction est suffisamment forte pour surmonter ces différentes approches » , a déclaré le Professeur Cornell . -" Nous aspirons à un avenir meilleur , à une nouvelle Égypte et à une autre république " , a déclaré Morsi lors d' une cérémonie solennelle qui s' est déroulée sur la télévision publique . « Nous aspirons à de meilleurs lendemains , une nouvelle Égypte et une deuxième république » , a déclaré le président Morsi au cours de la cérémonie solennelle diffusée en direct sur la télévision publique . -" Nous pouvons confirmer que a été tué lors d' une attaque par drone " , a déclaré un haut responsable des services de sécurité . " Nous pouvons confirmer que a été tué dans un tir de drone " , a dit un haut responsable de la sécurité . -" Nous pouvons maintenant aller de l' avant et nous concentrer sur l' avenir et sur les 90 % des actifs qui constituent une banque solide , et sur la création d' une grande banque pour nos clients et le Royaume-Uni " , a ajouté le nouveau directeur général , Ross , à la presse . " Nous pouvons maintenant aller de l' avant , nous préoccuper de l' avenir et des 90 % des actifs qui constituent une banque vraiment bonne et construire une grande banque pour la clientèle et pour le Royaume Uni " , a dit le nouveau directeur général Ross à la presse . -" Nous avons consolidé ou réduit certaines de nos activités à l' intérieur de l' institution afin de nous y attarder en attendant que les choses se . On a rationalisé ou diminué certaines de nos activités à l' intérieur de l' établissement pour palier , le temps que les choses se . -" Nous n' avons pas invité les rats " , a confié le père de la famille , qui a quitté sa maison et s' est installé dans un hôtel . " Nous n' avons pas convié les rats chez nous " , se défend le père de famille qui a quitté son logement pour s' installer à l' hôtel . -" Nous n' avons pas eu recours à une solution " , a expliqué le Dr . On n' a pas eu recours au dépannage , a expliqué Mme . -" Nous ne voulons pas le faire " , a-t-elle dit . « Nous ne voulons pas faire payer ce prix » , a-t-elle expliqué . -" Nous avons vécu dans la terreur [ du ] , nous sommes traumatisés " , a ajouté l' homme . " Nous avons vécu dans la terreur ( du ) , nous sommes traumatisés " , a ajouté cet homme . -" Nous vivons avec le pour un an et il semblait inimaginable qu' un jour nous soit libéré par l' armée " , a déclaré un habitant de la ville à la frontière avec l' Ouganda . " Nous avons vécu un an avec le et il nous paraissait inimaginable d' être un jour libérés par l' armée " , témoigne un habitant de cette localité limitrophe de l' Ouganda . -" Nous disons depuis environ deux ans et demi que l' idée de forcer les consommateurs à acheter de grands paquets de canaux ne fonctionne pas " , dit M. . " Ça fait à peu près deux ans et demi que nous disons que l' idée de forcer les consommateurs à acheter d' importants forfaits de chaînes , ça ne fonctionne pas " , a affirmé M. . -" Nous avons terminé les pourparlers , les médiateurs ougandais ont simplement besoin d' organiser la signature " , a déclaré Roger , vice-président de la délégation , bien que ces informations ne puissent être confirmées immédiatement par le gouvernement . " Nous avons terminé les pourparlers , il ne reste plus qu' à la médiation ougandaise à organiser la signature " , a affirmé Roger , vice-président de la délégation du , sans que cette information puisse être confirmée immédiatement du côté gouvernemental . -" Nous avons empêché les avions de décoller , les bâtiments ont été et les personnes assassinées , parce que nous avons pu rester informés avant ces attaques " , a déclaré le chef de la diplomatie américaine . " Nous avons de fait empêché que des avions ne tombent , que des immeubles n' explosent et que des gens soient assassinés , parce que nous étions en mesure d' être au courant en amont de ces projets " , a argumenté le patron de la diplomatie américaine . -" Nous ne devons pas leur permettre de se regrouper parce qu' ils ont fait trop souffrir le peuple congolais . Nous ne devons pas les laisser se réorganiser parce qu' ils le peuple congolais depuis trop longtemps . -" Nous devons nous adapter , changer ou résister au changement et à l' échec " , a souligné M. . " Il faut s' adapter et changer ou résister au changement et échouer " , a souligné M. . -" Nous devons être attentifs à la façon dont nous percevons le véritable départ de la puberté " , a déclaré le Dr Lawrence , pédiatrique à l' hôpital des enfants de , à , dans le New Jersey . « Nous devons être prudents quant à la façon dont nous identifions la véritable apparition de la puberté » , a dit le Dr Lawrence , un travaillant à l' hôpital pour enfants à , dans le New Jersey . -" Nous devons avoir un meilleur système " , a-t-il dit . « Nous avons besoin d' un meilleur système » , a-t-il déclaré . -" Nous avons reçu un certain nombre de plaintes de la part des clients , donc nous lui avons demandé de quitter le magasin " , a expliqué un porte-parole de la chaîne de . " Nous avons reçu plusieurs plaintes de clients , donc nous lui avons demandé de quitter le magasin " , a expliqué une porte-parole de la chaîne de supermarchés . -" Nous ne cherchons pas à obtenir des récompenses , à l' exception de la récompense de l' histoire que nous pouvons , à un moment critique , servir cette communauté " , dit-il . « Nous ne cherchons pas de récompense , excepté la récompense de l' histoire car nous devons servir la communauté à un moment critique » , a-t-il ajouté . -" Nous voulons la paix , nous voulons la stabilité " , a déclaré Vanessa de Sousa , chef de la direction d' une entreprise d' investissement . « Nous voulons que la paix revienne ; nous voulons de la stabilité » , a déclaré Vanessa de Sousa , directrice générale d' une société d' investissement . -" Nous aurons une présence accrue dans les campagnes électorales locales et nous de créer une liste . Nous allons être encore plus présents dans le cadre de la campagne municipale et nous allons réfléchir à présenter une liste . -" Nous allons poursuivre les et nous nous de là où ils se cachent parce qu' ils sont des criminels " , dit le colonel Olivier , porte-parole des FARDC , déclaré à Reuters . " Nous allons poursuivre le et l' partout où il se cache parce qu' il s' agit de criminels " , a déclaré à Reuters le colonel Olivier , porte-parole des FARDC . -" Nous serons bientôt et nous attendons une nouvelle maison au même prix que l' ancienne " . Nous sommes bientôt sans le sou et nous attendons un nouveau logement , au même tarif que l' ancien . -" Nous un examen par la CASA d' autoriser l' utilisation d' appareils électroniques parce que nous croyons vraiment que nous pouvons améliorer l' expérience du client maintenant que nous avons un message sur le vol radio ( sans fil ) dans nos avions " , a déclaré un porte-parole . « Nous favorablement une révision de la CASA qui permettrait l' utilisation des appareils électroniques car nous pensons vraiment que cela améliorerait l' expérience client maintenant que nous disposons ( du système de divertissement à bord utilisant la technologie Wi-Fi ) sur nos avions » , a indiqué un porte-parole . -" Ce que je peux vous dire , c' est qu' elle entrait dans le pays " . Ce que je peux vous dire , c' est qu' elle arrivait au pays . -" Quel est le sens de la vie ? " , se . " Quel est le sens de la vie " , . -" Quand je parle de banlieues , je me réfère à tous ceux qui se sentent exclus de la grande famille nationale " . Quand je parle de banlieue , je m' adresse à tous ceux qui sont exclus de la grande famille nationale . -" Où si nous commençons à demander des personnes à ? " demande Gérald , pour qui " le concept d' urgence " est " dangereux " et marque un " progrès médical à l' arrière " . " Où va-t-on si on demande aux gens de s' " , s' interroge Gérald pour qui " le concept d' urgence pas grave " est " dangereux " et marque une " régression médicale " . -" Alors que les livraisons d' avions les plus faibles étaient principalement prévues , nous sommes généralement déçus par la performance de la marge dans le transport " , a déclaré dans une note de client . « Bien que la baisse des livraisons d' avions ait été en grande partie anticipée , nous sommes vraiment déçus par la marge bénéficiaire dans la division Transport » , a déclaré Cameron dans une note aux clients . -" Bien qu' il existe de nombreuses preuves de la part de nombreux mammifères que les deux côtés du cerveau sont utilisés à des fins différentes , la plupart des détails doivent encore être - et les chiens ne font pas exception " , a-t-il dit . « Bien que de nombreux éléments , relevés chez différents mammifères , indiquent que les deux côtés du cerveau sont utilisés à des fins différentes , bon nombre de détails doivent encore être précisés – et les chiens ne font pas exception » , a-t-il déclaré . -" Bien que nous puissions modifier nos programmes de surveillance afin de mieux protéger les droits des Européens , nous devons également tenir compte du fait qu' ils ne sont pas les seuls à être exploités " , a déclaré M. Murphy . « Nous pouvons modifier nos programmes de surveillance afin de mieux protéger les droits des Européens , mais ils doivent aussi comprendre que nous ne sommes pas les seuls dans le monde à faire de l' espionnage » , a déclaré Murphy . -" Avec l' état actuel des connaissances , il pourrait être considéré comme le plus absolu des données biométriques " , confirme Sophie , directrice de la recherche , de l' innovation et des prévisions à la CNIL . " En l' état actuel des connaissances , il peut être considéré comme la donnée biométrique ultime " , confirme Sophie , directrice des études , de l' innovation et de la prospective à la CNIL . -" À partir de 2009 , nos 10 pays sont rentables et plusieurs d' entre eux sont en mesure de comptabiliser les gains d' année en année avant l' intérêt et les contributions à l' impôt , a-t-il dit . « Depuis le début de l' année , les 10 pays dans lesquels nous sommes présents sont rentables , et plusieurs d' entre eux sont en bonne voie de réaliser des résultats record sur l' ensemble de l' exercice avant déduction des intérêts et impôts » , a-t-il déclaré . -" Oui , dans certains cas , cela va trop loin de manière inappropriée " , a déclaré récemment la secrétaire d' État , qui a déjà fait une déclaration sur le scandale international lors d' une tournée à Paris , Londres et Rome la semaine dernière . " Et oui , dans certains cas , c' est allé trop loin de manière inappropriée " , a encore admis le secrétaire d' Etat , qui avait déjà dû s' exprimer sur ce scandale international lors d' une tournée la semaine dernière à Paris , Londres et Rome . -" Vous allez obtenir un meilleur accord " , dit-il . « Vous en gagnant » , a-t-il déclaré . -1 Date limite fixée par l' organisation à Damas pour détruire ou " " tous les équipements de production d' armes chimiques et les équipements permettant de mélanger des produits chimiques en gaz toxique et de purger des munitions . 1er novembre fixée par l' organisation basée à La Haye ordonnant à Damas de détruire ou « rendre inutilisables » toutes les unités de production d' armes chimiques et les machines destinées à mélanger les gaz toxiques et à remplir les munitions . - s Top Design Quick d' entreprise en Grand Style . Le top model de 22 ans ne commence pas sa carrière au cinéma par la petite porte . -300 pages plus tard , il les mots " The End " et chaque page à la main . 300 pages plus tard , il écrivait le mot " FIN " et chaque page au stylo . -66 % des athées ou des non-croyants y aspirent également . 66 % des athées ou non croyants la désirent aussi . -70 pour cent des établissements offrent aujourd' hui un encadrement du rite pour tenir les cérémonies en présence du corps : saluer la congrégation , nommer le défunt , lier les défunts à ce qui est arrivé , appeler la personne qui a l' air , le sens de la mort , organiser un adieu . , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , etc. 70 % des établissements proposent désormais des maîtres de cérémonies , qui procèdent , en présence du corps , au rituel suivant : accueillir l' assemblée , nommer le défunt , le relier aux autres , l' évoquer , donner sens à sa mort , organiser l' adieu . -77 % des Français veulent qu' ils soient proches , qu' ils soient religieux ( 53 % ) ou civils ( 24 % ) . 77 % des Français en veulent une , pour leurs proches , qu' elle soit religieuse ( 53 % ) , ou civile ( 24 % ) . -964 personnes ( dont 855 civils , 65 policiers et 44 soldats ) ont été tuées au cours d' actes de violence en octobre et 1 600 personnes ont été blessées . 964 personnes sont mortes dans les violences en octobre , dont 855 civils , 65 policiers et 44 soldats , et 1 600 personnes ont été blessées . - Un air de documentaire -La fille de , Katie , est née avec des chromosomes humains , mais a un qui s' appelle le syndrome d' insensibilité totale des . La fille du Dr , Katie , est née avec des chromosomes mâles , mais souffre d' un DDS appelé syndrome d' insensibilité complète aux . -La part du vote du diminue depuis 1992 , tandis qu' un autre , le Mouvement démocratique du Mozambique ( ) , qui a été créé par un ancien membre du , devrait améliorer sa représentation aux élections . La part des suffrages recueillie par la ne cesse de baisser depuis 1992 , tandis qu' un nouveau parti , le Mouvement démocratique du Mozambique ( ) qui fut créé par un ancien membre de la , devrait obtenir un plus grand nombre de suffrages aux élections . - , 42 , des informations aux criminels en raison des craintes que sa toxicomanie ne soit source de confusion , 42 ans , fournissait des informations aux criminels car il craignait que sa toxicomanie ne soit révélée au grand jour . - , et ont admis le piratage de téléphone . , et ont admis avoir effectué du piratage téléphonique . - , par exemple , n' émet que des transmissions minimes lorsqu' elles tournent une page . Les lecteurs de livres électroniques , par exemple , émettent seulement des transmissions minimales lorsqu' on tourne une page . - , Les grands exploits et les prouesses La comédie musicale , cette bête de scène - , qui milite pour des réformes électorales , a déjà déclaré qu' il les élections municipales qui devaient avoir lieu en novembre . La , qui est en faveur de réformes électorales , avait déjà dit qu' elle les élections municipales devant avoir lieu en novembre . - , qui lutte pour l' indépendance du Turkestan oriental - l' ancien nom du Xinjiang chinois - a été classé par l' ONU en 2002 comme l' une des organisations affiliées à Al-Qaida . L' , qui déclare se battre pour l' indépendance du Turkestan oriental , ancien nom du Xinjiang chinois , a été classée par l' ONU en 2002 parmi les organisations affiliées à Al-Qaïda . - , qui a traduit de et écrit le Livre des et des British Folk Stories , a déclaré : " Vous avez peut-être des intentions , mais vous faites mieux de les garder loin de la réalité . , qui a traduit de l' anglo-saxon et a écrit The Penguin Book of et British Folk Tales , a déclaré : « Vous avez peut-être de bonnes intentions , mais vous préférez les garder hors de vue . » - , qui supervise la mise en oeuvre de la Care Act , dit que le lancement du marché en ligne est devenu " " depuis octobre . Mme , qui supervise la mise en œuvre de l' Care Act , a indiqué que le lancement en octobre du marché en ligne avait échoué « lamentablement » . - : d' eau de source Explosion de l' Assemblée Explosion d' une maison à : un chauffe-eau défectueux à l' origine de l' explosion -La mort de a été annoncée plusieurs fois par le passé . La mort d' a été annoncée à plusieurs reprises déjà par le passé . - , 42 ans , l' information au criminel du chef de guerre Neil , 42 ans , donnait des informations au seigneur du crime Neil - , 42 , des informations confidentielles pour le seigneur de crime Neil à propos des enquêtes sur ses activités de trafic de drogue après avoir été compromise par son habitude de cocaïne . , 42 ans , donnait des informations sensibles au seigneur du crime Neil concernant des enquêtes sur ses activités liées au trafic de drogue après s' être compromis à cause de son habitude de consommer de la cocaïne . - , attaché régional à 50 ans , prétend aujourd' hui être " 100 % apolitique " , bien qu' il n' ait pas toujours été neutre . , 50 ans , attaché territorial , se revendique aujourd' hui comme " 100 % apolitique " même s' il n' a pas toujours été neutre . - 9 States of Rights reproche à 9 États américains de violer les droits homosexuels - prétend qu' il y a déjà sept noms sur la liste de candidats en France continentale et en Martinique . assure avoir déjà sept têtes de liste en France métropolitaine et en Martinique . - de l' Université de Montréal sera le directeur scientifique . , de l' Université de Montréal , en sera la directrice scientifique . - a été blessé dans un accident de voiture de 1981 près de Yaoundé , capitale du Cameroun . a été blessé dans un accident de voiture en 1981 près de Yaoundé , la capitale du Cameroun . -L' Église baptiste de , où il a été ministre , a déclaré qu' il était profondément attristé par les nouvelles . L' église baptiste de , où le pasteur , se dit profondément par cette nouvelle . - de la résistance du Gouvernement confiant dans la longévité de son gouvernement -Les enquêteurs du de comté de ont conclu que Johnson était mort dans un accident de type , mais les problèmes de la famille de ces derniers le sont . Les enquêteurs du shérif du Comté de ont conclu que Johnson était décédé dans des circonstances insolites , mais la famille du jeune garçon de 17 ans conteste ces conclusions . - due aux causes des À La Seyne A La Seyne , le relogement pour cause de rats tourne à la polémique -La est préoccupée par les conséquences pour les futures générations si elle est adoptée . S' il est adopté , s' inquiète des conséquences sur les prochaines générations . - du leader pakistanais des talibans Le chef des taliban pakistanais tué par un drone -Le député de , Clive Palmer , affirme que le Premier ministre Tony Abbott a un conflit d' intérêts concernant son régime de congé parental parce que ses filles pourraient s' en sortir et y gagner . Le député milliardaire Clive Palmer avance que le Premier ministre Tony Abbott a un conflit d' intérêt à propos du système de congé parental parce que ses filles pourraient être enceintes et profiter de ce système . - Mohamed Morsi a promis une " nouvelle Egypte " , car il a prêté serment pour devenir le premier président librement élu du pays , succédant à Hosni Moubarak qui avait été chassé il y a 16 mois . L' islamiste Mohamed Morsi a promis l' émergence d' une « nouvelle Égypte » lorsqu' il a prêté le serment d' investiture à la présidence , devenant ainsi le premier président librement élu et succédant à Hosni Moubarak qui fut chassé du pouvoir 16 mois plus tôt . - le portique de démonté -Théâtre à Narbonne le jeudi soir Une salle du de Narbonne évacuée jeudi soir - : Les demandes adressées à Brigitte par téléphone au Feux : un député demande la démission de Brigitte - sur l' avenir de son pays , elle a changé sa marque professionnelle pour un T-shirt avec " demande de sécurité " en portugais lorsqu' elle s' est jointe à la foule sur la Place de l' Indépendance de la capitale , jeudi . pour l' avenir de son pays , elle a échangé ses vêtements de travail contre un tee-shirt portant la mention « nous voulons la sécurité » en portugais avant de rejoindre la foule sur la place de l' Indépendance de la capitale jeudi . - al-Maliki demande l' aide des États-Unis . al-Maliki souhaite une aide des Etats-Unis . - al-Maliki s' est adressé à l' Institut américain de la paix , une institution indépendante créée par le Congrès . al-Maliki s' exprimait à l' Institut des Etats-Unis pour la paix , une institution indépendante créée par le Congrès . - et Brooks ont refusé de avec d' autres pour des téléphones entre le 3 octobre 2000 et le 9 août 2006 . et Brooks nient avoir avec d' autres pour des téléphones entre le 3 octobre 2000 et le 9 août 2006 . -Les et les excès de toutes sortes sont les mots clés de cette histoire folle . et excès en tout genre sont les maîtres mots de cette folle histoire . - et son équipe passent presque tous les week-ends à voir des centaines de patients Dr et son équipe passent presque tous leurs week-ends à voir des centaines de patients - apparaît , une autre caractéristique du travail de Chouinard , qui porte le nez rouge , dans les parades très des hommes et des femmes . Des apparaissent , aussi familiers de Chouinard , arborant le nez rouge , dans des parades amoureuses hommes-femmes très . - comme le " tambour de roue " de , son prestige personnel s' est amélioré depuis sa victoire et la remise d' un discours de la présidence de Singapour qui lui a présenté le candidat non seulement pour les islamistes , mais aussi pour tous ceux qui veulent achever le travail du soulèvement de 2011 contre le régime autoritaire de Moubarak . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; » Considéré comme la « roue de secours » peu charismatique des Frères musulmans , son prestige personnel s' est considérablement renforcé depuis sa victoire et son discours de vendredi au cours duquel il a essayé de se présenter comme le candidat non seulement des mais de tous ceux qui veulent finir le travail entamé lors du soulèvement de 2011 contre l' autoritaire président Moubarak . - attaque un de missiles russes dans la ville portuaire de , un officiel . Des avions de combat attaquent un stock de missiles russes dans la ville portuaire de , indique un responsable . - par son travail , les universitaires s' efforcent depuis longtemps de définir et de placer Pierre Gélinas . , les universitaires ont longtemps à définir et situer Pierre Nora . - par les libéraux et les de la Chambre des communes , M. Nicholson a assuré que , avant leur départ , les militaires avaient fait l' objet d' un plan de transition en collaboration avec leurs supérieurs . par les libéraux et les à la Chambre des communes , M. Nicholson a assuré qu' avant leur libération , les membres de l' armée suivaient un plan de transition en collaboration avec leurs supérieurs . - dit qu' il s' agit d' un frais d' utilisateur . appelle cela des frais d' utilisation au kilométrage . - ne peut pas excuser les problèmes du site , les attaques par drone L' n' excuse pas les difficultés du site , ni les attaques de drones -Les appels téléphoniques par téléphone cellulaire continueront également d' être interdits . Les téléphones portables à bord continueront également à être interdits . -Les enfants posent un dilemme éthique . Les enfants posent un dilemme éthique . - soutient que la prémisse qui retient le plus de poids est l' épidémie d' obésité . affirme que l' hypothèse la plus crédible est celle de l' épidémie d' obésité . - a refusé des actes répréhensibles et a déclaré : " Si je suis un avocat corrompu , que je n' ai pas , et que je voulais fournir des renseignements à M. Scarborough , je n' pas 15 heures , je vais le faire immédiatement " . a nié tout agissement illégal et a déclaré : « Si j' étais un avocat corrompu , ce que je ne suis pas , et si j' avais voulu fournir des informations à M. Scarborough , je n' aurais pas attendu 15 heures , je l' aurais fait immédiatement . » - nie être influencée par les systèmes d' observation de la Terre ( AIS ) nie être influencé par Obeid - Tunnel de la drogue avec son propre chemin de fer Découverte d' un tunnel de la drogue entre les États-Unis et le Mexique disposant de sa propre voie ferrée - combler cette lacune . Les spectacles musicaux comblent ce ­ manque . - de sa position de chef des services d' urgence au début du mois de juillet pour s' opposer au projet d' assainissement , il se qualifie de " " face à une " décision cynique " du " personnel de l' administration médicale technique " . de son poste de chef du début juillet pour avoir mené la contre le projet de réorganisation , il se définit comme un " lanceur d' alerte " face à une décision " cynique " prise par " la " . - se maintient pendant des décennies sous quelque forme que ce soit , mais les arrêts enregistrés ont considérablement augmenté sous l' administration du maire indépendant de Bloomberg Michael à un niveau élevé en 2011 de , majoritairement des hommes noirs et hispaniques . Ce système de contrôle et de fouille existe depuis des décennies sous une certaine forme , mais le nombre de contrôles enregistrés a augmenté de façon spectaculaire sous l' administration du maire indépendant Michael Bloomberg , pour atteindre son plus haut niveau historique en 2011 avec 684 330 contrôles qui concernaient principalement des hommes noirs et d' origine hispanique . -Les se sont adaptés à ce changement de société . Les se sont adaptés à cette évolution de la société . -Les ont pris part à des groupes environnementaux en lobbying afin de permettre au gouvernement d' utiliser les petites cases pour suivre les milles que vous avez tirés et , éventuellement , où vous les conduisez - puis utilisez l' information pour rédiger un projet de loi . Les ont rejoint des groupes écologistes pour faire pression afin que le gouvernement utilise les petites boîtes pour garder la trace des miles que vous parcourez , et éventuellement de la route sur laquelle vous , puis utiliser les informations pour rédiger un projet de loi fiscal . -Les ont considéré ce travail comme un " gigantesque " marché des puces , présenté par un commentateur intelligent et animé par un oeil attentif , mais qui est un " " . Les langues ont pu y voir un gigantesque marché aux puces , présenté par un commentateur intelligent , doté d' un vif coup d' oeil et d' érudition , mais . - en France , elle raconte l' histoire sinistre de Suzanne , un jeune envoyé à un couvent par sa famille . Se déroulant dans la France des années 1760 , elle raconte la sombre histoire de Suzanne , une jeune envoyée dans un couvent par sa famille . - dans Le Children avec Gun à l' occasion de l' Halloween Un accueille des enfants avec un fusil au pour Halloween - est un menteur et un traître . " est un menteur et un traître . - fait appel de la sentence et a bénéficié d' une libération sous caution de 50 000 baht . a fait appel du jugement et a été mise en liberté après le versement d' une caution de 50 000 bahts . -La est prudente en ce qui concerne les médicaments , mais reconnaît que " la suppression de la puberté peut atténuer les problèmes de comportement et les sentiments des filles d' être différents des pairs " . est prudent avec les médicaments mais reconnaît que « le fait de contrôler la puberté peut atténuer certains problèmes de comportement et la sensation pour les jeunes filles d' être différentes des autres » . -Les vendeurs à jet d' encre ont même des mesures de bande pour vérifier la mise en place de dispositifs concurrents . Les vendeurs d' avions qui font le tour du monde emportent même avec eux un mètre pour vérifier les configurations concurrentes . -Les bijoux ne sont pas simplement une solution rapide à l' argent de terre - 1 300 dollars de S dans le cas de Mme - mais sont à peu près aussi bon marché que les prêts bancaires non garantis . Mettre en gage des bijoux n' est pas seulement un moyen rapide d' obtenir de l' argent liquide – 1 300 $ de Singapour dans le cas de Mme – mais c' est presque aussi bon marché que les prêts bancaires non garantis . - avocat qui a M. Big au sujet de l' enquête de la police est emprisonné L' avocat qui a informé un parrain de la drogue d' une enquête de police a été emprisonné -Selon des sources juridiques non officielles , était représentée par un avocat local à Phuket mais elle a prévenu que l' appel pouvait porter le jugement à une peine de prison de deux ans au maximum et l' obliger à se rendre dans une prison pour adultes . Des sources judiciaires thaïlandaises ont indiqué que était représentée par un avocat local de Phuket , mais ont prévenu que l' appel pourrait conduire à un alourdissement de la peine pouvant aller jusqu' à deux ans d' emprisonnement et une incarcération dans une prison pour adultes . -La police d' a fait aujourd' hui l' objet d' un avis d' extradition concernant le fondateur de l' organisation , Julian , qui s' est réfugié dans l' ambassade de l' Équateur à Londres et a demandé l' asile . La police britannique a aujourd' hui délivré un mandat d' extradition à l' encontre du fondateur de WikiLeaks Julian , qui s' est réfugié à l' Ambassade de l' Équateur à Londres et a demandé l' asile politique . - a réuni plus de 1.000 chefs d' entreprise , représentant largement les secteurs , secteurs et régions de la taille de l' entreprise . a interrogé plus de 1 000 chefs d' entreprise , globalement représentatifs de toutes les tailles d' entreprises , des secteurs et régions de Grande-Bretagne . - a déclaré que son organisation appuie la loi australienne parce qu' elle " permet aux adultes de choisir d' être reconnus dans un troisième sexe " . a déclaré que son organisation soutenait la loi australienne car « elle autorise les adultes à choisir d' être reconnu sous l ’ appellation troisième sexe » . -M. a dit que certains médecins sont encore en train de " pratiquer ce modèle " . a déclaré que certains médecins « répètent toujours ce modèle » . - dit que le gouvernement a lancé les derniers affrontements en lançant une attaque contre ses membres dans la province de Sofala , bastion de , le 17 octobre . La déclare que le gouvernement est à l' origine des récents affrontements et qu' il a lancé une attaque contre ses membres dans la province de Sofala , un bastion traditionnel de la , le 17 octobre . - à Singapour comme classe moyenne se sentant Les prêteurs sur gage de Singapour tirent profit des classes moyennes contraintes à se serrer la ceinture -Des histoires sur les contrôles des passeports à et une ligne de fer le long du Mur d' . Et voilà qu' on entend des histoires de contrôle des passeports à et d' une frontière de fils barbelés le long du mur d' . - a dit au Sunday Telegraph à Londres que toute la famille était " extrêmement préoccupée par le sort de sa fille et qu' elle a voulu la retrouver en Australie . a déclaré au Sunday Telegraph de Sydney que toute la famille était « extrêmement préoccupée » du bien-être de sa fille et voulait qu' elle rentre en Australie . - a utilisé le téléphone pirate pour vérifier la précision . a utilisé le piratage téléphonique pour vérifier un tuyau - voulait " avertir la communauté internationale que les choses ne fonctionnaient pas bien au Mozambique " , a déclaré M. . La voulait « avertir la communauté internationale que les choses ne vont pas bien au Mozambique » , a indiqué M. . - a été arrêté plus tard , mais a nié avoir consommé de la cocaïne et a déclaré qu' il s' était adressé au trafiquant présumé parce qu' il était client et que ses discussions avaient fait l' objet d' un " privilège juridique " . a été arrêté plus tard mais a nié consommer de la cocaïne . Il a déclaré qu' il avait parlé au trafiquant de drogue présumé car c' était son client et a fait valoir que leurs discussions étaient assujetties à un « privilège juridique » . - était autrefois un mouvement rebelle notoire , initialement soutenu par la Rhodésie du Sud et maintenant le gouvernement de l' apartheid en Afrique du Sud dans le cadre des efforts visant à déstabiliser le gouvernement indépendant du pays . La était autrefois un mouvement rebelle célèbre , soutenu au départ par la Rhodésie gouvernée par des blancs puis le gouvernement d' apartheid d' Afrique du Sud dans le cadre des efforts pour déstabiliser le gouvernement indépendant du pays . - a été condamné à purger la peine de quinze jours d' emprisonnement dans un centre de détention de courte durée à la périphérie de Phuket , au lieu d' une prison pour adultes . a été condamnée à une peine d' emprisonnement de 15 jours dans un centre de détention de faible sécurité dans la périphérie de Phuket plutôt que dans une prison pour femmes . -Des ont été signalés dans les collines proches de , la dernière ville des rebelles à tomber cette semaine , et autour de , une colline où la rébellion a commencé en 2012 . Des accrochages sont signalés dans les collines qui dominent , la dernière localité aux mains des insurgés à être tombée cette semaine , ainsi qu' aux alentours de , une colline où la rébellion du a vu le jour en 2012 . -Un test " d' authenticité " pour les travailleurs étrangers sur 457 visas est actuellement examiné par le gouvernement alors qu' il envisage d' étendre la répression . Le gouvernement étudie actuellement la possibilité de mettre en place un test « d' authenticité » destiné aux travailleurs étrangers en possession d' un visa 457 alors qu' il envisage d' étendre les mesures de répression . -Une " petite quantité de dommages " , confinée à l' espace de la chambre du réacteur , s' est ensuite produite sur le de la flotte française . Une " petite avarie " , circonscrite à l' espace de confinement du réacteur , s' était alors produite sur le navire amiral de la flotte française . -Une femme de 21 ans de Sydney , condamnée à une peine de 15 jours de prison à Phuket pour avoir faussement déclaré qu' elle avait été agressée par un chauffeur de taxi , a plaidé pour le verdict et a obtenu la libération sous caution . Une de Sydney de 21 ans , condamnée à 15 jours d' emprisonnement à Phuket pour avoir prétendu à tort avoir été agressée par un chauffeur de taxi , a fait appel du jugement et a été mise en liberté sous caution . -Un homme de 38 ans qui a pris un enfant à l' école de Gabrielle Roy , à Surrey , doit faire face à six accusations , selon la Gendarmerie royale du Canada . L' homme de 37 ans qui a pris un enfant en otage à l' école Gabrielle Roy de Surrey fait face à six chefs d' accusation , selon la Gendarmerie royale du Canada . -Un entrepreneur britannique a créé la première de glace à l' échelle planétaire , utilisant les méduses . Un entrepreneur britannique a créé la première glace fluorescente au monde à base de . -Un journal chinois qui lance un appel à la presse pour la libération d' un journaliste accusé de diffamation doit être révisé , un organisme de presse judiciaire . Un journal chinois qui a lancé un appel en première page pour la libération d' un journaliste accusé de diffamation va être remanié , déclare un régulateur de la presse . -Une source du Hamas a expliqué que les combats avaient éclaté dans la région après que des chars israéliens eurent franchi la frontière et avaient été pris pour cible par des tirs de mortier palestiniens . Une source issue du Hamas a expliqué que des combats ont éclaté dans cette région , car des chars israéliens ont franchi la frontière et ont été visé par tes tirs de mortiers palestiniens . -Une équipe de sauvetage technique de la station de sapeurs-pompiers de s' est rendue sur la scène et a pour secourir Ruby et a utilisé un pour le transporter sur la falaise . Une équipe de secours technique de la caserne de pompiers de s' est ruée sur les lieux et est descendue en rappel pour sauver Ruby en utilisant une civière pour animaux pour la transporter jusqu' en haut de la falaise . -Un responsable américain a confirmé qu' il y avait eu une " grève israélienne " , mais n' a pas donné de détails sur l' objectif . Un responsable américain a confirmé une " frappe israélienne " mais n' a pas donné de détails sur la cible . -Une meilleure gestion des affaires britanniques est possible , et il est de plus en plus nécessaire que la zone euro s' ouvre sur la voie d' une union économique et fiscale plus étroite . Il est possible , et de plus en plus nécessaire , de mieux représenter les intérêts des entreprises britanniques , puisque la zone euro s' engage sur la voie d' une union économique et fiscale plus étroite . -Un grand changement se produit dans l' informatique des nuages où les dirigeants européens sont plus conscients que les données stockées aux États-Unis sont soumises à cette compétence et donc potentiellement vulnérables . Un gros changement est en train de se produire dans l' informatique en nuage puisque les responsables européens ont pris conscience du fait que les données conservées aux États-Unis sont régies en vertu des lois de cette juridiction et donc potentiellement vulnérables . -Un homme qui a été abattu à terre après avoir été épuisé par une centrale de forage en mer du Nord a été renvoyé dans la nature . Un oiseau , après avoir été trouvé épuisé sur une plate-forme pétrolière en mer du Nord , a été relâché dans la nature . -Une boîte noire dans votre voiture ? Une boîte noire dans votre voiture ? -Un membre du conseil d' administration d' une société allemande de premier plan a déclaré qu' en ce qui concerne l' espionnage économique , " les Français sont les pires " . Un membre du conseil d' administration d' une société allemande renommée estimait que lorsqu' il s' agit d' espionnage économique , « les Français sont les pires » . -Un chargement de missiles sol-air sol-air à courte portée a été ciblé et détruit . Une cargaison de missiles sol-air de courte portée aurait été ciblée et détruite . -Un cas de rage dans une chienne illégalement importée de Gambie a été signalé en 2008 . Un cas de rage sur une chienne importée illégalement de Gambie avait été enregistré en 2008 . -Un enfant de cinq ans a été gravement blessé à la suite d' un accident survenu à Disneyland Paris . Un enfant de cinq ans a été grièvement blessé à la suite d' un accident sur une attraction à Disneyland Paris . -Un clown revient sur la scène alors que le public quitte . Un clown revient en scène quand le public quitte . -Une combinaison de circonstances nous a mis dans une situation difficile l' année dernière " . Il y a un concours de circonstances qui a fait qu' on était dans une mauvaise situation l' année passée . -Un cours combiné de littérature anglaise et de cours de langue sera mis en échec . Un cours combinant littérature et langue anglaises sera supprimé . -Un ouvrier de la construction est mort après avoir été enseveli sous des dizaines de blocs de béton à Montréal jeudi après-midi . Un travailleur de la construction est mort enseveli sous des dizaines de blocs de béton , jeudi , à Montréal . -Un différend contractuel entre Gazprom et ses homologues en Ukraine , qui abrite la plus grande partie du gaz russe pour l' Europe , ajoute une couche de risque aux itinéraires conventionnels , disent les responsables . Un différend contractuel entre Gazprom et son homologue d ’ Ukraine , pays par lequel transite la plus grande partie du gaz russe destiné à l' Europe , accroît le niveau de risque par rapport aux voies conventionnelles , ajoute le directeur . -Un festin pour les fans Un festin pour ses fans -Une cour d' appel fédérale du Texas a réintroduit certaines restrictions sur le droit à l' interruption volontaire de grossesse , qui avait été bloquée par un juge de ce procès cette semaine . Une cour fédérale d' appel a rétabli au Texas certaines restrictions au droit d' interruption volontaire de grossesse qui avaient été bloquées par un juge de première instance cette semaine . -Une cour d' appel fédérale conteste la décision d' un juge selon laquelle la tactique controversée de la constitue une discrimination à l' encontre des minorités . Une cour d' appel fédérale bloque la décision d' une juge selon laquelle la tactique controversée du est discriminatoire à l' égard des minorités . -Un tribunal d' appel fédéral a suspendu l' ordonnance d' un juge exigeant des modifications du programme de police du de New York et a retiré le juge de l' affaire . Une cour d' appel fédérale a bloqué jeudi la décision d' une juge exigeant des changements dans le programme de contrôle et de fouille du département de police de New York et l' a du dossier . -Un combattant de l' aile armée du Hamas a été tué ce soir et un autre blessé par des tirs israéliens dans la bande de Gaza , des sources médicales et de sécurité dans le rapport de Gaza . Un combattant de la branche armée du Hamas a été tué ce soir et un autre blessé par un tir de char israélien dans la bande de Gaza , a-t-on appris de sources médicales et sécuritaires à Gaza . -Un incendie a causé de graves dommages à un magasin du district de LaSalle , à Montréal , le jeudi soir . Un incendie a gravement endommagé un commerce de l' arrondissement de LaSalle , à Montréal , dans la nuit de jeudi à vendredi . -Un incendie qui a commencé à midi le vendredi dans le Vieux-Québec a été rapidement maîtrisé . Un incendie qui s' est déclenché vendredi midi dans le Vieux-Québec a rapidement été maîtrisé . -Un formulaire qui , sans erreur , est tout aussi restrictif : Mais ne nous y trompons pas , tout aussi contraignante . -Un ancien de la province de , dans le nord-est de la Chine , a été condamné à des peines d' emprisonnement à perpétuité pour corruption le vendredi . Un ancien de la province de , dans le nord-est de la Chine , a été condamné vendredi à la prison à vie pour corruption . - , ancien ministre des Affaires étrangères et du Commerce international ( ) , a été condamné à une peine de trois ans de prison ferme et à une peine de trois ans d' emprisonnement . Un document de travail du gouvernement a été dévoilé aujourd' hui alors que la députée travailliste dénonçait le discours gouvernemental sur les travailleurs étrangers , en disant qu' il pourrait offenser les voisins de l' Australie . -Une maison historique construite en 1850 dans le vieux Québec a été sous les flammes le vendredi midi . Une maison historique construite en 1850 été la proie des flammes dans le Vieux-Québec vendredi midi . -Une action en justice a été intentée en 2004 par quatre hommes , par toutes les minorités , et est devenue un recours collectif . Une action judiciaire a été introduite en 2004 par quatre hommes , tous provenant de minorités , et s ’ est transformée en action collective . -Une fuite de vapeur radioactive , détectée sur l' un des deux nucléaires sur le à la mi-octobre quand le navire était en mer , " n' était pas une menace pour les marins " , mais ce n' est pas une mince affaire pour la Marine . La fuite de vapeur radioactive détectée sur l' une des deux nucléaires du , à la mi-octobre et alors que le bateau se trouvait en mer , était " sans danger pour les marins " , mais n' est pas sans conséquence pour la Marine . -Une lettre de M. , présentée aux médias à Berlin vendredi soir , a déclaré : " Bien que les résultats de mes efforts positifs aient été positifs , mon gouvernement continue de traiter l' opposition comme une farce et tend à criminaliser le discours politique à l' égard de crimes graves qui ne font pas l' objet de défense " . Dans une lettre que le député a présentée aux médias à Berlin vendredi , M. disait : « Bien que le résultat de mes efforts ait été de toute évidence positif , mon gouvernement continue de traiter la différence d' opinion comme une fuite et cherche à pénaliser le discours politique avec des accusations contre lesquelles il est impossible de se défendre . » -Un peu plus tard , sur la route menant à , la frontière avec l' Ouganda , des soldats blessés par des civils ont reçu un lance-roquettes sur un nouveau camion appartenant aux FARDC , destiné à prendre le relais d' un autre appareil qui les positions des dans les collines . Un peu plus tôt , sur la route menant à , avec l' Ouganda , des militaires aidés de civils un lance-roquettes multiple monté sur un camion flambant neuf des FARDC , devant assurer la relève d' un autre engin les positions du sur les collines . -Une petite fille a été blessée par balle et trois autres ont été blessées , l' un d' entre eux ayant été gravement touché et évacué par les FARDC , le témoin cité par les AFP . Une fillette a été blessée par balle , et trois autres personnes ont été blessées , dont une grièvement , évacuée par les FARDC , a ajouté ce témoin cité par l' AFP . -Un commandant local des Brigades , Abou Hamad , et un autre officier de l' aile armée du Hamas , , ont été tués par un obus de char au cours d' une incursion menée par l' armée israélienne à Khan , dans le sud de la bande de Gaza , selon des sources médicales locales . Un commandant local des Brigades , Khaled Abou Bakr , et un autre cadre de la branche armée du Hamas , , ont été tués par un tir d' obus de char lors d' une incursion de l' armée israélienne à l' est de Khan , dans le sud de la bande de Gaza , selon des sources médicales locales . -La majorité des actionnaires d' Oracle ont voté contre le projet de rémunération pour le fondateur et directeur exécutif Larry Ellison jeudi en raison de la performance financière de son groupe . Une majorité d' actionnaires d' Oracle ont voté jeudi contre la proposition de rémunération pour le fondateur et directeur général Larry Ellison , au vu des performances financières de son groupe . -Un homme vêtu d' un uniforme de l' uniforme nazi et d' un a été appelé à déposer un camion anglais après avoir reçu des clients du directeur de la boutique , qui avait appelé la police , selon le magasin le vendredi . Un homme vêtu d' un uniforme nazi barré d' un avec la croix a été prié de quitter un supermarché anglais , des clients s' étant plaints auprès de la direction de l' établissement qui a appelé la police , a indiqué le magasin vendredi . -Un homme de Cambridge a revendiqué la responsabilité de l' acte sur son compte Twitter , où il a posté des photos d' Adolf Hitler . Un homme de Cambridge a revendiqué la responsabilité de cet acte sur son compte Twitter , où il a posté des images d' Adolf Hitler . -Un homme a ouvert le feu à 10 heures , heure locale . Il était 10h00 du matin heure locale quand un homme a ouvert le feu . -Un ministre responsable des enfants dans la région de Halifax , en Nouvelle-Écosse , a été accusé de pornographie juvénile . Un pasteur qui s' occupait des jeunes , dans la région d' Halifax en Nouvelle-Écosse , est accusé de pornographie juvénile . -Une progression plus rapide peut exiger des tests d' un pour écarter de graves problèmes tels que les tumeurs ou les kystes . Une évolution plus rapide peut justifier des tests réalisés par un pour écarter des problèmes graves comme des tumeurs ou des kystes . -Un automobiliste a été condamné à payer une amende de 1,000 livres pour la conduite jusqu' à ( / h ) avec une boisson chaude équilibrée entre ses jambes . Un automobiliste s' est vu infliger une amende de 1 000 £ pour avoir roulé à 210 km / h avec une boisson chaude posée en équilibre entre ses jambes . -Une nouvelle action de revendication collective a été introduite contre les frères de la Congrégation de la Croix de , qui auraient fait l' objet d' agressions sexuelles , ce jour-là , à l' Oratoire Saint-Joseph et à plusieurs orphelinats , collèges et écoles . Une nouvelle demande de recours collectif a été déposée contre les frères Sainte-Croix , concernant des agressions sexuelles qui se seraient déroulées cette fois à l' oratoire Saint-Joseph ainsi que dans plusieurs orphelinats , collèges et écoles . -En février et en août 2007 , M. et M. , M. Steve et M. , ont fait état d' un communiqué de vote , un qui était arrivé au ministère des ports après avoir travaillé sous la responsabilité de M. Valverde dans le secteur de la pêche . La transcription de conversations téléphoniques présentée à l' a montré des appels passés en août et septembre 2007 entre M. Obeid , M. et Steve Dunn , un haut fonctionnaire qui était arrivé au ministère des Ports après avoir travaillé sous les ordres de M Obeid au ministère des Pêches . -Une audition plénière sur la question doit avoir lieu en janvier . Une audience plénière sur cette question est prévue pour le mois de janvier . -Une enquête préliminaire a révélé que le s New News Group avait publié plusieurs faux reportages sur la société cotée au cours de la période de septembre 2012 à août 2013 . Une enquête préliminaire a montré que le New Express , appartenant au groupe News , avait publié plusieurs articles erronés sur la société cotée entre septembre 2012 et août 2013 . -Un chiot a eu une chance de se sauver après que les équipes de pompiers ont été amenées à la afin de lui assurer une sécurité lorsqu' elle s' est retrouvée coincée sur une falaise précaire . Un chiot a eu la chance d' échapper au pire après que les pompiers ont été appelés pour le récupérer en toute sécurité alors qu' il était perché à 15 m au-dessus du sol sur la saillie instable d' une falaise . -Un rapport présenté à la Commission européenne en 2011 a décrit les comme appartenant à des personnes ou transgenres , car leur statut n' est pas spécifiquement lié à la condition de la femme , mais se rapporte à leur maquillage biologique , qui n' est pas exclusivement masculin ou féminin , mais il est également familier d' une manière ou d' une autre seulement clairement définie . Un rapport soumis à la Commission européenne en 2011 décrivait les personnes comme différentes des personnes transsexuelles ou transgenres , puisque leur statut n' est pas lié au sexe mais plutôt à leur constitution biologique , qui n' est ni exclusivement celle d' un homme ni exclusivement celle d' une femme , mais est en général les deux en même temps et pas clairement définie comme étant l' une ou l' autre . -Un représentant du Bureau du shérif du comté de n' a pas été immédiatement disponible pour commenter lorsqu' il a été contacté jeudi . Le représentant du bureau du shérif du comté de n' était pas disponible pour nous donner son opinion lorsque nous l' avons contacté jeudi . -Un accident de la route sur le chemin de Bruxelles , près de , en direction de Waterloo , à , le vendredi soir , a laissé une personne grièvement blessée . Un accident de la route a fait un blessé grave , vendredi matin vers 5h30 , sur le ring intérieur de Bruxelles à hauteur de , en direction de Waterloo . -Une ligne de conduite a été mise en place entre les leaders d' avions sur la taille des sièges de sur les vols à grande distance , donnant le ton pour une confrontation acharnée lors du salon aéronautique de Dubaï en 2009 . La dispute fait rage entre les grands constructeurs aéronautiques à propos de la largeur des sièges de la classe touriste sur les vols long-courriers , ouvrant la voie à une confrontation amère lors du salon aéronautique de Dubaï qui a lieu de mois-ci . -Un scandale concernant la présence de viande de cheval dans les repas préparés a été relevé en Europe au début de l' année , après les tests effectués en Irlande . Un scandale sur la présence de viande de cheval dans des plats cuisinés a éclaté en Europe au début de l' année , à la suite de tests effectués en Irlande . -Cependant , une deuxième autopsie a fait état d' une autre cause de décès , selon Moore . Toutefois , une seconde autopsie a identifié une autre cause de décès , selon Moore . -Un négociateur principal a dit que malgré le volume important de transactions boursières quotidiennes , la dévaluation de la liquidité entre les différentes plates-formes de négociation et l' utilisation croissante des banques par leurs propres plateformes internes , " on peut commencer à avoir un impact sur le marché à des prix très faibles des petits billets " . Un trader senior a déclaré qu' en dépit du volume élevé des transactions de change quotidiennes , la fragmentation de la liquidité entre les plates-formes de gestion des échanges et l' utilisation de plus en plus fréquente par les banques de leurs propres plates-formes internes « explique qu' il est possible de commencer à avoir un impact sur le marché à peu de frais » . -Un accident grave s' est produit entre une moto et une voiture sur la rue à à 15 heures le vendredi . Vendredi , vers 15 heures , une violente collision entre une moto et une voiture s' est produite rue de à . -Un mur de fondation commun avait été construit ces derniers jours . Un mur de fondation venait d' être érigé au cours des derniers jours . -Une technique similaire est appliquée à et à , en compagnie de la société , en partenariat avec les banques et les grands détaillants : - au paiement d' une carte dans la boutique , le client ne passe pas un code - il place un doigt dans le lecteur , qui scanne les veines . Une technique similaire est testée à et à Angoulême par la société Natural Security en partenariat avec des banques et des enseignes de la grande distribution : au moment de payer en magasin avec sa carte bancaire , le client ne tape pas son code , mais introduit son doigt dans un lecteur qui scanne les veines . -Une seule dose du vaccin a été en mesure de protéger les souris contre la dépendance à la nicotine pour la vie . Une seule dose du vaccin a pu protéger à vie des souris de la dépendance à la nicotine . -Un juge du sud de la Géorgie , le mercredi , a ordonné aux autorités de libérer toutes les vidéos de surveillance que les enquêteurs avaient examinées . Un juge de Géorgie du sud a ordonné mercredi aux autorités de remettre tous les enregistrements de vidéosurveillance que les enquêteurs avaient examinés . -Une espèce est considérée menacée ou menacée sur la base du nombre de ses membres vivant sur la planète , connus sous le nom de stocks , et de la possibilité pour les espèces de se reconstituer à l' aide de stocks voisins . Une espèce est considérée menacée ou en voie d' extinction en fonction du nombre d' individus existants sur Terre , c' est-à-dire du stock , et de la possibilité qu' elle a à se rétablir à partir des stocks voisins . -Un porte-parole du annuel a déclaré : " est seulement trois , mais il sait comment composer son " . Un porte-parole de la manifestation équestre annuelle a déclaré : « n' a que 3 ans mais il sait comment conduire son . » -Un porte-parole de la Conférence des évêques catholiques des États-Unis a confirmé que le document était authentique et que chaque évêque déciderait de la manière appropriée d' approcher les paroissiens . Une porte-parole de la Conférence des évêques catholiques des États-Unis a confirmé que le document était authentique et que chaque évêque déciderait de la façon appropriée de sonder les paroissiens . -Une déclaration de l' OIAC , qui travaille étroitement avec l' Organisation des Nations Unies , a indiqué que son équipe était maintenant " convaincue qu' elle avait vérifié - et avait été détruite - toutes les installations de production et d' équipement de production et de transformation déclarées de Syrie " . Une déclaration de l' OIAC , qui travaille en étroite collaboration avec l' ONU , indiquait que son équipe était « maintenant satisfaite d' avoir contrôlé – et vu détruites – toute la production critique déclarée et les machines destinées au mélange et au remplissage » . -Une étude visant à augmenter les avantages pour l' Écosse du projet ferroviaire a été annoncée par le gouvernement britannique . Une étude destinée à multiplier les avantages du projet ferroviaire pour l' Écosse a été lancée par le gouvernement britannique . -Une étude sur la façon dont les chiens réagissent aux vrais chiens a pu être utile , explique-t-il . Une étude sur la façon dont les chiens réagissent face à de vrais chiens pourrait être utile , a-t-il expliqué . -Une enquête publiée par le célèbre public des médias au début de l' année a révélé une explosion dans l' utilisation d' appareils mobiles par les jeunes enfants aux États-Unis : 38 % des enfants de moins de 2 ans utilisent déjà un ou un téléphone portable et 72 % en , contre 10 % et 38 % il y a deux ans respectivement . Un sondage publié en début de semaine par Common Sense Media montrait une explosion de l' usage des appareils mobiles par les jeunes enfants aux États-Unis : 38 % des moins de 2 ans se sont déjà servis d' une tablette ou d' un téléphone , et 72 % des moins de 8 ans , contre respectivement 10 % et 38 % il y a deux ans . -Une bande de , prise de son téléphone à au moins trois occasions , a été prise à la maison de en août 2006 . Une bande , sur laquelle étaient enregistrés des messages vocaux piratés sur son téléphone à au moins trois reprises , a été saisie au domicile de en août 2006 . -Une équipe de biologistes travaillant pour la Société de conservation de la faune de New York a identifié une nouvelle espèce de dauphin à bosse vivant sur la côte nord de l' Australie . Une équipe de biologistes dirigée par la Wildlife Conservation Society de New York a identifié une nouvelle espèce de dauphin à bosse au large des côtes septentrionales de l' Australie . -Une commission de trois personnes sera créée pour veiller à ce que l' État assure les services de la manière la plus efficace et la plus efficiente possible . Une commission composée de trois personnes sera constituée pour garantir que l' État fournit les services de la façon la plus efficace et efficiente possible . -Un avocat de haut rang qui a dit à une drogue M. Big une importante enquête de police , parce qu' il craint que sa drogue secrète ne soit contaminée , a été emprisonnée pendant trois ans . Un grand avocat de la défense , qui avait informé un parrain de la drogue d' une importante enquête de police car il avait peur que son addiction secrète à la drogue ne soit révélée , a été condamné à trois ans de prison . -Un groupe de l' industrie du tourisme s' est réjoui de ces changements , les qualifiant d' arrangements de bon goût pour un public en déplacement qui se avec la technologie . Un groupe de l' industrie du voyage s' est félicité des changements , les appelant des arrangements pleins de bon sens pour des voyageurs équipés de technologies . -Une vidéo envoyée aux autorités semble confirmer ces soupçons . Une vidéo remise à la justice semble appuyer ces soupçons . -Une jeune équipe de visionnaires était dirigée vers la Maison Blanche , et la nation était prête pour le changement . Une jeune équipe de visionnaires avançait vers la Maison-Blanche et la nation était prête pour le changement . -Aaron , âgé de 30 ans , a été arrêté le vendredi matin à la suite d' une perquisition à son domicile et au bureau du Conseil national de la recherche où il travaille . Aaron , âgé de 30 ans , a été arrêté vendredi matin suivant des perquisitions à son domicile et au bureau du Conseil national de la recherche où il travaille . -Le droit en matière d' avortement Remise en cause du droit à l' avortement au Texas -Il y a environ une dizaine d' années , les scientifiques ont reconnu deux espèces qui appartenaient à cette espèce : les dauphins de bosse du Pacifique et les baleines à bosse . Il y a une dizaine d' années , la communauté scientifique reconnaissait deux espèces appartenant à cette : les dauphins à bosse du Pacifique et les baleines à bosse de l' Atlantique . -Environ deux ans après le retrait des troupes américaines , le niveau de violence les craintes d' agitation alors que le voisin du pays , la Syrie , est en proie à la guerre civile . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Deux ans environ après le retrait des troupes américaines , le niveau des violences fait craindre un nouvel embrasement alors que la Syrie voisine est en proie à une guerre civile . -Selon la police chinoise , trois membres d' une Chine ouïghoure appartenant essentiellement à la communauté ouïghoure du Xinjiang , partageant la majorité avec plusieurs pays d' Asie centrale , ont volé une voiture transportant des drogues à l' abri de l' entrée de la Cité interdite à Pékin , dans un attentat-suicide qui a fait deux morts et plusieurs blessés . Selon la police chinoise , trois Ouïghours d' une même famille de la région à dominante musulmane du Xinjiang , frontalière de plusieurs pays d' Asie centrale , ont précipité leur voiture chargée de bidons d' essence contre l' entrée de la Cité interdite à Pékin , dans une qui a fait deux morts et 40 blessés . -Selon Detroit News , la reine de Soul se produira à la Salle Sound du Casino Hotel le 21 décembre . Le journal Detroit News rapporte que la reine du soul se produira le 21 décembre à la salle Sound Board de l' hôtel Casino . -Selon Google , les costumes les plus recherchés sont les zombies , Batman , les pirates et les sorcières . Selon Google , les les plus recherchés sont les zombies , Batman , les pirates et les sorcières . -Selon des sources palestiniennes de sécurité , les quatre menaient une opération de surveillance dans la zone frontalière entre l' enclave palestinienne et Israël . Selon des sources sécuritaires palestiniennes , les quatre combattants conduisaient une opération de surveillance dans la zone frontalière entre l' enclave palestinienne et Israël . -Selon le journal The Star , le nouveau projet le gouvernement sous un coup de pied " les médias " , tandis que la loi stipule que la démocratie et la liberté d' expression au Kenya ont été " un coup dur " et le projet de loi " draconienne " . Selon le journal The Star , le nouveau projet de loi permettra au gouvernement d' avoir « la mainmise sur les médias » . The Standard a , quant à lui , indiqué que la démocratie et la liberté d' opinion au Kenya « avaient subi un coup dur » et a le projet de loi le qualifiant de « » . -Selon une source à la Mission des Nations Unies pour la stabilisation en République démocratique du Congo ( ) , les combats sont entrés dans la " phase finale " : les FARDC ont " encerclé les derniers postes de pour les arracher " . Selon une source à la Mission des Nations unies pour la stabilisation de la RDC , les combats sont entrés " dans une phase finale " , les FARDC ayant " encerclé les positions du résiduelles pour les déloger " . -D' après une enquête réalisée par la Jet Trading Alliance , un organisme de la marine , environ 10 % des membres non américains ont annulé l' utilisation d' un réseau de satellites américains après avoir entendu des informations sur le programme de récupération des diamants de l' Amérique du Nord . Selon une enquête réalisée par la Cloud Security Alliance , un organisme professionnel , quelque 10 % des membres non américains ont renoncé à leurs projets d' utiliser un fournisseur américain de services de stockage dans le cloud après les révélations sur le programme américain d' exploration des données Prism . -D' après les chiffres publiés vendredi par les manufacturiers , cinq des plus de 6 points qui ont enregistré des résultats en termes d' attentes , la baisse de 16 jours de la plupart des services administratifs fédéraux a diminué la croissance des ventes de voitures aux États-Unis en octobre . La fermeture pendant 16 jours de la plupart des administrations fédérales a pesé sur la croissance des ventes de voitures aux Etats-Unis en octobre , montrent les chiffres publiés vendredi par les constructeurs , cinq des six principaux ayant réalisé des performances inférieures aux attentes . -Selon elle , le a été invité à une médiation et a demandé une autre période d' examen . Selon elle , la a été invitée à une médiation et elle a demandé un délai supplémentaire pour y réfléchir . -D' après les enquêtes de la Commission européenne , la France est la plus touchée par la présence de ce type de viande dans des produits qui ne sont destinés à contenir que du boeuf . Selon des examens de la Commission européenne , la France a été le pays le plus touché par la présence de ce type de viande dans des produits censés contenir uniquement du boeuf . -Selon l' un des documents , environ 181 millions d' articles ont été collectés au cours du mois de janvier seulement - allant de la à des courriels jusqu' à des éléments textuels ou à des fichiers audio et vidéo . A en croire un des documents , quelque 181 millions d' éléments avaient été collectés au cours du seul mois de janvier dernier - allant de métadonnées sur des e-mails , à des éléments de texte ou des documents audio ou vidéo . -Selon des informations émanant de quelques médias grecs , des membres de Golden Dawn ont affirmé que les victimes les locaux de la partie . Certains médias grecs ont rapporté que des membres d' Aube doré avaient soutenu que les victimes étaient des gardes de leur local . -Selon le quotidien , " même une amende suffit à paralyser la plupart des stations FM " . Selon le Daily Nation , « même une seule amende suffit à paralyser la plupart des stations FM » . -Selon le député , " il n' y a actuellement aucune différence dans la continuité visuelle entre la Flandre et Bruxelles . Selon le député , il n' y a désormais " plus de différence de continuité visuelle entre la Flandre et Bruxelles " . -Selon la marine , l' équipage n' a été exposé à aucune contamination radioactive . D' après la Marine nationale , l' équipage n' avait pas été exposé à une éventuelle contamination radioactive . -Selon le Gouvernement de l' État , ces nouvelles procédures en grande partie les récentes coupes . Selon le gouvernement de l' État , ces procédures non urgentes supplémentaires en grande partie les récentes coupes . -Selon le correspondant politique du public du billet blanc de Jonathan Brown , le général travaille à l' examen des efforts déployés par les hauts fonctionnaires de la campagne et de la Maison Blanche , l' ancien rédacteur en chef d' un parti d' Etat , Bill , pour mesurer l' effet de l' échange d' un ancien secrétaire d' État Clinton pour le vice-président dans les sondages . Selon le correspondant politique du Times Jonathan Martin , le livre offre une analyse détaillée du travail des hauts responsables de la campagne et de la Maison-Blanche , à savoir l' ancien secrétaire général de la Maison-Blanche Bill , visant à mesurer quel effet le remplacement du vice-président par l' ancienne secrétaire d' État Clinton pourrait avoir sur le scrutin . -D' après les médiateurs ougandais , les pourparlers entre le gouvernement de Kinshasa et le ont repris à Kampala , mercredi . Ces derniers , selon les médiateurs ougandais , ont repris mercredi à Kampala entre le gouvernement de Kinshasa et le . -Selon le Washington Post , la NSA a intercepté des centaines de millions d' utilisateurs de Google et de Yahoo . D' après le Washington Post , la NSA des données de centaines de millions d' utilisateurs de Google et Yahoo . -Selon les autorités , la victime est un homme dans les années cinquante , qui a travaillé pour une entreprise de coffrage . Selon les autorités , la victime est un homme âgé dans la cinquantaine qui travaillait pour une entreprise de coffrage . -Selon les mêmes sources et les mêmes témoins , un char israélien et un bulldozer blindé ont effectué une incursion d' une centaine de mètres sur le territoire avant de se replier . D' après les mêmes sources et des témoins , un char israélien et un bulldozer blindé ont fait une incursion d' une centaine de mètres à l' intérieur du territoire avant de se retirer ensuite . -Selon ces sources , les militants menaient une opération de surveillance dans la zone frontalière entre le territoire palestinien et Israël , lorsqu' ils ont été bombardés par un char israélien . Selon ces sources , ces activistes conduisaient une opération de surveillance dans la zone frontalière entre le territoire palestinien et Israël lorsqu' ils ont essuyé un tir d' obus d' un blindé israélien . -Selon ce témoin , qui a choisi de rester anonymes , les soldats ont " passé la nuit " à , avant de passer à une opération de " " . Selon ce témoin souhaitant rester anonyme , les soldats avaient " passé la nuit " à avant de monter au front pour une " opération de ratissage " . -Dans toute l' UE , les parties qui se sont engagées à s' opposer à l' UE ou à l' euro en tant que monnaie commune gagnent du terrain . À travers l ’ Europe , les partis qui s' opposent à l' UE ou qui militent pour l' abandon de l' euro en tant que devise commune , gagnent du terrain . -En fait , j' ai souvent eu l' habitude d' écouter la langue arabe pendant mon enfance et de parler du Maroc où elle vit depuis des décennies avant de revenir en France quand elle est devenue indépendante . En fait , je l' entendais souvent parler arabe dans mon enfance , et parler du Maroc , là où elle avait vécu pendant des décennies - avant de regagner la France au moment de l' indépendance . -À cela s' ajoute , plus récemment , des contrôles de présence . S' y est ajouté , plus récemment , le contrôle de présence . -De plus , le Défi Defender offrira une journée de formation et d' essai en février , ainsi que la possibilité de participer à des compétitions en Afrique du Nord et au Moyen-Orient . De plus , le Defender Challenge proposera une journée de formation et de tests en février et la possibilité de participer à des événements dans les déserts d ’ Afrique du nord et du Moyen-Orient . -Pour ce qui est de ce qu' il a appelé la " surprise dans les nouvelles " sur les annulations , Obama a invité les Américains à recevoir ces billets pour se procurer une nouvelle couverture sur le marché . Parlant de ce qu' il appelle « une effervescence dans l' actualité » au sujet des , le président Obama a incité les Américains qui recevaient ces avis à chercher une nouvelle assurance sur le marché . -Le bénéfice ajusté par action est demeuré inchangé à 9 cents . Le résultat par action ajusté est resté inchangé à 9 cents . -Les revenus tirés de l' exploitation minière ont chuté de 13 % à 2 milliards de dollars . Les revenus de l' Aéronautique ont baissé de 13 % pour atteindre 2 Md $ . -Les filles d' origine afro-américaine et hispanique ont tendance à atteindre la puberté plus tôt que leurs homologues blancs , les résultats de la recherche . Des études montrent que les jeunes filles et hispaniques ont tendance à atteindre l' âge de la puberté plus tôt que les jeunes filles blanches . -Après Broadway et Londres , Paris trouve enfin sa voix . Après Broadway et Londres , Paris trouve enfin sa voix . -Après un accord de paix de 1992 , il est devenu un parti d' opposition . Après un accord de paix signé en 1992 , elle est devenue un parti d' opposition . -Après avoir reçu des fournitures , on s' attend à ce qu' ils quittent la grotte le soir avec l' aide des sauveteurs . , ils devraient ressortir de la cavité avec l' aide des secouristes dans la soirée . -Après avoir jeté un coup d " œil lourd dans la livraison en janvier dernier de la position exclusive de dans le North Barrington , Olivier , George Smith , directeur général de la famille des pneus américains , Anthony , déclare maintenant qu' il est prêt à sauver 333 des de la société . Après avoir jeté l' éponge avec fracas en janvier pour la reprise partielle du site Goodyear d' promis à la fermeture , Maurice Taylor , le PDG du américain Titan , se déclare , aujourd' hui , prêt à sauver 333 emplois sur les 1 137 que compte l' usine . -Après cinq années de forte croissance depuis la crise financière mondiale , et la faible consommation de crédit alimentée par une politique monétaire laxiste dans les économies avancées , les familles à revenu faible ou moyen se tournent vers les magasins à puces pour faire la différence , alors que leur économie ralentit . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Après cinq années de croissance solide depuis la crise financière mondiale , et le crédit bon marché alimenté par une politique monétaire trop laxiste dans les économies avancées , les familles à revenu faible et moyen se tournent vers les prêteurs sur gage pour combler la différence alors que leurs économies connaissent un ralentissement . -Après avoir abandonné le plan d' acheter l' usine en janvier , aujourd' hui vous êtes revenus . Après avoir renoncé au projet d' achat de l' usine en janvier , vous revenez aujourd' hui . -Après qu' il a été arrêté , son journal a publié deux articles parus dans la presse pour sa libération , affirmant qu' il soutenait son journalisme . Après son arrestation , son journal a publié à deux reprises un appel en première page pour sa libération , écrivant qu' il soutenait son journalisme . -Après ses billets de mangue jaune et de Black , elle a fini par les vêtements à Bruxelles jaune , bleu foncé et noir . Après ses taxis , elle a finalement elle-même en les poteaux de Bruxelles . -Après avoir appris la vérité , il est revenu à un homme , prenant des de testostérone et a retiré ses seins . Après avoir appris la vérité , il est redevenu un homme , en suivant un traitement à la testostérone et s' est fait enlever les seins . -Après avoir perdu sur les marchés de Mega sur les brevets de Nortel , Google a acquis l' acquisition de Motorola Mobility pour 12,5 milliards de dollars , une transaction qui a été en partie prise par la bibliothèque de brevets de Motorola . Après avoir perdu les brevets Nortel face à Rockstar , Google a acquis par la suite Motorola Mobility pour un montant de 12,5 Md $ , un marché en partie basé sur la bibliothèque de brevets de Motorola . -Après dix jours de scandale , de révélations et de entre les États-Unis et ses alliés européens , c' est la première fois qu' un représentant du gouvernement américain reconnaît explicitement les activités controversées de l' interception des communications et des données par la CIA en Europe . Après dix jours de scandale , de révélations et de entre les Etats-Unis et leurs alliés européens , c' est la première fois qu' un responsable gouvernemental américain admet explicitement des pratiques controversées dans l' interception par la NSA de communications et de données en Europe . -Après dix années de silence ( son dernier record , la réalité , sorti en 2003 ) et de très nombreuses apparitions publiques , le musicien britannique a prouvé qu' il pouvait encore éclairer la scène pop . Après dix ans de silence ( le dernier disque , Reality , date de 2003 ) et de trop rares apparitions en public , le musicien anglais prouvait qu' on pouvait encore compter sur lui pour faire la scène pop . -Après l' intervalle , le est plus riche . Les , après l' , est plus riche . -Après la confirmation par la police qu' ils ont remis sur la copie d' une vidéo démontrant qu' il y avait Rob Ford Tobacco , Rod Clayton Wheeler a soutenu qu' il n' avait jamais vu une enquête " si " . Après que la police eut confirmé avoir mis la main sur une copie de la vidéo qui montrerait Rob Ford fumant du crack , Me Clayton Ruby a soutenu n' avoir jamais vu une enquête " aussi " . -Après le premier vol de l' avion d' essai il y a environ un mois et demi , il n' a fait que trois fois plus de temps , soulevant des questions sur la question de savoir si la phase d' essai était sur le point de se dérouler . Après le vol inaugural de l' avion d' essai il y a environ un mois et demi , ce dernier n' a volé que trois fois , et la question s' est alors posée de savoir si la phase d' essai était en bonne voie . -Après la guerre , l' élite politique française considérait l' Europe comme un véhicule pour maintenir l' Allemagne en position de contrôle , et pour donner à Paris la parité dans le monde avec Washington . Après la guerre , l' élite politique française a considéré l' UE comme un moyen de contrôler l' Allemagne et de mettre Paris sur un pied d' égalité avec Washington . -Après cette période difficile , semble être un peu plus facile pour le service de pharmacie du . Après cette période ardue , l' année s' annonce nettement plus facile pour le département de pharmacie du . -Après s' être déplacé dans des zones souvent , il a passé un mois à sa machine à utiliser la roue à Tokyo de 1976 , " toutes les pièces ont été remplacées " . Après ces voyages dans des zones souvent troublées , il s' pour un mois derrière sa machine à écrire IBM à datant de 1976 " dont toutes les pièces ont été changées " . -Raid aérien contre des installations militaires en Syrie Raid aérien contre des installations militaires en Syrie -Airbus et Boeing ne fournissent pas de sièges , mais offrent un catalogue de fournisseurs pour les compagnies aériennes . Airbus et Boeing ne fournissent pas de sièges , mais proposent un catalogue de fournisseurs parmi lesquels les compagnies aériennes doivent choisir . -Airbus propose un A350 , mais affirme qu' il ne l' a pas encore vendu . Airbus offre un A350 configuré avec 10 sièges par rangée mais indique qu' il ne l' a pas encore vendu . -Airbus affirme qu' elle a commandé des travaux de recherche suggérant qu' un pouce supplémentaire dans la largeur du siège améliore la qualité du sommeil de 53 % . Airbus ajoute qu' elle a commandé une étude qui indique qu' un pouce supplémentaire au niveau de la largeur des sièges améliorerait la qualité du sommeil de 53 % . -Airbus affirme que son rival est en phase avec un concept de siège des années 50 , lorsque la moyenne du plus récent " jet set " a été plus restreinte . Airbus déclare que son rival s' accroche à un concept de siège qui date des années 1950 , lorsque la circonférence de l' appareil fraîchement baptisé « jet set » était plus étroite . -Airbus affirme que la version concurrente de son A350 va emporter 350 personnes sur le économique fixé à 9 sur 12 . Airbus indique que la version concurrente de son A350 transportera en classe économique 350 personnes dans des sièges de 18 pouces de large configurés en rangées de 9 . -Airbus a lancé cette semaine pour une norme de l' industrie qui devrait prévoir un siège d' au moins 18 pouces ( 46 cm ) dans les cabines économiques , mais ses US Boeing stipulent que c' est aux compagnies aériennes de décider . Cette semaine , Airbus a appelé l' industrie aéronautique à mettre en place une norme imposant une taille de siège d' au moins 18 pouces ( cm ) dans les classes économiques , mais son grand rival américain Boeing déclare que ce devrait être aux compagnies aériennes de décider . -Règlements électroniques de l' appareil requis pour rester en vigueur en Australie pour l' instant Les règles relatives à l' utilisation d' appareils électroniques à bord restent pour l' instant en vigueur en Australie -Les passagers de la compagnie aérienne pourront utiliser leurs appareils électroniques pour lire , travailler , jouer aux jeux , lire des films et écouter de la musique - mais ne pas parler sur leurs téléphones cellulaires - dans de très nombreuses nouvelles directives annoncées jeudi par la Federal Aviation Administration . Les passagers des compagnies aériennes pourront utiliser leurs appareils électroniques de porte à porte pour lire , travailler , jouer à des jeux , regarder des films et écouter de la musique – mais pas parler sur leur téléphone portable – en vertu des nouvelles directives très attendues publiées jeudi par la Federal Aviation Administration . -Les compagnies aériennes ont commencé à facturer les premiers et deuxièmes bagages enregistrés en 2008 . Les compagnies aériennes ont commencé à faire payer les premier et second bagages enregistrés en 2008 . -Depuis plusieurs années , les compagnies aériennes assurent l' utilisation d' une connexion Wi-Fi à des altitudes de croisière . Les compagnies aériennes permettent à leurs passagers d' utiliser le Wi-Fi aux altitudes de croisière depuis plusieurs années . -Les compagnies aériennes paient des vendeurs de billets en ligne comme les de 10 à 25 $ pour chaque billet vendu . Les compagnies aériennes payent aux vendeurs de voyages en ligne comme 10 $ à 25 $ pour chaque billet vendu . -Les compagnies aériennes devront montrer à la FAA comment elles respectent les nouvelles directives et ont mis à jour les manuels de formation de leurs équipages et les règles relatives à l' arrimage des dispositifs afin de tenir compte des nouvelles lignes directrices . Les compagnies aériennes devront prouver à la FAA que leurs avions respectent les nouvelles directives et qu' elles ont mis à jour les manuels de formation des membres d' équipage et les règles concernant le rangement des appareils pour être en conformité . -Tous les vols à destination et en provenance de l' aéroport ont été suspendus . Tous les vols à l' arrivée et au départ de l' aéroport ont été suspendus . -Les quatre suspects étaient surveillés par la police au cours d' une enquête majeure sur un trafic d' héroïne et de cocaïne qui opérait à travers le , le , le , le et le West Yorkshire . Les quatre suspects étaient surveillés à l' époque par la police dans le cadre d' une importante enquête sur un trafic d' héroïne et de cocaïne se déroulant dans les comtés de , , , et West Yorkshire . -Tout ce qu' il a fallu , c' est mettre l' accent sur ses erreurs et , conformément à , la question sera confiée aux préfets et aux - les interprètes autorisés . Il ne fallait qu' en déployer les accidents , et l' affaire , oblige , était confiée aux préfets et aux , interprètes autorisés . -Tous les candidats nous ont dit qu' ils allaient s' attaquer à cette dette scandaleuse . Tous les candidats nous ont dit qu' ils s' à la dette scandaleuse . -Tous les locataires sont arrivés au moment de l' incendie . Tous les locataires étaient absents lors de l' incendie . -Tous les gels de salaires dans le monde ne pas que l' industrie pétrochimique européenne soit par un gaz de schiste américain bon marché . Tous les gels de salaires au monde n' empêcheront pas l' industrie pétrochimique européenne d' être pénalisée par un gaz de schiste américain bon marché . -Tous les billets sont valables pour ces spectacles . Tous les billets restent valables pour ces concerts . -Presque tous les vendredis , jusqu' à 30 personnes se dans des fourgonnettes , des fournitures médicales aux toits et se déplacent en terrain accidenté pour visiter les villages dans le besoin . Presque tous les vendredis , avec une trentaine de personnes , ils s' dans des fourgonnettes après avoir fixé le matériel médical sur le toit et traversent des terrains difficiles pour se rendre dans les villages qui ont le plus besoin d' aide . -Il y a presque trente ans , mon mari est mort de 33 ans . Voila bientôt trente ans que mon mari est décédé à l' âge de 33 ans . -Déjà , l' Australie et le Népal permettent aux adultes de marquer le nom des hommes , des femmes ou des " troisième sexes " sur leurs documents officiels . L' Australie et le Népal autorisent déjà les adultes à inscrire sexe masculin , féminin ou « troisième sexe » sur leurs documents officiels . -À l' heure actuelle , les droits de timbre pour les maisons de Westminster et de Kensington et de Cork , qui s' élèvent à 6 millions de livres sterling au cours de l' exercice 2012 / 13 , sont supérieurs à ceux de l' ensemble du territoire d' Irlande du Nord , du pays de Galles , de l' Écosse , du Nord-Est , du West Yorkshire et du Yorkshire et du Humber . Le droit de timbre s' applique déjà aux propriétés résidentielles des arrondissements de Westminster et Kensington & Chelsea , et représente 708 M £ sur l' exercice fiscal 2012 / 13 , ce qui dépasse le montant total de l' Irlande du Nord , du pays de Galles , de l' Écosse , du Nord-Est , du Nord-Ouest et de la région Yorkshire et Humber réunis . -De plus , M. a besoin d' un engagement de la part de Titan avant d' essayer pour que la CGT se joigne à Goodyear . Aussi M. a-t-il besoin d' un engagement de Titan avant d' essayer d' amener la CGT à s' asseoir à la table avec Goodyear . -De plus , les investisseurs continuent de faire preuve de prudence , car ils croient que la Réserve fédérale américaine pourrait mettre fin à sa politique d' assouplissement quantitatif plus tôt que prévu . Par ailleurs , les investisseurs restent prudents car ils estiment que la Réserve fédérale américaine pourrait dénouer sa politique d' assouplissement quantitatif plus tôt que prévu . -De même , la France peut , par exemple , transmettre des blocs entiers de données sur la région du Sahel aux Américains et , en retour , comme cela a déjà été mentionné brièvement , les Américains peuvent fournir des informations aux Français sur d' autres parties du monde . La France peut être conduite , par exemple , à transmettre des blocs entiers de données sur la région du Sahel aux services américains , et , en contrepartie - on l' a déjà rapidement dit - , les Américains peuvent donner des informations aux Français sur d' autres régions du monde . -Jeudi également , un groupe de militants syriens a déclaré que plus de 120 000 personnes avaient été tuées depuis le début de la guerre civile du pays il y a près de trois ans . Un groupe de militants syriens a également déclaré jeudi que plus de 120 000 personnes ont été tuées depuis le début de la guerre civile dans le pays il y a près de 3 ans . -Parmi les morts , il y avait également soldats de l' et quelques d' opposition , dont un grand nombre d' étrangers . Parmi les morts , on estime également que 2 202 étaient des déserteurs et quelque 5 375 des combattants de l' opposition , dont bon nombre étaient étrangers . -Bien que Google travaille actuellement sur un modèle de Google Glass équipé de correcteurs correcteurs , aucune date pour un lancement à grande échelle n' a encore été annoncée . Bien que Google travaille actuellement sur un modèle de Google Glass équipé de verres de correction , aucune date de commercialisation à grande échelle n' a encore été confirmée . -American Media and Entertainment Group de Disney a décidé de donner priorité aux PC Tablet sur ses propres chaînes de télévision pour la prochaine émission d' une nouvelle série pour les enfants . Le groupe américain de médias et de divertissement Disney a décidé de privilégier les tablettes à ses propres chaînes de télévision pour la sortie prochaine d' une nouvelle série pour enfants . -Les Américains n' achètent pas autant de gaz qu' ils ne l' ont fait . Les Américains n' achètent plus autant d' essence qu' avant . -Les Américains dépensent 1,9 milliards de dollars pour cette année , selon The Nielsen Company . Les Américains vont dépenser 1,9 Md $ en bonbons cette année , selon le cabinet Nielsen . -Au cours d' une vague de critiques , le président Obama a hier sa promesse répétée et sans équivoque que " si vous aimez votre régime de santé , vous pouvez le garder " . Sous une avalanche de critiques , le président Obama est revenu hier sur sa promesse sans ambiguïté et souvent répétée selon laquelle « si vous êtes satisfait de votre régime de santé , vous pouvez le garder » . -Au cours du tumulte , Murphy a déclaré que son bureau devrait organiser le voyage du Congrès , ce qui devrait se produire cette année , et espère que la délégation comprendra des membres des deux parties et des deux chambres . Dans cette atmosphère de tumulte , Murphy a indiqué que son bureau organisait un voyage du Congrès , qui devrait avoir lieu cette année , et qu' il espérait que la délégation comprendrait des membres des deux partis et des deux chambres . -Parmi les personnes qui militent en faveur d' un assouplissement des restrictions sur l' utilisation par les passagers des appareils ont été . fait partie de ceux qui font pression pour un assouplissement des restrictions concernant l' utilisation des appareils électroniques par les passagers . -De grandes quantités de pétrole brut ont pesé sur le prix au cours des dernières semaines . L' offre abondante de pétrole brut a pesé sur le prix au cours des dernières semaines . -Un dessin d' Henri est projeté sur l' écran . Sur l' écran , une encre d' Henri est projetée . -Le cri de désespoir de l' élection d' Obama Le cri de désespoir d' un électeur d' Obama - partisan de Jean-Pierre en 2002 , il a par la suite soutenu Dominique de Villepin dans le district de 2010 à 2011 . en 2002 aux côtés de Jean-Pierre , il a ensuite soutenu localement Dominique de Villepin entre 2010 et 2011 . -Plus de 2.600 opérations chirurgicales , y compris la chirurgie orthopédique et la cataracte , permettront de résorber l' arriéré . 2 600 opérations supplémentaires , notamment dans le domaine de la chirurgie orthopédique et de la cataracte , aideront à rattraper le retard . -Un privilège d' admission est le droit d' un médecin d' admettre un patient dans un hôpital ou un centre médical pour effectuer un diagnostic ou une procédure , sur la base de son statut de travailleur de santé dans un hôpital . Le privilège d' admission est le droit d' un médecin , en vertu de son statut de membre soignant d' un hôpital , d' admettre un patient dans un hôpital ou un centre médical afin d' y délivrer un diagnostic ou un traitement . -Un assistant du maire est allé à Londres pour demander à la Grande-Bretagne d' approuver cette demande . Un adjoint au maire de l' île est allé à Londres demander à la Grande-Bretagne de valider cette demande . -On estime qu' un enfant de 2 000 ans né chaque année n' est ni un garçon ni une fille . On estime qu' un enfant sur 2 000 nés chaque année , n' est ni un garçon ni une fille . -Un a appelé cette phrase , animée par un chaud , à un assistant qui lui a été assigné à l' audience de la séance d' hier à la suite d' un litige relatif à l' inscription de Foday Chi Long , , si elle devait être obligée de s' inscrire à . . C' est une Kathleen qui a prononcé cette phrase , alors qu' elle croyait son micro coupé , en s' adressant à un assistant assis derrière elle lors de la séance de l' Assemblée d' hier suite à un échange de points de vue litigieux avec Billy Long , représentant républicain du Missouri , pour savoir si elle devrait adhérer à l' . -Un extrait de a suscité beaucoup d' excitation sur Internet : il montre Martin au milieu d' un au plafond avec des . Un extrait de a fait monter la température sur la Toile : Martin y apparaît au lit et en pleine extase avec . -Une taxe supplémentaire ou annuelle pourrait être imposée aux conducteurs d' hybrides et d' autres dont les véhicules n' utilisent pas beaucoup de gaz , et ils paient donc leur juste part . Une taxe unique ou annuelle supplémentaire pourrait être imposée sur les conducteurs de véhicules hybrides et sur d' autres dont les véhicules ne consomment pas beaucoup d' essence , de sorte que chacun paie sa juste part . -Un comité consultatif de l' industrie créé par la FAA pour examiner la question a recommandé le mois dernier que le gouvernement autorise une utilisation plus large des appareils électroniques personnels . Un comité consultatif industriel créé par la FAA pour examiner le problème a recommandé le mois dernier que le gouvernement autorise une utilisation plus large des appareils électroniques personnels . -Un inspecteur de la Commission de la santé et de la sécurité au travail a été envoyé au site pour enquêter sur les circonstances de la tragédie . Un inspecteur de la Commission de la santé et de la sécurité du travail a été dépêché sur les lieux afin d' enquêter sur les circonstances entourant ce drame . -Une enquête est en cours pour trouver la cause de l' incendie . Une enquête est en cours pour trouver la cause de cet incendie . -Une enquête sera menée pour déterminer la cause , bien que la brigade des pompiers ait exclu la possibilité qu' elle soit un acte criminel . Une enquête sera menée pour en déterminer l' origine , mais les sapeurs ont écarté la thèse d' un acte criminelle . -Un magasin en ligne proposant une gamme d' accessoires pour Google Glass vient d' être lancé pour les milliers de développeurs qui possèdent un prototype des lunettes ® . Une boutique en ligne , proposant de multiples accessoires dédiés à Google Glass , vient d' être mise à disposition des milliers de développeurs possédant un prototype des lunettes connectées signées Google . -Un appel verbal sur l' appel de la ville est prévu après le 14 mars 2014 . L' audition de l' appel interjeté par la ville devrait avoir lieu après le 14 mars 2014 . -Selon les analystes , il est peu probable que le pays retombe dans un conflit réel , mais les événements récents ont attiré des investisseurs étrangers et des résidents locaux . Les analystes pensent que le pays ne devrait pas retomber dans un conflit ouvert , mais les récents évènements ont ébranlé les investisseurs étrangers et la population locale . -Les analystes avaient prévu des bénéfices de 10 cents par action et revenus de milliards de dollars , selon Thomson Reuters ( E / B / S ) . Les analystes s' attendaient à un résultat de 10 cents par action et des revenus de Md $ , selon Thomson Reuters I / B / E / S. -Les analystes ont prévu une moyenne des ventes de 15,4 millions d' unités ( taux annualisé , ajusté selon les variations saisonnières ) , mais le marché a atteint 15,3 millions , selon des estimations d' Adam Jonas , analyste à Morgan Stanley . Les analystes en moyenne 15,4 millions de ventes en rythme annualisé et corrigé des variations saisonnières mais le marché s' est établi à 15,3 millions selon les estimations d' Adam Jonas , analyste de Morgan Stanley . -Les analystes interrogés par Reuters comptaient sur un profit moyen de $ par action . Les analystes interrogés par Reuters en moyenne sur un bénéfice par action de dollars . -Et je lutte , à côté d' autres personnes , pour la reconnaissance du statut juridique des otages . Surtout , je me , avec d' autres , pour la reconnaissance du statut juridique de l' otage . -Et , plus récemment , elle ne peut excuser l' incapacité de concevoir un site Web simple depuis plus de trois ans depuis l' adoption de la Loi sur les soins à domicile . Et , plus récemment , elle ne peut pas excuser l' impossibilité de concevoir un simple site Internet pendant plus de trois années après que la loi sur l' Care Act a été ratifiée . -Et , en plus , il y a le portrait de l' historien , qui devrait contenir quelques surprises . Avec , en prime , le portrait de l' historien , qui devrait réserver quelques surprises . -Et , oui , je effectivement un garçon . Et bien sûr , ce fut un petit garçon dans mon ventre . -Et a reçu des notes élevées pour son apparence et sa personnalité . Et a reçu de très bonnes notes pour sa présentation et sa personnalité . -Et je m' adresse à l' Europe pour leur dire clairement que nous devons continuer à travailler ensemble à la lutte contre le terrorisme , en dépit de leur colère vis-à-vis de ces programmes . Et je vais me rendre en Europe pour bien leur faire comprendre que nous devons continuer à travailler ensemble pour lutter contre le terrorisme , malgré la colère provoquée par les programmes de la NSA . -Et je pense à mon père . Et je pense à mon père . -Et il y a un peu plus de dix ans , la communauté médicale considérait le sexe comme une qui pouvait être effacée et . Et il y a à peine une dizaine d' années , la communauté médicale considérait le genre comme une ardoise , que l' on peut effacer et sur laquelle on peut redessiner . -Et même s' ils pouvaient voir un médecin , de nombreux n' en avaient pas les moyens . Et même s' ils pouvaient voir un médecin , de nombreux ne pourraient pas payer la consultation . -Et chaque matin , je regarde dehors . Et tous les matins , je détourne les yeux . -Et enfin , il donne un avant-goût de sa nouvelle piste , , qui est en général dans le style de la prochaine journée , avec des guitares très bien et un métal électrique finement maîtrisé . Et , surtout , il donne un avant-goût de son nouveau titre , typiquement dans la veine de The Next Day , avec des guitares très en avant et une électricité rock savamment maîtrisée . -À partir de là , il y a une nation , divisée entre la gauche et la droite , laïque et catholique , l' adoration et la haine pour la Révolution . Et , de là , une nation binaire , partagée entre gauche et droite , laïcité et catholicisme , adoration et haine de la révolution . -Et il a lancé une association sans but lucratif , , qui se rend dans les zones rurales les fins de semaine pour fournir des soins médicaux gratuits . Et il a créé une association humanitaire à but non lucratif , , qui se rend dans les zones rurales pendant les week-ends pour dispenser des soins de santé gratuits . -Et les appareils plus lourds comme les portables devront continuer à être parce qu' ils risquent de blesser quelqu' un s' ils volent dans la cabine . Et les appareils plus lourds comme les ordinateurs portables devront toujours être rangés car ils pourraient blesser quelqu' un s' ils à travers la cabine . -Dans un autre document de recherche de l' Université de Victoria au Canada , il a dit : " Les chiens sont plus susceptibles de parler avec un chien quand son chien " est venu à la plutôt qu' à la droite , plutôt que de devenir anxieux - le contraire inverse à l' étude italienne . Et dans une autre étude scientifique réalisée par l' université de Victoria au Canada , il a déclaré : « Les chiens allaient plus facilement vers un chien robot lorsque sa queue vers la gauche plutôt que vers la droite , au lieu de devenir anxieux – l' inverse de ce qui figure dans l' étude italienne . » -Et n' est-il pas étonnant de lire dans les pages du Monde , d' une part , la reproduction d' une correspondance diplomatique avec les États-Unis et , d' autre part , la condamnation de l' espionnage du Ministère des affaires étrangères sur le Quai d' Orsay , dans les semaines qui viennent . & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; N' est-il pas surprenant de lire dans les colonnes du Monde à quelques semaines d' intervalle d' une part la reproduction de la correspondance diplomatique américaine et d' autre part une condamnation des écoutes du Quai d' Orsay par la NSA ? -Et c' est aussi la première fois depuis 1996 qu' il n' y a pas de groupe armé allié au Rwanda présent dans l' est du RDC . Et ce serait aussi la première fois depuis 1996 qu' aucun groupe armé allié au Rwanda ne serait présent dans l' est de la RDC . -Et ce qui est tellement long dans la dent qu' elle est devenue : je me suis d' abord dit que " l' indépendance est le contrôle des passeports " il il y a quarante ans au moins . Et une histoire si ancienne qu' elle est maintenant figée : J' ai entendu pour la première fois le selon lequel « l' indépendance signifie le contrôle des passeports » il y a plus de 40 ans . -Et elle aime prendre des risques - la jeune fille qui n' a pas hésité à s' habiller nu sur papier nu apparaîtra dans des situations bien pires sur le grand écran . Sans oublier un certain goût du risque : celle qui n' avait pas hésité à poser nue sur du papier glacé va s' afficher dans des situations bien plus sur grand écran . -Et il le fait , mais dans une forme moins formelle , plus explosive et plus subtile . Certes sous une forme moins raide , plus éclatée et plus subtile . -Et les actes réussis tels que " Cabaret " ou " Les " ont eu un air de mystère à leur égard . Et les réussites comme Cabaret ou Les portaient l' du mystère . -Et cela doit être ; il a été révélé par une histoire plutôt que par une déclaration . Et il doit être ; révélé à travers une histoire plutôt que déclaré . -De plus , la Coalition , qui regroupe 17 ministères de la sécurité du littoral le long de la côte est ( y compris le Maryland , la Pennsylvanie , la Virginie et la Floride ) , étudie comment elle pourrait s' appliquer à la mise en oeuvre du changement . Et la coalition , qui comprend 17 agences impliquées dans le transport routier dans des États longeant la côte est américaine ( comprenant le Maryland , la Pennsylvanie , la Virginie et la Floride ) étudie comment elle pourrait mettre en œuvre le changement . -Et la Marine assure que " cela n' a en aucune manière retardé le [ [ ] ] plan d' activités des navires " . Et la Marine nationale de certifier que " ça ne remet nullement en cause le programme d' activité du bateau " à propulsion nucléaire . -Et le président est résolu à essayer de clarifier et de clarifier les choses pour les gens et fait maintenant un examen approfondi pour que personne n' ait le sentiment d' abus . Et le Président est résolu à tenter de clarifier et préciser clairement aux gens qu ’ il procédait maintenant à un réexamen approfondi de ces pratiques afin que personne ne se sente trompé . -Et l' homme qui a rejeté une idée d' une nation plus d' un siècle plus tard , mais qui a échappé à l' exercice de l' exercice du travail philosophique , confie en nous , cum , qu' il a fini par le faire tout . Et celui qui avait déposé , voici plus d' un demi-siècle , un sujet sur l' idée de nation , mais s' était soustrait à l' exercice initiatique de la thèse , nous confie , cum , qu' il aura fini par la faire . -Et le plus emblématique est clairement celui qui a divisé l' histoire nationale en deux : Et la plus emblématique est évidemment celle qui en deux l' histoire nationale . -Et puis je termine le livre . Et puis je ferme le livre . -Et elle se alors en me montrant des aspects de moi-même que je n' avais jamais vus auparavant , cachés par ma relation avec elle . Et puis , elle se révélait tout en me dévoilant mes propres facettes que je n' avais pas encore perçues , masquées par ma relation avec elle . -Et il y avait beaucoup de médias . Et beaucoup de médias . -Et cette forme est définitive de toute la pièce : il s' agit d' une série de représentations physiques des dessins . Et cette forme sera celle de toute la pièce : une d' imitations physiques des dessins . -Et que se passe-t-il quand le virus du terrorisme est vivant . Et que se passe-t-il quand un virus du terrorisme vit ? il se répand . -Et même si le Congrès ne peut s' entendre sur la manière de procéder , plusieurs États ne s' y attendent pas . Et tandis que les membres du Congrès n' arrivent pas à se mettre d' accord pour savoir s' il faut continuer , plusieurs États n' ont pas attendu . -Et avec la pauvreté , ils ne peuvent pas profiter de leur vie . Et en raison de la pauvreté , ils ne peuvent pas profiter de la vie . -Andrew , 35 ans , de , Essex , a été repéré en train de conduire sa Mercedes Benz sur le à le 27 mai . Andrew , 35 ans , de dans l' Essex , a été repéré conduisant sa Mercedes Benz sur l' à hauteur de le 27 mai . -La colère dans les rangs des islamistes d' Égypte . La colère dans les rangs des islamistes égyptiens . - la colère sur la bombe de feu de Bali provoquée par la peine prononcée à l' encontre d' un des de bombe de Bali -Ann Hidalgo , candidate du Parti socialiste , a répété lundi matin à France Inter qu' elle était en faveur d' un moratoire pour empêcher la fermeture le 4 novembre . Anne Hidalgo , la candidate PS , a quant à elle de nouveau rappelé lundi matin sur France Inter qu' elle s' était prononcée en faveur d' un moratoire pour qu' il n' y ait pas de fermeture le 4 novembre . -Anne Hidalgo , candidate socialiste à Paris , devrait assister et présentera ultérieurement ses propositions sur les questions de lors d' une conférence de presse . Anne Hidalgo , candidate socialiste à la mairie de Paris doit y assister , et expliquera ensuite , lors d' une conférence de presse ses propositions en matière de funéraire . -Autre : Bien que les garçons aient des cheveux et à l' âge avancé , la tendance à la puberté précoce n' est pas aussi prononcée que chez les filles . Autre problème : Bien que les garçons aient des poils sur le visage et dans la zone plus jeunes , la tendance envers une puberté précoce véritable n' est pas aussi prononcée que chez les filles . -Une autre journée , une nouvelle histoire d' indépendance de l' aviation . Un autre jour , une autre histoire inquiétante d' indépendance . -Un autre effet pourrait être plus commercial : à la lumière des révélations , de plus en plus de pays d' Amérique du Sud et d' Amérique latine sont invités à accepter la divulgation de leurs données confidentielles aux fournisseurs américains qui sont soumis à la législation américaine et donc à la . Un autre effet peut être lui plus économique : de plus en plus d' entreprises européennes ou rechignent , à la lumière des révélations , à confier leurs données confidentielles à des prestataires américains soumis aux lois américaines , et donc à l' emprise de la NSA . -Un autre dirigeant a indiqué que Barack Obama avait donné l' ordre de mettre fin à ces pratiques ces dernières semaines . Un autre officiel a , lui , indiqué que Barack Obama avait donné ordre de cesser ces pratiques au cours des semaines passées . -Une autre fois , quand je me suis réveillé tranquillement au milieu de la nuit , j' ai regardé devant sa montre , que j' ai toujours avec moi , couché sur la table de chevet . Une autre fois , un doux réveil en pleine nuit pour regarder sa montre qui ne me quitte plus , posée sur la table de nuit . -Une autre utilisation est de prendre une photo d' une personne qui s' exécute dans une boutique , de suivre ses progrès à l' aide de la reconnaissance faciale et de faire le temps d' attente . Autre utilisation : prendre en photo un qui fait la queue dans un magasin , suivre sa progression grâce à la reconnaissance faciale et calculer le temps d' attente . -La police antiterroriste s' est rendue à l' avenue principale de , où l' incident s' est produit et a fermé la zone . Des policiers du service antiterroriste se sont dépêchés sur l' avenue centrale de la banlieue , où l' incident a eu lieu et ont bouclé le quartier . -Plus que je ne le ferai " . Pas plus que moi . -Apparemment , la version finale du film durera 2 heures 45 minutes . Il semblerait que le film dure finalement . -Les singes de l' Arctique le marathon de Vienne en raison de la tragédie d' Alex Turner Arctic reporte un concert à Glasgow en raison de la maladie d' Alex Turner - Franklin Back at Stage en décembre Franklin sur scène en décembre -Arnaud , ministre de la relance industrielle , avait déjà annoncé le lundi 21 octobre . Arnaud , ministre du redressement productif , l' avait déjà annoncé , lundi 21 octobre . -Environ 70 personnes ont été évacuées de l' appareil . 70 personnes environ sont sorties de la séance . -Alors que les planificateurs de la route de l' Amérique tentent de trouver de l' argent pour réparer un barrage en plein air , nombreux sont ceux qui commencent à trouver une solution dans une petite boîte noire qui s' aligne parfaitement sur le tableau de bord de votre voiture . Alors que les planificateurs du réseau routier des États-Unis ont du mal à trouver l' argent nécessaire pour réparer l' infrastructure en , nombreux sont ceux qui une solution sous forme d' une petite boîte noire qui se fixe au-dessus du tableau de bord de votre voiture . -Comme Reuters l' a signalé pour la première fois en juillet , la fixation du siège est exactement ce qui fait la bataille entre les derniers jets . Comme Reuters l' a indiqué pour la première fois en juillet , la configuration des sièges est exactement ce qui alimente la bataille entre les tout derniers appareils . -Par conséquent , des recherches ont été menées au cours des vingt dernières années dans d' autres méthodes , comme la reconnaissance faciale ou l' iris ( partie colorée de l' oeil ) . Conséquence , des recherches sont menées depuis une vingtaine d' années sur d' autres approches , comme la reconnaissance du visage ou de l' iris ( la partie colorée de l' oeil ) . -En conséquence , la révolte se , surtout en France , une fois de plus le principal de l' UE . En conséquence , la révolte , en particulier en France , l' un des grands meneurs de l' UE . -En conséquence , certains de ses livres de James ont été : un mois avant l' attaque d' un centre de commandement du gouvernement syrien , où plusieurs hauts responsables avaient été tués , il avait déjà raconté l' histoire au " Chemin de Damas " . Du coup , nombre de ses SAS sont souvent : ainsi , un mois avant l' attaque d' un centre de commandement du régime syrien ayant tué plusieurs hauts responsables , il avait raconté l' histoire dans " Le Chemin de Damas " . -Par conséquent , les ne respectent pas les normes fédérales de sécurité routière sur les appuie-tête . En conséquence , les dossiers ne respectent pas les normes fédérales de sécurité des véhicules concernant les . -En raison des politiques énergétiques " vertes " imposées par Bruxelles - le code pour subventionner les industries françaises et allemandes de l' énergie pour les consommateurs - l' industrie européenne paie deux fois plus pour l' électricité et quatre fois plus pour le gaz que pour les États-Unis . Suite aux politiques énergétiques « vertes » imposées par Bruxelles – qui sont une raison pour subventionner les sociétés françaises et allemandes du secteur de l' énergie aux frais du consommateur – l' industrie européenne paie deux fois plus pour l' électricité et quatre fois plus pour le gaz que les États-Unis . -En tant qu' expert inquiet d' une , il se promène dans une forêt de symboles , s' arrête après chaque étape pour examiner un objet sous la forme d' un puzzle , par des questions , Analyste inquiet d' une familière , il dans une forêt de symboles , arrêté à chaque pas par un objet en forme de , tourmenté de questions . -Au fur et à mesure que des foules d' experts se réunissaient à Cardiff pour les affronter à cheval pour l' Année , ils savaient que la compétition serait rude . Lorsque de nombreux experts en manifestations équestres se sont réunis à Cardiff pour s' affronter et remporter le trophée du Cheval de l' année , ils savaient que la compétition serait rude . -Au fur et à mesure que les régimes changent , les gens ont plus de sièges , mais les sièges d' avion n' ont pas radicalement changé . Comme les habitudes alimentaires changent , les gens , mais les sièges dans les avions n' ont pas radicalement changé . -Comme chaque année , le premier spectacle d' audition a donné naissance à un grand nombre de jeunes chanteurs dans le mélange , parfois pour les meilleurs et parfois pas . Comme chaque année , les premières émissions d' audition un large choix de chanteurs en herbe , pour le meilleur et parfois pour le pire . -En ce qui concerne les hommes , je pense que nous allons voir beaucoup de zombies , grâce à The Dead Back , et je parie que hommes Mag Space le feront dans nos feed cette année . Pour ce qui est des hommes , je m' attends à voir de nombreux zombies , grâce à la série télévisée The Walking Dead , et je parie que les hommes de l' espace de Punk figureront parmi nos photos cette année . -Dès que les premiers pompiers sont arrivés , ils pouvaient clairement voir la fumée . Aussitôt que nos premiers pompiers sont arrivés , ils ont vraiment vu de la fumée apparente . -Au fur et à mesure que le festival s' élargit , le monde de la comédie . Le festival a pris de l' ampleur , tout comme le monde de la comédie . -Au moment où le procès est en cours , l' du Nevada a mis en garde sur son site Web : " il serait relativement facile de transformer ces appareils en outils de traçage à part entière " . Dès le début de l' essai , l' du Nevada sur son site : « Ce serait relativement facile de transformer ces appareils en dispositifs de localisation à part entière . » - de la façon dont il a développé son personnage , l' acteur et chanteur Justin Timberlake a découvert " qu' il a grandi dans le Tennessee , baigné dans le blues et la musique country " . Interrogé sur la façon dont il a composé son personnage , l' acteur et chanteur Justin Timberlake avait rappelé avoir " grandi dans le Tennessee , baigné par le blues et la country " . -À la question de savoir s' il était retourné au poste de Premier ministre , M. Blair a été cité par la de Londres comme suit : " Oui , bien sûr , mais il n' est pas probable que cela arrive , " ... Quand on lui a demandé s' il aimerait retrouver le poste de Premier ministre , M. Blair aurait répondu , selon l' Standard de Londres : « Oui , bien sûr , mais il peut probable que cela arrive , donc ... » -Les troupes d' Assad ont combattu des rebelles , dont beaucoup sont liés à des groupes d' Al-Qaida , en pendant des semaines . Les troupes d' el-Assad se battent contre les rebelles , dont bon nombre sont liés à des groupes du réseau Al-Qaïda , à depuis des semaines . - est soupçonné d' avoir fait l' objet d' une extradition devant la Suède au sujet d' allégations de crimes sexuels , après avoir épuisé ses moyens en vertu de la loi britannique lorsque la Cour suprême a renversé son appel à l' extradition au début du mois . pourrait être extradé en Suède suite aux allégations de crime sexuel , après avoir épuisé toutes les voies possibles en vertu de la loi britannique après que la Cour suprême a invalidé l' appel qu' il a interjeté contre son extradition au début du mois . - craint qu' il soit extradé de Suède vers les États-Unis pour des accusations d' espionnage , après avoir libéré plus de 250 secrets diplomatiques américains sur le site internet de WikiLeaks . craint d' être extradé de la Suède vers les États-Unis où il pourrait être accusé d' espionnage , après avoir dévoilé plus de 250 000 câbles diplomatiques américains sur le site WikiLeaks luttant contre les informations secrètes . -Les attaques contre les rebelles se sont intensifiées alors que les forces gouvernementales ont attaqué les bases de et ont tenté de tuer Afonso , le dirigeant du groupe , Fernando , porte-parole de , a déclaré au Financial Times . Les assauts sur les anciens rebelles ont alors dégénéré lorsque les forces gouvernementales ont attaqué les bases de la et tenté de tuer Afonso , le leader du groupe , et Fernando , le porte-parole de la , selon le Financial Times . -À 18 ans , il a été mis en place pour un . À 18 ans , il a subi une . -À , sur la partie ougandaise de la frontière , à environ 5 000 personnes qui se sont réfugiées lundi et mercredi après les Nations unies , le processus de retour qui a commencé le matin a été inversé à midi . À , du côté ougandais de la frontière , où s' étaient réfugiées environ 5000 personnes entre lundi et mercredi selon l' ONU , le mouvement de retour vers la RDC qui s' était enclenché au matin s' est inversé à la mi-journée . -À la Cour de justice de Colchester , a reconnu avoir pris des mesures de conduite sans qu' il y ait eu de soins ni d' attention suffisante . Au Tribunal de première instance de Colchester , a admis l' infraction de conduite sans la prudence et l' attention requises . -Dans un du centre commercial à Singapour , un , une bague et deux boucles d' oreilles pour payer les frais de scolarité de ses filles . Chez un prêteur sur gage du centre commercial de à Singapour , dépose un bracelet , une bague et une paire de boucles d' oreille en or pour payer les frais de scolarité de ses filles . -Lors d' une réunion de campagne , il a dit à son personnel qu' il était nécessaire de se remettre sur la route et de rester fidèle à l' objectif de sa cause . Lors d' une réunion d' , il a dit à son équipe qu' il fallait qu' ils se remettent sur la bonne voie et restent fidèles à leur cause . - . Vers mercredi , une personne à la vue qui son chien a repéré Ruby sur une corniche dans la carrière , coincée à 15 m au-dessus du sol . -Au début , j' étais dans le déni de sa mort , et je l' ai parlé dans le présent . Au début j' ai nié sa mort , je parlais de lui au présent , -Lors de son investiture devant la Cour suprême constitutionnelle , Morsi est également devenu le premier président arabe musulman librement élu et le cinquième chef d' État de l' Égypte depuis le renversement de la monarchie il y a environ 60 ans . Lors de son investiture devant la Haute cour constitutionnelle , M. Morsi est également devenu le premier président islamiste librement élu du monde arabe et le cinquième chef d' État d' Égypte depuis le renversement de la monarchie il y a 60 ans . -C' est du moins ce que j' ai pensé lorsque j' ai vu la coquille de son corps fort sous les feuilles du lit d' hôpital . C' est du moins ce que je pensais le jour où je trouvais son corps vide , sur ce lit d ’ hôpital aux draps . -Au moins un groupe en ligne a déclaré qu' il était responsable de la fermeture du site NSA avec une attaque . Au moins un groupe d' en ligne a affirmé être responsable de la panne du site de la NSA par une attaque DSD . -Au moins six autorités européennes - la Commission européenne , , l' autorité de la concurrence suisse , la , le ministère de la Justice des États-Unis et l' Autorité monétaire de Hong Kong - se penchent sur les allégations selon lesquelles les banquiers pour faire bouger le marché des devises . Dans le monde , au moins six autorités – la Commission européenne , , l' autorité de la concurrence suisse , la FCA , le Département américain de la justice et l' Autorité monétaire de Hong Kong – examinent les allégations faites concernant des banquiers qui se seraient entendus pour manipuler les taux sur le marché des changes . -Au moins deux fois plus tôt cette année , Israël a lancé des frappes aériennes sur les livraisons de missiles en Syrie . Au moins deux fois cette année , Israël a mené des frappes aériennes contre des cargaisons de missiles en Syrie . -Au moins deux personnes ont été blessées , selon la police locale . Selon la police locale , il y aurait au moins 2 blessés . -A ce stade , Titan pourrait choisir de transférer les machines à la Pologne ou à tout autre pays de l' Union européenne qui possède encore sa propre monnaie . Dès lors , Titan pourrait déménager les machines vers la Pologne ou vers tout autre pays de l' Union européenne qui a encore sa propre monnaie . -À ce propos , tout porte à croire que le quatrième bout du monde est différent , les galaxies , les danseurs s' , les spectateurs , les voix , la fumée , le passage , l' écoute des applaudissements , le battage autour d' un désordre librement développé qui reste sensuel . Là , tout devient permis : le quatrième mur tombe , les univers explorés se superposent , les danseurs , descendent parmi les spectateurs , , fument , se , jouent des codes des applaudissements , la sauce dans un désordre savamment calculé qui reste sensuel . -À ce moment , je n' ai plus peur de lui et j' ai pu l' aimer . Là , je n' ai plus eu peur de lui et j' ai pu l' aimer . -À cette époque , le président américain , élu sur la promesse de mettre fin à l' engagement militaire des Etats-Unis dans le pays , a brossé un tableau optimiste de la situation . À l' époque , le président américain , élu sur la promesse de mettre fin à l' engagement militaire dans ce pays , avait dressé un tableau optimiste de la situation . -Au , les fonctionnaires préfèrent le terme " transformation " ou " changement de continuité " à la fermeture . Même si , à l' , on préfère employer le terme de " transformation " ou de " changement dans la continuité " . -Au School Board , 62 % des élèves ont une langue maternelle autre que le français . À la commission scolaire , 62 % des élèves ont une langue maternelle autre que le français . -À l' , des stagiaires spécialisés en situations d' urgence se rendront pour cinq médecins généralistes lundi . A l' , les internes spécialisés en urgences laisseront place lundi à cinq internes de médecine générale . -À la Metropolitan Transportation Commission de la baie de San Francisco , les représentants du Congrès ont dit que le Congrès pourrait tout simplement s' occuper du Fonds pour l' interdiction de la route en faisant augmenter les taxes sur le gaz . À la Metropolitan Transportation Commission de la zone de la baie de San Francisco , les responsables disent que le Congrès pourrait très simplement gérer la faillite du Highway Trust Fund en augmentant les taxes sur l' essence . -À la , " La Belle et la " pourrait être l' un des succès de la saison . À , " La Belle et la " pourrait bien être un des succès de la saison . -Lors de la conférence annuelle de l' American Academy of Pediatrics cette semaine à Orlando , en Floride , , le Dr Paul , a expliqué que les premiers changements physiques sont très fréquents chez les filles américaines et représentent une nouvelle norme . Lors de la conférence annuelle de l' American Academy of Pediatrics qui s' est tenue cette semaine à Orlando , en Floride , le Dr Paul , , a expliqué que ces changements physiques précoces sont assez courants parmi les jeunes Américaines et représentent une nouvelle norme . -À la fin de l' après-midi , plusieurs dizaines de soldats bien équipés se dirigeaient vers l' avant . En fin d' après-midi , plusieurs dizaines de soldats , bien approvisionnés en munitions , montaient en direction de la ligne de front . -Pour l' instant , il y a une sorte de parti pris à l' égard de l' Europe et surtout de l' Italie . Pour le moment , il y a une sorte de tendance à privilégier l' Europe et en particulier l' Italie . -Au piano , les danseurs prennent des tours pour jouer les notes . Au piano , des danseurs se relaient pour jouer les partitions . -Au prix d' un paquet de cigarettes , le lecteur reçoit un livre portant la couverture traditionnelle , avec une photo d' une jeune femme dotée d' un grand corps , munie d' un pistolet ou d' un fusil d' assaut . Pour le prix d' un paquet de cigarettes , le lecteur a droit à la traditionnelle couverture avec la photo d' une jeune femme à la poitrine avantageuse , portant un pistolet ou un fusil d' assaut . -Au début de cette semaine , une famille a abandonné son appartement du rez-de-chaussée dans le bâtiment D2 en raison de la présence de rats . En début de semaine , une famille a quitté son appartement en rez-de-chaussée de l' immeuble D2 , en raison de la présence de rats . -À leur sommet de novembre , les insurgés ont envahi Goma , la capitale de la province du Nord-Kivu , en profitant du retrait de la garnison du gouvernement et de l' inaction des bérets rouges . A leur apogée en novembre , les insurgés ont occupé Goma , chef-lieu de la province du Nord-Kivu , profitant de la fuite de la garnison gouvernementale et de la passivité des casques bleus de la . -Attaque sur un tunnel creusé par des Palestiniens Une attaque contre un tunnel creusé par les Palestiniens -Des tentatives de réanimation ont été faites par les services d' urgence , mais sans succès . Des manoeuvres de réanimation ont été tentées par les services d' urgence , mais sans succès . -Les d' Akihabara lancent le " Party de la " . Un habitant d' Aubervilliers lance son Parti de la banlieue - ne s' inquiète pas de cette fluctuation du chiffre d' affaires de la société . M. ne s' inquiète pas de voir le nombre de clients fluctuer de la sorte . -Les passagers des compagnies aériennes australiennes devront continuer à leurs et leurs téléphones intelligents pendant le décollage et l' atterrissage , malgré les mesures prises aux États-Unis pour assouplir la réglementation sur les appareils . Les passagers aériens australiens devront continuer à éteindre leurs tablettes et leurs smartphones pendant le décollage et l' atterrissage malgré le geste des États-Unis visant à assouplir la réglementation relative aux appareils électroniques . -Les autorités australiennes ont précipité l' homme à l' hôpital d' Christmas , où il est décédé . L' homme a été transféré d' urgence par les autorités australiennes à l' hôpital de l' île Christmas , où il est décédé . -Les transporteurs australiens examinent la décision , qui exige des transporteurs américains qu' ils accomplissent un énorme travail pour satisfaire aux exigences , mais ils ont déclaré qu' ils n' avaient pas l' intention de changer leurs procédures . Les transporteurs australiens examinent la décision qui oblige les transporteurs américains à effectuer un travail important en vue de respecter les exigences , mais ont indiqué qu' ils n' ont pas l' intention dans l' immédiat de changer leurs procédures . -Les diplomates australiens ont joué un rôle clé dans la promotion des " objectifs de développement durable " pour remplacer les Objectifs du Millénaire pour le développement , qui sont arrivés à échéance en 2015 , devant le sommet des Nations Unies sur le développement durable qui a commencé à Rio de Janeiro . Les diplomates australiens ont joué un rôle clé dans la mise en avant des « objectifs de développement durable » destinés à remplacer les OMD , qui expireront en 2015 , avant le Sommet des Nations Unies sur le développement durable qui a commencé la veille à Rio de Janeiro . -La australienne fait appel de la peine de prison en Thaïlande Une fait appel de sa peine d' emprisonnement en Thaïlande -L' authentification portait initialement sur les aspects physiques ( aux frontières , dans les lieux protégés , dans une cafétéria , etc. ) ou sur une présence numérique ( se connectant à un ordinateur ) . L' authentification concernait au départ les accès physiques ( frontières , locaux protégés , cantines ... ) ou logiques ( ouverture d' un ordinateur ) . -Les autorités ont saisi tonnes de marijuana et de 327 livres de cocaïne dans le cadre de la découverte du tunnel , selon les registres judiciaires . Les autorités ont saisi 8,5 tonnes de marijuana et 148 kg de cocaïne après la découverte du tunnel , selon les dossiers judiciaires . -Bagdad a publié des chiffres officiels vendredi : 964 personnes ont perdu la vie le mois dernier : 855 civils , 65 policiers et 44 soldats . Bagdad a publié ce vendredi des chiffres officiels : 964 personnes ont perdu la vie le mois dernier : 855 civils , 65 policiers et 44 soldats . -Les banques sont par des messages instantanés et des courriels d' une année à l' autre pour rechercher des exemples d' actes répréhensibles . Les banques les messages instantanés et les emails envoyés depuis des années pour trouver des cas d' actes répréhensibles . -Barack Obama a ordonné à l' Agence de la sécurité nationale ( ) de cesser d' utiliser les grilles du Fonds monétaire international et de la Banque mondiale dans le cadre de ses activités de renseignement , a déclaré un des représentants officiels de l' ONU jeudi . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Barack Obama a donné ordre à l' Agence nationale de sécurité ( NSA ) de mettre fin aux écoutes qu' elle pratiquait sur le Fonds monétaire international et la Banque mondiale dans le cadre de ses activités de renseignement , indique un responsable américain , jeudi . -Barack Obama pour rencontrer le Premier Ministre iraquien en tant que Barack Obama reçoit le premier ministre irakien , en pleine flambée de violence -La baronne Kramer a dit : " Notre objectif pour est d' établir un réseau véritablement national qui rapproche le Royaume-Uni et ses villes . La baronne Kramer a déclaré : « Notre objectif pour porte sur un réseau véritablement national qui rapprochera le Royaume-Uni et ses villes . » -Quoi qu' il en soit , son projet , qui a été présenté lors d' une conférence de presse dans sa ville natale d' Aubervilliers le 8 juin , est le fruit d' une volonté de promouvoir le multiculturalisme - le grand creuset culturel qui , plus que tout , caractérise ces districts . Il n' empêche , son projet , présenté jeudi en conférence de presse dans sa commune natale d' Aubervilliers , est né de la volonté de mieux défendre le multiculturalisme , ce grand pot culturel qui caractérise avant tout les quartiers . -Parce que ses récepteurs sont défectueux , Katie a développé des caractéristiques féminines . En raison de la défaillance de ses récepteurs d' , Katie a développé des caractéristiques féminines . -En raison du coût de ces programmes , les coûts d' investissement annuels ont augmenté de 7 millions de dollars en deux ans et devraient atteindre millions de dollars en 2013 . En raison du coût de ces projets , les dépenses d' investissement annuelles ont augmenté de sept milliards de dollars en deux ans et devraient atteindre 36,7 milliards de dollars en 2013 . -Derrière le différend est une course à l' ordre des commandes avec au moins $ des activités estimées à la cotation des prix dans les prochaines années , dans les temps à attirer la balance des exportations américaines et des exportations européennes . Derrière la dispute se cache une course aux commandes d' avions pour un montant estimé d' au moins 700 Md $ au prix du marché dans les décennies à venir , suffisamment pour faire pencher la balance des exportations américaines et européennes . -Derrière les écrans de télévision , il y a toujours ceux qui sont battus pour des chanteurs réussis , saison après saison . Derrière les écrans de télévision , les vocations de chanteur à succès sont toujours là , saison après saison . -Mouvement de libération du Pakistan ( ) Pékin met en cause un mouvement islamique turkmène -Pékin Groupe d' invasion de Morelos Pékin accuse un groupe de l' attaque de Tiananmen -Les supporters belges de qui veulent se rendre au match de la Ligue des Champions devront utiliser le système de transport fourni par . Les supporters belges du qui voudront se rendre à cette rencontre de la Ligue des Champions devront obligatoirement utiliser le système de déplacement mis en place par le . -Ben : Le dixième anniversaire du New York Festival : The New Ben : Dixième anniversaire du New York Festival : The New -Le chiffre d' affaires brut des États-Unis pour la livraison de décembre a diminué de 14 cents à $ le baril à la fin du mois de février en Europe dans le commerce électronique sur le New York Exchange . Le prix du brut de référence américain pour livraison en décembre a baissé de 14 cents à $ le baril en fin de matinée en Europe lors des échanges électroniques sur le New York Exchange ( ) . -Bernard , ancien directeur de l' agence de renseignement interne française , a été cité dans une interview de ce mois-ci : " Les services savent très bien que tous les pays , même lorsqu' ils coopèrent dans le cadre de la lutte anti-terroriste , à leurs alliés . & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; Bernard , ancien directeur du renseignement intérieur français , la , a déclaré dans un entretien ce mois-ci : « Les services savent parfaitement bien que tous les pays , même s' ils coopèrent dans la lutte contre le terrorisme , leurs alliés . » -Bernard rappelle également la pression qui s' sur ses épaules , surtout lors des dîners d' État à l' Elysée , où il devait servir " entre 200 et 250 personnes à une heure " . Bernard se souvient aussi de la pression qui pesait sur ses épaules , en particulier lors des dîners d' Etat à l' Elysée : il faut servir " entre 200 et 250 personnes en une heure environ " . -Bernard Didier , vice-président de , pense que c' est le " siècle de la biométrie " . Bernard Didier , directeur général adjoint de , estime même que " le siècle sera biométrique " . -En plus du multiculturalisme , a élaboré son logo , qui peut être consulté sur l' Internet , qui peut être consulté sur divers sujets destinés aux habitants des banlieues , en particulier les jeunes : le droit de voter pour les étrangers , la légalisation du cannabis , l' abaissement de l' âge de 16 ans , le rétablissement de la fonction de gardien , etc. Outre le multiculturalisme , a bâti son programme sur à partir de nombreux thèmes censés parler aux habitants de banlieue et notamment aux jeunes : le droit de vote pour les étrangers , la légalisation du cannabis , la majorité civile à 16 ans , la réhabilitation de la fonction de gardien d' immeuble ... -En plus de ce spectacle , la Société d' Art du Royaume Uni les " Destinations " , un programme qui a été à La à Chicoutimi . Outre ce spectacle , la Société d' art lyrique du Royaume reviendra avec ses Destinations lyriques , une formule qui fait désormais salle comble à la de Chicoutimi . -Entre 1995 et 2001 , , également Président de la Banque de , institution publique , a permis aux entreprises et aux gestionnaires de gagner des contrats , des prêts et des promotions en échange d' investissements ou de cadeaux , le Tribunal primaire intermédiaire de Beijing a déclaré à son . Entre 1995 et 2001 , le , qui était également le président de la Banque de , un établissement public , a aidé entreprises et responsables à obtenir contrats , prêts ou promotions en échange d' argent ou de cadeaux , a déclaré le premier tribunal intermédiaire de Pékin sur son . -Entre le début des années 70 , lorsque le Boeing 747 a défini les voyages long-courriers modernes et le temps du siècle , le poids de l' Américain moyen de 40 à 60 ans a augmenté de 10 % , selon les données du U.S. Department of Health . Entre le début des années 1970 , lorsque le 747 de Boeing a défini le voyage long-courrier moderne , et le tournant du siècle , le poids de l' Américain moyen de 40 à 49 ans a augmenté de 10 % , selon les données du département américain de la Santé . -Au-delà du budget actuel et des problèmes monétaires , il y a un malaise économique croissant pour la santé . Au-delà des problèmes budgétaires et monétaires actuels , il existe un malaise plus profond lié à la productivité en Europe . - a également publié un document de la mairie du quartier de Mitte , indiquant qu' il avait été enterré dans le cimetière juif du district . publie également un document de la mairie d' arrondissement du quartier de Mitte à Berlin indiquant qu' il a été enterré dans le cimetière juif du quartier . -La biométrie sera le seul moyen de garantir l' identité d' une personne effectuant des opérations dans un monde aussi transversal et transnational que Internet . Seule la biométrie permettra de s' assurer de l' identité d' une personne réalisant des transactions dans un univers aussi et transnational qu' Internet . -La biométrie s' appuiera sur de nouvelles technologies qui permettront de fournir de nouveaux services aux citoyens et aux consommateurs . La biométrie va s' appuyer sur de nouvelles techniques , qui permettront de rendre de nouveaux services aux citoyens et aux consommateurs . -Bird to safety from North Sea to be back in Un oiseau d' une plate-forme en mer du Nord a été relâché dans la nature -Les prétentions de rendement de Boeing reposent en partie sur la comparaison de l' avec un design original de 777 . Les performances annoncées par Boeing dépendent en partie de la comparaison entre le configuré avec 10 sièges par rangée et un modèle 777 d' origine configuré en rangées de 9 . -Boeing conteste les chiffres d' Airbus concernant les mesures des sièges et déclare qu' il n' appartient pas aux constructeurs de prendre des décisions sur la manière dont les compagnies aériennes fixent les tarifs et les installations . Boeing conteste les chiffres d' Airbus en ce qui concerne la largeur des sièges et affirme que ce n' est pas aux constructeurs de décider de la façon dont les compagnies aériennes équilibrent le prix des billets et les équipements . -Selon Boeing , son " " 406 personnes sur la base d' une économie de plus de 17 pouces de large et 10 dans chaque rangée . Boeing déclare que son « » réaménagé pourra accueillir en classe économique 406 personnes dans des sièges de plus de 17 pouces de large , et sera configuré avec 10 sièges par rangée . -Bombardier , la plus grande du monde , a déclaré que les revenus de cette division ont augmenté de près de 11 % pour s' établir à 2,1 milliards de dollars . Bombardier , le plus grand constructeur ferroviaire au monde , a déclaré que les revenus de cette division avaient augmenté de près de 11 % à 2,1 Md $ . -Bombardier espère que la famille d' aéronefs pourra la dans le bas d' un marché qui est aujourd' hui dominé par Boeing et Airbus . Bombardier espère que la famille d' avions pourra la dans le segment inférieur d' un marché aujourd' hui dominé par Boeing et Airbus . - des bénéfices de Bombardier à la suite de la livraison d' aéronefs , les commandes prennent fin . Le bénéfice de Bombardier en baisse , alors que les livraisons et les commandes d' avions reculent -Bombardier a déclaré qu' il évaluerait le calendrier de l' ( DIE ) et qu' il fournira une mise à jour dans les prochains mois . Bombardier a indiqué qu' elle était en train d' évaluer le calendrier d' entrée en service et fournira de nouvelles informations au cours des prochains mois . -Des marchés de jets de grève , des mosquées , des mariages et des cérémonies funéraires . Les bombes frappent des marchés , des mosquées , des mariages et des funérailles . -Né le 24 octobre 1953 à La , l' homme avec un visage ouvert , une tête et des verres de l' intellectuel est un titulaire d' un certificat de compétence professionnelle dans le domaine de la confiserie . L' homme au visage ouvert , crâne et lunettes d' intellectuel , né le 24 octobre 1953 à la est titulaire d' un CAP de pâtisserie . -Tant les que les insistent pour qu' ils évitent la guerre . Le et la insistent pour dire qu' ils souhaitent tous les deux éviter la guerre . -Airbus et Boeing demandent 20 % plus d' efficacité par siège dans leurs plus récentes long-courriers que le leader du marché dans ce segment , le Boeing . Airbus et Boeing revendiquent tous les deux un rendement par siège dans leurs derniers modèles de long-courriers 20 % supérieur à celui du leader du marché dans ce segment , le Boeing de 365 sièges . -Tant les que les , ainsi que d' autres jeunes avec des voix étranges ( le Anthony , le , W. Bell ) , sont des produits du monde de la comédie américaine . et , ainsi que d' autres jeunes comédiens à la voix singulière ( le pessimiste surréaliste Anthony , l' ironique défenseur de la justice raciale W. Bell ) , sont des purs produits du monde décentralisé de la comédie américaine . -Les garçons - comme les filles - atteignent la puberté plus tôt . Les garçons , comme les filles , atteignent leur puberté plus tôt . -L' entreprise brésilienne Embraer , la troisième plus importante commerciale au monde et le rival le plus proche de Bombardier , a enregistré une baisse de 10 % du chiffre d' affaires trimestriel . La société brésilienne Embraer SA , troisième constructeur aéronautique au monde et concurrent le plus proche de Bombardier , a indiqué jeudi une baisse de 10 % de son bénéfice trimestriel . -Brent Gross , une référence pour le brut international également utilisé par les raffineries américaines , a chuté de 26 à $ le baril sur l' ICE à Londres . Le brut Brent , une référence pour le brut international déjà utilisé par les raffineries américaines , a perdu 26 cents pour tomber à $ le baril sur l' Exchange ( ICE ) à Londres . -La police britannique avec notification d' extradition La police britannique délivre un mandat d' extradition à l' encontre d' -Les scientifiques britanniques ont dit que les résultats étaient intéressants , mais ont souligné que de nombreuses recherches étaient nécessaires . Des scientifiques britanniques ont déclaré que les résultats étaient intéressants mais ont signalé que des recherches bien plus approfondies étaient nécessaires . -Bruce est né avec un qui a empêché son corps de produire suffisamment de testostérone pour développer correctement ses organes génitaux . Bruce est né avec un désordre du développement sexuel ( DDS ) qui a empêché son corps de produire suffisamment de testostérone pour que ses organes sexuels puissent se développer . -Les consommateurs bulgares recevront du gaz de South Stream à un tarif réduit une fois que le projet aura démarré en 2015 . Les consommateurs bulgares recevront le gaz de South Stream à un tarif réduit dès que l' ensemble du projet sera mis en service en 2015 . -Business for the Britain a été lancé en avril , avec une promesse de partager les affaires et de définir ce que la richesse et les créateurs d' emplois du Royaume-Uni veulent voir changer dans nos relations avec l' UE . La campagne Business for Britain a été lancée en avril avec la promesse de réunir les entreprises et de définir ce que les créateurs de richesses et d' emplois veulent voir changer dans notre relation avec l' UE . -Les dirigeants d' entreprises sont sceptiques à ce sujet . Les dirigeants d' entreprise sont sceptiques à ce sujet . -Les entreprises peuvent également être tenues de prendre en compte le nombre de 457 titulaires de visa après que d' autres entreprises ont commencé à se tourner vers un petit nombre de travailleurs qui ont alors employé des centaines d' autres . Les entreprises pourraient également être tenues de déclarer le nombre de détenteurs de visas 457 après que certaines d ’ entre elles , ayant eu l' intention de un petit nombre de travailleurs , en ont en fait employé des centaines . -Mais , après les innombrables moments où je me suis servi de la défense de l' administration Obama à ses collègues et à ses pairs , j' ai atteint une limite aux explications que je peux fournir . Mais après avoir maintes fois défendu l' administration d' Obama face à mes pairs et mes collègues , je n ’ arrive plus à fournir d' explications . -Mais , comme je vous l' ai dit , la CGT et Goodyear doivent d' abord parvenir à un accord sur l' indemnité de licenciement . Mais , comme je vous l' ai dit , la CGT et Goodyear doivent d' abord se mettre d' accord sur des indemnités de départ . -Mais si le prix est juste et que les travailleurs sont qualifiés , il vaut la peine d' essayer . Mais , avec un prix correct et des travailleurs compétents , cela vaut le coup d' essayer . -Mais dans l' esprit de nos anciens dirigeants , l' identité française était une essence intemporelle . Mais dans l' esprit péremptoire de nos dirigeants d' alors , l' identité française était une essence intemporelle . -Mais , de plus en plus , l' authentification implique également des transactions , en particulier celles effectuées à l' aide de dispositifs connectés à l' Internet . Mais , de plus en plus , l' authentification va concerner aussi les transactions , en particulier celles réalisées à partir d' appareils connectés à Internet . -Mais sur Twitter , les gens veulent un spectacle , l' émotion , l' originalité , la musique ... Or , sur Twitter , on veut du spectacle , de l' émotion , de l' originalité , de la musique ... -Mais Anne , directrice exécutive du groupe juridique californien pour les conseils avisés , s' inquiète de ce que la loi allemande " incite à l' étiquetage et à la stigmatisation " . Mais Anne , directrice générale de l' organisation à but non lucratif basée en Californie Advocates for Choice , craint que la loi allemande « invite à l' étiquetage et à la stigmatisation » . -Mais Berlin n' a plus besoin de Paris en tant que passeport pour la légitimité politique et a imposé sa propre politique économique à l' Europe , laissant l' économie française battue en difficulté . Mais Berlin n' a plus besoin de Paris comme passeport de sa légitimité politique et a imposé sa propre politique économique en Europe , laissant l' économie française mise à mal se . -Mais l' analyse de l' ADN prend encore beaucoup de temps et coûte cher . Mais l' analyse de l' ADN est encore très longue et très coûteuse . -Mais Dr Arlene , un de la Fédération de , qui a une fille qui souffre d' un trouble du développement sexuel et qui a aidé des centaines d' autres membres d' un groupe de soutien , a déclaré que la loi allemande " les parents et les enfants " . Mais le Dr Arlene , spécialisée dans les mammographies à Pittsburgh , dont la fille est atteinte d' un désordre du développement sexuel et qui aide des milliers de personnes dans un groupe de soutien , a déclaré que la loi allemande « » les parents et les enfants . -Mais je n' ai pas de préjugés à propos de la France . Mais je n' ai pas de préjugés sur la France . -Mais M. Chen a par la suite admis à la télévision qu' il avait pris des pots-de-vin pour fabriquer des articles sur une société appartenant à l' État . Mais M. Chen a par la suite admis à la télévision qu' il avait accepté des pots-de-vin pour fabriquer des histoires au sujet d' une société . -Mais M. Chen s' est alors rendu à la télévision de l' État déclarant qu' il avait publié de fausses histoires pour de l' argent . Mais M. Chen est ensuite apparu sur une chaîne de télévision publique où il a admis avoir publié de fausses histoires contre de l' argent . -Mais M. Hare a rejeté la suggestion selon laquelle le YMCA avait un problème culturel qui empêchait le personnel de signaler les manquements à la sécurité de l' enfant commis par lord . Mais M. Hare a rejeté l' idée selon laquelle le YMCA aurait un problème culturel empêchant le personnel de signaler les manquements de Lord envers la sécurité des enfants . -Mais M. Palmer a affirmé que ce ne sont que les ministres qui ont eu un conflit d' intérêts et que les filles d' Abbott ont personnellement profité des politiques . Mais M. Palmer a prétendu que seuls des ministres pourraient avoir un conflit d' intérêt et a indiqué que les filles de M. Abbott étaient prêtes à bénéficier personnellement de ces politiques . -Mais le projet de loi ( ) , président de la commission des transports de la Chambre des communes , a également estimé qu' il s' agissait là de la solution la plus viable à long terme . Mais le représentant Bill ( . ) , président du Comité des transports de la Chambre des représentants , a déclaré qu' il le considérait aussi comme l' alternative la plus viable à long terme . -Mais une proposition d' augmentation de la demande est venue le vendredi de deux rapports sur la fabrication chinoise qui montrent un dans l' activité . Mais deux rapports sur le secteur manufacturier chinois arrivés vendredi , qui montrent une croissance de l' activité , suggèrent une hausse de la demande . -Mais après l' audience , Lee , de , de la police de , a déclaré : " M. a été renvoyé de l' avocat criminel à un criminel dès le moment où il a commencé à recevoir des drogues de groupes criminels " . Mais après l' audience , le Lee de la Police du a déclaré : « M. est passé d' avocat à criminel lui-même dès lors qu' il a commencé à se procurer de la drogue auprès d' organisations criminelles . » -Mais à 38 ans , il s' est suicidé . Mais à 38 ans , il s' est suicidé . -Mais en acceptant de prendre le serment officiel devant le tribunal , plutôt que devant le Parlement comme étant coutumier , il s' accroche à la volonté de l' armée pour montrer que le concours pour le pouvoir se poursuivra . Mais en acceptant de prêter officiellement serment devant la Cour , plutôt que devant le Parlement comme il est d' usage , il s' incline devant la volonté de l' armée en montrant que la lutte pour le pouvoir n ’ est pas terminée . -Mais les inquiétudes se sont accrues après que M. eut dit que avait abandonné l' accord de paix de 1992 . Mais l' inquiétude a grandi après que M. a déclaré que la l' accord de paix de 1992 . -Mais la connexion à Internet pour surfer , échanger des courriels , le texte ou les données de téléchargement sera toujours interdite à moins de 10,000 pieds , a déclaré l' agence . Mais il sera toujours interdit de se connecter à Internet pour surfer , échanger des emails et des SMS ou télécharger des données en dessous de 10 000 pieds , a déclaré l' agence . -Mais déprimé et connaissant quelque chose de mal , il a exigé des dossiers médicaux . Mais déprimé et se rendant compte que quelque chose n' allait pas , il a demandé son dossier médical . -Mais chacune de ces techniques a ses limites . Mais chacune de ces techniques possède ses contraintes . -Mais depuis que Edward , l' homme dirigeant , a commencé à son dernier trésor de secrets de surveillance américains , les gouvernements et les chefs d' entreprise européens ne sont plus tenus de prendre le directeur à sa parole . & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; Mais depuis qu' Edward , le consultant devenu , a commencé à dévoiler son inépuisable trésor d' informations sur les secrets de la surveillance des États-Unis , les gouvernements européens et les dirigeants d' entreprises ne savent plus s' il faut croire le directeur sur parole . -Mais le porte-parole de la santé publique , Peter McLean , estime que l' annonce d' aujourd ' hui est une " solution " . Mais le porte-parole de l' opposition en matière de santé Peter Dutton pense que l' annonce d' aujourd' hui est une « une solution de fortune » . -Mais les commandes fermes du sont à peu près au point 177 , car les acheteurs potentiels attendent les résultats des essais de vol pour confirmer les affirmations de la société quant à l' efficacité du nouveau système d' alimentation en huile et au potentiel d' économies . Mais le nombre de commandes fermes pour les avions est modeste jusqu' à présent à hauteur de 177 . Il semble que les acheteurs potentiels attendent les résultats des essais en vol pour valider les affirmations de la société concernant l' efficacité énergétique et le potentiel d' économies du nouvel avion de ligne . -Mais il a nié les changements intervenus à la demande de M. Obeid , qui , M. , a reconnu qu' il était urgent de modifier la politique de location du gouvernement . Mais il a nié avoir changé sa politique à la demande de M. Obeid qui , a reconnu M. , réclamait un changement de la politique gouvernementale en matière de baux . -Mais sa prochaine création sera encore plus ambitieuse . Mais sa prochaine création s' annonce encore plus ambitieuse . -Mais dans sa déclaration liminaire à la Commission , elle a déclaré qu' elle n' avait jamais " traité d' incident sexuel impliquant des enfants à l' intérieur de son organisation " , a déclaré la Commission . Mais dans sa déclaration d' ouverture devant la Commission , le YMCA a déclaré qu' il « n' avait jamais eu à traiter d' un problème d' agression sexuelle sur des enfants au sein de son organisation » . -Mais dans un cas de 1966 , connu sous le nom de " John / Joan " , ses théories sont devenues controversées . Mais dans un cas de 1966 , connu sous le nom de « John / Joan » , ses théories ont donné lieu à controverses . -Mais dans les années 1990 , avec l' arrivée de l' Internet , les survivants de ces interventions sexuelles se sont dits " pas contents du résultat " . Mais dans les années 1990 , avec l' arrivée d' Internet , ceux qui ont survécu à ces opérations de changement de sexe disent « [ qu ' ] ils ne sont pas satisfaits du résultat » . -Mais il s' agit aussi de la connectivité , dans tout le Royaume-Uni 18 villes comme Glasgow et Édimbourg seront mieux connectés en raison de . Mais il s' agit également de développer la connectivité entre les 18 grandes villes britanniques , y compris Glasgow et Édimbourg , afin que les liaisons entre elles soient meilleures grâce à . -Mais les souris qui n' avaient pas reçu le vaccin " " , c' est-à-dire les chercheurs , montraient que la nicotine avait atteint leur cerveau . Mais les souris qui n' avaient pas été vaccinées « étaient apaisées » , ont déclaré les chercheurs , signe que la nicotine avait atteint leur cerveau . -Mais il est plus probable qu' il soit utilisé par les fumeurs pour arrêter de fumer . Mais il est plus probable qu' il soit utilisé par les fumeurs qui veulent arrêter de fumer . -Mais personne n' était prêt pour un bébé de 3 ans , Kirkland . Mais personne ne s ’ attendait à voir gagner Kirkland , un enfant de trois ans . -Mais à l' heure actuelle , plusieurs d' entre nous remettent en question le fondement éthique de l' intervention chirurgicale , sachant que l' identité des sexes est très complexe , et que les médecins peuvent parfois se tromper , sans savoir comment un enfant peut se sentir à l' aise lorsqu' il s' agit d' élever sa femme . Mais désormais , nombreux sont ceux qui remettent en cause le fondement éthique de la chirurgie , sachant que l' identité sexuelle est complexe et que parfois les médecins peuvent se tromper , ne sachant pas comment un enfant sa sexuelle lorsqu' il . -Mais aujourd' hui , les scientifiques disent que les autres peuvent frapper et répondre à ces subtiles différences de . Mais désormais les scientifiques disent que les chiens peuvent détecter ces différences subtiles et y répondre . -Mais quand ils arrivent à Yaoundé , leur conseil , transport et instruction sont couverts par . Mais une fois à Yaoundé , leur pension , leur transport et l' enseignement sont pris en charge par le docteur . -Mais une partie de son camp , qui se concentre sur le cercle présidentiel du Parti de la liberté sociale ( ) , , continue de soutenir le gouvernement , comme il l' a montré le 2 octobre lorsqu' il a refusé de suivre les ordres de Berlusconi sur une tentative précédente de mettre le cabinet . Mais une partie de son camp , regroupée autour du secrétaire national du Peuple de la liberté ( PDL ) , continue à soutenir le gouvernement , comme l' a démontré le 2 octobre dernier son refus de céder aux injonctions de Berlusconi qui souhaitait déjà faire chuter le cabinet . -Mais les passagers ne devraient pas s' attendre à ce que les changements surviennent immédiatement . Mais les passagers ne devraient pas s' attendre à des changements immédiats . -Mais les projets en cours comme " Rent " , " Le de la Femme " ou " L' du Printemps " ont des difficultés à trouver des finances . Mais des projets en cours comme " Rent " , " Le de la femme araignée " ou " L' Éveil du ­ printemps " ont du mal à être financés . -Mais elle continue d' influencer ma vie , surtout quand je suis à des moments difficiles , ou quand je dois prendre une décision importante . Mais elle continue d' influencer ma vie , notamment dans les moments où je me trouve en difficulté , ou bien lorsque je dois prendre des décisions importantes . -Mais elle est maintenant bonne . Mais elle va bien maintenant . -Mais les statistiques montrent que je suis un mouton noir quand il s' agit d' Halloween . Mais les stats montrent que je suis la brebis lorsqu' on parle d' Halloween . -Mais le procureur général du Texas , le républicain Greg Abbott , qui travaille pour le rôle du gouverneur , a demandé à la cour d' appel d' annuler le jugement du tribunal qui l' application de la loi . Mais l' Attorney General du Texas , le républicain Greg Abbott , qui le poste de gouverneur , a demandé à la cour d' appel d' annuler le jugement de première instance qui l' application de la loi . -Mais les dirigeants de la Chambre ont pris la tête de cette proposition , invoquant les préoccupations des législateurs ruraux représentant des électeurs dont la vie quotidienne implique souvent des milliers de kilomètres pour se rendre à la ville ou à la ville . Mais la présidence de la Chambre a tué la proposition , répondant aux préoccupations des législateurs ruraux représentant des électeurs qui doivent parcourir de nombreux miles tous les jours pour se rendre à leur travail ou en ville . -Mais le bleu choisi par le ministre est " trop sombre " , selon le député de Bruxelles , Richard De , qui négocie le " " de la capitale et exige la démission de Brigitte . Mais le bleu choisi par la ministre est " trop foncé " , selon le député bruxellois Emmanuel De qui parle de " " de la capitale et exige la " démission " de Brigitte . -Mais l' approche chorégraphique manque de composition . Mais l' approche chorégraphique manque de composition . -Mais la controverse entre les Américains et les Européens a continué de croître cette semaine avec de nouvelles révélations dans la presse . Mais la controverse entre Américains et Européens a continué d' cette semaine avec de nouvelles révélations dans la presse . -Mais le coût du nouveau vaccin sera probablement beaucoup plus bas , car il transforme les cellules du foie en laboratoires d' anticorps . Mais le coût du nouveau vaccin devrait être bien inférieur car il transforme les cellules du foie en usines à anticorps . -Mais le tribunal a entendu dire qu' il des grands acteurs de la chaîne d' approvisionnement en médicaments , dont certains auparavant , après des arrestations clés , lui ce que les détectives savaient de lui . Mais le tribunal a entendu des témoignages selon lesquels il appelait des acteurs majeurs dans le commerce de la drogue , dont certains qu' il avait représentés , après des arrestations clés pour leur dire quels enquêteurs étaient au courant . -Mais la décision de fermer le service d' urgence relève de l' Office régional de la santé ( ARS ) . Mais c' est à l' ARS qu' il appartient de décider la fermeture du service d' accueil des urgences . -Mais les appareils font appel à des tests pour les planificateurs de l' autoroute parce qu' ils n' utilisent pas le GPS et fournissent un nombre limité d' informations , qui sont régulièrement mises à jour par modem . Mais les appareils qu' elle teste ont plu aux planificateurs des grands axes car ils n' utilisent pas le GPS et fournissent une quantité limitée d' informations , téléchargées périodiquement par modem . -Mais la pièce prend vraiment le dessus après l' arc . Mais la pièce prend son réel envol après le salut . -Mais les services secrets étaient totalement erronés . Les services secrets se totalement . -Mais il y a aussi des niveaux élevés de taxation et de chômage en France . Mais la France a un haut niveau d' impôt et de chômage aussi . -Mais il n' y a eu qu' un léger changement dans l' âge des premières règles ( la première période ) au cours des quatre dernières décennies . Mais il n' y a eu qu' une légère avancée de l' âge des premières règles au cours des quatre dernières décennies . -Mais elles sont régulièrement saturées . Or ils sont régulièrement . -Mais cela n' a jamais été . Mais ce n' était jamais . -Mais cela est allé au-delà des limites habituelles d' une relation . Mais cela est allé bien au-delà des limites ordinaires d' une relation entre un avocat et son client . -Mais le temps a atténué la douleur . Puis , le temps a apaisé la peine . -Mais les transmissions sont plus fortes lorsque les appareils téléchargent ou envoient des données . Mais les transmissions sont plus fortes lorsque les appareils téléchargent ou envoient des données . -Mais est-ce le reflet des attentes ? Mais le spectacle était-il au rendez-vous ? -Mais nous nous sommes retrouvés dans la partie la plus difficile . Mais on a fait le plus dur . -Mais nous n' en sommes pas là auparavant - tout est complètement nouveau . Mais nous n' avons jamais connu une telle situation – elle est totalement nouvelle . -Mais nous n' avons pas vu comment la loi va jouer , et tout ce que nous pouvons faire , c' est spéculer . Mais nous n' avons pas vu ce que la loi allait donner , donc nous ne pouvons qu' avancer des hypothèses . -Mais lorsqu' ils arrivent , ils reçoivent un véritable cadeau des héros : un festin , un chant et une danse , et les meilleurs hébergements que la communauté puisse offrir . Mais lorsqu' ils arrivent , ils sont reçus en héros : la communauté organise une fête , chante et danse et leur offre les meilleurs logements disponibles . -Mais lorsqu' ils ont vu une queue se vers la gauche ( avant tout du point de vue du chien ) , leur taux cardiaque s' est élevé et ils ont éveillé l' anxiété . Mais lorsqu' ils ont repérés une queue dévier principalement vers la gauche ( encore une fois du point de vue du chien qui la queue ) , leur fréquence cardiaque s' est accélérée et ils avaient l' air anxieux . -Mais qui dit que les otages peuvent aussi avoir accès à la justice ? Mais qui dit que ces otages pourraient peut-être avoir eux aussi accès à la justice ? -Mais pourquoi un tel optimisme pour certains et le pessimisme pour les autres ? Les raisons d' un tel optimisme , chez les uns , et pessimisme , chez les autres ? -Mais pourquoi ont-ils été si déçus ? Mais pourquoi tant de déception ? -Mais avec Nicolas Sarkozy , qui a interdit le fromage , les plaques sont toujours vides ! Mais avec Nicolas Sarkozy , qui avait supprimé le fromage , les assiettes revenaient vides ! -Par un vote à cinq points , les juges ont adopté la loi sur le service à la personne et à la protection gratuite des patients - qui oblige les citoyens à acheter de l' assurance maladie d' ici 2014 ou à payer une pénalité - relève de la responsabilité du gouvernement en matière d' imposition . Par 5 votes contre 4 , les juges ont validé le mandat individuel du Patient Protection and Care Act ( loi sur la protection des malades et les soins abordables ) – qui exige que les citoyens souscrivent une assurance santé d' ici 2014 , sous peine de devoir payer une amende – et ont déclaré qu' il était constitutionnel en vertu du pouvoir d' imposition du gouvernement . -Dans tous les comptes , les travaux de fondation se sont déroulés sur le site depuis un certain temps . Selon toute vraisemblance , des travaux effectués sur les fondations se déroulaient à cet endroit depuis un certain temps . -À titre de comparaison , les écoles australiennes fournissent en moyenne 143 heures par an et les élèves environ 138 heures à Singapour . En comparaison , les écoles australiennes proposent en moyenne 143 heures par an , et les élèves de Singapour en font environ 138 heures . -En votant , nous envoyons un message clair . En allant voter , nous envoyons un message clair . -En les médias , les politiciens savent qu' ils peuvent faire ce qu' ils veulent , en toute impunité . En réduisant les médias au silence , les hommes politiques savent qu' ils peuvent faire ce qu' ils veulent en toute impunité . -D' ici à la fin de l' année , les médecins généralistes indépendants devraient également participer à la mise en oeuvre d' un " service de soins ambulatoires " . D' ici à la fin de l' année , des médecins généralistes libéraux devraient également venir participer à la mise en place d' une " permanence de soins ambulatoires " . -Les Centers for Disease Control and Prevention ( CDC ) publient des directives sur les allergies à l' intention des enfants . Le CDC publie des conseils sur les allergies chez l' enfant à destination des écoles -Les économistes de la Californie se tournent vers le système , dans la mesure où ils conçoivent des stratégies pour atteindre les objectifs fixés dans les lois ambitieuses sur le réchauffement de la planète adoptées par l' État . Les planificateurs de Californie s' intéressent au système puisqu' ils élaborent des stratégies pour atteindre les objectifs fixés dans les lois ambitieuses de l' État sur le réchauffement climatique . -Cameron , analyste de la Banque Financière Nationale , a réduit sa cote de " performance " de " " plus tard que jeudi , étant donné que le portefeuille a un rendement limité au cours de l' un ou l' autre des trimestres . Cameron , analyste au sein de la National Bank Financial , a abaissé jeudi sa notation , passant de « » à « performance de secteur » en raison du faible potentiel de hausse de la bourse au cours du prochain trimestre ou des deux prochains trimestres . -Pouvons-nous nous passer de collaborer avec les Américains ? Peut-on se priver de la collaboration américaine ? -Pouvez-vous le confirmer ? Le ? -Le fabricant d' avions de transport et de train Bombardier a annoncé une réduction de 15 % du bénéfice net du jeudi , une réduction des commandes d' aéronefs et des livraisons au troisième trimestre et des problèmes contractuels dans le secteur de la construction . Bombardier Inc , constructeur aéronautique et ferroviaire canadien , a signalé une chute de 15 % de son bénéfice net jeudi , sous la pression d' une baisse des commandes et des livraisons d' avions au cours du troisième trimestre et de problèmes contractuels dans sa division Trains . -Les cardinaux sont le plus haut dignitaire religieux de l' Église catholique sous le nom de pape , et c' est eux qui les papes ; c' est pourquoi ils leur premier groupe d' hommes qui , à leur tour , le successeur de l' un de ses successeurs . , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , etc. Les cardinaux sont les hommes d' église les plus haut placés dans l' Église catholique derrière le pape , et ceux qui l ’ élisent , donc François nommera son premier groupe d' hommes qui choisiront en définitive son successeur . -Pour sa part , Brad a remplacé les de Daisy dans le " Grand " pour les vêtements moins . Sa voix vocale est plutôt douce , mais sa langue est très quand elle parle . Quant à Carey , elle a quitté les riches atours de Daisy dans " le magnifique " pour des vêtements moins , une voix douce quand elle chante , mais le débité à la quand elle parle . -Les transporteurs devront prouver que leurs appareils peuvent tolérer l' interférence de la radio par les appareils mobiles et réviser les manuels , les matériaux de formation , les dossiers des bagages de cabine et les avis de passagers . Les transporteurs devront démontrer que leurs avions peuvent tolérer des interférences radio provenant d' appareils électroniques , de même que revoir les manuels , le matériel de formation , les programme de transport de bagages à main et les consignes fournies aux passagers . -Les voitures vont de plus en plus de miles au gallon . Les voitures peuvent parcourir plus de miles avec un gallon . - votre vote ici ou par l' intermédiaire de votre appareil mobile Votez ici ou via votre appareil mobile -Le directeur du centre , Colin , a déclaré : " Ce train d' eau est probablement un immigrant de l' Europe du Nord qui a été pris dans des vents forts en mer du Nord " . Le directeur du centre , Colin , a déclaré : « Ce d' eau était probablement un qui venait d' Europe du nord et s' est retrouvé pris au milieu de vents violents en mer du Nord . » -Cérémonie de Mémoire de la Une cérémonie en mémoire des défunts -Le changement s' est effectivement produit , dans les années qui ont suivi mon premier vote . Le changement s' est vraiment produit dans les années qui ont suivi la première fois où j' ai voté . - Transporteur aérien " " Le porte-avions en " indisponibilité " -L' entreprise expérimentale de Charlie , basée à Bristol , est célèbre pour ses saveurs inhabituelles , dont la bière , le fromage , le boeuf et la feuille d' or . La société à la démarche expérimentale de Charlie , basée à Bristol , est connue pour ses parfums de glace inhabituels tels que la bière , le fromage , le bœuf et la feuille d' or . -Charlie , fondateur de la compagnie " I Cool Cream " , a déclaré : " Il y a des choses incroyables , mais encore très tôt dans le domaine de la production , alors que £ 200 vous donne environ de la matière " . Charlie , le fondateur de la société de glaces « Me I' m Delicious » , a déclaré : « C' est quelque chose d' incroyable mais nous n ’ en sommes encore qu ’ aux débuts en termes de production , et 2 g de ce truc coûte environ 200 £ . » -Charlie Brown nous a présenté la Great quand nous étions des enfants , et la d' un est comme décorer un sapin de Noël - c' est quelque chose que nous avons fait parce que nous étions peu . Charlie Brown nous a présenté La Grosse quand on était , et sculpter une citrouille en forme de lanterne , c' est comme décorer un arbre de Noël – c' est quelque chose qu' on fait depuis qu' on est tout petit . -Charlie Francis a les propriétés de l' animal marin pour développer la . Charlie Francis a tiré parti des propriétés fluorescentes de cet animal marin pour développer une collation . -Jetez un coup d' oeil sur le site de et voyez comment vous aider . Visitez le site pour voir comment apporter votre aide . -Le principal concurrent de Chevron , Exxon , a également annoncé une chute des bénéfices nets jeudi , malgré l' augmentation de sa production pétrolière et gazière . Ainsi le principal concurrent de Chevron , Exxon Mobil a fait état jeudi d' un bénéfice net en baisse , malgré la hausse de sa production de gaz et de pétrole . -Chevron , la deuxième plus grande compagnie pétrolière d' Amérique , a annoncé une baisse des bénéfices trimestriels vendredi , en raison de la réduction de ses marges de raffinage , bien que sa production de pétrole et de gaz ait augmenté et qu' elle reste en deçà de l' objectif du groupe . Chevron , deuxième compagnie pétrolière américaine , a fait état vendredi d' une baisse de son bénéfice trimestriel en raison de la baisse de ses marges de raffinage , alors que sa production de pétrole et de gaz a augmenté tout en restant en retrait par rapport aux objectifs du groupe . -Le chef de la direction , Pierre Beaudoin , s' est assuré que Bombardier sa cible de commande de 300 jours au moment où le premier jet est mis en service . Pierre Beaudoin , président et chef de la direction , est convaincu que Bombardier atteindra son objectif de 300 commandes fermes avant la mise en service commercial du premier avion . -Le juge en chef John Roberts s' est rallié aux quatre membres les plus libéraux du tribunal alors que les juges , Thomas , et Kennedy étaient opposés . Le président de la Cour suprême , John Roberts , s' est rallié aux quatre membres les plus libéraux de la Cour , tandis que les juges , Thomas , et Kennedy ont exprimé leur désaccord . -Le chef de l' exécutif , M. Phillip Hare , a été interrogé sur un cas où un employé du Rotary a été accusé d' avoir commis des infractions à la pornographie juvénile , et un autre enseignant de la salle de gymnastique au YMCA de l' école a été reconnu coupable d' infractions sexuelles à l' égard d' enfants à l' égard de ses enfants en 1991 . Le directeur général Phillip Hare a été interrogé au sujet de deux affaires : un employé du YMCA a été accusé d' infractions relatives à la pornographie enfantine et un moniteur d' éducation physique du YMCA Hall a été reconnu coupable d' agressions sexuelles sur des enfants dans le centre où il travaillait en 1991 . -Enfant gravement blessé au dos de chez Disneyland Paris Un enfant grièvement blessé dans une attraction à Disneyland Paris -Les enfants ont besoin de stabilité et de certitude . Les enfants ont besoin de stabilité et de certitudes . -Les enfants jouent sur un char qui fait partie des rebelles , à , dans l' est de la RDC , le 31 octobre . Des enfants jouent sur un blindé qui appartenait aux rebelles du , à , le 31 octobre 2013 , dans l' Est de la RDC . -Les enfants devraient apprendre les mythes et les légendes comme des " modèles pour un mode de vie " , dit l' auteure . Il faudrait enseigner les mythes et légendes aux enfants comme « des modèles de style de vie » , déclare l' auteur . -Document de presse chinois " à revoir " Un journal chinois qui a lancé un appel « va être remanié » -Chirac , le Premier ministre , était là . Chirac , Premier ministre , est là . -Chris dit que la crème glacée réagit quand elle réagit avec la chaleur de la bouche , ce qui signifie que plus vous , plus il devient brillant . Chris explique que , puisque la crème glacée commence à quand elle réagit à la chaleur de la bouche , plus on lèche , plus elle devient brillante . -Citant les " sources exclusives " qu' il n' a pas mentionnées , le canal a indiqué que " les bombes ont visé l' envoi de missiles sol-air destinés au Hezbollah au Liban " , faisant allusion au puissant mouvement chiite libanais qui combat les rebelles aux côtés des forces syriennes . Citant des " sources exclusives " qu' elle n' a pas nommées , la chaîne à capitaux saoudiens a indiqué que " le bombardement a visé une cargaison de missiles sol-air qui était destinée au Hezbollah au Liban " , en référence au puissant mouvement chiite libanais qui combat les rebelles aux côtés des forces syriennes . -Citant les attaques du 11 septembre 2004 , les attaques menées à Madrid en mars 2004 et celles à Londres en juillet 2005 , John Kerry a assuré que les autorités américaines ont depuis lors évité de nombreuses attaques planifiées en raison de l' interception des communications et de la collecte de données . Invoquant les attentats du 11 Septembre 2001 , les attaques de Madrid en mars 2004 et celles de Londres en juillet 2005 , John Kerry a assuré que les autorités américaines avaient depuis déjoué de nombreux projets d' attentats , grâce à l' interception de communications et la collecte d' informations . -Des citoyens partout dans le monde , de même que des hauts fonctionnaires - y compris aux États-Unis - ont estimé que la mise en place d' un système de surveillance sans surveillance était un service public . Les citoyens du monde entier ainsi que les hauts responsables – y compris aux États-Unis – ont jugé que la révélation d' un système inexplicable de surveillance permanente était d' utilité publique . - que la panne a été causée par une attaque de déni de service distribuée ( ) , ce n' est pas vrai . Les allégations selon lesquelles la panne était due à une attaque de déni de service distribué [ DSD ] sont fausses . -Classique , sophistiquée , populaire , en anglais ou en français , les productions de chansons et de danse commencent à prendre fin . Classiques , sophistiquées , populaires , en anglais ou en français , les productions et prennent leur envol . -Chant classique pour les adolescents Du chant classique pour adolescents -Cliquez sur ci-dessus pour voir la cérémonie . Pour suivre la cérémonie , cliquez ci-dessus . -Clive Reynolds , MP , Tony Abbott a des conflits d' intérêt concernant le congé parental Clive Palmer prétend que le député Tony Abbott a un conflit d' intérêt à propos du système de congé parental . -Près de 50,000 maisons dans toute la province sont demeurées sans électricité peu de temps après midi le vendredi , en raison de vents violents touchant de nombreuses régions le long du fleuve Saint-Laurent . Près de 50 000 foyers sont privés d' électricité sur l' ensemble de la province , peu après midi vendredi , en raison des vents violents touchant de nombreuses régions le long du fleuve Saint-Laurent . -La coopération avec les pays partageant des points de vue similaires sera facilitée dans une Europe non fédérale des Nations Unies . La coopération avec les autres pays aux vues similaires serait plus facile dans une Europe des Nations non fédérale . - Ruby était en panne alors qu' elle se trouvait dans un petit accident de route . Ruby , un , s' était échappée après un accident de la route sans gravité - Brothers " to Folk Music L' hommage au folk des frères - Cable Très bientôt pour offrir Interactive TV ? Câble pourrait bientôt offrir la télé interactive - Cable suit donc le chemin de ses concurrents , comme Bell , même si la filiale de n' a pas encore de date précise de lancement de cette nouvelle plate-forme . Câble ainsi le pas à ses concurrents , comme Bell , même si la filiale de n' a pas encore de date précise en ce qui a trait à cette nouvelle plateforme . - Les abonnés du câble pourraient bientôt avoir accès à des applications comme Facebook , Twitter et , enfin , le service de vidéo vidéo à leur demande par la télévision dans un futur pas trop éloigné . Les abonnés de Câble pourraient bien avoir accès à des applications comme Facebook , Twitter et ultimement le service de vidéo sur demande via leur télévision dans un avenir qui ne semble pas si lointain . - les étapes de l' avion , retrouver mon fils et mes parents , mes amis , c' était tout à fait émotionnel . La descente de l' escalier de l' avion , les retrouvailles avec mon fils et mes parents , mes amis , tout cela est extrêmement fort en émotion . - . Sur la piste , des gens qui ne le connaissaient pas pensaient qu' il était si phénoménal qu' ils voulaient se faire photographier avec lui . -Des affrontements ont éclaté à Alexandrie , où la police a utilisé du gaz lacrymogène et 60 manifestants ont été arrêtés . Des affrontements ont éclaté à Alexandrie , la police a fait usage de gaz lacrymogène , 60 manifestants ont été arrêtés . -Armée congolaise les rebelles L' armée congolaise les rebelles du -À l' inverse , ces spéculations le marché des obligations dans la zone euro . Ces même conjectures soutiennent à l' inverse le marché obligataire de la zone euro . -Cooper a demandé à Martin s' il avait sérieusement pensé que Obama n' était pas au courant de la recherche sur le livre Guinness du billet . Cooper a demandé à Martin s' il pensait sérieusement qu' Obama ne savait pas que l' éviction de Biden de la liste présidentielle était examinée . -Les chefs d' entreprise ne sont généralement pas enclins à se vanter des mesures qu' ils ont prises , en l' occurrence un avantage pour un attaquant . Les dirigeants d' entreprise sont généralement peu enclins à révéler les mesures de prévention qu' ils ont mises en place , au cas où cela laisserait l' avantage à un adversaire . -On s' attend à ce que les costumes représentent 1,2 milliard de dollars sur les 6,9 milliards de dollars dépensés , selon la . Les devraient représenter 1,2 Md $ sur les 6,9 milliards dépensés , selon la NRF . -Les couples suivent la scène comme des paires d' amour , très sensuelle , même érotique . Sur scène , passent les couples , en duos amoureux , très , même érotiques . -Le tribunal n' a pas compétence pour statuer sur la politique de La cour bloque une décision relative à la politique de contrôle et de fouille du -Les grues sont arrivées sur le site juste après 10h , et la circulation sur la route principale a été détournée par la suite . Des grues sont arrivées sur place peu après 10 heures , et la circulation sur la nationale a été détournée dans la foulée . -Créé et supervisé par Caroline Bolton , fondateur de l' institution , ce festival fête son dixième anniversaire cette année avec plus de soixante spectacles dans les petits clubs et les grands théâtres . Créé et supervisé par Caroline Hirsch , la fondatrice du club de , le festival célèbre son dixième anniversaire cette année , avec plus de 60 spectacles présentés dans des petits clubs et des grands théâtres . -La création d' une telle réserve permettrait au Royaume-Uni de protéger les , à condition qu' ils aient les moyens de s' assurer qu' ils sont contrôlés - ce qui n' est pas le cas . La création d' une telle réserve marine permettrait au Royaume-Uni de ces km2 , à condition toutefois d' avoir les moyens d' en assurer la surveillance , ce qui n' est pas le cas . -Des critiques comme Claire , , affirment qu' il n' y a pas de motif de sécurité valide pour les interdictions . Des détracteurs tels que la sénatrice Claire , . , soutiennent qu' il n' existe aucune raison de sécurité valable pour ces interdictions . -Actuellement , les passagers doivent éteindre leurs , leurs et d' autres appareils une fois que la porte d' un avion ferme . Actuellement , les passagers doivent éteindre leurs smartphones , tablettes et autres appareils dès que les portes de l' avion sont fermées . -À l' heure actuelle , l' action combine les plaintes de 25 personnes qui prétendent avoir été agressées par les frères de la Sainte Croix . À l' heure actuelle , le recours réunit les plaintes de 25 personnes qui affirment avoir été agressées par les frères Sainte-Croix . - au présent est un réveil brutal , comme d' un rêve . Le retour au présent est un réveil difficile , c' est comme sortir d' un rêve . -Les attaques sont conçues pour inonder un site cible avec du trafic jusqu' à ce que les serveurs soient surchargés et que le site s' effondre . Les attaques DSD visent à inonder un site ciblé jusqu' à ce que les serveurs soient surchargés et que le site s' effondre . -L' ADN , un marqueur unique et , suscite aussi l' espoir et l' inquiétude . unique et , l' ADN suscite également espoirs et inquiétudes . -RDC : Army s s RDC : l' armée attaque le dernier fief des rebelles -RDC : L' armée prépare de nouveaux attentats RDC : l' armée prépare un nouvel assaut -Danielle , qui était dirigée par la et demandant une formation à la protection de l' enfant , s' est dite préoccupée par son comportement . Elle a indiqué que le gestionnaire des services pour les enfants de qui supervisait le Seigneur " très intimidant et très difficile d' apprendre beaucoup de temps " . Danielle , qui était encadrée par Lord et qui avait demandé une formation en matière de protection des enfants car elle s' inquiétait de son comportement , a indiqué qu' elle trouvait la responsable des services à l' enfance du YMCA , , qui Lord « très intimidante et difficile à approcher la plupart du temps » . -David Bowie : Quatre chansons non signées David Bowie : quatre inédits en écoute -De Villiers , il est régulièrement critiqué par des groupes féministes pour le chauvinisme des hommes et par les organisations de défense des droits de l' homme pour racisme - accusations qu' il en deux phrases : Régulièrement pour par des ligues féministes et pour racisme par des organisations des droits de l' Homme , Gérard de Villiers écartait ces accusations en deux phrases . -Décider d' apprendre l' arabe et de faire de l' intérêt pour le Moyen-Orient , au point de prendre une maîtrise dans une université importante , à l' époque , n' étaient pas des décisions . Apprendre l' arabe et m' intéresser au Moyen-Orient , au point d' en faire un master dans une grande école , n' ont pas été , avec le recul , des choix . -Les défenseurs de l' avortement mettent l' accent sur le fait que le contenu de la loi peut conduire à la fermeture immédiate d' un tiers des cliniques dans l' État , ces cliniques n' ayant pas obtenu le privilège d' admission de leurs praticiens . Les défenseurs de l' IVG font valoir que le contenu de la loi risque de se traduire par une fermeture immédiate d' un tiers des cliniques de l' Etat car ces cliniques n' ont pas réussi à obtenir ce " privilège d' admission " pour leurs praticiens . -Certainement pas entre moi et M. Obeid . En tout cas certainement pas entre moi et M. Obeid . -La mine 21 / 2004 Le Delta Centre-Ville maintenant fermé -L' hôtel Delta Centre-Ville à Montréal a fermé ses portes le jeudi après 36 ans d' existence . L' hôtel Delta Centre-Ville a fermé ses portes jeudi à Montréal , après 36 ans d' existence . -Delta et étaient parmi les avions qui ont déjà présenté des plans . Delta et font partie des compagnies qui ont déjà soumis des plans . -Delta s' est engagé à ne pas abandonner ses employés . Delta promet de ne pas laisser tomber ses employés . -Malgré les instructions du ministre , la fermeture devrait néanmoins prendre effet à cette date . Malgré les consignes ministérielles , la fermeture devrait tout de même être effective à cette date . -Malgré la précarité de la situation dans laquelle de nombreuses institutions culturelles sont confrontées , l' avenir semble prometteur , même à long terme . Malgré la précarité qui est le lot de tant d' institutions culturelles , l' avenir se dessine bien , même à long terme . -Malgré l' intervention rapide des services d' urgence , il a succombé à la collision frontale . Malgré l' intervention rapide des secours , il est décédé lors de cette collision frontale . -La destruction de l' équipement signifie que la Syrie ne pourra plus produire de nouvelles armes chimiques . Grâce à la destruction des équipements , la Syrie ne pourra plus produire de nouvelles armes chimiques . -La découverte de cette nouvelle espèce de dauphin à bosse vous surprendra ? La découverte de cette nouvelle espèce de dauphin à bosse vous ? - difficile pour les pharmaciens Une année difficile pour les pharmaciens -La découverte des quartiers , de notre architecture , de notre environnement , sont des points de référence ; La découverte de quartiers , notre architecture , nos lieux sont des repères . -Disney pour lancer New Series sur Tablet Disney mise sur les tablettes pour lancer une série animée - suffisamment d' artistes capables de chanter , d' agir et de danser , comme sur Broadway ? Avons-nous assez d' artistes capables comme à Broadway de chanteur , jouer et danser ? -Les médecins ne pouvaient pas s' assurer qu' il avait un grand clitoris ou un petit pénis et qu' il était convaincu qu' il ne pouvait jamais vivre une " vie satisfaisante " en tant qu' homme . Les médecins ne pouvaient pas vraiment dire s' il avait un large clitoris ou un petit pénis et étaient convaincus qu' il ne pourrait jamais avoir une « vie satisfaisante » en tant qu' homme . -Est-ce qu' il sait parler de piratage par téléphone ? A-t-il connaissance du piratage téléphonique ? - un arbitre du travail d' autrui à partir de son bureau dans la Rue ? Arbitre des oeuvres d' autrui dans son bureau de la rue ? -La vérité vous ? La vérité vous ? -L' expert en comportement des chiens , John McNamara , professeur invité à l' école de médecine de l' Université de Bristol , a déclaré que ce n' était pas la première étude à examiner la question de la gauche ou du droit . L' expert en comportement John Bradshaw , professeur invité à l' École des sciences vétérinaires de l' université de Bristol , a précisé que ce n' était pas la première étude à examiner l' importance de la gauche et de la droite chez l' espèce canine . -Douglas , de l' Association nationale des passagers aériens , a déclaré qu' il est d' avis que l' interférence des appareils est sincère même si le risque est minime . Douglas de la National Association of Airline a déclaré qu' il pensait que les interférences provenant des appareils étaient réelles , même si le risque était minime . -Des dizaines de personnes sur place Des dizaines de personnes sur place -Le Dr Georges et son équipe de bénévoles ont réalisé 700 chirurgies gratuites au cours de l' année écoulée . Le Dr . Georges et son équipe de bénévoles ont pratiqué plus de 700 actes chirurgicaux gratuitement au cours de l' année dernière . -Le Dr Georges apporte des soins de santé gratuits aux villages ruraux du Cameroun . Le Dr Georges dispense des soins de santé gratuits dans les villages du Cameroun -Le Dr Georges est l' un des 10 meilleurs photographes de MTV de 2010 . Le Dr Georges est l' un des 10 Héros CNN de l' année 2013 . -Drew , directeur général de Motorsport , a déclaré : " Le retour des participants au a changé . Drew , le directeur général de Motorsport a déclaré : « Les amateurs de rallyes qui rejoignent ont changé . » - de vitesse à avec une boisson chaude entre les jambes imposée à une amende de £ 1 000 . Un conducteur roulant à 210 km / h avec une boisson chaude entre les jambes s' est vu infliger une amande de 1 000 £ -En raison des salles du 25 Décembre , sa sortie arrivera juste à temps pour les Oscars . En salle le 25 décembre , il pourra participer de justesse à la course aux . -En raison de la protection de l' environnement et des préoccupations de santé publique , les produits d' du colon sont quatre fois moins efficaces que par le passé , mais nous ne pas tuer un enfant pour deux rats . Pour des raisons de protection de l' environnement et de santé publique , les produits de sont quatre fois moins efficaces que par le passé , certes , mais pour deux rats , on ne va pas tuer un . -Lors de la transition d' Obama en 2008 , il a obtenu une note d' approbation de 82 % . Durant la période de transition d' Obama avant sa prise de fonction en 2008 , il avait 82 % d' opinions favorables . -Lors d' une entrevue avec une radio australienne cette semaine , la pop star a dit avoir chanté pour la chanson de la chanson " Old Die " de 2004 . Au cours d' un entretien lors d' une émission de radio australienne cette semaine , la pop star a déclaré qu' elle était sur la chanson de « Old Die Hard » en 2004 . -Au cours de son arrestation , s' est emparé de son portefeuille et a tenté de retirer plusieurs cartes de crédit , mais toutes ont été saisies et un échantillon de cheveux a été à l' intérieur . Au cours de son arrestation , a ramassé son portefeuille et a tenté de retirer plusieurs cartes de crédit , mais elles ont toutes été saisies et on lui a prélevé un échantillon de cheveux . -Au cours de l' opération , le Hamas a lancé un engin explosif ciblant les " forces " ( l' armée israélienne ) et blessé cinq soldats . Pendant l' opération , le Hamas a déclenché un engin explosif visant les forces de ( l' armée israélienne ) et a blessé 5 soldats . -Pendant la période à l' examen , les grands-parents de Mme ont reçu des appels anonymes demandant des informations à son sujet , a déclaré M. . Pendant la période où le détective a enquêté sur elle , les grands-parents de Mlle ont reçu des appels anonymes où on leur demandait des informations sur elle , a ajouté M. . -Au cours du procès , un agent du FBI a déclaré que des informations fournies par le Service de renseignement financier montraient que était au Pakistan pour rencontrer Ben Laden dans le but de rétablir des liens entre des groupes terroristes d' Asie du Sud-Est et Al-Qaida . Pendant le procès , un agent du FBI a déclaré que des rapports des services de renseignement avaient révélé que était au Pakistan pour rencontrer ben Laden dans le but de rétablir des liens entre les groupes terroristes d' Asie du sud-est et Al-Qaïda . -Au cours des années de discussions entamées en 2005 , le gouvernement a insisté pour que les baux soient soumis à des appels d' offres publics . Au cours des années de discussions entamées en 2005 , le gouvernement a fait pression pour que les baux fassent l' objet d' un appel d' offres . -Chaque livre suit la même formule : une grande dose de géopolitique et d' exotisme , plusieurs scènes , une touche de violence et de torture . A chaque livre , la même recette : une grande dose de géopolitique et d' exotisme , quelques scènes de sexe hard , un zeste de violences et de tortures . -Chaque épisode contient deux histoires . Chaque épisode comprend deux histoires de 11 minutes . -Chacune de ces cliniques de fin de semaine offre un large éventail de soins médicaux . Dans cet hôpital itinérant , divers soins de santé sont prodigués . -Hier , une ONG syrienne , l' Observatoire syrien des droits de l' homme , a signalé que plusieurs explosions avaient été lancées le mercredi à l' aube dans une base aérienne de défense au , près de , sur la côte syrienne . Plus tôt , une ONG syrienne , l' Observatoire syrien des droits de l' Homme , avait fait état de plusieurs explosions entendues mercredi à l' aube dans une base de défense aérienne à , près de , sur la côte syrienne . -Auparavant , un porte-parole de NSA avait déclaré à ABC News que le réseau interne et confidentiel de l' organisation était " pas du tout compromis " . Plus tôt , un porte-parole de la NSA a indiqué à ABC News que le réseau interne sensible de l' agence n' était « pas du tout » compromis . -Des recherches antérieures avaient révélé que les chiens heureux leurs queues plus à droite ( du point de vue du chien ) , alors que les chiens nerveux ont un . De précédentes études avaient révélé que les chiens heureux davantage leur queue vers la droite ( du point de vue du chien ) , tandis que les chiens nerveux la plus vers la gauche . -Plus tôt dans le présent communiqué , les serveurs du site ont constaté que le site Web de la NSA avait été fermé pendant au moins six heures , et que le site continuait d' être inaccessible pour certains utilisateurs . Plus tôt ce soir , les de serveurs en ligne ont noté que le site de la NSA avait été en panne pendant au moins 6 heures , et certains utilisateurs ne peuvent toujours pas accéder au site . -Au début de cette semaine , les inspecteurs ont déclaré avoir achevé leur première série de travaux de vérification , en se rendant dans 21 des 23 lieux déclarés par Damas . Plus tôt dans la semaine , les inspecteurs ont déclaré qu' ils avaient achevé la première étape du travail de vérification , en visitant 21 des 23 sites déclarés par Damas . -Plus tôt cette année , Lord a été reconnu coupable d' infractions sexuelles à l' encontre de 12 garçons au cours des deux années qu' il a passées au YMCA . Plus tôt dans l ’ année , Lord a été condamné pour agressions sexuelles sur douze garçons pendant les deux années où il a travaillé au YMCA . - précoce : vieillir plus tôt que d' habitude . précoce : plus tôt -Tôt ce soir , des dizaines de résidents traversaient la frontière , dont certains ont des matelas sur la tête , pour passer la nuit en Ouganda . En début de soirée , des dizaines d' habitants traversaient la frontière , certains avec un matelas sur la tête , pour passer la nuit en Ouganda . -Les économistes sont également préoccupés par les niveaux élevés de la dette des consommateurs en Thaïlande , qui , cette semaine , se sont rapprochés de la récession technique . Les économistes craignent également le niveau élevé d' endettement des consommateurs en Thaïlande , qui est sorti de peu cette semaine d' une récession technique . -Edward , le de l' intelligence américaine , a déclaré qu' il était prêt à se rendre à Berlin pour faire des déclarations au parlement allemand si l' Agence de la sécurité nationale de l' UE et son directeur Keith Alexander ne parviennent pas à répondre à ses activités . Edward , le des services de renseignements américains , a déclaré qu' il était disposé à se rendre à Berlin pour témoigner devant le parlement allemand si la National Security Agency des États-Unis et son directeur Keith Alexander ne fournissaient pas des réponses sur leurs activités . -Égypte le premier président librement élu L' Égypte demande au premier président librement élu de prêter serment -Les islamistes égyptiens se dirigent vers pour faire échec à Morsi Les islamistes égyptiens dans la rue pour dénoncer le procès de Morsi -Quoi qu' il en soit , la direction du théâtre a décidé de mettre en oeuvre le principe de précaution et de mettre en avant la sécurité de ses clients . Toujours est-il que la direction du cinéma a joué le principe de précaution et la sécurité de ses clients à fond . -Quoi qu' il en soit , avec cette voie joyeuse , le chanteur semble être dans son élément quand ses pieds ne sont plus sur le sol . Avec ce titre , le chanteur semble en tout cas dans son élément quand il n' a plus les pieds sur terre . -Les employeurs sont venus rencontrer les employés sur place et ont rencontré les employés individuellement pour évaluer leurs besoins . Des employeurs sont venus rencontrer des employés directement sur place , on a rencontré individuellement les employés pour évaluer leurs besoins . -Fin de la route pour l' Elysée Head Fin de service pour le chef cuisinier de l' Elysée -Les cours de littérature anglaise permettront aux élèves d' étudier au moins une pièce de Shakespeare , un roman du dix-neuvième siècle , une poésie romantique et une littérature anglaise contemporaine à partir de 1914 . Dans le cadre des cours de littérature anglaise , les élèves devront étudier au moins une pièce de Shakespeare , un roman du XIXème siècle , un poème romantique et une œuvre de fiction britannique contemporaine de 1914 à nos jours . -Surtout le débat sur les rançons . Le débat sur les rançons notamment . -Fondée en 1908 comme bastion de l' enseignement laïque , l' Université du Caire devint par la suite un bastion des groupes d' étudiants islamistes dans les années 1970 . Fondée en 1908 et ancien bastion de l' éducation laïque , l' université du Caire est devenue plus tard un fief des groupes d' étudiants islamistes dans les années 1970 . -Marchés européens , à l' exception de Londres , automne à Les Bourses européennes en baisse à la , sauf Londres -Même les alliés les plus proches gardent les choses les uns contre les autres - et travaillent en divers points pour découvrir ce qui reste à faire . Même les alliés proches se cachent des choses – et regardent sous tous les angles pour découvrir ce qui n' a pas été dit . -Même en Allemagne , le parti eurosceptique alternatif pour l' Allemagne - qui a été fondé cette année - n' est arrivé qu' à partir de près de 5 millions de votes lors des élections fédérales du mois de septembre , faisant ainsi de l' opposition aux démocrates libres ( à l' instar de notre propre ) du Bundestag . Même en Allemagne , le parti eurosceptique Alternative pour l' Allemagne , fondé seulement cette année et arrivé de nulle part , a rassemblé presque cinq millions de voix lors des élections fédérales de septembre et a sorti le Parti libéral démocrate ( équivalent de nos ) du Bundestag . -Malgré tout , le a décidé de ne pas recourir à une main-d ' oeuvre indépendante , ce qui pourrait coûter jusqu' à trois fois plus cher que la prise d' un pharmacien à plein temps . Toutefois , le a décidé de ne pas engager de main-d' œuvre indépendante , qui peut coûter jusqu' à trois fois plus cher qu' un pharmacien engagé à temps plein . -Malgré tout , le nombre de clients de Cable a augmenté de pour l' exercice financier de 2013 . Toutefois , pour l' année financière 2013 , le nombre de clients de Câble est en hausse . -Même si Davis n' a jamais existé , les deux réalisateurs , qui sont des amateurs de musique populaire , ont fondé leurs histoires sur les vraies personnes du temps , comme le musicien folk Dave Van . Si Davis n' a pas existé , les deux cinéastes fans de musique folk ont bâti leur histoire sur de vraies personnalités de l' époque comme le musicien folk Dave Von . -Même si elle gagne 3 000 dollars ( 2 400 dollars ) par mois comme administrateur et que son mari travaille , les revenus familiaux mensuels ne sont pas suffisants , dit-elle . Bien qu' elle gagne 3 000 $ de Singapour ( 2 400 $ ) par mois en tant qu' administratrice et que son mari travaille aussi , le revenu familial mensuel ne suffit pas , dit-elle . -Même s' il y avait des signes et que nous avions parlé de la menstruation , elle n' avait pas été préparée émotionnellement . Même s' il y avait des signes et que nous avions parlé des règles , elle n' était pas préparée émotionnellement . -Même si nous sommes des citadins , c' est notre maison spirituelle . Et même si nous sommes des citadins , c' est notre maison spirituelle . -Chaque fabricant le livre d' une manière différente . Chaque constructeur planifie de façon différente . -Chaque matin , je vois ses regards tournés vers lui , je vois les cernes sombres qui lui donnent un coup , les rides autour des lèvres par les cigarettes , les lignes qui marquent le front sur les jours d' inquiétude . Tous les matins , je vois ses yeux inquiets se poser sur moi , je vois les cernes qui alourdissent son regard , les autour des lèvres que creusent les cigarettes , les plis qui marquent le front les jours d' inquiétude . -Tout , mais l' industrie des technologies de l' information . Tout sauf l' industrie des TI . -Tout ne peut pas être réglé en un seul rendez-vous , mais vous savez que vous pouvez compter sur le psychiatre , lui demander si c' est nécessaire , vous n' avez pas été abandonné à vous-même . Ce n' est pas une séance qui fait tout mais on sait qu' on peut compter sur lui , le rappeler si besoin , qu' on n' est pas livrés à nous-mêmes . -Tout est subventionné en Allemagne , du charbon , aux voitures et aux agriculteurs . Tout est subventionné en Allemagne , du charbon aux voitures , en passant par les agriculteurs . -La preuve auprès de la Commission est que le YMCA a été avisé qu' il avait reçu le deuxième chiffre le plus bas de toutes les quatre évaluations possibles dans le cadre d' un audit du Ministère de la culture et des services publics en août de cette année . Il a été démontré devant la Commission que le YMCA avait été informé qu' il avait reçu les secondes plus mauvaises appréciations sur quatre établissements lors d' un audit qualité du Department of Education and Communities réalisé en août de cette année . -Il y a exactement cinq ans , ma grand-mère est morte d' un cancer . Il y a précisément cinq ans , ma grand-mère des suites d' un cancer . -Sauf ma femme . Sauf à ma femme . -Les dirigeants ont également rassuré les analystes et les médias le jeudi , le programme avance en fonction de l' échéancier . Les dirigeants ont également rassurés les analystes et les médias jeudi en leur indiquant que le programme avançait conformément au calendrier . -Les cadres ont indiqué que de nouvelles orientations seraient fournies au quatrième trimestre . Les dirigeants ont annoncé que de nouvelles indications seraient fournies au cours du quatrième trimestre . -Compte tenu d' un déficit budgétaire , le pays demande de l' aide du Fonds monétaire international pour les mois suivants , mais en l' an 2000 le pays a octroyé un crédit de 15 millions de , mais cette fois il n' en a débloqué que 3 , 3 millions . Le pays , en déficit budgétaire , réclame depuis des mois l' aide du Fonds monétaire international , qui lui avait accordé en 2010 un crédit de 15,3 milliards de dollars et n' en a débloqué pour l' instant que 3,4 milliards . -Les experts affirment que les aveux sont toujours systématiquement détenus , en dépit d' une modification de la loi intervenue au début de cette année interdisant aux autorités de contraindre quiconque à s' incriminer lui-même . Les spécialistes disent que les personnes sont systématiquement contraintes à faire leurs aveux , malgré un changement dans la loi qui a été voté plus tôt dans l' année interdisant aux autorités de forcer quiconque à s' incriminer lui-même . -Les experts disent que la violence a laissé 14 adultes et sept enfants morts , ce n' est rien d' autre qu' un hasard hasard , et non un signe de violence croissante en Amérique . Des experts disent que la violence qui a causé la mort de 14 adultes et 7 enfants n' est rien d' autre qu' un malheureux hasard , et non le signe d' une escalade de violence en Amérique . -Les experts pensent que l' épidémie d' obésité pourrait être un déclencheur de la puberté précoce . Les experts pensent que l' épidémie d' obésité pourrait être un déclencheur de la puberté précoce -FAA : Les passagers aériens peuvent désormais utiliser des gadgets sur des avions ( mais ne prennent pas de téléphone cellulaire ) FAA : Les passagers aériens peuvent désormais utiliser des gadgets à bord des avions ( mais pas passer un appel avec leur téléphone portable ) -Les membres du comité consultatif de la FAA ont exprimé des sentiments partagés quant à la question de savoir si l' utilisation des dispositifs présente des risques . Cela a inspiré aux membres du comité consultatif de la FAA des sentiments mitigés quant à la question de savoir si l' utilisation des appareils présente ou non un risque . -Les FAA peuvent lever l' interdiction de certains dispositifs électroniques au moment du décollage et de l' atterrissage . La FAA peut lever l' interdiction sur certains appareils électroniques pendant le décollage et l' atterrissage -Les pages Facebook pour ces présentations sont gérées habilement par des professionnels qui répondent aux questions . Les pages Facebook de ces spectacles sont savamment ­ tenues par des professionnels qui répondent aux questions . -Une crème glacée ? Vous avez envie d' une glace ? -L' innovation de a été étendue au fil des ans , tout récemment par le New York Festival . L' innovation de s ’ est au fil des ans , jusqu' à être reprise plus récemment par le New York Festival . -Craignant que Stockholm les aux États-Unis , il a demandé refuge à l' ambassade de l' Équateur à Londres le 19 juin , demandant au pays d' asile politique de l' Amérique du Sud . Craignant que Stockholm autorise son transfert vers les États-Unis , il a trouvé refuge à l' Ambassade de l' Équateur à Londres le 19 juin , demandant à ce pays d' Amérique du sud l' asile politique . -La ministre de la Santé fédérale , Tanya , a annoncé que le gouvernement du Burkina Faso devrait prendre des " mesures d' urgence " pour contrer une crise causée par le vieillissement de la population de l' île , des maladies chroniques et des contraintes de système . La ministre fédérale de la Santé Tanya a annoncé que le Commonwealth prend actuellement des « mesures urgentes » pour mettre fin à une crise causée par le vieillissement de la population de l' état insulaire , le taux élevé de maladies chroniques et les contraintes du système . -Les frais sont un moyen de permettre aux passagers d' apporter moins à bord . Ces frais sont un moyen de s' assurer que les passagers transportent moins de choses à bord . - est un animal sauvage - il aime les chevaux , les tracteurs et les fermes et il a reçu deux poulets qu' il a . est fou d' animaux – il adore les chevaux , les tracteurs et le monde rural et a deux poulets dont il s' occupe . - a été en tant que troisième anniversaire d' existence en mars dernier et se produit avec l' des Shetland depuis ce temps . a eu en cadeau pour son 3e anniversaire en mars dernier et s' est entraîné depuis avec son Shetland tous les jours . - a été loué par les juges pour ses compétences de manipulation naturelle bien au-delà de ses années . a été salué par les juges pour ses qualités naturelles de dressage peu communes pour un enfant de son âge . -Peu de productions théâtrales sont en tournée et la plupart sont habituellement des " " , destinées aux adultes et non aux adolescents . Peu de pièces de théâtre partent en tournée et c' est souvent du ­ théâtre de boulevard destiné aux adultes et non aux jeunes adultes . - les larmes , elle a dit que aurait dû être condamnée à mort . Au bord des larmes , elle a déclaré que aurait dû être condamné à mort . -Enfin , et ce qui est le plus important , c' est que nos collègues de l' administration des affaires ont trouvé une grande majorité claire voulant que la Grande-Bretagne poursuive un processus de modification des traités et une relation avec l' UE qui repose sur le commerce , et non sur la politique . & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; Enfin , et ce qui est plus parlant encore , notre enquête auprès des chefs d' entreprise a révélé qu' une importante majorité voulait voir la Grande-Bretagne poursuivre les initiatives en faveur d' un changement du traité et du développement d' une relation avec l' UE basée sur le commerce et non la politique . -Enfin , les . Enfin , parlons de . -Enfin , on peut rassurer le public en mettant l' accent sur les mesures mises en place pour protéger les bases de données . Et , enfin , rassurer en mettant en avant les précautions prises pour protéger les bases de données . -Enfin , l' exercice généralisé des révélations menées par les médias dans le monde entier , qui contribue à la mise en place d' un débat sur les pratiques de surveillance par des services de renseignements qui sont jusqu' à présent quasi invisibles , pourrait obliger les acteurs - y compris ceux de l' Amérique - à reconsidérer les pouvoirs qu' ils ont donnés à leurs services de renseignement . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; ] Dernier élément : le vaste mouvement de révélations engagé par des médias du monde entier , qui contribue à enclencher un débat sur les pratiques de surveillance des services de renseignement jusqu' alors quasiment inexistant , pourrait pousser les législateurs , y compris américains , à reconsidérer les pouvoirs qu' ils ont donnés à leurs services de renseignement . - que l' un des nazis les plus brutaux a été enterré dans un cimetière juif est horrible , a-t-il déclaré . Que l' un des nazis les plus brutaux soit enterré dans un cimetière juif , c' est une énormité de mauvais goût . -Les équipes de lutte contre les incendies ont demandé de sauver des survivants après qu' ils ont derrière le sol sur des précaires dans une carrière Les pompiers ont été appelés pour secourir un chiot perdu après être resté coincé à 15 m au-dessus du sol sur une corniche instable dans une carrière -Incendie en 1850 House of Québec Feu dans une maison construite en 1850 à Québec -Les pompiers vers le bas de la falaise pour blesser le chien à partir de sa mort . Les pompiers sont descendus en rappel le long de la falaise pour récupérer la chienne et la sauver d' une mort certaine -Les pompiers convoqués sur les lieux du drame ont confirmé qu' ils avaient répondu à un appel signalant des " blessures multiples " . Des pompiers appelés sur les lieux du drame ont affirmé avoir répondu à un appel faisant état de " multiples blessés " . -Tout d' abord , il y a les " spectacles musicaux " qui évoluent selon la volonté des producteurs ... Il y a d' abord les " spectacles musicaux " qui évoluent au gré des producteurs ... -Il y a cinq ans , mon père est décédé . Il y a cinq ans déjà que mon père s' est effacé du monde des vivants . -Le centre de vol semble la tendance . Flight Centre semble renverser la tendance . -Les parts du centre de vol étaient à $ hier . Les actions de Flight Centre ont augmenté de 3 cents pour atteindre $ hier . -Des fleurs ont été posées sur deux des trois tombes militaires placées dans le cimetière par le maire Jean-Claude et d' autres personnalités . Des ont été déposées dans deux des trois carrés militaires du cimetière par le Jean-Claude et diverses personnalités . -À la suite des excuses de Chen , New AI a fait des excuses à la première page , affirmant qu' il n' avait pas vérifié correctement ses rapports . Suite aux excuses de M. Chen , New Express a publié ses excuses en première page , écrivant qu' il n' avait pas correctement vérifié ses articles . -Suite à la prochaine journée , publiée en janvier , il a produit une de luxe prévue pour le week-end de novembre , contenant plusieurs séries non publiées . À partir de The Next Day , paru en janvier , il a concocté une réédition de luxe prévue pour le 4 novembre , avec plusieurs inédits . -À la suite d' une mission menée dans le pays , le FMI a noté cette semaine que " les besoins importants en matière de financement extérieur " constituaient une " faiblesse " , même s' il y avait des " signes d' amélioration économique " . A l' issue d' une nouvelle mission infructueuse dans le pays , le FMI a constaté cette semaine que " les importants besoins en financement extérieur " constituaient " une vulnérabilité " , tout en relevant " des signes d' amélioration économique " . -Après des examens médicaux à l' hôpital militaire de mercredi après-midi , ils sont désormais réunis avec leur famille . Après des examens médicaux à l' hôpital militaire du mercredi après-midi , ils ont pu , depuis , retrouver leurs familles . -À la suite du départ de toutes les industries ( principaux consommateurs d' eau ) et de la forte diminution de la population , l' utilisation de l' eau dans l' ensemble du district a considérablement diminué . À la suite du départ de toutes les industries ( grandes consommatrices d' eau ) et au grand déclin de la population , l' utilisation de l' eau a considérablement diminué partout sur le territoire . -À la suite de l' indignation des États européens , et même si les révélations sur la presse américaine indiquent que le président américain n' est pas au courant de ces activités d' espionnage , Barack Obama a refusé de faire des commentaires sur la question en invoquant la sécurité nationale . & # 160 ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Face à la colère d' Etats européens et alors que des fuites dans la presse américaine affirmaient que le président américain n' était pas au courant de telles écoutes , Barack Obama a refusé de s' exprimer à ce sujet , invoquant la sécurité nationale . -Pendant 21 ans , Georges a vu son père malade descendre et sortir de la conscience , voyageant dans des hôpitaux qui n' étaient pas équipés pour l' aider . Pendant 21 ans , Georges a regardé son père malade alterner les périodes de conscience et d' inconscience et se rendre dans des hôpitaux qui n' étaient pas équipés pour lui venir en aide . -Pour , la charge de travail n' est pas une difficulté . Pour Dr , la charge de travail relativement constante n' est pas une difficulté . -Pour François , directeur général des Services de la Ville de Paris , pour faire face à l' incinération , il faut donner à la défunte une cérémonie aussi belle qu' elle aurait pu l' être avec une . la crémation , c' est pour François , directeur général des Services funéraires de la Ville de Paris , prévoir pour le défunt une cérémonie aussi digne que celle à laquelle il aurait droit avec une inhumation . -Pour les transporteurs américains , les répercussions de la décision varieront d' une compagnie aérienne à l' autre et dépendront de l' âge de leur flotte . Pour les transporteurs américains , l' impact de la décision variera d' une compagnie aérienne à l' autre et dépendra de l' âge de leur flotte . -Pendant longtemps , il est resté chez moi - quand j' ai arrêté de fumer , quand j' ai eu peur , quand j' étais malade ... Pendant longtemps , il m' a accompagné : quand j' ai arrêté de fumer , quand j' ai eu peur , quand j' ai eu mal ... -Pendant longtemps , la plupart des tentatives , de " Notre-Dame de Paris " à " Mozart , l' Opéra Rock " , ont été . Longtemps , de Notre-Dame de Paris à Mozart l' opéra rock , la plupart des tentatives s' dans le ridicule . -Par souci de prudence , le rapport aux États-Unis est un médecin pour 413 personnes . À titre de comparaison , le ratio aux États-Unis est d ' 1 médecin pour 413 habitants . -Par exemple , la France a été informée de la présence de Mohammed dans les zones tribales de la par l' intermédiaire des ressources de la NSA . A titre d' exemple , la présence de Mohammed dans les zones tribales à a été signalée aux Français grâce aux moyens de la NSA . -Je pense par exemple à la définition de la résilience . Je m' interroge par exemple sur la définition de la résilience . -Par exemple , il y a parfois neuf heures d' attente à l' unité d' urgence à . Il y a par exemple parfois neuf heures d' attente aux urgences de . -Pour les , il s' agit d' une plus grande marge de , mais pour les fournisseurs , c' est de plus en plus une question qui pourrait affecter les gains . Pour les passagers , il s' agit davantage d' avoir une certaine liberté de mouvement , mais pour les fournisseurs , c' est de plus en plus une question qui pourrait affecter leurs résultats . -Pour lui , " si les autres services d' urgence de Paris pouvaient absorber l' excédent , il n' y aurait pas de problème " . Pour lui , " si les autres services d' urgences parisiens pouvaient absorber le surplus , il n' y aurait pas de problème " . -Pour les grandes entreprises , le message a été depuis longtemps le maison qui tient la possession d' un USB à un salon commercial , ou laisser un ordinateur non portatif dans une chambre d' hôtel est déraisonnable , pour le moins . Les grandes sociétés depuis longtemps le message selon lequel il est pour le moins imprudent de récupérer une clé USB gratuite lors d' un salon professionnel ou de laisser un ordinateur portable sans surveillance dans une chambre d' hôtel . -Pour les étudiants en médecine et en soins infirmiers , comme O' Malley , qui viennent des États-Unis et de l' Europe pour rejoindre sur leurs missions , il s' agit d' une occasion pratique qu' ils ne jamais chez eux . Pour les étudiants en médecine et les étudiants infirmiers comme Katie O' Malley , qui viennent des États-Unis et d' Europe pour rejoindre M. dans ses missions , c' est une occasion d' apprentissage sur le terrain qu' ils n' ont pas chez eux . -Depuis plus d' un an , j' ai remarqué le mécontentement profond de personnes que je rencontre partout : la réévaluation vertigineuse de leurs biens , un déluge d' impôts , le coût de toutes sortes de permis , des frais administratifs illimités , etc. Depuis plus d' un an , partout , je suis à même de constater un fort mécontentement de la part des gens que je rencontre : réévaluation monstrueuse de leurs propriétés , déluge de taxes , coût des permis de toutes sortes , frais administratifs à n' en plus finir . -Depuis plus de 40 ans , nous sommes essentiellement occupés par des costumes de grande qualité , et je ne crois pas qu' un éleveur puisse faire pire . Nous sommes gérés depuis plus de 40 ans par des avec de beaux diplômes pour la plupart et je ne crois pas qu' un éleveur de moutons fasse pire . -En l' espace de dix jours , un grand nombre de journaux importants en France , en Allemagne , en Espagne et en Italie ont fait savoir que la NSA avait intercepté de nombreuses quantités de données et de communications émanant d' alliés des États-Unis et de leurs dirigeants , en particulier la chancelière allemande Angela Merkel . Depuis dix jours , plusieurs grands journaux en France , en Allemagne , en Espagne ou en Italie , ont révélé que la NSA aurait intercepté massivement des données et communications émanant d' alliés des Etats-Unis et de leurs dirigeants , notamment la chancelière allemande Angela Merkel . -Pour le gouvernement écossais , Keith Brown a demandé à M. de " s' opposer " à l' inclusion de l' Écosse dans le réseau . Pour le gouvernement écossais , Keith Brown a appelé M. à soutenir « sans équivoque » l' inclusion de l' Écosse dans le réseau . -Pour la deuxième année consécutive , l' une de ces cérémonies a été retransmise en direct , pour laquelle on n' a pas pu y assister . Pour la deuxième année consécutive , l' une de ces cérémonies est retransmise , à l' attention de ceux qui ne pourraient pas faire le déplacement . -Pour le troisième trimestre terminé le 30 septembre , le bénéfice net de Bombardier est passé à 147 millions de dollars , soit 8 cents par action , soit 172 millions de dollars , soit 9 cents par action un an plus tôt . Au troisième trimestre clos le 30 septembre , le bénéfice net de Bombardier a chuté à 147 M $ , ou 8 cents par action , par rapport à 172 M $ , ou 9 cents par action , un an plus tôt . -Depuis deux semaines , il y a presque quotidiennement des rapports d' affrontements entre les forces gouvernementales et les , certaines des pires escarmouches depuis un accord de paix il y a plus de 20 ans . Pendant deux semaines , il y a eu pratiquement tous les jours des rapports faisant état de conflits entre les forces gouvernementales et la , les pires accrochages depuis l' accord de paix qui a été signé il y a plus de 20 ans . -Ford , qui tire encore profit du succès de ses camionnettes , entre autres , a vu les ventes augmenter de 14 % par rapport à l' année dernière pour atteindre véhicules , contre une attente de . Ford , qui continue de bénéficier entre autres du succès de ses pick-up , a vu ses ventes progresser de 14 % sur un an à 191 985 véhicules , alors que le consensus attendait 194 301 . -Les travailleurs étrangers sur 457 visas pouvaient faire l' objet d' une épreuve d' authenticité . Les travailleurs étrangers en possession d' un visa 457 pourraient avoir à passer un test « d' authenticité » -Oubliez le sens et votre ticket . Oubliez d' être raisonnable et sortez vos billets . -L' ancien reporter de CNN Pat est journaliste spécialisé dans les questions de santé et de famille . L' ancien correspondant de CNN Pat est un journaliste spécialisé dans les questions familiales et de santé chez l' enfant . -Ancien chef de la tribu dans le cimetière juif L' de la enterré dans un cimetière juif -L' ancien au Liban dit que " Getting is difficile to " Un ancien otage au Liban : " Le retour est assez complexe à gérer " -L' ancien ministre de l' Intérieur de la Nouvelle-Galles du Sud , Joe , sera interrogé par le responsable de la lutte contre la corruption de l' État . L' ancien ministre travailliste de Nouvelle-Galles du Sud Joe fera l' objet d' une enquête du régulateur anti-corruption de l' État . -L' ancien ministre de la ville de , Joe , a refusé de modifier le contrat de pêche politique à la demande de son conseiller tactique Eddie , qui avait des intérêts cachés dans trois propriétés sur des terres contrôlées par le gouvernement . L' ancien ministre de Nouvelle-Galles du Sud Joe a nié avoir modifié la politique de baux commerciaux à la demande de son mentor politique Eddie Obeid , qui avait des intérêts cachés dans trois biens situés dans la zone contrôlée par le gouvernement . -L' ancien rédacteur en chef du journal international Andy aurait utilisé " le téléphone , la surveillance et l' affrontement " pour tenter de confirmer un faux argument sur une affaire impliquant le secrétaire de Charles Clarke . L' ancien rédacteur en chef de News of the World Andy aurait utilisé « le piratage téléphonique , la surveillance et la confrontation » pour tenter de confirmer un tuyau bidon sur une liaison impliquant le ministre de l' Intérieur alors en exercice , Charles Clarke . -Heureusement , un petit nombre d' entre eux l' ont fait . Heureusement , quelques uns y ont trouvé leur compte . -Quarante pour cent des catholiques romains sont en Amérique du Sud , mais ils n' ont qu' un nombre infime de cardinaux . 40 % des Catholiques romains vivent en Amérique du Sud , mais un petit nombre de cardinaux les représentent . -Quatre ( d' Al-Qaïda au Maghreb islamique ) , Daniel , Thierry , Pierre Legrand et Marc ont été libérés mardi après avoir passé plus de 1000 jours . Détenus plus de 1 000 jours , les quatre otages d' ( Al-Qaida au Maghreb islamique ) Daniel , Thierry , Pierre Legrand et Marc ont été libérés mardi . -Quatre soldats du Hamas ont été tués et cinq soldats israéliens ont été blessés la nuit du 31 octobre , lors d' un violent affrontement sur la frontière entre Gaza et Israël , l' incident le plus grave sur le territoire palestinien pendant un an . Quatre combattants du Hamas ont été tués et cinq soldats israéliens blessés jeudi 31 octobre 2013 au soir lors d' un violent accrochage à la frontière entre Gaza et Israël , l' incident le plus sérieux dans le territoire palestinien depuis un an . -Quatre d' entre eux ont déjà été publiés sur l' Internet . Quatre ont déjà émergé sur la Toile . -La France , pour sa part , doit déjà procéder à la surveillance aérienne très coûteuse et maritime de 5 millions de kilomètres carrés de la ZEE de la Polynésie française , adjacente à Pitcairn . Quant à la France , elle doit déjà mener la très coûteuse surveillance aérienne et maritime des cinq millions de km2 de la ZEE de Polynésie française , voisine de Pitcairn . -La France , porte-parole du service de protection contre les incendies du Québec , a déclaré que l' intervention rapide des pompiers a permis d' éviter que le feu se déplace vers les bâtiments adjacents , un défi majeur dans cette partie de la ville où les bâtiments sont très proches . France , porte-parole du Service de protection contre les incendies de Québec , a précisé que l' intervention rapide des pompiers a permis d' éviter la propagation des flammes aux bâtiments adjacents , un défi de taille pour ce secteur de la ville où les immeubles sont construits de manière rapprochée . - et , et , catholiques et protestants : les forces de division dans ce pays sont très anciennes . Francs et , et , catholiques et protestants : les forces de division , ici , sont très anciennes . -François Hollande , qui a réintroduit le fromage , est " un homme qui aime manger " et " il n' est pas très " . Quant à François Hollande , qui a rétabli le fromage , " c' est quelqu' un qui aime manger " et " il n' y a pas grand-chose qu' il n' aime pas " . - La police française arrêtera les supporters d' venus autrement qu' avec les cars du -De 2005 à 2007 , il a été l' hôte d' un spectacle sur la télévision terrestre intitulé " with David Steinberg " . Entre 2005 et 2007 , il a animé une émission sur TV Land qui s' appelait « Sit Down with David Steinberg » . -À partir de 2015 , les élèves devront prendre un individuel en langue , avec de fortes incitations à choisir la littérature anglaise comme une qualification distincte . À partir de 2015 , les élèves devront passer un de langue indépendant , et seront fortement incités à choisir une qualification séparée en littérature anglaise . -De à Charlotte par Nicole Kidman , Lars von Trier a tendance à pousser ses actrices jusqu' à la limite , toujours dans le but d' atteindre le meilleur . De à Charlotte , en passant par Nicole Kidman , Lars von Trier a pour habitude de pousser ses actrices dans leurs derniers , toujours pour le meilleur . -De , bastion politique et dernier bastion de la rébellion à tomber le mercredi , un journaliste de l' AFP pourrait entendre parler de l' explosion des armes lourdes . De , fief politique et dernière place forte de la rébellion tombée mercredi , une journaliste de l' AFP pouvait entendre des d' armes lourdes . -À partir du 4 novembre , les unités de lutte contre les incendies , qui constituent le quart des 40 000 personnes envoyées à l' hôpital d' urgence de l' , ont toutes reçu l' instruction de prendre en charge chaque année une trentaine de cas graves à des unités d' urgence d' autres hôpitaux à Paris . A compter du 4 novembre , les pompiers – qui assuraient environ un quart des 40 000 passages annuels aux urgences de l' – auront tous reçu pour consigne d' emmener la trentaine de cas graves par jour vers les urgences d' autres hôpitaux parisiens . -Du point de vue de la conservation , il est essentiel de connaître les différences génétiques pour qu' une espèce et , par conséquent , sa variabilité génétique puissent être protégées . D' un point de vue conservation , il est essentiel de connaître ces différences génétiques , afin de protéger l' espèce et donc sa variabilité génétique . -Le fait de forcer le paiement des frais de port de cabine s' il n' y a pas d' achat direct de la compagnie est le plus récent effort pour orienter les clients vers leur propre site Web . La décision de Frontier de faire payer des frais de bagages à main lorsque les passagers n' achètent pas directement leur billet auprès de la compagnie aérienne constitue son dernier effort pour inciter les clients à aller sur son site . -Les nouveaux frais de cabine ne commenceront pas avant l' été , bien qu' une date n' ait pas été fixée . Les frais de bagages à main de Frontier ne seront pas appliqués avant l' été , bien qu' aucune date n' ait été fixée . -Frontier Airlines a l' intention de facturer jusqu' à 100 $ pour le transport des bagages de cabine à bord de leur vol. Frontier Airlines envisage de faire payer jusqu' à 100 $ aux passagers qui transportent des bagages à main sur ses vols . -Frontier Airlines a l' intention de commencer à facturer jusqu' à 100 $ pour un sac à main et 2 $ pour le café ou le soda , mais son avis de mercredi laisse croire que les passagers pourront conserver l' ensemble . Frontier Airlines envisage de commencer à faire payer jusqu' à 100 $ pour un bagage à main et 2 $ pour un café ou un soda , bien que dans son annonce mercredi , elle indiquait que les passagers pourraient emporter la non ouverte avec eux à la descente de l' avion . - Airlines pour les bagages à main Frontier Airlines envisage de faire payer les bagages à main -Frontier a une base fidèle de clients dans sa ville natale de Denver , mais son activité diminue et perd de l' argent . Frontier a une clientèle fidèle dans sa ville d' origine , Denver , mais son activité recule et elle perd de l' argent . -Cependant , la frontière est la plus avancée dans ce domaine . Toutefois , Frontier est allée plus loin dans ce domaine . -Frontier suit également les frais de 2 $ pour le café , le thé , le soda ou le jus . Frontier fait également comme Spirit et applique des frais de 2 $ pour un café , un thé , un soda ou un jus de fruit . -La frontière dit qu' elle paiera 25 $ si la taxe est payée à l' avance , 100 $ si les voyageurs attendent de payer jusqu' à ce qu' ils soient à la porte . Frontier a déclaré qu' elle ferait payer 25 $ si les frais étaient payés à l' avance et 100 $ si les voyageurs ne payaient qu' une fois à la porte d' embarquement . -Selon la frontière , les passagers qui consomment de la soude ou du jus peuvent garder tout le temps nécessaire , et ils peuvent emporter gratuitement des recharges de café . Frontier a indiqué que les passagers qui prennent un soda ou un jus de fruit peuvent emporter la non ouverte avec eux , et ceux qui prennent du café peuvent être gratuitement . -La direction de la pétition , Kate O' Malley , a dit que les 100 $ de frais nécessaires pour faire payer les passagers à l' avance . La porte-parole de Frontier , Kate O' Malley a indiqué que les 100 $ étaient prévus pour inciter les voyageurs à payer les frais à l' avance . -D' autres tests sont nécessaires avant le début des essais humains , ce qui prendra plusieurs années , mais Michael Crystal Ph . , du Cornell Medical College à New York , a dit que les premiers signes étaient bons . D' autres tests sont nécessaires avant de commencer les essais sur l' être humain , mais le Professeur Ronald Crystal du Cornell Medical College de New York a déclaré que les premiers signes étaient encourageants . -En outre , Israël a attaqué une base militaire du nord-ouest de la Syrie , ciblant un envoi de missiles destinés au mouvement libanais chiite Hezbollah , le satellite , communiqué le jeudi 31 octobre . Par ailleurs , Israël a frappé une base militaire aérienne dans le nord-ouest de la Syrie , visant une cargaison de missiles destinée au mouvement chiite libanais Hezbollah , a rapporté jeudi 31 octobre la chaîne satellitaire . -En outre , John Kerry a déclaré dans son allocution aux Européens : " Je vous assure qu' aucun innocent n' a été induit en erreur au cours de ce processus " . Et , a affirmé John Kerry à l' adresse des Européens , " je vous assure que dans ce processus des personnes innocentes n' ont pas été " . -En outre , à cet égard , la France et les États-Unis ont mis en place des procédures , parfois décrites comme essentielles , pour la coopération et l' échange d' informations d' une manière presque quotidienne . D' ailleurs , dans ce domaine , la France et les Etats-Unis notamment ont mis en place des procédures de coopération et d' échanges d' informations quasi quotidiens et qui sont décrits de part et d' autre comme essentiels . -De plus , la détection et la vérification d' un réseau de veines ou de formes nécessitent un geste volontaire et spécifique de la part des particuliers . Et le recueil ainsi que la vérification du réseau veineux et du contour de la main supposent un geste volontaire et précis des personnes . -À Wall Street , on prévoit que le cours d' ouverture des actions américaines augmentera , après deux sessions consécutives , marquées par une tendance à la baisse . Les futures de Wall Street laissent présager une ouverture des actions américaines en hausse , après deux séances consécutives de repli . -Le lancement du groupe GM a débuté en juin et représente le plus important lancement de véhicules pour le N ° 1 américain depuis sa restructuration en 2009 . Le déploiement des de GM a commencé en juin et représente le lancement de véhicules le plus important pour le premier constructeur automobile américain depuis sa faillite et sa restructuration en 2009 . -GM rappelle qu' un certain nombre de camionnettes neuves ont été mises à la disposition des États-Unis en vue de réduire les émissions de gaz à effet de serre . GM rappelle certains de ses nouveaux aux États-Unis pour réparer le dossier des sièges -Les parts de GM ont augmenté de 1,4 pour cent à $ à la Bourse de New York le vendredi après-midi . Le cours des actions de GM a grimpé de 1,4 % pour atteindre $ à la bourse de New York vendredi après-midi . -GM a informé les propriétaires de camions de la défectuosité du premier semestre d' octobre . GM a informé les propriétaires de du défaut au cours de la première quinzaine d' octobre . -Gazprom de Gazprom dit que le pipeline en Bulgarie commence une nouvelle ère de gaz Alexei Miller de Gazprom déclare que le gazoduc de Bulgarie inaugure une nouvelle ère du gaz -Gazprom a déclaré que la construction devrait commencer dans d' autres pays en aval d' ici la fin de l' année . Gazprom a indiqué que la construction devrait démarrer dans d' autres pays situés en aval d' ici la fin de l' année . -L' égalité des sexes n' est toujours pas bien comprise , mais la plupart des experts des États-Unis pensent que lorsque le sexe n' est pas reconnu , il est préférable d' utiliser les meilleures informations disponibles pour qu' elles parviennent à attendre et à surveiller le développement psychologique et physique de l' enfant avant d' entreprendre une opération de contraception , si c' est le cas . L' identification sexuelle n' est toujours pas bien comprise , mais la plupart des spécialistes aux États-Unis disent que lorsque le sexe ne peut pas être déterminé , il vaut mieux utiliser les meilleures informations disponibles pour l' attribuer , puis attendre et suivre le développement psychologique et physique de l' enfant avant d' envisager un acte chirurgical , le cas échéant . -Les garçons non ont maintenant un camp spécial . Les jeunes garçons de genre ont désormais des camps qui leur sont spécialement destinés . -General Motors Co . General Motors Co rappelle près de 19 000 de ses tout nouveaux Chevrolet et GMC Sierra 2014 pour régler un problème avec les dossiers manuellement , selon une déclaration vendredi de l' organisme américain chargé de la réglementation de la sécurité des véhicules . -George : L' éclatement de l' Europe donne le choix à l' écossais George : L' éclatement de l' Europe donne le choix aux Écossais -Gérard , né le 8 décembre 1950 à Paris , a publié , le 4 décembre 1970 , des livres sterling , et il ne savait pas combien de livres il avait vendus depuis la publication " SAS à Istanbul " , la première de la série , près d' un demi-siècle en 1965 . Avec quatre SAS publiés par an , Gérard de Villiers , né le 8 décembre 1929 à Paris , assurait ignorer le nombre exact de livres vendus depuis 1965 et la publication de " SAS à Istanbul " , le premier de la série , il y a près d' un demi siècle . -Gérard de Villiers , " SAS " Série Auteur , Dies . Gérard de Villiers , l' auteur de " SAS " , est mort . -Gérard de Villiers , un phénomène de la littérature française , qui est décédé le jeudi à 83 heures , vient de publier son 200e livre " " , La du Kremlin . de l' édition française , Gérard de Villiers , décédé jeudi à 83 ans , venait tout juste de publier son 200e SAS , " La du Kremlin " . -Gérard de Villiers a dit lui-même : " Comme tous les héros , est sans âge . Gérard de Villiers l' avait dit : " , comme tous les héros , n' a pas d' âge " . -German Journalists to Google and Yahoo Les journalistes allemands invités à fuir Google et Yahoo -Le député allemand jeudi a rencontré M. en Russie , où il a obtenu l' asile , pour discuter de son président en Allemagne . Le député allemand a rencontré jeudi M. en Russie , où il bénéficie du droit d' asile , pour discuter de son témoignage en Allemagne . -Les entreprises allemandes estiment que les Etats-Unis représentent presque aussi un risque aussi élevé que la Chine en ce qui concerne l' espionnage industriel et le vol des données , selon une enquête publiée en juillet par , la consultation . Les sociétés allemandes pensent que les États-Unis représentent un risque aussi important que la Chine lorsqu' il s' agit d' espionnage industriel et de vol de données , selon une enquête publiée en juillet par le cabinet de conseil . -Le pilote allemand , Sebastian Vettel , Formula 1 Champion du Monde , a reçu un nouveau casque spécial à l' aide d' un américain de 13 jours dans le cadre de la séance de qualifications pour le Grand Prix d' Abu Dhabi le vendredi . L' Allemand Sebastian Vettel , quadruple champion du monde de Formule 1 , a pour les essais libres du Grand Prix d' Abou Dhabi vendredi un casque spécial dont le dessin a été réalisé par un fan mexicain de 21 ans . -La loi allemande a pour but de l' attribuer à la naissance . La loi allemande concerne l' attribution du sexe à la naissance . -Les médias allemands disent que le espère identifier la présence d' antennes d' écoute et l' action déclenche un échange entre les États-Unis et le ministère allemand des Affaires étrangères à Berlin . Les médias allemands expliquent que le espérait identifier la présence d' antennes d' écoute et que l' opération a suscité un échange entre les États-Unis et le ministère des Affaires étrangères allemand à Berlin . -Les passeports allemands auront une troisième désignation autre que M ou F - X , pour , selon le ministère de l' Intérieur . Les passeports allemands comporteront une troisième désignation à côté de M ou F – X pour , selon le ministère de l' Intérieur . -Le de l' Allemagne conseille aux dirigeants d' envisager d' utiliser des téléphones mobiles prépayés en cas de voyages à l' étranger en raison du risque que des téléphones intelligents soient compromis . & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; Le d' Allemagne conseille à ses dirigeants d' envisager d' utiliser de simples téléphones portables prépayés lors de leurs voyages à l' étranger en raison du risque d' atteinte à l' intégrité des smartphones . -L' Allemagne et quelques économies satellites pourraient conserver l' euro , mais la France et le Sud de l' Europe leur propre monnaie . L' Allemagne et quelques-unes de ses économies satellites pourraient conserver l' euro , mais la France et l' Europe du sud retrouveront leur propre devise . -L' Allemagne est devenue la première nation européenne à reconnaître un troisième sexe pour les enfants nés avec des organes génitaux sensibles . L' Allemagne est devenue la première nation européenne à reconnaître un troisième sexe pour les nourrissons nés avec des organes sexuels ambigus . -Les filles qui ont tôt fait d' être doivent s' assurer que , même lorsqu' elles se présentent à l' avance , le processus est une partie normale de la vie . Les filles dont la puberté est précoce doivent être sur le fait que , même lorsque ça arrive plus tôt que prévu , le processus fait partie de la vie . -À l' échelle mondiale , les formes de puberté précoce semblent influencées par tous les aspects , de la situation économique au climat des gènes . À l' échelle mondiale , les mécanismes de la puberté précoce semblent être influencés par tout , des conditions économiques au climat , en passant par les gènes . -Allez voter , c' est la meilleure façon de s' exprimer et de dire qu' il suffit de le faire . Allez voter , c' est la meilleure façon de vous exprimer et de dire que c' est assez . -Google , Samsung , a intenté des poursuites contre les brevets de Nortel Google , Samsung et font l' objet de poursuites judiciaires concernant les brevets Nortel -Boutique de verre de Namur ( en anglais ) Lancement d' une boutique d' accessoires pour Google Glass -Google a augmenté sa soumission à plusieurs reprises , ce qui lui a permis de réaliser jusqu' à 4,4 milliards de dollars . Google a augmenté son offre à plusieurs reprises , pour finalement proposer 4,4 Md $ . -Google est accusé d' avoir enfreint sept brevets . Google est accusé d' avoir violé sept brevets . -Peu à peu , à cause de petites et légères apparences ou au cours de la journée et de la nuit , elle est revenue de nouveau dans ma vie , après avoir repris sa place dans le paysage de ma pensée qui le toujours en deuil . Peu à peu , par petites apparitions diurnes ou nocturnes , un peu timides , un peu effacées , elle est vite revenue dans mon esprit , évoluant au fur et à mesure qu' elle reprenait sa place dans le paysage de ma pensée qui se croyait en deuil . -Grand opposition à la mort de au Care Committee Beaucoup d' opposition à l' aide médicale à mourir au Congrès des soins palliatifs -Grèce : Two Dead in Shooting Games Project Grèce : deux morts dans une fusillade près d' un local du parti -Le travail de groupe , les petits en déplacement , en groupe , en , rendent la représentation moins . Travail de groupe , petits solos de transitions , ensembles qui se déplacent , choeurs , l' écriture est moins binaire . -L' adaptation de William du roman de Denis présente un design et un temps de production exceptionnels , mais il est aussi plus lourd qu' il ne devrait l' être . L' adaptation par Guillaume du roman de Denis se targue d' une direction artistique remarquable et d ’ une minutieuse reconstitution d' époque , mais elle est également plus laborieuse qu' elle ne devrait l' être . -La HSBC , Citigroup , et le FMI ont également lancé des internes ou reçu des demandes d' information de la part des organismes de réglementation , qui connaissent bien la situation . HSBC , Citigroup , et Crédit Suisse ont également lancé des enquêtes internes ou reçu des demandes d' informations des régulateurs , ont indiqué des sources proches du dossier . -Halloween 2013 : Par le nombre Halloween 2013 : En chiffres -L' Halloween est un festival célébré le jour avant la Toussaint , principalement dans les pays anglophones . Halloween est une fête païenne célébrée à la veille de la Toussaint , essentiellement dans les pays anglo-saxons . -Le Hamas garantit que Gaza sera l' enfer pour l' occupant " , a déclaré le porte-parole . " Le Hamas assure que Gaza sera un enfer pour l' occupant " , a menacé le porte-parole . -Le Hamas a défendu son utilisation des tunnels dans la lutte contre Israël , en affirmant qu' il visait à capturer des soldats israéliens afin qu' ils puissent être échangés contre des prisonniers palestiniens . Le Hamas a revendiqué l' usage de tunnels pour lutter contre Israël , précisant que le but était d' enlever des soldats israéliens pour les échanger contre des prisonniers palestiniens . -Le Hamas a loué les " héros qui sont morts pour défendre le territoire contre une incursion à Khan par l' ennemi " . Le Hamas félicite les héros d' qui sont tombés en défendant le territoire lors d' une incursion de l' occupant sioniste à Khan . -Hans-Peter Friedrich , ministre allemand de l' Intérieur , a déclaré à Time Online : " Si M. est disposé à parler aux autorités allemandes , nous trouverons des moyens pour y parvenir " . Hans-Peter Friedrich , ministre de l' Intérieur allemand , a déclaré au Zeit Online : « Si M. est prêt à parler aux responsables allemands , nous ferons en sorte de rendre cela possible . » -La France a-t-elle bénéficié des informations fournies par la NSA concernant les opérations terroristes à l' encontre de nos intérêts ? La France a-t-elle bénéficié d' informations fournies par la NSA concernant des opérations terroristes visant nos intérêts ? -Ayant été expulsé du Parti communiste en juillet 2012 , Tian a été accusé d' avoir reçu 19 millions de yuan en pots-de-vin , selon des représentants des médias en Chine . du Parti communiste en juillet 2012 , Tian était accusé d' avoir touché plus de 19 millions de yuans de pots-de-vin , rapportent les médias officiels chinois . -Après avoir entré la des cuisines au Palais de l' Elysée comme assistant après avoir travaillé dans les ambassades , le cuisinier a grimpé à l' échelle pour devenir chef cuisinier il y a neuf ans . Entré dans les 500 m2 des cuisines comme commis après avoir travaillé dans des ambassades , le cuisinier a gravi tous les échelons jusqu' à devenir chef , il y a neuf ans . -Il a ajouté : " Nous n' avons pas pu repartir à zéro ; nous avons donc été contactés pour obtenir de l' aide . Il a ajouté : « Il était incapable de s' envoler à nouveau , nous avons donc été contactés pour apporter notre aide . » -Il a ajouté : " Nous sommes dans un nouveau monde où les gens sont prêts à s' enfuir " . Il a ajouté : « Nous vivons dans un monde nouveau où les gens sont prêts à se faire sauter . » -Il a ajouté que les auteurs et les enseignants étaient " " , c' est-à-dire qu' il y avait " abandon complet " pour les enfants , et que les messages étaient " " dans des histoires agréables . Il a ajouté que le fait que les auteurs et les enseignants soient « ouvertement didactiques » fait que les enfants « complètement » , les messages étant « » dans des histoires agréables . -Il a ajouté que , tout comme les humains , pour les chiens , le côté du cerveau était responsable du mouvement et inversement , et les deux hémisphères ont joué des rôles différents dans les émotions . Il a ajouté que de même que chez l' homme , chez les chiens le côté droit du cerveau était responsable du mouvement vers la gauche et vice versa , et que les deux hémisphères jouaient un rôle différent dans les émotions . -Il a reconnu la possession de cocaïne à une audience antérieure . Il avait admis la possession de cocaïne au cours d' une audience précédente . -Il a informé les parents d' un garçon dont le pénis avait été dans une excision pour que l' enfant ait complètement , lui aussi ses testicules , et pour qu' il l' élève comme fille . Il avait conseillé aux parents d' un petit garçon dont le pénis avait été coupé lors d' une circoncision de le faire entièrement , en lui enlevant également les testicules , et de l' élever comme une fille . -Il m' a donné ma vie privée . Il me laissait mon intimité . -Il a également déclaré qu' une femme et un enfant avaient été tués à la suite de la soirée précédente et avaient été enterrés . Selon lui , une femme et son enfant ont été tués par les combats de la veille et ont été enterrées . -Il apparaît dans l' une de leurs publicités , diffusée à partir du 10 novembre . Il apparaîtra dans un de leurs spots publicitaires , diffusé à partir du 10 novembre . -Il est devenu médecin , travaillant comme chirurgien vasculaire à l' hôpital central de Yaoundé . Il est devenu médecin lui-même , travaillant comme chirurgien vasculaire à l' hôpital central de Yaoundé . -Il a rédigé l' idée après avoir lu un document de recherche sur les méduses et a convaincu les scientifiques chinois de recréer chimiquement la protéine . Il en a eu l' idée après avoir lu une étude sur les méduses et a convaincu des scientifiques chinois de recréer chimiquement la protéine brillante . -Il a inventé l' expression " identité sexuelle " et a affirmé que les indices sociaux et environnementaux - comment les parents d' un enfant - ont tissé des liens avec les gènes et les hormones d' un enfant afin de déterminer si la personne était considérée comme mâle ou femelle . Il a consacré le terme « identité sexuelle » et soutenait que les signaux sociaux et environnementaux – la façon dont les parents élèvent un enfant – interagissent avec les gènes et les hormones d' un enfant pour le façonner afin qu' il soit identifié comme étant de sexe masculin ou féminin . -Il a admis que la déclaration liminaire du YMCA à la commission était également inexacte en déclarant qu' il y a eu " des vérifications externes du YMCA qui reconnaissent que le YMCA est à l' avant-garde de la sécurité des enfants " . Il a également admis que la déclaration d' ouverture du YMCA était inexacte en soulignant , « des contrôles externes du YMCA ont été effectués et ont indiqué que la sécurité des enfants était une priorité absolue pour l' organisation . » -Il pourrait être libéré dans les 15 ans s' il obtenait la libération conditionnelle . Il pourrait être libéré au bout de 15 ans sous libération conditionnelle . -Il ne croit pas que l' Ontario ira de l' avant . Il ne croit pas que l' Ontario le pas . -Il ne se considère pas comme un héritier d' un éternel ; il refuse la définition . Il ne se pose pas en héritier d' une éternelle , il se refuse à la définition . -Il a estimé , dans un rapport de décembre 2011 , que les frais de cabine pour les animaux de compagnie s' élèvent à 50 millions de dollars par année . Il a estimé , dans un rapport de décembre 2011 , que les frais de bagages à main de Spirit rapportaient 50 M $ par an . -Il a même attiré l' attention des services de renseignements européens qui auraient pu bénéficier de ces communications avant de les transmettre à la NSA . Il a même renvoyé la balle aux services de renseignement européens qui se seraient saisis de ces communications , avant de les fournir à la NSA . -Il donne sa vie à cette organisation , et son désir d' aider le Cameroun est éternel . Il donne sa vie pour cette association , et son envie d' aider le peuple camerounais n' est pas près de s' éteindre . -Il trop près de certains clients , en particulier Scarborough , et il lui permettait d' être compromis . Il s' est trop rapproché de certains clients , en particulier Scarborough , et il a laissé compromettre son indépendance . -Il avait évité de se cacher pendant près d' une décennie , mais a finalement été arrêté en janvier 2011 dans la ville pakistanaise de , où les forces américaines ont tué l' ancien chef d' Al-Qaïda Osama Ben Laden au moins quatre mois plus tard . Il avait réussi à éviter de se faire capturer pendant presque 10 ans mais a finalement été arrêté en janvier 2011 dans la ville pakistanaise de , où les forces américaines l' ancien chef d' Al-Qaïda , Oussama ben Laden , moins de quatre mois plus tard . -Il était en mauvaise santé parce qu' il avait eu une crise cardiaque majeure en décembre 2010 , et , à ce moment-là , il avait utilisé un cadre de marche . physiquement par un très grave accident cardiaque en décembre 2010 , il se déplaçait durant ce voyage avec un . -Il avait juste passé dix jours en Afghanistan , à la création des numéros 198 et 199 de sa célèbre série d' oeuvres d' art . Il venait alors de passer dix jours en Afghanistan , théâtre des 198 et opus de la célèbre série de romans d' espionnage . -Il a servi six présidents français , Georges Pompidou à Nicolas Sarkozy et François , y compris dans leurs foyers de vacances , et nourri un nombre incalculable de personnes dans le monde entier , entouré d' un nuage d' anonymat qu' il n' a pas vu depuis aujourd' hui , au moment où il prend sa retraite à l' âge de 60 ans . Il a servi six présidents français , de Georges Pompidou à François Hollande en passant par Nicolas Sarkozy , y compris sur leurs lieux de vacances , et nourri un nombre incalculable de grands de ce monde dans un qu' il ne brise qu' aujourd' hui , à l' heure de la retraite , à 60 ans . -Il a aidé à réaliser en 2001 une étude de 6 à 9 ans sur les jeunes filles qui relient la graisse corporelle au moment de la puberté . Il a participé à une étude menée sur des filles âgées de 6 à 9 ans qui lie le pourcentage de masse grasse au déclenchement de la puberté . -Il est toujours à l' hôpital ou essaie de faire de l' argent pour l' organisation afin qu' il puisse faire campagne . Il est toujours à l' hôpital ou en train d' essayer de trouver de l' argent pour son association afin de pouvoir mener ces campagnes . -Il est tombé dans un coma d' un mois à la suite de l' attentat et a été " mort " pendant trois mois dans le cadre de son soutien à la vie . Il a sombré dans le coma au lendemain de l' attentat et est resté inconscient pendant un mois , et il est « mort » trois fois alors qu' il était sous assistance respiratoire . -Il n' a jamais su connaître le sens du mot " retraite " . Il n' aura jamais connu le mot retraite . -Il a néanmoins félicité le Gouvernement pour avoir répondu à sa demande d' aide d' urgence qu' il avait d' abord adressée au Premier Ministre au début du mois de mai . Il a néanmoins félicité le gouvernement pour avoir répondu à la demande d' aide urgente qu' il a présentée au Premier ministre début mai . -Il a réuni un véritable festin pour ses fans pour marquer l' occasion . Pour l' occasion , il a concocté un vrai festin pour ses fans . -Il a accepté avec réticence les derniers rôles de dans les studios . Il accepte à reculons les remplacements au pied levé dans les studios . -Il continue de violer ses conditions de mise en liberté sous caution . Il continue à enfreindre les conditions de sa mise en liberté sous caution . -Il a pris sa retraite en 2008 , mais en 2004 il a participé à la création d' une association de défense des otages , " du monde " ( otage du " monde " ) , qu' il depuis plusieurs années . depuis 2008 , il a participé en 2004 à la création de l' association de défense des otages " du monde " , qu' il préside depuis plusieurs années . -Il a dit : " Nous sommes libres , mais je ne suis pas sûr et j' ai pensé que nous pourrions toujours être tués . Il me dit " on est libérés " , mais je n' en suis pas sûr , et me dis qu' on peut encore être tués . -Il a dit : " Je veux vraiment développer une crème glacée invisible " . Il a ajouté : « Je voudrais vraiment développer une glace invisible . » -Il a dit : " Si plusieurs réunions sont organisées avec d' autres chiens , et que la plupart d' entre elles sont souvent par une attitude de plus en plus forte , le comportement inverse donne un comportement moins amical , et on répond de la même façon " . Il a ajouté : « Si vous rencontrez à plusieurs reprises d' autres chiens et si vous remarquez que , la plupart du temps , le mouvement de leur queue vers un côté est associé à une attitude plus amicale et que le côté droit entraîne une attitude moins amicale , vous en fonction de cette expérience . » -Il a dit : " Dans certains cas , il a été trop mal géré " . Il a déclaré : « Dans certains cas , c' est allé trop loin de manière inappropriée . » -Il a dit que les dirigeants européens doivent être honnêtes avec leur propre peuple au sujet des programmes d' espionnage qu' ils utilisent depuis des années . Il a expliqué que les chefs d' État européens devaient être honnêtes avec leurs citoyens au sujet du type de programmes d' espionnage qu' ils appliquent eux-mêmes depuis des années . -Il a dit qu' il était " tout à fait possible " que François demande aux Cardinaux d' obtenir des conseils . Il a ajouté qu' il était « tout à fait possible » que François demande conseil aux cardinaux . -Il a dit qu' il avait l' intention d' éviter la contamination des éléments de preuve , mais qu' il s' agissait d' un " " excessif et d' une exécution médiocre . Il a déclaré que cette décision était destinée à éviter toute contamination des éléments de preuve mais qu' il s' agissait d' un « excès de zèle » et qu' elle avait été mal mise en œuvre . -Il a dit que le a utilisé trois moyens pour enquêter sur les histoires : le piratage téléphonique , la surveillance et la confrontation . Il a déclaré que utilisait trois moyens pour enquêter sur les sujets : le piratage téléphonique , la surveillance et la confrontation . -Il a dit que les différences peuvent être parce que les chiens dans les différentes études ne pas interpréter complètement les animaux dans les films ou comme . Il a indiqué que les différences pouvaient venir du fait que les chiens ayant participé aux différentes études ne considéraient pas entièrement les animaux des films ou les chiens robots comme des chiens . -Il a dit que l' itinéraire est toujours en cours . Il a indiqué que l' itinéraire restait à définir . -Il a déclaré que le nouveau tribunal des médias " sera toujours partial parce qu' il s' agit d' une extension du gouvernement " , et que les restrictions sur le contenu et la publicité la place du Kenya dans l' économie mondiale . Il a indiqué que le nouveau tribunal des médias « sera toujours partial car il s' agit d' un prolongement du gouvernement » et que les restrictions relatives au contenu et à la publicité à la place du Kenya dans l' économie mondiale . -Il a déclaré qu' il y avait de graves problèmes dans le recrutement , le dépistage , l' induction , la formation et la supervision du personnel . Il a indiqué qu' il y avait des « problèmes graves » dans le recrutement , le contrôle , l' initiation , la formation et la supervision du personnel . -Il a dit en général qu' un pape autant de cardinaux que nécessaire pour augmenter le nombre d' électeurs cardinaux à 120 , et de nombreux cardinaux âgés de plus de 80 ans comme il le souhaitait . Il a déclaré qu' un pape nommait en général autant de cardinaux qu' il le fallait pour que le nombre de cardinaux électeurs atteigne 120 et autant de cardinaux âgés de plus de 80 ans qu' il voulait . -Selon lui , les efforts louables ont fait preuve d' un manque de soutien de la part du gouvernement fédéral face au Il dit qu' Obama a fait des efforts louables par l' du Parti républicain -Il dit qu' il ne le fait pas , dit : " Oh oui , il a fait ça " . Il affirme le contraire , mais nous disons « Bien sûr que si » . -Il comparaîtra devant le tribunal provincial à Halifax en décembre . Il comparaîtra en cour provinciale , à Halifax , en décembre . -Il doit encore trouver des candidats pour dresser des listes et défendre ses idées dans le domaine politique . " Il est probable qu' il sera possible de le faire à temps pour les élections municipales de 2014 " , a-t-il ajouté . Reste à trouver des candidats pour former des listes et défendre ses idées sur la scène politique " les élections municipales de 2014 , cela risque d' être juste , " . -Il n' a eu qu' un bras cassé au premier abord , mais une infection s' est développée et a circulé sur son cerveau , créant un qui l' a affecté pour le reste de sa vie . Au début , il n' a eu qu' un bras cassé , mais il a contracté une infection qui s' est propagée jusqu' à son cerveau , créant un dont il allait porter les séquelles le restant de sa vie . -Il est alors devenu le premier membre de haut rang du gouvernement américain à admettre que l' espionnage américain avait franchi la ligne , mais il a souligné que ses droits n' avaient pas été bafoués . Il est ensuite devenu le premier membre de haut rang du gouvernement des États-Unis à admettre que l' espionnage américain avait dépassé les bornes , mais a insisté sur le fait que les droits de chacun n' avaient pas été violés . -Il a l' enquête de la police autant que possible pour qu' elle puisse poursuivre ses activités criminelles . Il a entravé l' enquête de la police autant qu' il a pu pour lui permettre de poursuivre ses activités criminelles . -Il a dit au FT qu' il voulait dire que l' accord n' était plus respecté par . Il a déclaré au FT qu' il voulait dire que l' accord n' était plus respecté par le . -Il a pris part à la Conférence de en janvier 1942 , où " la solution finale " a été prise et a été notamment placé sous le commandement d' Adolf , responsable de la " logistique " de l' extermination des Juifs et condamnés à mort et exécutés en Israël en 1962 . Il participa à la conférence de en janvier 1942 , qui décida de la " solution finale " et eut notamment sous ses ordres Adolf , responsable de la " logistique " de l' extermination des juifs , condamné à mort et exécuté en Israël en 1962 . -Il a prêté serment devant les 18 juges du tribunal de dans son siège construit pour ressembler à un temple égyptien ancien . Il a prêté serment devant les 18 juges en robe noire de la Cour dont le bâtiment situé en bordure du Nil a été construit pour ressembler à un temple égyptien . -Il porte le voile et le montre en même temps , et s' adresse à des personnalités différentes , comme il l' a fait tout au long de sa carrière , surtout avec ses personnalités : et . Il dissimule et révèle à la fois , joue à adopter différentes personnalités , comme il l' a fait tout au long de sa carrière , notamment avec ses personnages et . -Il est très souvent " " dans mes rêves , d' une manière très particulière , si bien que lorsque je me réveille , je suis triste quand j' ai réalisé que c' était seulement un rêve . Très souvent il " débarque " dans mes rêves d' une façon si précise , si vivante qu' au réveil je suis encore plus triste de constater que ce n' était qu' un rêve . -Il était et , par conséquent , il pouvait descendre en bas de l' escalier qui était resté intact , bien que la paroi arrière de la maison fût entièrement détruite . et sauf , il a pu rejoindre le rez-de-chaussée par l' escalier resté intact tandis que le mur arrière de la maison était totalement détruit . -Il était un peu comme un alter ego , nous n' avions même pas besoin de parler . C' était un peu comme un , il n' y avait pas besoin de parler entre nous . -Il était accompagné de deux journalistes allemands lors de sa visite en Russie . Lors de sa visite en Russie , il était accompagné de deux journalistes allemands . -Il a également été reconnu coupable d' un certain nombre d' autres accusations liées au terrorisme , dont une vague d' attentats à la bombe contre des églises en Indonésie à la veille de Noël en 2000 . Il a également été reconnu coupable de nombreux autres chefs d' accusation en relation avec le terrorisme , notamment une vague d' attentats visant des églises dans toute l' Indonésie à la veille de Noël en 2000 . -Il était plein de louanges . Il a dit des éloges . -Il a reçu un accueil officiel par une troupe de l' armée qui a joué l' hymne national comme s' il s' agissait de l' attention . Il a été officiellement accueilli par une fanfare militaire qui a joué l' hymne national alors qu' il se mettait au . -On lui a donné des hormones féminines à l' âge de 12 ans . On lui a fait prendre des hormones féminines à l' âge de 12 ans . -Il a été emprisonné pour une durée minimale de six ans . Il a été condamné à une peine de 6 ans de prison minimum . -Il sur le sable sur toute la longueur de l' arène , et même s' il avait été si petit qu' il a fait un beau travail . Il a parcouru toute la longueur de la piste en courant et même s' il semblait être tout petit , il a fait un merveilleux travail . -Il m' a dit que , quand il est là , il me parfois une allée entre deux bâtiments et je pense à lui , Hey , si j' ai perdu tout mon argent , peut-être je vais habiter ici . Il me disait que , lorsqu' il était ici , parfois il une ruelle entre deux immeubles et se disait Eh bien , si je perds tout mon argent , peut-être que je vivre ici . -Il portait " un uniforme général " . Il portait " un uniforme de général " . -Il ne mourra pas et ne prendra pas sa retraite . Il ne mourra pas et ne partira pas en retraite . -Il aura probablement un autre casque spécial pour la course du dimanche , destinée à marquer son quatrième titre mondial consécutif . Il est probable qu' il aura un autre casque spécial dimanche , pour la course , destiné à marquer son 4e titre mondial consécutif . -Il a travaillé comme les grands reporters d' avant-guerre , comme Albert Londres , qui s' est rendu sur place et a reçu avec succès des enquêtes sérieuses et longues " . Il travaillait " comme les grands reporters d' avant guerre , du type Albert Londres , qui allaient sur place et revenaient avec de vraies et longues enquêtes " . -Il m' a parlé , m' a constamment encouragé , il a vécu dans mon corps . Il me parlait , m' encourageait constamment , il habitait mon corps . -Heinrich Muller , qui n' a jamais été découvert après avoir disparu à la fin de la Seconde Guerre mondiale , a été enterré dans une tombe commune dans un cimetière juif à Berlin , le chef du Mémorial de la Résistance allemande , le professeur Johannes , confirmé par . Müller , qui a disparu à la fin de la Seconde Guerre mondiale sans qu' on ne retrouve jamais sa trace , a en fait été enterré dans une fosse commune d' un cimetière juif de Berlin , affirme le dirigeant du Mémorial de la résistance allemande , le professeur Johannes , dans . -Heinrich Muller fut l' un des grands personnages du Troisième Reich à ne jamais être capturé . Heinrich Müller fait partie des personnalités importantes du Troisième Reich qui n' ont jamais été capturées . -Aider les autres à vivre une vie plus heureuse , en honorant une promesse qu' il a faite à son père , c' est quelque chose qui lui apporte une grande joie . Aider les autres à vivre heureux en tenant la promesse qu' il a faite à son père est quelque chose qui lui apporte une immense joie . -Par conséquent , en cas de mortalité à grande échelle dans la région , causée par une épidémie virale ou une capture accidentelle , nous savons que nous l' extinction d' une espèce plutôt que la disparition locale d' une population appartenant à une espèce beaucoup plus large . Dès lors , si un problème de mortalité massive causée par une épidémie virale ou de capture accidentelle se pose dans la région , nous pourrions avoir affaire à l' extinction d' une espèce , plutôt qu' à une disparition locale d' une population d' une espèce plus largement répandue sur Terre . -Ainsi , des millions de personnes dans la région se tournent vers parce que les familles se sentent par l' augmentation du coût de la vie et par la flambée de la dette des ménages et des consommateurs . Par conséquent , des millions de personnes à travers la région se tournent vers les prêteurs sur gage alors que les familles ressentent la pression qu' exerce l' augmentation du coût de la vie et le surendettement des ménages et des consommateurs . -D' où la question de savoir si le fond de la question des n' est pas la capacité ou le droit d' un pays d' utiliser des instruments d' interception , à savoir la question de l' absence totale de débat préalable - en particulier au sein des parlements - sur la justification de ces régimes , sur la mesure dans laquelle ils devraient être utilisés et , en fin de compte , sur la violation des libertés . Donc la question de fond derrière cette affaire NSA n' est pas tant la capacité ou le droit des pays de se doter d' outils d' interception , que la question de l' absence totale de débat préalable , notamment au sein des Parlements , sur la justification de tels systèmes , le périmètre qui doit être le leur , et , en fin de compte , la question des atteintes aux libertés . -D' où le choc que sa perte causa dans le jeune diplômé alors j' étais . C' est dire le choc que sa disparition a provoqué chez le jeune que j' étais à l' époque . -C' est pourquoi nous avons effectué 64 opérations depuis l' hiver dernier . Nous avons ainsi effectué , depuis la fin de l' hiver dernier , 64 interventions . -Son nouvel album , " " , a été lancé au de cette semaine . Son nouvel album , « Prism » , a débarqué à la première place des charts cette semaine . -Ses propriétaires Scott , 25 ans , et sa petite amie Hall , 20 ans , dans la carrière de à , dans le Sud-Ouest de l' île , ont été réunis avec Ruby et ont remercié le service de lutte contre l' incendie et le sauvetage de l' ouest du Yorkshire . Ses propriétaires , Scott , 25 ans , et sa petite amie Hall , 20 ans , se sont rendus à la carrière de à , dans le West Yorkshire , pour retrouver Ruby et ont remercié le Service d' incendie et de secours de West Yorkshire . -Voici mon point de vue : un sentiment de plus en plus et de l' euro en Europe se pour tuer l' euro . & # 160 ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Voilà ce que je pense : très bientôt , le sentiment et unique en Europe va s' unifier et finira par tuer l' euro . -Ici , les gens étaient par une vieille puissance indigène , et ils étaient encore plus radicaux pour eux . Ici , des hommes d' un Ancien Régime indigène , et d' autant plus radicaux . -Voici les vraies nouvelles de l' Europe : le grand projet de l' de rejoindre l' Europe est enfin en panne . Voici la véritable actualité concernant l' Europe : le grand plan d' après-guerre visant à unifier l' Europe est définitivement au point mort . -Nous sommes à nouveau confrontés à un débat , secoué par un nouveau type d' immigration , menacé par l' afflux de protestations des minorités , qui est absorbé dans le cadre européen . La voici à nouveau en débat , par une immigration d' un type nouveau , menacée par la montée de minorités , absorbée dans l' espace européen . -Ici , nous avons essayé de l' examiner à d' autres espèces . Ici , nous avons essayé de voir ce qu' il en était chez d' autres espèces . -Sa dépendance à la cocaïne l' a laissé définitivement et a été vulnérable aux motivations des dirigeants des groupes criminels organisés qui lui ont permis d' obtenir des informations précieuses sur les enquêtes policières . Son addiction à la cocaïne l' a compromis à jamais et l' a rendu vulnérable aux des membres dirigeants des groupes criminels organisés qui l' ont chargé d' obtenir des informations précieuses concernant des enquêtes de police . -Son arrivée a été accueillie par des chants , " L' armée et le peuple sont une main " , des centaines de personnes réunies dans la salle principale de l' université . À son arrivée , il a été accueilli par le chant « Le peuple et l' armée , main dans la main ... » par des milliers de personnes rassemblées dans le grand amphithéâtre de l' université . -Son affaire fait suite à l' intervention du caporal Glen Shwe , qui a déclaré devant une commission parlementaire le mois dernier qu' il avait été forcé de s' absenter avant qu' il ne soit prêt parce qu' il n' avait pas respecté la règle de l' universalité du service . Son cas est survenu après celui du caporal Glen Kirkland , qui a déclaré devant une commission parlementaire , le mois dernier , qu' il avait été poussé vers la sortie avant d' être prêt parce qu' il ne répondait pas à la règle d' universalité du service . -Son salaire fixe est de un dollar nominal . Son salaire fixe n' est que d' un dollar symbolique . -Sa vie n' est pas en danger , mais il est toujours hospitalisé . Son pronostic vital n' est plus engagé mais il est toujours hospitalisé . -Sa mère Donna , 30 ans , a déclaré : " et sont un grand double acte . Sa mère Donna , 30 ans , a déclaré : « et font un excellent duo . » -Sa mère était un cuisinier dans un château de la . Sa mère était elle-même cuisinière dans un château de . -Son état a récemment mis en place des dispositifs de suivi sur 500 véhicules pour tester un système . Son État a récemment fait installer des traceurs sur 500 voitures pour tester un système de paiement au mile . -Son arme a été saisie . Son arme a été saisie . -Historiquement , les enfants nés à la fois d' hommes et de femmes ont été appelés , nommés pour le beau dieu grec possédant une double sexualité . Historiquement , les enfants nés sans organes sexuels masculins ou féminins étaient appelés , d' après le nom du magnifique dieu grec qui avait un double sexe . -Historiquement , cela signifiait que c' était le travail blanc qui devait être protégé - et si , dans certaines régions , on voyait les échos de cet artefact historique , je ne serais pas surpris . Historiquement , cela signifiait que le travail des Blancs devait être protégé – et je ne serais pas surprise que certains pays de la région y voient comme un écho à cet historique . -Historiquement , les prix élevés de l' or ces deux dernières années s' ajoutent à la ruée vers l' achat de biens personnels , alors que les gens ont l' occasion d' investir dans la valeur de leurs bijoux familiaux . Les prix historiquement élevés de l' or au cours des deux dernières années ont fait que les gens se sont empressés de mettre en gage leurs effets personnels pour toucher la valeur en espèces de leurs bijoux . - dans les produits du boeuf Du cheval dans des produits au boeuf -La viande de cheval a été détectée dans des produits alimentaires en vente libre vendus par deux petits détaillants de produits laitiers à faible prix , l' Food Standards Agency l' a annoncé aujourd ' hui . De la viande de cheval a été décelée dans des boîtes de conserve de produits à base de boeuf , vendues par deux petites enseignes britanniques de distribution à bas prix , a annoncé aujourd' hui l' agence chargée de l' hygiène alimentaire . -House in in the Old Quebec Une maison la proie des flammes dans le Vieux-Québec -Comment la génétique vie Comment la biométrie va envahir nos vies -Comment les comédies musicales françaises ? Comment évoluent les comédies musicales françaises ? -Comment pouvons-nous offrir des garanties sur la durée de l' emploi lorsqu' il n' y a pas de salariés sur place ? Comment pouvons-nous donner des garanties de durée d' emploi quand il ne reste plus d' employés sur le site ? -Comment les examens médicaux et le compte-rendu avec le ( Direction générale de la sécurité extérieure ) sont-ils allés ? Comment se sont passés les examens médicaux et le ? -Comment pouvez-vous réagir aujourd' hui à la libération des quatre otages au Niger ? Comment , aujourd' hui , à la libération des quatre otages du Niger ? -Le directeur de l' Administration des FAA , M. Philippe , mettra l' accent sur la rapidité de la modification de la nouvelle version à l' occasion d' une conférence de presse . La rapidité avec laquelle le changement sera mis en place variera d' une compagnie aérienne à l' autre , a déclaré l' administrateur de la FAA Michael lors d' une conférence de presse . -La façon dont le dos de l' avion est prévu - en particulier s' il y a des sièges à 9 ou 10 - est le point central des évaluations de performance économique qui s' appliquent à de nouveaux modèles d' avion à réaction " " . La façon dont les sièges seront disposés à l' arrière de l' avion – en particulier , y aura-t-il 9 ou 10 sièges de front – est essentielle pour le rendement économique du segment des « » . -Comment avez-vous été reçu lorsque vous avez atteint le sol français ? Comment s' est déroulée votre arrivée sur le sol français ? -Toutefois , les ont l' avantage de ne pas exiger de chèques de crédit ou de preuve de salaire , et peuvent organiser des prêts plus rapidement que les banques . Cependant , les prêteurs sur gage ont l' avantage de ne pas demander de vérification de solvabilité ou de preuve de salaire , et peuvent débloquer les prêts plus rapidement que les banques . -Cependant , n' est pas la seule voie à avoir été sortie . Mais n' est pas le seul titre à dévoiler ses charmes . -Toutefois , des enregistrements de CCTV ont ensuite révélé qu' elle était retournée à son hôtel en toute sécurité après avoir été séparée de son ami australien . Cependant , les enregistrements de vidéosurveillance ont plus tard révélé qu' elle était rentrée à son hôtel en toute sécurité après s' être séparée de son petit ami australien . -Cependant , Damas doit encore commencer à détruire les armes et les stocks existants . Cependant , Damas doit encore démarrer la destruction des armes et des réserves existantes . -Cependant , je ne peux rien faire ; je perds du sommeil et je me demande constamment ce qu' il pense . Pour moi , c' est impossible d' aller dans un sens contraire , je n' en pas et je me demande toujours ce qu' il en . -Cependant , je me réfère à l' évocation de la transformation des anciens en une sorte d' incubateur industriel , mais à quel coût pour les contribuables . & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; Je donne cependant une mention honorable à la transformation de l' ancienne en une sorte d' incubateur industriel , mais à quel coût pour les payeurs de taxes ? -Cependant , l' intérêt croissant de Washington dans le Pacifique suggère que la Grande-Bretagne sera laissée dans le froid de l' Atlantique . Toutefois , l' intérêt croissant de Washington pour le Pacifique laisse penser que la Grande-Bretagne sera laissée pour compte dans l' Atlantique . -Cependant , Yahoo et Google ont nié toute implication dans ces pratiques . Mais Yahoo et Google ont nié tout lien avec ces pratiques . -Toutefois , à la suite du meurtre récent d' un agent de voyages australien , Michelle Smith , à Phuket , la Thaïlande pourrait également essayer de réparer son image touristique ternie , ce qui lui a valu un acquittement . Toutefois , après le meurtre récent à Phuket de Michelle Smith , une agente de voyage australienne , la Thaïlande pourrait également chercher à redorer son image détériorée auprès des touristes , ce qui pourrait conduire à un acquittement . -Toutefois , pour l' économiste , , du Centre d' études politiques , " on ne peut s' attendre à une amélioration à court terme " . Cependant , pour l' économiste , du Centre des Etudes politiques , " on ne peut pas attendre une amélioration à court terme " . -Cependant , étant donné la facilité avec laquelle leur comportement peut être enregistré , il ne sera probablement pas assez long avant de comprendre pourquoi leurs queues suivent parfois un chemin , parfois l' autre . Toutefois , étant donné la facilité avec laquelle leurs comportements peuvent être enregistrés , il ne faudra sûrement pas longtemps avant que nous comprenions pourquoi leur queue bouge parfois d' un côté et parfois de l' autre . -Cependant , étant donné l' absence d' études approfondies , il est impossible de savoir si cette ligne de conduite réellement les utilisateurs . Mais , faute d' études approfondies , il est impossible de savoir si ces discours convainquent vraiment les utilisateurs . -Cependant , il veut garder l' usine à Amiens avec au moins 333 employés bien payés . Or , il veut garder l' usine d' Amiens avec au moins 333 emplois bien payés . -Cependant , il souffrait d' une inflammation au genou gauche à la suite de son retour de l' ordre international de ce mois . Il souffrait néanmoins d' une inflammation du genou gauche lors de son retour de sélection , en début de mois . -Cependant , si aucune nouvelle commande n' est annoncée dans les prochains mois , nous que le marché sera plus sceptique sur le programme . Toutefois , si aucune nouvelle commande n' est annoncée au cours des prochains mois , nous nous attendons à ce que le marché devienne plus sceptique au sujet du programme . -Cependant , le personnel du renseignement , le personnel de l' armée et les militants politiques ont confirmé cette fois qu' il avait perdu sa vie lors de la grève qui avait été menée dans la région du . Mais des responsables du renseignement , de l' armée ainsi que de la mouvance activiste ont affirmé cette fois qu' il avait perdu la vie dans cette frappe , menée dans la région du . -Cependant , la chanteuse du chanteur d' enfance , Alex Turner , les a obligés à reporter le spectacle . Cependant , sa a forcé le chanteur Alex Turner a reprogrammer le spectacle . -Toutefois , lorsqu' un cardinal atteint 80 , il n' est plus autorisé à participer à l' élection d' un pape , ce qui revient à un groupe de 120 électeurs " " . Toutefois , dès qu' un cardinal atteint l' âge de 80 ans , il ne peut plus participer à l' élection d' un pape – cela revient donc à un groupe de 120 « cardinaux électeurs » . -Cependant , une fois de plus , il y a une tendance à l' , à l' intellectuel , à la en , comme si les deux pieds sont liés , d' un sujet à l' autre . On y retrouve pourtant , une fois de plus , la tentation de l' , le vagabondage intellectuel , la mise à sauter comme à pieds joints d' un sujet à l' autre . -Toutefois , notre familiarité avec les États-Unis ne fait que souligner encore plus une chose qui nous tient particulièrement à coeur . Mais le américain ne fait que mieux souligner ce qui nous est propre . -Toutefois , parler de vérité n' est pas un crime . Cependant , dire la vérité n' est pas un crime . -Cependant , le parti de la devrait être au moins représenté à Aubervilliers par ma candidature et dans d' autres villes j' espère , M. . Le Parti de la banlieue devrait quand même être représenté à Aubervilliers à travers ma candidature et j' espère aussi dans d' autres villes . -Cependant , le comité a recommandé à la FAA de permettre aux pilotes d' ordonner aux passagers d' éteindre les dispositifs pendant les atterrissages en cas de visibilité faible . Toutefois , le comité a recommandé que la FAA autorise les pilotes à demander aux passagers d' éteindre leurs appareils pendant les atterrissages aux instruments dans des conditions de faible visibilité . -Cependant , la crise économique actuelle s' est avérée un point de mire . Toutefois , la crise économique actuelle a marqué un tournant . -Toutefois , la famille a refusé " pour des raisons financières " et la situation se trouve dans l' impasse . Pourtant , la famille refuse " pour raison financière " et la situation est bloquée . -Toutefois , la situation reste précaire . La situation demeure toutefois délicate . -Cependant , il y a eu une époque où les Français croyaient qu' ils pouvaient réparer les dommages causés à leur histoire et surmonter la malédiction du numéro deux . Il y a pourtant eu une époque où les Français ont cru pouvoir la robe déchirée de leur histoire , et vaincre la malédiction du chiffre deux . -Toutefois , on craint que les petites et moyennes entreprises restent vulnérables au piratage et à la surveillance . Toutefois , on craint que les petites et moyennes entreprises restent vulnérables au piratage et à la surveillance . -Cependant , ils ont aussi critiqué chiites Al , affirmant qu' il est partiellement responsable de la reprise de la violence à cause de ses " politiques sectaires et autoritaires " . Mais ces derniers ont aussi critiqué Nouri Al , un chiite , lui attribuant la responsabilité partielle de la reprise des violences par sa " politique sectaire et autoritaire " . -Cependant , nous sommes toujours intéressés par l' évolution de la réglementation qui pourrait profiter aux passagers et nous suivrons de près la décision de la FAA et les raisons qui l' ont motivé . Toutefois , nous sommes toujours intéressés par les évolutions réglementaires qui pourraient bénéficier aux passagers , et nous examinerons de près la décision de la FAA et les raisons de celle-ci . -Toutefois , il a dit qu' il était encouragé par la réponse mondiale à " mon acte d' expression politique " . Toutefois , il a dit qu' il avait été encouragé par la réaction mondiale face à « mon acte d' expression politique » . -Beaucoup de choses que nous dénonçons les États-Unis , je ne vois pas comment elles peuvent être sanctionnées . On a beau les dénoncer , je ne vois pas de quelle manière ils pourront être punis . - a également été condamné à des frais de £ 90 et à une amende de 100 livres . a également été condamné à payer des frais s' élevant à 90 £ et une surtaxe pour victime de 100 £ . -Des centaines de soldats et de policiers ont le bâtiment , car Morsi est arrivé peu après des heures de dans un petit convoi . Des centaines de soldats et policiers gardaient le bâtiment lorsque M. Morsi est arrivé peu après 11 heures , heure locale , suivi d' un petit convoi . - . Sixième édition du Play ce samedi à Hyères -Je suis 58 et j' ai toujours vécu seul en le perdant , il est et restera l' amour de ma vie . J' ai 58 ans , j' ai toujours vécue seule depuis son départ , il est et restera l' amour de ma vie . -Je suis en train d' écouter ce qui est dit . J' écoute ce qui se dit . -Je n' essaie pas de plaire à quiconque . Je n' essaye de faire plaisir à personne . -Je suis en train de lire un livre terriblement triste ces jours-ci . Je suis en train de lire un livre triste ces jours-ci . -Je suis désolé si mes propos ont offensé quelqu' un . Si j' ai blessé quelqu' un par mes paroles , j' en suis désolé . -Je suis sûr que a été aidé par ses beaux vêtements , il a vraiment regardé la partie . Je suis sûr que ses beaux vêtements ont aidé , il avait vraiment fière allure . -J' ai atteint un point de détresse politique . Je suis au bord du désespoir politique . -Je me suis rendu compte de mes actes répréhensibles . Je me suis rendu compte de mon erreur . -J' ai également rencontré les services de renseignement . J' ai aussi rencontré les services de renseignement . -Je me souviens aussi d' une courte vidéo passée à parler à Roger et . Je me souviens aussi d' une courte nuit à discuter avec Roger et . -J' ai hâte de gagner ma confiance . J' ai hâte de regagner votre confiance . -Je suis comme mon père , " à l' intérieur et à l' extérieur " , semble-t-il . Je ressemble à mon père , " au dedans comme au dehors " . -Je vous assure qu' il n' y a pas de victimes innocentes dans ce processus . Je vous assure que les personnes innocentes ne seront pas victimes d' abus dans le cadre de ce processus . -Je ne me souviens pas de ce qui a été discuté , mais ce n' est certainement pas le cas . Je ne me rappelle pas de quoi nous avons discuté mais en tout cas , ce n' était pas de ça . -Je ne peux que fermer les yeux et les ouvrir à nouveau ... Je ne peux que fermer les yeux et les rouvrir lentement ... -Je le rêve toujours , peut-être pas tous les soirs , mais plusieurs fois par semaine pour tous . Je rêve constamment d' eux , peut-être pas toutes les nuits mais plusieurs fois par semaine c' est certain . -Je ne pouvais pas croire qu' elle était là pour la première fois . Je n' ai d' abord pas cru qu' elle était là-haut . -Je pourrais encore mener une vie personnelle . J' ai pu avoir une vie personnelle . -J' ai du crédit à d' importantes et belles réalisations , à partir de l' adoption de la loi sur les services de secours à la sortie de notre aéroport militaire iraquien , la fin de ce que l' on dit : " ne nous demandez pas de dire " , à propos de l' assassinat d' Oussama Ben Laden . Je mets au crédit d' Obama de grandes réalisations , depuis le passage de l' Care Act ( loi sur les soins abordables ) jusqu' au départ d' Irak de notre armée , en passant par la fin du « ni vu ni connu » et l' assassinat d' Oussama ben Laden . -Je l' ai fait principalement parce que je après l' argent et la renommée . Je l' ai fait principalement parce que je après l' argent et la célébrité . -Je ne sais pas si j' ai adopté sa façon de penser ou si je suis tout simplement le mien , est-ce qu' il est génétique ? Je ne sais pas si j' ai adopté sa façon d' être ou si je suis simplement comme lui : est-ce génétique ? -Je ne sais pas si c' est ce que je devrais attendre . Je ne sais pas si c' est ce à quoi je m' attendais . -Je ne sais pas qui a été dit que ceux qui meurent ne sont pas oubliés , mais invisibles . Je ne sais plus qui disait que les personnes décédées ne sont pas des oubliés mais des invisibles . -J' attends du Royaume-Uni qu' il se de ce projet , espérant profiter aux Etats-Unis . Je m' attends à ce que le Royaume-Uni prenne ses distances par rapport à ce projet , et j' espère qu' il fera les yeux doux aux États-Unis . -Je me sens sa présence et cela me rend heureux . Je ressens sa présence et cela me rend heureuse . -Je ressens sa présence dans chaque salle et cela me rend heureux . Dans toutes les pièces je ressens sa présence et cela me rend heureuse . -J' ai pensé qu' un second mandat pour Obama , libéré des pressions des futures élections , permettrait de réaliser l' espoir que nous avons entendu depuis si longtemps . Je sentais qu' Obama , élu pour un second mandat en étant libéré de la pression d' élections futures , tiendrait les promesses dont on entendait parler depuis si longtemps . -Je pense qu' il n' y a rien de mal à faire de la musique traditionnelle . Je suppose qu' il n' y a pas de mal à rester traditionnel . -J' avais une relation difficile avec lui jusqu' à ce qu' il devienne vieux et malade . J' ai eu des relations difficiles avec lui jusqu' à ce qu' il devienne vieux , malade . -J' ai dû subir un premier examen médical en Corse . J' avais eu droit à un premier contrôle médical en Corse . -On me l' a toujours dit . On me l' a toujours dit . -J' ai toujours été bien accueilli en Afrique , où j' ai un très grand nombre de lecteurs . Je suis toujours bien accueilli en Afrique où je compte de très nombreux lecteurs . -Je le préconise depuis vingt ans . Je plaide en faveur de ce projet depuis 20 ans . -J' ai même eu des problèmes de travail en raison de ces convictions . J' ai d' ailleurs eu des problèmes dans mon travail pour ces convictions . -Je dois vivre chaque jour sans voir un plus grand nombre de petits-enfants et mes filles . Je dois vivre tous les jours sans avoir d' autres petits-enfants , ni voir mes filles . -Je n' ai pas rejoint les deux dernières années pour donner des bonbons dans mon appartement et cela ne sera probablement pas cette année . Ces deux dernières années , je n' ai pas distribué un seul bonbon et je ne le ferai certainement pas plus cette année . -J' espère qu' il verra ce que je fais . J' espère qu' il voit ce que je fais . -Je ne peux tout simplement pas me croire là où elle était . Je n' arrive pas à croire qu' elle était là-haut . -Je sais qu' il aimerait une garantie de quatre ans . Je sais qu' il voudrait une garantie de quatre ans . -Je me réjouis d' examiner le rapport de l' enquête avec les ministres britanniques l' année prochaine et de décider ensemble des prochaines étapes . « Je suis impatient de lire le rapport d' examen avec les ministres britanniques l' année prochaine et nous déciderons alors ensemble des prochaines étapes . » -J' ai perdu mon père le 22 août 2008 au sujet du cancer de l' amiante . J' ai perdu mon père le 22 août 2008 des suites d' un cancer de l' amiante . -J' ai aimé partager ce moment avec lui et il a aimé parler de cette mémoire partagée qui a eu lieu après sa mort . J' ai aimé partager ce moment avec lui et je prends plaisir à raconter ce souvenir commun vécu après sa mort . -Je ne l' ai jamais cru à l' époque . Je ne l' ai jamais cru à l' époque . -Je rêve souvent des derniers moments que j' ai rencontrés avant qu' ils ne soient trop tard , bien qu' il y ait encore une chose qui m' empêche de profiter du moment . Je rêve souvent des derniers instants que je dois partager avec eux avant qu' il soit trop tard , seulement , il y a toujours une chose qui m' empêche d' accomplir ce moment . -Je me réveille souvent en détresse parce qu' elle me frappe et je me sens profondément oublié . Je me réveille souvent lorsque ça m' arrive et je ressens profondément leur absence . -J' ai vraiment le sentiment qu' il devrait suivre les traces des autres . Je pense vraiment qu' il aurait dû suivre le même chemin que les autres hommes . -Je me souviens d' avoir pris un bain d' une heure , Je me souviens avoir pris un bain pendant une heure . -Je vois ses yeux se reposer sur moi . Je vois ses yeux se poser sur moi . -Je dois dire que je suis et que je travaille avec des responsabilités , je suis et , je vais chanter , je suis une grand-mère , j' ai beaucoup d' amis , mais mon coeur et mon âme sont sur lui , je ne parle jamais que de lui , à l' exception de mon fils , et je n' irai jamais au cimetière . Je tiens à préciser que je suis sociable avec un métier à responsabilité , je ris et je chante , je sors , je suis grand-mère , j' ai plein d' amis , mais mon cœur et mon âme lui appartiennent et je ne parle jamais de lui sauf avec mon fils , et je ne vais jamais au cimetière . -Je vis toujours dans la même maison et , souvent , je me sens en train de le voir , je tourne rond et je vois qu' il n' est pas là , mais je sais qu' il le fait , et je lui parle . Je vis toujours dans la même maison , et souvent , je me sens observée et je me retourne pour voir s' il n' est pas là , je sais que c' est lui et je lui parle . -Je soupçonne que l' Écosse pourrait bien s' en tenir à un arrangement européen plus souple , à condition que nous disposions de notre propre monnaie . Je crois que l' Écosse pourrait s' accommoder d' un accord européen plus libre à condition que nous notre propre devise . -Je pense que M. le sait . Je pense que M. sait cela . -Je pense qu' il est concevable que ces données soient utilisées à des fins d' avantages mutuels . J' estime qu' il est concevable que ces données soient utilisées dans leur intérêt mutuel . -Je pense que c' est justifié , légitime , juste . Ça me semble pourtant fondé , légitime , frappé du bon sens . -Je pense qu' il y a un problème de sémantique . Il y a selon moi un problème de sémantique . -Je pensais qu' elle avait disparu . Je qu' elle était partie . -J' essaie de distinguer les questions de mon passé en otage et mon rôle de président de l' association " du Monde " , qui me permet de garder une distance et me semble plus intéressant à traiter . J' essaie de faire la part des choses entre les questions sur mon passé en tant qu' otage et ma casquette de président de l' association " du monde " qui me permet de mettre de la distance , et me semble bien plus intéressante à aborder . -J' ai voté pour Obama à nouveau en 2012 , mais pas parce que j' étais excité par sa candidature . J' ai à nouveau voté pour Obama en 2012 , mais pas parce que j' étais enthousiasmé par sa candidature . -J' ai voté pour le président Obama deux fois , partageant l' espoir dans la possibilité d' un changement J' ai voté deux fois pour le président Obama , croyant en la possibilité d' un changement -J' avais 28 ans et notre fils avait 6 ans . J' avais 28 ans et notre fils 6 ans . -J' avais peur de l' oublier , ou peut-être je ne savais pas comment je pourrais continuer à " passer du temps avec lui " . J' avais tellement peur de l' oublier ou plutôt je ne savais pas comment j' allais continuer à le " fréquenter " . -J' attendais de voir des de dents et un combat se lever à la porte . Je m' attendais à voir des de dents et une bagarre éclater à la porte . -J' étais sur la news cette même soirée , et à la télévision le lendemain aussi . Le , je suis au 20h , le lendemain aussi je suis à la télé . -J' étais stérilisée à la naissance et personne ne m' a jamais dit . J' ai été stérilisé à la naissance – et personne ne me l' avait jamais dit . -Je n' ai pu assister à son enterrement , et trois semaines après sa mort , j' ai donné naissance à un petit garçon . Je n' ai pas pu assister à son inhumation et trois semaines après son décès , je naissance à un petit garçon . -Je n' étais pas satisfait , mais il était vraiment difficile de poser des questions . Je n' étais pas heureux , mais c' était vraiment difficile de poser des questions . -J' étais très proche de lui , j' ai toujours agi selon ce qu' il aurait cru ou aurait aimé . J' étais très proche de lui , j' ai toujours agi en fonction de ce qu' il pensait ou de ce qu' il appréciait . -J' ai traversé toutes les étapes : l' incompréhension , la colère , le chagrin , les larmes qui allaient se produire , n' importe où , à n' importe quel moment , dans des endroits inhabituels , à des moments . Je suis passée par toutes les phases : l' incompréhension , la colère , la douleur , les pleurs qui se déclenchent seuls , n' importe où , n' importe quand , dans des lieux insolites , des moments . -Je vais suivre les faits , où qu' ils aillent . Je vais donner suite aux faits , où qu' ils mènent . -Je resterai le patchwork créé par des médecins , et marqués . Je resterai le patchwork qu' ont créé les médecins , et . -Je n' aucun effort pour remplir le mandat que m' ont confié les États membres de réaliser notre vision des objectifs de développement durable qui s' appuient sur le succès des OMD . Je ferai le maximum pour m' acquitter du mandat qui m' a été confié par les États membres pour réaliser notre vision des objectifs de développement durable qui s' appuient sur le succès des OMD . -J' aurais honte de défendre un tel bilan . J' aurais honte de défendre un tel bilan . -Je les mains quand je la mienne ; je lui mon corps . Je voyais ses mains en regardant les miennes , je lui mon corps . -Je n' ai pas quitté cette maison pour n' importe quoi dans le monde . Pour rien au monde je ne cette maison où nous avons été heureux et ou son esprit vit avec moi . -L' identité est une préoccupation contemporaine . Identité : c' est une préoccupation d' époque . -Si , comme cela est très possible , les partis sont bien parvenus à ces élections , un tel bloc pourrait dominer le Parlement européen pour la première fois . Si les partis font bonne figure à ces élections , ce qui est très possible , un tel bloc pourrait contrôler le Parlement européen pour la première fois . -Si , à l' âge de 18 ans , je pensais que la gouvernance pouvait être en contradiction avec les campagnes politiques qui la précèdent . Si , à 18 ans , j' étais supposé comprendre que la gouvernance pouvait contredire les campagnes politiques qui la précèdent . -Si la personne la plus puissante de l' Europe peut être ciblée , alors les dirigeants d' entreprises sont également des cibles potentielles . Si la personne la plus puissante d' Europe peut être visée , alors les dirigeants d' entreprise sont sûrement aussi des cibles potentielles . -Si Goodyear avait offert aux employés une bonne prime de licenciement après avoir annoncé que l' usine était fermée , je crois que 100 % des employés l' auraient acceptée . Si , depuis l' annonce de la fermeture de l' usine , Goodyear avait offert aux salariés une bonne indemnité de départ , je pense que 100 % des employés l' auraient acceptée . -Si M. a témoigné en Allemagne , il aurait besoin d' être assuré qu' il était " sûr " là-bas , a déclaré le député . Si M. témoignait en Allemagne , il faudrait lui donner l' assurance qu' il serait en « sécurité » là-bas , a déclaré le député . -Si M. obtient la compétence de la CGT et de Goodyear pour parvenir à un accord et que Titan achète l' usine , nous avons toute intention de rester à Amiens Nord depuis plus de quatre ans . Si M. parvient à ce que la CGT et Goodyear se mettent d' accord et que Titan achète l' usine , nous avons bien l' intention de rester à plus de quatre ans . -Si tous les employés le acceptent , le projet d' acheter Goodyear Amiens commencera par n' importe quel employé . Si tous les salariés les acceptent , le projet d' achat de l' usine démarre avec zéro employé . -Si elle est concluante , les lecteurs biométriques pourraient bientôt être vus dans les boutiques en France . Si elle est concluante , les lecteurs biométriques pourraient bientôt arriver chez les commerçants français . -Si elle le faisait , elle son désaccord évident , a souligné Bruno , son porte-parole . Si c' était le cas , elle marquerait son désaccord net , souligne Bruno , son porte-parole . -Si l' on voulait une industrie européenne des technologies de l' information , alors on ne devrait pas lui laisser mourir il y a 20 ans . Si on voulait une industrie européenne des TI forte , alors il ne fallait pas la laisser mourir il y a 20 ans . -Si quelqu' un vous a vraiment aimés , leur absence ne peut pas votre cœur ou vos souvenirs . Si un être vous a aimé très fort , l' absence ne peut vous l' arracher du cœur et des souvenirs . -Si le critère de " l' authenticité " était adopté , un candidat à la naturalisation pourrait être interrogé sur la question de savoir si la nomination serait réelle dans les cas où le candidat serait une relation ou un associé personnel d' un propriétaire ou d' une personne intéressée de l' entreprise de parrainage . Si le critère « d' authenticité » était adopté , un demandeur de visa pourrait être contrôlé pour savoir « si l' offre d' emploi est authentique dans le cas où le candidat est une relation ou un associé personnel du propriétaire ou de la personne concernée au sein de l' entreprise qui le » . -Si les directives actuelles sont respectées , elles représenteront une croissance de 12 à 17 % du résultat de millions de dollars de bénéfices atteint en 2011 / 12 . Si les prévisions actuelles sont atteintes , cela représentera 12 à 17 % de croissance par rapport au bénéfice record de M $ atteint en 2011 / 12 . -Si le propriétaire paie 5,000 $ dans les impôts municipaux , il divise cette somme entre chaque maison . Quand le propriétaire paie 5000 $ en taxes municipales , il divise ce montant entre chaque logement . -S' ils ne satisfont pas aux critères de déploiement outre-mer , ils peuvent être contraints de quitter l' armée . S' ils ne répondent pas aux critères pour les déploiements à l' étranger , ils peuvent être forcés de quitter l' armée . -Si nous le faisons , des centaines de millions de conducteurs seront préoccupés par leur vie privée et par une foule d' autres choses . Si on fait ça , des centaines de millions d' automobilistes s' de la protection de leur vie privée et d' un tas d' autres choses . -Si vous êtes à la recherche d' un crédit supplémentaire , appelez Tim et Susan à Napa , en Californie , et essayez de leur de maïs ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! Si vous cherchez à vous faire remarquer davantage , appelez Tim et Susan à Napa , en Californie , et essayez de sculpter leur citrouille de 922 kg . -Si vous pouvez le faire , dit Khan , le public se sent plus à l' aise . Si on peut faire ça , a ajouté Khan , les citoyens seront plus à l' aise . -Si vous restez calme , votre enfant répondra généralement bien . Si vous restez calme , votre enfant réagira bien en général . -L' Illinois s' y emploie de façon limitée avec des camions . L' Illinois tente un essai sur un nombre limité de camions . -Le ministre de l' Immigration , Brendan O' Connor , hier à Sri Lanka , où il a rencontré des responsables de la traite des personnes , a mis en oeuvre cinq des modifications recommandées avec le reste de l' examen . Le ministre de l' Immigration Brendan O' Connor , qui était au Sri Lanka pour rencontrer des hauts responsables afin de discuter du trafic d' êtres humains , a mis en œuvre cinq des changements recommandés , les autres étant en cours d' examen . -Amélioration de la situation Situation améliorée -Dans " Les de Benghazi " , il a été le premier à révéler l' existence d' un centre secret de la CIA dans la ville , berceau de la révolte libyenne . Dans " Les de Benghazi " , il avait été le premier à révéler l' existence d' un centre de commandement secret de la CIA dans cette ville , berceau de la révolte libyenne . -En 1980 , il a décrit l' assassinat du président égyptien Anouar dans " Le du " un an avant l' attaque . En 1980 , il mettait en scène l' assassinat du président égyptien Anouar dans " Le complot du Caire " , un an avant l' attentat . -En 2009 , les baux pour les entreprises de la Quay , qui ont gagné le environ 2,5 millions de dollars par année , ont été renouvelés sans être soumis à un appel public . En 2009 , les baux concernant les entreprises de Quay , qui rapportaient aux Obeid environ 2,5 M $ par an , ont été renouvelés sans qu ’ il n' y ait d' appel d' offres . -En 2010 , les autorités ont trouvé un passage à deux voies , équipé de voies ferrées qui s' étendait de la cuisine d' une maison de Tijuana à deux entrepôts de San Diego . En 2010 , les autorités ont trouvé un passage d' environ 700 équipé de voies ferrées qui partait de la cuisine d' une maison de Tijuana et allait jusqu ’ à deux entrepôts de San Diego . -En 2011 , les responsables de la compagnie ont commandé un avion de ligne rempli de leur et l' ont pour vérifier les problèmes , mais n' en ont trouvé aucun . En 2011 , les dirigeants de la société ont chargé un avion de ligne de lecteurs de livres électroniques et l' ont fait voler pour voir s' il y avait des problèmes , mais il n' y en a eu aucun . -En 2011 , des dizaines de législateurs français issus de ce pays fortement catholique ont signé une pétition pour que la " théorie des genres " soit retirée des manuels scolaires . En 2011 , des douzaines de législateurs de ce pays foncièrement catholique ont signé une pétition pour que la « théorie des genres » soit retirée des livres scolaires . -En 2011 , Google a réservé une première tranche de 900 millions de dollars pour les brevets de Nortel . En 2011 , Google a fait une offre initiale de 900 M $ pour les brevets de Nortel . -En 2012 , a acheté un câble aux États-Unis , Wide Western Pacific , pour million CAD . a acquis en 2012 le Atlantic Broadband , établi aux États-Unis , pour milliard . -En 2012 , l' Agence des services frontaliers du Québec a procédé à saisies de stupéfiants . En 2012 , les agents des services frontaliers de la région du Québec ont effectué un total de 1653 saisies de stupéfiants . -À , un contact local qui avait déclaré qu' il des combats depuis le jeudi , a déclaré que la situation était " calme depuis ce matin " . A , un habitant qui avait fait état de combats tout proches jeudi pendant toute la journée a indiqué que la situation était " calme depuis ce matin " . -En août , la ville de New York a accepté de mettre un terme à la pratique consistant à stocker les noms et adresses des personnes dont les plaintes sont après un arrêt de la police . En août , la ville de New York a accepté de mettre fin à la pratique consistant à conserver les noms et adresses des personnes dont le cas est rejeté après un contrôle de police . -En Australie , les entreprises de loisirs ont rebondi au cours de la seconde moitié , ce qui a compensé un marché intérieur de la petite entreprise légèrement plus intense . En Australie , le secteur des voyages de loisirs a rebondi pendant le second semestre , ce qui a permis de compenser un marché domestique des voyages d' affaires en légère baisse . -Au Cameroun , il n' y a qu' un seul médecin pour 5 000 habitants , selon l' Organisation mondiale de la santé . Au Cameroun , il y a seulement 1 médecin pour 5 000 habitants , selon l' Organisation mondiale de la santé . -Au Canada , le maire de Toronto est soupçonné d' avoir recours à la drogue , selon plusieurs médias . Au Canada , le maire de Toronto est soupçonné de se , selon plusieurs médias . -En février , le New York Times l' a surnommé " le romancier espion qui connaissait trop " . En février , le New York Times l' avait consacré comme " l' auteur de romans d' espionnage qui en savait trop " . -Rien qu' en France seulement , en 2011 , la Commission Nationale de la Technologie de l' Information et des Libertés Publiques a approuvé 774 modèles de reconnaissance , pour la reconnaissance des empreintes digitales , la forme de la main ou des réseaux de veines dans la réalité , pour les entreprises , les institutions , les cafétérias , etc. Rien qu' en 2011 et en France , la Commission nationale de l' informatique et des libertés a autorisé 774 systèmes de reconnaissance des empreintes , de la forme de la main ou du réseau veineux de la main dans des entreprises , des administrations , des cantines ... -En France aujourd' hui , les données biométriques les plus utilisées sont les empreintes digitales , la géométrie des mains et le réseau de veines dans la paume des doigts ou les doigts . Aujourd' hui , en France , les données biométriques les plus utilisées sont les empreintes digitales , le contour de la main et le réseau veineux de la paume ou des doigts . -En Allemagne , beaucoup de ces entreprises sont des leaders mondiaux du marché dans leur créneau particulier . En Allemagne , bon nombre de PME sont des leaders mondiaux dans leur niche particulière . -En Inde , en Chine et dans bien d' autres pays , les gens travaillent entre 10 et 12 heures par jour . En Inde , en Chine et dans plein d' autres pays , on travaille dix à douze heures par jour . -Au Japon , il est déjà possible de retirer de l' argent de certaines machines automatiques en insérant votre carte et en plaçant votre lecteur sur un lecteur biométrique : ce geste remplace l' utilisation d' un NIP . Au Japon , il est déjà possible de retirer de l' argent à certains distributeurs en introduisant sa carte et en posant sa main sur un lecteur biométrique : ce geste remplace la frappe du code secret . -En juin , Rio a suspendu l' utilisation de son chemin de fer pendant environ une semaine après que ait menacé d' attaquer la ligne . En juin , Rio n' a plus pu utiliser le chemin de fer pendant environ une semaine après que la a menacé d' attaquer la ligne ferroviaire . -En juin , un Australien âgé de 52 ans , , est devenu la première " " reconnue à l' échelle mondiale après avoir reçu un appel à la justice pour avoir un statut " non spécifié " pour la vie . En juin , une de 52 ans , , est devenue la première personne au monde à avoir été reconnue « neutre » après avoir gagné son procès en appel qui l' autorise à garder un sexe « non spécifié » à vie . -En juin , le gouvernement a révisé le coût estimatif de la construction de la liaison à grande vitesse entre Londres et le nord de l' Angleterre , de à livres . En juin , le gouvernement a revu à la hausse le coût estimatif de la construction de la liaison à grande vitesse entre Londres et le nord de l' Angleterre , passant de 32,7 Md £ à 42,6 Md £ . -En mai , ils ont voté pour la levée des sanctions imposées par la Commission nationale des salaires et rétabli leurs salaires les plus élevés d' environ livres par mois à l' abri de l' impôt , classés parmi les plus élevés du monde . En mai , ils ont pris position contre les coupes ordonnées par la Commission nationale en charge des salaires et ont rétabli leurs salaires d' environ 532 000 shillings par mois exonérés d' impôt – qui figurent parmi les plus élevés au monde . -Dans l' équipe de Nathalie , Roger Vincent , un partisan de l' UMP à Paris et du quatrième arrondissement du Bundestag , dit que " même si l' UMP à Paris soutient la poursuite des services d' urgence à l' , il est techniquement et financièrement impossible de les rouvrir si l' on considère les devoirs " . Dans l' équipe de Nathalie , Vincent Roger , conseiller UMP de Paris et élu du 4e arrondissement , annonce clairement que " même si l' UMP parisienne était pour le maintien des urgences à l' , ce serait techniquement et financièrement impossible de les rouvrir si on revient aux responsabilités " . -Au Nevada , où environ 50 voitures bénévoles étaient équipées de ces appareils , les conducteurs n' étaient pas au courant du fait que le gouvernement était en mesure de surveiller chaque mouvement . Dans le Nevada , où environ 50 automobilistes volontaires ont récemment équipé leur voiture d' un appareil , les conducteurs étaient inquiets que le gouvernement puisse suivre leurs moindres faits et gestes . -À New York , des milliers de personnes en costumes ont participé à la parade organisée dans le district de Greenwich Village . A New York , des milliers de personnes ont participé à la parade organisée dans le quartier de " Greenwich village " . -À New York , les représentants des services de transport chercheront à mettre au point un système d' ATS qui serait également utilisé pour payer les frais de stationnement , fournir des assurances " " et créer un pool de données en temps réel sur les coûts des autres conducteurs que les automobilistes pourraient utiliser pour éviter le trafic . Dans la ville de New York , les responsables des transports cherchent à développer un dispositif de taxation qui serait également équipé pour payer les frais de stationnement , fournir une assurance « paiement à la conduite » et créer un ensemble de données sur la vitesse en temps réel en provenance des autres conducteurs que les automobilistes pourraient utiliser pour éviter les embouteillages . -À , en Arizona , les contrebandiers s' approvisionnent dans de vastes réseaux souterrains de drainage . À , en Arizona , les trafiquants exploitent de vastes canaux de drainage souterrains . -En octobre , GM a vendu des camionnettes et Sierra . En octobre , GM a vendu 42 660 et 16 503 Sierra . -En octobre 2012 , dans " à Bamako " , il a mentionné des colonnes de lanceurs de vers la capitale malienne . En octobre 2012 , dans " à Bamako " , il les colonnes de 4X4 de qui fondaient sur la capitale malienne . -En Oregon , les planificateurs essaient de donner aux conducteurs des choix différents . Dans l' Oregon , les planificateurs tentent l' expérience en offrant aux automobilistes différents choix . -A Paris , où il y a huit comédies musicales en saison , c' est une bonne année . À Paris , quand il y a huit spectacles musicaux dans une saison , c' est une grande année . -En septembre , il a commencé à donner la moitié de plus de milles miles parcourus à des clients qui ont acheté par l' intermédiaire d' une agence de voyages en ligne . En septembre , elle a commencé à n' offrir que la moitié de miles de fidélité aux clients qui achetaient leur billet via une agence de voyage en ligne . -À , malgré la fermeture de toutes les grandes entreprises , il n' y a aucune gêne pour les dépenses extravagantes et le maintien des éléphants blancs , tels que le centre culturel fragile , les subventions énormes pour la Cité de l' Énergie , etc. À , malgré la fermeture de toutes les entreprises majeures , on ne se gêne pas quant aux dépenses et à l' entretien des canards boiteux : centre culturel déficitaire , forte subvention à la Cité de l' Énergie , etc. -À Singapour , environ 70 % des points aux 200 points d' du forum sont de l' or . À Singapour , environ 70 % des articles mis en gage dans les 200 boutiques de prêt sur gage de la sont en or . -En Thaïlande , le plus gros opérateur de , , a enregistré une augmentation de 20 % du nombre de clients utilisant ses points de vente au cours des derniers mois . En Thaïlande , le plus grand opérateur de boutiques de prêt sur gage , , a enregistré une hausse de 20 % du nombre de clients utilisant ses boutiques au cours des derniers mois . -Lors d' une grande victoire de l' administration Obama , la Cour suprême des Etats-Unis a décidé aujourd' hui que le gouvernement de Taiwan avait signé la loi sur les soins de santé . Grande victoire pour l' administration Obama : la Cour suprême américaine a déclaré aujourd' hui que la loi de réforme du système de santé ratifiée par Obama était constitutionnelle . -Dans une note aux clients de la banque , il précise que ce léger ralentissement peut s' expliquer principalement par " l' arrêt du gouvernement et son impact sur les sentiments des consommateurs au cours de la première moitié du mois " . Dans une note aux clients de la banque , il estime que ce léger coup de frein s' explique en premier lieu par " le ' ' gouvernemental et son impact sur le sentiment du consommateur au cours de la première quinzaine du mois " . -En un mot , il est facile de faire de grandes choses avec l' argent des autres . En résumé , il est bien facile de réaliser de grandes choses avec l' argent des autres . -Lors d' un entretien avec la police , il a déclaré qu' il avait un bureau à son domicile , ainsi que son lieu de travail et que les clients à sa maison sur des affaires juridiques . Lors d' un interrogatoire , il a déclaré qu' il possédait un bureau à son domicile , de même qu' à son lieu de travail et qu' il arrivait que des clients l' appellent chez lui pour des questions d' ordre juridique . -Lors d' une conférence de presse qui a eu lieu jeudi , M. Blair a déclaré qu' il n' y avait rien dans cette vidéo qui puisse constituer une " motivation raisonnable " susceptible d' entraîner des poursuites pénales contre le maire . En conférence de presse , jeudi , M. Blair a affirmé qu' il n' y avait rien dans cette vidéo qui puisse constituer des " motifs raisonnables " pouvant mener au dépôt d' une accusation criminelle contre le maire . -Dans un communiqué de presse , le club a déclaré qu' il était " très déçu " et que l' appel était tout à fait justifié . Dans un communiqué , son club s' est dit " très déçu " , estimant que son appel était entièrement justifié . -Au cours d' un déménagement séparé , , le régulateur de l' examen , la structure du , avec un nouveau système de notation et moins de cours . Par ailleurs , , le Bureau régulateur des examens , va dévoiler un remaniement de la structure des , avec un nouveau système de notation et moins de cours . -Dans une déclaration , un porte-parole du Hamas , Sami Abou , a rendu hommage aux quatre " héros " et a déclaré que des soldats israéliens étaient morts au cours de l' affrontement . Dans un communiqué , un porte-parole du Hamas , Sami Abou , a rendu hommage aux quatre " héros " et affirmé que des soldats israéliens avaient trouvé la mort lors de la confrontation . -Dans une déclaration , des responsables de l' Église ont déclaré que M. avait démissionné de son poste . Dans un communiqué , les responsables de l' église indiquent que M. a démissionné de son poste . -Dans une déclaration faite jeudi , un représentant de , 70 ans , affirme qu' il " nie catégoriquement qu' il n' ait jamais fait un passage à Perry " . Dans une déclaration jeudi , un représentant de , 70 ans , a déclaré qu' il « nie catégoriquement avoir fait des avances à Perry » . -Dans une déclaration annonçant les nouvelles , le père Federico , porte-parole du Vatican , a déclaré qu' une réunion de tous les Cardinaux inscrits serait organisée avant la cérémonie pour élever les nouveaux cardinaux , connus sous le nom de . Dans une déclaration annonçant la nouvelle , le père Federico , un porte-parole du Vatican , a dit qu' une réunion rassemblant tous les cardinaux existants se tiendrait avant la cérémonie au cours de laquelle des évêques seront élevés à la dignité de cardinal , connue sous le nom de . -Dans une déclaration sur son site Web officiel , les singes de l' Arctique ont déclaré : " Compte tenu de la décision d' ajourner le bureau du LG Arena à partir de ce soir et après avoir consulté les spécialistes de la nature , les singes de l' Arctique doivent aussi reporter le spectacle à la Glasgow Hydro , le vendredi 1er novembre . Dans une déclaration sur leur site officiel , les Arctic ont déclaré : « Suite à la décision de reporter le concert à la LG Arena de Birmingham ce soir et après avoir demandé un avis médical , les Arctic doivent également reporter le spectacle à l' Hydro de Glasgow le vendredi 1er novembre . » -D' une certaine façon , la personne que vous aimez est votre énergie intérieure . D' une certaine façon , l' être aimé devient votre énergie intérieure . -Conformément à la loi , la police , qui a elle-même été victime de transactions suspectes , aurait pu intercepter le véhicule de Ford , arrêter le maire et procéder à une fouille - même sans mandat . En vertu de la loi , la police , ayant été témoin de transactions suspectes en mains propres , aurait très bien pu intercepter le véhicule de Rob Ford , arrêter le maire , et effectuer une perquisition - même sans mandat . -De plus , les efforts déployés par le ministère pour maintenir le personnel ont été payés , puisque les quatre étudiants qui se sont déjà installés à Laval ont décidé de rester au . De plus , les efforts déployés par le département afin de retenir ont porté des fruits , car les quatre élèves présentement en résidence à Laval ont également décidé de rester à l' emploi du . -En plus de tenir ces cliniques de fin de semaine et de travailler comme chirurgien de l' hôpital , travaille aussi des nuits dans des cliniques médicales privées autour de Yaoundé . En plus de faire vivre cet hôpital itinérant et de travailler comme chirurgien à l' hôpital , le Dr travaille aussi la nuit dans des cabinets médicaux privés autour de Yaoundé . -En plus de ces événements , il y a eu le , le concert tenu en août dernier , avec le concours de la soprano , . À ces activités s' est ajouté l' lyrique , le tenu en août dernier , avec le concours de la soprano . -En tout , près de 22 000 femmes seront privées d' accès à ces établissements . Au total , ce serait près de 22.000 femmes qui seraient privées d' un accès à ces établissements . -Dans toute la documentation par M. , cependant , il n' y a jamais eu de preuves jusqu' à ce jour que les États-Unis aient transmis les secrets commerciaux des entreprises étrangères à leurs propres sociétés . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Cependant dans tous les documents dévoilés par M. , rien n' indique à ce jour que les États-Unis aient transmis des secrets industriels d' entreprises étrangères à des sociétés américaines . -Dans une entrevue qui a eu lieu la veille pour marquer le cinquième anniversaire de son départ , le a fait part de ses vues sur diverses politiques nationales . Dans un entretien la nuit dernière marquant le 5e anniversaire de son départ à la fin de son mandat , l' homme de 59 ans a fait connaître son opinion sur diverses politiques nationales . -Lors d' une entrevue avec Martin , a confirmé que l' administration avait en fait décidé de remplacer les avec Hillary Clinton . Dans un entretien avec Martin , a confirmé que l' administration avait effectivement envisagé de remplacer Biden par Clinton . -Dans une interview avec le quotidien La , le chef du centre a déclaré qu' il avait " l' intention " de continuer à gouverner avec sa coalition jusqu' aux prochaines élections législatives de 2015 . Dans une interview au quotidien La , le dirigeant de centre gauche dit avoir " fermement l' intention " de continuer à gouverner avec sa coalition jusqu' aux prochaines élections législatives de 2015 . -En tout cas , Bowie aime jouer le dans ces : à son tour , un , un , peut-être un ... Bowie s' amuse en tout cas à jouer au avec ces titres : tour à tour informateur , homme à bord d' une fusée , potentiel ... -En décrivant les méthodes américaines d' interception des données en rapport avec la représentation diplomatique française aux États-Unis , nous ne voulons pas exprimer l' indignation à propos de cette pratique , mais plutôt décrire le monde tel qu' il est . Lorsque nous décrivons les systèmes d' interception américains à l' encontre de la diplomatie française aux Etats-Unis , ce n' est en aucun cas pour nous de cette pratique , c' est pour décrire le monde tel qu' il est . -En mourant , cette femme avait achevé sa vie et libéré la personne que j' aimais , et elle est maintenant finie , complète et accomplie . En mourant , cette femme avait parfait sa vie et libéré la personne que j' aimais , et voici que je la , pleine et entière . -Dans tous les cas , à chaque instant de ma vie , je sens leur présence , et je me réfère toujours à eux : qu' est-ce qu' ils diraient , ce qu' ils pensent , que ? Chaque événement , chaque instant de ma vie me fait ressentir leur présence car je m' y réfère toujours : qu' auraient-ils dit ? qu' auraient-ils pensé ? qu' ils fait ? -En fait , les Américains dépensent 330 millions de dollars sur les costumes d' artisanat cette année , selon la . En fait , les Américains vont dépenser 330 M $ en pour animaux cette année , selon la NRF . -En fait , il y a eu 1.200 maisons aux États-Unis , qui ont fait des tirages gratuits de 500 millions de dollars US , ce qui est vrai en Amérique latine , ce qui inclut les photos que vous de ce petit ami que vous sur votre ordinateur et que vous ne pouvez pas tomber là , alors que le gars que vous voyez peut voir la photo et laisser un commentaire comme un " beau visage " . , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ... En fait , il existe 1 200 maisons reconnues officiellement aux États-Unis , qui génèrent environ 500 M $ de recettes , selon l' America , et cela inclut ces formidables photos de vous en train de vous dessus que vos amis mettent sur Facebook et que vous ne pouvez pas enlever et sur lesquelles un type que vous aimez bien laisse un commentaire du style « jolie tête » . -En fait , nous achetons et , qui nous , 90 millions de livres de chocolat pendant l' Halloween . En fait , nous allons acheter et , ne nous pas d' histoires , consommer 41 millions de kg de chocolat pendant Halloween . -Dans son livre , La Mort en , le cher Yvon Le philosophe et du Comité National d' du met l' accent sur la " violence " que représente la crémation pour ceux qui sont à l' arrière . Dans son livre La Mort en cendres , Damien Le Guay , philosophe et vice-président du Comité national d' éthique du funéraire , insiste sur la " violence " que constitue la crémation , pour les survivants . -Dans son discours , et en présence de son homologue britannique , William Hague , le chef de la diplomatie américaine a insisté à plusieurs reprises sur les pratiques de renseignement et la collecte de données dans le cadre de la nécessaire lutte contre le terrorisme et la prévention des attaques éventuelles . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; » Dans son intervention retransmise , en présence de son homologue britannique William Hague , le chef de la diplomatie américaine a longuement justifié les pratiques de renseignements et de collecte d' informations par la nécessaire lutte antiterroriste et la prévention contre d' éventuels attentats . -Dans celui-ci , il présente chacun des disques et les accessoires fournis avec eux : des photos exclusives et des , un cahier pour partager vos impressions , un livret de paroles , etc. Il y introduit à la fois chacun des disques , mais aussi les accessoires fournis avec : des photos et pochettes exclusives , un carnet pour partager ses propres impressions , un livret avec les paroles ... -À la suite de la rébellion qui a pris plusieurs mois à l' égard d' un certain nombre de membres du groupe et de plusieurs syndicats , dont la CGT , le ministre de la Santé , Touraine , a décidé le 10 juillet de " remettre le calendrier de la mise en oeuvre du projet et , en particulier , la fermeture de l' unité d' urgence qui n' est pas prévue pour le 4 novembre " . Face à la menée depuis plusieurs mois par une partie des personnels et plusieurs syndicats , dont la CGT , la ministre de la santé , Touraine , avait décidé le 10 juillet de " le calendrier de mise en oeuvre du projet , et en particulier la date de fermeture des urgences qui ne pourra intervenir le 4 novembre " . -Dans la plupart des cas , le processus ralentit ou - quelque chose qu' un pédiatre peut surveiller de près . Dans la plupart des cas , le développement ralentit ou s' arrête – c' est quelque chose qu' un pédiatre peut suivre de près . -Selon moi , il y a deux niveaux de réponse du gouvernement français . Selon moi , il y a deux niveaux de réponse de la part du gouvernement français . -Dans la France voisine , les questions liées à l' égalité des genres sont encore controversées , selon un rapport d' information sur la France 24 . En France voisine , les questions de genre font toujours l' objet de controverses , selon un reportage diffusé sur France 24 . -Dans d' autres , le chef de l' a déclaré qu' il y a eu une forte explosion mercredi à l' intérieur d' une base de défense aérienne dans la province côtière de de . D' autre part , le directeur de l' Observatoire syrien des droits de l' Homme a déclaré qu' il y avait eu une forte explosion mercredi dans une base de défense aérienne dans la province côtière syrienne de . -Dans de rares cas de visibilité réduite , l' équipage ordonne aux passagers d' éteindre leurs appareils pendant l' atterrissage . Dans les rares cas de faible visibilité , l' équipage demandera aux passagers d' éteindre leurs appareils pendant l' atterrissage . -En bref , il est là tous les jours . Bref , il est là au quotidien . -En bref , le illimité de l' argent de nos poches . Bref , sans limite de nos poches . -Malgré tout , j' ai dit mon assez rapidement . J' ai malgré tout réalisé mon deuil assez rapidement . -En ce qui a trait aux résultats , a indiqué qu' elle a enregistré un bénéfice net de 43,8 millions de au quatrième trimestre , soit 82 ¢ par action . Du côté des résultats , a indiqué avoir enregistré un bénéfice net de 43,8 millions au quatrième trimestre , ou 82 ¢ par action . -En matière de revenus , a connu une croissance de 41,5 % au quatrième trimestre , atteignant millions de dollars . Quant aux revenus de , ils ont connu une croissance de 41,5 % au quatrième trimestre pour atteindre millions . -Lors de tests , les souris vaccinées , qui ont ensuite reçu la nicotine , ont poursuivi leur activité normale . Lors des tests , les souris vaccinées à qui on a donné ensuite de la nicotine ont poursuivi leur activité normalement . -Dans les années 1980 , Jordan se lance dans une arnaque qui lui rendra un homme très riche . Dans les années 80 , Jordan Belfort monte alors une arnaque qui fera de lui un homme riche , très riche . -Au Royaume-Uni , les évêques ont publié un sondage sur Internet et demandé aux catholiques de participer . Au Royaume-Uni , des évêques ont publié le sondage sur Internet en demandant aux catholiques de participer . -Aux États-Unis , l' âge moyen est de 12,5 ans , contre en 1970 . Aux États-Unis , l' âge moyen est de 12 ans et demi , alors qu' il était de 12 ans trois quarts en 1970 . -Aux États-Unis , le berceau de la musique , c' est très différent . Aux États-Unis , berceau de la comédie musicale , c' est très différent . -Aux États-Unis , le service vidéo populaire parle de la possibilité de faire de son service par l' intermédiaire de ses plateformes de diffusion avec certaines grandes entreprises de câblodistribution . Aux États-Unis , le populaire service de vidéo dit être en discussions avec certains câblodistributeurs majeurs afin que son service soit disponible via leurs plateformes de diffusion . -Au début du XIXe siècle , animaux classés selon les différences morphologiques entre les espèces . Au début du 19ème siècle , les ont classé les animaux , sur base des différences morphologiques entre les espèces . -En fin de compte , il s' est senti à la fois confortable et rassurant d' être compris , encouragé , conseillé . Finalement , c' était confortable et rassurant d' être comprise , encouragée , conseillée . -Le soir , l' équipe effectuera des chirurgies simples avec l' anesthésie locale . Le soir , l' équipe pratique des actes chirurgicaux simples sous anesthésie locale . -Dans la première action , la Congrégation de la Sainte Croix a accepté de se mettre d' accord sur la question avant de procéder à des auditions préliminaires . Dans le cas du premier recours , la congrégation de Sainte-Croix avait accepté de régler à l' amiable avant que des auditions soient menées sur le fond . -Au cœur du cimetière de , des civils , des religieux et des militaires , ainsi que de nombreuses associations patriotiques , ont rendu hommage à tous les soldats et victimes de toutes les guerres le vendredi après-midi . Au cœur du cimetière du , autorités civiles , religieuses et militaires , ainsi que de nombreuses associations patriotiques , ont rendu , ce vendredi après-midi , un hommage aux soldats et défunts de toutes les guerres . -Dans sa lettre , M. a déclaré que le soutien de la communauté internationale pourrait convaincre le gouvernement américain d' abandonner les poursuites pénales contre lui . Dans sa lettre , M. a écrit qu' il pensait que le soutien de la communauté internationale pourrait persuader le gouvernement américain d' abandonner les charges pénales retenues contre lui . -Dans le local de son petit appartement dans l' immeuble situé sur l' avenue , à quelques pas de l' avenue Leon , avec son chat au ventre et son à dos de poitrine dans les cheveux blancs , il a décrit certaines des langues que les aventures de la SAS ont été enregistrées : italien , allemand , russe , grec , japonais et coréen . , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , etc. Son chat birman sur les genoux , dans le salon de son immense appartement d' un immeuble de l' avenue Foch à deux pas de l' Arc de Triomphe , l' oeil malicieux sous ses cheveux blancs , il citait quelques unes des langues dans lesquelles les aventures de SAS ont été traduites : italien , allemand , russe , grec , japonais ou coréen . -Dans le passé , les espèces ont été décrites et identifiées sur la base de la morphologie , de l' anatomie et de la géographie . Aujourd ' hui , le développement de nouveaux outils comme la génétique permet d' améliorer notre connaissance de la science de la classification des espèces . Si par le passé , les espèces ont été décrites et identifiées sur base morphologique , anatomique et géographique , aujourd' hui , le développement de nouveaux outils tels que la génétique nous permettent d' améliorer nos connaissances en sciences de la classification des espèces . -Dans le passé , le Pape décidait tout seul . Auparavant , le pape décidait tout seul . -Dans ces villages , les soins médicaux gratuits sont vraiment une cause de célébration , et - avec son grand sourire et son énergie infinie - est plus que heureux de se joindre à l' amusement . Dans ces villages , les soins de santé gratuits sont une bonne raison de faire la fête , et le Dr – avec son grand sourire et son énergie débordante – est plus qu' heureux de se joindre aux festivités . -Dans ce contexte , l' euro continue de baisser par rapport au dollar américain et , au cours de la session , a atteint un niveau record de 36 semaines , à dollars . Dans ce contexte , l' euro poursuit son repli contre la monnaie américaine et a touché en séance un plus bas de deux semaines à dollar . -Dans ce pays , les filles d' origine afro-américaine et hispanique tendent à avoir la puberté plus tôt que leurs homologues blancs . Dans ce pays , les jeunes filles et hispaniques ont tendance à atteindre l' âge de la puberté plus tôt que les jeunes filles blanches . -À ce jour , l' âge avec un crayon semble extraordinaire . De nos jours , avoir un crayon de papier semble extraordinaire . -Dans ce film à l' humour , où la musique est un personnage essentiel , le théâtre est rempli d' émissions , et l' écrivain Isaac s' est révélé être un musicien populaire et chanteur prolifique , tandis que Jill , son ami dans le cinéma , s' est mis à abandonner la pop music à la joie . Dans ce film bourré d' humour où la musique est un personnage central et les chansons interprétées en direct , Oscar Isaac se révèle un musicien et chanteur de folk accompli , tandis que Timberlake , son ami dans le film , quitte la pop pour la musique folk avec une contagieuse . -Dans cette performance , la musique est forte , accompagnée de percussions et de guitares électriques . Ici , la musique est forte , percussions et guitares électriques . -À cet égard , le Comité du renseignement du Sénat a approuvé le renforcement des contrôles des programmes de surveillance du Conseil jeudi , mais les a autorisés à poursuivre . Dans ce contexte , la Commission sénatoriale du renseignement a approuvé jeudi un renforcement des contrôles sur les programmes gouvernementaux de surveillance mais a autorisé leur poursuite . -Compte tenu de l' urgence de la situation , le Directeur du Bureau du Sud de la terre rurale a offert d' offrir un logement provisoire exceptionnel au couple et à ses trois enfants dans une nouvelle maison de . Devant l' urgence de la situation , le directeur de l' office Terres du Sud Habitat , a proposé , de manière exceptionnelle et provisoire , de reloger le couple et ses trois enfants dans un T4 neuf . -En effet , c' est la demande dans certaines parties de l' Asie du Sud-Est - où la dette des ménages est en hausse - que , où elle effectue sa transaction , est devenue la troisième à figurer sur la Bourse de Singapour . En fait , la demande est telle dans certaines parties d' Asie du sud-est – où l' endettement des ménages augmente – que , où elle a effectué sa transaction , est devenu cette semaine la troisième société de prêts sur gage à être introduite à la bourse de Singapour . -Les producteurs français indépendants essaient de faire une percée . Les ­ producteurs français indépendants tentent une ­ percée . -L' Indiana , la Géorgie , la Floride , le Mississippi , la Louisiane , l' Oklahoma , la Caroline du Sud et la Virginie occidentale , suivis par le refus , selon un haut responsable de la Défense . Il a été rejoint dans son refus par l' Indiana , la Géorgie , la Floride , le Mississippi , la Louisiane , l' Oklahoma , la Caroline du Sud et la Virginie occidentale , selon un haut responsable de la Défense . -L' Indonésie a convoqué l' ambassadeur de l' Australie , dont la mission a été accusée d' être utilisée par les Américains dans le cadre d' un vaste réseau international d' espionnage , qui a également suscité la colère de la Chine . L' Indonésie a convoqué l' ambassadeur d' Australie , dont la mission est accusée d' être utilisée par les Américains dans le cadre d' un vaste réseau d' espionnage international qui a également suscité l' de la Chine . -Les investigations ont révélé que était un " utilisateur régulier " de la drogue de la classe A après des tests de traces de cocaïne dans ses cheveux , sur son porte-monnaie et sur ses cartes de crédit . Les enquêtes ont révélé que était un « consommateur régulier » de cette drogue de classe A après que des tests ont montré des traces de cocaïne dans ses cheveux , sur son portefeuille et ses cartes de crédit . -A l' intérieur de l' , Son Altesse le prince ( ) , un indépendant et anonyme de l' Autriche pour le financement de la CIA ( pour payer les factures de la maison de la famille ) , est très loin de tous les des communistes - dans les années 70 et 80 , les de la défense depuis les années 1990 & # 160 ; : , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ... A l' intérieur , son ( SAS ) le prince , autrichien et agent contractuel de la CIA pour payer les réparations du château de famille , se lance aux trousses de tous les méchants de la terre , communistes des années 70 et 80 , puis à partir des années 90 . -Au lieu de cela , a passé des années à escorter son père dans des dispensaires et des hôpitaux surpeuplés , obtenant ainsi tout ce qu' ils pouvaient lui offrir . Au lieu de cela , Dr a passé des années à accompagner son père dans des cliniques et des hôpitaux bondés , pour qu' il obtienne n' importe quel traitement dont il pouvait bénéficier . -Au lieu de cela , il croit que les chiens ont appris de l' expérience qu' ils devraient faire et ne devraient pas s' inquiéter . Il pense plutôt que les chiens savent par expérience les mouvements dont ils devraient ou ne devraient pas s' inquiéter . -Enquête sur le maire Rob Ford , juriste en détachement L' enquête sur le maire Rob Ford a été , soutient un avocat -Les enquêteurs ont également trouvé des entrées sur l' ordinateur de l' enquêteur privé qui avait choisi Mme et sa soeur comme " projets " . Les enquêteurs ont également trouvé des saisies sur l' ordinateur du détective privé qui concernaient Mlle et sa sœur rangées dans un dossier « Projets » . -L' élite européenne est-elle prête à faire des affaires avec la Grande-Bretagne ? L' élite européenne est-elle prête à faire des affaires avec la Grande-Bretagne ? -Est-ce qu' il est professeur dans des salles de cours et dans des salles de classe ? Professeur dans les et les salles de cours ? -Est-ce l' héritier de la chaire de à l' Académie Française ? du fauteuil qu' occupa à l' Académie française ? -Est-ce parce qu' ils peuvent le résultat si quelqu' un ne l' aime pas ? Est-ce pour pouvoir effacer le résultat si quelqu' un n' est pas d' accord ? -Est-ce la fin des mots de passe pour accéder à un smartphone ou payer pour les achats ? Fini les mots de passe à taper pour accéder à son smartphone ou payer ses achats ? -N' y a-t-il pas un élément d' hypocrisie de votre part ? N' y aurait-il pas comme une vague hypocrisie de votre part ? -Est-ce que cela veut dire remercier M. ? Est-ce pour faire plaisir à M. ? -Assassinat du Hamas tué par l' armée israélienne à Gaza L' armée israélienne tue un membre du Hamas à -Les avions israéliens sont entrés dans l' espace aérien libanais tôt le mercredi après-midi , mais n' ont pas fait d' attaques jusqu' au soir , selon l' armée libanaise . Selon l' armée libanaise , des avions israéliens sont entrés dans l' espace aérien libanais mercredi en début d' après-midi , mais les frappes ont eu lieu dans la soirée . -Attaque israélienne dans le nord de la Syrie Attaque israélienne dans le nord de la Syrie -Des responsables gouvernementaux israéliens ont refusé de confirmer toute information relative à l' attaque . Des responsables du gouvernement israélien ont eux refusé de confirmer toute information concernant une telle attaque . -Les responsables israéliens n' ont pas tenté de cacher leur colère lorsque Washington a révélé cette attaque , au risque de forcer le président Assad à réagir . Les responsables israéliens n' avaient pas caché leur irritation de voir Washington révéler l' attaque , au risque d' obliger le président Assad à réagir . -Au cours de la journée , des milliers de personnes ont été tuées par des actes de violence contre des civils et des personnes déplacées à l' intérieur de leur pays d' origine . Des tirs de chars israéliens ont tué jeudi un militant palestinien islamiste du Hamas et en a grièvement blessé un autre dans le sud de la bande de Gaza , a-t-on appris après de sources des services de santé . -L' objectif de l' attaque aérienne israélienne à l' intérieur de la Syrie est le suivant : Des avions de guerre israéliens ciblent la Syrie , déclare un responsable -Des avions de combat israéliens ont atteint la cible dans la ville portuaire de vendredi , un haut responsable de l' administration a confirmé à Fox News . Des avions de guerre israéliens ont atteint une cible dans la ville portuaire syrienne de jeudi soir , confirme un responsable militaire à Fox News . -C' est un rythme , mais les musiciens du village aident habituellement à maintenir l' équipe . C' est un rythme , mais les musiciens des villages aident en général l' équipe à rester motivée . -C' est un peu compliqué , un peu de choc , assez difficile à gérer . C' est un peu compliqué , un peu violent , assez complexe à gérer . -C' est un livre que je pensais me au milieu de la terminale de médecine de deuxième classe , et en moi un sentiment d' espérance . C' est un livre qui , je pensais , m' encouragerait pendant la morosité de la 2e année de médecine et me de l' espoir . -Il s' agit d' un retour en noir et blanc pour Marie Chouinard , qui forme Marie Danse , deux de ses nouvelles compositions , la première à partir des dessins d' encre et des poèmes d' Henri , le second inspiré par les solos de piano de de Satie . Retour en noir et blanc pour Marie Chouinard , qui présente à Danse Danse ses deux nouvelles chorégraphies de groupe : la première inspirée des encres et poèmes d' Henri , la deuxième par les fameux solos de piano de Satie . -C' est un mystère que je n' ai pas encore résolu . C' est un mystère que je ne suis pas encore arrivé à élucider . -C' est une révolution : la musique française se donne comme un genre de réussite . C' est une révolution : le musical à la française s' affirme comme un genre à succès . -Il s' agit des clients les plus fidèles de Frontier , qui savent très bien qu' il est devenu de plus en plus difficile de trouver de l' espace . Il s' agit de faire comprendre aux clients de Frontier les plus fidèles qu' il devient de plus en plus difficile de trouver de la place dans les compartiments supérieurs . -Il semble que la poursuite de la campagne d' Israël continue d' empêcher les armes de proliférer au Moyen-Orient . Ce serait apparemment la poursuite de la campagne israélienne visant à empêcher la prolifération des armes dans le Moyen-Orient . -C' est ce que l' on appelle " to Win " , et c' est un des souvenirs de la campagne présidentielle de Barack Obama en 2008 . Il s' appelle « The to Win » et il s ’ agit des mémoires de la campagne présidentielle de Barack Obama de 2008 . -Cinq ans plus tard , Monsieur le Président , je ne suis pas d' accord avec vous . Nous voici 5 ans après , M. le Président , et je ne suis plus d' accord avec vous . -C' est une bonne nouvelle , mais ça ne m' a pas surpris , surtout . C' est une bonne nouvelle , mais cela ne m' étonne pas particulièrement . -Il est difficile de se souvenir de ces jours d' optimisme -- ils semblent une mémoire lointaine , un triste rappel des occasions qui se sont écoulées . Il est difficile de se rappeler ces jours pleins d' optimisme , ce sont comme de lointains souvenirs , un triste rappel des opportunités du passé . -Il est difficile de comprendre ce revirement . On ne comprend pas cette volte-face . -C' est comme le mystère de la raison pour laquelle les gens sont homosexuels . C' est comme le mystère qui entoure les personnes homosexuelles . -Ce n' est pas un procès , c' est une farce " . Ce n' est pas un procès , c' est une farce . -Il s' agit maintenant d' enquêter sur les allégations de M. qui connaissaient l' intérêt secret d' Europol pour les biens , après les témoignages faits par l' ancien adjoint du chef de cabinet , M. Lynne , jeudi . Elle enquête désormais sur les allégations selon lesquelles M. était au courant de l' intérêt secret de M. Obeid dans ces propriétés , suite au témoignage de l' ancienne secrétaire générale adjointe de M. , Lynne , jeudi . -C' est détendu , c' est cool . C' est bon enfant , c' est cool . -C' est la première fois en 20 ans que la belle blonde a eu des cheveux courts , et nous aimons le changement . C' est la première fois en 20 ans que la jolie blonde se fait couper les cheveux , et on est fan du changement discret . -C' est la même chose avec mes opinions : je m' efforce de résister à la générosité et à la justesse de son esprit en tant que fervent partisan . Il en va de même pour mes attitudes : j' essaie d' être à la hauteur de la générosité et de la droiture d' esprit de celle qui était une fervente . -C' est ce deuxième travail , a-t-il dit , qui finance environ 60 % de son ; le reste est pris en charge par des dons privés . C' est ce deuxième emploi , explique-t-il , qui finance à 60 % son association ; le reste est couvert par des dons privés . -Il n' est cependant pas difficile de savoir si Obama savait que son équipe recherchait l' échange . Cependant , on ne sait pas si Obama était au courant que son équipe examinait le remplacement . -Elle a ajouté que " d' autres activités d' inspection ne sont pas prévues actuellement " . Elle a ajouté que « aucune autre activité d' inspection n' était prévue pour le moment » . -Il permet également aux passagers de choisir leur siège à l' avance seulement s' ils achètent directement sur le site de la Frontier Convention . Elle permet également aux passagers de choisir leur siège à l' avance seulement s' ils achètent directement leur billet sur le site de Frontier . -Elle recommande également aux écoles d' être - la marque la plus communément utilisée -- pour réagir à une réaction fatale . & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; Il recommande également aux écoles d' avoir un stock d' – l' de marque étant le plus couramment utilisé – pour pouvoir réagir rapidement en cas d' potentiellement mortelle . -Il a également dit que les mesures pourraient avoir un effet dévastateur sur ce qu' il décrit comme la " blogosphère animée " du pays . Il dit également que les mesures pourraient avoir un effet dévastateur sur ce qu' il a décrit comme la « blogosphère animée » du Kenya . -Il est également dit que la recherche montre que l' expérience de la cabine dépend plus que de la largeur d' un siège . Il dit également que l' étude montre que l' expérience en cabine ne dépend pas seulement de la largeur des sièges . -Il montre également que les entreprises s' attendent à ce que cette renégociation un changement important dans l' équilibre actuel du pouvoir vers le Royaume-Uni . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Elle montre également que les entreprises espèrent que la renégociation constituera un changement significatif dans l' équilibre actuel du pouvoir en faveur d' un retour aux mains du Royaume-Uni . -Selon les informations contenues dans le rapport britannique de Powell , " les services de la Force de sécurité fédérale en Irlande ( ) et le United States Command and Communications Bureau ( ) en Grande-Bretagne ont rassemblé des renseignements grâce à l' infiltration des réseaux internationaux , ce qui a permis aux deux entités de synchroniser leurs serveurs . Elle juge " scandaleuses " les informations du Washington Post , selon lequel l' Agence nationale de sécurité américaine ( NSA ) et le Government Communications Headquarters ( ) britannique ont collecté des masses d' informations en les réseaux internationaux qui permettent aux deux firmes de synchroniser leurs serveurs . -Elle ne peut justifier l' assassinat d' Anwar , un citoyen américain , sans procès , ni contre le financement public et les limites de dépenses au cours des campagnes présidentielles . Elle ne peut pas justifier l' assassinat d' Anwar , un citoyen américain , sans procès , ni l' évasion des financements publics et le dépassement des dépenses autorisées pendant les campagnes présidentielles . -Il ne peut pas expliquer que l' Agence nationale de sécurité recueille des données sur la vie privée des Américains , ni ne surveille les dénonciateurs d' abus . Elle ne peut pas expliquer que la National Security Agency recueille des données sur la vie privée des Américains , ni la mise en accusation des dénonciateurs qui révèlent les méfaits du gouvernement . -Elle ne peut justifier les conclusions d' un rapport qui dit que les efforts de la Maison Blanche pour couper les médias sont les " les plus agressifs ... depuis l' administration Nixon " . Elle ne peut pas justifier les conclusions d' un rapport qui dit que les efforts de la Maison-Blanche pour réduire les médias au silence sont les « plus agressives ... depuis l' administration de Nixon » . -Cela ne m' a pas dérangé . Ça ne m' a pas gêné . -Il permet aux enfants de prendre la décision pour eux-mêmes . Cela autorise l' enfant à prendre la décision pour lui-même . -Il leur donne le temps de procéder à des tests et de les situer et de passer un peu de temps avant qu' ils n' aient " masculins " ou " féminins " . Cela leur laisse le temps de faire certains tests et de les comprendre , et un délai avant d' inscrire « masculin » ou « féminin » . -Il a également ordonné au Groupe de travail de de procéder à une correction complète de New Express , et a recommandé d' enquêter sur les personnes responsables de New Deal et de faire une nouvelle mise à jour de l' équipe de direction de New Express . Il a également « ordonné au groupe News d' entreprendre un remaniement complet de New Express , et a recommandé qu' il mène une enquête sur les personnes responsables chez New Express et immédiatement l' équipe de direction du journal » . -Il avait quitté Toulon à la mi-octobre pour un stage de formation . Il avait de Toulon à la mi-octobre pour un entraînement . -Elle devait soit réinvestir dans le bâtiment pour continuer à l' utiliser , ce qui exigerait des investissements de plusieurs dizaines de millions de dollars , alors que la concurrence est féroce puisque de nombreux nouveaux hôtels sont nés à Montréal . Soit il dans la bâtisse pour continuer à l' opérer , donc il y avait des investissements de dizaines de millions alors que la compétition est très féroce , il y a beaucoup de nouveaux hôtels qui sont apparus à Montréal . -Elle a toujours eu lieu . Il y en a toujours eu . -Il est britannique depuis son occupation par les mutins de Bounty , un épisode de l' histoire maritime britannique qui a été transformé en trois films . Il est britannique depuis son occupation en 1790 par les du Bounty , épisode de l' histoire de la marine anglaise qui donna lieu à trois adaptations . -Il n' a pas mis à jour ce chiffre depuis . Elle n' a pas actualisé ce chiffre depuis . -C' est un privilège de savoir que les gens que nous aimons ne nous quittent jamais . C' est un privilège de savoir que les gens aimés ne nous quittent jamais . -Tout cela fait partie du processus . Ça fait partie de la procédure . -Il est également regrettable de noter l' achat obligatoire de nombreuses propriétés pour faire place à un grand complexe industriel qui n' est jamais arrivé . Il est navrant aussi de constater l' expropriation de plusieurs propriétés afin d' accueillir une industrie majeure qui n' est jamais venue . -Il est également possible d' obtenir un casque de protection ou un pour le même prix . Pour le même prix , il est aussi possible d' acquérir une housse de protection fabriquée en ou un . -C' est à la fois un mécanisme de sécurité - ces données ne peuvent être saisies sans connaissance - et un inconvénient - la procédure doit parfois se répéter et le temps passe . C' est à la fois une sécurité - on ne peut pas capter ces données à leur insu - et un inconvénient : la procédure doit parfois être répétée et prend donc du temps . -Il est grand temps que le cheval de Troie flamand soit arrêté . Il est grand temps d' arrêter le cheval de Troie de la Flandre . -C' est dans ce contexte que , depuis le début de l' année 2010 , le Père du organise un certain nombre de cérémonies , laïques et , à l' occasion de la Journée des saints , auxquelles il invite les familles de tous ceux qui ont été au cours de l' année . C' est dans ce cadre que le du Père organise , depuis 2010 , le jour de La Toussaint , plusieurs cérémonies du souvenir , laïques et non religieuses , auxquelles il invite les familles des personnes qui ont été dans l' année . -Il est en effet impossible à cause de la réfraction causée par les cristaux de glace qui composent la crème glacée , mais je pense que nous trouverons un moyen de le faire . C ’ est intrinsèquement impossible en raison de la réfraction causée par les cristaux de glace qui constituent la crème glacée , mais je pense que je un moyen d ’ y arriver . -Il n' est pas surprenant que l' idée se tourne vers les libéraux urbains , car les impôts pourraient être utilisés pour modifier les modes de comportement de manière à contribuer à réduire la congestion et les gaz à effet de serre , par exemple . Il n' est pas surprenant que l' idée les libéraux citadins , puisque les taxes pourraient servir à modifier les habitudes de conduite de manière à permettre de réduire les encombrements et les émissions de gaz , par exemple . -Ce n' est pas au Trésor public d' investir dans de tels projets , mais au secteur privé en fonction de la demande des consommateurs . Ce n' est pas aux paliers publics à investir dans de telles entreprises , mais bien au secteur privé en se basant sur la demande du consommateur . -Il est maintenant possible d' identifier un visage ou une iris mobile . Il est désormais possible d' identifier un visage ou un iris en mouvement . -C' est probablement la crème glacée la plus chère que j' ai faite parce que la des méduses est quatre fois plus chère que l' or . C' est probablement la glace la plus chère que j' ai créée car la des méduses vaut quatre fois plus cher que l' or . -Il rappelle le " du matin " , où , sous forme de solos , Chouinard offre une série de spectacles sur des ententes et s' devant l' avant de la scène . On pense à " Gloire du matin " , où , en solo , Chouinard effectuait une série de tâches écrites sur des cartels , posés en ligne à l' avant-scène . -C' est le contraire pour les funérailles d' un être aimé - les Français préfèrent les ( 53 % contre 47 % ) . Pour les obsèques d' un proche , c' est l' inverse : les Français préfèrent l' inhumation ( 53 % contre 47 % ) . -C' est le même principe dans une version PowerPoint . C' est le même principe , version PowerPoint géant . -Il n' est donc pas surprenant qu' il prenne un masque dans la photographie promotionnelle pour L' au Voyage , par Louis Vuitton , dont il est le nouveau visage . Rien d' étonnant à ce qu' il ait donc un masque à la main dans la photographie de la campagne L' Invitation au Voyage , de la marque Louis Vuitton , dont il incarne le nouveau visage . -On ne sait pas très bien quelle est la capacité de , mais il a mené des attaques contre des postes de police et des véhicules sur une autoroute nord-sud centrale . On ne sait pas exactement quelle est la capacité de la , mais elle a mené des attaques éclair perturbatrices sur les commissariats de police et les véhicules sur un axe autoroutier nord-sud majeur . -Cela me rend pour les nombreuses promesses qui ont disparu . Il me fait désirer les nombreuses promesses qui se sont envolées . -Cela signifie qu' ils peuvent maintenant obtenir des écouteurs , un chargeur ou même un boîtier de stockage . Ils peuvent ainsi acquérir un , un chargeur ou bien encore un étui de rangement . -Il s' est récemment demandé ce que pourrait être le début d' une nouvelle guerre d' annonce en montrant à des financiers une puce montrant trois personnes dans un restaurant , intitulé " Vous pouvez souscrire à cette devise " ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? On a pu se rendre compte récemment de ce qui pourrait être le début d' une nouvelle guerre publicitaire avec la présentation aux financiers d ’ une photo montrant trois personnes les unes contre les autres au restaurant , intitulée « Est-ce que vous ça ? » . -Il révèle que les ont aimé rester à la maison . On y découvre que le était un . -Elle a déclaré qu' elle avait décidé " d' imposer une sanction administrative à Chen en suspendant la délivrance de son autorisation de journaliste " . Il a indiqué qu' il avait décidé « d' imposer une sanction de nature administrative à Chen en lui retirant sa carte de presse » . -Elle a indiqué que la Malaisie , la Thaïlande et Singapour avaient le taux d' endettement des ménages le plus élevé par rapport au produit intérieur brut en Asie . Elle a déclaré que la Malaisie , la Thaïlande et Singapour enregistraient le ratio endettement des ménages / produit intérieur brut le plus élevé d' Asie . -Il semble que l' oiseau a été épuisé et a réussi à trouver refuge sur la plate-forme pétrolière . Il semble que l' oiseau épuisé aurait réussi à trouver refuge sur la plate-forme pétrolière . -Il convient de noter que le milieu marin est le moins connu des environnements . Il faut savoir que le milieu marin est le milieu le moins bien connu . -Rappelons que le venait de revenir d' une période de maintenance de six mois . Rappelons que le porte-avions sort d' une période d' entretien intermédiaire de six mois . -Elle précise que " les deux cas identifiés en mai 2013 restent les seuls cas confirmés en France jusqu' à maintenant " . Il précise que " les deux cas identifiés en mai 2013 restent donc les deux seuls cas confirmés en France à ce jour " . -C' est assez beau , mais c' est vrai . Mais elle a plutôt bon goût ! -Il les pompiers de qui se sont précipités sur la scène presque une heure pour faire tomber les flammes . La trentaine de pompiers qui ont été dépêchés sur les lieux ont mis près d' une heure à maîtriser le . -C' était un peu grossier , C' est un peu brutal . -C' était incroyable de le ramener dans nos bras . C' était surprenant de l' avoir à nouveau dans les bras . -Il était au milieu d' une guerre et les gens qui nous ont conduits étaient eux aussi trop nombreux . C' est la guerre , et ceux qui nous conduisent sont eux aussi très tendus . -C' était normal . C' était logique . -Il n' était pas clair si l' installation de était l' un des deux sites que les inspecteurs de l' OIAC n' avaient pas pu visiter . Il était difficile de savoir à première vue si l' unité de était l' un des deux sites que les inspecteurs de l' OIAC n' ont pas pu visiter . -Ce n' était pas quelque chose que les gens voulaient . Ce n' est pas quelque chose que les gens veulent . -Ce n' était que la deuxième fois qu' il était en compétition avec et nous étions tous quand il a gagné . C' était seulement la deuxième fois qu' il avec et nous étions fous de joie qu' il ait gagné . -C' est vraiment ce qu' ils ont fait . Ce qu' ils ont fait était vraiment courageux . -Elle a été reprise par les Forces armées de la République démocratique du Congo ( FARDC ) mercredi . Elle a été reprise mercredi par les Forces armées de la République démocratique du Congo mercredi . -Elle cherchait à s' introduire dans un marché émergent de l' assurance automobile , dans lequel les conducteurs paieraient en fonction de leur kilométrage . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Elle cherchait à percer dans le marché émergent de l' assurance auto , dans lequel les conducteurs paieraient en fonction de leur kilométrage . -C' était tout simplement rien que j' aurais pu imaginer . Cela n' avait rien à voir avec ce que j' avais imaginé . -Il était encore tendu parce qu' il n' y avait aucun moyen d' évacuer l' aéroport de Chypre à Larnaca . La tension est toujours là car nous n' avons pas de moyens pour nous évacuer vers l' aéroport de Larnaca à Chypre . -Il fera des changements dans d' autres matières premières l' an prochain . Il procèdera à des changements dans d' autres matières fondamentales l' année prochaine . -Il ne sera probablement pas long avant que nous comprenions bien pourquoi leurs queues parfois un moyen , parfois l' autre Il ne faudra sûrement pas longtemps avant que nous comprenions pourquoi leur queue va parfois d' un côté et parfois de l' autre . -Il donnera encore de l' eau . Elle distribuera toujours de l' eau gratuitement . -Il serait ridicule pour l' Écosse de voter pour l' indépendance seulement pour accepter l' austérité imposée par Berlin et Bruxelles . Il serait ridicule maintenant pour l' Écosse de voter pour l' indépendance si c' est pour accepter l' austérité imposée par Berlin et Bruxelles . -Il ne s' arrêtait pas et nous étions bien placés pour savoir ... Ça n' arrête pas , nous sommes bien placés pour le savoir ... -L' entreprise américaine a récupéré ses pertes pendant la première moitié de la période t et elle devrait livrer son troisième bénéfice consécutif complet . Sa filiale aux États-Unis a récupéré ses pertes au cours du premier semestre où la demande saisonnière est plus faible et devait tirer un bénéfice sur l' ensemble de l' exercice pour la troisième fois consécutive . -Sa dette , qui est encore relativement modeste , a explosé ces dernières années : le S & P estime à 33,5 % le PIB , contre 10 % avant la crise de 2008-2009 . Son endettement , encore relativement modeste , a explosé ces dernières années et S & P l' estime à 33,5 % du PIB , contre moins de 10 % avant la crise de 2008-2009 . -Son bénéfice net pour le troisième trimestre est passé à millions de dollars , soit $ par action , comparativement à millions de dollars , soit $ par action , l' année précédente . Son bénéfice net du troisième trimestre a diminué à milliards de dollars , soit dollar par action , contre 5,25 milliards , soit dollars par action , un an auparavant . -Son chiffre d' affaires s' élève à 1 800 millions de dollars pour l' exercice en cours . Ils sont de 1,8 milliard pour l' exercice financier en cours . -Jack , de la Voix des catholiques , a dit qu' en février , le nombre d' électeurs cardinaux aurait probablement chuté . Jack des Voix catholiques a déclaré que d' ici février , le nombre de cardinaux électeurs aura probablement diminué . -Jacques Chirac avait certainement un " palais plus raffiné " . Jacques Chirac avait certes " un appétit un peu plus développé " . -James , directeur américain du renseignement national , a réitéré en septembre que les États-Unis n' avaient pas recours à des moyens de renseignement étrangers pour " voler les secrets des entreprises étrangères au nom de sociétés américaines afin de renforcer leur compétitivité internationale ou d' accroître leur rentabilité " . James , le directeur des services de renseignement américains , a insisté à nouveau en septembre sur le fait que les États-Unis n' utilisent pas leurs capacités en matière de renseignement étranger pour « voler les secrets industriels des entreprises étrangères pour le compte de sociétés américaines afin d' accroître leur compétitivité internationale ou augmenter leurs bénéfices » . -Jenny est une vedette de la série " Saturday Night Live " et " Parks and Recreation " , bien qu' elle soit la plus connue pour sa série vidéo " Iris the Shell with Gift On " . Jenny a participé à l' émission « Saturday Night Live » et a fait partie du casting de la série « Parks and Recreation » , mais elle est pourtant plus connue pour sa série de vidéos virales « Marcel the Shell with Shoes On » . -Les sur la largeur des sièges avec des commandes importantes Les se au sujet de la largeur des sièges alors que de grosses commandes sont en jeu -Jim , exécutif à SAP , a déclaré : " Nous avons une nouvelle question à propos des clients qui n' ont pas vu le jour il y a un an - c' est à ce jour que mes données sont stockées et nous vous garantissons qu' elles peuvent demeurer physiquement dans cette catégorie . Jim , de SAP , déclare : « Nos clients nous posent aujourd' hui une question qu' ils ne nous posaient pas il y a un an : où mes données sont-elles conservées et pouvez-vous me garantir qu' elles resteront physiquement dans cette juridiction ? » . -Joel et , dont la a récemment fait l' objet d' une revue à la Cinémathèque française , ont fait la Palme d' Or à Cannes en 1991 avec Barton Labs . Joel et , dont la filmographie vient de faire l' objet d' une rétrospective à la Cinémathèque française , avaient la Palme d' or à Cannes en 1991 pour Barton Fink . -John Kerry a fait preuve d' un assouplissement de la position défensive des États-Unis à l' égard de ses programmes de surveillance avec une admission sans précédent dans laquelle , à son avis , l' espionnage a " atteint un point de vue trop " . John Kerry a indiqué un de l' attitude défensive des États-Unis vis-à-vis de ses programmes de surveillance en admettant , fait sans précédent , qu' à plusieurs occasions , l' espionnage « était allé trop loin de manière inappropriée » . -John Kerry affirme que l' espionnage américain " a atteint un niveau beaucoup trop bas " dans un appel sans précédent John Kerry a admis , fait sans précédent , que l' espionnage américain « était allé trop loin de manière inappropriée » -Le journaliste Jean-Louis a été kidnappé le 6 mars 1984 et , accompagné de trois membres de son équipe de 2 qui étaient venus filmer une manifestation du Hezbollah , il a été libéré 21 mois plus tard , le 27 novembre 1987 . à Beyrouth le 8 mars 1986 avec trois membres de son équipe d ' " Antenne 2 " venue filmer une manifestation du Hezbollah , le journaliste Jean-Louis a été libéré près de 21 mois plus tard , le 27 novembre 1987 . -A peine 10 ans , il a déjà gagné sa place dans les championnats européens de cette année à Copenhague , et il n' a rien à envier à des concurrents presque deux fois sa taille ! A seulement 10 ans , il a déjà conquis cette année sa place pour les championnats d' Europe à Copenhague et il n' aime rien tant qu' en à des concurrents qui le dépassent de trois têtes . -Juste à 4 heures du matin , un conducteur qui se dirigeait vers le nord en direction de a perdu le contrôle de son véhicule et s' est écrasé dans une autre voiture qui circulait dans la direction opposée . Peu avant 4h , un automobiliste qui circulait en direction nord , à , a perdu la maîtrise de son véhicule et a percuté une voiture qui arrivait dans l' autre direction . -Pas plus tard que l' année dernière , Ellison a eu un vote négatif sur son paquet de rémunération . L' an dernier déjà , Ellison avait subi un vote négatif sur sa rémunération . - Johnson , de , Ga , a été retrouvé le 11 janvier devant un tapis rouge appuyé derrière les de son centre de formation secondaire . Johnson , de en Géorgie , a été retrouvé le 11 janvier coincé dans un tapis appuyé debout derrière les gradins du gymnase de son lycée . -Les médias du Kenya ont fait état de la colère des autorités à l' aide de caméras de surveillance de la sécurité des membres de l' armée qui ont été envoyées sur les lieux de l' attentat qui aurait prétendument volé le centre commercial . Les médias kenyans ont suscité la colère des autorités en diffusant des images de caméras de vidéosurveillance sur lesquelles les troupes déployées sur la scène de l' attaque soi-disant le centre commercial de luxe . -Les législateurs kényans ont été la cible de la colère du public dans le passé . Les législateurs kenyans ont été la cible de la colère de la population dans le passé . -La presse kenyane à un procès controversé de la loi sur les médias La presse kenyane s' indigne de la loi controversée sur les médias -Les enfants peuvent dire : " Ce qui me plaît , c' est de parler de ma langue " , " Ma langue me semble qu' il y a des poils " , ou " ma langue est " . Les enfants peuvent dire , « J' ai l' impression que quelque chose me pique la langue » , « J' ai l' impression d' avoir un cheveu sur la langue » ou « J' ai des sur la langue » . -Le groupe chargé des crimes graves et de la criminalité organisée a mené l' enquête à l' encontre de M. , qui l' a également reconnu coupable de trois chefs d' accusation de possession de cocaïne et qui a pratiquement le mode de justice , preuve de notre engagement à traduire les criminels en justice . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; L' unité chargée de la criminalité grave et organisée du a mené l' enquête sur M. , qui a été reconnu coupable des trois chefs d' accusation de possession de cocaïne et maintenant d' entrave à la justice , témoignant de son engagement à poursuivre les criminels en justice . -Le Caporal David Hawkins , un réserviste de London ( Ontario ) , a été diagnostiqué d' un trouble de stress post-traumatique et a été libéré de l' armée , bien qu' il ait demandé à être maintenu pendant une autre année pour obtenir une pension . Le caporal David Hawkins , un réserviste de London , en Ontario , atteint du syndrome de stress post-traumatique , a été libéré de l' armée malgré sa demande de rester un an de plus afin de pouvoir recevoir une retraite pleinement indexée . -Série de véhicules Land Rover annonce Land Rover annonce son parrainage d' une série de rallyes -Le mois dernier , le Conseil national de la sécurité des transports , Mark , un expert de la sécurité nationale de l' alimentation , a déclaré que les téléphones cellulaires sont toujours considérés comme un risque . Le mois dernier , Mark du National Transportation Safety Board , expert national en sécurité des transports sur CBS News , a déclaré que les téléphones portables étaient toujours considérés comme présentant un risque . -Le mois dernier , des personnes proches de la situation ont déclaré que la avait transféré les dossiers des courriers électroniques et des messages instantanés à l' Autorité de régulation britannique , l' Autorité de gestion des marchés , envoyée à un ancien courtier . Le mois dernier , des sources proches du dossier ont déclaré que avait transmis des dossiers d' emails et de messages instantanés , envoyés à et par un ancien trader , au régulateur britannique , la Financial Conduct Authority . -La semaine dernière , Kigali a évoqué la possibilité de représailles militaires après que les obus aient atterri sur le territoire rwandais . La semaine dernière , Kigali avait évoqué la possibilité de représailles militaires à la suite des obus tombés en territoire rwandais . -L' an dernier , les festivités de l' Halloween sur la côte Est des États-Unis ont été annulées parce que la chute de Sandy était en cours . L' an dernier , les festivités d' Halloween sur la côte est des Etats-Unis avaient été annulées en raison du passage de l' ouragan Sandy . -L' an dernier , une équipe de l' Université de Lincoln a découvert que les chiens leur tête à la gauche lorsqu' ils regardaient un chien agressif et à droite lorsqu' ils regardaient un chien heureux . L' année dernière , une équipe de l' université de Lincoln a découvert que les chiens tournaient la tête vers la gauche lorsqu' ils regardaient un chien agressif et vers la droite lorsqu' il s' agissait d' un chien joyeux . -Laura , présidente de l' Association of Professional , s' est félicitée des changements . Laura , présidente de l' Association of Professional Flight , s' est dite heureuse de ces changements . -Les dirigeants des ont déclaré qu' ils les villes sous pression diplomatique , et Bertrand , chef politique de la Coalition , a déclaré que ces reculs militaires ne pas les exigences des pourparlers de paix en aucune manière . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Des responsables du ont expliqué avoir évacué les localités sous la pression diplomatique et Bertrand , responsable politique de la rébellion , a affirmé sur RFI que ces revers militaires ne modifieraient en rien ses revendications aux pourparlers de paix . - sa voix le 18 juin 2005 , lui a dit : " J' ai une histoire que nous avons l' intention de courir demain que je voudrais vraiment parler à Charles " . Dans un message vocal qu' il lui a laissé le 18 juin 2005 , disait : « J' ai une histoire que nous prévoyons de publier demain dont j' aimerais vraiment parler à Charles . » -Leonardo va prendre son souffle dans son rôle de trader fou . Leonardo va vous dans son rôle de trader . -N' oublions pas que Brigitte a été élue par votes , soit 0,5 % des habitants de Bruxelles " . " Pour rappel , Brigitte n' a été élue qu' avec 2 245 voix soit 0,5 % des ! " , conclut M. De . -Que cette affaire serve d' avertissement aux criminels que personne n' est hors de la portée de la loi . Que cette affaire serve d' avertissement aux criminels pour qu' ils sachent que personne n' est au-dessus de la loi . -Peine de mort pour Prison à vie pour un ancien chinois -Comme Davis , qui refuse de transiger sur sa musique , la pop star dit que , dans une carrière , " la chose la plus importante consiste à éviter d' être prise au piège par des choses qui nous empêchent de nous exprimer " . Comme Davis qui refuse tout compromis sur sa musique , la star pop explique que dans une carrière , " le plus important c' est de ne pas se laisser par tout ce qui peut nous empêcher de nous exprimer " . -De même , deux espèces très semblables à la morphologie se distinguent par leur génétique . De la même manière , deux espèces très proches ont été distinguées grâce à la génétique . -M. , qui avait déjà été accusé de trafic de drogue , était souvent en contact avec le maire . Alexander , précédemment accusé de trafic de drogues , était en contact fréquent avec le maire . -Vivre ensemble en France est le défi qui se pose à la Commission ( Marguerite School Board ) . Vivre ensemble en français , le défi de la commission scolaire -La chaîne locale ABC a révélé qu' une personne était évacuée sur une civière et une deuxième dans un fauteuil roulant . L' antenne locale d' ABC montrait une personne évacuée sur un , et une deuxième transportée sur un fauteuil roulant . -Aéroport de Los Angeles L' aéroport de Los Angeles évacué après une fusillade -Lou Reed Masques à l' occasion de la Journée du Tai Chi Lou Reed est mort pendant une séance de chi -Lou Reed avait subi une greffe de foie en mai dernier . Lou Reed avait subi en mai dernier une greffe du foie . -Lou Reed était un maître de l' art chinois martial connu sous le nom de chi . Lou Reed était maître chi , un art martial chinois . -Lou est un prince et un combattant , et je sais que ses chansons sur la douleur et la beauté dans le monde de nombreuses personnes avec la joie de vivre incroyable qu' il ressent " , a-t-elle ajouté . " Lou était un prince et un combattant , et je sais que ses chansons sur la douleur et la beauté du monde transmettront à de nombreuses personnes la formidable joie de vivre qu' il avait " , ajoute-t-elle . - pour nous , le " bébé dans une tendance de la citrouille " a commencé seulement l' année dernière grâce à , de sorte que la plupart d' entre nous avons grandi de ces bandes qui ne se sont pas assis à eux . Heureusement pour nous , la « tendance du » n' a commencé que l' année dernière grâce à , donc la plupart d' entre nous a sculpté ces et n' a pas eu à s' asseoir dedans . -Le vainqueur vainqueur , Jake , a combiné les couleurs de la marque automobile et du drapeau allemand , tout en intégrant les notions de vitesse et de chaleur sur la piste . L' heureux élu , Jake , a combiné les couleurs de la marque automobile et du drapeau allemand , tout en intégrant les notions de vitesse et de chaleur au niveau de la piste . -Ne vous y trompez pas , Ne nous trompons surtout pas . -La Malaisie a dépassé 80 % du PIB , contre 60 % en 2008 . La Malaisie arrive en tête de liste avec 80 % du PIB , alors que ce taux s' élevait à 60 % en 2008 . -Man of Il se rend en uniforme nazi au supermarché , avant d' en être chassé -Le directeur général , Gordon Turner , a déclaré que le centre de ventes de Toronto avait fait un bénéfice de 8 % au cours de la première moitié et a amorcé la deuxième moitié de son activité en particulier en ce qui concerne les voyages non commerciaux en Australie et au Royaume-Uni . Le directeur général , Graham Turner , a déclaré que Flight Centre avait réalisé 8 % de bénéfices au premier semestre et que le second semestre avait démarré en force , en particulier en ce qui concerne les voyages autres que les voyages d' affaires en Australie et au Royaume-Uni . -De nombreux dirigeants allemands soutiennent que les derniers rapports sont simplement la confirmation de ce qu' ils savent déjà : les États puissants veulent voler leurs secrets les plus chers et ces données doivent donc être gardées à tout prix . De nombreux dirigeants allemands indiquent que les derniers rapports ne sont qu' une confirmation de ce qu' ils savaient déjà , à savoir que les états puissants veulent voler leurs secrets les plus précieux et que ces données doivent par conséquent être gardées à tout prix . -De nombreux ennemis ont été tués ou blessés durant l' opération . Plusieurs ennemis ont été tués et blessés au cours de l' opération . -Beaucoup de gens de Wall Street considèrent les frais de port comme un signe que les compagnies aériennes facturent suffisamment d' argent pour couvrir les frais de transport aérien après des années de pertes . Bon nombre de personnes à Wall Street considèrent l' ajout de frais de bagages comme un signe qui prouve que les compagnies aériennes font payer assez cher pour couvrir le coût du voyage aérien après des années de pertes . -De nombreuses personnes sont traitées pour le paludisme , la tuberculose , la malnutrition , le diabète , les parasites et les maladies sexuellement transmissibles . De nombreuses personnes reçoivent un traitement contre le paludisme , la tuberculose , la malnutrition , le diabète , les parasites et les maladies sexuellement transmissibles . -Beaucoup de gens ont essayé " La Nouvelle Star " , et certains ont même réussi . Ils sont nombreux à avoir tenté la Nouvelle Star , et parfois même à être allés jusqu' au bout . -Les mammifères marins sont situés au sommet de la chaîne alimentaire . Les mammifères marins sont situés en haut de la chaîne alimentaire . -Martin a publié des extraits de son prochain film dans lequel le titre de rôle , un ambitieux de New York , est joué par Leonardo . Martin laisse échapper quelques images de son prochain film dont le , celui d' un courtier new-yorkais ambitieux , est incarné par Leonardo . -Martin nous raconte une histoire sincère à propos d' un " courtier de l' Est " qui refuse de collaborer avec les autorités au sujet d' un système de corruption massive sur Wall Street . Martin nous relate l' histoire vraie d' un " courtier en Bourse de Long Island , qui refuse de collaborer avec les autorités dans le cadre d' une vaste affaire de corruption à Wall Street " . -Martin a ajouté que " Double Down " ne répond pas définitivement à la question de savoir si l' analyse politique a atteint le bureau d' Obama . Martin a ajouté que « Double Down » ne répondait pas de manière définitive à la question de savoir si les sondages politiques étaient arrivés sur le bureau d' Obama . -Martin a dit à CNN qu' il avait demandé à M. si son connaissait le . Martin a dit sur CNN qu' il avait demandé à si son patron d' alors était au courant du remaniement éventuel . -Entre-temps , le personnel des FARDC a été remanié et l' armée s' est lancée dans l' offensive contre les , changeant le cours de la guerre . De son côté , l' état-major des FARDC a été remanié et l' armée est passée à l' offensive contre le , faisant changer le cours de la guerre . -Dans le même temps , M. Kerry devrait être présent ce week-end au Moyen-Orient et en Pologne pour aborder la question des stratégies américaines en Syrie , en Égypte et en Iran , ainsi que les activités de surveillance américaines . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Entre-temps , M. Kerry a prévu de se rendre ce week-end au Moyen-Orient et en Pologne pour gérer la engendrée par les stratégies américaines en Syrie , en Égypte et en Iran , ainsi que par les activités de surveillance des États-Unis . -Dans le même temps , les législateurs américains vont venir en aide à l' Europe pour s' attaquer à des problèmes d' espionnage américains et convaincre les Européens de la nécessité de poursuivre les efforts conjoints en matière de lutte contre le terrorisme avec les États-Unis , le président d' un sous-comité du Sénat sur les affaires européennes , jeudi . Entre-temps , les législateurs américains se rendront en Europe pour répondre aux préoccupations concernant l' affaire d ’ espionnage américain présumé et convaincre les Européens de la nécessité de poursuivre les efforts conjoints de lutte contre le terrorisme avec les États-Unis , a expliqué jeudi le président du sous-comité du Sénat sur les affaires européennes . -Dans le même temps , un demandeur d' asile sri-lankais âgé de 35 ans est mort d' une attaque cardiaque imminente après avoir été amenée sur un bateau d' asile à l' île Christmas cette semaine . Entre-temps , un demandeur d' asile sri-lankais de 35 ans serait mort d' une crise cardiaque après être arrivé sur un bateau transportant des demandeurs d' asile à l' île Christmas cette semaine . -Pendant ce temps , l' ancien reporter principal Neville et ancien journaliste James a supervisé la surveillance des mouvements de Mme . Entre-temps , l' ancien reporter en chef Neville et l' ancien reporter James ont supervisé la surveillance des moindres faits et gestes de Mlle . -Entre-temps , quatre projets d' hôtel d' une valeur totale de près de 600 chambres seront mis en oeuvre dans les deux prochaines années . Entre-temps , quatre projets hôteliers totalisant près de 600 chambres verront le jour d' ici deux ans . -Entre-temps , le député libéral de Gatineau , Regina , pense que le concept de fin de vie doit être clarifié avant que le projet de loi puisse être adopté . De son côté , la députée libérale de Gatineau , Stéphanie Vallée , affirme que la notion de fin de vie devra être clarifiée avant que le projet de loi ne soit adopté . - , membre du Bureau des politiques du Parti communiste chinois ( PCC ) , a fait la déclaration à Tachkent lors d' une visite officielle en Ouzbékistan . , qui est membre du Bureau politique du Parti communiste chinois ( PCC ) , s' exprimait à Tachkent lors d' une visite officielle en Ouzbékistan . - , membre du chargé des questions de sécurité intérieure , a accusé le Mouvement islamique du Turkestan oriental d' être l' instigateur de l' attaque . , membre du Bureau politique en charge des questions de sécurité intérieure , a accusé le mouvement islamique du Turkestan oriental d' être l' instigateur de cette attaque . -Menton réduit le coût de la tempête de Noël Menton réduit le coût de ses de Noël -Michel , de l' Institut du Nouveau Monde ( Institut Nouveau Monde ) a identifié un certain nombre de défis pour le conseil scolaire . Michel , de l' Institut du Nouveau Monde , a identifié plusieurs défis pour la commission scolaire . -Mick dit qu' il n' a jamais touché Perry à 18 ans . Mick dit qu' il n' a jamais Perry lorsqu' elle avait 18 ans . -Mike Warren , maire de Pitcairn , estime que la création d' une telle zone protégée serait le premier pas vers une plus grande autonomie financière pour l' île . Mike Warren , le maire de Pitcairn , estime que la création d' une telle zone protégée serait un premier pas vers plus d' autonomie financière pour son île . -Le général Hussein , commandant de la branche militaire , était présent . Le maréchal Hussein a assisté à la cérémonie . -Selon M. Miller , le lien direct avec la Bulgarie , membre de l' Union européenne , signifie que les risques géopolitiques associés aux pays de transit sont éliminés " à jamais " . Miller a ajouté que la jonction directe vers la Bulgarie , membre de l' Union européenne , signifie que les risques géopolitiques associés aux pays de transit sont éliminés « pour toujours » . -Des millions de dollars pourraient être sauvés grâce à l' introduction d' un système fiscal plus respectueux des contribuables . Des millions de dollars pourraient être épargnés afin de permettre une taxation beaucoup respectueuse de ceux qui paient . -Accusé de pornographie juvénile en Nouvelle-Écosse Pasteur accusé de pornographie juvénile en Nouvelle-Écosse -Mlle Hall , de Halifax , dans le West Yorkshire , a dit : " Regardez le sauvetage est terrifiant " . Mlle Hall , d' Halifax dans le West Yorkshire , a déclaré : « J' étais en observant les secours . » - Romney a présenté une alternative confuse et peu qui ne semble pas mettre fin à ses politiques ou à ses positions . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Romney présentait une alternative confuse et peu raffinée qui ne semblait pas permettre de déterminer ses politiques ou ses positions . -Mohamed Morsi prête serment , mais son jour de triomphe n' augure pas de la fin des troubles politiques en Égypte . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Mohamed Morsi a prêté le serment d' investiture , mais il est peu probable que son jour de triomphe marque la fin des conflits politiques en Égypte . -L' argent recueilli de force par les contribuables réduit leur revenu disponible et contribue à leur appauvrissement . Tout argent enlevé de force aux contribuables diminue leur liberté de consommation et contribue à leur appauvrissement . -Bombardier Bombardier n' a pas non plus publié de données d' essais de vol pour ses appareils neufs ou il offre une mise à jour sur la question de savoir si l' appareil atteindra son calendrier ambitieux de se rendre dans le secteur des services commerciaux d' ici septembre prochain . Bombardier , basé à Montréal , n' a pas non plus fourni de données d' essai en vol pour son tout nouvel avion ni d' informations permettant de savoir si l' avion respectera son calendrier ambitieux de mise en service commercial avant septembre prochain . -Moore ajoute que s' il découvre suffisamment de preuves pour justifier l' ouverture d' une enquête pénale ou civile sur la mort de Johnson , il demandera au FBI de le faire . Moore a ajouté que s' il met au jour des éléments de nature à justifier une enquête civile ou pénale sur le décès de Johnson , il demandera au FBI de la mener . -Moore a déclaré lors d' une conférence de presse jeudi après-midi . Moore s' est exprimé lors d' une conférence de presse jeudi après-midi . -D' après Moore , les auteurs de l' autopsie ont déclaré Johnson être morts suite à une " asphyxie " . Moore a déclaré aux journalistes que la première autopsie indiquait que Johnson était décédé par « asphyxie » . -De plus en plus de Français choisissent la crémation pour leurs propres , contre % contre 47 % , selon un sondage effectué les 6 et 7 septembre pour 1 personnes . Pour leurs propres obsèques , de plus en plus de Français choisissent la crémation , plutôt que l' inhumation : 53 % , selon une étude réalisée les 6 et 7 septembre auprès de 1009 personnes , au lieu de 47 % . -Davantage d' argent est consacré à la recherche de la vie sur d' autres planètes que la recherche marine , et il est grand temps que les choses changent . On dépense plus d' argent dans la recherche de vie sur les autres planètes que dans la recherche marine et il est grand temps que cela change . -Plus d' une trentaine de pompiers s' à 9 rue Hébert . Plus d' une trentaine de pompiers ont été dépêchés au 9 de la rue Hébert . -Plus de trois heures à l' audience , a répété à maintes reprises pourquoi l ' " architecte de la Loi sur les soins à domicile " n' a pas volontairement acheté l' assurance gouvernementale pour acheter un plan par le biais de , qu' elle utilise maintenant pour des millions d' Américains . Pendant plus de trois heures , Billy Long n' a cessé de demander avec insistance à Mme pourquoi « l' architecte » de l' Care Act n' avait pas volontairement renoncé à l' assurance parrainée par le gouvernement pour souscrire un régime via , qu' elle vante désormais à des millions d' Américains . -En outre , David Bowie a présenté cette par une vidéo . David Bowie a d' ailleurs fait les présentations de ce beau coffret dans une vidéo . -En outre , je pense que l' obstruction partisane a trop des efforts pour pousser notre nation à l' avenir : réforme de l' immigration , option publique pour les soins de santé , et fermeture de la base de Guantanamo Bay , entre autres . De plus , je crois que l' partisan a détruit de trop nombreux efforts visant à faire avancer notre nation dans le domaine des réformes de l' immigration , des soins de santé publics et de la fermeture de Guantanamo , entre autres choses . -En outre , Kiev a besoin d' argent liquide pour payer ses importations de gaz en Russie , ce qui l' accuse de ne pas avoir payé un billet de 882 millions de dollars . Kiev a besoin par ailleurs de liquidités pour régler ses importations de gaz à la Russie , qui l' accuse de ne pas avoir payé une facture de 882 millions de dollars . -De plus , l' acte reconnaît que la Congrégation de la sainte et de l' Oratoire Saint-Joseph , qui est un " acte isolé " qui permet aux agresseurs sexuels d' avoir des enfants , a " exercé des pressions morales , psychologiques et religieuses sur les victimes " , et a déclaré " qu' ils ont été victimes d' agressions sexuelles , qu' ils restent silencieux " et " qu' ils ont délibérément et consciemment décidé d' ignorer le problème " . Le recours souligne par ailleurs que la congrégation de Sainte-Croix et l' oratoire Saint-Joseph , qui est une entité distincte , " ont permis que des sévices sexuels soient perpétrés à l' encontre d' enfants " , qu' ils " ont exercé une contrainte morale , religieuse et psychologique sur les victimes " , qu' ils " étaient au courant des sévices sexuels perpétrés et les ont néanmoins " , et qu' ils ont " sciemment et consciemment choisi d' ignorer la problématique " . -De plus , il est possible que ces avancées en génétique et en biologie moléculaire remettent en question la classification ou l' appartenance d' une espèce à un genre particulier , à la famille ou même à l' ordre . Il n' est d' ailleurs pas impossible que ces progrès en génétique et en biologie moléculaire remettent en cause la classification ou l' appartenance d' une espèce à un genre , une famille , voire même un ordre . -Le discours de Morsi sur la place Tahrir a été rempli de gestes populistes odieux . Le discours de M. Morsi sur la place Tahrir s' est accompagné de gestes populistes spectaculaires . -Morsi s' est par la suite rendu à l' Université du Caire où il a prononcé son discours d' investiture . Plus tard , M. Morsi s' est rendu à l' université du Caire où il a prononcé son discours d' investiture . -Morsi devrait essayer pour les massacres de , les massacres de et le massacre de la Garde républicaine . Morsi doit juger pour les massacres de , les massacres de , et le massacre de la Garde . -Morsi a prêté un serment symbolique à la place de la place Tahrir , la semaine du carnaval , qui a mis fin au régime autoritaire de l' OLP l' année dernière , et a promis de récupérer les pouvoirs présidentiels démis de ses fonctions par le conseil militaire qui avait pris le pouvoir du dirigeant renversé . & # 160 ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; M. Morsi a fait un serment symbolique vendredi sur la place Tahrir , d' où est parti le soulèvement qui a mis fin au régime autoritaire de Moubarak l' année dernière , et a juré de récupérer les pouvoirs présidentiels dont le conseil militaire , qui a remplacé l' ancien président évincé , l' a . -La plupart n' ont cependant pas touché les frais de bagages de cabine . Pourtant bon nombre d' entre elles n' ont pas touché aux frais de bagages . -La plupart des nouveaux avions de ligne et d' autres avions ont été modifiés de sorte que les passagers puissent utiliser Wifi à des altitudes supérieures doivent satisfaire aux critères . La plupart des nouveaux avions de ligne et des autres avions qui ont été modifiés afin que les passagers puissent utiliser le Wi-Fi à des altitudes plus élevées , devraient satisfaire aux critères . -La plupart des escarmouches ont eu lieu dans la province de Sofala , qui est située à plusieurs centaines de kilomètres au nord de Maputo , mais alimente Beira , le port que des mineurs , y compris Rio Verde et Vale , utilisent pour exporter du charbon . La plupart des accrochages ont eu lieu dans la province de Sofala , qui est située à plusieurs centaines de kilomètres au nord de Maputo , mais où se trouve Beira , le port d' où les sociétés minières , notamment Rio Tinto et Vale , exportent le charbon . -La plupart des gens seront en mesure d' obtenir des plans de soins de meilleure qualité pour le même prix , voire moins bien que prévu . La plupart des gens pourront obtenir des régimes de santé complets et mieux adaptés pour le même prix , voire pour encore moins cher que prévu . - aux vétérans de la guerre à Menton hommage rendu aux anciens combattants à Menton -Les préoccupations en matière de sécurité au Mozambique sont vives en raison de la présence de puissants affrontements L' inquiétude liée à la sécurité au Mozambique grandit en même temps que des conflits éclatent entre des personnalités puissantes -M. , député du parti écologiste allemand , a publié une photo de M. sur son fil Twitter . M. , député Vert allemand , a publié une photo de lui avec M. sur son compte Twitter . -M. , de , a ajouté : " Nous avions peur qu' elle ne tombe en panne , mais elle y est restée " . M. , de , a ajouté : « On a eu peur qu' elle tombe mais elle n' a pas bougé . » -M. , qui était ministre des ports de février 2006 à novembre 2009 , a été initialement en faveur des appels d' offres publics . M. , qui a été ministre des Ports de février 2006 à novembre 2009 , était au départ favorable aux appels d' offres . -M. a trahi cette confiance et tenté de se cacher derrière le vernis de sa profession . M. a trahi cette confiance et tenté de s' abriter derrière le vernis de sa profession . -M. espère être à samedi . M. prévoit de se rendre samedi a . -M. est accusé d' abuser de sa fonction de procureur criminel , qui s' est déjà impliqué dans des clients spécifiques . M. est accusé d' avoir abusé de sa position en tant qu' avocat , et de s' être trop impliqué auprès de clients particuliers . -Selon M. , l' implication de dans l' histoire a suivi le même modèle que les autres grands hommes , comme l' ancien secrétaire d' origine , David . M. a expliqué que l' implication de dans l' histoire a suivi le même schéma qu' avec d' autres hommes importants , tels que l' ancien ministre de l' Intérieur David . -M. a déclaré que , selon l' avocat de l' ancien fournisseur de la NSA , M. ne serait pas en mesure de revenir en Russie s' il quittait son pays . M. a déclaré que , selon l' avocat de l' ancien consultant de la NSA , M. ne pourrait pas retourner en Russie s' il quittait le pays . -M. dit que M. , qui doit se retirer lorsque son second mandat prendra fin l' année prochaine , veut détruire la démocratie du pays . M. a dit que M. – qui devrait se retirer à la fin de son second mandat l' année prochaine – voulait détruire la démocratie dans le pays . -M. a déclaré à l' AFP , l' agence de presse française , mercredi , que M. était considéré comme un " perdant " désireux d' utiliser " tout ce qui restait de ses forces pour essayer de prouver qu' il pourrait imposer au gouvernement ses propres décisions " . M. a déclaré mercredi à l' AFP , l' agence de presse française , que M. se considérait comme un « perdant » qui voulait utiliser « toutes les forces qu' il lui restait pour tenter de prouver qu' il pouvait imposer ses décisions au gouvernement » . -M. a été interrogé sur la question de savoir si cette menace . M. était faussement timide lorsqu' on lui a demandé si la cette menace . -M. ne pas sa profession , mais l' . M. n' a pas fait honneur à sa profession , mais l' a . -M. Brown a déclaré : " Le rail de grande vitesse peut apporter des avantages économiques considérables pour l' Écosse , mais il ajoute aussi le poids économique de l' Écosse à l' ensemble de la réglementation pour les trains à grande vitesse dans toute la Grande-Bretagne " . M. Brown a déclaré : « Le train à grande vitesse a le potentiel d' offrir des avantages économiques considérables à l' Écosse , mais il apporte également le poids économique de l' Écosse dans le projet global de réseau ferroviaire à grande vitesse dans toute la Grande-Bretagne . -M. Chen a écrit plusieurs articles pour le New Express en alléguant des irrégularités financières dans une société dénommée . M. Chen a écrit plusieurs articles pour le New Express sur les irrégularités financières présumées d' un fabricant d' engins de chantier appelé . -M. Cook ajoute : " Compte tenu de l' augmentation des frais de timbre dans les maisons à forte valeur et de l' introduction de la législation modifiée , il est très difficile d' affirmer que les biens à valeur élevée sont , sans égard à l' effet du régime fiscal des conseils de bande " . M. Cook a déclaré : « Suite à l' augmentation du droit de timbre sur les propriétés de grande valeur et l' introduction de la loi en la matière , il est très difficile d' avancer que les propriétés de grande valeur sont indépendamment des effets du système de taxe d' habitation obsolète . » -M. Dutton a demandé à Mme de veiller à ce qu' aucun dollar du budget de sauvetage ne soit affecté à une bureaucratie supplémentaire . M. Dutton a rendu visite à Mme pour garantir qu' aucun dollar du plan de sauvetage ne sera dépensé en bureaucratie supplémentaire . -M. Hare , qui a commencé avec le YMCA lorsqu' il a 21 ans , a accepté la gestion " de moi " parce qu' il n' a pas réussi à recruter du Seigneur et a fait en sorte que le personnel soit certain de ses obligations de signalement des manquements à la sécurité de l' État . M. Hare , qui a commencé à travailler au YMCA à l' âge de 21 ans , a admis que la gestion du personnel a été un échec avec le recrutement de Lord et qu' il avait omis de vérifier que les employés savaient qu' ils étaient tenus de signaler toute violation de la politique de sécurité des enfants . -M. Hare a indiqué qu' il avait dit à l' assemblée du YMCA de Calgary que la formule utilisée par l' organisation " Fox Frank " était " sans rapport avec le signalement " par le personnel et que le conseil était d' accord avec lui . M. Hare a déclaré qu' il avait exprimé son point de vue devant le conseil du YMCA de Nouvelle-Galles du Sud disant que , pour l' organisation , la leçon tirée de « l' incident Jonathan Lord » ne devait pas concerner « le signalement » par le personnel , et le conseil était d' accord avec lui . -Selon M. Hare , la décision de mettre le personnel à signer des accords de confidentialité a peu de temps après que les allégations ont été soulevées par le directeur général des services jeunesse du YMCA , Liam . M. Hare a dit que la décision de faire signer au personnel des accords de confidentialité , peu après que les allégations ont été évoquées , a été prise par le directeur général du YMCA chargé des services à l' enfance , Liam . -M. Hare a dit à Gail , conseiller auprès de la Commission , qu' il connaissait le premier cas mais qu' il ne connaissait pas le second . M. Hare a déclaré à Gail , avocat-conseil auprès de la Commission , qu' il avait eu connaissance de la première affaire mais qu' il n' était pas au courant de la seconde . -M. Hughes a déclaré que aurait dû partager le même sort que trois autres membres de la cellule terroriste responsable du carnage - , et Imam - qui a été exécuté il y a quatre ans . M. Hugues a déclaré que aurait dû connaître le même sort que les trois autres membres de la cellule terroriste de responsable du carnage – , et Imam – qui ont été exécutés il y a 4 ans . -M. Kerry insiste toutefois sur le fait que la NSA est une force de bonne volonté et que ses opérations de surveillance ont sauvé de nombreuses vies . M. Kerry a insisté , toutefois , sur le fait que la NSA était une force agissant pour le bien et que ses opérations de surveillance avaient sauvé de nombreuses vies . -M. Palmer , 59 ans , a déclaré que ses politiques comprenaient un aéroport international pour la Sunshine Coast et qu' il prendrait " très au sérieux " son nouvel emploi . M. Palmer , 59 ans , a déclaré que son projet politique comprenait un aéroport international pour la Sunshine Coast et qu' il prendrait ses nouvelles fonctions « très au sérieux » . -M. Palmer a également critiqué l' utilisation de crayons pour marquer les votes . M. Palmer a aussi critiqué l' utilisation de crayons de papier pour marquer les bulletins de vote . -M. Palmer a demandé une révision du décompte des voix afin d' accélérer le processus . M. Palmer a avancé l' idée d' un remaniement du système de dépouille des suffrages pour accélérer le processus . -M. a dit dans sa lettre qu' il avait été confronté à une " campagne de persécution " sérieuse et soutenue qui l' a forcé de quitter son domicile . M. a écrit dans sa lettre qu' il avait fait l' objet d' une campagne de persécution « grave et soutenue » qui l' avait forcé à quitter son pays . -Le groupe de message instantané de M. était composé de banquiers à Barclays et à Citigroup , des personnes proches de la réalité . Le dossier de M. comprenait des messages instantanés envoyés à et par des banquiers travaillant chez Barclays et Citigroup , ont ajouté des sources proches du dossier . -M. a déclaré aujourd ' hui que les 325 millions de dollars à peu près " à moins qu' il ne soit suivi d' une réforme réelle visant à mettre le système de santé publique de la Tasmanie sur une base plus durable " . M. a dit aujourd' hui que les 325 M $ ne compteront pas pour grand-chose « à moins qu' ils soient suivis d' une réforme véritable permettant de consolider le système de santé public de Tasmanie » . -M. a déclaré que vous étiez prêt à garantir ces 333 emplois pendant quatre ans . M. a dit que vous étiez prêt à garantir ces 333 emplois durant quatre ans . -M. est un charmant jeune homme qui tente de sauver certains des emplois industriels les plus rémunérateurs . M. , est un charmant jeune homme qui tente de sauver des emplois industriels parmi les mieux payés . -M. Hawkins a souligné qu' un soldat pouvait être préparé pour son départ , avec des séances de planification et de consultation , mais qu' il était tout à fait différent de vouloir quitter l' armée . M. Hawkins a souligné qu' un soldat pouvait être préparé à partir , avec des plans et des séances de consultation , mais que cela était totalement différent que le fait de vouloir quitter les rangs . -Mme , qui est également directrice du Home de Gatineau , pense que le service sera sans aucun doute utile . Mme , aussi directrice de la maison à Gatineau , estime que le service serait de toute façon inutile . -Mme Franklin a été occupée en novembre pour un album de Clive Mac et de Sony Music , produite par Don et Kenny " " . Mme Franklin s' affaire en novembre à l' enregistrement d' un album pour Clive Davis et Sony Music , produit par Don Was et Kenny " " . -Mme a dit : " Tout le monde est électrique et tout le monde a et pris des applaudissements " . Mme a déclaré : « Il y avait de l' électricité dans l' air et tout le monde l' encourageait et l' . » -Muller n' a pas survécu à la guerre . Müller n' a pas survécu à la fin de la guerre . -Mes camarades de classe ont ri avec joie , et mes parents ont sauvé tous les journaux qu' ils pouvaient trouver . Mes camarades ont pleuré de joie et mes parents ont conservé tous les journaux qu' ils ont trouvés . -Ma mère est décédée il y a dix-neuf ans . Ma mère est décédée il y a dix-neuf ans maintenant . -Ma mère a pris possession de son visage , et chaque matin , elle me regarde dans le miroir . Ma mère a pris possession de mon visage et elle me regarde tous les matins dans mon miroir . -Mon objectif est de découvrir la vérité . Mon objectif est de découvrir la vérité . -Mes parents ne sont plus là , mais je me sens plus proche de moi . Mes parents ne sont plus mais je les sens constamment près de moi . -Ma soeur et moi avons été autorisés à manger des bonbons , à rester en retard et à jouer pour le quartier . Ma sœur et moi étions autorisés à manger des bonbons , à aller nous coucher tard et à nous déguiser pour aller dans le voisinage . -Ma femme et la mère de mes trois enfants sont morts du cancer à 43 ans . Ma femme et la mère de mes 3 enfants est décédée d' un cancer à 43 ans . -NEC offre à la police judiciaire un sac portable d' une valeur de 90.000 euros qui peut analyser des échantillons d' ADN sur une scène du crime en moins d' une heure . NEC propose aux services de police judiciaire une valise coûtant 90.000 euros pour analyser des échantillons d' ADN sur le lieu d' un crime et en une heure . -NEC propose donc la reconnaissance VIP à l' entrée d' un hôtel ou d' un magasin . NEC propose ainsi la reconnaissance de VIP à l' entrée d' un hôtel ou d' un magasin . -La NHTSA n' a pu examiner la lettre de déclaration des propriétaires en raison de la fermeture de 16 mois du gouvernement , ce qui a provoqué une baisse de la croissance des ventes d' automobiles en octobre . La NHTSA n' a pas pu examiner la lettre d' information aux propriétaires en raison de l' arrêt de 16 jours des activités gouvernementales , ce qui a ralenti la croissance des ventes de véhicules en octobre . -NSA " Internal error " , for crash Internet La NSA met en cause une « erreur interne » et non des pirates informatiques pour la panne de son site - : " Les États-Unis sont allés trop loin " , a déclaré Kerry de la NSA : les Etats-Unis sont " allés trop loin " , admet Kerry -NSA Événements sans fond de débats sur les renseignements L' affaire NSA souligne l' absence totale de débat sur le renseignement -Des révélations de sécurité renforcent la paranoïa des entreprises en ce qui concerne la surveillance de l' État Les révélations de la NSA accroissent la paranoïa des entreprises à l' égard de la surveillance d' État -Les noms d' autres législateurs participants devaient être publiés dans les prochains jours . Les noms des autres législateurs participants devraient être dévoilés dans les jours à venir . - P. Morris est un étudiant de deuxième année à la Harvard Medical School . P. Morris est étudiant en 2e année de médecine à la Faculté de médecine de Harvard . -Près de 100 migrants africains qui espèrent voyager en Algérie meurent de soif après que leurs deux camions se retrouvent au milieu du Sahara . Presque 100 migrants africains espérant se rendre en Algérie sont morts de soif après que leurs deux camions sont tombés en au milieu du Sahara . -Nécessité d' en faire encore plus Besoins toujours croissants -Ni l' un ni l' autre ne sont de bons négociateurs , et aucun d' entre eux n' est prêt à faire le genre de concessions nécessaires . Aucun d' eux ne sait bien négocier et aucun d' eux n' est prêt à faire les concessions nécessaires . -Les commandes nettes sont tombées à 26 avions , à partir de 83 . Les commandes nettes sont tombées à 26 avions , contre 83 auparavant . -Le Nevada a déjà achevé un projet pilote . Le Nevada a déjà mené à terme un projet pilote . -Le Nevada fait partie de plusieurs états qui se aujourd' hui pour trouver des technologies abordables qui permettraient à l' état de surveiller le nombre de kilomètres parcourus par une automobile , mais qui ne sont pas exactement là où et à quel moment . Le Nevada fait partie des quelques États qui se désormais pour trouver une technologie abordable qui permettrait à l' État de savoir combien de miles parcourt une voiture , mais sans connaître le lieu ni l' heure exacts . -Néanmoins , Cable a connu une croissance de 45 % , atteignant millions de CAD . Les revenus de Câble ont cependant progressé de 45 % pour atteindre millions . -Néanmoins , GM a fait état d' une augmentation d' environ 16 % des ventes le mois dernier avec véhicules , contre prévus par consensus . GM a néanmoins fait état d' une progression de près de 16 % de ses ventes le mois dernier à 226 402 véhicules , contre 211 563 attendues par le consensus . -Néanmoins , il espère que les consultations menées par le Conseil de la radiodiffusion et des télécommunications canadiennes ( CRTC ) donneront lieu à des recommandations intéressantes . Il espère cependant que les consultations menées par le Conseil de la radiodiffusion et des télécommunications canadiennes ( CRTC ) vont déboucher sur des recommandations intéressantes . -Néanmoins , on comprend vite que les objets disparates rassemblés dans ce livre très beau conduisent le " vendeur de biens d' occasion " à une passion constante , celle de découvrir le coeur qui incarne l' identité française . Mais on comprend vite que les objets disparates rassemblés dans ce livre au si beau titre ramènent le à une seule et même passion : celle de découvrir le noeud constitutif de l' identité française . -Néanmoins , le gouvernement n' a pas déclaré son intention de réformer son système de lutte contre la corruption , par exemple en créant un organe indépendant de la Partie . Mais le gouvernement n' a aucunement manifesté l' intention de réformer le système de lutte contre la corruption en créant par exemple un organisme indépendant du parti . -Le nouveau vaccin pourrait prendre le plaisir de fumer . Le nouveau vaccin pourrait supprimer le plaisir de fumer -Demande d' indemnisation présentée par une nouvelle classe contre la Croix de la Trinité Nouvelle demande de recours collectif contre les frères Sainte-Croix -La ville de New York en étudie un . La ville de New York en envisage un . -Le psychiatre de la ville de New York , Jack , qui se spécialise dans les questions d' identification des sexes , a déclaré que la nouvelle loi allemande " ressemble à une bonne chose " . Le Dr Jack , psychiatre à New York et spécialiste des questions d' identification sexuelle , a déclaré que la nouvelle loi allemande « semble être une bonne idée » . -Les nouveaux avions de ligne sont beaucoup plus tributaires des systèmes électriques que les générations précédentes d' aéronefs , mais ils sont aussi conçus et approuvés par la FAA pour résister aux interférences électroniques . Les nouveaux avions de ligne dépendent beaucoup plus des systèmes électriques que les générations précédentes d' avions , mais ils sont également conçus pour résister aux interférences électroniques et approuvés par la FAA . -Nouvelles : la Bourse de Tokyo enregistre % La Bourse de Tokyo finit en baisse de % , Actualités -Les nouvelles sur les montres ont les traders dans un secteur qui a été l' un des plus importants atouts des banques de portefeuille des activités commerciales au cours des dernières années , mais qui a été contestée cette année parce que la volatilité faible des devises réduit les possibilités pour les spéculateurs . & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; Les nouvelles concernant ces enquêtes ont ébranlé les traders dans un secteur qui a été l' un des plus gros moteurs de croissance des unités de trading des banques d' investissement au cours des dernières années , mais qui a été remis en cause cette année en raison de la faible volatilité des devises qui limite les opportunités pour les spéculateurs . -Selon certaines informations , la NSA a maintenu des millions de téléphones aériens en Europe , a entretenu des relations avec certains alliés américains , bien que le chef de l' agence ait déclaré cette semaine qu' ils étaient inexacts et qu' ils ne comprenaient pas une mauvaise compréhension des liens que les alliés de l' OTAN ont recueillis et auxquels ils ont communiqué avec les États-Unis . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; » La presse rapporte que la NSA a capté des millions de communications en Europe , ce qui a mis à mal les relations avec certains alliés des États-Unis , cependant le directeur de l' agence a déclaré cette semaine que c' était inexact et qu' il s' agissait d' un malentendu car cela concernait des métadonnées que les alliés de l' OTAN avaient collectées et partagées avec les États-Unis . -La de l' an prochain serait importante parce qu' elle sera la première depuis la nomination de Francis en mars de cette année , a déclaré . Le de l' année prochaine sera important car ce sera le premier depuis l' élection du pape François en mars de cette année , a dit . -Nick a gagné en importance sur une ( s " The League " ) et a maintenant son propre disque de théâtre public . Nick a fait son apparition sur le devant de la scène dans une sur une chaîne câblée ( « The League » sur le thème du fantasy football joyeusement de la chaîne ) et a désormais son propre spectacle de sur Central . -Neuf Ruby a échoué après avoir été victime d' un accident de la route dimanche après-midi et a survécu trois jours seulement avant d' être secouru d' une carrière mercredi . Ruby , un de 9 mois , s' est échappée après un accident de la route dimanche après-midi et a survécu pendant trois jours seule avant d' être sauvée dans une carrière mercredi . -Nissan , qui a été obligé de procéder à la révision , a également annoncé une restructuration de la direction . Nissan , qui a dû procéder à une telle révision , a également annoncé un remaniement de sa direction . -Aucune information classifiée n' est en danger , a déclaré le porte-parole . Aucune information classifiée n' est en danger , a expliqué le porte-parole . -Aucune contribution financière n' a été demandée avant les changements apportés au dernier budget . Avant les changements apportés au dernier budget , aucune contribution financière n' était réclamée . -Les nouveau-nés ne seront plus assignés de manière rigide à un homme ou à une femme . Le sexe masculin ou féminin des nouveaux nés ne leur sera plus attribué , de manière rigide , à la naissance . -Personne ne peut répondre aux questions sur la raison de cette situation . Personne ne peut répondre à la question de savoir pourquoi cela se produit . -Aucun des candidats ne semble vraiment avoir pris la première place . Aucun candidat ne semble avoir réellement marqué lors de ce premier prime . -Aucun de ces négociants n' a été accusé d' un quelconque acte répréhensible . Aucun de ces traders n' a fait l' objet d' aucune accusation . -Ce ne sont pas non plus les types de personnes qui utilisent le uniquement sur le plan financier . Les personnes qui se rendent chez les prêteurs sur gage ne sont pas non plus seulement ceux qui ont des problèmes financiers . -Il n' a pas non plus fait remarquer que les États membres de l' UE passent beaucoup de temps à plier leurs règles formelles si elles le veulent . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Personne non plus n' a dit que les États membres de l' UE consacrent une grande partie de leur temps à contourner leurs règles formelles lorsque ça les arrange . -Le Nord-Kivu est l' une des zones les plus densément peuplées de la RDC , et ses terres regorgent de richesses minières bien connues . Le Nord-Kivu est l' une des régions les plus densément peuplées de la RDC , et son sous-sol regorge de ressources minières . -Ce n' est pas tout le monde qui se réjouit de la décision du ministre Brigitte de donner un coup d' éclat aux écrans de la circulation à Bruxelles . Le projet de la ministre Brigitte de " " des poteaux des feux de signalisation à Bruxelles ne plaît pas à tout le monde . -Il n' y a pas si longtemps , on nous demandait de le définir collectivement . Voici peu , nous avons été collectivement conviés à en donner une définition . -Non pas que tout doit être simplifié ou à la bonne fin , mais qu' il y ait un sens inné du bien et du mal . Non pas que tout doive être simplifié ou finir bien , mais il existe une sens inné du bien et du mal . -Pas trop fatigué de faire des surprises , David Bowie a eu plus d' un coup de manches lors de la prochaine journée : Pas las des surprises , David Bowie avait plus d' un tour dans son sac avec The Next Day . -Pas encore à l' école et juste quelques mois à la fin de ses premières démarches , le et son Shetland par les trois tours avec le plus facile à prendre le meilleur - laisser ses 30 joueurs d' âge adulte derrière le but . , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ... Pas encore à l' école et seulement quelques mois après avoir fait ses premières armes , le et son Shetland ont pendant les trois tours avec une grande facilité et ont remporté le premier prix , laissant derrière eux les 30 autres concurrents adultes . -Rien n' est plus drôle d' Halloween que les maisons . Rien n' est plus représentatif d' Halloween que les maisons . -Nouri Al veut une " guerre mondiale " contre Al-Qaida . Nouri Al veut une " guerre mondiale " contre Al-Qaida . -Le 4 novembre est également le jour où la formation de l' hôpital reprend . Cette date du 4 novembre correspond également au renouvellement des internats . -Airbus s' adresse aujourd' hui au public avant le salon aéronautique de Dubaï , où le devrait dominer avec plus de 100 commandes . Aujourd' hui , Airbus en appelle directement au public avant le salon aéronautique de Dubaï , où le devrait prendre le pas sur ses concurrents avec plus de 100 commandes . -Maintenant , imaginez que Titan achète une usine fermée de Goodyear . Maintenant , imaginons que Titan achète à Goodyear l' usine fermée . -À l' âge de 29 ans , Katie est mariée et à l' Université de Pennsylvanie , un résident de la psychiatrie infantile . Aujourd' hui âgée de 29 ans , Katie est mariée et interne en psychiatrie infantile à l' université de Pennsylvanie . -Aujourd' hui , au cours de sa troisième année , la guerre civile est à l' origine des principaux rebelles musulmans sunnites contre le gouvernement d' Assad et ses forces de sécurité , qui sont avec des membres de sa secte , une de l' Islam chiite . Maintenant dans sa troisième année , la guerre civile oppose principalement les rebelles musulmans sunnites au gouvernement d' el-Assad et ses forces de sécurité , qui sont soutenus par les membres de la secte des , une branche de l' islam chiite . -Maintenant , le nouveau " Express " doit faire l' objet d' une " rectification complète " , a déclaré l' organisme de réglementation . À présent , le New Express va subir un « remaniement complet » , a déclaré le régulateur . -Maintenant , il y a juste l' écart , le vide silencieux , la nécessité de lui me confier , de me rassurer dans la paume de sa douceur . N' est resté que le manque , ce vide sans réponse , ce besoin d' elle , de me confier , de me rassurer au creux de sa douceur . -Maintenant , ils peuvent tenter graduellement de reprendre leur vie . Et tenter , peu à peu , de reprendre le cours de leur vie . -Aujourd' hui , 73 % des personnes qui choisissent l' incinération veulent qu' une cérémonie soit organisée . Désormais , 73 % des personnes ayant choisi une crémation souhaitent l' organisation d' une cérémonie . -Aujourd' hui , je suis devenu un . Aujourd' hui , je suis devenu plus . -Aujourd' hui , je la considère comme une présence quotidienne , une force bienveillante , un esprit d' épargne . Désormais , je la ressens comme une présence quotidienne , une âme bienveillante , un esprit . -De nombreux Ouïghours , une minorité au Xinjiang , appellent cette province chinoise du Turkestan oriental . De nombreux , une minorité installée au Xinjiang , appellent cette province chinoise le Turkestan oriental . -De nombreuses motos ont suivi notre voiture , conduite par , depuis l' aéroport jusqu' à notre domicile . Entre l' aéroport et chez moi , de nombreuses motos suivent notre voiture que conduit . -D' après le point de contrôle latéral , le site de Frontier City et les procédures d' enregistrement sont modifiés afin de s' assurer que les passagers sont au courant de la taxe avant qu' ils arrivent à la porte d' embarquement . O' Malley a expliqué que le site et les procédures d' enregistrement de Frontier sont en train de changer pour garantir que les passagers connaissent l' existence de ces frais avant d' arriver à la porte . -M. O' Malley a dit que la nouvelle accusation n' est pas de recueillir de l' argent . O' Malley a déclaré que ces nouveaux frais ne serviront pas vraiment à collecter de l' argent . -Le bilan de Pékin en 2007 est un triste rappel de ce qui aurait pu être Le mémoire de la campagne 2008 d' Obama est un triste rappel de ce qui aurait pu se passer -Le retour en arrière de la politique de santé Obama fait marche arrière sur sa réforme du système de santé -Obama sur le FMI et la Banque mondiale Obama met fin aux écoutes visant le FMI et la Banque mondiale -Les signes évidents de développement , tels que les seins , les seins et les aisselles , et l' odeur du corps apparaissent plus tôt chez les filles . Les signes évidents de développement , tels que le des seins , les poils sous les bras et dans la zone et les odeurs corporelles , apparaissent plus tôt chez les filles . -Évidemment , les élections électives ne sont pas un calendrier prévisible , car le parti politique opposé et les événements aléatoires , comme le massacre de , notre conversation publique . Évidemment , la fonction électorale n' est pas prévisible , car le parti politique d' opposition et les évènements qui se produisent par hasard , comme le massacre de Newton , influencent notre conversation publique . -Évidemment , il connaît la seconde , la surveillance , qu' il doit faire . Il est évident qu' il est au courant du deuxième , la surveillance , il ne peut pas en être autrement . -Octobre : Mois d' en Irak en 2008 Irak : octobre a été le mois le plus sanglant depuis 2008 -Le mois d' octobre a été le mois le plus sanglant en Irak cinq ans et demi , selon les statistiques publiées par les autorités irakiennes le vendredi 1er novembre . L' Irak a connu en octobre son mois le plus meurtrier en cinq ans et demi , selon des chiffres publiés vendredi 1er novembre par les autorités irakiennes . -Octobre a été le mois le plus sanglant d' Iraq depuis avril 2008 Octobre a été en Irak le mois le plus meurtrier depuis avril 2008 . -Bien sûr , je lui parle à la nuit morte quand l' absence de trop de choses tombe , et elle arrive dans mes rêves quand mes esprits sont un peu bas . Bien sur , je lui parle dans le creux de la nuit quand l' absence fait trop mal , elle s' invite dans mes rêves quand le moral est un peu atteint . -Bien sûr , mais il met l' accent sur son intérêt pour le , à ( Institut des sciences politiques français ) et les Hautes Etudes ( School for Advanced Studies in Social Sciences ) . Sans doute , mais en soulignant son faible pour les chemins de traverse , , les Hautes Etudes . -De ce nombre , 200 n' ont toujours pas trouvé un nouvel emploi . Parmi elles , 200 n' ont toujours pas trouvé de nouvel emploi . -Parmi eux , il y a eu civils , dont enfants . Parmi eux , on estime qu' il y avait 61 067 civils , dont 6 365 enfants . -En offrant aux jeunes ( souvent talentueux ) des jeunes un rôle de premier plan dans une musique , ils garantissent l' adoration des jeunes filles et souvent leur famille tout entière . Proposer des beaux jeunes hommes ( souvent talentueux ) en tête d' affiche d' un spectacle musical , c' est l' assurance d' un engouement de ces jeunes filles et souvent de leur famille entière . -Des policiers ont , 32 , ont déplacé trois kilos de cocaïne dans le domicile de l' avocat à Blackburn , , alors qu' il était dans un dîner de la société de droit avec des collègues . Les fonctionnaires de police ont repéré , 32 ans , en train de déposer trois sacs de cocaïne au domicile de l' avocat à Blackburn , dans le , alors qu' il assistait à un dîner du Barreau avec des confrères . -Les responsables ont annoncé le jeudi 31 octobre qu' un cas de rage avait été détecté dans un à . Les autorités ont annoncé jeudi 31 octobre qu' un cas de rage avait été détecté chez un chaton dans le Oise , manifestement d' origine étrangère alors que la France est indemne de cas autochtones depuis 2001 . -Le pétrole s' abaisse pour atteindre un baril de 96 dollars Le prix du pétrole continue à baisser et se rapproche de 96 $ le baril -Omar , un démonstrateur , a dit que les gens voulaient juste la stabilité . Omar , un manifestant , a dit que les gens voulaient seulement de la stabilité . -Omar Hassan est toujours en détention et doit comparaître devant le tribunal le vendredi . Omar Hassan est toujours en détention et devra se présenter en cour vendredi . -Omar Moustapha Hassan est accusé de prise d' otage , de provocation , de menaces verbales , d' arme ayant une intention dangereuse , d' enlèvement et de non-respect d' une ordonnance . Omar Moustapha Hassan a été accusé de prise d' otage , séquestration , de menaces , port d' arme dans un dessein dangereux , enlèvement , et défaut de se conformer à une ordonnance . -Le 8 janvier 2013 , date de son anniversaire , il annonce qu' un nouvel album sera publié en mars . Le jour de son anniversaire , le 8 janvier 2013 , il avait déclaré qu' un nouvel album sortirait en mars . -Né d' un , David Bowie une fois de plus son : il aurait pu partir d' une autre planète ? Sur Born In A , David Bowie fait une nouvelle fois référence à son : d' une autre planète ? -Vendredi dernier , les actes de violence les plus récents ont tué quatre personnes dans le nord de l' Iraq , une journée après que 26 personnes ont été tuées dans une série d' attentats , dont cinq voitures piégées qui ont explosé dans le nord de Bagdad . Vendredi , de nouvelles violences ont tué quatre personnes dans le nord de l' Irak , au lendemain de la mort d' au moins 26 personnes dans une série d' attentats , dont l' explosion de cinq voitures piégées au nord de Bagdad . -Vendredi , l' armée congolaise préparait une nouvelle attaque contre les bastions définitifs des rebelles de près de , dans l' est de la République démocratique du Congo . L' armée congolaise se préparait vendredi à un nouvel assaut contre les derniers bastions des rebelles du près de , dans l' Est de la RDC . -Le 25 juillet , l' ONU estime qu' environ 100 000 personnes sont mortes dans le conflit depuis mars 2011 . Le 25 juillet , l' ONU a estimé que 100 000 personnes étaient mortes dans le conflit depuis mars 2011 . -Lundi , 20 000 policiers ont été déployés devant l' Académie de la police du Caire , où Mohamed Morsi sera jugé . Lundi , 20 000 policiers seront déployés devant l' académie de police du Caire , où sera jugé Mohamed Morsi . -Lundi , la veille de l' entrée en vigueur de la loi , un juge de première instance a estimé que les dispositions relatives à l' octroi de privilèges étaient inconstitutionnelles . Lundi , un juge de premier instance avait estimé à la veille de l' entrée en vigueur de la loi que les dispositions traitant du privilège d' admission étaient . -Dimanche , l' équipe revient à la ville , fatiguée , mais fière de leur travail . Le dimanche , l' équipe retourne en ville , fatiguée mais fière du travail qu' elle a accompli . -Jeudi , l' OIAC a déclaré que les deux sites étaient , selon la Syrie , " abandonnés et que les programmes d' armes chimiques qu' ils contenaient avaient été déplacés vers d' autres sites déclarés , qui avaient été inspectés " . Jeudi , l' OIAC a dit que les deux sites étaient , selon la Syrie , « abandonnés et ... que les éléments du programme d' armes chimiques qu' ils contenaient avaient été déplacés vers d' autres sites déclarés , qui ont été inspectés » . -Jeudi , les données montraient une faiblesse persistante des exportations et un affaiblissement de la demande des consommateurs . Jeudi , des données ont montré une faiblesse continue des exportations et un ralentissement de la demande des consommateurs . -Jeudi , il a plaidé que la communauté internationale devait mener une " troisième guerre mondiale " contre Al-Qaida . Jeudi , il a plaidé pour que la communauté internationale mène une " troisième guerre mondiale " contre Al-Qaida . -Le jeudi soir , il a affirmé que le président Obama avait " résolu de clarifier la question et a commencé un réexamen de ces pratiques afin de s' assurer que personne n' ait induit l' erreur " . Il a assuré jeudi soir que le président Obama était " résolu à tenter de clarifier et qu' il procédait à un réexamen de ces pratiques afin que personne ne se sente trompé " . -Mercredi , les Centers for Disease Control and Prevention ont publié un ensemble de lignes directrices pour gérer les allergies alimentaires chez les enfants à l' école . Mercredi , le Centre américain de contrôle et de prévention des maladies a publié une série de directives indiquant comment gérer les allergies alimentaires des enfants à l' école . -Mercredi , les habitants de sont descendus dans les rues de la ville pour se féliciter de l' entrée des FARDC . Mercredi , les habitants de ont envahi les rues de la bourgade pour saluer l' entrée des soldats des FARDC . -Le mercredi , le porte-parole libéral des anciens membres du personnel militaire , Jim , a demandé le retour du caporal Hawkins , tandis que le Jack Harris a demandé une cessation immédiate de cette " pratique honteuse " . Le porte-parole libéral en matière d' anciens combattants , Jim , a demandé mercredi la réintégration du caporal Hawkins , tandis que le Jack Harris a réclamé la fin immédiate de " cette pratique honteuse " . -Le mercredi , elle a réduit de 25 % le kilométrage parcouru . Mercredi , elle a réduit les primes de miles de 25 % sur un voyage . -Un jour léger à la fin août un hélicoptère de police allemand peu au-dessus du consulat des Etats-Unis à Francfort , la capitale financière de l' Allemagne . Par une belle journée de fin août , un hélicoptère de la police allemande a survolé à basse altitude le Consulat britannique de Francfort , la capitale financière de l' Allemagne . -Sur son site web a donné une séance de questions et de réponses sur ce qu' il a dit , son travail de fantaisie serait un avocat qui représente des clients à File Row en Amérique , son ultime invité comme Mohammed Ali et l' inégalité en tant que motivation pour le travail de . Sur son site , a organisé une séance de questions / réponses sur lui-même au cours de laquelle il disait que son job de rêve serait d' être avocat pour représenter des clients dans le couloir de la mort en Amérique , son dernier invité à dîner serait Mohammed Ali et l' inégalité serait ce qui le à travailler . -De son côté , Chrysler , qui est sous le contrôle de Fiat , a annoncé une augmentation de 11 % des ventes de véhicules , contre un prévu . De son côté , Chrysler , contrôlé par Fiat a annoncé une hausse de 11 % de ses ventes à 140 véhicules , contre 143 536 attendues . -De son côté , Vodafone ( + % ) continue de prendre la direction du dans le communiqué en réponse à un communiqué de presse indiquant que AT & T envisageait une éventuelle offre de rachat . De son côté , Vodafone ( + % ) se maintient en tête des hausses de l' en réaction à une information de presse voulant qu' AT & T étudie une éventuelle OPA . -Sur place , je rencontre des journalistes , dont ceux de l' AFP , des diplomates et du personnel des services , dont je connais depuis vingt ou trente ans . Sur place , je rencontre des journalistes , dont ceux de l' AFP , des diplomates , des gens des services que je connais pour certains depuis vingt ou trente ans . -À d' autres étapes , des comédies musicales telles que " 1789 : les de la Bastille " et de petites productions telles que " Disco " ou " Life and Time " sont en train de diminuer avec le succès et la qualité enregistrés jusqu' à présent . Sur les autres scènes , les spectacles musicaux comme " 1789 " , " les amants de la Bastille " et les petites comédies musicales comme " Disco " ou " Life and Times " se mettent au diapason du succès et de la qualité . -Sur certains des camions , les sièges avant peuvent avoir un défaut dans le mécanisme . Sur certains , les sièges avant pouvaient présenter un défaut au niveau du mécanisme d' inclinaison . -À la Bourse de New York , les actions GM ont augmenté de 1,4 % à 16 : 35 GMT , tandis que les actions de Ford ont perdu 1,1 % . A la Bourse de New York , l' action GM gagnait 1,4 % à GMT tandis que Ford 1,1 % . -Sur le tapis , qui est aussi blanc , dans l' immense espace du Théâtre Maisonneuve , un corps , apparaît dans le profil , aussi mince qu' une ligne et habillé en noir . Sur le tapis , aussi blanc , dans l' immense espace qu' est le Théâtre Maisonneuve , s' avance , de profil , fin comme une ligne et vêtu de noir , un corps . -Le lendemain de mon échographie , quand j' ai découvert le genre de bébé que je , je me suis réveillé et je me suis de mon lit , et mon père m' attendait à l' atterrissage : il me sourit et je me réjouis de recevoir un petit garçon . La veille de mon échographie où je devais savoir le sexe de mon bébé , j' ai rêvé que je me , je me hors du lit , mon père m' attendait sur la palier et il me et se réjouissait que j' attende un petit garçon . -Le soir , vous avez été libre , vous êtes apparu sur le news sur 2 . Dès le soir de votre libération , vous étiez sur le plateau du 20h d ' " Antenne 2 " . -Sur le marché des changes , la spéculation sur un changement de la politique monétaire de la BCE est monnaie courante , comme l' a témoigné John Hardy , un stratège de Saxo Bank . Sur le marché des changes , les spéculations sur l' évolution de la politique monétaire de la BCE vont bon train , comme en témoigne John Hardy , stratège chez Saxo Bank . -Du côté gouvernemental , il s' agit de qui sont membres des forces armées du Président Bachar el Assad , sont des combattants et 187 sont des militants du Hezbollah libanais . Du côté du gouvernement , on estime que 29 954 membres des forces armées du président Bachar el-Assad ont trouvé la mort , dont 18 678 étaient des combattants des forces et 187 des militants du Hezbollah libanais . -Sur le terrain , à Beyrouth ? Sur place notamment , à Beyrouth . -Sur instruction du Bureau de la protection de la Constitution ( ) , l' agence allemande de renseignement , sa mission consistait à photographier le toit de l' avant-poste américain situé à moins de 5 km de la Banque centrale européenne et de la Bundesbank . Sur ordre de l' Office fédéral de protection de la Constitution ( ) , l' agence nationale de renseignements allemande , sa mission était de photographier le toit de l' avant-poste américain , situé à moins de 5 km de la Banque centrale européenne et de la Bundesbank . -D' un autre côté , les analystes disent que les cabines sont de plus en plus fréquentes , ce qui laisse à penser que de nombreux passagers ont voté pour la formule plus dense , qui peut aller de pair avec des tarifs moins chers . D' un autre côté , les analystes disent que les cabines configurées avec 10 sièges par rangée dans les 777 existants suggèrent que de nombreux passagers se prononcent en faveur d' une configuration plus dense , qui peut aller de pair avec des tarifs moins chers . -D' autre part , à la suite de l' annonce d' une inflation de 0,7 % dans l' ensemble de la zone euro , l' idée que la Banque centrale européenne ( BCE ) puisse assouplir sa politique monétaire s' est répandue parmi les acteurs du marché . A l' inverse , après l' annonce d' une inflation à 0,7 % pour l' ensemble de la zone euro , l' idée se répand parmi les acteurs du marché que la Banque centrale européenne ( BCE ) pourrait assouplir sa politique monétaire . -D' un autre côté , les écosystèmes plus proches sont favorables à l' isolement génétique , ce qui peut en fin de compte entraîner la création de nouvelles espèces . A l' inverse , les écosystèmes plus fermés favorisent l' isolement génétique , qui peut engendrer à terme la création de nouvelles espèces . -D' un autre côté , le chef de la puissante NSA , le général Keith Alexander , a nié que son service de renseignement ait capturé des dizaines de millions de communications de citoyens européens . Par contre , le patron de la puissante NSA , le général Keith Alexander , a lui démenti que son agence de renseignement ait capté des dizaines de millions de communications de citoyens européens . -Sur le marché des valeurs mobilières , Renault ( % ) a enregistré la plus forte baisse avec le CAC 40 , par l' avertissement de rendement émis par son partenaire , Nissan Motor , vendredi . Aux valeurs , Renault ( % ) enregistre le plus fort repli du CAC 40 , par l' avertissement sur résultat de son partenaire Nissan Motor vendredi . -Sur les murs de son bureau , il y a des fusils d' assaut AK-47 , des photographies érotiques et des photos de l' auteur avec des seigneurs de guerre africains . Aux murs de son bureau sont accrochés des fusils d' assaut AK-47 , des photos érotiques et des photos de l' auteur avec des seigneurs de guerre africains . -Une fois que la Cour l' aura déclarée recevable , elle se rendra à la deuxième étape , à savoir les audiences préliminaires . Une fois la recevabilité admise par la Cour , on procède à la deuxième étape , soit les auditions sur le fond . -Une mère du Maryland d' un nouveau-né a également déclaré à la Zone pour bébés qu' elle aimerait voir des bébés accoucher de sexe à la naissance . Dans le Maryland , la mère d' un nourrisson a également déclaré sur Baby Zone qu' elle préfèrerait qu' un sexe soit attribué aux bébés à leur naissance . -Un blessé dans Accident avec Road Driver sur Bruxelles Ring Route Un blessé grave dans un accident avec un conducteur fantôme sur le ring de Bruxelles -On pourrait penser que la ville de New York a été la naissance d' une comédie en Amérique : il y a près de cent ans , le Frank , qui a été le maître des cérémonies du Palais du cinéma , à Broadway , a commencé à raconter des plaisanteries directement à la foule , de manière . On pourrait dire que la ville de New York est le lieu où est né le en Amérique : il y a presque 100 ans , l' acteur de Frank , qui a tenu le rôle de maître de cérémonie au Palace Theatre , sur Broadway , commençait à raconter des blagues directement devant le public , sur le ton de la conversation . -Un jour , il est mort . Un jour , il est mort . -On a l' impression que , au lieu de chercher au travail de , le chorégraphe s' est mis à l' utiliser avec son . On a l' impression que la chorégraphe , au lieu d' aller à la rencontre de l' oeuvre de , s' est imposée , avec ses . -Un enfant sur 20 aux États-Unis souffre maintenant d' allergies alimentaires . Un enfant sur 20 aux États-Unis souffre aujourd' hui d' une allergie alimentaire . -Selon la BBC , une personne a déclaré ces années plus tard : " Je ne suis ni homme ni femme " . Selon la BBC , une personne aurait dit des années plus tard , « Je ne suis ni un homme ni une femme . » -Un homme a expliqué que la chirurgie qu' il avait reçue lui permettrait de reprendre son travail . Un homme a expliqué que l' opération gratuite qu' il avait subie pour soigner une hernie lui permettrait de travailler à nouveau . -L' un des plus grands succès du festival sera une interview : David Steinberg en relation avec Larry David . L' un des principaux facteurs d' attraction du festival cette année sera un entretien : David Steinberg discutant avec Larry David . -Un des tunnels de contrebande les plus sophistiqués entre les États-Unis et le Mexique a été découvert , de même que ses propres systèmes d' éclairage , de ventilation et de transport électrique . L' un des tunnels les plus sophistiqués servant pour le trafic de drogue entre les États-Unis et le Mexique , disposant de ses propres systèmes d' éclairage , de ventilation et de rails électriques , a été découvert . -Une chose que j' attends avec impatience chaque Halloween est les tendances . Ce que j' attends avec impatience à chaque fête d' Halloween , ce sont les tendances . -L' un était employé à la U.S. Transportation Security Administration ( TSA ) , et l' autre était le tireur . Il s' agirait d' un employé travaillant pour l' Administration pour la sécurité des transports des États-Unis ( TSA ) et de l' auteur des coups de feu . -Un an auparavant , elle avait voyagé avec ma famille à Cuba . Un an auparavant , elle était en voyage avec ma famille à Cuba . -Seulement 15 % de ceux qui perdent un enfant choisissent l' incinération . S' ils ont le malheur de perdre un enfant , ils ne sont que 15 % à choisir la crémation . -Ce n' est qu' à la fin , qui est négative , que les danseurs ont des lumières et deviennent des symboles blancs dans un théâtre sombre , qu' il touche à la magie . Seule la fin , en négatif , avec ses danseurs par une lumière et les signes blancs dans un théâtre soudainement noir , touche à la magie . -Une demi-douzaine seulement des conseillers principaux du président , y compris M. , l' ancien chef de la diplomatie Barack Obama , et les anciens conseillers de la Première Commission , David et David , savaient que le changement était à l' étude . Seule une demi-douzaine de proches conseillers du président – notamment , l' ancien chef de campagne d' Obama Jim et les anciens conseillers à la Maison-Blanche David et David – savaient que le changement était à l' étude . -Une partie seulement de son dos était visible , c' est vraiment triste " . On voyait seulement une partie de son dos , c' est tellement malheureux . -Seules quelques centaines de sympathisants se sont rassemblés pour protester contre le président du pays et , à la faveur de la fin de la présidence libyenne des dernières élections , le couvre-feu n' a été que peu de temps pour permettre à son cortège de voitures de traverser la rue très occupée reliant le centre-ville à ses banlieues sud . Seuls quelques centaines de supporteurs s' étaient rassemblés devant la Cour pour le nouveau président et , se du faste présidentiel des années Moubarak , la circulation n' a été que brièvement interrompue pour permettre le passage de son convoi dans les rues très fréquentées reliant le centre-ville aux banlieues sud . -Les opérations sont habituellement menées dans une école , une école municipale ou une maison ; après la procédure , les patients se rendent à la zone de récupération pour aller chercher la personne suivante . Les opérations sont en général pratiquées dans une école , une mairie ou une maison ; après l' intervention , le patient se lève et se rend dans la salle de réveil pour laisser sa place à la personne suivante . -Les opposants disent que les changements le moral de la police , mais non le crime , le gaspillage de l' argent et ne réglerait pas un problème plus vaste d' une force policière sous pression après avoir diminué de milliers de policiers au cours de la dernière décennie . Les opposants disent que ces changements le moral de la police sans faire baisser la criminalité , représenteraient un gaspillage d' argent et ne pas le problème plus large de forces de police sous pression après que les effectifs de policiers ont été fortement réduits au cours de la dernière décennie . -Parmi les options , on trouve les freins , un système de prestation de services qui permet de relier Works , le soutien logistique et le stockage des véhicules entre les événements . Parmi les options , citons notamment des freins améliorés , un service donnant accès aux mécaniciens Works , au soutien logistique et à l' entreposage du véhicule entre deux évènements . -Ou bien ils peuvent choisir de ne pas disposer d' un dispositif , en choisissant de payer un tarif fixe basé sur le nombre moyen de milles parcourus par les habitants de tous les États . Ils peuvent encore choisir de ne pas avoir d' appareil du tout , préférant à la place payer un forfait basé sur le nombre moyen de miles parcourus par tous les résidents de l' État . -Ou s' agit-il d' un incident involontaire ? Ou incident involontaire ? -Ou bien nous pouvons défendre nos propres intérêts économiques , c' est-à-dire parler à Berlin et à Bruxelles où il faut aller . Ou nous pouvons défendre nos propres intérêts économiques , ce qui inclut d' envoyer promener Berlin et Bruxelles . -Oracle à DM Les actionnaires d' Oracle par le salaire d' Ellison -Les organisations consacrées à la protection de la vie privée prennent note de cette fascination pour la biométrie , mais elles le sont . Les organisations dédiées à la protection de la vie privée confirment cet engouement pour la biométrie , mais s' en inquiètent . -Organisé par la section Unity de l' association , le concours devrait réunir les meilleurs patineurs de la région . Organisée par la section Unity de l' association , la compétition devrait regrouper les meilleurs de la région . -D' autres membres du comité ont dit qu' il n' y a que des cas anecdotiques de rapports de pilotes pour soutenir que les appareils peuvent interférer avec les systèmes d' aéronefs , et la plupart de ces rapports sont très vieux . D' autres membres du comité ont déclaré qu' il n' existe que des rapports anecdotiques émanant de pilotes qui indiquent que les appareils peuvent interférer avec les systèmes d' un avion , et que la plupart d ’ entre eux sont très anciens . -D' autres médecins participant à la conférence du PAA ont renforcé les complexités du sujet . D' autres médecins participant à la conférence de l' AAP ont insisté sur la complexité du sujet . -D' autres conclusions appuient cette conclusion , mais il y a beaucoup d' autres facteurs . D' autres découvertes soutiennent cette conclusion , mais il existe de nombreux autres facteurs influents . -D' autres hôtels à Montréal seront également convertis dans les prochains mois , comme le Crown Plaza , qui deviendra un foyer pour les personnes âgées . D' autres hôtels de Montréal seront aussi convertis dans les prochains mois , comme le Crown Plaza qui deviendra une résidence pour personnes âgées . -D' autres endroits espèrent vendre le concept à un public averti en faisant pousser les dispositifs plus , pas moins . D' autres États espèrent vendre le concept à des citoyens méfiants en permettant aux appareils d' en faire plus , plutôt que pas assez . -D' autres révélations ont révélé des documents dissimulés par que la FSA a contrôlé le téléphone de la chancelière allemande Angela Merkel et celui de jusqu' à 34 autres dirigeants du monde . D' autres révélations ont fait état de documents divulgués par selon lesquels la NSA avait intercepté des données et des communications émanant du téléphone portable de la chancelière allemande Angela Merkel et de ceux de 34 autres chefs d' État . -D' autres sont là pour vous faire rire . D' autres ont juste l' ambition de faire rire . -D' autres reviennent en arrière quinze ans , sous le prétexte que vous pouvez mettre tout ce que vous voulez quand Matt sur le projet de loi . D' autres font un bond de quinze ans en arrière sous prétexte qu' on peut servir à peu près ­ n' importe quoi quand on a Matt ­ en tête d' affiche . -D' autres peuvent recevoir des béquilles , une paire de lunettes aveugles ou des permis de naissance gratuits - documents exigés pour l' école , mais que de nombreuses familles pauvres ne peuvent tout simplement pas se permettre . D' autres peuvent recevoir des béquilles , une paire de lunettes collectée ou un certificat de naissance gratuit – document nécessaire pour aller à l' école , mais de nombreuses familles pauvres ne peuvent tout simplement pas se permettre de l' acheter . -Sinon , nous devrions envisager d' imiter la Norvège et de conserver notre indépendance économique . Sinon , nous devrions imiter la Norvège et conserver notre indépendance économique . -Ottawa veut forcer les fournisseurs de services de télévision par câble et par satellite à offrir aux clients l' option d' avoir accès à des services payants . Ottawa désire forcer les fournisseurs de télévision par câble et satellite à offrir aux clients la possibilité de payer les services à la carte . -Notre mère nous a quittés après une lutte acharnée contre le cancer . Notre mère nous a quittés après son combat épuisant contre le cancer . - , il s' installe dans des partis , des femmes et des médicaments . Sans limites , l' homme enchaîne les fêtes , les femmes , les drogues . -À l' extérieur du bâtiment , des manifestants protestaient contre le dirigeant iraquien . A l' extérieur du bâtiment , des manifestants protestaient contre le dirigeant irakien . -Au cours d' une période de huit mois entre janvier et août 2011 , il a cherché à obtenir des renseignements illégaux sur les arrestations de deux hommes au nom de Scarborough ainsi que d' un de ses associés . Sur une période de huit mois entre janvier et août 2011 , il a cherché à obtenir de manière illicite des informations sur l' arrestation de deux hommes pour le compte de Scarborough ainsi que l' un de ses associés . -Pendant cinq semaines , les approvisionnements ont augmenté de plus de 25 millions de barils . Sur cinq semaines , les approvisionnements ont augmenté de plus de 25 millions de barils . -Là , les gens ont abandonné leurs anciens dirigeants en Angleterre et n' ont plus à s' en préoccuper . Là-bas , des hommes qui avaient laissé leur Ancien Régime en Angleterre et n' avaient plus à s' en soucier . -Là , la révolution a été consensuelle , alors qu' elle a engendré des drames et des conflits . Là une révolution consensuelle quand ici elle a semé le drame et le confit . -Le PSG , qui est le plus haut de la hiérarchie de Monaco à la différence de l' objectif , a publié la grille de l' équipe pour le match , qui est le 12 janvier de la saison , sur laquelle le 32 n' a pas comparu . Le PSG , leader du championnat devant Monaco à la différence de buts , a diffusé la liste des joueurs retenu pour ce match de la 12e journée , sans qu' y figure l' attaquant de 32 ans . -Les Palestiniens et les Israéliens sur la frontière avec la bande de Gaza entre Palestiniens et Israéliens à la frontière de Gaza - s a été fait célèbre par son rôle dans sexy TV show . Les boucles blondes de Pamela sont devenues célèbres grâce à son rôle dans la série télévisée sexy Alerte à Malibu . -Pamela Anderson les griffes de ces superbes , de nouvelles coupes . Pamela Anderson coupe ses emblématiques boucles blondes et adopte une spectaculaire coupe . -Pamela Anderson est la dernière star à avoir les fans avec un nouveau . Pamela Anderson est la dernière célébrité à choquer ses fans avec une nouvelle coiffure spectaculaire . -Parents de Kids Select " Gender " Les parents d' enfants peuvent choisir « sexe indéterminé » -Les parents de la Géorgie qui sont morts dans un " accident " pensent que le fils a été assassiné . Les parents d' un adolescent de Géorgie , qui est décédé dans des circonstances insolites , pensent que leur fils a été assassiné -Les parents ne devraient pas hésiter à recevoir des conseils de leur pédiatre sur la façon de parler avec leur enfant . Les parents ne devraient pas hésiter à demander conseil à leur pédiatre pour savoir comment parler avec leur enfant . -A la veille de l' ouverture de la Coupe du monde de football en France , l' équipe de France s' est engagée en faveur d' un accord sur le changement climatique . Le Paris Saint-Germain recevra Lorient vendredi sans son atout maître , le Suédois , qui est blessé , annonce le club de Ligue 1 dans un communiqué . -Les passagers sont censés remettre leurs appareils en marche , l' avion s' vers la terre et ne les pas jusqu' à ce que l' avion soit sur le sol . Les passagers sont supposés éteindre leurs appareils lors de la descente de l' avion avant l' atterrissage et ne pas les rallumer jusqu' à ce que l' avion se soit posé . -Les passagers peuvent déjà faire cela pendant la majeure partie d' un vol , mais beaucoup de gens trouvent difficile d' être incapables d' accéder à leurs livres électroniques pendant les et les atterrissages . Les passagers peuvent déjà le faire pendant la plus grande partie du vol , mais de nombreuses personnes trouvent pénible de ne pas pouvoir accéder à leurs livres électroniques pendant le décollage et l' atterrissage . -Les passagers se plaignent souvent des frais de bagages et d' autres frais , mais les compagnies aériennes les aiment . Les passagers souvent à propos des frais de bagages et autres frais , mais les compagnies aériennes les adorent . -Les passagers qui essaient d' éviter ces frais sont autant qu' ils peuvent dans les bagages de cabine dissimulés dans les , ce qui signifie que ces bacs sont souvent hors de l' espace . Les passagers qui essayaient d' éviter ces frais mettaient autant de choses qu' ils pouvaient dans les bagages à main rangés dans les compartiments supérieurs , ainsi il n' y avait souvent plus de place dans ces compartiments . -Les passagers qui achètent leurs billets sur le site Internet de la compagnie n' auront pas à payer . Les passagers qui achètent leur billet sur le site de la compagnie aérienne n' auront pas à payer . -On leur demandera de changer leurs smartphones , leurs tablettes et d' autres appareils en mode avion . Les passagers devront mettre leurs smartphones , tablettes et autres appareils en mode avion . - , qui a passé près de dix ans en tant que l' un des plus d' Asie du Sud-Est , a été condamné hier à 20 ans de prison pour son rôle dans la construction des dispositifs explosifs utilisés lors de l' attentat . , qui a passé presque 10 ans en fuite en étant l' un des hommes les plus recherchés d' Asie du sud-est , a été hier condamné à une peine de 20 ans de prison pour son rôle dans la fabrication des engins explosifs utilisés lors des attentats . - est le dernier des de Bali à être soumis à la justice . est le dernier des de bombe de Bali à être traduit en justice . - peut encore interjeter appel de sa sentence . peut toutefois faire appel de sa condamnation . -Patrick , auteur du livre de référence " de " et conférencier au Théâtre du Châtelet , analyse les raisons du succès durement remporté . Patrick , auteur du livre de référence " Histoires de comédies musicales " et conférencier au Théâtre du Châtelet , analyse les raisons du succès si durement conquis . -Des personnes sont attaquées dans la rue et même à la maison , et les forces de sécurité sont souvent les cibles des attaques . Des hommes sont abattus en pleine rue ou même chez eux , et les forces de sécurité sont également la cible d' attaques fréquentes . -Les gens ne sont pas stupides , ils iront là où les meilleurs soins sont offerts " . Les gens ne sont pas fous , ils vont aller là où il y a la meilleure offre de soins . -Les gens paient plus directement dans ce qu' ils gagnent . Les gens paient plus directement pour les avantages qui leur sont procurés . -Les gens disent que " le prix de l' or est bon , la chaîne d' or de Eli et la récupérer le mois prochain . Les gens disent , « le prix de l' or est élevé , mettons en gage la chaîne en or de grand-mère et on reviendra la chercher le mois prochain » . -Les gens entendent le message , mais ne écoutez pas , et cela me choque . Ce message est entendu , mais pas écouté , et cela me choque . -Les gens sont . Les gens étaient bouche . -Peut-être la grande différence entre un auteur adulte auteur d' un adulte et un auteur adulte écrit pour un enfant est la nécessité d' un certain sens de l' espérance . Peut-être que la grande différence entre un auteur adulte écrivant pour les adultes et un auteur adulte écrivant pour les enfants est la nécessité d' offrir de l' espoir . -Perry a déclaré qu' elle avait le dîner avec le vétéran et que " elle m' a frappé quand j' avais 18 ans " . Perry a déclaré avoir avec le vétéran du rock et expliqué , « il m' a quand j' avais 18 ans » . -Perry a été l' un des chanteurs à faire un exposé sur la carrière des Rolling Stones cette année . Perry était l' une des chanteuses à faire une apparition lors de la tournée des Rolling Stones cette année . -Peter Hughes était au barreau de où un kamikaze a fouillé un sac à dos contenant des explosifs au moins 20 secondes plus tôt . Peter Hughes était au s Bar lorsqu' un kamikaze y a fait sauter un sac à dos chargé d' explosifs à peine 20 secondes plus tôt . -La phase 1 consistera en une nouvelle ligne ferroviaire à grande vitesse entre Londres et les Midlands de l' Ouest . La Phase 1 portera sur une nouvelle ligne ferroviaire à grande vitesse entre Londres et les West Midlands . -La police de Phuket a interrogé pendant deux jours avant qu' elle n' ait avoué la fabrication de l' article . La police de Phuket a interrogé pendant deux jours avant qu' elle avoue avoir inventé l' histoire de toutes pièces . -Les changements physiques ne signifient pas que la puberté est imminente Les changements physiques ne signifient pas que la puberté est imminente - : Au moins quatre en Pia : au moins quatre blessés dans une violente bagarre -Pierre s Two Cavendish Les deux France de Pierre Nora -André II a montré à la fois qu' il s' est montré très intéressé par la Nouvelle République et qu' il s' est montré prêt à aider ceux qui ont commencé à réparer le fossé créé par la Révolution en enseignant à tout le monde dans cet ancien pays que la France a ouvert la voie à la France moderne et en la dotant d' une version unifiée de son histoire . A cette IIIe République , moment central et créateur , Pierre Nora a montré beaucoup d' intérêt et même de tendresse : saluant ceux qui se sont alors employés à réparer la fracture révolutionnaire , en enseignant aux écoliers tout ce qui dans l' ancienne France préparait la France moderne et en leur proposant une version unifiée de leur histoire . -L' île de Pitcairn a pour objectif de créer la plus grande réserve marine dans le monde , le journal News Les Nouvelles de Tahiti a annoncé jeudi , suite à des nouvelles de Radio Australia . L' île de Pitcairn souhaite créer la plus grande réserve marine du monde , a annoncé jeudi le quotidien Les Nouvelles de Tahiti , reprenant une information de Radio Australie . -Pitcairn pour créer la réserve marine la plus grande du monde Pitcairn veut créer la plus grande réserve marine du monde -Les géants font souvent du commerce sur des questions techniques par la publicité dans la presse commerciale . Les géants de l' aéronautique échangent souvent des coups sur des questions techniques en faisant de la publicité dans la presse professionnelle . -Les appareils modifiés pour les systèmes Wi-Fi sont aussi plus résistants aux interférences . Les avions modifiés pour accueillir des systèmes Wi-Fi résistent également mieux aux interférences . -Veuillez contacter les services à la clientèle du bureau de poste où vous avez acheté vos billets pour toute autre assistance . Merci de contacter le service client à la billetterie où vous avez acheté vos billets pour toute aide complémentaire . -Le chef de la police , David , a réagi en nommant deux journalistes et un responsable des médias pour interrogatoire , bien que la citation ait été levée suite à un tollé . Le chef de la police , David , a réagi en convoquant deux journalistes et un responsable des médias pour les interroger , bien que la convocation ait été retirée suite à une levée de boucliers dans les médias . -Les policiers disent qu' ils ont une vidéo qui semble indiquer au maire Rob Ford de fumer une pipe de fumée . La police déclare qu' elle est en possession d' une vidéo semblant montrer le Maire Rob Ford en train de fumer du crack . -Les politiciens ne veulent pas augmenter l' impôt même un cent lorsque les prix du gaz sont élevés . Les hommes politiques sont réticents à augmenter la taxe , même d ' 1 cent , alors que les prix de l' essence sont élevés . -Les responsables politiques se sont dits préoccupés par le fait que l' UE manque de capacités en matière d' informatique et de TI et qu' elle devrait s' efforcer de réduire sa dépendance vis-à-vis des États-Unis . Les hommes politiques ont fait part de leur inquiétude à l' idée que l' UE ne dispose pas de capacités informatiques et Internet suffisantes et pensent qu' elle devrait s' efforcer de réduire sa dépendance vis-à-vis des États-Unis . -Pape François pour nommer les premiers cardinaux en février Le pape François nommera ses premiers cardinaux en février -Le pape François va créer de nouvelles cardinaux de l' Église catholique pour la première fois le 22 février , et le Vatican annoncé jeudi . Le pape François nommera de nouveaux cardinaux de l' Église catholique pour la première fois le 22 février , a annoncé le Vatican jeudi . -Préparation à la gestion d' une classe dans un contexte nord-américain et québécois . La préparation à gérer une classe dans un contexte nord-américain , québécois . -Le Président Armando s' est efforcé de faire fi de l' instabilité . Le président Armando a cherché à minimiser les inquiétudes concernant l' instabilité . -Les membres du conseil d' administration du président Barack Obama ont décidé de remplacer le vice-président Joe Biden par Hillary Clinton au sujet du billet de 2012 , selon le New York Times . Les conseillers les plus proches du président Barack Obama ont envisagé , dans le plus grand secret , de remplacer le vice-président Joe Biden par Hillary Clinton sur la liste de 2012 , selon le New York Times . -Le président Xi , qui a pris ses fonctions en mars dernier , a fait de la lutte contre la corruption une priorité nationale , estimant que ce phénomène constitue une menace à l' existence même du parti communiste . Le président Xi , qui a pris ses fonctions en mars dernier , a fait de la lutte contre la corruption une priorité nationale , estimant que le phénomène constituait une menace à l' du Parti communiste . -Les élections présidentielles et parlementaires sont prévues pour l' année prochaine . Les élections présidentielles et législatives sont prévues l' année prochaine . -La pression exercée sur la FAA ces dernières années a contribué à assouplir les restrictions à leur utilisation . La pression pesant sur la FAA s' est renforcée au cours des dernières années pour assouplir les restrictions concernant leur utilisation . -Les pourparlers précédents entre les parties ont fait peu pour calmer les tensions suscitées par une série d' affrontements cette année . Les précédentes discussions entre les parties n' ont pas réussi à apaiser les tensions alimentées par une série de conflits cette année . -Les vaccins antérieurs ont échoué parce qu' ils contenaient des anticorps . Les précédents vaccins contre le tabac se sont avérés inefficaces car ils contenaient des anticorps . -Fonctions non publiées récemment libérées Des inédits déjà en écoute -Les prix au centre-ville de Londres ont continué de représenter une croissance constante de 5,6 % , mais ils ont été par un " marché intérieur " florissant au sud de l' ouest , au nord ( 7,4 % ) et à l' est ( 6,5 % ) tous , selon la recherche de . Les prix dans le centre de Londres affichent une croissance régulière d' année en année de 5,6 % mais ont été par un « marché intérieur » en pleine croissance , le sud-ouest de la ville , le nord ( 7,4 % ) et l' est ( 6,5 % ) connaissant une reprise , selon les études de . -Le bien immobilier - les 5 à 10 % du marché du logement dans la fourchette - dans la grande banlieue londonienne du Sud-Ouest , qui s " étend de à Wimbledon , a augmenté de 11,8 % au cours de l' année écoulée . L' immobilier de prestige – les 5 à 10 % des prix les plus élevés du marché de l' immobilier – dans le secteur prospère situé au sud-ouest de Londres qui s' étire de à Wimbledon , a augmenté d' un pourcentage record de 11,8 % au cours de l' an dernier . -Avant son arrivée en Afghanistan , il était également allé en Libye , en Russie , au Liban et au Mali . Et avant l' Afghanistan , il était aussi reparti en Libye , en Russie , au Liban et au Mali . -Les spécialistes de la protection de la vie privée continuent de s' inquiéter . Les spécialistes de la protection de la vie privée , eux , continuent de s' inquiéter . -Le professeur , de l' Université de Trente , a déclaré : " Il est bien connu de l' homme que la gauche et le côté gauche du cerveau sont impliqués dans des stimuli qui évoquent des émotions positives ou négatives " . Le Professeur , un de l' université de Trente , a déclaré : « Tout le monde sait que , chez l' homme , le côté gauche et le côté droit du cerveau sont impliqués de manière différente dans les stimuli qui éveillent des émotions positives ou négatives . » -Le professeur a dit : " Nous avons présenté des chiens pour des films de chiens , soit une version naturelle et une silhouette pour enlever toutes les autres sortes de délires , et nous sommes en mesure de conseiller le mouvement de la queue et de présenter la queue plus à gauche ou à droite " . Le Professeur a expliqué : « Nous avons présenté à des chiens des films où figuraient d' autres chiens – soit une version naturaliste soit une silhouette pour éviter tout autre facteur de confusion – et nous avons pu le mouvement de la queue et le présenter comme allant plus à gauche ou à droite . » -Le professeur a dit qu' il ne croyait pas que les chiens spontanément entre eux au cours de ces mouvements . Le Professeur a dit qu' il ne pensait pas que les chiens communiquaient délibérément entre eux grâce à ces mouvements . -Le professeur Crystal a dit que si un vaccin à l' avenir était complètement sûr , il pourrait être mis à la disposition des enfants avant qu' ils ne soient tentés d' essayer une cigarette , empêchant ainsi la dépendance à la nicotine . Le Professeur Crystal a déclaré que si le futur vaccin à usage humain était complètement sûr , on pourrait l' administrer aux enfants avant qu' ils soient tentés d' essayer une cigarette , ce qui empêcherait toute addiction à la nicotine . - , Anne a déclaré : " Si quelqu' un doit savoir qu' il ne s' agit pas d' une infraction à la loi , il s' agit d' un avocat criminel . Le procureur Anne a déclaré : « Si quelqu' un doit savoir qu' il ne faut pas violer la loi , c' est bien un avocat . » -Le procureur Andrew a fait savoir au East Bailey que les nouvelles du monde avaient révélé , en mai 2005 , une mauvaise rumeur que M. Clarke voyait son " conseiller spécial " , Hannah . Le Procureur Andrew QC a déclaré devant le tribunal de l' Old Bailey que News of the World avait entendu une fausse rumeur en mai 2005 selon laquelle Clarke avait une relation avec Hannah , sa « jolie conseillère spéciale » . -Les procureurs ont demandé une peine de vie , même s' ils ont peut-être poussé l' homme appelé " Man " à sa réputation de maître condamné à mort . Les procureurs ont demandé une peine de réclusion à perpétuité , pourtant ils auraient pu demander à ce que l' homme , qualifié de « Man » pour sa réputation d' expert en fabrication de bombes , soit condamné à mort . -Les aspects psychosociaux sont également importants . Les aspects psychosociaux sont également importants . -La fonction publique porte sur le service public . L' exercice de fonctions publiques est une question de service public . - et ont déclaré vendredi qu' ils n' avaient pas reçu d' opposition des autorités américaines à leur projet de fusion , rapprochant ainsi la création de la plus grande agence de publicité du monde . et ont dit vendredi n' avoir reçu aucune objection de la part des autorités américaines à leur fusion , se rapprochant ainsi de la création de la première agence de publicité mondiale . -Qantas a déclaré qu' elle respecterait les règles actuelles pour le moment . Qantas a déclaré qu' elle respectait pour le moment les règles actuelles . -La a également déclaré qu' elle une provision de £ 4 millions pour l' amortissement des créances douteuses pour le trimestre , une inscription liée à la mise en place de la structure . a également dit qu' elle inscrirait une provision pour dépréciation des créances douteuses supplémentaire de quatre à 4,5 milliards de livres sur le trimestre , une écriture liée à la constitution de la structure de cantonnement . - a également précisé que cette restructuration interne 10 millions de livres de capital , renforçant ainsi sa capacité de prêt . a précisé aussi que cette restructuration interne de 10 à 11 milliards de livres de capital , renforçant ainsi sa capacité à prêter . - suspend deux vendeurs de devises suspend deux traders sur le marché des changes - in Cat in of Oise ( en anglais ) Un cas de rage détecté chez un chat dans le Oise -Il a plutôt dit que leurs amitiés avec le Seigneur leurs jugements à propos de leur rapport . Au lieu de ça , il a déclaré que les relations amicales qu' ils entretenaient avec Lord ont leur jugement , ce qui fait qu' ils n' ont pas signalés ses agissements . -Au contraire , l' euro est enfermé dans ses pays aux taux de change favorables aux exportateurs allemands - quelque chose que les politiciens allemands veulent garder . Le système européen a plutôt fait peser sur ses membres des taux de change favorables aux exportateurs allemands – ce que les hommes politiques allemands veulent conserver . -Au lieu d' être par le retrait du gouvernement américain , les investisseurs ont concentré leur attention sur ce qui est probablement plus important : la Réserve fédérale . Au lieu d' être par l' arrêt des activités gouvernementales aux États-Unis , les investisseurs sont restés concentrés sur ce qui est probablement le plus important : la Réserve fédérale . -L' agence de notation Standard & Poor' s a réduit sa cote de crédit pour le mois de mai 2009 , ce qui a mis en doute la capacité de l' ancienne république soviétique , qui est en récession depuis un an , de répondre à ses obligations financières . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; L' agence de notation Standard & Poor' s a abaissé vendredi la note de solvabilité de l' Ukraine , mettant en doute la capacité de l' soviétique , en récession depuis plus d' un an , à faire face à ses obligations financières . -Vraiment , ce gars devrait recevoir la peine de mort devant quelqu' un . En réalité , cet homme aurait dû être condamné à mort avant les autres . -Des analyses récentes ont permis de distinguer quatre espèces . Les récentes analyses ont permis de distinguer quatre espèces . -Les recommandations pour la thérapie médicamenteuse ou hormonale sont fondées sur l' âge de l' enfant , le taux de développement , le taux de croissance et la maturité affective . Les recommandations en matière de traitement médicamenteux et hormonal sont basées sur l' âge de l' enfant , le taux de développement , le taux de croissance et la maturité affective . -Les inscriptions auront lieu le samedi matin à partir de 9h et les étapes de qualification seront suivies peu après 10 h , la finale étant fixée à 15h et la à 16h . Les inscriptions sont prises sur place samedi matin à partir de 9 heures et les phases de qualification débuteront dans la foulée , à 10 heures , avant la finale prévue à 15 heures et la remise des prix à 16 heures . -L' autorité réglementaire sur les appels téléphoniques appartient à la Federal Communications Commission , et non à la FAA . Le pouvoir réglementaire concernant les téléphones portables appartient à la Federal Communications Commission et non à la FAA . -Les relations entre les États-Unis et l' Allemagne ont été à la suite des accusations selon lesquelles la avait manipulé le téléphone de la chancelière Angela Merkel . Les relations entre les États-Unis et l' Allemagne ont été mises à rude épreuve à la suite de plaintes selon lesquelles la NSA avait mis sur écoute le téléphone portable de la chancelière allemande Angela Merkel . -Renault Share price Through Tony Controls Renault en Bourse par l' avertissement de Nissan -Renault a annoncé la plus grosse chute sur l' indice 120 à Paris vendredi après son partenaire , Nissan Motors , a annoncé qu' elle avait réduit son avance de bénéfices nette de près de 20 % , le chiffre d' affaires des traders . Renault accuse vendredi la plus forte baisse de l' indice 120 à Paris après que son partenaire Nissan Motor a annoncé avoir diminué de près de 20 % sa prévision de bénéfice net annuel , rapportent des traders . -Rapport : Obama a estimé qu' il s' agissait d' une affaire de scandale pour Hillary Clinton Rapport : La campagne d' Obama considérée comme laissant tomber Biden pour Hillary Clinton -Les rapports de l' Australie faisaient savoir qu' entre-temps , elle faisait des séjours dans la zone de villégiature de Krabi , dans le sud de la Thaïlande . En Australie , la presse a déclaré qu' en attendant , elle était en vacances dans la région de Krabi , au sud de la Thaïlande . -Selon certaines informations , au moins quatre personnes ont été légèrement blessées . Quatre blessés légers au moins seraient à déplorer . -Des informations indiquent que l' agence de la sécurité nationale américaine sur la société mère brésilienne et a profité de l' accès à des données provenant des émetteurs américains de virus , y compris Google et Yahoo , qui ont la paranoïa des entreprises face à la surveillance de l' Etat à de nouveaux sommets . & # 160 ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; Des rapports , selon lesquels la National Security Agency américaine a la compagnie pétrolière brésilienne et a eu accès à des données détenues par des fournisseurs américains de services de stockage dans le Cloud , notamment Google et Yahoo , ont plus que jamais suscité la paranoïa des entreprises à l' égard de la surveillance d' État . -Les représentants de Samsung , de , de HTC et de Rockstar n' ont pas pu être retrouvés . Il n' a pas été possible de joindre les représentants de Samsung , , HTC et Rockstar . -La République a essayé de régler les problèmes financiers de la compagnie aérienne dans le cadre de la vente de la compagnie aérienne . Republic a tenté de redresser les finances de Frontier en vendant la compagnie aérienne . -L' républicain ne peut pas expliquer la possibilité d' écoutes de responsables étrangers , ni de drones qui tuent des enfants innocents à l' étranger . L' obstruction républicaine ne peut pas expliquer la mise sur écoute des dirigeants étrangers , ni les attaques d' enfants étrangers innocents par des drones . -Les chercheurs sont connus depuis longtemps pour cette population , mais ils ne savaient pas qu' ils faisaient partie d' une nouvelle espèce qui ne vit que dans cet endroit . Si cette population était déjà connue depuis longtemps par les chercheurs , ils ne savaient pas qu' elle faisait partie d' une nouvelle espèce , présente seulement à cet endroit . -Résultat : le Front de la Mer du , le Front national du vient d' être élu lors d' une élection partielle de haut niveau , amenant les dirigeants du parti socialiste à la troisième place . Résultat : le parti d' extrême droite de Marine Le Pen , le Front National , vient de remporter une élection partielle cruciale , renvoyant les socialistes au pouvoir à la troisième place . -Les résultats n' ont pas dépassé les prévisions et ont expédié des actions à plus de 8 % sur la Bourse de Toronto . Les résultats n' ont pas été à la hauteur des prévisions et le cours des actions a reculé de 8 % à la Bourse de Toronto . -Les recettes ont légèrement diminué , passant de 4,2 milliards de dollars à 4,1 milliards de dollars . Les revenus ont légèrement reculé de 4,2 à 4,1 Md $ . -Les revenus ont chuté de 9 % et sa capacité de vol a chuté de près de 13 % au premier trimestre , en raison des résultats financiers annoncés mercredi par la compagnie de chemin de fer Virgin Holdings Holdings Inc . Les recettes ont chuté de 9 % et sa capacité de vol a diminué de presque 13 % au cours du premier trimestre , selon les résultats financiers publiés mercredi par sa société mère , Republic Airways Holdings Inc . - . Le groupe de rock Arctic a reporté un concert à Glasgow après que l' on a diagnostiqué une à son chanteur . -Rockstar , le consortium qui a racheté les brevets de Nortel pour 4,5 milliards de dollars , a poursuivi Samsung Electronics Trading Ltd , LG Corporation , et quatre autres compagnies pour contrefaçon de brevet au tribunal de district des États-Unis au Texas . Rockstar , le consortium qui a acquis les brevets Nortel pour 4,5 Md $ , a déposé plainte à l' encontre de Samsung Electronics Co Ltd , HTC Corp , et quatre autres sociétés pour violation de brevet devant la Cour d' assises des États-Unis au Texas . -Rockstar est détenue conjointement par Apple , Microsoft , Blackberry , Ericsson et Sony . Rockstar est détenu conjointement par Apple , Microsoft , Blackberry , Ericsson et Sony . -Rockstar est à la recherche de dommages-intérêts supplémentaires contre Google , car il prétend que la violation par Google des brevets est intentionnelle , selon la plainte . Rockstar réclame des dommages et intérêts plus important à l' encontre de Google , car il prétend que la violation de brevets de Google est obstinée , selon le plaignant . -Rohan , directeur du commerce extérieur européen à , a pris la fuite cette semaine , tandis que Matt Gardiner , ancien grossiste à la Bourse de Barclays et UBS , a été suspendu par la Standard Chartered cette semaine . Rohan , responsable européen du trading de change au comptant chez , a été mis en congé cette semaine , tandis que Matt Gardiner , trader senior de devises chez Barclays et UBS , a été suspendu par Standard Chartered cette semaine . -Des tests systématiques ont révélé que les produits traités en Roumanie en janvier et vendus par des magasins de la chaîne Home et des chaînes de conservation de la qualité couvraient entre 1 et 5 % de l' ADN des chevaux . Des tests de routine ont révélé que les produits , élaborés en janvier en Roumanie et vendus par les chaînes Home et Quality Save , contenaient de l' ADN de cheval à hauteur de 1 à 5 % . - de la clôture des unités d' intervention d' urgence de l' autour de la fermeture des urgences de l' -La Royal Bank of Scotland a suspendu deux agences de change de sa division commerciale en raison de deux cas connus de la part d' un autre , le fait que l' ensemble de la chaîne des régulateurs dans la tentative de manipulation du marché des devises gagne rapidement de la vigueur . Royal Bank of Scotland a suspendu deux traders de sa division marché des changes selon deux sources proches du dossier , ce qui montre une nouvelle fois que l' enquête mondiale des régulateurs sur les suspicions de manipulations du marché des devises commence à porter ses fruits . -Ruby affirme qu' il n' est pas " stupide " que la police ait jamais cherché à fouiller le véhicule ou la maison de Bell , ou qu' elle ait pris contact avec son téléphone , déclarant que le chef de la police de Bill Blair a sciemment décidé de ne pas agir contre le maire . Me Ruby a qualifié d ' " inexplicable " le fait que la police n' ait jamais fouillé le véhicule de Rob Ford ou son domicile , ou mis sur écoute son téléphone , soutenant que le chef de police Bill Blair a choisi en connaissance de cause de ne pas agir contre le maire . -Il a été emmené chez les vétérinaires pour un examen de contrôle et il a été considéré comme étant d' ordre autre que l' épuisement et la déshydratation . Ruby a été conduite chez le vétérinaire pour un contrôle ; son état général était satisfaisant , mis à part l' épuisement et la déshydratation . -Des rumeurs concernant une affaire de Clarke ont été soulevées en premier lieu par les bureaux de la où une personne qui a été sexuellement intéressée par Mme a déclaré : " Ne perdez pas votre temps , elle est avec Charles " . Les rumeurs sur une liaison impliquant Clarke ont été reprises en premier par le bureau de la rédaction de lorsqu' une source , éprouvant une attirance sexuelle envers Mlle , a déclaré qu' on lui avait dit : « Ne perds pas ton temps avec elle , elle est avec Charles . » -La Russie , qui est responsable d' un quart des exportations ukrainiennes , a fait valoir que , si une zone de libre-échange avait été créée entre l' UE et Kiev , elle devrait renforcer ses contrôles des frontières pour les produits importés . La Russie , qui représente le quart des exportations ukrainiennes , a prévenu qu' en cas de création d' une zone de libre échange entre l' UE et Kiev , elle devrait renforcer ses contrôles à la frontière pour les marchandises importées . -Par ailleurs , la note verbale note " l' incertitude " liée à la signature éventuelle d' un accord d' association entre l' Ukraine et l' UE qui , pour Bruxelles , est subordonné à la libération du leader de l' opposition , . S & P relève en outre l ' " incertitude " liée à la possible signature d' un accord d' association entre l' Ukraine et l' UE , que Bruxelles conditionne à la libération de l' opposante Timochenko . -Les ventes des camions et Sierra , qui ont été révisées pour l' année modèle 2014 , ont augmenté d' environ 20 pour cent au cours des 10 premiers mois de l' année , GM a déclaré vendredi . Les ventes de et Sierra , qui ont été pour le modèle 2014 , ont augmenté d' environ 20 % au cours des 10 premiers mois de l' année , a déclaré GM vendredi . -Samsung , et HTC tous les téléphones qui fonctionnent sur le système d' exploitation Android de Google , qui rivalisent fortement avec les produits Apple et les produits mobiles de Microsoft . Samsung , et HTC fabriquent des téléphones qui fonctionnent sous le système d' exploitation Android de Google , qui livre une concurrence féroce aux produits mobiles Apple et Microsoft . -Dire beaucoup de mythes serait " trop sauvage , beaucoup trop scandaleux et , dans bien des cas , trop pour être enseigné dans les écoles " , préconise une " sélection soigneuse " d' activités adaptées à l' âge . que de nombreux mythes seraient « bien trop violents , bien trop scandaleux et dans certains cas , bien trop obscènes pour être enseignés à l' école » , a préconisé une « sélection minutieuse » des œuvres adaptées à l' âge . -Scarborough lui-même a été condamné à quatorze ans d' emprisonnement après avoir plaidé coupable de complot en vue de fournir de l' héroïne , de la cocaïne et du cannabis . Scarborough a lui-même été emprisonné pendant 14 ans après avoir plaidé coupable de complicité de trafic d' héroïne , de cocaïne et de cannabis . -Le transport scolaire : plainte Transport d' écolier : la plainte jugée recevable -Les écoles ont recommandé de se concentrer davantage sur les mathématiques , l' orthographe et la grammaire . Les écoles sont encouragées à se concentrer davantage sur les mathématiques , l' orthographe et la grammaire -Les scientifiques ont développé un vaccin qui pourrait prendre le plaisir de fumer une cigarette . Des scientifiques ont développé un vaccin qui pourrait supprimer le plaisir que l' on prend à fumer une cigarette . -Les scientifiques ont mis plus de lumière sur la façon dont les mouvements de la queue d' un chien sont liés à son humeur . Des scientifiques viennent de mettre en lumière la façon dont les mouvements de la queue d' un chien sont liés à son humeur . -Scotland Yard a " donné un avis de mise en garde à un homme de 40 ans qui lui a demandé d' assister à un poste de police à la date et à l' heure de notre choix " , a déclaré un porte-parole . Scotland Yard a « délivré un avis de remise à l' encontre d' un homme de 40 ans qui lui demande de se présenter à un commissariat de police , à la date et l' heure de son choix » , a déclaré un porte-parole . -Scotland Yard a déclaré qu' il avait reçu un " avis de mise en garde " sur l' autre de l' Australie qui l' avait obligé à comparaître à un poste de police , ajoutant que le fait de ne pas l' avoir fait pourrait l' obliger à s' arrêter . Scotland Yard a déclaré qu' elle avait délivré un « avis de remise » à l' encontre de l' Australien de 40 ans , lui demandant de se présenter à un commissariat de police , ajoutant que s' il ne le faisait pas , il serait susceptible d' être arrêté . -Scott Brown , capitaine de l' équipe celtique de Glasgow , a reçu son appel et n' a pas remporté les deux matchs de qualification de son prochain match de la Ligue , contre Ajax et Milan AC . Scott Brown , le capitaine du Celtic Glasgow , a vu son appel rejeté et sera bien suspendu pour les deux prochains matches de Ligue des champions de son club , contre l' Ajax et l' AC Milan . -Scott Brown Opposition L' appel de Scott Brown rejeté -Le secrétaire écossais a ajouté : " L' annonce d' aujourd' hui est une bonne nouvelle pour l' Écosse " . Le secrétaire d' État britannique pour l' Écosse , , a ajouté : « L' annonce d' aujourd' hui est une bonne nouvelle pour l' Écosse . » -Le ministre des Transports allemand , Keith Brown , a déclaré qu' il était " excité " à travailler avec le gouvernement britannique sur le plan . Le ministre écossais des Transports , Keith Brown s' est dit « excité » à l ’ idée de travailler avec le gouvernement britannique sur le projet . -Voyant son père et un grand nombre de ses concitoyens , est résolu à faire quelque chose . Après avoir vu son père et autant de ses compatriotes souffrir , le Dr était déterminé à essayer de changer le cours des choses . -Le sénateur Chris Murphy du Connecticut a déclaré qu' il s' était entretenu avec les membres du Parlement européen et les autres membres de cette semaine et s' inquiétait de leurs menaces d' arrêter de participer à des organisations antiterroristes en raison de la frustration suscitée par la surveillance de l' Agence nationale de sécurité . Le sénateur Chris Murphy du Connecticut a indiqué qu' il s' était exprimé devant les membres du Parlement européen et d' autres cette semaine et qu' il était inquiet des menaces qu' ils avaient proférées concernant un possible retrait des organisations anti-terroristes en raison du sentiment de frustration que suscite la surveillance de la National Security Agency . -Feu grave à la boutique Important incendie dans un commerce -Sept points ajoutés à sa licence lui ont valu une interdiction de conduire six mois . Les sept points retirés sur son permis ont conduit à un retrait de permis de 6 mois . -Plusieurs dizaines d' habitants , dont la majorité sont des descendants des mutins du Bounty , vivent encore sur Pitcairn , avec 95 % de leurs revenus selon la générosité de Londres . Quelques dizaines d' habitants , en majorité des descendants des du Bounty , habitent encore Pitcairn , et leurs revenus dépendent à 95 % de la générosité de Londres . -Plusieurs facteurs , comme la pénurie de pharmaciens dans les hôpitaux ou la prédominance des jeunes femmes dans la profession , font que des situations comme celles vécues en 2012 sont difficiles à prévoir . Plusieurs facteurs , comme la pénurie de pharmaciens en hôpital ou une prédominance de jeunes femmes dans le métier , rendent des situations telles que celle vécue en 2012 difficiles à prévoir . -Plusieurs suspects notoires ont récemment fait l' objet d' aveux à la télévision . Plusieurs suspects importants ont récemment fait des aveux télévisés . -Plusieurs des comédiens qui ont participé au festival cette année ont vu le jour par des canaux non traditionnels , comme des spectacles sur de petits réseaux , comme Central , Silver et Spike . Plusieurs des comédiens participant au festival de cette année sont passés par des circuits non traditionnels , en présentant par exemple des spectacles sur de petites chaînes de télévision , comme Central , FX et Spike . -Plusieurs associations de soins palliatifs ont profité de l' occasion pour réitérer leur désapprobation . C' est l' occasion pour plusieurs associations de soins palliatifs de réaffirmer leur désaccord . -Plusieurs États et villes sont néanmoins en train de faire marche arrière . Plusieurs États et grandes villes font néanmoins des progrès de leur côté . -Plusieurs commerçants ont indiqué que les actions de Renault avaient été touchées par l' avertissement de ce matin . Plusieurs traders ont rapporté que l' action Renault était affectée ce matin par le " warning " . -Partagez avec vous vos réflexions dans les commentaires ci-dessous . Dites-nous ce que vous en pensez dans les commentaires ci-dessous . -Les parts de Bombardier , qui ont également annoncé que le directeur général de Microsoft , Patrick , s' à ce poste , ont chuté de 8,5 % à $ CA lors de l' ouverture du journal le jeudi . Le cours des actions de Bombardier , qui a également annoncé que Patrick , chef de la direction financière de Google Inc , au conseil d' administration , a chuté de 8,5 % à dollars canadiens dans l' après-midi de jeudi . -Elle a ajouté : " C' était il y a longtemps , et c' est très gentil " . Elle a ajouté , « C' était il y a longtemps , et il a été très gentil . » -Elle a nommé un moniteur extérieur pour surveiller les changements importants , y compris les réformes politiques , la formation et la supervision , et elle a commandé un programme pilote pour tester les caméras dans certaines enceintes où la plupart des arrêts se sont produits . Elle a désigné un contrôleur extérieur pour superviser les modifications , notamment la réforme des politiques , de la formation et de la supervision , et elle a ordonné la mise en place d' un programme pilote pour tester des caméras portées sur le corps dans les quartiers où se déroulent le plus grand nombre de contrôles . -Elle est revenue de l' école à la peur et à l' agitation pour être le premier parmi ses amis . Elle est rentrée de l' école et bouleversée d' être la première parmi ses amies . -Elle n' a pas déplacé de muscle pendant le secours - elle a été gelée solide . Elle n' a absolument pas bougé pendant le sauvetage – elle était gelée . -Elle est décédée après avoir parlé au téléphone . Morte après m' avoir parlé au téléphone . -Elle a un vagin , mais pas d' utérus ou d' ovaires . Elle a un vagin , mais pas d' utérus ni d' ovaires . -Elle est la fille de l' ancien joueur de la ligue australienne Peter , qui a appelé le Ministère des affaires étrangères à Canberra à aider sa fille . Il s ’ agit de la fille de l' ancien joueur de la Ligue australienne de rugby Peter , qui a fait appel au ministère des Affaires étrangères de Canberra pour aider sa fille . -Elle a dit qu' elle était " heureuse de revenir " . Elle affirme être " heureuse d' être de retour " . -Elle veut utiliser les élections de l' année prochaine de l' UE pour créer un bloc à travers le Parlement européen . Elle veut se servir des élections européennes de l' année prochaine pour créer un bloc unique et au sein du Parlement européen . -Elle alors des biscuits . Elle a mangé des biscuits après . -Elle a été détenue dans des cellules de la police locale avant l' audience . Elle a été détenue dans une cellule du commissariat local avant l' audience devant le tribunal . -Elle était juste au milieu de la falaise - du haut et du bas . Elle était en plein milieu de la falaise , à 15 m du sommet et 15 m du sol . -Elle a été repérée trois jours plus tard par un coincé dans la carrière . Elle a été repérée trois jours plus tard par une personne qui son chien coincée dans la carrière -Si une mère est si sa fille commence à pousser des bourgeons du sein et des cheveux à 7 ou 8 ? Une mère doit-elle s' inquiéter si les seins et les poils de sa fille commencent à pousser à 7 ou 8 ans ? -Cette élection doit-elle être décidée deux mois après que nous nous sommes abstenus de voter ? Cette élection devrait-elle être décidée deux mois après que le vote est terminé ? -Silvio Berlusconi menace de retirer son soutien au gouvernement si le Sénat , dans lequel il a plus d' opposition que les partisans , décide de l' alléger de ses fonctions . Silvio Berlusconi menace de retirer son soutien au gouvernement si le Sénat , où il compte plus d' adversaires que de partisans , prononce sa destitution . -De même , ses funérailles étaient un adieu centré sur le peu de lumière des bougies autour de son cercueil . De même , ses funérailles furent un adieu glacial focalisé autour de la maigre lueur des chandelles qui son cercueil . -De même , au Royaume-Uni , les activités du centre de loisirs de vol ont bien fonctionné , tandis que les entreprises clientes ont dépensé moins . De même au Royaume-Uni , le secteur des voyages de loisirs de Flight Centre a enregistré de bons résultats tandis que les clients d' affaires ont dépensé moins . -Depuis 2008 , M. et son groupe de volontaires ont aidé près de 32,000 personnes . Depuis 2008 , lui et son groupe de bénévoles ont aidé près de 32 000 personnes . -Étant donné que l' Écosse n' est pas dans l' espace Schengen , la poursuite de la peine de mort serait une solution de rechange pour que Bruxelles puisse bénéficier de tout ce que l' on attend de l' écossais d' Ulster . & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; Puisque l' Écosse ne fait pas partie de l' espace Schengen pour le moment , un non-respect continu des règles serait une mince concession que Bruxelles pourrait offrir en contrepartie de tout ce qu' elle veut obtenir des Écossais . -Depuis son arrivée à Washington le mercredi , Nouri Al a rencontré plusieurs membres du gouvernement et des membres du Congrès . Depuis son arrivée mercredi à Washington , Nouri Al a multiplié les rencontres avec l' exécutif , et les élus du Congrès . -Depuis qu' il a pris sa retraite en juin 2007 au terme d' une dizaine d' années comme chef , M. Blair a évité beaucoup de discuter de la politique britannique , limitant la plupart de ses remarques aux affaires étrangères et son rôle d' envoyé du Quartet des de paix au Moyen-Orient . Depuis son départ en juin 2007 après une décennie à la tête du pays , M. Blair a largement évité de discuter de la politique britannique , limitant ses commentaires aux affaires étrangères et à son rôle d' envoyé spécial du Quatuor pour le processus de paix au Moyen-Orient . -Comme le Gouvernement fédéral reconnaît le mariage entre personnes du même sexe , " tous les conjoints de militaires ont droit à un titre de carte d' identité du Ministère de la défense et aux avantages connexes " , a déclaré le secrétaire à la défense à New York , dans un discours à l' League pour la lutte contre l' antisémitisme . Depuis que l' État fédéral a reconnu le mariage homosexuel , " tous les conjoints de militaires ont droit à une carte d' identité du département de la Défense et aux prestations qui viennent avec " , a rappelé le ministre à New York dans une allocution devant l' League de lutte contre l' antisémitisme . -Depuis le début de l' année scolaire , le Conseil scolaire de la région métropolitaine de Sherbrooke ( ) a demandé un montant de 150 $ par élève ( jusqu' à 300 $ par famille ) pour les étudiants qui utilisent le transport scolaire pour se rendre à deux adresses , un service que l' organisation peut offrir lorsqu' elle est en mesure de le faire . La Commission scolaire de la ( ) exige depuis le début de l' année scolaire des frais de 150 $ par élève ( jusqu' à un maximum de 300 $ par famille ) pour les élèves bénéficiant du transport scolaire à deux résidences , un service que l' organisation offre lorsqu' elle est en mesure de le faire . -Depuis la reprise des hostilités entre les FARDC et les rebelles le 25 octobre , les deux parties n' ont donné aucune indication sur les pertes . Depuis la reprise des combats entre FARDC et rebelles le 25 octobre , les deux belligérants n' ont donné aucun bilan des pertes en vies humaines . -Depuis la reprise des affrontements du 23 mars entre le ( ) et l' armée , n' a pas participé directement aux combats , mais elle fournit aux troupes du gouvernement un soutien fort en matière de renseignement , d' observation et de planification . Depuis la reprise , vendredi , des affrontements entre le Mouvement du 23 Mars ( ) et l' armée , la ne participe pas directement aux combats , mais elle fournit aux troupes gouvernementales un soutien déterminant en matière de renseignement , d' observation et de planification . -Depuis le début de 2013 , l' Agence des services frontaliers du Canada à l' aéroport de a procédé à 173 saisies de drogues , dont 10 saisies de cocaïne totalisant 44 kilogrammes . Depuis le début de 2013 , les agents des services frontaliers à l' aéroport ont effectué 173 saisies de drogues , dont 10 saisies de cocaïne pour un total de 44 kilogrammes . -Depuis le début des combats importants en Afghanistan , l' armée s' est efforcée de déterminer quelle latitude elle peut accorder aux soldats blessés qui veulent rester dans l' armée , mais qui ne sont pas capables de se battre . Depuis le début des combats majeurs en Afghanistan , l' armée peine à déterminer quelle latitude elle peut accorder aux soldats blessés qui veulent rester dans les rangs , mais qui ne sont pas aptes au combat . -Depuis lors , plusieurs centaines de ont été dans les collines agricoles de , et , près de et de la ville voisine de , à une altitude de près de 2.000 mètres . Depuis , quelques centaines d' irréductibles du , sont retranchés à près de 2.000 mètres d' altitude sur les collines agricoles de , et , proches de et de la localité voisine de -Depuis qu' ils n' ont pas pu prendre le soleil avec Bob ou Richard , Daniel et Benoît , John et ont choisi un film américain d' avant-garde , Oscar Isaac , âgé de 33 ans , et l' actrice américaine Justin Timberlake . À défaut de pouvoir embaucher Dylan ou le trio Peter , Paul and Mary , Joel et ont jeté leur dévolu sur l' étoile montante du cinéma américain , Oscar Isaac , 33 ans , mais aussi sur la star planétaire pop Justin Timberlake . -Le journaliste Lou Reed a été tué alors que ses camarades de travail , son épouse , Laurie Anderson , publiée dans une lettre publiée par le journal régional East Hampton , pour laquelle il était destiné aux résidents de la ville , la localité où le couple avait une résidence . Le chanteur Lou Reed est mort alors qu' il faisait des exercices de chi , a révélé jeudi son épouse Laurie Anderson dans une lettre publiée par le journal régional " East Hampton Star " et destinée aux habitants de la ville de Springs , où le couple avait une maison . - a fait la transparence de la collaboration intensive entre les services de renseignement américains et les entreprises . a rendu transparente la collaboration intensive entre les services de renseignement américains et les entreprises . - prêts à " coopérer " avec l' Allemagne à propos de la surveillance des États-Unis est prêt à « coopérer » avec l' Allemagne sur la question de la surveillance américaine -Ainsi , un voyage de 1.000 milles acheté auprès d' une agence de voyages en ligne 250 milles . Ainsi , un voyage Frontier 000 miles acheté auprès d' une agence de voyage en ligne permettrait de gagner 250 miles . -Donc , un , alors . Il s' agit donc d' une histoire qui n' en est pas une . -Donc encore , No Words , Friends of The Online Game , acteur Button jouait dans son casino favori en 1912 alors qu' il avait été acclamé par un jet d' American Airlines pour avoir refusé de voler le dispositif alors que l' avion était à la porte . Donc toujours pas de Words With Friends , le jeu en ligne style auquel jouait l' acteur Baldwin sur son smartphone en 2011 lorsqu' il a été expulsé à grand bruit d' un avion d' American Airlines après avoir refusé d' éteindre son appareil alors que l' appareil était garé à la porte . -Ainsi , lorsque la bouche réchauffe la protéine , elle augmentera le niveau de pH et la crème glacée se . Donc lorsque votre bouche réchauffe la protéine , le niveau de pH augmente et la glace se met à briller . -Donc chaque me coûte environ £ 140 . Chaque boule me coûte donc environ 140 £ . -Peu après sa naissance en 1976 , le corps et les testicules de l' ange furent enlevés par chirurgie et il fut élevé en tant que fille . Donc peu de temps après sa naissance en 1976 , Bruce a subi une ablation chirurgicale de l' organe externe et des testicules et a été élevé comme une fille . -Ainsi , nous pourrons combler les lacunes que nous avons connues l' an dernier " . Cela viendra donc combler nos départs de l' année passée . -Donc cette année , le l' a fait . Cette année , les se sont donc . -Nous sommes donc très heureux de travailler en partenariat avec le gouvernement britannique en vue d' étudier les options pour le transport ferroviaire à grande vitesse en Écosse , afin de créer des avantages pour tous et de compléter la ligne que le gouvernement écossais met déjà en place . « Nous sommes donc excités à l' idée de travailler en partenariat avec le gouvernement britannique afin d' examiner les options visant à amener le réseau de train à grande vitesse jusqu' en Écosse , créant un avantage pour tous et complétant la ligne que le gouvernement écossais est déjà en train de planifier . -Alors , que faire si vous avez été capable d' intercepter cela et de l' arrêter avant que cela ne se produise ? Et si vous pouviez intercepter leurs communications et les arrêter avant qu' ils n' agissent ? -Les procureurs doivent respecter les normes les plus rigoureuses en matière d' intégrité et donner confiance en la confiance du public . Les avocats devraient observer les règles d' intégrité les plus strictes et instaurer un climat de confiance auprès du public . -Quelque 10.000 candidats se sont présentés aux auditions pour présenter le jury pour cette 10e saison : le , le Sinclair , le André et Olivier Bas . Quelque 10 000 candidats se sont présentés aux auditions de cette 10e saison , pour affronter le jury : , le méchant Sinclair , le perché André et Olivier Bas . -Certains Français nous ont emmenés à l' Ambassade de France . Des Français nous amènent à l' ambassade de France . -Certains banquiers ont essayé de s' emparer de l' affaire en disant que le vaste marché des changes liquide et presque inexistant est pratiquement impossible à manipuler , mais les traders de haut niveau disent que cela n' est pas nécessairement vrai . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Certains banquiers ont essayé de minimiser l' affaire en disant qu' il est presque impossible de manipuler le vaste marché des changes fortement liquide , mais les traders seniors expliquent que cela n' est pas nécessairement vrai . -Certains commentateurs ont interprété son recours à la force comme la tentative d' un mouvement en difficulté pour gagner des concessions et des profits financiers du gouvernement . Certains commentateurs ont interprété son utilisation de la force comme la tentative d' un mouvement malade d' obtenir des concessions et un trésor de guerre de la part du gouvernement . -Certaines de ses communications ont sans aucun doute été légitimes car il s' agissait de son avocat . Certaines de ses communications seront sans aucun doute des communications légitimes car c' était son avocat . -Certains des plus grands tunnels ont été découverts après la récolte de cannabis du Nicaragua en juillet dernier , qui présente des cartels de la drogue avec un défi de faire passer rapidement leur production aux consommateurs . Certains des plus grands tunnels ont été découverts suite à la récolte de marijuana dans le centre du Mexique en octobre , posant ainsi un défi pour les cartels de la drogue qui doivent rapidement livrer leurs produits aux consommateurs . -Certaines des plus grandes banques du monde , comme UBS , Barclays , la Deutsche Bank et , ont confirmé leur volonté de coopérer avec les régulateurs dans les enquêtes sur le plus grand marché financier au monde , où $ les mains chaque jour . Certaines des plus grandes banques au monde , notamment UBS , Barclays , Deutsche Bank et , ont confirmé qu' elles coopéraient avec les régulateurs dans le cadre des enquêtes menées sur le marché financier le plus important au monde , où 5,3 billions de dollars US changent de main chaque jour . -Certaines personnes peuvent mourir parce qu' elles attendent . Des gens peuvent mourir parce qu' ils attendent trop longtemps . -Certaines études indiquent que plus d' un tiers des passagers oublient ou ignorent les instructions pour éteindre leurs appareils . Certaines études indiquent qu' au moins un tiers des passagers oublient d' éteindre leurs appareils , ou le font délibérément . -Certaines choses me dérangent , d' autres moins . Certaines choses m' , d' autres moins . -Certains planificateurs de transport , cependant , s' interrogent sur le fait que tout ce que l' on peut dire au juste le mille , c' est simplement une distraction géante . Toutefois , certains planificateurs des transports se demandent si tout le débat sur le paiement au mile n ’ est pas simplement une énorme diversion . -Certains utilisent de jolies projections d' images et une vraie histoire , comme " 1789 : les de la Bastille " . Certains utilisent de ­ belles projections d' images et une ­ véritable histoire comme dans 1789 , les de la Bastille . -Quelqu' un du parlement allemand nous dit qu' il faut construire un Allemand Google . Un député du parlement allemand dit que nous devrions créer un Google allemand . -Parfois , certains rêves me touchent différemment et ont une impression sur mon esprit ; en fait , ils continuent de vivre et de faire partie de ma vie . Parfois , certains rêves me portent au contraire et s' inscrivent dans mon histoire et effectivement , ils continuent à vivre et à partager ma vie . -Parfois , mes croyances sont différentes de lui , alors je me demande toujours si ce que je fais est conforme à sa façon de voir les choses . Si j' ai mes convictions parfois différentes des siennes , je me demande toujours si mes agissements sont conformes à sa vision des choses . -Parfois , ils ont tout simplement besoin de l' argent très rapidement . Parfois , ils ont juste besoin d' argent très rapidement . -Sony a chuté de plus de 11 % à yens . Sony a chuté de plus de 11 % à 1 668 yens . -Peu après la descente , Scarborough était en contact régulier avec qui était au dîner dans le stade de curling Red , à Park . Peu de temps après le dépôt , Scarborough a été régulièrement en contact téléphonique avec qui était à un dîner au stade de foot d' Park , enceinte du club des Blackburn . -Sorensen , ancien responsable de la société Midwest Airlines , a récemment pris l' avion et s' est demandé ce qu' il faisait à la porte , car les passagers avaient les frais de bagages inhabituels de l' équipage . Sorensen , un ancien cadre de Midwest Airlines , a récemment voyagé sur Spirit et s' est demandé ce qu' il allait trouver à la porte d' embarquement au moment où les passagers les frais de bagages à main inhabituels imposés par la compagnie . -South Stream est destiné à ajouter de la diversité aux routes d' exportation de la Russie via l' Europe . South Stream devrait permettre de diversifier les voies d' exportation de la Russie vers l' Europe . -L' espace , par comparaison , a été beaucoup plus solennel . Space , à côté , était beaucoup plus grave . -Prenant la parole lors de l' été dernier , Michael , secrétaire de l' éducation , a déclaré qu' il y avait un consensus général selon lequel nous devions réformer notre système d' examen afin de rétablir la confiance du public , ajoutant que " l' " serait plus difficile , plus ambitieux et plus rigoureux . S' exprimant au cours de l' été , Michael , ministre de l' Éducation , a déclaré qu' il y avait un « large consensus quant au besoin de réformer notre système d' examens pour restaurer la confiance du public » , en insistant sur le fait que les seraient « plus stimulants , plus ambitieux et plus rigoureux » . -S' exprimant au nom d' une conférence publique ouverte à Londres en , M. a dit : " Je suis certain que le président et le gouvernement d' un autre pays ont connu des choses qui se sont toutes déroulées dans le cadre d' un pilote automatique , parce que cela a eu lieu à la fin de la Seconde Guerre mondiale et dans les années difficiles de la guerre froide , et bien sûr , 9 / 11 " . S' exprimant lors d' une conférence intergouvernementale à Londres par liaison vidéo , M. Kerry a déclaré : « Il est indéniable que le Président , moi-même et d' autres membres du gouvernement avons pris connaissance de certaines choses en mode pilote automatique parce que nous en avions la possibilité , dès la Seconde guerre mondiale et jusqu' aux années difficiles de la Guerre froide , puis bien sûr le 11 septembre . » -Richard , spécialiste en sauvetage technique , a déclaré : " C' est dans une situation précaire . Le spécialiste des secours techniques Andy Clayton a déclaré : « Elle était dans une situation précaire . » -Le spécialiste des services techniques de sauvetage Peter Lau a déclaré : " Ruby a connu une très grande chance d' évasion " . Le spécialiste des secours techniques Peter Lau a déclaré : « Ruby a eu beaucoup de chance de s' en sortir indemne . » -Une compagnie de défense navale spécialisée a confirmé que ses équipes , avec , ont été envoyées au département de et travaillent actuellement sur le porte-avions . De son côté , l' entreprise , spécialiste du naval de défense , assure que ses équipes et celles d' , dépêchées dans le Var pour l' occasion , " interviennent actuellement sur le porte-avions " . - Jump Plus à Bogota saut en " " au-dessus de Bogota -Spirit Airlines Inc. a commencé les premiers frais de transport de passagers il y a trois ans , et Air l' a suivi plus tard . Spirit Airlines Inc. avait appliqué les premiers frais de bagages à main il y a trois ans , et la compagnie low-cost a suivi un peu plus tard . - a sauté d' un hélicoptère au-dessus de Bogotá , la capitale de la Colombie , jeudi . Le sportif a sauté jeudi d' un hélicoptère au-dessus de Bogota , la capitale colombienne . -La ressemblance de avec sa soeur aînée est frappante : une , une couleur et . La ressemblance de Martin avec son aînée est frappante : une silhouette de , un teint et des origines . - Martin n' est pas fait comme le sujet des conversations . Martin n' a pas fini de faire parler d' elle . -Le personnel a établi des preuves qu' il n' y avait pas de violations , dont le Seigneur , seul , avec des enfants , en les en privé , en les tenant sur le genoux , en faisant savoir qu' il aimait et en leur laissant jouer avec son téléphone portable . Les membres du personnel ont témoigné qu' ils avaient observé certains manquements , notamment le fait que Lord restait seul avec les enfants , qu' il les gardait en privé , les faisait asseoir sur ses genoux , leur disait qu' il les aimait et les laissait jouer avec son téléphone portable . -Stage Animation variété de comédies musicales anglaises en les adaptant au français . Stage ­ Entertainment des comédies musicales de variété en les adaptant en français . -Les organes législatifs de l' État ont récemment mis à jour les règles pour permettre aux écoles d' accroître l' appât du gain . Les assemblées législatives des États ont récemment actualisé les règlements pour permettre aux écoles d' avoir plus facilement de l' en stock . -Steinberg a débuté en tant qu' humoriste , mais est devenu un réalisateur de télévision et de cinéma , ainsi qu' un historien de l' humour non officiel . Steinberg a commencé en tant que comédien de , mais il est devenu un réalisateur de cinéma et de télévision accompli , ainsi qu' un historien officieux de la comédie . - . Rochelle a d' abord été jugée coupable par un Tribunal de Province de Phuket le 15 juin de fausses déclarations après avoir dit à la police thaïlandaise qu' un chauffeur de taxi local , accompagné de deux autres hommes qui la , l' avait agressée tôt le dimanche 10 juin . -Néanmoins , les analystes sont sceptiques : le premier client peut commencer à exploiter un avion 12 mois après son premier vol . Pourtant , les analystes sont sceptiques quant au fait que le premier client puisse faire voler un avion seulement 12 mois après son vol inaugural . -Pourtant , comme la note d' Obama a chuté en dessous de 45 % cette semaine , le retour en 2008 par ce livre est devenu beaucoup plus difficile . Pourtant , alors que l' indice d' opinions favorables a plongé sous les 45 % cette semaine , il est encore plus difficile de revenir à l' année 2008 à travers ce livre . -Néanmoins , la portée et la sophistication technique des agences américaines d' espionnage mises en lumière par les révélations sont devenues un choc pour certaines sociétés qui avaient pensé que le plus grand risque de surveillance était la Chine . & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; Pourtant , l' ampleur et la sophistication des agences d' espionnage américaines exposées par les révélations de ont été un choc pour certaines entreprises qui pensaient jusqu' alors que la Chine représentait le plus gros risque en termes de surveillance . -Des études montrent que les écoles anglaises consacrent moins de temps aux maths - 116 heures par an ou trois heures par semaine au temps - que dans la plupart des pays . Des études montrent que les écoles anglaises consacrent moins de temps aux maths ( 116 heures par an , soit 3 heures par semaine pendant l' année scolaire ) que la plupart des autres pays . -Une controverse de plus en plus répandue à propos de l' agence a conduit à la conclusion précoce que l' incident de ce soir était le résultat d' une ciblée . Cette controverse croissante autour de l' agence a provoqué beaucoup de spéculations selon lesquelles l' incident de ce soir était le résultat d' une ciblée . -Il s' agit de la croissance de l' entreprise , qui le de et de deux autres à Singapour , a l' intention de travailler non seulement dans la Malaisie voisine - où il y a quatre magasins , mais aussi à l' extérieur de l' Asie , dit M. Hein Jin Lee , directeur exécutif de la société ( ) , de , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , etc. La hausse est telle dans le secteur du prêt sur gage que , opérateur de la boutique à et de 15 autres boutiques comme celle-ci à Singapour , prévoit de se développer non seulement dans la Malaisie voisine – où il compte déjà quatre boutiques – mais également en dehors de l' Asie , déclare Lee Ching , sa directrice générale . -Une telle perte de patrimoine génétique serait beaucoup plus problématique . Cette perte de patrimoine génétique serait beaucoup plus problématique . -Ces personnes ne comprendraient pas si quelque chose s' est mal passé . Ces gens-là ne comprendraient pas qu' il y ait des erreurs . -C' est le cas de Jim Bruce , un écrivain de 36 ans du Montana , qui est né avec des chromosomes mâles , mais qui sont sujets à des ramifications génitales . Tel est le cas de Jim Bruce , un écrivain de 36 ans du Montana , qui est né avec des chromosomes mâles mais des organes sexuels ambigus . -Les partisans de la modification du programme de indiquent que les changements mettront fin aux pratiques déloyales , une police de police plus efficace et plus efficace et pourraient influer sur la façon dont les autres services de police utilisent la police . Les partisans prônant des changements dans le programme de contrôle et de fouille du disent qu' ils mettront fin aux pratiques injustes , permettront d ’ avoir des forces de police plus efficaces et auxquelles on fait confiance , et pourront changer la façon dont les départements de police usent de la politique . -La Cour suprême confirme le droit à la santé d' Obama La Cour suprême valide la loi de réforme du système de santé d' Obama -La chirurgie le rendait stérile . La chirurgie l' a rendu stérile . -Les victimes et les proches des 202 personnes tuées lors de l' attentat à la bombe de 2002 à Seattle ont réagi avec colère face à la dernière condamnation de ceux qui ont répondu à l' appel de la justice , étant donné que Umar doit être considéré comme un peloton d' exécution . & # 160 ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; Les survivants et les proches des 202 personnes tuées dans les attentats de Bali en 2002 ont réagi avec colère à la peine prononcée à l' encontre du dernier des conspirateurs à être traduit en justice , disant qu' Umar devrait être par un peloton d' exécution . -La Syrie a détruit des équipements essentiels pour la production d' armes chimiques et de munitions explosives toxiques , et le Service de lutte contre les armes chimiques a déclaré jeudi des affrontements acharnés dans le nord du pays , près de l' un des endroits où des agents toxiques semblent être stockés . La Syrie a détruit les équipements essentiels à la production d' armes chimiques et de munitions remplies de gaz toxique , a déclaré l' organisme international de surveillance des armes chimiques jeudi , alors que de violents affrontements faisaient rage dans le nord du pays , à proximité de l' un des sites où des agents toxiques sont censés être stockés . -La Syrie a détruit son potentiel en matière d' armes chimiques , a déclaré le Groupe de surveillance . La Syrie a détruit sa capacité de production d' armes chimiques , déclare l' organisme de surveillance -La Syrie a présenté un plan pour la destruction totale de ses armes chimiques qui doivent être approuvées le mois prochain par le Comité exécutif de l' OIAC . La Syrie a soumis un plan de destruction totale de ses armes chimiques qui a été approuvé le mois suivant par le Comité exécutif de l' OIAC . -Profitant de l' Halloween , une femme a tenté de faire sauter deux kilos de cocaïne dans le pays à l' aéroport de jeudi matin . Profitant de la journée de l' Halloween , une femme a tenté de faire entrer au pays deux kilos de cocaïne dissimulés dans des , jeudi matin , à l' aéroport . -Devant la scène vers les hommes et les femmes dans des chapeaux intelligents , il son capuchon plat à un angle et deux enfants de deux ans autour de l' anneau . Se présentant contre des hommes et des femmes en chapeaux melons élégants , il portait sa casquette plate un peu sur le côté de manière décontractée pour faire défiler son âgé de 2 ans autour de la piste . -Tales of Thor montre que " la force brute ne peut pas être comparée à une subtile " , tandis que les légendes révèlent l' importance d' avoir un rêve . La Légende de Thor pourrait montrer que « la force brutale ne fait pas le poids face à la ruse subtile » , tandis que les légendes révèlent l' importance d' avoir un rêve . -Parlant sous le couvert de l' anonymat , le fonctionnaire ne précise pas si cette surveillance a eu lieu dans le passé . S' exprimant sous le sceau de l' anonymat , ce responsable n' a pas précisé si une telle surveillance avait été mise en place par le passé . -Taxe sur les propriétaires étrangers à abattre la bulle de Londres Un impôt sur les propriétaires immobiliers étrangers pour faire éclater la bulle de Londres -Les adolescents ont la possibilité de se familiariser avec le chant classique à travers cinq petits spectacles créés et réalisés par des artistes régionaux - des spectacles qui contiennent une grande dose d' humour . À travers cinq imaginées et interprétées par des artistes de la région , des vignettes comportant une forte dose d' humour , on permet à des adolescents d' apprivoiser le chant classique . -Les locataires doivent se présenter , car chacun paie en moyenne 100 $ par mois pour le coût de son logement qui va à la ville en taxes . Les locataires doivent aller voter , car chacun d' entre eux paient en moyenne 100 $ par mois sur le prix du logement qui va en taxes à la Ville . -Les tests effectués par l' Institut Pasteur sur un patient soupçonné d' être infecté par le coronavirus se sont révélés négatifs , le ministère de la Santé l' a annoncé . Les tests pratiqués par l' Institut Pasteur sur un patient soupçonné d' être infecté par le coronavirus se sont révélés négatifs , a annoncé le ministère de la santé . -Grâce à l' amélioration de la précision des capteurs et de la capacité de calcul des ordinateurs utilisés pour analyser ces données , ces techniques sont devenues pratiques . Grâce à l' augmentation de la précision des capteurs et de la capacité de calcul des ordinateurs nécessaires à l' analyse des données , ces techniques deviennent opérationnelles . -C' est beaucoup de . Ça fait beaucoup de chiens curieux déguisés en . -Il s' agit d' environ 9 milliards de graines de maïs . Cela représente environ 9 milliards de grains de maïs . -C' est environ 600 millions de livres de barres d' Hershey , , Milk , et Clark Bars . Cela représente environ 272 millions de kg de barres Hershey , , Milk , et barres Clark . -C' est une excellente nouvelle pour les 41 millions de à prendre en charge nos quartiers , selon le Département du Commerce des États-Unis . C' est une super nouvelle pour les 41 millions d' enfants qui vont à la chasse aux bonbons dans leur voisinage , selon le département américain du Commerce . -C' est ma principale lutte . C' est mon principal combat . -Ce n' est pas quelque chose que vous pourriez faire en Amérique comme étudiant en médecine de deuxième classe . Ce n' est pas quelque chose que vous avez l' occasion de faire en Amérique lorsque vous êtes étudiant en médecine en deuxième année . -L' espionnage économique n' est pas une surprise . Il n' est pas surprenant qu' il y ait de l' espionnage économique . -Cela a permis à toutes les compagnies aériennes d' inciter les passagers à acheter directement des billets plutôt que de passer par une agence de voyage en ligne . Cela a incité toutes les compagnies aériennes à encourager les passagers à acheter directement auprès d' eux au lieu de s' adresser à une agence de voyage en ligne . -C' est un inconvénient de coût , comme nous l' avons déjà vu à . Il s' agit d' un inconvénient en termes de coûts , comme nous l' avons déjà vu à . -Ce n' est pas un combat que les jeunes enfants devraient prendre à ce stade . Ce n' est pas une bataille que les jeunes enfants devraient mener à cet . -C' est ce qui m' inquiète le plus . C' est ce qui me dérange le plus . -Cela a duré longtemps , puis un jour il est parti . Ça a duré longtemps puis un jour il est parti . -Cela signifie qu' un passager en ligne à la porte de la frontière pourrait se procurer un sac gratuitement , alors que la personne à la ligne suivante pourrait payer 100 $ pour un même sac . Ainsi , un passager faisant la queue à une porte d' embarquement Frontier pourrait transporter un sac gratuitement , tandis que la personne suivante dans la queue pourrait devoir payer 100 $ pour un sac semblable . -Cette personne recevra 250,000 $ pour poursuivre son travail extraordinaire . Cette personne recevra 250 000 $ pour poursuivre son travail extraordinaire . -Cela suggère que la reprise économique de la Chine pourrait se poursuivre après que la croissance s' est redressée pour revenir à 7,8 % au troisième trimestre , à partir d' un creux à la baisse au cours du premier trimestre de l' année précédente . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Cela laisse supposer que la reprise économique de la Chine pourrait continuer à se raffermir après que la croissance a rebondi à 7,8 % au troisième trimestre , après une baisse que l' on n' avait pas connue depuis deux décennies au trimestre précédent . -C' est quand j' ai fait mon premier appel à mes parents , à ma famille , à mes amis , à la presse , etc. C' est le temps des premiers coups de fil aux parents , à la famille , aux amis , à la presse ... -Cela se produira tout au long du programme de vol. Cela se reproduira tout au long du programme de vol. -Le projet de 325 millions de dollars comprend une subvention d' urgence de millions de dollars . Le plan de 325 M $ comprend une campagne de 31 M $ pour les opérations chirurgicales non urgentes . -L' ordre de rectification est venu de l' Administration de la presse et de la publication , de la radio , du cinéma et de la télévision de Guangdong . L' ordre de « remaniement » émane de l' administration de pour la Presse et l' Édition , la Radio , le Film et la Télévision . -La Cour d' appel du deuxième circuit des États-Unis a déclaré que les décisions du juge seront maintenues en attendant le résultat d' un appel de la ville . La cour d' appel des États-Unis ( deuxième circuit ) a indiqué que la décision de la juge était suspendue jusqu' à l' issue de l' appel interjeté par la ville . - . Mardi , ce patient de 43 ans avait été soupçonné d' être atteint , après être rentré d' Arabie saoudite , un pays où la maladie a déjà fait une cinquantaine de morts . -L' homme de 45 ans a été reconnu coupable de meurtre en masse pour l' attaque de deux boîtes de nuit dans le quartier touristique de , qui a fait 202 morts , dont 88 Australiens , et qui a fait plus de blessés . L' homme de 45 ans a été reconnu coupable de tuerie de masse pour les attentats visant deux boîtes de nuit dans le quartier touristique populaire de , qui ont causé la mort de 202 personnes , dont 88 Australiens , et ont fait de nombreux blessés . -Le New Express a fait un rare appel public à la libération du journaliste Chen . Le New Express basé à Guangzhou a publié un rare appel public pour la libération du journaliste Chen . -Le conseil scolaire de a créé un centre de recherche qui fournira des outils aux enseignants qui , eux-mêmes , viennent d' ailleurs d' ailleurs . La commission scolaire a créé un centre de recherche qui donnera des outils aux professeurs qui , eux aussi parfois , viennent d' ailleurs . -L' actrice a révélé son dernier look à l' âge et à la rue à mercredi et a partagé un peu sur sa page Twitter . L' actrice de 46 ans a dévoilé son nouveau look alors qu' elle se dans L.A. mercredi et a partagé une photo sur sa page Twitter . -Les départements d' incendie de et de Namur ainsi que la direction des sont intervenus . Les pompiers d' et de Namur et la police des sont intervenus . -Le semble être au bord de la défaite , après avoir été chassée des villes du nord-est de la République démocratique du Congo ( RDC ) , qu' il contrôlait depuis le début de la rébellion il y a 20 mois . Le apparaît sur le point d' être battu après avoir été chassé des localités du nord-est de la République démocratique du Congo ( RDC ) qu' il contrôlait depuis le début , il y a 20 mois , de l' insurrection . -The babe has her her long blonde salope to for a Chinese . L' ancienne jolie nana d' Alerte à Malibu a laissé tomber ses longs cheveux contre une coupe blonde platine . -Le bâtiment situé à 15 rue Hébert a posé trois alarmes au service d' incendie . L' immeuble de quatre logements , situé au 15 de la rue Hébert , a fait l' objet de trois alarmes auprès du service d' incendie . -Les services de protection civile ont stabilisé les deux bâtiments . La protection civile de a stabilisé les deux bâtiments . -Le duo semble avoir redécouvert la magie qui les a réunis dans l' île . Le couple semble avoir retrouvé la magie qui les avait unis dans Island . -Les ont dû être fréquemment afin de maintenir les niveaux d' anticorps qui se sont révélés être coûteux et peu pratiques . Les injections devaient être répétées si fréquemment pour que les niveaux d' anticorps restent élevés qu' ils étaient trop onéreux et peu pratiques . -L' homme a dit à la police qu' il avait peur . Le vieil homme , âgé de 86 ans , a expliqué aux gendarmes qu' il avait eu peur . -Les du , qui ont plusieurs centaines d' habitants , se sont retrouvés à une altitude de 2 000 mètres carrés dans les collines agricoles de , et , à proximité de et de , deux villes situées à 80 kilomètres au nord de Goma , la capitale de la province du Nord-Kivu . Les irréductibles du , soit quelques centaines de combattants , étaient retranchés à près de 2000 mètres d' altitude sur les collines agricoles de , et , proches de et , deux localités situées à environ 80 km au nord de Goma , la capitale de la province du Nord-Kivu . -Le à , où une manifestation violente contre l' impôt a eu lieu samedi , a été jeudi matin . Le portique de , autour duquel s' est déroulée samedi une violente manifestation d' opposants à cette taxe , a été démonté jeudi matin . -Le lentement sur votre porche avant est probablement venu de l' Illinois , qui a fait pousser 542 millions de livres de citrouille cette année . Les qui se doucement sur le porche de votre maison venaient probablement de l' Illinois , où 542 millions de ont été cultivées cette année . -Le détient encore 25 % du capital du groupe logiciel qu' il il y a 40 ans . Le milliardaire détient encore 25 % du capital du groupe de logiciels qu' il a il y a 40 ans . -La tactique de a été critiquée par un certain nombre de défenseurs des droits civils . La tactique du contrôle et de la fouille a été critiquée par de nombreux défenseurs des droits civils . -Le est né d' une mutinerie d' avril 2008 menée par des anciens Tutsis , principalement des Tutsis , qui ont été incorporés dans l' armée en 2009 à la suite d' un accord de paix . Le est né d' une mutinerie , en avril 2012 , d' anciens rebelles , essentiellement , intégrés dans l' armée en 2009 après un accord de paix . -L' ARS , comme le , défend le " nouveau modèle d' hôpital " qui a débuté le 6 octobre et devrait continuer à faire passer de 30,000 à 35,000 patients par année . A l' ARS comme à l' , on défend le " nouveau modèle hospitalier " déjà mis en route depuis le 7 octobre et qui devrait continuer d' accueillir 30 000 à 35 000 patients par an . -L' ARS souligne que 30 patients " envoyés " à plusieurs endroits n' ont pas le risque de devenir une " avalanche " pour les autres services d' urgence , où les ressources humaines ont été renforcées de toute façon . A l' ARS , on fait valoir que trente patients " " sur plusieurs sites ne risquent pas de représenter une " avalanche " dans les autres services d' urgences , dont les moyens humains auront par ailleurs été renforcés . -L' American Civil Liberties Union est également profondément préoccupée par la question de la protection de la vie privée . L' American Civil Liberties Union est elle aussi très préoccupée et exprime son inquiétude concernant la protection de la vie privée . -Le Président américain recevra le Premier ministre irakien , Nouri al-Maliki , qui cherche à aider les États-Unis dans la lutte contre la plus forte vague de violence en cinq ans , le 1er novembre de cette année . Le président américain devait recevoir vendredi 1er novembre 2013 le premier ministre irakien Nouri Al , en quête d' aide des États-Unis pour lutter contre la plus forte vague de violence depuis cinq ans . -Les Américains nous sur le niveau commercial et industriel parce que nous les aussi , parce qu' il est dans l' intérêt national de défendre nos entreprises . Les Américains nous sur le plan commercial et industriel tout comme nous les , car il est dans l' intérêt national de défendre nos entreprises . -L' Associated Press rapporte que la cible était missiles . L' Associated Press rapporte que la cible était des missiles de fabrication russe . -Le Ministre britannique des affaires étrangères , William Hague , a invité le président du Conseil de l' Europe , M. Paul Kagame , à faire preuve de retenue , un porte-parole du Foreign Office a annoncé son intention de le faire . Le chef de la diplomatie britannique , William Hague , a invité le président rwandais Paul Kagame à faire preuve de retenue , a annoncé un porte-parole du Foreign Office . -Le musicien britannique est plein de surprises cette année . Le musicien anglais n' a pas fini de surprendre cette année . -Les CDC ont constaté que la prévalence des allergies alimentaires chez les enfants a augmenté de 18 % entre 1997 et 2007 . Le CDC a découvert que la prévalence des allergies alimentaires chez l' enfant avait augmenté de 18 % entre 1997 et 2007 . -Le chef de la direction de et Cable s' est également félicité de l' annonce faite par le gouvernement Harper lors de son discours du Trône le 16 octobre . Le de et Câble a également salué l' annonce effectuée par le gouvernement Harper lors de son discours du Trône , le 16 octobre dernier . -Le chef de la direction a déclaré qu' il n' avait pas accepté les témoignages du personnel qu' il était mal à l' aise de faire rapport à ses supérieurs . Le directeur général a dit qu' il rejetait les témoignages du personnel selon lesquels ils étaient réticents à l' idée de signaler ces manquements à leur supérieur hiérarchique . -Le CRTC organise des consultations avec le public depuis la semaine dernière , et celles-ci seront poursuivies avec l' industrie au printemps prochain . Le CRTC mène des consultations auprès du public depuis la semaine dernière et elles devraient se poursuivre avec l' industrie au printemps prochain . -La Californie a l' intention de remettre en question ce qui pourrait être une citation , car les lunettes rendent la navigation plus facile . La envisage de contester ce qui pourrait être la première citation à comparaître en son genre , en disant que les lunettes connectées à Internet rendent la navigation plus facile . -L' Agence des services frontaliers du Canada ( ASFC ) n' a pas révélé où était la femme qui avait voyagé après avoir été interceptée . L' Agence des services frontaliers du Canada ( ASFC ) n' a pas révélé d' où arrivait la femme lorsqu' elle a été interceptée . -Le Congrès canadien des soins palliatifs , qui se tient à Ottawa cette semaine , arrive plusieurs jours après le vote en faveur du projet de loi sur la mort de au Québec . Le Congrès canadien de soins palliatifs , qui se déroule cette semaine à Ottawa , survient quelques jours après le vote en faveur du principe du projet de loi sur l' aide médicale à mourir au Québec . -L' Église catholique n' a commencé à le tolérer qu' en 1963 , ce qui a limité les efforts déployés pour y faire face . Elle n' est tolérée que depuis 1963 par l' Eglise catholique , ce qui a limité les tentatives de l' apprivoiser . -Le président du Réseau québécois des soins palliatifs rejette également le projet de loi sur la mort de . La présidente du Réseau des soins palliatifs du Québec rejette aussi le projet de loi sur l' aide médicale à mourir . -Le gouvernement chinois estime que le mouvement est responsable des fréquentes flambées de violence dans la province , lesquelles sont déclenchées par des exigences d' indépendance . Pour le gouvernement chinois , ce mouvement est responsable des fréquentes éruptions de violence qui s' y produisent animées par des revendications indépendantistes . -La police chinoise a identifié le conducteur du véhicule , dont le nom lui semblait d' origine , et a déclaré que sa femme et sa mère étaient avec lui dans le véhicule . La police chinoise a identifié le chauffeur du véhicule dont le nom suggère qu' il est d' origine ouïghoure et a précisé que sa femme et sa mère se trouvaient avec lui dans la voiture . -L' Administration de la sécurité de l' aviation civile a également indiqué qu' elle avait examiné l' annonce , mais a souligné que des restrictions concernant l' utilisation d' appareils électroniques dans les phases critiques de vol étaient encore en place en Australie . La Civil Aviation Safety Authority a également déclaré qu' elle examinait l' annonce mais a souligné que les restrictions portant sur l' utilisation des appareils électroniques pendant les phases de vol critiques étaient toujours en vigueur en Australie . -La magie des frères continue de travailler en intégrant un personnage qui est toujours debout à l' écran : un merveilleux chat mignon , , qui est le compagnon de choix du de C. Davis . ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! La magie des frères opère encore en intégrant un drôle de personnage qui traverse le film en permanence : un magnifique chat roux au regard , compagnon de fortune de Davis . -La filiale de a indiqué jeudi qu' elle effectue actuellement des tests préliminaires de la version bêta de cette plate-forme avec certains de ses utilisateurs . La filiale de a indiqué jeudi qu' elle menait actuellement des tests préliminaires de la version beta de cette plateforme avec certains de ses usagers . -Le Président congolais , Joseph Kabila , a exhorté les rebelles le mercredi à " volontairement " ou " " par la force , tout en continuant d' ouvrir le processus de Kampala , où Kinshasa et sont toujours en pourparlers depuis le mois de décembre sans qu' aucun résultat n' ait été obtenu à ce jour . Le président congolais , Joseph Kabila , a appelé mercredi soir une nouvelle fois les rebelles à " se démobiliser volontairement " sous peine d' être désarmés " par la force " , mais a laissé ouvert le processus de Kampala , où Kinshasa et le discutent depuis décembre sans résultat pour l' instant . -L' armée congolaise ( FARDC ) a annoncé jeudi que ses unités les rebelles de jusqu' à leurs bases situées dans les forêts et les montagnes du Nord-Kivu , qui le Rwanda et l' Ouganda . L' armée congolaise ( FARDC ) a annoncé jeudi que ses unités allaient les rebelles du jusque dans leurs bases situées dans les forêts et les montagnes du Nord-Kivu limitrophe du Rwanda et de l' Ouganda . -Le Département de l' éducation doit publier les nouveaux programmes en anglais et les mathématiques demain - les premiers sujets à subir un remaniement radical . Le ministère de l' Éducation doit publier demain ses nouveaux programmes d' anglais et de maths – les premières matières à subir une révision radicale . -Le mandat unique de la BCE a toujours tourné autour de l' inflation , donc Mario Draghi et son équipe ont toutes les raisons de prendre des mesures lors de la réunion de la semaine prochaine . Le mandat unique de la BCE a toujours porté sur l' inflation , donc Mario Draghi et son équipe ont davantage de raisons d' agir lors de la réunion de la semaine prochaine . -L' UE ne disparaîtra pas , mais elle reviendra sur quelque chose de plus que la " " de l' Europe ( souveraine ) prônée par le général de Gaulle . L' UE ne disparaîtra pas , mais elle reviendra à quelque chose qui ressemblera davantage à « l' Europe des Nations ( souveraines ) » libre privilégiée par le Général de Gaulle . -Le Mouvement islamique du Turkestan oriental est considéré par les États-Unis et l' Organisation comme une organisation terroriste . Le mouvement islamique du Turkestan oriental est considéré comme une organisation terroriste par les Etats-Unis et les Nations unies . -La Commission électorale a étudié les options de vote électronique et a récemment publié un document de discussion commun avec la Nouvelle-Zélande . La Commission électorale a étudié les options possibles en ce qui concerne le vote électronique et a récemment publié un document de réflexion conjoint avec la Nouvelle-Zélande . -Le ministère de l' Énergie a déclaré mercredi que les approvisionnements américains ont augmenté de 4,1 millions de barils la semaine dernière . Le département américain de l' énergie a indiqué mercredi que les approvisionnements américains avaient augmenté de 4,1 millions de barils la semaine dernière . -La FAA a commencé à restreindre l' utilisation des dispositifs portatifs en 1966 en réponse à des rapports d' interférence dans l' utilisation de la navigation et du matériel de communication lorsque les passagers ont commencé à transporter des radios FM , les gadgets de pointe de leur journée . La FAA a commencé à limiter l' utilisation des appareils électroniques en 1996 suite aux rapports sur la présence d' interférences avec les instruments de navigation et de communication lorsque les passagers transportaient des radios FM , le gadget high-tech d' alors . -La FAA est en train d' assouplir les restrictions à l' utilisation de gadgets électroniques dans les avions - bien que les discussions sur les téléphones cellulaires continueront d' être interdites . La FAA assouplit les restrictions portant sur l' utilisation des gadgets électroniques à bord des avions – mais il sera toujours interdit de passer des appels sur les téléphones portables . -La décision de la FAA s' applique aux compagnies aériennes américaines . La décision de la FAA s' applique aux compagnies aériennes américaines . -La FAA a indiqué qu' elle avait déjà reçu des plans de certaines compagnies aériennes d' étendre l' utilisation de dispositifs électroniques portatifs sur les aéronefs . La FAA a déclaré qu' elle avait déjà reçu des plans de certaines compagnies aériennes visant à élargir l' utilisation des appareils électroniques sur leurs avions . -Le Gouvernement fédéral insiste sur le fait qu' un plan de sauvetage de 325 millions de dollars pour le système de santé en difficulté de la Tasmanie a des conditions difficiles qui permettront à l' État de ne pas gaspiller les fonds . Le gouvernement fédéral maintient que le plan de sauvetage de 325 M $ destiné au système de santé est assorti de conditions strictes visant à garantir que le gouvernement de l' État ne pas les fonds . -La police française a décidé d' imposer des règles strictes avant le match entre le Royal Sporting Club d' et le Paris Saint-Germain , prévu pour jeudi , Frank , porte-parole du district de police de , a déclaré vendredi . La police française a décidé de poser des règles strictes concernant la rencontre du Royal Sporting Club d' contre le Paris Saint-Germain fixée mardi , a communiqué Marie Verbeke , la porte-parole de la zone de police , vendredi . -Le Front est désormais le parti le plus populaire en France avec 24 % du vote - une mise en garde rapide au public britannique qu' ils ne peuvent pas prendre part à une scission sur le droit de s' accorder automatiquement la gauche . & # 160 ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Le Front National est désormais le parti le plus populaire en France avec 24 % des suffrages – un avertissement opportun pour le Parti travailliste britannique qui lui rappelle qu' il ne peut pas partir du principe qu' une scission de la droite favorisera automatiquement la gauche . -Le service secret allemand , le , déclarait à l' été 1949 que Muller était à Karlovy Vary , puis en Tchécoslovaquie , selon un document obtenu par . Les services secrets allemands , le , assuraient ainsi que durant l' été 1949 , Müller se trouvait à Karlovy Vary , alors en Tchécoslovaquie , selon un document obtenu par . -Le FMI refuse de payer des sommes supplémentaires jusqu' à ce que le pays adopte des réformes impopulaires visant à réduire son déficit , en particulier en augmentant le prix du gaz pour la population . Mais le Fonds refuse de verser toute nouvelle tranche tant que le pays n' aura pas adopté de réformes impopulaire pour réduire son déficit , notamment en augmentant les prix du gaz à la population . -La Commission indépendante de la lutte contre la corruption ( ) a ouvert la discussion sur le point de savoir si M. avait demandé à plusieurs ministres d' État de souscrire des concessions à quai de quai de , où les deux avaient un restaurant et un café , renouvelés sans avoir à se présenter à l' avance à l' expiration de leur mandat en août 2005 . Vendredi , l' Independent Commission Against Corruption ( ) a élargi son enquête pour savoir si M. Obeid avait fait pression sur plusieurs ministres de l' État pour renouveler ses baux dans Quay , où les Obeid possèdent deux restaurants et un café , après leur expiration en août 2005 , sans qu ’ il n' y ait eu d' appel d' offres . -L' , Comme A Rocket Man et Born In A est également disponible sur le net . The Informer , Like A Rocket Man , Born In A sont eux aussi disponibles sur la toile . -Le DLC est doublement - une illusion d' intro suivie d' une bonne dose de bruit qui ralentit lentement pour faire une ballade en pop . The Informer est à : une intro inquiétante avant une lumineuse qui ralentit progressivement pour laisser place à une ballade pop . -L' armée israélienne , quant à elle , a déclaré que l' objectif de son opération était initialement d' une section d' un vaste tunnel creusé dans le territoire israélien de l' enclave palestinienne , qui avait été découvert le 7 octobre et que , selon l' armée , elle était destinée à des " activités terroristes " . De son côté , l' armée israélienne a indiqué que la cible de son opération était au départ une section d' un large tunnel creusé en territoire israélien depuis l' enclave palestinienne , découvert le 7 octobre et destiné , selon l' armée , à des " activités terroristes " . -L' armée israélienne a simplement parlé de " mesures ciblées " dans la région proche de ce que l' État juif appelait un tunnel " terroriste " creusé à la frontière , dont la découverte a été signalée à la mi-octobre . L' armée israélienne s' est contentée d' évoquer une " action ciblée " dans la zone proche d' un tunnel qualifié de " terroriste " par l' Etat hébreu , creusé à la frontière et dont la découverte a été annoncée milieu octobre . -Le L.A. rapporte qu' un agent de la US Security Transportation Administration ( TSA ) et un suspect ont été blessés lors d' un échange de coups de feu . Le L.A. Times rapporte qu' un agent de l' Agence américaine de sécurité des Transports , TSA , et un suspect ont été blessés durant l' échange de coups de feu . -La Bourse de Londres a fermé jeudi , les prix étant faibles par mauvais résultats de Shell le lendemain de l' annonce par la Fed qu' elle maintiendrait son soutien à l' économie , comme prévu . La Bourse de Londres a clôturé en baisse jeudi , par les mauvaise résultats de Shell et au lendemain d' une annonce par la Fed du maintien en l' état de son soutien à l' économie , comme prévu . -Le Los Angeles Times a déclaré que les terminaux 2 et 3 étaient évacués . Le Los Angeles Times précise que les 2 et 3 sont en cours d' évacuation . -Les encres , comme les formes de , procurent un degré d' imagination plus vaste et plus riche que la danse , plus poreuse . Les encres de , comme des formes de , ouvrent un imaginaire plus large et que la danse , plus poreux . -Le ministre de la Défense , Rob Nicholson , a insisté sur le fait que les soldats blessés ne sont pas libérés sommairement des Forces armées canadiennes et ont souligné que tous les soldats sont soumis à un processus de transition avant leur retour à la vie civile . Le ministre de la Défense , Rob Nicholson , a insisté pour dire que les soldats blessés n' étaient pas sommairement libérés des Forces armées canadiennes , et a souligné qu' un processus de transition était suivi par tous les soldats avant leur retour à la vie civile . -Le ministre des Travaux publics et des Transports à Bruxelles a lancé un projet pilote dans le centre de Bruxelles le jeudi , à savoir la fabrication de 16 points de dans les couleurs jaunes et bleues de la région bruxelloise . Pour rappel , la ministre bruxelloise des Travaux Publics et des Transports a lancé ce jeudi au centre de Bruxelles un projet test consistant à 16 poteaux de feux de signalisation de la région aux couleurs jaune et bleu de Bruxelles . -Le Ministère de l' agriculture indique que le dernier cas enregistré de " cas d' espèce " a été déclaré en décembre 1998 dans un renard et que la France a été officiellement déclarée indemne de cette maladie par l' Organisation mondiale de la santé animale ( OIE ) en novembre 2001 . Le ministère de l' Agriculture précise que le dernier cas " autochtone " de rage recensé remonte à décembre 1998 sur un renard et que " la France est déclarée officiellement indemne de cette maladie en novembre 2001 par l' Office international des épizooties ( OIE ) " . -La société de Montréal a également fait appel à One Network Enterprises , un fournisseur d' accès Internet basé à Vancouver , pour un coût de 526 millions de dollars canadiens en décembre dernier . En décembre dernier , la société montréalaise a également acheté Peer 1 Network Entreprises , un fournisseur Internet établi à Vancouver , pour la somme de 526 millions . -L' entreprise montréalaise déclare que cette baisse est attribuable aux dépenses d' amortissement liées aux acquisitions récentes . L' entreprise établie à Montréal explique ce recul par des coûts d' amortissements reliés à de nouvelles acquisitions . -La prochaine avait un total de 10 morceaux supplémentaires par rapport à l' album original : les trois chansons de l' édition de la saison , cinq chansons spécialement sélectionnées pour l' occasion , et deux . The Next Day Extra contiendra au total dix titres supplémentaires par rapport à l' album original : les trois morceaux de l' édition Deluxe , cinq chansons spécialement dévoilées pour l' occasion , ainsi que deux . -L' indice a chuté de points ( % ) à et le est de points ( % ) , à . L' indice a perdu points ( % ) à 14 et le a cédé points ( % ) à 1 . -L' Observatoire a déclaré qu' il y avait des victimes tous les deux jours , mais qu' il n' avait pas de détails précis . L' Observatoire a déclaré jeudi qu' il y avait des pertes des deux côtés , mais n' a donné aucune précision . -Le PDG d' Oracle a renoncé à un bonus de 1,2 million de dollars pour l' exercice financier 2013 , qui s' est terminé en mai , en raison des faibles performances du groupe , qui n' ont pas atteint ses objectifs de croissance , mais qui a rapporté un total de 77 millions de dollars liés à des options d' achat d' actions . Le patron d' Oracle a renoncé à un bonus de 1,2 million de dollars pour l' exercice 2013 clos en mai en raison des mauvaises performances du groupe , qui a manqué ses objectifs de croissance , mais il a perçu environ 77 millions de dollars liés à des . -Le groupe Palmer pourrait contrôler jusqu' à quatre voix au Sénat , ce qui serait déterminant pour décider si les taxes sur le carbone et l' exploitation minière seraient éliminées . Le Palmer United Party pourrait contrôler jusqu' à quatre voix au Sénat , ce qui pourrait être crucial au moment de décider si les taxes carbone et minières doivent être supprimées . -Le Pentagone estime que la population concernée par la reconnaissance du mariage homosexuel compte environ personnes , soit 17 000 si la Garde nationale , la réserve et les personnes retraitées sont incluses . Le Pentagone estime la population concernée par la reconnaissance du mariage homosexuel à environ personnes active , 17 000 en y incluant la Garde nationale , la réserve et les retraités . -Le président du Conseil italien , Enrico , pense que son gouvernement restera jusqu' en 2015 , malgré de fortes tensions entre les ailes gauche et droite au sein de la coalition au pouvoir dans le cadre du vote du Sénat sur le licenciement de Silvio Berlusconi de son mandat . Le président du Conseil italien Enrico juge que son gouvernement tiendra jusqu' en 2015 malgré les fortes tensions entre gauche et droite au sein de la coalition au pouvoir à l' approche du vote du Sénat pour Silvio Berlusconi de son mandat . -Le Premier ministre devrait sans aucun doute être présent : ce sondage montre que les milieux britanniques soutiennent son projet de renégocier les termes de l' adhésion de la Grande-Bretagne à l' Union européenne . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Le Premier ministre ne devrait pas douter un seul instant : cette enquête montre que les entreprises britanniques soutiennent ce plan visant à renégocier les termes de l' adhésion de la Grande-Bretagne à l' UE . -Le Père du Père Blanc organise une à 11 heures en souvenir de tous ceux qui ont été à l' établissement cette année . Le du Père organise , à 11 heures , une cérémonie laïque , en mémoire de tous les défunts dans son établissement , au cours de l' année . -La Croix Rouge a été demandée parce que les habitants de l' un des appartements peuvent avoir besoin d' un logement temporaire , selon le Département des incendies . La Croix-Rouge a été demandée , car les résidents d' un des appartements pourraient devoir être logés provisoirement , a indiqué le SIM . -La Royal Bank of Scotland ( ) a enregistré le taux le plus bas de la au cours de la même période , après avoir signalé une baisse des bénéfices de cette journée et annoncé la création d' une banque bancaire intérieure contenant 38 milliards de de son actif à risque le plus élevé . Royal Bank of Scotland ( % ) marque la plus mauvaise performance de l' , après avoir également publié des résultats en baisse ce matin et annoncé la création en interne d' une structure de ( " bad bank " ) regroupant 38 milliards de livres d' actifs les plus risqués . -La Banque Royale de l' Allemagne créera une structure bancaire interne pour couvrir 38 milliards de livres ( 45 millions d' euros ) de ses actifs à risque les plus élevés , un pas qui vise à améliorer ses relations avec la Ville et à accélérer son . Royal Bank of Scotland créera en interne une structure de ( " bad bank " ) regroupant 38 milliards de livres ( 45 milliards d' euros ) d' actifs les plus risqués , mesure destinée à détendre ses relations avec Londres et à accélérer sa . -Le gouvernement des SNP en Écosse est le - le plus fort mouvement politique raciste en Europe - ayant remporté une victoire spectaculaire en 2011 en raison de l' opposition aux réductions proposées ( et mises en œuvre ) par le chancelier travailliste de Darling et la coalition . Le gouvernement SNP d ’ Écosse est – et c' est à noter – le mouvement politique qui a le mieux réussi en Europe , puisqu' il a remporté une majorité spectaculaire en 2011 fondée sur son opposition aux coupes sombres proposées ( et mises en œuvre ) par le Chancelier de l' échiquier travailliste , Darling , et la coalition conservateurs / qui a suivi . -Le Club Sari a été frappé lorsque une bombe massive placée dans un fourgon stationné à l' extérieur a explosé juste après minuit , le 12 octobre 2002 . Le Sari Club a été entièrement détruit lorsqu' une énorme bombe , chargée dans une fourgonnette à l' extérieur , a explosé juste après 23 h le 12 octobre 2002 . -Le conseil scolaire vient de dévoiler sa vision de " Vivre ensemble en français " . La commission scolaire vient de dévoiler sa vision du " Vivre ensemble en français " . -Le match de football en Écosse a été repoussé pour une dispute opposant le au match contre le FC Barcelone , et l' UEFA a suspendu sa mise en place pour trois matchs . contre le pour avoir adressé un coup à , le milieu écossais a vu sa suspension étendue à trois rencontres par l' UEFA . -Le Secrétaire à la défense a déclaré qu' il avait ordonné au chef de la Garde nationale , le général Frank , de veiller à l' application de la loi fédérale . Le secrétaire à la Défense a dit avoir ordonné au chef de la Garde nationale , le général Frank , de s' assurer de la mise en oeuvre de la loi fédérale . -Le Secrétaire d' État a également reconnu qu' il aurait été coupable , avec Barack Obama , d' être " pilote automatique " à la suite de révélations incendiaires faites par Edward sur les activités d' espionnage menées par le NSA . Le Secrétaire d' État a également admis qu' il avait , tout comme Barack Obama , gouverné sur « pilote automatique » lorsqu' une série de révélations incendiaires du Edward au sujet des activités d' espionnage de la NSA a vu le jour . -Le groupe de Sheffield devait se présenter au centre d' Hydro dans la ville le vendredi . Le groupe de Sheffield devait se produire vendredi dans la salle de concert The Hydro au centre-ville . -Le conseil scolaire de la région de Sherbrooke ( ) n' a pas voulu commenter la question . La Commission scolaire de la ( ) n' a pas voulu commenter . -Le porte-parole de la TSA n' a pas pu confirmer sur place si un de ses employés avait été blessé . Le porte-parole de la TSA n' était pas en mesure de confirmer dans l' immédiat si l' un de ses employés était blessé . -Le Times a obtenu un exemplaire du communiqué suivant et a déclaré jeudi matin que les principaux collaborateurs du président avaient " travaillé à fond et à la retraite à la fin de 2011 " pour déterminer si le scandale pouvait aider à renforcer les espoirs de réélection d' Obama . & nbsp ; & nbsp ; & nbsp ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; – & nbsp ; & nbsp ; » Le Times a obtenu une copie du livre à paraître et a raconté jeudi soir que les proches collaborateurs du président avaient mené des « travaux approfondis en groupes et des sondages fin 2011 » pour voir si l' éviction de Biden pourrait aider Obama en vue de sa réélection alors que ses espoirs s' . -La bourse de Tokyo est tombée à la fin du mois de vendredi , malgré les bons chiffres de la production en provenance de Chine . La Bourse de Tokyo a fini en baisse vendredi en dépit de bonnes statistiques manufacturières chinoises . -Le ministère du Trésor a procédé à l' allégement budgétaire de la BCE , mais il a estimé qu' une décision finale de M. Osborne , qui , dans le budget de 1995 , prévoyait un taux de 7 % de droits de timbre pour les ménages de plus de 2 millions de livres sterling et des frais annuels pour les acheteurs qui choisissent de placer des maisons dans une entreprise plutôt que des particuliers . Le Trésor a provisoirement chiffré la mesure de l' impôt sur les gains en capital mais attend une décision définitive de M. Osborne qui , dans le budget de 2012 , a inscrit un droit de timbre de 7 % pour les propriétés coûtant plus de 2 M £ et des frais annuels pour les acheteurs qui choisissent de détenir des propriétés dans le cadre d' une société plutôt qu' à titre individuel . -Le Sénat américain a approuvé un projet pilote de $ l' année dernière qui aurait impliqué environ 10,000 voitures . Le Sénat américain a approuvé un projet pilote de 90 M $ l' année dernière qui aurait porté sur environ 10 000 voitures . -Le site Web allemand English Online s' est également opposé à la loi allemande , qui dit : " Lorsque le monde est poussé dans un nouvel état , où le sexe est un choix , mais l' activité sexuelle n' est pas , nous deux autres piliers de la civilisation " . Le site américain Catholic Online s' est également opposé à la loi allemande , écrivant que « puisque le monde est traîné dans un nouveau règne où le genre est un choix mais où l' activité sexuelle n' en est pas un , on renverse deux autres piliers de la civilisation . » -Le gouvernement britannique , qui a tenu des discussions avec Transports Scotland , a demandé à Ltd de se pencher sur d' autres capacités de transport ferroviaire et d' amélioration du temps pour le nord de l' Angleterre et de l' Écosse . Le gouvernement britannique , qui est en pourparlers avec Transport Scotland , a demandé à Ltd d' étudier le renforcement de la capacité ferroviaire et l' amélioration de la durée des trajets pour le nord de l' Angleterre et l' Écosse . -L' ONU et l' Ouganda accusent régulièrement l' Ouganda et le Rwanda de soutenir le . L' ONU et Kinshasa accusent régulièrement l' Ouganda et le Rwanda de soutenir le , ce que réfutent Kigali et Kampala . -L' ONU a signalé des chiffres encore plus élevés , avec 979 morts et blessés . L' ONU donne un bilan même plus élevé avec 979 morts et 1 902 blessés . -La Federal Aviation Administration ( FAA ) des États-Unis a autorisé les compagnies américaines à modifier leurs procédures afin que les clients puissent lire des livres électroniques , regarder des vidéos ou des jeux sur leurs appareils pendant les phases critiques de vol , à condition de rester en " avion " . La Federal Aviation Administration américaine a laissé la porte ouverte aux transporteurs américains pour un changement de leurs procédures afin que les passagers puissent lire des livres électroniques , regarder des vidéos ou jouer à des jeux sur leurs appareils pendant les phases de vol critiques à condition qu' ils soient en mode « avion » . -L' US National Intelligence a prévu en février la France à la frontière avec la Russie et Israël dans un deuxième niveau de criminels qui se livrent à la désinformation en matière d' intelligence économique , derrière la Chine , selon le Washington Post . Le National Intelligence ( ) réalisé en février par les États-Unis , citait la France , de même que la Russie et Israël , coupables au second degré d' avoir effectué des actes de piratage de renseignements économiques , derrière la Chine , selon The Washington Post . -Les États-Unis mettent en garde leurs propres entreprises contre l' espionnage économique d' autres pays . Les États-Unis leurs propres sociétés sur les activités d' espionnage économiques réalisées par d' autres pays . -L' ONU doit commencer à travailler immédiatement sur un nouvel ensemble d' objectifs visant à remplacer les objectifs du Millénaire pour le développement , qui ont été mis en place il y a 12 ans pour lutter contre la pauvreté dans le monde . Les Nations Unies vont se mettre dès à présent à travailler sur une nouvelle série d' objectifs destinés à remplacer les objectifs du millénaire pour le développement ( OMD ) , qui avaient été mis en place il y a 12 ans pour lutter contre la pauvreté dans le monde . -Les États-Unis ont parfois été " trop loin " à l' occasion de leurs activités d' espionnage , le ministre d' État John Kerry a reconnu , lors de la première entrée en fonction de Washington , une profonde controverse avec l' Europe sur la collecte massive de données par l' Agence de sécurité nationale ( NSA ) . Les Etats-Unis sont parfois allés " trop loin " en matière d' espionnage , a reconnu le secrétaire d' Etat John Kerry , dans ce premier aveu de Washington en pleine polémique avec l' Europe sur la collecte massive de données par l' Agence nationale de sécurité ( NSA ) . -La Chambre haute du parlement italien se réunira ce mois-ci pour décider du sort de l' ancien président qui a été reconnu coupable d' évasion fiscale en août . La chambre haute du parlement italien se réunira dans le courant du mois pour décider du sort de l' ancien président du Conseil condamné pour fraude fiscale en août . -Le Vatican est en train de réaliser un grand sondage dans le monde entier sur la manière dont les paroisses traitent des questions sensibles telles que la contraception , le divorce et les couples de même sexe . Le Vatican a entrepris de mener un grand sondage à travers le monde sur la façon dont les paroisses gèrent les dossiers sensibles comme la contraception , le divorce et les couples formés de personnes de même sexe . -Le Vatican veut savoir comment les paroisses catholiques du monde entier gèrent des questions délicates telles que la contraception , le divorce et les couples homosexuels . Le Vatican veut savoir comment les paroisses catholiques du monde entier gèrent les questions délicates comme la contraception , le divorce et les couples homosexuels . -Le loup de Wall Street : Deuxième Le Loup de Wall Street : sortie du second trailer -L' accident s' est produit juste après que la victime était arrivée sur le site pour recueillir ses outils . L' accident s' est produit au moment où la victime venait d' arriver sur le chantier afin de récupérer ses outils . -La précision de la reconnaissance faciale s' est améliorée au cours des cinq dernières années , dit Cyrille , directeur de recherche de R & D auprès d' Accenture en Europe . La précision de la reconnaissance faciale a été multipliée par dix au cours des cinq dernières années , calcule Cyrille , directeur des laboratoires de R & D d' Accenture en Europe . -Le groupe militant , qui fait le bilan du nombre de victimes par le biais d' un réseau de militants en Syrie , a déclaré jeudi que personnes sont mortes . Le groupe de militants , qui suit l' évolution du bilan humain grâce à un réseau de militants basés en Syrie , a déclaré jeudi que 120 296 personnes étaient mortes . -L' administration a dit qu' il ne faut pas s' étonner que cinq pour cent de la population qui achète des assurances puisse changer de régime parce que leur protection ne répond pas aux nouvelles normes requises en vertu de la Loi sur les soins à la personne abordable . L' Administration a annoncé que l' on ne devrait s' étonner si les 5 % de la population qui contractent eux-mêmes leur assurance se retrouvaient forcés de changer de régime car leur couverture ne satisfait pas aux nouvelles normes exigées en vertu de l' Care Act . -L' Office note que les réserves de change de l' Ukraine ont diminué de 26 % entre septembre 2012 et septembre 2013 , et que la tendance devrait se maintenir . L' agence relève que les réserves de change de l' Ukraine ont chuté de 26 % entre septembre 2012 et septembre 2013 et prévoit que la tendance va continuer . -Le but de l' assaut est de la des collines qui entourent . L' objectif est de déloger le des collines qui surplombent . -Le but de la loi était de faire pression sur les parents qui pourraient prendre des décisions hâtives sur la chirurgie des nouveau-nés et de lutter contre la discrimination à l' encontre des . Le but de la loi était d' enlever la pression qui pesait sur les épaules des parents qui pouvaient prendre des décisions précipitées concernant une chirurgie de sexuelle pour leur nourrisson , et pour lutter contre la discrimination à l' encontre des personnes . -L' objectif est de " renforcer la sécurité " et de " renforcer l' identité de Bruxelles " . Le but étant à la fois " d' accroître la sécurité " et " l' identité bruxelloise " . -Le transporteur aérien a été élevé et sec à Toulon . A Toulon , le porte-avions est en rade . -L' aéroport a également indiqué que l' incident avait eu lieu à environ heure locale ( heure de Montréal ) à l' aérogare 3 de l' aéroport . L' incident a eu lieu vers 9h30 locales ( 12h30 à Montréal ) au terminal 3 de l' aéroport , a également précisé l' aéroport . -L' aéroport est actuellement évacué et le trafic aérien a été suspendu . L' aéroport est en cours d' évacuation et le trafic aérien est interrompu . -Les actions présumées ont eu lieu en général avant celles citées dans la première action , la plupart des institutions concernées étant fermées dans les années 60 . Les faits allégués sont en général plus anciens que ceux mentionnés dans le premier recours , puisque plusieurs des établissements visés ont fermé leurs portes dans les années 1960 . -L' animation , destinée aux enfants de 2 à 7 ans , est consacrée aux aventures du chat , , le shérif d' une ville dans l' Ouest sauvage où elle maintient le droit de l' ordre à l' aide d' une magique . Le dessin animé , destiné aux enfants de 2 à 7 ans , raconte les aventures de la chatte , shérif d' une ville de l' ouest américain où elle fait régner l' ordre avec un magique . -L' annonce faite par l' Organisation pour l' interdiction des armes chimiques est arrivée à la fin de l' année . L' annonce de l' Organisation pour l' interdiction des armes chimiques ( OIAC ) est arrivée avec un jour d' avance sur la date limite du -L' annonce a été marquée par des combats qui ont eu lieu jeudi dans la ville de , où des experts ont dit qu' ils étaient à l' origine d' une installation de production d' armes chimiques , ainsi que des sites de stockage , a déclaré l' Observatoire syrien des droits de l' homme de . L' annonce a été faite alors que les combats faisaient rage jeudi dans la ville de qui , selon les experts , abrite une unité de production d' armes chimiques ainsi que des sites de stockage , a rapporté l' Observatoire syrien des droits de l' Homme basé en Grande-Bretagne . -L' annonce est une mauvaise nouvelle pour le gouvernement ukrainien dans une période de tensions sérieuses avec son voisin russe , qui est par la volonté de Kiev de signer un accord d' association avec l' UE à la fin du mois de novembre . Cette annonce constitue une mauvaise nouvelle pour le pouvoir ukrainien dans une période de vives tensions avec son voisin russe , furieux de la volonté de Kiev de signer un accord d' association avec l' UE fin novembre . -L' annonce que David Bowie avait sorti un nouvel album avait le monde . L' annonce de la parution d' un nouvel album de David Bowie avait laissé tout le monde . -L' enquête annuelle montre la force de l' attachement à la disposition de la cérémonie . L' enquête annuelle d' montre un attachement très fort à l' organisation de la cérémonie . -Les anticorps les molécules de nicotine dans le sang , les avant d' atteindre le cerveau , empêchant ainsi un fumeur de toucher à la nicotine . Les anticorps toutes les molécules de nicotine présentes dans le système sanguin et les avant qu' elles atteignent le cerveau , ce qui empêche le fumeur d' avoir sa dose de nicotine . -La cour d' appel a entendu les plaidoiries le mardi sur le sursis demandé . La cour d' appel a entendu mardi les arguments portant sur la demande de suspension . -La cour d' appel a déclaré que le juge d' appel devait être écarté parce qu' il était nécessaire de revoir le code de conduite des juges américains , en empêchant le juge d' éviter un sens de partialité en partie à cause d' une série d' entretiens avec les médias et de déclarations publiques répondant publiquement à des critiques du tribunal . La cour d' appel a déclaré que la juge devait être du dossier car elle a enfreint le code de conduite des juges en compromettant la nécessité pour un juge d' éviter toute apparence de partialité , en partie en raison d' une série d' entretiens accordés aux médias et de déclarations publiques dans lesquels elle réagissait publiquement aux critiques de la cour . -L' apparence de l' acné et des poils de l' état est fréquente , même chez les nourrissons et les tout-petits . L' apparition de l' acné et des poils est courante , même chez les nourrissons et les tout-petits . -La région n' a pas été abandonnée et les rats ne sont pas dans la ville . Le quartier n' est pas laissé à l' abandon , et les rats ne pas dans la cité . -La zone est remplie d' entrepôts , ce qui facilite la manipulation des camions chargés de la drogue . La zone regorge d' entrepôts , ce qui permet de dissimuler facilement les camions chargés de drogue . -L' arrangement sera le même pour le retour . Le déroulement sera le même pour le retour . -L' arrivée de Jean-Luc au Châtelet et à la a changé tout cela . L' arrivée de Jean-Luc au Châtelet et de Stage ­ ­ Entertainment à a changé la donne . -L' artiste est décédé dimanche dernier à l' âge de 71 ans . L' artiste est décédé dimanche dernier , à l' âge de 71 ans . -L' attaque a eu lieu au moment où l' Egypte n' a jamais reçu de touristes , car l' armée a destitué le président Arafat , Mohamed Morsi , au début du mois de juillet et les manifestations de ses partisans . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Cette attaque survient alors que l' Égypte ne reçoit quasiment plus aucun touriste depuis que l' armée a destitué le président islamiste Mohamed Morsi début juillet et réprime dans le sang les manifestations de ses partisans . -Les assaillants ont fui , selon le porte-parole du Ministère de l' intérieur , le général de police Abdel Latif . Les assaillants ont pris la fuite , a précisé le porte-parole du ministère de l' Intérieur , le général de police Abdel Latif . -The staff were very friendly and helpful . The room was very comfortable and the staff were very friendly and helpful . Le public a aimé , l' accueil fut chaleureux . -L' austérité imposée par Berlin et la Banque centrale européenne , ainsi que le carcan imposé aux économies nationales par l' adhésion à la monnaie commune , ont amené beaucoup de gens à penser que le projet Europe est allé trop loin . L' austérité imposée par Berlin et la Banque centrale européenne , associée au carcan pesant sur les économies nationales par l' intermédiaire de l' adhésion à la monnaie unique , a conduit de nombreuses personnes à penser que le Projet Europe est allé trop loin . -Le fond est un écran blanc . Un écran blanc sert de mur de fond . -L' arriéré de la division aérospatiale était de 32,9 milliards de dollars au 30 septembre , inchangé par rapport au 31 décembre . Le retard dans la division Aéronautique représentait 32,9 Md $ au 30 septembre , chiffre inchangé par rapport au 31 décembre de l' année précédente . -L' annonce du groupe a eu lieu après qu' ils ont été obligés de reporter de nouveau le concert du LG Arena à Birmingham jeudi . L' annonce a été faite après que le groupe a été contraint de reporter de même un concert à la LG Arena de Birmingham jeudi . -La banque , qui a refusé de commenter les suspensions , a confirmé ce mois qu' elle recevait des demandes d' information de la part des organismes de réglementation . La banque , qui a refusé de commenter les suspensions , a confirmé ce mois-ci avoir reçu des demandes d' informations de la part des régulateurs . -La banque et la ville ont souligné que la " mauvaise banque " aurait permis à un du passé , notamment le gouvernement , d' être accusé d' interférer dans la gestion du . La banque et Londres ont souligné que la " bad bank " permettrait de tirer un trait sur le passé , le gouvernement ayant en particulier été accusé de s' ingérer dans la gestion de . -La banque a rapidement avisé les autorités concernées et coopéré avec ses régulateurs . La banque a rapidement informé les autorités compétentes et a coopéré avec les régulateurs . -La banque veut réduire la proportion d' actifs toxiques sur ses livres de 55 % à 70 % au cours des deux prochaines années et espère y nettoyer complètement trois ans . La banque veut réduire de 55 % à 70 % la proportion des actifs douteux dans son bilan dans les deux années à venir et espère avoir nettoyé complètement celui-ci dans les trois ans . -Le procureur a déclaré à la presse canadienne qu' il croyait que la police avait " ignoré ou minimisé les éléments de preuve " contre le maire . Me Ruby a dit croire à La Presse Canadienne que la police avait " ignoré ou minimisé " les éléments de preuve contre le maire . -Les vélos étaient pour s' approcher et prendre des photos , je me suis rendu à la maison , où j' ai dû prendre des mesures de maintien de l' ordre pour empêcher certaines d' entre elles d' escalader . Les motards se tirent la bourre pour faire des photos et me suivent jusqu' à chez moi où je dois faire un peu la police pour empêcher certains de monter . -Le corps , un simple accident , prend sa forme . Le corps , simple trait , prend sa forme . -Les mouvements du corps , le rire , les petits mots admirablement les , la grâce et l' absurdité des corps en , des êtres se mariant les uns avec les autres . La gestuelle , les rires et les petits cris l' archaïque , la grâce et le ridicule des corps en coït , des êtres qui se fondent . -Le livre offre donc un portrait fascinant de la France à la sortie et un plan de la France dans la construction . Le livre offre donc à la fois le portrait fascinant de la France qui s' en va et l' esquisse de la France qui vient . -Le garçon était avec son père sur le bateau " The of boat " quand il est tombé . Au moment de la chute , le garçon était avec son père sur un bateau de l' attraction " Pirates des Caraïbes " . -Le terrain propice au talent est aussi important que le nombre de postes disponibles . Le vivier de talents est aussi conséquent que le nombre de postes à pourvoir . -Le bâtiment endommagé par l' incendie contenait quatre appartements , mais il n' y avait personne à la maison lorsque l' incendie a commencé . La maison incendiée abritait quatre logements , mais aucun locataire ne se trouvait sur les lieux au moment de l' incendie . -Les appels n' interviennent qu' en raison de la nécessité de remédier aux problèmes causés par la discorde dans l' histoire de la France , déclare Nora . Ils ne sont si pressants , suggère Pierre Nora , que pour conjurer le trouble né des de l' histoire française . -La voiture conduite par la femme blessée a été frappée par un chauffeur qui se déplaçait dans la mauvaise direction et a tiré sur . La voiture d' une conductrice a été par un conducteur fantôme , qui lui est indemne . -Les Cardinaux seront également les premiers à être élus depuis la création du Conseil des Cardinaux , un groupe de huit cardinaux de tous les coins du monde , chargé de rechercher les moyens de réformer l' Église . Les cardinaux seront également les premiers à être choisis depuis que François a formé le conseil des cardinaux , un groupe de huit cardinaux du monde entier chargés d' examiner les moyens de réformer l' église . -La cause de l' explosion n' était pas connue , a-t-il dit . La cause de l' explosion reste inconnue , a-t-il ajouté . -La cause du feu n' est pas connue , mais il n' y a personne dans le restaurant lorsque les pompiers sont arrivés sur les lieux . La cause de l' incendie était inconnue , mais il n' y avait personne au restaurant lorsque les pompiers sont arrivés sur la scène . -Les accusations portées par le département de la justice des États-Unis incluent l' espionnage et le vol de biens de l' État . Dans le cadre de la plainte déposée par le ministère américain de la justice , il est accusé d' espionnage et de vol de biens de l' État . -Le chef sera remplacé par son adjoint , âgé de ans , qui est à l' Elysée depuis dix-sept ans . Le chef sera remplacé par son adjoint , Guillaume Gomez , 35 ans , qui est pour sa part depuis dix-sept ans à l' Elysée . -Le chef de la police n' a pas abordé cette question , mais son porte-parole , Mark , a décrit les déclarations de Ruby comme " une tentative évidente et désespérée " de se tourner vers les médias dans une affaire qui ne le concerne pas du tout . Le chef de police n' a pas commenté , mais son porte-parole Mark a qualifié les propos de Me Ruby de " tentative évidente et désespérée " de se faire valoir auprès des médias dans une affaire qui ne le concerne aucunement . -Le choix se fait de plus en plus . L' offre s' est . -Le cinéma a été en mesure de se montrer en bonne forme et les clients peuvent continuer à profiter des événements ... Le cinéma a pu reprendre ses droits et les clients retrouver la plaisir des images . -Le cinéma a été ventilé et tout le monde est revenu en bon ordre . La salle a été aérée et tout est rentré dans l' ordre . -La ville a fait appel de ses conclusions et de ses ordonnances de redressement , y compris la décision d' affecter un surveillant pour aider le service de police à modifier son programme de politiques et de formation . La ville a fait appel de ses conclusions et de ses ordonnances correctives , notamment une décision visant à attribuer un contrôleur pour aider le département de police à modifier sa politique et le programme de formation qui lui est associé . -La réclamation se fonde sur le témoignage d' un " " , connu sous le nom de J. A. , qui avait été dans les années 1950 , d' abord par son , frère de , puis par son , le père Bernard , à l' Oratoire Saint-Joseph , où il aurait été un garçon d' autel et où son père a travaillé comme peintre . La requête s' appuie sur le témoignage du requérant , identifié comme " J. J " . , qui aurait été dans les années 1950 , d' abord à par son professeur le frère Soumis , puis par son , le père Bernard , à l' oratoire Saint-Joseph , où il était servant de messe et où son père travaillait comme peintre . -L' action collective doit d' abord être jugée recevable par la Cour supérieure . Le recours collectif doit d' abord passer l' étape de la recevabilité en Cour supérieure . -L' action collective qui vient d' être introduite a précisément permis d' inclure des plaignants de toute institution où des membres de la Congrégation de la Croix peuvent avoir été impliqués dans des abus . Le recours collectif qui vient d' être déposé a pour particularité de permettre l' ajout de plaignants de n' importe quel établissement où des sévices auraient pu être perpétrés par des membres de la congrégation de Sainte-Croix . -La fermeture du Delta intervient à un moment de grande concurrence sur le marché de l' hôtellerie . La fermeture du Delta intervient dans un marché hôtelier très concurrentiel . -La cocaïne a été entre trois qui avaient été auparavant . La cocaïne avait été répartie dans trois ayant été préalablement . -Le Comité a introduit de nouvelles restrictions sur les données que les services de renseignement ont été autorisées à percevoir et ont imposé une limite de cinq ans à la durée de conservation de ces données . La commission a instauré de nouvelles restrictions sur les données que les agences de renseignement sont autorisées à collecter et a imposé une limite de cinq ans pour la conservation de ces informations . -L' entreprise a amélioré ses prévisions de bénéfices à long terme et s' attend à produire des recettes provenant de vacanciers en Australie et au Royaume-Uni . La société a revu ses prévisions de bénéfice net à la hausse et espère des résultats record grâce aux vacanciers d' Australie et du Royaume-Uni . -L' entreprise cible millions de barils par jour pour cette année , avec une augmentation de 25 % de la production prévue en 2017 . La société vise millions de pour cette année , avec une croissance de 25 % de la production attendue d' ici 2017 . -La société a perdu clients au cours du quatrième trimestre . L' entreprise a perdu 15 237 clients au cours du quatrième trimestre . -La société va financer cette opération en utilisant 60 % de son chiffre d' affaires de $ qu' elle a augmenté cette semaine dans le cadre d' une cotation à la Bourse de Singapour . La société les financera en utilisant 60 % des 66 M $ de Singapour qu' elle a levés cette semaine lors de son introduction à la bourse de Singapour . -L' achèvement de ce qui est essentiellement le stade initial de la destruction est un jalon important dans un calendrier ambitieux qui vise à détruire toutes les armes chimiques de Damas par . L' achèvement de ce qui est essentiellement la phase de destruction initiale est une étape importante du calendrier ambitieux visant à détruire les armes chimiques de Damas d' ici . -Le concept n' est pas un défi universel . Le concept n' est pas un succès universel . -Les conclusions du sondage seront une surprise pour beaucoup . Les conclusions de l' enquête seront une surprise pour beaucoup . -L' état des deux victimes demeure critique . Le pronostic vital des deux victimes est toujours engagé . -Le conflit a forcé quelque 2 millions de personnes à fuir le pays . Quelque 2 millions de personnes ont dû fuir le pays en raison du conflit . -L' affrontement a duré une demi-heure , selon des témoins . L' affrontement a duré une demi-heure , selon des témoins . -Le contrat a chuté de 39 cents le jeudi , ce qui lui a fait baisser de 5,8 % le mois d' octobre . Le contrat a perdu 39 cents jeudi , le laissant avec une baisse de 5,8 % pour le mois d' octobre . -Le pays compterait environ 1 000 tonnes de substances chimiques et d' armes , y compris de gaz moutarde et de sarin . On pense que le pays dispose d' environ 1 000 tonnes de produits et d' armes chimiques , notamment du gaz moutarde et l' agent sarin . -Le tribunal a également confirmé les articles restants de la Loi sur les et a ajouté que la loi sur les soins de santé exige que les États imposent l' admissibilité au supplément de Medicaid ou qu' ils perdent tout le financement du Service Medicaid fédéral n' est pas contraire à la loi contraire à la loi . La Cour a également validé les autres sections de la loi de 2 700 pages et a , par ailleurs , considéré que la disposition de la loi de réforme du système de santé , qui stipule qu' il faut abaisser le seuil d' éligibilité à Medicaid sous peine de perdre tous les financements fédéraux du programme , était coercitive conformément à la constitution . -Le tribunal a entendu un " utilisateur régulier " de cocaïne après avoir découvert des traces de la drogue de la classe A dans ses cheveux , son portefeuille et sur ses cartes de crédit . Le tribunal a entendu que était un « consommateur régulier » de cocaïne après que des tests ont montré des traces de cette drogue de classe A dans ses cheveux , sur son portefeuille et ses cartes de crédit . -Le tribunal de La Haye a été créé à cette fin . Un tribunal à La Haye est fait pour ça . -Le créateur de SAS a déclaré qu' il était en train de " fabriquer une sorte d' avion de comédie géopolitique " . Le père de SAS disait " faire un genre de feuilleton géopolitique " . -Le nous a autorisé à le diffuser . Le nous a autorisés à la relayer . -La crise de l' euro a peu à voir avec les gouvernements nationaux qui ont des déficits budgétaires excessifs - ce n' était vrai que pour la Grèce . La crise de l' euro n' a pas grand-chose à voir avec les gouvernements nationaux qui enregistrent des déficits budgétaires excessifs – cela était uniquement vrai pour la Grèce . -Le chef de l' État actuel , qui a sur le livre qu' il a perdu pendant la campagne présidentielle , " n' a pas d' obligation alimentaire spécifique " . L' actuel chef de l' Etat , qui a regagné les kilos perdus avant la campagne présidentielle , n' a " pas de demande de régime spécifique " . -La est relativement et les attaques ne sont pas censées pénétrer dans le réseau interne du système cible . La n' est pas très sophistiquée et les attaques ne sont pas destinées à pénétrer dans le réseau interne du système visé . -Les dommages étaient assez limités dans l' ensemble , même si l' eau a causé un dommage . Les dommages ont été somme toute limités , même si l' eau a causé des dégâts . -Les danseurs , tous habillés en noir , qui ne portent que les mains et les regards , attendent de chaque côté de la scène ; ils se demandent quand le prochain graphique s' affiche , s' , puis s' . Les danseurs , entièrement vêtus de noir , seuls les mains et le visage découverts , attendent de chaque côté de l' aire de jeu , se précipitent quand la prochaine apparaît , l' incarnent , ressortent en courant . -Les données présentées au Parlement pour l' année dernière indiquent que sur les soldats qui ont été libérés pour des raisons médicales , 199 n' avaient pas atteint la durée de service requise pour obtenir des prestations de retraite . Des données présentées l' an dernier au Parlement indiquent que parmi les soldats libérés pour des raisons médicales , 199 n' avaient pas atteint la durée de service requise pour obtenir des prestations de retraite . -Le jour où j' étais libre , j' étais au décollage d' une voiture où j' ai rencontré Roger - mais je n' ai pas pu le voir parce que c' était sombre . Quand je suis libéré , je me retrouve dans le coffre d' une voiture où je rencontre - sans le voir car il fait noir - Roger . -La décision de la cour d' appel signifie que les médecins pratiquant l' avortement devront avoir un " privilège d' admission " avec les hôpitaux locaux . L' arrêt rendu par la cour d' appel signifie que les médecins qui des IVG devront disposer d' un " privilège d' admission " auprès des hôpitaux locaux . -La décision a ravivé les souvenirs douloureux de la grand-mère de June , qui a perdu sa fille , âgée de 20 ans , Jane et Jenny , lors de la répression de et de son , il y a près de dix ans . La décision a ravivé des souvenirs pénibles pour June , une mère de Perth qui a perdu ses deux jumelles de 39 ans , Jane et Jenny , dans les destructions perpétrées par et les autres conspirateurs il y a près de 10 ans . -La décision signifie qu' un texte juridique adopté dans l' État en juillet dernier , qui met en doute le droit à l' avortement , pourra entrer en vigueur . Ce jugement signifie qu' un texte de loi adopté en juillet dans cet Etat et remettant largement en cause le droit à l' avortement va pouvoir entrer en vigueur . -La décision de prendre le , annoncée par la police du mercredi , a été prise par , l' entreprise gérant le portique . La décision du démantèlement , annoncée par la préfecture du Finistère mercredi , a été prise par la société , gestionnaire du portique . -Le départ de près de 10 pharmaciens du Centre for Health and Social Services ( ) de Laval a causé des troubles au sein des dirigeants de l' hôpital de l' année 2012-2013 . Le départ de près de 10 pharmaciens au Centre de santé et services sociaux ( ) de Laval a causé tout un émoi à la direction de l' Hôpital de la , et ce , lors de l' année 2012-2013 . -Les détracteurs accusent le gouvernement de forcer l' argent à faire des économies parce qu' ils ne permettent pas aux soldats blessés - qui ne respectent pas la règle de l ' " universalité " de l' armée - que le personnel puisse accomplir un certain nombre de tâches différentes , afin de pouvoir atteindre la période de dix ans requise pour les prestations de retraite . Les détracteurs accusent le gouvernement de vouloir économiser de l' argent en ne permettant pas aux militaires blessés - qui ne répondent pas à la règle de " l' universalité du service " de l' armée exigeant que le personnel soit en mesure d' effectuer une série de tâches variées - d' atteindre la période d' admissibilité de 10 ans requise pour les prestations de retraite . -Les dispositifs , qui les automobilistes et transmettent les informations aux bureaucrates , sont au centre d' une tentative controversée des bureaux de l' État et de la planification de l' État de réformer le système dépassé pour le financement des principales routes de l' Amérique . Les appareils , qui enregistrent tous les miles parcourus par un automobiliste et transmettent les informations aux fonctionnaires , sont au centre d' une tentative controversée à Washington et dans les bureaux gouvernementaux de la planification de remanier le système obsolète de financement des principales routes américaines . -Le diagnostic de la rage a été confirmé par l' Institut Pasteur . Le diagnostic de rage a été confirmé par l' institut Pasteur . -Le réalisateur Lars von Trier l' a choisi pour jouer la jeune Charlotte dans son prochain film , . Le réalisateur Lars von Trier l' a choisie pour incarner Charlotte jeune dans , son prochain film . -Le directeur de l' Association des soins palliatifs de l' Ontario , Rick Firth , estime que le projet de Québec est confus pour ce qui est de l' objectif des soins palliatifs . Le directeur de l' Association des soins palliatifs de l' Ontario , Rick Firth , croit que le projet de loi du Québec la population au sujet de la vocation des soins palliatifs . -La discorde s' est propagée à l' Asie vendredi . Et la polémique s' étendait vendredi à l' Asie . -Le document de travail fait état de 12 mesures qui ont été précédemment examinées par l' ancien ministre de l' Immigration , M. Stephen Bowen . Le document de travail fait état des 12 mesures qui avaient précédemment été envisagées par l' ancien ministre de l' Immigration Chris Bowen . -Le litige se produit lorsque les dirigeants de l' avion tentent de vendre des versions de plus en plus longues de leurs avions à grand rayon d' action , ce qui pourrait permettre d' enregistrer les commandes attendues à l' occasion de l' événement du 21 novembre . Le différend a éclaté alors que les cherchent à vendre des versions encore plus grandes de leurs avions long-courriers , en espérant éventuellement un record de commandes lors de l' évènement qui se déroulera du 17 au 21 novembre . -Le différend se concentre sur la largeur des sièges offerts sur les vols long-courriers pour l' économie d' énergie - pas toujours ceux qui sont les plus demandés par les compagnies aériennes , mais dont les places spatiales bénéficient de la garantie d' efficacité pour les jets d' affaires les plus récents offerts par Airbus SAS et Boeing Co . , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , etc. Le conflit porte sur la largeur des sièges proposés sur les vols long-courriers aux passagers de la classe économique – qui ne sont pas toujours les plus par les compagnies aériennes , mais auxquels l' espace alloué est essentiel pour augmenter les gains d' efficacité dans les derniers appareils présentés par Airbus SAS et Boeing Co . -Le conducteur du véhicule , sa femme et sa mère sont tous morts au feu dans la voiture . Le conducteur de la voiture , son épouse et sa mère ont péri dans l' incendie de la voiture . -Les drogues ont été détectées lors de l' inspection des bagages . C' est lors de l' examen des bagages de la passagère que la drogue a pu être détectée . -Les drogues ont été ensuite transmises au bureau de la Gendarmerie royale du Canada ( GRC ) , qui a alors pris l' enquête . La drogue a ensuite été transportée au bureau de la Gendarmerie royale du Canada ( GRC ) qui a pris le relais dans cette enquête . -La gravité profonde des risques de narration fait également de lui un calvaire pour les spectateurs . La consciencieuse solennité de la narration pourrait également devenir une pénitence pour le public . -Le rédacteur en chef est personnellement impliqué dans le troisième . Le rédacteur en chef est personnellement impliqué dans le troisième . -L' élection s' est avérée un glissement de terrain , et les vedettes de l' information se sont pour réfléchir à la nature historique de l' heure . L' élection s' est terminée par une victoire écrasante et les présentateurs des journaux d' actualités ont pris le temps de réfléchir à la nature historique du moment . -L' ambassade a refusé de faire des observations au sujet de l' avis de la police . L' Ambassade a refusé de commenter la délivrance de l' avis par la police . -Les services d' urgence ont été lancés vers 1 heure du matin pour un incendie qui a eu lieu dans le sous-sol d' un restaurant indien à l' avenue , près de la jonction avec la rue . Les secours ont été appelés vers 1h , vendredi , en raison d' un feu qui s' était déclaré au sous-sol d' un restaurant de cuisine indienne sur l' avenue , près de l' intersection avec la rue . -L' énergie est grande , et le rythme rapide . L' énergie est haute , le rythme toujours rapide . -L' examen comportera également des " textes non lus " pour encourager une lecture plus large ; L' examen présentera également des « textes nouveaux » pour encourager une lecture plus diversifiée . -L' excitation de trouver leur programme préféré après avoir diminué , beaucoup sont déçus . l' excitation de retrouver leur programme favori , beaucoup ont été déçus . -L' expérience devrait durer six mois . L' expérimentation est prévue pour durer six mois . -Les expériences sont décrites dans la revue Medicine Science . Les expériences sont décrites dans la revue Science Medicine . -L' expert , qui a été dépêché sur les lieux par le parquet , a déclaré que la cause de l' explosion était purement accidentelle . L' expert dépêché sur place par le parquet assure que l' origine de l' explosion est purement accidentelle . -L' expiration de la période d' enquête prévue par la aux États-Unis et les décisions d' autorisation rendues dans les autres pays répondent à bon nombre des conditions nécessaires à la mise sur le marché . L' expiration du délai d' examen prévu par le aux Etats-Unis et les décisions d' autorisation délivrées dans les autres juridictions satisfont plusieurs des conditions nécessaires à la l' opération . -Le fait que le chien ait été repéré est incroyable . C' est incroyable que quelqu' un ait remarqué le chien . -La chute de Goma a conduit la Mission des Nations Unies en République démocratique du Congo , le pays le plus important du monde , à renforcer son mandat et à constituer une force d' intervention rapide composée de soldats de l' Afrique du Sud , du Malawi et de la Tanzanie . La chute de Goma a amené la Mission des Nations unies en RDC , la plus importante au monde en terme d' effectifs , à renforcer son mandat et à former une brigade rapide d' intervention composée de soldats sud-africains , et tanzaniens . -L' impôt fédéral lui-même , 18,4 cents par gallon , n' a pas augmenté en 20 ans . La taxe fédérale elle-même , qui est de 18,4 cents par gallon , n' a pas augmenté depuis 20 ans . -Les combats ont mis en lumière les dangers que les inspecteurs des armes chimiques doivent affronter lorsqu' ils sont confrontés à des délais serrés dans leur mission de débarrasser la Syrie de l' arsenal toxique au cours d' une guerre civile en cours . Les combats ont souligné l' importance des dangers auxquels sont confrontés les inspecteurs chargés des armes chimiques alors qu' ils doivent achever leur mission , qui consiste à débarrasser la Syrie de son arsenal toxique au milieu d' une guerre civile qui perdure , dans des délais serrés . -Le film a le sentiment d' un documentaire , de l' record de disques et d' une ancienne radio , aux choix musicaux des studios du jour et aux cafés où les chanteurs ont joué . Le film prend alors des allures de documentaire , du label de musique poussiéreux envahi de invendus habité par une vieille secrétaire , aux choix musicaux des studios de l' époque et aux cafés où se produisaient les chanteurs . -Le film ne glisse jamais en ou en - et c' est le problème . Le film ne verse jamais dans la ni le , et c' est bien le problème . -La dernière goutte s' est produite quand on a appris que le téléphone de la chancelière Angela Merkel avait été , peut-être pendant près d' une décennie . La révélation selon laquelle le téléphone de la chancelière allemande Angela Merkel avait été piraté , peut-être même pendant une décennie , a été la goutte d' eau qui a fait déborder le vase . -L' incendie " a causé des dommages importants à la structure du bâtiment " , a déclaré le chef des opérations du département des incendies de Montréal , Richard Breton . Le feu a " causé d' importants dommages à la structure du bâtiment " , a indiqué le chef aux opérations du Service de sécurité incendie de Montréal , Richard Bordeaux . -Le feu a commencé à l' avant , mais s' est étendu aux trois étages du bâtiment . Le feu s' est déclaré à l' avant , mais s' est élevé sur les trois étages de l' immeuble . -Le feu a commencé dans une maison sur quatre étages , située à l' arrière de la maison du Séminaire du Québec . Les flammes se sont déclarées dans une maison de trois logements répartis sur quatre étages située derrière la cour du Séminaire de Québec . -L' incendie a été maîtrisé une heure après que les pompiers arrivés à leur place pour intervenir . Le feu a été maîtrisé une heure après l' arrivée des pompiers , qui ont été nombreux à intervenir . -Les pompiers étaient extraordinaires . Les pompiers ont été . -Les pompiers étaient brillants . Les pompiers ont été super . -L' entreprise n' était pas à l' origine pour aider les contribuables à prendre des mesures fiscales . À l' origine , la société ne se pas à aider les automobilistes qui payent des taxes aux États . -La première est d' expliquer comment elle peut gagner du temps . La première est d' expliquer qu' elle peut faire gagner du temps . -La première est que lorsque leurs alliés - comme cela a été le cas récemment - ont appris que leurs gouvernements étaient , parfois au plus haut niveau . Le premier , c' est lorsque leurs alliés - et ça a été le cas récemment - apprennent que leurs dirigeants , parfois au plus haut sommet de leur Etat , ont été surveillés . -Les neuf premiers épisodes de s Wild West seront disponibles depuis le 24 novembre sur le site ou via sa demande de téléphones cellulaires et de comprimés . Les neuf premiers épisodes de s Wild West seront disponibles à partir du 24 novembre sur le site ou via son application pour téléphones et tablettes . -La première représentation d' un film dans le Mega à Narbonne a été interrompue jeudi de la fin de la récréation après que les auditeurs ont signalé une sensation de dans la gorge . Jeudi soir , la première séance d' un film a été interrompue au de Narbonne , par mesure de précaution , à la suite de à la gorge ressentis par les spectateurs . -La première étape est que la CGT et la un accord sur l' indemnité de départ pour tous les salariés . La première étape est que la CGT et Goodyear un accord sur les indemnités de départ pour tous les employés . -Le premier avion d' essai a été dévoilé en mars et a pris le vol pour la première fois en septembre après des mois de retard . Le premier avion d' essai a été dévoilé en mars et s' est pour la première fois en septembre après des mois de retard . -La première fois que j' ai eu ce sentiment très fort de sa présence , il était dans le siège passager pendant que je . La première fois ce fut le très fort ressenti de sa présence sur le siège passager alors que je . -Le lendemain , j' avais le sentiment qu' il était venu me voir dans mon rêve pour me dire qu' il partageait la joie que je me sentais . Le lendemain , j' avais le sentiment qu' il m' avait rendu visite dans mon sommeil pour me signifier qu' il partageait avec joie ce qui m' arrivait . -La première est son excellent " Fair Lady " ce soir et a annoncé la production mondiale de " Un à Paris " en guise de suivi . La première reprend , ce Noël , son excellent My Fair Lady et annonce pour le suivant la création mondiale de " Un Américain à Paris " . -L' ancienne agence secrète , autrefois connue sous le nom de , a fait preuve d' un manque de transparence et , au cours des derniers jours , de critiques très virulentes à l' égard de programmes de surveillance de l' étranger et de programmes nationaux de grande envergure , elle a été remise en question par les fichiers secrets secrets de l' Agence et par l' arrivée de l' , membre de l' Autorité palestinienne , qui s' est retrouvé à l' insu de l' auteur de la plainte . , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , etc. La NSA autrefois ultra secrète , un temps surnommée la « No Agency » , s' est retrouvée sous le feu des projecteurs et fortement critiquée au cours des derniers mois suite à une vague de révélations à propos de ses vastes programmes de surveillance des communications au niveau national et à l' étranger – qui font partie des fichiers secrets de la NSA volés à l' agence et divulgués par l' ancien consultant de la NSA , Edward . -Les quatre marques du groupe ont vu leur performance commerciale s' améliorer par rapport à octobre 2012 . Les quatre marques du groupe ont vu leurs performances commerciales s' améliorer par rapport à octobre 2012 . -Les quatre occupants des deux véhicules ont été blessés , mais pas . Les quatre occupants des deux véhicules ont été blessés , mais l' on ne craignait pas pour leur vie . -Le gratuit à la Fondation de la Raison est aussi passionné par le paiement des chauffeurs par mile parcouru . Les partisans du marché libre de la fondation Reason apprécient également l' idée de faire payer les conducteurs au mile . -Le financement , sur une période de quatre ans , a été décidé après des consultations gouvernementales avec le député Andrew Andrew de la République de Corée . Le financement , prévu sur 4 ans , a été décidé après des consultations du gouvernement avec le député indépendant de Tasmanie , Andrew . -Le gain dans les coûts unitaires est par rapport aux actuellement utilisés . Le gain en termes de coûts unitaires est faible comparé à l' appareil configuré avec 10 sièges par rangée actuellement en service . -Le lancement mondial sur les chaînes de groupe Disney n' est pas prévu jusqu' en 2014 , selon le communiqué de presse de sa junior division de Disney . Son lancement mondial sur les chaînes du groupe Disney n' est prévu qu' en 2014 , détaille le communiqué de sa division Disney Junior . -Le gouvernement accuse d' avoir déclenché les affrontements , l' accusant d' attaquer des soldats . Le gouvernement rend la responsable d' avoir déclenché les conflits , et l' accuse d' attaquer des soldats . -L' indice de la dette publique a été ramené à , dans la catégorie des investissements spéculatifs . La note des titres de dette publique du pays passe à " " , s' dans la catégorie des investissements spéculatifs . -Le grand-père et le locataire de la maison voisine , qui ont été affaiblis par l' explosion , ont été logés ailleurs par le chef des services sociaux . Le grand-père et le locataire de la maison voisine , fragilisée par l' explosion , ont été par le président du CPAS . -La grand-mère aurait allumé une cigarette alors qu' il y avait une fuite de gaz , La grand-mère aurait allumé une cigarette lors d' une fuite de gaz . -La grand-mère une cigarette lorsqu' il y avait une fuite de gaz et une accumulation de gaz . La grand-mère a allumé une cigarette alors qu' il y avait une fuite et une accumulation de gaz . -La grand-mère a été jetée dans la pièce par l' explosion et gravement brûlée . La grand-mère a été projetée par l' explosion et est grièvement brûlée . -Le groupe - une combinaison de médecins camerounais et d' étudiants en médecine étrangère - a effectué 700 chirurgies gratuites au cours de la dernière année , et ils savent que leur aide peut faire changer le monde de ceux qui en ont besoin . Le groupe , composé de médecins camerounais et d' étudiants en médecine , a pratiqué 700 actes chirurgicaux gratuitement au cours de l' année dernière , et il sait que leur présence peut faire toute la différence pour ceux à qui ils viennent en aide . -Le groupe a également recommandé que les dispositifs plus lourds soient placés en toute sécurité sous des sièges ou dans des au décollage et à l' atterrissage . Le groupe a également recommandé que les appareils plus lourds soient rangés en toute sécurité sous les sièges ou dans les compartiments supérieurs pendant le décollage et l' atterrissage . -Le groupe a produit millions de barils par jour au cours du trimestre , soit une augmentation par rapport au chiffre de millions de barils par jour enregistré un an auparavant . Le groupe a produit millions de barils d' équivalent pétrole par jour au cours du trimestre , en hausse par rapport aux millions de produits un an auparavant . -Le groupe qui détient des milliers d' anciens éditeurs de Nortel a fait l' objet de plusieurs poursuites judiciaires jeudi à l' encontre des fabricants de téléphones cellulaires , dont Google , l' entreprise qu' il a dans la vente aux enchères de Nortel . Le groupe qui possède des milliers d' anciens brevets Nortel a entamé une avalanche de poursuites judiciaires en matière de brevets jeudi à l' encontre de fabricants de téléphones portables , notamment Google , la société qu' il a au poteau lors de la vente aux enchères des brevets qui a eu lieu après la faillite de Nortel . -Le mécontentement croissant de la minorité sunnite , qui détient le pouvoir sous Saddam Hussein et qui se plaint désormais d' être politiquement marginalisée et d' être l' objectif poursuivi d' arrestations arbitraires , est favorable à cette explosion de violence . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Le mécontentement croissant de la minorité sunnite , au pouvoir sous Saddam Hussein , qui se plaint d' être marginalisée politiquement et d' être la cible d' arrestations injustes , a favorisé cette flambée de violences . -Le guide contient de l' information sur les écoles afin de sensibiliser les enseignants et le personnel aux allergies alimentaires des enfants et de les aider à faire face à une réaction allergique . Le guide contient des informations destinées aux écoles sur la façon de sensibiliser le corps professoral et le personnel aux allergies alimentaires chez l' enfant , et sur la façon de les traiter en cas de réaction allergique . -Les tirs semblent se rapprocher de la population . On a l' impression que les coups de feu se rapprochent des populations . -Dans le cadre de l' enquête sur la situation des droits de l' homme au Tibet , le président de la République de Serbie , M. , a fait part de son inquiétude à l' égard de la situation des droits de l' homme dans la région de l' Europe du Sud-Est . Le chef de la sécurité intérieure chinoise a accusé vendredi un groupe de séparatistes du Xinjiang d' être derrière l' attentat à la voiture piégée qui a fait cinq morts sur la place Tiananmen dans le centre de Pékin , lundi . -Le chef d' État a promis que la justice serait tout aussi ferme avec les puissants dictateurs que les " mouches " , les plus faibles , même si seulement une poignée de hauts fonctionnaires auraient été condamnés , y compris les anciens dirigeants du géant pétrolier . & nbsp ; & nbsp ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; – & # 160 ; & # 160 ; & # 160 ; & # 160 ; » Si le chef de l' Etat a promis que la justice serait aussi avec les puissants " tigres " qu' avec les " mouches " - les fonctionnaires subalternes - seule une poignée de responsables de haut rang ont été condamnés , parmi lesquels d' anciens cadres du géant pétrolier . -L' audience a été ajournée jusqu' au 20 décembre . L' audience a été ajournée jusqu' au 20 décembre . -L' hôtel compte trois cent cinquante employés . Trois cent cinquante personnes travaillaient à l' hôtel . -La chasse à cette technologie a conduit certains organismes d' Etat à un petit démarrage de la Californie appelé real . La recherche de cette technologie a amené certaines agences d' État à faire appel à une petite start-up de Californie du nom de True . -La glace les propriétés fluorescentes d' une , synthétisée par des scientifiques chinois La glace utilise les propriétés fluorescentes de la synthétisées par des scientifiques chinois . -La crème glacée réagit avec la langue du patient - augmentant le niveau de pH dans la protéine et la rendre . La crème glacée réagit avec la langue en augmentant le pH de la protéine pour la faire briller . -L' idée est de tous les feux de circulation à Bruxelles , à un coût estimé à un million d' euros . L' idée est , à terme , de tous les feux de signalisation bruxellois , pour un coût estimé d' environ un million d' euros . -L' idée était de présenter le moment précédant l' apparition de Greenwich Village , le quartier de New York qui a commencé à être l' épicentre du " de la ville " qui a créé les stars numériques , selon le journaliste Wald , un ami de Van . L' idée était de montrer le moment précédant l' avènement de Greenwich village , ce quartier de New York appelé à devenir " l' épicentre de l' essor de la musique folk qui engendrerait des stars internationales " comme l' écrit le journaliste Wald , ami de Van . -L' immense douleur qui m' a frappé m' a bien sûr échappé , mais il est toujours proche de moi . La peine immense qui m' a alors envahie s' est bien sûr atténuée mais il est toujours près de moi . -L' incident arrive quelques semaines après que six membres de Golden Dawn , dont le chef et le fondateur du parti , ont été responsables de faire partie d' une " organisation criminelle " , dans le cadre d' une opération dirigée contre le parti à la suite du meurtre d' un musicien par un de ses membres . L' incident intervient quelques semaines après l' inculpation de six députés d' Aube dorée , dont le chef et fondateur du parti , pour participation à " une organisation criminelle " dans le cadre d' une offensive contre ce parti lancée après la mort d' un musicien par un de ses membres . -L' incident a fait 42 victimes . L' incident a fait 42 blessés . -L' incident s' est produit dans le terminal 3 , provoquant une panique . L' incident se serait produit au terminal 3 , provoquant une vague de panique . -L' incident s' est produit jeudi - Halloween - lorsqu' il est habituel de s' habiller comme un monstre , mais il est difficile de croire qu' il s' agissait d' un . Cet incident s' est produit jeudi , jour de la fête d' Halloween , où il est de coutume de se déguiser en monstre , mais il était difficile de savoir s' il s' agissait d' un . -Les informations seront utilisées lors d' une grande assemblée sur la famille que le pape entend organiser l' année prochaine . L' information servira à un important rassemblement sur la famille que le pape François prévoit organiser l' an prochain . -La personne blessée a été immédiatement emmenée à l' hôpital , la même source l' a déclaré . La personne blessée a aussitôt été hospitalisée , selon la même source . -La paire inséparable - qui est la même hauteur - a été félicitée pour l' enregistrement , le style et le comportement lors du concours annuel organisé par le Centre équestre de , à près de Cardiff . Les deux inséparables – qui sont de la même taille – ont été salués pour la présentation , l' attitude et le style lors du concours annuel qui s' est déroulé au centre équestre , à près de Cardiff . -L' instabilité a ajouté aux frustrations avec le gouvernement , a déclaré Fernando Lima , responsable de , une entreprise de médias indépendante , qui a également beaucoup de victimes de la corruption , de la lenteur du développement et d' une vague récente d' enlèvements . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; L' instabilité a renforcé les frustrations que suscite le gouvernement , déclare Fernando Lima , directeur de , une société de médias indépendante , de nombreuses personnes étant également préoccupées par la corruption , la lenteur du développement et la récente vague d' enlèvements . -L' instruction a été donnée presque en même temps que la fin de l' du siège des Nations Unies à New York . Cette instruction a été donnée à peu près au même moment que celle mettant fin aux écoutes du quartier général de l' Onu à New York . -L' intérieur possède des sièges de course et des ceintures de 6 places , ainsi qu' un système d' interphone . L' intérieur dispose de sièges et de ceintures harnais à 6 points , ainsi que d' un système . -Le fonds d' investissement qui appartenait à l' immeuble devait faire un choix . Le fonds d' investissement qui était propriétaire de cette avait des choix à faire . -Le fonds d' investissement qui appartient à l' immeuble l' a vendu aux promoteurs qui la convertissent en résidences d' étudiants . Le fonds d' investissement qui possédait l' édifice l' a vendu à des promoteurs qui vont le transformer en résidences étudiantes . -Le Conseil de l' île a voté à l' unanimité en faveur de la création d' une réserve marine autour du minuscule archipel , qui sera , correspondant à sa zone économique exclusive . Le Conseil de l' île a voté à l' unanimité en faveur de la création d' une réserve marine autour du petit archipel , soit km2 qui correspondent à sa zone économique exclusive . -L' île , qui se trouve à l' est de l' archipel ( Polynésie française ) , est le dernier territoire britannique du Pacifique Sud , à mi-chemin entre la Nouvelle-Zélande et le Chili . Cette île , à l' est de l' archipel des ( Polynésie française ) , est le dernier territoire britannique du Pacifique Sud , à mi-chemin entre la Nouvelle-Zélande et le Chili . -La question est abordée dans les règlements qui exigent que les exploitants d' aéronefs veillent à ce que la sécurité soit maintenue en tout temps et que les passagers soient tenus de se conformer aux consignes de sécurité données par les membres de l' équipage . La question fait l' objet d' une réglementation qui oblige les exploitants d' aéronefs à garantir le maintien de la sécurité en permanence et les passagers à respecter les consignes de sécurité données par les membres d' équipage . -La question sera réglée ce soir . Le problème sera résolu ce soir . -La méthode journalistique n' est pas d' adopter une position morale , mais d' examiner l' importance et la pertinence de l' information et de permettre à chaque citoyen de prendre une opinion . La démarche journalistique n' est pas un positionnement moral , mais la recherche de l' intérêt et de la pertinence d' informations qui permettent à chaque citoyen de se forger une opinion . -Le juge a statué en août , la ville a violé la Constitution dans le cadre de son programme d' arrestation et de mise en examen des personnes . La juge avait statué en août que la ville avait violé la Constitution suite à la mise en œuvre de son programme visant à contrôler et interroger les citoyens . -Le juge a jugé que les agents de police violaient les droits civils de dizaines de milliers de personnes en mettant délibérément en cause des Noirs et des Hispaniques dans le cadre de leur programme . La juge avait statué que les officiers de police violaient les droits civils de dizaines de milliers de personnes en ciblant à tort des hommes noirs et d' origine hispanique dans le cadre de leur programme de contrôle et de fouille . -La juge a souligné qu' elle n' avait pas mis fin à la pratique , qui était constitutionnelle , mais qui a mis un terme à la façon dont le a mis en oeuvre ses arrêts . La juge a souligné qu' elle ne mettait pas fin aux pratiques de contrôle et de fouille , qui sont constitutionnelles , mais qu' elle la façon dont le mettait en œuvre ses contrôles . -Les juges ont marqué et au sujet de la façon dont ils ont été et de la façon dont ils ont été présentés dans l' anneau de présentation . Les juges ont remarqué que et présentaient bien et la façon dont ils ont défilé sur la piste . -Le jury a entendu jeudi que avait rencontré M. à propos d' une affaire avec une femme mariée alors qu' il s' était lui-même retrouvé avec Brooks , qui était mariée à l' époque . Le jury a entendu jeudi que avait confronté M. au sujet d' une liaison avec une femme mariée alors que lui-même fréquentait la Brooks , qui était également mariée à cette époque . -Le chaton a été trouvé à Argenteuil le 25 octobre et est décédé le 28 octobre . Le chaton a été trouvé le 25 octobre à Argenteuil et il est décédé le 28 octobre . -Ces dernières accusations reposent en partie sur des lettres écrites par l' avocat des frères de la Dominique , M. Emile Fournier , dans les années 80 , et sur les recherches effectuées dans ces archives au sujet du témoignage de M. Samuel Wilson Kennedy , ancien frère de la Croix qui a publiquement dénoncé les abus . Ces dernières accusations se basent entre autres sur des lettres rédigées par l' avocat des frères Sainte-Croix , Me Émile Perrin , dans les années 1990 , mais aussi par les recherches faites dans les archives à ce sujet par le frère Wilson Kennedy , un ancien frère de Sainte-Croix qui a dénoncé publiquement les sévices . -La loi a été vigoureusement opposée à la candidature de chacun des principaux candidats républicains à l' élection présidentielle de 2002 , y compris le candidat désigné Romney . Tous les grands candidats républicains à l' élection présidentielle en lice au cours des primaires de 2012 , y compris le candidat désigné Romney , se sont farouchement opposés à la loi . -La chute de l' avocat est survenue après que la police de Scarborough a découvert qu' il était en contact régulier avec en février 2011 . La chute de l' avocat est survenue après que la police , enquêtant sur Scarborough , a découvert qu' il avait été régulièrement en contact téléphonique avec en février 2011 . -L' avocat est retourné chez lui pour découvrir les drogues et il y a eu neuf communications entre eux . L' avocat est retourné chez lui , a découvert la drogue et ils se sont à neuf reprises . -Le chef du mouvement des talibans pakistanais , , a été tué lors d' une frappe aérienne américaine au Pakistan vendredi , selon les services de sécurité . Le chef du mouvement des taliban pakistanais , , a été tué dans une frappe d' un drone américain vendredi au Pakistan , a-t-on appris auprès des services de sécurité . -Les fuites ont placé le gouvernement américain au centre d' une tempête diplomatique avec ses alliés . Les fuites ont mis le gouvernement américain au centre d' une tempête diplomatique avec ses alliés . -Les locataires étaient contre cela et souhaitaient des conditions plus longues . Les locataires étaient contre et voulaient également des durées plus longues . -La lettre étend l' offre de coopérer avec les autorités allemandes " lorsque les difficultés de cette situation humanitaire ont été résolues " . La lettre avance une offre de coopération avec les autorités allemandes « lorsque les difficultés de cette situation humanitaire auront été résolues » . -La liste a souffert de la chute du dollar par rapport au yen , qui est nuisible pour les valeurs à l' exportation , et de l' avertissement du rendement annuel émis par Sony le jeudi . La cote a pâti du recul du dollar face au yen , préjudiciable aux valeurs exportatrices et de l' avertissement sur son résultat annuel lancé par Sony jeudi . -Le petit garçon , qui était resté avec son grand-père et son partenaire , était de son côté et a subi moins de brûlures graves . Le petit garçon , qui chez son grand-père et sa compagne , était à ses côtés et a été brûlé plus légèrement . -L' objectif principal de l' association est de permettre aux otages de porter plainte et d' amener leurs ravisseurs à la Cour pénale internationale . Son objectif principal : obtenir la possibilité pour les otages de porter plainte et d' amener leurs preneurs d' otages devant la Cour pénale internationale . -Les principaux marchés boursiers européens , à l' exception de Londres , sont en chute libre à ce vendredi . A l' exception de Londres , les principales Bourses européennes évoluent en baisse vendredi à , par certaines nouvelles de sociétés décevantes , tandis que est attendue en hausse . -La majorité des Américains - 158 millions d' entre eux en fait - célébreront l' Halloween cette année , réalisant un total de 6,9 milliards de dollars sur les bonbons , les costumes et les décorations , selon la National Retail Federation . La majorité des Américains – 158 millions d' entre eux en fait – Halloween cette année , dépensant un total de 6,9 Md $ en bonbons , et décorations , selon la National Retail Federation . -La majorité des foyers de soins palliatifs refuseront d' aider les patients à mourir , selon le vice-président de l' Alliance of Care Homes , Suzanne . La majorité des maisons de soins palliatifs refuseront d' aider les patients à mourir , selon la vice-présidente de l' Alliance des maisons de soins palliatifs , Suzanne . -La majeure partie de la croissance dans les années à venir viendra de ses systèmes de gaz naturel liquéfié en Australie . La majeur partie de la croissance dans les années à venir proviendra de ses projets de gaz naturel liquéfié en Australie . -La majorité de la violence est attribuée à l' État islamique d' Iraq et au ( ) , groupe affilié à Al-Qaida ( extrémistes sunnites ) , qui est également impliqué dans la guerre civile en Syrie . Une bonne partie des violences a été imputée à l' État islamique en Irak et au Levant ( ) , un groupe affilié à Al-Qaida ( extrémistes sunnites ) , également impliqué dans la guerre civile en Syrie . -Le fils violent d' un homme âgé de neuf ans s' est rendu en Australie avec lui et a été depuis le décès de son père , le mercredi , par un cousin adulte qui était également sur le navire . Le fils de l' homme , âgé de 9 ans , se rendait en Australie avec lui et a trouvé quelque réconfort depuis le décès de son père mercredi auprès d' un cousin adulte qui était également à bord . -L' homme concerné , Rob Ford , a toujours nié avoir pris le crack , mais a admis être en partie du cannabis . L' intéressé , Rob Ford , a toujours nié avoir consommé du crack , tout en reconnaissant avoir un penchant pour le cannabis . -L' homme de nombreuses curiosités est - un peu - centré sur une idée unique . L' homme des curiosités multiples tournait , légèrement , autour d' une même idée . -L' homme a succombé à ses blessures graves . L' homme n' a pas survécu à ses graves blessures . -Les maîtres de la cérémonie sont souvent des personnes qui ont déménagé dans ce nouveau genre d' emploi . Les maîtres de cérémonie sont souvent des gens qui se sont . -La pression des médias était énorme . La pression médiatique est énorme . -La réunion a lieu à l' hôtel de ville , au centre de Manhattan . Le rendez-vous a lieu à l' Hôtel de Ville , dans le centre de Manhattan . -La fusion réunit la deuxième plus grande agence du monde , , et la troisième plus importante , la . La fusion rapproche en effet la deuxième agence mondiale , , et la troisième , . -La région métropolitaine de Montréal connaît également sa part de pannes , avec près de 7,000 maisons dans la ville et à Laval , sans pouvoir . La grande région de Montréal connaît aussi son lot de pannes avec près de 7000 foyers privés de courant dans la métropole et à Laval . -Le géant de l' exploitation minière , qui conteste le paiement d' une taxe de vente au carbone de 6 millions de dollars , a fait la demande en tentant de s' interroger sur la question de savoir s' il avait un conflit . Le magnat de l' extraction minière , qui conteste le paiement d' une taxe carbone d' un montant de 6 M $ , a fait valoir cet argument pour essayer d' écarter les questions concernant un éventuel conflit . -Le ministre a été libéré sous condition . Le pasteur a été relâché sous conditions . -Le ministre a insisté sur le fait que le gouvernement de la Tasmanie serait soumis à un régime strict de déclaration et de responsabilisation . La ministre a affirmé que le gouvernement de Tasmanie serait confronté à un régime strict de préparation des rapports et de reddition de comptes . -Les ministères demandent à toute personne qui a été , , ou à la suite d' une membrane ou d' une peau sur la peau ou qui a été infectée par un animal qui pourrait avoir des contacts entre le 00 et le 28 mars , au contact du groupe de 19 à 95 V 00 entre et 18h du 1er novembre au 31 décembre 2010 . Pour plus de renseignements , veuillez contacter le service de l' assistance à la clientèle au ou par téléphone au ( 514 ) ou par téléphone au ( 514 ) ou par téléphone au ( 514 ) ou par téléphone au ( 514 ) ou par téléphone au ( 514 ) , poste , , , , , , , , , , , , , , , , , , , , , , , , , , , , , et . Les ministères appellent à présent les personnes qui auraient été , , , ou sur une muqueuse ou sur une peau lésée par ce chaton ou dont l' animal aurait été en contact avec ce chaton entre le 8 et le 28 octobre à contacter le entre 10 heures et 18 heures à partir du 1er novembre . -Les mineurs , âgés de 9 à 13 ans , sortaient de la porte de la porte pour réclamer des friandises le soir de la fête nocturne , lorsque : Les mineurs âgés de 9 et 13 ans faisaient du porte à porte pour réclamer des bonbons le soir d' Halloween . -Le monumental " de mémoire " ( sites de mémoire ) , auquel son nom est attaché , contribue également à cette image floue . A cette image incertaine le monument des " Lieux de mémoire " , auquel il a attaché son nom , a aussi contribué . -Les plus enthousiastes sont l' Oregon , qui recrute 5,000 pilotes dans la plus grande expérience du pays . Le plus déterminé est l' Oregon , qui a mobilisé 5 000 conducteurs pour mener l' expérience la plus importante du pays . -L' affaire la plus récente a été celle de l' ancien dirigeant du PCC à Chongqing , Bo , qui a été condamné à la prison pour la vie en septembre pour corruption et abus de pouvoir . Le cas récent le plus spectaculaire a été celui de l' ancien chef du PCC à Chongqing , Bo , condamné à vie en septembre pour corruption et abus de pouvoir après avoir visé les plus hautes fonctions au sein de l' Etat . -Le , Jonathan , âgé de 26 ans , de , ne portait pas de casque . Le , Jonathan , 26 ans , de , ne portait pas de casque . -Le mouvement est également soutenu par deux anciens secrétaires des transports des États-Unis qui , dans un rapport de 2011 , ont exhorté le Congrès à aller de l' avant dans la direction de l' . Le mouvement est également soutenu par deux anciens secrétaires américains aux Transports qui , dans un rapport de 2011 , le Congrès à aller dans la voie du paiement au mile . -Le mouvement est souvent accusé par les autorités chinoises de troubles sporadiques au Xinjiang , bien que de nombreux experts remettent en cause leur véritable influence . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Ce mouvement est souvent désigné par les autorités chinoises comme étant responsable des troubles sporadiques au Xinjiang , mais son influence réelle est mise en doute par plusieurs experts . -La musique recommence , et la séquence se poursuit . Et la musique de repartir , et la séquence de reprendre . -Les comédies musicales sont préparées à Paris et appréciées dans les provinces . La mayonnaise est montée à Paris et par les provinciaux . -Le directeur du service national , James , a défendu l' espionnage à l' encontre des alliés comme il le fallait et a dit qu' il était courant de part et d' autre . Le directeur des services de renseignement , James , a justifié l' espionnage des alliés en indiquant qu' il était nécessaire et qu' il s' agissait d' une pratique courante des deux côtés . -Le bénéfice net de sa filiale principale , Câble , s' est établi à 43,9 M $ , ou à 90 ¢ par action , soit M CAD , soit 93 ¢ par , pour la même période l' an dernier . Le bénéfice net de sa principale filiale , Câble , a été de 43,9 millions , ou 90 ¢ par action , en recul par rapport aux 45,7 millions , ou 93 ¢ par action , de la même période l' an dernier . -La nouvelle parution de est en retard , parce que l' édition a pris plus de temps que prévu . La sortie du nouveau a pris du retard en raison de son montage trop long . -Les nouveaux frais de port sont pour les sacs dans la , de sorte que de petits sacs sous le siège seront toujours libres . Les nouveaux frais de bagages à main concernent les sacs rangés dans le compartiment supérieur , donc les petits sacs installés sous le siège seront toujours gratuits . -La nouvelle loi n' oblige pas les parents à déclarer un sexe de ces enfants , ce qui permet aux parents de déclarer " non valides " ou " non spécifiés " dans leur certificat de naissance . Dans le cadre de la nouvelle loi , les parents ne seront plus tenus de déclarer le sexe de leur enfant , les autorisant à inscrire un sexe « indéterminé » ou « non spécifié » sur son certificat de naissance . -Le nouveau vaccin contient un virus inoffensif qui a été conçu pour transmettre l' information génétique afin de produire des anticorps . Le nouveau vaccin contient un virus inoffensif qui a été modifié pour transporter les informations génétiques nécessaires pour concevoir des anticorps . -La journée s' est terminée le même jour que le Credit Suisse a annoncé qu' il avait donné un coup de fil à son groupe de négociation de fonds de change à Londres cette semaine après qu' il eut subi une perte de près de 6 millions de dollars à la fin de l' année dernière . La nouvelle est arrivée le jour même où le Crédit Suisse a annoncé avoir renvoyé un trader travaillant dans son bureau des fonds négociés la Bourse de Londres cette semaine après qu' il a causé une perte de 6 M $ en fin d' année dernière . -Les nouvelles ont été publiées le jeudi par le Catholic National Reporter , un journal catholique indépendant aux États-Unis . La nouvelle a été rapportée jeudi par le " National Catholic Reporter " , un journal catholique indépendant des États-Unis . -Le journal , qui cite des documents publiés par l' ancien consultant irlandais , a dit que le programme " " , réalisé conjointement avec le bureau britannique de la NSA , les , a permis aux deux organismes de recueillir des données des câbles à fibre optique utilisés par les géants de l' Internet . Le journal , qui cite des documents obtenus auprès de l' de la NSA Edward , affirme que le programme baptisé " " , et mené avec l' homologue britannique de la NSA , le , permet à ces deux agences de récupérer des données depuis les fibres optiques utilisées par les géants d' internet . -Le journal a confié à l' enquêteur privé Glenn à la de et " " lui même , mais a également appelé et a laissé son , la cour a entendu . Il a été signalé à la cour que le journal a chargé Glenn , un détective privé , de la boîte vocale de et de « planter en permanence devant sa porte » , mais l' a également appelée et lui a laissé des messages vocaux . -Le lendemain matin , l' équipe commence à rencontrer des centaines de patients . Le lendemain matin , l' équipe commence à recevoir des centaines de patients . -La nuit précédente serait pratiquement vide . La nuit qui précède est quasiment blanche . -L' ancien chef de l' Elysée , qui a pris un verre le mardi dernier avec son équipe , sa famille et ses amis , a fini par un de douleur " , a déclaré François Hollande . Le désormais ancien chef des cuisines de l' Elysée , qui a pris un verre mardi soir avec son équipe , sa famille et des amis , part avec un pincement au cœur " Hollande est venu me saluer " . -Le fonctionnaire n' a pas précisé ce que signifiait la cible , mais a déclaré qu' il y en avait au moins un . Le responsable n' a pas précisé quelle était la cible , mais a dit qu' il y en avait au moins une . -La vieille fonction consistant à ne pas dire . La base connue de ce que l' on montre et que l' on ne dit pas . -La seule chose que nous ne voulons pas consommer , de maïs , et pourtant près de 35 millions de livres sont vendus à l' Halloween , selon la National Association . Les bonbons maïs sont la seule chose qu' on ne veut pas manger ; et pourtant presque 16 millions de kg sont vendus pour Halloween , selon la National Association . -Celui qui aux bords était debout au centre du centre . Le de se tenait au centre du centre . -Le seul espoir est d' améliorer la situation internationale . Le seul espoir , c' est une amélioration de la conjoncture internationale . -Le seul numéro que j' ai mentionné au ministre est 333 . Le seul nombre que j' ai mentionné au ministre , c' est 333 . -La seule autre compagnie aérienne à avoir une telle redevance est la Air de la Hongrie , a déclaré l' expert-conseil en vol Jay Sorensen , qui suit de près les frais supplémentaires . La seule autre compagnie imposant de tels frais est la compagnie hongroise Air , a déclaré le consultant auprès de compagnies aériennes Jay Sorensen , qui suit de près les frais en supplément . -La seule centrale électrique de la bande de Gaza a cessé de fonctionner le vendredi 1er novembre , à la suite de l' épuisement de sa réserve de carburant , l' Autorité palestinienne pour l' enclave palestinienne annoncée . La seule centrale électrique de la bande de Gaza a cessé de fonctionner vendredi 1er novembre après un épuisement de ses stocks de carburant , a annoncé l' Autorité de l' énergie de l' enclave palestinienne . -Le seul recours consiste à réduire les salaires et les dépenses publiques - encouragés par Berlin . Le seul recours est de faire des coupes sombres dans les salaires et les dépenses publiques – encouragées par Berlin . -L' opération avait pour but d' empêcher les futures attaques terroristes en utilisant ce tunnel , a expliqué une déclaration militaire israélienne . L' opération visait à empêcher de futures attaques terroristes utilisant ce tunnel , a expliqué un communiqué militaire israélien . -Les opinions exprimées dans ce commentaire sont celles de Morris . Les points de vue exprimés dans ce commentaire sont uniquement ceux de Morris . -L' arriéré de commandes de l' unité de transport s' établissait à 32,6 milliards de dollars au 30 septembre , soit une augmentation de 36 % par rapport au 31 décembre . Le carnet de commandes de la division Transport totalisait 32,6 Md $ au 30 septembre , soit une légère hausse par rapport au 31 décembre de l' année précédente . -L' organisation , appelée Vision , organise des écoles au cours de la dernière année pour aider les élèves de toutes les origines à trouver des références communes qui ne se limitent pas à la langue française . L' organisme Vision diversité se promène depuis un an dans les écoles pour aider les élèves de toutes origines à découvrir des repères communs qui ne se limitent pas à la langue française . -L' organisation a simplement indiqué que la médiation faisait partie d' un processus découlant d' une plainte . L' organisation a seulement signifié que la médiation fait partie du processus lors d' une plainte . -L' autre grande question est compréhensible : les parents ne veulent pas que leurs jeunes filles aient des périodes de repos . L' autre grand problème est compréhensible : Les parents ne veulent tout simplement pas que leurs très jeunes filles aient leurs règles . -L' autre soir , il m' a demandé de me dire : je ne sais pas , et je lui ai dit qu' il était " sorti " , mais dans une voix si grave que je me suis réveillé et se , et je me suis tourné vers la lampe de chevet et ai regardé autour de lui , convaincu qu' il allait venir . L' autre nuit , il m' a demandé si j' allais bien , je lui ai dit non , il me réponds " je " mais d' une voix tellement réelle que je me suis en sursaut , et j' ai allumé la lampe de chevet en regardant partout , persuadée qu' il allait arriver . -Les autres tests ont été effectués à dans les jours suivant mon retour : des radiographies , des examens de tous types et une nomination avec un psychiatre . Le reste a été fait au les jours suivants mon retour : radios , examens en tous genres et rencontre avec un psychiatre . -Les parents d' un adolescent de Géorgie , dont le corps a été retrouvé dans un de lutte contre la violence dans son centre d' entraînement de l' école secondaire , ont affirmé que son fils avait été assassiné , a déclaré l' avocat de la famille jeudi . Les parents d' un adolescent de Géorgie , dont le corps a été retrouvé dans un tapis de lutte roulé dans le gymnase de son lycée , pensent que leur fils a été assassiné , a déclaré l' avocat de la famille jeudi . -L' adoption de ce projet de loi s' inscrit dans une série de mesures visant à renforcer la sécurité nationale à la suite de l' attentat perpétré en septembre contre des tireurs islamistes dans le centre commercial de . L' adoption du projet de loi intervient en même temps qu' un train de mesures visant à renforcer la sécurité nationale dans le sillage de l' attaque perpétrée par des islamistes armés dans le centre commercial de en septembre dernier . -Les brevets couvrent la technologie qui aide à faire correspondre les possibilités de recherche Internet avec la publicité pertinente , le procès défendu , qui est le cœur de la recherche de Google . Les brevets contiennent des technologies qui permettent d ’ afficher une publicité en rapport avec des termes de recherche sur Internet , ont indiqué les conclusions du procès , ce qui est le cœur de métier de Google . -Le département des pharmacies a été sérieusement suite à de multiples départs dus à la retraite , au congé de maternité ou , tout simplement , à la démission . En raison de multiples départs à la retraite , de congés de maternité ou de départs tout court , le département de pharmacie s' est trouvé fortement déficitaire en personnel . -Le pipeline est conçu pour une capacité annuelle de 2,2 billions de pieds cubes de gaz naturel . Le gazoduc a été conçu pour une capacité annuelle de 63 milliards de mètres cubes de gaz naturel . -L' avion était complet , a-t-il dit , " il est monté à bord " . L' avion était plein , a-t-il ajouté , « les passagers sont montés à bord à la vitesse de l' éclair » . -La police , qui avait été avertie , a réussi à séparer les belligérants et s' assurer que les blessés reçoivent des soins médicaux . , les gendarmes se sont rendus sur place pour séparer les belligérants et veiller à ce que les blessés soient pris en charge . -La police n' est actuellement pas en mesure de fournir des informations sur l' identité des victimes ni sur leur affiliation politique . La police n' était pas en mesure pour l' instant de donner des informations sur l' identité des victimes et leur éventuelle appartenance politique . -La police est intervenue très rapidement et l' individu suspecté a été arrêté sur le toit d' un parking de l' aéroport . Très vite les forces de police sont intervenues et un homme suspecté d' être le tireur a été appréhendé sur le toit d' un parking de l' aéroport . -La police a indiqué qu' elle avait également indiqué qu' elle des colis à Rob Ford , conformément à de nouveaux documents de la cour . La police a dit avoir aussi observé cet homme délivrant des colis à Rob Ford , indiquent de nouveaux documents de cour . -Les dirigeants politiques de l' espèrent parvenir à un accord . Sur place , les dirigeants politiques du espèrent parvenir à un accord . -Le sondage a été envoyé à chaque conférence nationale des évêques à la mi-octobre , avec des instructions pour assurer un maximum de réponses possibles . Le sondage a été envoyé à la mi-octobre à chaque conférence nationale des évêques , avec la consigne d' obtenir le plus de réponses possible . -Le sondage demandera aux participants comment les prêtres traitent avec les couples gais et leurs enfants , et comment ils traitent les hommes et les femmes qui vivent ensemble sans être unis par des liens de mariage . Le sondage demande aux participants comment les prêtres s' occupent des couples gais et de leurs enfants , et comment ils traitent les hommes et les femmes qui vivent ensemble sans être unis par les liens du mariage . -Le changement possible était un secret bien gardé à l' intérieur de l' infrastructure de campagne de Chicago et à l' intérieur du bureau ovale . La possibilité d' un échange est un secret qui a été farouchement gardé au sein du QG de campagne de Chicago et à l' intérieur du Bureau ovale . -Le potentiel était là où elle aurait pu être gravement blessée ou pire . Elle aurait pu être gravement blessée ou pire . -Les téléphones mobiles prépayés sont ensuite supprimés par la suite . Les téléphones portables prépayés sont ensuite jetés . -Le président du Conseil central des Juifs d' Allemagne , Dieter , a déclaré qu' il avait été choqué par la révélation . Le président du Conseil central des juifs d' Allemagne , Dieter , s' est déclaré choqué par cette révélation . -Le communiqué de presse a simplement déclaré qu' il était " blessé " , sans dire ce qu' il en était . Le communiqué indique simplement qu' il est " blessé " , sans préciser la nature de la blessure . -La théorie générale sur la façon de traiter les enfants atteints d' un trouble est proposée par le Dr John Money , de l' Université Johns Hopkins , qui affirme que le sexe est . La théorie dominante sur la façon de traiter les enfants pourvus d' organes sexuels ambigus a été lancée par le Dr John Money , de l' université , qui considérait que le genre est . -La veille , le Premier ministre pakistanais , Sharif , a informé le gouvernement britannique lors d' un voyage à Londres que des discussions avaient été entamées avec et le mouvement taliban pakistanais ( ) , le mouvement des talibans pakistanais . La veille , le Premier ministre pakistanais Sharif , en déplacement à Londres , avait informé le gouvernement britannique que des discussions avaient été engagées avec le e Taliban Pakistan ( ) , le mouvement des taliban pakistanais . -Le prix du pétrole a continué de chuter le vendredi , en raison des inquiétudes suscitées par les excédents de production , ce qui témoigne d' un rapport que le secteur chinois de la fabrication d' armes à haut risque est en train de renforcer . & nbsp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; & amp ; Le prix du pétrole a continué à baisser vendredi alors que les préoccupations concernant l ’ importance des stocks compensent un rapport montrant que le secteur manufacturier chinois gourmand en électricité est en train de se renforcer . -Le principe de l' aide à la sécurité pour l' Irak est soutenu par des sénateurs républicains et démocrates influents . Le principe d' une aide accrue à l' Irak en matière de sécurité est soutenu par d' influents sénateurs républicains et démocrates . -La priorité doit être l' emploi et la croissance en Grande-Bretagne et , comme le montrent les résultats de nos enquêtes , pour les affaires , cela signifie une nouvelle concentration sur le commerce et une modification fondamentale de l' approche de la politique de Bruxelles . La priorité doit être l' emploi et la croissance en Grande-Bretagne et , comme les conclusions de notre enquête le montrent , pour les entreprises cela signifie mettre à nouveau l' accent sur le commerce et un changement fondamental de l' approche réglementaire de Bruxelles . -Les bénéfices des activités de production ont diminué légèrement au cours du troisième trimestre , tandis que les bénéfices des activités en aval ( y compris le raffinage et la production chimique ) ont chuté de 45 % à 380 millions de dollars . Les bénéfices tirés des activités de production ont légèrement diminué au troisième trimestre , tandis que les profits des activités en aval ( dont le raffinage et la production chimique ) ont chuté de 45 % à 380 millions de dollars . -Le projet devra être ratifié par le gouverneur de Pitcairn , qui est basé en Nouvelle-Zélande , et le gouvernement britannique . Le projet doit être accepté par le gouverneur anglais de Pitcairn , résidant en Nouvelle-Zélande , et par le gouvernement britannique . -La protéine que nous utilisons dans la glace réagit avec votre langue au pH neutre . La protéine que nous utilisons dans la glace réagit avec la langue à pH neutre . -Le cinéaste provocateur a peut-être demandé à ses victimes de s' attaquer à une partie de son visage , mais il a dû s' adresser à des professionnels du cinéma pour les scènes les plus noires , avant de fusionner les corps numériques : la moitié principale est l' étoile , la moitié inférieure est le double . Si le cinéaste provocateur a demandé à ses acteurs de mettre leur pudeur au placard , il a eu recours à des professionnels du X pour les scènes de sexe les plus , avant de fusionner tous les corps numériquement : au-dessus de la ceinture , c' est la star ; au-dessous , la doublure . -Le public pourra profiter des prouesses techniques de jeunes patineurs , dont certains , comme la jeune étoile de , Lorenzo , ont déjà participé à des compétitions de haut niveau . Le public pourra admirer les prouesses techniques de jeunes qui , pour certains , fréquentent déjà les compétitions au plus haut niveau , à l' instar du jeune prodige Lorenzo . -La publication a été publiée le jour où le Premier Ministre iraquien a rencontré le Président américain . Cette publication a lieu le jour où le Premier ministre irakien est reçu par le Président américain . -La marche à suivre est le fonds d' affectation spéciale de la Route du Pacifique , financé par les taxes que les Américains paient à la pompe à gaz , est cassée . & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; La pression vient du fait que le Highway Trust Fund du pays , financé avec les taxes que les Américains paient à la pompe , est financièrement à sec. -Le raid a eu lieu dans des circonstances presque identiques à celles du 5 juillet : à cette occasion , il y avait aussi un représentant américain avisé qui avait confirmé à CNN une attaque israélienne qui visait les missiles sol-air à destination de Damas par la Russie . Le raid s' est déroulé dans des circonstances à peu près identiques à celui qui avait eu lieu le 5 juillet : à l' époque , c' est aussi un responsable américain anonyme qui avait confirmé à CNN une attaque israélienne , visant cette fois-ci des missiles , livrés par la Russie à Damas . -La compagnie d' extermination du rat réalise deux opérations par an , et nous prenons en charge chaque fois que nous recevons une plainte de la part de la population . L' entreprise de réalise deux passages par an , et en plus , nous intervenons , à notre charge , à chaque demande des habitants . -Le rythme des progrès des troupes gouvernementales est sans précédent . Le rythme de progression des troupes gouvernementales est aujourd' hui sans précédent . -Le des tirs d' armes légères pourrait bien être entendu à l' arrière-plan . On entendait très nettement derrière lui des nourris d' armes légères . -Les modèles rappelés ont été construits entre le 1er août et le 10 septembre . Les modèles rappelés ont été construits entre le 1er août et le 10 septembre . -La réduction des marges de raffinage affecte l' ensemble du secteur . Cette baisse des marges de raffinage touche l' ensemble du secteur . -Le refus des neuf États a commencé avec le Texas , qui a refusé de mettre en œuvre ces mesures dans les installations de la Garde nationale en raison d' un conflit entre la loi et la loi fédérale sur le mariage entre personnes du même sexe . La des neuf États a débuté par le Texas qui refuse de mettre en oeuvre ces prestations sur les implantations de la Garde nationale , en raison d' un conflit entre la loi et la loi fédérale sur le mariage homosexuel . -La relation que nous parlons n' est pas simplement un trafiquant de drogue , mais un trafiquant de drogue qui donne son propre avocat à la drogue . La relation dont nous parlons n' est pas simplement un trafiquant de drogue , mais un trafiquant de drogue fournissant de la drogue à son propre avocat . -Le représentant ajoute : " Peut-être la confondre avec quelqu' un d' autre . Le représentant a ajouté : « Peut-être qu' elle le confond avec quelqu' un d' autre . » -Le rapport donne également un aperçu de la discrimination dont sont victimes les et les transsexuels dans le domaine de l' emploi , ainsi que les niveaux de harcèlement , de violence et de crimes de discrimination . Le rapport donnait également un aperçu de la discrimination à laquelle étaient confrontées les personnes et transgenres en matière d' emploi , ainsi que l' ampleur des actes de harcèlement et de violence et des crimes motivés par des préjugés . -Le rapport comprend également une liste des symptômes typiques signalés par les enfants qui ont une réaction allergique . Le rapport comprend également une liste des symptômes typiques communiqués par les enfants qui ont une réaction allergique . -La recherche abordera l' inclusion sous tous les angles : la langue , l' éducation , le social et la culture . Les recherches porteront sur l' inclusion sous tous ses angles : linguistique , scolaire , social et culturel . -Les chercheurs disent que les résultats pourraient amener les propriétaires , les vétérinaires et les formateurs à mieux comprendre les émotions de leur animal . Les chercheurs indiquent que les résultats pourraient permettre aux propriétaires , aux vétérinaires et aux de mieux appréhender les émotions de leurs animaux . -Les habitants de Bruxelles méritent mieux que de voir leur argent perdu par un ministre démocrate chrétien et flamand , qui exécute le nouveau programme de l' Alliance elle-même . Les méritent mieux que de voir leur argent gaspillé par une ministre CD & V qui réalise elle-même le programme de la . -Les restrictions sont également devenues de plus en plus difficiles à appliquer à mesure que l' utilisation des dispositifs est devenue omniprésente . Les restrictions sont également de plus en plus difficiles à appliquer car l' utilisation des appareils est devenue omniprésente . -Le résultat est des corps qui semblent disposés en avant des dessins . Résultat : les corps semblent à plat devant les dessins . -La révélation est la plus notable des images de Mark et de la toute première campagne de 2012 de John , " Double Down : Game Change 2012 " . Cette révélation est l' évènement le plus marquant qui figure dans le très attendu livre de la campagne 2012 de Mark et John , « Double Down : Game Change 2012 » . -La course a été fermée jusqu' à nouvel ordre . Elle a été fermée jusqu' à nouvel ordre . -Le risque couru par les Américains pourrait être de deux ordres . Le risque couru par les Américains peut être double . -Le parti politique , qui est la force politique dominante depuis 1975 , et le se sont mutuellement opposés à la tension . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Le , le parti au pouvoir et la principale force politique depuis 1975 , et la se rejettent mutuellement la responsabilité des tensions . -La connaissance de la façon dont un enfant conçoit une identité liée au sexe n' est pas très précise . La science qui consiste à savoir comment un enfant développera une identité sexuelle n' est pas très précise . -La seconde est d' offrir des services personnalisés : dans quelques années , un serveur vocal pourra reconnaître votre voix et vous proposer des solutions personnalisées . La deuxième est d' offrir des services personnalisés : dans quelques années , un serveur vocal pourra reconnaître votre voix et vous proposer des options sur mesure . -Le second des cinq avions d' essai devrait prendre le vol dans les semaines à venir , et les autres suivront peu après , selon les dires de l' entreprise . Le deuxième des cinq avions d' essai devrait s' envoler dans les semaines à venir , les autres suivront peu de temps après , a indiqué la société . -La seconde a essayé de tuer le premier , mais elle n' a pas réussi à l' effacer ; au contraire , elle a su se moquer de son caractère sacré et être si ardemment attachée à l' unité et à l' indivisibilité qu' elle a coupé la tête de la personne qui était l' incarnation puissante et des deux . . La seconde a voulu la mort de la première , mais sans parvenir à la nier ; soucieuse au contraire d' en retrouver la et d' autant plus avide d' unité et d' indivisibilité qu' elle avait coupé la tête à celui qui en était l' évidente et puissante incarnation . -La section du chemin périphérique où l' accident a eu lieu a été fermée à la circulation jusqu' à 9h pour la réouverture de la route et pour qu' un expert puisse déterminer les circonstances de l' accident . Le ring a été fermé à la circulation , jusqu' à 9h00 , à l' endroit de l' accident , le temps que la route soit et qu' un expert détermine les circonstances de l' accident . -Les secteurs les plus touchés sont les Laurentides , la Montérégie et l' Outaouais , dont les , les et les ont respectivement plongé dans les ténèbres . Les secteurs les plus touchés sont les Laurentides avec 15 042 abonnés plongés dans le noir , la Montérégie avec 13 464 et l' Outaouais avec . -La mise en place d' une technologie d' interception à la fine pointe de la technologie au cours des dix dernières années par les États-Unis - et par la France - a été officiellement justifiée par la lutte contre le terrorisme . La mise en place depuis en gros dix ans d' outils technologiques d' interception très puissants par les Etats-Unis , mais aussi par la France , a officiellement été justifiée par la lutte contre le terrorisme . -L' agence de sécurité nationale a déclaré vendredi dernier qu' il s' agissait d' un bug qui a fait passer son site Web public à quelques heures , et non pas des pirates informatiques comme ceux annoncés en ligne . La National Security Agency œuvrant dans l' ombre a déclaré vendredi que c' était un bug qui avait causé la panne de son site public pendant quelques heures , et non des hackers comme certains l' avaient prétendu en ligne . -La boutique , qui est strictement réservé aux développeurs qui ont déjà Google de verre , a quelques accessoires , comme un chargeur et un câble USB , pour 50 $ . Cette boutique , strictement réservée aux développeurs disposant déjà de Google Glass , propose quelques accessoires tels qu' un chargeur et son câble USB , pour 50 $ . -Le magasin appelait la police à l' aide , mais " au moment où il arrivait déjà qu' il avait déjà quitté une scène " , a-t-elle ajouté . Le magasin a appelé la police à la rescousse , mais " le temps qu' elle arrive , il était déjà parti sans " , a-t-elle ajouté . -Le spectacle du LG Arena à Birmingham aura lieu le 20 novembre et le spectacle aura lieu à Glasgow Hydro le 21 novembre . Le spectacle à la LG Arena de Birmingham aura lieu le 20 novembre et celui à l' Hydro de Glasgow se tiendra le 21 novembre . -Le spectacle est conçu pour être présenté dans tout théâtre majeur en France , de la même manière qu' à Paris . Le show est conçu pour entrer dans toutes les grandes salles de France sous la même forme qu' à Paris . -Les spectacles sont très attendus au moment où ils atteignent un provincial ( théâtre ) . Ces spectacles sont très attendus quand ils arrivent dans un Zénith de province . -La fermeture aux États-Unis a ralenti le marché automobile . Le aux USA a freiné la hausse du marché automobile -La femme du chanteur a également déclaré que , une semaine avant sa mort , elle avait promis à son mari qu' elle l' de l' hôpital et le à son domicile à ( Long Island ) . L' épouse du chanteur explique également qu' elle avait promis à son mari , la semaine avant son décès , de le faire sortir de l' hôpital et de l' emmener dans leur maison de Springs ( Long Island ) . -La situation n' a pas beaucoup changé depuis le décès du père en 2002 . La situation n' a pas beaucoup changé depuis le décès du père du Dr en 2002 . -La sixième édition de Play se tiendra au parc de patinage à samedi , le 02 novembre . La sixième édition du Play a lieu ce samedi 2 novembre au skate park à Hyères . -La petite ville de Pia a connu une hauteur inhabituelle de fièvre le jeudi soir . La petite commune de Pia a connu un inhabituel excès de fièvre jeudi soir . -L' image souriante de mon père reste avec moi pendant mes activités quotidiennes . C' est l' image de mon père souriant qui m' accompagne dans mes gestes au quotidien . -L' avocat , surnommé " " et reconnu par les criminels comme étant un " excellent point de départ " , a été arrêté à son domicile en 2011 à la suite d' une opération de surveillance policière à Scarborough , qu' il avait rencontrée dans un essai de drogue antérieur . L' avocat , qui était surnommé « » et considéré par les criminels comme un « dossier de première importance » , a été arrêté chez lui en 2011 à la suite d' opérations d' écoute de la police dirigées vers Scarborough , qu' il avait représenté lors d' un précédent procès en relation avec le trafic de stupéfiants . -L' avocat craignait qu' il ne soit exposé à la toxicomanie secrète . L' avocat avait peur que son addiction secrète à la drogue ne soit révélée au grand jour -Les riffs de guitare font que vous voulez quitter la terre . Les riffs de guitare , , donnent envie de quitter la Terre . -La scène est partiellement en gris , les danseurs et le piano sont couverts en tissu , comme les meubles dans une maison abandonnée . La scène est en partie de gris , les danseurs et un piano sont recouverts de tissus , comme des meubles dans une maison délaissée . -Les enjeux sont importants - l' obtention d' un accord sur les réformes et l' amélioration de la position de la Grande-Bretagne font que le référendum sur l' adhésion à l' UE dans le cadre d' un référendum marque une hausse de 16 % . Les enjeux sont élevés – parvenir à changer le traité et mieux représenter les intérêts de la Grande-Bretagne permettrait de voir un basculement de 16 % des votes en faveur du maintien du pays dans l' UE lors d' un référendum . -L' enjeu est important pour l' essor rapide de l' économie , comme la découverte de grandes réserves de gaz à effet de serre et les gisements de charbon dans le Nord-Ouest d' ici à plus de $ d' investissements dans les prochaines années par des entreprises telles que Rio Tinto , Vale du Brésil , d' Italie et des États-Unis . & nbsp ; & nbsp ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; amp # 160 ; & # 160 ; & # 160 ; & # 160 ; » & # 160 ; » L' enjeu est considérable pour l' économie en pleine croissance puisque la découverte d' énormes réserves de gaz offshore et de gisements de charbon dans le nord-ouest pourrait attirer plus de 50 Md $ d' investissements au cours des prochaines années provenant de sociétés comme Rio Tinto , Vale du Brésil , d' Italie et des États-Unis . -Le lancement de la construction du gazoduc South Stream en Bulgarie marque le lancement de l' un des plus grands projets énergétiques de l' Europe , a déclaré le chef de Gazprom . Le début de la construction du gazoduc South Stream en Bulgarie marque le lancement de l' un des plus grands projets énergétique d' Europe , a déclaré le directeur de Gazprom . -L' État devrait maintenir les niveaux de financement actuels pour recevoir les fonds du Commonwealth et faire rapport chaque mois sur l' affectation des fonds supplémentaires . L' État devra maintenir les niveaux de financement actuels afin de recevoir de l' argent du Commonwealth et préparer des rapports mensuels sur les secteurs où il dépense les fonds supplémentaires . -Le stock des espèces de Sousa , dont la population découverte vient de se reproduire , s' est progressivement affaiblie , ce qui la rend encore plus vulnérable . Pour ce qui est de l' espèce Sousa , dont la population de la nouvelle espèce découverte est issue , son stock est automatiquement devenu plus faible , la rendant ainsi plus vulnérable . -Le corps étudiant a été , à chaque fois , la cible d' une initiative qui a eu beaucoup de succès la saison dernière . À chaque fois , c' est la clientèle scolaire qui sera ciblée , une initiative qui avait connu beaucoup de succès au cours de la saison précédente . -Les étudiants volontaires paient souvent leur propre chemin vers le Cameroun , souvent avec des dons médicaux . Les étudiants bénévoles paient en général leur voyage jusqu' au Cameroun , et arrivent souvent avec du matériel médical collecté . -L' action visant à bloquer cette loi a été lancée par 26 États et la Fédération nationale des entreprises indépendantes . La procédure visant à bloquer la loi a été engagée par 26 États et la National Federation of Independent Business . -Le programme mettra davantage l' accent sur les " problèmes du monde réel " , y compris les mathématiques financières . Le programme mettra davantage l' accent sur les « problèmes du monde réel » , notamment les mathématiques financières . -La fête du thé est . Le Tea Party est . -La technologie est là pour le faire . La technologie est là pour le faire . -Les parents de l' adolescent ont dit espérer que la vidéo des indices de la mort . Les parents de l' adolescent ont déclaré qu' ils espéraient que les images vidéo contiennent des indices sur les circonstances de sa mort . -Le géant des télécommunications Rogers a déjà indiqué qu' il pourrait offrir si certains détails techniques ( qu' il ne cite pas ) pourraient être corrigés . Le géant des télécommunications Rogers a déjà indiqué qu' il pourrait offrir si certains détails techniques , qui n' ont pas été cités , pouvaient être réglés . -Le territoire mesure , dont trois autres petites îles proches . Ce territoire mesure 47 km2 , si l' on compte trois autres petites îles proches . -L' essai , si tel était le cas , serait appliqué à l' aide de critères permettant d' éviter que les servent à occuper des postes non qualifiés ou comme moyen de locomotion pour déplacer la famille et les amis vers l' Australie . Le test , s' il est adopté , serait appliqué selon un critère destiné à empêcher les visas 457 d' être utilisés pour occuper des postes non qualifiés ou comme un moyen détourné pour faire venir sa famille et ses amis en Australie . -Le directeur du théâtre a immédiatement commencé une procédure d' évacuation et a appelé la brigade des pompiers à découvrir une odeur suspecte . Immédiatement , le directeur de l' établissement a fait procéder à l' évacuation de la salle et a prévenu les pompiers pour une odeur suspecte . -Les trois principaux fabricants de Detroit ont réalisé une croissance de plus de 10 points par rapport à octobre 2012 , mais seulement General Motors a fait des progrès meilleurs que prévu . Les trois grands constructeurs de Detroit affichent une croissance à deux chiffres par rapport à octobre 2012 mais seul General Motors a fait mieux qu' attendu . -Le moment est venu de rétablir la paix " . Le temps est venu de ramener la paix . -Le nombre total de morts en octobre est le plus élevé depuis avril 2008 , année où personnes ont été tuées . Le nombre total de morts en octobre est le plus élevé depuis avril 2008 , quand 1 073 personnes avaient été tuées . -La ville , qui est un bastion politique et le dernier bastion de la rébellion , se trouve à la frontière avec l' Ouganda , à environ 80 km au nord de Goma . politique et dernière place forte de la rébellion , cette localité est située à la frontière ougandaise , à environ 80 km au nord de Goma . -La ville a décidé d' utiliser des DEL pour réduire les coûts . Pour alléger ses coûts , la ville a recours à des . -Les habitants ont été heureux d' apprendre que le lac à la pêche et le lac des Cèdres continueront de fournir de l' eau potable à la ville . Les citoyens ont été heureux d' apprendre que les lacs à la Pêche et des Piles continueront d' approvisionner la ville en eau potable . -La tragédie s' est produite à environ 11 heures sur la rue , près de la jonction avec la rue , dans le quartier de . Le drame est survenu vers 11 h , sur la rue , tout près de l' intersection de la rue , dans l' arrondissement de . -Le transfert de lits de médecine intérieure est prévu pour un certain temps au cours du mois . Quant au transfert des lits de médecine interne , il est programmé pour le courant du mois . -Les marges de division du transport ont été affectées par les problèmes d' exécution dans quelques grands contrats . Les marges de la division Transport ont souffert des problèmes d' exécution concernant quelques gros contrats . -Le grand-père traumatisé était à l' étage au moment de l' explosion . , le grand-père était à l' étage au moment de l' explosion . -La compagnie de voyages s' attend maintenant à ce que son investissement à long terme avant impôt s' élève à 325 millions de dollars et 340 millions de dollars , comparativement à un montant de 305 millions de dollars à 315 millions de dollars qu' il avait prévu à l' origine . L' agence de voyage s' attend désormais à réaliser un bénéfice sous-jacent sur l' ensemble de l' exercice situé entre 325 et 340 M $ avant impôts , contre une précédente prévision de 305 à 315 M $ . -La tendance à la puberté précoce n' est pas aussi prononcée que chez les garçons La tendance vers une puberté précoce n' est pas aussi prononcée chez les garçons -Le tribunal a également le pouvoir de saisir les biens d' un délinquant si une amende n' est pas payée . Le tribunal a également le pouvoir de saisir les biens d' un contrevenant si celui-ci n' a pas payé son amende . -Le tunnel , qui la longueur de presque six terrains de football , relie les entrepôts près de Tijuana , Mexico et San Diego , USA . Le tunnel , sur une longueur équivalente à près de six terrains de football de long , relie des entrepôts près de Tijuana , au Mexique , à San Diego , aux États-Unis . -Le tunnel est le huitième grand passage découvert à San Diego depuis 2006 . Le tunnel est le huitième passage important découvert à San Diego depuis 2006 . -Le tunnel a été fermé avant que les médicaments ne soient détectés par les autorités . Le tunnel a été fermé avant que les drogues n' en soient ressorties en passant inaperçues , ont déclaré les autorités . -Les deux ont été trouvés tard le vendredi après-midi . En fin d' après-midi ce vendredi , les deux ont été localisés . -Les deux hommes , un autre , l' inconnu , n' ont pas le métro le jeudi à , dans le but de la des , dans le district de . , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , . Les deux hommes , l' un à la pratique et l' autre non , étaient partis sous terre jeudi vers 21h30 afin d' effectuer une traversée de la de , situé sur la commune de . -Les deux hommes , qui sont des soldats du 13e bataillon des troupes françaises stationnés à Chambéry , ont été trouvés " épuisés mais " . Les deux militaires au 13e bataillon des alpins de Chambéry , ont été retrouvés " épuisés , mais non blessés " . -Les deux commerçants seront les premiers travailleurs du à être suspendus dans l' extension qui se reflète dans le scandale de manipulation des crédits interbancaires . Les deux traders sont les premiers employés de à être suspendus dans le cadre de l' élargissement de l' enquête qui fait écho au scandale des manipulations du taux interbancaire . -Les deux ouvrages publiés par Nora il y a près de deux ans ont déjà prouvé que même s' il ne semblait pas écrire , il a écrit : et bien . Les deux ouvrages publiés par Nora voici bientôt deux ans ont déjà établi que celui qui passait pour ne pas écrire avait beaucoup et bien écrit . -Le but ultime est toujours d' offrir des outils dont nos clients n' ont pas encore accès . Le but ultime demeure d' offrir des outils auxquels nos clients n' ont pas accès actuellement . -L' Union des journalistes allemands a exhorté ses membres à cesser d' utiliser les services de Yahoo et de Google en ligne jeudi , après de nouvelles révélations sur les activités des services de renseignement américains et britanniques . Le syndicat des journalistes allemands a invité jeudi ses adhérents à cesser d' utiliser les services en ligne de Google et de Yahoo au lendemain de nouvelles révélations sur les activités des services de renseignements américains et britanniques . -Les utilisations de la biométrie se divisent en deux groupes : l' identification ( reconnaissance d' une personne parmi d' autres ) et l' authentification ( confirmation d' une personne qu' elle est censée dire ) . Les utilisations de la biométrie se répartissent en deux groupes : l' identification ( reconnaître une personne parmi d' autres ) et l' authentification ( s' assurer que la personne est celle qu' elle prétend être ) . -La scène généralement de la planification de la route a soudainement suscité un débat intense et des alliances colorées . Le secteur généralement sans intérêt de la planification des grands axes a soudain provoqué un débat fort animé et des alliances . -Les valeurs qu' elle à moi sont toujours là , et je me donc de me demander ce que je veux lui faire à un moment donné . Les valeurs qu' elle m' a sont toujours présentes , si bien qu' il m' arrive de penser à comment elle voudrait que j' agisse à un moment donné . -La grande majorité des avions de ligne devraient être admissibles à un plus grand usage de dispositifs électroniques au titre des nouvelles lignes directrices , a déclaré . La vaste majorité des avions de ligne devraient permettre une utilisation plus large des appareils électroniques en vertu des nouvelles directives , a déclaré . -Le véhicule , un , s' est concentré sur la foule dans la fameuse place de la capitale chinoise , symbole de la répression sanglante de 1989 , avant de brûler le feu , tuant ses trois occupants et deux passants . Le véhicule , un SUV , a foncé sur la foule qui se trouvait sur la célèbre place de la capitale chinoise , symbole de la répression meurtrière menée en 1989 , et a pris feu tuant ses trois passagers et deux passants . -Le véhicule portait également des conteneurs remplis d' essence et un drapeau avec des écrits religieux orthodoxes . Cette dernière contenait également des récipients remplis d' essence et un drapeau avec des inscriptions religieuses orthodoxes . -Le verdict vient à l' horizon du dixième anniversaire de l' attaque qui se déroulera cette année , qui sera marquée par des cérémonies à Bali et en Australie . Le verdict a été rendu à la veille du 10e anniversaire des attentats qui sera célébré plus tard cette année et marqué par des cérémonies à Bali et en Australie . -Le vétérinaire a dit que si elle était devenue trop lâche et , elle aurait probablement pu tomber . Le vétérinaire a dit que si elle avait été trop épuisée et qu' elle s' était , elle serait probablement tombée . -Les téléspectateurs ont partagé leurs pensées sur Twitter . Sur Twitter , les téléspectateurs ont partagé leurs commentaires . -La violence est au centre des discussions des États-Unis avec le Premier ministre Nouri Al , qui sera reçu par le Président Barack Obama le 2 novembre , après leur dernière réunion , le 12 décembre 2011 . Les violences sont au centre des entretiens aux États-Unis du premier ministre Nouri Al qui sera reçu vendredi par le président Barack Obama , deux ans après leur précédente rencontre , le 12 décembre 2011 . -La violence est de plus en plus meurtrière à la suite des mesures de sécurité renforcées et des opérations militaires de grande envergure menées ces derniers mois par le gouvernement d' Nouri , dominé par les chiites . Ces violences sont de plus en plus meurtrières en dépit de mesures de sécurité renforcées et d' opérations militaires d' envergure lancées depuis des mois par le gouvernement de Nouri Al , dominé par les chiites . -Le virus infecte de façon sélective les cellules du foie , lesquelles commencent alors à produire un flux régulier des anticorps . Le virus infecte de manière sélective les cellules du foie , qui commence alors à produire un flux régulier d' anticorps . -Le vote n' est pas contraignant , mais il l' image de , le troisième homme le plus riche du monde , qui est encore dans la victoire de son bateau dans la Coupe des Amériques . Le vote n' est pas contraignant mais vient ternir l' image de Larry Ellison , troisième fortune mondiale tout juste de la victoire de son bateau à la Coupe de l' America . -La taille du américain moyen est de pouces , selon les statistiques américaines sur la santé . Le tour de taille de l' Américain moyen du XXIe siècle est de 39,7 pouces , selon les statistiques sanitaires américaines . -Le rail d' eau était convenable et bien au moment où il a été libéré . Le d' eau était en bonne santé lorsqu' il a été relâché . -Le chemin de fer d' eau a été mis sur un hélicoptère à Aberdeen le mois dernier avant d' être repris à la santé par la écossaise dans son centre de sauvetage à . On a fait monter le d' eau à bord d' un hélicoptère en direction d' Aberdeen le mois dernier avant qu' il soit remis sur pieds par la écossaise dans son centre de sauvetage d' . -La façon dont il a commencé à faire son apparition sera au Cheval de l' année avant le long terme - et je suis sûr qu' il fera bien . De la façon dont il a commencé , il participera sûrement bientôt à la manifestation du Cheval de l' année – et je suis sûre qu' il fera une belle prestation . -Le gagnant a été invité au Grand Prix à Abu Dhabi avec un ami et a pu se rapprocher du champion allemand , sur la piste et dans les . Il a été invité suivre le GP d' Abou Dhabi , avec un ami , et peut donc voir de près le champion allemand , sur la piste et dans son stand . -Les travaux de Ltd suggèrent des services à grande vitesse pour l' Écosse et le nord de l' Angleterre dès le lancement de la phase 1 en 2026 . Le travail réalisé par Ltd laisse penser que les services à grande vitesse jusqu' à l' Écosse et le nord de l' Angleterre dès l' ouverture de la Phase 1 en 2026 . -Le quatrième du monde indique qu' il a livré 45 avions au cours du trimestre , contre 57 l' année précédente . Le quatrième constructeur aéronautique au monde a indiqué qu' il avait livré 45 avions au cours du trimestre , contre 57 un an plus tôt . -Le jeune , du village de , près de Vale , au sud de la campagne , suit les traces de sa tante Sharon , qui a fait des courses depuis plus de 10 ans . Ce jeune garçon , du village de , près d' Vale , dans le sud du pays de Galles , suit les traces de sa tante Sharon , qui participe à des concours équestres depuis plus de 10 ans . -Leurs corps ont été découverts plus tard . Leurs corps ont été découverts plus tard . -Leurs sont , ou . Les amours sont , ou joueurs . -Leur chance ne s' arrête pas toujours . La chance ne les suit pas toujours . -Leurs tâches incluent la modification d' une pompe sur les défectueux . Il s' agit pour eux , entre autres , de changer une pompe sur la chaufferie défaillante . -Leur © de la mГ © © sente la vraie © . Leurs en musique , , donnent une vraie fragilité . -Puis , une sorte de réajustement très doux à la vie a commencé , à un ou deux mois qui était un peu comme des vacances . Commence ensuite une sorte de réadaptation à la vie , très doucement , pendant un ou deux mois qui sont un peu comme des vacances . -Ensuite , seule la colline de sera laissée . Il ne restera plus que la colline de . -Puis , les signes ont commencé à apparaître . Puis des signes sont apparus . -Ensuite , il y a les comédies musicales avec les somptueuses productions du Châtelet , qui nous permettent d' abandonner l' âge d' or de Broadway à la vie et de redécouvrir l' univers de Stephen , le plus grand compositeur vivant . Après , on a les comédies musicales avec les ­ productions somptueuses du Châtelet qui nous font revivre l' âge d' or de Broadway et découvrir l' univers de Stephen , le plus grand ­ compositeur vivant . -Ensuite , il n' y aura plus de salariés à l' usine . Dès lors , il n' y aura plus d' employés dans l' usine . -Il y a aussi des fonds pour des cliniques privées à et , des services de soins de santé améliorés , une formation spécialisée , des services de santé mentale et la mise en place de systèmes de santé électroniques personnels dans les hôpitaux locaux . Des fonds sont également prévus pour les cliniques sans rendez-vous à et , de meilleurs soins externes , la formation des cadres médicaux , des services voués à la santé mentale et le déploiement de systèmes de dossiers médicaux électroniques dans les hôpitaux locaux . -Il y a une acceptation sociale accrue de la en tant que moyen de garantir un financement garanti à court terme . La mise en gage est de mieux en mieux acceptée socialement comme moyen d' obtenir un financement garanti à court terme . -Rien n' indique que des hormones ou d' autres produits chimiques soient à blâmer Rien ne prouve que les hormones ou que d' autres substances chimiques sont responsables -Là , les pères fondateurs sont toujours honorés , mais ici , nos ancêtres révolutionnaires ne sont guère utilisés comme modèles , mais ils se sont les uns les autres . Là des pères fondateurs toujours honorés , et ici des ancêtres révolutionnaires dont on peine à faire des modèles , et qui se sont du reste . -Là , il y a cohérence institutionnelle ; il y a eu un torrent de constitutions , autant d' erreurs à corriger et des tests à répéter : Là une constance institutionnelle , et ici une cascade de constitutions , comme autant d' erreurs à rattraper et d' essais à reprendre . -Il y a beaucoup moins de façons coûteuses et moins intrusives de le faire . Il existe des moyens bien moins onéreux et des méthodes bien moins intrusives de procéder . -Il y a beaucoup de patients . Il y a beaucoup de patients . -Il y a aussi des anciens prêtres catholiques . Il y a aussi d' anciens prêtres catholiques . -Il y a actuellement 403 commandes et engagements de 15 clients et opérateurs . À l' heure actuelle , la société a enregistré un total de 403 commandes et engagements d' achat auprès de 15 clients et opérateurs . -Il existe de nombreuses théories sur les causes de la puberté précoce . Il existe de nombreuses théories portées à la connaissance du public sur les causes de la puberté précoce . -Il y a aujourd' hui 201 cardinaux . Il y a à l' heure actuelle 201 cardinaux . -Il y a de nombreux emplois dans les théâtres régionaux , les compagnies de tournée , les théâtres de Broadway , les théâtres de Broadway , etc. Les emplois sont nombreux entre les théâtres régionaux , tournées , Broadway , le ... -Il y a beaucoup de belles - la création d' images a toujours été un des points forts de Chouinard - comme les cheveux qui sont ou le tissu noir qui les lignes . Il y a quelques beaux - la création d' images a toujours été une force chez Chouinard - comme ces cheveux qui s' ou ces tissus noirs qui allongent les lignes . -Il y a trois façons de rendre la biométrie attrayante pour le grand public . Il existe trois manières de rendre la biométrie séduisante aux yeux du grand public . -Il existe des traitements pour retarder les règles précoces et pour interrompre une autre cause : le vieillissement prématuré des os qui peut en fin de compte entraîner un retard de croissance et être comme un adulte . Il existe des traitements pour retarder l' apparition des premières règles et prévenir d' autres conséquences : un vieillissement prématuré des os qui peut freiner la croissance et une petite taille à l' âge adulte . -Il y a des explications différentes . Il existe diverses explications . -Il n' y a pas beaucoup de médecins dans le pays d' Afrique de l' Ouest ; un seul pour 5 000 personnes . Il n' y a pas beaucoup de médecins dans ce pays d' Afrique occidentale , à peine 1 pour 5 000 habitants . -Il y a eu aussi un changement dans la gestion du réseau d' éclairage public et des décorations de Noël . Mais aussi une gestion différenciée du réseau de l' éclairage public et des décorations de Noël . -Il y a toujours eu une opposition interne au projet de création d' une Europe fédérale . Il y a toujours eu une opposition nationale au projet de création d' une Europe fédérale . -Des manifestations ont eu lieu dans toute l' Égypte pour exiger le rétablissement du premier président démocratiquement élu du pays . Une mobilisation dans toute l' Egypte pour réclamer le rétablissement du premier président démocratiquement élu du pays . -Il y a eu de grandes erreurs dans certains de ces programmes pilotes d' État . De grosses erreurs ont été commises dans les programmes pilotes de certains États . -Il y a une pause quand Carol Zola se couche sous le tapis de danse , le microphone , pour ne pas offrir , sous la forme d' une ou d' une réserve , d' un poème , dans la voix rythmique d' une vente aux enchères , continue de danser . Il y aura cassure quand Carol Prieur se sous le tapis de danse , pour livrer , micro en main , sans sobriété ni retenue , une part du poème de , et monotone dans la criée , et poursuivra en dansant . -Il y a une pénurie de personnel de près de 30 % , ce qui fait de l' exercice " très difficile " , selon le chef du département , Gillian . Il manquait près de 30 % des effectifs nécessaires , rendant cette année financière " très difficile " , selon la chef du département , Gillian . -Il y a aussi une " violence symbolique " qui se réfère à " l' effacement de la singularité de la personne et des symboles distinctifs " , qui est réduite à " l' anonymat " des cendres . Il y a aussi " une violence symbolique " , qui tient à l ' " effacement de la singularité et des signes distinctifs " réduits à l ' " anonymat " des cendres . -Il y a débat sur ce qui constitue le début de la puberté , mais il est considéré comme " précoce " lorsque l' élargissement du sein est accompagné d' une poussée de croissance avant l' âge de 8 ans . Il y a un débat concernant ce qui constitue l' apparition réelle de la puberté , mais elle est considérée comme « précoce » lorsque le développement de la poitrine s' accompagne d' une croissance soudaine avant l' âge de 8 ans . -Il y aura un changement dans la façon dont nous allons payer ces taxes . Il va y avoir du changement dans la façon dont nous payons ces taxes . -Il n' y a pas d' aéroport à Pitcairn , ce qui rend impossible de trouver un ou plusieurs appareils pour surveiller la zone , et il n' y a pas de port où l' on puisse installer des patrouilles de défense . Il n' y a pas d' aéroport à Pitcairn , ce qui rend impossible d' y baser un ou plusieurs avions dédiés à la surveillance de la zone , et il n' y a pas de port où placer des navires militaires de surveillance . -Il n' y a pas de formule , pas de méthode pour traverser le mur de l' invisible pour être avec vos êtres chers . Il n' existe pas de recette , de mode d' emploi pour passer le mur de l' invisible et retrouver les siens . -Il n' est pas nécessaire de mettre en place une énorme infrastructure technologique coûteuse , qui sera inévitablement agrandie pour tenir les gens au courant des allées et venues du pays . Il n' est pas nécessaire de construire une infrastructure technologique énorme et qui sera inévitablement élargie pour conserver les enregistrements des allées et venues quotidiennes des gens . -Il y a un extrémisme radical dans le monde qui est et déterminé à essayer de tuer les gens et de frapper des individus et d' attaquer les gouvernements . Il existe un extrémisme radical dans le monde qui est déterminé à tenter de tuer des gens , les faire exploser et attaquer les gouvernements . -Il y a eu un tir à l' aéroport international de Los Angeles . Une fusillade a eu lieu à l' aéroport international de Los Angeles . -Il y a eu un mécontentement général vis-à-vis du marché unique , les entreprises ont déclaré que les coûts du règlement de Bruxelles aujourd' hui les avantages de l' appartenance à la zone commerciale de l' Europe - même si 40 % des grandes entreprises , traditionnellement les plus des entreprises , étaient d' accord . Le marché unique a occasionné un mécontentement général , les entreprises disant que le coût du règlement Bruxelles est disproportionné par rapport aux avantages qu' il y a à faire partie de la zone d' échanges européenne – 40 % des grandes entreprises , traditionnellement les plus , sont même d' accord . -Il n' y a pas eu de nouvelles après cela , a déclaré la même source . On était alors sans nouvelle d' eux depuis , a précisé la même source . -Il y avait des soldats syriens là-bas . Il y a des militaires syriens . -Il n' y a pas eu de victimes , mais près de 20 appartements sur les premier et deuxième étages de ce quartier des magasins ont dû être évacués . Il n' y a pas eu de blessés , mais près de vingt logements , situés au premier et deuxième étages de cette rangée de locaux commerciaux , ont dû être évacués . -Ces s' appuieront sur nos progrès dans le cadre des objectifs du Millénaire pour le développement , et ils feront partie intégrante du cadre de développement . Ces s' appuieront sur les progrès réalisés dans le cadre des objectifs du millénaire pour le développement et feront partie intégrante du cadre de développement . -Ce sont les mêmes jeunes qui aiment la télévision de réalité et les étoiles éphémères qu' il produit ... Ce sont ces mêmes jeunes qui aiment la et ce qu' elle produit comme stars éphémères ... -Ces installations ne sont pas rentables et devraient donc être vendues à des entreprises privées ou démolies . Si ces installations ne sont pas rentables , qu' on les vende à l' entreprise privée ou qu' on les . -Ces caractéristiques peuvent se manifester dans des caractéristiques sexuelles secondaires , telles que la masse musculaire , la distribution de cheveux , les seins et la taille , les caractéristiques sexuelles primaires telles que les organes génitaux et les organes génitaux , ou les structures et les hormones hormonales . Ces caractéristiques peuvent se manifester d' elles-mêmes dans les caractères sexuels secondaires , ( comme la masse musculaire , la , la poitrine et la stature ) , les caractères sexuels primaires ( comme les organes reproducteurs et les organes sexuels ) ou la structure des chromosomes et les hormones . -Ces interceptions auraient apparemment lieu à l' extérieur des États-Unis . Ces interceptions auraient lieu en dehors des Etats-Unis . -Ces réserves , que les autorités ont largement utilisées pour soutenir la devise locale , le , sont en train de s' effondrer , ce qui conduit à une augmentation probable de la charge de la part de la Commission , ce qui la dette extérieure du pays . & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; Ces réserves , auxquelles les autorités ont recours massivement pour soutenir la monnaie locale , la , s' , l' agence juge de plus en plus probable une dévaluation , ce qui sa dette extérieure . -Ils ne sont pas tous des coureurs expérimentés , mais des gens qui recherchent l' excitation et l' aventure , et un chemin possible vers des événements de classe mondiale . Ils ne sont pas tous expérimentés , certains recherchent l ’ excitation et l ’ aventure et un accès réaliste à des événements internationaux . -Ils ne sont pas censés les reprendre jusqu' à ce que les plans atteignent les 10,000 mètres et que le capitaine donne le feu vert . Ils ne sont pas censés les rallumer avant que l' avion n' atteigne une altitude de 10 000 pieds et que le commandant donne son feu vert . -Ils ont dû pousser les véhicules à travers des rivières et des boues plus d' une fois . Ils ont maintes fois dû pousser les véhicules pour traverser des rivières ou des pistes . -Par la suite , 32 et plus d' autres hommes ont été incarcérés après que la police eut saisi l' héroïne et la cocaïne d' une valeur de livres et d' une valeur de plus de 200 000 livres sterling au cours d' une série de raids . Avec 32 autres hommes , ils ont été par la suite emprisonnés après que la police a saisi de l' héroïne et de la cocaïne pour un montant de 1,5 M £ , ainsi que plus de 200 000 £ en espèces au cours d' une série de raids . -Il s' agit d' une équipe formidable . Ils forment une bonne équipe . -Ils viennent de 60 kilomètres autour du village , et ils viennent à pied . Ils habitent parfois à 60 km du village , et ils viennent à pied . -Ils explorent comment , au cours de la prochaine décennie , ils peuvent se déplacer vers un système où les conducteurs paient le quart des routes qu' ils empruntent . Ils cherchent comment , au cours de la prochaine décennie , ils pourront passer à un système permettant aux conducteurs de payer en fonction du nombre de miles parcourus . -Ils sont , une partie d' une soixantaine de sujets qui sont le résultat d' un diagnostic de troubles du développement sexuel , d' un terme générique pour tous les types de , de ( ovaires ou testicules ) , ou d' organes génitaux mal développés . Ils sont , l' faisant partie du groupe de la soixantaine de maladies diagnostiquées comme désordres du développement sexuel , un terme générique désignant les personnes possédant des chromosomes ou des ( ovaires ou testicules ) atypiques ou des organes sexuels anormalement développés . -Ils affirment que les bagages coûtent de l' argent et que les passagers qui veulent le service devraient payer . Elles estiment que les coûts de manutention des bagages sont importants et que les passagers qui veulent ce service devraient le payer . -Ils m' ont posé de nombreuses questions sur le . Ils m' ont posé des questions sur les preneurs d' otages . -Ils peuvent choisir un appareil avec ou sans GPS . Ils peuvent choisir un appareil avec ou sans GPS . -Ils auraient également pu demander un mandat de perquisition immédiat , si cela était jugé nécessaire . Elle aurait également pu requérir un mandat de perquisition instantané , si jugé nécessaire . -Ils ne veulent pas que nous leur ce qui les rend rentables . Elles ne veulent pas qu' on leur dise ce qui leur permettra d' être rentables . -Ils nous ont sur un trottoir . Ils nous déposent sur le bord d' un trottoir . -Ils la joie et la folie ; c' est l' art de Chouinard qui contamine le public . La joie et la folie , et c' est l' art de Chouinard , là , de contaminer le public . -Ils sont condamnés à des peines de prison allant de 10 ans d' emprisonnement à perpétuité s' ils sont reconnus coupables . Ils encourent une peine de prison allant de 10 ans à la réclusion à perpétuité s' ils sont condamnés . -Ils ont été perdus dans le réseau , mais ils ont repris les marches en attendant d' obtenir de l' aide , a déclaré la police . Ils se sont perdus dans le réseau , sont retournés sur leurs pas en attendant les secours , a précisé la préfecture . -Ils avaient fait des recherches pour leur chien disparu et avaient envoyé des appels sur des sites de réseautage social après qu' ils eurent pénétré dans la carrière à la suite de l' accident mineur . Ils avaient cherché désespérément leur chienne perdue et lancé un appel sur les sites de réseaux sociaux après qu' elle a disparu dans la carrière suite à un accident sans gravité . -Ils ont également demandé à Barack Obama de faire comprendre à Al Thani que " l' influence pernicieuse de l' Iran au sein du gouvernement irakien constitue un sérieux problème dans la relation entre nos deux pays " . Ils ont aussi exigé de Barack Obama qu' il fasse comprendre à Nouri Al que " l' influence pernicieuse de l' Iran au sein du gouvernement irakien constitue un problème sérieux dans notre relation bilatérale " . -Ils ont eu le plaisir de le reconnaître comme un grand , une force motrice , un de leur travail , mais ils ont remarqué qu' il semblait peu disposé à produire ce travail lui-même . Ils ont volontiers reconnu en lui un grand , animateur et fédérateur de leurs travaux , mais en faisant observer qu' il se montrait peu enclin à en produire lui-même . -Ils ont signalé en particulier deux cas signalés dans La Presse Canadienne , un cas impliquant un soldat mis en liberté vendredi dernier . Ils ont notamment cité deux cas rapportés par La Presse Canadienne , dont celui d' un soldat libéré vendredi dernier . -Ils ont les meilleurs noms , comme " derrière les murs " ( qui se trouve dans une prison réelle ) , " " et " The House of " . Elles portent les meilleurs noms , comme « Terror Behind the » ( qui est en fait une vraie prison ) , « » et « The House of » . -Ils connaissent mieux leurs affaires que n' importe qui d' autre . Elles connaissent leur entreprise mieux que personne . -Ils vivent dans des habitats homogènes et ouverts , où il y a de nombreux échanges génétiques entre populations et entre individus . Ils vivent dans des habitats homogènes , ouverts , dans lesquels il y a de nombreux échanges génétiques entre les populations et les individus . -Ils recherchent le bon travail d' équipe entre le et le - et sont les meilleurs dans le ring . Ils recherchent un travail d' équipe efficace entre le et le – et ont été les meilleurs sur la piste . -Ils ont mesuré le taux cardiaque chez les animaux et ont analysé leur comportement . Ils ont mesuré la fréquence cardiaque des animaux et analysé leur comportement . -Ils n' ont jamais cru qu' il était mort de la façon dont le shérif avait conclu . Ils n' ont jamais pensé qu' il était mort comme cela est indiqué dans les conclusions du shérif . -Ils ont précisé qu' ils avaient également reçu les autorisations nécessaires du Canada , de l' Inde et de la Turquie , outre celles de l' Afrique du Sud et de la Corée du Sud . Ils précisent qu' ils ont aussi reçu les autorisations nécessaires au Canada , en Inde et en Turquie , après l' Afrique du Sud et la Corée du Sud . -Ils estiment qu' il s' agit d' une dissimulation visant à protéger la personne ou les personnes responsables du décès de leur fils . Ils sont persuadés que ces conclusions ont été livrées dans le seul but de couvrir la ou les personnes responsables de la mort de leur fils . -Ils veulent voir le gouvernement décider de nouveaux liens commerciaux avec les Chinois , l' Inde et le Brésil , plutôt que de s' enliser dans le long et difficile processus de réforme des institutions de l' Union . Ils veulent voir le gouvernement donner la priorité à de nouveaux liens commerciaux avec des pays comme la Chine , l' Inde et le Brésil plutôt que de s' enliser dans le processus long et ardu de réforme des arcanes institutionnels européens . -Ils étaient armés de et de lance-roquettes . Ils étaient armés de et de lance-roquettes . -Ils des pancartes qui l' accusent et d' autres d' être des meurtriers et appelaient les États-Unis à refuser de lui venir en aide . Ils des pancartes l' accusant , entre autres , d' être un meurtrier et appelant les Etats-Unis à refuser de lui fournir de l' aide . -Ils ont été examinés , et un contrôle approfondi du cinéma n' a révélé rien . Les personnes ont été examinées et une reconnaissance complète de la salle n' a rien donné . -Ils ont été retrouvés le vendredi après-midi . Ils ont été localisés ce . -Ils ont été inclus dans la version finale du document , qui sera approuvée par les dirigeants mondiaux , dont Mme au cours du sommet . Ils figuraient dans l' avant-projet final du document , qui sera adopté par les dirigeants mondiaux , y compris Mme , au cours du sommet . -Ils n' ont pu se rendre sur les deux sites en raison de préoccupations en matière de sécurité , a déclaré les inspecteurs . Ils n' ont pas pu visiter les deux derniers sites en raison de préoccupations concernant leur sécurité , ont indiqué les inspecteurs . -Ils étaient contre tous les , mais les deux se sont avec la tasse d' or et la . Ils se sont présentés envers et contre tous , mais les deux ont la coupe en or et la . - , responsable de la mission scientifique à l' Observatoire de la Conservation et de l' Étude des et des ( ) , a discuté de cette découverte avec L' Express et décrit le long processus d' identification des nouvelles espèces . , chargé de mission scientifique à l' Observatoire pour la Conservation et l' Etude des Animaux et ( ) revient pour L' Express sur cette découverte et explique le long processus d' identification de nouvelles espèces . -Trente cinq autres hommes impliqués dans cette course ont été emprisonnés pour un total de 153 ans d' emprisonnement pour infraction à la législation sur les stupéfiants . Trente-cinq autres personnes impliquées dans le trafic ont été condamnées à un total de 153 ans de prison pour trafic de stupéfiants . -Ceci , parallèlement à la dimension artistique , a été un événement bénéfique qui aide l' organisation à but non lucratif à obtenir un bilan financier qui sonne comme apaisant pour l' oreille sous la direction de Die . Il s' agissait d' une , parallèlement à la dimension artistique , et elle a aidé l' organisme sans but lucratif à produire un bilan financier qui est aussi doux aux oreilles qu' un air de La chauve-souris . -Cet ensemble Henri avec du noir et du blanc est constitué d' une série de solos , dans un groupe en tandem ( une messe en solo ) , un grand groupe de douze solos , où chaque individu donne la représentation de son propre signe . Ce jeu noir et blanc de Henri se fait en relais de solos , en groupe à l' unisson ( une masse qui fait solo ) , en grand groupe de douze solos où chacun exécute son signe . -Ce jeudi , la police a arrêté l' ami de Rob Ford et son chauffeur , Alexander , âgé de 35 ans , et l' a accusé d' extorsion en rapport avec la vidéo . Ce jeudi , la police a arrêté l' ami et chauffeur occasionnel de Rob Ford , Alexander , âgé de 35 ans , et l' a accusé d' extorsion relativement à la vidéo . -Ce jeudi , il a conduit un certain nombre de journalistes de sa propriété lorsqu' ils sont venus pour l' interviewer . Ce jeudi , il a chassé de sa propriété plusieurs journalistes venus l' interroger . -Ce mélange de gènes limite la création de nouvelles espèces . Ce brassage de gènes limite la création de nouvelles espèces . -Ce livre devrait définitivement le traduire en justice . Le livre que voici devrait lui rendre définitivement justice . -Cela complique le remboursement du crédit à l' étranger . Cela complique le remboursement des crédits pris à l' étranger . -Cela concerne les " opérations militaires " dans les pays où les alliés de l' OTAN coopèrent avec les États-Unis et qui , en fin de compte , n' ont pas visé l' Europe , selon le Général Alexander . Cela concernerait des " opérations militaires " dans des pays où ces alliés de l' Otan coopèrent avec les Etats-Unis et cela ne viserait absolument pas l' Europe , a affirmé le général Alexander . -Ce jour d' Halloween , Pia a eu un jour trouble marqué par l' apparition d' une émeute impliquant un certain nombre de personnes autour du bureau de poste . En ce jour d' Halloween , la soirée a été agitée du côté de Pia où une impliquant plusieurs personnes a éclaté du côté de la Poste . -Cette décision fait partie des tentatives de la Maison Blanche de reprendre le contrôle de l' affaire NSA à la suite des révélations de l' ancien analyste , Edward , qui s' est réfugié en Russie . Cette décision fait partie de la tentative de la Maison blanche de reprendre la main dans l' affaire des écoutes de la NSA après les révélations faites par l' ancien analyste Edward , réfugié en Russie . -Ce détour dans le chemin de ma vie est toujours la chose la plus inattendue et la plus belle . Ce détour du chemin de ma vie reste le plus inattendu et le plus beau . -Cette constatation , qui a été reflétée dans les tailles et les grands groupes d' entreprises , montre que les entreprises font pression pour un " changement significatif " qui confère des pouvoirs au Royaume-Uni . Cette conclusion , qui se retrouve dans tous les grands groupes et les entreprises de toutes tailles , montre que le monde de l' entreprise insiste pour un « changement significatif » qui redonne le pouvoir au Royaume-Uni . -Cela inclut l' Australie et le Royaume-Uni , qui sont généralement nos plus gros générateurs de profit . Cela concerne l' Australie et le Royaume-Uni , qui sont généralement les pays qui génèrent les plus gros profits . -Il s' agissait de trois véhicules de police , de deux ambulances et d' un groupe d' une trentaine de personnes , selon les rapports . Trois véhicules de gendarmerie , deux ambulances et un d' une trentaine de personnes selon les témoignages recueillis ... -Il s' agit d' une baisse par rapport au bénéfice net de millions de CAD , ou de 83 ¢ par action , enregistré au cours de la même période l' an dernier . Il s' agit d' un recul comparativement au bénéfice net de millions , ou 83 ¢ par action , de la même période l' an dernier . -C' est une soirée qui met en vedette tout le spectre de Chouinard , de l' art à l' et de la noirceur de l' encre à la blancheur . Une soirée qui permet de voir tout le prisme de Chouinard , de l' art aux et du noir d' encre au blanc . -C' est parce que l' Amérique et la France ont toutes deux une révolution , ont établi une déclaration de droits et ont essayé de trouver une nouvelle société . Car l' Amérique et la France ont l' une et l' autre fait une révolution , rédigé une déclaration des droits , voulu fonder une société neuve . -C' est le cas au Brésil et en Allemagne , deux pays où les relations diplomatiques avec les États-Unis sont tendues . C' est le cas du Brésil et de l' Allemagne , deux pays où les relations diplomatiques avec les Etats-Unis se sont tendues . -C' est le cas des nouvelles espèces qui viennent d' être identifiées . C' est le cas de la nouvelle espèce qui vient d' être identifiée . -Il s' agit de la première série de directives que le gouvernement américain a mises en place , car le nombre d' enfants d' âge scolaire atteints d' allergies alimentaires s' accroît . Il s' agit de la première série de lignes directrices que le gouvernement américain publie , puisque le nombre d' enfants en âge scolaire souffrant d' allergies alimentaires a fortement grimpé . -C' est la première fois qu' un haut fonctionnaire chinois a nommé une organisation spécifique à la suite de l' attaque lundi . C' est la première fois qu' un responsable chinois désigne une organisation particulière après l' attentat de lundi . -C' est la première fois que la surveillance du FMI et de la Banque mondiale par l' agence de renseignement est mentionnée depuis le début du scandale . C' est la première fois qu' est mentionnée la surveillance du FMI et de la Banque mondiale par l' agence de renseignement depuis le début du scandale . -Ceci devrait inclure la possibilité de temps de voyage possible de Glasgow et d' Édimbourg à Londres de trois heures ou moins . Cela doit inclure la possibilité de durées de voyage entre Glasgow et Édimbourg de trois heures au plus . -C' est ce qui se passe au du Père depuis 1998 . C' est le cas au du Père , depuis 1998 . -C' est ce qui a inspiré un exercice de consultation entre les parents , les étudiants , les enseignants et le personnel du département de l' éducation il y a un an pour réfléchir sur les moyens d' améliorer l' intégration des élèves qui ont des racines dans plusieurs cultures . C' est ce qui a motivé , il y a un an , une consultation auprès des parents , élèves , professeurs et membres du personnel de l' éducation , pour réfléchir aux moyens de mieux intégrer les élèves qui vivent à cheval entre plusieurs cultures . -Ce jugement signifie qu' il sera possible d' appliquer une loi qui a été adoptée dans cet État en juillet et qui soulève en grande partie la question du droit à l' avortement . Ce jugement signifie qu' un texte de loi adopté en juillet dans cet État et remettant largement en cause le droit à l' avortement va pouvoir entrer en vigueur . -Ce type d' expérience fait partie des efforts de Disney pour " étendre la vie de ses séries et créer de nouvelles relations avec les auditoires par le biais de plateformes numériques qui deviennent de plus en plus importantes " , ajoute-t-il . Ce type d' expérience entre dans le cadre des efforts de Disney pour " étendre la durée de vie de ses séries et construire de nouvelles relations avec son public grâce à des plateformes numériques qui sont de plus en plus importantes " , a-t-il ajouté . -Ce dernier raid effectué par les forces aériennes israéliennes sur le territoire syrien ( le sixième depuis le début de l' année , selon le quotidien israélien Ha' aretz ) n' a été confirmé ni par Israël ni par la Syrie . Ce nouveau raid de l' aviation de en territoire syrien ( le sixième depuis le début de l' année , selon le quotidien israélien ) n' a été confirmé ni par Israël ni par la Syrie . -Cette organisation , qui dépend d' un réseau de militants et de sources médicales , n' a pas été en mesure d' identifier l' origine des explosions . Cette organisation s' appuyant sur un réseau de militants et sources médicales n' avait pas été en mesure d' identifier l' origine des explosions . -Ce drame pornographique dans huit chapitres qui sera publié en deux parties ( le 1er janvier et le 8 ) traite des souvenirs érotiques d' un coincé sur le sexe depuis son adolescence . Ce drame pornographique , en huit chapitres qui sortiront en deux volets ( les 1er et 8 janvier 2014 ) , les souvenirs érotiques d' une accro au sexe depuis son adolescence . -Ce principe de comparaison peut être utilisé à d' autres fins . Ce principe de comparaison peut avoir d' autres applications . -Ce projet est un élément clé de la sécurité énergétique de l' ensemble du continent européen . Ce projet est un élément clé de la sécurité énergétique de l' ensemble du continent européen . -Cette notation est accompagnée d' une perspective négative , avec S & P au moins une chance dans trois d' entre eux de ramener la notation à nouveau dans l' année à venir . Elle est assortie d' une perspective négative , S & P voyant au moins une chance sur trois qu' elle soit de nouveau abaissée d' ici un an . -Cette , intitulée The Next Half Day , a été présentée sous la forme de trois disques : l' album original , les séances de studio non publiées et les , ainsi qu' un DVD contenant les quatre clips qui ont été dévoilés . Cette réédition , intitulée The Next Day Extra , se présentera sous la forme de trois disques : l' album original , des sessions studio inédites et des , ainsi qu' un DVD contenant les quatre clips déjà dévoilés . -C' est vraiment une nécessité pour notre nation . Cela est vraiment indispensable pour notre nation . -Cette révélation vient à 68 ans après la chute du régime nazi d' Adolf Hitler et un des grands mystères de l' après-guerre . Cette révélation , 68 ans après la fin du régime nazi d' Adolf Hitler , répondrait à l' une des grandes énigmes de l' . -Cependant , cette situation n' a pas créé de car l' accident a eu lieu à la sortie de . Cette situation n' a toutefois guère provoqué d' embouteillages car l' accident a eu lieu à hauteur de la sortie . - ( si c' est possible ) le bruit de l' énergie et de l' énergie finit par être assourdissant et , Cette décharge sonore et énergique , longue , si cela se peut , finit par et . -Cette fois , nous sommes prévenus qu' une Écosse indépendante serait obligée d' adhérer à la zone à l' échelle européenne comme condition de l' adhésion à l' UE . Cette fois-ci , on nous prévient qu' une Écosse indépendante serait tenue de rejoindre la zone européenne de libre circulation pour pouvoir adhérer à l' UE . -Ce négociateur , Richard , a quitté en 2010 et est entendu qu' il sera laissé de côté de sa position actuelle en tant que leader européen du commerce de billets d' argent à . Ce trader , Richard , a quitté en 2010 et aurait été mis suspendu de son poste de responsable européen du trading au comptant pour les devises chez . -Il s' agissait d' un appel très inhabituel . C' était un appel très inhabituel . -Il s' agissait d' une demande officielle de " ne pas faire de risque à l' égard des services d' urgence à Paris au début de l' hiver " , mais aussi d' éviter d' entraver la campagne du parti socialiste quelques mois avant les élections locales . Une demande officiellement faite " pour ne courir aucun risque sur la prise en charge des urgences à Paris au début de l' hiver " mais qui visait aussi à ne pas gêner la campagne du PS à quelques mois des municipales . -Il s' agissait de sa promesse sur son lit de mort ; alors , de temps en temps , nous avons appris à nouveau à sourire , et dire que son nom n' est plus tabou , mais un réconfort . C' était sa promesse sur son lit de douleur , alors on a petit à petit à sourire , et prononcer son nom n' est plus tabou mais un réconfort . -Cela a été rendu public le lendemain de la publication de statistiques officielles montrant que l' économie du pays avait connu sa cinquième baisse trimestrielle entre juillet et septembre . Elle a été rendue publique au lendemain de la publication de statistiques officielles montrant que l' économie du pays avait subi entre juillet et septembre son cinquième trimestre de suite de contraction . -Il s' agissait de la première acquisition majeure de l' entreprise après l' échec de sa tentative de au Portugal . Il s' agissait de la première acquisition majeure de l' entreprise après celle qui avait échoué au Portugal . -Il s' agissait de la dernière de trois toujours en service dans le département du Finistère , les deux autres ayant été ou . Il s' agit du dernier des trois portiques en état de fonctionner dans le département du Finistère , les deux autres ayant été démontés ou . -De cette façon , vous êtes OK - élever l' enfant , aimer l' enfant . Ainsi , tout va bien – vous pouvez élever votre enfant et l' aimer . -Cette semaine , j' ai lu la partie du livre expliquant comment Obama a subi une perte considérable à Clinton dans la Pennsylvanie primaire . Cette semaine , j' ai lu la partie du livre qui décrit comment Obama a subi d' énormes pertes au profit de Clinton aux de Pennsylvanie . -Cette semaine Standard & Poor , l' agence de notation , a cité l' augmentation de l' influence des ménages , principalement de la hausse des hypothèques , en tant que facteur de risque pour la solvabilité des banques asiatiques . Cette semaine , l' agence de notation Standard & Poor' s a cité l' endettement croissant des ménages , découlant principalement de la hausse des prêts hypothécaires , comme un facteur de risque pour la solvabilité des banques asiatiques . -Cela garantira que quelqu' un n' essaie pas d' usurper l' identité d' une autre personne . Cela permet de s' assurer qu' une personne ne tente pas d' usurper l' identité de quelqu' un d' autre . -Cela serait historique - c' est la première fois que le gouvernement de Kinshasa serait en mesure d' annuler une importante insurrection . Ce serait historique - ce serait la première fois que le gouvernement de Kinshasa réussit à battre une insurrection majeure . -Cette année , l' événement se déroule sous les noms de , Kathy Griffin et Bill Maher , ainsi que de " for Heroes " , un grand prix pour les vétérans militaires de la région , au Madison Square Garden , qui comprend , entre autres , Bruce , Jon Stewart , Francisco Lois , et Bill . L' évènement de cette année verra la participation de , Kathy Griffin et Bill Maher et la présentation de « Stand Up for Heroes » , une soirée de comédie et de musique au profit des anciens combattants , organisée au Madison Square Garden , à laquelle seront présents , entre autres , Bruce , Jon Stewart , Roger Waters et Bill . -Cette année , les Américains environ 106 millions de dollars sur les , selon le Census Bureau américain . Cette année , les Américains environ 106 M $ en , selon l' U.S. Census Bureau . -Cette année , les objets sexy sont tous de la rage . Cette année , les objets sexy font fureur . -Thomas , député responsable de la commission parlementaire chargée de surveiller le renseignement , a déclaré que s' il était possible d' entendre M. comme un témoin " sans le craindre et risquer complètement de déstabiliser les relations avec les États-Unis " , il faut le faire . Thomas , le député qui dirige le groupe parlementaire supervisant les services secrets étrangers , a indiqué que s' il y avait une chance d' entendre M. à titre de témoin « sans le mettre en danger et complètement ruiner les relations avec les États-Unis » , il fallait en profiter . -Ces conducteurs paieront bientôt les frais de kilométrage plutôt que les taxes sur les gaz à l' État . Ces automobilistes paieront bientôt à l' État des frais au lieu des taxes sur l' essence . -Bien qu' elle soit stérile , elle aspire à devenir mère en adoptant ou en adoptant une maternité . Bien qu' elle soit stérile , elle espère avoir des enfants par l' entremise de services d' adoption ou de mères porteuses . - les agents de voyages était une chose du passé grâce à l' internet ? Vous pensiez que les agences de voyage appartenaient au passé à cause d' Internet ? -Des milliers d' automobilistes ont déjà pris les boîtes noires , dont certaines ont une surveillance GPS , pour un lecteur d' essai . Des milliers d' automobilistes ont déjà embarqué ces boîtes noires , parfois équipées d' un système de surveillance par GPS , pour une virée expérimentale . -Trois hommes qui ont dit qu' ils travaillaient parce que les conducteurs ont été accusés de posséder de la marijuana et de la cocaïne avec l' intention de les distribuer . Trois hommes qui , selon les autorités , travaillaient comme chauffeurs , ont été accusés de possession de marijuana et de cocaïne avec intention de vente . -Trois pharmaciens sont déjà retournés au travail après un congé de maternité et trois autres ont été engagés au cours des derniers mois . Déjà , trois sont de retour au travail après leur congé de maternité et trois autres ont été embauchées au cours des derniers mois . -Au cours de l' exercice 2013 , le bénéfice net de Oracle a progressé de 3,5 % , tandis que son chiffre d' affaires a augmenté de 27,5 % , surpassant l' indice S & qui a augmenté de 24 % au cours de la même période . Sur l' ensemble de l' exercice décalé 2013 , le bénéfice net d' Oracle a progressé de 3,5 % alors que le cours de Bourse à bondi de 27,5 % , l' indice S & qui a pris 24 % dans le même temps . -Ainsi , j' ai appris et j' ai compris que je n' avais pas perdu ma mère seule , une femme que je ne connaissais pas très bien , une femme qui l' a incarné pendant son séjour sur cette terre . J' ai donc appris et compris que je n' avais aucunement perdu la personne de ma mère mais simplement une femme que je ne connaissais pas vraiment , une femme qui incarnait cette personne , durant son séjour dans la vie . -Si vous aimez le travail , vous devez être à la hauteur ... La chance de faire le tour de votre carrière ... et de plus en plus ... " , n' hésitez pas à nous le faire savoir dès aujourd' hui ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! Pour Justin Timberlake , si le travail est nécessaire pour " être considéré comme bon " , " le hasard peut aussi lancer la carrière de quelqu' un " , et " de plus en plus aujourd' hui " . -Le temps est le test le plus précis de la façon dont la puberté va progresser . Le temps est le test le plus précis pour savoir comment va évoluer la puberté . -Titan n' a pas besoin d' acheter cette usine . Titan n' a pas besoin d' acheter cette usine . -Titan a accepté de les recruter parmi les 1200 personnes qui travaillent actuellement pour Goodyear . Titan accepte de les recruter parmi les quelque 1 200 salariés actuels de Goodyear . -Pour être précis , il n' est pas autorisé à communiquer avec des personnes de moins de 18 ans ou à accéder à Internet . Il ne doit notamment pas communiquer avec des personnes de moins de 18 ans et ne pas accéder à Internet . -Pour en savoir plus sur la façon dont les chiens réagissent à la queue d' autres chiens , les chercheurs ont surveillé les animaux pendant qu' ils regardaient des films d' autres chiens . Pour en savoir plus sur la façon dont les chiens réagissent aux mouvements de queue d' autres chiens , les chercheurs ont suivi des animaux en train de regarder des films où figuraient d' autres chiens . -Avoir 20 ans , après avoir tué 202 personnes et blessé plusieurs centaines de personnes , ce n' est pas grand chose . Passer 20 ans en prison après avoir tué 202 personnes et blessé des centaines d' autres , ce n' est pas beaucoup . -Pour le maintenir en vie , il n' y a pas de raison de le garder vivant . Le garder en vie , pourquoi , il n' y a aucune raison de le garder en vie . -Pour que les gens , pour réduire la douleur , c' est pour ça que je fais ça . Faire rire les gens et soulager la douleur sont les raisons qui me poussent à faire cela . -À cette fin , nous avons commandé le plus grand et le plus complet des chefs d' entreprise britanniques qui leur demandaient des idées sur la Grande-Bretagne , les entreprises et l' UE . Pour ce faire , nous avons commandé l' enquête la plus large et la plus complète auprès des chefs d' entreprise britanniques et leur avons demandé ce qu' ils pensaient de la Grande-Bretagne , des entreprises et de l' UE . -Pour comprendre cet aspect unique de l' identité française , il n' y a rien de mieux qu' un court voyage en dehors de la métropole , si bien que la part de l' entreprise de Paul Montana pourrait être le coeur de son livre . & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; Pour faire sentir cette singularité de l' identité française , rien ne vaut les quatre pas hors de l' , si bien que l' escapade américaine de Pierre Nora pourrait bien être le coeur de son livre . -Les appareils électroniques d' aujourd' hui émettent en général de très faibles charges de transmission radio que les générations précédentes d' appareils . Les appareils électroniques d' aujourd' hui émettent en général des transmissions radio de puissance bien plus faible que les générations précédentes d' appareils . -Aujourd ' hui , on s' attend à ce que tombe aussi . Aujourd' hui , devrait tomber d' elle-même . -Aujourd ' hui , il défend les autres dans une organisation appelée le Projet d' interface , essayant de normaliser la perception de ceux qui sont . Aujourd' hui , il défend la cause des autres dans une organisation appelée Interface Project , qui essaie de normaliser la perception des personnes . -Aujourd' hui , une fois de plus , tout le monde est emballé dans la compassion , l' émotion et se réjouit que les otages soient libres . Aujourd' hui , une fois de plus , tout le monde est dans la compassion , dans l' émotion , et se réjouit de la libération de ces otages . -Aujourd' hui , certaines de ces classifications ont été remises en question à la suite des progrès de la génétique . Aujourd' hui , certains de ces classements sont remis en cause en raison des progrès de la génétique . -Aujourd' hui , ces deux scènes des performances telles que des hits . Aujourd' hui , ces deux scènes les titres comme des tubes à succès . -Aujourd' hui à Liverpool Crown Court , qui travaille en tant que membre du conseil d' administration de , siégeant au conseil de Blackburn , a été victime de honte , après avoir été jugé coupable de deux manquements à l' emprise de la justice , à la suite d' un procès de trois semaines à la Crown Court de Liverpool . Aujourd' hui , à la cour d' assises de Liverpool , , qui travaille dans le cabinet d' avocats Forbes situé à Blackburn , est tombé en disgrâce après avoir été reconnu coupable sur deux chefs d' accusation d' entrave à la justice après un procès de trois semaines . -Les filles d' Adam , Frances et Bridget . Les filles de Tony Abbott , Frances et Bridget . -Tony Blair a déclaré qu' il pourrait saisir la possibilité de revenir en tant que Premier ministre britannique , mais il est peu probable qu' un retour soit perçu . Tony Blair a déclaré qu' il l' occasion de redevenir Premier ministre , mais reconnaît que son retour est peu probable . -Le maire de la ville de Toronto Off Journalists qui cherche à s' entretenir avec lui de l' affaire Drogue Le maire de Toronto chasse des journalistes venus l' interroger sur une affaire de drogue -Toyota et Nissan ont également déclaré des ventes inférieures aux attentes , malgré des hausses de 8,8 % et de 14,2 % respectivement sur une année . Toyota et Nissan affichent également des ventes inférieures au consensus , malgré des hausses sur un an de 8,8 % et 14,2 % respectivement . -Le trafic est retourné à la normale à 6 heures du matin . La circulation était revenue à la normale vers 6h , vendredi . -Le trafic a donc été détourné par la sortie / l' entrée du périphérique . La circulation a dès lors pu être par cette sortie / entrée du ring . -La ministre des Transports , Mme la députée , a déclaré que le projet " le Royaume-Uni " . La ministre des Transports baronne Kramer a déclaré que le projet « rassemblerait le Royaume-Uni » . -Les passagers et le personnel se sont précipités vers les portes ou vers l' aire de trafic . Les voyageurs et le personnel se sont vers les sorties ou sur le . -Faire du trekking dans la boue , les rivières et la jungle pour assurer des soins médicaux gratuits Il marche dans la boue , la jungle et traverse des rivières pour offrir une assistance médicale gratuite -Il est vrai que le document a souligné les avantages économiques potentiels d' une plus grande liberté de circulation avec le reste de l' Europe , bien que prévisible - ce qui n' a pas figuré dans les manchettes . S' il est vrai que le journal de a indiqué les éventuelles retombées économiques d' une plus grande liberté de circulation avec le reste de l' Europe , cela ne figurait pourtant pas dans les gros titres ( et c' était à prévoir ) . -Vingt secouristes , dix membres civils de l' organisation française , ainsi que des membres de la police , des services de secours en montagne et des pompiers étaient présents . Une vingtaine de secouristes , dix civils membres du secours français , ainsi que des gendarmes , CRS et pompiers étaient sur place . -Deux âgés de 23 et 27 ans ont été portés disparus dans une grotte à de jeudi soir , selon un rapport de l' Agence de sauvegarde des Alpes de l' le vendredi . Deux de 23 et 27 ans étaient portés disparus dans un gouffre sous la de depuis jeudi soir , a-t-on appris vendredi auprès du secours de l' Isère . -Deux employés du YMCA de Nouvelle-Galles du Sud ont été accusés d' infractions à l' égard des enfants avant qu' il n' ait été porté à la connaissance de l' agent de la protection de la jeunesse , le fils de Victoria , en 1972 , l' affaire de la violence sexuelle que la Commission royale d' enquête a entendue . Deux employés du YMCA de Nouvelle-Galles du Sud ont été accusés d' agressions sexuelles contre des enfants suite à des allégations formulées à l' encontre de l' éducateur de , Jonathan Lord , en 2011 , lors d' une audience de la Commission royale chargée des abus sexuels sur les enfants . -Deux employés du Ministère de la condition de la femme ayant commis des délits contre le racisme avant l' âge de 13 ans de la part de la Reine , la commission royale s' interroge : Deux employés du YMCA accusés d' agressions sexuelles suite à des allégations formulées à l' encontre de Jonathan Lord , lors d' une audience de la Commission royale . -Deux jours avant le début du procès du président déchu , Mohamed Morsi , ils se sont introduits dans la rue . Deux jours avant l' ouverture du procès du président déchu Mohamed Morsi , ils sont descendus dans les rues . -Deux ont le suspect et l' ont aperçu à la sortie de la maison de et ont vu les drogues qu' il détenait d' une pureté de 60 % sous les vases d' avocat dans un gant de golf noir . Deux enquêteurs ont suivi le suspect et l' ont repéré arrivant au domicile de et plaçant de la drogue , pure à 60 % , sous les poubelles de l' avocat dans un gant de golf noir . -Deux mois après l' élection , la commission électorale a officiellement déclaré M. Palmer , le gagnant du siège de la Sunshine Coast de , par 53 voix , après un . Deux mois après son élection , la commission électorale a officiellement déclaré que M. Palmer a remporté le siège de Sunshine Coast de par 53 voix , après . -Deux autres concerts auront lieu au cours de l' été 2014 , ce qui donnera aux artistes de la région l' occasion de montrer ce qu' ils sont . Deux autres concerts auront lieu à l' été 2014 , ce qui donne la chance à des artistes de la région de montrer de quoi ils sont capables . -Deux autres agents locaux des Brigades , Mohammed et Mohammed , ont été tués lorsqu' un hélicoptère israélien a ouvert le feu dans la même zone . Deux autres responsables locaux des Brigades , Mohammed et Mohammed , ont trouvé la mort lorsqu' un hélicoptère israélien a ouvert le feu dans la même région . -Deux personnes sur cinq dans le pays vivent en dessous du seuil de pauvreté , et près des trois quarts des dépenses de santé du pays sont privées . Deux personnes sur cinq dans le pays vivent en dessous du seuil de pauvreté , et presque trois quarts des dépenses de santé du pays sont réalisées dans le secteur privé . -Deux personnes ont été tuées et un autre grièvement a été tué le lendemain par des coups de feu tirés par deux personnes sur une moto qui avait ouvert un bureau à l' aide d' un tireur d' élite , Golden Dawn , dans la banlieue ouest d' Athènes , une source de police a déclaré . Deux personnes ont été tuées et une autre a été grièvement blessée vendredi soir par des coups de feu tirés par deux personnes qui se trouvaient à bord d' une moto qui passait devant un local du parti Aube dorée dans la banlieue ouest d' Athènes , a-t-on appris de source policière . -Deux véhicules entrent en collision sur la route 131 dans la région de Lanaudière , laissant quatre blessés le vendredi matin . Une collision entre deux véhicules a fait quatre blessés , tôt vendredi , sur la route 131 , dans Lanaudière . -En règle générale , applique à Singapour un taux annuel effectif de 17 % , soit juste au-dessus du niveau de 15,4 % offert à la United Overseas Bank , un prêteur local disposant d' une succursale dans le même centre commercial . En général , les prêteurs sur gage de Singapour prélèvent un taux d' intérêt annuel effectif de 17 % , juste au-dessus des 15,4 % offerts par l' United Overseas Bank , un bailleur de fonds local ayant une succursale dans le même centre commercial . -Le procureur américain Michael Moore a déclaré jeudi qu' il menait une enquête officielle sur la mort de Johnson , notant que plusieurs questions clés demeurent sans réponse . Le procureur des États-Unis Michael Moore a déclaré jeudi qu' il menait une enquête officielle sur le décès de Johnson , soulignant que plusieurs questions importantes restaient sans réponse . -UBS a déclaré cette semaine qu' elle avait agi à l' encontre de certains de ses employés à la suite de l' accord de l' agence suisse de surveillance des marchés , , qui avait fait état d' une enquête sur la manipulation du marché des changes dans plusieurs banques suisses . UBS a expliqué cette semaine qu' elle avait intenté une action à l' encontre de certains de ses employés après que le régulateur suisse , , a déclaré qu' il enquêtait sur des suspicions de manipulations du marché des changes dans un certain nombre de banques suisses . -Le secrétaire général des Nations unies , M. Ban Ki-moon , a déclaré lors du sommet de cette nuit que le moment était venu de " dépasser les intérêts nationaux " . Le secrétaire général de l' ONU Ban Ki-moon a dit à la veille du sommet qu' il était maintenant temps de « dépasser les intérêts nationaux » . -Les Nations Unies saluent de nouveaux objectifs en matière de lutte contre la pauvreté . L' ONU salue les nouveaux objectifs en faveur de la lutte contre la pauvreté -Les chercheurs américains , français et japonais prédisent un avenir où nous serons reconnus par des capteurs biométriques sur les téléphones et les ordinateurs . Les chercheurs américains , français ou japonais nous prédisent un avenir où nous serons reconnus par les capteurs biométriques des téléphones et des ordinateurs . -Fusion par les États-Unis Feu vert des Etats-Unis à la fusion -US Airways a tenté une fois de plus de facturer des boissons en 2008 , mais a finalement refusé sept mois plus tard après que les passagers ont été informés et qu' aucun autre transporteur principal n' a suivi . US Airways a brièvement tenté de faire payer pour les boissons en 2008 mais est revenue en arrière sept mois plus tard après que les passagers se sont plaints et alors qu' aucune autre grande compagnie aérienne n' avait suivi . -Le président américain Barack Obama a célébré la tradition de l' Halloween . Aux Etats-Unis , le président Obama n' a pas dérogé à la tradition d' Halloween . -Les autorités américaines ont décrit les quatre pieds de trois comme l' un des passages les plus secrets jamais découverts . Les autorités américaines ont décrit le tunnel de 4 pieds par 3 comme l' un des passages secrets les plus sophistiqués qu' elles avaient jamais vu . -L \ de l \ de l \ L' Ukraine s' approche de la faillite -Ulrich , membre du conseil d' administration de Audi , déclare qu' il est pratique courante depuis des années d' envoyer des téléphones mobiles avant les réunions du conseil d' administration afin de ne pas être utilisés comme appareils d' écoute à distance . Ulrich , membre du conseil d' administration du constructeur automobile Audi , déclare que la collecte des téléphones portables avant les réunions du conseil , afin qu' ils ne puissent pas être utilisés comme appareils d' écoute à distance , est une pratique courante depuis des années . -En fin de compte , l' enfant décidera de quel sexe il se sent plus à l' aise avec lui - et c' est une chose merveilleuse . Enfin , l' enfant décidera avec quel sexe il / elle se sent le / la plus à l' aise – et c' est quelque chose de formidable . -Dans le cadre de l' alliance Renault-Nissan , Renault détient 43,4 % de la motorisation du Japon et le constructeur japonais , 15 % de la société française , selon des données sur le site de Nissan . Dans le cadre de l' alliance Renault-Nissan , Renault détient 43,4 % du capital de Nissan et le constructeur japonais 15 % du français , selon les données du site internet de Nissan . -Selon les règles actuelles , les soldats gravement blessés ont jusqu' à trois ans pour se rétablir . En vertu des règles actuelles , les soldats grièvement blessés ont jusqu' à trois ans pour se rétablir . -En vertu du nouveau projet de loi , les médias peuvent être condamnés à une amende pouvant aller jusqu' à 20 millions de shillings kényans et à des journalistes d' au moins un million , le risque étant de " " ou de ne pas être autorisés à recevoir la presse écrite . En vertu du nouveau projet de loi , les médias pourront être passibles d' amendes allant jusqu' à 20 millions de shillings kenyans et les journalistes jusqu' à 1 million avec le risque supplémentaire d' être « radié » ou de ne plus pouvoir recevoir d' accréditation presse officielle . -Selon les nouvelles lignes directrices , les compagnies aériennes dont les aéronefs sont parfaitement protégés contre les intrusions électroniques peuvent permettre aux passagers d' utiliser les appareils au cours des , atterrissages et , selon la FAA . En vertu des nouvelles directives , les compagnies aériennes dont les avions sont correctement protégés des interférences électroniques pourront autoriser les passagers à utiliser leurs appareils pendant les , les atterrissages et le roulement au sol , a déclaré la FAA . -Les experts de l' ONU accusent le Rwanda d' avoir fourni un appui militaire à la , qui s' est d' abord constituée de anciens soldats congolais , ce que le Rwanda a nié catégoriquement . Des experts des Nations unies accusent le Rwanda , qui le nie farouchement , de soutenir militairement le , constitué au départ d' anciens militaires congolais . -États-Unis pour l' Halloween Les Etats-Unis aux couleurs d' Halloween - Open Fire on Hotel at Cairo Des inconnus tirent sur un hôtel près des pyramides du Caire -Des inconnus ont ouvert le feu sur un hôtel près des pyramides du Caire , en Égypte , le vendredi , ce qui n' a pas été le cas dans l' incident , ce qui s' explique par un argument impliquant des travailleurs qui ont été licenciés . & # 160 ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Des inconnus ont ouvert le feu ce vendredi sur un hôtel proche des pyramides du Caire en Égypte sans faire de victimes , conséquence d' une apparente querelle avec des salariés licenciés . -Jusqu' à tout récemment , il n' était pas vraiment désolé de la douleur qu' il avait causée à d' autres personnes . Jusqu' à récemment , il se du chagrin qu' il avait pu causer à d' autres personnes . -Jusqu' à présent , l' identification a consisté à émettre des documents d' identité : les systèmes biométriques ( empreintes digitales , photos , iris , etc. ) doivent avoir accès à l' information que l' État détient sur chaque citoyen déjà enregistré . L' identification concernait jusqu' à présent la délivrance de papiers d' identité : il faut confronter les éléments biométriques ( empreintes , photo , iris ) avec les informations que l' Etat possède sur tous les citoyens déjà . -Jusqu' à récemment , Airbus mettait l' accent sur la nécessité d' améliorer la cabine en proposant des sièges plus larges sur certains de ses jets . Jusqu' à récemment , Airbus soulignait la nécessité d' une plus grande personnalisation des cabines en offrant des sièges plus larges côté allée sur certains de ses avions . - a dit que les papes ont généralement élevé les évêques de grands lieux à la position de cardinal , mais que François est " plein de surprises - bien sûr , il ne sait pas qui le nommer " . a indiqué que les papes élevaient en général des évêques représentant de vastes communautés au rang de cardinal , mais que François était « plein de surprises – donc nous ne savons pas qui il nommera » . -Enquête chrétienne sur la contraception , le divorce et l' homosexualité Le Vatican sonde sur la contraception , le divorce et l' homosexualité -Nouveau casque spécial utilisé à Abu Dhabi Vettel un casque spécial à Abou Dhabi - a choisi l' entrée gagnante à partir de 1,500 exemplaires provenant de tous les coins du monde dans le cadre d' un concours organisé par l' un des sponsors , une voiture de marque dans son écurie . Vettel a choisi parmi 1.500 dessins envoyés du monde entier , dans le cadre d' un concours lancé par l' un des sponsors , une marque automobile , de son écurie . -On s' attendait à ce que les pneus portent le casque lors des séances d' essais du vendredi et du samedi . Vettel devait porter ce casque vendredi et samedi , aux essais . -Virgin , qui s' est déjà adressé à la CASA à propos de l' extension de son système de divertissement en ligne sans fil , a pu s' adapter à un changement , mais il a dit qu' il devrait prendre le relais de l' autorité de régulation . Virgin , qui a déjà parlé à la CASA de son intention d' étendre l' utilisation de son système de divertissement à bord utilisant la technologie Wi-Fi , était ouverte à un changement , mais a déclaré qu' elle suivrait le régulateur . -Wallis Simpson a peut-être été . Wallis Simpson aurait pu être . -Est-ce que Bowie essaie de faire référence à John Rocket de l' homme , ou même à la gravité , dans son propre ? Sur Like A Rocket Man , Bowie a-t-il voulu faire une petite référence au Rocket Man d' John , voire même à ? - été condamné à une peine de prison de trois ans à la Crown Court de Liverpool Il a été condamné à une peine d' emprisonnement de trois ans par la cour d' assises de Liverpool -Est-ce que son décès est le résultat d' un crime ? Son décès est-il le résultat d' un acte criminel ? -Est-ce une mauvaise blague impliquant des gaz lacrymogènes ? Mauvaise plaisanterie sur fond de jet lacrymogène ? -Moyens de rassurer les utilisateurs Des pistes pour rassurer les utilisateurs -Nous sommes heureux de pouvoir vous offrir ce mode de partenariat en partenariat avec Land Rover et le CPA , et la preuve est que le nouveau style est un moyen de découvrir d' autres disciplines de rallye au Royaume-Uni et à l' étranger , et de préparer les participants pour les rigueurs et les réalités du rallye de Raid . Nous sommes vraiment heureux d ’ offrir une telle approche en partenariat avec Land Rover et la MSA , et nous sommes convaincus que ce sera une nouvelle façon de s ’ essayer à différentes formes de rallyes au Royaume-Uni et à l ’ international , et de se préparer aux rigueurs et aux réalités des rallyes raids . -Nous sommes tout à fait fatigués de sentir comme des " " lorsqu' il s' agit de ce problème . Nous sommes franchement fatigués de nous sentir comme des « surveillants de salle » lorsqu' il s' agit de cette question . -Nous devons nous souvenir de qui nous sommes " . Nous devons nous rappeler qui nous sommes . -Nous avons vraiment aimé développer le Defender Call Challenge - c' est un championnat vraiment fun . Nous avons eu beaucoup de plaisir à développer le véhicule du Defender Challenge – le championnat va vraiment être palpitant . » -Nous pensons aussi que parfois les photos n' ont pas besoin d' explication ou de commentaire . Nous pensons aussi que parfois les images n' ont pas besoin d' explication ou de commentaire . -Nous sentons toujours qu' elle nous protège , rien de mauvais ne se produira à nous . On sent en permanence sa protection , rien de grave ne peut nous arriver . -Nous craignons que cela n' encourage l' intervention . Nous avons peur qu' elle encourage les interventions . -Nous sommes en train de faire marche arrière , parce que les avantages qu' il apporte sont énormes . Nous le projet car les avantages qu' il présente sont énormes . -Nous sommes contraints d' augmenter le loyer pour nous aider à payer les nouvelles taxes . Nous sommes obligés d' augmenter les loyers pour nous aider à payer les nouvelles taxes . -Nous prévoyons une chute très probable de l' euro . Nous anticipons un fort potentiel de baisse de l' euro . -Nous sommes tout à fait reconnaissants et chacun d' entre eux était absolument extraordinaire . On leur est tellement reconnaissants , chacun d' eux a été absolument formidable . -Nous croyons que Bombardier recevra de nouvelles commandes pour le à mesure que le programme d' essais de vol. Nous pensons que Bombardier enregistrera de nouvelles commandes pour ses avions au fur et à mesure que le programme d' essais en vol avancera . -Nous avons construit ce festival comme un moyen de continuer à travailler avec eux . Nous avons créé ce festival dans le but de pouvoir continuer à travailler avec eux . -Nous pouvons choisir d' être une région de ( essentiellement ) la Petite Angleterre . Nous pouvons décider d' être une région de la petite Angleterre ( essentiellement ) . -Nous avons certainement travaillé dur pour essayer de persuader nos jeunes résidents [ de pharmacie ] de venir ici . C' est sûr qu' on a travaillé fort pour intéresser nos jeunes [ résidents en pharmacie ] pour qu' ils viennent et restent ici . -Nous n' avons pas à nous précipiter dans une chirurgie qui est irréversible . Vous n ’ avez pas à vous précipiter pour une intervention chirurgicale qui est irréversible . -Nous avons nos chiens l' année dernière et , à mon étonnement , nous n' étions pas seuls . Nous avons déguisé nos chiens l' année dernière et , à mon grand étonnement , ils n' étaient pas les seuls . -Nous nous sentons soutenus et protégés dans des moments difficiles , et le passage du temps nous a permis de constater qu' elle était le chef d' orchestre , avec nous pour nous avancer sur la voie qu' elle nous avait tracée . Nous nous sentons portés et protégés dans les moments difficiles , le recul du temps nous fait réaliser combien elle était chef d' orchestre et nous essayons de rester dans la voie qu' elle nous avait tracée . -Nous avons constaté que la grande majorité des entreprises cherchent maintenant à exporter en dehors de l' Europe , en se concentrant sur les pays qui sont en train de se moderniser et de se développer tandis que les pays de l' UE stagnent . Nous avons découvert que la vaste majorité des entreprises cherchent désormais à exporter en dehors de l' Europe , en se concentrant sur les pays qui se et se développent tandis que les États de l' UE stagnent . -Nous sommes arrivés par hélicoptère et avons pris un avion privé vers la France via Corfou et , où Pasqua est arrivé . Arrivés là-bas en hélicoptère , on prend un avion privé qui nous ramène en France via Corfou et , où Pasqua embarque . -Nous avions décidé de faire un premier vol et de faire une mise à jour , et c' est ce que nous avons fait . Nous avions décidé de procéder à un premier vol et de prévoir une période de mise à jour , et c' est ce que nous avons fait . -Nous avons 200 artistes talentueux qui vont du projet au projet . Nous avons 200 artistes de talent qui tournent de projets en projets . -Nous avons effectivement empêché les avions de descendre , les bâtiments sont en train d' exploser et les gens ont été assassinés parce que nous avons été capables d' apprendre à l' avance les plans . Nous avons bel et bien empêché des avions de tomber , des immeubles d' exploser et gens d' être assassinés parce que nous avons pu prendre connaissance des faits à l' avance . -Nous l' avons testé au cours des derniers mois et il semble parfait de le partager avec Halloween parce qu' il a donné cet effet merveilleux de la lumière . Nous l' avons testée au cours des derniers mois et il semblait opportun de la partager avec tous au moment d' Halloween car elle donne un extraordinaire effet brillant . -Nous avons un support plus solide et stable . Nous avons des appuis plus solides et constants . -Nous avons atterri à Orly . On atterrit à Orly . -Nous devons changer les choses dans la ville . Nous avons besoin de faire changer les choses à la Ville . -Nous devons admettre que les prises d' otages sont des prises d' otages politiques - c' est le plus souvent - et permettre aux otages d' avoir accès à la justice , de porter des accusations de presse et d' amener leurs ravisseurs à la Cour pénale internationale . Il faut qualifier la prise d' otages de prise d' otages " politique " - ce qui est le cas la plupart du temps - et permettre aux otages d' avoir accès à la justice , de porter plainte et d' amener leurs preneurs d' otages devant la Cour pénale internationale . -Nous n' avons plus les moyens de payer 100.000 dollars pour les feux d' artifice , mais 4 à 5 millions de dollars pour la terre juste en cas ... Nous n' avons plus les moyens de nous payer des feux d' artifice à 100 000 $ , des terrains de 4 à 5 millions $ pour au cas où ... -Nous préférons envoyer des robots à Mars plutôt que de savoir ce qui se trouve dans la rivière , à seulement 11 kilomètres sous la surface de l' océan . On préfère envoyer des robots sur Mars , que de savoir ce qui vit dans la fosse des Mariannes , à seulement 11 kilomètres de la surface de la mer . -Nous en avons vraiment besoin " . On a vraiment besoin de ça . -Nous voyons des clients de tous les milieux . Nous voyons des clients de toutes les couches de la société . -Nous voyons sa montée fulgurante , mais aussi la chute vertigineuse qui s' ensuit . L' occasion de suivre son incroyable ascension , mais aussi la chute vertigineuse qui s' en suivra . -Nous pensons qu' un meilleur processus consiste à attribuer un sexe masculin ou féminin , puis à attendre . Nous pensons qu' il vaudrait mieux attribuer un sexe masculin ou féminin , et attendre . -Nous avons ensuite été amenés à l' hôtel où la presse s' est rassemblée . On arrive ensuite sur le site de l' hôtel où la presse est présente . -Nous étions très excités , mais aussi très tendue . On est très excités , mais très tendus aussi . -Nous vous trouverons et vous devant les tribunaux . Nous vous trouverons et vous devant les tribunaux . -Nous tenons à nous excuser auprès de tous les détenteurs de billets pour les inconvénients que cela a causé . Nous souhaitons nous excuser auprès de tous les détenteurs de billet pour tout désagrément causé . -Nous l' aurions emmené du Cameroun si nous avions de l' argent . Nous l' aurions emmené à l' étranger si nous avions eu assez d' argent . -Les riches à Singapour utilisent également des , des barres d' or ou des montres Rolex , qui peuvent commander jusqu' à 60 % de leur prix d' achat en liquide . Les gens aisés de Singapour vont également dans les boutiques de prêt sur gage pour mettre en gage des lingots d' or ou des montres Rolex , qui peuvent représenter jusqu' à 60 % de leur prix d' achat en espèces . - une , il sur le célèbre sanctuaire de à / h. Le sanctuaire est situé à une altitude de plus de 3000 mètres et de nombreux spectateurs se sont rassemblés pour admirer son exploit . Equipé d' un ( une combinaison munie d' ailes ) , il est passé à 160 km / h au-dessus du célèbre sanctuaire , situé à plus de 3 000 mètres d' altitude , où de nombreux s' étaient rassemblés pour observer son exploit . -De plus , contrairement à de nombreuses communes , le projet de loi est respecté par les contribuables plutôt que par des associations professionnelles . Facture qui , contrairement à de nombreuses communes , est en plus réglée par les contribuables , et non pas par les associations de commerçants . -Ce que je vois est une usine qui fabrique de bons pneus agricoles , possède un bon équipement , un bon emplacement et beaucoup de place pour la croissance . Ce que je vois , c' est une usine qui fabrique de bons pneus agricoles , a de bons équipements , est bien située et dispose d' espaces pour s' agrandir . -Ce que Pierre Stein cherche , c' est autre chose . Celle que traque Pierre Nora est tout autre . -Qu' en est-il de la première ? Mais qu' en est-il du premier ? -Quels sont vos souvenirs d' être libérés aujourd' hui ? Quels sont aujourd' hui vos souvenirs de votre libération ? -Que pensez-vous des cheveux de ? Que pensez-vous de la coiffure de ? -Ce qu' il a découvert est terrifiant . Ce qu' il a découvert l' a horrifié . -Qu' est-ce que Le Pen fait avec sa nouvelle popularité parmi les classes blanches françaises ? Que fait Marine Le Pen avec sa nouvelle popularité auprès de la classe ouvrière blanche française ? -Quels risques les États-Unis en réalité , leur image ? Que risquent réellement les Etats-Unis ? une dégradation de leur image ? -Quel genre d' accord entre la CGT et Goodyear ? Quel type d' accord attendez-vous entre la CGT et Goodyear ? -Quelle a été la cause de la mort ? Quelle est la cause de la mort ? -Quoi qu' il en soit , il sera toujours mon point de référence . En tout état de cause , il sera toujours ma référence . -Lorsque François Hollande téléphone Barack Obama , ou lorsque le ministre des affaires étrangères , Laurent , convoque l' ambassadeur des États-Unis , il répond à une véritable découverte , celle de l' ampleur de la surveillance américaine des communications en France en général . & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; Lorsque François Hollande téléphone à Barack Obama ou quand le ministre des affaires étrangères Laurent convoque l' ambassadeur des Etats-Unis , ils réagissent à une vraie découverte , qui est celle de l' ampleur de la surveillance américaine sur l' ensemble des communications en France . -Quand je suis fini avec mon patient à la nuit et se coucher , le livre me retourne à un moment où la politique a inspiré des millions et des discours peuvent sonner . Quand j' ai fini les comptes-rendus médicaux de mes patients le soir et que je me couche , ce livre me à l' époque où les politiques inspiraient des millions de gens et où les discours pouvaient vous couper le souffle . -Quand j' ai quitté le pays , il y avait trois chaînes de télévision : Quand je suis parti il y avait trois chaînes de télévision . -Quand j' étais petite , Halloween était magique . Lorsque j' étais petit , Halloween était magique . -Lorsque la phase 2 sera terminée , les lignes seront acheminées à Manchester et à Leeds . Lorsque la Phase 2 sera achevée , les lignes Manchester et Leeds . -Lorsque WikiLeaks révèle le contrôle de l' administration américaine sur des questions politiques et autres quelque part dans le monde , nous estimons qu' il s' agit là d' une leçon importante pour le gouvernement américain . & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; & # 160 ; Lorsque WikiLeaks lève le voile sur l' analyse par la diplomatie américaine d' enjeux politiques ou autres dans le monde entier , nous considérons en effet que , au regard de la puissance américaine , cela constitue un éclairage important . -Interrogé sur ce point , un responsable de l' administration américaine a répondu : " Les États-Unis ne mènent pas de surveillance électronique aux bureaux de la Banque mondiale et du FMI à Washington " . Interrogé sur le sujet , un responsable de l' administration américaine a répondu : " les Etats-Unis ne mènent pas de surveillance électronique visant les sièges de la Banque mondiale et du FMI à Washington " . -Lorsqu' on leur a demandé des points de vue sur des domaines d' action précis , allant de la réglementation monopolistique aux lois sur les produits , la majorité des chefs d' entreprise pensaient que le contrôle de ces compétences clés devait être renvoyé à Westminster . Lorsqu' on leur demande leur avis sur des points politiques spécifiques – allant de la réglementation des monopoles aux lois sur les produits – la majorité des chefs d' entreprise pensent que le contrôle de ces compétences clés devrait retourner aux mains de Westminster . -Quand il s' agit de bonbons , nous ne nous pas . Lorsqu' il s' agit de bonbons , on ne pas . -Lorsque la police l' a arrêtée , ils ont découvert la boisson à emporter entre les jambes . Lorsque la police l' a arrêté , ils ont découvert la boisson à emporter entre ses jambes . -Lorsqu' elle se rebelle , elle est soumise à la cruauté extrême des mains d' une mère très et devient un objet de fascination érotique pour un autre . Lorsqu' elle se rebelle , elle fait face à la cruauté sans borne d' une Mère Supérieure délibérément et devient un objet de fascination érotique pour une autre . -Quand les animaux ont vu un chien déplacer leur queue vers la droite ( du point de vue du chien ) , ils sont restés parfaitement détendu . Lorsque les animaux ont vu un chien sans expression remuer la queue vers la droite ( du point de vue du chien qui la queue ) , ils sont restés parfaitement . -Lorsque le garçon était âgé d' environ 15 ans , il est retourné à un garçon et a épousé une femme . À l' âge d' environ 15 ans , il est redevenu un garçon et s' est marié avec une femme . -Lorsque ces enfants sont nés , vous n' avez pas appelé le psychiatre , vous avez appelé un chirurgien . Lorsque ces enfants , on n' appelait pas le psychiatre , on au chirurgien . -Lorsqu' ils sont cultivés , ils peuvent prendre des décisions concernant leurs propres corps . Lorsqu' ils , ils pourront prendre des décisions concernant leur propre corps . -Où est-ce que cela quitte l' Écosse ? Et l' Écosse , dans tout ça ? -Alors que avait qualifié la recherche de " diligence raisonnable " , Martin a dit à Anderson Anderson que la semaine dernière , la campagne de souscription avait fait un important investissement pour déterminer si le mouvement pouvait rapporter des dividendes dans les sondages . Tandis que qualifiait l' examen de « concerté » , Martin a indiqué à Anderson Cooper de CNN que la campagne de réélection avait beaucoup investi pour découvrir si le changement serait bénéfique lors du scrutin . -Bien que M. ne dise pas que le président était " conscient du changement potentiel " , l' ancien chef du personnel a admis qu' il était « & # 160 ; possible & # 160 ; » Obama . Bien que ait dit qu' il ne pensait pas que le président « ait été au courant » du changement éventuel , l' ancien secrétaire général a admis qu' il était « possible » qu' Obama ait su . -Alors que Murphy a dit que le but du voyage était d' aider à améliorer les relations , il a dit qu' un " amour difficile " serait également . Murphy a déclaré que le but de ce voyage était de permettre d' améliorer les relations , mais également de faire preuve de « fermeté » . -Tout en bénéficiant du confort , tous les constructeurs offrent également des avions avec des cartes à haute densité pour les compagnies aériennes à bas coûts et les voyages régionaux . Tout en se de proposer des appareils confortables , tous les constructeurs offrent également des avions avec des configurations à forte densité aux compagnies aériennes low-cost et pour les voyages régionaux . -Bien que certains de crédit aient recours à des taux d' intérêt élevés , Mme Yes dit que non seulement les offrent des taux moins élevés que les autres prêteurs , mais qu' ils n' ajoutent pas directement à la dette . Tandis que certains prêteurs à taux réduit ont essuyé les critiques pour les taux d' intérêt exorbitants qu' ils pratiquent , Mlle indique que la mise en gage offre non seulement des taux moins élevés que les autres prêteurs , mais qu ’ en plus elle n' augmente pas directement la dette . -Bien qu' il n' y ait pas d' obligation formelle de consacrer plus de temps au calcul des mathématiques , les membres de la coalition ont déclaré que le grand de mathématiques , combiné à une plus grande pondération pour le sujet dans les tableaux de la ligue , aurait probablement incité les écoles à dispenser un enseignement supplémentaire . Bien qu' il n' y ait aucune obligation formelle de consacrer plus d' heures de l' emploi du temps aux maths , des sources proches du gouvernement de coalition ont indiqué que le de maths approfondi – de même que l' importance plus grande accordée aux matières figurant dans les classements internationaux – encouragerait vraisemblablement les écoles à assurer plus d' heures de cours . -Bien qu' il ne s' agisse pas de gaspillage , la ville ne sera jamais épargnée et la réglementation sévère de son utilisation risque d' être considérablement réduite . Sans vouloir dire de la gaspiller , la ville ne manquera jamais d' eau et la sévère réglementation quant à son utilisation pourrait être grandement . -Qui sont nos constructeurs ? Qui sont nos bâtisseurs ? -Pourquoi les tournées provinciales sont-elles si fructueuses ? Pourquoi les tournées en province ont-elles autant de succès ? -Pourquoi Titan a-t-elle besoin de cette usine aussi ? Pourquoi Titan a-t-il tant besoin de cette usine ? -Pourquoi est-il rare de découvrir de nouvelles espèces de mammifères marins ? Pourquoi est-il rare de découvrir de nouvelles espèces de mammifères marins ? -Pourquoi nous ne devrions pas avoir un système où vous pouvez mettre en marche , vos données dans un ordinateur , votez immédiatement et avoir un résultat à 6.30 la nuit ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Pourquoi ne pourrions-nous pas avoir un système dans lequel vous pouvez entrer , saisir vos coordonnées dans un ordinateur , voter immédiatement et avoir le résultat à 6h30 le lendemain matin ? -Pourquoi est-ce qu' il s' agit d' une découverte importante comme l' a dit la Société de conservation de la faune ? Pourquoi s' agit-il d' une découverte d' importance comme l' a affirmé la Wildlife Conservation Society ? -Pourquoi cela est-il accepté sans difficulté dans certains pays nordiques et protestants , mais reste encore mal en France ? Pourquoi ce qui est accepté sans peine dans certains pays nordiques et protestants , est-il encore mal vécu en France ? -Pourquoi une Ecosse indépendante devrait-elle être en mesure de faire appel à l' Europe ? Pourquoi une Écosse indépendante devrait-elle suivre les instructions de l' Europe , en fait ? -Pourquoi ne pas au : " Vous avez violé les règles de la guerre et celles de toutes les conventions de Genève , nous allons nous tourner vers la justice ? " Pourquoi ne pas aux preneurs d' otages " vous avez bafoué les règles de la guerre , comme celles de toutes les conventions de Genève , on va vous amener en justice " ? -Pourquoi l' attitude du gouvernement français : d' une part , elle publiquement et , d' autre part , l' ambassadeur des États-Unis à l' article 21 et , d' autre part , l' interdiction du parlement de la Grèce d' entrer son espace aérien sur la base d' une rumeur selon laquelle Edward se tenait à bord ? Comment expliquer l' attitude contradictoire du gouvernement français , qui d' un coté s' en public en convoquant l' ambassadeur des Etats-Unis le 21 octobre , et de l' autre interdit le survol du territoire par l' avion présidentiel bolivien , sur la base de la rumeur de la présence à son bord d' Edward ? -Pourquoi ces appels radicaux à l' unité nationale ? Pourquoi ces appels dramatiques à l' unité nationale ? -Pourquoi le libéralisme de Londres pour celui de Bruxelles , surtout aujourd' hui ? Pourquoi échanger le joug de Londres contre celui de Bruxelles , en particulier maintenant ? -Avec ses 420 designs d' inspiration et ses 2 kilomètres de plage de front de mer dans un décor de couleurs , le projet de à Menton pourrait vous donner un petit coup de ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! 420 motifs , 2,2 kilomètres de bord de mer recouverts d' un manteau de lumière : on pourrait le coup de chaud à Menton avec la facture des . -Avec une dette brute de plus de 200 millions de dollars , cette ville n' est même plus capable d' acheter un stylo sans passer par un emprunt . Avec une dette brute de plus de 200 millions $ , cette ville n' est même plus capable d' acheter un stylo sans passer par un règlement d' emprunt . -Avec une statue de Machel , la présidente fondatrice du Mozambique , qui les a retrouvés , des milliers de personnes se sont rassemblées dans le centre-ville de Maputo pour des slogans de paix dans une manifestation publique rare . Aux pieds de la statue de Machel , président fondateur du Mozambique , en train de les , des milliers de personnes se sont rassemblées dans le centre de Maputo pour des slogans pacifiques lors d' une rare manifestation publique . -Avec tout le respect des chefs d' État qui semblent relativement indifférents aux fruits de la terre , " ils sont tous de la nourriture " Bernard a rendu volontairement . N' en déplaise à des chefs d' Etat qui affichaient une relative indifférence aux nourritures terrestres , " ils sont tous gourmands " , dénonce gentiment Bernard . -Avec l' incinération , il y a un " sentiment de violence contre le corps d' un être aimé " , qui sera réduit à un " tas de cendres " au plus tard au lieu d' un processus de décomposition qui " accompagnera les moments de la douleur " . Il y a , avec la crémation , " une violence faite au corps aimé " , qui va être " réduit à un tas de cendres " en très peu de temps , et non après un processus de décomposition , qui " accompagnerait les phases du deuil " . -Avec des centaines de milliers de personnes qui reçoivent des avis d' annulation de la part de leurs fournisseurs , les Républicains ont le président ces derniers jours pour tromper le public américain . Après que des centaines de milliers de personnes ont reçu des avis de résiliation de leurs prestataires , les républicains ont critiqué le président au cours des derniers jours en l' accusant de tromper le peuple américain . -N' ayant nulle part où vivre , il dort sur les canapés des amis qui le encore . Sans logement , il fait la tournée des canapés de ses amis qui veulent bien encore le recevoir . -Avec notre aide , le Royaume-Uni et les Pays-Bas ont mis en place des systèmes automatisés de saisie des passeports qui utilisent la reconnaissance faciale . Avec notre aide , le Royaume-Uni et la Hollande ont déployé des automatiques utilisant la reconnaissance faciale . -Avec la crise de l' euro , le projet Europe est officiellement mort . Avec la crise de l' euro , le Projet Europe est officiellement mort . -À l' exception de l' opéra , les provinces demeurent le parent pauvre de la culture en France . À l' exception de l' opéra , la province reste le parent pauvre de la culture en France . -Avec le générateur de groupe qui l' équipement d' exploitation et de nettoyage , et ses bénévoles travaillent les premières heures du dimanche matin . Grâce à l' éclairage du groupe électrogène , la salle d' opération et la stérilisation du matériel , le Dr et ses bénévoles travaillent jusqu' aux premières heures du dimanche matin . -Sans donner de chiffres précis , le président du Conseil d' administration , Yves Bergeron , a pris la parole lors de la conférence de presse tenue hier à Chicoutimi pour obtenir des assurances . Sans livrer de chiffres précis , le président du conseil d' administration , Yves Bergeron , a profité de la rencontre de presse tenue hier , à Chicoutimi , pour se montrer rassurant . -Sans cela , nous sommes confrontés à une crise de capacité sur notre réseau ferroviaire . Sans lui , nous sommes confrontés à un problème de capacité de notre réseau ferroviaire . -Sans préciser la maladie qu' elle souffrait , l' de " Respect " a confirmé aux médias le 16 octobre que les effets secondaires d' un traitement qu' elle recevait étaient " difficiles " à traiter . Sans préciser la maladie dont elle souffrait , la célèbre interprète de Respect avait affirmé aux médias le 16 octobre que les effets secondaires d' un traitement qu' elle recevait étaient " difficiles " . -Sans la possibilité d' une dévaluation de la monnaie nationale , le Sud de l' Europe se trouve confronté à un désavantage de productivité lié à l' Allemagne . Sans la possibilité d' une dévaluation de la devise nationale , les pays d' Europe du sud se retrouvent avec un problème de productivité intrinsèque vis-à-vis de l' Allemagne . -Sans le soutien de la deuxième poignée de grands jets modernes , les experts disent qu' il est peu probable que l' appel à une nouvelle norme de l' industrie soit peu probable , mais qu' il risque de détourner l' attention d' une vague de ventes à l' échelle de la planète . & # 160 ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; Sans le soutien du seul autre grand constructeur de gros appareils modernes , les experts disent qu' il est peu probable que son appel à l' industrie aéronautique en vue de mettre en place une norme de taille pour les sièges aboutisse , mais qu' il pourrait faire oublier la vague de ventes de . -Sans ces cartes , ces personnes ne peuvent bénéficier de nombreux services sociaux ou de soins de santé dans ces bases , ni accéder aux magasins de ces derniers . Sans ces cartes , ces personnes ne peuvent bénéficier des nombreux services sociaux ou de santé fournis dans ces bases , ou accéder à leurs magasins . - DE DE Elle tente d' importer 2 kg de cocaïne dans des -Les femmes n' ont pas à être des professionnels plus sexy ; elles peuvent également être des aliments sexy comme la pizza , les hamburgers et les carottes . Les femmes n' ont plus besoin d' être des professionnelles sexy ; elles peuvent également être des aliments sexy comme des pizzas , des hamburgers et des carottes . - DANS Un travailleur meurt enseveli sous des blocs de béton -Travailler sept heures par jour , lorsque les gens d' autres pays travaillent huit heures , revient à la France . Travailler sept heures par jour quand dans d' autres pays on travaille huit heures handicape la France . -Le YMCA de n' était pas une organisation à but non lucratif alors que Lord God était utilisé entre 2009 et 2011 , un professeur de violence sexuelle à l' égard des enfants , le professeur Stephen , de l' Université Griffith , a déclaré à la commission . Le YMCA de Nouvelle-Galles du Sud n' était pas une organisation sûre pour les enfants à l' époque où Jonathan Lord y était employé , à savoir entre 2009 et 2011 , a déclaré l' expert en matière de violences sexuelles sur les enfants , le Professeur Stephen de l' université de Griffith devant la Commission . -Oui , mais en ignorant son travail moins connu et son écriture souterraine en tant qu' éditeur . En effet , mais en ignorant le travail caché et l' écriture souterraine de l' éditeur . -Oui encore , mais il sait qu' un tel honneur ne peut remplacer le véritable , l' oeuvre philosophique , qu' il a . Certes encore , mais en sachant qu' un tel sacre ne saurait remplacer l' huile sainte véritable , celle de la thèse , dont il s' est passé . -Hier , Obama a modifié son engagement initial . Hier , Barack Obama a ajusté sa promesse d' origine . -Hier , nous avons pris la colline de , qui donne sur la colline de . Hier , nous avons pris la colline de , qui surplombe celle de . -Hier soir , son épouse et lui ont remis des bonbons à des centaines d' enfants invités dans les jardins de la Maison Blanche à Washington . Avec son épouse , hier soir , il a distribué des friandises à des centaines d' enfants invités dans les jardins de la Maison-Blanche à Washington . -Pourtant , de tous les exemples que j' ai énumérés ci-dessus , ils semblent surtout être le choix de l' administration . Cependant , tous les exemples que j' ai répertoriés ci-dessus semblent être en grande partie le choix de l' administration . -Pourtant , elle est là , une présence silencieuse , me regarde . Pourtant , elle est là , présence silencieuse , elle est là et me regarde . -Pourtant , le niveau de l' émotion et de la fatigue l' a rendue difficile à penser clairement . Mais l' état d' émotion et de fatigue est tel qu' il est difficile d' y voir clair . -Pourtant , il n' existe aucune preuve cohérente que les hormones dans le lait ou dans d' autres aliments , les produits chimiques présents dans l' environnement ou les messages sexuels dans les médias sont à blâmer . Pourtant , il n' y a aucune théorie cohérente selon laquelle les hormones contenues dans le lait ou d' autres aliments , les produits chimiques présents dans l' environnement ou les messages sexuels véhiculés par les médias sont responsables . -Pourtant , cette identité a pris sa place . Mais cette identité n' a eu qu' un temps . -Pourtant , l' Amérique s' efforce de rassembler des données . Mais nous nous efforçons de rassembler des informations . -Pourtant , il y a un point intéressant qui a été oublié lors de cette d' un conte . Pourtant , on a oublié un point intéressant dans cette interprétation d' un vieux conte poussiéreux . -Un gouvernement d' État n' épargne pas près de $ et le Commonwealth a investi 300 millions de dollars et fait valoir qu' il s' agit d' un bon jour de bonne nouvelle . & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; On ne peut pas avoir un gouvernement de l' État qui arrache presque un demi-milliard de dollars et le Commonwealth qui met 300 M $ , et prétendre que c' est un jour faste . -Vous pouvez quitter la maison à 5 heures du matin , à l' hôpital pour être le premier , et vous n' êtes pas le premier . Vous pouvez quitter la maison à 5 h , vous à l' hôpital pour être le premier , et vous n' êtes pas le premier . -Vous pouvez faire une distinction entre les sexes sans chirurgie , puis voir comment l' identité se développe . Il est possible de faire une assignation sexuelle sans opération , et voir comment l' identité se développe . -Vous pouvez voter pour lui , ou l' un des 10 meilleurs héros , pour être un héros de l' animation de l' année . Vous pouvez voter pour lui , ou l' un des 10 autres héros , pour l' élire « Héros CNN de l' année » . -Vous d' un trou et vous êtes soudainement sous le feu des projecteurs . On sort d' un trou et on est tout à coup sous les feux médiatiques . -Vous ne vous pas en public comme si vous ne vouliez pas attirer l' attention " , a déclaré un client , , âgé de 60 ans , aux Cambridge News . " Vous ne sortez pas ainsi dans la rue à moins que vous vouliez attirer l' attention " , a témoigné une cliente , , 60 ans , au journal Cambridge News . -Vous avez provoqué des attaques violentes et des insultes , en parlant de " travailleurs " qui travaillent " trois heures par jour " , et de " fous " , qui ciblent la CGT . Vous aviez prononcé de violentes attaques et des insultes , en parlant de " soi-disant ouvriers " , qui " travaillent trois heures " par jour , de syndicats " fous " , en visant la CGT . -Vous avez un bébé merveilleux et vous amuser . Vous avez un bébé magnifique et profitez du plaisir d' être parent . -Il faut se rappeler , à ce moment-là , que le président était en forme , alors nous étions comme " le Christ saint , qu' est-ce que nous faisons ? " . Rappelez-vous qu' à ce moment-là , le président n' était pas en forme , et donc on se disait , « Seigneur , que pouvons-nous faire ? » . -Les jeunes acteurs sont formés à ces compétences dans de nombreuses écoles . Les jeunes sont formés à ces métiers dans de nombreuses écoles . -et dîner dans un T-shirt à l' Ambassade de France . Et avoir en t-shirt à l' ambassade de France . -comme les noms des grands écrivains et artistes , qu' ils soient d' origine française ou venant d' ailleurs . Des grands noms d' écrivains , d' artistes , qu' ils soient de souche ou qu' ils soient venus d' ailleurs . -aussi bien la régularité du rythme visuel . Comme le fait aussi la régularité du rythme visuel . -parce que , au-delà de la forme imposée au cours de l' université , ce travail philosophique parle de la nécessité intérieure d' une vie . Car bien plus que la figure imposée du cursus universitaire , cette dit la nécessité intérieure d' une vie . -mais seulement à l' écran . Sur écran uniquement . -Dans l' esprit de son fondateur , , le " Parti de la " ne cible pas seulement les résidents des banlieues . Dans l' esprit d' , son fondateur , le " Parti de la banlieue " ne s' adresse pas uniquement aux . -À la suite d' une explosion dans une maison à qui a causé la mort de deux personnes grièvement blessées le vendredi 10 juillet , la grand-mère mineure de Iasi et son fils de 5 ans ont été par une fuite de gaz dans un à mauvais feu , l' avocat du parquet l' a dit vendredi soir . L' explosion d' une habitation à , qui a fait deux blessés graves vendredi matin , une grand-mère de 52 ans et son petit-fils de 5 ans , est due à une fuite de gaz provenant d' un chauffe-eau défectueux , a indiqué vendredi soir la du procureur du Roi . -Un des habitants de la rivière , les enfants appelés dans le quartier de Le ouvrit la porte portant son fusil de chasse . L' un des riverains du quartier du visité par les enfants a ouvert la porte en tenant son fusil de chasse . -Ou la vendre à quelqu' un d' autre , c' est ce qui s' est passé " , explique Paul Arsenault , titulaire de la chaire de Transat au Tourisme à l' École de gestion de l' UQAM . " Ou soit il le vendait à quelqu' un d' autre et c' est ce qui s' est passé " , explique Paul Arsenault , titulaire de la Chaire de tourisme Transat à l' École des sciences de la gestion de l' UQAM . -principalement pour la qualification des nouveaux pilotes de chasse . Ce , notamment , en vue de qualifier les jeunes pilotes de chasse . -Depuis le bouleversement de la révolution , les Français ont eu deux histoires et deux nations - un , l' autre révolutionnaire . A compter du séisme de la Révolution , les Français ont deux histoires et ils ont deux nations , l' une , l' autre révolutionnaire . -comme cela , qui découle de l' engagement lui-même : pourquoi , dans un pays aussi solidement implanté à l' intérieur de ces frontières , doté d' une structure aussi ancienne et si solidement construite , de ces appels massifs pour l' unité & nbsp ; ? & nbsp ; ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Dont celle-ci , qui naît de l' entreprise elle-même : pourquoi , dans un pays si durablement installé dans ses frontières , si anciennement et solidement , entend-on ces appels dramatiques à l' unité ? -la France à la sortie , la France en train de se faire . La France qui s' en va , la France qui vient -Le White Duke envisageait également de l' album le 04 novembre . Le White Duke a ainsi prévu de cet album , le 4 novembre . -La bataille a été incroyable . Il y a une incroyable . -Quand je suis revenue , il y avait des charges . Je reviens et il y en a plein . diff --git a/lingvo/core/ops/text_packing.cc b/lingvo/core/ops/text_packing.cc deleted file mode 100644 index e51c2505..00000000 --- a/lingvo/core/ops/text_packing.cc +++ /dev/null @@ -1,103 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/text_packing.h" - -#include - -#include "tensorflow/core/platform/logging.h" - -namespace tensorflow { -namespace lingvo { - -TextPacking::TextPacking(int columns, int batch, std::vector times, - int align, bool pack, int spread_first_n, - bool use_last_fit) - : columns_(columns), - batch_(batch), - times_(std::move(times)), - align_(align), - pack_(pack), - spread_first_n_(std::min(spread_first_n, batch)), - use_last_fit_(use_last_fit), - wpos_(batch, std::vector(columns, 0)), - seq_(batch, 0), - last_fit_(0), - counter_(0) { - CHECK_EQ(columns_, times_.size()) << "The size of `times` must be `columns`"; -} - -bool TextPacking::Add(const std::vector& lens, PackingIndex* p) { - CHECK_EQ(columns_, lens.size()); - CHECK(p); - // Start searching for batch position where 'lens' could fit. - // If we find a fit, on next call we start searching from the same position. - // Because if we always start from 0 this loop becomes O(N^2) in batch size. - // TODO(krikun): add a benchmark for very large batch sizes - if (!use_last_fit_) { - last_fit_ = 0; - } - // b is the index of the row on which we see if the current sequence fits. - int b = last_fit_; - for (int i = 0; i < batch_; i++, b++) { - if (counter_ < spread_first_n_) { - b = counter_; - last_fit_ = 0; - } - b %= batch_; - bool fits = true; - for (int c = 0; c < columns_; c++) { - if (wpos_[b][c] + lens[c] > times_[c]) { - fits = false; - break; - } - } - if (fits) { - last_fit_ = b; - p->batch = b; - p->time.resize(columns_); - for (int c = 0; c < columns_; c++) { - p->time[c] = wpos_[b][c]; - wpos_[b][c] += lens[c]; - if (align_ > 1) { - int r = wpos_[b][c] % align_; - if (r) wpos_[b][c] += (align_ - r); - } - if (!pack_) { - wpos_[b][c] = times_[c]; - } - } - seq_[b]++; - p->seq = seq_[b]; - counter_++; - return true; - } - } - return false; -} - -void TextPacking::Reset() { - for (int b = 0; b < batch_; b++) { - seq_[b] = 0; - for (int c = 0; c < columns_; c++) { - wpos_[b][c] = 0; - } - } - last_fit_ = 0; - counter_ = 0; -} - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/text_packing.h b/lingvo/core/ops/text_packing.h deleted file mode 100644 index d4c8c2d1..00000000 --- a/lingvo/core/ops/text_packing.h +++ /dev/null @@ -1,112 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#ifndef THIRD_PARTY_PY_LINGVO_CORE_OPS_TEXT_PACKING_H_ -#define THIRD_PARTY_PY_LINGVO_CORE_OPS_TEXT_PACKING_H_ - -#include - -namespace tensorflow { -namespace lingvo { - -// Text packing. -// -// It takes tuples of variable-length sequences and packs into batches of -// size [batch, time_i] for different time_i per each column. -// -// For example this picture describes packing five 2-tuples of sequences -// into 2-tuple of matrices (batch, time) with batch=2, time=10, and align=2: -// -// column 0 column1 -// batch 0 [ a a a - b b b b b - ] [ A A A A B B B - - -] -// batch 1 [ c c c c c - d - e - ] [ C C - D - E - - - -] -// -// Each tuple is assigned PackedIndex as following: -// (a a a, A A A A) // p.batch=0 p.time=(0, 0) p.seq=1 -// (b b b b b, B B B) // p.batch=0 p.time=(4, 4) p.seq=2 -// (c c c c, C C) // p.batch=1 p.time=(0, 0) p.seq=1 -// (d, D) // p.batch=1 p.time=(6, 4) p.seq=2 -// (e, E) // p.batch=1 p.time=(8, 6) p.seq=3 -class TextPacking { - public: - // columns: number of columns - // batch: batch dimension - // times: time dimension, one element per each column. - // align: align sequence start position modulo n - // pack: set to false to disable packing - // spread_first_n: The first n added sequences will be assigned to the first n - // rows. Note that n is at most `batch`. If a value larger than `batch` is - // provided, `batch` will be used for n instead. - // use_last_fit: Whether to enable an optimization where Add() starts its - // search from the row of the previous successful Add(). When disabled (by - // default), we always start from 0, resulting in O(N^2) in batch size. - TextPacking(int columns, int batch, std::vector times, int align, - bool pack, int spread_first_n, bool use_last_fit = false); - - // Same as above, except that all columns share the same `time` and - // `spread_first_n` is set to 0. - TextPacking(int columns, int batch, int time, int align, bool pack) - : TextPacking(columns, batch, std::vector(columns, time), align, - pack, /*spread_first_n=*/0, /*use_last_fit=*/false) {} - - // Describes the location of a packed item in the batch. - struct PackingIndex { - // The (row) index of this item in the packed batch. - int batch; - - // A vector of size `columns`, one per column, indicating the starting - // positions of this item. - std::vector time; - - // Sequence index: a one-based index indicating how many packed items come - // before this item on the same row. - int seq; - }; - - // Adds a new item with sequence lengths 'lens' to the batch. - // Returns false if the item does not fit. - bool Add(const std::vector& lens, PackingIndex* p); - - // Resets internal state, forgetting any previously packed items. - void Reset(); - - // Current writing position at batch index 'b' column 'c'. - int wpos(int b, int c) const { return wpos_[b][c]; } - - private: - const int columns_; - const int batch_; - const std::vector times_; - const int align_; - const bool pack_; - const int spread_first_n_; - const bool use_last_fit_; - - // Current write position in each column in each row in batch. - std::vector> wpos_; - // Current sequence index in each row in batch. - std::vector seq_; - - // Row index of the last successfully packed segment. - int last_fit_; - - // How many sequences have been added. - int counter_; -}; - -} // namespace lingvo -} // namespace tensorflow - -#endif // THIRD_PARTY_PY_LINGVO_CORE_OPS_TEXT_PACKING_H_ diff --git a/lingvo/core/ops/text_packing_test.cc b/lingvo/core/ops/text_packing_test.cc deleted file mode 100644 index bcdc3bdf..00000000 --- a/lingvo/core/ops/text_packing_test.cc +++ /dev/null @@ -1,282 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/text_packing.h" - -#include - -#include -#include - -namespace tensorflow { -namespace lingvo { -namespace { - -// column 0 column1 -// batch 0 [ a a a - b b b b b - ] [ A A A A B B B - - -] -// batch 1 [ c c c c c - d - e - ] [ C C C - D - E - - -] -TEST(TextPackingTest, TestPacking) { - TextPacking pack(2, 2, 10, 2, true); - std::vector p(5); - EXPECT_TRUE(pack.Add({3, 4}, &p[0])); // (a a a, A A A A) - EXPECT_TRUE(pack.Add({5, 3}, &p[1])); // (b b b b b, B B B) - EXPECT_TRUE(pack.Add({5, 3}, &p[2])); // (c c c c c, C C C) - EXPECT_TRUE(pack.Add({1, 1}, &p[3])); // (d, D) - EXPECT_TRUE(pack.Add({1, 1}, &p[4])); // (e, E) - EXPECT_EQ(p[0].batch, 0); - EXPECT_EQ(p[0].time, std::vector({0, 0})); - EXPECT_EQ(p[0].seq, 1); - EXPECT_EQ(p[1].batch, 0); - EXPECT_EQ(p[1].time, std::vector({4, 4})); - EXPECT_EQ(p[1].seq, 2); - EXPECT_EQ(p[2].batch, 1); - EXPECT_EQ(p[2].time, std::vector({0, 0})); - EXPECT_EQ(p[2].seq, 1); - EXPECT_EQ(p[3].batch, 1); - EXPECT_EQ(p[3].time, std::vector({6, 4})); - EXPECT_EQ(p[3].seq, 2); - EXPECT_EQ(p[4].batch, 1); - EXPECT_EQ(p[4].time, std::vector({8, 6})); - EXPECT_EQ(p[4].seq, 3); -} - -TEST(TextPackingTest, TestReset) { - TextPacking pack(2, 2, 10, 2, true); - std::vector p(5); - EXPECT_TRUE(pack.Add({3, 4}, &p[0])); // (a a a, A A A A) - EXPECT_TRUE(pack.Add({5, 3}, &p[1])); // (b b b b b, B B B) - EXPECT_TRUE(pack.Add({5, 3}, &p[2])); // (c c c c c, C C C) - pack.Reset(); - EXPECT_TRUE(pack.Add({1, 1}, &p[3])); // (d, D) - EXPECT_TRUE(pack.Add({1, 1}, &p[4])); // (e, E) - EXPECT_EQ(p[0].batch, 0); - EXPECT_EQ(p[0].time, std::vector({0, 0})); - EXPECT_EQ(p[0].seq, 1); - EXPECT_EQ(p[1].batch, 0); - EXPECT_EQ(p[1].time, std::vector({4, 4})); - EXPECT_EQ(p[1].seq, 2); - EXPECT_EQ(p[2].batch, 1); - EXPECT_EQ(p[2].time, std::vector({0, 0})); - EXPECT_EQ(p[2].seq, 1); - // after reset - EXPECT_EQ(p[3].batch, 0); - EXPECT_EQ(p[3].time, std::vector({0, 0})); - EXPECT_EQ(p[3].seq, 1); - EXPECT_EQ(p[4].batch, 0); - EXPECT_EQ(p[4].time, std::vector({2, 2})); - EXPECT_EQ(p[4].seq, 2); -} -TEST(TextPackingTest, TestDoesNotFit) { - TextPacking pack(2, 2, 10, 2, true); - TextPacking::PackingIndex p; - - EXPECT_FALSE(pack.Add({11, 1}, &p)); - EXPECT_FALSE(pack.Add({1, 11}, &p)); - - EXPECT_TRUE(pack.Add({3, 4}, &p)); - - EXPECT_TRUE(pack.Add({5, 3}, &p)); - - EXPECT_TRUE(pack.Add({5, 3}, &p)); - EXPECT_FALSE(pack.Add({5, 1}, &p)); - EXPECT_FALSE(pack.Add({1, 7}, &p)); - - EXPECT_TRUE(pack.Add({1, 1}, &p)); - EXPECT_FALSE(pack.Add({3, 1}, &p)); - EXPECT_FALSE(pack.Add({1, 5}, &p)); - - EXPECT_TRUE(pack.Add({1, 1}, &p)); - EXPECT_FALSE(pack.Add({3, 1}, &p)); - - EXPECT_FALSE(pack.Add({1, 1}, &p)); -} - -// column 0 column1 -// batch 0 [ a a a b b b b b d e ] [ A A A A B B B D E -] -// batch 1 [ c c c c c - - - - - ] [ C C C - - - - - - -] -TEST(TextPackingTest, TestDoNotAlign) { - TextPacking pack(2, 2, 10, 1, true); - std::vector p(5); - EXPECT_TRUE(pack.Add({3, 4}, &p[0])); // (a a a, A A A A) - EXPECT_TRUE(pack.Add({5, 3}, &p[1])); // (b b b b b, B B B) - EXPECT_TRUE(pack.Add({5, 3}, &p[2])); // (c c c c c, C C C) - EXPECT_TRUE(pack.Add({1, 1}, &p[3])); // (d, D) - EXPECT_TRUE(pack.Add({1, 1}, &p[4])); // (e, E) - EXPECT_EQ(p[0].batch, 0); - EXPECT_EQ(p[0].time, std::vector({0, 0})); - EXPECT_EQ(p[0].seq, 1); - EXPECT_EQ(p[1].batch, 0); - EXPECT_EQ(p[1].time, std::vector({3, 4})); - EXPECT_EQ(p[1].seq, 2); - EXPECT_EQ(p[2].batch, 1); - EXPECT_EQ(p[2].time, std::vector({0, 0})); - EXPECT_EQ(p[2].seq, 1); - EXPECT_EQ(p[3].batch, 0); - EXPECT_EQ(p[3].time, std::vector({8, 7})); - EXPECT_EQ(p[3].seq, 3); - EXPECT_EQ(p[4].batch, 0); - EXPECT_EQ(p[4].time, std::vector({9, 8})); - EXPECT_EQ(p[4].seq, 4); -} - -TEST(TextPackingTest, TestDoNotPack) { - TextPacking pack(2, 2, 10, 0, false); - std::vector p(5); - EXPECT_TRUE(pack.Add({3, 4}, &p[0])); // (a a a, A A A A) - EXPECT_TRUE(pack.Add({5, 3}, &p[1])); // (b b b b b, B B B) - EXPECT_FALSE(pack.Add({5, 3}, &p[2])); // (c c c c c, C C C) - EXPECT_FALSE(pack.Add({1, 1}, &p[3])); // (d, D) - EXPECT_FALSE(pack.Add({1, 1}, &p[4])); // (e, E) - EXPECT_EQ(p[0].batch, 0); - EXPECT_EQ(p[0].time, std::vector({0, 0})); - EXPECT_EQ(p[0].seq, 1); - EXPECT_EQ(p[1].batch, 1); - EXPECT_EQ(p[1].time, std::vector({0, 0})); - EXPECT_EQ(p[1].seq, 1); -} - -// column 0 column1 -// batch 0 [ a a a - ] [ A A A A - - ] -// batch 1 [ b b c - ] [ B C C C C C ] -TEST(TextPackingTest, TestDifferentTimesPerColumn) { - TextPacking pack(2, 2, {4, 6}, 1, true, 0); - std::vector p(5); - EXPECT_TRUE(pack.Add({3, 4}, &p[0])); // (a a a, A A A A) - EXPECT_TRUE(pack.Add({2, 1}, &p[1])); // (b b, B) - EXPECT_TRUE(pack.Add({1, 5}, &p[2])); // (c, C C C C C) - EXPECT_EQ(p[0].batch, 0); - EXPECT_EQ(p[0].time, std::vector({0, 0})); - EXPECT_EQ(p[0].seq, 1); - EXPECT_EQ(p[1].batch, 1); - EXPECT_EQ(p[1].time, std::vector({0, 0})); - EXPECT_EQ(p[1].seq, 1); - EXPECT_EQ(p[2].batch, 1); - EXPECT_EQ(p[2].time, std::vector({2, 1})); - EXPECT_EQ(p[2].seq, 2); - EXPECT_FALSE(pack.Add({2, 1}, &p[3])); - EXPECT_FALSE(pack.Add({1, 3}, &p[4])); -} - -// column 0 column1 -// batch 0 [ a a d - ] [ A A A A D - ] -// batch 1 [ b b - - ] [ B - - - - - ] -// batch 2 [ c - - - ] [ C C C C - - ] -TEST(TextPackingTest, TestSpreadFirstN) { - // First 3 sequences will be assigned different rows. - TextPacking pack(2, 3, {4, 6}, 1, true, 3); - std::vector p(4); - TextPacking::PackingIndex unused; - EXPECT_TRUE(pack.Add({2, 4}, &p[0])); // (a a a, A A A A) - EXPECT_FALSE(pack.Add({5, 1}, &unused)); - EXPECT_TRUE(pack.Add({2, 1}, &p[1])); // (b b, B) - EXPECT_FALSE(pack.Add({1, 7}, &unused)); - EXPECT_TRUE(pack.Add({1, 4}, &p[2])); // (c, C C C C C) - EXPECT_TRUE(pack.Add({1, 1}, &p[3])); // (d, D) - EXPECT_EQ(p[0].batch, 0); - EXPECT_EQ(p[0].time, std::vector({0, 0})); - EXPECT_EQ(p[0].seq, 1); - EXPECT_EQ(p[1].batch, 1); - EXPECT_EQ(p[1].time, std::vector({0, 0})); - EXPECT_EQ(p[1].seq, 1); - EXPECT_EQ(p[2].batch, 2); - EXPECT_EQ(p[2].time, std::vector({0, 0})); - EXPECT_EQ(p[2].seq, 1); - EXPECT_EQ(p[3].batch, 0); - EXPECT_EQ(p[3].time, std::vector({2, 4})); - EXPECT_EQ(p[3].seq, 2); -} - -// column 0 column1 -// batch 0 [ a a d - ] [ A A A A D - ] -// batch 1 [ b b c - ] [ B C C C C - ] -// batch 2 [ - - - - ] [ - - - - - - ] -TEST(TextPackingTest, TestSpreadFirstNSmall) { - // First 2 sequences will be assigned different rows. - TextPacking pack(2, 3, {4, 6}, 1, true, 2); - std::vector p(4); - EXPECT_TRUE(pack.Add({2, 4}, &p[0])); // (a a, A A A A) - EXPECT_TRUE(pack.Add({2, 1}, &p[1])); // (b b, B) - EXPECT_TRUE(pack.Add({1, 4}, &p[2])); // (c, C C C C C) - EXPECT_TRUE(pack.Add({1, 1}, &p[3])); // (d, D) - EXPECT_EQ(p[0].batch, 0); - EXPECT_EQ(p[0].time, std::vector({0, 0})); - EXPECT_EQ(p[0].seq, 1); - EXPECT_EQ(p[1].batch, 1); - EXPECT_EQ(p[1].time, std::vector({0, 0})); - EXPECT_EQ(p[1].seq, 1); - EXPECT_EQ(p[2].batch, 1); - EXPECT_EQ(p[2].time, std::vector({2, 1})); - EXPECT_EQ(p[2].seq, 2); - EXPECT_EQ(p[3].batch, 0); - EXPECT_EQ(p[3].time, std::vector({2, 4})); - EXPECT_EQ(p[3].seq, 2); -} - -// With use_last_fit set: -// column 0 column1 -// batch 0 [ a d d d ] [ A A A A A D ] -// batch 1 [ b b c - ] [ B B C - - - ] -// -// With use_last_fit unset: -// column 0 column1 -// batch 0 [ a c - - ] [ A A A A A C ] -// batch 1 [ b b - - ] [ B B - - - - ] -TEST(TextPackingTest, TestUseLastFit) { - { - TextPacking pack(2, 2, {4, 6}, 1, true, 0, /*use_last_fit=*/true); - std::vector p(5); - EXPECT_TRUE(pack.Add({1, 5}, &p[0])); // (a, A A A A A) - EXPECT_TRUE(pack.Add({2, 2}, &p[1])); // (b b, B B) - // starts searching in batch 1. - EXPECT_TRUE(pack.Add({1, 1}, &p[2])); // (c, C) - // wraps around to search in batch 0. - EXPECT_TRUE(pack.Add({3, 1}, &p[3])); // (d d d, D) - EXPECT_FALSE(pack.Add({3, 1}, &p[4])); // (e e e, E) - EXPECT_EQ(p[0].batch, 0); - EXPECT_EQ(p[0].time, std::vector({0, 0})); - EXPECT_EQ(p[0].seq, 1); - EXPECT_EQ(p[1].batch, 1); - EXPECT_EQ(p[1].time, std::vector({0, 0})); - EXPECT_EQ(p[1].seq, 1); - EXPECT_EQ(p[2].batch, 1); - EXPECT_EQ(p[2].time, std::vector({2, 2})); - EXPECT_EQ(p[2].seq, 2); - EXPECT_EQ(p[3].batch, 0); - EXPECT_EQ(p[3].time, std::vector({1, 5})); - EXPECT_EQ(p[3].seq, 2); - } - { - TextPacking pack(2, 2, {4, 6}, 1, true, 0, /*use_last_fit=*/false); - std::vector p(4); - EXPECT_TRUE(pack.Add({1, 5}, &p[0])); // (a, A A A A A) - EXPECT_TRUE(pack.Add({2, 2}, &p[1])); // (b b, B B) - // starts search in batch 0. - EXPECT_TRUE(pack.Add({1, 1}, &p[2])); // (c, C) - EXPECT_FALSE(pack.Add({3, 1}, &p[3])); // (d d d, D) - EXPECT_EQ(p[0].batch, 0); - EXPECT_EQ(p[0].time, std::vector({0, 0})); - EXPECT_EQ(p[0].seq, 1); - EXPECT_EQ(p[1].batch, 1); - EXPECT_EQ(p[1].time, std::vector({0, 0})); - EXPECT_EQ(p[1].seq, 1); - EXPECT_EQ(p[2].batch, 0); - EXPECT_EQ(p[2].time, std::vector({1, 5})); - EXPECT_EQ(p[2].seq, 2); - } -} - -} // namespace -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/tokenizer_op_headers.h b/lingvo/core/ops/tokenizer_op_headers.h deleted file mode 100644 index c629104c..00000000 --- a/lingvo/core/ops/tokenizer_op_headers.h +++ /dev/null @@ -1,151 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#ifndef LINGVO_CORE_OPS_TOKENIZER_OP_HEADERS_H_ -#define LINGVO_CORE_OPS_TOKENIZER_OP_HEADERS_H_ - -#include -#include -#include -#include - -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" -#include "tensorflow/core/lib/strings/str_util.h" -#include "tensorflow/core/platform/env.h" - -namespace tensorflow { -namespace lingvo { -namespace { - -template -class LabelToTokenIdOp : public OpKernel { - public: - explicit LabelToTokenIdOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, ctx->GetAttr("append_eos", &append_eos_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("maxlen", &maxlen_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("pad_to_maxlen", &pad_to_maxlen_)); - } - - void Compute(OpKernelContext* ctx) override { - const Tensor& labels = ctx->input(0); - OP_REQUIRES(ctx, TensorShapeUtils::IsVector(labels.shape()), - errors::InvalidArgument("labels must be a vector, but get ", - labels.shape().DebugString())); - const int batch = labels.NumElements(); - auto Tlabels = labels.flat(); - Tensor token_ids(DT_INT32, TensorShape({batch, maxlen_})); - auto Ttoken_ids = token_ids.matrix(); - Ttoken_ids.setZero(); // Sanity - Tensor target_ids(DT_INT32, TensorShape({batch, maxlen_})); - auto Ttarget_ids = target_ids.matrix(); - Ttarget_ids.setZero(); // Sanity - Tensor paddings(DT_FLOAT, TensorShape({batch, maxlen_})); - auto Tpaddings = paddings.matrix(); - Tpaddings.setZero(); // Sanity - int actual_maxlen = pad_to_maxlen_ ? maxlen_ : 0; - for (int i = 0; i < batch; ++i) { - VLOG(1) << i << " " << Tlabels(i); - std::vector ids = TokenizerClass::StringToIds(Tlabels(i)); - if (ids.size() + 1 > maxlen_) { - LOG(WARNING) << "Too long target " << ids.size() << " " << Tlabels(i); - ids.resize(maxlen_ - 1); - } - const int id_size = ids.size(); - const int32 kSOS = 1; - const int32 kEOS = 2; - Ttoken_ids(i, 0) = kSOS; - for (int j = 0; j < id_size; ++j) { - Ttoken_ids(i, j + 1) = ids[j]; - Ttarget_ids(i, j) = ids[j]; - Tpaddings(i, j) = 0.0; // padding = false - } - Ttarget_ids(i, id_size) = kEOS; - Tpaddings(i, id_size) = append_eos_ ? 0.0 : 1.0; - actual_maxlen = std::max(actual_maxlen, id_size + 1); - for (int j = id_size + 1; j < maxlen_; ++j) { - Ttoken_ids(i, j) = kEOS; - Ttarget_ids(i, j) = kEOS; - Tpaddings(i, j) = 1.0; // padding = true - } - } - - Tensor out_token_ids(DT_INT32, TensorShape({batch, actual_maxlen})); - Tensor out_target_ids(DT_INT32, TensorShape({batch, actual_maxlen})); - Tensor out_paddings(DT_FLOAT, TensorShape({batch, actual_maxlen})); - - typedef const Eigen::DSizes DSize2; - out_token_ids.matrix() = - Ttoken_ids.slice(DSize2{0, 0}, DSize2{batch, actual_maxlen}); - out_target_ids.matrix() = - Ttarget_ids.slice(DSize2{0, 0}, DSize2{batch, actual_maxlen}); - out_paddings.matrix() = - Tpaddings.slice(DSize2{0, 0}, DSize2{batch, actual_maxlen}); - - OP_REQUIRES_OK(ctx, ctx->set_output("token_ids", out_token_ids)); - OP_REQUIRES_OK(ctx, ctx->set_output("target_ids", out_target_ids)); - OP_REQUIRES_OK(ctx, ctx->set_output("paddings", out_paddings)); - } - - private: - bool append_eos_ = true; - int maxlen_ = 0; - bool pad_to_maxlen_ = true; -}; - -template -class IdToTokenOp : public OpKernel { - public: - explicit IdToTokenOp(OpKernelConstruction* ctx) : OpKernel(ctx) {} - - void Compute(OpKernelContext* ctx) override { - const Tensor& ids = ctx->input(0); - OP_REQUIRES(ctx, TensorShapeUtils::IsMatrix(ids.shape()), - errors::InvalidArgument("token_ids must be a matrix, but get ", - ids.shape().DebugString())); - const Tensor& seq_lens = ctx->input(1); - OP_REQUIRES(ctx, TensorShapeUtils::IsVector(seq_lens.shape()), - errors::InvalidArgument("seq_lens must be a vector, but get ", - seq_lens.shape().DebugString())); - const int batch = seq_lens.NumElements(); - OP_REQUIRES(ctx, batch == ids.dim_size(0), - errors::InvalidArgument( - "batch size has to match between token_ids and seq_lens. ", - ids.shape().DebugString(), " vs. ", - seq_lens.shape().DebugString())); - - Tensor* out; - OP_REQUIRES_OK(ctx, ctx->allocate_output(0, TensorShape({batch}), &out)); - const auto& t_ids = ids.matrix(); - const auto& t_seq_lens = seq_lens.vec(); - auto t_out = out->template vec(); - for (int i = 0; i < batch; ++i) { - const int len_i = std::max(0, t_seq_lens(i)); - std::vector ids_i(len_i); - for (int j = 0; j < len_i; ++j) { - ids_i[j] = t_ids(i, j); - } - std::vector labels = TokenizerClass::IdToStrings(ids_i); - t_out(i) = TokenizerClass::JoinLabels(labels); - } - } -}; - -} // namespace -} // namespace lingvo -} // namespace tensorflow - -#endif // LINGVO_CORE_OPS_TOKENIZER_OP_HEADERS_H_ diff --git a/lingvo/core/ops/tokenizer_ops_kernels.cc b/lingvo/core/ops/tokenizer_ops_kernels.cc deleted file mode 100644 index 18d49f80..00000000 --- a/lingvo/core/ops/tokenizer_ops_kernels.cc +++ /dev/null @@ -1,374 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ -#include -#include -#include -#include - -#include "lingvo/core/ops/ascii_tokenizer.h" -#include "lingvo/core/ops/simple_vocab.h" -#include "lingvo/core/ops/tokenizer_op_headers.h" -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" -#include "tensorflow/core/lib/strings/numbers.h" -#include "tensorflow/core/lib/strings/str_util.h" -#include "tensorflow/core/platform/env.h" - -namespace tensorflow { -namespace lingvo { -namespace { - -REGISTER_KERNEL_BUILDER(Name("AsciiToTokenId").Device(DEVICE_CPU), - LabelToTokenIdOp); - -REGISTER_KERNEL_BUILDER(Name("IdToAscii").Device(DEVICE_CPU), - IdToTokenOp); - -class StrToVocabTokensOp : public OpKernel { - public: - explicit StrToVocabTokensOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, ctx->GetAttr("append_eos", &append_eos_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("maxlen", &maxlen_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("pad_to_maxlen", &pad_to_maxlen_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("vocab_filepath", &vocab_filepath_)); - bool load_token_ids_from_vocab; - OP_REQUIRES_OK(ctx, ctx->GetAttr("load_token_ids_from_vocab", - &load_token_ids_from_vocab)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("delimiter", &delimiter_)); - CHECK_GT(maxlen_, 0); - OP_REQUIRES_OK(ctx, - vocab_.Load(vocab_filepath_, load_token_ids_from_vocab)); - } - - void Compute(OpKernelContext* ctx) override { - const Tensor* labels; - OP_REQUIRES_OK(ctx, ctx->input("labels", &labels)); - const auto& t_label = labels->vec(); - const int32 b_size = labels->dim_size(0); - Tensor token_ids(DT_INT32, TensorShape({b_size, maxlen_})); - Tensor target_ids(DT_INT32, TensorShape({b_size, maxlen_})); - Tensor paddings(DT_FLOAT, TensorShape({b_size, maxlen_})); - - auto t_token_ids = token_ids.tensor(); - auto t_target_ids = target_ids.tensor(); - auto t_paddings = paddings.tensor(); - t_token_ids.setZero(); - t_target_ids.setZero(); - t_paddings.setZero(); - - int actual_maxlen = pad_to_maxlen_ ? maxlen_ : 0; - for (int i = 0; i < b_size; ++i) { - t_token_ids(i, 0) = vocab_.sos_id(); - - string label(t_label(i)); - VLOG(1) << "Label " << label; - std::vector tokens; - if (delimiter_.length() > 0) { - tokens = str_util::Split(label, delimiter_, str_util::SkipWhitespace()); - } else { - // Split by the empty delimiter. - for (int i = 0; i < label.size(); ++i) { - tokens.push_back(string(1, label[i])); - } - } - - VLOG(1) << "#Tokens " << tokens.size() << " " - << str_util::Join(tokens, "/"); - int cur_char = 0; - for (const auto& token : tokens) { - const int token_id = vocab_.TokenToId(token); - t_target_ids(i, cur_char) = token_id; - t_paddings(i, cur_char) = 0.0; - // If the number of tokens is longer than the max length - truncate. - if (cur_char + 1 >= maxlen_) { - cur_char++; - LOG(INFO) << "Label: \"" << label << "\" contained " << tokens.size() - << " tokens, and was truncated to size: " << maxlen_ << " (" - << tokens.size() - maxlen_ << " tokens were ignored)."; - break; - } - t_token_ids(i, cur_char + 1) = token_id; - cur_char++; - } - if (cur_char < maxlen_) { - // There was no truncation, t_token_ids is ahead by 1 over t_target_ids - // and t_paddings - t_target_ids(i, cur_char) = vocab_.eos_id(); - t_paddings(i, cur_char) = append_eos_ ? 0.0 : 1.0; - ++cur_char; - } - actual_maxlen = std::max(actual_maxlen, cur_char); - for (; cur_char < maxlen_; ++cur_char) { - t_token_ids(i, cur_char) = vocab_.eos_id(); - t_target_ids(i, cur_char) = vocab_.eos_id(); - t_paddings(i, cur_char) = 1.0; - } - } - - Tensor out_token_ids(DT_INT32, TensorShape({b_size, actual_maxlen})); - Tensor out_target_ids(DT_INT32, TensorShape({b_size, actual_maxlen})); - Tensor out_paddings(DT_FLOAT, TensorShape({b_size, actual_maxlen})); - - typedef const Eigen::DSizes DSize2; - out_token_ids.matrix() = - t_token_ids.slice(DSize2{0, 0}, DSize2{b_size, actual_maxlen}); - out_target_ids.matrix() = - t_target_ids.slice(DSize2{0, 0}, DSize2{b_size, actual_maxlen}); - out_paddings.matrix() = - t_paddings.slice(DSize2{0, 0}, DSize2{b_size, actual_maxlen}); - - OP_REQUIRES_OK(ctx, ctx->set_output("token_ids", out_token_ids)); - OP_REQUIRES_OK(ctx, ctx->set_output("target_ids", out_target_ids)); - OP_REQUIRES_OK(ctx, ctx->set_output("paddings", out_paddings)); - } - - private: - string vocab_filepath_; - bool append_eos_ = true; - int maxlen_ = 0; - bool pad_to_maxlen_ = true; - string delimiter_; - Vocab vocab_; -}; - -REGISTER_KERNEL_BUILDER(Name("StrToVocabTokens").Device(DEVICE_CPU), - StrToVocabTokensOp); - -class NgramIdToTokenOp : public OpKernel { - public: - explicit NgramIdToTokenOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, ctx->GetAttr("ngram_vocab_filepath", &vocab_filepath_)); - OP_REQUIRES_OK(ctx, vocab_.Load(vocab_filepath_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("ngram_separator", &ngram_separator_)); - } - - void Compute(OpKernelContext* ctx) override { - const Tensor* token_ids; - const Tensor* seq_lengths; - OP_REQUIRES_OK(ctx, ctx->input("token_ids", &token_ids)); - OP_REQUIRES_OK(ctx, ctx->input("seq_lengths", &seq_lengths)); - OP_REQUIRES(ctx, TensorShapeUtils::IsMatrix(token_ids->shape()), - errors::InvalidArgument("token_ids must be a matrix, but get ", - token_ids->shape().DebugString())); - OP_REQUIRES( - ctx, TensorShapeUtils::IsVector(seq_lengths->shape()), - errors::InvalidArgument("seq_lengths must be a vector, but get ", - seq_lengths->shape().DebugString())); - const int batch = seq_lengths->NumElements(); - OP_REQUIRES( - ctx, batch == token_ids->dim_size(0), - errors::InvalidArgument("batch size has to match between token_ids and " - "seq_lengths")); - - Tensor* out; - OP_REQUIRES_OK(ctx, ctx->allocate_output(0, TensorShape({batch}), &out)); - const auto& t_ids = token_ids->matrix(); - const auto& t_seq_lens = seq_lengths->vec(); - auto t_out = out->template vec(); - for (int i = 0; i < batch; ++i) { - const int len_i = std::max(0, t_seq_lens(i)); - std::vector ids_i(len_i); - for (int j = 0; j < len_i; ++j) { - ids_i[j] = t_ids(i, j); - } - auto labels = vocab_.IdsToTokens(ids_i); - t_out(i) = str_util::Join(labels, ngram_separator_.c_str()); - } - } - - private: - string vocab_filepath_; - Vocab vocab_; - string ngram_separator_; -}; - -REGISTER_KERNEL_BUILDER(Name("NgramIdToToken").Device(DEVICE_CPU), - NgramIdToTokenOp); - -class BpeIdsToWordsOp : public OpKernel { - public: - explicit BpeIdsToWordsOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, ctx->GetAttr("vocab_filepath", &vocab_filepath_)); - string contents; - OP_REQUIRES_OK(ctx, ReadFileToString( - Env::Default(), vocab_filepath_, &contents)); - std::vector lines = str_util::Split(contents, '\n', - str_util::SkipEmpty()); - for (const string& line : lines) { - std::vector parts = str_util::Split(line, ' '); - id_to_string_map_.push_back(parts[0]); - } - } - - void Compute(OpKernelContext* ctx) override { - const Tensor* token_ids; - const Tensor* seq_lengths; - OP_REQUIRES_OK(ctx, ctx->input("token_ids", &token_ids)); - OP_REQUIRES_OK(ctx, ctx->input("seq_lengths", &seq_lengths)); - OP_REQUIRES(ctx, TensorShapeUtils::IsMatrix(token_ids->shape()), - errors::InvalidArgument("token_ids must be a matrix, but get ", - token_ids->shape().DebugString())); - OP_REQUIRES( - ctx, TensorShapeUtils::IsVector(seq_lengths->shape()), - errors::InvalidArgument("seq_lengths must be a vector, but get ", - seq_lengths->shape().DebugString())); - const int batch = seq_lengths->NumElements(); - OP_REQUIRES( - ctx, batch == token_ids->dim_size(0), - errors::InvalidArgument("batch size has to match between token_ids and " - "seq_lengths")); - - Tensor* out; - OP_REQUIRES_OK(ctx, ctx->allocate_output(0, TensorShape({batch}), &out)); - const auto& t_ids = token_ids->matrix(); - const auto& t_seq_lens = seq_lengths->vec(); - auto t_out = out->template vec(); - for (int i = 0; i < batch; ++i) { - const int len_i = std::max(0, t_seq_lens(i)); - std::vector labels; - for (int j = 0; j < len_i; ++j) { - string label = id_to_string_map_[t_ids(i, j)]; - std::size_t pos = label.find("@@"); - if (pos == std::string::npos) - label = label + " "; - else - label.erase(pos, 2); - labels.push_back(label); - } - t_out(i) = str_util::Join(labels, ""); - } - } - - private: - string vocab_filepath_; - std::vector id_to_string_map_; -}; - -REGISTER_KERNEL_BUILDER(Name("BpeIdsToWords").Device(DEVICE_CPU), - BpeIdsToWordsOp); - -class BpeWordsToIdsOp : public OpKernel { - public: - explicit BpeWordsToIdsOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, ctx->GetAttr("append_eos", &append_eos_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("maxlen", &maxlen_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("tokenization_filepath", - &tokenization_filepath_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("sos_id", &sos_id_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("eos_id", &eos_id_)); - CHECK_GT(maxlen_, 0); - string contents; - OP_REQUIRES_OK(ctx, ReadFileToString(Env::Default(), tokenization_filepath_, - &contents)); - std::vector lines = str_util::Split(contents, '\n', - str_util::SkipEmpty()); - for (const string& line : lines) { - // Each line: - // string int1,int2,int3,...,intn - std::vector parts = str_util::Split(line, ' '); - std::vector split_parts_1 = str_util::Split(parts[1], ','); - std::vector ids; - for (const string& str_id : split_parts_1) { - int32 id; - strings::safe_strto32(str_id, &id); - ids.push_back(id); - } - string_to_ids_map_[parts[0]] = ids; - } - } - - void Compute(OpKernelContext* ctx) override { - const Tensor* labels; - OP_REQUIRES_OK(ctx, ctx->input("labels", &labels)); - const auto& t_label = labels->vec(); - const int32 b_size = labels->dim_size(0); - Tensor* token_ids = nullptr; - Tensor* target_ids = nullptr; - Tensor* paddings = nullptr; - OP_REQUIRES_OK( - ctx, ctx->allocate_output("token_ids", TensorShape({b_size, maxlen_}), - &token_ids)); - OP_REQUIRES_OK( - ctx, ctx->allocate_output("target_ids", TensorShape({b_size, maxlen_}), - &target_ids)); - OP_REQUIRES_OK( - ctx, ctx->allocate_output("paddings", TensorShape({b_size, maxlen_}), - &paddings)); - - token_ids->flat().setConstant(eos_id_); - target_ids->flat().setConstant(eos_id_); - paddings->flat().setConstant(1.0f); - auto t_token_ids = token_ids->tensor(); - auto t_target_ids = target_ids->tensor(); - auto t_paddings = paddings->tensor(); - - for (int i = 0; i < b_size; ++i) { - t_token_ids(i, 0) = sos_id_; - - string label(t_label(i)); - VLOG(1) << "Label " << label; - std::vector tokens = - str_util::Split(label, ' ', str_util::SkipWhitespace()); - VLOG(1) << "#Tokens " << tokens.size() << " " - << str_util::Join(tokens, "/"); - int cur_char = 0; - for (const auto& token : tokens) { - if (cur_char >= maxlen_) { - break; - } - const std::vector token_ids = string_to_ids_map_[token]; - for (const auto& token_id : token_ids) { - t_target_ids(i, cur_char) = token_id; - t_paddings(i, cur_char) = 0.0f; - // If the number of tokens is longer than the max length - truncate. - if (cur_char + 1 >= maxlen_) { - cur_char++; - int num_token_ids = 0; - for (const auto& t : tokens) { - num_token_ids += string_to_ids_map_[t].size(); - } - LOG(INFO) << "Label: \"" << label << "\" had " << num_token_ids - << " tokens, and was truncated to size: " << maxlen_ - << " (" << num_token_ids - maxlen_ << " tokens ignored)."; - break; - } - t_token_ids(i, cur_char + 1) = token_id; - cur_char++; - } - } - if (cur_char < maxlen_) { - // There was no truncation, t_token_ids is ahead by 1 over t_target_ids - // and t_paddings - t_target_ids(i, cur_char) = eos_id_; - t_paddings(i, cur_char) = append_eos_ ? 0.0f : 1.0f; - } - } - } - - private: - string tokenization_filepath_; - bool append_eos_ = true; - int maxlen_ = 0; - int sos_id_ = 1; - int eos_id_ = 2; - std::unordered_map > string_to_ids_map_; -}; - -REGISTER_KERNEL_BUILDER(Name("BpeWordsToIds").Device(DEVICE_CPU), - BpeWordsToIdsOp); -} // namespace -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/tokenizer_ops_test.py b/lingvo/core/ops/tokenizer_ops_test.py deleted file mode 100644 index 1cc64f31..00000000 --- a/lingvo/core/ops/tokenizer_ops_test.py +++ /dev/null @@ -1,336 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for tokenizer_ops.""" - -from lingvo import compat as tf -from lingvo.core import ops -from lingvo.core import test_helper -from lingvo.core import test_utils -import six - - -class TokenizerOpsTest(test_utils.TestCase): - - def testLabelsToTokenId(self): - with self.session(use_gpu=False): - token_ids, target_ids, paddings = self.evaluate( - ops.ascii_to_token_id([ - 'hElLo', 'sIr', 'What a day', 'america\'s', - ' early', '1:00 AM', 'morning' - ], - append_eos=True, - maxlen=10)) - self.assertAllEqual(token_ids, [ - [1, 12, 9, 16, 16, 19, 2, 2, 2, 2], - [1, 23, 13, 22, 73, 2, 2, 2, 2, 2], - [1, 27, 12, 5, 24, 3, 5, 3, 0, 3], - [1, 5, 17, 9, 22, 13, 7, 5, 32, 23], - [1, 4, 3, 9, 5, 22, 16, 29, 2, 2], - [1, 40, 34, 39, 39, 3, 5, 17, 2, 2], - [1, 74, 17, 19, 22, 18, 13, 18, 11, 2], - ]) - self.assertAllEqual( - target_ids, - [[12, 9, 16, 16, 19, 2, 2, 2, 2, 2], [23, 13, 22, 73, 2, 2, 2, 2, 2, 2], - [27, 12, 5, 24, 3, 5, 3, 0, 3, 2], [5, 17, 9, 22, 13, 7, 5, 32, 23, 2], - [4, 3, 9, 5, 22, 16, 29, 2, 2, 2], [40, 34, 39, 39, 3, 5, 17, 2, 2, 2], - [74, 17, 19, 22, 18, 13, 18, 11, 2, 2]]) - self.assertAllEqual( - paddings, - [[0, 0, 0, 0, 0, 0, 1, 1, 1, 1], [0, 0, 0, 0, 0, 1, 1, 1, 1, 1], [ - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - ], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 1, 1], - [0, 0, 0, 0, 0, 0, 0, 0, 1, 1], [0, 0, 0, 0, 0, 0, 0, 0, 0, 1]]) - - def testLabelsToTokenIdAppendEOSFalse(self): - with self.session(use_gpu=False): - token_ids, target_ids, paddings = self.evaluate( - ops.ascii_to_token_id([ - 'hElLo', 'sIr', 'What a day', 'america\'s', - ' early', '1:00 AM', '100%' - ], - append_eos=False, - maxlen=10)) - self.assertAllEqual( - token_ids, - [[1, 12, 9, 16, 16, 19, 2, 2, 2, 2], [1, 23, 13, 22, 73, 2, 2, 2, 2, 2], - [1, 27, 12, 5, 24, 3, 5, 3, 0, 3], [1, 5, 17, 9, 22, 13, 7, 5, 32, 23], - [1, 4, 3, 9, 5, 22, 16, 29, 2, 2], [1, 40, 34, 39, 39, 3, 5, 17, 2, 2], - [1, 40, 39, 39, 52, 2, 2, 2, 2, 2]]) - self.assertAllEqual( - target_ids, - [[12, 9, 16, 16, 19, 2, 2, 2, 2, 2], [23, 13, 22, 73, 2, 2, 2, 2, 2, 2], - [27, 12, 5, 24, 3, 5, 3, 0, 3, 2], [5, 17, 9, 22, 13, 7, 5, 32, 23, 2], - [4, 3, 9, 5, 22, 16, 29, 2, 2, 2], [40, 34, 39, 39, 3, 5, 17, 2, 2, 2], - [40, 39, 39, 52, 2, 2, 2, 2, 2, 2]]) - self.assertAllEqual( - paddings, - [[0, 0, 0, 0, 0, 1, 1, 1, 1, 1], [0, 0, 0, 0, 1, 1, 1, 1, 1, 1], [ - 0, 0, 0, 0, 0, 0, 0, 0, 0, 1 - ], [0, 0, 0, 0, 0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 0, 0, 1, 1, 1], - [0, 0, 0, 0, 0, 0, 0, 1, 1, 1], [0, 0, 0, 0, 1, 1, 1, 1, 1, 1]]) - - def testLabelsToTokenIdNoPadToMaxlen(self): - with self.session(use_gpu=False): - token_ids, target_ids, paddings = self.evaluate( - ops.ascii_to_token_id([ - 'hElLo', 'sIr', 'What a day', 'america\'s', - ' early', '1:00 AM', 'morning' - ], - append_eos=True, - maxlen=20, - pad_to_maxlen=False)) - self.assertAllEqual(token_ids, [ - [1, 12, 9, 16, 16, 19, 2, 2, 2, 2, 2, 2, 2], - [1, 23, 13, 22, 73, 2, 2, 2, 2, 2, 2, 2, 2], - [1, 27, 12, 5, 24, 3, 5, 3, 0, 3, 8, 5, 29], - [1, 5, 17, 9, 22, 13, 7, 5, 32, 23, 2, 2, 2], - [1, 4, 3, 9, 5, 22, 16, 29, 2, 2, 2, 2, 2], - [1, 40, 34, 39, 39, 3, 5, 17, 2, 2, 2, 2, 2], - [1, 74, 17, 19, 22, 18, 13, 18, 11, 2, 2, 2, 2], - ]) - self.assertAllEqual(target_ids, [ - [12, 9, 16, 16, 19, 2, 2, 2, 2, 2, 2, 2, 2], - [23, 13, 22, 73, 2, 2, 2, 2, 2, 2, 2, 2, 2], - [27, 12, 5, 24, 3, 5, 3, 0, 3, 8, 5, 29, 2], - [5, 17, 9, 22, 13, 7, 5, 32, 23, 2, 2, 2, 2], - [4, 3, 9, 5, 22, 16, 29, 2, 2, 2, 2, 2, 2], - [40, 34, 39, 39, 3, 5, 17, 2, 2, 2, 2, 2, 2], - [74, 17, 19, 22, 18, 13, 18, 11, 2, 2, 2, 2, 2], - ]) - self.assertAllEqual(paddings, [ - [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1], - [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1], - [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1], - [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1], - [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1], - [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1], - ]) - - def testIdToToken(self): - with self.session(use_gpu=False): - token_ids = [[12, 9, 16, 16, 19, 2, 2, 2, 2, - 2], [23, 13, 22, 73, 2, 2, 2, 2, 2, - 2], [27, 12, 5, 24, 3, 5, 3, 0, 3, - 2], [5, 17, 9, 22, 13, 7, 5, 32, 23, 2], - [4, 3, 9, 5, 22, 16, 29, 2, 2, - 2], [40, 34, 39, 39, 3, 5, 17, 2, 2, - 2], [52, 2, 2, 2, 2, 2, 2, 2, 2, 2]] - seq_lens = [5, 4, 9, 9, 7, 7, 1] - tokens = self.evaluate(ops.id_to_ascii(token_ids, seq_lens)) - - self.assertEqual(tokens.tolist(), [ - b'hello', b'sir', b'what a ', b"america's", - b' early', b'1:00 am', b'%' - ]) - - def testStrToVocabToken(self): - vocab = test_helper.test_src_dir_path('core/ops/testdata/test_vocab.txt') - with self.session(use_gpu=False): - token_ids, target_ids, paddings = self.evaluate( - ops.str_to_vocab_tokens([ - 'a b c d e', - ' ', - 'øut über ♣ 愤青 ←', - ], - append_eos=True, - maxlen=10, - vocab_filepath=vocab)) - self.assertEqual( - token_ids.tolist(), - [[1, 5, 6, 7, 8, 9, 2, 2, 2, 2], [1, 0, 1, 2, 3, 2, 2, 2, 2, 2], - [1, 10, 11, 12, 13, 3, 2, 2, 2, 2]]) - self.assertEqual( - target_ids.tolist(), - [[5, 6, 7, 8, 9, 2, 2, 2, 2, 2], [0, 1, 2, 3, 2, 2, 2, 2, 2, 2], - [10, 11, 12, 13, 3, 2, 2, 2, 2, 2]]) - self.assertEqual(paddings.tolist(), - [[0., 0., 0., 0., 0., 0., 1., 1., 1., 1.], [ - 0., 0., 0., 0., 0., 1., 1., 1., 1., 1. - ], [0., 0., 0., 0., 0., 0., 1., 1., 1., 1.]]) - - def testStrToVocabTokenAppendEOSFalse(self): - vocab = test_helper.test_src_dir_path('core/ops/testdata/test_vocab.txt') - with self.session(use_gpu=False): - token_ids, target_ids, paddings = self.evaluate( - ops.str_to_vocab_tokens([ - 'a b c d e', - ' ', - 'øut über ♣ 愤青 ←', - ], - append_eos=False, - maxlen=10, - vocab_filepath=vocab)) - self.assertEqual( - token_ids.tolist(), - [[1, 5, 6, 7, 8, 9, 2, 2, 2, 2], [1, 0, 1, 2, 3, 2, 2, 2, 2, 2], - [1, 10, 11, 12, 13, 3, 2, 2, 2, 2]]) - self.assertEqual( - target_ids.tolist(), - [[5, 6, 7, 8, 9, 2, 2, 2, 2, 2], [0, 1, 2, 3, 2, 2, 2, 2, 2, 2], - [10, 11, 12, 13, 3, 2, 2, 2, 2, 2]]) - self.assertEqual(paddings.tolist(), - [[0., 0., 0., 0., 0., 1., 1., 1., 1., 1.], [ - 0., 0., 0., 0., 1., 1., 1., 1., 1., 1. - ], [0., 0., 0., 0., 0., 1., 1., 1., 1., 1.]]) - - def testStrToVocabTokenTruncates(self): - vocab = test_helper.test_src_dir_path('core/ops/testdata/test_vocab.txt') - with self.session(use_gpu=False): - token_ids, target_ids, paddings = self.evaluate( - ops.str_to_vocab_tokens(['a b c d e ' * 1000], - append_eos=True, - maxlen=5, - vocab_filepath=vocab)) - self.assertEqual(token_ids.tolist(), [[1, 5, 6, 7, 8]]) - self.assertEqual(target_ids.tolist(), [[5, 6, 7, 8, 9]]) - self.assertEqual(paddings.tolist(), [[0., 0., 0., 0., 0.]]) - - def testStrToVocabTokenNoPadToMaxlen(self): - vocab = test_helper.test_src_dir_path('core/ops/testdata/test_vocab.txt') - with self.session(use_gpu=False): - token_ids, target_ids, paddings = self.evaluate( - ops.str_to_vocab_tokens([ - 'a b c d e', - ' ', - 'øut über ♣ 愤青 ←', - ], - append_eos=True, - maxlen=10, - pad_to_maxlen=False, - vocab_filepath=vocab)) - self.assertEqual( - token_ids.tolist(), - [[1, 5, 6, 7, 8, 9], [1, 0, 1, 2, 3, 2], [1, 10, 11, 12, 13, 3]]) - self.assertEqual( - target_ids.tolist(), - [[5, 6, 7, 8, 9, 2], [0, 1, 2, 3, 2, 2], [10, 11, 12, 13, 3, 2]]) - self.assertEqual(paddings.tolist(), - [[0., 0., 0., 0., 0., 0.], [0., 0., 0., 0., 0., 1.], - [0., 0., 0., 0., 0., 0.]]) - - def testStrToVocabTokenCustomDelimiter(self): - custom_delimiter = '_' - vocab = test_helper.test_src_dir_path('core/ops/testdata/test_vocab.txt') - with self.session(use_gpu=False): - token_ids, target_ids, paddings = self.evaluate( - ops.str_to_vocab_tokens([custom_delimiter.join('abcde')], - append_eos=True, - maxlen=8, - vocab_filepath=vocab, - delimiter=custom_delimiter)) - self.assertEqual(token_ids.tolist(), [[1, 5, 6, 7, 8, 9, 2, 2]]) - self.assertEqual(target_ids.tolist(), [[5, 6, 7, 8, 9, 2, 2, 2]]) - self.assertEqual(paddings.tolist(), [[0., 0., 0., 0., 0., 0., 1., 1.]]) - - def testStrToVocabTokenSplitToCharacters(self): - custom_delimiter = '' - vocab = test_helper.test_src_dir_path('core/ops/testdata/test_vocab.txt') - with self.session(use_gpu=False): - token_ids, target_ids, paddings = self.evaluate( - ops.str_to_vocab_tokens(['abcde'], - append_eos=True, - maxlen=8, - vocab_filepath=vocab, - delimiter=custom_delimiter)) - self.assertEqual(token_ids.tolist(), [[1, 5, 6, 7, 8, 9, 2, 2]]) - self.assertEqual(target_ids.tolist(), [[5, 6, 7, 8, 9, 2, 2, 2]]) - self.assertEqual(paddings.tolist(), [[0., 0., 0., 0., 0., 0., 1., 1.]]) - - def testNgramIdToToken(self): - vocab = test_helper.test_src_dir_path('core/ops/testdata/test_ngrams.txt') - with self.session(use_gpu=False): - ngram_ids = [[14, 11, 6, 24, 7, 3, 13, 82, 2, 2], - [57, 3, 73, 17, 22, 9, 2, 2, 2, 2]] - lengths = [8, 6] - scripts = ops.ngram_id_to_token( - ngram_ids, lengths, ngram_vocab_filepath=vocab) - scripts_expected = [b'pn?o"{twe', b'gh{rtlcr'] - self.assertEqual(scripts_expected, scripts.eval().tolist()) - - def testMlPerf(self): - tf.logging.info(dir(ops)) - vocab = test_helper.test_src_dir_path( - 'core/ops/testdata/mlperf.ende.subwords.vocab') - with self.session(use_gpu=False): - lengths = [26, 33] - inputs = [[ - 264, 4, 288, 189, 4138, 30725, 14, 1461, 2761, 243, 28, 2692, 9, 1679, - 2, 1218, 9, 4, 15190, 7, 427, 497, 7, 147, 3, 1, 0, 0, 0, 0, 0, 0, 0 - ], - [ - 275, 52, 11, 1434, 215, 30372, 1546, 6997, 6495, 291, 802, - 10, 9986, 6, 10, 1857, 12726, 13624, 2, 53, 18, 87, 15934, - 5618, 12321, 127, 1565, 4209, 885, 777, 335, 3, 1 - ]] - expected = [ - six.ensure_binary( - 'If the market “misbehaves,” farmers could be reduced to poverty,' - ' leading to the neglect of large areas of Europe.'), - six.ensure_binary( - 'Wenn sich der Markt „daneben benimmt“ könnten die Bauern in die ' - 'Armut abgleiten, was zu einer Vernachlässigung großer Teile ' - 'Europas führen würde.') - ] - decode = ops.ml_perf_subword_id_to_string( - inputs, lengths, vocab_filepath=vocab) - self.assertEqual(expected, decode.eval().tolist()) - - def testNgramIdToTokenSeparator(self): - vocab = test_helper.test_src_dir_path('core/ops/testdata/test_ngrams.txt') - with self.session(use_gpu=False): - ngram_ids = [[14, 11, 6, 24, 7, 3, 13, 82, 2, 2], - [57, 3, 73, 17, 22, 9, 2, 2, 2, 2]] - lengths = [8, 6] - scripts = ops.ngram_id_to_token( - ngram_ids, lengths, ngram_vocab_filepath=vocab, ngram_separator='.') - scripts_expected = [b'p.n.?.o.".{.t.we', b'gh.{.rt.l.c.r'] - self.assertEqual(scripts_expected, scripts.eval().tolist()) - - def testBpeTokenization(self): - word_vocab = test_helper.test_src_dir_path( - 'core/ops/testdata/bpe_words.vocab') - code_vocab = test_helper.test_src_dir_path( - 'core/ops/testdata/bpe_codes.vocab') - sentences = [ - 'GIVE ME A PENNY', 'THEY LIVED ALONE', 'THEY GIVE ME A PENNY ALONE' - ] - expected_sentences = [ - b'GIVE ME A PENNY ', - b'THEY LIVED ALONE ', - b'THEY GIVE ME A PENNY ', - ] - expected_token_ids = [ - [27, 9, 30, 14, 28, 14, 52, 11, 4, 6, 6, 10, 2, 2, 2], - [16, 4, 10, 12, 9, 30, 24, 7, 12, 49, 14, 2, 2, 2, 2], - [16, 4, 10, 27, 9, 30, 14, 28, 14, 52, 11, 4, 6, 6, 10], - ] - with self.session(use_gpu=False): - label_tensor = tf.constant(sentences) - _, token_ids, paddings = ops.bpe_words_to_ids( - label_tensor, tokenization_filepath=word_vocab, maxlen=15) - seq_lens = tf.cast( - tf.round(tf.reduce_sum(1 - paddings, axis=-1)), tf.int32) - - target_string = ops.bpe_ids_to_words( - token_ids, seq_lengths=seq_lens, vocab_filepath=code_vocab) - self.assertEqual(expected_sentences, target_string.eval().tolist()) - self.assertEqual(expected_token_ids, token_ids.eval().tolist()) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/ops/versioned_file_set.proto b/lingvo/core/ops/versioned_file_set.proto deleted file mode 100644 index d9c30ad2..00000000 --- a/lingvo/core/ops/versioned_file_set.proto +++ /dev/null @@ -1,24 +0,0 @@ -syntax = "proto2"; - -package tensorflow.lingvo; -option cc_enable_arenas = true; - -// Protocol buffer representing the data version. -message VersionedFileSet { - // The latest version of this fileset. - optional FileSet current = 1; - - // A list of recent past versions of this fileset. - repeated FileSet history = 2; -} - -message FileSet { - // Each file_pattern is a file glob pattern or a sharded file spec - // file_pattern is intended to store a relative path, to enable moving - // datasets around without having to update the pbtxt file storing the - // VersionedFileSet information. - repeated string file_pattern = 1; - - // Seconds since epoch indicating when this version of fileset is created. - optional double create_timestamp = 2; -} diff --git a/lingvo/core/ops/weighted_mix_record_yielder.cc b/lingvo/core/ops/weighted_mix_record_yielder.cc deleted file mode 100644 index 341e8534..00000000 --- a/lingvo/core/ops/weighted_mix_record_yielder.cc +++ /dev/null @@ -1,87 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/weighted_mix_record_yielder.h" - -#include - -namespace tensorflow { -namespace lingvo { - -WeightedMixRecordYielder::WeightedMixRecordYielder( - const int64 seed, const std::vector& yielders, - const std::vector& input_source_weights) - : rnd_(seed), - sample_distribution_(input_source_weights.begin(), - input_source_weights.end()), - yielders_(yielders) { - if (yielders.size() != input_source_weights.size()) { - LOG(FATAL) << "Unable to create WeightedMixRecordYielder: every yielder " - << "should have a corresponding weight. " << yielders.size() - << " yielders and " << input_source_weights.size() - << " weights were " - << "provided."; - } - if (yielders.empty()) { - LOG(FATAL) << "There should be at least one yielder provided."; - } - - for (float x : input_source_weights) { - if (x < 0) { - LOG(FATAL) << "All weights should be greater or equal to zero. Got " << x; - } - } -} - -WeightedMixRecordYielder* WeightedMixRecordYielder::New( - const int64 seed, const std::vector& yielders, - const std::vector& input_source_weights) { - WeightedMixRecordYielder* yielder = - new WeightedMixRecordYielder(seed, yielders, input_source_weights); - return yielder; -} - -WeightedMixRecordYielder::~WeightedMixRecordYielder() {} - -void WeightedMixRecordYielder::Close() { - for (RecordYielder* yielder : yielders_) { - yielder->Close(); - } - LOG(INFO) << this << "Weighted mix record yielder exit"; - delete this; -} - -Status WeightedMixRecordYielder::Yield(Record* record) { - size_t yielder_idx = 0; - { - absl::MutexLock l(&mu_); - yielder_idx = sample_distribution_(rnd_); - // Release the lock immediately once we fix which yielder to use. - } - while (true) { - // Retry indefinitely until we get an Ok status from the specific yielder. - // This will stall the training if there is any unrecoverable error with - // the child yielder. - Status s = yielders_.at(yielder_idx)->Yield(record); - if (!s.ok()) { - LOG(WARNING) << s; - continue; - } - return s; - } -} - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/weighted_mix_record_yielder.h b/lingvo/core/ops/weighted_mix_record_yielder.h deleted file mode 100644 index 19f11b8c..00000000 --- a/lingvo/core/ops/weighted_mix_record_yielder.h +++ /dev/null @@ -1,88 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ -#ifndef LINGVO_CORE_OPS_WEIGHTED_MIX_RECORD_YIELDER_H_ -#define LINGVO_CORE_OPS_WEIGHTED_MIX_RECORD_YIELDER_H_ - -#include "lingvo/core/ops/record_yielder.h" - -namespace tensorflow { -namespace lingvo { - -// WeightedMixRecordYielder is a special RecordYielder that mixes examples from -// yielders in a random order maintaining the mix ratio specified by -// input_source_weights. -// -// Usage example: -// BasicRecordYielder::Options opts1; -// opts1.file_pattern = ; -// opts1.seed = 301; -// opts1.bufsize = 1000000; // A randomized buffer with 1M records. -// opts1.parallelism = 8; // Use 8 iterators. -// RecordYielder* yielder1 = BasicRecordYielder::New(opts1); -// BasicRecordYielder::Options opts2; -// opts2.file_pattern = ; -// opts2.seed = 301; -// opts2.bufsize = 1000000; // A randomized buffer with 1M records. -// opts2.parallelism = 8; // Use 8 iterators. -// RecordYielder* yielder2 = BasicRecordYielder::New(opts2); -// // Use records from yielder1 for 70% of the yields. -// WeightedMixRecordYielder* yielder = WeightedMixRecordYielder::New( -// opts1.seed, {yielder1, yielder2}, {0.7, 0.3}); -// Record record; -// while (true) { -// yielder->Yield(&record); -// // process record. -// } -// yielder->Close(); -// -// WeightedMixRecordYielder can be accessed by multiple threads concurrently. -class WeightedMixRecordYielder : public RecordYielder { - public: - ~WeightedMixRecordYielder() override; - void Close() override; - Status Yield(Record* record) override; - - // Creates new WeightedMixRecordYielder and takes ownership over yielders - // provided. Those yielders should be properly initialized already and will be - // closed once WeightedMixRecordYielder is closed. Caller is responsible - // closing the WeightedMixRecordYielder returned by this function. Caller - // should not delete the yielder as it will be handled internally. - static WeightedMixRecordYielder* New( - const int64 seed, - const std::vector& yielders, - const std::vector& input_source_weights); - - protected: - WeightedMixRecordYielder( - const int64 seed, - const std::vector& yielders, - const std::vector& input_source_weights); - - - private: - mutable absl::Mutex mu_; - - // PRG used for randomization. - std::mt19937_64 rnd_ ABSL_GUARDED_BY(mu_); - - std::discrete_distribution sample_distribution_; - - // A list of child yielders used as an input to the mixer. - std::vector yielders_; -}; - -} // namespace lingvo -} // namespace tensorflow -#endif // LINGVO_CORE_OPS_WEIGHTED_MIX_RECORD_YIELDER_H_ diff --git a/lingvo/core/ops/weighted_mix_record_yielder_test.cc b/lingvo/core/ops/weighted_mix_record_yielder_test.cc deleted file mode 100644 index f0e7bfb9..00000000 --- a/lingvo/core/ops/weighted_mix_record_yielder_test.cc +++ /dev/null @@ -1,241 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/core/ops/weighted_mix_record_yielder.h" - -#include - -#include -#include -#include "lingvo/core/ops/input_common.h" -#include "lingvo/core/ops/record_yielder.h" -#include "lingvo/core/ops/yielder_test_helper.h" -#include "tensorflow/core/lib/core/stringpiece.h" -#include "tensorflow/core/lib/io/path.h" -#include "tensorflow/core/platform/env.h" - -namespace tensorflow { -namespace lingvo { - -TEST(RecordYielderTest, WeightedMixerBasicTest) { - const int N = 10; - const int M = 1000; - GeneratePlainTextTestData("yielder1", N, M); - GeneratePlainTextTestData("yielder2", N, M); - - BasicRecordYielder::Options opts1; - opts1.file_pattern = - strings::StrCat("text:", io::JoinPath("/tmp", "yielder1.*")); - opts1.seed = 301; - opts1.bufsize = 2000; - opts1.parallelism = 1; - BasicRecordYielder* yielder1 = BasicRecordYielder::New(opts1); - - BasicRecordYielder::Options opts2; - opts2.file_pattern = - strings::StrCat("text:", io::JoinPath("/tmp", "yielder2.*")); - opts2.seed = 301; - opts2.bufsize = 2000; - opts2.parallelism = 1; - BasicRecordYielder* yielder2 = BasicRecordYielder::New(opts2); - WeightedMixRecordYielder* yielder = - WeightedMixRecordYielder::New(301, {yielder1, yielder2}, {0.5, 0.5}); - - std::vector vals; - Record record; - record.source_id = kDefaultSourceId; - for (int i = 0; i < 2 * N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - VLOG(1) << i << " " << record.value; - vals.emplace_back(string(record.value)); - } - - auto input_source_distribution = ComputeInputSourceDistribution(vals); - ASSERT_NEAR(input_source_distribution["yielder1"], 0.5, 0.01); - ASSERT_NEAR(input_source_distribution["yielder2"], 0.5, 0.01); - - std::sort(vals.begin(), vals.end()); - auto new_end = std::unique(vals.begin(), vals.end()); - - // Duplicates should be rare in the epoch thanks to 0.5/0.5 mix and equally - // sized input sources. - ASSERT_LT(vals.end() - new_end, vals.size() / 100); - - yielder->Close(); -} - -TEST(RecordYielderTest, WeightedMixerUnevenMixTest) { - const int N = 10; - const int M = 1000; - GeneratePlainTextTestData("yielder1", N, M); - GeneratePlainTextTestData("yielder2", N, M); - - BasicRecordYielder::Options opts1; - opts1.file_pattern = - strings::StrCat("text:", io::JoinPath("/tmp", "yielder1.*")); - opts1.seed = 301; - opts1.bufsize = 2000; - opts1.parallelism = 1; - opts1.source_id = 0; - BasicRecordYielder* yielder1 = BasicRecordYielder::New(opts1); - - BasicRecordYielder::Options opts2; - opts2.file_pattern = - strings::StrCat("text:", io::JoinPath("/tmp", "yielder2.*")); - opts2.seed = 301; - opts2.bufsize = 2000; - opts2.parallelism = 1; - opts2.source_id = 1; - BasicRecordYielder* yielder2 = BasicRecordYielder::New(opts2); - WeightedMixRecordYielder* yielder = - WeightedMixRecordYielder::New(301, {yielder1, yielder2}, {0.3, 0.7}); - - std::vector vals; - std::vector source_ids; - Record record; - record.source_id = kDefaultSourceId; - for (int i = 0; i < 2 * N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - VLOG(1) << i << " " << record.value; - vals.emplace_back(string(record.value)); - source_ids.emplace_back(record.source_id); - } - - auto input_source_distribution = ComputeInputSourceDistribution(vals); - ASSERT_NEAR(input_source_distribution["yielder1"], 0.3, 0.01); - ASSERT_NEAR(input_source_distribution["yielder2"], 0.7, 0.01); - - int32 sum_of_elems = std::accumulate(source_ids.begin(), source_ids.end(), 0); - float ratio = (float)(sum_of_elems) / (float)(source_ids.size()); - ASSERT_NEAR(ratio, 0.7, 0.01); - - // Take couple 1024-sized batches from the vals, they should have roughly the - // same distribution. - for (int i = 0; i < 5; ++i) { - auto batch_input_source_distribution = - ComputeInputSourceDistribution(std::vector( - vals.begin() + i * 1024, vals.begin() + (i + 1) * 1024)); - ASSERT_NEAR(input_source_distribution["yielder1"], 0.3, 0.01); - ASSERT_NEAR(input_source_distribution["yielder2"], 0.7, 0.01); - } - - yielder->Close(); -} - -TEST(RecordYielderTest, WeightedMixerUnevenInputSourcesTest) { - const int N = 10; - const int M = 1000; - GeneratePlainTextTestData("yielder1", N, M); - // Second input source is 4 times larger than the first one. - GeneratePlainTextTestData("yielder2", 4 * N, M); - - BasicRecordYielder::Options opts1; - opts1.file_pattern = - strings::StrCat("text:", io::JoinPath("/tmp", "yielder1.*")); - opts1.seed = 301; - opts1.bufsize = 2000; - opts1.parallelism = 1; - BasicRecordYielder* yielder1 = BasicRecordYielder::New(opts1); - - BasicRecordYielder::Options opts2; - opts2.file_pattern = - strings::StrCat("text:", io::JoinPath("/tmp", "yielder2.*")); - opts2.seed = 301; - opts2.bufsize = 2000; - opts2.parallelism = 1; - BasicRecordYielder* yielder2 = BasicRecordYielder::New(opts2); - WeightedMixRecordYielder* yielder = - WeightedMixRecordYielder::New(301, {yielder1, yielder2}, {0.5, 0.5}); - - std::vector vals; - Record record; - record.source_id = kDefaultSourceId; - // Iterate 8 times the total record count. - for (int i = 0; i < 8 * 5 * N * M; ++i) { - TF_CHECK_OK(yielder->Yield(&record)); - VLOG(1) << i << " " << record.value; - vals.emplace_back(string(record.value)); - } - auto input_source_distribution = ComputeInputSourceDistribution(vals); - ASSERT_NEAR(input_source_distribution["yielder1"], 0.5, 0.01); - ASSERT_NEAR(input_source_distribution["yielder2"], 0.5, 0.01); - - // Each child yielder will be called half the time to maintain 50/50% split. - // So out of (8 * 5 * N * M) total yields (4 * 5 * N * M) will be yielded by - // each yielder. - // Second yielder produces 4 * N * M elements per epoch so it should be at the - // end of the 5th epoch | start of the 6th epoch. - EXPECT_TRUE(yielder2->current_epoch() == 5 || yielder2->current_epoch() == 6); - // First yielder produces 4 times less elements per epoch (N * M) so it - // should be at the end of the 20th epoch | start of the 21st epoch. - EXPECT_TRUE(yielder1->current_epoch() == 20 || - yielder1->current_epoch() == 21); - yielder->Close(); -} - -TEST(RecordYielderTest, RecordYielderRetryLoop) { - MockRecordYielder yielder1; - MockRecordYielder yielder2; - // Yielder1 always returns OK. Yielder2 returns DEADLINE_EXCEEDED 3 times in a - // row and then returns OK. - // Each of them yields max of 5 records and then saturates. - EXPECT_CALL(yielder1, Yield(testing::_)) - .Times(5) - .WillRepeatedly(testing::Return(Status::OK())); - EXPECT_CALL(yielder2, Yield(testing::_)) - .Times(5) - .WillRepeatedly(testing::Return(Status::OK())); - EXPECT_CALL(yielder2, Yield(testing::_)) - .Times(3) - .WillRepeatedly(testing::Return(Status(error::DEADLINE_EXCEEDED, ""))) - .RetiresOnSaturation(); - - WeightedMixRecordYielder* yielder = - WeightedMixRecordYielder::New(304, {&yielder1, &yielder2}, {0.5, 0.5}); - - Record record; - record.source_id = kDefaultSourceId; - for (int i = 0; i < 10; ++i) { - // Thanks to the seed selected every child yielder will be selected exactly - // 5 times. - TF_CHECK_OK(yielder->Yield(&record)); - } - EXPECT_CALL(yielder1, Close()); - EXPECT_CALL(yielder2, Close()); - yielder->Close(); -} - -TEST(RecordYielderDeathTest, WeightedMixerInconsistentYieldersAndWeights) { - RecordYielder* yielder1 = nullptr; // won't ever be used. - RecordYielder* yielder2 = nullptr; // won't ever be used. - ASSERT_DEATH(WeightedMixRecordYielder::New(301, {yielder1, yielder2}, {0.5}), - "2 yielders and 1 weights were provided"); -} - -TEST(RecordYielderDeathTest, WeightedMixerNoYielders) { - ASSERT_DEATH(WeightedMixRecordYielder::New(301, {}, {}), - "There should be at least one yielder provided"); -} - -TEST(RecordYielderDeathTest, WeightedMixerNegativeWeights) { - RecordYielder* yielder1 = nullptr; // won't ever be used. - RecordYielder* yielder2 = nullptr; // won't ever be used. - ASSERT_DEATH( - WeightedMixRecordYielder::New(301, {yielder1, yielder2}, {0.3, -0.1}), - "All weights should be greater or equal to zero. Got -0.1"); -} - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/x_ops.cc b/lingvo/core/ops/x_ops.cc deleted file mode 100644 index a5def996..00000000 --- a/lingvo/core/ops/x_ops.cc +++ /dev/null @@ -1,1027 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "tensorflow/core/framework/common_shape_fns.h" -#include "tensorflow/core/framework/op.h" -#include "tensorflow/core/framework/shape_inference.h" -#include "tensorflow/core/framework/types.pb.h" -#include "x_ops_helper.h" - -namespace tensorflow { -namespace { - -REGISTER_OP("AssertShapeMatch") - .Input("x: int32") - .Input("y: int32") - .Attr("msg: string = ''") - .SetShapeFn([](shape_inference::InferenceContext* c) { - return Status::OK(); - }) - .Doc(R"doc( -Asserts that shape vector x and y matches. - -The i-th dimension matches iff x[i] == y[i] || x[i] == -1 || y[i] == -1. - -x: A shape vector. -y: A shape vector. -msg: The error message generated when the assertion failed. -)doc"); - -REGISTER_OP("AssertSameDim0") - .Input("x: types") - .Attr("msg: string = ''") - .Attr("types: list(type)") - .SetShapeFn([](shape_inference::InferenceContext* c) { - return Status::OK(); - }) - .Doc(R"doc( -Asserts that all input tensors are non-scalar and have the same 0-th dim size. - -x: The list of tensors. -msg: The error message generated when the assertion failed. -)doc"); - -REGISTER_OP("RandomPermutationSequence") - .Attr("num: int = 1") - .Attr("batch: int = 1") - .Attr("repeat: bool = false") - .Attr("seed: int = 0") - .Output("out: int32") - .SetIsStateful() - .Doc(R"doc( -Generate random samples from [0..num-1] without replacements. - -num: The number of ids. -batch: Each output is a vector of size up to batch. Right now, - the last batch from one epoch is the only one that can be - smaller than a full batch. -repeat: If true, this op keep generating random samples after one - epoch. If false, this op errors with `tf.errors.OutOfRangeError` when an - epoch finishes. -seed: The random seed. -out: Each output is a vector of size up to batch. -)doc"); - -REGISTER_OP("BestStep") - .Output("best_step: int64") - .Attr("hist_file: string") - .Attr("tol: float = 0.0") - .Attr("minimize: bool = true") - .Attr("metric: string = \"\"") - .SetIsStateful() - .Doc(R"doc( - -Determines the best global step from a history file. - -best_step: Shape [2]. best_step[0] is scalar value for best global step. - best_step[1] is scalar value for last global step. -hist_file: A text file containing 'step score' records, or a file pattern that - matches tf event files in the format of /path_to_file/events.out.tfevents*. -tol: Difference between previous best score and current score must be greater -than this amount to trigger update. -minimize: If the metric is being minimized. Recorded in hist_file, smaller - scores are better if True, and bigger scores are better if False. -metric: The name of the metric being tracked. -)doc"); - -REGISTER_OP("BeamSearchStep") - .Input("scores: float32") - .Input("atten_probs: float32") - .Input("best_scores: float32") - .Input("cumulative_scores: float32") - .Input("in_scores: float32") - .Input("in_hyps: int32") - .Input("in_prev_hyps: int32") - .Input("in_done_hyps: string") - .Input("in_atten_probs: float32") - .Input("is_last_chunk: bool") - .Input("cur_step: int32") - .Output("out_best_scores: float32") - .Output("out_cumulative_scores: float32") - .Output("out_scores: float32") - .Output("out_hyps: int32") - .Output("out_prev_hyps: int32") - .Output("out_done_hyps: string") - .Output("out_atten_probs: float32") - .Output("all_done: bool") - .Attr("eoc_id: int = -1") - .Attr("eos_id: int") - .Attr("beam_size: float") - .Attr("num_hyps_per_beam: int") - .Attr("valid_eos_max_logit_delta: float = 5.0") - .Attr("local_eos_threshold: float = -100.0") - .Attr("merge_paths: bool = false") - .Attr("allow_empty_terminated_hyp: bool = true") - .Attr("ensure_full_beam: bool = false") - .Attr("force_eos_in_last_step: bool = false") - .SetShapeFn([](shape_inference::InferenceContext* c) { - c->set_output(0, c->input(2)); - c->set_output(1, c->input(3)); - c->set_output(2, c->input(4)); - c->set_output(3, c->input(5)); - c->set_output(4, c->input(6)); - c->set_output(5, c->input(7)); - c->set_output(6, c->input(8)); - c->set_output(7, c->Scalar()); - return Status::OK(); - }) - .Doc(R"doc( - -Move forward one step in beam search. - -Let "b" be the number of beams, "k" be the number hyps in each beam, "t" be the -maximum decoding steps. - -The following data structures are allocated before the first decoding step and -are passed along from cur step to the next step: - -in_scores - A tensor of shape [t, b * k]. in_scores[i, j] is the local - score of the j-th hyp at the i-th decoding step. -in_hyps - A tensor of shape [t, b * k]. in_hyps[i, j] is the token id of the - j-th hyp at the i-th decoding step. -in_prev_hyps - A tensor of shape [t, b * k]. in_prev_hyps[i, j] stores a - pointer of the j-th hyp at time step i to the hyp at previous timestep - (i - 1). 0 <= in_prev_hyps[i, j] < b * k. -in_done_hyps - A tensor of shape [t, b * k]. in_done_hyps[i, j] can be either an - empty string, or a serialized Hypothesis proto. Terminated hyps are removed - from the beam and are moved to the corresponding in_done_hyps slot. -in_atten_probs - A tensor of shape [t, b * k, s_len]. in_atten_probs[i, j, ...] - is the attention probs over the source words for the j-th hyp at the i-th - timestep. - -Those tensors are modified (with content for the cur_step timestep being filled -in) within this op invocation and are passed to the corresponding output -tensors. - -is_last_chunk: A tensor of shape [b * k]. Used by neural transducer, determine - whether the current hypothesis reaches the last chunk and should treat the - next end-of-chunk symbol as end-of-sentence. -scores: A matrix of shape [b * k, vocab_size], where b is the number of - active beams, and k is the number of hyps in each beam. Local scores for the - current timestep. -atten_probs: A matrix of shape [b * k, source_len]. Attention probabilities - for the current timestep. -best_scores: A vector of size [b], best scores of terminated hyps so far in - each of the beams. -cumulative_scores: A vector of size [b * k]. The cumulative score of each - active hyp before the current step. -in_scores: As explained above. -in_hyps: As explained above. -in_prev_hyps: As explained above. -in_done_hyps: As explained above. -in_atten_probs: As explained above. -cur_step: Current step id. -out_best_scores: - Updated best scores for each of the beams. -out_cumulative_scores: - A vector of size [b * k]. The cumulative score of the new hyps after the - current decoding step. -out_scores: - As explained above. -out_hyps: - As explained above. -out_prev_hyps: - As explained above. -out_done_hyps: - As explained above. -out_atten_probs: - As explained above. -all_done: - A scalar, whether decoding should terminate for all beams. -eoc_id: Token id of the special end of chunk token. -eos_id: Token id of the special end of sequence token. -beam_size: Search terminates if the delta between the scores of the active hyps - in a beam and the best scores exceeds this threashold. -num_hyps_per_beam: Number of hyps in a beam. -valid_eos_max_logit_delta: We allow to terminate a hyp only if its logit - is no more than `valid_eos_max_logit_delta` away from the logit of the best - candidate. -local_eos_threshold: We allow to terminate a hyp if the local score for - is greater than local_eos_threshold. -merge_paths: If true, hyps which are identical when epsilons are removed will - be combined into a single hyp. The probability for that combined hyp will - be the sum of the probabilities of the component hyps. This can only be - applied for epsilon-emitting models (RNN-T and NT). -allow_empty_terminated_hyp: Whether it is okay to consider a hyp that consists - only of epsilons as terminated. By default this is true, as an - utterance may consist of silence. It should be set to false when EMBR - training epsilon-emitting models (e.g., RNN-T), which are prone to emit - all-epsilon hyps even in the absence of silence. Note that a hyp that - terminates in EOS is not considered empty, so this flag has no effect for - non-epsilon-emitting models. -ensure_full_beam: If True, we will not set the all_done output to True until we - have found 'num_hyps_per_beam' terminated hyps AND no active hyps have a - score within 'beam_size' of the best terminated hyp. If False, only the - second condition must be satisfied. Generally this should be False unless - beam search is being run as part of minimum word error rate training. -force_eos_in_last_step: If true, then if decode does not terminate even after - (max - 1) steps, eos symbol is injected into the result and partial - hypotheses (with a valid eos symbol in the end) are returned. all_done - is set to true for these partials. If false, which is the default behavior, - empty hypothesis are returned and all_done is set to false at termination. -)doc"); - -REGISTER_OP("TopKTerminatedHyps") - .Input("in_done_hyps: string") - .Input("src_seq_lengths: int32") - .Output("out_topk_hyps: string") - .Attr("k: int") - .Attr("num_hyps_per_beam: int") - .Attr("length_normalization: float") - .Attr("coverage_penalty: float") - .Attr("target_seq_length_ratio: float=1.0") - .Attr("eoc_id: int=-1") - .Attr("merge_paths: bool = false") - .SetShapeFn([](shape_inference::InferenceContext* c) { - auto batch_size = c->Dim(c->input(1), 0); - int k; - TF_RETURN_IF_ERROR(c->GetAttr("k", &k)); - shape_inference::DimensionOrConstant k_dim = c->UnknownDim(); - if (k > 0) { - k_dim = k; - } - c->set_output(0, c->Matrix(batch_size, k_dim)); - return Status::OK(); - }) - .Doc(R"doc( - -Compute the top k terminated hyps based on normalized score for each beam. - -Let "b" be the number of beams, "h" be the number hyps in each beam, "t" be the -maximum decoding steps. - -in_done_hyps: A tensor of shape [t, h * b]. Each string in in_done_hyps can be - either an empty string, or a serialized Hypothesis proto. If not empty, - in_done_hyps[t, i * num_beams + j] represents the i-th hypothesis for beam - j that terminates at step t. -src_seq_lengths: A tensor of shape [b] of the src sequence lengths. -out_topk_hyps: A string tensor of shape [b, k]. topk_hyps[i: ] contains - top k terminated hyp for beam 'i', each hyp could be either an empty string - or a serialized `Hypothesis` proto. -k: number of highest scoring hyps to be returned for each beam. -num_hyps_per_beam: Number of hyps per beam in the input `in_done_hyps`. -length_normalization: The length normalization ratio. -coverage_penalty: The alpha value for coverage penalty. -target_seq_length_ratio: Ratio of the average target sequence length - over the average source sequence length. -eoc_id: Token id of the special end of chunk or blank (epsilon) token. -1 means - this model does not use epsilon. -merge_paths: If true, hyps which are identical when epsilons are removed will - be combined into a single hyp. The probability for that combined hyp will - be the sum of the probabilities of the component hyps. This can only be - applied for epsilon-emitting models (RNN-T and NT). -)doc"); - -REGISTER_OP("UnpackHyp") - .Input("in_hyps: string") - .Output("out_ids: int32") - .Output("out_seq_lens: int32") - .Output("out_scores: float32") - .Attr("max_seq_length: int = 0") - .SetShapeFn([](shape_inference::InferenceContext* c) { - auto batch_size = c->NumElements(c->input(0)); - int k; - TF_RETURN_IF_ERROR(c->GetAttr("max_seq_length", &k)); - shape_inference::DimensionOrConstant k_dim = c->UnknownDim(); - if (k > 0) { - k_dim = k; - } - c->set_output(0, c->Matrix(batch_size, k_dim)); - c->set_output(1, c->Vector(batch_size)); - c->set_output(2, c->Vector(batch_size)); - return Status::OK(); - }) - .Doc(R"doc( -Unpacks hyps into tensors of ids, seq_len and scores. - -in_hyps: A vector of serialized `Hypothesis` protos. -out_ids: - Output sequences, a matrix of shape (batch_size, max_seq_length). - Sequences shorter than max_seq_length are padded with 0s. If max_seq_length is 0, derive it from the longest sequence in input_hyps. -out_seq_lens: - Length of each of the output sequence, a vector of size `batch_size`. -out_scores: - Scores for each of the output sequence, a vector of `batch_size`. -)doc"); - -REGISTER_OP("HypsFromBeamSearchOuts") - .Input("hyps: int32") - .Input("prev_hyps: int32") - .Input("done_hyps: bool") - .Input("scores: T") - .Input("atten_probs: T") - .Input("eos_scores: T") - .Input("eos_atten_probs: T") - .Output("out_hyps: string") - .Attr("T: {float, bfloat16} = DT_FLOAT") - .Attr("eos_id: int") - .Attr("num_hyps_per_beam: int") - .SetShapeFn([](shape_inference::InferenceContext* c) { - c->set_output(0, c->input(0)); - return ::tensorflow::Status::OK(); - }) - .Doc(R"doc( - -Generates `Hypothesis` protos from output of a beam search step. - -hyps: A tensor of shape [t, b * k] with ids of the token selected. -prev_hyps: A tensor of shape [t, b * k] with index to the previous hyps which - was selected. -done_hyps: A boolean tensor of shape [t, b * k] where value indicates if hyps - was terminated. -scores: A tensor of shape [t, b * k]. in_scores[i, j] is the local score of - the j-th hyp at the i-th decoding step. -atten_probs: A tensor of shape [t, b * k, s_len]. atten_probs[i, j, ...] - is the attention probs over the source words for the j-th hyp at the i-th - timestep. -eos_scores: A tensor of shape [t, b * k]. eos_scores[i, j] is the local - score of the EOS token at the j-th hyp at the i-th decoding step. -eos_atten_probs: A tensor of shape [t, b * k, s_len]. - eos_atten_probs[i, j, ...] is the attention probs over the source words - for the j-th terminated hyp at the i-th timestep. -out_hyps: A tensor of shape [t, b * k] with terminated hyps. -eos_id: Token id of the special end of sequence token. -num_hyps_per_beam: Number of hyps per beam. -)doc"); - -REGISTER_OP("CachedCall") - .Output("output: T") - .Attr("f: func") - .Attr("T: list(type) >= 0") - .SetIsStateful() - .SetShapeFn(shape_inference::UnknownShape) - .Doc(R"doc( -Invokes function f once and memorize its output. - -output: A list of output tensors whose types are T. -f: A function that returns a list of tensors (T). -)doc"); - -REGISTER_OP("VocabTokenToId") - .Input("token: string") - .Output("id: int32") - .Attr("vocab: list(string)") - .Attr("load_token_ids_from_vocab: bool = false") - .SetIsStateful() - .SetShapeFn(shape_inference::UnchangedShape) - .Doc(R"doc( -Looks up the token in the vocab and return its id. - -token: A scalar or list of strings. -id: A scalar or list of ints. -vocab: A list of strings. -load_token_ids_from_vocab: Whether token ids are present in vocab (i.e. vocab - contains two colums, one for IDs and one for words). If false, line numbers - are used. -)doc"); - -REGISTER_OP("VocabIdToToken") - .Input("id: int32") - .Output("token: string") - .Attr("vocab: list(string)") - .Attr("load_token_ids_from_vocab: bool = false") - .SetIsStateful() - .SetShapeFn(shape_inference::UnchangedShape) - .Doc(R"doc( -Looks up the token at the given id from a vocab. - -id: A scalar or list of ints. -token: A scalar or list of strings. -vocab: A list of strings. -load_token_ids_from_vocab: Whether token ids are present in vocab (i.e. vocab - contains two colums, one for IDs and one for words). If false, line numbers - are used. -)doc"); - -REGISTER_OP("TokenInVocab") - .Input("token: string") - .Output("result: bool") - .Attr("vocab: list(string)") - .Attr("load_token_ids_from_vocab: bool = false") - .SetIsStateful() - .SetShapeFn(shape_inference::UnchangedShape) - .Doc(R"doc( -Checks whether the provided token is in the vocab. - -token: A scalar or list of strings. -result: A scalar or list of bools. -vocab: A list of strings. -load_token_ids_from_vocab: Whether token ids are present in vocab (i.e. vocab - contains two colums, one for IDs and one for words). If false, line numbers - are used. -)doc"); - -REGISTER_OP("AsciiToTokenId") - .Input("labels: string") - .Output("token_ids: int32") - .Output("target_ids: int32") - .Output("paddings: float") - .Attr("append_eos: bool = true") - .Attr("maxlen: int = 300") - .Attr("pad_to_maxlen: bool = true") - .SetShapeFn([](shape_inference::InferenceContext* c) { - auto batch_size = c->Dim(c->input(0), 0); - int maxlen; - TF_RETURN_IF_ERROR(c->GetAttr("maxlen", &maxlen)); - c->set_output(0, c->Matrix(batch_size, maxlen)); - c->set_output(1, c->Matrix(batch_size, maxlen)); - c->set_output(2, c->Matrix(batch_size, maxlen)); - return Status::OK(); - }) - .Doc(R"doc( -Converts ASCII label strings into token ids. - -labels: A vector of shape [batch]. -token_ids: - A matrix of shape [batch, maxlen]. - token_ids[i, j] is the i-th sample's j-th token id. - token_ids[i, 0] is always . -target_ids: - A matrix of shape [batch, maxlen]. - target_ids[i, j] is the i-th sample's j-th prediction label id. -paddings: - A matrix of shape [batch, maxlen]. - paddings[i, j] == 1.0 indicates that i-th training example' - j-th target token is padded and should be ignored. -append_eos: Whether to append at the end and treat it as a non-padded - label. -maxlen: an integer, sequence length of the output tensors. -pad_to_maxlen: Whether to pad the output to maxlen. -)doc"); - -REGISTER_OP("StrToVocabTokens") - .Input("labels: string") - .Output("token_ids: int32") - .Output("target_ids: int32") - .Output("paddings: float") - .Attr("append_eos: bool = true") - .Attr("maxlen: int = 300") - .Attr("pad_to_maxlen: bool = true") - .Attr("vocab_filepath: string") - .Attr("load_token_ids_from_vocab: bool = true") - .Attr("delimiter: string = ' '") - .SetShapeFn([](shape_inference::InferenceContext* c) { - auto batch_size = c->Dim(c->input(0), 0); - int maxlen; - TF_RETURN_IF_ERROR(c->GetAttr("maxlen", &maxlen)); - c->set_output(0, c->Matrix(batch_size, maxlen)); - c->set_output(1, c->Matrix(batch_size, maxlen)); - c->set_output(2, c->Matrix(batch_size, maxlen)); - return Status::OK(); - }) - .Doc(R"doc( -Tokenizes string into white space separated tokens according to a vocab file. - -labels: A vector of shape [batch]. -token_ids: - A matrix of shape [batch, maxlen]. - token_ids[i, j] is the i-th sample's j-th token id. - token_ids[i, 0] is always . -target_ids: - A matrix of shape [batch, maxlen]. - target_ids[i, j] is the i-th sample's j-th prediction label id. -paddings: - A matrix of shape [batch, maxlen]. - paddings[i, j] == 1.0 indicates that i-th training example's - j-th target token is padded and should be ignored. -append_eos: Whether to append at the end and treat it as a non-padded - label. -maxlen: an integer, sequence length of the output tensors. -pad_to_maxlen: Whether to pad the output to maxlen. -vocab_filepath: a string, filepath to the vocab file. -load_token_ids_from_vocab: Whether token ids are present in vocab (i.e. vocab - contains two colums, one for IDs and one for words). If false, line numbers - are used. -delimiter: The delimiter to split the labels to tokens by. -)doc"); - -REGISTER_OP("IdToAscii") - .Input("token_ids: int32") - .Input("seq_lengths: int32") - .Output("sequence: string") - .SetShapeFn([](shape_inference::InferenceContext* c) { - c->set_output(0, c->Vector(c->Dim(c->input(0), 0))); - return Status::OK(); - }) - .Doc(R"doc( -Converts sequences from token ids to actual ASCII tokens. - -token_ids: A matrix of shape [batch, seq_len]. -seq_lengths: A vector of shape [batch]. seq_lengths[i] is the length of the - i-th sequence. Only the first seq_lengths[i] tokens in token_ids[i] are - valid tokens for the i-th sequence. -sequence: A vector of shape [batch]. The converted string sequence. -)doc"); - -REGISTER_OP("NgramIdToToken") - .Input("token_ids: int32") - .Input("seq_lengths: int32") - .Output("sequences: string") - .SetShapeFn([](shape_inference::InferenceContext* c) { - c->set_output(0, c->input(1)); - return Status::OK(); - }) - .Attr("ngram_vocab_filepath: string") - .Attr("ngram_separator: string = \"\"") - .Doc(R"doc( -Converts sequences from token ids to actual tokens. - -token_ids: A matrix of shape [batch, seq_len]. -seq_lengths: A vector of shape [batch]. seq_lengths[i] is the length of the - i-th sequence. Only the first seq_lengths[i] tokens in token_ids[i] are - valid tokens for the i-th sequence. -sequences: A vector of shape [batch]. The converted string sequence. -ngram_vocab_filepath: filepath to the ngram vocab file. -ngram_separator: separator to use when joining ngrams into string. -)doc"); - -REGISTER_OP("BpeWordsToIds") - .Input("labels: string") - .Output("token_ids: int32") - .Output("target_ids: int32") - .Output("paddings: float") - .Attr("append_eos: bool = true") - .Attr("maxlen: int = 300") - .Attr("sos_id: int = 1") - .Attr("eos_id: int = 2") - .Attr("tokenization_filepath: string") - .SetShapeFn([](shape_inference::InferenceContext* ctx) { - const auto batch_size = ctx->Dim(ctx->input(0), 0); - int maxlen; - TF_RETURN_IF_ERROR(ctx->GetAttr("maxlen", &maxlen)); - ctx->set_output(0, ctx->Matrix(batch_size, maxlen)); - ctx->set_output(1, ctx->Matrix(batch_size, maxlen)); - ctx->set_output(2, ctx->Matrix(batch_size, maxlen)); - return Status::OK(); - }) - .Doc(R"doc( -A tokenizer to convert string to BPE ids. - -This op is especially convenient for mapping string text to a sequenec of word BPE -ids. The `labels` strings are tokenized via BPE. This op is typically used in conjunction with BpeIdsToWords. -As the vocabulary file it receives the mapping from each word to the series of BPE ids. -An example of lines in the vocabulary file: -... -AARON 10005,16,29 -AARON'S 10005,16,3466 -AARONSON 10005,16,17,447 -... - -The output tensor `token_ids` is a sequence of integer ids with prepended. -The output tensor `target_ids` is a sequence of integer ids with appended. - -labels: The batch of tf.String tensors. Expected shape is [batch_size]. -token_ids: - The ids with . The shape is [batch_size, maxlen]. -target_ids: - The ids with . The shape is [batch_size, maxlen]. -paddings: - The paddings. The shape is [batch_size, maxlen]. -maxlen: Maximum length of token_ids/target_ids/paddings. -tokenization_filepath: A path to a text file where each line is a word separated with space form a list of ids which are separated by ','. -)doc"); - -REGISTER_OP("BpeIdsToWords") - .Input("token_ids: int32") - .Input("seq_lengths: int32") - .Output("sequences: string") - .Attr("vocab_filepath: string") - .SetShapeFn([](shape_inference::InferenceContext* ctx) { - const auto batch_size = ctx->Dim(ctx->input(0), 0); - ctx->set_output(0, ctx->Vector(batch_size)); - return Status::OK(); - }) - .Doc(R"doc( -A tokenizer to map BPE ids to strings. - -This op is to map a sequence of integer ids to a string. The op is typically -used in conjunction with BpeWordsToIds. - -The op will consume `seq_lengths` of tokens from `token_ids` and convert it to -string `sequences`. A space character will be interested inbetween tokens. We do -not filter any tokens (i.e., and are not treated specially). - -token_ids: The ids (can include paddings; length is determined by seq_lengths). The shape is [batch_size, maxlen]. -seq_lengths: The length of the ids. The shape is [batch_size]. -sequences: The string sequences. The shape is [batch_size]. -vocab_filepath: A path to a text file where each line is a BPE string token. -)doc"); - -REGISTER_OP("GenericInput") - .Output("out: out_types") - .INPUT_ATTRS // Common input attributes. - .Attr("out_types: list(type)") - .Attr("processor: func") - .Attr("dynamic_padding_dimensions: list(int) = []") - .Attr("dynamic_padding_constants: list(int) = []") - .Doc(R"doc( -Produces examples from processed from records. - -out: The 1st dimension of every tensor is the batch dimension. -)doc" INPUT_DOCS - R"doc( -out_types: A list of tensor types. -processor: A function that processes a string (one record) and returns - a list of tensors. The last tensor must be a int32 scalar, which is - used in conjunction with `bucket_upper_bound` to bucket all the - samples. The other tensors belongs to one sample. They have the - respective `out_types`. These tensors' first dimension are _not_ the - batch dimension. Instead, when multiple samples are merged into a - batch, GenericInput's implementation expand the batch dimension (dim - 0) and concatenate the corresponding tensors into one tensor. -dynamic_padding_dimensions: If not empty, must be the same length as out. - Specifies the 0-indexed dimension to pad dynamically for each output. - The output is padded to the longest tensor in the batch along the dimension. - The first (0-th) dimension is _not_ the batch dimension. A value of -1 - indicates the specified output should not be padded, eg. if the output is a - scalar rather than a sequence. -dynamic_padding_constants: Must be set if `dynamic_padding_dimension` is - provided. The constant value to use for padding. -)doc"); - -REGISTER_OP("StaticMapStringInt") - .Input("x: string") - .Output("y: int32") - .Attr("keys: list(string)") - .Attr("vals: list(int) = []") - .Attr("unk: int = -1") - .SetShapeFn([](shape_inference::InferenceContext* c) { - c->set_output(0, c->input(0)); - return ::tensorflow::Status::OK(); - }) - .Doc(R"doc( -Maps every element of x according a static mapping. - -x: A Tensor of type string. -y: A Tensor of type int32. Same shape of x. -keys: The list of keys. -vals: The list of values. If empty, defaults to [0 .. len(keys)). -unk: The value when the key is not found. -)doc"); - -REGISTER_OP("StaticMapIntString") - .Input("x: int32") - .Output("y: string") - .Attr("keys: list(int) = []") - .Attr("vals: list(string)") - .Attr("unk: string = ''") - .SetShapeFn([](shape_inference::InferenceContext* c) { - c->set_output(0, c->input(0)); - return ::tensorflow::Status::OK(); - }) - .Doc(R"doc( -Maps every element of x according a static mapping. - -x: A Tensor of type int32. -y: A Tensor of type string. Same shape of x. -keys: The list of keys. If empty, defaults to [0 .. len(keys)). -vals: The list of values. -unk: The value when the key is not found. -)doc"); - -REGISTER_OP("StaticMapIntInt") - .Input("x: int32") - .Output("y: int32") - .Attr("keys: list(int) = []") - .Attr("vals: list(int)") - .Attr("unk: int = -1") - .SetShapeFn([](shape_inference::InferenceContext* c) { - c->set_output(0, c->input(0)); - return ::tensorflow::Status::OK(); - }) - .Doc(R"doc( -Maps every element of x according a static mapping. - -x: A Tensor of type int32. -y: A Tensor of type int32. Same shape of x. -keys: The list of keys. If empty, defaults to [0 .. len(keys)). -vals: The list of values. -unk: The value when the key is not found. -)doc"); - -REGISTER_OP("ComputePreconditioners") - .Input("inputs: num_tensors * float32") - .Input("exponents: num_tensors * float32") - .Input("global_step: int32") - .Attr("preconditioner_compute_graphdef: string") - .Attr("keys: list(string)") - .Attr("sync: bool = false") - .Attr("num_tensors: int >= 1") - .SetIsStateful() - .SetShapeFn([](shape_inference::InferenceContext* c) { - return Status::OK(); - }) - .Doc(R"doc( -Compute preconditioners for Shampoo optimizer. - -inputs: A list of Tensors of type float32, of statistic matrices. -exponents: A list of scalar Tensors of type float32, exponent for matrix power. -global_step: A scalar Tensor of type int32 which indicates the global step. -preconditioner_compute_graphdef: A graphdef which indicates the function to run. -keys: A list of keys indicating the name of preconditioners. -sync: Boolean indicating whether to run preconditioning in synchronous mode. -num_tensors: Number of tensor inputs. -)doc"); - - -REGISTER_OP("GetPreconditioners") - .Input("shapes: num_tensors * Tshape") - .Output("outputs: num_tensors * float32") - .Output("statuses: num_tensors * bool") - .Attr("preconditioner_compute_graphdef: string") - .Attr("keys: list(string)") - .Attr("Tshape: {int32, int64} = DT_INT32") - .Attr("num_tensors: int >= 1") - .SetIsStateful() - .SetShapeFn([](shape_inference::InferenceContext* c) { - std::vector shapes; - if (c->input("shapes", &shapes).ok()) { - for (int i = 0; i < shapes.size(); ++i) { - shape_inference::ShapeHandle out; - TF_RETURN_IF_ERROR(c->MakeShapeFromShapeTensor(i, &out)); - c->set_output(i, out); - c->set_output(shapes.size() + i, c->Scalar()); - } - } - return Status::OK(); - }) - .Doc(R"doc( -Get preconditioners for Shampoo optimizer. - -shapes: A list of Tensors of type Tshape indicating the size of preconditioner. -outputs: A list of Tensors of type float32 which are the preconditioners. -statuses: A list of Tensors of type bool which are the preconditioner status. -preconditioner_compute_graphdef: A graphdef which indicates the function to run. -keys: A list of keys indicating the name of preconditioners. -Tshape: The data-type to use for shape. -num_tensors: Number of tensor inputs. -)doc"); - -REGISTER_OP("MlPerfSubwordIdToString") - .Input("token_ids: int32") - .Input("seq_lengths: int32") - .Output("sequences: string") - .SetShapeFn([](shape_inference::InferenceContext* c) { - c->set_output(0, c->input(1)); - return Status::OK(); - }) - .Attr("vocab_filepath: string") - .Doc(R"doc( -Converts sequences from subword token ids to strings - -token_ids: A matrix of shape [batch, seq_len]. -seq_lengths: A vector of shape [batch]. seq_lengths[i] is the length of the - i-th sequence. Only the first seq_lengths[i] tokens in token_ids[i] are - valid tokens for the i-th sequence. -sequences: A vector of shape [batch]. The converted string sequence. -vocab_filepath: filepath to the MLPerf subword vocab file. -)doc"); - -REGISTER_OP("PackSequences") - .Input("src_actual_seq_len: int32") - .Input("tgt_actual_seq_len: int32") - .Attr("packed_batch_size: int") - .Attr("packed_src_seq_len: int") - .Attr("packed_tgt_seq_len: int") - .SetIsStateful() // TODO(navari): disable when packed_batch_size==0? - .Output("src_segment_ids: int32") - .Output("src_segment_pos: int32") - .Output("src_indices_in_input: int32") - .Output("tgt_segment_ids: int32") - .Output("tgt_segment_pos: int32") - .Output("tgt_indices_in_input: int32") - .SetShapeFn([](shape_inference::InferenceContext* c) { - int packed_batch_size, packed_src_seq_len, packed_tgt_seq_len; - TF_RETURN_IF_ERROR(c->GetAttr("packed_batch_size", &packed_batch_size)); - shape_inference::DimensionOrConstant batch_dim = c->UnknownDim(); - if (packed_batch_size > 0) { - batch_dim = packed_batch_size; - } - TF_RETURN_IF_ERROR(c->GetAttr("packed_src_seq_len", &packed_src_seq_len)); - TF_RETURN_IF_ERROR(c->GetAttr("packed_tgt_seq_len", &packed_tgt_seq_len)); - c->set_output(0, c->Matrix(batch_dim, packed_src_seq_len)); - c->set_output(1, c->Matrix(batch_dim, packed_src_seq_len)); - c->set_output(2, c->Matrix(batch_dim, packed_src_seq_len)); - c->set_output(3, c->Matrix(batch_dim, packed_tgt_seq_len)); - c->set_output(4, c->Matrix(batch_dim, packed_tgt_seq_len)); - c->set_output(5, c->Matrix(batch_dim, packed_tgt_seq_len)); - return Status::OK(); - }) - .Attr("seed: int = 0") - .Doc(R"doc( -Produces a packing pattern for the (src, tgt) input pair with the provided -lengths, according to the given packed shape. - -For example, the following input:: - - src_actual_seq_len = [3, 2, 1] - tgt_actual_seq_len = [4, 1, 5] - packed_batch_size = 2 - packed_src_seq_len = 5 - packed_tgt_seq_len = 5 - -will result in:: - - src_segment_ids = [ [1, 1, 1, 2, 2], [1, 0, 0, 0, 0] ] - src_segment_pos = [ [0, 1, 2, 0, 1], [0, 0, 0, 0, 0] ] - src_indices_in_input = [ [0, 0, 0, 1, 1], [2, 0, 0, 0, 0] ] - tgt_segment_ids = [ [1, 1, 1, 1, 2], [1, 1, 1, 1, 1] ] - tgt_segment_pos = [ [0, 1, 2, 3, 0], [0, 1, 2, 3, 4] ] - tgt_indices_in_input = [ [0, 0, 0, 0, 1], [2, 2, 2, 2, 2] ] - -The packed sequence length can be different between src and tgt. For example, -the following input:: - - src_actual_seq_len = [3, 2, 1] - tgt_actual_seq_len = [4, 1, 5] - packed_batch_size = 2 - packed_src_seq_len = 4 - packed_tgt_seq_len = 6 - -will result in:: - - src_segment_ids = [ [1, 1, 1, 0], [1, 1, 2, 0] ] - src_segment_pos = [ [0, 1, 2, 0], [0, 1, 0, 0] ] - src_indices_in_input = [ [0, 0, 0, 0], [1, 1, 2, 0] ] - tgt_segment_ids = [ [1, 1, 1, 1, 0, 0], [1, 2, 2, 2, 2, 2] ] - tgt_segment_pos = [ [0, 1, 2, 3, 0, 0], [0, 0, 1, 2, 3, 4] ] - tgt_indices_in_input = [ [0, 0, 0, 0, 0, 0], [1, 2, 2, 2, 2, 2] ] - -If there are too few input sequences to pack into `output_shape`, the op pads -the remaining elements in the output. - -If there are too many input sequences to pack into `output_shape`, the op drops -input sequences. The dropping is done randomly uniformly on the input sequences -to not bias the distribution of sequence lengths in the packed output. - -src_actual_seq_len: A tensor of shape [N], where N is the input batch size. - This tensor contains the actual lengths for the src sequence. -tgt_actual_seq_len: A tensor of shape [N], where N is the input batch size. - This tensor contains the actual lengths for the tgt sequence. -packed_batch_size: A scalar. The output batch size. The packed output will - be of shape [packed_batch_size, packed_{src,tgt}_seq_len] for src and tgt, - respectively. if this value is set to 0, output will be of variable batch - size, determined by the number of row needed to pack all given inputs. -packed_src_seq_len: A scalar. The output sequence length for src. A src input - with shape [N, src_input_seq_len] will be packed into an output with shape - [packed_batch_size, packed_src_seq_len]. -packed_tgt_seq_len: A scalar. The output sequence length for tgt. A tgt input - with shape [N, tgt_input_seq_len] will be packed into an output with shape - [packed_batch_size, packed_tgt_seq_len]. -src_segment_ids: - A tensor of shape [packed_batch_size, packed_src_seq_len]. Incrementing from 1 - to indicate each segment in the packed output for src. Zero is reserved for - indicating padding at the end of each row. -tgt_segment_ids: - A tensor of shape [packed_batch_size, packed_tgt_seq_len]. Incrementing from 1 - to indicate each segment in the packed output for tgt. Zero is reserved for - indicating padding at the end of each row. -src_segment_pos: - A tensor of shape [packed_batch_size, packed_src_seq_len]. Zero-based index to - indicate relative position within each segment for src. Zero is also used to - indicate padding. When needed, use `src_segment_ids` to disambiguate. -tgt_segment_pos: - A tensor of shape [packed_batch_size, packed_tgt_seq_len]. Zero-based index to - indicate relative position within each segment for tgt. Zero is also used to - indicate padding. When needed, use `tgt_segment_ids` to disambiguate. -src_indices_in_input: - A tensor of shape [packed_batch_size, packed_src_seq_len]. For each segment in - the packed output, it contains the original (zero-based) row index of each - segment found in `src_actual_seq_len`. Zero is also used to indicate padding. - When needed, use `src_segment_ids` to disambiguate. -tgt_indices_in_input: - A tensor of shape [packed_batch_size, packed_tgt_seq_len]. For each segment in - the packed output, it contains the original (zero-based) row index of each - segment found in `tgt_actual_seq_len`. Zero is also used to indicate padding. - When needed, use `tgt_segment_ids` to disambiguate. -seed: Seed for random number generator, which is used when we need to drop - excessive input sequences. If seed is zero, use completely random seed. -)doc"); - -REGISTER_OP("ApplyPacking") - .Input("input: T") - .Input("padding: T") - .Input("segment_ids: int32") - .Input("indices_in_input: int32") - .Output("output: T") - .SetShapeFn([](shape_inference::InferenceContext* c) { - DataType dtype; - TF_RETURN_IF_ERROR(c->GetAttr("T", &dtype)); - if (dtype == DT_STRING) { - const auto batch_size = c->Dim(c->input(2), 0); - c->set_output(0, c->Vector(batch_size)); - } else { - c->set_output(0, c->input(2)); - } - return Status::OK(); - }) - .Attr("T: type") - .Doc(R"doc( -Applies a packing pattern on the input to obtain a packed output. - -A slightly different semantics when T is tf.string type: the output joins the -strings that are packed on the same row, separated by `padding`. - -The inputs `segment_ids` and `indices_in_input` can be obtained from the outputs -of an `PackSequence` op (though only the src or the tgt tensors are needed). - -Note that ApplyPacking is done on a per column basis (either on the src or on -the tgt), as opposed to in PackSequences, when both src and tgt columns must be -processed together within the same op. - -input: A tensor of shape [N, seq_len]. The input to apply the packing to. - For tf.string typed input, a vector of shape [N] is expected. -padding: A scalar to indicate the padding value. This is typically the zero - value of T, but may not always be the case, e.g. when the input is a paddings - tensor, in which case caller should set padding=1. - For tf.string typed input, padding is used as a separator to join all the - strings on the same row in the output. -segment_ids: A rank 2 tensor of shape `output_shape`. -indices_in_input: A rank 2 tensor of shape `output_shape`. - -output: - A tensor of shape `output_shape`. For tf.string typed input, the output - is a vector of strings where its length is the same as the number of rows in - `output_shape`. -)doc"); - -REGISTER_OP("Mass") - .Input("ids: int32") - .Input("weights: float32") - .Input("actual_seq_len: int32") - .Attr("mask_id: int") - .Attr("mask_ratio: float = 0.5") - .Attr("mask_minlen: int = 0") - .Attr("span_len: int = 100000") - .Attr("random_start_prob: float = 0.6") - .Attr("keep_prob: float = 0.1") - .Attr("rand_prob: float = 0.1") - .Attr("mask_prob: float = 0.8") - // TODO(alisonlui): This flag is rarely used; remove after verification. - .Attr("mask_target: bool = True") - .Attr("vocab_size: int") - .Attr("first_unreserved_id: int = 4") - .Output("src_ids: int32") - .Output("tgt_ids: int32") - .Output("tgt_labels: int32") - .Output("tgt_weights: float32") - .Doc(R"doc( -Applies masking to implement MASS. - -ids: Tensor of shape [batch_size, max_seq_len] containing the token ids. - Should include EOS token . -weights: Tensor of shape [batch_size, max_seq_len]. -actual_seq_len: Tensor of shape [batch_size]. - -mask_id: The id to use for the mask token. -mask_ratio: Proportion of src to mask. -mask_minlen: Skip sentences too short to mask at least this many tokens. -span_len: Split mask_len into segments of this size and randomly distribute -those across the src. -random_start_prob: The probability that the placement of masked segments will be - entirely random. The remaining cases are split evenly between masking at the - beginning and at the end of the src. -keep_prob: The probability that a token to be masked will be unchanged. - `keep_prob + rand_prob + mask_prob` must sum to 1. -rand_prob: The probability that a token to be masked will be replaced with a - random token in the vocab. `keep_prob + rand_prob + mask_prob` must sum to 1. -mask_prob: The probability that a token to be masked will be replaced with the - mask_id. `keep_prob + rand_prob + mask_prob` must sum to 1. -mask_target: whether to mask the target (the mask will be the inverse of that of - the src). -vocab_size: Vocab size used when selecting a random token to replace a masked - token. -first_unreserved_id: Tokens greater than or equal to this may be selected at - random to replace a masked token. - -src_ids: - Masked ids. E.g. `s1 s2 s3 m m ` -tgt_ids: - Right-shifted ids with BOS token added, where the mask is the - positional inverse of that of the source unless mask_target=False. - E.g. `m m m s3 s4 m` -tgt_labels: - E.g. `s1 s2 s3 s4 s5 ` -tgt_weights: - weights are zeroed wherever the target is masked. -)doc"); - -} // namespace -} // namespace tensorflow diff --git a/lingvo/core/ops/x_ops_helper.h b/lingvo/core/ops/x_ops_helper.h deleted file mode 100644 index b6552b5d..00000000 --- a/lingvo/core/ops/x_ops_helper.h +++ /dev/null @@ -1,75 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#ifndef LINGVO_CORE_OPS_X_OPS_HELPER_H_ -#define LINGVO_CORE_OPS_X_OPS_HELPER_H_ - -#define INPUT_ATTRS \ - Output("bucket_keys: int32") \ - .Attr("file_pattern: string") \ - .Attr("input_source_weights: list(float) = []") \ - .Attr("file_random_seed: int = 301") \ - .Attr("file_buffer_size: int = 10000") \ - .Attr("file_buffer_size_in_seconds: int = 0") \ - .Attr("file_parallelism: int = 16") \ - .Attr("bucket_upper_bound: list(int)") \ - .Attr("bucket_batch_limit: list(int)") \ - .Attr("bucket_adjust_every_n: int = 0") \ - .Attr("flush_every_n: int = 0") \ - .Attr("num_threads: int = 1") \ - .Attr("require_sequential_order: bool = False") \ - .Attr("repeat_count: int = -1") \ - .Attr("use_chaining: bool = False") \ - .SetIsStateful() - -#define INPUT_DOCS \ - R"( \ -file_pattern: A comma-separated list of glob patterns or sharded file patterns\ - for the data files. A sharded file pattern looks like /path/name@100 or \ - /path/name@*.\ -input_source_weights: A list of input sources weights that control the input\ - example mix. The records will be sampled from inputs proportionally to these\ - weights. When empty list is provided, no mix weighting will be done.\ - Defaults to empty list.\ -file_random_seed: Random seeds used to produce randomized records.\ -file_buffer_size: The randomization shuffling buffer.\ -file_buffer_size_in_seconds: Number of records the shuffling buffer should\ - contain, measured in seconds (the number of records demanded by the trainer\ - in this many seconds).\ -file_parallelism: How many sstables are opened and concurrently iterated over.\ -bucket_upper_bound: Bucketing scheme. Specifies each bucket's upper bound.\ -bucket_batch_limit: Batching scheme. Specifies each bucket's maximum batch\ - size.\ -bucket_adjust_every_n: If non-zero, optimize the values of bucket_upper_bound\ - except the last one after every N records based on the current input length\ - distribution.\ -flush_every_n: If non-zero, flushes all batches buffered so far every these\ - many records are yielded.\ -num_threads: Number of threads to use for the record batcher. Each thread\ - fills separate batches based on bucket limits.\ -require_sequential_order: If true, the input op is required to process the file\ - glob as well as the contents of each file in a deterministic sequential order.\ - Setting this automatically disables file_random_seed, file_buffer_size,\ - file_parallelism, num_threads, and requires a single file_pattern.\ -repeat_count: Number of repetitions of a dataset before throwing OutOfRange\ - error when using require_sequential_order. Must only be set if\ - require_sequential_order is True.)\ -use_chaining: If true, the input op is outputing records from file patterns in \ - order. That is, first all records from first file pattern will be yielded, \ - then all records from the second file pattern and so on. If false, the \ - records from different file patterns will be mixed.\ -)" - -#endif // LINGVO_CORE_OPS_X_OPS_HELPER_H_ diff --git a/lingvo/core/ops/yielder_test_helper.cc b/lingvo/core/ops/yielder_test_helper.cc deleted file mode 100644 index 7ff994e8..00000000 --- a/lingvo/core/ops/yielder_test_helper.cc +++ /dev/null @@ -1,85 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ -#include "lingvo/core/ops/yielder_test_helper.h" - -#include "tensorflow/core/lib/io/path.h" -#include "tensorflow/core/lib/strings/stringprintf.h" - -namespace tensorflow { -namespace lingvo { - -void GeneratePlainTextTestData(const string& prefix, int n, int m) { - for (int i = 0; i < n; ++i) { - std::unique_ptr file; - TF_CHECK_OK(Env::Default()->NewWritableFile( - io::JoinPath("/tmp", strings::StrCat(prefix, ".", i)), - &file)); - for (int j = 0; j < m; ++j) { - TF_CHECK_OK(file->Append( - strings::Printf("%s:%010d\n", prefix.c_str(), m * i + j))); - } - } -} - -void GenerateCheckpointPlainTextTestData(const string& prefix, int m) { - std::unique_ptr ckpt_file; - std::unique_ptr data_file; - TF_CHECK_OK(Env::Default()->NewWritableFile( - io::JoinPath("/tmp", prefix), &ckpt_file)); - TF_CHECK_OK(ckpt_file->Append( - strings::Printf("current: {file_pattern:\"data-0.txt\"}"))); - - string data_file_path = io::JoinPath("/tmp", "data-0.txt"); - TF_CHECK_OK(Env::Default()->NewWritableFile(data_file_path, &data_file)); - for (int j = 0; j < m; ++j) { - TF_CHECK_OK(data_file->Append( - strings::Printf("%s:%010d\n", prefix.c_str(), m + j))); - } -} - -void UpdateCheckpointPlainTextTestData(const string& prefix, int m) { - std::unique_ptr ckpt_file; - std::unique_ptr data_file; - TF_CHECK_OK(Env::Default()->NewWritableFile( - io::JoinPath("/tmp", prefix), &ckpt_file)); - TF_CHECK_OK(ckpt_file->Append( - strings::Printf("current: {file_pattern:\"data-1.txt\"}"))); - - string data_file_path = io::JoinPath("/tmp", "data-1.txt"); - TF_CHECK_OK(Env::Default()->NewWritableFile(data_file_path, &data_file)); - for (int j = 0; j < m; ++j) { - TF_CHECK_OK(data_file->Append( - strings::Printf("%s:%010d\n", prefix.c_str(), 2 * m + j))); - } -} - -std::unordered_map ComputeInputSourceDistribution( - const std::vector& vals) { - std::unordered_map input_source_distribution; - for (const string& val : vals) { - const auto prefix_end = val.find(':'); - if (prefix_end != string::npos) { - input_source_distribution[val.substr(0, prefix_end)] += 1.0; - } - } - for (auto it = input_source_distribution.begin(); - it != input_source_distribution.end(); ++it) { - it->second /= vals.size(); - } - return input_source_distribution; -} - -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/core/ops/yielder_test_helper.h b/lingvo/core/ops/yielder_test_helper.h deleted file mode 100644 index f85e1981..00000000 --- a/lingvo/core/ops/yielder_test_helper.h +++ /dev/null @@ -1,48 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ -#ifndef LINGVO_CORE_OPS_YIELDER_TEST_HELPER_H_ -#define LINGVO_CORE_OPS_YIELDER_TEST_HELPER_H_ - -#include -#include -#include "lingvo/core/ops/record_yielder.h" - -namespace tensorflow { -namespace lingvo { - -class MockRecordYielder : public BasicRecordYielder { - public: - MockRecordYielder() : BasicRecordYielder() {} - MOCK_METHOD1(Yield, Status(Record* record)); - MOCK_METHOD0(Close, void()); - MOCK_CONST_METHOD0(current_epoch, int64()); -}; - -// Generates n plain text files with m lines each. -void GeneratePlainTextTestData(const string& prefix, int n, int m); - -// Generates checkpoint file and data file for test. -void GenerateCheckpointPlainTextTestData(const string& prefix, int m); -void UpdateCheckpointPlainTextTestData(const string& prefix, int m); - -// Computes input source distribution of lines read from plain text data -// generated by GeneratePlainTextTestData function. -std::unordered_map ComputeInputSourceDistribution( - const std::vector& vals); - -} // namespace lingvo -} // namespace tensorflow - -#endif // LINGVO_CORE_OPS_YIELDER_TEST_HELPER_H_ diff --git a/lingvo/core/optimizer.py b/lingvo/core/optimizer.py deleted file mode 100644 index ad091d18..00000000 --- a/lingvo/core/optimizer.py +++ /dev/null @@ -1,605 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Optimizers.""" - -import re - -import lingvo.compat as tf -from lingvo.core import adagraft -from lingvo.core import base_layer -from lingvo.core import distributed_shampoo -from lingvo.core import py_utils -from lingvo.core import summary_utils - - -class Base(base_layer.BaseLayer): - """Base class for all optimizers.""" - - @classmethod - def Params(cls): - p = super().Params() - p.name = cls.__name__ - p.Define( - 'use_bf16_gradients_ar', False, - 'Whether to use bfloat16 dtype for gradients all-reduce. ' - 'This applies to TPU only.') - return p - - def GetOptimizer(self, lr): - """Returns the TF optimizer object.""" - raise NotImplementedError('Abstract method') - - def AddSummary(self, lr, optimizer, var_grad): - """Adds summary if needed.""" - pass - - def ComputeGradients(self, loss, vmap, *args, **kwargs): - """Allows subclasses control computation of gradients.""" - kwargs['use_bf16_gradients_ar'] = self.params.use_bf16_gradients_ar - return py_utils.ComputeGradients(loss, vmap, *args, **kwargs) - - def VarReuseForSlotVars(self): - """Multi-task models require AUTO_REUSE for var sharing.""" - var_reuse = False - if py_utils.GetOpportunisticVariableReuse(): - var_reuse = tf.AUTO_REUSE - return var_reuse - - def Apply(self, lr, var_grad): - """Applies the gradient to the variable. - - Args: - lr: A scalar. The base learning rate. - var_grad: A `.NestedMap` of (var, grad) pairs. - - Returns: - The variable update op. - """ - optimizer = self.GetOptimizer(lr) - - def _Apply(): - if self.params.use_bf16_gradients_ar: - return optimizer.apply_gradients( - [(tf.cast(g, tf.float32), v) for (v, g) in var_grad.Flatten()], - name='meta_backprop') - else: - return optimizer.apply_gradients( - [(g, v) for (v, g) in var_grad.Flatten()], name='meta_backprop') - - if not py_utils.use_resource_variables(): - var_update_op = _Apply() - else: - # Many optimizers, e.g., Adam, Adagrad, etc., create - # variables. We need to ensure name scope and variable scope are - # cleared. Otherwise, tpu.batch_parallel does not work. - with tf.name_scope(None): - with tf.variable_scope( - tf.VariableScope( - use_resource=True, reuse=self.VarReuseForSlotVars())): - var_update_op = _Apply() - self.AddSummary(lr, optimizer, var_grad) - return var_update_op - - def ApplyPostTrainingLoop(self, global_step): - """Applies any computation to run after each tpu trainining loop. - - Args: - global_step: Global step variable. - - Returns: - Ops to run after training loop ends. - """ - return tf.no_op() - - -class CompositeOptimizer(Base): - """Composite Optimizer. - - A composite optimizer is composed of one or more Lingvo Optimizer objects - where regex specifies which variables should use which optimizer. The - optimizer_map dictionary must specify a default_optimizer regex to a - (Lingvo Optimizer, learning rate) tuple which will be applied to all variables - which do not match an earlier regex. - - For example, - - optimizer_map = {'a': Adam, 'b': Adagrad, 'default_optimizer': SGD} - - will apply Adam to all variables which contain an 'a' in their name, apply - Adagrad to all variables which contain a 'b' in their name, and apply SGD to - the variables which do not contain either 'a' or 'b'. - - If a non-default_optimizer matches more than one variable -- in this example - variables with both 'a' and 'b' in their name -- an exception is thrown. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'optimizer_map', None, - 'Mapping of variable regex to (Lingvo Optimizer, learning rate) tuple.') - return p - - def __init__(self, params): - super().__init__(params) - self._optimizer_map = {} - self._lr_map = {} - for index, regex in enumerate(params.optimizer_map): - sub_optimizer, learning_rate = params.optimizer_map[regex] - self.CreateChild('sub_{}_{}'.format(sub_optimizer.name, index), - sub_optimizer) - self._optimizer_map[regex] = self.children['sub_{}_{}'.format( - sub_optimizer.name, index)] - self._lr_map[regex] = learning_rate - - if 'default_optimizer' not in self._optimizer_map: - raise KeyError('default_optimizer is not found in optimizer_map. Please ' - 'specify a default_optimizer regex and its associated ' - '(Lingvo Optimizer, learning rate) tuple.') - - def GetOptimizer(self, lr): - """Returns a dictionary of regex to TF optimizer objects.""" - return { - k: v.GetOptimizer(self._lr_map[k]) - for k, v in self._optimizer_map.items() - } - - def Apply(self, lr, var_grad): - """For each optimizer, apply the gradient to the variable. - - Args: - lr: A scalar. The base learning rate. - var_grad: A `.NestedMap` of (var, grad) pairs. - - Returns: - The variable update op. - - Raises: - Exception: When the regex overlaps with or does not cover all variables. - """ - # Override inherited GetOptimizer even though learning rate is unused. - tf_optimizer_map = self.GetOptimizer(0) - var_grad_map = {regex: [] for regex in self._optimizer_map} - - for (v, g) in var_grad.Flatten(): - regex_match = 0 - for regex in self._optimizer_map: - if re.match(regex, v.name): - var_grad_map[regex].append((g, v)) - regex_match += 1 - if regex_match == 0: - var_grad_map['default_optimizer'].append((g, v)) - if regex_match > 1: - raise Exception('Variable {} is matched {} times by regex {}'.format( - v.name, regex_match, list(self._optimizer_map.keys()))) - - def _Apply(): - """Use the matched optimizer to apply the gradients.""" - train_ops = [] - non_default_regex = [ - regex for regex in self._optimizer_map if regex != 'default_optimizer' - ] - for regex in self._optimizer_map: - if var_grad_map[regex]: - opt = tf_optimizer_map[regex] - train_ops.append(opt.apply_gradients(var_grad_map[regex])) - # pylint: disable=cell-var-from-loop, g-long-lambda - if regex == 'default_optimizer': - filtered_var_grad = var_grad.FilterKeyVal(lambda k, v: any( - [re.match(i, v.var.name) for i in non_default_regex])) - else: - filtered_var_grad = var_grad.FilterKeyVal( - lambda k, v: (re.match(regex, v.var.name))) - # pylint: enable=cell-var-from-loop, g-long-lambda - self._optimizer_map[regex].AddSummary(self._lr_map[regex], opt, - filtered_var_grad) - return tf.group(*train_ops, name='composite_optimizer_train_op') - - if not py_utils.use_resource_variables(): - var_update_op = _Apply() - else: - # Many optimizers, e.g., Adam, Adagrad, etc., create - # variables. We need to ensure name scope and variable scope are - # cleared. Otherwise, tpu.batch_parallel does not work. - var_reuse = False - if py_utils.GetOpportunisticVariableReuse(): - var_reuse = tf.AUTO_REUSE - with tf.name_scope(None): - with tf.variable_scope( - tf.VariableScope(use_resource=True, reuse=var_reuse)): - var_update_op = _Apply() - return var_update_op - - def ApplyPostTrainingLoop(self, global_step): - """Apply any computation to run after each tpu training loop for each optimizer. - - Args: - global_step: Global step variable. - - Returns: - Ops to run after training loop ends. - """ - post_training_ops = [ - opt.ApplyPostTrainingLoop(global_step) - for _, opt in self._optimizer_map.items() - ] - return tf.group(*post_training_ops) - - -class SGD(Base): - """SGD.""" - - def GetOptimizer(self, lr): - return tf.train.GradientDescentOptimizer(lr) - - def AddSummary(self, lr, optimizer, var_grad): - summary_utils.scalar('sgd_lr', lr) - - -class Momentum(Base): - """Momentum optimizer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'alpha', 0.9, 'The damping factor in the momentum ' - 'optimizer. This controls how the velocity (averaged ' - 'past gradients) is decayed over time.') - p.Define('use_nesterov', False, 'True iff use Nesterov') - return p - - def GetOptimizer(self, lr): - p = self.params - return tf.train.MomentumOptimizer( - learning_rate=lr, momentum=p.alpha, use_nesterov=p.use_nesterov) - - def AddSummary(self, lr, optimizer, var_grad): - summary_utils.scalar('momentum_lr', lr) - - -class RMSProp(Base): - """RMSProp optimizer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('decay', 0.9, 'Discounting factor for the history/coming gradient') - p.Define('momentum', 0.9, 'Momentum in RMSProp.') - p.Define( - 'epsilon', 1.0, - 'Epsilon term for RMSProp. Small value to avoid zero denominator.') - return p - - def GetOptimizer(self, lr): - p = self.params - return tf.train.RMSPropOptimizer( - lr, p.decay, momentum=p.momentum, epsilon=p.epsilon) - - def AddSummary(self, lr, optimizer, var_grad): - summary_utils.scalar('rmsprop_lr', lr) - - -class Adagrad(Base): - """Adagrad.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('initial_accumulator_value', 1.0, - "Adagrad's initial_accumulator_value.") - return p - - def GetOptimizer(self, lr): - p = self.params - return tf.train.AdagradOptimizer( - learning_rate=lr, initial_accumulator_value=p.initial_accumulator_value) - - def AddSummary(self, lr, optimizer, var_grad): - summary_utils.scalar('adagrad_lr', lr) - for v, _ in var_grad.Flatten(): - slot = optimizer.get_slot(v, 'accumulator') - assert slot is not None - summary_utils.scalar('optimizer/adagrad_accum_%s' % v.name, - tf.reduce_mean(slot)) - - -class AdaDelta(Base): - """AdaDelta optimizer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('decay', 0.95, - 'Discounting factor for the history/coming gradient') - p.Define( - 'epsilon', 1e-8, - 'Epsilon term for AdaDelta. Small value to avoid zero denominator.') - return p - - def GetOptimizer(self, lr): - p = self.params - return tf.train.AdadeltaOptimizer( - learning_rate=lr, rho=p.decay, epsilon=p.epsilon) - - def AddSummary(self, lr, optimizer, var_grad): - summary_utils.scalar('adadelta_lr', lr) - - -class Adam(Base): - """Adam.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('beta1', 0.9, 'Beta1 for Adam.') - p.Define('beta2', 0.999, 'Beta2 for Adam.') - p.Define('epsilon', 1e-6, 'Epsilon for Adam.') - p.name = 'Adam' - return p - - @classmethod - def ParamsA(cls): - """Convenient method for a commonly used Adam config.""" - return cls.Params().Set(beta1=0.9, beta2=0.997, epsilon=1e-9) - - @classmethod - def ParamsB(cls): - """Convenient method for another commonly used Adam config.""" - return cls.Params().Set(beta1=0.9, beta2=0.98, epsilon=1e-9) - - def GetOptimizer(self, lr): - p = self.params - return tf.train.AdamOptimizer( - learning_rate=lr, - beta1=p.beta1, - beta2=p.beta2, - epsilon=p.epsilon, - name=p.name) - - def AddSummary(self, lr, optimizer, var_grad): - summary_utils.scalar('adam_lr', lr) - - -class Accumulator(Base): - """Gradient accumulator wrapper.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('optimizer_tpl', Adam.Params(), - 'Params for the wrapped optimizer.') - p.Define( - 'accum_steps', 5, 'Number of gradient accumulation steps' - ' before invoking wrapped optimizer.') - p.name = 'Accumulator' - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self.CreateChild('_opt', p.optimizer_tpl) - - def Apply(self, lr, var_grad): - p = self.params - - def _Acc(vg): - """Updating accumulators.""" - - v, g = vg - with tf.variable_scope(v.op.name): - a = py_utils.CreateVariable( - 'grad_accumulator', - py_utils.WeightParams(v.get_shape(), - py_utils.WeightInit.Constant(0.0), - self.params.dtype), - trainable=False) - a = tf.assign_add(a, g) - - return py_utils.VarGrad(v, a) - - var_grad = var_grad.Transform(_Acc) - - def _ApplyAndReset(): - with tf.control_dependencies([ - self._opt.Apply( - lr, py_utils.ApplyGradMultiplier(var_grad, 1. / p.accum_steps)) - ]): - return tf.group( - *[tf.assign(a, tf.zeros_like(a)) for _, a in var_grad.Flatten()]) - - return tf.cond( - tf.equal( - tf.math.floormod(self.global_step, p.accum_steps), - p.accum_steps - 1), _ApplyAndReset, lambda: tf.group(tf.no_op())) - - def GetOptimizer(self, lr): - return self._opt.GetOptimizer(lr) - - def AddSummary(self, lr, optimizer, var_grad): - return self._opt.AddSummary(lr, optimizer, var_grad) - - -class DistributedShampoo(Base): - """Approximates full-matrix AdaGrad per layer. - - Approximates full-matrix AdaGrad with kronecker-products of two statistics - matrices based on only the first-order gradients of the layer. - - "Second-order optimization made practical.", 2019 - Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, Yoram Singer. - """ - - @classmethod - def Params(cls): - params = super().Params() - params.Define('momentum', 0.9, 'Momentum parameter.') - params.Define('start_preconditioning_steps', 1000, - 'When to start approximate full matrix preconditioning.') - params.Define('initial_accumulator_value', 0.0, - 'Initial accumulator value.') - params.Define('block_size', 4096, 'Block size for partitioning.') - params.Define('block_partition_threshold_size', 1000000, - 'Threshold for block partitioning.') - params.Define('max_any_dim', 8192, - 'max dimension before skipping preconditioning altogether.') - params.Define('matrix_epsilon', 1e-6, - 'Minimum eigen value used to improve the conditioning.') - params.Define( - 'second_moment_averaging', 1.0, - 'Averaging coefficient, with special case of (1.0) means sum ' - 'of squares while less than 1.0 is RMSProp style moving' - ' average modification.') - params.Define( - 'fallback_to_diagonal_dim', 4096, - 'If any dimension is larger than this value, the optimizer falls back' - ' to the diagonal.') - params.Define( - 'statistics_computation_frequency', 1, - 'How often to compute statistics. Greater than 1 speeds up training.') - return params - - def GetOptimizer(self, lr): - params = self.params - return distributed_shampoo.DistributedShampoo( - learning_rate=lr, - momentum=params.momentum, - start_preconditioning_steps=params.start_preconditioning_steps, - initial_accumulator_value=params.initial_accumulator_value, - matrix_epsilon=params.matrix_epsilon, - statistics_computation_frequency=( - params.statistics_computation_frequency), - second_moment_averaging=params.second_moment_averaging, - max_any_dim=params.max_any_dim, - block_size=params.block_size, - global_step=self.global_step) - - def Apply(self, lr, var_grad): - """Applies the gradient to the variable. - - Args: - lr: A scalar. The base learning rate. - var_grad: A `.NestedMap` of (var, grad) pairs. - - Returns: - The variable update op. - """ - self._optimizer = self.GetOptimizer(lr) - - def _Apply(): - return self._optimizer.apply_gradients( - [(g, v) for (v, g) in var_grad.Flatten()], name='meta_backprop') - - if not py_utils.use_resource_variables(): - var_update_op = _Apply() - else: - # Many optimizers, e.g., Adam, Adagrad, etc., create - # variables. We need to ensure name scope and variable scope are - # cleared. Otherwise, tpu.batch_parallel does not work. - with tf.name_scope(None): - with tf.variable_scope( - tf.VariableScope( - use_resource=True, reuse=self.VarReuseForSlotVars())): - var_update_op = _Apply() - self.AddSummary(lr, self._optimizer, var_grad) - return var_update_op - - def ApplyPostTrainingLoop(self, global_step): - """Applies any computation to run after each tpu trainining loop. - - Args: - global_step: Global step variable. - - Returns: - Ops to run after training loop ends. - """ - invoke_async_ops = self._optimizer.invoke_async_preconditioner_computation( - tf.cast(global_step, tf.int32)) - assign_ops = self._optimizer.assign_preconditioner_to_host_vars() - return tf.group(*[invoke_async_ops, assign_ops]) - - def AddSummary(self, lr, optimizer, var_grad): - summary_utils.scalar('distributed_shampoo', lr) - - -class AdaGraft(Base): - """Optimizer which combines step size and direction of two optimizers. - - - Disentangling Adaptive Gradient Methods from Learning Rates - Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, Cyril Zhang - https://arxiv.org/abs/2002.11803 - """ - - @classmethod - def Params(cls): - params = super().Params() - - params.Define('magnitude_optimizer', None, - 'Instantiated Optimizer layer providing the step size.') - params.Define('direction_optimizer', None, - 'Instantiated Optimizer layer providing the step direction.') - params.Define( - 'direction_optimizer_lr', None, - 'Custom constant learning rate passed to direction ' - 'optimizer. If None, then pass scheduled lr for both.') - params.Define('use_global_norm', False, 'Whether to graft global l2 norm.') - params.Define('diagnostic', False, 'Whether to record norm measurements.') - - params.name = 'AdaGraft' - return params - - def GetOptimizer(self, lr): - params = self.params - - if params.direction_optimizer_lr is None: - dir_lr = lr - else: - dir_lr = params.direction_optimizer_lr - - magnitude_tf_optimizer = params.magnitude_optimizer.GetOptimizer(lr=lr) - direction_tf_optimizer = params.direction_optimizer.GetOptimizer(lr=dir_lr) - - return adagraft.AdaGraftOptimizer( - 1.0, - magnitude_tf_optimizer, - direction_tf_optimizer, - use_global_norm=params.use_global_norm, - diagnostic=params.diagnostic) - - def AddSummary(self, lr, optimizer, var_grad): - summary_utils.scalar('adagraft_lr', lr) - - if self.params.diagnostic: # verbose option - m_step_norm_total = 0.0 - d_step_norm_total = 0.0 - - for v, _ in var_grad.Flatten(): # record layer-wise gradient norms - m_step_norm = optimizer.get_slot(v, 'm_step_norm') - d_step_norm = optimizer.get_slot(v, 'd_step_norm') - summary_utils.scalar('optimizer/m_step_norm_%s' % v.name, m_step_norm) - summary_utils.scalar('optimizer/d_step_norm_%s' % v.name, d_step_norm) - m_step_norm_total += m_step_norm**2 - d_step_norm_total += d_step_norm**2 - - # record global gradient norms - m_step_norm_total **= 0.5 - d_step_norm_total **= 0.5 - summary_utils.scalar('optimizer/m_step_norm', m_step_norm_total) - summary_utils.scalar('optimizer/d_step_norm', d_step_norm_total) - summary_utils.scalar('optimizer/norm_correction', - m_step_norm_total / d_step_norm_total) diff --git a/lingvo/core/optimizer_test.py b/lingvo/core/optimizer_test.py deleted file mode 100644 index ea1ce33a..00000000 --- a/lingvo/core/optimizer_test.py +++ /dev/null @@ -1,215 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for optimizer.""" - -import lingvo.compat as tf -from lingvo.core import layers -from lingvo.core import optimizer -from lingvo.core import py_utils -from lingvo.core import test_utils -import numpy as np - - -class OptimizerTest(test_utils.TestCase): - - def testCompositeOptimizer(self): - adam_op = optimizer.Adam.Params() - rmsprop_op = optimizer.RMSProp.Params() - adam_rmsprop_opt = optimizer.CompositeOptimizer.Params().Set( - optimizer_map={ - 'fc/w': (adam_op, 1.), - 'fc/b': (rmsprop_op, 1.), - 'default_optimizer': (adam_op, 1.) - }).Instantiate() - - adam_op_2 = optimizer.Adam.Params().Set(name='adam_2') - unspecified_comp_opt = optimizer.CompositeOptimizer.Params().Set( - optimizer_map={ - 'fc/w': (adam_op_2, 1.), - 'default_optimizer': (adam_op_2, 1.) - }).Instantiate() - - sgd_op = optimizer.SGD.Params() - adagrad_op = optimizer.Adagrad.Params() - overlapping_comp_opt = optimizer.CompositeOptimizer.Params().Set( - optimizer_map={ - 'fc/w': (sgd_op, 1.), - '.': (adagrad_op, 1.), - 'default_optimizer': (adagrad_op, 1.) - }).Instantiate() - - params = layers.FCLayer.Params() - params.name = 'fc' - params.dtype = tf.float64 - params.input_dim = 3 - params.output_dim = 2 - params.batch_norm = False - fc_layer = layers.FCLayer(params) - - inputs = tf.placeholder(shape=[2, 4, 3], dtype=tf.float64) - output = fc_layer.FPropDefaultTheta(inputs) - loss = tf.reduce_sum(output) - var_grads = py_utils.ComputeGradients(loss, fc_layer.vars) - - self.assertIn('composite_optimizer_train_op', - adam_rmsprop_opt.Apply(1e-1, var_grads).name) - self.assertIn('composite_optimizer_train_op', - unspecified_comp_opt.Apply(1e-1, var_grads).name) - with self.assertRaisesRegex( - Exception, - 'Variable fc/w/var:0 is matched 2 times by regex', - ): - overlapping_comp_opt.Apply(1e-1, var_grads) - - def testAccumulator(self): - # testAccumulator compares - # - explicit averaging of independently computed var_grads1 and - # var_grads2, - # - Accumulator(SGD) optimizer effectively doing this over 2 steps. - np.random.seed(12345) - np_input1 = np.random.normal(0.1, 0.5, [2, 4, 3]) - np.random.seed(12346) - np_input2 = np.random.normal(0.1, 0.5, [2, 4, 3]) - - with self.session(use_gpu=True, graph=tf.Graph()) as sess: - tf.random.set_seed(123456) - params = layers.ProjectionLayer.Params() - params.name = 'proj' - params.dtype = tf.float64 - params.input_dim = 3 - params.output_dim = 2 - params.params_init = py_utils.WeightInit.Gaussian(0.01, 123456) - - params.batch_norm = False - proj_layer = layers.ProjectionLayer(params) - inputs1 = tf.placeholder(shape=[2, 4, 3], dtype=tf.float64) - in_padding1 = tf.zeros([2, 4, 1], dtype=tf.float64) - inputs2 = tf.placeholder(shape=[2, 4, 3], dtype=tf.float64) - in_padding2 = tf.zeros([2, 4, 1], dtype=tf.float64) - output1 = proj_layer.FPropDefaultTheta(inputs1, in_padding1) - output2 = proj_layer.FPropDefaultTheta(inputs2, in_padding2) - loss1 = tf.reduce_sum(output1) - loss2 = tf.reduce_sum(output2) - var_grads1 = py_utils.ComputeGradients(loss1, proj_layer.vars) - var_grads2 = py_utils.ComputeGradients(loss2, proj_layer.vars) - op = optimizer.SGD.Params() - opt = op.Instantiate() - lr = 1e-1 - with tf.control_dependencies([loss1, loss2]): - var_update_op1 = opt.Apply( - lr, py_utils.ApplyGradMultiplier(var_grads1, 1. / 2.)) - with tf.control_dependencies([var_update_op1]): - var_update_op2 = opt.Apply( - lr, py_utils.ApplyGradMultiplier(var_grads2, 1. / 2.)) - - self.evaluate(tf.global_variables_initializer()) - vars1 = self.evaluate(proj_layer.vars.Flatten()) - loss1_1, grads1_1, loss1_2, grads1_2 = sess.run( - [ - loss1, - var_grads1.Transform(tuple), loss2, - var_grads2.Transform(tuple) - ], - feed_dict={ - inputs1: np_input1, - inputs2: np_input2, - }, - ) - sess.run([var_update_op2], - feed_dict={ - inputs1: np_input1, - inputs2: np_input2, - }) - vars1_1 = self.evaluate(proj_layer.vars.Flatten()) - - with self.session(use_gpu=True, graph=tf.Graph()) as sess: - tf.random.set_seed(123456) - params = layers.ProjectionLayer.Params() - params.name = 'proj' - params.dtype = tf.float64 - params.input_dim = 3 - params.output_dim = 2 - params.params_init = py_utils.WeightInit.Gaussian(0.01, 123456) - - params.batch_norm = False - proj_layer = layers.ProjectionLayer(params) - in_padding1 = tf.zeros([2, 4, 1], dtype=tf.float64) - inputs1 = tf.placeholder(shape=[2, 4, 3], dtype=tf.float64) - output1 = proj_layer.FPropDefaultTheta(inputs1, in_padding1) - loss = tf.reduce_sum(output1) - var_grads = py_utils.ComputeGradients(loss, proj_layer.vars) - op = optimizer.Accumulator.Params().Set( - accum_steps=2, dtype=tf.float64, optimizer_tpl=optimizer.SGD.Params()) - opt = op.Instantiate() - lr = 1e-1 - var_update_op = opt.Apply(lr, var_grads) - increment_global_step_op = tf.assign_add( - py_utils.GetOrCreateGlobalStepVar(), 1) - - self.evaluate(tf.global_variables_initializer()) - vars2 = self.evaluate(proj_layer.vars.Flatten()) - loss2_1, grads2_1 = sess.run([loss, var_grads.Transform(tuple)], - feed_dict={ - inputs1: np_input1, - }) - loss2_2, grads2_2 = sess.run([loss, var_grads.Transform(tuple)], - feed_dict={ - inputs1: np_input2, - }) - acc_0 = self.evaluate( - [v for v in tf.global_variables() if 'grad_accumulator' in v.name])[0] - sess.run([var_update_op], feed_dict={ - inputs1: np_input1, - }) - acc_1 = self.evaluate( - [v for v in tf.global_variables() if 'grad_accumulator' in v.name])[0] - vars2_intermediate = self.evaluate(proj_layer.vars.Flatten()) - self.evaluate(increment_global_step_op) - sess.run([var_update_op], feed_dict={ - inputs1: np_input2, - }) - acc_2 = self.evaluate( - [v for v in tf.global_variables() if 'grad_accumulator' in v.name])[0] - vars2_1 = self.evaluate(proj_layer.vars.Flatten()) - - self.assertAllClose(vars1, vars2) - - self.assertAllClose(acc_0, np.zeros_like(acc_0)) - self.assertAllClose(acc_1, grads2_1['w'][1]) - self.assertAllClose(acc_2, np.zeros_like(acc_0)) - - self.assertAllClose(loss1_1, loss2_1) - self.assertAllClose(loss1_2, loss2_2) - self.assertAllClose(grads1_1, grads2_1) - self.assertAllClose(grads1_2, grads2_2) - - self.assertAllClose(vars1, vars2_intermediate) - - self.assertAllClose(vars2[0], grads2_1['w'][0]) - self.assertAllClose(vars2[0], grads2_2['w'][0]) - - self.assertAllClose( - vars1[0] - 0.5 * lr * (grads1_1['w'][1] + grads1_2['w'][1]), vars1_1[0]) - - self.assertAllClose( - vars2[0] - 0.5 * lr * (grads2_1['w'][1] + grads2_2['w'][1]), vars2_1[0]) - - self.assertAllClose(vars2, vars2_intermediate) - self.assertAllClose(vars1_1, vars2_1) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/plot.py b/lingvo/core/plot.py deleted file mode 100644 index 9b6dbbb0..00000000 --- a/lingvo/core/plot.py +++ /dev/null @@ -1,498 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Utilities for generating image summaries using matplotlib.""" - -import collections -import functools -import traceback - -import lingvo.compat as tf -from lingvo.core import py_utils -from matplotlib.backends import backend_agg -import matplotlib.gridspec as gridspec -import matplotlib.pyplot as plt -import numpy as np -import six - - -def ToUnicode(text): - if not isinstance(text, str): - text = six.ensure_text(text, 'utf-8') - return text - - -def AddPlot(unused_fig, - axes, - data, - title=u'', - xlabel=u'', - ylabel=u'', - fontsize='small', - xlim=None, - ylim=None, - suppress_xticks=False, - suppress_yticks=False): - """Convenience function to add a plot.""" - axes.plot(data) - axes.set_title(ToUnicode(title), size=fontsize) - axes.set_xlabel(ToUnicode(xlabel), size=fontsize) - axes.set_ylabel(ToUnicode(ylabel), size=fontsize) - if xlim: - axes.set_xlim(xlim) - if ylim: - axes.set_ylim(ylim) - if suppress_xticks: - axes.set_xticks([]) - if suppress_yticks: - axes.set_yticks([]) - - -def AddImage(fig, - axes, - data, - cmap='bone_r', - clim=None, - show_colorbar=True, - title=u'', - xlabel=u'', - ylabel=u'', - fontsize='small', - origin='lower', - suppress_xticks=False, - suppress_yticks=False, - aspect='auto', - vmin=None, - vmax=None): - """Convenience function to plot data as an image on the given axes.""" - image = axes.imshow( - data, - cmap=cmap, - origin=origin, - aspect=aspect, - interpolation='nearest', - vmin=vmin, - vmax=vmax) - if show_colorbar: - fig.colorbar(image) - if clim is not None: - image.set_clim(clim) - axes.set_title(ToUnicode(title), size=fontsize) - axes.set_xlabel(ToUnicode(xlabel), size=fontsize) - axes.set_ylabel(ToUnicode(ylabel), size=fontsize) - if suppress_xticks: - axes.set_xticks([]) - if suppress_yticks: - axes.set_yticks([]) - - -def AddScatterPlot(unused_fig, - axes, - xs, - ys, - title=u'', - xlabel=u'', - ylabel=u'', - fontsize='small', - xlim=None, - ylim=None, - suppress_xticks=False, - suppress_yticks=False, - **kwargs): - """Convenience function to add a scatter plot.""" - # For 3D axes, check to see whether zlim is specified and apply it. - if 'zlim' in kwargs: - zlim = kwargs.pop('zlim') - if zlim: - axes.set_zlim(zlim) - - axes.scatter(xs, ys, **kwargs) - axes.set_title(ToUnicode(title), size=fontsize) - axes.set_xlabel(ToUnicode(xlabel), size=fontsize) - axes.set_ylabel(ToUnicode(ylabel), size=fontsize) - if xlim: - axes.set_xlim(xlim) - if ylim: - axes.set_ylim(ylim) - if suppress_xticks: - axes.set_xticks([]) - if suppress_yticks: - axes.set_yticks([]) - - -_SubplotMetadata = collections.namedtuple('_SubplotMetadata', - ['tensor_list', 'plot_func']) - - -class MatplotlibFigureSummary: - """Helper to minimize boilerplate in creating a summary with several subplots. - - Typical usage:: - - >>> fig_helper = plot.MatplotlibFigureSummary( - ... 'summary_name', shared_subplot_kwargs={'xlabel': 'Time'}) - >>> fig_helper.AddSubplot([tensor1], title='tensor1') - >>> fig_helper.AddSubplot([tensor2], title='tensor2', ylabel='Frequency') - >>> image_summary = fig_helper.Finalize() - - Can also be used as a context manager if the caller does not need the return - value from Finalize(), e.g. - - >>> with plot.MatplotlibFigureSummary('figure') as fig: - ... fig.AddSubplot([tensor1]) - """ - - def __init__(self, - name, - figsize=(8, 10), - max_outputs=3, - subplot_grid_shape=None, - gridspec_kwargs=None, - plot_func=AddImage, - shared_subplot_kwargs=None): - """Creates a new MatplotlibFigureSummary object. - - Args: - name: A string name for the generated summary. - figsize: A 2D tuple containing the overall figure (width, height) - dimensions in inches. - max_outputs: The maximum number of images to generate. - subplot_grid_shape: A 2D tuple containing the height and width dimensions - of the subplot grid. height * width must be >= the number of subplots. - Defaults to (num_subplots, 1), i.e. a vertical stack of plots. - gridspec_kwargs: A dict of extra keyword args to use when initializing the - figure's gridspec, as supported by matplotlib.gridspec.GridSpec. - plot_func: A function shared across all subplots used to populate a single - subplot. See the docstring for AddSubplot for details. - shared_subplot_kwargs: A dict of extra keyword args to pass to the plot - function for all subplots. This is useful for specifying properties - such as 'clim' which should be consistent across all subplots. - """ - self._name = name - self._figsize = figsize - self._max_outputs = max_outputs - self._subplot_grid_shape = subplot_grid_shape - self._gridspec_kwargs = gridspec_kwargs if gridspec_kwargs else {} - self._plot_func = plot_func - self._shared_subplot_kwargs = ( - shared_subplot_kwargs if shared_subplot_kwargs else {}) - self._subplots = [] - - def __enter__(self): - return self - - def __exit__(self, unused_exc_type, unused_exc_value, unused_tb): - self.Finalize() - - def AddSubplot(self, tensor_list, plot_func=None, **kwargs): - r"""Adds a subplot from tensors using plot_fun to populate the subplot axes. - - Args: - tensor_list: A list of tensors to be realized as numpy arrays and passed - as arguments to plot_func. The first dimension of each tensor in the - list corresponds to batch, and must be the same size for each tensor. - plot_func: A function with signature f(fig, axes, data1, data2, ..., - datan, \*\*kwargs) that will be called with the realized data from - tensor_list to plot data on axes in fig. This function is called - independently on each element of the batch. Overrides plot_func passed - in to the constructor. - **kwargs: A dict of additional non-tensor keyword args to pass to - plot_func when generating the plot, overridding any - shared_subplot_kwargs. Useful for e.g. specifying a subplot's title. - """ - merged_kwargs = dict(self._shared_subplot_kwargs, **kwargs) - if plot_func is None: - plot_func = self._plot_func - plot_func = functools.partial(plot_func, **merged_kwargs) - self._subplots.append(_SubplotMetadata(tensor_list, plot_func)) - - def FinalizeImage(self): - """Finishes creation of the overall figure, returning the image tensor.""" - subplot_grid_shape = self._subplot_grid_shape - if subplot_grid_shape is None: - subplot_grid_shape = (len(self._subplots), 1) - - # AddMatplotlibFigureSummary (due to restrictions of py_func) only supports - # flattened list of tensors so we must do some bookkeeping to maintain a - # mapping from _SubplotMetadata object to flattened_tensors. - subplot_slices = [] - flattened_tensors = [] - for subplot in self._subplots: - start = len(flattened_tensors) - subplot_slices.append((start, start + len(subplot.tensor_list))) - flattened_tensors.extend(subplot.tensor_list) - - def PlotFunc(fig, *numpy_data_list): - gs = gridspec.GridSpec(*subplot_grid_shape, **self._gridspec_kwargs) - for n, subplot in enumerate(self._subplots): - axes = fig.add_subplot(gs[n]) - start, end = subplot_slices[n] - subplot_data = numpy_data_list[start:end] - subplot.plot_func(fig, axes, *subplot_data) - - func = functools.partial(_RenderMatplotlibFigures, self._figsize, - self._max_outputs, PlotFunc) - batch_sizes = [tf.shape(t)[0] for t in flattened_tensors] - num_tensors = len(flattened_tensors) - with tf.control_dependencies([ - tf.assert_equal( - batch_sizes, [batch_sizes[0]] * num_tensors, summarize=num_tensors) - ]): - return tf.py_func( - func, flattened_tensors, tf.uint8, name='RenderMatplotlibFigures') - - def Finalize(self): - """Finishes creation of the overall figure, returning the image summary.""" - rendered = self.FinalizeImage() - return tf.summary.image(self._name, rendered, max_outputs=self._max_outputs) - - -def _RenderOneMatplotlibFigure(fig, plot_func, *numpy_data_list): - fig.clear() - plot_func(fig, *numpy_data_list) - fig.canvas.draw() - ncols, nrows = fig.canvas.get_width_height() - image = np.frombuffer(fig.canvas.tostring_rgb(), dtype=np.uint8) - return image.reshape(nrows, ncols, 3) - - -def _RenderMatplotlibFigures(figsize, max_outputs, plot_func, *numpy_data_list): - r"""Renders a figure containing several subplots using matplotlib. - - This is an internal implementation detail of MatplotlibFigureSummary.Finalize - and should not be called directly. - - The unconventional function signature is used to work around the behavior of - `tf.py_func` which always passes in different tensors as positional arguments. - - Args: - figsize: A 2D tuple containing the overall figure (width, height) dimensions - in inches. - max_outputs: The maximum number of images to generate. - plot_func: A function with signature f(fig, data1, data2, ..., datan) that - will be called with \*numpy_data_list to plot data in fig. - *numpy_data_list: A list of numpy matrices to plot specified as separate - arguments. - - Returns: - A numpy 4D array of type np.uint8 which can be used to generate a - `tf.math.image_summary` when converted to a tf tensor. - """ - batch_size = numpy_data_list[0].shape[0] - max_outputs = min(max_outputs, batch_size) - images = [] - - # Use plt.Figure instead of plt.figure to avoid a memory leak (matplotlib - # keeps global references to every figure created with plt.figure). When not - # using plt.figure we have to create a canvas manually. - fig = plt.Figure(figsize=figsize, dpi=100, facecolor='white') - backend_agg.FigureCanvasAgg(fig) - - for b in range(max_outputs): - data = [numpy_data[b] for numpy_data in numpy_data_list] - try: - images.append(_RenderOneMatplotlibFigure(fig, plot_func, *data)) - except Exception as e: # pylint: disable=broad-except - tf.logging.warning( - 'Error rendering example %d using matplotlib: %s\n%s', b, e, - traceback.format_exc()) - if len(images) == max_outputs: - break - plt.close(fig) - - # Pad with dummy black images in case there were too many rendering errors. - while len(images) < max_outputs: - image_shape = (1, 1, 1) - if images: - image_shape = images[0].shape - images.append(np.ones(image_shape, dtype=np.uint8)) - - return np.array(images) - - -def FigureToSummary(name, fig): - """Create tf.Summary proto from matplotlib.figure.Figure. - - Args: - name: Summary name. - fig: A matplotlib figure object. - - Returns: - A `tf.Summary` proto containing the figure rendered to an image. - """ - canvas = backend_agg.FigureCanvasAgg(fig) - fig.canvas.draw() - ncols, nrows = fig.canvas.get_width_height() - png_file = six.BytesIO() - canvas.print_figure(png_file) - png_str = png_file.getvalue() - return tf.Summary(value=[ - tf.Summary.Value( - tag='%s/image' % name, - image=tf.Summary.Image( - height=nrows, - width=ncols, - colorspace=3, - encoded_image_string=png_str)) - ]) - - -def Image(name, figsize, image, setter=None, **kwargs): - """Plot an image in numpy and generates tf.Summary proto for it. - - Args: - name: Image summary name. - figsize: A 2D tuple containing the overall figure (width, height) dimensions - in inches. - image: A 2D/3D numpy array in the format accepted by pyplot.imshow. - setter: A callable taking (fig, axes). Useful to fine-tune layout of the - figure, xlabel, xticks, etc. - **kwargs: Additional arguments to AddImage. - - Returns: - A `tf.Summary` proto contains one image visualizing 'image. - """ - assert image.ndim in (2, 3), '%s' % image.shape - fig = plt.Figure(figsize=figsize, dpi=100, facecolor='white') - axes = fig.add_subplot(1, 1, 1) - # Default show_colorbar to False if not explicitly specified. - show_colorbar = kwargs.pop('show_colorbar', False) - # Default origin to 'upper' if not explicitly specified. - origin = kwargs.pop('origin', 'upper') - AddImage( - fig, axes, image, origin=origin, show_colorbar=show_colorbar, **kwargs) - if setter: - setter(fig, axes) - return FigureToSummary(name, fig) - - -def Scatter(name, figsize, xs, ys, setter=None, **kwargs): - """Plot a scatter plot in numpy and generates tf.Summary proto for it. - - Args: - name: Scatter plot summary name. - figsize: A 2D tuple containing the overall figure (width, height) dimensions - in inches. - xs: A set of x points to plot. - ys: A set of y points to plot. - setter: A callable taking (fig, axes). Useful to fine-tune layout of the - figure, xlabel, xticks, etc. - **kwargs: Additional arguments to AddScatterPlot. - - Returns: - A `tf.Summary` proto contains one image visualizing 'image. - """ - fig = plt.Figure(figsize=figsize, dpi=100, facecolor='white') - - # If z data is provided, use 3d projection. - # - # This requires the mplot3d toolkit (e.g., from mpl_toolkits import mplot3d) - # to be registered in the program. - if 'zs' in kwargs: - axes = fig.add_subplot(111, projection='3d') - else: - axes = fig.add_subplot(1, 1, 1) - AddScatterPlot(fig, axes, xs, ys, **kwargs) - if setter: - setter(fig, axes) - return FigureToSummary(name, fig) - - -Matrix = Image # pylint: disable=invalid-name - - -def Curve(name, figsize, xs, ys, setter=None, **kwargs): - """Plot curve(s) to a `tf.Summary` proto. - - Args: - name: Image summary name. - figsize: A 2D tuple containing the overall figure (width, height) dimensions - in inches. - xs: x values for matplotlib.pyplot.plot. - ys: y values for matplotlib.pyplot.plot. - setter: A callable taking (fig, axes). Useful to fine-control layout of the - figure, xlabel, xticks, etc. - **kwargs: Extra args for matplotlib.pyplot.plot. - - Returns: - A `tf.Summary` proto contains the line plot. - """ - fig = plt.Figure(figsize=figsize, dpi=100, facecolor='white') - axes = fig.add_subplot(1, 1, 1) - axes.plot(xs, ys, '.-', **kwargs) - if setter: - setter(fig, axes) - return FigureToSummary(name, fig) - - -def AddMultiCurveSubplot(fig, - tensors, - paddings, - labels, - xlabels=None, - **kwargs): - """Adds a multi curve subplot to Matplotlib figure. - - Plots one line for each entry in tensors and assigns a plot label legend. - - Args: - fig: The Matplotlib figure. - tensors: List of tensors of shape [batch, length] - paddings: Paddings for 'tensors' with shape [batch, length] with 0. in valid - positions and 1. in invalid. - labels: A list of tensor names (strings) of the same length as 'tensors'. - xlabels: A string tensor of shape [batch] with an xlabel per batch. - **kwargs: With optional, title, xlabel, ylabel, fontsize. - """ - data = [] - row_labels = [] - for t, l in zip(tensors, labels): - if t is not None: - data.append(py_utils.ApplyPadding(paddings, t)) - row_labels.append(l) - shape = py_utils.GetShape(data[0], 2) - data = tf.reshape(tf.concat(data, -1), [shape[0], len(data), shape[1]]) - - args = [data, py_utils.LengthsFromPaddings(paddings)] - if xlabels is not None: - args.append(xlabels) - fig.AddSubplot( - args, plot_func=_AddMultiCurveRowPlots, row_labels=row_labels, **kwargs) - - -def _AddMultiCurveRowPlots(fig, - axes, - data, - length, - x_label_override=None, - row_labels=None, - title=u'', - xlabel=u'', - ylabel=u'', - fontsize='small'): - """Add a plot per row in data and cut the plot by the length.""" - del fig - colors = ['b-', 'r-', 'g-', 'm-', 'y-'] - for row in range(data.shape[0]): - label = row_labels[row] if row_labels else '{}'.format(row) - axes.plot(data[row, :length], colors[row % len(colors)], label=label) - axes.set_xlim([0, length]) - axes.legend() - axes.set_title(ToUnicode(title), size=fontsize) - if x_label_override: - axes.set_xlabel(ToUnicode(x_label_override), size='x-small', wrap=True) - else: - axes.set_xlabel(ToUnicode(xlabel), size=fontsize) - axes.set_ylabel(ToUnicode(ylabel), size=fontsize) diff --git a/lingvo/core/plot_test.py b/lingvo/core/plot_test.py deleted file mode 100644 index 9cf305ef..00000000 --- a/lingvo/core/plot_test.py +++ /dev/null @@ -1,275 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for plot.""" - -import lingvo.compat as tf -from lingvo.core import plot -from lingvo.core import test_utils -import numpy as np - - -class PlotTest(test_utils.TestCase): - - def testToUnicode(self): - str_str = 'pójdź kińże tę chmurność w głąb flaszy' - uni_str = u'pójdź kińże tę chmurność w głąb flaszy' - - self.assertEqual(plot.ToUnicode(str_str), uni_str) - self.assertEqual(plot.ToUnicode(str_str), plot.ToUnicode(uni_str)) - - def testMatrix(self): - summary = plot.Matrix('summary', (4, 4), np.random.rand(10, 10)) - self.assertEqual(len(summary.value), 1) - value = summary.value[0] - self.assertGreater(value.image.width, 0) - self.assertGreater(value.image.height, 0) - - def testScatter(self): - summary = plot.Scatter( - 'summary', (4, 4), xs=np.random.rand(10), ys=np.random.rand(10)) - self.assertEqual(len(summary.value), 1) - value = summary.value[0] - self.assertGreater(value.image.width, 0) - self.assertGreater(value.image.height, 0) - - -class MatplotlibFigureSummaryTest(test_utils.TestCase): - - FIGSIZE = (8, 4) - EXPECTED_DPI = 100 - DEFAULT_DATA = tf.ones((5, 4)) - - def setUp(self): - with self.session() as s: - fig = plot.MatplotlibFigureSummary('DEFAULT', self.FIGSIZE, max_outputs=1) - batched_data = tf.expand_dims(self.DEFAULT_DATA, 0) # Batch size 1. - fig.AddSubplot([batched_data]) - im = fig.Finalize() - summary_str = s.run(im) - summary = tf.summary.Summary.FromString(summary_str) - self.default_encoded_image = summary.value[0].image.encoded_image_string - - def testBasic(self): - with self.session() as s: - fig = plot.MatplotlibFigureSummary( - 'matplotlib_figure', self.FIGSIZE, max_outputs=1) - batched_data = tf.expand_dims(self.DEFAULT_DATA, 0) # Batch size 1. - fig.AddSubplot([batched_data]) - im = fig.Finalize() - summary_str = s.run(im) - summary = tf.summary.Summary.FromString(summary_str) - self.assertEqual(len(summary.value), 1) - value = summary.value[0] - self.assertEqual(value.tag, 'matplotlib_figure/image') - self.assertEqual(value.image.width, self.EXPECTED_DPI * self.FIGSIZE[0]) - self.assertEqual(value.image.height, self.EXPECTED_DPI * self.FIGSIZE[1]) - self.assertEqual(value.image.colorspace, 3) - self.assertEqual(value.image.encoded_image_string, - self.default_encoded_image) - - def testCanUseAsContextManager(self): - with self.session() as s: - with plot.MatplotlibFigureSummary( - 'context_manager_figure', self.FIGSIZE, max_outputs=1) as fig: - batched_data = tf.expand_dims(self.DEFAULT_DATA, 0) # Batch size 1. - fig.AddSubplot([batched_data]) - summary_str = s.run(tf.summary.merge_all(scope='context_manager_figure')) - summary = tf.summary.Summary.FromString(summary_str) - self.assertEqual(len(summary.value), 1) - value = summary.value[0] - self.assertEqual(value.image.width, self.EXPECTED_DPI * self.FIGSIZE[0]) - self.assertEqual(value.image.height, self.EXPECTED_DPI * self.FIGSIZE[1]) - self.assertEqual(value.image.colorspace, 3) - self.assertEqual(value.image.encoded_image_string, - self.default_encoded_image) - - def testUnicodeText(self): - with self.session() as s: - fig = plot.MatplotlibFigureSummary( - 'matplotlib_uni', self.FIGSIZE, max_outputs=1) - batched_data = tf.expand_dims(self.DEFAULT_DATA, 0) # Batch size 1. - fig.AddSubplot([batched_data], xlabel=u'bździągwa', ylabel='żółć') - im = fig.Finalize() - summary_str = s.run(im) - summary = tf.summary.Summary.FromString(summary_str) - self.assertEqual(len(summary.value), 1) - value = summary.value[0] - self.assertEqual(value.tag, 'matplotlib_uni/image') - self.assertEqual(value.image.width, self.EXPECTED_DPI * self.FIGSIZE[0]) - self.assertEqual(value.image.height, self.EXPECTED_DPI * self.FIGSIZE[1]) - self.assertEqual(value.image.colorspace, 3) - - def testMultipleSubplots(self): - batch_size = 4 - tensors = [tf.ones((batch_size, 3, 5)), tf.ones((batch_size, 2, 2))] - with self.session() as s: - max_outputs = 3 - fig = plot.MatplotlibFigureSummary('fig', self.FIGSIZE, max_outputs) - for t in tensors: - fig.AddSubplot([t]) - im = fig.Finalize() - summary_str = s.run(im) - summary = tf.summary.Summary.FromString(summary_str) - self.assertEqual(len(summary.value), max_outputs) - for n, value in enumerate(summary.value): - self.assertEqual(value.tag, 'fig/image/%d' % n) - self.assertEqual(value.image.width, self.EXPECTED_DPI * self.FIGSIZE[0]) - self.assertEqual(value.image.height, self.EXPECTED_DPI * self.FIGSIZE[1]) - self.assertEqual(value.image.colorspace, 3) - self.assertNotEqual(value.image.encoded_image_string, - self.default_encoded_image) - - def testCustomPlotFunc(self): - batch_size = 3 - data = tf.ones((batch_size, 3, 5)) - trim = tf.constant([[3, 2], [2, 5], [3, 3]]) - titles = tf.constant(['batch1', 'batch2', 'batch3']) - - def TrimAndAddImage(fig, axes, data, trim, title, **kwargs): - plot.AddImage(fig, axes, data[:trim[0], :trim[1]], title=title, **kwargs) - - with self.session() as s: - fig = plot.MatplotlibFigureSummary( - 'fig_custom_plotfunc', - max_outputs=batch_size, - plot_func=TrimAndAddImage) - fig.AddSubplot([data, trim, titles]) - im = fig.Finalize() - summary_str = s.run(im) - summary = tf.summary.Summary.FromString(summary_str) - self.assertEqual(len(summary.value), batch_size) - for n, value in enumerate(summary.value): - self.assertEqual(value.tag, 'fig_custom_plotfunc/image/%d' % n) - self.assertGreater(value.image.width, 1) - self.assertGreater(value.image.height, 1) - self.assertEqual(value.image.colorspace, 3) - self.assertNotEqual(value.image.encoded_image_string, - self.default_encoded_image) - - def testDoesNotDieOnMatplotlibError(self): - invalid_dim_data = tf.ones((5,)) - with self.session() as s: - fig = plot.MatplotlibFigureSummary('summary', self.FIGSIZE, max_outputs=1) - fig.AddSubplot([invalid_dim_data]) - im = fig.Finalize() - summary_str = s.run(im) - summary = tf.summary.Summary.FromString(summary_str) - self.assertEqual(len(summary.value), 1) - value = summary.value[0] - # Generates dummy 1-pixel image. - self.assertEqual(value.image.width, 1) - self.assertEqual(value.image.height, 1) - - def testLargerBatch(self): - batch_size = 4 - tensors = [tf.ones((batch_size, 3, 5)), tf.ones((batch_size, 2, 2))] - with self.session() as s: - fig = plot.MatplotlibFigureSummary( - 'larger_batch', self.FIGSIZE, max_outputs=batch_size) - for t in tensors: - fig.AddSubplot([t]) - im = fig.Finalize() - summary_str = s.run(im) - summary = tf.summary.Summary.FromString(summary_str) - self.assertEqual(len(summary.value), batch_size) - for n, value in enumerate(summary.value): - self.assertEqual(value.tag, u'larger_batch/image/%d' % n) - self.assertEqual(value.image.width, self.EXPECTED_DPI * self.FIGSIZE[0]) - self.assertEqual(value.image.height, self.EXPECTED_DPI * self.FIGSIZE[1]) - self.assertEqual(value.image.colorspace, 3) - self.assertNotEqual(value.image.encoded_image_string, - self.default_encoded_image) - - def testCanChangeFigsize(self): - figsize = (self.FIGSIZE[0], 2 * self.FIGSIZE[1]) - with self.session() as s: - fig = plot.MatplotlibFigureSummary('summary', figsize, max_outputs=1) - batched_data = tf.expand_dims(self.DEFAULT_DATA, 0) # Batch size 1. - fig.AddSubplot([batched_data]) - im = fig.Finalize() - summary_str = s.run(im) - - summary_str = s.run(im) - summary = tf.summary.Summary.FromString(summary_str) - self.assertEqual(len(summary.value), 1) - value = summary.value[0] - self.assertEqual(value.image.width, self.EXPECTED_DPI * figsize[0]) - self.assertEqual(value.image.height, self.EXPECTED_DPI * figsize[1]) - self.assertNotEqual(value.image.encoded_image_string, - self.default_encoded_image) - - def testEnforcesConsistentBatchSize(self): - batch_size = 4 - tensors = [tf.ones((batch_size, 3, 5)), tf.ones((batch_size - 2, 2, 2))] - with self.session() as s: - fig = plot.MatplotlibFigureSummary('summary', self.FIGSIZE, max_outputs=1) - for t in tensors: - fig.AddSubplot([t]) - im = fig.Finalize() - with self.assertRaises(tf.errors.InvalidArgumentError): - s.run(im) - - def testOnlyPlotsFirstMaxOutputImages(self): - batch_size = 4 - tensors = [tf.ones((batch_size, 3, 5)), tf.ones((batch_size, 2, 2))] - with self.session() as s: - fig = plot.MatplotlibFigureSummary('summary', self.FIGSIZE, max_outputs=2) - for t in tensors: - fig.AddSubplot([t]) - im = fig.Finalize() - summary_str = s.run(im) - summary = tf.summary.Summary.FromString(summary_str) - self.assertEqual(len(summary.value), 2) - - def testLimitsOutputImagesIfBatchIsSmall(self): - batch_size = 1 - tensors = [tf.zeros((batch_size, 3, 5)), tf.ones((batch_size, 2, 2))] - with self.session() as s: - fig = plot.MatplotlibFigureSummary('summary', self.FIGSIZE, max_outputs=3) - for t in tensors: - fig.AddSubplot([t]) - im = fig.Finalize() - summary_str = s.run(im) - summary = tf.summary.Summary.FromString(summary_str) - self.assertEqual(len(summary.value), 1) - - def testAddMultiCurveSubplot(self): - with self.session(graph=tf.Graph(), use_gpu=False): - fig = plot.MatplotlibFigureSummary('XXX') - batch_size = 2 - tensor = tf.ones([batch_size, 3]) - paddings = tf.constant([[0., 0., 0.], [0., 1., 1.]]) - plot.AddMultiCurveSubplot( - fig, [tensor, tensor], - paddings, - labels=['label1', 'label2'], - xlabels=tf.constant(['a', 'b']), - title='Title', - ylabel='Ylabel') - summary_str = self.evaluate(fig.Finalize()) - - summary = tf.Summary.FromString(summary_str) - self.assertEqual(len(summary.value), batch_size) - for n, value in enumerate(summary.value): - self.assertEqual(value.tag, 'XXX/image/%d' % n) - self.assertGreater(value.image.width, 0) - self.assertGreater(value.image.height, 0) - self.assertNotEqual(value.image.encoded_image_string, - self.default_encoded_image) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/predictor.py b/lingvo/core/predictor.py deleted file mode 100644 index a35fac6c..00000000 --- a/lingvo/core/predictor.py +++ /dev/null @@ -1,307 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Predictor does inference using a saved inference graph. - -Example:: - - params = model_registry.GetParams('MyModel', 'Test') - inference_graph = inference_graph_exporter.InferenceGraphExporter.Export( - params) - pred = Predictor(inference_graph=inference_graph) - pred.Load("/tmp/logdir/train/ckpt-00000000") - [topk_hyps] = pred.Run(["topk_hyps"], src_strings=["Hello World"]) -""" -import threading -import time -from lingvo import model_imports -import lingvo.compat as tf -from lingvo.core import inference_graph_pb2 -from lingvo.core import py_utils - -from google.protobuf import text_format - - -def LoadInferenceGraph(path, clear_device_placement=False): - """Parse the given path as an InferenceGraph proto. - - Args: - path: The path to the file to load. - clear_device_placement: If true, clears device field from nodes in graph. - - Returns: - An InferenceGraph object. - """ - inference_graph = inference_graph_pb2.InferenceGraph() - with tf.io.gfile.GFile(path, "r") as f: - text_format.Parse(f.read(), inference_graph) - if clear_device_placement: - for node in inference_graph.graph_def.node: - node.ClearField("device") - for function in inference_graph.graph_def.library.function: - for node_def in function.node_def: - node_def.ClearField("device") - return inference_graph - - -class Predictor: - """Loads a model and does inference. - - See model.Inference() documentation for list of fetches and feeds. - """ - - def __init__(self, - inference_graph, - subgraph_name=None, - checkpoint=None, - device_type="gpu", - tf_master="", - session_config=None, - clear_device_placement=False): - """Constructor. - - Args: - inference_graph: A saved InferenceGraph proto. - subgraph_name: The subgraph to use for prediction. - checkpoint: An optional checkpoint to load. - device_type: Device type string. Either "cpu", "gpu", or "tpu". - tf_master: The tf_master. - session_config: A tf.SessionConfig to use. By default - py_utils.SessionConfig() is used. - clear_device_placement: If set, clears device field of loaded inference - graph. - """ - assert device_type in ["cpu", "gpu", "tpu"] - subgraph_name = subgraph_name or "default" - if isinstance(inference_graph, str): - tf.logging.info("Reading inference graph from %s.", inference_graph) - inference_graph = LoadInferenceGraph(inference_graph, - clear_device_placement) - self._inference_graph = inference_graph - self._checkpoint = checkpoint - self._device_type = device_type - self._tf_master = tf_master - self._session_config = session_config - - self._graph = tf.Graph() - with self._graph.as_default(): - tf.logging.info( - "Loading inference graph for prediction subgraph_name={}.".format( - subgraph_name)) - with tf.device("/%s:0" % "cpu" if device_type == "tpu" else device_type): - tf.import_graph_def(inference_graph.graph_def, name="") - if device_type == "tpu": - # If no tpu init op exists, create it here. - try: - self._graph.get_operation_by_name("tpu_init_op") - except KeyError: - tf.group(tf.tpu.initialize_system(), name="tpu_init_op") - - self._graph.finalize() - - if inference_graph.subgraphs: - if subgraph_name not in inference_graph.subgraphs: - raise ValueError( - "Subgraph %s not defined. Valid subgraphs: %s" % - (subgraph_name, list(inference_graph.subgraphs.keys()))) - subgraph = inference_graph.subgraphs[subgraph_name] - self._fetches = subgraph.fetches - self._feeds = subgraph.feeds - else: - self._fetches = inference_graph.fetches - self._feeds = inference_graph.feeds - - # Lock for creating new sessions. - self._sess_lock = threading.Lock() - self._cur_sess_id = 0 - self._CreateNewSession() - - @property - def fetch_keys(self): - return list(self._fetches.keys()) - - @property - def feed_keys(self): - return list(self._feeds.keys()) - - @property - def feed_shapes(self): - # Conversion from dict to NestedMap required. - return py_utils.Transform( - lambda x: self._graph.get_tensor_by_name(x).shape.as_list(), - py_utils.NestedMap(self._feeds)) - - @property - def fetch_shapes(self): - # Conversion from dict to NestedMap required. - return py_utils.Transform( - lambda x: self._graph.get_tensor_by_name(x).shape.as_list(), - py_utils.NestedMap(self._fetches)) - - @py_utils.RetryOnTransientTfError() - def _CreateNewSession(self): - """Updates self._sess with a new session.""" - config = self._session_config - if not config: - config = py_utils.SessionConfig() - sess = tf.Session(self._tf_master, graph=self._graph, config=config) - - try: - sess.run(self._graph.get_operation_by_name("init_all_tables")) - except KeyError: - tf.logging.info("Could not find tables initializer in graph.") - if self._device_type == "tpu": - sess.run(self._graph.get_operation_by_name("tpu_init_op")) - if self._checkpoint: - sess.run(self._inference_graph.saver_def.restore_op_name, { - self._inference_graph.saver_def.filename_tensor_name: self._checkpoint - }) - - else: - try: - init_op = self._graph.get_operation_by_name("init_all_variables") - sess.run(init_op) - except KeyError: - tf.logging.warning( - "No checkpoint provided and the graph has no default " - "variable_init op.") - tf.logging.info("Created new predictor session.") - self._sess = sess - - def _MaybeCreateNewSession(self, sess_id): - """Create a new session if sess_id is the current session. - - Args: - sess_id: The id of a session that no longer works. - """ - with self._sess_lock: - if sess_id == self._cur_sess_id: - self._CreateNewSession() - self._cur_sess_id += 1 - tf.logging.info("Current session id {}.".format(self._cur_sess_id)) - - @py_utils.RetryOnTransientTfError() - def _RunWithValidSession(self, fn, *args, **kwargs): - """Ensures `fn` is called while self._sess is a valid session.""" - sess_id = self._cur_sess_id - try: - return fn(self._sess, *args, **kwargs) - except py_utils.transient_tf_errors: - # self._sess is invalid, most likely due to the worker being preempted. - # Make sure a new session is created before re-raising the exception and - # triggering the py_utils.Retry loop. - self._MaybeCreateNewSession(sess_id) - raise - - def Load(self, checkpoint): - """Loads parameters from a checkpoint if self._sess is a valid session. - - Args: - checkpoint: The checkpoint path to restore. - """ - if checkpoint != self._checkpoint: - sess_id = self._cur_sess_id - try: - self._sess.run( - self._inference_graph.saver_def.restore_op_name, - {self._inference_graph.saver_def.filename_tensor_name: checkpoint}) - except py_utils.transient_tf_errors: - # self._sess is invalid, most likely due to the worker being preempted. - # Make sure a new session is created before re-raising the exception and - # triggering the py_utils.Retry loop. - self._MaybeCreateNewSession(sess_id) - raise - self._checkpoint = checkpoint - - def Run(self, - fetch_keys, - validate_fetches=True, - session_run_options=None, - run_metadata=None, - time_session_run=False, - **kwargs): - """Runs predictor. - - Args: - fetch_keys: dict_keys object or a list of keys in the fetch dictionary to - fetch. - validate_fetches: if True, raises a KeyError if a specified fetch is - invalid. If False, returns None for invalid fetches instead. - session_run_options: Optional tf.RunOptions() to use in the session. - run_metadata: Optional tf.RunMetadata() to use in the session. - time_session_run: Optional bool, if True, additionally return the - execution time of session.run. Defaults to False. - **kwargs: a dict of inputs to feed. - - Returns: - A list of predictions corresponding to the order of fetch_keys and, if - time_session_run is True, the run time in seconds. - - Raises: - InvalidArgumentError: the number of inputs does not meet requirements. - KeyError: a feed specified in kwargs is invalid, or a fetch in fetch_keys - is invalid and validate_fetches is True. - """ - single_fetch = False - if not isinstance(fetch_keys, (list, type(dict().keys()))): - single_fetch = True - fetch_keys = [fetch_keys] - - if validate_fetches: - for x in fetch_keys: - if x not in self._fetches: - raise KeyError( - "%s is not in the list of available fetches. Available keys: %s" % - (x, list(self._fetches.keys()))) - valid_fetch_idxs, valid_fetches = zip(*[(i, self._fetches[k]) - for i, k in enumerate(fetch_keys) - if k in self._fetches.keys()]) - - for k in kwargs: - if k not in self._feeds: - raise KeyError( - "%s is not in the list of available feeds. Available keys: %s" % - (k, list(self._feeds.keys()))) - feeds = {self._feeds[k]: v for k, v in kwargs.items()} - - run_options = tf.RunOptions(report_tensor_allocations_upon_oom=False) - if session_run_options: - run_options = session_run_options - - start = time.time() - fetched_results = self._RunWithValidSession( - tf.Session.run, - valid_fetches, - feed_dict=feeds, - options=run_options, - run_metadata=run_metadata) - duration = time.time() - start - results = [None] * len(fetch_keys) - for i, fetch in zip(valid_fetch_idxs, fetched_results): - results[i] = fetch - if single_fetch: - results = results[0] - return (results, duration) if time_session_run else results - - -def main(_): - import IPython # pylint: disable=g-import-not-at-top - IPython.start_ipython(argv=["--colors", "NoColor"], user_ns=globals()) - - -if __name__ == "__main__": - tf.disable_eager_execution() - model_imports.ImportAllParams() - tf.app.run(main) diff --git a/lingvo/core/predictor_runner_base.py b/lingvo/core/predictor_runner_base.py deleted file mode 100644 index a16c5130..00000000 --- a/lingvo/core/predictor_runner_base.py +++ /dev/null @@ -1,290 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Interface for binaries built around predictor. - -To use: subclass PredictorRunnerBase, implement the InputGenerator and RunBatch -functions, and call Run(). - -To run on TPU, set: - --device_type=tpu - --xla_device=tpu - --tf_master=url/to/tpu/server - --inference_threads=num_tpu_cores -""" - -import concurrent.futures -import itertools -import os -import re -import threading -import time - -from absl import flags -from lingvo import compat as tf -from lingvo.core import predictor -from lingvo.core import py_utils -import six - -flags.DEFINE_string( - 'checkpoint', None, 'Either a checkpoint file to load,' - ' or a directory containing multiple checkpoints, where' - ' the latest checkpoint will be loaded.') -flags.DEFINE_string( - 'inference_graph', None, 'Path to an inference graph. ' - 'If not specified, will be inferred from the checkpoint path.') -flags.DEFINE_string( - 'inference_subgraph_name', '', 'The name of the inference subgraph to use. ' - 'Defaults to the default subgraph.') -flags.DEFINE_enum('device_type', 'gpu', ['cpu', 'gpu', 'tpu'], 'Device type.') -flags.DEFINE_string('tf_master', 'local', 'tf_master for predictor session.') -flags.DEFINE_integer('inference_threads', '1', 'Number of inference threads.') -flags.DEFINE_integer('batch_size', 64, 'Batch size.') -flags.DEFINE_integer( - 'prediction_step_interval', 3000, 'Number of steps between outputs. ' - 'Only meaningful if FLAGS.checkpoint is a directory.') - -flags.DEFINE_integer('max_inputs', 0, 'Only process the first n inputs.') -flags.DEFINE_list( - 'input_id_filter', [], - 'If not empty, only process the input ids in the given list.') -flags.DEFINE_string( - 'output_dir', None, 'Output directory. ' - 'If FLAGS.checkpoint is a directory, a subdirectory will be created ' - 'for each checkpoint evaluated.') -flags.DEFINE_integer( - 'output_num_shards', 1, 'Each replica generates one shard of output ' - 'according to --output_shard_id.') -flags.DEFINE_integer( - 'output_shard_id', 0, - 'The output shard id in range [0, output_num_shards - 1].') - -FLAGS = flags.FLAGS - -_RETRY_SLEEP_SECONDS = 10 - - -class PredictorRunnerBase: - """Manages state for running predictor. - - Abstract class. Subclasses should override `InputGenerator` and `RunBatch`. - Call `Subclass().Run()` in `main()` function to run. - """ - - def __init__(self, - checkpoint, - output_dir=None, - inference_graph=None, - inference_subgraph_name='', - device_type='cpu', - output_num_shards=1, - output_shard_id=0, - max_inputs=0, - input_id_filter=None, - tf_master='local', - inference_threads=1, - batch_size=64, - prediction_step_interval=3000): - """Constructor. - - Args: - checkpoint: Either a checkpoint file to load, or a directory containing - multiple checkpoints, where the latest checkpoint will be loaded. - output_dir: Output directory. If `checkpoint` is a directory, a - subdirectory will be created for each checkpoint evaluated. - inference_graph: Path to an inference graph. If not specified, will be - inferred from the checkpoint path. - inference_subgraph_name: The name of the inference subgraph to use. - Defaults to the default subgraph. - device_type: Device type, either cpu, gpu, or tpu. - output_num_shards: Each replica generates one shard of output according to - `output_shard_id`. - output_shard_id: The output shard id in range `[0, output_num_shards - - 1]`. - max_inputs: Only process the first n inputs. 0 means process all inputs. - input_id_filter: If not empty, only process the input ids in the given - list. - tf_master: tf_master for predictor session. - inference_threads: Number of inference threads. - batch_size: Batch size. - prediction_step_interval: Number of steps between outputs. Only meaningful - if `checkpoint` is a directory. - """ - self._checkpoint = checkpoint - self._output_dir = output_dir - self._output_num_shards = output_num_shards - self._output_shard_id = output_shard_id - self._max_inputs = max_inputs - input_id_filter = input_id_filter or [] - self._input_id_filter = [str(x) for x in input_id_filter] - self._batch_size = batch_size - self._prediction_step_interval = prediction_step_interval - - if device_type == 'tpu' and FLAGS.xla_device != 'tpu': - raise ValueError('xla_device=tpu should be set with device_type=tpu!') - - if tf.io.gfile.isdir(self._checkpoint): - initial_checkpoint = tf.train.latest_checkpoint(self._checkpoint) - while (not initial_checkpoint or - not tf.io.gfile.exists(initial_checkpoint + '.index')): - tf.logging.log_first_n(tf.logging.INFO, - 'Waiting for checkpoint to be available.', - 10) - time.sleep(_RETRY_SLEEP_SECONDS) - initial_checkpoint = tf.train.latest_checkpoint(self._checkpoint) - else: - initial_checkpoint = self._checkpoint - if not tf.io.gfile.exists(initial_checkpoint + '.index'): - raise ValueError('Could not find checkpoint %s' % initial_checkpoint) - - # Use saved inference graph. - if inference_graph: - self._inference_graph = inference_graph - else: - checkpoint_dir = self._checkpoint - if not tf.io.gfile.isdir(checkpoint_dir): - checkpoint_dir = os.path.dirname(checkpoint_dir) - logdir = os.path.dirname(checkpoint_dir) - inference_graph_filename = 'inference.pbtxt' - if device_type == 'tpu': - inference_graph_filename = 'inference_tpu.pbtxt' - self._inference_graph = os.path.join(logdir, 'inference_graphs', - inference_graph_filename) - self._predictor = predictor.Predictor( - inference_graph=self._inference_graph, - subgraph_name=inference_subgraph_name, - checkpoint=initial_checkpoint, - device_type=device_type, - tf_master=tf_master) - self._threadpool = concurrent.futures.ThreadPoolExecutor(inference_threads) - self._locks = [threading.Lock() for _ in range(inference_threads)] - - @classmethod - def FromFlags(cls, **kwargs): - """Constructs an instance of this class from FLAGS.""" - return cls( - checkpoint=FLAGS.checkpoint, - output_dir=FLAGS.output_dir, - inference_graph=FLAGS.inference_graph, - inference_subgraph_name=FLAGS.inference_subgraph_name, - device_type=FLAGS.device_type, - output_num_shards=FLAGS.output_num_shards, - output_shard_id=FLAGS.output_shard_id, - max_inputs=FLAGS.max_inputs, - input_id_filter=FLAGS.input_id_filter, - tf_master=FLAGS.tf_master, - inference_threads=FLAGS.inference_threads, - batch_size=FLAGS.batch_size, - prediction_step_interval=FLAGS.prediction_step_interval, - **kwargs) - - def _ShouldProcessInputId(self, input_id): - if self._max_inputs > 0 and input_id >= self._max_inputs: - return False - if self._input_id_filter and str(input_id) not in self._input_id_filter: - return False - return input_id % self._output_num_shards == self._output_shard_id - - def _OutputFilename(self, output_dir, name): - assert self._output_shard_id >= 0 - assert self._output_shard_id < self._output_num_shards - return '%s-%.5d-of-%.5d' % (os.path.join( - output_dir, name), self._output_shard_id, self._output_num_shards) - - def InputGenerator(self): - """Generator that yields the next input. - - Must yield in a deterministic order or raise an exception when - self._output_num_shards > 1. - """ - raise NotImplementedError('Abstract method.') - - def RunBatch(self, output_dir, batch): - """Runs predictor on a single batch of data. - - Args: - output_dir: the output directory. - batch: a list of (input_id, element) pairs, where element is yielded from - InputGenerator and input_id is a unique counter starting from 0. - """ - raise NotImplementedError('Abstract method.') - - def _PredictOneCheckpoint(self, checkpoint, output_dir): - """Runs predictor.""" - tf.logging.info('Processing checkpoint %s.', checkpoint) - self._predictor.Load(checkpoint) - - def LockedRunBatch(batch, batch_id): - """TPU inference runs the i-th batch on the i%num_cores-th core. - - Make sure that core is available before scheduling the next batch on it. - - Args: - batch: The input to be passed to RunBatch. - batch_id: The id of this batch, which determins which core it runs on. - """ - with self._locks[batch_id % len(self._locks)]: - self.RunBatch(output_dir, batch) - - batch_id = 0 - batch = [] - futures = [] - # Iterate through the input and process it one batch at a time. - it = self.InputGenerator() - if self._max_inputs > 0: - it = itertools.islice(it, self._max_inputs) - for next_id, element in enumerate(it): - if self._ShouldProcessInputId(next_id): - batch.append((next_id, element)) - if len(batch) == self._batch_size: - futures.append( - self._threadpool.submit(LockedRunBatch, batch, batch_id)) - batch_id += 1 - batch = [] - # Last batch. - if batch: - futures.append(self._threadpool.submit(LockedRunBatch, batch, batch_id)) - # Wait for completion. - for f in futures: - f.result() - - def _PredictContinuously(self): - """Waits for new checkpoints and runs predictor continuously.""" - prev_step = -1000000 - while True: - # TODO(jonathanasdf): how to determine when training finished? - path = tf.train.latest_checkpoint(self._checkpoint) - step_str = re.search(r'ckpt-(\d{8})', six.ensure_str(path)).group(1) - step = int(step_str) - if step - prev_step >= self._prediction_step_interval: - if not self._output_dir: - raise ValueError( - 'output_dir must be specified for _PredictContinuously.') - output_dir = os.path.join(self._output_dir, 'step_' + step_str) - tf.io.gfile.makedirs(output_dir) - self._PredictOneCheckpoint(path, output_dir) - prev_step = step - tf.logging.info('Waiting for next checkpoint...') - time.sleep(_RETRY_SLEEP_SECONDS) - - @py_utils.RetryOnTransientTfError() - def Run(self): - """Monitor checkpoints and runs predictor.""" - if self._output_dir: - tf.io.gfile.makedirs(self._output_dir) - if tf.io.gfile.isdir(self._checkpoint): - self._PredictContinuously() - else: - self._PredictOneCheckpoint(self._checkpoint, self._output_dir) diff --git a/lingvo/core/predictor_test.py b/lingvo/core/predictor_test.py deleted file mode 100644 index b3c8cc62..00000000 --- a/lingvo/core/predictor_test.py +++ /dev/null @@ -1,81 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lingvo.core.predictor.""" - -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import base_model -from lingvo.core import inference_graph_exporter -from lingvo.core import inference_graph_pb2 -from lingvo.core import predictor -from lingvo.core import test_utils - - -class DummyModel(base_model.BaseTask): - - def Inference(self): - with tf.name_scope('inference'): - feed1 = tf.placeholder(name='feed1_node', dtype=tf.float32, shape=[1]) - fetch1 = tf.identity(feed1, name='fetch1_node') - inference_graph = inference_graph_pb2.InferenceGraph() - subgraph = inference_graph.subgraphs['default'] - subgraph.feeds['feed1'] = feed1.name - subgraph.fetches['fetch1'] = fetch1.name - return inference_graph - - -class PredictorTest(test_utils.TestCase): - - def _testInferenceGraph(self): - p = base_model.SingleTaskModel.Params(DummyModel.Params().Set(name='test')) - p.input = base_input_generator.BaseInputGenerator.Params().Set(name='test') - inference_graph = inference_graph_exporter.InferenceGraphExporter.Export(p) - return inference_graph - - def testPredictorFeedShapes(self): - pred = predictor.Predictor(self._testInferenceGraph()) - self.assertEqual([1], pred.feed_shapes.feed1) - - def testPredictorFetchShapes(self): - pred = predictor.Predictor(self._testInferenceGraph()) - self.assertEqual([1], pred.fetch_shapes.fetch1) - - def testPredictor(self): - pred = predictor.Predictor(self._testInferenceGraph()) - fetch1 = pred.Run('fetch1', feed1=[12345]) - self.assertEqual(12345, fetch1) - - def testMissingFeedRaisesInvalidArgumentError(self): - pred = predictor.Predictor(self._testInferenceGraph()) - with self.assertRaisesRegex(tf.errors.InvalidArgumentError, 'feed1'): - pred.Run('fetch1') - - def testInvalidFetchRaisesKeyError(self): - pred = predictor.Predictor(self._testInferenceGraph()) - with self.assertRaisesRegex(KeyError, 'nonexistent'): - pred.Run(['fetch1', 'nonexistent'], feed1=[12345]) - - def testInvalidFetchWithoutValidateFetchesReturnsNone(self): - pred = predictor.Predictor(self._testInferenceGraph()) - fetch1, nonexistent = pred.Run(['fetch1', 'nonexistent'], - feed1=[12345], - validate_fetches=False) - self.assertEqual(12345, fetch1) - self.assertIsNone(nonexistent) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/program.py b/lingvo/core/program.py deleted file mode 100644 index 42b8c73d..00000000 --- a/lingvo/core/program.py +++ /dev/null @@ -1,1201 +0,0 @@ -# Lint as: python3 -# Lint as: python2, python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Programs for interleaving execution on TPU.""" - -import multiprocessing.dummy -import os -import time - -import lingvo.compat as tf -from lingvo.core import base_model -from lingvo.core import checkpointer -from lingvo.core import cluster_factory -from lingvo.core import hyperparams -from lingvo.core import metrics -from lingvo.core import ml_perf_log as mlp_log -from lingvo.core import py_utils -from lingvo.core import summary_utils - -# pylint:disable=g-direct-tensorflow-import -from tensorflow.core.protobuf.tpu import compilation_result_pb2 as tpu_compilation_result -from tensorflow.python.tpu import tpu -from tensorflow.python.tpu import tpu_function -from tensorflow.python.tpu import training_loop as tpu_training_loop -from tensorflow.python.tpu.ops import tpu_ops -# pylint:enable=g-direct-tensorflow-import - - -class BaseProgram: - """A Program associated with a Task. - - This is inspired by the "program" multi-tenancy that TPUs - support. Essentially, each program corresponds with a - sub-graph can exist in the same Graph/Session. - - Upon first execution, it is XLA/JIT compiled and is subsequently - available to be executed on demand without significant delay. - - Program's provides the following functionality: - - - Builds a sub-graph - - Writes summaries - - Runs for pre-determined `steps_per_loop` steps with appropriate infeeds - """ - - @classmethod - def Params(cls): - """"Defaults parameters for Programs.""" - p = hyperparams.InstantiableParams(cls) - p.Define('task', None, 'Underlying task') - p.Define('logdir', None, 'Log directory') - p.Define('num_splits_per_client', None, '') - p.Define('steps_per_loop', None, 'Number of steps to run.') - p.Define('dataset_name', None, - 'Dataset the program is operating on, eg: "Test"') - p.Define('name', 'base_program', 'Program name.') - p.Define('task_name', None, - 'If multi-task, what the high-level task name is') - p.Define('num_threads', 1, 'Number of threads in multiprocessing pool.') - p.Define('spmd', False, 'Whether program is running under SPMD mode.') - return p - - def __init__(self, params, shared_model=None): - self.params = params.Copy() - p = self.params - self._task_params = p.task - self._logdir = p.logdir - self._task_name = p.task_name - self._program_name = '' - self._shared_model = shared_model - - # Program dirs are where the summaries are written to. - if p.task_name: - program_dir_name = ( - p.task_name + '_' + p.name + '_' + p.dataset_name.lower()) - else: - program_dir_name = p.name + '_' + p.dataset_name.lower() - self._program_dir = os.path.join(self._logdir, program_dir_name) - # Initialized on use; access via self._summary_writer property only. - self._summary_writer_obj = None - - tf.io.gfile.makedirs(self._logdir) - # Just a standard spot that all programs may restore from. - self._checkpoint_dir = os.path.join(self._logdir, 'train') - tf.io.gfile.makedirs(self._checkpoint_dir) - - self._steps_per_loop = p.steps_per_loop - self.num_splits_per_client = p.num_splits_per_client - self.data_parallelism = p.num_splits_per_client - - # Thread Pool for infeed. - self._infeed_pool = multiprocessing.dummy.Pool(p.num_threads) - - self._compile_op = None - self._status_msg_fn = None - - @property - def _summary_writer(self): - """Returns the FileWriter object to use for summaries.""" - # Initialize on first use, so that subclasses can override the - # implementation without creating a default FileWriter in the constructor. - if self._summary_writer_obj is None: - self._summary_writer_obj = tf.summary.FileWriter(self._program_dir) - return self._summary_writer_obj - - def _SummarizeValue(self, steps, tag, value): - self._summary_writer.add_summary( - metrics.CreateScalarSummary(tag, value), steps) - - def _WriteSummaries(self, job_name, global_step, summaries): - """Write summaries to be viewed by TensorBoard. - - Args: - job_name: The name of this job ('trainer', 'evaler', etc.) - global_step: Integer number of trainer steps (not a tensor). - summaries: Dict of {summary_name: tf.Summary()}. - """ - if not summaries: - return - for unused_name, summary in sorted(summaries.items()): - self._summary_writer.add_summary(summary, global_step) - if summary.value: - for value in summary.value: - if value.HasField('simple_value'): - tf.logging.info('%s summary on checkpoint@%d %s = %.8g', job_name, - global_step, value.tag, value.simple_value) - self._summary_writer.flush() - - def _InfeedLoop(self, sess): - tf.logging.info('_InfeedLoop start') - try: - for i in range(self._steps_per_loop): - tf.logging.vlog(1, '_InfeedLoop %d', i) - sess.run(self._task.input.tpu_infeed_op) - # Hmm... so how does self._steps_per_loop get set? - tf.logging.info('_InfeedLoop done') - except Exception as e: - tf.logging.info('_InfeedLoop exception %r %s', e, e) - raise - - def BuildTpuSubgraph(self): - """Sub classes should construct a model/graph to be executed by Run. - - Specific to TPU execution, this may involve a - @tpu_function.on_device_training_loop etc. - """ - raise NotImplementedError() - - def SetStatusMessageFn(self, fn): - """Workaround since we instantiate programs via Params.""" - self._status_msg_fn = fn - - def SetStatusMessage(self, msg): - """Write to borglet status.""" - tf.logging.info('Status: %s', msg) - if self._status_msg_fn: - self._status_msg_fn(msg) - - def Compile(self, sess): - """Compile the program using the given session handle.""" - if self._compile_op is not None: - self.SetStatusMessage('Compiling %s' % self._program_name) - result = sess.run(self._compile_op) - proto = tpu_compilation_result.CompilationResultProto() - proto.ParseFromString(result) - if proto.status_error_message: - tf.logging.fatal('Compilation failed: {}'.format( - proto.status_error_message)) - tf.logging.info('Compiling %s done.', self._program_name) - - def Run(self, sess): - """Execute the program using the given session handle. - - Args: - sess: TF Session. - - Returns: - done: Whether to end all execution. - """ - raise NotImplementedError() - - def CreateCheckpointer(self): - self._checkpointer = checkpointer.Checkpointer(self._checkpoint_dir, - self._model) - - def RestoreIfNeeded(self, sess): - self._checkpointer.RestoreIfNeeded(sess) - - def _InstantiateTaskModel(self, task_params): - """Instantiates a model object for a particular task. - - MultiTaskModels can accept a shared_model parameter, but SingleTaskModels - cannot, so we handle them separately here. - - Args: - task_params: An params instance that constructs either a SingleTaskModel - or a MultiTaskSubModel. - - Returns: - An instantiated object based on task_params. - """ - if issubclass(task_params.cls, base_model.MultiTaskSubModel): - return task_params.Instantiate(shared_model=self._shared_model) - return task_params.Instantiate() - - -class TrainProgram(BaseProgram): - """TrainProgram trains a single task and handles checkpoints.""" - - def __init__(self, params, shared_model=None): - super().__init__(params, shared_model=shared_model) - self._step_rate_tracker = summary_utils.StepRateTracker() - self._program_name = 'TrainProgram' - - def _OutfeedEnqueue(self, per_example_tensors): - if not per_example_tensors: - return tf.no_op() - per_example_tensors = py_utils.NestedMap(per_example_tensors) - device = tpu.core(0) if self.spmd else '' - with tf.device(device): - return tpu_ops.outfeed_enqueue_tuple(per_example_tensors.Flatten()) - - def _OutfeedDequeueLoop(self, per_example_tensors, num_loops, num_devices): - """Process all per-example tensor outfeed data for a TPU sess.run. - - Args: - per_example_tensors: dict of key -> tensor as generated by TpuTrainStep. - num_loops: number of times that TpuTrainStep will be executed by TpuTrain. - num_devices: number of TPU cores assigned to this process. - - Returns: - A dict of per-example tensors from the latest TpuTrainStep. - """ - if not per_example_tensors: - return tf.no_op() - - tensor_shapes = [ - py_utils.GetShape(per_example_tensors[key]) - for key in sorted(per_example_tensors) - ] - tensor_types = [ - tf.as_dtype(per_example_tensors[key].dtype) - for key in sorted(per_example_tensors) - ] - - def LoopBody(i, *input_arrays): - """Process outfeed data for a single TpuTrainStep. - - Args: - i: current loop index. - *input_arrays: One tf.TensorArray per outfeed tensor. - - Returns: - i+1 (new index) plus post-write tf.TensorArray handles. - """ - # Outfeed ops execute on each JF node, so they must be located on the - # nodes. - outfeed_devices = [] - device_assignment = py_utils.GetTpuDeviceAssignment() - assert device_assignment - for replica in range(device_assignment.num_replicas): - num_cores_per_replica = 1 if self.spmd else ( - device_assignment.num_cores_per_replica) - for core in range(num_cores_per_replica): - with tf.device(device_assignment.host_device(replica, core)): - outfeed_devices.append( - tpu_ops.outfeed_dequeue_tuple( - tensor_types, - tensor_shapes, - device_ordinal=device_assignment.tpu_ordinal(replica, - core))) - offset = i * num_devices - output_arrays = list(input_arrays) - # Each output_array holds a different per-example tensor. We get results - # for each tensor from each TPU for each TpuTrainStep call. - for j in range(len(output_arrays)): - for k in range(len(outfeed_devices)): - output_arrays[j] = output_arrays[j].write(offset + k, - outfeed_devices[k][j]) - - return tuple([i + 1] + output_arrays) - - def LoopCond(i, *output_arrays): - del output_arrays - return i < num_loops - - output_arrays = [] - for i in range(len(tensor_shapes)): - output_arrays.append( - tf.TensorArray( - tensor_types[i], - size=num_loops * num_devices, - element_shape=tensor_shapes[i])) - # Loop once for each time that TpuTrainStep runs. - output_arrays = tf.while_loop( - LoopCond, LoopBody, [0] + output_arrays, parallel_iterations=1)[1:] - concatenated_arrays = [array.concat() for array in output_arrays] - return dict(zip(sorted(per_example_tensors), concatenated_arrays)) - - def BuildTpuSubgraph(self): - tf.logging.info('TrainProgram BuildTpuSubGraph') - self.spmd = ( - self.params.spmd or - self._task_params.input.use_partitioned_infeed_queue) - - self._eval_metrics = metrics.TpuEvalMetrics() - data_parallelism = self.data_parallelism - - with cluster_factory.SetImmediatelyInstantiateVariables(False): - self._model = self._InstantiateTaskModel(self._task_params) - self._task = self._model.GetTask() - self._task.input.InstantiateVariables() - self._task.input.CreateTpuEnqueueOps() - - def TpuTrainStep(*args): - """Train a shard of a batch on a single TPU core. - - Args: - *args: metrics values from previous steps. - - Returns: - New summed metrics values and a train_op. - """ - with tf.name_scope('tpu_train'): - with py_utils.OpportunisticVariableReuseScope(True): - self._model.InstantiateVariables() - self._model.ConstructFPropBPropGraph() - per_step_eval_metrics = self._eval_metrics.SetMetrics( - self._task.eval_metrics, args) - outfeed_op = self._OutfeedEnqueue(self._task.per_example_tensors) - summed_metrics = [] - assert len(per_step_eval_metrics) == len(args) - with tf.control_dependencies([outfeed_op]): - for x, y in zip(per_step_eval_metrics, args): - summed_metrics.append(x + y) - return summed_metrics + [self._task.train_op] - - @tpu_function.on_device_training_loop - def TpuTrain(): - loop_result = tpu_training_loop.repeat( - self._steps_per_loop, - TpuTrainStep, - inputs=self._eval_metrics.initial_values, - name='train_loop') - # Final metrics are the avg across self._steps_per_loop steps. - return self._eval_metrics.FinalizeMetrics(loop_result) - - self._compile_op, batch_parallel_res = tpu.split_compile_and_shard( - TpuTrain, - num_shards=data_parallelism, - device_assignment=py_utils.GetTpuDeviceAssignment()) - outfeed_dequeue_op = self._OutfeedDequeueLoop( - self._task.per_example_tensors, self._steps_per_loop, - self.num_splits_per_client) - - # Get metric result from a single replica; they are all same here. - - def _ConstructPostTrainingLoop(train_loop_op, outfeed_dequeue_op): - """Returns the op for tpu training with tail cpu computation.""" - # Adds a tail computation that is run after the tpu_training loop - # step finishes. This allows us to run certain computation that - # acts on the variable between tpu_train_loop iterations and - # amortizing the cost of the operations. Alternative of running - # tpu.outside_compilation & using tf.cond is expenseive. - with tf.control_dependencies(train_loop_op): - self._model.ConstructPostTrainingLoop() - with tf.control_dependencies([self._task.post_training_loop_op]): - return ([[tf.identity(o) for o in train_loop_op], outfeed_dequeue_op]) - - # Get metric result from a single replica; they are all same here. - all_tpu_ops = [t[0] for t in batch_parallel_res] - self.tpu_ops = (_ConstructPostTrainingLoop(all_tpu_ops, outfeed_dequeue_op)) - self._model_analysis, self._total_num_params = summary_utils.ModelAnalysis( - self._model) - try: - with tf.io.gfile.GFile( - os.path.join(self._program_dir, 'model_analysis.txt'), 'w') as f: - f.write(self._model_analysis) - except tf.errors.NotFoundError as e: - tf.logging.info('Failed to write model analysis %s', e) - - return self.tpu_ops - - def Run(self, sess): - global_step = sess.run(self._model.global_step) - self.SetStatusMessage('Executing train program at step %d' % global_step) - infeed_future = self._infeed_pool.apply_async( - self._InfeedLoop, args=(sess,)) - ary = sess.run(self.tpu_ops) - infeed_future.wait() - - values = ary[0] - outfeeds = ary[1] - - self._eval_metrics.PackMetricsValues(values) - eval_metrics = self._eval_metrics.metrics - - global_step = sess.run(self._model.global_step) - step_rate, example_rate, total_examples = ( - self._step_rate_tracker.ComputeStepRate( - global_step, - eval_metrics['num_samples_in_batch'][0] * self._steps_per_loop)) - self._SummarizeValue(global_step, 'global_step/sec', step_rate) - self._SummarizeValue(global_step, 'examples/sec', example_rate) - self._SummarizeValue(global_step, 'total_samples', total_examples) - self._SummarizeValue(global_step, 'total_num_params', - self._total_num_params) - for key, (val, _) in sorted(eval_metrics.items()): - self._SummarizeValue(global_step, key, val) - - task_global_step = sess.run(self._task.global_step) - summaries = self._task.ProcessFPropResults(sess, task_global_step, - eval_metrics, outfeeds) - self._WriteSummaries( - os.path.basename(self._program_dir), global_step, summaries) - - # Simpler version of _ShouldStop without early stopping. - if task_global_step >= self._task_params.train.max_steps: - tf.logging.info('ShouldStop: step:%6d params.train.max_steps:%6d', - task_global_step, self._task_params.train.max_steps) - return True - - return False - - -class EvalProgram(BaseProgram): - """Evaluation program. - - Note that this currently has different infeed semantics compared to - the existing Evaler as the input generator is not recreated - per-eval. Thus different random samples are selected each - evaluation. - """ - - def __init__(self, params, shared_model=None): - super().__init__(params, shared_model=shared_model) - self._program_name = 'EvalProgram' - - def BuildTpuSubgraph(self): - tf.logging.info('EvalProgram BuildTpuSubGraph') - with cluster_factory.SetEval(True): - self._eval_metrics = metrics.TpuEvalMetrics() - data_parallelism = self.data_parallelism - with cluster_factory.SetImmediatelyInstantiateVariables(False): - self._model = self._InstantiateTaskModel(self._task_params) - self._task = self._model.GetTask() - self._task.input.InstantiateVariables() - self._task.input.CreateTpuEnqueueOps() - - def TpuEvalStep(*args): - """Eval a shard of a batch on a single TPU core. - - Args: - *args: metrics values from previous steps. - - Returns: - Summed eval metrics. - """ - with tf.name_scope('tpu_eval'): - with py_utils.OpportunisticVariableReuseScope(True): - self._model.InstantiateVariables() - self._model.ConstructFPropGraph() - per_step_eval_metrics = self._eval_metrics.SetMetrics( - self._task.eval_metrics, args) - summed_metrics = [] - for x, y in zip(per_step_eval_metrics, args): - summed_metrics.append(x + y) - return summed_metrics - - @tpu_function.on_device_training_loop - def TpuEval(): - loop_result = tpu_training_loop.repeat( - self._steps_per_loop, - TpuEvalStep, - inputs=self._eval_metrics.initial_values, - name='eval_loop') - # Final metrics are the avg across self._steps_per_loop steps. - return self._eval_metrics.FinalizeMetrics(loop_result) - - self._compile_op, batch_parallel_res = tpu.split_compile_and_shard( - TpuEval, - num_shards=data_parallelism, - device_assignment=py_utils.GetTpuDeviceAssignment()) - # Get metric result from a single replica; they are all same here. - self.tpu_ops = [[t[0] for t in batch_parallel_res]] - - return self.tpu_ops - - def Run(self, sess): - global_step = sess.run(self._model.global_step) - self.SetStatusMessage('Executing eval program at step %d' % global_step) - - infeed_future = self._infeed_pool.apply_async( - self._InfeedLoop, args=(sess,)) - ary = sess.run(self.tpu_ops) - infeed_future.wait() - values = ary[0] - self._eval_metrics.PackMetricsValues(values) - for key, (val, _) in sorted(self._eval_metrics.metrics.items()): - self._SummarizeValue(global_step, key, val) - self._summary_writer.flush() - return False - - -class DecodeProgram(BaseProgram): - """DecodeProgram. - - Note that this currently has different infeed semantics compared to - the existing Decoder as the input generator is not recreated - per-eval. Thus different random samples are selected each - decoder run. - """ - - def __init__(self, params, shared_model=None): - super().__init__(params, shared_model=shared_model) - self._program_name = 'DecodeProgram' - - def _CompileDecodeFn(self): - """Wrap the DecodeFn with split_compile_and_shard.""" - with cluster_factory.SetImmediatelyInstantiateVariables(False): - self._model = self._InstantiateTaskModel(self._task_params) - self._task = self._model.GetTask() - self._task.input.InstantiateVariables() - self._task.input.CreateTpuEnqueueOps() - - def _DecodeFn(): - """Decode call to be compiled for TPU.""" - with py_utils.OpportunisticVariableReuseScope(True): - self._model.InstantiateVariables() - input_batch = self._task.input.TpuDequeueBatch() - # Bingo! self._task.Decode. This defers to DecodeWithTheta - metrics_dict = self._task.Decode(input_batch) - self.metrics_nm = py_utils.NestedMap(metrics_dict) - return self.metrics_nm.Flatten() - - self._compile_op, batch_parallel_res = tpu.split_compile_and_shard( - _DecodeFn, - num_shards=self.data_parallelism, - device_assignment=py_utils.GetTpuDeviceAssignment()) - - self.metrics = py_utils.NestedMap(self.metrics_nm) - self.metrics = self.metrics.Pack(batch_parallel_res) - - def BuildTpuSubgraph(self): - tf.logging.info('DecodeProgram BuildTpuSubGraph') - py_utils.ResetStepSeed() - with cluster_factory.SetEval(True): - self._CompileDecodeFn() - return None - - def Run(self, sess): - global_step = sess.run(self._model.global_step) - self.SetStatusMessage('Executing decode program at step %d' % global_step) - infeed_future = self._infeed_pool.apply_async( - self._InfeedLoop, args=(sess,)) - dec_metrics = self._task.CreateDecoderMetrics() - start_time = time.time() - buffered_decode_out = [] - # self._steps_per_loop is tricky to set. - for i in range(self._steps_per_loop): - metrics_values = sess.run(self.metrics) - # It seems like this is the right place to write to disk. PostProcessDecodeOut - decode_out = self._task.PostProcessDecodeOut(metrics_values, dec_metrics) - tf.logging.info('step: %d %f' % - (i, dec_metrics['num_samples_in_batch'].total_value)) - if decode_out: - # So buffered_decode_out should be a list of something. Could cause - buffered_decode_out.extend(decode_out) - infeed_future.wait() - - num_examples_metric = dec_metrics['num_samples_in_batch'] - summaries = {k: v.Summary(k) for k, v in dec_metrics.items()} - elapsed_secs = time.time() - start_time - example_rate = num_examples_metric.total_value / elapsed_secs - summaries['examples/sec'] = tf.Summary( - value=[tf.Summary.Value(tag='examples/sec', simple_value=example_rate)]) - self._WriteSummaries( - os.path.basename(self._program_dir), global_step, summaries) - decode_out_path = os.path.join(self._program_dir, - 'decoder_out_%09d' % global_step) - # Experimental one doesn't have these finalize calls. So let's use non-experimental for now... - decode_finalize_args = base_model.DecodeFinalizeArgs( - decode_out_path=decode_out_path, decode_out=buffered_decode_out) - self._task.DecodeFinalize(decode_finalize_args) - return False - - -# Note: Inherits from DecodeProgram -class ExperimentalDecodeProgram(DecodeProgram): - """DecodeProgram in a tpu loop. - - TODO(huangyp) test this for beam search decoders and replace the - default DecodeProgram. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.num_threads = 2 - return p - - def _CompileDecodeLoop(self): - """Wrap the DecodeLoop with split_compile_and_shard.""" - device_assignment = py_utils.GetTpuDeviceAssignment() - with cluster_factory.SetImmediatelyInstantiateVariables(False): - self._model = self._InstantiateTaskModel(self._task_params) - self._task = self._model.GetTask() - self._task.input.InstantiateVariables() - self._task.input.CreateTpuEnqueueOps() - - def _DecodeStep(): - """Decode call to be compiled for TPU.""" - with py_utils.OpportunisticVariableReuseScope(True): - self._model.InstantiateVariables() - input_batch = self._task.input.TpuDequeueBatch() - # defers to DecodeWithTheta - metrics_dict = self._task.Decode(input_batch) - self.metrics_nm = py_utils.NestedMap(metrics_dict) - device = tpu.core(0) if self.spmd else '' - with tf.device(device): - outfeed_enqueue = tpu_ops.outfeed_enqueue_tuple( - self.metrics_nm.Flatten()) - return [outfeed_enqueue] - - @tpu_function.on_device_training_loop - def DecodeLoopFn(): - return tpu_training_loop.repeat( - self._steps_per_loop, _DecodeStep, inputs=[]) - - self._compile_op, self.decode_loop = tpu.split_compile_and_shard( - DecodeLoopFn, - num_shards=self.data_parallelism, - device_assignment=device_assignment) - - # Get a list of outfeed ops. - self.metrics = self._OutfeedDequeue() - # Pack the list of outfeed ops with structure in self.metrics_nm. - self.metrics = tf.nest.pack_sequence_as(self.metrics_nm, self.metrics) - - def BuildTpuSubgraph(self): - tf.logging.info('DecodeProgram BuildTpuSubGraph') - py_utils.ResetStepSeed() - self.spmd = ( - self.params.spmd or - self._task_params.input.use_partitioned_infeed_queue) - with cluster_factory.SetEval(True): - self._CompileDecodeLoop() - return - - def _OutfeedDequeue(self): - """Collect outfeed dequeue from all devices.""" - num_outfeeds = len(self.metrics_nm.Flatten()) - outfeed_ops = [[]] * num_outfeeds - device_assignment = py_utils.GetTpuDeviceAssignment() - assert device_assignment - for replica in range(device_assignment.num_replicas): - num_cores_per_replica = 1 if self.spmd else ( - device_assignment.num_cores_per_replica) - for core in range(num_cores_per_replica): - with tf.device(device_assignment.host_device(replica, core)): - outfeeds_per_core = tpu_ops.outfeed_dequeue_tuple( - dtypes=[x.dtype for x in self.metrics_nm.Flatten()], - shapes=[x.shape for x in self.metrics_nm.Flatten()], - device_ordinal=device_assignment.tpu_ordinal(replica, core)) - for idx_outfeed, out_feed in enumerate(outfeeds_per_core): - outfeed_ops[idx_outfeed] = outfeed_ops[idx_outfeed] + [out_feed] - return [tf.concat(per_outfeed, 0) for per_outfeed in outfeed_ops] - - def _DecodeLoop(self, sess): - sess.run(self.decode_loop) - - def Run(self, sess): - global_step = sess.run(self._model.global_step) - self.SetStatusMessage('Executing decode program at step %d' % global_step) - infeed_future = self._infeed_pool.apply_async( - self._InfeedLoop, args=(sess,)) - decode_future = self._infeed_pool.apply_async( - self._DecodeLoop, args=(sess,)) - - dec_metrics = self._task.CreateDecoderMetrics() - start_time = time.time() - # set this very high - for _ in range(self._steps_per_loop): - metrics_values = sess.run(self.metrics) - self._task.PostProcessDecodeOut(metrics_values, dec_metrics) - decode_future.wait() - infeed_future.wait() - summaries = {k: v.Summary(k) for k, v in dec_metrics.items()} - elapsed_secs = time.time() - start_time - num_examples_metric = dec_metrics['num_samples_in_batch'] - example_rate = num_examples_metric.total_value / elapsed_secs - summaries['examples/sec'] = tf.Summary( - value=[tf.Summary.Value(tag='examples/sec', simple_value=example_rate)]) - self._WriteSummaries( - os.path.basename(self._program_dir), global_step, summaries) - - return False - - -class MLPerfTrainDecodeProgram(BaseProgram): - """Run train/decode in a single session run.""" - - @classmethod - def Params(cls): - """"Defaults parameters for Programs.""" - p = super().Params() - p.Define('train_task', None, 'Underlying task') - p.Define('decode_task', None, 'Underlying task') - p.Define('train_dataset_name', None, '') - p.Define('decode_dataset_name', None, '') - p.Define('train_steps_per_loop', 0, '') - p.Define('decode_steps_per_loop', 0, '') - p.Define('ml_perf', None, 'MLPerf config') - return p - - def __init__(self, params, shared_model=None): - super().__init__(params, shared_model=shared_model) - p = self.params - if p.ml_perf is not None and p.ml_perf.benchmark_name is not None: - self._ml_perf_log = True - self._ml_perf = p.ml_perf - self._ml_perf_epoch = -1 - else: - self._ml_perf_log = False - self._program_name = 'TrainAndDecodeProgram' - self._train_steps_per_loop = params.train_steps_per_loop - self._decode_steps_per_loop = params.decode_steps_per_loop - assert self._decode_steps_per_loop == 1, ('Only supports a single decode ' - 'step right now.') - self._train_task_params = params.train_task - self._decode_task_params = params.decode_task - self._run_start = None - self._run_stop = None - self._train_pool = multiprocessing.dummy.Pool(1) - self._warmup_seconds = 60 - - def BuildTpuSubgraph(self): - if self._ml_perf_log: - mlp_log.mlperf_print('global_batch_size', self._ml_perf.global_batch_size) - mlp_log.mlperf_print('max_sequence_length', - self._ml_perf.max_sequence_length) - mlp_log.mlperf_print('opt_name', self._ml_perf.optimizer_name) - mlp_log.mlperf_print('opt_base_learning_rate', - self._ml_perf.base_learning_rate) - mlp_log.mlperf_print('opt_learning_rate_warmup_steps', - self._ml_perf.warmup_steps) - - self._eval_metrics = metrics.TpuEvalMetrics() - data_parallelism = self.data_parallelism - with cluster_factory.SetImmediatelyInstantiateVariables(False): - self._train_model = self._train_task_params.Instantiate() - self._train_task = self._train_model.GetTask() - self._train_task.input.InstantiateVariables() - self._train_task.input.CreateTpuEnqueueOps() - self._model = self._train_model - - def TpuTrainStep(): - """Train a shard of a batch on a single TPU core. - - Do not calculate loss metrics. - - Returns: - [train_op]. - """ - with py_utils.OpportunisticVariableReuseScope(True): - self._train_model.InstantiateVariables() - self._train_model.ConstructFPropBPropGraph() - return [self._train_task.train_op] - - def TpuTrain(): - loop_result = tpu_training_loop.repeat( - self._train_steps_per_loop, - TpuTrainStep, - inputs=[], - name='train_loop') - return loop_result - - py_utils.ResetStepSeed() - - with cluster_factory.SetImmediatelyInstantiateVariables(False): - self._decode_model = self._InstantiateTaskModel(self._decode_task_params) - self._decode_task = self._decode_model.GetTask() - self._decode_task.input.InstantiateVariables() - self._decode_task.input.CreateTpuEnqueueOps() - - def _DecodeFn(): - """Decode call to be compiled for TPU.""" - with py_utils.OpportunisticVariableReuseScope(True): - with cluster_factory.SetEval(True): - self._decode_model.InstantiateVariables() - input_batch = self._decode_task.input.TpuDequeueBatch() - metrics_dict = self._decode_task.Decode(input_batch) - self.metrics_nm = py_utils.NestedMap(metrics_dict) - return self.metrics_nm.Flatten() - - @tpu_function.on_device_training_loop - def TrainAndDecode(): - with tf.control_dependencies([TpuTrain()]): - return _DecodeFn() - - self._compile_op, batch_parallel_res = tpu.split_compile_and_shard( - TrainAndDecode, - num_shards=data_parallelism, - device_assignment=py_utils.GetTpuDeviceAssignment()) - - self.metrics = py_utils.NestedMap(self.metrics_nm) - self.metrics = self.metrics.Pack(batch_parallel_res) - return None - - def _InfeedLoop(self, sess): - tf.logging.info('_InfeedLoop start') - try: - for i in range(self._train_steps_per_loop): - tf.logging.vlog(1, '_InfeedLoop %d', i) - sess.run(self._train_task.input.tpu_infeed_op) - if self._ml_perf_log: - mlp_log.mlperf_print( - 'eval_start', - None, - metadata={ - 'first_epoch_num': self._ml_perf_epoch + 1, - 'epoch_count': 1 - }) - for i in range(self._decode_steps_per_loop): - tf.logging.vlog(1, '_InfeedLoop %d', i) - sess.run(self._decode_task.input.tpu_infeed_op) - tf.logging.info('_InfeedLoop done') - except Exception as e: - tf.logging.info('_InfeedLoop exception %r %s', e, e) - raise - - def _TrainAndDecode(self, sess): - metrics_values = sess.run(self.metrics) - self._decode_task.PostProcessDecodeOut(metrics_values, self.dec_metrics) - - def Run(self, sess): - global_step = sess.run(self._model.global_step) - self.dec_metrics = self._decode_task.CreateDecoderMetrics() - # Start TPU program thread. - train_future = self._train_pool.apply_async( - self._TrainAndDecode, args=(sess,)) - - if self._warmup_seconds > 0: - # The first execution of the TPU program has a warm-up - # so we delay feeding data yet as that's when the MLPerf timing - # starts. This way, when we actually infeed, the TPU program - # is immediately ready to execute/dequeue data. - tf.logging.info('Waiting before first infeed.') - time.sleep(self._warmup_seconds) - self._warmup_seconds = 0 - - if self._ml_perf_log: - if not self._run_start: - mlp_log.mlperf_print(key='init_stop', value=None) - self._run_start = mlp_log.mlperf_print(key='run_start', value=None) - steps_per_epoch = self._ml_perf.steps_per_epoch - epoch = int(global_step) // steps_per_epoch - if epoch > self._ml_perf_epoch: - self._ml_perf_epoch = epoch - mlp_log.mlperf_print( - 'block_start', - None, - metadata={ - 'first_epoch_num': epoch + 1, - 'epoch_count': 1 - }) - self.SetStatusMessage('MLPerf epoch: %d' % self._ml_perf_epoch) - # Start infeed thread. - infeed_future = self._infeed_pool.apply_async( - self._InfeedLoop, args=(sess,)) - - infeed_future.wait() - train_future.wait() - - if self._ml_perf_log: - mlp_log.mlperf_print( - 'eval_stop', None, metadata={'epoch_num': (epoch + 1)}) - mlperf_metric = self._ml_perf.decoder_metric_name - mlperf_metric_value = float(self.dec_metrics[mlperf_metric].value) - mlp_log.mlperf_print( - 'eval_accuracy', mlperf_metric_value, metadata={'epoch_num': epoch}) - if mlperf_metric_value > self._ml_perf.decoder_metric_success_threshold: - tf.logging.info('ml_perf_final_threshold: %f exceeded', - self._ml_perf.decoder_metric_success_threshold) - if not self._run_stop: - self._run_stop = mlp_log.mlperf_print( - 'run_stop', None, metadata={'status': 'success'}) - self.SetStatusMessage('MLPerf run_time: %.2f' % - (self._run_stop - self._run_start)) - return True - return False - - -class MultiTaskProgramSchedule: - """Container for ProgramSchedules for a MultiTask model.""" - - @classmethod - def Params(cls): - p = hyperparams.InstantiableParams(cls) - p.Define('program_schedule_dict', None, - 'task_name -> ProgramScheduleParams') - return p - - -class SimpleProgramSchedule: - """A schedule of programs associated with a single task. - - Simple sequence is: - This is the behavior! - Run train_executions_per_eval * train_program - Run all the eval_programs - """ - - @classmethod - def Params(cls): - """Params for a SimpleProgramSchedule.""" - p = hyperparams.InstantiableParams(cls) - p.Define('task_dict', None, 'dataset_name -> task params') - p.Define('task_name', None, 'High level task name') - p.Define('logdir', None, 'Log directory') - p.Define('train_program', None, 'Train program params') - p.Define('train_executions_per_eval', 1, '') - p.Define('eval_programs', [], 'List of eval program params.') - p.Define('num_splits_per_client', None, '') - p.Define('dataset_names', [], 'List of all dataset names.') - - # TODO(blee): Clean these up. - p.Define('ml_perf', hyperparams.Params(), 'MlPerf configuration.') - mlp = p.ml_perf - mlp.Define('benchmark_name', None, 'Benchmark name for compliance log.') - return p - - def __init__(self, params, shared_model=None): - self.params = params.Copy() - p = self.params - self._shared_model = shared_model - - # Propagate run-time parameters to programs: - p.train_program.logdir = p.logdir - if p.train_program.dataset_name not in p.task_dict: - tf.logging.error('could not find %s in %s' % - (p.train_program.dataset_name, p.task_dict)) - print("GALVEZ:task_dict=", list(p.task_dict.keys())) - import sys; sys.stdout.flush() - p.train_program.task = p.task_dict[p.train_program.dataset_name] - p.train_program.num_splits_per_client = p.num_splits_per_client - p.train_program.task_name = p.task_name - - for eval_program_params in p.eval_programs: - eval_program_params.logdir = p.logdir - eval_program_params.task = p.task_dict[eval_program_params.dataset_name] - eval_program_params.task_name = p.task_name - eval_program_params.num_splits_per_client = p.num_splits_per_client - - self.eval_programs = [] - self.train_program = p.train_program.Instantiate(shared_model=shared_model) - for eval_program in p.eval_programs: - self.eval_programs.append( - eval_program.Instantiate(shared_model=shared_model)) - - self._programs = [] - self._programs.append(self.train_program) - self._programs += self.eval_programs - - def Programs(self): - return self._programs - - def Run(self, sess): - p = self.params - for _ in range(p.train_executions_per_eval): - done = self.train_program.Run(sess) - if done: - break - for eval_program in self.eval_programs: - eval_program.Run(sess) - return False - - -def SimpleProgramScheduleForTask(train_dataset_name, - train_steps_per_loop, - eval_dataset_names, - eval_steps_per_loop, - decode_steps_per_loop, - experimental_decoder=False, - train_program_cls=TrainProgram, - eval_program_cls=EvalProgram): - """Convenient helper method for common case. - - Args: - train_dataset_name: Name of the training dataset, eg: 'Train' - train_steps_per_loop: Number of steps to execute the training program. - eval_dataset_names: List of eval dataset_name strings, eg: ['Train']. - eval_steps_per_loop: Number of steps to execute the eval program. - decode_steps_per_loop: Number of steps to execute the decode program. - experimental_decoder: bool. Whether to use experimental deocder which is - placed in a tpu loop. - train_program_cls: The class to use for training programs. Defaults - to TrainProgram. - eval_program_cls: The class to use for eval programs. Defaults to - EvalProgram. - - Returns: - A populated SimpleProgramSchedule.Params() - """ - - program_schedule_params = SimpleProgramSchedule.Params() - train_program_params = train_program_cls.Params() - train_program_params.name = 'train' - train_program_params.steps_per_loop = train_steps_per_loop - train_program_params.dataset_name = train_dataset_name - program_schedule_params.train_program = train_program_params - - program_schedule_params.dataset_names = [] - - for dataset_name in eval_dataset_names: - program_schedule_params.dataset_names.append(dataset_name) - if eval_steps_per_loop > 0: - eval_program_params = eval_program_cls.Params() - eval_program_params.name = 'eval_tpu' - # TODO(blee): This should be derived from the Dataset size. - eval_program_params.steps_per_loop = eval_steps_per_loop - eval_program_params.dataset_name = dataset_name - program_schedule_params.eval_programs.append(eval_program_params) - - if decode_steps_per_loop > 0: - decoder = ( - ExperimentalDecodeProgram if experimental_decoder else DecodeProgram) - decode_program_params = decoder.Params() - decode_program_params.name = 'decode_tpu' - # TODO(blee): This should be derived from the Dataset size. - decode_program_params.steps_per_loop = decode_steps_per_loop - decode_program_params.dataset_name = dataset_name - program_schedule_params.eval_programs.append(decode_program_params) - - return program_schedule_params - -def DecodeProgramSchedule(eval_dataset_names, - decode_steps_per_loop, - experimental_decoder=False, - train_program_cls=TrainProgram): - """Convenient helper method for common case. - - Args: - train_dataset_name: Name of the training dataset, eg: 'Train' - train_steps_per_loop: Number of steps to execute the training program. - eval_dataset_names: List of eval dataset_name strings, eg: ['Train']. - eval_steps_per_loop: Number of steps to execute the eval program. - decode_steps_per_loop: Number of steps to execute the decode program. - experimental_decoder: bool. Whether to use experimental deocder which is - placed in a tpu loop. - train_program_cls: The class to use for training programs. Defaults - to TrainProgram. - eval_program_cls: The class to use for eval programs. Defaults to - EvalProgram. - - Returns: - A populated SimpleProgramSchedule.Params() - """ - - program_schedule_params = SimpleProgramSchedule.Params() - train_program_params = train_program_cls.Params() - train_program_params.name = 'train' - train_program_params.steps_per_loop = 0 - train_program_params.dataset_name = "Train" - program_schedule_params.train_program = train_program_params - program_schedule_params.train_executions_per_eval = 0 - - program_schedule_params.dataset_names = [] - - for dataset_name in eval_dataset_names: - program_schedule_params.dataset_names.append(dataset_name) - if decode_steps_per_loop > 0: - decoder = ( - ExperimentalDecodeProgram if experimental_decoder else DecodeProgram) - decode_program_params = decoder.Params() - # Yes! This is what I want! - decode_program_params.name = 'decode_tpu' - # TODO(blee): This should be derived from the Dataset size. - # GALVEZ: Yes I agree - decode_program_params.steps_per_loop = decode_steps_per_loop - decode_program_params.dataset_name = dataset_name - program_schedule_params.eval_programs.append(decode_program_params) - return program_schedule_params - - -class MLPerfProgramSchedule: - """Program schedule for ML Perf benchmark.""" - - @classmethod - def Params(cls): - """Params for a MLPerfProgramSchedule.""" - p = hyperparams.InstantiableParams(cls) - - p.Define('task_dict', None, 'dataset_name -> task params') - p.Define('task_name', None, 'High level task name') - p.Define('logdir', None, 'Log directory') - p.Define('train_program', None, 'Train program params') - p.Define('train_executions_per_eval', 1, '') - p.Define('dataset_names', [], 'List of all dataset names.') - p.Define('num_splits_per_client', None, '') - - p.Define('ml_perf', hyperparams.Params(), 'MlPerf configuration.') - - mlp = p.ml_perf - mlp.Define('benchmark_name', None, 'Benchmark name for compliance log.') - mlp.Define('decoder_metric_name', None, - 'Name of the decoder metric to report for compliance log.') - mlp.Define('decoder_metric_success_threshold', None, - 'Benchmark run must exceed this value to succeeed.') - mlp.Define('steps_per_epoch', None, 'Number of training steps per epoch.') - mlp.Define('global_batch_size', None, 'Global batch size.') - mlp.Define('max_sequence_length', None, 'Maximum sequence length.') - mlp.Define('optimizer_name', None, 'Optimizer used.') - mlp.Define('base_learning_rate', None, 'Base learning rate.') - mlp.Define('warmup_steps', None, 'Number of warm-up steps.') - - return p - - def __init__(self, params, shared_model=None): - self.params = params.Copy() - p = self.params - self._shared_model = shared_model - - # Propagate run-time parameters to programs: - p.train_program.logdir = p.logdir - if p.train_program.train_dataset_name not in p.task_dict: - tf.logging.error('could not find %s in %s' % - (p.train_program.train_dataset_name, p.task_dict)) - - if p.train_program.decode_dataset_name not in p.task_dict: - tf.logging.error('could not find %s in %s' % - (p.train_program.decode_dataset_name, p.task_dict)) - - p.train_program.train_task = p.task_dict[p.train_program.train_dataset_name] - p.train_program.decode_task = p.task_dict[ - p.train_program.decode_dataset_name] - - p.train_program.num_splits_per_client = p.num_splits_per_client - p.train_program.task_name = p.task_name - p.train_program.ml_perf = p.ml_perf.Copy() - - self.train_program = p.train_program.Instantiate(shared_model=shared_model) - self._programs = [] - self._programs.append(self.train_program) - - def Programs(self): - return self._programs - - def Run(self, sess): - p = self.params - for _ in range(p.train_executions_per_eval): - program_done = self.train_program.Run(sess) - if program_done: - return True - return False - - -def MLPerfProgramScheduleForTask(train_dataset_name, train_steps_per_loop, - decode_dataset_name, decode_steps_per_loop): - """Populate MLPerfProgramSchedule params. - - Args: - train_dataset_name: Name of the training dataset, eg: 'Train'. - train_steps_per_loop: Number of steps to execute the training program. - decode_dataset_name: Eg: 'Test'. - decode_steps_per_loop: Number of steps to execute the decode program. - - Returns: - A populated MLPerfProgramSchedule.Params() - """ - - program_schedule_params = MLPerfProgramSchedule.Params() - train_program_params = MLPerfTrainDecodeProgram.Params() - train_program_params.name = 'train_and_decode' - train_program_params.train_steps_per_loop = train_steps_per_loop - train_program_params.decode_steps_per_loop = decode_steps_per_loop - - train_program_params.dataset_name = train_dataset_name - train_program_params.train_dataset_name = train_dataset_name - train_program_params.decode_dataset_name = decode_dataset_name - - program_schedule_params.train_program = train_program_params - - program_schedule_params.dataset_names = [ - train_dataset_name, decode_dataset_name - ] - - return program_schedule_params diff --git a/lingvo/core/pruning_utils.py b/lingvo/core/pruning_utils.py deleted file mode 100644 index 14a30668..00000000 --- a/lingvo/core/pruning_utils.py +++ /dev/null @@ -1,42 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Utilities for pruning.""" - -import lingvo.compat as tf -from model_pruning.python import pruning - - -def AddToPruningCollections(weight, - mask, - threshold, - gradient=None, - old_weight=None, - old_old_weight=None): - """Add mask, threshold, and weight vars to their respective collections.""" - if mask not in tf.get_collection(pruning.MASK_COLLECTION): - tf.add_to_collection(pruning.WEIGHT_COLLECTION, weight) - tf.add_to_collection(pruning.MASK_COLLECTION, mask) - tf.add_to_collection(pruning.THRESHOLD_COLLECTION, threshold) - - # Add gradient, old_weight, and old_old_weight to collections approximating - # gradient and hessian, where old_weight is the weight tensor one step - # before and old_old_weight is the weight tensor two steps before. - if gradient is not None: - assert old_weight is not None - assert old_old_weight is not None - tf.add_to_collection(pruning.WEIGHT_GRADIENT_COLLECTION, gradient) - tf.add_to_collection(pruning.OLD_WEIGHT_COLLECTION, old_weight) - tf.add_to_collection(pruning.OLD_OLD_WEIGHT_COLLECTION, old_old_weight) diff --git a/lingvo/core/py_utils.py b/lingvo/core/py_utils.py deleted file mode 100644 index ad2a3f46..00000000 --- a/lingvo/core/py_utils.py +++ /dev/null @@ -1,5428 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Common utilities.""" - -# ============================================================================== -# Note: Avoid adding dependencies to py_utils beyond standard python packages -# and tensorflow. -# ============================================================================== - -import collections as py_collections -import contextlib -import functools -import hashlib -import inspect -import math -import numbers -import os -import pkgutil -import re -import threading -import traceback - -import lingvo.compat as tf -from lingvo.core import hyperparams -from lingvo.core import ops -from lingvo.core import retry -from lingvo.core import symbolic -from lingvo.core import tshape -import numpy as np -import six - -# pylint: disable=g-direct-tensorflow-import -from tensorflow.core.framework import node_def_pb2 -from tensorflow.core.protobuf import rewriter_config_pb2 -from tensorflow.python.framework import func_graph -from tensorflow.python.framework import function -from tensorflow.python.ops import init_ops -from tensorflow.python.ops import stateless_random_ops -from tensorflow.python.tpu import topology as tf_topology -from tensorflow.python.tpu import tpu_function -from tensorflow.python.util import deprecation -# pylint: enable=g-direct-tensorflow-import - -tf.flags.DEFINE_bool('enable_asserts', True, - 'If False, we disable all asserts.') - -tf.flags.DEFINE_bool('enable_check_numerics', True, - 'If False, we bypass calls to CheckNumerics.') - -tf.flags.DEFINE_bool('print_debug_tensors', False, - 'Whether to print debug tensors.') - -tf.flags.DEFINE_string( - 'xla_device', '', 'If non-empty, can be cpu, gpu, or tpu (case sensitive)') - -tf.flags.DEFINE_bool( - 'use_resource_var', True, - 'Use ResourceVariable instead of RefVariable; this option is ' - 'enabled by default and will be removed in the future.') - -tf.flags.DEFINE_bool( - 'tpu_compatible', False, 'Create variables in a way compatible with TPU. ' - 'This should be true for any job that will interact ' - 'with variables or a checkpoint that will be produced ' - 'or consumed by TPU') - -tf.flags.DEFINE_bool( - 'tflite_compatible', False, - 'Uses tflite converter-friendly ops at applicable places. This so far ' - '(08/2020) is a only best-effort option.') - -tf.flags.DEFINE_bool( - 'pin_vars_to_cpu', False, - 'Pin variables to cpu:0. This is useful for weight-sharing / multi-core ' - 'inference on TPUs in which TPU core variables are managed via ' - 'TPUPartitionedCallOp.') - -tf.flags.DEFINE_bool( - 'no_identity_on_vars', False, - 'Do not add tf.identity() on vars. This allows TPUPartitionedCallOp to use' - 'variable handles directly for weight-sharing / multi-core ' - 'inference on TPUs.') - -tf.flags.DEFINE_bool('disable_py_utils_debug', False, - 'If True disables all py_utils.Debug() logs.') - -# TODO(laigd): remove these after the migration. -tf.flags.DEFINE_bool('if_use_tf_function', False, - 'If True use tf.function for py_utils.If().') -tf.flags.DEFINE_bool('while_loop_use_tf_function', False, - 'If True use tf.function for py_utils.WhileLoop().') -tf.flags.DEFINE_bool('call_defun_use_tf_function', False, - 'If True use tf.function for py_utils.CallDefun().') - -tf.flags.DEFINE_bool( - 'stateless_vars_init', False, - 'Use stateless TensorFlow random number generators (RNG) (e.g. ' - 'tf.random.stateless_uniform) to initialize variables instead of the ' - 'default ones (e.g. tf.random.uniform). This is useful to make variable ' - 'initialization deterministic on different replicas such as on TPUs, ' - 'since XLA does not fully respect the contract with respect to ' - 'user-specified seeds, when using TensorFlow stateful RNGs.') - -# NOTE: Using absl flags in libraries are frowned upon for several reasons: -# -# 1) They require app.run() or explicit flag parsing, preventing the use of -# these libraries in environments that don't look like normal binaries (colab -# notebooks). -# -# 2) They are process-level globals that cannot be scoped or configured except -# once during binary startup. -# -# Because py_utils is a library, no more flags should be used in this file; the -# existing flags are present for backwards compatibility. Instead, consider -# using a stack-scoped configuration object such as the Cluster object. We guard -# against issue 1 above by using _FromGlobal below, which uses the default value -# of the FLAG even if flags are unparsed. - -FLAGS = tf.flags.FLAGS - - -def _FromGlobal(field_name): - """Get 'field_name' from a global configuration object. - - Currently the global configuration object used is FLAGS, but this may - change to Cluster() or an equivalent stack-scoped config object. - - Args: - field_name: The string field name to look up. - - Returns: - The value associated with the global configuration string 'field_name'. - """ - # TODO(b/145831327): check the field name in the current cluster object. - # If explicitly set, use that value instead of using the FLAG value. - - # Now check the FLAGS object for backwards compatibility. - # - # If not explicitly set, get the field from the FLAGS object. If FLAGS - # have not been parsed yet, the default value of the flag will be used. - return FLAGS[field_name].value - - -ENQUEUE_OPS = '__lingvo_enqueue_ops' - -TPU_EMBEDDING_LOAD_OPS = '__lingvo_tpu_embedding_load_ops' -TPU_EMBEDDING_RETRIEVE_OPS = '__lingvo_tpu_embedding_retrieve_ops' -TPU_EMBEDDING = '__tpu_embedding' -TPU_EMBEDDING_ACTIVATIONS = '__tpu_embedding_activations' - -# pylint: disable=protected-access -deprecation._PRINT_DEPRECATION_WARNINGS = False - -# pylint: enable=protected-access - - -class ThreadLocalStack(threading.local): - - def __init__(self): - super().__init__() - self.stack = [] - - -def Assert(condition, data, *args, **kwargs): - if _FromGlobal('enable_asserts'): - return tf.Assert(condition, data, *args, **kwargs) - else: - return tf.no_op() - - -def assert_equal(*args, **kwargs): # pylint: disable=invalid-name - if _FromGlobal('enable_asserts'): - return tf.assert_equal(*args, **kwargs) - else: - return tf.no_op() - - -def assert_greater_equal(*args, **kwargs): # pylint: disable=invalid-name - if _FromGlobal('enable_asserts'): - return tf.debugging.assert_greater_equal(*args, **kwargs) - else: - return tf.no_op() - - -def assert_greater(*args, **kwargs): # pylint: disable=invalid-name - if _FromGlobal('enable_asserts'): - return tf.assert_greater(*args, **kwargs) - else: - return tf.no_op() - - -def assert_less_equal(*args, **kwargs): # pylint: disable=invalid-name - if _FromGlobal('enable_asserts'): - return tf.debugging.assert_less_equal(*args, **kwargs) - else: - return tf.no_op() - - -def assert_less(*args, **kwargs): # pylint: disable=invalid-name - if _FromGlobal('enable_asserts'): - return tf.assert_less(*args, **kwargs) - else: - return tf.no_op() - - -def assert_between(x, l, r, *args, **kwargs): # pylint: disable=invalid-name - x = tf.convert_to_tensor(x) - l = tf.cast(tf.convert_to_tensor(l), x.dtype) - r = tf.cast(tf.convert_to_tensor(r), x.dtype) - return tf.group([ - assert_greater_equal(x, l, *args, **kwargs), - assert_less(x, r, *args, **kwargs) - ]) - - -def assert_shape_match(*args, **kwargs): # pylint: disable=invalid-name - if _FromGlobal('enable_asserts'): - filepath, line, func, _ = traceback.extract_stack(limit=3)[-2] - kwargs['msg'] = 'LINGVO ASSERT %s:%s(%s)' % (re.sub( - r'.*/', '', filepath), line, func) - return ops.assert_shape_match(*args, **kwargs) - else: - return tf.no_op() - - -def assert_same_dim0(xs, *args, **kwargs): # pylint: disable=invalid-name - if _FromGlobal('enable_asserts'): - return ops.assert_same_dim0(xs, *args, **kwargs) - else: - return tf.no_op() - - -def assert_even_divide(denorm, num): # pylint: disable=invalid-name - """Asserts that denorm is evenly divided by num.""" - denorm = tf.convert_to_tensor(denorm) - num = tf.convert_to_tensor(num) - - if denorm.dtype not in (tf.int32, tf.int64): - raise ValueError('denorminator.dtype is not tf.int32 or tf.int64.') - if num.dtype not in (tf.int32, tf.int64): - raise ValueError('numerator.dtype is not tf.int32 or tf.int64.') - - num = HasShape(num, GetShape(denorm)) - - quo = denorm // num - return assert_equal(quo * num, denorm) - - -def AssertIdShape(expected_ids_shape_pattern, ids_shape, *args): - """Asserts shape expected_ids_shape_pattern matches all other input shapes.""" - - def AssertFn(inputs): - dependencies = [ - assert_shape_match(inputs.ids_shape, inputs.expected_ids_shape_pattern) - ] + [ - assert_shape_match(inputs.ids_shape, x_shape) for x_shape in inputs.args - ] - return with_dependencies(dependencies, inputs.ids_shape) - - inputs = NestedMap( - expected_ids_shape_pattern=expected_ids_shape_pattern, - ids_shape=ids_shape, - args=args) - return CallDefun(AssertFn, Transform(tf.convert_to_tensor, inputs)) - - -def _CheckNumerics(x, message=None, *args, **kwargs): - if x.dtype.is_floating: - if 'name' not in kwargs: - kwargs['name'] = re.sub(r':\d+', '', x.name) + '_CheckNumerics' - return tf.debugging.check_numerics(x, message if message else x.name, *args, - **kwargs) - else: - return x - - -def CheckNumerics(inp, message=None, *args, **kwargs): - """Check numerics for tensors in inp.""" - if not _FromGlobal('enable_check_numerics'): - return inp - if isinstance(inp, list): - return [_CheckNumerics(x, message, *args, **kwargs) for x in inp] - if isinstance(inp, tuple): - return tuple(_CheckNumerics(x, message, *args, **kwargs) for x in inp) - return _CheckNumerics(inp, message, *args, **kwargs) - - -def with_dependencies(dependencies, output_tensor): # pylint: disable=invalid-name - with tf.control_dependencies(dependencies): - return tf.identity(output_tensor) - - -@contextlib.contextmanager -def _PrintOptions(*args, **kwargs): - original = np.get_printoptions() - np.set_printoptions(*args, **kwargs) - try: - yield - finally: - np.set_printoptions(**original) - - -def _Print(name, x): - with _PrintOptions(linewidth=1000): - tf.logging.info('%s = %s', name, np.array_repr(x)) - - -def Log(value, prefix, **kwargs): - """Prints out values of tensors. - - Useful for debugging. E.g., - x = ... a tf.Tensor ... - y = ... a tf.Tensor ... - z = compute(x, y) - z = Log(z, 'debug compute()', x=x, y=y) - - Args: - value: A Tensor. Log happens after this tensor's computed. - prefix: Every tensor is logged with this prefix. - **kwargs: keywords and tensors. Tensors are logged in the sort order of - these keywards. - - Returns: - value is returned. - """ - - # Ensures tensors are printed in order. - last = value - for k in sorted(kwargs): - with tf.control_dependencies([last]): - last = tf.py_func(_Print, [prefix + ' : ' + k, kwargs[k]], []) - with tf.control_dependencies([last]): - return tf.identity(value) - - -def Debug(tensor, message='', enabled=True, summarize=100, more=None): - """Wrapper around tf.Print() and tf.logging.info() to simplify debug printing. - - x = py_utils.Debug(x) - - When the graph is built a regular log info line will be printed: - -DBG- py_utils_test.py:429 x=Tensor(... - - Then when the tensor node is evaluated it will print lines like: - -DBG- py_utils_test.py:429 x Const:0[x.shape=][2 2][x=][[1 2][3 4]] - - WARNING: The code that parses local variable names can fail. E.g. don't write - two Debug() calls on one line or a Debug() call that spans more than one line. - - Args: - tensor: A tensor to print. - message: A message to print. - enabled: To enable the debugging. - summarize: Integer with number of tensor values to print. - more: An optional list of additional tensors. - - Returns: - The tensor. - """ - if not enabled or _FromGlobal('disable_py_utils_debug'): - return tensor - - if more is None: - more = [] - - stack = inspect.stack()[1][0] - caller = inspect.getframeinfo(stack) - - caller_var = '' - caller_more_vars = [] - if caller.code_context: - # Rough and likely to fail. But better than nothing. - caller_var = re.compile(r'Debug\((.*?)(\)|,).*$').search( - caller.code_context[0]).groups()[0] - if more: - more_vars = re.compile(r'more=\[(.*?)\].*$').search( - caller.code_context[0]).groups()[0] - caller_more_vars = more_vars.split(',') - - the_class = '' - if 'self' in stack.f_locals: - the_class = stack.f_locals['self'].__class__.__name__ - header = '-DBG- {}:{}:{}:{} {} '.format( - os.path.basename(caller.filename), the_class, caller.function, - caller.lineno, message) - - info = '{}{}={}'.format(header, caller_var, tensor) - for name, val in zip(caller_more_vars, more): - info += ' {}={}'.format(name.strip(), val) - tf.logging.info(info) - - if isinstance(tensor, tf.Tensor): - tensors = [] - tensors += [tf.constant('{}.shape='.format(caller_var)), tf.shape(tensor)] - for name, val in zip(caller_more_vars, more): - tensors += [tf.constant('{}.shape='.format(name.strip())), tf.shape(val)] - - tensors += [tf.constant('{}='.format(caller_var)), tensor] - for name, val in zip(caller_more_vars, more): - tensors += [tf.constant('{}='.format(name.strip())), val] - - info = '{}{} {}'.format(header, caller_var, tensor.name) - return tf.Print(tensor, tensors, info, summarize=summarize) - - return tensor - - -def _Save(steps, prefix, key, val): - filename = '%s.%08d.%s.npy' % (six.ensure_text(prefix), steps, - six.ensure_text(key)) - with tf.io.gfile.GFile(filename, 'w') as outfile: - np.save(outfile, val) - - -def Save(value, filename_prefix, **kwargs): - """Saves values of tensors into files. - - Useful for debugging. E.g., - x = ... a tf.Tensor ... - y = ... a tf.Tensor ... - z = compute(x, y) - z = Save(z, '/path/tmp', x=x, y=y, z=z) - - Args: - value: A Tensor. Saving happens after this tensor is computed. - filename_prefix: Every tensor is saved with this filename prefix. - **kwargs: keywords and tensors. Tensors are logged in the sort order of - these keywards. - - Returns: - value is returned. - """ - last = value - steps = GetGlobalStep() - for k in sorted(kwargs): - with tf.control_dependencies([last]): - last = tf.py_func(_Save, [steps, filename_prefix, k, kwargs[k]], []) - with tf.control_dependencies([last]): - return tf.identity(value) - - -def HasRank(tensor, expected_rank): - """Syntactic sugar for asserting that tensor has the expected rank.""" - if tensor.shape.ndims is not None and isinstance(expected_rank, int): - assert tensor.shape.ndims == expected_rank, ( - 'Ranks did not match, got %d, ' - 'expected %d') % (tensor.shape.ndims, expected_rank) - return tensor - if _FromGlobal('enable_asserts'): - return with_dependencies([tf.assert_equal(tf.rank(tensor), expected_rank)], - tensor) - else: - return tensor - - -def HasAtLeastRank(tensor, expected_rank): - """Syntactic sugar for asserting that tensor has rank >= expected_rank.""" - if tensor.shape.ndims is not None and isinstance(expected_rank, int): - assert tensor.shape.ndims >= expected_rank, ( - 'Rank of tensor %d did not exceed the expected value %d.') % ( - tensor.shape.ndims, expected_rank) - return tensor - if _FromGlobal('enable_asserts'): - return with_dependencies( - [tf.debugging.assert_greater_equal(tf.rank(tensor), expected_rank)], - tensor) - else: - return tensor - - -def GetRank(tensor): - """Returns tensor's rank as an int if it's available, otherwise a Tensor. - - Args: - tensor: The input tensor. - - Returns: - Either an int or a Tensor for the rank of the input tensor. - """ - if tensor.shape.ndims is not None: - return tensor.shape.ndims # int - else: - return tf.rank(tensor) # Tensor - - -def GetShape(tensor, ndims=None): - """Returns tensor's shape as a list which can be unpacked, unlike tf.shape. - - Tries to return static shape if it's available. Note that this means - some of the outputs will be ints while the rest will be Tensors. - - Args: - tensor: The input tensor. - ndims: If not None, returns the shapes for the first `ndims` dimensions. - """ - tensor = tf.convert_to_tensor(tensor) - dynamic_shape = tf.shape(tensor) - - # Early exit for unranked tensor. - if tensor.shape.ndims is None: - if ndims is None: - return dynamic_shape - else: - return [dynamic_shape[x] for x in range(ndims)] - - # Ranked tensor. - if ndims is None: - ndims = tensor.shape.ndims - else: - ndims = min(ndims, tensor.shape.ndims) - - # Return mixture of static and dynamic dims. - static_shape = tensor.shape.as_list() - shapes = [ - static_shape[x] if static_shape[x] is not None else dynamic_shape[x] - for x in range(ndims) - ] - return shapes - - -def HasShape(tensor, expected_shape, ndims=None): - """Syntactic sugar for asserting that tensor has the expected shape. - - Args: - tensor: A Tensor. - expected_shape: A Python list or a 1D tensor. Elements of expected_shape can - be -1 which indicate that any size is valid for that dimension. - ndims: If not None, check only the first `ndims` dimensions of `tensor`. - Must be equal to the length of `expected_shape` if not None. - - Returns: - The input `tensor` with control dependencies that will raise a runtime - error if dynamic shape checks fail. - - Raises: - ValueError: A value error if the assertion fails at static shape checks. - """ - if not _FromGlobal('enable_asserts'): - return tensor - - filepath, line, func, _ = traceback.extract_stack(limit=3)[-2] - msg = 'LINGVO ASSERT %s:%s(%s)' % (re.sub(r'.*/', '', - filepath), line, func) - - tensor_shape = GetShape(tensor) - if ndims is not None: - tensor_shape = tensor_shape[:ndims] - - # TODO(jngiam): Attempt to switch back to tf.Assert after it has better - # support on GPUs. - assert_op = ops.assert_shape_match(tensor_shape, expected_shape, msg=msg) - - # If expected_shape is a Tensor, then we are unable to perform static checks. - # In this case, we can do a dynamic check and return. - if isinstance(expected_shape, tf.Tensor): - return with_dependencies([assert_op], tensor) - - # Infer ranks from the inputs. - expected_rank = len(expected_shape) - if isinstance(tensor_shape, tf.Tensor): - tensor_rank = tensor.shape.ndims - else: - tensor_rank = len(tensor_shape) - - # If ndims is None, then either one of the ranks should not be None, or they - # should both match. If both ranks are None, then they are both tensors and - # should be caught by the earlier short-circuit. - if ndims is None: - if (tensor_rank is not None) and (expected_rank != tensor_rank): - raise ValueError('Tensor does not match rank of expected shape.\n' - 'Tensor shape: {} Expected shape: {}'.format( - tensor_shape, expected_shape)) - # Both tensors can be assumed to be of same rank. - ndims = expected_rank - else: - if (tensor_rank is not None) and (tensor_rank < ndims): - raise ValueError('Tensor has fewer dimensions than ndims.\n' - 'Tensor shape: {} ndims: {}'.format(tensor_shape, ndims)) - if expected_rank != ndims: - raise ValueError( - 'Expected shape must have number of dimensions equal to ndims.\n' - 'Expected shape: {} ndims: {}'.format(expected_shape, ndims)) - - # Ensure that both tensor_shape and expected_shape are both lists. - tensor_shape = tensor_shape[:ndims] - if isinstance(tensor_shape, tf.Tensor): - tensor_shape = tf.unstack(tensor_shape, num=ndims) - - # Map tf.Dimension values to their held values. - tensor_shape = [ - v.value if isinstance(v, tf.Dimension) else v for v in tensor_shape - ] - expected_shape = [ - v.value if isinstance(v, tf.Dimension) else v for v in expected_shape - ] - - all_static_checks = True - for idx, (dim, expected_dim) in enumerate(zip(tensor_shape, expected_shape)): - if expected_dim == -1: - continue - if isinstance(dim, tf.Tensor) or isinstance(expected_dim, tf.Tensor): - all_static_checks = False - elif dim != expected_dim: - raise ValueError('Tensor does not match expected shape on dimension {}.\n' - 'Tensor shape: {} Expected shape: {}'.format( - idx, tensor_shape, expected_shape)) - - if all_static_checks: - return tf.convert_to_tensor(tensor) - else: - return with_dependencies([assert_op], tensor) - - -def GetSize(tensor): - shape = GetShape(tensor) - if (isinstance(shape, tf.Tensor) or - any([isinstance(x, tf.Tensor) for x in shape])): - return tf.size(tensor) - return np.prod(shape) - - -def CausalSelfAttenPadding(seqlen, dtype): - """Wraps tf.linalg.band_part() for tflite compatibility.""" - if FLAGS.tflite_compatible: - # [N, 1] - rows = tf.expand_dims(tf.range(seqlen), -1) - # [1, N] - cols = tf.expand_dims(tf.range(seqlen), 0) - row_cols = rows - cols - return tf.where(row_cols < 0, tf.ones([seqlen, seqlen], dtype), - tf.zeros([seqlen, seqlen], tf.float32)) - else: - return 1.0 - tf.linalg.band_part( - tf.ones([seqlen, seqlen], dtype=dtype), -1, 0) - - -def use_xla(): # pylint: disable=invalid-name - res = _FromGlobal('xla_device') - if res: - assert res in ('', 'cpu', 'gpu', 'tpu') - return res - - -def use_tpu(): # pylint: disable=invalid-name - res = _FromGlobal('xla_device') == 'tpu' - if res: - assert not _FromGlobal('enable_asserts') # asserts not supported on tpu - return res - - -def tpu_compat(): # pylint: disable=invalid-name - return use_tpu() or _FromGlobal('tpu_compatible') - - -def use_resource_variables(): # pylint: disable=invalid-name - return _FromGlobal('use_resource_var') or tpu_compat() - - -def outside_all_rewrites(): # pylint: disable=invalid-name - return tf.control_dependencies(None) - - -def use_stateless_vars_init(): # pylint: disable=invalid-name - return _FromGlobal('stateless_vars_init') - - -# TODO(jamesqin): remove once b/147439702 is fixed. -_OUTSIDE_COMPILATION = threading.local() - - -def RunOnTpuHost(func, *args, **kwargs): - r"""Runs the given function call on TPU host. - - Invokes func(\*args, \*\*kwargs) directly if not running on tpu. - - Args: - func: the function to invoke. - *args: args of func - **kwargs: kwargs of func - - Returns: - The function return value. - """ - if use_tpu() and not getattr(_OUTSIDE_COMPILATION, 'on', False): - _OUTSIDE_COMPILATION.on = True - res = tf.tpu.outside_compilation(func, *args, **kwargs) - _OUTSIDE_COMPILATION.on = False - else: - res = func(*args, **kwargs) - return res - - -def tpu_host(func): # pylint: disable=invalid-name - r"""Decorates a python function to only run on TPU hosts. - - This function has no effect when running on CPU/GPU. - - Example:: - - @py_utils.tpu_host() - def ComputeWER(self): - # Call a custom op computing WER. - - Args: - func: the function to invoke - - Returns: - A TPU-host only function - """ - - def Wrapped(*args, **kwargs): - return RunOnTpuHost(func, *args, **kwargs) - - return Wrapped - - -_tpu_device_assignment = None - - -def SetTpuDeviceAssignment(tpu_device_assignment): - global _tpu_device_assignment - if _tpu_device_assignment is not None: - tf.logging.warning('tpu_device_assignment was already set, ' - 'overwriting with new assignment.') - _tpu_device_assignment = tpu_device_assignment - - -# This function should called in unittest only. -def ClearTpuDevice(): - global _tpu_device_assignment - _tpu_device_assignment = None - - -def GetTpuDeviceAssignment(): - return _tpu_device_assignment - - -def SessionConfig(soft_placement=True, - inline=True, - cluster_def=None, - disable_meta_optimizer=False): - """Returns a session config proto. - - Args: - soft_placement: Turns allow_soft_placement on iff True. - inline: Turns do_function_inlining on iff True. - cluster_def: A tf.train.ClusterDef describing the cluster. - disable_meta_optimizer: Turns off grappler/metagraph optimizer. - - Returns: - A TF session config proto. - """ - session_config = tf.config_pb2.ConfigProto( - allow_soft_placement=soft_placement, - graph_options=tf.GraphOptions( - optimizer_options=tf.OptimizerOptions( - opt_level=tf.OptimizerOptions.L1, do_function_inlining=inline)), - cluster_def=cluster_def) - - if disable_meta_optimizer: - # Useful if start-up time is critical. - session_config.graph_options.rewrite_options.disable_meta_optimizer = True - # Disable layout optimizer which increases GPU memory usage. - session_config.graph_options.rewrite_options.layout_optimizer = ( - rewriter_config_pb2.RewriterConfig.OFF) - return session_config - - -def AssertIsCompatible(a, b): - assert a.IsCompatible(b), ('%s vs %s' % (a, b)) - - -def SetShapes(dst_nmap, src_nmap): - """Set shapes in dst_nmap using those in src_nmap.""" - AssertIsCompatible(src_nmap, dst_nmap) - for src, dst in zip(src_nmap.Flatten(), dst_nmap.Flatten()): - dst.set_shape(src.shape) - - -def Dtypes(nmap_list): - """Returns all tensors' data types in a list.""" - return [v.dtype for v in Flatten(nmap_list)] - - -def Flatten(x): - """Flattens 'x' by extracting tensors from nested structures to a list.""" - return tf.nest.flatten(x) - - -def Pack(tmpl, values): - """Packs 'values' according to 'tmpl'.""" - return tf.nest.pack_sequence_as(tmpl, values) - - -def Transform(fn, *v): - """Replaces every nested value x in 'v' with fn(x) and returns the result.""" - return tf.nest.map_structure(fn, *v) - - -def ConvertNoneGradientToZeros(xs, dxs): - """Sanitize dxs so that None becomes zeros appropriately. - - Args: - xs: A list of tensors. - dxs: A list of tensors. dxs[i] corresponds to xs[i]'s gradient. - - Returns: - A `.NestedMap` same as dxs with None replaced by a zero tensor. - """ - fn = lambda x, dx: tf.zeros_like(x) if dx is None else dx - return Transform(fn, xs, dxs) - - -def IsCompatible(lhs, rhs): - """Returns true if lhs and rhs are compatible.""" - try: - tf.nest.assert_same_structure(lhs, rhs) - return True - except (ValueError, TypeError): - return False - - -_NAME_PATTERN = re.compile('[A-Za-z_][A-Za-z0-9_]*') - - -class NestedMap(dict): - """A simple helper to maintain a dict. - - It is a sub-class of dict with the following extensions/restrictions: - - It supports attr access to its members (see examples below). - - Member keys have to be valid identifiers. - - E.g.:: - - >>> foo = NestedMap() - >>> foo['x'] = 10 - >>> foo.y = 20 - >>> assert foo.x * 2 == foo.y - """ - - # Disable pytype attribute checking. - _HAS_DYNAMIC_ATTRIBUTES = True - # keys in this list are not allowed in a NestedMap. - _RESERVED_KEYS = frozenset(dir(dict)) - # sentinel value for deleting keys used in Filter. - _DELETE = object() - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - for key in self.keys(): - assert isinstance(key, str), ( - 'Key in a NestedMap has to be a six.string_types. Currently type: %s,' - ' value: %s' % (str(type(key)), str(key))) - NestedMap.CheckKey(key) - assert key not in NestedMap._RESERVED_KEYS, ('%s is a reserved key' % key) - - def __setitem__(self, key, value): - # Make sure key is a valid expression and is not one of the reserved - # attributes. - assert isinstance(key, str), ( - 'Key in a NestedMap has to be a six.string_types. Currently type: %s, ' - 'value: %s' % (str(type(key)), str(key))) - NestedMap.CheckKey(key) - assert key not in NestedMap._RESERVED_KEYS, ('%s is a reserved key' % key) - super().__setitem__(key, value) - - def __setattr__(self, name, value): - self.__setitem__(name, value) - - def __getattr__(self, name): - try: - return self[name] - except KeyError as e: - raise AttributeError('%s; available attributes: %s' % - (e, sorted(list(self.keys())))) - - def __delattr__(self, name): - try: - del self[name] - except KeyError as e: - raise AttributeError('%s; available attributes: %s' % - (e, sorted(list(self.keys())))) - - def copy(self): # Don't delegate w/ super: dict.copy() -> dict. - return NestedMap(self) - - def __deepcopy__(self, unused_memo): - """Deep-copies the structure but not the leaf objects.""" - return self.DeepCopy() - - def DeepCopy(self): - """Deep-copies the structure but not the leaf objects.""" - return self.Pack(self.Flatten()) - - @staticmethod - def FromNestedDict(x): - """Converts every dict in nested structure 'x' to a NestedMap.""" - if isinstance(x, dict): - res = NestedMap() - for k, v in x.items(): - res[k] = NestedMap.FromNestedDict(v) - return res - elif isinstance(x, (list, tuple)): - return type(x)(NestedMap.FromNestedDict(v) for v in x) - else: - return x - - @staticmethod - def CheckKey(key): - """Asserts that key is valid NestedMap key.""" - if not (isinstance(key, str) and _NAME_PATTERN.match(key)): - raise ValueError('Invalid NestedMap key \'{}\''.format(key)) - - def GetItem(self, key): - """Gets the value for the nested `key`. - - Note that indexing lists is not supported, names with underscores will be - considered as one key. - - Args: - key: str of the form - `([A-Za-z_][A-Za-z0-9_]*)(.[A-Za-z_][A-Za-z0-9_]*)*.`. - - Returns: - The value for the given nested key. - - Raises: - KeyError if a key is not present. - """ - current = self - # Note: This can't support lists. List keys are ambiguous as underscore is - # not reserved for list indexing but also allowed to be used in keys. - # E.g., this is a valid nested map where the key 'a_0' is not well defined - # {'a_0': 3, 'a': [4]}. - for k in key.split('.'): - current = current[k] - return current - - def Get(self, key, default=None): - """Gets the value for nested `key`, returns `default` if key does not exist. - - Note that indexing lists is not supported, names with underscores will be - considered as one key. - - Args: - key: str of the form - `([A-Za-z_][A-Za-z0-9_]*)(.[A-Za-z_][A-Za-z0-9_]*)*.`. - default: Optional default value, defaults to None. - - Returns: - The value for the given nested key or `default` if the key does not exist. - """ - try: - return self.GetItem(key) - # TypeError is raised when an intermediate item is a list and we try to - # access an element of it with a string. - except (KeyError, TypeError): - return default - - def Set(self, key, value): - """Sets the value for a nested key. - - Note that indexing lists is not supported, names with underscores will be - considered as one key. - - Args: - key: str of the form - `([A-Za-z_][A-Za-z0-9_]*)(.[A-Za-z_][A-Za-z0-9_]*)*.`. - value: The value to insert. - - Raises: - ValueError if a sub key is not a NestedMap or dict. - """ - current = self - sub_keys = key.split('.') - for i, k in enumerate(sub_keys): - self.CheckKey(k) - # We have reached the terminal node, set the value. - if i == (len(sub_keys) - 1): - current[k] = value - else: - if k not in current: - current[k] = NestedMap() - if not isinstance(current[k], (dict, NestedMap)): - raise ValueError('Error while setting key {}. Sub key "{}" is of type' - ' {} but must be a dict or NestedMap.' - ''.format(key, k, type(current[k]))) - current = current[k] - - def _RecursiveMap(self, fn, flatten=False): - """Traverse recursively into lists, dicts, and NestedMaps applying `fn`. - - Args: - fn: The function to apply to each item (leaf node). - flatten: If true, the result should be a single flat list. Otherwise the - result will have the same structure as this NestedMap. - - Returns: - The result of applying fn. - """ - - def Recurse(v, key=''): - """Helper function for _RecursiveMap.""" - if isinstance(v, dict): - ret = [] if flatten else type(v)() - deleted = False - for k in sorted(v.keys()): - res = Recurse(v[k], key + '.' + k if key else k) - if res is self._DELETE: - deleted = True - continue - elif flatten: - ret += res - else: - ret[k] = res - if not ret and deleted: - return self._DELETE - return ret - elif isinstance(v, list): - ret = [] - deleted = False - for i, x in enumerate(v): - res = Recurse(x, '%s[%d]' % (key, i)) - if res is self._DELETE: - deleted = True - continue - elif flatten: - ret += res - else: - ret.append(res) - if not ret and deleted: - return self._DELETE - return ret - else: - ret = fn(key, v) - if flatten: - ret = [ret] - return ret - - res = Recurse(self) - if res is self._DELETE: - return [] if flatten else NestedMap() - return res - - def Flatten(self): - """Returns a list containing the flattened values in the `.NestedMap`. - - Unlike py_utils.Flatten(), this will only descend into lists, dicts, and - NestedMaps and not tuples, or namedtuples. - """ - return self._RecursiveMap(lambda _, v: v, flatten=True) - - def FlattenItems(self): - """Flatten the `.NestedMap` and returns pairs in a list. - - Returns: - A list of pairs, where keys for nested entries will be - represented in the form of `foo.bar[10].baz`. - """ - return self._RecursiveMap(lambda k, v: (k, v), flatten=True) - - def Pack(self, lst): - """Returns a copy of this with each value replaced by a value in lst.""" - assert len(self.FlattenItems()) == len(lst), f"{lst} vs {self.FlattenItems()}" - v_iter = iter(lst) - return self._RecursiveMap(lambda unused_k, unused_v: next(v_iter)) - - def Transform(self, fn): - """Returns a copy of this `.NestedMap` with fn applied on each value.""" - return self._RecursiveMap(lambda _, v: fn(v)) - - def TransformWithKey(self, fn): - """Returns a copy of this `.NestedMap` with fn applied on each key/value.""" - return self._RecursiveMap(fn) - - def IsCompatible(self, other): - """Returns true if self and other are compatible. - - If x and y are two compatible `.NestedMap`, `x.Pack(y.Flatten())` produces y - and vice versa. - - Args: - other: Another `.NestedMap`. - """ - items = self._RecursiveMap(lambda k, _: k, flatten=True) - other_items = other._RecursiveMap(lambda k, _: k, flatten=True) # pylint: disable=protected-access - return items == other_items - - def Filter(self, fn): - """Returns a copy with entries where fn(entry) is True.""" - return self.FilterKeyVal(lambda _, v: fn(v)) - - def FilterKeyVal(self, fn): - """Returns a copy of this `.NestedMap` filtered by fn. - - If fn(key, entry) is True, the entry is copied into the returned NestedMap. - Otherwise, it is not copied. - - Args: - fn: a callable of (string, entry)->boolean. - - Returns: - A `.NestedMap` contains copied entries from this `'.NestedMap`. - """ - return self._RecursiveMap(lambda k, v: v if fn(k, v) else self._DELETE) - - def _ToStrings(self): - """Returns debug strings in a list for this `.NestedMap`.""" - kv = self.FlattenItems() - maxlen = max([len(k) for k, _ in kv]) if kv else 0 - return sorted([k + ' ' * (4 + maxlen - len(k)) + str(v) for k, v in kv]) - - def DebugString(self): - """Returns a debug string for this `.NestedMap`.""" - return '\n'.join(self._ToStrings()) - - def VLog(self, level=None, prefix=None): - """Logs the debug string at the level.""" - if level is None: - level = 0 - if prefix is None: - prefix = 'nmap: ' - for l in self._ToStrings(): - tf.logging.vlog(level, '%s %s', prefix, l) - - def __dir__(self): - """dir() that includes flattened keys in returned output.""" - keys = self._RecursiveMap(lambda k, v: k, flatten=True) - return keys + super().__dir__() - - -class _Unique: - """A helper to uniqify variables in a NestedMap.""" - - def __init__(self): - self._vset = set() - - def __call__(self, v): - if (v is None) or (id(v) in self._vset): - return False - else: - self._vset.add(id(v)) - return True - - -def ToUniqueList(nmap): - """Returns the flattened `nmap` with duplicates removed.""" - return nmap.Filter(_Unique()).Flatten() - - -def ReadOnlyAttrDictView(backing): - """Wraps a dict to provide a read-only view of its contents. - - Dict keys can also be accessed by attribute. - - Args: - backing: Dict-like object to wrap. - - Returns: - Read-only Mapping that can be accessed by index (['foo']) or attr (d.foo). - """ - - class Wrapper: - """Wrapper object.""" - - # Disable pytype attribute checking. - _HAS_DYNAMIC_ATTRIBUTES = True - - def __getitem__(self, key): - return backing[key] - - def __len__(self): - return len(backing) - - def __iter__(self): - return iter(backing) - - def __getattr__(self, key): - return backing[key] - - def __hasattr__(self, key): - return key in backing - - def __setattr__(self, key, value): - raise AttributeError('Dictionary is read-only.') - - def __setitem__(self, key, value): - raise AttributeError('Dictionary is read-only.') - - return Wrapper() - - -def ToStaticShape(shape): - """Converts 'shape' to a static shape.""" - if isinstance(shape, (list, tuple)): - shape = [ - dim.value if isinstance(dim, tf.Dimension) else dim for dim in shape - ] - static_shape = [] - for dim in shape: - if symbolic.IsExpr(dim): - static_shape.append(symbolic.ToStatic(dim)) - else: - static_shape.append(dim) - return static_shape - else: - return shape.value if isinstance(shape, tf.Dimension) else shape - - -def Zeros(shape, *args, **kwargs): - return tf.zeros(ToStaticShape(shape), *args, **kwargs) - - -class UniformSampler: - """A reservoir sampler. - - This class implements reservoir sampling: Given a limit of `num_samples` total - samples, this class maintains a uniform probability (1 / `num_samples`) of - keeping any item dynamically added to the sampler. - - See https://en.wikipedia.org/wiki/Reservoir_sampling for details. - """ - - def __init__(self, num_samples): - assert num_samples > 0 - self._num_samples = num_samples - self._num_seen_items = 0 - self._samples = [] - - def Add(self, item): - """Add item to sampler.""" - self._num_seen_items += 1 - - if len(self._samples) < self._num_samples: - self._samples.append(item) - return - - index = np.random.randint(0, self._num_seen_items) - if index < self._num_samples: - self._samples[index] = item - - @property - def samples(self): - """Fetch the current samples from the sampler.""" - return self._samples - - -class RNNCellStateInit: - """State initialization functions for RNN cell init state.""" - - @staticmethod - def _Params(method, seed): - p = hyperparams.Params() - p.Define('method', method, - 'Initialization method. Should be one of zeros, random_normal.') - p.Define('seed', seed, 'Random seed used to generate initial values.') - p.Freeze() - return p - - @staticmethod - def Zeros(): - """tf.zeros().""" - return RNNCellStateInit._Params('zeros', seed=None) - - @staticmethod - def RandomNormal(seed=None): - """tf.random.normal().""" - return RNNCellStateInit._Params('random_normal', seed) - - -def DefaultRNNCellStateInit(): - return RNNCellStateInit.Zeros() - - -def InitRNNCellState(shape, init=None, dtype=None, name=None, is_eval=False): - """Initial state definitions for RNN cell implementations. - - Args: - shape: A array of ints/symbols for specifying the shape of the state. - init: Hyperparameters as returned by one of the static implemetaitons in - RNNCellStateInit. - dtype: The dype of the states. Defaults to tf.float32. - name: A name for the operation. If --stateless_vars_init is set, this name - is used to generate a seed on a per-variable basis. Otherwise, this name - is optional. - is_eval: Bool, set to True if we need special behavior in eval mode. - - Returns: - A Tensor of the specified shape, and sampled from the distribution as - defined by the init parameters. - """ - shape = ToStaticShape(shape) - - if init is None: - init = DefaultRNNCellStateInit() - if dtype is None: - dtype = tf.float32 - - method = init.method - if ((method in ['zeros']) or (method in ['random_normal'] and is_eval)): - init_state = tf.zeros(shape=shape, dtype=dtype, name=name) - elif method in ['random_normal']: - if use_stateless_vars_init(): - if name is None: - raise ValueError('InitRNNCellState() requires a `name` argument when ' - '--stateless_vars_init is enabled.') - seed = _GenerateStatelessRngSeed(name, init.seed) - init_state = stateless_random_ops.stateless_random_normal( - shape=shape, dtype=dtype, name=name, seed=seed) - else: - init_state = tf.random.normal( - shape=shape, dtype=dtype, name=name, seed=init.seed) - else: - raise ValueError('Initialization method (%s) not supported.' % method) - - return init_state - - -class WeightInit: - """Static class providing weight initialization config params.""" - - @staticmethod - def _Params(method, scale, seed): - """Parameters of this class.""" - p = hyperparams.Params() - p.Define('method', method, 'Initialization method.') - p.Define('scale', scale, 'Initialization scale.') - p.Define('seed', seed, 'Random seed used to generate initial values.') - p.Freeze() - return p - - @staticmethod - def Gaussian(scale=1.0, seed=None): - """scale * tf.random.normal(0, 1.0).""" - return WeightInit._Params('gaussian', scale, seed) - - @staticmethod - def Uniform(scale=1.0, seed=None): - """scale * tf.random.uniform(-1.0, 1.0).""" - return WeightInit._Params('uniform', scale, seed) - - @staticmethod - def UniformPositive(scale=1.0, seed=None): - """scale * tf.random.uniform(0., 1.0).""" - return WeightInit._Params('uniform_positive', scale, seed) - - @staticmethod - def Xavier(scale=1.0, seed=None): - """Xavier initialization (x = sqrt(6. / (in + out)); [-x, x]).""" - return WeightInit._Params('xavier', scale, seed) - - @staticmethod - def XavierGaussian(scale=1.0, seed=None): - """tf.random.normal(0, sqrt(2.0 / (in + out))).""" - return WeightInit._Params('xavier_gaussian', scale, seed) - - @staticmethod - def XavierWithFixupParams(scale=1.0, - depth=1.0, - layers_per_residual_block=1.0, - seed=None): - """Xavier initialization with Fixup.""" - scale = scale * math.pow(depth, (-1.0 / (2 * layers_per_residual_block))) - return WeightInit._Params('xavier', scale, seed) - - @staticmethod - def GeoMeanXavier(scale=1.0, seed=None): - """A variant of Xavier (x = sqrt(3. / sqrt(in * out)); [-x, x]).""" - return WeightInit._Params('geo_mean_xavier', scale, seed) - - @staticmethod - def Constant(scale=1.0): - """scale.""" - return WeightInit._Params('constant', scale, 0) - - @staticmethod - def TruncatedGaussian(scale=1.0, seed=None): - """scale * tf.random.truncated_normal(0, 1.0).""" - return WeightInit._Params('truncated_gaussian', scale, seed) - - @staticmethod - def GaussianSqrtDim(scale=1.0, seed=None): - """scale * tf.random.normal(0, 1 / sqrt(dim0)).""" - return WeightInit._Params('gaussian_sqrt_dim', scale, seed) - - @staticmethod - def GaussianSqrtFanIn(scale=1.0, seed=None): - """scale * tf.random.normal(0, 1 / sqrt(fan_in)).""" - return WeightInit._Params('gaussian_sqrt_fanin', scale, seed) - - @staticmethod - def GaussianSqrtFanOut(scale=1.0, seed=None): - """scale * tf.random.normal(0, 1 / sqrt(fan_out)).""" - return WeightInit._Params('gaussian_sqrt_fanout', scale, seed) - - @staticmethod - def UniformSqrtDim(scale=1.0, seed=None): - """scale * tf.uniform(-1 / sqrt(dim0), 1 / sqrt(dim0)).""" - return WeightInit._Params('uniform_sqrt_dim', scale, seed) - - @staticmethod - def UniformUnitScaling(scale=1.0, seed=None): - """scale * sqrt(3) / sqrt(dim0) * tf.uniform(-1, 1).""" - return WeightInit._Params('uniform_unit_scaling', scale, seed) - - @staticmethod - def UniformUnitScalingFanAvg(scale=1.0, seed=None): - """Same as tf.variance_scaling_initializer() ... - - Samples are drawn from a uniform distribution within [-limit, limit], with - limit = sqrt(3 * scale / n) - - where - n = max(1., (fan_in + fan_out) / 2). - See tf.keras.initializers.VarianceScaling for details. - - Args: - scale: A Python float. - seed: A Python int or None. - - Returns: - A WeightInit param. - """ - return WeightInit._Params('uniform_unit_scaling_fan_avg', scale, seed) - - @staticmethod - def TruncatedGaussianSqrtDim(scale=1.0, seed=None): - """scale * tf.random.truncated_normal(0, 1 / sqrt(dim0)).""" - return WeightInit._Params('truncated_gaussian_sqrt_dim', scale, seed) - - @staticmethod - def TruncatedGaussianSqrtFanIn(scale=1.0, seed=None): - """scale * tf.random.truncated_normal(0, 1 / sqrt(fan_in)).""" - return WeightInit._Params('truncated_gaussian_sqrt_fanin', scale, seed) - - @staticmethod - def TruncatedGaussianSqrtFanOut(scale=1.0, seed=None): - """scale * tf.random.truncated_normal(0, 1 / sqrt(fan_out)).""" - return WeightInit._Params('truncated_gaussian_sqrt_fanout', scale, seed) - - @staticmethod - def KaimingUniformFanInRelu(scale=1.0, seed=None): - return WeightInit._Params('kaiming_uniform_fanin_relu', scale, seed) - - @staticmethod - def KaimingUniformFanInLeakyRelu(scale=np.sqrt(5.), seed=None): - return WeightInit._Params('kaiming_uniform_fanin_leakyrelu', scale, seed) - - -_DEFAULT_XAVIER_INIT = 1.000001 - - -def DefaultParamInit(): - # Here we use 1.000001 as a signature for user picking up the - # default param initializer. - return WeightInit.Xavier(_DEFAULT_XAVIER_INIT) - - -# TODO(rpang, jonathanasdf): explore adding _is_default to hyperparams.Param. -def IsDefaultParamInit(p): - return (p.method == 'xavier' and - abs(p.scale - _DEFAULT_XAVIER_INIT) < 1e-7 and p.seed is None) - - -def WeightParams(shape, init=None, dtype=None, collections=None): - """Returns a hyperparams for a weight variable given the shape/init/dtype.""" - if init is None: - init = WeightInit.Xavier(_DEFAULT_XAVIER_INIT) - if dtype is None: - dtype = tf.float32 - if collections is None: - collections = [] - p = hyperparams.Params() - p.Define('dtype', dtype, 'The weight data type.') - p.Define('shape', shape, 'The weight shape.') - p.Define('init', init, 'Initialization method.') - p.Define('collections', collections, - 'Variable collections this weight belongs to.') - return p - - -def FindNeeded(endpoints): - """List names of tensors and operations required to compute endpoints.""" - names_seen = set() - queue = [] - for e in Flatten(endpoints): - if isinstance(e, tf.Operation): - queue.append(e) - else: - queue.append(e.op) - while queue: - op = queue.pop() - name = op.name - if name not in names_seen: - names_seen.add(name) - names_seen.update((o.name for o in op.outputs)) - queue.extend(i.op for i in op.inputs) - queue.extend(op.control_inputs) - return names_seen - - -def FindNeededInList(tensor_list, endpoints): - """Return tensors from tensor_list needed to compute any of endpoints.""" - all_needed = FindNeeded(endpoints) - return [t for t in tensor_list if t.name in all_needed] - - -class _CollectionGetter: - """Get graph local value from a defined collection.""" - - def __init__(self, key, default_factory): - self._key = key - self._default_factory = default_factory - - def __call__(self): - collection = tf.get_collection(self._key) - if collection: - assert len(collection) == 1 - return collection[0] - value = self._default_factory() - tf.add_to_collection(self._key, value) - return value - - -def SanitizeScopeKey(key): - """Removes invalid symbols from name_scope keys.""" - if key.startswith('_'): - key = key[1:] - return key.replace('[', '_').replace(']', '') - - -# Global variable to control multitask variable reuse -# If False (default) the default tf.get_variable is used, that is: -# - Reusing scopes only allow getting existing variables -# - Non-reusing scopes only allow getting new variables -# With GetOpportunisticVariableReuse() == True: -# - Reusing scopes only allow getting existing variables, as usual -# - Non-reusing scopes reuse new variables or get new ones -_OPPORTUNISTIC_VARIABLE_REUSE = ThreadLocalStack() - - -@contextlib.contextmanager -def OpportunisticVariableReuseScope(enable_opportunistic_reuse=True): - _OPPORTUNISTIC_VARIABLE_REUSE.stack.append(enable_opportunistic_reuse) - try: - yield - finally: - _OPPORTUNISTIC_VARIABLE_REUSE.stack.pop() - - -def GetOpportunisticVariableReuse(): - """Get the current variable reuse setting.""" - return (_OPPORTUNISTIC_VARIABLE_REUSE.stack[-1] - if _OPPORTUNISTIC_VARIABLE_REUSE.stack else False) - - -_VARIABLE_RENAME_RULES = ThreadLocalStack() - - -@contextlib.contextmanager -def VariableRenameScope(renames): - """Append the renaming rules to the stack of renames. - - Args: - renames: pairs of (regexp, new_name_format). If the regexp matches, the - new_name_format will be interpolated using the matched groups. - - Yields: - scope in which the renaming rules are applied - """ - _VARIABLE_RENAME_RULES.stack.append(renames) - try: - yield - finally: - _VARIABLE_RENAME_RULES.stack.pop() - - -def GetVariableName(name): - """Get variable name after application of all renaming rules. - - Args: - name: untransformed variable name with scope_name prepended - - Returns: - name possibly modified using renaming rules - """ - matched = False - new_name = name - for renames in _VARIABLE_RENAME_RULES.stack: - for regexp, name_format in renames: - match = re.match(regexp, name) - if match: - if matched: - tf.logging.warning('Multiple matches for: %s', name) - matched = True - new_name = name_format % match.groups() - if new_name != name: - tf.logging.info("WARNING!!! Renaming variable '%s' to '%s'", name, new_name) - return new_name - - -def GenerateSeedFromName(name): - """Generate a random seed from a name string.""" - md5 = hashlib.md5() - md5.update(six.ensure_binary(name)) - return int(md5.hexdigest(), 16) % (2**31 - 1) - - -# To keep track of all the variables ever gets created by the CreateVariable -# routine below. -_ALL_VARS_KEY = ('__lingvo_all_vars',) - -_get_all_vars = _CollectionGetter(_ALL_VARS_KEY, lambda: {}) - -_VARIABLE_SHAPE_PREFIXES = ThreadLocalStack() - - -@contextlib.contextmanager -def VariableShapePrefixContext(shape_prefix): - """Add a shape prefix to variable created by CreateVariable(). - - Args: - shape_prefix: a positive integer of shape prefix. - - Yields: - None. - """ - assert shape_prefix > 0, ('%s' % shape_prefix) - _VARIABLE_SHAPE_PREFIXES.stack.append(shape_prefix) - try: - yield - finally: - _VARIABLE_SHAPE_PREFIXES.stack.pop() - - -def GetVariableShapePrefixes(): - """Return the list of shape prefixes for CreateVariable().""" - return _VARIABLE_SHAPE_PREFIXES.stack - - -def GetFanInFanOut(shape): - """Returns (fan_in, fan_out) of a weight variable of the give shape.""" - if not shape: - return None, None - if len(shape) < 1: - return 1, 1 - elif len(shape) == 1: - # Following _compute_fans() from TF's init_ops.py. - return shape[0], shape[0] - else: - receptive_field_size = 1 - for s in shape[:-2]: - receptive_field_size *= s - fan_in = shape[-2] * receptive_field_size - fan_out = shape[-1] * receptive_field_size - return fan_in, fan_out - - -_VARIABLE_CREATOR_STACK = ThreadLocalStack().stack - - -def _DefaultVariableCreator(**kwargs): - kwargs.pop('var_name', None) - kwargs.pop('var_params', None) - return tf.get_variable(**kwargs) - - -def _GetVariableCreator(): - fn = _DefaultVariableCreator - for wrapper in reversed(_VARIABLE_CREATOR_STACK): - fn = functools.partial(wrapper, fn) - return fn - - -@contextlib.contextmanager -def VariableCreatorScope(variable_creator): - """Yields a context around a variable_creator, used by `CreateVariable()`. - - The function must have the following signature:: - - def variable_creator(next_creator, **kwargs) - - The function may delegate variable creation to the next variable creator, or - return its own tf.Variable. - - This differs from tf.variable_creator_scope in that tf.variable_creator_scope - modifies a tf.Variable() call while this modifies a tf.get_variable() call. As - the code is migrated to TF2 and tf.get_variable() is deprecated, this may be - upgraded to using tf.variable_creator_scope instead. - - This differs from tf.variable_scope(custom_getter=variable_creator) in that - the kwargs passed can be manipulated. - - Variable creators are resolved from the outermost towards the innermost. - - The innermost variable creator function is tf.get_variable. - - The passed in kwargs must conform to what tf.get_variable accepts, with the - addition of `var_name` and `var_params`. - - Args: - variable_creator: A variable creator function. - """ - _VARIABLE_CREATOR_STACK.append(variable_creator) - try: - yield - finally: - _VARIABLE_CREATOR_STACK.pop() - - -# TODO(yonghui): Add support for partitioned Variables. -def CreateVariable(name, - params, - reuse=None, - trainable=True, - collections=None, - default_seed=None, - synchronization=tf.VariableSynchronization.AUTO, - aggregation=tf.VariableAggregation.NONE): - """Creates tf.Variable according to param_config. - - Args: - name: A string, name of the variable. - params: A WeightParams specifying the details of how this variable should be - constructed and initialized. - reuse: Whether or not to reuse an existing variable. It has the same - semantics as the reuse arg in tf.variable_scope. - trainable: Whether or not the variable is trainable. - collections: Override the default variable collection ( - tf.GraphKeys.GLOBAL_VARIABLES). - default_seed: Seed to use for initialization if not specified in params. - Used for deterministic initialization in tests. - synchronization: Indicates when a distributed a variable will be aggregated. - Accepted values are constants defined in the class - tf.VariableSynchronization. By default the synchronization is set to AUTO - and the current DistributionStrategy chooses when to synchronize. - aggregation: Indicates how a distributed variable will be aggregated. - Accepted values are constants defined in the class tf.VariableAggregation. - - Returns: - The created variable. - """ - if use_stateless_vars_init(): - return _CreateVariableStateless(name, params, reuse, trainable, collections, - default_seed, synchronization, aggregation) - else: - return _CreateVariableStateful(name, params, reuse, trainable, collections, - default_seed, synchronization, aggregation) - - -def _CreateVariableStateful(name, - params, - reuse=None, - trainable=True, - collections=None, - default_seed=None, - synchronization=tf.VariableSynchronization.AUTO, - aggregation=tf.VariableAggregation.NONE): - """Creates tf.Variable using TF stateful RNGs according to param_config. - - Args: - name: A string, name of the variable. - params: A WeightParams specifying the details of how this variable should be - constructed and initialized. - reuse: Whether or not to reuse an existing variable. It has the same - semantics as the reuse arg in tf.variable_scope. - trainable: Whether or not the variable is trainable. - collections: Override the default variable collection ( - tf.GraphKeys.GLOBAL_VARIABLES). - default_seed: Seed to use for initialization if not specified in params. - Used for deterministic initialization in tests. - synchronization: Indicates when a distributed a variable will be aggregated. - Accepted values are constants defined in the class - tf.VariableSynchronization. By default the synchronization is set to AUTO - and the current DistributionStrategy chooses when to synchronize. - aggregation: Indicates how a distributed variable will be aggregated. - Accepted values are constants defined in the class tf.VariableAggregation. - - Returns: - The created variable. - """ - p = params.Copy() - shape = tf.TensorShape(ToStaticShape(p.shape)).as_list() - if shape: - assert all([dim_size > 0 for dim_size in shape]), shape - dim0 = shape[0] - else: - dim0 = 1 - assert p.init.method == 'constant' or np.all(np.asarray(p.init.scale) >= 0) - method = p.init.method - scale = p.init.scale - seed = p.init.seed - - if IsDefaultParamInit(p.init): - tf.logging.warning( - 'WARNING!!! var %s is using the default xavier initializer.' - ' Make sure this is intended.', name) - - with tf.variable_scope(name) as scope: - var_name = GetVariableName(scope.name) - - if tf.get_default_graph().seed is not None: - # We are in a program/test which need determistic randomization. - if seed is None: - if default_seed is not None: - seed = default_seed - else: - # We are not given a per-variable random seed. We use hash of - # variable name as a stable random seed. - seed = GenerateSeedFromName(var_name) - - init_dtype = p.dtype.real_dtype - v_init = _CreateVarInitStateful(name, method, shape, dim0, seed, scale, - init_dtype) - - if p.dtype == tf.complex64: - - def ComplexWrapper(init): - - def _Wrapper(shape, dtype, partition_info): - del dtype - # A more complex alternative may be to use the init function for - # magnitudes and uniform random for phases instead. - shape = [2] + shape - value = init(shape, init_dtype, partition_info) - return tf.complex(value[0], value[1]) - - return _Wrapper - - v_init = ComplexWrapper(v_init) - - # Variable creators. - def MaybePinVarsToCpu(next_creator, **kwargs): - if _FromGlobal('pin_vars_to_cpu'): - with tf.device('/cpu:0'): - return next_creator(**kwargs) - return next_creator(**kwargs) - - def MaybeOpportunisticVariableReuse(next_creator, **kwargs): - try: - return next_creator(**kwargs) - except ValueError: # Possibly the variable already exists - if GetOpportunisticVariableReuse(): - with tf.variable_scope(tf.get_variable_scope(), reuse=tf.AUTO_REUSE): - return next_creator(**kwargs) - else: - raise - - def LingvoVariableCreator(next_creator, **kwargs): - """Lingvo variable creator.""" - # TODO(yonghui): Possibly get away from variable_scope and implement our own - # variable sharing mechanism. - with tf.variable_scope(name) as scope: - var_scope = tf.VariableScope( - scope.reuse, - custom_getter=scope.custom_getter, - caching_device=scope.caching_device, - use_resource=scope.use_resource or use_resource_variables()) - with tf.variable_scope(var_scope), tf.variable_scope(var_name, reuse=reuse): - var = next_creator(**kwargs) - - var_ref = var.experimental_ref() # For key in dict/set. - all_vars = _get_all_vars() - if var_ref in all_vars: - tf.logging.info('Reusing var %s', var.name) - cached = all_vars[var_ref] - assert cached == p.ToText(), ('Cached config:\n %s vs new config:\n %s' % - (cached, p.ToText())) - else: - tf.logging.info('Creating var %s shape=%s on device %s', var.name, - var.shape, var.device) - all_vars[var_ref] = p.ToText() - for col in p.collections: - tf.add_to_collection(col, var) - return var - - with VariableCreatorScope(LingvoVariableCreator): - with VariableCreatorScope(MaybeOpportunisticVariableReuse): - with VariableCreatorScope(MaybePinVarsToCpu): - var = _GetVariableCreator()( - var_name=var_name, - var_params=p, - name='var', - shape=GetVariableShapePrefixes() + list(shape), - dtype=p.dtype, - initializer=v_init, - collections=collections, - trainable=trainable, - validate_shape=True, - synchronization=synchronization, - aggregation=aggregation) - - return var - - -def _CreateVariableStateless(name, - params, - reuse=None, - trainable=True, - collections=None, - default_seed=None, - synchronization=tf.VariableSynchronization.AUTO, - aggregation=tf.VariableAggregation.NONE): - """Creates tf.Variable using TF stateless RNGs according to `params`. - - Args: - name: A string, name of the variable. - params: A WeightParams specifying the details of how this variable should be - constructed and initialized. - reuse: Whether or not to reuse an existing variable. It has the same - semantics as the reuse arg in tf.variable_scope. - trainable: Whether or not the variable is trainable. - collections: Override the default variable collection ( - tf.GraphKeys.GLOBAL_VARIABLES). - default_seed: Seed to use for initialization if not specified in params. - Used for deterministic initialization in tests. - synchronization: Indicates when a distributed a variable will be aggregated. - Accepted values are constants defined in the class - tf.VariableSynchronization. By default the synchronization is set to AUTO - and the current DistributionStrategy chooses when to synchronize. - aggregation: Indicates how a distributed variable will be aggregated. - Accepted values are constants defined in the class tf.VariableAggregation. - - Returns: - The created variable. - """ - p = params.Copy() - shape = tf.TensorShape(ToStaticShape(p.shape)).as_list() - if shape: - assert all([dim_size > 0 for dim_size in shape]), shape - dim0 = shape[0] - else: - dim0 = 1 - assert p.init.method == 'constant' or np.all(np.asarray(p.init.scale) >= 0) - method = p.init.method - scale = p.init.scale - seed = p.init.seed - - if IsDefaultParamInit(p.init): - tf.logging.warning( - 'WARNING!!! var %s is using the default xavier initializer.' - ' Make sure this is intended.', name) - - with tf.variable_scope(name) as scope: - var_name = GetVariableName(scope.name) - - user_seed = seed if seed is not None else default_seed - seed = _GenerateStatelessRngSeed(var_name, user_seed) - - init_dtype = p.dtype.real_dtype - v_init = _CreateVarInitStateless(name, method, shape, dim0, seed, scale, - init_dtype) - - if p.dtype == tf.complex64: - raise TypeError( - 'Stateless variable initialization does not support tf.complex64.') - - def LingvoVariableCreator(next_creator, **kwargs): - """Lingvo variable creator.""" - # TODO(yonghui): Possibly get away from variable_scope and implement our own - # variable sharing mechanism. - with tf.variable_scope(name) as scope: - var_scope = tf.VariableScope( - scope.reuse, - custom_getter=scope.custom_getter, - caching_device=scope.caching_device, - use_resource=scope.use_resource or use_resource_variables()) - with tf.variable_scope(var_scope), tf.variable_scope(var_name, reuse=reuse): - var = next_creator(**kwargs) - - var_ref = var.experimental_ref() # For key in dict/set. - all_vars = _get_all_vars() - if var_ref in all_vars: - tf.logging.info('Reusing var %s', var.name) - cached = all_vars[var_ref] - assert cached == p.ToText(), ('Cached config:\n %s vs new config:\n %s' % - (cached, p.ToText())) - else: - tf.logging.info('Creating var %s shape=%s on device %s', var.name, - var.shape, var.device) - all_vars[var_ref] = p.ToText() - for col in p.collections: - tf.add_to_collection(col, var) - return var - - with VariableCreatorScope(LingvoVariableCreator): - var = _GetVariableCreator()( - var_name=var_name, - var_params=p, - name='var', - shape=GetVariableShapePrefixes() + list(shape), - dtype=p.dtype, - initializer=v_init, - collections=collections, - trainable=trainable, - validate_shape=True, - synchronization=synchronization, - aggregation=aggregation) - - return var - - -def _RandomXavierUniformInitializer(method, scale, seed): - """Creates a random Xavier uniform initializer.""" - - def XavierUniform(shape, dtype, partition_info): - """Xavier initialization (x = sqrt(6. / (in + out)); scale*[-x, x]).""" - del partition_info # Unused. - if not shape: - raise ValueError('\'shape\' must not be \'None\' or 0 for XavierUniform') - fan_in, fan_out = GetFanInFanOut(shape) - if method == 'xavier': - limit = math.sqrt(6. / (fan_in + fan_out)) - elif method == 'geo_mean_xavier': - limit = math.sqrt(3. / math.sqrt(fan_in * fan_out)) - return scale * tf.random.uniform(shape, -limit, limit, dtype, seed) - - return XavierUniform - - -def _CreateVarInitStateful(name, method, shape, dim0, seed, scale, init_dtype): - """Creates variable initialization function for a stateful RNG.""" - if (method in [ - 'gaussian_sqrt_dim', 'uniform_sqrt_dim', 'truncated_gaussian_sqrt_dim' - ]): - if len(shape) > 2: - # This is probably not the right method to use when len(shape) > 2, - # e.g. dim0 will be 3 with a 3x3 conv2d kernel. - tf.logging.warning( - 'Initializing %s of shape %s with method %s: dim0=%s. ' - 'Make sure that it is intended.', name, shape, method, dim0) - scale *= 1.0 / math.sqrt(dim0) - - if method in ['gaussian_sqrt_fanin', 'truncated_gaussian_sqrt_fanin']: - fan_in, _ = GetFanInFanOut(shape) - if fan_in is not None: - scale *= 1.0 / math.sqrt(fan_in) - if method in ['gaussian_sqrt_fanout', 'truncated_gaussian_sqrt_fanout']: - _, fan_out = GetFanInFanOut(shape) - if fan_out is not None: - scale *= 1.0 / math.sqrt(fan_out) - if method == 'xavier_gaussian': - fan_in, fan_out = GetFanInFanOut(shape) - if fan_in is not None and fan_out is not None: - scale *= math.sqrt(2.0 / (fan_in + fan_out)) - - if method in [ - 'gaussian', 'gaussian_sqrt_dim', 'gaussian_sqrt_fanin', - 'gaussian_sqrt_fanout', 'xavier_gaussian' - ]: - v_init = init_ops.random_normal_initializer( - mean=0.0, stddev=scale, seed=seed, dtype=init_dtype) - elif method in ['uniform', 'uniform_sqrt_dim']: - v_init = init_ops.random_uniform_initializer( - minval=-scale, maxval=scale, seed=seed, dtype=init_dtype) - elif method in ['uniform_positive']: - v_init = init_ops.random_uniform_initializer( - minval=0.0, maxval=scale, seed=seed, dtype=init_dtype) - elif method in ['uniform_unit_scaling']: - v_init = init_ops.uniform_unit_scaling_initializer( - factor=scale, seed=seed, dtype=init_dtype) - elif method in ['uniform_unit_scaling_fan_avg']: - v_init = tf.variance_scaling_initializer( - scale=scale, - mode='fan_avg', - distribution='uniform', - seed=seed, - dtype=init_dtype) - elif method in [ - 'truncated_gaussian', 'truncated_gaussian_sqrt_dim', - 'truncated_gaussian_sqrt_fanin', 'truncated_gaussian_sqrt_fanout' - ]: - v_init = init_ops.truncated_normal_initializer( - mean=0.0, stddev=scale, seed=seed, dtype=init_dtype) - elif method in ['constant']: - v_init = init_ops.constant_initializer(value=scale, dtype=init_dtype) - elif method in ['xavier', 'geo_mean_xavier']: - def XavierUniform(shape, dtype, partition_info): - """Xavier initialization (x = sqrt(6. / (in + out)); scale*[-x, x]).""" - del partition_info # Unused. - if not shape: - raise ValueError( - '\'shape\' must not be \'None\' or 0 for XavierUniform') - fan_in, fan_out = GetFanInFanOut(shape) - if method == 'xavier': - limit = math.sqrt(6. / (fan_in + fan_out)) - elif method == 'geo_mean_xavier': - limit = math.sqrt(3. / math.sqrt(fan_in * fan_out)) - return scale * tf.random.uniform(shape, -limit, limit, dtype, seed) - - v_init = XavierUniform - elif method in [ - 'kaiming_uniform_fanin_relu', 'kaiming_uniform_fanin_leakyrelu' - ]: - fan_in = np.prod(shape[:-1]) - if method == 'kaiming_uniform_fanin_leakyrelu': - # Assume the 'a' parameter is the 'scale' argument. - gain = np.sqrt(2. / (1 + scale**2)) - else: - gain = np.sqrt(2.) - std_dev = gain / np.sqrt(fan_in) - bound = np.sqrt(3.0) * std_dev - v_init = init_ops.random_uniform_initializer( - minval=-bound, maxval=bound, seed=seed, dtype=init_dtype) - else: - assert False, 'init_type `%s` not supported.' % method - - return v_init - - -def _GenerateStatelessRngSeed(name, seed): - """Generates a 2-tuple seed for a stateless variable initializer. - - We want to ensure that different variables end up with different random values - even when they are passed the same seed and shape. To this aim, this function - generates a pseudo-unique seed by hashing the variable name and mapping it - into a scalar seed. More specifically, the returned value is a 2-tuple of - tf.int32 scalar, where the first element is the user-provided seed and the - second element is obtained by hashing the variable name. - - Args: - name: The variable name for which to generate a stateless-like seed. - seed: The user-specified scalar seed. - - Returns: - A 2-tuple seed of tf.int32 values (for TPU compatibility). - """ - seed0 = seed or 0 - seed1 = GenerateSeedFromName(name) - return tf.constant([seed0, seed1], dtype=tf.int32) - - -def _DeterministicRandomNormalInitializer(seed, mean, stddev): - """Creates a random normal initializer.""" - - def DeterministicNormal(shape, dtype, partition_info): - del partition_info # Unused. - return stateless_random_ops.stateless_random_normal( - shape=shape, seed=seed, mean=mean, stddev=stddev, dtype=dtype) - - return DeterministicNormal - - -def _DeterministicRandomUniformInitializer(seed, minval, maxval): - """Creates a random uniform initializer.""" - - def DeterministicUniform(shape, dtype, partition_info): - del partition_info # Unused. - return stateless_random_ops.stateless_random_uniform( - shape=shape, seed=seed, minval=minval, maxval=maxval, dtype=dtype) - - return DeterministicUniform - - -def _DeterministicRandomTruncatedNormalInitializer(seed, mean, stddev): - """Creates a random truncated normal initializer.""" - - def DeterministicTruncatedNormal(shape, dtype, partition_info): - del partition_info # Unused. - return stateless_random_ops.stateless_truncated_normal( - shape=shape, seed=seed, mean=mean, stddev=stddev, dtype=dtype) - - return DeterministicTruncatedNormal - - -def _DeterministicRandomUniformUnitScalingInitializer(seed, factor): - """Creates a random uniform unit scaling initializer.""" - - def DeterministicUniformUnitScaling(shape, dtype, partition_info): - # The following logic is originally from (UniformUnitScaling.__call__()) - # in TensorFlow: python/ops/init_ops.py - scale_shape = shape - if partition_info is not None: - scale_shape = partition_info.full_shape - - input_size = 1.0 - # Estimating input size is not possible to do perfectly, but we try. - # The estimate, obtained by multiplying all dimensions but the last one, - # is the right thing for matrix multiply and convolutions (see above). - for dim in scale_shape[:-1]: - input_size *= float(dim) - # Avoid errors when initializing zero-size tensors. - input_size = max(input_size, 1.0) - maxval = math.sqrt(3 / input_size) * factor - return stateless_random_ops.stateless_random_uniform( - shape=shape, seed=seed, minval=-maxval, maxval=maxval, dtype=dtype) - - return DeterministicUniformUnitScaling - - -def _DeterministicRandomVarianceScalingInitializer(scale, mode, distribution, - seed): - """Creates a variance scaling initializer.""" - - if scale <= 0.: - raise ValueError('`scale` must be positive float.') - if mode not in {'fan_in', 'fan_out', 'fan_avg'}: - raise ValueError('Invalid `mode` argument:', mode) - distribution = distribution.lower() - if distribution not in { - 'normal', 'uniform', 'truncated_normal', 'untruncated_normal' - }: - raise ValueError('Invalid `distribution` argument:', distribution) - - def DeterministicVarianceScaling(shape, dtype, partition_info): - # This is originally from TensorFlow: python/ops/init_ops.py - scale_shape = shape - if partition_info is not None: - scale_shape = partition_info.full_shape - # Handle special case of empty list as shape, since fan_in and fan_out - # are numerically added below. Without this, GetFanInFanOut() would - # return None, None instead. - if isinstance(scale_shape, (list, tuple)) and not scale_shape: - fan_in, fan_out = 1, 1 - else: - fan_in, fan_out = GetFanInFanOut(scale_shape) - if mode == 'fan_in': - scale_inner = scale / max(1., fan_in) - elif mode == 'fan_out': - scale_inner = scale / max(1., fan_out) - else: - scale_inner = scale / max(1., (fan_in + fan_out) / 2.) - if distribution == 'normal' or distribution == 'truncated_normal': - # constant taken from scipy.stats.truncnorm.std( - # a=-2, b=2, loc=0., scale=1.) - stddev = math.sqrt(scale_inner) / .87962566103423978 - return stateless_random_ops.stateless_truncated_normal( - shape=shape, seed=seed, mean=0.0, stddev=stddev, dtype=dtype) - elif distribution == 'untruncated_normal': - stddev = math.sqrt(scale_inner) - return stateless_random_ops.stateless_random_normal( - shape=shape, seed=seed, mean=0.0, stddev=stddev, dtype=dtype) - else: - limit = math.sqrt(3.0 * scale_inner) - return stateless_random_ops.stateless_random_uniform( - shape=shape, seed=seed, minval=-limit, maxval=limit, dtype=dtype) - - return DeterministicVarianceScaling - - -def _DeterministicRandomXavierUniformInitializer(method, scale, seed): - """Creates a variance scaling initializer.""" - - def XavierUniform(shape, dtype, partition_info): - """Xavier initialization (x = sqrt(6. / (in + out)); scale*[-x, x]).""" - del partition_info # Unused. - if not shape: - raise ValueError('\'shape\' must not be \'None\' or 0 for XavierUniform') - fan_in, fan_out = GetFanInFanOut(shape) - if method == 'xavier': - limit = math.sqrt(6. / (fan_in + fan_out)) - elif method == 'geo_mean_xavier': - limit = math.sqrt(3. / math.sqrt(fan_in * fan_out)) - return scale * stateless_random_ops.stateless_random_uniform( - shape, seed, -limit, limit, dtype) - - return XavierUniform - - -def _CreateVarInitStateless(name, method, shape, dim0, seed, scale, init_dtype): - """Creates variable initialization function for a stateless RNG.""" - if (method in [ - 'gaussian_sqrt_dim', 'uniform_sqrt_dim', 'truncated_gaussian_sqrt_dim' - ]): - if len(shape) > 2: - # This is probably not the right method to use when len(shape) > 2, - # e.g. dim0 will be 3 with a 3x3 conv2d kernel. - tf.logging.warning( - 'Initializing %s of shape %s with method %s: dim0=%s. ' - 'Make sure that it is intended.', name, shape, method, dim0) - scale *= 1.0 / math.sqrt(dim0) - - if method in ['gaussian_sqrt_fanin', 'truncated_gaussian_sqrt_fanin']: - fan_in, _ = GetFanInFanOut(shape) - if fan_in is not None: - scale *= 1.0 / math.sqrt(fan_in) - if method in ['gaussian_sqrt_fanout', 'truncated_gaussian_sqrt_fanout']: - _, fan_out = GetFanInFanOut(shape) - if fan_out is not None: - scale *= 1.0 / math.sqrt(fan_out) - if method == 'xavier_gaussian': - fan_in, fan_out = GetFanInFanOut(shape) - if fan_in is not None and fan_out is not None: - scale *= math.sqrt(2.0 / (fan_in + fan_out)) - - if method in [ - 'gaussian', 'gaussian_sqrt_dim', 'gaussian_sqrt_fanin', - 'gaussian_sqrt_fanout', 'xavier_gaussian' - ]: - v_init = _DeterministicRandomNormalInitializer( - seed=seed, mean=0., stddev=scale) - elif method in ['uniform', 'uniform_sqrt_dim']: - v_init = _DeterministicRandomUniformInitializer( - seed=seed, minval=-scale, maxval=scale) - elif method in ['uniform_positive']: - v_init = _DeterministicRandomUniformInitializer( - seed=seed, minval=0., maxval=scale) - elif method in ['uniform_unit_scaling']: - v_init = _DeterministicRandomUniformUnitScalingInitializer( - seed=seed, factor=scale) - elif method in ['uniform_unit_scaling_fan_avg']: - v_init = _DeterministicRandomVarianceScalingInitializer( - scale=scale, mode='fan_avg', distribution='uniform', seed=seed) - elif method in [ - 'truncated_gaussian', 'truncated_gaussian_sqrt_dim', - 'truncated_gaussian_sqrt_fanin', 'truncated_gaussian_sqrt_fanout' - ]: - v_init = _DeterministicRandomTruncatedNormalInitializer( - seed=seed, mean=0., stddev=scale) - elif method in ['constant']: - v_init = init_ops.constant_initializer(value=scale, dtype=init_dtype) - elif method in ['xavier', 'geo_mean_xavier']: - v_init = _DeterministicRandomXavierUniformInitializer(method, scale, seed) - elif method in [ - 'kaiming_uniform_fanin_relu', 'kaiming_uniform_fanin_leakyrelu' - ]: - fan_in = np.prod(shape[:-1]) - if method == 'kaiming_uniform_fanin_leakyrelu': - # Assume the 'a' parameter is the 'scale' argument. - gain = np.sqrt(2. / (1 + scale**2)) - else: - gain = np.sqrt(2.) - std_dev = gain / np.sqrt(fan_in) - bound = np.sqrt(3.0) * std_dev - v_init = _DeterministicRandomUniformInitializer( - seed=seed, minval=-bound, maxval=bound) - else: - assert False, 'init_type %s not supported.' % method - - return v_init - - -_global_variable_scope = None - - -def GetGlobalVariableScope(): - """Gets the global variable scope (as if no variable_scope has been set). - - Returns: - The VariableScope corresponding to as if no tf.variable_scope is in effect. - """ - if not _global_variable_scope: - # Each thread gets its own default global variable scope, and we take - # advantage of that in order to get a top-level scope. This avoids the - # need to call tf.get_variable_scope() at the module level, which allows - # this module to be imported without modifying global state (i.e. creating - # the default graph). It is important to not mutate the global state at - # module load time, because it let's us flip flags after import that affect - # core TensorFlow behavior. - def Initialize(): - global _global_variable_scope - _global_variable_scope = tf.get_variable_scope() - - t = threading.Thread(target=Initialize) - t.start() - t.join() - return _global_variable_scope - - -_GLOBAL_STEP_STACK = ThreadLocalStack() - - -@contextlib.contextmanager -def GlobalStepContext(global_step_tensor): - _GLOBAL_STEP_STACK.stack.append(global_step_tensor) - try: - yield - finally: - _GLOBAL_STEP_STACK.stack.pop() - - -def GetGlobalStep(): - """Return the global_step.""" - if _GLOBAL_STEP_STACK.stack: - return _GLOBAL_STEP_STACK.stack[-1] - return tf.train.get_global_step() - - -def GetOrCreateGlobalStepVar(): - """Return the global_step variable, creating it if it does not exist. - - Prefer GetGlobalStep if a tensor rather than a tf.Variable is sufficient. - - Returns: - The global_step variable, or a new created one if it does not exist. - """ - with tf.variable_scope( - GetGlobalVariableScope(), use_resource=use_resource_variables()): - return tf.train.get_or_create_global_step() - - -def LogMultiLines(label, lines): - if not isinstance(lines, (list, tuple)): - lines = lines.split('\n') - for line in lines: - tf.logging.info('%s: %s', label, line) - - -def _LogPlacement(label, theta, copy): - """Logs theta and its copy's device placement.""" - - def GetDevices(m): - """Flatten a `.NestedMap` m and extracts each value's device.""" - return [x.device for x in m.Flatten()] - - tf.logging.info('=== %s ===', label) - LogMultiLines( - label, - theta.Pack([('%s -> %s' % (x[0], x[1])) - for x in zip(GetDevices(theta), GetDevices(copy)) - ]).DebugString()) - tf.logging.info('==========') - - -def CreateLocalTheta(theta, device_list=None, label=None): - """Creates local copy of theta and shards across devices device list. - - Leaves variables intact. - - Args: - theta: a `.NestedMap` of variables. - device_list: list of devices to shard across. If None, defaults to a list - ['']. - label: Logging label. - - Returns: - A `.NestedMap` of identity() wrapped theta - """ - - class AddIdentity: - """Helper class.""" - - def __init__(self, device_list): - self._list = device_list if device_list else [''] - self._index = 0 - - def __call__(self, x): - if isinstance(x, tf.Variable): - return x - with tf.device(self._list[self._index % len(self._list)]): - self._index += 1 - return tf.identity(x) - - copy = theta.Transform(AddIdentity(device_list)) - _LogPlacement(label, theta, copy) - return copy - - -def _GetVarsToLoad(all_vars, variable_loading_rules, var_ignore_rules): - """Determines variables to load and their names in checkpoint.""" - # This list contains mappings from var names as they appear in the checkpoint - # to the vars in our model they correspond to. - vars_to_load = [] - for model_var in all_vars: - for regexp, name_format in variable_loading_rules: - match = re.match(regexp, model_var.name) - # Skip if var doesn't match the loading rules, or if it should be ignored. - if not match or any( - re.match(r, model_var.name) for r in var_ignore_rules): - continue - checkpoint_var_name = name_format % match.groups() - if checkpoint_var_name.endswith(':0'): - checkpoint_var_name = checkpoint_var_name[:-2] - tf.logging.info('Loading %s from %s', model_var, checkpoint_var_name) - vars_to_load.append((checkpoint_var_name, model_var)) - break - return vars_to_load - - -def OverrideVarsFromCheckpoint(sess, all_vars, checkpoint_path, - variable_loading_rules, var_ignore_rules): - """Overrides variables from a provided checkpoint.""" - vars_to_load = _GetVarsToLoad(all_vars, variable_loading_rules, - var_ignore_rules) - if not vars_to_load: - raise ValueError(('Variable loading rules did not match any vars. ' - 'All known: %r') % [v.name for v in all_vars]) - load_var_names = sorted([v.name for _, v in vars_to_load]) - tf.logging.info('Overriding vars from checkpoint: %r', load_var_names) - - while vars_to_load: - # When restoring, it's possible the same value in the checkpoint - # can be restored to multiple variables (e.g. during - # distillation). However, tf.train.Saver, since it's used for - # both saving and restoring, requires the name in the checkpoint - # to be unique for each variable. So, we call it multiple times - # with a unique set of names each time. - unique_vars_to_load = {} - remaining_vars_to_load = [] - for k, v in vars_to_load: - if k not in unique_vars_to_load: - unique_vars_to_load[k] = v - else: - remaining_vars_to_load.append((k, v)) - tf.train.Saver(var_list=unique_vars_to_load).restore(sess, checkpoint_path) - vars_to_load = remaining_vars_to_load - - -def OverrideVarsFromCheckpoints(session, all_vars, ckpts_loading_rules): - """Overrides model variables from checkpoints. - - Args: - session: Tensorflow session. - all_vars: List of all the parameters in the model. - ckpts_loading_rules: A dictionary of checkpoint path: loading rules. - Checkpoint path must be a path to a pretrained model, and loading rules is - expected to be a tuple of two lists. The first consisting of tuples of - strings defining (regex to match parameter names in the model to override, - format string to determine the corresponding var in the checkpoint), and - the second list consisting of a list of regexes to match parameter names - in the model which should not be overridden, even if they match those in - the loading rules. - - Returns: - A list of overwritten variables. - - Raises: - ValueError: if colliding vars exist or loading rules is not a list. - """ - if len(ckpts_loading_rules) > 1: - tf.logging.info('Overriding vars from multiple checkpoints.') - - var_refs_overridden = set() - var_names_overridden = set() - for ckpt_path, loading_rules in ckpts_loading_rules.items(): - tf.logging.info('Overriding vars from checkpoint: %s', ckpt_path) - - if not isinstance(loading_rules, tuple): - raise ValueError('Loading rules for %s must be a tuple of two lists!' % - ckpt_path) - if len(loading_rules) != 2 or not all( - isinstance(l, list) for l in loading_rules): - raise ValueError('Loading rules for %s must be a tuple of two lists!' % - ckpt_path) - - # Filter the model variables to be overridden. - var_refs_to_override = [ - var[1].experimental_ref() - for var in _GetVarsToLoad(all_vars, loading_rules[0], loading_rules[1]) - ] - var_names_to_override = [ - var[1].name - for var in _GetVarsToLoad(all_vars, loading_rules[0], loading_rules[1]) - ] - - overlap_refs = set.intersection(var_refs_overridden, var_refs_to_override) - if overlap_refs: - raise ValueError('Colliding variables to override: %s' % overlap_refs) - - OverrideVarsFromCheckpoint(session, all_vars, ckpt_path, loading_rules[0], - loading_rules[1]) - var_refs_overridden.update(var_refs_to_override) - var_names_overridden.update(var_names_to_override) - tf.logging.info('Model variables overridden: %s', var_refs_overridden) - return var_names_overridden - - -def ComputeGradientsSimple(loss, all_vars, grad_aggregation_method, - colocate_gradients_with_ops, gate_gradients): - return tf.gradients( - loss, - all_vars, - aggregation_method=grad_aggregation_method, - colocate_gradients_with_ops=colocate_gradients_with_ops, - gate_gradients=gate_gradients) - - -def ComputeTpuEmbeddingGradients(loss, activation_dict, tpu_embedding): - """Returns a TpuEmbedding SendGradient op. - - Args: - loss: The loss to backprop from. - activation_dict: String feature -> embedding activations dict. - tpu_embedding: TPUEmbedding instance. - """ - - # Scale the loss to account for the full batch size. - shards = tpu_function.get_tpu_context().number_of_shards - loss *= tf.constant(1.0 / shards, dtype=loss.dtype) - - grads = tf.gradients(loss, list(activation_dict.values())) - feature_to_gradient_dict = py_collections.OrderedDict( - zip(list(activation_dict.keys()), grads)) - send_gradient_op = tpu_embedding.generate_send_gradients_op( - feature_to_gradient_dict) - return send_gradient_op - - -def _ComputeGradientsTpu(loss, - all_vars, - grad_aggregation_method, - colocate_gradients_with_ops, - gate_gradients, - skip_zero_gradients=None, - use_bf16_gradients_ar=False, - defer_crs_to_apply_grad=False): - """Computes gradients for local loss across whole TPU cluster. - - This implementation specializes for the case where weight params maybe used - for different number of times in the forward computation, so that gradients - should be normalized by the actual number of times they are being computed. - - TODO(yonghui): Maybe merge this implementation with the _ComputeGradientsTpu - one. - - Args: - loss: The loss to backprop from. - all_vars: Vars with respect to which gradients are to be computed. - grad_aggregation_method: aggregation method to use when calling - tf.gradients. - colocate_gradients_with_ops: boolean, whether or not to colocate gradient op - with the original op. - gate_gradients: boolean, flag to be passed to tf.gradients. - skip_zero_gradients: whether to skip zero gradients during aggregation. - use_bf16_gradients_ar: Whether to use bfloat16 dtype for gradients - all-reduce. - defer_crs_to_apply_grad: Whether to defer gradient cross replica sum to - apply_gradient. This helps reducing the number of gradient all-reduces - when doing gradient accumulation, which does gradient cross replica sum - only every k steps in a tf.cond. Currently this works only when - skip_zero_gradients is None. - - Returns: - Gradients to be passed back. - - Raises: - ValueError: upon invalid arguments. - """ - if not skip_zero_gradients: - # Scale the loss to account for the full batch size. - shards = tpu_function.get_tpu_context().number_of_shards - assert shards - loss *= tf.constant(1.0 / shards, dtype=loss.dtype) - - # Computes the gradients. - # Sum the grads so that we can compute statistics across the whole batch. - all_grads = ComputeGradientsSimple(loss, all_vars, grad_aggregation_method, - colocate_gradients_with_ops, - gate_gradients) - - # NOTE: We can't use tpu_optimizer.CrossShardOptimizer since - # we need to scale the grads *after* the cross_replica_sum to - # match GPU version! - - # TODO(cwhipkey): should we do something different here? - we could do - # some operations on the gradients before the aggregation (see comments in - # tensorflow/contrib/tpu/python/tpu/tpu_optimizer.py - see compute_gradients - - # for some more details). - - aggregated_grads = [] - for g in all_grads: - if g is None: - aggregated_grads.append(None) - continue - if use_bf16_gradients_ar: - g = tf.cast(g, tf.bfloat16) - with tf.ops.colocate_with(g): - if skip_zero_gradients is None: - # loss is already scaled by 1/shards. - if defer_crs_to_apply_grad: - normalized_g = tf.convert_to_tensor(g) - else: - normalized_g = tf.tpu.cross_replica_sum(g) - else: - # Compute the cross-replica mean of 'g', skipping zero gradients. - - # Q(yonghui): Is there a better way to detect a non-zero gradient? - # Note(yonghui): gradient of a weight can be zero if that - # weight is not used in the forward computation, e.g. as in - # switchable layers in neural architecture search, pruned by channel - # mask, or sparsified. - if skip_zero_gradients == 'weight': - # Same shape as 'g'. - g_is_non_zero = tf.cast(tf.math.abs(g) > 1e-8, g.dtype) - elif skip_zero_gradients == 'variable': - # A variable-wide 0/1 scalar. - g_is_non_zero = tf.cast( - tf.reduce_sum(tf.math.abs(g)) > 1e-24, g.dtype) - else: - raise ValueError('Unknown skip_zero_gradients: %s' % - skip_zero_gradients) - num_updates = tf.maximum(tf.tpu.cross_replica_sum(g_is_non_zero), 1.0) - normalized_g = tf.tpu.cross_replica_sum(g) / num_updates - aggregated_grads.append(normalized_g) - return aggregated_grads - - -class VarGrad: - """A class that holds a variable and a gradient.""" - - _VAR_GRAD = py_collections.namedtuple('VarGradNamedTuple', ['var', 'grad']) - - def __init__(self, *args, **kwargs): - self._var_grad = self._VAR_GRAD(*args, **kwargs) - - def __getitem__(self, key): - return self._var_grad[key] - - def __getattr__(self, key): - return getattr(self._var_grad, key) - - def __iter__(self): - return iter(self._var_grad) - - def __repr__(self): - return 'VarGrad(%r, %r)' % (self._var_grad.var, self._var_grad.grad) - - -def SkipNoneGradients(var_grads): - """Removes pairs whose grad is None.""" - for key, (_, g) in var_grads.FlattenItems(): - if g is None: - tf.logging.info('ComputeGradients drops %s', key) - return var_grads.Filter(lambda var_grad: var_grad.grad is not None) - - -def ComputeGradients( - loss, - vmap, - grad_aggregation_method=tf.AggregationMethod.EXPERIMENTAL_TREE, - colocate_gradients_with_ops=True, - gate_gradients=False, - compute_gradients_fn=None, - skip_zero_gradients=None, - use_bf16_gradients_ar=False, - skip_none_gradients=True, - defer_crs_to_apply_grad=False): - """Computes gradients of variables in vmap w.r.t loss. - - Args: - loss: A scalar Tensor. - vmap: A `.NestedMap` of variables. - grad_aggregation_method: Specifies the method used to combine gradient - terms. Accepted values are constants defined in the class - AggregationMethod. - colocate_gradients_with_ops: If True, try colocating gradients with the - corresponding op. - gate_gradients: If True, add a tuple around the gradients returned for an - operations. This avoids some race conditions. - compute_gradients_fn: Function to use to compute gradients. If None, use - default. compute_gradients_fn should have the same signature as this - function, but without the last argument. - skip_zero_gradients: Whether to skip aggregating zero gradients. This helps - in case where some weights may not be used in forward computation, e.g., - sparsely activated networks or switchable layers in neural architectural - search. Only applicable on TPU. - Possible values are: - - * None: do not skip zero gradients; - * `variable`: skip if the entire variable's gradients are almost zero; - reduce_sum(abs(grads)) < 1e-8. - * `weight`: skip if the individual weight's gradients are almost zero: - abs(grad) < 1e-8. - use_bf16_gradients_ar: Whether to use bfloat16 dtype for gradients - all-reduce. This applies to TPU only. - skip_none_gradients: Whether to skip gradients that are None. - defer_crs_to_apply_grad: Whether to defer gradient cross replica sum to - apply_gradient. This applies to TPU only. - - Returns: - var_grad - a `.NestedMap` of VarGrad. You can view - var_grad as an ordered list of (key, (var, grad)) tuples. Every - key of var_grad exists in vmap. Every variable in vmap that - contributes to loss must exist in var_grad. Every var of var_grad - must exist in vmap. grad is the corresponding gradient computed - for var. grad is guaranteed to be not None. - """ - loss = HasRank(loss, 0) - assert isinstance(vmap, NestedMap) - assert skip_zero_gradients in (None, 'variable', 'weight') - - # Uniqify and remove None. - filtered_vmap = vmap.Filter(_Unique()) - assert filtered_vmap is not None - - # Filter out variables not contributing to 'loss'. - trainable_variables = set(tf.trainable_variables()) - dependent_ops_and_tensors = set(FindNeeded([loss])) - - def Needed(v): - if isinstance(v, tf.Variable): - if v not in trainable_variables: - # Skip non-trainable variables. Otherwise, - # tf.Optimizer.apply_gradients throws up an exception instead - # of skipping the update. - return False - return True - - filtered_vmap = filtered_vmap.Filter(Needed) - assert filtered_vmap is not None - filtered_vlist = filtered_vmap.Flatten() - - # Use caller-supplied gradient function if supplied. - if compute_gradients_fn is not None: - take_grad = compute_gradients_fn - else: - # tpu vs non-tpu is slightly different. - if use_tpu(): - take_grad = functools.partial( - _ComputeGradientsTpu, - skip_zero_gradients=skip_zero_gradients, - use_bf16_gradients_ar=use_bf16_gradients_ar, - defer_crs_to_apply_grad=defer_crs_to_apply_grad) - else: - take_grad = ComputeGradientsSimple - - grads = take_grad(loss, filtered_vlist, grad_aggregation_method, - colocate_gradients_with_ops, gate_gradients) - - # Formulate pairs of (var, grad) and pack them into the same - # structure as filtered_vmap. - var_grads = filtered_vmap.Pack( - [VarGrad(v, g) for v, g in zip(filtered_vlist, grads)]) - - # TPU training is not compatible with the variable name check below when - # control flow v2 is enabled. The main reason is the body function will be - # encapsulated as a TF function while variables will be lifted out, and as a - # result dependent_ops_and_tensors will not contain any variables. See - # b/150689507 for more info. - if not tf.compat.v1.control_flow_v2_enabled(): - # Check that gradients for variables that are not needed by current task is - # empty. - def CheckGrad(vg): - if vg.var.name not in dependent_ops_and_tensors and vg.grad is not None: - err_msg = ('Variable %s is not a dependent of %s, expect ' - 'gradient be None, but got %s. This should not happen, ' - 'please contact the owner of b/150689507 for further ' - 'investigation.' % (str(vg.var), str(loss), str(vg.grad))) - assert False, err_msg - return True - - var_grads = var_grads.Filter(CheckGrad) - - if skip_none_gradients: - var_grads = SkipNoneGradients(var_grads) - - return var_grads - - -def MaskGradients(var_grad, grad_mask): - """Computes gradients of non-masked variables in vmap w.r.t loss. - - Args: - var_grad: A `.NestedMap` of (variable, gradient) - grad_mask: A dict of (variable name, mask). - - Returns: - var_grad - a `.NestedMap` of (variable, mask * gradient). - """ - - def ApplyMask(entry): - var, grad = entry - mask = grad_mask[var.name] - if isinstance(grad, tf.IndexedSlices): - return VarGrad(var, tf.IndexedSlices(grad.values * mask, grad.indices)) - else: - return VarGrad(var, grad * mask) - - return var_grad.Transform(ApplyMask) - - -def ApplyGradMultiplier(vs_gs, grad_scale=None): - """Scale gradients by grad_scale on same device as corresponding variables. - - Args: - vs_gs: A `.NestedMap` of VarGrad. - grad_scale: If None, each vs_gs entry has the scale. Otherwise, grad_scale - applies to every entry. - - Returns: - A `.NestedMap` of (variable, gradient * grad_scale). In particular, if - grad_scale is 0, the result gradient is always 0, even if the input - gradient is inf or nan. - """ - - def ScaleOrZero(var, grad, scale): - grad = CheckNumerics(grad, 'Gradient for %s is not finite.' % var.name) - return tf.where( - tf.equal(scale, 0.), tf.zeros_like(grad), - tf.cast(scale, grad.dtype) * grad) - - def Scale(item): - """Scales the gradient.""" - var, grad = item - assert grad is not None, ('No grad found for ', var.name) - if grad_scale is None: - scale = item.scale - else: - scale = grad_scale - with tf.device(var.device): - if isinstance(grad, tf.IndexedSlices): - grad = tf.IndexedSlices( - ScaleOrZero(var, grad.values, scale), grad.indices, - grad.dense_shape) - else: - grad = ScaleOrZero(var, grad, scale) - return VarGrad(var, grad) - - return vs_gs.Transform(Scale) - - -def HasNanOrInfGradient(var_grads): - """Returns a bool tensor to indicate if `var_grads` contains NaNs or Infs. - - Args: - var_grads: A `.NestedMap` with (var, grad) tuple as the map value. - - Returns: - A bool scalar tensor to indicate if the `var_grads` contains NaNs or Infs. - """ - - def HasNanOrInf(x): - if isinstance(x, tf.IndexedSlices): - x = x.values - with tf.device(x.device): - if x.dtype.is_complex: - return tf.reduce_any( - [HasNanOrInf(tf.math.real(x)), - HasNanOrInf(tf.math.imag(x))]) - return tf.reduce_any( - tf.math.logical_or(tf.math.is_nan(x), tf.math.is_inf(x))) - - return tf.reduce_any([HasNanOrInf(g) for (_, g) in var_grads.Flatten()]) - - -def ApplyGradNormClipping(vs_gs, norm=1.0): - """Clip gradients to norm on same device as corresponding variables. - - Args: - vs_gs: A `.NestedMap` of VarGrad. - norm: Each tensor's gradient will be scaled down to have a maximum L2-norm - value of `norm`. - - Returns: - A `.NestedMap` of VarGrad(variable, scaled_gradient). In particular, if - grad_scale is 0, the result gradient is always 0, even if the input - gradient is inf or nan. - """ - - def ClipByNorm(var, grad, norm): - grad = CheckNumerics(grad, 'Gradient for %s is not finite.' % var.name) - return tf.clip_by_norm(grad, norm) - - def Clip(item): - """Scales the gradient.""" - var, grad = item - assert grad is not None, ('No grad found for ', var.name) - with tf.device(var.device): - if isinstance(grad, tf.IndexedSlices): - grad = tf.IndexedSlices( - ClipByNorm(var, grad.values, norm), grad.indices, grad.dense_shape) - else: - grad = ClipByNorm(var, grad, norm) - return VarGrad(var, grad) - - return vs_gs.Transform(Clip) - - -SKIP_LP_REGULARIZATION = '__lingvo_skip_lp_regularization' - - -def AdjustGradientsWithLpLoss(var_grads, lp_regularizer_weight, p=2.0): - """Adjusts the map of (var, grad) with Lp regularization, where p=1.0 or 2.0. - - Args: - var_grads: a `.NestedMap` or list of (variable, gradient). - lp_regularizer_weight: Lp regularization weight. - p: For now we support 1.0 or 2.0. - - Returns: - A tuple (lp_loss, var_grads). - - - lp_loss: A scalar. The lp loss. - - var_grads: a `.NestedMap` or list of (variable, gradient) regulated by Lp. - """ - # TODO(yuancao): For now we support p=1 or 2, but this can be extended to - # lp-norm in general. - - assert p in [2.0, 1.0], 'For now we only support L1/L2 regularization.' - - def GetVar(item): - var, grad = item - if isinstance(grad, tf.IndexedSlices): - with tf.device(var.device): - ids = HasRank(grad.indices, 1) - uniq_ids = tf.unique(ids).y - return tf.gather(var, uniq_ids) - else: - return var - - def ShouldAdjust(v): - return v not in tf.get_collection(SKIP_LP_REGULARIZATION) - - filtered_var_grads = [ - var_grad for var_grad in Flatten(var_grads) if ShouldAdjust(var_grad.var) - ] - filtered_vars = Transform(GetVar, filtered_var_grads) - for v in filtered_vars: - tf.logging.info('AdjustGradientsWithLpLoss: %s', v.name) - - if p == 2.0: - lp_loss = 0.5 * lp_regularizer_weight * SumSquared(filtered_vars) - elif p == 1.0: - lp_loss = lp_regularizer_weight * SumAbs(filtered_vars) - - def LpGrad(var_grad): - """Adjusts item's grad w/ Lp loss term.""" - var, grad = var_grad - if isinstance(grad, tf.IndexedSlices): - # Question(rpang): do we apply Lp loss here even if 'var' is in - # SKIP_LP_REGULARIZATION? - # - # Note: IndexedSlces appears for embedding lookups. - # Embedding lookup ids can have duplicate. For duplicated ids, we - # only want to consider once for each ids. - with tf.device(var.device): - emb = HasRank(var, 2) - vocab_size = tf.shape(emb)[0] - ids = HasRank(grad.indices, 1) - values = tf.gather(emb, ids) # [#ids, dims] - with tf.device(grad.device): - # Counts is a vector of size vocab_size. counts[i] is i-th words - # occurances in 'ids'. - counts = tf.math.unsorted_segment_sum( - tf.ones_like(ids, dtype=values.dtype), ids, vocab_size) - - # Gradients for duplicated ids will be summed when they get - # applied, and hence we account for that by first dividing - # gradient resulting from lp loss by how many times the id is - # duplicated. - # - # For each id in 'ids', we know counts[id] is non-zero, - # hence, it's always safe to take reciprocal. - weights = tf.math.reciprocal(tf.gather(counts, ids)) - weights = tf.expand_dims(weights, -1) # [#ids, 1] - if p == 2.0: - grad_v = values - elif p == 1.0: - grad_v = tf.sign(values) - delta = lp_regularizer_weight * weights * grad_v - grad = tf.IndexedSlices(grad.values + delta, ids) - elif var not in tf.get_collection(SKIP_LP_REGULARIZATION): - with tf.device(var.device): - if p == 2.0: - grad_v = var - elif p == 1.0: - grad_v = tf.sign(var) - delta = lp_regularizer_weight * grad_v - with tf.device(grad.device): - grad += delta - return VarGrad(var, grad) - - return lp_loss, Transform(LpGrad, var_grads) - - -def SplitRecursively(x, num_splits, axis=-1): - """Splits Tensors in 'x' recursively. - - Args: - x: a Tensor, or a list or NestMap containing Tensors to split. - num_splits: number of splits per Tensor. - axis: the split axis. - - Returns: - A list of split values of length 'num_splits'. - - - If 'x' is a Tensor, a list of split Tensors. - - If 'x' is a list, a list of lists, where each sublist has the same length - as 'x' and the k'th element in each sublist corresponds to a split of the - k'th element from 'x'. - - If 'x' is a `.NestedMap`, a list of `.NestedMap`, where each field - corresponds to a split from the same field of 'x'. - """ - if isinstance(x, tf.Tensor): - return tf.split(x, num_splits, axis=axis) - elif isinstance(x, list): - splits = [SplitRecursively(element, num_splits, axis) for element in x] - splits = list(zip(*splits)) - return [list(t) for t in splits] - elif isinstance(x, NestedMap): - results = [NestedMap() for _ in range(num_splits)] - for key, val in x.items(): - val_splits = SplitRecursively(val, num_splits, axis) - for i in range(num_splits): - results[i][key] = val_splits[i] - return results - else: - raise TypeError('Unexpected type for SplitRecursively: %s' % type(x)) - - -def ConcatRecursively(splits, axis=-1): - """Concatenates tensors from 'splits'. - - This is the inverse function of SplitRecursively. - - Args: - splits: a list of splits to concatenate, where elements can be Tensors, - lists, or `.NestedMap`. The elements must share the same type and - structure. For example, list elements must have the same length; - `.NestedMap` must have the same set of fields. - axis: the concatenation axis. - - Returns: - Concatenated data. - - - If input 'splits' are Tensors, returns a concatenated Tensor. - - If input 'splits' are lists, returns a list of the same length where the - k'th element represents concatenated data of the k'th element from each - split. - - If input 'splits' are `.NestedMap`, returns a `.NestedMap` with each field - concatenated from corresponding fields of input splits. - - Raises: - TypeError: if 'splits' is not a list or elements of 'splits' do not have - known or matching types. - ValueError: if 'splits' is empty or elements of 'splits' do not have - matching structures. - """ - if not isinstance(splits, list): - raise TypeError('Non-list inputs for ConcatRecursively: %s' % splits) - if not splits: - raise ValueError('Empty inputs for ConcatRecursively: %s' % splits) - - tmpl = splits[0] - - if isinstance(tmpl, tf.Tensor): - return tf.concat(splits, axis=axis) - elif isinstance(tmpl, list): - if not all(isinstance(split, list) for split in splits): - raise TypeError('Type mismatch for ConcatRecursively: %s' % splits) - if not all(len(split) == len(tmpl) for split in splits): - raise ValueError('Length mismatch for ConcatRecursively: %s' % splits) - return [ - ConcatRecursively([split[i] - for split in splits], axis) - for i in range(len(tmpl)) - ] - elif isinstance(tmpl, NestedMap): - if not all(isinstance(split, NestedMap) for split in splits): - raise TypeError('Type mismatch for ConcatRecursively: %s' % splits) - results = NestedMap() - for key in tmpl: - results[key] = ConcatRecursively([split[key] for split in splits], axis) - return results - else: - raise TypeError('Unexpected type for ConcatRecursively: %s' % type(splits)) - - -def WeightedAvg(values, weights, sum_reduction_fn=tf.reduce_sum, name=''): - """Computes weighted average of values from a tensor. - - Args: - values: a tensor of values - weights: a tensor of weights - sum_reduction_fn: called to reduce the values and weights to single value - name: name of metric. - - Returns: - A tuple (avg, total_weight). - - - avg: weighted average value - - total_weight: sum of all weights - """ - msg = 'shape of values and weights tensors must match for metric ' + name - values = with_dependencies( - [assert_equal(tf.shape(values), tf.shape(weights), message=msg)], values) - total_weight = sum_reduction_fn(weights) - # divide_no_nan only supports tf.{float,complex}*. - dtype = values.dtype if values.dtype is tf.float64 else tf.float32 - avg = tf.math.divide_no_nan( - sum_reduction_fn(tf.cast(values, dtype) * tf.cast(weights, dtype)), - tf.cast(total_weight, dtype)) - return tf.cast(avg, values.dtype), total_weight - - -def WeightedAvgOfMetrics(metrics): - """Computes the weighted average of metrics in the list. - - Args: - metrics: list of dictionaries of metrics - - Returns: - ret_dict - dictionary of weighted averages of each metrics. - """ - ret_dict = {} - lists_of_metrics = {} - for m in metrics: - for name, (value, weight) in m.items(): - if name not in lists_of_metrics: - lists_of_metrics[name] = [] - lists_of_metrics[name].append((value, weight)) - - for name, values_and_weights in sorted(lists_of_metrics.items()): - values = tf.stack([x[0] for x in values_and_weights]) - weights = tf.stack([x[1] for x in values_and_weights]) - ret_dict[name] = WeightedAvg(values, weights, tf.reduce_sum, name) - - return ret_dict - - -def ConcatPerExampleTensors(per_example): - """Concatenate per-example tensors from many hosts into one large block. - - Args: - per_example: list of dictionaries of per-example tensors. - - Returns: - ret_dict - string -> concatenated tensors. - """ - ret_dict = {} - lists_of_per_example = {} - for m in per_example: - for name, value in m.items(): - if name not in lists_of_per_example: - lists_of_per_example[name] = [] - lists_of_per_example[name].append(value) - - for name, values in sorted(lists_of_per_example.items()): - ret_dict[name] = tf.concat(values, 0) - - return ret_dict - - -def CombineMetrics(loss_metric_weight_pairs): - """Combines metrics from `loss_metric_weight_pairs` according to weights. - - Keys must either exist in all metrics, in which it will be processed as a - weighted sum, or exist in only one metrics, in which case it will be copied. - - Args: - loss_metric_weight_pairs: a list of (metrics, weight) pairs, where each - weight is a float and each metrics is a dict with str keys and - (metric_value, target_weight) values. - - Returns: - A dict with the same set of keys as input metrics and values of - (weighted_sum(metric_value), weighted_sum(target_weight)). - - Raises: - ValueError: if there exists a metric that exists in more than one element - of `loss_metric_weight_pairs` but not in all of them. - """ - all_keys = set( - [k for loss_metrics, _ in loss_metric_weight_pairs for k in loss_metrics]) # pylint: disable=g-complex-comprehension - result = {} - for k in all_keys: - count = 0 - for loss_metrics, weight in loss_metric_weight_pairs: - if k in loss_metrics: - count += 1 - if count > 1 and count != len(loss_metric_weight_pairs): - raise ValueError('Found metric %s which exists in more than one' - 'but not all loss metrics.' % k) - - total_val = 0 - total_target_weight = 0 - for loss_metrics, weight in loss_metric_weight_pairs: - if k in loss_metrics: - val, target_weight = loss_metrics[k] - if count == 1: - # Single metric, don't multiply by weight. - total_val = val * target_weight - total_target_weight = target_weight - else: - # Total weighted sum of all predictions. - total_val += weight * val * target_weight - total_target_weight += weight * target_weight - - result[k] = (total_val / total_target_weight, total_target_weight) - return result - - -def _AddVN(p, x, step=None): - assert p.vn.scale is not None - seed = p.vn.seed - if seed and step: - seed += step * 203984 - # So a seed is provided based on the step. So how does TPU generate - # random numbers then? Would be good to dig intot he xla implementation. - noises = tf.cast(p.vn.scale, x.dtype) * tf.random.normal( - tf.shape(x), stddev=1.0, seed=seed, dtype=x.dtype) - return x + noises - - -def AddGlobalVN(params, weights): - """Adds variational noise to weights if specified by params.""" - p = params - if p.vn.global_vn: - weights = _AddVN(p, weights) - return weights - - -def AddPerStepVN(params, weights, step=None): - """Adds per-setp variational noise to weights if specified by params.""" - p = params - if p.vn.per_step_vn: - weights = _AddVN(p, weights, step) - return weights - - -def VariationalNoiseParams(scale, - global_vn=False, - per_step_vn=False, - seed=None): - """Returns a hyperparams for variational noise.""" - p = hyperparams.Params() - p.Define( - 'scale', scale, - 'Std of the variational noise to apply . This can be a scalar,' - ' or a scalar tensor.') - p.Define('global_vn', global_vn, - 'Adds global variational noise every training setp iff True.') - p.Define('per_step_vn', per_step_vn, - 'Adds per-timesetp variational noise iff True.') - p.Define('seed', seed, 'Random seed used to generate noise.') - return p - - -def DefaultVN(): - return VariationalNoiseParams(None, False, False) - - -# To disable VN of a layer, we use 1.0 in the first input parameter -# of the following function because otherwise it is the same to DefaultVN() -# which will be updated by parent configuration in CopyBaseParams() -def DisableVN(): - return VariationalNoiseParams(1.0, False, False) - - -def GetStepSeed(): - """Gets step_seed.""" - step_seed_tensors = tf.get_default_graph().get_collection_ref('step_seed') - if not step_seed_tensors: - ResetStepSeed() - return GetStepSeed() - elif len(step_seed_tensors) == 1: - return step_seed_tensors[0] - else: - raise ValueError('Multiple tensors in step_seed collection.') - - -def ResetStepSeed(seed=0): - """Resets step_seed to specified value.""" - new_step_seed = tf.convert_to_tensor(seed, dtype=tf.int64) - step_seed_tensors = tf.get_default_graph().get_collection_ref('step_seed') - if len(step_seed_tensors) == 1: - step_seed_tensors[0] = new_step_seed - elif not step_seed_tensors: - tf.add_to_collection('step_seed', new_step_seed) - else: - raise ValueError('Multiple tensors in step_seed collection.') - - -def GetIncStepSeed(): - """Returns and increments the step_seed.""" - step_seed = GetStepSeed() - # TODO(lepikhin): introduce a routine filling a queue of uint32 random seeds - # independent of underlying PRNG used by tensorflow. - ResetStepSeed(step_seed + 1) - return step_seed - - -def GenerateStepSeedPair(p, global_step, op_seed=None): - """Generates a seed pair for deterministic random operations in ... - - functional loops. - - This function retrieves a unique seed pair on each call, based off the current - global step and step seed. The step seed ensures this function returns a - unique seed pair on each call: calling this function automatically increments - the step seed. The step seed is automatically reset at the beginning of each - global step in the model's FProp and works transparently through recurrent.py. - - Args: - p: A hyperparams.Params object, containing keys 'random_seed' and - 'is_inference'. - global_step: The global step. - op_seed: An additional operation-level seed to apply. - - Returns: - A size 2 tensor of op seeds to use for stateless_random ops. - """ - seed_dtype = tf.int32 if use_tpu() else tf.int64 - if p.is_inference and p.random_seed is None: - # Ensure GetIncStepSeed is called even inside the shortcut. - # This ensures if p.random_seed is set for other ops that use this function - # that they will get the same seed pair whether or not p.random_seed is set - # for this specific call. - GetIncStepSeed() - # Unlike tf.random*, stateless random ops are completely determined by the - # passed-in seeds. This means at inference time the same inputs will produce - # the same outputs, even if the model is supposed to have randomness such as - # dropout during inference. We inject additional randomness only during - # inference if the graph is exported with random_seed=None as a workaround. - return tf.random.uniform([2], maxval=seed_dtype.max, dtype=seed_dtype) - - global_step = tf.cast(global_step, seed_dtype) - step_seed = tf.cast(GetIncStepSeed(), seed_dtype) - seeds = tf.stack([global_step, step_seed]) - - if p.random_seed is not None: - seeds += p.random_seed - if op_seed is not None: - seeds += op_seed - return seeds - - -def DeterministicDropout(x, keep_prob, seeds, noise_shape=None, name=None): - """Similar to `tf.nn.dropout()`, but fully deterministic. - - Args: - x: A float Tensor on which to apply dropout. - keep_prob: A scalar `Tensor` of keep probability. - seeds: A Tensor of shape [2]. 2 seeds for deterministic random number - generator. - noise_shape: A 1-D `Tensor` of type `int32`, representing the shape for - randomly generated keep/drop flags. - name: An optional name for this operation. - - Returns: - A Tensor with the same shape as `x`. - - Raises: - InvalidArgumentError: if keep_prob is invalid. - """ - if isinstance(keep_prob, numbers.Real): - if keep_prob <= 0 or keep_prob > 1: - raise tf.errors.InvalidArgumentError( - 'keep_prob must be in range (0, 1]. Value: {}'.format(keep_prob)) - - if keep_prob == 1: - return x - with tf.name_scope(name, 'dropout', [x]) as name: - if use_tpu(): - seeds = tf.cast(seeds, tf.int32) - keep_prob = tf.convert_to_tensor( - keep_prob, dtype=tf.float32, name='keep_prob') - # uniform in [keep_prob, 1.0 + keep_prob) - # StatelessRandomUniform op does not support non-float (e.g. bfloat16) dtype - # and non-int32 seed types. - noise_shape = noise_shape or GetShape(x) - random_tensor = keep_prob + tf.random.stateless_uniform( - noise_shape, seed=seeds, dtype=tf.float32) - # 0. if [keep_prob, 1.0) and 1. if [1.0, 1.0 + keep_prob) - binary_tensor = tf.floor(random_tensor) - if x.dtype != tf.float32: - binary_tensor = tf.cast(binary_tensor, x.dtype) - keep_prob = tf.cast(keep_prob, dtype=x.dtype) - result = tf.div(x, keep_prob) * binary_tensor - result.set_shape(x.get_shape()) - return result - - -def DeterministicVN(params, seeds, noise_shape, mean=0.0, std=1.0, name=None): - """Produces Fully deterministic Gaussian noise from shape, mean and std. - - Args: - params: Nested map of params. - seeds: A Tensor of shape [2]. 2 seeds for deterministic random number - generator. - noise_shape: A 1-D `Tensor` of type `int32`, representing the shape for - randomly generated Gaussian noise. - mean: Mean for the Gaussian noise. - std: Standard deviation for noise. - name: An optional name for this operation. - - Returns: - A Tensor with the shape noise_shape and type fprop_dtype. - """ - - with tf.name_scope(name, 'gaussian_noise') as name: - if use_tpu(): - seeds = tf.cast(seeds, tf.int32) - random_tensor = mean + ( - std * tf.random.stateless_normal(noise_shape, seed=seeds)) - if FPropDtype(params) != tf.float32: - random_tensor = tf.cast(random_tensor, FPropDtype(params)) - return random_tensor - - -BATCH_NORM_UPDATES = 'batch_norm_updates' - -_BATCH_NORM_UPDATES_DICT = '__batch_norm_update_dict' -_get_batch_norm_updates_dict = _CollectionGetter(_BATCH_NORM_UPDATES_DICT, - lambda: {}) - - -def UpdateBatchNormVars(batch_norm_var, batch_norm_stats, decay): - """Update batch normalization moving averages.""" - with tf.name_scope( - 'AssignMovingAvg', values=[ - batch_norm_var, - batch_norm_stats, - decay, - ]) as scope: - with tf.ops.colocate_with(batch_norm_var): - decay = tf.convert_to_tensor( - 1.0 - decay, dtype=batch_norm_var.dtype.base_dtype) - update_delta = (batch_norm_var - tf.cast( - batch_norm_stats, batch_norm_var.dtype.base_dtype)) * decay - has_nan_or_inf = tf.reduce_any( - tf.math.logical_or( - tf.math.is_nan(update_delta), tf.math.is_inf(update_delta))) - update_delta = tf.cond(has_nan_or_inf, - lambda: tf.zeros_like(update_delta), - lambda: update_delta) - bn_update = tf.assign_sub(batch_norm_var, update_delta, name=scope) - tf.add_to_collection(BATCH_NORM_UPDATES, bn_update) - bn_update_dict = _get_batch_norm_updates_dict() - assert bn_update.name not in bn_update_dict - bn_update_dict[bn_update.name] = (batch_norm_var, batch_norm_stats) - return bn_update - - -def FindRelevantBatchNormUpdates(loss, batch_norm_updates): - """Finds and returns a list of relevant batch-normalization updates. - - Args: - loss: The loss that is being optimized for. A tensor or a list of tensors. - batch_norm_updates: A list of batch normalization updates. - - Returns: - A pair of lists. The first list contains all the batch normalization updates - that are relevant to the loss being optimized, and the second list contains - all in batch_norm_updates but not in the first list. - """ - dependent_ops_and_tensors = set(FindNeeded(loss)) - relevant_updates = [] - irrelevant_updates = [] - - bn_update_dict = _get_batch_norm_updates_dict() - for bn_update in batch_norm_updates: - assert bn_update.name in bn_update_dict, ( - '%s is probably not a valid batch normalization update op.' - ' Make sure batch normalization is done through calling' - ' the py_utils.UpdateBatchNormVars helper routine.') - bn_stat_name = bn_update_dict[bn_update.name][1].name - if bn_stat_name in dependent_ops_and_tensors: - # If a batch normalization stat is computed in the forward pass in - # computing loss, then the corresponding batch normalization update is - # relevant. Otherwise, it is not. - relevant_updates.append(bn_update) - else: - irrelevant_updates.append(bn_update) - return relevant_updates, irrelevant_updates - - -_SAMPLE_STEP_STACK = ThreadLocalStack() - - -@contextlib.contextmanager -def SampleStep(step): - """A context for a sample step during decoding. - - Example usage:: - - with py_utils.SampleStep(step): - sample = self.DecodeOneStep() - - Args: - step: the step tensor. - - Yields: - a context manager for the step scope. - """ - try: - _SAMPLE_STEP_STACK.stack.append(step) - yield step - finally: - _SAMPLE_STEP_STACK.stack.pop() - - -def _GetSampleStep(): - return _SAMPLE_STEP_STACK.stack[-1] if _SAMPLE_STEP_STACK.stack else None - - -def AddDebugTensor(tensor, summarize=None, name=None): - """Adds `tensor` to the debug collection. - - Prints the tensor if `--print_debug_tensors` is True. - - Args: - tensor: A tensor. - summarize: Only print this many entries of each tensor. If None, then a - maximum of 3 elements are printed per input tensor. - name: An optional name for the tensor. - - Returns: - A Tensor that evaluates to the same value as the input tensor. - """ - if _FromGlobal('print_debug_tensors'): - step = _GetSampleStep() - tensors_to_print = ([] if step is None else [step]) + [tensor] - with tf.name_scope(name) as s: - tensor = tf.Print( - tensor, - tensors_to_print, - message='DEBUG tensor %s' % s, - name=name, - summarize=summarize) - return tensor - - -def ArgMax(inputs): - """tf.argmax wrapper. - - Args: - inputs: A tensor, whose last dimension is being reduced on. - - Returns: - A tensor of rank tf.rank(logits)-1. If i == ret[indices], - logits[indices, i] is the maximum among logits[indices, :]. - """ - if use_tpu(): - return tf.argmax(inputs, axis=-1, output_type=tf.int32) - else: - return tf.argmax(inputs, axis=-1) - - -def _EnsureMatrixShape(x): - if x.shape.ndims is None: - x.set_shape([None, None]) - else: - assert x.shape.ndims == 2 - return x - - -def Matmul(x, y, *args, **kwargs): - """tf.matmul wrapper expecting x and y are actually matrices.""" - x = _EnsureMatrixShape(x) - y = _EnsureMatrixShape(y) - return tf.matmul(x, y, *args, **kwargs) - - -def clip_by_value(t, clip_value_min, clip_value_max, name=None): # pylint: disable=invalid-name - if t.dtype.is_complex: - return tf.complex( - tf.clip_by_value( - tf.math.real(t), clip_value_min, clip_value_max, '%s_real' % name), - tf.clip_by_value( - tf.math.imag(t), clip_value_min, clip_value_max, '%s_imag' % name)) - return tf.clip_by_value(t, clip_value_min, clip_value_max, name) - - -def _TransformAndSum(tensor_list, transform): - with tf.name_scope('TransformAndSum'): - sum_transform = [] - for t in tensor_list: - with tf.device(t.device): - if isinstance(t, tf.IndexedSlices): - sum_transform += [tf.reduce_sum(transform(t.values))] - else: - sum_transform += [tf.reduce_sum(transform(t))] - return tf.add_n(sum_transform) - - -def SumSquared(tensor_list): - return _TransformAndSum(tensor_list, lambda v: tf.abs(v)**2) - - -def SumAbs(tensor_list): - return _TransformAndSum(tensor_list, tf.abs) - - -def PiecewiseConstant(x_in, boundaries, values, vdtype): - """Returns the piecewise value of x_in.""" - x_in = tf.cast(tf.convert_to_tensor(x_in), tf.float32) - assert len(values) == len(boundaries) + 1 - assert sorted(boundaries) == list(boundaries) - bs = tf.convert_to_tensor(boundaries, dtype=tf.float32) - vs = tf.convert_to_tensor(values, dtype=vdtype) - # The following is equivalent to 'return vs[index]'. - index = tf.reduce_sum(tf.cast(tf.greater_equal(x_in, bs), tf.int32)) - one_hot_vec = tf.one_hot( - tf.expand_dims(index, 0), depth=len(values), dtype=vdtype) - return Matmul(tf.reshape(vs, (1, -1)), tf.transpose(one_hot_vec))[0][0] - - -def PadBatchDimension(x, batch_size, pad_val): - rank = tf.rank(x) - with tf.control_dependencies([assert_greater_equal(rank, 1)]): - current_batch_size = tf.shape(x)[0] - with tf.control_dependencies([assert_less_equal(current_batch_size, batch_size)]): - fake_samples_in_batch = batch_size - current_batch_size - pad = tf.scatter_nd([[0, 1]], [fake_samples_in_batch], [rank, 2]) - x = tf.pad(x, pad, constant_values=pad_val) - return x - - -def PadSequenceDimension(x, length, pad_val, shape=None): - """Pads x to `length` using `pad_val` along the second dim. - - Assumes `x` is a tensor with rank >= 2, and it only pads `x` to `length` - along the second dim. Explicitly sets the returned tensor shape to `shape` if - given. Raises runtime errors if x.shape[1] > length or x.shape[i] != shape[i] - where i != 1. - - Args: - x: the tensor to be padded with shape [batch, seq_len, ...]. - length: an int to specify the length to pad x to. - pad_val: an int or float used to pad x. - shape: an int array specifying the shape of the padded tensor if specified. - - Returns: - The padded tensor with shape [batch, seq_len, ...], where - ret[:, :seq_len, ...] == x. - """ - if x.shape.ndims is not None: - rank = x.shape.ndims - assert rank >= 2 - slen = GetShape(x, rank)[1] - pad_len = length - slen - pad = [[0, 0] for _ in range(rank)] - pad[1][1] = pad_len - else: - rank = tf.rank(x) - with tf.control_dependencies([assert_greater_equal(rank, 2)]): - slen = tf.shape(x)[1] - pad_len = length - slen - pad = tf.scatter_nd([[1, 1]], [pad_len], [rank, 2]) - x = tf.pad(x, pad, constant_values=pad_val) - if x.shape.ndims is not None and isinstance(length, int): - static_shape = x.shape.as_list() - static_shape[1] = length - x.set_shape(static_shape) - - if shape: - if not isinstance(shape, (list, tuple)): - raise TypeError('Shape must be a list or tuple.') - x = HasRank(x, len(shape)) - x = tf.ensure_shape(x, shape) - return x - - -def PadSequenceTo(xs, padding, length, pad_val): - """Pads `xs` and `padding` to `length` using `pad_val` along the 2nd dim. - - Pads `xs` to `length` using `pad_val`, and `padding` using 1. - Raise error if `x.shape[:2]` and `padding.shape` are not the same. - - Args: - xs: A Tensor or a list of Tensors of shape [batch, seqlen] or [batch, - seqlen, ...]. - padding: A 0/1 Tensor of shape [batch, seqlen]. 1 is for padded locations. - length: A Python int, the length to pad to. - pad_val: A Python numeric, used for padding x. - - Returns: - A tuple of padded xs and padding. - """ - if not isinstance(xs, (list, tuple)): - new_xs = [xs] - else: - new_xs = xs - - res = [] - for x in new_xs: - batch, slen = GetShape(x, 2) - - padding = HasRank(padding, 2) - padding = HasShape(padding, [batch, slen]) - - new_x = PadSequenceDimension(x, length, pad_val) - res.append(new_x) - padding = PadSequenceDimension(padding, length, tf.cast(1, padding.dtype)) - - if not isinstance(xs, (list, tuple)): - assert len(res) == 1 - return res[0], padding - else: - return tuple(res), padding - - -def ApplyPadding(padding, x, padded=None, broadcast=True, use_select=True): - """Applies padding to a tensor. - - This is preferable to using arithmetic means for masking out padded values - such as:: - - # Equiv to ApplyPadding(padding, x)) - x *= 1.0 - padding - # Equiv to ApplyPadding(padding, new, old) - new = old * padding + new * (1 - padding) - - Aside from just being easier to read and reason about, using this function - is friendly to quantized representations because it does not mix arithmetic - on the padding values with the values in the tensor being padded (which can - have a very different range than the 0..1 padding tensor). - - In addition, this works around issues in quantized schemes where we are - guaranteed to have an exact 0 but not necessarily any other number (i.e. 1). - - Args: - padding: Tensor of padding values where 0 == keep and 1 == pad. - x: Tensor to apply padding to. - padded: Optional. Values to include for padded elements. Defaults to zeros. - Must be the same shape as 'x' if specified. - broadcast: Whether to broadcast the padding shape to the shape of 'x'. You - almost certainly want this to be true as it matches how padding would be - expanded if applied arithmetically. - use_select: Controls whether padding is applied with a select-mask - (True/default) or arithmetically (False). Some platforms have a - sensitivity to one or the other and this is used to work around such - issues. - - Returns: - A tensor with the same shape as x with padded values masked. - """ - # Interesting dependencies usage - padding = with_dependencies([ - Assert( - tf.reduce_all( - tf.math.logical_or( - tf.equal(padding, 0.0), tf.equal(padding, 1.0))), [padding]) - ], padding) - if use_select: - if padded is None: - padded = tf.zeros_like(x) - if broadcast: - # Broadcast padding to the full shape. - padding = tf.cast(padding, x.dtype) * tf.ones_like(x) - return tf.where(padding > tf.zeros_like(padding), padded, x) - else: - result = x * tf.cast(1.0 - padding, x.dtype) - if padded is not None: - result += padded * tf.cast(padding, padded.dtype) - return result - - -def LengthsFromPaddings(paddings): - """Computes lengths of each sequence in a batch, ignoring trailing padding. - - Args: - paddings: a tensor with shape [batch, length]. - - Returns: - lengths tensor shaped [batch] containing the unpadded length of each - sequence in the batch. - """ - paddings = HasRank(paddings, 2) - paddings = tf.cast(paddings, tf.int32) - # Find the last unpadded value. - # Cannot just use tf.reduce_sum because there might be leading paddings. - # Everything after the last unpadded value has 1.0 - paddings == 0.0, so in - # the cumsum below they will have the same value. - cumsum = tf.cumsum(1 - paddings, axis=1) - same_as_last_element = tf.equal(cumsum, cumsum[:, -1:]) - # Counting the number of elements with the same value gives us num_padded + 1 - # and so counting the number that differs gives us num_padded - 1. - length = tf.reduce_sum( - 1 - tf.cast(same_as_last_element, tf.int32), axis=1) + 1 - # Special case for all 0 paddings. - all_zero_paddings = tf.equal(tf.reduce_sum(1 - paddings, axis=1), 0) - return tf.where(all_zero_paddings, tf.zeros_like(length), length) - - -def TrimTrailingPaddings(inputs, paddings): - """Trims trailing paddings from inputs. - - Since the number of dimensions is not fixed, this will not work on TPU. - - Args: - inputs: a tensor with shape [batch, length, ...]. - paddings: a tensor with shape [batch, length]. - - Returns: - Trimmed inputs and paddings. For compatibility reasons, the trimmed tensors - will always have length at least 1. - """ - paddings = HasRank(paddings, 2) - max_length = tf.maximum(tf.reduce_max(LengthsFromPaddings(paddings)), 1) - output_shape = tf.shape(inputs) - output_shape = tf.concat([[output_shape[0], max_length], output_shape[2:]], - axis=0) - outputs = tf.slice(inputs, tf.zeros_like(output_shape), output_shape) - out_paddings = tf.slice(paddings, [0, 0], - tf.stack([output_shape[0], max_length])) - return outputs, out_paddings - - -def ReversePaddedSequence(inputs, paddings): - """Reverse inputs based on paddings. - - Only reverse the unpadded portion of `inputs`. It assumes inputs are only - padded in the end. - - Args: - inputs: a tensor of [seq_length, batch_size, num_input_nodes]. - paddings: a tensor of float32/float64 zero or one of shape [seq_length, - batch_size, 1]. - - Returns: - A reversed tensor of the same shape as `inputs`. - """ - inversed_paddings = 1.0 - tf.squeeze(paddings, 2) - inputs_length = tf.cast( - tf.math.rint(tf.reduce_sum(inversed_paddings, axis=0)), tf.int32) - return tf.reverse_sequence(inputs, inputs_length, seq_axis=0, batch_axis=1) - - -def ConcatenatePaddedSequences(input0, input1, padding0, padding1, seq_dim=1): - """Concatenates input sequences with varying lenghts as defined by paddings. - - This is a helper function for concatenating 2 batches of input sequences, - where each example in the batch can have different lengths, as defined by - the corresponding paddings. To concatenate correctly, it makes use of - tf.reverse_sequence to partially reverse the sequences before - concatenating them together. - - NOTE: We assume that the tensors have no leading paddings. - - Args: - input0: A tensor of size [batch, max_length, ...] or [max_length, batch, - ...] depending on the value set for axis. - input1: A tensor of size [batch, max_length, ...] or [max_length, batch, - ...] depending on the value set for axis. - padding0: A Tensor of size [batch, max_length] or [max_length, batch] - corresponding to the padding for input0. - padding1: A Tensor of size [batch, max_length] or [max_length, batch] - corresponding to the padding for input1. - seq_dim: int, the time axis along which the tensors will be concatenated. - Should be 0 or 1. Assumes that batch_dim is 1 - seq_dim. - - Returns: - The concatenation of input0 and input1, and the corresponding padding. - - Raises: - tf.errors.InvalidArgumentError when seq_dim is not 0 or 1. - """ - if seq_dim != 0 and seq_dim != 1: - raise tf.errors.InvalidArgumentError(None, None, 'seq_dim must be 0 or 1.') - batch_dim = 1 - seq_dim - # inpu0 and input1 should have the same batch size and same rank. - input0 = with_dependencies([ - assert_equal(GetShape(input0)[batch_dim], - GetShape(input1)[batch_dim]), - assert_equal(GetRank(input0), GetRank(input1)) - ], input0) - - batch_size = GetShape(padding0)[batch_dim] - # batch dimension of inputs and paddings should match. - input0 = with_dependencies([ - assert_equal(GetShape(input0)[batch_dim], batch_size), - assert_equal(GetShape(padding1)[batch_dim], batch_size) - ], input0) - input0_seq_dim = tf.cast( - tf.tile([tf.shape(padding0)[seq_dim]], [batch_size]), dtype=tf.int32) - input1_seq_dim = tf.cast( - tf.tile([tf.shape(padding1)[seq_dim]], [batch_size]), dtype=tf.int32) - # LengthsFromPaddings assumes that paddings is of size [batch, max_length]. - if seq_dim == 1: - seq_length0 = LengthsFromPaddings(padding0) - seq_length1 = LengthsFromPaddings(padding1) - else: - seq_length0 = LengthsFromPaddings(tf.transpose(padding0)) - seq_length1 = LengthsFromPaddings(tf.transpose(padding1)) - # We assume that the tensors have no leading paddings. - # TODO(arunnt): Concatenate tensors with leading paddings correctly. - seq_length0 = with_dependencies([ - assert_equal( - seq_length0, - tf.cast(tf.reduce_sum(1.0 - padding0, seq_dim), dtype=tf.int32)) - ], seq_length0) - seq_length1 = with_dependencies([ - assert_equal( - seq_length1, - tf.cast(tf.reduce_sum(1.0 - padding1, seq_dim), dtype=tf.int32)) - ], seq_length1) - # Concatenate input sequences. - reversed_input0 = tf.reverse_sequence( - input0, seq_length0, seq_axis=seq_dim, batch_axis=batch_dim) - reversed_input1 = tf.reverse_sequence( - input1, input1_seq_dim, seq_axis=seq_dim, batch_axis=batch_dim) - reversed_concat = tf.concat([reversed_input1, reversed_input0], axis=seq_dim) - concat_inputs = tf.reverse_sequence( - reversed_concat, - seq_length0 + input1_seq_dim, - seq_axis=seq_dim, - batch_axis=batch_dim) - # Concatenate paddings. Note that paddings are always a Tensor of 0s and 1s, - # so, unlike the inputs, we don't have to reverse padding1, we can simply - # concatenate reversed padding0 and padding1. - reversed_padding0 = tf.reverse_sequence( - padding0, input0_seq_dim, seq_axis=seq_dim, batch_axis=batch_dim) - reversed_concat_padding = tf.concat([reversed_padding0, padding1], - axis=seq_dim) - concat_paddings = tf.reverse_sequence( - reversed_concat_padding, - input0_seq_dim + seq_length1, - seq_axis=seq_dim, - batch_axis=batch_dim) - return concat_inputs, concat_paddings - - -def ShiftLeft(tensor, shift_size, pad_val=0): - """Shifts the values in a tensor to the left along the time dimension (dim 1). - - The first shift_size values are dropped, and the tensor is padded on the - right with pad_val. - - Args: - tensor: the input tensor to modify shaped [batch, time, ...]. - shift_size: the number of frames >= 0 to shift. - pad_val: the value to pad on the right of the tensor. - - Returns: - A left shifted tensor. - """ - with tf.control_dependencies([ - assert_greater_equal(tensor.shape.rank, 2), - assert_greater_equal(shift_size, 0) - ]): - time = GetShape(tensor)[1] - return PadSequenceDimension(tensor[:, shift_size:], time, pad_val) - - -def Retry(*args, **kwargs): - return retry.Retry(*args, **kwargs) - - -# FailedPreconditionError: variables are not initialized. -# AbortedError: processes restarts. -# UnavailableError: Bad hardware status: 0x1 -transient_tf_errors = (tf.errors.FailedPreconditionError, - tf.errors.AbortedError, tf.errors.UnavailableError) - - -def RetryOnTransientTfError(*args, **kwargs): - return Retry(transient_tf_errors, *args, **kwargs) - - -def PadOrTrimTo(x, shape, pad_val=0, pad_after_contents=True): - """Pad and slice x to the given shape. - - Args: - x: A tensor. - shape: The shape of the returned tensor. - pad_val: An int or float used to pad x. - pad_after_contents: Whether to pad and trim after the original contents - of each dimension. - - Returns: - 'x' is padded with pad_val and sliced so that the result has the given - shape. - - Raises: - ValueError: if shape is a tf.TensorShape and not fully defined. - """ - if isinstance(shape, (list, tuple)): - expected_rank = len(shape) - elif isinstance(shape, tf.TensorShape): - if not shape.is_fully_defined(): - raise ValueError('shape %s padding %s must be fully defined.' % - (shape, x)) - expected_rank = shape.rank - else: - shape = HasRank(shape, 1) - expected_rank = tf.size(shape) - x = HasRank(x, expected_rank) - - pad = shape - tf.minimum(tf.shape(x), shape) - zeros = tf.zeros_like(pad) - if pad_after_contents: - # If dim_i is less than shape[i], pads after contents. - paddings = tf.stack([zeros, pad], axis=1) - # If dim_i is larger than shape[i], we slice [0:shape[i]] for dim_i. - slice_begin = zeros - else: - # If dim_i is less than shape[i], pads before contents. - paddings = tf.stack([pad, zeros], axis=1) - # If dim-i is larger than shape[i], we slice [dim_i - shape[i]:dim_i] - # for dim_i. - slice_begin = tf.shape(x) + pad - shape - - x = tf.pad(x, paddings, constant_values=pad_val) - x = tf.slice(x, slice_begin, shape) - - return tf.reshape(x, shape) - - -def RepeatDim(tensor, multiple, axis): - """Copies elements in tensor's axis "multiple" times, like np.repeat.""" - # x = [[1, 2, 3], [4, 5, 6]] - # RepeatDim(x, multiple=2, axis=1) gives: - # [[1, 1, 2, 2, 3, 3]. [4, 4, 5, 5, 6, 6]] - # As a comparison tf.tile(x, multiples=[1, 2]) gives:\ - # [[1, 2, 3, 1, 2, 3], [4, 5, 6, 4, 5, 6]] - - if multiple == 1: - return tensor - t_shape = tf.shape(tensor) - tensor_dims = tf.concat( - [t_shape[:axis], [t_shape[axis] * multiple], t_shape[axis + 1:]], 0) - multiple_dims = tf.concat([ - tf.fill([axis + 1], 1), [multiple], - tf.fill([tf.rank(tensor) - axis - 1], 1) - ], 0) - return tf.reshape( - tf.tile(tf.expand_dims(tensor, axis + 1), multiple_dims), tensor_dims) - - -def StackTensorsRecursively(values): - """Recursively stacks Tensors in a list of `.NestedMap`. - - Args: - values: a list of `.NestedMap` or Tensors to stacks. - - Returns: - A `.NestedMap` with stacked values or a stacked Tensor. - """ - flatten = [w.Flatten() for w in values] - stacked = [] - for i in range(len(flatten[0])): - stacked += [tf.stack([flatten[j][i] for j in range(len(flatten))])] - ret = values[0].Pack(stacked) - return ret - - -def MixByWeight(inputs, weights, seed=None): - """Returns a weighted random choice and bprop type from the give inputs. - - Args: - inputs: a list of callables, where each callable returns a tf.Tensor or a - nested structure containing tf.Tensor. Function return types must be - consistent across elements. The tf.Operation to compute the result tensor - will only be invoked for one input at a time. For example, if each fn - represents an input record stream, a record will be drawn only from a - selected stream while the other streams will remain unchanged. - weights: a 1D tensor of float > 0 of the same length as inputs. - seed: random seed. - - Returns: - A probablistic sample from the inputs proportional to the weights. The - return type will be the same as return type of individual 'fn' from the - inputs. - A one-hot vector of the source selected. - """ - weights = tf.convert_to_tensor(weights, dtype=tf.float32) - weights = with_dependencies([ - assert_equal(tf.shape(weights), [len(inputs)]), - assert_greater_equal(tf.reduce_min(weights), 0.0) - ], weights) - - lower = tf.cumsum(weights, exclusive=True) - upper = tf.cumsum(weights, exclusive=False) - r = tf.random.uniform(shape=[], maxval=upper[-1], seed=seed) - return_input = tf.case( - [(tf.math.logical_and(lower[i] <= r, r < upper[i]), inputs[i]) - for i in range(len(inputs))], - exclusive=True) - selected_index = tf.case( - [(tf.math.logical_and(lower[i] <= r, r < upper[i]), lambda i=i: i) - for i in range(len(inputs))], - exclusive=True) - bprop_index = tf.one_hot(selected_index, len(inputs), dtype=tf.float32) - return return_input, bprop_index - - -def CheckShapes(shapes): - """Asserts that shapes is a tuple of NestedMap or tshape.Shape.""" - assert isinstance(shapes, tuple), str(shapes) - for s in shapes: - if isinstance(s, NestedMap): - assert all([isinstance(t, tshape.Shape) for t in Flatten(s) - ]), '{} contains non-tensor value.'.format(s) - else: - assert isinstance(s, tshape.Shape), '{}: {}'.format(type(s), s) - - -def FPropDtype(params): - return params.fprop_dtype if params.fprop_dtype is not None else params.dtype - - -def UpdateFpropDtype(params, fprop_dtype): - """Recursively update the fprop_dtype of the Params.""" - # Handle the case when the input "params" is not an instance of hyperparams - # For example, when UpdateDtype is called recursively for all the items in - # the "sub" list of SequentialLayer (see 1st elif below) - if not isinstance(params, hyperparams.Params): - return - - for key, val in params.IterParams(): - if isinstance(val, hyperparams.Params): - UpdateFpropDtype(val, fprop_dtype) - elif isinstance(val, (list, tuple)): - for item in val: - UpdateFpropDtype(item, fprop_dtype) - elif key == 'fprop_dtype': - params.fprop_dtype = fprop_dtype - - -def UpdateDtype(params, dtype): - """Recursively update the dtype of the Params.""" - # Handle the case when the input "params" is not an instance of hyperparams - # For example, when UpdateDtype is called recursively for all the items in - # the "sub" list of SequentialLayer (see 1st elif below) - if not isinstance(params, hyperparams.Params): - return - - for key, val in params.IterParams(): - if isinstance(val, hyperparams.Params): - UpdateDtype(val, dtype) - elif isinstance(val, (list, tuple)): - for item in val: - UpdateDtype(item, dtype) - elif key == 'dtype': - params.dtype = dtype - - -def NameScopeDecorator(name_scope): - """Decorates a python function to introduce a tf.name_scope. - - Example:: - - @py_utils.NameScopeDecorator('foobar') - def MyFoobarMethod(self): - # ... Do TF things - - Args: - name_scope: The name scope to introduce. - - Returns: - A function decorator. - """ - - def Decorator(f): - - def Wrapped(*args, **kwargs): - with tf.name_scope(name_scope): - return f(*args, **kwargs) - - return Wrapped - - return Decorator - - -def SequencesToDebugStrings(ids, lens, summarize=5): - """Returns debug strings for the given sequences. - - Args: - ids: int32 of [batch, len]. - lens: int32 of [batch]. - summarize: number of ids to summarize per sequence. - - Returns: - A string tensor of [batch]. - """ - num_seqs = tf.shape(lens)[0] - - def _Body(i, result): - line = tf.strings.format('{}', ids[i, :lens[i]], summarize=summarize) - return i + 1, tf.concat([result, tf.reshape(line, [1])], axis=0) - - i0 = tf.zeros(shape=[], dtype=tf.int32) - result0 = tf.constant('', shape=[0], dtype=tf.string) - _, strs = tf.while_loop( - lambda i, result: i < num_seqs, - _Body, (i0, result0), - shape_invariants=(i0.shape, tf.TensorShape([None]))) - return strs - - -def RematerializeFn(fn, *xs): - """Calls fn and rematerializes fn in the backward pass. - - `fn(*xs) -> ys`, where xs and ys can be a single tensor or a tuple of tensors. - - Args: - fn: A python function to be rematerialized in the backprop pass. - *xs: A single tensor or a list/tuple of tensors. `xs` are input args to the - fn function. - - Returns: - `fn(*xs)` - """ - initial_step_seed = GetStepSeed() - final_step_seed = GenerateSeedFromName(tf.no_op(name='new_step_seed').name) - - def Backward(fwd_xs, fwd_ys, d_fwd_ys): - """The backward function that rematerializes forward outputs.""" - del fwd_ys - always_true = tf.random.uniform([]) < 2.0 - # Alternatively, can do this: - # tf.where(tf.math.is_nan(x), - # tf.constant(float('nan'), dtype=x.dtype) * tf.ones_like(x), - # x) - bak_xs = [tf.where(always_true, x, tf.zeros_like(x)) for x in fwd_xs.xs] - for dst, src in zip(bak_xs, xs): - dst.set_shape(src.shape) - ResetStepSeed(initial_step_seed) - ys = fn(*bak_xs) - ResetStepSeed(final_step_seed) - dxs = tf.gradients(ys, bak_xs, grad_ys=d_fwd_ys) - dxs_final = [] - for dx, x in zip(dxs, bak_xs): - if dx is None: - dxs_final.append(tf.zeros_like(x)) - else: - dxs_final.append(dx) - assert len(dxs_final) == len(bak_xs) - return NestedMap( - initial_step_seed=tf.zeros_like(initial_step_seed), xs=dxs_final) - - ys_shapes = [] - - # TODO(huangyp, yonghui): Check Forward doesn't use any stateful random ops. - def Forward(fwd_xs): - """Forward function plus sanity checks.""" - for dst, src in zip(fwd_xs.xs, xs): - dst.set_shape(src.shape) - ResetStepSeed(fwd_xs.initial_step_seed) - ys = fn(*fwd_xs.xs) - # Some sanity check. - assert not GetExtraInputs() - assert not GetExtraArgs() - assert not GetExtraVars() - if isinstance(ys, tuple): - for y in ys: - assert isinstance(y, tf.Tensor) - ys_shapes.append(y.shape) - else: - assert isinstance(ys, tf.Tensor) - ys_shapes.append(ys.shape) - return ys - - ys = CallDefun( - Forward, - NestedMap(initial_step_seed=initial_step_seed, xs=xs), - bak=Backward) - if isinstance(ys, tuple): - for y, s in zip(ys, ys_shapes): - y.set_shape(s) - else: - ys.set_shape(ys_shapes[0]) - # TODO(b/129159299): The ResetStepSeed below is needed to work around this - # bug, which is a problem with global tensors being shared by different - # inference graphs. It should be replaced with the new step seed value - # returned from the Forward function when the bug is fixed. - ResetStepSeed(final_step_seed) - return ys - - -# A set of names of stateful random number generator ops. -# See tensorflow/core/ops/random_ops.cc -_STATEFUL_RANDOM_OPS = frozenset({ - # pyformat: disable - 'RandomUniform', - 'RandomUniformInt', - 'RandomStandardNormal', - 'ParameterizedTruncatedNormal', - 'TruncatedNormal', - 'RandomShuffle', - 'Multinomial', - 'RandomGamma', - 'RandomPoisson', - 'RandomPoissonV2', - # pyformat: enable -}) - - -def StatefulRandomOpsInDefun(func, graph=None): - """Checks whether the Defun depends on stateful random number ops. - - Stateful random number generator ops should be avoid in Recurrent() call. - Otherwise, these ops produce inconsistent values between FProp and BProp. - - Args: - func: a _DefinedFunction or ConcreteFunction to check. - graph: a Graph. Set None to use the default graph. - - Returns: - A list of names of the stateful random ops. - - Raises: - InvalidArgumentError: if the input func/graph is invalid. - """ - if graph is None: - graph = tf.get_default_graph() - func.add_to_graph(graph) - graph_def = graph.as_graph_def() - - # A dict from function name to FunctionDef. - func_defs = {x.signature.name: x for x in graph_def.library.function} - - if isinstance(func, function._DefinedFunction): # pylint: disable=protected-access - if func.definition.signature.name not in func_defs: - raise tf.errors.InvalidArgumentError( - None, None, 'Defun {} is not in the graph .'.format( - func.definition.signature.name)) - nodes = py_collections.deque(func.definition.node_def) - else: - nodes = py_collections.deque(func.function_def.node_def) - - stateful_ops = [] - - # Recursively search for stateful random op. - while nodes: - node = nodes.pop() - assert isinstance(node, node_def_pb2.NodeDef), node - - if node.op in _STATEFUL_RANDOM_OPS: - stateful_ops.append(node.op) - continue - - def _AddDefunNodes(func_name): - """If the given func_name is a Defun, add its sub-nodes into nodes.""" - if func_name in func_defs: - nodes.extend(func_defs[func_name].node_def) - - # For functional.{While|For|If} ops, add their Defun attr into search. - if node.op == 'While': - _AddDefunNodes(node.attr['body'].func.name) - _AddDefunNodes(node.attr['cond'].func.name) - elif node.op == 'For': - _AddDefunNodes(node.attr['body'].func.name) - elif node.op == 'If': - _AddDefunNodes(node.attr['then_branch'].func.name) - _AddDefunNodes(node.attr['else_branch'].func.name) - elif node.op == 'StatefulPartitionedCall': - _AddDefunNodes(node.attr['f'].func.name) - elif node.op != 'PartitionedCall': - # For other op, check whether itself is a Defun op. - _AddDefunNodes(node.op) - - return stateful_ops - - -def ToPlaceholders(nmap, dtype=None): - """Converts every Tensor in nmap to a placeholder.""" - - def _ToPlacerholder(x): - shape = [None for _ in x.shape[:-1]] + [x.shape[-1]] - return tf.placeholder(dtype=dtype or x.dtype, shape=shape) - - return nmap.Transform(_ToPlacerholder) - - -def SoftmaxCrossEntropyFocalLoss(logits, - label_ids=None, - label_probs=None, - alpha=None, - gamma=None): - u"""Focal loss for multinomial (softmax) logistic loss. - - [1] Focal loss https://arxiv.org/abs/1708.02002 - - Args: - logits: [..., C]. Logits for the multinomial logistic regression. C is the - number of classes. - label_ids: [...]. Each entry in labels must be an index in [0, C). - label_probs: [..., C]. Each vector along last dimension must be a valid - probability distribution. - alpha: [C]. The weighting factor alpha. Eq (3) in [1]. - gamma: []. Tunable focusing parameter. Eq (4) in [1]. - - Returns: - loss[i..., j] = FL(pₜ) = - αₜ(1-pₜ)ˠlog(pₜ) Eq (5) in [1]. - """ - if label_probs is not None: - log_probs = tf.nn.log_softmax(logits) - loss = -(label_probs * log_probs) - if gamma is not None and gamma != 0: - probs = tf.exp(log_probs) - loss *= tf.pow(1.0 - probs, gamma) - if alpha is not None: - loss *= tf.reshape( - alpha, tf.concat([tf.ones(tf.rank(loss) - 1, tf.int32), [-1]], - axis=0)) - loss = tf.reduce_sum(loss, axis=-1) - else: - loss = tf.nn.sparse_softmax_cross_entropy_with_logits( - labels=label_ids, logits=logits) - if gamma is not None and gamma != 0: - probs = tf.exp(-loss) - loss *= tf.pow(1.0 - probs, gamma) - if alpha is not None: - loss *= tf.gather(alpha, label_ids) - return loss - - -def SigmoidCrossEntropyFocalLoss(logits, labels, alpha=None, gamma=None): - u"""Focal loss for binary (sigmoid) logistic loss. - - [1] Focal loss https://arxiv.org/abs/1708.02002 - - Args: - logits: [..., C]. Logits for the sigmoid logistic regression. - labels: [..., C]. 0/1 labels. - alpha: The weighting factor alpha. Eq (3) in [1]. - gamma: Tunable focusing parameter. Eq (4) in [1]. - - Returns: - loss[i..., j] = FL(pₜ) = - αₜ(1-pₜ)ˠlog(pₜ) Eq (5) in [1]. - """ - - # [1] Eq (4). - # - # The numerically-stable way to compute - # log(p) for positives; - # log(1 - p) for negatives. - loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=labels, logits=logits) - - if gamma is not None and gamma != 0: - # The modulating factor. Note that - # (1 - p)ˠ = [1 - σ(x)]ˠ = [σ(-x)]ˠ, for positives. - # pˠ = [σ(x)]ˠ, for negatives. - loss *= tf.pow(tf.sigmoid(logits * (1 - labels * 2)), gamma) - - if alpha is not None: - # [1] Eq (3) - loss *= (alpha * labels + (1 - alpha) * (1 - labels)) - - return loss - - -_RECORD_FORMAT_RE = re.compile('(^[A-Za-z]+):(.*)') - - -def RecordFormatFromFilePattern(file_pattern): - """Return the record format string for a Lingvo file pattern. - - Lingvo file patterns take the form of: - tfrecord:/path/to/bar -> tfrecord is the record_format. - - This function takes a file pattern and returns a string indicating - which format the filepattern implies. - - Args: - file_pattern: String file pattern. - - Returns: - Tuple (string, string): - - - record_format: String record format, e.g., "tfrecord", etc. - - file_pattern: The file pattern without any prefixes. - """ - result = re.match(_RECORD_FORMAT_RE, file_pattern) - - if result is None: - # TODO(vrv): Fix all callers so that file_pattern must contain - # the record format prefix. - return 'sstable', file_pattern - - # regexp ensures that a match implies there are two groups: - # the record format and then the file pattern. - return result.groups() - - -def ReadFileLines(file_path): - """Read a text file and return the lines. - - If the file cannot be found at the given path, attempt to load it from the - Lingvo package (useful for data dependencies in par files). - - Args: - file_path: path to file, either absolute or relative to the bazel workspace. - - Returns: - A list of lines from the file. - """ - if not tf.io.gfile.exists(file_path): - try: - lines = pkgutil.get_data( - 'lingvo', file_path.replace('lingvo/', '', - 1)).splitlines(True) - except IOError: - # If pkgutil can't find the file, continue and let GFile raise the error. - lines = None - else: - lines = None - - if not lines: - with tf.io.gfile.GFile(file_path, 'r') as f: - lines = f.readlines() - - return lines - - -# Partially borrowed from -# https://github.com/tensorflow/tensor2tensor/blob/32929305e1a4ec926eff24123758b794df35492b/tensor2tensor/layers/common_layers.py#L349 -def CumSum(x, axis=0, exclusive=False): - """A TPU efficient implementation of tf.cumsum(). - - This is equivalent to tf.cumsum and is faster on TPU as of 08/2019 unless - the axis dimension is very large. The current Tensorflow implementation is - based on scanning and reducing which is not efficient on TPU. - - Args: - x: An input Tensor. - axis: An int for the axis. - exclusive: A bool for performing exclusive cumsum. - - Returns: - A Tensor of the same shape as x. - - Raises: - ValueError: if the input axis is invalid. - """ - if x.dtype not in (tf.float32, tf.bfloat16) or not use_tpu(): - # Fallback to tf.cumsum when inputs are not floats or not running on TPU. - return tf.cumsum(x, axis=axis, exclusive=exclusive) - - rank = GetRank(x) - # Needs to know the rank for the final transpose if axis is not the last - # dimension. Otherwise, falls back to tf.cumsum. - if not isinstance(rank, int) and axis != -1: - return tf.cumsum(x, axis=axis, exclusive=exclusive) - - if axis < -1: - if axis + rank < 0: - raise ValueError('Unexpected axis: %d (rank = %d)' % (axis, rank)) - axis += rank - - length = GetShape(x)[axis] - my_range = tf.range(length) - comparator = tf.less if exclusive else tf.less_equal - mask = tf.cast( - comparator(tf.expand_dims(my_range, 1), tf.expand_dims(my_range, 0)), - x.dtype) - result = tf.tensordot(x, mask, axes=[[axis], [0]]) - if axis != -1 and axis != rank - 1: - result = tf.transpose( - result, - list(range(axis)) + [rank - 1] + list(range(axis, rank - 1))) - return result - - -def ProjectLastDim(inputs, weight, input_dim, output_dim): - """Linear projection on the last dim of the input tensor. - - This is a TPU efficient implementation to avoid reshaping inputs to Rank-2 - tensor by using Einsum for the compute. - - Args: - inputs: An input Tensor, the last dimension of which is input_dim. - weight: A weight matrix with shape [input_dim, output_dim]. - input_dim: An integer or a symbolic dim, the last dimension of the inputs. - output_dim: An integer or a symbolic dim, the last dimension of the outputs. - - Returns: - An output Tensor of the same rank as inputs, the last dimension is - output_dim. - """ - input_dim = int( - symbolic.ToStatic(input_dim) if symbolic.IsExpr(input_dim) else input_dim) - output_dim = int( - symbolic.ToStatic(output_dim) if symbolic.IsExpr(output_dim - ) else output_dim) - - # Assert input_dim and output_dim - inputs = with_dependencies([assert_equal(GetShape(inputs)[-1], input_dim)], - inputs) - weight = with_dependencies([ - assert_equal(GetShape(weight)[0], input_dim), - assert_equal(GetShape(weight)[-1], output_dim) - ], weight) - - if (use_tpu() and inputs.shape is not None and - inputs.shape.rank is not None and inputs.shape.rank < 26): - # Avoids reshape if feasible and uses Einsum. - if inputs.shape.rank == 2: - outputs = tf.matmul(inputs, weight) - else: - s = ''.join([chr(x) for x in range(97, 123)]) # abc...xyz - r = inputs.shape.rank - outputs = tf.einsum('{0}y,yz->{0}z'.format(s[:r - 1]), inputs, weight) - else: - outputs = Matmul(tf.reshape(inputs, ToStaticShape([-1, input_dim])), weight) - outputs = tf.reshape( - outputs, - tf.concat([ - tf.cast(GetShape(inputs)[:-1], tf.int32), - ToStaticShape([output_dim]) - ], - axis=0)) - - return outputs - - -@contextlib.contextmanager -def RemoveAssertContext(remove=True): - """Hacks to replace certain unwanted tensorflow ops.""" - # TODO(zhifengc/huangyp): Consider implementing assert_equal - # op replacement for lingvo. As assert_equal doesn't support String on GPUs. - # Hack to replace tf.assert_equal - # TODO(b/136040013): Remove this after migration to tf.function. - if remove: - saved_assert_equal = tf.check_ops.assert_equal - - def NoOP(*args, **kwargs): # pylint: disable=unused-argument - return tf.no_op() - - tf.check_ops.assert_equal = NoOP # Make assert_equal a no op. - try: - yield - finally: - tf.check_ops.assert_equal = saved_assert_equal - else: - yield - - -def _AssertInputsMatch(op, args, implicit_captures): - """Assert that op's inputs match with args and implicit_captures. - - Args: - op: The operation to check. - args: A nested structure representing the explicit arguments of 'op'. - implicit_captures: A nested structure representing the implicitly captured - inputs of 'op'. - - Raises: - ValueError: if the number of inputs mismatch. - """ - expected_inputs = Flatten([args, implicit_captures]) - expected_num_inputs = len(expected_inputs) - if len(op.inputs) > expected_num_inputs: - raise ValueError(('Too many inputs. The most likely cause is that fwd ' - 'captures additional tensors: extra inputs %r vs %r ' - 'captures=%r') % (list(op.inputs), list(expected_inputs), - list(Flatten(implicit_captures)))) - if len(op.inputs) < expected_num_inputs: - raise ValueError(('Mismatched inputs to fwd: Found %d vs expected %d: %r' - '. Implicit captures(%d) = %r') % - (len(op.inputs), expected_num_inputs, list(op.inputs), - len(Flatten(implicit_captures)), implicit_captures)) - - -def _TensorSpecs(nmap): - """Transforms tensors in the input nested structure to TensorSpecs.""" - return Transform(lambda t: tf.TensorSpec(t.shape, t.dtype), nmap) - - -def _DefineDefun(fwd, - fwd_sig, - bak=None, - bak_as_function=False, - device=None): - """Wraps fwd in a defun with custom gradient bak. - - Args: - fwd: A callable xs: Nested Structure -> ys: Nested Structure. - fwd_sig: A Nested Structure of tf.Tensor representing the input signature of - fwd. - bak: A callable xs, ys, dys: Nested Structure -> dxs[, dcapture]: Nested - Structure. The custom backprop function for fwd. bak needs to return - dcapture if fwd uses any implicitly captured tensors, whose gradients are - dcapture. - bak_as_function: Whether to create a TF graph function for bak. - device: the device on which to run fwd and bak. - - Returns: - A function that wraps fwd. - """ - assert fwd is not None - - # fwd signature (tf.Tensor dtypes). - get_dtype = lambda x: x.dtype - arg_dtypes = Flatten(Transform(get_dtype, fwd_sig)) - - get_shape = lambda x: x.shape - arg_shapes = Flatten(Transform(get_shape, fwd_sig)) - - compiled = use_xla() - noinline = not compiled - - # Used to hold: - # - the output dtypes/shapes of fwd, which will be set in Forward. - # - the real Backward function used by Grad, which will be defined later. - sigs = NestedMap() - - python_grad_func = None - implicit_captures = NestedMap() # Will be set after Forward is defined. - if bak: - - def Grad(op, *args): - """Gradient function for the forward function. - - Args: - op: The forward operation. - *args: Gradients wrt op.outputs. - - Returns: - Tuple of derivatives. - """ - _AssertInputsMatch(op, fwd_sig, implicit_captures) - # Ensure dys contains no None. - args = ConvertNoneGradientToZeros(list(op.outputs), list(args)) - xs = op.inputs[:len(arg_dtypes)] # The rest are captures. - return sigs.backward(*Flatten([xs, op.outputs, args])) - - python_grad_func = Grad - - def _SetShape(dst_list, shape_list): - for dst, shape in zip(dst_list, shape_list): - if isinstance(dst, tf.Tensor): - dst.set_shape(shape) - - @tf.Defun(*arg_dtypes, python_grad_func=python_grad_func, noinline=noinline) - def Forward(*args): - """The forward function.""" - _SetShape(args, arg_shapes) - with RemoveAssertContext(remove=noinline), tf.device(device): - rets = fwd(Pack(fwd_sig, args)) - sigs.ret_dtypes = Transform(get_dtype, rets) - sigs.ret_shapes = Transform(get_shape, rets) - return Flatten(rets) - - # Invokes fwd() to get sigs.ret_dtypes and sigs.ret_shapes. - Forward.add_to_graph(tf.get_default_graph()) - implicit_captures.captured = Forward.captured_inputs - - if bak: - - def Backward(*args): - """The backward function.""" - xs_len = len(arg_dtypes) - ys_len = len(Flatten(sigs.ret_dtypes)) - xs = args[:xs_len] - ys = args[xs_len:(xs_len + ys_len)] - dys = args[xs_len + ys_len:] - assert len(dys) == ys_len - - _SetShape(xs, arg_shapes) - _SetShape(ys, Flatten(sigs.ret_shapes)) - _SetShape(dys, Flatten(sigs.ret_shapes)) - - xs = Pack(fwd_sig, xs) - ys = Pack(sigs.ret_dtypes, ys) - dys = Pack(sigs.ret_dtypes, dys) - with RemoveAssertContext(remove=noinline), tf.device(device): - dxs = bak(xs, ys, dys) - return Flatten(dxs) - - if bak_as_function: - sigs.backward = tf.Defun( - *Flatten([arg_dtypes, sigs.ret_dtypes, sigs.ret_dtypes]), - noinline=noinline)( - Backward) - - sigs.backward.add_to_graph(tf.get_default_graph()) - else: - sigs.backward = Backward - - def Call(args): - """Wrapper of fwd.""" - flat_rets = Forward(*Flatten(args)) - if not isinstance(flat_rets, (tuple, list)): - flat_rets = [flat_rets] - _SetShape(flat_rets, Flatten(sigs.ret_shapes)) - return Pack(sigs.ret_dtypes, flat_rets) - - return Call - - -# Global variable to control rendezvous sharing in tf.function. -# If False (default) rendezvous sharing is disabled in tf.function, that is, the -# function body use a separate rendezvous and can't communicate with parent -# graph via send/recv. -# With _GetSharedRendezvous() == True, the function body share the same -# rendezvous with the parent graph and can talk to it using send/recv. This is -# useful for layers like StackedRecurrent. -_SHARED_RENDEZVOUS = ThreadLocalStack() - - -@contextlib.contextmanager -def _SharedRendezvousScope(shared_rendezvous=True): - _SHARED_RENDEZVOUS.stack.append(shared_rendezvous) - try: - yield - finally: - _SHARED_RENDEZVOUS.stack.pop() - - -def _GetSharedRendezvous(): - """Get the current rendezvous sharing setting.""" - return _SHARED_RENDEZVOUS.stack[-1] if _SHARED_RENDEZVOUS.stack else False - - -def _ApplySharedRendezvous(func): - """Apply the rendezvous sharing setting on the given tf.function func.""" - # pylint: disable=protected-access - func._shared_rendezvous = _GetSharedRendezvous() - # pylint: enable=protected-access - - -def _WrapFunction(func=None, input_signature=None): - """Wraps func as a tf.function to be used in If/WhileLoop/CallDefun.""" - if input_signature is None: - input_signature = [] - - def Decorated(fn): - - @tf.function(input_signature=input_signature, autograph=False) - def Fn(*args): - # TODO(b/163904067): mimic Defun' behavior and reset the step seed to - # avoid it being used as an implicit capture. This is not a desired - # behavir, it should take the step seed from parent graph instead. - ResetStepSeed() - return fn(*args) - - _ApplySharedRendezvous(Fn) - - # Add the function to the graph so it'll be traced under the current - # context. This is necessary if the function body captures any non-tensor - # values from the environment, like symbolic maps. - cf = Fn.get_concrete_function() - cf.add_to_graph() - return cf - - # For the `foo = _WrapFunction(foo, ...)` use case. - if func is not None: - return Decorated(func) - - # For the `@_WrapFunction(...)` use case. - return Decorated - - -def _DefineFunction(fwd, - fwd_sig, - bak=None, - bak_as_function=False, - device=None): - """Wraps fwd in a defun with custom gradient bak. - - Args: - fwd: A callable xs: Nested Structure -> ys: Nested Structure. - fwd_sig: A Nested Structure of tf.Tensor representing the input signature of - fwd. - bak: A callable xs, ys, dys: Nested Structure -> dxs[, dcapture]: Nested - Structure. The custom backprop function for fwd. bak needs to return - dcapture if fwd uses any implicitly captured tensors, whose gradients are - dcapture. - bak_as_function: Whether to create a TF graph function for bak. - device: the device on which to run fwd and bak. - - Returns: - A function that wraps fwd. - """ - assert fwd is not None - noinline = not use_xla() - fwd_sig = _TensorSpecs(fwd_sig) - - # Only used to hold the output signature of fwd in sigs.rets, which will be - # set in Forward. - sigs = NestedMap() - - @_WrapFunction(input_signature=Flatten(fwd_sig)) - def Forward(*args): - """The forward function.""" - with RemoveAssertContext(remove=noinline), tf.device(device): - xs = Pack(fwd_sig, args) - rets = fwd(xs) - sigs.rets = _TensorSpecs(rets) - return Flatten(rets) - - shared_rendezvous = _GetSharedRendezvous() - implicit_captures = Forward.captured_inputs - - if not bak: - forward = Forward - else: - - def Backward(*args): - xs_len = len(Flatten(fwd_sig)) - ys_len = len(Flatten(sigs.rets)) - xs = Pack(fwd_sig, args[:xs_len]) - ys = Pack(sigs.rets, args[xs_len:(xs_len + ys_len)]) - dys = Pack(sigs.rets, args[xs_len + ys_len:]) - with RemoveAssertContext(remove=noinline), tf.device(device): - dxs = bak(xs, ys, dys) - return Flatten(dxs) - - if bak_as_function: - backward_cf = _WrapFunction( - Backward, input_signature=Flatten([fwd_sig, sigs.rets, sigs.rets])) - else: - - def BackwardWithSharedRendezvous(*args): - with _SharedRendezvousScope(shared_rendezvous): - return Backward(*args) - - backward_cf = BackwardWithSharedRendezvous - - @tf.custom_gradient - def ForwardWithGrad(*args): - """Forward function and its custom gradient.""" - # Note that `args` includes implicit_captures. This is required by - # tf.custom_gradient so that when the Grad() outputs include gradients to - # implicit captures, they match the inputs to ForwardWithGrad(). - # - # However, Forward doesn't take implicit_captures as input, so we exclude - # them here. - fwd_args = args[:(len(args) - len(Flatten(implicit_captures)))] - op = NestedMap(inputs=args, outputs=Forward(*fwd_args)) - - def Grad(*args, **kwargs): - """Gradient function for the forward function. - - Args: - *args: Gradients wrt op.outputs. - **kwargs: Additional arguments from tf.custom_gradient. - - Returns: - Tuple of derivatives. - """ - if kwargs: - tf.logging.warning( - 'Ignoring additional arguments used by tf.custom_gradient: %s', - str(kwargs)) - - _AssertInputsMatch(op, fwd_sig, implicit_captures) - - # Ensure dys contains no None. - args = ConvertNoneGradientToZeros(list(op.outputs), list(args)) - - xs, _ = Pack([fwd_sig, implicit_captures], op.inputs) - return backward_cf(*Flatten([xs, op.outputs, args])) - - return op.outputs, Grad - - forward = lambda *args: ForwardWithGrad(*Flatten([args, implicit_captures])) - - def Call(args): - """Wrapper of fwd.""" - flat_rets = forward(*Flatten(args)) - if not isinstance(flat_rets, (tuple, list)): - flat_rets = [flat_rets] - return Pack(sigs.rets, flat_rets) - - return Call - - -def CallDefun(fwd, - args, - bak=None, - bak_as_function=False, - device=None): - """Wraps fwd in a defun with custom gradient bak and calls it with args. - - Args: - fwd: A callable xs: Nested Structure -> ys: Nested Structure. - args: A Nested Structure of tf.Tensor. - bak: A callable xs, ys, dys: Nested Structure -> dxs[, dcapture]: Nested - Structure. The custom backprop function for fwd. bak needs to return - dcapture if fwd uses any implicitly captured tensors, whose gradients are - dcapture. - bak_as_function: Whether to create a TF graph function for bak. - device: the device on which to run fwd and bak. - - Returns: - A Nested Structure equivalent to what fwd(args) computes. - """ - if _FromGlobal('call_defun_use_tf_function'): - fn = _DefineFunction - else: - fn = _DefineDefun - call = fn(fwd, args, bak, bak_as_function, device) - return call(args) - - -def If(cond, inputs, then_branch, else_branch): - """Helper to construct an if/else statement. - - Args: - cond: A scalar `Tensor` that can be converted to boolean. - inputs: A flattenable representing the input tensors of the if/else - statement. - then_branch: A callable 'inputs' -> flattenable. The returned value - should be compatible with what 'else_branch' returns. - else_branch: A callable 'inputs' -> flattenable. The returned value - should be compatible with what 'then_branch' returns. - - Returns: - Output returned by the call to either 'then_branch' or 'else_branch'. - """ - ret_dtypes = NestedMap() - get_dtype = lambda x: x.dtype - - if _FromGlobal('if_use_tf_function'): - - @_WrapFunction(input_signature=Flatten(_TensorSpecs(inputs))) - def ThenBranch(*args): - inp = Pack(inputs, args) - out = then_branch(inp) - ret_dtypes.then_out = Transform(get_dtype, out) - return Flatten(out) - - @_WrapFunction(input_signature=Flatten(_TensorSpecs(inputs))) - def ElseBranch(*args): - inp = Pack(inputs, args) - out = else_branch(inp) - ret_dtypes.else_out = Transform(get_dtype, out) - return Flatten(out) - - ret = tf.If( - cond=cond, - inputs=Flatten(inputs), - then_branch=ThenBranch, - else_branch=ElseBranch) - else: - dtypes = Flatten(Transform(lambda x: x.dtype, inputs)) - - @tf.Defun(*dtypes) - def ThenBranch(*args): - for dst, src in zip(args, Flatten(inputs)): - dst.set_shape(src.shape) - inp = Pack(inputs, args) - out = then_branch(inp) - ret_dtypes.then_out = Transform(get_dtype, out) - return Flatten(out) - - @tf.Defun(*dtypes) - def ElseBranch(*args): - for dst, src in zip(args, Flatten(inputs)): - dst.set_shape(src.shape) - inp = Pack(inputs, args) - out = else_branch(inp) - ret_dtypes.else_out = Transform(get_dtype, out) - return Flatten(out) - - ret = tf.If( - cond=cond, - inputs=Flatten(inputs), - then_branch=ThenBranch, - else_branch=ElseBranch) - - assert IsCompatible(ret_dtypes.then_out, ret_dtypes.else_out), ( - 'Outputs of then_branch and else_branch are not compatible: {} vs {}' - .format(ret_dtypes.then_out, ret_dtypes.else_out)) - return Pack(ret_dtypes.then_out, ret) - - -def _Itype(): - """Loop iterator data type.""" - return tf.int32 if use_xla() else tf.int64 - - -def WhileLoop(cond, body, loop_state): - """Helper to construct a while loop. - - Args: - cond: A callable NestedMap -> tf.bool. - body: A callable NestedMap -> NestedMap. - loop_state: A flattenable (NestedMap, list, tuple, etc.) representing the - loop state. - - Returns: - The final loop state in the same structure as loop_state. - """ - state = NestedMap(loop_state=loop_state) - - if _FromGlobal('while_loop_use_tf_function'): - - @_WrapFunction(input_signature=Flatten(_TensorSpecs(state))) - def LoopCond(*args): - s = state.Pack(args) - return cond(s.loop_state) - - @_WrapFunction(input_signature=Flatten(_TensorSpecs(state))) - def LoopBody(*args): - s = state.Pack(args) - s.loop_state = body(s.loop_state) - return s.Flatten() - - new_state = tf.While(input_=state.Flatten(), cond=LoopCond, body=LoopBody) - else: - dtypes = state.Transform(lambda x: x.dtype).Flatten() - - @tf.Defun(*dtypes) - def LoopCond(*args): - s = state.Pack(args) - return cond(s.loop_state) - - @tf.Defun(*dtypes) - def LoopBody(*args): - s = state.Pack(args) - s.loop_state = body(s.loop_state) - return s.Flatten() - - new_state = tf.While(input_=state.Flatten(), cond=LoopCond, body=LoopBody) - return state.Pack(new_state).loop_state - - -def ForLoop(body, start, limit, delta, loop_state): - """Helper to construct a for loop. - - Args: - body: A callable (tf.int, NestedMap) -> NestedMap. - start: Loop variable's initial value. - limit: Loop variable's limit value. - delta: Loop variable's change per iteration. - loop_state: A flattenable (NestedMap, list, tuple, etc.) representing the - loop state. - - Returns: - The final loop state in the same structure as loop_state. - """ - state = NestedMap( - iter=tf.cast(start, _Itype()), - limit=tf.cast(limit, _Itype()), - delta=tf.cast(delta, _Itype()), - loop_state=loop_state) - - def LoopCond(state): - return tf.less(state.iter, state.limit) - - def LoopBody(state): - state.loop_state = body(state.iter, state.loop_state) - state.iter = tf.add(state.iter, state.delta) - return state - - return WhileLoop(LoopCond, LoopBody, state).loop_state - - -def TopK(x_in, k): - """Equivalent to tf.math.top_k(x_in, k) but more efficient on tpu.""" - assert k <= 2, 'This implementation is only efficient for small k.' - # TODO(yonghui): Try out an alternative idea where we first reshape x_in as a - # 2d tensor, then call tf.math.top_k, and then reshape back. - x_in_shape = x_in.shape - x_rank = x_in_shape.rank - assert x_rank and x_in_shape.as_list()[x_rank - 1] > 0 - last_dim_size = x_in_shape.as_list()[x_rank - 1] - min_value = tf.math.reduce_min(x_in) - 1.0 - - out_indices = [] - out_values = [] - - for unused_i in range(k): - index_i = tf.math.argmax(x_in, axis=-1, output_type=tf.int32) - mask_i = tf.one_hot(index_i, last_dim_size) - # TODO(yonghui): Would tf.gather be more efficient and numerically stable - # here? - value_i = tf.reduce_sum(mask_i * x_in, -1, keepdims=True) - x_in = (1.0 - mask_i) * x_in + mask_i * min_value - out_indices.append(tf.expand_dims(index_i, -1)) - out_values.append(value_i) - - if k == 1: - return out_values[0], out_indices[0] - else: - return tf.concat(out_values, x_rank - 1), tf.concat(out_indices, x_rank - 1) - - -def ReadVariable(var_op): - """Returns the value of the given variable operation. - - Args: - var_op: The variable's TF `Operation`. It could be one of VarHandleOp, - Variable and VariableV2. - - Returns: - A `Tensor` containing the value of the variable. - """ - if var_op.type == 'VarHandleOp': - # Filter out the ReadVariableOps that have control dependencies to avoid - # side-effects when the user runs it. - filter_fn = lambda op: op.type == 'ReadVariableOp' and not op.control_inputs - var_readers = list(filter(filter_fn, var_op.outputs[0].consumers())) - assert var_readers - return var_readers[0].outputs[0] - - assert var_op.type in ['Variable', 'VariableV2'] - return var_op.outputs[0] - - -_TPU_SUMMARY_TENSORS_KEY = ('__lingvo_tpu_summary_tensors') - -_get_tpu_summary_tensors = _CollectionGetter(_TPU_SUMMARY_TENSORS_KEY, - lambda: []) - - -def AddTpuSummaryTensor(name, value, weight=1.0): - """Adds tensor to global collection of summaries. - - This needs to be used in situations where tf.summary() could be used but - currently tf.summary is not supported. Use py_utils.AddTpuSummaryTensor() in - low level code to add summary tensors to global collection of summaries. - Then recover all summary tensors from global collection by calling - py_utils.GetTpuSummaryTensors() from top level code (for example from - ComputeLoss method of BaseTask). - - In addition to 'name' argument, current tensorflow name scope is also - captured and added to the metric name. This way for example summaries from - a repeated layer will appear as separate graphs in the tensorboard. - - Weight argument is optional and defaults to 1.0. See BaseTask.ComputeLoss for - the exact definition of weight for eval metrics. - - Args: - name: metric name - value: metric value tensor - weight: weight tensor for weighted metrics - """ - tpu_summary_tensors = _get_tpu_summary_tensors() - x = NestedMap() - x.name = name - x.value = value, tf.convert_to_tensor(weight) - x.name_scope = tf.get_default_graph().get_name_scope() - tpu_summary_tensors.append(x) - - -def GetTpuSummaryTensors(): - """Returns summary tensors from global collection. - - Returns: - A dict containing str keys and (metric, weight) pairs as values - """ - tpu_summary_tensors = _get_tpu_summary_tensors() - return { - '%s/%s' % (x.name, SanitizeScopeKey(x.name_scope)): x.value - for x in tpu_summary_tensors - } - - -def ClearTpuSummaryTensors(): - tpu_summary_tensors = _get_tpu_summary_tensors() - del tpu_summary_tensors[:] - - -def ComputationShape(split_size, topology=None): - """Decides the computation shape based on the split_size.""" - if topology: - topology_info = tf_topology.Topology(serialized=topology) - computation_shape = None - if split_size == 1: - # GALV: Lingvo assumes you're running on a group of TPUs that is a - # power of four. We are running on 8 TPUv3's, so we reduce the - # shape to length 3. - computation_shape = [1, 1, 1] - elif split_size == 2: - computation_shape = [1, 1, 1, 2] - elif split_size == 4: - computation_shape = [1, 2, 1, 2] - elif split_size == 8: - computation_shape = [2, 2, 1, 2] - elif split_size == 16: - computation_shape = [4, 2, 1, 2] - elif split_size == 32: - computation_shape = [4, 4, 1, 2] - elif split_size == 64: - if topology and topology_info.mesh_shape[1] == 32: - # Fwd within-replica all-reduces is performed along column; - # Bwd gradient cross-replica all-reduces is performed along row. - # This currently has better performance than the strided patten. - computation_shape = [1, 32, 1, 2] - else: - computation_shape = [4, 8, 1, 2] - elif split_size == 128: - computation_shape = [8, 8, 1, 2] - elif split_size == 256: - computation_shape = [8, 16, 1, 2] - elif split_size == 512: - computation_shape = [16, 16, 1, 2] - elif split_size == 1024: - computation_shape = [16, 32, 1, 2] - elif split_size == 2048: - computation_shape = [32, 32, 1, 2] - elif split_size == 4096: - computation_shape = [64, 32, 1, 2] - elif split_size == 8192: - computation_shape = [128, 32, 1, 2] - else: - assert False, ('Model parallelism with %d devices is currently not' - ' supported.' % split_size) - assert computation_shape is not None - return computation_shape - - -def GetExtraVars(): - """Returns the captured variables by the function.""" - g = tf.get_default_graph() - if isinstance(g, func_graph.FuncGraph): - return g.variable_captures - return function.get_extra_vars() - - -def GetExtraInputs(): - """Returns the captured input tensors by the function.""" - g = tf.get_default_graph() - if isinstance(g, func_graph.FuncGraph): - return g.external_captures - return function.get_extra_inputs() - - -def GetExtraArgs(): - """Returns the corresponding function arguments for the captured inputs.""" - g = tf.get_default_graph() - if isinstance(g, func_graph.FuncGraph): - return g.internal_captures - return function.get_extra_args() - -def LengthsFromBitMask(padding_bitmask, time_axis: int): - # T, B - assert padding_bitmask.dtype == tf.float32, \ - "This assert is not necessary, but I would like to know when the condition isn't true." - lengths = tf.cast(tf.reduce_sum(1.0 - padding_bitmask, axis=time_axis), tf.int32) - return lengths - -def BitMaskFromLengths(padding_lengths): - max_length = tf.shape(padding_lengths)[0] - padding = 1.0 - tf.sequence_mask(padding_lengths, max_length, tf.float32) - return padding - -def ShardedFilePatternToGlob(file_pattern): - """Converts a file pattern path@shards to path-?????-of-shards.""" - if ',' in file_pattern: - raise ValueError( - 'ShardedFilePatternToGlob does not support multiple file patterns.') - if '@' not in file_pattern: - return file_pattern - path, shards = file_pattern.split('@') - if shards == '*': - return f'{path}-?????-of-*' - return f'{path}-?????-of-{int(shards):05}' diff --git a/lingvo/core/py_utils_test.py b/lingvo/core/py_utils_test.py deleted file mode 100644 index cffd5c3c..00000000 --- a/lingvo/core/py_utils_test.py +++ /dev/null @@ -1,3340 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for py_utils.""" - -import collections -import copy -import itertools -import os -import sys - -from absl.testing import flagsaver -from absl.testing import parameterized -from lingvo import model_registry -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import builder_layers -from lingvo.core import cluster_factory -from lingvo.core import hyperparams -from lingvo.core import py_utils -from lingvo.core import recurrent -from lingvo.core import symbolic -from lingvo.core import test_helper -from lingvo.core import test_utils -from lingvo.tasks.image.params import mnist # pylint: disable=unused-import - -import mock -import numpy as np - -from tensorflow.python.ops import functional_ops # pylint:disable=g-direct-tensorflow-import - -FLAGS = tf.flags.FLAGS - - -class PyUtilsTest(test_utils.TestCase, parameterized.TestCase): - - def testIsDefaultParamInit(self): - p = py_utils.DefaultParamInit() - self.assertTrue(py_utils.IsDefaultParamInit(p)) - p = hyperparams.Params.FromProto(p.ToProto()) - self.assertTrue(py_utils.IsDefaultParamInit(p)) - p = py_utils.WeightInit.Xavier(scale=1.) - self.assertFalse(py_utils.IsDefaultParamInit(p)) - - def testNestedMapFromNestedDict(self): - a = {'a1': 1, 'a2': 2} - b = {'b1': 1, 'b2': 2} - c = {'a': a, 'b': b, 'ab': [a, b]} - d = py_utils.NestedMap(c) - e = py_utils.NestedMap.FromNestedDict(c) - self.assertIsInstance(d, py_utils.NestedMap) - self.assertNotIsInstance(d.a, py_utils.NestedMap) - self.assertIsInstance(e.a, py_utils.NestedMap) - self.assertIsInstance(e.ab[0], py_utils.NestedMap) - self.assertEqual(e.ab[0], e.a) - self.assertEqual(e.ab[1], e.b) - - def testNestedMapGetItem(self): - nested_map = py_utils.NestedMap() - nested_map['a'] = py_utils.NestedMap({'x': 3}) - nested_map['b'] = py_utils.NestedMap({'y_0': 4, 'y': [3]}) - self.assertEqual(nested_map.GetItem('a.x'), nested_map.a.x) - self.assertEqual(nested_map.GetItem('b.y_0'), 4) - with self.assertRaises(KeyError): - nested_map.GetItem('o') - - def testNestedMapGet(self): - nested_map = py_utils.NestedMap({'a': {'b': 0}}) - self.assertEqual(nested_map.Get('a.b'), 0) - self.assertIsNone(nested_map.Get('a.b.c')) - self.assertIsNone(nested_map.Get('x')) - self.assertEqual(nested_map.Get('x', 0), 0) - - def testNestedMapDir(self): - nested_map = py_utils.NestedMap({'a': {'b': 0}}) - nested_map.c = '1' - nested_map_dir = dir(nested_map) - # Flattened keys are in the dir - self.assertIn('a.b', nested_map_dir) - self.assertIn('c', nested_map_dir) - # So are method APIs. - self.assertIn('Get', nested_map_dir) - - def testNestedMapSet(self): - nested_map = py_utils.NestedMap.FromNestedDict({'a': {'b': 0}}) - self.assertEqual(nested_map.a.b, 0) - # Test if overriding an existing value works. - nested_map.Set('a.b', 1) - self.assertEqual(nested_map.a.b, 1) - # Test if ValueError is raised if an existing intermediate value is not a - # NestedMap. - with self.assertRaises(ValueError): - nested_map.Set('a.b.c', 2) - nested_map.Set('a.b', py_utils.NestedMap()) - # Verify that non-existing intermediate keys are set to NestedMap. - nested_map.Set('a.b.x.y', 2) - self.assertIsInstance(nested_map.a.b.x, py_utils.NestedMap) - self.assertEqual(nested_map.a.b.x.y, 2) - - @parameterized.named_parameters( - ('_stateful', False, [tf.float32, tf.float64, tf.complex64]), - ('_stateless', True, [tf.float32, tf.float64])) - def testCreateVariableBasics(self, stateless_vars_init, dtypes): - with flagsaver.flagsaver(stateless_vars_init=stateless_vars_init): - with self.session(use_gpu=False, graph=tf.Graph()): - methods = [ - py_utils.WeightInit.Gaussian, - py_utils.WeightInit.Uniform, - py_utils.WeightInit.Constant, - py_utils.WeightInit.TruncatedGaussian, - py_utils.WeightInit.GaussianSqrtDim, - py_utils.WeightInit.UniformSqrtDim, - py_utils.WeightInit.UniformUnitScaling, - py_utils.WeightInit.UniformUnitScalingFanAvg, - py_utils.WeightInit.TruncatedGaussianSqrtDim, - py_utils.WeightInit.TruncatedGaussianSqrtFanIn, - py_utils.WeightInit.TruncatedGaussianSqrtFanOut, - py_utils.WeightInit.XavierGaussian, - ] - shapes = [[], [3], [2, 4], [3, 3, 2, 4]] - col = ['col1', 'col2'] - - all_vars = [] - for i, (m, dt, - sp) in enumerate(itertools.product(methods, dtypes, shapes)): - pc = py_utils.WeightParams(sp, m(), dt, col) - all_vars.append(py_utils.CreateVariable('var_%d' % i, pc)) - - # To reuse existing variables - tf.get_variable_scope().reuse_variables() - - self.assertEqual(len(tf.trainable_variables()), len(all_vars)) - - all_vars_copy = [] - for i, (m, dt, - sp) in enumerate(itertools.product(methods, dtypes, shapes)): - pc = py_utils.WeightParams(sp, m(), dt, col) - all_vars_copy.append(py_utils.CreateVariable('var_%d' % i, pc)) - - self.evaluate(tf.global_variables_initializer()) - for v1, v2 in zip(all_vars, all_vars_copy): - v1_v = v1.eval() - v2_v = v2.eval() - self.assertAllEqual(v1_v, v2_v) - - def testCreateVariableWithSymbols(self): - with self.session(use_gpu=False, graph=tf.Graph()): - dim_symbol = symbolic.Symbol('dim') - shape = [2, 3, dim_symbol * 2] - - with symbolic.SymbolToValueMap(symbolic.STATIC_VALUES, {dim_symbol: 2}): - pc = py_utils.WeightParams(shape, py_utils.WeightInit.Gaussian(), - tf.float32, ['col1', 'col2']) - var = py_utils.CreateVariable('var', pc) - - # To reuse existing variables - tf.get_variable_scope().reuse_variables() - - new_dim_symbol = symbolic.Symbol('new_dim') - # Same shape as above but from different symbol. - shape = [2, 3, new_dim_symbol * 2] - with symbolic.SymbolToValueMap(symbolic.STATIC_VALUES, - {new_dim_symbol: 2}): - pc = py_utils.WeightParams(shape, py_utils.WeightInit.Gaussian(), - tf.float32, ['col1', 'col2']) - var_copy = py_utils.CreateVariable('var', pc) - - self.evaluate(tf.global_variables_initializer()) - self.assertAllEqual(var.eval(), var_copy.eval()) - - def testCreateVariableUniform(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(12345678) - methods = [ - py_utils.WeightInit.Uniform, - py_utils.WeightInit.UniformSqrtDim, - py_utils.WeightInit.UniformUnitScaling, - ] - dtypes = [tf.float32, tf.complex64] - shapes = [[2, 3]] - all_vars = [] - for i, (dt, m, - sp) in enumerate(itertools.product(dtypes, methods, shapes)): - pc = py_utils.WeightParams(sp, m(0.1), dt) - all_vars.append(py_utils.CreateVariable('var_%d' % i, pc)) - - v1_v_expted = [[0.069674, -0.072278, -0.021777], - [-0.052155, -0.050274, 0.086218]] - v2_v_expted = [[0.005361, 0.036109, -0.036575], - [0.058314, 0.031438, 0.049196]] - v4_v_expted = [ - [0.015448 + 0.068295j, -0.098710 - 0.054435j, 0.037030 - 0.048017j], - [-0.047435 + 0.035301j, 0.041994 + 0.000279j, -0.029097 + 0.084902j], - ] - - self.evaluate(tf.global_variables_initializer()) - v1_v = all_vars[0].eval() - v2_v = all_vars[1].eval() - v4_v = all_vars[3].eval() - self.assertAllClose(v1_v_expted, v1_v.tolist()) - self.assertAllClose(v2_v_expted, v2_v.tolist()) - self.assertAllClose(v4_v_expted, v4_v.tolist()) - - def testCreateVariableNormal(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(832124) - methods = [ - py_utils.WeightInit.Gaussian, - py_utils.WeightInit.GaussianSqrtDim, - ] - dtypes = [tf.float32, tf.complex64] - shapes = [[2, 3]] - all_vars = [] - for i, (dt, m, - sp) in enumerate(itertools.product(dtypes, methods, shapes)): - pc = py_utils.WeightParams(sp, m(), dt) - all_vars.append(py_utils.CreateVariable('var_%d' % i, pc)) - - v1_v_expted = [[-1.472208, 0.960204, -0.192588], - [-0.461884, 1.018134, 0.063719]] - v2_v_expted = [[-0.862255, -0.688153, 0.82515], - [-0.07671, 0.613031, -0.020327]] - v3_v_expted = [ - [1.005469 + 0.827639j, 1.249896 + 0.802671j, -0.026286 - 0.813836j], - [0.865386 + 0.301172j, 0.876698 - 0.907293j, 1.996337 + 1.840192j], - ] - - self.evaluate(tf.global_variables_initializer()) - v1_v = all_vars[0].eval() - v2_v = all_vars[1].eval() - v3_v = all_vars[2].eval() - self.assertAllClose(v1_v_expted, v1_v.tolist()) - self.assertAllClose(v2_v_expted, v2_v.tolist()) - self.assertAllClose(v3_v_expted, v3_v.tolist()) - - def testCreateVariableSqrtFanInOut(self): - with self.session(): - tf.random.set_seed(832124) - methods = [ - py_utils.WeightInit.GaussianSqrtFanIn, - py_utils.WeightInit.TruncatedGaussianSqrtFanIn, - py_utils.WeightInit.GaussianSqrtFanOut, - py_utils.WeightInit.TruncatedGaussianSqrtFanOut, - py_utils.WeightInit.XavierGaussian, - ] - dtypes = [tf.float32] - shapes = [[1, 1, 2, 3]] - all_vars = [] - for i, (dt, m, - sp) in enumerate(itertools.product(dtypes, methods, shapes)): - pc = py_utils.WeightParams(sp, m(scale=2), dt) - all_vars.append(py_utils.CreateVariable('var_%d' % i, pc)) - - self.evaluate(tf.global_variables_initializer()) - var_values = self.evaluate(all_vars) - tf.logging.info('var_values=%s', var_values) - self.assertAllClose( - [ - # GaussianSqrtFanIn. - [[[[-2.08201575, 1.35793388, -0.27236053], - [-0.65320235, 1.43985856, 0.09011276]]]], - # TruncatedGaussianSqrtFanIn. - [[[[-1.72450912, -1.37630582, 1.65029943], - [-0.15342039, -0.7636584, -0.97026265]]]], - # GaussianSqrtFanOut. - [[[[1.16101539, 1.4432559, -0.03035267], - [0.9992612, 1.01232362, 2.30517101]]]], - # TruncatedGaussianSqrtFanOut. - [[[[-0.049076, -0.25183302, -1.79192507], - [0.93166995, -0.83121753, -1.40264213]]]], - # XavierGaussian. - [[[[-0.59545106, 1.232773, -0.2630677], - [0.11635535, 0.9317614, 0.16670291]]]], - ], - var_values) - - def testCreateVariableException(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(832124) - pc = py_utils.WeightParams([2, 3], py_utils.WeightInit.Gaussian()) - var1 = py_utils.CreateVariable('var1', pc) - - tf.get_variable_scope().reuse_variables() - # Reuses an existing variable. - var2 = py_utils.CreateVariable('var1', pc) - - # An exception should be thrown in this case. - pc = py_utils.WeightParams([2, 3], py_utils.WeightInit.Gaussian(2.0)) - with self.assertRaises(AssertionError): - py_utils.CreateVariable('var1', pc) - - self.evaluate(tf.global_variables_initializer()) - self.assertAllEqual(var1.eval(), var2.eval()) - - def testCreateVariableDifferentSeed(self): - with self.session(use_gpu=False): - tf.random.set_seed(3251343) - pc = py_utils.WeightParams([2, 3], py_utils.WeightInit.Gaussian()) - with tf.variable_scope('layer0'): - w0 = py_utils.CreateVariable('w', pc) - with tf.variable_scope('layer1'): - w1 = py_utils.CreateVariable('w', pc) - self.evaluate(tf.global_variables_initializer()) - - # w0_val, w1_val should be sufficient different. - w0_val, w1_val = self.evaluate([w0, w1]) - print(['diff = ', w0_val - w1_val]) - self.assertGreater(np.max(np.abs(w0_val - w1_val)), 0.1) - - def testXavier(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(1618) - methods = [py_utils.WeightInit.Xavier] - dtypes = [tf.float32, tf.float16, tf.complex64] - shapes = [[2, 3]] - all_vars = [] - for i, (m, dt, - sp) in enumerate(itertools.product(methods, dtypes, shapes)): - pc = py_utils.WeightParams(sp, m(), dt) - all_vars.append(py_utils.CreateVariable('var_%d' % i, pc)) - - v1_v_expted = [[1.051236, -0.959198, 0.796091], - [-0.685691, 0.230933, -1.006293]] - v3_v_expted = [ - [0.149996 - 0.064369j, 0.689145 + 0.017257j, -0.502070 - 0.367683j], - [0.519782 + 0.470412j, 0.738902 - 0.054006j, 0.028603 + 0.471832j], - ] - - self.evaluate(tf.global_variables_initializer()) - v1_v = all_vars[0].eval() - v3_v = all_vars[2].eval() - self.assertAllClose(v1_v_expted, v1_v.tolist()) - self.assertAllClose(v3_v_expted, v3_v.tolist()) - - def testXavier1D(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(1618) - methods = [py_utils.WeightInit.Xavier] - dtypes = [tf.float32, tf.float16, tf.complex64] - shapes = [[2]] - all_vars = [] - for i, (m, dt, - sp) in enumerate(itertools.product(methods, dtypes, shapes)): - pc = py_utils.WeightParams(sp, m(), dt) - all_vars.append(py_utils.CreateVariable('var_%d' % i, pc)) - - v1_v_expted = [1.175317, -1.072416] - - self.evaluate(tf.global_variables_initializer()) - v1_v = all_vars[0].eval() - self.assertAllClose(v1_v_expted, v1_v.tolist()) - - def testXavier3D(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(1618) - methods = [py_utils.WeightInit.Xavier] - dtypes = [tf.float32, tf.float16, tf.complex64] - shapes = [[1, 1, 2]] - all_vars = [] - for i, (m, dt, - sp) in enumerate(itertools.product(methods, dtypes, shapes)): - pc = py_utils.WeightParams(sp, m(), dt) - all_vars.append(py_utils.CreateVariable('var_%d' % i, pc)) - - v1_v_expted = [[[1.357139, -1.23832]]] - - self.evaluate(tf.global_variables_initializer()) - v1_v = all_vars[0].eval() - self.assertAllClose(v1_v_expted, v1_v.tolist()) - - def testVariableShapePrefix(self): - with self.session(use_gpu=False, graph=tf.Graph()): - shape = [3, 2] - pc = py_utils.WeightParams( - shape=shape, init=py_utils.WeightInit.Constant(0.0), dtype=tf.float32) - with py_utils.VariableShapePrefixContext(5): - with py_utils.VariableShapePrefixContext(4): - var = py_utils.CreateVariable('var', pc) - self.assertEqual([5, 4, 3, 2], var.shape.as_list()) - - def testGeoMeanXavier(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(1618) - methods = [py_utils.WeightInit.GeoMeanXavier] - dtypes = [tf.float32, tf.float16, tf.complex64] - shapes = [[2, 3]] - all_vars = [] - for i, (m, dt, - sp) in enumerate(itertools.product(methods, dtypes, shapes)): - pc = py_utils.WeightParams(sp, m(), dt) - all_vars.append(py_utils.CreateVariable('var_%d' % i, pc)) - - v1_v_expted = [[1.062019, -0.969037, 0.804257], - [-0.692724, 0.233301, -1.016615]] - v3_v_expted = [[ - 0.151534 - 0.065029j, 0.696214 + 0.017434j, -0.507220 - 0.371455j - ], [0.525114 + 0.475238j, 0.746481 - 0.05456j, 0.028896 + 0.476672j]] - - self.evaluate(tf.global_variables_initializer()) - v1_v = all_vars[0].eval() - v3_v = all_vars[2].eval() - self.assertAllClose(v1_v_expted, v1_v.tolist()) - self.assertAllClose(v3_v_expted, v3_v.tolist()) - - def testCheckNumerics(self): - xv = [[1, 2], [3, 4]] - yv = [10] * 4 - with self.session(): - x = tf.constant(xv, tf.float32) - y = tf.constant(yv) - z = tf.reduce_mean(tf.constant([], tf.float32)) - self.assertAllClose(xv, self.evaluate(py_utils.CheckNumerics(x))) - self.assertAllClose(yv, self.evaluate(py_utils.CheckNumerics(y))) - actual_xv, actual_yv = self.evaluate(py_utils.CheckNumerics([x, y])) - self.assertAllClose(xv, actual_xv) - self.assertAllClose(yv, actual_yv) - actual_xv, actual_yv = self.evaluate(py_utils.CheckNumerics((x, y))) - self.assertAllClose(xv, actual_xv) - self.assertAllClose(yv, actual_yv) - - with self.assertRaisesRegex(tf.errors.InvalidArgumentError, 'NaN'): - self.evaluate(py_utils.CheckNumerics(z)) - - def testLog(self): - with self.session(): - x = tf.constant([[1, 2], [3, 4]]) - y = tf.constant([10] * 4) - x = py_utils.Log(x, 'testLog', x=x, y=y) - self.assertAllEqual(x.eval(), [[1, 2], [3, 4]]) - - def testDebug(self): - with self.session(): - x = tf.constant([[1, 2], [3, 4]]) - y = tf.constant([11] * 4) - z = tf.constant([22] * 4) - x = py_utils.Debug(x, 'msg') - self.assertAllEqual(x.eval(), [[1, 2], [3, 4]]) - - x = py_utils.Debug(x, 'msg', more=[y, z]) - self.assertAllEqual(x.eval(), [[1, 2], [3, 4]]) - - def testSave(self): - with self.session(): - x = tf.constant([[1, 2], [3, 4]]) - y = tf.constant([10] * 4) - x = py_utils.Save(x, '%s/test' % self.get_temp_dir(), x=x, y=y) - self.evaluate(tf.global_variables_initializer()) - self.assertAllEqual(self.evaluate(x), [[1, 2], [3, 4]]) - - # Reads npy files and check the values. - read_x = np.load('%s/test.%08d.x.npy' % (self.get_temp_dir(), 0)) - read_y = np.load('%s/test.%08d.y.npy' % (self.get_temp_dir(), 0)) - self.assertAllEqual(read_x, [[1, 2], [3, 4]]) - self.assertAllEqual(read_y, [10] * 4) - - def testTensorRank(self): - a = tf.constant([1]) - self.assertIsInstance(py_utils.HasRank(a, 1), tf.Tensor) - self.assertIsInstance(py_utils.HasAtLeastRank(a, 1), tf.Tensor) - - b = tf.constant([[1, 2]]) - self.assertIsInstance(py_utils.HasRank(b, 2), tf.Tensor) - self.assertIsInstance(py_utils.HasAtLeastRank(b, 1), tf.Tensor) - self.assertIsInstance(py_utils.HasAtLeastRank(b, 2), tf.Tensor) - with self.assertRaises(Exception): - py_utils.HasAtLeastRank(b, 3) - - c = tf.placeholder(tf.int32, shape=None) - d = py_utils.HasAtLeastRank(c, 3) - with self.session() as sess: - d_v = sess.run(d, feed_dict={c: np.array([[[1, 2]]])}) - self.assertAllEqual([[[1, 2]]], d_v) - with self.assertRaises(Exception): - sess.run(d, feed_dict={c: np.array([[1, 2]])}) - - def testTensorRankDisableAsserts(self): - with flagsaver.flagsaver(enable_asserts=False): - c = tf.placeholder(tf.int32, shape=None) - d = py_utils.HasAtLeastRank(c, 3) - with self.session() as sess: - d_v = sess.run(d, feed_dict={c: np.array([[1, 2]])}) - self.assertAllEqual([[1, 2]], d_v) - - def testGetShape(self): - a = tf.constant([1]) - self.assertEqual(py_utils.GetShape(a), [1]) - self.assertEqual(py_utils.GetShape(a, 1), [1]) - self.assertEqual(py_utils.GetShape(a, 3), [1]) - - b = tf.constant([[1, 2]]) - self.assertEqual(py_utils.GetShape(b), [1, 2]) - self.assertEqual(py_utils.GetShape(b, 1), [1]) - self.assertEqual(py_utils.GetShape(b, 2), [1, 2]) - self.assertEqual(py_utils.GetShape(b, 3), [1, 2]) - - c = tf.zeros([1, a[0], a.shape[0], tf.shape(a)[0]]) - self.assertEqual(py_utils.GetShape(c)[0], 1) - self.assertEqual(py_utils.GetShape(c)[1], 1) - self.assertEqual(py_utils.GetShape(c)[2], 1) - self.assertEqual(py_utils.GetShape(c)[3], 1) - - d = tf.placeholder(tf.float32, shape=(1, None)) - self.assertEqual(py_utils.GetShape(d)[0], 1) - self.assertIsInstance(py_utils.GetShape(d)[1], tf.Tensor) - - e = tf.zeros([d.shape[0], tf.shape(d)[0], tf.shape(d)[1]]) - self.assertEqual(py_utils.GetShape(e)[0], 1) - self.assertIsInstance(py_utils.GetShape(e)[1], tf.Tensor) - self.assertIsInstance(py_utils.GetShape(e)[2], tf.Tensor) - - f = tf.placeholder(tf.float32) - self.assertIsNone(f.shape.ndims) - # GetShape() will return a Tensor. - self.assertIsInstance(py_utils.GetShape(f), tf.Tensor) - - def testGetSize(self): - a = tf.constant([1]) - self.assertEqual(py_utils.GetSize(a), 1) - - b = tf.constant([[1, 2]]) - self.assertEqual(py_utils.GetSize(b), 2) - - d = tf.placeholder(tf.float32, shape=(1, None)) - self.assertIsInstance(py_utils.GetSize(d), tf.Tensor) - - shape = tf.placeholder(tf.int32) - f = py_utils.GetSize(tf.reshape(d, shape)) - self.assertIsInstance(f, tf.Tensor) - - with self.session() as sess: - f_v = sess.run(f, feed_dict={d: np.array([[1, 2]]), shape: [2]}) - self.assertEqual(2, f_v) - - def testUpdateFpropDtype(self): - network_p = builder_layers.SequentialLayer.Params() - linear_layer_p = builder_layers.LinearLayer.Params() - linear_layer_p.input_dims = 5 - linear_layer_p.output_dims = 6 - network_p.sub.append(linear_layer_p) - - py_utils.UpdateFpropDtype(network_p, tf.bfloat16) - self.assertEqual(network_p.sub[0].fprop_dtype, tf.bfloat16) - - def testUpdateDtype(self): - network_p = builder_layers.SequentialLayer.Params() - linear_layer_p = builder_layers.LinearLayer.Params() - linear_layer_p.input_dims = 5 - linear_layer_p.output_dims = 6 - network_p.sub.append(linear_layer_p) - - py_utils.UpdateDtype(network_p, tf.bfloat16) - self.assertEqual(network_p.sub[0].dtype, tf.bfloat16) - - def testGetRank(self): - a = tf.constant([1]) - self.assertEqual(py_utils.GetRank(a), 1) - - b = tf.constant([[1, 2]]) - self.assertEqual(py_utils.GetRank(b), 2) - - c = tf.zeros([1, a[0], a.shape[0], tf.shape(a)[0]]) - self.assertEqual(py_utils.GetRank(c), 4) - - d = tf.placeholder(tf.float32, shape=(1, None)) - self.assertEqual(py_utils.GetRank(d), 2) - - e = tf.zeros([d.shape[0], tf.shape(d)[0], tf.shape(d)[1]]) - self.assertEqual(py_utils.GetRank(e), 3) - - f = tf.placeholder(tf.float32) - self.assertIsNone(f.shape.ndims) - # GetRank() will return a Tensor. - self.assertIsInstance(py_utils.GetRank(f), tf.Tensor) - - def testRenamingRules(self): - pc = py_utils.WeightParams([3, 3]) - with tf.variable_scope('model'): - v1 = py_utils.CreateVariable('v1', pc) - with py_utils.VariableRenameScope([('model/(.*)', 'data/%s')]): - v2 = py_utils.CreateVariable('v2', pc) - v3 = py_utils.CreateVariable('v3', pc) - - self.assertEqual(v1.name, 'model/v1/var:0') - self.assertEqual(v2.name, 'data/v2/var:0') - self.assertEqual(v3.name, 'model/v3/var:0') - - def testOpportunisticReuse(self): - pc = py_utils.WeightParams([3, 3]) - v1 = py_utils.CreateVariable('v1', pc) - with self.assertRaises(Exception): - py_utils.CreateVariable('v1', pc) - with py_utils.OpportunisticVariableReuseScope(True): - v2 = py_utils.CreateVariable('v1', pc) - x1 = py_utils.CreateVariable('x1', pc) - with py_utils.OpportunisticVariableReuseScope(False): - with self.assertRaises(Exception): - py_utils.CreateVariable('v1', pc) - v3 = py_utils.CreateVariable('v1', pc) - with self.assertRaises(Exception): - py_utils.CreateVariable('v1', pc) - - for v in [v2, v3]: - self.assertIs(v1, v) - self.assertIsNot(v1, x1) - - def testGetOrCreateGlobalStepVar(self): - with tf.variable_scope('s1'): - with tf.name_scope('s2'): - gs1 = py_utils.GetOrCreateGlobalStepVar() - gs2 = tf.train.get_global_step() - gs3 = py_utils.GetOrCreateGlobalStepVar() - gs4 = tf.train.get_global_step() - gs5 = py_utils.GetOrCreateGlobalStepVar() - gs6 = tf.train.get_global_step() - for gs in [gs2, gs3, gs4, gs5, gs6]: - self.assertIs(gs1, gs) - self.assertEqual(gs1.name, 'global_step:0') - - def testCreateLocalTheta(self): - methods = [py_utils.WeightInit.Gaussian, py_utils.WeightInit.Uniform] - dtypes = [tf.float32, tf.complex64] - shapes = [[2, 4], [3]] - - test_vars = py_utils.NestedMap() - for i, (m, dt, sp) in enumerate(itertools.product(methods, dtypes, shapes)): - pc = py_utils.WeightParams(sp, m(), dt, 'col1') - var = py_utils.CreateVariable('var_%d' % i, pc) - with tf.device(var.device): - test_vars['var_%d' % i] = tf.identity(var) - - test_devices = [ - '/job:worker/replica:0/device:GPU:0', - '/job:worker/replica:0/device:GPU:1' - ] - - sharded_local_vars = py_utils.CreateLocalTheta(test_vars, test_devices) - sharded_local_vars_list = sharded_local_vars.Flatten() - - # assert proper device placement - for i, v in enumerate(sharded_local_vars_list): - expected_device = test_devices[i % len(test_devices)] - self.assertEqual(v.device, expected_device) - - def testComputeGradient(self): - with self.session(use_gpu=False): - a = tf.get_variable('a', []) - b = tf.get_variable('b', [], trainable=False) - c = tf.get_variable('c', []) - e = tf.get_variable('e', []) - l = a + b + tf.stop_gradient(c) - vmap = py_utils.NestedMap( - a=a, b=b, c=c, d=None, n=py_utils.NestedMap(aa=a, e=e)) - var_grads = py_utils.ComputeGradients(l, vmap) - print('var_grads = ', var_grads.DebugString()) - # Only 'a' matters. b is not trainable; c has stop_gradient; d - # is None; e is not computed by l and aa is a duplicated. - self.assertEqual([_[0] for _ in var_grads.FlattenItems()], ['a']) - self.assertEqual(var_grads.a.var.name, 'a:0') - - def testVarGradNestFlatten(self): - a = tf.get_variable('a', []) - b = tf.get_variable('b', []) - vs_gs = py_utils.NestedMap( - a=py_utils.VarGrad(a, - tf.ones_like(a) * 10.0), - b=py_utils.VarGrad(b, - tf.ones_like(b) * 0.5)) - flattened = tf.nest.flatten(vs_gs) - self.assertLen(flattened, 2) - for x in flattened: - self.assertIsInstance(x, py_utils.VarGrad) - - def testClipSingleTensorGradients(self): - a = tf.get_variable('a', []) - b = tf.get_variable('b', []) - vs_gs = py_utils.NestedMap( - a=py_utils.VarGrad(a, - tf.ones_like(a) * 10.0), - b=py_utils.VarGrad(b, - tf.ones_like(b) * 0.5)) - clipped = py_utils.ApplyGradNormClipping(vs_gs, norm=1.0) - with self.session(use_gpu=False): - self.evaluate(tf.global_variables_initializer()) - clipped_np = self.evaluate(clipped.Transform(tuple)) - # Each variable is clipped indipendently to grad scale of 1. - self.assertAllClose(clipped_np.a[1], 1.0) - self.assertAllClose(clipped_np.b[1], 0.5) - - def testMaskGradient(self): - with self.session(use_gpu=False): - a = tf.get_variable('a', []) - b = tf.get_variable('b', []) - c = tf.get_variable('c', []) - d = tf.get_variable('d', []) - e = tf.get_variable('e', []) - l = a + b + c + d - zeros = tf.zeros(3, dtype=tf.float32) - select = tf.one_hot(1, 3, dtype=tf.float32) - vmap = py_utils.NestedMap( - a=a, b=b, c=c, d=d, n=py_utils.NestedMap(aa=a, e=e)) - grad_mask = py_utils.NestedMap() - grad_mask['a:0'] = zeros - grad_mask['b:0'] = zeros - grad_mask['c:0'] = select - grad_mask['d:0'] = select - grad_onehot = tf.one_hot(1, 3, dtype=tf.float32) - grad_mask = { - k: tf.tensordot(v, grad_onehot, 1) for k, v in grad_mask.items() - } - var_grads = py_utils.ComputeGradients(l, vmap) - var_grads_mask = py_utils.MaskGradients(var_grads, grad_mask) - self.evaluate(tf.global_variables_initializer()) - _, var_grads_mask_vals = self.evaluate( - [var_grads.Transform(tuple), - var_grads_mask.Transform(tuple)]) - # 'a' and 'b' are masked, while 'c' and 'd' are not. - self.assertEqual(var_grads_mask_vals['a'][1], 0) - self.assertEqual(var_grads_mask_vals['b'][1], 0) - self.assertEqual(var_grads_mask_vals['c'][1], 1) - self.assertEqual(var_grads_mask_vals['d'][1], 1) - - def testSkipL2Regularization(self): - with self.session(use_gpu=False): - beta = tf.get_variable( - 'beta', - initializer=tf.constant(np.arange(10).reshape([1, 10]), tf.float32)) - tf.add_to_collection(py_utils.SKIP_LP_REGULARIZATION, beta) - gamma = tf.get_variable( - 'gamma', - initializer=tf.constant(np.arange(10).reshape([1, 10]), tf.float32)) - act = tf.constant(np.arange(10).reshape([1, 10]), tf.float32) - pred = act * gamma + beta - loss = tf.reduce_sum(pred) - vmap = py_utils.NestedMap(beta=beta, gamma=gamma) - var_grads = py_utils.ComputeGradients(loss, vmap) - self.assertCountEqual(var_grads.keys(), ['beta', 'gamma']) - l2_loss, var_grads_with_l2 = py_utils.AdjustGradientsWithLpLoss( - var_grads, 0.1, p=2.0) - - self.evaluate(tf.global_variables_initializer()) - var_grads_vals, l2_loss_val, var_grads_with_l2_vals = self.evaluate([ - var_grads.Transform(tuple), l2_loss, - var_grads_with_l2.Transform(tuple) - ]) - print('var_grads_vals = ', var_grads_vals) - print('var_grads_with_l2_vals = ', var_grads_with_l2_vals) - self.assertAllEqual(var_grads_vals.beta[0], - var_grads_with_l2_vals.beta[0]) - self.assertAllEqual(var_grads_vals.gamma[0], - var_grads_with_l2_vals.gamma[0]) - self.assertAllEqual( - l2_loss_val, 0.5 * 0.1 * np.sum(np.square(var_grads_vals.gamma[0]))) - - # With l2, gradients of be gamma are adjusted. - self.assertAllClose( - var_grads_with_l2_vals.gamma[1], - var_grads_vals.gamma[1] + 0.1 * var_grads_vals.gamma[0]) - self.assertAllClose(var_grads_with_l2_vals.beta[1], - var_grads_vals.beta[1]) - - def testAdjustGradientsWithL2Loss(self): - with self.session(use_gpu=False): - emb = tf.get_variable( - 'emb', - initializer=tf.constant(np.arange(100).reshape([10, 10]), tf.float32)) - act = tf.gather(emb, [2, 5, 2, 2, 5]) - weight = tf.get_variable( - 'w', initializer=tf.constant(np.ones([10, 1]), tf.float32)) - bias = tf.get_variable('b', initializer=tf.constant([0.217])) - pred = tf.matmul(act, weight) + tf.stop_gradient(bias) - loss = tf.reduce_sum(pred) - vmap = py_utils.NestedMap(emb=emb, weight=weight, bias=bias) - var_grads = py_utils.ComputeGradients(loss, vmap) - self.assertCountEqual(var_grads.keys(), ['emb', 'weight']) - for mode in ('NestedMap', 'list'): - if mode == 'NestedMap': - l2_loss, var_grads_with_l2 = py_utils.AdjustGradientsWithLpLoss( - var_grads, 0.1, p=2.0) - else: - l2_loss, var_grads_with_l2 = py_utils.AdjustGradientsWithLpLoss( - var_grads.Flatten(), 0.1, p=2.0) - var_grads_with_l2 = py_utils.Pack(var_grads, var_grads_with_l2) - - self.evaluate(tf.global_variables_initializer()) - var_grads_vals, l2_loss_val, var_grads_with_l2_vals = self.evaluate([ - var_grads.Transform(tuple), l2_loss, - var_grads_with_l2.Transform(tuple) - ]) - print('var_grads_vals = ', var_grads_vals) - print('var_grads_with_l2_vals = ', var_grads_with_l2_vals) - self.assertAllEqual(var_grads_vals.emb[0], - var_grads_with_l2_vals.emb[0]) - self.assertAllEqual(var_grads_vals.weight[0], - var_grads_with_l2_vals.weight[0]) - self.assertAllEqual( - l2_loss_val, - 0.5 * 0.1 * (np.sum(np.square(var_grads_vals.weight[0])) + - np.sum(np.square(var_grads_vals.emb[0][2, :])) + - np.sum(np.square(var_grads_vals.emb[0][5, :])))) - - # With l2, gradients of emb and weight are adjusted. - self.assertAllClose( - var_grads_with_l2_vals.weight[1], - var_grads_vals.weight[1] + 0.1 * var_grads_vals.weight[0]) - self.assertAllClose(var_grads_with_l2_vals.emb[1].indices, - var_grads_vals.emb[1].indices) - self.assertAllClose(var_grads_with_l2_vals.emb[1].indices, - [2, 5, 2, 2, 5]) - self.assertAllClose( - var_grads_with_l2_vals.emb[1].values, var_grads_vals.emb[1].values + - 0.1 * np.array([[1 / 3.], [1 / 2.], [1 / 3.], [1 / 3.], [1 / 2.]]) * - var_grads_vals.emb[0][[2, 5, 2, 2, 5], :]) - - def testSkipL1Regularization(self): - with self.session(use_gpu=False): - beta = tf.get_variable( - 'beta', - initializer=tf.constant(np.arange(10).reshape([1, 10]), tf.float32)) - tf.add_to_collection(py_utils.SKIP_LP_REGULARIZATION, beta) - gamma = tf.get_variable( - 'gamma', - initializer=tf.constant(np.arange(10).reshape([1, 10]), tf.float32)) - act = tf.constant(np.arange(10).reshape([1, 10]), tf.float32) - pred = act * gamma + beta - loss = tf.reduce_sum(pred) - vmap = py_utils.NestedMap(beta=beta, gamma=gamma) - var_grads = py_utils.ComputeGradients(loss, vmap) - self.assertCountEqual(var_grads.keys(), ['beta', 'gamma']) - l1_loss, var_grads_with_l1 = py_utils.AdjustGradientsWithLpLoss( - var_grads, 0.1, p=1.0) - - self.evaluate(tf.global_variables_initializer()) - var_grads_vals, l1_loss_val, var_grads_with_l1_vals = self.evaluate([ - var_grads.Transform(tuple), l1_loss, - var_grads_with_l1.Transform(tuple) - ]) - print('var_grads_vals = ', var_grads_vals) - print('var_grads_with_l1_vals = ', var_grads_with_l1_vals) - self.assertAllEqual(var_grads_vals.beta[0], - var_grads_with_l1_vals.beta[0]) - self.assertAllEqual(var_grads_vals.gamma[0], - var_grads_with_l1_vals.gamma[0]) - self.assertAllEqual(l1_loss_val, - 0.1 * np.sum(np.abs(var_grads_vals.gamma[0]))) - - def testAdjustGradientsWithL1Loss(self): - with self.session(use_gpu=False): - emb = tf.get_variable( - 'emb', - initializer=tf.constant(np.arange(100).reshape([10, 10]), tf.float32)) - act = tf.gather(emb, [2, 5, 2, 2, 5]) - weight = tf.get_variable( - 'w', initializer=tf.constant(np.ones([10, 1]), tf.float32)) - bias = tf.get_variable('b', initializer=tf.constant([0.217])) - pred = tf.matmul(act, weight) + tf.stop_gradient(bias) - loss = tf.reduce_sum(pred) - vmap = py_utils.NestedMap(emb=emb, weight=weight, bias=bias) - var_grads = py_utils.ComputeGradients(loss, vmap) - self.assertCountEqual(var_grads.keys(), ['emb', 'weight']) - l1_loss, var_grads_with_l1 = py_utils.AdjustGradientsWithLpLoss( - var_grads, 0.1, p=1.0) - - self.evaluate(tf.global_variables_initializer()) - var_grads_vals, l1_loss_val, var_grads_with_l1_vals = self.evaluate([ - var_grads.Transform(tuple), l1_loss, - var_grads_with_l1.Transform(tuple) - ]) - print('var_grads_vals = ', var_grads_vals) - print('var_grads_with_l1_vals = ', var_grads_with_l1_vals) - self.assertAllEqual(var_grads_vals.emb[0], var_grads_with_l1_vals.emb[0]) - self.assertAllEqual(var_grads_vals.weight[0], - var_grads_with_l1_vals.weight[0]) - self.assertAllEqual( - l1_loss_val, 0.1 * (np.sum(np.abs(var_grads_vals.weight[0])) + - np.sum(np.abs(var_grads_vals.emb[0][2, :])) + - np.sum(np.abs(var_grads_vals.emb[0][5, :])))) - - # With l1, gradients of emb and weight are adjusted. - self.assertAllClose( - var_grads_with_l1_vals.weight[1], - var_grads_vals.weight[1] + 0.1 * var_grads_vals.weight[0]) - self.assertAllClose(var_grads_with_l1_vals.emb[1].indices, - var_grads_vals.emb[1].indices) - - def testSplitAndConcat(self): - with self.session(): - # Split a Tensor. - m3x4 = tf.constant(np.arange(12).reshape([3, 4])) - splits = py_utils.SplitRecursively(m3x4, 2) - self.assertLen(splits, 2) - for split in splits: - self.assertIsInstance(split, tf.Tensor) - self.assertAllClose([[0, 1], [4, 5], [8, 9]], splits[0].eval()) - self.assertAllClose([[2, 3], [6, 7], [10, 11]], splits[1].eval()) - concatenated = py_utils.ConcatRecursively(splits) - self.assertAllClose(m3x4.eval(), concatenated.eval()) - - # Split along axis 0. - splits = py_utils.SplitRecursively(m3x4, 3, axis=0) - self.assertLen(splits, 3) - concatenated = py_utils.ConcatRecursively(splits, axis=0) - self.assertAllClose(m3x4.eval(), concatenated.eval()) - self.assertAllClose([[0, 1, 2, 3]], splits[0].eval()) - - # Split a list. - list_3 = [m3x4] * 3 - splits = py_utils.SplitRecursively(list_3, 2) - for split in splits: - self.assertIsInstance(split, list) - for x in splits[0]: - self.assertAllClose([[0, 1], [4, 5], [8, 9]], x.eval()) - for x in splits[1]: - self.assertAllClose([[2, 3], [6, 7], [10, 11]], x.eval()) - concatenated = py_utils.ConcatRecursively(splits) - self.assertAllClose([x.eval() for x in list_3], - [x.eval() for x in concatenated]) - - # Split a NestedMap. - map_ab = py_utils.NestedMap(a=m3x4, b=list_3) - splits = py_utils.SplitRecursively(map_ab, 2) - for split in splits: - self.assertIsInstance(split, py_utils.NestedMap) - self.assertIsInstance(split.a, tf.Tensor) - self.assertIsInstance(split.b, list) - for x in splits[0].b: - self.assertAllClose([[0, 1], [4, 5], [8, 9]], x.eval()) - concatenated = py_utils.ConcatRecursively(splits) - self.assertAllClose(map_ab.a.eval(), concatenated.a.eval()) - self.assertAllClose([x.eval() for x in map_ab.b], - [x.eval() for x in concatenated.b]) - - def testFindNeeded(self): - phs = [ - tf.placeholder('float32', shape=(), name='p%d' % (i + 1)) - for i in range(4) - ] - p1, p2, p3, p4 = phs - - z1 = p1 + p2 - z2 = z1 * p3 - - z1_needed = set(py_utils.FindNeededInList(phs, z1)) - z2_needed = set(py_utils.FindNeededInList(phs, [z2])) - z2_p4_needed = set(py_utils.FindNeededInList(phs, [z2, p4])) - - self.assertEqual(set([p1, p2]), z1_needed) - self.assertEqual(set([p1, p2, p3]), z2_needed) - self.assertEqual(set([p1, p2, p3, p4]), z2_p4_needed) - - def testArgMax(self): - - def Compute(x): - with self.session(graph=tf.Graph()): - x = tf.constant(x) - y = py_utils.ArgMax(x) - return self.evaluate([x, y]) - - np.random.seed(426421) - x, y = Compute(np.random.uniform(size=(3, 5, 10))) - self.assertAllEqual(np.argmax(x, axis=-1), y) - - x, y = Compute(np.array([[1, 5, 3, 4, 5], [1, 5, 3, 5, 0]])) # Has dups. - self.assertAllEqual(np.argmax(x, axis=-1), y) - - def testPiecewiseConstant(self): - boundaries = (1000, 2000, 3000) - values = (1e-3, 2e-4, 3e-5, 4e-6) - - def _Eval(x): - with self.session(use_gpu=False): - result = py_utils.PiecewiseConstant( - x, boundaries, values, vdtype=tf.float32) - return self.evaluate(result) - - self.assertAlmostEqual(1e-3, _Eval(0)) - self.assertAlmostEqual(1e-3, _Eval(999)) - self.assertAlmostEqual(2e-4, _Eval(1000)) - self.assertAlmostEqual(2e-4, _Eval(1001)) - self.assertAlmostEqual(2e-4, _Eval(1999)) - self.assertAlmostEqual(3e-5, _Eval(2000)) - self.assertAlmostEqual(4e-6, _Eval(3000)) - self.assertAlmostEqual(4e-6, _Eval(4000)) - - def testRepeatDim(self): - # Create a tensor shaped [time (2), batch(2), depth(3)] - x = tf.constant([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]]) - # [batch, time, depth] - y = tf.transpose(x, [1, 0, 2]) - # [depth, batch, time] - z = tf.transpose(x, [2, 1, 0]) - repeat_inner_dim0 = py_utils.RepeatDim(x, 2, 0) - repeat_inner_dim1 = py_utils.RepeatDim(y, 2, 1) - repeat_inner_dim2 = py_utils.RepeatDim(z, 2, 2) - - with self.session(use_gpu=False): - [repeat_inner_dim0, repeat_inner_dim1, repeat_inner_dim2] = self.evaluate( - [repeat_inner_dim0, repeat_inner_dim1, repeat_inner_dim2]) - self.assertAllEqual( - repeat_inner_dim0, - [[[1, 2, 3], [4, 5, 6]], [[1, 2, 3], [4, 5, 6]], - [[7, 8, 9], [10, 11, 12]], [[7, 8, 9], [10, 11, 12]]]) - repeat_inner_dim = np.transpose(repeat_inner_dim0, [1, 0, 2]) - self.assertAllEqual(repeat_inner_dim1, repeat_inner_dim) - repeat_inner_dim = np.transpose(repeat_inner_dim0, [2, 1, 0]) - self.assertAllEqual(repeat_inner_dim2, repeat_inner_dim) - - def testStackTensorsRecursively(self): - with self.session(use_gpu=False, graph=tf.Graph()): - stacked = py_utils.StackTensorsRecursively([ - py_utils.NestedMap( - x=tf.constant([1, 2]), - y=py_utils.NestedMap(), - z=py_utils.NestedMap(a=tf.constant([1, 2]),), - ), - py_utils.NestedMap( - x=tf.constant([3, 4]), - y=py_utils.NestedMap(), - z=py_utils.NestedMap(a=tf.constant([10, 20]),), - ), - ]) - self.evaluate(tf.global_variables_initializer()) - self.assertAllEqual(stacked.x, tf.constant([[1, 2], [3, 4]])) - self.assertAllEqual(stacked.z.a, tf.constant([[1, 2], [10, 20]])) - - def testCumSum(self): - with self.session(use_gpu=False), mock.patch( - 'lingvo.core.py_utils.use_tpu', return_value=True): - np.random.seed(12345) - x = tf.constant(np.random.rand(2, 4, 8), dtype=tf.float32) - - # If rank is non-static, py_utils.CumSum falls back to tf.cumsum. Make - # sure it's not the case. - rank = py_utils.GetRank(x) - self.assertIsInstance(rank, int) - - self.assertAllClose(py_utils.CumSum(x, 0).eval(), tf.cumsum(x, 0).eval()) - self.assertAllClose(py_utils.CumSum(x, 1).eval(), tf.cumsum(x, 1).eval()) - self.assertAllClose(py_utils.CumSum(x, 2).eval(), tf.cumsum(x, 2).eval()) - self.assertAllClose( - py_utils.CumSum(x, -1).eval(), - tf.cumsum(x, -1).eval()) - self.assertAllClose( - py_utils.CumSum(x, -2).eval(), - tf.cumsum(x, -2).eval()) - self.assertAllClose( - py_utils.CumSum(x, -3).eval(), - tf.cumsum(x, -3).eval()) - with self.assertRaises(ValueError): - py_utils.CumSum(x, -4).eval() - - def testProjectLastDim(self): - np.random.seed(12345) - input_dim = 4 - output_dim = 6 - inputs_p = np.random.rand(2, 5, input_dim) - weight_p = np.random.rand(input_dim, output_dim) - - with self.session(use_gpu=False), mock.patch( - 'lingvo.core.py_utils.use_tpu', return_value=True): - inputs = tf.constant(inputs_p, dtype=tf.float32) - weight = tf.constant(weight_p, dtype=tf.float32) - outputs = py_utils.ProjectLastDim(inputs, weight, input_dim, output_dim) - - self.assertAllClose(outputs.eval(), - np.einsum('bti,io->bto', inputs_p, weight_p)) - - def testAssertEvenDivide(self): - with self.session(): - op = py_utils.assert_even_divide(4, 2) - self.evaluate(op) - - op = py_utils.assert_even_divide(4, 3) - with self.assertRaises(tf.errors.InvalidArgumentError): - self.evaluate(op) - - @mock.patch.object(tf.tpu, 'outside_compilation', autospec=True) - def testTpuHostDecorator(self, mock_outside_compilation): - - with self.session(use_gpu=False), mock.patch( - 'lingvo.core.py_utils.use_tpu', return_value=True): - - def noop_outside_compilation(func, *args, **kwargs): # pylint:disable=invalid-name - return func(*args, **kwargs) - - mock_outside_compilation.side_effect = noop_outside_compilation - - @py_utils.tpu_host - def foo(x): # pylint:disable=invalid-name - return bar(x) * x - - @py_utils.tpu_host - def bar(x): # pylint:disable=invalid-name - return tf.math.log(x) - - x = tf.random.uniform([]) - y = foo(x) - unused_z = y * y - - self.assertTrue(py_utils.use_tpu()) - self.assertEqual(1, mock_outside_compilation.call_count) - - def testRemoveAssertContext(self): - - @tf.Defun(tf.float32, noinline=True) - def Op(x): - with py_utils.RemoveAssertContext(remove=True): - x = py_utils.with_dependencies( - [tf.assert_equal(0, 1, message='assert not removed')], x) - x = py_utils.with_dependencies( - [tf.check_ops.assert_equal(0, 1, message='assert not removed')], x) - return x - - with self.session(use_gpu=True): - - x = tf.ones((2, 2)) - y = Op(x) - _ = self.evaluate(y) - - def testDefaultVnParams(self): - default_vn = py_utils.DefaultVN() - disable_vn = py_utils.DisableVN() - self.assertNotEqual(default_vn, disable_vn) - - def testShardedFilePatternToGlob(self): - file_pattern = '/some/path/to/file@8' - self.assertEqual('/some/path/to/file-?????-of-00008', - py_utils.ShardedFilePatternToGlob(file_pattern)) - - file_pattern = '/some/path/to/file@000008' - self.assertEqual('/some/path/to/file-?????-of-00008', - py_utils.ShardedFilePatternToGlob(file_pattern)) - - file_pattern = '/some/path/to/file@888888' - self.assertEqual('/some/path/to/file-?????-of-888888', - py_utils.ShardedFilePatternToGlob(file_pattern)) - - file_pattern = '/some/path/to/file' - self.assertEqual('/some/path/to/file', - py_utils.ShardedFilePatternToGlob(file_pattern)) - - file_pattern = '/some/path/to/file*' - self.assertEqual('/some/path/to/file*', - py_utils.ShardedFilePatternToGlob(file_pattern)) - - file_pattern = '/some/path/to/file1@8,/some/path/to/file2@8' - with self.assertRaises(ValueError): - py_utils.ShardedFilePatternToGlob(file_pattern) - - -class DeterministicDropoutTest(test_utils.TestCase): - - def testDeterministicDropoutTest(self): - x = tf.ones([4, 6], dtype=tf.float32) - x = py_utils.DeterministicDropout(x, keep_prob=0.7, seeds=[1234, 5678]) - with self.session(): - x_val = self.evaluate(x) - self.assertAllClose([ - [1.0 / 0.7, 0.0000000, 0.0000000, 0.0000000, 1.0 / 0.7, 1.0 / 0.7], - [1.0 / 0.7, 1.0 / 0.7, 1.0 / 0.7, 1.0 / 0.7, 1.0 / 0.7, 1.0 / 0.7], - [1.0 / 0.7, 0.0000000, 0.0000000, 1.0 / 0.7, 1.0 / 0.7, 0.0000000], - [1.0 / 0.7, 0.0000000, 0.0000000, 1.0 / 0.7, 1.0 / 0.7, 1.0 / 0.7], - ], x_val) - self.assertAllClose(22.85714, np.sum(x_val)) - self.assertEqual(x_val.dtype, np.float32) - - -class DeterministicVNTest(test_utils.TestCase): - - def testDeterministicVNTest(self): - x = py_utils.DeterministicVN( - py_utils.NestedMap({'fprop_dtype': tf.float32}), - seeds=[1234, 5678], - noise_shape=np.asarray([3, 3])) - with self.session(): - x_val = self.evaluate(x) - self.assertAllClose( - [[0.686219, 1.360352, 0.197617], [1.866415, 0.326923, -0.781743], - [1.130218, -0.726336, 0.667041]], x_val) - self.assertAllClose(4.726707, np.sum(x_val)) - self.assertEqual(x_val.dtype, np.float32) - - -class WeightedAvgTest(test_utils.TestCase): - - def testWeightedAvg(self): - with self.session(use_gpu=False): - losses = tf.constant([5.6, 4.6, 1.5, 3.4]) - weights = tf.constant([10, 9, 2, 8]) - loss, weight = py_utils.WeightedAvg(losses, weights) - expected = [4.4, 29] - actual = self.evaluate([loss, weight]) - self.assertAllClose(actual, expected) - - def testWeightedAvgOfMetrics(self): - with self.session(use_gpu=False): - metrics = [{ - 'a': (2.0, 0.5), - 'b': (5.0, 1.5) - }, { - 'a': (9.0, 3.0), - 'b': (4.0, 0.5) - }] - expected = {'a': (8.0, 3.5), 'b': (4.75, 2.0)} - weighted_avg = py_utils.WeightedAvgOfMetrics(metrics) - actual = self.evaluate(weighted_avg) - self.assertDictEqual(actual, expected) - - def testConcatPerExampleTensors(self): - with self.session(use_gpu=False): - per_example_1 = { - 'a': - tf.constant([[1.0, 2.0, 3.0], [12.0, 13.0, 14.0]], - dtype=tf.float32), - 'b': - tf.constant([[1.5, 2.5, 3.5, 4.5]], dtype=tf.float32), - } - per_example_2 = { - 'a': - tf.constant([[3.0, 4.0, 5.0], [9.0, 10.0, 11.0]], - dtype=tf.float32), - 'b': - tf.constant([[3.5, 4.5, 5.5, 6.5]], dtype=tf.float32), - } - expected = { - 'a': [[1.0, 2.0, 3.0], [12.0, 13.0, 14.0], [3.0, 4.0, 5.0], - [9.0, 10.0, 11.0]], - 'b': [[1.5, 2.5, 3.5, 4.5], [3.5, 4.5, 5.5, 6.5]] - } - stacked = py_utils.ConcatPerExampleTensors([per_example_1, per_example_2]) - actual = self.evaluate(stacked) - self.assertAllClose(actual['a'], expected['a']) - self.assertAllClose(actual['b'], expected['b']) - self.assertEqual(2, len(actual)) - - def testCombineMetrics(self): - a = py_utils.NestedMap() - a['a'] = (1, 1) - a['loss'] = (100, 10) - b = py_utils.NestedMap() - b['b'] = (2, 2) - b['loss'] = (50, 20) - c = py_utils.NestedMap() - c['loss'] = (60, 15) - combined = py_utils.CombineMetrics([(a, 0.7), (b, 0.3), (c, 1.5)]) - self.assertEqual(combined['a'], (1, 1)) - self.assertEqual(combined['b'], (2, 2)) - total_loss = combined['loss'][0] * combined['loss'][1] - self.assertEqual(total_loss, 100 * 10 * 0.7 + 50 * 20 * 0.3 + 60 * 15 * 1.5) - - def testCombineMetricsKeyNotInAllMetrics(self): - a = py_utils.NestedMap() - a['a'] = (1, 1) - b = py_utils.NestedMap() - b['b'] = (2, 2) - b['loss'] = (50, 20) - c = py_utils.NestedMap() - c['loss'] = (60, 15) - with self.assertRaises(ValueError): - py_utils.CombineMetrics([(a, 0.7), (b, 0.3), (c, 1.5)]) - - -class OverrideVarsFromCheckpointsTest(test_utils.TestCase): - - def _GetLeNetVarsFirstVal(self, sess): - with tf.variable_scope('lenet5', reuse=True): - conv0 = tf.get_variable('conv0/w/var') - conv1 = tf.get_variable('conv1/w/var') - fc_bias = tf.get_variable('fc/b/var') - conv0_val, conv1_val, fc_bias_val = self.evaluate([conv0, conv1, fc_bias]) - return conv0_val[0][0][0][0], conv1_val[0][0][0][0], fc_bias_val[0] - - def testOverrideVarsFromCheckpoint(self): - - with self.session(use_gpu=False) as sess: - tf.random.set_seed(8372749040) - cfg = model_registry.GetParams('image.mnist.LeNet5', 'Train') - with cluster_factory.ForTestingWorker(mode='sync', job='trainer_client'): - cfg.Instantiate() - self.evaluate(tf.global_variables_initializer()) - self.assertAllClose( - # These are initialized values before overriding with checkpoint. - self._GetLeNetVarsFirstVal(sess), - [-0.005945, -0.036722, 0.0]) - checkpoint_path = test_helper.test_src_dir_path( - 'core/testdata/lenet_test_model') - variable_loading_rules = [('lenet5/conv0/w/var', 'lenet5/conv0/w/var'), - ('lenet5/conv1/w/var', 'lenet5/conv1/w/var')] - variable_ignore_rules = [] - py_utils.OverrideVarsFromCheckpoint(sess, tf.all_variables(), - checkpoint_path, - variable_loading_rules, - variable_ignore_rules) - self.assertAllClose( - # Now conv weights have been overwritten but fc bias has not. - self._GetLeNetVarsFirstVal(sess), - [0.043092, -0.024082, 0.0]) - - def testOverrideVarsFromCheckpointWithIgnoreRules(self): - - with self.session(use_gpu=False) as sess: - tf.random.set_seed(8372749040) - cfg = model_registry.GetParams('image.mnist.LeNet5', 'Train') - with cluster_factory.ForTestingWorker(mode='sync', job='trainer_client'): - cfg.Instantiate() - self.evaluate(tf.global_variables_initializer()) - self.assertAllClose( - # These are initialized values before overriding with checkpoint. - self._GetLeNetVarsFirstVal(sess), - [-0.005945, -0.036722, 0.0]) - checkpoint_path = test_helper.test_src_dir_path( - 'core/testdata/lenet_test_model') - variable_loading_rules = [('lenet5/conv0/w/var', 'lenet5/conv0/w/var'), - ('lenet5/conv1/w/var', 'lenet5/conv1/w/var')] - variable_ignore_rules = ['lenet5/conv1/w/var'] - py_utils.OverrideVarsFromCheckpoint(sess, tf.all_variables(), - checkpoint_path, - variable_loading_rules, - variable_ignore_rules) - self.assertAllClose( - # Now only conv0 weights have been overridden. - self._GetLeNetVarsFirstVal(sess), - [0.043092, -0.036722, 0.0]) - - -def _AddOne(x): - return None if x is None else x + type(x)(1) - - -class NestedMapTest(test_utils.TestCase): - - _TUPLE = collections.namedtuple('Tuple', ['x', 'y']) - - def _get_basic_test_inputs(self): - m = py_utils.NestedMap() - m.foo = [1, 20, [32]] - m.bar = py_utils.NestedMap() - m.bar.x = 100 - m.bar.y = [200, py_utils.NestedMap(z='abc')] - return m - - def _get_advanced_test_inputs(self): - m = py_utils.NestedMap() - m.w = None - m.y = (200, py_utils.NestedMap(z='abc')) - m.x = {'foo': 1, 'bar': 'def'} - m.z = self._TUPLE(5, 'xyz') - m.zz = [] - return m - - def testBasic(self): - x = py_utils.NestedMap() - self.assertLen(x, 0) - x['foo'] = 100 - self.assertEqual(100, x.foo) - self.assertEqual(100, x['foo']) - x.bar = py_utils.NestedMap({'baz': 200}) - self.assertEqual(200, x.bar.baz) - self.assertNotIn('flatten', x) - - def testPrint(self): - self.assertEqual(py_utils.NestedMap().DebugString(), '') - - expected = """bar.x 100 -bar.y[0] 200 -bar.y[1].z abc -foo[0] 1 -foo[1] 20 -foo[2][0] 32""" - m = self._get_basic_test_inputs() - self.assertEqual(m.DebugString(), expected) - - m = self._get_advanced_test_inputs() - res = m.DebugString() - w, x1, x2, y, z = res.split('\n') - self.assertEqual(w, 'w None') - self.assertEqual(x1, 'x.bar def') - self.assertEqual(x2, 'x.foo 1') - self.assertEqual(y, "y (200, {'z': 'abc'})") - self.assertEqual(z, "z Tuple(x=5, y='xyz')") - - def testTransformBasic(self): - n = py_utils.Transform(_AddOne, py_utils.NestedMap()) - self.assertEqual(n.DebugString(), '') - n = py_utils.NestedMap().Transform(_AddOne) - self.assertEqual(n.DebugString(), '') - - expected = """bar.x 101 -bar.y[0] 201 -bar.y[1].z abc1 -foo[0] 2 -foo[1] 21 -foo[2][0] 33""" - m = self._get_basic_test_inputs() - n = py_utils.Transform(_AddOne, m) - self.assertEqual(n.DebugString(), expected) - n = m.Transform(_AddOne) - self.assertEqual(n.DebugString(), expected) - - # Original has not been modified. - expected = """bar.x 100 -bar.y[0] 200 -bar.y[1].z abc -foo[0] 1 -foo[1] 20 -foo[2][0] 32""" - self.assertEqual(m.DebugString(), expected) - - def testTransformAdvanced(self): - m = self._get_advanced_test_inputs() - original = [ - ('w', None), - ('x.bar', 'def'), - ('x.foo', 1), - ('y', (200, { - 'z': 'abc' - })), - ('z', self._TUPLE(x=5, y='xyz')), - ] - self.assertEqual(m.FlattenItems(), original) - - expected = [ - ('w', None), - ('x.bar', 'def1'), - ('x.foo', 2), - ('y', (201, { - 'z': 'abc1' - })), - ('z', self._TUPLE(x=6, y='xyz1')), - ] - n = py_utils.Transform(_AddOne, m) - self.assertEqual(n.zz, []) - self.assertNotEqual(expected, original) - self.assertEqual(n.FlattenItems(), expected) - - with self.assertRaises(TypeError): - m.Transform(_AddOne) - - def _AddOneIgnoreError(x): - try: - return _AddOne(x) - except TypeError: - return x - - expected = [ - ('w', None), - ('x.bar', 'def1'), - ('x.foo', 2), - ('y', (200, { - 'z': 'abc' - })), - ('z', self._TUPLE(x=5, y='xyz')), - ] - n = m.Transform(_AddOneIgnoreError) - self.assertEqual(n.zz, []) - self.assertNotEqual(expected, original) - self.assertEqual(n.FlattenItems(), expected) - - # Original has not been modified. - self.assertEqual(m.FlattenItems(), original) - - def testFlattenBasic(self): - self.assertEqual(py_utils.Flatten(py_utils.NestedMap()), []) - self.assertEqual(py_utils.NestedMap().Flatten(), []) - self.assertEqual(py_utils.NestedMap().FlattenItems(), []) - - expected = [100, 200, 'abc', 1, 20, 32] - m = self._get_basic_test_inputs() - self.assertEqual(py_utils.Flatten(m), expected) - self.assertEqual(m.Flatten(), expected) - - expected_keys = [ - 'bar.x', 'bar.y[0]', 'bar.y[1].z', 'foo[0]', 'foo[1]', 'foo[2][0]' - ] - self.assertEqual(m.FlattenItems(), list(zip(expected_keys, expected))) - - def testFlattenAdvanced(self): - m = self._get_advanced_test_inputs() - - expected = [None, 'def', 1, 200, 'abc', 5, 'xyz'] - self.assertEqual(py_utils.Flatten(m), expected) - - expected = [ - None, - 'def', - 1, - (200, { - 'z': 'abc' - }), - self._TUPLE(x=5, y='xyz'), - ] - self.assertEqual(m.Flatten(), expected) - - expected = [ - ('w', None), - ('x.bar', 'def'), - ('x.foo', 1), - ('y', (200, { - 'z': 'abc' - })), - ('z', self._TUPLE(x=5, y='xyz')), - ] - self.assertEqual(m.FlattenItems(), expected) - - def testPackBasic(self): - n = py_utils.Pack(py_utils.NestedMap(), []) - self.assertEqual(n.DebugString(), '') - n = py_utils.NestedMap().Pack([]) - self.assertEqual(n.DebugString(), '') - - expected = """bar.x 0 -bar.y[0] 1 -bar.y[1].z 2 -foo[0] 3 -foo[1] 4 -foo[2][0] 5""" - m = self._get_basic_test_inputs() - n = py_utils.Pack(m, list(range(6))) - self.assertEqual(n.DebugString(), expected) - n = m.Pack(list(range(6))) - self.assertEqual(n.DebugString(), expected) - - # Original has not been modified. - expected = """bar.x 100 -bar.y[0] 200 -bar.y[1].z abc -foo[0] 1 -foo[1] 20 -foo[2][0] 32""" - self.assertEqual(m.DebugString(), expected) - - def testPackAdvanced(self): - m = self._get_advanced_test_inputs() - - expected = [ - ('w', 0), - ('x.bar', 1), - ('x.foo', 2), - ('y', (3, { - 'z': 4 - })), - ('z', self._TUPLE(x=5, y=None)), - ] - n = py_utils.Pack(m, list(range(6)) + [None]) - self.assertEqual(n.zz, []) - self.assertEqual(n.FlattenItems(), expected) - - expected = [('w', 0), ('x.bar', 1), ('x.foo', 2), ('y', 3), ('z', None)] - n = m.Pack(list(range(4)) + [None]) - self.assertEqual(n.zz, []) - self.assertEqual(n.FlattenItems(), expected) - - # Original has not been modified. - expected = [ - ('w', None), - ('x.bar', 'def'), - ('x.foo', 1), - ('y', (200, { - 'z': 'abc' - })), - ('z', self._TUPLE(x=5, y='xyz')), - ] - self.assertEqual(m.FlattenItems(), expected) - - def testIsCompatible(self): - empty = py_utils.NestedMap() - self.assertTrue(empty.IsCompatible(empty)) - self.assertTrue(py_utils.IsCompatible(empty, empty)) - self.assertTrue(empty.IsCompatible(py_utils.NestedMap(x=[]))) - self.assertFalse(py_utils.IsCompatible(empty, py_utils.NestedMap(x=[]))) - self.assertTrue(empty.IsCompatible(py_utils.NestedMap(x=empty))) - self.assertFalse(py_utils.IsCompatible(empty, py_utils.NestedMap(x=empty))) - self.assertTrue(empty.IsCompatible(py_utils.NestedMap(x={}))) - self.assertFalse(py_utils.IsCompatible(empty, py_utils.NestedMap(x={}))) - x = py_utils.NestedMap( - a='a', b='b', c=py_utils.NestedMap(d='d', e=[1, 2, 4])) - y = py_utils.NestedMap(a=1, b=2, c=py_utils.NestedMap(d=3, e=[10, 20, 30])) - z = py_utils.NestedMap( - a=1, b=[10, 20, 30], c=py_utils.NestedMap(d=3, e=['x', 'y', 'z'])) - self.assertTrue(x.IsCompatible(y)) - self.assertTrue(py_utils.IsCompatible(x, y)) - self.assertFalse(x.IsCompatible(z)) - self.assertFalse(py_utils.IsCompatible(x, z)) - - def testFilter(self): - x = py_utils.NestedMap( - a=100, - b=200, - c=300, - d=py_utils.NestedMap(foo=38, bar=192, ok=[200, 300], ko=[10, 20])) - y = x.Filter(lambda v: v > 150) - self.assertEqual(y.FlattenItems(), [('b', 200), ('c', 300), ('d.bar', 192), - ('d.ok[0]', 200), ('d.ok[1]', 300)]) - self.assertNotIn('ko', y.d) - - y = x.Filter(lambda v: v > 500) - self.assertLen(y.FlattenItems(), 0) - - def testFilterKeyVal(self): - x = py_utils.NestedMap( - a=100, - b=200, - c=300, - d=py_utils.NestedMap(foo=38, bar=192, ok=[200, 300], ko=[10, 20])) - selected = {'a', 'd.foo', 'd.ok[1]'} - - def Sel(k, _): - return k in selected - - y = x.FilterKeyVal(Sel) - self.assertEqual(y.FlattenItems(), [('a', 100), ('d.foo', 38), - ('d.ok[0]', 300)]) - - def testCopy(self): - # This is not a copy. - x = py_utils.NestedMap( - a='a', b='b', c=py_utils.NestedMap(d='d', e=[1, 2, 4])) - y = x - y.a = 'y' - self.assertEqual('y', y.a) - self.assertEqual('y', x.a) - - # This is a (shallow) copy. - x = py_utils.NestedMap( - a='a', b='b', c=py_utils.NestedMap(d='d', e=[1, 2, 4])) - y = py_utils.NestedMap(x) - self.assertNotEqual(id(x), id(y)) - y.a = 'y' - y.c.d = 'z' - self.assertEqual('y', y.a) - self.assertEqual('a', x.a) - self.assertEqual('z', y.c.d) - self.assertEqual('z', x.c.d) - - # This is also a (shallow) copy. - x = py_utils.NestedMap( - a='a', b='b', c=py_utils.NestedMap(d='d', e=[1, 2, 4])) - y = x.copy() - self.assertNotEqual(id(x), id(y)) - y.a = 'y' - y.c.d = 'z' - self.assertEqual('y', y.a) - self.assertEqual('a', x.a) - self.assertEqual('z', y.c.d) - self.assertEqual('z', x.c.d) - - def testDeepCopy(self): - - class SomeObj: - - def __init__(self): - self.foo = 'foo' - - x = py_utils.NestedMap( - a='a', - b='b', - c=py_utils.NestedMap(d='d', e=[1, 2, 4], obj=SomeObj()), - f=[], - g={}, - h=py_utils.NestedMap(), - i=None) - # Perform a deep copy. - y = copy.deepcopy(x) - # Objects are different. - self.assertNotEqual(id(x), id(y)) - - # modify deep copy, even nested version. - y.a = 'x' - y.c.e[0] = 'y' - y.c.obj.foo = 'bar' - y.f.append(5) - y.h.foo = 'bar' - - # x values are the originals. - self.assertEqual('a', x.a) - self.assertEqual(1, x.c.e[0]) - self.assertEqual([], x.f) - self.assertEqual({}, x.g) - self.assertLen(x.h, 0) - - # y values are updated. - self.assertEqual('x', y.a) - self.assertEqual('y', y.c.e[0]) - self.assertEqual([5], y.f) - self.assertEqual({}, y.g) - self.assertLen(y.h, 1) - self.assertEqual('bar', y.h.foo) - self.assertIsNone(y.i) - - # but leaf objects are the shared. - self.assertEqual('bar', x.c.obj.foo) - self.assertEqual('bar', y.c.obj.foo) - self.assertEqual(id(x.c.obj), id(y.c.obj)) - - def testAttrAccess(self): - a = py_utils.NestedMap() - a.a1 = 10 - self.assertEqual(10, a.a1) - self.assertEqual(10, a['a1']) - self.assertEqual(10, a.get('a1')) - self.assertEqual(10, a.get(u'a1')) - self.assertEqual(10, getattr(a, 'a1')) - - a[u'a1'] = 20 - self.assertEqual(20, a.a1) - self.assertEqual(20, a['a1']) - self.assertEqual(20, a.get('a1')) - self.assertEqual(20, a.get(u'a1')) - self.assertEqual(20, getattr(a, 'a1')) - - with self.assertRaisesRegex(AttributeError, 'available attributes'): - print(a.a2) - with self.assertRaises(KeyError): - print(a['a2']) - - # 'get' is a reserved key. - with self.assertRaisesRegex(AssertionError, 'is a reserved key'): - a.get = 10 - with self.assertRaisesRegex(AssertionError, 'is a reserved key'): - a['get'] = 10 - with self.assertRaisesRegex(AssertionError, 'is a reserved key'): - _ = py_utils.NestedMap(get=2) - - del a.a1 - with self.assertRaisesRegex(AttributeError, 'available attributes'): - print(a.a1) - - with self.assertRaisesRegex(AttributeError, 'available attributes'): - del a.a2 - - -class ReadOnlyAttrDictViewTest(test_utils.TestCase): - - def testWrapping(self): - backing = dict() - view = py_utils.ReadOnlyAttrDictView(backing) - backing['test'] = 1 - - self.assertEqual(1, view['test']) - self.assertEqual(1, view.test) - # Item assign. - with self.assertRaises(AttributeError): - view['test'] = 2 - self.assertEqual(1, view['test']) - # Attr assign. - with self.assertRaises(AttributeError): - view.test = 2 - self.assertEqual(1, view['test']) - # Delete attr. - with self.assertRaises(AttributeError): - del view.test - self.assertEqual(1, view['test']) - # Delete item. - with self.assertRaises(AttributeError): - del view['test'] - self.assertEqual(1, view['test']) - - -class PadPadSequenceToTest(test_utils.TestCase): - - def test2DInputs(self): - with self.session(use_gpu=False, graph=tf.Graph()): - x = tf.random.normal(shape=(3, 3), seed=123456) - padding = tf.constant([[0, 0, 0], [0, 0, 1], [0, 1, 1]], tf.float32) - length = 6 - new_xs, new_padding = py_utils.PadSequenceTo([x, x], padding, length, 0) - - real_xs, real_padding = self.evaluate([new_xs, new_padding]) - expected_x = [ - [0.38615, 2.975221, -0.852826, 0., 0., 0.], - [-0.571142, -0.432439, 0.413158, 0., 0., 0.], - [0.255314, -0.985647, 1.461641, 0., 0., 0.], - ] - expected_padding = [ - [0., 0., 0., 1., 1., 1.], - [0., 0., 1., 1., 1., 1.], - [0., 1., 1., 1., 1., 1.], - ] - self.assertAllClose([expected_x, expected_x], real_xs) - self.assertAllClose(expected_padding, real_padding) - - def testSingleInput(self): - with self.session(use_gpu=False, graph=tf.Graph()): - x = tf.random.normal(shape=(3, 3), seed=123456) - padding = tf.constant([[0, 0, 0], [0, 0, 1], [0, 1, 1]], tf.float32) - length = 6 - new_x, new_padding = py_utils.PadSequenceTo(x, padding, length, 0) - - real_x, real_padding = self.evaluate([new_x, new_padding]) - expected_x = [ - [0.38615, 2.975221, -0.852826, 0., 0., 0.], - [-0.571142, -0.432439, 0.413158, 0., 0., 0.], - [0.255314, -0.985647, 1.461641, 0., 0., 0.], - ] - expected_padding = [ - [0., 0., 0., 1., 1., 1.], - [0., 0., 1., 1., 1., 1.], - [0., 1., 1., 1., 1., 1.], - ] - self.assertAllClose(expected_x, real_x) - self.assertAllClose(expected_padding, real_padding) - - -class PadSequenceDimensionTest(test_utils.TestCase): - - def testPadSequenceDimension_2D(self): - with self.session(use_gpu=False, graph=tf.Graph()): - x = tf.random.normal(shape=(3, 3), seed=123456) - length = 6 - padded_x = py_utils.PadSequenceDimension(x, length, 0) - self.assertEqual(padded_x.shape.as_list(), [3, 6]) - real_x = self.evaluate(padded_x) - expected_x = [ - [0.38615, 2.975221, -0.852826, 0., 0., 0.], - [-0.571142, -0.432439, 0.413158, 0., 0., 0.], - [0.255314, -0.985647, 1.461641, 0., 0., 0.], - ] - self.assertAllClose(expected_x, real_x) - - def testPadSequenceDimension_2D_UnknownShape(self): - with self.session(use_gpu=False, graph=tf.Graph()) as sess: - shape = tf.placeholder(tf.int32) - x = tf.random.normal(shape=shape, seed=123456) - length = 6 - padded_x = py_utils.PadSequenceDimension(x, length, 0) - self.assertEqual(padded_x.shape, tf.TensorShape(None)) - real_x = sess.run(padded_x, feed_dict={shape: [3, 3]}) - expected_x = [ - [0.38615, 2.975221, -0.852826, 0., 0., 0.], - [-0.571142, -0.432439, 0.413158, 0., 0., 0.], - [0.255314, -0.985647, 1.461641, 0., 0., 0.], - ] - self.assertAllClose(expected_x, real_x) - - def testPadSequenceDimension_ShortPaddingLength(self): - x = tf.random.normal(shape=(3, 8), seed=123456) - length = 6 - with self.assertRaisesRegex(ValueError, 'Paddings must be non-negative'): - py_utils.PadSequenceDimension(x, length, 0) - - def testPadSequenceDimension_4D(self): - with self.session(use_gpu=False, graph=tf.Graph()): - x = tf.random.normal(shape=(2, 2, 2, 2), seed=123456) - length = 4 - padded_x = py_utils.PadSequenceDimension(x, length, 1) - real_x = self.evaluate(padded_x) - expected_x = [ - [[[0.38614973, 2.97522092], [-0.85282576, -0.57114178]], - [[-0.43243945, 0.41315758], [0.2553139, -0.98564667]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]], - [[[1.46164131, 0.12003655], [-0.0986772, 0.60644895]], - [[0.03092973, -0.96897006], [-1.27853918, -0.44018385]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]], - ] - self.assertAllClose(expected_x, real_x) - - def testPadSequenceDimension_UnmatchedShape(self): - with self.session(use_gpu=False, graph=tf.Graph()): - x = tf.random.normal(shape=(2, 2, 2, 2), seed=123456) - length = 4 - self.assertRaises(ValueError, py_utils.PadSequenceDimension, x, length, 0, - (32, 3, 4, 5)) - - -class PadOrTrimToTest(test_utils.TestCase): - - def test2DConstantShapePad(self): - with self.session(use_gpu=False, graph=tf.Graph()): - x = tf.random.normal(shape=(3, 3), seed=123456) - shape = [4, 6] - padded_x_right = py_utils.PadOrTrimTo(x, shape, pad_val=0) - padded_x_left = py_utils.PadOrTrimTo( - x, shape, pad_val=0, pad_after_contents=False) - self.assertEqual(padded_x_right.shape.as_list(), [4, 6]) - self.assertEqual(padded_x_left.shape.as_list(), [4, 6]) - real_x_right, real_x_left = self.evaluate([padded_x_right, padded_x_left]) - expected_x_right = [ - [0.38615, 2.975221, -0.852826, 0., 0., 0.], - [-0.571142, -0.432439, 0.413158, 0., 0., 0.], - [0.255314, -0.985647, 1.461641, 0., 0., 0.], - [0., 0., 0., 0., 0., 0.], - ] - self.assertAllClose(expected_x_right, real_x_right) - expected_x_left = [ - [0., 0., 0., 0., 0., 0.], - [0., 0., 0., 0.38615, 2.975221, -0.852826], - [0., 0., 0., -0.571142, -0.432439, 0.413158], - [0., 0., 0., 0.255314, -0.985647, 1.461641], - ] - self.assertAllClose(expected_x_left, real_x_left) - - def test2DConstantShapeTrim(self): - with self.session(use_gpu=False, graph=tf.Graph()): - x = tf.random.normal(shape=(3, 3), seed=123456) - shape = [1, 3] - trimmed_x_right = py_utils.PadOrTrimTo(x, shape, pad_val=0) - trimmed_x_left = py_utils.PadOrTrimTo( - x, shape, pad_val=0, pad_after_contents=False) - self.assertEqual(trimmed_x_right.shape.as_list(), [1, 3]) - self.assertEqual(trimmed_x_left.shape.as_list(), [1, 3]) - real_x_right, real_x_left = self.evaluate( - [trimmed_x_right, trimmed_x_left]) - expected_x_right = [[0.38615, 2.975221, -0.852826]] - self.assertAllClose(expected_x_right, real_x_right) - expected_x_left = [[0.255314, -0.985647, 1.461641]] - self.assertAllClose(expected_x_left, real_x_left) - - def test2DStaticShape(self): - with self.session(use_gpu=False, graph=tf.Graph()): - x = tf.random.normal(shape=(3, 3), seed=123456) - y = tf.zeros(shape=(4, 6)) - padded_x = py_utils.PadOrTrimTo(x, y.shape, pad_val=0) - self.assertEqual(padded_x.shape.as_list(), [4, 6]) - real_x = self.evaluate(padded_x) - expected_x = [ - [0.38615, 2.975221, -0.852826, 0., 0., 0.], - [-0.571142, -0.432439, 0.413158, 0., 0., 0.], - [0.255314, -0.985647, 1.461641, 0., 0., 0.], - [0., 0., 0., 0., 0., 0.], - ] - self.assertAllClose(expected_x, real_x) - - def test2DDynamicShape(self): - with self.session(use_gpu=False, graph=tf.Graph()) as sess: - x = tf.random.normal(shape=(3, 3), seed=123456) - y = tf.placeholder(dtype=tf.float32) - padded_x = py_utils.PadOrTrimTo(x, tf.shape(y), pad_val=0) - self.assertEqual(padded_x.shape, tf.TensorShape(None)) - real_x = sess.run(padded_x, feed_dict={y: np.zeros((4, 6))}) - expected_x = [ - [0.38615, 2.975221, -0.852826, 0., 0., 0.], - [-0.571142, -0.432439, 0.413158, 0., 0., 0.], - [0.255314, -0.985647, 1.461641, 0., 0., 0.], - [0., 0., 0., 0., 0., 0.], - ] - self.assertAllClose(expected_x, real_x) - - def testDynamicTensorShapeRaises(self): - tensor = tf.zeros(shape=[3, 2]) - shape = tf.TensorShape([3, None]) - with self.assertRaises(ValueError): - py_utils.PadOrTrimTo(tensor, shape) - - def test4D(self): - with self.session(use_gpu=False, graph=tf.Graph()): - x = tf.random.normal(shape=(2, 2, 2, 2), seed=123456) - shape = (1, 1, 3, 3) - padded_x = py_utils.PadOrTrimTo(x, shape, pad_val=1) - real_x = self.evaluate(padded_x) - expected_x = [[[ - [0.38615, 2.975221, 1.], - [-0.852826, -0.571142, 1.], - [1., 1., 1.], - ]]] - self.assertAllClose(expected_x, real_x) - - -class ApplyPaddingTest(test_utils.TestCase): - - def testApplyPaddingToZeroWithBroadcast(self): - with self.session(): - y = py_utils.ApplyPadding( - tf.convert_to_tensor([[0.0], [1.0], [0.0]]), - tf.convert_to_tensor([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])).eval() - self.assertAllClose(y, [[1.0, 2.0], [0.0, 0.0], [5.0, 6.0]]) - - def testApplyPaddingToConstWithBroadcast(self): - with self.session(): - y = py_utils.ApplyPadding( - tf.convert_to_tensor([[0.0], [1.0], [0.0]]), - tf.convert_to_tensor([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]), - tf.convert_to_tensor([[1.0, 2.0], [9.0, 10.0], [5.0, 6.0]])).eval() - self.assertAllClose(y, [[1.0, 2.0], [9.0, 10.0], [5.0, 6.0]]) - - def testApplyPaddingToZeroWithoutBroadcast(self): - with self.session(): - y = py_utils.ApplyPadding( - tf.convert_to_tensor([[0.0, 0.0], [1.0, 0.0], [0.0, 1.0]]), - tf.convert_to_tensor([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])).eval() - self.assertAllClose(y, [[1.0, 2.0], [0.0, 4.0], [5.0, 0.0]]) - - def testApplyPaddingToZeroWithBroadcastArithmetic(self): - with self.session(): - y = py_utils.ApplyPadding( - tf.convert_to_tensor([[0.0], [1.0], [0.0]]), - tf.convert_to_tensor([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]), - use_select=False).eval() - self.assertAllClose(y, [[1.0, 2.0], [0.0, 0.0], [5.0, 6.0]]) - - def testApplyPaddingToZeroWithoutBroadcastArithmetic(self): - with self.session(): - y = py_utils.ApplyPadding( - tf.convert_to_tensor([[0.0, 0.0], [1.0, 0.0], [0.0, 1.0]]), - tf.convert_to_tensor([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]), - use_select=False).eval() - self.assertAllClose(y, [[1.0, 2.0], [0.0, 4.0], [5.0, 0.0]]) - - -class LengthsFromPaddingsTest(test_utils.TestCase): - - def testBasic(self): - with self.session(): - paddings = np.array([ - [0.0, 0.0, 0.0, 0.0, 0.0, 0.0], - [0.0, 0.0, 0.0, 1.0, 1.0, 1.0], - [1.0, 1.0, 0.0, 0.0, 0.0, 1.0], - [1.0, 1.0, 1.0, 1.0, 1.0, 1.0], - ]) - lengths = py_utils.LengthsFromPaddings( - tf.convert_to_tensor(paddings)).eval() - self.assertAllEqual([6, 3, 5, 0], lengths) - - def testZeroLength(self): - with self.session(): - paddings = np.zeros([4, 0]) - lengths = py_utils.LengthsFromPaddings( - tf.convert_to_tensor(paddings)).eval() - self.assertAllEqual([0, 0, 0, 0], lengths) - - def testBFloat16(self): - with self.session(): - actual_lengths = [1, 255, 256, 1024, 2048] - paddings = 1.0 - tf.sequence_mask( - actual_lengths, maxlen=actual_lengths[-1], dtype=tf.bfloat16) - lengths = py_utils.LengthsFromPaddings(paddings).eval() - self.assertAllEqual(actual_lengths, lengths) - - -class TrimTrailingPaddingsTest(test_utils.TestCase): - - def test2D(self): - with self.session(use_gpu=False, graph=tf.Graph()): - np.random.seed(123456) - x = np.random.normal(size=(3, 6)) - padding = np.array([ - [1.0, 1.0, 0.0, 0.0, 0.0, 1.0], - [1.0, 0.0, 0.0, 0.0, 1.0, 1.0], - [0.0, 0.0, 0.0, 1.0, 1.0, 1.0], - ]) - trimmed_x, trimmed_padding = self.evaluate( - py_utils.TrimTrailingPaddings(x, tf.convert_to_tensor(padding))) - self.assertAllEqual(x[:, :5], trimmed_x) - self.assertAllEqual(padding[:, :5], trimmed_padding) - - def test2D_UnknownShape(self): - with self.session(use_gpu=False, graph=tf.Graph()) as sess: - shape = tf.placeholder(tf.int32) - x = tf.random.normal(shape=shape, seed=123456) - padding = np.array([ - [1.0, 1.0, 0.0, 0.0, 0.0, 1.0], - [1.0, 0.0, 0.0, 0.0, 1.0, 1.0], - [0.0, 0.0, 0.0, 1.0, 1.0, 1.0], - ]) - actual_x, (trimmed_x, trimmed_padding) = sess.run( - [x, - py_utils.TrimTrailingPaddings(x, tf.convert_to_tensor(padding))], - feed_dict={shape: [3, 6]}) - self.assertAllEqual(actual_x[:, :5], trimmed_x) - self.assertAllEqual(padding[:, :5], trimmed_padding) - - def test4D(self): - with self.session(use_gpu=False, graph=tf.Graph()): - np.random.seed(123456) - x = np.random.normal(size=(3, 6, 3, 3)) - padding = np.array([ - [1.0, 1.0, 0.0, 0.0, 0.0, 1.0], - [1.0, 0.0, 0.0, 0.0, 1.0, 1.0], - [0.0, 0.0, 0.0, 1.0, 1.0, 1.0], - ]) - trimmed_x, trimmed_padding = self.evaluate( - py_utils.TrimTrailingPaddings(x, tf.convert_to_tensor(padding))) - self.assertAllEqual(x[:, :5], trimmed_x) - self.assertAllEqual(padding[:, :5], trimmed_padding) - - def testNoPadding(self): - with self.session(use_gpu=False, graph=tf.Graph()): - np.random.seed(123456) - x = np.random.normal(size=(3, 6)) - padding = np.array([ - [0.0, 0.0, 0.0, 0.0, 0.0, 0.0], - [0.0, 0.0, 0.0, 0.0, 0.0, 0.0], - [0.0, 0.0, 0.0, 0.0, 0.0, 0.0], - ]) - trimmed_x, trimmed_padding = self.evaluate( - py_utils.TrimTrailingPaddings(x, tf.convert_to_tensor(padding))) - self.assertAllEqual(x, trimmed_x) - self.assertAllEqual(padding, trimmed_padding) - - def testLeadingPaddingOnly(self): - with self.session(use_gpu=False, graph=tf.Graph()): - np.random.seed(123456) - x = np.random.normal(size=(3, 6)) - padding = np.array([ - [1.0, 1.0, 0.0, 0.0, 0.0, 0.0], - [1.0, 0.0, 0.0, 0.0, 0.0, 0.0], - [0.0, 0.0, 0.0, 0.0, 0.0, 0.0], - ]) - trimmed_x, trimmed_padding = self.evaluate( - py_utils.TrimTrailingPaddings(x, tf.convert_to_tensor(padding))) - self.assertAllEqual(x, trimmed_x) - self.assertAllEqual(padding, trimmed_padding) - - def testAllPadded(self): - with self.session(use_gpu=False, graph=tf.Graph()): - np.random.seed(123456) - x = np.random.normal(size=(3, 6)) - padding = np.array([ - [1.0, 1.0, 1.0, 1.0, 1.0, 1.0], - [1.0, 1.0, 1.0, 1.0, 1.0, 1.0], - [1.0, 1.0, 1.0, 1.0, 1.0, 1.0], - ]) - trimmed_x, trimmed_padding = self.evaluate( - py_utils.TrimTrailingPaddings(x, tf.convert_to_tensor(padding))) - self.assertAllEqual([3, 1], trimmed_x.shape) - self.assertAllEqual(padding[:, :1], trimmed_padding) - - -class ReversePaddedSequenceTest(test_utils.TestCase): - - def testReversePaddedSequence(self): - with self.session(use_gpu=False): - # inputs is [seq_length, batch_size, input_dim] = [4, 3, 2] - # The length of each batch is [2, 3, 4] - inputs = tf.constant( - [[[1, 2], [3, 4], [5, 6]], [[11, 12], [13, 14], [15, 16]], - [[0, 0], [23, 24], [25, 26]], [[0, 0], [0, 0], [35, 36]]], - dtype=tf.float32) - paddings = tf.constant( - [[[0], [0], [0]], [[0], [0], [0]], [[1], [0], [0]], [[1], [1], [0]]], - dtype=tf.float32) - actual_output = py_utils.ReversePaddedSequence(inputs, paddings).eval() - expected_output = np.array([[[11, 12], [23, 24], [35, 36]], - [[1, 2], [13, 14], [25, 26]], - [[0, 0], [3, 4], [15, 16]], - [[0, 0], [0, 0], [5, 6]]]).astype('float32') - self.assertAllClose(expected_output, actual_output) - - -class ConcatenatePaddedSequencesTest(test_utils.TestCase): - - def _ComputeFloatOutputAndVerify(self, - input0, - input1, - seq_lens0, - seq_lens1, - tranpose_input=False): - with self.session(use_gpu=False): - expected_output_seq_lens = seq_lens0 + seq_lens1 - batch_size, input0_seq_dim = input0.shape - input1_seq_dim = input1.shape[1] - padding0 = 1.0 - tf.sequence_mask( - seq_lens0, maxlen=input0_seq_dim, dtype=tf.float32) - padding1 = 1.0 - tf.sequence_mask( - seq_lens1, maxlen=input1_seq_dim, dtype=tf.float32) - - if tranpose_input: - seq_dim = 0 - tf_input0 = tf.constant(np.transpose(input0)) - tf_input1 = tf.constant(np.transpose(input1)) - tf_padding0 = tf.transpose(padding0) - tf_padding1 = tf.transpose(padding1) - else: - seq_dim = 1 - tf_input0 = tf.constant(input0) - tf_input1 = tf.constant(input1) - tf_padding0 = padding0 - tf_padding1 = padding1 - - actual_outputs = self.evaluate( - py_utils.ConcatenatePaddedSequences( - tf_input0, - tf_input1, - padding0=tf_padding0, - padding1=tf_padding1, - seq_dim=seq_dim)) - - if tranpose_input: - actual_outputs = (np.transpose(actual_outputs[0]), - np.transpose(actual_outputs[1])) - - for batch in range(batch_size): - expected_output = np.concatenate((input0[batch, :seq_lens0[batch]], - input1[batch, :seq_lens1[batch]])) - self.assertAllClose( - expected_output, - actual_outputs[0][batch, :expected_output_seq_lens[batch]]) - expected_padding = np.ones( - (input0_seq_dim + input1_seq_dim,)).astype('float32') - expected_padding[:(seq_lens0[batch] + seq_lens1[batch])] = 0.0 - self.assertAllClose(expected_padding, actual_outputs[1][batch, :]) - - def testConcatenateFloatFeatures(self): - input0 = np.array([[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]]).astype('float32') - seq_lens0 = np.array([2, 3]) - input1 = np.array([[11, 12, 13, 14, 15, 16], [16, 17, 18, 19, 20, - 21]]).astype('float32') - seq_lens1 = np.array([4, 5]) - batch_size, input0_seq_dim = input0.shape - input1_seq_dim = input1.shape[1] - - no_padding_seq_lens0 = np.array([input0_seq_dim] * batch_size) - no_padding_seq_lens1 = np.array([input1_seq_dim] * batch_size) - - self._ComputeFloatOutputAndVerify(input0, input1, no_padding_seq_lens0, - no_padding_seq_lens1, False) - self._ComputeFloatOutputAndVerify(input0, input1, no_padding_seq_lens0, - no_padding_seq_lens1, True) - - self._ComputeFloatOutputAndVerify(input0, input1, seq_lens0, seq_lens1, - False) - self._ComputeFloatOutputAndVerify(input0, input1, seq_lens0, seq_lens1, - True) - - -class RetryTest(test_utils.TestCase): - - def testRetry(self): - max_retries = 5 - - @py_utils.Retry(max_retries=max_retries) - def Foo(state): - tf.logging.error('foo retried %s', state) - state['count'] += 1 - raise ValueError('test') - - try: - state = {'count': 0, 'msg': 'test'} - Foo(state) - except Exception as e: # pylint: disable=broad-except - tf.logging.error('%s', e) - - self.assertEqual(1 + max_retries, state['count']) - - -class MixByWeightTest(test_utils.TestCase): - - def testMixByWeight(self): - var_a = tf.get_variable('a', trainable=False, initializer=0) - var_b = tf.get_variable('b', trainable=False, initializer=0) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - - def _AddFn(var): - return lambda: tf.assign_add(var, 1) - - op, _ = py_utils.MixByWeight([_AddFn(var_a), _AddFn(var_b)], [0.7, 0.3], - seed=12345) - for _ in range(100): - self.evaluate(op) - a, b = self.evaluate([var_a, var_b]) - self.assertEqual(100, a + b) - self.assertGreater(a, 50) - self.assertLess(b, 50) - - def testMixByWeightWithDynamicWeights(self): - var_a = tf.get_variable('a', trainable=False, initializer=0) - var_b = tf.get_variable('b', trainable=False, initializer=0) - var_w = tf.get_variable('w', trainable=False, dtype=tf.float32, shape=[2]) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - - def _AddFn(var): - return lambda: tf.assign_add(var, 1) - - op, _ = py_utils.MixByWeight([_AddFn(var_a), _AddFn(var_b)], var_w) - - # all weight goes to 'a' - self.evaluate([tf.assign(var_w, [1.0, 0.0])]) - for _ in range(10): - self.evaluate(op) - a, b = self.evaluate([var_a, var_b]) - self.assertEqual(10, a) - self.assertEqual(0, b) - - # all weight goes to 'b' - self.evaluate([tf.assign(var_w, [0.0, 1.0])]) - for _ in range(10): - self.evaluate(op) - a, b = self.evaluate([var_a, var_b]) - self.assertEqual(10, a) - self.assertEqual(10, b) - - def testMixByWeightAndBpropType(self): - var_a = tf.get_variable('a', trainable=False, initializer=0) - var_b = tf.get_variable('b', trainable=False, initializer=0) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - - def _AddFn(var): - return lambda: tf.assign_add(var, 1) - - op, bprop = py_utils.MixByWeight( - [_AddFn(var_a), _AddFn(var_b)], [1.0, 0.0]) - for _ in range(10): - self.evaluate(op) - bprop_v, a, b = self.evaluate([bprop, var_a, var_b]) - self.assertEqual(10, a) - self.assertEqual(0, b) - self.assertAllClose(np.array([1, 0]), np.squeeze(bprop_v)) - - op, bprop = py_utils.MixByWeight( - [_AddFn(var_a), _AddFn(var_b)], [0.0, 1.0]) - for _ in range(10): - self.evaluate(op) - bprop_v, a, b = self.evaluate([bprop, var_a, var_b]) - self.assertEqual(10, a) - self.assertEqual(10, b) - self.assertAllClose(np.array([0, 1]), np.squeeze(bprop_v)) - - -class SequencesToDebugStrings(test_utils.TestCase): - - def testSequencesToDebugStrings(self): - with self.session(): - self.assertAllEqual([b'[1 2 3]', b'[100 200]'], - py_utils.SequencesToDebugStrings( - tf.constant([[1, 2, 3], [100, 200, 300]], - dtype=tf.int32), - tf.constant([3, 2], dtype=tf.int32)).eval()) - - -class StepSeedTest(test_utils.TestCase): - - def _testStepSeedHelper(self, sess, step_fn, expected_starting_step_seed): - state0 = py_utils.NestedMap( - input=tf.constant(0, dtype=tf.int64), - seed_pair=tf.zeros(2, dtype=tf.int64)) - inputs = py_utils.NestedMap(input=tf.range(10, dtype=tf.int64)) - - p = base_layer.BaseLayer.Params().Set(name='test') - accumulated_states, _ = recurrent.Recurrent(p.Instantiate().theta, state0, - inputs, step_fn) - - self.evaluate(tf.global_variables_initializer()) - accumulated_states = accumulated_states.Pack( - self.evaluate(accumulated_states.Flatten())) - self.assertAllEqual(np.arange(10), accumulated_states.input) - expected_step_seeds = expected_starting_step_seed + np.arange(10) - self.assertAllEqual( - np.stack((np.zeros(10), expected_step_seeds), axis=1), - accumulated_states.seed_pair) - - def testStepSeed(self): - p = base_layer.BaseLayer.Params() - - def RecurrentStep(theta, unused_state0, inputs): - state1 = py_utils.NestedMap() - state1.input = inputs.input - state1.seed_pair = py_utils.GenerateStepSeedPair(p, theta.global_step) - return state1, py_utils.NestedMap() - - with self.session(graph=tf.Graph()) as sess: - self._testStepSeedHelper(sess, RecurrentStep, 0) - # Second recurrent inside the same graph has different step_seeds. - self._testStepSeedHelper(sess, RecurrentStep, 111107348) - - # After a reset, the step_seeds are the same even with a slightly - # different RecurrentStep function. - def RecurrentStep2(theta, state0, inputs): - with tf.control_dependencies([tf.no_op()]): - return RecurrentStep(theta, state0, inputs) - - with self.session(graph=tf.Graph()) as sess: - self._testStepSeedHelper(sess, RecurrentStep2, 0) - self._testStepSeedHelper(sess, RecurrentStep2, 111107348) - - with self.session(graph=tf.Graph()) as sess: - # But a different name_scope changes it. - with tf.name_scope('test'): - self._testStepSeedHelper(sess, RecurrentStep2, 0) - self._testStepSeedHelper(sess, RecurrentStep2, 1553244033) - - -class WeightParamsTest(test_utils.TestCase): - - def testShapeModification(self): - """Tests that WeightParams.shape can be modified.""" - pc = py_utils.WeightParams([20, 30], - py_utils.WeightInit.UniformPositive(1.0), - tf.float32) - pc.shape = [10, 30] - var = py_utils.CreateVariable('var', pc) - self.assertEqual(var.shape, [10, 30]) - - -class WeightInitTest(test_utils.TestCase): - - def testModification(self): - """Tests that WeightInit cannot be modified.""" - w_init = py_utils.WeightInit.UniformPositive(1.0) - with self.assertRaisesRegex(TypeError, 'immutable'): - w_init.scale = 2.0 - - def testUniformPositive(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(12345678) - pc = py_utils.WeightParams([20, 30], - py_utils.WeightInit.UniformPositive(1.0), - tf.float32) - var = py_utils.CreateVariable('var', pc) - self.evaluate(tf.global_variables_initializer()) - var_v = var.eval() - self.assertTrue(np.all(var_v >= 0.0)) - self.assertTrue(np.all(var_v <= 1.0)) - - def testKaimingUniformRelu(self): - with self.session(use_gpu=False, graph=tf.Graph()): - pc = py_utils.WeightParams( - [2, 10, 30], py_utils.WeightInit.KaimingUniformFanInRelu(1.0), - tf.float32) - var = py_utils.CreateVariable('var', pc) - self.evaluate(tf.global_variables_initializer()) - var_v = var.eval() - # With Relu initialization, uniform bounds are - # sqrt(3) * sqrt(2) / sqrt(fan_in) - bound = np.sqrt(3.) * np.sqrt(2.) / np.sqrt(20) - self.assertTrue(np.all(var_v >= -bound)) - self.assertTrue(np.all(var_v <= bound)) - - def testKaimingUniformLeakyRelu(self): - with self.session(use_gpu=False, graph=tf.Graph()): - pc = py_utils.WeightParams( - [2, 10, 30], py_utils.WeightInit.KaimingUniformFanInLeakyRelu(), - tf.float32) - var = py_utils.CreateVariable('var', pc) - self.evaluate(tf.global_variables_initializer()) - var_v = var.eval() - # With LeakyRelu initialization, uniform bounds are - # sqrt(3) * sqrt(2 / (1 + scale**2)) / sqrt(fan_in) - # - # scale = sqrt(5) by default. - bound = np.sqrt(3.) * np.sqrt(2. / 6.) / np.sqrt(20) - self.assertTrue(np.all(var_v >= -bound)) - self.assertTrue(np.all(var_v <= bound)) - - -class RNNCellStateInitTest(test_utils.TestCase): - - def testZeros(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(12345678) - zero_state = py_utils.InitRNNCellState( - [2, 3], init=py_utils.RNNCellStateInit.Zeros(), dtype=tf.float32) - self.evaluate(tf.global_variables_initializer()) - zero_state_v = zero_state.eval() - expected_zero_state = [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]] - self.assertAllClose(zero_state_v, expected_zero_state) - - def testRandomNormal(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(12345678) - zero_state = py_utils.InitRNNCellState( - [2, 3], - init=py_utils.RNNCellStateInit.RandomNormal(seed=12345), - dtype=tf.float32) - self.evaluate(tf.global_variables_initializer()) - zero_state_v = zero_state.eval() - expected_zero_state = [[1.621003, -1.097501, 0.493424], - [-1.048426, 2.73048, 0.091445]] - self.assertAllClose(zero_state_v, expected_zero_state) - - @flagsaver.flagsaver(stateless_vars_init=True) - def testRandomNormalStatelessVarsInit(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(12345678) - zero_state = py_utils.InitRNNCellState( - [2, 3], - name='RNNCellStateInit', - init=py_utils.RNNCellStateInit.RandomNormal(seed=12345), - dtype=tf.float32) - self.evaluate(tf.global_variables_initializer()) - zero_state_v = zero_state.eval() - expected_zero_state = [[-0.887855, 0.993745, -0.439152], - [0.312563, 0.923067, -1.952364]] - self.assertAllClose(zero_state_v, expected_zero_state) - zero_state_v_bis = zero_state.eval() - self.assertAllClose(zero_state_v_bis, expected_zero_state) - - def testRandomNormalInEval(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(12345678) - zero_state = py_utils.InitRNNCellState( - [2, 3], - init=py_utils.RNNCellStateInit.RandomNormal(seed=12345), - dtype=tf.float32, - is_eval=True) - self.evaluate(tf.global_variables_initializer()) - zero_state_v = zero_state.eval() - expected_zero_state = [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]] - self.assertAllClose(zero_state_v, expected_zero_state) - - -class RematerializeFnTest(test_utils.TestCase): - - def testRandomNormal(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(12345678) - a = tf.random.normal([2, 3]) - b = tf.random.normal([3, 4]) - - def Fn(a, b): - c = tf.matmul(a, b) - d = tf.nn.sigmoid(c) - e = tf.nn.tanh(c) - return d, e - - d1, e1 = Fn(a, b) - d2, e2 = py_utils.RematerializeFn(Fn, a, b) - self.assertEqual(d2.shape.as_list(), [2, 4]) - self.assertEqual(e2.shape.as_list(), [2, 4]) - da1, db1 = tf.gradients([d1, e1], [a, b]) - da2, db2 = tf.gradients([d2, e2], [a, b]) - self.evaluate(tf.global_variables_initializer()) - v1, v2, v3, v4 = self.evaluate([da1, db1, da2, db2]) - self.assertAllEqual(v1, v3) - self.assertAllEqual(v2, v4) - - -def WrapFunction(use_tf_function, *dtypes): - """Wrap a python function as a Defun or tf.function.""" - if use_tf_function: - - def Decorated(fn): - - @tf.function( - input_signature=[tf.TensorSpec(shape=None, dtype=t) for t in dtypes]) - def Fn(*args): - return fn(*args) - - return Fn.get_concrete_function() - - return Decorated - else: - return tf.Defun(*dtypes) - - -def StatefulRandomOpsInDefunTestParameters(test_fn): - - def WrappedTestFn(self, use_tf_function): - # TODO(laigd): remove this check when 312743821 is in the release. - if use_tf_function and tf.compat.v1.__version__ < '2.3.0': - return - test_fn(self, use_tf_function) - - decorator = parameterized.named_parameters( - ('_defun', False), - ('_function', True), - ) - return decorator(WrappedTestFn) - - -class StatefulRandomOpsInDefunTest(test_utils.TestCase, parameterized.TestCase): - - @StatefulRandomOpsInDefunTestParameters - def testFunctionWithStatelessOp(self, use_tf_function): - - @WrapFunction(use_tf_function) - def FunctionWithStatelessOp(): - return tf.constant(42.0) - - self.assertAllEqual( - [], py_utils.StatefulRandomOpsInDefun(FunctionWithStatelessOp)) - - @StatefulRandomOpsInDefunTestParameters - def testFunctionWithStatefulOp(self, use_tf_function): - - @WrapFunction(use_tf_function) - def FunctionWithStatefulOp(): - return tf.random.uniform([100], maxval=10, dtype=tf.int32) - - self.assertAllEqual( - ['RandomUniformInt'], - py_utils.StatefulRandomOpsInDefun(FunctionWithStatefulOp)) - - @StatefulRandomOpsInDefunTestParameters - def testFunctionWithStatelessFunctionCall(self, use_tf_function): - - @WrapFunction(use_tf_function) - def FunctionWithStatelessOp(): - return tf.constant(42.0) - - @WrapFunction(use_tf_function) - def FunctionWithStatelessFunctionCall(): - return FunctionWithStatelessOp() - - self.assertAllEqual( - [], - py_utils.StatefulRandomOpsInDefun(FunctionWithStatelessFunctionCall)) - - @StatefulRandomOpsInDefunTestParameters - def testFunctionWithStatefulFunctionCall(self, use_tf_function): - - @WrapFunction(use_tf_function) - def FunctionWithStatefulOp(): - return tf.random.uniform([100], maxval=10, dtype=tf.int32) - - @WrapFunction(use_tf_function) - def FunctionWithStatefulFunctionCall(): - return FunctionWithStatefulOp() - - self.assertAllEqual( - ['RandomUniformInt'], - py_utils.StatefulRandomOpsInDefun(FunctionWithStatefulFunctionCall)) - - @StatefulRandomOpsInDefunTestParameters - def testFunctionWithStatefulFunctionalWhile(self, use_tf_function): - - @WrapFunction(use_tf_function) - def FunctionWithStatefulFunctionalWhile(): - - @WrapFunction(use_tf_function, tf.float32, tf.int32) - def Cond(result, i): - del result - return tf.less(i, 4) - - @WrapFunction(use_tf_function, tf.float32, tf.int32) - def Body(result, i): - return (result + tf.random.uniform(tf.shape(result)), i + 1) - - return functional_ops.While([tf.zeros([2, 2]), 0], cond=Cond, body=Body) - - self.assertAllEqual( - ['RandomUniform'], - py_utils.StatefulRandomOpsInDefun(FunctionWithStatefulFunctionalWhile)) - - @StatefulRandomOpsInDefunTestParameters - def testFunctionWithStatefulFunctionalIf(self, use_tf_function): - - @WrapFunction(use_tf_function) - def FunctionWithStatefulFunctionalIf(): - - @WrapFunction(use_tf_function, tf.float32) - def ThenFn(x): - return tf.abs(x) - - @WrapFunction(use_tf_function, tf.float32) - def ElseFn(x): - return tf.random.uniform(tf.shape(x)) - - return functional_ops.If( - tf.greater(tf.eye(2), 0.5), [tf.eye(2)], ThenFn, ElseFn) - - self.assertAllEqual( - ['RandomUniform'], - py_utils.StatefulRandomOpsInDefun(FunctionWithStatefulFunctionalIf)) - - @StatefulRandomOpsInDefunTestParameters - def testFunctionWithStatefulFunctionalFor(self, use_tf_function): - - @WrapFunction(use_tf_function) - def FunctionWithStatefulFunctionalFor(): - - @WrapFunction(use_tf_function, tf.float32) - def Body(result): - return [ - result + tf.random.uniform(tf.shape(result)) + - tf.random.poisson(shape=tf.shape(result), lam=[0.5, 1.5]) - ] - - return functional_ops.For( - start=0, limit=4, delta=1, inputs=[tf.eye(2)], body=Body) - - self.assertAllEqual(['RandomPoissonV2', 'RandomUniform'], - sorted( - py_utils.StatefulRandomOpsInDefun( - FunctionWithStatefulFunctionalFor))) - - @StatefulRandomOpsInDefunTestParameters - def testFunctionWithStatelessFunctionalFor(self, use_tf_function): - - @WrapFunction(use_tf_function) - def FunctionWithStatelessFunctionalFor(): - - @WrapFunction(use_tf_function, tf.float32) - def Body(result): - return [ - result + - tf.random.stateless_normal(tf.shape(result), seed=tf.stack([0, 1])) - ] - - return functional_ops.For( - start=0, limit=4, delta=1, inputs=[tf.eye(2)], body=Body) - - self.assertAllEqual( - [], - py_utils.StatefulRandomOpsInDefun(FunctionWithStatelessFunctionalFor)) - - -class RecordFormatTest(test_utils.TestCase): - - def testRecordFormatFromFilePattern(self): - record_format, path = py_utils.RecordFormatFromFilePattern( - 'tfrecord:/path/to/bar') - self.assertEqual(record_format, 'tfrecord') - self.assertEqual(path, '/path/to/bar') - - record_format, path = py_utils.RecordFormatFromFilePattern( - 'custom:/path/to/baz') - self.assertEqual(record_format, 'custom') - self.assertEqual(path, '/path/to/baz') - - -class ReadFileLinesTest(test_utils.TestCase): - - def testReadFileLines(self): - contents = [ - 'hello', - 'world', - 'foo', - 'bar', - ] - outpath = os.path.join(tf.test.get_temp_dir(), 'test.txt') - with tf.io.gfile.GFile(outpath, 'w') as f: - f.write('\n'.join(contents)) - - lines = [line.strip() for line in py_utils.ReadFileLines(outpath)] - self.assertAllEqual(lines, contents) - - def testReadFilesLinesFromPackage(self): - # py_utils.py is at lingvo/core relative to the working - # directory, so it will load from the lingvo package instead - lines = py_utils.ReadFileLines('core/py_utils.py') - self.assertIsNotNone(lines) - - def testReadFileLinesWithInvalidFile(self): - path = os.path.join(tf.test.get_temp_dir(), 'fake.txt') - - with self.assertRaises(tf.errors.NotFoundError): - py_utils.ReadFileLines(path) - - -class FocalLossTest(test_utils.TestCase): - - def _testNpFL(self, logits, labels, alpha, gamma): - self.assertEqual(logits.shape, labels.shape) - shape = labels.shape - logits = logits.reshape([-1]) - labels = labels.reshape([-1]) - - def _Sigmoid(x): - return 1.0 / (1.0 + np.exp(-x)) - - def _CrossEntropy(prob, label): - if label > 0: - return -np.log(prob) - else: - return -np.log(1 - prob) - - probabilities = _Sigmoid(logits) - ans = np.empty(probabilities.shape) - for i, (l, p) in enumerate(zip(labels, probabilities)): - ce = _CrossEntropy(p, l) - pt = (l * p) + ((1 - l) * (1 - p)) - if alpha is not None: - ce *= (l * alpha) + ((1 - l) * (1 - alpha)) - if gamma is not None: - ce *= np.power(1 - pt, gamma) - ans[i] = ce - return ans.reshape(shape) - - def _testTfFL(self, logits, labels, alpha, gamma): - g = tf.Graph() - with g.as_default(): - x = tf.placeholder(tf.float32) - y = tf.placeholder(tf.float32) - z = py_utils.SigmoidCrossEntropyFocalLoss(x, y, alpha, gamma) - with self.session(graph=g) as sess: - return sess.run(z, feed_dict={x: logits, y: labels}) - - def testSigmoidCrossEntropyFocalLoss(self): - logits = np.random.normal(scale=10, size=(2, 3, 5)) - labels = np.floor(np.random.uniform(size=(2, 3, 5)) + 0.2) - for (alpha, gamma) in [(None, None), (0.25, 2), (0.1, 0), (1, 5)]: - np_result = self._testNpFL(logits, labels, alpha, gamma) - tf_lingvo_result = self._testTfFL(logits, labels, alpha, gamma) - self.assertAllClose(np_result, tf_lingvo_result) - - def _testNpSCEFL(self, logits, labels, alpha, gamma): - probs = np.exp(logits - np.max(logits, axis=-1, keepdims=True)) - probs = probs / np.sum(probs, axis=-1, keepdims=True) - - shape = probs.shape[:-1] - probs = probs.reshape([-1, probs.shape[-1]]) - ans = np.empty(probs.shape[:-1]) - - if labels.shape != logits.shape: - # convert labels to class probabilities - label_probs = np.zeros(probs.shape) - label_probs[np.arange(labels.size), labels.reshape([-1])] = 1.0 - else: - label_probs = labels.reshape([-1, labels.shape[-1]]) - for i, (lp, p) in enumerate(zip(label_probs, probs)): - ce = lp * -np.log(p) - if alpha is not None: - ce *= alpha - if gamma is not None: - ce *= np.power(1 - p, gamma) - ans[i] = ce.sum() - ans = ans.reshape(shape) - return ans - - def _testTfSCEFLLabelIds(self, logits, labels, alpha, gamma): - g = tf.Graph() - with g.as_default(): - x = tf.placeholder(tf.float32) - y = tf.placeholder(tf.int32) - z = py_utils.SoftmaxCrossEntropyFocalLoss( - x, label_ids=y, alpha=alpha, gamma=gamma) - with self.session(graph=g) as sess: - return sess.run(z, feed_dict={x: logits, y: labels}) - - def _testTfSCEFLLabelProbs(self, logits, labels, alpha, gamma): - g = tf.Graph() - with g.as_default(): - x = tf.placeholder(tf.float32) - y = tf.placeholder(tf.float32) - z = py_utils.SoftmaxCrossEntropyFocalLoss( - x, label_probs=y, alpha=alpha, gamma=gamma) - with self.session(graph=g) as sess: - return sess.run(z, feed_dict={x: logits, y: labels}) - - def testSoftmaxCrossEntropyFocalLoss(self): - num_classes = 7 - logits = np.random.normal(scale=10, size=(2, 3, 5, num_classes)) - label_ids = np.random.randint(num_classes, size=(2, 3, 5)) - label_probs = np.random.uniform(size=(2, 3, 5, num_classes)) - label_probs /= label_probs.sum(axis=-1, keepdims=True) - for (alpha, gamma) in [ - (None, None), - (np.random.uniform(size=[num_classes]).astype(np.float32), 2), - (np.random.uniform(size=[num_classes]).astype(np.float32), 0), - (np.random.uniform(size=[num_classes]).astype(np.float32), 5) - ]: - self.assertAllClose( - self._testNpSCEFL(logits, label_ids, alpha, gamma), - self._testTfSCEFLLabelIds(logits, label_ids, alpha, gamma)) - self.assertAllClose( - self._testNpSCEFL(logits, label_probs, alpha, gamma), - self._testTfSCEFLLabelProbs(logits, label_probs, alpha, gamma)) - - -class UniformSamplerTest(test_utils.TestCase): - - def testUniformSamplerSamples(self): - sampler = py_utils.UniformSampler(5) - for i in range(5): - sampler.Add(i) - # Up to the total number of samples, no sampling is performed. - self.assertEqual([0, 1, 2, 3, 4], sampler.samples) - - def testUniformSampler(self): - # Run a bunch of trials sampling 10 items out of 100 ids. - np.random.seed(123456) - state_space = 100 - num_samples = 10 - num_trials = 10000 - counts = np.zeros([state_space]) - for _ in range(num_trials): - sampler = py_utils.UniformSampler(num_samples) - # Add an element for each item in the state space. - for i in range(state_space): - sampler.Add(i) - samples = sampler.samples - self.assertEqual(num_samples, len(samples)) - for value in samples: - counts[value] += 1 - distribution = counts / np.sum(counts) - - # We expect that over the course of many trials, each item in - # the state space gets selected roughly an equal number of times, - # implying that the reservoir sampler is not biased based on the order - # in which items were added to the sampler. - self.assertGreater(min(distribution), 0.009) - self.assertLess(max(distribution), 0.011) - - -class FromGlobalTest(test_utils.TestCase): - - def testAccessAssertFlagWhenUnparsed(self): - tf.flags.FLAGS.unparse_flags() - # Accessing the flag value directly fails. - with self.assertRaises(tf.flags._exceptions.UnparsedFlagAccessError): - result = FLAGS.enable_asserts - result = py_utils._FromGlobal('enable_asserts') - # Default value of this flag is True. - self.assertTrue(result) - # Reparse args. - tf.flags.FLAGS(sys.argv) - - -def CallDefunTestParameters(test_fn): - - def WrappedTestFn(self, use_tf_function, bak_as_function): - # TODO(laigd): remove this check when 312743821 is in the release. - if use_tf_function and tf.compat.v1.__version__ < '2.3.0': - return - with flagsaver.flagsaver(call_defun_use_tf_function=use_tf_function): - test_fn(self, bak_as_function) - - decorator = parameterized.named_parameters( - ('_defun', False, False), - ('_defun_bakasfunction', False, True), - ('_function', True, False), - ('_function_bakasfunction', True, True), - ) - return decorator(WrappedTestFn) - - -class CallDefunTest(test_utils.TestCase, parameterized.TestCase): - - @CallDefunTestParameters - def testSimple(self, bak_as_function): - with self.session(): - - def Bak(xs, ys, dys): - del ys - w, x = xs - return (tf.matmul(dys, tf.transpose(x)) + 100., - tf.matmul(tf.transpose(w), dys) + 200.) - - def Fwd(args): - w, x = args - return tf.matmul(w, x) - - a = np.array([[1.0, 2.0], [0.0, -3.0]]) - b = np.array([[2.0, 0.0], [1.0, 1.0]]) - xs = [tf.constant(a), tf.constant(b)] - ys = py_utils.CallDefun(Fwd, xs, bak=Bak, bak_as_function=bak_as_function) - loss = tf.reduce_sum(tf.square(ys)) - dw, dx, dy = tf.gradients(xs=xs + [ys], ys=loss) - y, dw, dx, dy = self.evaluate([ys, dw, dx, dy]) - self.assertAllEqual(y, a.dot(b)) - self.assertAllEqual(dy, 2 * y) - self.assertAllEqual(dw, (2 * y).dot(b.T) + 100) - self.assertAllEqual(dx, a.T.dot(2 * y) + 200) - - @CallDefunTestParameters - def testPreserveStaticShape(self, bak_as_function): - with self.session(): - - def Bak(x, y, dy): - del x, y - return dy - - def Fwd(args): - x = args - shape = py_utils.GetShape(x) - if isinstance(shape, tf.Tensor): - return tf.ones_like(x) - else: - for dim in shape: - if isinstance(dim, tf.Tensor): - return tf.ones_like(x) + 1 - return tf.zeros_like(x) - - a = np.array([[1.0, 2.0], [0.0, -3.0]]) - x = tf.constant(a) - y = self.evaluate( - py_utils.CallDefun(Fwd, x, bak=Bak, bak_as_function=bak_as_function)) - self.assertAllEqual(y, np.zeros_like(a)) - - @CallDefunTestParameters - def testNestedMap(self, bak_as_function): - with self.session(): - - def Bak(xs, ys, dys): - del ys - return py_utils.NestedMap( - w=tf.matmul(dys.y, tf.transpose(xs.x)) + 100., - x=tf.matmul(tf.transpose(xs.w), dys.y) + 200.) - - def Fwd(xs): - return py_utils.NestedMap(y=tf.matmul(xs.w, xs.x)) - - a = np.array([[1.0, 2.0], [0.0, -3.0]]) - b = np.array([[2.0, 0.0], [1.0, 1.0]]) - xs = py_utils.NestedMap(w=tf.constant(a), x=tf.constant(b)) - ys = py_utils.CallDefun(Fwd, xs, bak=Bak, bak_as_function=bak_as_function) - loss = tf.reduce_sum(tf.square(ys.y)) - dw, dx, dy = tf.gradients(xs=xs.Flatten() + ys.Flatten(), ys=loss) - y, dw, dx, dy = self.evaluate([ys.y, dw, dx, dy]) - self.assertAllEqual(y, a.dot(b)) - self.assertAllEqual(dy, 2 * y) - self.assertAllEqual(dw, (2 * y).dot(b.T) + 100) - self.assertAllEqual(dx, a.T.dot(2 * y) + 200) - - @CallDefunTestParameters - def testImplicitInput(self, bak_as_function): - with self.session() as sess: - w = tf.placeholder(tf.float32) - - def Fwd(xs): - ret = py_utils.NestedMap(y=tf.matmul(w, xs.x)) - assert py_utils.GetExtraArgs() - return ret - - def Bak(xs, ys, dys): - del ys - dw = tf.matmul(dys.y, tf.transpose(xs.x)) + 100. - ret = py_utils.NestedMap(x=tf.matmul(tf.transpose(w), dys.y) + 200.) - if bak_as_function: - assert py_utils.GetExtraArgs() - return ret, dw - - a = np.array([[1.0, 2.0], [0.0, -3.0]]) - b = np.array([[2.0, 0.0], [1.0, 1.0]]) - xs = py_utils.NestedMap(x=tf.constant(b, dtype=tf.float32)) - ys = py_utils.CallDefun(Fwd, xs, bak=Bak, bak_as_function=bak_as_function) - loss = tf.reduce_sum(tf.square(ys.y)) - dw, dx, dy = tf.gradients(xs=[w] + xs.Flatten() + ys.Flatten(), ys=loss) - y, dw, dx, dy = sess.run([ys.y, dw, dx, dy], feed_dict={w: a}) - self.assertAllEqual(y, a.dot(b)) - self.assertAllEqual(dy, 2 * y) - self.assertAllEqual(dw, (2 * y).dot(b.T) + 100) - self.assertAllEqual(dx, a.T.dot(2 * y) + 200) - - -def IfTestParameters(test_fn): - - def WrappedTestFn(self, use_tf_function): - # TODO(laigd): remove this check when 313682500 is in the release. - if use_tf_function and tf.compat.v1.__version__ < '2.3.0': - return - with flagsaver.flagsaver(if_use_tf_function=use_tf_function): - test_fn(self) - - decorator = parameterized.named_parameters( - ('_defun', False), - ('_tf_function', True), - ) - return decorator(WrappedTestFn) - - -class IfTest(test_utils.TestCase, parameterized.TestCase): - - @IfTestParameters - def testNestedMapInput(self): - g = tf.Graph() - with g.as_default(): - - def ThenBody(nmap): - nmap.value -= 1. - return nmap - - def ElseBody(nmap): - nmap.value += 1. - return nmap - - inputs = py_utils.NestedMap(value=tf.constant(0.)) - true_out = py_utils.If(True, inputs, ThenBody, ElseBody) - false_out = py_utils.If(False, inputs, ThenBody, ElseBody) - - with self.session(graph=g): - true_out = self.evaluate(true_out) - false_out = self.evaluate(false_out) - - self.assertEqual(-1., true_out.value) - self.assertEqual(1., false_out.value) - - @IfTestParameters - def testScalarInput(self): - g = tf.Graph() - with g.as_default(): - - def ThenBody(value): - return value - 1. - - def ElseBody(value): - return value + 1. - - inputs = tf.constant(0.) - true_out = py_utils.If(True, inputs, ThenBody, ElseBody) - false_out = py_utils.If(False, inputs, ThenBody, ElseBody) - - with self.session(graph=g): - true_out = self.evaluate(true_out) - false_out = self.evaluate(false_out) - - self.assertEqual(-1., true_out) - self.assertEqual(1., false_out) - - @IfTestParameters - def testListInput(self): - g = tf.Graph() - with g.as_default(): - - def ThenBody(values): - return values[0] - 1., values[1] + 1. - - def ElseBody(values): - return values[0] + 1., values[1] - 1. - - inputs = [tf.constant(0.), tf.constant(0.)] - true_out = py_utils.If(True, inputs, ThenBody, ElseBody) - false_out = py_utils.If(False, inputs, ThenBody, ElseBody) - - with self.session(graph=g): - true_out = self.evaluate(true_out) - false_out = self.evaluate(false_out) - - self.assertEqual((-1., 1.), true_out) - self.assertEqual((1., -1.), false_out) - - -class ForLoopTest(test_utils.TestCase): - - def testSimple(self): - g = tf.Graph() - with g.as_default(): - - # Basel problem. \sum_1 1/i^2 = pi ^ 2 / 6. A slow convergent series. - def Body(i, state): - state.value = state.value + 1.0 / tf.square(tf.cast(i, tf.float32)) - return state - - state = py_utils.NestedMap(value=tf.constant(0.)) - state = py_utils.ForLoop(Body, 1, 10000, 1, state) - - with self.session(graph=g): - value = self.evaluate(state.value) - - self.assertAllClose(np.pi * np.pi / 6, value, rtol=1e-3) - - -class TopKTest(test_utils.TestCase): - - def test_top_2(self): - with self.session(): - x_in = tf.random.normal([4, 5, 6, 8]) - top2_value_a, top2_index_a = py_utils.TopK(x_in, 2) - top2_value_b, top2_index_b = tf.math.top_k(x_in, 2) - v1, v2 = self.evaluate([top2_value_a, top2_value_b]) - v3, v4 = self.evaluate([top2_index_a, top2_index_b]) - self.assertAllEqual(v1, v2) - self.assertAllEqual(v3, v4) - - def test_top_1(self): - with self.session(): - x_in = tf.random.normal([4, 5, 6, 8]) - top1_value_a, top1_index_a = py_utils.TopK(x_in, 1) - top1_value_b, top1_index_b = tf.math.top_k(x_in, 1) - v1, v2 = self.evaluate([top1_value_a, top1_value_b]) - v3, v4 = self.evaluate([top1_index_a, top1_index_b]) - self.assertAllEqual(v1, v2) - self.assertAllEqual(v3, v4) - - -class TpuSummaryTensorsTest(test_utils.TestCase): - - def testTpuSummaryTensors(self): - with self.session(): - with tf.name_scope('fprop'): - with tf.name_scope('tower_0_0'): - with tf.name_scope('fprop'): - with tf.name_scope('my_model'): - with tf.name_scope('layer_001'): - with tf.name_scope('fprop'): - x = tf.constant(0., name='inputs') - py_utils.AddTpuSummaryTensor('mean_x', tf.reduce_mean(x)) - with tf.name_scope('layer_002'): - with tf.name_scope('fprop'): - x = tf.identity(x, name='inputs') - py_utils.AddTpuSummaryTensor('mean_x', tf.reduce_mean(x)) - tpu_summary_tensors = py_utils.GetTpuSummaryTensors() - actual_value = self.evaluate([tpu_summary_tensors]) - expected_value = [{ - 'mean_x/fprop/tower_0_0/fprop/my_model/layer_001/fprop': (0., 1.), - 'mean_x/fprop/tower_0_0/fprop/my_model/layer_002/fprop': (0., 1.), - }] - self.assertAllEqual(expected_value, actual_value) - - -class HasShapeTest(test_utils.TestCase): - - def testFullyDynamicShapesMatchesOk(self): - x_pl = tf.placeholder(tf.float32) - y_pl = tf.placeholder(tf.float32) - x = py_utils.HasShape(x_pl, py_utils.GetShape(y_pl)) - with self.session() as sess: - sess.run( - x, - feed_dict={ - x_pl: np.random.rand(1, 2, 3), - y_pl: np.random.rand(1, 2, 3), - }) - - def testFullyDynamicShapesMismatchRaisesError(self): - x_pl = tf.placeholder(tf.float32) - y_pl = tf.placeholder(tf.float32) - x = py_utils.HasShape(x_pl, py_utils.GetShape(y_pl)) - with self.session() as sess: - with self.assertRaisesRegexp(tf.errors.InvalidArgumentError, - r'.*mismatch shape:.*'): - sess.run( - x, - feed_dict={ - x_pl: np.random.rand(1, 2, 3), - y_pl: np.random.rand(4, 5, 6), - }) - - def testFullyDynamicRankMismatchRaisesError(self): - x_pl = tf.placeholder(tf.float32) - y_pl = tf.placeholder(tf.float32) - x = py_utils.HasShape(x_pl, py_utils.GetShape(y_pl)) - with self.session() as sess: - with self.assertRaisesRegexp(tf.errors.InvalidArgumentError, - r'.*mismatch shape:.*'): - sess.run( - x, - feed_dict={ - x_pl: np.random.rand(1, 2), - y_pl: np.random.rand(4, 5, 6), - }) - - def testFullyConstantShapesMatchesOk(self): - x_pl = tf.placeholder(tf.float32) - x = py_utils.HasShape(x_pl, tf.constant([1, 2, -1])) - with self.session() as sess: - sess.run( - x, feed_dict={ - x_pl: np.random.rand(1, 2, 3), - }) - - def testFullyConstantShapesMismatchRaisesError(self): - x_pl = tf.placeholder(tf.float32) - x = py_utils.HasShape(x_pl, tf.constant([1, 2, -1])) - with self.session() as sess: - with self.assertRaisesRegexp(tf.errors.InvalidArgumentError, - r'.*mismatch shape:.*'): - sess.run( - x, feed_dict={ - x_pl: np.random.rand(2, 2, 3), - }) - - def testRankMismatchRaisesError(self): - with self.assertRaisesRegexp( - ValueError, r'Tensor does not match rank of expected shape.*'): - py_utils.HasShape(tf.random.uniform((1, 2, 3)), [1, 2]) - - def testTensorRankLessThanNDimsRaisesError(self): - with self.assertRaisesRegexp(ValueError, - r'Tensor has fewer dimensions than ndims.*'): - py_utils.HasShape(tf.random.uniform((1, 2, 3)), [1, 2, 3, 4], ndims=4) - - def testExpectedShapeRankLessThanNDimsRaisesError(self): - with self.assertRaisesRegexp( - ValueError, - r'Expected shape must have number of dimensions equal to ndims.*'): - py_utils.HasShape(tf.random.uniform((1, 2, 3, 4)), [1, 2, 3], ndims=4) - - def testTensorStaticShapeMismatchRaisesError(self): - x_pl = tf.placeholder(tf.float32, (None, 2, 3)) - y_pl = tf.placeholder(tf.float32, (3, 1, None)) - with self.assertRaisesRegexp( - ValueError, r'Tensor does not match expected shape on dimension 1.*'): - py_utils.HasShape(x_pl, py_utils.GetShape(y_pl)) - - def testTensorShapeMatchesOk(self): - x_pl = tf.placeholder(tf.float32, (None, 2, 3, None)) - y_pl = tf.placeholder(tf.float32, (3, 2, None, None)) - x = py_utils.HasShape(x_pl, py_utils.GetShape(y_pl)) - with self.session() as sess: - sess.run( - x, - feed_dict={ - x_pl: np.random.rand(3, 2, 3, 4), - y_pl: np.random.rand(3, 2, 3, 4), - }) - - def testTensorShapeMatchesWithMinus1Ok(self): - x_pl = tf.placeholder(tf.float32, (None, 2, 3, None)) - x = py_utils.HasShape(x_pl, [-1, -1, 3, -1]) - with self.session() as sess: - sess.run( - x, feed_dict={ - x_pl: np.random.rand(3, 2, 3, 4), - }) - - def testTensorShapeWithMinus1MismatchRaises(self): - x_pl = tf.placeholder(tf.float32, (None, 2, 3, None)) - x = py_utils.HasShape(x_pl, [-1, -1, 3, 5]) - with self.session() as sess: - with self.assertRaisesRegexp(tf.errors.InvalidArgumentError, - r'.*mismatch shape:.*'): - sess.run( - x, feed_dict={ - x_pl: np.random.rand(3, 2, 3, 4), - }) - - def testTensorShapeMatchesWithTensorExpectedShape(self): - x_pl = tf.placeholder(tf.float32, (None, 2, 3, None)) - x = py_utils.HasShape(x_pl, tf.constant([-1, -1, 3, -1])) - with self.session() as sess: - sess.run( - x, feed_dict={ - x_pl: np.random.rand(3, 2, 3, 4), - }) - - def testTensorShapeMismatchWithTensorExpectedShapeRaises(self): - x_pl = tf.placeholder(tf.float32, (None, 2, 3, None)) - x = py_utils.HasShape(x_pl, [-1, tf.constant(-1), 3, tf.constant(5)]) - with self.session() as sess: - with self.assertRaisesRegexp(tf.errors.InvalidArgumentError, - r'.*mismatch shape:.*'): - sess.run( - x, feed_dict={ - x_pl: np.random.rand(3, 2, 3, 4), - }) - - def testTensorStaticShapeMatchDynamicMismatchRaises(self): - x_pl = tf.placeholder(tf.float32, (None, None, 2, 3, None)) - y_pl = tf.placeholder(tf.float32, (None, 3, 2, None, None)) - x = py_utils.HasShape(x_pl, py_utils.GetShape(y_pl)) - with self.session() as sess: - with self.assertRaisesRegexp(tf.errors.InvalidArgumentError, - r'.*mismatch shape:.*'): - sess.run( - x, - feed_dict={ - x_pl: np.random.rand(1, 3, 2, 3, 4), - y_pl: np.random.rand(2, 3, 2, 5, 4), - }) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/quant_utils.py b/lingvo/core/quant_utils.py deleted file mode 100644 index 428fef70..00000000 --- a/lingvo/core/quant_utils.py +++ /dev/null @@ -1,1186 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Utilities for model quantization.""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import hyperparams -from lingvo.core import py_utils -from lingvo.core import summary_utils -import numpy as np - - -class QuantizableLayer(base_layer.BaseLayer): - """A layer that supports various forms of quantization. - - It is always safe to extend QuantizableLayer instead of BaseLayer (i.e. at - the base of layer inheritance hierarchies) if any layer in the hierarchy - may be quantized. Unless if configured/used, all quantization behavior - is disabled by default. - - Most quantization strategies employed at training time fall into the - "fake quantization" category, where we add various constraints in the - forward propagation to quantify and simulate the effect of quantization. - Within that, we have two major approaches: - - - Active clipping: Usually via a schedule, tensors are actively - clipped to fall into ranges that we know apriori that the model should - be able to deal with. - - Passive tracking and simulation: Passively track the min/max ranges - of tensors and insert special ops at training and eval time that - constrain to those ranges. - - The tensors of interest for both approaches are top-level inputs (or - embeddings), outputs of arithmetic operations (add, mul, tanh, etc) and - weights. While the actual process of quantizing can be quite complex and - involve an end to end view of the system, from a modeling perspective, it - can be thought of as providing tags/decorators to arithmetic inputs/outputs. - It would be appropriate to think of these as casts which alter the way that - the arithmetic operation is tracked and quantize (if Python/Tensorflow were - a more strongly typed environment, they would indeed represent types in the - type system but given the loose typing, it is just an honor system). - - The "decorators" are: - - - QWeight: Tags a tensor (typically a var) as a weight quantized type. - - QR* (QRTanh, QRSigmoid, QRSoftmax, etc): Tags a tensor as the result - of a fixed activation function with a known output range (the range - is implied in the name). - - QRPadding: Tags a tensor as containing a padding value (as we define - them as 0..1). While such values are numeric, they generally exist with - very different ranges from the rest of the graph and should not be - arithmetically combined with tensors that may have a different/variable - range. - - QTensor: Tags a tensor as a generic quantized intermediate value. - These are also tagged with a layer-unique name. All QTensors with the - same name will be considered the same from a numerical range/precision - perspective. - - Tagging things in this way allows us to, via hyperparameters, associate - one or more quantization domains (QDomain) with the layer that will - actually perform the necessary tracking and transformations needed at - training and inference time to ensure that the layer can operate in low - bit inference engines that only have quantized numeric representations. - See the SampleQuantizedProjectionLayer in the unit test for an example layer - that has had these tags applied. - - As a note on terminology, domain/QDomain here refers to a distinct set of - quantization rules to apply to a subset of tensors. Most layers will only - have one QDomain (default). The concept exists for layers which have been - specially created to operate in more than one quantized precision (i.e. an - RNN cell that uses 8bit quantization for inputs/outputs and 16bit - quantization for internal state arithmetic). Such uses should be rare. - - - **Convenience functions:** - - The layer adds a number of convenience functions to the layer's 'fns' - function library. These mirror similarly named functions in TensorFlow but - automatically add the necessary annotations. All such functions take the - following named parameters: - - - qt: Name of QTensor (setup with TrackQTensor) for dynamic range tracking. - - qmin/qmax/qdomain: Constant min/max range plus optional QDomain name to - resolve against. Typically, only qmin/qmax are used. - - Functions that have a natural output range will have default values for - qmin/qmax so that they just work. Functions that do not have a natural - output range must have either qt or qmin/qmax specified manually. - - Natural range functions - - - qtanh - - qsigmoid - - qsoftmax - - Dynamic range functions: - - - qadd - - qmultiply - - qmatmul (defers to `.py_utils.Matmul` and only accepts rank-2 tensors) - - qbatchmatmul (defers to `tf.matmul` directly) - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('qdomain', hyperparams.Params(), - 'Container for quantization domains.') - p.qdomain.Define('default', None, 'Default quantization domain.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - self._tracked_tensors = dict() # tracked t_name -> (QDomain) - self._qstate = None # t_name -> Tensor - - # Instantiate quantization domains. - self._qdomains = dict() # Dict of qdname -> QDomain or None - for qdname in dir(p.qdomain): - qdparams = p.qdomain.Get(qdname) - if qdparams is None: - continue - assert issubclass( - qdparams.cls, - QDomain), ('Expected quantized domain %s to extend QDomain' % qdname) - qdchild_name = 'qdomain_' + qdname - self.CreateChild(qdchild_name, qdparams) - self._qdomains[qdname] = self.children[qdchild_name] - self._AddQuantizationFunctions() - - def _CreateChildrenVariables(self): - # Backwards compatibility: child.InstantiateVariables() in custom scope. - p = self.params - with tf.variable_scope(p.name + '/q'): - for qdomain in self._qdomains.values(): - qdomain.InstantiateVariables() - super()._CreateChildrenVariables() - - def QRTanh(self, t, domain='actf'): - """Quantizes the output of a tanh (-1.0, 1.0).""" - qd = self.GetQDomain(domain) - return qd.QuantizeNaturalRange(t, -1.0, 1.0) if qd else t - - def QRSigmoid(self, t, domain='actf'): - """Quantizes the output of a sigmoid (0, 1.0).""" - qd = self.GetQDomain(domain) - return qd.QuantizeNaturalRange(t, 0.0, 1.0) if qd else t - - def QRSoftmax(self, t, domain='softmax', narrow_to_asym_bit_depth=False): - """Quantizes the output of a softmax (0, 1.0 - 1.0/2^-bits).""" - qd = self.GetQDomain(domain) - # Override based on TFLite softmax support. - softmax_max = 1.0 - if qd is not None and narrow_to_asym_bit_depth: - softmax_max = (2**qd.bits - 1) / (2**qd.bits) - return qd.QuantizeNaturalRange(t, 0.0, softmax_max) if qd else t - - def QRRelu(self, t, domain='relu'): - """Quantizes the output of a relu (0, 1.0).""" - qd = self.GetQDomain(domain) - return qd.QuantizeNaturalRange(t, 0.0, 1.0) if qd else t - - def QRRelu6(self, t, domain='relu6'): - """Quantizes the output of a relu6 (0, 6.0).""" - qd = self.GetQDomain(domain) - return qd.QuantizeNaturalRange(t, 0.0, 6.0) if qd else t - - def QRPadding(self, t, domain='padding'): - """Quantizes the padding.""" - qd = self.GetQDomain(domain) - return qd.QuantizeConstantRange(t, 0.0, 1.0) if qd else t - - def TrackQTensor(self, *t_names, **kwargs): - r"""Creates one or more QTensors for later use. - - Any tensor that will later be quantized must be created first, preferably - in _CreateLayerVariables(). - - Along with a list of tensor names to create, they can be associated with - a 'domain'. Most layers are simple enough to only have a single quantization - domain (QDomain), typically 'default'. However, additional QDomains can - be defined as parameters to control fine grained aspects of quantization. - - If no explicit domain is passed, then the domain ('tensor\_' + t_name) is - tried. If that is not defined, then 'default'. - - Args: - *t_names: Positional parameters are taken to be QTensor names to create. - **kwargs: Can contain an explicit 'domain'. Written this way due to - python2 limitations. - """ - domain_override = kwargs['domain'] if 'domain' in kwargs else None - for t_name in t_names: - domain = domain_override - if domain is None: - domain = 'tensor_' + t_name - qd = self.GetQDomain(domain) - self._tracked_tensors[t_name] = qd - if qd: - qd.CreateTensor(t_name) - - def QTensor(self, t_name, t, eval_only=False): - """Quantizes a general tensor input/output in one step. - - t_name must have been previously created via TrackQTensor. - - Args: - t_name: Previously created QTensor t_name to quantize to. - t: Tensor to quantize. - eval_only: Whether to only apply quantization pressure at eval time. - Returns: - The tensor, quantized. - """ - return self.QTensorMulti(t_name, t, eval_only=eval_only)[0] - - def QTensorMulti(self, t_name, *ts, **kwargs): - """Quantizes multiple tensors simultaneously. - - t_name must have been previously created via TrackQTensor. - - This is different from multiple calls to QTensor because each of the - tensors will contribute to the min/max of the same constraint. - Typically used for tensors that are being added together. - - Args: - t_name: Previously created QTensor t_name to quantize to. - *ts: Tensor to quantize. - **kwargs: Additional kwargs as per QTensor. - Returns: - Tuple of quantized tensors. - """ - assert t_name in self._tracked_tensors, ( - ('Call to QTensor without first calling TrackQTensor: %s ' - '(all known = %r)') % (t_name, list(self._tracked_tensors.keys()))) - eval_only = kwargs['eval_only'] if 'eval_only' in kwargs else False - qd = self._tracked_tensors[t_name] - if not qd: - return ts - return qd.QuantizeTensors(t_name, ts, eval_only=eval_only) - - def GetQTensorRange(self, t_name, ts): - """Returns the range for a quantized tensor. - - t_name must have been previously create via TrackQTensor and t should be - previously quantized. - - Args: - t_name: Preivously created QTensor t_name to fetch range from. - t: Tensor to retrieve range from. - - Returns: - The (min, max) range of the quantized tensor. - """ - qd = self._tracked_tensors[t_name] - return qd.GetTensorRange(t_name, ts) - - def QWeight(self, w, domain='weight'): - """Quantizes a weight. - - Args: - w: The weight tensor. - domain: Custom domain to match (defaults to 'weight' or 'default'). - Returns: - The weights quantized. - """ - qd = self.GetQDomain(domain) - return qd.QuantizeWeight(w) if qd else w - - def GetQDomain(self, domain): - """Gets the QDomain matching a given domain name. - - Args: - domain: User specified domain name. - Returns: - The requested QDomain, the 'default' QDomain or None. - """ - qd = self._qdomains.get(domain) - if qd: - return qd - qd = self._qdomains.get('default') - return qd - - def _AddQuantizationFunctions(self): - """Adds standard quantization functions against the given layer.""" - - def WrapOp(fnname, f, default_qmin=None, default_qmax=None): - """Adds a wrapper op to the layer's fns.""" - - def Wrapped(*args, **kwargs): - """Wraps a native op.""" - # Validate and pop args 'qt', 'qmin', 'qmax' and 'qdomain'. - qt = kwargs.get('qt') - if qt is not None: - del kwargs['qt'] - qmin = kwargs.get('qmin') - if qmin is not None: - del kwargs['qmin'] - qmax = kwargs.get('qmax') - if qmax is not None: - del kwargs['qmax'] - qdomain = kwargs.get('qdomain') - if qdomain is not None: - del kwargs['qdomain'] - narrow_to_asym_bit_depth = kwargs.get('narrow_to_asym_bit_depth') - if narrow_to_asym_bit_depth is not None: - del kwargs['narrow_to_asym_bit_depth'] - if qmin is None: - qmin = default_qmin - if qmax is None: - qmax = default_qmax - assert qt is not None or (qmin is not None and qmax is not None), ( - ('Quantized function "%s" requires either qt (QTensor name) or ' - 'qmin/qmax to be set.') % fnname) - - # Provide a better default name if none provided. - if 'name' not in kwargs and qt is not None: - kwargs['name'] = '%s_%s' % (fnname, qt) - - # Invoke original. - y = f(*args, **kwargs) - - # Handle the output. - if qt is not None: - y = self.QTensor(qt, y) - else: - qd = self.GetQDomain(qdomain) - if qd: - if narrow_to_asym_bit_depth: - qrange = qmax - qmin - qmax = qmin + qrange * (2**qd.bits - 1) / (2**qd.bits) - y = qd.QuantizeNaturalRange(y, qmin, qmax) - return y - - self.AddFunction(fnname, Wrapped) - - # Supported quantized functions. - WrapOp('qadd', tf.add) - WrapOp('qsubtract', tf.subtract) - WrapOp('qmultiply', tf.multiply) - WrapOp('qmatmul', py_utils.Matmul) - WrapOp('qbatchmatmul', tf.matmul) - WrapOp('qconv1d', tf.nn.conv1d) - WrapOp('qtanh', tf.tanh, default_qmin=-1.0, default_qmax=1.0) - WrapOp('qsigmoid', tf.sigmoid, default_qmin=0.0, default_qmax=1.0) - WrapOp('qsoftmax', tf.nn.softmax, default_qmin=0.0, default_qmax=1.0) - WrapOp('qlogsoftmax', tf.nn.log_softmax) - WrapOp('qrelu', tf.nn.relu) - WrapOp('qrelu6', tf.nn.relu6, default_qmin=0.0, default_qmax=6.0) - WrapOp( - 'qrandom_uniform', - tf.random.uniform, - default_qmin=0.0, - default_qmax=1.0) - - # Convenience for quantizing weights. - self.AddFunction('qweight', self.QWeight) - - -class BaseClippingCapSchedule(base_layer.BaseLayer): - """Base class for clipping cap schedules.""" - - def __init__(self, params): - super().__init__(params) - - @property - def is_quantized(self): - return False - - def GetEndRange(self): - """Public method to get the final range as a constant. - - Note that this returns the "ideal" end range (i.e. -1..1) as opposed to - the actual range, which has its upper bound slightly adjusted based on - the bit depth of the quantized type. In this sense, this value is a lie, - but it is a consistent lie that can be corrected for downstream by the - inference engine once it has inferred the actual quantized types being - used. - - Note that this also assumes the default start/end caps. Some internal - parts may use altered caps or bit depths. - - Returns: - Tuple of (min, max) for the final range. - """ - raise NotImplementedError('Abstract Method: GetEndRange') - - def GetQuantizedEndRange(self): - """Gets the quantized ending range. - - Unlike GetEndRange(), this takes quantization effects into account. - The default implementation just returns self.GetEndRange(). Subclasses - can include additional keyword arguments, tightly coupling them to callers - of specific types. - - Returns: - Tuple of (min, max) for the final range. - """ - assert not self.is_quantized - return self.GetEndRange() - - def ApplyConstantClip(self, x, min_value, max_value): - """Applies a constant clip with the clipping op for the implementation. - - This is a special case which allows applying a custom clipping range to - constants that are not used arithmetically. This exists to support padding. - - Args: - x: Tensor to clip. - min_value: Minimum value. - max_value: Maximum value. - Returns: - Tensor clipped. - """ - raise NotImplementedError('Abstract method: ApplyConstantClip') - - def GetState(self, theta): - """Gets a state tensor that can be used to calculate clipping. - - The state will be a float32 tensor that is safe to pass to TF functions. - - Args: - theta: Layer theta. - Returns: - An opaque tensor to be passed to ApplyClippingWithState(). - """ - raise NotImplementedError('Abstract method: GetState') - - def ApplyClipping(self, theta, x, **kwargs): - """Applies clipping to x. - - Args: - theta: Layer theta. - x: Input tensor to clip. - **kwargs: Additional implementation specific kwargs. - Returns: - Clipped (or identity) x. - """ - return self.ApplyClippingWithState(self.GetState(theta), x, **kwargs) - - def ApplyClippingWithState(self, state, x): - """Applies clipping to x. - - Args: - state: A previously obtained value of GetState(). - x: Input tensor to clip. - Returns: - Clipped (or identity) x. - """ - raise NotImplementedError('Abstract Method: ApplyClippingWithState') - - -class IdentityClippingCapSchedule(BaseClippingCapSchedule): - """Dummy cc schedule (useful in some cases instead of None).""" - - def GetEndRange(self): - np_dtype = self.params.dtype.as_numpy_dtype - np_info = np.finfo(np_dtype) - return (np_info.min, np_info.max) - - def ApplyConstantClip(self, x, min_value, max_value): - return x - - def GetState(self, theta): - return tf.zeros([1], tf.float32) - - def ApplyClippingWithState(self, state, x): - return x - - -class LinearClippingCapSchedule(BaseClippingCapSchedule): - """Class for linear clipping cap decay.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('start_step', 0, - 'We start gradually narrowing clipping cap from start_step.') - p.Define('end_step', 15000, - 'We reach end_cap by end_step.') - p.Define('start_cap', 8.0, - 'We gradually narrow the clipping range over the course of ' - 'training. This is the clipping range we apply when training ' - 'starts.') - p.Define('end_cap', 1.0, - 'We gradually narrow the clipping range over the course of ' - 'training. This is the clipping range we apply towards the end ' - 'of training.') - p.name = 'CCSchedule' - return p - - @property - def is_quantized(self): - return False - - def ApplyConstantClip(self, x, min_value, max_value): - return tf.clip_by_value(x, min_value, max_value) - - def GetState(self, theta): - return self._Value(theta.global_step) - - def ApplyClippingWithState(self, state, x): - """Applies clipping to x. - - Args: - state: Clipping state. - x: Input tensor to clip. - Returns: - Clipped (or identity) x. - """ - cap = tf.cast(state, x.dtype) - return tf.clip_by_value(x, -cap, cap) - - def GetEndRange(self): - """Returns the range of values that are clipped towards the end of training. - - This is always a constant and is used by downstream systems. - - Returns: - Tuple of (min, max). - """ - return (-self.params.end_cap, self.params.end_cap) - - def _Value(self, current_step): - """Returns the current clipping cap.""" - p = self.params - start_step = tf.cast(p.start_step, tf.float32) - end_step = tf.cast(p.end_step, tf.float32) - current_step = tf.cast(current_step, tf.float32) - steps_ratio = ( - tf.minimum(end_step - start_step, current_step - start_step)/ - (end_step - start_step)) - rmax_tensor = ( - steps_ratio * p.end_cap + (1.0 - steps_ratio) * p.start_cap) - return tf.cond( - tf.less(current_step, - p.start_step), lambda: tf.cast(p.start_cap, tf.float32), - lambda: tf.cast(rmax_tensor, tf.float32)) - - def PostTrainingStepUpdate(self, global_step): - summary_utils.scalar('cap', self._Value(global_step)) - return tf.no_op() - - -class FakeQuantizationSchedule(BaseClippingCapSchedule): - """Manages application of fake quantization via a schedule. - - This implementation is a general-purpose clipping cap schedule but also - works with the Fake Quantization approach used by mobile inference engines. - It is tightly coupled to the FakeQuantizedLSTMCell. See more exhaustive - documentation and links there. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'FQSchedule' - p.Define('clip_start_step', 0, - 'We start gradually narrowing clipping cap from start_step.') - p.Define('clip_end_step', 15000, 'We reach end_cap by end_step.') - p.Define('quant_start_step', 15000, - 'Step at which we begin to apply quantization.') - p.Define('start_cap', 8.0, 'Default clipping/quant start cap.') - p.Define('end_cap', 1.0, 'Default clipping/quant end cap.') - p.Define('bits', 8, 'Default quantized bit depth.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - # We may relax this constraint at some point to allow gradual quantization - # but enforce for now as it is easy to mess up and we have not evaluated - # how it would work otherwise. - assert p.quant_start_step >= p.clip_end_step, ( - 'quant_start_step must be >= clip_end_step') - - @property - def is_quantized(self): - return True - - @property - def bits(self): - p = self.params - return p.bits - - def GetEndRange(self): - """Public method to get the final range as a constant. - - Note that this returns the "ideal" end range (i.e. -1..1) as opposed to - the actual range, which has its upper bound slightly adjusted based on - the bit depth of the quantized type. In this sense, this value is a lie, - but it is a consistent lie that can be corrected for downstream by the - inference engine once it has inferred the actual quantized types being - used. - - Note that this also assumes the default start/end caps. Some internal - parts may use altered caps or bit depths. - - Returns: - Tuple of (min, max) for the final range. - """ - p = self.params - return (-p.end_cap, p.end_cap) - - def GetQuantizedEndRange(self, end_cap=None, bits=None): - """Gets the quantized ending range. - - Unlike GetEndRange(), this takes quantization effects into account. - - Args: - end_cap: Override end_cap value. - bits: Override bits value. - Returns: - Tuple of (min, max) for the final range. - """ - p = self.params - if end_cap is None: - end_cap = p.end_cap - if bits is None: - bits = p.bits - return self._GetQuantizedRangeForCap(end_cap, bits) - - def ApplyConstantClip(self, x, min_value, max_value): - return tf.quantization.fake_quant_with_min_max_vars( - x, min_value, max_value, num_bits=self.params.bits) - - def GetState(self, theta): - """Gets the state from theta.""" - p = self.params - if p.is_inference: - # State is not used for inference. Just return dummy. - return tf.zeros([1], tf.float32) - else: - # Calculations/vars need to be float but these can be ints in the params. - clip_end_step = tf.cast(p.clip_end_step, tf.float32) - clip_start_step = tf.cast(p.clip_start_step, tf.float32) - quant_start_step = tf.cast(p.quant_start_step, tf.float32) - global_step = tf.cast(theta.global_step, tf.float32) - - # Will be negative if before clipping starts. - clip_ratio = ( - tf.minimum(clip_end_step - clip_start_step, - global_step - clip_start_step) / - tf.maximum(1.0, clip_end_step - clip_start_step)) - # Currently fq is either on (1.0) or off (-1.0). Progressive quantization - # may later occupy 0..1.0. - fq_ratio = tf.where(global_step < quant_start_step, -1.0, 1.0) - - return tf.stack([clip_ratio, fq_ratio]) - - def _GetQuantizedRangeForCap(self, current_cap, bits): - """Gets the range for the given cap and number of bits. - - Args: - current_cap: Cap to compute against. - bits: Number of bits (8, 16, etc). - Returns: - If current_cap is a python float, the result will be a float. If a Tensor - scalar, then a Tensor scalar. - """ - dt_max = 2**(bits - 1) # i.e. 8bit = 128, 16bit = 32768 - return -current_cap, current_cap * (dt_max - 1) / dt_max - - def _GetCurrentMinMax(self, - state, - start_cap, - end_cap, - bits, - fixate_to_end_state=False): - """Gets the current min/max for the bit depth and caps. - - Args: - state: Clipping state. - start_cap: Starting cap. - end_cap: Ending cap once clipping saturates. - bits: Number of bits of the quantized datatype. - fixate_to_end_state: Whether to fixate the cap to the end state. - Returns: - (min_value, max_value) as python scalars or 0D Tensors ( - if not fixate_to_end_state). - """ - if fixate_to_end_state: - current_cap = end_cap - else: - clip_ratio = state[0] if not fixate_to_end_state else 1.0 - current_cap = clip_ratio * end_cap + (1.0 - clip_ratio) * start_cap - return self._GetQuantizedRangeForCap(current_cap, bits) - - def ApplyClippingWithState(self, - state, - x, - start_cap=None, - end_cap=None, - bits=None): - """Applies clipping. - - The start_cap, end_cap and bits can be set explicitly and take the default - if None. - - Args: - state: Clipping state. - x: Tensor to clip. - start_cap: Clipping value at the start of the ramp. - end_cap: Clipping value at the end of the ramp. - bits: Number of bits to quantize to. - Returns: - x with clipping applied. - """ - p = self.params - if start_cap is None: - start_cap = p.start_cap - if end_cap is None: - end_cap = p.end_cap - if bits is None: - bits = p.bits - if p.is_inference: - # For inference, we assume that both clipping and quantization have - # saturated and just output a saturated quant op. - min_value, max_value = self._GetCurrentMinMax( - state, start_cap, end_cap, bits, fixate_to_end_state=True) - # Note that the inference version uses the *_args variant, which requires - # constants for min/max. The _GetCurrentMinMax will return (python) - # constants if fixating. This is fragile but works around a Toco bug - # if trying to run on the *_vars form because it can't seem to read - # 0D tensors. This form has the benefit of blowing up at export time - # if the min/max aren't constant. - return _CopyShape( - x, - tf.quantization.fake_quant_with_min_max_args( - x, min_value, max_value, num_bits=bits)) - - # Non-inference. - def Clipped(): - clip_ratio = state[0] - min_value, max_value = self._GetCurrentMinMax(state, start_cap, end_cap, - bits) - min_value = tf.stop_gradient(min_value) - max_value = tf.stop_gradient(max_value) - return tf.where(clip_ratio >= 0.0, - (lambda: tf.clip_by_value(x, min_value, max_value))(), - (lambda: x)()) - - def Quantized(): - min_value, max_value = self._GetCurrentMinMax(state, start_cap, end_cap, - bits) - min_value = tf.stop_gradient(min_value) - max_value = tf.stop_gradient(max_value) - return tf.quantization.fake_quant_with_min_max_vars( - x, min_value, max_value, num_bits=bits) - - # Quantization will implicitly clip, so if we are in the quant phase, just - # do that. Otherwise, clip (which will return identity if not in that - # phase yet). - fq_ratio = state[1] - # return _CopyShape(x, Clipped()) - return _CopyShape(x, tf.where(fq_ratio <= 0.0, Clipped(), Quantized())) - - -class QDomain(base_layer.BaseLayer): - """Base class for a quantization domain layer. - - This implementation doubles as a no-op quantization domain. - """ - - @property - def bits(self): - """Retrieves the bits used by this quantization layer. - - Returns: - The number of bits available to this qdomain or None if unquantized. - """ - return None - - def QuantizeWeight(self, w): - """Quantizes a weight. - - Args: - w: Weight tensor to quantize. - Returns: - Quantized weight. - """ - return w - - def QuantizeConstantRange(self, t, min_value, max_value): - """Quantizes a true-constant range that is not used for arithmetic. - - This supports special values like padding that should have a precise - range that we do not deviate from. - - Args: - t: Tensor to quantize. - min_value: Min of the range. - max_value: Max of the range. - Returns: - Quantized tensor. - """ - return t - - def QuantizeNaturalRange(self, t, min_value, max_value): - """Quantizes a tensor with a known, natural range. - - Args: - t: Tensor to quantize. - min_value: Min value of the range. - max_value: Max value of the range. - Returns: - Quantized tensor. - """ - return t - - def CreateTensor(self, t_name): - """Creates a QTensor with t_name. - - Args: - t_name: Unique name (within layer) for this tensor. - """ - pass - - def QuantizeTensors(self, t_name, ts, eval_only=False): - """Quantizes a tensor with t_name previously created with CreateTensor. - - If applicable, each of the passed tensors contributes to a shared - range. - - Args: - t_name: Tensor name. - ts: List of tensors to quantize. - eval_only: Whether to only apply quantization pressure at eval time. - Returns: - Quantized tensors. - """ - return ts - - def GetTensorRange(self, t_name, ts): - """Retrieves the range of a tensor given the t_name used by CreateTensor. - - Note, this computes the batch range across the list of tensors at training - time but fetches the stored tensor over time. This depends on - QuantizeTensors updating the appropriate value. - - Args: - t_name: Tensor name. - ts: Tensor to determine the range for. - - Returns: - A min-max pair that represents the tensor range. - """ - raise NotImplementedError('Abstract method: NormalizeTensors') - - -class SymmetricScheduledClipQDomain(QDomain): - """A quantization domain that does symmetric scheduled clipping. - - This contains a BaseClippingCapSchedule which handles the actual clipping. It - defaults to a FakeQuantizationSchedule. - - This clipping domain will aid in quantizing layers that are known to tolerate - operation within known ranges (such as LSTM cells). The clipping range will - converge over a range of steps and is setup to match ideal, symmetric ranges - for quantized types. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('cc_schedule', FakeQuantizationSchedule.Params(), - 'Quantization clipping schedule.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - self.CreateChild('cc_schedule', p.cc_schedule) - - @property - def bits(self): - return self.cc_schedule.bits - - def QuantizeWeight(self, w): - return self.cc_schedule.ApplyClipping(self.theta.cc_schedule, w) - - def QuantizeNaturalRange(self, t, min_value, max_value): - # Note: We apply the scheduled clip here, completely overriding the - # known natural range. This is intentional and assumes that when this - # layer is used for symmetric clipping, it is applied uniformly to all - # active elements. - return self.cc_schedule.ApplyClipping(self.theta.cc_schedule, t) - - def QuantizeConstantRange(self, t, min_value, max_value): - # Constant ranges, such as padding are handled separately. They are merely - # constrained to the given range and assumed to be quantizable as-is. - # This is used for padding. - return tf.clip_by_value(t, min_value, max_value) - - def QuantizeTensors(self, t_name, ts, eval_only=False): - if eval_only and not self.do_eval: - return ts - else: - return [ - self.cc_schedule.ApplyClipping(self.theta.cc_schedule, t) for t in ts - ] - - -class _CountedMinMaxAccumulator(base_layer.Accumulator): - """Accumulator for a counted min/max. - - Represented as a tensor of shape [count, min, max]. Every update - increases the count and expands the min/max (initially zeros). - """ - - def __init__(self, dtype): - super().__init__() - self.dtype = dtype - - def DefaultValue(self): - return tf.zeros([3], dtype=self.dtype, name='qstate_zero') - - def Update(self, new_value): - state0 = self.GetValue() - state1 = tf.stack([ - state0[0] + new_value[0], - tf.minimum(state0[1], new_value[1]), - tf.maximum(state0[2], new_value[2]), - ]) - self.SetValue(state1) - - -class PassiveAsymQDomain(QDomain): - """A quantization domain that does passive, asymmetric quantization. - - See: https://arxiv.org/abs/1712.05877 - - This quantization domain will adjust to min/max ranges during training - time, recording them into vars via an exponential moving average and then - applying them at eval/inference time. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('bits', 8, 'Default quantized bit depth.') - p.Define('ema_decay', 0.99, 'Moving average decay.') - p.Define('default_min', -1.0, - 'Default minimum value (so initial graphs are valid).') - p.Define('default_max', 1.0, - 'Default maximum value (so initial graphs are valid).') - p.Define('quantize_weight_epsilon', 0.0, - 'Default epsilon for weight quantization to prevent zero range.') - p.Define( - 'delay_start_steps', 0, - 'Delays applying quantization at training time until after ' - 'this many steps. 0 = start immediately. -1 = start never. ' - 'This is often needed to allow the model to reach some level ' - 'of convergence prior to applying quantization. Only affects ' - 'training (not eval/inference).') - return p - - def __init__(self, params): - super().__init__(params) - - self._t_names = set() # set of known t_name (from CreateTensor) - self._qvars = py_utils.NestedMap() # var_name -> tf.Variable - - def _CreateLayerVariables(self): - # Save a scope for lazily created variables. - with tf.variable_scope('q'): - self._qvars_scope = tf.get_variable_scope() - - def _MaybeFakeQuant(self, inputs, min_v, max_v, num_bits): - p = self.params - - def Apply(): - return tf.quantization.fake_quant_with_min_max_vars( - inputs, min_v, max_v, num_bits=num_bits) - - if p.delay_start_steps != 0 and not self.do_eval: - if p.delay_start_steps == -1: - return inputs - return tf.where(self.theta.global_step >= p.delay_start_steps, Apply(), - inputs) - else: - return Apply() - - @property - def bits(self): - p = self.params - return p.bits - - def QuantizeWeight(self, w): - p = self.params - w_min = tf.reduce_min(w) - w_max = tf.reduce_max(w) - # NOTE: We force a small, non-zero range because otherwise, zero weights - # can cause downstream inference engines to blow up. - w_min = tf.minimum(w_min, -p.quantize_weight_epsilon) - w_max = tf.maximum(w_max, p.quantize_weight_epsilon) - quant_w = self._MaybeFakeQuant(w, w_min, w_max, num_bits=p.bits) - if self.do_eval: - return quant_w - else: - # If quantizing during training, skip quantization if it produces - # NANs. Sometimes early in the training process, things are unstable - # and ranges can produce numerical instability that makes it - # impossible to perform a fake_quant. - quant_w_has_nans = tf.math.is_nan(quant_w) - return tf.where(quant_w_has_nans, w, quant_w) - - def QuantizeNaturalRange(self, t, min_value, max_value): - p = self.params - return self._MaybeFakeQuant(t, min_value, max_value, num_bits=p.bits) - - def QuantizeConstantRange(self, t, min_value, max_value): - p = self.params - return self._MaybeFakeQuant(t, min_value, max_value, num_bits=p.bits) - - def CreateTensor(self, t_name): - p = self.params - assert t_name not in self._t_names, ( - 'QTensor already registered: %s' % t_name) - self._t_names.add(t_name) - - # Create accumulator - accumulator_name = self._GetAccumulatorNameForTensor(t_name) - self.RegisterAccumulator(accumulator_name, - _CountedMinMaxAccumulator(p.dtype)) - # Register vars. - min_pc = py_utils.WeightParams((), - py_utils.WeightInit.Constant(p.default_min), - p.dtype) - max_pc = py_utils.WeightParams((), - py_utils.WeightInit.Constant(p.default_max), - p.dtype) - self._CreateQStateVar(t_name, 'min', min_pc) - self._CreateQStateVar(t_name, 'max', max_pc) - - def QuantizeTensors(self, t_name, ts, eval_only=False): - p = self.params - # Always straddle a real zero point. - if self.do_eval: - # At eval/inference time, use the memorized range. - # Important: Don't capture these variables in training mode so as to - # avoid extra/unnecessary captures. - min_var = self._GetQStateVar(t_name, 'min') - max_var = self._GetQStateVar(t_name, 'max') - return [ - self._MaybeFakeQuant(t, min_var, max_var, num_bits=p.bits) for t in ts - ] - else: - # At training time, use the batch calculated min/max. - accumulator_name = self._GetAccumulatorNameForTensor(t_name) - # Calculate min/max for all tensors. - batch_min = 0.0 - batch_max = 0.0 - for t in ts: - batch_min = tf.minimum(tf.reduce_min(t), batch_min) - batch_max = tf.maximum(tf.reduce_max(t), batch_max) - - # New state. - state1 = tf.stack([1.0, batch_min, batch_max]) - self.accumulators[accumulator_name].Update(state1) - - # Results. - ts_out = [] - for i, t in enumerate(ts): - if eval_only: - # If only quantizing at eval time, still record ranges as above - # but don't quantize. - quant_t = t - else: - # If quantizing during training, skip quantization if it produces - # NANs. Sometimes early in the training process, things are unstable - # and ranges can produce numerical instability that makes it - # impossible to perform a fake_quant. - quant_t = self._MaybeFakeQuant( - t, batch_min, batch_max, num_bits=p.bits) - # TODO(laurenzo): Plumb quant_t_has_nans through state and report. - quant_t_has_nans = tf.math.is_nan(quant_t) - quant_t = tf.where(quant_t_has_nans, t, quant_t) - ts_out.append(quant_t) - summary_utils.histogram( - '%s/%s_%d' % (self._qvars_scope.name, t_name, i), t) - return ts_out - - def GetTensorRange(self, t_name, ts): - # Always straddle a real zero point. - if self.do_eval: - # At eval/inference time, use the memorized range. - # Important: Don't capture these variables in training mode so as to - # avoid extra/unnecessary captures. - min_var = tf.stop_gradient(self._GetQStateVar(t_name, 'min')) - max_var = tf.stop_gradient(self._GetQStateVar(t_name, 'max')) - return (min_var, max_var) - # Calculate min/max for all tensors. - batch_min = tf.minimum(tf.reduce_min(ts), 0.0) - batch_max = tf.maximum(tf.reduce_max(ts), 0.0) - return (tf.stop_gradient(batch_min), tf.stop_gradient(batch_max)) - - def PostTrainingStepUpdate(self, global_step): - ops = [super().PostTrainingStepUpdate(global_step)] - for t_name in self._t_names: - ops.extend(self._RecordTensor(t_name)) - self._SummarizeTensor(t_name) - return tf.group(ops) - - def _CreateQStateVar(self, t_name, suffix, params): - name = t_name + '_' + suffix - assert name not in self._qvars, 'QState var already exists: %s' % name - var_name = self._qvars_scope.name + '/' + name - with tf.variable_scope(py_utils.GetGlobalVariableScope()): - v = py_utils.CreateVariable(var_name, params, trainable=False) - self._qvars[name] = v - return v - - def _GetAccumulatorNameForTensor(self, t_name): - return 'qtensor_' + t_name - - def _GetQStateVar(self, t_name, suffix): - v = self._qvars[t_name + '_' + suffix] - return v - - def _SummarizeTensor(self, t_name): - min_var = self._GetQStateVar(t_name, 'min') - max_var = self._GetQStateVar(t_name, 'max') - # foo/q/somet_min:0 -> foo/q/somet_min - summary_name_min = min_var.name.split(':')[0] - summary_name_max = max_var.name.split(':')[0] - summary_utils.scalar(summary_name_min, min_var) - summary_utils.scalar(summary_name_max, max_var) - - def _RecordTensor(self, t_name): - p = self.params - if self.do_eval: - return [] - - accumulator_name = self._GetAccumulatorNameForTensor(t_name) - accumulator = self.accumulators[accumulator_name] - min_var = self._GetQStateVar(t_name, 'min') - max_var = self._GetQStateVar(t_name, 'max') - - # Unpack state tensor. - current_value = accumulator.GetValue() - count = current_value[0] - min_value = current_value[1] - max_value = current_value[2] - accumulator.Reset() - - def Ema(variable, value): - return (1.0 - p.ema_decay) * (variable - value) - - # Note that small floating point issues can cause ranges that naturally - # begin or end at zero to move slightly past, causing hard failures - # downstream (checks that all ranges straddle zero). We therefore repeat - # the straddling constraint here. - return [ - tf.assign( - min_var, - tf.minimum( - 0., - min_var - tf.where(count > 0., Ema(min_var, min_value), 0.))), - tf.assign( - max_var, - tf.maximum( - 0., - max_var - tf.where(count > 0., Ema(max_var, max_value), 0.))), - ] - - -def _CopyShape(from_t, to_t): - if isinstance(from_t, tf.Tensor) and isinstance(to_t, tf.Tensor): - to_t.set_shape(from_t.shape) - return to_t diff --git a/lingvo/core/quant_utils_test.py b/lingvo/core/quant_utils_test.py deleted file mode 100644 index b0a6e4d6..00000000 --- a/lingvo/core/quant_utils_test.py +++ /dev/null @@ -1,444 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for quant_utils.""" - -# pylint: disable=bad-whitespace -# pylint: disable=bad-continuation - - -import lingvo.compat as tf -from lingvo.core import py_utils -from lingvo.core import quant_utils -from lingvo.core import test_utils -import numpy as np - - -class SampleQuantizedProjectionLayer(quant_utils.QuantizableLayer): - """Simple projection layer to demonstrate quantization.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', 2, 'Depth of the input.') - p.Define('output_dim', 3, 'Depth of the output.') - return p - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - - w_pc = py_utils.WeightParams( - shape=[p.input_dim, p.output_dim], - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - self.CreateVariable('w', w_pc) - - self.TrackQTensor('inputs', 'transformed') - - def FProp(self, theta, inputs, paddings): - p = self.params - fns = self.fns - - # It is the most important that weights and top-level activations - # be tagged for quantization: - # - Weights use the self.QWeight() decorator - # - Inputs/activations are decorated with self.QTensor(). In general, - # the provided name should match a call to self.TrackQTensor in the - # constructor. This creates an tensor that is individually accounted - # for. - w = fns.qweight(theta.w) - inputs = self.QTensor('inputs', inputs) - - # Note the use of the qmatmul from the function library. This will - # automatically track the output against the qtensor 'transformed'. - out = fns.qmatmul( - tf.reshape(inputs, [-1, p.input_dim]), w, qt='transformed') - out = tf.reshape(out, tf.concat([tf.shape(inputs)[:-1], [p.output_dim]], 0)) - - # Decorate outputs of simple activation functions with their corresponding - # range decorator. This will ensure that the result does not exceed the - # precision of the underlying representation. - out = fns.qtanh(out) - - # Perform padding manipulation via booleans instead of: - # out *= 1.0 - paddings - # Because the paddings can exist in entirely different numeric ranges than - # the tensor they are being applied to, it is best to not perform - # arithmetic directly between them. Instead, broadcast them to the needed - # size (if different) and perform an exact mask with tf.where. - # For added numeric range protection, the QRPadding decorator ensures - # the correct range. This is mostly needed for cases where padding is - # dynamic at inference time. - paddings = self.QRPadding(paddings) - paddings *= tf.ones_like(out) # Broadcast to 'out' size. - out = tf.where(paddings > 0.0, tf.zeros_like(out), out) - - return out - - -class QuantizableLayerTest(test_utils.TestCase): - # pyformat: disable - NO_QDOMAIN_EXPECTED = [ - [[ 0.00071405, -0.03868543, -0.01999986, -0.00994987], - [ 0.08905827, 0.13636404, -0.03180931, 0.06056439], - [ 0. , 0. , 0. , 0. ], - [-0.0208858 , -0.17595209, -0.05192588, 0.02618068]], - [[ 0. , 0. , 0. , 0. ], - [ 0. , 0. , 0. , 0. ], - [-0.02125708, -0.10454545, -0.01147466, 0.06903321], - [ 0.0276652 , -0.14823943, -0.09726462, 0.01415125]]] - # pyformat: enable - - def testOpWrapperArgChecking(self): - with self.session(): - p = SampleQuantizedProjectionLayer.Params() - p.name = 'test' - l = p.Instantiate() - l.TrackQTensor('test') - fns = l.fns - - # Just testing one dynamic and one const op. - # Dynamic. - fns.qadd(1, 1, qt='test') - fns.qadd(1, 1, qmin=-1.0, qmax=1.0) - with self.assertRaises(AssertionError): - fns.qadd(1, 1) # No range args. - with self.assertRaises(AssertionError): - fns.qadd(1, 1, qmin=-1.0) # Incomplete range args. - with self.assertRaises(AssertionError): - fns.qadd(1, 1, qmax=-1.0) # Incomplete range args. - with self.assertRaisesRegex(AssertionError, 'first calling TrackQTensor'): - fns.qadd(1, 1, qt='non_existing') # Test that qt is resolved. - - # Const. - fns.qtanh(6.0) # No min/max. - fns.qtanh(6.0, qmin=-5.0, qmax=6.0) # Min/max - fns.qtanh(6.0, qt='test') - with self.assertRaisesRegex(AssertionError, 'first calling TrackQTensor'): - fns.qtanh(6.0, qt='non_existing') # Test that qt has precedence. - - def testLayerWithNoQDomain(self): - with self.session(): - p = SampleQuantizedProjectionLayer.Params() - self._testLayerHelper('testLayerWithNoQDomain', p, - self.NO_QDOMAIN_EXPECTED) - - def testLayerWithIdentityQDomain(self): - with self.session(): - p = SampleQuantizedProjectionLayer.Params() - p.qdomain.default = quant_utils.QDomain.Params() - self._testLayerHelper('testLayerWithIdentityQDomain', p, - self.NO_QDOMAIN_EXPECTED) - - def testLayerWithPassiveAsymQDomain(self): - # pyformat: disable - expected = [ - [[ 0. , -0.03921568, -0.02352941, -0.00784314], - [ 0.0862745 , 0.13333333, -0.03137255, 0.06274509], - [ 0. , 0. , 0. , 0. ], - [-0.02352941, -0.17254901, -0.05490196, 0.02352941]], - [[ 0. , 0. , 0. , 0. ], - [ 0. , 0. , 0. , 0. ], - [-0.02352941, -0.10196078, -0.00784314, 0.07058823], - [ 0.02352941, -0.1490196 , -0.09411764, 0.01568627]]] - # pyformat: enable - - with self.session(): - p = SampleQuantizedProjectionLayer.Params() - p.qdomain.default = quant_utils.PassiveAsymQDomain.Params() - l = self._testLayerHelper( - 'testLayerWithPassiveAsymQDomain', p, expected=expected) - init_minmax_vars = l.qdomain_default._qvars.Transform(lambda x: x.eval()) - print('Initial Minmax vars:', init_minmax_vars) - # Record. - self.evaluate([l.PostTrainingStepUpdate(16)]) - minmax_vars = l.qdomain_default._qvars.Transform(lambda x: x.eval()) - print('Minmax vars:', minmax_vars) - - # Make sure that the vars have moved from their defaults. - for k in minmax_vars: - self.assertNotEqual(init_minmax_vars[k], minmax_vars[k]) - - def testLayerWithPassiveAsymQDomainTrainQuantDisabledInital(self): - p = SampleQuantizedProjectionLayer.Params() - p.qdomain.default = quant_utils.PassiveAsymQDomain.Params() - p.qdomain.default.delay_start_steps = -1 - with self.session(): - self._testLayerHelper( - 'testLayerWithPassiveAsymQDomainTrainQuantDisabledInital', - p, - expected=self.NO_QDOMAIN_EXPECTED) - - def testLayerWithPassiveAsymQDomainTrainQuantDisabledStep16(self): - p = SampleQuantizedProjectionLayer.Params() - p.qdomain.default = quant_utils.PassiveAsymQDomain.Params() - p.qdomain.default.delay_start_steps = -1 - with self.session(): - self._testLayerHelper( - 'testLayerWithPassiveAsymQDomainTrainQuantDisabledStep16', - p, - expected=self.NO_QDOMAIN_EXPECTED, - global_step=16) - - def testLayerWithPassiveAsymQDomainEvalQuantDisabled(self): - with self.session(), self.SetEval(True): - p = SampleQuantizedProjectionLayer.Params() - p.qdomain.default = quant_utils.PassiveAsymQDomain.Params() - p.qdomain.default.delay_start_steps = -1 - self._testLayerHelper( - 'testLayerWithPassiveAsymQDomainEvalQuantDisabled', - p, - not_expected=self.NO_QDOMAIN_EXPECTED) - - def testLayerWithPassiveAsymQDomainTrainQuantDelayNotSatisfied(self): - p = SampleQuantizedProjectionLayer.Params() - p.qdomain.default = quant_utils.PassiveAsymQDomain.Params() - p.qdomain.default.delay_start_steps = 8 - with self.session(): - self._testLayerHelper( - 'testLayerWithPassiveAsymQDomainTrainQuantDelayNotSatisfied', - p, - expected=self.NO_QDOMAIN_EXPECTED, - global_step=3) - - def testLayerWithPassiveAsymQDomainTrainQuantDelaySatisfied(self): - p = SampleQuantizedProjectionLayer.Params() - p.qdomain.default = quant_utils.PassiveAsymQDomain.Params() - p.qdomain.default.delay_start_steps = 8 - with self.session(): - self._testLayerHelper( - 'testLayerWithPassiveAsymQDomainTrainQuantDelaySatisfied', - p, - not_expected=self.NO_QDOMAIN_EXPECTED, - global_step=8) - - def testLayerWithPassiveAsymQDomainTrainQuantDelaySatisfiedPlusOne(self): - p = SampleQuantizedProjectionLayer.Params() - p.qdomain.default = quant_utils.PassiveAsymQDomain.Params() - p.qdomain.default.delay_start_steps = 8 - with self.session(): - self._testLayerHelper( - 'testLayerWithPassiveAsymQDomainTrainQuantDelaySatisfied', - p, - not_expected=self.NO_QDOMAIN_EXPECTED, - global_step=9) - - def testLayerWithSymmetricScheduledClipQDomain(self): - # pyformat: disable - expected = [ - [[ 0. , -0.0390625, -0.015625 , -0.0078125], - [ 0.0859375, 0.140625 , -0.0234375, 0.0625 ], - [ 0. , 0. , 0. , 0. ], - [-0.0234375, -0.171875 , -0.0546875, 0.0234375]], - [[ 0. , 0. , 0. , 0. ], - [ 0. , 0. , 0. , 0. ], - [-0.0234375, -0.1015625, -0.015625 , 0.0703125], - [ 0. , -0.125 , -0.0625 , 0. ]]] - # pyformat: enable - - with self.session(): - p = SampleQuantizedProjectionLayer.Params() - p.qdomain.default = quant_utils.SymmetricScheduledClipQDomain.Params() - p.qdomain.default.cc_schedule.Set( - clip_start_step=0, - clip_end_step=5, - quant_start_step=10, - ) - self._testLayerHelper( - 'testLayerWithSymmetricScheduledClipQDomain', - p, - expected=expected, - global_step=16) - - def _testLayerHelper(self, - test_case, - p, - expected=None, - not_expected=None, - global_step=-1): - tf.random.set_seed(398847392) - np.random.seed(12345) - p.name = 'proj' - p.input_dim = 3 - p.output_dim = 4 - p.params_init = py_utils.WeightInit.Gaussian(0.1) - l = p.Instantiate() - in_padding = tf.zeros([2, 4, 1], dtype=tf.float32) - in_padding = tf.constant( - [[[0], [0], [1], [0]], [[1], [1], [0], [0]]], dtype=tf.float32) - inputs = tf.constant( - np.random.normal(0.1, 0.5, [2, 4, 3]), dtype=tf.float32) - output = l.FPropDefaultTheta(inputs, in_padding) - self.evaluate(tf.global_variables_initializer()) - - if global_step >= 0: - self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), global_step)) - - output = output.eval() - print('QuantizableLayerTest output', test_case, ':\n', - np.array_repr(output)) - if expected is not None: - self.assertAllClose(output, expected) - if not_expected is not None: - self.assertNotAllClose(output, not_expected) - return l - - -class ClippingCapScheduleTest: - - def testLinearClippingCapSchedule(self): - p = quant_utils.LinearClippingCapSchedule.Params() - p.start_step = 50 - p.end_step = 100 - p.start_cap = 6.0 - p.end_cap = 1.0 - cc_schedule = p.Instantiate() - with self.session(): - self.assertAllClose(cc_schedule._Value(25).eval(), 6.0) - self.assertAllClose(cc_schedule._Value(50).eval(), 6.0) - self.assertAllClose(cc_schedule._Value(60).eval(), 5.0) - self.assertAllClose(cc_schedule._Value(70).eval(), 4.0) - self.assertAllClose(cc_schedule._Value(80).eval(), 3.0) - self.assertAllClose(cc_schedule._Value(90).eval(), 2.0) - self.assertAllClose(cc_schedule._Value(100).eval(), 1.0) - self.assertAllClose(cc_schedule._Value(110).eval(), 1.0) - - def _ClipExample(self, cc_schedule, v): - """Returns a tuple of (neg, pos) for clipped neg/pos values of v.""" - v = float(v) - clipped = ( - cc_schedule.ApplyClipping(cc_schedule.theta, -v).eval(), - cc_schedule.ApplyClipping(cc_schedule.theta, v).eval(), - ) - print('Clipped +-', v, ' ->', clipped) - return clipped - - def testFakeQuantizationScheduleFromDefun(self): - p = quant_utils.FakeQuantizationSchedule.Params() - p.clip_start_step = 5 - p.clip_end_step = 10 - p.quant_start_step = 15 - p.start_cap = 6.0 - p.end_cap = 1.0 - with self.session(): - cc_schedule = p.Instantiate() - self.evaluate(tf.global_variables_initializer()) - # Move to fully quantized part of schedule - self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), 16)) - - @tf.function(autograph=False) - def ExampleFunction8(x, cc_state): - return cc_schedule.ApplyClippingWithState(cc_state, x, bits=8) - - @tf.function(autograph=False) - def ExampleFunction16(x, cc_state): - return cc_schedule.ApplyClippingWithState(cc_state, x, bits=16) - - a = tf.constant(1.0) - b = tf.constant(0.5) - - # 8bit value. - v = ExampleFunction8(a * b, cc_schedule.GetState(cc_schedule.theta)) - self.assertAllClose(v.eval(), 0.5) - - # 16bit value. - v = ExampleFunction16(a * b, cc_schedule.GetState(cc_schedule.theta)) - self.assertAllClose(v.eval(), 0.5) - - # An incomplete implementation requires special case gradient logic. - # This tests it, specifically in a Defun, which caused issues. - # 8bit gradient. - g = tf.gradients( - ExampleFunction8(a * b, cc_schedule.GetState(cc_schedule.theta)), - [a, b]) - g = [t.eval() for t in g] - print('Gradient8:', g) - self.assertAllClose(g, (0.5, 1.0)) - - # 16bit gradient. - g = tf.gradients( - ExampleFunction16(a * b, cc_schedule.GetState(cc_schedule.theta)), - [a, b]) - g = [t.eval() for t in g] - print('Gradient16:', g) - self.assertAllClose(g, (0.5, 1.0)) - - def testFakeQuantizationScheduleTraining(self): - p = quant_utils.FakeQuantizationSchedule.Params() - p.clip_start_step = 5 - p.clip_end_step = 10 - p.quant_start_step = 15 - p.start_cap = 6.0 - p.end_cap = 1.0 - with self.session(): - cc_schedule = p.Instantiate() - self.evaluate(tf.global_variables_initializer()) - # Step 0: No clipping. - self.assertAllClose( - self._ClipExample(cc_schedule, 100.0), (-100.0, 100.0)) - self.assertAllClose( - self._ClipExample(cc_schedule, 0.123456), - (-0.123456, 0.123456)) # Not Quantized. - - # Step 5: Clipping active but not yet quantizing. - self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), 5)) - self.assertAllClose( - self._ClipExample(cc_schedule, 100.0), - (-6.0, 5.953125)) # 6 * 127/128 - self.assertAllClose( - self._ClipExample(cc_schedule, 0.123456), - (-0.123456, 0.123456)) # Not Quantized. - - # Step 7: Middle of clipping range. - self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), 7)) - self.assertAllClose( - self._ClipExample(cc_schedule, 100.0), (-4.0, 3.96875)) # 4 * 127/128 - self.assertAllClose( - self._ClipExample(cc_schedule, 0.123456), - (-0.123456, 0.123456)) # Not Quantized. - - # Step 10: End of clipping range. - self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), 10)) - self.assertAllClose( - self._ClipExample(cc_schedule, 100.0), - (-1.0, 0.9921875)) # 1 * 127/128 - self.assertAllClose( - self._ClipExample(cc_schedule, 0.123456), - (-0.123456, 0.123456)) # Not Quantized. - - # Step 11: No more clipping but not yet quantizing. - self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), 11)) - self.assertAllClose( - self._ClipExample(cc_schedule, 100.0), - (-1.0, 0.9921875)) # 1 * 127/128 - self.assertAllClose( - self._ClipExample(cc_schedule, 0.123456), - (-0.123456, 0.123456)) # Not Quantized. - - # Step 15-16: Quantizing at full clip. - for step in (15, 16): - self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), step)) - self.assertAllClose( - self._ClipExample(cc_schedule, 100.0), - (-1.0, 0.9921875)) # 1 * 127/128 - self.assertAllClose( - self._ClipExample(cc_schedule, 0.123456), - (-0.125, 0.125)) # Quantized. - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/recurrent.py b/lingvo/core/recurrent.py deleted file mode 100644 index f7e41106..00000000 --- a/lingvo/core/recurrent.py +++ /dev/null @@ -1,1634 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Recurrent neural nets. - -The main interface of this module is Recurrent(). -This expects the caller to describe the recurrent neural net by specifying: - - - theta: the "weights" each RNN uses. - - state0: the initial state of each RNN. - - cell_fn: A python function describing RNN cell. It must have the following - signature:: - - cell_fn: (theta, state0, inputs) -> (state1, extras) - - state1 is the next RNN state, extras are computed by cell_fn - and the library forwards extras to cell_fn's gradient function. - - cell_grad: An optional python function describing the backprop gradient - function for the RNN cell. It must have the following signature:: - - cell_grad: (theta, state0, inputs, extras, dstate1) -> - (dtheta, dstate0, dinputs) - - dstate1 is what the backprop algorithm provides representing - gradients of the final loss w.r.t. state1. - -All of `theta`, `state0`, `inputs`, `extras` and `dstate1` are -`.NestedMap` so that they can carry a bunch of tensors around. - -Recurrent computes, roughly:: - - state = state0 - for t in inputs' sequence length: - state = cell_fn(theta, state, inputs[t, :]) - accumulate_state[t, :] = state - return accumulate_state, state - -The main advantage to using Recurrent instead of tf.while_loop is in -memory savings. In order to compute the gradient for cell_fn, a tf.while_loop -implementation will try to save all of the intermediate tensor values in the -forward pass. For long input sequences this can add up to a very large amount -of memory space. - -Recurrent saves only the state output from cell_fn, not any of the intermediate -tensors generated within cell_fn. This saves lots of memory in the forward -pass, but there is a cost: we have to recompute those intermediate tensors -in the backward pass in order to compute the gradient. This recomputation -is why we require that cell_fn be stateless: Recurrent calls cell_fn both -in the forward pass and in the backward pass, and both of those invocations -need to be the same in order for training to work properly. - -When using Recurrent, then, we need to store state for the whole training -sequence (in accumulate_state), as well as all of the intermediate tensors -for a single step of cell_fn. Without Recurrent, we would store all of the -intermediate tensors for all of the steps. - -We prefer that all of the inputs to cell_fn be passed in using theta, state0, -or inputs. But sometimes you may have code with other inputs; for instance, this -cell_fn references tensor my_tensor, even though it was never passed in as an -input:: - - my_tensor = tf.constant(5) - def cell_fn(inputs): - return inputs.input * my_tensor - -We say that my_tensor was implicitly captured by cell_fn. By default, -Recurrent doesn't allow this, but you can change that behavior by setting the -allow_implicit_captures flag. -""" - -import collections -import lingvo.compat as tf -from lingvo.core import cluster_factory -from lingvo.core import py_utils -from lingvo.core import sendrecv -from lingvo.core import symbolic - -DevicePair = collections.namedtuple('DevicePair', ['send', 'recv']) - - -def _AssertSameTensors(list_a, list_b): - """Asserts that two lists of tensors are the same tensors.""" - assert len(list_a) == len(list_b), ( - 'Expected equal tensor lists but different lengths: %r vs %r' % - (list_a, list_b)) - for a, b in zip(list_a, list_b): - assert a is b, ( - 'Expected equal tensor lists but at least one differs: %r vs %r' % - (list_a, list_b)) - - -def _Index(nmap, index): - """Returns a `.NestedMap` with x[index, :] for each tensor x in nmap. - - Args: - nmap: A `.NestedMap` of tensors. - index: A tf scalar integer. Performance is better if 'index' is on the host - memory. - - Returns: - A `.NestedMap` of tensors. For each key in nmap:: - - rets.key = nmap.key[index, :] - """ - index = tf.convert_to_tensor(index) - index.get_shape().assert_has_rank(0) - return nmap.Transform(lambda x: tf.gather(x, index)) - - -def _Update(nmap_acc, nmap_x, t): - """Updates t-th row in accumulators. - - Args: - nmap_acc: A `.NestedMap` of tensors. The accumulators. - nmap_x: A `.NestedMap` of tensors. The update values. - t: A scalar integer. Performance is better if 't' is on the device memory. - - Returns: - A `.NestedMap` of tensors. Say, ret is returned. For each key, we have:: - - ret[key] = nmap_acc[key]; - ret[key][t, :] = nmap_x[key] - """ - acc_lst = nmap_acc.Flatten() - kx_lst = nmap_x.FlattenItems() - t = tf.cast([t], tf.int32) # tf.cast casts on-device tensors. - lst = [] - for acc, (key, x) in zip(acc_lst, kx_lst): - with tf.name_scope('update_%s' % py_utils.SanitizeScopeKey(key)): - lst += [tf.InplaceUpdate(acc, t, tf.expand_dims(x, 0))] - return nmap_acc.Pack(lst) - - -def _SeqLenDim(nmap): - """Returns the 0-th dim size of tensors in nmap. - - This is the max sequence length according to the shape of the inputs. - - Args: - nmap: A `.NestedMap` of tensors. Every tensor's 0-th dim has the same size. - - Returns: - A scalar tensor which is the size of 0-th dim of every tensors in nmap. - """ - keys, values = zip(*nmap.FlattenItems()) - assert values, 'nmap is empty.' - with tf.control_dependencies([ - py_utils.assert_same_dim0( - values, msg='recurrent._SeqLen: %s' % list(keys)) - ]): - return tf.shape(values[0])[0] - - -def FlattenPadding(padding): - """Returns padding reduced to have only the time dimension.""" - if padding is None: - return padding - r = tf.rank(padding) - return tf.reduce_min(padding, axis=tf.range(1, r)) - - -def _SeqPaddingLength(inputs_nmap): - """Returns the lengths of paddings at the beginning and end of the sequence. - - Args: - inputs_nmap: A `.NestedMap` of tensors that may have 'padding' Every - tensor's 0-th dim has the same size. - - Returns: - padding length at the beginning, padding length at the end - """ - padding = inputs_nmap.get('padding') - if padding is None: - return [0, 0] - time = tf.shape(padding)[0] - pad_1d = FlattenPadding(padding) - mask = tf.cast(tf.equal(pad_1d, 0), tf.int32) # [time], 1s/0s - mask_reverse = tf.cast(tf.equal(tf.reverse(pad_1d, [0]), 0), tf.int32) - numbers = tf.range(1, time + 1) - padding_end = time - tf.reduce_max(mask * numbers) - padding_begin = tf.where( - tf.equal(padding_end, time), 0, - time - tf.reduce_max(mask_reverse * numbers)) - return [padding_begin, padding_end] - - -def _EmptyAcc(slen, nmap): - """Creates a set of accumulators for tensors in nmap. - - Args: - slen: A scalar tensor. - nmap: A `.NestedMap` of tensors. - - Returns: - A `.NestedMap` with the same keys as nmap. ret.key, a tensor, has the - same dtype as nmap.key. The tensor's shape has 1 more dimension - than the tensor nmap.key. The extra 0-th dimension is of size - slen. E.g., if slen=10 and nmap.key's shape is [3, 5], then, - ret.key's shape is [10, 3, 5]. - """ - - def Fill(x): - return tf.Empty( - tf.concat([[slen], tf.shape(x)], axis=0), x.dtype, init=True) - - return nmap.Transform(Fill) - - -def _EmptyWithFixShape(shape, nmap): - """Creates a set of empty initialized tensors with fixed shape. - - Args: - shape: A list of integers to describe the output tensor shape. - nmap: A `.NestedMap` of tensors. - - Returns: - A `.NestedMap` with the same keys as nmap. ret.key, a tensor, has the - same dtype as nmap.key, but with the fixed shape. - """ - - return nmap.Transform(lambda x: tf.Empty(shape, dtype=x.dtype, init=True)) - - -def _EmptyLike(nmap): - """Creates a set of empty initialized tensors. - - Args: - nmap: A `.NestedMap` of tensors. - - Returns: - A `.NestedMap` of tensors. Each tensor has the same shape and dtype as - its corresponding tensor in nmap. And each tensor is initialized. - """ - return nmap.Transform(lambda x: tf.EmptyLike(x, init=True)) - - -def _Add(nmap_x, nmap_y): - """Adds tensors in nmap_x with respective tensors in nmap_y. - - Args: - nmap_x: A `.NestedMap` of tensors. - nmap_y: A `.NestedMap` of tensors. - - Returns: - A `.NestedMap` of tensors. ret.key = nmap_x.key + nmap_y.key for every key. - """ - return py_utils.Transform(tf.add, nmap_x, nmap_y) - - -def _TransformDType(nmap): - return nmap.Transform(lambda x: tf.cast(x, tf.int64) - if x.dtype == tf.int32 else x) - - -class _Recurrent: - """A helper class to construct a recurrent neural net.""" - - def __init__(self, - cell_fn, - cell_grad, - stop_fn, - theta, - state0, - inputs, - extras, - cell_type=None, - accumulator_layer=None, - implicit_captures=None, - unused_acc_state=None): - """RNN helper class. - - Args: - cell_fn: A python function which computes: - state1, extras = cell_fn(theta, state0, inputs[t, :]) - cell_grad: A python function which computes: - dtheta, dstate0, dinputs[t, :] = cell_grad( - theta, state0, inputs[t, :], extras, dstate1) - stop_fn: A python function which computes: should_stop = stop_fn(t, theta, - state0) - theta: weights. A `.NestedMap`. - state0: initial state. A `.NestedMap`. - inputs: inputs. A `.NestedMap`. - extras: A `.NestedMap` of Tensors. The 2nd return value of every - invocation of cell_fn is a `.NestedMap` with matching keys and shapes of - this 'extras'. - cell_type: Cell type used in this class. - accumulator_layer: If provided, then accumulators on this layer will be - managed such that they carry to the final state in `FProp` and are - disabled for gradients. Uses the state key `accumulators`. - implicit_captures: A `.NestedMap` corresponding to implicit captures of - the cell_fn. If empty/None, implicit captures are either not present or - disallowed. - unused_acc_state: If None, we assume every field of acc_state is consumed - in the following timestamps. If True, None of the acc_state is consumed. - And we reduce_sum each timestep's new state into a scalar. Note, this - feature should be used with StackedRecurrent where we send out the new - state to the other devices. - """ - self._theta = theta - self._state = state0 - self._inputs = inputs - self._cell_fn = _DecorateCellFn(cell_fn, accumulator_layer) - self._cell_grad = _DecorateCellGrad(cell_grad, accumulator_layer) - self._stop_fn = stop_fn - self._extras = extras - if cell_type is not None: - self._cell_type = cell_type - else: - self._cell_type = 'UnknownType' - self._accumulator_layer = accumulator_layer - self._implicit_captures = implicit_captures - self._unused_acc_state = unused_acc_state - - # NOTE: TF Function (Fwd, Bak, ForwardLoopBody, BackwardLoopBody, - # Forward and Backward defined below) simply takes a list of - # Tensors and returns a list of Tensors. When we pass in a - # structure (a list of NestedMap of Tensors), we use Flatten to - # convert the structure into a list of tensor. Conversely, the - # following code often uses Pack to formulate a structure from a - # list of tensors based on a "template". - - compiled = py_utils.use_xla() - noinline = not compiled - - # state1, extras = cell_fn(theta, state0, inputs) - def Fwd(theta, state0, inputs): - py_utils.SetShapes(theta, self._theta) - state1, extras = self._cell_fn(theta, state0, inputs) - py_utils.AssertIsCompatible(state1, self._state) - py_utils.AssertIsCompatible(extras, self._extras) - return state1, extras - - # Wraps cell_fn in a TF Function as a for-loop's body. - # - # The loop state is composed of: - # t: The loop variable on the device. Timestep id. - # theta: the recurrent net's weights. - # state0: the previous recurrent state. - # inputs: inputs to the recurrent net. inputs[t, :] are for the timestep t. - # acc_state: Each timestep's computed new state is also stashed into - # acc_state. - # acc_extras: Each timestep's computed extras is stashed into acc_extras - - def ForwardLoopCond(loop_state): - """The condition of forward loop.""" - should_continue = loop_state.t < loop_state.limit - if self._stop_fn: - should_continue = tf.math.logical_and( - should_continue, - tf.reduce_any( - tf.math.logical_not( - self._stop_fn(loop_state.t, loop_state.theta, - loop_state.state0)))) - return should_continue - - def ForwardLoopBody(loop_state): - """The body of forward loop.""" - t = loop_state.t - # external input at time step t. - inputs_t = _Index(loop_state.inputs, t) - loop_state.state0, extras = Fwd(loop_state.theta, loop_state.state0, - inputs_t) - # Saves state1 and extras in their accumulators. - if not self._unused_acc_state: - loop_state.acc_state = _Update(loop_state.acc_state, loop_state.state0, - t) - loop_state.acc_extras = _Update(loop_state.acc_extras, extras, t) - loop_state.t = tf.add(t, 1) - return loop_state - - # Forward calls ForwardLoopBody n times. Each time computes one - # time step of the recurrent net. - def Forward(args): - """Forward pass of the recurrent net.""" - # The sequence length. - pad_begin, pad_end = _SeqPaddingLength(args.inputs) - slen_dim = _SeqLenDim(args.inputs) - limit = slen_dim - pad_end - - # Creates accumulators for state0 and extras. - if self._unused_acc_state: - acc_state = _EmptyWithFixShape([slen_dim], args.state0) - else: - acc_state = _EmptyAcc(slen_dim, args.state0) - acc_extras = _EmptyAcc(slen_dim, args.extras) - - if compiled: - t = tf.cast(pad_begin, tf.int32) - limit = tf.cast(limit, tf.int32) - else: - t = tf.cast(pad_begin, tf.int64) - limit = tf.cast(limit, tf.int64) - - with py_utils.RemoveAssertContext(remove=noinline): - run = py_utils.WhileLoop( - ForwardLoopCond, - ForwardLoopBody, - loop_state=py_utils.NestedMap( - t=t, - limit=limit, - theta=args.theta, - state0=args.state0, - inputs=args.inputs, - acc_state=acc_state, - acc_extras=acc_extras)) - return py_utils.NestedMap( - limit=run.t, - final_state=run.state0, - acc_state=run.acc_state, - acc_extras=run.acc_extras) - - # The per-step backward computes: - # d_theta, d_state0, d_inputs = cell_grad( - # theta, state0, inputs, extras, d_state1) - # where d_state1 is the backprop-ed gradient for state1, and - # extras is the computed by the forward step to facilitate the - # backward step. - def Bak(theta, state0, inputs, extras, d_state1): - """Backward step.""" - py_utils.SetShapes(theta, self._theta) - (dtheta, dstate0, dinputs, - dcaptures) = self._cell_grad(theta, state0, inputs, extras, d_state1) - py_utils.AssertIsCompatible(dtheta, self._theta) - py_utils.AssertIsCompatible(dstate0, self._state) - py_utils.AssertIsCompatible(dinputs, self._inputs) - if dcaptures is None: - # NOTE: Custom gradient fns can return None if they do not support - # captured tensors. The return value is reserved for the future when - # that may be supported. - dcaptures = _EmptyLike(self._implicit_captures) - py_utils.AssertIsCompatible(dcaptures, self._implicit_captures) - - # Make sure this function didn't capture anything different than the - # cell_fn when reflected on at the beginning. Must come after the call - # to cell_grad() which adds to the captured list. - _AssertSameTensors(py_utils.GetExtraInputs(), - self._implicit_captures.Flatten()) - - return [dtheta, dstate0, dinputs, dcaptures] - - # Wraps cell_grad gradient function in a TF Function as a - # for-loop's body for the Backward pass. - # - # The loop state is composed of: - # t: The loop variable on the device. Timestep id. - # state0: the initial state for the entire backward loop. - # theta: the recurrent net's weights. - # inputs: inputs to the recurrent net. inputs[t, :] are for the timestep t. - # acc_state: Each timestep's computed new state was stashed into - # acc_state by the Forward pass. - # acc_extras: Each timestep's computed extras was stashed into - # acc_extras by the Forward pass. - # d_theta: All timestep's gradient for theta is accumulated (added) into - # d_theta. - # d_state1: The backprop-ed gradient for the new stated computed by - # timestep t. - # d_inputs: d_inputs[t, :] is populated by the backward time step t. - # d_acc_state: The backprop-ed gradient for acc_state. - # d_captured: All timestep's gradient for theta is accumulated (added) - # into d_captured. - - def BackwardLoopCond(loop_state): - """Backward loop condition function.""" - return loop_state.t >= loop_state.limit - - def BackwardLoopBody(loop_state): - """Backward loop body function.""" - t = loop_state.t - # The input recurrent state for time step t is previous time step's - # output, or the original state0 when on time step 0. - state_from_acc = _Index(loop_state.acc_state, - tf.maximum(tf.constant(0, t.dtype), t - 1)) - state0 = py_utils.If( - tf.equal(t, tf.constant(0, t.dtype)), - inputs=py_utils.NestedMap( - orig_state0=loop_state.state0, state_from_acc=state_from_acc), - then_branch=lambda nmap: nmap.orig_state0, - else_branch=lambda nmap: nmap.state_from_acc) - - # The external inputs for time step t. - inputs_t = _Index(loop_state.inputs, t) - # The extras for time step t. - extras_t = _Index(loop_state.acc_extras, t) - - d_state1 = _Add(_Index(loop_state.d_acc_state, t), loop_state.d_state1) - (d_theta_t, loop_state.d_state1, d_inputs_t, - d_captured_t) = Bak(loop_state.theta, state0, inputs_t, extras_t, - d_state1) - - if self._unused_acc_state: - # XLA IF op requires the same shape for if and else branches. - loop_state.d_state1 = loop_state.d_state1.Transform(tf.reduce_sum) - loop_state.d_theta = _Add(loop_state.d_theta, d_theta_t) - loop_state.d_inputs = _Update(loop_state.d_inputs, d_inputs_t, t) - loop_state.d_captured = _Add(loop_state.d_captured, d_captured_t) - loop_state.t = tf.subtract(t, 1) - - # Make sure this function didn't capture anything different than the - # cell_fn when reflected on at the beginning. Must come after the call - # to Bak() which adds to the captured list. - _AssertSameTensors(py_utils.GetExtraInputs(), - self._implicit_captures.Flatten()) - - return loop_state - - # Backward calls BackwardLoopBody n times. Each time computes the backprop - # for one time step of the recurrent net. - def Backward(xs, ys, dys): - """Backward pass for the recurrent net. - - Args: - xs: inputs to the forward operation. - ys: outputs of the forward operation. - dys: gradients to the outputs of the forward operation. - - Returns: - Gradients to the inputs of the forward operation. - """ - # Accumulators for gradients. - d_theta = _EmptyLike(xs.theta) - d_inputs = _EmptyLike(xs.inputs) - d_captured = _EmptyLike(self._implicit_captures) - - # The sequence length. - pad_begin, _ = _SeqPaddingLength(xs.inputs) - limit = pad_begin - - if compiled: - limit = tf.cast(limit, tf.int32) - else: - limit = tf.cast(limit, tf.int64) - - state0 = xs.state0 - d_state1 = dys.final_state - if self._unused_acc_state: - # XLA While op requires the same shape for the init and carry on - # values. - state0 = state0.Transform(tf.reduce_sum) - d_state1 = d_state1.Transform(tf.reduce_sum) - - with py_utils.RemoveAssertContext(remove=noinline): - run = py_utils.WhileLoop( - cond=BackwardLoopCond, - body=BackwardLoopBody, - loop_state=py_utils.NestedMap( - t=ys.limit - 1, - limit=limit, - theta=xs.theta, - state0=state0, - inputs=xs.inputs, - acc_state=ys.acc_state, - acc_extras=ys.acc_extras, - d_theta=d_theta, - d_state1=d_state1, - d_inputs=d_inputs, - d_acc_state=dys.acc_state, - d_captured=d_captured)) - - d_state0 = run.d_state1 - if self._unused_acc_state: - # Match the shape of gradient of the init_state. - d_state0 = self._state.Transform(tf.zeros_like) - - # The `extra` input in the Forward function is actually an output of the - # function. It was supplied as an input only to create acc_extras with - # proper shape, so its gradients should be zero. - return py_utils.NestedMap( - d_theta=run.d_theta, - d_state0=d_state0, - d_inputs=run.d_inputs, - d_extras=_EmptyLike(self._extras)), run.d_captured - - # Forward arguments. - self._fwd_args = py_utils.NestedMap( - theta=self._theta, - state0=self._state, - inputs=self._inputs, - extras=self._extras) - - # pylint: disable=protected-access - device_funcs = tf.get_default_graph()._device_functions_outer_to_inner - self._caller_device = device_funcs[-1] if device_funcs else None - # pylint: enable=protected-access - - self._forward = Forward - self._backward = Backward - - def Compute(self): - """Run the computation.""" - run = py_utils.CallDefun( - self._forward, - self._fwd_args, - bak=self._backward, - device=self._caller_device) - - if self._accumulator_layer: - # Restore the accumulators from the final recurrent state. - self._accumulator_layer.SetAccumulatorValues(run.final_state.accumulators) - del run.acc_state.accumulators - del run.final_state.accumulators - - del run.acc_state['_step_seed'] - py_utils.ResetStepSeed(run.final_state.pop('_step_seed')) - - return run.acc_state, run.final_state - - -def _ReflectOnCellFn(cell_fn, - theta, - state0, - inputs, - accumulator_layer=None, - check_stateful_ops=False, - allow_implicit_capture=False): - """Reflects on the cell_fn, applying asserts and returning needed info. - - Args: - cell_fn: A python function that computes: - state1, extras = cell_fn(theta, state0, inputs[t, :]) - theta: weights. A `.NestedMap`. - state0: initial state. A `.NestedMap`. - inputs: inputs. A `.NestedMap`. - accumulator_layer: Whether the cell function must be run in the context of - the given accumulator layer. - check_stateful_ops: if True, raise a `ValueError` if cell_fn is stateful. - allow_implicit_capture: Whether to allow the `cell_fn` to implicitly capture - tensors. - - Returns: - `.NestedMap` of implicit captures that the cell_fn takes. - Raises: - ValueError: cell_fn is stateful. - """ - # Reset the augmented state entries as we may be running in a special - # disabled context and we want state0 to reflect that. - state0 = _AugmentState( - state0.DeepCopy(), accumulator_layer, allow_overwrite=True) - - fwd_sig = [theta, state0, inputs] - input_signature = py_utils.Transform(lambda t: tf.TensorSpec(None, t.dtype), - fwd_sig) - - @py_utils._WrapFunction(input_signature=py_utils.Flatten(input_signature)) # pylint: disable=protected-access - def Fwd(*args): - # tf.function inherits the step seed collection from parent graph, we need - # to reset it to mimic Defun's behavior. - py_utils.ResetStepSeed() - - (theta, state0, inputs) = py_utils.Pack(fwd_sig, args) - py_utils.SetShapes(theta, fwd_sig[0]) - state1, extras = cell_fn(theta, state0, inputs) - return py_utils.Flatten([state1, extras]) - - # Get the stateful ops used in cell_fn. Logic borrowed from - # _EagerDefinedFunction.__init__(). - input_ops = set(arg.op for arg in Fwd.graph.inputs) - operations = [op for op in Fwd.graph.get_operations() if op not in input_ops] - stateful_ops = tuple(op for op in operations if op._is_stateful) # pylint: disable=protected-access - - # Asserts about the function. - if stateful_ops: - if check_stateful_ops: - raise ValueError('cell_fn contains stateful ops: %s' % stateful_ops) - else: - tf.logging.warning('cell_fn contains stateful ops: %s', stateful_ops) - - if cluster_factory.Current().job in {'trainer', 'trainer_client'}: - stateful_random_ops = py_utils.StatefulRandomOpsInDefun(Fwd) - if stateful_random_ops: - raise tf.errors.InvalidArgumentError( - None, None, 'cell_fn depends on stateful random ops: {}'.format( - stateful_random_ops)) - - ret = py_utils.NestedMap() - captured_inputs = list(Fwd.captured_inputs) - if captured_inputs: - if not allow_implicit_capture: - raise ValueError('Recurrent cell_fn implicitly captures tensors but ' - 'implicit capture is disabled or a custom cell_grad fn ' - 'is in use. Captured tensors: %r' % captured_inputs) - ret.captured = captured_inputs - return ret - - -def _GetCellGrad(cell_fn, - cell_grad, - theta, - state0, - inputs, - accumulator_layer, - check_stateful_ops=False, - allow_implicit_capture=False): - """Returns the gradient function for cell_fn. - - Args: - cell_fn: The recurrent neural net's cell function. - cell_grad: If not None, cell_fn's gradient function. - theta: weights. A `.NestedMap`. - state0: initial state. A `.NestedMap`. - inputs: inputs. A `.NestedMap`. - accumulator_layer: Whether the cell function must be run in the context of - the given accumulator layer. - check_stateful_ops: if True, raise a `ValueError` if cell_fn is stateful. - allow_implicit_capture: Whether to allow the `cell_fn` to implicitly capture - tensors. - - Returns: - Returns (cell_grad, implicit_captures). The passed in cell_grad is returned - as-is if not None. Otherwise, assume cell_fn is a python function - representing the recurrent neural net's cell function, i.e.:: - - cell_fn: (theta, state0, inputs) -> (state1, extra) - - returns its default gradient python function, i.e.:: - - cell_grad: (theta, state0, inputs, extras, captured, dstate1) -> - (dtheta, dstate0, dinputs) - """ - implicit_captures = _ReflectOnCellFn(cell_fn, theta, state0, inputs, - accumulator_layer, check_stateful_ops, - allow_implicit_capture) - - if not cell_grad: - - def CellGrad(theta, state0, inputs, unused_extras, dstate1): - """Default gradient function for cell_fn.""" - # NOTE: The default grad function recomputes the forward - # function and does not take advantage of 'extras' returned by - # the forward function. - state1, _ = cell_fn(theta, state0, inputs) - assert isinstance(state1, py_utils.NestedMap), ('%s' % state1) - - # Assert that if captured inputs were given, they match the actual - # tensors passed to the function we are compiled into. Must come after - # the call to cell_fn, which does the capture. - _AssertSameTensors(py_utils.GetExtraInputs(), implicit_captures.Flatten()) - - # Extract the internal captured tensor placeholders within the Defun - # we are running in. - captured = py_utils.Pack(implicit_captures, py_utils.GetExtraArgs()) - ys = py_utils.Flatten(state1) - xs = py_utils.Flatten([theta, state0, inputs, captured]) - grad_ys = py_utils.Flatten(dstate1) - grads = tf.gradients(ys=ys, xs=xs, grad_ys=grad_ys) - grads = py_utils.ConvertNoneGradientToZeros(xs, grads) - return py_utils.Pack([theta, state0, inputs, captured], grads) - - cell_grad = CellGrad - - return cell_grad, implicit_captures - - -def _AugmentState(state0, accumulator_layer, allow_overwrite=False): - """Augments state0 with additional state.""" - if accumulator_layer: - if 'accumulators' in state0 and not allow_overwrite: - raise ValueError('accumulators is a private state key used by Recurrent.') - state0.accumulators = accumulator_layer.GetAccumulatorValues() - - # _step_seed is used for seeding stateless random ops. - # See py_utils.GenerateStepSeedPair for more details. - if '_step_seed' in state0 and not allow_overwrite: - raise ValueError('_step_seed is a private state key used by Recurrent.') - state0['_step_seed'] = py_utils.GetStepSeed() - - return state0 - - -def _WrapAccumulatorCellFn(accumulator_layer, cell_fn): - """Wrap a cell_fn to propagate accumulators.""" - - def WrappedCellFn(theta, state0, inputs): - """cell_fn wrapped to propagate accumulators.""" - accumulator_layer.SetAccumulatorValues(state0.accumulators) - # The underlying cell_fn has no knowledge of accumulator state so - # delete it. - state0_accumulators = state0.pop('accumulators') - state1, extras = cell_fn(theta, state0, inputs) - state0.accumulators = state0_accumulators - # Propagate new accumulator state forward. - state1.accumulators = accumulator_layer.GetAccumulatorValues() - # Reset: make sure nothing escapes. - accumulator_layer.accumulators.Transform(lambda x: x.Reset()) - return state1, extras - - return WrappedCellFn - - -def _WrapAccumulatorCellGradFn(accumulator_layer, cell_grad): - """Wrap a cell grad function to disable accumulators.""" - - def WrappedCellGradFn(theta, state0, inputs, extras, dstate1): - """cell_grad wrapped to disable accumulators.""" - # Compute the cell grad function with accumulators disabled. - accumulator_layer.accumulators.Transform(lambda x: x.Disable()) - # The underlying cell_grad has no knowledge of accumulator state so - # delete it. - state0_accumulators = state0.pop('accumulators') - dstate1_accumulators = dstate1.pop('accumulators') - dtheta, dstate0, dinputs, dcaptures = cell_grad(theta, state0, inputs, - extras, dstate1) - state0.accumulators = state0_accumulators - dstate0.accumulators = dstate1_accumulators - dstate1.accumulators = dstate1_accumulators - accumulator_layer.accumulators.Transform(lambda x: x.Enable()) - return dtheta, dstate0, dinputs, dcaptures - - return WrappedCellGradFn - - -def _WrapCellFnWithStepSeed(cell_fn): - """Wrap a cell_fn to initialize the step seed.""" - - def WrappedCellFn(theta, state0, *args, **kwargs): - """The wrapper function.""" - # The _step_seed state should be transparent to cell_fn. - state0_step_seed = state0.pop('_step_seed') - py_utils.ResetStepSeed(state0_step_seed) - state1, extras = cell_fn(theta, state0, *args, **kwargs) - state0['_step_seed'] = state0_step_seed - state1['_step_seed'] = py_utils.GetStepSeed() - return state1, extras - - return WrappedCellFn - - -def _WrapCellGradFnWithStepSeed(cell_grad): - """Wrap a cell grad function to handle step seed in state.""" - - def WrappedCellGradFn(theta, state0, inputs, extras, dstate1): - """The wrapper function.""" - # The _step_seed state should be transparent to cell_grad. - state0_step_seed = state0.pop('_step_seed') - dstep_seed = dstate1.pop('_step_seed') - py_utils.ResetStepSeed(state0_step_seed) - dtheta, dstate0, dinputs, dcaptures = cell_grad(theta, state0, inputs, - extras, dstate1) - state0['_step_seed'] = state0_step_seed - dstate0['_step_seed'] = dstep_seed - dstate1['_step_seed'] = dstep_seed - return dtheta, dstate0, dinputs, dcaptures - - return WrappedCellGradFn - - -def _WrapCellFnWithSymbolValues(cell_fn, symbol_to_tensor_map): - """Wrap a cell_fn to propagate symbol values.""" - - def WrappedCellFn(theta, state0, inputs): - """cell_fn wrapped to propagate accumulators.""" - theta = theta.copy() - symbols = list(symbol_to_tensor_map.keys()) - symbol_values = theta.pop('_symbol_values') - inner_symbol_to_tensor_map = dict(zip(symbols, symbol_values)) - if symbols: - tf.logging.info('_WrapCellFnWithSymbolValues: %s', symbols) - with symbolic.SymbolToValueMap(symbolic.TENSOR_VALUES, - inner_symbol_to_tensor_map): - state1, extras = cell_fn(theta, state0, inputs) - return state1, extras - - return WrappedCellFn - - -def _WrapCellGradFnWithSymbolValues(cell_grad, cell_fn, symbol_to_tensor_map): - """Wrap a cell grad function to propagate symbol values.""" - - def WrappedCellGradFn(theta, state0, inputs, extras, dstate1): - """The wrapper function.""" - symbols = list(symbol_to_tensor_map.keys()) - symbol_values = theta['_symbol_values'] - inner_symbol_to_tensor_map = dict(zip(symbols, symbol_values)) - if symbols: - tf.logging.info('_WrapCellGradFnWithSymbolValues: %s', symbols) - with symbolic.SymbolToValueMap(symbolic.TENSOR_VALUES, - inner_symbol_to_tensor_map): - dtheta, dstate0, dinputs, dcaptures = cell_grad(theta, state0, inputs, - extras, dstate1) - # cell_grad may have populated dtheta by applying tf.gradients() on - # theta.Flatten(). - if '_symbol_values' not in dtheta: - state1, _ = cell_fn(theta, state0, inputs) - dxs = tf.gradients( - ys=state1.Flatten(), xs=symbol_values, grad_ys=dstate1.Flatten()) - dtheta['_symbol_values'] = py_utils.ConvertNoneGradientToZeros( - symbol_values, dxs) - return dtheta, dstate0, dinputs, dcaptures - - return WrappedCellGradFn - - -def _DecorateCellFn(cell_fn, accumulator_layer): - """Decorates cell_fn with additional state information.""" - if accumulator_layer: - # Wrap the cell_fn so that it knows how to propagate accumulators. - cell_fn = _WrapAccumulatorCellFn(accumulator_layer, cell_fn) - cell_fn = _WrapCellFnWithStepSeed(cell_fn) - return cell_fn - - -def _DecorateCellGrad(cell_grad, accumulator_layer): - """Decorates cell_grad with additional state information.""" - if accumulator_layer: - # Wrap the cell_grad so it disables accumulators. - cell_grad = _WrapAccumulatorCellGradFn(accumulator_layer, cell_grad) - cell_grad = _WrapCellGradFnWithStepSeed(cell_grad) - return cell_grad - - -def _IsSingleTimeStep(inputs): - """Returns True only if the time dimension of inputs is 1.""" - for x in inputs.Flatten(): - if x.shape.dims is None or x.shape.as_list()[0] != 1: - return False - return True - - -def _RecurrentSingleTimeStep(theta, state0, inputs, cell_fn): - """Short-cut for the single timestep without explicit cell_grad case.""" - # The seqlen length is staticly known as 1. Hence, we just need to - # call cell_fn once without putting it into a loop. - # Since we are not looping, there is no need to specially manage - # accumulators. - inputs = inputs.Transform(lambda x: tf.squeeze(x, axis=0)) - state1, _ = cell_fn(theta, state0, inputs) - acc_state = state1.Transform(lambda x: tf.expand_dims(x, axis=0)) - return acc_state, state1 - - -def Recurrent(theta, - state0, - inputs, - cell_fn, - cell_grad=None, - cell_type=None, - stop_fn=None, - extras=None, - check_stateful_ops=False, - accumulator_layer=None, - allow_implicit_capture=False): - """Compute a recurrent neural net. - - Roughly, `Recurrent()` computes the following:: - - state = state0 - for t in inputs' sequence length: - state = cell_fn(theta, state, inputs[t, :]) - accumulate_state[t, :] = state - return accumulate_state, state - - `theta`, `state`, `inputs` are all `.NestedMap` objects. - - `inputs[t, :]` means taking a slice out from every tensor in the - `.NestedMap` `inputs`. - - `accumulate_state[t, :] = state` means that we stash every tensor in - `state` into a slice of the corresponding tensor in - `accumulate_state`. - - `cell_fn` is a python callable computing (building up a TensorFlow - graph) the recurrent neural network's one forward step. `cell_fn` must not - contain any stateful ops. Two calls of `cell_fn` must describe two identical - computations. - - By construction, `Recurrent()`'s backward computation does not access - any intermediate values computed by `cell_fn` during forward - computation. We may extend `Recurrent()` to support that by taking a - customized backward function of `cell_fn`. - - Args: - theta: weights. A `.NestedMap`. - state0: initial state. A `.NestedMap`. - inputs: inputs. A `.NestedMap`. - cell_fn: A python function which computes:: - state1, extras = cell_fn(theta, state0, inputs[t, :]) - cell_grad: A python function which computes:: - dtheta, dstate0, dinputs[t, :], dcaptured = cell_grad( - theta, state0, inputs[t, :], extras, dstate1) If there are no - captured tensors in `cell_fn`, `dcaptured` can be returned as - None. Captured tensors with custom `cell_grad` is currently - unsupported so this return value is reserved for future expansion. - cell_type: Cell name to be used. - stop_fn: If not None, a python function which computes:: should_stop = - stop_fn(t, theta, state0) The function determines whether the recurrent - loop should terminate. - extras: A `.NestedMap` of Tensors. The 2nd return value of every invocation - of `cell_fn` is a `.NestedMap` with matching keys and shapes of `extras`. - check_stateful_ops: if True, raise a `ValueError` if `cell_fn` is stateful. - accumulator_layer: If provided, then accumulators on this layer will be - managed such that they carry to the final state in `FProp` and are - disabled for gradients. Uses the state key `accumulators`. - allow_implicit_capture: Whether to allow the `cell_fn` to implicitly capture - tensors. Only allowed if an explicit `cell_grad` is not given. - - Returns: - `accumulate_state` and the final state. - """ - symbol_to_tensor_map = symbolic.SymbolToValueMap.Get(symbolic.TENSOR_VALUES) - if symbol_to_tensor_map: - theta = theta.copy() # Do not modify the caller's 'theta'. - theta['_symbol_values'] = list( - tf.convert_to_tensor(v) for v in symbol_to_tensor_map.values()) - cell_fn = _WrapCellFnWithSymbolValues(cell_fn, symbol_to_tensor_map) - if cell_grad: - cell_grad = _WrapCellGradFnWithSymbolValues(cell_grad, cell_fn, - symbol_to_tensor_map) - - inputs = _TransformDType(inputs) - if cell_grad is not None: - allow_implicit_capture = False - - # Short-cut for the single timestep with default grad function case. - if cell_grad is None and _IsSingleTimeStep(inputs): - return _RecurrentSingleTimeStep(theta, state0, inputs, cell_fn) - - # Disable accumulators since cell_fn needs to be called a few times and those - # aren't real calls to cell_fn. They will be re-enabled just prior to - # calling _Recurrent. - if accumulator_layer: - accumulator_layer.accumulators.Transform(lambda x: x.Disable()) - - cell_grad, implicit_captures = _GetCellGrad(cell_fn, cell_grad, theta, state0, - inputs, accumulator_layer, - check_stateful_ops, - allow_implicit_capture) - - with tf.name_scope('recurrent_cellfn_extras'): - # Derives 'extras' so that we can allocate extras' accumulator. - # Not a real call to cell_fn, so make sure it doesn't affect step_seed. - step_seed = py_utils.GetStepSeed() - # Make sure not to modify the original state0. - _, actual_extras = cell_fn(theta, state0.DeepCopy(), _Index(inputs, 0)) - py_utils.ResetStepSeed(step_seed) - if extras is None: - extras = actual_extras.Transform(tf.zeros_like) - else: - if not extras: - # Forces the extras to be an empty map if an empty 'extras' is provided. - extras = py_utils.NestedMap() - py_utils.AssertIsCompatible(extras, actual_extras) - - # Enable accumulators. Note that this must happen prior to the initial - # _AugmentState() below or it will initialize with defaults. - if accumulator_layer: - accumulator_layer.accumulators.Transform(lambda x: x.Enable()) - - acc_state, final_state = _Recurrent( - cell_fn=cell_fn, - cell_grad=cell_grad, - cell_type=cell_type, - stop_fn=stop_fn, - theta=theta, - state0=_AugmentState(state0.DeepCopy(), accumulator_layer), - inputs=inputs, - extras=extras, - accumulator_layer=accumulator_layer, - implicit_captures=implicit_captures).Compute() - - # TODO(b/129159299): The ResetStepSeed below is needed to work around this - # bug, which is a problem with global tensors being shared by different - # inference graphs. It should be removed once the bug is fixed. - py_utils.ResetStepSeed( - py_utils.GenerateSeedFromName(tf.no_op(name='new_step_seed').name)) - - return acc_state, final_state - - -class _Link: - """A link is a pair of channels.""" - - def __init__(self, t, dpair): - # Uses a unique name scope to name the channel. - with tf.name_scope('fwd') as scope: - self.fwd = sendrecv.Channel(t.dtype, t.shape, dpair.send, dpair.recv, - scope) - with tf.name_scope('bak') as scope: - self.bak = sendrecv.Channel(t.dtype, t.shape, dpair.recv, dpair.send, - scope) - - -def _CreateLinks(nmap, dpair): - """Creates links between the send/recv devices for every tensor in nmap.""" - return nmap.Transform(lambda t: _Link(t, dpair)) - - -def _Join(nmap_x, nmap_y, fn): - return py_utils.Transform(fn, nmap_x, nmap_y).Flatten() - - -class _Input: - """Input layers.""" - - def __init__(self, - cell_fn, - cell_out, - cell_grad, - cell_out_grad, - theta, - state0, - accumulator_layer, - inputs, - extras, - out_links, - unused_acc_state=False): - self._cell_fn = cell_fn - self._cell_out = cell_out - self._cell_grad, self._implicit_captures = _GetCellGrad( - cell_fn, - cell_grad, - theta, - state0, - inputs, - accumulator_layer, - allow_implicit_capture=True) - self._cell_out_grad = cell_out_grad - self._theta = theta - self._state0 = state0 - self._accumulator_layer = accumulator_layer - self._inputs = inputs - self._extras = extras - self._out_links = out_links - self._unused_acc_state = unused_acc_state - assert self._extras is not None - - def Compute(self): - """Compute the input layer.""" - - def InputFn(theta, state0, inputs): - state1, extras = self._cell_fn(theta, state0, inputs) - py_utils.AssertIsCompatible(state1, state0) - py_utils.AssertIsCompatible(extras, self._extras) - out = self._cell_out(state1) - sends = _Join(self._out_links, out, lambda l, x: l.fwd.Send(x)) - with tf.control_dependencies(sends): - return state1.Transform(tf.identity), extras.Transform(tf.identity) - - def InputGrad(theta, state0, inputs, extras, dstate1): - """Gradient function for InputFn.""" - recv_dout = self._out_links.Transform(lambda l: l.bak.Recv()) - dstate1 = _Add(dstate1, self._cell_out_grad(recv_dout)) - dtheta, dstate0, dinputs, dcaptures = self._cell_grad( - theta, state0, inputs, extras, dstate1) # pylint: disable=unbalanced-tuple-unpacking - py_utils.AssertIsCompatible(dtheta, self._theta) - py_utils.AssertIsCompatible(dstate0, state0) - py_utils.AssertIsCompatible(dinputs, self._inputs) - if dcaptures is None: - # NOTE: Custom gradient fns can return None if they do not support - # captured tensors. The return value is reserved for the future when - # that may be supported. - dcaptures = _EmptyLike(self._implicit_captures) - py_utils.AssertIsCompatible(dcaptures, self._implicit_captures) - return dtheta, dstate0, dinputs, dcaptures - - return _Recurrent( - cell_fn=InputFn, - cell_grad=InputGrad, - stop_fn=None, - theta=self._theta, - state0=self._state0, - inputs=self._inputs, - extras=self._extras, - accumulator_layer=self._accumulator_layer, - implicit_captures=self._implicit_captures, - unused_acc_state=self._unused_acc_state).Compute() - - -class _Middle: - """Middle layers.""" - - def __init__(self, cell_fn, cell_out, cell_grad, cell_out_grad, theta, state0, - accumulator_layer, in_links, padding, slen_dim, per_step_inputs, - extras, out_links, unused_acc_state): - self._cell_fn = cell_fn - self._cell_out = cell_out - self._cell_grad, self._implicit_captures = _GetCellGrad( - cell_fn, - cell_grad, - theta, - state0, - per_step_inputs, - accumulator_layer, - allow_implicit_capture=True) - self._cell_out_grad = cell_out_grad - self._theta = theta - self._state0 = state0 - self._accumulator_layer = accumulator_layer - self._in_links = in_links - self._padding = padding - self._slen_dim = slen_dim - self._per_step_inputs = per_step_inputs - self._extras = extras - assert self._extras is not None - self._out_links = out_links - self._unused_acc_state = unused_acc_state - - def Compute(self): - """Compute the middle layer.""" - - def MiddleFn(theta, state0, inputs): - del inputs - inputs = self._in_links.Transform(lambda l: l.fwd.Recv()) - state1, extras = self._cell_fn(theta, state0, inputs) - py_utils.AssertIsCompatible(state1, state0) - py_utils.AssertIsCompatible(extras, self._extras) - out = self._cell_out(state1) - sends = _Join(self._out_links, out, lambda l, x: l.fwd.Send(x)) - with tf.control_dependencies(sends): - return (state1.Transform(tf.identity), - py_utils.NestedMap(inputs=inputs, - cell_fn_extras=extras).Transform( - tf.identity)) - - def MiddleGrad(theta, state0, inputs, extras, dstate1): - """Gradient function for MiddleFn.""" - recv_dout = self._out_links.Transform(lambda l: l.bak.Recv()) - dstate1 = _Add(dstate1, self._cell_out_grad(recv_dout)) - dtheta, dstate0, dinputs, dcaptures = self._cell_grad( - theta, state0, extras.inputs, extras.cell_fn_extras, dstate1) # pylint: disable=unbalanced-tuple-unpacking - py_utils.AssertIsCompatible(dtheta, self._theta) - py_utils.AssertIsCompatible(dstate0, state0) - py_utils.AssertIsCompatible(dinputs, self._per_step_inputs) - if dcaptures is None: - # NOTE: Custom gradient fns can return None if they do not support - # captured tensors. The return value is reserved for the future when - # that may be supported. - dcaptures = _EmptyLike(self._implicit_captures) - py_utils.AssertIsCompatible(dcaptures, self._implicit_captures) - sends = _Join(self._in_links, dinputs, lambda l, x: l.bak.Send(x)) - with tf.control_dependencies(sends): - return (dtheta.Transform(tf.identity), dstate0.Transform(tf.identity), - inputs.Transform(tf.zeros_like), - dcaptures.Transform(tf.identity)) - - fake_inputs = py_utils.NestedMap( - fake_input=tf.zeros([self._slen_dim], tf.float32)) - if self._padding is not None: - fake_inputs['padding'] = self._padding - - return _Recurrent( - cell_fn=MiddleFn, - cell_grad=MiddleGrad, - stop_fn=None, - theta=self._theta, - state0=self._state0, - inputs=fake_inputs, - extras=py_utils.NestedMap( - inputs=self._per_step_inputs, cell_fn_extras=self._extras), - accumulator_layer=self._accumulator_layer, - implicit_captures=self._implicit_captures, - unused_acc_state=self._unused_acc_state).Compute() - - -class _Output: - """Output layers.""" - - def __init__(self, cell_fn, cell_grad, theta, state0, accumulator_layer, - in_links, padding, slen_dim, per_step_inputs, extras): - self._cell_fn = cell_fn - self._cell_grad, self._implicit_captures = _GetCellGrad( - cell_fn, - cell_grad, - theta, - state0, - per_step_inputs, - accumulator_layer, - allow_implicit_capture=True) - self._theta = theta - self._state0 = state0 - self._accumulator_layer = accumulator_layer - self._in_links = in_links - self._padding = padding - self._slen_dim = slen_dim - self._per_step_inputs = per_step_inputs - self._extras = extras - assert self._extras is not None - - def Compute(self): - """Compute the output layer.""" - - def OutputFn(theta, state0, inputs): - del inputs - inputs = self._in_links.Transform(lambda l: l.fwd.Recv()) - state1, extras = self._cell_fn(theta, state0, inputs) - py_utils.AssertIsCompatible(state1, state0) - py_utils.AssertIsCompatible(extras, self._extras) - return state1, py_utils.NestedMap(inputs=inputs, cell_fn_extras=extras) - - def OutputGrad(theta, state0, inputs, extras, dstate1): - """Gradient function for OutputFn.""" - dtheta, dstate0, dinputs, dcaptures = self._cell_grad( - theta, state0, extras.inputs, extras.cell_fn_extras, dstate1) # pylint: disable=unbalanced-tuple-unpacking - py_utils.AssertIsCompatible(dtheta, self._theta) - py_utils.AssertIsCompatible(dstate0, state0) - py_utils.AssertIsCompatible(dinputs, self._per_step_inputs) - if dcaptures is None: - # NOTE: Custom gradient fns can return None if they do not support - # captured tensors. The return value is reserved for the future when - # that may be supported. - dcaptures = _EmptyLike(self._implicit_captures) - py_utils.AssertIsCompatible(dcaptures, self._implicit_captures) - sends = _Join(self._in_links, dinputs, lambda l, x: l.bak.Send(x)) - with tf.control_dependencies(sends): - return (dtheta.Transform(tf.identity), dstate0.Transform(tf.identity), - inputs.Transform(tf.zeros_like), - dcaptures.Transform(tf.identity)) - - fake_inputs = py_utils.NestedMap( - fake_input=tf.zeros([self._slen_dim], tf.float32)) - if self._padding is not None: - fake_inputs['padding'] = self._padding - - return _Recurrent( - cell_fn=OutputFn, - cell_grad=OutputGrad, - stop_fn=None, - theta=self._theta, - state0=self._state0, - inputs=fake_inputs, - extras=py_utils.NestedMap( - inputs=self._per_step_inputs, cell_fn_extras=self._extras), - accumulator_layer=self._accumulator_layer, - implicit_captures=self._implicit_captures, - unused_acc_state=False).Compute() - - -def _DependsOn(xs, ys): - """Every x in xs should depend on every y in ys via a data edge.""" - - # TODO(zhifengc): Using the following ops is likely more robust because - # algebra simplifier may remove s - s, t + 0, etc. - # nil: list -> 0 - # first: x, list -> x - # - # If we have nil & first, we can write - # zero = nil(py_utils.Flatten(ys)) - # return [x.Transform(lambda t: first(t, zero)) for x in xs] - def MakeZero(x): - s = tf.reduce_sum(x) - return tf.cast(s - s, tf.float32) - - def SumToZero(nmap_list): - return tf.add_n([MakeZero(x) for x in py_utils.Flatten(nmap_list)]) - - ys_zero = SumToZero(ys) - return [x.Transform(lambda t: t + tf.cast(ys_zero, t.dtype)) for x in xs] - - -def StackedRecurrent(devices, - cell_fns, - cell_grads, - cell_outs, - cell_out_grads, - thetas, - init_states, - inputs, - accumulator_layers=None, - unused_acc_state=False): - """Computes stacked recurrent neural nets placed on various devices. - - Conceptually, StackedRecurrent() computes the following:: - - for (device, cell_fn, cell_out, cell_grad, theta, state0) in zip( - (devices, cell_fns, cell_outs, cell_grads, thetas, init_states): - with tf.device(device): - state1, _ = Recurrent(theta, state0, inputs, cell_fn, cell_grad) - outputs = cell_out(state1) - inputs = outputs # Next layer's input is this layer's output - return outputs - - The only difference is that StackedRecurrent implements a model parallelism - so that all layers computation can happen concurrently. - - Args: - devices: A list of N tensorflow device names. - cell_fns: If a list of N recurrent cell function, cell_fns[i] must meet the - same requirement as Recurrent() requires its cell_fn argument. Otherwise, - applies to all layers. - cell_grads: If a list of N recurrent cell gradient function, cell_grads[i] - must meet the same requirement as Recurrent() requires its cell_grad - argument. Otherwise, applies to all layers. - cell_outs: If a list of N function, cell_outs[i] takes the state computed by - cell_fns[i] and returns the input for the next layer. These functions are - expected to be simple and just do renaming of fields. Otherwise, applies - to all layers. - cell_out_grads: If a list of N function, cell_out_grads[i] is often the - reverse of cell_outs[i]. Otherwise, applies to all layers. - thetas: A list of N weights NestedMap. thetas[i] must meet the same - requirement as Recurrent() requires its theta argument. - init_states: A list of N initial state NestedMap. init_states[i] must meet - the same requirement as Recurrent() requires its state0 argument. - inputs: Inputs to the 1st layer of the stacked recurrent neural nets. A - NestedMap. - accumulator_layers: A list of layers whose accumulators will be managed such - that they carry to the output state in `FProp` and are disabled for - gradients. Uses the state key `accumulators`. Default to None where no - accumulator values will be carried. - unused_acc_state: If True, we shink all the layer's acc_state to [num_ts] - except the last layer(_Output). - - Returns: - Tuple (output, states): - - - The last layer's output (accumulated states). - - The list of final state NestedMap. One for each layer. - """ - # Enable rendezvous sharing when using tf.function, since it needs to do - # send/recv across function boundary. - # pylint: disable=protected-access - with py_utils._SharedRendezvousScope(shared_rendezvous=True): - return _StackedRecurrent(devices, cell_fns, cell_grads, cell_outs, - cell_out_grads, thetas, init_states, inputs, - accumulator_layers, unused_acc_state) - # pylint: enable=protected-access - - -def _StackedRecurrent(devices, cell_fns, cell_grads, cell_outs, cell_out_grads, - thetas, init_states, inputs, accumulator_layers, - unused_acc_state): - """Implementation of StackedRecurrent, see StackedRecurrent for details.""" - num_layers = len(devices) - assert num_layers - - def _MakeList(fns): - if not isinstance(fns, (list, tuple)): - return [fns] * num_layers - else: - assert num_layers == len(fns) - return fns - - cell_fns = _MakeList(cell_fns) - cell_grads = _MakeList(cell_grads) - cell_outs = _MakeList(cell_outs) - cell_out_grads = _MakeList(cell_out_grads) - accumulator_layers = accumulator_layers or [None] * num_layers - assert num_layers == len(thetas) - assert all(isinstance(x, py_utils.NestedMap) for x in thetas) - assert num_layers == len(init_states) - assert all(isinstance(x, py_utils.NestedMap) for x in init_states) - assert isinstance(inputs, py_utils.NestedMap) - - if py_utils.use_tpu(): - # If this error happens, the number of splits must be increased (e.g. - # worker_split_size in trainer/tpu.sh), or the number of rnn layers - # decreased. - # TODO(cwhipkey): lift this restriction by grouping layers by device and - # having a device handle a contiguous run of layers, and have them loop - # over the layers in the cell fns. - assert len(devices) == len(set(devices)), ( - 'StackedRecurrent must provide a different device for each layer ' - 'when run on TPU. devices passed were: %s' % str(devices)) - - if num_layers == 1: - # Simple case, just use Recurrent() directly. - with tf.device(devices[0]): - acc_states, final = Recurrent( - theta=thetas[0], - state0=init_states[0], - inputs=inputs, - cell_fn=cell_fns[0], - cell_grad=cell_grads[0], - accumulator_layer=accumulator_layers[0]) - # Just the accumulated states. - return cell_outs[0](acc_states), final - - # We add explicit data dependencies between layer-i's theta/state0 - # and layer-(i-1)'s theta/state0, layer-0's theta/state0 has an - # explicit data dependency on inputs. These extra data dependencies - # ensure that if layer-i's theta/state0 is used in tf.gradient, all - # layers above's backprop are triggered. - prev = [inputs] - for i in range(num_layers): - with tf.device(devices[i]): - thetas[i], init_states[i] = _DependsOn([thetas[i], init_states[i]], prev) - prev = [thetas[i], init_states[i]] - - def ExpectedOutputOfLayers(): - """Estimate what tensor dtypes and shapes output by each layer.""" - - def ZerosLikeRequireShape(t): - assert t.shape.is_fully_defined() - return tf.zeros_like(t) - - if py_utils.use_tpu(): - transform_fn = ZerosLikeRequireShape - else: - transform_fn = tf.zeros_like - - expected_output_by_layers = [] - xs = _Index(inputs, 0) - for i in range(num_layers): - # Disable accumulators and step_seed since this is not a real call to - # cell_fns[i]. They will be re-enabled in _Recurrent. - if accumulator_layers[i]: - accumulator_layers[i].accumulators.Transform(lambda x: x.Disable()) - step_seed = py_utils.GetStepSeed() - state1, extras = cell_fns[i](thetas[i], init_states[i], xs) - py_utils.ResetStepSeed(step_seed) - # only dtype and shape is needed. - xs = cell_outs[i](state1) - expected_output_by_layers += [ - py_utils.NestedMap( - xs=xs.Transform(transform_fn), - extras=extras.Transform(transform_fn)) - ] - return expected_output_by_layers - - expected_output_by_layers = ExpectedOutputOfLayers() - - # Sequence length. We assume it's a grid we are building. - slen_dim = _SeqLenDim(inputs) - - assert num_layers >= 2 - layers = [] - - padding = FlattenPadding(inputs.get('padding', None)) - - # Builds the input layer. - out_links = _CreateLinks(expected_output_by_layers[0].xs, - DevicePair(devices[0], devices[1])) - - # Enable accumulators. Note that this must happen prior to the initial - # _AugmentState() below or it will initialize with defaults. - for accumulator_layer in accumulator_layers: - if accumulator_layer: - accumulator_layer.accumulators.Transform(lambda x: x.Enable()) - - inp_l = _Input( - cell_fn=cell_fns[0], - cell_grad=cell_grads[0], - cell_out=cell_outs[0], - cell_out_grad=cell_out_grads[0], - theta=thetas[0], - state0=_AugmentState(init_states[0].DeepCopy(), accumulator_layers[0]), - accumulator_layer=accumulator_layers[0], - inputs=inputs, - extras=expected_output_by_layers[0].extras, - out_links=out_links, - unused_acc_state=unused_acc_state) - layers += [inp_l] - - # Builds the intermediate layers. - for i in range(1, num_layers - 1): - in_links = out_links - out_links = _CreateLinks(expected_output_by_layers[i].xs, - DevicePair(devices[i], devices[i + 1])) - mid_l = _Middle( - cell_fn=cell_fns[i], - cell_grad=cell_grads[i], - cell_out=cell_outs[i], - cell_out_grad=cell_out_grads[i], - theta=thetas[i], - state0=_AugmentState(init_states[i].DeepCopy(), accumulator_layers[i]), - accumulator_layer=accumulator_layers[i], - in_links=in_links, - padding=padding, - slen_dim=slen_dim, - per_step_inputs=expected_output_by_layers[i - 1].xs, - extras=expected_output_by_layers[i].extras, - out_links=out_links, - unused_acc_state=unused_acc_state) - layers += [mid_l] - - # Builds the final output layer. - in_links = out_links - del out_links - out_l = _Output( - cell_fn=cell_fns[-1], - cell_grad=cell_grads[-1], - theta=thetas[-1], - state0=_AugmentState(init_states[-1].DeepCopy(), accumulator_layers[-1]), - accumulator_layer=accumulator_layers[-1], - in_links=in_links, - padding=padding, - slen_dim=slen_dim, - per_step_inputs=expected_output_by_layers[-2].xs, - extras=expected_output_by_layers[-1].extras) - layers += [out_l] - - assert len(layers) == num_layers - - anchor = 0 - final_states = [] - for (dev, layer) in zip(devices, layers): - # Computes each layer on their designated device. - with tf.device(dev): - acc_states, final = layer.Compute() # Don't care of final state yet. - final_states.append(final) - - # We add every number output by the layer (s) and computes a - # zero scalar: (s - s), as an anchor. Anchors are added - # sequentially and added to the final layer's output. This way, - # we ensure that the final output depends on every previous - # layer through data dependencies. This is a hack to ensure that - # tf.gradient will follow some data dependencies path to start - # the Backward loop for each layer. - # - # TODO(zhifengc): We can write, if we have nil & first ops: - # anchor += [nil(py_utils.Flatten(acc_states))] - # And finally, - # return acc_states.Transform(lambda x: first(x, anchor)) - def ComputeAnchor(x): - # For each - s = tf.add_n([tf.reduce_sum(_) for _ in x.Flatten()]) - return s - s - - anchor = ComputeAnchor(acc_states) + anchor - - # The last layer's output is the real output that matters. However, - # to make the previous layers backprop work, we need to make sure - # the returned value has data dependencies on the previous layers. - # 'anchor' is guaranteed to be a scalar 0 and hence adding it to the - # final output does not change its numerical value. - with tf.device(devices[-1]): - outputs = cell_outs[-1](acc_states.Transform(lambda x: x + anchor)) - - # TODO(b/129159299): The ResetStepSeed below is needed to work around this - # bug, which is a problem with global tensors being shared by different - # inference graphs. It should be removed once the bug is fixed. - py_utils.ResetStepSeed( - py_utils.GenerateSeedFromName(tf.no_op(name='new_step_seed').name)) - - return outputs, final_states diff --git a/lingvo/core/recurrent_gpu_test.py b/lingvo/core/recurrent_gpu_test.py deleted file mode 100644 index be778c2f..00000000 --- a/lingvo/core/recurrent_gpu_test.py +++ /dev/null @@ -1,59 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lingvo recurrent on gpu.""" - -import lingvo.compat as tf -from lingvo.core import py_utils -from lingvo.core import recurrent -from lingvo.core import test_utils - - -class RecurrentGpuTest(test_utils.TestCase): - - def testRecurrent(self): - """Run on GPU locally with --test_output=streamed --config=cuda.""" - - def Sum(theta, state, inputs): - next_state = py_utils.NestedMap() - v = tf.reduce_sum(tf.one_hot(inputs.one_hot, depth=2) * theta.x, axis=0) - next_state.sum = state.sum + v - return next_state, py_utils.NestedMap() - - with self.session(use_gpu=True): - - theta = py_utils.NestedMap() - theta.x = tf.constant([-1.0, 2.0]) - state = py_utils.NestedMap() - state.sum = tf.constant(0.0) - inputs = py_utils.NestedMap() - inputs.one_hot = tf.constant([0, 1, 1], dtype=tf.int32) - - # sum = -1 + 2 + 2 - ret = recurrent.Recurrent(theta, state, inputs, Sum) - - acc, state = self.evaluate(ret) - self.assertAllClose(acc.sum, [-1., 1., 3.]) - self.assertAllClose(state.sum, 3.) - - y = ret[1].sum - dx, d_inputs = tf.gradients(ys=[y], xs=[theta.x, inputs.one_hot]) - tf.logging.info('d(inputs) = %s', d_inputs) - dx_val = self.evaluate(dx) - self.assertAllClose(dx_val, [1, 2]) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/recurrent_test.py b/lingvo/core/recurrent_test.py deleted file mode 100644 index f8302d98..00000000 --- a/lingvo/core/recurrent_test.py +++ /dev/null @@ -1,911 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for recurrent.""" - -from absl.testing import parameterized -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils -from lingvo.core import recurrent -from lingvo.core import symbolic -from lingvo.core import test_utils -import numpy as np - -FLAGS = tf.flags.FLAGS - - -def _ApplyPadding(padding, v_no_pad, v_pad): - if padding is not None: - padding = tf.cast(padding, v_no_pad.dtype) - return (1 - padding) * v_no_pad + padding * v_pad - return v_no_pad - - -def _ReferenceStaticUnroll(theta, state0, inputs, cell_fn): - """Statically unrolls a `cell_fn` wrt inputs as `recurrent.Recurrent()` does. - - This is not a complete implementation but should work reasonably for - calls that do not have a custom cell_grad, extras or accumulators and - can help check expectations. - - Args: - theta: weights. A `.NestedMap`. - state0: initial state. A `.NestedMap`. - inputs: inputs. A `.NestedMap`. Tensors must have static shape. - cell_fn: A python function, which computes:: - state1, extras = cell_fn(theta, state0, inputs[t, :]) - - Returns: - accumulate and final state. - """ - flat_inputs = inputs.Flatten() - flat_state = state0.Flatten() - step_count = flat_inputs[0].shape[0] - acc_state = [[None] * step_count for t in flat_state] - for i in range(step_count): - flat_step_inputs = [t[i] for t in flat_inputs] - step_inputs = inputs.Pack(flat_step_inputs) - state1, unused_extras = cell_fn(theta, state0, step_inputs) - for j, state_step in zip(range(len(acc_state[0])), state1.Flatten()): - acc_state[j][i] = state_step - state0 = state1 - return state0.Pack(acc_state), state1 - - -def _Poly(theta, state, inputs): - next_state = py_utils.NestedMap() - next_state.value = state.value + inputs.coeff * state.x_power - next_state.x_power = state.x_power * theta.x - return next_state, py_utils.NestedMap() - - -class _IncrementAccumulator(base_layer.Accumulator): - - def DefaultValue(self): - return tf.convert_to_tensor(0.0) - - def Update(self, increment_by): - initial = self.GetValue() - self.SetValue(initial + tf.convert_to_tensor(increment_by)) - - -class _SampleAccumulatorLayer(base_layer.BaseLayer): - - def __init__(self, params): - super().__init__(params) - self.accumulator_name = 'sample_accumulator' - self.RegisterAccumulator(self.accumulator_name, _IncrementAccumulator()) - - -def Elman(theta, state0, inputs): - h0, w, b, x = state0.h, theta.w, theta.b, inputs.x - xw = py_utils.Matmul(tf.concat([x, h0], axis=1), w) # 1st part - # 2nd part - padding = inputs.get('padding', None) - h1 = _ApplyPadding(padding, v_no_pad=tf.sigmoid(xw + b), v_pad=state0.h) - - state1 = py_utils.NestedMap(h=h1) - if padding is not None: - state1.padding = inputs.padding - - return (state1, py_utils.NestedMap(h=h1)) - - -def ElmanGrad(theta, state0, inputs, extras, dstate1): - - @tf.function(autograph=False) - def Grad(h0, w, b, x, padding, h1, dh1): - del b - dh1_orig = dh1 - dh1 = _ApplyPadding(padding, dh1, tf.zeros_like(dh1, dtype=dh1.dtype)) - - # We hand-roll the gradient for the 2nd half of the cell as a demo. - # h1 = tf.sigmoid(xw + b) - # 𝛔'(x) = ((1 - 𝛔(x)) * 𝛔(x)) - dxwb = (dh1 * (1 - h1) * h1) - dxw, db = dxwb, tf.reduce_sum(dxwb, axis=0) - - # Uses tf.gradient for the 1nd half of the cell as a demo. - xw = py_utils.Matmul(tf.concat([x, h0], axis=1), w) - dh0, dx, dw = tf.gradients(ys=[xw], xs=[h0, x, w], grad_ys=[dxw]) - - dh0 = _ApplyPadding(padding, dh0, dh1_orig) - - return dh0, dx, dw, db - - dh0, dx, dw, db = Grad(state0.h, theta.w, theta.b, inputs.x, - inputs.get('padding', 0), extras.h, dstate1.h) - dstate0 = py_utils.NestedMap(h=dh0) - dinputs = py_utils.NestedMap(x=dx) - if 'padding' in dstate1: - dstate0.padding = dstate1.padding - dinputs.padding = dstate1.padding - return (py_utils.NestedMap(w=dw, b=db), dstate0, dinputs, None) - - -def ElmanOut(state1): - return py_utils.NestedMap(x=state1.h, padding=state1.padding) - - -def ElmanOutGrad(dout): - return py_utils.NestedMap(h=dout.x, padding=dout.padding) - - -def RecurrentTestParameters(test_fn): - - def WrappedTestFn(self, use_tf_function): - # TODO(laigd): remove this check when 312743821 and 313682500 are in the - # release. - if use_tf_function and tf.compat.v1.__version__ < '2.3.0': - return - FLAGS.if_use_tf_function = use_tf_function - FLAGS.while_loop_use_tf_function = use_tf_function - FLAGS.call_defun_use_tf_function = use_tf_function - test_fn(self) - - decorator = parameterized.named_parameters( - ('_defun', False), - ('_function', True), - ) - return decorator(WrappedTestFn) - - -class RecurrentTest(test_utils.TestCase, parameterized.TestCase): - - @RecurrentTestParameters - def testBasic(self): - - with self.session(): - - theta = py_utils.NestedMap() - theta.x = tf.constant(2.0) - state = py_utils.NestedMap() - state.value = tf.constant(0.0) - state.x_power = tf.constant(1.0) - inputs = py_utils.NestedMap() - inputs.coeff = tf.constant([1., 2., 3.]) - - # x = 2 - # 1 + 2*x + 3*x^2 - ret = recurrent.Recurrent(theta, state, inputs, _Poly) - - acc, state = self.evaluate(ret) - self.assertAllClose(acc.value, [1., 5., 17.]) - self.assertAllClose(acc.x_power, [2., 4., 8.]) - self.assertAllClose(state.value, 17.) - self.assertAllClose(state.x_power, 8.) - - y = ret[1].value - dx, d_coeff = tf.gradients(ys=[y], xs=[theta.x, inputs.coeff]) - dx_val, d_coeff_val = self.evaluate([dx, d_coeff]) - - # 2 + 6*x - self.assertAllClose(dx_val, 14.) - self.assertAllClose(d_coeff_val, [1., 2., 4.]) - - # acc = [1, 1+2x, 1+2x+3x^2] - # sum(acc) = 3 + 4x + 3x^2 - acc = ret[0].value - dx, d_coeff = tf.gradients( - ys=[tf.reduce_sum(acc)], xs=[theta.x, inputs.coeff]) - dx_val, d_coeff_val = self.evaluate([dx, d_coeff]) - # 4 + 6*x - self.assertAllClose(dx_val, 16.) - self.assertAllClose(d_coeff_val, [3., 4., 4.]) - - @RecurrentTestParameters - def testBasicWithAccumulator(self): - - with self.session(): - - p = _SampleAccumulatorLayer.Params() - p.name = 'sample' - accum_layer = _SampleAccumulatorLayer(p) - accum_obj = accum_layer.accumulators[accum_layer.accumulator_name] - - theta = py_utils.NestedMap() - theta.x = tf.constant(2.0) - state = py_utils.NestedMap() - state.value = tf.constant(0.0) - state.x_power = tf.constant(1.0) - inputs = py_utils.NestedMap() - inputs.coeff = tf.constant([1., 2., 3.]) - - def _CellFn(theta, state, inputs): - print('TEST ACCUM WITHIN CellFn = ', accum_obj.GetValue()) - accum_obj.Update(inputs.coeff) - return _Poly(theta, state, inputs) - - # By doing one accumulate prior to recurrent, we ensure that incoming - # recurrent state is preserved. - accum_obj.Update(10.) - - # x = 2 - # 1 + 2*x + 3*x^2 - ret = recurrent.Recurrent( - theta, state, inputs, _CellFn, accumulator_layer=accum_layer) - - # Verify bprop. - y = ret[1].value - dx, d_coeff = tf.gradients(ys=[y], xs=[theta.x, inputs.coeff]) - dx_val, d_coeff_val = self.evaluate([dx, d_coeff]) - - # 2 + 6*x - self.assertAllClose(dx_val, 14.) - self.assertAllClose(d_coeff_val, [1., 2., 4.]) - - # acc = [1, 1+2x, 1+2x+3x^2] - # sum(acc) = 3 + 4x + 3x^2 - acc = ret[0].value - dx, d_coeff = tf.gradients( - ys=[tf.reduce_sum(acc)], xs=[theta.x, inputs.coeff]) - dx_val, d_coeff_val = self.evaluate([dx, d_coeff]) - # 4 + 6*x - self.assertAllClose(dx_val, 16.) - self.assertAllClose(d_coeff_val, [3., 4., 4.]) - - # Verify fprop. - (acc, state), accum_obj_value = self.evaluate((ret, accum_obj.GetValue())) - - # Verify that accumulators don't change fprop results. - self.assertAllClose(acc.value, [1., 5., 17.]) - self.assertAllClose(acc.x_power, [2., 4., 8.]) - self.assertAllClose(state.value, 17.) - self.assertAllClose(state.x_power, 8.) - - # Verify accumulator (should be 10 (initial increment) + 1 + 2 + 3). - self.assertEqual(0, accum_obj._disable_count) - self.assertAllClose([accum_obj_value], [16.0]) - - @RecurrentTestParameters - def testTimeBasedStopFn(self): - - with self.session(): - - def StopFn(t, unused_theta, unused_state): - # This stops after 3 iterations. - return t >= 3 - - theta = py_utils.NestedMap() - theta.x = tf.constant(2.0) - state = py_utils.NestedMap() - state.value = tf.constant(0.0) - state.x_power = tf.constant(1.0) - inputs = py_utils.NestedMap() - inputs.coeff = tf.constant([1., 2., 3., 4.]) - - # x = 2 - # 1 + 2*x + 3*x^2 - ret = recurrent.Recurrent(theta, state, inputs, _Poly, stop_fn=StopFn) - - acc, state = self.evaluate(ret) - self.assertAllClose([1., 5., 17., 0.], acc.value) - self.assertAllClose([2., 4., 8., 0.], acc.x_power) - self.assertAllClose(17., state.value) - self.assertAllClose(8., state.x_power) - - y = ret[1].value - dx, d_coeff = tf.gradients(ys=[y], xs=[theta.x, inputs.coeff]) - dx_val, d_coeff_val = self.evaluate([dx, d_coeff]) - - # 2 + 6*x - self.assertAllClose(14., dx_val) - self.assertAllClose([1., 2., 4., 0.], d_coeff_val) - - # acc = [1, 1+2x, 1+2x+3x^2] - # sum(acc) = 3 + 4x + 3x^2 - acc = ret[0].value - dx, d_coeff = tf.gradients( - ys=[tf.reduce_sum(acc)], xs=[theta.x, inputs.coeff]) - dx_val, d_coeff_val = self.evaluate([dx, d_coeff]) - # 4 + 6*x - self.assertAllClose(16., dx_val) - self.assertAllClose([3., 4., 4., 0.], d_coeff_val) - - @RecurrentTestParameters - def testStateBasedStopFn(self): - - with self.session(): - - def StopFn(unused_t, unused_theta, state): - # This stops after 3 iterations. - return state.value >= 15. - - theta = py_utils.NestedMap() - theta.x = tf.constant(2.0) - state = py_utils.NestedMap() - state.value = tf.constant(0.0) - state.x_power = tf.constant(1.0) - inputs = py_utils.NestedMap() - inputs.coeff = tf.constant([1., 2., 3., 4.]) - - # x = 2 - # 1 + 2*x + 3*x^2 - ret = recurrent.Recurrent(theta, state, inputs, _Poly, stop_fn=StopFn) - - acc, state = self.evaluate(ret) - self.assertAllClose([1., 5., 17., 0.], acc.value) - self.assertAllClose([2., 4., 8., 0.], acc.x_power) - self.assertAllClose(17., state.value) - self.assertAllClose(8., state.x_power) - - y = ret[1].value - dx, d_coeff = tf.gradients(ys=[y], xs=[theta.x, inputs.coeff]) - dx_val, d_coeff_val = self.evaluate([dx, d_coeff]) - - # 2 + 6*x - self.assertAllClose(14., dx_val) - self.assertAllClose([1., 2., 4., 0.], d_coeff_val) - - # acc = [1, 1+2x, 1+2x+3x^2] - # sum(acc) = 3 + 4x + 3x^2 - acc = ret[0].value - dx, d_coeff = tf.gradients( - ys=[tf.reduce_sum(acc)], xs=[theta.x, inputs.coeff]) - dx_val, d_coeff_val = self.evaluate([dx, d_coeff]) - # 4 + 6*x - self.assertAllClose(16., dx_val) - self.assertAllClose([3., 4., 4., 0.], d_coeff_val) - - @RecurrentTestParameters - def testStopFnNotTriggeredBeforeEOS(self): - - with self.session(): - - def StopFn(t, unused_theta, unused_state): - # The input sequence is only length 4, so this is never true. - # However, the Recurrent call should still terminate after iteration 4. - return t >= 5 - - theta = py_utils.NestedMap() - theta.x = tf.constant(2.0) - state = py_utils.NestedMap() - state.value = tf.constant(0.0) - state.x_power = tf.constant(1.0) - inputs = py_utils.NestedMap() - inputs.coeff = tf.constant([1., 2., 3., 4.]) - - # x = 2 - # 1 + 2*x + 3*x^2 + 4*x^3 - ret = recurrent.Recurrent(theta, state, inputs, _Poly, stop_fn=StopFn) - - acc, state = self.evaluate(ret) - self.assertAllClose([1., 5., 17., 49.], acc.value) - self.assertAllClose([2., 4., 8., 16.], acc.x_power) - self.assertAllClose(49., state.value) - self.assertAllClose(16., state.x_power) - - y = ret[1].value - dx, d_coeff = tf.gradients(ys=[y], xs=[theta.x, inputs.coeff]) - dx_val, d_coeff_val = self.evaluate([dx, d_coeff]) - - # 2 + 6*x + 12*x^2 - self.assertAllClose(62., dx_val) - self.assertAllClose([1., 2., 4., 8.], d_coeff_val) - - # acc = [1, 1+2x, 1+2x+3x^2, 1+2x+3x^2+4x^3] - # sum(acc) = 4 + 6x + 6x^2 + 4x^3 - acc = ret[0].value - dx, d_coeff = tf.gradients( - ys=[tf.reduce_sum(acc)], xs=[theta.x, inputs.coeff]) - dx_val, d_coeff_val = self.evaluate([dx, d_coeff]) - # 6 + 12*x + 12*x^2 - self.assertAllClose(78., dx_val) - self.assertAllClose([4., 6., 8., 8.], d_coeff_val) - - @RecurrentTestParameters - def testCapture(self): - - with self.session(): - - theta = py_utils.NestedMap() - theta.x = tf.constant(2.0) - state0 = py_utils.NestedMap() - state0.value = tf.constant(0.0) - inputs = py_utils.NestedMap() - inputs.coeff = tf.constant([1., 2., 3.]) - captured = tf.constant(1.2, name='captured_const') - - def CellFn(theta, state, inputs): - next_state = py_utils.NestedMap() - # Captured is pulled from outside of the function and captured via the - # internal Defun. - next_state.value = state.value + inputs.coeff * captured * theta.x - return next_state, py_utils.NestedMap() - - # Run static fprop reference implementation. - ref_acc, ref_staten = _ReferenceStaticUnroll(theta, state0, inputs, - CellFn) - ref_acc_values, ref_staten_values = self.evaluate((ref_acc, ref_staten)) - print('Ref fprop: acc =', ref_acc, ', stateN =', ref_staten) - print('Ref fprop values: acc =', ref_acc_values, ', stateN =', - ref_staten_values) - self.assertAllClose(ref_acc_values.value, [2.4, 7.2, 14.4]) - self.assertAllClose(ref_staten_values.value, 14.4) - - # Run real fprop implementation. - real_acc, real_staten = recurrent.Recurrent( - theta, state0, inputs, CellFn, allow_implicit_capture=True) - real_acc_values, real_staten_values = self.evaluate( - (real_acc, real_staten)) - print('Real fprop: acc =', real_acc, ', stateN =', real_staten) - print('Real fprop values: acc =', real_acc_values, ', stateN =', - real_staten_values) - self.assertAllClose(ref_acc_values.value, real_acc_values.value) - self.assertAllClose(ref_staten_values.value, real_staten_values.value) - - # BProp real vs ref of stateN. - ref_dx, ref_dcaptured = tf.gradients( - ys=[ref_staten.value], xs=[theta.x, captured]) - ref_dx_values, ref_dcaptured_values = self.evaluate( - [ref_dx, ref_dcaptured]) - real_dx, real_dcaptured = tf.gradients( - ys=[real_staten.value], xs=[theta.x, captured]) - real_dx_values, real_dcaptured_values = self.evaluate( - [real_dx, real_dcaptured]) - print('Ref Dstate/[dx,dcaptured] =', ref_dx_values, ', ', - ref_dcaptured_values) - print('Real Dstate/[dx,dcaptured] =', real_dx_values, ', ', - real_dcaptured_values) - self.assertAllClose(ref_dx_values, 7.2) - self.assertAllClose(ref_dcaptured_values, 12.0) - self.assertAllClose(ref_dx_values, real_dx_values) - self.assertAllClose(ref_dcaptured_values, real_dcaptured_values) - - @RecurrentTestParameters - def testCaptureDisallowed(self): - - with self.session() as unused_sess: - - theta = py_utils.NestedMap() - theta.x = tf.constant(2.0) - state0 = py_utils.NestedMap() - state0.value = tf.constant(0.0) - inputs = py_utils.NestedMap() - inputs.coeff = tf.constant([1., 2., 3.]) - captured = tf.constant(1.2, name='captured_const') - - def CellFn(theta, state, inputs): - next_state = py_utils.NestedMap() - # Captured is pulled from outside of the function and captured via the - # internal Defun. - next_state.value = state.value + inputs.coeff * captured * theta.x - return next_state, py_utils.NestedMap() - - # Run real fprop implementation. - with self.assertRaisesRegex(ValueError, 'implicit capture is disabled'): - unused_real_acc, unused_real_staten = recurrent.Recurrent( - theta, state0, inputs, CellFn, allow_implicit_capture=False) - - @RecurrentTestParameters - def testSymbolToTensorMap(self): - """Tests that cell_fn can rely on the contextual symbol-to-tensor map.""" - - x = symbolic.Symbol('x') - y = symbolic.Symbol('y') - - def PlusWXT(theta, state, inputs): - """state.value += theta.w * x * inputs.t.""" - next_state = py_utils.NestedMap() - x_tensor = symbolic.EvalExpr(symbolic.TENSOR_VALUES, x) - next_state.value = state.value + theta.w * x_tensor * inputs.t - return next_state, py_utils.NestedMap() - - def PlusWXTGrad(theta, state0, inputs, extras, dstate1): - """Gradient function for PlusWXT.""" - del state0, extras - x_tensor = symbolic.EvalExpr(symbolic.TENSOR_VALUES, x) - dtheta = py_utils.NestedMap(w=dstate1.value * x_tensor * inputs.t) - dstate0 = py_utils.NestedMap(value=dstate1.value) - dinputs = py_utils.NestedMap(t=dstate1.value * theta.w * x_tensor) - return dtheta, dstate0, dinputs, None - - with self.session(): - theta = py_utils.NestedMap(w=tf.constant(1., name='w')) - state0 = py_utils.NestedMap(value=tf.constant(0., name='value')) - inputs = py_utils.NestedMap(t=tf.constant([1., 2., 3.], name='t')) - - # With automatic cell_grad. - with symbolic.SymbolToValueMap(symbolic.TENSOR_VALUES, { - x: tf.constant(7., name='x7'), - y: 8 - }): - x_tensor = symbolic.EvalExpr(symbolic.TENSOR_VALUES, x) - _, state1 = recurrent.Recurrent(theta, state0, inputs, PlusWXT) - dw = tf.gradients(ys=[state1.value], xs=[theta.w])[0] - dx = tf.gradients(ys=[state1.value], xs=[x_tensor])[0] - final_value, x_val, dx_val, dw_val = self.evaluate( - [state1.value, x_tensor, dx, dw]) - self.assertEqual(x_val, 7) - self.assertEqual(final_value, x_val * (1. + 2. + 3.)) - self.assertEqual(dw_val, x_val * (1. + 2. + 3.)) - self.assertEqual(dx_val, (1. + 2. + 3.)) - - # With manual cell_grad. - with symbolic.SymbolToValueMap(symbolic.TENSOR_VALUES, - {x: tf.constant(5., name='x5')}): - x_tensor = symbolic.EvalExpr(symbolic.TENSOR_VALUES, x) - _, state1 = recurrent.Recurrent( - theta, state0, inputs, PlusWXT, cell_grad=PlusWXTGrad) - dw = tf.gradients(ys=[state1.value], xs=[theta.w])[0] - dx = tf.gradients(ys=[state1.value], xs=[x_tensor])[0] - final_value, x_val, dx_val, dw_val = self.evaluate( - [state1.value, x_tensor, dx, dw]) - self.assertEqual(x_val, 5) - self.assertEqual(final_value, x_val * (1. + 2. + 3.)) - self.assertEqual(dw_val, x_val * (1. + 2. + 3.)) - self.assertEqual(dx_val, (1. + 2. + 3.)) - - @RecurrentTestParameters - def testStatefulCellFn(self): - - def Rand(theta, state, inputs): - del theta - next_state = py_utils.NestedMap() - next_state.value = ( - state.value + - inputs.coeff * tf.random.uniform(shape=[], dtype=state.value.dtype)) - return next_state, py_utils.NestedMap() - - with self.session(): - theta = py_utils.NestedMap() - state = py_utils.NestedMap() - state.value = tf.constant(0.0) - inputs = py_utils.NestedMap() - inputs.coeff = tf.constant([1., 2., 3.]) - - with self.assertRaisesRegex(ValueError, 'stateful.*random_uniform'): - recurrent.Recurrent(theta, state, inputs, Rand, check_stateful_ops=True) - - @RecurrentTestParameters - def testNestedCellFn(self): - """Tests when cell_fn calls another function.""" - - @tf.function(autograph=False) - def RandWithCoeff(coeff): - return coeff * tf.random.uniform(shape=[], dtype=coeff.dtype) - - def Rand(theta, state, inputs): - del theta - next_state = py_utils.NestedMap() - next_state.value = state.value + RandWithCoeff(inputs.coeff) - return next_state, py_utils.NestedMap() - - @tf.function(autograph=False) - def Coeff(coeff): - return coeff * 2 - - def Deterministic(theta, state, inputs): - del theta - next_state = py_utils.NestedMap() - next_state.value = state.value + Coeff(inputs.coeff) - return next_state, py_utils.NestedMap() - - with self.session(): - theta = py_utils.NestedMap() - state = py_utils.NestedMap() - state.value = tf.constant(0.0) - inputs = py_utils.NestedMap() - inputs.coeff = tf.constant([1., 2., 3.]) - - recurrent.Recurrent( - theta, state, inputs, Deterministic, check_stateful_ops=True) - with self.assertRaisesRegex(ValueError, 'cell_fn contains stateful ops:'): - recurrent.Recurrent(theta, state, inputs, Rand, check_stateful_ops=True) - - @RecurrentTestParameters - def testSeqLenActual(self): - for value, expected in [([[1.0], [1.0], [1.0]], - [0, 3]), ([[1.0], [1.0], [0.0]], - [2, 0]), ([[1.0], [0.0], [1.0]], [1, 1]), - ([[1.0], [0.0], [0.0]], - [1, 0]), ([[0.0], [1.0], [1.0]], - [0, 2]), ([[0.0], [1.0], [0.0]], [0, 0]), - ([[0.0], [0.0], [1.0]], [0, 1]), ([[0.0], [0.0], - [0.0]], [0, 0])]: - with self.session(): - inputs = py_utils.NestedMap() - inputs.padding = tf.constant(value) - - slen_op = recurrent._SeqPaddingLength(inputs) - slen, = self.evaluate([slen_op]) - self.assertEqual(expected, slen) - - @staticmethod - def Rand(shape): - return tf.random.uniform(shape, minval=-0.2, maxval=0.2, dtype=tf.float64) - - def _testElmanHelper(self, seqlen, use_grad, stop_fn=None): - with self.session(): - tf.random.set_seed(342462) - - batch = 3 - dims = 4 - theta = py_utils.NestedMap() - theta.w = self.Rand([2 * dims, dims]) - theta.b = self.Rand([dims]) - state0 = py_utils.NestedMap() - state0.h = self.Rand([batch, dims]) - inputs = py_utils.NestedMap() - inputs.x = self.Rand([seqlen, batch, dims]) - - # Static unrolled. - s = state0 - out = [] - for i in range(seqlen): - inp = py_utils.NestedMap() - inp.x = inputs.x[i, :] - s, _ = Elman(theta, s, inp) - out += [s.h] - if stop_fn and stop_fn(i + 1, theta, s): - out += [tf.zeros_like(out[-1]) for _ in range(seqlen - i - 1)] - break - acc0, final0 = tf.stack(out), s.h - loss0 = tf.reduce_sum(acc0) + tf.reduce_sum(final0) - (dw0, db0, dh0, - di0) = tf.gradients(loss0, [theta.w, theta.b, state0.h, inputs.x]) - - # Uses the Recurrent() library. - acc1, final1 = recurrent.Recurrent( - theta=theta, - state0=state0, - inputs=inputs, - cell_fn=Elman, - cell_grad=ElmanGrad if use_grad else None, - stop_fn=stop_fn) - acc1, final1 = acc1.h, final1.h - loss1 = tf.reduce_sum(acc1) + tf.reduce_sum(final1) - (dw1, db1, dh1, - di1) = tf.gradients(loss1, [theta.w, theta.b, state0.h, inputs.x]) - - # Fetches a bunch of values and compare them. - (acc0, acc1, final0, final1, dw0, dw1, db0, db1, dh0, dh1, di0, - di1) = self.evaluate( - [acc0, acc1, final0, final1, dw0, dw1, db0, db1, dh0, dh1, di0, di1]) - self.assertAllClose(acc0, acc1) - self.assertAllClose(final0, final1) - self.assertAllClose(dw0, dw1) - self.assertAllClose(db0, db1) - self.assertAllClose(dh0, dh1) - self.assertAllClose(di0, di1) - - @RecurrentTestParameters - def testElman(self): - self._testElmanHelper(1, False) - self._testElmanHelper(1, True) - self._testElmanHelper(7, False) - self._testElmanHelper(7, True) - - def StopFn(t, unused_theta, unused_state): - return t >= 4 - - self._testElmanHelper(7, False, StopFn) - self._testElmanHelper(7, True, StopFn) - - @RecurrentTestParameters - def testSetShape(self): - dst = py_utils.NestedMap( - a=tf.placeholder(tf.int32, shape=None), - b=py_utils.NestedMap( - b1=tf.placeholder(tf.int32, shape=None), - b2=tf.placeholder(tf.int32, shape=None))) - src = py_utils.NestedMap( - a=tf.constant(0, shape=[2, 4], dtype=tf.int32), - b=py_utils.NestedMap( - b1=tf.constant(0, shape=[1, 3], dtype=tf.int32), - b2=tf.constant(0, shape=[5, 8], dtype=tf.int32))) - py_utils.SetShapes(dst, src) - self.assertAllClose( - [2, 4], - py_utils.GetShape(dst.a, 2), - ) - self.assertAllClose([1, 3], py_utils.GetShape(dst.b.b1, 2)) - self.assertAllClose([5, 8], py_utils.GetShape(dst.b.b2, 2)) - - -class StackedRecurrentTest(test_utils.TestCase, parameterized.TestCase): - - @staticmethod - def Poly(theta, state0, inputs): - x = theta.x - s = state0.s - c = inputs.c - return py_utils.NestedMap(s=s * x + c), py_utils.NestedMap() - - @staticmethod - def Identity(theta, state0, inputs): - del theta, state0 - return py_utils.NestedMap(s=inputs.s), py_utils.NestedMap() - - @RecurrentTestParameters - def testSimpleStacked(self): - g = tf.Graph() - with g.as_default(): - devices = ['/cpu:0'] * 3 - cell_fns = [self.Poly, self.Identity, self.Identity] - cell_grads = [None] * 3 - cell_outs = [lambda x: x] * 3 - cell_out_grads = [lambda x: x] * 3 - w0 = tf.constant(2.) - w1 = tf.constant(0.) - w2 = tf.constant(0.) - thetas = [ - py_utils.NestedMap(x=w0), - py_utils.NestedMap(x=w1), - py_utils.NestedMap(x=w2) - ] - init_states = [py_utils.NestedMap(s=tf.constant(0.))] * 3 - inputs = py_utils.NestedMap( - c=tf.constant([1., 2., 1., 0.]), - padding=tf.constant([0., 0., 0., 1.])) - output, _ = recurrent.StackedRecurrent( - devices=devices, - cell_fns=cell_fns, - cell_grads=cell_grads, - cell_outs=cell_outs, - cell_out_grads=cell_out_grads, - thetas=thetas, - init_states=init_states, - inputs=inputs) - dw0, dw1, dw2 = tf.gradients(tf.reduce_sum(output.s), [w0, w1, w2]) - - with self.session(graph=g): - (output, dw0, dw1, dw2) = self.evaluate([output.s, dw0, dw1, dw2]) - - self.assertAllClose(output, [1., 4., 9., 0.]) - self.assertAllClose(dw2, 0.) - self.assertAllClose(dw1, 0.) - self.assertAllClose(dw0, 7.) - - def _BuildStackedRecurrentElman(self, seqlen, trailing_pad_len, batch, dims, - layers): - tf.random.set_seed(342462) - np.random.seed(32540) - - seqlen += trailing_pad_len - dtype = tf.float64 - - def CreateTheta(): - return py_utils.NestedMap( - w=tf.constant( - np.random.uniform(0, 0.2, (2 * dims, dims)), dtype=dtype), - b=tf.constant(np.random.uniform(0, 0.2, (dims,)), dtype=dtype)) - - def CreateState0(): - return py_utils.NestedMap( - h=tf.constant(np.random.uniform(0, 0.2, (batch, dims)), dtype=dtype), - padding=tf.constant([[0]] * batch, dtype=dtype)) - - devices = ['/cpu:0'] * layers - cell_fns = [Elman] * layers - cell_grads = [ElmanGrad] * layers - cell_outs = [ElmanOut] * layers - cell_out_grads = [ElmanOutGrad] * layers - thetas = [CreateTheta() for _ in range(layers)] - init_states = [CreateState0() for _ in range(layers)] - padding = np.zeros((seqlen, batch, 1)) - padding[-trailing_pad_len:, :, :] = 1. - padding[-trailing_pad_len - 3:-trailing_pad_len - 1, :, :] = 1. - inputs = py_utils.NestedMap( - x=tf.constant( - np.random.uniform(0, 0.2, (seqlen, batch, dims)), dtype=dtype), - padding=tf.constant(padding, dtype=dtype)) - output, _ = recurrent.StackedRecurrent( - devices=devices, - cell_fns=cell_fns, - cell_grads=cell_grads, - cell_outs=cell_outs, - cell_out_grads=cell_out_grads, - thetas=thetas, - init_states=init_states, - inputs=inputs) - o = output.x - if 'padding' in inputs: - o *= (1 - inputs.padding) - loss = tf.reduce_sum(tf.square(o)) - - xs = py_utils.Flatten(thetas + [py_utils.NestedMap(x=inputs.x)]) - dxs = tf.gradients(ys=loss, xs=xs) - - # Reference implementation using Recurrent(). - ref = inputs - for i in range(layers): - ref = ElmanOut( - recurrent.Recurrent( - cell_fn=cell_fns[i], - cell_grad=cell_grads[i], - theta=thetas[i], - state0=init_states[i], - inputs=ref)[0]) - return ref.x, output.x, loss, xs, dxs - - def _LogDiff(self, x, y): - tf.logging.info('max(abs(x - y)) = %s', np.max(np.abs(x - y))) - - def _CompareStackedElman(self, seqlen, batch, dims, layers): - """Tests that StackedRecurrent computest the same output as Recurrent().""" - trailing_pad_len = 2 - g = tf.Graph() - with g.as_default(): - ref, output, _, _, _ = self._BuildStackedRecurrentElman( - seqlen, trailing_pad_len, batch, dims, layers) - ref = ref[:-trailing_pad_len] - output = output[:-trailing_pad_len] - with self.session(graph=g): - ref_val, out_val = self.evaluate([ref, output]) - self._LogDiff(ref_val, out_val) - self.assertAllClose(ref_val, out_val) - - @RecurrentTestParameters - def testStackedElman_2(self): - self._CompareStackedElman(4, 3, 8, 2) - - @RecurrentTestParameters - def testStackedElman_4(self): - self._CompareStackedElman(8, 5, 8, 4) - - @RecurrentTestParameters - def testStackedElman_8(self): - self._CompareStackedElman(11, 1, 4, 8) - - def _TestStackedElmanGradient(self, num, seqlen=7, batch=5): - """Tests a stacked Elman recurrent network with num layers.""" - g = tf.Graph() - with g.as_default(): - # Sequence length, batdh size, hidden dimension - trailing_pad_len, dims, layers = 2, 8, num - _, _, loss, xs, dxs = self._BuildStackedRecurrentElman( - seqlen, trailing_pad_len, batch, dims, layers) - - # Fetches all gradients (dxs) in one session run and compare - # them with their respective numerical gradient. - with self.session(graph=g) as sess: - s_dxs = self.evaluate(dxs) - for (x, s_dx) in zip(xs, s_dxs): - n_dx = test_utils.ComputeNumericGradient(sess, loss, x) - self._LogDiff(n_dx, s_dx) - self.assertAllClose(n_dx, s_dx) - - # Randomly pick a few (x, dx) pairs, and fetch dx via one self.evaluate - # and compare with its numerical gradient. - xs_dxs = list(zip(xs, dxs)) - np.random.shuffle(xs_dxs) - with self.session(graph=g) as sess: - for (x, dx) in xs_dxs[:4]: - s_dx = self.evaluate(dx) - n_dx = test_utils.ComputeNumericGradient(sess, loss, x) - self._LogDiff(n_dx, s_dx) - self.assertAllClose(n_dx, s_dx) - - @RecurrentTestParameters - def testStackedElmanGrad_1(self): - self._TestStackedElmanGradient(1) - - @RecurrentTestParameters - def testStackedElmanGrad_2(self): - self._TestStackedElmanGradient(2) - - @RecurrentTestParameters - def testStackedElmanGrad_4(self): - self._TestStackedElmanGradient(4) - - @RecurrentTestParameters - def testStackedElmanGrad_8(self): - self._TestStackedElmanGradient(8, seqlen=5, batch=3) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/retry.py b/lingvo/core/retry.py deleted file mode 100644 index f3232e0b..00000000 --- a/lingvo/core/retry.py +++ /dev/null @@ -1,71 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Retry on exception.""" - -import functools -import itertools -import random -import sys -import time -import traceback - -import lingvo.compat as tf - - -def Retry(retry_value=Exception, - max_retries=None, - initial_delay_sec=1.0, - delay_growth_factor=1.5, - delay_growth_fuzz=0.1, - max_delay_sec=60): - """Returns a retry decorator.""" - if max_retries is None: - max_retries = 2**30 # Effectively forever. - - if delay_growth_factor < 1.0: - raise ValueError("Invalid delay_growth_factor: %f" % delay_growth_factor) - - def _Retry(func): - - @functools.wraps(func) - def Wrapper(*args, **kwargs): - """Decorator wrapper.""" - delay = initial_delay_sec - for retries in itertools.count(0): - try: - return func(*args, **kwargs) - except retry_value as e: - if retries >= max_retries: - raise - time.sleep(delay) - fuzz_factor = 1.0 + random.random() * delay_growth_fuzz - delay += delay * (delay_growth_factor - 1) * fuzz_factor - delay = min(delay, max_delay_sec) - - e_desc_str = "".join(traceback.format_exception_only(e.__class__, e)) - stack_traceback_str = "".join(traceback.format_stack()[:-2]) - e_traceback = sys.exc_info()[2] - e_traceback_str = "".join(traceback.format_tb(e_traceback)) - tf.logging.info( - "Retry: caught exception: %s while running %s. " - "Call failed at (most recent call last):\n%s" - "Traceback for above exception (most recent call last):\n%s" - "Waiting for %.2f seconds before retrying.", func.__name__, - e_desc_str, stack_traceback_str, e_traceback_str, delay) - - return Wrapper - - return _Retry diff --git a/lingvo/core/retry_test.py b/lingvo/core/retry_test.py deleted file mode 100644 index d7f660c4..00000000 --- a/lingvo/core/retry_test.py +++ /dev/null @@ -1,90 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for retry.""" - -import lingvo.compat as tf -from lingvo.core import retry -from lingvo.core import test_utils - - -class RetryTest(test_utils.TestCase): - - def testRetry(self): - - @retry.Retry(max_retries=5) - def _TestFunc(): - _TestFunc.calls += 1 - raise ValueError() - - _TestFunc.calls = 0 - - with self.assertRaises(ValueError): - _TestFunc() - self.assertEqual(_TestFunc.calls, 6) - - def testZeroMaxRetries(self): - - @retry.Retry(max_retries=0) - def _TestFunc(): - _TestFunc.calls += 1 - raise ValueError() - - _TestFunc.calls = 0 - - with self.assertRaises(ValueError): - _TestFunc() - self.assertEqual(_TestFunc.calls, 1) - - def testArgsAndReturnValue(self): - - @retry.Retry() - def _TestFunc(val): - if _TestFunc.calls > 2: - return val - else: - _TestFunc.calls += 1 - raise ValueError() - - _TestFunc.calls = 0 - - self.assertEqual(_TestFunc(2), 2) - self.assertEqual(_TestFunc.calls, 3) - - def testSpecificExceptions(self): - - @retry.Retry(max_retries=1, retry_value=(ValueError, KeyError)) - def _TestFunc(e): - _TestFunc.calls += 1 - raise e() - - _TestFunc.calls = 0 - with self.assertRaises(ValueError): - _TestFunc(ValueError) - self.assertEqual(_TestFunc.calls, 2) - - _TestFunc.calls = 0 - with self.assertRaises(KeyError): - _TestFunc(KeyError) - self.assertEqual(_TestFunc.calls, 2) - - _TestFunc.calls = 0 - with self.assertRaises(AttributeError): - _TestFunc(AttributeError) - self.assertEqual(_TestFunc.calls, 1) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/reversible_layers.py b/lingvo/core/reversible_layers.py deleted file mode 100644 index 710829ff..00000000 --- a/lingvo/core/reversible_layers.py +++ /dev/null @@ -1,233 +0,0 @@ -# Lint as: python3 -# coding=utf-8 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Reversible layers for memory efficient Backpropagation. - -[1] The Reversible Residual Network: Backpropagation Without Storing Activations - https://arxiv.org/abs/1707.04585 -""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils - - -class RevNetLayer(base_layer.BaseLayer): - """A reversible residual block. - - Computes y1 = x1 + f(x2), y2 = x2 + g(y1), where f and g can be arbitrary - functions that retain the input tensor shape. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('f_params', None, 'Layer params for the f block.') - p.Define('g_params', None, 'Layer params for the g block.') - return p - - def __init__(self, params): - super().__init__(params) - p = params - assert p.name - assert p.f_params - assert p.g_params - self.CreateChild('f_block', p.f_params) - self.CreateChild('g_block', p.g_params) - - def ReverseAndGrad(self, theta, outputs, d_outputs, f_seed, g_seed, - *extra_inputs): - """Implements Algorithm 1 in the revnet paper. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - outputs: A NestedMap: .split1 and .split2 corresponding to y1 and y2. - d_outputs: A NestedMap: .split1 and .split2 corresponding to dy1 and dy2, - the total derivatives. - f_seed: Scalar tensor. The step seed used in forward for the f block. - g_seed: Scalar tensor. The step seed used in forward for the g block. The - step seeds are needed for deterministic randomness, e.g. to ensure - dropout generate the same random mask in forward and reverse_grad. - *extra_inputs: additional inputs that will be passed to both f and g. No - gradient will be computed for these inputs. - - Returns: - A tuple of NestedMaps - - - inputs: .split1 and .split2 corresponding to x1 and x2. - - d_inputs: .split1 and .split2 corresponding to dx1 and dx2, the total - derivatives with respect to inputs. - - d_theta: has the same structure as theta. The total derivatives with - respect to weights. - - """ - - # Stop gradient on the outputs to avoid circular symbolic dependency. - y1 = tf.stop_gradient(outputs.split1) - y2 = tf.stop_gradient(outputs.split2) - dy1 = d_outputs.split1 - dy2 = d_outputs.split2 - - # Computes the reverse. - z1 = y1 - py_utils.ResetStepSeed(g_seed) - gz1 = self.g_block.FProp(theta.g_block, z1, *extra_inputs) - x2 = y2 - gz1 - py_utils.ResetStepSeed(f_seed) - fx2 = self.f_block.FProp(theta.f_block, x2, *extra_inputs) - x1 = z1 - fx2 - - # Computes the gradients. - dz1 = dy1 + tf.gradients(gz1, z1, dy2)[0] - dx2 = dy2 + tf.gradients(fx2, x2, dz1)[0] - - dgw = tf.gradients( - gz1, - theta.g_block.Flatten(), - dy2, - unconnected_gradients=tf.UnconnectedGradients.ZERO) - dgw = theta.g_block.Pack(dgw) - - dfw = tf.gradients( - fx2, - theta.f_block.Flatten(), - dz1, - unconnected_gradients=tf.UnconnectedGradients.ZERO) - dfw = theta.f_block.Pack(dfw) - - return (py_utils.NestedMap(split1=x1, split2=x2), - py_utils.NestedMap(split1=dz1, split2=dx2), - py_utils.NestedMap( - f_block=dfw, - g_block=dgw, - global_step=tf.zeros_like(theta.global_step))) - - def FProp(self, theta, inputs, *extra_inputs): - """Forward pass. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - inputs: A NestedMap: .split1 and .split2 corresponding to x1 and x2. - *extra_inputs: additional inputs that will be passed to both f and g. No - gradient will be computed for these inputs. - - Returns: - outputs: A NestedMap: .split1 and .split2 corresponding to y1 and y2. - f_seed: Scalar tensor. The step seed used in forward for the f block. - g_seed: Scalar tensor. The step seed used in forward for the g block. - - """ - - f_seed = py_utils.GetStepSeed() - f_out = self.f_block.FProp(theta.f_block, inputs.split2, *extra_inputs) - z1 = inputs.split1 + f_out - g_seed = py_utils.GetStepSeed() - g_out = self.g_block.FProp(theta.g_block, z1, *extra_inputs) - y2 = inputs.split2 + g_out - # This is essential to make dy1 independent to y2. - y1 = tf.identity(z1) - return py_utils.NestedMap(split1=y1, split2=y2), f_seed, g_seed - - -class StackedRevNetLayer(base_layer.BaseLayer): - """Stacked RevNet layers with custom gradient. - - The standard backpropagation has peak memory footprint of - - Θ(num_layers x (activation_size_per_layer + param_size_per_layer)), - - which is reduced to - - Θ(activation_size_per_layer + num_layers x param_size_per_layer)), - - in the custom gradient implmentation, at the cost of extra computation for the - reverse and potential accumulated numerical errors. See Section 3.2 in the - revnet paper for the full discussion. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('sub_layer_params', [], 'A list of RevNetLayer params.') - p.Define( - 'custom_gradient', True, 'If True, use the custom gradient over' - 'the standard TF gradient. Useful for unit test and benchmarks') - return p - - def __init__(self, params): - super().__init__(params) - p = params - assert p.name - assert p.sub_layer_params - self.CreateChildren('sub_layers', p.sub_layer_params) - - def FProp(self, theta, inputs, *extra_inputs): - - initial_step_seed = py_utils.GetStepSeed() - final_step_seed = py_utils.GenerateSeedFromName( - tf.no_op(name='new_step_seed').name) - num_layers = len(self.sub_layers) - - def Bak(inputs, outputs, d_outputs): - """Backward step.""" - del inputs # unused - output_acts, step_seeds = outputs - d_outputs = d_outputs[0] - - d_layer_thetas = [] - for layer_idx in reversed(range(num_layers)): - f_seed, g_seed = step_seeds[layer_idx] - layer = self.sub_layers[layer_idx] - layer_theta = theta.sub_layers[layer_idx] - - input_acts, d_inputs, d_theta = layer.ReverseAndGrad( - layer_theta, output_acts, d_outputs, f_seed, g_seed, *extra_inputs) - - d_layer_thetas.append(d_theta) - # Passes reconstructed inputs to the previous layer. - output_acts = input_acts - d_outputs = d_inputs - py_utils.ResetStepSeed(final_step_seed) - d_theta = py_utils.NestedMap(global_step=tf.zeros_like(initial_step_seed)) - d_theta.sub_layers = list(reversed(d_layer_thetas)) - - extra_grads = [tf.zeros_like(t) for t in extra_inputs] - return [tf.zeros_like(initial_step_seed), d_theta, d_inputs, extra_grads] - - def Fwd(xs): - """Forward pass.""" - initial_step_seed, theta, acts, extra_inputs = xs - - py_utils.ResetStepSeed(initial_step_seed) - layer_step_seeds = [] - - for layer_theta, layer in zip(theta.sub_layers, self.sub_layers): - acts, f_seed, g_seed = layer.FProp(layer_theta, acts, *extra_inputs) - layer_step_seeds += [(f_seed, g_seed)] - return [acts, layer_step_seeds] - - if self.params.custom_gradient: - acts, _ = py_utils.CallDefun( - Fwd, [initial_step_seed, theta, inputs, extra_inputs], Bak) - py_utils.ResetStepSeed(final_step_seed) - return acts - else: - acts = inputs - for layer_theta, layer in zip(theta.sub_layers, self.sub_layers): - acts, _, _ = layer.FProp(layer_theta, acts, *extra_inputs) - return acts diff --git a/lingvo/core/reversible_layers_test.py b/lingvo/core/reversible_layers_test.py deleted file mode 100644 index 2fc78f99..00000000 --- a/lingvo/core/reversible_layers_test.py +++ /dev/null @@ -1,191 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for reversible layers.""" - -from absl.testing import parameterized -from lingvo import compat as tf -from lingvo.core import layers -from lingvo.core import py_utils -from lingvo.core import reversible_layers -from lingvo.core import test_utils - -import numpy as np - -FLAGS = tf.flags.FLAGS - - -class RevNetLayerTest(test_utils.TestCase): - """Test single revnet layer.""" - - def testRevNetLayerFProp(self): - with self.session(): - tf.random.set_seed(321) - input_1 = tf.random.normal([5, 3], seed=89122) - input_2 = tf.random.normal([5, 3], seed=19438) - p = reversible_layers.RevNetLayer.Params() - p.name = 'revnet_simple' - p.f_params = layers.FCLayer.Params().Set(input_dim=3, output_dim=3) - p.g_params = layers.FCLayer.Params().Set(input_dim=3, output_dim=3) - revnet_layer = p.Instantiate() - - h, _, _ = revnet_layer.FPropDefaultTheta( - py_utils.NestedMap(split1=input_1, split2=input_2)) - self.evaluate(tf.global_variables_initializer()) - actual_layer_output = self.evaluate(h) - expected_split1 = np.array([[-0.7262997, 0.9276514, -0.20907314], - [-0.7089523, 0.24923629, 2.5001974], - [1.6766014, 0.26847264, -0.2510258], - [0.9629222, -0.57908165, 0.0485389], - [2.7580009, 0.17540382, 1.6282884]], - dtype=np.float32) - expected_split2 = np.array([[1.1282716, 1.4266306, -0.16530532], - [-0.3836313, 0.04922554, 0.25543338], - [0.03718817, 1.5488712, 2.1594636], - [-2.1252284, 3.2059612, 0.1118355], - [3.4058936, -0.63690275, -0.95291173]], - dtype=np.float32) - - self.assertAllClose(expected_split1, actual_layer_output.split1) - self.assertAllClose(expected_split2, actual_layer_output.split2) - - def testRevNetLayerReverseGrad(self): - with self.session() as sess: - tf.random.set_seed(321) - input_1 = np.random.normal(size=[5, 3]) - input_2 = np.random.normal(size=[5, 3]) - x1 = tf.placeholder(dtype=tf.float32) - x2 = tf.placeholder(dtype=tf.float32) - p = reversible_layers.RevNetLayer.Params() - p.name = 'revnet_simple' - p.f_params = layers.FCLayer.Params().Set(input_dim=3, output_dim=3) - p.g_params = layers.FCLayer.Params().Set(input_dim=3, output_dim=3) - revnet_layer = p.Instantiate() - - outputs, f_seed, g_seed = revnet_layer.FPropDefaultTheta( - py_utils.NestedMap(split1=x1, split2=x2)) - loss = tf.reduce_sum(outputs.split1 + outputs.split2) - - # Computes tensorflow gradients. - dy1, dy2 = tf.gradients(loss, [outputs.split1, outputs.split2]) - dx1, dx2 = tf.gradients(loss, [x1, x2]) - dw = tf.gradients( - loss, - revnet_layer.theta.Flatten(), - unconnected_gradients=tf.UnconnectedGradients.ZERO) - - # Computes custom gradients. - inputs_reconstruct, dinputs, d_theta = revnet_layer.ReverseAndGrad( - revnet_layer.theta, - outputs, - py_utils.NestedMap(split1=dy1, split2=dy2), - f_seed, - g_seed, - ) - - self.evaluate(tf.global_variables_initializer()) - - # Tests the reverse. - x1r, x2r = sess.run( - [inputs_reconstruct.split1, inputs_reconstruct.split2], { - x1: input_1, - x2: input_2 - }) - self.assertAllClose(input_1, x1r) - self.assertAllClose(input_2, x2r) - - # Tests the gradient. - dx1_tf, dx2_tf, dx1_custom, dx2_custom, dw_tf, dw_custom = sess.run( - [dx1, dx2, dinputs.split1, dinputs.split2, dw, - d_theta.Flatten()], { - x1: input_1, - x2: input_2 - }) - self.assertAllClose(dx1_tf, dx1_custom) - self.assertAllClose(dx2_tf, dx2_custom) - self.assertAllClose(dw_tf, dw_custom) - - -class StackedRevNetLayerTest(test_utils.TestCase, parameterized.TestCase): - """Test stacked layers.""" - - def _SimpleRevNetParams(self, name, dropout, custom_gradient): - """Construct a simple 3-layers RevNet.""" - layer_tpl = reversible_layers.RevNetLayer.Params() - layer_tpl.f_params = layers.FeedForwardNet.Params().Set( - input_dim=3, hidden_layer_dims=[3, 3]) - layer_tpl.g_params = layers.FeedForwardNet.Params().Set( - input_dim=3, hidden_layer_dims=[3, 3]) - if dropout: - layer_tpl.f_params.dropout = layers.DeterministicDropoutLayer.Params() - layer_tpl.f_params.dropout.keep_prob = 0.7 - layer_tpl.g_params.dropout = layers.DeterministicDropoutLayer.Params() - layer_tpl.g_params.dropout.keep_prob = 0.8 - - stacked_p = reversible_layers.StackedRevNetLayer.Params() - stacked_p.name = name - stacked_p.custom_gradient = custom_gradient - for idx in range(3): - layer_p = layer_tpl.Copy() - layer_p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - layer_p.name = 'layer_{}'.format(idx) - stacked_p.sub_layer_params.append(layer_p) - return stacked_p - - def _RunModel(self, input_1_val, input_2_val, dropout, custom_grad): - self._ClearCachedSession() - tf.reset_default_graph() - with self.session() as sess: - tf.random.set_seed(321) - input_1 = tf.placeholder(tf.float32) - input_2 = tf.placeholder(tf.float32) - - revnet_params = self._SimpleRevNetParams('revnet', dropout, custom_grad) - revnet_params.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - revnet = revnet_params.Instantiate() - - h = revnet.FPropDefaultTheta( - py_utils.NestedMap(split1=input_1, split2=input_2)) - - dfx = tf.gradients(h.Flatten(), [input_1, input_2]) - dfw = tf.gradients( - h.Flatten(), - revnet.theta.Flatten(), - unconnected_gradients=tf.UnconnectedGradients.ZERO) - - self.evaluate(tf.global_variables_initializer()) - dfx_val, dfw_val, h_val = sess.run([dfx, dfw, h], - feed_dict={ - input_1: input_1_val, - input_2: input_2_val, - }) - return h_val, dfx_val, dfw_val - - @parameterized.named_parameters(('nodropout', False), ('dropout', True)) - def testStackedRevNetLayer(self, dropout): - input_1_val = np.random.normal(size=[5, 3]) - input_2_val = np.random.normal(size=[5, 3]) - # input_padding_val = np.random.randint(low=0, high=2, size=[5, 1]) - - h, dfx, dfw = self._RunModel(input_1_val, input_2_val, dropout, True) - h2, dfx2, dfw2 = self._RunModel(input_1_val, input_2_val, dropout, False) - - self.assertAllClose(h, h2) - self.assertAllClose(dfx, dfx2) - self.assertAllClose(dfw, dfw2) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/rnn_cell.py b/lingvo/core/rnn_cell.py deleted file mode 100644 index 07beb4ab..00000000 --- a/lingvo/core/rnn_cell.py +++ /dev/null @@ -1,2586 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""RNN cells (e.g., LSTM, GRU) that the Lingvo model uses.""" - -import math -import lingvo.compat as tf -from lingvo.core import hyperparams -from lingvo.core import layers -from lingvo.core import pruning_utils -from lingvo.core import py_utils -from lingvo.core import quant_utils -from lingvo.core import summary_utils - -from tensorflow.python.util import deprecation as tf_deprecation # pylint: disable=g-direct-tensorflow-import - - -def _HistogramSummary(p, name, v): - """Adds a histogram summary for 'v' into the default tf graph.""" - summary_utils.histogram(name, tf.cast(v, tf.float32)) - - -RNN_CELL_WT = 'rnn_cell_weight_variable' - - -class RNNCell(quant_utils.QuantizableLayer): - # pylint: disable=line-too-long - """RNN cells. - - RNNCell represents recurrent state in a `.NestedMap`. - - `zero_state(theta, batch_size)` returns the initial state, which is defined - by each subclass. From the state, each subclass defines `GetOutput()` - to extract the output tensor. - - `RNNCell.FProp` defines the forward function:: - - (theta, state0, inputs) -> state1, extras - - All arguments and return values are `.NestedMap`. Each subclass defines - what fields these `.NestedMap` are expected to have. `extras` is a - `.NestedMap` containing some intermediate results `FProp` computes to - facilitate the backprop. - - `zero_state(theta, batch_size)`, `state0` and `state1` are all compatible - `.NestedMap` (see `.NestedMap.IsCompatible`). - I.e., they have the same keys recursively. Furthermore, the corresponding - tensors in these `.NestedMap` have the same shape and dtype. - """ - # pylint: enable=line-too-long - - @classmethod - def Params(cls): - p = super().Params() - p.Define('inputs_arity', 1, - 'number of tensors expected for the inputs.act to FProp.') - p.Define('num_input_nodes', 0, 'Number of input nodes.') - p.Define( - 'num_output_nodes', 0, - 'Number of output nodes. If num_hidden_nodes is 0, also used as ' - 'cell size.') - p.Define( - 'reset_cell_state', False, - ('Set True to support resetting cell state in scenarios where multiple ' - 'inputs are packed into a single training example. The RNN layer ' - 'should provide reset_mask inputs in addition to act and padding if ' - 'this flag is set.')) - p.Define( - 'zero_state_init_params', py_utils.DefaultRNNCellStateInit(), - 'Parameters that define how the initial state values are set ' - 'for each cell. Must be one of the static functions defined in ' - 'py_utils.RNNCellStateInit.') - return p - - def __init__(self, params): - """Initializes RnnCell.""" - super().__init__(params) - assert not self.params.vn.per_step_vn, ( - 'We do not support per step VN in RNN cells.') - - def _VariableCollections(self): - return [RNN_CELL_WT, '%s_vars' % self.__class__.__name__] - - def zero_state(self, theta, batch_size): - """Returns the initial state given the batch size.""" - raise NotImplementedError('Abstract method') - - def GetOutput(self, state): - """Returns the output value given the current state.""" - raise NotImplementedError('Abstract method') - - def batch_size(self, inputs): - """Given the inputs, returns the batch size.""" - raise NotImplementedError('Abstract method') - - def FProp(self, theta, state0, inputs): - """Forward function. - - The default implementation here assumes the cell forward - function is composed of two functions:: - - _Gates(_Mix(theta, state0, inputs), theta, state0, inputs) - - The result of `_Mix` is stashed in `extras` to facilitate backprop. - - `_ResetState` is optionally applied if `reset_cell_state` is True. The RNN - layer should provide `reset_mask` inputs in addition to other inputs. - `reset_mask` inputs are expected to be 0 at timesteps where state0 should be - reset to default (zeros) before running `_Mix()` and `_Gates()`, and 1 - otherwise. This is meant to support use cases like packed inputs, where - multiple samples are fed in a single input example sequence, and need to be - masked from each other. For example, if the two examples packed together - are ['good', 'day'] -> ['guten-tag'] and ['thanks'] -> ['danke'] - to produce ['good', 'day', 'thanks'] -> ['guten-tag', 'danke'], the - source reset_masks would be [1, 1, 0] and target reset masks would be - [1, 0]. These ids are meant to enable masking computations for - different examples from each other. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - state0: The previous recurrent state. A `.NestedMap`. - inputs: The inputs to the cell. A `.NestedMap`. - - Returns: - A tuple (state1, extras). - - state1: The next recurrent state. A `.NestedMap`. - - extras: Intermediate results to faciliate backprop. A `.NestedMap`. - """ - assert isinstance(inputs.act, list) - assert self.params.inputs_arity == len(inputs.act) - if self.params.reset_cell_state: - state0_modified = self._ResetState(state0.DeepCopy(), inputs) - else: - state0_modified = state0 - xmw = self._Mix(theta, state0_modified, inputs) - state1 = self._Gates(xmw, theta, state0_modified, inputs) - return state1, py_utils.NestedMap() - - def _ZoneOut(self, - prev_v, - cur_v, - padding_v, - zo_prob, - is_eval, - random_uniform, - qt=None, - qdomain=''): - """Apply ZoneOut regularlization to cur_v. - - Implements ZoneOut regularization as described in - https://arxiv.org/abs/1606.01305 - - Args: - prev_v: A tensor, values from the previous timestep. - cur_v: A tensor, values from the current timestep. - padding_v: A tensor, the paddings vector for the cur timestep. - zo_prob: A float, probability at which to apply ZoneOut regularization. - is_eval: A bool, whether or not in eval mode. - random_uniform: a tensor of random uniform numbers. This can be None if - zo_prob=0.0 - qt: A string, name of the qtensor for zone out math. - qdomain: A string, name of the qdomain for quantized zone out math. - - Returns: - cur_v after ZoneOut regularization has been applied. - """ - prev_v = tf.convert_to_tensor(prev_v) - cur_v = tf.convert_to_tensor(cur_v) - padding_v = tf.convert_to_tensor(padding_v) - if zo_prob == 0.0: - # Special case for when ZoneOut is not enabled. - return py_utils.ApplyPadding(padding_v, cur_v, prev_v) - - if is_eval: - # We take expectation in the eval mode. - # - fns = self.fns - # This quantized mixed operation should probably occur as fused kernel to - # avoid quantized-math rounding errors. Current accuracy has not been - # verified. - prev_weight = self.QWeight(zo_prob, domain=qdomain) - new_weight = self.QWeight(1.0 - prev_weight, domain=qdomain) - if qt is None: - mix_prev = tf.multiply(tf.fill(tf.shape(prev_v), prev_weight), prev_v) - mix_curr = tf.multiply(tf.fill(tf.shape(cur_v), new_weight), cur_v) - mix = tf.add(mix_prev, mix_curr) - else: - mix_prev = fns.qmultiply( - self.QWeight( - tf.fill(tf.shape(prev_v), prev_weight), domain=qdomain), - prev_v, - qt=qt) - mix_curr = fns.qmultiply( - self.QWeight(tf.fill(tf.shape(cur_v), new_weight), domain=qdomain), - cur_v, - qt=qt) - mix = fns.qadd(mix_prev, mix_curr, qt=qt) - - # If padding_v is 1, it always carries over the previous state. - return py_utils.ApplyPadding(padding_v, mix, prev_v) - else: - assert random_uniform is not None - random_uniform = py_utils.HasShape(random_uniform, tf.shape(prev_v)) - zo_p = tf.cast(random_uniform < zo_prob, padding_v.dtype) - zo_p += padding_v - # If padding_v is 1, we always carry over the previous state. - zo_p = tf.minimum(zo_p, 1.0) - zo_p = tf.stop_gradient(zo_p) - return py_utils.ApplyPadding(zo_p, cur_v, prev_v) - - -class LSTMCellSimple(RNNCell): - """Simple LSTM cell. - - theta: - - - wm: the parameter weight matrix. All gates combined. - - b: the combined bias vector. - - state: - - - m: the lstm output. [batch, cell_nodes] - - c: the lstm cell state. [batch, cell_nodes] - - inputs: - - - act: a list of input activations. [batch, input_nodes] - - padding: the padding. [batch, 1]. - - reset_mask: optional 0/1 float input to support packed input training. - Shape [batch, 1] - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'num_hidden_nodes', 0, 'Number of projection hidden nodes ' - '(see https://arxiv.org/abs/1603.08042). ' - 'Set to 0 to disable projection.') - p.Define( - 'cell_value_cap', 10.0, 'LSTM cell values are capped to be within ' - ' [-cell_value_cap, +cell_value_cap] if the value is not None. ' - 'It can be a scalar, a scalar tensor or None. When set to None, ' - 'no capping is applied.') - p.Define('forget_gate_bias', 0.0, 'Bias to apply to the forget gate.') - p.Define('output_nonlinearity', True, - 'Whether or not to apply tanh non-linearity on lstm output.') - p.Define('zo_prob', 0.0, - 'If > 0, applies ZoneOut regularization with the given prob.') - p.Define('enable_lstm_bias', True, 'Enable the LSTM Cell bias.') - p.Define( - 'couple_input_forget_gates', False, - 'Whether to couple the input and forget gates. Just like ' - 'tf.contrib.rnn.CoupledInputForgetGateLSTMCell') - p.Define('apply_pruning', False, 'Whether to prune the weights while ' - 'training') - p.Define('apply_pruning_to_projection', False, - 'Whether to prune the projection matrix while ' - 'training') - p.Define('gradient_pruning', False, 'Whether to gradient prune the model') - p.Define('bias_init', py_utils.WeightInit.Constant(0.0), - 'Initialization parameters for bias') - - # Non-default quantization behaviour. - p.qdomain.Define('weight', None, 'Quantization for the weights') - p.qdomain.Define('c_state', None, 'Quantization for the c-state.') - p.qdomain.Define('m_state', None, 'Quantization for the m-state.') - p.qdomain.Define('fullyconnected', None, - 'Quantization for fully connected node.') - return p - - def __init__(self, params): - """Initializes LSTMCellSimple.""" - super().__init__(params) - assert isinstance(params, hyperparams.Params) - p = self.params - assert isinstance(p.cell_value_cap, - (int, float)) or p.cell_value_cap is None - - assert p.cell_value_cap is None or p.qdomain.default is None - - self._timestep = -1 - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - # Define weights. - wm_pc = py_utils.WeightParams( - shape=[ - p.num_input_nodes + self.output_size, - self.num_gates * self.hidden_size - ], - init=p.params_init, - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('wm', wm_pc, self.AddGlobalVN) - if p.apply_pruning: - mask_pc = py_utils.WeightParams(wm_pc.shape, - py_utils.WeightInit.Constant(1.0), - p.dtype) - threshold_pc = py_utils.WeightParams([], - py_utils.WeightInit.Constant(0.0), - tf.float32) - self.CreateVariable('mask', mask_pc, theta_fn=None, trainable=False) - self.CreateVariable( - 'threshold', threshold_pc, theta_fn=None, trainable=False) - # for gradient based pruning - # gradient and weight snapshots - grad_pc = py_utils.WeightParams(wm_pc.shape, - py_utils.WeightInit.Constant(0.0), - p.dtype) - if p.gradient_pruning: - self.CreateVariable('gradient', grad_pc, theta_fn=None, trainable=False) - self.CreateVariable( - 'old_weight', grad_pc, theta_fn=None, trainable=False) - self.CreateVariable( - 'old_old_weight', grad_pc, theta_fn=None, trainable=False) - - if p.num_hidden_nodes: - w_proj = py_utils.WeightParams( - shape=[self.hidden_size, self.output_size], - init=p.params_init, - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('w_proj', w_proj, self.AddGlobalVN) - if p.apply_pruning_to_projection: - proj_mask_pc = py_utils.WeightParams(w_proj.shape, - py_utils.WeightInit.Constant(1.0), - p.dtype) - proj_threshold_pc = py_utils.WeightParams( - [], py_utils.WeightInit.Constant(0.0), tf.float32) - self.CreateVariable( - 'proj_mask', proj_mask_pc, theta_fn=None, trainable=False) - self.CreateVariable( - 'proj_threshold', proj_threshold_pc, trainable=False) - # for gradient based pruning - # gradient and weight snapshots - proj_grad_pc = py_utils.WeightParams(w_proj.shape, - py_utils.WeightInit.Constant(0.0), - p.dtype) - if p.gradient_pruning: - self.CreateVariable('proj_gradient', proj_grad_pc, trainable=False) - self.CreateVariable('proj_old_weight', proj_grad_pc, trainable=False) - self.CreateVariable( - 'proj_old_old_weight', proj_grad_pc, trainable=False) - - if p.enable_lstm_bias: - bias_pc = py_utils.WeightParams( - shape=[self.num_gates * self.hidden_size], - init=p.bias_init, - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('b', bias_pc, self.AddGlobalVN) - - if p.apply_pruning: - if p.gradient_pruning: - pruning_utils.AddToPruningCollections(self.vars.wm, self.vars.mask, - self.vars.threshold, - self.vars.gradient, - self.vars.old_weight, - self.vars.old_old_weight) - else: - pruning_utils.AddToPruningCollections(self.vars.wm, self.vars.mask, - self.vars.threshold) - if p.num_hidden_nodes: - if p.apply_pruning_to_projection: - if p.gradient_pruning: - pruning_utils.AddToPruningCollections(self.vars.w_proj, - self.vars.proj_mask, - self.vars.proj_threshold, - self.vars.proj_gradient, - self.vars.proj_old_weight, - self.vars.proj_old_old_weight) - else: - pruning_utils.AddToPruningCollections(self.vars.w_proj, - self.vars.proj_mask, - self.vars.proj_threshold) - - self.TrackQTensor( - 'zero_m', - 'm_output', - 'm_output_projection', - 'm_zoneout', - domain='m_state') - self.TrackQTensor( - 'zero_c', - 'mixed', - 'c_couple_invert', - 'c_input_gate', - 'c_forget_gate', - 'c_output_gate', - 'c_zoneout', - domain='c_state') - self.TrackQTensor('add_bias', domain='fullyconnected') - - # Collect some stats. - scope = tf.get_variable_scope() - w = self.vars.wm - if p.couple_input_forget_gates: - i_i, f_g, o_g = tf.split( - value=w, num_or_size_splits=self.num_gates, axis=1) - else: - i_i, i_g, f_g, o_g = tf.split( - value=w, num_or_size_splits=self.num_gates, axis=1) - _HistogramSummary(p, scope.name + '/wm_i_g', i_g) - _HistogramSummary(p, scope.name + '/wm_i_i', i_i) - _HistogramSummary(p, scope.name + '/wm_f_g', f_g) - _HistogramSummary(p, scope.name + '/wm_o_g', o_g) - - @property - def output_size(self): - return self.params.num_output_nodes - - @property - def hidden_size(self): - return self.params.num_hidden_nodes or self.params.num_output_nodes - - @property - def num_gates(self): - return 3 if self.params.couple_input_forget_gates else 4 - - def batch_size(self, inputs): - return tf.shape(inputs.act[0])[0] - - def zero_state(self, theta, batch_size): - p = self.params - zero_m = py_utils.InitRNNCellState((batch_size, self.output_size), - init=p.zero_state_init_params, - dtype=py_utils.FPropDtype(p), - is_eval=self.do_eval) - zero_c = py_utils.InitRNNCellState((batch_size, self.hidden_size), - init=p.zero_state_init_params, - dtype=py_utils.FPropDtype(p), - is_eval=self.do_eval) - if p.is_inference: - zero_m = self.QTensor('zero_m', zero_m) - zero_c = self.QTensor('zero_c', zero_c) - return py_utils.NestedMap(m=zero_m, c=zero_c) - - def _ResetState(self, state, inputs): - state.m = inputs.reset_mask * state.m - state.c = inputs.reset_mask * state.c - return state - - def GetOutput(self, state): - return state.m - - def _GetBias(self, theta): - """Gets the bias vector to add. - - Includes adjustments like forget_gate_bias. Use this instead of the 'b' - variable directly as including adjustments in this way allows const-prop - to eliminate the adjustments at inference time. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - - Returns: - The bias vector. - """ - p = self.params - if p.enable_lstm_bias: - b = theta.b - else: - b = tf.zeros([self.num_gates * self.hidden_size], dtype=p.dtype) - if p.forget_gate_bias != 0.0: - # Apply the forget gate bias directly to the bias vector. - if not p.couple_input_forget_gates: - # Normal 4 gate bias (i_i, i_g, f_g, o_g). - adjustment = ( - tf.ones([4, self.hidden_size], dtype=p.dtype) * tf.expand_dims( - tf.constant([0., 0., p.forget_gate_bias, 0.], dtype=p.dtype), - axis=1)) - else: - # 3 gates with coupled input/forget (i_i, f_g, o_g). - adjustment = ( - tf.ones([3, self.hidden_size], dtype=p.dtype) * tf.expand_dims( - tf.constant([0., p.forget_gate_bias, 0.], dtype=p.dtype), - axis=1)) - adjustment = tf.reshape(adjustment, [self.num_gates * self.hidden_size]) - b = b + adjustment - - return b - - def _Mix(self, theta, state0, inputs): - assert isinstance(inputs.act, list) - if self.params.apply_pruning: - wm = self.QWeight(tf.multiply(theta.wm, theta.mask, 'masked_weights')) - else: - wm = self.QWeight(theta.wm) - concat = tf.concat(inputs.act + [state0.m], 1) - # Defer quantization until after adding in the bias to support fusing - # matmul and bias add during inference. - return tf.matmul(concat, wm) - - def _Gates(self, xmw, theta, state0, inputs): - """Compute the new state.""" - i_i, i_g, f_g, o_g = self._RetrieveAndSplitGates(xmw, theta) - return self._GatesInternal(theta, state0, inputs, i_i, i_g, f_g, o_g) - - def _RetrieveAndSplitGates(self, xmw, theta): - p = self.params - b = self.QWeight(tf.expand_dims(self._GetBias(theta), 0), domain='fc') - xmw = self.fns.qadd(xmw, b, qt='add_bias') - gates = tf.split(value=xmw, num_or_size_splits=self.num_gates, axis=1) - if p.couple_input_forget_gates: - gates = gates[0], None, gates[1], gates[2] - return gates - - def _GatesInternal(self, theta, state0, inputs, i_i, i_g, f_g, o_g): - p = self.params - fns = self.fns - if not p.couple_input_forget_gates: - assert i_g is not None - forget_gate = fns.qmultiply(tf.sigmoid(f_g), state0.c, qt='c_input_gate') - # Sigmoid / tanh calls are not quantized under the assumption they share - # the range with c_input_gate and c_forget_gate. - input_gate = fns.qmultiply( - tf.sigmoid(i_g), tf.tanh(i_i), qt='c_forget_gate') - new_c = fns.qadd(forget_gate, input_gate, qt='c_output_gate') - else: - assert i_g is None - # Sigmoid / tanh calls are not quantized under the assumption they share - # the range with c_input_gate and c_forget_gate. - forget_gate = fns.qmultiply(tf.sigmoid(f_g), state0.c, qt='c_input_gate') - - # input_gate = tanh(i_i) - tanh(i_i) * tf.sigmoid(f_g) - # equivalent to (but more stable in fixed point): - # (1.0 - sigmoid(f_g)) * tanh(i_i) - tanh_i_i = tf.tanh(i_i) - input_gate = fns.qsubtract( - tanh_i_i, - fns.qmultiply(tanh_i_i, tf.sigmoid(f_g), qt='c_couple_invert'), - qt='c_forget_gate') - - new_c = fns.qadd(forget_gate, input_gate, qt='c_output_gate') - - new_c = self._ProcessNewC(theta, new_c) - - # Clip the cell states to reasonable value. - if p.cell_value_cap is not None: - new_c = py_utils.clip_by_value(new_c, -p.cell_value_cap, p.cell_value_cap) - if p.output_nonlinearity: - new_m = fns.qmultiply(tf.sigmoid(o_g), tf.tanh(new_c), qt='m_output') - else: - new_m = fns.qmultiply(tf.sigmoid(o_g), new_c, qt='m_output') - if p.num_hidden_nodes: - if p.apply_pruning_to_projection: - w_proj = self.QWeight( - tf.multiply(theta.w_proj, theta.proj_mask, 'masked_projection'), - domain='m_state') - else: - w_proj = self.QWeight(theta.w_proj, domain='m_state') - - new_m = fns.qmatmul(new_m, w_proj, qt='m_output_projection') - - # Apply Zoneout. - return self._ApplyZoneOut(state0, inputs, new_c, new_m) - - def _ProcessNewC(self, theta, new_c): - return new_c - - def _ApplyZoneOut(self, state0, inputs, new_c, new_m): - """Apply Zoneout and returns the updated states.""" - p = self.params - if p.zo_prob > 0.0: - assert not py_utils.use_tpu(), ( - 'LSTMCellSimple does not support zoneout on TPU. Switch to ' - 'LSTMCellSimpleDeterministic instead.') - c_random_uniform = tf.random.uniform(tf.shape(new_c), seed=p.random_seed) - m_random_uniform = tf.random.uniform(tf.shape(new_m), seed=p.random_seed) - else: - c_random_uniform = None - m_random_uniform = None - - new_c = self._ZoneOut( - state0.c, - new_c, - self.QRPadding(inputs.padding), - p.zo_prob, - self.do_eval, - c_random_uniform, - qt='c_zoneout', - qdomain='c_state') - new_m = self._ZoneOut( - state0.m, - new_m, - self.QRPadding(inputs.padding), - p.zo_prob, - self.do_eval, - m_random_uniform, - qt='m_zoneout', - qdomain='m_state') - new_c.set_shape(state0.c.shape) - new_m.set_shape(state0.m.shape) - return py_utils.NestedMap(m=new_m, c=new_c) - - -class LSTMCellGrouped(RNNCell): - """LSTM cell with groups. - - Grouping: based on "Factorization tricks for LSTM networks". - https://arxiv.org/abs/1703.10722. - - Shuffling: adapted from "ShuffleNet: An Extremely Efficient Convolutional - Neural Network for Mobile Devices". https://arxiv.org/abs/1707.01083. - - theta: - - - groups: a list of child LSTM cells. - - state: - - A `.NestedMap` containing 'groups', a list of `.NestedMap`, each with: - - - m: the lstm output. [batch, cell_nodes // num_groups] - - c: the lstm cell state. [batch, cell_nodes // num_groups] - - inputs: - - - act: a list of input activations. [batch, input_nodes] - - padding: the padding. [batch, 1]. - - reset_mask: optional 0/1 float input to support packed input training. - Shape [batch, 1] - """ - - @classmethod - def Params(cls, child_cell_cls=LSTMCellSimple): - p = super().Params() - p.Define('child_lstm_tpl', child_cell_cls.Params(), - 'Template of child LSTM cells.') - p.Define('num_hidden_nodes', 0, 'Number of hidden nodes.') - p.Define( - 'split_inputs', True, 'If true, split the inputs into N groups. ' - 'If false, each group gets all inputs.') - p.Define('num_groups', 0, 'Number of LSTM cell groups.') - p.Define('num_shuffle_shards', 1, - 'If > 1, number of shards for cross-group shuffling.') - return p - - def __init__(self, params): - """Initializes LSTMCellGrouped.""" - super().__init__(params) - assert isinstance(params, hyperparams.Params) - p = self.params - assert p.num_input_nodes > 0 - assert p.num_output_nodes > 0 - assert p.num_groups > 0 - assert p.num_shuffle_shards > 0 - assert p.num_input_nodes % p.num_groups == 0 - assert p.num_output_nodes % (p.num_shuffle_shards * p.num_groups) == 0 - - child_params = [] - for i in range(p.num_groups): - child_p = self.params.child_lstm_tpl.Copy() - child_p.name = 'group_%d' % i - assert child_p.num_input_nodes == 0 - assert child_p.num_output_nodes == 0 - if p.split_inputs: - child_p.num_input_nodes = p.num_input_nodes // p.num_groups - else: - child_p.num_input_nodes = p.num_input_nodes - child_p.num_output_nodes = p.num_output_nodes // p.num_groups - child_p.num_hidden_nodes = p.num_hidden_nodes // p.num_groups - child_p.reset_cell_state = p.reset_cell_state - child_params.append(child_p) - self.CreateChildren('groups', child_params) - - def batch_size(self, inputs): - return self.groups[0].batch_size(inputs) - - def zero_state(self, theta, batch_size): - return py_utils.NestedMap(groups=[ - child.zero_state(child_theta, batch_size) - for child, child_theta in zip(self.groups, theta.groups) - ]) - - # TODO(rpang): avoid split and concat between layers with the same number of - # groups, if necessary. - def GetOutput(self, state): - p = self.params - # Assuming that GetOutput() is stateless, we can just use the first child. - outputs = [ - child.GetOutput(child_state) - for child, child_state in zip(self.groups, state.groups) - ] - split_output = [] - # Split each output to num_shuffle_shards. - for output in outputs: - split_output.extend( - py_utils.SplitRecursively(output, p.num_shuffle_shards)) - # Shuffle and concatenate shards. - return py_utils.ConcatRecursively(self._ShuffleShards(split_output)) - - def FProp(self, theta, state0, inputs): - """Forward function. - - Splits state0 and inputs into N groups (N=num_groups), runs child - LSTM cells on each group, and concatenates the outputs with optional - shuffling between groups. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - state0: The previous recurrent state. A `.NestedMap`. - inputs: The inputs to the cell. A `.NestedMap`. - - Returns: - A tuple (state1, extras). - - state1: The next recurrent state. A list. - - extras: An empty `.NestedMap`. - """ - p = self.params - if p.split_inputs: - split_inputs_act = py_utils.SplitRecursively(inputs.act, p.num_groups) - else: - split_inputs_act = [inputs.act] * p.num_groups - state1 = py_utils.NestedMap(groups=[]) - for child, child_theta, child_state0, child_inputs_act in zip( - self.groups, theta.groups, state0.groups, split_inputs_act): - child_inputs = inputs.copy() - child_inputs.act = child_inputs_act - child_state1, child_extras = child.FProp(child_theta, child_state0, - child_inputs) - assert not child_extras - state1.groups.append(child_state1) - return state1, py_utils.NestedMap() - - def _ShuffleShards(self, shards): - """Shuffles shards across groups. - - Args: - shards: a list of length num_shuffle_shards (S) * num_groups (G). The - first S shards belong to group 0, the next S shards belong to group 1, - etc. - - Returns: - A shuffled list of shards such that shards from each input group are - scattered across output groups. - - For example, if we have 3 groups, each with 4 shards: - - | Group 0: 0_0, 0_1, 0_2, 0_3 - | Group 1: 1_0, 1_1, 1_2, 1_3 - | Group 2: 2_0, 2_1, 2_2, 2_3 - - The shuffled output will be: - - | Group 0: 0_0, 1_1, 2_2, 0_3 - | Group 1: 1_0, 2_1, 0_2, 1_3 - | Group 2: 2_0, 0_1, 1_2, 2_3 - """ - p = self.params - assert len(shards) == (p.num_shuffle_shards * p.num_groups) - shuffled_shards = [] - for group_i in range(p.num_groups): - for shuffle_i in range(p.num_shuffle_shards): - shuffled_shards.append(shards[( - (group_i + shuffle_i) % p.num_groups) * p.num_shuffle_shards + - shuffle_i]) - return shuffled_shards - - -# TODO(yonghui): Merge this cell with the LSTMCellSimple cell. -class LSTMCellSimpleDeterministic(LSTMCellSimple): - """Same as LSTMCellSimple, except this cell is completely deterministic.""" - - @classmethod - def Params(cls): - p = super().Params() - return p - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - self.CreateVariable( - name='lstm_step_counter', - var_params=py_utils.WeightParams([], py_utils.WeightInit.Constant(0), - tf.int64), - trainable=False) - vname = self.vars.lstm_step_counter.name - self._prng_seed = tf.constant( - py_utils.GenerateSeedFromName(vname), dtype=tf.int64) - if p.random_seed: - self._prng_seed += p.random_seed - - def zero_state(self, theta, batch_size): - p = self.params - zero_m = tf.zeros((batch_size, self.output_size), - dtype=py_utils.FPropDtype(p)) - zero_c = tf.zeros((batch_size, self.hidden_size), - dtype=py_utils.FPropDtype(p)) - if p.is_inference: - zero_m = self.QTensor('zero_m', zero_m) - zero_c = self.QTensor('zero_c', zero_c) - - # The first random seed changes for different layers and training steps. - random_seed1 = self._prng_seed + theta.lstm_step_counter - # The second random seed changes for different unroll time steps. - random_seed2 = tf.constant(0, dtype=tf.int64) - random_seeds = tf.stack([random_seed1, random_seed2]) - return py_utils.NestedMap(m=zero_m, c=zero_c, r=random_seeds) - - def _ApplyZoneOut(self, state0, inputs, new_c, new_m): - """Apply Zoneout and returns the updated states.""" - p = self.params - random_seed1 = state0.r[0] - random_seed2 = state0.r[1] - if p.zo_prob > 0.0: - # Note(yonghui): It seems that currently TF only supports int64 as the - # random seeds, however, TPU will support int32 as the seed. - # TODO(yonghui): Fix me for TPU. - c_seed = tf.stack([random_seed1, 2 * random_seed2]) - m_seed = tf.stack([random_seed1, 2 * random_seed2 + 1]) - if py_utils.use_tpu(): - c_random_uniform = tf.random.stateless_uniform( - py_utils.GetShape(new_c, 2), tf.cast(c_seed, tf.int32)) - m_random_uniform = tf.random.stateless_uniform( - py_utils.GetShape(new_m, 2), tf.cast(m_seed, tf.int32)) - else: - c_random_uniform = tf.random.stateless_uniform( - py_utils.GetShape(new_c, 2), c_seed) - m_random_uniform = tf.random.stateless_uniform( - py_utils.GetShape(new_m, 2), m_seed) - else: - c_random_uniform = None - m_random_uniform = None - - new_c = self._ZoneOut( - state0.c, - new_c, - inputs.padding, - p.zo_prob, - self.do_eval, - c_random_uniform, - qt='zero_c', - qdomain='c_state') - new_m = self._ZoneOut( - state0.m, - new_m, - inputs.padding, - p.zo_prob, - self.do_eval, - m_random_uniform, - qt='zero_m', - qdomain='m_state') - # TODO(yonghui): stop the proliferation of tf.stop_gradient - r = tf.stop_gradient(tf.stack([random_seed1, random_seed2 + 1])) - new_c.set_shape(state0.c.shape) - new_m.set_shape(state0.m.shape) - r.set_shape(state0.r.shape) - return py_utils.NestedMap(m=new_m, c=new_c, r=r) - - def PostTrainingStepUpdate(self, global_step): - """Update the global_step value.""" - p = self.params - with tf.name_scope(p.name): - summary_utils.scalar('step_counter', self.vars.lstm_step_counter) - return self.vars.lstm_step_counter.assign(tf.cast(global_step, tf.int64)) - - -class QuantizedLSTMCell(RNNCell): - """Simplified LSTM cell used for quantized training. - - There is no forget_gate_bias, no output_nonlinearity and no bias. Right now - only clipping is performed. - - theta: - - - wm: the parameter weight matrix. All gates combined. - - cap: the cell value cap. - - state: - - - m: the lstm output. [batch, cell_nodes] - - c: the lstm cell state. [batch, cell_nodes] - - inputs: - - - act: a list of input activations. [batch, input_nodes] - - padding: the padding. [batch, 1]. - - reset_mask: optional 0/1 float input to support packed input training. - [batch, 1] - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('cc_schedule', quant_utils.LinearClippingCapSchedule.Params(), - 'Clipping cap schedule.') - return p - - def __init__(self, params): - """Initializes QuantizedLSTMCell.""" - super().__init__(params) - assert isinstance(params, hyperparams.Params) - p = self.params - - self.CreateChild('cc_schedule', p.cc_schedule) - - self._timestep = -1 - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - - # Define weights. - wm_pc = py_utils.WeightParams( - shape=[p.num_input_nodes + p.num_output_nodes, 4 * p.num_output_nodes], - init=p.params_init, - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('wm', wm_pc, self.AddGlobalVN) - - scope = tf.get_variable_scope() - # Collect some stats - i_i, i_g, f_g, o_g = tf.split( - value=self.vars.wm, num_or_size_splits=4, axis=1) - _HistogramSummary(p, scope.name + '/wm_i_i', i_i) - _HistogramSummary(p, scope.name + '/wm_i_g', i_g) - _HistogramSummary(p, scope.name + '/wm_f_g', f_g) - _HistogramSummary(p, scope.name + '/wm_o_g', o_g) - - def batch_size(self, inputs): - return tf.shape(inputs.act[0])[0] - - def zero_state(self, theta, batch_size): - p = self.params - zero_m = py_utils.InitRNNCellState((batch_size, p.num_output_nodes), - init=p.zero_state_init_params, - dtype=py_utils.FPropDtype(p), - is_eval=self.do_eval) - zero_c = py_utils.InitRNNCellState((batch_size, p.num_output_nodes), - init=p.zero_state_init_params, - dtype=py_utils.FPropDtype(p), - is_eval=self.do_eval) - return py_utils.NestedMap(m=zero_m, c=zero_c) - - def GetOutput(self, state): - return state.m - - def _ResetState(self, state, inputs): - state.m = inputs.reset_mask * state.m - state.c = inputs.reset_mask * state.c - return state - - def _Mix(self, theta, state0, inputs): - assert isinstance(inputs.act, list) - return py_utils.Matmul(tf.concat(inputs.act + [state0.m], 1), theta.wm) - - def _Gates(self, xmw, theta, state0, inputs): - """Compute the new state.""" - i_i, i_g, f_g, o_g = tf.split(value=xmw, num_or_size_splits=4, axis=1) - - new_c = tf.sigmoid(f_g) * state0.c + tf.sigmoid(i_g) * tf.tanh(i_i) - new_c = self.cc_schedule.ApplyClipping(theta.cc_schedule, new_c) - new_m = tf.sigmoid(o_g) * new_c - - # Respect padding. - new_m = state0.m * inputs.padding + new_m * (1 - inputs.padding) - new_c = state0.c * inputs.padding + new_c * (1 - inputs.padding) - - new_c.set_shape(state0.c.shape) - new_m.set_shape(state0.m.shape) - return py_utils.NestedMap(m=new_m, c=new_c) - - -class LayerNormalizedLSTMCell(RNNCell): - """DEPRECATED: use LayerNormalizedLSTMCellSimple instead. - - Simple LSTM cell with layer normalization. - - Implements normalization scheme as described in - https://arxiv.org/pdf/1607.06450.pdf - - theta: - - - wm: the parameter weight matrix. All gates combined. - - b: the combined bias vector. - - state: - - - m: the lstm output. [batch, cell_nodes] - - c: the lstm cell state. [batch, cell_nodes] - - inputs: - - - act: a list of input activations. [batch, input_nodes] - - padding: the padding. [batch, 1]. - - reset_mask: optional 0/1 float input to support packed input training. - Shape [batch, 1] - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'cell_value_cap', 10.0, 'LSTM cell values are capped to be within ' - ' [-cell_value_cap, +cell_value_cap]. This can be a' - ' scalar or a scalar tensor.') - p.Define('forget_gate_bias', 0.0, 'Bias to apply to the forget gate.') - p.Define('output_nonlinearity', True, - 'Whether or not to apply tanh non-linearity on lstm output.') - p.Define('zo_prob', 0.0, - 'If > 0, applies ZoneOut regularization with the given prob.') - p.Define('layer_norm_epsilon', 1e-8, 'Tiny value to guard rsqr against.') - p.Define('cc_schedule', None, 'Clipping cap schedule.') - p.Define('use_fused_layernorm', False, 'Whether to use fused layernorm.') - return p - - @tf_deprecation.deprecated( - date=None, - instructions='New models should use LayerNormalizedLSTMCellSimple.') - def __init__(self, params): - """Initializes LayerNormalizedLSTMCell.""" - super().__init__(params) - params = self.params - if not isinstance(params.cell_value_cap, (int, float)): - raise ValueError('Cell value cap must of type int or float!') - - if params.cc_schedule: - self.CreateChild('cc_schedule', params.cc_schedule) - - self._timestep = -1 - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - - # Define weights. - wm_pc = py_utils.WeightParams( - shape=[p.num_input_nodes + p.num_output_nodes, 4 * p.num_output_nodes], - init=p.params_init, - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('wm', wm_pc, self.AddGlobalVN) - # This bias variable actually packs the initial lstm bias variables as - # well as various layer norm scale and bias variables. We pack multiple - # variables into one so that we can still unroll this lstm using the FRNN - # layer defined in layers.py. - bias_pc = py_utils.WeightParams( - shape=[4 * p.num_output_nodes + 4 * p.num_output_nodes], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('b', bias_pc, self.AddGlobalVN) - - # Collect some stats - scope = tf.get_variable_scope() - i_i, i_g, f_g, o_g = tf.split( - value=self.vars.wm, num_or_size_splits=4, axis=1) - _HistogramSummary(p, scope.name + '/wm_i_i', i_i) - _HistogramSummary(p, scope.name + '/wm_i_g', i_g) - _HistogramSummary(p, scope.name + '/wm_f_g', f_g) - _HistogramSummary(p, scope.name + '/wm_o_g', o_g) - # TODO(yonghui): Add more summaries here. - - def batch_size(self, inputs): - return tf.shape(inputs.act[0])[0] - - @property - def output_size(self): - return self.params.num_output_nodes - - @property - def hidden_size(self): - return self.params.num_output_nodes - - def zero_state(self, theta, batch_size): - p = self.params - return py_utils.NestedMap( - m=py_utils.InitRNNCellState([batch_size, p.num_output_nodes], - init=p.zero_state_init_params, - dtype=py_utils.FPropDtype(p), - is_eval=self.do_eval), - c=py_utils.InitRNNCellState([batch_size, p.num_output_nodes], - init=p.zero_state_init_params, - dtype=py_utils.FPropDtype(p), - is_eval=self.do_eval)) - - def GetOutput(self, state): - return state.m - - def _ResetState(self, state, inputs): - state.m = inputs.reset_mask * state.m - state.c = inputs.reset_mask * state.c - return state - - def _Mix(self, theta, state0, inputs): - if not isinstance(inputs.act, list): - raise ValueError('Input activations must be of list type!') - return py_utils.Matmul(tf.concat(inputs.act + [state0.m], 1), theta.wm) - - def _Gates(self, xmw, theta, state0, inputs): - """Compute the new state.""" - # Unpack the variables (weight and bias) into individual variables. - params = self.params - - def BiasSlice(dim, num_dims, start_ind): - s = [] - for i in range(num_dims): - s.append(theta.b[start_ind + i * dim:start_ind + (i + 1) * dim]) - start_ind += dim * num_dims - return s, start_ind - - # Unpack the bias variable. - slice_start = 0 - bias_lstm, slice_start = BiasSlice(params.num_output_nodes, 4, slice_start) - ln_scale, slice_start = BiasSlice(params.num_output_nodes, 4, slice_start) - assert slice_start == 8 * params.num_output_nodes - - def _LayerNorm(x, last_dim): - """Normalize the last dimension.""" - if params.use_fused_layernorm: - counts, means_ss, variance_ss, _, = tf.nn.sufficient_statistics( - x, axes=[last_dim], keepdims=True) - mean, variance = tf.nn.normalize_moments(counts, means_ss, variance_ss, - None) - else: - mean = tf.reduce_mean(x, axis=[last_dim], keepdims=True) - variance = tf.reduce_mean( - tf.square(x - mean), axis=[last_dim], keepdims=True) - return (x - mean) * tf.math.rsqrt(variance + params.layer_norm_epsilon) - - state_split = tf.split(xmw, num_or_size_splits=4, axis=1) - for i in range(4): - state_split[i] = _LayerNorm(state_split[i], 1) * tf.expand_dims( - ln_scale[i] + 1.0, 0) + tf.expand_dims(bias_lstm[i], 0) - - i_i, i_g, f_g, o_g = state_split - - if params.forget_gate_bias != 0.0: - f_g += params.forget_gate_bias - new_c = tf.sigmoid(f_g) * state0.c + tf.sigmoid(i_g) * tf.tanh(i_i) - - # Clip the cell states to reasonable value. - if params.cc_schedule: - cap = self.cc_schedule.GetState(theta.cc_schedule) - else: - cap = params.cell_value_cap - new_c = py_utils.clip_by_value(new_c, -cap, cap) - - if params.output_nonlinearity: - new_m = tf.sigmoid(o_g) * tf.tanh(new_c) - else: - new_m = tf.sigmoid(o_g) * new_c - - if params.zo_prob > 0.0: - c_random_uniform = tf.random.uniform( - tf.shape(new_c), seed=params.random_seed) - m_random_uniform = tf.random.uniform( - tf.shape(new_m), seed=params.random_seed) - else: - c_random_uniform = None - m_random_uniform = None - - new_c = self._ZoneOut(state0.c, new_c, inputs.padding, params.zo_prob, - self.do_eval, c_random_uniform) - new_m = self._ZoneOut(state0.m, new_m, inputs.padding, params.zo_prob, - self.do_eval, m_random_uniform) - new_c.set_shape(state0.c.shape) - new_m.set_shape(state0.m.shape) - return py_utils.NestedMap(m=new_m, c=new_c) - - -class LayerNormalizedLSTMCellSimple(LSTMCellSimple): - """An implementation of layer normalized LSTM based on LSTMCellSimple. - - Implements normalization scheme as described in - https://arxiv.org/pdf/1607.06450.pdf - - theta: - - - wm: the parameter weight matrix. All gates combined. - - b: the combined bias vector. - - state: - - - m: the lstm output. [batch, cell_nodes] - - c: the lstm cell state. [batch, cell_nodes] - - inputs: - - - act: a list of input activations. [batch, input_nodes] - - padding: the padding. [batch, 1]. - - reset_mask: optional 0/1 float input to support packed input training. - Shape [batch, 1] - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('layer_norm_epsilon', 1e-8, 'Tiny value to guard rsqr against.') - return p - - def _CreateLayerVariables(self): - """Initializes LayerNormalizedLSTMCellSimple.""" - super()._CreateLayerVariables() - p = self.params - - add_biases = ['add_bias_{}'.format(i) for i in range(self.num_gates)] - self.TrackQTensor(*add_biases, domain='fullyconnected') - - ln_scale_pc = py_utils.WeightParams( - shape=[self.num_gates * self.hidden_size], - init=py_utils.WeightInit.Constant(1.0), - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('ln_scale', ln_scale_pc, self.AddGlobalVN) - - def _Gates(self, xmw, theta, state0, inputs): - """Compute the new state.""" - - def _LayerNorm(x): - """Applies layer normalization on the last dimension of 'x'. - - Args: - x: activation tensor, where the last dimension represents channels. - - Returns: - Layer normalized 'x', with the same shape as the input. - """ - mean = tf.reduce_mean(x, axis=-1, keepdims=True) - variance = tf.reduce_mean(tf.square(x - mean), axis=-1, keepdims=True) - return (x - mean) * tf.math.rsqrt(variance + p.layer_norm_epsilon) - - p = self.params - b = self.QWeight(tf.expand_dims(self._GetBias(theta), 0), domain='fc') - - bs = tf.split(b, num_or_size_splits=self.num_gates, axis=1) - ln_scales = tf.split( - theta.ln_scale, num_or_size_splits=self.num_gates, axis=0) - gates = tf.split(xmw, num_or_size_splits=self.num_gates, axis=1) - - for i in range(self.num_gates): - # i_g is None when p.couple_input_forget_gates is True. - if gates[i] is not None: - gates[i] = _LayerNorm(gates[i]) * tf.expand_dims(ln_scales[i], 0) - gates[i] = self.fns.qadd(gates[i], bs[i], qt='add_bias_{}'.format(i)) - - if not p.couple_input_forget_gates: - i_i, i_g, f_g, o_g = gates - else: - i_i, i_g, f_g, o_g = gates[0], None, gates[1], gates[2] - return self._GatesInternal(theta, state0, inputs, i_i, i_g, f_g, o_g) - - -class NormalizedLSTMCellSimple(LSTMCellSimple): - """LSTM Cell that allows customzied normalization. - - theta: - - - wm: the parameter weight matrix. All gates combined. - - b: the combined bias vector. - - state: - - - m: the lstm output. [batch, cell_nodes] - - c: the lstm cell state. [batch, cell_nodes] - - inputs: - - - act: a list of input activations. [batch, input_nodes] - - padding: the padding. [batch, 1]. - - reset_mask: optional 0/1 float input to support packed input training. - Shape [batch, 1] - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('norm_layer_tpl', - layers.LayerNorm.Params().Set(epsilon=1e-8), - 'The normalization layer param') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - gates_name = ['i_i', 'i_g', 'f_g', 'o_g'] - for gate in gates_name: - norm_layer_p = p.norm_layer_tpl.Copy().Set(input_dim=self.hidden_size) - self.CreateChild('norm_' + gate, norm_layer_p) - - def _Gates(self, xmw, theta, state0, inputs): - # Retrieve i_i, i_g, f_g, o_g - gates = self._RetrieveAndSplitGates(xmw, theta) - gates_name = ['i_i', 'i_g', 'f_g', 'o_g'] - for i, gate_name in enumerate(gates_name): - norm_layer_name = 'norm_' + gate_name - if gates[i] is not None: - gates[i] = self.children.get(norm_layer_name).FProp( - theta.get(norm_layer_name), gates[i]) - - i_i, i_g, f_g, o_g = gates - return self._GatesInternal(theta, state0, inputs, i_i, i_g, f_g, o_g) - - -class LayerNormalizedLSTMCellLean(RNNCell): - """A very lean layer normalized LSTM cell. - - This version is around 20% faster on TPU than LayerNormalizedLSTMCellSimple as - it avoids certain reshape ops which are not free on TPU. - - Note, this version doesn't support all the options as implemented in - LayerNormalizedLSTMCellSimple, such as quantization, zoneout regularization - and etc. - - For the overlapping options, an incomplete list of differences from - LayerNormalizedLSTMCellSimple include: - - c_state is layer-normalized for computing new_m (if enable_ln_on_c=True) - - ln_scale has a fixed offset of 1. - - Please use the other version if you even need those options. - - state: - - - m: the lstm output. [batch, cell_nodes] - - c: the lstm cell state. [batch, cell_nodes] - - inputs: - - - act: a list of input activations. [batch, input_nodes] - - padding: the padding. [batch, 1]. - - reset_mask: optional 0/1 float input to support packed input training. - Shape [batch, 1] - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'num_hidden_nodes', 0, 'Number of projection hidden nodes ' - '(see https://arxiv.org/abs/1603.08042). ' - 'Set to 0 to disable projection.') - p.Define('layer_norm_epsilon', 1e-8, 'Tiny value to guard rsqrt against.') - p.Define( - 'enable_ln_on_c', True, - 'Whether to apply layer normalization on state.c. ' - 'If false, LayerNormalizedLSTMCellLean will behave exactly as ' - 'LayerNormalizedLSTMCellSimple.') - p.Define( - 'cell_value_cap', None, 'LSTM cell values are capped to be within ' - ' [-cell_value_cap, +cell_value_cap] if the value is not None. ' - 'It can be a scalar, a scalar tensor or None. When set to None, ' - 'no capping is applied.') - p.Define('enable_lstm_bias', False, 'Enable the LSTM Cell bias.') - p.Define('bias_init', py_utils.WeightInit.Constant(0.0), - 'Initialization parameters for bias') - p.Define('use_ln_bias', True, 'If to include a bias term for layer norm.') - - # TODO(yonghui): Get rid of the following two params. - p.Define('output_nonlinearity', True, - 'Whether or not to apply tanh non-linearity on lstm output.') - p.Define('zo_prob', 0.0, - 'If > 0, applies ZoneOut regularization with the given prob.') - return p - - def __init__(self, params): - """Initializes LayerNormalizedLSTMCellLean.""" - super().__init__(params) - assert isinstance(params, hyperparams.Params) - p = self.params - - assert p.output_nonlinearity - assert p.zo_prob == 0.0 - if p.cell_value_cap is not None and not isinstance(p.cell_value_cap, - (int, float)): - raise ValueError( - 'p.cell_value_cap should be a int/float if not None, but got {}' - .format(p.cell_value_cap)) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - # Define weights. - wm_pc = py_utils.WeightParams( - shape=[p.num_input_nodes + self.output_size, 4 * self.hidden_size], - init=p.params_init, - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('wm', wm_pc, self.AddGlobalVN) - - if p.num_hidden_nodes: - w_proj = py_utils.WeightParams( - shape=[self.hidden_size, self.output_size], - init=p.params_init, - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('w_proj', w_proj, self.AddGlobalVN) - - if p.enable_lstm_bias: - bias_pc = py_utils.WeightParams( - shape=[4 * self.hidden_size], - init=p.bias_init, - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('b', bias_pc, self.AddGlobalVN) - - pc = py_utils.WeightParams( - shape=[self.hidden_size], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=self._VariableCollections()) - ln_gates = ['i_g', 'i_i', 'f_g', 'o_g'] - if p.enable_ln_on_c: - ln_gates += ['c'] - for ln_name in ln_gates: - self.CreateVariable('ln_scale_' + ln_name, pc, self.AddGlobalVN) - if p.use_ln_bias: - self.CreateVariable('bias_' + ln_name, pc, self.AddGlobalVN) - - @property - def output_size(self): - return self.params.num_output_nodes - - @property - def hidden_size(self): - return self.params.num_hidden_nodes or self.params.num_output_nodes - - def batch_size(self, inputs): - return tf.shape(inputs.act[0])[0] - - def zero_state(self, theta, batch_size): - p = self.params - zero_m = py_utils.InitRNNCellState((batch_size, self.output_size), - init=p.zero_state_init_params, - dtype=py_utils.FPropDtype(p), - is_eval=self.do_eval) - zero_c = py_utils.InitRNNCellState((batch_size, self.hidden_size), - init=p.zero_state_init_params, - dtype=py_utils.FPropDtype(p), - is_eval=self.do_eval) - return py_utils.NestedMap(m=zero_m, c=zero_c) - - def _ResetState(self, state, inputs): - state.m = inputs.reset_mask * state.m - state.c = inputs.reset_mask * state.c - return state - - def GetOutput(self, state): - return state.m - - def _Mix(self, theta, state0, inputs): - assert isinstance(inputs.act, list) - mixed = tf.matmul(tf.concat(inputs.act + [state0.m], 1), theta.wm) - return mixed - - def _LayerNormGate(self, theta, gate_name, x): - """Applies layer normalization on the last dimension of 'x'. - - Args: - theta: a NestedMap of layer params. - gate_name: the name of the gate, e.g., 'i_i', 'f_g', 'c', etc. - x: activation tensor, where the last dimension represents channels. - - Returns: - Layer normalized 'x', with the same shape as the input. - """ - p = self.params - if gate_name == 'c' and not p.enable_ln_on_c: - return x - mean = tf.reduce_mean(x, axis=[1], keepdims=True) - centered = x - mean - variance = tf.reduce_mean(tf.square(centered), axis=[1], keepdims=True) - normed = centered * tf.math.rsqrt(variance + p.layer_norm_epsilon) - scale = theta['ln_scale_%s' % gate_name] + 1.0 - if p.use_ln_bias: - bias = theta['bias_%s' % gate_name] - return normed * scale + bias - else: - return normed * scale - - def _Gates(self, xmw, theta, state0, inputs): - """Compute the new state.""" - p = self.params - i_i, i_g, f_g, o_g = tf.split(value=xmw, num_or_size_splits=4, axis=1) - i_i = self._LayerNormGate(theta, 'i_i', i_i) - i_g = self._LayerNormGate(theta, 'i_g', i_g) - f_g = self._LayerNormGate(theta, 'f_g', f_g) - o_g = self._LayerNormGate(theta, 'o_g', o_g) - if p.enable_lstm_bias: - # LayerNormalizedLSTMCellLean applies biases after LN. - biases = tf.split(theta.b, num_or_size_splits=4) - i_i += biases[0] - i_g += biases[1] - f_g += biases[2] - o_g += biases[3] - new_c = tf.sigmoid(f_g) * state0.c + tf.sigmoid(i_g) * tf.tanh(i_i) - if p.cell_value_cap is not None: - new_c = py_utils.clip_by_value(new_c, -p.cell_value_cap, p.cell_value_cap) - # new_c_normed is only used for computing 'new_m'. We use the un-normalized - # new_cc as cell state to keep the residual property of lstm cell. - new_c_normed = self._LayerNormGate(theta, 'c', new_c) - new_m = tf.sigmoid(o_g) * tf.tanh(new_c_normed) - - if p.num_hidden_nodes: - new_m = tf.matmul(new_m, theta.w_proj) - - # Now take care of padding. - padding = inputs.padding - new_m = py_utils.ApplyPadding(padding, new_m, state0.m) - new_c = py_utils.ApplyPadding(padding, new_c, state0.c) - - return py_utils.NestedMap(m=new_m, c=new_c) - - -class DoubleProjectionLSTMCell(RNNCell): - """A layer normalized LSTM cell that support input and output projections. - - Note, this version doesn't support all the options as implemented in - LayerNormalizedLSTMCellSimple, like quantization, zoneout regularization, - etc. Please use the other version if you need those options and do not need - input projection. - - It also uses separate variables for weight matrices between gates - ('wm_{i_i, i_g, f_g, o_g}') instead of a single variable ('wm'). This allows - the initialization to use the default GeoMeanXavier(). - - state: - - - m: the lstm output. [batch, cell_nodes] - - c: the lstm cell state. [batch, cell_nodes] - - inputs: - - - act: a list of input activations. [batch, input_nodes] - - padding: the padding. [batch, 1]. - - reset_mask: optional 0/1 float input to support packed input training. - Shape [batch, 1] - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'num_input_hidden_nodes', 0, - 'Project all inputs, include m, to a hidden vector this size before ' - 'projecting to num_gates * |c|. Must be > 0.') - p.Define( - 'num_hidden_nodes', 0, 'Number of projection hidden nodes ' - '(see https://arxiv.org/abs/1603.08042). ' - 'Set to 0 to disable projection.') - p.Define('layer_norm_epsilon', 1e-8, 'Tiny value to guard rsqrt against.') - p.Define('enable_ln_on_c', True, - 'Whether to apply layer normalization on state.c.') - p.params_init = py_utils.WeightInit.GeoMeanXavier() - return p - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - - def _WeightInit(shape): - return py_utils.WeightParams( - shape=shape, - init=p.params_init, - dtype=p.dtype, - collections=self._VariableCollections()) - - self.CreateVariable( - 'w_input_proj', - _WeightInit( - [p.num_input_nodes + self.output_size, p.num_input_hidden_nodes]), - self.AddGlobalVN) - - self.CreateVariable('w_output_proj', - _WeightInit([self.hidden_size, self.output_size]), - self.AddGlobalVN) - - for gate_name in self.gates: - self.CreateVariable( - 'wm_%s' % gate_name, - _WeightInit([p.num_input_hidden_nodes, self.hidden_size]), - self.AddGlobalVN) - - pc = py_utils.WeightParams( - shape=[self.hidden_size], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=self._VariableCollections()) - ln_gates = self.gates - if p.enable_ln_on_c: - ln_gates += ['c'] - for ln_name in ln_gates: - self.CreateVariable('ln_scale_' + ln_name, pc, self.AddGlobalVN) - self.CreateVariable('bias_' + ln_name, pc, self.AddGlobalVN) - - @property - def output_size(self): - return self.params.num_output_nodes - - @property - def hidden_size(self): - return self.params.num_hidden_nodes - - def batch_size(self, inputs): - return tf.shape(inputs.act[0])[0] - - @property - def gates(self): - return ['i_g', 'i_i', 'f_g', 'o_g'] - - def zero_state(self, theta, batch_size): - p = self.params - zero_m = py_utils.InitRNNCellState((batch_size, self.output_size), - init=p.zero_state_init_params, - dtype=py_utils.FPropDtype(p), - is_eval=self.do_eval) - zero_c = py_utils.InitRNNCellState((batch_size, self.hidden_size), - init=p.zero_state_init_params, - dtype=py_utils.FPropDtype(p), - is_eval=self.do_eval) - return py_utils.NestedMap(m=zero_m, c=zero_c) - - def _ResetState(self, state, inputs): - state.m = inputs.reset_mask * state.m - state.c = inputs.reset_mask * state.c - return state - - def GetOutput(self, state): - return state.m - - def _ProcessInputProj(self, theta, input_proj): - return input_proj - - def _Mix(self, theta, state0, inputs): - assert isinstance(inputs.act, list) - concat = tf.concat(inputs.act + [state0.m], 1) - input_proj = tf.matmul(concat, theta.w_input_proj) - input_proj = self._ProcessInputProj(theta, input_proj) - gate_map = {} - for gate_name in self.gates: - g = tf.matmul(input_proj, theta.get('wm_%s' % gate_name)) - g = self._LayerNormGate(theta, gate_name, g) - gate_map[gate_name] = g - return gate_map - - def _LayerNormGate(self, theta, gate_name, x): - """Applies layer normalization on the last dimension of 'x'. - - Args: - theta: a NestedMap of layer params. - gate_name: the name of the gate, e.g., 'i_i', 'f_g', 'c', etc. - x: activation tensor, where the last dimension represents channels. - - Returns: - Layer normalized 'x', with the same shape as the input. - """ - p = self.params - if gate_name == 'c' and not p.enable_ln_on_c: - return x - mean = tf.reduce_mean(x, axis=[1], keepdims=True) - centered = x - mean - variance = tf.reduce_mean(tf.square(centered), axis=[1], keepdims=True) - normed = centered * tf.math.rsqrt(variance + p.layer_norm_epsilon) - scale = theta['ln_scale_%s' % gate_name] + 1.0 - bias = theta['bias_%s' % gate_name] - return normed * scale + bias - - def _Gates(self, xmw, theta, state0, inputs): - """Compute the new state.""" - new_c = tf.sigmoid(xmw['f_g']) * state0.c + tf.sigmoid( - xmw['i_g']) * tf.tanh(xmw['i_i']) - new_c_normed = self._LayerNormGate(theta, 'c', new_c) - new_m = tf.sigmoid(xmw['o_g']) * tf.tanh(new_c_normed) - new_m = tf.matmul(new_m, theta.w_output_proj) - - # Now take care of padding. - padding = inputs.padding - new_m = py_utils.ApplyPadding(padding, new_m, state0.m) - new_c = py_utils.ApplyPadding(padding, new_c, state0.c) - - return py_utils.NestedMap(m=new_m, c=new_c) - - -class ConvLSTMCell(RNNCell): - """Convolution LSTM cells. - - theta: - - - wm: the parameter weight matrix. All gates combined. - - b: the combined bias vector. - - state: - - - m: the lstm output. cell_shape - - c: the lstm cell state. cell_shape - - inputs: - - - act: a list of input activations. input_shape. - - padding: the padding. [batch]. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'inputs_shape', [None, None, None, None], - 'The shape of the input. It should be a list/tuple of size four.' - ' Elements are in the order of batch, height, width, channel.') - p.Define( - 'cell_shape', [None, None, None, None], - 'The cell shape. It should be a list/tuple of size four.' - ' Elements are in the order of batch, height, width, channel.' - ' Height and width of cell_shape should match that of' - ' inputs_shape.') - p.Define( - 'filter_shape', [None, None], - 'Shape of the convolution filter. This should be a pair, in the' - ' order height and width.') - p.Define( - 'cell_value_cap', 10.0, 'LSTM cell values are capped to be within ' - ' [-cell_value_cap, +cell_value_cap]. This can be a' - ' scalar or a scalar tensor.') - p.Define('output_nonlinearity', True, - 'Whether or not to apply tanh non-linearity on lstm output.') - p.Define('zo_prob', 0.0, - 'If > 0, applies ZoneOut regularization with the given prob.') - - return p - - def __init__(self, params): - """Initializes ConvLSTMCell.""" - assert isinstance(params, hyperparams.Params) - super().__init__(params) - - p = self.params - assert p.reset_cell_state is False, ('ConvLSTMCell currently doesnt ' - 'support resetting cell state.') - assert p.inputs_shape[1] == p.cell_shape[1] - assert p.inputs_shape[2] == p.cell_shape[2] - assert isinstance(p.cell_value_cap, (int, float)) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - - in_channels = p.inputs_shape[3] + p.cell_shape[3] - out_channels = p.cell_shape[3] - # Define weights. - var_shape = [ - p.filter_shape[0], p.filter_shape[1], in_channels, 4 * out_channels - ] - wm_pc = py_utils.WeightParams( - shape=var_shape, - init=p.params_init, - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('wm', wm_pc, self.AddGlobalVN) - - bias_pc = py_utils.WeightParams( - shape=[4 * out_channels], - init=py_utils.WeightInit.Constant(0.0), - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('b', bias_pc, self.AddGlobalVN) - - def batch_size(self, inputs): - return tf.shape(inputs.act[0])[0] - - @property - def output_size(self): - # There is no projection. - return self.hidden_size - - @property - def hidden_size(self): - p = self.params - return p.cell_shape[1:] - - def zero_state(self, theta, batch_size): - p = self.params - height = p.inputs_shape[1] - width = p.inputs_shape[2] - out_channels = p.cell_shape[3] - return py_utils.NestedMap( - m=py_utils.InitRNNCellState( - tf.stack([batch_size, height, width, out_channels]), - init=p.zero_state_init_params, - dtype=p.dtype, - is_eval=self.do_eval), - c=py_utils.InitRNNCellState( - tf.stack([batch_size, height, width, out_channels]), - init=p.zero_state_init_params, - dtype=p.dtype, - is_eval=self.do_eval)) - - def GetOutput(self, state): - return state.m - - def _Mix(self, theta, state0, inputs): - assert isinstance(inputs.act, list) - # Concate on channels. - xm = tf.concat(inputs.act + [state0.m], 3) - # TODO(yonghui): Possibly change the data_format to NCHW to speed - # up conv2d kernel on gpu. - xmw = tf.nn.conv2d(xm, theta.wm, [1, 1, 1, 1], 'SAME', data_format='NHWC') - return xmw - - def _Gates(self, xmw, theta, state0, inputs): - """Compute the new state.""" - p = self.params - # Bias is applied to channels. - bias = tf.reshape(theta.b, [1, 1, 1, -1]) - i_i, i_g, f_g, o_g = tf.split( - value=xmw + bias, num_or_size_splits=4, axis=3) - new_c = tf.sigmoid(f_g) * state0.c + tf.sigmoid(i_g) * tf.tanh(i_i) - # Clip the cell states to reasonable value. - new_c = py_utils.clip_by_value(new_c, -p.cell_value_cap, p.cell_value_cap) - if p.output_nonlinearity: - new_m = tf.sigmoid(o_g) * tf.tanh(new_c) - else: - new_m = tf.sigmoid(o_g) * new_c - padding = tf.reshape(inputs.padding, [-1, 1, 1, 1]) - new_c = state0.c * padding + new_c * (1.0 - padding) - new_m = state0.m * padding + new_m * (1.0 - padding) - if p.zo_prob > 0.0: - c_random_uniform = tf.random.uniform(tf.shape(new_c), seed=p.random_seed) - m_random_uniform = tf.random.uniform(tf.shape(new_m), seed=p.random_seed) - else: - c_random_uniform = None - m_random_uniform = None - new_c = self._ZoneOut(state0.c, new_c, padding, p.zo_prob, self.do_eval, - c_random_uniform) - new_m = self._ZoneOut(state0.m, new_m, padding, p.zo_prob, self.do_eval, - m_random_uniform) - new_c.set_shape(state0.c.shape) - new_m.set_shape(state0.m.shape) - return py_utils.NestedMap(m=new_m, c=new_c) - - -class SRUCell(RNNCell): - """SRU cell. - - From this paper: https://arxiv.org/abs/1709.02755 - - This is a simple implementation that can be used as a drop-in replacement for - another RNN. It doesn't do the performance tricks that an SRU is capable of, - like unrolling matrix computations over time. This is just a basic - implementation. It does the 4-matrix implementation found in appendix C. - - theta: - - - wm: the parameter weight matrix. All gates combined. - - b: the combined bias vector. - - state: - - - m: the sru output. [batch, cell_nodes] - - c: the sru cell state. [batch, cell_nodes] - - inputs: - - - act: a list of input activations. [batch, input_nodes] - - padding: the padding. [batch, 1]. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'num_hidden_nodes', 0, 'Number of projection hidden nodes ' - '(see https://arxiv.org/abs/1603.08042). ' - 'Set to 0 to disable projection.') - p.Define( - 'cell_value_cap', 10.0, 'SRU cell values are capped to be within ' - ' [-cell_value_cap, +cell_value_cap]. This can be a' - ' scalar or a scalar tensor.') - p.Define('zo_prob', 0.0, - 'If > 0, applies ZoneOut regularization with the given prob.') - p.Define('couple_input_forget_gates', True, - 'Whether to couple the input and forget gates.') - p.Define('apply_layer_norm', False, 'Apply layer norm to the variables') - p.Define( - 'layer_norm_epsilon', 1e-8, 'Tiny value to guard rsqr against.' - 'value is necessary only if apply_layer_norm is True') - p.Define('apply_pruning', False, 'Whether to prune the weights while' - 'training') - p.Define('apply_pruning_to_projection', False, - 'Whether to prune the weights in the projection layer') - p.Define('gradient_pruning', False, 'Whether to gradient prune the model') - p.Define('bias_init', py_utils.WeightInit.Constant(0.0), - 'Initialization parameters for bias') - # Add cell-recursive vector into the SRU cells (arxiv.org/abs/1709.02755). - p.Define( - 'pointwise_peephole', False, 'Whether c_{t-1} should be used to' - 'calculate gate values by aggregating gate calculations with its ' - 'point-wise dot product with a weight vector.') - p.Define( - 'hidden_scaling_factor', False, - 'scaling factor alpha for hidden layer. See details on alpha in' - 'section 3.2 of https://arxiv.org/pdf/1709.02755.pdf') - p.Define( - 'uniform_heuristic_init', False, - 'When set to True, initialize the weight params with uniform ' - 'distribution of [-sqrt(3/hidden_nodes), +sqrt(3/hidden_nodes)], aka' - 'UniformUnitScaling. This initialization has proven to help NLP tasks' - 'in arxiv.org/abs/1709.02755. This impacts 1) input weight matrices for' - 'gates, 2) projection weight matrices, and 3) cell recursion vectors.') - return p - - def __init__(self, params): - """Initializes SRUCell.""" - super().__init__(params) - assert isinstance(params, hyperparams.Params) - p = self.params - assert p.reset_cell_state is False, ('SRUCell currently doesnt support ' - 'resetting cell state.') - assert isinstance(p.cell_value_cap, (int, float)) - if p.uniform_heuristic_init: - # Setting init = sqrt(3) / sqrt(hidden) * tf.uniform(-1, 1). - p.params_init = py_utils.WeightInit.Uniform( - scale=(math.sqrt(3.0 / float(self.hidden_size)))) - - self._timestep = -1 - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - # Define weights. - wm_pc = py_utils.WeightParams( - shape=[p.num_input_nodes, self.num_gates * self.hidden_size], - init=p.params_init, - dtype=p.dtype, - collections=self._VariableCollections()) - - self.CreateVariable('wm', wm_pc, self.AddGlobalVN) - if p.apply_pruning: - mask_pc = py_utils.WeightParams(wm_pc.shape, - py_utils.WeightInit.Constant(1.0), - p.dtype) - threshold_pc = py_utils.WeightParams([], - py_utils.WeightInit.Constant(0.0), - tf.float32) - self.CreateVariable('mask', mask_pc, theta_fn=None, trainable=False) - self.CreateVariable( - 'threshold', threshold_pc, theta_fn=None, trainable=False) - - # for gradient based pruning - # gradient and weight snapshots - grad_pc = py_utils.WeightParams(wm_pc.shape, - py_utils.WeightInit.Constant(0.0), - p.dtype) - if p.gradient_pruning: - self.CreateVariable('gradient', grad_pc, theta_fn=None, trainable=False) - self.CreateVariable( - 'old_weight', grad_pc, theta_fn=None, trainable=False) - self.CreateVariable( - 'old_old_weight', grad_pc, theta_fn=None, trainable=False) - - bias_pc = py_utils.WeightParams( - shape=[self.num_gates * self.hidden_size], - init=p.bias_init, - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('b', bias_pc, self.AddGlobalVN) - - if p.num_hidden_nodes: - w_proj = py_utils.WeightParams( - shape=[self.hidden_size, self.output_size], - init=p.params_init, - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('w_proj', w_proj, self.AddGlobalVN) - if p.apply_pruning_to_projection: - proj_mask_pc = py_utils.WeightParams(w_proj.shape, - py_utils.WeightInit.Constant(1.0), - p.dtype) - proj_threshold_pc = py_utils.WeightParams( - [], py_utils.WeightInit.Constant(0.0), tf.float32) - self.CreateVariable( - 'proj_mask', proj_mask_pc, theta_fn=None, trainable=False) - self.CreateVariable( - 'proj_threshold', proj_threshold_pc, trainable=False) - # for gradient based pruning - # gradient and weight snapshots - proj_grad_pc = py_utils.WeightParams(w_proj.shape, - py_utils.WeightInit.Constant(0.0), - p.dtype) - if p.gradient_pruning: - self.CreateVariable('proj_gradient', proj_grad_pc, trainable=False) - self.CreateVariable('proj_old_weight', proj_grad_pc, trainable=False) - self.CreateVariable( - 'proj_old_old_weight', proj_grad_pc, trainable=False) - - # TODO(yuansg): b/136014373 investigate the layer norm initialization and - # implementation, try skipping LP regularization on layer norm and bias. - if p.apply_layer_norm: - f_t_ln_scale = py_utils.WeightParams( - shape=[self.hidden_size], - init=py_utils.WeightInit.Constant(1.0), - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('f_t_ln_scale', f_t_ln_scale, self.AddGlobalVN) - r_t_ln_scale = py_utils.WeightParams( - shape=[self.hidden_size], - init=py_utils.WeightInit.Constant(1.0), - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('r_t_ln_scale', r_t_ln_scale, self.AddGlobalVN) - c_t_ln_scale = py_utils.WeightParams( - shape=[self.hidden_size], - init=py_utils.WeightInit.Constant(1.0), - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('c_t_ln_scale', c_t_ln_scale, self.AddGlobalVN) - if not p.couple_input_forget_gates: - i_t_ln_scale = py_utils.WeightParams( - shape=[self.hidden_size], - init=py_utils.WeightInit.Constant(1.0), - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('i_t_ln_scale', i_t_ln_scale, self.AddGlobalVN) - - if p.pointwise_peephole: - f_t_vector_cell = py_utils.WeightParams( - shape=[self.hidden_size], - init=p.params_init, - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('f_t_vector_cell', f_t_vector_cell, self.AddGlobalVN) - r_t_vector_cell = py_utils.WeightParams( - shape=[self.hidden_size], - init=p.params_init, - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('r_t_vector_cell', r_t_vector_cell, self.AddGlobalVN) - if not p.couple_input_forget_gates: - i_t_vector_cell = py_utils.WeightParams( - shape=[self.hidden_size], - init=p.params_init, - dtype=p.dtype, - collections=self._VariableCollections()) - self.CreateVariable('i_t_vector_cell', i_t_vector_cell, - self.AddGlobalVN) - - if p.apply_pruning: - if p.gradient_pruning: - pruning_utils.AddToPruningCollections(self.vars.wm, self.vars.mask, - self.vars.threshold, - self.vars.gradient, - self.vars.old_weight, - self.vars.old_old_weight) - else: - pruning_utils.AddToPruningCollections(self.vars.wm, self.vars.mask, - self.vars.threshold) - if p.num_hidden_nodes: - if p.apply_pruning_to_projection: - if p.gradient_pruning: - pruning_utils.AddToPruningCollections(self.vars.w_proj, - self.vars.proj_mask, - self.vars.proj_threshold, - self.vars.proj_gradient, - self.vars.proj_old_weight, - self.vars.proj_old_old_weight) - else: - pruning_utils.AddToPruningCollections(self.vars.w_proj, - self.vars.proj_mask, - self.vars.proj_threshold) - - scope = tf.get_variable_scope() - # Collect some stats. - if p.couple_input_forget_gates: - x_t2, resized, f_t, r_t = tf.split( - value=self.vars.wm, num_or_size_splits=self.num_gates, axis=1) - else: - x_t2, resized, i_t, f_t, r_t = tf.split( - value=self.vars.wm, num_or_size_splits=self.num_gates, axis=1) - _HistogramSummary(p, scope.name + '/wm_i_t', i_t) - _HistogramSummary(p, scope.name + '/wm_x_t2', x_t2) - _HistogramSummary(p, scope.name + '/wm_resized', resized) - _HistogramSummary(p, scope.name + '/wm_f_t', f_t) - _HistogramSummary(p, scope.name + '/wm_r_t', r_t) - - @property - def output_size(self): - return self.params.num_output_nodes - - @property - def hidden_size(self): - return self.params.num_hidden_nodes or self.params.num_output_nodes - - @property - def num_gates(self): - return 4 if self.params.couple_input_forget_gates else 5 - - def batch_size(self, inputs): - return tf.shape(inputs.act[0])[0] - - def zero_state(self, theta, batch_size): - p = self.params - zero_m = py_utils.InitRNNCellState((batch_size, self.output_size), - init=p.zero_state_init_params, - dtype=py_utils.FPropDtype(p), - is_eval=self.do_eval) - zero_c = py_utils.InitRNNCellState((batch_size, self.hidden_size), - init=p.zero_state_init_params, - dtype=py_utils.FPropDtype(p), - is_eval=self.do_eval) - return py_utils.NestedMap(m=zero_m, c=zero_c) - - def GetOutput(self, state): - return state.m - - def LayerNorm(self, theta, gate_name, x, bias): - """Applies layer normalization on the last dimension of 'x'. - - Args: - theta: a NestedMap of layer params. - gate_name: the name of the gate, e.g., 'i_i', 'f_g', 'c', etc. - x: activation tensor, where the last dimension represents channels. - bias: the bias tensor of the gate. - - Returns: - Layer normalized 'x', with the same shape as the input. - """ - p = self.params - if p.apply_layer_norm: - mean = tf.reduce_mean(x, axis=[1], keepdims=True) - centered = x - mean - variance = tf.reduce_mean(tf.square(centered), axis=[1], keepdims=True) - normed = centered * tf.math.rsqrt(variance + p.layer_norm_epsilon) - scale = theta['%s_ln_scale' % gate_name] - x = normed * scale - return x + bias - - def _Mix(self, theta, state0, inputs): - assert isinstance(inputs.act, list) - if self.params.apply_pruning: - wm = tf.multiply(theta.wm, theta.mask) - else: - wm = theta.wm - return py_utils.Matmul(tf.concat(inputs.act, 1), wm) - - def _Gates(self, xmw, theta, state0, inputs): - """Compute the new state.""" - p = self.params - if p.couple_input_forget_gates: - x_t2, resized, f_t, r_t = tf.split( - value=xmw, num_or_size_splits=4, axis=1) - b_t2, b_resized, b_f, b_r = tf.split( - value=tf.expand_dims(theta.b, 0), num_or_size_splits=4, axis=1) - if p.pointwise_peephole: - f_t = f_t + tf.multiply(state0.c, theta.f_t_vector_cell) - f_t = self.LayerNorm(theta, 'f_t', f_t, b_f) - f_t = tf.nn.sigmoid(f_t) - i_t = 1.0 - f_t - else: - x_t2, resized, i_t, f_t, r_t = tf.split( - value=xmw + tf.expand_dims(theta.b, 0), num_or_size_splits=5, axis=1) - b_t2, b_resized, b_i, b_f, b_r = tf.split( - value=tf.expand_dims(theta.b, 0), num_or_size_splits=5, axis=1) - if p.pointwise_peephole: - f_t = f_t + tf.multiply(state0.c, theta.f_t_vector_cell) - i_t = i_t + tf.multiply(state0.c, theta.i_t_vector_cell) - f_t = self.LayerNorm(theta, 'f_t', f_t, b_f) - f_t = tf.nn.sigmoid(f_t) - i_t = self.LayerNorm(theta, 'i_t', i_t, b_i) - i_t = tf.nn.sigmoid(i_t) - - if p.pointwise_peephole: - r_t = r_t + tf.multiply(state0.c, theta.r_t_vector_cell) - r_t = self.LayerNorm(theta, 'r_t', r_t, b_r) - r_t = tf.nn.sigmoid(r_t) - - c_t = f_t * state0.c + i_t * x_t2 - c_t = self.LayerNorm(theta, 'c_t', c_t, 0) - - resized = tf.add(resized, b_resized) - x_t2 = tf.add(x_t2, b_t2) - # Clip the cell states to reasonable value. - if p.cell_value_cap is not None: - c_t = py_utils.clip_by_value(c_t, -p.cell_value_cap, p.cell_value_cap) - # Calculate state outputs. - g_c_t = tf.nn.tanh(c_t) - # Apply scaling factor if needed. - alpha = 1.0 - if p.hidden_scaling_factor: - # For the derivations of alpha please refer to variance computation of - # hidden cells h with respect to the variance of input x in appendix A.3 - # https://arxiv.org/pdf/1709.02755.pdf. - alpha = tf.sqrt(1.0 + tf.exp(p.bias_init.scale * 2.0)) - h_t = r_t * g_c_t + (1.0 - r_t) * resized * alpha - - if p.num_hidden_nodes: - if p.apply_pruning_to_projection: - w_proj = tf.multiply(theta.w_proj, theta.proj_mask) - else: - w_proj = theta.w_proj - h_t = tf.matmul(h_t, w_proj) - - return self._ApplyZoneOut(state0, inputs, c_t, h_t) - - def _ApplyZoneOut(self, state0, inputs, new_c, new_m): - """Apply ZoneOut and returns updated states.""" - p = self.params - if p.zo_prob > 0.0: - assert not py_utils.use_tpu(), ( - 'SRUCell does not support zoneout on TPU yet.') - c_random_uniform = tf.random.uniform(tf.shape(new_c), seed=p.random_seed) - m_random_uniform = tf.random.uniform(tf.shape(new_m), seed=p.random_seed) - else: - c_random_uniform = None - m_random_uniform = None - - new_c = self._ZoneOut(state0.c, new_c, inputs.padding, p.zo_prob, - self.do_eval, c_random_uniform) - new_m = self._ZoneOut(state0.m, new_m, inputs.padding, p.zo_prob, - self.do_eval, m_random_uniform) - new_c.set_shape(state0.c.shape) - new_m.set_shape(state0.m.shape) - return py_utils.NestedMap(m=new_m, c=new_c) - - -class QRNNPoolingCell(RNNCell): - """This implements just the "pooling" part of a quasi-RNN or SRU. - - From these papers: - - - https://arxiv.org/abs/1611.01576 - - https://arxiv.org/abs/1709.02755 - - The pooling part implements gates for recurrence. These architectures split - the transform (conv or FC) from the gating/recurrent part. This cell can - do either the quasi-RNN style or SRU style pooling operation based on params. - - If you want all of the functionality in one RNN cell, use `SRUCell` instead. - - theta: - - Has the trainable zero state. Other weights are done outside the recurrent - loop. - - state: - - - m: the qrnn output. [batch, cell_nodes] - - c: the qrnn cell state. [batch, cell_nodes] - - inputs: - - - act: a list of input activations. [batch, input_nodes * num_rnn_matrices] - - padding: the padding. [batch, 1]. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'cell_value_cap', 10.0, 'LSTM cell values are capped to be within ' - ' [-cell_value_cap, +cell_value_cap] if the value is not None. ' - 'It can be a scalar, a scalar tensor or None. When set to None, ' - 'no capping is applied.') - p.Define('zo_prob', 0.0, - 'If > 0, applies ZoneOut regularization with the given prob.') - p.Define('pooling_formula', 'INVALID', - 'Options: quasi_ifo, sru. Which pooling math to use') - return p - - def __init__(self, params): - """Initializes quasi-RNN Cell.""" - super().__init__(params) - assert isinstance(params, hyperparams.Params) - p = self.params - assert p.reset_cell_state is False, ('QRNNPoolingCell currently doesnt ' - 'support resetting cell state.') - assert p.pooling_formula in ('quasi_ifo', 'sru') - assert isinstance(p.cell_value_cap, - (int, float)) or p.cell_value_cap is None - - self._timestep = -1 - - def batch_size(self, inputs): - return tf.shape(inputs.act[0])[0] - - def zero_state(self, theta, batch_size): - p = self.params - zero_m = py_utils.InitRNNCellState((batch_size, p.num_output_nodes), - init=p.zero_state_init_params, - dtype=p.dtype, - is_eval=self.do_eval) - zero_c = py_utils.InitRNNCellState((batch_size, p.num_output_nodes), - init=p.zero_state_init_params, - dtype=p.dtype, - is_eval=self.do_eval) - return py_utils.NestedMap(m=zero_m, c=zero_c) - - def GetOutput(self, state): - return state.m - - def _Mix(self, theta, state0, inputs): - assert isinstance(inputs.act, list) - # Just do identity. The convolution part of the QRNN has to be done earlier. - return inputs.act - - def _Gates(self, xmw, theta, state0, inputs): - """Compute the new state.""" - p = self.params - if p.pooling_formula == 'quasi_ifo': - z_t, i_t, f_t, o_t = tf.split( - value=tf.concat(xmw, 1), num_or_size_splits=4, axis=1) - # Quasi-RNN "ifo" pooling - c_t = f_t * state0.c + i_t * z_t - h_t = o_t * c_t - elif p.pooling_formula == 'sru': - x_t2, resized, f_t, r_t = tf.split( - value=tf.concat(xmw, 1), num_or_size_splits=4, axis=1) - c_t = f_t * state0.c + (1.0 - f_t) * x_t2 - # TODO(otaviogood): Optimization - Since state doesn't depend on these - # ops, they can be moved outside the loop. - g_c_t = tf.nn.tanh(c_t) - h_t = r_t * g_c_t + (1.0 - r_t) * resized - else: - raise ValueError('Invalid pooling_formula: %s', p.pooling_formula) - - new_c = c_t - new_m = h_t - - # Clip the cell states to reasonable value. - if p.cell_value_cap is not None: - new_c = py_utils.clip_by_value(new_c, -p.cell_value_cap, p.cell_value_cap) - - # Apply Zoneout. - return self._ApplyZoneOut(state0, inputs, new_c, new_m) - - def _ApplyZoneOut(self, state0, inputs, new_c, new_m): - """Apply Zoneout and returns the updated states.""" - p = self.params - if p.zo_prob > 0.0: - c_random_uniform = tf.random.uniform(tf.shape(new_c), seed=p.random_seed) - m_random_uniform = tf.random.uniform(tf.shape(new_m), seed=p.random_seed) - else: - c_random_uniform = None - m_random_uniform = None - - new_c = self._ZoneOut(state0.c, new_c, inputs.padding, p.zo_prob, - self.do_eval, c_random_uniform) - new_m = self._ZoneOut(state0.m, new_m, inputs.padding, p.zo_prob, - self.do_eval, m_random_uniform) - new_c.set_shape(state0.c.shape) - new_m.set_shape(state0.m.shape) - return py_utils.NestedMap(m=new_m, c=new_c) - - -class GRUCell(RNNCell): - """ Gated Recurrent Unit cell. - - implemented: layer normalization, gru_biasing, gru_cell cap, - not yet implemented: pruning, quantization, zone-out (enforced to 0.0 now) - reference: https://arxiv.org/pdf/1412.3555.pdf - - theta: - - - w_n: the parameter weight matrix for the input block. - - w_u: the parameter weight matrix for the update gate - - w_r: the parameter weight matrix for the reset gate - - b_n: the bias vector for the input block - - b_u: the bias vector for the update gate - - b_r: the bias vector for the reset gate - - state: - - - m: the GRU output. [batch, output_cell_nodes] - - c: the GRU cell state. [batch, hidden_cell_nodes] - - inputs: - - - act: a list of input activations. [batch, input_nodes] - - padding: the padding. [batch, 1]. - - reset_mask: optional 0/1 float input to support packed input training. - Shape [batch, 1] - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'num_hidden_nodes', 0, 'Number of projection hidden nodes ' - '(see https://arxiv.org/abs/1603.08042). ' - 'Set to 0 to disable projection.') - p.Define( - 'cell_value_cap', 10.0, 'GRU cell values are capped to be within ' - ' [-cell_value_cap, +cell_value_cap] if the value is not None. ' - 'It can be a scalar, a scalar tensor or None. When set to None, ' - 'no capping is applied.') - p.Define('enable_gru_bias', False, 'Enable the GRU Cell bias.') - p.Define('bias_init', py_utils.WeightInit.Constant(0.0), - 'Initialization parameters for GRU Cell bias') - p.Define('zo_prob', 0.0, - 'If > 0, applies ZoneOut regularization with the given prob.') - p.Define('apply_layer_norm', True, 'Apply layer norm to the variables') - p.Define( - 'layer_norm_epsilon', 1e-8, 'Tiny value to guard rsqr against.' - 'value is necessary only if apply_layer_norm is True') - return p - - def __init__(self, params): - """Initializes GRUCell.""" - super().__init__(params) - assert isinstance(params, hyperparams.Params) - p = self.params - assert isinstance(p.cell_value_cap, - (int, float)) or p.cell_value_cap is None - assert p.zo_prob == 0.0 - - self._timestep = -1 - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - p = self.params - - def CreateVarHelper(variable_name, shape_to_init, params_to_init): - """Utility function to initialize variables. - - Args: - variable_name: the name of the variable - shape_to_init: shape of the variables to be initialized. - params_to_init: p.params_init, p.bias_init, or otherwise specified - returns: initialized variable with name "$variable_name" - """ - return self.CreateVariable( - variable_name, - py_utils.WeightParams( - shape=shape_to_init, - init=params_to_init, - dtype=p.dtype, - collections=self._VariableCollections()), self.AddGlobalVN) - - # Define weights. - # Weight for block input - CreateVarHelper('w_n', - [p.num_input_nodes + self.output_size, self.hidden_size], - p.params_init) - # Weight for update gate - CreateVarHelper('w_u', - [p.num_input_nodes + self.output_size, self.hidden_size], - p.params_init) - # Weight for reset gate - CreateVarHelper('w_r', - [p.num_input_nodes + self.output_size, self.output_size], - p.params_init) - - if p.num_hidden_nodes: - # Set up projection matrix - CreateVarHelper('w_proj', [self.hidden_size, self.output_size], - p.params_init) - CreateVarHelper('b_proj', [self.output_size], p.bias_init) - - if p.enable_gru_bias: - # Bias for the block input - CreateVarHelper('b_n', [self.hidden_size], p.bias_init) - # Bias for update gate - CreateVarHelper('b_u', [self.hidden_size], p.bias_init) - # Bias for the reset gate - CreateVarHelper('b_r', [self.output_size], p.bias_init) - - if p.apply_layer_norm: - assert p.layer_norm_epsilon is not None - ln_unit = py_utils.WeightInit.Constant(0.0) - CreateVarHelper('bn_ln_scale', [self.hidden_size], ln_unit) - CreateVarHelper('bu_ln_scale', [self.hidden_size], ln_unit) - CreateVarHelper('br_ln_scale', [self.output_size], ln_unit) - - @property - def output_size(self): - return self.params.num_output_nodes - - @property - def hidden_size(self): - return self.params.num_hidden_nodes or self.params.num_output_nodes - - def batch_size(self, inputs): - return tf.shape(inputs.act[0])[0] - - def zero_state(self, theta, batch_size): - p = self.params - zero_m = py_utils.InitRNNCellState((batch_size, self.output_size), - init=p.zero_state_init_params, - dtype=py_utils.FPropDtype(p), - is_eval=self.do_eval) - zero_c = py_utils.InitRNNCellState((batch_size, self.hidden_size), - init=p.zero_state_init_params, - dtype=py_utils.FPropDtype(p), - is_eval=self.do_eval) - return py_utils.NestedMap(m=zero_m, c=zero_c) - - def _ResetState(self, state, inputs): - state.m = inputs.reset_mask * state.m - state.c = inputs.reset_mask * state.c - return state - - def GetOutput(self, state): - return state.m - - def LayerNorm(self, x, scale): - """Applies layer normalization on the last dimension of 'x'. - - Args: - x: activation tensor, where the last dimension represents channels. - scale: the scale tensor of the layer normalization - - Returns: - Layer normalized 'x', with the same shape as the input. - """ - p = self.params - mean = tf.reduce_mean(x, axis=[1], keepdims=True) - centered = x - mean - variance = tf.reduce_mean(tf.square(centered), axis=[1], keepdims=True) - normed = centered * tf.math.rsqrt(variance + p.layer_norm_epsilon) - return normed * scale - - def FProp(self, theta, state0, inputs): - """Forward function. - - GRU has coupled reset gate in the candidate actiavation function for output. - See equation 5 and above in https://arxiv.org/pdf/1412.3555.pdf. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - state0: The previous recurrent state. A `.NestedMap`. - inputs: The inputs to the cell. A `.NestedMap`. - - Returns: - A tuple (state1, extras). - - state1: The next recurrent state. A `.NestedMap`. - - extras: Intermediate results to faciliate backprop. A `.NestedMap`. - """ - - p = self.params - assert isinstance(inputs.act, list) - - # Update all gates - # Compute r_g. r_g has size [batch, output] - r_g = tf.matmul(tf.concat(inputs.act + [state0.m], 1), theta.w_r) - if p.apply_layer_norm: - r_g = self.LayerNorm(r_g, theta.br_ln_scale + 1.0) - if p.enable_gru_bias: - r_g = r_g + theta.b_r - r_g = tf.sigmoid(r_g) - - # Compute u_g and n_g. Both have size [batch, hidden]. - # u_g has size [batch, hidden] - u_g = tf.matmul(tf.concat(inputs.act + [state0.m], 1), theta.w_u) - # size of n_g is [batch, hidden] - n_g = tf.matmul( - tf.concat(inputs.act + [tf.multiply(r_g, state0.m)], 1), theta.w_n) - if p.apply_layer_norm: - u_g = self.LayerNorm(u_g, theta.bu_ln_scale + 1.0) - n_g = self.LayerNorm(n_g, theta.bn_ln_scale + 1.0) - if p.enable_gru_bias: # Add biases to u_g and n_g if needed - u_g = u_g + theta.b_u - n_g = n_g + theta.b_n - - u_g = tf.sigmoid(u_g) - n_g = tf.tanh(n_g) - - new_c = (1.0 - u_g) * (state0.c) + u_g * n_g - - # Clip the cell states to reasonable value. - if p.cell_value_cap is not None: - new_c = py_utils.clip_by_value(new_c, -p.cell_value_cap, p.cell_value_cap) - - # Apply non-linear output is necessary - new_m = new_c - # Apply projection matrix if necessary - if p.num_hidden_nodes: - new_m = tf.matmul(new_m, theta.w_proj) + theta.b_proj - # Apply padding. - new_m = py_utils.ApplyPadding(inputs.padding, new_m, state0.m) - new_c = py_utils.ApplyPadding(inputs.padding, new_c, state0.c) - return py_utils.NestedMap(m=new_m, c=new_c), py_utils.NestedMap() diff --git a/lingvo/core/rnn_cell_test.py b/lingvo/core/rnn_cell_test.py deleted file mode 100644 index 9992c02c..00000000 --- a/lingvo/core/rnn_cell_test.py +++ /dev/null @@ -1,1447 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for rnn_cell.""" - -from absl.testing import parameterized -import lingvo.compat as tf -from lingvo.core import py_utils -from lingvo.core import quant_utils -from lingvo.core import rnn_cell -from lingvo.core import test_utils -import numpy as np - -_INIT_RANDOM_SEED = 429891685 -_NUMPY_RANDOM_SEED = 12345 -_RANDOM_SEED = 98274 - - -class RNNCellTest(test_utils.TestCase, parameterized.TestCase): - - # pyformat: disable - @parameterized.named_parameters( - ('_NoInlineNoGruBias', False, False, - [[-1.085402, 2.161964], [-0.933972, 1.995606], [-0.892969, 2.059967]], - [[0.500292, 0.732436], [0.34267, 0.732542], [0.341799, 0.815305]]), - ('_NoInlineGruBias', False, True, - [[-1.206088, 2.558667], [-1.024555, 2.359131], [-1.006608, 2.385566]], - [[0.726844, 0.932083], [0.537847, 0.932127], [0.536803, 0.967041]]), - ('_InlineNoGruBias', True, False, - [[-1.085402, 2.161964], [-0.933972, 1.995606], [-0.892969, 2.059967]], - [[0.500292, 0.732436], [0.34267, 0.732542], [0.341799, 0.815305]]), - ('_InlineGruBias', True, True, - [[-1.206088, 2.558667], [-1.024555, 2.359131], [-1.006608, 2.385566]], - [[0.726844, 0.932083], [0.537847, 0.932127], [0.536803, 0.967041]])) - # pyformat: enable - def testGRUCell(self, inline, enable_gru_bias, m_expected, c_expected): - params = rnn_cell.GRUCell.Params().Set( - name='gru_rnn', - params_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - bias_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - num_input_nodes=2, - num_output_nodes=2, - num_hidden_nodes=2, - enable_gru_bias=enable_gru_bias) - params.vn.global_vn = False # do not set variational noise - params.vn.per_step_vn = False # do not set step wise noise - - gru = rnn_cell.GRUCell(params) - - tf.logging.info('gru vars = %s', gru.vars) - self.assertIn('w_r', gru.vars.w_r.name) - self.assertIn('w_u', gru.vars.w_u.name) - self.assertIn('w_n', gru.vars.w_n.name) - - if enable_gru_bias: - self.assertIn('b_n', gru.vars.b_n.name) - self.assertIn('b_r', gru.vars.b_r.name) - self.assertIn('b_u', gru.vars.b_u.name) - - self.assertEqual( - gru.theta.w_n.get_shape(), - tf.TensorShape([ - params.num_input_nodes + params.num_output_nodes, - params.num_hidden_nodes, - ])) - self.assertEqual( - gru.theta.w_u.get_shape(), - tf.TensorShape([ - params.num_input_nodes + params.num_output_nodes, - params.num_hidden_nodes, - ])) - self.assertEqual( - gru.theta.w_r.get_shape(), - tf.TensorShape([ - params.num_input_nodes + params.num_output_nodes, - params.num_output_nodes, - ])) - - if enable_gru_bias: - self.assertEqual(gru.theta.b_n.get_shape(), - tf.TensorShape([params.num_hidden_nodes])) - self.assertEqual(gru.theta.b_u.get_shape(), - tf.TensorShape([params.num_hidden_nodes])) - self.assertEqual(gru.theta.b_r.get_shape(), - tf.TensorShape([params.num_output_nodes])) - - # Start feeding in inputs. - np.random.seed(_NUMPY_RANDOM_SEED) - inputs = py_utils.NestedMap( - act=[tf.constant(np.random.uniform(size=(3, 2)), tf.float32)], - padding=tf.zeros([3, 1])) - c_values = tf.constant(np.random.uniform(size=(3, 2)), tf.float32) - m_values = tf.constant(np.random.uniform(size=(3, 2)), tf.float32) - state0 = py_utils.NestedMap(c=c_values, m=m_values) - state1, _ = gru.FPropDefaultTheta(state0, inputs) - - with self.session( - use_gpu=False, config=py_utils.SessionConfig(inline=inline)): - self.evaluate(tf.global_variables_initializer()) - - variable_count = 11 if enable_gru_bias else 8 - wts = tf.get_collection('GRUCell_vars') - self.assertLen(wts, variable_count) - - self.assertAllClose(m_expected, state1.m.eval()) - self.assertAllClose(c_expected, state1.c.eval()) - - # pyformat: disable - @parameterized.named_parameters( - ('_NoInlineNoCIFGNoLSTMBias', False, False, False, - [[0.095727, 0.476658], [0.04662, 0.180589], [0.001656, 0.374141]], - [[0.241993, 0.820267], [0.086863, 0.349722], [0.003176, 0.655448]]), - ('_NoInlineNoCIFGLSTMBias', False, False, True, - [[0.007753, 0.66843], [-0.029904, 0.485617], [-0.026663, 0.654127]], - [[0.033096, 1.013467], [-0.086807, 0.748031], [-0.08087, 1.04254]]), - ('_NoInlineCIFGNoLSTMBias', False, True, False, - [[0.22088, 0.244225], [0.123647, 0.25378], [0.163328, 0.214796]], - [[0.355682, 0.711696], [0.313728, 0.633475], [0.485248, 0.961122]]), - ('_NoInlineCIFGLSTMBias', False, True, True, - [[0.342635, 0.182102], [0.140832, 0.210234], [0.224034, 0.155077]], - [[0.499417, 0.701774], [0.278458, 0.697437], [0.51618, 0.964456]]), - ('_InlineNoCIFGNoLSTMBias', True, False, False, - [[0.095727, 0.476658], [0.04662, 0.180589], [0.001656, 0.374141]], - [[0.241993, 0.820267], [0.086863, 0.349722], [0.003176, 0.655448]]), - ('_InlineNoCIFGLSTMBias', True, False, True, - [[0.007753, 0.66843], [-0.029904, 0.485617], [-0.026663, 0.654127]], - [[0.033096, 1.013467], [-0.086807, 0.748031], [-0.08087, 1.04254]]), - ('_InlineCIFGNoLSTMBias', True, True, False, - [[0.22088, 0.244225], [0.123647, 0.25378], [0.163328, 0.214796]], - [[0.355682, 0.711696], [0.313728, 0.633475], [0.485248, 0.961122]]), - ('_InlineCIFGLSTMBias', True, True, True, - [[0.342635, 0.182102], [0.140832, 0.210234], [0.224034, 0.155077]], - [[0.499417, 0.701774], [0.278458, 0.697437], [0.51618, 0.964456]])) - # pyformat: enable - def testLSTMSimple_P1(self, inline, couple_input_forget_gates, - enable_lstm_bias, m_expected, c_expected): - params = rnn_cell.LSTMCellSimple.Params().Set( - name='lstm', - params_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - bias_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - num_input_nodes=2, - num_output_nodes=2, - couple_input_forget_gates=couple_input_forget_gates, - enable_lstm_bias=enable_lstm_bias) - params.vn.global_vn = False - params.vn.per_step_vn = False - - lstm = rnn_cell.LSTMCellSimple(params) - tf.logging.info('lstm vars = %s', lstm.vars) - self.assertIn('wm', lstm.vars.wm.name) - - if enable_lstm_bias: - self.assertIn('b', lstm.vars.b.name) - - num_param_vectors = 6 if couple_input_forget_gates else 8 - self.assertEqual(lstm.theta.wm.get_shape(), - tf.TensorShape([4, num_param_vectors])) - - if enable_lstm_bias: - self.assertEqual(lstm.theta.b.get_shape(), - tf.TensorShape([num_param_vectors])) - - np.random.seed(_NUMPY_RANDOM_SEED) - inputs = py_utils.NestedMap( - act=[tf.constant(np.random.uniform(size=(3, 2)), tf.float32)], - padding=tf.zeros([3, 1])) - state0 = py_utils.NestedMap( - c=tf.constant(np.random.uniform(size=(3, 2)), tf.float32), - m=tf.constant(np.random.uniform(size=(3, 2)), tf.float32)) - - state1, _ = lstm.FPropDefaultTheta(state0, inputs) - - with self.session( - use_gpu=False, config=py_utils.SessionConfig(inline=inline)): - self.evaluate(tf.global_variables_initializer()) - - variable_count = 2 if enable_lstm_bias else 1 - wts = tf.get_collection('LSTMCellSimple_vars') - self.assertLen(wts, variable_count) - - # pyformat: disable - # xmw_expected = [ - # [-0.74310219, 1.10182762, 0.67478961, 0.62169313, 0.77394271, - # -0.1691505, -0.39185536, 0.87572402], - # [-0.78952235, 0.04464795, 0.00245538, -0.34931657, 0.22463873, - # 0.02745318, 0.15253648, 0.14931624], - # [-1.58246589, 0.03950393, 0.18513964, -0.25745165, 0.73317981, - # 0.68082684, 0.08576801, 0.62040436]] - # pyformat: enable - self.assertAllClose(m_expected, state1.m.eval()) - self.assertAllClose(c_expected, state1.c.eval()) - - # pyformat: disable - @parameterized.named_parameters( - ('_Masked', 0, 2, True, False, - [[0.095727, 0.476658], [0.04662, 0.180589], [0.001656, 0.374141]], - [[0.241993, 0.820267], [0.086863, 0.349722], [0.003176, 0.655448]]), - ('_MaskedProjections', 2, 1, True, True, - [[0.414049], [0.076521], [0.356313]], - [[0.270425, 0.840373], [0.349856, 0.440421], [0.261243, 0.889804]]), - ('_Projections', 2, 1, False, False, [[0.414049], [0.076521], [0.356313]], - [[0.270425, 0.840373], [0.349856, 0.440421], [0.261243, 0.889804]])) - # pyformat: enable - def testLSTMSimple_P2(self, num_hidden_nodes, num_output_nodes, apply_pruning, - apply_pruning_to_projection, m_expected, c_expected): - params = rnn_cell.LSTMCellSimple.Params().Set( - name='lstm', - params_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - num_input_nodes=2, - num_hidden_nodes=num_hidden_nodes, - num_output_nodes=num_output_nodes, - apply_pruning=apply_pruning, - apply_pruning_to_projection=apply_pruning_to_projection) - params.vn.global_vn = False - params.vn.per_step_vn = False - lstm = rnn_cell.LSTMCellSimple(params) - - tf.logging.info('lstm vars = %s', lstm.vars) - self.assertIn('wm', lstm.vars.wm.name) - self.assertIn('b', lstm.vars.b.name) - if apply_pruning: - self.assertIn('mask', lstm.vars.mask.name) - self.assertIn('threshold', lstm.vars.threshold.name) - if apply_pruning_to_projection: - self.assertIn('w_proj', lstm.vars.w_proj.name) - self.assertIn('proj_mask', lstm.vars.proj_mask.name) - self.assertIn('proj_threshold', lstm.vars.proj_threshold.name) - - num_io = params.num_input_nodes + num_output_nodes - num_param_vectors = 8 - self.assertEqual(lstm.theta.wm.get_shape(), - tf.TensorShape([num_io, num_param_vectors])) - self.assertEqual(lstm.theta.b.get_shape(), - tf.TensorShape([num_param_vectors])) - if apply_pruning_to_projection: - self.assertEqual(lstm.theta.w_proj.get_shape(), tf.TensorShape([2, 1])) - self.assertEqual(lstm.theta.proj_mask.get_shape(), tf.TensorShape([2, 1])) - - np.random.seed(_NUMPY_RANDOM_SEED) - inputs = py_utils.NestedMap( - act=[tf.constant(np.random.uniform(size=(3, 2)), tf.float32)], - padding=tf.zeros([3, 1])) - state0 = py_utils.NestedMap( - c=tf.constant(np.random.uniform(size=(3, 2)), tf.float32), - m=tf.constant( - np.random.uniform(size=(3, num_output_nodes)), tf.float32)) - - state1, _ = lstm.FPropDefaultTheta(state0, inputs) - - with self.session( - use_gpu=False, config=py_utils.SessionConfig(inline=False)): - self.evaluate(tf.global_variables_initializer()) - - if num_hidden_nodes > 0: - variable_count = 3 # weights, biases, projection. - else: - variable_count = 2 # weights, biases. - wts = tf.get_collection('LSTMCellSimple_vars') - self.assertLen(wts, variable_count) - - if apply_pruning: - num_vars = 2 if apply_pruning_to_projection else 1 - masks = tf.get_collection('masks') - self.assertLen(masks, num_vars) - - threshold = tf.get_collection('thresholds') - self.assertLen(threshold, num_vars) - - self.assertAllClose(m_expected, state1.m.eval()) - self.assertAllClose(c_expected, state1.c.eval()) - - # pyformat: disable - @parameterized.named_parameters( - ('_GlobalNoise', True, False, tf.float32, True, False, 2, 0., - [[0.080182, 0.4585], [0.050852, 0.245296], [0.023557, 0.382329]], - [[0.387777, 0.819644], [-0.160634, 0.513716], [-0.179584, 0.862915]]), - ('_Double', True, False, tf.float64, True, False, 2, 0., - [[0.3472136, 0.11880029], [0.14214374, 0.33760977], - [0.1168568, 0.32053401]], - [[1.46477364, 0.43743008], [0.57051592, 0.14892339], - [0.6949858, 0.16326128]]), - ('_NoOutputNonlinearity', False, False, tf.float64, False, False, 2, 0., - [[0.532625, 0.083511], [0.118662, 0.110532], [0.121542, 0.084161]], - [[0.789908, 0.312811], [0.192642, 0.207369], [0.167591, 0.172713]]), - ('_ByPass', False, False, tf.float64, False, True, 2, 0., None, None), - ('_WithForgetGateBias', True, False, tf.float64, True, False, 2, -1., - [[0.19534954, 0.10979363], [0.02134449, 0.2821926], - [-0.02530111, 0.25382254]], - [[1.29934979, 0.31769676], [0.41655035, 0.05172589], - [0.58909841, -0.00438461]])) - # pyformat: enable - def testLSTMSimple_P3(self, global_vn, couple_input_forget_gates, dtype, - output_nonlinearity, bypass, num_output_nodes, - forget_gate_bias, m_expected, c_expected): - params = rnn_cell.LSTMCellSimple.Params().Set( - name='lstm', - couple_input_forget_gates=couple_input_forget_gates, - output_nonlinearity=output_nonlinearity, - params_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - num_input_nodes=2, - num_output_nodes=num_output_nodes, - dtype=dtype, - forget_gate_bias=forget_gate_bias) - params.vn.seed = 8820495 - params.vn.global_vn = global_vn - params.vn.per_step_vn = False - params.vn.scale = 0.5 - - lstm = rnn_cell.LSTMCellSimple(params) - - np.random.seed(_NUMPY_RANDOM_SEED) - padding = tf.ones([3, 1]) if bypass else tf.zeros([3, 1]) - inputs = py_utils.NestedMap( - act=[tf.constant(np.random.uniform(size=(3, 2)), dtype)], - padding=padding) - state0 = py_utils.NestedMap( - c=tf.constant(np.random.uniform(size=(3, 2)), dtype), - m=tf.constant(np.random.uniform(size=(3, 2)), dtype)) - - state1, _ = lstm.FPropDefaultTheta(state0, inputs) - - with self.session(use_gpu=False): - self.evaluate(tf.global_variables_initializer()) - - wts = tf.get_collection('LSTMCellSimple_vars') - self.assertLen(wts, 2) - - if bypass: - m_expected = state0.m.eval() - c_expected = state0.c.eval() - self.assertAllClose(m_expected, state1.m.eval()) - self.assertAllClose(c_expected, state1.c.eval()) - - # pyformat: disable - @parameterized.named_parameters( - ('', False, - [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 2.1, 2.1, 2.1, 0.1, 0.1, 0.1]), - ('_CoupledInputForget', True, - [0.1, 0.1, 0.1, 2.1, 2.1, 2.1, 0.1, 0.1, 0.1])) - # pyformat: enable - def testLSTMSimpleWithForgetGateInitBias(self, couple_input_forget_gates, - b_expected): - params = rnn_cell.LSTMCellSimple.Params().Set( - name='lstm', - params_init=py_utils.WeightInit.Constant(0.1), - couple_input_forget_gates=couple_input_forget_gates, - num_input_nodes=2, - num_output_nodes=3, - forget_gate_bias=2.0, - bias_init=py_utils.WeightInit.Constant(0.1), - dtype=tf.float64) - - lstm = rnn_cell.LSTMCellSimple(params) - - np.random.seed(_NUMPY_RANDOM_SEED) - with self.session(use_gpu=False): - self.evaluate(tf.global_variables_initializer()) - b_value = lstm._GetBias(lstm.theta).eval() - tf.logging.info('testLSTMSimpleWithForgetGateInitBias b = %s', - np.array_repr(b_value)) - self.assertAllClose(b_value, b_expected) - - # pyformat: disable - @parameterized.named_parameters( - ('_NoLSTMBias', False, - [[0.09375, 0.460938], [0.046875, 0.179688], [0.039062, 0.375]], - [[0.234375, 0.789062], [0.09375, 0.351562], [0.078125, 0.664062]]), - ('_LSTMBias', True, - [[0.039062, 0.476562], [0.007812, 0.445312], [0.015625, 0.546875]], - [[0.148438, 0.78125], [0.023438, 0.710938], [0.039062, 0.984375]])) - # pyformat: enable - def testQuantizedLSTMSimple(self, enable_lstm_bias, m_expected, c_expected): - params = rnn_cell.LSTMCellSimple.Params().Set( - name='lstm', - params_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - bias_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - num_input_nodes=2, - num_output_nodes=2, - couple_input_forget_gates=False, - enable_lstm_bias=enable_lstm_bias) - params.vn.global_vn = False - params.vn.per_step_vn = False - - cc_schedule = quant_utils.FakeQuantizationSchedule.Params().Set( - clip_start_step=0, # Step 0 is unclipped. - clip_end_step=0, - quant_start_step=0, - start_cap=1.0, - end_cap=1.0) - params.qdomain.default = quant_utils.SymmetricScheduledClipQDomain.Params( - ).Set(cc_schedule=cc_schedule.Copy()) - params.qdomain.c_state = quant_utils.SymmetricScheduledClipQDomain.Params( - ).Set(cc_schedule=cc_schedule.Copy()) - params.qdomain.m_state = quant_utils.SymmetricScheduledClipQDomain.Params( - ).Set(cc_schedule=cc_schedule.Copy()) - params.qdomain.fullyconnected = ( - quant_utils.SymmetricScheduledClipQDomain.Params().Set( - cc_schedule=cc_schedule.Copy())) - - params.cell_value_cap = None - - lstm = rnn_cell.LSTMCellSimple(params) - - tf.logging.info('lstm vars = %s', lstm.vars) - self.assertIn('wm', lstm.vars.wm.name) - - if enable_lstm_bias: - self.assertIn('b', lstm.vars.b.name) - - num_param_vectors = 8 - self.assertEqual(lstm.theta.wm.get_shape(), - tf.TensorShape([4, num_param_vectors])) - - if enable_lstm_bias: - self.assertEqual(lstm.theta.b.get_shape(), - tf.TensorShape([num_param_vectors])) - - np.random.seed(_NUMPY_RANDOM_SEED) - inputs = py_utils.NestedMap( - act=[tf.constant(np.random.uniform(size=(3, 2)), tf.float32)], - padding=tf.zeros([3, 1])) - state0 = py_utils.NestedMap( - c=tf.constant(np.random.uniform(size=(3, 2)), tf.float32), - m=tf.constant(np.random.uniform(size=(3, 2)), tf.float32)) - - state1, _ = lstm.FPropDefaultTheta(state0, inputs) - - with self.session( - use_gpu=False, config=py_utils.SessionConfig(inline=False)): - self.evaluate(tf.global_variables_initializer()) - - variable_count = 2 if enable_lstm_bias else 1 - wts = tf.get_collection('LSTMCellSimple_vars') - self.assertLen(wts, variable_count) - - self.assertAllClose(m_expected, state1.m.eval()) - self.assertAllClose(c_expected, state1.c.eval()) - - @parameterized.named_parameters(('_WithoutOutputShuffling', 1), - ('_WithOutputShuffling', 2)) - def testLSTMCellGrouped(self, num_shuffle_shards): - params = rnn_cell.LSTMCellGrouped.Params().Set( - name='lstm', - num_input_nodes=8, - num_output_nodes=8, - num_groups=4, - num_shuffle_shards=num_shuffle_shards) - child_p = params.child_lstm_tpl - child_p.output_nonlinearity = True - child_p.params_init = py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED) - child_p.vn.global_vn = False - child_p.vn.per_step_vn = False - - lstm = params.Instantiate() - - tf.logging.info('lstm vars = %s', lstm.vars) - for child_lstm in lstm.groups: - self.assertIn('wm', child_lstm.vars.wm.name) - self.assertIn('b', child_lstm.vars.b.name) - - self.assertEqual(child_lstm.theta.wm.get_shape(), tf.TensorShape([4, 8])) - self.assertEqual(child_lstm.theta.b.get_shape(), tf.TensorShape([8])) - - np.random.seed(_NUMPY_RANDOM_SEED) - inputs = py_utils.NestedMap( - act=[tf.constant(np.random.uniform(size=(3, 8)), tf.float32)], - padding=tf.zeros([3, 1])) - state0 = py_utils.NestedMap( - groups=py_utils.SplitRecursively( - py_utils.NestedMap( - c=tf.constant(np.random.uniform(size=(3, 8)), tf.float32), - m=tf.constant(np.random.uniform( - size=(3, 8)), tf.float32)), params.num_groups)) - - state1, _ = lstm.FPropDefaultTheta(state0, inputs) - self.assertLen(state1.groups, params.num_groups) - out1 = lstm.GetOutput(state1) - - with self.session( - use_gpu=False, config=py_utils.SessionConfig(inline=False)): - self.evaluate(tf.global_variables_initializer()) - - variable_count = 2 * params.num_groups # one for weights, one for biases. - wts = tf.get_collection('LSTMCellSimple_vars') - self.assertLen(wts, variable_count) - - state1 = py_utils.ConcatRecursively(state1.groups) - m_actual = state1.m.eval() - c_actual = state1.c.eval() - out_actual = out1.eval() - - tf.logging.info('m_actual = %s', np.array_repr(m_actual)) - tf.logging.info('c_actual = %s', np.array_repr(c_actual)) - tf.logging.info('out_actual = %s', np.array_repr(out_actual)) - - # pyformat: disable - # pylint: disable=bad-whitespace,bad-continuation - m_expected = [ - [ - -0.07857136, 0.43932292, 0.11373602, 0.16337454, - 0.01618987, 0.09685542, -0.20168062, 0.52612996, - ], - [ - 0.07929622, 0.18910739, -0.11084013, 0.32307294, - 0.03500029, -0.05823045, 0.16963124, 0.27039385, - ], - [ - 0.11623365, 0.38104215, 0.00935007, 0.22124135, - -0.17368057, 0.10859803, -0.06948104, 0.10925373, - ], - ] - c_expected = [ - [ - -0.23670214, 0.66260374, 0.24650344, 0.28946888, - 0.03051668, 0.15143034, -0.52736223, 0.88325077, - ], - [ - 0.16262427, 0.28568456, -0.19542629, 0.52116692, - 0.06872599, -0.1123996, 0.31477568, 0.49881396, - ], - [ - 0.19667494, 0.68746102, 0.02078706, 0.30816019, - -0.36376655, 0.16003416, -0.16141629, 0.16648693, - ], - ] - # pylint: enable=bad-whitespace,bad-continuation - # pyformat: enable - out_expected = m_expected - if num_shuffle_shards > 1: - - def _ShuffleShards(x): - return [[row[i] for i in (0, 3, 2, 5, 4, 7, 6, 1)] for row in x] - - assert num_shuffle_shards == 2 - out_expected = _ShuffleShards(out_expected) - self.assertAllClose(m_expected, m_actual) - self.assertAllClose(c_expected, c_actual) - self.assertAllClose(out_expected, out_actual) - - def testLSTMCellGroupedNoInputSplit(self): - params = rnn_cell.LSTMCellGrouped.Params().Set( - name='lstm', - num_input_nodes=8, - num_output_nodes=8, - num_hidden_nodes=16, - num_groups=4, - num_shuffle_shards=1, - split_inputs=False) - child_p = params.child_lstm_tpl - child_p.output_nonlinearity = True - child_p.params_init = py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED) - child_p.vn.global_vn = False - child_p.vn.per_step_vn = False - - lstm = params.Instantiate() - - tf.logging.info('lstm vars = %s', lstm.vars) - for child_lstm in lstm.groups: - self.assertIn('wm', child_lstm.vars.wm.name) - self.assertIn('b', child_lstm.vars.b.name) - self.assertIn('w_proj', child_lstm.vars.w_proj.name) - - # 10 = 8 layer inputs + 2 recurrent - # 16 = 4 gates * 4 hidden units/group - self.assertEqual(child_lstm.theta.wm.get_shape(), tf.TensorShape([10, - 16])) - self.assertEqual(child_lstm.theta.b.get_shape(), tf.TensorShape([16])) - # Projection from 4 hidden units (16 total / 4 groups) to 2 outputs - # (8 total / 4 groups) - self.assertEqual(child_lstm.theta.w_proj.get_shape(), - tf.TensorShape([4, 2])) - - np.random.seed(_NUMPY_RANDOM_SEED) - inputs = py_utils.NestedMap( - act=[tf.constant(np.random.uniform(size=(3, 8)), tf.float32)], - padding=tf.zeros([3, 1])) - state0 = py_utils.NestedMap( - groups=py_utils.SplitRecursively( - py_utils.NestedMap( - c=tf.constant(np.random.uniform(size=(3, 16)), tf.float32), - m=tf.constant(np.random.uniform( - size=(3, 8)), tf.float32)), params.num_groups)) - - state1, _ = lstm.FPropDefaultTheta(state0, inputs) - self.assertLen(state1.groups, params.num_groups) - out1 = lstm.GetOutput(state1) - - with self.session( - use_gpu=False, config=py_utils.SessionConfig(inline=False)): - self.evaluate(tf.global_variables_initializer()) - - variable_count = 3 * params.num_groups # [wm, b, w_proj] for each group. - wts = tf.get_collection('LSTMCellSimple_vars') - self.assertLen(wts, variable_count) - - state1 = py_utils.ConcatRecursively(state1.groups) - m_actual = state1.m.eval() - c_actual = state1.c.eval() - out_actual = out1.eval() - - tf.logging.info('m_actual = %s', np.array_repr(m_actual)) - tf.logging.info('c_actual = %s', np.array_repr(c_actual)) - tf.logging.info('out_actual = %s', np.array_repr(out_actual)) - - # pyformat: disable - # pylint: disable=bad-whitespace,bad-continuation - m_expected = [ - [ - 0.61734521, 0.02338588, 0.19424279, 0.31576008, - 0.18000039, 0.1672723, 0.44075012, -0.06824636, - ], - [ - 0.44694018, -0.01717547, 0.49302083, -0.27330822, - 0.35382932, -0.1967615, 0.44225505, -0.04489155, - ], - [ - 0.66018867, 0.09434807, 0.643556, 0.0383133, - 0.74754262, -0.01860991, 0.48671043, 0.29460859, - ], - ] - c_expected = [ - [ - -0.52246463, 0.67389512, 0.58692968, 0.75484836, - -0.21763092, 0.45671225, -0.33593893, 1.03087521, - -0.15525842, 0.31072262, 0.14663902, 0.64976436, - -0.40176213, 0.36785093, 0.52653724, 0.73124039, - ], - [ - -0.27722716, 0.90508962, 0.39852297, 0.01676523, - -0.7724061, 0.40351537, 0.20194794, 0.08798298, - -0.39136624, 0.26601788, 0.21635406, -0.05538163, - -0.36326468, 0.64099556, 0.25886536, -0.09711652, - ], - [ - -0.63169837, 0.99831283, 0.53726614, 0.77321815, - -0.67881596, 1.01512539, 0.38799196, 0.26393941, - -0.87696433, 1.29881907, 0.60203284, 0.42675141, - -0.24902672, 1.15422893, 0.70180357, 0.12213309, - ], - ] - out_expected = [ - [ - 0.61734521, 0.02338588, 0.19424279, 0.31576008, - 0.18000039, 0.1672723, 0.44075012, -0.06824636, - ], - [ - 0.44694018, -0.01717547, 0.49302083, -0.27330822, - 0.35382932, -0.1967615, 0.44225505, -0.04489155, - ], - [ - 0.66018867, 0.09434807, 0.643556, 0.0383133, - 0.74754262, -0.01860991, 0.48671043, 0.29460859, - ], - ] - # pylint: enable=bad-whitespace,bad-continuation - # pyformat: enable - self.assertAllClose(m_expected, m_actual) - self.assertAllClose(c_expected, c_actual) - self.assertAllClose(out_expected, out_actual) - - # pyformat: disable - @parameterized.named_parameters( - ('_NoInline', False, False, - [0.4144063, 0.88831079, 0.56665027, 0.30154669, 0.2818037], - [4.72228432, 3.9454143, 3.77556086, 2.76972866, 1.87397099]), - ('_Inline', True, False, - [0.4144063, 0.88831079, 0.56665027, 0.30154669, 0.2818037], - [4.72228432, 3.9454143, 3.77556086, 2.76972866, 1.87397099]), - ('_GlobalNoise', True, True, - [0.21634784, 0.40635043, 0.12228709, 0.51806468, 0.02064975], - [5.21427298, 4.5560832, 4.24992609, 3.85193706, 2.35372424])) - # pyformat: enable - def testConvLSTM(self, inline, global_vn, m_expected, c_expected): - params = rnn_cell.ConvLSTMCell.Params().Set( - name='conv_lstm', - params_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - inputs_shape=[None, 4, 2, 3], - cell_shape=[None, 4, 2, 2], - filter_shape=[3, 2]) - params.vn.seed = 8820495 - params.vn.scale = 0.5 - params.vn.global_vn = global_vn - params.vn.per_step_vn = False - - lstm = rnn_cell.ConvLSTMCell(params) - lstm_vars = lstm.vars - tf.logging.info('lstm vars = %s', lstm_vars) - self.assertIn('wm', lstm_vars.wm.name) - self.assertIn('b', lstm_vars.b.name) - - w = lstm.theta.wm - b = lstm.theta.b - - self.assertEqual(w.get_shape(), tf.TensorShape([3, 2, 5, 8])) - self.assertEqual(b.get_shape(), tf.TensorShape([8])) - - np.random.seed(_NUMPY_RANDOM_SEED) - inputs = py_utils.NestedMap( - act=[tf.constant(np.random.uniform(size=(5, 4, 2, 3)), tf.float32)], - padding=tf.zeros([5, 1])) - state0 = py_utils.NestedMap( - c=tf.constant(np.random.uniform(size=(5, 4, 2, 2)), tf.float32), - m=tf.constant(np.random.uniform(size=(5, 4, 2, 2)), tf.float32)) - state1, _ = lstm.FPropDefaultTheta(state0, inputs) - m1 = tf.reduce_sum(state1.m, [1, 2, 3]) - c1 = tf.reduce_sum(state1.c, [1, 2, 3]) - - with self.session( - use_gpu=False, config=py_utils.SessionConfig(inline=inline)): - self.evaluate(tf.global_variables_initializer()) - - wts = tf.get_collection('ConvLSTMCell_vars') - self.assertLen(wts, 2) - - self.assertAllClose(m_expected, m1.eval()) - self.assertAllClose(c_expected, c1.eval()) - - @parameterized.named_parameters( - ('_Disabled', 0., False, [[0.2, 0.], [0., 2.4], [0.2, 3.4]], False), - ('_Enabled', 0.5, True, [[0.2, 0.], [0., 0.4], [0.2, 0.5]], False), - ('_Eval', 0.5, False, [[0.2, 0.], [0.05, 1.4], [0.15, 1.95]], True)) - def testZoneOut(self, zo_prob, enable_random_uniform, v_expected, is_eval): - params = rnn_cell.LSTMCellSimple.Params().Set( - name='lstm', - params_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - num_input_nodes=2, - num_output_nodes=2) - lstm = rnn_cell.LSTMCellSimple(params) - if enable_random_uniform: - random_uniform = tf.random.uniform([3, 2], seed=98798202) - else: - random_uniform = None - prev_v = [[0.2, 0.0], [0.1, 0.4], [0.1, 0.5]] - cur_v = [[0.3, 1.0], [0.0, 2.4], [0.2, 3.4]] - padding_v = [[1.0], [0.0], [0.0]] - new_v = lstm._ZoneOut( - prev_v, - cur_v, - padding_v, - zo_prob=zo_prob, - is_eval=is_eval, - random_uniform=random_uniform) - - with self.session(use_gpu=False): - # In eval mode, if padding[i] == 1, new_v equals prev_v. - # Otherwise, new_v = zo_prob * prev_v + (1.0 - zo_prob) * cur_v - new_v_evaled = new_v.eval() - tf.logging.info('new_v_evaled = %s', np.array_repr(new_v_evaled)) - self.assertAllClose(v_expected, new_v_evaled) - - # pyformat: disable - @parameterized.named_parameters( - ('_LSTMSimple', rnn_cell.LSTMCellSimple, True, - [[0.0083883, 0.10644437], [0.04662009, 0.18058866], - [0.0016561, 0.37414068]], - [[0.96451449, 0.65317708], [0.08686253, 0.34972212], - [0.00317609, 0.6554482]]), - ('_LSTMSimpleDeterministic', rnn_cell.LSTMCellSimpleDeterministic, False, - [[-0.145889, 0.], [-0.008282, 0.073219], [-0.041057, 0.]], - [[0., 0.532332], [-0.016117, 0.13752], [0., 0.]])) - # pyformat: enable - def testCellWithZoneOut(self, cell_cls, manual_state, m_expected, c_expected): - params = cell_cls.Params().Set( - name='lstm', - params_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - num_input_nodes=2, - num_output_nodes=2, - zo_prob=0.5, - random_seed=_RANDOM_SEED) - params.vn.global_vn = False - params.vn.per_step_vn = False - lstm = cell_cls(params) - - np.random.seed(_NUMPY_RANDOM_SEED) - inputs = py_utils.NestedMap( - act=[tf.constant(np.random.uniform(size=(3, 2)), tf.float32)], - padding=tf.zeros([3, 1])) - if manual_state: - state0 = py_utils.NestedMap( - c=tf.constant(np.random.uniform(size=(3, 2)), tf.float32), - m=tf.constant(np.random.uniform(size=(3, 2)), tf.float32)) - else: - state0 = lstm.zero_state(lstm.theta, 3) - state1, _ = lstm.FPropDefaultTheta(state0, inputs) - - with self.session(use_gpu=False): - self.evaluate(tf.global_variables_initializer()) - - m_v = state1.m.eval() - c_v = state1.c.eval() - - tf.logging.info('m_v = %s', np.array_repr(m_v)) - tf.logging.info('c_v = %s', np.array_repr(c_v)) - self.assertAllClose(m_expected, m_v) - self.assertAllClose(c_expected, c_v) - - # pyformat: disable - @parameterized.named_parameters( - ('LSTMCell', rnn_cell.LayerNormalizedLSTMCell, None, - [[0.03960676, 0.26547235], [-0.00677715, 0.09782403], - [-0.00272907, 0.31641623]], - [[0.14834785, 0.3804915], [-0.00927538, 0.38059634], - [-0.01014781, 0.46336061]]), - ('LSTMCellSimple', rnn_cell.LayerNormalizedLSTMCellSimple, None, - [[0.03960676, 0.26547235], [-0.00677715, 0.09782403], - [-0.00272907, 0.31641623]], - [[0.14834785, 0.3804915], [-0.00927538, 0.38059634], - [-0.01014781, 0.46336061]]), - ('NormLSTMCellSimple', rnn_cell.NormalizedLSTMCellSimple, None, - [[0.03960676, 0.26547235], [-0.00677715, 0.09782403], - [-0.00272907, 0.31641623]], - [[0.14834785, 0.3804915], [-0.00927538, 0.38059634], - [-0.01014781, 0.46336061]]), - ('LSTMCellLean', rnn_cell.LayerNormalizedLSTMCellLean, None, - [[-0.20482419, 0.55676991], [-0.55648255, 0.20511301], - [-0.20482422, 0.55676997]], - [[0.14834785, 0.3804915], [-0.00927544, 0.38059637], - [-0.01014781, 0.46336061]]), - ('LSTMCellProj', rnn_cell.LayerNormalizedLSTMCellSimple, 4, - [[0.39790073, 0.28511256], [0.41482946, 0.28972796], - [0.47132283, 0.03284446]], - [[-0.3667627, 1.03294277, 0.24229962, 0.43976486], - [-0.15832338, 1.22740746, 0.19910297, -0.14970526], - [-0.57552528, 0.9139322, 0.41805002, 0.58792269]]), - ('NormLSTMCellProj', rnn_cell.NormalizedLSTMCellSimple, 4, - [[0.39790073, 0.28511256], [0.41482946, 0.28972796], - [0.47132283, 0.03284446]], - [[-0.3667627, 1.03294277, 0.24229962, 0.43976486], - [-0.15832338, 1.22740746, 0.19910297, -0.14970526], - [-0.57552528, 0.9139322, 0.41805002, 0.58792269]]), - ('LSTMCellLeanProj', rnn_cell.LayerNormalizedLSTMCellLean, 4, - [[0.51581347, 0.22646663], [0.56025136, 0.16842051], - [0.58704823, -0.07126484]], - [[-0.36676273, 1.03294277, 0.24229959, 0.43976486], - [-0.15832338, 1.22740746, 0.19910295, -0.14970522], - [-0.57552516, 0.9139322, 0.41805002, 0.58792269]])) - # pyformat: enable - def testLN(self, cell_cls, num_hidden_nodes, m_expected, c_expected): - m_v, c_v = self._testLNLSTMCell(cell_cls.Params(), num_hidden_nodes) - self.assertAllClose(m_expected, m_v) - self.assertAllClose(c_expected, c_v) - - def testLNLSTMCellLeanNoLnOnC(self): - """LayerNormalizedLSTMCellLean without normalization on 'c'.""" - m_v, c_v = self._testLNLSTMCell( - rnn_cell.LayerNormalizedLSTMCellLean.Params().Set(enable_ln_on_c=False)) - m_expected = [[0.039607, 0.265472], [-0.006777, 0.097824], - [-0.002729, 0.316416]] - c_expected = [[0.14834785, 0.3804915], [-0.00927544, 0.38059637], - [-0.01014781, 0.46336061]] - self.assertAllClose(m_expected, m_v) - self.assertAllClose(c_expected, c_v) - - @parameterized.named_parameters(('Enable', True), ('Disable', False)) - def testLNLSTMCellLeanLSTMBias(self, enable): - m_expected, c_expected = self._testLNLSTMCell( - rnn_cell.LayerNormalizedLSTMCellSimple.Params().Set( - cell_value_cap=None, - enable_lstm_bias=enable, - bias_init=py_utils.WeightInit.Constant(1.0))) - m, c = self._testLNLSTMCell( - rnn_cell.LayerNormalizedLSTMCellLean.Params().Set( - enable_ln_on_c=False, - enable_lstm_bias=enable, - bias_init=py_utils.WeightInit.Constant(1.0))) - self.assertAllClose(m_expected, m) - self.assertAllClose(c_expected, c) - - @parameterized.named_parameters(('HighThreshold', 0.5), - ('LowThreshold', 5e-4)) - def testLNLSTMCellLeanCellValueCap(self, cell_value_cap): - m_expected, c_expected = self._testLNLSTMCell( - rnn_cell.LayerNormalizedLSTMCellSimple.Params().Set( - enable_lstm_bias=False, cell_value_cap=cell_value_cap)) - m, c = self._testLNLSTMCell( - rnn_cell.LayerNormalizedLSTMCellLean.Params().Set( - enable_ln_on_c=False, - enable_lstm_bias=False, - cell_value_cap=cell_value_cap)) - self.assertAllClose(m_expected, m) - self.assertAllClose(c_expected, c) - - def testLNLSTMCellLeanFeatureParity(self): - """Tests feature parity with LayerNormalizedLSTMCellSimple ... - - under the same configuration. - """ - m_expected, c_expected, grads_expected = self._testLNLSTMCellFPropBProp( - rnn_cell.LayerNormalizedLSTMCellSimple.Params().Set( - enable_lstm_bias=True, cell_value_cap=5e-4)) - m, c, grads = self._testLNLSTMCellFPropBProp( - rnn_cell.LayerNormalizedLSTMCellLean.Params().Set( - enable_lstm_bias=True, - cell_value_cap=5e-4, - enable_ln_on_c=False, - use_ln_bias=False)) - self.assertAllClose(m_expected, m) - self.assertAllClose(c_expected, c) - tf.logging.info('grads_expected: %r', grads_expected) - tf.logging.info('grads_actual: %r', grads) - self.assertAllClose(grads_expected.wm, grads.wm) - self.assertAllClose(grads_expected.b, grads.b) - self.assertAllClose( - grads_expected.ln_scale, - np.concatenate([ - grads.ln_scale_i_i, grads.ln_scale_i_g, grads.ln_scale_f_g, - grads.ln_scale_o_g - ])) - - def _testLNLSTMCellHelper(self, params, num_hidden_nodes): - params = params.Copy().Set( - name='lstm', - params_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - num_input_nodes=2, - num_output_nodes=2, - random_seed=_RANDOM_SEED) - if num_hidden_nodes is not None: - params.num_hidden_nodes = num_hidden_nodes - params.vn.global_vn = False - params.vn.per_step_vn = False - lstm = params.Instantiate() - np.random.seed(_NUMPY_RANDOM_SEED) - inputs = py_utils.NestedMap( - act=[tf.constant(np.random.uniform(size=(3, 2)), tf.float32)], - padding=tf.zeros([3, 1])) - state0 = py_utils.NestedMap( - c=tf.constant( - np.random.uniform(size=(3, lstm.hidden_size)), tf.float32), - m=tf.constant( - np.random.uniform(size=(3, lstm.output_size)), tf.float32)) - state1, _ = lstm.FPropDefaultTheta(state0, inputs) - return lstm, state0, state1 - - def _testLNLSTMCell(self, params, num_hidden_nodes=None): - tf.reset_default_graph() - _, _, state1 = self._testLNLSTMCellHelper(params, num_hidden_nodes) - with self.session(use_gpu=False): - self.evaluate(tf.global_variables_initializer()) - m_v = state1.m.eval() - c_v = state1.c.eval() - tf.logging.info('m_v = %s', np.array_repr(m_v)) - tf.logging.info('c_v = %s', np.array_repr(c_v)) - return m_v, c_v - - def _testLNLSTMCellFPropBProp(self, params, num_hidden_nodes=None): - tf.reset_default_graph() - lstm, _, state1 = self._testLNLSTMCellHelper(params, num_hidden_nodes) - loss = -tf.math.log( - tf.sigmoid( - tf.reduce_sum(tf.square(state1.m)) + - tf.reduce_sum(state1.m * state1.c * state1.c))) - grads = tf.gradients(loss, lstm.vars.Flatten()) - - with self.session(use_gpu=False): - self.evaluate(tf.global_variables_initializer()) - m_v, c_v, grads_v = self.evaluate([state1.m, state1.c, grads]) - - tf.logging.info('m_v = %s', np.array_repr(m_v)) - tf.logging.info('c_v = %s', np.array_repr(c_v)) - grads_val = py_utils.NestedMap() - for (n, _), val in zip(lstm.vars.FlattenItems(), grads_v): - tf.logging.info('%s : %s', n, np.array_repr(val)) - grads_val[n] = val - return m_v, c_v, grads_val - - # pyformat: disable - @parameterized.named_parameters( - ('_NoLnOnC', False, [[-0.606178], [0.599713], [0.657852]], - [[1.261887, -0.029158], [-0.00341, 1.034558], [-0.003731, 1.259534]]), - ('_LnOnC', True, [[-0.751002], [0.784634], [0.784634]], - [[1.261887, -0.029158], [-0.00341, 1.034558], [-0.003731, 1.259534]])) - # pyformat: enable - def testDoubleProjectionLSTMCell(self, enable_ln_on_c, m_expected, - c_expected): - params = rnn_cell.DoubleProjectionLSTMCell.Params().Set( - name='lstm', - params_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - enable_ln_on_c=enable_ln_on_c, - num_input_nodes=2, - num_output_nodes=1, - num_hidden_nodes=2, - num_input_hidden_nodes=2) - params.vn.global_vn = False - params.vn.per_step_vn = False - - lstm = params.Instantiate() - tf.logging.info('lstm vars = %s', lstm.vars) - - # Input projection. - self.assertEqual( - lstm.theta.w_input_proj.get_shape(), - tf.TensorShape([ - params.num_input_nodes + params.num_output_nodes, - params.num_input_hidden_nodes, - ])) - # W, LN, bias for the gates. - for gate in ['i_i', 'i_g', 'f_g', 'o_g']: - self.assertEqual( - lstm.theta.get('wm_%s' % gate).get_shape(), - tf.TensorShape( - [params.num_input_hidden_nodes, params.num_hidden_nodes])) - self.assertEqual( - lstm.theta.get('ln_scale_%s' % gate).get_shape(), - tf.TensorShape([params.num_hidden_nodes])) - self.assertEqual( - lstm.theta.get('bias_%s' % gate).get_shape(), - tf.TensorShape([params.num_hidden_nodes])) - # LN and bias for 'c'. - if enable_ln_on_c: - self.assertEqual(lstm.theta.ln_scale_c.get_shape(), - tf.TensorShape([params.num_hidden_nodes])) - self.assertEqual(lstm.theta.bias_c.get_shape(), - tf.TensorShape([params.num_hidden_nodes])) - else: - self.assertNotIn('ln_scale_c', lstm.theta) - self.assertNotIn('bias_c', lstm.theta) - # Output projection. - self.assertEqual( - lstm.theta.w_output_proj.get_shape(), - tf.TensorShape([params.num_hidden_nodes, params.num_output_nodes])) - - np.random.seed(_NUMPY_RANDOM_SEED) - inputs = py_utils.NestedMap( - act=[ - tf.constant( - np.random.uniform(size=(3, params.num_input_nodes)), - tf.float32), - ], - padding=tf.zeros([3, 1])) - state0 = py_utils.NestedMap( - c=tf.constant( - np.random.uniform(size=(3, params.num_hidden_nodes)), tf.float32), - m=tf.constant( - np.random.uniform(size=(3, params.num_output_nodes)), tf.float32)) - - state1, _ = lstm.FPropDefaultTheta(state0, inputs) - - with self.session( - use_gpu=False, config=py_utils.SessionConfig(inline=False)): - self.evaluate(tf.global_variables_initializer()) - - wts = tf.get_collection('DoubleProjectionLSTMCell_vars') - if enable_ln_on_c: - self.assertLen(wts, 2 + 3 * 4 + 2) - else: - self.assertLen(wts, 2 + 3 * 4) - - self.assertAllClose(m_expected, state1.m.eval()) - self.assertAllClose(c_expected, state1.c.eval()) - - # pyformat: disable - @parameterized.named_parameters( - ('', tf.zeros, - lambda: tf.constant(np.random.uniform(size=(3, 2)), tf.float32), - [[0.097589, 0.579055], [0.046737, 0.187892], [0.001656, 0.426245]], - [[0.241993, 0.820267], [0.086863, 0.349722], [0.003176, 0.655448]]), - ('_Padding', tf.ones, lambda: tf.zeros([3, 2], tf.float32), - [[0., 0.], [0., 0.], [0., 0.]], [[0., 0.], [0., 0.], [0., 0.]])) - # pyformat: enable - def testQuantizedLSTMCell(self, padding_fn, state0_fn, m_expected, - c_expected): - params = rnn_cell.QuantizedLSTMCell.Params().Set( - name='lstm', - params_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - num_input_nodes=2, - num_output_nodes=2, - cc_schedule=quant_utils.LinearClippingCapSchedule.Params().Set( - start_step=0, end_step=2, start_cap=5.0, end_cap=1.0)) - params.vn.global_vn = False - params.vn.per_step_vn = False - - lstm = rnn_cell.QuantizedLSTMCell(params) - lstm_vars = lstm.vars - tf.logging.info('lstm vars = %s', lstm_vars) - self.assertIn('wm', lstm_vars.wm.name) - - wm = lstm.theta.wm - self.assertEqual(wm.get_shape(), tf.TensorShape([4, 8])) - - np.random.seed(_NUMPY_RANDOM_SEED) - inputs = py_utils.NestedMap( - act=[tf.constant(np.random.uniform(size=(3, 2)), tf.float32)], - padding=padding_fn([3, 1])) - state0 = py_utils.NestedMap(c=state0_fn(), m=state0_fn()) - state1, _ = lstm.FPropDefaultTheta(state0, inputs) - - with self.session(use_gpu=False): - self.evaluate(tf.global_variables_initializer()) - - self.assertAllClose(m_expected, state1.m.eval()) - self.assertAllClose(c_expected, state1.c.eval()) - self.assertEqual(5.0, - lstm.cc_schedule.GetState(lstm.theta.cc_schedule).eval()) - self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), 1)) - self.assertEqual(3.0, - lstm.cc_schedule.GetState(lstm.theta.cc_schedule).eval()) - - def testQuantizedLayerNormalizedLSTMCell(self): - params = rnn_cell.LayerNormalizedLSTMCell.Params().Set( - name='lstm', - params_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - num_input_nodes=2, - num_output_nodes=2, - random_seed=_RANDOM_SEED, - cc_schedule=quant_utils.LinearClippingCapSchedule.Params().Set( - start_step=0, end_step=2, start_cap=5.0, end_cap=1.0)) - params.vn.global_vn = False - params.vn.per_step_vn = False - - lstm = rnn_cell.LayerNormalizedLSTMCell(params) - lstm_vars = lstm.vars - tf.logging.info('lstm vars = %s', lstm_vars) - self.assertIn('wm', lstm_vars.wm.name) - - wm = lstm.theta.wm - self.assertEqual(wm.get_shape(), tf.TensorShape([4, 8])) - - np.random.seed(_NUMPY_RANDOM_SEED) - inputs = py_utils.NestedMap( - act=[tf.constant(np.random.uniform(size=(3, 2)), tf.float32)], - padding=tf.zeros([3, 1])) - state0 = py_utils.NestedMap( - c=tf.constant(np.random.uniform(size=(3, 2)), tf.float32), - m=tf.constant(np.random.uniform(size=(3, 2)), tf.float32)) - state1, _ = lstm.FPropDefaultTheta(state0, inputs) - - with self.session(use_gpu=False): - self.evaluate(tf.global_variables_initializer()) - m_expected = [[0.03960676, 0.26547235], [-0.00677715, 0.09782403], - [-0.00272907, 0.31641623]] - c_expected = [[0.14834785, 0.3804915], [-0.00927538, 0.38059634], - [-0.01014781, 0.46336061]] - self.assertAllClose(m_expected, state1.m.eval()) - self.assertAllClose(c_expected, state1.c.eval()) - - self.assertEqual(5.0, - lstm.cc_schedule.GetState(lstm.theta.cc_schedule).eval()) - self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), 1)) - self.assertEqual(3.0, - lstm.cc_schedule.GetState(lstm.theta.cc_schedule).eval()) - - # pyformat: disable - @parameterized.named_parameters( - ('_TrainingUnclipped', False, False, 0, None, - [[0.097589, 0.579055], [0.046737, 0.187892], [0.001656, 0.426245]], - [[0.241993, 0.820267], [0.086863, 0.349722], [0.003176, 0.655448]]), - ('_Training', False, True, 0, [[0.0], [0.0], [1.0]], - [[0.09375, 0.5625], [0.046875, 0.1875], [0.809813, 0.872176]], - [[0.23288, 0.806], [0.090057, 0.355591], [0.747715, 0.961307]]), - ('_HiddenNodes', False, True, 4, None, - [[0.382812, 0.296875], [0.164062, 0.171875], [0.3125, -0.039062]], - [[-0.160339, 0.795929, 0.449707, 0.347534], - [-0.049194, 0.548279, -0.060852, -0.106354], - [-0.464172, 0.345947, 0.407349, 0.430878]]), - ('_Inference', True, False, 0, None, - [[0.09375, 0.5625], [0.046875, 0.1875], [0., 0.429688]], - [[0.23288, 0.806], [0.090057, 0.355591], [-0.003937, 0.662567]])) - # pyformat: enable - def testLSTMCellSimpleQuantized(self, is_inference, set_training_step, - num_hidden_nodes, padding, m_expected, - c_expected): - params = rnn_cell.LSTMCellSimple.Params().Set( - name='lstm', - is_inference=is_inference, - params_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - num_input_nodes=2, - num_output_nodes=2, - num_hidden_nodes=num_hidden_nodes, - output_nonlinearity=False, - cell_value_cap=None, - enable_lstm_bias=False) - params.vn.global_vn = False - params.vn.per_step_vn = False - - cc_schedule = quant_utils.FakeQuantizationSchedule.Params().Set( - clip_start_step=1, # Step 0 is unclipped. - clip_end_step=2, - quant_start_step=2, - start_cap=5.0, - end_cap=1.0) - # Default quantization. - qdomain = quant_utils.SymmetricScheduledClipQDomain.Params().Set( - cc_schedule=cc_schedule) - params.qdomain.default = qdomain - # M state uses the default 8-bit quantziation. - cc_schedule = cc_schedule.Copy() - qdomain = quant_utils.SymmetricScheduledClipQDomain.Params().Set( - cc_schedule=cc_schedule) - params.qdomain.m_state = qdomain - # C state uses 16 bit quantization. - cc_schedule = cc_schedule.Copy().Set(bits=16) - qdomain = quant_utils.SymmetricScheduledClipQDomain.Params().Set( - cc_schedule=cc_schedule) - params.qdomain.c_state = qdomain - # Fully connected layer clips slightly differently. - cc_schedule = cc_schedule.Copy().Set(start_cap=64.0, end_cap=8.0) - qdomain = quant_utils.SymmetricScheduledClipQDomain.Params().Set( - cc_schedule=cc_schedule) - params.qdomain.fullyconnected = qdomain - - lstm = rnn_cell.LSTMCellSimple(params) - lstm_vars = lstm.vars - tf.logging.info('lstm vars = %s', lstm_vars) - self.assertIn('wm', lstm_vars.wm.name) - if num_hidden_nodes: - self.assertIn('w_proj', lstm_vars.w_proj.name) - else: - self.assertNotIn('w_proj', lstm_vars) - - np.random.seed(_NUMPY_RANDOM_SEED) - if padding is None: - padding = tf.zeros([3, 1]) - else: - padding = tf.constant(padding, dtype=tf.float32) - inputs = py_utils.NestedMap( - act=[tf.constant(np.random.uniform(size=(3, 2)), tf.float32)], - padding=padding) - state0 = py_utils.NestedMap( - c=tf.constant( - np.random.uniform(size=(3, lstm.hidden_size)), tf.float32), - m=tf.constant( - np.random.uniform(size=(3, lstm.output_size)), tf.float32)) - state1, _ = lstm.FPropDefaultTheta(state0, inputs) - - with self.session(use_gpu=False), self.SetEval(is_inference): - self.evaluate(tf.global_variables_initializer()) - if set_training_step: - # Get it into the fully clipped/quantized part of the schedule. - self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), 5)) - - # Outputs. - self.assertAllClose(m_expected, state1.m.eval()) - self.assertAllClose(c_expected, state1.c.eval()) - - # Cell reported zeros. - cell_zero_state = lstm.zero_state(lstm.theta, batch_size=3) - self.assertAllEqual(cell_zero_state.m.eval(), - tf.zeros_like(state0.m).eval()) - self.assertAllEqual(cell_zero_state.c.eval(), - tf.zeros_like(state0.c).eval()) - - # pyformat: disable - @parameterized.named_parameters( - ('_Regular', None, True, 0, False, False, False, False, - [[0.58657289, 0.70520258], [0.32375532, 0.29133356], - [0.58900255, 0.58398587]], - [[0.45297623, 0.88433027], [0.42112729, 0.47023624], - [0.50483131, 0.89583319]]), - ('_WithCellCap', 0.4, True, 0, False, False, False, False, - [[0.569023, 0.472208], [0.314541, 0.260022], [0.543097, 0.379774]], - [[0.4, 0.4], [0.4, 0.4], [0.4, 0.4]]), - ('_WithInputGates', None, False, 0, False, False, False, False, - [[0.18558595, 0.81267989], [0.30404502, 0.35851872], - [0.51972485, 0.77677751]], - [[-0.19633068, 0.78370988], [0.29389063, 0.39952573], - [0.1221304, 0.67298377]]), - ('_WithProjection', None, True, 3, False, False, False, False, - [[0.04926362, 0.54914111], [-0.0501487, 0.32742232], - [-0.19329719, 0.28332305]], - [[-0.101138, 0.8266117, 0.75368524], - [0.31730127, 0.58325875, 0.64149243], - [0.09471729, 0.48504758, 0.53909004]]), - ('_WithNonZeroBiasInit', None, True, 3, True, False, False, False, - [[-0.012749, 0.788564], [-0.215866, 0.891125], [-0.420648, 0.713777]], - [[-0.30021, 0.728136, 0.751739], [0.211836, 0.665222, 0.755629], - [0.112793, 0.281477, 0.44823]]), - ('_WithLayerNormalization', None, True, 0, False, True, False, False, - [[0.301535, 0.744214], [0.38422, -0.524064], [0.328908, 0.658833]], - [[-1., 1.], [0.999999, -0.999999], [-1., 1.]]), - ('_Masked', None, True, 0, False, False, True, False, - [[0.586573, 0.705203], [0.323755, 0.291334], [0.589002, 0.583986]], - [[0.452976, 0.88433], [0.421127, 0.470236], [0.504831, 0.895833]]), - ('_MaskedProjection', None, True, 3, False, False, True, True, - [[0.049264, 0.549141], [-0.050149, 0.327422], [-0.193297, 0.283323]], - [[-0.10113806, 0.8266117, 0.75368524], - [0.31730127, 0.58325875, 0.64149243], - [0.0947173, 0.48504758, 0.53909004]])) - # pyformat: enable - def testSRUCell(self, cell_value_cap, couple_input_forget_gates, - num_hidden_nodes, enable_bias_init, apply_layer_norm, - apply_pruning, apply_pruning_to_projection, m_expected, - c_expected): - params = rnn_cell.SRUCell.Params().Set( - name='sru', - couple_input_forget_gates=couple_input_forget_gates, - params_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - num_input_nodes=2, - num_hidden_nodes=num_hidden_nodes, - num_output_nodes=2, - random_seed=_RANDOM_SEED, - apply_layer_norm=apply_layer_norm, - apply_pruning=apply_pruning, - apply_pruning_to_projection=apply_pruning_to_projection) - if cell_value_cap is not None: - params.cell_value_cap = 0.4 # cell cap set to low level - if enable_bias_init: - params.bias_init = py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED) - sru = rnn_cell.SRUCell(params) - - self.assertIn('wm', sru.vars.wm.name) - self.assertIn('b', sru.vars.b.name) - if apply_pruning: - self.assertIn('mask', sru.vars.mask.name) - self.assertIn('threshold', sru.vars.threshold.name) - if params.apply_pruning_to_projection: - self.assertIn('w_proj', sru.vars.w_proj.name) - self.assertIn('proj_mask', sru.vars.proj_mask.name) - self.assertIn('proj_threshold', sru.vars.proj_threshold.name) - - num_gates = 4 if couple_input_forget_gates else 5 - if num_hidden_nodes > 0: - num_nodes = num_hidden_nodes - else: - num_nodes = params.num_output_nodes - num_param_vectors = num_gates * num_nodes - self.assertEqual(sru.theta.wm.get_shape(), - tf.TensorShape([2, num_param_vectors])) - self.assertEqual(sru.theta.b.get_shape(), - tf.TensorShape([num_param_vectors])) - if params.apply_pruning_to_projection: - self.assertEqual(sru.theta.w_proj.get_shape(), tf.TensorShape([3, 2])) - self.assertEqual(sru.theta.proj_mask.get_shape(), tf.TensorShape([3, 2])) - - np.random.seed(_NUMPY_RANDOM_SEED) - inputs = py_utils.NestedMap( - act=[tf.constant(np.random.uniform(size=(3, 2)), tf.float32)], - padding=tf.zeros([3, 1])) - if params.num_hidden_nodes > 0: - c_dim = params.num_hidden_nodes - else: - c_dim = params.num_output_nodes - state0 = py_utils.NestedMap( - c=tf.constant(np.random.uniform(size=(3, c_dim)), tf.float32), - m=tf.constant(np.random.uniform(size=(3, 2)), tf.float32)) - state1, _ = sru.FPropDefaultTheta(state0, inputs) - - with self.session(use_gpu=False): - self.evaluate(tf.global_variables_initializer()) - m_v = state1.m.eval() - c_v = state1.c.eval() - - tf.logging.info('m_v = %s', np.array_repr(m_v)) - tf.logging.info('c_v = %s', np.array_repr(c_v)) - self.assertAllClose(m_expected, m_v) - self.assertAllClose(c_expected, c_v) - - # pyformat: disable - @parameterized.named_parameters( - ('', 'quasi_ifo', - [[0.40564698, 0.32611847], [0.16975531, 0.45970476], - [0.60286027, 0.24542703]], - [[0.42057115, 0.49928033], [0.56830668, 0.7003305], - [1.28926766, 0.75380397]]), - ('_InSRUMode', 'sru', - [[0.55884922, 0.40396619], [0.71426249, 0.70820922], - [0.823457, 0.60304904]], - [[0.65144706, 0.56252223], [0.75096267, 0.65605044], - [0.79761195, 0.36905324]])) - # pyformat: enable - def testQRNNPoolingCell(self, pooling_formula, m_expected, c_expected): - params = rnn_cell.QRNNPoolingCell.Params().Set( - name='QuasiRNN', - params_init=py_utils.WeightInit.Uniform(1.24, _INIT_RANDOM_SEED), - num_input_nodes=2, - num_output_nodes=2, - zo_prob=0.0, - random_seed=_RANDOM_SEED, - pooling_formula=pooling_formula) - qrnn = rnn_cell.QRNNPoolingCell(params) - - np.random.seed(_NUMPY_RANDOM_SEED) - num_rnn_matrices = 4 - inputs = py_utils.NestedMap( - act=[ - tf.constant( - np.random.uniform( - size=(3, params.num_input_nodes * num_rnn_matrices)), - tf.float32) - ], - padding=tf.zeros([3, 1])) - state0 = py_utils.NestedMap( - c=tf.constant( - np.random.uniform(size=(3, params.num_output_nodes)), tf.float32), - m=tf.constant( - np.random.uniform(size=(3, params.num_output_nodes)), tf.float32)) - state1, _ = qrnn.FPropDefaultTheta(state0, inputs) - - with self.session(use_gpu=False): - self.evaluate(tf.global_variables_initializer()) - m_v = state1.m.eval() - c_v = state1.c.eval() - - tf.logging.info('m_v = %s', np.array_repr(m_v)) - tf.logging.info('c_v = %s', np.array_repr(c_v)) - self.assertAllClose(m_expected, m_v) - self.assertAllClose(c_expected, c_v) - - # pyformat: disable - @parameterized.named_parameters( - ('_FnZeros', False, False, [[0.0, 0.0, 0.0]], [[0.0, 0.0, 0.0]]), - ('_FnRandomNormal', True, False, [[-0.630551, -1.208959, -0.348799]], - [[-0.630551, -1.208959, -0.348799]]), - ('_FnRandomNormalInEval', True, True, [[0.0, 0.0, 0.0]], - [[0.0, 0.0, 0.0]])) - # pyformat: enable - def testLSTMZeroState(self, random_zero_state, is_eval, m_expected, - c_expected): - if random_zero_state: - zero_state_init = py_utils.RNNCellStateInit.RandomNormal(seed=12345) - else: - zero_state_init = py_utils.RNNCellStateInit.Zeros() - params = rnn_cell.LSTMCellSimple.Params().Set( - name='lstm', - params_init=py_utils.WeightInit.Constant(0.1), - num_input_nodes=2, - num_output_nodes=3, - forget_gate_bias=2.0, - bias_init=py_utils.WeightInit.Constant(0.1), - dtype=tf.float64, - zero_state_init_params=zero_state_init) - lstm = rnn_cell.LSTMCellSimple(params) - - with self.session(use_gpu=False), self.SetEval(is_eval): - self.evaluate(tf.global_variables_initializer()) - init_state_value = self.evaluate(lstm.zero_state(lstm.theta, 1)) - - tf.logging.info('testLSTMSimpleWithStateInitializationFn m = %s', - np.array_repr(init_state_value['m'])) - tf.logging.info('testLSTMSimpleWithStateInitializationFn c = %s', - np.array_repr(init_state_value['c'])) - self.assertAllClose(init_state_value['m'], m_expected) - self.assertAllClose(init_state_value['c'], c_expected) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/rnn_layers.py b/lingvo/core/rnn_layers.py deleted file mode 100644 index d1e18ab7..00000000 --- a/lingvo/core/rnn_layers.py +++ /dev/null @@ -1,1436 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Lingvo RNN layers.""" - -import math -import lingvo.compat as tf -from lingvo.core import attention -from lingvo.core import base_layer -from lingvo.core import layers -from lingvo.core import py_utils -from lingvo.core import quant_utils -from lingvo.core import recurrent -from lingvo.core import rnn_cell - - -def GeneratePackedInputResetMask(segment_id, is_reverse=False): - """Generates mask inputs for RNN cells from segment_id. - - Args: - segment_id: A tensor of shape [time, batch_size, 1]. - is_reverse: True if inputs are fed to the RNN in reverse order. - - Returns: - reset_mask - a tensor of shape [time, batch_size, 1]. Set to 0 for samples - where state needs to be reset (at example boundaries), and 1 otherwise. - """ - segment_id_left = segment_id[:-1] - segment_id_right = segment_id[1:] - - # Mask is a [t-1, bs, 1] tensor. - reset_mask = tf.cast( - tf.equal(segment_id_left, segment_id_right), dtype=segment_id.dtype) - mask_padding_shape = tf.concat( - [tf.ones([1], dtype=tf.int32), - tf.shape(segment_id)[1:]], axis=0) - mask_padding = tf.ones(mask_padding_shape, dtype=segment_id.dtype) - if is_reverse: - reset_mask = tf.concat([reset_mask, mask_padding], axis=0) - else: - reset_mask = tf.concat([mask_padding, reset_mask], axis=0) - return reset_mask - - -class IdentitySeqLayer(base_layer.BaseLayer): - """A no-op sequence layer.""" - - def __init__(self, params): - super().__init__(params) - - def zero_state(self, theta, batch_size): - return py_utils.NestedMap() - - def FPropFullSequence(self, theta, inputs, paddings): - return inputs - - -class RNN(base_layer.BaseLayer): - """Statically unrolled RNN.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('cell', rnn_cell.LSTMCellSimple.Params(), - 'Configs for the RNN cell.') - p.Define( - 'sequence_length', 0, - 'Sequence length to unroll. If > 0, then will unroll to this fixed ' - 'size. If 0, then will unroll to accommodate the size of the inputs ' - 'for each call to FProp.') - p.Define('reverse', False, - 'Whether or not to unroll the sequence in reversed order.') - p.Define('packed_input', False, 'To reset states for packed inputs.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert not p.packed_input, ('Packed inputs are currently not supported by ' - 'Static RNN') - p.cell.reset_cell_state = p.packed_input - assert p.sequence_length >= 0 - self.CreateChild('cell', p.cell) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - self.cell.InstantiateVariables() - super()._CreateChildrenVariables() - - def zero_state(self, theta, batch_size): - return self.cell.zero_state(theta.cell, batch_size) - - def FProp(self, theta, inputs, paddings, state0=None): - """Compute RNN forward pass. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - inputs: A single tensor or a tuple of tensors with cardinality equal to - rnn_cell.inputs_arity. For every input tensor, the first dimension is - assumed to be time, second dimension batch, and third dimension depth. - paddings: A tensor. First dim is time, second dim is batch, and third dim - is expected to be 1. - state0: If not None, the initial rnn state in a `.NestedMap`. Defaults - to the cell's zero-state. - - Returns: - A tensor of [time, batch, dims]. - The final recurrent state. - """ - p = self.params - rcell = self.cell - assert isinstance(rcell, (rnn_cell.RNNCell)) - if p.sequence_length == 0: - if isinstance(inputs, (tuple, list)): - sequence_length = len(inputs) - else: - sequence_length = py_utils.GetShape(inputs)[0] - else: - sequence_length = p.sequence_length - assert sequence_length >= 1, ('Sequence length must be defined or inputs ' - 'must have fixed shapes.') - with tf.name_scope(p.name): - inputs_sequence = tf.unstack(inputs, num=sequence_length) - paddings_sequence = tf.unstack(paddings, num=sequence_length) - # We start from all 0 states. - if state0: - state = state0 - else: - inputs0 = py_utils.NestedMap( - act=[inputs_sequence[0]], padding=paddings_sequence[0]) - state = rcell.zero_state(theta.cell, rcell.batch_size(inputs0)) - outputs = [None] * sequence_length - if p.reverse: - sequence = range(sequence_length - 1, -1, -1) - else: - sequence = range(0, sequence_length, 1) - for idx in sequence: - cur_input = py_utils.NestedMap(act=[inputs[idx]], padding=paddings[idx]) - state, _ = rcell.FProp(theta.cell, state, cur_input) - outputs[idx] = rcell.GetOutput(state) - return tf.stack(outputs), state - - -class StackedRNNBase(base_layer.BaseLayer): - """Stacked RNN base class.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_layers', 1, 'The number of RNN layers.') - p.Define( - 'num_input_nodes', -1, - 'If >= 0, overrides cell_tpl.num_input_nodes for the first layer.') - p.Define( - 'num_output_nodes', -1, - 'If >= 0, overrides cell_tpl.num_output_nodes for the last layer.') - p.Define('skip_start', 1, 'The first layer start skip connection.') - p.Define( - 'cell_tpl', rnn_cell.LSTMCellSimple.Params(), - 'Configs for the RNN cell(s). ' - 'If cell_tpl is not a list/tuple, the same cell config is used ' - 'for all layers. Otherwise, cell_tpl[i] is the config for ' - 'i-th layer and cell_tpl[-1] is used for the rest of layers.') - p.Define('dropout', layers.DropoutLayer.Params(), - 'Dropout applied to each layer.') - p.Define('packed_input', False, 'To reset states for packed inputs.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert not p.packed_input, ('Packed inputs are currently not supported by ' - 'Static RNN Base') - - def _GetCellTpls(self): - p = self.params - if not isinstance(p.cell_tpl, (tuple, list)): - cell_tpls = [p.cell_tpl] * p.num_layers - else: - cell_tpls = list(p.cell_tpl) - assert len(p.cell_tpl) <= p.num_layers - last = cell_tpls[-1] - while len(cell_tpls) < p.num_layers: - cell_tpls.append(last) - for cell_tpl in cell_tpls: - cell_tpl.reset_cell_state = p.packed_input - return cell_tpls - - -class StackedFRNNLayerByLayer(StackedRNNBase, quant_utils.QuantizableLayer): - """An implemention of StackedRNNBase which computes layer-by-layer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('rnn_tpl', FRNN.Params(), 'Rnn cell default params.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - rnn_params = [] - with tf.name_scope(p.name): - for (i, cell_tpl) in enumerate(self._GetCellTpls()): - params = p.rnn_tpl.Copy() - params.packed_input = p.packed_input - params.allow_implicit_capture = p.allow_implicit_capture - params.name = 'frnn_%d' % i - params.cell = cell_tpl.Copy() - params.cell.name = '%s_%d' % (p.name, i) - if p.num_input_nodes > 0 and i == 0: - params.cell.num_input_nodes = p.num_input_nodes - if p.num_output_nodes > 0 and i == p.num_layers - 1: - params.cell.num_output_nodes = p.num_output_nodes - rnn_params.append(params) - - for i in range(len(rnn_params) - 1): - # Because one layer's output needs to be fed into the next layer's - # input, hence, we have this assertion. We can relax it later by - # allowing more parameterization of the layers. - assert (rnn_params[i].cell.num_output_nodes == rnn_params[i + 1].cell - .num_input_nodes) - - self.CreateChildren('rnn', rnn_params) - self.CreateChild('dropout', p.dropout) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - self.TrackQTensor('residual') - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - for rnn in self.rnn: - rnn.InstantiateVariables() - self.dropout.InstantiateVariables() - super()._CreateChildrenVariables() - - def zero_state(self, theta, batch_size): - p = self.params - ret = py_utils.NestedMap(rnn=[]) - for i in range(p.num_layers): - state0 = self.rnn[i].zero_state(theta.rnn[i], batch_size) - ret.rnn.append(state0) - return ret - - def FProp(self, theta, inputs, paddings, state0=None): - """Compute RNN forward pass. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - inputs: A single tensor of shape [time, batch, dims]. - paddings: A single tensor of shape [time, batch, 1]. - state0: If not None, the initial rnn state in a `.NestedMap`. Defaults - to the init state. - - Returns: - (outputs, state1) - outputs: A tensor of [time, batch, dims]. - state1: The final state. - """ - p = self.params - if not state0: - state0 = self.zero_state(theta, tf.shape(inputs)[1]) - xs = inputs - state1 = py_utils.NestedMap(rnn=[None] * p.num_layers) - for i in range(p.num_layers): - ys, state1.rnn[i] = self.rnn[i].FProp(theta.rnn[i], xs, paddings, - state0.rnn[i]) - ys = self.dropout.FProp(theta.dropout, ys) - if (p.skip_start >= 0 and i >= p.skip_start and - (p.num_input_nodes <= 0 or i != 0) and - (p.num_output_nodes <= 0 or i != p.num_layers - 1)): - ys = self.fns.qadd(ys, xs, qt='residual') - xs = ys - return xs, state1 - - def FPropFullSequence(self, theta, inputs, paddings): - return self.FProp(theta, inputs, paddings)[0] - - -class StackedBiFRNNLayerByLayer(StackedRNNBase, quant_utils.QuantizableLayer): - """An implemention of StackedRNNBase with bidirection RNN layers.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('frnn_tpl', BidirectionalFRNN.Params(), 'Rnn cell default params.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - rnn_params = [] - feature_dim = None - with tf.name_scope(p.name): - for (i, cell_tpl) in enumerate(self._GetCellTpls()): - rnn_p = cell_tpl.Copy() - if i > 0: - rnn_p.num_input_nodes = feature_dim - if p.num_input_nodes > 0 and i == 0: - rnn_p.num_input_nodes = p.num_input_nodes - if p.num_output_nodes > 0 and i == p.num_layers - 1: - rnn_p.num_output_nodes = p.num_output_nodes // 2 - frnn_param = p.frnn_tpl.Copy() - frnn_param.name = 'bidi_rnn_%d' % i - frnn_param.fwd = rnn_p.Copy().Set(name='f_rnn_%d' % i) - frnn_param.bak = rnn_p.Copy().Set(name='b_rnn_%d' % i) - rnn_params.append(frnn_param) - feature_dim = 2 * rnn_p.num_output_nodes - - self.CreateChildren('rnn', rnn_params) - self.CreateChild('dropout', p.dropout) - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - self.TrackQTensor('residual') - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - for rnn in self.rnn: - rnn.InstantiateVariables() - self.dropout.InstantiateVariables() - super()._CreateChildrenVariables() - - def FProp(self, theta, inputs, paddings): - """Compute the forward pass. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: A single tensor of shape [time, batch, dims]. - paddings: A single tensor of shape [time, batch, 1]. - - Returns: - A tensor of [time, batch, dims]. - """ - p = self.params - xs = inputs - for i in range(p.num_layers): - ys = self.rnn[i].FProp(theta.rnn[i], xs, paddings) - ys = self.dropout.FProp(theta.dropout, ys) - if (p.skip_start >= 0 and i >= p.skip_start and - (p.num_input_nodes <= 0 or i != 0) and - (p.num_output_nodes <= 0 or i != p.num_layers - 1)): - ys = self.fns.qadd(ys, xs, qt='residual') - xs = ys - return xs - - def FPropFullSequence(self, theta, inputs, paddings): - return self.FProp(theta, inputs, paddings) - - -class FRNN(base_layer.BaseLayer): - """Functional while based RNN.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('cell', rnn_cell.LSTMCellSimple.Params(), - 'Configs for the RNN cell.') - p.Define('reverse', False, - 'Whether or not to unroll the sequence in reversed order.') - p.Define('packed_input', False, 'To reset states for packed inputs.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - p.cell.reset_cell_state = p.packed_input - self.CreateChild('cell', p.cell) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - self.cell.InstantiateVariables() - super()._CreateChildrenVariables() - - @property - def rnn_cell(self): - return self.cell - - def zero_state(self, theta, batch_size): - return self.cell.zero_state(theta.cell, batch_size) - - def FProp(self, theta, inputs, paddings, state0=None, segment_id=None): - """Compute RNN forward pass. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - inputs: A single tensor or a tuple of tensors with cardinality equal to - rnn_cell.inputs_arity. For every input tensor, the first dimension is - assumed to be time, second dimension batch, and third dimension depth. - paddings: A tensor. First dim is time, second dim is batch, and third dim - is expected to be 1. - state0: If not None, the initial rnn state in a `.NestedMap`. Defaults - to the cell's zero-state. - segment_id: A tensor to support packed inputs. First dim is time, second - dim is batch, and third dim is expected to be 1. - - Returns: - A tensor of [time, batch, dims]. - The final recurrent state. - """ - p = self.params - rcell = self.cell - assert isinstance(rcell, (rnn_cell.RNNCell)) - - def FlipUpDown(x): - # Reverse the first dimension (time) - return tf.reverse(x, [0]) - - if not isinstance(inputs, (list, tuple)): - inputs = [inputs] - - if p.packed_input: - assert segment_id is not None - reset_mask = GeneratePackedInputResetMask( - segment_id, is_reverse=p.reverse) - reset_mask = py_utils.HasShape(reset_mask, tf.shape(paddings)) - else: - reset_mask = tf.zeros_like(paddings) - - if p.reverse: - inputs = [FlipUpDown(x) for x in inputs] - paddings = FlipUpDown(paddings) - reset_mask = FlipUpDown(reset_mask) - - if not state0: - inputs0 = py_utils.NestedMap( - act=[x[0] for x in inputs], - padding=paddings[0, :], - reset_mask=reset_mask[0, :]) - state0 = rcell.zero_state(theta.cell, rcell.batch_size(inputs0)) - - inputs = py_utils.NestedMap( - act=inputs, padding=paddings, reset_mask=reset_mask) - - acc_state, final_state = recurrent.Recurrent( - theta=theta.cell, - state0=state0, - inputs=inputs, - cell_fn=rcell.FProp, - cell_type=rcell.layer_type, - accumulator_layer=self, - allow_implicit_capture=p.allow_implicit_capture) - - act = rcell.GetOutput(acc_state) - if p.reverse: - act = FlipUpDown(act) - return act, final_state - - -class BidirectionalFRNN(base_layer.BaseLayer): - """Bidirectional functional RNN.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('fwd', rnn_cell.LSTMCellSimple.Params(), - 'Configs for the forward RNN cell.') - p.Define('bak', rnn_cell.LSTMCellSimple.Params(), - 'Configs for the backward RNN cell.') - p.Define('rnn', FRNN.Params(), 'Config for underlying RNNs') - p.Define('packed_input', False, 'To reset states for packed inputs.') - return p - - def __init__(self, params): - super().__init__(params) - p = params - - params_forward = p.rnn.Copy() - params_forward.name = 'fwd' - params_forward.dtype = p.dtype - params_forward.reverse = False - params_forward.packed_input = p.packed_input - params_forward.cell = p.fwd.Copy() - self.CreateChild('fwd_rnn', params_forward) - - params_backward = p.rnn.Copy() - params_backward.name = 'bak' - params_backward.dtype = p.dtype - params_backward.reverse = True - params_backward.packed_input = p.packed_input - params_backward.cell = p.bak.Copy() - self.CreateChild('bak_rnn', params_backward) - - def _CreateChildrenVariables(self): - if py_utils.use_tpu() and self.cluster.num_devices_per_split > 1: - fwd_device = self.cluster.WorkerDeviceInModelSplit(0) - bwd_device = self.cluster.WorkerDeviceInModelSplit(1) - else: - fwd_device = '' - bwd_device = '' - with tf.device(fwd_device): - self.fwd_rnn.InstantiateVariables() - with tf.device(bwd_device): - self.bak_rnn.InstantiateVariables() - super()._CreateChildrenVariables() - - def FProp(self, theta, inputs, paddings, segment_id=None): - """Compute bidi-RNN forward pass. - - `rcell_forward` unroll the sequence in the forward direction and - `rcell_backward` unroll the sequence in the backward direction. The - outputs are concatenated in the last output dim. - - See `FRNN.FProp` for more details. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - inputs: A single tensor or a tuple of tensors with cardinality equal to - rnn_cell.inputs_arity. For every input tensor, the first dimension is - assumed to be time, second dimension batch, and third dimension depth. - paddings: A tensor. First dim is time, second dim is batch, and third dim - is expected to be 1. - segment_id: A tensor to support packed inputs. First dim is time, second - dim is batch, and third dim is expected to be 1. - - Returns: - A tensor of [time, batch, dims]. - """ - p = self.params - with tf.name_scope(p.name): - - def Fwd(): - """Run the forward pass.""" - output_forward, _ = self.fwd_rnn.FProp( - theta.fwd_rnn, inputs, paddings, segment_id=segment_id) - return output_forward - - def Bwd(): - """Run the backward pass. - - Returns: - A tensor of [time, batch, dims]. The final recurrent state. - """ - output_backward, _ = self.bak_rnn.FProp( - theta.bak_rnn, inputs, paddings, segment_id=segment_id) - # TODO(yonghui/zhifengc): In the current implementation, we copy - # output_forward from gpu:0 to gpu:1, and then copy the concatenated - # output from gpu:1 to gpu:0 to enable next layer computation. It might - # be more efficient to only copy output_backward from gpu:1 to gpu:0 to - # reduce cross-gpu data transfer. - return output_backward - - # On TPU, we run both direction's RNNs on one device to reduce memory - # usage. - if self.cluster.num_devices_per_split > 1: - fwd_device = self.cluster.WorkerDeviceInModelSplit(0) - bwd_device = self.cluster.WorkerDeviceInModelSplit(1) - else: - fwd_device = '' - bwd_device = '' - with tf.device(fwd_device): - output_forward = Fwd() - with tf.device(bwd_device): - output_backward = Bwd() - with tf.device(fwd_device): - return tf.concat([output_forward, output_backward], -1) - - -class BidirectionalRNN(base_layer.BaseLayer): - """Statically unrolled bidirectional RNN.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('fwd', rnn_cell.LSTMCellSimple.Params(), - 'Configs for the forward RNN cell.') - p.Define('bak', rnn_cell.LSTMCellSimple.Params(), - 'Configs for the backward RNN cell.') - p.Define('sequence_length', 0, 'Sequence length.') - p.Define('rnn', RNN.Params(), 'Config for underlying RNNs') - p.Define('packed_input', False, 'To reset states for packed inputs.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert not p.packed_input, ('Packed input is currently not supported by ' - 'BiDirectionalRNN') - params_forward = p.rnn.Copy() - params_forward.name = '%s_forward' % p.name - params_forward.cell = p.fwd.Copy() - params_forward.sequence_length = p.sequence_length - self.CreateChild('fwd_rnn', params_forward) - params_backward = p.rnn.Copy() - params_backward.name = '%s_backward' % p.name - params_backward.cell = p.bak.Copy() - params_backward.sequence_length = p.sequence_length - params_backward.reverse = True - self.CreateChild('bak_rnn', params_backward) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - self.fwd_rnn.InstantiateVariables() - self.bak_rnn.InstantiateVariables() - super()._CreateChildrenVariables() - - def FProp(self, theta, inputs, paddings): - """Compute bidi-RNN forward pass. - - `rcell_forward` is responsible for unrolling the sequence in the forward - direction and `rcell_backward` in the backward direction. Output from - forward and backward rnns are concatenated on the last output dim. - - See `RNN.FProp()` for more details. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - inputs: A single tensor or a tuple of tensors with cardinality equal to - rnn_cell.inputs_arity. For every input tensor, the first dimension is - assumed to be time, second dimension batch, and third dimension depth. - paddings: A tensor. First dim is time, second dim is batch, and third dim - is expected to be 1. - - Returns: - A tensor of [time, batch, dims]. - """ - p = self.params - - with tf.name_scope(p.name): - outputs_forward, _ = self.fwd_rnn.FProp(theta.fwd_rnn, inputs, paddings) - outputs_backward, _ = self.bak_rnn.FProp(theta.bak_rnn, inputs, paddings) - return tf.concat([outputs_forward, outputs_backward], axis=-1) - - -class BidirectionalRNNV2(base_layer.BaseLayer): - """Statically unrolled bidirectional RNN.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('fwd', rnn_cell.LSTMCellSimple.Params(), - 'Configs for the forward RNN cell.') - p.Define('bak', rnn_cell.LSTMCellSimple.Params(), - 'Configs for the backward RNN cell.') - p.Define('sequence_length', 0, 'Sequence length.') - p.Define('packed_input', False, 'To reset states for packed inputs.') - return p - - def __init__(self, params): - super().__init__(params) - assert not self.params.packed_input, ('Packed input is currently not ' - 'supported by BiDirectionalRNNV2') - p = BidirectionalRNN.Params() - p.packed_input = self.params.packed_input - p.name = '%s_brnn' % self.params.name - p.fwd = self.params.fwd.Copy() - p.bak = self.params.bak.Copy() - p.sequence_length = self.params.sequence_length - self.CreateChild('brnn', p) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - self.brnn.InstantiateVariables() - super()._CreateChildrenVariables() - - def _PadSequenceToLength(self, t_input, length, pad_value): - t_input = py_utils.with_dependencies( - [py_utils.assert_less_equal(tf.shape(t_input)[0], length)], t_input) - pad_shape = tf.concat([[length - tf.shape(t_input)[0]], - tf.shape(t_input)[1:]], 0) - padding = tf.zeros(shape=pad_shape, dtype=t_input.dtype) + pad_value - return tf.concat([t_input, padding], 0) - - def FProp(self, theta, inputs, paddings): - """Compute bidi-RNN forward pass. - - rcell_forward is responsible for unrolling the sequence in the forward - direction and rcell_backward in the backward direction. Output from forward - and backward rnns are concatenated on the last output dim. - - See RNN.FProp() for more details. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - inputs: A single tensor or a tuple of tensors with cardinality equal to - rnn_cell.inputs_arity. For every input tensor, the first dimension is - assumed to be time, second dimension batch, and third dimension depth. - paddings: A tensor. First dim is time, second dim is batch, and third dim - is expected to be 1. - - Returns: - A tensor of [time, batch, dims]. - """ - p = self.params - if not isinstance(inputs, (list, tuple)): - inputs = [inputs] - seq_len = tf.shape(paddings)[0] - inputs = [ - self._PadSequenceToLength(x, p.sequence_length, 0) for x in inputs - ] - inputs_sequence = [ - list(tf.unstack(x, num=p.sequence_length)) for x in inputs - ] - if len(inputs_sequence) > 1: - inputs_sequence = list(zip(inputs_sequence)) - else: - inputs_sequence = inputs_sequence[0] - paddings_sequence = list( - tf.unstack( - self._PadSequenceToLength(paddings, p.sequence_length, 1.0), - num=p.sequence_length)) - assert len(inputs_sequence) == p.sequence_length - assert len(paddings_sequence) == p.sequence_length - out = self.brnn.FProp(theta.brnn, inputs_sequence, paddings_sequence) - return tf.stack(out)[:seq_len,] - - -def _ConcatLastDim(*args): - """Concatenates all args along the last dimension.""" - return tf.concat(args, tf.rank(args[0]) - 1) - - -def _ShiftRight(x0, xs): - """Shifts xs[:-1] one step to the right and attaches x0 on the left.""" - return tf.concat([[x0], xs[:-1]], axis=0) - - -def _ShiftRightWithMasking(x0, xs, mask): - """Shifts xs[:-1] one step to the right and attaches x0 on the left.""" - return tf.concat([[x0], xs[:-1] * mask[1:]], axis=0) - - -class FRNNWithAttention(base_layer.BaseLayer): - """An RNN layer intertwined with an attention layer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('cell', rnn_cell.LSTMCellSimple.Params(), - 'Configs for the RNN cell.') - p.Define('attention', attention.AdditiveAttention.Params(), - 'Attention used by this layer.') - p.Define( - 'output_prev_atten_ctx', False, - 'If True, output previous attention context for each position.' - 'Otherwise, output current attention context.') - p.Define( - 'input_prev_atten_ctx', True, - 'If True, concat previous attention context and user input as input.' - 'Otherwise, this layer will still produce attention context but will ' - 'not use previous context as input.') - p.Define( - 'use_zero_atten_state', False, - 'To use zero attention state instead of computing attention with ' - 'zero query vector.') - p.Define('atten_context_dim', 0, 'Size of attention context.') - p.Define('packed_input', False, 'To reset states for packed inputs.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.use_zero_atten_state: - assert p.atten_context_dim > 0, ( - 'atten_context_dim needs to be set when ' - 'initializing attention state and context with 0.') - if p.packed_input: - assert p.use_zero_atten_state, ( - 'Packed input is only supported when ' - 'training with zero initial attention states.') - p.cell.reset_cell_state = p.packed_input - self.CreateChild('cell', p.cell) - p.attention.packed_input = p.packed_input - # Set p.attention.atten_dropout_deterministic to True by default. - p.attention.atten_dropout_deterministic = True - self.CreateChild('atten', p.attention) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - self.cell.InstantiateVariables() - self.atten.InstantiateVariables() - super()._CreateChildrenVariables() - - @property - def rnn_cell(self): - return self.cell - - @property - def attention(self): - return self.atten - - def InitForSourcePacked(self, - theta, - src_encs, - src_paddings, - src_contexts=None, - src_segment_id=None): - """A wrapper of InitForSourcePacked of child attention layer. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - src_encs: A tensor of shape [source_seq_length, batch_size, source_dim]. - src_paddings: A tensor of shape [source_seq_length, batch_size]. - src_contexts: [Optional] If specified, must be a tensor of shape - [source_seq_length, batch_size, some_dim]. When specified, this tensor - will be used as the source context vectors when computing attention - context, and src_ends will be only used to compute the attention score - for each context. If set to None, the 'src_encs' will be used as - source context. - src_segment_id: A tensor of shape [source_seq_length, batch_size], to - support packed inputs. - - Returns: - packed_src - A `.NestedMap` containing packed source. - """ - atten = self.atten - - if src_contexts is None: - src_contexts = src_encs - - if self.params.packed_input: - assert src_segment_id is not None - - # Initial attention state. - return atten.InitForSourcePacked( - theta=theta.atten, - source_vecs=src_encs, - source_contexts=src_contexts, - source_padding=src_paddings, - source_segment_id=src_segment_id) - - def zero_state(self, - theta, - src_encs, - packed_src, - batch_size, - atten_state_dim=1): - """Initial state of this layer. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - src_encs: A tensor of shape [source_seq_length, batch_size, source_dim]. - packed_src: A `.NestedMap` containing packed source. - batch_size: Batch size. - atten_state_dim: Attention state dim when using zero_atten_state - - Returns: - state0 - A `.NestedMap` containing initial states of RNN and attention. - """ - - p = self.params - atten = self.atten - # Initial RNN states. - state0 = py_utils.NestedMap( - rnn=self.cell.zero_state(theta.cell, batch_size)) - - s_seq_len = tf.shape(src_encs)[0] - - zero_atten_state = atten.ZeroAttentionState(s_seq_len, batch_size) - if p.use_zero_atten_state: - zero_atten_context = tf.zeros([batch_size, p.atten_context_dim], - dtype=py_utils.FPropDtype(p)) - state0.atten = zero_atten_context - state0.atten_state = zero_atten_state - state0.atten_probs = tf.zeros([batch_size, s_seq_len], - dtype=py_utils.FPropDtype(p)) - else: - state0.atten, state0.atten_probs, state0.atten_state = ( - atten.ComputeContextVectorWithSource( - theta.atten, packed_src, - tf.zeros([batch_size, p.cell.num_output_nodes], - dtype=py_utils.FPropDtype(p)), zero_atten_state)) - return state0 - - def reset_atten_state(self, theta, state, inputs): - state.atten = inputs.reset_mask * state.atten - state.atten_state = inputs.reset_mask * state.atten_state - state.atten_probs = inputs.reset_mask * state.atten_probs - return state - - def AccumulateStates(self, - theta, - src_encs, - src_paddings, - inputs, - paddings, - src_contexts=None, - state0=None, - src_segment_id=None, - segment_id=None): - """Sets up and runs the recurrence, returning the raw accumulated states. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - src_encs: A tensor of shape [source_seq_length, batch_size, source_dim]. - src_paddings: A tensor of shape [source_seq_length, batch_size]. - inputs: A tensor of [time, batch, dims]. - paddings: A tensor of [time, batch, 1]. - src_contexts: [Optional] If specified, must be a tensor of shape - [source_seq_length, batch_size, some_dim]. When specified, this tensor - will be used as the source context vectors when computing attention - context, and src_ends will be only used to compute the attention score - for each context. If set to None, the 'src_encs' will be used as source - context. - state0: [Optional] If not None, the initial rnn state and attention - context in a `.NestedMap`. Defaults to the cell's zero-state. - src_segment_id: A tensor of shape [source_seq_length, batch_size] to - support masking with packed inputs. - segment_id: A tensor of [time, batch, 1]. - - Returns: - Tuple of 3 NestedMaps: - - - accumulated_state: a NestedMap of accumulated states from recurrence. - - final_state: final_state: The final recurrent state. - - side_info: NestedMap of intermediate results needed for post-processing. - """ - p = self.params - dtype = p.dtype - rcell = self.cell - atten = self.atten - assert dtype == rcell.params.dtype - assert dtype == atten.params.dtype - assert rcell.params.inputs_arity == 1 or rcell.params.inputs_arity == 2 - if segment_id is None: - segment_id = tf.zeros_like(paddings) - - inputs_shape = tf.shape(inputs) - _, batch = inputs_shape[0], inputs_shape[1] - input_dim = tf.shape(inputs)[2] - - packed_src = self.InitForSourcePacked(theta, src_encs, src_paddings, - src_contexts, src_segment_id) - if state0 is None: - zero_atten_state_dim = p.cell.num_input_nodes - input_dim - state0 = self.zero_state(theta, src_encs, packed_src, batch, - zero_atten_state_dim) - else: - assert not p.packed_input, ('packed input is only supported with default ' - 'initial states.') - - def CellFn(theta, state0, inputs): - """Computes one step forward.""" - if p.packed_input: - state0_mod = state0.DeepCopy() - state0_mod = self.reset_atten_state(theta, state0_mod, inputs) - else: - state0_mod = state0 - state1 = py_utils.NestedMap() - if p.input_prev_atten_ctx: - if rcell.params.inputs_arity == 1: - act = [_ConcatLastDim(inputs.act, state0_mod.atten)] - else: - act = [inputs.act, state0_mod.atten] - else: - act = [inputs.act] - state1.rnn, _ = rcell.FProp( - theta.rnn, state0_mod.rnn, - py_utils.NestedMap( - act=act, padding=inputs.padding, reset_mask=inputs.reset_mask)) - - state1.atten, state1.atten_probs, state1.atten_state = ( - atten.ComputeContextVectorWithSource( - theta.atten, - theta.packed_src, - rcell.GetOutput(state1.rnn), - state0_mod.atten_state, - query_segment_id=tf.cast( - tf.squeeze(inputs.segment_id, 1), py_utils.FPropDtype(p)))) - return state1, py_utils.NestedMap() - - if p.packed_input: - reset_mask = GeneratePackedInputResetMask(segment_id) - else: - reset_mask = tf.zeros_like(paddings) - - acc_state, final_state = recurrent.Recurrent( - theta=py_utils.NestedMap( - rnn=theta.cell, - packed_src=packed_src, - atten=theta.atten, - global_step=theta.global_step), - state0=state0, - inputs=py_utils.NestedMap( - act=inputs, - padding=paddings, - reset_mask=reset_mask, - segment_id=segment_id), - cell_fn=CellFn, - accumulator_layer=self, - allow_implicit_capture=p.allow_implicit_capture) - - side_info = py_utils.NestedMap(state0=state0, reset_mask=reset_mask) - return acc_state, final_state, side_info - - def PostProcessStates(self, acc_state, side_info): - """Post-process accumulated states to fulfill FProp's interface. - - Args: - acc_state: a NestedMap of the raw accumulated states from the recurrence. - side_info: side-information collected by AccumulateStates. - - Returns: - A tuple (atten_context, rnn_output, atten_probs). - - - atten_context: a tensor of [time, batch, attention.context_dim]. - - rnn_output: a tensor of [time, batch, rcell.num_output_nodes]. - - atten_probs: a tensor of [time, batch, source_seq_length]. - """ - - p = self.params - rcell = self.cell - state0 = side_info.state0 - reset_mask = side_info.reset_mask - if p.output_prev_atten_ctx: - # Add the initial attention context in and drop the attention context - # in the last position so that the output atten_ctx is previous - # attention context for each target position. - if p.packed_input: - # Note: Assumes first element of mask is padding, as generated by - # GeneratePackedInputResetMask - atten_ctx = _ShiftRightWithMasking(state0.atten, acc_state.atten, - reset_mask) - atten_probs = _ShiftRightWithMasking(state0.atten_probs, - acc_state.atten_probs, reset_mask) - else: - atten_ctx = _ShiftRight(state0.atten, acc_state.atten) - atten_probs = _ShiftRight(state0.atten_probs, acc_state.atten_probs) - else: - atten_ctx = acc_state.atten - atten_probs = acc_state.atten_probs - - return atten_ctx, rcell.GetOutput(acc_state.rnn), atten_probs - - def FProp(self, - theta, - src_encs, - src_paddings, - inputs, - paddings, - src_contexts=None, - state0=None, - src_segment_id=None, - segment_id=None): - """Forward propagate through a rnn layer with attention. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - src_encs: A tensor of shape [source_seq_length, batch_size, source_dim]. - src_paddings: A tensor of shape [source_seq_length, batch_size]. - inputs: A tensor of [time, batch, dims]. - paddings: A tensor of [time, batch, 1]. - src_contexts: [Optional] If specified, must be a tensor of shape - [source_seq_length, batch_size, some_dim]. When specified, this tensor - will be used as the source context vectors when computing attention - context, and src_ends will be only used to compute the attention score - for each context. If set to None, the 'src_encs' will be used as source - context. - state0: [Optional] If not None, the initial rnn state and attention - context in a `.NestedMap`. Defaults to the cell's zero-state. - src_segment_id: A tensor of shape [source_seq_length, batch_size] to - support masking with packed inputs. - segment_id: A tensor of [time, batch, 1]. - - Returns: - A tuple (atten_context, rnn_output, atten_probs, final_state). - - - atten_context: a tensor of [time, batch, attention.context_dim]. - - rnn_output: a tensor of [time, batch, rcell.num_output_nodes]. - - atten_probs: a tensor of [time, batch, source_seq_length]. - - final_state: The final recurrent state. - """ - - acc_state, final_state, side_info = self.AccumulateStates( - theta, src_encs, src_paddings, inputs, paddings, src_contexts, state0, - src_segment_id, segment_id) - - atten_context, rnn_output, atten_probs = self.PostProcessStates( - acc_state, side_info) - return atten_context, rnn_output, atten_probs, final_state - - -class MultiSourceFRNNWithAttention(base_layer.BaseLayer): - """RNN layer intertwined with an attention layer for multiple sources. - - Allows different attention params per source, if attention is not shared. - """ - - @classmethod - def Params(cls): - """Params for this MultiSourceFRNNWithAttention class.""" - p = super().Params() - p.Define( - 'cell', - rnn_cell.LSTMCellSimple.Params().Set( - params_init=py_utils.WeightInit.Uniform(0.04)), - 'Configs for the RNN cell.') - p.Define( - 'attention_tpl', attention.AdditiveAttention.Params(), - 'Attention used by this attention layer, can be overridden by ' - 'source_name_to_attention_params.') - p.Define( - 'atten_merger', attention.MergerLayer.Params(), - 'Merger layer config for combining context vectors computed for ' - 'different source encodings.') - p.Define('source_names', None, 'List of source names.') - p.Define('share_attention', False, 'If set single attention layer shared.') - p.Define( - 'source_name_to_attention_params', None, - 'Can be set if share_attention is False. Allows defining ' - 'different attention params per source in a dictionary, eg. ' - '{"src1": atten_tpl1, "src2": atten_tpl2}') - p.Define('packed_input', False, 'To reset states for packed inputs.') - return p - - @property - def rnn_cell(self): - """Reference to the RNN cell of this layer.""" - return self.cell - - @property - def attention(self): - """Reference to the attention layer(s) of this layer.""" - return self.attentions - - def __init__(self, params): - """Constructs a MultiSourceFRNNWithAttention layer with params.""" - super().__init__(params) - p = self.params - assert not p.packed_input, ('packed input is not supported for ' - 'MultiSourceFRNNWithAttention') - if p.atten_merger is None: - raise ValueError('Merger layer cannot be none!') - if not isinstance(p.source_names, list) or not p.source_names: - raise ValueError('Source names must be a non-empty list.') - if p.share_attention and p.source_name_to_attention_params: - raise ValueError( - 'Cant specify source_name_to_attention_params with share_attention.') - self.CreateChild('cell', p.cell) - - # Initialize attention layer(s). - params_atten = [] - self._source_dims = [] - src_to_att = p.source_name_to_attention_params - for src_name in p.source_names: - if src_to_att and src_name in src_to_att: - att_params = src_to_att[src_name] - att_params.name = 'atten_%s' % src_name - else: - att_params = p.attention_tpl.Copy() - if p.share_attention: - att_params.name = 'atten_shared' - else: - att_params.name = 'atten_%s' % src_name - if att_params.params_init is None: - att_params.params_init = py_utils.WeightInit.Gaussian( - 1. / math.sqrt(att_params.source_dim + att_params.query_dim), - seed=p.random_seed) - att_params.atten_dropout_deterministic = True - params_atten.append(att_params) - self._source_dims.append(att_params.source_dim) - if p.share_attention: - break - self.CreateChildren('attentions', params_atten) - - # Initialize merger layer for attention layer(s). - params = p.atten_merger.Copy() - params.name = 'atten_merger' - self.CreateChild('atten_merger', params) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - self.cell.InstantiateVariables() - for atten in self.attentions: - atten.InstantiateVariables() - self.atten_merger.InstantiateVariables() - super()._CreateChildrenVariables() - - def InitAttention(self, theta, src_encs, src_paddings, batch_size): - """Computes initial states for attention layer(s). - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - src_encs: A `.NestedMap` object containing source encoding tensors, - each of shape [source_seq_length, batch_size, source_dim]. Children - names of the `.NestedMap` is defined by source_names. - src_paddings: A `.NestedMap` object contraining source padding tensors, - each of shape [source_seq_length, batch_size]. Children names of the - `.NestedMap` is defined by source_names. - batch_size: Scalar Tensor of type int, for initial state shape. - - Returns: - state0 - Initial attention-rnn state in a `.NestedMap`. Zeros for the rnn - initial state, and merger output for attention initial state. - - Transformed source vectors and transposed source vectors. - """ - p = self._params - rcell = self.cell - - # Initial RNN states, theta and auxiliary variables. - state0 = py_utils.NestedMap(rnn=rcell.zero_state(theta.cell, batch_size)) - query_vec0 = tf.zeros([batch_size, p.cell.num_output_nodes], - py_utils.FPropDtype(p)) - - ctxs0 = [] - packed_srcs = py_utils.NestedMap() - for i, src_name in enumerate(p.source_names): - att_idx = (0 if p.share_attention else i) - - packed_srcs[src_name] = self.attentions[att_idx].InitForSourcePacked( - theta.attentions[att_idx], src_encs[src_name], src_encs[src_name], - src_paddings[src_name]) - - # Initial attention state. - s_seq_len = tf.shape(src_encs[src_name])[0] - zero_atten_state = self.attentions[att_idx].ZeroAttentionState( - s_seq_len, batch_size) - ctxs0.append(self.attentions[att_idx].ComputeContextVectorWithSource( - theta.attentions[att_idx], packed_srcs[src_name], query_vec0, - zero_atten_state)[0]) - - # Initial attention state is the output of merger-op. - state0.atten = self.atten_merger.FProp(theta.atten_merger, ctxs0, - query_vec0) - return state0, packed_srcs - - def FProp(self, theta, src_encs, src_paddings, inputs, paddings): - """Forward propagate through a RNN layer with attention(s). - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - src_encs: A `.NestedMap` object containing source encoding tensors, each - of shape [source_seq_length, batch_size, source_dim]. Children names of - the `.NestedMap` is defined by source_names. - src_paddings: A `.NestedMap` object contraining source padding tensors, - each of shape [source_seq_length, batch_size]. Children names of the - `.NestedMap` is defined by source_names. - inputs: A tensor of [time, batch, dims]. - paddings: A tensor of [time, batch, 1]. - - Returns: - A tuple (atten_context, rnn_output) - - - atten_context: a tensor of [time, batch, attention.hidden_dim]. - - rnn_output: a tensor of [time, batch, p.cell.num_output_nodes]. - - Raises: - ValueError: dtype mismatch of attention layers. - """ - p = self.params - dtype = p.dtype - rcell = self.cell - attentions = self.attentions - assert rcell.params.inputs_arity == 1 or rcell.params.inputs_arity == 2 - assert dtype == rcell.params.dtype - for atten in attentions: - if dtype != atten.params.dtype: - raise ValueError('Data type mismatch!') - - # Check if all batch sizes and depths match for source encs and paddings. - src_name_0 = p.source_names[0] - src_encs[src_name_0] = py_utils.with_dependencies([ - py_utils.assert_shape_match( - [tf.shape(src_encs[src_name_0])[1], source_dim], - tf.shape(src_encs[src_name_i])[-2:]) - for src_name_i, source_dim in zip(p.source_names, self._source_dims) - ], src_encs[src_name_0]) - src_paddings[src_name_0] = py_utils.with_dependencies([ - py_utils.assert_equal( - tf.shape(src_paddings[src_name_0])[-1], - tf.shape(src_paddings[src_name_i])[-1]) - for src_name_i in p.source_names[1:] - ], src_paddings[src_name_0]) - - # Compute source transformations and initial rnn states. - state0, packed_src = self.InitAttention(theta, src_encs, src_paddings, - tf.shape(inputs)[1]) - - # Collect individual attention parameters for CellFn. - attens_theta = py_utils.NestedMap({ - src_name: theta.attentions[0 if p.share_attention else i] - for i, src_name in enumerate(p.source_names) - }) - - def CellFn(theta, state0, inputs): - """Computes one step forward.""" - state1 = py_utils.NestedMap() - state1.rnn, _ = rcell.FProp( - theta.rnn, state0.rnn, - py_utils.NestedMap( - act=[_ConcatLastDim(inputs.act, state0.atten)], - padding=inputs.padding)) - - # The ordering in local_ctxs follows p.source_names. - local_ctxs = [] - query_vec = rcell.GetOutput(state1.rnn) - for i, src_name in enumerate(p.source_names): - att_idx = (0 if p.share_attention else i) - local_ctxs.append(attentions[att_idx].ComputeContextVectorWithSource( - theta.attens[src_name], - theta.packed_src[src_name], - query_vec, - state0.atten, - )[0]) - state1.atten = self.atten_merger.FProp(theta.atten_merger, local_ctxs, - query_vec) - return state1, py_utils.NestedMap() - - # Note that, we have a NestedMap for each parameter. - acc_state, _ = recurrent.Recurrent( - theta=py_utils.NestedMap( - rnn=theta.cell, - attens=attens_theta, - packed_src=packed_src, - atten_merger=theta.atten_merger), - state0=state0, - inputs=py_utils.NestedMap(act=inputs, padding=paddings), - cell_fn=CellFn, - accumulator_layer=self, - allow_implicit_capture=p.allow_implicit_capture) - - return acc_state.atten, rcell.GetOutput(acc_state.rnn) - - -class BidirectionalFRNNQuasi(base_layer.BaseLayer): - """Bidirectional functional Quasi-RNN. - - This is very similar to BidirectionalFRNN except the input is a list of the - forward and backward inputs. It is split because quasi-rnns do the - matrix/convolution unrolled over time outside of the recurrent part. Also, - this uses quasi-rnn instead of LSTM. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('fwd', rnn_cell.QRNNPoolingCell.Params(), - 'Configs for the forward RNN cell.') - p.Define('bak', rnn_cell.QRNNPoolingCell.Params(), - 'Configs for the backward RNN cell.') - p.Define('packed_input', False, 'To reset states for packed inputs.') - return p - - def __init__(self, params): - super().__init__(params) - p = params - assert not p.packed_input, ('packed input is not supported for ' - 'BidirectionalFRNNQuasi') - params_forward = FRNN.Params() - params_forward.name = 'fwd' - params_forward.dtype = p.dtype - params_forward.reverse = False - params_forward.cell = p.fwd.Copy() - self.CreateChild('fwd_rnn', params_forward) - - params_backward = FRNN.Params() - params_backward.name = 'bak' - params_backward.dtype = p.dtype - params_backward.reverse = True - params_backward.cell = p.bak.Copy() - self.CreateChild('bak_rnn', params_backward) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - self.fwd_rnn.InstantiateVariables() - self.bak_rnn.InstantiateVariables() - super()._CreateChildrenVariables() - - def FProp(self, theta, inputs, paddings): - """Compute bidi-quasi-RNN forward pass. - - fwd_rnn unroll the sequence in the forward direction and - bak_rnn unroll the sequence in the backward direction. The - outputs are concatenated in the last output dim. - - See `FRNN.FProp` for more details. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - inputs: A list of the fwd and bak tensors. Each item in the list should be - A single tensor or a tuple of tensors with cardinality equal to - rnn_cell.inputs_arity. For every input tensor, the first dimension is - assumed to be time, second dimension batch, and third dimension depth. - paddings: A tensor. First dim is time, second dim is batch, and third dim - is expected to be 1. - - Returns: - A tensor of [time, batch, dims]. - """ - p = self.params - with tf.name_scope(p.name): - fwd_device = self.cluster.WorkerDeviceInModelSplit(0) - bwd_device = self.cluster.WorkerDeviceInModelSplit(1) - with tf.device(fwd_device): - output_forward, _ = self.fwd_rnn.FProp(theta.fwd_rnn, inputs[0], - paddings) - with tf.device(bwd_device): - output_backward, _ = self.bak_rnn.FProp(theta.bak_rnn, inputs[1], - paddings) - output_forward = py_utils.HasShape(output_forward, - tf.shape(output_backward)) - out_rank = tf.rank(output_forward) - 1 - # TODO(yonghui/zhifengc): In the current implementation, we copy - # output_forward from gpu:0 to gpu:1, and then copy the concatenated - # output from gpu:1 to gpu:0 to enable next layer computation. It might - # be more efficient to only copy output_backward from gpu:1 to gpu:0 to - # reduce cross-gpu data transfer. - return tf.concat([output_forward, output_backward], out_rank) diff --git a/lingvo/core/rnn_layers_test.py b/lingvo/core/rnn_layers_test.py deleted file mode 100644 index 1fc7f8de..00000000 --- a/lingvo/core/rnn_layers_test.py +++ /dev/null @@ -1,1560 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for rnn_layers.""" - -import types -from absl.testing import parameterized -import lingvo.compat as tf -from lingvo.core import attention -from lingvo.core import base_layer -from lingvo.core import cluster_factory -from lingvo.core import py_utils -from lingvo.core import rnn_cell -from lingvo.core import rnn_layers -from lingvo.core import test_utils -import numpy as np - -from tensorflow.python.ops import inplace_ops - -FLAGS = tf.flags.FLAGS - - -class TimestepAccumulator(base_layer.Accumulator): - """Simple accumulator for counting timesteps.""" - - def DefaultValue(self): - return tf.convert_to_tensor(0.0) - - def Increment(self): - self.SetValue(self.GetValue() + 1.0) - - -def AddTimestepAccumulator(layer): - orig_fprop = layer.FProp - - def WrappedFProp(*args, **kwargs): - layer.accumulators.ts_count.Increment() - return orig_fprop(*args, **kwargs) - - layer.FProp = WrappedFProp - - layer.RegisterAccumulator('ts_count', TimestepAccumulator()) - - -class LayersTestBase(test_utils.TestCase): - - def _testStackedFRNNHelper(self, - cls, - dtype, - trailing_pad_len=0, - keep_prob=1.0, - bi_directional=False, - input_dim=-1, - output_dim=-1): - tf.random.set_seed(123456) - batch = 3 - dims = 16 - slen = 10 + trailing_pad_len - num_layers = 4 - with tf.Graph().as_default() as g: - with self.session(use_gpu=True, graph=g): - params = rnn_cell.LSTMCellSimple.Params() - params.name = 'lstm' - params.output_nonlinearity = True - params.dtype = dtype - params.params_init = py_utils.WeightInit.Uniform(1.24, 429891685) - params.vn.global_vn = True - params.vn.per_step_vn = False - params.vn.seed = 2938482 - params.vn.scale = 0.1 - params.num_input_nodes = dims - params.num_output_nodes = dims // 2 if bi_directional else dims - - sfrnn_params = cls.Params() - sfrnn_params.name = 'sfrnn' - sfrnn_params.dtype = dtype - sfrnn_params.random_seed = 123456 - sfrnn_params.cell_tpl = params - sfrnn_params.num_layers = num_layers - sfrnn_params.skip_start = 2 - sfrnn_params.dropout.keep_prob = keep_prob - sfrnn_params.num_input_nodes = input_dim - sfrnn_params.num_output_nodes = output_dim - with tf.name_scope('sfrnn'): - sfrnn = sfrnn_params.Instantiate() - - np.random.seed(12345) - input_dim = input_dim if input_dim > 0 else dims - output_dim = output_dim if output_dim > 0 else dims - inputs = tf.constant( - np.random.uniform(size=(slen, batch, input_dim)), dtype) - paddings = np.zeros([slen, batch, 1]) - if trailing_pad_len > 0: - paddings[-trailing_pad_len:, :] = 1.0 - paddings[-trailing_pad_len - 3:-trailing_pad_len - 1, :] = 1.0 - paddings = tf.constant(paddings, dtype) - - self.evaluate(tf.global_variables_initializer()) - if bi_directional: - sfrnn_outputs = sfrnn.FPropFullSequence(sfrnn.theta, inputs, paddings) - sfrnn_outputs = py_utils.HasShape(sfrnn_outputs, - [slen, batch, output_dim]) - return self.evaluate(sfrnn_outputs) - else: - sfrnn_outputs, sfrnn_final = sfrnn.FPropDefaultTheta(inputs, paddings) - sfrnn_outputs = py_utils.HasShape(sfrnn_outputs, - [slen, batch, output_dim]) - return self.evaluate([sfrnn_outputs, sfrnn_final]) - - def _testStackedFRNNGradHelper(self, cls, bi_directional=False): - trailing_pad_len = 2 - dtype = tf.float64 - batch = 3 - dims = 16 - slen = 10 + trailing_pad_len - num_layers = 4 - with self.session(use_gpu=True, graph=tf.Graph()) as sess: - params = rnn_cell.LSTMCellSimple.Params() - params.name = 'lstm' - params.output_nonlinearity = True - params.dtype = dtype - params.params_init = py_utils.WeightInit.Uniform(1.24, 429891685) - params.vn.global_vn = False - params.vn.per_step_vn = False - params.vn.seed = 2938482 - params.vn.scale = 0.1 - params.num_input_nodes = dims - params.num_output_nodes = dims // 2 if bi_directional else dims - - sfrnn_params = cls.Params() - sfrnn_params.name = 'sfrnn' - sfrnn_params.dtype = dtype - sfrnn_params.cell_tpl = params - sfrnn_params.num_layers = num_layers - sfrnn_params.skip_start = 2 - with tf.name_scope('sfrnn'): - sfrnn = sfrnn_params.Instantiate() - - np.random.seed(12345) - inputs = tf.constant(np.random.uniform(size=(slen, batch, dims)), dtype) - paddings = np.zeros([slen, batch, 1]) - paddings[-trailing_pad_len:, :] = 1.0 - paddings[-trailing_pad_len - 3:-trailing_pad_len - 1, :] = 1.0 - paddings = tf.constant(paddings, dtype) - - if bi_directional: - sfrnn_outputs = sfrnn.FPropDefaultTheta(inputs, paddings) - loss = tf.reduce_sum(sfrnn_outputs) - else: - sfrnn_outputs, sfrnn_final = sfrnn.FPropDefaultTheta(inputs, paddings) - loss = tf.reduce_sum(sfrnn_outputs) - for fin in sfrnn_final.rnn: - loss += tf.reduce_sum(fin.m) + tf.reduce_sum(fin.c) - xs = sfrnn.vars.Flatten() + [inputs] - dxs = tf.gradients(loss, xs) - - # Compares the sym grad against the numeric grads. - self.evaluate(tf.global_variables_initializer()) - grad_step = 17 - sym_grads = self.evaluate(dxs) - sym_grads = [test_utils.PickEveryN(_, grad_step) for _ in sym_grads] - num_grads = [ - test_utils.PickEveryN( - test_utils.ComputeNumericGradient( - sess, loss, v, delta=1e-4, step=grad_step), grad_step) - for v in xs - ] - for (sym, num) in zip(sym_grads, num_grads): - self.assertFalse(np.any(np.isnan(sym))) - self.assertFalse(np.any(np.isnan(num))) - print('max = ', np.max(np.abs(sym))) - self.assertAllClose(sym, num) - - -class LayersTest(LayersTestBase, parameterized.TestCase): - - def testIdentitySeqLayer(self): - with self.session(use_gpu=False): - rnn_params = rnn_layers.IdentitySeqLayer.Params() - rnn_params.name = 'no_op' - rnn = rnn_params.Instantiate() - - np.random.seed(12345) - inputs_sequence = [] - paddings_sequence = [] - for _ in range(5): - inputs_sequence.append( - tf.constant(np.random.uniform(size=(3, 2)), tf.float32)) - paddings_sequence.append(tf.zeros([3, 1])) - - paddings_sequence[-1] = tf.constant([[1.0], [1.0], [1.0]]) - paddings_sequence[-2] = tf.constant([[1.0], [1.0], [1.0]]) - - inputs, paddings = tf.stack(inputs_sequence), tf.stack(paddings_sequence) - outputs = rnn.FPropFullSequence(rnn.theta, inputs, paddings) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - - inputs_v, outputs_v = self.evaluate([inputs, outputs]) - self.assertAllEqual(inputs_v, outputs_v) - - def testRNN(self): - with self.session(use_gpu=False): - params = rnn_cell.LSTMCellSimple.Params() - params.name = 'lstm' - params.output_nonlinearity = True - params.params_init = py_utils.WeightInit.Uniform(1.24, 429891685) - params.num_input_nodes = 2 - params.num_output_nodes = 2 - - rnn_params = rnn_layers.RNN.Params() - rnn_params.name = 'rnn' - rnn_params.vn.global_vn = True - rnn_params.vn.per_step_vn = False - rnn_params.vn.seed = 2938482 - rnn_params.vn.scale = 0.1 - rnn_params.cell = params - rnn_params.sequence_length = 10 - rnn = rnn_layers.RNN(rnn_params) - - np.random.seed(12345) - inputs_sequence = [] - paddings_sequence = [] - for _ in range(rnn_params.sequence_length): - inputs_sequence.append( - tf.constant(np.random.uniform(size=(3, 2)), tf.float32)) - paddings_sequence.append(tf.zeros([3, 1])) - - paddings_sequence[-1] = tf.constant([[1.0], [1.0], [1.0]]) - paddings_sequence[-2] = tf.constant([[1.0], [1.0], [1.0]]) - - inputs, paddings = tf.stack(inputs_sequence), tf.stack(paddings_sequence) - outputs, final = rnn.FPropDefaultTheta(inputs, paddings) - - outputs *= paddings - sum_outputs = tf.reduce_sum(outputs, axis=0) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - - actual = self.evaluate(py_utils.NestedMap(sum=sum_outputs, **final)) - - # In case this test ever breaks, you can uncomment the line below, copy - # out the golden values, and then comment out the line below again. - # - sum_expected = [[-0.46614376, 0.86599183], [-0.63463444, 0.57043159], - [-0.64659989, 0.72516292]] - c_expected = [[-0.63455635, 0.76446551], [-0.59985822, 0.6631192], - [-0.63043576, 0.77522433]] - m_expected = [[-0.23307188, 0.43299592], [-0.31731722, 0.2852158], - [-0.32329994, 0.36258146]] - self.assertAllClose(sum_expected, actual.sum) - self.assertAllClose(m_expected, actual.m) - self.assertAllClose(c_expected, actual.c) - - def testRNNGradientChecker(self): - with self.session(use_gpu=False) as sess: - params = rnn_cell.LSTMCellSimple.Params() - params.name = 'lstm' - params.output_nonlinearity = True - params.params_init = py_utils.WeightInit.Uniform(1.24, 429891685) - params.num_input_nodes = 2 - params.num_output_nodes = 2 - - rnn_params = rnn_layers.RNN.Params() - rnn_params.name = 'rnn' - rnn_params.dtype = tf.float64 - rnn_params.vn.scale = 0.1 - rnn_params.vn.global_vn = False - rnn_params.vn.per_step_vn = False - rnn_params.cell = params - rnn_params.sequence_length = 10 - rnn = rnn_layers.RNN(rnn_params) - - np.random.seed(12345) - inputs_sequence = [] - paddings_sequence = [] - for _ in range(rnn_params.sequence_length): - inputs_sequence.append( - tf.constant(np.random.uniform(size=(3, 2)), tf.float64)) - paddings_sequence.append(tf.zeros([3, 1], dtype=tf.float64)) - - paddings_sequence[-1] = tf.constant( - [[1.0], [1.0], [1.0]], dtype=tf.float64) - paddings_sequence[-2] = tf.constant( - [[1.0], [1.0], [1.0]], dtype=tf.float64) - - inputs, paddings = tf.stack(inputs_sequence), tf.stack(paddings_sequence) - outputs, final = rnn.FPropDefaultTheta(inputs, paddings) - outputs *= paddings - sum_outputs = tf.reduce_sum(outputs, axis=0) - - loss = tf.reduce_sum(sum_outputs) + tf.reduce_sum(final.m + final.c) - all_vars = tf.get_collection('LSTMCellSimple_vars') - assert len(all_vars) == 2 - - grads = tf.gradients(loss, all_vars) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - - symbolic_grads = [gd.eval() for gd in grads] - numerical_grads = [] - for v in all_vars: - numerical_grads.append(test_utils.ComputeNumericGradient(sess, loss, v)) - for x, y in zip(symbolic_grads, numerical_grads): - self.assertAllClose(x, y) - - def testRNNReversed(self): - """Test an RNN layer with reverse=true. - - This should yield the same output as feeding reversed input into - the same RNN with reverse=false (except the output is in reversed order). - """ - - timesteps = 10 - padding_steps = 2 - batch_size = 2 - depth = 3 - with self.session(use_gpu=True): - lstm_params = rnn_cell.LSTMCellSimple.Params() - lstm_params.output_nonlinearity = True - lstm_params.num_input_nodes = depth - lstm_params.num_output_nodes = depth - - rnn_params = rnn_layers.RNN.Params() - rnn_params.vn.global_vn = False - rnn_params.vn.per_step_vn = False - rnn_params.vn.seed = 2938482 - rnn_params.vn.scale = 0.1 - rnn_params.cell = lstm_params - rnn_params.sequence_length = timesteps - - fwd_rnn_params = rnn_params.Copy() - fwd_rnn_params.name = 'fwd' - fwd_rnn_params.cell.name = 'fwd_lstm' - fwd_rnn = rnn_layers.RNN(fwd_rnn_params) - - bak_rnn_params = rnn_params.Copy() - bak_rnn_params.name = 'bak' - bak_rnn_params.cell.name = 'bak_lstm' - bak_rnn_params.reverse = True - bak_rnn = rnn_layers.RNN(bak_rnn_params) - - # Create 8 timesteps of random input, 2 timesteps of zeros, and paddings - # to match. - fwd_inputs = tf.constant( - np.concatenate( - (np.random.uniform( - size=(timesteps - padding_steps, batch_size, depth)), - np.zeros(shape=(padding_steps, batch_size, depth))), - axis=0), tf.float32) - fwd_paddings = tf.concat( - (tf.zeros(shape=(timesteps - padding_steps, batch_size, depth)), - tf.ones(shape=(padding_steps, batch_size, depth))), - axis=0) - bak_inputs = tf.reverse(fwd_inputs, [0]) - bak_paddings = tf.reverse(fwd_paddings, [0]) - - # Run the forward rnn with reversed inputs - reversed_outputs, _ = fwd_rnn.FProp(fwd_rnn.theta, bak_inputs, - bak_paddings) - reversed_outputs = tf.reverse(reversed_outputs, [0]) - - # Run the backward rnn with forward inputs. Note we reuse the fwd_rnn - # theta so the results should match - bak_outputs, _ = bak_rnn.FProp(fwd_rnn.theta, fwd_inputs, fwd_paddings) - - self.evaluate(tf.global_variables_initializer()) - actual_reversed_outputs, actual_bak_outputs = self.evaluate( - [reversed_outputs, bak_outputs]) - self.assertAllClose(actual_reversed_outputs, actual_bak_outputs) - - def testRNNWithConvLSTMCell(self): - with self.session(use_gpu=False): - params = rnn_cell.ConvLSTMCell.Params() - params.name = 'conv_lstm' - params.output_nonlinearity = True - params.params_init = py_utils.WeightInit.Uniform(1.24, 429891685) - params.inputs_shape = [None, 4, 2, 3] - params.cell_shape = [None, 4, 2, 2] - params.filter_shape = [3, 2] - - rnn_params = rnn_layers.RNN.Params() - rnn_params.name = 'rnn' - rnn_params.vn.global_vn = True - rnn_params.vn.per_step_vn = False - rnn_params.vn.seed = 2938482 - rnn_params.vn.scale = 0.1 - rnn_params.cell = params - rnn_params.sequence_length = 10 - rnn = rnn_layers.RNN(rnn_params) - - np.random.seed(12345) - inputs_sequence = [] - paddings_sequence = [] - for _ in range(rnn_params.sequence_length): - inputs_sequence.append( - tf.constant(np.random.uniform(size=(3, 4, 2, 3)), tf.float32)) - paddings_sequence.append(tf.zeros([3, 1])) - - paddings_sequence[-1] = tf.constant([[1.0], [1.0], [1.0]]) - paddings_sequence[-2] = tf.constant([[1.0], [1.0], [1.0]]) - - inputs, paddings = tf.stack(inputs_sequence), tf.stack(paddings_sequence) - outputs, final = rnn.FPropDefaultTheta(inputs, paddings) - outputs *= tf.reshape(1.0 - paddings, - [rnn_params.sequence_length, -1, 1, 1, 1]) - - sum_outputs = tf.reduce_sum(tf.reduce_sum(outputs, axis=0), [1, 2, 3]) - sum_final_m = tf.reduce_sum(final.m, [1, 2, 3]) - sum_final_c = tf.reduce_sum(final.c, [1, 2, 3]) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - actual = self.evaluate( - py_utils.NestedMap(sum=sum_outputs, m=sum_final_m, c=sum_final_c)) - - # In case this test ever breaks, you can uncomment the line below, copy - # out the golden values, and then comment out the line below again. - # - # print('sum_outputs', np.array_repr(actual.sum)) - # print('final_m', np.array_repr(actual.m)) - # print('final_c', np.array_repr(actual.c)) - - sum_expected = [1.59282923, 0.85964835, -1.09797788] - m_expected = [0.32634005, 0.38098311, 0.23331133] - c_expected = [2.72942424, 2.72087693, 2.97179723] - self.assertAllClose(sum_expected, actual.sum) - self.assertAllClose(m_expected, actual.m) - self.assertAllClose(c_expected, actual.c) - - def _testFRNNWithConvLSTMCell(self): - with self.session(use_gpu=True): - params = rnn_cell.ConvLSTMCell.Params() - params.name = 'conv_lstm' - params.output_nonlinearity = True - params.params_init = py_utils.WeightInit.Uniform(1.24, 429891685) - params.inputs_shape = [None, 4, 2, 3] - params.cell_shape = [None, 4, 2, 2] - params.filter_shape = [3, 2] - - rnn_params = rnn_layers.RNN.Params() - rnn_params.name = 'rnn' - rnn_params.vn.global_vn = True - rnn_params.vn.per_step_vn = False - rnn_params.vn.seed = 2938482 - rnn_params.vn.scale = 0.1 - rnn_params.cell = params - rnn_params.sequence_length = 10 - - with tf.variable_scope('rnn'): - rnn = rnn_layers.RNN(rnn_params) - - np.random.seed(12345) - inputs_sequence = [] - paddings_sequence = [] - for _ in range(rnn_params.sequence_length): - inputs_sequence.append( - tf.constant(np.random.uniform(size=(3, 4, 2, 3)), tf.float32)) - paddings_sequence.append(tf.zeros([3, 1])) - - paddings_sequence[-1] = tf.constant([[1.0], [1.0], [1.0]]) - paddings_sequence[-2] = tf.constant([[1.0], [1.0], [1.0]]) - - inputs, paddings = tf.stack(inputs_sequence), tf.stack(paddings_sequence) - outputs, final = rnn.FPropDefaultTheta(inputs, paddings) - outputs *= tf.reshape(1.0 - paddings, - [rnn_params.sequence_length, -1, 1, 1, 1]) - rnn_outputs = outputs - - frnn_params = rnn_layers.FRNN.Params() - frnn_params.name = 'frnn' - frnn_params.cell = params - frnn_params.vn = rnn_params.vn - - with tf.variable_scope('frnn'): - frnn = frnn_params.Instantiate() - - frnn_outputs, frnn_final = frnn.FPropDefaultTheta( - tf.stack(inputs_sequence), tf.stack(paddings_sequence)) - paddings = tf.stack(paddings_sequence) - frnn_outputs *= tf.reshape(1.0 - paddings, - tf.concat([tf.shape(paddings), [1, 1]], 0)) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - - (rnn_outputs_v, rnn_final_v, frnn_outputs_v, - frnn_final_v) = self.evaluate( - [rnn_outputs, final, frnn_outputs, frnn_final]) - - self.assertAllClose(rnn_outputs_v, frnn_outputs_v) - self.assertAllClose(rnn_final_v.m, frnn_final_v.m) - self.assertAllClose(rnn_final_v.c, frnn_final_v.c) - - def testFRNNWithConvLSTMCell4(self): - self._testFRNNWithConvLSTMCell() - - def testRNNWithConvLSTMCellGradientChecker(self): - with self.session(use_gpu=True) as sess: - params = rnn_cell.ConvLSTMCell.Params() - params.name = 'conv_lstm' - params.output_nonlinearity = True - params.params_init = py_utils.WeightInit.Uniform(1.24, 429891685) - params.inputs_shape = [None, 4, 2, 3] - params.cell_shape = [None, 4, 2, 2] - params.filter_shape = [3, 2] - - rnn_params = rnn_layers.RNN.Params() - rnn_params.name = 'rnn' - rnn_params.vn.global_vn = False - rnn_params.vn.per_step_vn = False - rnn_params.vn.seed = 2938482 - rnn_params.vn.scale = 0.1 - rnn_params.cell = params - rnn_params.sequence_length = 10 - rnn = rnn_layers.RNN(rnn_params) - - np.random.seed(12345) - inputs_sequence = [] - paddings_sequence = [] - for _ in range(rnn_params.sequence_length): - inputs_sequence.append( - tf.constant(np.random.uniform(size=(3, 4, 2, 3)), tf.float32)) - paddings_sequence.append(tf.zeros([3, 1])) - - paddings_sequence[-1] = tf.constant([[1.0], [1.0], [1.0]]) - paddings_sequence[-2] = tf.constant([[1.0], [1.0], [1.0]]) - - inputs, paddings = tf.stack(inputs_sequence), tf.stack(paddings_sequence) - outputs, final = rnn.FPropDefaultTheta(inputs, paddings) - outputs *= tf.reshape(1.0 - paddings, - [rnn_params.sequence_length, -1, 1, 1, 1]) - loss = tf.reduce_sum(tf.reduce_sum( - outputs, axis=0)) + tf.reduce_sum(final.m + final.c) - all_vars = tf.trainable_variables() - assert len(all_vars) == 2 - - grads = tf.gradients(loss, all_vars) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - - symbolic_grads = [gd.eval() for gd in grads] - numerical_grads = [] - for v in all_vars: - numerical_grads.append(test_utils.ComputeNumericGradient(sess, loss, v)) - for x, y in zip(symbolic_grads, numerical_grads): - self.assertAllClose(x, y, rtol=0.1, atol=0.1) - - def _testFRNNWithConvLSTMCellGradientChecker(self): - with self.session(use_gpu=True) as sess: - params = rnn_cell.ConvLSTMCell.Params() - params.name = 'conv_lstm' - params.output_nonlinearity = True - params.params_init = py_utils.WeightInit.Uniform(1.24, 429891685) - params.inputs_shape = [None, 4, 2, 3] - params.cell_shape = [None, 4, 2, 2] - params.filter_shape = [3, 2] - - frnn_params = rnn_layers.FRNN.Params() - frnn_params.name = 'rnn' - frnn_params.vn.global_vn = False - frnn_params.vn.per_step_vn = False - frnn_params.vn.seed = 2938482 - frnn_params.vn.scale = 0.1 - frnn_params.cell = params - frnn = rnn_layers.FRNN(frnn_params) - - np.random.seed(12345) - inputs_sequence = tf.constant( - np.random.uniform(size=(10, 3, 4, 2, 3)), tf.float32) - paddings = inplace_ops.inplace_update( - tf.zeros([10, 3, 1]), 1, [[1.0], [0.0], [1.0]]) - - outputs, final = frnn.FPropDefaultTheta(inputs_sequence, paddings) - outputs *= tf.reshape(paddings, [10, 3, 1, 1, 1]) - loss = tf.reduce_sum(outputs) + tf.reduce_sum(final.m + final.c) - all_vars = tf.trainable_variables() - assert len(all_vars) == 2 - - grads = tf.gradients(loss, all_vars) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - - symbolic_grads = [gd.eval() for gd in grads] - numerical_grads = [] - for v in all_vars: - numerical_grads.append(test_utils.ComputeNumericGradient(sess, loss, v)) - for x, y in zip(symbolic_grads, numerical_grads): - self.assertAllClose(x, y, rtol=0.1, atol=0.1) - - def testFRNNWithConvLSTMCellGradientChecker(self): - self._testFRNNWithConvLSTMCellGradientChecker() - - def testFRNNWithLSTMCellSimpleDeterministicGradientChecker(self): - with self.session(use_gpu=True) as sess: - params = rnn_cell.LSTMCellSimpleDeterministic.Params() - params.name = 'conv_lstm' - params.output_nonlinearity = True - params.params_init = py_utils.WeightInit.Uniform(1.24, 429891685) - params.zo_prob = 0.25 - params.num_input_nodes = 4 - params.num_output_nodes = 6 - params.dtype = tf.float64 - - frnn_params = rnn_layers.FRNN.Params() - frnn_params.dtype = tf.float64 - frnn_params.name = 'rnn' - frnn_params.cell = params - old_enable_asserts = FLAGS.enable_asserts - FLAGS.enable_asserts = False - frnn = rnn_layers.FRNN(frnn_params) - FLAGS.enable_asserts = old_enable_asserts - - np.random.seed(12345) - inputs_sequence = tf.constant( - np.random.uniform(size=(10, 3, 4)), tf.float64) - paddings = inplace_ops.inplace_update( - tf.zeros([10, 3, 1], tf.float64), 1, [[1.0], [0.0], [1.0]]) - - outputs, _ = frnn.FPropDefaultTheta(inputs_sequence, paddings) - outputs *= (1.0 - tf.reshape(paddings, [10, 3, 1])) - loss = tf.reduce_sum(outputs) - - all_vars = tf.trainable_variables() - assert len(all_vars) == 2 - - grads = tf.gradients(loss, all_vars) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - - symbolic_grads = [gd.eval() for gd in grads] - numerical_grads = [] - for v in all_vars: - numerical_grads.append(test_utils.ComputeNumericGradient(sess, loss, v)) - for x, y in zip(symbolic_grads, numerical_grads): - self.assertAllClose(x, y, rtol=0.00001, atol=0.00001) - - def _testFRNNHelper(self, config=None): - dtype = tf.float32 - batch = 3 - dims = 16 - slen = 10 - with self.session(use_gpu=True, config=config): - params = rnn_cell.LSTMCellSimple.Params() - params.name = 'lstm' - params.output_nonlinearity = True - params.dtype = dtype - params.params_init = py_utils.WeightInit.Uniform(1.24, 429891685) - params.vn.global_vn = True - params.vn.per_step_vn = False - params.vn.seed = 2938482 - params.vn.scale = 0.1 - params.num_input_nodes = dims - params.num_output_nodes = dims - - frnn_params = rnn_layers.FRNN.Params() - frnn_params.name = 'frnn' - frnn_params.dtype = dtype - frnn_params.cell = params - with tf.variable_scope('frnn'): - frnn = rnn_layers.FRNN(frnn_params) - AddTimestepAccumulator(frnn.cell) - - rnn_params = rnn_layers.RNN.Params() - rnn_params.name = 'rnn' - rnn_params.dtype = dtype - rnn_params.sequence_length = slen - rnn_params.cell = params - with tf.variable_scope('rnn'): - rnn = rnn_layers.RNN(rnn_params) - - np.random.seed(12345) - inputs = tf.constant( - np.random.uniform(size=(slen, batch, dims)), tf.float32) - paddings = np.zeros([slen, batch, 1]) - paddings[-3:-1, :] = 1.0 - paddings = tf.constant(paddings, tf.float32) - - frnn_outputs, frnn_final = frnn.FPropDefaultTheta(inputs, paddings) - rnn_outputs, rnn_final = rnn.FPropDefaultTheta( - tf.unstack(inputs), tf.unstack(paddings)) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - frnn_out, frnn_final, rnn_out, rnn_final = self.evaluate( - [frnn_outputs, frnn_final, rnn_outputs, rnn_final]) - self.assertAllClose(frnn_out, rnn_out) - self.assertAllClose(frnn_final.m, rnn_final.m) - self.assertAllClose(frnn_final.c, rnn_final.c) - - def testFRNNNoInline(self): - self._testFRNNHelper(py_utils.SessionConfig(inline=False)) - - def testFRNNInline(self): - self._testFRNNHelper(py_utils.SessionConfig(inline=True)) - - def _testFRNNGradHelper(self, config): - dtype = tf.float64 # More stable using float64. - batch = 3 - dims = 16 - slen = 10 - with self.session(use_gpu=True, config=config) as sess: - params = rnn_cell.LSTMCellSimple.Params() - params.name = 'lstm' - params.output_nonlinearity = True - params.params_init = py_utils.WeightInit.Uniform(0.02, 429891685) - params.vn.global_vn = False - params.vn.per_step_vn = False - params.num_input_nodes = dims - params.num_output_nodes = dims - params.dtype = dtype - - frnn_params = rnn_layers.FRNN.Params() - frnn_params.name = 'frnn' - frnn_params.dtype = dtype - frnn_params.cell = params - frnn = rnn_layers.FRNN(frnn_params) - AddTimestepAccumulator(frnn.cell) - w, b = frnn.theta.cell.wm, frnn.theta.cell.b - - np.random.seed(12345) - inputs = tf.constant( - np.random.uniform(-0.02, 0.02, size=(slen, batch, dims)), dtype) - paddings = np.zeros([slen, batch, 1]) - paddings[-3:-1, :] = 1.0 - paddings = tf.constant(paddings, dtype) - frnn_outputs, frnn_final = frnn.FPropDefaultTheta(inputs, paddings) - loss = tf.reduce_sum(frnn_outputs) + tf.reduce_sum( - frnn_final.m) + tf.reduce_sum(frnn_final.c) - dw, db, dinputs = tf.gradients(loss, [w, b, inputs]) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - grad_step = 7 - sym_grads = self.evaluate([db, dw, dinputs]) - sym_grads = [test_utils.PickEveryN(_, grad_step) for _ in sym_grads] - num_grads = [ - test_utils.PickEveryN( - test_utils.ComputeNumericGradient( - sess, loss, v, delta=1e-4, step=grad_step), grad_step) - for v in [b, w, inputs] - ] - for (sym, num) in zip(sym_grads, num_grads): - self.assertFalse(np.any(np.isnan(sym))) - self.assertFalse(np.any(np.isnan(num))) - print('max = ', np.max(np.abs(sym))) - self.assertAllClose(sym, num) - - def testFRNNGradNoInline(self): - self._testFRNNGradHelper(py_utils.SessionConfig(inline=False)) - - def testFRNNGradInline(self): - self._testFRNNGradHelper(py_utils.SessionConfig(inline=True)) - - def testStackedFRNNDropout(self): - v1_out, _ = self._testStackedFRNNHelper( - rnn_layers.StackedFRNNLayerByLayer, - tf.float32, - trailing_pad_len=0, - keep_prob=0.5) - if tf.test.is_gpu_available(): - rtol = 1e-5 - else: - rtol = 1e-6 - self.assertAllClose([175.974121], [np.sum(v1_out * v1_out)], rtol=rtol) - - def testStackedFRNNInputOutputDims(self): - v1_out, _ = self._testStackedFRNNHelper( - rnn_layers.StackedFRNNLayerByLayer, - tf.float32, - trailing_pad_len=0, - keep_prob=0.5, - input_dim=5, - output_dim=7) - if tf.test.is_gpu_available(): - rtol = 1e-5 - else: - rtol = 1e-6 - self.assertAllClose([32.743263], [np.sum(v1_out * v1_out)], rtol=rtol) - - def testStackedFRNNLayerByLayerGrad(self): - self._testStackedFRNNGradHelper(rnn_layers.StackedFRNNLayerByLayer) - - def testStackedBiFRNNDropout(self): - v1_out = self._testStackedFRNNHelper( - rnn_layers.StackedBiFRNNLayerByLayer, - tf.float32, - trailing_pad_len=0, - keep_prob=0.5, - bi_directional=True) - if tf.test.is_gpu_available(): - rtol = 1e-5 - else: - rtol = 1e-6 - self.assertAllClose([305.774384], [np.sum(v1_out * v1_out)], rtol=rtol) - - def testStackedBiFRNNInputOutputDims(self): - v1_out = self._testStackedFRNNHelper( - rnn_layers.StackedBiFRNNLayerByLayer, - tf.float32, - trailing_pad_len=0, - keep_prob=0.5, - bi_directional=True, - input_dim=5, - output_dim=8) - if tf.test.is_gpu_available(): - rtol = 1e-5 - else: - rtol = 1e-6 - self.assertAllClose([8.116007], [np.sum(v1_out * v1_out)], rtol=rtol) - - def testStackedBiFRNNLayerByLayerGrad(self): - self._testStackedFRNNGradHelper( - rnn_layers.StackedBiFRNNLayerByLayer, bi_directional=True) - - def _testBidirectionalFRNNHelper(self, - trailing_pad_len=0, - cluster_params=None): - batch = 3 - dims = 16 - slen = 10 + trailing_pad_len - with self.session( - use_gpu=True, - config=tf.config_pb2.ConfigProto(allow_soft_placement=True)): - params = rnn_cell.LSTMCellSimple.Params() - params.name = 'lstm_forward' - params.output_nonlinearity = True - params.params_init = py_utils.WeightInit.Uniform(0.02, 429891685) - params.vn.global_vn = True - params.vn.per_step_vn = False - params.vn.seed = 2938482 - params.vn.scale = 0.1 - params.num_input_nodes = dims - params.num_output_nodes = dims - lstm_forward = params.Copy() - params.name = 'lstm_backward' - params.params_init = py_utils.WeightInit.Uniform(0.02, 83820209838) - lstm_backward = params.Copy() - - frnn_params = rnn_layers.BidirectionalFRNN.Params() - frnn_params.name = 'bifrnn' - frnn_params.fwd = lstm_forward.Copy() - frnn_params.bak = lstm_backward.Copy() - with tf.variable_scope('frnn'): - frnn = rnn_layers.BidirectionalFRNN(frnn_params) - - rnn_params = rnn_layers.BidirectionalRNN.Params() - rnn_params.name = 'rnn' - rnn_params.fwd = lstm_forward.Copy() - rnn_params.bak = lstm_backward.Copy() - rnn_params.sequence_length = slen - - with cluster_factory.Cluster(cluster_params if cluster_params else - cluster_factory.Cluster.Params()): - with tf.variable_scope('rnn'): - rnn = rnn_layers.BidirectionalRNN(rnn_params) - - np.random.seed(12345) - inputs = tf.constant( - np.random.uniform(size=(slen, batch, dims)), tf.float32) - paddings = np.zeros([slen, batch, 1]) - paddings[-trailing_pad_len:, :] = 1.0 - paddings[-trailing_pad_len - 3:-trailing_pad_len - 1, :] = 1.0 - paddings = tf.constant(paddings, tf.float32) - - frnn_outputs = frnn.FPropDefaultTheta(inputs, paddings) - rnn_outputs = rnn.FPropDefaultTheta( - tf.unstack(inputs), tf.unstack(paddings)) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - rnn_outputs_val, frnn_outputs_val = [ - x[:-trailing_pad_len] - for x in self.evaluate([rnn_outputs, frnn_outputs]) - ] - self.assertAllClose(rnn_outputs_val, frnn_outputs_val) - - def testBidirectionalFRNN(self): - self._testBidirectionalFRNNHelper() - - def testBidirectionalFRNNTrailingPadding(self): - self._testBidirectionalFRNNHelper(trailing_pad_len=2) - - def testBidirectionalFRNNSplit(self): - cluster_params = cluster_factory.Cluster.Params() - cluster_params.worker.Set( - gpus_per_replica=2, devices_per_split=2, name='/job:localhost') - self._testBidirectionalFRNNHelper(cluster_params=cluster_params) - - def _testBidirectionalFRNNGradientHelper(self): - dtype = tf.float64 # More stable using float64. - batch = 3 - dims = 16 - slen = 10 - with self.session(use_gpu=True) as sess: - params = rnn_cell.LSTMCellSimple.Params() - params.name = 'lstm_forward' - params.output_nonlinearity = True - params.params_init = py_utils.WeightInit.Uniform(0.02, 429891685) - params.vn.global_vn = False - params.vn.per_step_vn = False - params.vn.scale = 0.1 - params.dtype = dtype - params.num_input_nodes = dims - params.num_output_nodes = dims - lstm_forward = params.Copy() - params.name = 'lstm_backward' - params.params_init = py_utils.WeightInit.Uniform(0.02, 83820209838) - params.dtype = dtype - lstm_backward = params.Copy() - - frnn_params = rnn_layers.BidirectionalFRNN.Params() - frnn_params.name = 'bifrnn' - frnn_params.dtype = dtype - frnn_params.fwd = lstm_forward.Copy() - frnn_params.bak = lstm_backward.Copy() - frnn = rnn_layers.BidirectionalFRNN(frnn_params) - w0, b0 = (frnn.theta.fwd_rnn.cell.wm, frnn.theta.fwd_rnn.cell.b) - w1, b1 = (frnn.theta.bak_rnn.cell.wm, frnn.theta.bak_rnn.cell.b) - - np.random.seed(12345) - inputs = tf.constant(np.random.uniform(size=(slen, batch, dims)), dtype) - paddings = np.zeros([slen, batch, 1]) - paddings[-3:-1, :] = 1.0 - paddings = tf.constant(paddings, dtype) - frnn_outputs = frnn.FPropDefaultTheta(inputs, paddings) - loss = tf.reduce_sum(frnn_outputs) - - dw0, db0, dw1, db1, dinputs = tf.gradients(loss, [w0, b0, w1, b1, inputs]) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - grad_step = 13 - sym_grads = self.evaluate([dw0, db0, dw1, db1, dinputs]) - sym_grads = [test_utils.PickEveryN(_, grad_step) for _ in sym_grads] - num_grads = [ - test_utils.PickEveryN( - test_utils.ComputeNumericGradient( - sess, loss, v, delta=1e-4, step=grad_step), grad_step) - for v in [w0, b0, w1, b1, inputs] - ] - for (sym, num) in zip(sym_grads, num_grads): - self.assertFalse(np.any(np.isnan(sym))) - self.assertFalse(np.any(np.isnan(num))) - print('max = ', np.max(np.abs(sym))) - self.assertAllClose(sym, num) - - def testBidirectionalFRNNGrad(self): - self._testBidirectionalFRNNGradientHelper() - - def _MultiSourceFRNNWithAttentionInputs(self, - single_source=False, - single_source_length=True, - dtype=tf.float32): - np.random.seed(12345) - if single_source: - src_names = ['en'] - slens = [10] - sdepths = [4] - elif single_source_length: - src_names = ['en1', 'en2', 'de'] - slens = [11, 10, 9] - sdepths = [4, 4, 4] - else: - src_names = ['en1', 'en2', 'de'] - slens = [11, 10, 9] - sdepths = [4, 4, 3] - sbatch = 3 - tlen = 7 - tbatch = 6 - dims = 4 - - src_encs = py_utils.NestedMap() - src_paddings = py_utils.NestedMap() - for sdepth, slen, sname in zip(sdepths, slens, src_names): - src_encs[sname] = tf.constant( - np.random.uniform(size=[slen, sbatch, sdepth]), dtype) - src_paddings[sname] = tf.constant(np.zeros([slen, sbatch]), dtype) - inputs = tf.constant(np.random.uniform(size=(tlen, tbatch, dims)), dtype) - paddings = tf.constant(np.zeros([tlen, tbatch, 1]), dtype) - return (src_encs, src_paddings, inputs, paddings) - - def _MultiSourceFRNNWithAttentionParams(self, - single_source=False, - single_source_length=True, - dtype=tf.float32): - dims = 4 - alt_depth = 3 - if single_source: - src_names = ['en'] - else: - src_names = ['en1', 'en2', 'de'] - - p = rnn_cell.LSTMCellSimple.Params() - p.name = 'lstm' - p.dtype = dtype - p.output_nonlinearity = True - p.params_init = py_utils.WeightInit.Uniform(0.02, 429891685) - p.vn.global_vn = False - p.vn.per_step_vn = False - p.num_input_nodes = dims * 2 - p.num_output_nodes = dims - lstm_params = p - - p = attention.AdditiveAttention.Params() - p.name = 'atten' - p.dtype = dtype - p.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - p.source_dim = dims - p.query_dim = dims - p.hidden_dim = dims - p.vn.global_vn = False - p.vn.per_step_vn = False - attention_tpl = p - - p = attention.MergerLayer.Params() - p.name = 'merger' - p.dtype = dtype - p.merger_op = ('mean' if single_source else 'atten') - p.source_dim = dims - p.query_dim = dims - p.hidden_dim = dims - merger_tpl = p - - p = rnn_layers.MultiSourceFRNNWithAttention.Params() - p.name = 'msrc_frnn_with_atten' - p.dtype = dtype - p.cell = lstm_params - p.attention_tpl = attention_tpl - p.atten_merger = merger_tpl - p.source_names = src_names - - if not single_source_length: - de_atten = attention_tpl.Copy() - de_atten.source_dim = alt_depth - p.source_name_to_attention_params = {'de': de_atten} - merger_tpl.pre_proj_input_dims = [dims, dims, alt_depth] - merger_tpl.pre_proj_output_dims = [dims, dims, dims] - merger_tpl.proj_tpl.batch_norm = False - merger_tpl.proj_tpl.weight_norm = True - - return p - - def testMultiSourceFRNNWithAttention(self): - with self.session(use_gpu=True): - p = self._MultiSourceFRNNWithAttentionParams() - msrc_frnn = p.Instantiate() - - (src_encs, src_paddings, inputs, - paddings) = self._MultiSourceFRNNWithAttentionInputs() - a, m = msrc_frnn.FPropDefaultTheta(src_encs, src_paddings, inputs, - paddings) - msrc_frnn_out = tf.concat([a, m], 2) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - ys = self.evaluate([msrc_frnn_out])[0] - self.assertEqual(ys.shape, (7, 6, 8)) - print(np.sum(ys, axis=(1, 2)), np.sum(ys, axis=(0, 1)), - np.sum(ys, axis=(0, 2))) - # pyformat: disable - # pylint: disable=bad-whitespace - self.assertAllClose( - np.sum(ys, axis=(1, 2)), [ - 11.87568951, 11.8436203 , 11.80368233, 11.80167198, - 11.82034779, 11.80246162, 11.80818748 - ]) - self.assertAllClose( - np.sum(ys, axis=(0, 1)), [ - 21.41802788, 20.86244965, 21.48164749, 19.95701981, - -0.54706949, 0.07046284, -0.50449395, 0.0176318 - ]) - self.assertAllClose( - np.sum(ys, axis=(0, 2)), [ - 13.29822254, 14.01552773, 14.04851151, 13.28098106, - 14.05391502, 14.0585041 - ]) - # pyformat: enable - # pylint: enable=bad-whitespace - - def testMultiSourceFRNNWithAttentionMultiDepth(self): - with self.session(use_gpu=True): - p = self._MultiSourceFRNNWithAttentionParams(single_source_length=False) - msrc_frnn = p.Instantiate() - - (src_encs, src_paddings, inputs, paddings - ) = self._MultiSourceFRNNWithAttentionInputs(single_source_length=False) - a, m = msrc_frnn.FPropDefaultTheta(src_encs, src_paddings, inputs, - paddings) - msrc_frnn_out = tf.concat([a, m], 2) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - ys = self.evaluate([msrc_frnn_out])[0] - self.assertEqual(ys.shape, (7, 6, 8)) - print(np.sum(ys, axis=(1, 2)), np.sum(ys, axis=(0, 1)), - np.sum(ys, axis=(0, 2))) - # pyformat: disable - # pylint: disable=bad-whitespace - self.assertAllClose( - np.sum(ys, axis=(1, 2)), [ - 5.976197, 5.932313, 5.917447, 5.907898, 5.907385, 5.90272 , - 5.890248 - ]) - self.assertAllClose( - np.sum(ys, axis=(0, 1)), [ - 2.635296e+01, 3.177989e+00, 1.024462e+01, 2.403777e+00, - -4.908564e-01, 1.006475e-01, -3.303704e-01, -2.455414e-02 - ]) - self.assertAllClose( - np.sum(ys, axis=(0, 2)), [ - 6.610287, 6.657996, 7.452699, 6.626875, 6.60216 , 7.484191 - ]) - # pyformat: enable - # pylint: enable=bad-whitespace - - def testMultiSourceFRNNWithAttentionSingleSource(self, dtype=tf.float32): - with self.session( - use_gpu=True, config=py_utils.SessionConfig(inline=False)): - p = self._MultiSourceFRNNWithAttentionParams( - single_source=True, dtype=dtype) - frnn = p.Instantiate() - - (src_encs, src_paddings, inputs, - paddings) = self._MultiSourceFRNNWithAttentionInputs( - single_source=True, dtype=dtype) - - a, m = frnn.FPropDefaultTheta(src_encs, src_paddings, inputs, paddings) - frnn_out = tf.concat([a, m], 2) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - ys, = self.evaluate([frnn_out]) - self.assertEqual(ys.shape, (7, 6, 8)) - print(np.sum(ys, axis=(1, 2)), np.sum(ys, axis=(0, 1)), - np.sum(ys, axis=(0, 2))) - - # These values are identical with FRNNWithAttention. - expected_sum12 = [ - 13.07380962, 13.03321552, 12.99956226, 13.00612164, 13.01202011, - 12.99347878, 12.98680687 - ] - expected_sum01 = [ - 2.41238327e+01, 2.11899853e+01, 2.45926647e+01, 2.22827835e+01, - -5.62886238e-01, 2.42760777e-02, -5.79716980e-01, 3.40666063e-02 - ] - expected_sum02 = [ - 12.74695969, 16.13114548, 16.66101837, 12.74922562, 16.16581345, - 16.65085411 - ] - - self.assertAllClose(np.sum(ys, axis=(1, 2)), expected_sum12) - self.assertAllClose(np.sum(ys, axis=(0, 1)), expected_sum01) - self.assertAllClose(np.sum(ys, axis=(0, 2)), expected_sum02) - - def testMultiSourceFRNNWithAttentionGradSingleSource(self, dtype=tf.float64): - with self.session( - use_gpu=True, config=py_utils.SessionConfig(inline=False)) as sess: - - p = self._MultiSourceFRNNWithAttentionParams( - single_source=True, dtype=dtype) - frnn = p.Instantiate() - - (src_encs, src_paddings, inputs, - paddings) = self._MultiSourceFRNNWithAttentionInputs( - single_source=True, dtype=dtype) - - # Fetch all the parameters. - w0, b0 = (frnn.theta.cell.wm, frnn.theta.cell.b) - att0h, att0q, att0s = (frnn.theta.attentions[0].hidden_var, - frnn.theta.attentions[0].query_var, - frnn.theta.attentions[0].source_var) - - out, _ = frnn.FPropDefaultTheta(src_encs, src_paddings, inputs, paddings) - loss = tf.reduce_sum(out) - - parameters = [w0, b0, inputs, att0h, att0q, att0s] - grads = tf.gradients(loss, parameters) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - sym_grads = self.evaluate(grads) - num_grads = [ - test_utils.ComputeNumericGradient(sess, loss, v, delta=1e-5) - for v in parameters - ] - for i, (sym, num) in enumerate(zip(sym_grads, num_grads)): - print([ - i, sym.shape, num.shape, - np.max(np.abs(sym)), - np.max(np.abs(sym - num)), - np.max(np.abs(sym - num) / np.abs(sym)) - ]) - - def Compare(name, sym, num, rtol=1e-5): - print(['name = ', name]) - self.assertFalse(np.any(np.isnan(sym))) - self.assertFalse(np.any(np.isnan(num))) - self.assertAllClose(sym, num, rtol=rtol, atol=1e-8) - - for i, (sym, num) in enumerate(zip(sym_grads, num_grads)): - Compare(parameters[i].name, sym, num) - - def testMultiSourceFRNNWithAttentionGrad(self, dtype=tf.float64): - with self.session( - use_gpu=True, config=py_utils.SessionConfig(inline=False)) as sess: - - p = self._MultiSourceFRNNWithAttentionParams(dtype=dtype) - frnn = p.Instantiate() - - # Fetch all the parameters. - w0, b0 = (frnn.theta.cell.wm, frnn.theta.cell.b) - mh, mq, ms = (frnn.theta.atten_merger.atten.hidden_var, - frnn.theta.atten_merger.atten.query_var, - frnn.theta.atten_merger.atten.source_var) - att0h, att0q, att0s = (frnn.theta.attentions[0].hidden_var, - frnn.theta.attentions[0].query_var, - frnn.theta.attentions[0].source_var) - att1h, att1q, att1s = (frnn.theta.attentions[1].hidden_var, - frnn.theta.attentions[1].query_var, - frnn.theta.attentions[1].source_var) - att2h, att2q, att2s = (frnn.theta.attentions[2].hidden_var, - frnn.theta.attentions[2].query_var, - frnn.theta.attentions[2].source_var) - - (src_encs, src_paddings, inputs, - paddings) = self._MultiSourceFRNNWithAttentionInputs(dtype=dtype) - - out, _ = frnn.FPropDefaultTheta(src_encs, src_paddings, inputs, paddings) - loss = tf.reduce_sum(out) - - parameters = [ - w0, b0, inputs, mh, mq, ms, att0h, att0q, att0s, att1h, att1q, att1s, - att2h, att2q, att2s - ] - grads = tf.gradients(loss, parameters) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - sym_grads = self.evaluate(grads) - num_grads = [ - test_utils.ComputeNumericGradient(sess, loss, v, delta=1e-5) - for v in parameters - ] - for i, (sym, num) in enumerate(zip(sym_grads, num_grads)): - print([ - i, sym.shape, num.shape, - np.max(np.abs(sym)), - np.max(np.abs(sym - num)), - np.max(np.abs(sym - num) / np.abs(sym)) - ]) - - def Compare(name, sym, num, rtol=1e-5): - print(['name = ', name]) - self.assertFalse(np.any(np.isnan(sym))) - self.assertFalse(np.any(np.isnan(num))) - self.assertAllClose(sym, num, rtol=rtol, atol=1e-8) - - for i, (sym, num) in enumerate(zip(sym_grads, num_grads)): - Compare(parameters[i].name, sym, num) - - def testMultiSourceFRNNWithAttentionGradMultiDepth(self, dtype=tf.float64): - with self.session( - use_gpu=True, config=py_utils.SessionConfig(inline=False)) as sess: - - p = self._MultiSourceFRNNWithAttentionParams( - single_source_length=False, dtype=dtype) - frnn = p.Instantiate() - - # Fetch all the parameters. - w0, b0 = (frnn.theta.cell.wm, frnn.theta.cell.b) - mh, mq, ms, mw0, mw1, mw2 = (frnn.theta.atten_merger.atten.hidden_var, - frnn.theta.atten_merger.atten.query_var, - frnn.theta.atten_merger.atten.source_var, - frnn.theta.atten_merger.pre_proj[0].w, - frnn.theta.atten_merger.pre_proj[1].w, - frnn.theta.atten_merger.pre_proj[2].w) - att0h, att0q, att0s = (frnn.theta.attentions[0].hidden_var, - frnn.theta.attentions[0].query_var, - frnn.theta.attentions[0].source_var) - att1h, att1q, att1s = (frnn.theta.attentions[1].hidden_var, - frnn.theta.attentions[1].query_var, - frnn.theta.attentions[1].source_var) - att2h, att2q, att2s = (frnn.theta.attentions[2].hidden_var, - frnn.theta.attentions[2].query_var, - frnn.theta.attentions[2].source_var) - - (src_encs, src_paddings, inputs, - paddings) = self._MultiSourceFRNNWithAttentionInputs( - single_source_length=False, dtype=dtype) - - out, _ = frnn.FPropDefaultTheta(src_encs, src_paddings, inputs, paddings) - loss = tf.reduce_sum(out) - - parameters = [ - w0, b0, inputs, mh, mq, ms, att0h, att0q, att0s, att1h, att1q, att1s, - att2h, att2q, att2s, mw0, mw1, mw2 - ] - grads = tf.gradients(loss, parameters) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - sym_grads = self.evaluate(grads) - num_grads = [ - test_utils.ComputeNumericGradient(sess, loss, v, delta=1e-5) - for v in parameters - ] - for i, (sym, num) in enumerate(zip(sym_grads, num_grads)): - print([ - i, sym.shape, num.shape, - np.max(np.abs(sym)), - np.max(np.abs(sym - num)), - np.max(np.abs(sym - num) / np.abs(sym)) - ]) - - def Compare(name, sym, num, rtol=1e-5): - print(['name = ', name]) - self.assertFalse(np.any(np.isnan(sym))) - self.assertFalse(np.any(np.isnan(num))) - self.assertAllClose(sym, num, rtol=rtol, atol=1e-8) - - for i, (sym, num) in enumerate(zip(sym_grads, num_grads)): - Compare(parameters[i].name, sym, num) - - def _CreateFRNNWithAttentionParams(self, - dtype, - dims, - slen, - sbatch, - tlen, - tbatch, - input_prev_atten_ctx=True, - output_prev_atten_ctx=False): - # Create RNN Layer. - p = rnn_cell.LSTMCellSimple.Params() - p.name = 'lstm' - p.dtype = dtype - p.output_nonlinearity = True - p.params_init = py_utils.WeightInit.Uniform(0.02, 429891685) - p.vn.global_vn = False - p.vn.per_step_vn = False - p.num_input_nodes = dims * 2 if input_prev_atten_ctx else dims - p.num_output_nodes = dims - lstm_params = p - - # Create Attention Layer. - p = attention.AdditiveAttention.Params() - p.name = 'atten' - p.dtype = dtype - p.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - p.source_dim = dims - p.query_dim = dims - p.hidden_dim = dims - p.vn.global_vn = False - p.vn.per_step_vn = False - atten = p - - p = rnn_layers.FRNNWithAttention.Params() - p.name = 'frnn_with_atten' - p.dtype = dtype - p.cell = lstm_params - p.attention = atten - p.input_prev_atten_ctx = input_prev_atten_ctx - p.output_prev_atten_ctx = output_prev_atten_ctx - return p - - @parameterized.parameters((False, False), (False, True), (True, False), - (True, True)) - def testFRNNWithAttentionSeparateSourceContextIdenticalToSourceEnc( - self, input_prev_atten_ctx, output_prev_atten_ctx): - dtype = tf.float32 - dims = 4 - slen = 10 - sbatch = 3 - tlen = 7 - tbatch = 6 - - with self.session(use_gpu=True, config=py_utils.SessionConfig(inline=True)): - np.random.seed(12345) - p = self._CreateFRNNWithAttentionParams( - dtype=dtype, - dims=dims, - slen=slen, - sbatch=sbatch, - tlen=tlen, - tbatch=tbatch, - input_prev_atten_ctx=input_prev_atten_ctx, - output_prev_atten_ctx=output_prev_atten_ctx) - - frnn = p.Instantiate() - - src_encs = tf.constant( - np.random.uniform(size=[slen, sbatch, dims]), dtype) - src_paddings = tf.constant(np.zeros([slen, sbatch]), dtype) - - inputs = tf.constant(np.random.uniform(size=(tlen, tbatch, dims)), dtype) - paddings = tf.constant(np.zeros([tlen, tbatch, 1]), dtype) - - # Run without specifying source context vectors. - atten_ctx, rnn_out, atten_prob, _ = frnn.FPropDefaultTheta( - src_encs, src_paddings, inputs, paddings) - frnn_out = tf.concat([atten_ctx, rnn_out, atten_prob], 2) - - # Run after providing separate source context vectors set to the src_encs - # should provide the same answer. - (atten_ctx_src_ctx, rnn_out_src_ctx, atten_prob_src_ctx, - _) = frnn.FPropDefaultTheta( - src_encs, src_paddings, inputs, paddings, src_contexts=src_encs) - frnn_out_src_ctx = tf.concat( - [atten_ctx_src_ctx, rnn_out_src_ctx, atten_prob_src_ctx], 2) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - frnn_out_v, frnn_out_src_ctx_v = self.evaluate( - [frnn_out, frnn_out_src_ctx]) - - # Expected last dimensions for atten_ctx_src_ctx, rnn_out_src_ctx, - # atten_prob_src_ctx are respectively, (dims, dims, slen). - self.assertEqual(frnn_out_v.shape, (tlen, tbatch, 2 * dims + slen)) - self.assertEqual(frnn_out_src_ctx_v.shape, frnn_out_v.shape) - - self.assertAllClose(frnn_out_v, frnn_out_src_ctx_v) - - @parameterized.parameters((False, False), (False, True), (True, False), - (True, True)) - def testFRNNWithAttentionSeparateSourceContextDifferentFromSourceEnc( - self, input_prev_atten_ctx, output_prev_atten_ctx): - dtype = tf.float32 - dims = 4 - slen = 10 - sbatch = 3 - tlen = 7 - tbatch = 6 - - with self.session(use_gpu=True, config=py_utils.SessionConfig(inline=True)): - np.random.seed(12345) - p = self._CreateFRNNWithAttentionParams( - dtype=dtype, - dims=dims, - slen=slen, - sbatch=sbatch, - tlen=tlen, - tbatch=tbatch, - input_prev_atten_ctx=input_prev_atten_ctx, - output_prev_atten_ctx=output_prev_atten_ctx) - - frnn = p.Instantiate() - - src_encs = tf.constant( - np.random.uniform(size=[slen, sbatch, dims]), dtype) - src_paddings = tf.constant(np.zeros([slen, sbatch]), dtype) - - # We create src_contexts with even dimensions (0, 2) set to all zero, the - # rest are set randomly. - src_contexts = np.random.uniform(size=[slen, sbatch, dims]) - src_contexts[:, :, 0:dims:2] = 0.0 - src_contexts = tf.constant(src_contexts, dtype=dtype) - - inputs = tf.constant(np.random.uniform(size=(tlen, tbatch, dims)), dtype) - paddings = tf.constant(np.zeros([tlen, tbatch, 1]), dtype) - - # Run after providing separate source context vectors set to the src_encs - # should provide the same answer. - atten_ctx, _, _, _ = frnn.FPropDefaultTheta( - src_encs, src_paddings, inputs, paddings, src_contexts=src_contexts) - - # Initialize all the variables, and then run one step. - self.evaluate(tf.global_variables_initializer()) - atten_ctx_v = self.evaluate(atten_ctx) - - self.assertEqual(atten_ctx_v.shape, (tlen, tbatch, dims)) - # Verify that the output also has zeros in the locations that the - # source context has zeros. - self.assertAllClose( - np.zeros(shape=(tlen, tbatch, dims // 2)), - atten_ctx_v[:, :, 0:dims:2]) - - def _testFRNNWithAttentionUseZeroAttenState(self, zero_atten_state_fn): - dtype = tf.float32 - dims = 5 - slen = 4 - tlen = 3 - sbatch = 2 - tbatch = 6 - - with self.session(use_gpu=True): - p = self._CreateFRNNWithAttentionParams( - dtype=dtype, - dims=dims, - slen=slen, - sbatch=sbatch, - tlen=tlen, - tbatch=tbatch) - p.use_zero_atten_state = True - p.atten_context_dim = dims - frnn = p.Instantiate() - - # Override the ZeroAttentionState to have the desired output type - frnn.atten.ZeroAttentionState = types.MethodType(zero_atten_state_fn, - frnn.atten) - - src_encs = tf.constant( - np.random.uniform(size=[slen, sbatch, dims]), dtype) - src_paddings = tf.constant(np.zeros([slen, sbatch]), dtype) - - inputs = tf.constant(np.random.uniform(size=(tlen, tbatch, dims)), dtype) - paddings = tf.constant(np.zeros([tlen, tbatch, 1]), dtype) - - atten_ctx, rnn_out, atten_prob, _ = frnn.FPropDefaultTheta( - src_encs, src_paddings, inputs, paddings) - - self.evaluate(tf.global_variables_initializer()) - atten_ctx, rnn_out, atten_prob = self.evaluate( - [atten_ctx, rnn_out, atten_prob]) - - # Check shapes - self.assertEqual(atten_ctx.shape, (tlen, tbatch, dims)) - self.assertEqual(rnn_out.shape, (tlen, tbatch, dims)) - self.assertEqual(atten_prob.shape, (tlen, tbatch, slen)) - - def testFRNNWithAttentionUseZeroAttenStateTensor(self): - - def _TensorZeroAttenState(self, source_seq_length, decoder_batch_size): - del source_seq_length - p = self.params - zs = tf.zeros([decoder_batch_size, 1], dtype=py_utils.FPropDtype(p)) - return zs - - self._testFRNNWithAttentionUseZeroAttenState(_TensorZeroAttenState) - - def testFRNNWithAttentionUseZeroAttenStateNestedMap(self): - - def _NestedMapZeroAttenState(self, source_seq_length, decoder_batch_size): - del source_seq_length - p = self.params - zs = tf.zeros([decoder_batch_size, 1], dtype=py_utils.FPropDtype(p)) - return py_utils.NestedMap(z=zs) - - self._testFRNNWithAttentionUseZeroAttenState(_NestedMapZeroAttenState) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/saver.py b/lingvo/core/saver.py deleted file mode 100644 index 9e16042f..00000000 --- a/lingvo/core/saver.py +++ /dev/null @@ -1,333 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""A saver that we use to save and restore variables. - -The implementation mimics tf.train.Saver. Meanwhile, it allows us -to carry out extra sanity checks on the checkpoint. -""" - -import re -import time -from lingvo import compat as tf -# pylint: enable=g-direct-tensorflow-import -from lingvo.core import py_utils -import numpy as np -from google.protobuf import text_format -# pylint: disable=g-direct-tensorflow-import -from tensorflow.python.lib.io import file_io -from tensorflow.python.ops import io_ops -from tensorflow.python.training.checkpoint_state_pb2 import CheckpointState - - -class SanityCheck: - - def Check(self, *args): - """Returns true iff the sanity check passes.""" - raise NotImplementedError() - - -class InRange(SanityCheck): - """Sanity check a value is within [low, high].""" - - def __init__(self, low, high): - self._low = low - self._high = high - - def __str__(self): - return "InRange({}, {})".format(self._low, self._high) - - def Check(self, value): - return self._low <= value <= self._high - - -class IsFinite(SanityCheck): - - def __str__(self): - return "IsFinite" - - def Check(self, value): - return np.all(np.isfinite(value)) - - -def _VarKey(var): - return var.name[:-2] # strip :0 - - -class Saver: - """Simpler version of tf.train.Saver with extra sanity checks.""" - - def __init__(self, - logdir, - variables, - sanity_checks=None, - keep_latest_n=None, - keep_every_n_hours=None): - self._logdir = logdir - self._state_file = "{}/checkpoint".format(self._logdir) - self._vars = variables - assert not sanity_checks or all( - isinstance(x[1], SanityCheck) for x in sanity_checks) - self._sanity_checks = sanity_checks - self._keep_latest_n = keep_latest_n - self._keep_every_n_hours = keep_every_n_hours - self._re_pattern = re.compile(r"^.*/ckpt-(\d+).*$") - self._logdir_ph = tf.placeholder(tf.string, shape=[]) - self._restore_prefix_ph = tf.placeholder(tf.string, shape=[]) - self._BuildSave() - self._BuildRestore() - tf.logging.info("Saver: %s %s %s", self._logdir, self._keep_latest_n, - self._keep_every_n_hours) - - def _BuildSave(self): - """Builds save ops.""" - self._save_global_step = py_utils.GetGlobalStep() - self._save_prefix = tf.strings.join([ - self._logdir_ph, "/ckpt-", - tf.as_string(self._save_global_step, width=8, fill="0") - ]) - self._save_op = io_ops.save_v2( - prefix=self._save_prefix, - tensor_names=[_VarKey(v) for v in self._vars], - tensors=[v.read_value() for v in self._vars], - shape_and_slices=[""] * len(self._vars)) - - def _BuildRestore(self): - """Builds restore ops.""" - assign_ops = [] - for var in self._vars: - val, = io_ops.restore_v2( - prefix=self._restore_prefix_ph, - tensor_names=[_VarKey(var)], - shape_and_slices=[""], - dtypes=[var.dtype]) - assign_ops.append(var.assign(val)) - self._restore_op = tf.group(*assign_ops) - - def _GetState(self): - """Returns the latest checkpoint id.""" - state = CheckpointState() - if file_io.file_exists(self._state_file): - content = file_io.read_file_to_string(self._state_file) - text_format.Parse(content, state) - return state - - def _SetState(self, state): - file_io.atomic_write_string_to_file(self._state_file, - text_format.MessageToString(state)) - - def _GetCheckpointId(self, filename_prefix): - match = self._re_pattern.match(filename_prefix) - assert match, "Unexpected {} does not match re({})".format( - filename_prefix, self._re_pattern.pattern) - return int(match.group(1)) - - def _GarbageCollect(self): - """Garbage collect obsolete checkpoint files.""" - state = self._GetState() - - valid_ids = set() - if state.model_checkpoint_path: - valid_ids.add(self._GetCheckpointId(state.model_checkpoint_path)) - for path in state.all_model_checkpoint_paths: - valid_ids.add(self._GetCheckpointId(path)) - - existing_files = tf.io.gfile.glob(r"{}/ckpt-*".format(self._logdir)) - # Filter to make sure we catch only the ckpt files. - existing_files = [f for f in existing_files if self._re_pattern.match(f)] - for filename in existing_files: - if self._GetCheckpointId(filename) not in valid_ids: - # TODO(zhifengc): May need to find a bulk delete method. - tf.logging.info("Garbage collecting %s", filename) - tf.io.gfile.remove(filename) - - def _DoSanityCheck(self, prefix): - """Sanity-check the content of the checkpoint.""" - if not self._sanity_checks: - return - reader = tf.train.NewCheckpointReader(prefix) - content = {} - for variables, rule in self._sanity_checks: - args = [] - for v in variables: - key = _VarKey(v) - if key in content: - args.append(content[key]) - else: - value = reader.get_tensor(key) - content[key] = value - args.append(value) - if not rule.Check(*args): - # TODO(zhifengc): Maybe should return an explicit signal - # so that the caller (the controller loop) can Restore() - # the latest checkpoint before raise the error. - msg = "Checkpoint sanity check failed: {} {} {}\n".format( - prefix, ",".join([_VarKey(v) for v in variables]), rule) - # Also saves the error messge into a file. - file_io.write_string_to_file("{}.failed".format(prefix), msg) - raise tf.errors.AbortedError(None, None, msg) - - def Save(self, sess): - """Generate a new checkpoint. - - Args: - sess: A session with tf.Graph under which this object is constructred. - - Returns: - If the checkpoint is successfully generated, returns its global step - and file prefix. Otherwise, raises an Aborted error. - """ - # Garbage collect. Do so before generates the checkpoint - # in case we repeatedly fails the sanity checks. - self._GarbageCollect() - - _, global_step, prefix = sess.run( - fetches=[self._save_op, self._save_global_step, self._save_prefix], - feed_dict={self._logdir_ph: self._logdir}) - prefix = tf.compat.as_text(prefix) - - # Many users expect this as the tf.train.Saver does this by default. - meta_graph_filename = prefix + ".meta" - tf.train.export_meta_graph(filename=meta_graph_filename) - - # We can do extra sanity checks. - self._DoSanityCheck(prefix) - - # Commit new state. - self._UpdateState(prefix) - - tf.logging.info("Saved %d %s", global_step, prefix) - return global_step, prefix - - def _UpdateState(self, prefix): - """Updates the checkpoint state with the new checkpoint prefix.""" - # The checkpoint looks OK. Commit it to the state. - state = self._GetState() - - # If a previous checkpoint exists and it was generated long - # after the checkpoint before it, we preserve it in - # all_model_checkpoint_{paths,timestamps}. - if state.model_checkpoint_path and ( - not state.all_model_checkpoint_timestamps or - not self._keep_every_n_hours or - (state.last_preserved_timestamp - - state.all_model_checkpoint_timestamps[-1] > - 3600. * self._keep_every_n_hours)): - state.all_model_checkpoint_paths.append(state.model_checkpoint_path) - state.all_model_checkpoint_timestamps.append( - state.last_preserved_timestamp) - - # Record the checkpoint we just generated. - state.model_checkpoint_path = prefix - state.last_preserved_timestamp = time.time() - - # Applies the count-based GC policy. - if self._keep_latest_n: - if self._keep_latest_n == 1: - # Use [:] to prevent "Assignment not allowed to repeated field" error. - state.all_model_checkpoint_paths[:] = [] - state.all_model_checkpoint_timestamps[:] = [] - else: - n = self._keep_latest_n - 1 - state.all_model_checkpoint_paths[:] = ( - state.all_model_checkpoint_paths[-n:]) - state.all_model_checkpoint_timestamps[:] = ( - state.all_model_checkpoint_timestamps[-n:]) - self._SetState(state) - - def Restore(self, sess, checkpoint_id=None): - """Restore variables from a checkpoint. - - Args: - sess: A session with tf.Graph under which this object is constructred. - checkpoint_id: If None, restore from the latest checkpoint. Otherwise, - restore from the specific checkpoint. - - Returns: - If no checkpoint has been generated, returns (0, ""). Otherwise, try to - restore from the specified or the latest checkpoint. If the checkpoint is - successfully restored, returns the checkpoint's global step and file - prefix. Otherwise, raises an error. - """ - - if checkpoint_id: - prefix = "{}/ckpt-{:08d}".format(self._logdir, checkpoint_id) - else: - prefix = self._GetState().model_checkpoint_path - if not prefix: - return 0, "" - - sess.run( - fetches=[self._restore_op], feed_dict={self._restore_prefix_ph: prefix}) - global_step = self._GetCheckpointId(prefix) - tf.logging.info("Restored %d %s", global_step, prefix) - return global_step, prefix - - -def WriteNpArrays(file_prefix, nmap): - """Writes a NestedMap of numpy arrays into a TF checkpoint. - - Args: - file_prefix: A TF checkpoint filename prefix. - nmap: A NestedMap of numpy arrays. - """ - g = tf.Graph() - with g.as_default(): - - def Wrap(val): - dtype = tf.as_dtype(val.dtype) - assert dtype != tf.string # tf.string is not supported by py_func. - return tf.py_func(lambda: val, [], dtype) - - names, values = [], [] - for k, v in nmap.FlattenItems(): - names.append(k) - assert isinstance(v, np.ndarray) - values.append(Wrap(v)) - - save = io_ops.save_v2( - prefix=file_prefix, - tensor_names=names, - tensors=values, - shape_and_slices=[""] * len(names)) - - with tf.Session(graph=g) as sess: - sess.run(save) - - -def ReadNpArrays(file_prefix, nmap): - """Reads from a tf checkpoint to fill in values of a NesteMap. - - Args: - file_prefix: A TF checkpoint filename prefix. - nmap: A NestedMap of numpy dtypes. - - Returns: - A NestedMap with numpy arrays compatible w/ nmap. - """ - g = tf.Graph() - with g.as_default(): - reads = [] - for name, dtype in nmap.FlattenItems(): - reads.append( - io_ops.restore_v2( - prefix=file_prefix, - tensor_names=[name], - shape_and_slices=[""], - dtypes=[dtype])[0]) - - with tf.Session(graph=g) as sess: - vals = sess.run(reads) - - return nmap.Pack(vals) diff --git a/lingvo/core/saver_test.py b/lingvo/core/saver_test.py deleted file mode 100644 index a61d6822..00000000 --- a/lingvo/core/saver_test.py +++ /dev/null @@ -1,123 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for saver.""" - -import os -import tempfile -from lingvo import compat as tf -from lingvo.core import cluster_factory -from lingvo.core import py_utils -from lingvo.core import saver -from lingvo.core import test_utils -from lingvo.tasks.image.params import mnist -import numpy as np - - -class SaverTest(test_utils.TestCase): - - def testBasic(self): - logdir = tempfile.mkdtemp() - # Create a dummy file that looks like a checkpoint that shouldn't - # be touched. - with tf.io.gfile.GFile(logdir + '/ckpt-foo', 'w') as f: - f.write('contents') - - g = tf.Graph() - with g.as_default(): - p = mnist.LeNet5().Task() - p.input = mnist.LeNet5().Train() - with cluster_factory.ForTestingWorker(mode='sync', job='controller'): - _ = p.Instantiate() - gsv = py_utils.GetOrCreateGlobalStepVar() - inc = gsv.assign_add(1) - variables = tf.all_variables() - sanity_checks = [([gsv], saver.InRange(0, 10))] - for var in variables: - sanity_checks.append(([var], saver.IsFinite())) - sav = saver.Saver( - logdir, - variables, - sanity_checks, - keep_latest_n=5, - keep_every_n_hours=1e-9) - - with self.session(graph=g) as sess: - # Creates a few checkpoints. - sess.run(tf.global_variables_initializer()) - for _ in range(10): - sess.run(inc) - _ = sav.Save(sess) - - # Restore to the latest. - sess.run(tf.global_variables_initializer()) - _ = sav.Restore(sess) - - # Restore to a specific checkpoint. - sess.run(tf.global_variables_initializer()) - _ = sav.Restore(sess, 6) - - # Increments global_step out of range, Save() fails. - for _ in range(5): - sess.run(inc) - with self.assertRaises(tf.errors.AbortedError): - _ = sav.Save(sess) - - filenames = tf.io.gfile.glob('{}/*'.format(logdir)) - filenames = [x[len(logdir) + 1:] for x in filenames] - print('\n'.join(filenames)) - self.assertIn('checkpoint', filenames) - - meta_files = [] - for f in filenames: - if f.endswith('.meta'): - meta_files.append(f) - # A .meta for each checkpoint. - self.assertEqual(len(meta_files), 6) - - # 1 for checkpoint. 3 files per checkpoint. 5 good checkpoints, 1 bad. - # 1 extra file contains the error message, and 1 dummy file - self.assertEqual(len(filenames), 1 + (5 + 1) * 3 + 1 + 1) - - def testSingleCheckpoint(self): - logdir = tempfile.mkdtemp() - g = tf.Graph() - with g.as_default(): - _ = py_utils.GetOrCreateGlobalStepVar() - sav = saver.Saver(logdir, tf.all_variables(), [], keep_latest_n=1) - with self.session(graph=g) as sess: - sess.run(tf.global_variables_initializer()) - _ = sav.Save(sess) - - def testWriteReadNpArrays(self): - prefix = os.path.join(tempfile.mkdtemp(), 'nptest') - nmap = py_utils.NestedMap() - nmap.train = np.random.normal(size=(3, 3)) - nmap.test = np.random.normal(size=(1, 3)) - nmap.foo = py_utils.NestedMap() - nmap.foo.bar = np.arange(10).astype(np.int32).reshape([2, 5]) - saver.WriteNpArrays(prefix, nmap) - files = tf.io.gfile.glob(prefix + '*') - self.assertEqual(len(files), 2) - self.assertEqual(files[0], prefix + '.data-00000-of-00001') - self.assertEqual(files[1], prefix + '.index') - read_nmap = saver.ReadNpArrays(prefix, nmap.Transform(lambda x: x.dtype)) - self.assertTrue(nmap.IsCompatible(read_nmap)) - self.assertAllEqual(nmap.train, read_nmap.train) - self.assertAllEqual(nmap.test, read_nmap.test) - self.assertAllEqual(nmap.foo.bar, read_nmap.foo.bar) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/schedule.py b/lingvo/core/schedule.py deleted file mode 100644 index 6a14cd71..00000000 --- a/lingvo/core/schedule.py +++ /dev/null @@ -1,824 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Learning rate schedule.""" - -import math -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import early_stop -from lingvo.core import ops -from lingvo.core import py_utils - - -class BaseSchedule(base_layer.BaseLayer): - """Base class for learning rate decay algorithms.""" - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'LRSched' - return p - - def __init__(self, params): - super().__init__(params) - self.SetVariableFree() - - def Value(self, current_step=None): - """Returns the current learning rate schedule value. - - Args: - current_step: The current global step value. - - Returns: - Returns the current learning rate schedule value given the - current training global step. Typically, the base learning rate - multiplied by the returned schedule value is used as the - effective learning rate. - """ - if current_step is None: - current_step = self.global_step - return self.FPropDefaultTheta(current_step) - - -class Constant(BaseSchedule): - """A schedule that always returns a constant value.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('value', 1., 'The constant value.') - return p - - def FProp(self, theta, current_step): - del theta, current_step - return tf.constant(self.params.value, self.params.dtype) - - -class ConstantOne(Constant): - """A lr schedule remains constant 1.""" - pass - - -class PiecewiseConstantSchedule(BaseSchedule): - """Piecewise constants rate decay.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('boundaries', None, 'Boundaries at which learning rate drops.') - p.Define('values', None, 'Values in each interval.') - return p - - def FProp(self, theta, current_step): - p = self.params - return py_utils.PiecewiseConstant(current_step, p.boundaries, p.values, - p.dtype) - - -class ContinuousSchedule(BaseSchedule): - """Continuous learning rate decay.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('initial_value', 1.0, 'Initial decay value.') - p.Define('start_step', 400000, - 'Starts to decay the learning rate from this step.') - p.Define('half_life_steps', 100000, - 'Halve the learning rate every this many steps after start_step.') - p.Define('min', 0.01, 'Minimum relative learning rate.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - q = ExponentialSchedule.Params().Set( - start=(p.start_step, 1.0), - limit=(p.start_step + - p.half_life_steps * math.log(p.min) / math.log(0.5), p.min)) - self.CreateChild('exp', q) - - def FProp(self, theta, current_step): - """Returns the current learning rate decay.""" - return self.params.initial_value * self.exp.Value(current_step) - - -class PolynomialSchedule(BaseSchedule): - """Polynomial learning rates. - - If x < x0, returns y0. If x >= x1, returns y1. Otherwise, - interpolate with a polynomial between (x0, y0) and (x1, y1). - - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('power', 1, 'Polynomial power.') - p.Define('start', (0, 1.), '(x0, y0)') - p.Define('limit', (1, 1.), '(x1, y1)') - p.Define('origin', 'start', 'Origin of the polynomial. Can be "start" or ' - '"limit".') - return p - - def __init__(self, params): - super().__init__(params) - - def Polynomial(x): - """Polynomial function of x.""" - p = self.params - x0, y0 = p.start - x1, y1 = p.limit - - assert x0 < x1, '%s must be < %s' % (x0, x1) - - x0 = tf.cast(x0, dtype=x.dtype) - x1 = tf.cast(x1, dtype=x.dtype) - y0 = tf.cast(y0, dtype=x.dtype) - y1 = tf.cast(y1, dtype=x.dtype) - - ratio = (x - x0) / (x1 - x0) - if p.origin == 'start': - f_x = ratio**p.power - elif p.origin == 'limit': - f_x = 1 - (1 - ratio)**p.power - else: - raise ValueError('Invalid parameter origin: %s' % p.origin) - y = y0 + f_x * (y1 - y0) - return tf.where(x < x0, y0, tf.where(x >= x1, y1, y)) - - self._polynomial = Polynomial - - def FProp(self, theta, current_step): - return py_utils.CallDefun(self._polynomial, - tf.cast(current_step, dtype=self.params.dtype)) - - -class LinearSchedule(PolynomialSchedule): - """Linear learning rate schedule. - - If x < x0, returns y0. If x >= x1, returns y1. Otherwise, - interpolate linearly between (x0, y0) and (x1, y1). - - """ - - @classmethod - def Params(cls): - p = super().Params().Set(power=1) - return p - - -class ExponentialSchedule(BaseSchedule): - """Linear learning rate schedule. - - If x < x0, returns y0. If x >= x1, returns y1. Otherwise, - interpolate exponentially between (x0, y0) and (x1, y1). - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('start', (0, 1.), '(x0, y0)') - p.Define('limit', (1, 0.5), '(x1, y1)') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - x0, y0 = p.start - x1, y1 = p.limit - assert x0 < x1, '%s must be < %s' % (x0, x1) - assert y0 > 0, '%s must be > 0' % y0 - assert y1 > 0, '%s must be > 0' % y1 - - self.CreateChild( - 'linear', - LinearSchedule.Params().Set( - start=(x0, math.log(y0)), limit=(x1, math.log(y1)))) - - def Exp(x): - return tf.exp(self.linear.Value(x)) - - self._exp = Exp - - def FProp(self, theta, current_step): - return py_utils.CallDefun(self._exp, - tf.cast(current_step, dtype=self.params.dtype)) - - -class StepwiseExponentialSchedule(BaseSchedule): - """Exponential decay every N steps.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('decay', 0.99, 'Decay factor.') - p.Define('num_steps_per_decay', 1000, 'Number of steps between decays.') - return p - - def FProp(self, theta, current_step): - p = self.params - num_decays = tf.floor( - tf.div(tf.cast(current_step, tf.float32), float(p.num_steps_per_decay))) - return tf.pow(p.decay, num_decays) - - -class CombinedMinimumSchedule(BaseSchedule): - """Combine a few learning rate decay schedules and takes the min.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('schedules', [LinearSchedule.Params()], - 'A list of learning rate schedule params.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self.CreateChildren('schedules', p.schedules) - - def Combined(x): - ys = [s.Value(x) for s in self.schedules] - return tf.reduce_min(tf.stack(ys), axis=0) - - self._combined = Combined - - def FProp(self, theta, current_step): - return py_utils.CallDefun(self._combined, - tf.convert_to_tensor(current_step)) - - -class TransformerSchedule(BaseSchedule): - """Inverse-decay learning rate until warmup_steps, then decay.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'warmup_steps', 4000, 'Increase the learning rate linearly for ' - 'the first warmup_steps training steps.') - p.Define( - 'model_dim', 512, 'Model dimension that applies to embedding ' - 'layers and all Transformer layers.') - p.Define('worker_replicas', 1, 'Number of worker replicas.') - p.Define('decay_end', None, 'Ends the learning rate decay at ' - 'decay_end-th step.') - return p - - def FProp(self, theta, current_step): - """Returns the current learning rate decay.""" - p = self.params - current_step = tf.cast(current_step, tf.float32) - warmup_steps = tf.cast(p.warmup_steps * p.worker_replicas, tf.float32) - if p.decay_end is not None: - current_step = tf.where(current_step < p.decay_end, current_step, - tf.cast(p.decay_end, tf.float32)) - return p.model_dim**-0.5 * tf.minimum( - (current_step + 1) * warmup_steps**-1.5, (current_step + 1)**-0.5) - - -class TransformerMLPerfSchedule(BaseSchedule): - """learning rate with linear warmup, then rsqrt decay.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'warmup_steps', 4000, 'Increase the learning rate linearly for ' - 'the first warmup_steps training steps.') - p.Define( - 'model_dim', 512, 'Model dimension that applies to embedding ' - 'layers and all Transformer layers.') - return p - - def FProp(self, theta, current_step): - """Returns the current learning rate decay.""" - p = self.params - current_step = tf.cast(current_step, tf.float32) - warmup_steps = tf.cast(p.warmup_steps, tf.float32) - linear_warmup = tf.minimum(1.0, current_step / warmup_steps) - rsqrt_decay = tf.math.rsqrt(tf.maximum(current_step, warmup_steps)) - return p.model_dim**-0.5 * linear_warmup * rsqrt_decay - - -class TransformerScheduleNoWarmUp(BaseSchedule): - """Fixed learning rate until decay_start, then decay. - - This learning rate schedule is identical to TransformerSchedule - except in the warm-up phase, where this learning rate schedule uses a fixed - learning rate (peak-learning rate of TransformerSchedule) for the - original warm-up phase. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('decay_start', 4000, 'It is used to estimate peak-lr.') - p.Define('decay_end', None, 'Ends the learning rate decay at ' - 'decay_end-th step.') - p.Define( - 'model_dim', 512, 'Model dimension that applies to embedding ' - 'layers and all Transformer layers.') - p.Define('worker_replicas', 1, 'Number of worker replicas.') - return p - - def __init__(self, params): - super().__init__(params) - tf.logging.info('Peak lr: %f', (self.params.decay_start * - self.params.worker_replicas)**-0.5) - - def FProp(self, theta, current_step): - """Returns the current learning rate decay.""" - params = self.params - warmup_steps = tf.cast(params.decay_start * params.worker_replicas, - tf.float32) - current_step = tf.cast(current_step, tf.float32) - if params.decay_end is not None: - current_step = tf.where(current_step < params.decay_end, current_step, - tf.cast(params.decay_end, tf.float32)) - peak_learning_rate = (warmup_steps**-0.5) - return (params.model_dim**-0.5) * tf.minimum( - tf.minimum((current_step + 1), - (current_step + 1)**-0.5), peak_learning_rate) - - -class LinearRampupExponentialDecayScaledByNumSplitSchedule(BaseSchedule): - """A learning rate schedule that does the following. - - 1. The peak learning rate multiplier is scaled by num splits, - (often the same as #replicas during batch splitting synchronous - training). - 2. The multiplier ramps up linearly from 1 to the peak initially. - 3. The multiplier stays constant until the exponential decay starts. - 4. The multiplier is capped at max. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'warmup', 300, 'Increases the learning rate linearly ' - 'before warmup * num_splits steps.') - p.Define('warmup_init', 1.0, 'The initial value of the warm-up phase.') - p.Define('decay_start', 70000, 'Starts the learning rate decay at ' - 'decay_start-th step.') - p.Define('decay_end', 100000, 'Ends the learning rate decay at ' - 'decay_end-th step.') - p.Define('min', 0.01, 'After decay_end, the multiplier stays at min.') - p.Define( - 'max', 1e8, 'The schedule is never larger than this value. ' - 'By default, 1e8 effectively means there is no cap.') - p.Define( - 'num_splits', 0, 'Specifies the intended number of splits for the ' - 'LR. Overrides num_splits_per_client if non-zero.') - return p - - def __init__(self, params): - super().__init__(params) - - p = self.params - - # We always compute lr schedule from the trainer's perspective. - # Also note that this schedule makes sense to sync training only. - if p.num_splits: - splits = p.num_splits - else: - # Infer num_splits from cluster. - cluster_params = self.cluster.params.Copy() - cluster_params.task = 0 - assert cluster_params.mode == 'sync' - cluster_params.job = 'trainer_client' - my_cluster = cluster_params.Instantiate() - splits = my_cluster.num_splits_per_client - - warmup_end = p.warmup * splits - decay_start = max(warmup_end + 1.0, p.decay_start / splits) - peak = 1.0 * splits - tf.logging.info('Peak lr: %f', peak) - decay_end = max(decay_start + 1.0, p.decay_end / splits) - schedules = [ - LinearSchedule.Params().Set( - start=(warmup_end, peak), limit=(decay_start, peak)), - ExponentialSchedule.Params().Set( - start=(decay_start, peak), limit=(decay_end, p.min)), - LinearSchedule.Params().Set(start=(0, p.max), limit=(decay_end, p.max)), - ] - # Only include a warm up schedule if the warmup_end exceeds 0.0. Note that - # linear schedules must have x1 > x0 strictly. - if warmup_end > 0.0: - schedules = [ - LinearSchedule.Params().Set( - start=(0., p.warmup_init), limit=(warmup_end, peak)) - ] + schedules - self.CreateChild('combine', - CombinedMinimumSchedule.Params().Set(schedules=schedules)) - - def FProp(self, theta, current_step): - return self.combine.Value(current_step) - - -class LinearRampupExponentialDecay( - LinearRampupExponentialDecayScaledByNumSplitSchedule): - """A schedule that does the following... - - 1. Linearly ramps up from `p.warmup_init` to 1.0 initially; - 2. Stays at constant 1.0 until the decay starts; - 3. Exponential decays from 1.0 to `p.min`. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.num_splits = 1 - p.warmup_init = 0.0 - return p - - def __init__(self, params): - assert params.num_splits == 1 - super().__init__(params) - - -class LinearRampupSqrtDecayByBatchSizeAndReplicas(BaseSchedule): - """Linearly increase learning rate until warmup_examples, then sqrt decay. - - Same as the Transformer learning schedule, except that learning rate - multiplier and time scales are adjusted according to the aggregate batch size - (batch_size * num_replicas). - - The implied peak learning rate multiplier given by this schedule is: - sqrt(batch_size * num_replicas / warmup_examples). - - Can only be used with synchronized trainers. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'warmup_examples', 256 * 2**20, - 'Increase the learning rate linearly for the first warmup_examples ' - 'training examples, and then starts square-root decay.') - p.Define('batch_size', None, - 'Norminal (per-replica) batch size. Must be provided.') - p.Define( - 'num_replicas', None, 'Number of worker replicas. If None, ' - 'determined automatically (and error if this fails).') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.batch_size > 0 - if p.num_replicas: - self._num_replicas = p.num_replicas - else: - # Infer from cluster. - cluster_params = self.cluster.params.Copy() - cluster_params.task = 0 - assert cluster_params.mode == 'sync' - cluster_params.job = 'trainer_client' - my_cluster = cluster_params.Instantiate() - self._num_replicas = my_cluster.num_splits_per_client - assert self._num_replicas > 0 - - def FProp(self, theta, current_step): - """Returns the current learning rate decay.""" - p = self.params - current_step = tf.cast(current_step, tf.float32) - warmup_steps = tf.cast( - p.warmup_examples / (p.batch_size * self._num_replicas), tf.float32) - return tf.minimum((current_step + 1) * warmup_steps**-1.5, - (current_step + 1)**-0.5) - - -class LinearRampupPiecewiseConstantSchedule(BaseSchedule): - """A learning rate schedule that does the following. - - 1. The learning rate is scaled by #split * lrs[i] - (often #split is the same as #replicas during batch splitting synchronous - training). - 2. The multiplier ramps up linearly from 0 to the peak(lrs[0]) at - boundaries[0]. - 3. After peak, the multiplier stays lrs[i] when step falls into - [boundaries[i], boundaries[i+1]) - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('boundaries', [], 'Boundaries at which learning rate changes.') - p.Define('lrs', [], 'A list of learning rate multiplers.') - p.Define( - 'num_splits', 0, 'Specifies the intended number of num_splits for ' - 'LR. Overrides num_splits if non-zero.') - return p - - def __init__(self, params): - super().__init__(params) - - p = self.params - assert len(p.boundaries) >= 2 and len(p.boundaries) == len(p.lrs) - # We always compute lr schedule from the trainer's perspective. - # Also note that this schedule makes sense to sync training only. - if p.num_splits: - splits = p.num_splits - else: - # Infer num_splits from cluster. - cluster_params = self.cluster.params.Copy() - cluster_params.task = 0 - assert cluster_params.mode == 'sync' - cluster_params.job = 'trainer_client' - my_cluster = cluster_params.Instantiate() - splits = my_cluster.num_splits_per_client - - assert splits >= 1 - splits = float(splits) - boundaries = [step / splits for step in p.boundaries] - lrs = [step * splits for step in p.lrs] - - tf.logging.info('splits: {}\n boundaries: {}\n lrs: {} '.format( - splits, boundaries, lrs)) - - schedules = [ - LinearSchedule.Params().Set( - start=(0., 0.), limit=(boundaries[0], lrs[0])), - PiecewiseConstantSchedule.Params().Set( - boundaries=boundaries, values=[1e8] + lrs) - ] - self.CreateChild('combine', - CombinedMinimumSchedule.Params().Set(schedules=schedules)) - - def FProp(self, theta, current_step): - return self.combine.Value(current_step) - - -class LinearRampupCosineSchedule(BaseSchedule): - """A cosine decaying learning rate schedule with a linear rampup phase.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('warmup_init', 0, 'The initial lr value of the warm-up phase.') - p.Define('warmup_steps', 0, 'Number of warm up steps.') - p.Define('initial_value', 1.0, 'Initial decay value.') - p.Define('final_value', 0., 'Final decay value.') - p.Define('total_steps', 0, 'Number of steps to reach full decay.') - p.Define( - 'num_splits', 1, 'Specifies the intended number of splits for the ' - 'LR. Overrides num_splits_per_client if non-zero. Uses ' - 'num_splits_per_client as num_splits if zero or negative.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - # We always compute lr schedule from the trainer's perspective. - # Also note that this schedule makes sense to sync training only. - if p.num_splits > 0: - splits = p.num_splits - else: - # Infer num_splits from cluster. - cluster_params = self.cluster.params.Copy() - cluster_params.task = 0 - assert cluster_params.mode == 'sync' - cluster_params.job = 'trainer_client' - my_cluster = cluster_params.Instantiate() - splits = my_cluster.num_splits_per_client - - schedules = [ - LinearSchedule.Params().Set( - start=(0., p.warmup_init * splits), - limit=(p.warmup_steps // splits, p.initial_value * splits)), - CosineSchedule.Params().Set( - initial_value=p.initial_value * splits, - final_value=p.final_value * splits, - total_steps=p.total_steps // splits), - ] - self.CreateChild('combine', - CombinedMinimumSchedule.Params().Set(schedules=schedules)) - - def FProp(self, theta, current_step): - return self.combine.Value(current_step) - - -class DevBasedSchedule(BaseSchedule): - """Decay triggered by lack of improvement on the dev set. - - This reads a file containing a history of values of a selected metric versus - global step (file is recorded by the evaler loop in the trainer). Decay - depends on these variables: - - - best_step - step at which optimum metric value occurred in history file - - last_step - last step recorded in history file - - ref_step - most recent decay step or best_step - - cur_factor - current multiplier on initial learning rate - - The decay algorithm is:: - - ref_step = max(ref_step, best_step) - if last_step - ref_step > window: - cur_factor = max(cur_factor * decay, min_factor) - ref_step = last_step - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('metric_history', early_stop.MetricHistory.Params(), - 'Metric to monitor for stopping.') - p.Define('tolerance', 0.0, 'Minimum significant difference in metric.') - p.Define('window', 10000, - 'Steps since most recent decay or best_step before decaying.') - p.Define('decay', 0.5, - 'Factor by which learning rate multiplier is decayed.') - p.Define('min_factor', 0.01, 'Minimum learning rate multiplier.') - - return p - - def __init__(self, params): - super().__init__(params) - - p = self.params - - with tf.variable_scope(p.name): - wp = py_utils.WeightParams( - shape=[], - init=py_utils.WeightInit.Constant(1.0), - collections=['DevBasedSchedule_vars'], - dtype=tf.float32) - self._cur_factor = py_utils.CreateVariable( - 'cur_factor', wp, trainable=False) - wp = py_utils.WeightParams( - shape=[], - init=py_utils.WeightInit.Constant(0), - collections=['DevBasedSchedule_vars'], - dtype=tf.int64) - self._ref_step = py_utils.CreateVariable('ref_step', wp, trainable=False) - self._metric_history = early_stop.MetricHistory(p.metric_history) - self._best_step = ops.best_step(self._metric_history.hist_file, - p.tolerance) - - def FProp(self, theta, current_step): - p = self.params - with tf.name_scope(p.name): - - steps = self._best_step - best_step = steps[0] - last_step = steps[1] - - ref_step = tf.maximum(self._ref_step, best_step) - f = self._cur_factor - - # Decay if no improvement within window. - new_factor = tf.where(last_step - ref_step < p.window, f, - tf.maximum(p.min_factor, f * p.decay)) - # Update ref_step if we decayed. - new_step = tf.where(tf.equal(new_factor, f), ref_step, last_step) - update_step = tf.assign(self._ref_step, new_step) - with tf.control_dependencies([update_step]): - return tf.assign(self._cur_factor, new_factor) - - -class CosineSchedule(BaseSchedule): - """Cosine learning rate decay. - - First proposed in https://arxiv.org/pdf/1608.03983.pdf, which only uses - multiple cycles with angle from 0 to pi/2. Later people use only one cycle - with angle from 0 to pi (e.g., https://arxiv.org/pdf/1711.09224.pdf), which - is implemented here. - - where: - angle = pi * min(1, current_step / total_steps) - decay_gap = initial_value - final_value - value = final_value + decay_gap * (1 + cosine(angle)) / 2 - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('initial_value', 1.0, 'Initial decay value.') - p.Define('final_value', 0., 'Final decay value.') - p.Define('total_steps', 0, 'Number of steps to reach full decay.') - return p - - def FProp(self, theta, current_step): - p = self.params - assert p.total_steps > 0 - assert p.initial_value > p.final_value - with tf.name_scope(p.name): - decay_gap = p.initial_value - p.final_value - return p.final_value + 0.5 * decay_gap * (1 + tf.cos(math.pi * tf.minimum( - 1.0, - tf.cast(current_step, tf.float32) / p.total_steps))) - - -class PiecewiseSchedule(BaseSchedule): - """Piecewise schedule composed of sub-schedules.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('boundaries', None, 'Boundaries between subschedules.') - p.Define( - 'schedules', None, 'A list of sub-schedules. ' - 'The length must be len(boundaries) + 1. ' - 'schedules[i] starts at boundaries[i-1] (inclusive) and ends at ' - 'boundaries[i] (exclusive). ' - 'The *relative* step in each interval will be passed to the ' - 'sub-schedule for FProp.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - prev_boundary = 0 - for boundary in p.boundaries: - if boundary < prev_boundary: - raise ValueError('Invalid boundary %s < %s' % (boundary, prev_boundary)) - prev_boundary = boundary - if len(p.schedules) != len(p.boundaries) + 1: - raise ValueError('len(schedules) != len(boundaries) + 1: %s vs %s' % - (len(p.schedules), len(p.boundaries))) - self.CreateChildren('schedules', p.schedules) - - def FProp(self, theta, current_step): - p = self.params - current_step = tf.cast(current_step, tf.int64) - interval_starts = [0] + p.boundaries - values = [] - for interval_start, schedule, schedule_theta in zip( - interval_starts, self.schedules, theta.schedules): - relative_step = tf.maximum( - tf.cast(0, current_step.dtype), - current_step - tf.cast(interval_start, current_step.dtype)) - values.append(schedule.FProp(schedule_theta, relative_step)) - - return py_utils.PiecewiseConstant(current_step, p.boundaries, values, - values[0].dtype) - - -class SqrtDecay(BaseSchedule): - """Sqrt decay schedule.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('warmup_steps', 10000, 'Number of warm up steps.') - p.Define('multiplier', 1.0, 'Multiplier.') - return p - - def FProp(self, theta, current_step): - p = self.params - step_num = tf.cast(current_step, tf.float32) - learning_rate = tf.math.rsqrt(tf.maximum(step_num, p.warmup_steps)) - learning_rate *= p.multiplier - return learning_rate - - -class CycleSchedule(BaseSchedule): - """Piecewise schedule composed of sub-schedules in a cycle.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'schedules', None, 'A list of sub-schedules. Unlike PiecewiseSchedule, ' - 'the absolute step is passed to the sub-schedule.') - p.Define('steps', None, 'The number of steps to run each sub-schedule.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if len(p.schedules) != len(p.steps): - raise ValueError('len(schedules) != len(steps): %s vs %s' % - (len(p.schedules), len(p.steps))) - self.CreateChildren('schedules', p.schedules) - boundaries = [0] - for step in p.steps: - boundaries.append(boundaries[-1] + step) - self._period = boundaries[-1] - self._boundaries = boundaries[1:-1] - - def FProp(self, theta, current_step): - values = [] - for schedule, schedule_theta in zip(self.schedules, theta.schedules): - values.append(schedule.FProp(schedule_theta, current_step)) - relative_step = tf.math.mod(current_step, self._period) - return py_utils.PiecewiseConstant(relative_step, self._boundaries, values, - values[0].dtype) diff --git a/lingvo/core/schedule_test.py b/lingvo/core/schedule_test.py deleted file mode 100644 index 83acdabd..00000000 --- a/lingvo/core/schedule_test.py +++ /dev/null @@ -1,572 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lr_schedule.""" - -import math -import os -import lingvo.compat as tf -from lingvo.core import cluster_factory -from lingvo.core import early_stop -from lingvo.core import schedule -from lingvo.core import test_utils - - -class LearningRateScheduleTest(test_utils.TestCase): - - def testConstant(self): - with self.session(use_gpu=False): - p = schedule.Constant.Params().Set(value=5) - lrs = p.Instantiate() - for x in [0, 10, 100, 1000000]: - self.assertAllClose(lrs.Value(x).eval(), 5.0) - - def testConstantOne(self): - with self.session(use_gpu=False): - p = schedule.ConstantOne.Params() - lrs = p.Instantiate() - for x in [0, 10, 100, 1000000]: - self.assertAllClose(lrs.Value(x).eval(), 1.0) - - def testPiecewiseConstant(self): - cls = schedule.PiecewiseConstantSchedule - with self.session(use_gpu=False): - bs = [300000, 400000, 500000] - vs = [1.0, 0.1, 0.01, 0.001] - x_ins = [tf.constant(x) for x in [299999, 399999, 499999, 599999]] - outs = [] - for x in x_ins: - lrs = cls.Params().Set(boundaries=bs, values=vs).Instantiate() - outs.append(lrs.Value(x).eval()) - self.assertAllClose([1.0, 0.1, 0.01, 0.001], outs) - - def testContinuousSchedule(self): - p = schedule.ContinuousSchedule.Params() - p.start_step = 1000 - p.half_life_steps = 100 - p.min = 0.1 - decay = p.Instantiate() - with self.session(): - self.assertAllClose(decay.Value(0).eval(), 1.0) - self.assertAllClose(decay.Value(500).eval(), 1.0) - self.assertAllClose(decay.Value(1000).eval(), 1.0) - self.assertAllClose(decay.Value(1100).eval(), 0.5) - self.assertAllClose(decay.Value(1200).eval(), 0.25) - self.assertAllClose(decay.Value(1300).eval(), 0.125) - self.assertAllClose(decay.Value(1400).eval(), 0.1) - self.assertAllClose(decay.Value(2000).eval(), 0.1) - - # Tests that the decay consistently decreases by half per 100 - # steps. - for step in range(1000, 1200, 25): - self.assertGreater( - decay.Value(step).eval(), - decay.Value(step + 10).eval()) - self.assertAllClose( - decay.Value(step).eval(), - decay.Value(step + 100).eval() * 2.) - - def testContinuousSchedule_CanOverrideStart(self): - p = schedule.ContinuousSchedule.Params() - p.initial_value = 2.0 - p.start_step = 1000 - p.half_life_steps = 100 - decay = p.Instantiate() - with self.session(): - self.assertAllClose(decay.Value(0).eval(), 2.0) - self.assertAllClose(decay.Value(1000).eval(), 2.0) - self.assertAllClose(decay.Value(1100).eval(), 1.0) - self.assertAllClose(decay.Value(1200).eval(), 0.5) - self.assertAllClose(decay.Value(1300).eval(), 0.25) - - def testStepwiseExponentialSchedule(self): - p = schedule.StepwiseExponentialSchedule.Params() - p.decay = 0.5 - p.num_steps_per_decay = 1000 - decay = p.Instantiate() - with self.session(): - self.assertAllClose(decay.Value(0).eval(), 1.0) - self.assertAllClose(decay.Value(999).eval(), 1.0) - self.assertAllClose(decay.Value(1000).eval(), 0.5) - self.assertAllClose(decay.Value(1999).eval(), 0.5) - self.assertAllClose(decay.Value(2000).eval(), 0.25) - - def testTransformerSchedule(self): - p = schedule.TransformerSchedule.Params() - p.warmup_steps = 4000 - p.model_dim = 512 - lrs = p.Instantiate() - with self.session(): - print(lrs.Value(0).eval()) - print(lrs.Value(1000).eval()) - print(lrs.Value(2000).eval()) - print(lrs.Value(3000).eval()) - print(lrs.Value(4000).eval()) - print(lrs.Value(4500).eval()) - print(lrs.Value(5000).eval()) - self.assertAllClose(lrs.Value(0).eval(), 1.74693e-07) - self.assertAllClose(lrs.Value(1000).eval(), 0.000174867) - self.assertAllClose(lrs.Value(2000).eval(), 0.00034956) - self.assertAllClose(lrs.Value(3000).eval(), 0.000524253) - self.assertAllClose(lrs.Value(4000).eval(), 0.000698684) - self.assertAllClose(lrs.Value(4500).eval(), 0.000658735) - self.assertAllClose(lrs.Value(5000).eval(), 0.000624937) - # Tests that the schedule peaks at 4000 steps. - self.assertGreater(lrs.Value(4000).eval(), lrs.Value(3990).eval()) - self.assertGreater(lrs.Value(4000).eval(), lrs.Value(4010).eval()) - - # Tests that the schedule increases linearly before 4000 steps. - for step in range(300, 4000, 200): - self.assertAllClose( - lrs.Value(step).eval() * 2., - lrs.Value(step + 10).eval() + lrs.Value(step - 10).eval()) - - def testTransformerScheduleWithDecayEnd(self): - p = schedule.TransformerSchedule.Params() - p.warmup_steps = 4000 - p.model_dim = 512 - p.decay_end = 5000 - lrs = p.Instantiate() - with self.session(): - self.assertAllClose(lrs.Value(0).eval(), 1.74693e-07) - self.assertAllClose(lrs.Value(3000).eval(), 0.000524253) - self.assertAllClose(lrs.Value(5000).eval(), 0.000624937) - - # Tests that the schedule peaks at 4000 steps. - self.assertGreater(lrs.Value(4000).eval(), lrs.Value(3990).eval()) - self.assertGreater(lrs.Value(4000).eval(), lrs.Value(4010).eval()) - - # Tests that the schedule increases linearly before 4000 steps. - for step in range(300, 4000, 200): - self.assertAllClose( - lrs.Value(step).eval() * 2., - lrs.Value(step + 10).eval() + lrs.Value(step - 10).eval()) - - print(lrs.Value(4999).eval()) - print(lrs.Value(5000).eval()) - print(lrs.Value(5001).eval()) - print(lrs.Value(6000).eval()) - # Tests that the schedule is fixed after decay end steps. - self.assertGreater(lrs.Value(4999).eval(), lrs.Value(5000).eval()) - self.assertAllClose(lrs.Value(5000).eval(), lrs.Value(5001).eval()) - self.assertAllClose(lrs.Value(5000).eval(), lrs.Value(6000).eval()) - - def testTransformerScheduleNoWarmUp(self): - params = schedule.TransformerScheduleNoWarmUp.Params().Set( - decay_start=4000, model_dim=512) - lrs = params.Instantiate() - - base_params = schedule.TransformerSchedule.Params().Set( - warmup_steps=4000, model_dim=512) - base_lrs = base_params.Instantiate() - - with self.session(): - - # Tests that the schedule is flat up until 4000 steps. - self.assertAllClose(lrs.Value(0).eval(), 0.000698684) - self.assertAllClose(lrs.Value(1000).eval(), 0.000698684) - self.assertAllClose(lrs.Value(2000).eval(), 0.000698684) - self.assertAllClose(lrs.Value(3000).eval(), 0.000698684) - self.assertAllClose(lrs.Value(4000).eval(), 0.000698684) - self.assertAllClose(lrs.Value(4500).eval(), 0.000658735) - self.assertAllClose(lrs.Value(5000).eval(), 0.000624937) - - # Test that the schedule is identical with transformer-lr after 4k steps - self.assertAllClose(base_lrs.Value(4000).eval(), lrs.Value(4000).eval()) - self.assertAllClose(base_lrs.Value(4010).eval(), lrs.Value(4010).eval()) - self.assertAllClose(base_lrs.Value(5000).eval(), lrs.Value(5000).eval()) - - def testPolynomialLRSchedule(self): - p = schedule.PolynomialSchedule.Params().Set( - power=2, start=(0, 0.), limit=(20000, 2.)) - with self.session(): - lrs = p.Instantiate() - pts = [[i, lrs.Value(i).eval()] for i in [0, 10000, 20000]] - self.assertAllClose( - pts, - [ - [0, 0.0], - [10000, 0.5], # 2 * (0.5 ** 2) - [20000, 2.0], - ]) - self.assertEqual(len(lrs.Value(42).shape), 0) - - def testPolynomialLimitOriginLRSchedule(self): - p = schedule.PolynomialSchedule.Params().Set( - power=2, start=(0, 0.), limit=(20000, 2.), origin='limit') - with self.session(): - lrs = p.Instantiate() - pts = [[i, lrs.Value(i).eval()] for i in [0, 5000, 10000, 15000, 20000]] - self.assertAllClose( - pts, - [ - [0, 0.0], - [5000, 0.875], # 2 * (1 - (1 - 0.25) ** 2) - [10000, 1.5], # 2 * (1 - (1 - 0.5) ** 2) - [15000, 1.875], # 2 * (1 - (1 - 0.75) ** 2) - [20000, 2.0], - ]) - self.assertEqual(len(lrs.Value(42).shape), 0) - - def testCombinedLRSchedule(self): - p = schedule.CombinedMinimumSchedule.Params().Set(schedules=[ - schedule.LinearSchedule.Params().Set( - start=(0., 1.), limit=(2000000, 8.)), - schedule.LinearSchedule.Params().Set( - start=(2000000., 8.), limit=(4000000, 8.)), - schedule.ExponentialSchedule.Params().Set( - start=(4000000., 8.), limit=(8000000, 0.5)) - ]) - with self.session(): - lrs = p.Instantiate() - pts = [[i, lrs.Value(i).eval()] for i in range(0, 10000000, 1000000)] - self.assertAllClose( - pts, - [ - # Linear increasing. - [0, 1.0], - [1000000, 4.5], - # Constant - [2000000, 8.0], - [3000000, 8.0], - # Exponentially decreasing. - [4000000, 8.0], - [5000000, 4.0], - [6000000, 2.0], - [7000000, 1.0], - [8000000, 0.5], - [9000000, 0.5] - ]) - - def testLinearRampupExponentialDecayScaledByNumSplitSchedule(self): - p = schedule.LinearRampupExponentialDecayScaledByNumSplitSchedule.Params( - ).Set( - warmup=250000, decay_start=32000000, decay_end=64000000, min=0.5) - with self.session(), cluster_factory.ForTestingWorker( - mode='sync', job='trainer_client', gpus=8): - lrs = p.Instantiate() - pts = [[i, lrs.Value(i).eval()] for i in range(0, 10000000, 1000000)] - self.assertAllClose( - pts, - [ - # Linear increasing. - [0, 1.0], - [1000000, 4.5], - # Constant - [2000000, 8.0], - [3000000, 8.0], - # Exponentially decreasing. - [4000000, 8.0], - [5000000, 4.0], - [6000000, 2.0], - [7000000, 1.0], - [8000000, 0.5], - [9000000, 0.5] - ]) - - def testLinearRampupExponentialDecayScaledByNumSplitScheduleWarmUpInit(self): - p = schedule.LinearRampupExponentialDecayScaledByNumSplitSchedule.Params( - ).Set( - warmup_init=0, - warmup=250000, - decay_start=32000000, - decay_end=64000000, - min=0.5) - with self.session(), cluster_factory.ForTestingWorker( - mode='sync', job='trainer_client', gpus=8): - lrs = p.Instantiate() - pts = [[i, lrs.Value(i).eval()] for i in range(0, 10000000, 1000000)] - self.assertAllClose( - pts, - [ - # Linear increasing from warmup_init=0. - [0, 0], - [1000000, 4.0], - # Constant - [2000000, 8.0], - [3000000, 8.0], - # Exponentially decreasing. - [4000000, 8.0], - [5000000, 4.0], - [6000000, 2.0], - [7000000, 1.0], - [8000000, 0.5], - [9000000, 0.5] - ]) - - def testLinearRampupExponentialDecayScaledByNumSplitScheduleWithCap(self): - p = schedule.LinearRampupExponentialDecayScaledByNumSplitSchedule.Params( - ).Set( - warmup=250000, - decay_start=32000000, - decay_end=64000000, - min=0.5, - max=5.0) - with self.session(), cluster_factory.ForTestingWorker( - mode='sync', job='trainer_client', gpus=8): - lrs = p.Instantiate() - pts = [[i, lrs.Value(i).eval()] for i in range(0, 10000000, 1000000)] - self.assertAllClose( - pts, - [ - # Linear increasing. - [0, 1.0], - [1000000, 4.5], - # Constant - [2000000, 5.0], - [3000000, 5.0], - # Exponentially decreasing. - [4000000, 5.0], - [5000000, 4.0], - [6000000, 2.0], - [7000000, 1.0], - [8000000, 0.5], - [9000000, 0.5] - ]) - - def testLinearRampupExponentialDecayScaledByNumSplitScheduleWithNumSplits( - self): - p = schedule.LinearRampupExponentialDecayScaledByNumSplitSchedule.Params( - ).Set( - warmup=250000, - decay_start=32000000, - decay_end=64000000, - min=0.5, - max=5.0, - num_splits=8) - # Increases the number of splits to 32. - with self.session(), cluster_factory.ForTestingWorker( - mode='sync', job='trainer_client', gpus=8, split_size=4): - lrs = p.Instantiate() - pts = [[i, lrs.Value(i).eval()] for i in range(0, 10000000, 1000000)] - # Values are copied from - # testLinearRampupExponentialDecayScaledByNumSplitScheduleWithCap. - self.assertAllClose( - pts, - [ - # Linear increasing. - [0, 1.0], - [1000000, 4.5], - # Constant - [2000000, 5.0], - [3000000, 5.0], - # Exponentially decreasing. - [4000000, 5.0], - [5000000, 4.0], - [6000000, 2.0], - [7000000, 1.0], - [8000000, 0.5], - [9000000, 0.5] - ]) - - def testLinearRampupExponentialDecayScaledByNumSplitScheduleNoWarmUp(self): - p = schedule.LinearRampupExponentialDecayScaledByNumSplitSchedule.Params( - ).Set( - warmup=0, decay_start=32000000, decay_end=64000000, min=0.5) - with self.session(), cluster_factory.ForTestingWorker( - mode='sync', job='trainer_client', gpus=8): - lrs = p.Instantiate() - pts = [[i, lrs.Value(i).eval()] for i in range(0, 10000000, 1000000)] - self.assertAllClose( - pts, - [ - # Constant - [0, 8.0], - [1000000, 8.0], - [2000000, 8.0], - [3000000, 8.0], - # Exponentially decreasing. - [4000000, 8.0], - [5000000, 4.0], - [6000000, 2.0], - [7000000, 1.0], - [8000000, 0.5], - [9000000, 0.5] - ]) - - def testLinearRampupExponentialDecayScaledByNumSplitScheduleExpOnly(self): - p = schedule.LinearRampupExponentialDecayScaledByNumSplitSchedule.Params( - ).Set( - warmup=0, decay_start=0, decay_end=32000000, min=0.5) - with self.session(), cluster_factory.ForTestingWorker( - mode='sync', job='trainer_client', gpus=8): - lrs = p.Instantiate() - pts = [[i, lrs.Value(i).eval()] for i in range(0, 6000000, 1000000)] - self.assertAllClose( - pts, - [ - # Exponentially decreasing. - [0, 8.0], - [1000000, 4.0], - [2000000, 2.0], - [3000000, 1.0], - [4000000, 0.5], - [5000000, 0.5] - ]) - - def testLinearRampupSqrtDecayByBatchSizeAndReplicasSchedule(self): - p = schedule.LinearRampupSqrtDecayByBatchSizeAndReplicas.Params().Set( - warmup_examples=100000, batch_size=100) - with self.session(), cluster_factory.ForTestingWorker( - mode='sync', job='trainer_client', gpus=10): - lrs = p.Instantiate() - self.assertAllClose(lrs.Value(-1).eval(), 0.0) - self.assertAllClose(lrs.Value(49).eval(), 0.05) - self.assertAllClose(lrs.Value(99).eval(), 0.1) - self.assertAllClose(lrs.Value(399).eval(), 0.05) - self.assertAllClose(lrs.Value(1599).eval(), 0.025) - - def testDevBasedSchedule(self): - logdir = tf.test.get_temp_dir() - tf.io.gfile.mkdir(os.path.join(logdir, 'eval_dev')) - - p = schedule.DevBasedSchedule.Params() - p.tolerance = 1.0 - p.window = 2 - p.decay = 0.5 - p.min_factor = 0.20 - early_stop.MetricHistory.SetLogdirInMetricHistories(p, logdir) - - lrs = p.Instantiate() - mh = lrs._metric_history - mh.params.local_filesystem = True - with self.session(): - self.evaluate(tf.global_variables_initializer()) - mh.ConditionalAppend(mh.params.jobname, mh.params.metric, 1, 10.0) - # best = 1 - self.assertAllClose(lrs.Value(0).eval(), 1.0) - - mh.ConditionalAppend(mh.params.jobname, mh.params.metric, 2, 5.0) - # best = 2 - self.assertAllClose(lrs.Value(0).eval(), 1.0) - - mh.ConditionalAppend(mh.params.jobname, mh.params.metric, 5, 4.0) - # best = 2, out of window - self.assertAllClose(lrs.Value(0).eval(), 0.5) - - mh.ConditionalAppend(mh.params.jobname, mh.params.metric, 6, 4.0) - # best = 2, ref = 5, in window - self.assertAllClose(lrs.Value(0).eval(), 0.5) - - mh.ConditionalAppend(mh.params.jobname, mh.params.metric, 9, 4.0) - # best = 2, ref = 5, out of window - self.assertAllClose(lrs.Value(0).eval(), 0.25) - - mh.ConditionalAppend(mh.params.jobname, mh.params.metric, 10, 3.9) - # best = 10 - self.assertAllClose(lrs.Value(0).eval(), 0.25) - - mh.ConditionalAppend(mh.params.jobname, mh.params.metric, 13, 3.0) - # best = 10, out of window, min factor - self.assertAllClose(lrs.Value(0).eval(), 0.20) - - def testLinearRampupPiecewiseConstantSchedule(self): - p = schedule.LinearRampupPiecewiseConstantSchedule.Params().Set( - boundaries=[40, 64, 80, 96], - lrs=[1.0, 0.1, 0.01, 0.001], - ) - with self.session(), cluster_factory.ForTestingWorker( - mode='sync', job='trainer_client', tpus=8): - lrs = p.Instantiate() - pts = [[i, lrs.Value(i).eval()] for i in range(0, 15, 1)] - - self.assertAllClose( - pts, [[0, 0.0], [1, 1.6], [2, 3.2], [3, 4.8], [4, 6.4], [5, 8.0], - [6, 8.0], [7, 8.0], [8, 0.8], [9, 0.8], [10, 0.08], [11, 0.08], - [12, 0.008], [13, 0.008], [14, 0.008]]) - - def testCosineSchedule(self): - p = schedule.CosineSchedule.Params().Set( - initial_value=3.0, final_value=1.0, total_steps=400000) - with self.session(): - lrs = p.Instantiate() - pts = [[i, lrs.Value(i).eval()] for i in range(0, 600000, 100000)] - self.assertAllClose( - pts, - [ - [0, 3.0], - [100000, math.cos(math.pi / 4) + 2.], # angle=pi/4 - [200000, 2.0], # angle=pi/2, half-way - [300000, math.cos(math.pi * 3 / 4) + 2.], # angle=pi*3/4 - [400000, 1.0], - [500000, 1.0], - ]) - - def testLinearRampupCosineSchedule(self): - p = schedule.LinearRampupCosineSchedule.Params().Set( - warmup_steps=200, - initial_value=3.0, - final_value=1.0, - total_steps=400000, - num_splits=1) - with self.session(): - lrs = p.Instantiate() - - pts = [[i, lrs.Value(i).eval()] - for i in [0, 100, 200, 100000, 200000, 300000, 400000]] - self.assertAllClose( - pts, - [ - [0, 0.0], - [100, 1.5], - [200, 3.0], - [100000, math.cos(math.pi / 4) + 2.], # angle=pi/4 - [200000, 2.0], # angle=pi/2, half-way - [300000, math.cos(math.pi * 3 / 4) + 2.], # angle=pi*3/4 - [400000, 1.0], - ]) - - def testPiecewiseSchedule(self): - # Linear ramp-up in 20000 steps, cosine decay in 40000 steps. - p0 = schedule.LinearSchedule.Params().Set(start=(0, 0.), limit=(20000, 2.)) - p1 = schedule.CosineSchedule.Params().Set( - initial_value=2.0, total_steps=40000) - p = schedule.PiecewiseSchedule.Params().Set( - boundaries=[20000], schedules=[p0, p1]) - with self.session(): - lrs = p.Instantiate() - pts = [[i, lrs.Value(i).eval()] for i in range(0, 70000, 10000)] - self.assertAllClose( - pts, - [ - [0, 0.0], - [10000, 1.0], # half-way in linear ramp-up. - [20000, 2.0], # completed linear ramp-up. - [30000, math.cos(math.pi / 4) + 1.], # pi/4. - [40000, 1.0], # pi/2. - [50000, math.cos(math.pi * 3 / 4) + 1.], # pi*3/4. - [60000, 0.0], # pi. - ]) - - def testCycleSchedule(self): - p0 = schedule.LinearSchedule.Params().Set(start=(0, 0.), limit=(1000, 1.)) - p1 = schedule.Constant.Params().Set(value=5.0) - p = schedule.CycleSchedule.Params().Set(schedules=[p0, p1], steps=[4, 1]) - with self.session(): - lrs = p.Instantiate() - pts = [[i, lrs.Value(i).eval()] for i in [0, 1, 4, 5, 998, 999, 1000]] - self.assertAllClose(pts, [ - [0, 0.0], - [1, 1.0 / 1000.0], - [4, 5.0], - [5, 5.0 / 1000.0], - [998, 998.0 / 1000.0], - [999, 5.0], - [1000, 1.0], - ]) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/scorers.py b/lingvo/core/scorers.py deleted file mode 100644 index cd962428..00000000 --- a/lingvo/core/scorers.py +++ /dev/null @@ -1,132 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Helper classes for computing scores.""" - -import collections -import math -import six - - -def _ToUnicode(line): - return six.ensure_text(line, 'utf-8') - - -def _Tokenize(string): - return _ToUnicode(string).split() - - -def NGrams(lst, order): - """Generator that yields all n-grams of the given order present in lst.""" - return (lst[i:i + order] for i in range(len(lst) - order + 1)) - - -class Unsegmenter: - """Un-segments (merges) segmented strings. - - Used to retain back the original surface form of strings that are encoded - using byte-pair-encoding (BPE), word-piece-models (WPM) or - sentence-piece-models (SPM). - """ - - _BPE_SEPARATOR = _ToUnicode('@@ ') - _WPM_SEPARATOR = _ToUnicode('\xe2\x96\x81') # Same for SPM. - - def __init__(self, separator_type=None): - self._separator_type = separator_type - - def _UnsegmentWpm(self, line): - return _ToUnicode(line).replace(' ', '').replace(self._WPM_SEPARATOR, - ' ').strip() - - def _UnsegmentBpe(self, line): - return _ToUnicode(line).replace(self._BPE_SEPARATOR, '').strip() - - def __call__(self, line): - if self._separator_type == 'bpe': - return self._UnsegmentBpe(line) - elif self._separator_type in ['wpm', 'spm']: - return self._UnsegmentWpm(line) - else: - return line - - -class BleuScorer: - """Scorer to compute BLEU scores to measure translation quality. - - The BLEU score is the geometric average precision of all token n-grams of - order 1 to max_ngram across all sentences. - - Successive calls to AddSentence() accumulate statistics which are converted to - an overall score on calls to ComputeOverallScore(). - - Example usage: - >>> scorer = BleuScorer(max_ngram=4) - >>> scorer.AddSentence("hyp matches ref str", "hyp matches ref str") - >>> scorer.AddSentence("almost right", "almost write") - >>> print(scorer.ComputeOverallScore()) - 0.6687... - """ - - def __init__(self, max_ngram=4, separator_type=None): - self._max_ngram = max_ngram - self._hyp_ngram_matches = [0 for _ in range(max_ngram)] - self._hyp_ngram_counts = [0 for _ in range(max_ngram)] - self._num_ref_tokens = 0 - self._num_hyp_tokens = 0 - self._unsegmenter = Unsegmenter(separator_type) - - @property - def unsegmenter(self): - return self._unsegmenter - - def AddSentence(self, ref_str, hyp_str): - """Accumulates ngram statistics for the given ref and hyp string pair.""" - ref_tokens = tuple(_Tokenize(self._unsegmenter(ref_str))) - self._num_ref_tokens += len(ref_tokens) - hyp_tokens = tuple(_Tokenize(self._unsegmenter(hyp_str))) - self._num_hyp_tokens += len(hyp_tokens) - for order_idx in range(self._max_ngram): - ref_counts = collections.Counter(NGrams(ref_tokens, order_idx + 1)) - hyp_matches = collections.Counter() - hyp_count = 0 - for x in NGrams(hyp_tokens, order_idx + 1): - hyp_count += 1 - count = ref_counts[x] - if count: - # Clip hyp_matches so ngrams that are repeated more frequently in hyp - # than ref are not double counted. - hyp_matches[x] = min(hyp_matches[x] + 1, count) - self._hyp_ngram_matches[order_idx] += sum(hyp_matches.values()) - self._hyp_ngram_counts[order_idx] += hyp_count - - def ComputeOverallScore(self): - """Computes overall BLEU score from the statistics accumulated so far.""" - score = 0.0 - num_nonzero_orders = 0 - for order_idx in range(self._max_ngram): - matches = self._hyp_ngram_matches[order_idx] - total = self._hyp_ngram_counts[order_idx] - if matches > 0.0 and total > 0.0: - score += math.log(matches / total) - num_nonzero_orders += 1 - if not num_nonzero_orders: - return 0.0 - precision = math.exp(score / num_nonzero_orders) - - brevity_penalty = 1.0 - if self._num_hyp_tokens < self._num_ref_tokens: - brevity_penalty = math.exp(1 - self._num_ref_tokens/self._num_hyp_tokens) - return brevity_penalty * precision diff --git a/lingvo/core/scorers_test.py b/lingvo/core/scorers_test.py deleted file mode 100644 index b888cee8..00000000 --- a/lingvo/core/scorers_test.py +++ /dev/null @@ -1,98 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for scorers.""" - -import math -import lingvo.compat as tf -from lingvo.core import scorers -from lingvo.core import test_helper -from lingvo.core import test_utils - - -class BleuScorerTest(test_utils.TestCase): - - def testNGrams(self): - words = 'a b c d e'.split(' ') - self.assertEqual([['a'], ['b'], ['c'], ['d'], ['e']], - list(scorers.NGrams(words, 1))) - - self.assertEqual([['a', 'b'], ['b', 'c'], ['c', 'd'], ['d', 'e']], - list(scorers.NGrams(words, 2))) - - self.assertEqual([['a', 'b', 'c'], ['b', 'c', 'd'], ['c', 'd', 'e']], - list(scorers.NGrams(words, 3))) - - def testBleuScorerDocTest(self): - scorer = scorers.BleuScorer(max_ngram=4) - scorer.AddSentence('hyp matches ref str', 'hyp matches ref str') - self.assertAlmostEqual((4/4 * 3/3 * 2/2 * 1/1) ** (1/4), - scorer.ComputeOverallScore()) - scorer.AddSentence('almost right', 'almost write') - self.assertAlmostEqual((5/6 * 3/4 * 2/2 * 1/1) ** (1/4), - scorer.ComputeOverallScore()) - - def testBleuScorerClipsExtraHypNGrams(self): - scorer = scorers.BleuScorer(max_ngram=4) - scorer.AddSentence('a b c d', 'a a b c d') - self.assertAlmostEqual((4/5 * 3/4 * 2/3 * 1/2) ** (1/4), - scorer.ComputeOverallScore()) - - def testBleuScorerSentencesShorterThanMaxNGram(self): - scorer = scorers.BleuScorer(max_ngram=4) - scorer.AddSentence('', '') - self.assertAlmostEqual(0.0, scorer.ComputeOverallScore()) - scorer.AddSentence('a', 'a') - self.assertAlmostEqual(1.0, scorer.ComputeOverallScore()) - scorer.AddSentence('a b', 'a b') - self.assertAlmostEqual(1.0, scorer.ComputeOverallScore()) - scorer.AddSentence('a b c', 'a b c') - self.assertAlmostEqual(1.0, scorer.ComputeOverallScore()) - scorer.AddSentence('a b c d', 'a b c d') - self.assertAlmostEqual(1.0, scorer.ComputeOverallScore()) - - def testBleuScorerBrevityPenalty(self): - scorer = scorers.BleuScorer(max_ngram=4) - scorer.AddSentence('1 2 3 4 5', '1 2 3 4 -2') - # No penalty. - expected_score = (4/5 * 3/4 * 2/3 * 1/2) ** (1/4) - self.assertAlmostEqual(0.6687403, expected_score) - self.assertAlmostEqual(expected_score, scorer.ComputeOverallScore()) - - scorer = scorers.BleuScorer(max_ngram=4) - scorer.AddSentence('1 2 3 4 5', '1 2 3 4') - expected_score = math.exp(1 - 5/4) * (4/4 * 3/3 * 2/2 * 1/1) ** (1/4) - self.assertAlmostEqual(0.7788008, expected_score) - self.assertAlmostEqual(expected_score, scorer.ComputeOverallScore()) - - def testBleuScorerMatchesCppImplementationOneExample(self): - scorer = scorers.BleuScorer() - scorer.AddSentence( - 'It is obvious that this will have a certain influence .', - 'It is clear that this will have a certain influence .') - self.assertAlmostEqual(0.74194467, scorer.ComputeOverallScore()) - - def testBleuScorerMatchesCppImplementation(self): - filename = test_helper.test_src_dir_path('core/ops/testdata/wmt/sm18.txt') - scorer = scorers.BleuScorer() - with open(filename, 'rb') as fp: - for line in fp: - hyp, ref = line[:-1].split(b'\t') - scorer.AddSentence(ref, hyp) - self.assertAlmostEqual(0.313776, scorer.ComputeOverallScore(), places=5) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/self_attention_layer.py b/lingvo/core/self_attention_layer.py deleted file mode 100644 index bd19951a..00000000 --- a/lingvo/core/self_attention_layer.py +++ /dev/null @@ -1,101 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Self-attention layers. - -[1] Attention is all you need. - https://arxiv.org/pdf/1706.03762.pdf Section 3. -[2] Pay Less Attention with Lightweight and Dynamic Convolutions. - https://arxiv.org/abs/1901.10430 -""" - -from lingvo.core import base_layer -from lingvo.core import batch_major_attention -from lingvo.core import py_utils - -MultiHeadedSelfAttention = batch_major_attention.MultiHeadedAttention - - -# pyformat: disable -class Builder(batch_major_attention.Builder): - """Builder for self-attention layers.""" - - def SelfAttention(self, name): - p = self.params - input_to_add = ( - 'i.vec' if p.selfatten_add_unnormalized_input else 'after_ln') - - attention_inputs = 'after_ln,after_ln,after_ln,i.paddings' - if p.packed_input: - attention_inputs += ',i.segment_mask' - - sub_list = [ - ('i.vec->after_ln', self._LN('LN', p.model_dim)), - ('{}->after_att,unused_prob'.format(attention_inputs), self._MultiHeadedAtten('atten')), - ('after_att->after_dropout', - self._Dropout('dropout', p.residual_dropout_prob)), - ('{},after_dropout->o.vec'.format(input_to_add), self._Add('add')), - ('i.paddings->o.paddings', self._Id('id')), - ] - - if p.packed_input: - sub_list.append( - ('i.segment_mask->o.segment_mask', self._Id('mask'))) - - return self._Graph( - name, - ['i'], # input NestedMap with {vec, paddings, segment_mask} - ['o'], # output NestedMap with {vec, paddings, segment_mask} - *sub_list) - - def _TransformerLayerBlock(self, name): - """(inputs, paddings) -> (encoded, paddings).""" - return self._Seq( - name, - self.SelfAttention('self_atten'), - self.Feedforward('ff')) - - def TransformerStack(self, name, num_layers=1): - """Returns a stack of num_layers self-attention layers.""" - blocks = [self._TransformerLayerBlock( - 'block_{}'.format(d)) for d in range(num_layers)] - return self._MaybeSplit(name, blocks) or ( - self._Rep(name, num_layers, self._TransformerLayerBlock('block'))) -# pyformat: enable - - -# TODO(huangyp): remove this layer after transition to nested maps is complete. -class StackedTransformerEncoderLayers(base_layer.BaseLayer): - """Wrapper class for layers returned by Builder.TransformerStack.""" - - @classmethod - def Cast(cls, params): - # Cast params returned from the builder to params in this class. - params.Define('base_cls', params.cls, 'Store the base class in params.') - params.cls = cls - return params - - def __init__(self, params): - # Make this class a sub-class of params.base_cls - self.__class__ = type(self.__class__.__name__, (params.base_cls, object), - dict(self.__class__.__dict__)) - super(self.__class__, self).__init__(params) # pylint: disable=bad-super-call - - def FProp(self, theta, vec, paddings, segment_mask=None): - outputs = super(self.__class__, self).FProp( # pylint: disable=bad-super-call - theta, - py_utils.NestedMap( - vec=vec, paddings=paddings, segment_mask=segment_mask)) - return outputs.vec, outputs.paddings diff --git a/lingvo/core/self_attention_layer_test.py b/lingvo/core/self_attention_layer_test.py deleted file mode 100644 index dcce2e0e..00000000 --- a/lingvo/core/self_attention_layer_test.py +++ /dev/null @@ -1,167 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for batch_major self attention.""" - -from absl.testing import parameterized -from lingvo import compat as tf -from lingvo.core import batch_major_attention as mt_attention -from lingvo.core import py_utils -from lingvo.core import self_attention_layer as self_attention -from lingvo.core import test_utils -import numpy as np - - -class BuilderTest(test_utils.TestCase, parameterized.TestCase): - - @parameterized.named_parameters( - { - 'testcase_name': '_baseline', - 'num_splits': 1, - 'num_micro_batches': 1 - }, { - 'testcase_name': '_two_splits', - 'num_splits': 2, - 'num_micro_batches': 2 - }, { - 'testcase_name': '_one_split_two_micro_batches', - 'num_splits': 1, - 'num_micro_batches': 2 - }) - - def testTransformerStack(self, num_splits, num_micro_batches): - with self.session(use_gpu=False) as sess: - bs = 2 - sl = 21 - d = 16 - tf.random.set_seed(12345) - deterministic_dropout = num_splits > 1 or num_micro_batches > 1 - atten_builder = self_attention.Builder.Params().Set( - model_dim=d, - num_heads=2, - ff_hidden_dim=5, - deterministic_dropout=deterministic_dropout, - num_splits=num_splits, - num_micro_batches=num_micro_batches) - p = atten_builder.Instantiate().TransformerStack('atten', 6) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - input_embs = tf.constant( - np.random.random(size=[bs, sl, d]), dtype=np.float) - paddings = tf.zeros([bs, sl]) - segment_mask = tf.zeros([bs, 1, sl, sl]) - - out = l.FPropDefaultTheta( - py_utils.NestedMap( - vec=input_embs, paddings=paddings, segment_mask=segment_mask)) - enc_out = out.vec - tf.logging.info('enc_out={}'.format(enc_out.shape)) - enc_out_sum = tf.reduce_sum(enc_out) - - tf.global_variables_initializer().run() - actual_enc_out, actual_enc_out_sum = sess.run([enc_out, enc_out_sum]) - print('actual_enc_out_sum=', actual_enc_out_sum) - - self.assertAllEqual(actual_enc_out.shape, [bs, sl, d]) - self.assertAllClose(386.16741943359375, actual_enc_out_sum, atol=1e-5) - - -class TransformerLayerTest(test_utils.TestCase, parameterized.TestCase): - - def _TransformerAttentionLayerInputs(self, input_dim=4, dtype=tf.float32): - np.random.seed(6348575) - query_vec = tf.transpose( - tf.stack([ - tf.constant(np.random.rand(2, input_dim), dtype=dtype) - for _ in range(5) - ]), [1, 0, 2]) - paddings = tf.constant([[0, 0, 1, 1, 0], [1, 0, 0, 0, 1]], dtype=dtype) - aux_vec = tf.transpose( - tf.stack([ - tf.constant(np.random.rand(2, input_dim), dtype=dtype) - for _ in range(7) - ]), [1, 0, 2]) - aux_paddings = tf.constant([[0, 1, 0, 1, 0, 1, 0], [1, 0, 1, 0, 1, 0, 1]], - dtype=dtype) - - segment_mask = tf.zeros([2, 1, 5, 5]) - return query_vec, paddings, aux_vec, aux_paddings, segment_mask - - def testTransformerAttentionLayerFPropCaseEncoder(self): - with self.session(use_gpu=True) as sess: - query_vec, paddings, _, _, segment_mask = self._TransformerAttentionLayerInputs( - ) - - expected_p = mt_attention.TransformerAttentionLayer.Params().Set( - name='transformer_self_atten', - input_dim=4, - is_masked=False, - num_heads=2) - expected_p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - expected_l = expected_p.Instantiate() - expected_ctx_vec, _ = expected_l.FProp(expected_l.theta, query_vec, None, - paddings) - - atten_builder = self_attention.Builder.Params().Set( - model_dim=4, num_heads=2, selfatten_add_unnormalized_input=True) - p = atten_builder.Instantiate().SelfAttention('self_atten') - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - l_out = l.FProp( - l.theta, - py_utils.NestedMap( - vec=query_vec, paddings=paddings, segment_mask=segment_mask)) - ctx_vec = l_out.vec - tf.global_variables_initializer().run() - actual_ctx = sess.run(ctx_vec) - expected_ctx = sess.run(expected_ctx_vec) - self.assertAllClose(expected_ctx, actual_ctx) - - def testTransformerLayerFProp(self): - with self.session(use_gpu=True) as sess: - src_vec, _, _, _, segment_mask = self._TransformerAttentionLayerInputs() - paddings = tf.zeros([2, 5]) - p = mt_attention.TransformerLayer.Params() - p.name = 'transformer_encoder_layer' - p.input_dim = 4 - p.tr_fflayer_tpl.hidden_dim = 7 - p.tr_atten_tpl.num_heads = 2 - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - l = p.Instantiate() - enc_vec, _ = l.FProp(l.theta, src_vec, paddings, None, None) - - self_attention_p = self_attention.Builder.Params().Set( - model_dim=4, - num_heads=2, - ff_hidden_dim=7, - selfatten_add_unnormalized_input=True) - self_attention_p = self_attention_p.Instantiate().TransformerStack( - name='transformer_encoder_layer', num_layers=1) - self_attention_p.params_init = py_utils.WeightInit.Xavier( - scale=1.0, seed=0) - self_attention_l = self_attention_p.Instantiate() - self_attention_enc_out = self_attention_l.FProp( - self_attention_l.theta, - py_utils.NestedMap( - vec=src_vec, paddings=paddings, segment_mask=segment_mask)) - self_attention_enc_vec = self_attention_enc_out.vec - tf.global_variables_initializer().run() - enc_vec = sess.run(enc_vec) - self_attention_enc_vec = sess.run(self_attention_enc_vec) - self.assertAllClose(enc_vec, self_attention_enc_vec) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/sendrecv.py b/lingvo/core/sendrecv.py deleted file mode 100644 index f489a41c..00000000 --- a/lingvo/core/sendrecv.py +++ /dev/null @@ -1,112 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Send/Recv ops. - -The following _Send()/_Recv() are adapted from python op wrappers -generated by python_op_gen_main. python_op_gen_main.cc's -PrintAllPythonOps needs to be updated to export internal ops. -""" - -from lingvo import compat as tf - -# pylint: disable=g-direct-tensorflow-import -from tensorflow.compiler.tf2xla.python import xla -# pylint: enable=g-direct-tensorflow-import - - -def _TpuCore(device): - """Returns the TPU core represented by , or -1 if not TPU.""" - prefix = "device:TPU_REPLICATED_CORE:" - if prefix in device: - return int(device[len(prefix):]) - return -1 - - -class Channel: - """A communication channel to transfer tensors in order.""" - - def __init__(self, dtype, shape, send_device, recv_device, name=None): - """Construct a channel. - - Args: - dtype: The dtype of tensors sent through the channel. - shape: The shape of tensors sent through the channel. Must be a fully - defined shape for TPUs. - send_device: A fully-specified tensorflow device. - recv_device: A fully-specified tensorflow device. - name: A name for the channel (optional). - """ - current_graph = tf.get_default_graph() - assert current_graph, "A channel is scoped within a tf.Graph" - self._dtype = dtype - self._send_device = send_device - self._recv_device = recv_device - self._name = current_graph.unique_name(name if name else "channel") - - assert shape is not None - shape = tf.TensorShape(shape) - - self._shape = shape - self._send_tpu_core = _TpuCore(send_device) - self._recv_tpu_core = _TpuCore(recv_device) - self._send_called = False - self._recv_op = None - assert ((self._send_tpu_core == -1) == (self._recv_tpu_core == -1)), ( - "Mixing TPU and non-TPU: %s and %s" % (send_device, recv_device)) - if self._send_tpu_core >= 0: - assert self._shape.is_fully_defined(), ( - "TPU channel must have fully defined shape. Name: %s, shape: %s" % - (self._name, self._shape)) - assert self._send_tpu_core != self._recv_tpu_core, ( - "TPU send/recv must be cross-core: %s and %s" % - (send_device, recv_device)) - - def Send(self, tensor): - """Sends a tensor through the channel.""" - assert tensor.dtype == self._dtype - assert not self._send_called, ("Send called multiple times for %s" % - self._name) - self._send_called = True - if self._send_tpu_core == -1: - return tf.raw_ops.Send( - tensor=tensor, - tensor_name=self._name, - send_device=self._send_device, - send_device_incarnation=0, - recv_device=self._recv_device) - else: - with tf.device(self._send_device): - return xla.send( - tensor, tensor_name=self._name, name="Send_" + self._name) - - def Recv(self): - """Receives a tensor from the channel.""" - if self._send_tpu_core == -1: - received = tf.raw_ops.Recv( - tensor_type=self._dtype, - tensor_name=self._name, - send_device=self._send_device, - send_device_incarnation=0, - recv_device=self._recv_device) - received.set_shape(self._shape) - return received - else: - with tf.device(self._recv_device): - return xla.recv( - self._dtype, - tensor_name=self._name, - shape=self._shape, - name="Recv_" + self._name) diff --git a/lingvo/core/sendrecv_test.py b/lingvo/core/sendrecv_test.py deleted file mode 100644 index 91637a1c..00000000 --- a/lingvo/core/sendrecv_test.py +++ /dev/null @@ -1,105 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for sendrecv.""" - -from absl.testing import parameterized -import lingvo.compat as tf -from lingvo.core import py_utils -from lingvo.core import sendrecv -from lingvo.core import test_utils -import numpy as np - -FLAGS = tf.flags.FLAGS - - -def _ListDevices(target): - with tf.Session(target) as sess: - devices = sess.list_devices() - return [_.name for _ in devices] - - -def _Target(): - return "" - - -class SendrecvTest(test_utils.TestCase, parameterized.TestCase): - - def testBasic(self): - devices = _ListDevices(_Target()) - print("\n".join(devices)) - sender, recver = devices[0], devices[-1] - shape = [] - - for dtype in tf.float32, tf.complex64: - to_send = np.array(3.1415 + 2j).astype(dtype.as_numpy_dtype) - g = tf.Graph() - with g.as_default(): - ch = sendrecv.Channel(dtype, shape, sender, recver, "test") - with tf.device(sender): - src_val = tf.constant(to_send) - send_op = ch.Send(src_val) - with tf.device(recver): - recv_val = ch.Recv() - - with tf.Session(_Target(), graph=g): - _, val = self.evaluate([send_op, recv_val]) - - self.assertAllClose(to_send, val) - - @parameterized.named_parameters( - ("_defun", False), - ("_function", True), - ) - def testInsideFunction(self, use_tf_function): - FLAGS.call_defun_use_tf_function = use_tf_function - devices = _ListDevices(_Target()) - sender, recver = devices[0], devices[-1] - shape = [] - - def SendRecv(graph, dtype): - to_send = np.array(3.1415 + 2j).astype(dtype.as_numpy_dtype) - with graph.as_default(): - ch = sendrecv.Channel(dtype, shape, sender, recver, "test") - with tf.device(sender): - - # py_utils.CallDefun requires non-empty inputs. Same below. - def Send(_): - src_val = tf.constant(to_send) - ch.Send(src_val) - return tf.convert_to_tensor(1.0) - - send_op = py_utils.CallDefun(Send, tf.convert_to_tensor(0)) - - with tf.device(recver): - - def Recv(_): - return ch.Recv() - - recv_val = py_utils.CallDefun(Recv, tf.convert_to_tensor(0)) - return send_op, recv_val, to_send - - for dtype in tf.float32, tf.complex64: - g = tf.Graph() - send_op, recv_val, sent_val = SendRecv(g, dtype) - - with tf.Session(_Target(), graph=g): - _, val = self.evaluate([send_op, recv_val]) - - self.assertAllClose(sent_val, val) - - -if __name__ == "__main__": - tf.test.main() diff --git a/lingvo/core/spectrum_augmenter.py b/lingvo/core/spectrum_augmenter.py deleted file mode 100644 index 18623328..00000000 --- a/lingvo/core/spectrum_augmenter.py +++ /dev/null @@ -1,845 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Lingvo layers that used for spectrum augmentation.""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils - -_SPECAUGMENT_ARGS = ( - 'freq_mask_max_bins', - 'freq_mask_count', - 'use_dynamic_time_mask_max_frames', - 'time_mask_max_frames', - 'time_mask_count', - 'time_mask_max_ratio', - 'time_masks_per_frame', - 'time_warp_bound', - 'time_warp_max_frames', - 'time_warp_max_ratio', -) - - -def _random_uniform_op(use_stateless_op): - return tf.random.stateless_uniform if use_stateless_op else tf.random.uniform - - -def _random_normal_op(use_stateless_op): - return tf.random.stateless_normal if use_stateless_op else tf.random.normal - - -def _global_seed_from_inputs(input_floats): - """Generates a random seed tensor based on input floats and mode key. - - Args: - input_floats: a set of float input tensors that are derived from the input - data (for example, input tokens). The important thing is that these are - usually different for each batch. - - Returns: - A tensor of shape=[2] with integer seed tensors derived from the inputs. - """ - timestamp = tf.math.floormod( - tf.cast(tf.timestamp(), dtype=tf.int64), 10000000) - input_sum = tf.cast(tf.reduce_sum(tf.math.abs(input_floats)), dtype=tf.int64) - return tf.stack([timestamp + input_sum, timestamp - input_sum], axis=-1) - - -def _hat(x): - """Hat function. - - The hat function is a piecewise linear function defined such that - 1) x < -1: _hat(x) = 0 - 2) -1 <= x < 0: _hat(x) = x + 1 - 3) 0 <= x < 1: _hat(x) = -x + 1 - 4) x > 1 : _hat(x) = 0 - - Args: - x: A tensor. - - Returns: - Tensor obtained by element-wise application of the hat function. - """ - return tf.nn.relu(x + 1) - 2 * tf.nn.relu(x) + tf.nn.relu(x - 1) - - -class SpectrumAugmenter(base_layer.BaseLayer): - """Performs data augmentation as according to the SpecAug paper. - - https://arxiv.org/pdf/1904.08779.pdf - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('freq_mask_max_bins', 15, - 'Maximum number of frequency bins of frequency masking.') - p.Define('freq_mask_count', 1, - 'Number of times we apply masking on the frequency axis.') - # TODO(danielspark): Deprecate 'use_dynamic_time_mask_max_frames' and - # introduce enum parameter to replace it. - p.Define( - 'use_dynamic_time_mask_max_frames', False, - 'If true, time_mask_max_frames is determined by ' - 'time_mask_max_ratio * utterance_length.') - p.Define( - 'time_mask_max_frames', 50, 'Maximum number of frames of time masking. ' - 'Overridden when use_dynamic_time_mask_max_frames = True.') - p.Define( - 'time_mask_count', 1, - 'Number of times we apply masking on the time axis. ' - 'Acts as upper-bound when time_masks_per_frame > 0.') - p.Define('time_mask_max_ratio', 1.0, - 'Maximum portion allowed for time masking.') - p.Define( - 'time_masks_per_frame', 0.0, - 'Ratio of number of time masks to be applied against the number ' - 'of frames. If > 0, multiplicity of the time mask is determined by ' - 'min(time_masks_per_frame * utterance_length, time_mask_count).') - p.Define( - 'time_warp_bound', 'static', - 'To be set to either `dynamic` or `static`. ' - 'If `dynamic`, time warp bound is determined by ' - 'time_warp_max_ratio * utterance_length. ' - 'If `static`, time warp bound is determined by ' - 'min(time_warp_max_frames, time_warp_max_ratio * utterance_length).') - p.Define('time_warp_max_frames', 0, - 'Maximum number of frames for shifting in time warping.') - p.Define('time_warp_max_ratio', 0.0, - 'Maximum portion of frames for shifting in time warping.') - p.Define('use_noise', False, 'Whether to noisify the time masked region.') - p.Define('gaussian_noise', False, 'Use Gaussian distribution for noise.') - p.Define('unstack', False, - 'Whether to unstack features before applying SpecAugment.') - p.Define('stack_height', 3, - 'Number of frames stacked on top of each other.') - p.Define( - 'domain_ids', [0], - 'If domain ids was given, this parameters describe which domain ' - 'will be augmented, e.g. ' - 'p.domain_ids = [2, 7, 1] ' - 'p.time_mask_count = [1, 2, 0] ' - 'implies domain 2 will have 1, 7 has 2 and 1 has 0 time masks. ' - 'All other domain will not augmented if it exists.') - p.Define( - 'use_input_dependent_random_seed', False, - 'Whether to use stateless random TensorFlow ops, with seeds' - 'determined by the input features. This feature is necessary for' - 'applications including federated learning.') - return p - - def __init__(self, params): - super().__init__(params) - self.SetVariableFree() - p = self.params - num_domains = len(p.domain_ids) - for field in _SPECAUGMENT_ARGS: - v = getattr(p, field) - if isinstance(v, (list, tuple)): - assert len(v) == num_domains, ( - 'Length: %d of field: %s does not match total domains: %d' % - (len(v), field, num_domains)) - else: - setattr(p, field, [v] * num_domains) - assert p.freq_mask_max_bins[0] > -1 - assert p.time_mask_max_frames[0] > -1 - assert p.time_warp_max_frames[0] > -1 - - def EinsumBBmBm(self, a, b, name=None): - return tf.einsum('b,bm->bm', a, b, name=name) - - def EinsumBmtBmBt(self, a, b, name=None): - return tf.einsum('bmt,bm->bt', a, b, name=name) - - def EinsumBxycByBxyc(self, a, b, name=None): - return tf.einsum('bxyc,by->bxyc', a, b, name=name) - - def EinsumBxycBxBxyc(self, a, b, name=None): - return tf.einsum('bxyc,bx->bxyc', a, b, name=name) - - def EinsumBxyBxBxy(self, a, b, name=None): - return tf.einsum('bxy,bx->bxy', a, b, name=name) - - def EinsumBxycBzxBzyc(self, a, b, name=None): - return tf.einsum('bxyc,bzx->bzyc', a, b, name=name) - - def _GetMask(self, - batch_size, - choose_range, - mask_size, - global_seed, - max_length=None, - masks_per_frame=0.0, - multiplicity=1, - dtype=tf.float32, - max_ratio=1.0): - """Returns fixed size multi-masks starting from random positions. - - A multi-mask is a mask obtained by applying multiple masks. - - This function when max_length is given: - 1) Sample random mask lengths less than max_length with shape - (batch_size, multiplicity). - 2) Truncate lengths to a max of (choose_range * max_ratio), - so that each mask is fully contained within the corresponding sequence. - 3) Random sample start points of shape (batch_size, multiplicity) - with in (choose_range - lengths). - 4) For each batch, multiple masks (whose number is given by the - multiplicity) are constructed. - 5) Return a mask of shape (batch_size, mask_size) where masks are - obtained by composing the masks constructed in step 4). - If masks_per_frame > 0, the number is given by - min(masks_per_frame * choose_range, multiplicity). - If not, all the masks are composed. The masked regions are set to zero. - - This function when max_length is not given: - 1) Sample random mask lengths less than (choose_range * max_ratio) - with shape (batch_size, multiplicity). - 2) Proceed to steps 3), 4) and 5) of the above. - - Args: - batch_size: Batch size. Integer number. - choose_range: Range within which the masked entries must lie. Tensor of - shape (batch_size,). - mask_size: Size of the mask. Integer number. - global_seed: an integer seed tensor for stateless random ops. - max_length: Maximum number of allowed consecutive masked entries. Integer - number or None. - masks_per_frame: Number of masks per frame. Float number. If > 0, the - multiplicity of the mask is set to be masks_per_frame * choose_range. - multiplicity: Maximum number of total masks. Integer number. - dtype: Data type. - max_ratio: Maximum portion of the entire range allowed to be masked. Float - number. - - Returns: - mask: a fixed size multi-mask starting from a random position with shape - (batch_size, mask_size). - """ - p = self.params - # Non-empty random seed values are only used for testing or when using - # stateless random ops. seed_1 and seed_2 are set separately to avoid - # correlation of mask size and mask position. - if p.use_input_dependent_random_seed: - seed_1 = global_seed + 1 - seed_2 = global_seed + 2 - elif p.random_seed: - seed_1 = p.random_seed + 1 - seed_2 = 2 * p.random_seed - else: - seed_1 = p.random_seed - seed_2 = p.random_seed - # Sample lengths for multiple masks. - if max_length and max_length > 0: - max_length = tf.broadcast_to(tf.cast(max_length, dtype), (batch_size,)) - else: - max_length = tf.cast(choose_range, dtype=dtype) * max_ratio - random_uniform = _random_uniform_op(p.use_input_dependent_random_seed) - masked_portion = random_uniform( - shape=(batch_size, multiplicity), - minval=0.0, - maxval=1.0, - dtype=dtype, - seed=seed_1) - masked_frame_size = self.EinsumBBmBm(max_length, masked_portion) - masked_frame_size = tf.cast(masked_frame_size, dtype=tf.int32) - # Make sure the sampled length was smaller than max_ratio * length_bound. - # Note that sampling in this way was biased - # (shorter sequence may over-masked.) - choose_range = tf.expand_dims(choose_range, -1) - choose_range = tf.tile(choose_range, [1, multiplicity]) - length_bound = tf.cast(choose_range, dtype=dtype) - length_bound = tf.cast(max_ratio * length_bound, dtype=tf.int32) - length = tf.minimum(masked_frame_size, tf.maximum(length_bound, 1)) - - # Choose starting point. - random_start = random_uniform( - shape=(batch_size, multiplicity), maxval=1.0, seed=seed_2) - start_with_in_valid_range = random_start * tf.cast( - (choose_range - length + 1), dtype=dtype) - start = tf.cast(start_with_in_valid_range, tf.int32) - end = start + length - 1 - - # Shift starting and end point by small value. - delta = tf.constant(0.1) - start = tf.expand_dims(tf.cast(start, dtype) - delta, -1) - start = tf.tile(start, [1, 1, mask_size]) - end = tf.expand_dims(tf.cast(end, dtype) + delta, -1) - end = tf.tile(end, [1, 1, mask_size]) - - # Construct pre-mask of shape (batch_size, multiplicity, mask_size). - diagonal = tf.expand_dims( - tf.expand_dims(tf.cast(tf.range(mask_size), dtype=dtype), 0), 0) - diagonal = tf.tile(diagonal, [batch_size, multiplicity, 1]) - pre_mask = tf.cast( - tf.math.logical_and(diagonal < end, diagonal > start), dtype=dtype) - - # Sum masks with appropriate multiplicity. - if masks_per_frame > 0: - multiplicity_weights = tf.tile( - tf.expand_dims(tf.range(multiplicity, dtype=dtype), 0), - [batch_size, 1]) - multiplicity_tensor = masks_per_frame * tf.cast(choose_range, dtype=dtype) - multiplicity_weights = tf.cast( - multiplicity_weights < multiplicity_tensor, dtype=dtype) - pre_mask = self.EinsumBmtBmBt(pre_mask, multiplicity_weights) - else: - pre_mask = tf.reduce_sum(pre_mask, 1) - mask = tf.cast(1.0 - tf.cast(pre_mask > 0, dtype=dtype), dtype=dtype) - - if p.fprop_dtype is not None and p.fprop_dtype != p.dtype: - mask = tf.cast(mask, p.fprop_dtype) - - return mask - - def _GetWarpMatrix(self, - batch_size, - choose_range, - matrix_size, - global_seed, - max_warp_frames=None, - dtype=tf.float32, - max_ratio=1.0): - """Returns warp matrices starting from random positions. - - In this function when max_warp_frames != None: - 1) Sample random warp displacements from the interval - [-max_warp_frames, max_warp_frames) to yield shift tensor - with shape (batch_size,). - 2) Truncate lengths to a maximum magnitude of (choose_range * max_ratio), - so that each shift is fully contained within the - corresponding sequence. - 3) Random sample origin points of shape (batch_size, multiplicity) - with in [shift, choose_range - shift). - 4) Return a batch of 1-D linear maps that fix the boundary points and - shift the origin point by the shift. - - When max_warp_frames == None: - 1) Sample random warp displacements with magnitudes less than - (choose_range * max_ratio) to yield shift tensor with - shape (batch_size,). - 2) Proceed through steps 3), 4). - - Args: - batch_size: Batch size. Integer number. - choose_range: Range within which the warp reference points must lie. - Tensor of shape (batch_size,). - matrix_size: Dimension of vector space warp matrix is applied to. Integer - number. - global_seed: an integer seed tensor for stateless random ops. - max_warp_frames: Upper-bound on the warp distance. Integer or None. - dtype: Data type. - max_ratio: Maximum ratio between the shift distance and choose_range. - Float number. - - Returns: - warp_matrix: An array of fixed size warp matrices with shape - (batch_size, matrix_size, matrix_size). - """ - p = self.params - # Non-empty random seed values are only used for testing or when using - # stateless random ops. seed_3, seed_4, and seed_5 are set separately to - # avoid correlation of warp magnitude and origin position. - if p.use_input_dependent_random_seed: - seed_3 = global_seed + 3 - seed_4 = global_seed + 4 - seed_5 = global_seed + 5 - elif p.random_seed: - seed_3 = p.random_seed - 1 - seed_4 = p.random_seed - 1 - seed_5 = 2 * p.random_seed + 1 - else: - seed_3 = p.random_seed - seed_4 = p.random_seed - seed_5 = p.random_seed - - choose_range_dtype = tf.cast(choose_range, dtype=dtype) - length_upper_bound = tf.cast(max_ratio * choose_range_dtype, dtype=tf.int32) - # Set shift length. - - random_uniform = _random_uniform_op(p.use_input_dependent_random_seed) - - if max_warp_frames and max_warp_frames > 0: - shift = random_uniform( - shape=(batch_size,), - minval=-1 * max_warp_frames, - maxval=max_warp_frames + 1, - dtype=tf.int32, - seed=seed_3) - else: - random_ratio = random_uniform( - shape=(batch_size,), - minval=-1.0, - maxval=1.0, - dtype=dtype, - seed=seed_4) - shift = tf.cast(random_ratio * tf.cast(length_upper_bound, dtype=dtype), - tf.int32) - # Make sure the sampled length was smaller than max_ratio * length_bound. - # Note that sampling in this way is biased. - # (Shorter sequence may over-masked.) - final_shift = tf.maximum(-length_upper_bound, - tf.minimum(shift, length_upper_bound)) - # Choose origin anchor point. - mid_range = tf.cast(choose_range, dtype=tf.int32) - mid_range = tf.maximum(choose_range - 2, 0) - random_origin = random_uniform(shape=(batch_size,), maxval=1.0, seed=seed_5) - origin_with_in_valid_range = random_origin * tf.cast(mid_range, dtype=dtype) - origin = tf.cast(origin_with_in_valid_range, tf.int32) + 1 - # Set destination point of the origin anchor point under the warp map. - destination = origin + final_shift - # Cast origin and destination. - origin = tf.cast(origin, dtype=dtype) - destination = tf.cast(destination, dtype=dtype) - - return self._ConstructWarpMatrix( - batch_size=batch_size, - matrix_size=matrix_size, - origin=origin, - destination=destination, - choose_range=choose_range_dtype, - dtype=dtype) - - def _ConstructWarpMatrix(self, batch_size, matrix_size, origin, destination, - choose_range, dtype): - """Returns warp matrices according to origin, destination and choose_range. - - This function constructs a batch of warp matrices which maps the batch - of origin points to the batch of destination points with fixed boundary - coordinates at 0 and choose_range. - - The warping function, defined by the origin anchor point `origin`, - the destination of the origin anchor point `destination` and the - length of the domain in the warping axis `choose_range` is a piecewise - linear map that fixes the points 0 and `choose_range` and maps - `origin` to `destination`. - - For the warping matrix to be non-singular, destination must lie in the - range 1<= destination <= choose_range - 1, so a destination - out of this range is adjusted to be in this range before the warping - matrix is constructed. - - The warping map can be explicitly written by first defining the slopes: - 1) slope_0 = origin / destination. - 2) slope_1 = (choose_range - origin) / (choose_range - destination). - 3) slope_2 = 1.0. - - Then the origin point orig_i of the mapped coordinate i is given by: - 1) i < destination: orig_i = slope_0 * i. - 2) destination <= i < choose_range: - orig_i = slope_1 * i - (slope_1 - slope_0) * destination. - 3) i >= choose_range: orig_i = i. - - Denoting n_i = ceil(orig_i), the warp matrix element warp[i][j] is given by: - 1) j = n_i: 1 - n_i + orig_i. - 2) j = n_i - 1: n_i - orig_i. - 3) Otherwise: 0. - - Applying the warp matrix to an array of pixels, i.e., - warped_pixel[i] = sum_j warp[i][j] * pixel[j], one would get - warped_pixel[i] = (n_i-orig_i) pixel[n_i-1] + (1-n_i+orig_i) pixel[n_i]. - - Args: - batch_size: Batch size. Integer number. - matrix_size: Dimension of the vector space the warp matrix is applied to. - Integer number. - origin: Origin anchor point for warping. Tensor of shape (batch_size,) and - data type dtype. - destination: Destination of the origin anchor point upon warping. Tensor - of shape (batch_size,) and data type dtype. - choose_range: Range within which the warp reference points must lie. - Tensor of shape (batch_size,) data type dtype. - dtype: Data type of origin, destination, choose_range and the output warp - matrix. - - Returns: - warp_matrix: An array of fixed size warp matrices with shape - (batch_size, matrix_size, matrix_size). - """ - p = self.params - - # Entries of destination must be in the range - # 1 <= destination <= choose_range - 1 - # for warp matrix to have non-singular values. - destination = tf.minimum(tf.maximum(destination, 1.0), choose_range - 1.0) - - # Construct piece-wise linear function fixing boundary points - # specified by zero, choose_range and matrix size and maps - # the origin anchor point to the destination. - destination_bc = tf.broadcast_to(destination, (matrix_size, batch_size)) - destination_bc = tf.transpose(destination_bc) - choose_range_bc = tf.broadcast_to(choose_range, (matrix_size, batch_size)) - choose_range_bc = tf.transpose(choose_range_bc) - - # Slopes of piece-wise linear function. - slope_0 = origin / destination - slope_1 = (choose_range - origin) / (choose_range - destination) - slope_2 = 1.0 - - # x is a batch of origin matrices. - # The origin matrix is the matrix such that - # origin[i][j] = Origin coordinate of coordinate i for the warp map. - # Denoting the destination of the origin anchor point in the - # warp map as "dest," the origin coordinate of point i is given by: - # 1) i < dest: slope_0 * i. - # 2) dest <= i < choose_range: slope_1 * i - (slope_1 - slope_0) * dest. - # 3) i >= choose_range: i. - x = tf.broadcast_to( - tf.cast(tf.range(matrix_size), dtype=dtype), (batch_size, matrix_size)) - x = ( - self.EinsumBBmBm(slope_0, x) + - self.EinsumBBmBm(slope_1 - slope_0, tf.nn.relu(x - destination_bc)) + - self.EinsumBBmBm(slope_2 - slope_1, tf.nn.relu(x - choose_range_bc))) - x = tf.broadcast_to(x, (matrix_size, batch_size, matrix_size)) - x = tf.transpose(x, perm=[1, 2, 0]) - - # y is a batch of coordinate matrices. - # A coordinate matrix is a matrix such that - # coordinate[i][j] = j. - y = tf.broadcast_to( - tf.cast(tf.range(matrix_size), dtype=dtype), - (batch_size, matrix_size, matrix_size)) - # Warp matrix is obtained by applying hat function element-wise to (x-y). - # Denoting the origin point of i under the warp map as orig_i, - # and n_i = ceil(orig_i), the warp matrix element warp[i][j] is given by: - # 1) j = n_i: 1 - n_i + orig_i. - # 2) j = n_i - 1: n_i - orig_i. - # 3) Otherwise: 0. - # Applying the warp matrix to pixels, i.e., - # warped_pixel[i] = sum_j warp[i][j] * original_pixel[j], one would get - # warped_pixel[i] = (n_i - orig_i) * original_pixel[n_i-1] - # + (1 - n_i + orig_i) * original_pixel[n_i]. - warp_matrix = x - y - warp_matrix = _hat(warp_matrix) - if p.fprop_dtype is not None and p.fprop_dtype != dtype: - warp_matrix = tf.cast(warp_matrix, p.fprop_dtype) - - return warp_matrix - - def _FrequencyMask(self, - inputs, - global_seed, - dtype=tf.float32, - domain_id_index=0): - """Applies frequency masking with given degree to inputs. - - Args: - inputs: Batch of input features of shape (batch_size, time_length, - num_freq, channels). - global_seed: an integer seed tensor for stateless random ops. - dtype: Data type. - domain_id_index: domain id index. - - Returns: - Inputs with random frequency masking applied. - """ - p = self.params - - # Mask parameters. - freq_mask_max_bins = p.freq_mask_max_bins[domain_id_index] - multiplicity = p.freq_mask_count[domain_id_index] - - # If masking length or count is zero, do nothing. - if freq_mask_max_bins == 0 or multiplicity == 0: - return inputs - - # Arguments to pass to mask generator. - batch_size, _, num_freq, _ = py_utils.GetShape(inputs) - choose_range = tf.cast( - tf.broadcast_to(num_freq, (batch_size,)), dtype=tf.int32) - # Create masks in frequency direction and apply. - block_arrays = self._GetMask( - tf.shape(inputs)[0], - choose_range=choose_range, - mask_size=num_freq, - global_seed=global_seed, - max_length=freq_mask_max_bins, - masks_per_frame=0.0, - multiplicity=multiplicity, - dtype=dtype, - max_ratio=1.0) - return self.EinsumBxycByBxyc(inputs, block_arrays) - - def _TimeMask(self, - inputs, - seq_lengths, - global_seed, - noisify=False, - gaussian_noise=False, - dtype=tf.float32, - domain_id_index=0): - """Applies time masking with given degree to inputs. - - Args: - inputs: Batch of input features of shape (batch_size, time_length, - num_freq, channels). - seq_lengths: The actual sequence lengths which mask been sampled of shape - (batch_size,). - global_seed: an integer seed tensor for stateless random ops. - noisify: Whether to noisify the masked out regions. - gaussian_noise: Whether to use gaussian noise when noisifying. - dtype: Data type. - domain_id_index: domain id index. - - Returns: - Inputs with random time masking applied. - """ - p = self.params - - # Get time masking parameters. - time_mask_max_frames = p.time_mask_max_frames[domain_id_index] - time_masks_per_frame = p.time_masks_per_frame[domain_id_index] - use_dynamic_time_mask_max_frames = \ - p.use_dynamic_time_mask_max_frames[domain_id_index] - multiplicity = p.time_mask_count[domain_id_index] - max_ratio = p.time_mask_max_ratio[domain_id_index] - - # If maximum mask length is zero, do nothing. - if ((time_mask_max_frames == 0 and not use_dynamic_time_mask_max_frames) or - max_ratio <= 0.0): - return inputs - if multiplicity == 0: - return inputs - seq_lengths = tf.cast(seq_lengths, tf.int32) - batch_size, time_length, _, _ = py_utils.GetShape(inputs) - - # When using dynamic time mask size, discard upper-bound on - # maximum allowed frames for time mask. - if use_dynamic_time_mask_max_frames: - time_mask_max_frames = None - # Create masks in time direction and apply. - block_arrays = self._GetMask( - batch_size, - choose_range=seq_lengths, - mask_size=time_length, - global_seed=global_seed, - max_length=time_mask_max_frames, - masks_per_frame=time_masks_per_frame, - multiplicity=multiplicity, - dtype=dtype, - max_ratio=max_ratio) - - # Non-empty random seed values are only used for testing or when using - # stateless random ops. seed_6 and seed_7 are set separately to avoid - # correlation of warp magnitude and origin position. - if p.use_input_dependent_random_seed: - seed_6 = global_seed + 6 - seed_7 = global_seed + 7 - else: - seed_6 = p.random_seed - seed_7 = p.random_seed - - outputs = self.EinsumBxycBxBxyc( - inputs, block_arrays, name='einsum_formasking') - if noisify: - # Sample noise with standard deviation with factor * 0.1 + 0.0001 - # TODO(ngyuzh): Make sure this won't affect EOS. - if gaussian_noise: - stddev = 1.0 - else: - random_uniform = _random_uniform_op(p.use_input_dependent_random_seed) - factor = random_uniform( - shape=(), minval=1.0, maxval=2.0, dtype=dtype, seed=seed_6) - stddev = factor * 0.1 + 0.0001 - random_normal = _random_normal_op(p.use_input_dependent_random_seed) - noise = random_normal( - shape=[tf.shape(inputs)[0], - tf.shape(inputs)[1], - tf.shape(inputs)[2]], - stddev=stddev, - seed=seed_7) - if p.fprop_dtype is not None and p.fprop_dtype != p.dtype: - noise = tf.cast(noise, p.fprop_dtype) - outputs_mask = self.EinsumBxyBxBxy( - noise, 1.0 - block_arrays, name='einsum_fornoisymasking') - outputs = outputs + tf.expand_dims(outputs_mask, -1) - - return outputs - - def _TimeWarp(self, - inputs, - seq_lengths, - global_seed, - dtype=tf.float32, - domain_id_index=0): - """Applies time warping with given degree to inputs. - - Args: - inputs: Batch of input features of shape (batch_size, time_length, - num_freq, channels). - seq_lengths: The actual sequence lengths which mask been sampled of shape - (batch_size,). - global_seed: an integer seed tensor for stateless random ops. - dtype: Data type. - domain_id_index: Domain ID index. - - Returns: - Inputs with random time warping applied. - """ - p = self.params - batch_size, time_length, _, _ = py_utils.GetShape(inputs) - - # Get parameters for warping. - time_warp_max_frames = p.time_warp_max_frames[domain_id_index] - max_ratio = p.time_warp_max_ratio[domain_id_index] - time_warp_bound = p.time_warp_bound[domain_id_index] - assert time_warp_bound in ('static', 'dynamic') - - # If maximum warp length is zero, do nothing. - if ((time_warp_max_frames == 0 and time_warp_bound == 'static') or - max_ratio <= 0.0): - return inputs - seq_lengths = tf.cast(seq_lengths, tf.int32) - - # Discard upper-bound on time-warp frames when - # dynamic time warping is used. - if time_warp_bound == 'dynamic': - time_warp_max_frames = None - - # Create warping matrix in time direction and apply - warp_matrix = self._GetWarpMatrix( - batch_size, - choose_range=seq_lengths, - matrix_size=time_length, - global_seed=global_seed, - max_warp_frames=time_warp_max_frames, - dtype=dtype, - max_ratio=max_ratio) - - return self.EinsumBxycBzxBzyc(inputs, warp_matrix, name='einsum_forwarping') - - def UnstackFeatures(self, src_inputs, src_paddings): - """Unstacks src_input and src_paddings based off stack height.""" - sh = self.params.stack_height - bs, old_series_length, _, channels = py_utils.GetShape(src_inputs) - unstacked_series_length = old_series_length * sh - src_inputs = tf.reshape(src_inputs, - [bs, unstacked_series_length, -1, channels]) - content = 1 - src_paddings - lengths = tf.cast(sh * tf.reduce_sum(content, axis=1), tf.int32) - mask = tf.sequence_mask(lengths, maxlen=unstacked_series_length) - src_paddings = 1 - tf.cast(mask, tf.int32) - return src_inputs, src_paddings - - def _AugmentationNetwork(self, - series_length, - inputs, - paddings, - global_seed, - domain_id_index=0): - """Returns augmented features. - - Args: - series_length: Total length of time series. - inputs: Batch of input features of shape (batch_size, time_length, - num_freq, channels). - paddings: Batch of padding vectors of shape (batch_size, time_length). - global_seed: an integer seed tensor for stateless random ops. - domain_id_index: domain id index. - - Returns: - Batch of output features of shape (batch_size, time_length, num_freq, - channels) obtained by applying random augmentations to inputs. - """ - p = self.params - dtype = p.dtype - - # Unstack the features. - if p.unstack: - inputs, paddings = self.UnstackFeatures(inputs, paddings) - - lengths = tf.reduce_sum(1 - paddings, 1) - inputs = self._TimeWarp( - inputs, - lengths, - global_seed=global_seed, - dtype=dtype, - domain_id_index=domain_id_index) - inputs = self._TimeMask( - inputs, - lengths, - global_seed=global_seed, - noisify=p.use_noise, - gaussian_noise=p.gaussian_noise, - dtype=dtype, - domain_id_index=domain_id_index) - inputs = self._FrequencyMask( - inputs, - global_seed=global_seed, - dtype=dtype, - domain_id_index=domain_id_index) - - # Restack the features after applying specaugment. - if p.unstack: - inputs = tf.reshape( - inputs, [tf.shape(inputs)[0], series_length, -1, - tf.shape(inputs)[3]]) - - return inputs - - def FProp(self, theta, inputs, paddings, domain_ids=None): - """Applies data augmentation by randomly mask spectrum in inputs. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - inputs: A tensor of shape [batch, time, freq, num_channels]. - paddings: A 0/1 tensor of shape [batch, time]. - domain_ids: input domain_ids of shape [batch, time]. - - Returns: - A pair of 2 tensors: - - - augmented_inputs: A tensor of shape [batch, time, freq, num_channels]. - - paddings: A 0/1 tensor of shape [batch, time]. - """ - p = self.params - - global_seed = None # A tensor seed in case stateless random ops are needed. - if p.use_input_dependent_random_seed: - global_seed = _global_seed_from_inputs(inputs) - - batch_size, series_length, _, _ = py_utils.GetShape(inputs) - if len(p.domain_ids) > 1: - augmented_inputs = tf.zeros_like(inputs) - original_inputs = inputs - for i, domain_id in enumerate(p.domain_ids): - augmented_domain = self._AugmentationNetwork( - series_length, - inputs, - paddings, - global_seed=global_seed, - domain_id_index=i) - target_domain = tf.cast( - tf.expand_dims(tf.tile([domain_id], [batch_size]), -1), - dtype=p.dtype) - # [batch, time]. - domain_mask = tf.cast( - tf.equal(domain_ids, target_domain), dtype=p.dtype) - augmented_domain = self.EinsumBxycBxBxyc( - augmented_domain, domain_mask, name='einsum_domainmasking') - original_inputs = self.EinsumBxycBxBxyc( - original_inputs, 1.0 - domain_mask, name='einsum_domainmasking2') - augmented_inputs = augmented_domain + augmented_inputs - augmented_inputs = original_inputs + augmented_inputs - else: - augmented_inputs = self._AugmentationNetwork( - series_length, - inputs, - paddings, - global_seed=global_seed, - domain_id_index=0) - return augmented_inputs, paddings diff --git a/lingvo/core/spectrum_augmenter_on_device.py b/lingvo/core/spectrum_augmenter_on_device.py deleted file mode 100644 index 6be9a8b9..00000000 --- a/lingvo/core/spectrum_augmenter_on_device.py +++ /dev/null @@ -1,57 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Lingvo layers that are used for spectrum augmentation on-device.""" - -import lingvo.compat as tf -from lingvo.core import spectrum_augmenter - - -class SpectrumAugmenterOnDevice(spectrum_augmenter.SpectrumAugmenter): - """Performs data augmentation as according to the SpecAug paper. - - This implementation uses portable replacements for the tf.einsum ops. - - https://arxiv.org/pdf/1904.08779.pdf - """ - - def EinsumBBmBm(self, a, b, name=None): - """Portable replacement for tf.einsum('b,bm->bm', a, b).""" - return tf.math.multiply(tf.expand_dims(a, axis=-1), b, name=name) - - def EinsumBmtBmBt(self, a, b, name=None): - """Portable replacement for tf.einsum('bmt,bm->bt', a, b).""" - return tf.linalg.matvec(a, b, transpose_a=True, name=name) - - def EinsumBxycByBxyc(self, a, b, name=None): - """Portable replacement for tf.einsum('bxyc,by->bxyc', a, b).""" - expanded_b = tf.expand_dims(tf.expand_dims(b, axis=1), axis=3) - return tf.math.multiply(a, expanded_b, name=name) - - def EinsumBxycBxBxyc(self, a, b, name=None): - """Portable replacement for tf.einsum('bxyc,bx->bxyc', a, b).""" - expanded_b = tf.expand_dims(tf.expand_dims(b, axis=2), axis=3) - return tf.math.multiply(a, expanded_b, name=name) - - def EinsumBxyBxBxy(self, a, b, name=None): - """Portable replacement for tf.einsum('bxy,bx->bxy', a, b).""" - return tf.math.multiply(a, tf.expand_dims(b, axis=2), name=name) - - def EinsumBxycBzxBzyc(self, a, b, name=None): - """Portable replacement for tf.einsum('bxyc,bzx->bzyc', a, b).""" - expanded_a = tf.expand_dims(a, axis=1) - expanded_b = tf.expand_dims(tf.expand_dims(b, axis=-1), axis=-1) - return tf.reduce_sum( - tf.math.multiply(expanded_a, expanded_b, name=name), axis=2) diff --git a/lingvo/core/spectrum_augmenter_on_device_test.py b/lingvo/core/spectrum_augmenter_on_device_test.py deleted file mode 100644 index 8547be3f..00000000 --- a/lingvo/core/spectrum_augmenter_on_device_test.py +++ /dev/null @@ -1,492 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for spectrum augmenter layer.""" - -import lingvo.compat as tf -from lingvo.core import spectrum_augmenter -from lingvo.core import spectrum_augmenter_on_device -from lingvo.core import test_utils -import numpy as np - - -class SpectrumAugmenterTest(test_utils.TestCase): - - def testSpectrumAugmenterWithTimeMask(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(127) - batch_size = 5 - inputs = tf.ones([batch_size, 20, 2, 2], dtype=tf.float32) - paddings = [] - for i in range(batch_size): - paddings.append( - tf.concat([tf.zeros([1, i + 12]), - tf.ones([1, 8 - i])], axis=1)) - paddings = tf.concat(paddings, axis=0) - hs = [] - for p in [ - spectrum_augmenter.SpectrumAugmenter.Params(), - spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - ]: - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_frames = 5 - p.time_mask_count = 2 - p.time_mask_max_ratio = 1.0 - p.random_seed = 23456 - specaug_layer = p.Instantiate() - - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - hs.append(h) - layer_output, layer_output_on_device = self.evaluate(hs) - self.assertAllClose(layer_output, layer_output_on_device) - - def testSpectrumAugmenterDynamicSizeTimeMask(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(127) - batch_size = 3 - inputs = tf.ones([batch_size, 20, 2, 2], dtype=tf.float32) - paddings = [] - for i in range(batch_size): - paddings.append( - tf.concat([tf.zeros([1, 8 * i + 3]), - tf.ones([1, 17 - 8 * i])], - axis=1)) - paddings = tf.concat(paddings, axis=0) - hs = [] - for p in [ - spectrum_augmenter.SpectrumAugmenter.Params(), - spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - ]: - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_ratio = 0.4 - p.time_mask_count = 1 - p.use_dynamic_time_mask_max_frames = True - p.random_seed = 12345 - specaug_layer = p.Instantiate() - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - hs.append(h) - layer_output, layer_output_on_device = self.evaluate(hs) - self.assertAllClose(layer_output, layer_output_on_device) - - def testSpectrumAugmenterDynamicMultiplicityTimeMask(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(127) - batch_size = 4 - inputs = tf.ones([batch_size, 22, 2, 2], dtype=tf.float32) - paddings = [] - for i in range(batch_size): - paddings.append( - tf.concat([tf.zeros([1, 5 * i + 5]), - tf.ones([1, 16 - 5 * i])], - axis=1)) - paddings = tf.concat(paddings, axis=0) - hs = [] - for p in [ - spectrum_augmenter.SpectrumAugmenter.Params(), - spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - ]: - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_frames = 5 - p.time_mask_count = 10 - p.time_masks_per_frame = 0.2 - p.random_seed = 67890 - specaug_layer = p.Instantiate() - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - hs.append(h) - layer_output, layer_output_on_device = self.evaluate(hs) - self.assertAllClose(layer_output, layer_output_on_device) - - def testSpectrumAugmenterDynamicSizeAndMultiplicityTimeMask(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(127) - batch_size = 4 - inputs = tf.ones([batch_size, 22, 2, 2], dtype=tf.float32) - paddings = [] - for i in range(batch_size): - paddings.append( - tf.concat([tf.zeros([1, 5 * i + 5]), - tf.ones([1, 16 - 5 * i])], - axis=1)) - paddings = tf.concat(paddings, axis=0) - hs = [] - for p in [ - spectrum_augmenter.SpectrumAugmenter.Params(), - spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - ]: - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_frames = 5 - p.time_mask_count = 10 - p.time_masks_per_frame = 0.2 - p.time_mask_max_ratio = 0.4 - p.use_dynamic_time_mask_max_frames = True - p.random_seed = 67890 - specaug_layer = p.Instantiate() - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - hs.append(h) - layer_output, layer_output_on_device = self.evaluate(hs) - self.assertAllClose(layer_output, layer_output_on_device) - - def testSpectrumAugmenterWithFrequencyMask(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(1234) - inputs = tf.ones([3, 5, 10, 1], dtype=tf.float32) - paddings = tf.zeros([3, 5]) - hs = [] - for p in [ - spectrum_augmenter.SpectrumAugmenter.Params(), - spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - ]: - p.name = 'specAug_layers' - p.freq_mask_max_bins = 6 - p.freq_mask_count = 2 - p.time_mask_max_frames = 0 - p.random_seed = 34567 - specaug_layer = p.Instantiate() - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - hs.append(h) - layer_output, layer_output_on_device = self.evaluate(hs) - self.assertAllClose(layer_output, layer_output_on_device) - - def testSpectrumAugmenterWarpMatrixConstructor(self): - with self.session(use_gpu=False, graph=tf.Graph()): - inputs = tf.broadcast_to(tf.cast(tf.range(10), dtype=tf.float32), (4, 10)) - origin = tf.cast([2, 4, 4, 5], dtype=tf.float32) - destination = tf.cast([3, 2, 6, 8], dtype=tf.float32) - choose_range = tf.cast([4, 8, 8, 10], dtype=tf.float32) - outputs = [] - for p in [ - spectrum_augmenter.SpectrumAugmenter.Params(), - spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - ]: - p.name = 'specAug_layers' - specaug_layer = p.Instantiate() - warp_matrix = specaug_layer._ConstructWarpMatrix( - batch_size=4, - matrix_size=10, - origin=origin, - destination=destination, - choose_range=choose_range, - dtype=tf.float32) - output = tf.einsum('bij,bj->bi', warp_matrix, inputs) - outputs.append(output) - layer_output, layer_output_on_device = self.evaluate(outputs) - self.assertAllClose(layer_output, layer_output_on_device) - - def testSpectrumAugmenterWithTimeWarping(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(1234) - inputs = tf.broadcast_to(tf.cast(tf.range(10), dtype=tf.float32), (3, 10)) - inputs = tf.expand_dims(tf.expand_dims(inputs, -1), -1) - paddings = [] - for i in range(3): - paddings.append( - tf.concat([tf.zeros([1, i + 7]), - tf.ones([1, 3 - i])], axis=1)) - paddings = tf.concat(paddings, axis=0) - hs = [] - for p in [ - spectrum_augmenter.SpectrumAugmenter.Params(), - spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - ]: - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_frames = 0 - p.time_warp_max_frames = 8 - p.time_warp_max_ratio = 1.0 - p.time_warp_bound = 'static' - p.random_seed = 34567 - specaug_layer = p.Instantiate() - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - hs.append(h) - layer_output, layer_output_on_device = self.evaluate(hs) - self.assertAllClose(layer_output, layer_output_on_device) - - def testSpectrumAugmenterWithDynamicTimeWarping(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(1234) - inputs = tf.broadcast_to(tf.cast(tf.range(10), dtype=tf.float32), (3, 10)) - inputs = tf.expand_dims(tf.expand_dims(inputs, -1), -1) - paddings = [] - for i in range(3): - paddings.append( - tf.concat([tf.zeros([1, 2 * i + 5]), - tf.ones([1, 5 - 2 * i])], - axis=1)) - paddings = tf.concat(paddings, axis=0) - hs = [] - for p in [ - spectrum_augmenter.SpectrumAugmenter.Params(), - spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - ]: - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_frames = 0 - p.time_warp_max_ratio = 0.5 - p.time_warp_bound = 'dynamic' - p.random_seed = 34567 - specaug_layer = p.Instantiate() - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - hs.append(h) - layer_output, layer_output_on_device = self.evaluate(hs) - self.assertAllClose(layer_output, layer_output_on_device) - - def testSpectrumAugmenterUnstacking(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(1234) - inputs = tf.ones([3, 5, 10, 1], dtype=tf.float32) - paddings = tf.zeros([3, 5]) - hs = [] - for p in [ - spectrum_augmenter.SpectrumAugmenter.Params(), - spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - ]: - p.name = 'specAug_layers' - p.unstack = True - p.stack_height = 2 - p.freq_mask_max_bins = 5 - p.time_mask_max_frames = 8 - p.random_seed = 12345 - specaug_layer = p.Instantiate() - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - hs.append(h) - layer_output, layer_output_on_device = self.evaluate(hs) - self.assertAllClose(layer_output, layer_output_on_device) - - def testSpectrumAugmenterWithPerDomainPolicyFreqMask(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(1234) - inputs = tf.ones([6, 5, 4, 2], dtype=tf.float32) - input_domain_ids = tf.constant( - [[1] * 5, [2] * 5, [0] * 5, [2] * 5, [0] * 5, [1] * 5], - dtype=tf.float32) - paddings = tf.zeros([3, 5]) - hs = [] - for p in [ - spectrum_augmenter.SpectrumAugmenter.Params(), - spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - ]: - p.name = 'specAug_layers' - p.domain_ids = [0, 1, 2] - p.freq_mask_max_bins = [0, 3, 8] - p.time_mask_max_frames = 0 - p.random_seed = 1234 - specaug_layer = p.Instantiate() - h, _ = specaug_layer.FPropDefaultTheta( - inputs, paddings, domain_ids=input_domain_ids) - hs.append(h) - layer_output, layer_output_on_device = self.evaluate(hs) - self.assertAllClose(layer_output, layer_output_on_device) - - def testSpectrumAugmenterNoisify(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(127) - batch_size = 2 - inputs = tf.ones([batch_size, 20, 2, 2], dtype=tf.float32) - paddings = [] - for i in range(batch_size): - paddings.append( - tf.concat([tf.zeros([1, 8 * i + 3]), - tf.ones([1, 17 - 8 * i])], - axis=1)) - paddings = tf.concat(paddings, axis=0) - hs = [] - for p in [ - spectrum_augmenter.SpectrumAugmenter.Params(), - spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - ]: - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_ratio = 0.4 - p.time_mask_count = 1 - p.use_dynamic_time_mask_max_frames = True - p.use_noise = True - p.gaussian_noise = False - p.random_seed = 12345 - specaug_layer = p.Instantiate() - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - hs.append(h) - layer_output, layer_output_on_device = self.evaluate(hs) - self.assertAllClose(layer_output, layer_output_on_device) - - def testSpectrumAugmenterGaussianNoisify(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(127) - batch_size = 2 - inputs = tf.ones([batch_size, 20, 2, 2], dtype=tf.float32) - paddings = [] - for i in range(batch_size): - paddings.append( - tf.concat([tf.zeros([1, 8 * i + 3]), - tf.ones([1, 17 - 8 * i])], - axis=1)) - paddings = tf.concat(paddings, axis=0) - hs = [] - for p in [ - spectrum_augmenter.SpectrumAugmenter.Params(), - spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - ]: - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_ratio = 0.4 - p.time_mask_count = 1 - p.use_dynamic_time_mask_max_frames = True - p.use_noise = True - p.gaussian_noise = True - p.random_seed = 12345 - specaug_layer = p.Instantiate() - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - hs.append(h) - layer_output, layer_output_on_device = self.evaluate(hs) - self.assertAllClose(layer_output, layer_output_on_device) - - def testSpectrumAugmenterWithStatelessRandomOps(self): - with self.session(use_gpu=False, graph=tf.Graph()): - batch_size = 5 - inputs1 = tf.random.uniform( - shape=[batch_size, 20, 2, 2], minval=0, maxval=1, dtype=tf.float32) - inputs2 = tf.random.uniform( - shape=[batch_size, 20, 2, 2], minval=0, maxval=1, dtype=tf.float32) - paddings = [] - for i in range(batch_size): - paddings.append( - tf.concat([tf.zeros([1, i + 12]), - tf.ones([1, 8 - i])], axis=1)) - paddings = tf.concat(paddings, axis=0) - - p = spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - p.name = 'specAug_layers' - p.freq_mask_count = 1 - p.freq_mask_max_bins = 1 - p.time_mask_max_frames = 5 - p.time_mask_count = 2 - p.time_mask_max_ratio = 1.0 - p.use_input_dependent_random_seed = True - specaug_layer = p.Instantiate() - h1, _ = specaug_layer.FPropDefaultTheta(inputs1, paddings) - h2, _ = specaug_layer.FPropDefaultTheta(inputs2, paddings) - actual_layer_output1, actual_layer_output2 = self.evaluate([h1, h2]) - self.assertAllEqual( - np.shape(actual_layer_output1), np.array([5, 20, 2, 2])) - self.assertNotAllEqual(actual_layer_output1, actual_layer_output2) - - def testGraphContainsOnDeviceOps(self): - """Checks that einsum and stateful random ops are not used on-device.""" - model_graph = tf.Graph() - with model_graph.as_default(): - batch_size = 5 - inputs = tf.random.stateless_uniform( - shape=[batch_size, 20, 2, 2], - minval=0, - maxval=1, - seed=tf.constant([123, 123]), - dtype=tf.float32) - paddings = [] - for i in range(batch_size): - paddings.append( - tf.concat([tf.zeros([1, i + 12]), - tf.ones([1, 8 - i])], axis=1)) - paddings = tf.concat(paddings, axis=0) - p = spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - p.name = 'specAug_layers' - p.freq_mask_count = 1 - p.freq_mask_max_bins = 1 - p.time_mask_max_frames = 5 - p.time_mask_count = 2 - p.use_noise = True - p.gaussian_noise = True - p.time_mask_max_ratio = 1.0 - p.use_input_dependent_random_seed = True - specaug_layer = p.Instantiate() - _, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - # A list of ops that are not compatible with on-device training. - unsupported_on_device_nodes = [ - 'RandomUniform', 'RandomStandardNormal', 'Einsum' - ] - for node in model_graph.as_graph_def().node: - self.assertNotIn(node.op, unsupported_on_device_nodes) - - def testEinsumReplacementBBmBm(self): - with self.session(use_gpu=False, graph=tf.Graph()): - a = tf.random.uniform(shape=[20], minval=0, maxval=1, dtype=tf.float32) - b = tf.random.uniform( - shape=[20, 10], minval=0, maxval=1, dtype=tf.float32) - einsum = tf.einsum('b,bm->bm', a, b) - p = spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - p.name = 'specAug_layers' - specaug_layer = p.Instantiate() - replacement = specaug_layer.EinsumBBmBm(a, b) - einsum, replacement = self.evaluate([einsum, replacement]) - self.assertAllClose(einsum, replacement) - - def testEinsumReplacementBxycByBxyc(self): - with self.session(use_gpu=False, graph=tf.Graph()): - a = tf.random.uniform( - shape=[20, 5, 7, 4], minval=0, maxval=1, dtype=tf.float32) - b = tf.random.uniform(shape=[20, 7], minval=0, maxval=1, dtype=tf.float32) - einsum = tf.einsum('bxyc,by->bxyc', a, b) - p = spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - p.name = 'specAug_layers' - specaug_layer = p.Instantiate() - replacement = specaug_layer.EinsumBxycByBxyc(a, b) - einsum, replacement = self.evaluate([einsum, replacement]) - self.assertAllClose(einsum, replacement) - - def testEinsumReplacementBxycBxBxyc(self): - with self.session(use_gpu=False, graph=tf.Graph()): - a = tf.random.uniform( - shape=[20, 5, 7, 4], minval=0, maxval=1, dtype=tf.float32) - b = tf.random.uniform(shape=[20, 5], minval=0, maxval=1, dtype=tf.float32) - einsum = tf.einsum('bxyc,bx->bxyc', a, b) - p = spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - p.name = 'specAug_layers' - specaug_layer = p.Instantiate() - replacement = specaug_layer.EinsumBxycBxBxyc(a, b) - einsum, replacement = self.evaluate([einsum, replacement]) - self.assertAllClose(einsum, replacement) - - def testEinsumReplacementBxyBxBxy(self): - with self.session(use_gpu=False, graph=tf.Graph()): - a = tf.random.uniform( - shape=[20, 7, 4], minval=0, maxval=1, dtype=tf.float32) - b = tf.random.uniform(shape=[20, 7], minval=0, maxval=1, dtype=tf.float32) - einsum = tf.einsum('bxy,bx->bxy', a, b) - p = spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - p.name = 'specAug_layers' - specaug_layer = p.Instantiate() - replacement = specaug_layer.EinsumBxyBxBxy(a, b) - einsum, replacement = self.evaluate([einsum, replacement]) - self.assertAllClose(einsum, replacement) - - def testEinsumReplacementBxycBzxBzyc(self): - with self.session(use_gpu=False, graph=tf.Graph()): - a = tf.random.uniform( - shape=[20, 7, 4, 3], minval=0, maxval=1, dtype=tf.float32) - b = tf.random.uniform( - shape=[20, 5, 7], minval=0, maxval=1, dtype=tf.float32) - einsum = tf.einsum('bxyc,bzx->bzyc', a, b) - p = spectrum_augmenter_on_device.SpectrumAugmenterOnDevice.Params() - p.name = 'specAug_layers' - specaug_layer = p.Instantiate() - replacement = specaug_layer.EinsumBxycBzxBzyc(a, b) - einsum, replacement = self.evaluate([einsum, replacement]) - self.assertAllClose(einsum, replacement) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/spectrum_augmenter_test.py b/lingvo/core/spectrum_augmenter_test.py deleted file mode 100644 index a8fa6286..00000000 --- a/lingvo/core/spectrum_augmenter_test.py +++ /dev/null @@ -1,785 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for spectrum augmenter layer.""" - -import lingvo.compat as tf -from lingvo.core import spectrum_augmenter -from lingvo.core import test_utils -import numpy as np - - -class SpectrumAugmenterTest(test_utils.TestCase): - - def testSpectrumAugmenterWithTimeMask(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(127) - batch_size = 5 - inputs = tf.ones([batch_size, 20, 2, 2], dtype=tf.float32) - paddings = [] - for i in range(batch_size): - paddings.append( - tf.concat([tf.zeros([1, i + 12]), - tf.ones([1, 8 - i])], axis=1)) - paddings = tf.concat(paddings, axis=0) - - p = spectrum_augmenter.SpectrumAugmenter.Params() - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_frames = 5 - p.time_mask_count = 2 - p.time_mask_max_ratio = 1.0 - p.random_seed = 23456 - specaug_layer = p.Instantiate() - expected_output = np.array([[[[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]], - [[[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[0., 0.], [0., 0.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]], - [[[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]], - [[[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]], - [[[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]]]) - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - actual_layer_output = self.evaluate(h) - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testSpectrumAugmenterDynamicSizeTimeMask(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(127) - batch_size = 3 - inputs = tf.ones([batch_size, 20, 2, 2], dtype=tf.float32) - paddings = [] - for i in range(batch_size): - paddings.append( - tf.concat([tf.zeros([1, 8 * i + 3]), - tf.ones([1, 17 - 8 * i])], - axis=1)) - paddings = tf.concat(paddings, axis=0) - - p = spectrum_augmenter.SpectrumAugmenter.Params() - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_ratio = 0.4 - p.time_mask_count = 1 - p.use_dynamic_time_mask_max_frames = True - p.random_seed = 12345 - specaug_layer = p.Instantiate() - expected_output = np.array([[[[1., 1.], [1., 1.]], [[0., 0.], [0., 0.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]], - [[[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]], - [[[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]]]) - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - actual_layer_output = self.evaluate(h) - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testSpectrumAugmenterDynamicMultiplicityTimeMask(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(127) - batch_size = 4 - inputs = tf.ones([batch_size, 22, 2, 2], dtype=tf.float32) - paddings = [] - for i in range(batch_size): - paddings.append( - tf.concat([tf.zeros([1, 5 * i + 5]), - tf.ones([1, 16 - 5 * i])], - axis=1)) - paddings = tf.concat(paddings, axis=0) - - p = spectrum_augmenter.SpectrumAugmenter.Params() - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_frames = 5 - p.time_mask_count = 10 - p.time_masks_per_frame = 0.2 - p.random_seed = 67890 - specaug_layer = p.Instantiate() - expected_output = np.array([[[[1., 1.], [1., 1.]], [[0., 0.], [0., 0.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]], - [[[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]], - [[[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[1., 1.], [1., 1.]], - [[0., 0.], [0., 0.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]], - [[[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]]]) - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - actual_layer_output = self.evaluate(h) - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testSpectrumAugmenterDynamicSizeAndMultiplicityTimeMask(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(127) - batch_size = 4 - inputs = tf.ones([batch_size, 22, 2, 2], dtype=tf.float32) - paddings = [] - for i in range(batch_size): - paddings.append( - tf.concat([tf.zeros([1, 5 * i + 5]), - tf.ones([1, 16 - 5 * i])], - axis=1)) - paddings = tf.concat(paddings, axis=0) - - p = spectrum_augmenter.SpectrumAugmenter.Params() - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_frames = 5 - p.time_mask_count = 10 - p.time_masks_per_frame = 0.2 - p.time_mask_max_ratio = 0.4 - p.use_dynamic_time_mask_max_frames = True - p.random_seed = 67890 - specaug_layer = p.Instantiate() - expected_output = np.array([[[[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]], - [[[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]], - [[[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[1., 1.], [1., 1.]], - [[0., 0.], [0., 0.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]], - [[[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[1., 1.], [1., 1.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[0., 0.], [0., 0.]], - [[0., 0.], [0., 0.]], [[1., 1.], [1., 1.]], - [[1., 1.], [1., 1.]], [[1., 1.], [1., 1.]]]]) - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - actual_layer_output = self.evaluate(h) - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testSpectrumAugmenterWithFrequencyMask(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(1234) - inputs = tf.ones([3, 5, 10, 1], dtype=tf.float32) - paddings = tf.zeros([3, 5]) - p = spectrum_augmenter.SpectrumAugmenter.Params() - p.name = 'specAug_layers' - p.freq_mask_max_bins = 6 - p.freq_mask_count = 2 - p.time_mask_max_frames = 0 - p.random_seed = 34567 - specaug_layer = p.Instantiate() - # pyformat: disable - # pylint: disable=bad-whitespace,bad-continuation - expected_output = np.array( - [[[[1.], [1.], [1.], [0.], [0.], [0.], [0.], [0.], [0.], [1.]], - [[1.], [1.], [1.], [0.], [0.], [0.], [0.], [0.], [0.], [1.]], - [[1.], [1.], [1.], [0.], [0.], [0.], [0.], [0.], [0.], [1.]], - [[1.], [1.], [1.], [0.], [0.], [0.], [0.], [0.], [0.], [1.]], - [[1.], [1.], [1.], [0.], [0.], [0.], [0.], [0.], [0.], [1.]]], - [[[0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.], [1.], [1.]], - [[0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.], [1.], [1.]], - [[0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.], [1.], [1.]], - [[0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.], [1.], [1.]], - [[0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.], [1.], [1.]]], - [[[1.], [1.], [0.], [0.], [1.], [1.], [0.], [1.], [1.], [1.]], - [[1.], [1.], [0.], [0.], [1.], [1.], [0.], [1.], [1.], [1.]], - [[1.], [1.], [0.], [0.], [1.], [1.], [0.], [1.], [1.], [1.]], - [[1.], [1.], [0.], [0.], [1.], [1.], [0.], [1.], [1.], [1.]], - [[1.], [1.], [0.], [0.], [1.], [1.], [0.], [1.], [1.], [1.]]]]) - # pylint: enable=bad-whitespace,bad-continuation - # pyformat: enable - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - actual_layer_output = self.evaluate(h) - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testSpectrumAugmenterWarpMatrixConstructor(self): - with self.session(use_gpu=False, graph=tf.Graph()): - inputs = tf.broadcast_to(tf.cast(tf.range(10), dtype=tf.float32), (4, 10)) - origin = tf.cast([2, 4, 4, 5], dtype=tf.float32) - destination = tf.cast([3, 2, 6, 8], dtype=tf.float32) - choose_range = tf.cast([4, 8, 8, 10], dtype=tf.float32) - p = spectrum_augmenter.SpectrumAugmenter.Params() - p.name = 'specAug_layers' - specaug_layer = p.Instantiate() - # pyformat: disable - # pylint: disable=bad-whitespace,bad-continuation - expected_output = np.array( - [[0.0000000, 0.6666667, 1.3333333, 2.0000000, 4.0000000, - 5.0000000, 6.0000000, 7.0000000, 8.0000000, 9.0000000], - [0.0000000, 2.0000000, 4.0000000, 4.6666667, 5.3333333, - 6.0000000, 6.6666667, 7.3333333, 8.0000000, 9.0000000], - [0.0000000, 0.6666667, 1.3333333, 2.0000000, 2.6666667, - 3.3333333, 4.0000000, 6.0000000, 8.0000000, 9.0000000], - [0.0000000, 0.6250000, 1.2500000, 1.8750000, 2.5000000, - 3.1250000, 3.7500000, 4.3750000, 5.0000000, 7.5000000]]) - # pylint: enable=bad-whitespace,bad-continuation - # pyformat: enable - warp_matrix = specaug_layer._ConstructWarpMatrix( - batch_size=4, - matrix_size=10, - origin=origin, - destination=destination, - choose_range=choose_range, - dtype=tf.float32) - outputs = tf.einsum('bij,bj->bi', warp_matrix, inputs) - actual_layer_output = self.evaluate(outputs) - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testSpectrumAugmenterWithTimeWarping(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(1234) - inputs = tf.broadcast_to(tf.cast(tf.range(10), dtype=tf.float32), (3, 10)) - inputs = tf.expand_dims(tf.expand_dims(inputs, -1), -1) - paddings = [] - for i in range(3): - paddings.append( - tf.concat([tf.zeros([1, i + 7]), - tf.ones([1, 3 - i])], axis=1)) - paddings = tf.concat(paddings, axis=0) - p = spectrum_augmenter.SpectrumAugmenter.Params() - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_frames = 0 - p.time_warp_max_frames = 8 - p.time_warp_max_ratio = 1.0 - p.time_warp_bound = 'static' - p.random_seed = 34567 - specaug_layer = p.Instantiate() - # pyformat: disable - # pylint: disable=bad-whitespace,bad-continuation - expected_output = np.array( - [[[[0.0000000]], [[0.6666667]], [[1.3333334]], [[2.0000000]], - [[2.6666667]], [[3.3333335]], [[4.0000000]], [[7.0000000]], - [[8.0000000]], [[9.0000000]]], - [[[0.0000000]], [[3.0000000]], [[6.0000000]], [[6.3333334]], - [[6.6666665]], [[7.0000000]], [[7.3333334]], [[7.6666667]], - [[8.0000000]], [[9.0000000]]], - [[[0.0000000]], [[0.5000000]], [[1.0000000]], [[1.5000000]], - [[2.0000000]], [[3.4000000]], [[4.8000000]], [[6.2000000]], - [[7.6000000]], [[9.0000000]]]]) - # pylint: enable=bad-whitespace,bad-continuation - # pyformat: enable - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - actual_layer_output = self.evaluate(h) - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testSpectrumAugmenterWithDynamicTimeWarping(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(1234) - inputs = tf.broadcast_to(tf.cast(tf.range(10), dtype=tf.float32), (3, 10)) - inputs = tf.expand_dims(tf.expand_dims(inputs, -1), -1) - paddings = [] - for i in range(3): - paddings.append( - tf.concat([tf.zeros([1, 2 * i + 5]), - tf.ones([1, 5 - 2 * i])], - axis=1)) - paddings = tf.concat(paddings, axis=0) - p = spectrum_augmenter.SpectrumAugmenter.Params() - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_frames = 0 - p.time_warp_max_ratio = 0.5 - p.time_warp_bound = 'dynamic' - p.random_seed = 34567 - specaug_layer = p.Instantiate() - # pyformat: disable - # pylint: disable=bad-whitespace,bad-continuation - expected_output = np.array( - [[[[0.0000000]], [[1.0000000]], [[2.0000000]], [[3.0000000]], - [[4.0000000]], [[5.0000000]], [[6.0000000]], [[7.0000000]], - [[8.0000000]], [[9.0000000]]], - [[[0.0000000]], [[0.8333333]], [[1.6666666]], [[2.5000000]], - [[3.3333333]], [[4.1666665]], [[5.0000000]], [[7.0000000]], - [[8.0000000]], [[9.0000000]]], - [[[0.0000000]], [[2.0000000]], [[2.8750000]], [[3.7500000]], - [[4.6250000]], [[5.5000000]], [[6.3750000]], [[7.2500000]], - [[8.1250000]], [[9.0000000]]]]) - # pylint: enable=bad-whitespace,bad-continuation - # pyformat: enable - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - actual_layer_output = self.evaluate(h) - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testSpectrumAugmenterUnstacking(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(1234) - inputs = tf.ones([3, 5, 10, 1], dtype=tf.float32) - paddings = tf.zeros([3, 5]) - p = spectrum_augmenter.SpectrumAugmenter.Params() - p.name = 'specAug_layers' - p.unstack = True - p.stack_height = 2 - p.freq_mask_max_bins = 5 - p.time_mask_max_frames = 8 - p.random_seed = 12345 - specaug_layer = p.Instantiate() - # pyformat: disable - # pylint: disable=bad-whitespace,bad-continuation - expected_output = np.array( - [[[[1.], [1.], [0.], [1.], [1.], [1.], [1.], [0.], [1.], [1.]], - [[1.], [1.], [0.], [1.], [1.], [0.], [0.], [0.], [0.], [0.]], - [[0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.]], - [[1.], [1.], [0.], [1.], [1.], [1.], [1.], [0.], [1.], [1.]], - [[1.], [1.], [0.], [1.], [1.], [1.], [1.], [0.], [1.], [1.]]], - [[[1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.]], - [[1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.]], - [[1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.]], - [[1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.]], - [[1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.]]], - [[[1.], [1.], [0.], [1.], [1.], [1.], [1.], [0.], [1.], [1.]], - [[1.], [1.], [0.], [1.], [1.], [1.], [1.], [0.], [1.], [1.]], - [[1.], [1.], [0.], [1.], [1.], [0.], [0.], [0.], [0.], [0.]], - [[0.], [0.], [0.], [0.], [0.], [1.], [1.], [0.], [1.], [1.]], - [[1.], [1.], [0.], [1.], [1.], [1.], [1.], [0.], [1.], [1.]]]]) - # pylint: enable=bad-whitespace,bad-continuation - # pyformat: enable - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - actual_layer_output = self.evaluate(h) - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testSpectrumAugmenterWithPerDomainPolicyFreqMask(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(1234) - inputs = tf.ones([6, 5, 4, 2], dtype=tf.float32) - input_domain_ids = tf.constant( - [[1] * 5, [2] * 5, [0] * 5, [2] * 5, [0] * 5, [1] * 5], - dtype=tf.float32) - paddings = tf.zeros([3, 5]) - p = spectrum_augmenter.SpectrumAugmenter.Params() - p.name = 'specAug_layers' - p.domain_ids = [0, 1, 2] - p.freq_mask_max_bins = [0, 3, 8] - p.time_mask_max_frames = 0 - p.random_seed = 1234 - specaug_layer = p.Instantiate() - expected_output = np.array([[[[0., 0.], [0., 0.], [1., 1.], [1., 1.]], - [[0., 0.], [0., 0.], [1., 1.], [1., 1.]], - [[0., 0.], [0., 0.], [1., 1.], [1., 1.]], - [[0., 0.], [0., 0.], [1., 1.], [1., 1.]], - [[0., 0.], [0., 0.], [1., 1.], [1., 1.]]], - [[[1., 1.], [0., 0.], [0., 0.], [0., 0.]], - [[1., 1.], [0., 0.], [0., 0.], [0., 0.]], - [[1., 1.], [0., 0.], [0., 0.], [0., 0.]], - [[1., 1.], [0., 0.], [0., 0.], [0., 0.]], - [[1., 1.], [0., 0.], [0., 0.], [0., 0.]]], - [[[1., 1.], [1., 1.], [1., 1.], [1., 1.]], - [[1., 1.], [1., 1.], [1., 1.], [1., 1.]], - [[1., 1.], [1., 1.], [1., 1.], [1., 1.]], - [[1., 1.], [1., 1.], [1., 1.], [1., 1.]], - [[1., 1.], [1., 1.], [1., 1.], [1., 1.]]], - [[[0., 0.], [0., 0.], [0., 0.], [0., 0.]], - [[0., 0.], [0., 0.], [0., 0.], [0., 0.]], - [[0., 0.], [0., 0.], [0., 0.], [0., 0.]], - [[0., 0.], [0., 0.], [0., 0.], [0., 0.]], - [[0., 0.], [0., 0.], [0., 0.], [0., 0.]]], - [[[1., 1.], [1., 1.], [1., 1.], [1., 1.]], - [[1., 1.], [1., 1.], [1., 1.], [1., 1.]], - [[1., 1.], [1., 1.], [1., 1.], [1., 1.]], - [[1., 1.], [1., 1.], [1., 1.], [1., 1.]], - [[1., 1.], [1., 1.], [1., 1.], [1., 1.]]], - [[[1., 1.], [0., 0.], [0., 0.], [1., 1.]], - [[1., 1.], [0., 0.], [0., 0.], [1., 1.]], - [[1., 1.], [0., 0.], [0., 0.], [1., 1.]], - [[1., 1.], [0., 0.], [0., 0.], [1., 1.]], - [[1., 1.], [0., 0.], [0., 0.], [1., 1.]]]]) - h, _ = specaug_layer.FPropDefaultTheta( - inputs, paddings, domain_ids=input_domain_ids) - actual_layer_output = self.evaluate(h) - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testSpectrumAugmenterNoisify(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(127) - batch_size = 2 - inputs = tf.ones([batch_size, 20, 2, 2], dtype=tf.float32) - paddings = [] - for i in range(batch_size): - paddings.append( - tf.concat([tf.zeros([1, 8 * i + 3]), - tf.ones([1, 17 - 8 * i])], - axis=1)) - paddings = tf.concat(paddings, axis=0) - - p = spectrum_augmenter.SpectrumAugmenter.Params() - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_ratio = 0.4 - p.time_mask_count = 1 - p.use_dynamic_time_mask_max_frames = True - p.use_noise = True - p.gaussian_noise = False - p.random_seed = 12345 - specaug_layer = p.Instantiate() - expected_output = np.array([[[[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[-0.00113627, -0.00113627], - [0.08975883, 0.08975883]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]]], - [[[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[0.09341543, 0.09341543], - [-0.11914382, -0.11914382]], - [[0.04238122, 0.04238122], - [0.115249, 0.115249]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]]]]) - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - actual_layer_output = self.evaluate(h) - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testSpectrumAugmenterGaussianNoisify(self): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(127) - batch_size = 2 - inputs = tf.ones([batch_size, 20, 2, 2], dtype=tf.float32) - paddings = [] - for i in range(batch_size): - paddings.append( - tf.concat([tf.zeros([1, 8 * i + 3]), - tf.ones([1, 17 - 8 * i])], - axis=1)) - paddings = tf.concat(paddings, axis=0) - - p = spectrum_augmenter.SpectrumAugmenter.Params() - p.name = 'specAug_layers' - p.freq_mask_max_bins = 0 - p.time_mask_max_ratio = 0.4 - p.time_mask_count = 1 - p.use_dynamic_time_mask_max_frames = True - p.use_noise = True - p.gaussian_noise = True - p.random_seed = 12345 - specaug_layer = p.Instantiate() - expected_output = np.array([[[[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[-0.00798237, -0.00798237], - [0.6305642, 0.6305642]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]]], - [[[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[0.6562522, 0.6562522], - [-0.83699656, -0.83699656]], - [[0.29773206, 0.29773206], - [0.8096351, 0.8096351]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]], - [[1.00000000, 1.00000000], - [1.00000000, 1.00000000]]]]) - h, _ = specaug_layer.FPropDefaultTheta(inputs, paddings) - actual_layer_output = self.evaluate(h) - print(np.array_repr(actual_layer_output)) - self.assertAllClose(actual_layer_output, expected_output) - - def testSpectrumAugmenterWithStatelessRandomOps(self): - with self.session(use_gpu=False, graph=tf.Graph()): - batch_size = 5 - inputs1 = tf.random.uniform( - shape=[batch_size, 20, 2, 2], minval=0, maxval=1, dtype=tf.float32) - inputs2 = tf.random.uniform( - shape=[batch_size, 20, 2, 2], minval=0, maxval=1, dtype=tf.float32) - paddings = [] - for i in range(batch_size): - paddings.append( - tf.concat([tf.zeros([1, i + 12]), - tf.ones([1, 8 - i])], axis=1)) - paddings = tf.concat(paddings, axis=0) - - p = spectrum_augmenter.SpectrumAugmenter.Params() - p.name = 'specAug_layers' - p.freq_mask_count = 1 - p.freq_mask_max_bins = 1 - p.time_mask_max_frames = 5 - p.time_mask_count = 2 - p.time_mask_max_ratio = 1.0 - p.use_input_dependent_random_seed = True - specaug_layer = p.Instantiate() - h1, _ = specaug_layer.FPropDefaultTheta(inputs1, paddings) - h2, _ = specaug_layer.FPropDefaultTheta(inputs2, paddings) - actual_layer_output1, actual_layer_output2 = self.evaluate([h1, h2]) - self.assertAllEqual( - np.shape(actual_layer_output1), np.array([5, 20, 2, 2])) - self.assertNotAllEqual(actual_layer_output1, actual_layer_output2) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/step.py b/lingvo/core/step.py deleted file mode 100644 index 1a2f631f..00000000 --- a/lingvo/core/step.py +++ /dev/null @@ -1,693 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""An abstract layer for processing sequences step-by-step. - -E.g.:: - - def ProcessSeq(step, external_inputs, input_batch): - prepared_inputs = step.PrepareExternalInputs( - step.theta, external_inputs) - batch_size, T = tf.shape(input_batch.paddings)[:2] - state = step.ZeroState( - step.theta, prepared_inputs, batch_size) - for t in range(T): - step_inputs = input_batch.Transform(lambda x: x[:, i, ...]) - step_outputs, state = step.FProp( - step.theta, prepared_inputs, step_inputs, state) - (processing step_outputs...) -""" - -import collections - -from lingvo import compat as tf -from lingvo.core import base_layer -from lingvo.core import builder_layers -from lingvo.core import py_utils -from lingvo.core import recurrent - - -class Step(base_layer.BaseLayer): - """A layer that processes input sequences step-by-step. - - This can be seen as an RNNCell extended with optional external inputs. - """ - - def PrepareExternalInputs(self, theta, external_inputs): - """Returns the prepared external inputs, e.g., packed_src for attention.""" - if not external_inputs: - external_inputs = py_utils.NestedMap() - packed = external_inputs.DeepCopy() - for name, child in self.children.items(): - child_external_inputs = external_inputs.get(name, py_utils.NestedMap()) - if isinstance(child, (tuple, list)): - output = [] - for i, sub in enumerate(child): - if isinstance(sub, Step): - output.append( - sub.PrepareExternalInputs(theta[name][i], - child_external_inputs)) - if output: - if len(output) != len(child): - raise ValueError('Expecting child list to be instances of Step.') - packed[name] = type(child)(output) - elif isinstance(child, Step): - packed[name] = child.PrepareExternalInputs(theta[name], - child_external_inputs) - return packed - - def ZeroState(self, theta, prepared_inputs, batch_size): - """Returns the initial state given external inputs and batch size. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - prepared_inputs: External inputs returned by PrepareExternalInputs(). - batch_size: An int scalar representing the batch size of per-step inputs. - - Returns: - A `.NestedMap` representing the initial state, which can be passed to - FProp() for processing the first time step. - """ - state0 = py_utils.NestedMap() - for name, child in self.children.items(): - if isinstance(child, (tuple, list)): - output = [] - for i, sub in enumerate(child): - if isinstance(sub, Step): - output.append( - sub.ZeroState(theta[name][i], prepared_inputs[name][i], - batch_size)) - if output: - if len(output) != len(child): - raise ValueError('Expecting child list to be instances of Step.') - state0[name] = type(child)(output) - elif isinstance(child, Step): - state0[name] = child.ZeroState(theta[name], prepared_inputs[name], - batch_size) - return state0 - - def FProp(self, theta, prepared_inputs, step_inputs, padding, state0): - """Forward function. - - step_inputs, state0, step_outputs, and state1 should each be a `.NestedMap` - of tensor values. Each tensor must be of shape [batch_size ...]. The - structure of NestedMaps are determined by the implementation. state0 and - state1 must have exactly the same structure and tensor shapes. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - prepared_inputs: External inputs returned by PrepareExternalInputs(). - step_inputs: The inputs for this time step. - padding: A 0/1 float tensor of shape [batch_size]; 1.0 means that this - batch element is empty in this step. - state0: The previous recurrent state. - - Returns: - A tuple (step_outputs, state1). - - - outputs: The outputs of this step. - - state1: The next recurrent state. - """ - raise NotImplementedError(type(self)) - - -class StatelessLayerStep(Step): - """Allows BaseLayer subclasses to be used as Steps. - - Layers used with this class should be stateless: they should not return - anything that must be passed back in the next invocation. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('layer', None, 'Params for the layer that this step wraps.') - return p - - def __init__(self, params): - super().__init__(params) - p = params - self.CreateChild('layer', p.layer) - - def FProp(self, theta, prepared_inputs, step_inputs, padding, state0): - """Perform inference on a stateless layer. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - prepared_inputs: unused. - step_inputs: A NestedMap containing 'inputs', which are passed directly to - the layer. - padding: A 0/1 float tensor of shape [batch_size]; 1.0 means that this - batch element is empty in this step. - state0: unused. - - Returns: - (output, state1), where output is the output of the layer, and - state1 is an empty NestedMap. - """ - del state0 - del prepared_inputs - args = {} - if padding is not None: - args['padding'] = padding - output = self.layer.FProp(theta.layer, step_inputs.inputs, **args) - return output, py_utils.NestedMap() - - -class StackStep(Step): - """A stack of steps. - - Each sub-step is assumed to accept step_inputs of type NestedMap(inputs=[]) - and return a primary output of type NestedMap(output=tensor). The - output of layer n-1 is sent to input of layer n. - - Per-step context vectors and per-sequence context vectors can also be - supplied; see FProp for more details. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'sub', [], 'A list of sub-stack params. Each layer is ' - 'expected to accept its input as NestedMap(inputs=[]), and ' - 'produce output as NestedMap(output=tensor). ' - 'The external_inputs parameter is passed directly to the ' - 'PrepareExternalInputs method of each sub-step. ') - p.Define( - 'residual_start', -1, 'An index of the layer where residual ' - 'connections start. Setting this parameter to a negative value turns ' - 'off residual connections.' - 'More precisely, when i >= residual_start, the output of each step ' - 'is defined as: ' - 'output[i] = output[i - residual_stride] + sub[i](output[i - 1]) ' - 'where output[-1] is the step input.') - p.Define( - 'residual_stride', 1, 'If residual connections are active, this ' - 'is the number of layers that each connection skips. For ' - 'instance, setting residual_stride = 2 means the output of layer ' - 'n is added to layer n + 2') - return p - - def __init__(self, params): - super().__init__(params) - p = params - self.sub_steps = [] - self.CreateChildren('sub', p.sub) - - def PrepareExternalInputs(self, theta, external_inputs): - """Delegates external inputs preparation to sub-layers. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - external_inputs: A `.NestedMap` object. The structure of the internal - fields is defined by the sub-steps. - - Returns: - A `.NestedMap` containing a pre-processed version of the external_inputs, - one per sub-step. - """ - packed = py_utils.NestedMap(sub=[]) - for i in range(len(self.sub)): - packed.sub.append(self.sub[i].PrepareExternalInputs( - theta.sub[i], external_inputs)) - return packed - - def ZeroState(self, theta, prepared_inputs, batch_size): - """Computes a zero state for each sub-step. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - prepared_inputs: An output from PrepareExternalInputs. - batch_size: The number of items in the batch that FProp will process. - - Returns: - A `.NestedMap` containing a state0 object for each sub-step. - """ - state = py_utils.NestedMap(sub=[]) - for i in range(len(self.sub)): - state.sub.append(self.sub[i].ZeroState(theta.sub[i], prepared_inputs, - batch_size)) - return state - - def FProp(self, theta, prepared_inputs, step_inputs, padding, state0): - """Performs inference on the stack of sub-steps. - - There are three possible ways to feed input to the stack: - - * step_inputs.inputs: These tensors are fed only to the lowest layer. - * step_inputs.context: [Optional] This tensor is fed to every layer. - * prepared_inputs: [Optional] This tensor is fed to every layer and - is assumed to stay constant over all steps. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - prepared_inputs: An output from PrepareExternalInputs. - step_inputs: A `.NestedMap` containing a list called 'inputs', an - optionally a tensor called 'context'. - padding: A 0/1 float tensor of shape [batch_size]; 1.0 means that this - batch element is empty in this step. - state0: The previous recurrent state. - - Returns: - A tuple (output, state1): - - - output: A `.NestedMap` containing the output of the top-most step. - - state1: The recurrent state to feed to next invocation of this graph. - """ - state1 = py_utils.NestedMap(sub=[]) - inputs = list(step_inputs.inputs) - # We pretend that the input is the output of layer -1 for the purposes - # of residual connections. - residual_inputs = [tf.concat(inputs, axis=1)] - additional = [] - if 'context' in step_inputs: - additional.append(step_inputs.context) - for i in range(len(self.sub)): - sub_inputs = py_utils.NestedMap(inputs=inputs + additional) - sub_output, state1_i = self.sub[i].FProp(theta.sub[i], - prepared_inputs.sub[i], - sub_inputs, padding, - state0.sub[i]) - state1.sub.append(state1_i) - output = sub_output.output - if i >= self.params.residual_start >= 0: - # residual_inputs contains the step input at residual_inputs[0]. - assert i + 1 - self.params.residual_stride < len(residual_inputs) - output += residual_inputs[i + 1 - self.params.residual_stride] - residual_inputs.append(output) - inputs = [output] - return py_utils.NestedMap(output=output), state1 - - -class ParallelStep(Step): - """Runs many steps on the same input and concatenates their outputs.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'sub', [], 'A list of step params. Each step is ' - 'expected to accept its input as NestedMap(inputs=[]), and ' - 'produce output as NestedMap(output=tensor). ' - 'The external_inputs parameter is passed directly to the ' - 'PrepareExternalInputs method of each sub-step. ') - return p - - def __init__(self, params): - super().__init__(params) - p = params - self.CreateChildren('sub', p.sub) - - def FProp(self, theta, prepared_inputs, step_inputs, padding, state0): - """Performs inference on N steps at once and concatenates the result. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - prepared_inputs: An output from PrepareExternalInputs. - step_inputs: A `.NestedMap` containing a list called 'inputs'. - padding: A 0/1 float tensor of shape [batch_size]; 1.0 means that this - batch element is empty in this step. - state0: The previous recurrent state. - - Returns: - A tuple (output, state1): - - - output: A `.NestedMap` containing the output of the top-most step. - - state1: The recurrent state to feed to next invocation of this graph. - """ - state1 = py_utils.NestedMap(sub=[None] * len(self.sub)) - outputs = [None] * len(self.sub) - - for i in range(len(self.sub)): - outputs[i], state1.sub[i] = self.sub[i].FProp(theta.sub[i], - prepared_inputs.sub[i], - step_inputs, padding, - state0.sub[i]) - - output = py_utils.NestedMap(output=tf.concat(outputs, axis=1)) - return output, state1 - - -# signature: A GraphSignature string defining the input and output parameters -# of this step. For example, (inputs=[a,b])->c means that step_inputs -# should be NestedMap(inputs=[a,b]), and the output of FProp should be -# stored in c. -# external_signature: A GraphSignature string defining the input to -# PrepareExternalInputs. For example, 'external_inputs.foo' means that -# the tensor external_inputs.foo should be the 'external_inputs' parameter -# when calling PrepareExternalInputs on this sub-step. -# params: The parameters to use when constructing the sub-step. -SubStep = collections.namedtuple('SubStep', - ['signature', 'external_signature', 'params']) - - -class GraphStep(Step): - r"""A step that connects sub-steps in a simple data flow graph. - - This is an adaptation of builder_layers.GraphLayer to support steps. - - Params.sub specifies a list of Specs that define each sub-step. - - A spec contains: - - * step_inputs: The signature describing how to assemble the input and output - for this step. The input part describes the 'step_inputs' parameter, - while the output part describes the name of the output. The state0 - input and state1 output are handled automatically and should not be - specified. - * external_inputs: if this Step requires external_inputs, this - is the signature describing how to find those inputs. - This value can also be set to None. - * params: the params used to construct the sub-step. - - The format of signature strings is defined in detail in the GraphSignature - class documentation. - - All inputs to a layer must have been produced by some previous layer. No - cycles are allowed. All outputs must be uniquely named; no overwriting - of previous names is allowed. - - Example - ('(act=[layer_0.output,step_inputs.context])->layer_1', - 'external_inputs.extra', - step_params) - - This constructs the step defined by step_params. Its FProp method will be - called with {act=[layer_0.output,step_inputs.context]} as the step_inputs - parameter. Its PrepareExternalInputs method will be called with - 'external_inputs.extra' as the external_inputs parameter. The output of that - method will be passed to ZeroState and FProp. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('output_signature', '', 'Signature of the step output.') - p.Define('sub', [], 'A list of SubSteps (defined above).') - p.Define('dict_type', py_utils.NestedMap, 'Type of nested dicts.') - return p - - _seq = collections.namedtuple( - '_Seq', ['name', 'signature', 'external_signature', 'step']) - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - self._seq = [] - for i, (signature, external_signature, sub_params) in enumerate(p.sub): - assert signature - sig = builder_layers.GraphSignature(signature) - assert len(sig.inputs) == 1 - assert sig.outputs - external_sig = None - if external_signature: - external_sig = builder_layers.GraphSignature(external_signature) - assert len(external_sig.inputs) == 1 - assert not external_sig.outputs - name = sub_params.name - if not name: - name = '%s_%02d' % (sig.outputs[0], i) - sub_params.name = name - self.CreateChild(name, sub_params) - self._seq.append( - GraphStep._seq(name, sig, external_sig, self.children[name])) - self.output_signature = builder_layers.GraphSignature(p.output_signature) - - def PrepareExternalInputs(self, theta, external_inputs): - """Prepares external inputs for each sub-step. - - The external_inputs parameter of this method is processed by the - external_inputs of each sub-step, then processed by the sub-step's - PrepareExternalInputs method. - - Args: - theta: variables used by sub-steps. - external_inputs: A NestedMap of [n_batch, ...] tensors. - - Returns: - A NestedMap of prepared inputs, where the keys are the names of - each sub-step. - """ - graph_tensors = builder_layers.GraphTensors() - graph_tensors.StoreTensor('external_inputs', external_inputs) - prepared_inputs = py_utils.NestedMap() - with tf.name_scope(self.params.name): - for seq in self._seq: - if seq.external_signature: - template = py_utils.NestedMap(inputs=seq.external_signature.inputs) - packed = template.Transform(graph_tensors.GetTensor) - seq_external_inputs = packed.inputs[0] - prepared_inputs[seq.name] = seq.step.PrepareExternalInputs( - theta[seq.name], seq_external_inputs) - else: - prepared_inputs[seq.name] = py_utils.NestedMap() - return prepared_inputs - - def ZeroState(self, theta, prepared_inputs, batch_size): - """Creates a zero state NestedMap for this step. - - Args: - theta: variables used by sub-steps. - prepared_inputs: Output from a call to PrepareExternalInputs. - batch_size: The number of items in the batch that FProp will process. - - Returns: - A NestedMap of ZeroState results for each sub-step. - """ - state0 = py_utils.NestedMap() - with tf.name_scope(self.params.name): - for seq in self._seq: - state0[seq.name] = seq.step.ZeroState(theta[seq.name], - prepared_inputs[seq.name], - batch_size) - return state0 - - def FProp(self, theta, prepared_inputs, step_inputs, padding, state0): - """A single inference step for this step graph. - - Args: - theta: variables used by sub-steps. - prepared_inputs: A NestedMap containing external_inputs that were - pre-processed by the PrepareExternalInputs method of each sub-step. The - keys are the names of the sub-steps. - step_inputs: A NestedMap of [batch, ...] tensors. The structure of this - depends on the graph implementation. - padding: A 0/1 float tensor of shape [batch_size]; 1.0 means that this - batch element is empty in this step. - state0: A NestedMap of state variables produced by either ZeroState or a - previous invocation of this FProp step. The keys are the names of the - sub-steps. - - Returns: - (output, state1), both of which are NestedMaps. - output is implementation-dependent and is defined by the output_signature - parameter. - state1 is a NestedMap where the keys are names of sub-steps and the values - are state outputs from their FProp methods. - """ - p = self.params - graph_tensors = builder_layers.GraphTensors() - graph_tensors.StoreTensor('prepared_inputs', prepared_inputs) - graph_tensors.StoreTensor('step_inputs', step_inputs) - state1 = py_utils.NestedMap() - with tf.name_scope(p.name): - for seq in self._seq: - tf.logging.vlog(1, 'GraphStep: call %s', seq.name) - external = None - if seq.external_signature: - external = prepared_inputs[seq.name] - template = py_utils.NestedMap(inputs=seq.signature.inputs) - packed = template.Transform(graph_tensors.GetTensor) - input_args = packed.inputs[0] - out, seq_state1 = seq.step.FProp(theta[seq.name], external, input_args, - padding, state0[seq.name]) - graph_tensors.StoreTensor(seq.signature.outputs[0], out) - state1[seq.name] = seq_state1 - template = py_utils.NestedMap(inputs=self.output_signature.inputs) - output_tensors = template.Transform(graph_tensors.GetTensor).inputs[0] - return output_tensors, state1 - - -class IteratorStep(Step): - """An iterator over the time dimension of some tensors. - - It's common to have a tensor of shape [batch, time, ...] or - [time, batch, ...]. This object will step through the time dimension, - producing tensors of shape [batch, ...] in succession. - - The input tensors are passed to PrepareExternalInputs. The step_inputs - argument of FProp is unused. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('axis', 1, 'The time dimension of the tensors.') - return p - - def PrepareExternalInputs(self, theta, external_inputs): - """Prepares the input for iteration. - - Args: - theta: unused. - external_inputs: A NestedMap containing tensors. The time axis of each - tensor should be params.axis. - - Returns: - A prepared NestedMap (current the same as the input). - """ - return external_inputs - - def ZeroState(self, theta, prepared_inputs, batch_size): - """Returns the initial iterator state. - - Args: - theta: unused. - prepared_inputs: Output from a call to PrepareExternalInputs. - batch_size: The number of items in the batch that FProp will process. - - Returns: - An initial state NestedMap. - """ - return py_utils.NestedMap(t=tf.constant(0, dtype=tf.int32)) - - def FProp(self, theta, prepared_inputs, step_inputs, padding, state0): - """Returns a A single inference step for this step graph. - - Args: - theta: unused. - prepared_inputs: Output from a call to PrepareExternalInputs. - step_inputs: unused. - padding: unused. - state0: A NestedMap of state variables produced by either ZeroState or a - previous invocation of this FProp step. - - Returns: - (output, state1), both of which are NestedMaps. - output is implementation-dependent and is defined by the output_signature - parameter. - state1 is a NestedMap where the keys are names of sub-steps and the values - are state outputs from their FProp methods. - """ - del theta - del step_inputs - del padding - - def _Slice(tensor): - """Return a slice of this tensor at time=state0.t.""" - shape = py_utils.GetShape(tensor) - # All zeros except for t in the time dimension. - # e.g. if params.axis=1, begin is [0, t, 0, 0, 0, ...] - begin = tf.one_hot(self.params.axis, tf.rank(tensor), on_value=state0.t) - # Same as shape, but with a 1 in the time dimension. - # e.g. if params.axis=1, shape is [shape[0], 1, shape[2], shape[3], ...] - size = tf.concat([ - shape[0:self.params.axis], - tf.constant([1], dtype=tf.int32), shape[self.params.axis + 1:] - ], - axis=0) - # Make a slice where the time dimension is fixed at state0.t. - time_slice = tf.slice(tensor, begin, size) - # Remove the time dimension. - return tf.squeeze(time_slice, axis=self.params.axis) - - output = prepared_inputs.Transform(_Slice) - state1 = py_utils.NestedMap(t=state0.t + 1) - return output, state1 - - -class RecurrentStepWrapper(base_layer.BaseLayer): - """A layer that wraps a step in a recurrent.Recurrent call.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('step', None, 'The step params that this class wraps.') - return p - - def __init__(self, params): - super().__init__(params) - self.CreateChild('step', self.params.step) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - self.step.InstantiateVariables() - super()._CreateChildrenVariables() - - def PrepareExternalInputs(self, theta, external_inputs): - """See Step.PrepareExternalInputs.""" - return self.step.PrepareExternalInputs(theta.step, external_inputs) - - def ZeroState(self, theta, prepared_inputs, batch_size): - """See Step.ZeroState.""" - return self.step.ZeroState(theta.step, prepared_inputs, batch_size) - - def FProp(self, theta, prepared_inputs, inputs, padding, state0, **kwargs): - """Runs a Step layer over multiple timesteps using Recurrent. - - Args: - theta: A NestedMap containing weights' values of this layer and its - children layers. - prepared_inputs: External inputs returned by Step.PrepareExternalInputs(). - inputs: A NestedMap of inputs of shape [time, batch_size, dim]. - padding: A 0/1 float tensor of shape [time, batch_size]; 1.0 means that - this batch element is empty in this step. - state0: A NestedMap containing the initial recurrent state. - **kwargs: Additional kwargs to pass to Recurrent. - - Returns: - A tuple (outputs, state1). - - - outputs: A NestedMap containing the accumulated outputs of all steps, - containing Tensors shaped [time, batch_size, dim]. - - state1: A NestedMap containing the accumulated recurrent states, - containing Tensors shaped [time, batch_size, dim]. - """ - - def RnnStep(recurrent_theta, recurrent_state0, recurrent_inputs): - """Compute a single timestep.""" - output, state1 = self.step.FProp( - theta=recurrent_theta.theta, - prepared_inputs=recurrent_theta.prepared_inputs, - step_inputs=recurrent_inputs.inputs, - padding=recurrent_inputs.padding, - state0=recurrent_state0.state) - recurrent_state1 = py_utils.NestedMap(output=output, state=state1) - return recurrent_state1, py_utils.NestedMap() - - # In order to pass Step outputs through Recurrent, they need to be - # included as part of state. - output0, _ = self.step.FProp(theta.step, prepared_inputs, - inputs.Transform(lambda x: x[0]), padding[0], - state0) - - accumulated_states, _ = recurrent.Recurrent( - theta=py_utils.NestedMap( - theta=theta.step, prepared_inputs=prepared_inputs), - state0=py_utils.NestedMap(output=output0, state=state0), - inputs=py_utils.NestedMap(inputs=inputs, padding=padding), - cell_fn=RnnStep, - **kwargs) - - return accumulated_states.output, accumulated_states.state diff --git a/lingvo/core/step_test.py b/lingvo/core/step_test.py deleted file mode 100644 index 23d80483..00000000 --- a/lingvo/core/step_test.py +++ /dev/null @@ -1,220 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for lingvo.core.step.""" - -from lingvo import compat as tf -from lingvo.core import builder_layers -from lingvo.core import py_utils -from lingvo.core import step -from lingvo.core import test_utils - - -class TextStep(step.Step): - """TextStep is a fake step used for testing. - - A real step would perform numeric operations on tensors. Instead, we return - text strings from these functions. This causes the result of our tests - to be long text strings that encode the sequence of operations that - were performed. - """ - - def PrepareExternalInputs(self, theta, external_inputs): - return external_inputs - - def ZeroState(self, theta, prepared_inputs, batch_size): - return self.params.name - - def FProp(self, theta, external_inputs, step_inputs, padding, state0): - return py_utils.NestedMap( - output=':'.join(step_inputs.inputs + [external_inputs]) + state0), ( - state0 + ':'.join(step_inputs.inputs)) - - -class StepTest(test_utils.TestCase): - - def testStatelessLayerStep(self): - with self.session(): - p = step.StatelessLayerStep.Params() - p.name = 'sum' - p.layer = builder_layers.FnLayer.Params().Set(fn=lambda x: x[0] + x[1]) - s = p.Instantiate() - - self.assertEqual(py_utils.NestedMap(), - s.PrepareExternalInputs(s.theta, None)) - self.assertEqual(py_utils.NestedMap(), s.ZeroState(s.theta, None, 8)) - - out, state1 = s.FProp( - s.theta, None, - py_utils.NestedMap(inputs=[ - tf.constant([4], dtype=tf.int32), - tf.constant([11], dtype=tf.int32) - ]), None, None) - self.assertEqual(py_utils.NestedMap(), state1) - value = self.evaluate(out) - self.assertEqual([15], value) - - def testGraphStep(self): - p = step.GraphStep.Params() - p.name = 'graphtest' - p.sub = [ - ('(inputs=[step_inputs.a])->output0', 'external_inputs.x', - TextStep.Params().Set(name='step0')), - ('(inputs=[output0.output])->output1', 'external_inputs.y', - TextStep.Params().Set(name='step1')), - ] - p.output_signature = 'output1' - s = p.Instantiate() - - prepared = s.PrepareExternalInputs(s.theta, - py_utils.NestedMap(x='1', y='2')) - self.assertEqual({'step0': '1', 'step1': '2'}, prepared) - - state0 = s.ZeroState(s.theta, prepared, 8) - self.assertEqual({'step0': 'step0', 'step1': 'step1'}, state0) - - output, state1 = s.FProp(s.theta, prepared, - py_utils.NestedMap(a='aa', b='bb'), - tf.constant([0.0], dtype=tf.float32), state0) - self.assertEqual({'output': 'aa:1step0:2step1'}, output) - self.assertEqual({'step1': 'step1aa:1step0', 'step0': 'step0aa'}, state1) - - def testStackStep(self): - p = step.StackStep.Params() - p.name = 'stack' - p.sub = [ - TextStep.Params().Set(name='text0'), - TextStep.Params().Set(name='text1'), - TextStep.Params().Set(name='text2'), - ] - s = p.Instantiate() - - prepared = s.PrepareExternalInputs(s.theta, 'z') - self.assertEqual({'sub': ['z', 'z', 'z']}, prepared) - state0 = s.ZeroState(s.theta, prepared, 1) - self.assertEqual({'sub': ['text0', 'text1', 'text2']}, state0) - output, state1 = s.FProp(s.theta, prepared, - py_utils.NestedMap(inputs=['in']), [0.0], state0) - # This output encodes the computations that were performed in the stack: - # input = in - # layer0: state=text0, external_inputs=z - # in -> in:ztext0 - # layer1: state=text1, external_inputs=z - # in:ztext0 -> in:ztext0:ztext1 - # layer2: state=text2, external_inputs=z - # in:ztext0:ztext1 -> in:ztext0:ztext1:ztext2 - self.assertEqual({'output': 'in:ztext0:ztext1:ztext2'}, output) - # The state1 of each sub-step is equal to its input plus its current state. - # layer0: state=text0, input=in - # layer1: state=text1, input=in:ztext0 - # layer2: state=text2, input=in:ztext0:ztext1 - self.assertEqual( - {'sub': ['text0in', 'text1in:ztext0', 'text2in:ztext0:ztext1']}, state1) - - def testStackStepWithResidualConnections(self): - p = step.StackStep.Params() - p.name = 'stack' - p.sub = [ - TextStep.Params().Set(name='text0'), - TextStep.Params().Set(name='text1'), - TextStep.Params().Set(name='text2'), - ] - p.residual_start = 1 - s = p.Instantiate() - - prepared = s.PrepareExternalInputs(s.theta, 'z') - self.assertEqual({'sub': ['z', 'z', 'z']}, prepared) - state0 = s.ZeroState(s.theta, prepared, 1) - self.assertEqual({'sub': ['text0', 'text1', 'text2']}, state0) - output, state1 = s.FProp(s.theta, prepared, - py_utils.NestedMap(inputs=['in']), [0.0], state0) - # This output encodes the computations that were performed in the stack: - # input=in - # layer0: state=text0, external_inputs=z - # in -> in:ztext0 - # layer1: state=text1, external_inputs=z - # in:ztext0 -> in:ztext0:ztext1 + in:ztext0 (residual) - # layer2: state=text2, external_inputs=z - # in:ztext0:ztext1in:ztext0 -> in:ztext0:ztext1in:ztext0:ztext2 + - # in:ztext0:ztext1in:ztext0 (residual) - self.assertEqual( - {'output': 'in:ztext0:ztext1in:ztext0:ztext2in:ztext0:ztext1in:ztext0'}, - output) - # The state1 of each sub-step is equal to its input plus its current state. - # layer0: state=text0, input=in - # layer1: state=text1, input=in:ztext0 - # layer2: state=text2, input=in:ztext0:ztext1in:ztext0 - self.assertEqual( - { - 'sub': - ['text0in', 'text1in:ztext0', 'text2in:ztext0:ztext1in:ztext0'] - }, state1) - - def testParallelStep(self): - - def PlusConstantParams(n): - p = step.StatelessLayerStep.Params() - p.name = 'sum' - p.layer = builder_layers.FnLayer.Params().Set(fn=lambda x: x + n) - return p - - with self.session(): - p = step.ParallelStep.Params() - p.name = 'concat' - p.sub = [PlusConstantParams(1), PlusConstantParams(2)] - concat = p.Instantiate() - - prepared = concat.PrepareExternalInputs(concat.theta, - py_utils.NestedMap()) - state0 = concat.ZeroState(concat.theta, prepared, 3) - step_inputs = py_utils.NestedMap( - inputs=tf.constant([[5], [10], [15]], dtype=tf.float32)) - output, _ = concat.FProp(concat.theta, prepared, step_inputs, None, - state0) - output = self.evaluate(output) - # Input is batch size 3: [[5], [10], [15]]. - # Two separate steps run on each input, +1 and +2, and the result is - # concatenated. - self.assertAllClose(output.output, [[6, 7], [11, 12], [16, 17]]) - - def testIteratorStep(self): - with self.session(): - p = step.IteratorStep.Params() - p.name = 'iterator' - iterator = p.Instantiate() - - prepared = iterator.PrepareExternalInputs( - iterator.theta, - py_utils.NestedMap( - a=tf.constant( - [[[1, 2], [3, 4], [5, 6]], [[7, 8], [9, 1], [2, 3]]], - dtype=tf.int32))) - state = iterator.ZeroState(iterator.theta, prepared, 2) - - outputs = [] - for _ in range(3): - out, state = iterator.FProp(iterator.theta, prepared, - py_utils.NestedMap(), None, state) - outputs.append(out) - outputs = self.evaluate(outputs) - self.assertAllClose([{ - 'a': [[1, 2], [7, 8]] - }, { - 'a': [[3, 4], [9, 1]] - }, { - 'a': [[5, 6], [2, 3]] - }], outputs) - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/steps/BUILD b/lingvo/core/steps/BUILD deleted file mode 100644 index 8892207b..00000000 --- a/lingvo/core/steps/BUILD +++ /dev/null @@ -1,85 +0,0 @@ -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -py_library( - name = "attention_steps", - srcs = ["attention_steps.py"], - srcs_version = "PY3", - deps = [ - ":rnn_steps", - "//lingvo:compat", - "//lingvo/core:attention", - "//lingvo/core:py_utils", - "//lingvo/core:step", - ], -) - -py_test( - name = "attention_steps_test", - size = "small", - srcs = ["attention_steps_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":attention_steps", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - ], -) - -py_library( - name = "embedding_steps", - srcs = ["embedding_steps.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:layers", - "//lingvo/core:py_utils", - "//lingvo/core:step", - ], -) - -py_test( - name = "embedding_steps_test", - size = "small", - srcs = ["embedding_steps_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":embedding_steps", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - ], -) - -py_library( - name = "rnn_steps", - srcs = ["rnn_steps.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:rnn_cell", - "//lingvo/core:step", - ], -) - -py_test( - name = "rnn_steps_test", - size = "small", - srcs = ["rnn_steps_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":rnn_steps", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:step", - "//lingvo/core:test_utils", - ], -) diff --git a/lingvo/core/steps/attention_steps.py b/lingvo/core/steps/attention_steps.py deleted file mode 100644 index 5d81717a..00000000 --- a/lingvo/core/steps/attention_steps.py +++ /dev/null @@ -1,277 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Steps for attention computation.""" - -from lingvo import compat as tf -from lingvo.core import attention -from lingvo.core import py_utils -from lingvo.core import step -from lingvo.core.steps import rnn_steps - - -class AttentionStep(step.Step): - """AttentionStep wraps an attention layer in the Step interface. - - An attention algorithm outputs a targeted summary of a set of input vectors. - - At each step, the query vector (input to FProp as step_inputs.input) - describes what data should be returned. The attention algorithm compares the - query vector with the source vectors (external_inputs.src) to compute - weights (attention_probs). - - The result is the weighted sum (using attention_probs) of a set of vectors. - By default that set of vectors is also external_inputs.src, but it can - optionally be external_inputs.context if a context tensor is specified. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'attention_step' - p.Define('atten', attention.AdditiveAttention.Params(), - 'Params of a subclass of BaseAttentionLayer.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self.CreateChild('atten', p.atten) - - def PrepareExternalInputs(self, theta, external_inputs): - """Prepare encoded source data for processing. - - In some attention algorithms, this step will pre-process the source - feature data so that FProp runs faster. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - external_inputs: A NestedMap containing tensors: - - - src: a [time, batch, depth] tensor that forms the input to the - attention layer. - - padding: a [time, batch] 0/1 tensor indicating which parts of - src contain useful information. - - context: Optional. See the class documentation for more details. - - Returns: - state0, a state parameter to pass to FProp on its first invocation. - """ - packed_inputs = external_inputs.DeepCopy() - if 'context' in external_inputs: - del packed_inputs['context'] - context = external_inputs.context - else: - context = external_inputs.src - packed_inputs.packed_src = self.atten.InitForSourcePacked( - theta.atten, external_inputs.src, context, external_inputs.padding) - return packed_inputs - - def _GetMaxSeqLength(self, src_encs): - """Compute the maximum sequence length of the encoded source sequence. - - Args: - src_encs: Encoded source sequence pre-processed by using - PrepareExternalInputs. It can be either a [time, batch, depth] tensor - (when there is only one source) or a NestedMap of [time, batch, depth] - tensors (when there are more than one source). - - Returns: - max_seq_length: the maximum sequence length of the encoded source - sequence. It can be either a scalar (when there is only one source) or - a NestedMap of scalars (when there are more than one source). - """ - # TODO(shaojinding): Create a MultiSourceAttentionStep class for the - # scenarios when there are multiple sources to avoid the use of if/else. - if isinstance(src_encs, py_utils.NestedMap): - max_seq_length = py_utils.NestedMap() - for key in src_encs: - max_seq_length[key] = py_utils.GetShape(src_encs[key], 3)[0] - else: - max_seq_length = py_utils.GetShape(src_encs, 3)[0] - return max_seq_length - - def ZeroState(self, theta, prepared_inputs, batch_size): - """Produce a zero state for this step. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - prepared_inputs: A set of inputs pre-processed by using - PrepareExternalInputs. - batch_size: Number of elements in the batched input. - - Returns: - state0, a state parameter to pass to FProp on its first invocation. - """ - max_seq_length = self._GetMaxSeqLength(prepared_inputs.src) - atten_state = self.atten.ZeroAttentionState(max_seq_length, batch_size) - (new_atten_context, _, - new_atten_states) = self.atten.ComputeContextVectorWithSource( - theta.atten, - prepared_inputs.packed_src, - tf.zeros([batch_size, self.params.atten.query_dim], - dtype=py_utils.FPropDtype(self.params)), - attention_state=atten_state) - return py_utils.NestedMap( - atten_context=new_atten_context, atten_state=new_atten_states) - - def FProp(self, theta, prepared_inputs, step_inputs, padding, state0): - """Produces a context vector from the attention algorithm. - - The context vector is a summary of the inputs from external_inputs - which the attention algorithm has determined would be useful for decoding - the next output. - - Args: - theta: A NestedMap containing weights' values of this layer and its - children layers. - prepared_inputs: A set of encoded tensors that have been pre-processed by - PrepareExternalInputs. - step_inputs: A NestedMap containing an 'inputs' tensor with the query - vector to use. - padding: A [batch, 1] 0/1 float tensor, where 1.0 means that this batch - slot is not used. - state0: A NestedMap of state, either produced by ZeroState or a previous - invocation of this graph. - - Returns: - output, state1, defined as follows: - - output: a NestedMap containing a query tensor, a context tensor, and - cum_atten_probs, the log of attention probabilities for each input - vector. - - state1: a NestedMap of state to be used in subsequent invocations of - this graph. - """ - (new_atten_context, new_atten_probs, - new_atten_states) = self.atten.ComputeContextVectorWithSource( - theta.atten, - prepared_inputs.packed_src, - tf.concat(step_inputs.inputs, axis=1), - attention_state=state0.atten_state) - new_atten_probs = py_utils.ApplyPadding(padding, new_atten_probs) - output = py_utils.NestedMap( - context=new_atten_context, probs=new_atten_probs) - state1 = py_utils.NestedMap( - atten_context=new_atten_context, atten_state=new_atten_states) - return output, state1 - - -class AttentionBlockStep(step.Step): - """Computes attention queries and context vectors. - - An attention algorithm produces a summary of a set of input vectors. - A query vector is used as an input to the process; we can think of this as - describing what information we're hoping to retrieve in the summary. - The summary output is called a context vector. - - This class uses attention as a way to view the input to a decoder. - In each step, we hope to read a summary of the encoded input that would be - most useful for generating the next output from the decoder. - - To do this, we combine a query generator and an attention algorithm. - The query generator's job is to build a query vector that represents the - current decoder state. The attention algorithm then uses that query vector - as a key to decide which encoded inputs are most important, and it combines - those into a context vector. - - The query generator takes two inputs: one is the context vector output by - the attention algorithm in the previous step, and the other is the label - output by the decoder in the previous step. It combines this information - in an implementation-dependent way to produce a query vector. - - In previous implementations, the query generator was just the first layer - of a stack of RNN layers (rnn_cell[0]). In this implementation, the query - generator can be any suitable Step, but in practice is one or more RNN layers. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'attention_block' - p.Define('query_generator', rnn_steps.RnnStackStep.Params(), - 'Query generator params.') - p.Define( - 'attention', AttentionStep.Params(), - 'Attention params. This can be either an AttentionStep.Params()' - 'or a list of them.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self.CreateChild('query_generator', p.query_generator) - self.CreateChild('attention', p.attention) - - def ZeroState(self, theta, prepared_inputs, batch_size): - """Produce a zero state for this step. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - prepared_inputs: A set of inputs pre-processed by using - PrepareExternalInputs. - batch_size: Number of elements in the batched input. - - Returns: - state0, a state parameter to pass to FProp on its first invocation. - """ - query_state0 = self.query_generator.ZeroState( - theta.query_generator, prepared_inputs.query_generator, batch_size) - atten_state0 = self.attention.ZeroState(theta.attention, - prepared_inputs.attention, - batch_size) - state0 = py_utils.NestedMap( - query_state=query_state0, atten_state=atten_state0) - return state0 - - def FProp(self, theta, prepared_inputs, step_inputs, padding, state0): - """Produces a query vector and a context vector for the next decoder step. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - prepared_inputs: A set of encoded tensors that have been pre-processed by - PrepareExternalInputs. - step_inputs: Unused. All of the input for this step comes from - external_inputs and previous step state. - padding: A [batch, 1] 0/1 float tensor, where 1.0 means that this batch - slot is not used. - state0: A NestedMap of state, either produced by ZeroState or a previous - invocation of this graph. - - Returns: - output, state1, are defined as follows. - output, a NestedMap containing an atten_query tensor, - an atten_context tensor, and atten_probs, attention probabilities for each - input vector. - state1, a NestedMap of state to be used in subsequent invocations of this - graph. - """ - query_output, query_state1 = self.query_generator.FProp( - theta.query_generator, prepared_inputs.query_generator, - py_utils.NestedMap(inputs=[state0.atten_state.atten_context]), padding, - state0.query_state) - atten_input = py_utils.NestedMap(inputs=[query_output.output]) - atten_output, atten_state1 = self.attention.FProp(theta.attention, - prepared_inputs.attention, - atten_input, padding, - state0.atten_state) - state1 = py_utils.NestedMap( - atten_state=atten_state1, query_state=query_state1) - return py_utils.NestedMap( - atten_context=atten_output.context, - atten_query=query_output.output, - atten_probs=atten_output.probs), state1 diff --git a/lingvo/core/steps/attention_steps_test.py b/lingvo/core/steps/attention_steps_test.py deleted file mode 100644 index ff9b547b..00000000 --- a/lingvo/core/steps/attention_steps_test.py +++ /dev/null @@ -1,581 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for third_party.py.lingvo.core.steps.attention_steps.""" - -from lingvo import compat as tf -from lingvo.core import attention -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.core.steps import attention_steps -import numpy as np - - -class AttentionStepsTest(test_utils.TestCase): - - def testAttentionStep(self): - with self.session(use_gpu=False): - np.random.seed(12345) - src_batch_size = 3 - target_batch_size = 6 - src_length = 5 - src_context_dim = 4 - query_dim = 5 - src_dim = 4 - source_vecs = tf.constant( - np.random.rand(src_length, src_batch_size, src_dim), dtype=tf.float32) - source_contexts = tf.constant( - np.random.rand(src_length, src_batch_size, src_context_dim), - dtype=tf.float32) - source_padding = tf.zeros([src_length, target_batch_size], - dtype=tf.float32) - query_vec = tf.constant( - np.random.rand(target_batch_size, query_dim), dtype=tf.float32) - - p = attention_steps.AttentionStep.Params() - p.atten.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - p.atten.source_dim = src_dim - p.atten.query_dim = query_dim - p.atten.hidden_dim = query_dim - p.atten.vn.global_vn = False - p.atten.vn.per_step_vn = False - p.atten.packed_input = True - step = p.Instantiate() - - external_inputs = py_utils.NestedMap( - src=source_vecs, - context=source_contexts, - padding=source_padding) - packed = step.PrepareExternalInputs(step.theta, external_inputs) - state0 = step.ZeroState(step.theta, packed, target_batch_size) - - step_inputs = py_utils.NestedMap(inputs=[query_vec]) - step_padding = tf.zeros([target_batch_size, 1], dtype=tf.float32) - output, state1 = step.FProp(step.theta, packed, step_inputs, step_padding, - state0) - - self.evaluate(tf.global_variables_initializer()) - output, state1 = self.evaluate([output, state1]) - - self.assertAllClose( - output, { - 'context': [[0.41788787, 0.5865286, 0.58267754, 0.21218117], - [0.42178467, 0.5067202, 0.5413259, 0.6616881], - [0.71586907, 0.6303425, 0.52290946, 0.694283], - [0.41789612, 0.58647645, 0.5826333, 0.21220288], - [0.421697, 0.5068262, 0.5411844, 0.66167986], - [0.7156511, 0.63033843, 0.5228955, 0.69437]], - 'probs': - [[0.20118009, 0.19332525, 0.20120151, 0.2022583, 0.20203482], - [0.20019522, 0.20133461, 0.19572362, 0.2025276, 0.2002189], - [0.20116101, 0.20004824, 0.20221081, 0.19645905, 0.20012087], - [0.20123273, 0.19319996, 0.20131132, 0.20220752, 0.2020485], - [0.2002011, 0.2015253, 0.19534773, 0.20260131, 0.20032457], - [0.20097165, 0.19993119, 0.20225787, 0.19671878, 0.20012051]] - }) - self.assertAllClose( - state1, { - 'atten_state': [[0.], [0.], [0.], [0.], [0.], [0.]], - 'atten_context': [[0.41788787, 0.5865286, 0.58267754, 0.21218117], - [0.42178467, 0.5067202, 0.5413259, 0.6616881], - [0.71586907, 0.6303425, 0.52290946, 0.694283], - [0.41789612, 0.58647645, 0.5826333, 0.21220288], - [0.421697, 0.5068262, 0.5411844, 0.66167986], - [0.7156511, 0.63033843, 0.5228955, 0.69437]] - }) - - def testAttentionStepMultiSourceSame(self): - with self.session(use_gpu=False): - np.random.seed(12345) - src_batch_size = 3 - target_batch_size = 6 - src_length = 5 - query_dim = 5 - src_dim = 4 - source_vecs_0 = tf.constant( - np.random.rand(src_length, src_batch_size, src_dim), dtype=tf.float32) - source_vecs_1 = tf.constant( - np.random.rand(src_length, src_batch_size, src_dim), dtype=tf.float32) - sources = py_utils.NestedMap( - source_0=source_vecs_0, source_1=source_vecs_1) - - source_padding_0 = tf.zeros([src_length, src_batch_size], - dtype=tf.float32) - source_padding_1 = tf.zeros([src_length, src_batch_size], - dtype=tf.float32) - source_paddings = py_utils.NestedMap( - source_0=source_padding_0, source_1=source_padding_1) - query_vec = tf.constant( - np.random.rand(target_batch_size, query_dim), dtype=tf.float32) - - p = attention_steps.AttentionStep.Params() - - # Setup MultiSourceAttention - p.atten = attention.MultiSourceAttention.Params() - p.atten.source_dim = src_dim - p.atten.query_dim = query_dim - - add_atten_params = attention.AdditiveAttention.Params() - add_atten_params.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - add_atten_params.source_dim = src_dim - add_atten_params.query_dim = query_dim - add_atten_params.hidden_dim = query_dim - add_atten_params.vn.global_vn = False - add_atten_params.vn.per_step_vn = False - add_atten_params.packed_input = True - - p.atten.source_atten_tpls = [('source_0', add_atten_params), - ('source_1', add_atten_params)] - - step = p.Instantiate() - - external_inputs = py_utils.NestedMap(src=sources, padding=source_paddings) - packed = step.PrepareExternalInputs(step.theta, external_inputs) - state0 = step.ZeroState(step.theta, packed, target_batch_size) - - step_inputs = py_utils.NestedMap(inputs=[query_vec]) - step_padding = tf.zeros([target_batch_size, 1], dtype=tf.float32) - output, state1 = step.FProp(step.theta, packed, step_inputs, step_padding, - state0) - - self.evaluate(tf.global_variables_initializer()) - output, state1 = self.evaluate([output, state1]) - - self.assertAllClose( - output, { - 'context': [[0.9590156, 0.8653384, 1.1668519, 0.697219], - [1.175648, 1.1199431, 1.2219069, 1.1452408], - [1.3191833, 1.0350775, 1.1315871, 1.3297331], - [0.95910096, 0.86546516, 1.1669571, 0.6971649], - [1.175647, 1.1201943, 1.222264, 1.1451368], - [1.3188481, 1.034915, 1.1314276, 1.3297772]], - 'probs': - [[0.20118009, 0.19332525, 0.20120151, 0.2022583, 0.20203482], - [0.20019522, 0.20133461, 0.19572362, 0.2025276, 0.2002189], - [0.20116101, 0.20004824, 0.20221081, 0.19645905, 0.20012087], - [0.20123273, 0.19319996, 0.20131132, 0.20220752, 0.2020485], - [0.2002011, 0.2015253, 0.19534773, 0.20260131, 0.20032457], - [0.20097165, 0.19993119, 0.20225787, 0.19671878, 0.20012051]] - }) - self.assertAllClose( - state1, { - 'atten_state': { - 'source_0': [[0.], [0.], [0.], [0.], [0.], [0.]], - 'source_1': [[0.], [0.], [0.], [0.], [0.], [0.]] - }, - 'atten_context': [[0.9590156, 0.8653384, 1.1668519, 0.697219], - [1.175648, 1.1199431, 1.2219069, 1.1452408], - [1.3191833, 1.0350775, 1.1315871, 1.3297331], - [0.95910096, 0.86546516, 1.1669571, 0.6971649], - [1.175647, 1.1201943, 1.222264, 1.1451368], - [1.3188481, 1.034915, 1.1314276, 1.3297772]] - }) - - def testAttentionStepMultiSourceSameWithGmmAttention(self): - with self.session(use_gpu=False): - np.random.seed(12345) - src_batch_size = 3 - target_batch_size = 6 - src_length = 5 - query_dim = 5 - src_dim = 4 - source_vecs_0 = tf.constant( - np.random.rand(src_length, src_batch_size, src_dim), dtype=tf.float32) - source_vecs_1 = tf.constant( - np.random.rand(src_length, src_batch_size, src_dim), dtype=tf.float32) - sources = py_utils.NestedMap( - source_0=source_vecs_0, source_1=source_vecs_1) - - source_padding_0 = tf.zeros([src_length, src_batch_size], - dtype=tf.float32) - source_padding_1 = tf.zeros([src_length, src_batch_size], - dtype=tf.float32) - source_paddings = py_utils.NestedMap( - source_0=source_padding_0, source_1=source_padding_1) - query_vec = tf.constant( - np.random.rand(target_batch_size, query_dim), dtype=tf.float32) - - p = attention_steps.AttentionStep.Params() - - # Setup MultiSourceAttention - p.atten = attention.MultiSourceAttention.Params() - p.atten.source_dim = src_dim - p.atten.query_dim = query_dim - - gmm_atten_params = attention.GmmMonotonicAttention.Params() - gmm_atten_params.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - gmm_atten_params.source_dim = src_dim - gmm_atten_params.query_dim = query_dim - gmm_atten_params.hidden_dim = query_dim - gmm_atten_params.vn.global_vn = False - gmm_atten_params.vn.per_step_vn = False - gmm_atten_params.packed_input = True - - p.atten.source_atten_tpls = [('source_0', gmm_atten_params), - ('source_1', gmm_atten_params)] - - step = p.Instantiate() - - external_inputs = py_utils.NestedMap(src=sources, padding=source_paddings) - packed = step.PrepareExternalInputs(step.theta, external_inputs) - state0 = step.ZeroState(step.theta, packed, target_batch_size) - - step_inputs = py_utils.NestedMap(inputs=[query_vec]) - step_padding = tf.zeros([target_batch_size, 1], dtype=tf.float32) - output, state1 = step.FProp(step.theta, packed, step_inputs, step_padding, - state0) - - self.evaluate(tf.global_variables_initializer()) - output, state1 = self.evaluate([output, state1]) - - self.assertAllClose( - output, { - 'context': [[0.8048796, 0.9554154, 1.2422264, 0.82598877], - [1.1976988, 0.9226365, 1.1311831, 1.1287751], - [1.2583418, 0.96984935, 0.8972859, 1.2939383], - [0.8055052, 0.9545301, 1.2421954, 0.824931], - [1.1980952, 0.9227077, 1.1313919, 1.13009], - [1.2582378, 0.96980226, 0.8973369, 1.2938937]], - 'probs': - [[0.05302628, 0.20965888, 0.3661108, 0.26998273, 0.08293614], - [0.05321905, 0.20958655, 0.36570197, 0.270003, 0.08308904], - [0.05327733, 0.20919749, 0.36514452, 0.27033207, 0.08349889], - [0.05328987, 0.20906723, 0.3648241, 0.27042356, 0.08376145], - [0.05301215, 0.21013679, 0.36650375, 0.26960865, 0.08261178], - [0.05328071, 0.20917267, 0.36505368, 0.27032903, 0.08357814]] - }) - self.assertAllClose( - state1, { - 'atten_state': { - 'source_0': [[[2.4243412, 1.2218076, 1.0122609, 0.18427502], - [1.9546769, 0.9721461, 1.0469768, 0.19244196], - [1.7934805, 0.8947478, 1.2158467, 0.18101364], - [2.2727895, 1.1433213, 0.969053, 0.21098366], - [2.1986299, 1.0997422, 1.3713341, 0.23128569]], - [[2.4298353, 1.227302, 1.0116383, 0.18391277], - [1.9476058, 0.96507514, 1.0462759, 0.19275317], - [1.793545, 0.89481235, 1.220826, 0.1817861], - [2.2800756, 1.1506072, 0.96794796, 0.21093304], - [2.194984, 1.0960963, 1.3741415, 0.23061496]], - [[2.4273272, 1.2247936, 1.0106387, 0.18302175], - [1.9522938, 0.96976304, 1.0510013, 0.19241981], - [1.7976122, 0.8988795, 1.2246737, 0.18208173], - [2.2875524, 1.1580843, 0.97309643, 0.21170339], - [2.1904838, 1.0915961, 1.3786552, 0.23077331]], - [[2.4339817, 1.2314482, 1.0118915, 0.18239658], - [1.9538436, 0.9713129, 1.050209, 0.19243228], - [1.7997689, 0.90103614, 1.2248727, 0.18208562], - [2.286818, 1.15735, 0.9776513, 0.2125737], - [2.1872034, 1.0883157, 1.3807379, 0.23051178]], - [[2.4258854, 1.223352, 1.0136935, 0.18414007], - [1.9573982, 0.9748675, 1.0445031, 0.19239089], - [1.7965381, 0.89780533, 1.2112961, 0.18159895], - [2.2637806, 1.1343125, 0.9743988, 0.21178932], - [2.1948628, 1.0959752, 1.366173, 0.23008086]], - [[2.435421, 1.2328876, 1.0118036, 0.18307444], - [1.9479709, 0.96544015, 1.0476727, 0.19277772], - [1.795729, 0.8969963, 1.224472, 0.18180896], - [2.2865427, 1.1570745, 0.9713619, 0.211611], - [2.1911612, 1.0922736, 1.3791639, 0.2307278]]], - 'source_1': [[[2.4243412, 1.2218076, 1.0122609, 0.18427502], - [1.9546769, 0.9721461, 1.0469768, 0.19244196], - [1.7934805, 0.8947478, 1.2158467, 0.18101364], - [2.2727895, 1.1433213, 0.969053, 0.21098366], - [2.1986299, 1.0997422, 1.3713341, 0.23128569]], - [[2.4298353, 1.227302, 1.0116383, 0.18391277], - [1.9476058, 0.96507514, 1.0462759, 0.19275317], - [1.793545, 0.89481235, 1.220826, 0.1817861], - [2.2800756, 1.1506072, 0.96794796, 0.21093304], - [2.194984, 1.0960963, 1.3741415, 0.23061496]], - [[2.4273272, 1.2247936, 1.0106387, 0.18302175], - [1.9522938, 0.96976304, 1.0510013, 0.19241981], - [1.7976122, 0.8988795, 1.2246737, 0.18208173], - [2.2875524, 1.1580843, 0.97309643, 0.21170339], - [2.1904838, 1.0915961, 1.3786552, 0.23077331]], - [[2.4339817, 1.2314482, 1.0118915, 0.18239658], - [1.9538436, 0.9713129, 1.050209, 0.19243228], - [1.7997689, 0.90103614, 1.2248727, 0.18208562], - [2.286818, 1.15735, 0.9776513, 0.2125737], - [2.1872034, 1.0883157, 1.3807379, 0.23051178]], - [[2.4258854, 1.223352, 1.0136935, 0.18414007], - [1.9573982, 0.9748675, 1.0445031, 0.19239089], - [1.7965381, 0.89780533, 1.2112961, 0.18159895], - [2.2637806, 1.1343125, 0.9743988, 0.21178932], - [2.1948628, 1.0959752, 1.366173, 0.23008086]], - [[2.435421, 1.2328876, 1.0118036, 0.18307444], - [1.9479709, 0.96544015, 1.0476727, 0.19277772], - [1.795729, 0.8969963, 1.224472, 0.18180896], - [2.2865427, 1.1570745, 0.9713619, 0.211611], - [2.1911612, 1.0922736, 1.3791639, 0.2307278]]] - }, - 'atten_context': [[0.8048796, 0.9554154, 1.2422264, 0.82598877], - [1.1976988, 0.9226365, 1.1311831, 1.1287751], - [1.2583418, 0.96984935, 0.8972859, 1.2939383], - [0.8055052, 0.9545301, 1.2421954, 0.824931], - [1.1980952, 0.9227077, 1.1313919, 1.13009], - [1.2582378, 0.96980226, 0.8973369, 1.2938937]] - }) - - def testAttentionStepMultiSourceDifferent(self): - with self.session(use_gpu=False): - np.random.seed(12345) - src_batch_size = 3 - target_batch_size = 6 - src_length = 5 - query_dim = 5 - src_dim = 4 - source_vecs_0 = tf.constant( - np.random.rand(src_length, src_batch_size, src_dim), dtype=tf.float32) - source_vecs_1 = tf.constant( - np.random.rand(src_length, src_batch_size, src_dim), dtype=tf.float32) - sources = py_utils.NestedMap( - source_0=source_vecs_0, source_1=source_vecs_1) - - source_padding_0 = tf.zeros([src_length, src_batch_size], - dtype=tf.float32) - source_padding_1 = tf.zeros([src_length, src_batch_size], - dtype=tf.float32) - source_paddings = py_utils.NestedMap( - source_0=source_padding_0, source_1=source_padding_1) - query_vec = tf.constant( - np.random.rand(target_batch_size, query_dim), dtype=tf.float32) - - p = attention_steps.AttentionStep.Params() - - # Setup MultiSourceAttention - p.atten = attention.MultiSourceAttention.Params() - p.atten.source_dim = src_dim - p.atten.query_dim = query_dim - - add_atten_params = attention.AdditiveAttention.Params() - add_atten_params.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - add_atten_params.source_dim = src_dim - add_atten_params.query_dim = query_dim - add_atten_params.hidden_dim = query_dim - add_atten_params.vn.global_vn = False - add_atten_params.vn.per_step_vn = False - add_atten_params.packed_input = True - - gmm_atten_params = attention.GmmMonotonicAttention.Params() - gmm_atten_params.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - gmm_atten_params.source_dim = src_dim - gmm_atten_params.query_dim = query_dim - gmm_atten_params.hidden_dim = query_dim - gmm_atten_params.vn.global_vn = False - gmm_atten_params.vn.per_step_vn = False - gmm_atten_params.packed_input = True - - p.atten.source_atten_tpls = [('source_0', add_atten_params), - ('source_1', gmm_atten_params)] - - step = p.Instantiate() - - external_inputs = py_utils.NestedMap(src=sources, padding=source_paddings) - packed = step.PrepareExternalInputs(step.theta, external_inputs) - state0 = step.ZeroState(step.theta, packed, target_batch_size) - - step_inputs = py_utils.NestedMap(inputs=[query_vec]) - step_padding = tf.zeros([target_batch_size, 1], dtype=tf.float32) - output, state1 = step.FProp(step.theta, packed, step_inputs, step_padding, - state0) - - self.evaluate(tf.global_variables_initializer()) - output, state1 = self.evaluate([output, state1]) - - self.assertAllClose( - output, { - 'context': [[0.9140804, 0.8979037, 1.1033492, 0.70460725], - [1.1748682, 1.0488822, 1.2771418, 1.0938747], - [1.2568944, 1.0808113, 0.9878455, 1.4196949], - [0.9142588, 0.8978502, 1.1039352, 0.7042637], - [1.174994, 1.0493405, 1.2779118, 1.0942582], - [1.2567302, 1.0806134, 0.98783255, 1.4195559]], - 'probs': - [[0.20118009, 0.19332525, 0.20120151, 0.2022583, 0.20203482], - [0.20019522, 0.20133461, 0.19572362, 0.2025276, 0.2002189], - [0.20116101, 0.20004824, 0.20221081, 0.19645905, 0.20012087], - [0.20123273, 0.19319996, 0.20131132, 0.20220752, 0.2020485], - [0.2002011, 0.2015253, 0.19534773, 0.20260131, 0.20032457], - [0.20097165, 0.19993119, 0.20225787, 0.19671878, 0.20012051]] - }) - self.assertAllClose( - state1, { - 'atten_state': { - 'source_0': [[0.], [0.], [0.], [0.], [0.], [0.]], - 'source_1': [[[2.4243412, 1.2218076, 1.0122609, 0.18427502], - [1.9546769, 0.9721461, 1.0469768, 0.19244196], - [1.7934805, 0.8947478, 1.2158467, 0.18101364], - [2.2727895, 1.1433213, 0.969053, 0.21098366], - [2.1986299, 1.0997422, 1.3713341, 0.23128569]], - [[2.4298353, 1.227302, 1.0116383, 0.18391277], - [1.9476058, 0.96507514, 1.0462759, 0.19275317], - [1.793545, 0.89481235, 1.220826, 0.1817861], - [2.2800756, 1.1506072, 0.96794796, 0.21093304], - [2.194984, 1.0960963, 1.3741415, 0.23061496]], - [[2.4273272, 1.2247936, 1.0106387, 0.18302175], - [1.9522938, 0.96976304, 1.0510013, 0.19241981], - [1.7976122, 0.8988795, 1.2246737, 0.18208173], - [2.2875524, 1.1580843, 0.97309643, 0.21170339], - [2.1904838, 1.0915961, 1.3786552, 0.23077331]], - [[2.4339817, 1.2314482, 1.0118915, 0.18239658], - [1.9538436, 0.9713129, 1.050209, 0.19243228], - [1.7997689, 0.90103614, 1.2248727, 0.18208562], - [2.286818, 1.15735, 0.9776513, 0.2125737], - [2.1872034, 1.0883157, 1.3807379, 0.23051178]], - [[2.4258854, 1.223352, 1.0136935, 0.18414007], - [1.9573982, 0.9748675, 1.0445031, 0.19239089], - [1.7965381, 0.89780533, 1.2112961, 0.18159895], - [2.2637806, 1.1343125, 0.9743988, 0.21178932], - [2.1948628, 1.0959752, 1.366173, 0.23008086]], - [[2.435421, 1.2328876, 1.0118036, 0.18307444], - [1.9479709, 0.96544015, 1.0476727, 0.19277772], - [1.795729, 0.8969963, 1.224472, 0.18180896], - [2.2865427, 1.1570745, 0.9713619, 0.211611], - [2.1911612, 1.0922736, 1.3791639, 0.2307278]]] - }, - 'atten_context': [[0.9140804, 0.8979037, 1.1033492, 0.70460725], - [1.1748682, 1.0488822, 1.2771418, 1.0938747], - [1.2568944, 1.0808113, 0.9878455, 1.4196949], - [0.9142588, 0.8978502, 1.1039352, 0.7042637], - [1.174994, 1.0493405, 1.2779118, 1.0942582], - [1.2567302, 1.0806134, 0.98783255, 1.419555]] - }) - - def testAttentionBlockStep(self): - with self.session(use_gpu=False): - np.random.seed(12345) - src_batch_size = 3 - target_batch_size = 6 - src_length = 5 - query_dim = 5 - context_dim = 8 - hidden_dim = 7 - src_dim = context_dim - source_vecs = tf.constant( - np.random.rand(src_length, src_batch_size, src_dim), dtype=tf.float32) - source_padding = tf.zeros([src_length, target_batch_size], - dtype=tf.float32) - - p = attention_steps.AttentionBlockStep.Params() - p.attention.atten.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - p.attention.atten.source_dim = src_dim - p.attention.atten.query_dim = query_dim - p.attention.atten.hidden_dim = hidden_dim - p.attention.atten.vn.global_vn = False - p.attention.atten.vn.per_step_vn = False - p.attention.atten.packed_input = True - p.query_generator.step_input_dim = context_dim - p.query_generator.rnn_cell_dim = query_dim - step = p.Instantiate() - - external_inputs = py_utils.NestedMap( - attention=py_utils.NestedMap(src=source_vecs, padding=source_padding)) - packed = step.PrepareExternalInputs(step.theta, external_inputs) - state0 = step.ZeroState(step.theta, packed, target_batch_size) - - step_padding = tf.zeros([target_batch_size, 1], dtype=tf.float32) - output, state1 = step.FProp(step.theta, packed, None, step_padding, - state0) - - self.evaluate(tf.global_variables_initializer()) - output, state1 = self.evaluate([output, state1]) - - self.assertAllClose( - output, { - 'atten_query': - np.array([ - [ - 0.1142175, 0.00020437, 0.02718649, -0.06030316, - 0.02916641 - ], - [ - 0.09362462, 0.07093287, 0.10184045, -0.0228882, - 0.06189567 - ], - [ - 0.12866478, 0.0121689, 0.05557573, -0.04107622, - 0.0543875 - ], - [ - 0.1142175, 0.00020437, 0.02718649, -0.06030316, - 0.02916641 - ], - [ - 0.09362462, 0.07093287, 0.10184045, -0.0228882, - 0.06189567 - ], - [ - 0.12866478, 0.0121689, 0.05557573, -0.04107622, - 0.0543875 - ], - ]), - 'atten_context': - np.array([ - [ - 0.55453926, 0.55162865, 0.62239933, 0.26001987, - 0.51269007, 0.555924, 0.54857075, 0.51340824 - ], - [ - 0.6495046, 0.42096642, 0.605386, 0.79519784, - 0.39852753, 0.30938083, 0.53797, 0.43651274 - ], - [ - 0.66645885, 0.56522155, 0.67393464, 0.6224826, - 0.66094846, 0.6098963, 0.52270895, 0.5319694 - ], - [ - 0.55453926, 0.55162865, 0.62239933, 0.26001987, - 0.51269007, 0.555924, 0.54857075, 0.51340824 - ], - [ - 0.6495046, 0.42096642, 0.605386, 0.79519784, - 0.39852753, 0.30938083, 0.53797, 0.43651274 - ], - [ - 0.66645885, 0.56522155, 0.67393464, 0.6224826, - 0.66094846, 0.6098963, 0.52270895, 0.5319694 - ], - ]), - 'atten_probs': - np.array([ - [ - 0.20132412, 0.19545832, 0.20277032, 0.19362292, - 0.20682438 - ], - [ - 0.20172212, 0.20001633, 0.20166671, 0.20218876, - 0.19440602 - ], - [ - 0.20540778, 0.20792785, 0.19377577, 0.19288684, - 0.20000176 - ], - [ - 0.20132412, 0.19545832, 0.20277032, 0.19362292, - 0.20682438 - ], - [ - 0.20172212, 0.20001633, 0.20166671, 0.20218876, - 0.19440602 - ], - [ - 0.20540778, 0.20792785, 0.19377577, 0.19288684, - 0.20000176 - ], - ]) - }) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/steps/embedding_steps.py b/lingvo/core/steps/embedding_steps.py deleted file mode 100644 index 1d122449..00000000 --- a/lingvo/core/steps/embedding_steps.py +++ /dev/null @@ -1,65 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Step classes for embedding tables.""" - -from lingvo.core import layers -from lingvo.core import py_utils -from lingvo.core import step - - -class EmbeddingStep(step.Step): - """A simple wrapper around EmbeddingLayer and its subclasses. - - This class can be used to insert an embedding lookup at the input side - of a GraphStep or StackStep. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'emb_step' - p.Define('emb', layers.EmbeddingLayer.Params(), 'Embedding layer params.') - return p - - def __init__(self, params): - super().__init__(params) - p = params - self.CreateChild('emb', p.emb) - - def FProp(self, theta, prepared_inputs, step_inputs, padding, state0): - """Looks up a list of embeddings from an EmbeddingLayer. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - prepared_inputs: unused. - step_inputs: A NestedMap containing a list called inputs. This list should - contain a single integer tensor of shape [batch], where each integer - represents an index into the embedding table. (By convention, all Steps - that can be used with StackStep must store inputs in - step_inputs.inputs[], but in this step it does not make sense for that - list to have more than one tensor in it). - padding: unused. - state0: unused. - - Returns: - A params.dtype tensor of shape [batch, embedding_dim]. - """ - del prepared_inputs - del state0 - assert len(step_inputs.inputs) == 1 - - output = self.emb.EmbLookup(theta.emb, step_inputs.inputs[0]) - return py_utils.NestedMap(output=output), py_utils.NestedMap() diff --git a/lingvo/core/steps/embedding_steps_test.py b/lingvo/core/steps/embedding_steps_test.py deleted file mode 100644 index 6980d873..00000000 --- a/lingvo/core/steps/embedding_steps_test.py +++ /dev/null @@ -1,57 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for lingvo.core.steps.embedding_steps.""" - -from lingvo import compat as tf -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.core.steps import embedding_steps - - -class EmbeddingStepsTest(test_utils.TestCase): - - def testEmbeddingStep(self): - with self.session(use_gpu=False): - tf.random.set_seed(398847392) - p = embedding_steps.EmbeddingStep.Params() - p.name = 'emb_step' - p.emb.vocab_size = 10 - p.emb.embedding_dim = 2 - p.emb.max_num_shards = 1 - p.emb.params_init = py_utils.WeightInit.Gaussian(0.01) - p.emb.vn.global_vn = False - p.emb.vn.per_step_vn = False - emb = p.Instantiate() - - # Verify that nothing bad happens when these methods are called. - packed = emb.PrepareExternalInputs(None, None) - state0 = emb.ZeroState(None, None, None) - - out1, state1 = emb.FProp( - emb.theta, packed, - py_utils.NestedMap(inputs=[tf.constant([4, 3], tf.int32)]), - tf.constant([0.0], dtype=tf.float32), state0) - - self.evaluate(tf.global_variables_initializer()) - out1, state1 = self.evaluate([out1, state1]) - - self.assertEqual({}, state1) - self.assertAllClose( - out1.output, - [[-5.9790569e-03, -8.7367110e-03], [-2.1643407e-06, 1.4426162e-02]]) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/steps/rnn_steps.py b/lingvo/core/steps/rnn_steps.py deleted file mode 100644 index d905f368..00000000 --- a/lingvo/core/steps/rnn_steps.py +++ /dev/null @@ -1,260 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Step APIs for RNN layers.""" - -from lingvo.core import py_utils -from lingvo.core import rnn_cell -from lingvo.core import step - - -class RnnStep(step.Step): - """A step containing an RNNCell.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('cell', rnn_cell.LSTMCellSimple.Params(), - 'Params for the RNN cell.') - return p - - def __init__(self, params): - super().__init__(params) - p = params - self.CreateChild('cell', p.cell) - - def PrepareExternalInputs(self, theta, external_inputs): - """Does not modify the external_inputs parameter. - - This parameter, if provided, is assumed to be a vector that should be - concatenated with the other vectors in step_inputs.inputs. - - Args: - theta: unused. - external_inputs: Either a tensor or None. - - Returns: - external_inputs, unmodified. - """ - return external_inputs - - def ZeroState(self, theta, prepared_inputs, batch_size): - """Returns the zero_state for the RNN cell. - - Args: - theta: Variables used by the RNNCell. - prepared_inputs: unused. - batch_size: An int scalar representing the batch size of per-step inputs. - - Returns: - The zero state of the RNNCell. - """ - return self.cell.zero_state(theta.cell, batch_size) - - def FProp(self, theta, prepared_inputs, step_inputs, padding, state0): - """Performs one inference step on the RNN cell. - - If external_inputs is not None, it is added as another act input - to the RNNCell. - - Args: - theta: Variables used by the RNNCell. - prepared_inputs: If not None, concatenated with step_inputs.input. A - tensor of shape [batch_size, external_input_dim]. - step_inputs: A NestedMap containing an 'input' list of [batch_size, dim] - where the sum of dim (including external_inputs) is - p.cell.num_input_nodes. - padding: A 0/1 float tensor of shape [batch_size]; 1.0 means that this - batch element is empty in this step. - state0: A NestedMap of state, either produced by ZeroState or a previous - invocation of FProp. - - Returns: - (output, state1), where output is the cell output (GetOutput(state1)) - of shape [batch_size, p.cell.num_output_nodes], and state1 is the cell's - recurrent state. - """ - cell_inputs = py_utils.NestedMap(act=step_inputs.inputs) - # An empty NestedMap can act as a None value here. - if prepared_inputs is not None and not isinstance(prepared_inputs, - py_utils.NestedMap): - cell_inputs.act.append(prepared_inputs) - cell_inputs.padding = padding - state1, extra = self.cell.FProp(theta.cell, state0, cell_inputs) - return py_utils.NestedMap( - output=self.cell.GetOutput(state1), extra=extra, - padding=padding), state1 - - -class RnnStackStep(step.Step): - """A stack of RnnSteps. - - Three types of inputs are supported: - step_inputs.input: This is the standard input. It is expected to change - on every step of the sequence, and it is fed only to the first layer. - step_inputs.context: This input changes for each step of the sequence, but - is fed to every layer. - external_inputs: This input is fixed at the beginning of the sequence. - It is fed to every layer. - - Residual connections are also supported. When residual_start >= 0, the output - of layer i (i >= residual_start) is added to the output of layer - i - residual_stride. - """ - - @classmethod - def Params(cls): - """Constructs Params for an RnnStackStep.""" - p = super().Params() - p.Define( - 'rnn_cell_tpl', rnn_cell.LSTMCellSimple.Params(), - 'RNNCell params template. ' - 'Can be a single param or ' - 'a list of rnn_layers params, one for each layer.') - p.Define( - 'external_input_dim', 0, 'Size of the external input. ' - 'The external input is given at the start of the sequence ' - 'and is given to every layer at every step.') - p.Define( - 'step_input_dim', 0, 'Size of the step input. ' - 'This input is only given to the first layer and is expected to ' - 'be different for each step.') - p.Define( - 'context_input_dim', 0, 'Size of the context input. ' - 'This input is given to every layer and is expected to be ' - 'different for each step.') - p.Define( - 'rnn_cell_dim', 0, 'Size of the rnn cells. ' - 'This may be overridden by parameters set in rnn_cell_tpl.') - p.Define( - 'rnn_cell_hidden_dim', 0, 'internal size of the rnn cells. When ' - 'set to > 0 it enables a projection layer at the output of the ' - 'rnn cell. This may be overridden by parameters set in rnn_cell_tpl.') - p.Define('rnn_layers', 1, 'Number of rnn layers.') - p.Define( - 'residual_start', -1, - 'Start residual connections from this layer. For this and higher ' - 'layers, the layer output is the sum of the RNN cell output and ' - 'input; if the layer also normalizes its output, then the ' - 'normalization is done over this sum. Set to -1 to disable ' - 'residual connections.') - p.Define('residual_stride', 1, - 'Number of lstm layers to skip per residual connection.') - return p - - def __init__(self, params): - super().__init__(params) - p = params - sub = [] - - # Users can either provide a single rnn_cell_tpl or one per layer. - # If only one is provided, we replicate it for each layer. - rnn_cell_tpls = p.rnn_cell_tpl - if not isinstance(rnn_cell_tpls, list): - rnn_cell_tpls = [p.rnn_cell_tpl] * p.rnn_layers - - # We may provide up to three tensors as input to the RnnStep: - # the normal input, the context input (from step_inputs.context), - # and the external input (from external_inputs). - arity = 1 - if p.context_input_dim: - arity += 1 - if p.external_input_dim: - arity += 1 - extra_dim = p.context_input_dim + p.external_input_dim - - # The first layer's input comes from step_inputs.input. Later layers - # will get their inputs from the previous layer's output. - input_nodes = p.step_input_dim - for i in range(p.rnn_layers): - step_i = RnnStep.Params() - step_i.name = 'rnn_%d' % i - step_i.cell = rnn_cell_tpls[i].Copy() - step_i.cell.num_input_nodes = input_nodes + extra_dim - step_i.cell.inputs_arity = arity - # The dimensions of each cell may be specified in the cell template - # but most users will specify them in the stack params. - if step_i.cell.num_output_nodes == 0: - step_i.cell.num_output_nodes = p.rnn_cell_dim - if step_i.cell.num_hidden_nodes == 0: - step_i.cell.num_hidden_nodes = p.rnn_cell_hidden_dim - input_nodes = step_i.cell.num_output_nodes - sub.append(step_i) - - stack_params = step.StackStep.Params() - stack_params.name = p.name - stack_params.sub = sub - stack_params.residual_start = p.residual_start - stack_params.residual_stride = p.residual_stride - self.CreateChild('stack', stack_params) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - self.stack.InstantiateVariables() - super()._CreateChildrenVariables() - - def PrepareExternalInputs(self, theta, external_inputs): - """Delegates external inputs preparation to sub-layers. - - Args: - theta: A `.NestedMap` object containing weight values of this layer and - its children layers. - external_inputs: A `.NestedMap` object. The structure of the internal - fields is defined by the sub-steps. - - Returns: - A `.NestedMap` containing a pre-processed version of the external_inputs, - one per sub-step. - """ - return self.stack.PrepareExternalInputs(theta.stack, external_inputs) - - def ZeroState(self, theta, prepared_inputs, batch_size): - """Computes a zero state for each sub-step. - - Args: - theta: A `.NestedMap` object containing weight values of this layer and - its children layers. - prepared_inputs: An output from PrepareExternalInputs. - batch_size: The number of items in the batch that FProp will process. - - Returns: - A `.NestedMap` containing a state0 object for each sub-step. - """ - return self.stack.ZeroState(theta.stack, prepared_inputs, batch_size) - - def FProp(self, theta, prepared_inputs, step_inputs, padding, state0): - """Performs inference on the stack of sub-steps. - - See the documentation for StackStep for the particulars of passing context - information to layers. - - Args: - theta: A `.NestedMap` object containing weight values of this layer and - its children layers. - prepared_inputs: An output from PrepareExternalInputs. - step_inputs: A `.NestedMap` containing a list called 'inputs', an - optionally a tensor called 'context'. - padding: A 0/1 float tensor of shape [batch_size]; 1.0 means that this - batch element is empty in this step. - state0: The previous recurrent state. - - Returns: - (output, state1): - - - output: A `.NestedMap` containing the output of the top-most step. - - state1: The recurrent state to feed to next invocation of this graph. - """ - return self.stack.FProp(theta.stack, prepared_inputs, step_inputs, padding, - state0) diff --git a/lingvo/core/steps/rnn_steps_test.py b/lingvo/core/steps/rnn_steps_test.py deleted file mode 100644 index c79b0290..00000000 --- a/lingvo/core/steps/rnn_steps_test.py +++ /dev/null @@ -1,233 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for lingvo.core.steps.rnn_steps.""" - -from lingvo import compat as tf -from lingvo.core import py_utils -from lingvo.core import step -from lingvo.core import test_utils -from lingvo.core.steps import rnn_steps - - -class RnnStepsTest(test_utils.TestCase): - - def testRnnStep(self): - with self.session(use_gpu=False): - p = rnn_steps.RnnStep.Params() - p.name = 'rnn_step' - p.cell.name = 'rnn_step_cell' - p.cell.output_nonlinearity = True - p.cell.params_init = py_utils.WeightInit.Uniform(1.24, 429891685) - p.cell.bias_init = py_utils.WeightInit.Uniform(1.24, 429891685) - p.cell.vn.global_vn = False - p.cell.vn.per_step_vn = False - p.cell.num_input_nodes = 2 - p.cell.num_output_nodes = 2 - p.cell.inputs_arity = 2 - - rnn_step = p.Instantiate() - external = tf.constant([[3]], tf.float32) - packed = rnn_step.PrepareExternalInputs(rnn_step.theta, external) - - state0 = rnn_step.ZeroState(rnn_step.theta, packed, 1) - _, state1 = rnn_step.FProp( - rnn_step.theta, packed, - py_utils.NestedMap(inputs=[tf.constant([[4]], tf.float32)]), - tf.constant([0.0], dtype=tf.float32), state0) - out2, state2 = rnn_step.FProp( - rnn_step.theta, packed, - py_utils.NestedMap(inputs=[tf.constant([[4]], tf.float32)]), - tf.constant([0.0], dtype=tf.float32), state1) - - self.evaluate(tf.global_variables_initializer()) - out2, state2 = self.evaluate([out2, state2]) - - self.assertAllClose(state2.m, [[-0.32659757, 0.87739915]]) - self.assertAllClose(state2.c, [[-1.9628618, 1.4194499]]) - self.assertAllClose(out2.output, [[-0.32659757, 0.87739915]]) - self.assertAllClose(out2.padding, [0.0]) - - def testRnnStepRecurrent(self): - with self.session(use_gpu=False): - p = rnn_steps.RnnStep.Params() - p.name = 'rnn_step' - p.cell.name = 'rnn_step_cell' - p.cell.output_nonlinearity = True - p.cell.params_init = py_utils.WeightInit.Uniform(1.24, 429891685) - p.cell.bias_init = py_utils.WeightInit.Uniform(1.24, 429891685) - p.cell.vn.global_vn = False - p.cell.vn.per_step_vn = False - p.cell.num_input_nodes = 2 - p.cell.num_output_nodes = 2 - p.cell.inputs_arity = 2 - - recurrent_p = step.RecurrentStepWrapper.Params().Set( - name='recurrent_wrapper', step=p) - recurrent_step = recurrent_p.Instantiate() - - external = tf.constant([[3]], tf.float32) - packed = recurrent_step.PrepareExternalInputs(recurrent_step.theta, - external) - inputs = py_utils.NestedMap( - inputs=[tf.constant([[[4]], [[4]]], tf.float32)]) - padding = tf.constant([[0.0], [0.0]], dtype=tf.float32) - state0 = recurrent_step.ZeroState(recurrent_step.theta, packed, 1) - - output, state2 = recurrent_step.FProp( - recurrent_step.theta, - prepared_inputs=packed, - inputs=inputs, - padding=padding, - state0=state0) - - self.evaluate(tf.global_variables_initializer()) - output, state2 = self.evaluate([output, state2]) - self.assertAllClose(state2.m, - [[[-0.20144, 0.741861]], [[-0.326598, 0.877399]]]) - self.assertAllClose(state2.c, - [[[-0.980946, 0.973391]], [[-1.962862, 1.41945]]]) - self.assertAllClose(output.output, - [[[-0.20144, 0.741861]], [[-0.326598, 0.877399]]]) - - def testRnnStackStep(self): - with self.session(use_gpu=False): - p = rnn_steps.RnnStackStep.Params() - p.name = 'rnn_stack_step' - p.rnn_cell_tpl.params_init = py_utils.WeightInit.Uniform(1.24, 429891685) - p.rnn_cell_tpl.bias_init = py_utils.WeightInit.Uniform(1.24, 429891685) - p.rnn_cell_tpl.vn.global_vn = False - p.rnn_cell_tpl.vn.per_step_vn = False - p.external_input_dim = 2 - p.step_input_dim = 1 - p.context_input_dim = 1 - p.rnn_cell_dim = 3 - p.rnn_cell_hidden_dim = 3 - p.rnn_layers = 2 - rnn_stack = p.Instantiate() - - external = tf.constant([[1, 2]], tf.float32) - packed = rnn_stack.PrepareExternalInputs(rnn_stack.theta, external) - state0 = rnn_stack.ZeroState(rnn_stack.theta, packed, 1) - output1, state1 = rnn_stack.FProp( - rnn_stack.theta, packed, - py_utils.NestedMap( - inputs=[tf.constant([[4]], tf.float32)], - context=tf.constant([[5]], tf.float32)), - tf.constant([0.0], dtype=tf.float32), state0) - - self.evaluate(tf.global_variables_initializer()) - output1, state1 = self.evaluate([output1, state1]) - - self.assertAllClose(output1.output, - [[0.43175745, -0.39472747, -0.36191428]]) - self.assertAllClose( - state1, { - 'sub': [{ - 'm': [[-0.4587491, 0.56409806, 0.23025148]], - 'c': [[5.6926787e-01, 6.3084178e-02, 4.1969700e-04]] - }, { - 'm': [[0.43175745, -0.39472747, -0.36191428]], - 'c': [[0.00158867, 0.57818687, -0.98025495]] - }] - }) - - def testRnnStackStepResidual(self): - with self.session(use_gpu=False): - p = rnn_steps.RnnStackStep.Params() - p.name = 'rnn_stack_step' - p.rnn_cell_tpl.params_init = py_utils.WeightInit.Uniform(1.24, 429891685) - p.rnn_cell_tpl.bias_init = py_utils.WeightInit.Uniform(1.24, 429891685) - p.rnn_cell_tpl.vn.global_vn = False - p.rnn_cell_tpl.vn.per_step_vn = False - p.external_input_dim = 2 - p.step_input_dim = 1 - p.context_input_dim = 1 - p.rnn_cell_dim = 3 - p.rnn_cell_hidden_dim = 3 - p.rnn_layers = 3 - p.residual_start = 0 - rnn_stack = p.Instantiate() - - external = tf.constant([[1, 2]], tf.float32) - packed = rnn_stack.PrepareExternalInputs(rnn_stack.theta, external) - state0 = rnn_stack.ZeroState(rnn_stack.theta, packed, 1) - output1, state1 = rnn_stack.FProp( - rnn_stack.theta, packed, - py_utils.NestedMap( - inputs=[tf.constant([[4]], tf.float32)], - context=tf.constant([[5]], tf.float32)), - tf.constant([0.0], dtype=tf.float32), state0) - - self.evaluate(tf.global_variables_initializer()) - output1, state1 = self.evaluate([output1, state1]) - - # Because there are residual connections, we expect the output to - # be equal to sum(rnn_state.m for rnn_state in state1.sub). - self.assertAllClose(output1.output, [[3.5457525, 4.576743, 4.2290816]]) - self.assertAllClose( - state1, { - 'sub': [{ - 'm': [[-0.4587491, 0.56409806, 0.23025148]], - 'c': [[5.6926787e-01, 6.3084178e-02, 4.1969700e-04]] - }, { - 'm': [[0.00226134, 0.00633766, -0.00058896]], - 'c': [[0.00088014, 0.07547389, 0.00625527]] - }, { - 'm': [[0.00224019, 0.00630755, -0.00058099]], - 'c': [[0.00088081, 0.07505893, 0.00621293]] - }] - }) - - def testRnnStackStepNoContext(self): - with self.session(use_gpu=False): - p = rnn_steps.RnnStackStep.Params() - p.name = 'rnn_stack_step' - p.rnn_cell_tpl.params_init = py_utils.WeightInit.Uniform(1.24, 429891685) - p.rnn_cell_tpl.bias_init = py_utils.WeightInit.Uniform(1.24, 429891685) - p.rnn_cell_tpl.vn.global_vn = False - p.rnn_cell_tpl.vn.per_step_vn = False - p.step_input_dim = 1 - p.rnn_cell_dim = 3 - p.rnn_cell_hidden_dim = 3 - p.rnn_layers = 2 - p.residual_start = 0 - rnn_stack = p.Instantiate() - - packed = rnn_stack.PrepareExternalInputs(rnn_stack.theta, - py_utils.NestedMap()) - state0 = rnn_stack.ZeroState(rnn_stack.theta, packed, 1) - output1, state1 = rnn_stack.FProp( - rnn_stack.theta, packed, - py_utils.NestedMap(inputs=[tf.constant([[4]], tf.float32)]), - tf.constant([0.0], dtype=tf.float32), state0) - - self.evaluate(tf.global_variables_initializer()) - output1, state1 = self.evaluate([output1, state1]) - - self.assertAllClose(output1.output, [[5.900284, 3.0231729, 3.0207822]]) - self.assertAllClose( - state1, { - 'sub': [{ - 'm': [[1.1416901, -0.32166323, -0.5909376]], - 'c': [[-0.98086286, 0.9052862, 0.10041453]] - }, { - 'm': [[0.7585938, -0.655164, -0.3882802]], - 'c': [[-8.3011830e-01, 1.8685710e-01, 1.0723456e-04]] - }] - }) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/summary_utils.py b/lingvo/core/summary_utils.py deleted file mode 100644 index 6b0efa49..00000000 --- a/lingvo/core/summary_utils.py +++ /dev/null @@ -1,381 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Common utility functions for generating summaries.""" - -import time -import lingvo.compat as tf -from lingvo.core import cluster_factory -from lingvo.core import plot -from lingvo.core import py_utils -import numpy as np - - -def _ShouldAddSummary(): - return cluster_factory.Current().add_summary - - -def scalar(*args, **kwargs): # pylint: disable=invalid-name - if _ShouldAddSummary(): - tf.summary.scalar(*args, **kwargs) - - -def histogram(*args, **kwargs): # pylint: disable=invalid-name - if _ShouldAddSummary(): - tf.summary.histogram(*args, **kwargs) - - -def image(*args, **kwargs): # pylint: disable=invalid-name - if _ShouldAddSummary(): - tf.summary.image(*args, **kwargs) - - -def SequenceLength(padding): - """Computes the length of a sequence based on binary padding. - - Args: - padding: A tensor of binary paddings shaped [batch, seqlen]. - - Returns: - seq_lens, A tensor of shape [batch] containing the non-padded length of each - element of plot_tensor along the batch dimension. - """ - seq_lens = tf.cast(tf.round(tf.reduce_sum(1 - padding, axis=1)), tf.int32) - # Get rid of any extra dimensions. - batch_size = tf.shape(padding)[0] - seq_lens = tf.reshape(seq_lens, [batch_size], name='seq_lens') - return seq_lens - - -def TrimPaddingAndPlotSequence(fig, axes, seq_matrix, seq_len, **kwargs): - """Trims the time axis of seq_matrix with shape (dim, time) and plots it. - - For use as a plot function with MatplotlibFigureSummary. - - Args: - fig: A matplotlib figure handle. - axes: A matplotlib axes handle. - seq_matrix: A 2D ndarray shaped (num_rows, time). - seq_len: Integer length to use to trim the time axis of seq_matrix. - **kwargs: Additional keyword args to pass to plot.AddImage. - """ - plot.AddImage(fig, axes, seq_matrix[:, :seq_len], **kwargs) - - -def TrimPaddingAndPlotAttention(fig, - axes, - atten_matrix, - src_len, - tgt_len, - transcript=None, - **kwargs): - """Trims axes of atten_matrix with shape (tgt_time, src_time) and plots it. - - For use as a plot function with MatplotlibFigureSummary. - - Args: - fig: A matplotlib figure handle. - axes: A matplotlib axes handle. - atten_matrix: A 2D ndarray shaped (tgt_time, src_time). - src_len: Integer length to use to trim the src_time axis of atten_matrix. - tgt_len: Integer length to use to trim the tgt_time axis of atten_matrix. - transcript: transcript for the target sequence. - **kwargs: Additional keyword args to pass to plot.AddImage. - """ - plot.AddImage( - fig, axes, atten_matrix[:tgt_len, :src_len], clim=(0, 1), **kwargs) - if transcript is not None: - if isinstance(transcript, np.ndarray): - transcript = ' '.join(transcript[:src_len]) - axes.set_xlabel(plot.ToUnicode(transcript), size='x-small', wrap=True) - - -def AddAttentionSummary(name, - attention_tensors, - src_paddings, - tgt_paddings, - transcripts=None, - max_outputs=3): - """Adds an image summary showing the attention probability matrix and state. - - Tensors are in sequence tensor format with the batch dimension in axis 1. - - Args: - name: Summary name. - attention_tensors: A list of 3D tensors shaped [target_len, batch_size, - source_len] where attention[i, j, k] is the probability for the i-th - output attending to the k-th input for element j in the batch. - src_paddings: A tensor of binary paddings shaped [source_len, batch] for the - source sequence. Or a list of tensors of the same length as - attention_tensors with a separate paddings for each entry in - attention_tensors. - tgt_paddings: A tensor of binary paddings shaped [target_len, batch] for the - target sequence. Or a list of tensors of the same length as - attention_tensors with a separate paddings for each entry in - attention_tensors. - transcripts: Optional, transcripts shaped [batch, source_len] for the source - sequence. - max_outputs: Integer maximum number of elements of the batch to plot. - """ - - def Transpose(paddings): - paddings = paddings if isinstance(paddings, list) else [paddings] - return [tf.transpose(p) for p in paddings] - - AddAttentionSummaryBatchMajor( - name, [tf.transpose(a, [1, 0, 2]) for a in attention_tensors], - Transpose(src_paddings), Transpose(tgt_paddings), transcripts, - max_outputs) - - -def AddAttentionSummaryBatchMajor(name, - attention_tensors, - src_paddings, - tgt_paddings, - transcripts=None, - max_outputs=3): - """Adds an image summary showing the attention probability matrix and state. - - As opposed to AddAttentionSummary() takes all tensors with batch dimension in - axis 0. - - Args: - name: Summary name. - attention_tensors: A list of 3D tensors shaped [batch_size, target_len, - source_len] where attention[b, i, j] is the probability for the i-th - output attending to the j-th input for element b in the batch. - src_paddings: A tensor of binary paddings shaped [batch, source_len] for the - source sequence. Or a list of tensors of the same length as - attention_tensors with a separate paddings for each entry in - attention_tensors. - tgt_paddings: A tensor of binary paddings shaped [batch, target_len] for the - target sequence. Or a list of tensors of the same length as - attention_tensors with a separate paddings for each entry in - attention_tensors. - transcripts: Optional, transcripts shaped [batch, source_len] for the source - sequence. - max_outputs: Integer maximum number of elements of the batch to plot. - """ - - def VerifyLen(paddings): - length = len(paddings) if isinstance(paddings, list) else 1 - if length != 1 and length != len(attention_tensors): - raise ValueError('Bad length of paddings list {}'.format(length)) - - VerifyLen(src_paddings) - VerifyLen(tgt_paddings) - - if not _ShouldAddSummary(): - return - - def ToLengths(paddings): - paddings = paddings if isinstance(paddings, list) else [paddings] - return [SequenceLength(p) for p in paddings] - - def Get(lengths, i): - return lengths[0 if len(lengths) == 1 else i] - - src_lens = ToLengths(src_paddings) - tgt_lens = ToLengths(tgt_paddings) - - with plot.MatplotlibFigureSummary( - name + '/Attention', - max_outputs=max_outputs, - gridspec_kwargs={'hspace': 0.3}) as fig: - for n, atten in enumerate(attention_tensors): - # Diagnostic metric that decreases as attention picks up. - max_entropy = tf.math.log(tf.cast(Get(src_lens, n), tf.float32)) - max_entropy = tf.expand_dims(tf.expand_dims(max_entropy, -1), -1) - atten_normalized_entropy = -atten * tf.math.log(atten + - 1e-10) / max_entropy - scalar(name + '/Attention/average_normalized_entropy/%d' % n, - tf.reduce_mean(atten_normalized_entropy)) - args = [atten, Get(src_lens, n), Get(tgt_lens, n)] - if transcripts is not None and n == 0: - args.append(transcripts) - fig.AddSubplot( - args, - TrimPaddingAndPlotAttention, - title=name, - xlabel='Input', - ylabel='Output') - - -def AddNormSummary(name, vs_gs): - """"Returns and creates summary for norms of vs and their gradients gs. - - Args: - name: A name string for summary. - vs_gs: A `.NestedMap` or a list of `.NestedMap` of (variable, gradient). - - Returns: - norm of variables, and norm of gradients. - """ - flatten = py_utils.Flatten(vs_gs) - v_norm = tf.sqrt(py_utils.SumSquared([v for (v, _) in flatten])) - scalar('var_norm/%s' % name, v_norm) - g_norm = tf.sqrt(py_utils.SumSquared([g for (_, g) in flatten])) - scalar('grad_norm/%s' % name, g_norm) - return v_norm, g_norm - - -def CollectVarHistogram(vs_gs): - """Adds histogram summaries for variables and gradients.""" - - for name, (var, grad) in vs_gs.FlattenItems(): - name = py_utils.SanitizeScopeKey(name) - with tf.device(var.device), tf.name_scope(name + '/summary'): - if isinstance(grad, tf.IndexedSlices): - var = tf.gather(var, grad.indices) - grad = grad.values - if var.dtype.is_complex: - var = tf.abs(var) - grad = tf.abs(grad) - - histogram('var_hist/' + name, var) - histogram('grad_hist/' + name, grad) - - -def PrepareSequenceForPlot(tensor, padding, name): - """Prepares a sequence feature for plotting. - - The sequence feature is transposed and channels are flattened. - - Args: - tensor: A n-D Tensor of shape [batch, time, ...]. - padding: A Tensor of shape [batch, time]. - name: A string as the name of the reshaped Tensor, which will be used as the - subcaption for plotting. - - Returns: - A tuple of: - reshaped_tensor: A 3-D Tensor of shape [batch, dim, time]. - sequence_length: A 1-D Tensor of shape [batch]. - """ - # Flatten any dimensions beyond the third into the third. - batch_size, max_len = py_utils.GetShape(tensor, 2) - plot_tensor = tf.reshape(tensor, [batch_size, max_len, -1]) - plot_tensor = tf.transpose(plot_tensor, [0, 2, 1], name=name) - return (plot_tensor, SequenceLength(padding)) - - -def PlotSequenceFeatures(plots, name, **kwargs): - """Plots a stack of sequence features. - - Args: - plots: A list of tuple (tensor, seq_len), as returned by - PrepareSequenceForPlot(). - name: A string for the caption of the plot. - **kwargs: Keyword arguments passed to AddSubplot(). - """ - if not _ShouldAddSummary(): - return - - with plot.MatplotlibFigureSummary(name, figsize=(8, len(plots) * 3.5)) as fig: - for tensor, seq_len in plots: - fig.AddSubplot([tensor, seq_len], - TrimPaddingAndPlotSequence, - title=tensor.name, - **kwargs) - - -class StatsCounter: - """A single counter in TF.""" - - def __init__(self, name): - self._name = name - self._var = py_utils.CreateVariable( - name=name, - params=py_utils.WeightParams([], py_utils.WeightInit.Constant(0), - tf.int64), - trainable=False) - self._value = self._var.value() + 0 # Makes a copy. - - def Value(self): - """Returns the current counter value.""" - return self._value - - def IncBy(self, delta): - """Increment the counter by delta and return the new value.""" - # NOTE: We must ensure _value is computed (_var + 0) before - # updating _var with delta. - delta = tf.cast(delta, tf.int64) - with tf.control_dependencies([self._value]): - scalar(self._name, self._value) - return tf.identity(tf.assign_add(self._var, delta)) - - -class StepRateTracker: - """A class that tracks step/example rate.""" - - def __init__(self): - self._time_steps = [] # History of (timestamp, global_step, total_examples) - - def ComputeStepRate(self, current_steps, total_examples): - """Computes the overall step rate.""" - if self._time_steps: - total_examples += self._time_steps[-1][-1] - self._time_steps.append((time.time(), current_steps, total_examples)) - # Keeps a relative long history to compute a smooth steps/second. - # Removes duplicate stats for step = 0 to get rid of the warm-up period. - while (self._time_steps[-1][1] - self._time_steps[0][1] > 10000 or - (len(self._time_steps) > 1 and - self._time_steps[0][1] == self._time_steps[1][1])): - del self._time_steps[0] - (t0, s0, e0), (t1, s1, e1) = self._time_steps[0], self._time_steps[-1] - rate = 0.0 - example_rate = 0.0 - if t1 > t0 + 1: - elapsed_secs = t1 - t0 - rate = (s1 - s0) / elapsed_secs - example_rate = (e1 - e0) / elapsed_secs - tf.logging.info('Steps/second: %f, Examples/second: %f', rate, - example_rate) - return rate, example_rate, total_examples - - -def ModelAnalysis(model): - """Returns a text showing variable sizes and their total size.""" - - class Analyzer: - """Helper class.""" - - def __init__(self): - self._seen_var = {} - self.total = 0 - - def __call__(self, v): - assert isinstance(v, tf.Variable) - # pylint: disable=protected-access - if not v.shape.is_fully_defined(): - # Only Cudnn RNN params lack static shapes. - if hasattr(v, 'approx_size'): - size = v.approx_size - else: - return '%-20s %10s %s' % (v.shape, 'n/a', v._shared_name) - else: - size = v.shape.num_elements() - if v._shared_name not in self._seen_var: - self._seen_var[v._shared_name] = size - self.total += size - return '%-20s %10d %s' % (v.shape, size, v._shared_name) - - analyzer = Analyzer() - output = '\n' - output += model.vars.Transform(analyzer).DebugString() - output += '\n' - output += '=' * 100 - output += f'\ntotal #params: {analyzer.total:,}\n' - return output, analyzer.total diff --git a/lingvo/core/summary_utils_test.py b/lingvo/core/summary_utils_test.py deleted file mode 100644 index 0050a001..00000000 --- a/lingvo/core/summary_utils_test.py +++ /dev/null @@ -1,44 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for summary_utils.""" - -from lingvo import compat as tf -from lingvo.core import cluster_factory -from lingvo.core import summary_utils -from lingvo.core import test_utils - - -class SummaryUtilsTest(test_utils.TestCase): - - def testStatsCounter(self): - with self.session(): - with cluster_factory.ForTestingWorker(add_summary=True): - foo = summary_utils.StatsCounter('foo') - val = foo.Value() - inc = foo.IncBy(100) - - self.evaluate(tf.global_variables_initializer()) - self.assertAllEqual(0, val.eval()) - self.assertAllEqual(100, self.evaluate(inc)) - self.assertAllEqual(100, val.eval()) - self.assertAllEqual([100, 200], self.evaluate([val, inc])) - self.assertAllEqual([200, 300], self.evaluate([val, inc])) - summary = tf.Summary.FromString(self.evaluate(tf.summary.merge_all())) - self.assertTrue(any('foo' in v.tag for v in summary.value)) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/symbolic.py b/lingvo/core/symbolic.py deleted file mode 100644 index e9ee1e28..00000000 --- a/lingvo/core/symbolic.py +++ /dev/null @@ -1,127 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Utilities for symbolic computation.""" - -import threading -import sympy - - -class Symbol(sympy.Dummy): - pass - - -def IsSymbol(x): - return isinstance(x, Symbol) - - -def IsExpr(x): - return isinstance(x, sympy.Expr) - - -STATIC_VALUES = 'static' -TENSOR_VALUES = 'tensor' -VALUE_TYPES = (STATIC_VALUES, TENSOR_VALUES) - - -class _LocalSymbolToValueStack(threading.local): - """A thread-local stack of symbol-to-value dicts.""" - - def __init__(self): - super().__init__() - self.stack = {} - for value_type in VALUE_TYPES: - self.stack[value_type] = [{}] - - -class SymbolToValueMap: - """A symbol-to-value mapping. - - Usage: - - with SymbolToValueMap('static', {symbol1: value1, symbol2: value2, ...}): - with SymbolToValueMap('tensor', {symbol1: value1, symbol2: value2, ...}): - ... = EvalExpr(value_type, symbolic_expr) - - Multiple SymbolToValueMap context can be nested inside one another. The inner - contexts take precedence over outer ones when multiple contexts provide - values for the same symbol. - """ - - _local_stack = _LocalSymbolToValueStack() - - def __init__(self, value_type, symbol_to_value_map): - """Creates a new symbol to value map. - - Args: - value_type: the type of values in 'symbol_to_value_map'. - symbol_to_value_map: a dict from Symbol to values. - """ - assert value_type in VALUE_TYPES - self.value_type = value_type - self.merged = dict(self.Stack(value_type)[-1]) - self.merged.update(symbol_to_value_map) - - @staticmethod - def Stack(value_type): - return SymbolToValueMap._local_stack.stack[value_type] - - def __enter__(self): - self.Stack(self.value_type).append(self.merged) - - def __exit__(self, type_arg, value_arg, traceback_arg): - stack = self.Stack(self.value_type) - assert stack - assert stack[-1] is self.merged - stack.pop() - - @staticmethod - def Get(value_type): - """Returns a symbol-to-value mapping merged from Stack().""" - return SymbolToValueMap.Stack(value_type)[-1] - - -def EvalExpr(value_type, x): - """Evaluates x with symbol_to_value_map within the current context. - - Args: - value_type: the target value type (see VALUE_TYPE). - x: a sympy.Expr, an object, or a list/tuple of Exprs and objects. - - Returns: - Evaluation result of 'x'. - """ - if isinstance(x, (list, tuple)): - return type(x)(EvalExpr(value_type, y) for y in x) - elif isinstance(x, sympy.Expr): - symbol_to_value_map = SymbolToValueMap.Get(value_type) - if not symbol_to_value_map: - return x - # In theory the below should be equivalent to: - # y = x.subs(symbol_to_value_map). - # In practice subs() doesn't work for when values are Tensors. - k, v = list(zip(*(list(symbol_to_value_map.items())))) - y = sympy.lambdify(k, x)(*v) - return y - else: - return x - - -def ToStatic(expr): - return EvalExpr(STATIC_VALUES, expr) - - -def ToTensor(expr): - return EvalExpr(TENSOR_VALUES, expr) diff --git a/lingvo/core/symbolic_test.py b/lingvo/core/symbolic_test.py deleted file mode 100644 index fc364a68..00000000 --- a/lingvo/core/symbolic_test.py +++ /dev/null @@ -1,68 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lingvo.core.symbolic.""" - -import lingvo.compat as tf -from lingvo.core import symbolic -from lingvo.core import test_utils -import sympy - - -class SymbolicTest(test_utils.TestCase): - - def testGetSymbol(self): - x = symbolic.Symbol('x') - self.assertIsInstance(x, sympy.Expr) - - def testEvalExpr(self): - x = symbolic.Symbol('x') - y = symbolic.Symbol('y') - xy = x * y - - # Without symbol-to-value map. - self.assertEqual(xy, symbolic.ToStatic(xy)) - self.assertEqual(xy, symbolic.ToTensor(xy)) - - with symbolic.SymbolToValueMap(symbolic.STATIC_VALUES, {x: 2, y: 3}): - self.assertEqual(symbolic.ToStatic(xy), 6) - # The inner map overrides the outer map. - with symbolic.SymbolToValueMap(symbolic.STATIC_VALUES, {x: 5, y: 6}): - self.assertEqual(symbolic.ToStatic(xy), 30) - # Back to the outer map. - self.assertEqual(symbolic.ToStatic(xy), 6) - - # EvalExpr can also evaluate a symbolic expression to a - # Tensor. - a = tf.placeholder(tf.float32) - b = tf.placeholder(tf.float32) - with symbolic.SymbolToValueMap(symbolic.TENSOR_VALUES, {x: a, y: b}): - with symbolic.SymbolToValueMap(symbolic.STATIC_VALUES, {x: 2, y: 3}): - # Value maps of different types do not affect each other. - self.assertEqual(symbolic.ToStatic(xy), 6) - ab = symbolic.ToTensor(xy) - self.assertIsInstance(ab, tf.Tensor) - with self.session() as sess: - self.assertEqual(12, sess.run(ab, {a: 3, b: 4})) - - # EvalExpr supports partial evaluation. - with symbolic.SymbolToValueMap(symbolic.STATIC_VALUES, {y: 3}): - x3 = symbolic.ToStatic(xy) - with symbolic.SymbolToValueMap(symbolic.STATIC_VALUES, {x: 9}): - self.assertEqual(27, symbolic.ToStatic(x3)) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/target_sequence_sampler.py b/lingvo/core/target_sequence_sampler.py deleted file mode 100644 index aed6ac3c..00000000 --- a/lingvo/core/target_sequence_sampler.py +++ /dev/null @@ -1,168 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""A library to sample target sequences given a decoder and decoder inputs. - -The sampled sequences can be used for training, e.g., with scheduled sampling, -OCD, second-pass deliberation. -""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils -from lingvo.core import recurrent - - -def _ComputePaddings(ids, eos_id): - is_eos = tf.cast(tf.equal(ids, eos_id), tf.int32) - # eos_in_prefix[i, j] = any(ids[i, k] == eos_id for k in range(j)) - eos_in_prefix = tf.cumsum(is_eos, axis=-1, exclusive=True) - return tf.where( - tf.equal(eos_in_prefix, 0), tf.zeros_like(ids), tf.ones_like(ids)) - - -class TargetSequenceSampler(base_layer.BaseLayer): - """Helper class for sampling target sequences with a decoder from inputs.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('target_sos_id', 1, 'Id of the start of sentence token.') - p.Define('target_eos_id', 2, 'Id of the end of sentence token.') - p.Define('target_eoc_id', -1, 'Id of the end of chunk token.') - p.Define('target_seq_len', 0, 'Maximum allowed target seq length.') - p.Define( - 'temperature', 1., 'If > 1, a smoother distribution than logits; ' - 'if < 1, a sharper distribution than logits. ' - 'Must be > 0.') - p.Define( - 'use_stop_fn', False, 'If True, use a stop_fn that causes the ' - 'sampler to early terminate when all samples in the batch end with ' - 'the target_eos_id token.') - p.name = 'target_sequence_sampler' - return p - - def Sample(self, decoder_theta, encoder_outputs, random_seed, - init_state_callback, pre_step_callback, post_step_callback): - """Samples target sequences, one target sequence per source sequence. - - (Please see beam_search_helper.py for description of decoder callbacks.) - - Args: - decoder_theta: A NestedMap object containing weights' values of the - decoder layer and its children layers, to be passed to decoder - callbacks. - encoder_outputs: the outputs of the encoder, to be passed to callbacks. - random_seed: a scalar int32 tensor representing the random seed. - init_state_callback: decoder._InitBeamSearchStateCallback. - pre_step_callback: decoder._PreBeamSearchStepCallback. - post_step_callback: decoder._PostBeamSearchStepCallback. - - Returns: - A NestedMap containing the following tensors - - - 'logits': [batch, max_target_length, vocab_size], representing the - distribution from which target sequences are sampled. - - 'ids': [batch, max_target_length] of int32, representing the target - sequence ids, not including target_sos_id, but maybe ending with - target_eos_id if end-of-sequence is reached before target_seq_len. - - 'paddings': [batch, max_target_length] of 0/1, where 1 represents - a padded timestep. - """ - p = self.params - assert p.temperature > 0 - if getattr(encoder_outputs, 'segment_id', 1) is None: - # Remove None values, which are not supported by recurrent. - del encoder_outputs['segment_id'] - # init_state_callback may modify 'encoder_outputs', e.g., by inserting - # 'packed_src'. - bs_result, bs_state = init_state_callback( - decoder_theta, encoder_outputs, num_hyps_per_beam=1) - # 'recurrent_theta' represents all cross-timestep information used by the - # recurrent loop below, including layer theta and encoder outputs. - recurrent_theta = py_utils.NestedMap( - theta=decoder_theta, - random_seed=random_seed, - encoder_outputs=encoder_outputs) - batch = tf.shape(bs_result.log_probs)[0] - recurrent_state0 = py_utils.NestedMap( - timestep=tf.zeros(shape=[], dtype=tf.int32), - logits=bs_result.log_probs, - # Start with target_sos_id. - ids=tf.fill([batch], tf.cast(p.target_sos_id, tf.int32)), - bs_state=bs_state) - inputs = py_utils.NestedMap(dummy=tf.zeros([p.target_seq_len, batch])) - - def Step(recurrent_theta, state0, inputs): - """Computes one decoder step.""" - del inputs - with tf.name_scope('single_sampler_step'): - # Compute logits and states. - bs_result, bs_state1 = pre_step_callback( - recurrent_theta.theta, - recurrent_theta.encoder_outputs, - tf.expand_dims(state0.ids, 1), # [batch, 1]. - state0.bs_state, - num_hyps_per_beam=1) - batch = tf.shape(bs_result.log_probs)[0] - state1 = py_utils.NestedMap(timestep=state0.timestep + 1) - state1.logits = bs_result.log_probs - # Sample ids from logits. [batch]. - state1.ids = tf.reshape( - tf.random.stateless_categorical( - state1.logits / p.temperature, - num_samples=1, - seed=tf.stack([recurrent_theta.random_seed, state0.timestep]), - dtype=state0.ids.dtype, - name='sample_next_id'), [batch]) - if 'is_last_chunk' in bs_result and p.target_eoc_id >= 0: - state1.ids = tf.where( - tf.math.logical_and(bs_result.is_last_chunk, - tf.equal(state1.ids, p.target_eoc_id)), - tf.fill(tf.shape(state1.ids), p.target_eos_id), state1.ids) - state1.bs_state = post_step_callback(recurrent_theta.theta, - recurrent_theta.encoder_outputs, - state1.ids, bs_state1) - return state1, py_utils.NestedMap() - - def StopFn(t, theta, state): - del t, theta # Unused: this stop function only uses the state ids. - return tf.equal(state.ids, p.target_eos_id) - - if p.use_stop_fn: - stop_fn = StopFn - else: - stop_fn = None - - accumulated_states, _ = recurrent.Recurrent( - recurrent_theta, - recurrent_state0, - inputs, - Step, - stop_fn=stop_fn, - allow_implicit_capture=True) - result = py_utils.NestedMap( - logits=tf.transpose(accumulated_states.logits, [1, 0, 2]), - ids=tf.transpose(accumulated_states.ids)) - result.paddings = tf.cast( - _ComputePaddings(result.ids, p.target_eos_id), result.logits.dtype) - # Force ids to be eos_id if the timestep is padded. - result.ids = tf.where( - tf.equal(result.paddings, 0), result.ids, - tf.fill(tf.shape(result.ids), p.target_eos_id)) - static_batch_size = bs_result.log_probs.shape[0] - result.ids.set_shape([static_batch_size, p.target_seq_len]) - result.paddings.set_shape([static_batch_size, p.target_seq_len]) - return result diff --git a/lingvo/core/target_sequence_sampler_test.py b/lingvo/core/target_sequence_sampler_test.py deleted file mode 100644 index b8a6c144..00000000 --- a/lingvo/core/target_sequence_sampler_test.py +++ /dev/null @@ -1,258 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for target_sequence_sampler.""" - -import lingvo.compat as tf -from lingvo.core import py_utils -from lingvo.core import target_sequence_sampler -from lingvo.core import test_utils -import numpy as np - - -class TargetSequenceSamplerTest(test_utils.TestCase): - - def testTargetSequenceSampler(self): - with self.session(use_gpu=False): - np.random.seed(9384758) - tf.random.set_seed(8274758) - vocab_size = 12 - src_len = 5 - tgt_len = 7 - batch_size = 2 - - def InitBeamSearchCallBack(unused_theta, unused_encoder_outputs, - num_hyps_per_beam): - self.assertEqual(1, num_hyps_per_beam) - logits = tf.zeros((batch_size, vocab_size), dtype=tf.float32) - return (py_utils.NestedMap(log_probs=logits), - py_utils.NestedMap(step=tf.constant(0))) - - def PreBeamSearchStepCallback(unused_theta, unused_encoder_outputs, - unused_step_ids, states, num_hyps_per_beam): - self.assertEqual(1, num_hyps_per_beam) - logits = tf.random.stateless_normal([batch_size, vocab_size], - seed=[8273747, 9]) - return (py_utils.NestedMap(log_probs=logits), - py_utils.NestedMap(step=states.step + 1)) - - def PostBeamSearchStepCallback(unused_theta, unused_encoder_outputs, - unused_new_step_ids, states): - return states - - src_enc = tf.random.stateless_normal([src_len, batch_size, 8], - seed=[982774838, 9]) - src_enc_padding = tf.constant( - [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 1.0], [1.0, 1.0]], - dtype=tf.float32) - encoder_outputs = py_utils.NestedMap( - encoded=src_enc, padding=src_enc_padding) - - theta = py_utils.NestedMap() - random_seed = tf.constant(123) - p = target_sequence_sampler.TargetSequenceSampler.Params().Set( - name='bsh', target_seq_len=tgt_len) - seq_sampler = p.Instantiate() - decoder_output = seq_sampler.Sample( - theta, encoder_outputs, random_seed, InitBeamSearchCallBack, - PreBeamSearchStepCallback, PostBeamSearchStepCallback) - - ids, lens = self.evaluate([ - decoder_output.ids, - tf.reduce_sum(1 - decoder_output.paddings, 1), - ]) - print(np.array_repr(ids)) - print(np.array_repr(lens)) - expected_ids = [[9, 0, 2, 2, 2, 2, 2], [0, 0, 11, 8, 1, 0, 7]] - expected_lens = [3, 7] - self.assertAllEqual(expected_ids, ids) - self.assertAllEqual(expected_lens, lens) - - p = target_sequence_sampler.TargetSequenceSampler.Params().Set( - name='bsh', target_seq_len=tgt_len, temperature=0.2) - seq_sampler = p.Instantiate() - decoder_output = seq_sampler.Sample( - theta, encoder_outputs, random_seed, InitBeamSearchCallBack, - PreBeamSearchStepCallback, PostBeamSearchStepCallback) - - ids, lens = self.evaluate([ - decoder_output.ids, - tf.reduce_sum(1 - decoder_output.paddings, 1), - ]) - print(np.array_repr(ids)) - print(np.array_repr(lens)) - expected_ids = [[0, 0, 0, 0, 0, 0, 9], [0, 0, 11, 7, 1, 0, 7]] - expected_lens = [7, 7] - self.assertAllEqual(expected_ids, ids) - self.assertAllEqual(expected_lens, lens) - - def testTargetSequenceSamplerWithEOC(self): - with self.session(use_gpu=False): - np.random.seed(9384758) - tf.random.set_seed(8274758) - vocab_size = 4 - src_len = 5 - tgt_len = 20 - batch_size = 2 - p = target_sequence_sampler.TargetSequenceSampler.Params().Set( - name='bsh', target_seq_len=tgt_len, target_eoc_id=0) - seq_sampler = p.Instantiate() - - def InitBeamSearchCallBack(unused_theta, unused_encoder_outputs, - num_hyps_per_beam): - self.assertEqual(1, num_hyps_per_beam) - logits = tf.zeros((batch_size, vocab_size), dtype=tf.float32) - is_last_chunk = tf.constant(False, shape=[batch_size]) - result = py_utils.NestedMap( - log_probs=logits, is_last_chunk=is_last_chunk) - states = py_utils.NestedMap( - step=tf.constant(0), - src_step=tf.zeros([batch_size], dtype=tf.int32)) - return result, states - - def PreBeamSearchStepCallback(unused_theta, unused_encoder_outputs, - unused_step_ids, states, num_hyps_per_beam): - self.assertEqual(1, num_hyps_per_beam) - logits = tf.random.stateless_normal([batch_size, vocab_size], - seed=[8273747, 9]) - # Make it never predict . - logits -= tf.one_hot([p.target_eos_id], vocab_size, 1e30) - is_last_chunk = tf.equal(states.src_step, src_len - 1) - result = py_utils.NestedMap( - log_probs=logits, is_last_chunk=is_last_chunk) - return result, states - - def PostBeamSearchStepCallback(unused_theta, unused_encoder_outputs, - new_step_ids, states): - return py_utils.NestedMap( - step=states.step + 1, - src_step=states.src_step + tf.cast( - tf.equal(new_step_ids, p.target_eoc_id), dtype=tf.int32)) - - src_enc = tf.random.stateless_normal([src_len, batch_size, 8], - seed=[982774838, 9]) - src_enc_padding = tf.constant( - [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 1.0], [1.0, 1.0]], - dtype=tf.float32) - encoder_outputs = py_utils.NestedMap( - encoded=src_enc, padding=src_enc_padding) - - theta = py_utils.NestedMap() - random_seed = tf.constant(123) - decoder_output = seq_sampler.Sample( - theta, encoder_outputs, random_seed, InitBeamSearchCallBack, - PreBeamSearchStepCallback, PostBeamSearchStepCallback) - - ids, lens = self.evaluate([ - decoder_output.ids, - tf.reduce_sum(1 - decoder_output.paddings, 1), - ]) - print(np.array_repr(ids)) - print(np.array_repr(lens)) - expected_ids = [ - [0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2], - [0, 0, 3, 3, 1, 0, 3, 0, 1, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2], - ] - expected_lens = [5, 11] - self.assertAllEqual(expected_ids, ids) - self.assertAllEqual(expected_lens, lens) - - # Now do the same, except with use_stop_fn=True. - p = target_sequence_sampler.TargetSequenceSampler.Params().Set( - name='bsh', target_seq_len=tgt_len, target_eoc_id=0, use_stop_fn=True) - seq_sampler = p.Instantiate() - decoder_output = seq_sampler.Sample(theta, encoder_outputs, random_seed, - InitBeamSearchCallBack, - PreBeamSearchStepCallback, - PostBeamSearchStepCallback) - - ids, lens = self.evaluate([ - decoder_output.ids, - tf.reduce_sum(1 - decoder_output.paddings, 1), - ]) - print(np.array_repr(ids)) - print(np.array_repr(lens)) - expected_ids = [ - [0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2], - [0, 0, 3, 3, 1, 0, 3, 0, 1, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2], - ] - expected_lens = [5, 11] - self.assertAllEqual(expected_ids, ids) - self.assertAllEqual(expected_lens, lens) - - def testTargetSequenceSamplerWithVariables(self): - with self.session(use_gpu=False): - np.random.seed(9384758) - tf.random.set_seed(8274758) - hidden_dim = 8 - vocab_size = 12 - src_len = 5 - tgt_len = 7 - batch_size = 2 - keras_layer = py_utils.NestedMap(dense=None) - - def InitBeamSearchCallBack(unused_theta, unused_encoder_outputs, - num_hyps_per_beam): - self.assertEqual(1, num_hyps_per_beam) - logits = tf.zeros((batch_size, vocab_size), dtype=tf.float32) - return (py_utils.NestedMap(log_probs=logits), - py_utils.NestedMap(step=tf.constant(0))) - - def PreBeamSearchStepCallback(unused_theta, unused_encoder_outputs, - unused_step_ids, states, num_hyps_per_beam): - self.assertEqual(1, num_hyps_per_beam) - hidden = tf.random.stateless_normal([batch_size, hidden_dim], - seed=[8273747, 9]) - # A stateful 'dense' layer. - if not keras_layer.dense: - keras_layer.dense = tf.keras.layers.Dense( - vocab_size, input_shape=hidden.shape) - log_probs = tf.nn.log_softmax(keras_layer.dense(hidden)) - return (py_utils.NestedMap(log_probs=log_probs), - py_utils.NestedMap(step=states.step + 1)) - - def PostBeamSearchStepCallback(unused_theta, unused_encoder_outputs, - unused_new_step_ids, states): - return states - - src_enc = tf.random.stateless_normal([src_len, batch_size, 8], - seed=[982774838, 9]) - src_enc_padding = tf.constant( - [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 1.0], [1.0, 1.0]], - dtype=tf.float32) - encoder_outputs = py_utils.NestedMap( - encoded=src_enc, padding=src_enc_padding) - - theta = py_utils.NestedMap() - random_seed = tf.constant(123) - p = target_sequence_sampler.TargetSequenceSampler.Params().Set( - name='bsh', target_seq_len=tgt_len) - seq_sampler = p.Instantiate() - decoder_output = seq_sampler.Sample(theta, encoder_outputs, random_seed, - InitBeamSearchCallBack, - PreBeamSearchStepCallback, - PostBeamSearchStepCallback) - - self.evaluate(tf.global_variables_initializer()) - ids, lens = self.evaluate([ - decoder_output.ids, - tf.reduce_sum(1 - decoder_output.paddings, 1), - ]) - print(np.array_repr(ids)) - print(np.array_repr(lens)) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/task_scheduler.py b/lingvo/core/task_scheduler.py deleted file mode 100644 index 94597e33..00000000 --- a/lingvo/core/task_scheduler.py +++ /dev/null @@ -1,324 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Multi-task task sampling schedules.""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import early_stop -import numpy as np - - -class TaskScheduler(base_layer.BaseLayer): - """Generic multi-task scheduler. - - Subclasses should override the `Sample` method to return a task string given - a step. All of the task strings as well as additional hyperparameters needed - by `Sample` should be exposed and stored in the params. `Sample` should also - update `cur_probs`. - """ - - @classmethod - def Params(cls): - """Parameters for this task scheduler.""" - p = super().Params() - p.name = 'task_scheduler' - return p - - def __init__(self, params): - super().__init__(params) - self.cur_probs = None - self.SetVariableFree() - - def Sample(self, current_step): - raise NotImplementedError('Abstract method') - - -class AdaptiveScheduler(TaskScheduler): - """Tasks with low scores will be sampled more often. - - Scores are expected to be non-negative. Larger scores are better.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('tasks', [], 'List of tasks') - p.Define('expected', [], 'List of final expected scores') - p.Define('mh_a', early_stop.MetricHistory.Params(), '') - p.Define('mh_b', early_stop.MetricHistory.Params(), '') - p.Define( - 'epsilon', 0.05, 'Regularizarion term. A large epsilon will lead' - 'to a more uniform task distribution.') - p.Define('alpha', 1.0, 'Normalized task scores are raised to this power.') - return p - - def __init__(self, params): - super().__init__(params) - if len(self.params.tasks) != 2 or len(self.params.expected) != 2: - raise ValueError('Only two tasks are supported by this scheduler.') - - if self.params.epsilon < 0: - raise ValueError('Epsilon should be positive.') - - self.tasks = self.params.tasks - - self.last_scores = [0.0] * 2 - - self._metric_histories = [ - early_stop.MetricHistory(self.params.mh_a), - early_stop.MetricHistory(self.params.mh_b) - ] - - def getMetricHistories(self): - # If too slow, consider another implementation. - # TODO(sebjean) Time file reading and change behaviour if too long. - for index, mh in enumerate(self._metric_histories): - try: - with tf.io.gfile.GFile(mh.hist_file) as f: - lines = f.readlines() - except tf.errors.NotFoundError: - tf.logging.warning('File not found. ' - 'Expected at start of training only.') - score, lines = 0.0, [] - if lines: - try: - score = lines[-1].split()[-1] - except IndexError: - tf.logging.warning( - 'IndexError. Your history file may be corrupted.') - score = 0.0 - self.last_scores[index] = float(score) - - -class SimpleAdaptiveScheduler(AdaptiveScheduler): - """Simple adaptive scheduler. - - A task with a normalized score of `s` is approximately weighted as `1 - s`. - """ - - def Sample(self, current_step): - """Sample a task. - - The unnormalized probability of a task if given by - 1 + epsilon - min(1, score / expected)**alpha. - - Args: - current_step: Unused. - - Returns: - str, the name of the sampled task. - """ - del current_step # Unused - - self.getMetricHistories() - - alpha, eps = self.params.alpha, self.params.epsilon - probs = [ - 1 + eps - min(1, score / self.params.expected[index])**alpha - for index, score in enumerate(self.last_scores) - ] - probs = tuple(probs / np.sum(probs)) - sampled_task = np.random.choice(self.params.tasks, p=probs) - self.cur_probs = probs - return sampled_task - - -class InverseRatioAdaptiveScheduler(AdaptiveScheduler): - """Inverse ratio adaptive scheduler. - - Tasks are approximately weighed as the inverse of their normalized scores. - """ - - def Sample(self, current_step): - """Sample a task. - - The unnormalized probability of a task if given by - 1 / (min(1, score / expected)**alpha + epsilon) - - Args: - current_step: Unused. - - Returns: - str, the name of the sampled task. - """ - del current_step # Unused - - self.getMetricHistories() - - alpha, eps = self.params.alpha, self.params.epsilon - probs = [ - 1.0 / (min(1, score / self.params.expected[index])**alpha + eps) - for index, score in enumerate(self.last_scores) - ] - probs = tuple(probs / np.sum(probs)) - sampled_task = np.random.choice(self.params.tasks, p=probs) - self.cur_probs = probs - return sampled_task - - -class ShiftedExponentialScheduler(TaskScheduler): - """The unnormalized score of each task follows a shifted exponential function. - - Generalizes the constant, exponential and sigmoid - schedules described in "Scheduled Multi-Task Learning: From Syntax to - Translation" (Kiperwasser and Ballesteros). - https://arxiv.org/pdf/1804.08915.pdf - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'alpha', 0, 'Controls the rate at which the schedule changes. ' - 'A large alpha will lead to fast convergence toward final values.') - p.Define( - 'task_probs', [], 'List of 2-tuples (task, prob). For non-constant' - 'schedulers, prob is a tuple of the form (init_prob, final_prob).') - return p - - def __init__(self, params): - super().__init__(params) - assert isinstance(self.params.task_probs, list) - self.tasks = [] - self._descriptors = [] - - def Sample(self, current_step): - """Sample a task. - - Given an input [a, b] and a rate `alpha`, the unnormalized - score of eack task is a + b * exp(-alpha * t). - - Args: - current_step: int. Current time step. - - Returns: - str, the name of the sampled task. - """ - probs = [ - a + b * np.exp(-self.params.alpha * current_step) - for a, b in self._descriptors - ] - probs = tuple(probs / np.sum(probs)) - sampled_task = np.random.choice(self.tasks, p=probs) - self.cur_probs = probs - return sampled_task - - -class ConstantScheduler(ShiftedExponentialScheduler): - """Constant schedule. Tasks are sampled from a fixed probability distribution. - """ - - def __init__(self, params): - super().__init__(params) - - for key, value in self.params.task_probs: - self.tasks.append(key) - self._descriptors.append((value, 0)) - - -class ExponentialScheduler(ShiftedExponentialScheduler): - """Exponential schedule. - - For a task with initial and final probabilities p_0 and p_1 respectively, - its unnormalized score is given by - `p_1 + (p_0 - p_1) * exp(-alpha * current_step)`. - """ - - def __init__(self, params): - super().__init__(params) - - for key, value in self.params.task_probs: - self.tasks.append(key) - self._descriptors.append((value[1], value[0] - value[1])) - - -class SigmoidScheduler(ShiftedExponentialScheduler): - """Sigmoid schedule. - - For a task with initial and final probabilities p_0 and p_1 respectively, - its unnormalized score is given by - `p_1 + (2 * p_0 - p_1) * exp(-alpha * current_step)`. - """ - - def __init__(self, params): - super().__init__(params) - - for key, value in self.params.task_probs: - self.tasks.append(key) - self._descriptors.append((value[1], 2 * value[0] - value[1])) - - -class RoundRobinScheduler(TaskScheduler): - """Deterministic sequential schedule.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('tasks', [], 'List of task names. No repetitions allowed.') - return p - - def __init__(self, params): - super().__init__(params) - assert isinstance(self.params.tasks, list) - self.tasks = sorted(self.params.tasks) - self.n_tasks = len(self.tasks) - self.cur_probs = [1. / self.n_tasks] * self.n_tasks # For summary - self.next_task_idx = 0 - - def Sample(self, current_step): - """Sample a task.""" - sampled_task = self.tasks[self.next_task_idx] - self.next_task_idx = (self.next_task_idx + 1) % self.n_tasks - return sampled_task - - -class SequentialScheduler(TaskScheduler): - """Deterministic schedule that stays a fixed number of steps on each task.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'task_steps', [], 'List of tuples of (task_name, steps_for_task). Goes ' - 'through list sequentially in the specified order, staying ' - 'steps_for_task steps on task_name. On completing the schedule, ' - 'remains on the final task for the rest of the time. Assumes ' - 'p.task_global_step is False.') - return p - - def __init__(self, params): - super().__init__(params) - assert isinstance(self.params.task_steps, list) - assert self.params.task_steps - self.task_steps = [] - for (name, steps) in self.params.task_steps: - assert steps > 0 - if self.task_steps: - self.task_steps.append((name, steps + self.task_steps[-1][1])) - else: - self.task_steps.append((name, steps)) - self.n_tasks = len(self.task_steps) - self.task_idx = 0 - self.cur_probs = [1] + [0] * (self.n_tasks - 1) # For summary - - def Sample(self, current_step): - """Sample a task.""" - sampled_task, to_step = self.task_steps[self.task_idx] - if current_step >= to_step and self.task_idx < self.n_tasks - 1: - self.task_idx += 1 - sampled_task = self.task_steps[self.task_idx][0] - self.cur_probs[self.task_idx - 1] = 0 - self.cur_probs[self.task_idx] = 1 - return sampled_task diff --git a/lingvo/core/task_scheduler_test.py b/lingvo/core/task_scheduler_test.py deleted file mode 100644 index e892ea75..00000000 --- a/lingvo/core/task_scheduler_test.py +++ /dev/null @@ -1,198 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for task_scheduler.""" - -import os -import lingvo.compat as tf -from lingvo.core import early_stop -from lingvo.core import task_scheduler -from lingvo.core import test_utils -import numpy as np - -_NUMPY_RANDOM_SEED = 9885784 - - -class SchedulerTests(test_utils.TestCase): - - def _TestSchedulerHelper(self, schedule, global_step, count_a): - np.random.seed(_NUMPY_RANDOM_SEED) - - task_counts = {'a': 0, 'b': 0} - for _ in range(100): - task = schedule.Sample(global_step) - task_counts[task] += 1 - self.assertEqual(task_counts['a'], count_a) - self.assertEqual(task_counts['b'], 100 - count_a) - - def testConstantScheduler(self): - """Approximate expected probabilities: (a:0.8, b:0.2).""" - p = task_scheduler.ConstantScheduler.Params() - p.task_probs = [('a', 0.8), ('b', 0.2)] - - schedule = p.Instantiate() - - self._TestSchedulerHelper(schedule, 0, 83) - - def testExponentialScheduler(self): - """Test exponential scheduler. - - Approximate probabilities: - t=0: (a:0, b:1) - t=1e5: (a:0.63, b:0.37) - t=1e10: (a:1, b:0) - """ - p = task_scheduler.ExponentialScheduler.Params() - p.alpha = 1e-5 - p.task_probs = [('a', (0, 1)), ('b', (1, 0))] - - schedule = p.Instantiate() - - self._TestSchedulerHelper(schedule, global_step=0, count_a=0) - self._TestSchedulerHelper(schedule, global_step=1e5, count_a=63) - self._TestSchedulerHelper(schedule, global_step=1e10, count_a=100) - - def testSigmoidScheduler(self): - """Test sigmoid scheduler. - - Approximate probabilities: - t=0: (a:0.5, b:0.5) - t=1e5: (a:0.73, b:0.27) - t=1e10: (a:1, b:0) - """ - p = task_scheduler.SigmoidScheduler.Params() - p.alpha = 1e-5 - p.task_probs = [('a', (0.5, 1)), ('b', (0.5, 0))] - - schedule = p.Instantiate() - - self._TestSchedulerHelper(schedule, global_step=0, count_a=54) - self._TestSchedulerHelper(schedule, global_step=1e5, count_a=73) - self._TestSchedulerHelper(schedule, global_step=1e10, count_a=100) - - def _setupTestAdaptiveScheduler(self, p): - logdir = tf.test.get_temp_dir() - tf.io.gfile.mkdir(os.path.join(logdir, 'decoder_dev_a')) - tf.io.gfile.mkdir(os.path.join(logdir, 'decoder_dev_b')) - - early_stop.MetricHistory.SetLogdirInMetricHistories(p, logdir) - - p.epsilon = 0.05 - p.tasks = ['a', 'b'] - p.expected = [0.3, 0.5] - - mh_a = early_stop.MetricHistory.Params() - mh_a.jobname = 'decoder_dev_a' - mh_a.metric = 'corpus_bleu' - mh_a.logdir = logdir - mh_a.local_filesystem = True - - mh_b = early_stop.MetricHistory.Params() - mh_b.jobname = 'decoder_dev_b' - mh_b.metric = 'corpus_bleu' - mh_b.logdir = logdir - mh_b.local_filesystem = True - - p.mh_a = mh_a - p.mh_b = mh_b - - schedule = p.Instantiate() - - early_stop.MetricHistory.ConditionalAppend(mh_a.jobname, mh_a.metric, 1, - 0.05) - early_stop.MetricHistory.ConditionalAppend(mh_b.jobname, mh_b.metric, 1, - 0.25) - - return schedule - - def testSimpleAdaptiveScheduler(self): - """Test simple adaptive schedule. - - Probability of task a: - (1.05 - 0.05/0.3) / ((1.05 - 0.05/0.3) + (1.05 - 0.25/0.5)) /approx 0.616 - """ - np.random.seed(_NUMPY_RANDOM_SEED) - - p = task_scheduler.SimpleAdaptiveScheduler.Params() - schedule = self._setupTestAdaptiveScheduler(p) - - self._TestSchedulerHelper(schedule, 0, 63) - - def testInverseRatioAdaptiveScheduler(self): - """Test simple adaptive schedule. - - Probability of task a: - 1.05/(13/60.) / (1.05/(13/60.) + 1.05/(11/20.)) /approx 0.717 - """ - np.random.seed(_NUMPY_RANDOM_SEED) - - p = task_scheduler.InverseRatioAdaptiveScheduler.Params() - schedule = self._setupTestAdaptiveScheduler(p) - self._TestSchedulerHelper(schedule, 0, 71) - - def testRoundRobinScheduler(self): - """Test round-robin scheduler.""" - p = task_scheduler.RoundRobinScheduler.Params() - p.tasks = ['a', 'b'] - - schedule = p.Instantiate() - for global_step in range(20): - task = schedule.Sample(global_step) - if global_step % 2 == 0: - self.assertEqual('a', task) - else: - self.assertEqual('b', task) - - def testRoundRobinSchedulerEvenStep(self): - """Should work regardless of step increment.""" - p = task_scheduler.RoundRobinScheduler.Params() - p.tasks = ['a', 'b'] - - schedule = p.Instantiate() - tasks = [] - - for global_step in [0, 2, 10, 17]: - tasks.append(schedule.Sample(global_step)) - self.assertEqual(['a', 'b', 'a', 'b'], tasks) - - def testSequentialScheduler(self): - """Test sequential scheduler.""" - p = task_scheduler.SequentialScheduler.Params() - p.task_steps = [('a', 8), ('b', 10), ('c', 2)] - - schedule = p.Instantiate() - tasks = [] - - for global_step in range(25): - tasks.append(schedule.Sample(global_step)) - expected_tasks = ['a'] * 8 + ['b'] * 10 + ['c'] * 2 + ['c'] * 5 - self.assertEqual(expected_tasks, tasks) - - def testSequentialSchedulerUnevenStep(self): - """Sequential schedule uses global_step even with uneven step increments.""" - p = task_scheduler.SequentialScheduler.Params() - p.task_steps = [('a', 8), ('b', 10), ('c', 2)] - - schedule = p.Instantiate() - tasks = [] - - for global_step in [0, 2, 10, 17, 21]: - tasks.append(schedule.Sample(global_step)) - expected_tasks = ['a', 'a', 'b', 'b', 'c'] - self.assertEqual(expected_tasks, tasks) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/test_helper.py b/lingvo/core/test_helper.py deleted file mode 100644 index ba80e3c2..00000000 --- a/lingvo/core/test_helper.py +++ /dev/null @@ -1,23 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Helpers for unittests.""" - -import os - - -def test_src_dir_path(relative_path): - return os.path.join(os.environ['TEST_SRCDIR'], '__main__/lingvo', - relative_path) diff --git a/lingvo/core/test_trainer_utils.py b/lingvo/core/test_trainer_utils.py deleted file mode 100644 index a2f1665a..00000000 --- a/lingvo/core/test_trainer_utils.py +++ /dev/null @@ -1,83 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Helpers for trainer-based tests.""" - -import os - -from absl.testing import parameterized -from lingvo import trainer as trainer_lib -import lingvo.compat as tf -from lingvo.core import test_utils - -FLAGS = tf.flags.FLAGS - - -def _ModelTuples(model_classes): - return [(model_class.__name__, model_class) for model_class in model_classes] - - -def MakeModelValidatorTestCase(model_classes): - """Returns a TestCase that validates the training for `model_classes`. - - Usage in a test: - - MakeModelValidatorTestCase([path.to.module.MyRegisteredModel1, - path.to.module.MyRegisteredModel2]) - - Args: - model_classes: A list of model classes. - """ - - class _ModelValidator(parameterized.TestCase, test_utils.TestCase): - """TestCase template for validating training and decoding models.""" - - def TrainerBuilds(self, model): - tmpdir = os.path.join(FLAGS.test_tmpdir, model.__name__) - - # Trainer should probably create these directories in the future. - tf.io.gfile.makedirs(os.path.join(tmpdir, 'train')) - - model_params = model() - cfg = model_params.Model() - cfg.input = model_params.GetDatasetParams('Train') - cfg.cluster.mode = 'sync' - cfg.cluster.job = 'trainer_client' - _ = trainer_lib.Trainer(cfg, '', tmpdir, tf_master='') - - def DecoderBuilds(self, model): - tmpdir = os.path.join(FLAGS.test_tmpdir, model.__name__) - tf.io.gfile.makedirs(tmpdir) - - model_params = model() - cfg = model_params.Model() - cfg.input = model_params.GetDatasetParams('Train') - cfg.cluster.mode = 'sync' - cfg.cluster.job = 'decoder' - cfg.cluster.task = 0 - cfg.cluster.decoder.replicas = 1 - _ = trainer_lib.Decoder('train', cfg, '', tmpdir, tf_master='') - - # Each of these take about 10-20 seconds to run, to build the model and - # execute the forward and backward pass building. - @parameterized.named_parameters(_ModelTuples(model_classes)) - def testTrain(self, model): - self.TrainerBuilds(model) - - @parameterized.named_parameters(_ModelTuples(model_classes)) - def testDecoder(self, model): - self.DecoderBuilds(model) - - return _ModelValidator diff --git a/lingvo/core/test_utils.py b/lingvo/core/test_utils.py deleted file mode 100644 index ffe8cd56..00000000 --- a/lingvo/core/test_utils.py +++ /dev/null @@ -1,212 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Helpers for unittests.""" - -import inspect -import re - -import lingvo.compat as tf -from lingvo.core import cluster_factory -from lingvo.core import py_utils -import numpy as np - -FLAGS = tf.flags.FLAGS - -# Enable tf.function when eager execution is on-by-default, which is the case -# when: -# - the test target doesn't depend on the disable_tf2 target, and -# - --define=tf_api_version=1 is not specified during the build. -# -# TODO(laigd): remove TF version check when 312743821 and 313682500 are in the -# release. -if tf.executing_eagerly() and tf.compat.v1.__version__ >= '2.3.0': - try: - FLAGS.if_use_tf_function = True - FLAGS.while_loop_use_tf_function = True - FLAGS.call_defun_use_tf_function = True - except tf.flags.UnrecognizedFlagError: - pass - -# Disable eager execution for all tests. -tf.disable_eager_execution() - -tf.flags.DEFINE_boolean( - 'update_goldens', False, - 'Update the goldens, rather than diffing against them.') - - -class TestCase(tf.test.TestCase): - """TestCase that performs Lingvo-specific setup.""" - - def setUp(self): - super().setUp() - # Ensure the global_step variable is created in the default graph. - py_utils.GetOrCreateGlobalStepVar() - - def _create_session(self, *args, **kwargs): - sess = super()._create_session(*args, **kwargs) - with sess.graph.as_default(): - # Ensure the global_step variable is created in every new session. - py_utils.GetOrCreateGlobalStepVar() - return sess - - def SetEval(self, mode): - return cluster_factory.SetEval(mode=mode) - - -def _ReplaceOneLineInFile(fpath, linenum, old, new): - """Replaces a line for the input file.""" - lines = [] - lines = open(fpath).readlines() - assert lines[linenum] == old, ( - 'Expected "%s" at line %d in file %s, but got "%s"' % - (lines[linenum], linenum + 1, fpath, old)) - tf.logging.info('Replacing {}:{}.'.format(fpath, linenum)) - lines[linenum] = new - with open(fpath, 'w') as f: - for l in lines: - f.write(l) - - -def ReplaceGoldenSingleFloat(old, float_value): - m = re.match( - r'(?P.*)\bCompareToGoldenSingleFloat\(' - r'(?P[^,]+), *' - r'[-+]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][-+]?\d+)?, *' - r'(?P.*)\)(?P.*)\n', old) - assert m - return ('%sCompareToGoldenSingleFloat(%s, %f, %s)%s\n' % - (m.group('prefix'), m.group('testobj'), float_value, m.group('v2'), - m.group('postfix'))) - - -def ReplaceGoldenStackAnalysis(new_float_value): - """Analyze the stack trace to figure out how to update the golden value.""" - src_file_frame = None - for frame in inspect.stack(): - if frame[-2] and 'CompareToGoldenSingleFloat' in frame[-2][0]: - src_file_frame = frame - break - assert src_file_frame - runfiles_pattern = 'runfiles/[^/]+/' - fpath = src_file_frame[1].split(runfiles_pattern)[-1] - line_num = src_file_frame[2] - 1 - old_line = src_file_frame[4][0] - new_line = ReplaceGoldenSingleFloat(old_line, new_float_value) - return fpath, line_num, old_line, new_line - - -def CompareToGoldenSingleFloat(testobj, v1, v2, *args, **kwargs): - """Compare golden value with real value. - - When running the bazel tests with FLAGS.update_goldens to be True, this - function automatically updates the golden value in the test file if there is a - mismatch and the calling site of CompareToGoldenSingleFloat is a 1-liner. E.g. - Code:: - - test_utils.CompareToGoldenSingleFloat(self, 0.3232, input_batch.label) - - works but this will not:: - - test_utils.CompareToGoldenSingleFloat(self, - 0.3232, - input_batch.label) - - Args: - testobj: A test object, such as tf.test.TestCase or test_utils.TestCase. - v1: the golden value to compare against. - v2: the returned value. - *args: extra args - **kwargs: extra args - """ - if not FLAGS.update_goldens: - testobj.assertAllClose(v1, v2, *args, **kwargs) - else: - _ReplaceOneLineInFile(*ReplaceGoldenStackAnalysis(v2)) - - -def PickEveryN(np_arr, step=1): - """Flattens `np_arr` and keeps one value every step values.""" - return np_arr.flatten()[::step] - - -def ComputeNumericGradient(sess, - y, - x, - delta=1e-4, - step=1, - extra_feed_dict=None): - """Compute the numeric gradient of y wrt to x. - - Args: - sess: The TF session constructed with a graph containing x and y. - y: A scalar TF Tensor in the graph constructed in sess. - x: A TF Tensor in the graph constructed in sess. - delta: Gradient checker's small perturbation of x[i]. - step: Only compute numerical gradients for a subset of x values. I.e. - dy/dx[i] is computed if i % step == 0. - extra_feed_dict: Additional feed_dict of tensors to keep fixed during the - gradient checking. - - Returns: - A Tensor of the same shape and dtype as x. If x[i] is not chosen - to compute the numerical gradient dy/x[i], the corresponding - value is set to 0. - """ - - x_data = sess.run(x) - x_size = x_data.size - x_shape = x_data.shape - - numeric_grad = np.zeros(x_size, dtype=x_data.dtype) - - # For variables we need to issue an assignment operation in order to update - # the value of the variable. This is because with resource variables x will be - # pointing to the handle rather than its value. - feed_dict = extra_feed_dict or {} - ph = tf.placeholder(x_data.dtype, x_shape) - x_assign = x.assign(ph) if isinstance(x, tf.Variable) else None - - for i in range(0, x_size, step): - x_pos = x_data.copy() - if x_size == 1: - x_pos += delta - else: - x_pos.flat[i] += delta - if x_assign is None: - feed_dict.update(dict([(x, x_pos)])) - else: - sess.run(x_assign, feed_dict={ph: x_pos}) - y_pos = sess.run(y, feed_dict=feed_dict) - - x_neg = x_data.copy() - if x_size == 1: - x_neg -= delta - else: - x_neg.flat[i] -= delta - if x_assign is None: - feed_dict.update(dict([(x, x_neg)])) - else: - sess.run(x_assign, feed_dict={ph: x_neg}) - y_neg = sess.run(y, feed_dict=feed_dict) - numeric_grad[i] = (y_pos - y_neg) / (2 * delta) - - # Restore the variable back to its original value to avoid breaking any - # further test code that operates on the graph. - if x_assign is not None: - sess.run(x_assign, feed_dict={ph: x_data}) - - return numeric_grad.reshape(x_shape) diff --git a/lingvo/core/test_utils_test.py b/lingvo/core/test_utils_test.py deleted file mode 100644 index 7325c0b9..00000000 --- a/lingvo/core/test_utils_test.py +++ /dev/null @@ -1,53 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for test_utils.""" - -import lingvo.compat as tf -from lingvo.core import test_utils - - -class TestUtilsTest(test_utils.TestCase): - - def testReplaceGoldenSingleFloat(self): - old_line = ' CompareToGoldenSingleFloat(self, 1.489712, vs[0])\n' - expected = ' CompareToGoldenSingleFloat(self, 1.000000, vs[0])\n' - actual = test_utils.ReplaceGoldenSingleFloat(old_line, 1.0) - self.assertEqual(expected, actual) - - old_line = ('test_utils.CompareToGoldenSingleFloat(self, -2.e-3, vs[0])' - ' # pylint: disable=line-too-long\n') - expected = ('test_utils.CompareToGoldenSingleFloat(self, 1.000000, vs[0])' - ' # pylint: disable=line-too-long\n') - actual = test_utils.ReplaceGoldenSingleFloat(old_line, 1.0) - self.assertEqual(expected, actual) - - def CompareToGoldenSingleFloat(self, unused_v1, v2): - return test_utils.ReplaceGoldenStackAnalysis(v2) - - def testReplaceGoldenStackAnalysis(self): - v2 = 2.0 - result = TestUtilsTest.CompareToGoldenSingleFloat(self, 1.0, v2) - self.assertTrue(result[0].endswith('test_utils_test.py')) - old_line = (' result = TestUtilsTest.CompareToGoldenSingleFloat(' - 'self, 1.0, v2)\n') - new_line = (' result = TestUtilsTest.CompareToGoldenSingleFloat(' - 'self, 2.000000, v2)\n') - self.assertEqual(old_line, result[2]) - self.assertEqual(new_line, result[3]) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/testdata/BUILD b/lingvo/core/testdata/BUILD deleted file mode 100644 index 825ef81d..00000000 --- a/lingvo/core/testdata/BUILD +++ /dev/null @@ -1,10 +0,0 @@ -package(default_visibility = ["//visibility:public"]) - -licenses(["notice"]) # Apache 2.0 - -filegroup( - name = "lenet_test_model", - data = glob([ - "lenet_test_model*", - ]), -) diff --git a/lingvo/core/testdata/lenet_test_model.data-00000-of-00001 b/lingvo/core/testdata/lenet_test_model.data-00000-of-00001 deleted file mode 100644 index 9ba4d73b..00000000 Binary files a/lingvo/core/testdata/lenet_test_model.data-00000-of-00001 and /dev/null differ diff --git a/lingvo/core/testdata/lenet_test_model.index b/lingvo/core/testdata/lenet_test_model.index deleted file mode 100644 index 15a7d663..00000000 Binary files a/lingvo/core/testdata/lenet_test_model.index and /dev/null differ diff --git a/lingvo/core/tokenizers.py b/lingvo/core/tokenizers.py deleted file mode 100644 index 3f3496e9..00000000 --- a/lingvo/core/tokenizers.py +++ /dev/null @@ -1,339 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tokenizers.""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import ops -from lingvo.core import py_utils -from lingvo.core import wpm_encoder - - -class BaseTokenizer(base_layer.BaseLayer): - """The base tokenizer.""" - - @classmethod - def Params(cls): - """Defaults params for tokenizers.""" - p = super().Params() - p.name = 'tokenizer' - p.Define('vocab_size', 64, 'The size of the vocabuary.') - p.Define( - 'append_eos', True, 'Whether to append at the end and treat ' - 'it as a non-padded label.') - p.Define('pad_to_max_length', True, - 'If True, output ids will be padded to max_length.') - # TODO(ciprianchelba): there should be a check in __init__ that the ids - # below are consistent with the ones assigned by the vocabulary. - p.Define('target_unk_id', 0, 'Target unknown token id.') - p.Define('target_sos_id', 1, 'Target start of sequence id.') - p.Define('target_eos_id', 2, 'Target end of sequence id.') - p.Define('target_wb_id', -1, 'Target word boundary id.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - self.sos_id = p.target_sos_id # - self.eos_id = p.target_eos_id # - self.unk_id = p.target_unk_id # - - self.SetVariableFree() - - def StringsToIds(self, - strs, - max_length, - external_append_eos=None, - languages=None): - """Tokenize strs into vocab ids. - - Args: - strs: A vector of strings. - max_length: An int providing the max_length for strs. - external_append_eos: Bool or None. If None, will be ignored and - `params.append_eos` will be used. If bool, will determine if an eos - symbol will be added to tokens. - languages: A vector of strings with the same length as `strs`. - - Returns: - A tuple (ids, labels, paddings) with the same shape [batch, maxlen]. - - - ids[i, j] is the input token id of i-th sample for j-th step. - - labels[i, j] is the target token id of i-th sample for j-th step. - - paddings[i, j] is 1 iff i-th sample's j-th step is padded. - - Raises: - ValueError: If unknown token type. - """ - p = self.params - - if external_append_eos is None: - append_eos = p.append_eos - else: - append_eos = external_append_eos - - return self._StringsToIdsImpl(strs, max_length, append_eos, languages) - - def _StringsToIdsImpl(self, strs, max_length, append_eos, languages): - raise NotImplementedError('Abstract method.') - - def IdsToStrings(self, ids, lens, languages=None): - """Converts ids back to strings. - - Args: - ids: A matrix of shape [batch, seqlen]. ids[i, :] is the i-th sample's - ids. - lens: A vector of shape [batch]. lens[i] is the sequence length of the - i-th sample. Only the first lens[i] tokens in ids[i, :] are valid tokens - for the i-th sequence. - languages: A vector of strings of shape [batch]. - - Returns: - sequences - A vector of shape [batch]. The converted string sequence. - - Raises: - ValueError: If unknown token type. - """ - raise NotImplementedError('Abstract method.') - - -class AsciiTokenizer(BaseTokenizer): - """A simple grapheme tokenizer. - - Maps a small vocabulary of character tokens for (lower case) letters, digits, - and punctuation symbols. - """ - - def _StringsToIdsImpl(self, strs, max_length, append_eos, languages): - p = self.params - return ops.ascii_to_token_id( - strs, - maxlen=max_length, - pad_to_maxlen=p.pad_to_max_length, - append_eos=append_eos) - - def IdsToStrings(self, ids, lens): - return ops.id_to_ascii(ids, lens) - - -class VocabFileTokenizer(BaseTokenizer): - """Tokenizers that use vocab files for look-up.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('token_vocab_filepath', None, - 'If set, specifies a filepath to the token vocab file.') - p.Define('ngram_vocab_filepath', None, - 'If set, specifies a filepath to the Ngram vocab file.') - p.Define('ngram_separator', '', - 'string separator to use when joining ngrams.') - p.Define('tokens_delimiter', ' ', - 'The delimiter to split a string to tokens with.') - p.Define( - 'load_token_ids_from_vocab', True, - 'Whether token ids are present in vocab (i.e. vocab contains two ' - 'colums, one for IDs and one for words). If false, line numbers ' - 'are used.') - return p - - @property - def _vocab_file_params(self): - return ['token_vocab_filepath', 'ngram_vocab_filepath'] - - def _CheckParams(self): - p = self.params - num_params_specified = sum( - [getattr(p, x) is not None for x in self._vocab_file_params]) - if num_params_specified != 1: - raise ValueError('Exactly one vocab file should be specified!') - - def _StringsToIdsImpl(self, strs, max_length, append_eos, languages): - self._CheckParams() - p = self.params - - if p.token_vocab_filepath: - return ops.str_to_vocab_tokens( - strs, - maxlen=max_length, - pad_to_maxlen=p.pad_to_max_length, - append_eos=append_eos, - vocab_filepath=p.token_vocab_filepath, - load_token_ids_from_vocab=p.load_token_ids_from_vocab, - delimiter=p.tokens_delimiter) - elif p.ngram_vocab_filepath: - raise NotImplementedError('ngram vocab StringsToIds is not supported.') - - def IdsToStrings(self, ids, lens): - self._CheckParams() - p = self.params - if p.token_vocab_filepath: - ngram_vocab_filepath = p.token_vocab_filepath - ngram_separator = p.tokens_delimiter - elif p.ngram_vocab_filepath: - ngram_vocab_filepath = p.ngram_vocab_filepath - ngram_separator = p.ngram_separator - - return ops.ngram_id_to_token( - token_ids=ids, - seq_lengths=lens, - ngram_vocab_filepath=ngram_vocab_filepath, - ngram_separator=ngram_separator) - - -class BpeTokenizer(BaseTokenizer): - """Tokenizers that use BPE vocab files and word to id lists for look-up.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('codes_filepath', None, - 'Specifies a filepath to the list of bpe codes vocab file.') - p.Define('words_to_ids_filepath', None, - 'Specifies a filepath to the word bpe vocab file.') - return p - - def _StringsToIdsImpl(self, strs, max_length, append_eos, languages): - p = self.params - - return ops.bpe_words_to_ids( - strs, - maxlen=max_length, - append_eos=append_eos, - tokenization_filepath=p.words_to_ids_filepath) - - def IdsToStrings(self, ids, lens): - p = self.params - - return ops.bpe_ids_to_words( - token_ids=ids, seq_lengths=lens, vocab_filepath=p.codes_filepath) - - -class WpmTokenizer(BaseTokenizer): - """Tokenizer for word-piece models.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'vocab_filepath', None, - 'Specifies a filepath to the WPM vocab. The vocab is sorted by ' - 'descending merge score.') - p.Define( - 'merge_prob', 1., - 'Probability of merging WPMs. If less than 1, then decomposition ' - 'of words into wordpieces will no longer be deterministic, and ' - 'result in longer ID sequences. At 0, it will be graphemes.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self._wpm_encoder = wpm_encoder.WpmEncoder(p.vocab_filepath, p.merge_prob) - assert p.target_unk_id == self._wpm_encoder.unk_id - assert p.target_sos_id == self._wpm_encoder.sentence_start_id - assert p.target_eos_id == self._wpm_encoder.sentence_end_id - - def _StringsToIdsImpl(self, strs, max_length, append_eos, languages): - """Takes a tensor of strings and returns id/padding tensors. - - This generates `token_ids`, `target_ids`, and `paddings` in the format that - is expected for tokenizers. This performs padding to a fixed length and - appends the end-of-sentence token as appropriate. - - Args: - strs: a string Tensor. - max_length: a python integer. The second dimension of the returned arrays. - All sequences are padded or truncated to that length. - append_eos: a python bool. See `BaseTokenizer` for explanation. - languages: A vector of strings with the same length as `strs`. - - Returns: - A tuple of 3 tensors: - - - token_ids: a tensor of sequences of WPM ids starting with SOS. Sequences - always end with EOS unless the sequence exceeds the maximum length. - Always padded with EOS. - - target_ids: a tensor of sequences of WPM ids not starting with SOS - but ending with EOS. Always padded with EOS. - - paddings: a tensor of floats indicating, at each position, whether - the corresponding position is padded. - """ - p = self.params - if append_eos is None: - append_eos = p.append_eos - - batch_size = py_utils.GetShape(strs)[0] - token_ids_ta = tf.TensorArray(tf.int32, batch_size) - target_ids_ta = tf.TensorArray(tf.int32, batch_size) - paddings_ta = tf.TensorArray(tf.float32, batch_size) - - def _TokenizeOneSentence(i, strs, token_ids_ta, target_ids_ta, paddings_ta): - """Tokenizes a single sentence.""" - ids, _ = self._wpm_encoder.Encode(strs[i]) - - if append_eos: - ids = tf.concat([ids, [self.eos_id]], axis=0) - - # This truncates after the eos is added, so some sentences might - # not have at the end. - token_ids_ta = token_ids_ta.write( - i, - py_utils.PadOrTrimTo( - tf.concat([[self.sos_id], ids], axis=0), [max_length], - self.eos_id)) - target_ids_ta = target_ids_ta.write( - i, py_utils.PadOrTrimTo(ids, [max_length], self.eos_id)) - paddings_ta = paddings_ta.write( - i, - py_utils.PadOrTrimTo( - tf.zeros_like(ids, dtype=tf.float32), [max_length], 1.)) - - return i + 1, strs, token_ids_ta, target_ids_ta, paddings_ta - - _, _, token_ids_ta, target_ids_ta, paddings_ta = tf.while_loop( - lambda i, *_: i < batch_size, - _TokenizeOneSentence, - loop_vars=(tf.constant(0, tf.int32), strs, token_ids_ta, target_ids_ta, - paddings_ta), - parallel_iterations=30, - back_prop=False) - - token_ids = token_ids_ta.stack() - target_ids = target_ids_ta.stack() - paddings = paddings_ta.stack() - - if not p.pad_to_max_length: - maxlen = tf.cast( - tf.round(tf.reduce_max(tf.reduce_sum(1.0 - paddings, axis=1))), - tf.int32) - token_ids = token_ids[:, :maxlen] - target_ids = target_ids[:, :maxlen] - paddings = paddings[:, :maxlen] - - return token_ids, target_ids, paddings - - def IdsToStrings(self, ids, lens): - """Takes integer matrices and returns vectors of strings.""" - ids = py_utils.with_dependencies([py_utils.assert_same_dim0([ids, lens])], - ids) - return tf.map_fn( - lambda inputs: self._wpm_encoder.Decode(inputs[0][:inputs[1]]), - (ids, lens), - dtype=tf.string, - parallel_iterations=30, - back_prop=False) diff --git a/lingvo/core/tpu_embedding_layers.py b/lingvo/core/tpu_embedding_layers.py deleted file mode 100644 index 8d63c2c2..00000000 --- a/lingvo/core/tpu_embedding_layers.py +++ /dev/null @@ -1,547 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""TPU embedding layers.""" - -import math -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils - -# pylint:disable=g-direct-tensorflow-import -from tensorflow.python.tpu import tpu_embedding as tpu_embedding_lib -# pylint:enable=g-direct-tensorflow-import - - -class _TPUEmbeddingOptimizer(base_layer.BaseLayer): - """Base class for TPUEmbeddingLayer, TPUEmbeddingTable optimizers.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('learning_rate', None, 'Used for updating embedding table.') - p.Define('clip_weight_min', None, - 'The minimum value to clip by; None means -infinity.') - p.Define('clip_weight_max', None, - 'The maximum value to clip by; None means +infinity.') - p.Define( - 'weight_decay_factor', None, - 'Amount of weight decay to apply; None means that the weights are not ' - 'decayed.') - p.Define( - 'multiply_weight_decay_factor_by_learning_rate', None, - 'If true, weight_decay_factor is multiplied by the current learning ' - 'rate.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - - @property - def tpu_embedding_optimizer_parameters(self): - return self._tpu_embedding_optimizer_parameters - - def CreateSlotVariablesAndOps(self, table_vars, tpu_embedding_table): - """Create slot variables and infeed/retrieval ops. - - Args: - table_vars: A list of all embedding table shard variables. - tpu_embedding_table: Parent TPUEmbeddingTable layer. - - Returns: - List of load ops - List of retrieve ops - """ - return NotImplementedError() - - -class TPUEmbeddingSGDOptimizer(_TPUEmbeddingOptimizer): - """SGD optimizer for TPUEmbeddingLayer, TPUEmbeddingTable.""" - - def __init__(self, params): - super().__init__(params) - p = self.params - self._tpu_embedding_optimizer_parameters = ( - tpu_embedding_lib.StochasticGradientDescentParameters( - learning_rate=p.learning_rate, - clip_weight_min=p.clip_weight_min, - clip_weight_max=p.clip_weight_max, - weight_decay_factor=p.weight_decay_factor, - multiply_weight_decay_factor_by_learning_rate=p - .multiply_weight_decay_factor_by_learning_rate)) - - def CreateSlotVariablesAndOps(self, table_vars, tpu_embedding_table): - load_op_list = [] - retrieve_op_list = [] - - num_tpu_hosts = tpu_embedding_table.params.num_tpu_hosts - table_name = tpu_embedding_table.table_name - - for host_id, table_var in zip(range(num_tpu_hosts), table_vars): - # The slot vars should be on the same device as the table var. - device_name = tpu_embedding_table.GetDeviceName(host_id) - with tf.device(device_name), py_utils.outside_all_rewrites(): - # Only the Trainer needs these ops. - if py_utils.use_tpu(): - # TPU Embedding load/retrieve ops need to be in the outer graph scope. - with tf.init_scope(): - tf.logging.info('creating load and retrieve ops.') - load_parameters_op = ( - tpu_embedding_lib.tpu_ops - .load_tpu_embedding_stochastic_gradient_descent_parameters( - parameters=table_var, - table_name=table_name, - num_shards=num_tpu_hosts, - shard_id=host_id)) - load_op_list.append(load_parameters_op) - - retrieved_table = ( - tpu_embedding_lib.tpu_ops - .retrieve_tpu_embedding_stochastic_gradient_descent_parameters( - table_name=table_name, - num_shards=num_tpu_hosts, - shard_id=host_id)) - retrieve_parameters_op = tpu_embedding_lib.control_flow_ops.group( - tf.assign(table_var, retrieved_table)) - retrieve_op_list.append(retrieve_parameters_op) - - return load_op_list, retrieve_op_list - - -class TPUEmbeddingAdagradOptimizer(_TPUEmbeddingOptimizer): - """Adagrad optimizer for TPUEmbeddingLayer, TPUEmbeddingTable.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('initial_accumulator', 0.1, - 'Initial value of Adagrad accumulator.') - p.Define( - 'use_gradient_accumulation', True, - 'Setting this to False makes embedding gradients calculation less ' - 'accurate but faster. See tpu_embedding_lib for more details.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self._tpu_embedding_optimizer_parameters = ( - tpu_embedding_lib.AdagradParameters( - learning_rate=p.learning_rate, - initial_accumulator=p.initial_accumulator, - clip_weight_min=p.clip_weight_min, - clip_weight_max=p.clip_weight_max, - weight_decay_factor=p.weight_decay_factor, - multiply_weight_decay_factor_by_learning_rate=p - .multiply_weight_decay_factor_by_learning_rate)) - - def CreateSlotVariablesAndOps(self, table_vars, tpu_embedding_table): - p = self.params - - load_op_list = [] - retrieve_op_list = [] - - num_tpu_hosts = tpu_embedding_table.params.num_tpu_hosts - table_name = tpu_embedding_table.table_name - slot_var_collections = [tpu_embedding_table.__class__.__name__ + '_vars'] - - for host_id, table_var in zip(range(num_tpu_hosts), table_vars): - # The slot vars should be on the same device as the table var. - device_name = tpu_embedding_table.GetDeviceName(host_id) - with tf.device(device_name), py_utils.outside_all_rewrites(): - w_ada = py_utils.WeightParams( - shape=table_var.shape.as_list(), - init=py_utils.WeightInit.Constant(p.initial_accumulator), - dtype=p.dtype, - collections=slot_var_collections) - var_name = tpu_embedding_table.GetVariableName(host_id) - tpu_embedding_table.CreateVariable( - '%s/Adagrad' % var_name, w_ada, trainable=False) - accumulator_var = tpu_embedding_table.vars['%s/Adagrad' % var_name] - - # Only the Trainer needs these ops. - if py_utils.use_tpu(): - # TPU Embedding load/retrieve ops need to be in the outer graph scope. - with tf.init_scope(): - tf.logging.info('creating load and retrieve ops.') - load_parameters_op = ( - tpu_embedding_lib.tpu_ops.load_tpu_embedding_adagrad_parameters( - parameters=table_var, - accumulators=accumulator_var, - table_name=table_name, - num_shards=num_tpu_hosts, - shard_id=host_id)) - load_op_list.append(load_parameters_op) - - retrieved_table, retrieved_accumulator = ( - tpu_embedding_lib.tpu_ops - .retrieve_tpu_embedding_adagrad_parameters( - table_name=table_name, - num_shards=num_tpu_hosts, - shard_id=host_id)) - retrieve_parameters_op = tpu_embedding_lib.control_flow_ops.group( - tf.assign(table_var, retrieved_table), - tf.assign(accumulator_var, retrieved_accumulator)) - retrieve_op_list.append(retrieve_parameters_op) - - return load_op_list, retrieve_op_list - - -class TPUEmbeddingTable(base_layer.BaseLayer): - """An embedding table controlled by TPUEmbeddingLayer. - - Note that all input_keys needs to be declared upfront. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('vocab_size', 0, 'Depth of the input.') - p.Define('embedding_dim', 0, 'Depth of the output.') - p.Define('input_keys', None, 'Name of inputs in InputBatch.') - p.Define( - 'combiner', 'mean', - 'Must be "sum", "sqrtn", "mean" or None in the case of a ' - '"sequence embedding "') - p.Define( - 'max_sequence_length', None, - 'If not None or 0, embedding lookup will return a ' - '"sequence embedding" of shape ' - '`[batch, max_sequence_length, embedding_dim]` without applying a ' - 'sequence reducing combiner') - p.Define('num_tpu_hosts', 0, 'Total number of TPU hosts.') - p.Define( - 'optimizer', None, - 'Table optimizer parameters. Will override the optimizer parameters ' - 'defined in this table\'s TPUEmbeddingLayer.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.vocab_size > 0 - assert p.embedding_dim > 0 - assert p.input_keys - assert p.name - assert p.num_tpu_hosts > 0 - if p.combiner is None: - assert p.max_sequence_length - if p.max_sequence_length is not None and p.max_sequence_length > 0: - assert p.combiner is None - - self._ids_per_shard = int(math.ceil(float(p.vocab_size) / p.num_tpu_hosts)) - self._padded_vocab_size = self._ids_per_shard * p.num_tpu_hosts - self._input_keys = p.input_keys - - self._max_sequence_length = 0 - if p.max_sequence_length: - self._max_sequence_length = p.max_sequence_length - - self.CreateChild('optimizer', p.optimizer) - - self._table_name = '{}_table'.format(p.name) - self._table_config = tpu_embedding_lib.TableConfig( - self._padded_vocab_size, - p.embedding_dim, - combiner=p.combiner, - optimization_parameters=self.optimizer - .tpu_embedding_optimizer_parameters) - - self._load_op_list = [] - self._retrieve_op_list = [] - - def _CreateLayerVariables(self): - p = self.params - w_pc = py_utils.WeightParams( - shape=[self._ids_per_shard, p.embedding_dim], - init=p.params_init, - dtype=p.dtype, - collections=[self.__class__.__name__ + '_vars']) - - embedding_table_vars = [] - for i in range(p.num_tpu_hosts): - device_name = self.GetDeviceName(i) - with tf.device(device_name), py_utils.outside_all_rewrites(): - var_name = self.GetVariableName(i) - self.CreateVariable(var_name, w_pc) - embedding_var = self.vars[var_name] - embedding_table_vars.append(embedding_var) - # Remove from _private_vars / _private_thetas to be added later as wm. - del self._private_vars[var_name] - del self._private_theta[var_name] - - if not py_utils.use_tpu(): - # We don't want to add this for TrainerTpu, otherwise the identity - # reference leads to copying the embedding to the TPU for no reason. - # However, this is needed for CPU (eval/decode/controller). - self._private_vars['wm'] = embedding_table_vars - self._private_theta['wm'] = [tf.identity(v) for v in embedding_table_vars] - - # Only trainer and controller need slot variables and load/retrieve ops. - if not self.do_eval: - self._load_op_list, self._retrieve_op_list = ( - self.optimizer.CreateSlotVariablesAndOps(embedding_table_vars, self)) - - # Return device to place sharded variables on. - def GetDeviceName(self, host_id): - if self.do_eval: - return None - else: - return '{}/replica:0/task:{}/device:CPU:0'.format( - self.cluster.params.worker.name, host_id) - - # Return variable name for embedding table shards. - def GetVariableName(self, host_id): - return 'var_%d' % host_id - - @property - def table_config(self): - return self._table_config - - @property - def table_name(self): - return self._table_name - - @property - def retrieve_op_list(self): - return self._retrieve_op_list - - @property - def load_op_list(self): - return self._load_op_list - - @property - def input_keys(self): - return self._input_keys - - @property - def max_sequence_length(self): - return self._max_sequence_length - - def CpuEmbLookup(self, ids_map): - """CPU evaluation embedding lookup. - - Args: - ids_map: A dict of `input_key` string -> [batch, sequence] int32 Tensor. - -1 is used as a padding id. - - Returns: - An activations dict of string -> float32 Tensor. - For non-sequence embeddings: [batch, 1, embedding_dim] - For sequence embeddings: [batch, max_sequence_length, embedding_dim] - - """ - p = self.params - rets = py_utils.NestedMap() - if self.max_sequence_length > 0: - # "Sequence embedding", no combiner case - for k, ids in ids_map.items(): - embs = tf.nn.embedding_lookup(self.theta.wm, tf.reshape(ids, [-1])) - out_shape = tf.concat([tf.shape(ids), [p.embedding_dim]], 0) - rets[k] = tf.reshape(embs, out_shape) - else: - # Non-"Sequence embedding", combiner case - for k, ids in ids_map.items(): - # Dense to sparse. - dense_shape = tf.shape(ids, out_type=tf.int64) - sample_indices = tf.cast(tf.where(tf.not_equal(ids, -1)), tf.int64) - embedding_indices = tf.cast(tf.gather_nd(ids, sample_indices), tf.int64) - sparse_ids = tf.SparseTensor( - indices=sample_indices, - values=embedding_indices, - dense_shape=dense_shape) - # [?, embedding_dim] - # For tf.nn.embedding_lookup_sparse, output.dim0 might be different from - # sparse_ids.dense_shape.dim0. - # In fact, the '?' is the smallest span starting from the index=0 that - # covers all the results. - embs = tf.nn.embedding_lookup_sparse( - self.theta.wm, - sparse_ids, - None, # sp_weights - combiner=p.combiner) - batch_size = dense_shape[0] - # Explicitly pad results to maintain dim0=batch. - dim0_padlen = tf.cast(batch_size, tf.int32) - tf.shape(embs)[0] - embs = tf.pad(embs, [[0, dim0_padlen], [0, 0]]) - # [batch, 1, embedding_dim] - embs = py_utils.HasShape(embs, [batch_size], ndims=1) - rets[k] = tf.expand_dims(embs, 1) - return rets - - -class TPUEmbeddingLayer(base_layer.BaseLayer): - """Monolithic interface to TPU embedding. - - This layer has some important caveats, due to the interface of the - TPU embedding hardware. Its behavior most closely mimics that of - tf.nn.embedding_lookup_sparse. - - Supports multiple tables and multiple input_keys per table. - Requires its own optimizer parameters. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('tables', None, 'TPUEmbeddingTables') - p.Define('pipeline_execution_with_tensor_core', False, - 'Set to True to be faster. See tpu_embedding.py for details.') - p.Define('batch_size', 0, 'Per-core batch size.') - p.Define( - 'optimizer', TPUEmbeddingAdagradOptimizer.Params(), - 'Layer optimizer parameters. Will be used for any TPUEmbeddingTables ' - 'with None optimizer parameters.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - assert p.tables - assert p.batch_size > 0 - assert p.name - - num_tpu_hosts = p.tables[0].num_tpu_hosts - assert all([t.num_tpu_hosts == num_tpu_hosts for t in p.tables]) - - # Stop if a table has no optimizer parameters and the layer also has no - # optimizer parameters - table_optimizer_missing = any( - table_params.optimizer is None for table_params in p.tables) - if not p.optimizer and table_optimizer_missing: - raise ValueError( - 'A table is missing optimizer parameters, and no layer-level ' - 'optimizer parameters were given.') - elif table_optimizer_missing: - for table_params in p.tables: - if table_params.optimizer is None: - table_params.optimizer = p.optimizer.Copy() - - self.CreateChildren('tables', p.tables) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - for table in self.tables: - table.InstantiateVariables() - super()._CreateChildrenVariables() - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - - load_op_list = [] - retrieve_op_list = [] - - # At the feature level, track which are associated - # with "sequence embeddings". - self._sequence_features = {} - - if py_utils.use_tpu(): - num_cores = self.cluster.params.worker.tpus_per_replica - global_batch_size = ( - self.params.batch_size * self.cluster.num_splits_per_client) - table_to_config_dict = {} - feature_to_config_dict = {} - for table in self.tables: - table_to_config_dict[table.table_name] = table.table_config - load_op_list += table.load_op_list - retrieve_op_list += table.retrieve_op_list - for feature in table.input_keys: - if table.max_sequence_length > 0: - self._sequence_features[feature] = True - feature_to_config_dict[feature] = tpu_embedding_lib.FeatureConfig( - table.table_name, max_sequence_length=table.max_sequence_length) - mode = tpu_embedding_lib.TRAINING - device_config = tpu_embedding_lib.DeviceConfig( - num_cores=num_cores, - num_hosts=self.params.tables[0].num_tpu_hosts, - job_name=self.cluster.params.worker.name) - self._tpu_embedding = tpu_embedding_lib.TPUEmbedding( - table_to_config_dict, - feature_to_config_dict, - global_batch_size, - mode, - master=None, - pipeline_execution_with_tensor_core=( - self.params.pipeline_execution_with_tensor_core), - device_config=device_config) - tf.add_to_collection(py_utils.TPU_EMBEDDING, self._tpu_embedding) - - tf.logging.info('adding load and retrieve ops to collection.') - tf.add_to_collection(py_utils.TPU_EMBEDDING_LOAD_OPS, load_op_list) - tf.add_to_collection(py_utils.TPU_EMBEDDING_RETRIEVE_OPS, - retrieve_op_list) - - def EmbLookup(self, ids_map): - """Looks up embedding vectors for each entry in ids_map. - - Since the TPUEmbedding is monolothic, and consulted once per - FProp/BPRop, we must centralize the lookup. Thus, for multiple - features, we contain them into a single-lookup rather than allowing - the caller to call Lookup multiple times. - - Currently, there's also an implied combination step which combines - the sequence into a single set of activations by sum, mean or - sqrtn. - - Args: - ids_map: A dict of `input_key` string -> [batch, sequence] int32 Tensor. - -1 is used as a padding id. - - Returns: - Activations dict of string -> - For non-sequence embeddings: [batch, 1, embedding_dim], - For sequence embeddings: [batch, max_sequence_length, embedding_dim] - float32 Tensor. - """ - - def TpuEmbLookup(ids_map): - """TPU Embedding lookup.""" - del ids_map - activations = self._tpu_embedding.get_activations() - tf.add_to_collection(py_utils.TPU_EMBEDDING_ACTIVATIONS, activations) - ret = py_utils.NestedMap() - for k, v in activations.items(): - if k in self._sequence_features: - ret[k] = v - else: - # Non-sequence embeddings, we fill the "time" dimension with 1. - ret[k] = tf.expand_dims(v, axis=[1]) - return ret - - def CpuEmbLookup(ids_map): - """CPU evaluation embedding lookup.""" - rets = py_utils.NestedMap() - for table in self.tables: - table_id_map = {} - for key in table.input_keys: - table_id_map[key] = ids_map[key] - table_rets = table.CpuEmbLookup(table_id_map) - # Merge table_rets with rets - for k, v in table_rets.items(): - rets[k] = v - return rets - - if self.do_eval: - return CpuEmbLookup(ids_map) - elif not py_utils.use_tpu(): - # Contoller - return CpuEmbLookup(ids_map) - else: - # TPU Trainer - return TpuEmbLookup(ids_map) diff --git a/lingvo/core/tpu_summary.py b/lingvo/core/tpu_summary.py deleted file mode 100644 index 97a345ae..00000000 --- a/lingvo/core/tpu_summary.py +++ /dev/null @@ -1,227 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Alternative to tf.summary for TPU without host call and outside compilation. - -To add a summary, in layer implementation code call:: - - tpu_summary.scalar('name', value) - -Then in trainer code:: - - my_model = my_model_params.Instantiate() - with tpu_summary.context(): - my_model.ConstructFPropBPropGraph() - summaries = tpu_summary.merge_all() - -If some summaries are created inside a while loop:: - - my_model = my_model_params.Instantiate() - with tpu_summary.context(rewrite_while_loop=True): - output = my_model.GreedyDecodeIds() - summaries = tpu_summary.merge_all() - -Note that you must call tpu_summary.merge_all() inside tpu_summary.context(), -or else it will return nothing. - -Note also that merge_all() returns a dict of tensors, not a serialized proto. -As such, merge_all() can be used inside tpu rewrite context. -It is left up to the caller to decide how to return summary tensors from TPU -and what to do next. -""" - -import contextlib - -from lingvo import compat as tf - - -class TpuSummaryScalar: - """Plain data object. Much better than namedtuple.""" - name = None - value = None - name_scope = None - while_loop_reduce = None - - -class TpuSummaryContext: - """Non-reentrant context that holds the list of summary tensors.""" - - _global_stack = [] - - @classmethod - def current(cls): - """Returns current context or None.""" - if cls._global_stack: - return cls._global_stack[-1] - else: - return None - - def __init__(self): - self.summary_tensors = [] - - def __enter__(self): - assert not self._global_stack, 'no re-entry' - self._global_stack.append(self) - - def __exit__(self, *args): - self._global_stack.pop() - - -class RewriteLoopContext: - """Context manager. Rewrites tf.while_loop to propagate summary tensors.""" - - def __init__(self, max_loop_vars=16, dtype=tf.float32): - self.max_loop_vars = max_loop_vars - self.dtype = dtype - self.tf_while_loop = None - - def __enter__(self): - self.tf_while_loop = tf.while_loop - tf.while_loop = self.while_loop - - def __exit__(self, *args): - tf.while_loop = self.tf_while_loop - self.tf_while_loop = None - - def while_loop(self, cond, body, loop_vars, **kwargs): - """Wrapper for tf.while_loop that adds summary_tensors to loop vars.""" - ctx = TpuSummaryContext.current() - assert ctx is not None, 'must be inside TpuSummaryContext' - assert self.tf_while_loop, 'must be inside self RewriteLoopContext' - - outer_summary_tensors = ctx.summary_tensors - ctx.summary_tensors = [] - - accumulators = [ - tf.constant(0, dtype=self.dtype) for _ in range(self.max_loop_vars) - ] - size = kwargs['maximum_iterations'] # as if not None - arrays = [ - tf.TensorArray(self.dtype, size) for _ in range(self.max_loop_vars) - ] - loop_count = tf.constant(0, dtype=tf.int32) - - def loop_body(loop_vars, accumulators, arrays, loop_count): - loop_vars = body(*loop_vars) - del ctx.summary_tensors[self.max_loop_vars:] - for i, x in enumerate(ctx.summary_tensors): - if x.while_loop_reduce == 'stack': - arrays[i] = arrays[i].write(loop_count, tf.cast(x.value, self.dtype)) - else: - accumulators[i] += tf.cast(x.value, self.dtype) - loop_count += 1 - return loop_vars, accumulators, arrays, loop_count - - def loop_cond(loop_vars, accumulators, arrays, loop_count): - del accumulators, arrays, loop_count - return cond(*loop_vars) - - loop_vars, accumulators, arrays, loop_count = self.tf_while_loop( - cond=loop_cond, - body=loop_body, - loop_vars=(loop_vars, accumulators, arrays, loop_count), - **kwargs) - - for i, x in enumerate(ctx.summary_tensors): - if x.while_loop_reduce == 'stack': - x.value = arrays[i].stack() - elif x.while_loop_reduce == 'mean': - denominator = tf.cast(tf.math.maximum(1, loop_count), self.dtype) - x.value = accumulators[i] / denominator - else: - x.value = accumulators[i] - - ctx.summary_tensors = outer_summary_tensors + ctx.summary_tensors - return loop_vars - - -def scalar(name, value, while_loop_reduce='mean'): - """Adds summary scalar. - - Outside of tpu_summary.context() does nothing. - - Args: - name: string name - value: scalar tensor value - while_loop_reduce: optional argument, determines what to do when this - summary appears inside a tf.while_loop. Can be 'mean' or 'sum'. - """ - assert while_loop_reduce in ('mean', 'sum') - ctx = TpuSummaryContext.current() - if ctx is None: - return - x = TpuSummaryScalar() - x.name = str(name) - x.value = tf.convert_to_tensor(value) - if x.value.shape != (): # pylint: disable=g-explicit-bool-comparison - raise ValueError('use tpu_summary.tensor() instead: %r' % value) - x.name_scope = tf.get_default_graph().get_name_scope() - x.while_loop_reduce = while_loop_reduce - ctx.summary_tensors.append(x) - - -def tensor(name, value): - """Adds summary tensor. Similar to scalar() but allows other shapes.""" - ctx = TpuSummaryContext.current() - if ctx is None: - return - x = TpuSummaryScalar() - x.name = str(name) - x.value = tf.convert_to_tensor(value) - x.name_scope = tf.get_default_graph().get_name_scope() - x.while_loop_reduce = 'stack' - ctx.summary_tensors.append(x) - - -def merge_all(): - """Returns all summary tensors as a dict of {name: tensor}. - - Note that this is not the same return type as tf.summary.merge_all - which returns a serialized proto string. - - Outside of tpu_summary.context() returns {} - """ - ctx = TpuSummaryContext.current() - if ctx is None: - return {} - g = tf.get_default_graph() - ret = {} - for x in ctx.summary_tensors: - if x.value.graph is not g: - raise ValueError('Tensor %r %r is not an element of this graph.' % - (x.name, x.value)) - ret['%s/%s' % (x.name, x.name_scope)] = x.value - return ret - - -# pylint: disable=g-doc-return-or-yield -@contextlib.contextmanager -def context(rewrite_while_loop=False, - max_loop_vars=16, - loop_vars_dtype=tf.float32): - """TPU summary context. - - Args: - rewrite_while_loop: rewrite tf.while_loop to propagate summaries - max_loop_vars: number of loop vars added by rewrite_while_loop - loop_vars_dtype: dtype of loop vars added by rewrite_while_loop - """ - if not rewrite_while_loop: - with TpuSummaryContext(): - yield - else: - with TpuSummaryContext(), RewriteLoopContext(max_loop_vars, - loop_vars_dtype): - yield diff --git a/lingvo/core/tpu_summary_test.py b/lingvo/core/tpu_summary_test.py deleted file mode 100644 index f07a29ba..00000000 --- a/lingvo/core/tpu_summary_test.py +++ /dev/null @@ -1,224 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for tpu_summary.""" - -from lingvo import compat as tf - -from lingvo.core import test_utils -from lingvo.core import tpu_summary - - -class MockTransformer: - - def FProp(self, x, y): - for i in range(3): - with tf.name_scope('encoder%03d' % i): - x = tf.identity(x) - y = tf.identity(y) - x = x + 1 - tpu_summary.scalar('x_mean', tf.reduce_mean(x)) - tpu_summary.scalar('y_mean', tf.reduce_mean(y)) - for i in range(3): - with tf.name_scope('decoder%03d' % i): - x = tf.identity(x) - y = tf.identity(y) - y = y + 1 - tpu_summary.scalar('x_mean', tf.reduce_mean(x)) - tpu_summary.scalar('y_mean', tf.reduce_mean(y)) - return x, y - - def BeamSearch(self, x, y, decoder_reduce_sum=False): - for i in range(3): - with tf.name_scope('encoder%03d' % i): - x = tf.identity(x) - y = tf.identity(y) - x = x + 1 - tpu_summary.scalar('x_mean', tf.reduce_mean(x)) - tpu_summary.scalar('y_mean', tf.reduce_mean(y)) - - def DecoderStep(x, y): - for i in range(3): - with tf.name_scope('decoder%03d' % i): - x = tf.identity(x) - y = tf.identity(y) - y = y + 1 - if decoder_reduce_sum: - tpu_summary.scalar( - 'x_mean', tf.reduce_mean(x), while_loop_reduce='sum') - tpu_summary.scalar( - 'y_mean', tf.reduce_mean(y), while_loop_reduce='sum') - else: - tpu_summary.scalar('x_mean', tf.reduce_mean(x)) - tpu_summary.scalar('y_mean', tf.reduce_mean(y)) - return x, y - - def DecoderCond(x, y): - del x, y - return True - - (x, y) = tf.while_loop( - cond=DecoderCond, - body=DecoderStep, - loop_vars=(x, y), - maximum_iterations=10) - return x, y - - -class TpuSummaryTest(test_utils.TestCase): - - def testNoContext(self): - with self.session() as sess: - model = MockTransformer() - x = tf.constant(0, dtype=tf.float32) - y = tf.constant(0, dtype=tf.int64) - x, y = model.FProp(x, y) - x, y = sess.run((x, y)) - self.assertEqual((3.0, 3), (x, y)) - - def _CanonicalizeSummaryName(self, summaries): - ret = dict() - for k in summaries: - ret[k.replace('/while', '')] = summaries[k] - return ret - - def testMergeAll(self): - with self.session() as sess: - model = MockTransformer() - x = tf.constant(0, dtype=tf.float32) - y = tf.constant(0, dtype=tf.int64) - with tpu_summary.context(): - x, y = model.FProp(x, y) - summaries = tpu_summary.merge_all() - x, y, summaries = sess.run((x, y, summaries)) - self.assertEqual((3.0, 3), (x, y)) - expected = { - 'x_mean/decoder000': 3.0, - 'x_mean/decoder001': 3.0, - 'x_mean/decoder002': 3.0, - 'x_mean/encoder000': 1.0, - 'x_mean/encoder001': 2.0, - 'x_mean/encoder002': 3.0, - 'y_mean/decoder000': 1, - 'y_mean/decoder001': 2, - 'y_mean/decoder002': 3, - 'y_mean/encoder000': 0, - 'y_mean/encoder001': 0, - 'y_mean/encoder002': 0, - } - self.assertEqual(expected, self._CanonicalizeSummaryName(summaries)) - - def testWhileLoopNoMergeAll(self): - with self.session() as sess: - model = MockTransformer() - x = tf.constant(0, dtype=tf.float32) - y = tf.constant(0, dtype=tf.int64) - with tpu_summary.context(): - x, y = model.BeamSearch(x, y) - x, y = sess.run((x, y)) - self.assertEqual((3.0, 30), (x, y)) - - def testWhileLoopNoRewrite(self): - with self.session() as sess: - model = MockTransformer() - x = tf.constant(0, dtype=tf.float32) - y = tf.constant(0, dtype=tf.int64) - with tpu_summary.context(): - x, y = model.BeamSearch(x, y) - # ValueError: Tensor decoder000/Mean:0 is not an element of this graph. - with self.assertRaises(ValueError): - summaries = tpu_summary.merge_all() - x, y, summaries = sess.run((x, y, summaries)) - - def testWhileLoopRewrite(self): - with self.session() as sess: - model = MockTransformer() - x = tf.constant(0, dtype=tf.float32) - y = tf.constant(0, dtype=tf.int64) - with tpu_summary.context(rewrite_while_loop=True): - x, y = model.BeamSearch(x, y) - summaries = tpu_summary.merge_all() - tf.logging.info('summaries=%r', summaries) - x, y, summaries = sess.run((x, y, summaries)) - self.assertEqual((3.0, 30), (x, y)) - expected = { - 'x_mean/encoder000': 1.0, - 'x_mean/encoder001': 2.0, - 'x_mean/encoder002': 3.0, - 'y_mean/encoder000': 0, - 'y_mean/encoder001': 0, - 'y_mean/encoder002': 0, - 'x_mean/decoder000': 3.0, - 'x_mean/decoder001': 3.0, - 'x_mean/decoder002': 3.0, - 'y_mean/decoder000': 14.5, - 'y_mean/decoder001': 15.5, - 'y_mean/decoder002': 16.5, - } - self.assertEqual(expected, self._CanonicalizeSummaryName(summaries)) - - def testWhileLoopRewriteMaxVarsLimit(self): - with self.session() as sess: - model = MockTransformer() - x = tf.constant(0, dtype=tf.float32) - y = tf.constant(0, dtype=tf.int64) - with tpu_summary.context(rewrite_while_loop=True, max_loop_vars=2): - x, y = model.BeamSearch(x, y) - summaries = tpu_summary.merge_all() - tf.logging.info('summaries=%r', summaries) - x, y, summaries = sess.run((x, y, summaries)) - self.assertEqual((3.0, 30), (x, y)) - expected = { - 'x_mean/encoder000': 1.0, - 'x_mean/encoder001': 2.0, - 'x_mean/encoder002': 3.0, - 'y_mean/encoder000': 0, - 'y_mean/encoder001': 0, - 'y_mean/encoder002': 0, - 'x_mean/decoder000': 3.0, - 'y_mean/decoder000': 14.5, - } - self.assertEqual(expected, self._CanonicalizeSummaryName(summaries)) - - def testWhileLoopReduceSum(self): - with self.session() as sess: - model = MockTransformer() - x = tf.constant(0, dtype=tf.float32) - y = tf.constant(0, dtype=tf.int64) - with tpu_summary.context(rewrite_while_loop=True): - x, y = model.BeamSearch(x, y, decoder_reduce_sum=True) - summaries = tpu_summary.merge_all() - tf.logging.info('summaries=%r', summaries) - x, y, summaries = sess.run((x, y, summaries)) - self.assertEqual((3.0, 30), (x, y)) - expected = { - 'x_mean/encoder000': 1.0, - 'x_mean/encoder001': 2.0, - 'x_mean/encoder002': 3.0, - 'y_mean/encoder000': 0, - 'y_mean/encoder001': 0, - 'y_mean/encoder002': 0, - 'x_mean/decoder000': 30.0, - 'x_mean/decoder001': 30.0, - 'x_mean/decoder002': 30.0, - 'y_mean/decoder000': 145.0, - 'y_mean/decoder001': 155.0, - 'y_mean/decoder002': 165.0, - } - self.assertEqual(expected, self._CanonicalizeSummaryName(summaries)) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/trainer_test_utils.py b/lingvo/core/trainer_test_utils.py deleted file mode 100644 index 1939bea4..00000000 --- a/lingvo/core/trainer_test_utils.py +++ /dev/null @@ -1,137 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Toy models and input generation tools for testing trainer code.""" - -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import base_model -from lingvo.core import py_utils -from lingvo.core import summary_utils - - -class CountingInputGenerator(base_input_generator.BaseInputGenerator): - """Produces deterministic inputs for IdentityRegressionModel. - - src_ids increment by 1, so a 2x2 batch would look like: - [[0, 1], [2, 3]] - and the next batch would be: - [[4, 5], [6, 7]] - - Targets are the sum of the src_ids: - [1, 5] - next batch: - [9, 13] - - Since `sum(src_ids) = target`, we expect that the regression model of - `target = sum(m * src_ids) + b` will learn `m = 1` and `b = 0`. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Delete('batch_size') - p.Define('batch_size', 2, 'batch size') - p.Define('shape', [2, 2], 'source shape.') - return p - - def __init__(self, params): - super().__init__(params) - self.shape = params.shape - - def _InputBatch(self): - length = tf.reduce_prod(self.shape) - counter = summary_utils.StatsCounter('CountingInputGenerator') - new_value = tf.cast(counter.IncBy(length), dtype=tf.int32) - length - new_value = tf.stop_gradient(new_value) - values = new_value + tf.range(length) - shaped_values = tf.reshape(tf.cast(values, dtype=tf.float32), self.shape) - targets = tf.reduce_sum(shaped_values, axis=0) - return py_utils.NestedMap(src_ids=shaped_values, tgt_ids=targets) - - -class IdentityRegressionTask(base_model.BaseTask): - """A simple regression task for testing.""" - - def __init__(self, params): - super().__init__(params) - self.global_steps = [] - self.metrics = [] - self.result_per_example_tensors = [] - - def _CreateLayerVariables(self): - super()._CreateLayerVariables() - self.CreateVariable( - 'm', - py_utils.WeightParams(shape=[], init=py_utils.WeightInit.Uniform())) - self.CreateVariable( - 'b', - py_utils.WeightParams(shape=[], init=py_utils.WeightInit.Uniform())) - - def ComputePredictions(self, theta, input_batch): - """sum(m * x) + b.""" - return tf.reduce_sum(theta.m * input_batch.src_ids, axis=1) + theta.b - - def ComputeLoss(self, theta, predicted, input_batch): - diff = predicted - input_batch.tgt_ids - per_example_loss = diff * diff - batch_dim = py_utils.GetShape(per_example_loss)[0] - - def replicate_var(name): - return tf.convert_to_tensor( - [self._private_vars[name]] * batch_dim, dtype=tf.float32) - - metrics = {'loss': (tf.reduce_sum(per_example_loss), batch_dim)} - per_example_tensors = { - 'input': input_batch.src_ids, - 'loss': per_example_loss, - 'diff': diff, - 'm': replicate_var('m'), - 'b': replicate_var('b'), - } - return metrics, per_example_tensors - - def FilterPerExampleTensors(self, per_example): - return per_example - - def ProcessFPropResults(self, sess, global_step, metrics, - per_example_tensors): - self.global_steps.append(global_step) - self.metrics.append(metrics) - self.result_per_example_tensors.append(per_example_tensors) - - -class IdentityRegressionModel(base_model.SingleTaskModel): - """Simple regression model for testing.""" - - def __init__(self, params): - super().__init__(params) - self.global_steps = [] - self.metrics = [] - self.result_per_example_tensors = [] - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'IdentityRegressionModel' - p.input = CountingInputGenerator.Params() - p.task = IdentityRegressionTask.Params() - return p - - def ProcessFPropResults(self, sess, global_step, metrics, - per_example_tensors): - self.global_steps.append(global_step) - self.metrics.append(metrics) - self.result_per_example_tensors.append(per_example_tensors) diff --git a/lingvo/core/tshape.py b/lingvo/core/tshape.py deleted file mode 100644 index cd34d1d0..00000000 --- a/lingvo/core/tshape.py +++ /dev/null @@ -1,117 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Symbolic representation of tensor shapes.""" - -import lingvo.compat as tf -import sympy - - -class Shape: - """Shape represents a tensor's symbolic shape.""" - - def __init__(self, dims): - """Constructs a shape whose i-th dim is dims[i]. - - Each dim can be one of the following types: - integer: represents the dimension is a known and fixed. - string: represents the dimension is an unknown and a sympy dummy symbol is - used to represent it. Also note that contents of strings only matter for - logging/printing. Even if the same string is given on multiple - dimensions, it doesn't mean that they are the same. - sympy expression: represents a dimension which possibly - depends on dimensions of other shapes. - - Args: - dims: A list of either integer, string or sympy.Symbol. - """ - self._shape = [] - for x in dims: - assert x is not None, str(dims) - if isinstance(x, str): - # NOTE: Dummy(x) creates a unique symbol. I.e., the value of x has no - # meaning except for printing, etc. - self._shape.append(sympy.Dummy(x, integer=True)) - else: - # Converts x to a sympy type. E.g., int to sympy.Integer. - self._shape.append(sympy.sympify(x)) - self._size = sympy.prod(self._shape) - - @property - def rank(self): - """Returns the rank of the tensor.""" - return len(self._shape) - - @property - def size(self): - """Returns the size (num of elements) of the tensor.""" - return self._size - - def num_elements(self): # pylint: disable=invalid-name - """Returns the size (num of elements) of the tensor.""" - return self.size - - def __getitem__(self, key): - """Returns one dimension or a shape from a slice of dimensions.""" - if isinstance(key, int): - return self._shape[key] - elif isinstance(key, slice): - return Shape(self._shape[key]) - else: - raise TypeError("Invalid argument type.") - - def __add__(self, other): - """Concatenates two shapes into one.""" - # pylint: disable=protected-access - if isinstance(other, Shape): - return Shape(self._shape + other._shape) - elif isinstance(other, list): - return Shape(self._shape + Shape(other)._shape) - else: - raise NotImplementedError - - def __radd__(self, other): - """Concatenates two shapes into one.""" - # pylint: disable=protected-access - if isinstance(other, Shape): - return Shape(other._shape + self._shape) - elif isinstance(other, list): - return Shape(Shape(other)._shape + self._shape) - else: - raise NotImplementedError - - def __str__(self): - return str(self._shape) - - def Subs(self, bindings): - """Substitute symbols with new values. - - Args: - bindings: key/value items correspond to old/new pairs for substitution. - - Returns: - The Shape with symbols substituted according to bindings. - """ - return Shape([x.subs(bindings) for x in self._shape]) - - def ToTensorShape(self): - """Converts to a possibly partially specified tf.TensorShape.""" - dims = [] - for d in self._shape: - if d.is_number and d.is_integer: - dims.append(int(d)) - else: - dims.append(None) - return tf.TensorShape(dims) diff --git a/lingvo/core/tshape_test.py b/lingvo/core/tshape_test.py deleted file mode 100644 index 5c2e4020..00000000 --- a/lingvo/core/tshape_test.py +++ /dev/null @@ -1,49 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lingvo.core.tshape.""" - -import lingvo.compat as tf -from lingvo.core import test_utils -from lingvo.core.tshape import Shape - - -class TshapeTest(test_utils.TestCase): - - def testShape(self): - w1 = Shape([3, 3, 'd0', 'd1']) - w2 = Shape([5, 5, w1[2], 'd2']) - self.assertEqual(w2.ToTensorShape().as_list(), [5, 5, None, None]) - s2 = w2.Subs({w1[2]: 8, w1[3]: 16, w2[3]: 32}) - self.assertEqual(s2.ToTensorShape().as_list(), [5, 5, 8, 32]) - - # __getitem__ - inner = w1[-2:] - self.assertIsInstance(inner, Shape) - - # unpack - d0, d1 = w1[-2:] - self.assertEqual((d0 * d1).subs({d0: 3, d1: 5}), 15) - - # __add__ - self.assertEqual(str(w1 + w2), '[3, 3, _d0, _d1, 5, 5, _d0, _d2]') - - # __radd_ - self.assertEqual(str([7] + w1), '[7, 3, 3, _d0, _d1]') - self.assertEqual(str(w1[-2:] + w2[-1:]), '[_d0, _d1, _d2]') - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/wpm_encoder.py b/lingvo/core/wpm_encoder.py deleted file mode 100644 index 131b4b60..00000000 --- a/lingvo/core/wpm_encoder.py +++ /dev/null @@ -1,197 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Encode using wordpiece models. - -Implements the segmentation algorithm described in the last paragraph of -p. 5150, in the following publication: - -M. Schuster and K. Nakajima, "Japanese and Korean voice -search," 2012 IEEE International Conference on Acoustics, -Speech and Signal Processing, 2012 - -https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37842.pdf -""" -import lingvo.compat as tf -from lingvo.core import ops -from lingvo.core import py_utils -import six - -# Must be a large ID. -NO_TOKEN = 1 << 31 - 1 -NO_TOKEN_STRING = '' - -SENTENCE_START_STRING = '' -SENTENCE_END_STRING = '' - -BOW_STR = '▁' - - -class WpmEncoder: - """WPM encoder.""" - - def __init__(self, wpm_filepath, merge_prob=1.): - """Create a WPM encoder. - - Args: - wpm_filepath: a path to the file containing the vocabulary. - merge_prob: the probability of merging tokens while encoding. - """ - # Load vocabulary file. - lines = py_utils.ReadFileLines(wpm_filepath) - - self._pieces = [] - for line in lines: - if isinstance(line, bytes): - line = six.ensure_text(line, 'utf-8') - piece = line.strip().split('\t')[0] - self._pieces.append(piece) - self._merge_prob = merge_prob - - def _TokenToString(self, token): - return ops.vocab_id_to_token(token, vocab=self._pieces) - - def _StringToToken(self, tokstr): - return tf.where( - ops.token_in_vocab(tokstr, vocab=self._pieces), - ops.vocab_token_to_id(tokstr, vocab=self._pieces), - tf.broadcast_to(NO_TOKEN, tf.shape(tokstr))) - - def _MergeTokens(self, tokens): - return self._StringToToken( - self._TokenToString(tokens[0]) + self._TokenToString(tokens[1])) - - def _EncodeToIds(self, word): - # Below: - # * a token is a wordpiece ID. - # * the tokens array will be merged in-place. - # * the candidates array is an array of size len(tokens) - 1. - # It contains the token for the merged wordpiece, if it exists, - # -1 otherwise. For instance, candidate[3] = id(token[3] + token[4]). - # First, split into basic UTF-8 characters (letters). - chars = tf.strings.unicode_split(word, 'UTF-8') - tokens = self._StringToToken(chars) - tokens = tf.where( - tf.equal(tokens, NO_TOKEN), - # Unseen character. - tf.broadcast_to(self.unk_id, tf.shape(tokens)), - tokens) - # Create initial candidate list. - candidates = tf.map_fn( - self._MergeTokens, (tokens[:-1], tokens[1:]), dtype=tokens.dtype) - - def _ShouldMerge(unused_tokens, candidates): - """Merge until not possible, or we abort early according to merge_prob.""" - return tf.math.logical_and( - tf.reduce_any(tf.not_equal(candidates, NO_TOKEN)), - tf.random.uniform([]) < self._merge_prob) - - def _MergeOneToken(tokens, i): - return tf.expand_dims( - self._MergeTokens((tokens[i], tokens[i + 1])), axis=-1) - - def _MergeCandidates(tokens, candidates): - """Merge in the reverse binary tree.""" - best_id = tf.argmin(candidates, output_type=tf.int32) - # Perform the merge at position best_id. - tokens = tf.concat( - [tokens[:best_id], [candidates[best_id]], tokens[best_id + 2:]], - axis=0) - # Recompute the merge candidates. - # Only the neighbors of best_id need to be recomputed. - empty = tf.zeros([0], dtype=candidates.dtype) - - def _MergeLeft(): - return tf.concat( - [candidates[:best_id - 1], - _MergeOneToken(tokens, best_id - 1)], - axis=0) - - left_candidates = tf.cond(tf.equal(best_id, 0), lambda: empty, _MergeLeft) - - def _MergeRight(): - return tf.concat( - [_MergeOneToken(tokens, best_id), candidates[best_id + 2:]], axis=0) - - right_candidates = tf.cond( - tf.greater_equal(best_id, - tf.size(tokens) - 1), lambda: empty, _MergeRight) - - candidates = tf.concat([left_candidates, right_candidates], axis=0) - return tokens, candidates - - return tf.while_loop( - _ShouldMerge, - _MergeCandidates, (tokens, candidates), - parallel_iterations=1, - back_prop=False)[0] - - def Encode(self, text): - """Converts string `text` to integer ids and the encoded string. - - Encoding includes prefixing the beginning-of-word token to each word. - - Returns: - (ids, tokens) where ids is the encoded integer ids and tokens is the - encoded string. - """ - words = tf.sparse.to_dense(tf.strings.split([text]), default_value='')[0] - num_words = tf.size(words) - ids_ta = tf.TensorArray(tf.int32, 0, dynamic_size=True) - - def _WordsToIds(i, words, ids_ta): - encoded_ids = self._EncodeToIds(BOW_STR + words[i]) - ids_ta = ids_ta.scatter( - tf.range(ids_ta.size(), - ids_ta.size() + tf.size(encoded_ids)), encoded_ids) - return i + 1, words, ids_ta - - _, _, ids_ta = tf.while_loop( - lambda i, *_: i < num_words, - _WordsToIds, - loop_vars=(tf.constant(0, tf.int32), words, ids_ta), - parallel_iterations=30, - back_prop=False) - - ids = ids_ta.stack() - return ids, self._TokenToString(ids) - - def Decode(self, ids): - txt = tf.strings.reduce_join(self._TokenToString(ids)) - txt = tf.strings.regex_replace(txt, BOW_STR, ' ') - # Note that this strips spaces from the end of the input as well. - # We assume no inputs rely on the existence of trailing whitespace. - txt = tf.strings.strip(txt) - return txt - - @property - def sentence_start_id(self): - return self._pieces.index(SENTENCE_START_STRING) - - @property - def sentence_start_string(self): - return SENTENCE_START_STRING - - @property - def sentence_end_id(self): - return self._pieces.index(SENTENCE_END_STRING) - - @property - def sentence_end_string(self): - return SENTENCE_END_STRING - - @property - def unk_id(self): - return self._pieces.index(NO_TOKEN_STRING) diff --git a/lingvo/core/wpm_encoder_test.py b/lingvo/core/wpm_encoder_test.py deleted file mode 100644 index 5a57d56b..00000000 --- a/lingvo/core/wpm_encoder_test.py +++ /dev/null @@ -1,104 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for wpm_encoder.""" - -import os -import lingvo.compat as tf -from lingvo.core import test_utils -from lingvo.core import wpm_encoder - - -class WpmEncoderTest(test_utils.TestCase): - - def _CreateVocab(self): - outpath = os.path.join(tf.test.get_temp_dir(), 'wpm.voc') - with tf.io.gfile.GFile(outpath, 'w') as f: - contents = [ - '', - '', - '', - 't', - 'i', - 'o', - 'f', - 'r', - 'D', - 'it', - 'or', - 'for', - 'itt', - 'to', - 'i-', - 'tt', - 'f.', - 'o-', - 'o.', - 'fo', - 'ø', # \xC3\xB8 - 'ö', # \xC3\xB6 - '\\', - '▁', - ] - f.write('\n'.join(contents)) - return outpath - - def setUp(self): - voc = self._CreateVocab() - self._enc = wpm_encoder.WpmEncoder(voc) - - def testDitto(self): - with tf.Session(): - ids, strs = self._enc.Encode('Ditto') - self.assertEqual(u'▁ D itt o'.encode('utf-8'), - tf.strings.reduce_join(strs, separator=' ').eval()) - self.assertEqual(b'Ditto', self._enc.Decode(ids).eval()) - ids, strs = self._enc.Encode('Ditto Ditto') - self.assertEqual(u'▁ D itt o ▁ D itt o'.encode('utf-8'), - tf.strings.reduce_join(strs, separator=' ').eval()) - self.assertEqual(b'Ditto Ditto', self._enc.Decode(ids).eval()) - - def testMergeProb(self): - voc = self._CreateVocab() - enc = wpm_encoder.WpmEncoder(voc, merge_prob=0.) - with tf.Session(): - ids, strs = enc.Encode('Ditto') - self.assertEqual(u'▁ D i t t o'.encode('utf-8'), - tf.strings.reduce_join(strs, separator=' ').eval()) - self.assertEqual(b'Ditto', self._enc.Decode(ids).eval()) - - def testEmpty(self): - with tf.Session(): - ids, strs = self._enc.Encode('') - self.assertEqual(b'', tf.strings.reduce_join(strs, separator=' ').eval()) - self.assertEqual(b'', self._enc.Decode(ids).eval()) - - def testWithBackslash(self): - with tf.Session(): - ids, strs = self._enc.Encode('\\') - self.assertEqual(u'▁ \\'.encode('utf-8'), - tf.strings.reduce_join(strs, separator=' ').eval()) - self.assertEqual(b'\\', self._enc.Decode(ids).eval()) - - def testWithUnicode(self): - with tf.Session(): - ids, strs = self._enc.Encode('føö') - self.assertEqual(u'▁ f ø ö'.encode('utf-8'), - tf.strings.reduce_join(strs, separator=' ').eval()) - self.assertEqual(u'føö'.encode('utf-8'), self._enc.Decode(ids).eval()) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/core/wpm_tokenizer_test.py b/lingvo/core/wpm_tokenizer_test.py deleted file mode 100644 index 69bfc14d..00000000 --- a/lingvo/core/wpm_tokenizer_test.py +++ /dev/null @@ -1,66 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for WPM tokenizer (which is not in ops test).""" - -import lingvo.compat as tf -from lingvo.core import test_helper -from lingvo.core import test_utils -from lingvo.core import tokenizers -import numpy as np - - -class WpmTokenizerTest(test_utils.TestCase): - - def testStringsTokenIds(self): - p = tokenizers.WpmTokenizer.Params() - p.vocab_filepath = test_helper.test_src_dir_path('tasks/mt/wpm-ende.voc') - p.vocab_size = 32000 - wpm_tokenizer = p.Instantiate() - with self.session(use_gpu=False): - token_ids, target_ids, paddings = self.evaluate( - wpm_tokenizer.StringsToIds( - tf.constant(['would that it were so simple', 'this is it', ''], - dtype=tf.string), 6, True)) - self.assertAllEqual( - token_ids, [[1, 926, 601, 560, 1273, 721], [1, 647, 470, 560, 2, 2], - [1, 2, 2, 2, 2, 2]]) - self.assertAllEqual(target_ids, - [[926, 601, 560, 1273, 721, 5490], - [647, 470, 560, 2, 2, 2], [2, 2, 2, 2, 2, 2]]) - self.assertAllEqual(paddings, - [[0., 0., 0., 0., 0., 0.], [0., 0., 0., 0., 1., 1.], - [0., 1., 1., 1., 1., 1.]]) - - def testIdsToStrings(self): - p = tokenizers.WpmTokenizer.Params() - p.vocab_filepath = test_helper.test_src_dir_path('tasks/mt/wpm-ende.voc') - p.vocab_size = 32000 - wpm_tokenizer = p.Instantiate() - with self.session(use_gpu=False): - ref = tf.constant([ - 'would that it were so simple', - 'this is it', - '', - ]) - _, target_ids, paddings = self.evaluate( - wpm_tokenizer.StringsToIds(ref, 100, True)) - lens = np.argmax(paddings > 0.0, axis=1) - 1 - found = self.evaluate(wpm_tokenizer.IdsToStrings(target_ids, lens)) - self.assertAllEqual(ref, found) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/datasets.py b/lingvo/datasets.py deleted file mode 100644 index f551ab33..00000000 --- a/lingvo/datasets.py +++ /dev/null @@ -1,28 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Utilities for dataset information.""" -import inspect - - -def GetDatasets(cls): - """Returns the list of dataset functions (e.g. Train, Dev, ...).""" - datasets = [] - for name, _ in inspect.getmembers( - cls, lambda x: inspect.isfunction(x) or inspect.ismethod(x)): - if name not in ['GetDatasetParams', 'Model', 'Task', 'ProgramSchedule' - ] and not name.startswith('_'): - datasets += [name] - return datasets diff --git a/lingvo/executor.py b/lingvo/executor.py deleted file mode 100644 index 6f7000e3..00000000 --- a/lingvo/executor.py +++ /dev/null @@ -1,347 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""An experimental new unified TPU executor.""" - -import os - -from lingvo import compat as tf -from lingvo.core import base_model -from lingvo.core import checkpointer -from lingvo.core import cluster_factory -from lingvo.core import ml_perf_log as mlp_log -from lingvo.core import multitask_model -from lingvo.core import py_utils -from lingvo.core import task_scheduler - -from lingvo import base_runner -from tensorflow.python.tpu import device_assignment as device_assignment_lib # pylint: disable=g-direct-tensorflow-import - -tf.flags.DEFINE_bool( - 'cluster_placer_in_executor', False, - 'If True, cluster.GetPlacer() is used in Executor. ' + - 'When running on TPU model weights can be distributed ' + - 'across TPU hosts, for outrageously large models this ' + - 'enables sharded checkpointing and reduces host memory ' + - 'requirements, see _LeastLoadedPlacer in cluster.py.') - -tf.flags.DEFINE_bool( - 'disable_meta_optimizer_in_executor', False, - 'Disabling the grappler meta_optimizer improves start-up time.') - -tf.flags.DEFINE_bool( - 'lingvo_executor_skip_saving_upon_stop', False, - 'When we are just running decoding or inference, saving is not necessary,' - 'since no training has occurred.' -) - -FLAGS = tf.flags.FLAGS - - -def GetExecutorParams(model_name, cluster_params, model_registry): - """Get the params needed to instantiate the Executor. - - Args: - model_name: A model name regsitered in the ModelRegistry. - cluster_params: A cluster hyperparams object. - model_registry: A ModelRegistry object. - - Returns: - A tuple (dict, Params): - - - ps_params_dict: High-level task name -> ProgramScheduleParams - - train_cfg: A SingleTaskModelParams or MultiTaskModelParams. - """ - - ps_params_dict = {} - with cluster_factory.Cluster(cluster_params): - ps_cfg = model_registry.GetProgramSchedule(model_name) - train_cfg = model_registry.GetParams(model_name, 'Train') - train_cfg.cluster = cluster_params - - if issubclass(train_cfg.cls, base_model.MultiTaskModel): - multi_task_train_cfg = train_cfg - # Create MultiTaskSubModel params from a MultiTaskModelParams. - for k, _ in multi_task_train_cfg.task_params.IterParams(): - train_task_params = base_model.MultiTaskSubModel.Params() - train_task_params.name = k + '_executor_train_task' - train_task_params.task_name = k - train_task_params.cluster = multi_task_train_cfg.cluster - train_task_params.input = multi_task_train_cfg.input.Get(k).Copy() - train_task_params.train = multi_task_train_cfg.task_params.Get(k).train - if k not in ps_cfg.program_schedule_dict: - tf.logging.fatal( - 'Could not find %s in ps_cfg.program_schedule_dict: %s', k, - ps_cfg) - program_schedule_params = ps_cfg.program_schedule_dict[k] - - program_schedule_params.task_dict = {'Train': train_task_params} - - for eval_dataset_name in program_schedule_params.dataset_names: - multi_task_eval_cfg = model_registry.GetParams( - model_name, eval_dataset_name) - eval_task_params = base_model.MultiTaskSubModel.Params() - eval_task_params.name = ( - k + '_' + eval_dataset_name + '_executor_eval_task') - eval_task_params.task_name = k - eval_task_params.cluster = multi_task_eval_cfg.cluster - eval_task_params.input = multi_task_eval_cfg.input.Get(k).Copy() - program_schedule_params.task_dict[ - eval_dataset_name] = eval_task_params - ps_params_dict[k] = program_schedule_params - else: - program_schedule_params = ps_cfg - # So, what if I just remove the Train dictionary? - program_schedule_params.task_dict = {'Train': train_cfg} - for eval_dataset_name in program_schedule_params.dataset_names: - task_eval_params = model_registry.GetParams(model_name, - eval_dataset_name) - task_eval_params.cluster = train_cfg.cluster - program_schedule_params.task_dict[eval_dataset_name] = task_eval_params - ps_params_dict[''] = program_schedule_params - - return ps_params_dict, train_cfg - - -class ExecutorTpu(base_runner.BaseRunner): - """An experimental runner that does arbitrary multi-program execution on TPU. - - Overview of operation: - - - During construction, all programs construct their sub-graphs, in a sense - creating a mega-graph. - - A sequence of programs is then executed in-whole associated with that task. - eg: [train x 1000 steps, checkpoint, eval 4 steps, decode 2 steps] - - In this manner, programs and higher-level tasks cooperatively time-slice - share the TPU. - """ - - def __init__(self, train_cfg, ps_params_dict, model_task_name, logdir, - tf_master, **kwargs): - """Construct an ExecutorTpu BaseRunner. - - Args: - train_cfg: SingleTaskModelParams or MultiTaskModelParams - ps_params_dict: A dict of top-level task name -> ProgramSchedule params, - if train_cfg is a SingleTaskModelParams, we expect only one entry. - model_task_name: An override for multi-task models, currently unused. - logdir: String path to the log directory to output to. - tf_master: String path to the master job, e.g. 'local'. - **kwargs: keyword args to pass through to BaseRunner. - """ - super().__init__(train_cfg, model_task_name, logdir, tf_master, **kwargs) - - self._cluster_def = self._cluster.worker_cluster_def - - # There is a single Executor task - assert self._cluster.num_replicas == 1 - data_parallelism = self._cluster.num_splits_per_client - - assert data_parallelism - num_devices_per_split = self._cluster.num_devices_per_split - tf.logging.info('data_parallelism: %d, num_devices_per_split: %d', - data_parallelism, num_devices_per_split) - - self.task_scheduler = None - self._checkpoint_dir = os.path.join(logdir, 'train') - - self._variable_renaming_rules = [] - - self._ml_perf = None - - # If this is a multi-task model, grab the params for the TaskScheduler. - if issubclass(train_cfg.cls, base_model.SingleTaskModel): - tf.logging.info('single_task_model') - assert len(ps_params_dict) == 1 - self._model_task_name = list(ps_params_dict.keys())[0] - self._single_task_mode = True - elif issubclass(train_cfg.cls, base_model.MultiTaskModel): - tf.logging.info('multi_task_model') - - if issubclass(train_cfg.cls, multitask_model.RegExSharedVariableModel): - self._variable_renaming_rules = train_cfg.variable_renaming_rules - - if train_cfg.task_schedule is None: - task_schedule_params = task_scheduler.ConstantScheduler.Params() - task_schedule_params.task_probs = sorted( - list(train_cfg.task_probs.IterParams())) - else: - task_schedule_params = train_cfg.task_schedule - self.task_scheduler = task_schedule_params.Instantiate() - self._single_task_mode = False - else: - tf.logging.fatal( - 'Model %s is not a sub-class of SingleTaskModel or MultiTaskModel', - train_cfg.cls) - - tf.logging.info('train_cfg.cls: %s', train_cfg.cls) - - self._WriteToLog(train_cfg.ToText(), self._checkpoint_dir, - 'trainer_params.txt') - if self._ml_perf is not None: - self._ml_perf_log = True - mlp_log.mlperf_print(key='benchmark', value=self._ml_perf.benchmark_name) - else: - self._ml_perf_log = False - - # BaseRunner legacy - self.enqueue_ops = None - - @py_utils.RetryOnTransientTfError() - def _WaitTillInit(): - """Wait until the model is ready.""" - try: - with self._graph.as_default(), self._GetSession( - cluster_def=self._cluster_def, - disable_meta_optimizer=FLAGS.disable_meta_optimizer_in_executor - ) as sess: - topology = sess.run( - tf.tpu.initialize_system(embedding_config=None, job=None)) - device_assignment = device_assignment_lib.device_assignment( - topology, - computation_shape=py_utils.ComputationShape( - num_devices_per_split, topology), - num_replicas=data_parallelism) - py_utils.SetTpuDeviceAssignment(device_assignment) - tf.logging.info('device_assignment.core_assignment: %s', - str(device_assignment.core_assignment)) - tf.logging.info( - 'device_assignment.topology.device_coordinates: %s', - str(device_assignment.topology.device_coordinates)) - except py_utils.transient_tf_errors as e: - tf.logging.info('TPU initialization failed: %s', e) - raise - - if self._ml_perf_log: - mlp_log.mlperf_print(key='init_start', value=None) - _WaitTillInit() - - train_cfg = self.params - shared_model = self._MaybeConstructSharedModel(train_cfg) - - self._program_schedule_dict = {} - self._programs = [] - - for task_string, program_schedule_params in ps_params_dict.items(): - program_schedule_params.logdir = logdir - program_schedule_params.num_splits_per_client = data_parallelism - program_schedule_params.task_name = task_string - # If the model was created above, we'll inject it here as a shared_model. - ps = program_schedule_params.Instantiate(shared_model=shared_model) - self._program_schedule_dict[task_string] = ps - tf.logging.info('program_schedule_params: %s', - program_schedule_params.ToText()) - self._programs += ps.Programs() - if program_schedule_params.ml_perf.benchmark_name is not None: - self._ml_perf = program_schedule_params.ml_perf - - tf.logging.info('num_programs: %d', len(self._programs)) - - with self._graph.as_default(), tf.container(self._container_id): - with self._cluster, tf.device( - self._cluster.job_spec.name if not FLAGS.cluster_placer_in_executor - else self._cluster.GetPlacer()): - with py_utils.VariableRenameScope(self._variable_renaming_rules): - _ = py_utils.GetOrCreateGlobalStepVar() - for program in self._programs: - program.BuildTpuSubgraph() - py_utils.ClearTpuSummaryTensors() - for program in self._programs: - program.SetStatusMessageFn(self._SetStatusMessage) - program.CreateCheckpointer() - self._initialize_tables = tf.tables_initializer() - self._initialize_local_vars = tf.local_variables_initializer() - - self.save_only_checkpointer = checkpointer.Checkpointer( - self._checkpoint_dir, - model=None, - train_params=train_cfg.train, - save_only=True) - - def _MaybeConstructSharedModel(self, train_cfg): - """Construct a single shared copy of the model if this is a MultiTaskModel. - - For MultiTaskModels, we create a MultiTaskSubModel for each task, but - construct the model only once. - - Args: - train_cfg: The params for a SingleTaskModel or MultiTaskModel. - - Returns: - A MultiTaskModel, if train_cfg is a MultiTaskModel params object. - """ - if not issubclass(train_cfg.cls, base_model.MultiTaskModel): - return None - - with self._graph.as_default(), tf.container(self._container_id): - with self._cluster, tf.device( - self._cluster.job_spec.name if not FLAGS.cluster_placer_in_executor - else self._cluster.GetPlacer()): - with py_utils.VariableRenameScope(self._variable_renaming_rules): - _ = py_utils.GetOrCreateGlobalStepVar() - shared_model = train_cfg.Instantiate() - shared_model.InstantiateVariables() - - return shared_model - - def Start(self): - # Run training. - self._RunLoop('executor_tpu', self._Loop) - - def _Loop(self): - with tf.container(self._container_id), self._GetSession( - cluster_def=self._cluster_def, - disable_meta_optimizer=FLAGS.disable_meta_optimizer_in_executor - ) as sess: - # Initialize the variables first, if needed. - for program in self._programs: - program.RestoreIfNeeded(sess) - program.Compile(sess) - sess.run(self._initialize_tables) - sess.run(self._initialize_local_vars) - - # How to stop? - while True: - global_step = sess.run(py_utils.GetGlobalStep()) - if self._ShouldStop(sess, global_step): - tf.logging.info('Training finished.') - if not self._ml_perf_log and not FLAGS.lingvo_executor_skip_saving_upon_stop: - self.save_only_checkpointer.Save(sess, global_step) - return - - # If a task is explicitly selected, only run the programs associated - # with that task. - if self._single_task_mode or self._model_task_name: - tf.logging.info('Single task mode: %s', self._model_task_name) - program_schedule = self._program_schedule_dict[self._model_task_name] - else: - # Otherwise, sample a task. - model_task = self.task_scheduler.Sample(global_step) - tf.logging.info('Sampled %s', model_task) - program_schedule = self._program_schedule_dict[model_task] - - done = program_schedule.Run(sess) - if done: - tf.logging.info('Program schedule told us to stop.') - return - - # global_step local variable above is a result of sess.run, not a - # tf variable, so when we do save_only_checkpointer.Save(...) here - # py_utils.GetGlobalStep() is ahead of it by - # (train_executions_per_eval * train_steps_per_loop) - # steps ahead already, due to program_schedule.Run(sess). - # - if not self._ml_perf_log and not FLAGS.lingvo_executor_skip_saving_upon_stop: - self.save_only_checkpointer.MaybeSave(sess, py_utils.GetGlobalStep()) diff --git a/lingvo/ipython_kernel.py b/lingvo/ipython_kernel.py deleted file mode 100644 index 28e72913..00000000 --- a/lingvo/ipython_kernel.py +++ /dev/null @@ -1,44 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Starts an IPython kernel with Lingvo deps. - -jupyter_http_over_ws must be installed and activated. See -https://research.google.com/colaboratory/local-runtimes.html for more details. - -To use: - bazel run -c opt //lingvo:ipython_kernel -""" - -from absl import app -from IPython.html.notebookapp import NotebookApp - - -def main(_): - notebookapp = NotebookApp.instance() - notebookapp.open_browser = False - notebookapp.ip = "0.0.0.0" - notebookapp.port = 8888 - notebookapp.port_retries = 0 - notebookapp.allow_origin_pat = "https://colab\\.[^.]+\\.google.com" - notebookapp.allow_root = True - notebookapp.token = "" - notebookapp.disable_check_xsrf = True - notebookapp.initialize() - notebookapp.start() - - -if __name__ == "__main__": - app.run(main) diff --git a/lingvo/lingvo.bzl b/lingvo/lingvo.bzl deleted file mode 100644 index 7c9d0ee1..00000000 --- a/lingvo/lingvo.bzl +++ /dev/null @@ -1,173 +0,0 @@ -"""Implements custom rules for Lingvo.""" - -load("@subpar//:subpar.bzl", "par_binary") - -def tf_copts(): - # TODO(drpng): autoconf this. - return ["-D_GLIBCXX_USE_CXX11_ABI=0", "-Wno-sign-compare", "-mavx"] + select({ - "//lingvo:cuda": ["-DGOOGLE_CUDA=1"], - "//conditions:default": [], - }) - -def lingvo_cc_library(name, srcs = [], hdrs = [], deps = [], testonly = 0): - native.cc_library( - name = name, - copts = tf_copts(), - srcs = srcs, - hdrs = hdrs, - deps = [ - "@tensorflow_includes//:includes", - ] + deps, - testonly = testonly, - ) - -def lingvo_cc_test_library(name, srcs = [], hdrs = [], deps = []): - lingvo_cc_library( - name = name, - srcs = srcs, - hdrs = hdrs, - deps = deps + ["@com_google_googletest//:gtest"], - testonly = 1, - ) - -def lingvo_cc_binary(name, srcs = [], deps = []): - native.cc_binary( - name = name, - copts = tf_copts(), - srcs = srcs, - deps = [ - "@tensorflow_includes//:includes", - "@tensorflow_solib//:framework_lib", - ] + deps, - ) - -def lingvo_cc_test(name, srcs, deps = [], **kwargs): - native.cc_test( - name = name, - copts = tf_copts(), - srcs = srcs, - deps = [ - "@tensorflow_includes//:includes", - "@tensorflow_solib//:framework_lib", - "@com_google_googletest//:gtest_main", - ] + deps, - **kwargs - ) - -def lingvo_py_binary(*args, **kwargs): - # Extract all files to a temporary directory is required to allow access to - # the shared object containing lingvo ops. - kwargs["zip_safe"] = False - par_binary(*args, **kwargs) - -def custom_kernel_library(name, op_def_lib, srcs, hdrs = [], deps = []): - native.cc_library( - name = name, - srcs = srcs, - hdrs = hdrs, - copts = tf_copts(), - deps = [ - "@tensorflow_includes//:includes", - ] + deps + op_def_lib, - alwayslink = 1, - ) - -def gen_op_cclib(name, srcs, deps = []): - # TODO(drpng): gpu. - native.cc_library( - name = name, - srcs = srcs, - deps = [ - "@tensorflow_includes//:includes", - "@tensorflow_solib//:framework_lib", - ] + deps, - alwayslink = 1, - copts = tf_copts(), - ) - -def gen_op_pylib(name, cc_lib_name, srcs, kernel_deps, py_deps = [], **kwargs): - native.cc_binary( - name = cc_lib_name + ".so", - deps = [cc_lib_name] + kernel_deps, - linkshared = 1, - copts = tf_copts(), - **kwargs - ) - - native.py_library( - name = name, - srcs = srcs, - srcs_version = "PY3", - data = [cc_lib_name + ".so"], - deps = py_deps, - **kwargs - ) - -def lingvo_cuda_py_test(name, tags = [], deps = [], **kwargs): - native.py_test( - name = name, - tags = tags + ["requires-gpu"], - deps = deps, - **kwargs - ) - -def _proto_gen_cc_src(name, basename): - native.genrule( - name = name, - srcs = [basename + ".proto"], - outs = [basename + ".pb.cc", basename + ".pb.h"], - tools = [ - "@protobuf_protoc//:protoc_bin", - "//lingvo:tf_dot_protos", - ], - # TODO(drpng): only unpack if tf_proto dependency is requested. - cmd = """ - mkdir -p $(@D)/tf_proto.$$$$; - tar -C $(@D)/tf_proto.$$$$ -xf $(location //lingvo:tf_dot_protos); - $(location @protobuf_protoc//:protoc_bin) --proto_path=$(@D)/tf_proto.$$$$ --proto_path=. --cpp_out=$(GENDIR) $(<); - rm -rf $(@D)/tf_proto.$$$$ - """, - ) - -def _proto_gen_py_src(name, basename): - native.genrule( - name = name, - srcs = [basename + ".proto"], - outs = [basename + "_pb2.py"], - tools = [ - "@protobuf_protoc//:protoc_bin", - "//lingvo:tf_dot_protos", - ], - # TODO(drpng): only unpack if tf_proto dependency is requested. - cmd = """ - mkdir -p $(@D)/tf_proto.$$$$; - tar -C $(@D)/tf_proto.$$$$ -xf $(location //lingvo:tf_dot_protos); - $(location @protobuf_protoc//:protoc_bin) --proto_path=$(@D)/tf_proto.$$$$ --proto_path=. --python_out=$(GENDIR) $(<); - rm -rf $(@D)/tf_proto.$$$$ - """, - ) - -def lingvo_proto_cc(name, src, deps = []): - # TODO(drpng): only works with proto with no deps within lingvo. - _unused = [deps] - basename = src.replace(".proto", "") - _proto_gen_cc_src(name + "_gencc", basename) - lingvo_cc_library( - name = name, - srcs = [basename + ".pb.cc"], - hdrs = [basename + ".pb.h"], - ) - lingvo_cc_library( - name = "%s_cc" % name, - deps = [":%s" % name], - ) - -def lingvo_proto_py(name, src, deps = []): - # TODO(drpng): only works with proto with no deps within lingvo. - _unused = [deps] - basename = src.replace(".proto", "") - _proto_gen_py_src(name + "_genpy", basename) - native.py_library( - name = name, - srcs = [basename + "_pb2.py"], - ) diff --git a/lingvo/model_import_test.py b/lingvo/model_import_test.py deleted file mode 100644 index f9948002..00000000 --- a/lingvo/model_import_test.py +++ /dev/null @@ -1,23 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Test model imports.""" - -from lingvo import model_imports -import lingvo.compat as tf - -if __name__ == '__main__': - model_imports.ImportAllParams() - tf.test.main() diff --git a/lingvo/model_imports.py b/lingvo/model_imports.py deleted file mode 100644 index 58ad0439..00000000 --- a/lingvo/model_imports.py +++ /dev/null @@ -1,104 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Global import for model hyper-parameters. - -Using this module any ModelParams can be accessed via GetParams. -""" - -import importlib -import re -import sys - - -def _Import(name): - """Imports the python module of the given name.""" - print('model_imports.py: Importing %s' % name, file=sys.stderr) - try: - importlib.import_module(name) - return True - except ModuleNotFoundError as e: - missing_module = re.match("No module named '(.*?)'", e.msg).group(1) - if not name.startswith(missing_module): - raise - return False - - -def _InsertParams(module): - """Try inserting 'params' everywhere in the module.""" - left = [] - right = module.split('.') - while right: - left.append(right.pop(0)) - yield '.'.join(left + ['params'] + right) - - -_TASK_ROOT = 'lingvo.tasks' - - -# LINT.IfChange(task_dirs) -_TASK_DIRS = ( - 'asr', - 'car', - 'image', - 'lm', - 'mt', - 'punctuator', -) -# LINT.ThenChange(tasks/BUILD:task_dirs) - - -def ImportAllParams(task_root=_TASK_ROOT, - task_dirs=_TASK_DIRS, - require_success=False): - """Import all ModelParams to add to the global registry.""" - success = False - for task in task_dirs: - # By our code repository convention, there is a params.py under the task's - # params directory. params.py imports _all_ modules that may registers a - # model param. - success = _Import('{}.{}.params.params'.format(task_root, task)) or success - if require_success and not success: - raise LookupError('Could not import any task params. Make sure task params ' - 'are linked into the binary.') - return success - - -def ImportParams(model_name, - task_root=_TASK_ROOT, - require_success=True): - """Attempts to only import the files that may contain the model.""" - # 'model_name' follows .. - if '.' not in model_name: - raise ValueError('Invalid model name %s' % model_name) - model_module = model_name.rpartition('.')[0] - # Try importing the module directly, in case it's a local import. - success = _Import(model_module) - # Try all locations of inserting params. - for module_with_params in _InsertParams(model_module): - success = _Import(module_with_params) or success - - # Try built-in tasks imports. - task_model_module = f'{task_root}.{model_module}' - success = _Import(task_model_module) or success - # Try all locations of inserting params. - for module_with_params in _InsertParams(task_model_module): - success = _Import(module_with_params) or success - - if require_success and not success: - raise LookupError( - f'Could not find any valid import paths for module {model_module}. ' - f'Make sure the relevant params files are linked into the binary.') - return success diff --git a/lingvo/model_registry.py b/lingvo/model_registry.py deleted file mode 100644 index dd06bb10..00000000 --- a/lingvo/model_registry.py +++ /dev/null @@ -1,272 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Convenience helpers for managing Params for datasets and models. - -Typical usage will be to define and register a subclass of ModelParams -for each dataset. -""" - -import inspect -from lingvo import model_imports -import lingvo.compat as tf -from lingvo.core import base_model_params - -tf.flags.DEFINE_string( - 'model_params_override', '', 'Optional text specifying' - ' model_class.Model() params to be overridden in' - ' Params.FromText format. Each param must be separated by' - ' a newline or a semicolon. This is only intended for' - ' quick experimentation. Only one of' - ' --model_params_override and --model_params_file_override' - ' may be specified.') -tf.flags.DEFINE_string( - 'model_params_file_override', '', 'Optional text file for' - ' overwriting model_class.Model() params in Params.FromText' - ' format. Each param must occur on a single line. Only one' - ' of --model_params_override and' - ' --model_params_file_override may be specified.') - -FLAGS = tf.flags.FLAGS - - -class _ModelRegistryHelper: - """Helper class.""" - - _MODEL_PARAMS_ALLOW_REDEF = False - - # Global dictionary mapping subclass name to registered ModelParam subclass. - _MODEL_PARAMS = {} - # Global set of modules from which ModelParam subclasses have been registered. - _REGISTERED_MODULES = set() - - @classmethod - def _ClassPathPrefix(cls): - return 'lingvo.tasks.' - - @classmethod - def _ModelParamsClassKey(cls, src_cls): - """Returns a string key used for `src_cls` in the model registry. - - The returned key is a period separated string. E.g., image.mnist.LeNet5. It - roughly reflects how params files are organized. We put some of the - directory information into the key to avoid future model name conflicts. - - Args: - src_cls: A subclass of `~.base_model.BaseModel`. - """ - path = src_cls.__module__ - # Removes the prefix. - path_prefix = cls._ClassPathPrefix() - path = path.replace(path_prefix, '') - - # Removes 'params.' if exists. - if 'params.' in path: - path = path.replace('params.', '') - if inspect.getfile(src_cls).endswith('test.py'): - return 'test.{}'.format(src_cls.__name__) - return '{}.{}'.format(path, src_cls.__name__) - - @classmethod - def _GetSourceInfo(cls, src_cls): - """Gets a source info string given a source class.""" - return '%s@%s:%d' % (cls._ModelParamsClassKey(src_cls), - inspect.getsourcefile(src_cls), - inspect.getsourcelines(src_cls)[-1]) - - @classmethod - def _RegisterModel(cls, wrapper_cls, src_cls): - """Registers a ModelParams subclass in the global registry.""" - key = cls._ModelParamsClassKey(src_cls) - module = src_cls.__module__ - if not cls._MODEL_PARAMS_ALLOW_REDEF and key in cls._MODEL_PARAMS: - raise ValueError('Duplicate model registered for key {}: {}.{}'.format( - key, module, src_cls.__name__)) - - tf.logging.debug('Registering model %s', key) - # Log less frequently (once per module) but at a higher verbosity level. - if module not in cls._REGISTERED_MODULES: - tf.logging.info('Registering models from module: %s', module) - cls._REGISTERED_MODULES.add(module) - - # Decorate param methods to add source info metadata. - cls._MODEL_PARAMS[key] = wrapper_cls - return key - - @classmethod - def _CreateWrapperClass(cls, src_cls): - """Creates a wrapper class for model params that adds source info.""" - - # Python2 class decorators that return a different class are fundamentally - # broken (technically, they are fine but canonical use of super() is - # broken). Also, fallback mechanisms don't exist in python3. So, we only - # decorate the version of the class that we register, but any decorators - # should return the original class for maximum compatibility. - # When the python3 super() is used, it should be possible to return this - # from the decorators too. - - registered_source_info = cls._GetSourceInfo(src_cls) - - class Registered(src_cls): - """Registered model wrapper.""" - - @property - def _registered_source_info(self): - return registered_source_info - - # Extend model to annotate source information. - def Model(self): - """Wraps BaseTask params into SingleTaskModel params.""" - p = super().Model() - p.model = self._registered_source_info - return p - - # So things show up in messages well. - Registered.__name__ = src_cls.__name__ - return Registered - - @classmethod - def MaybeUpdateParamsFromFlags(cls, cfg): - """Updates Model() Params from flags if set.""" - if FLAGS.model_params_override and FLAGS.model_params_file_override: - raise ValueError('Only one of --model_params_override and' - ' --model_params_file_override may be specified.') - - if FLAGS.model_params_override: - params_override = FLAGS.model_params_override.replace(';', '\n') - tf.logging.info('Applying params overrides:\n%s\nTo:\n%s', - params_override, cfg.ToText()) - cfg.FromText(params_override) - if (FLAGS.model_params_file_override and - tf.io.gfile.exists(FLAGS.model_params_file_override)): - params_override = tf.io.gfile.GFile(FLAGS.model_params_file_override, - 'r').read() - tf.logging.info('Applying params overrides from file %s:\n%s\nTo:\n%s', - FLAGS.model_params_file_override, params_override, - cfg.ToText()) - # Okay, there we go. This is where parameters are applied - cfg.FromText(params_override) - - @classmethod - def RegisterSingleTaskModel(cls, src_cls): - """Class decorator that registers a `.SingleTaskModelParams` subclass.""" - if not issubclass(src_cls, base_model_params.SingleTaskModelParams): - raise TypeError('src_cls %s is not a SingleTaskModelParams!' % - src_cls.__name__) - cls._RegisterModel(cls._CreateWrapperClass(src_cls), src_cls) - return src_cls - - @classmethod - def RegisterMultiTaskModel(cls, src_cls): - """Class decorator that registers a `.MultiTaskModelParams` subclass.""" - if not issubclass(src_cls, base_model_params.MultiTaskModelParams): - raise TypeError('src_cls %s is not a MultiTaskModelParams!' % - src_cls.__name__) - cls._RegisterModel(cls._CreateWrapperClass(src_cls), src_cls) - return src_cls - - @staticmethod - def GetAllRegisteredClasses(): - """Returns global registry map from model names to their param classes.""" - all_params = _ModelRegistryHelper._MODEL_PARAMS - if not all_params: - tf.logging.warning('No classes registered.') - return all_params - - @classmethod - def GetClass(cls, class_key): - """Returns a ModelParams subclass with the given `class_key`. - - Args: - class_key: string key of the ModelParams subclass to return. - - Returns: - A subclass of `~.base_model_params._BaseModelParams`. - - Raises: - LookupError: If no class with the given key has been registered. - """ - all_params = cls.GetAllRegisteredClasses() - if class_key not in all_params: - for k in sorted(all_params): - tf.logging.info('Known model: %s', k) - raise LookupError('Model %s not found from list of above known models.' % - class_key) - return all_params[class_key] - - @classmethod - def GetParams(cls, class_key, dataset_name): - """Constructs a `Params` object for given model and dataset, obeying flags. - - In case of default model, params may be updated based on the flags - `--model_params_override` or `--model_params_file_override`. - - Args: - class_key: String class key (i.e. `image.mnist.LeNet5`). - dataset_name: Method to generate dataset params (i.e. 'Test'). - - Returns: - Full `~.hyperparams.Params` for the model class. - """ - model_params_cls = cls.GetClass(class_key) - model_params = model_params_cls() - cfg = model_params.Model() - cfg.input = model_params.GetDatasetParams(dataset_name) - - cls.MaybeUpdateParamsFromFlags(cfg) - return cfg - - @classmethod - def GetProgramSchedule(cls, class_key): - """Retrieve the ProgramSchedule and a dict of task params. - - Args: - class_key: String class key (i.e. `image.mnist.LeNet5`). - - Returns: - ProgramSchedule.Params() - """ - model_params_cls = cls.GetClass(class_key) - model_params = model_params_cls() - program_schedule_cfg = model_params.ProgramSchedule() - return program_schedule_cfg - - -# pyformat: disable -# pylint: disable=invalid-name -RegisterSingleTaskModel = _ModelRegistryHelper.RegisterSingleTaskModel -RegisterMultiTaskModel = _ModelRegistryHelper.RegisterMultiTaskModel -# pylint: enable=invalid-name -# pyformat: enable - - -def GetAllRegisteredClasses(): - model_imports.ImportAllParams() - return _ModelRegistryHelper.GetAllRegisteredClasses() - - -def GetClass(class_key): - model_imports.ImportParams(class_key) - return _ModelRegistryHelper.GetClass(class_key) - - -def GetParams(class_key, dataset_name): - model_imports.ImportParams(class_key) - return _ModelRegistryHelper.GetParams(class_key, dataset_name) - - -def GetProgramSchedule(class_key): - model_imports.ImportParams(class_key) - return _ModelRegistryHelper.GetProgramSchedule(class_key) diff --git a/lingvo/model_registry_test.py b/lingvo/model_registry_test.py deleted file mode 100644 index 28be6630..00000000 --- a/lingvo/model_registry_test.py +++ /dev/null @@ -1,149 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for model_registry.""" - -from lingvo import model_registry -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import base_model -from lingvo.core import base_model_params -from lingvo.core import test_utils - -FLAGS = tf.flags.FLAGS - - -@model_registry.RegisterSingleTaskModel -class DummyModel(base_model_params.SingleTaskModelParams): - - def Train(self): - p = base_input_generator.BaseInputGenerator.Params() - p.name = 'Train' - return p - - def Dev(self): - p = base_input_generator.BaseInputGenerator.Params() - p.name = 'Dev' - return p - - def Test(self): - p = base_input_generator.BaseInputGenerator.Params() - p.name = 'Test' - return p - - def Task(self): - p = base_model.BaseTask.Params() - p.name = 'DummyModel' - return p - - -class ModelRegistryTest(test_utils.TestCase): - - def setUp(self): - FLAGS.model_params_override = '' - - def testGetClass(self): - - mp_cls = model_registry.GetClass('test.DummyModel') - mp = mp_cls() - self.assertEqual('Train', mp.Train().name) - self.assertEqual('Dev', mp.Dev().name) - self.assertEqual('Test', mp.Test().name) - self.assertIsNotNone(mp.Task()) - self.assertIsNotNone(mp.Model()) - - with self.assertRaises(LookupError): - # Not yet registered. - model_registry.GetClass('something.does.not.exist') - - def testGetParams(self): - cfg = model_registry.GetParams('test.DummyModel', 'Test') - self.assertIsNotNone(cfg) - self.assertEqual(DummyModel().Test(), cfg.input) - cfg.input = None - # Registered version adds model source info but direct does not. - cfg.model = None - self.assertEqual(DummyModel().Model(), cfg) - - with self.assertRaises(LookupError): - # Not yet registered. - cfg = model_registry.GetParams('something.does.not.exist', 'Test') - - with self.assertRaises(base_model_params.DatasetError): - cfg = model_registry.GetParams('test.DummyModel', 'UnknownDataset') - - def testGetParamsCanOverrideWithFlags(self): - cfg = model_registry.GetParams('test.DummyModel', 'Train') - - FLAGS.model_params_override = ( - 'train.max_steps: 10; train.ema_decay: 0.9\n' - 'train.init_from_checkpoint_rules : {"ckpt": (["abc", "def"], [])}\n') - cfg2 = model_registry.GetParams('test.DummyModel', 'Train') - - self.assertNotEqual(cfg.train.max_steps, 10) - self.assertEqual(cfg2.train.max_steps, 10) - self.assertNotEqual(cfg.train.ema_decay, 0.9) - self.assertEqual(cfg2.train.ema_decay, 0.9) - self.assertNotEqual(cfg.train.init_from_checkpoint_rules, - {'ckpt': (['abc', 'def'], [])}) - self.assertEqual(cfg2.train.init_from_checkpoint_rules, - {'ckpt': (['abc', 'def'], [])}) - - def testGetParamsCanOverrideWithFlagsRaises(self): - FLAGS.model_params_override = 'task.SOME_UNKNOWN_PARAM : 10' - with self.assertRaises(AttributeError): - _ = model_registry.GetParams('test.DummyModel', 'Train') - - def testGetParamsCanOverrideWithFlagsBadSyntax(self): - FLAGS.model_params_override = 'task.SOME_UNKNOWN_PARAM=10' - with self.assertRaises(ValueError): - _ = model_registry.GetParams('test.DummyModel', 'Train') - - def testGetParamsCanOverrideInputParamsWithFlags(self): - cfg = model_registry.GetParams('test.DummyModel', 'Train') - FLAGS.model_params_override = 'input.num_samples: 100' - - cfg2 = model_registry.GetParams('test.DummyModel', 'Train') - self.assertNotEqual(cfg.input.num_samples, 100) - self.assertEqual(cfg2.input.num_samples, 100) - - def testModelParamsIncludeSourceInfo(self): - path = 'lingvo/model_registry_test.py' - # NOTE: Only the registered version has source info. - self.assertIn(path, - model_registry.GetParams('test.DummyModel', 'Test').model) - - def testDoubleRegister(self): - - def CreateDuplicate(): - # pylint: disable=unused-variable - # pylint: disable=function-redefined - @model_registry.RegisterSingleTaskModel - class DummyDupl(DummyModel): - pass - - @model_registry.RegisterSingleTaskModel - class DummyDupl(DummyModel): - pass - - # pylint: enable=unused-variable - # pylint: enable=function-redefined - - with self.assertRaises(ValueError): - CreateDuplicate() - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/models_test.py b/lingvo/models_test.py deleted file mode 100644 index ef309985..00000000 --- a/lingvo/models_test.py +++ /dev/null @@ -1,39 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for models.""" - -from lingvo import model_imports # pylint: disable=unused-import -from lingvo import model_registry -# Import DummyModel -from lingvo import model_registry_test # pylint: disable=unused-import -from lingvo import models_test_helper -import lingvo.compat as tf -from lingvo.core import base_model - - -class ModelsTest(models_test_helper.BaseModelsTest): - - def testGetModelParams(self): - name = 'test.DummyModel' - p = model_registry.GetParams(name, 'Train') - self.assertTrue(issubclass(p.cls, base_model.SingleTaskModel)) - - -ModelsTest.CreateTestMethodsForAllRegisteredModels(model_registry) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/models_test_helper.py b/lingvo/models_test_helper.py deleted file mode 100644 index 5db6e708..00000000 --- a/lingvo/models_test_helper.py +++ /dev/null @@ -1,181 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Helper for models_test.""" - -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import base_model -from lingvo.core import bn_layers -from lingvo.core import py_utils -from lingvo.core import test_utils - - -def _StubOutCreateVariable(variable_cache): - """Stub out py_utils.CreateVariable to not spend time creating variables. - - Args: - variable_cache: a dict from unique shapes to a dummy tensor of that shape. - """ - - def _CreateVariableStub(name, - params, - reuse=None, - trainable=True, - collections=None, - default_seed=None, - synchronization=None, - aggregation=None): - """Return a zero tensor of the right shape instead of creating variable.""" - del reuse - del default_seed - del synchronization - del aggregation - dtype = params.dtype - shape = py_utils.ToStaticShape(params.shape) - # For total samples counters we have to actually create variables so that - # we can access the 'value' attribute during construction. - if 'total_samples' in name: - var = tf.get_variable( - name, - shape, - dtype, - tf.constant_initializer(0), - collections=collections, - trainable=trainable, - validate_shape=True) - else: - key = (tf.get_default_graph(), tuple(shape)) - if key in variable_cache: - var = variable_cache[key] - else: - var = tf.zeros(shape, dtype) - variable_cache[key] = var - return var - - py_utils.CreateVariable = _CreateVariableStub - - -def TraverseLayer(layer, fn): - """Traverses the layer tree and invokes fn(node) on each node. - - Args: - layer: a BaseLayer. - fn: a function of (layer, layer_theta) -> None. - """ - if isinstance(layer, (list, tuple)): - for layer_i in layer: - TraverseLayer(layer_i, fn) - return - - with tf.name_scope(layer.params.name): - fn(layer) - # Traverse all children in alphabetical order. - for _, child in sorted(layer.children.items()): - TraverseLayer(child, fn) - - -class BaseModelsTest(test_utils.TestCase): - """Base model test class which does not define any test methods of its own.""" - - def setUp(self): - super().setUp() - self._variable_cache = {} - _StubOutCreateVariable(self._variable_cache) - - def _ValidateEMA(self, name, mdl): - if not mdl.ema: - return - self.assertIsInstance(mdl, base_model.SingleTaskModel) - for task in mdl.tasks: - tp = task.params.train - # If the model has explicitly specified ema_decay_moving_vars to - # True or False, then we assume they understand the implication - # of that choice. - if tp.ema_decay_moving_vars is not None: - # ema_decay_moving_vars is set explicitly. - continue - # Otherwise the model should not contain any BatchNormLayer. - # - # If a model fails this test, the user should explicitly specify - # ema_decay_moving_vars, or ensure no BatchNormLayers are in their model. - all_layers = [] - TraverseLayer(task, all_layers.append) - batch_norm_layers = [ - layer.path - for layer in all_layers - if isinstance(layer, bn_layers.BatchNormLayer) - ] - self.assertEqual([], batch_norm_layers) - - def _testOneModelParams(self, registry, name): - with tf.Graph().as_default(): - p = registry.GetParams(name, 'Train') - self.assertTrue(issubclass(p.cls, base_model.BaseModel)) - self.assertIsNot(p.model, None) - p.cluster.mode = 'sync' - p.cluster.job = 'decoder' - p.cluster.decoder.replicas = 1 - with p.cluster.Instantiate(): - # Instantiate the params class, to help catch errors in layer - # constructors due to misconfigurations. - mdl = p.Instantiate() - self._ValidateEMA(name, mdl) - p = mdl.params - - for dataset in ('Train', 'Dev', 'Test'): - input_p = registry.GetParams(name, dataset).input - if issubclass(p.cls, base_model.SingleTaskModel): - self.assertTrue( - issubclass(input_p.cls, base_input_generator.BaseInputGenerator), - 'Error in %s' % dataset) - if (dataset != 'Train') and issubclass( - input_p.cls, - base_input_generator.BaseSequenceInputGenerator) and ( - input_p.num_samples != 0): - self.assertEqual( - input_p.num_batcher_threads, 1, - 'num_batcher_threads too large in %s. Decoder ' - 'or eval runs over this set might not span ' - 'exactly one epoch.' % dataset) - else: - self.assertTrue(issubclass(p.cls, base_model.MultiTaskModel)) - for _, v in input_p.IterParams(): - self.assertTrue( - issubclass(v.cls, base_input_generator.BaseInputGenerator), - 'Error in %s' % dataset) - - @classmethod - def CreateTestMethodsForAllRegisteredModels(cls, - registry, - task_prefix_filter='', - exclude_prefixes=None): - """Programmatically defines test methods for each registered model.""" - model_names = list(registry.GetAllRegisteredClasses().keys()) - for model_name in sorted(model_names): - if task_prefix_filter and not model_name.startswith(task_prefix_filter): - tf.logging.info('Skipping tests for registered model: %s', - model_name) - continue - if exclude_prefixes and any( - [model_name.startswith(prefix) for prefix in exclude_prefixes]): - tf.logging.info( - 'Explicitly excluding tests for registered model: %s', model_name) - continue - - def _Test(self, name=model_name): - self._testOneModelParams(registry, name) # pylint: disable=protected-access - - setattr(cls, 'testModelParams_%s' % model_name, _Test) diff --git a/lingvo/repo.bzl b/lingvo/repo.bzl deleted file mode 100644 index 0a59eb27..00000000 --- a/lingvo/repo.bzl +++ /dev/null @@ -1,243 +0,0 @@ -"""Setup autoconf repo for tensorflow.""" - -load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") -load("//third_party:repo.bzl", "third_party_http_archive") - -def _find_tf_include_path(repo_ctx): - exec_result = repo_ctx.execute( - [ - "python3", - "-c", - "import tensorflow as tf; import sys; " + - "sys.stdout.write(tf.sysconfig.get_include())", - ], - quiet = True, - ) - if exec_result.return_code != 0: - fail("Could not locate tensorflow installation path.") - return exec_result.stdout.splitlines()[-1] - -def _find_tf_lib_path(repo_ctx): - exec_result = repo_ctx.execute( - [ - "python3", - "-c", - "import tensorflow as tf; import sys; " + - "sys.stdout.write(tf.sysconfig.get_lib())", - ], - quiet = True, - ) - if exec_result.return_code != 0: - fail("Could not locate tensorflow installation path.") - return exec_result.stdout.splitlines()[-1] - -def _eigen_archive_repo_impl(repo_ctx): - tf_include_path = _find_tf_include_path(repo_ctx) - repo_ctx.symlink(tf_include_path, "tf_includes") - repo_ctx.file( - "BUILD", - content = """ -cc_library( - name = "includes", - hdrs = glob(["tf_includes/Eigen/**/*.h", - "tf_includes/Eigen/**", - "tf_includes/unsupported/Eigen/**/*.h", - "tf_includes/unsupported/Eigen/**"]), - # https://groups.google.com/forum/#!topic/bazel-discuss/HyyuuqTxKok - includes = ["tf_includes"], - visibility = ["//visibility:public"], -) -""", - executable = False, - ) - -def _absl_includes_repo_impl(repo_ctx): - tf_include_path = _find_tf_include_path(repo_ctx) - repo_ctx.symlink( - tf_include_path + "/absl", - "absl", - ) - repo_ctx.file( - "BUILD", - content = """ -cc_library( - name = "includes", - hdrs = glob(["absl/**/*.h", - "absl/**/*.inc"]), - includes = ["absl"], - visibility = ["//visibility:public"], -) -""", - executable = False, - ) - -def _zlib_includes_repo_impl(repo_ctx): - tf_include_path = _find_tf_include_path(repo_ctx) - repo_ctx.symlink( - tf_include_path + "/external/zlib", - "zlib", - ) - repo_ctx.file( - "BUILD", - content = """ -cc_library( - name = "includes", - hdrs = glob(["zlib/**/*.h"]), - includes = ["zlib"], - visibility = ["//visibility:public"], -) -""", - executable = False, - ) - -def _protobuf_includes_repo_impl(repo_ctx): - tf_include_path = _find_tf_include_path(repo_ctx) - repo_ctx.symlink(tf_include_path, "tf_includes") - repo_ctx.file( - "BUILD", - content = """ -cc_library( - name = "includes", - hdrs = glob(["tf_includes/google/protobuf/*.h", - "tf_includes/google/protobuf/*.inc", - "tf_includes/google/protobuf/**/*.h", - "tf_includes/google/protobuf/**/*.inc"]), - includes = ["tf_includes"], - visibility = ["//visibility:public"], -) -""", - executable = False, - ) - -def _tensorflow_includes_repo_impl(repo_ctx): - tf_include_path = _find_tf_include_path(repo_ctx) - repo_ctx.symlink(tf_include_path, "tensorflow_includes") - repo_ctx.file( - "BUILD", - content = """ -cc_library( - name = "includes", - hdrs = glob(["tensorflow_includes/**/*.h", - "tensorflow_includes/third_party/eigen3/**"]), - includes = ["tensorflow_includes"], - deps = ["@absl_includes//:includes", - "@eigen_archive//:includes", - "@protobuf_archive//:includes", - "@zlib_includes//:includes",], - visibility = ["//visibility:public"], -) -""", - executable = False, - ) - -def _tensorflow_solib_repo_impl(repo_ctx): - tf_lib_path = _find_tf_lib_path(repo_ctx) - repo_ctx.symlink(tf_lib_path, "tensorflow_solib") - repo_ctx.file( - "BUILD", - content = """ -cc_library( - name = "framework_lib", - srcs = ["tensorflow_solib/libtensorflow_framework.so.2"], - visibility = ["//visibility:public"], -) -""", - ) - -def cc_tf_configure(): - """Autoconf pre-installed tensorflow repo.""" - make_eigen_repo = repository_rule(implementation = _eigen_archive_repo_impl) - make_eigen_repo(name = "eigen_archive") - make_absl_repo = repository_rule( - implementation = _absl_includes_repo_impl, - ) - make_absl_repo(name = "absl_includes") - make_zlib_repo = repository_rule( - implementation = _zlib_includes_repo_impl, - ) - make_zlib_repo(name = "zlib_includes") - make_protobuf_repo = repository_rule( - implementation = _protobuf_includes_repo_impl, - ) - make_protobuf_repo(name = "protobuf_archive") - make_tfinc_repo = repository_rule( - implementation = _tensorflow_includes_repo_impl, - ) - make_tfinc_repo(name = "tensorflow_includes") - make_tflib_repo = repository_rule( - implementation = _tensorflow_solib_repo_impl, - ) - make_tflib_repo(name = "tensorflow_solib") - -def lingvo_testonly_deps(): - if not native.existing_rule("com_google_googletest"): - http_archive( - name = "com_google_googletest", - build_file_content = """ -cc_library( - name = "gtest", - srcs = [ - "googletest/src/gtest-all.cc", - "googlemock/src/gmock-all.cc", - ], - copts = ["-D_GLIBCXX_USE_CXX11_ABI=0"], - hdrs = glob([ - "**/*.h", - "googletest/src/*.cc", - "googlemock/src/*.cc", - ]), - includes = [ - "googlemock", - "googletest", - "googletest/include", - "googlemock/include", - ], - linkopts = ["-pthread"], - visibility = ["//visibility:public"], -) - -cc_library( - name = "gtest_main", - copts = ["-D_GLIBCXX_USE_CXX11_ABI=0"], - srcs = ["googlemock/src/gmock_main.cc"], - linkopts = ["-pthread"], - visibility = ["//visibility:public"], - deps = [":gtest"], -) -""", - urls = [ - "https://github.com/google/googletest/archive/release-1.8.0.tar.gz", - ], - sha256 = "58a6f4277ca2bc8565222b3bbd58a177609e9c488e8a72649359ba51450db7d8", - strip_prefix = "googletest-release-1.8.0", - ) - -def lingvo_protoc_deps(): - http_archive( - name = "protobuf_protoc", - build_file_content = """ -filegroup( - name = "protoc_bin", - srcs = ["bin/protoc"], - visibility = ["//visibility:public"], -) -""", - urls = [ - "https://github.com/protocolbuffers/protobuf/releases/download/v3.8.0/protoc-3.8.0-linux-x86_64.zip", - ], - sha256 = "717903f32653f07cd895cfe89ca18ff4ca35f825afa7fe17bcb5cb13bf628be0", - ) - -def icu(): - third_party_http_archive( - name = "icu", - strip_prefix = "icu-release-64-2", - sha256 = "dfc62618aa4bd3ca14a3df548cd65fe393155edd213e49c39f3a30ccd618fc27", - urls = [ - "https://storage.googleapis.com/mirror.tensorflow.org/github.com/unicode-org/icu/archive/release-64-2.zip", - "https://github.com/unicode-org/icu/archive/release-64-2.zip", - ], - build_file = "//third_party/icu:BUILD.bazel", - system_build_file = "//third_party/icu:BUILD.system", - patch_file = "//third_party/icu:udata.patch", - ) diff --git a/lingvo/tasks/BUILD b/lingvo/tasks/BUILD deleted file mode 100644 index e98e1a23..00000000 --- a/lingvo/tasks/BUILD +++ /dev/null @@ -1,20 +0,0 @@ -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -py_library( - name = "all_params", - srcs_version = "PY3", - # LINT.IfChange(task_dirs) - deps = [ - "//lingvo/tasks/asr/params", - "//lingvo/tasks/car/params", - "//lingvo/tasks/image/params", - "//lingvo/tasks/lm/params", - "//lingvo/tasks/mt/params", - "//lingvo/tasks/punctuator/params", - ], - # LINT.ThenChange(../model_imports.py:task_dirs) -) diff --git a/lingvo/tasks/__init__.py b/lingvo/tasks/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/lingvo/tasks/asr/BUILD b/lingvo/tasks/asr/BUILD deleted file mode 100644 index 4f392698..00000000 --- a/lingvo/tasks/asr/BUILD +++ /dev/null @@ -1,286 +0,0 @@ -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -py_library( - name = "frontend", - srcs = ["frontend.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:py_utils", - ], -) - -py_test( - name = "frontend_test", - srcs = ["frontend_test.py"], - data = [ - "//lingvo/tools/testdata:audio_data", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":frontend", - # Additional FFT kernels dependency. - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_helper", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "input_generator", - srcs = ["input_generator.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_input_generator", - "//lingvo/core:generic_input", - "//lingvo/core:py_utils", - ], -) - -py_test( - name = "input_generator_test", - srcs = ["input_generator_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":input_generator", - "//lingvo:compat", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "encoder", - srcs = ["encoder.py", "encoder_v2.py", "blocks.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:layers", - # "//lingvo/core:conformer_layer", - "//lingvo/core:model_helper", - "//lingvo/core:plot", - "//lingvo/core:py_utils", - "//lingvo/core:rnn_cell", - "//lingvo/core:rnn_layers", - "//lingvo/core:spectrum_augmenter", - "//lingvo/core:summary_utils", - ], -) - -py_test( - name = "encoder_test", - srcs = ["encoder_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":encoder", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "fusion", - srcs = ["fusion.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:layers", - "//lingvo/core:py_utils", - "//lingvo/tasks/lm:layers", - ], -) - -py_library( - name = "decoder_utils", - srcs = ["decoder_utils.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:symbolic", - # Implicit six dependency. - ], -) - -py_test( - name = "decoder_utils_test", - srcs = ["decoder_utils_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":decoder", - ":decoder_utils", - "//lingvo:compat", - "//lingvo/core:rnn_cell", - "//lingvo/core:symbolic", - "//lingvo/core:test_utils", - ], -) - -py_library( - name = "contextualizer_base", - srcs = ["contextualizer_base.py"], - srcs_version = "PY3", - deps = [ - "//lingvo/core:base_layer", - ], -) - -py_library( - name = "decoder", - srcs = ["decoder.py"], - srcs_version = "PY3", - deps = [ - ":contextualizer_base", - ":decoder_utils", - ":fusion", - "//lingvo:compat", - "//lingvo/core:attention", - "//lingvo/core:base_decoder", - "//lingvo/core:cluster_factory", - "//lingvo/core:layers", - "//lingvo/core:plot", - "//lingvo/core:py_utils", - "//lingvo/core:recurrent", - "//lingvo/core:rnn_cell", - "//lingvo/core:summary_utils", - "//lingvo/core:symbolic", - # Implicit matplotlib dependency. - ], -) - -py_test( - name = "decoder_test", - size = "large", - srcs = ["decoder_test.py"], - python_version = "PY3", - shard_count = 10, - srcs_version = "PY3", - deps = [ - ":decoder", - # Implicit python proto dependency. - "//lingvo:compat", - "//lingvo/core:cluster_factory", - "//lingvo/core:layers", - "//lingvo/core:py_utils", - "//lingvo/core:symbolic", - "//lingvo/core:test_utils", - "//lingvo/core/ops:hyps_py_pb2", - # Implicit numpy dependency. - ], -) - -py_library( - name = "model_test_input_generator", - srcs = ["model_test_input_generator.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_input_generator", - "//lingvo/core:py_utils", - ], -) - -py_library( - name = "model", - srcs = ["model.py"], - srcs_version = "PY3", - deps = [ - ":decoder", - ":decoder_utils", - ":encoder", - ":frontend", - "//lingvo:compat", - "//lingvo/core:base_model", - "//lingvo/core:metrics", - "//lingvo/core:py_utils", - "//lingvo/core:schedule", - "//lingvo/tools:audio_lib", - ], -) - -py_library( - name = "ctc_model", - srcs = ["ctc_model.py"], - srcs_version = "PY2AND3", - deps = [ - ":encoder", - ":frontend", - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:base_model", - "//lingvo/core:metrics", - "//lingvo/core:py_utils", - "//lingvo/core:schedule", - "//lingvo/tools:audio_lib", - # Implicit six dependency. - ], -) - -py_test( - name = "model_test", - srcs = ["model_test.py"], - data = ["//lingvo/tools/testdata:audio_data"], - python_version = "PY3", - shard_count = 15, - srcs_version = "PY3", - deps = [ - ":decoder", - ":input_generator", - ":model", - ":model_test_input_generator", - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:cluster_factory", - "//lingvo/core:py_utils", - "//lingvo/core:schedule", - "//lingvo/core:summary_utils", - "//lingvo/core:test_helper", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_test( - name = "ctc_model_test", - srcs = ["ctc_model_test.py"], - data = ["//lingvo/tools/testdata:audio_data"], - python_version = "PY3", - shard_count = 1, - srcs_version = "PY3", - deps = [ - ":decoder", - ":input_generator", - ":ctc_model", - ":model_test_input_generator", - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:cluster_factory", - "//lingvo/core:py_utils", - "//lingvo/core:schedule", - "//lingvo/core:summary_utils", - "//lingvo/core:test_helper", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -filegroup( - name = "wpm_files", - srcs = ["wpm_16k_librispeech.vocab"], -) diff --git a/lingvo/tasks/asr/__init__.py b/lingvo/tasks/asr/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/lingvo/tasks/asr/align_input_generator.py b/lingvo/tasks/asr/align_input_generator.py deleted file mode 100644 index 83feb731..00000000 --- a/lingvo/tasks/asr/align_input_generator.py +++ /dev/null @@ -1,147 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Speech recognition input generator.""" - -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import generic_input -from lingvo.core import py_utils - -from tensorflow.python.ops import inplace_ops # pylint:disable=g-direct-tensorflow-import - - -class AsrInput(base_input_generator.BaseSequenceInputGenerator): - """Input generator for ASR.""" - - @classmethod - def Params(cls): - """Defaults params for AsrInput.""" - p = super().Params() - p.Define('frame_size', 40, 'The number of coefficients in each frame.') - p.Define('append_eos_frame', True, 'Append an all-zero frame.') - p.Define('static_vocabulary_table_path', '', 'Path to new-line delimited table of sample id keys') - p.source_max_length = 3000 - return p - - def _DataSourceFromFilePattern(self, file_pattern): - - def Proc(record): - """Parses a serialized tf.Example record.""" - # There we go! string, string, float32. I hope frames is allowed - # to be a waveform directly... - features = [ - ('uttid', tf.io.VarLenFeature(tf.int64)), - # Would like to change this to tf.int16 in the future, if that is possible (would have to read from - ('frames', tf.io.VarLenFeature(tf.float32)), - ] - example = tf.io.parse_single_example(record, dict(features)) - fval = {k: v.values for k, v in example.items()} - # Reshape the flattened vector into its original time-major - # representation. - fval['frames'] = tf.reshape( - fval['frames'], shape=[-1, self.params.frame_size]) - # Input duration determines the bucket. - bucket_key = tf.cast(tf.shape(fval['frames'])[0], tf.int32) - if self.params.append_eos_frame: - bucket_key += 1 - src_paddings = tf.zeros([tf.shape(fval['frames'])[0]], dtype=tf.float32) - return [ - fval['uttid'], fval['frames'], src_paddings - ], bucket_key - - return generic_input.GenericInput( - file_pattern=file_pattern, - processor=Proc, - dynamic_padding_dimensions=[0] * 3, - dynamic_padding_constants=[0] * 2 + [1], - **self.CommonInputOpArgs()) - - def _MaybePadSourceInputs(self, src_inputs, src_paddings): - p = self.params - if not p.append_eos_frame: - return src_inputs, src_paddings - - per_src_len = tf.reduce_sum(1 - src_paddings, 1) - per_src_len += 1 - max_src_len = tf.reduce_max(per_src_len) - input_shape = tf.shape(src_inputs) - input_len = tf.maximum(input_shape[1], tf.cast(max_src_len, tf.int32)) - pad_steps = input_len - input_shape[1] - src_inputs = tf.concat([ - src_inputs, - tf.zeros( - inplace_ops.inplace_update(input_shape, 1, pad_steps), - src_inputs.dtype) - ], 1) - src_paddings = 1 - tf.sequence_mask( - tf.reshape(per_src_len, [input_shape[0]]), tf.reshape(input_len, []), - src_paddings.dtype) - return src_inputs, src_paddings - - def __init__(self, params): - super().__init__(params) - p = self.params - - (utt_ids, src_frames, src_paddings), self._bucket_keys = self._BuildDataSource() - - self._sample_ids = utt_ids - - src_frames, src_paddings = self._MaybePadSourceInputs( - src_frames, src_paddings) - - # We expect src_inputs to be of shape - # [batch_size, num_frames, feature_dim, channels]. - src_frames = tf.expand_dims(src_frames, axis=-1) - - if p.pad_to_max_seq_length: - assert p.source_max_length - assert p.target_max_length - - if all(x == p.bucket_batch_limit[0] for x in p.bucket_batch_limit): - # Set the input batch size as an int rather than a tensor. - src_frames_shape = (self.InfeedBatchSize(), p.source_max_length, - p.frame_size, 1) - src_paddings_shape = (self.InfeedBatchSize(), p.source_max_length) - else: - tf.logging.warning( - 'Could not set static input shape since not all bucket batch sizes ' - 'are the same:', p.bucket_batch_limit) - src_frames_shape = None - src_paddings_shape = None - - src_frames = py_utils.PadBatchDimension(src_frames, self.InfeedBatchSize(), 0) - src_paddings = py_utils.PadBatchDimension(src_paddings, self.InfeedBatchSize(), - 1) - self._sample_ids = py_utils.PadBatchDimension(self._sample_ids, self.InfeedBatchSize(), - self._sample_ids.min) - - src_frames = py_utils.PadSequenceDimension( - src_frames, p.source_max_length, 0, shape=src_frames_shape) - src_paddings = py_utils.PadSequenceDimension( - src_paddings, p.source_max_length, 1, shape=src_paddings_shape) - self._sample_ids = tf.ensure_shape(self._sample_ids, self.InfeedBatchSize()) - - src = py_utils.NestedMap(src_inputs=src_frames, paddings=src_paddings) - - self._src = src - - def _InputBatch(self): - batch = py_utils.NestedMap() - batch.bucket_keys = self._bucket_keys - batch.src = self._src - # Bingo. TPUs don't support tf.string... If anyone there were some way to avoid enqueueing and dequeueing this in the first place... - batch.sample_ids = self._sample_ids - - return batch diff --git a/lingvo/tasks/asr/blocks.py b/lingvo/tasks/asr/blocks.py deleted file mode 100644 index 9b6e20b6..00000000 --- a/lingvo/tasks/asr/blocks.py +++ /dev/null @@ -1,315 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Encoders for the speech model.""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import layers -from lingvo.core import model_helper -from lingvo.core import py_utils -from lingvo.core import rnn_cell -# from lingvo.core import conformer_layer - - -class LSTMBlock(base_layer.BaseLayer): - - @classmethod - def Params(cls): - """Configs for AsrEncoder.""" - p = super().Params() - p.Define('lstm_tpl', rnn_cell.LSTMCellSimple.Params(), - 'Configs template for the RNN layer.') - p.Define('lstm_cell_size', 256, 'LSTM cell size for the RNN layer.') - p.Define('input_feats', None, 'Number of features input to first LSTM.') - p.Define('lstm_type', 'fwd', 'fwd or bidi') - p.Define('num_lstm_layers', 3, 'Number of rnn layers to create') - p.Define('dropout', layers.DropoutLayer.Params(), - 'Apply dropout between LSTM') - p.Define('bidi_rnn_type', 'func', - 'Options: func. func: BidirectionalFRNN. ') - p.Define('unidi_rnn_type', 'func', - 'Options: func. func: UnidirectionalFRNN. ') - # Default config for the rnn layer. - p.lstm_tpl.params_init = py_utils.WeightInit.Uniform(0.1) - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - params_rnn_layers = [] - lstm_out_size = p.lstm_cell_size - - assert p.lstm_type in {'bidi', 'fwd'}, 'Only fwd, bidi allowed' - if p.lstm_type == 'bidi': - lstm_out_size *= 2 - - output_dim = p.input_feats - for i in range(p.num_lstm_layers): - input_dim = output_dim - forward_p = p.lstm_tpl.Copy() - forward_p.name = 'fwd_rnn_L%d' % i - forward_p.num_input_nodes = input_dim - forward_p.num_output_nodes = p.lstm_cell_size - - if p.lstm_type == 'bidi': - backward_p = forward_p.Copy() - backward_p.name = 'bak_rnn_L%d' % i - rnn_p = self.CreateBidirectionalRNNParams(forward_p, backward_p) - rnn_p.name = 'brnn_L%d' % i - else: - rnn_p = self.CreateUnidirectionalRNNParams(forward_p) - rnn_p.name = 'rnn_L%d' % i - - params_rnn_layers.append(rnn_p) - output_dim = lstm_out_size - - if p.dropout: - self.CreateChildren('dropout', p.dropout) - - self.CreateChildren('rnn', params_rnn_layers) - - @property - def output_dim(self): - multiplier = 2 if self.params.lstm_type == 'bidi' else 1 - return self.params.lstm_cell_size * multiplier - - def CreateBidirectionalRNNParams(self, forward_p, backward_p): - return model_helper.CreateBidirectionalRNNParams(self.params, forward_p, - backward_p) - - def CreateUnidirectionalRNNParams(self, forward_p): - return model_helper.CreateUnidirectionalRNNParams(self.params, forward_p) - - def FProp(self, theta, inputs, paddings): - """ - inputs: (B, T, F, 1) - paddings: (B, T) - outputs: encoded, padding: (T, B, F), (T, B) - """ - - p = self.params - rnn_in = tf.transpose(tf.squeeze(inputs, axis=3), [1, 0, 2]) - rnn_padding = tf.expand_dims(tf.transpose(paddings), 2) - # rnn_in is of shape [time, batch, depth] - # rnn_padding is of shape [time, batch, 1] - - for i in range(p.num_lstm_layers): - rnn_out = self.rnn[i].FProp(theta.rnn[i], rnn_in, rnn_padding) - - if p.lstm_type == 'fwd': - rnn_out, _ = rnn_out - - if p.dropout: - rnn_out = self.dropout.FProp(theta.dropout, rnn_out) - - rnn_in = rnn_out - - encoded = rnn_out * (1.0 - rnn_padding) - padding = tf.squeeze(rnn_padding, [2]) - return encoded, padding - - -class ConvolutionalDownsampler(base_layer.BaseLayer): - """ Use convolution with striding to achieve stacking effect """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('cnn_tpl', layers.ConvLayer.Params(), - 'Configs template for the conv layer.') - p.Define('num_cnn_layers', None, 'Number of CNN layers') - p.Define('conv_filter_shapes', None, 'Filter shapes for each conv layer.') - p.Define('conv_filter_strides', None, - 'Filter strides for each conv layer (T, F).') - p.Define('input_shape', [None, None, 80, 1], - 'Shape of the input. This should a TensorShape with rank 4.') - - p.num_cnn_layers = 2 - p.conv_filter_shapes = [(5, 5, 1, 32), (5, 5, 32, 32)] - p.conv_filter_strides = [(1, 2), (3, 2)] # (T, F) - p.cnn_tpl.params_init = py_utils.WeightInit.TruncatedGaussian(0.1) - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - assert p.num_cnn_layers == len(p.conv_filter_shapes) - assert p.num_cnn_layers == len(p.conv_filter_strides) - params_conv_layers = [] - for i in range(p.num_cnn_layers): - conv_p = p.cnn_tpl.Copy() - conv_p.name = 'conv_L%d' % i - conv_p.filter_shape = p.conv_filter_shapes[i] - conv_p.filter_stride = p.conv_filter_strides[i] - params_conv_layers.append(conv_p) - - self.CreateChildren('conv', params_conv_layers) - - assert p.input_shape is not None - conv_output_shape = p.input_shape - for i in range(p.num_cnn_layers): - conv_output_shape = self.conv[i].OutShape(conv_output_shape) - - assert len(conv_output_shape) == 4 # batch, height, width, channel. - feat_dim = conv_output_shape[-1] * conv_output_shape[-2] - self.conv_output_shape = (*conv_output_shape[:2], feat_dim, 1) - - @property - def output_dim(self): - return self.conv_output_shape[2] - - def FProp(self, theta, inputs, paddings): - # BTF1 -> BTF1 - - conv_out = inputs - out_padding = paddings - - for i, conv_layer in enumerate(self.conv): - conv_out, out_padding = conv_layer.FProp(theta.conv[i], conv_out, - out_padding) - - b, t, f1, f2 = py_utils.GetShape(conv_out) - conv_out = tf.reshape(conv_out, [b, t, f1 * f2, 1]) - return conv_out, out_padding - - -class InputStackingDownsampler(base_layer.BaseLayer): - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_shape', [None, None, 80, 1], - 'Shape of the input. This should a TensorShape with rank 4.') - p.Define('stacking', layers.StackingOverTime.Params(), 'Stacking params') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self.CreateChild('stacking', p.stacking.Copy()) - self.num_feats = p.input_shape[-2] * self.stacking.window_size - - @property - def output_dim(self): - return self.num_feats - - def FProp(self, theta, inputs, paddings): - # BTF1 -> BTF1 - inputs = tf.squeeze(inputs, axis=3) # BTF1 -> BTF - paddings = tf.expand_dims(paddings, axis=2) # BT -> BT1 - encoded, padding = self.stacking.FProp(inputs, paddings) - encoded = tf.expand_dims(encoded, axis=3) # BTF -> BTF1 - padding = tf.squeeze(padding, axis=2) # BT1 -> BT - return encoded, padding - - -# Lingvo's conformer models work with TF2.3 but TPUs have issues with TF2.3 -# class ConformerBlock(base_layer.BaseLayer): - -# @ classmethod -# def Params(cls): -# p = super().Params() - -# p.Define('input_feats', None, 'Required') -# p.Define('dropout_prob', 0., '') - -# p.Define('atten_num_heads', 4, '') -# p.Define('atten_local_context', 3, '') -# p.Define('kernel_size', 32, '') -# p.Define('fflayer_activation', 'SWISH', '') -# p.Define('layer_order', 'mhsa_before_conv', '') -# p.Define('num_conformer_blocks', 1, 'Number of conformer layers') - -# return p - -# def __init__(self, params): -# super().__init__(params) -# p = self.params -# assert p.input_feats is not None - -# conformer_tpl = conformer_layer.ConformerLayer.CommonParams( -# input_dim=p.input_feats, -# atten_num_heads=p.atten_num_heads, -# atten_local_context=p.atten_local_context, -# kernel_size=p.kernel_size, -# fflayer_hidden_dim=p.input_feats // 2) - -# for i in range(p.num_conformer_blocks): -# self.CreateChild(f'conformer_{i}', conformer_tpl.Copy()) - -# @property -# def output_dim(self): -# return self.params.input_feats - -# def FProp(self, theta, input, paddings): -# """ -# inputs: (B, T, F, 1) -# paddings: (B, T) -# outputs: encoded, padding: (T, B, F), (T, B) -# same as LSTM block -# """ -# p = self.params - -# input = tf.squeeze(input, 3) # BTF1 -> BTF - -# for i in range(p.num_conformer_blocks): -# conf_block = getattr(self, f'conformer_{i}') -# conf_theta = getattr(theta, f'conformer_{i}') -# encoded, padding = conf_block.FProp(conf_theta, input, paddings) -# input, paddings = encoded, padding - -# encoded = tf.transpose(encoded, [1, 0, 2]) -# padding = tf.transpose(padding) -# return encoded, padding - - -class VocabProjectionBlock(base_layer.BaseLayer): - - @classmethod - def Params(cls): - p = super().Params() - p.Define('input_dim', None, 'Required') - p.Define('vocab_size', None, 'Required') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - pp = layers.FCLayer.Params() - pp.activation = 'NONE' - pp.output_dim = min(p.vocab_size, 96) - pp.input_dim = p.input_dim - pp.params_init = py_utils.WeightInit.Uniform(0.1) - self.CreateChild('projection', pp.Copy()) - - remaining = p.vocab_size - 96 - if remaining > 0: - rp = layers.FCLayer.Params() - rp.activation = 'NONE' - rp.output_dim = p.vocab_size - 96 - rp.input_dim = p.input_dim - rp.params_init = py_utils.WeightInit.Uniform(0.01) - self.CreateChild('lowpri_projection', rp.Copy()) - - def FProp(self, theta, inputs, paddings): - encoded = self.projection.FProp(theta.projection, inputs) - - if hasattr(self, 'lowpri_projection'): - encoded2 = self.lowpri_projection.FProp(theta.lowpri_projection, inputs) - encoded = tf.concat([encoded, encoded2], axis=2) - - return encoded, paddings diff --git a/lingvo/tasks/asr/contextualizer_base.py b/lingvo/tasks/asr/contextualizer_base.py deleted file mode 100644 index 5c6ff7d4..00000000 --- a/lingvo/tasks/asr/contextualizer_base.py +++ /dev/null @@ -1,126 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""API for context injection into a speech decoder.""" - -from lingvo.core import base_layer - - -class ContextualizerBase(base_layer.BaseLayer): - """Base class for a contextualizer. - - Typical usage of a contextualizer is:: - - contextualizer.SetContextMap(context_map, theta) # Set context map. - contextualizer.InitAttention(...) # Initalize attn sources. - context = contextualizer.ZeroAttention(...) # Initalize attn state. - context = contextualizer.QueryAttention(...) # Call on each decoding step. - - `context` is a per-decoding-step context vector that augments the standard - LAS model with additional context. - - `context_map` can include data needed for initialization. - - After parameters of the contextualizer are set, these accessors can be used: - - - contextualizer.GetContextDim() - """ - - def SetContextMap(self, context_map, theta): - """Set the context map. - - Args: - context_map: A NestedMap object containing the context from which - attention vectors will be computed. - theta: NestedMap, parameters needed for embedding. - """ - raise NotImplementedError('SetContextMap') - - def InitAttention(self, theta, packed_src, misc_states): - """Initialized the contextualizer's attention. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - packed_src: A NestedMap object into which attention source vectors will be - placed. - misc_states: A NestedMap object into which attention states will be - placed. - """ - raise NotImplementedError('InitAttention') - - def ZeroAttention(self, theta, dec_bs, misc_states, audio_context, - packed_src): - """Creates the contextualizer 'zero' context vector. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - dec_bs: int32, the decoding batch size. - misc_states: A NestedMap object into which attention states will be - placed. - audio_context: Tensor of shape [dec_bs, aud_dim] representing the - audio-based context vector. - packed_src: A NestedMap object into which attention source vectors will be - placed. - - Returns: - A 'zero' context vector of shape [dec_bs, aud_dim + context_dim] - """ - raise NotImplementedError('ZeroAttention') - - def QueryAttention(self, theta, attn_query, misc_states, audio_context, - packed_src): - """Query the contextualizer's attention. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - attn_query: Tensor of shape [dec_bs, ...] representing the query vectors. - misc_states: A NestedMap object into which attention states will be - placed. - audio_context: Tensor of shape [dec_bs, aud_dim] representing the - audio-based context vector. - packed_src: A NestedMap object into which attention source vectors will be - placed. - - Returns: - A context vector of shape [dec_bs, aud_dim + context_dim] - """ - raise NotImplementedError('QueryAttention') - - def GetContextDim(self): - """Returns the context dimension.""" - raise NotImplementedError('GetContextDim') - - -class NullContextualizer(ContextualizerBase): - """An 'empty' or no-op contextualizer.""" - - def SetContextMap(self, context_map, theta): - pass - - def InitAttention(self, theta, packed_src): - pass - - def ZeroAttention(self, theta, dec_bs, misc_states, audio_context, - packed_src): - return audio_context - - def QueryAttention(self, theta, attn_query, misc_states, audio_context, - packed_src): - return audio_context - - def GetContextDim(self): - return 0 diff --git a/lingvo/tasks/asr/ctc_model.py b/lingvo/tasks/asr/ctc_model.py deleted file mode 100644 index ad655d39..00000000 --- a/lingvo/tasks/asr/ctc_model.py +++ /dev/null @@ -1,333 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""CTC model.""" - -from collections import defaultdict -import numpy as np -import os - -import lingvo.compat as tf -from lingvo.core import base_model -from lingvo.core import metrics -from lingvo.core import py_utils -from lingvo.core import schedule -from lingvo.tasks.asr import blocks -from lingvo.tasks.asr import encoder_v2 -from lingvo.tasks.asr import decoder_utils -from lingvo.tasks.asr import input_generator - -# https://stackoverflow.com/a/2912455 -class keydefaultdict(defaultdict): - def __missing__(self, key): - if self.default_factory is None: - raise KeyError(key) - else: - ret = self[key] = self.default_factory(key) # pylint: disable=not-callable - return ret - -class CTCModel(base_model.BaseTask): - """ - CTC model without a language model. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - # Need to set frontend appropriately - 'frontend', None, - 'ASR frontend to extract features from input. Defaults to no frontend ' - 'which means that features are taken directly from the input.') - - p.Define('encoder_v2', encoder_v2.AsrEncoder.Params(), 'Encoder V2 Params') - - # Defaults based on graphemes / ascii_tokenizer.cc - p.Define('vocab_size', 76, - 'Vocabulary size, *including* the blank symbol.') - p.Define( - 'blank_index', 73, 'Index assigned to epsilon, aka blank for CTC. ' - 'This should never appear in the label sequence. Reconsider this.') - - p.Define('inference_compute_only_log_softmax', False, - 'At inference time, compute the output of log softmax, rather ' - 'than running any sort of CTC decoder.') - p.Define('log_softmax_output_directory', '', - 'Path to which to dump log_softmax values') - - tp = p.train - tp.lr_schedule = (schedule.PiecewiseConstantSchedule.Params().Set( - boundaries=[350000, 450000, 600000], values=[1.0, 0.1, 0.01, 0.001])) - - tp.vn_start_step = 20_000 - tp.vn_std = 0.075 - tp.l2_regularizer_weight = 1e-6 - tp.clip_gradient_norm_to_value = 1.0 - tp.grad_norm_to_clip_to_zero = 100.0 - tp.tpu_steps_per_loop = 100 - - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - if not p.encoder_v2.name: - p.encoder_v2.name = 'enc' - self.CreateChild('encoder', p.encoder_v2) - - if p.frontend: - self.CreateChild('frontend', p.frontend) - - projection_p = blocks.VocabProjectionBlock.Params() - projection_p.vocab_size = p.vocab_size - projection_p.input_dim = self.encoder.output_dim - self.CreateChild('project_to_vocab_size', projection_p) - - if p.inference_compute_only_log_softmax: - assert p.log_softmax_output_directory, 'Must provide non-empty path to which to dump log softmax values' - # `ulimit -n` gives me 1024 on this machine. We may need to - # increase the number of open file descriptors allowed in this - # process - class WriterAndRecordsWritten: - records_written: int - file_handle: tf.io.TFRecordWriter - def __init__(self, write_path: str): - self.records_written = 0 - self.file_handle = tf.io.TFRecordWriter(write_path) - self._int64_audio_document_id_to_file_handle = keydefaultdict( - WriterAndRecordsWritten) - - def ComputePredictions(self, theta, input_batch): - return self._FrontendAndEncoderFProp(theta, input_batch.src) - - def DecodeWithTheta(self, theta, input_batch): - """Constructs the inference graph.""" - p = self.params - # from IPython import embed; embed() - with tf.name_scope('decode'), tf.name_scope(p.name): - with tf.name_scope('encoder'): - encoder_outputs = self._FrontendAndEncoderFProp(theta, input_batch.src) - if p.inference_compute_only_log_softmax: - global_step = tf.train.get_global_step() - increment_global_step = tf.assign(global_step, global_step + 1) - with tf.control_dependencies([increment_global_step]): - log_probabilities = tf.transpose(tf.nn.log_softmax(encoder_outputs.encoded, axis=2), - perm=(1, 0, 2)) - with tf.name_scope('decoder'): - decoder_outs = self._DecodeCTC(encoder_outputs) - # encoder_outputs's shape is [T,B,F] - return {'log_probabilities': log_probabilities, - 'log_probabilities_lengths': - py_utils.LengthsFromBitMask(encoder_outputs.padding, 0), - 'int64_uttid': input_batch.sample_ids, - 'int64_audio_document_id': input_batch.audio_document_ids, - 'num_utterances_in_audio_document': input_batch.num_utterances_in_audio_document, - 'transcripts': decoder_outs.transcripts, - } - with tf.name_scope('decoder'): - decoder_outs = self._DecodeCTC(encoder_outputs) - - decoder_metrics = py_utils.RunOnTpuHost(self._CalculateErrorRates, decoder_outs, input_batch) - return decoder_metrics - - def PostProcessDecodeOut(self, decode_out_dict, dec_metrics_dict): - p = self.params - if p.inference_compute_only_log_softmax: - mini_batch_size = decode_out_dict['log_probabilities'].shape[0] - dec_metrics_dict['num_samples_in_batch'].Update(mini_batch_size) - - for i in range(mini_batch_size): - int64_uttid = decode_out_dict['int64_uttid'][i] - int64_audio_document_id = decode_out_dict['int64_audio_document_id'][i] - if int64_uttid == input_generator.RawAsrInputIntegerUttIds.PAD_INDEX: - continue - length = decode_out_dict['log_probabilities_lengths'][i] - flat_log_probabilities = decode_out_dict['log_probabilities'][i, :length, :].flatten() - record_bytes = tf.train.Example( - features=tf.train.Features(feature={ - 'int64_uttid': tf.train.Feature( - int64_list=tf.train.Int64List(value=[int64_uttid])), - 'log_probabilities': tf.train.Feature( - float_list=tf.train.FloatList(value=flat_log_probabilities)), - 'transcripts': tf.train.Feature( - bytes_list=tf.train.BytesList(value=[decode_out_dict['transcripts'][i]])), - }) - ).SerializeToString() - output_path = os.path.join(p.log_softmax_output_directory, f'int64_audio_document_id={int64_audio_document_id}/shard.tfrecord') - pair = self._int64_audio_document_id_to_file_handle[output_path] - pair.file_handle.write(record_bytes) - pair.records_written += 1 - if pair.records_written == decode_out_dict['num_utterances_in_audio_document'][i]: - pair.file_handle.close() - del self._int64_audio_document_id_to_file_handle[output_path] - return - - gt_transcripts = decode_out_dict['target_transcripts'] - hyp_transcripts = decode_out_dict['decoded_transcripts'] - if not py_utils.use_tpu(): - utt_id = decode_out_dict['utt_id'] - else: - utt_id = [["TPU does not know utt_id, sorry"]] * len(gt_transcripts) - - for i in range(len(gt_transcripts)): - # tf.logging.info('utt_id : %s', utt_id[i][0]) - tf.logging.info('ref_str: %s', gt_transcripts[i]) - tf.logging.info('hyp_str: %s', hyp_transcripts[i]) - - total_word_err = np.sum(decode_out_dict['num_wrong_words']) - total_ref_words = np.sum(decode_out_dict['num_ref_words']) - total_char_err = np.sum(decode_out_dict['num_wrong_chars']) - total_ref_chars = np.sum(decode_out_dict['num_ref_chars']) - - dec_metrics_dict['num_samples_in_batch'].Update(len(gt_transcripts)) - dec_metrics_dict['wer'].Update(total_word_err / max(1., total_ref_words), - total_ref_words) - dec_metrics_dict['cer'].Update(total_char_err / max(1., total_ref_chars), - total_ref_chars) - tf.logging.info(' ]]] CER: %.3f ]]] WER: %.3f', - dec_metrics_dict['cer'].value, - dec_metrics_dict['wer'].value) - - def ComputeLoss(self, theta, predictions, input_batch): - output_batch = predictions - assert self.params.blank_index == 31 - ctc_loss = tf.nn.ctc_loss( - input_batch.tgt.labels, - output_batch.encoded, - py_utils.LengthsFromBitMask(input_batch.tgt.paddings, 1), - py_utils.LengthsFromBitMask(output_batch.padding, 0), - logits_time_major=True, - blank_index=self.params.blank_index) - - # ctc_loss.shape = (B) - total_loss = tf.reduce_mean(ctc_loss) - per_sequence_loss = {'loss': ctc_loss} - return dict(loss=(total_loss, 1.0)), per_sequence_loss - - def CreateDecoderMetrics(self): - base_metrics = { - 'wer': metrics.AverageMetric(), - 'cer': metrics.AverageMetric(), - 'num_samples_in_batch': metrics.AverageMetric() - } - return base_metrics - - @classmethod - def FPropMeta(cls, params, *args, **kwargs): - raise NotImplementedError('No FPropMeta available.') - - def _FrontendAndEncoderFProp(self, theta, input_batch_src): - p = self.params - - if p.frontend: - input_batch_src = self.frontend.FProp(theta.frontend, input_batch_src) - - encoder_out = self.encoder.FProp(theta.encoder, input_batch_src) - - encoded, padding = self.project_to_vocab_size(encoder_out.encoded, - encoder_out.padding) - outputs = py_utils.NestedMap(encoded=encoded, padding=padding) - return outputs - - def _DecodeCTC(self, output_batch): - tok_logits = output_batch.encoded # (T, B, F) - - # GALVEZ: Make beam_width a tunable parameter! - # (decoded,), _ = py_utils.RunOnTpuHost(tf.nn.ctc_beam_search_decoder, tok_logits, - # py_utils.LengthsFromBitMask( - # output_batch.padding, 0), - # beam_width=100) - # (decoded,), _ = tf.nn.ctc_beam_search_decoder(tok_logits, - # py_utils.LengthsFromBitMask( - # output_batch.padding, 0), - # beam_width=100) - return py_utils.RunOnTpuHost(cpu_tf_graph_decode, - tok_logits, output_batch.padding, - self.params.blank_index, self.input_generator) - - def _CalculateErrorRates(self, dec_outs_dict, input_batch): - # return {'stuff': dec_outs_dict.sparse_ids.values} - gt_seq_lens = py_utils.LengthsFromBitMask(input_batch.tgt.paddings, 1) - gt_transcripts = py_utils.RunOnTpuHost(self.input_generator.IdsToStrings, - input_batch.tgt.labels, - gt_seq_lens) - - # token error rate - char_dist = tf.edit_distance(tf.string_split(dec_outs_dict.transcripts, - sep=''), - tf.string_split(gt_transcripts, sep=''), - normalize=False) - - ref_chars = tf.strings.length(gt_transcripts) - num_wrong_chars = tf.reduce_sum(char_dist) - num_ref_chars = tf.cast(tf.reduce_sum(ref_chars), tf.float32) - cer = num_wrong_chars / num_ref_chars - - # word error rate - word_dist = decoder_utils.ComputeWer(dec_outs_dict.transcripts, - gt_transcripts) # (B, 2) - num_wrong_words = tf.reduce_sum(word_dist[:, 0]) - num_ref_words = tf.reduce_sum(word_dist[:, 1]) - wer = num_wrong_words / num_ref_words - ret_dict = { - 'target_ids': input_batch.tgt.ids, - 'target_labels': input_batch.tgt.labels, - 'target_weights': input_batch.tgt.weights, - 'target_paddings': input_batch.tgt.paddings, - 'target_transcripts': gt_transcripts, - 'decoded_transcripts': dec_outs_dict.transcripts, - 'wer': wer, - 'cer': cer, - 'num_wrong_words': num_wrong_words, - 'num_ref_words': num_ref_words, - 'num_wrong_chars': num_wrong_chars, - 'num_ref_chars': num_ref_chars - } - if not py_utils.use_tpu(): - ret_dict['utt_id'] = input_batch.sample_ids - - return ret_dict - -def cpu_tf_graph_decode(tok_logits, padding, blank_index, input_generator): - # (T, B, F) - # ctc_beam_search_decoder assumes blank_index=0 - assert blank_index == 31 - - # TODO: Consider making beam_width larger - (decoded,), _ = tf.nn.ctc_beam_search_decoder(tok_logits, - py_utils.LengthsFromBitMask(padding, 0), - beam_width=100) - # Could easily use blank_index here as well, right? - invalid = tf.constant(-1, tf.int64) - dense_dec = tf.sparse_to_dense(decoded.indices, - decoded.dense_shape, - decoded.values, - default_value=invalid) - - batch_segments = decoded.indices[:, 0] - times_in_each_batch = decoded.indices[:, 1] - - decoded_seq_lengths = tf.cast(tf.math.segment_max(times_in_each_batch, batch_segments) + 1, tf.int32) - # What happens if an empty sequence is output??? Then pad appropriately, tada! - decoded_seq_lengths = py_utils.PadBatchDimension(decoded_seq_lengths, tf.shape(tok_logits)[1], 0) - - hyp_str = py_utils.RunOnTpuHost(input_generator.IdsToStrings, - tf.cast(dense_dec, tf.int32), - decoded_seq_lengths) - - hyp_str = tf.strings.regex_replace(hyp_str, '()+', '') - hyp_str = tf.strings.regex_replace(hyp_str, '()+', '') - hyp_str = tf.strings.regex_replace(hyp_str, '()+', '') - return py_utils.NestedMap(sparse_ids=decoded, transcripts=hyp_str) diff --git a/lingvo/tasks/asr/ctc_model_test.py b/lingvo/tasks/asr/ctc_model_test.py deleted file mode 100644 index 78825f6b..00000000 --- a/lingvo/tasks/asr/ctc_model_test.py +++ /dev/null @@ -1,75 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for Asr Model.""" - -import lingvo.compat as tf -from lingvo.core import test_utils -from lingvo.tasks.asr import ctc_model -from lingvo.tasks.asr import model_test_input_generator as tig - - -class AsrCtcModelTest(test_utils.TestCase): - # bazel test //lingvo/tasks/asr:ctc_model_test --test_output=all - - def _testParams(self): - input_shape = [12, 16, 80, 1] # (B, T, F, 1) - p = ctc_model.CTCModel.Params() - p.vocab_size = 76 - p.blank_index = 73 - - # Initialize encoder params. - p.encoder = None - ep = p.encoder_v2 - ep.use_specaugment = True - - ep.stacking_subsampler = None - sub = ep.conv_subsampler - sub.input_shape = input_shape - - elp = ep.lstm_block - elp.input_feats = None # leave as none to infer automatically - elp.lstm_cell_size = 128 - elp.num_lstm_layers = 5 - elp.lstm_type = 'fwd' - elp.dropout.keep_prob = 0.8 - - # ep.lstm_block = None - # ecp = ep.conformer_block - # ecp.num_conformer_blocks = 5 - # ecp.name = 'conformer_layer' - - p.input = tig.TestInputGenerator.Params() - p.input.target_max_length = 5 - p.input.source_shape = input_shape - p.input.target_shape = [12, 5] - p.name = 'test_ctc_mdl' - return p - - def testFProp_conv_sub(self): - with self.session(use_gpu=False): - tf.random.set_seed(93820985) - p = self._testParams() - mdl = p.Instantiate() - - # FPropDefaultTheta -> FPropTower -> { ComputePredictions ; ComputeLoss; } - metrics, _ = mdl.FPropDefaultTheta() - self.evaluate(tf.global_variables_initializer()) - - ctc, _ = metrics['loss'] - test_utils.CompareToGoldenSingleFloat(self, 76.710762, ctc.eval()) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/asr/decoder.py b/lingvo/tasks/asr/decoder.py deleted file mode 100644 index 214fbc54..00000000 --- a/lingvo/tasks/asr/decoder.py +++ /dev/null @@ -1,1556 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Decoders for the speech model.""" - -import collections -import math -import lingvo.compat as tf -from lingvo.core import attention -from lingvo.core import base_decoder -from lingvo.core import cluster_factory -from lingvo.core import layers -from lingvo.core import plot -from lingvo.core import py_utils -from lingvo.core import recurrent -from lingvo.core import rnn_cell -from lingvo.core import summary_utils -from lingvo.core import symbolic -from lingvo.tasks.asr import contextualizer_base -from lingvo.tasks.asr import decoder_utils -from lingvo.tasks.asr import fusion -from matplotlib import font_manager - - -def _ToTensorArray(name, v, max_seq_length, clear_after_read=None): - """Create TensorArray from v, of size max_seq_length.""" - ta = tf.TensorArray( - v.dtype, max_seq_length, name=name, clear_after_read=clear_after_read) - ta = ta.unstack(v) - return ta - - -def _NewTensorArray(name, max_seq_length, dtype=None): - """Create empty TensorArray which can store max_seq_length elements.""" - return tf.TensorArray(dtype, max_seq_length, name=name) - - -class AsrDecoderBase(base_decoder.BaseBeamSearchDecoder): - """Base class for RNN-with-attention speech decoders. - - The decoder takes encoder_outputs, a NestedMap generated by the encoder, as - input. The NestedMap is expected to contain the following fields: - - - 'encoded': encoded features - - 'padding': padding for encoded features - - The decoder operates in a 'step-by-step' fashion. The model encapsulates all - information which should persist from one step to the next in the - DecoderStepState NestedMap, which provides a 'misc_states' NestedMap which can - store arbitrary information required by the specific decoder sub-class. - - A 'step' in training consists of the following sequence of steps which compute - the outputs from the decoder given the current input target (and the state of - the model after making the previous predictions): - - 1. Compute the input target at the current time step:: - - cur_target_info = self.TargetsToBeFedAtCurrentDecodeStep(...) - - 2. Update state and compute outputs by running SingleDecodeStep:: - - step_outs, new_state = self.SingleDecodeStep(...) - - 3. Update state based on the logits computed at this step:: - - new_state = self.PostStepDecoderStateUpdate(old_state, logits) - - 4. Display summaries based on the accumulated information across all steps:: - - self.AddAdditionalDecoderSummaries(seq_out_tas) - - Sub-classes can customize behavior by implementing the following functions, - which will modify the behavior of the decoder: - - For beam search decoder: - - _InitBeamSearchStateCallback - - _PreBeamSearchStepCallback - - _PostBeamSearchStepCallback - For EMBR training: - - ComputeHypsWithBeamSearch - - - MiscZeroState: NestedMap which represents the initial state for the - 'misc_states' in the DecoderStepState. The default implementation returns - an empty NestedMap. - - - SingleDecodeStep: This corresponds to the computation which happens in each - step of the model. The function should return the outputs of the decoder - as well as the updated state. - - - PostStepDecoderStateUpdate: A function which updates the DecoderStepState - after the output logits from the decoder have been computed. By default, - this returns the DecoderStepState unchanged. - - - TargetsToBeFedAtCurrentDecodeStep: Returns a TargetInfo namedtuple, which - represents information about the targets which should be input at the - current step, as well as the output label which should be predicted. - The default implementation uses the values in the batched 'targets' - provided by the InputGenerator. - - - AddAdditionalDecoderSummaries: A function which can be used to add any - decoder specific information as part of the summaries displayed during - training. By default this is a no-op. - - - CreateTargetInfoMisc: A function which can be used to store arbitrary - information as required by a sub-classes in the target info arrays used - to determine the current label at each step during training. By default, - this creates an empty NestedMap. - - A few other functions that control how the decoder initializes and computes - attention during the initial step, and during each step can also be - modified, if need be: - - - _GetAttenContextDim: The dimensionality of the attention context vector. - - _CreateAtten: Controls how the attention module is configured. Most - subclasses will not have to change this unless it changes how attention - works. - - BaseZeroState: Returns initial state of RNNs, and attention. - - _InitAttention: Initializes Tensors used by the attention module. - - _GetInitialSeqStateTensorArrays: Get intitial tensor arrays for - ComputePredictionsDynamic. - - _GetNewAttenProbs: Update atten probs for a timestep and return the - updated tensor array. - """ - # pylint: disable=invalid-name - # DecoderStepState encapsulates everything that needs to persist from one - # 'step' of next label prediction to the next. This interface is also used - # while performing inference in the model, so any changes here should be - # undertaken with care. Although the presence of the fields listed below is - # not enforced explicitly in the code, all sub-classes should use the same - # structure. - # - # rnn_states: List of NestedMaps, corresponding to states of all RNNs in the - # decoder. - # atten_context, atten_probs, atten_states: See attention.py for details. - # misc_states: NestedMap, which can contain anything the decoder needs to - # persist from one step to another. - # - # DecoderStepState = py_utils.NestedMap( - # rnn_states=..., - # atten_context=..., - # atten_probs=..., - # atten_states=..., - # misc_states=..., - # ) - - # TargetInfo encapsulates information about the input target sequence - # available during training. - # These are only used during training, so sub-classes are free to add any - # additional target specific information in the 'misc' field, which can be - # used to represent any model specific information. - # misc: NestedMap, that can contain any model specific target information. By - # default, this is an empty NestedMap. - TargetInfo = collections.namedtuple( - 'TargetInfo', ['id', 'label', 'weight', 'emb', 'padding', 'misc']) - - # SequenceOutTensorArrays encapsulates the various outputs generated as we - # step through the decoder. These are used to display statistics during - # training, and the information in these arrays can be used to modify the - # decoder steps, e.g., by using previously predicted outputs to modify targets - # fed at the next step in the ScheduledSampling decoder. - # These are only used during training, so modifications to these to add - # additional components are fine. In particular, the 'misc' field is a list of - # TensorArrays corresponding to the 'misc_states' in the DecoderStepState - # across all steps. - SequenceOutTensorArrays = collections.namedtuple( - 'SequenceOutTensorArrays', - ['rnn_outs', 'step_outs', 'atten_probs', 'logits', 'fusion', 'misc']) - # pylint: enable=invalid-name - - @classmethod - def Params(cls): - p = super().Params() - p.Define('dropout_prob', 0.0, 'Prob at which we do dropout.') - p.Define('emb', layers.EmbeddingLayer.Params(), 'Embedding layer params.') - p.Define('emb_dim', 0, 'dimension of the embedding layer.') - p.Define('label_smoothing', None, 'Label smoothing class.') - p.Define( - 'rnn_cell_tpl', rnn_cell.LSTMCellSimple.Params(), - 'RNNCell params template. ' - 'Can be a single param or ' - 'a list of rnn_layers params, one for each layer.') - p.Define('rnn_cell_dim', 0, 'size of the rnn cells.') - p.Define( - 'rnn_cell_hidden_dim', 0, 'internal size of the rnn cells. When ' - 'set to > 0 it enables a projection layer at the output of the ' - 'rnn cell (see call to SetRnnCellNodes).') - p.Define('attention', attention.AdditiveAttention.Params(), - 'Additive attention params.') - p.Define('softmax', layers.SimpleFullSoftmax.Params(), 'Softmax params.') - p.Define('softmax_uses_attention', True, - 'Controls whether attention is fed to the softmax or not.') - p.Define('source_dim', 0, 'Dimension of the source encodings.') - p.Define('atten_context_dim', 0, - 'Depth of the attention context vector output.') - p.Define( - 'attention_plot_font_properties', '', - 'Adds font properties for the given file if set. Required ' - 'for displaying east-Asian character sets on plot axes.') - p.Define('rnn_layers', 1, 'Number of rnn layers.') - p.Define( - 'residual_start', 0, - 'Start residual connections from this layer. For this and higher ' - 'layers, the layer output is the sum of the RNN cell output and ' - 'input; if the layer also normalizes its output, then the ' - 'normalization is done over this sum. Set to 0 to disable ' - 'residual connections.') - p.Define('fusion', fusion.NullFusion.Params(), 'Fusion class params.') - p.Define('parallel_iterations', 30, - 'Max number of iterations to run in parallel for while loop.') - p.Define( - 'per_token_avg_loss', True, - 'Use per-token average loss when set to True (default); when set ' - 'to False use sequence average loss (sum logP across tokens in an ' - 'output sequence) and average across all sequences in the batch.') - p.Define( - 'token_normalized_per_seq_loss', False, - 'Whether or not to normalize the per-sequence loss by the sequence ' - 'length.') - # Configs for scheduled sampling. - p.Define( - 'min_ground_truth_prob', 1.0, - 'The min probability of using the ground truth as the previous ' - 'prediction.') - p.Define('min_prob_step', 1e6, 'Step to reach min_ground_truth_prob.') - p.Define( - 'prob_decay_start_step', 1e4, - 'The step to starts linearly decrease the probability of sampling ' - 'ground truth.') - p.Define( - 'use_while_loop_based_unrolling', True, - 'Whether or not to use while loop based unrolling for training.' - ' If false, we use a functional while based unrolling.') - p.Define( - 'logit_types', {'logits': 1.0}, - 'A dict of logit_name -> loss_weight. logit_name must be a field in ' - 'the predictions NestedMap. loss_weight should add up to 1.0.') - p.Define( - 'use_unnormalized_logits_as_log_probs', True, - 'If true, decoder beam search may return unnormalized logits as ' - 'log_probs. Used for backwards-compatibility.') - p.Define( - 'contextualizer', contextualizer_base.NullContextualizer.Params(), - 'A contextualizer that can be used' - 'to inject context into the decoder. The default NullContextualizer ' - 'does not add parameters to the model nor changes the ' - 'computation.') - p.Define('focal_loss_alpha', None, 'The weighting factor alpha.') - p.Define('focal_loss_gamma', None, 'Tunable focusing parameter.') - p.Define('adapter_layer_tpl', layers.MultitaskAdapterLayer.Params(), - 'Params for domain/language adatper layer.') - p.Define( - 'adapter_task_id_field', None, - 'Setting this will enable the use of adapter layers. This is the name ' - 'of the field in the encoder_outputs to extract the tasks IDs for ' - 'adatper layers.') - - # Set some reasonable default values. - # Default config for the embedding layer. - vocab = 96 - p.emb_dim = 96 - p.emb.vocab_size = vocab - p.emb.max_num_shards = 1 - p.emb.params_init = py_utils.WeightInit.Uniform(1.0) - # Default config for the rnn layer. - p.rnn_cell_dim = 256 - p.rnn_cell_tpl.params_init = py_utils.WeightInit.Uniform(0.1) - # Default config for the attention model. - p.attention.hidden_dim = 128 - p.attention.params_init = py_utils.WeightInit.UniformSqrtDim(math.sqrt(3.0)) - # Default config for the softmax part. - p.softmax.num_classes = vocab - p.softmax.params_init = py_utils.WeightInit.Uniform(0.1) - # LM config, if used. - p.fusion.lm.vocab_size = vocab - # Other configs. - p.target_seq_len = 300 - p.source_dim = 512 - p.adapter_layer_tpl.data_format = 'TBC' - return p - - @classmethod - def UpdateTargetVocabSize(cls, p, vocab_size, wpm_model=None): - """Updates params with the vocab size and wpm model. - - Args: - p: model params. - vocab_size: size of the vocabulary. - wpm_model: file name prefix pointing to a wordpiece model. - - Returns: - Model params updated with the vocab size and wpm model. - """ - p.emb.vocab_size = vocab_size - p.softmax.num_classes = vocab_size - p.fusion.lm = p.fusion.lm.cls.UpdateTargetVocabSize(p.fusion.lm, vocab_size, - wpm_model) - if p.label_smoothing: - p.label_smoothing.num_classes = vocab_size - return p - - def __init__(self, params): - params = params.Copy() - if params.min_ground_truth_prob < 1: - # Move embedding lookup onto worker. - params.emb.on_ps = False - super().__init__(params) - - p = self.params - assert not p.packed_input, ('Packed inputs are not yet supported for ' - 'AsrDecoderBase.') - - self._max_label_prob = 1 - p.min_ground_truth_prob - self._decay_interval = p.min_prob_step - p.prob_decay_start_step - if self._decay_interval <= 0: - raise ValueError('min_prob_step (%d) <= prob_decay_start_step (%d)' % - (p.min_prob_step, p.prob_decay_start_step)) - - if p.attention_plot_font_properties: - self._font_properties = font_manager.FontProperties( - fname=p.attention_plot_font_properties) - else: - self._font_properties = font_manager.FontProperties() - - name = p.name - self.CreateChild('contextualizer', p.contextualizer) - atten_context_dim = self._GetAttenContextDim() - assert symbolic.IsExpr(atten_context_dim) or atten_context_dim > 0 - - p.emb.dtype = p.dtype - p.emb.embedding_dim = p.emb_dim - self.CreateChild('emb', p.emb) - - params_rnn_cells = [] - params_adapter_layers = [] - feat_dim = p.emb_dim - for i in range(p.rnn_layers): - if isinstance(p.rnn_cell_tpl, (list, tuple)): - assert len(p.rnn_cell_tpl) == p.rnn_layers - rnn_cell_params = p.rnn_cell_tpl[i].Copy() - else: - rnn_cell_params = p.rnn_cell_tpl.Copy() - rnn_cell_params.dtype = p.dtype - rnn_cell_params.inputs_arity = 2 - decoder_utils.SetRnnCellNodes(p, rnn_cell_params) - rnn_cell_params.num_input_nodes = feat_dim + atten_context_dim - if i == 0: - rnn_cell_params.name = 'rnn_cell' - else: - rnn_cell_params.name = 'rnn_cell_%d' % i - feat_dim = rnn_cell_params.num_output_nodes - params_rnn_cells.append(rnn_cell_params) - if p.adapter_task_id_field is not None: - adapter_p = p.adapter_layer_tpl.Copy() - adapter_p.name = 'adapter_%d' % i - adapter_p.input_dim = feat_dim - params_adapter_layers.append(adapter_p) - self.CreateChildren('rnn_cell', params_rnn_cells) - self.CreateChildren('adapters', params_adapter_layers) - - p.softmax.dtype = p.dtype - p.softmax.input_dim = feat_dim - if p.softmax_uses_attention: - p.softmax.input_dim += atten_context_dim - self.CreateChild('softmax', p.softmax) - - if p.fusion: - p.fusion.base_model_logits_dim = p.softmax.input_dim - self.CreateChild('fusion', p.fusion) - - self._CreateAtten() - - if p.label_smoothing is not None: - p.label_smoothing.name = 'smoother' - if p.label_smoothing.num_classes == 0: - p.label_smoothing.num_classes = p.softmax.num_classes - elif p.label_smoothing.num_classes != p.softmax.num_classes: - raise ValueError('label_smoothing.num_classes ({}) does not match ' - 'softmax.num_classes ({})'.format( - p.label_smoothing.num_classes, - p.softmax.num_classes)) - self.CreateChild('smoother', p.label_smoothing) - - def _CreateAtten(self): - p = self.params - p.attention.dtype = p.dtype - p.attention.source_dim = p.attention.source_dim or p.source_dim - p.attention.query_dim = ( - p.attention.query_dim or self.rnn_cell[0].params.num_output_nodes) - self.CreateChild('atten', p.attention) - - def _GetAttenContextDim(self): - p = self.params - audio_context_dim = ( - p.atten_context_dim if p.atten_context_dim else p.source_dim) - additional_context_dim = self.contextualizer.GetContextDim() - return audio_context_dim + additional_context_dim - - def _ApplyDropout(self, theta, x_in, deterministic=False, extra_seed=None): - p = self.params - assert 0 <= p.dropout_prob and p.dropout_prob < 1.0 - if self.do_eval or p.dropout_prob == 0.0: - return x_in - - if deterministic: - seeds = py_utils.GenerateStepSeedPair(p, theta.global_step) - if extra_seed: - seeds += extra_seed - return py_utils.DeterministicDropout(x_in, 1.0 - p.dropout_prob, seeds) - else: - seed = p.random_seed - if seed and extra_seed: - seed += extra_seed - return tf.nn.dropout(x_in, rate=p.dropout_prob, seed=seed) - - def _InitAttention(self, theta, encoder_outputs): - """Intializes attention and returns a NestedMap with those values.""" - packed_src = self.atten.InitForSourcePacked( - theta.atten, - encoder_outputs.encoded, # src - encoder_outputs.encoded, # context - encoder_outputs.padding) - self.contextualizer.InitAttention(theta.contextualizer, packed_src) - return packed_src - - def _GetEncoderPaddings(self, encoder_outputs): - """Get Encoder Paddings from encoder_outputs.""" - if encoder_outputs and isinstance(encoder_outputs.padding, tf.Tensor): - return encoder_outputs.padding - else: - return None - - def BaseZeroState(self, - theta, - encoder_outputs, - bs, - misc_zero_states, - per_step_source_padding=None): - """Returns initial state of RNNs, and attention.""" - p = self.params - rnn_states = [] - for i in range(p.rnn_layers): - rnn_states.append(self.rnn_cell[i].zero_state(theta.rnn_cell[i], bs)) - - packed_src = self._InitAttention(theta, encoder_outputs) - zero_atten_state = self.atten.ZeroAttentionState( - tf.shape(encoder_outputs.padding)[0], bs) - (atten_context, atten_probs, atten_states) = ( - self.atten.ComputeContextVectorWithSource( - theta.atten, - packed_src, - py_utils.Zeros([bs, self.rnn_cell[0].params.num_output_nodes], - dtype=py_utils.FPropDtype(p)), - zero_atten_state, - per_step_source_padding=per_step_source_padding)) - atten_context = self.contextualizer.ZeroAttention( - theta.contextualizer, bs, misc_zero_states, atten_context, packed_src) - - return rnn_states, atten_context, atten_probs, atten_states, packed_src - - def AddAdditionalDecoderSummaries(self, encoder_outputs, targets, seq_out_tas, - softmax_input): - """Additional model-specific summaries which should be displayed.""" - pass - - def DecoderStepZeroState(self, theta, encoder_outputs, target_ids, bs): - misc_zero_states = self.MiscZeroState(theta, encoder_outputs, target_ids, - bs) - rnn_states, atten_context, atten_probs, atten_states, packed_src = ( - self.BaseZeroState(theta, encoder_outputs, bs, misc_zero_states)) - return py_utils.NestedMap( - rnn_states=rnn_states, - atten_context=atten_context, - atten_probs=atten_probs, - atten_states=atten_states, - fusion_states=self.fusion.zero_state(theta.fusion, bs), - misc_states=misc_zero_states), packed_src - - def _AddDecoderActivationsSummary(self, - encoder_outputs, - targets, - atten_probs, - rnn_outs, - softmax_input, - additional_atten_probs=None, - target_alignments=None): - """Adds summary about decoder activations. - - For each of the args, a TensorArray can also be a Tensor representing - the stacked array. - - Args: - encoder_outputs: a NestedMap computed by encoder. - targets: a NestedMap, usually input_batch.tgt. - atten_probs: a TensorArray of max_target_length elements, each of shape - [batch, max_source_length]. - rnn_outs: a list of TensorArray, one for each RNN layer. Each - TensorArray has max_target_length elements, each of shape [batch, - rnn_output_dim]. - softmax_input: a Tensor of shape [batch, max_target_length, vocab_size]. - additional_atten_probs: an optional list of (name, TensorArray) to display - along with atten_probs. - target_alignments: an optional Tensor of shape [batch, max_target_length] - where every value is an int32 in the range of [1, max_source_length], - representing number of source frames by which a target label should be - emitted. - - Returns: - A finalized figure. - """ - source_encs = encoder_outputs.encoded - source_paddings = encoder_outputs.padding - if not self.cluster.add_summary: - return - - def _ToTensor(t): - return t.stack() if isinstance(t, tf.TensorArray) else t - - atten_probs = _ToTensor(atten_probs) - rnn_outs = [_ToTensor(ta) for ta in rnn_outs] - if additional_atten_probs: - additional_atten_probs = [ - (name, _ToTensor(ta)) for name, ta in additional_atten_probs - ] - - num_cols = 2 + len(rnn_outs) - fig = plot.MatplotlibFigureSummary( - 'decoder_example', - figsize=(2.3 * (3 + num_cols - 1), 6), - max_outputs=1, - subplot_grid_shape=(2, num_cols), - gridspec_kwargs=dict( - width_ratios=[3] + [1] * (num_cols - 1), height_ratios=(4, 1))) - - # Attention needs a custom plot_func to allow for clean y-axis label for - # very long transcripts - def PlotAttention(fig, axes, transcript, atten_probs, title): - plot.AddImage(fig, axes, atten_probs, title=title) - axes.set_ylabel( - plot.ToUnicode(transcript) + '\nOutput token', - size='x-small', - wrap=True, - fontproperties=self._font_properties) - - index = 0 - if 'transcripts' not in targets: - return - transcript = targets.transcripts[:index + 1] - - srclen = tf.cast( - tf.round(tf.reduce_sum(1 - source_paddings[:, index])), tf.int32) - tgtlen = tf.cast( - tf.round(tf.reduce_sum(1 - targets.paddings[index, :])), tf.int32) - - def PlotAttentionForOneExample(atten_probs, - target_fig, - title, - alignments=None): - """Plots attention for one example.""" - tf.logging.info('Plotting attention for %s: %s %s', title, - atten_probs.shape, alignments) - atten_probs = atten_probs[:tgtlen, index, :srclen] - if alignments is not None: - # [tgtlen]. - alignment_positions = alignments[index, :tgtlen] - 1 - # [tgtlen, srclen]. - alignment_probs = tf.one_hot(alignment_positions, depth=srclen, axis=-1) - - # The summary image will use red bars to represent target label - # alignments and purple shades for attention probabilities. - atten_probs = 1 - tf.stack( - [ - atten_probs, - # Overlay atten_probs and alignment_probs on the green channel - # so that colors are visible on a white background. - tf.minimum(atten_probs + alignment_probs, 1.), - alignment_probs - ], - axis=-1) - probs = tf.expand_dims(atten_probs, 0) - target_fig.AddSubplot([transcript, probs], PlotAttention, title=title) - - PlotAttentionForOneExample(atten_probs, fig, title=u'atten_probs') - # rnn_outs and softmax_input have transposed shapes of [tgtlen, dim] - # compared to source_encs [dim, srclen]. - for i in range(len(rnn_outs)): - rnn_out = tf.expand_dims(rnn_outs[i][:tgtlen, index, :], 0) - fig.AddSubplot([rnn_out], title=u'rnn_outs/%d' % i) - fig.AddSubplot( - [softmax_input[:index + 1, :tgtlen, :]], title=u'softmax_input') - source_encs = tf.expand_dims( - tf.transpose(source_encs[:srclen, index, :]), 0) - fig.AddSubplot([source_encs], title=u'source_encs', xlabel=u'Encoder frame') - finalized_fig = fig.Finalize() - - if additional_atten_probs: - all_atten_probs = [('atten_probs', atten_probs)] + additional_atten_probs - num_atten_images = len(all_atten_probs) - atten_fig = plot.MatplotlibFigureSummary( - 'decoder_attention', figsize=(6, 3 * num_atten_images), max_outputs=1) - for key, probs in all_atten_probs: - PlotAttentionForOneExample( - probs, atten_fig, title=key, alignments=target_alignments) - atten_fig.Finalize() - return finalized_fig - - def _ComputeMetrics(self, - logits, - target_labels, - target_weights, - target_probs=None): - """Compute loss and misc metrics. - - Args: - logits: Tensor of shape [batch, time, num_classes]. - target_labels: Tensor of shape [batch, time]. - target_weights: Tensor of shape [batch, time]. - target_probs: Tensor of shape [batch, time, num_classes]. - Returns: - A (metrics, per_sequence_loss) pair. - """ - p = self.params - target_weights_sum = tf.reduce_sum(target_weights) - # add 0.000001 to avoid divide-by-zero. - target_weights_sum_eps = target_weights_sum + 0.000001 - correct_preds = tf.cast( - tf.equal(tf.argmax(logits, 2, output_type=tf.int32), target_labels), - py_utils.FPropDtype(p)) - correct_next_preds = tf.reduce_sum(correct_preds * target_weights) - accuracy = tf.identity( - correct_next_preds / target_weights_sum_eps, - name='fraction_of_correct_next_step_preds') - # Pad zeros so that we can stack them. - per_example_loss = py_utils.SoftmaxCrossEntropyFocalLoss( - logits=logits, - label_ids=target_labels, - label_probs=target_probs, - alpha=p.focal_loss_alpha, - gamma=p.focal_loss_gamma) - - per_sequence_loss = tf.reduce_sum(per_example_loss * target_weights, 1) - per_token_avg_loss = ( - tf.reduce_sum(per_sequence_loss) / target_weights_sum_eps) - if p.token_normalized_per_seq_loss: - per_seq_length = tf.reduce_sum(target_weights, 1) - # +0.001 to avoid possible divide by 0. - per_sequence_loss /= (per_seq_length + 0.001) - if p.per_token_avg_loss: - loss = per_token_avg_loss - loss_weight = target_weights_sum - else: # per-sequence average loss - loss = tf.reduce_mean(per_sequence_loss) - loss_weight = tf.shape(per_sequence_loss)[0] - metrics = { - 'loss': (loss, loss_weight), - # add log_pplx for compatibility with the mt/decoder.py - 'log_pplx': (per_token_avg_loss, target_weights_sum), - 'token_normed_prob': (tf.exp(-per_token_avg_loss), target_weights_sum), - } - metrics['fraction_of_correct_next_step_preds'] = (accuracy, - target_weights_sum) - return metrics, per_sequence_loss - - def InitDecoder(self, theta, encoder_outputs, dec_bs): - decoder_step_zero_state, packed_src = self.DecoderStepZeroState( - theta, encoder_outputs, - tf.ones([dec_bs, 1], dtype=tf.int32) * self.params.target_sos_id, - dec_bs) - - return (decoder_step_zero_state.rnn_states, - decoder_step_zero_state.atten_context, - decoder_step_zero_state.atten_probs, - decoder_step_zero_state.atten_states, - decoder_step_zero_state.fusion_states, - decoder_step_zero_state.misc_states, packed_src) - - def _InitBeamSearchStateCallback(self, theta, encoder_outputs, - num_hyps_per_beam): - raise NotImplementedError('_InitBeamSearchStateCallback') - - def _PreBeamSearchStepCallback(self, theta, encoder_outputs, step_ids, states, - num_hyps_per_beam): - raise NotImplementedError('_PreBeamSearchStepCallback') - - def _PostBeamSearchStepCallback(self, theta, encoder_outputs, new_step_ids, - states): - raise NotImplementedError('_PostBeamSearchStepCallback') - - def ComputeLoss(self, theta, predictions, targets): - """Computes loss metrics and per-sequence losses. - - Args: - theta: A NestedMap object containing weights' values of this - layer and its children layers. - predictions: A NestedMap containing logits (and possibly other fields). - targets: A dict of string to tensors representing the targets one is - trying to predict. Each tensor in targets is of shape [batch, time]. - - Returns: - (metrics, per_sequence_loss), where metrics is a dictionary containing - metrics for the xent loss and prediction accuracy. per_sequence is a - dictionary containing 'loss', a (-log(p)) vector of size [bs]. - """ - p = self.params - with tf.name_scope(p.name): - if 'probs' in targets: - target_probs = targets.probs - elif p.label_smoothing is not None: - target_probs = self.smoother.FProp(theta.smoother, targets.paddings, - targets.labels, targets.ids) - else: - target_probs = None - merged_metrics = {} - merged_per_sequence_loss = 0. - - def AddToMetric(acc, scale, metric): - assert len(acc) == 2 - assert len(metric) == 2 - return (acc[0] + scale * tf.cast(metric[0], py_utils.FPropDtype(p)), - acc[1] + scale * tf.cast(metric[1], py_utils.FPropDtype(p))) - - for logit_name, loss_weight in p.logit_types.items(): - metrics, per_sequence_loss = self._ComputeMetrics( - getattr(predictions, logit_name), targets.labels, targets.weights, - target_probs) - for k, v in metrics.items(): - tf.logging.info('Merging metric %s: %s', k, v) - merged_metrics[k + '/' + logit_name] = v - if k not in merged_metrics: - merged_metrics[k] = (tf.zeros( - shape=[], dtype=py_utils.FPropDtype(p)), - tf.zeros( - shape=[], dtype=py_utils.FPropDtype(p))) - merged_metrics[k] = AddToMetric(merged_metrics[k], loss_weight, v) - merged_per_sequence_loss += loss_weight * per_sequence_loss - return merged_metrics, {'loss': merged_per_sequence_loss} - - def CreateTargetInfoMisc(self, targets): - """Return a NestedMap corresponding to the 'misc' field in TargetInfo.""" - if 'fst_bias_probs' in targets: - return py_utils.NestedMap({ - 'fst_bias_probs': targets.fst_bias_probs, - }) - else: - return py_utils.NestedMap() - - def ComputePredictions(self, theta, encoder_outputs, targets): - """Computes logits. - - Args: - theta: A NestedMap object containing weights values of this layer and its - child layers. - encoder_outputs: a NestedMap computed by encoder. - targets: A dict of string to tensors representing the targets one is - trying to predict. Each tensor in targets is of shape [batch, time]. - - Returns: - A NestedMap object containing logit tensors as values, each of shape - [target_batch, max_target_length, vocab_size]. One of the keys must be - 'logits'. - """ - assert getattr(encoder_outputs, 'src_segment_id', None) is None - p = self.params - self.contextualizer.SetContextMap(targets, theta.contextualizer) - if 'weights' not in targets and 'paddings' in targets: - targets.weights = 1.0 - targets.paddings - if p.use_while_loop_based_unrolling: - predictions = self.ComputePredictionsDynamic(theta, encoder_outputs, - targets) - else: - predictions = self.ComputePredictionsFunctional(theta, encoder_outputs, - targets) - encoder_paddings = self._GetEncoderPaddings(encoder_outputs) - if isinstance(encoder_paddings, tf.Tensor): - # source_padding is of shape [time, batch]. Compute source_enc_len, which - # is used for computing attention loss. - predictions.source_enc_len = tf.reduce_sum(1 - encoder_paddings, axis=0) - if 'paddings' in targets: - source_batch = py_utils.GetShape(encoder_paddings)[1] - target_batch = py_utils.GetShape(targets.paddings)[0] - multiplier = target_batch // source_batch - source_len = py_utils.RepeatDim( - predictions.source_enc_len, multiplier, axis=0) - target_len = tf.reduce_sum(1 - targets.paddings, axis=1) - target_source_length_ratio = target_len / tf.maximum(source_len, 1.0) - summary_utils.scalar('avg_target_source_length_ratio', - tf.reduce_mean(target_source_length_ratio)) - return predictions - - def _GetInitialSeqStateTensorArrays(self, max_seq_length, - decoder_step_state_zero_fusion_flat, - decoder_step_state_zero_misc_flat): - """Get intitial tensor arrays for ComputePredictionsDynamic.""" - p = self.params - # TensorArrays for sequence outputs. - return AsrDecoder.SequenceOutTensorArrays( - rnn_outs=[ - _NewTensorArray( - name='rnn%d_outs' % i, - max_seq_length=max_seq_length, - dtype=py_utils.FPropDtype(p)) for i in range(p.rnn_layers) - ], - step_outs=_NewTensorArray( - name='step_outs', - max_seq_length=max_seq_length, - dtype=py_utils.FPropDtype(p)), - atten_probs=_NewTensorArray( - name='atten_probs', - max_seq_length=max_seq_length, - dtype=py_utils.FPropDtype(p)), - logits=_NewTensorArray( - name='logits', - max_seq_length=max_seq_length, - dtype=py_utils.FPropDtype(p)), - fusion=[ - _NewTensorArray( - name='fusion_states%d' % i, - max_seq_length=max_seq_length, - dtype=decoder_step_state_zero_fusion_flat[i].dtype) - for i in range(len(decoder_step_state_zero_fusion_flat)) - ], - misc=[ - _NewTensorArray( - name='misc_states%d' % i, - max_seq_length=max_seq_length, - dtype=decoder_step_state_zero_misc_flat[i].dtype) - for i in range(len(decoder_step_state_zero_misc_flat)) - ]) - - def _GetNewAttenProbs(self, seq_out_tas, time, decoder_step_state): - """Update atten probs for a timestep and return the updated tensor array.""" - return seq_out_tas.atten_probs.write(time, decoder_step_state.atten_probs) - - def _UpdateSequenceOutTensorArrays(self, decoder_step_state, time, step_outs, - seq_out_tas): - """Update SequenceOutTensorArrays at each time step.""" - new_rnn_outs = [] - assert len(seq_out_tas.rnn_outs) == len(decoder_step_state.rnn_states) - for i in range(len(seq_out_tas.rnn_outs)): - new_rnn_outs.append(seq_out_tas.rnn_outs[i].write( - time, decoder_step_state.rnn_states[i].m)) - new_logits_ta = seq_out_tas.logits.write(time, decoder_step_state.logits) - new_step_outs_ta = seq_out_tas.step_outs.write(time, step_outs) - new_atten_probs_ta = self._GetNewAttenProbs(seq_out_tas, time, - decoder_step_state) - new_seq_outs_fusion_states = [] - new_fusion_states_flat = decoder_step_state.fusion_states.Flatten() - for i in range(len(new_fusion_states_flat)): - new_seq_outs_fusion_states.append(seq_out_tas.fusion[i].write( - time, new_fusion_states_flat[i])) - new_seq_outs_misc_states = [] - new_misc_states_flat = decoder_step_state.misc_states.Flatten() - for i in range(len(new_misc_states_flat)): - new_seq_outs_misc_states.append(seq_out_tas.misc[i].write( - time, new_misc_states_flat[i])) - - return AsrDecoder.SequenceOutTensorArrays( - rnn_outs=new_rnn_outs, - step_outs=new_step_outs_ta, - atten_probs=new_atten_probs_ta, - logits=new_logits_ta, - fusion=new_seq_outs_fusion_states, - misc=new_seq_outs_misc_states) - - def _GetAttenProbsFromSequenceOutTensorArrays(self, atten_probs): - return tf.transpose(atten_probs.stack(), [1, 0, 2]) - - def _GetPredictionFromSequenceOutTensorArrays(self, seq_out_tas): - return py_utils.NestedMap( - # softmax_input is of shape [time, batch, dim] for compatibility. - softmax_input=seq_out_tas.step_outs.stack(), - # logits is of shape [batch, time, dim]. - logits=tf.transpose(seq_out_tas.logits.stack(), [1, 0, 2]), - attention=py_utils.NestedMap( - probs=self._GetAttenProbsFromSequenceOutTensorArrays( - seq_out_tas.atten_probs))) - - def _GetInitialTargetInfo(self, targets, max_seq_length, target_embs): - return AsrDecoderBase.TargetInfo( - id=_ToTensorArray( - 'target_ids_ta', - tf.transpose(targets.ids), - max_seq_length, - clear_after_read=False), - label=_ToTensorArray( - 'target_labels_ta', - tf.transpose(targets.labels), - max_seq_length, - clear_after_read=False), - weight=_ToTensorArray('target_weights_ta', - tf.transpose(targets.weights), max_seq_length), - emb=_ToTensorArray('target_embs_ta', tf.transpose( - target_embs, [1, 0, 2]), max_seq_length), - padding=_ToTensorArray( - 'target_paddings_ta', - tf.expand_dims(tf.transpose(targets.paddings), -1), max_seq_length), - misc=self.CreateTargetInfoMisc(targets), - ) - - def ComputePredictionsDynamic(self, theta, encoder_outputs, targets): - p = self.params - with tf.name_scope(p.name): - # Create TensorArrays corresponding to the targets to be used for - # training. - dec_bs = tf.shape(targets.ids)[0] - max_seq_length = tf.shape(targets.ids)[1] - - target_embs = self.emb.EmbLookup(theta.emb, tf.reshape(targets.ids, [-1])) - target_embs = tf.reshape(target_embs, [dec_bs, max_seq_length, p.emb_dim]) - target_embs = self._ApplyDropout(theta, target_embs) - target_info_tas = self._GetInitialTargetInfo(targets, max_seq_length, - target_embs) - - # Initialize all loop variables. - time = tf.constant(0, tf.int32) - # Decoder state. - decoder_step_state_zero, packed_src = self.DecoderStepZeroState( - theta, encoder_outputs, targets.ids, dec_bs) - decoder_step_state_zero_fusion_flat = ( - decoder_step_state_zero.fusion_states.Flatten()) - decoder_step_state_zero_misc_flat = ( - decoder_step_state_zero.misc_states.Flatten()) - - # TensorArrays for sequence outputs. - seq_out_tas = self._GetInitialSeqStateTensorArrays( - max_seq_length, decoder_step_state_zero_fusion_flat, - decoder_step_state_zero_misc_flat) - - def _LoopContinue(time, decoder_step_state, target_info_tas, seq_out_tas): - del decoder_step_state, target_info_tas, seq_out_tas - return time < max_seq_length - - def _LoopBody(time, old_decoder_step_state, target_info_tas, seq_out_tas): - """Computes decoder outputs and updates decoder_step_state.""" - cur_target_info = self.TargetsToBeFedAtCurrentDecodeStep( - time, theta, old_decoder_step_state, target_info_tas, seq_out_tas) - - step_outs, decoder_step_state = self.SingleDecodeStep( - theta, packed_src, cur_target_info, old_decoder_step_state) - - step_outs, decoder_step_state.fusion_states = self.fusion.FProp( - theta.fusion, old_decoder_step_state.fusion_states, step_outs, - cur_target_info.id, cur_target_info.padding) - - # Compute logits. - xent_loss = self.softmax.FProp( - theta.softmax, [step_outs], - class_weights=cur_target_info.weight, - class_ids=cur_target_info.label) - - decoder_step_state = self.PostStepDecoderStateUpdate( - decoder_step_state, xent_loss.logits) - - decoder_step_state.logits = self.fusion.ComputeLogitsWithLM( - decoder_step_state.fusion_states, decoder_step_state.logits) - - # Update SequenceOutTensorArrays. - new_seq_out_tas = self._UpdateSequenceOutTensorArrays( - decoder_step_state, time, step_outs, seq_out_tas) - del decoder_step_state.logits - return (time + 1, decoder_step_state, target_info_tas, new_seq_out_tas) - - loop_vars = time, decoder_step_state_zero, target_info_tas, seq_out_tas - # NOTE(skyewm): this could be more specific, but for now don't verify - # while_loop input/output shapes at all. - shape_invariants = tf.nest.map_structure(lambda t: tf.TensorShape(None), - loop_vars) - - (time, _, target_info_tas, seq_out_tas) = tf.while_loop( - _LoopContinue, - _LoopBody, - loop_vars=loop_vars, - shape_invariants=shape_invariants, - parallel_iterations=p.parallel_iterations, - swap_memory=False) - - softmax_input = seq_out_tas.step_outs.stack() - softmax_input = tf.transpose(softmax_input, [1, 0, 2]) - self._AddDecoderActivationsSummary(encoder_outputs, targets, - seq_out_tas.atten_probs, - seq_out_tas.rnn_outs, softmax_input) - self.AddAdditionalDecoderSummaries(encoder_outputs, targets, seq_out_tas, - softmax_input) - return self._GetPredictionFromSequenceOutTensorArrays(seq_out_tas) - - def ComputePredictionsFunctional(self, theta, encoder_outputs, targets): - p = self.params - # Currently, scheduled sampling is not supported. - assert p.min_ground_truth_prob == 1.0 - with tf.name_scope(p.name): - dec_bs = tf.shape(targets.ids)[0] - - # Decoder state. - state0, packed_src = self.DecoderStepZeroState(theta, encoder_outputs, - targets.ids, dec_bs) - - atten_context_dim = self._GetAttenContextDim() - rnn_output_dim = self.rnn_cell[-1].params.num_output_nodes - out_dim = rnn_output_dim + atten_context_dim - state0.step_outs = py_utils.Zeros([dec_bs, out_dim], - dtype=py_utils.FPropDtype(p)) - target_embs = self.emb.EmbLookup(theta.emb, targets.ids) - target_embs = self._ApplyDropout(theta, target_embs) - inputs = py_utils.NestedMap( - id=tf.transpose(targets.ids), - label=tf.transpose(targets.labels), - weight=tf.transpose(targets.weights), - emb=tf.transpose(target_embs, [1, 0, 2]), - padding=tf.expand_dims(tf.transpose(targets.paddings), -1), - misc=self.CreateTargetInfoMisc(targets), - ) - - # If the theta in the recurrent loop contains fusion related variables, - # it will allocate a large amount of memory even though it is not being - # used and exceed current TPU HBM limit. Thus remove fusion theta from - # the recurrent loop, and performs fusion outside the recurrent loop. - theta_no_fusion = theta.copy() - del theta_no_fusion.fusion - recurrent_theta = py_utils.NestedMap( - theta=theta_no_fusion, packed_src=packed_src) - state0_no_fusion = state0.copy() - del state0_no_fusion.fusion_states - - def RnnStep(recurrent_theta, state0, inputs): - """Computes one rnn step.""" - with tf.name_scope('single_decode_step'): - step_outs, state1 = self.SingleDecodeStep( - recurrent_theta.theta, - recurrent_theta.packed_src, - inputs, - state0, - use_deterministic_random=True) - state1.step_outs = step_outs - # TODO(syzhang, tsainath): Add SS into Functional Decoder, which - # requires computing softmax logits. - state1 = self.PostStepDecoderStateUpdate(state1, inputs.label) - return state1, py_utils.NestedMap() - - accumulated_states, _ = recurrent.Recurrent( - recurrent_theta, state0_no_fusion, inputs, RnnStep) - - if not p.softmax_uses_attention: - step_out, _ = tf.split( - accumulated_states.step_outs, [rnn_output_dim, atten_context_dim], - axis=-1) - else: - step_out = accumulated_states.step_outs - softmax_input, state0.fusion_states = self.fusion.FProp( - theta.fusion, state0.fusion_states, step_out, inputs.id, - inputs.padding, inputs.misc) - # TODO(syzhang): understand why we have to construct softmax outside the - # recurrent loop; otherwise, the BProp numbers don't match. - seq_logits = self._ComputeLogits(theta, softmax_input) - # TODO(syzhang): supports AddAdditionalDecoderSummaries(). - atten_states = accumulated_states.atten_states - if isinstance(atten_states, py_utils.NestedMap): - additional_atten_probs = sorted( - [(name, tensor) - for name, tensor in atten_states.FlattenItems() - if name.endswith('probs')]) - else: - additional_atten_probs = [] - rnn_outs = [ - cell.GetOutput(accumulated_states.rnn_states[i]) - for i, cell in enumerate(self.rnn_cell) - ] - self._AddDecoderActivationsSummary( - encoder_outputs, - targets, - accumulated_states.atten_probs, - rnn_outs, - softmax_input, - additional_atten_probs=additional_atten_probs, - target_alignments=getattr(targets, 'alignments', None)) - # seq_logits: [time, batch, num_classes]. - adjusted_logits = self.fusion.ComputeLogitsWithLM(state0.fusion_states, - seq_logits) - predictions = py_utils.NestedMap( - # Transpose to [batch, time, num_classes]. - logits_without_bias=tf.transpose(seq_logits, [1, 0, 2]), - logits=tf.transpose(adjusted_logits, [1, 0, 2]), - # softmax_input is of shape [time, batch, dim] for compatibility. - softmax_input=softmax_input) - attention_map = py_utils.NestedMap(probs=accumulated_states.atten_probs) - for k, v in additional_atten_probs: - attention_map[k] = v - # Transpose attention probs from [target_length, batch, source_length] to - # [batch, target_length, source_length]. - predictions.attention = attention_map.Transform( - lambda x: tf.transpose(x, [1, 0, 2])) - return predictions - - def _ComputeLogits(self, theta, softmax_input): - if isinstance(self.softmax, layers.ConvSoftmax): - return self.softmax.Logits(theta.softmax, softmax_input) - else: - # SoftmaxLayer.Logits() may not support 3-D inputs. So use FProp() with - # fake labels. - xent_loss = self.softmax.FProp( - theta.softmax, [softmax_input], - class_weights=tf.ones( - shape=tf.shape(softmax_input)[:-1], dtype=softmax_input.dtype), - class_ids=tf.ones(shape=tf.shape(softmax_input)[:-1], dtype=tf.int32)) - return xent_loss.logits - - def SingleDecodeStep(self, - theta, - packed_src, - cur_target_info, - decoder_step_state, - per_step_src_padding=None, - use_deterministic_random=False): - """Computes one 'step' of computation for the decoder. - - Must be implemented by sub-classes. Residual connections must also be taken - care of in sub-classes. - - Args: - theta: A NestedMap object containing weights' values of this - layer and its children layers. - packed_src: A NestedMap to represent the packed source tensors generated - by the attention model. - cur_target_info: TargetInfo namedtuple, which represents the targets - which represents information about the target at this step. It is up - to the various sub-classes to determine how to process the current - target. - decoder_step_state: DecoderStepState which encapsulates the state of the - decoder before computing outputs at the current step. - per_step_src_padding: Optional padding to be applied to the source_encs - which overrides the default padding in source_paddings. Used, for - example, by the Neural Transducer (NT) decoder. - use_deterministic_random: whether to use deterministic random numbers when - needed. Must be set to True if called from functional recurrent. - - Returns: - A tuple (step_out, new_decoder_state) which represent the outputs of the - decoder (usually logits), and the new decoder state after processing the - current step. - """ - # TODO(syzhang): unify the API to always pass in packed_src. - raise NotImplementedError('Must be implemented by sub-classes.') - - def MiscZeroState(self, theta, encoder_outputs, target_ids, bs): - """Returns initial state for other miscellaneous states, if any.""" - misc_zero_state = py_utils.NestedMap() - p = self.params - if self._max_label_prob > 0: - misc_zero_state.prev_predicted_ids = tf.reshape(target_ids[:, 0], [bs]) - step = tf.cast(theta.global_step, tf.float32) - sampling_p = (step - p.prob_decay_start_step) / self._decay_interval - groundtruth_p = 1 - (self._max_label_prob * sampling_p) - groundtruth_p = tf.maximum(groundtruth_p, p.min_ground_truth_prob) - groundtruth_p = tf.minimum(groundtruth_p, 1.0) - summary_utils.scalar('ground_truth_sampling_probability', groundtruth_p) - misc_zero_state.groundtruth_p = groundtruth_p - if p.adapter_task_id_field: - # encoder_outputs.encoded: [time, batch, dim] - source_bs = py_utils.GetShape(encoder_outputs.encoded, 2)[1] - # Only task_ids of shape [batch] is supported - task_ids = tf.reshape( - encoder_outputs.Get(p.adapter_task_id_field), [source_bs]) - multiplier = bs // source_bs - task_ids = tf.tile(task_ids, [multiplier]) - misc_zero_state.Set(p.adapter_task_id_field, task_ids) - return misc_zero_state - - def TargetsToBeFedAtCurrentDecodeStep(self, time, theta, decoder_step_state, - target_info_tas, seq_out_tas): - del seq_out_tas - - target_id = target_info_tas.id.read(time) - label = target_info_tas.label.read(time) - weight = tf.squeeze(target_info_tas.weight.read(time)) - emb = target_info_tas.emb.read(time) - padding = target_info_tas.padding.read(time) - misc = py_utils.NestedMap() - - # Use different id and embedding for scheduled sampling. - if self._max_label_prob > 0: - dec_bs = tf.shape(decoder_step_state.misc_states.prev_predicted_ids)[0] - pick_groundtruth = tf.less( - tf.random.uniform([dec_bs], seed=self.params.random_seed), - decoder_step_state.misc_states.groundtruth_p) - emb = tf.where( - pick_groundtruth, target_info_tas.emb.read(time), - self.emb.EmbLookup( - theta.emb, - tf.stop_gradient( - decoder_step_state.misc_states.prev_predicted_ids))) - target_id = tf.where(pick_groundtruth, target_info_tas.id.read(time), - decoder_step_state.misc_states.prev_predicted_ids) - return AsrDecoderBase.TargetInfo( - id=target_id, - label=label, - weight=weight, - emb=emb, - padding=padding, - misc=misc) - - def PostStepDecoderStateUpdate(self, decoder_step_state, logits=None): - """Update decoder states and logits after SingleDecodeStep. - - Args: - decoder_step_state: A NestedMap object which encapsulates decoder states. - logits: a tensor, predicted logits. - - Returns: - decoder_step_state. - - Raises: - ValueError: if scheduled sampling is used for functional unrolling or - if logits is None for while loop based unrolling. - """ - if not self.params.use_while_loop_based_unrolling: - if self.params.min_ground_truth_prob < 1.0: - raise ValueError('SS is not yet supported') - else: - if logits is None: - raise ValueError('logits cannot be None') - decoder_step_state.logits = logits - - if self._max_label_prob > 0: - bs = tf.shape(logits)[0] - # log_probs: [bs, num_classes] - log_probs = tf.nn.log_softmax(logits) - # log_prob_sample: [bs, 1] - log_prob_sample = tf.multinomial( - log_probs, 1, seed=self.params.random_seed) - # pred_ids: [bs] - pred_ids = tf.reshape(tf.cast(log_prob_sample, tf.int32), [bs]) - decoder_step_state.misc_states.prev_predicted_ids = pred_ids - return decoder_step_state - - -class AsrDecoder(AsrDecoderBase): - """Step-by-step decoder with LM fusion.""" - - @classmethod - def Params(cls): - p = super().Params() - return p - - def AddAdditionalDecoderSummaries(self, encoder_outputs, targets, seq_out_tas, - softmax_input): - """Add summaries not covered by the default activations summaries. - - Args: - encoder_outputs: a NestedMap computed by encoder. - targets: a NestedMap containing target info. - seq_out_tas: a SequenceOutTensorArrays. - softmax_input: a tensor of shape [batch, time, vocab_size]. - """ - if cluster_factory.Current().add_summary: - self.fusion.AddAdditionalDecoderSummaries( - encoder_outputs.encoded, encoder_outputs.padding, targets, - seq_out_tas, softmax_input) - - def _ComputeAttention(self, - theta, - rnn_out, - packed_src, - attention_state, - per_step_src_padding=None, - query_segment_id=None): - """Runs attention and computes context vector. - - Can be overridden by a child class if attention is computed differently. - - Args: - theta: A NestedMap object containing weights for the attention layers. - Expects a member named 'atten'. - rnn_out: A Tensor of shape [batch_size, query_dim]; output of the - first layer of decoder RNN, which is the query vector used for - attention. - packed_src: A NestedMap returned by self.atten.InitForSourcePacked. - attention_state: The attention state computed at the previous timestep. - Varies with the type of attention, but is usually a Tensor or a - NestedMap of Tensors of shape [batch_size, ]. - per_step_src_padding: Source sequence padding to apply at this step. - query_segment_id: a tensor of shape [batch_size]. - - Returns: - A tuple of 3 tensors: - - - The attention context vector: shaped [batch_size, context_dim]. - - The attention probability vector: shaped [batch_size, seq_len] - - The attention state: A Tensor or a NestedMap of Tensors of shape - [batch_size, ]. - """ - return self.atten.ComputeContextVectorWithSource( - theta.atten, - packed_src, - rnn_out, - attention_state=attention_state, - per_step_source_padding=per_step_src_padding, - query_segment_id=query_segment_id) - - def SingleDecodeStep(self, - theta, - packed_src, - cur_target_info, - decoder_step_state, - per_step_src_padding=None, - use_deterministic_random=False): - """Decode one step. - - Note that the implementation of attention here follows the model in - https://arxiv.org/pdf/1609.08144.pdf, detailed more in - https://arxiv.org/pdf/1703.08581.pdf. - - Args: - theta: A NestedMap object containing weights' values of this - layer and its children layers. - packed_src: A NestedMap to represent the packed source tensors generated - by the attention model. - cur_target_info: TargetInfo namedtuple, which represents the targets - which represents information about the target at this step. It is up - to the various sub-classes to determine how to process the current - target. - decoder_step_state: DecoderStepState which encapsulates the state of the - decoder before computing outputs at the current step. - per_step_src_padding: Optional padding to be applied to the source_encs - which overrides the default padding in source_paddings. Used, for - example, by the Neural Transducer (NT) decoder. - use_deterministic_random: whether to use deterministic random numbers when - needed. Must be set to True if called from functional recurrent. - - Returns: - A tuple (step_out, new_decoder_state) which represent the outputs of the - decoder (usually logits), and the new decoder state after processing the - current step. - """ - p = self.params - misc_states = decoder_step_state.misc_states - new_rnn_states = [] - new_rnn_states_0, _ = self.rnn_cell[0].FProp( - theta.rnn_cell[0], decoder_step_state.rnn_states[0], - py_utils.NestedMap( - act=[cur_target_info.emb, decoder_step_state.atten_context], - padding=cur_target_info.padding)) - new_rnn_states.append(new_rnn_states_0) - rnn_out = self.rnn_cell[0].GetOutput(new_rnn_states_0) - if p.adapter_task_id_field: - # rnn_out is [batch, dim], adapter layers requires [time, batch, dim] - rnn_out = self.adapters[0].FProp(theta.adapters[0], - tf.expand_dims(rnn_out, [0]), - misc_states.Get(p.adapter_task_id_field)) - rnn_out = tf.squeeze(rnn_out, [0]) - - (new_atten_context, new_atten_probs, - new_atten_states) = self._ComputeAttention( - theta, - rnn_out, - packed_src, - decoder_step_state.atten_states, - per_step_src_padding=per_step_src_padding) - # Here the attention context is being updated according to the - # contextualizer (the default contextualizer is a no-op). - new_atten_context = self.contextualizer.QueryAttention( - theta.contextualizer, rnn_out, misc_states, new_atten_context, - packed_src) - for i, cell in enumerate(self.rnn_cell[1:], 1): - new_rnn_states_i, _ = cell.FProp( - theta.rnn_cell[i], decoder_step_state.rnn_states[i], - py_utils.NestedMap( - act=[rnn_out, new_atten_context], - padding=cur_target_info.padding)) - new_rnn_states.append(new_rnn_states_i) - new_rnn_out = cell.GetOutput(new_rnn_states_i) - if p.adapter_task_id_field: - new_rnn_out = self.adapters[i].FProp( - theta.adapters[i], tf.expand_dims(new_rnn_out, [0]), - misc_states.Get(p.adapter_task_id_field)) - new_rnn_out = tf.squeeze(new_rnn_out, [0]) - new_rnn_out = self._ApplyDropout( - theta, - new_rnn_out, - deterministic=use_deterministic_random, - extra_seed=i * 1000) - if i + 1 >= self.params.residual_start > 0: - rnn_out += new_rnn_out - else: - rnn_out = new_rnn_out - - step_out = tf.concat([rnn_out, new_atten_context], 1) - - return step_out, py_utils.NestedMap( - rnn_states=new_rnn_states, - atten_context=new_atten_context, - atten_probs=new_atten_probs, - atten_states=new_atten_states, - misc_states=misc_states) - - def _GetNumHypsForBeamSearch(self, source_encs, num_hyps_per_beam): - """Returns number of hypothesis times batch_size. - - This function can be overridden by a child class if the total number of - hyps are to be computed in a different way, e.g., when the format of inputs - change. - Args: - source_encs: A Tensor of [time, batch, dim] with source encodings. - num_hyps_per_beam: Int, the number of hypothesis per example in the beam. - Returns: - A Tensor with value batch * num_hyps_per_beam. - """ - return tf.shape(source_encs)[1] * num_hyps_per_beam - - def _PostProcessAttenProbsForBeamSearch(self, atten_probs): - """Returns the attention probabilities after optional post processing. - - This is a noop for the base class. But this function can be overridden - by a child class, e.g., when the format of probabilities change. - Args: - atten_probs: A Tensor of [batch, source_len] dimension with atten probs. - Returns: - A Tensor with processed atten_probs. The same as input in this case. - """ - return atten_probs - - def _InitBeamSearchStateCallback(self, theta, encoder_outputs, - num_hyps_per_beam): - p = self.params - num_hyps = self._GetNumHypsForBeamSearch(encoder_outputs.encoded, - num_hyps_per_beam) - (rnn_states, atten_context, atten_probs, atten_states, - fusion_states, misc_states, packed_src) = self.InitDecoder( - theta, encoder_outputs, num_hyps) - # Throw away packed_src. We re-compute it in _PreBeamSearchStepCallback - # because we cannot pass 'packed_src' through 'states'. beam_search_helper - # assumes that all Tensors in 'states' have 'target_batch' as the first - # dimension. - del packed_src - atten_probs = self._PostProcessAttenProbsForBeamSearch(atten_probs) - all_atten_states = py_utils.NestedMap({ - 'atten_context': atten_context, - 'atten_probs': atten_probs, - 'atten_states': atten_states, - }) - - initial_results = py_utils.NestedMap({ - 'log_probs': - tf.nn.log_softmax( - tf.zeros([num_hyps, p.softmax.num_classes], - dtype=py_utils.FPropDtype(p))), - 'atten_probs': - atten_probs, - }) - other_states = py_utils.NestedMap({ - 'time_step': tf.constant(0), - 'rnn_states': rnn_states, - 'all_atten_states': all_atten_states, - 'fusion_states': fusion_states, - 'misc_states': misc_states, - }) - return initial_results, other_states - - def _PreBeamSearchStepCallback(self, theta, encoder_outputs, step_ids, states, - num_hyps_per_beam): - p = self.params - step_paddings = tf.zeros(tf.shape(step_ids), dtype=p.dtype) - embs = self.emb.EmbLookup(theta.emb, tf.reshape(step_ids, [-1])) - prev_rnn_states = states.rnn_states - prev_atten_states = states.all_atten_states.atten_states - prev_atten_context = states.all_atten_states.atten_context - prev_atten_probs = states.all_atten_states.atten_probs - prev_fusion_states = states.fusion_states - prev_misc_states = states.misc_states - - prev_decoder_step_state = py_utils.NestedMap( - rnn_states=prev_rnn_states, - atten_context=prev_atten_context, - atten_probs=prev_atten_probs, - atten_states=prev_atten_states, - misc_states=prev_misc_states) - # TODO(prabhavalkar): Must handle CreateMiscTargetInfo during beam search - # eval. - cur_target_info = AsrDecoderBase.TargetInfo( - id=tf.reshape(step_ids, [-1]), - label=None, - weight=None, - emb=embs, - padding=step_paddings, - misc=py_utils.NestedMap()) - - packed_src = self._InitAttention(theta, encoder_outputs) - step_out, new_decoder_step_state = self.SingleDecodeStep( - theta, - packed_src, - cur_target_info=cur_target_info, - decoder_step_state=prev_decoder_step_state) - (atten_context, atten_probs, rnn_states, atten_states, - misc_states) = (new_decoder_step_state.atten_context, - new_decoder_step_state.atten_probs, - new_decoder_step_state.rnn_states, - new_decoder_step_state.atten_states, - new_decoder_step_state.misc_states) - - if p.softmax_uses_attention: - # [batch, dims] - softmax_input = step_out - else: - # Strip the attention context from the last dimension of softmax_input. - # TODO(prabhavalkar): This currently assumes that the context is appended - # to the end, see tf.concat in - # AsrDecoderBase.ComputePredictionsFunctional().RnnStep(). Refactor the - # code so as to remove this assumption. - atten_context_dim = self._GetAttenContextDim() - rnn_output_dim = self.rnn_cell[-1].params.num_output_nodes - softmax_input, _ = tf.split( - step_out, [rnn_output_dim, atten_context_dim], axis=-1) - - softmax_input, fusion_states = self.fusion.FProp( - theta.fusion, prev_fusion_states, softmax_input, cur_target_info.id, - cur_target_info.padding) - - logits = self._ComputeLogits(theta, softmax_input) - logits = self.fusion.ComputeLogitsWithLM( - fusion_states, logits, is_eval=True) - if p.use_unnormalized_logits_as_log_probs: - log_probs = logits - else: - log_probs = tf.nn.log_softmax(logits) - - atten_probs = self._PostProcessAttenProbsForBeamSearch(atten_probs) - bs_results = py_utils.NestedMap({ - 'atten_probs': atten_probs, - 'log_probs': log_probs, - }) - all_atten_states = py_utils.NestedMap({ - 'atten_context': atten_context, - 'atten_probs': atten_probs, - 'atten_states': atten_states, - }) - new_states = py_utils.NestedMap({ - 'time_step': states.time_step + 1, - 'rnn_states': rnn_states, - 'all_atten_states': all_atten_states, - 'fusion_states': fusion_states, - 'misc_states': misc_states, - }) - return bs_results, new_states - - def _PostBeamSearchStepCallback(self, theta, encoder_outputs, new_step_ids, - states): - del encoder_outputs, new_step_ids - return states diff --git a/lingvo/tasks/asr/decoder_test.py b/lingvo/tasks/asr/decoder_test.py deleted file mode 100644 index 9460a238..00000000 --- a/lingvo/tasks/asr/decoder_test.py +++ /dev/null @@ -1,765 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for speech decoder.""" - -import lingvo.compat as tf -from lingvo.core import cluster_factory -from lingvo.core import layers as lingvo_layers -from lingvo.core import py_utils -from lingvo.core import symbolic -from lingvo.core import test_utils -from lingvo.core.ops.hyps_pb2 import Hypothesis -from lingvo.tasks.asr import decoder -import numpy as np - -from google.protobuf import text_format - -FLAGS = tf.flags.FLAGS - - -class DecoderTest(test_utils.TestCase): - - def _DecoderParams(self, - vn_config, - num_classes=32, - num_rnn_layers=1): - """Create a small decoder for testing.""" - p = decoder.AsrDecoder.Params() - p.random_seed = 12345 - - p.name = 'decoder' - uniform_init = py_utils.WeightInit.Uniform(0.1, seed=12345) - - # Set up embedding params. - p.emb.vocab_size = num_classes - p.emb.max_num_shards = 1 - p.emb.params_init = uniform_init - - # Set up decoder RNN layers. - p.rnn_layers = num_rnn_layers - rnn_params = p.rnn_cell_tpl - rnn_params.params_init = uniform_init - - # Set up attention. - p.attention.hidden_dim = 16 - p.attention.params_init = uniform_init - - # Set up final softmax layer. - p.softmax.num_classes = num_classes - p.softmax.params_init = uniform_init - - # Set up variational noise params. - p.vn = vn_config - p.vn.scale = tf.constant(0.1) - - p.target_seq_len = 5 - p.source_dim = 8 - p.emb_dim = 2 - p.rnn_cell_dim = 4 - - return p - - def _getDecoderFPropMetrics(self, params): - """Creates decoder from params and computes metrics with random inputs.""" - dec = params.Instantiate() - src_seq_len = 5 - src_enc = tf.random.normal([src_seq_len, 2, 8], - seed=982774838, - dtype=py_utils.FPropDtype(params)) - src_enc_padding = tf.constant( - [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 1.0], [1.0, 1.0]], - dtype=py_utils.FPropDtype(params)) - encoder_outputs = py_utils.NestedMap( - encoded=src_enc, padding=src_enc_padding) - # shape=[4, 5] - target_ids = tf.transpose( - tf.constant([[0, 1, 2, 3], [1, 2, 3, 4], [10, 11, 12, 15], [5, 6, 7, 8], - [10, 5, 2, 5]], - dtype=tf.int32)) - # shape=[4, 5] - target_labels = tf.transpose( - tf.constant([[0, 1, 2, 3], [1, 2, 3, 4], [10, 11, 12, 13], - [5, 7, 8, 10], [10, 5, 2, 4]], - dtype=tf.int32)) - # shape=[4, 5] - target_paddings = tf.transpose( - tf.constant([[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 1, 0], [0, 1, 0, 0], - [1, 1, 1, 0]], - dtype=py_utils.FPropDtype(params))) - target_transcripts = tf.constant(['abcd', 'bcde', 'klmp', 'fghi', 'kfcf']) - target_weights = 1.0 - target_paddings - # ids/labels/weights/paddings are all in [batch, time] shape. - targets = py_utils.NestedMap({ - 'ids': target_ids, - 'labels': target_labels, - 'weights': target_weights, - 'paddings': target_paddings, - 'transcripts': target_transcripts, - }) - decoder_outputs = dec.FPropDefaultTheta(encoder_outputs, targets) - return decoder_outputs.metrics, decoder_outputs.per_sequence['loss'] - - def _testDecoderFPropHelper(self, params): - metrics, per_sequence_loss = self._getDecoderFPropMetrics(params) - return metrics['loss'], per_sequence_loss - - def _testDecoderFPropFloatHelper(self, - func_inline=False, - num_decoder_layers=1, - target_seq_len=5, - residual_start=0): - """Computes decoder from params and computes loss with random inputs.""" - cluster = cluster_factory.ForTestingWorker(add_summary=True) - config = tf.config_pb2.ConfigProto( - graph_options=tf.GraphOptions( - optimizer_options=tf.OptimizerOptions( - do_function_inlining=func_inline))) - with cluster, self.session(use_gpu=False, config=config): - tf.random.set_seed(8372749040) - vn_config = py_utils.VariationalNoiseParams(None, False, False) - p = self._DecoderParams(vn_config) - p.rnn_layers = num_decoder_layers - p.residual_start = residual_start - p.target_seq_len = target_seq_len - dec = p.Instantiate() - src_seq_len = 5 - src_enc = tf.random.normal([src_seq_len, 2, 8], seed=9283748) - src_enc_padding = tf.constant( - [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 1.0], [1.0, 1.0]], - dtype=tf.float32) - encoder_outputs = py_utils.NestedMap( - encoded=src_enc, padding=src_enc_padding) - target_ids = tf.transpose( - tf.constant([[0, 1, 2, 3], [1, 2, 3, 4], [10, 11, 12, 15], - [5, 6, 7, 8], [10, 5, 2, 5]], - dtype=tf.int32)) - target_labels = tf.transpose( - tf.constant([[0, 1, 2, 3], [1, 2, 3, 4], [10, 11, 12, 13], - [5, 7, 8, 10], [10, 5, 2, 4]], - dtype=tf.int32)) - target_paddings = tf.transpose( - tf.constant([[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 1, 0], [0, 1, 0, 0], - [1, 1, 1, 1]], - dtype=tf.float32)) - target_transcripts = tf.constant(['abcd', 'bcde', 'klmp', 'fghi', 'kfcf']) - target_weights = 1.0 - target_paddings - targets = py_utils.NestedMap({ - 'ids': target_ids, - 'labels': target_labels, - 'weights': target_weights, - 'paddings': target_paddings, - 'transcripts': target_transcripts, - }) - metrics = dec.FPropDefaultTheta(encoder_outputs, targets).metrics - loss = metrics['loss'][0] - correct_predicts = metrics['fraction_of_correct_next_step_preds'][0] - summaries = tf.summary.merge(tf.get_collection(tf.GraphKeys.SUMMARIES)) - - self.evaluate(tf.global_variables_initializer()) - loss_v, _ = self.evaluate([loss, correct_predicts]) - - summaries.eval() - - return loss_v - - # Actual tests follow. - - def testDecoderConstruction(self): - """Test that decoder can be constructed from params.""" - p = self._DecoderParams( - vn_config=py_utils.VariationalNoiseParams( - None, True, False, seed=12345)) - _ = decoder.AsrDecoder(p) - - def testDecoderFProp(self): - """Create decoder with default params, and verify that FProp runs.""" - with self.session(use_gpu=False): - tf.random.set_seed(8372749040) - - p = self._DecoderParams( - vn_config=py_utils.VariationalNoiseParams( - None, True, False, seed=12345)) - - metrics, per_sequence_loss = self._getDecoderFPropMetrics(params=p) - self.assertIn('fraction_of_correct_next_step_preds', metrics) - self.evaluate(tf.global_variables_initializer()) - metrics_val, per_sequence_loss_val = self.evaluate( - [metrics, per_sequence_loss]) - tf.logging.info('metrics=%s, per_sequence_loss=%s', metrics_val, - per_sequence_loss_val) - - self.assertEqual(metrics_val['loss'], metrics_val['log_pplx']) - # Target batch size is 4. Therefore, we should expect 4 here. - self.assertEqual(per_sequence_loss_val.shape, (4,)) - - def testDecoderFPropWithMeanSeqLoss(self): - """Create and fprop a decoder with different dims per layer.""" - with self.session(use_gpu=False): - tf.random.set_seed(8372749040) - - p = self._DecoderParams( - vn_config=py_utils.VariationalNoiseParams( - None, True, False, seed=12345)) - p.token_normalized_per_seq_loss = True - p.per_token_avg_loss = False - - metrics, per_sequence_loss = self._getDecoderFPropMetrics(params=p) - self.evaluate(tf.global_variables_initializer()) - metrics_val, per_sequence_loss_val = self.evaluate( - [metrics, per_sequence_loss]) - tf.logging.info('metrics=%s, per_sequence_loss=%s', metrics_val, - per_sequence_loss_val) - - self.assertNotEqual(metrics_val['loss'][0], metrics_val['log_pplx'][0]) - self.assertAllClose(metrics_val['loss'], (3.484608, 4.0)) - self.assertAllClose(metrics_val['log_pplx'], (3.496482, 15.0)) - # Target batch size is 4. Therefore, we should expect 4 here. - self.assertEqual(per_sequence_loss_val.shape, (4,)) - - def testDecoderFPropWithProjection(self): - """Create decoder with projection layers, and verify that FProp runs.""" - with self.session(use_gpu=False): - tf.random.set_seed(8372749040) - - p = self._DecoderParams( - vn_config=py_utils.VariationalNoiseParams( - None, True, False, seed=12345)) - rnn_cell_tpl = p.rnn_cell_tpl - p.rnn_cell_tpl = [ - rnn_cell_tpl.Copy().Set( - num_output_nodes=i + 2, num_hidden_nodes=i + 5) - for i in range(p.rnn_layers) - ] - p.rnn_cell_dim = -1 - p.rnn_cell_hidden_dim = -1 - - loss, per_sequence_loss = self._testDecoderFPropHelper(params=p) - self.evaluate(tf.global_variables_initializer()) - loss_val, per_sequence_loss_val = self.evaluate([loss, per_sequence_loss]) - - print('loss = ', loss_val, 'per sequence loss = ', per_sequence_loss_val) - # Target batch size is 4. Therefore, we should expect 4 here. - self.assertEqual(per_sequence_loss_val.shape, (4,)) - - def testDecoderFPropWithPerLayerDims(self): - """Create and fprop a decoder with different dims per layer.""" - with self.session(use_gpu=False): - tf.random.set_seed(8372749040) - - p = self._DecoderParams( - vn_config=py_utils.VariationalNoiseParams( - None, True, False, seed=12345)) - p.rnn_cell_hidden_dim = 6 - - loss, per_sequence_loss = self._testDecoderFPropHelper(params=p) - self.evaluate(tf.global_variables_initializer()) - loss_val, per_sequence_loss_val = self.evaluate([loss, per_sequence_loss]) - - print('loss = ', loss_val, 'per sequence loss = ', per_sequence_loss_val) - # Target batch size is 4. Therefore, we should expect 4 here. - self.assertEqual(per_sequence_loss_val.shape, (4,)) - - def testDecoderFPropDtype(self): - """Create decoder with different fprop_type, and verify that FProp runs.""" - with self.session(use_gpu=False): - tf.random.set_seed(8372749040) - - p = self._DecoderParams( - vn_config=py_utils.VariationalNoiseParams( - None, True, False, seed=12345)) - p.fprop_dtype = tf.float64 - - loss, per_sequence_loss = self._testDecoderFPropHelper(params=p) - self.evaluate(tf.global_variables_initializer()) - loss_val, per_sequence_loss_val = self.evaluate([loss, per_sequence_loss]) - - print('loss = ', loss_val, 'per sequence loss = ', per_sequence_loss_val) - # Target batch size is 4. Therefore, we should expect 4 here. - self.assertEqual(per_sequence_loss_val.shape, (4,)) - - def testDecoderFPropDeterministicAttentionDropout(self): - """Verify that attention dropout is deterministic given fixed seeds.""" - with self.session(use_gpu=False): - tf.random.set_seed(8372749040) - p = self._DecoderParams( - py_utils.VariationalNoiseParams(None, True, False, seed=1792)) - - p.use_while_loop_based_unrolling = False - p.attention.atten_dropout_prob = 0.5 - p.attention.atten_dropout_deterministic = True - - loss, per_sequence_loss = self._testDecoderFPropHelper(params=p) - global_step = py_utils.GetGlobalStep() - self.evaluate(tf.global_variables_initializer()) - loss_val, per_sequence_loss_val, global_steps_val = self.evaluate( - [loss, per_sequence_loss, global_step]) - - print('loss = ', loss_val, 'per sequence loss = ', per_sequence_loss_val) - self.assertAllClose([3.587372, 15.0], loss_val) - self.assertAllClose([14.171288, 9.965696, 10.221684, 19.451914], - per_sequence_loss_val) - self.assertEqual(0, global_steps_val) - - # Run another step to test global_step and time_step are incremented - # correctly. - self.evaluate(tf.assign_add(global_step, 1)) - loss_val, per_sequence_loss_val, global_steps_val = self.evaluate( - [loss, per_sequence_loss, global_step]) - - print('loss = ', loss_val, 'per sequence loss = ', per_sequence_loss_val) - self.assertAllClose([3.626164, 15.0], loss_val) - self.assertAllClose([14.70993, 10.572938, 10.516836, 18.592758], - per_sequence_loss_val) - self.assertEqual(1, global_steps_val) - - def testLabelSmoothing(self): - """Verify that loss computation with label smoothing is as expected..""" - with self.session(use_gpu=False): - tf.random.set_seed(8372749040) - - p = self._DecoderParams(vn_config=py_utils.VariationalNoiseParams(None)) - p.label_smoothing = lingvo_layers.LocalizedLabelSmoother.Params() - p.label_smoothing.offsets = [-2, -1, 1, 2] - p.label_smoothing.weights = [0.015, 0.035, 0.035, 0.015] - - loss, _ = self._testDecoderFPropHelper(params=p) - self.evaluate(tf.global_variables_initializer()) - loss_val = self.evaluate(loss[0]) - - print('loss = ', loss_val) - test_utils.CompareToGoldenSingleFloat(self, 3.471763, loss_val) - - def testDecoderFPropFloatNoInline(self): - actual_value = self._testDecoderFPropFloatHelper(func_inline=False) - test_utils.CompareToGoldenSingleFloat(self, 3.458980, actual_value) - - def testDecoderFPropFloatNoInlinePadTargetsToLongerLength(self): - actual_value = self._testDecoderFPropFloatHelper( - func_inline=False, target_seq_len=10) - test_utils.CompareToGoldenSingleFloat(self, 3.458980, actual_value) - - def testDecoderFPropFloatInline(self): - actual_value = self._testDecoderFPropFloatHelper(func_inline=True) - test_utils.CompareToGoldenSingleFloat(self, 3.458980, actual_value) - - def testDecoderFPropFloatNoInline2Layers(self): - actual_value = self._testDecoderFPropFloatHelper( - func_inline=False, num_decoder_layers=2) - test_utils.CompareToGoldenSingleFloat(self, 3.457761, actual_value) - - def testDecoderFPropFloatInline2Layers(self): - actual_value = self._testDecoderFPropFloatHelper( - func_inline=True, num_decoder_layers=2) - test_utils.CompareToGoldenSingleFloat(self, 3.457761, actual_value) - - def testDecoderFPropFloat2LayersResidual(self): - actual_value = self._testDecoderFPropFloatHelper( - num_decoder_layers=2, residual_start=2) - test_utils.CompareToGoldenSingleFloat(self, 3.458294, actual_value) - - def testDecoderFPropDouble(self): - with self.session(use_gpu=False): - tf.random.set_seed(8372749040) - np.random.seed(827374) - - p = self._DecoderParams( - vn_config=py_utils.VariationalNoiseParams(None, False, False)) - p.dtype = tf.float64 - - dec = decoder.AsrDecoder(p) - src_seq_len = 5 - src_enc = tf.constant( - np.random.uniform(size=(src_seq_len, 2, 8)), tf.float64) - src_enc_padding = tf.constant( - [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 1.0], [1.0, 1.0]], - dtype=tf.float64) - target_ids = tf.transpose( - tf.constant([[0, 1, 2, 3], [1, 2, 3, 4], [10, 11, 12, 15], - [5, 6, 7, 8], [10, 5, 2, 5]], - dtype=tf.int32)) - target_labels = tf.transpose( - tf.constant([[0, 1, 2, 3], [1, 2, 3, 4], [10, 11, 12, 13], - [5, 7, 8, 10], [10, 5, 2, 4]], - dtype=tf.int32)) - target_paddings = tf.transpose( - tf.constant([[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 1, 0], [0, 1, 0, 0], - [1, 1, 1, 1]], - dtype=tf.float64)) - target_transcripts = tf.constant(['abcd', 'bcde', 'klmp', 'fghi', 'kfcf']) - target_weights = 1.0 - target_paddings - targets = py_utils.NestedMap({ - 'ids': target_ids, - 'labels': target_labels, - 'weights': target_weights, - 'paddings': target_paddings, - 'transcripts': target_transcripts, - }) - encoder_outputs = py_utils.NestedMap( - encoded=src_enc, padding=src_enc_padding) - metrics = dec.FPropDefaultTheta(encoder_outputs, targets).metrics - loss = metrics['loss'][0] - - self.evaluate(tf.global_variables_initializer()) - - test_utils.CompareToGoldenSingleFloat(self, 3.467679, loss.eval()) - # Second run to make sure the function is determistic. - test_utils.CompareToGoldenSingleFloat(self, 3.467679, loss.eval()) - - def _testDecoderFPropGradientCheckerHelper(self, func_inline=False): - config = tf.config_pb2.ConfigProto( - graph_options=tf.GraphOptions( - optimizer_options=tf.OptimizerOptions( - do_function_inlining=func_inline))) - with self.session(use_gpu=False, config=config) as sess: - tf.random.set_seed(8372749040) - np.random.seed(274854) - vn_config = py_utils.VariationalNoiseParams(None, False, False) - p = self._DecoderParams(vn_config) - p.dtype = tf.float64 - - dec = p.Instantiate() - src_seq_len = 5 - src_enc = tf.constant( - np.random.uniform(size=(src_seq_len, 2, 8)), tf.float64) - src_enc_padding = tf.constant( - [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 1.0], [1.0, 1.0]], - dtype=tf.float64) - encoder_outputs = py_utils.NestedMap( - encoded=src_enc, padding=src_enc_padding) - target_ids = tf.transpose( - tf.constant([[0, 1, 2, 3], [1, 2, 3, 4], [10, 11, 12, 15], - [5, 6, 7, 8], [10, 5, 2, 5]], - dtype=tf.int32)) - target_labels = tf.transpose( - tf.constant([[0, 1, 2, 3], [1, 2, 3, 4], [10, 11, 12, 13], - [5, 7, 8, 10], [10, 5, 2, 4]], - dtype=tf.int32)) - target_paddings = tf.transpose( - tf.constant([[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 1, 0], [0, 1, 0, 0], - [1, 1, 1, 1]], - dtype=tf.float64)) - target_transcripts = tf.constant(['abcd', 'bcde', 'klmp', 'fghi', 'kfcf']) - target_weights = 1.0 - target_paddings - - targets = py_utils.NestedMap({ - 'ids': target_ids, - 'labels': target_labels, - 'weights': target_weights, - 'paddings': target_paddings, - 'transcripts': target_transcripts, - }) - metrics = dec.FPropDefaultTheta(encoder_outputs, targets).metrics - loss = metrics['loss'][0] - all_vars = tf.trainable_variables() - grads = tf.gradients(loss, all_vars) - - def DenseGrad(var, grad): - if isinstance(grad, tf.Tensor): - return grad - elif isinstance(grad, tf.IndexedSlices): - return tf.math.unsorted_segment_sum(grad.values, grad.indices, - tf.shape(var)[0]) - - dense_grads = [DenseGrad(x, y) for (x, y) in zip(all_vars, grads)] - - self.evaluate(tf.global_variables_initializer()) - - test_utils.CompareToGoldenSingleFloat(self, 3.458078, loss.eval()) - # Second run to make sure the function is determistic. - test_utils.CompareToGoldenSingleFloat(self, 3.458078, loss.eval()) - - symbolic_grads = [x.eval() for x in dense_grads if x is not None] - numerical_grads = [] - for v in all_vars: - numerical_grads.append(test_utils.ComputeNumericGradient(sess, loss, v)) - - for x, y in zip(symbolic_grads, numerical_grads): - self.assertAllClose(x, y) - - def testDecoderFPropGradientCheckerNoInline(self): - self._testDecoderFPropGradientCheckerHelper(func_inline=False) - - def testDecoderFPropGradientCheckerInline(self): - self._testDecoderFPropGradientCheckerHelper(func_inline=True) - - def _testDecoderBeamSearchDecodeHelperWithOutput(self, - params, - src_seq_len=None, - src_enc_padding=None): - config = tf.config_pb2.ConfigProto( - graph_options=tf.GraphOptions( - optimizer_options=tf.OptimizerOptions(do_function_inlining=False))) - p = params - with self.session(use_gpu=False, config=config), self.SetEval(True): - tf.random.set_seed(837274904) - np.random.seed(837575) - p.beam_search.num_hyps_per_beam = 4 - p.dtype = tf.float32 - p.target_seq_len = 5 - - dec = p.Instantiate() - if src_seq_len is None: - src_seq_len = 5 - src_enc = tf.constant( - np.random.uniform(size=(src_seq_len, 2, 8)), tf.float32) - if src_enc_padding is None: - src_enc_padding = tf.constant( - [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 1.0], [1.0, 1.0]], - dtype=tf.float32) - - encoder_outputs = py_utils.NestedMap( - encoded=src_enc, padding=src_enc_padding) - done_hyps = dec.BeamSearchDecode(encoder_outputs).done_hyps - self.evaluate(tf.global_variables_initializer()) - - softmax_wts = self.evaluate(dec.vars.softmax) - print('softmax wts = ', softmax_wts) - - done_hyps_serialized = self.evaluate([done_hyps])[0] - hyp = Hypothesis() - print('done hyps shape = ', done_hyps_serialized.shape) - for i in range(5): - for j in range(8): - print(i, j, len(done_hyps_serialized[i, j])) - hyp.ParseFromString(done_hyps_serialized[2, 5]) - print('hyp = ', hyp) - return hyp - - def _VerifyHypothesesMatch(self, hyp1, hyp2): - tf.logging.info('hyp1 = %s', hyp1) - tf.logging.info('hyp2 = %s', hyp2) - self.assertEqual(hyp1.beam_id, hyp2.beam_id) - self.assertEqual(list(hyp1.ids), list(hyp2.ids)) - self.assertAllClose(hyp1.scores, hyp2.scores) - self.assertEqual(len(hyp1.atten_vecs), len(hyp2.atten_vecs)) - for av1, av2 in zip(hyp1.atten_vecs, hyp2.atten_vecs): - self.assertAllClose(av1.prob, av2.prob) - - def testDecoderBeamSearchDecode(self): - np.random.seed(837575) - - p = self._DecoderParams( - vn_config=py_utils.VariationalNoiseParams(None, False, False), - num_classes=8) - p.beam_search.num_hyps_per_beam = 4 - p.dtype = tf.float32 - p.target_seq_len = 5 - - expected_str = """ - beam_id: 1 - ids: 0 - ids: 6 - ids: 2 - scores: -2.021608 - scores: -2.000098 - scores: -2.036338 - atten_vecs { - prob: 0.330158 - prob: 0.342596 - prob: 0.327246 - prob: 0.0 - prob: 0.0 - } - atten_vecs { - prob: 0.330158 - prob: 0.342597 - prob: 0.327245 - prob: 0.0 - prob: 0.0 - } - atten_vecs { - prob: 0.330158 - prob: 0.342597 - prob: 0.327245 - prob: 0.0 - prob: 0.0 - } - """ - expected_hyp = Hypothesis() - text_format.Parse(expected_str, expected_hyp) - - decoded_hyp = self._testDecoderBeamSearchDecodeHelperWithOutput(params=p) - self._VerifyHypothesesMatch(expected_hyp, decoded_hyp) - - def testDecoderSampleTargetSequences(self): - p = self._DecoderParams( - vn_config=py_utils.VariationalNoiseParams(None, False, False), - num_classes=8) - p.target_seq_len = 5 - p.random_seed = 1 - config = tf.config_pb2.ConfigProto( - graph_options=tf.GraphOptions( - optimizer_options=tf.OptimizerOptions(do_function_inlining=False))) - with self.session(use_gpu=False, config=config): - tf.random.set_seed(8372740) - np.random.seed(35315) - dec = p.Instantiate() - source_sequence_length = 5 - batch_size = 4 - source_encodings = tf.constant( - np.random.normal( - size=[source_sequence_length, batch_size, p.source_dim]), - dtype=tf.float32) - source_encoding_padding = tf.constant( - [[0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 1.0], [0.0, 1.0, 1.0, 1.0], - [0.0, 1.0, 1.0, 1.0], [0.0, 1.0, 1.0, 1.0]], - dtype=tf.float32) - encoder_outputs = py_utils.NestedMap( - encoded=source_encodings, padding=source_encoding_padding) - sampled_sequences = dec.SampleTargetSequences( - dec.theta, encoder_outputs, random_seed=tf.cast(123, tf.int32)) - self.assertAllEqual([batch_size, p.target_seq_len], - sampled_sequences.ids.shape) - self.evaluate(tf.global_variables_initializer()) - decoder_output = self.evaluate(sampled_sequences) - print('ids=%s' % np.array_repr(decoder_output.ids)) - lens = np.sum(1 - decoder_output.paddings, axis=1) - print('lens=%s' % lens) - # pyformat: disable - # pylint: disable=bad-whitespace,bad-continuation - expected_ids = [[6, 2, 2, 2, 2], - [0, 0, 7, 5, 1], - [6, 1, 5, 1, 5], - [6, 7, 7, 4, 4]] - # pylint: enable=bad-whitespace,bad-continuation - # pyformat: enable - expected_lens = [2, 5, 5, 5] - self.assertAllEqual(expected_lens, lens) - self.assertAllEqual(expected_ids, decoder_output.ids) - - # Sample again with the same random seed. - decoder_output2 = self.evaluate( - dec.SampleTargetSequences( - dec.theta, encoder_outputs, random_seed=tf.cast(123, tf.int32))) - # Get the same output. - self.assertAllEqual(decoder_output.ids, decoder_output2.ids) - self.assertAllEqual(decoder_output.paddings, decoder_output2.paddings) - - # Sample again with a different random seed. - decoder_output3 = self.evaluate( - dec.SampleTargetSequences( - dec.theta, encoder_outputs, random_seed=tf.cast(123456, - tf.int32))) - # Get different sequences. - self.assertNotAllClose(expected_ids, decoder_output3.ids) - - def testDecoderFPropWithSymbolicShape(self): - """Create decoder with default params, and verify that FProp runs.""" - with self.session(): - p = self._DecoderParams( - vn_config=py_utils.VariationalNoiseParams( - None, True, False, seed=12345)) - p.rnn_cell_dim = symbolic.Symbol('rnn_cell_dim') - - with symbolic.SymbolToValueMap(symbolic.STATIC_VALUES, - {p.rnn_cell_dim: 6}): - loss, per_sequence_loss = self._testDecoderFPropHelper(params=p) - self.evaluate(tf.global_variables_initializer()) - loss_val, per_sequence_loss_val = self.evaluate( - [loss, per_sequence_loss]) - - print('loss = ', loss_val, 'per sequence loss = ', per_sequence_loss_val) - # Target batch size is 4. Therefore, we should expect 4 here. - self.assertEqual(per_sequence_loss_val.shape, (4,)) - - def testUpdateTargetVocabSize(self): - p = self._DecoderParams( - vn_config=py_utils.VariationalNoiseParams( - None, True, False, seed=12345)) - p.label_smoothing = lingvo_layers.LocalizedLabelSmoother.Params() - p.label_smoothing.num_classes = p.softmax.num_classes - - vocab_size = 1024 - self.assertNotEqual(p.emb.vocab_size, vocab_size) - self.assertNotEqual(p.softmax.num_classes, vocab_size) - self.assertNotEqual(p.fusion.lm.vocab_size, vocab_size) - self.assertNotEqual(p.label_smoothing.num_classes, vocab_size) - p = p.cls.UpdateTargetVocabSize(p, vocab_size) - dec = p.Instantiate() - self.assertEqual(vocab_size, dec.params.emb.vocab_size) - self.assertEqual(vocab_size, dec.params.softmax.num_classes) - self.assertEqual(vocab_size, p.fusion.lm.vocab_size) - self.assertEqual(vocab_size, p.label_smoothing.num_classes) - - def testDecoderFPropWithAdapters(self): - """Create decoder with adapters, and verify that FProp runs.""" - with self.session(use_gpu=False): - tf.random.set_seed(8372749040) - - params = self._DecoderParams( - num_rnn_layers=2, - vn_config=py_utils.VariationalNoiseParams( - None, True, False, seed=12345)) - params.rnn_cell_dim = 3 - params.adapter_layer_tpl.Set( - bottleneck_dim=4, - num_tasks=16, - projection_params_init=py_utils.WeightInit.Gaussian(0.01)) - params.adapter_task_id_field = 'domain_ids' - - dec = params.Instantiate() - src_seq_len = 5 - src_enc = tf.random.normal([src_seq_len, 2, 8], - seed=982774838, - dtype=py_utils.FPropDtype(params)) - src_enc_padding = tf.constant( - [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 1.0], [1.0, 1.0]], - dtype=py_utils.FPropDtype(params)) - domain_ids = tf.constant(np.random.randint(low=0, high=16, size=[2])) - encoder_outputs = py_utils.NestedMap( - encoded=src_enc, padding=src_enc_padding, domain_ids=domain_ids) - # shape=[4, 5] - target_ids = tf.transpose( - tf.constant([[0, 1, 2, 3], [1, 2, 3, 4], [10, 11, 12, 15], - [5, 6, 7, 8], [10, 5, 2, 5]], - dtype=tf.int32)) - # shape=[4, 5] - target_labels = tf.transpose( - tf.constant([[0, 1, 2, 3], [1, 2, 3, 4], [10, 11, 12, 13], - [5, 7, 8, 10], [10, 5, 2, 4]], - dtype=tf.int32)) - # shape=[4, 5] - target_paddings = tf.transpose( - tf.constant([[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 1, 0], [0, 1, 0, 0], - [1, 1, 1, 0]], - dtype=py_utils.FPropDtype(params))) - target_transcripts = tf.constant(['abcd', 'bcde', 'klmp', 'fghi', 'kfcf']) - target_weights = 1.0 - target_paddings - # ids/labels/weights/paddings are all in [batch, time] shape. - targets = py_utils.NestedMap({ - 'ids': target_ids, - 'labels': target_labels, - 'weights': target_weights, - 'paddings': target_paddings, - 'transcripts': target_transcripts, - }) - decoder_outputs = dec.FPropDefaultTheta(encoder_outputs, targets) - metrics = decoder_outputs.metrics - per_sequence_loss = decoder_outputs.per_sequence['loss'] - - self.assertIn('fraction_of_correct_next_step_preds', metrics) - self.evaluate(tf.global_variables_initializer()) - metrics_val, per_sequence_loss_val = self.evaluate( - [metrics, per_sequence_loss]) - tf.logging.info('metrics=%s, per_sequence_loss=%s', metrics_val, - per_sequence_loss_val) - - self.assertEqual(metrics_val['loss'], metrics_val['log_pplx']) - # Target batch size is 4. Therefore, we should expect 4 here. - self.assertEqual(per_sequence_loss_val.shape, (4,)) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/asr/decoder_utils.py b/lingvo/tasks/asr/decoder_utils.py deleted file mode 100644 index ed9efa7a..00000000 --- a/lingvo/tasks/asr/decoder_utils.py +++ /dev/null @@ -1,168 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Common utilities for ASR decoders.""" - -import copy -import lingvo.compat as tf -from lingvo.core import py_utils -from lingvo.core import symbolic -import six - - -def _IsSymbolOrPositive(dim): - return symbolic.IsSymbol(dim) or dim > 0 - - -def SetRnnCellNodes(decoder_params, rnn_cell_params): - if _IsSymbolOrPositive(decoder_params.rnn_cell_dim): - rnn_cell_params.num_output_nodes = decoder_params.rnn_cell_dim - if _IsSymbolOrPositive(decoder_params.rnn_cell_hidden_dim): - if not hasattr(rnn_cell_params, 'num_hidden_nodes'): - raise ValueError( - 'num_hidden_nodes not supported by the RNNCell: %s' % rnn_cell_params) - rnn_cell_params.num_hidden_nodes = decoder_params.rnn_cell_hidden_dim - - -def Tokenize(string): - """Returns a list containing non-empty tokens from the given string.""" - if not isinstance(string, str): - string = six.ensure_text(string, 'utf-8') - return string.split() - - -def ComputeWer(hyps, refs): - """Computes word errors in hypotheses relative to reference transcripts. - - Args: - hyps: Hypotheses, represented as string tensors of shape [N]. - refs: References, represented as string tensors of shape [N]. - - Returns: - An int64 tensor, word_errs, of size [N, 2] where word_errs[i, 0] corresponds - to the number of word errors in hyps[i] relative to refs[i]; word_errs[i, 1] - corresponds to the number of words in refs[i]. - """ - - def _NormalizeWhitespace(s): - return tf.strings.regex_replace(tf.strings.strip(s), r'\s+', ' ') - - hyps = _NormalizeWhitespace(hyps) - refs = _NormalizeWhitespace(refs) - - hyps = py_utils.HasRank(hyps, 1) - refs = py_utils.HasRank(refs, 1) - hyps = py_utils.HasShape(hyps, tf.shape(refs)) - - word_errors = tf.cast( - tf.edit_distance( - tf.string_split(hyps), tf.string_split(refs), normalize=False), - tf.int64) - - # Count number of spaces in reference, and increment by 1 to get total number - # of words. - ref_words = tf.cast( - tf.strings.length(tf.strings.regex_replace(refs, '[^ ]', '')) + 1, - tf.int64) - # Set number of words to 0 if the reference was empty. - ref_words = tf.where( - tf.equal(refs, ''), tf.zeros_like(ref_words, tf.int64), ref_words) - - return tf.concat( - [tf.expand_dims(word_errors, -1), - tf.expand_dims(ref_words, -1)], axis=1) - - -def EditDistance(ref_str, hyp_str): - """Computes Levenshtein edit distance between reference and hypotheses. - - Args: - ref_str: A string of the ref sentence. - hyp_str: A string of one actual hyp. - - Returns: - (ins, subs, del, total): - - - ins: number of insertions. - - subs: number of substitutions. - - del: number of deletions. - - total: total difference length. - """ - - class ErrorStats: - """Class to keep track of error counts.""" - - def __init__(self, ins, dels, subs, tot): - self.ins, self.dels, self.subs, self.total_cost = ins, dels, subs, tot - - def __repr__(self): - return 'ErrorStats(ins=%d, dels=%d, subs=%d, tot=%d)' % ( - self.ins, self.dels, self.subs, self.total_cost) - - # temp sequence to remember error type and stats. - e, cur_e = [], [] - lst_ref = Tokenize(ref_str) - for i in range(len(lst_ref) + 1): - e.append(ErrorStats(0, i, 0, i)) - cur_e.append(ErrorStats(0, 0, 0, 0)) - - lst_hyp = Tokenize(hyp_str) - for hyp_index in range(1, len(lst_hyp) + 1): - cur_e[0] = copy.copy(e[0]) - cur_e[0].ins += 1 - cur_e[0].total_cost += 1 - - for ref_index in range(1, len(lst_ref) + 1): - ins_err = e[ref_index].total_cost + 1 - del_err = cur_e[ref_index - 1].total_cost + 1 - sub_err = e[ref_index - 1].total_cost - if lst_hyp[hyp_index - 1] != lst_ref[ref_index - 1]: - sub_err += 1 - - if sub_err < ins_err and sub_err < del_err: - cur_e[ref_index] = copy.copy(e[ref_index - 1]) - if lst_hyp[hyp_index - 1] != lst_ref[ref_index - 1]: - cur_e[ref_index].subs += 1 - cur_e[ref_index].total_cost = sub_err - elif del_err < ins_err: - cur_e[ref_index] = copy.copy(cur_e[ref_index - 1]) - cur_e[ref_index].total_cost = del_err - cur_e[ref_index].dels += 1 - else: - cur_e[ref_index] = copy.copy(e[ref_index]) - cur_e[ref_index].total_cost = ins_err - cur_e[ref_index].ins += 1 - - for i in range(len(e)): - e[i] = copy.copy(cur_e[i]) - - return e[-1].ins, e[-1].subs, e[-1].dels, e[-1].total_cost - - -def EditDistanceInIds(ref_ids, hyp_ids): - ref_ids = ['%d' % x for x in ref_ids] - hyp_ids = ['%d' % x for x in hyp_ids] - ref_str = ' '.join(ref_ids) - hyp_str = ' '.join(hyp_ids) - return EditDistance(ref_str, hyp_str) - - -def FilterEpsilon(string): - """Filters out tokens from the given string.""" - return ' '.join(Tokenize(string.replace('', ' '))) - - -def FilterNoise(string): - """Filters out tokens from the given string.""" - return ' '.join(t for t in Tokenize(string) if t != '') diff --git a/lingvo/tasks/asr/decoder_utils_test.py b/lingvo/tasks/asr/decoder_utils_test.py deleted file mode 100644 index 5e81f764..00000000 --- a/lingvo/tasks/asr/decoder_utils_test.py +++ /dev/null @@ -1,212 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for decoder utility functions.""" - -import lingvo.compat as tf -from lingvo.core import rnn_cell -from lingvo.core import symbolic -from lingvo.core import test_utils -from lingvo.tasks.asr import decoder -from lingvo.tasks.asr import decoder_utils - -FLAGS = tf.flags.FLAGS - - -class DecoderUtilsSetRnnCellNodesTest(test_utils.TestCase): - - def testSetRnnCellNodes(self): - decoder_p = decoder.AsrDecoder.Params() - base_rnn_p = rnn_cell.LSTMCellSimple.Params().Set(num_output_nodes=4) - - # rnn_cell_dim > 0. - decoder_p.rnn_cell_dim = 8 - rnn_p = base_rnn_p.Copy() - decoder_utils.SetRnnCellNodes(decoder_p, rnn_p) - self.assertEqual(rnn_p.num_output_nodes, decoder_p.rnn_cell_dim) - - # rnn_cell_dim <= 0. - decoder_p.rnn_cell_dim = 0 - rnn_p = base_rnn_p.Copy() - decoder_utils.SetRnnCellNodes(decoder_p, rnn_p) - self.assertEqual(rnn_p.num_output_nodes, base_rnn_p.num_output_nodes) - - # rnn_cell_dim is a symbol. - decoder_p.rnn_cell_dim = symbolic.Symbol("rnn_cell_dim") - rnn_p = base_rnn_p.Copy() - decoder_utils.SetRnnCellNodes(decoder_p, rnn_p) - self.assertIs(rnn_p.num_output_nodes, decoder_p.rnn_cell_dim) - - # rnn_cell_hidden_dim > 0. - decoder_p.rnn_cell_hidden_dim = 16 - rnn_p = base_rnn_p.Copy() - decoder_utils.SetRnnCellNodes(decoder_p, rnn_p) - self.assertEqual(rnn_p.num_hidden_nodes, decoder_p.rnn_cell_hidden_dim) - - # rnn_cell_hidden_dim <= 0. - decoder_p.rnn_cell_hidden_dim = 0 - rnn_p = base_rnn_p.Copy() - decoder_utils.SetRnnCellNodes(decoder_p, rnn_p) - self.assertEqual(rnn_p.num_hidden_nodes, base_rnn_p.num_hidden_nodes) - - # rnn_cell_hidden_dim is a symbol. - decoder_p.rnn_cell_hidden_dim = symbolic.Symbol("rnn_cell_hidden_dim") - rnn_p = base_rnn_p.Copy() - decoder_utils.SetRnnCellNodes(decoder_p, rnn_p) - self.assertIs(rnn_p.num_hidden_nodes, decoder_p.rnn_cell_hidden_dim) - - -class DecoderUtilsTokenizeTest(test_utils.TestCase): - - def testTokenize(self): - s = "onetoken" - self.assertEqual(["onetoken"], decoder_utils.Tokenize(s)) - - s = "two tokens" - self.assertEqual(["two", "tokens"], decoder_utils.Tokenize(s)) - - s = " extra spaces are filtered " - self.assertEqual(["extra", "spaces", "are", "filtered"], - decoder_utils.Tokenize(s)) - - -class DecoderUtilsComputeWerTest(test_utils.TestCase): - - def testInvalidInputsExtraHyps(self): - with self.session(): - with self.assertRaises(Exception): - decoder_utils.ComputeWer(hyps=["one", "two"], refs=["one"]).eval() - - def testInvalidInputsExtraRefs(self): - with self.session(): - with self.assertRaises(Exception): - decoder_utils.ComputeWer(hyps=["one"], refs=["one", "two"]).eval() - - def testInvalidInputsWrongRank(self): - with self.session(): - with self.assertRaises(Exception): - decoder_utils.ComputeWer( - hyps=[["one"], ["two"]], refs=[["one"], ["two"]]).eval() - - def testBasic(self): - with self.session(): - self.assertAllEqual( - decoder_utils.ComputeWer(hyps=["one"], refs=["one"]).eval(), [[0, 1]]) - self.assertAllEqual( - decoder_utils.ComputeWer(hyps=["one two"], refs=["one two"]).eval(), - [[0, 2]]) - - def testMultiples(self): - with self.session(): - wer = decoder_utils.ComputeWer( - hyps=["one", "two pigs"], refs=["one", "three pink pigs"]) - self.assertAllEqual(wer.shape, [2, 2]) - self.assertAllEqual(wer.eval(), [[0, 1], [2, 3]]) - - def testConsecutiveWhiteSpace(self): - with self.session(): - wer = decoder_utils.ComputeWer( - hyps=["one two", "one two", "two pigs"], - refs=["one two", "one two ", "three pink pigs"]) - self.assertAllEqual(wer.shape, [3, 2]) - self.assertAllEqual(wer.eval(), [[0, 2], [0, 2], [2, 3]]) - - def testEmptyRefsAndHyps(self): - with self.session(): - wer = decoder_utils.ComputeWer( - hyps=["", "one two", ""], refs=["", "", "three four five"]) - self.assertAllEqual(wer.shape, [3, 2]) - self.assertAllEqual(wer.eval(), [[0, 0], [2, 0], [3, 3]]) - - def testDifferencesInCaseAreCountedAsErrors(self): - with self.session(): - wer = decoder_utils.ComputeWer( - hyps=["ONE two", "one two"], refs=["one two", "ONE two"]) - self.assertAllEqual(wer.shape, [2, 2]) - self.assertAllEqual(wer.eval(), [[1, 2], [1, 2]]) - - -class DecoderUtilsFilterTest(test_utils.TestCase): - - def testFilterEpsilon(self): - s = "no epsilon" - self.assertEqual(s, decoder_utils.FilterEpsilon(s)) - - s = "epsilon tokens areremoved" - self.assertEqual("epsilon tokens are removed", - decoder_utils.FilterEpsilon(s)) - - def testFilterNoise(self): - s = "no noise" - self.assertEqual(s, decoder_utils.FilterNoise(s)) - - s = " noise tokens are removed " - self.assertEqual("noise tokens are removed", decoder_utils.FilterNoise(s)) - - -class DecoderUtilsEditDistanceTest(test_utils.TestCase): - - def testEditDistance1(self): - ref = "a b c d e f g h" - hyp = "a b c d e f g h" - self.assertEqual((0, 0, 0, 0), decoder_utils.EditDistance(ref, hyp)) - - ref = "a b c d e f g h" - hyp = "a b d e f g h" - self.assertEqual((0, 0, 1, 1), decoder_utils.EditDistance(ref, hyp)) - - ref = "a b c d e f g h" - hyp = "a b c i d e f g h" - self.assertEqual((1, 0, 0, 1), decoder_utils.EditDistance(ref, hyp)) - - ref = "a b c d e f g h" - hyp = "a b c i e f g h" - self.assertEqual((0, 1, 0, 1), decoder_utils.EditDistance(ref, hyp)) - - ref = "a b c d e f g j h" - hyp = "a b c i d e f g h" - self.assertEqual((1, 0, 1, 2), decoder_utils.EditDistance(ref, hyp)) - - ref = "a b c d e f g j h" - hyp = "a b c i e f g h k" - self.assertEqual((1, 1, 1, 3), decoder_utils.EditDistance(ref, hyp)) - - ref = "" - hyp = "" - self.assertEqual((0, 0, 0, 0), decoder_utils.EditDistance(ref, hyp)) - ref = "" - hyp = "a b c" - self.assertEqual((3, 0, 0, 3), decoder_utils.EditDistance(ref, hyp)) - - ref = "a b c d" - hyp = "" - self.assertEqual((0, 0, 4, 4), decoder_utils.EditDistance(ref, hyp)) - - def testEditDistanceInIds(self): - ref = [0, 1, 2, 3, 9] - hyp = [0, 2, 3, 5, 6] - self.assertEqual((1, 1, 1, 3), decoder_utils.EditDistanceInIds(ref, hyp)) - - def testEditDistanceSkipsEmptyTokens(self): - ref = "a b c d e f g h" - hyp = "a b c d e f g h" - self.assertEqual((0, 0, 0, 0), decoder_utils.EditDistance(ref, hyp)) - - ref = "a b c d e f g h" - hyp = "a b c d e f g h" - self.assertEqual((0, 0, 0, 0), decoder_utils.EditDistance(ref, hyp)) - - -if __name__ == "__main__": - tf.test.main() diff --git a/lingvo/tasks/asr/encoder.py b/lingvo/tasks/asr/encoder.py deleted file mode 100644 index 7bf619b9..00000000 --- a/lingvo/tasks/asr/encoder.py +++ /dev/null @@ -1,464 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Encoders for the speech model.""" - -import collections -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import layers -from lingvo.core import model_helper -from lingvo.core import py_utils -from lingvo.core import rnn_cell -from lingvo.core import rnn_layers -from lingvo.core import spectrum_augmenter -from lingvo.core import summary_utils - -from tensorflow.python.ops import inplace_ops - -ConvLSTMBlock = collections.namedtuple('ConvLSTMBlock', ('rnn', 'cnn')) - - -class AsrEncoder(base_layer.BaseLayer): - """Speech encoder version 1.""" - - @classmethod - def Params(cls): - """Configs for AsrEncoder.""" - p = super().Params() - p.Define('specaugment_network', - spectrum_augmenter.SpectrumAugmenter.Params(), - 'Configs template for the augmentation network.') - p.Define('use_specaugment', False, 'Use specaugmentation or not.') - p.Define('lstm_tpl', rnn_cell.LSTMCellSimple.Params(), - 'Configs template for the RNN layer.') - p.Define('cnn_tpl', layers.ConvLayer.Params(), - 'Configs template for the conv layer.') - p.Define('proj_tpl', layers.ProjectionLayer.Params(), - 'Configs template for the projection layer.') - p.Define( - 'highway_skip', False, - 'If set, residual connections from different layers are gated. ' - 'Will only be used if residual_start is enabled.') - p.Define('highway_skip_tpl', layers.HighwaySkipLayer.Params(), - 'Configs template for the highway skip layer.') - p.Define('conv_lstm_tpl', rnn_cell.ConvLSTMCell.Params(), - 'Configs template for ConvLSTMCell.') - p.Define( - 'after_conv_lstm_cnn_tpl', layers.ConvLayer.Params(), - 'Configs template for the cnn layer immediately follow the' - ' convlstm layer.') - p.Define('conv_filter_shapes', None, 'Filter shapes for each conv layer.') - p.Define('conv_filter_strides', None, 'Filter strides for each conv layer.') - p.Define('input_shape', [None, None, None, None], - 'Shape of the input. This should a TensorShape with rank 4.') - p.Define('lstm_cell_size', 256, 'LSTM cell size for the RNN layer.') - p.Define('num_cnn_layers', 2, 'Number of conv layers to create.') - p.Define('num_conv_lstm_layers', 1, 'Number of conv lstm layers to create.') - p.Define('num_lstm_layers', 3, 'Number of rnn layers to create') - p.Define('project_lstm_output', True, - 'Include projection layer after each encoder LSTM layer.') - p.Define('pad_steps', 6, - 'Extra zero-padded timesteps to add to the input sequence. ') - p.Define( - 'residual_start', 0, 'Start residual connections from this lstm layer. ' - 'Disabled if 0 or greater than num_lstm_layers.') - p.Define('residual_stride', 1, - 'Number of lstm layers to skip per residual connection.') - p.Define('bidi_rnn_type', 'func', 'Options: func. ' - 'func: BidirectionalFRNN. ') - p.Define( - 'extra_per_layer_outputs', False, - 'Whether to output the encoding result from each encoder layer besides ' - 'the regular final output. The corresponding extra outputs are keyed ' - 'by "${layer_type}_${layer_index}" in the encoder output NestedMap, ' - 'where layer_type is one of: "conv", "conv_lstm" and "rnn".') - p.Define('stacking_layer_tpl', layers.StackingOverTime.Params(), - 'Configs template for the stacking layer over time.') - p.Define( - 'layer_index_before_stacking', -1, - 'The (0-based) index of the lstm layer after which the stacking layer ' - 'will be inserted. Negative value means no stacking layer will be ' - 'used.') - - # TODO(yonghui): Maybe move those configs to a separate file. - # Set some reasonable default values. - # - # NOTE(yonghui): The default config below assumes the following encoder - # architecture: - # - # cnn/batch-norm/relu -> - # cnn/batch-norm/relu -> - # bidirectional conv-lstm -> - # cnn/batch-norm/relu - # bidirectional lstm -> - # projection/batch-norm/relu -> - # bidirectional lstm -> - # projection/batch-norm/relu -> - # bidirectional lstm - # - # Default config for the rnn layer. - p.lstm_tpl.params_init = py_utils.WeightInit.Uniform(0.1) - - # Default config for the convolution layer. - # What is this 3 for? Delta and delta-delta? - p.input_shape = [None, None, 80, 3] - p.conv_filter_shapes = [(3, 3, 3, 32), (3, 3, 32, 32)] - p.conv_filter_strides = [(2, 2), (2, 2)] - p.cnn_tpl.params_init = py_utils.WeightInit.TruncatedGaussian(0.1) - # TODO(yonghui): Disable variational noise logic. - # NOTE(yonghui): Fortunately, variational noise logic is currently not - # implemented for ConvLayer yet (as of sep 22, 2016). - - # Default config for the projection layer. - p.proj_tpl.params_init = py_utils.WeightInit.TruncatedGaussian(0.1) - # TODO(yonghui): Disable variational noise logic. - # NOTE(yonghui): Fortunately, variational noise logic is currently not - # implemented for ProjectionLayer yet (as of sep 22, 2016). - - p.conv_lstm_tpl.filter_shape = [1, 3] # height (time), width (frequency) - p.conv_lstm_tpl.inputs_shape = [None, None, None, None] - p.conv_lstm_tpl.cell_shape = [None, None, None, None] - p.conv_lstm_tpl.params_init = py_utils.WeightInit.TruncatedGaussian(0.1) - p.after_conv_lstm_cnn_tpl.filter_shape = [3, 3, None, None] - p.after_conv_lstm_cnn_tpl.params_init = ( - py_utils.WeightInit.TruncatedGaussian(0.1)) - p.after_conv_lstm_cnn_tpl.filter_stride = [1, 1] - - # See https://arxiv.org/pdf/1610.03022.pdf, section 2.2. - p.proj_tpl.batch_norm = True - p.proj_tpl.activation = 'RELU' - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - # Use specAugment or not. - if p.use_specaugment: - self.CreateChild('specaugment', p.specaugment_network.Copy()) - # First create the conv layers. - - assert p.num_cnn_layers == len(p.conv_filter_shapes) - assert p.num_cnn_layers == len(p.conv_filter_strides) - params_conv_layers = [] - for i in range(p.num_cnn_layers): - conv_p = p.cnn_tpl.Copy() - conv_p.name = 'conv_L%d' % i - conv_p.filter_shape = p.conv_filter_shapes[i] - conv_p.filter_stride = p.conv_filter_strides[i] - params_conv_layers.append(conv_p) - self.CreateChildren('conv', params_conv_layers) - - conv_output_shape = p.input_shape - for i in range(p.num_cnn_layers): - conv_output_shape = self.conv[i].OutShape(conv_output_shape) - assert len(conv_output_shape) == 4 # batch, height, width, channel. - - params_conv_lstm_rnn = [] - params_conv_lstm_cnn = [] - for i in range(p.num_conv_lstm_layers): - # NOTE(yonghui): We assume that output from ConvLSTMBlock has the same - # shape as its input. - _, _, width, in_channel = conv_output_shape - f_conv_lstm_p = p.conv_lstm_tpl.Copy() - f_conv_lstm_p.name = 'f_conv_lstm_%d' % i - f_conv_lstm_p.inputs_shape = [None, 1, width, in_channel] - f_conv_lstm_p.cell_shape = [None, 1, width, in_channel] - b_conv_lstm_p = f_conv_lstm_p.Copy() - b_conv_lstm_p.name = 'b_conv_lstm_%d' % i - conv_lstm_rnn_p = self.CreateConvLstmLayerParams() - conv_lstm_rnn_p.name = 'conv_lstm_rnn' - conv_lstm_rnn_p.fwd = f_conv_lstm_p - conv_lstm_rnn_p.bak = b_conv_lstm_p - params_conv_lstm_rnn.append(conv_lstm_rnn_p) - cnn_p = p.after_conv_lstm_cnn_tpl.Copy() - cnn_p.name = 'conv_lstm_cnn_%d' % i - cnn_p.filter_shape[2] = 2 * in_channel - cnn_p.filter_shape[3] = in_channel - params_conv_lstm_cnn.append(cnn_p) - # TODO(yonghui): Refactor ConvLSTMBlock into a layer. - self.CreateChildren('conv_lstm_rnn', params_conv_lstm_rnn) - self.CreateChildren('conv_lstm_cnn', params_conv_lstm_cnn) - - (self._first_lstm_input_dim, - self._first_lstm_input_dim_pad) = self.FirstLstmLayerInputDimAndPadding( - conv_output_shape, pad_to_multiple=16) - - # Now create all the rnn layers and projection layers. - # TODO(yonghui): take care of device placement. - params_rnn_layers = [] - params_proj_layers = [] - params_highway_skip_layers = [] - output_dim = self._first_lstm_input_dim - for i in range(p.num_lstm_layers): - input_dim = output_dim - forward_p = p.lstm_tpl.Copy() - forward_p.name = 'fwd_rnn_L%d' % i - forward_p.num_input_nodes = input_dim - forward_p.num_output_nodes = p.lstm_cell_size - backward_p = forward_p.Copy() - backward_p.name = 'bak_rnn_L%d' % i - rnn_p = self.CreateBidirectionalRNNParams(forward_p, backward_p) - rnn_p.name = 'brnn_L%d' % i - params_rnn_layers.append(rnn_p) - output_dim = 2 * p.lstm_cell_size - - if p.project_lstm_output and (i < p.num_lstm_layers - 1): - proj_p = p.proj_tpl.Copy() - proj_p.input_dim = 2 * p.lstm_cell_size - proj_p.output_dim = 2 * p.lstm_cell_size - proj_p.name = 'proj_L%d' % i - params_proj_layers.append(proj_p) - - # add the skip layers - residual_index = i - p.residual_start + 1 - if p.residual_start > 0 and residual_index >= 0 and p.highway_skip: - highway_skip = p.highway_skip_tpl.Copy() - highway_skip.name = 'enc_hwskip_%d' % len(params_highway_skip_layers) - highway_skip.input_dim = 2 * p.lstm_cell_size - params_highway_skip_layers.append(highway_skip) - # Adds the stacking layer. - if p.layer_index_before_stacking == i: - stacking_layer = p.stacking_layer_tpl.Copy() - stacking_layer.name = 'stacking_%d' % i - self.CreateChild('stacking', stacking_layer) - stacking_window_len = ( - p.stacking_layer_tpl.left_context + 1 + - p.stacking_layer_tpl.right_context) - output_dim *= stacking_window_len - - self.CreateChildren('rnn', params_rnn_layers) - self.CreateChildren('proj', params_proj_layers) - self.CreateChildren('highway_skip', params_highway_skip_layers) - - @property - def _use_functional(self): - return True - - @property - def input_shape(self): - return self.params.input_shape - - def CreateBidirectionalRNNParams(self, forward_p, backward_p): - return model_helper.CreateBidirectionalRNNParams(self.params, forward_p, - backward_p) - - def CreateConvLstmLayerParams(self): - return rnn_layers.BidirectionalFRNN.Params() - - def FirstLstmLayerInputDimAndPadding(self, - conv_output_shape, - pad_to_multiple=16): - lstm_input_shape = conv_output_shape - # Makes sure the lstm input dims is multiple of 16 (alignment - # requirement from FRNN). - first_lstm_input_dim_unpadded = lstm_input_shape[2] * lstm_input_shape[3] - - if self._use_functional and (first_lstm_input_dim_unpadded % pad_to_multiple - != 0): - first_lstm_input_dim = int( - (first_lstm_input_dim_unpadded + pad_to_multiple - 1) / - pad_to_multiple) * pad_to_multiple - else: - first_lstm_input_dim = first_lstm_input_dim_unpadded - - first_lstm_input_dim_padding = ( - first_lstm_input_dim - first_lstm_input_dim_unpadded) - return first_lstm_input_dim, first_lstm_input_dim_padding - - @property - def supports_streaming(self): - return False - - def zero_state(self, theta, batch_size): - return py_utils.NestedMap() - - def FProp(self, theta, batch, state0=None): - """Encodes source as represented by 'inputs' and 'paddings'. - - Args: - theta: A NestedMap object containing weights' values of this - layer and its children layers. - batch: A NestedMap with fields: - - - src_inputs - The inputs tensor. It is expected to be of shape [batch, - time, feature_dim, channels]. - - paddings - The paddings tensor. It is expected to be of shape [batch, - time]. - state0: Recurrent input state. Not supported/ignored by this encoder. - - Returns: - A NestedMap containing - - - 'encoded': a feature tensor of shape [time, batch, depth] - - 'padding': a 0/1 tensor of shape [time, batch] - - 'state': the updated recurrent state - - '${layer_type}_${layer_index}': The per-layer encoder output. Each one - is a NestedMap containing 'encoded' and 'padding' similar to regular - final outputs, except that 'encoded' from conv or conv_lstm layers are - of shape [time, batch, depth, channels]. - """ - p = self.params - inputs, paddings = batch.src_inputs, batch.paddings - outputs = py_utils.NestedMap() - with tf.name_scope(p.name): - # Adding specAugmentation. - if p.use_specaugment and not self.do_eval: - inputs, paddings = self.specaugment.FProp(theta.specaugment, inputs, - paddings) - # Add a few extra padded timesteps at the end. This is for ensuring the - # correctness of the conv-layers at the edges. - if p.pad_steps > 0: - # inplace_update() is not supported by TPU for now. Since we have done - # padding on the input_generator, we may avoid this additional padding. - assert not py_utils.use_tpu() - inputs_pad = tf.zeros( - inplace_ops.inplace_update(tf.shape(inputs), 1, p.pad_steps), - inputs.dtype) - paddings_pad = tf.ones( - inplace_ops.inplace_update(tf.shape(paddings), 1, p.pad_steps), - paddings.dtype) - inputs = tf.concat([inputs, inputs_pad], 1, name='inputs') - paddings = tf.concat([paddings, paddings_pad], 1) - - plots = [ - summary_utils.PrepareSequenceForPlot( - tf.transpose(inputs, [0, 1, 3, 2]), paddings, 'inputs') - ] - - conv_out = inputs - out_padding = paddings - for i, conv_layer in enumerate(self.conv): - conv_out, out_padding = conv_layer.FProp(theta.conv[i], conv_out, - out_padding) - if p.extra_per_layer_outputs: - conv_out *= (1.0 - out_padding[:, :, tf.newaxis, tf.newaxis]) - outputs['conv_%d' % i] = py_utils.NestedMap( - encoded=tf.transpose(conv_out, [1, 0, 2, 3]), # to [t, b, d, c] - padding=tf.transpose(out_padding)) - plots.append( - summary_utils.PrepareSequenceForPlot( - tf.transpose(conv_out, [0, 1, 3, 2]), out_padding, - 'conv_%d_out' % i)) - - def TransposeFirstTwoDims(t): - first_dim = tf.shape(t)[0] - second_dim = tf.shape(t)[1] - t_new = tf.transpose( - tf.reshape(t, [first_dim, second_dim, -1]), [1, 0, 2]) - t_shape_new = tf.concat([[second_dim], [first_dim], tf.shape(t)[2:]], 0) - return tf.reshape(t_new, t_shape_new) - - # Now the conv-lstm part. - conv_lstm_out = conv_out - conv_lstm_out_padding = out_padding - for i, (rnn, cnn) in enumerate( - zip(self.conv_lstm_rnn, self.conv_lstm_cnn)): - conv_lstm_in = conv_lstm_out - # Move time dimension to be the first. - conv_lstm_in = TransposeFirstTwoDims(conv_lstm_in) - conv_lstm_in = tf.expand_dims(conv_lstm_in, 2) - conv_lstm_in_padding = tf.expand_dims( - tf.transpose(conv_lstm_out_padding), 2) - lstm_out = rnn.FProp(theta.conv_lstm_rnn[i], conv_lstm_in, - conv_lstm_in_padding) - # Move time dimension to be the second. - cnn_in = TransposeFirstTwoDims(lstm_out) - cnn_in = tf.squeeze(cnn_in, 2) - cnn_in_padding = conv_lstm_out_padding - cnn_out, cnn_out_padding = cnn.FProp(theta.conv_lstm_cnn[i], cnn_in, - cnn_in_padding) - conv_lstm_out, conv_lstm_out_padding = cnn_out, cnn_out_padding - if p.extra_per_layer_outputs: - conv_lstm_out *= ( - 1.0 - conv_lstm_out_padding[:, :, tf.newaxis, tf.newaxis]) - outputs['conv_lstm_%d' % i] = py_utils.NestedMap( - encoded=tf.transpose(conv_lstm_out, - [1, 0, 2, 3]), # to [t, b, d, c] - padding=tf.transpose(conv_lstm_out_padding)) - plots.append( - summary_utils.PrepareSequenceForPlot(conv_lstm_out, - conv_lstm_out_padding, - 'conv_lstm_%d_out' % i)) - - # Need to do a reshape before starting the rnn layers. - conv_lstm_out = py_utils.HasRank(conv_lstm_out, 4) - conv_lstm_out_shape = tf.shape(conv_lstm_out) - new_shape = tf.concat([conv_lstm_out_shape[:2], [-1]], 0) - conv_lstm_out = tf.reshape(conv_lstm_out, new_shape) - if self._first_lstm_input_dim_pad: - conv_lstm_out = tf.pad( - conv_lstm_out, - [[0, 0], [0, 0], [0, self._first_lstm_input_dim_pad]]) - - conv_lstm_out = py_utils.HasShape(conv_lstm_out, - [-1, -1, self._first_lstm_input_dim]) - - # Transpose to move the time dimension to be the first. - rnn_in = tf.transpose(conv_lstm_out, [1, 0, 2]) - rnn_padding = tf.expand_dims(tf.transpose(conv_lstm_out_padding), 2) - # rnn_in is of shape [time, batch, depth] - # rnn_padding is of shape [time, batch, 1] - - # Now the rnn layers. - num_skips = 0 - for i in range(p.num_lstm_layers): - rnn_out = self.rnn[i].FProp(theta.rnn[i], rnn_in, rnn_padding) - residual_index = i - p.residual_start + 1 - if p.residual_start > 0 and residual_index >= 0: - if residual_index % p.residual_stride == 0: - residual_in = rnn_in - if residual_index % p.residual_stride == p.residual_stride - 1: - # Highway skip connection. - if p.highway_skip: - rnn_out = self.highway_skip[num_skips].FProp( - theta.highway_skip[num_skips], residual_in, rnn_out) - num_skips += 1 - else: - # Residual skip connection. - rnn_out += py_utils.HasShape(residual_in, tf.shape(rnn_out)) - if p.project_lstm_output and (i < p.num_lstm_layers - 1): - # Projection layers. - rnn_out = self.proj[i].FProp(theta.proj[i], rnn_out, rnn_padding) - if i == p.num_lstm_layers - 1: - rnn_out *= (1.0 - rnn_padding) - if p.extra_per_layer_outputs: - rnn_out *= (1.0 - rnn_padding) - outputs['rnn_%d' % i] = py_utils.NestedMap( - encoded=rnn_out, padding=tf.squeeze(rnn_padding, [2])) - # Stacking layer connection. - if p.layer_index_before_stacking == i: - # Stacking layer expects input tensor shape as [batch, time, feature]. - # So transpose the tensors before and after the layer. - rnn_out, rnn_padding = self.stacking.FProp( - tf.transpose(rnn_out, [1, 0, 2]), - tf.transpose(rnn_padding, [1, 0, 2])) - rnn_out = tf.transpose(rnn_out, [1, 0, 2]) - rnn_padding = tf.transpose(rnn_padding, [1, 0, 2]) - - plots.append( - summary_utils.PrepareSequenceForPlot( - tf.transpose(rnn_out, [1, 0, 2]), - tf.transpose(rnn_padding, [1, 0, 2]), 'rnn_%d_out' % i)) - rnn_in = rnn_out - final_out = rnn_in - - summary_utils.PlotSequenceFeatures( - list(reversed(plots)), 'encoder_example', xlabel='Time') - - outputs['encoded'] = final_out - outputs['padding'] = tf.squeeze(rnn_padding, [2]) - outputs['state'] = py_utils.NestedMap() - return outputs diff --git a/lingvo/tasks/asr/encoder_test.py b/lingvo/tasks/asr/encoder_test.py deleted file mode 100644 index ebbd8355..00000000 --- a/lingvo/tasks/asr/encoder_test.py +++ /dev/null @@ -1,337 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for ASR encoder.""" - -import lingvo.compat as tf -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.tasks.asr import encoder -import numpy as np - - -class EncoderTest(test_utils.TestCase): - - def _EncoderParams(self, vn_config): - p = encoder.AsrEncoder.Params() - p.name = 'encoder' - vn_config.scale = tf.constant(0.1) - params_init = py_utils.WeightInit.Uniform(0.05) - - rnn_params = p.lstm_tpl - rnn_params.vn = vn_config - rnn_params.params_init = params_init - - cnn_params = p.cnn_tpl - cnn_params.vn = vn_config - cnn_params.params_init = params_init - - proj_params = p.proj_tpl - proj_params.vn = vn_config - proj_params.params_init = params_init - - p.conv_filter_shapes = [[3, 3, 3, 6], [3, 3, 6, 6]] - p.conv_filter_strides = [[2, 2], [2, 2]] - p.input_shape = [None, None, 16, 3] - - p.conv_lstm_tpl.vn = vn_config - p.after_conv_lstm_cnn_tpl.vn = vn_config - - p.num_cnn_layers = 2 - p.num_lstm_layers = 2 - p.lstm_cell_size = 16 - p.num_conv_lstm_layers = 0 - - return p - - def _ForwardPass(self, p): - tf.random.set_seed(8372749040) - stt_enc = encoder.AsrEncoder(p) - batch = py_utils.NestedMap() - batch.src_inputs = tf.random.normal([2, 20, 16, 3], seed=92837472) - batch.paddings = tf.zeros([2, 20]) - return stt_enc.FPropDefaultTheta(batch) - - def testEncoderConstruction(self): - vn_config = py_utils.VariationalNoiseParams(None, True, False, seed=12345) - p = self._EncoderParams(vn_config) - _ = encoder.AsrEncoder(p) - - def testForwardPass(self): - with self.session(use_gpu=False): - vn_config = py_utils.VariationalNoiseParams(None, False, False) - p = self._EncoderParams(vn_config) - enc_out = self._ForwardPass(p).encoded - enc_out_sum = tf.reduce_sum(enc_out, 0) - self.evaluate(tf.global_variables_initializer()) - - # pyformat: disable - # pylint: disable=bad-whitespace - expected_enc_out = [ - [-2.63900943e-02, -4.88980189e-02, 1.78375337e-02, - -9.66763496e-03, -1.45432353e-02, 3.63842538e-03, - -4.93378285e-03, 9.87463910e-03, -1.98941268e-02, - -2.31636949e-02, -6.76718354e-03, -1.01988772e-02, - 4.81432397e-03, 9.02220048e-03, 1.31793215e-03, - -1.39696691e-02, -2.36637704e-02, -5.25583047e-04, - -3.79295787e-03, 1.09998491e-02, 8.54234211e-03, - -2.43989471e-02, -6.27756910e-03, -1.64192859e-02, - 1.54568311e-02, 3.69091239e-03, 1.27634332e-02, - 2.50437222e-02, 3.77510749e-02, 1.71656217e-02, - 1.94890760e-02, 4.31961473e-03], - [-1.61839426e-02, 1.27755934e-02, -1.96352396e-02, - 1.04363225e-02, 6.10197056e-03, -5.08408714e-03, - -9.20344493e-04, 2.55419128e-02, -3.58198807e-02, - -4.18110676e-02, 9.45025682e-03, -7.00431701e-04, - 2.31945589e-02, -6.53471798e-05, -1.94577798e-02, - -1.53421704e-02, -1.50274234e-02, 1.06492080e-03, - 8.32110923e-03, -1.38334394e-03, 2.02696323e-02, - 2.13975199e-02, 2.23143250e-02, -1.54133392e-02, - 1.83746461e-02, 8.25020485e-03, -1.64317098e-02, - 1.46762179e-02, 1.89543713e-03, -3.36170895e-03, - 3.14423591e-02, -2.64923554e-02 ]] - # pylint: enable=bad-whitespace - # pyformat: enable - enc_out_sum_val = enc_out_sum.eval() - print('enc_out_sum_val', np.array_repr(enc_out_sum_val)) - self.assertAllClose(expected_enc_out, enc_out_sum_val) - - def testForwardPassWithConvLSTM(self): - with self.session(use_gpu=False): - vn_config = py_utils.VariationalNoiseParams(None, False, False) - p = self._EncoderParams(vn_config) - p.num_conv_lstm_layers = 1 - enc_out = self._ForwardPass(p).encoded - enc_out_sum = tf.reduce_sum(enc_out) - self.evaluate(tf.global_variables_initializer()) - - enc_out_sum_val = enc_out_sum.eval() - print('expected enc_out_sum_val', enc_out_sum_val) - test_utils.CompareToGoldenSingleFloat( - self, -0.1091638132929802, enc_out_sum_val) - - def testForwardPassWithConvLSTM2Layers(self): - with self.session(use_gpu=False): - vn_config = py_utils.VariationalNoiseParams(None, False, False) - p = self._EncoderParams(vn_config) - p.num_conv_lstm_layers = 2 - enc_out = self._ForwardPass(p).encoded - enc_out_sum = tf.reduce_sum(enc_out) - self.evaluate(tf.global_variables_initializer()) - - enc_out_sum_val = enc_out_sum.eval() - print('expected enc_out_sum_val', enc_out_sum_val) - test_utils.CompareToGoldenSingleFloat( - self, 0.024680551141500473, enc_out_sum_val) - - def testForwardPassWithResidualStart1(self): - with self.session(use_gpu=False): - vn_config = py_utils.VariationalNoiseParams(None, False, False) - p = self._EncoderParams(vn_config) - p.residual_start = 1 - enc_out = self._ForwardPass(p).encoded - enc_out_sum = tf.reduce_sum(enc_out) - self.evaluate(tf.global_variables_initializer()) - - enc_out_sum_val = enc_out_sum.eval() - print('expected enc_out_sum_val', enc_out_sum_val) - test_utils.CompareToGoldenSingleFloat( - self, 119.61833190917969, enc_out_sum_val) - - def testForwardPassWithResidualStart2(self): - with self.session(use_gpu=False): - vn_config = py_utils.VariationalNoiseParams(None, False, False) - p = self._EncoderParams(vn_config) - p.residual_start = 2 - enc_out = self._ForwardPass(p).encoded - enc_out_sum = tf.reduce_sum(enc_out) - self.evaluate(tf.global_variables_initializer()) - - enc_out_sum_val = enc_out_sum.eval() - print('expected enc_out_sum_val', enc_out_sum_val) - test_utils.CompareToGoldenSingleFloat( - self, 13.103919982910156, enc_out_sum_val) - - def testForwardPassWithResidualStart1Interval2(self): - with self.session(use_gpu=False): - vn_config = py_utils.VariationalNoiseParams(None, False, False) - p = self._EncoderParams(vn_config) - p.residual_start = 1 - p.residual_stride = 2 - enc_out = self._ForwardPass(p).encoded - enc_out_sum = tf.reduce_sum(enc_out) - self.evaluate(tf.global_variables_initializer()) - - enc_out_sum_val = enc_out_sum.eval() - print('expected enc_out_sum_val', enc_out_sum_val) - test_utils.CompareToGoldenSingleFloat( - self, 92.58077239990234, enc_out_sum_val) - - def testForwardPassWithStackingAfterFinalLayer(self): - with self.session(use_gpu=False): - vn_config = py_utils.VariationalNoiseParams(None, False, False) - p = self._EncoderParams(vn_config) - p.stacking_layer_tpl.left_context = 1 - p.stacking_layer_tpl.right_context = 0 - p.stacking_layer_tpl.stride = 2 - p.layer_index_before_stacking = 1 - enc_out = self._ForwardPass(p).encoded - enc_out_sum = tf.reduce_sum(enc_out, 0) - self.evaluate(tf.global_variables_initializer()) - # pyformat: disable - # pylint: disable=bad-whitespace - expected_enc_out = [ - [-1.25796525e-02, -2.32883729e-02, 7.40477070e-03, -4.51436592e-03, - -5.84740378e-03, 2.30195466e-03, -3.08505213e-03, 4.05658083e-03, - -8.12252797e-03, -1.08030904e-02, -4.17955732e-03, -3.73707339e-03, - 6.97144482e-04, 2.79850606e-03, 8.33133236e-04, -5.75614115e-03, - -1.10648498e-02, -1.20132393e-03, -1.69872947e-03, 6.97519444e-03, - 2.46211258e-03, -1.28190573e-02, -8.66306946e-05, -6.09322963e-03, - 7.14540575e-03, -5.67986863e-05, 5.17684873e-03, 1.18097477e-02, - 1.74862407e-02, 9.13049746e-03, 7.31027778e-03, 4.83186450e-05, - -1.38104409e-02, -2.56096497e-02, 1.04327593e-02, -5.15327370e-03, - -8.69584084e-03, 1.33647269e-03, -1.84873224e-03, 5.81806153e-03, - -1.17716007e-02, -1.23606063e-02, -2.58761784e-03, -6.46180846e-03, - 4.11718246e-03, 6.22369815e-03, 4.84800315e-04, -8.21352564e-03, - -1.25989169e-02, 6.75740885e-04, -2.09423108e-03, 4.02465323e-03, - 6.08023722e-03, -1.15798926e-02, -6.19094400e-03, -1.03260633e-02, - 8.31142440e-03, 3.74771934e-03, 7.58658582e-03, 1.32339774e-02, - 2.02648211e-02, 8.03512800e-03, 1.21787926e-02, 4.27130330e-03], - [-5.94401825e-03, 4.23503201e-03, -7.39302021e-03, 3.84659087e-03, - 2.92047067e-03, -2.28955783e-03, 7.80778937e-05, 7.74920732e-03, - -1.29534695e-02, -1.44997425e-02, 3.00848205e-03, -1.33561785e-04, - 7.31927902e-03, -2.24683899e-03, -6.27679843e-03, -5.35295857e-03, - -5.39031485e-03, -4.90641687e-05, 4.03603073e-03, -1.08133641e-03, - 9.59445070e-03, 9.81783494e-03, 8.77558347e-03, -5.13678743e-03, - 7.19959754e-03, 3.93835502e-03, -6.01979066e-03, 6.13247836e-03, - 1.39782019e-03, 4.60287556e-04, 1.04263611e-02, -9.61792190e-03, - -1.02399308e-02, 8.54056142e-03, -1.22422148e-02, 6.58972748e-03, - 3.18149826e-03, -2.79453350e-03, -9.98417381e-04, 1.77927073e-02, - -2.28664111e-02, -2.73113251e-02, 6.44177478e-03, -5.66864444e-04, - 1.58752780e-02, 2.18148530e-03, -1.31809842e-02, -9.98921506e-03, - -9.63711366e-03, 1.11398206e-03, 4.28507291e-03, -3.02007422e-04, - 1.06751733e-02, 1.15796775e-02, 1.35387452e-02, -1.02765551e-02, - 1.11750513e-02, 4.31185029e-03, -1.04119312e-02, 8.54373723e-03, - 4.97616245e-04, -3.82199232e-03, 2.10159980e-02, -1.68744288e-02]] - # pylint: enable=bad-whitespace - # pyformat: enable - enc_out_sum_val = enc_out_sum.eval() - print('expected enc_out_sum_val', enc_out_sum_val) - self.assertAllClose(expected_enc_out, enc_out_sum_val) - - def testForwardPassWithStackingAfterMiddleLayer(self): - with self.session(use_gpu=False): - vn_config = py_utils.VariationalNoiseParams(None, False, False) - p = self._EncoderParams(vn_config) - p.stacking_layer_tpl.left_context = 1 - p.stacking_layer_tpl.right_context = 0 - p.stacking_layer_tpl.stride = 2 - p.layer_index_before_stacking = 0 - enc_out = self._ForwardPass(p).encoded - enc_out_sum = tf.reduce_sum(enc_out, 0) - - self.evaluate(tf.global_variables_initializer()) - - # pyformat: disable - # pylint: disable=bad-whitespace - expected_enc_out = [ - [0.00102275, -0.02697385, 0.01709868, -0.00939053, -0.01576837, - 0.0070826, -0.00626193, 0.01143604, -0.01742513, -0.00529445, - 0.00284249, -0.01362027, -0.00490865, 0.0216262, -0.01344598, - -0.00460993, -0.01329017, 0.01379208, -0.00850593, 0.0193335, - 0.01134925, -0.00131254, 0.00375953, -0.00588882, 0.01347932, - -0.00252493, 0.01274828, 0.01027388, 0.02657663, 0.02644286, - 0.0286899, -0.00833998], - [-0.01801126, 0.0115137, 0.01355767, 0.00113954, 0.00986663, - -0.0128988, 0.00794239, -0.00524312, 0.00246279, -0.00575782, - -0.00213567, -0.01528412, 0.00186096, 0.00253562, -0.00411006, - -0.00390748, -0.01001569, -0.00344393, -0.01211706, 0.00387725, - 0.02194905, 0.02578988, -0.00255773, 0.00690117, 0.00976908, - 0.01935913, 0.01131854, 0.0013859, -0.01567556, 0.01858256, - 0.02251371, -0.0185001]] - # pylint: enable=bad-whitespace - # pyformat: enable - enc_out_sum_val = enc_out_sum.eval() - print('enc_out_sum_val', np.array_repr(enc_out_sum_val)) - self.assertAllClose(expected_enc_out, enc_out_sum_val) - - def testForwardPassWithHighwaySkip(self): - with self.session(use_gpu=False): - vn_config = py_utils.VariationalNoiseParams(None, False, False) - p = self._EncoderParams(vn_config) - p.residual_start = 1 - p.residual_stride = 2 - p.highway_skip = True - enc_out = self._ForwardPass(p).encoded - enc_out_sum = tf.reduce_sum(enc_out) - self.evaluate(tf.global_variables_initializer()) - - enc_out_sum_val = enc_out_sum.eval() - print('expected enc_out_sum_val', enc_out_sum_val) - test_utils.CompareToGoldenSingleFloat( - self, 65.77313995361328, enc_out_sum_val) - - - def testForwardPassWithExtraPerLayerOutputs(self): - with self.session(use_gpu=False): - vn_config = py_utils.VariationalNoiseParams(None, False, False) - p = self._EncoderParams(vn_config) - p.num_conv_lstm_layers = 1 - p.extra_per_layer_outputs = True - enc_out = self._ForwardPass(p) - regular_encoded_sum = tf.reduce_sum(enc_out.encoded) - conv_0_encoded_sum = tf.reduce_sum(enc_out.conv_0.encoded) - conv_1_encoded_sum = tf.reduce_sum(enc_out.conv_1.encoded) - conv_lstm_0_encoded_sum = tf.reduce_sum(enc_out.conv_lstm_0.encoded) - rnn_0_encoded_sum = tf.reduce_sum(enc_out.rnn_0.encoded) - - self.evaluate(tf.global_variables_initializer()) - - # pyformat: disable - self.assertAllEqual(tf.shape(enc_out.conv_0.encoded).eval(), - [13, 2, 8, 6]) - self.assertAllEqual(tf.shape(enc_out.conv_0.padding).eval(), - [13, 2]) - self.assertAllEqual(tf.shape(enc_out.conv_1.encoded).eval(), - [7, 2, 4, 6]) - self.assertAllEqual(tf.shape(enc_out.conv_1.padding).eval(), - [7, 2]) - self.assertAllEqual(tf.shape(enc_out.conv_lstm_0.encoded).eval(), - [7, 2, 4, 6]) - self.assertAllEqual(tf.shape(enc_out.conv_lstm_0.padding).eval(), - [7, 2]) - self.assertAllEqual(tf.shape(enc_out.rnn_0.encoded).eval(), - [7, 2, 32]) - self.assertAllEqual(tf.shape(enc_out.rnn_0.padding).eval(), - [7, 2]) - self.assertAllEqual(tf.shape(enc_out.encoded).eval(), - [7, 2, 32]) - self.assertAllEqual(tf.shape(enc_out.padding).eval(), - [7, 2]) - # pyformat: enable - - test_utils.CompareToGoldenSingleFloat(self, 371.75390625, - conv_0_encoded_sum.eval()) - test_utils.CompareToGoldenSingleFloat(self, 92.5332946777, - conv_1_encoded_sum.eval()) - test_utils.CompareToGoldenSingleFloat(self, 80.975112915, - conv_lstm_0_encoded_sum.eval()) - test_utils.CompareToGoldenSingleFloat(self, 10.9648704529, - rnn_0_encoded_sum.eval()) - test_utils.CompareToGoldenSingleFloat(self, 0.0522322505713, - regular_encoded_sum.eval()) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/asr/encoder_v2.py b/lingvo/tasks/asr/encoder_v2.py deleted file mode 100644 index 1a5a21af..00000000 --- a/lingvo/tasks/asr/encoder_v2.py +++ /dev/null @@ -1,127 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Encoders for the speech model.""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils -from lingvo.core import spectrum_augmenter -from lingvo.tasks.asr import blocks - - -class AsrEncoder(base_layer.BaseLayer): - """Speech encoder version 2.""" - - @classmethod - def Params(cls): - """Configs for AsrEncoder.""" - p = super().Params() - - # spec-augment - p.Define('specaugment_network', - spectrum_augmenter.SpectrumAugmenter.Params(), - 'Configs template for the augmentation network.') - p.Define('use_specaugment', False, 'Use specaugmentation or not.') - - # temporal downsampling, use one of the two - p.Define('conv_subsampler', blocks.ConvolutionalDownsampler.Params(), - 'Convolution subsampling layer params') - p.Define('stacking_subsampler', blocks.InputStackingDownsampler.Params(), - 'Stacking subsampling layer params') - p.Define('use_conv_subsampler', False, 'Enable p.conv_subsampler') - p.Define('use_stacking_subsampler', False, 'Enable p.stacking_subsampler') - - # actual encoding layers, use one of these - p.Define('lstm_block', blocks.LSTMBlock.Params(), 'LSTM layer params') - # p.Define('conformer_block', blocks.ConformerBlock.Params(), 'Conformer specs') - - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - ##### Use specAugment or not #### - if p.use_specaugment: - self.CreateChild('specaugment', p.specaugment_network.Copy()) - - ##### handle sub-sampling #### - - assert not (p.use_conv_subsampler and p.use_stacking_subsampler), \ - 'Please use only one form of time subsampling' - - if p.use_conv_subsampler: - self.CreateChild('sub', p.conv_subsampler.Copy()) - else: - assert p.use_stacking_subsampler, 'Need one stacking module' - self.CreateChild('sub', p.stacking_subsampler.Copy()) - stack_out_feats = self.sub.output_dim - - ##### handle encoding ##### - if p.lstm_block is not None: - if p.lstm_block.input_feats is None: - p.lstm_block.input_feats = stack_out_feats - - assert p.lstm_block.input_feats == stack_out_feats - self.CreateChildren('enc', p.lstm_block.Copy()) - - @property - def output_dim(self): - return self.enc.output_dim - - @property - def _use_functional(self): - return True - - @property - def supports_streaming(self): - return False - - def zero_state(self, theta, batch_size): - return py_utils.NestedMap() - - def FProp(self, theta, batch, state0=None): - """Encodes source as represented by 'inputs' and 'paddings'. - Args: - theta: A NestedMap object containing weights' values of this - layer and its children layers. - batch: A NestedMap with fields: - - src_inputs - The inputs tensor. It is expected to be of shape [batch, - time, feature_dim, channels]. - - paddings - The paddings tensor. It is expected to be of shape [batch, - time]. - state0: Recurrent input state. Not supported/ignored by this encoder. - Returns: - A NestedMap containing - - 'encoded': a feature tensor of shape [time, batch, depth] - - 'padding': a 0/1 tensor of shape [time, batch] - - 'state': the updated recurrent state - """ - p = self.params - inputs, paddings = batch.src_inputs, batch.paddings - - with tf.name_scope(p.name): - - if p.use_specaugment and not self.do_eval: - inputs, paddings = self.specaugment.FProp(theta.specaugment, inputs, - paddings) - - inputs, paddings = self.sub.FProp(theta.sub, inputs, paddings) - - encoded, padding = self.enc.FProp(theta.enc, inputs, paddings) - - return py_utils.NestedMap(encoded=encoded, - padding=padding, - state=py_utils.NestedMap()) diff --git a/lingvo/tasks/asr/frontend.py b/lingvo/tasks/asr/frontend.py deleted file mode 100644 index 1184bb10..00000000 --- a/lingvo/tasks/asr/frontend.py +++ /dev/null @@ -1,480 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Layers to construct an ASR frontend.""" - -import collections -import math -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils - - -# AsrFrontendConfig which defines characteristics of the frontend that may -# be relevant to interfacing code which needs to reason about inputs and -# outputs. -# Fields: -# is_null: Whether this is the NullAsrFrontend. -# src_type: Interpretation of the src_inputs. Can be one of 'none' or 'pcm'. -# src_pcm_scale: If src_type is 'pcm', then this is the scale of each sample. -# If normalized, this should be 1.0. If unnormalized from int16, then it -# should be 32768.0. -# src_pcm_sample_rate: Sample rate of the expected src PCM frames. -# output_dim: Dimension of the output. Typically the number of mel bands -# or equiv. May be -1 for unknown. -# input_frame_ratio: Approximate ratio of the number of -# input_frames / output_frames. Intended to be multiplied by output frames -# (i.e. as part of bucket_bounds to arrive at input frames to the frontend). -AsrFrontendConfig = collections.namedtuple('AsrFrontendConfig', [ - 'is_null', - 'src_type', - 'src_pcm_scale', - 'src_pcm_sample_rate', - 'output_dim', - 'input_frame_ratio', -]) - - -def _NextPowerOfTwo(i): - return math.pow(2, math.ceil(math.log(i, 2))) - - -class BaseAsrFrontend(base_layer.BaseLayer): - """Base class for ASR frontends. - - An ASR frontend is responsible for performing feature extraction from the - input in the cases where features are not precomputed as part of the - dataset. In such cases, it would be typical for the input to consist of - waveform data in some form. - """ - - @property - def config(self): - """Returns the AsrFrontendConfig namedtuple for this instance.""" - return self.GetConfigFromParams(self.params) - - @staticmethod - def GetConfigFromParams(params): - """Returns an AsrFrontendConfig namedtuple with vital config settings.""" - raise NotImplementedError() - - def FProp(self, theta, input_batch): - """Generates ASR features for a batch. - - Shapes of the input_batch and output are dependent on the implementation - and should be paired with the model's input format and encoder expectations. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - input_batch: A NestedMap with fields: - - - 'src_inputs' - The inputs tensor, - compatible with model input. Expected to be of shape - [batch, time, ...]. - - 'paddings' - The paddings tensor. It is expected to be of shape - [batch, time]. - - Returns: - NestedMap of encoder inputs which can be passed directly to a - compatible encoder and contains: - - - 'src_inputs': inputs to the encoder, minimally of shape - [batch, time, ...]. - - 'paddings': a 0/1 tensor of shape [batch, time]. - """ - raise NotImplementedError() - - -class NullAsrFrontend(BaseAsrFrontend): - """ASR frontend that just returns its input as FProp output.""" - - @staticmethod - def GetConfigFromParams(params): - """Returns an AsrFrontendConfig namedtuple with vital config settings.""" - return AsrFrontendConfig( - is_null=True, - src_type='none', - src_pcm_sample_rate=-1, - src_pcm_scale=1.0, - output_dim=-1, - input_frame_ratio=1.0) - - def FProp(self, theta, input_batch): - return input_batch.DeepCopy() - - -class MelAsrFrontend(BaseAsrFrontend): - """An AsrFrontend that implements mel feature extraction from PCM frames. - - This is expressed in pure TensorFlow and without reference to external - resources. - - The frontend implements the following stages: - `Framer -> Window -> FFT -> FilterBank -> MeanStdDev -> SubSample` - - The FProp input to this layer can either have rank 3 or rank 4 shape: - [batch_size, timestep, packet_size, channel_count] - [batch_size, timestep * packet_size, channel_count] - - For compatibility with existing code, 2D [batch_size, timestep] mono shapes - are also supported. - - In the common case, the packet_size is 1. The 4D variant is accepted for - glueless interface to input generators that frame their input samples in - some way. The external framing choice does not influence the operation of - this instance, but it is accepted. - - TODO(laurenzo): Refactor call sites to uniformly use the 4D variant and - eliminate fallback logic in this class. - - Only 1 channel is currently supported. - TODO(laurenzo): Refactor this class to operate on multi-channel inputs. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'frontend' - p.Define('sample_rate', 16000.0, 'Sample rate in Hz') - p.Define('channel_count', 1, 'Number of channels.') - p.Define('frame_size_ms', 25.0, - 'Amount of data grabbed for each frame during analysis') - p.Define('frame_step_ms', 10.0, 'Number of ms to jump between frames') - p.Define('num_bins', 80, 'Number of bins in the mel-spectrogram output') - p.Define('lower_edge_hertz', 125.0, - 'The lowest frequency of the mel-spectrogram analsis') - p.Define('upper_edge_hertz', 7600.0, - 'The highest frequency of the mel-spectrogram analsis') - p.Define('preemph', 0.97, - 'The first-order filter coefficient used for preemphasis') - p.Define('noise_scale', 8.0, - 'The amount of noise (in 16-bit LSB units) to add') - p.Define('window_fn', 'HANNING', - 'Window function to apply (valid values are "HANNING", and None)') - p.Define( - 'pad_end', False, - 'Whether to pad the end of `signals` with zeros when the provided ' - 'frame length and step produces a frame that lies partially past ' - 'its end.') - p.Define( - 'per_bin_mean', None, - 'Per-bin (num_bins) means for normalizing the spectrograms. ' - 'Defaults to zeros.') - p.Define('per_bin_stddev', None, - 'Per-bin (num_bins) standard deviations. Defaults to ones.') - p.Define('stack_left_context', 0, 'Number of left context frames to stack.') - p.Define('stack_right_context', 0, - 'Number of right context frames to stack.') - p.Define('frame_stride', 1, 'The frame stride for sub-sampling.') - - return p - - @staticmethod - def GetConfigFromParams(params): - """Returns an AsrFrontendConfig namedtuple with vital config settings.""" - context_size = params.stack_left_context + params.stack_right_context + 1 - subsample_factor = params.num_bins * context_size - frame_step = round(params.sample_rate * params.frame_step_ms / 1000.0) - return AsrFrontendConfig( - is_null=False, - src_type='pcm', - src_pcm_scale=32768.0, - src_pcm_sample_rate=16000.0, - output_dim=subsample_factor, - input_frame_ratio=frame_step * subsample_factor) - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.frame_stride < 1: - raise ValueError('frame_stride must be positive.') - - assert p.channel_count == 1, 'Only 1 channel currently supported.' - # Make sure key params are in floating point. - p.sample_rate = float(p.sample_rate) - p.frame_step_ms = float(p.frame_step_ms) - p.frame_size_ms = float(p.frame_size_ms) - p.lower_edge_hertz = float(p.lower_edge_hertz) - p.upper_edge_hertz = float(p.upper_edge_hertz) - - self._frame_step = int(round(p.sample_rate * p.frame_step_ms / 1000.0)) - self._frame_size = (int(round(p.sample_rate * p.frame_size_ms / 1000.0)) + 1 - ) # +1 for the preemph - # Overdrive means double FFT size. - # Note: 2* because of overdrive - self._fft_size = 2 * int(max(512, _NextPowerOfTwo(self._frame_size))) - - self._CreateWindowFunction() - - # Mean/stddev. - if p.per_bin_mean is None: - p.per_bin_mean = [0.0] * p.num_bins - if p.per_bin_stddev is None: - p.per_bin_stddev = [1.0] * p.num_bins - assert len(p.per_bin_mean) == p.num_bins - assert len(p.per_bin_stddev) == p.num_bins - - def _CreateWindowFunction(self): - p = self.params - if p.window_fn is None: - self._window_fn = None - elif p.window_fn == 'HANNING': - - def _HanningWindow(frame_size, dtype): - return tf.signal.hann_window(frame_size, dtype=dtype) - - self._window_fn = _HanningWindow - else: - raise ValueError('Illegal value %r for window_fn param' % p.window_fn) - - @property - def window_frame_size(self): - return self._frame_size - - @property - def window_frame_step(self): - return self._frame_step - - def _RemoveChannelDim(self, pcm_audio_data): - if pcm_audio_data.shape.rank == 3: - pcm_audio_data = tf.squeeze(pcm_audio_data, 2) - assert pcm_audio_data.shape.rank == 2, ( - 'MelAsrFrontend only supports one channel') - return pcm_audio_data - - def _ReshapeToMono2D(self, pcm_audio_data, paddings): - """Reshapes a 3D or 4D input to 2D. - - Since the input to FProp can be 3D or 4D (see class comments), this will - collapse it back to a 2D, mono shape for internal processing. - - Args: - pcm_audio_data: 2D, 3D or 4D audio input. See class comments. Must have a - rank. - paddings: Original paddings shaped to the first two dims of - pcm_audio_data. - - Returns: - Tuple of 2D [batch_size, timestep] mono audio data, new paddings. - """ - shape = py_utils.GetShape(pcm_audio_data) - rank = len(shape) - if rank == 2: - return pcm_audio_data, paddings - elif rank == 3: - # [batch, time, channel] - with tf.control_dependencies([tf.assert_equal(shape[2], 1)]): - return tf.squeeze(pcm_audio_data, axis=2), paddings - elif rank == 4: - # [batch, time, packet, channel] - batch_size, orig_time, orig_packet_size, channel = shape - time = orig_time * orig_packet_size - with tf.control_dependencies([tf.assert_equal(channel, 1)]): - pcm_audio_data = tf.reshape(pcm_audio_data, (batch_size, time)) - # Transform paddings into the new time base with a padding per time - # step vs per packet by duplicating each packet. - paddings = tf.reshape( - tf.tile(tf.expand_dims(paddings, axis=2), [1, 1, orig_packet_size]), - (batch_size, time)) - return pcm_audio_data, paddings - else: - raise ValueError('Illegal pcm_audio_data shape') - - def FProp(self, theta, input_batch): - """Perform signal processing on a sequence of PCM data. - - NOTE: This implementation does not currently support paddings, and they - are accepted for compatibility with the super-class. - - TODO(laurenzo): Rework this to support paddings. - - Args: - theta: Layer theta. - input_batch: PCM input map: - - # int16 and float32 are both permissible - - 'src_inputs': int16 or float32 tensor of PCM audio data, scaled to - +/-32768 (versus [-1..1)!). See class comments for supported input - shapes. - - 'paddings': per frame 0/1 paddings. Shaped: [batch, frame]. - Returns: - NestedMap of encoder inputs which can be passed directly to a - compatible encoder and contains: - - - 'src_inputs': inputs to the encoder, minimally of shape - [batch, time, ...]. - - 'paddings': a 0/1 tensor of shape [batch, time]. - """ - - return self._FPropDefault(input_batch) - - def _FPropDefault(self, input_batch): - pcm_audio_data, pcm_audio_paddings = self._ReshapeToMono2D( - input_batch.src_inputs, input_batch.paddings) - - mel_spectrogram, mel_spectrogram_paddings = self._FPropChunk( - pcm_audio_data, pcm_audio_paddings) - - mel_spectrogram, mel_spectrogram_paddings = self._PadAndReshapeSpec( - mel_spectrogram, mel_spectrogram_paddings) - - return py_utils.NestedMap( - src_inputs=mel_spectrogram, paddings=mel_spectrogram_paddings) - - def _StackSignal(self, signal, stack_size, stride): - signal = tf.signal.frame( - signal=signal, - frame_length=stack_size, - frame_step=stride, - pad_end=False, - axis=1, - ) - signal = tf.reshape(signal, py_utils.GetShape(signal)[:2] + [-1]) - return signal - - def _PadAndReshapeSpec(self, mel_spectrogram, mel_spectrogram_paddings): - p = self.params - # Stack and sub-sample. - stack_size = 1 - if p.stack_left_context > 0: - # Since left context is leading, pad the left by duplicating the first - # frame. - stack_size += p.stack_left_context - mel_spectrogram = tf.concat( - [mel_spectrogram[:, 0:1, :]] * p.stack_left_context + - [mel_spectrogram], - axis=1) - mel_spectrogram_paddings = tf.concat( - [mel_spectrogram_paddings[:, 0:1]] * p.stack_left_context + - [mel_spectrogram_paddings], - axis=1) - - if p.stack_right_context > 0: - stack_size += p.stack_right_context - mel_spectrogram = tf.concat( - [mel_spectrogram] + - [mel_spectrogram[:, -1:, :]] * p.stack_right_context, - axis=1) - mel_spectrogram_paddings = tf.concat( - [mel_spectrogram_paddings] + - [mel_spectrogram_paddings[:, -1:]] * p.stack_right_context, - axis=1) - - if p.stack_left_context or p.stack_right_context: - mel_spectrogram = self._StackSignal(mel_spectrogram, stack_size, - p.frame_stride) - mel_spectrogram_paddings = self._StackSignal(mel_spectrogram_paddings, - stack_size, p.frame_stride) - # After stacking paddings, pad if any source frame was padded. - # Stacks into [batch_size, stacked_frame_dim, stack_size] like the - # spectrogram stacking above, and then reduces the stack_size dim - # to the max (effectively, making padding = 1.0 if any of the pre-stacked - # frames were 1.0). Final shape is [batch_size, stacked_frame_dim]. - mel_spectrogram_paddings = tf.reduce_max(mel_spectrogram_paddings, axis=2) - - # Add feature dim. Shape = [batch, time, features, 1] - mel_spectrogram = tf.expand_dims(mel_spectrogram, -1) - return mel_spectrogram, mel_spectrogram_paddings - - def _ApplyPreemphasis(self, framed_signal): - p = self.params - preemphasized = ( - framed_signal[:, :, 1:] - p.preemph * framed_signal[:, :, 0:-1]) - return preemphasized - - def _GetMelPadding(self, pcm_audio_paddings): - p = self.params - # shape: [batch, time, _frame_size] - framed_paddings = tf.signal.frame(pcm_audio_paddings, self._frame_size, - self._frame_step, p.pad_end) - # Pad spectrograms that have any padded frames. - mel_spectrogram_paddings = tf.reduce_max(framed_paddings, axis=2) - return mel_spectrogram_paddings - - def _FPropChunk(self, pcm_audio_chunk, pcm_audio_paddings): - p = self.params - pcm_audio_chunk = tf.cast(pcm_audio_chunk, tf.float32) - # shape: [batch, time, _frame_size] - framed_signal = tf.signal.frame(pcm_audio_chunk, self._frame_size, - self._frame_step, p.pad_end) - - # Pre-emphasis. - if p.preemph != 1.0: - preemphasized = self._ApplyPreemphasis(framed_signal) - else: - preemphasized = framed_signal[:-1] - - # Noise. - if p.noise_scale > 0.0: - noise_signal = tf.random.normal( - tf.shape(preemphasized), - stddev=p.noise_scale, - mean=0.0, - seed=p.random_seed) - else: - noise_signal = 0.0 - - # Apply window fn. - windowed_signal = preemphasized + noise_signal - if self._window_fn is not None: - window = self._window_fn(self._frame_size - 1, framed_signal.dtype) - windowed_signal *= window - - mel_spectrogram = self._MelSpectrogram(windowed_signal) - - output_floor = 1.0 - mel_spectrogram_log = tf.math.log( - tf.maximum(float(output_floor), mel_spectrogram)) - - # Mean and stddev. - mel_spectrogram_norm = ( - (mel_spectrogram_log - tf.convert_to_tensor(p.per_bin_mean)) / - tf.convert_to_tensor(p.per_bin_stddev)) - return mel_spectrogram_norm, self._GetMelPadding(pcm_audio_paddings) - - def _MelSpectrogram(self, signal): - """Computes the mel spectrogram from a waveform signal. - - Args: - signal: f32 Tensor, shaped [batch_size, num_samples] - - Returns: - f32 features Tensor, shaped [batch_size, num_frames, mel_channels] - """ - p = self.params - # FFT. - real_frequency_spectrogram = tf.signal.rfft(signal, [self._fft_size]) - magnitude_spectrogram = tf.abs(real_frequency_spectrogram) - - # Shape of magnitude_spectrogram is num_frames x (fft_size/2+1) - # Mel_weight is [num_spectrogram_bins, num_mel_bins] - mel_weight_matrix = tf.signal.linear_to_mel_weight_matrix( - num_mel_bins=p.num_bins, - num_spectrogram_bins=self._fft_size // 2 + 1, - sample_rate=p.sample_rate, - lower_edge_hertz=p.lower_edge_hertz, - upper_edge_hertz=p.upper_edge_hertz, - dtype=tf.float32) - # Weight matrix implemented in the magnitude domain. - batch_size, num_frames, fft_channels = py_utils.GetShape( - magnitude_spectrogram, 3) - mel_spectrogram = tf.matmul( - tf.reshape(magnitude_spectrogram, - [batch_size * num_frames, fft_channels]), mel_weight_matrix) - mel_spectrogram = tf.reshape(mel_spectrogram, - [batch_size, num_frames, p.num_bins]) - - return mel_spectrogram diff --git a/lingvo/tasks/asr/frontend_test.py b/lingvo/tasks/asr/frontend_test.py deleted file mode 100644 index 458f82a4..00000000 --- a/lingvo/tasks/asr/frontend_test.py +++ /dev/null @@ -1,480 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for asr frontend.""" - -import lingvo.compat as tf -from lingvo.core import py_utils -from lingvo.core import test_helper -from lingvo.core import test_utils -from lingvo.tasks.asr import frontend -import numpy as np - - -class AsrFrontendTest(test_utils.TestCase): - - def _GetPcm(self): - """Gets sample wav file pcm samples. - - Returns: - (sample_rate, mono_audio) where mono_audio is of shape - [batch (=1), samples]. - """ - with open( - test_helper.test_src_dir_path('tools/testdata/gan_or_vae.wav'), - 'rb') as f: - wavdata = f.read() - result = tf.audio.decode_wav(wavdata) - # Remove the last dimension: channel is 1. - audio = py_utils.HasShape(result.audio, [75900, 1]) - audio = tf.squeeze(audio, axis=1) - # Returns audio as batch-major data with a single batch. - return result.sample_rate, tf.expand_dims(audio, axis=0) - - def _CreateFrontendParams(self): - p = frontend.MelAsrFrontend.Params() - p.sample_rate = 24000. - p.num_bins = 2 - p.noise_scale = 0. - self.params = p - - def testNullAsrFrontendConfig(self): - p = frontend.NullAsrFrontend.Params() - p.name = 'null' - fe = p.Instantiate() - config = fe.config - self.assertTrue(config.is_null) - self.assertEqual(config.src_type, 'none') - self.assertEqual(config.output_dim, -1) - self.assertEqual(config.input_frame_ratio, 1.0) - - def testMelFeaturesUnstackedConfig(self): - self._CreateFrontendParams() - p = self.params - fe = p.Instantiate() - config = fe.config - self.assertFalse(config.is_null) - self.assertEqual(config.src_type, 'pcm') - self.assertEqual(config.src_pcm_scale, 32768.0) - self.assertEqual(config.src_pcm_sample_rate, 16000.0) - self.assertEqual(config.output_dim, 2) - # Approx 34 output frames per second. - self.assertEqual(config.input_frame_ratio, 480.0) - - def testMelFeaturesLeftStackedConfig(self): - self._CreateFrontendParams() - p = self.params - p.stack_left_context = 2 - p.frame_stride = p.stack_left_context + 1 - fe = p.Instantiate() - config = fe.config - self.assertFalse(config.is_null) - self.assertEqual(config.src_type, 'pcm') - self.assertEqual(config.src_pcm_scale, 32768.0) - self.assertEqual(config.src_pcm_sample_rate, 16000.0) - self.assertEqual(config.output_dim, 6) - # Approx 12 output frames per second. - self.assertEqual(config.input_frame_ratio, 1440.0) - - def testMelFeaturesLeftRightStackedConfig(self): - self._CreateFrontendParams() - p = self.params - p.stack_right_context = 2 - p.stack_left_context = 2 - p.frame_stride = p.stack_right_context + p.stack_left_context + 1 - fe = p.Instantiate() - config = fe.config - # Approx 6 output frames per second. - self.assertEqual(config.input_frame_ratio, 2400.0) - - def testMelFeaturesUnstacked(self): - self._CreateFrontendParams() - p = self.params - mel_frontend = p.Instantiate() - sample_rate, pcm = self._GetPcm() - pcm *= 32768 - - # Convert to 4D [batch, time, packet, channels]. - sample_count = tf.shape(pcm)[1] - packet_size = 11 # A non-round number. - trimmed_pcm = pcm[:, 0:(sample_count // packet_size) * packet_size] - src_inputs = tf.reshape(trimmed_pcm, (1, -1, packet_size, 1)) - paddings = tf.zeros(tf.shape(src_inputs)[0:2]) - - outputs = mel_frontend.FPropDefaultTheta( - py_utils.NestedMap(src_inputs=src_inputs, paddings=paddings)) - log_mel = outputs.src_inputs - paddings = outputs.paddings - with self.session(): - pcm = self.evaluate(pcm) - tf.logging.info('pcm: ~ %s = %s', pcm.shape, pcm) - self.assertGreater(33000, np.amax(pcm)) - self.assertGreater(np.amax(pcm), 2.) - log_mel, paddings, sample_rate = self.evaluate( - [log_mel, paddings, sample_rate]) - self.assertEqual(sample_rate, p.sample_rate) - self.assertEqual(paddings.shape, log_mel.shape[0:2]) - self.assertAllEqual(paddings, np.zeros_like(paddings)) - # log_mel ~ [batch, time, feature_size, channel] - tf.logging.info('mel ~ %s', log_mel.shape) - self.assertEqual(log_mel.shape[2], 2) # 2 bins - # Squeeze the batch and channel dimensions out. - log_mel = np.squeeze(log_mel, axis=(0, 3)) - t = log_mel.shape[0] - mu = np.sum(log_mel, axis=0) / t - d = log_mel - mu - v = np.sum(d * d, axis=0) / (t - 1) - s = np.sqrt(v) - tf.logging.info('Found mean = %s', mu) - tf.logging.info('Found stddev = %s', s) - ref_unstacked_mean = [13.46184731, 13.30099297] - ref_unstacked_stddev = [1.3840059, 1.24434352] - self.assertAllClose(mu, ref_unstacked_mean, atol=1e-4) - self.assertAllClose(s, ref_unstacked_stddev, atol=1e-3) - - def testMelFeaturesLeftStacked(self): - self._CreateFrontendParams() - p = self.params - p.stack_left_context = 2 - p.frame_stride = p.stack_left_context + 1 - mel_frontend = p.Instantiate() - sample_rate, pcm = self._GetPcm() - pcm *= 32768 - - # Convert to 4D [batch, time, packet, channels]. - sample_count = tf.shape(pcm)[1] - packet_size = 11 # A non-round number. - trimmed_pcm = pcm[:, 0:(sample_count // packet_size) * packet_size] - src_inputs = tf.reshape(trimmed_pcm, (1, -1, packet_size, 1)) - paddings = tf.zeros(tf.shape(src_inputs)[0:2]) - - outputs = mel_frontend.FPropDefaultTheta( - py_utils.NestedMap(src_inputs=src_inputs, paddings=paddings)) - log_mel = outputs.src_inputs - paddings = outputs.paddings - with self.session(): - pcm = self.evaluate(pcm) - tf.logging.info('pcm: ~ %s = %s', pcm.shape, pcm) - self.assertGreater(33000, np.amax(pcm)) - self.assertGreater(np.amax(pcm), 2.) - log_mel, paddings, sample_rate = self.evaluate( - [log_mel, paddings, sample_rate]) - self.assertEqual(sample_rate, p.sample_rate) - self.assertEqual(paddings.shape, log_mel.shape[0:2]) - self.assertAllEqual(paddings, np.zeros_like(paddings)) - # log_mel ~ [batch, time, feature_size, channel] - tf.logging.info('mel ~ %s', log_mel.shape) - # Squeeze the batch and channel dimensions out. - log_mel = np.squeeze(log_mel, axis=(0, 3)) - t = log_mel.shape[0] - mu = np.sum(log_mel, axis=0) / t - d = log_mel - mu - v = np.sum(d * d, axis=0) / (t - 1) - s = np.sqrt(v) - tf.logging.info('Found mean = %s', mu) - tf.logging.info('Found stddev = %s', s) - ref_mean = (13.38236332, 13.2698698, 13.45229626, 13.26469517, - 13.46731281, 13.31649303) - ref_stddev = (1.52104115, 1.27433181, 1.41266346, 1.27072334, 1.41251481, - 1.28583682) - self.assertAllClose(mu, ref_mean, atol=1e-4) - self.assertAllClose(s, ref_stddev, atol=1e-3) - - def testMelFeaturesRightStacked(self): - self._CreateFrontendParams() - p = self.params - p.stack_right_context = 2 - p.frame_stride = p.stack_right_context + 1 - mel_frontend = p.Instantiate() - sample_rate, pcm = self._GetPcm() - pcm *= 32768 - - # Convert to 4D [batch, time, packet, channels]. - sample_count = tf.shape(pcm)[1] - packet_size = 11 # A non-round number. - trimmed_pcm = pcm[:, 0:(sample_count // packet_size) * packet_size] - src_inputs = tf.reshape(trimmed_pcm, (1, -1, packet_size, 1)) - paddings = tf.zeros(tf.shape(src_inputs)[0:2]) - - outputs = mel_frontend.FPropDefaultTheta( - py_utils.NestedMap(src_inputs=src_inputs, paddings=paddings)) - log_mel = outputs.src_inputs - paddings = outputs.paddings - with self.session(): - pcm = self.evaluate(pcm) - tf.logging.info('pcm: ~ %s = %s', pcm.shape, pcm) - self.assertGreater(33000, np.amax(pcm)) - self.assertGreater(np.amax(pcm), 2.) - log_mel, paddings, sample_rate = self.evaluate( - [log_mel, paddings, sample_rate]) - self.assertEqual(sample_rate, p.sample_rate) - self.assertEqual(paddings.shape, log_mel.shape[0:2]) - self.assertAllEqual(paddings, np.zeros_like(paddings)) - # log_mel ~ [batch, time, feature_size, channel] - tf.logging.info('mel ~ %s', log_mel.shape) - # Squeeze the batch and channel dimensions out. - log_mel = np.squeeze(log_mel, axis=(0, 3)) - t = log_mel.shape[0] - mu = np.sum(log_mel, axis=0) / t - d = log_mel - mu - v = np.sum(d * d, axis=0) / (t - 1) - s = np.sqrt(v) - tf.logging.info('Found mean = %s', mu) - tf.logging.info('Found stddev = %s', s) - ref_mean = (13.46731281, 13.31649303, 13.41263676, 13.28540039, - 13.48256969, 13.2802248) - ref_stddev = (1.41251481, 1.28583682, 1.43964291, 1.23710775, 1.32300735, - 1.23345602) - self.assertAllClose(mu, ref_mean, atol=1e-4) - self.assertAllClose(s, ref_stddev, atol=1e-3) - - def testMelFeaturesPaddedLeftStacked(self): - self._CreateFrontendParams() - p = self.params - p.stack_left_context = 2 - p.frame_stride = p.stack_left_context + 1 - mel_frontend = p.Instantiate() - sample_rate, pcm = self._GetPcm() - pcm *= 32768 - - # Convert to 4D [batch, time, packet, channels]. - sample_count = tf.shape(pcm)[1] - packet_size = 11 # A non-round number. - trimmed_pcm = pcm[:, 0:(sample_count // packet_size) * packet_size] - src_inputs = tf.reshape(trimmed_pcm, (1, -1, packet_size, 1)) - - # Create paddings such that the first 455 packets are unpadded. - paddings = tf.concat([ - tf.zeros([1, 455], dtype=tf.float32), - tf.ones([1, tf.shape(src_inputs)[1] - 455], dtype=tf.float32) - ], - axis=1) - # frame_step=240, frame_size=601, +1202 left padded frames - # 455 packets * 11 frames rounds = 5005 frames, rounds down to 21 mel - # frames. Divide by 3 for stacking = 7. - expected_unpadded = 7 - - outputs = mel_frontend.FPropDefaultTheta( - py_utils.NestedMap(src_inputs=src_inputs, paddings=paddings)) - log_mel = outputs.src_inputs - paddings = outputs.paddings - with self.session(): - pcm = self.evaluate(pcm) - tf.logging.info('pcm: ~ %s = %s', pcm.shape, pcm) - self.assertGreater(33000, np.amax(pcm)) - self.assertGreater(np.amax(pcm), 2.) - log_mel, paddings, sample_rate = self.evaluate( - [log_mel, paddings, sample_rate]) - self.assertEqual(sample_rate, p.sample_rate) - self.assertEqual(paddings.shape, log_mel.shape[0:2]) - self.assertAllEqual(paddings[:, 0:expected_unpadded], - np.zeros([1, expected_unpadded])) - self.assertAllEqual(paddings[:, expected_unpadded:], - np.ones([1, paddings.shape[1] - expected_unpadded])) - # log_mel ~ [batch, time, feature_size, channel] - tf.logging.info('mel ~ %s', log_mel.shape) - # Squeeze the batch and channel dimensions out. - log_mel = np.squeeze(log_mel, axis=(0, 3)) - t = log_mel.shape[0] - mu = np.sum(log_mel, axis=0) / t - d = log_mel - mu - v = np.sum(d * d, axis=0) / (t - 1) - s = np.sqrt(v) - tf.logging.info('Found mean = %s', mu) - tf.logging.info('Found stddev = %s', s) - ref_mean = (13.38236332, 13.2698698, 13.45229626, 13.26469517, - 13.46731281, 13.31649303) - ref_stddev = (1.52104115, 1.27433181, 1.41266346, 1.27072334, 1.41251481, - 1.28583682) - self.assertAllClose(mu, ref_mean, atol=1e-4) - self.assertAllClose(s, ref_stddev, atol=1e-3) - - def testMelFeaturesPaddedRightStacked(self): - self._CreateFrontendParams() - p = self.params - p.stack_right_context = 2 - p.frame_stride = p.stack_right_context + 1 - mel_frontend = p.Instantiate() - sample_rate, pcm = self._GetPcm() - pcm *= 32768 - - # Convert to 4D [batch, time, packet, channels]. - sample_count = tf.shape(pcm)[1] - packet_size = 11 # A non-round number. - trimmed_pcm = pcm[:, 0:(sample_count // packet_size) * packet_size] - src_inputs = tf.reshape(trimmed_pcm, (1, -1, packet_size, 1)) - - # Create paddings such that the first 455 packets are unpadded. - paddings = tf.concat([ - tf.zeros([1, 455], dtype=tf.float32), - tf.ones([1, tf.shape(src_inputs)[1] - 455], dtype=tf.float32) - ], - axis=1) - # frame_step=240, frame_size=600, +1200 right padded frames - # 455 packets * 11 frames rounds = 5005 frames, rounds down to 21 mel - # frames. Divide by 3 for stacking = 7. - # TODO(talremez): Make sure with this makes sense. - expected_unpadded = 6 - - outputs = mel_frontend.FPropDefaultTheta( - py_utils.NestedMap(src_inputs=src_inputs, paddings=paddings)) - log_mel = outputs.src_inputs - paddings = outputs.paddings - - with self.session(): - pcm = self.evaluate(pcm) - tf.logging.info('pcm: ~ %s = %s', pcm.shape, pcm) - self.assertGreater(33000, np.amax(pcm)) - self.assertGreater(np.amax(pcm), 2.) - log_mel, paddings, sample_rate = self.evaluate( - [log_mel, paddings, sample_rate]) - self.assertEqual(sample_rate, p.sample_rate) - self.assertEqual(paddings.shape, log_mel.shape[0:2]) - self.assertAllEqual(paddings[:, 0:expected_unpadded], - np.zeros([1, expected_unpadded])) - self.assertAllEqual(paddings[:, expected_unpadded:], - np.ones([1, paddings.shape[1] - expected_unpadded])) - - def testMelMeanVarNormalization(self): - self._CreateFrontendParams() - p = self.params - p.stack_left_context = 2 - p.frame_stride = p.stack_left_context + 1 - ref_mean = (13.38236332, 13.2698698, 13.45229626, 13.26469517, 13.46731281, - 13.31649303) - ref_stddev = (1.52104115, 1.27433181, 1.41266346, 1.27072334, 1.41251481, - 1.28583682) - p.per_bin_mean = ref_mean[:p.num_bins] - p.per_bin_stddev = ref_stddev[:p.num_bins] - mel_frontend = p.Instantiate() - _, pcm = self._GetPcm() - pcm *= 32768 - - # Convert to 4D [batch, time, packet, channels]. - sample_count = tf.shape(pcm)[1] - packet_size = 11 # A non-round number. - trimmed_pcm = pcm[:, 0:(sample_count // packet_size) * packet_size] - src_inputs = tf.reshape(trimmed_pcm, (1, -1, packet_size, 1)) - paddings = tf.zeros(tf.shape(src_inputs)[0:2]) - - outputs = mel_frontend.FPropDefaultTheta( - py_utils.NestedMap(src_inputs=src_inputs, paddings=paddings)) - log_mel = outputs.src_inputs - with self.session(): - log_mel = self.evaluate(log_mel) - # log_mel ~ [batch, time, feature_size, channel] - tf.logging.info('mel ~ %s', log_mel.shape) - # Squeeze the batch and channel dimensions out. - log_mel = np.squeeze(log_mel, axis=(0, 3)) - t = log_mel.shape[0] - mu = np.sum(log_mel, axis=0) / t - d = log_mel - mu - v = np.sum(d * d, axis=0) / (t - 1) - s = np.sqrt(v) - # Only take the base bin values: - mu = mu[:p.num_bins] - s = s[:p.num_bins] - self.assertAllClose(mu, np.zeros_like(mu), atol=1e-4) - self.assertAllClose(s, np.ones_like(s), atol=1e-3) - - def testMelFeaturesUnstacked2D(self): - # TODO(laurenzo): Remove this test once 2D inputs support removed. - self._CreateFrontendParams() - p = self.params - mel_frontend = p.Instantiate() - sample_rate, pcm = self._GetPcm() - pcm *= 32768 - - # Leave in 2D [batch, time]. - src_inputs = pcm - paddings = tf.zeros(tf.shape(src_inputs)[0:2]) - - outputs = mel_frontend.FPropDefaultTheta( - py_utils.NestedMap(src_inputs=src_inputs, paddings=paddings)) - log_mel = outputs.src_inputs - paddings = outputs.paddings - with self.session(): - pcm = self.evaluate(pcm) - tf.logging.info('pcm: ~ %s = %s', pcm.shape, pcm) - self.assertGreater(33000, np.amax(pcm)) - self.assertGreater(np.amax(pcm), 2.) - log_mel, paddings, sample_rate = self.evaluate( - [log_mel, paddings, sample_rate]) - self.assertEqual(sample_rate, p.sample_rate) - self.assertEqual(paddings.shape, log_mel.shape[0:2]) - self.assertAllEqual(paddings, np.zeros_like(paddings)) - # log_mel ~ [batch, time, feature_size, channel] - tf.logging.info('mel ~ %s', log_mel.shape) - self.assertEqual(log_mel.shape[2], 2) # 2 bins - # Squeeze the batch and channel dimensions out. - log_mel = np.squeeze(log_mel, axis=(0, 3)) - t = log_mel.shape[0] - mu = np.sum(log_mel, axis=0) / t - d = log_mel - mu - v = np.sum(d * d, axis=0) / (t - 1) - s = np.sqrt(v) - tf.logging.info('Found mean = %s', mu) - tf.logging.info('Found stddev = %s', s) - ref_unstacked_mean = [13.46184731, 13.30099297] - ref_unstacked_stddev = [1.3840059, 1.24434352] - self.assertAllClose(mu, ref_unstacked_mean, atol=1e-4) - self.assertAllClose(s, ref_unstacked_stddev, atol=1e-3) - - def testMelFeaturesUnstacked3D(self): - # TODO(laurenzo): Remove this test once 3D inputs support removed. - self._CreateFrontendParams() - p = self.params - mel_frontend = p.Instantiate() - sample_rate, pcm = self._GetPcm() - pcm *= 32768 - - # Leave in 3D [batch, time, 1]. - src_inputs = tf.expand_dims(pcm, axis=2) - paddings = tf.zeros(tf.shape(src_inputs)[0:2]) - - outputs = mel_frontend.FPropDefaultTheta( - py_utils.NestedMap(src_inputs=src_inputs, paddings=paddings)) - log_mel = outputs.src_inputs - paddings = outputs.paddings - with self.session(): - pcm = self.evaluate(pcm) - tf.logging.info('pcm: ~ %s = %s', pcm.shape, pcm) - self.assertGreater(33000, np.amax(pcm)) - self.assertGreater(np.amax(pcm), 2.) - log_mel, paddings, sample_rate = self.evaluate( - [log_mel, paddings, sample_rate]) - self.assertEqual(sample_rate, p.sample_rate) - self.assertEqual(paddings.shape, log_mel.shape[0:2]) - self.assertAllEqual(paddings, np.zeros_like(paddings)) - # log_mel ~ [batch, time, feature_size, channel] - tf.logging.info('mel ~ %s', log_mel.shape) - self.assertEqual(log_mel.shape[2], 2) # 2 bins - # Squeeze the batch and channel dimensions out. - log_mel = np.squeeze(log_mel, axis=(0, 3)) - t = log_mel.shape[0] - mu = np.sum(log_mel, axis=0) / t - d = log_mel - mu - v = np.sum(d * d, axis=0) / (t - 1) - s = np.sqrt(v) - tf.logging.info('Found mean = %s', mu) - tf.logging.info('Found stddev = %s', s) - ref_unstacked_mean = [13.46184731, 13.30099297] - ref_unstacked_stddev = [1.3840059, 1.24434352] - self.assertAllClose(mu, ref_unstacked_mean, atol=1e-4) - self.assertAllClose(s, ref_unstacked_stddev, atol=1e-3) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/asr/fusion.py b/lingvo/tasks/asr/fusion.py deleted file mode 100644 index 7c06857a..00000000 --- a/lingvo/tasks/asr/fusion.py +++ /dev/null @@ -1,177 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Utilities for fusing language models with the decoder output.""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import layers -from lingvo.core import py_utils -from lingvo.tasks.lm import layers as lm_layers - - -class FusionBase(base_layer.BaseLayer): - """Base class for fusion with LMs.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('lm', lm_layers.NullLm.Params(), 'Language model params.') - p.Define( - 'base_model_logits_dim', None, - 'Dimension of base (i.e., the model being fused with the LM) model\'s ' - 'logits.') - return p - - def __init__(self, params): - """Initializes Fusion class.""" - try: - if isinstance(params.lm.emb, (layers.EmbeddingLayer)): - params.lm.emb.on_ps = False - except AttributeError: - pass - - super().__init__(params) - - p = self.params - self.CreateChild('lm', p.lm) - - def _CreateChildrenVariables(self): - # Backwards compatibility: manually call child.InstantiateVariables() - # outside of tf.variable_scope(p.name). - self.lm.InstantiateVariables() - super()._CreateChildrenVariables() - - def zero_state(self, theta, batch_size): - """Returns initial model state for fusion model.""" - state0 = py_utils.NestedMap() - state0.lm_states = self.lm.zero_state(theta.lm, batch_size) - return state0 - - def _FPropLm(self, theta, state0, ids, paddings, misc=None): - """LM FProp. - - Works for single step or entire seq. - - Args: - theta: A NestedMap object containing weights for the layer and its - children. - state0: A NestedMap of states (specific to the layer). - ids: Target ids, of shape [batch_size] for single step unrolling or - [batch_size, base_model_logits_dim] for the entire sequence. - paddings: Target paddings, of the same shape as 'ids'. - misc: NestedMap of miscellaneous items, which might be needed during - training. - - Returns: - (lm_output, state1): - - - lm_output: A NestedMap containing lm output. If 'ids' is 1-D, then - lm_output should have shape [batch_size, dim]; if it is 2-D then the - shape should be [seq_len, batch_size, dim]. - - state1: A NestedMap of updated states. - """ - state1 = state0.DeepCopy() - is_single_step = (ids.shape.ndims == 1) - if is_single_step: - seq_len = 1 - else: - seq_len = tf.shape(ids)[0] - - self._ModifyLmBeforeFProp(theta, state0, ids, paddings, misc) - - lm_output, state1.lm_states = self.lm.FProp( - theta.lm, tf.reshape(ids, [seq_len, -1]), - tf.reshape(paddings, [seq_len, -1]), state0.lm_states) - - if is_single_step: - # lm outputs have dimension [time, batch, dim]. Since this is only one - # step, remove time dimension. - lm_output = lm_output.Transform(lambda v: tf.squeeze(v, axis=0)) - - return lm_output, state1 - - def FProp(self, theta, state0, am_output, ids, paddings, misc=None): - """Real fusion logic happens here. - - Works for single step or for the entire sequence. - - Args: - theta: A NestedMap object containing weights for the layer and its - children. - state0: A NestedMap of states (specific to the layer). - am_output: The output from the speech model. 'am_output' can have shape - [batch_size, base_model_logits_dim] for a single step unrolling or - [seq_len, batch_size, base_model_logits_dim] for the entire sequence. - ids: Target ids, of shape [batch_size] for single step unrolling or - [batch_size, base_model_logits_dim] for the entire sequence. - paddings: Target paddings, of the same shape as 'ids'. - misc: NestedMap of miscellaneous items, which might be needed during - training. - - Returns: - (fused_output, state1): - - - fused_output: A tensor containing the fused result. If am_output is 2-D, - then the fused_output should have shape [batch_size, dim]; if - am_output is 3-D, then the shape should be [seq_len, batch_size, dim]. - - state1: a NestedMap of updated states (specific to the layer). - """ - del theta, state0, am_output, ids, paddings - raise NotImplementedError('Must be implemented by sub-classes.') - - def _ModifyLmBeforeFProp(self, theta, state0, ids, paddings, misc=None): - """Perform any LM modifications before LM FProp (no-op by default).""" - del theta, state0, ids, paddings, misc - - def ComputeLogitsWithLM(self, state, logits, is_eval=False): - """Compute resulting logits based on the fusion method. - - Args: - state: a NestedMap of states (specific to the layer). - logits: a tensor corresponds to AM logits. - is_eval: whether this is used in eval model (for example, beam search). - - Returns: - Resulting logits after fusion with the LM. - - Raises: - NotImplementedError: If method is not implemented. - """ - del state, logits, is_eval - raise NotImplementedError('Must be implemented by sub-classes.') - - def AddAdditionalDecoderSummaries(self, source_encs, source_paddings, targets, - seq_out_tas, softmax_input): - """Add any fusion related summaries (no-op by default). - - Args: - source_encs: A tensor of shape [time, batch_size, source_dim]. - source_paddings: A tensor of shape [time, batch_size]. - targets: A NestedMap containing target info. - seq_out_tas: A SequenceOutTensorArrays. - softmax_input: A tensor of shape [batch, time, vocab_size]. - """ - del source_encs, source_paddings, targets, seq_out_tas, softmax_input - - -class NullFusion(FusionBase): - """A trivial fusion layer which does nothing.""" - - def FProp(self, theta, state0, am_output, ids, paddings, misc=None): - del theta, ids, paddings - return am_output, state0 - - def ComputeLogitsWithLM(self, state, logits, is_eval=False): - return tf.nn.log_softmax(logits) if is_eval else logits diff --git a/lingvo/tasks/asr/input_generator.py b/lingvo/tasks/asr/input_generator.py deleted file mode 100644 index 6db0345b..00000000 --- a/lingvo/tasks/asr/input_generator.py +++ /dev/null @@ -1,357 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Speech recognition input generator.""" - -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import generic_input -from lingvo.core import py_utils - -from tensorflow.python.ops import inplace_ops # pylint:disable=g-direct-tensorflow-import - - -class AsrInput(base_input_generator.BaseSequenceInputGenerator): - """Input generator for ASR.""" - - @classmethod - def Params(cls): - """Defaults params for AsrInput.""" - p = super().Params() - p.Define('frame_size', 40, 'The number of coefficients in each frame.') - p.Define('append_eos_frame', True, 'Append an all-zero frame.') - p.source_max_length = 3000 - return p - - def _DataSourceFromFilePattern(self, file_pattern): - - def Proc(record): - """Parses a serialized tf.Example record.""" - # There we go! string, string, float32. I hope frames is allowed - # to be a waveform directly... - features = [ - ('uttid', tf.io.VarLenFeature(tf.string)), - ('transcript', tf.io.VarLenFeature(tf.string)), - ('frames', tf.io.VarLenFeature(tf.float32)), - ] - example = tf.io.parse_single_example(record, dict(features)) - fval = {k: v.values for k, v in example.items()} - # Reshape the flattened vector into its original time-major - # representation. - fval['frames'] = tf.reshape( - fval['frames'], shape=[-1, self.params.frame_size]) - # Input duration determines the bucket. - bucket_key = tf.cast(tf.shape(fval['frames'])[0], tf.int32) - if self.params.append_eos_frame: - bucket_key += 1 - tgt_ids, tgt_labels, tgt_paddings = self.StringsToIds(fval['transcript']) - src_paddings = tf.zeros([tf.shape(fval['frames'])[0]], dtype=tf.float32) - return [ - fval['uttid'], tgt_ids, tgt_labels, tgt_paddings, fval['frames'], - src_paddings - ], bucket_key - - return generic_input.GenericInput( - file_pattern=file_pattern, - processor=Proc, - dynamic_padding_dimensions=[0] * 6, - dynamic_padding_constants=[0] * 5 + [1], - **self.CommonInputOpArgs()) - - def _MaybePadSourceInputs(self, src_inputs, src_paddings): - p = self.params - if not p.append_eos_frame: - return src_inputs, src_paddings - - per_src_len = tf.reduce_sum(1 - src_paddings, 1) - per_src_len += 1 - max_src_len = tf.reduce_max(per_src_len) - input_shape = tf.shape(src_inputs) - input_len = tf.maximum(input_shape[1], tf.cast(max_src_len, tf.int32)) - pad_steps = input_len - input_shape[1] - src_inputs = tf.concat([ - src_inputs, - tf.zeros( - inplace_ops.inplace_update(input_shape, 1, pad_steps), - src_inputs.dtype) - ], 1) - src_paddings = 1 - tf.sequence_mask( - tf.reshape(per_src_len, [input_shape[0]]), tf.reshape(input_len, []), - src_paddings.dtype) - return src_inputs, src_paddings - - def __init__(self, params): - super().__init__(params) - p = self.params - - (utt_ids, tgt_ids, tgt_labels, tgt_paddings, src_frames, - src_paddings), self._bucket_keys = self._BuildDataSource() - - self._sample_ids = utt_ids - - src_frames, src_paddings = self._MaybePadSourceInputs( - src_frames, src_paddings) - - # We expect src_inputs to be of shape - # [batch_size, num_frames, feature_dim, channels]. - src_frames = tf.expand_dims(src_frames, axis=-1) - - # Convert target ids, labels, paddings, and weights from shape [batch_size, - # 1, num_frames] to [batch_size, num_frames] - tgt_ids = tf.squeeze(tgt_ids, axis=1) - tgt_labels = tf.squeeze(tgt_labels, axis=1) - tgt_paddings = tf.squeeze(tgt_paddings, axis=1) - - if p.pad_to_max_seq_length: - assert p.source_max_length - assert p.target_max_length - - if all(x == p.bucket_batch_limit[0] for x in p.bucket_batch_limit): - # Set the input batch size as an int rather than a tensor. - src_frames_shape = (self.InfeedBatchSize(), p.source_max_length, - p.frame_size, 1) - src_paddings_shape = (self.InfeedBatchSize(), p.source_max_length) - tgt_shape = (self.InfeedBatchSize(), p.target_max_length) - else: - tf.logging.warning( - 'Could not set static input shape since not all bucket batch sizes ' - 'are the same:', p.bucket_batch_limit) - src_frames_shape = None - src_paddings_shape = None - tgt_shape = None - - src_frames = py_utils.PadBatchDimension(src_frames, self.InfeedBatchSize(), 0) - src_paddings = py_utils.PadBatchDimension(src_paddings, self.InfeedBatchSize(), - 1) - tgt_ids = py_utils.PadBatchDimension(tgt_ids, self.InfeedBatchSize(), 0) - tgt_labels = py_utils.PadBatchDimension(tgt_labels, self.InfeedBatchSize(), 0) - tgt_paddings = py_utils.PadBatchDimension(tgt_paddings, self.InfeedBatchSize(), - 1) - - src_frames = py_utils.PadSequenceDimension( - src_frames, p.source_max_length, 0, shape=src_frames_shape) - src_paddings = py_utils.PadSequenceDimension( - src_paddings, p.source_max_length, 1, shape=src_paddings_shape) - tgt_ids = py_utils.PadSequenceDimension( - tgt_ids, p.target_max_length, 0, shape=tgt_shape) - tgt_labels = py_utils.PadSequenceDimension( - tgt_labels, p.target_max_length, 0, shape=tgt_shape) - tgt_paddings = py_utils.PadSequenceDimension( - tgt_paddings, p.target_max_length, 1, shape=tgt_shape) - - tgt = py_utils.NestedMap( - ids=tgt_ids, - labels=tgt_labels, - paddings=tgt_paddings, - weights=1.0 - tgt_paddings) - src = py_utils.NestedMap(src_inputs=src_frames, paddings=src_paddings) - - self._tgt = tgt - self._src = src - - def _InputBatch(self): - batch = py_utils.NestedMap() - batch.bucket_keys = self._bucket_keys - batch.src = self._src - batch.tgt = self._tgt - if not py_utils.use_tpu(): - batch.sample_ids = self._sample_ids - return batch - -class RawAsrInputIntegerUttIds(base_input_generator.BaseSequenceInputGenerator): - """Input generator for ASR. Assumes uttid is tf.int64""" - - PAD_INDEX = -1 - - @classmethod - def Params(cls): - """Defaults params for AsrInput.""" - p = super().Params() - p.Define('frame_size', 1, 'The number of coefficients in each frame.') - p.Define('append_eos_frame', True, 'Append an all-zero frame.') - # source_max_length is way too small by default when the length's time unit - # is in samples, given that we sample at 16kHz. - p.source_max_length = 3000 - return p - - def _DataSourceFromFilePattern(self, file_pattern): - - def Proc(record): - """Parses a serialized tf.Example record.""" - # There we go! string, string, float32. I hope frames is allowed - # to be a waveform directly... - features = [ - ('int64_uttid', tf.io.VarLenFeature(tf.int64)), - ('int64_audio_document_id', tf.io.VarLenFeature(tf.int64)), - ('num_utterances_in_audio_document', tf.io.VarLenFeature(tf.int64)), - ('transcript', tf.io.VarLenFeature(tf.string)), - ('frames', tf.io.FixedLenFeature((), tf.string)), - ] - example = tf.io.parse_single_example(record, dict(features)) - fval = {} - for k, v in example.items(): - if k == 'frames': - fval[k] = tf.cast(tf.io.decode_raw(v, tf.int16), tf.float32) - else: - assert isinstance(v, tf.SparseTensor) - fval[k] = v.values - # Reshape the flattened vector into its original time-major - # representation. - fval['frames'] = tf.reshape( - fval['frames'], shape=[-1, self.params.frame_size]) - # Input duration determines the bucket. - bucket_key = tf.cast(tf.shape(fval['frames'])[0], tf.int32) - if self.params.append_eos_frame: - bucket_key += 1 - tgt_ids, tgt_labels, tgt_paddings = self.StringsToIds(fval['transcript']) - src_paddings = tf.zeros([tf.shape(fval['frames'])[0]], dtype=tf.float32) - return [ - fval['int64_uttid'], fval['int64_audio_document_id'], - fval['num_utterances_in_audio_document'], tgt_ids, - tgt_labels, tgt_paddings, fval['frames'], - src_paddings - ], bucket_key - - return generic_input.GenericInput( - file_pattern=file_pattern, - processor=Proc, - dynamic_padding_dimensions=[0] * 8, - dynamic_padding_constants=[0] * 7 + [1], - **self.CommonInputOpArgs()) - - def _MaybePadSourceInputs(self, src_inputs, src_paddings): - p = self.params - if not p.append_eos_frame: - return src_inputs, src_paddings - - per_src_len = tf.reduce_sum(1 - src_paddings, 1) - per_src_len += 1 - max_src_len = tf.reduce_max(per_src_len) - input_shape = tf.shape(src_inputs) - input_len = tf.maximum(input_shape[1], tf.cast(max_src_len, tf.int32)) - pad_steps = input_len - input_shape[1] - src_inputs = tf.concat([ - src_inputs, - tf.zeros( - inplace_ops.inplace_update(input_shape, 1, pad_steps), - src_inputs.dtype) - ], 1) - src_paddings = 1 - tf.sequence_mask( - tf.reshape(per_src_len, [input_shape[0]]), tf.reshape(input_len, []), - src_paddings.dtype) - return src_inputs, src_paddings - - def __init__(self, params): - super().__init__(params) - p = self.params - - (utt_ids, audio_document_ids, num_utterances_in_audio_document, - tgt_ids, tgt_labels, tgt_paddings, src_frames, - src_paddings), self._bucket_keys = self._BuildDataSource() - - self._sample_ids = utt_ids - - src_frames, src_paddings = self._MaybePadSourceInputs( - src_frames, src_paddings) - - # We expect src_inputs to be of shape - # [batch_size, num_frames, feature_dim, channels]. - src_frames = tf.expand_dims(src_frames, axis=-1) - - # Convert target ids, labels, paddings, and weights from shape [batch_size, - # 1, num_frames] to [batch_size, num_frames] - tgt_ids = tf.squeeze(tgt_ids, axis=1) - tgt_labels = tf.squeeze(tgt_labels, axis=1) - tgt_paddings = tf.squeeze(tgt_paddings, axis=1) - - if p.pad_to_max_seq_length: - assert p.source_max_length - assert p.target_max_length - - if all(x == p.bucket_batch_limit[0] for x in p.bucket_batch_limit): - # Set the input batch size as an int rather than a tensor. - src_frames_shape = (self.InfeedBatchSize(), p.source_max_length, - p.frame_size, 1) - src_paddings_shape = (self.InfeedBatchSize(), p.source_max_length) - tgt_shape = (self.InfeedBatchSize(), p.target_max_length) - else: - tf.logging.warning( - 'Could not set static input shape since not all bucket batch sizes ' - 'are the same:', p.bucket_batch_limit) - src_frames_shape = None - src_paddings_shape = None - tgt_shape = None - - src_frames = py_utils.PadBatchDimension(src_frames, self.InfeedBatchSize(), 0) - src_paddings = py_utils.PadBatchDimension(src_paddings, self.InfeedBatchSize(), - 1) - tgt_ids = py_utils.PadBatchDimension(tgt_ids, self.InfeedBatchSize(), 0) - tgt_labels = py_utils.PadBatchDimension(tgt_labels, self.InfeedBatchSize(), 0) - tgt_paddings = py_utils.PadBatchDimension(tgt_paddings, self.InfeedBatchSize(), - 1) - self._sample_ids = py_utils.PadBatchDimension(self._sample_ids, self.InfeedBatchSize(), - type(self).PAD_INDEX) - # For reasons I don't understand, the shape of self._sample_ids after the above is - # [BatchSize, 1] rather than [BatchSize]. - self._sample_ids = tf.squeeze(self._sample_ids, axis=1) - self._sample_ids = tf.ensure_shape(self._sample_ids, self.InfeedBatchSize()) - - audio_document_ids = py_utils.PadBatchDimension(audio_document_ids, self.InfeedBatchSize(), - type(self).PAD_INDEX) - # For reasons I don't understand, the shape of audio_document_ids after the above is - # [BatchSize, 1] rather than [BatchSize]. - audio_document_ids = tf.squeeze(audio_document_ids, axis=1) - audio_document_ids = tf.ensure_shape(audio_document_ids, self.InfeedBatchSize()) - - num_utterances_in_audio_document = py_utils.PadBatchDimension(num_utterances_in_audio_document, self.InfeedBatchSize(), - type(self).PAD_INDEX) - # For reasons I don't understand, the shape of num_utterances_in_audio_document after the above is - # [BatchSize, 1] rather than [BatchSize]. - num_utterances_in_audio_document = tf.squeeze(num_utterances_in_audio_document, axis=1) - num_utterances_in_audio_document = tf.ensure_shape(num_utterances_in_audio_document, self.InfeedBatchSize()) - - - src_frames = py_utils.PadSequenceDimension( - src_frames, p.source_max_length, 0, shape=src_frames_shape) - src_paddings = py_utils.PadSequenceDimension( - src_paddings, p.source_max_length, 1, shape=src_paddings_shape) - tgt_ids = py_utils.PadSequenceDimension( - tgt_ids, p.target_max_length, 0, shape=tgt_shape) - tgt_labels = py_utils.PadSequenceDimension( - tgt_labels, p.target_max_length, 0, shape=tgt_shape) - tgt_paddings = py_utils.PadSequenceDimension( - tgt_paddings, p.target_max_length, 1, shape=tgt_shape) - - tgt = py_utils.NestedMap( - ids=tgt_ids, - labels=tgt_labels, - paddings=tgt_paddings, - weights=1.0 - tgt_paddings) - src = py_utils.NestedMap(src_inputs=src_frames, paddings=src_paddings) - - self._tgt = tgt - self._src = src - - self._audio_document_ids = audio_document_ids - self._num_utterances_in_audio_document = num_utterances_in_audio_document - - def _InputBatch(self): - batch = py_utils.NestedMap() - batch.bucket_keys = self._bucket_keys - batch.src = self._src - batch.tgt = self._tgt - batch.sample_ids = self._sample_ids - batch.audio_document_ids = self._audio_document_ids - batch.num_utterances_in_audio_document = self._num_utterances_in_audio_document - return batch diff --git a/lingvo/tasks/asr/input_generator_test.py b/lingvo/tasks/asr/input_generator_test.py deleted file mode 100644 index cf63baa5..00000000 --- a/lingvo/tasks/asr/input_generator_test.py +++ /dev/null @@ -1,176 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for asr.input_generator.""" - -import os -import lingvo.compat as tf -from lingvo.core import test_utils -from lingvo.tasks.asr import input_generator -import numpy as np - - -def _MakeBytesFeature(unicode_array): - value = [tf.compat.as_bytes(w) for w in unicode_array] - return tf.train.Feature(bytes_list=tf.train.BytesList(value=value)) - - -def _MakeInt64Feature(value): - return tf.train.Feature(int64_list=tf.train.Int64List(value=value)) - - -def _MakeFloatFeature(value): - return tf.train.Feature(float_list=tf.train.FloatList(value=value)) - - -FRAME_SIZE = 40 - - -def _MakeFrames(utt_len): - return np.random.normal(size=FRAME_SIZE * utt_len) - - -def _MakeTfExample(uttid, frames, text): - flat_frames = frames.flatten() - feature = { - 'uttid': _MakeBytesFeature([uttid]), - 'transcript': _MakeBytesFeature([text]), - 'frames': _MakeFloatFeature(flat_frames) - } - return tf.train.Example(features=tf.train.Features(feature=feature)) - - -class InputTest(test_utils.TestCase): - - def _GenerateExamples(self, output_filepath): - example_def = [(b'utt1', (1234, b'HELLO WORLD')), - (b'utt2', (568, b'TIRED WITH ALL THESE')), - (b'utt3', (778, b'WOULD THAT IT WERE SO EASY'))] - self._example_def = dict(example_def) - tf_examples = [] - for xdef in example_def: - tf_examples.append( - _MakeTfExample(xdef[0], _MakeFrames(xdef[1][0]), xdef[1][1])) - with tf.python_io.TFRecordWriter(output_filepath) as outf: - for ex in tf_examples: - outf.write(ex.SerializeToString()) - - def _GenerateSetup(self, append_eos_frame, pad_to_max_seq_length=False): - tfrecords_filepath = os.path.join(tf.test.get_temp_dir(), - 'simple.tfrecords') - self._GenerateExamples(tfrecords_filepath) - p = input_generator.AsrInput.Params() - p.file_pattern = 'tfrecord:' + tfrecords_filepath - p.frame_size = FRAME_SIZE - p.target_max_length = 30 - p.bucket_upper_bound = [2560] - p.bucket_batch_limit = [3] - p.append_eos_frame = append_eos_frame - p.pad_to_max_seq_length = pad_to_max_seq_length - return p - - def _AssertAllOnes(self, np_data): - self.assertAllEqual(np_data, np.ones_like(np_data)) - - def _AssertAllZeros(self, np_data): - self.assertAllEqual(np_data, np.zeros_like(np_data)) - - def _AssertShapesAsExpected(self, shapes, expected): - for key, expected_shape in sorted(expected.items()): - self.assertAllEqual( - shapes.GetItem(key), expected_shape, msg='Shape of %s' % key) - - def _TestAsrInput(self, params): - p = params - with self.session(use_gpu=False): - inp = input_generator.AsrInput(p) - batch = inp.GetPreprocessedInputBatch() - vals = self.evaluate(batch) - shapes = vals.Transform(lambda x: x.shape) - shapes.VLog(0, 'shapes: ') - # sample_ids (3, 1) - # src.src_inputs (3, 1235, 40, 1) - # src.paddings (3, 1235) - # tgt.ids (3, 30) - # tgt.labels (3, 30) - # tgt.paddings (3, 30) - # tgt.weights (3, 30) - batch_size = p.bucket_batch_limit[0] - if p.pad_to_max_seq_length: - max_num_frames = p.source_max_length - else: - max_num_frames = np.amax( - [xdef[0] for xdef in self._example_def.values()]) - if p.append_eos_frame: - max_num_frames += 1 - - tgt_shape = [batch_size, p.target_max_length] - self._AssertShapesAsExpected( - shapes, { - 'sample_ids': [batch_size, 1], - 'src.src_inputs': [batch_size, max_num_frames, p.frame_size, 1], - 'src.paddings': [batch_size, max_num_frames], - 'tgt.ids': tgt_shape, - 'tgt.labels': tgt_shape, - 'tgt.paddings': tgt_shape, - 'tgt.weights': tgt_shape - }) - for b in range(batch_size): - ex = vals.Transform(lambda x: x[b]) - ref = self._example_def[ex.sample_ids[0]] - # Check source. - ref_num_frames = ref[0] - if p.append_eos_frame: - ref_num_frames += 1 - ref_num_padding_frames = max_num_frames - ref_num_frames - zero_frames = ex.src.src_inputs[max_num_frames - - ref_num_padding_frames:] - self._AssertAllZeros(zero_frames) - zero_paddings = ex.src.paddings[:ref_num_frames] - one_paddings = ex.src.paddings[ref_num_frames:] - self._AssertAllZeros(zero_paddings) - self._AssertAllOnes(one_paddings) - # Check target. Something like this: - # ids: [1 a b c 2 2 2] - # labels: [a b c 2 2 2 2] - # paddings: [0 0 0 1 1 1 1] - ref_num_graphemes = len(ref[1]) - ref_tgt_ids_padding = p.target_max_length - ref_num_graphemes - 1 - self.assertEqual(ex.tgt.ids[0], p.tokenizer.target_sos_id) - self.assertAllEqual(ex.tgt.ids[-ref_tgt_ids_padding:], - [2] * ref_tgt_ids_padding) - self.assertAllEqual(ex.tgt.labels[-ref_tgt_ids_padding - 1:], - [2] * (ref_tgt_ids_padding + 1)) - self.assertAllEqual(ex.tgt.ids[1:1 + ref_num_graphemes], - ex.tgt.labels[:ref_num_graphemes]) - self._AssertAllZeros(ex.tgt.paddings[:ref_num_graphemes + 1]) - self._AssertAllOnes(ex.tgt.paddings[ref_num_graphemes + 1:]) - self._AssertAllOnes(ex.tgt.weights[:ref_num_graphemes + 1]) - self._AssertAllZeros(ex.tgt.weights[ref_num_graphemes + 1:]) - - def testAsrInput(self): - p = self._GenerateSetup(append_eos_frame=True) - self._TestAsrInput(p) - - def testAsrInputWithoutEosFrame(self): - p = self._GenerateSetup(append_eos_frame=False) - self._TestAsrInput(p) - - def testAsrInputWithStaticShape(self): - p = self._GenerateSetup(append_eos_frame=False, pad_to_max_seq_length=True) - self._TestAsrInput(p) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/asr/mlperf_compat_frontend.py b/lingvo/tasks/asr/mlperf_compat_frontend.py deleted file mode 100644 index 48d70b2c..00000000 --- a/lingvo/tasks/asr/mlperf_compat_frontend.py +++ /dev/null @@ -1,148 +0,0 @@ -class MLPerfCompatFrontend(BaseAsrFrontend): - @classmethod - def Params(cls): - p = super(MelAsrFrontend, cls).Params() - p.name = 'frontend' - p.Define('sample_rate', 16000.0, 'Sample rate in Hz') - p.Define('channel_count', 1, 'Number of channels.') - p.Define('frame_size_ms', 25.0, - 'Amount of data grabbed for each frame during analysis') - p.Define('frame_step_ms', 10.0, 'Number of ms to jump between frames') - p.Define('num_bins', 80, 'Number of bins in the mel-spectrogram output') - p.Define('lower_edge_hertz', 125.0, - 'The lowest frequency of the mel-spectrogram analsis') - p.Define('upper_edge_hertz', 7600.0, - 'The highest frequency of the mel-spectrogram analsis') - p.Define('preemph', 0.97, - 'The first-order filter coefficient used for preemphasis') - p.Define('noise_scale', 8.0, - 'The amount of noise (in 16-bit LSB units) to add') - p.Define('window_fn', 'HANNING', - 'Window function to apply (valid values are "HANNING", and None)') - p.Define( - 'pad_end', False, - 'Whether to pad the end of `signals` with zeros when the provided ' - 'frame length and step produces a frame that lies partially past ' - 'its end.') - p.Define( - 'per_bin_mean', None, - 'Per-bin (num_bins) means for normalizing the spectrograms. ' - 'Defaults to zeros.') - p.Define('per_bin_stddev', None, - 'Per-bin (num_bins) standard deviations. Defaults to ones.') - p.Define('stack_left_context', 0, 'Number of left context frames to stack.') - p.Define('stack_right_context', 0, - 'Number of right context frames to stack.') - p.Define('frame_stride', 1, 'The frame stride for sub-sampling.') - - return p - - @staticmethod - def GetConfigFromParams(params): - """Returns an AsrFrontendConfig namedtuple with vital config settings.""" - context_size = params.stack_left_context + params.stack_right_context + 1 - subsample_factor = params.num_bins * context_size - frame_step = round(params.sample_rate * params.frame_step_ms / 1000.0) - return AsrFrontendConfig( - is_null=False, - src_type='pcm', - src_pcm_scale=32768.0, - src_pcm_sample_rate=16000.0, - output_dim=subsample_factor, - input_frame_ratio=frame_step * subsample_factor) - - @base_layer.initializer - def __init__(self, params): - super(MelAsrFrontend, self).__init__(params) - p = self.params - if p.frame_stride < 1: - raise ValueError('frame_stride must be positive.') - - assert p.channel_count == 1, 'Only 1 channel currently supported.' - # Make sure key params are in floating point. - p.sample_rate = float(p.sample_rate) - p.frame_step_ms = float(p.frame_step_ms) - p.frame_size_ms = float(p.frame_size_ms) - p.lower_edge_hertz = float(p.lower_edge_hertz) - p.upper_edge_hertz = float(p.upper_edge_hertz) - - self._frame_step = int(round(p.sample_rate * p.frame_step_ms / 1000.0)) - self._frame_size = (int(round(p.sample_rate * p.frame_size_ms / 1000.0)) + 1 - ) # +1 for the preemph - # Overdrive means double FFT size. - # Note: 2* because of overdrive - self._fft_size = 2 * int(max(512, _NextPowerOfTwo(self._frame_size))) - - self._CreateWindowFunction() - - # Mean/stddev. - if p.per_bin_mean is None: - p.per_bin_mean = [0.0] * p.num_bins - if p.per_bin_stddev is None: - p.per_bin_stddev = [1.0] * p.num_bins - assert len(p.per_bin_mean) == p.num_bins - assert len(p.per_bin_stddev) == p.num_bins - - def _CreateWindowFunction(self): - p = self.params - if p.window_fn is None: - self._window_fn = None - elif p.window_fn == 'HANNING': - - def _HanningWindow(frame_size, dtype): - return tf.signal.hann_window(frame_size, dtype=dtype) - - self._window_fn = _HanningWindow - else: - raise ValueError('Illegal value %r for window_fn param' % (p.window_fn,)) - - def FProp(self, theta, input_batch): - """Perform signal processing on a sequence of PCM data. - - NOTE: This implementation does not currently support paddings, and they - are accepted for compatibility with the super-class. - - TODO(laurenzo): Rework this to support paddings. - - Args: - theta: Layer theta. - input_batch: PCM input map: - - - 'src_inputs': int16 or float32 tensor of PCM audio data, scaled to - +/-32768 (versus [-1..1)!). See class comments for supported input - shapes. - - 'paddings': per frame 0/1 paddings. Shaped: [batch, frame]. - Returns: - NestedMap of encoder inputs which can be passed directly to a - compatible encoder and contains: - - - 'src_inputs': inputs to the encoder, minimally of shape - [batch, time, ...]. - - 'paddings': a 0/1 tensor of shape [batch, time]. - """ - - return self._FPropDefault(input_batch) - - def _FPropDefault(self, input_batch): - pcm_audio_data, pcm_audio_paddings = self._ReshapeToMono2D( - input_batch.src_inputs, input_batch.paddings) - - mel_spectrogram, mel_spectrogram_paddings = self._FPropChunk( - pcm_audio_data, pcm_audio_paddings) - - mel_spectrogram, mel_spectrogram_paddings = self._PadAndReshapeSpec( - mel_spectrogram, mel_spectrogram_paddings) - - return py_utils.NestedMap( - src_inputs=mel_spectrogram, paddings=mel_spectrogram_paddings) - - def _StackSignal(self, signal, stack_size, stride): - signal = tf.signal.frame( - signal=signal, - frame_length=stack_size, - frame_step=stride, - pad_end=False, - axis=1, - ) - signal = tf.reshape(signal, py_utils.GetShape(signal)[:2] + [-1]) - return signal diff --git a/lingvo/tasks/asr/model.py b/lingvo/tasks/asr/model.py deleted file mode 100644 index 976cd6a7..00000000 --- a/lingvo/tasks/asr/model.py +++ /dev/null @@ -1,496 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Speech model.""" - -import collections -import lingvo.compat as tf -from lingvo.core import base_model -from lingvo.core import metrics -from lingvo.core import py_utils -from lingvo.core import schedule -from lingvo.tasks.asr import decoder -from lingvo.tasks.asr import decoder_utils -from lingvo.tasks.asr import encoder -from lingvo.tasks.asr import frontend as asr_frontend -from lingvo.tools import audio_lib - - -# hyps: [num_beams, num_hyps_per_beam] of serialized Hypothesis protos. -# ids: [num_beams * num_hyps_per_beam, max_target_length]. -# lens: [num_beams * num_hyps_per_beam]. -# scores: [num_beams, num_hyps_per_beam]. -# decoded: [num_beams, num_hyps_per_beam]. -DecoderTopK = collections.namedtuple( - 'topk', ['hyps', 'ids', 'lens', 'scores', 'decoded']) # pyformat: disable - - -class AsrModel(base_model.BaseTask): - """Speech model.""" - - @classmethod - def Params(cls): - p = super().Params() - p.encoder = encoder.AsrEncoder.Params() - p.decoder = decoder.AsrDecoder.Params() - p.Define( - 'frontend', None, - 'ASR frontend to extract features from input. Defaults to no frontend ' - 'which means that features are taken directly from the input.') - - p.Define( - 'include_auxiliary_metrics', True, - 'In addition to simple WER, also computes oracle WER, SACC, TER, etc. ' - 'Turning off this option will speed up the decoder job.') - - tp = p.train - tp.lr_schedule = ( - schedule.PiecewiseConstantSchedule.Params().Set( - boundaries=[350000, 500000, 600000], values=[1.0, 0.1, 0.01, - 0.001])) - tp.vn_start_step = 20000 - tp.vn_std = 0.075 - tp.l2_regularizer_weight = 1e-6 - tp.learning_rate = 0.001 - tp.clip_gradient_norm_to_value = 1.0 - tp.grad_norm_to_clip_to_zero = 100.0 - tp.tpu_steps_per_loop = 20 - - return p - - def __init__(self, params): - if not params.name: - raise ValueError('params.name not set.') - super().__init__(params) - p = self.params - - # Construct the model. - if p.encoder: - if not p.encoder.name: - p.encoder.name = 'enc' - self.CreateChild('encoder', p.encoder) - if p.decoder: - if not p.decoder.name: - p.decoder.name = 'dec' - self.CreateChild('decoder', p.decoder) - if p.frontend: - self.CreateChild('frontend', p.frontend) - - def _GetDecoderTargets(self, input_batch): - """Returns targets which will be forwarded to the decoder. - - Subclasses can override this method to change the target that is used by - the decoder. For example, a subclass could add additional targets that - can be forwared to the decoder. - - Args: - input_batch: a NestedMap which contains the targets. - - Returns: - a NestedMap corresponding to the target selected. - """ - return input_batch.tgt - - def _MakeDecoderTheta(self, theta, input_batch): - """Compute theta to be used by the decoder for computing metrics and loss. - - This method can be over-ridden by child classes to add values to theta that - is passed to the decoder. - - For example, to pass the one hot vector which indicates which data source - was selected a child class could over-ride this method as follows: - - def _MakeDecoderTheta(self, theta): - decoder_theta = super(MyModel, self)._MakeDecoderTheta(theta, input_batch) - decoder_theta.child_onehot = input_batch.source_selected - return decoder_theta - - Args: - theta: A `.NestedMap` object containing variable values used to compute - loss and metrics. - input_batch: NestedMap containing input data in the current batch. Unused - here. - - Returns: - A copy of the decoder theta. - """ - del input_batch # Unused - return theta.decoder.DeepCopy() - - def ComputePredictions(self, theta, input_batch): - input_batch_src = input_batch.src - encoder_outputs = self._FrontendAndEncoderFProp(theta, input_batch_src) - tgt = self._GetDecoderTargets(input_batch) - decoder_theta = self._MakeDecoderTheta(theta, input_batch) - return self.decoder.ComputePredictions(decoder_theta, encoder_outputs, tgt) - - def ComputeLoss(self, theta, predictions, input_batch): - tgt = self._GetDecoderTargets(input_batch) - decoder_theta = self._MakeDecoderTheta(theta, input_batch) - return self.decoder.ComputeLoss(decoder_theta, predictions, tgt) - - def _FrontendAndEncoderFProp(self, - theta, - input_batch_src, - initial_state=None): - """FProps through the frontend and encoder. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - input_batch_src: An input NestedMap as per `BaseAsrFrontend.FProp`. - initial_state: None or a NestedMap object containing the initial states. - - Returns: - A NestedMap as from `AsrEncoder.FProp`. - """ - p = self.params - if p.frontend: - input_batch_src = self.frontend.FProp(theta.frontend, input_batch_src) - if initial_state: - return self.encoder.FProp( - theta.encoder, input_batch_src, state0=initial_state) - else: - return self.encoder.FProp(theta.encoder, input_batch_src) - - def _GetTopK(self, decoder_outs, tag=''): - hyps = decoder_outs.topk_hyps - ids = tf.identity(decoder_outs.topk_ids, name='TopKLabelIds' + tag) - lens = tf.identity(decoder_outs.topk_lens, name='TopKLabelLengths' + tag) - scores = decoder_outs.topk_scores - decoded = decoder_outs.topk_decoded - - if ids is not None: - decoded = self.input_generator.IdsToStrings(ids, lens - 1) - decoded = tf.identity(decoded, name='top_k_decoded%s' % tag) - decoded = tf.reshape(decoded, tf.shape(hyps)) - if scores is not None and hyps is not None: - scores = tf.identity( - tf.reshape(scores, tf.shape(lens)), name='top_k_scores%s' % tag) - scores = tf.reshape(scores, tf.shape(hyps)) - return DecoderTopK(hyps, ids, lens, scores, decoded) - - def _ComputeNormalizedWER(self, hyps, refs): - # Filter out all '' tokens for norm_wer computation. - hyps_no_epsilon = tf.strings.regex_replace(hyps, '()+', ' ') - # norm_wer is size [num_transcripts * hyps_per_beam, 2] - norm_wer = decoder_utils.ComputeWer(hyps_no_epsilon, refs) - # Split into two tensors of size [num_transcripts * hyps_per_beam, 1] - norm_wer_errors, norm_wer_words = tf.split(norm_wer, [1, 1], 1) - shape = [-1, self.params.decoder.beam_search.num_hyps_per_beam] - norm_wer_errors = tf.reshape(norm_wer_errors, shape) - norm_wer_words = tf.reshape(norm_wer_words, shape) - - return norm_wer_errors, norm_wer_words - - def AddAdditionalDecoderMetricsToGraph(self, topk_hyps, filtered_hyps, - filtered_refs, input_batch, - decoder_outs): - """Returns a dict of metrics which should be computed from decoded hyps.""" - # The base class implementation returns an empty dictionary. Sub-classes can - # provide their own implementation. - return {} - - def DecodeWithTheta(self, theta, input_batch): - """Constructs the inference graph.""" - p = self.params - with tf.name_scope('decode'), tf.name_scope(p.name): - with tf.name_scope('encoder'): - encoder_outputs = self._FrontendAndEncoderFProp(theta, input_batch.src) - with tf.name_scope('beam_search'): - decoder_outs = self.decoder.BeamSearchDecodeWithTheta( - theta.decoder, encoder_outputs) - - if py_utils.use_tpu(): - # Decoder metric computation contains arbitrary execution - # that may not run on TPU. - decoder_metrics = py_utils.RunOnTpuHost(self._ComputeDecoderMetrics, - decoder_outs, input_batch) - else: - decoder_metrics = self._ComputeDecoderMetrics(decoder_outs, input_batch) - return decoder_metrics - - def _GetTargetForDecoderMetrics(self, input_batch): - """Returns targets which will be used to compute decoder metrics. - - Subclasses can override this method to change the target that is used when - calculating decoder metrics. - - Args: - input_batch: a NestedMap which contains the targets. - - Returns: - a NestedMap containing 'ids', 'labels', 'paddings', 'weights' - """ - return self._GetDecoderTargets(input_batch) - - def _ComputeDecoderMetrics(self, decoder_outs, input_batch): - """Computes metrics on output from decoder. - - Args: - decoder_outs: A `BeamSearchDecodeOutput`, a namedtuple containing the - decode results. - input_batch: A `NestedMap` of tensors representing the source, target, - and other components of the input batch. - - Returns: - A dict of Tensors containing decoder output and metrics. - """ - p = self.params - topk = self._GetTopK(decoder_outs) - tgt = self._GetTargetForDecoderMetrics(input_batch) - transcripts = self.input_generator.IdsToStrings( - tgt.labels, - tf.cast(tf.round(tf.reduce_sum(1.0 - tgt.paddings, 1) - 1.0), tf.int32)) - - # Filter out all isolated '' tokens. - noise_pattern = ' |^ | $|^$' - filtered_refs = tf.strings.regex_replace(transcripts, noise_pattern, ' ') - filtered_hyps = tf.strings.regex_replace(topk.decoded, noise_pattern, ' ') - # Compute translation quality scores for all hyps. - filtered_refs = tf.tile( - tf.reshape(filtered_refs, [-1, 1]), - [1, p.decoder.beam_search.num_hyps_per_beam]) - filtered_hyps = tf.reshape(filtered_hyps, [-1]) - filtered_refs = tf.reshape(filtered_refs, [-1]) - norm_wer_errors, norm_wer_words = self._ComputeNormalizedWER( - filtered_hyps, filtered_refs) - - ret_dict = { - 'target_ids': tgt.ids, - 'target_labels': tgt.labels, - 'target_weights': tgt.weights, - 'target_paddings': tgt.paddings, - 'transcripts': transcripts, - 'topk_decoded': topk.decoded, - 'topk_ids': topk.ids, - 'topk_lens': topk.lens, - 'topk_scores': topk.scores, - 'norm_wer_errors': norm_wer_errors, - 'norm_wer_words': norm_wer_words, - } - - if not py_utils.use_tpu(): - ret_dict['utt_id'] = input_batch.sample_ids - - ret_dict.update( - self.AddAdditionalDecoderMetricsToGraph(topk, filtered_hyps, - filtered_refs, input_batch, - decoder_outs)) - return ret_dict - - def CreateAdditionalDecoderMetrics(self): - """Returns a dictionary of additional metrics which should be computed.""" - # The base class implementation returns an empty dictionary. Sub-classes can - # provide their own implementation. - return {} - - def CreateDecoderMetrics(self): - base_metrics = { - 'num_samples_in_batch': metrics.AverageMetric(), - 'norm_wer': metrics.AverageMetric(), # Normalized word error rate. - 'corpus_bleu': metrics.CorpusBleuMetric(), - } - - if self.params.include_auxiliary_metrics: - base_metrics.update({ - 'wer': metrics.AverageMetric(), # Word error rate. - 'sacc': metrics.AverageMetric(), # Sentence accuracy. - 'ter': metrics.AverageMetric(), # Token error rate. - 'oracle_norm_wer': metrics.AverageMetric(), - }) - - # Add any additional metrics that should be computed. - base_metrics.update(self.CreateAdditionalDecoderMetrics()) - return base_metrics - - def UpdateAdditionalMetrics(self, dec_out_dict, dec_metrics_dict): - """Updates and returns a dictionary of metrics based on decoded hyps.""" - # Can be implemented in sub-classes to perform any model specific behavior. - # The default implementation just returns the metrics unchanged. - del dec_out_dict - return dec_metrics_dict - - # TODO(prabhavalkar): Add support to save out the decoded hypotheses. - def PostProcessDecodeOut(self, dec_out_dict, dec_metrics_dict): - p = self.params - assert 'topk_scores' in dec_out_dict, list(dec_out_dict.keys()) - topk_scores = dec_out_dict['topk_scores'] - topk_decoded = dec_out_dict['topk_decoded'] - transcripts = dec_out_dict['transcripts'] - if not py_utils.use_tpu(): - utt_id = dec_out_dict['utt_id'] - assert len(utt_id) == len(transcripts) - norm_wer_errors = dec_out_dict['norm_wer_errors'] - norm_wer_words = dec_out_dict['norm_wer_words'] - target_labels = dec_out_dict['target_labels'] - target_paddings = dec_out_dict['target_paddings'] - topk_ids = dec_out_dict['topk_ids'] - topk_lens = dec_out_dict['topk_lens'] - assert len(transcripts) == len(target_labels) - assert len(transcripts) == len(target_paddings) - assert len(transcripts) == len(topk_decoded) - assert (len(topk_ids) == p.decoder.beam_search.num_hyps_per_beam * - len(transcripts)) - assert len(norm_wer_errors) == len(transcripts) - assert len(norm_wer_words) == len(transcripts) - - dec_metrics_dict['num_samples_in_batch'].Update(len(transcripts)) - - def GetRefIds(ref_ids, ref_paddinds): - assert len(ref_ids) == len(ref_paddinds) - return_ids = [] - for i in range(len(ref_ids)): - if ref_paddinds[i] == 0: - return_ids.append(ref_ids[i]) - return return_ids - - total_norm_wer_errs = norm_wer_errors[:, 0].sum() - total_norm_wer_words = norm_wer_words[:, 0].sum() - - dec_metrics_dict['norm_wer'].Update( - total_norm_wer_errs / total_norm_wer_words, total_norm_wer_words) - - for ref_str, hyps in zip(transcripts, topk_decoded): - filtered_ref = decoder_utils.FilterNoise(ref_str) - filtered_ref = decoder_utils.FilterEpsilon(filtered_ref) - filtered_hyp = decoder_utils.FilterNoise(hyps[0]) - filtered_hyp = decoder_utils.FilterEpsilon(filtered_hyp) - dec_metrics_dict['corpus_bleu'].Update(filtered_ref, filtered_hyp) - - total_errs = 0 - total_oracle_errs = 0 - total_ref_words = 0 - total_token_errs = 0 - total_ref_tokens = 0 - total_accurate_sentences = 0 - key_value_pairs = [] - - if p.include_auxiliary_metrics: - for i in range(len(transcripts)): - ref_str = transcripts[i] - if not py_utils.use_tpu(): - tf.logging.info('utt_id: %s', utt_id[i]) - if self.cluster.add_summary: - tf.logging.info(' ref_str: %s', ref_str) - hyps = topk_decoded[i] - ref_ids = GetRefIds(target_labels[i], target_paddings[i]) - hyp_index = i * p.decoder.beam_search.num_hyps_per_beam - top_hyp_ids = topk_ids[hyp_index][:topk_lens[hyp_index]] - if self.cluster.add_summary: - tf.logging.info(' ref_ids: %s', ref_ids) - tf.logging.info(' top_hyp_ids: %s', top_hyp_ids) - total_ref_tokens += len(ref_ids) - _, _, _, token_errs = decoder_utils.EditDistanceInIds( - ref_ids, top_hyp_ids) - total_token_errs += token_errs - - assert p.decoder.beam_search.num_hyps_per_beam == len(hyps) - filtered_ref = decoder_utils.FilterNoise(ref_str) - filtered_ref = decoder_utils.FilterEpsilon(filtered_ref) - oracle_errs = norm_wer_errors[i][0] - for n, (score, hyp_str) in enumerate(zip(topk_scores[i], hyps)): - if self.cluster.add_summary: - tf.logging.info(' %f: %s', score, hyp_str) - filtered_hyp = decoder_utils.FilterNoise(hyp_str) - filtered_hyp = decoder_utils.FilterEpsilon(filtered_hyp) - ins, subs, dels, errs = decoder_utils.EditDistance( - filtered_ref, filtered_hyp) - # Note that these numbers are not consistent with what is used to - # compute normalized WER. In particular, these numbers will be - # inflated when the transcript contains punctuation. - tf.logging.info(' ins: %d, subs: %d, del: %d, total: %d', ins, subs, - dels, errs) - # Only aggregate scores of the top hypothesis. - if n == 0: - total_errs += errs - total_ref_words += len(decoder_utils.Tokenize(filtered_ref)) - if norm_wer_errors[i, n] == 0: - total_accurate_sentences += 1 - oracle_errs = min(oracle_errs, norm_wer_errors[i, n]) - total_oracle_errs += oracle_errs - - dec_metrics_dict['wer'].Update(total_errs / max(1., total_ref_words), - total_ref_words) - dec_metrics_dict['oracle_norm_wer'].Update( - total_oracle_errs / max(1., total_ref_words), total_ref_words) - dec_metrics_dict['sacc'].Update( - total_accurate_sentences / len(transcripts), len(transcripts)) - dec_metrics_dict['ter'].Update( - total_token_errs / max(1., total_ref_tokens), total_ref_tokens) - - # Update any additional metrics. - dec_metrics_dict = self.UpdateAdditionalMetrics(dec_out_dict, - dec_metrics_dict) - return key_value_pairs - - def Inference(self): - """Constructs inference subgraphs. - - Returns: - dict: A dictionary of the form ``{'subgraph_name': (fetches, feeds)}``. - Each of fetches and feeds is itself a dictionary which maps a string name - (which describes the tensor) to a corresponding tensor in the inference - graph which should be fed/fetched from. - """ - subgraphs = {} - with tf.name_scope('inference'): - subgraphs['default'] = self._InferenceSubgraph_Default() - return subgraphs - - def _InferenceSubgraph_Default(self): - """Constructs graph for offline inference. - - Returns: - (fetches, feeds) where both fetches and feeds are dictionaries. Each - dictionary consists of keys corresponding to tensor names, and values - corresponding to a tensor in the graph which should be input/read from. - """ - p = self.params - with tf.name_scope('default'): - # TODO(laurenzo): Once the migration to integrated frontends is complete, - # this model should be upgraded to use the MelAsrFrontend in its - # params vs relying on pre-computed feature generation and the inference - # special casing. - wav_bytes = tf.placeholder(dtype=tf.string, name='wav') - frontend = self.frontend if p.frontend else None - if not frontend: - # No custom frontend. Instantiate the default. - frontend_p = asr_frontend.MelAsrFrontend.Params() - frontend = frontend_p.Instantiate() - - # Decode the wave bytes and use the explicit frontend. - unused_sample_rate, audio = audio_lib.DecodeWav(wav_bytes) - audio *= 32768 - # Remove channel dimension, since we have a single channel. - audio = tf.squeeze(audio, axis=1) - # Add batch. - audio = tf.expand_dims(audio, axis=0) - input_batch_src = py_utils.NestedMap( - src_inputs=audio, paddings=tf.zeros_like(audio)) - input_batch_src = frontend.FPropDefaultTheta(input_batch_src) - - encoder_outputs = self.encoder.FPropDefaultTheta(input_batch_src) - decoder_outputs = self.decoder.BeamSearchDecode(encoder_outputs) - topk = self._GetTopK(decoder_outputs) - - feeds = {'wav': wav_bytes} - fetches = { - 'hypotheses': topk.decoded, - 'scores': topk.scores, - 'src_frames': input_batch_src.src_inputs, - 'encoder_frames': encoder_outputs.encoded - } - - return fetches, feeds diff --git a/lingvo/tasks/asr/model_test.py b/lingvo/tasks/asr/model_test.py deleted file mode 100644 index 378e1248..00000000 --- a/lingvo/tasks/asr/model_test.py +++ /dev/null @@ -1,430 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for Asr Model.""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import cluster_factory -from lingvo.core import py_utils -from lingvo.core import schedule -from lingvo.core import summary_utils -from lingvo.core import test_helper -from lingvo.core import test_utils -from lingvo.tasks.asr import decoder -from lingvo.tasks.asr import input_generator -from lingvo.tasks.asr import model -from lingvo.tasks.asr import model_test_input_generator as tig -import numpy as np - - -class DecoderForTest(decoder.AsrDecoder): - """Unit test class for AsrDecoder with functional.for based unrolling.""" - - @classmethod - def Params(cls): - p = super().Params() - p.use_while_loop_based_unrolling = False - return p - - -class AsrModelTest(test_utils.TestCase): - - def _testParams(self): - input_shape = [2, 16, 8, 3] - p = model.AsrModel.Params() - p.decoder.target_seq_len = 5 - p.encoder.input_shape = input_shape - p.input = tig.TestInputGenerator.Params() - p.input.target_max_length = 5 - p.input.source_shape = input_shape - p.input.target_shape = [2, 5] - p.name = 'test_mdl' - return p - - def testMakeDecoderTheta(self): - # Test that decoder theta returns a copy of theta.decoder without changes. - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(93820985) - p = self._testParams() - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - decoder_theta = mdl._MakeDecoderTheta(theta=mdl.theta, input_batch=None) - mdl.BProp() - self.assertEqual(decoder_theta, mdl.theta.decoder) - - def testFProp(self): - with self.session(use_gpu=False): - tf.random.set_seed(93820985) - p = self._testParams() - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - self.evaluate(tf.global_variables_initializer()) - test_utils.CompareToGoldenSingleFloat(self, 4.472597, mdl.loss.eval()) - - actual_var_names = [_.name for _ in tf.trainable_variables()] - print('all vars \n', '\n'.join(actual_var_names)) - expected_var_names = [ - 'test_mdl/enc/conv_L0/w/var:0', - 'test_mdl/enc/conv_L0/beta/var:0', - 'test_mdl/enc/conv_L0/gamma/var:0', - 'test_mdl/enc/conv_L1/w/var:0', - 'test_mdl/enc/conv_L1/beta/var:0', - 'test_mdl/enc/conv_L1/gamma/var:0', - 'test_mdl/enc/f_conv_lstm_0/wm/var:0', - 'test_mdl/enc/f_conv_lstm_0/b/var:0', - 'test_mdl/enc/b_conv_lstm_0/wm/var:0', - 'test_mdl/enc/b_conv_lstm_0/b/var:0', - 'test_mdl/enc/conv_lstm_cnn_0/w/var:0', - 'test_mdl/enc/conv_lstm_cnn_0/beta/var:0', - 'test_mdl/enc/conv_lstm_cnn_0/gamma/var:0', - 'test_mdl/enc/fwd_rnn_L0/wm/var:0', - 'test_mdl/enc/fwd_rnn_L0/b/var:0', - 'test_mdl/enc/bak_rnn_L0/wm/var:0', - 'test_mdl/enc/bak_rnn_L0/b/var:0', - 'test_mdl/enc/proj_L0/w/var:0', - 'test_mdl/enc/proj_L0/beta/var:0', - 'test_mdl/enc/proj_L0/gamma/var:0', - 'test_mdl/enc/fwd_rnn_L1/wm/var:0', - 'test_mdl/enc/fwd_rnn_L1/b/var:0', - 'test_mdl/enc/bak_rnn_L1/wm/var:0', - 'test_mdl/enc/bak_rnn_L1/b/var:0', - 'test_mdl/enc/proj_L1/w/var:0', - 'test_mdl/enc/proj_L1/beta/var:0', - 'test_mdl/enc/proj_L1/gamma/var:0', - 'test_mdl/enc/fwd_rnn_L2/wm/var:0', - 'test_mdl/enc/fwd_rnn_L2/b/var:0', - 'test_mdl/enc/bak_rnn_L2/wm/var:0', - 'test_mdl/enc/bak_rnn_L2/b/var:0', - 'test_mdl/dec/emb/var_0/var:0', - 'test_mdl/dec/rnn_cell/wm/var:0', - 'test_mdl/dec/rnn_cell/b/var:0', - 'test_mdl/dec/atten/source_var/var:0', - 'test_mdl/dec/atten/query_var/var:0', - 'test_mdl/dec/atten/hidden_var/var:0', - 'test_mdl/dec/softmax/weight_0/var:0', - 'test_mdl/dec/softmax/bias_0/var:0', - ] - self.assertCountEqual(expected_var_names, actual_var_names) - - def testDecode(self): - with self.session(use_gpu=False): - tf.random.set_seed(93820985) - p = self._testParams() - mdl = p.Instantiate() - input_batch = mdl.input_generator.GetPreprocessedInputBatch() - dec_out_dict = mdl.DecodeWithTheta(mdl.theta, input_batch) - self.evaluate(tf.global_variables_initializer()) - dec_out = self.evaluate(dec_out_dict) - print('dec_out', dec_out) - metrics_dict = mdl.CreateDecoderMetrics() - key_value_pairs = mdl.PostProcessDecodeOut(dec_out, metrics_dict) - - self.assertEqual(1.0, metrics_dict['wer'].value) - self.assertEqual(1.0, metrics_dict['norm_wer'].value) - self.assertEqual(1.0, metrics_dict['ter'].value) - self.assertEqual(0, len(key_value_pairs)) - - def testPostProcessDecodeOut(self): - p = self._testParams() - p.decoder.beam_search.num_hyps_per_beam = 2 - mdl = p.Instantiate() - fake_dec_out = { - 'utt_id': ['utt1', 'utt2'], - 'transcripts': ['a b c d', 'a'], - 'topk_decoded': [['a b c d', 'a b c d'], ['wrong', '']], - 'topk_scores': [[1.0, 0.9], [1.0, 0.9]], - 'topk_ids': [[1, 2, 3, 4], [2, 3, 4, 5], [3, 4, 5, 6], [4, 5, 6, 7]], - 'topk_lens': [2, 4, 4, 2], - 'target_labels': [[1, 2, 3, 4], [2, 3, 4, 5]], - 'target_paddings': [[0, 0, 0, 1], [0, 0, 0, 1]], - 'norm_wer_errors': [[0, 0], [1, 1]], - 'norm_wer_words': [[4, 4], [1, 1]], - } - fake_dec_out = {k: np.array(v) for k, v in fake_dec_out.items()} - metrics_dict = mdl.CreateDecoderMetrics() - key_value_pairs = mdl.PostProcessDecodeOut(fake_dec_out, metrics_dict) - - self.assertEqual(0 + 1, metrics_dict['wer'].total_value) - self.assertEqual(4 + 1, metrics_dict['wer'].total_weight) - self.assertEqual(0 + 1, metrics_dict['norm_wer'].total_value) - self.assertEqual(4 + 1, metrics_dict['norm_wer'].total_weight) - self.assertEqual(4, metrics_dict['ter'].total_value) - self.assertEqual(6, metrics_dict['ter'].total_weight) - self.assertEqual(2, metrics_dict['num_samples_in_batch'].total_value) - self.assertEqual(1.0, metrics_dict['num_samples_in_batch'].total_weight) - self.assertEqual((4 / 5 * 3 / 3 * 2 / 2 * 1 / 1)**(1 / 4), - metrics_dict['corpus_bleu'].value) - self.assertEqual((0 + 1) / 2, metrics_dict['sacc'].value) - self.assertEqual((0 + 1) / (4 + 1), metrics_dict['oracle_norm_wer'].value) - self.assertEqual(0, len(key_value_pairs)) - - def testPostProcessDecodeOutFiltersEpsilonTokensForWER(self): - p = self._testParams() - p.decoder.beam_search.num_hyps_per_beam = 1 - mdl = p.Instantiate() - fake_dec_out = { - 'utt_id': ['utt1', 'utt2'], - 'transcripts': ['a b c d', 'a b c'], - 'topk_decoded': [['a bc d'], ['a b']], - 'topk_scores': [[1.0], [1.0]], - 'topk_ids': [[1, 2, 3, 4], [2, 3, 4, 5]], - 'topk_lens': [3, 4], - 'target_labels': [[1, 2, 3, 4], [2, 3, 4, 5]], - 'target_paddings': [[0, 0, 0, 1], [0, 0, 1, 1]], - 'norm_wer_errors': [[0], [1]], - 'norm_wer_words': [[4], [3]], - } - fake_dec_out = {k: np.array(v) for k, v in fake_dec_out.items()} - metrics_dict = mdl.CreateDecoderMetrics() - kv_pairs = mdl.PostProcessDecodeOut(fake_dec_out, metrics_dict) - - self.assertEqual(0 + 1, metrics_dict['wer'].total_value) - self.assertEqual(7, metrics_dict['wer'].total_weight) - self.assertEqual(0 + 1, metrics_dict['norm_wer'].total_value) - self.assertEqual(7, metrics_dict['norm_wer'].total_weight) - self.assertEqual(0, len(kv_pairs)) - - def testPostProcessDecodeOutFiltersNoiseTokensForWER(self): - p = self._testParams() - p.decoder.beam_search.num_hyps_per_beam = 1 - mdl = p.Instantiate() - fake_dec_out = { - 'utt_id': ['utt1', 'utt2'], - 'transcripts': ['a b c d', 'a b c'], - 'topk_decoded': [['a b c d'], [' a b ']], - 'topk_scores': [[1.0], [1.0]], - 'topk_ids': [[1, 2, 3, 4], [2, 3, 4, 5]], - 'topk_lens': [3, 4], - 'target_labels': [[1, 2, 3, 4], [2, 3, 4, 5]], - 'target_paddings': [[0, 0, 0, 1], [0, 0, 1, 1]], - 'norm_wer_errors': [[0], [1]], - 'norm_wer_words': [[4], [3]], - } - fake_dec_out = {k: np.array(v) for k, v in fake_dec_out.items()} - metrics_dict = mdl.CreateDecoderMetrics() - kv_pairs = mdl.PostProcessDecodeOut(fake_dec_out, metrics_dict) - - self.assertEqual(0 + 1, metrics_dict['wer'].total_value) - self.assertEqual(7, metrics_dict['wer'].total_weight) - self.assertEqual(0 + 1, metrics_dict['norm_wer'].total_value) - self.assertEqual(7, metrics_dict['norm_wer'].total_weight) - self.assertEqual(0, len(kv_pairs)) - - def testPostProcessDecodeOutHandlesEmptyRef(self): - p = self._testParams() - p.decoder.beam_search.num_hyps_per_beam = 1 - mdl = p.Instantiate() - fake_dec_out = { - 'utt_id': ['utt1', 'utt2'], - 'transcripts': ['', 'a b c d'], - 'topk_decoded': [['a'], ['a b c d']], - 'topk_scores': [[1.0], [1.0]], - 'topk_ids': [[1, 2, 3, 4], [2, 3, 4, 5]], - 'topk_lens': [3, 4], - 'target_labels': [[1, 2, 3, 4], [2, 3, 4, 5]], - 'target_paddings': [[1, 1, 1, 1], [0, 0, 1, 1]], - 'norm_wer_errors': [[1], [0]], - 'norm_wer_words': [[0], [4]], - } - fake_dec_out = {k: np.array(v) for k, v in fake_dec_out.items()} - metrics_dict = mdl.CreateDecoderMetrics() - mdl.PostProcessDecodeOut(fake_dec_out, metrics_dict) - - self.assertEqual(1 + 0, metrics_dict['wer'].total_value) - self.assertEqual(0 + 4, metrics_dict['wer'].total_weight) - self.assertEqual(1 + 0, metrics_dict['norm_wer'].total_value) - self.assertEqual(0 + 4, metrics_dict['norm_wer'].total_weight) - - def testBProp(self): - with self.session(use_gpu=False): - tf.random.set_seed(93820985) - p = self._testParams() - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - mdl.BProp() - self.evaluate(tf.global_variables_initializer()) - test_utils.CompareToGoldenSingleFloat(self, 4.472597, mdl.loss.eval()) - mdl.train_op.run() - - def testBPropSmoothDecay(self): - with self.session(use_gpu=False): - tf.random.set_seed(93820985) - p = self._testParams() - p.train.lr_schedule = ( - schedule.ContinuousSchedule.Params().Set( - start_step=350000, half_life_steps=45000)) - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - mdl.BProp() - self.evaluate(tf.global_variables_initializer()) - test_utils.CompareToGoldenSingleFloat(self, 4.472597, mdl.loss.eval()) - mdl.train_op.run() - - def testAllLayerParams(self): - with self.session(use_gpu=False, graph=tf.Graph()): - p = self._testParams() - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - lps = base_layer.RecursiveFindLayerParams(mdl.params) - l_names = sorted([p.cls.__name__ for p in lps]) - expected_layers = sorted([ - 'Adam', - 'AdditiveAttention', - 'AsciiTokenizer', - 'AsrDecoder', - 'AsrEncoder', - 'AsrModel', - 'BatchNormLayer', - 'BeamSearchHelper', - 'GreedySearchHelper', - 'TargetSequenceSampler', - 'ConvLSTMCell', - 'Conv2DLayer', - 'Conv2DLayer', - 'EmbeddingLayer', - 'HighwaySkipLayer', - 'LSTMCellSimple', - 'LSTMCellSimple', - 'LayerNorm', - 'NullContextualizer', - 'NullFusion', - 'NullLm', - 'Learner', - 'MultitaskAdapterLayer', - 'PiecewiseConstantSchedule', - 'ProjectionLayer', - 'SimpleFullSoftmax', - 'SpectrumAugmenter', - 'StackingOverTime', - 'TestInputGenerator', - ]) - self.assertEqual(expected_layers, l_names) - - def testParamValueSumSquared(self): - with self.session(use_gpu=False, graph=tf.Graph()): - p = self._testParams() - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - all_vars = tf.trainable_variables() - py_utils.SumSquared(all_vars) - - def testCollectVarHistogram(self): - with self.session(use_gpu=False, graph=tf.Graph()): - p = self._testParams() - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - var_grads = py_utils.ComputeGradients(mdl.loss, mdl.vars) - summary_utils.CollectVarHistogram(var_grads) - - def testGradientMult(self): - with self.session(use_gpu=False, graph=tf.Graph()): - p = self._testParams() - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - var_grads = py_utils.ComputeGradients(mdl.loss, mdl.vars) - py_utils.ApplyGradMultiplier(var_grads, -1.1) - - def testLRDecay(self): - with self.session(use_gpu=False, graph=tf.Graph()): - p = self._testParams() - tp = p.train - tp.lr_schedule.boundaries = [300000, 400000, 500000] - tp.lr_schedule.values = [1.0, 0.1, 0.01, 0.001] - lrs = tp.lr_schedule.Instantiate() - steps = [299999, 300001, 399999, 400001, 499999, 500001] - fetches = [lrs.Value(_) for _ in steps] - values = self.evaluate(fetches) - self.assertAllClose([1.0, 0.1, 0.1, 0.01, 0.01, 0.001], values) - - def testBatchSplit(self): - - def Run(num_splits): - p = self._testParams() - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(93820981) - p.input.cur_iter_in_seed = False - p.input.bucket_batch_limit = [ - b * 2 / num_splits for b in p.input.bucket_batch_limit - ] - with cluster_factory.ForTestingWorker(gpus=num_splits, do_eval=True): - mdl = p.Instantiate() - metrics = mdl.FPropDefaultTheta()[0] - self.evaluate(tf.global_variables_initializer()) - return self.evaluate(metrics['loss']) - - res1, res2 = Run(1), Run(2) - self.assertAllClose(res1[0], res2[0]) - self.assertAllEqual(res1[1], res2[1]) - - def testInference(self): - - def _CreateModelParamsForTest(): - p = model.AsrModel.Params() - p.name = 'test_config' - - # Encoder params. - ep = p.encoder - ep.input_shape = [None, None, 80, 1] - ep.lstm_cell_size = 16 - ep.num_lstm_layers = 2 - ep.conv_filter_shapes = [(3, 3, 1, 32), (3, 3, 32, 32)] - ep.conv_filter_strides = [(2, 2), (2, 2)] - ep.num_conv_lstm_layers = 0 - # Initialize decoder params. - dp = p.decoder - dp.rnn_cell_dim = 16 - dp.rnn_layers = 2 - dp.source_dim = ep.lstm_cell_size * 2 - # Use functional while based unrolling. - dp.use_while_loop_based_unrolling = False - - p.input = input_generator.AsrInput.Params() - ip = p.input - ip.frame_size = 80 - ip.append_eos_frame = True - ip.pad_to_max_seq_length = False - return p - - with self.session( - use_gpu=False, graph=tf.Graph()) as sess, self.SetEval(True): - p = _CreateModelParamsForTest() - mdl = p.Instantiate() - subgraphs = mdl.Inference() - self.assertIn('default', subgraphs) - - fetches, feeds = subgraphs['default'] - self.assertIn('wav', feeds) - for name in ['hypotheses', 'scores', 'src_frames', 'encoder_frames']: - self.assertIn(name, fetches) - - with open( - test_helper.test_src_dir_path('tools/testdata/gan_or_vae.16k.wav'), - 'rb') as f: - wav = f.read() - self.evaluate(tf.global_variables_initializer()) - fetches = sess.run(fetches, {feeds['wav']: wav}) - - self.assertAllEqual((1, p.decoder.beam_search.num_hyps_per_beam), - fetches['hypotheses'].shape) - self.assertAllEqual((1, p.decoder.beam_search.num_hyps_per_beam), - fetches['scores'].shape) - self.assertAllEqual((1, 314, p.encoder.input_shape[2], 1), - fetches['src_frames'].shape) - self.assertAllEqual((80, 1, 2 * p.encoder.lstm_cell_size), - fetches['encoder_frames'].shape) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/asr/model_test_input_generator.py b/lingvo/tasks/asr/model_test_input_generator.py deleted file mode 100644 index 28baa243..00000000 --- a/lingvo/tasks/asr/model_test_input_generator.py +++ /dev/null @@ -1,268 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Simple input generator used for ASR model tests.""" - -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import py_utils - - -class TestInputGenerator(base_input_generator.BaseSequenceInputGenerator): - """A simple InputGenerator that delegate requests to another obj.""" - - @classmethod - def Params(cls): - p = super().Params() - p.random_seed = 20349582 - p.Define('feature_dims', 240, 'Feature dims') - p.Define('num_channels', 3, 'Data are preprocessed into these many ' - 'channels per timestep. E.g., feature_dims=240 is actually ' - '[80, 3], i.e., 3 channels, each with a 40-dim feature vector.') - p.Define('source_shape', [2, 10, 8, 3], 'source shape.') - p.Define('target_shape', [2, 5], 'targets shape.') - p.Define('fixed_target_labels', None, - 'If not None, use these as the targets instead of generating ' - 'random targets and set target padding to 0. Must have same ' - 'shape as target_shape.') - p.Define( - 'fixed_target_ids', None, - 'If not None, use these as the target_ids instead of generating ' - 'random targets. Must have same shape as target_shape.') - p.Define('cur_iter_in_seed', True, 'use current iter value in seed ' - 'computation.') - p.Define('integer_source_max', None, 'Generate integers as source values ' - 'with this value as an upper bound.') - p.Define( - 'float_source_max', None, 'Generate floats as source values ' - 'with this value as an upper bound.') - p.Define('for_mt', False, 'True if this is for mt models; ' - 'this affects some parts of batch generation') - p.Define('target_key', '', 'If non-empty, targets will be specified in ' - 'batch.additional_tgts[target_key] instead of batch.tgt.') - p.Define('target_key_target_shape', [2, 5], 'Shape of the targets stored ' - 'batch.additional_tgts[target_key].') - p.Define('set_tgt_and_additional_tgts', False, 'If true, ' - 'both batch.tgt and batch.additional_tgts[target_key] will ' - 'be set. target_key_target_shape must be specified.') - p.Define('target_language', 'ENGLISH', - 'The target language. Both language name (e.g. "ENGLISH") and ' - 'language code (e.g. "zh-CN") are acceptted.') - p.Define('align_label_with_frame', False, - 'Whether to generate label-frame alignments.') - p.Define( - 'bprop_filters', [], 'If set, simulates a multi source' - 'input and sets filters for each source i.e the first filter' - 'corresponds to the first source etc. The number of sources is set' - 'to the length of this param.') - p.Define( - 'number_sources', None, 'Integer which specifies the number of' - 'sources. Cannot be used along with bprop_filters.') - p.Define( - 'source_selected', None, 'Integer which specifies the index of the' - 'source selected. Corresponds to the data source that would be' - 'sampled by the input_generator when given multiple file_patterns.' - 'This has an effect only when number_sources is set and greater than 1.' - 'Can use either constant values or a tensor like' - 'tf.math.floormod(tf.train.get_or_create_global_step(), num_sources)') - p.Define('target_transcript', 'dummy_transcript', - 'Text to use for transcript.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self._bprop_variable_filters = [''] - self._bprop_onehot = tf.constant([1], dtype=tf.float32) - if p.target_key and not p.target_key_target_shape: - raise ValueError('target_key_target_shape must be set when ' - 'target_key (%s) is not empty.' % p.target_key) - if (p.set_tgt_and_additional_tgts and - (p.target_key_target_shape[0] != p.target_shape[0])): - raise ValueError('The first dimension of target_key_target_shape (%d) ' - 'should match the first dimension of target_shape ' - '(%d) when both have to be set.' % - (p.target_key_target_shape[0], p.target_shape[0])) - self._cur_iter = 0 - if p.bprop_filters and p.number_sources: - raise ValueError( - 'Number of sources will be set to length of bprop_filters, the param' - 'number_sources should not be used when bprop_filters is set.') - number_sources = p.number_sources - if p.bprop_filters: - self._bprop_variable_filters = p.bprop_filters - number_sources = len(p.bprop_filters) - if number_sources and number_sources > 1: - self._bprop_onehot = tf.one_hot( - p.source_selected, number_sources, dtype=tf.float32) - - def _check_paddings(self, paddings): - with tf.name_scope('check_paddings'): - unpacked_paddings = tf.unstack(paddings) - - non_decr = [] - for t in unpacked_paddings: - non_d = tf.math.is_non_decreasing(t) - non_decr.append(non_d) - all_non_decr = tf.stack(non_decr) - - paddings = py_utils.with_dependencies([ - tf.assert_equal( - tf.reduce_any(tf.equal(paddings, 0.0)), - True, - message='must have at least one zero value.'), - tf.assert_equal(all_non_decr, True, message='must be non-decreasing') - ], paddings) - return paddings - - def GetBpropParams(self): - return self._bprop_params - - def GetBpropType(self): - """Get the current bprop type of the input generator batch.""" - return self._bprop_onehot - - def SampleIds(self): - p = self.params - if p.cur_iter_in_seed: - random_seed = p.random_seed * 2000 * self._cur_iter - else: - random_seed = p.random_seed * 2000 - return tf.as_string(tf.random.uniform(p.target_shape[:1], seed=random_seed)) - - def _Sources(self): - p = self.params - if p.cur_iter_in_seed: - self._cur_iter += 1 - - if p.integer_source_max: - inputs = tf.random.uniform( - p.source_shape, - maxval=p.integer_source_max, - dtype=tf.int32, - seed=p.random_seed + 1000 * self._cur_iter) - elif p.float_source_max: - inputs = tf.random.uniform( - p.source_shape, - maxval=p.float_source_max, - seed=p.random_seed + 1000 * self._cur_iter) - else: - inputs = tf.random.normal( - p.source_shape, seed=p.random_seed + 1000 * self._cur_iter) - - paddings = tf.cast( - tf.cumsum( - tf.random.uniform( - p.source_shape[:2], seed=p.random_seed + 1001 * self._cur_iter), - axis=1) > 0.5 * p.source_shape[1], tf.float32) - - paddings = self._check_paddings(paddings) - - return inputs, paddings - - def _Targets(self, target_shape): - p = self.params - if p.cur_iter_in_seed: - self._cur_iter += 1 - random_seed = p.random_seed * 2000 * self._cur_iter - - if p.fixed_target_ids is None: - tids = tf.cast( - tf.random.uniform(target_shape, seed=random_seed) * - p.tokenizer.vocab_size, tf.int32) - else: - tids = p.fixed_target_ids - assert tids.shape_as_list() == target_shape - - if p.fixed_target_labels is None: - tlabels = tf.cast( - tf.random.uniform(target_shape, seed=random_seed + 1) * - p.tokenizer.vocab_size, tf.int32) - tpaddings = tf.cast( - tf.cumsum( - tf.random.uniform( - target_shape[:2], seed=p.random_seed + 1001 * self._cur_iter), - axis=1) > 0.4 * target_shape[1], tf.float32) - tpaddings = self._check_paddings(tpaddings) - else: - tlabels = p.fixed_target_labels - assert tlabels.shape_as_list() == target_shape - tpaddings = tf.constant(0.0, shape=target_shape) - tweights = 1.0 - tpaddings - d = { - 'ids': tids, - 'labels': tlabels, - 'weights': tweights, - 'paddings': tpaddings - } - if not p.for_mt: - d['transcripts'] = tf.constant( - p.target_transcript, shape=[target_shape[0]]) - if p.align_label_with_frame: - source_len = p.source_shape[1] - d['alignments'] = tf.cast( - tf.random.uniform(target_shape, seed=p.random_seed) * source_len, - tf.int32) - return d - - def GlobalBatchSize(self): - p = self.params - return tf.constant(p.target_shape[0]) - - def _InputBatch(self): - p = self.params - ret = py_utils.NestedMap() - ret.src = py_utils.NestedMap() - input_name = 'ids' if p.for_mt else 'src_inputs' - ret.src[input_name], ret.src.paddings = self._Sources() - - # Set tgts only when needed: If target_key is specified, and both tgt and - # additional_tgts are not needed, we only set additional_tgts. This is - # useful when testing a model that solely uses additional_tgts instead - # of tgt. - if not p.target_key or p.set_tgt_and_additional_tgts: - ret.tgt = py_utils.NestedMap(self._Targets(p.target_shape)) - else: - ret.tgt = None - if p.target_key: - ret.additional_tgts = py_utils.NestedMap() - ret.additional_tgts[p.target_key] = py_utils.NestedMap( - self._Targets(p.target_key_target_shape)) - ret.sample_ids = self.SampleIds() - - # Cast floating point tensors to the fprop dtype (default: float32). - def _CastFloats(v): - if v is None: - return None - return tf.cast(v, py_utils.FPropDtype(p)) if v.dtype.is_floating else v - - ret.source_selected = tf.tile( - tf.expand_dims(self._bprop_onehot, 0), [p.source_shape[0], 1]) - return ret.Transform(_CastFloats) - - def _GetSourceInputsAndLabels(self, data_source): - p = self.params - src_inputs, src_paddings, labels = data_source - # The data are laid out in the channel-major order. In order to move channel - # to the last dimension, a tf.transpose of the data is needed. - src_inputs = tf.transpose( - tf.reshape( - src_inputs, - tf.concat([tf.shape(src_inputs)[:-1], [p.num_channels, -1]], 0)), - [0, 1, 3, 2]) - return src_inputs, src_paddings, labels - - def SetBpropType(self): - """Get the current bprop type of the input generator batch.""" - self._bprop_index = tf.one_hot(1, 2) diff --git a/lingvo/tasks/asr/params/BUILD b/lingvo/tasks/asr/params/BUILD deleted file mode 100644 index 9a9e51ed..00000000 --- a/lingvo/tasks/asr/params/BUILD +++ /dev/null @@ -1,38 +0,0 @@ -# Params for ASR models. - -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -filegroup( - name = "param_files", - srcs = glob( - include = [ - "*.py", - ], - ), -) - -py_library( - name = "params", - srcs = [ - ":param_files", - ], - data = ["//lingvo/tasks/asr:wpm_files"], - srcs_version = "PY3", - tags = ["keep_dep"], - deps = [ - "//lingvo:model_registry", - "//lingvo/core:base_model_params", - "//lingvo/core:datasource", - "//lingvo/core:program_lib", - "//lingvo/core:py_utils", - "//lingvo/core:schedule", - "//lingvo/core:tokenizers", - "//lingvo/tasks/asr:input_generator", - "//lingvo/tasks/asr:model", - "//lingvo/tasks/asr:ctc_model", - ], -) diff --git a/lingvo/tasks/asr/params/README.md b/lingvo/tasks/asr/params/README.md deleted file mode 100644 index dcc7ab5b..00000000 --- a/lingvo/tasks/asr/params/README.md +++ /dev/null @@ -1,75 +0,0 @@ -# Setting up your machine for featurization / training / eval - -Set up conda environment and build lingvo using bazel on VM / local machine -```sh -# install ubuntu dependencies first -sudo apt install wget git screen neovim build-essential htop parallel sox libsox-fmt-mp3 - -# Download miniconda -wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -chmod +x Miniconda3-latest-Linux-x86_64.sh -./Miniconda3-latest-Linux-x86_64.sh - -# Create conda environment for lingvo, includes development tools (like yapf, pylint) as well as cuda -conda env create -n lingvo -f lingvo-copy/environment.yml -echo "conda activate lingvo" >> ~/.bashrc -bazel build -c opt //lingvo:trainer -``` -If you are running featurization you will need to create an extra volume when provisioning the VM. Then you will need to mount it. I have added the commands to `setup.sh` just in case, you can also find it in google cloud documentation. - -# Submit a training job on TPU -Make sure you are inside a google cloud cpu VM, and using the `lingvo` conda environment. -```sh -./submit.sh $device_name executor $class_model_name $fldr_name - -## Example usage: -./submit.sh ag-tpu1-1019 executor Grphm_DO_SpecAug_ConvStk_6x512Bidi_40batchsize 1127 -``` - -`device_name`: The `device_name` contains `-tpu`, script assumes you are training on a tpu. If a TPU by that name exists, the TPU is reused, if not, a new TPU is created with that name and the job is launched. `device_name` can also be `gpu` or `cpu`, and the job is launched locally in either of these cases. - -`executor`: Lingvo has a bunch of `runner` classes for managing the training loop. We use the `executor_tpu` for training on the tpu. Use `executor` when training on tpu, or `trainer` when running locally on cpu/gpu. Training on local machines is not fully tested, you may have to change batch size, data dir etc when training locally. -The tpu_executor runs a few iterations of training, then a few iterations of evaluation on the train and dev set, so you'll see 3 loss curves on tensorboard. -- train-train: training loss on the training dataset with augmentation (dropout, specuagment) -- eval-train: eval loss on the training dataset, without augmentation (should be better than training loss) -- eval-dev: eval loss on the dev dataset. I use this curve for picking which model to choose for evaluation. - -`class_model_name`: One of the fully specified model clases available in `lingvo/tasks/asr/params/peoplesspeech_ctc.py`. Use this class to specify all hyper-parameters. The class name in example above is our sota hyper-parameter spec for librispeech. You can also find the location of the training / dev / test data specified in this file. - -`fldr_name`: Google cloud storage suffix where to save the checkpoints and tensorboard events. *IMPORTANT*: Change the GS_BASE path in `tpu_submit` to your directory (right now it's pointing to `gs://the-peoples-speech-west-europe/PeoplesSpeech/ag_training`). If there are checkpoints of `$class_model_name` in `$fldr_name` directory, you will be restarting that job, and the folder will be reused for new checkpoints. - -# Eval a model on GPU/CPU: -We could not get the decoder / evaluation to run on TPU. Lingvo supports marking some functions to be only run on the tpu_host", but even this was not sufficient to make the decoding work, while the forward prop is running on the tpu. Therefore, I run the eval on my local gpu machine. You could also run on a cpu machine, but that is super slow. On a gtx1070 evaluation finishes in ~15 minutes. Make sure you set up the conda environment, just like above, and setup google cloud cli from https://cloud.google.com/sdk/docs/install - -Download the latest checkpoint, and tensorboard events of all the models in the experiment group to `$HOME_BASE` which is hardcoded in `download_models.sh`. -```sh -./download_models.sh ${fldr_name} - -## Example usage: -./download_models.sh 1127 -``` -Downloads from `gs://the-peoples-speech-west-europe/PeoplesSpeech/ag_training/1127` to `/home/anjali/data/PeoplesSpeech/models/1127`. Now you can evaluate the model on the dev set and see sample predictions, cer, wer using -```sh -./submit.sh gpu decode Grphm_DO_SpecAug_ConvStk_6x512Bidi_40batchsize 1127 -``` -Since I run locally on my home machine, I sometimes run into google cloud storage errors when evaling. So I have a copy of the devtest set locally - you may need this hack. You could also just get a small gpu vm for 1 hour and avoid all the model and dataset downloading. - -After one pass, the process will keep monitoring the checkpoint directory for new checkpoints. The decoder job dumps a lot of the decoded output to tensorboard, so you should not let the decoder continue past an epoch. - -# Featurizing a new dataset: -Make sure you have allocated a large disk on a large VM (>16 cores). See setup instructions above. -```sh -# Copy all the tarballs and csv files locally, takes a few hours -mkdir -p /mnt/disks/dataset/raw/v0.7/raw /mnt/disks/dataset/raw/v0.7/feats -gsutil -m rsync the-peoples-speech-west-europe/peoples-speech-v0.7 /mnt/disks/dataset/raw/v0.7/raw - -# featurize the dataset, usually takes ~24 hours -# ./featurize.sh $input_dir $output_dir -./featurize.sh /mnt/disks/dataset/raw/v0.7/raw /mnt/disks/dataset/raw/v0.7/feats - -# Upload the tfrecords back to the cloud -gsutil -m rsync /mnt/disks/dataset/raw/v0.7/feats gs://the-peoples-speech-west-europe/PeoplesSpeech/v0.7.1 -``` -`featurize.sh` calls `create_peoples_speech_asr_features.py` which first loads the csv file to read the audio filenames and transcripts (other metadata is thrown away) into a dictionary, keyed by the audio filepath. Then it makes a pass through the tarball, reading one audio file at a time, standardizing it (to 16Khz, single channel, 16 bit wav audio), calculates mel spectrograms (80 bins, 25ms window, 10ms step) and then saves it to tfrecord. - -The featurization outputs 1 tfrecord file each for dev and test set. Training dataset is split into 512 tfrecords, and we use 16 processes in parallel to do the featurization. Worker 0 outputs files 0..31, worker 1 outputs 32..63 and so on. I picked 512 files so that each tfrecord file is around 1GB, same as librispeech. Therefore, there are 512 output_shards, and 16 worker shards. diff --git a/lingvo/tasks/asr/params/inference_only.py b/lingvo/tasks/asr/params/inference_only.py deleted file mode 100644 index 2a23b869..00000000 --- a/lingvo/tasks/asr/params/inference_only.py +++ /dev/null @@ -1,94 +0,0 @@ -from lingvo import model_registry -from lingvo.core import base_model_params -from lingvo.core import datasource -from lingvo.core import program -from lingvo.core import py_utils -from lingvo.core import schedule -from lingvo.core import tokenizers -from lingvo.tasks.asr import input_generator -from lingvo.tasks.asr import ctc_model -from lingvo.tasks.asr import frontend as asr_frontend - -@model_registry.RegisterSingleTaskModel -class InferenceOnly(base_model_params.SingleTaskModelParams): - def RawInputParams(self): - """ - Reads in a raw waveform, where samples are float32 and in the range[-1,1] - """ - p = input_generator.AsrInput.Params() - - p.file_datasource = datasource.PrefixedDataSource.Params() - p.file_datasource.file_type = 'tfrecord' - p.file_datasource.file_pattern_prefix = 'gs://the-peoples-speech-west-europe/Librispeech' - - # We have only - p.frame_size = 1 - p.append_eos_frame = False - - p.pad_to_max_seq_length = True - p.file_random_seed = 0 - p.file_buffer_size = 10000 - # Set this to whatever - p.file_parallelism = 2 - - # Should be set very high... - p.source_max_length = 40_000 # 7412603 - 7389537 - p.bucket_upper_bound = [40_000] - - p.bucket_batch_limit = [1] - - return p - - def Inference(self): - p = self.RawInputParams() - p.file_datasource.file_pattern = '*.tfrecord' - # Keep it simple for now. This should be overridable - p.num_samples = 8 - return p - - def Task(self): - p = ctc_model.CTCModel.Params() - p.name = 'librispeech' - - # This is copied from audio_lib.py. Ideally these configurations - # would be split off into a function, but I want to minimize merge - # conflicts with lingvo for now. Of course, a merge conflict would - # indicate that parameters had changed, so my choice is clearly - # wrong, but oh well. - p.frontend = asr_frontend.MelAsrFrontend.Params() - pf = p.frontend - pf.sample_rate = 16000. - pf.frame_size_ms = 25. - pf.frame_step_ms = 10. - pf.num_bins = 80 - pf.lower_edge_hertz = 125. - pf.upper_edge_hertz = 7600. - pf.preemph = 0.97 - pf.noise_scale = 0. - pf.pad_end = False - - # No default encoder params in this class. - - tp = p.train - tp.learning_rate = 1e-4 - tp.lr_schedule = schedule.ContinuousSchedule.Params().Set( - start_step=25_000, half_life_steps=5_000, min=1e-6) - tp.scale_gradients = False - tp.l2_regularizer_weight = None - - # Setting p.eval.samples_per_summary to a large value ensures that dev, - # devother, test, testother are evaluated completely (since num_samples for - # each of these sets is less than 5000), while train summaries will be - # computed on 5000 examples. - p.eval.samples_per_summary = 2700 - p.eval.decoder_samples_per_summary = 2700 - - return p - - def ProgramSchedule(self): - return program.SimpleProgramScheduleForTask( - train_dataset_name='Train', - train_steps_per_loop=500, - eval_dataset_names=['Inference'], - eval_steps_per_loop=1, - decode_steps_per_loop=0) diff --git a/lingvo/tasks/asr/params/librispeech.py b/lingvo/tasks/asr/params/librispeech.py deleted file mode 100644 index 0e00e7eb..00000000 --- a/lingvo/tasks/asr/params/librispeech.py +++ /dev/null @@ -1,330 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Models for Librispeech dataset.""" - -from lingvo import model_registry -from lingvo.core import base_model_params -from lingvo.core import datasource -from lingvo.core import program -from lingvo.core import py_utils -from lingvo.core import schedule -from lingvo.core import tokenizers -from lingvo.tasks.asr import input_generator -from lingvo.tasks.asr import model - - -@model_registry.RegisterSingleTaskModel -class Librispeech960Base(base_model_params.SingleTaskModelParams): - """Base parameters for Librispeech 960 hour task.""" - - def _CommonInputParams(self, is_eval): - """Input generator params for Librispeech.""" - p = input_generator.AsrInput.Params() - - # Insert path to the base directory where the data are stored here. - # Generated using scripts in lingvo/tasks/asr/tools. - p.file_datasource = datasource.PrefixedDataSource.Params() - p.file_datasource.file_type = 'tfrecord' - p.file_datasource.file_pattern_prefix = '/tmp/librispeech' - - p.frame_size = 80 - p.append_eos_frame = True - - p.pad_to_max_seq_length = False - p.file_random_seed = 0 - p.file_buffer_size = 10000 - p.file_parallelism = 16 - - if is_eval: - p.source_max_length = 3600 - p.bucket_upper_bound = [639, 1062, 1275, 1377, 1449, 1506, 1563, 3600] - else: - p.source_max_length = 3000 - p.bucket_upper_bound = [639, 1062, 1275, 1377, 1449, 1506, 1563, 1710] - - p.bucket_batch_limit = [96, 48, 48, 48, 48, 48, 48, 48] - - return p - - def SetBucketSizes(self, params, bucket_upper_bound, bucket_batch_limit): - """Sets bucket sizes for batches in params.""" - params.bucket_upper_bound = bucket_upper_bound - params.bucket_batch_limit = bucket_batch_limit - return params - - def Train(self): - p = self._CommonInputParams(is_eval=False) - p.file_datasource.file_pattern = 'train/train.tfrecords-*' - p.num_samples = 281241 - return p - - def Dev(self): - p = self._CommonInputParams(is_eval=True) - p.file_datasource.file_pattern = ( - 'devtest/dev-clean.tfrecords-00000-of-00001') - p.num_samples = 2703 - return p - - def Devother(self): - p = self._CommonInputParams(is_eval=True) - p.file_datasource.file_pattern = ( - 'devtest/dev-other.tfrecords-00000-of-00001') - p.num_samples = 2864 - return p - - def Test(self): - p = self._CommonInputParams(is_eval=True) - p.file_datasource.file_pattern = ( - 'devtest/test-clean.tfrecords-00000-of-00001') - p.num_samples = 2620 - return p - - def Testother(self): - p = self._CommonInputParams(is_eval=True) - p.file_datasource.file_pattern = ( - 'devtest/test-other.tfrecords-00000-of-00001') - p.num_samples = 2939 - return p - - def Task(self): - p = model.AsrModel.Params() - p.name = 'librispeech' - - # Initialize encoder params. - ep = p.encoder - # Data consists 240 dimensional frames (80 x 3 frames), which we - # re-interpret as individual 80 dimensional frames. See also, - # LibrispeechCommonAsrInputParams. - ep.input_shape = [None, None, 80, 1] - ep.lstm_cell_size = 1024 - ep.num_lstm_layers = 4 - ep.conv_filter_shapes = [(3, 3, 1, 32), (3, 3, 32, 32)] - ep.conv_filter_strides = [(2, 2), (2, 2)] - ep.cnn_tpl.params_init = py_utils.WeightInit.Gaussian(0.001) - # Disable conv LSTM layers. - ep.num_conv_lstm_layers = 0 - - # Initialize decoder params. - dp = p.decoder - dp.rnn_cell_dim = 1024 - dp.rnn_layers = 2 - dp.source_dim = 2048 - # Use functional while based unrolling. - dp.use_while_loop_based_unrolling = False - - tp = p.train - tp.learning_rate = 2.5e-4 - tp.lr_schedule = schedule.ContinuousSchedule.Params().Set( - start_step=50000, half_life_steps=100000, min=0.01) - - # Setting p.eval.samples_per_summary to a large value ensures that dev, - # devother, test, testother are evaluated completely (since num_samples for - # each of these sets is less than 5000), while train summaries will be - # computed on 5000 examples. - p.eval.samples_per_summary = 5000 - p.eval.decoder_samples_per_summary = 0 - - # Use variational weight noise to prevent overfitting. - p.vn.global_vn = True - p.train.vn_std = 0.075 - p.train.vn_start_step = 20000 - - return p - - def ProgramSchedule(self): - return program.SimpleProgramScheduleForTask( - train_dataset_name='Train', - train_steps_per_loop=50, - eval_dataset_names=['Test'], - eval_steps_per_loop=5, - decode_steps_per_loop=0) - - -@model_registry.RegisterSingleTaskModel -class Librispeech960Grapheme(Librispeech960Base): - """Base params for Librispeech 960 hour experiments using grapheme models. - - With 8 workers using asynchronous gradient descent on 16 (8x2) GPUs, the model - achieves the following error rates after ~853.2K steps: - - ========= ===== - Dev 5.2% - DevOther 15.2% - Test 5.4% - TestOther 15.5% - ========= ===== - """ - - GRAPHEME_TARGET_SEQUENCE_LENGTH = 620 - GRAPHEME_VOCAB_SIZE = 76 - - def InitializeTokenizer(self, params): - """Initializes a grapheme tokenizer.""" - params.tokenizer = tokenizers.AsciiTokenizer.Params() - tokp = params.tokenizer - tokp.vocab_size = self.GRAPHEME_VOCAB_SIZE - tokp.append_eos = True - tokp.target_unk_id = 0 - tokp.target_sos_id = 1 - tokp.target_eos_id = 2 - - params.target_max_length = self.GRAPHEME_TARGET_SEQUENCE_LENGTH - return params - - def Train(self): - p = super().Train() - return self.InitializeTokenizer(params=p) - - def Dev(self): - p = super().Dev() - return self.InitializeTokenizer(params=p) - - def Devother(self): - p = super().Devother() - return self.InitializeTokenizer(params=p) - - def Test(self): - p = super().Test() - return self.InitializeTokenizer(params=p) - - def Testother(self): - p = super().Testother() - return self.InitializeTokenizer(params=p) - - def Task(self): - p = super().Task() - dp = p.decoder - dp.target_seq_len = self.GRAPHEME_TARGET_SEQUENCE_LENGTH - dp.emb_dim = self.GRAPHEME_VOCAB_SIZE - dp.emb.vocab_size = self.GRAPHEME_VOCAB_SIZE - dp.softmax.num_classes = self.GRAPHEME_VOCAB_SIZE - return p - - -@model_registry.RegisterSingleTaskModel -class Librispeech960GraphemeTpuV2(Librispeech960Grapheme): - """Librispeech 960 grapheme model for training on TPU V2.""" - - def _CommonInputParams(self, is_eval): - p = super()._CommonInputParams(is_eval) - - if py_utils.use_tpu(): - p.pad_to_max_seq_length = True - p.bucket_batch_limit = [48] * len(p.bucket_upper_bound) - p.source_max_length = p.bucket_upper_bound[-1] - - return p - - def Task(self): - p = super().Task() - - p.encoder.pad_steps = 0 - - return p - - -@model_registry.RegisterSingleTaskModel -class Librispeech960Wpm(Librispeech960Base): - """Base params for Librispeech 960 hour experiments using Word Piece Models. - - With 8 workers using asynchronous gradient descent on 16 (8x2) GPUs, the model - achieves the following error rates after ~632.6K steps: - - ========= ===== - Dev 4.3% - DevOther 13.0% - Test 4.5% - TestOther 13.2% - ========= ===== - """ - - # Set this to a WPM vocabulary file before training. By default, we use the - # pre-generated 16K word piece vocabulary checked in under 'tasks/asr/'. - WPM_SYMBOL_TABLE_FILEPATH = ( - 'lingvo/tasks/asr/wpm_16k_librispeech.vocab') - WPM_TARGET_SEQUENCE_LENGTH = 140 - WPM_VOCAB_SIZE = 16328 - - EMBEDDING_DIMENSION = 96 - NUM_TRAINING_WORKERS = 8 - - def InitializeTokenizer(self, params): - """Initializes a Word Piece Tokenizer.""" - params.tokenizer = tokenizers.WpmTokenizer.Params() - tokp = params.tokenizer - tokp.vocab_filepath = self.WPM_SYMBOL_TABLE_FILEPATH - tokp.vocab_size = self.WPM_VOCAB_SIZE - tokp.append_eos = True - tokp.target_unk_id = 0 - tokp.target_sos_id = 1 - tokp.target_eos_id = 2 - - params.target_max_length = self.WPM_TARGET_SEQUENCE_LENGTH - return params - - def Train(self): - p = super().Train() - return self.InitializeTokenizer(params=p) - - def Dev(self): - p = super().Dev() - return self.InitializeTokenizer(params=p) - - def Devother(self): - p = super().Devother() - return self.InitializeTokenizer(params=p) - - def Test(self): - p = super().Test() - return self.InitializeTokenizer(params=p) - - def Testother(self): - p = super().Testother() - return self.InitializeTokenizer(params=p) - - def Task(self): - p = super().Task() - dp = p.decoder - dp.target_seq_len = self.WPM_TARGET_SEQUENCE_LENGTH - dp.emb_dim = self.EMBEDDING_DIMENSION - dp.emb.vocab_size = self.WPM_VOCAB_SIZE - dp.emb.max_num_shards = self.NUM_TRAINING_WORKERS # One shard per worker. - dp.softmax.num_classes = self.WPM_VOCAB_SIZE - - return p - - -@model_registry.RegisterSingleTaskModel -class Librispeech960WpmTpuV2(Librispeech960Wpm): - """Librispeech 960 WPM model for training on TPU V2.""" - - def _CommonInputParams(self, is_eval): - p = super()._CommonInputParams(is_eval) - - if py_utils.use_tpu(): - p.pad_to_max_seq_length = True - p.bucket_batch_limit = [48] * len(p.bucket_upper_bound) - p.source_max_length = p.bucket_upper_bound[-1] - - return p - - def Task(self): - p = super().Task() - - p.encoder.pad_steps = 0 - p.decoder.emb.max_num_shards = 1 - - return p diff --git a/lingvo/tasks/asr/params/librispeech_ctc.py b/lingvo/tasks/asr/params/librispeech_ctc.py deleted file mode 100644 index d881e0bf..00000000 --- a/lingvo/tasks/asr/params/librispeech_ctc.py +++ /dev/null @@ -1,462 +0,0 @@ -from lingvo import model_registry -import lingvo.compat as tf -from lingvo.core import base_model_params -from lingvo.core import datasource -from lingvo.core import program -from lingvo.core import py_utils -from lingvo.core import schedule -from lingvo.core import tokenizers -from lingvo.tasks.asr import input_generator -from lingvo.tasks.asr import ctc_model -from lingvo.tasks.asr import frontend as asr_frontend - -VOCAB_FILEPATH = "gs://the-peoples-speech-west-europe/Librispeech/tokens2.txt" - -tf.flags.DEFINE_integer( - 'ctc_inference_model_num_samples', 0, 'Total number of samples to run inference on. Should be equal to input.num_samples (this is checked)') -tf.flags.DEFINE_integer( - 'ctc_inference_model_batch_size', 0, - '') -tf.flags.DEFINE_integer( - 'ctc_inference_model_number_of_tpus', 8, - '') - - - -FLAGS = tf.flags.FLAGS - - -def intialize_vocab_file_tokenizer_params(): - pt = tokenizers.VocabFileTokenizer.Params() - pt.append_eos = False - - # TODO: Don't hard-code this path! How can I make it relative? - pt.token_vocab_filepath = VOCAB_FILEPATH - pt.tokens_delimiter = "" - pt.load_token_ids_from_vocab = False - with tf.io.gfile.GFile(VOCAB_FILEPATH, mode='r') as fh: - lines = fh.readlines() - for i, line in enumerate(lines): - if line == "\n": - pt.target_unk_id = i - elif line == "\n": - pt.target_sos_id = i - elif line == "\n": - pt.target_eos_id = i - pt.vocab_size = len(lines) - assert pt.vocab_size == 32 - assert pt.target_unk_id == 0 - assert pt.target_sos_id == 1, pt.target_sos_id - assert pt.target_eos_id == 2, pt.target_eos_id - - return pt - -@model_registry.RegisterSingleTaskModel -class Librispeech960Base(base_model_params.SingleTaskModelParams): - """Base parameters for Librispeech 960 hour task.""" - - def _CommonInputParams(self, is_eval): - """Input generator params for Librispeech.""" - p = input_generator.AsrInput.Params() - - # Insert path to the base directory where the data are stored here. - # Generated using scripts in lingvo/tasks/asr/tools. - p.file_datasource = datasource.PrefixedDataSource.Params() - p.file_datasource.file_type = 'tfrecord' - p.file_datasource.file_pattern_prefix = 'gs://the-peoples-speech-west-europe/Librispeech' - - p.frame_size = 80 - # Interesting. First I've heard of this. - p.append_eos_frame = False - - if tf.flags.FLAGS.tpu: - p.pad_to_max_seq_length = True - else: - p.pad_to_max_seq_length = False - p.file_random_seed = 0 - p.file_buffer_size = 10000 - p.file_parallelism = 16 - - if is_eval: - p.source_max_length = 3600 - p.bucket_upper_bound = [639, 1062, 1275, 1377, 1449, 1506, 1563, 3600] - else: - # So it looks like - p.source_max_length = 1710 - p.bucket_upper_bound = [639, 1062, 1275, 1377, 1449, 1506, 1563, 1710] - - p.bucket_batch_limit = [48] * 8 - - - p.tokenizer = intialize_vocab_file_tokenizer_params() - - return p - - def SetBucketSizes(self, params, bucket_upper_bound, bucket_batch_limit): - """Sets bucket sizes for batches in params.""" - params.bucket_upper_bound = bucket_upper_bound - params.bucket_batch_limit = bucket_batch_limit - return params - - def Train(self): - p = self._CommonInputParams(is_eval=False) - p.file_datasource.file_pattern = 'train/train.tfrecords-*' - p.num_samples = 281241 - return p - - def Dev(self): - p = self._CommonInputParams(is_eval=True) - p.file_datasource.file_pattern = ( - 'devtest/dev-clean.tfrecords-00000-of-00001') - p.num_samples = 2703 - p.target_max_length = 516 - # p.source_max_length = 3600 - # p.bucket_upper_bound = [3600] - # p.bucket_batch_limit = [1] - return p - - def Devother(self): - p = self._CommonInputParams(is_eval=True) - p.file_datasource.file_pattern = ( - 'devtest/dev-other.tfrecords-00000-of-00001') - p.num_samples = 2864 - return p - - def Test(self): - p = self._CommonInputParams(is_eval=True) - p.file_datasource.file_pattern = ( - 'devtest/test-clean.tfrecords-00000-of-00001') - p.num_samples = 2620 - return p - - def Testother(self): - p = self._CommonInputParams(is_eval=True) - p.file_datasource.file_pattern = ( - 'devtest/test-other.tfrecords-00000-of-00001') - p.num_samples = 2939 - return p - - def Task(self): - p = ctc_model.CTCModel.Params() - p.name = 'librispeech' - - with tf.io.gfile.GFile(VOCAB_FILEPATH, mode='r') as fh: - lines = fh.readlines() - assert lines[-1] == "\n" - p.vocab_size = len(lines) - p.blank_index = 31 - - assert p.vocab_size == 32 - assert p.blank_index == 31 - - ep = p.encoder_v2 - ep.use_stacking_subsampler = True - - # No default encoder params in this class. - - tp = p.train - tp.learning_rate = 1e-4 - tp.lr_schedule = schedule.ContinuousSchedule.Params().Set( - start_step=25_000, half_life_steps=5_000, min=1e-6) - tp.scale_gradients = False - tp.l2_regularizer_weight = None - - # A value of 0 will force each eval and dev task to iterate over - # the entire dataset. - p.eval.samples_per_summary = 0 - p.eval.decoder_samples_per_summary = 0 - - return p - - # From what I can tell, this is used only by ExecutorTpu. So perhaps - # I could change it such that it runs only on the Train set... Right? - def ProgramSchedule(self): - return program.SimpleProgramScheduleForTask( - train_dataset_name='Train', - train_steps_per_loop=100, - # I want to compute WER... - eval_dataset_names=['Dev'], - eval_steps_per_loop=1, - decode_steps_per_loop=1,) - - -# @model_registry.RegisterSingleTaskModel -# class TpuDecoderLibrispeech960Base(Librispeech960Base): - -# SAMPLES_PER_SECOND = 16_000 - -# def _InferenceInputParams(self): -# """ -# Reads in a raw waveform, where samples are float32 and in the range[-1,1] -# """ -# p = input_generator.RawAsrInputIntegerUttIds.Params() - -# p.file_datasource = datasource.PrefixedDataSource.Params() -# p.file_datasource.file_type = 'tfrecord' -# p.file_datasource.file_pattern_prefix = 'REPLACE_ME' -# p.file_datasource.file_pattern = '*.tfrecord' - -# p.frame_size = 1 -# p.append_eos_frame = False - -# p.pad_to_max_seq_length = True -# p.file_random_seed = 0 -# p.file_buffer_size = 1 -# # Set this to whatever -# p.file_parallelism = 1 - -# # 15 seconds is our maximum utterance size -# max_sample_length = 15 * type(self).SAMPLES_PER_SECOND -# p.source_max_length = max_sample_length -# p.bucket_upper_bound = [p.source_max_length] - -# p.bucket_batch_limit = [1] - -# return p - -# def Dev(self): -# return self._InferenceInputParams() - -# def Task(self): -# p = super().Task() -# # This is copied from audio_lib.py. Ideally these configurations -# # would be split off into a function, but I want to minimize merge -# # conflicts with lingvo for now. Of course, a merge conflict would -# # indicate that parameters had changed, so my choice is clearly -# # wrong, but oh well. -# p.frontend = asr_frontend.MelAsrFrontend.Params() -# pf = p.frontend -# pf.sample_rate = float(type(self).SAMPLES_PER_SECOND) -# pf.frame_size_ms = 25. -# pf.frame_step_ms = 10. -# pf.num_bins = 80 -# pf.lower_edge_hertz = 125. -# pf.upper_edge_hertz = 7600. -# pf.preemph = 0.97 -# pf.noise_scale = 0. -# pf.pad_end = False - -# p.inference_compute_only_log_softmax = True -# return p - -# def ProgramSchedule(self): -# # This value must be parameterized -# number_of_inputs = FLAGS.ctc_inference_model_num_samples -# batch_size = 8 -# import math -# decode_steps_per_loop = math.ceil(number_of_inputs / batch_size) -# return program.DecodeProgramSchedule( -# eval_dataset_names=['Dev'], -# decode_steps_per_loop=decode_steps_per_loop, # 1, -# # I may want to reevaluate this -# experimental_decoder=True) - - -@model_registry.RegisterSingleTaskModel -class Librispeech960Grapheme(Librispeech960Base): - - GRAPHEME_TARGET_SEQUENCE_LENGTH = 620 - GRAPHEME_VOCAB_SIZE = 76 - BLANK_IDX = 73 - - # This seems bad. Does this undo my VocabFileTokenizer??? - def InitializeTokenizer(self, params): - """Initializes a grapheme tokenizer.""" - params.tokenizer = tokenizers.AsciiTokenizer.Params() - tokp = params.tokenizer - tokp.vocab_size = self.GRAPHEME_VOCAB_SIZE - tokp.append_eos = False - tokp.target_unk_id = 0 - tokp.target_sos_id = 1 - tokp.target_eos_id = 2 - - params.target_max_length = self.GRAPHEME_TARGET_SEQUENCE_LENGTH - return params - - def Train(self): - p = super().Train() - return self.InitializeTokenizer(params=p) - - def Dev(self): - p = super().Dev() - return self.InitializeTokenizer(params=p) - - def Devother(self): - p = super().Devother() - return self.InitializeTokenizer(params=p) - - def Test(self): - p = super().Test() - return self.InitializeTokenizer(params=p) - - def Testother(self): - p = super().Testother() - return self.InitializeTokenizer(params=p) - - def Task(self): - p = super().Task() - p.vocab_size = self.GRAPHEME_VOCAB_SIZE - p.blank_index = self.BLANK_IDX - - return p - -@model_registry.RegisterSingleTaskModel -class Grphm_DO_SpecAug_StackingSubSampler(Librispeech960Base): - - def Task(self): - p = super().Task() - - ep = p.encoder_v2 - ep.use_specaugment = True - ep.use_conv_subsampler = False - ep.use_stacking_subsampler = True - - ep.stacking_subsampler.stacking.left_context = 1 - ep.stacking_subsampler.stacking.right_context = 1 - ep.stacking_subsampler.stacking.stride = ( - ep.stacking_subsampler.stacking.left_context + - 1 + - ep.stacking_subsampler.stacking.right_context - ) - - elp = p.encoder_v2.lstm_block - elp.dropout.keep_prob = 0.8 - elp.lstm_cell_size = 1024 - elp.num_lstm_layers = 6 - elp.lstm_type = 'fwd' - - return p - - -@model_registry.RegisterSingleTaskModel -class Grphm_DO_SpecAug_ConvStk_6x512Bidi(Librispeech960Grapheme): - - def Task(self): - p = super().Task() - - ep = p.encoder_v2 - ep.use_specaugment = True - ep.use_conv_subsampler = True - ep.use_stacking_subsampler = False - - elp = p.encoder_v2.lstm_block - elp.dropout.keep_prob = 0.8 - elp.lstm_cell_size = 512 - elp.num_lstm_layers = 6 - elp.lstm_type = 'bidi' - - ep.stacking_subsampler = None - ecp = ep.conv_subsampler - ecp.input_shape = [None, None, 80, 1] - return p - - -@model_registry.RegisterSingleTaskModel -class Grphm_DO_SpecAug_ConvStk_6x512Bidi_40batchsize(Librispeech960Grapheme): - - def Train(self): - p = super().Train() - # OOM with 48 - p.bucket_batch_limit = [40] * 8 - return p - - def Task(self): - p = super().Task() - - ep = p.encoder_v2 - ep.use_specaugment = True - ep.use_conv_subsampler = True - ep.use_stacking_subsampler = False - - elp = p.encoder_v2.lstm_block - elp.dropout.keep_prob = 0.8 - elp.lstm_cell_size = 512 - elp.num_lstm_layers = 6 - elp.lstm_type = 'bidi' - - ep.stacking_subsampler = None - ecp = ep.conv_subsampler - ecp.input_shape = [None, None, 80, 1] - return p - -@model_registry.RegisterSingleTaskModel -class TpuDecoderGrphm_DO_SpecAug_StackingSubSampler(Grphm_DO_SpecAug_StackingSubSampler): - - SAMPLES_PER_SECOND = 16_000 - - def _InferenceInputParams(self): - """ - Reads in a raw waveform, where samples are float32 and in the range[-1,1] - """ - p = input_generator.RawAsrInputIntegerUttIds.Params() - - p.file_datasource = datasource.PrefixedDataSource.Params() - p.file_datasource.file_type = 'tfrecord' - p.file_datasource.file_pattern_prefix = 'REPLACE_ME' - p.file_datasource.file_pattern = '*.tfrecord' - - p.frame_size = 1 - p.append_eos_frame = False - - p.pad_to_max_seq_length = True - p.file_random_seed = 0 - p.file_buffer_size = 1 - # Set this to whatever - p.file_parallelism = 1 - - # 15 seconds is our maximum utterance size - max_sample_length = 15 * type(self).SAMPLES_PER_SECOND - # TODOL: change to max_sample_length - p.source_max_length = max_sample_length - p.bucket_upper_bound = [p.source_max_length] - - p.bucket_batch_limit = [FLAGS.ctc_inference_model_batch_size] - - # assert p.num_samples == FLAGS.ctc_inference_model_num_samples - - p.tokenizer = intialize_vocab_file_tokenizer_params() - - return p - - def Dev(self): - return self._InferenceInputParams() - - def Task(self): - p = super().Task() - # This is copied from audio_lib.py. Ideally these configurations - # would be split off into a function, but I want to minimize merge - # conflicts with lingvo for now. Of course, a merge conflict would - # indicate that parameters had changed, so my choice is clearly - # wrong, but oh well. - p.frontend = asr_frontend.MelAsrFrontend.Params() - pf = p.frontend - pf.sample_rate = float(type(self).SAMPLES_PER_SECOND) - pf.frame_size_ms = 25. - pf.frame_step_ms = 10. - pf.num_bins = 80 - pf.lower_edge_hertz = 125. - pf.upper_edge_hertz = 7600. - pf.preemph = 0.97 - pf.noise_scale = 0. - pf.pad_end = False - - p.inference_compute_only_log_softmax = True - - # TODO: Consider doing this - # ep = p.encoder_v2 - # ep.use_specaugment = False - return p - - def ProgramSchedule(self): - # TODO: Generalize this - number_of_inputs = FLAGS.ctc_inference_model_num_samples - assert number_of_inputs != 0 - assert FLAGS.ctc_inference_model_batch_size != 0 - batch_size = FLAGS.ctc_inference_model_batch_size * FLAGS.ctc_inference_model_number_of_tpus - import math - decode_steps_per_loop = math.ceil(number_of_inputs / batch_size) - return program.DecodeProgramSchedule( - eval_dataset_names=['Dev'], - decode_steps_per_loop=decode_steps_per_loop, # 1, - # I may want to reevaluate this - experimental_decoder=False) diff --git a/lingvo/tasks/asr/params/params.py b/lingvo/tasks/asr/params/params.py deleted file mode 100644 index 3cad45e6..00000000 --- a/lingvo/tasks/asr/params/params.py +++ /dev/null @@ -1,22 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""ASR model parameters.""" - -# Import ModelParams to ensure that they are added to the global registry. -# LAS model -import lingvo.tasks.asr.params.librispeech # pylint: disable=unused-import -import lingvo.tasks.asr.params.librispeech_ctc # pylint: disable=unused-import -import lingvo.tasks.asr.params.peoplesspeech_ctc # pylint: disable=unused-import \ No newline at end of file diff --git a/lingvo/tasks/asr/params/peoplesspeech_ctc.py b/lingvo/tasks/asr/params/peoplesspeech_ctc.py deleted file mode 100644 index 5c301ce7..00000000 --- a/lingvo/tasks/asr/params/peoplesspeech_ctc.py +++ /dev/null @@ -1,192 +0,0 @@ -import os - -from lingvo import model_registry -from lingvo.core import base_model_params -from lingvo.core import datasource -from lingvo.core import program -from lingvo.core import py_utils -from lingvo.core import schedule -from lingvo.core import tokenizers -from lingvo.tasks.asr import input_generator -from lingvo.tasks.asr import ctc_model - -@model_registry.RegisterSingleTaskModel -class Peoplesspeech100Base(base_model_params.SingleTaskModelParams): - """Base parameters for Peoplesspeech 100k hour task.""" - - def _CommonInputParams(self, is_eval): - """Input generator params for Peoplesspeech.""" - p = input_generator.AsrInput.Params() - - # Insert path to the base directory where the data are stored here. - # Generated using scripts in lingvo/tasks/asr/tools. - p.file_datasource = datasource.PrefixedDataSource.Params() - p.file_datasource.file_type = 'tfrecord' - p.file_datasource.file_pattern_prefix = 'gs://the-peoples-speech-west-europe/PeoplesSpeech/v0.7.1/' - - p.frame_size = 80 - # Interesting. First I've heard of this. - p.append_eos_frame = False - - p.pad_to_max_seq_length = True - p.file_random_seed = 0 - p.file_buffer_size = 10000 - # N1 standard 2 has only 2 vCPUs, so we may want a larger machine. - # https://cloud.google.com/compute/docs/machine-types#n1_standard_machine_types - p.file_parallelism = 16 - - if is_eval: - p.source_max_length = 3600 - p.bucket_upper_bound = [639, 1062, 1275, 1377, 1449, 1506, 1563, 3600] - else: - # So it looks like - p.source_max_length = 1710 - p.bucket_upper_bound = [639, 1062, 1275, 1377, 1449, 1506, 1563, 1710] - - p.bucket_batch_limit = [48] * 8 - - return p - - def SetBucketSizes(self, params, bucket_upper_bound, bucket_batch_limit): - """Sets bucket sizes for batches in params.""" - params.bucket_upper_bound = bucket_upper_bound - params.bucket_batch_limit = bucket_batch_limit - return params - - def Train(self): - p = self._CommonInputParams(is_eval=False) - p.file_datasource.file_pattern = 'train/train.tfrecords-*' - p.num_samples = 2292260 - return p - - def Dev(self): - p = self._CommonInputParams(is_eval=True) - p.file_datasource.file_pattern = ( - 'devtest/dev.tfrecords-00000-of-00001') - p.num_samples = 3000 - return p - - def Test(self): - p = self._CommonInputParams(is_eval=True) - p.file_datasource.file_pattern = ( - 'devtest/test.tfrecords-00000-of-00001') - p.num_samples = 3000 - return p - - def Task(self): - p = ctc_model.CTCModel.Params() - p.name = 'peoplesspeech' - - # No default encoder params in this class. - - tp = p.train - tp.learning_rate = 1e-4 - tp.lr_schedule = schedule.ContinuousSchedule.Params().Set( - start_step=25_000, half_life_steps=5_000, min=1e-6) - tp.scale_gradients = False - tp.l2_regularizer_weight = None - - # Setting p.eval.samples_per_summary to a large value ensures that dev, - # devother, test, testother are evaluated completely (since num_samples for - # each of these sets is less than 5000), while train summaries will be - # computed on 5000 examples. - p.eval.samples_per_summary = 2700 - p.eval.decoder_samples_per_summary = 2700 - - return p - - def ProgramSchedule(self): - return program.SimpleProgramScheduleForTask( - train_dataset_name='Train', - train_steps_per_loop=500, - eval_dataset_names=['Dev', 'Train'], - eval_steps_per_loop=50, - decode_steps_per_loop=0) - - -@model_registry.RegisterSingleTaskModel -class Peoplesspeech100Grapheme(Peoplesspeech100Base): - - GRAPHEME_TARGET_SEQUENCE_LENGTH = 620 - GRAPHEME_VOCAB_SIZE = 76 - BLANK_IDX = 73 - - def InitializeTokenizer(self, params): - """Initializes a grapheme tokenizer.""" - params.tokenizer = tokenizers.AsciiTokenizer.Params() - tokp = params.tokenizer - tokp.vocab_size = self.GRAPHEME_VOCAB_SIZE - tokp.append_eos = False - tokp.target_unk_id = 0 - tokp.target_sos_id = 1 - tokp.target_eos_id = 2 - - params.target_max_length = self.GRAPHEME_TARGET_SEQUENCE_LENGTH - return params - - def Train(self): - p = super().Train() - return self.InitializeTokenizer(params=p) - - def Dev(self): - p = super().Dev() - return self.InitializeTokenizer(params=p) - - def Test(self): - p = super().Test() - return self.InitializeTokenizer(params=p) - - def Task(self): - p = super().Task() - p.vocab_size = self.GRAPHEME_VOCAB_SIZE - p.blank_index = self.BLANK_IDX - - return p - - -@model_registry.RegisterSingleTaskModel -class Grphm_DO_SpecAug_ConvStk_6x512Bidi(Peoplesspeech100Grapheme): - - def Task(self): - p = super().Task() - - ep = p.encoder_v2 - ep.use_specaugment = True - - elp = p.encoder_v2.lstm_block - elp.dropout.keep_prob = 0.8 - elp.lstm_cell_size = 512 - elp.num_lstm_layers = 6 - elp.lstm_type = 'bidi' - - ep.stacking_subsampler = None - ecp = ep.conv_subsampler - ecp.input_shape = [None, None, 80, 1] - return p - - -@model_registry.RegisterSingleTaskModel -class Grphm_DO_SpecAug_ConvStk_6x512Bidi_40batchsize(Peoplesspeech100Grapheme): - - def Train(self): - p = super().Train() - # OOM with 48 - p.bucket_batch_limit = [40] * 8 - return p - - def Task(self): - p = super().Task() - - ep = p.encoder_v2 - ep.use_specaugment = True - - elp = p.encoder_v2.lstm_block - elp.dropout.keep_prob = 0.8 - elp.lstm_cell_size = 512 - elp.num_lstm_layers = 6 - elp.lstm_type = 'bidi' - - ep.stacking_subsampler = None - ecp = ep.conv_subsampler - ecp.input_shape = [None, None, 80, 1] - return p diff --git a/lingvo/tasks/asr/rnnt_model.py b/lingvo/tasks/asr/rnnt_model.py deleted file mode 100644 index 0cbf96ca..00000000 --- a/lingvo/tasks/asr/rnnt_model.py +++ /dev/null @@ -1,63 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""RNN-T model.""" - -import collections -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import base_model -from lingvo.core import metrics -from lingvo.core import py_utils -from lingvo.core import schedule -from lingvo.tasks.asr import decoder -from lingvo.tasks.asr import decoder_utils -from lingvo.tasks.asr import encoder -from lingvo.tasks.asr import frontend as asr_frontend -from lingvo.tools import audio_lib - -class RNNTModel(base_model.BaseTask): - - @classmethod - def Params(cls): - p = super().Params() - p.encoder = encoder.AsrEncoder.Params() - p.prediction = rnnt_prediction.Prediction.Params() - p.joint = rnnt_joint.RNNTJoint.Params() - p.Define( - 'frontend', None, - 'ASR frontend to extract features from input. Defaults to no frontend ' - 'which means that features are taken directly from the input.') - - p.Define('include_auxiliary_metrics', True, - 'In addition to simple WER, also computes oracle WER, SACC, TER, etc. ' - 'Turning off this option will speed up the decoder job.') - - tp = p.train - tp.lr_schedule = ( - schedule.PiecewiseConstantSchedule.Params().Set( - boundaries=[350_000, 500_000, 600_000], - values=[1.0, 0.1, 0.01, 0.001])) - tp.vn_start_step = 20_000 - tp.vn_std = 0.075 - tp.l2_regularizer_weight = 1e-6 - # Completely redundant with previous initial 1.0 value... which - # has higher precendence? - tp.learning_rate = 0.001 - tp.clip_gradient_norm_to_value = 1.0 - tp.grad_norm_to_clip_to_zero = 100.0 - # What does this mean? - tp.tpu_steps_per_loop = 20 - - diff --git a/lingvo/tasks/asr/rnnt_prediction.py b/lingvo/tasks/asr/rnnt_prediction.py deleted file mode 100644 index b7590ee7..00000000 --- a/lingvo/tasks/asr/rnnt_prediction.py +++ /dev/null @@ -1,155 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Prediction network for the RNNT model.""" -from lingvo.core import base_layer -from lingvo.core import layers - - -class Prediction(base_layer.BaseLayer): - - @classmethod - def Params(cls): - p = super().Params() - # This is input to an embedding layer - p.Define('input_shape', [None, None], 'Shape of the input. This should be a TensorShape with rank 2.') - p.Define('lstm_cell_size', 2048, 'LSTM cell size for the RNN layer.') - p.Define('num_lstm_layers', 8, 'Number of rnn layers to create.') - p.Define('project_lstm_output', True, - 'Include projection layer after each encoder LSTM layer.') - # How to use? - StackedFRNNLayerByLayer.Params() - p.Define('lstm_tpl', rnn_cell.LSTMCellSimple.Params(), - 'Configs template for the RNN layer.') - p.Define('proj_tpl', layers.ProjectionLayer.Params(), - 'Configs template for the projection layer.') - p.Define('stacking_layer_tpl', layers.StackingOverTime.Params(), - 'Configs template for the stacking layer over time.') - p.Define('embedding_tpl', layers.EmbeddingLayer.Params(), - 'Configs template for the embedding layer.') - p.Define( - 'layer_index_before_stacking', -1, - 'The (0-based) index of the lstm layer after which the stacking layer ' - 'will be inserted. Negative value means no stacking layer will be ' - 'used.') - - return p - - @base_layer.initializer - def __init__(self, params): - super().__init__(params) - p = self.params - name = p.name - - - (self._first_lstm_input_dim, - self._first_lstm_input_dim_pad) = self.FirstLstmLayerInputDimAndPadding( - p.embedding_tpl.embedding_dim, pad_to_multiple=16) - - - with tf.variable_scope(name): - params_rnn_layers = [] - params_proj_layers = [] - - embed_p = p.embedding_tpl.Copy() - embed_p.name = 'embed' - self.CreateChild('embed', embed_p) - - output_dim = self._first_lstm_input_dim - for i in range(p.num_lstm_layers): - input_dim = output_dim - lstm_p = p.lstm_tpl.Copy() - lstm_p.name = f"lstm_L{i}" - lstm_p.num_input_nodes = input_dim - lstm_p.num_output_nodes = p.lstm_cell_size - params_rnn_layers.append(lstm_p) - - if p.project_lstm_output and (i < p.num_lstm_layers - 1): - proj_p = p.proj_tpl.Copy() - proj_p.input_dim = p.lstm_cell_size - proj_p.output_dim = p.lstm_cell_size - proj_p.name = f'proj_L{i}' - params_proj_layers.append(proj_p) - - if p.layer_index_before_stacking == i: - stacking_layer = p.stacking_layer_tpl.Copy() - stacking_layer.name = f'stacking_L{i}' - self.CreateChild('stacking', stacking_layer) - stacking_window_len = ( - p.stacking_layer_tpl.left_context + 1 + - p.stacking_layer_tpl.right_context) - output_dim *= stacking_window_len - - self.CreateChildren('rnn', params_rnn_layers) - self.CreateChildren('proj', params_proj_layers) - - @property - def _use_functional(self): - return True - - @property - def input_shape(self): - return self.params.input_shape - - @property - def supports_streaming(self): - return True - - def FirstLstmLayerInputDimAndPadding(self, - lstm_input_shape, - pad_to_multiple=16): - # Makes sure the lstm input dims is multiple of 16 (alignment - # requirement from FRNN). - first_lstm_input_dim_unpadded = lstm_input_shape[2] - - if self._use_functional and (first_lstm_input_dim_unpadded % pad_to_multiple - != 0): - first_lstm_input_dim = int( - (first_lstm_input_dim_unpadded + pad_to_multiple - 1) / - pad_to_multiple) * pad_to_multiple - else: - first_lstm_input_dim = first_lstm_input_dim_unpadded - - first_lstm_input_dim_padding = ( - first_lstm_input_dim - first_lstm_input_dim_unpadded) - return first_lstm_input_dim, first_lstm_input_dim_padding - - def FProp(self, theta, batch, state0=None): - """Encodes source as represented by 'inputs' and 'paddings'. - - Args: - theta: A NestedMap object containing weights' values of this - layer and its children layers. - batch: A NestedMap with fields: - - - tokens - The inputs tensor. It is expected to be of shape [time, batch]. - - paddings - The paddings tensor. It is expected to be of shape [time, batch]. - state0: Recurrent input state. - - Returns: - A NestedMap containing - - - 'encoded': a feature tensor of shape [batch, time, depth] - - 'padding': a 0/1 tensor of shape [batch, time] - - 'state': the updated recurrent state - - '${layer_type}_${layer_index}': The per-layer encoder output. Each one - is a NestedMap containing 'encoded' and 'padding' similar to regular - final outputs, except that 'encoded' from conv or conv_lstm layers are - of shape [time, batch, depth, channels]. - """ - p = self.params - if len(batch) == 0: - - if state0 is None: - diff --git a/lingvo/tasks/asr/tools/BUILD b/lingvo/tasks/asr/tools/BUILD deleted file mode 100644 index 3abb1023..00000000 --- a/lingvo/tasks/asr/tools/BUILD +++ /dev/null @@ -1,43 +0,0 @@ -# tools for ASR task, including WER computation and breakdown analytics - -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -py_library( - name = "simple_wer", - srcs = ["simple_wer.py"], - srcs_version = "PY3", -) - -py_test( - name = "simple_wer_test", - srcs = ["simple_wer_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":simple_wer", - "//lingvo:compat", - "//lingvo/core:test_utils", - ], -) - -py_library( - name = "simple_wer_v2", - srcs = ["simple_wer_v2.py"], - srcs_version = "PY3", -) - -py_test( - name = "simple_wer_v2_test", - srcs = ["simple_wer_v2_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":simple_wer_v2", - "//lingvo:compat", - "//lingvo/core:test_utils", - ], -) diff --git a/lingvo/tasks/asr/tools/__init__.py b/lingvo/tasks/asr/tools/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/lingvo/tasks/asr/tools/librispeech.01.download_train.sh b/lingvo/tasks/asr/tools/librispeech.01.download_train.sh deleted file mode 100755 index f1ff4805..00000000 --- a/lingvo/tasks/asr/tools/librispeech.01.download_train.sh +++ /dev/null @@ -1,28 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. librispeech_lib.sh - -mkdir -p "${ROOT}/raw" - -# This will take about half an hour on a good connection. -echo " -${SOURCE}/train-clean-100.tar.gz -${SOURCE}/train-clean-360.tar.gz -${SOURCE}/train-other-500.tar.gz" \ - | aria2c -x16 -s16 --dir="${ROOT}/raw" -i - diff --git a/lingvo/tasks/asr/tools/librispeech.02.download_devtest.sh b/lingvo/tasks/asr/tools/librispeech.02.download_devtest.sh deleted file mode 100755 index 83df5103..00000000 --- a/lingvo/tasks/asr/tools/librispeech.02.download_devtest.sh +++ /dev/null @@ -1,28 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. librispeech_lib.sh - -mkdir -p "${ROOT}/raw" - -echo " -${SOURCE}/dev-clean.tar.gz -${SOURCE}/dev-other.tar.gz -${SOURCE}/test-clean.tar.gz -${SOURCE}/test-other.tar.gz" \ - | aria2c -x16 -s16 --dir="${ROOT}/raw" -i - diff --git a/lingvo/tasks/asr/tools/librispeech.03.parameterize_train.sh b/lingvo/tasks/asr/tools/librispeech.03.parameterize_train.sh deleted file mode 100755 index 0fe7a1a9..00000000 --- a/lingvo/tasks/asr/tools/librispeech.03.parameterize_train.sh +++ /dev/null @@ -1,98 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. librispeech_lib.sh - -# To save space, we don't unpack to intermediate files. The first pass collects -# all transcription files from the tarball. The second pass unpacks the audio, -# decompresses it, and encodes MFCC frames in memory, then writes a tf.Example -# with the accompanying transcription from the first pass. - -# This takes about 10 minutes per set. -for subset in train-clean-100 train-clean-360 train-other-500; do - echo "=== First pass, collecting transcripts: ${subset}" - qsub_run \ - $(git rev-parse --show-toplevel)/bazel-bin/lingvo/tools/create_asr_features \ - --logtostderr \ - --input_tarball="${ROOT}/raw/${subset}.tar.gz" --dump_transcripts \ - --transcripts_filepath="${ROOT}/train/${subset}.txt" -done - -# We are allocating as follows: -# num utts num_shards subset -# 28539 10 train-clean-100 -# 104014 40 train-clean-360 -# 148688 50 train-other-500 -# 281241 100 Total -# -# We expect a total of, say, 100GB, so we want 100 shards to get into the 1GB -# range. We use 10 processors, each alloted a range of 10 output shards. - -# Second pass: Create tf.Examples. It takes about 90 minutes. - -rm -f FAILED -subset=train-clean-100 -echo "=== Second pass, parameterization: ${subset}" -for subshard in $(seq 0 9); do - set -x - # nice -n 20 python3 -m lingvo.tools.create_asr_features \ - qsub_run \ - $(git rev-parse --show-toplevel)/bazel-bin/lingvo/tools/create_asr_features \ - --logtostderr \ - --input_tarball="${ROOT}/raw/${subset}.tar.gz" --generate_tfrecords \ - --shard_id="${subshard}" --num_shards=10 --num_output_shards=100 \ - --output_range_begin="${subshard}" --output_range_end="$((subshard + 1))" \ - --output_template="${ROOT}/train/train.tfrecords-%5.5d-of-%5.5d" || touch FAILED & - set +x -done -wait -! [ -f FAILED ] - -subset=train-clean-360 -echo "=== Second pass, parameterization: ${subset}" -for subshard in $(seq 0 9); do - set -x - qsub_run \ - $(git rev-parse --show-toplevel)/bazel-bin/lingvo/tools/create_asr_features \ - --logtostderr \ - --input_tarball="${ROOT}/raw/${subset}.tar.gz" --generate_tfrecords \ - --shard_id="${subshard}" --num_shards=10 --num_output_shards=100 \ - --output_range_begin="$((10 + 4 * subshard))" \ - --output_range_end="$((10 + 4 * subshard + 4))" \ - --output_template="${ROOT}/train/train.tfrecords-%5.5d-of-%5.5d" || touch FAILED & - set +x -done -wait -! [ -f FAILED ] - -subset=train-other-500 -echo "=== Second pass, parameterization: ${subset}" -for subshard in $(seq 0 9); do - set -x - qsub_run \ - $(git rev-parse --show-toplevel)/bazel-bin/lingvo/tools/create_asr_features \ - --logtostderr \ - --input_tarball="${ROOT}/raw/${subset}.tar.gz" --generate_tfrecords \ - --shard_id="${subshard}" --num_shards=10 --num_output_shards=100 \ - --output_range_begin="$((50 + 5 * subshard))" \ - --output_range_end="$((50 + 5 * subshard + 5))" \ - --output_template="${ROOT}/train/train.tfrecords-%5.5d-of-%5.5d" || touch FAILED & - set +x -done -wait -! [ -f FAILED ] diff --git a/lingvo/tasks/asr/tools/librispeech.04.parameterize_devtest.sh b/lingvo/tasks/asr/tools/librispeech.04.parameterize_devtest.sh deleted file mode 100755 index 0a90ed61..00000000 --- a/lingvo/tasks/asr/tools/librispeech.04.parameterize_devtest.sh +++ /dev/null @@ -1,32 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. librispeech_lib.sh - -for subset in {dev,test}-{clean,other}; do - set -x - # python3 -m lingvo.tools.create_asr_features \ - # --run_under=$(which ipdb3) - bazel_ run -c opt //lingvo/tools:create_asr_features -- \ - --generate_tfrecords \ - --input_tarball="${ROOT}/raw/${subset}.tar.gz" \ - --shard_id=0 --num_shards=1 --num_output_shards=1 \ - --output_range_begin=0 --output_range_end=1 \ - --output_template="${ROOT}/devtest/${subset}.tfrecords-%5.5d-of-%5.5d" - set +x -done diff --git a/lingvo/tasks/asr/tools/librispeech.05.train_ctc.sh b/lingvo/tasks/asr/tools/librispeech.05.train_ctc.sh deleted file mode 100644 index 1d833f50..00000000 --- a/lingvo/tasks/asr/tools/librispeech.05.train_ctc.sh +++ /dev/null @@ -1,17 +0,0 @@ -#!/bin/bash - -. ./librispeech_lib.sh - -set -euo pipefail - -cat < model_params.txt - -EOF - -bazel_ run //lingvo:trainer \ - --interactive \ - --mode=shell \ - --run_locally=cpu \ - --model= - --logdir=logs/ctc/ - --model_params_file_override=model_params.txt diff --git a/lingvo/tasks/asr/tools/librispeech_get_data.sh b/lingvo/tasks/asr/tools/librispeech_get_data.sh deleted file mode 100755 index 08bc6acd..00000000 --- a/lingvo/tasks/asr/tools/librispeech_get_data.sh +++ /dev/null @@ -1,22 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -./librispeech.01.download_train.sh -./librispeech.02.download_devtest.sh -./librispeech.03.parameterize_train.sh -./librispeech.04.parameterize_devtest.sh diff --git a/lingvo/tasks/asr/tools/librispeech_lib.sh b/lingvo/tasks/asr/tools/librispeech_lib.sh deleted file mode 100755 index 9c4ea92a..00000000 --- a/lingvo/tasks/asr/tools/librispeech_lib.sh +++ /dev/null @@ -1,55 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -# ROOT=/tmp/librispeech -ROOT=gs://the-peoples-speech-west-europe/Librispeech - -# From: -# http://www.openslr.org/12/ -SOURCE=http://www.openslr.org/resources/12 - -# If in China, use this mirror: -# http://cn-mirror.openslr.org/resources/12 - -# conda activate galvasr - -# My home directory in on an NFS mount, which bazel dislikes, so I need to do this. -bazel_() { - bazel --output_user_root=/home/ws15dgalvez/dgalvez-b02/.cache/bazel "$@" -} - -qsub_run() { - cat < qsub_script.sh -#!/bin/bash -#$ -cwd -#$ -j yes -#$ -l mem_free=5G,ram_free=5G,arch=*64* -#$ -v PATH - -. ./librispeech_lib.sh - -export CUDA_VISIBLE_DEVICES="" -source "$($CONDA_EXE info --base)/etc/profile.d/conda.sh" -conda activate galvasr - -$@ -EOF - -qsub qsub_script.sh -} - -# Never instantiate a cuda context. -export CUDA_VISIBLE_DEVICES="" diff --git a/lingvo/tasks/asr/tools/qsub_script.sh b/lingvo/tasks/asr/tools/qsub_script.sh deleted file mode 100644 index cf85f064..00000000 --- a/lingvo/tasks/asr/tools/qsub_script.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash -#$ -cwd -#$ -j yes -#$ -l mem_free=5G,ram_free=5G,arch=*64* -#$ -v PATH - -. ./librispeech_lib.sh - -export CUDA_VISIBLE_DEVICES="" -source "/export/bdc01/home2/dgalvez/miniconda3/etc/profile.d/conda.sh" -conda activate galvasr - -/export/b02/ws15dgalvez/galvASR2/third_party/lingvo/bazel-bin/lingvo/tools/create_asr_features --logtostderr --input_tarball=gs://the-peoples-speech-west-europe/Librispeech/raw/train-other-500.tar.gz --generate_tfrecords --shard_id=3 --num_shards=10 --num_output_shards=100 --output_range_begin=65 --output_range_end=70 --output_template=gs://the-peoples-speech-west-europe/Librispeech/train/train.tfrecords-%5.5d-of-%5.5d diff --git a/lingvo/tasks/asr/tools/run_model.sh b/lingvo/tasks/asr/tools/run_model.sh deleted file mode 100755 index 3dacec4a..00000000 --- a/lingvo/tasks/asr/tools/run_model.sh +++ /dev/null @@ -1,14 +0,0 @@ -set -eu - -. librispeech_lib.sh - -bazel_ run -c opt //lingvo:trainer -- \ - --model=asr.librispeech_ctc.Librispeech960Base \ - --run_locally=cpu \ - --logtostderr \ - --logdir=my_logs/ - - - # --gcp_project=the-peoples-speech \ - # --tpu_zone= \ - # --tpu= diff --git a/lingvo/tasks/asr/tools/simple_wer.py b/lingvo/tasks/asr/tools/simple_wer.py deleted file mode 100644 index d6835c04..00000000 --- a/lingvo/tasks/asr/tools/simple_wer.py +++ /dev/null @@ -1,340 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Stand-alone script to evalute the word error rate (WER) for ASR tasks. - -THIS SCRIPT IS NO LONGER SUPPORTED. PLEASE USE simple_wer_v2.py INSTEAD. - -Tensorflow and Lingvo are not required to run this script. - -Example of Usage:: - - python simple_wer.py file_hypothesis file_reference - python simple_wer.py file_hypothesis file_reference diagnosis_html - -where `file_hypothesis` is the file name for hypothesis text and -`file_reference` is the file name for reference text. -`diagnosis_html` (optional) is the html filename to diagnose the errors. - -Or you can use this file as a library, and call either of the following: - - - ``ComputeWER(hyp, ref)`` compute WER for one pair of hypothesis/reference - - ``AverageWERs(hyps, refs)`` average WER for a list of hypotheses/references - -Note to evaluate the ASR, we consider the following pre-processing: - - - change transcripts to lower-case - - remove punctuation: ``" , . ! ? ( ) [ ]`` - - remove extra empty spaces -""" - -import re -import sys - - -def ComputeEditDistanceMatrix(hs, rs): - """Compute edit distance between two list of strings. - - Args: - hs: the list of words in the hypothesis sentence - rs: the list of words in the reference sentence - - Returns: - Edit distance matrix (in the format of list of lists), where the first - index is the reference and the second index is the hypothesis. - """ - dr, dh = len(rs) + 1, len(hs) + 1 - dists = [[]] * dr - - # Initialization. - for i in range(dr): - dists[i] = [0] * dh - for j in range(dh): - if i == 0: - dists[0][j] = j - elif j == 0: - dists[i][0] = i - - # Do dynamic programming. - for i in range(1, dr): - for j in range(1, dh): - if rs[i - 1] == hs[j - 1]: - dists[i][j] = dists[i - 1][j - 1] - else: - tmp0 = dists[i - 1][j - 1] + 1 - tmp1 = dists[i][j - 1] + 1 - tmp2 = dists[i - 1][j] + 1 - dists[i][j] = min(tmp0, tmp1, tmp2) - - return dists - - -def PreprocessTxtBeforeWER(txt): - """Preprocess text before WER caculation.""" - - # Lowercase, remove \t and new line. - txt = re.sub(r'[\t\n]', ' ', txt.lower()) - - # Remove punctuation before space. - txt = re.sub(r'[,.\?!]+ ', ' ', txt) - - # Remove punctuation before end. - txt = re.sub(r'[,.\?!]+$', ' ', txt) - - # Remove punctuation after space. - txt = re.sub(r' [,.\?!]+', ' ', txt) - - # Remove quotes, [, ], ( and ). - txt = re.sub(r'["\(\)\[\]]', '', txt) - - # Remove extra space. - txt = re.sub(' +', ' ', txt.strip()) - - return txt - - -def _GenerateAlignedHtml(hyp, ref, err_type): - """Generate a html element to highlight the difference between hyp and ref. - - Args: - hyp: Hypothesis string. - ref: Reference string. - err_type: one of 'none', 'sub', 'del', 'ins'. - - Returns: - a html string where disagreements are highlighted. - - hyp highlighted in green, and marked with - - ref highlighted in yellow - - """ - - highlighted_html = '' - if err_type == 'none': - highlighted_html += '%s ' % hyp - - elif err_type == 'sub': - highlighted_html += """ - %s - %s """ % (hyp, ref) - - elif err_type == 'del': - highlighted_html += """ - %s """ % ( - ref) - - elif err_type == 'ins': - highlighted_html += """ - %s """ % ( - hyp) - - else: - raise ValueError('unknown err_type ' + err_type) - - return highlighted_html - - -def GenerateSummaryFromErrs(nref, errs): - """Generate strings to summarize word errors. - - Args: - nref: integer of total words in references - errs: dict of three types of errors. e.g. {'sub':10, 'ins': 15, 'del': 3} - - Returns: - Two strings: - - string summarizing total error, total word, WER, - - string breaking down three errors: deleting, insertion, substitute - """ - - total_error = sum(errs.values()) - str_sum = 'total error = %d, total word = %d, wer = %.2f%%' % ( - total_error, nref, total_error * 100.0 / nref) - - str_details = 'Error breakdown: del = %.2f%%, ins=%.2f%%, sub=%.2f%%' % ( - errs['del'] * 100.0 / nref, errs['ins'] * 100.0 / nref, - errs['sub'] * 100.0 / nref) - - return str_sum, str_details - - -def ComputeWER(hyp, ref, diagnosis=False): - """Computes WER for ASR by ignoring diff of punctuation, space, captions. - - Args: - hyp: Hypothesis string. - ref: Reference string. - diagnosis (optional): whether to generate diagnosis str (in html format) - - Returns: - A tuple of 3 elements: - - - dict of three types of errors. e.g. ``{'sub':0, 'ins': 0, 'del': 0}`` - - num of reference words, integer - - aligned html string for diagnois (empty if diagnosis = False) - """ - - hyp = PreprocessTxtBeforeWER(hyp) - ref = PreprocessTxtBeforeWER(ref) - - # Compute edit distance. - hs = hyp.split() - rs = ref.split() - distmat = ComputeEditDistanceMatrix(hs, rs) - - # Back trace, to distinguish different errors: insert, deletion, substitution. - ih, ir = len(hs), len(rs) - errs = {'sub': 0, 'ins': 0, 'del': 0} - aligned_html = '' - while ih > 0 or ir > 0: - err_type = '' - - # Distinguish error type by back tracking - if ir == 0: - err_type = 'ins' - elif ih == 0: - err_type = 'del' - else: - if hs[ih - 1] == rs[ir - 1]: # correct - err_type = 'none' - elif distmat[ir][ih] == distmat[ir - 1][ih - 1] + 1: # substitute - err_type = 'sub' - elif distmat[ir][ih] == distmat[ir - 1][ih] + 1: # deletion - err_type = 'del' - elif distmat[ir][ih] == distmat[ir][ih - 1] + 1: # insert - err_type = 'ins' - else: - raise ValueError('fail to parse edit distance matrix') - - # Generate aligned_html - if diagnosis: - if ih == 0 or not hs: - tmph = ' ' - else: - tmph = hs[ih - 1] - if ir == 0 or not rs: - tmpr = ' ' - else: - tmpr = rs[ir - 1] - aligned_html = _GenerateAlignedHtml(tmph, tmpr, err_type) + aligned_html - - # If no error, go to previous ref and hyp. - if err_type == 'none': - ih, ir = ih - 1, ir - 1 - continue - - # Update error. - errs[err_type] += 1 - - # Adjust position of ref and hyp. - if err_type == 'del': - ir = ir - 1 - elif err_type == 'ins': - ih = ih - 1 - else: # err_type == 'sub' - ih, ir = ih - 1, ir - 1 - - assert distmat[-1][-1] == sum(errs.values()) - - # Num of words. For empty ref we set num = 1. - nref = max(len(rs), 1) - - return errs, nref, aligned_html - - -def AverageWERs(hyps, refs, verbose=True, diagnosis=False): - """Computes average WER from a list of references/hypotheses. - - Args: - hyps: list of hypothesis strings. - refs: list of reference strings. - verbose: optional (default True) - diagnosis (optional): whether to generate list of diagnosis html - - Returns: - A tuple of 3 elements: - - - dict of three types of errors. e.g. ``{'sub':0, 'ins': 0, 'del': 0}`` - - num of reference words, integer - - list of aligned html string for diagnosis (empty if diagnosis = False) - - """ - totalw = 0 - total_errs = {'sub': 0, 'ins': 0, 'del': 0} - aligned_html_list = [] - - for hyp, ref in zip(hyps, refs): - errs_i, nref_i, diag_str = ComputeWER(hyp, ref, diagnosis) - if diagnosis: - aligned_html_list += [diag_str] - - totalw += nref_i - total_errs['sub'] += errs_i['sub'] - total_errs['ins'] += errs_i['ins'] - total_errs['del'] += errs_i['del'] - - if verbose: - str_summary, str_details = GenerateSummaryFromErrs(totalw, total_errs) - print(str_summary) - print(str_details) - - return total_errs, totalw, aligned_html_list - - -def main(argv): - - hyp = open(argv[1], 'r').read() - ref = open(argv[2], 'r').read() - if len(argv) == 4: - diagnosis = True - fn_output = argv[3] - else: - diagnosis = False - fn_output = None - - errs, nref, aligned_html = ComputeWER(hyp, ref, diagnosis) - str_summary, str_details = GenerateSummaryFromErrs(nref, errs) - print(str_summary) - print(str_details) - - if fn_output: - with open(fn_output, 'wt') as fp: - fp.write('') - fp.write('
%s
' % aligned_html) - fp.write('') - - -if __name__ == '__main__': - print('THIS SCRIPT IS NO LONGER SUPPORTED.' - 'PLEASE USE simple_wer_v2.py INSTEAD.') - - if len(sys.argv) < 3 or len(sys.argv) > 4: - print(""" -Example of Usage: - - python simple_wer.py file_hypothesis file_reference -or - python simple_wer.py file_hypothesis file_reference diagnosis_html - - where file_hypothesis is the file name for hypothesis text - file_reference is the file name for reference text. - diagnosis_html (optional) is the html filename to diagnose the errors. - -Or you can use this file as a library, and call either of the following - - ComputeWER(hyp, ref) to compute WER for one pair of hypothesis/reference - - AverageWERs(hyps, refs) to average WER for a list of hypotheses/references -""") - sys.exit(1) - - main(sys.argv) diff --git a/lingvo/tasks/asr/tools/simple_wer_test.py b/lingvo/tasks/asr/tools/simple_wer_test.py deleted file mode 100644 index 5edb605f..00000000 --- a/lingvo/tasks/asr/tools/simple_wer_test.py +++ /dev/null @@ -1,34 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for simple_wer.""" - -import lingvo.compat as tf -from lingvo.core import test_utils -from lingvo.tasks.asr.tools import simple_wer - - -class SimpleWerTest(test_utils.TestCase): - - def testWerIgnorePunctuation(self): - hyp = 'hello world. today is a good day, how are you' - ref = '(Hello world)! [Today] is a good day! How are you?' - errs, nw, _ = simple_wer.ComputeWER(hyp, ref) - self.assertEqual(sum(errs.values()), 0) - self.assertEqual(nw, 10) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/asr/tools/simple_wer_v2.py b/lingvo/tasks/asr/tools/simple_wer_v2.py deleted file mode 100644 index 369ef4ad..00000000 --- a/lingvo/tasks/asr/tools/simple_wer_v2.py +++ /dev/null @@ -1,437 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""The new version script to evalute the word error rate (WER) for ASR tasks. - -Tensorflow and Lingvo are not required to run this script. - -Example of Usage: - -a) `python simple_wer_v2.py file_hypothesis file_reference` -b) `python simple_wer_v2.py file_hypothesis file_reference file_keyphrases` - -where `file_hypothesis` is the filename for hypothesis text, -`file_reference` is the filename for reference text, and -`file_keyphrases` is the optional filename for important phrases -(one phrase per line). - -Note that the program will also generate a html to diagnose the errors, -and the html filename is `{$file_hypothesis}_diagnois.html`. - -Another way is to use this file as a stand-alone library, by calling class -SimpleWER with the following member functions: - -- AddHypRef(hyp, ref): Updates the evaluation for each (hyp,ref) pair. -- GetWER(): Computes word error rate (WER) for all the added hyp-ref pairs. -- GetSummaries(): Generates strings to summarize word and key phrase errors. -- GetKeyPhraseStats(): Measures stats for key phrases. - Stats include: - (1) Jaccard similarity: https://en.wikipedia.org/wiki/Jaccard_index. - (2) F1 score: https://en.wikipedia.org/wiki/Precision_and_recall. - -""" - -import re -import sys - - -def TxtPreprocess(txt): - """Preprocess text before WER caculation.""" - - # Lowercase, remove \t and new line. - txt = re.sub(r'[\t\n]', ' ', txt.lower()) - - # Remove punctuation before space. - txt = re.sub(r'[,.\?!]+ ', ' ', txt) - - # Remove punctuation before end. - txt = re.sub(r'[,.\?!]+$', ' ', txt) - - # Remove punctuation after space. - txt = re.sub(r' [,.\?!]+', ' ', txt) - - # Remove quotes, [, ], ( and ). - txt = re.sub(r'["\(\)\[\]]', '', txt) - - # Remove extra space. - txt = re.sub(' +', ' ', txt.strip()) - - return txt - - -def RemoveCommentTxtPreprocess(txt): - """Preprocess text and remove comments in the brancket, such as [comments].""" - - # Remove comments surrounded by box brackets: - txt = re.sub(r'\[\w+\]', '', txt) - - return TxtPreprocess(txt) - - -def HighlightAlignedHtml(hyp, ref, err_type): - """Generate a html element to highlight the difference between hyp and ref. - - Args: - hyp: Hypothesis string. - ref: Reference string. - err_type: one of 'none', 'sub', 'del', 'ins'. - - Returns: - a html string where disagreements are highlighted. - Note `hyp` is highlighted in green, and marked with - `ref` is highlighted in yellow. If you want html with nother styles, - consider to write your own function. - - Raises: - ValueError: if err_type is not among ['none', 'sub', 'del', 'ins']. - or if when err_type == 'none', hyp != ref - """ - - highlighted_html = '' - if err_type == 'none': - if hyp != ref: - raise ValueError('hyp (%s) does not match ref (%s) for none error' % - (hyp, ref)) - highlighted_html += '%s ' % hyp - - elif err_type == 'sub': - highlighted_html += """ - %s - %s """ % (hyp, ref) - - elif err_type == 'del': - highlighted_html += """ - %s """ % ( - ref) - - elif err_type == 'ins': - highlighted_html += """ - %s """ % ( - hyp) - - else: - raise ValueError('unknown err_type ' + err_type) - - return highlighted_html - - -def ComputeEditDistanceMatrix(hyp_words, ref_words): - """Compute edit distance between two list of strings. - - Args: - hyp_words: the list of words in the hypothesis sentence - ref_words: the list of words in the reference sentence - - Returns: - Edit distance matrix (in the format of list of lists), where the first - index is the reference and the second index is the hypothesis. - """ - reference_length_plus = len(ref_words) + 1 - hypothesis_length_plus = len(hyp_words) + 1 - edit_dist_mat = [[]] * reference_length_plus - - # Initialization. - for i in range(reference_length_plus): - edit_dist_mat[i] = [0] * hypothesis_length_plus - for j in range(hypothesis_length_plus): - if i == 0: - edit_dist_mat[0][j] = j - elif j == 0: - edit_dist_mat[i][0] = i - - # Do dynamic programming. - for i in range(1, reference_length_plus): - for j in range(1, hypothesis_length_plus): - if ref_words[i - 1] == hyp_words[j - 1]: - edit_dist_mat[i][j] = edit_dist_mat[i - 1][j - 1] - else: - tmp0 = edit_dist_mat[i - 1][j - 1] + 1 - tmp1 = edit_dist_mat[i][j - 1] + 1 - tmp2 = edit_dist_mat[i - 1][j] + 1 - edit_dist_mat[i][j] = min(tmp0, tmp1, tmp2) - - return edit_dist_mat - - -class SimpleWER: - """Compute word error rates after the alignment. - - Attributes: - key_phrases: list of important phrases. - aligned_htmls: list of diagnois htmls, each of which corresponding to a pair - of hypothesis and reference. - hyp_keyphrase_counts: dict. `hyp_keyphrase_counts[w]` counts how often a key - phrases `w` appear in the hypotheses. - ref_keyphrase_counts: dict. `ref_keyphrase_counts[w]` counts how often a key - phrases `w` appear in the references. - matched_keyphrase_counts: dict. `matched_keyphrase_counts[w]` counts how - often a key phrase `w` appear in the aligned transcripts when the - reference and hyp_keyphrase match. - wer_info: dict with four keys: 'sub' (substitution error), 'ins' (insersion - error), 'del' (deletion error), 'nw' (number of words). We can use - wer_info to compute word error rate (WER) as - (wer_info['sub']+wer_info['ins']+wer_info['del'])*100.0/wer_info['nw'] - """ - - def __init__(self, - key_phrases=None, - html_handler=HighlightAlignedHtml, - preprocess_handler=RemoveCommentTxtPreprocess): - """Initialize SimpleWER object. - - Args: - key_phrases: list of strings as important phrases. If key_phrases is - None, no key_phrases related metric will be computed. - html_handler: function to generate a string with html tags. - preprocess_handler: function to preprocess text before computing WER. - """ - self._preprocess_handler = preprocess_handler - self._html_handler = html_handler - self.key_phrases = key_phrases - self.aligned_htmls = [] - self.wer_info = {'sub': 0, 'ins': 0, 'del': 0, 'nw': 0} - if key_phrases: - # Pre-process key_phrase list - if self._preprocess_handler: - self.key_phrases = \ - [self._preprocess_handler(k) for k in self.key_phrases] - - # Init keyphrase_counts for every key phrase - self.ref_keyphrase_counts = {} - self.hyp_keyphrase_counts = {} - self.matched_keyphrase_counts = {} - for k in self.key_phrases: - self.ref_keyphrase_counts[k] = 0 - self.hyp_keyphrase_counts[k] = 0 - self.matched_keyphrase_counts[k] = 0 - else: - self.ref_keyphrase_counts = None - self.hyp_keyphrase_counts = None - self.matched_keyphrase_counts = None - - def AddHypRef(self, hypothesis, reference): - """Update WER when adding one pair of strings: (hypothesis, reference). - - Args: - hypothesis: Hypothesis string. - reference: Reference string. - - Raises: - ValueError: when the program fails to parse edit distance matrix. - """ - if self._preprocess_handler: - hypothesis = self._preprocess_handler(hypothesis) - reference = self._preprocess_handler(reference) - - # Compute edit distance. - hyp_words = hypothesis.split() - ref_words = reference.split() - distmat = ComputeEditDistanceMatrix(hyp_words, ref_words) - - # Back trace, to distinguish different erroref_words: ins, del, sub. - pos_hyp, pos_ref = len(hyp_words), len(ref_words) - wer_info = {'sub': 0, 'ins': 0, 'del': 0, 'nw': len(ref_words)} - aligned_html = '' - matched_ref = '' - while pos_hyp > 0 or pos_ref > 0: - err_type = '' - - # Distinguish error type by back tracking - if pos_ref == 0: - err_type = 'ins' - elif pos_hyp == 0: - err_type = 'del' - else: - if hyp_words[pos_hyp - 1] == ref_words[pos_ref - 1]: - err_type = 'none' # correct error - elif distmat[pos_ref][pos_hyp] == distmat[pos_ref - 1][pos_hyp - 1] + 1: - err_type = 'sub' # substitute error - elif distmat[pos_ref][pos_hyp] == distmat[pos_ref - 1][pos_hyp] + 1: - err_type = 'del' # deletion error - elif distmat[pos_ref][pos_hyp] == distmat[pos_ref][pos_hyp - 1] + 1: - err_type = 'ins' # insersion error - else: - raise ValueError('fail to parse edit distance matrix.') - - # Generate aligned_html - if self._html_handler: - if pos_hyp == 0 or not hyp_words: - tmph = ' ' - else: - tmph = hyp_words[pos_hyp - 1] - if pos_ref == 0 or not ref_words: - tmpr = ' ' - else: - tmpr = ref_words[pos_ref - 1] - aligned_html = self._html_handler(tmph, tmpr, err_type) + aligned_html - - # If no error, go to previous ref and hyp. - if err_type == 'none': - matched_ref = hyp_words[pos_hyp - 1] + ' ' + matched_ref - pos_hyp, pos_ref = pos_hyp - 1, pos_ref - 1 - continue - - # Update error. - wer_info[err_type] += 1 - - # Adjust position of ref and hyp. - if err_type == 'del': - pos_ref = pos_ref - 1 - elif err_type == 'ins': - pos_hyp = pos_hyp - 1 - else: # err_type == 'sub' - pos_hyp, pos_ref = pos_hyp - 1, pos_ref - 1 - - # Verify the computation of edit distance finishes - assert distmat[-1][-1] == wer_info['ins'] + \ - wer_info['del'] + wer_info['sub'] - - # Accumulate err_info before the next (hyp, ref). - for k in wer_info: - self.wer_info[k] += wer_info[k] - - # Collect aligned_htmls. - if self._html_handler: - self.aligned_htmls += [aligned_html] - - # Update key phrase info. - if self.key_phrases: - for w in self.key_phrases: - self.ref_keyphrase_counts[w] += reference.count(w) - self.hyp_keyphrase_counts[w] += hypothesis.count(w) - self.matched_keyphrase_counts[w] += matched_ref.count(w) - - def GetWER(self): - """Compute Word Error Rate (WER) to summarize word erroref_words. - - Note WER can be larger than 100.0, esp when there are many insertion errors. - - Returns: - WER as percentage number, usually between 0.0 to 100.0 - """ - nref = self.wer_info['nw'] - nref = max(1, nref) # non_zero value for division - total_error = self.wer_info['ins'] \ - + self.wer_info['del'] + self.wer_info['sub'] - return total_error * 100.0 / nref - - def GetKeyPhraseStats(self): - """Measure the Jaccard similarity of key phrases between hyps and refs. - - Returns: - jaccard_similarity: jaccard similarity, between 0.0 and 1.0 - F1_keyphrase: F1 score (=2/(1/prec + 1/recall)), between 0.0 and 1.0 - matched_keyphrases: num of matched key phrases. - ref_keyphrases: num of key phrases in the reference strings. - hyp_keyphrases: num of key phrases in the hypothesis strings. - """ - - matched_k = sum(self.matched_keyphrase_counts.values()) - ref_k = sum(self.ref_keyphrase_counts.values()) - hyp_k = sum(self.hyp_keyphrase_counts.values()) - joined_k = ref_k + hyp_k - matched_k - joined_k = max(1, joined_k) # non_zero value for division - jaccard_similarity = matched_k * 1.0 / joined_k - - f1_k = 2.0 * matched_k / max(ref_k + hyp_k, 1.0) - return (jaccard_similarity, f1_k, matched_k, ref_k, hyp_k) - - def GetSummaries(self): - """Generate strings to summarize word errors and key phrase errors. - - Returns: - str_sum: string summarizing total error, total word and WER. - str_details: string breaking down three error types: del, ins, sub. - str_str_keyphrases_info: string summarizing kerphrase information. - """ - nref = self.wer_info['nw'] - total_error = self.wer_info['ins'] \ - + self.wer_info['del'] + self.wer_info['sub'] - str_sum = 'total WER = %d, total word = %d, wer = %.2f%%' % ( - total_error, nref, self.GetWER()) - - str_details = 'Error breakdown: del = %.2f%%, ins=%.2f%%, sub=%.2f%%' % ( - self.wer_info['del'] * 100.0 / nref, self.wer_info['ins'] * 100.0 / - nref, self.wer_info['sub'] * 100.0 / nref) - - str_keyphrases_info = '' - if self.key_phrases: - jaccard_p, f1_p, matched_p, ref_p, hyp_p = self.GetKeyPhraseStats() - str_keyphrases_info = ('matched %d key phrases (%d in ref, %d in hyp), ' - 'jaccard similarity=%.2f, F1=%.2f') % \ - (matched_p, ref_p, hyp_p, jaccard_p, f1_p) - - return str_sum, str_details, str_keyphrases_info - - -def main(argv): - hypothesis = open(argv[1], 'r').read() - reference = open(argv[2], 'r').read() - - if len(argv) == 4: - phrase_lines = open(argv[3]).readlines() - keyphrases = [line.strip() for line in phrase_lines] - else: - keyphrases = None - - wer_obj = SimpleWER( - key_phrases=keyphrases, - html_handler=HighlightAlignedHtml, - preprocess_handler=RemoveCommentTxtPreprocess) - - wer_obj.AddHypRef(hypothesis, reference) - - str_summary, str_details, str_keyphrases_info = wer_obj.GetSummaries() - print(str_summary) - print(str_details) - print(str_keyphrases_info) - - try: - fn_output = argv[1] + '_diagnosis.html' - aligned_html = '
'.join(wer_obj.aligned_htmls) - with open(fn_output, 'wt') as fp: - fp.write('') - fp.write('
%s
' % aligned_html) - fp.write('') - except IOError: - print('failed to write diagnosis html') - - -if __name__ == '__main__': - if len(sys.argv) < 3 or len(sys.argv) > 4: - print(""" -Example of Usage: - - python simple_wer_v2.py file_hypothesis file_reference -or - python simple_wer_v2.py file_hypothesis file_reference file_keyphrases - - where file_hypothesis is the file name for hypothesis text - file_reference is the file name for reference text. - file_keyphrases (optional) is the filename of key phrases over which - you want to measure accuracy. - -Or you can use this file as a library, and call class SimpleWER - .AddHypRef(hyp, ref): add one pair of hypothesis/reference. You can call this - function multiple times. - .GetWER(): get the Word Error Rate (WER). - .GetKeyPhraseStats(): get stats for key phrases. The first value is Jaccard - Similarity of key phrases. - .GetSummaries(): generate strings to summarize word error and - key phrase errors. -""") - sys.exit(1) - - main(sys.argv) diff --git a/lingvo/tasks/asr/tools/simple_wer_v2_test.py b/lingvo/tasks/asr/tools/simple_wer_v2_test.py deleted file mode 100644 index cdb7bd05..00000000 --- a/lingvo/tasks/asr/tools/simple_wer_v2_test.py +++ /dev/null @@ -1,90 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for simple_wer_v2.""" - -import lingvo.compat as tf -from lingvo.core import test_utils -from lingvo.tasks.asr.tools import simple_wer_v2 as simple_wer - - -class SimpleWerTest(test_utils.TestCase): - - def testTxtPreprocess(self): - txt = 'abcd [ABC] (fg) ??? OK! ' - txt_preprocessed = simple_wer.TxtPreprocess(txt) - self.assertEqual(txt_preprocessed, 'abcd abc fg ok') - - def testRemoveCommentTxtPreprocess(self): - txt = 'abcd [ABC] (fg) ??? OK! ' - txt_preprocessed = simple_wer.RemoveCommentTxtPreprocess(txt) - self.assertEqual(txt_preprocessed, 'abcd fg ok') - - def testHighlightAlignedHtml(self): - hyp = 'thank you' - ref = 'thank you' - html = simple_wer.HighlightAlignedHtml(hyp, ref, 'none') - self.assertEqual(html.strip(), 'thank you') - - def testWerIgnoreCommentPunc(self): - ref = '(Hello world)! [pause] Today is a good day! How are you?' - hyp = 'hello world. today is a good day, how are you' - wer_obj = simple_wer.SimpleWER() - wer_obj.AddHypRef(hyp, ref) - err_info = wer_obj.wer_info - self.assertEqual(err_info['del'], 0) - self.assertEqual(err_info['sub'], 0) - self.assertEqual(err_info['ins'], 0) - self.assertEqual(err_info['nw'], 10) - - def testKeyPhraseCounts(self): - key_phrases = ['Google', 'Mars'] - wer_obj = simple_wer.SimpleWER(key_phrases=key_phrases) - - ref = 'Hey Google. I have a question about Mars, can I google it? ' - hyp = 'Hey Google! I have question about Mars, can I google it? ' - wer_obj.AddHypRef(hyp, ref) - self.assertEqual(sum(wer_obj.ref_keyphrase_counts.values()), 3) - self.assertEqual(sum(wer_obj.matched_keyphrase_counts.values()), 3) - - ref = 'Hey Google, could you tell me a story about Mars? ' - hyp = 'Hey Google, could you tell me a story about March? ' - wer_obj.AddHypRef(hyp, ref) - self.assertEqual(sum(wer_obj.ref_keyphrase_counts.values()), 5) - self.assertEqual(sum(wer_obj.matched_keyphrase_counts.values()), 4) - - def testKeyPhraseJaccardSimilarity(self): - key_phrases = ['Google', 'Mars and Earth'] - wer_obj = simple_wer.SimpleWER(key_phrases=key_phrases) - hyp = 'Hey Google, could you tell me a story about March and Earth?' - ref = 'Hey Google, could you tell me a story about Mars and Earth?' - wer_obj.AddHypRef(hyp, ref) - stats = wer_obj.GetKeyPhraseStats() - jaccard = stats[0] - self.assertEqual(jaccard, 0.5) - - def testKeyPhraseF1(self): - key_phrases = ['Google', 'Mars and Earth'] - wer_obj = simple_wer.SimpleWER(key_phrases=key_phrases) - hyp = 'Hey Google, could you tell me a story about March and Earth?' - ref = 'Hey Google, could you tell me a story about Mars and Earth?' - wer_obj.AddHypRef(hyp, ref) - stats = wer_obj.GetKeyPhraseStats() - f1 = stats[1] - self.assertAlmostEqual(f1, 0.66666666666666667, delta=0.01) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/asr/wpm_16k_librispeech.vocab b/lingvo/tasks/asr/wpm_16k_librispeech.vocab deleted file mode 100644 index b7b4fb2e..00000000 --- a/lingvo/tasks/asr/wpm_16k_librispeech.vocab +++ /dev/null @@ -1,16328 +0,0 @@ - - - - - -▁ -t -h -e -a -n -d -o -f -i -w -s -y -u -r -b -m -v -l -c -p -k -g -j -' -q -x -z -▅ -" -. -- -: -! -~ -` -; -# -$ -% -& -( -) -* -+ -, -/ -< -= -> -? -@ -[ -\ -] -^ -_ -{ -| -} -th -the -▁the -in -an -and -▁w -ing -▁h -ou -on -▁b -of -en -er -ed -▁and -▁of -or -▁f -▁wh -ch -re -▁s -▁a -▁m -▁th -is -▁c -▁in -▁p -▁t -▁to -ow -▁d -▁n -▁i -gh -▁sh -▁g -om -ar -at -▁that -▁for -▁no -as -▁was -ad -▁l -ll -ver -▁be -▁wi -▁with -▁it -you -▁you -ld -ould -ve -ther -▁his -▁had -▁he -es -▁her -ly -▁on -▁re -id -ion -im -▁him -ght -▁bu -ent -un -▁but -le -▁not -ight -it -ic -▁ha -▁have -ir -us -qu -al -ay -ac -▁is -▁this -▁an -▁r -▁e -▁ex -am -ain -ter -▁fr -▁from -ur -▁j -▁o -▁k -▁un -aid -▁said -▁she -▁ch -▁as -'s -▁whi -▁which -▁con -▁com -all -up -▁up -oun -▁ou -▁st -ble -ation -ak -▁kn -▁will -▁all -▁v -▁or -own -ough -▁pr -▁go -▁would -▁by -ce -ound -os -▁qu -ard -▁what -▁when -ess -ous -very -od -our -ind -▁who -ought -▁li -ish -end -▁so -ok -▁my -▁wer -▁were -▁we -▁do -▁could -▁ne -▁should -▁wor -ook -ct -ome -▁some -▁su -▁se -▁very -▁tr -▁tim -ag -ong -ul -il -ill -▁br -ack -▁know -▁now -el -sel -self -▁at -▁y -▁us -▁man -▁mis -▁lik -ust -▁out -out -▁me -▁mor -for -▁look -ive -▁if -▁other -bout -▁tw -▁one -ven -▁en -▁dis -▁de -ame -▁good -▁every -▁been -ence -▁over -▁they -▁gr -ect -▁sp -pp -ck -ut -est -ast -'t -op -ap -st -et -ant -ance -thing -able -der -▁under -▁their -▁about -▁any -▁gre -▁great -▁did -▁well -▁upon -way -ure -um -▁there -ff -ful -▁are -▁al -ol -ri -▁our -res -▁your -▁see -▁more -▁tak -▁never -▁old -▁befor -ace -▁hand -▁how -ity -itt -ittle -▁little -▁even -▁like -▁down -▁day -ave -ose -ost -ot -art -ist -ood -other -▁two -ather -aw -▁own -▁than -▁can -▁thought -ount -gain -▁again -ang -ig -▁just -▁ro -ove -fter -▁after -▁mister -ell -▁time -urn -▁such -ach -▁has -▁int -▁into -ward -▁only -▁long -▁himself -▁str -▁bo -▁back -▁pl -ink -▁think -▁bl -▁must -ick -▁ar -mp -▁exp -ally -ate -▁le -▁mad -▁mak -▁them -ment -▁before -▁per -ry -▁thr -▁through -▁cl -▁came -▁without -▁part -ious -irst -▁first -▁lif -▁say -▁made -one -▁way -on't -ull -uch -▁much -ress -▁turn -oll -▁wher -▁where -▁imp -▁may -▁nothing -ud -▁app -n't -ness -▁then -ab -are -ved -tain -▁comp -▁while -▁those -▁hous -▁am -▁miss -▁shall -▁don't -▁ob -▁place -ass -act -▁di -▁i' -▁cr -▁dr -ving -ge -age -oss -se -▁seem -ies -▁happ -igh -ie -ite -ide -ice -ep -▁rep -▁op -▁fe -▁room -▁let -▁pass -▁though -▁thing -▁thre -▁tell -▁still -▁pres -ber -▁come -ine -▁fl -ject -▁another -▁missus -aus -cc -ort -ock -▁most -▁ad -▁pre -▁mar -▁work -▁new -▁pro -▁night -▁sup -▁might -▁comm -▁something -ign -▁life -▁word -▁world -▁rem -▁diff -▁yet -▁get -air -ath -av -ail -au -aught -▁away -▁alway -▁house -ers -▁supp -ip -if -ond -▁end -▁ask -alk -▁make -▁count -▁love -▁last -▁gl -iz -▁off -ollow -ild -▁child -old -▁went -ause -▁acc -▁call -ction -ition -ear -▁year -▁appear -▁wom -▁enough -▁open -▁once -▁po -ople -eople -▁people -▁des -▁hun -▁wat -▁cour -▁cont -▁father -▁give -▁ent -ady -▁occ -ible -▁quite -▁ey -▁eyes -▁high -▁fri -▁friend -▁take -▁found -▁three -▁want -▁ind -ord -▁yes -ark -selve -▁woman -▁having -▁bec -▁because -▁face -▁tre -ire -▁right -▁home -ult -▁these -▁course -▁mother -▁saw -▁feel -▁sc -▁sm -▁follow -▁same -augh -ature -ise -ix -▁bre -▁lef -▁left -▁jo -▁heart -▁head -atter -▁against -▁sw -▁small -▁put -ince -ated -▁ever -▁arm -▁spe -▁speak -ue -gether -▁men -anc -▁ear -▁beg -▁among -▁each -▁white -ertain -▁certain -▁wr -▁present -▁kind -▁wind -▁mind -▁find -▁anything -▁many -leas -▁pleas -een -▁light -▁nor -▁eng -▁prop -ossible -▁country -ried -▁black -ning -▁morning -▁began -ian -ily -ty -oment -▁moment -▁kne -▁knew -▁few -turn -▁return -▁name -cl -less -▁ab -▁car -▁mean -▁brought -▁king -▁walk -▁talk -▁sub -▁always -cess -bl -lie -▁belie -▁sl -▁bet -▁better -▁betw -▁between -▁hal -▁half -pt -cept -▁dist -▁asked -▁order -▁gir -▁girl -ody -ary -▁cap -ither -▁here -▁door -▁oh -ffect -over -▁took -▁fell -▁char -▁near -▁suff -▁wish -▁god -▁leave -▁gave -▁youn -▁young -▁water -▁adv -▁hund -▁hundr -▁lady -▁whom -▁why -▁add -▁strang -em -man -ank -emp -orn -▁nex -▁att -▁themselve -▁myself -▁inter -▁matter -▁brother -▁believe -▁sec -▁disc -onder -▁wonder -ever -▁however -▁cons -▁conf -most -▁almost -▁live -ci -ied -iv -▁ev -▁herself -▁both -▁boy -uck -▁gu -▁hop -▁emp -▁remem -▁remember -▁hard -▁heard -▁second -lied -▁replied -▁poor -▁hor -▁poin -iver -▁seemed -▁underst -▁things -pect -amp -▁everything -ory -▁its -▁interest -ions -oke -▁del -▁done -▁hour -▁hum -ush -▁care -▁point -▁set -▁dark -▁war -▁el -▁rest -▁inst -▁far -▁wa -▁wait -eep -▁keep -▁looked -▁sure -▁dre -iness -▁captain -ully -fully -▁watch -▁remain -▁consid -▁side -aut -▁beaut -app -az -ular -▁four -▁opp -▁whose -ro -▁gen -▁gener -▁nature -land -les -dd -▁sudd -▁sudden -▁soon -▁stood -▁along -▁strong -ake -▁six -▁res -▁rec -▁sir -▁feet -▁next -▁known -▁quest -wer -swer -▁answer -▁years -osition -aster -ash -ason -▁reason -▁mon -▁became -▁become -▁draw -▁law -▁stre -▁fear -▁em -▁horse -▁above -▁eight -▁rather -▁ought -ief -oice -▁voice -▁necess -▁laugh -▁death -▁understand -fort -form -ently -▁got -aint -elt -▁felt -ase -ict -opp -▁i'll -▁i'm -'ll -▁master -▁subject -▁object -▁round -▁ve -▁wood -▁cried -▁therefor -▁going -▁letter -▁reach -▁play -ense -▁pers -▁person -▁perh -▁perhap -▁else -▁full -▁hundred -▁twent -▁dire -ical -▁ref -▁reg -▁read -▁wif -▁being -▁whole -ield -▁question -▁true -▁seen -▁remark -▁doct -▁doctor -▁dou -▁doub -oud -▁sou -▁thou -▁thous -▁thousand -▁express -▁contin -▁consider -▁stand -ower -▁power -▁window -▁continu -▁told -▁gold -▁adm -mm -▁form -▁thir -▁thus -▁taken -gg -hind -▁behind -▁chang -erv -▁expect -▁hands -▁perhaps -fect -▁perfect -▁since -▁prince -▁doubt -▁unt -▁until -▁sil -▁sat -▁serv -▁twenty -▁hold -▁wife -▁lo -▁show -▁fact -row -urp -▁lord -▁can't -▁sun -▁till -▁gone -▁alone -▁gi -▁given -▁called -▁book -▁land -ool -▁too -ph -▁ph -▁cre -▁ground -▁close -▁quick -▁col -▁street -▁plac -ready -▁already -▁sever -▁break -▁cond -body -▁body -ile -ia -ib -▁turned -▁temp -umber -▁childr -▁children -ott -▁together -led -▁rel -▁ready -▁prin -▁around -▁fre -▁less -▁cann -▁cannot -▁respect -▁either -▁lay -uth -ual -▁earth -▁does -▁princess -▁bur -▁start -▁dep -▁dear -arge -▁large -▁succ -▁vis -ister -▁prom -▁cle -▁clear -▁general -▁best -▁women -▁whe -▁whether -▁inde -▁indeed -▁compan -leep -▁deep -▁kep -▁step -▁prof -▁happen -'d -de -▁ide -▁really -▁rece -▁differ -▁human -▁dream -▁number -▁const -▁fire -af -▁mount -▁account -▁accord -lish -▁english -▁fall -▁wall -▁short -ial -▁i've -▁truth -▁fellow -og -ations -▁words -▁toward -▁soul -▁ext -ton -ret -▁secret -▁cur -▁imm -▁summ -ouch -ale -▁suppose -wh -▁therefore -▁unc -▁als -▁also -▁themselves -oney -▁money -▁mo -▁min -▁minut -'re -▁suddenly -▁joh -▁john -▁except -▁run -▁fam -▁family -▁carried -▁nat -▁natur -▁least -▁blood -▁looking -▁sk -▁charac -▁character -imp -▁simp -▁times -▁enter -▁didn't -▁mountain -▁trou -▁regard -▁success -▁five -▁gent -▁gentle -vent -lp -▁help -▁expl -▁act -▁neither -▁condition -▁diffic -▁difficult -▁direct -uct -uc -▁kept -▁sound -▁poss -▁possess -▁cross -▁pe -▁hope -▁effect -▁wild -▁gover -▁beauti -▁beautiful -▁fair -▁visit -▁govern -▁case -▁co -▁dress -▁excl -ances -▁contr -roach -ities -ubl -ublic -side -▁eighteen -▁sign -▁sleep -ately -ith -ich -▁rich -ived -▁received -▁disapp -▁disp -▁gentleman -▁won't -▁eas -▁ele -oot -▁foot -iet -▁quiet -gre -▁begin -ne -▁itself -ner -▁manner -utter -▁pret -▁pretty -▁art -enc -▁real -▁touch -▁several -▁jud -▁mil -▁daught -▁england -ens -ier -ience -ipp -▁mer -▁entire -▁spoke -ably -▁occas -▁comfort -▁har -▁par -▁bar -▁discover -▁ter -▁spir -▁spirit -▁hair -ific -ift -▁rock -▁es -▁kill -oken -eng -ength -▁different -▁evid -idd -ually -vel -▁yourself -▁state -quir -▁queen -▁bus -▁business -▁creature -▁direction -▁trouble -▁dead -▁def -▁lost -▁river -ub -hip -▁ship -jo -▁approach -ross -cross -▁dec -▁idea -▁blue -▁it's -▁pol -▁possible -▁road -oft -▁necessary -▁seven -ention -▁sent -▁soft -ek -ke -▁te -ore -▁surp -▁purp -▁surpr -▁purpose -▁rose -iew -▁view -▁cold -olut -atch -ates -▁command -▁intell -unt -▁sister -▁delight -▁neigh -▁need -▁soci -▁table -▁answered -ished -ison -▁prov -▁dev -▁equ -icular -▁particular -chool -▁terri -▁spr -▁week -▁lear -▁learn -▁smile -empt -▁attempt -▁mist -▁che -▁chief -▁sea -▁strange -▁decl -▁school -▁story -▁days -▁smil -day -▁attention -▁leg -ility -ative -▁court -▁lead -▁glad -▁hur -▁dur -▁during -osed -▁prison -▁pleasure -▁common -▁princ -▁princip -▁build -▁dang -▁prod -▁satis -▁satisf -ient -iff -uff -▁corn -▁ste -ude -ug -▁grand -▁carry -opped -ugg -▁list -▁listen -▁hon -▁feeling -itude -▁desc -▁desp -urch -▁church -▁cover -▁pap -▁eye -▁occasion -▁month -▁mus -▁exist -▁exc -▁import -▁sort -ting -▁coming -▁making -ling -▁bel -▁tear -▁big -▁living -▁lived -▁ang -▁sn -▁impossible -▁daughter -ans -▁trans -▁esc -▁escap -▁belong -▁bir -▁bird -▁hung -▁friends -aven -urther -▁further -▁cir -▁circ -▁circum -▁sugg -▁move -▁mag -outh -▁worth -▁enem -▁enjo -▁comple -resh -▁thirty -▁instant -▁mouth -▁ma -▁maj -ating -▁thank -though -▁although -▁bad -ised -ted -▁wide -▁guard -▁sur -▁wrong -▁sold -▁held -rib -▁arri -riage -▁free -▁servant -▁chr -▁christ -▁use -▁uncle -▁glass -aim -▁exclaim -▁utter -▁cause -▁son -▁gard -▁garden -▁ten -▁green -▁third -▁mark -▁accom -▁ver -▁ed -▁est -▁conver -cious -wn -▁town -▁clo -▁clothe -▁forg -▁slow -▁grow -▁neighb -▁whis -▁government -▁desire -▁phil -▁div -▁circumst -oc -▁conc -▁passed -▁dem -▁coll -▁tried -▁mod -▁explain -▁plain -▁drew -▁saf -▁silence -ah -▁struck -▁strength -'ve -▁public -▁chair -ffair -ledge -▁knowledge -▁obs -▁across -▁air -agg -▁suggest -br -band -usband -▁husband -par -▁prepar -▁accomp -ump -▁happy -▁happiness -▁meet -▁none -▁taking -▁z -▁drink -▁service -tern -▁sense -▁pie -▁inf -tim -▁sometim -▁sometimes -▁stud -▁hall -▁depart -▁fin -erve -▁inc -▁sight -▁bright -▁knight -▁getting -▁proper -▁peace -▁occup -ten -▁often -▁conduct -▁exper -▁beginning -▁profess -▁scar -▁scarce -▁hear -▁star -▁stay -▁stopped -▁quar -ford -▁result -ask -▁giving -▁ben -▁evening -▁heaven -▁suffer -ides -▁south -▁neck -▁length -▁loved -rup -▁cor -▁cut -▁lad -▁understood -▁city -▁running -ified -ifty -ung -▁conscious -▁pray -iously -▁certainly -ages -▁admir -▁pract -▁priv -▁prot -▁experience -ater -▁later -ents -▁er -▁danger -▁circumstances -▁returned -▁hast -▁past -▁cast -▁ho -▁demand -▁possib -▁grew -▁aw -▁trave -▁exact -▁exce -▁towards -▁piece -▁rob -itch -iss -ont -▁front -tle -attle -▁field -▁ill -ead -▁instead -head -▁dread -▁shad -▁shadow -▁whatever -▁chap -aith -▁faith -▁arrived -erved -▁observed -▁morrow -ender -ended -▁somewh -▁terrible -▁teach -oup -uthor -▁offic -icture -ney -▁stri -▁whisp -▁o' -▁sens -▁picture -▁broken -▁sevent -▁abs -ounc -avour -ments -▁disappear -aving -▁leaving -▁try -inner -▁dinner -▁easily -▁recog -▁mere -gree -▁cloud -▁fifty -▁prob -▁probably -▁societ -▁class -▁o'cl -▁grave -▁absolut -▁met -▁natural -▁hill -cul -lf -▁self -▁flo -▁flow -▁snow -▁others -▁squ -▁thy -▁thick -▁minutes -well -▁ac -▁accept -▁acqu -▁acquaint -igure -▁figure -▁ag -▁caught -▁fill -▁fine -itting -▁slight -ors -▁marg -▁margare -▁dog -ourney -▁journey -▁o'clock -▁chapter -mag -▁imag -mbl -te -▁complete -▁means -▁married -cially -▁bene -▁margaret -▁afterward -▁breath -▁beneath -▁low -▁crow -▁crowd -▁sitting -▁yell -▁win -▁winter -▁box -▁boat -▁esp -▁swe -▁sweet -oon -ized -ize -▁kiss -▁bank -▁opin -▁forget -▁bit -▁position -▁quarter -▁peter -▁laid -▁je -▁cry -raid -fraid -ract -▁sus -▁sy -▁symp -▁surprise -▁anim -▁animal -▁wit -▁hot -▁grace -▁mur -▁murder -▁occur -▁below -▁allow -▁chance -▁mill -uction -▁maid -▁ann -▁suffic -▁dri -▁infl -▁madame -▁vill -▁bott -▁sword -▁recogn -▁prec -asure -▁troubl -▁roll -▁excell -▁super -ffort -▁effort -▁eat -▁pat -▁opinion -ceived -▁indian -▁port -▁floor -▁moved -▁bring -▁sail -ron -▁fac -▁marriage -▁distance -▁concer -▁lip -▁according -▁miles -▁fool -▁fail -▁proce -▁obl -▁oblig -▁affect -ined -inct -lect -▁gather -▁fight -▁fright -▁fresh -▁caus -▁moon -▁shout -▁relig -▁silver -▁ass -▁vict -▁val -▁vir -▁presence -iately -▁dropped -▁bow -▁deter -▁determ -▁determined -▁ago -▁dut -▁strugg -▁stra -▁straight -▁continued -▁eag -▁reached -ov -▁arr -▁arrang -gr -▁describ -▁beauty -orrow -▁writt -itted -▁writ -▁brown -▁fast -▁write -▁refus -▁forward -▁makes -otion -olog -ifteen -teen -▁cal -▁camp -▁influ -bert -▁evil -▁conse -▁consequ -▁conv -▁concl -▁plan -▁address -▁threw -▁incre -▁attack -▁jack -ident -▁warm -men -▁gentlemen -irm -ired -ench -▁french -▁flowers -▁pleasant -▁twel -▁twelve -fortun -▁fish -▁honour -hood -▁fix -▁slowly -▁carriage -▁protect -▁satisfact -▁separ -▁produc -ically -▁finally -▁ret -▁requ -▁pull -▁pain -ike -▁longer -▁gaz -▁mention -ustom -▁ladies -▁heav -▁heavy -▁appearance -▁appeared -▁bill -▁born -▁bound -▁battle -▁cheek -▁nearly -asp -▁cent -▁arms -▁ran -▁prevent -▁afraid -▁becom -ope -▁tree -▁island -▁guess -▁board -ences -▁relie -▁trees -▁deal -▁early -ocket -▁pocket -▁impress -▁fort -▁fortun -int -under -▁commun -▁sit -bit -▁habit -▁ye -▁anne -▁anx -aken -▁written -▁slave -▁save -ches -ane -ote -▁wrote -ade -▁change -▁i'd -▁stream -▁contain -▁obtain -▁pay -▁declar -▁var -▁bear -▁train -▁inv -▁rev -▁stone -▁dest -▁yellow -▁enjoy -▁followed -▁sad -▁saint -alth -▁north -▁wound -▁believed -▁able -▁opport -▁opportun -▁food -▁enc -▁surprised -▁comes -ured -ivid -▁individ -▁individual -▁usual -▁whisper -▁quickly -▁jew -▁destro -▁doll -▁bell -iddle -ons -▁colon -▁pecul -▁wonderful -ination -▁conce -▁joy -▁intellig -▁intellect -▁hardly -▁silent -noon -▁afternoon -ann -▁tone -▁fanc -▁forgott -▁susp -▁stor -▁please -▁mem -▁member -▁liber -▁expression -▁famil -▁frequ -▁closed -▁outside -▁beside -▁repres -▁immed -▁immediately -eless -▁finished -▁bitter -▁music -▁yard -▁decid -▁trembl -▁author -▁send -▁middle -▁ser -iest -▁exclaimed -▁states -▁evidently -▁easy -▁safe -▁society -ey -arch -▁search -▁sing -▁single -▁simple -val -▁term -▁terr -▁you're -▁interrup -▁bed -▁happened -▁entered -▁det -▁extre -▁extrem -▁pur -▁pal -▁palace -▁various -ablish -▁establish -▁lab -▁bab -▁accomplish -▁press -▁tender -▁fun -▁throw -▁neg -▁bread -▁sal -ality -ution -▁broke -off -▁added -▁used -▁trust -▁influence -▁existence -▁espe -▁especially -▁shar -▁sharp -athe -▁ra -▁raised -▁promised -▁promise -ury -▁company -▁anxious -ism -▁priest -▁stair -▁seventeen -▁sufficient -▁line -▁courage -▁village -▁paper -ondon -▁london -yond -▁beyond -alley -▁ball -▁generally -inary -▁within -arrow -hold -▁spring -▁blow -▁blind -▁offer -oy -unk -▁assist -▁mine -▁vi -▁viol -ium -▁represent -▁colour -inc -ript -zz -▁narrow -▁valley -▁attract -▁satisfied -▁support -aterial -▁opened -red -ires -▁important -▁fifteen -▁couldn't -▁scarcely -▁compl -▁spoken -▁bottom -▁marry -▁mary -ited -▁ourselve -▁season -▁christian -▁sol -▁memory -▁elect -▁colonel -▁polit -velop -▁develop -▁pick -iar -▁peculiar -▁familiar -▁minute -ere -where -▁eager -▁worse -sembl -▁mass -▁grass -ute -▁darkness -avor -iment -▁crim -▁gall -ened -▁frightened -▁red -▁rush -▁ris -▁lie -▁hurried -ines -ames -▁calm -▁travell -oth -▁corner -▁firm -▁says -▁prev -▁collect -▁reflect -loy -▁employ -▁march -▁charm -▁distinct -▁situ -▁shook -▁gh -▁despair -▁office -▁walked -▁saying -▁perform -▁repe -▁repeated -▁gun -▁standing -▁ideas -▁claim -▁party -▁carefully -quired -▁conn -▁connect -▁proud -▁hours -ins -▁convers -▁conversation -▁judge -▁judg -▁affair -▁questions -▁furn -▁wick -▁bless -▁seems -eness -▁mess -▁confess -▁wanted -▁sorrow -▁spot -▁faint -▁favour -▁educ -▁knock -▁regular -ency -▁fash -▁fashion -▁property -▁farm -▁increas -ert -▁observ -▁slipp -▁constant -▁poet -▁golden -idden -▁entirely -▁stop -ham -amber -▁chamber -ament -▁main -▁earn -▁subst -▁proceed -▁nine -▁group -▁dollar -▁opportunity -▁rap -▁rapid -▁stick -▁lifted -▁inform -▁late -▁minister -while -ald -wise -▁simply -▁rid -▁duty -▁lower -▁loud -ission -usion -ius -olution -angu -▁langu -uit -▁fruit -▁ru -uce -▁intr -▁introd -▁instr -▁private -▁favor -▁serious -▁neighbour -▁angry -▁sigh -▁swift -▁supposed -▁receive -ra -▁extra -▁post -▁lips -▁stret -▁stretch -▁virt -▁aunt -▁handsome -▁die -ster -▁myster -▁suspect -▁motion -▁naturally -▁jane -▁jump -▁honor -▁honest -▁earnest -▁tears -▁instinct -▁oppos -▁opposite -sides -▁besides -▁advanc -▁sympath -uke -▁breakf -▁breakfast -▁somewhat -▁hat -ones -▁degree -▁latter -▁language -▁wise -▁ow -▁comb -iers -▁soldiers -▁soldier -▁forgotten -▁scen -▁president -▁harm -▁attend -▁piec -▁entertain -▁report -idence -ault -▁tall -▁path -▁pen -▁color -ony -▁confidence -▁cat -eth -▁police -come -uous -▁spite -▁wheel -▁unless -▁surr -▁surround -▁surf -▁surface -▁everybody -▁storm -amm -▁major -ounter -▁empty -▁ess -▁places -▁placed -▁game -▁merely -▁enemy -by -awn -▁magn -▁houses -light -▁spl -▁splend -▁suc -▁succe -▁roof -▁pale -▁sac -▁sick -▁burn -▁burst -▁appar -▁sacri -dom -▁kingdom -▁rom -oom -▁gloom -to -together -▁altogether -▁plant -▁absolute -ship -▁value -aph -▁excite -▁majest -▁sim -▁clim -ices -▁sand -▁oper -▁shap -ior -▁clean -▁concern -▁moving -▁filled -▁exam -▁exh -▁cheer -▁ce -▁social -cial -▁special -cient -▁ancient -▁spec -▁custom -▁mistress -gress -oop -oph -▁dar -▁brave -▁fortune -ric -▁suit -uel -▁cruel -▁measure -▁treasure -osoph -▁philosoph -let -tory -▁history -ilt -▁built -▁emb -▁delic -▁obey -▁key -▁scr -▁ceas -▁process -▁progress -▁hang -▁stepp -ries -▁miser -▁deliver -▁univer -▁phy -▁engag -▁begg -▁thrown -▁bold -▁occurred -▁army -ley -▁castle -▁fle -▁perfectly -▁advant -▁advantage -▁amer -▁americ -▁van -▁vol -▁nobody -▁shot -▁shut -uture -▁future -▁sky -▁west -▁flower -wood -iumph -▁summer -▁wouldn't -▁that's -▁exactly -▁seat -▁host -▁arch -▁sixteen -▁double -▁following -▁eld -▁escape -▁square -▁recover -▁accompan -▁climb -▁edge -▁catch -▁iron -▁wave -▁situation -▁gain -▁passion -▁yield -▁shore -▁extraord -▁murm -▁incl -▁drop -▁flash -▁liberty -▁temper -▁paid -▁frank -aft -▁extraordinary -▁sacrif -▁mid -▁ap -▁apart -▁tired -▁cab -▁cook -▁civ -▁civil -▁cup -▁cost -▁syst -▁system -▁american -▁wand -▁prove -▁lean -▁affection -▁arrival -▁constit -▁pros -dinary -▁breast -▁hurry -▁hurt -umb -umbl -▁song -▁examp -anch -▁gar -▁ge -▁geor -▁george -▁charge -▁lim -▁feature -▁clothes -▁force -▁seek -▁triumph -▁destroy -▁cott -▁difference -▁weak -akes -▁phys -▁hen -▁henry -▁den -▁superior -rew -read -▁spread -▁space -▁companion -▁companions -▁teeth -▁tea -▁spee -▁speech -ode -▁smoke -ven't -▁haven't -▁extreme -▁kitch -ax -olved -aved -oved -▁proved -▁sixty -▁doesn't -orry -▁sorry -▁wander -iety -▁rain -str -▁action -▁regret -▁pieces -▁und -▁seized -▁detail -▁tail -▁tong -▁tongue -ague -▁depend -▁resp -▁resolved -▁rub -▁proof -▁pop -▁pun -▁isn't -▁contrary -ables -arg -▁fat -▁clever -▁perm -▁curious -▁pref -▁material -▁meant -▁suspic -▁kitchen -use -▁waiting -▁pound -▁duke -▁nice -urb -▁disturb -▁accident -▁content -▁scene -▁imagination -▁imagine -▁egg -▁thor -▁thorough -▁you'll -▁bull -▁advent -▁fault -▁coat -▁treat -▁drawn -▁ordinary -▁luck -▁bro -▁broad -▁sides -▁serve -▁decided -▁eleven -▁twice -▁dick -▁simil -▁crown -▁grown -▁similar -arm -▁alarm -imm -▁threat -▁agree -olly -▁forth -▁fo -ob -▁lou -▁edward -▁somewhere -▁bush -▁gift -ork -▁york -▁appre -rel -▁tast -▁instance -▁flew -▁disappo -▁sav -▁sch -retch -▁wretch -▁labor -▁succeed -▁mir -▁stir -▁east -▁sunday -ues -▁excited -▁laughed -▁evidence -▁arg -▁splendid -▁accustom -▁protest -aff -▁occupied -▁there's -▁necessity -▁authority -▁mix -▁remained -▁sett -▁settled -▁former -▁fallen -▁contempt -chief -▁luc -▁lock -▁ghost -▁popular -▁merc -▁rough -ette -▁eliz -arily -▁cand -▁coup -▁cool -▁angel -▁adventure -▁usually -▁apparently -▁sprang -▁building -▁talking -▁magnific -▁movement -▁thinking -ending -▁started -▁turning -▁excuse -iction -▁glance -▁excitement -▁leap -▁meanwhile -itive -idge -uge -▁age -▁punish -▁rode -▁hors -▁horses -▁horr -ians -▁months -▁earl -ously -▁rang -▁events -erman -▁german -▁gate -▁native -▁detect -▁ven -▁desired -▁covered -inted -▁interrupted -▁paus -▁pack -▁paint -▁putting -▁perceived -▁sake -▁united -▁conceal -fford -▁witness -▁france -▁rejo -▁removed -▁record -▁shop -▁shoot -▁otherwise -▁shr -iod -▁period -▁anxiety -uss -▁discuss -▁trying -▁study -▁render -theless -▁nevertheless -▁fancy -▁baby -yp -▁typ -▁pride -▁pow -▁passage -▁spect -▁amount -▁ly -▁lying -▁thunder -▁guest -▁equal -▁equally -▁assum -▁lover -▁vain -dle -▁driven -key -▁strike -▁nin -▁ninet -▁nineteen -inite -igin -▁origin -▁d' -▁d'art -▁charles -▁mamm -▁reply -▁friendship -▁frequently -iently -▁precious -▁design -eign -ollect -▁recollect -urg -urope -▁europe -ery -acy -▁roy -▁royal -▁difficulty -ify -▁franc -▁glanc -▁milk -▁push -▁waters -▁stranger -▁d'artag -▁importance -▁forty -▁drove -▁intended -▁political -▁constitution -▁larg -ception -▁died -▁actually -▁joined -▁noble -▁smith -▁dry -▁hunting -▁thorn -▁stones -▁infinite -atic -terday -▁yesterday -▁excellent -▁asleep -olk -▁folk -▁cottage -▁required -▁intention -▁birth -▁moder -▁powers -▁maintain -▁prospect -▁troop -▁you've -▁physical -▁jul -▁danc -thod -▁method -▁loss -▁wealth -▁health -▁cow -▁advice -▁wear -▁keen -▁news -▁afterwards -▁aware -lex -ex -▁fox -▁contemp -▁farther -thern -onscious -▁unconscious -▁father's -▁virtue -▁wil -like -▁patient -▁huge -▁fier -▁fierce -▁purs -▁century -▁papa -▁papers -▁bun -▁buried -▁scre -▁check -▁example -▁changed -▁wine -▁devil -▁feather -▁weather -▁coff -▁cas -▁nur -▁faces -▁girls -▁encounter -▁unfortun -▁misfortun -▁sle -▁slept -▁satisfaction -▁possession -▁dreadful -▁silk -son -ants -raph -ograph -▁mistake -▁names -▁james -▁reck -▁rever -▁ut -▁enemies -ny -noc -gn -gnor -▁virg -erg -▁energ -sh -▁boys -▁we'll -▁entr -▁persu -▁project -▁alive -▁convinc -▁boil -▁learned -ided -▁killed -gan -▁urg -▁coast -▁immense -▁pleased -▁bod -▁bodies -oat -▁throat -▁drive -▁youth -ument -▁encour -▁vent -▁vast -▁southern -▁compar -sie -▁monsie -▁monsieur -▁elsie -▁taste -▁knif -itation -▁sufficiently -▁nation -igg -aces -▁takes -▁attach -▁paris -itary -▁military -▁lot -▁boot -▁la -▁lines -▁lose -▁concerning -▁burning -▁introduc -▁lake -▁presently -▁justice -▁fond -▁drunk -▁quietly -▁consist -▁utmost -▁speaking -▁expected -▁hidden -▁hide -▁hither -unch -▁travel -▁err -▁impr -▁stock -▁brain -▁grief -▁brief -▁ant -▁proport -▁ignor -irt -▁begun -▁pier -▁unh -▁unhapp -▁hath -▁emotion -▁elizab -▁elizabeth -▁wid -und -▁fit -arest -▁bob -be -▁seeing -▁prefer -▁religion -▁grad -▁gradually -▁gray -▁grant -▁race -▁prompt -father -▁grandfather -▁benef -▁dism -▁stup -▁drawing -▁visible -ocr -▁mach -▁man's -athed -aking -▁buck -▁tur -▁fur -▁fully -▁hole -▁terror -▁religious -asket -▁basket -▁task -aver -▁admitted -▁thin -▁skin -▁entrance -▁labour -▁observe -▁famous -▁cous -▁cousin -amond -▁diamond -oes -oses -osity -ancho -▁belief -iam -▁william -ik -iant -▁distant -▁distress -iosity -▁curiosity -ribut -agon -▁fixed -▁strik -ively -▁hyp -▁magic -▁communic -▁sell -▁officers -ares -ability -▁powder -▁discovered -uzz -▁doing -▁science -▁odd -▁hotel -▁america -▁message -att -▁capt -▁writing -▁foolish -▁anybody -▁declared -▁forest -▁russ -▁rail -▁descript -▁greatest -▁level -▁species -▁midst -▁ourselves -▁exhib -▁trib -▁provided -▁busy -▁showed -▁mort -▁intim -▁produce -▁ug -▁deck -▁lit -▁d'artagn -▁d'artagnan -▁unk -▁unkn -▁unknown -▁remind -▁patience -andon -▁abandon -▁parl -▁mart -▁mot -▁mysterious -gar -illiant -▁brilliant -▁relief -ibility -▁shame -▁desert -▁clos -▁amaz -▁murmured -▁education -▁organ -lection -▁reflection -▁dav -▁principle -▁sought -▁control -▁quarrel -▁personal -with -▁lamp -▁announc -▁allowed -▁consciousness -iving -▁forgive -▁motive -▁forgot -vest -▁books -▁choice -just -ugust -▁nurse -▁dwell -▁dust -▁dull -▁due -▁surely -▁club -rupt -▁card -▁shown -▁beast -▁argument -▁served -▁steps -▁moral -▁sacred -▁innoc -▁modern -▁perce -expect -▁lod -▁indiff -▁pot -▁pair -▁resist -istr -▁jun -▁beat -▁thrust -▁arth -▁arthur -▁mel -▁melancho -▁stuff -▁skill -▁nodd -▁fal -▁false -ital -▁capital -▁conqu -▁pet -▁noise -▁loc -▁indep -▁independ -▁notice -▁somebody -▁settle -▁everywhere -▁band -zen -▁dozen -▁relations -▁awful -▁appreci -▁shape -que -▁assured -▁inside -▁directly -in't -▁revolution -▁smok -ime -▁rule -▁cabin -▁tele -ining -▁shining -▁shock -▁hollow -▁thirteen -▁fourteen -▁evident -▁unto -io -▁dign -▁indign -▁dom -▁dim -▁dam -▁dare -▁bare -per -▁prosper -▁upper -▁perp -▁greater -▁unf -▁salt -▁information -▁seventy -▁bag -▁mistaken -▁smiled -▁smiling -ator -▁grat -▁spent -▁corres -how -▁request -▁nerv -▁dignity -▁passing -▁shoes -▁gently -▁grasp -▁corp -▁corre -▁correct -▁grey -▁greek -er's -▁dangerous -▁stupid -▁liter -▁hol -▁recall -▁glory -▁victory -▁couple -▁merch -▁sacrifice -▁practice -▁consci -oci -▁associ -▁consent -▁smo -▁smooth -▁conscience -▁neglect -▁unp -▁acquaintance -▁strain -▁majesty -iments -▁increase -▁recognized -▁intelligence -ves -▁wished -▁horror -▁pure -▁knees -▁indic -▁harry -▁hungry -▁delay -▁pursu -▁continue -lcome -▁welcome -▁cases -▁event -▁powerful -▁obliged -▁horri -▁horrible -▁genius -▁wol -▁possibly -▁cord -▁absence -▁fly -ival -▁rank -▁seated -▁seld -▁seldom -apped -ision -▁saved -▁smell -orth -worth -▁stories -rest -▁meeting -▁safet -▁ears -ouse -ole -vo -void -▁bridge -▁require -▁embr -▁rise -then -▁secure -ises -▁wis -▁wisdom -aged -▁principles -▁david -▁scream -▁nam -▁steam -▁loose -▁shoulders -ux -▁sex -▁singular -▁commission -▁dish -ows -▁parents -▁dece -ounce -▁permitted -▁fing -▁fingers -▁finger -▁condem -▁fever -▁match -▁flor -▁trem -▁depth -▁judgment -▁disappoint -▁arrest -▁fold -▁savage -▁dawn -▁indust -▁artic -▁worst -oices -▁voices -▁vague -▁conclusion -▁conclud -▁worship -▁beh -hab -▁ah -▁profound -▁awaken -▁advance -▁crack -▁laughing -▁stead -▁respons -▁poison -▁corresp -▁correspond -▁engage -▁wolf -▁robert -▁root -▁fourth -▁resembl -chang -▁exchang -▁exe -▁exec -▁exer -▁exerc -asc -ze -▁trade -ets -▁legs -▁risk -▁rising -▁arose -▁accompanied -▁moreover -▁horiz -▁blank -▁speed -▁etern -▁eff -▁innocent -▁hitherto -▁exercise -▁subm -▁oppress -▁instrument -▁substance -aughter -▁taught -▁thirst -▁interval -▁nervous -ensive -▁om -▁mat -▁reward -▁dressed -▁convent -itated -▁beloved -▁arrow -▁led -▁birds -isc -▁forc -▁trail -ulous -aul -▁paul -▁sole -▁solem -▁solemn -▁newsp -▁newspap -▁professor -▁neighbor -stair -▁walls -overy -▁discovery -▁hence -▁miserable -▁remarkable -▁considerable -habit -habitants -▁admit -▁retre -▁whispered -▁pursuit -▁retired -acious -▁mamma -▁sugar -▁sum -▁tom -▁fem -▁disappointed -▁merry -▁consequence -erence -▁officer -▁eg -▁disg -▁divine -▁divided -▁proportion -▁appo -▁appointed -▁size -▁holding -▁persist -▁privile -▁holy -▁instantly -uly -▁truly -ourn -▁pour -▁vig -▁hastily -▁dollars -▁delicate -▁supper -▁adam -▁learning -▁improve -▁constr -▁ir -▁sul -▁syl -▁sylv -▁sylvia -▁volum -▁confusion -▁erect -▁resum -▁appreh -▁emper -▁emperor -▁impression -▁env -▁relative -▁walking -ings -▁thoughts -▁type -▁arrange -▁track -▁lack -▁counc -▁council -▁deb -road -uments -▁virgin -▁feast -▁alice -▁calcul -▁roar -arr -▁embarr -▁whenever -ising -▁governor -▁buy -ancy -▁bas -▁intense -els -ssel -▁vessel -▁flat -▁higher -▁aloud -▁log -▁load -▁loving -▁aston -▁astonish -▁astonished -uished -ett -▁proposed -▁persons -▁letters -▁attitude -room -ges -enced -▁republic -▁reproach -▁interp -▁top -▁embarrass -▁wore -▁worn -night -▁midnight -▁flight -▁experi -▁negro -▁gratitude -▁execution -omp -▁wasn't -▁peas -ars -▁cloth -▁counten -▁countenance -den -▁sensible -▁useful -▁careful -▁faithful -▁explan -▁curtain -▁absolutely -▁includ -▁inclined -▁unexpect -▁dun -▁tor -order -▁precise -atherine -ned -▁reckon -▁coach -▁schem -▁dragg -abel -▁amuse -▁complex -▁throughout -▁considered -▁throne -▁consult -▁laughter -▁lives -▁knife -▁worthy -yr -▁ep -▁unpleas -field -▁particularly -▁oak -▁oce -▁ocean -▁plent -▁plenty -▁foreign -vereign -▁adver -▁enth -▁enthus -▁enthusi -▁animals -▁messeng -▁previous -▁prevail -▁addition -▁share -▁intellectual -▁engaged -ounsel -crib -▁rul -▁capable -ged -▁stern -▁fate -▁impat -▁philip -uish -ishop -▁bishop -▁supre -stone -oul -▁price -▁valu -▁engine -▁diver -▁weight -aily -▁daily -▁caused -▁offered -▁nearest -dy -▁somehow -cript -istic -ification -▁prepared -▁bottle -itiz -▁citiz -ization -▁caut -▁wings -▁alas -▁king's -▁cliff -▁stiff -▁cart -▁contract -▁hid -▁northern -▁scatter -▁liked -▁severe -▁female -▁freed -▁freedom -▁experienced -▁comfortable -lic -ler -iot -▁cave -▁named -▁adop -▁waist -▁mutter -▁profession -▁glow -▁forced -▁spark -▁rare -▁trad -▁consequently -▁mile -▁mult -▁cult -▁cultiv -▁med -▁mal -▁stairs -▁variety -▁services -▁rapidly -▁resolution -▁nose -kins -mother -▁grandmother -lexand -stand -▁station -▁flour -▁cho -▁choose -vey -▁reve -▁obv -▁obst -▁lux -▁lunch -▁max -▁sentiment -▁setting -ulation -▁wash -▁test -▁whir -▁treated -▁rubb -▁ease -▁pepp -ially -▁reveal -▁ideal -▁victim -▁horn -▁safety -▁completely -▁telegr -▁remembered -▁suspicion -▁descended -▁sympathy -ositive -▁explanation -▁tight -▁confirm -▁raise -▁proble -▁struggle -▁hind -▁softly -▁waited -▁barb -▁barbar -▁immedi -▁immediate -▁store -▁bore -▁convey -▁steady -▁arab -mond -▁official -▁trial -▁catherine -▁murmur -▁crime -▁touched -▁carp -osp -▁hosp -▁hospital -▁frag -▁indul -▁hom -omm -▁recomm -withstand -▁practical -▁awake -▁composed -▁instruction -▁lovely -▁purch -ners -▁burden -▁jeal -▁jealous -ckn -cknow -umn -▁bought -▁fought -▁temple -▁nearer -▁devot -etch -▁fetch -▁greet -▁household -▁behold -▁curr -ray -▁betray -▁ray -▁perplex -▁annoy -▁voy -onsense -▁nonsense -▁park -▁positive -▁consum -▁disposition -▁displ -▁display -▁ol -▁oliver -▁don -eld -▁beheld -ee -▁coffee -▁scot -▁unhappy -▁astonishment -ension -▁citizens -▁disting -▁distinguished -▁distinguish -▁distr -▁district -ina -ulf -▁anger -▁departure -▁circle -▁servants -ardon -▁pardon -▁pip -phere -osphere -▁fancied -▁mountains -▁comr -▁violent -▁darling -▁pity -▁gathered -abbit -ern -▁larger -▁exha -▁exhaust -▁illust -rop -▁dor -▁prisoner -▁physic -▁assistance -▁machine -▁meantime -▁widow -▁pan -▁we're -▁they're -▁leaves -▁duties -oseph -▁joseph -▁passeng -▁ox -▁meaning -▁original -pir -▁select -▁pierre -▁produced -▁univers -▁victor -▁suprem -ists -▁julia -▁whist -▁pra -▁praise -▁nay -▁releas -▁apolog -▁paused -▁realized -▁watched -▁watching -▁christm -▁christmas -▁opposition -▁million -▁disappeared -▁mood -▁flood -▁flung -▁growing -▁lies -▁column -▁vanished -▁hunt -▁problem -asm -▁enthusiasm -▁democr -▁affairs -▁appro -▁bal -▁bent -▁bay -▁magnificent -▁wicked -▁lone -▁ride -▁guide -▁abroad -▁abrupt -▁principal -▁thoroughly -▁span -incere -▁proc -▁electric -▁dance -iance -▁merchant -▁deli -▁deliber -▁eating -▁ec -▁cheerful -antic -▁committed -▁sour -▁source -▁memor -▁hunter -▁vision -amb -▁definite -▁roman -anches -▁branches -▁brush -▁ceased -work -▁specul -▁becomes -▁goes -▁occasions -allow -▁swallow -▁movements -▁whit -bles -▁leather -▁leaf -▁unfortunate -▁portion -▁lamb -na -▁trunk -onstr -ille -ville -illa -▁pill -▁marilla -▁polite -▁bloom -▁blush -▁flush -▁bride -▁aside -▁brit -▁british -▁maiden -▁remarked -udder -▁kindness -▁monster -acle -▁butter -▁alter -▁disgust -▁height -▁essent -▁amb -▁ambition -▁louis -▁awa -▁anyone -▁voyage -▁revol -▁proph -▁kitt -▁committ -▁committee -▁retain -view -▁interview -ev -▁nap -▁whil -▁whilst -▁colle -▁college -▁block -▁clock -▁blame -▁frame -▁product -▁chain -▁invited -▁drif -▁driving -▁cop -▁cell -achel -▁excess -▁strict -▁useless -▁lonely -▁pulled -▁wal -ral -▁countess -▁empire -▁sweep -▁swep -▁swept -▁weep -▁sheep -▁creep -▁ventured -▁venture -pe -▁egy -▁egypt -▁trick -▁countries -▁por -▁pause -▁pitch -▁libr -▁library -▁imagined -▁examined -▁drown -▁extended -▁extent -▁attended -vestig -▁continually -▁agreeable -fied -▁violence -▁absur -▁absurd -▁possibility -▁delightful -▁windows -▁necessarily -▁easter -▁thread -▁steadily -▁heavily -▁shaking -▁dearest -▁rope -▁incident -▁essential -▁counter -▁discour -▁mischief -▁fleet -pl -▁clasp -iced -▁noticed -▁ignorant -▁degre -▁agre -▁agreed -▁econ -▁econom -oard -board -▁thornton -▁shoulder -▁disposed -▁alike -▁punishment -▁grim -▁dy -▁consideration -▁bark -▁crept -▁crew -▁thither -▁centre -▁aff -▁robber -▁humble -ivity -▁trip -▁preach -▁spare -▁fare -▁farewell -▁purposes -eas -▁uneas -▁perpet -▁fairy -quis -▁row -▁domest -▁domestic -astic -▁active -mosphere -▁ice -▁respond -▁saddle -▁magistr -▁invent -agger -▁shudder -keep -▁spanish -▁admiration -acity -▁dying -▁conviction -▁deserve -▁desir -opping -▁reven -▁conven -iven -iveness -▁reign -aunt -▁volunt -▁steep -▁elder -▁connection -▁facult -▁advanced -▁stage -▁complain -▁amongst -▁gle -▁gleam -▁guil -▁guilty -▁perceive -▁clearly -▁surrender -▁silently -▁highest -▁manners -▁crush -▁crit -▁uncertain -reh -▁compreh -▁interpret -▁emph -her -▁parli -▁parliament -▁hans -▁quality -▁listened -▁explained -▁singing -▁experiment -▁multitude -▁compell -▁compass -▁ninety -▁desper -▁squire -▁warri -▁appoint -▁permission -▁scrap -▁prim -▁occasionally -▁laws -right -itable -▁circumstance -erchief -kerchief -▁handkerchief -▁straw -▁melancholy -▁destr -▁destruction -▁ugly -▁frequent -▁neighbourhood -▁refresh -▁rome -▁ros -osing -▁hastened -▁acquired -▁imper -▁impul -▁virginia -▁afford -▁sport -▁hale -▁haunt -erted -▁deserted -▁desires -othe -▁plat -▁platform -▁plans -▁kat -▁receiving -rain -▁chest -▁execut -▁hut -▁honey -▁repeat -▁confident -▁reading -▁tit -▁shone -▁everyone -▁sovereign -▁sank -▁sen -▁join -aced -▁succeeded -▁cloak -▁trif -▁weeks -▁likely -ingly -▁carrying -▁assert -▁vigor -▁absor -▁absorb -▁sin -▁careless -▁description -▁tin -▁tiny -▁tied -▁sang -▁secur -▁circl -▁rate -▁warning -▁limit -▁limb -▁reb -▁resolut -▁applied -▁stoop -ulated -▁hate -▁arrive -▁scient -▁scientific -▁tristr -▁tristram -▁sam -▁pear -▁stepped -▁id -▁solid -▁relation -▁feelings -ught -▁florence -▁napole -▁thom -▁thomas -▁instit -▁swear -▁carpet -acles -▁avoid -▁opening -▁ren -▁rabbit -▁sentence -ground -aked -a's -▁accordingly -▁constantly -▁characteristic -▁ident -▁anywhere -▁solitary -▁mercy -▁abund -▁narr -▁nich -▁managed -▁strip -▁hopeless -▁conquer -▁lieut -▁lieuten -▁lieutenant -▁forces -▁reduc -▁hopes -▁undert -pr -▁enterpr -▁terms -▁inquired -▁becoming -▁seriously -▁dismiss -ington -▁washington -▁mother's -▁threatened -▁sunsh -▁sunshine -▁irre -aries -▁gay -▁cattle -▁hamm -▁tramp -▁eagerly -▁formed -▁signific -▁knows -▁likewise -▁monkey -▁gown -▁misery -burg -▁pounds -▁families -▁entre -▁august -▁irr -▁irish -▁irres -▁spend -▁descend -▁literature -▁purple -▁ble -▁ken -▁greatly -▁sink -▁resign -▁demonstr -here -▁valuable -▁dw -▁dash -▁monarch -ew -▁renew -▁ende -▁endeavour -▁endeavor -esh -▁flesh -als -▁national -▁walter -▁glimp -▁giant -▁unpleasant -▁modest -▁mud -▁rust -▁prep -▁tal -▁unw -▁shouted -▁unus -▁unusual -▁spiritual -▁actual -▁mentioned -▁brand -▁branch -▁critic -uf -▁yonder -▁preserve -▁nerve -▁baron -▁degrees -aded -▁persuaded -▁exceed -▁talked -▁possessed -▁plung -▁ain't -atory -▁puzz -▁pil -▁attain -iform -▁uniform -▁failure -▁failed -▁assure -▁mich -▁mut -▁counsel -▁pink -▁appe -▁appeal -cknowled -▁ell -▁craft -▁demanded -cknowledge -▁publ -▁doubtless -oo -oe -oil -▁spoil -▁fro -port -▁tyr -▁tale -▁medic -▁flatter -▁politic -▁mystery -▁energy -▁pig -▁refuge -▁scorn -▁hills -▁fastened -▁jimm -▁activity -▁clothing -▁quiver -▁quant -▁quantity -▁eighty -▁gasp -udience -▁obsc -ifully -▁nav -ox -▁furnished -▁pretended -▁interf -▁protection -▁qual -▁qualities -▁que -▁queer -▁vex -▁title -▁difficulties -▁accus -utumn -▁startled -▁drank -▁dragon -▁regiment -▁emerg -▁candle -▁procl -▁sincere -▁representative -▁whither -▁scotland -▁market -▁raising -▁separate -▁separated -▁rad -▁ragg -▁central -▁blaz -▁gardens -com -omfort -▁outer -▁nichol -▁nicholas -▁invol -▁invitation -▁apartment -▁disappointment -▁daughters -▁robin -▁admin -▁administr -▁falling -▁idiot -ators -▁tort -▁nut -▁flame -▁flag -vell -▁retort -▁successful -▁ment -▁mental -▁current -fall -▁function -▁pipe -▁capacity -▁preserved -▁contempl -▁gaze -▁suffering -▁leading -▁disgu -▁survey -uchess -ifest -▁manifest -▁haste -▁finish -▁veil -▁disgr -▁disgrace -▁condemned -▁decision -▁civilization -▁cele -▁celebr -▁plainly -▁finding -▁tables -ilities -▁we've -unning -▁inquir -▁witch -▁trembling -▁retreat -▁craw -▁universal -▁jer -▁prud -▁pin -▁jim -▁billy -anny -▁richard -▁interesting -▁presum -▁permit -▁benefit -▁portra -athing -▁ignorance -▁earlier -▁union -illy -▁opinions -▁num -▁numer -▁numerous -▁nest -▁torn -▁lucy -▁population -lies -▁jewels -▁novel -▁virtues -▁rh -▁ash -▁desk -▁weakness -▁silly -ois -▁streets -▁ruin -▁patch -▁scratch -usk -kind -▁mankind -▁vivid -▁snatch -▁mont -▁monte -▁et -▁fulf -acc -▁wrapped -▁reput -▁repair -▁lodg -stairs -▁flutter -▁maxim -▁charming -▁shield -step -▁shade -▁local -▁dogs -▁impro -▁closely -▁stamp -▁consequences -▁telegraph -▁breathing -rell -▁brow -▁cunning -▁cav -▁congress -▁extremely -▁discip -▁ze -▁fee -▁feeble -▁ward -ture -▁mixture -▁jeff -▁intent -lly -▁wholly -▁shell -▁convict -lict -▁probable -▁caval -▁poem -▁pover -▁poverty -▁driver -▁shirt -▁ruth -▁conflict -▁maybe -upon -▁exceedingly -▁anxiously -▁glor -▁glorious -▁coward -▁im -▁image -▁palm -mith -orm -▁enorm -▁enormous -▁woods -▁wooden -▁frown -▁base -▁bond -▁unable -iable -▁prize -▁tremend -▁fortnight -▁tradition -▁articles -▁article -▁haw -▁peep -▁slender -▁proposition -▁declare -▁prepare -▁mirr -▁painful -▁relieved -▁supply -▁tower -▁manage -▁harsh -iday -▁holiday -▁desperate -en't -▁chris -▁observation -ehead -▁forehead -▁causes -▁reality -▁rachel -▁moonlight -▁commenced -▁involunt -▁katy -cy -▁stirred -▁indians -▁brook -▁uncomfort -▁jean -riek -▁shriek -▁arist -▁aristocr -▁twist -▁matters -▁reference -▁indifference -bury -coat -▁bones -▁refused -▁refuse -▁affirm -▁furnit -▁furniture -▁imperfect -▁thrill -▁cutting -▁marvell -▁marsh -▁thief -▁hy -▁hint -orthos -osom -▁bosom -▁examination -▁development -▁hoped -uk -▁monk -ecture -▁dwar -▁dwarf -reck -▁awoke -▁ech -▁madam -▁echo -▁ring -▁inj -▁happily -▁attribut -▁scarlet -▁accustomed -▁applic -▁await -▁pit -rived -▁hearing -▁knowing -▁joe -▁job -bly -▁doctr -▁instruments -▁instruct -▁construct -▁suffered -▁secured -▁pearl -riet -▁propriet -get -▁veget -▁generous -▁slightest -▁slightly -▁lev -▁revenge -▁stout -▁san -▁mighty -▁he's -▁what's -▁atmosphere -▁lat -more -▁rude -▁opposed -▁lightning -▁increased -▁suggested -▁stretched -▁arranged -▁liqu -▁natash -▁ing -▁cock -▁rage -▁sar -▁marvel -icip -▁anticip -▁tip -▁vic -▁shake -▁kindly -▁cease -▁ban -▁bath -▁cath -▁cathol -▁catholic -▁batter -▁cities -incol -incoln -▁lincoln -▁lest -▁wishes -▁proclaim -▁prisoners -▁ordered -▁transport -peare -akespeare -▁strangers -▁rout -▁railroad -▁reputation -▁louise -ountain -▁fountain -osen -▁chosen -osh -▁fairly -▁escaped -▁puzzled -ufact -▁overcome -avery -▁flying -▁hadn't -▁boast -▁contrast -▁marble -▁cries -▁rear -born -▁pictures -▁advantages -▁hatred -▁tun -▁supreme -▁gest -▁gesture -▁ham -▁pup -▁poetry -▁melt -▁dwelt -▁asham -▁ashamed -▁rushed -▁neighbors -voided -▁extern -▁buff -▁amid -▁warriors -uary -tic -athetic -▁devotion -▁sau -▁grie -▁smart -▁heads -▁orders -▁frozen -▁merit -▁spear -▁slavery -▁slip -▁june -▁reform -▁penc -▁wounded -▁mirror -▁rib -▁refer -▁border -▁disorder -▁displeas -▁begged -▁duchess -▁remote -ernard -▁bernard -▁ma' -▁pir -▁bedroom -▁wreck -▁informed -▁recognize -▁turkey -▁tax -▁keeping -ening -len -▁helen -▁wives -▁ard -▁tend -▁sooner -▁worry -▁spirits -▁complaint -▁suppress -ifth -▁fifth -▁illustr -▁rog -▁debt -▁sult -▁sof -▁inclination -▁determination -▁fatig -▁medicine -▁played -▁motionless -▁eternal -▁looks -▁rocks -▁readily -▁emily -▁sensation -▁sensitive -▁senses -ades -▁comrades -▁comrade -▁ma'am -aham -ibly -▁fatal -▁pupil -▁nodded -▁whence -▁od -▁sultan -▁mourn -▁mild -▁exquis -▁published -▁wrath -▁trap -▁timid -▁members -▁exquisite -▁encourage -▁dusk -▁clerg -bb -▁bree -▁breeze -▁operation -▁reluct -▁wagon -▁won -▁utterly -▁fields -▁brothers -ters -▁conditions -lter -▁shelter -▁emm -▁waves -▁picked -▁delighted -kin -▁apparent -▁fil -▁anti -▁doroth -▁penet -▁penetr -▁appet -▁appetite -▁ladder -antly -▁excellency -▁rif -ued -▁community -▁parts -▁spell -▁changes -▁absent -▁fighting -izing -▁listening -▁tones -▁tis -▁steel -▁attent -▁waved -▁wax -▁beaten -▁helpless -▁portrait -▁breathe -▁footstep -imney -▁chimney -▁domin -▁funny -▁staff -▁sauce -▁dutch -▁adopted -▁note -▁proceeded -bridge -▁detective -▁swell -▁swung -'n -jud -▁prejud -aud -▁expressed -▁shift -▁obser -▁features -xic -▁glanced -ology -ological -▁previously -▁bestow -▁stars -▁deeply -▁joyou -ipped -▁shiver -▁bushes -▁railway -▁troubles -liness -▁albert -▁telling -▁calling -▁pushed -▁venge -▁vengeance -▁related -▁incred -▁dancing -▁plate -▁mouse -house -▁troubled -wild -▁simplic -▁industry -▁ruined -▁leis -▁leisure -▁decide -▁treatment -▁crimson -▁johnson -▁hanging -▁bargain -▁effic -▁ridic -▁ridiculous -▁lodge -▁pool -zard -azz -▁ceter -cknowledged -▁simon -▁whisk -▁whip -▁kissed -▁tremendous -▁shakespeare -▁frog -▁steal -▁solitude -▁elsewhere -▁impulse -place -▁hook -▁accepted -▁conception -▁horizon -estern -▁western -▁altern -▁wake -▁approached -▁neat -▁ere -▁fund -▁created -▁schol -▁net -▁weary -▁lawy -▁satisfactory -▁peasant -istol -▁pistol -▁polly -▁porthos -▁gates -▁final -▁zeal -ovitch -▁consol -▁gallery -▁approaching -ambl -irty -▁dirty -▁unexpected -▁limited -▁exert -▁naked -▁romantic -▁penny -▁russian -▁edith -ered -▁fer -▁slipped -▁regarded -▁concerned -▁eggs -▁sore -▁flock -▁mock -agre -▁disagre -▁cheap -▁intimate -▁tomorrow -itor -▁holl -▁selfish -▁favorite -▁easier -bour -▁contrad -▁lift -▁thresh -▁threshold -▁imprison -▁physician -▁surprising -▁mutual -▁gloomy -inal -▁cardinal -▁criminal -▁lately -▁unfortunately -▁throng -▁romans -▁nail -▁remove -ule -chie -▁slaves -cel -▁precip -▁dian -▁diana -avern -▁compliment -▁range -▁grate -▁grateful -imate -▁depress -▁wretched -▁troops -▁creatures -▁furnish -▁propos -▁lawyer -erem -▁cerem -▁ceremony -▁agony -▁region -▁ult -▁suggestion -▁americans -▁restrain -▁bul -▁expense -▁oath -▁rum -▁alm -▁unl -▁neighborhood -▁childhood -▁advised -▁pressure -▁pressed -▁torture -▁ragged -▁rejoined -▁politeness -▁estate -▁chees -▁beautifully -▁spectacle -▁striking -▁exception -▁floating -▁anguish -▁province -▁henri -▁exposed -▁resumed -▁crossed -▁requires -▁tex -▁firmly -▁friendly -▁struggling -▁nations -▁holland -▁satisfy -▁perse -▁glimpse -▁interested -mber -▁brid -▁copp -▁bon -▁summon -▁legis -▁legisl -▁profit -▁boots -▁established -▁crying -iece -fric -▁completed -▁trace -▁roses -▁rival -▁jac -▁jacob -▁eaten -▁eastern -▁reserve -▁blew -▁page -▁undou -▁teleph -▁obvious -▁purchas -▁purchase -▁susan -elia -▁majority -▁gallop -▁millions -isions -▁provisions -▁series -ofty -▁rag -van -▁continent -▁cheeks -▁sarah -▁quoth -▁impatiently -▁impatience -▁impatient -▁convenient -▁fired -▁wire -▁emphas -▁independent -▁meat -▁fasc -▁creek -inction -▁distinction -▁section -▁shouldn't -whel -▁contents -▁contact -▁mortal -▁discussion -▁infant -eem -▁esteem -▁bade -▁exped -▁expedition -▁playing -ding -edding -▁wedding -▁working -▁asking -chan -▁mechan -▁cam -▁growth -▁oil -▁privilege -▁bol -action -▁convinced -▁devoted -▁ships -▁ins -▁insist -▁inspect -▁insult -▁clouds -▁brandon -▁barn -line -▁collar -▁kick -wick -▁obviously -▁lofty -▁lion -▁dart -▁abst -▁abstract -▁resc -▁rescue -used -▁confused -▁ca -▁humour -▁gazed -▁parties -▁dram -▁drag -▁ahead -▁jackson -▁wilson -▁engagement -▁bench -▁transf -▁jumped -▁pointed -▁stroke -▁sharply -▁pron -▁pronounc -ropri -▁appropri -pped -whelm -▁asc -▁ascertain -ained -▁dining -▁indiffer -▁indifferent -▁tempt -▁slain -▁terribly -▁der -▁derived -▁gro -▁groan -▁obtained -▁farmer -▁pierc -▁crist -▁cristo -▁rod -▁wherever -▁lap -▁chann -▁china -▁thine -▁examine -▁cetera -rid -▁corrid -pired -▁practically -▁thanks -▁intelligent -▁fires -▁properly -▁disagree -▁nurs -▁announced -▁anyhow -▁aren't -▁comparative -▁margu -ixote -▁quixote -▁gil -▁gilbert -▁surv -▁unm -mon -▁supplied -▁supposing -▁realize -▁garments -▁passengers -▁charged -▁indulg -▁cig -▁behav -▁waste -▁congr -▁conceived -▁deceived -▁carol -▁secretary -▁essence -▁ecc -▁eve -oisell -▁swimm -▁awk -▁awkward -▁gracious -engthen -ef -▁fatigue -▁elements -▁eleg -▁elev -▁election -▁element -▁preced -▁precaut -▁apple -▁ple -▁rifle -▁marked -▁uttered -▁employed -▁weap -▁weapons -▁universe -▁sne -▁sancho -▁oct -▁persuade -▁signal -▁roast -▁stole -▁sty -▁style -▁stuck -▁chuck -▁stopping -yn -▁preferred -▁deem -▁breathed -▁terrified -▁gained -▁ber -▁prayer -▁inhabitants -▁concluded -▁repent -▁isabel -▁yards -▁fortunate -▁fortunately -▁production -▁reject -▁innocence -▁temperature -▁stat -▁congrat -▁humanity -▁security -quet -ara -aly -arse -ural -▁external -▁assur -▁unwill -▁dismay -▁napoleon -▁commer -▁commercial -▁grove -▁barrel -▁backward -▁salut -▁comparison -▁instances -go -foot -▁literary -▁skirt -encies -uries -▁centuries -▁images -▁intend -▁pian -▁piano -▁caress -▁richmond -▁connected -vail -osure -▁depos -▁remembr -sem -sembly -▁despite -▁favourite -▁authorities -▁matthe -▁matthew -▁veh -▁vel -▁velve -▁forms -▁dared -▁genu -▁genuine -ulgar -▁pul -▁overwhelm -▁diss -▁endure -▁peasants -▁afflict -▁volume -▁transform -▁december -▁pole -▁presented -lin -▁wither -▁believing -▁misfortune -▁sprink -▁acquainted -▁advert -▁sisters -▁observing -▁distract -▁pony -quire -▁described -▁describe -▁mode -▁knee -▁pirates -▁fury -▁injury -▁chose -▁prose -▁recovered -▁specially -▁visitors -▁lifting -▁dropping -▁ingen -▁narrative -▁standard -▁interior -ember -ptember -▁september -▁wept -ption -▁division -▁contribut -▁you'd -▁undoub -▁undoubted -low -▁pillow -▁industri -▁serge -▁serp -▁muttered -▁marian -▁lin -▁linen -omen -enomen -▁phenomen -▁donkey -▁budd -▁tenderness -▁fitted -▁valj -▁grav -▁francis -▁hunger -▁ones -▁rat -▁arrangement -▁propose -▁peril -▁dan -▁friday -vember -▁violently -▁ways -▁uns -▁unre -▁captured -▁injured -▁announce -▁tomb -▁precisely -▁needed -▁eldest -▁income -▁restor -▁decor -▁ability -▁introduced -▁trump -▁kenned -▁pretend -▁bowed -▁immort -▁nancy -▁ashore -▁commerce -▁encl -▁timber -▁supplies -▁lament -▁frightful -▁hit -▁fulfill -▁summit -▁glove -▁vac -▁borrow -▁purse -▁clergy -▁deny -▁perpetual -ulate -▁uncomm -▁residence -▁providence -▁cream -▁tar -add -▁newspaper -▁newspapers -icken -▁raoul -▁mounted -umbled -ella -▁compelled -▁phot -▁photograph -▁cheese -▁bearing -▁belt -▁rolled -▁addressed -▁surrounded -▁feed -▁strok -▁jes -▁jesus -▁luxury -▁losing -▁lan -▁lantern -▁introduce -▁composition -▁louder -▁studied -▁waistcoat -▁leaning -▁plum -▁scheme -▁naught -cr -▁brick -▁sailors -▁sailor -▁spani -▁spaniard -▁clay -phew -▁nephew -alph -▁ralph -pher -pherd -▁alt -▁folly -urday -▁saturday -▁reckless -▁pace -uncel -▁launcel -▁laur -▁gallant -▁convention -chester -▁dreams -▁alexand -▁lyn -imony -▁launcelot -umbrell -▁umbrell -▁ur -▁experiences -▁phr -▁phrase -▁carri -▁manufact -▁understanding -▁sull -▁vulgar -▁unlike -▁extract -etic -▁aspect -▁hush -▁inspir -▁elb -▁returning -▁embraced -▁embrace -▁contradict -▁recommend -▁inspired -▁monstr -▁weapon -itter -▁glitter -olet -▁violet -▁duck -▁pluck -▁champ -apping -▁arguments -▁july -▁affected -▁preparations -▁reception -▁someone -▁deserved -▁desol -▁cautiously -▁elbow -▁kate -▁heat -▁hell -▁damp -▁defend -▁riding -▁encouraged -▁accur -▁efforts -▁score -▁resour -omach -▁stomach -▁treach -▁rules -▁guilt -▁induc -▁india -▁batt -▁fearful -▁embro -rog -isp -isk -▁brisk -▁musk -▁brut -▁brig -▁rig -▁constitut -wildered -▁safely -▁aboard -▁responsibility -▁scattered -▁musc -▁muscular -case -▁staircase -▁chase -chieve -▁wondered -▁prudence -ili -▁humili -▁punished -▁princes -▁angri -▁assembl -▁cryst -▁borne -▁objection -▁objects -▁disagreeable -▁flames -▁confined -▁denied -▁snake -look -moisell -▁concent -▁concentr -▁sighed -▁investig -▁sway -▁doorway -fore -▁glob -▁appears -ites -▁ages -oster -▁porter -▁porch -▁rebell -cast -▁faded -▁beard -ease -▁disease -▁sunlight -▁pilgr -▁awaited -▁amusement -▁seal -▁mademoisell -▁waver -▁resent -▁escort -ourse -▁indignation -▁citizen -▁doctrine -adow -▁tread -▁tad -ren -▁restless -becc -shire -aire -▁wants -▁bert -▁rebecc -▁revel -bur -burgh -▁institut -▁coarse -ighth -▁eighth -▁weigh -▁vest -▁faced -▁woman's -▁compet -▁worked -▁perceiving -▁celebrated -▁dig -▁bundle -▁resistance -▁tendency -▁sunk -▁highly -▁partly -▁morgan -▁cyr -▁discipline -▁bigg -▁telephone -▁prodig -▁artific -▁daylight -▁accompany -▁reserved -▁seize -▁sees -▁directed -▁exagger -▁ench -▁enchant -▁joke -▁leaned -▁julius -▁interfere -▁sentiments -▁gods -▁doors -▁chill -▁curiously -▁delicious -▁ruins -▁multip -▁refrain -▁indicated -▁locked -▁array -onday -▁sust -▁probability -▁glac -▁cos -▁cosette -▁mos -▁notwithstand -▁wheat -▁humor -▁swiftly -▁rigid -▁slid -▁introduction -▁musical -▁fog -▁fir -quirrel -▁wherefore -▁housekeep -▁enterprise -▁scenes -▁waving -bor -bacc -abb -▁patron -▁independence -▁shepherd -▁frost -sey -bacco -▁assurance -▁slaughter -▁balloon -oonful -fulness -▁anna -▁abruptly -oqu -▁eloqu -▁hideous -▁infer -▁inferior -▁embroid -win -inated -▁mask -▁mould -▁remains -▁moments -▁insect -▁johnny -▁referred -▁georg -▁decre -reland -▁tick -▁upstairs -omb -lars -▁danglars -▁pilgrim -▁horrid -▁specim -▁wilt -edge -▁destroyed -▁burg -verage -▁journ -▁institution -▁devour -▁struct -▁structure -▁stagger -▁communication -ka -▁grain -▁simplicity -atural -▁entreat -▁saving -▁points -▁bor -▁tempor -▁adjust -▁unjust -▁deprived -▁contrived -▁philosophy -worthy -inate -itate -▁adorn -▁dried -▁mission -▁britain -▁examining -▁chok -▁cher -▁marguer -▁marguerite -▁struggled -▁worm -▁rendered -▁patri -▁bitterly -▁steph -▁stephen -force -▁one's -▁pepper -▁knelt -▁breaking -▁velvet -▁slumber -▁hated -▁admire -nament -▁ornament -ilight -▁twilight -bled -▁twink -▁patiently -▁wet -▁jones -▁furious -onies -▁gazing -▁operations -▁pursued -▁cred -▁credit -▁classes -▁arrangements -▁harbour -▁cotton -▁naut -▁administration -▁channel -▁closer -▁kneel -▁moist -▁guests -ensible -▁assault -▁assign -asper -▁sensations -lor -▁stricken -aps -▁eagle -ique -▁fel -▁intentions -▁errand -▁error -▁administer -▁artist -▁heartily -▁pencil -▁tide -▁messenger -▁leader -▁retire -▁agitation -▁ali -▁villain -iscilla -▁whereupon -▁passions -▁disguise -▁seas -▁obscure -▁remonstr -alian -▁italian -▁speedily -▁conceive -ape -natural -▁spur -▁spain -▁invis -▁invisible -ceptible -▁throwing -habited -▁comment -▁puff -▁hoarse -▁parlour -▁torment -▁suck -▁incomp -▁incon -▁inconven -▁wolves -▁bloss -▁blossom -otted -▁tremble -▁testimony -ugh -▁stared -▁badly -▁coron -▁coal -agues -▁primitive -evitable -ribution -ribute -lyosh -lyosha -▁accomplished -itory -▁territory -umbling -▁arise -▁turtle -▁apost -▁apron -▁mc -ces -▁resources -▁male -▁moss -athedral -ovna -▁apprehension -▁machin -▁machinery -▁bidd -▁hero -▁floated -▁ribb -nes -azy -▁uneasy -▁dorothy -▁notwithstanding -▁respectable -▁emot -▁emotions -▁rejoiced -▁steward -▁upward -▁fred -▁freder -▁frederick -▁natasha -▁sofa -▁drum -▁launch -▁disch -ched -▁attached -itating -▁uncomfortable -ugged -ashed -▁kitty -▁statement -▁enjoyment -▁indisp -▁jolly -▁shadows -rov -▁alexandrov -▁pleaded -▁plead -▁attorn -▁attorney -▁generosity -▁norman -yard -▁entering -▁conject -aws -▁lav -▁cheerfully -▁regain -▁forever -▁department -ement -▁amazement -▁climbed -semble -▁editor -▁hay -▁wrink -▁wrought -▁breathless -▁treas -▁combat -▁combined -▁reader -ering -▁cler -▁clerk -▁writer -erable -▁eleph -▁stalk -▁harmony -▁bringing -uring -▁couch -▁contained -▁hatch -▁geograph -▁sem -▁corrupt -▁hood -▁brood -▁hugh -▁chiefly -▁distinctly -▁sadly -▁formerly -▁finest -▁procure -▁ashes -▁dimin -▁chin -▁emin -▁annie -▁irresist -atre -▁asserted -can -▁volcan -▁harr -ring -▁stirring -rav -▁extrav -▁extravag -▁risen -▁motor -▁trembled -ais -▁parad -▁paradise -▁advise -▁lands -▁landsc -▁landscape -▁obed -▁obedience -▁likeness -▁dragged -▁apples -smith -▁thumb -▁ridge -▁graham -▁seren -▁departed -sy -sych -▁psych -rey -▁inquiry -▁blown -▁pond -▁cannon -▁eyed -▁democratic -▁access -▁exchange -▁dale -▁sale -▁review -▁bust -▁worried -▁tub -▁slope -▁circular -▁cake -▁assent -▁slop -eg -▁dangers -▁prayers -▁stronger -▁delicacy -owl -▁fowl -vol -▁benevol -▁coffin -▁monday -▁magician -▁experiments -▁responsible -▁serpent -▁construction -▁valje -▁valjean -▁interrupt -▁downstairs -▁equality -▁prime -▁chall -▁earnestly -▁repose -▁exhausted -▁medium -▁medical -▁agent -▁slee -▁stroll -▁serving -▁jap -▁japan -▁visitor -▁scrup -▁threatening -▁despatch -▁carr -▁fanny -▁ceil -▁haught -▁hail -▁hare -▁hamp -▁nautil -▁discharg -▁discharge -▁agric -▁agricult -▁nautilus -▁reflected -▁painted -▁peak -▁fame -▁jerk -▁prosperity -rant -rous -▁durham -▁adventur -▁fellows -▁bind -▁bid -rig -▁showing -▁copy -▁mademoiselle -▁homes -▁bolt -▁forb -▁forbidden -▁mexic -▁meadow -▁candid -▁solic -▁solom -▁solomon -▁warned -▁planned -▁deliberately -▁droop -▁cru -▁god's -▁weeping -▁bonn -▁dil -▁soil -▁fogg -▁observations -▁details -▁rays -▁butterf -▁punct -▁defence -▁suitable -▁perfection -▁speck -▁defe -▁establishment -▁savages -▁deaf -▁forgetting -▁hypn -▁hypnot -▁interposed -▁ambass -▁balance -▁recognised -▁recognition -▁burnt -▁burned -▁rip -▁hypothe -▁hypocr -▁poured -▁insisted -▁vanity -▁charity -▁charl -▁charlott -aine -▁earliest -▁englishman -▁confound -▁priscilla -▁knocked -▁pine -▁ponder -▁crazy -▁dislike -▁plague -umult -▁tumult -▁bald -▁lessons -back -▁robb -▁joint -▁subtle -▁subs -▁subd -▁subdued -rence -▁cub -▁fisherman -▁leagues -▁promises -▁mail -sion -▁compassion -▁procession -▁complexion -▁improved -▁bone -▁smoking -justice -▁machines -▁reduced -▁succession -esc -▁condesc -▁crouch -▁crook -▁patriot -▁signs -▁shrink -phia -▁russia -▁polished -▁behaved -▁cruelty -apt -▁bapt -▁caution -▁superint -▁alight -course -▁intercourse -▁pred -▁prair -▁felic -▁regions -▁jury -▁lust -▁distrust -▁lane -▁revolver -▁liver -▁magistrate -▁knights -▁wip -▁elegant -▁educated -▁columb -▁dumb -▁dazz -▁ambassad -▁conclude -▁complic -▁arous -▁rous -▁torr -▁appr -▁approve -▁apply -▁appl -▁angrily -▁scrambl -▁sobb -▁persec -▁irreg -▁irregular -▁resolute -▁enthusiastic -▁gives -▁performance -▁audience -▁romance -▁remembrance -ram -▁marius -▁sergeant -▁bodily -▁vigorous -uster -▁metal -▁amused -▁goat -▁cousins -▁perman -▁artificial -▁let's -▁prosec -after -▁hull -▁prey -▁abandoned -▁helped -▁disturbed -▁draught -▁improvement -▁hedge -▁soup -▁merchants -▁halted -▁martin -pic -picuous -▁conspicuous -▁rav -▁copper -udding -▁knot -▁trot -▁traged -venue -▁sounds -▁damsel -▁sack -▁dignified -iency -▁induce -▁facts -▁hearts -▁destined -▁destin -▁patter -▁chatter -▁pant -▁assistant -▁pall -▁works -▁figures -▁radiant -▁belonged -▁guns -▁terrif -▁resort -▁sanct -▁rect -▁swarm -▁behave -▁mac -▁culture -▁graceful -iful -▁assail -▁anyway -▁recently -▁micha -▁michael -▁journal -▁pope -▁inex -ulating -▁squee -▁ratt -▁thence -▁memories -▁spectacles -▁thyself -cell -aker -▁promptly -▁contest -▁folks -▁observer -▁detain -▁deed -▁defect -▁woe -▁planet -▁navig -▁lemon -▁undertake -▁fortunes -tor -▁histor -▁involved -▁throb -▁altar -▁exhibited -▁exhibit -▁miracle -▁cavern -▁strengthen -▁wandered -▁wandering -▁ton -onc -▁lesson -▁prost -anqu -▁tranqu -▁tent -▁submit -▁abode -▁provision -▁provide -▁organization -▁wondering -▁sullen -▁pursue -▁germany -▁tales -▁scale -▁assumed -▁fundament -▁captive -▁glacier -▁jest -▁jen -▁jason -▁hiding -▁legislature -▁strongly -▁freely -▁moderate -▁suspended -▁colors -▁delivered -▁impart -wart -▁rooms -▁solution -▁employment -▁ecst -▁compared -umble -▁yo -▁suspected -abled -▁enabled -▁enl -▁quarters -▁generation -▁largest -▁strait -itud -itudes -▁plot -▁scare -ourning -▁mourning -wers -▁corners -▁weren't -▁pac -▁pacific -▁discl -cles -▁muscles -▁extensive -▁extend -▁perf -▁perfum -▁indef -▁indefinite -▁transl -▁suited -▁corpse -▁corpor -▁defeat -▁declined -ora -▁emma -▁literally -▁conclus -▁adj -▁relieve -▁mixed -▁cavalry -▁recent -▁tang -▁swamp -▁monstrous -▁winds -▁pile -▁limp -▁fragments -▁sher -▁sheriff -▁shaw -▁shor -▁shagg -▁shoe -▁bother -▁bible -▁prem -▁clut -▁tore -▁turk -ky -▁occupation -▁funer -▁funeral -▁concealed -dr -▁withdr -▁withdraw -raham -▁abraham -▁mosc -▁moscow -▁vital -ulliver -▁snapped -▁convince -▁convul -▁older -▁younger -▁potat -▁potatoes -▁faster -▁lanyard -▁insol -▁gw -▁herbert -▁rasc -sued -▁follows -▁resol -▁confession -▁lecture -▁reasonable -▁beggar -▁stayed -▁reconc -▁quot -▁quit -sitated -▁hesitated -▁agitated -▁interests -▁banks -▁shaggy -▁arriving -▁superintend -▁inflict -▁transpar -▁scott -▁magg -▁maggie -▁woke -▁impressions -▁overlook -▁criticism -▁unb -▁absorbed -▁speeches -▁unkind -▁situated -▁feared -▁printed -▁print -alt -▁shalt -ft -▁sympt -▁symptom -▁applause -▁treasury -▁undoubtedly -▁distribut -▁lordship -▁senator -▁undertaking -▁patsy -▁sleeping -▁staring -▁seeking -▁chicken -▁chances -aign -▁campaign -▁yourselve -▁stands -▁inquiries -▁apartments -olen -▁stolen -▁stable -▁stove -▁intelligible -▁incredible -▁adopt -▁button -▁cush -ird -▁bud -▁hawk -▁discourse -▁verse -berry -▁intens -ensions -▁combination -▁goodness -▁nursery -▁gratify -▁application -▁drake -▁he'll -▁sparg -▁granted -▁commanded -▁fled -▁levin -inations -▁eust -▁eustace -▁leaped -▁squirrel -▁sire -▁illum -▁persever -▁incap -▁envelop -▁cellar -▁slew -▁descent -▁unse -sequ -▁subsequ -▁minor -▁sketch -▁exclam -amed -▁sufferings -▁jealousy -▁holes -▁ju -▁parish -▁vault -▁remainder -▁battles -▁conj -▁owe -▁institutions -▁rou -▁affectionate -▁sunny -▁flee -▁fence -▁cigare -▁cigar -▁jar -▁jan -▁january -bruary -▁february -▁hencefor -▁venus -▁lily -igantic -▁gigantic -▁cordial -▁martyr -borough -▁recollection -▁kennedy -frica -▁contemptuous -▁virtuous -▁unreason -velope -▁dispute -▁bost -▁boston -▁bending -▁hostile -▁hound -▁letting -leness -▁landl -▁landlord -▁plants -▁carved -▁syn -▁hoof -▁antiqu -▁latin -▁proposal -▁gravely -eous -inous -▁goose -▁wink -▁enjoyed -▁occupy -acon -▁foul -▁advertise -▁tempted -▁script -▁archit -▁architecture -▁substit -▁smaller -▁veins -▁brass -▁surpass -▁parcel -▁guided -rage -▁center -rod -▁detach -▁beach -▁shill -▁sew -▁sympathetic -▁hunted -▁jewel -▁horseback -▁shaft -▁spargo -▁superst -▁crest -iter -▁teacher -▁fant -▁fantastic -leton -▁provok -▁cure -▁represented -▁ended -▁jail -ait -▁apr -▁april -▁skull -ikov -▁commander -▁tyranny -auvel -▁resolve -▁rolling -▁jimmy -▁jimmie -set -▁sunset -▁increasing -▁shores -▁recommended -ache -necess -▁unnecess -▁unacc -▁armed -alog -▁shooting -▁agnes -cules -▁embark -▁prow -▁capture -▁falsehood -▁calmly -▁gravity -inity -▁assass -▁blast -▁pinoc -▁pinocch -▁pinocchio -▁suspicious -itz -▁decay -▁chariot -idents -▁urged -▁prophet -oh -araoh -▁pharaoh -▁perception -▁warmth -▁elean -▁unnecessary -▁murderer -▁ger -▁gal -▁gang -ange -▁orange -▁gor -▁gorge -▁gulf -▁gas -geon -alty -▁betty -▁eleanor -▁liberal -▁tap -▁interm -▁danced -iding -▁civilized -▁raggedy -▁coloured -▁remaining -▁clergyman -▁plunged -▁prick -lfare -▁samuel -▁breton -▁breed -▁decent -▁depths -▁wail -▁likes -▁specific -▁visited -▁sons -thren -▁brethren -▁liquid -▁clover -▁economic -▁wag -▁inspector -▁crowded -▁closet -▁antagon -▁shortly -▁conjecture -▁mathe -▁uneasiness -▁vote -▁smote -▁irresistible -▁inexp -▁temptation -▁enlight -utom -▁surgeon -▁rent -▁exult -▁simult -▁bonnet -▁scholar -▁cherished -▁meal -▁worsh -▁sleeve -▁whoever -▁grin -▁valent -▁valentine -▁autumn -▁til -▁consumption -▁clothed -▁they've -▁relax -▁colored -▁fawn -▁intently -▁amiable -owed -▁barri -▁shrew -▁shrewd -▁shaken -▁appearances -▁scarec -▁scarecrow -▁venerable -drew -▁washed -▁francs -▁rud -▁exclus -about -▁ot -▁cathedral -atern -▁condemn -▁intern -▁corridor -▁simultane -▁picturesque -▁string -▁bowl -▁repl -▁replaced -▁imperial -▁inn -▁penn -▁lawn -▁provinc -▁amidst -▁ireland -▁transfer -ulations -▁generations -▁pave -▁pavement -▁appointment -▁spectators -▁link -equ -▁assume -▁roused -▁unusually -▁guardian -▁overhead -ucc -▁grind -▁partner -▁milton -▁sym -▁symb -▁symbol -▁crumb -▁crash -▁raft -▁collection -uffled -▁exercised -▁sounded -▁nigh -▁preparing -▁preparation -▁opportunities -▁swim -▁involuntarily -▁exercises -▁sparkling -▁permanent -▁scent -▁dew -▁retorted -anz -▁franz -▁perceval -▁students -▁explor -▁explos -▁mule -iting -▁exciting -▁tongues -▁substant -▁substances -▁hoop -▁lighted -▁pronounced -atically -▁niece -▁non -▁stores -▁travellers -iel -▁daniel -ih -izz -▁viz -izes -▁vot -▁reass -▁dost -ridge -▁issued -▁merri -▁avail -▁italy -▁stray -umm -▁conform -▁dismount -▁envelope -▁scold -▁incapable -▁esther -burn -▁recess -▁forgiven -▁norm -▁normal -▁legal -▁slay -▁circuit -▁drift -▁luxur -▁candles -oid -▁pilot -▁warrior -▁griev -▁grieved -▁eternity -▁polic -▁praised -▁entit -▁entitled -▁furiously -▁intimacy -▁reverence -▁groom -▁resemblance -▁temperament -▁exhibition -▁advoc -▁idol -▁uncommon -▁comprehend -▁certainty -▁actions -▁date -▁dense -▁habits -▁drinking -▁doubtful -▁significance -▁smiles -▁masses -▁ark -▁subjects -▁abb -nesday -▁fest -▁temporary -▁harbor -▁substantial -▁owing -▁psmith -▁misunder -▁plunder -▁plough -▁lucky -▁beasts -▁rebecca -▁juliet -anced -▁prairie -ieth -▁twentieth -▁stealth -▁wearing -▁reaching -▁changing -elphia -adelphia -▁philadelphia -▁digg -ishing -▁mathem -▁ceremonies -▁admission -▁prol -▁prolong -▁scotch -▁superf -▁scor -▁approved -▁shy -▁offended -▁marcell -▁hazard -joice -▁shower -rim -▁fitting -▁kid -▁forbid -▁harvest -ifications -▁pages -▁villages -▁emphasis -▁sustained -▁sustain -▁illness -▁enclosed -mouth -▁humm -▁ascend -▁ascended -▁henriett -▁revealed -▁sledge -▁games -▁critical -▁traver -▁jaws -▁november -▁temples -▁upright -▁thousands -▁tilney -▁quaint -▁hospit -▁hospitality -▁association -▁containing -▁wing -▁concert -▁needs -▁quarrell -▁controll -▁policy -▁happier -▁happiest -▁tempest -▁pirate -▁tig -▁gathering -▁stained -▁stated -▁stump -▁twas -▁crop -▁essentially -▁limbs -▁warrant -▁curse -▁thur -▁outward -▁nowhere -▁sphere -▁university -▁sailed -▁natured -▁jerry -▁estim -▁estr -▁estrall -▁stall -▁stim -▁aim -▁stool -▁wool -▁prostr -▁behal -▁behalf -▁juice -▁amelia -▁developed -▁envy -▁results -▁parson -▁watson -▁reasons -▁cough -▁din -▁shouting -oever -▁octo -▁october -ago -erson -▁dishon -▁descending -lcom -▁welcom -▁lazy -▁strictly -▁coachman -cription -uscript -▁cigaret -▁promising -▁subl -▁sublime -▁varied -▁hired -ola -▁abundance -▁monot -▁they'll -ouquet -zzie -▁flank -▁loudly -▁acquire -▁aladd -▁sources -▁duration -rom -▁destiny -▁procured -▁darcy -▁promin -oundrel -▁scoundrel -abella -▁bachel -▁compare -iscuit -▁rosy -▁chauvel -▁hasn't -▁edison -▁performed -▁clara -▁passionate -▁shrugged -▁obstacle -▁invention -▁sleeves -▁comprom -▁deeper -▁x -▁unworthy -▁restr -▁restraint -▁determine -▁impert -▁cap'n -▁happens -▁minds -▁degr -▁festival -▁costum -▁refined -▁ghast -bourne -▁marvellous -▁suspense -▁braz -▁magaz -▁dix -▁chron -▁wages -▁carriages -▁passages -▁fancies -▁scrut -▁cove -▁clung -▁cyril -▁embroidered -ieves -▁thieves -ogn -▁attempted -▁cares -▁eagerness -▁gregg -▁thoughtfully -▁quitted -▁submitted -▁peaceful -▁raw -▁andrew -▁screw -▁bury -▁creation -▁instinctively -▁corporal -▁distort -▁discont -▁wont -▁waking -▁dressing -▁salv -och -ech -lock -▁glances -▁rested -▁raven -▁gran -▁granite -▁followers -▁bend -upt -▁lights -▁shrill -▁nick -▁expedient -▁tie -▁brows -adows -▁rows -▁brute -▁bunch -uesday -▁analy -▁conseil -▁henriette -▁mustn't -▁needn't -▁sinister -▁register -▁positively -▁deceive -▁dishes -▁guards -▁pulling -gen -ena -▁screen -▁ellen -▁fern -▁efficiency -▁hyster -▁transm -▁irritated -▁obeyed -▁pho -▁phoen -▁phoenix -enz -▁dug -▁associated -cest -▁ancest -co -fold -neath -▁underneath -athan -▁meadows -flow -chor -▁anchor -▁seventh -▁righteous -▁apparat -▁apparatus -▁buzz -▁implor -▁conserv -▁fulfil -▁occurrence -▁flushed -▁bobby -▁carpent -▁torrent -▁desirable -▁desirous -▁glimm -▁drifted -▁despise -▁efficient -▁loaf -▁souls -▁heap -▁falter -▁bluff -▁jasper -▁compliments -▁fed -▁maz -▁francisc -▁attentively -▁haul -▁sulph -▁cupboard -▁restored -▁conspir -▁bulk -▁quart -▁superstition -▁assembly -▁accum -yster -▁oyster -▁blade -asured -▁withdrawn -▁offices -▁sober -mple -▁whirl -▁vessels -▁rarely -▁suffrage -▁pattern -▁student -▁rejoice -▁advancing -▁film -▁dict -▁attendants -▁garr -▁garrison -▁winchester -din -dain -▁disdain -▁measured -▁depended -▁exasper -▁ost -▁bough -▁davy -▁solemnly -▁mob -▁moan -▁almight -▁dreary -▁scripture -▁foam -▁team -nam -▁despised -▁tribe -▁teaching -▁sixth -▁unluck -▁glasses -ses -▁verses -▁caes -▁caesar -▁tribes -othing -▁belongs -▁belonging -acing -▁desolate -▁jok -▁bump -▁starting -▁vegetables -▁responded -▁madness -▁repeating -▁negative -▁negot -▁scrutin -eur -▁touching -joining -▁adjoining -▁lightly -▁constance -▁statue -▁plank -▁lively -▁strangely -▁whereas -▁legend -▁pomp -iqu -iences -▁defiance -▁defined -▁intensity -ulpt -▁sculpt -▁imitation -▁quantities -▁repul -▁dramatic -▁yours -▁blessed -▁julie -▁dilig -▁model -▁prior -▁accent -▁lizzie -▁thorpe -uz -▁jacket -▁acquaintances -▁ke -▁keith -▁kit -▁gladly -ugitive -▁elab -▁elabor -rehensible -▁awakened -▁folded -▁deposited -▁supposition -▁energetic -▁prover -▁proverb -▁perver -brew -▁assertion -▁boxes -▁consolation -▁ninth -▁loy -▁possibilities -▁traveling -▁admirable -▁ancestors -▁tonight -▁management -▁agreement -▁cents -▁cash -▁o'er -▁dreadfully -▁stub -▁stubborn -▁tulliver -▁pudding -▁ax -▁egyptian -▁ferv -▁wigg -▁doubled -▁restore -▁swore -gl -▁dougl -▁musket -▁policeman -▁flashed -▁trifle -▁trifling -uctive -▁dispatch -umerable -▁innumerable -▁lynde -▁manuscript -ondrous -▁merr -▁accomm -▁accommod -▁oxford -stead -▁including -▁sexual -▁harp -lets -▁embarrassment -▁beef -▁conquest -▁hercules -▁misfortunes -▁background -▁rand -▁scand -hand -forn -fornia -ifornia -▁california -lm -▁helm -▁fiery -▁agents -▁talents -▁talent -▁potent -▁county -▁justified -▁maidens -▁dame -▁oven -▁elbows -iour -▁behaviour -▁infantry -▁invar -▁harris -liest -▁prel -▁engineer -▁felicity -▁inher -▁inherit -▁globe -▁thee -▁crystal -▁courty -▁courtyard -chard -▁orchard -▁approval -gard -▁trait -▁scandal -▁jos -▁lace -▁professional -▁hoping -ories -▁idle -▁cigarette -inating -▁customary -▁prett -▁requis -▁daring -▁encouragement -▁entertainment -▁settlement -▁phenomena -▁rejoic -greg -▁congreg -▁faculties -▁faculty -▁kinds -▁imposed -▁expectation -▁frenz -▁hamil -▁hamilton -▁doom -ips -▁loyalty -▁expert -▁map -▁mul -▁numbers -▁islands -▁shows -▁correspondence -▁betrayed -▁behavior -▁phenomenon -▁eug -▁eugen -▁drunken -▁unseen -▁priests -▁aroused -retion -▁discretion -▁dipl -plaine -ynplaine -▁gwynplaine -▁diplom -▁tragedy -▁intervals -▁he'd -▁leaders -▁shrub -▁loun -▁whipped -▁stripped -▁striving -▁invari -▁invariably -▁reaction -▁blunt -▁dad -▁daddy -▁separation -▁dais -▁reminded -▁parall -▁parallel -roph -▁birthday -▁bottles -▁oppon -▁oppose -▁declaration -ational -▁rational -▁ador -▁cultivated -▁excur -▁umbrella -mph -▁attachment -▁aged -▁trout -▁scout -pping -▁vicinity -▁dirt -▁endless -▁assembled -▁apt -▁shaped -inter -▁marquis -▁estates -▁estimate -▁ultimate -▁gaily -▁float -▁artistic -▁calculated -▁nour -▁judges -▁japanes -▁japanese -▁aladdin -▁leon -▁leonora -▁spar -naments -▁abundant -▁personality -▁morality -▁personally -▁valiant -▁basin -▁axe -▁flick -▁flies -▁treason -▁feder -▁federal -▁effects -▁she's -▁cape -▁grunt -▁alyosha -▁chauvelin -▁chop -▁kar -lexey -▁dixon -ston -▁vehic -▁bub -▁radiance -▁fascinating -▁angle -▁transferred -▁pitiful -▁messages -▁languages -▁elm -▁eloquence -elly -▁cruelly -▁frenchman -▁vap -▁vice -▁vib -▁vibr -▁suspicions -▁owl -▁plates -▁detest -▁mike -lantic -▁skil -▁skilful -iless -▁jess -▁jam -▁bewildered -▁resentment -▁oars -▁travelling -fell -▁route -▁alert -▁arrested -▁considering -▁justify -▁directions -▁eyel -nel -▁respon -▁response -▁trous -▁trousers -▁tour -▁faintly -▁fugitive -▁coinc -▁disput -▁tuesday -▁betwix -▁lend -▁sprung -▁ming -▁mingled -▁writers -▁bullet -▁parlor -▁obstinate -▁obstr -▁withdrew -▁gree -▁greece -▁awe -▁awfully -ounded -▁facing -▁rice -▁prejudice -▁swimming -he -hov -▁proh -▁prohib -▁comfortably -▁considerably -bitious -▁bigger -▁wilder -▁wilderness -▁sickness -▁shatter -▁robe -borne -▁fert -▁examples -▁arbit -▁stif -▁nan -▁wheels -ma -▁surm -▁sie -▁siege -▁pav -▁intol -▁intolerable -▁intrig -▁moses -▁mast -▁glancing -▁vale -▁prudent -▁vizier -▁frock -▁appreciate -iate -▁release -▁estralla -maid -▁rhy -▁thankful -▁miracles -▁crust -▁career -▁harness -▁confron -▁fitz -▁contented -▁cared -▁guessed -▁attracted -▁attractive -▁drowned -▁supported -elled -▁mechanical -▁scal -▁vine -▁fashioned -▁scholars -▁startling -▁halt -▁fragrant -▁ungr -▁claws -▁rapture -▁discern -▁femin -▁feminine -ata -▁enlarg -▁foe -▁alarmed -▁jung -▁lung -▁feat -▁sweat -▁indignantly -▁indignant -▁basis -▁extremity -▁undertaken -cribed -▁thereupon -▁gn -mn -▁hymn -nie -master -▁healthy -▁chancell -▁defeated -▁defense -▁mansion -ging -omin -▁rein -▁especial -▁recovery -▁heavens -▁swing -▁metaph -▁monaster -▁plaster -▁disaster -boy -▁dant -▁sorts -▁serene -▁split -▁laura -▁identity -▁wondrous -▁upset -▁horrib -▁politics -▁fid -▁crisp -▁gorgeous -▁converse -▁eccles -▁studio -▁studies -▁jerus -▁jerusale -▁jerusalem -▁desperately -▁severely -▁incons -▁incense -▁rings -▁shillings -▁pockets -▁sacrificed -▁sacrifices -▁beating -▁cakes -▁dismal -▁arabella -▁shir -efort -▁villefort -▁summoned -▁telegram -▁calam -ignific -▁insignific -utor -fish -fe -▁strife -▁charlie -▁surrounding -▁architect -▁handle -▁freck -▁horace -▁dod -▁rubbed -rane -ranean -iterranean -rical -yl -▁steamer -▁financ -iled -▁vigil -▁patty -▁chancellor -ics -▁misunderstand -rection -urrection -▁consecr -▁treachery -▁renown -▁barric -▁barricade -▁joyous -▁feathers -▁mediterranean -▁rises -▁passes -▁league -▁feebly -▁surroundings -▁bidding -othed -▁syll -▁barr -dles -▁armies -bility -▁muse -▁museum -▁clasped -▁persuas -▁unwilling -▁housekeeper -▁barren -▁synony -▁practised -▁trim -▁haz -▁mas -▁boats -▁ceiling -crates -▁incur -ollen -▁challen -▁challenge -▁offence -▁seed -▁loaded -uthbert -▁comparatively -▁forbear -appear -▁reappear -▁charlotte -▁drifting -▁perish -amble -▁mah -▁marched -▁monument -▁fiercely -▁disob -utomob -▁dwelling -▁tightly -▁sending -▁shrank -▁shroud -▁yank -▁complicated -▁enable -▁favorable -inth -▁remarkably -▁incidents -aciously -▁nobility -▁blankets -▁illustri -▁highway -▁humbly -▁jungle -ases -▁phrases -anim -▁footsteps -▁contradiction -▁jelly -▁ambitious -▁extending -brow -▁eyebrow -▁dunb -▁dunbar -▁hob -icago -▁chicago -che -▁perch -▁cens -▁destructive -▁bite -▁sque -iated -▁anticipated -▁unconsciously -▁boldly -▁youngest -▁matt -▁liquor -▁tailor -▁scenery -an't -▁shan't -▁shed -lest -▁philosopher -▁biscuit -▁indicate -▁pencr -▁pencroft -▁chopped -▁rushing -sburg -▁petersburg -▁pump -▁plump -▁subord -▁challeng -▁dripping -▁impressed -▁indulgence -reak -break -▁originally -▁relate -▁mirth -▁henceforth -▁confine -▁dine -▁dares -▁acute -▁acad -▁academ -▁acid -▁acres -▁aid -▁alternative -▁wrest -▁matron -▁privileges -▁nervously -ustral -▁mabel -▁invite -▁inval -▁invalid -▁submission -▁remor -▁remorse -▁remed -▁remedy -▁induced -▁diamonds -▁triumphant -ulously -▁vacant -▁shawl -▁bliss -▁argue -▁excursion -▁discouraged -▁dex -▁dexter -▁harold -▁nerves -orpe -▁chines -▁chinese -▁wonderfully -▁rash -▁landlady -▁barrier -▁expressions -▁cautious -▁ingenious -poonful -▁tablespoonful -▁mistakes -▁marks -inder -▁drops -▁workmen -▁destination -seech -▁compact -▁eliza -▁prejudices -▁gifts -▁wiggily -▁wend -▁democracy -▁drain -▁drug -▁fain -▁unlock -▁eh -▁governess -▁ridicule -▁jacqu -▁needle -▁readiness -▁flora -ica -ilet -▁writes -▁pulse -itimate -▁legitimate -▁attacked -▁imaginary -▁pyr -▁retreated -▁animated -▁seam -▁sniff -▁preserv -▁welfare -▁internal -▁paces -▁languid -▁dissip -▁tavern -▁tank -▁petition -▁regarding -igr -▁mysteries -▁shine -▁traces -▁furnace -▁mathematic -▁aristocratic -▁tid -▁tidings -▁channels -▁heels -▁effective -▁carries -chieved -▁marth -▁martha -▁infirm -▁immortal -ki -▁gaunt -▁gambl -▁prominent -▁wig -▁commence -▁commit -▁u -▁ruff -▁interference -▁gross -thorpe -▁subsist -▁mate -▁slim -▁affections -▁percy -▁hann -▁bachelor -▁sins -▁factory -▁garment -▁substitute -ger -▁confed -▁confeder -▁tiger -▁crawled -oman -ronom -tronom -▁tribute -▁gratification -▁conscient -▁repress -▁influences -ichikov -▁revived -▁peop -▁rop -▁peered -▁peoples -▁padd -▁abbey -▁salvation -▁celia -▁illustrious -▁handed -▁instructions -▁tenderly -▁hurriedly -▁echoes -▁confid -▁formid -▁formidable -▁lid -▁winning -▁proceedings -▁transparent -ceptions -▁snuff -▁berth -▁eccent -▁eccentric -thenians -▁warn -▁senate -▁appropriate -▁otto -▁suffice -▁indisc -▁frantic -me -▁highness -▁strikes -▁sermon -▁bruce -▁faithfully -▁ling -▁lingered -auch -asmuch -▁frivol -▁ecclesi -▁sax -▁encourag -▁remarks -giving -▁roots -▁inner -▁gap -▁views -▁deject -ria -ripp -astroph -▁catastroph -rian -▁european -▁whale -▁abuse -▁edwin -▁edges -▁drap -▁blunder -▁differently -▁fig -▁ensued -mit -itious -▁grandeur -▁enclosure -▁buildings -enn -▁spoon -▁saloon -▁salute -▁aram -▁aramis -▁interven -▁empt -▁learnt -▁overt -▁significant -▁hurled -▁tunnel -▁travelled -▁traveller -▁exclamation -quest -▁trench -enched -▁yourselves -▁discontent -urity -▁obscurity -▁christianity -▁spin -▁spinning -▁paraly -▁confronted -▁converted -iggins -▁higgins -▁clutched -▁casual -▁relationship -▁signature -▁attendant -dess -▁goddess -▁skins -▁rost -▁rostov -▁gnaw -▁expecting -▁silvery -▁spotted -▁weariness -▁spray -▁essay -▁rugged -▁augustine -ivities -woman -▁lingering -▁fertile -▁expensive -▁simultaneously -▁vigorously -▁eminent -▁du -▁fairies -▁tech -▁techn -ching -▁whistle -rael -▁robins -insmore -▁dinsmore -▁childish -▁intending -▁rufford -▁crossing -ennox -▁ends -▁trumpet -ettle -▁kettle -▁uneasily -▁swallowed -▁acted -▁phant -▁phantom -▁nom -▁som -▁sanction -unction -▁damned -▁additional -▁clad -▁horizont -▁fears -▁sciences -▁obedient -regard -▁disregard -▁hack -▁supernatural -▁unnatural -▁carly -▁carlyle -▁fascinated -▁illuminated -▁disl -▁dislik -▁curl -▁adher -▁blessing -▁glittering -▁swee -▁expand -▁cats -▁acts -uded -▁murdered -▁collected -▁reluctantly -▁mechanically -▁mischie -▁insensible -▁provinces -▁fan -vid -interest -think -▁husband's -▁obstacles -▁obstin -▁measures -▁mare -▁uly -▁scrub -▁stain -▁voc -iation -▁loom -▁guine -ainty -▁dainty -▁vehement -▁decline -▁carb -arn -nut -picious -ware -▁cluster -▁ripe -▁ridden -▁rue -dyke -▁pointing -▁glowing -▁nun -▁pyram -▁pyramid -▁clam -cer -▁strings -▁stockings -▁prest -▁songs -▁thereby -its -atis -▁dissatis -▁bunny -▁jenny -▁miller -tel -▁dashed -▁bomb -ubert -▁cabinet -▁jehov -▁cass -▁stare -▁strode -▁splendour -▁occasional -▁superintendent -▁unfold -▁chem -▁chanced -▁expens -▁pledge -▁ivan -▁lovers -▁adventures -▁deer -acinth -▁hyacinth -▁stately -▁swam -▁improp -away -▁smother -▁avoided -▁closing -▁splash -▁dagger -▁feverish -▁scro -▁scroo -▁scrooge -▁urge -▁disappro -ifying -▁placid -▁placing -▁perspir -▁naples -▁girl's -▁deadly -▁neglected -▁deeds -▁disguised -▁rescued -▁yielded -▁ardent -▁forthwith -▁fireplace -▁commonplace -▁commonly -▁defarge -▁pasture -eslie -▁leslie -▁latest -▁cyn -▁twins -fresh -▁ascending -▁passionately -▁humph -ai -ji -▁marie -▁admired -▁admiral -▁mules -▁judged -▁apparition -▁thoughtful -▁chestnut -sitation -▁hesitation -▁stake -▁average -▁reef -▁acknowledged -▁sweetness -▁beck -▁damage -▁damn -▁adequ -▁secretly -▁torch -▁buffal -▁cards -▁brace -▁discomfort -▁fundamental -▁wrist -▁confirmed -▁ned -▁acting -icial -dled -▁foster -▁shirley -▁squeez -▁fisher -▁chapel -illery -▁artillery -▁tenth -▁circles -▁fuss -▁device -▁venice -seless -▁devote -▁troublesome -▁chat -▁bat -▁chick -▁nic -▁harmless -▁harmon -rance -▁girdle -▁begins -▁argued -▁whim -▁marianne -ester -▁bitterness -igger -version -▁appall -▁displayed -▁underground -▁devout -▁tobacco -▁lash -uity -▁ingenuity -gu -guard -▁emptied -▁twisted -▁released -▁indulged -▁indulge -▁elephant -bird -▁westward -▁downward -▁schoon -▁existed -▁shells -ased -▁deceased -▁recalled -▁crimes -ilian -▁volumes -▁wealthy -▁pad -▁falls -▁legg -▁decidedly -▁heavenly -▁pretending -▁hearth -▁jude -▁seizing -▁shutter -▁boiled -▁boiling -ading -▁pressing -▁painting -▁discre -▁beckon -▁equip -▁recip -▁equipped -▁douglas -icient -▁tyrant -▁overflow -▁ske -▁skeleton -▁smallest -▁lodged -▁briefly -▁mute -▁constitute -▁alexander -▁grinned -▁harriet -▁hamlet -▁hardship -▁ladyship -▁historical -▁malign -igneur -▁feign -▁woo -▁horrors -▁untr -▁trimm -▁ponies -▁torp -▁senior -▁distribution -▁abol -▁tranquill -▁tranquil -▁discreet -▁conference -▁eyelid -▁communicate -▁hostess -▁bart -▁impertin -▁controver -▁robbed -▁dependent -▁ghastly -▁sulphur -▁deform -▁divide -▁specimens -vas -▁canvas -▁lowest -▁babies -ipping -issipp -▁mississipp -▁mississippi -▁maintained -orous -▁contempor -▁twig -▁pillars -▁thanked -▁gasped -▁altered -aur -▁restaur -joy -thed -▁ethe -ithe -▁settling -▁scant -▁sect -▁tune -▁lodgings -▁gramm -▁reprod -▁muslin -▁ram -▁jews -▁kings -bs -▁methods -▁terrace -▁mity -▁mitya -▁wizard -▁attentive -▁wishing -▁evolution -▁russell -▁depends -▁resemble -▁polish -▁incomprehensible -▁boom -▁booth -itably -evitably -▁climate -ateau -▁chateau -▁awaiting -▁unaccount -▁breeches -▁diseas -▁towns -▁arbitr -rise -▁sunrise -▁wagg -▁ware -▁ecstas -▁carelessly -▁armour -ourt -▁composure -▁caroline -▁carolina -▁ultim -▁ultimately -▁addressing -▁comprehension -▁conducted -ifies -▁inquire -▁willing -▁lodging -▁perpetually -▁edition -▁receip -▁receiver -broken -▁glen -▁investigation -▁involuntary -▁wrinkled -onze -▁bronze -▁omn -▁cargo -▁rug -▁differences -▁verd -▁berries -▁bridle -▁create -▁mol -▁mascul -▁prelim -▁preliminary -▁included -▁ojo -▁pleasing -▁rubbing -▁crad -▁ruddy -▁gratified -▁secl -▁gentleness -▁requisite -▁thorndyke -aternal -▁endured -▁restrict -▁switch -▁cradle -▁calls -▁dazzling -▁continual -▁today -▁perplexity -▁strewn -▁streak -▁plentiful -▁mournful -▁drows -▁defended -▁reconciled -▁henrietta -▁loyal -▁rascal -▁verily -▁organized -▁quad -ada -▁fortress -▁mun -imble -▁worshipped -▁bang -angle -▁bi -▁willingly -▁thinks -▁peg -leg -▁volunte -▁murray -▁verdict -aves -▁tipto -▁communicated -▁banner -▁banquet -▁selection -▁cad -▁plaint -ino -▁peppino -▁influenced -▁tremulous -▁otter -unned -▁rack -▁races -▁bour -lay -▁perceptible -▁gait -occ -▁bucc -▁buccane -▁bug -▁alb -▁theory -icious -ks -lace -▁necklace -▁flint -uated -▁acknowledge -▁stamm -▁stammered -▁yoke -▁entrust -▁butterfly -▁butterflies -▁greeted -▁defiant -▁purely -▁rattle -▁wendy -▁hasty -▁hasten -▁brandy -▁beforehand -▁prosperous -ae -▁pang -▁sinking -▁parent -▁ascent -▁ribbons -▁garret -▁vehicle -▁accidents -▁rocky -▁pavl -▁splendor -▁dynam -stril -strong -▁africa -▁reinforce -emost -▁accused -▁bestowed -▁offering -▁superiority -▁spake -▁kent -▁kentuck -▁indispens -▁clement -upiter -occup -olph -▁erupt -▁suburb -▁bake -▁gab -▁gabri -▁gabriel -▁glare -▁youthful -▁vicar -▁pipes -▁characters -▁pickwick -▁swinging -▁jupiter -▁interrog -▁mice -ylum -▁rustling -▁magnetic -▁sadness -▁ate -▁presume -▁sod -▁trusted -▁tact -▁palaces -▁poly -▁wallace -▁imagin -▁crushed -▁prevented -▁hearted -▁jove -ives -▁attraction -▁bathed -▁boss -▁bacon -▁exclusively -▁painter -▁inconvenience -stretch -▁angels -▁envir -▁champion -▁renewed -▁summons -▁unhe -▁melted -▁phill -▁phillips -▁dispos -▁disposal -▁accordance -thodox -▁melody -▁hark -▁muddy -▁fidel -▁fidelity -▁teasp -▁undertook -▁stimul -▁motions -▁rebellion -▁chuckled -▁defin -▁definition -▁kin -▁peering -udge -▁impulses -▁jersey -▁believer -▁vary -▁tract -▁she'll -▁respectfully -▁wed -▁wednesday -▁traced -▁packed -coq -▁advis -▁hover -▁riches -▁strongest -ronsie -▁phronsie -▁encountered -ts -▁hud -▁bloody -▁lumber -▁jeffrey -▁femal -▁exertion -▁rend -▁overtook -▁whistling -▁lump -▁ign -▁physicians -▁felix -ez -▁costume -▁coolly -▁fishing -aming -▁convenience -▁mous -▁marvelous -▁atlantic -▁excused -▁magistrates -▁ambassador -▁lud -▁forgiveness -▁diminished -▁unequ -▁glided -otes -otesque -▁grotesque -▁marcella -▁pennsy -▁ribbon -▁moisture -▁sentences -▁disciples -▁struggles -▁turb -▁favourable -▁vegetable -▁fashionable -▁nois -▁noiseless -▁applaud -▁maximilian -▁lined -sue -▁issue -alogue -▁daresay -▁tray -▁trag -▁tragic -ager -▁pag -▁pagan -▁sag -cher -▁churchyard -▁churches -▁banished -▁cherry -▁annih -▁lu -▁bru -▁junior -▁announcement -▁turret -▁turf -▁expenses -▁notes -▁types -▁choosing -eon -▁luncheon -▁whistled -▁curtains -▁dreamed -lain -▁bye -▁fore -ye -▁owner -▁courteous -▁glar -▁gasc -▁republican -▁chew -▁tart -▁fuel -▁mason -▁loft -▁aloft -tal -▁total -▁shelf -▁outrage -▁coax -▁whiskers -▁lifeless -▁snap -▁pint -▁noon -▁wiser -▁doth -▁hurrying -▁dined -▁ditch -▁thrice -▁malice -▁lions -▁luke -▁lame -▁industrial -▁formal -▁brutal -▁blanket -▁flu -▁secre -▁secrecy -cliff -▁honourable -▁commod -▁wires -▁congratulate -▁masters -▁master's -▁senseless -▁germ -▁synonym -▁pains -elie -▁lakes -▁snakes -▁lark -▁imposing -enka -ushenka -▁grushenka -▁starved -▁unde -▁ruler -▁emphatically -▁wider -▁maria -▁eventually -▁proportions -▁recl -▁grip -▁recognise -▁nobl -bling -▁firing -▁loves -▁turns -▁bars -▁buddh -▁aristocracy -ander -seigneur -▁monseigneur -ieties -quisitive -▁enlist -▁franklin -▁frankly -▁vinegar -▁multiplied -▁trod -▁avenue -▁elaborate -▁twelf -▁flowing -▁philosophers -▁perched -▁excus -▁sed -▁ster -▁dismissed -▁reverse -▁extravagant -▁prayed -▁reported -▁explo -▁saviour -▁chival -▁lawyers -▁clar -▁clark -mark -▁checked -▁deemed -▁hereafter -▁cuthbert -▁cyl -▁insuff -▁lent -▁consented -▁greed -▁invest -▁rubber -▁attained -▁lick -▁sa -▁saunder -▁eric -athom -▁chronic -unge -▁plunge -▁southward -▁northward -▁homeward -▁rider -▁disturbance -▁pulp -▁exter -▁penalty -▁chichikov -▁flaming -▁clump -water -▁progr -▁contri -▁endowed -▁courtiers -▁knives -▁wo -▁outl -▁hyd -▁wagons -▁delicately -▁companionship -▁predict -▁sor -▁dresses -▁infern -▁dolly -▁sandy -▁unanim -▁gloves -▁bertram -▁labors -▁robes -▁visions -▁sworn -▁swollen -▁trained -▁almighty -ossack -▁cossack -▁coloss -▁chalk -▁confidential -▁sel -▁sequ -▁wipe -▁tuck -▁heroic -▁riot -▁inherited -▁habitual -loc -▁verloc -▁complac -▁stem -▁encamp -▁gild -▁matild -▁growled -▁curled -▁limits -▁ordin -▁ordinarily -▁disconc -lers -▁infancy -▁conclusions -▁barely -▁pushing -▁miriam -▁honorable -▁stag -▁genji -▁outline -▁proceeding -▁smoothed -▁pronounce -▁confer -▁notion -▁cushion -▁engross -▁decis -▁scamp -▁universally -▁parted -▁jehovah -▁gale -▁persian -▁roubles -▁roulet -▁regretted -▁prodigious -▁realm -▁publish -▁concentrated -▁babyl -▁babylon -▁demon -▁redou -▁stepmother -▁obstinacy -▁dantes -▁saul -▁sant -▁samp -craft -frey -▁surmount -▁tumbled -▁stumbled -▁peck -▁pict -▁dump -▁privately -ticoat -▁petticoat -boat -▁petty -▁mischiev -▁annihil -▁odour -▁financial -▁repentance -▁annoyance -▁holm -▁execute -▁bann -▁streams -▁constitu -▁aud -▁audible -▁ud -▁impud -▁giants -▁inheritance -▁nell -▁battery -▁framed -▁destit -▁destitute -▁allah -piece -▁horrified -▁practise -thm -▁rhythm -▁mechanism -power -▁overpower -▁chast -▁groups -don -uted -▁neut -▁neutral -▁echoed -▁assumption -abby -▁shabby -▁roads -▁alexandrovitch -▁twitch -ivial -▁trivial -▁obliv -▁farmers -▁zad -▁doctrines -▁mirac -▁traveled -▁protested -▁protestant -ovel -▁biggest -▁bridges -▁breach -▁achieve -▁landed -ession -▁session -▁sid -▁confessed -▁stooped -▁rainb -▁rainbow -▁climbing -▁preceding -▁calf -▁companies -▁decorated -▁excessive -▁massive -▁inspiration -▁nineteenth -▁odious -▁ingred -▁ingredient -▁curve -▁indispensable -▁twelfth -▁resource -▁planted -▁ripp -▁thomp -▁thompson -▁melod -▁imperative -▁hopkins -▁pegg -▁peer -▁roger -▁contemplation -▁fragrance -▁raymond -▁entertained -▁consisted -▁haughty -▁clapped -▁competition -arian -▁shorter -▁commend -▁merci -unci -ced -forced -▁clatter -▁earthqu -▁earthquake -▁wiped -▁fishermen -▁attire -▁bagg -▁amend -▁mend -▁selected -▁lur -▁lurk -▁executed -▁terrific -▁insufficient -▁hammer -▁fluid -▁keeper -▁epoch -▁inches -▁specimen -itten -▁wri -▁thicket -▁interpretation -▁monoton -▁monotonous -▁intoxic -▁israel -▁chor -▁chorus -uspicious -▁prevailed -▁discord -▁superficial -▁magazine -▁represents -▁strand -▁curate -▁searching -▁catching -▁theatre -▁spared -▁relating -▁relics -▁galley -▁compromise -athen -▁executive -▁bonds -▁killing -▁training -▁merciless -▁divor -▁divorce -▁grieve -▁wist -ombre -▁sombre -▁wre -▁wreath -▁fascination -▁disobey -iture -enditure -▁expenditure -▁demands -▁individuals -▁roughly -▁hudson -▁shared -▁irritation -▁painfully -▁monop -▁frail -▁lar -▁anton -▁decisive -▁coincidence -we -▁rans -▁successfully -▁unhappiness -eliness -▁os -▁osborne -▁fet -▁waverley -▁repor -▁reporter -▁softened -▁implied -▁punch -▁chairs -▁whirling -▁whirled -wich -▁abbe -▁abomin -▁throbb -▁paying -▁empress -▁colony -▁colonies -fl -▁amus -▁amusing -▁lennox -▁baggage -▁tranquillity -▁lob -▁holmes -▁shapes -▁weighed -fred -▁kindred -▁signed -▁cleared -▁welcomed -agraph -▁paragraph -▁cambridge -▁sternly -▁novelty -▁regiments -▁sparrow -▁impet -▁impetuous -▁illustration -▁motives -ats -atical -ashka -ukashka -▁considerations -achus -achusett -▁massachusett -▁viv -▁vag -▁eminence -▁fathom -▁conservative -▁subsequent -▁barbarous -aneous -▁carroll -▁inevitable -▁exists -sp -▁gosp -▁gospel -ingham -▁tortured -▁pictured -▁displeasure -▁lobster -▁recoil -▁mutton -▁fritz -vant -advant -▁disadvant -▁equival -▁forego -▁knit -▁blanch -▁honors -▁precautions -▁precaution -▁liable -▁testament -fare -▁vigour -cle -▁dup -▁proclam -▁durrance -▁tack -▁crab -▁searched -ula -▁bravely -▁namely -▁bum -▁bumble -▁deliberate -▁logical -combe -▁morn -net -▁sheet -▁lifet -▁lifetime -▁ledge -▁jewell -▁revolutionary -▁premises -▁burd -▁exterior -▁joining -▁academy -▁random -▁hinted -▁tink -▁sprinkle -▁females -▁tastes -▁oz -▁missionary -▁elevation -▁constructed -cloth -▁creator -▁gem -▁freight -▁fret -▁pert -try -olutions -▁nourish -▁teachers -▁inclinations -▁filling -▁ego -▁fi -▁fiend -▁perturb -▁possessions -▁humility -▁santa -▁stur -▁sturdy -iph -▁caliph -berries -▁imagining -▁miraculous -▁haunted -▁stamped -▁snatched -▁covering -▁penetrated -aylor -▁taylor -ushroom -otten -▁cripp -▁benefact -▁hundreds -▁currents -▁cliffs -▁reverend -▁existing -▁sailing -▁sails -▁chant -▁lil -▁meek -▁sleigh -▁husk -awning -▁tuf -▁ropes -▁sarc -▁whirlw -▁whirlwind -▁pleases -▁diame -▁cylinder -▁nostril -▁tribun -▁lucian -▁buffalo -▁guy -▁apprehended -▁apprehensions -▁pearls -▁remov -▁freckles -▁hart -▁doris -sis -▁analysis -▁hypothesis -▁cris -▁crisis -ris -▁fab -▁fabric -cent -▁patent -aware -▁raff -▁lilies -▁laden -▁natives -▁data -▁duel -▁enquir -▁ungrate -▁we'd -throp -▁beauties -long -▁majestic -▁respectful -▁merciful -▁puss -▁godfrey -▁coop -▁urgent -▁duly -▁wife's -▁refusal -▁constern -ball -▁contribute -▁attribute -▁restaurant -▁fry -▁voluntary -▁unex -ias -iary -▁starlight -▁oxen -▁dot -▁betwixt -▁illusion -▁exclusion -auchamp -▁speaker -▁cushions -▁knitting -▁creeping -ibilities -▁rubbish -irect -▁indirect -▁attentions -▁infall -▁insignificant -▁balc -▁balcony -theus -▁sweethe -▁sweetheart -▁anthe -▁anthea -▁consists -▁spy -▁divinity -▁lawrence -▁merriment -▁merrick -▁click -▁vanqu -▁vanquished -▁persecution -▁contriv -▁harrington -urance -▁recur -▁disappearance -▁artists -▁exchanged -omwell -▁cromwell -▁regularly -▁solely -▁litter -▁psychology -bishop -▁archbishop -oki -▁pok -ukoki -▁severity -▁picking -▁locks -swers -▁answers -▁milburgh -▁diameter -▁mammy -▁cyrus -uminous -▁luminous -▁blake -▁navy -mmy -ys -▁abys -▁metaphys -▁goods -▁symptoms -▁gets -▁junction -▁fist -▁unmist -▁unmistak -▁unchang -ama -▁charley -▁eastward -▁inward -▁headlong -▁undergo -▁plato -▁imprisonment -▁insolent -onathan -▁jonathan -▁nathan -▁latch -▁onward -▁scrib -▁scour -▁errors -▁preservation -▁rotten -▁ethel -▁sticks -utomobile -▁necessities -ocious -▁ferocious -▁nails -▁grows -▁advers -▁adversary -▁horsemen -▁deepest -▁insurrection -▁devices -▁poets -▁propriety -▁magnificence -▁olive -▁nobles -▁arrows -▁spaniards -▁helmet -▁packet -lessly -▁pretex -▁flowed -ossibility -▁impossibility -▁spending -▁ranks -▁vapour -▁witnesses -▁calamity -▁rapidity -▁ungrateful -▁toler -▁harder -▁charges -▁clum -▁clumsy -ounced -▁barbara -▁feud -▁cheval -▁chevalier -▁portions -▁utility -▁cha -▁lee -▁socrates -▁ravine -▁survived -▁ornaments -▁barber -▁text -▁curt -erville -uv -▁associations -uzov -utuzov -▁antiquity -erous -▁treacherous -▁immaterial -▁exile -▁manager -▁continuous -▁unique -▁stunned -▁economy -▁bleed -roll -▁identical -houn -▁calhoun -▁coun -▁nights -▁gidd -▁activities -▁sinc -▁sincerely -▁alien -▁frighten -▁eloquent -▁lun -▁buckingham -▁opera -▁operator -▁imprud -▁affl -▁affliction -▁fiction -▁fol -▁foli -▁foliage -▁fouquet -▁faire -▁fairest -▁annoyed -▁yawn -▁pont -▁pouring -▁wherein -▁nobleman -▁interruption -▁overtake -▁maker -▁sincer -▁sincerity -▁chickens -▁peb -▁pebbles -▁resting -▁annual -▁inviting -▁thwart -▁blows -▁blowing -▁brains -▁moust -▁moustache -wr -writing -▁dragging -▁praises -▁doc -▁document -▁documents -▁dipped -▁dizz -▁perished -▁engines -▁adding -▁prophe -▁prophecy -time -▁dolls -▁lang -▁lic -▁lukashka -▁delir -▁delusion -▁recourse -▁prostrate -▁gibb -▁despond -odor -yodor -▁fyodor -▁fyodorovitch -▁deput -olate -ocolate -▁betro -▁betrothed -▁morcer -▁morcerf -▁morris -▁politely -▁confided -▁uncover -erving -▁abide -▁stretches -▁javer -▁javert -ivable -ctic -▁beds -▁rivers -▁drama -▁dora -▁promote -▁hannah -▁grasped -tally -▁digging -▁mantle -▁dissatisfied -berg -▁arkady -▁arkadyev -▁ransom -▁montg -▁montgom -▁montgomery -▁correction -▁advances -▁injustice -▁naughty -▁enthusiast -anda -iana -▁sab -▁mercury -▁contains -▁transact -▁libert -▁muffled -▁liking -▁enjoying -▁greeting -▁eth -▁entreated -▁airship -▁neighbours -▁vices -▁blessings -raged -mitri -▁dmitri -▁hue -▁revenue -▁inexpress -▁inexplic -▁implic -▁filth -▁depression -▁conquered -▁darted -▁crooked -▁columbia -▁realised -itations -▁forests -▁grounds -▁convuls -▁tasted -▁ecclesiast -▁ecclesiastical -▁involve -▁feels -▁usurp -▁patric -▁baptism -▁circulation -▁circus -▁mow -castle -▁sorrowful -▁vass -▁injure -▁chaos -▁supple -▁exhort -heard -▁alph -▁alphab -▁alphabet -ustria -▁brings -▁spaces -▁spenc -▁ej -▁ejac -▁ejaculated -jah -▁bravery -▁stables -▁enchanted -ider -▁panic -▁purity -▁vegetation -▁revelation -▁zadig -▁elph -▁orph -▁soph -▁pretence -▁agriculture -cessantly -▁millionaire -▁tough -▁syd -▁sydney -▁lecoq -▁simplest -▁enraged -▁magnet -selfish -▁unselfish -▁unbroken -▁bog -aux -▁carn -▁available -▁quivering -▁treaty -▁strokes -▁shady -pire -▁slack -▁transgress -▁countless -▁counted -▁victorious -▁victoria -▁plains -▁lady's -▁exposure -▁ounce -▁alternately -▁wounds -▁parr -▁parrot -▁conventional -▁prettiest -▁vinc -▁vincent -ties -▁kutuzov -▁bount -▁console -▁inexper -▁luxuries -▁mound -hound -▁redeem -▁salary -▁flask -▁sobs -▁rot -▁sensibility -vory -▁gust -▁redoubled -▁missed -▁amazed -far -▁whar -▁preference -▁carav -zan -▁tarzan -▁pennsyl -▁pennsylvan -▁pennsylvania -▁imitate -▁concess -▁foundation -▁heavier -▁vaguely -hor -▁abhor -iah -▁proprietor -▁include -▁negroes -inda -▁trumpets -▁screamed -▁dreaded -▁sally -▁si -▁laure -▁manufacture -▁sculpture -▁wildly -▁bells -▁balls -rimony -▁matrimony -▁pigeon -▁francisco -▁remo -▁removing -onesome -▁lonesome -▁wanting -▁baked -irk -▁lever -▁leop -▁whiteness -▁seventeenth -▁diet -▁civility -▁superstitious -istinct -▁indistinct -▁colbert -▁stew -▁chocolate -soever -▁rese -▁perilous -▁purit -▁remn -▁diversion -▁clutching -▁confinement -▁slide -▁hospitable -▁humming -▁bags -▁precept -▁milan -▁glee -rie -▁elaps -▁pierced -▁warfare -▁vind -▁vow -▁vamp -▁pamp -▁chip -▁mall -▁mull -▁midd -▁sung -▁guar -▁guarant -▁lew -▁lewis -▁kl -▁fult -▁fulton -▁passive -▁persisted -▁brother's -▁fifteenth -▁compound -▁compens -▁compart -▁sandford -ingale -▁nightingale -▁cling -▁clinging -▁begging -▁laying -oneliness -▁loneliness -▁proudly -▁principally -▁impelled -▁embracing -▁memorable -▁tint -▁vexed -▁vexation -▁elderly -▁winterbourne -▁parker -▁brist -iston -▁conniston -▁glist -▁partial -▁infinitely -▁ounc -▁toby -oughby -▁landing -▁inspection -▁pork -▁curs -▁cursed -▁carbon -▁mushroom -▁simpson -istry -▁um -▁ticket -▁ast -▁rosamond -▁dimly -▁briskly -▁unlucky -gans -▁organs -▁repro -▁replies -▁seemingly -ifice -▁precipice -▁magnitude -▁angles -▁alfred -▁mars -▁boy's -▁glimpses -▁momentary -▁burton -▁fut -▁brightly -▁jenks -▁nought -ieu -▁adieu -ie's -▁adap -▁adapted -▁naval -▁situations -▁chemical -▁typical -▁arts -▁armor -▁trades -▁almah -▁cruc -▁appreciation -▁noisy -▁here's -anning -▁channing -usions -▁scared -▁martian -any -▁moham -▁mohamm -▁exal -▁exalted -▁moor -▁mole -▁moth -▁sealed -▁duncan -▁courtes -▁courtesy -▁unexpectedly -▁antip -▁miner -▁mexico -▁mexican -▁dispers -▁claw -o's -▁seats -▁enlightened -▁keys -▁dimensions -▁pretensions -▁producing -▁revolutions -▁urb -▁mortals -tery -bray -▁emer -▁emerald -▁proving -▁impass -ocity -▁gradual -▁precision -aday -▁nowaday -▁hebrew -quiet -▁sembl -▁ferdin -▁smashed -▁skirm -ometer -▁jug -▁eyebrows -▁agricultural -▁blink -▁opponent -▁mack -bourg -▁impost -▁disinterest -▁eames -icate -▁steadf -▁steadfast -▁despot -▁canoe -oor -ooroo -▁sanctuary -▁blaze -▁blazing -▁adel -ycle -▁sham -▁thirsty -inald -▁reginald -▁sixteenth -▁profitable -▁repet -▁repetition -ancourt -▁valancourt -▁baff -▁butler -▁lungs -▁swords -▁shrunk -▁distances -▁gravel -▁lime -▁territor -▁pitiless -▁popul -▁populace -▁denis -▁denial -▁martial -▁cordially -▁opponents -▁comte -▁squad -▁squadron -▁ruby -▁unpre -▁unaccustom -▁snapping -▁obstruct -gone -▁humphrey -▁loveliness -▁muzz -▁astronom -▁cel -▁celest -▁celestial -▁code -▁gretel -▁shades -osis -▁superfl -▁superfluous -▁wald -▁sprinkled -ardier -▁thenardier -▁headed -▁governed -▁physi -▁physically -▁ped -parable -▁pierce -▁environ -▁alexandrovna -▁breadth -▁carth -▁beth -areth -▁appare -▁corinth -▁lugg -▁luggage -▁courses -▁insolence -▁elphick -▁benevolence -▁eury -▁discrim -▁robinson -▁exhaustion -▁pupp -▁laurel -sitate -▁hesitate -olooroo -▁boolooroo -▁excellence -▁afforded -▁hurst -▁hurstwood -print -▁clue -▁twinkling -▁salmon -▁nodding -▁florid -▁explanations -▁badger -▁dome -▁statues -▁extinct -▁muscle -▁merrily -▁oppression -ustralia -rial -▁burial -▁bure -▁bureau -▁valet -wealth -▁poems -▁neatly -▁horns -▁deleg -▁keenly -▁mord -▁mordaunt -▁locality -▁crumbs -▁sports -▁trop -▁penetrate -▁penetrating -▁rifles -▁hopped -▁treasures -▁lounge -▁accuse -▁explosion -▁deplor -▁verge -▁mentally -▁cul -▁scottish -'n't -▁associates -▁routine -keeper -▁purchased -▁slopes -▁rattled -▁rattling -▁livid -▁ens -▁holiness -▁sterling -thouse -▁eighteenth -▁deserving -▁preserving -▁malady -utomatic -▁flourish -▁dreaming -▁frig -▁frances -▁nathaniel -▁rigg -▁rup -▁maud -▁mockery -▁slippery -▁bery -▁beryl -▁baz -azed -mates -▁gotten -▁pavlovna -▁stretching -▁sentimental -▁marshal -▁horribly -▁resolutely -thos -▁confidently -▁trapp -▁cabb -▁workman -▁appreciated -▁daybreak -▁void -▁voluntarily -▁temporarily -▁properties -▁attendance -▁trifles -▁guich -idious -▁pious -▁paws -▁poster -▁hog -▁technical -▁beseech -▁barrister -▁accurate -▁accuracy -▁accurately -▁fragment -▁elevated -▁gould -▁pupils -▁torrents -▁triumphantly -▁porridge -▁abilities -abilities -▁reports -▁inconsist -angled -▁fishes -escent -▁abyss -▁finds -▁regards -▁pinch -affold -▁scaffold -▁traff -▁traffic -▁utterance -▁overcoat -▁unreasonable -raved -lesome -▁jaw -▁ivanovna -▁crev -▁av -▁exerted -verted -▁futile -▁banners -▁factories -▁interchang -▁stealing -▁fidd -▁obligation -▁societies -escrib -▁indescrib -cribe -▁indescribable -▁witches -▁wits -▁hazel -▁obliter -▁rags -▁printing -▁merlin -▁insane -▁heroes -folk -wall -wig -▁kenwig -▁epis -▁episode -▁disclosed -question -▁unquestion -▁florent -▁soothing -vis -▁jarvis -▁rumour -▁malicious -▁warn't -▁roaring -▁burdens -▁solicitude -▁largely -▁based -▁basil -▁virgil -ropol -▁metropol -▁vrain -sieged -▁kenne -▁kenneth -▁troy -anna -▁tyrann -▁clown -e's -▁injuries -▁astray -▁coil -iles -▁prices -▁gill -▁inventor -▁invented -▁discussed -▁dungeon -▁identified -▁resignation -▁benevolent -▁superb -▁emergency -▁emerged -▁stepping -▁overheard -ios -▁semi -▁curved -▁staying -▁leads -▁tells -▁busily -ubb -▁ignomin -▁conspiracy -▁subsided -▁modesty -▁daisy -roar -▁uproar -▁extension -alm -▁rouletab -▁rouletabille -▁cultivation -▁reins -▁condens -▁texture -▁wilm -▁velocity -last -▁everlast -▁sobbed -▁sobbing -frican -an's -▁tube -▁tolerably -▁miserably -▁poles -▁antique -▁perpend -▁perpendicular -▁ferdinand -▁camel -▁mortim -▁excluded -▁starving -▁klaus -▁earnestness -▁clarence -held -▁ceaseless -rupted -▁bennet -▁chart -▁controversy -ipsy -▁gipsy -▁waken -▁strove -▁partners -▁partially -▁odor -▁sovereignty -▁wy -▁englishmen -▁masculine -▁tracks -▁wrigg -ustra -athustra -arathustra -▁zarathustra -▁width -restrial -▁ringing -▁rejoicing -▁cooking -▁electricity -▁particip -▁municip -▁confederate -▁burrow -▁overboard -▁spectator -▁maur -▁maurice -▁hagg -▁haggard -▁agg -▁michel -▁deference -▁ministers -▁exclusive -pool -▁liverpool -▁tops -▁inference -▁alongside -▁suppressed -▁tooth -▁pans -▁paw -▁cage -▁cutter -umberland -▁dispose -▁removal -ocation -▁pavlovitch -▁coin -▁grinning -▁bred -▁hermit -▁racing -▁crick -▁crude -rude -ertrude -▁gertrude -▁gord -▁na -▁commissioner -▁personage -▁accounts -apart -▁bonapart -▁grap -▁grapes -▁groc -ads -▁nuts -▁boh -▁bohem -▁conferred -▁snar -▁reluctance -▁reluctant -▁cant -▁reflections -▁license -▁stifled -▁berg -uniary -▁blamed -peak -▁pleasantly -▁davis -▁plight -▁gwend -▁gwendolen -ways -▁materials -▁divers -▁photop -▁photoplay -▁spots -▁claims -inster -▁associate -▁embarrassed -▁assisted -▁choked -▁lorry -▁rosal -▁endurance -▁schemes -▁child's -▁audacity -▁chooses -ulent -▁cogn -▁fruits -▁bills -▁expressive -▁mim -▁inconce -▁inconceivable -▁joyce -ridges -oux -▁compress -▁cec -▁alc -▁alp -▁fitzp -▁fitzpiers -▁hump -ha -▁paras -▁armstrong -▁cows -forward -▁trotted -▁arkadyevitch -▁refine -▁guarded -▁turkish -▁claimed -▁aggress -▁isabella -▁garland -▁morland -▁shah -▁providing -▁larry -▁presents -▁livery -▁disadvantage -▁belg -▁revolt -▁communities -▁communion -▁unpr -▁mumm -child -wright -▁loveliest -▁perchance -▁stress -▁undress -thest -arin -▁mazarin -▁mathematical -▁barrack -amel -▁interfered -▁bedside -▁invaded -▁degraded -▁dul -▁facility -▁vanish -▁aught -▁reassured -▁elastic -▁dispens -▁sill -▁knob -▁assigned -▁imaginative -▁owed -▁yach -▁yer -▁ominous -▁omitted -▁industrious -▁invas -▁invasion -▁environment -lessness -▁everlasting -thetic -▁susceptible -▁dinners -▁cater -▁caterp -▁caterpill -oir -▁sho -▁veter -▁drawer -▁molly -▁magnanim -▁deliverance -▁whereby -▁presby -▁subordinate -▁excessively -▁sioux -▁dand -▁commotion -▁patted -▁adorned -▁ferry -▁abject -▁cowboy -▁orphan -▁sciss -▁scissors -▁terrors -▁suggestions -incial -▁provincial -▁bathing -lorn -▁tangled -▁morsel -▁preoccup -▁phase -sip -ssip -▁gossip -yrinth -▁labyrinth -▁leopold -▁paste -▁plane -▁brun -▁programm -▁monastery -▁boff -▁boffin -▁relent -▁dissolved -▁gleaming -▁acquies -▁earthly -▁damon -▁retreating -▁sidney -▁elegance -▁international -▁mines -▁ministry -▁vicious -▁cornwall -▁replace -▁wholesome -olley -▁poll -▁perfume -▁deme -▁demean -▁demeanour -▁pathetic -▁unimp -▁dusty -▁slice -rice -fallen -leby -▁nickleby -▁unity -▁elmwood -▁prowess -▁colours -▁catastrophe -▁pew -▁berwin -▁warmly -alled -▁apert -▁aperture -▁cert -▁certific -▁gaw -▁gawaine -▁readers -▁denom -▁darkened -cile -▁advertisement -▁prescribed -▁restoration -▁luxurious -▁equipment -▁compartment -▁whatsoever -uart -▁stuart -uing -▁kissing -▁panting -ta -▁benches -▁shocked -▁ded -▁conveyed -▁blacksmith -▁footprint -▁monica -▁mountjoy -▁aspir -▁bleak -▁rebel -▁pel -▁pic -▁frivolous -▁entry -▁staggered -▁piled -▁piles -▁mayor -cuniary -▁pecuniary -▁leaping -▁sump -▁joyful -▁handful -▁eminently -▁lustre -▁shattered -▁protected -▁bears -icable -▁relatives -▁delirium -▁renounce -▁chopin -▁joyfully -▁adjac -▁adjacent -▁snail -▁mung -▁endeavoured -▁explore -▁tropical -▁enquired -▁deceit -▁mischievous -▁attempts -itors -ita -▁vit -▁dunde -▁sorcer -▁proclamation -▁posterity -▁commiss -norm -▁abnorm -▁fraud -sky -▁glistening -▁accost -▁bast -▁stillness -▁fantine -▁accidentally -▁rugg -▁jerome -▁pitt -urst -hurst -▁primary -▁waz -▁wazir -▁ecl -▁peach -▁babe -▁fleece -▁uncertainty -▁rights -▁suic -▁suicide -▁hubert -▁dazed -▁partic -▁particles -▁transmitted -onn -▁referring -▁expectations -aria -engaria -▁berengaria -▁canada -▁hoist -uber -▁squeeze -▁muzzle -▁whispering -icycle -▁baking -▁ba -▁bass -▁drill -▁footman -▁tents -▁resurrection -▁erected -▁elected -▁bicycle -▁heir -▁encir -▁encircl -▁burgund -▁kitten -▁baza -▁bazaar -▁wisest -▁vile -▁exaggerated -li -▁operate -uate -▁debate -ua -aitor -▁traitor -groom -▁bridegroom -▁trigger -▁truck -aska -▁cask -▁popp -▁roared -▁finer -▁bout -▁overwhelmed -▁karl -▁wield -▁swoon -▁arises -▁crawling -▁maxwell -▁keeps -▁nem -▁outstretch -▁gracefully -▁generously -▁judging -▁gyp -▁reel -▁nast -▁bleeding -▁intensely -▁dorothe -▁lords -▁groaned -▁robbers -▁improvements -▁adequate -▁birch -▁degen -▁clamber -▁selfishness -▁wickedness -▁cavalc -osc -▁transc -▁transcend -▁offend -▁aristot -rich -▁jewish -▁smugg -▁abominable -▁spont -▁spontaneous -▁indeb -▁indebted -▁magd -▁magdal -▁magdalen -▁conceptions -▁faults -▁balanced -▁presbyter -▁mattress -▁apology -ken -▁repay -▁doomed -burst -▁flashing -▁medicines -▁lizard -▁involves -▁strata -▁teaspoonful -▁picn -▁biograph -▁incom -▁kins -▁lads -▁candidate -▁persistent -▁arctic -▁arrog -▁switz -vy -▁strained -▁corpore -▁constrained -▁multitudes -vidrig -▁emphatic -▁vouch -▁envious -▁cit -▁ferocity -▁evidences -▁margin -▁baronet -▁lava -▁serpents -con -afe -▁dominion -▁planch -▁planchet -'er -▁ne'er -▁accumulated -fax -▁guarantee -ghai -▁glaciers -abouts -rogen -▁equivalent -▁ample -▁terry -▁thump -tered -▁noblest -▁capit -▁spraw -▁consternation -▁woven -▁aven -▁avenge -▁gau -▁eleventh -▁shutting -▁interrupting -▁nasty -▁stormy -▁humiliation -rift -▁comprehended -▁slugg -▁thickness -▁greatness -▁spoonful -▁importun -▁signifies -▁determining -gate -▁petrov -▁cob -itous -▁milit -▁tempting -▁righteousness -inguished -▁extinguished -▁letty -▁philosophical -▁schooner -▁predomin -on's -▁impl -▁springs -▁beatrice -▁romola -▁witnessed -▁chivalry -▁unsatis -dale -▁phine -▁phineas -▁costly -▁scept -ceptibly -▁sic -▁sicily -▁sage -▁saith -▁immortality -quiring -▁seeming -▁studying -iazz -▁piazz -uition -▁intuition -▁pleasures -▁poys -▁errant -▁harrison -down -▁herd -dained -▁chaise -▁misgiving -▁tearing -atrical -▁mukoki -▁folds -▁ecstasy -▁ty -▁jem -▁benj -▁benjam -beam -▁spacious -▁hire -▁submerg -▁ferg -▁fergus -▁submiss -▁arbitrary -▁tiss -▁tissue -▁secluded -▁slipping -▁slippers -▁varieties -▁steven -▁dwellings -▁foremost -▁melbury -▁shouts -▁travers -▁chains -ots -lo -▁sweeping -▁washing -▁chamberlain -▁lowered -▁questioned -▁confounded -▁elephants -▁orient -▁matilda -▁fulfilled -▁impressive -▁patr -▁retr -cients -▁queen's -▁devoid -▁fland -▁flanders -▁infernal -ruit -▁recruit -▁lyd -▁holds -oof -▁thereof -▁sparkled -▁paradox -▁bourge -▁bourgeois -▁kil -▁escaping -▁pigs -▁barred -▁cod -▁finn -▁darn -yce -▁bryce -▁produces -▁hawkins -▁envoy -▁frenzy -▁damasc -▁lain -▁hostility -▁unsucc -▁unsuccess -▁unspeak -▁presumption -▁councill -▁talb -▁talbot -▁arb -▁aris -▁arisen -▁retained -▁bullets -▁resolutions -▁shark -▁encroach -▁oppressive -▁oppressed -▁broom -▁brink -▁broth -▁evans -▁chased -▁apoll -▁ape -▁resume -▁aloof -▁sacr -▁sacrament -▁heav'n -▁tru -▁logic -▁charms -▁reflex -▁crater -▁mirrors -▁actors -▁beings -▁perv -▁bertucc -▁openly -▁monotony -▁bower -▁daughter's -y's -stood -▁misunderstood -▁russians -▁digest -▁spider -▁bewild -▁stooping -▁infam -▁beam -unition -munition -▁ammunition -▁columns -▁dishonest -▁dishonour -▁clamour -eries -▁galleries -▁energies -▁passenger -▁refreshment -▁spreading -▁helping -▁fanciful -▁diligence -▁highland -▁starve -▁solve -▁solved -▁tutor -thentic -▁feeding -▁rouse -▁hateful -▁bard -▁packard -lymp -▁olymp -▁emigr -▁skies -▁persuasion -▁bucket -▁titles -▁greyhound -▁grimly -▁depr -▁deprive -▁ramp -▁meditation -▁breez -▁firmness -▁brightness -hear -▁jumping -▁cork -▁humb -▁domain -▁rapt -▁delivery -▁usher -▁culpr -▁flourishing -▁reduce -▁dividing -▁dearly -udoff -ludoff -hludoff -khludoff -▁nekhludoff -▁pamph -▁abolition -▁whites -▁thanksgiving -▁outlines -▁outlaw -▁champag -▁champagne -▁depre -▁depict -▁treating -▁enchantment -▁hatter -▁hatteras -▁boughs -▁applies -▁messengers -▁perspiration -▁speculation -▁tebr -▁tebrick -phem -▁refusing -▁clearing -▁declaring -▁backwards -▁stealthily -▁subsequently -▁arc -▁arising -▁advertising -▁fatigued -▁garb -▁withered -▁sedley -itative -▁saxon -▁nib -ieces -▁successive -▁strive -ulture -▁gateway -▁marching -▁chaplain -▁gloomily -▁luckily -▁oriental -▁guid -▁guidance -▁overtaken -▁weaken -▁apprent -▁neighboring -▁curves -▁clapping -▁mortification -▁asylum -▁screech -izzard -▁sylvie -▁haired -▁chim -▁piercing -▁preceded -▁adams -▁ruled -▁rulers -▁nemo -▁undist -▁undisturb -ologist -▁afflicted -▁reckoned -▁detected -▁wan -▁famine -▁haze -etically -▁complained -▁remembering -ien -▁serf -▁sift -▁sall -▁engagements -▁jefferson -▁envied -agers -▁villagers -▁hag -▁pines -▁deliberation -▁formation -▁forming -▁plantation -▁entangled -▁injunction -▁distressed -▁spoiled -▁resigned -▁exemp -adian -▁canadian -▁flavor -▁sap -▁incor -▁regulations -▁voyages -▁stages -▁cottages -▁stature -▁seclusion -▁postp -▁records -▁crimin -▁criminals -oz -▁tri -▁bustle -▁clenched -▁shadowy -▁wester -akem -▁wakem -▁stragg -▁strat -▁occupations -▁therewith -▁handwriting -▁wilf -▁negl -▁neglig -▁pend -▁robbery -▁ohio -▁witchcraft -▁laud -▁dow -▁tries -▁aristotle -▁hipp -▁bows -▁og -▁consulted -▁slices -▁baffled -▁strap -▁schools -▁semblance -▁buoy -▁preval -uph -▁refinement -▁kneeling -▁genial -▁genie -▁yea -▁stony -thony -rat -▁ingrat -lestick -▁texas -▁perspect -▁visibly -▁propag -▁penit -ya -▁sonya -▁shovel -▁censure -▁anxieties -▁tapped -▁tatter -▁curls -▁expressing -▁monuments -▁monton -▁montoni -▁linton -▁shuddered -▁vitality -▁wegg -▁earning -▁planning -▁pa -▁paup -upp -▁draft -▁graces -▁votes -▁doctor's -▁coldly -▁gascoy -▁saluted -▁offensive -▁lavish -▁objections -ammed -ols -▁tools -annie -edges -▁marrying -▁enact -▁forefather -▁brim -▁explod -▁embod -▁jur -▁penrod -nikov -▁brill -▁preposter -▁wasted -▁huddled -▁resembled -▁renders -▁perseverance -▁writhing -▁attending -▁beggars -▁colleagues -▁harrow -▁furrow -▁leapt -ghthouse -▁lighthouse -▁anglo -▁carleton -▁receipt -▁rhyme -▁honestly -▁nik -▁ship's -▁iceland -▁contemplated -▁hunters -▁customs -▁commands -▁aggrav -stones -ste -▁barry -ufton -▁lufton -▁lance -▁settlers -quois -roquois -▁privileged -▁rampart -▁apiece -▁candlestick -▁forbidd -▁enemy's -▁pitcher -▁greedy -otting -▁mott -▁munch -▁foes -▁patches -▁wharf -▁croc -▁crocod -▁cramp -▁crut -▁divisions -▁comprehensive -▁rejected -▁nicely -▁demonstration -▁acceptance -▁balin -▁violin -▁lor -▁strolled -▁pitied -▁punctual -▁subter -▁subterranean -▁nether -▁teres -▁teresa -▁terrestrial -▁util -▁excuses -▁patriarch -▁tails -▁proposals -▁bridget -▁commanding -▁continues -▁vagab -▁vagabond -▁soap -▁kh -▁benjamin -thin -▁elin -▁elinor -▁nimble -▁awhile -▁hounds -▁donald -▁stitch -▁pillows -▁hearty -▁denz -▁denzil -▁abstain -▁wireless -▁dissol -▁allowance -onial -ential -▁consistent -yon -▁canyon -▁semyon -▁blur -▁injur -urable -▁ebb -▁survive -▁sandwich -▁morb -▁morbid -▁touches -▁sangu -▁retir -▁homer -▁unint -▁walks -▁loos -▁loosened -▁rosa -▁repast -▁dazzled -▁nett -▁enforced -▁thereafter -stein -agin -▁timidly -▁appeals -▁nostrils -oultry -▁vigilance -pole -▁supplic -▁rehear -▁countrymen -▁taxes -▁smash -▁bail -▁carthage -▁florentines -▁insens -▁watchful -▁mantel -▁admirably -▁admiring -▁untouch -▁targ -▁mournfully -▁imprisoned -▁earned -▁heroine -▁convert -▁bertuccio -▁han -▁rusty -▁scales -▁attributes -▁attributed -▁lydia -▁exalt -▁chil -▁chiltern -▁platter -▁plaus -cknowledg -agr -▁walnut -▁skiff -▁bailiff -▁courier -▁vines -▁spasm -▁thigh -▁pursuing -▁waiter -▁glenc -▁immeas -▁snob -▁snort -▁cavalier -▁esteemed -▁indication -▁commonwealth -▁rector -▁scope -escope -oscope -escence -▁impudence -elle -itchell -▁mitchell -▁rhe -▁rheum -▁interpre -▁suffolk -▁prolonged -▁coral -▁crawl -▁crawley -▁willoughby -▁reigned -▁responsibilities -▁kenn -▁peac -▁peacock -▁sarcastic -▁sarcasm -▁nora -▁cherish -▁plaintive -cibly -▁disconcerted -▁confederacy -▁reap -▁gloss -▁apparel -▁morton -▁venet -▁unintell -▁hurric -▁georgia -apse -verse -▁wolver -ouver -▁enforce -fellow -▁belinda -▁improper -▁graciously -leans -▁orleans -▁hurricane -▁inch -▁mature -▁premature -vidrigail -▁coals -▁alliance -▁brazen -▁ul -▁hinder -abbath -▁sabbath -▁potential -▁chaff -ees -▁flirt -▁continuing -▁evap -▁evapor -▁maturity -▁distemp -akov -▁perplexed -▁blushed -▁blushing -▁pite -▁gascoyne -▁illeg -▁truce -▁spruce -▁smoked -▁peculiarly -▁stack -▁smack -▁rak -▁riddle -▁arn -▁nowadays -▁obligations -▁oblige -iations -▁negotiations -▁guiche -▁silverbridge -elf -▁oneself -▁suite -▁faltered -oral -▁whal -quar -quarters -▁headquarters -▁mosqu -▁unaccountable -▁pinched -▁drained -▁tyrants -▁approb -▁approbation -▁approving -gged -▁corporeal -▁reasonably -▁disliked -▁depressed -sday -▁thursday -▁prosecutor -▁barbic -▁barbicane -▁pavil -▁congratulations -▁piety -▁amendment -▁descendants -▁figured -▁straightway -▁brigand -▁congregation -unciation -▁hoofs -sitating -▁hesitating -▁raz -▁runs -▁gliding -▁choir -▁lamps -▁discharged -iard -▁billiard -▁concur -▁concili -▁desperation -▁respects -▁prohibition -▁fourteenth -▁pell -ancouver -▁signify -▁signor -▁rendez -▁hens -▁manhood -▁boyhood -▁shilling -▁amazing -▁parting -▁saunders -▁separately -▁grop -▁widely -▁reed -▁guineas -▁migr -▁pod -▁juris -▁moderation -▁target -witch -▁secrets -▁undergone -▁neighbouring -▁client -▁claver -▁unequal -▁unmoved -▁discussing -▁tame -▁inhabited -▁propens -▁dejah -▁nourishment -▁concealment -▁seconds -▁shack -▁pulpit -ras -▁casting -▁chipm -▁chipmunk -▁chunk -▁flann -▁flannel -▁psychological -▁floors -▁anniver -▁farthest -chus -▁tab -▁lots -▁chasm -▁flavour -▁flown -▁forbade -▁memorial -▁coolness -▁deduction -▁kittens -▁unav -▁unavoid -ana -lift -▁torches -▁prince's -▁warlike -vidrigailov -▁borough -awares -mons -▁sermons -▁exertions -▁charco -▁resembling -unct -▁sumpt -▁sumptuous -▁dealt -▁bland -▁sized -▁shallow -▁bibl -▁wilfrid -▁demons -▁gurg -▁coroner -▁coronel -▁tel -▁telescope -▁tess -▁tons -eer -▁pretext -▁reflecting -▁causing -▁lull -▁repuls -▁deserves -▁trav -▁refract -drop -▁tickets -▁proclaimed -wreck -▁shipwreck -▁ours -▁legislation -▁navigation -yrcan -▁happening -ulsion -▁sluggish -▁thurston -iors -▁transformed -▁warden -▁wardro -▁wardrobe -▁mysteriously -inqu -▁relinqu -▁yacht -▁fib -▁fuzz -▁victims -▁unconc -▁uncomp -▁snug -▁worthless -▁senor -▁intrins -▁significantly -▁kennel -▁bait -▁prizes -▁moll -▁foll -▁fiddle -▁upwards -▁acquiring -▁inexplicable -▁accompaniment -urer -▁treasurer -idity -▁stupidity -▁minded -▁gardener -iresome -▁tiresome -▁reasoning -▁benny -▁unintellig -▁serenity -▁assuming -▁payment -▁whig -anthrop -▁philanthrop -ront -mont -▁nel -▁nelson -▁commencement -▁captiv -▁knocking -pence -elp -▁alighted -▁climax -▁fists -▁duc -▁fortified -▁constituents -▁couns -▁counsell -▁margaret's -▁devast -▁prohibited -▁totally -lar -▁recorded -▁baroness -▁corresponding -▁augustus -usive -▁answering -▁recomp -▁prophets -▁vainly -▁fortitude -▁stacy -▁supremacy -▁mostly -▁day's -'r -▁sewing -▁varying -▁imperturb -▁crocodile -orious -▁outlaws -▁outlined -▁bathe -▁forgetfulness -▁uttering -▁corral -▁qualified -▁enters -▁linley -▁embroidery -▁copies -mate -▁visiting -▁visits -▁insight -▁blight -▁cabbage -▁bored -ado -▁palp -▁sneered -▁appetites -▁ulys -▁ulysses -▁kicked -▁evinc -▁travilla -▁indications -▁intimately -▁assassin -▁cy -▁cycl -▁princesses -▁partition -▁intricate -▁arbaces -udged -▁panther -▁maloney -▁duplic -▁relay -▁tito -▁eaves -▁egyptians -▁hello -▁tricks -unuch -▁eunuch -▁saun -▁redd -▁reddish -▁grumbled -▁dulc -agged -▁crowned -▁shops -▁effectually -▁compose -▁complied -egiance -▁modified -▁collins -▁fetched -anned -▁expose -▁export -▁contemplate -▁rochester -oso -▁whiskey -rea -rep -▁paula -float -▁rushes -▁prophetic -▁runaway -▁beech -etery -emetery -▁cemetery -▁jesuit -▁moore -▁rogue -▁ceremonial -▁endeavored -▁outstretched -▁hunts -▁huntsman -vard -inting -▁genev -▁cloister -uthentic -▁grudge -▁bould -▁bouch -▁baker -▁antonia -▁antony -▁terrier -▁citoy -▁bristol -▁dub -▁kirk -▁pest -▁pension -not -▁sweeter -walk -▁anthony -▁apprehensive -▁symm -▁dedic -▁galah -▁galahad -▁inconvenient -▁sunbeam -noticed -▁winding -▁tumble -▁injurious -▁requiring -▁paths -etto -▁accidental -▁cere -▁cerebr -udg -▁workers -▁yawning -▁bits -▁cosm -▁accents -tes -▁ambassadors -▁pedd -▁hoopd -▁hoopdri -▁hoopdriver -▁improb -▁improving -▁oldest -▁sweetest -▁accompl -▁disquiet -▁disin -▁exclude -ervis -▁jervis -etr -enetr -▁impenetr -▁impenetrable -▁cricket -▁carpenter -ateg -▁resounded -▁abbot -▁banish -▁midday -▁unmistakable -▁comforted -▁traversed -▁hanged -cessant -▁fools -▁monks -▁sleepy -asphem -▁blasphem -▁grease -po -point -▁horizontal -ugn -▁repugn -▁hug -▁inscription -olated -▁memoir -▁respir -▁mullig -▁levell -▁shoem -▁accusation -▁reserv -▁moderately -arried -▁unmarried -▁enterprises -▁arouse -▁obtaining -▁adult -▁insects -▁constable -▁constancy -▁imb -▁meals -▁skirmish -▁admon -▁admitting -▁whart -▁gore -▁plausible -▁polar -▁rustle -▁caravan -▁puritan -▁vick -▁jake -▁kisses -▁hap -▁borrowed -▁delaware -▁brib -▁gables -▁langmore -▁wearied -▁habitually -▁savour -▁engraved -▁hain't -acit -▁tacit -▁tumbling -▁pike -▁jenkins -auc -▁claire -inburgh -▁edinburgh -▁perverse -▁quote -▁steed -▁rheumat -▁rheumatism -iliar -▁belle -▁charlest -▁translated -▁stairway -▁ambr -▁aye -▁fanatic -eseen -▁ga -▁gaiety -▁gum -▁gush -ona -▁guinea -▁frowned -▁unhappily -▁bruised -▁nile -▁junct -▁mousquet -▁encouraging -tical -▁decency -▁peel -▁yarn -▁incarn -▁scalp -ortion -▁mold -▁raid -ologists -▁implies -▁alley -▁massac -▁cooked -▁tinder -turned -▁taunt -▁capric -▁remedies -▁humbug -▁formula -▁electrical -▁avar -▁avarice -▁supplement -▁patriotism -▁towers -▁sara -rified -▁lustrous -▁nameless -▁they'd -▁she'd -▁caterpillar -some -▁bondage -ected -▁scrambled -▁regained -▁adventurous -▁stunt -▁lass -▁embass -▁standpoint -▁hitch -ssy -▁bessy -▁hardened -▁latitude -lium -▁opium -▁precinct -▁heathen -▁elapsed -▁ted -▁tedious -▁teddy -▁ody -dyakov -▁matches -▁reaches -▁devised -ilar -▁exhilar -▁inflamm -▁promotion -amor -amorph -▁metamorph -▁conqueror -iac -ancip -▁emancip -▁visual -▁shaving -▁liberties -▁attitudes -▁forlorn -▁possesses -▁copied -▁ecclesiastic -itiveness -▁easiest -▁wales -okes -ukes -▁dishe -ec -ecd -▁gallow -▁galloway -▁clair -▁laz -▁romarin -▁absurdity -▁kentucky -▁claus -▁flem -▁flex -▁historian -▁imperious -▁tumultuous -▁coins -▁hovel -onne -▁temperance -▁smoothing -▁wyn -▁yankee -▁invitations -vert -▁statements -▁bundles -olnikov -askolnikov -▁illustrated -▁fearfully -upport -▁unheard -▁temptations -▁dusky -hum -human -▁mohammed -▁sentin -▁sentinel -▁mary's -▁hydrogen -▁charging -▁charitable -▁harass -▁discoveries -abbed -▁discard -▁signals -▁doubted -▁inspire -xim -avia -▁hedges -▁greggory -▁baltim -▁profusion -▁admiralty -▁colonial -▁scanty -▁dat -▁threaten -▁peaks -▁rebels -▁tended -▁solemnity -▁crouched -egr -▁integr -eff -▁patting -▁parsonage -▁clash -urph -▁murph -eph -hire -▁tire -▁conscientious -▁magical -▁practicable -▁practices -▁practiced -▁unob -▁aristob -▁abound -▁fellowship -▁astonishing -iature -▁miniature -▁cracked -▁singularly -▁portug -▁dove -▁holly -▁acquis -▁acquisition -▁transition -▁ivory -enuous -▁strenuous -▁meddle -▁shrubber -▁affright -▁oblique -▁creed -▁paganel -▁degener -▁spun -▁cruise -rust -▁mistrust -▁mic -ico -icing -▁sacrificing -▁noticing -▁consisting -▁beware -▁canoes -▁rites -▁uncont -▁uncontroll -▁cecily -▁captivity -▁homage -ded -▁studded -▁pitched -▁assented -▁distributed -▁anticipation -▁tolerable -tty -▁misty -▁accomplishment -▁inserted -▁sledges -▁consumed -▁toil -▁tommy -▁entreaties -▁flound -▁swine -▁inscribed -▁noises -▁missing -▁fring -▁fluttering -▁fluttered -▁obstruction -▁attic -▁attired -▁expired -▁innocently -▁terminated -▁constitutional -▁mexicans -▁ounces -▁pail -▁wooz -▁malignant -▁candy -▁honesty -▁winterborne -▁symph -▁apprehend -eline -▁representatives -▁myst -▁mystic -▁frederic -▁briar -▁recount -▁recre -armed -▁innov -▁implac -▁exceeding -▁guiding -▁gordon -requ -lier -▁combinations -▁desolation -▁arches -▁growl -▁carter -▁brake -▁philipp -▁linger -▁nobler -▁conceit -▁conceivable -▁juncture -▁shines -bell -▁campbell -▁gull -▁shivered -▁cement -▁utop -▁daunt -▁nigger -▁enveloped -▁loses -▁martians -▁busied -▁baltimore -▁mowbray -▁debray -▁indies -▁package -▁uplift -▁obstinately -▁quarry -▁alma -▁peeped -▁piazza -▁phyll -▁speechless -▁regardless -▁garn -▁coco -▁owners -▁newly -▁hillside -vention -▁intervention -▁vindict -▁translation -▁gear -▁it'll -▁nod -▁boar -iquette -▁etiquette -▁assuredly -▁resided -▁groans -▁zealous -▁seaw -▁ranch -▁analog -▁located -arine -uating -uation -acies -▁contemporary -▁raced -▁raskolnikov -▁prosecution -▁irresistibly -▁scratched -▁connections -▁enlighten -▁plateau -▁hord -▁shaded -▁bere -▁shapeless -▁ogre -▁screaming -rescent -▁bray -▁hastening -▁conductor -▁continuance -▁amazon -▁exercising -loyd -▁lloyd -▁syllable -▁disheart -▁dock -▁recital -▁janet -▁jacques -illes -▁unbelie -▁latent -▁competent -▁correspondent -▁rebellious -▁ridd -▁provocation -▁cavalcant -▁luxuriant -▁diverted -▁pestil -▁wight -▁valor -▁deposit -▁deposits -▁affectionately -▁pooh -▁thrash -▁sidewalk -▁isolated -▁swayed -▁swaying -▁designed -▁imperceptible -dam -sterdam -▁ketch -▁sash -▁wisely -adok -▁kadok -okhov -▁laboratory -▁contrive -▁intrust -▁posture -▁cigarettes -forth -▁overth -▁overthrow -arth -▁garth -▁allison -▁emotional -▁fragile -▁compos -▁promont -▁yielding -journ -▁carl -illed -▁crystall -▁populous -▁windsor -▁indol -▁spong -▁vowed -▁casually -▁legged -▁castles -▁emerge -▁gestic -▁allowing -▁yale -▁infallible -▁file -▁reared -▁yelled -▁traddles -▁conceited -▁arranging -▁bully -▁choking -▁exclamations -voc -▁hampton -▁deception -▁scornfully -▁achievements -▁accomplishments -▁colonists -▁vehem -▁vehemence -▁research -▁diction -▁recompense -▁pretense -▁tense -▁laced -▁disgraced -▁l' -▁lind -▁frontier -▁falk -▁grizz -▁constituted -▁athenians -▁lounging -▁unear -▁committing -▁lunatic -▁idleness -▁sagacity -haw -▁meditated -aterially -lethorpe -▁oglethorpe -▁tendencies -▁independently -▁indec -▁apostles -▁apostle -▁earle -▁twinkle -▁twinkled -indle -▁garral -▁unimport -▁lena -▁contend -▁spencer -enhow -▁torpenhow -▁enliven -▁barbarians -▁pilgrimage -ighthood -▁sticking -inople -▁constantinople -▁ambush -▁digress -▁owen -▁politicians -▁familiarity -▁volley -▁transaction -▁repaired -▁mace -▁dialogue -▁diary -▁engaging -▁ambig -▁enig -▁governors -▁appearing -▁venom -▁preface -▁optim -▁rakit -▁controlled -▁peterkin -▁proff -▁proffered -▁sock -▁bah -ableness -▁shameful -▁improbable -offrey -▁geoffrey -▁syria -▁loath -▁loan -▁loop -▁contemplating -▁beaver -▁instructed -uced -▁distracted -▁sheltered -▁unfit -▁git -▁gul -▁gutter -▁weird -minster -▁westminster -▁exploits -iloqu -▁brazil -aub -▁stan -▁stanley -▁sting -▁waked -▁intercept -▁twin -▁cavalcanti -ini -▁brok -▁promot -▁pars -▁parsley -▁wingfold -▁pausing -▁exposition -▁dies -▁dip -▁sep -▁diseases -▁grammont -▁grammar -court -▁cooper -▁eternally -akeney -▁blakeney -ipper -▁oysters -▁ponderous -lves -▁vancouver -▁satin -▁anew -▁athos -▁volunteered -▁cleft -▁valued -▁morrel -strel -▁minstrel -▁impertinent -▁impertinence -opper -▁arduous -▁dances -▁realise -▁trough -▁dough -▁troll -▁misc -▁tenant -▁deuce -▁bung -▁brigitt -▁undone -▁undo -▁dubl -▁declares -vin -▁bin -▁bou -cessible -accessible -ecess -▁predecess -rac -▁syrac -▁thames -▁meditative -▁corpses -▁sexes -thesis -▁baba -▁rasp -▁cherries -▁complaints -brook -▁hats -▁freak -▁smelt -▁eagles -▁peddler -▁alexey -▁securing -▁chambers -▁picnic -▁oblivion -▁lapse -▁explaining -▁unarmed -rah -▁hurrah -▁requested -▁celebrate -▁logs -▁beams -oscop -▁silken -▁considers -theast -aston -ection -▁implacable -▁epist -▁nikol -▁nikolay -▁unfair -▁collapse -▁daint -▁dens -▁calculation -▁calculations -resses -▁diminish -▁geography -▁deceiving -▁chieftain -▁secondly -▁trinity -▁pussy -▁suggestive -▁capricious -▁spied -▁sinner -▁vein -▁hysterical -▁radical -▁cust -▁condemnation -▁plays -▁seeds -▁sorrows -▁wilmot -▁blot -▁svidrigailov -▁elders -thmetic -▁cerebral -▁brend -▁brenda -▁brack -▁integrity -▁witty -▁reconcili -▁ross -▁praying -▁wherewith -▁batteries -▁raises -▁multiplic -▁corruption -▁wright -▁wrung -▁strung -▁slung -▁stung -▁vo -▁rakitin -▁dublin -▁rim -▁brilli -▁brilliancy -▁ni -book -▁nurses -▁brushed -▁solicitor -▁christians -▁magp -▁magpie -▁dispersed -▁whispers -▁substituted -▁mounting -▁using -▁longing -▁longed -▁glaring -▁friend's -▁graz -▁swiss -evik -shevik -▁inhabit -▁exempl -▁dependence -▁verb -▁version -▁veranda -▁unearth -▁genus -▁euph -▁lac -▁thorac -▁loki -▁thoris -▁servad -▁servadac -▁servitude -▁salad -▁yolk -lan -▁monarchy -▁dominions -aternity -▁intrigue -▁scarf -▁josie -▁offers -lete -▁disgusted -▁darnay -▁moves -▁deign -▁cowardice -▁paced -▁rolls -▁calamities -▁jessica -encing -▁volupt -▁tempered -▁moment's -▁shares -▁squares -aches -▁adverse -main -▁delayed -▁winged -▁detained -▁horse's -▁cros -▁crosb -▁crosbie -▁stirrup -▁glencora -▁accl -▁terrify -▁provoked -▁switzer -▁switzerland -▁executioner -▁blossoms -▁affirmative -▁clem -▁exterm -mer -azad -razad -▁shahrazad -▁abnormal -▁regulated -▁kite -▁kay -ancing -▁problems -▁bidden -▁contracted -▁giles -▁indecision -▁richly -avors -▁proven -▁combatants -▁marqu -▁marquise -▁bands -▁greeks -▁cups -▁impuls -spread -▁virgilius -▁chatting -▁cowardly -▁rely -▁pruss -▁grade -▁quoted -▁octavia -▁eb -▁ebony -▁bony -▁suffoc -▁dive -grown -▁abused -▁dante -ele -▁steele -▁steer -▁sensitiveness -▁sensual -▁eyelids -▁flourished -▁bethought -▁battal -▁imitated -lant -▁indig -▁indo -▁indoors -ero -▁mortimer -▁persistently -▁hoard -▁shrieked -▁rick -▁perempt -▁thoracic -iciency -▁johnnie -▁nanny -▁barley -▁adapt -▁crowds -▁persuading -▁fading -▁knighthood -▁ties -▁clap -okesmith -▁recept -▁nieces -▁agency -▁scraps -▁incredulous -beard -▁cars -▁indulgent -▁pud -▁pantry -▁massachusetts -▁obscured -igured -▁rook -▁reinforcements -▁soliloqu -▁rustic -▁impending -▁grandchild -▁fray -▁perkins -▁recommendation -▁maze -▁mes -▁owes -▁elf -▁disf -▁shoved -▁pillar -▁pelop -▁geome -town -inn -▁vern -▁mulligan -▁magazines -uble -▁relish -▁speaks -▁sacram -▁vase -▁da -▁taciturn -▁zero -▁halting -▁devoured -▁dismounted -▁lasted -▁naz -▁ern -▁remonstrance -▁monkeys -▁stimulated -▁concerns -▁eruption -▁eclip -▁sheets -▁repulsive -▁manufactured -aneously -▁tremendously -▁stupend -▁stupendous -▁sherwin -▁sumter -▁doubts -▁debts -iser -▁palliser -▁martine -▁martineau -▁strangest -▁gilded -▁respected -▁uncon -▁uncanny -lyn -▁evelyn -▁shaved -▁particulars -▁wonders -▁ernest -▁bursting -▁gratefully -aging -tieth -▁thirtieth -▁porcel -▁concret -▁blues -▁edged -▁edifice -▁brooding -▁arnold -▁kang -▁kangar -▁kangaroo -▁tease -▁chilly -▁dean -▁respective -▁craving -▁ancients -▁anecd -▁paved -▁essex -▁splinter -▁vicom -▁vicomte -▁estimation -▁estimated -▁confirmation -▁contemporaries -▁males -▁descriptions -▁crescent -▁exempt -▁blond -enance -aintenance -▁extravagance -▁insensibility -▁permanently -▁squat -burton -▁warburton -▁dissipated -▁southw -▁brew -uptial -▁impartial -▁guides -▁tides -▁impudent -▁spright -▁dal -maker -▁contemptuously -▁chuckle -▁affront -▁profoundly -▁waterfall -▁nightfall -▁fearing -▁starv -▁starvation -▁vacation -▁compensation -▁secular -▁rawdon -▁loosen -▁giddy -ensively -▁unbounded -ills -cknowledgment -▁conjunction -▁protector -▁beckoned -▁lacking -▁tourn -▁cro -▁apollo -▁didst -▁industries -▁puck -▁redmain -▁cultivate -▁bewilder -▁bewilderment -▁frames -▁persecuted -▁extric -▁polona -▁disadvantages -▁heights -▁ducks -▁submarine -ulus -▁aristobulus -▁grated -▁richest -▁mainly -▁inflamed -▁custody -▁abundantly -▁ozma -▁sleek -▁organism -man's -▁heaped -▁skimm -▁baw -▁springing -▁straightforward -pton -▁immeasurable -▁failing -racticable -▁peculiarity -▁disfigured -▁corridors -▁oval -▁edna -enna -▁vienna -▁poultry -▁riders -▁peers -▁allusion -▁greg -▁gregory -▁burglar -shot -▁banker -▁mouths -▁adore -▁grigg -▁confide -▁diffused -▁denisov -▁consequent -ugging -▁stumbling -▁cylind -▁rendezv -hal -▁exceptional -▁exceptionally -▁grape -▁ivanovitch -▁pane -▁sane -▁paternal -▁conversations -▁safer -▁menace -▁fidget -▁alexandria -▁compliance -▁complais -▁hyrcan -▁sorely -▁indicating -▁indicates -▁raptur -▁unsuccessful -my -▁jock -▁mell -▁mellow -▁sow -▁repeatedly -▁ignore -▁sensibly -▁voluptuous -▁insin -▁goin -▁shin -▁ron -▁ronald -▁prosecute -▁sulk -▁poach -▁gravitation -pan -▁saucepan -▁gan -▁ganem -▁thong -▁embarked -▁bubble -▁amusements -▁makers -▁quadr -▁betsy -▁indifferently -▁ardently -▁edgar -▁sliced -▁slant -▁discourage -▁wanderings -▁milner -▁prompted -▁restrained -▁judgement -▁sue -▁muskrat -▁interpreter -▁fernand -▁cigars -▁displeased -▁consul -▁consort -▁reub -▁reuben -▁brighter -rians -▁strew -▁strengthened -▁stations -▁discovering -▁smoothly -▁blizzard -olic -▁adelaid -laid -oodoo -▁trotting -▁approxim -▁proxim -▁propped -▁parasol -▁dimm -▁shimm -▁blades -▁folliot -▁phoenic -▁pansy -aract -▁cataract -fact -▁unsatisfact -▁nero -▁pregn -▁sister's -▁governing -▁grasping -bler -▁crus -▁mosquit -▁unsatisfactory -▁athen -▁buttons -oons -▁diale -▁dialect -▁subsistence -▁fastidious -▁raged -▁gleamed -▁contemptible -▁louisa -▁plea -estra -chestra -viet -▁petrified -▁cair -▁irresolut -▁hammock -cock -▁mouthful -ene -▁yose -▁yosem -▁yosemite -moil -▁turmoil -▁topic -▁volcanic -▁volcano -▁schoolmaster -▁sacraments -▁maintenance -hall -▁metall -▁metallic -▁pavilion -▁dejection -▁carving -▁zepp -▁zeus -▁swoop -▁sloop -▁historic -▁histories -▁historians -▁modes -▁dizzy -▁transactions -▁administered -▁pallid -▁diverse -▁gallows -fellows -oeuv -owder -powder -▁gunpowder -'p -'pose -▁s'pose -▁site -▁tony -▁remnants -▁remnant -outhed -▁wharton -▁achievement -▁quilt -▁explic -▁categ -▁strateg -▁cordelia -▁chord -▁twain -▁orthodox -▁militia -▁postpon -▁newcom -▁clasping -▁torres -▁bruno -glass -▁mowgl -▁lobby -▁unexpl -▁valleys -▁constraint -▁taint -▁mink -▁unlo -▁unlim -▁unlimited -▁romay -▁romayne -▁ingenuous -▁stuffed -▁contributed -▁louisiana -▁representation -▁tumbl -▁charcoal -▁insurance -▁inevitably -▁popped -▁poign -▁tangle -uding -▁speculative -▁enchan -▁rendezvous -▁hauled -▁gestures -▁damascus -▁oliv -▁almond -▁adjourn -▁adjut -▁tut -utable -▁other's -ernel -impernel -▁describing -▁describes -ato -▁overh -▁overhang -▁porcelain -▁lawful -▁infect -▁primarily -▁wearily -▁vulture -isle -isance -▁alteration -▁disapproval -▁veteran -▁outburst -amiliar -▁unfamiliar -▁jet -▁prescription -▁presided -▁intrusion -▁imaginable -roy -▁gilroy -▁advocate -▁curd -▁smould -▁sooth -▁rosalind -ogue -▁compr -▁alac -▁alacr -▁alacrity -▁shrive -▁arguing -▁religions -▁mills -▁scow -▁scald -▁scull -▁scoff -▁andrea -▁revive -▁grandp -▁cashier -▁cable -▁revolving -▁hopeful -▁hypocris -▁goth -▁glimmering -▁decisions -▁colleges -▁lilac -▁droll -▁drought -▁nu -▁nuisance -▁wuz -▁functions -vonne -▁yvonne -▁nev -▁improv -▁yor -▁yorke -▁nursing -▁drooping -▁dundee -▁sometime -▁discons -▁disconsolate -▁discom -▁discomf -lane -ls -▁borls -▁borlsover -▁hindrance -▁clove -rove -▁clare -▁astr -▁chir -▁uninterest -▁unnoticed -▁unaware -▁unres -▁fried -▁unlocked -ustin -▁investigate -▁grumbling -▁bot -▁philamm -▁butcher -▁brightened -▁straightened -▁metals -▁installed -▁wench -▁rais -▁piteously -▁piteous -▁numb -▁appeas -▁acquitted -▁fruitless -▁fearless -▁jenning -▁jennie -▁minnie -▁acceptable -▁nuptial -▁majesty's -▁eugene -▁appealed -▁wyat -▁couplets -▁bracelets -▁doorstep -criptions -▁convulsions -▁fashions -▁appointments -▁requirements -▁contrivance -▁earthen -▁trunks -▁perspective -ierra -▁sierra -icorn -▁malicorn -▁maternal -▁declining -▁definitely -umbent -▁deport -▁depot -▁kara -▁beneficial -▁judicial -▁laund -▁elliot -ife -▁lashed -▁kennic -▁kennicott -cott -cass -▁poetic -▁epic -▁realization -▁location -▁reconstr -▁disinterested -▁professed -▁professors -▁people's -▁blister -▁dimp -▁whimp -▁curses -▁cured -▁curly -▁dancers -▁customers -▁selling -▁regularity -▁shameless -▁joam -▁joanna -iber -▁astound -▁bolted -▁shrug -▁stagn -oncile -▁reconcile -▁volition -▁animation -mare -▁nightmare -▁reeds -▁voiced -▁officials -▁sustenance -▁vicksburg -▁limitation -▁braid -▁awakening -▁infants -▁amory -▁conne -▁connex -▁abd -▁inflex -▁whoso -▁vanishing -▁mimi -▁trodden -▁irritable -▁irrational -▁irritating -▁parch -▁parchment -▁fum -▁dissolution -▁blindness -▁unbear -▁glug -▁illustrate -▁consulting -▁consultation -▁impulsively -▁rupert -▁sorrowfully -▁shrine -issy -▁variations -▁whereof -▁boundary -odora -▁profane -▁colossal -▁teas -▁meetings -▁skirts -▁pistols -▁incline -▁vivacity -▁camels -▁camer -▁huck -▁clamor -▁intoxicated -▁zadok -▁bolshevik -▁extraordin -▁extraordinarily -▁communicating -▁ugliness -▁ambros -▁dess -▁dessert -▁wailing -▁retiring -▁rye -▁rames -▁rambl -▁rudder -▁rinald -▁doubtfully -▁gypsy -▁bedstead -▁gib -▁foref -▁enlarge -▁lect -▁heaviest -▁renaud -▁infid -▁victual -▁speedy -▁cheerfulness -▁carelessness -▁dubious -▁unite -▁distorted -▁penniless -▁veritable -▁vertical -▁bates -▁grimaud -storm -tingham -▁seamen -▁citoyen -ampton -astrous -▁disastrous -▁gond -▁scanned -utomatically -▁proprietors -▁spilet -▁spilett -▁crucif -▁crucifix -▁indiscretion -▁constell -▁armand -interrupted -▁margery -▁quivered -▁vestig -▁carrie -▁entertaining -▁muff -ustr -▁distast -▁entail -▁robust -▁mused -▁successor -▁smelling -▁sixpence -▁vas -shaw -▁avow -uder -▁intruder -▁gentry -▁berry -▁scruples -▁needles -▁conclusive -▁perceptions -▁leopard -▁alarming -▁forefing -▁forefinger -▁dealing -▁avon -▁avonle -▁incomplete -▁grandson -▁rascals -▁penitent -▁folding -▁foreb -▁forebod -▁luther -ugment -▁carfax -▁carcass -dust -▁sawdust -erby -▁bounderby -▁football -itial -feit -▁dotted -▁pedest -▁men's -▁socialist -robor -▁corrobor -▁inquiring -▁recognizing -▁apothe -▁frere -▁survival -▁consecrated -▁exasperated -▁fathers -▁proofs -▁boister -▁brad -▁identify -▁pouch -▁abolished -▁dinah -▁crouching -▁recovering -came -▁woollen -rogr -▁maryland -amour -▁cabbages -▁degrad -▁hannib -valanch -▁leisurely -▁condescend -▁seals -▁crystals -▁shuff -▁shaven -▁popularity -▁yon -▁retic -▁sallie -▁benefactor -▁surpl -▁wars -▁procuring -isation -▁micr -▁glittered -▁enduring -▁rogers -▁gerald -▁swed -▁sweden -▁swagger -▁flickering -pring -spring -▁offspring -▁cartridges -▁creaking -▁permitting -irtier -▁shiny -▁sly -▁plut -▁glut -▁tribut -tering -▁bowing -▁grinding -▁snowy -▁mov -▁rebu -▁rebuke -▁desiring -▁herds -▁grapp -▁gren -▁grub -▁griff -▁reproduce -▁clavering -▁congratulated -▁dangling -▁harmonious -▁patriotic -▁preposterous -▁werner -haust -▁inexhaust -keepers -▁drawers -▁maids -▁collars -▁uses -▁bib -▁bibbs -▁balk -▁omnib -▁omnip -▁subdue -▁beauchamp -▁unsh -▁unin -▁headache -▁basement -▁detachment -▁inventions -▁imperson -▁impre -▁impot -▁impracticable -▁resear -▁researches -▁offense -▁bravest -▁canst -tically -athetically -▁discred -▁shred -▁voted -▁surveyed -affect -▁myles -▁dishonor -▁posts -allows -▁swallows -▁products -owerberry -▁virtually -▁thrush -▁bobb -kirt -▁ink -▁grandchildren -▁concentration -▁egot -▁pianof -▁pianofort -▁befallen -▁tipped -▁gripped -▁courageous -▁accommodation -▁walcott -▁yearning -▁doubly -▁tactics -▁smothered -▁scores -▁compel -▁countenances -▁shak -requently -▁gladness -▁shifted -▁hindered -▁remin -▁reminisc -▁needless -▁gaining -▁ruling -▁forcing -▁wrapp -▁indecent -▁darrow -anter -▁pleasanter -▁emble -▁emblem -▁pebble -▁nit -▁youngster -▁noiselessly -▁titan -▁cran -▁resembles -▁despotism -▁solace -▁penance -▁grating -▁von -▁charleston -▁surg -▁hoot -▁representing -lokhov -▁emperor's -▁cannib -▁frankness -archy -▁projectile -▁unimportant -▁hydra -cox -▁wilcox -▁dorm -▁balancing -▁bac -▁bacchus -▁poplar -▁phyllis -▁goldsmith -▁incomparable -▁oblong -▁gour -coming -▁legislative -▁provoke -▁invinc -▁invincible -erting -▁asserting -▁reverie -▁robart -▁saucy -▁glacial -▁nitrogen -▁negotiation -▁rosalie -osa -osph -▁phosph -▁degradation -▁stark -▁crock -▁phosphor -▁collector -▁overwhelming -▁counting -▁arabian -▁freem -▁surviv -rimonial -▁scrape -▁troph -▁suffers -▁quadrup -▁scientists -▁baile -▁haines -dry -▁sundry -▁jewelry -▁charmed -▁destroying -▁shivering -▁crushing -▁contentment -▁consign -▁enlarged -rupulous -▁syrup -▁marj -▁marjor -▁inherent -iating -▁ll -▁randolph -▁wren -▁burne -▁cupful -▁madman -▁nursed -▁profuse -▁knotted -▁advisable -▁ludov -▁overcame -▁unspeakable -▁hampshire -▁spade -adequ -adequate -▁tufts -▁poisoned -▁founded -▁critically -▁dose -▁impose -▁imply -▁imple -▁implements -▁sweetly -▁rightly -▁invariable -ifiable -▁persecut -a'n't -▁sha'n't -path -▁seasons -▁bricks -▁gulp -▁publicly -▁barefoot -▁objected -▁portugues -▁ford -▁occurs -▁commons -▁mattie -▁encomp -▁defending -▁recollections -▁humiliating -▁softer -▁dissent -▁disclosure -▁uppermost -▁hatchet -▁galloped -▁enm -▁enmity -▁chirp -▁influential -▁foc -▁smitten -vir -▁sagacious -▁scientist -▁boundless -▁professions -▁sowerberry -▁overthr -▁overthrown -owness -▁adored -ctor -▁invested -▁convincing -▁merchand -▁pollen -▁wintry -▁scruple -▁scrupulous -▁fabulous -▁solar -▁shod -▁dedicated -▁crowns -▁athens -▁settlements -▁epithe -▁triang -▁clang -▁thickly -▁denounced -▁uncouth -▁jane's -▁grandpap -▁neelie -▁relates -rithmetic -▁intrep -airy -▁dairy -chair -▁armchair -▁unintelligible -▁goats -▁dandy -▁amy -sed -ssed -▁tossed -▁roofs -▁shutters -▁hailed -▁thirteenth -▁conveniently -▁antagonist -comb -▁expans -▁exposing -▁interfering -lihood -retches -▁wretches -▁circling -▁packing -▁abashed -▁splashed -ree -▁lashes -▁constitutes -▁farmhouse -▁odys -▁odysse -▁odysseus -otic -▁cot -▁customer -▁quer -▁queried -pper -ker -▁traveler -▁patrol -▁dickens -note -▁footnote -▁anecdote -▁anecdotes -▁hardy -▁travelers -▁twould -▁succour -▁lucie -▁silas -▁collision -▁collaps -▁liar -▁siber -▁golf -▁sideways -▁convictions -▁wiping -▁storms -▁gilt -▁ging -▁ginger -▁clause -▁toilet -▁ardour -▁everyday -▁festivities -▁materially -▁gigg -▁excav -▁striped -▁overlooked -▁lair -▁leonard -▁profile -▁philist -▁philistines -▁slammed -▁plunging -▁shifting -▁belgium -▁illusions -▁marchion -▁marchioness -▁glowed -▁geneva -▁wyld -▁unanimous -▁infamous -▁galile -ygen -▁oxygen -in's -▁altitude -▁encompass -▁bitten -▁designs -▁crane -▁grievous -▁succeeding -▁josiana -▁trampled -▁handled -▁musicians -grow -▁exeter -▁accepting -herent -▁molten -▁fireside -wald -▁oswald -▁herald -▁wheeled -▁graduate -▁pointer -▁crusoe -▁ingredients -ileas -▁phileas -▁muttering -▁joking -▁jokes -▁congen -▁privacy -▁wines -▁hier -▁animosity -▁mocking -▁battered -▁freez -▁jerked -▁ruffled -▁rural -▁vasili -lit -▁dwind -▁sherwood -▁ruffian -▁bramble -▁scramble -▁sanguine -▁teap -▁teapot -▁metropolis -▁royalty -▁humpty -imentary -▁cynth -▁cynthia -▁gluck -▁currants -▁currency -▁avonlea -▁perry -▁programme -▁objective -▁murphy -▁prettier -respect -▁disrespect -uspect -▁unsuspect -▁mutually -▁knitted -▁returns -▁ardor -zevir -▁adherents -aments -erned -▁discerned -▁unconcerned -▁aspirations -▁crump -▁snub -▁samson -▁thomson -erd -imerd -▁shimerd -▁naom -▁naomi -▁koodoo -▁rafter -▁georgina -▁bounded -▁flickered -▁deacon -▁brav -▁mong -▁natty -▁dumpty -▁flattery -▁flattered -▁errone -▁coats -insula -▁peninsula -▁bonaparte -▁clearer -▁cheque -▁gravy -▁append -▁napkin -▁disapprove -inois -▁illinois -▁prisoner's -▁inquiringly -▁fringe -▁judicious -▁stride -▁corps -▁tus -▁tuske -▁tuskege -▁tuscan -▁discomfiture -▁greas -▁gret -▁grettel -▁complet -▁completion -▁combine -peck -▁blended -▁bankrupt -▁occupants -▁imprudent -▁mutil -▁disincl -▁gawain -▁crevice -▁grabbed -▁promontory -▁selina -▁frustr -▁weaker -col -▁malcol -▁jol -▁jars -▁malchus -▁amsterdam -▁calmness -equality -▁infinity -▁bridal -▁token -▁bohemia -▁peggot -▁peggotty -▁ridges -▁whims -▁traditions -▁replen -▁satan -▁belly -▁furthermore -▁soothe -▁undisturbed -▁trenches -▁d'ye -shop -shadow -owski -▁senators -▁complacency -▁physiogn -▁betook -▁rameses -▁huss -aterina -▁katerina -▁inexpressible -▁flatten -▁comed -▁comedy -▁poyser -▁cheery -▁cheering -▁peeping -▁travels -▁darwin -▁myth -growth -▁particle -▁mingle -esley -ulity -▁incredulity -▁oats -▁hir -▁hild -▁surfaces -▁pins -▁achieved -▁encumber -▁concurrence -▁untrue -▁untruth -▁icy -▁strides -▁knave -▁knuck -▁knoll -▁diplomacy -▁iceberg -▁locust -▁locom -▁termed -roc -▁buccaneers -▁breeding -▁lynx -▁implored -▁adjusted -▁yorkshire -▁tam -editary -▁hilt -▁nevada -▁heath -▁heathcliff -▁varn -▁variation -▁wilmington -▁wicker -▁expan -▁expanse -▁triangle -▁lining -▁tucked -▁peculiarities -▁horseman -▁chairman -▁raging -▁benefic -▁sentinels -▁dived -▁diseng -▁langdon -▁refreshed -▁mechanics -lled -▁contribution -▁contag -▁contention -agu -▁stork -▁stored -▁entrusted -▁lookout -▁gaston -▁afore -▁afresh -▁blocks -▁speculations -▁whimsical -▁blazed -▁squeezed -▁bounds -▁blindly -▁nar -▁quench -▁wrench -▁mingling -▁randal -aled -▁rewarded -▁shelves -▁cloudy -▁watches -stock -▁marine -thorn -▁drenched -▁corrupted -▁transient -▁aimed -▁vividly -▁steamboat -cular -▁guise -▁enquire -▁enri -▁enriched -▁abhorrence -▁prospects -▁fits -▁morals -▁pelopon -▁peloponnes -▁besieged -▁bearings -▁impregn -▁swelling -▁swelled -▁heaps -▁knaps -▁knapsack -▁anglic -▁despatched -ivoc -▁obsequ -tles -▁turtles -▁pursuers -▁riv -▁sparks -▁sympathetically -▁comments -zed -▁concentrate -▁wrenched -▁casket -umbr -▁precipitous -▁caprice -▁seaman -▁involving -▁potato -▁antipater -▁antipath -▁locksley -▁peev -▁peevish -▁microscop -▁dinars -bec -▁coqu -▁unquestionably -▁wares -▁rats -▁omnibus -▁plucked -▁ascertained -▁marsden -▁daoud -▁occasioned -▁owned -▁quartz -auff -auffeur -▁resisted -▁comprehending -▁whizz -▁blackness -▁boisterous -strained -▁unanimously -▁dejected -▁elim -▁elzevir -▁precipices -rose -▁scrupulously -▁rents -inished -▁unfinished -▁habitation -▁predic -▁predicament -▁footh -▁wroth -▁unavail -▁keel -▁avowed -▁shopping -▁shakes -▁precipitate -▁assurances -▁fasten -▁wistful -▁wistfully -▁accomplice -▁intrigues -▁bewitch -▁silenced -▁vampire -orily -▁satisfactorily -▁greedily -▁tremor -▁simpler -▁explicit -▁uncl -▁massacre -▁appertain -▁disgraceful -▁gig -▁recipient -brand -▁divert -meg -▁nutmeg -▁scroll -▁reiter -▁justly -▁contrivances -▁relatively -▁realizing -▁uncovered -▁rhod -▁expost -▁geron -▁geronim -▁geronimo -heim -▁sut -▁suzan -▁exit -▁lurking -▁mouthed -▁dictionary -▁groove -▁unaffect -▁drowsy -▁distribute -▁cope -▁core -▁spher -ipher -▁madden -▁launched -▁piqu -▁pose -▁respectability -▁carpets -▁assistants -▁protestants -▁harvard -▁sliding -▁bubbles -ofore -etofore -▁transcendent -▁insupport -▁institute -▁creditors -▁dorry -▁unsc -▁unscrupulous -▁osc -▁osmond -ienne -bon -▁irrevoc -▁observance -▁schoolboy -▁jub -endarm -▁gendarm -aug -▁cleanliness -▁deficient -▁reformation -▁skipper -▁paddle -▁unprof -▁martyrdom -▁inexhaustible -▁instincts -▁instinctive -▁processes -▁exuber -▁memorand -▁chum -▁chagr -▁hurons -▁bash -▁crumbling -▁alcoh -▁acorn -za -▁alg -▁embodied -▁stip -une -▁shun -▁shedding -▁blooming -▁tet -▁tete -▁tending -▁debauch -▁ceases -▁reversed -▁enables -▁aggre -▁aggregate -reast -▁abreast -▁inconsistent -▁exploit -▁spit -▁sph -▁sphin -▁sphinx -▁appalling -▁fling -▁filthy -▁monopo -▁monopoly -▁excitedly -▁schoolroom -▁woozy -▁patchwork -▁purest -▁snowdrop -▁individually -▁pinn -▁pinned -▁whereabouts -▁decorations -▁fade -romising -from -▁marvelled -▁conveniences -▁diminution -▁bee -▁scornful -▁scrutiny -▁incumbent -▁planets -▁raddle -▁conversion -▁sneer -▁nothin -▁gride -▁pensive -▁defensive -▁defer -▁defy -▁defied -▁cellars -▁missionaries -▁prone -▁zone -▁connexion -lem -▁enigm -▁meekly -▁equator -▁comply -▁shoemaker -▁derive -▁extort -▁barbarian -▁deafening -▁unearthly -▁practising -▁hilar -▁hive -▁adventurer -▁adventurers -▁superiors -▁loosed -▁securely -▁pimpernel -▁inwardly -▁sickly -▁correctly -▁recesses -▁friction -▁adjustment -▁persh -▁assim -▁assimil -▁vows -▁divined -▁glimmer -▁trimmed -▁rivalry -▁gallantry -▁agricola -▁oakes -▁redoub -▁captain's -▁gems -▁proceeds -▁fails -▁flocks -▁mists -▁gruff -▁saus -▁miranda -▁frogs -ilib -▁equilib -▁equilibri -▁equilibrium -ils -▁breaks -ante -▁kenwigs -▁agreeably -▁lavender -▁brimm -▁fras -▁fraser -▁retribution -▁amiss -▁mand -akers -▁jov -▁jovial -▁increases -▁preferable -▁gobl -▁fragoso -▁sentry -▁prettily -▁buying -▁pacing -▁ludic -▁ludicrous -▁psychic -▁urging -▁omnipot -▁walt -▁waltz -▁novels -▁mug -▁tension -▁percent -▁accessible -▁fixing -▁disturbing -▁affecting -▁drowning -▁bayon -▁binding -▁lanc -▁maple -ilghai -▁nilghai -▁devils -▁expended -▁enjoined -▁heroism -▁adjutant -▁shrugging -▁streaked -▁hereditary -▁lag -▁lagoon -▁spout -▁furt -▁clod -▁patronage -▁stiffly -▁derision -▁demure -▁demol -▁demoral -oles -▁includes -▁instig -▁ensuing -▁deity -▁contending -▁geographical -ton's -▁mik -▁greasy -lands -▁flashes -▁conversed -▁fainted -▁postpone -▁respecting -▁meditating -abble -▁absently -▁fervently -▁bathsh -▁subt -▁sample -▁steered -eb -▁bathsheb -▁seduced -▁vernon -▁joll -▁perforce -▁poised -▁advertised -▁skinner -▁hopelessly -▁rudely -▁modestly -▁pleading -▁clutch -thood -▁reek -▁gothic -▁footmen -▁bunder -▁bundercombe -▁melbourne -▁dissolve -▁graveyard -▁pamel -▁blouse -▁grouse -▁resident -▁mama -▁southwest -▁favors -▁vibration -▁bask -▁unr -▁vestib -▁vestibule -▁ballast -sham -▁feversham -▁gresham -▁worms -▁spears -▁proves -▁sherry -▁crag -▁crashed -mphithe -▁attempting -fit -▁outfit -iesthood -▁priesthood -▁authors -▁authorized -▁possum -▁deprec -▁depreci -civ -▁scur -iner -▁paler -▁gout -▁traps -▁ghosts -▁unprepar -▁corrected -▁concrete -▁unatt -▁pratt -▁rides -▁grandma -▁roadside -lliver -▁dye -▁cane -venture -adventure -▁unfavor -▁separating -▁athenian -▁blithe -▁thrilling -▁complacently -▁scorched -▁nazareth -▁estra -▁calend -▁calendar -▁suspend -▁suspension -▁thrilled -▁assailed -▁tom's -▁misle -▁prevailing -▁introducing -▁suzanne -▁morocc -▁pion -▁pioneer -▁poetical -▁appell -▁temporal -▁outrageous -▁preparatory -arken -▁dexterity -▁surgery -▁threats -▁woes -▁devise -▁treatise -▁cyp -▁spectre -▁reliable -▁erroneous -asuring -▁wisc -▁wiscons -▁wisconsin -▁wisp -▁therein -▁raisins -▁sord -▁sordid -▁sissy -▁warren -▁metaphysical -▁hypocrisy -▁astronomy -▁burgundy -▁servile -▁deportment -▁scarborough -ogne -lov -▁sown -▁darker -▁memorandum -▁boldness -▁tips -▁ruggles -▁strut -▁scut -▁plush -▁presty -faction -▁dissatisfaction -▁recreation -▁implicit -▁wilful -▁sprawling -▁carson -▁powerless -▁tho -▁thinner -▁thaw -▁praising -▁assemblage -belie -▁disbelie -▁greeb -▁veiled -▁curry -▁pricked -▁vapor -▁razor -▁rubies -▁doun -▁dounia -▁hullo -▁dissim -▁sah -▁hardships -▁livelihood -▁condescended -mann -▁paltry -elope -▁ful -ionys -▁dionys -▁dapp -▁conway -▁pathway -▁diligently -augur -▁finely -▁detestable -▁ribs -▁jord -▁jordan -▁frant -▁frantically -▁hobbled -▁vehicles -▁raffles -▁projected -▁tartar -▁dashing -▁jailer -▁catalogue -▁workmanship -▁confederation -▁smear -▁fulfilment -▁ryn -▁spat -▁spokes -▁sped -▁mit -▁surprises -▁afar -▁gaul -rocious -▁mythology -▁diplomatic -▁preaching -▁wrinkles -▁knightley -▁prejudiced -abul -abulary -▁vocabulary -onnected -▁grenville -thell -▁othell -▁ceasing -▁reckoning -imetheus -▁epimetheus -▁politician -▁decomp -▁marion -▁marionette -chamber -▁eclipse -▁provident -▁seating -▁chry -▁uncompromising -▁tankerville -▁stowed -▁stella -▁polonaise -▁hopping -▁curling -▁skir -▁individuality -▁denmark -▁assailants -▁tenants -▁louv -olf -▁insure -▁quack -▁confront -▁symmet -▁anita -▁finite -▁prefers -▁immov -▁merits -▁saints -inent -▁imminent -rent -▁fervent -▁screened -▁unwillingly -▁hetty -braska -▁nebraska -▁moraine -▁maslov -▁cheered -▁announcing -▁municipal -aboo -▁diving -▁preoccupied -▁shuttle -▁attacks -▁crippled -▁mitig -▁pallor -▁implore -▁cray -▁overjoy -▁offerings -▁mortar -▁plumage -▁reappeared -▁juan -▁ordinance -▁reassuring -▁capturing -esee -▁adela -daughter -▁abiding -▁bamb -▁bamboo -▁amazingly -▁venturing -▁undertone -▁undertaker -▁tarling -loe -▁chloe -issure -▁fissure -▁wrapping -▁chinam -▁mineral -▁inflicted -▁committees -▁kr -▁caps -▁disappearing -▁throbbing -▁poland -▁resid -▁reside -▁soda -▁swain -▁swans -▁beauteous -▁loaves -▁norah -▁tilted -▁frowning -▁reclining -▁melting -▁intrinsic -▁ethere -▁stimulating -▁tissues -▁inconveniences -▁astounded -▁verm -▁similarly -▁ester -▁esterbrook -▁opposing -cturn -▁carrier -bald -▁cald -▁trudged -borg -separable -▁omen -▁omit -▁exultation -▁supporting -▁depraved -▁micaw -▁micawber -oxys -▁paroxys -▁sophia -▁scented -▁editorial -▁geological -afty -▁crafty -▁homesick -▁ravages -people -▁florina -▁hintock -▁skirted -▁rejoinder -▁handling -▁humorous -ese -▁geese -▁dipping -▁whipping -▁sulky -▁laf -▁lafitt -▁impair -▁sloping -ety -▁scy -▁scythe -▁harlow -▁subscription -▁unbearable -▁delirious -▁attainment -▁ignored -▁loathsome -▁lighter -▁telford -▁ank -▁ankle -▁fork -▁puzzle -▁ankles -▁brus -▁brag -▁merchandise -▁surmise -▁despoil -▁ornamented -▁stiffened -▁bunches -▁role -▁rogues -▁weakened -▁indirectly -▁fortifications -▁dent -▁embank -▁woodpeck -▁roaming -▁beaming -▁admiringly -▁florida -▁hunch -▁lengthen -▁irrit -▁fruitful -▁impotent -utation -▁decoration -▁ingratitude -engthe -engthening -▁favoured -▁reced -▁turbulent -▁midwin -▁hyrcanus -▁gourville -▁behaving -▁ting -▁tinge diff --git a/lingvo/tasks/car/BUILD b/lingvo/tasks/car/BUILD deleted file mode 100644 index df8874a3..00000000 --- a/lingvo/tasks/car/BUILD +++ /dev/null @@ -1,562 +0,0 @@ -# Car related tasks. - -package(default_visibility = [ - "//visibility:public", -]) - -licenses(["notice"]) # Apache 2.0 - -py_library( - name = "car_layers", - srcs = ["car_layers.py"], - srcs_version = "PY3", - deps = [ - ":car_lib", - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:py_utils", - ], -) - -py_library( - name = "detection_3d_lib", - srcs = ["detection_3d_lib.py"], - srcs_version = "PY3", - deps = [ - ":geometry", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/tasks/car/ops", - # Implicit numpy dependency. - ], -) - -py_library( - name = "detection_decoder", - srcs = ["detection_decoder.py"], - srcs_version = "PY3", - deps = [ - ":detection_3d_lib", - "//lingvo:compat", - "//lingvo/core:py_utils", - ], -) - -py_test( - name = "detection_decoder_test", - srcs = ["detection_decoder_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":detection_decoder", - "//lingvo:compat", - "//lingvo/core:test_utils", - ], -) - -py_library( - name = "detection_3d_metrics", - srcs = ["detection_3d_metrics.py"], - srcs_version = "PY3", - deps = [ - ":summary", - ":transform_util", - # Implicit PIL dependency. - "//lingvo:compat", - "//lingvo/core:metrics", - "//lingvo/core:plot", - "//lingvo/core:py_utils", - # Implicit matplotlib dependency. - # Implicit numpy dependency. - # Implicit Tensorboard dependency. - ], -) - -py_test( - name = "detection_3d_metrics_test", - srcs = ["detection_3d_metrics_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":detection_3d_metrics", - ":transform_util", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_test( - name = "detection_3d_lib_test", - srcs = ["detection_3d_lib_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":detection_3d_lib", - "//lingvo:compat", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_test( - name = "car_layers_test", - srcs = ["car_layers_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":car_layers", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - ], -) - -py_library( - name = "ap_metric", - srcs = [ - "ap_metric.py", - ], - srcs_version = "PY3", - deps = [ - ":breakdown_metric", - "//lingvo:compat", - "//lingvo/core:hyperparams", - "//lingvo/core:metrics", - "//lingvo/core:py_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "calibration_processing", - srcs = [ - "calibration_processing.py", - ], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:plot", - # Implicit numpy dependency. - ], -) - -py_test( - name = "calibration_processing_test", - srcs = ["calibration_processing_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":calibration_processing", - ":kitti_metadata", - "//lingvo:compat", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "breakdown_metric", - srcs = [ - "breakdown_metric.py", - ], - srcs_version = "PY3", - deps = [ - ":calibration_processing", - "//lingvo:compat", - "//lingvo/core:hyperparams", - "//lingvo/core:plot", - # Implicit numpy dependency. - ], -) - -py_test( - name = "breakdown_metric_test", - srcs = ["breakdown_metric_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":breakdown_metric", - ":kitti_ap_metric", - ":kitti_metadata", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "builder_lib", - srcs = ["builder_lib.py"], - srcs_version = "PY3", - deps = [ - ":car_layers", - "//lingvo:compat", - "//lingvo/core:builder_layers", - "//lingvo/core:layers", - "//lingvo/core:py_utils", - # Implicit numpy dependency. - ], -) - -py_test( - name = "builder_lib_test", - srcs = ["builder_lib_test.py"], - python_version = "PY3", - shard_count = 2, - srcs_version = "PY3", - deps = [ - ":builder_lib", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "evaluation_metadata", - srcs = [ - "evaluation_metadata.py", - ], - srcs_version = "PY3", - deps = [ - # Implicit numpy dependency. - ], -) - -py_library( - name = "car_lib", - srcs = ["car_lib.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:py_utils", - ], -) - -py_library( - name = "geometry", - srcs = ["geometry.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:py_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "input_extractor", - srcs = ["input_extractor.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:py_utils", - "//lingvo/tasks/car:base_extractor", - ], -) - -py_test( - name = "input_extractor_test", - srcs = ["input_extractor_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":input_extractor", - ":input_preprocessors", - "//lingvo:compat", - "//lingvo/core:hyperparams", - "//lingvo/core:test_utils", - ], -) - -py_library( - name = "base_decoder", - srcs = ["base_decoder.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:py_utils", - "//lingvo/core/ops:record_py_pb2", - ], -) - -py_library( - name = "base_extractor", - srcs = ["base_extractor.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_input_generator", - "//lingvo/core:datasource", - "//lingvo/core:generic_input", - "//lingvo/core:hyperparams", - "//lingvo/core:py_utils", - ], -) - -py_library( - name = "kitti_ap_metric", - srcs = [ - "kitti_ap_metric.py", - ], - srcs_version = "PY3", - deps = [ - ":ap_metric", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/tasks/car/ops", - # Implicit numpy dependency. - ], -) - -py_library( - name = "kitti_decoder", - srcs = ["kitti_decoder.py"], - srcs_version = "PY3", - deps = [ - ":base_decoder", - ":detection_3d_lib", - ":detection_3d_metrics", - ":geometry", - ":kitti_ap_metric", - ":kitti_metadata", - ":transform_util", - "//lingvo:compat", - "//lingvo/core:metrics", - "//lingvo/core:py_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "kitti_metadata", - srcs = [ - "kitti_metadata.py", - ], - srcs_version = "PY3", - deps = [":evaluation_metadata"], -) - -py_library( - name = "lr_util", - srcs = ["lr_util.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:cluster_factory", - "//lingvo/core:py_utils", - "//lingvo/core:schedule", - ], -) - -py_test( - name = "lr_util_test", - size = "small", - srcs = ["lr_util_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":kitti_input_generator", - ":lr_util", - ":pillars", - "//lingvo:compat", - "//lingvo/core:test_utils", - ], -) - -py_library( - name = "pillars", - srcs = ["pillars.py"], - srcs_version = "PY3", - deps = [ - ":builder_lib", - ":detection_3d_lib", - ":geometry", - ":point_detector", - # Implicit enum dependency. - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:layers", - "//lingvo/core:optimizer", - "//lingvo/core:py_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "point_detector", - srcs = ["point_detector.py"], - srcs_version = "PY3", - deps = [ - ":detection_decoder", - ":kitti_decoder", - "//lingvo:compat", - "//lingvo/core:base_model", - "//lingvo/core:py_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "pointnet", - srcs = ["pointnet.py"], - srcs_version = "PY3", - deps = [ - ":builder_lib", - "//lingvo:compat", - "//lingvo/core:builder_layers", - "//lingvo/core:py_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "starnet", - srcs = ["starnet.py"], - srcs_version = "PY3", - deps = [ - ":builder_lib", - ":detection_3d_lib", - ":geometry", - ":point_detector", - # Implicit enum dependency. - "//lingvo:compat", - "//lingvo/core:py_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "summary", - srcs = ["summary.py"], - srcs_version = "PY3", - deps = [ - ":transform_util", - # Implicit PIL dependency. - "//lingvo:compat", - "//lingvo/core:plot", - # Implicit matplotlib dependency. - # Implicit numpy dependency. - ], -) - -py_test( - name = "pointnet_test", - srcs = ["pointnet_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":pointnet", - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - ], -) - -py_test( - name = "summary_test", - srcs = ["summary_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":summary", - "//lingvo:compat", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "transform_util", - srcs = [ - "transform_util.py", - ], - srcs_version = "PY3", - deps = [ - # Implicit numpy dependency. - ], -) - -py_test( - name = "transform_util_test", - size = "small", - srcs = [ - "transform_util_test.py", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":transform_util", - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_test( - name = "car_lib_test", - srcs = ["car_lib_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":car_lib", - "//lingvo:compat", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_test( - name = "geometry_test", - srcs = ["geometry_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":geometry", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "input_preprocessors", - srcs = ["input_preprocessors.py"], - srcs_version = "PY3", - deps = [ - ":car_lib", - ":detection_3d_lib", - ":geometry", - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:py_utils", - "//lingvo/tasks/car/ops", - # Implicit numpy dependency. - ], -) - -py_library( - name = "kitti_input_generator", - srcs = [ - "kitti_input_generator.py", - ], - srcs_version = "PY3", - deps = [ - ":geometry", - ":input_extractor", - ":input_preprocessors", - ":kitti_metadata", - "//lingvo:compat", - "//lingvo/core:datasource", - "//lingvo/core:hyperparams", - "//lingvo/core:py_utils", - "//lingvo/core/ops", - ], -) diff --git a/lingvo/tasks/car/README.md b/lingvo/tasks/car/README.md deleted file mode 100644 index ec83ec1a..00000000 --- a/lingvo/tasks/car/README.md +++ /dev/null @@ -1,158 +0,0 @@ -# Lingvo 3D pointcloud object detection - -This directory contains libraries and models for training 3D classification and -detection datasets using primarily pointcloud inputs. - -The repo contains the code needed to reproduce the results at -https://arxiv.org/abs/1908.11069. - -## Guide to reproduce StarNet on GCP - -The guide below is meant to illustrate how to train a StarNet model on GCP using -CloudTPUs for training and GPUs for evaluation, using GKE. - -### Setup - -First, you will need to setup a GCP account. - -Some useful resources: - -* https://cloud.google.com/kubernetes-engine/docs/tutorials/hello-app -* https://cloud.google.com/tpu/docs/kubernetes-engine-setup -* https://cloud.google.com/kubernetes-engine/docs/troubleshooting - -You may also consider reading the -[TensorFlow Minigo Cloud TPU](https://github.com/tensorflow/minigo/tree/master/cluster) -script repository for managing GCP resources. - -#### Cloud TPU cluster setup - -__IMPORTANT: These commands will result in VMs and other GCP resources being -created and will result in charges to your GCP account! Proceed with care!__ - -To properly create the CloudTPU cluster, one should follow the TPU documentation -above. An example might be: - - gcloud container clusters create tpu_v3_cluster --cluster-version=1.13 --scopes=cloud-platform,gke-default --enable-ip-alias --enable-tpu --zone=europe-west4-a - -where we create a TPU-compatible cluster in a zone that contains V3 TPU pods. We -only need to specify the size of the TPU pod we want when we create the job. - -#### GPU cluster setup - -__IMPORTANT: These commands will result in VMs and other GCP resources being -created and will result in charges to your GCP account! Proceed with care!__ - -First, create the cluster. You may need to get -[GPU quota](https://cloud.google.com/compute/quotas#gpus) in the zone you want -for the GPU type you want. - -For example, here we create a P100 cluster of size 1 using high-memory CPU -instances (which are useful for evaluating / decoding): - - gcloud container clusters create p100-europe-west4-a-nh16 --accelerator type=nvidia-tesla-p100,count=1 --num-nodes=1 --zone europe-west4-a --scopes=cloud-platform,gke-default --enable-ip-alias --cluster-version=1.13 --machine-type=n1-highmem-16 - -To be able to launch GPU jobs on this cluster, one must then install the -drivers. - - kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml - -To check that the GPUs have been fully initialized with these drivers, you can -run: - - kubectl get no -w -o yaml | grep -E 'hostname:|nvidia.com/gpu' - -#### Docker setup - -It is required to use Docker to be able to push Docker images for GKE to load. - -One should set up docker following instructions online. - -We have provided a lingvo-compatible dockerfile that comes with the lingvo pip -package pre-installed. One should build the docker with the GPU nvidia-docker -base for running on GPUs. For example, from the current repo: - - docker build --tag tensorflow:lingvo_lib_gpu --build-arg base_image=nvidia/cuda:10.0-cudnn7-runtime-ubuntu18.04 - < lingvo/docker/lib.dockerfile - -This will be the base docker image you will use to run lingvo jobs on GKE. - -#### Data prep - -To train our existing models without modification, one must first upload -TF.Example versions of the datasets to GCS. - -We have provided scripts to process the raw data in the tools/ and waymo/tools -subdirectories. For example, one can run tools/kitti_exporter.py to create the -TFRecord TFExample files from the raw KITTI data; one can then upload the -results to GCS under a bucket you own. - -## Launching the training job on a 4x4 Cloud TPU V3. - -We have provided a python script that launches / manages GKE jobs at -lingvo/tools/gke_launch.py. Let's say you want to reproduce the -model used for training the StarNet Pedestrian/Cyclist model on KITTI named -StarNetPedCycModel0704 in params/kitti.py. - -To launch training on TPU, decoding for the validation set on GPU, and a -TensorBoard to monitor it all, one can run: - - # Name of the registered model in params/kitti.py to run. - export MODEL=params.kitti.StarNetPedCycModel0704 - - # Environment variable containing path to input dataset. - export DATA_ENV="KITTI_DIR=gs://my-bucket/kitti" - - # For waymo, one can use the version generated via generate_waymo_tf.py - # graciously hosted by Waymo; you must have registered for access to - # the Waymo dataset to be able to access the following bucket. - # export DATA_ENV="WAYMO_DIR=gs://waymo_open_dataset_v_1_0_0_tf_example_lingvo/v.1.2.0/" - - export SPLIT=dev # Use 'minidev' for smaller Waymo validation set. - - # Base docker image name to use for packaging the - # code to run. - export DOCKER_IMAGE=gcr.io/my-project/lingvo-code - - # Location of where the code to be packaged into - # the docker file is located. - export CODE_DIR=/path/to/base/of/lingvo_repo - - # Log directory to write checkpoints to. - export LOGDIR=gs://my-bucket/logs/starnet.kitti.v0 - - # Name (or filter) of the TPU cluster you created above. - export TPU_CLUSTER_NAME=tpu_v3_cluster - - # Specify using a 4x4 Cloud TPU v3. - export TPU_TYPE=v3-32 - - # Name (or filter) of the GPU cluster you created above. - export GPU_CLUSTER_NAME=p100 - - # Specify using a p100 GPU. - export GPU_TYPE=p100 - - # The prefix name of the jobs to launch on GKE. - export EXP_NAME=kitti.starnet.pedcyc.v0 - - python3 lingvo/tools/gke_launch.py \ - --model=$MODEL \ - --base_image=tensorflow:lingvo_lib_gpu \ - --image=$DOCKER_IMAGE \ - --logdir=$LOGDIR \ - --tpu_type=$TPU_TYPE \ - --trainer_cell=$TPU_CLUSTER_NAME \ - --decoder_cell=$GPU_CLUSTER_NAME \ - --decoder_gpus=1 \ - --gpu_type=$GPU_TYPE \ - --decoder=$SPLIT \ - --extra_envs=$DATA_ENV \ - --name=$EXP_NAME \ - --build=$CODE_DIR/lingvo/tasks/car \ - reload all - -As the model trains, the TPU and GPU jobs will output events to the log -directory, which the TensorBoard job will visualize. - -To bring down the jobs, you can run the above command with "down" instead of -"reload". diff --git a/lingvo/tasks/car/__init__.py b/lingvo/tasks/car/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/lingvo/tasks/car/ap_metric.py b/lingvo/tasks/car/ap_metric.py deleted file mode 100644 index 1ef2242d..00000000 --- a/lingvo/tasks/car/ap_metric.py +++ /dev/null @@ -1,556 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Average precision metric interface.""" - -import functools -from lingvo import compat as tf -from lingvo.core import hyperparams -from lingvo.core import py_utils -from lingvo.core.metrics import BaseMetric -from lingvo.tasks.car import breakdown_metric -import numpy as np - - -# TODO(shlens): Consider making this internal data structure a list of NestedMap -# to simplify the implementation. -class Boxes3D: - """A container for a list of 7-DOF 3D boxes. - - Each 3D box is represented by the 7-tuple [x, y, z, dx, dy, dz, phi], - where x, y, z is the center of the box, dx, dy, dz represent the width, - length, and height of the box, and phi is the rotation of the box. - """ - - def __init__(self): - self._capacity = 0 - self._size = 0 - self._buf = py_utils.NestedMap( - imgids=np.empty([0]), - scores=np.empty([0]), - boxes=np.empty([0, 7]), - difficulties=np.empty([0]), - distances=np.empty([0]), - num_points=np.empty([0]), - rotations=np.empty([0]), - heights_in_pixels=np.empty([0]), - speeds=np.empty([0, 2])) - - def Add(self, img_id, score, box, difficulty, distance, num_points, rotation, - height_in_pixels, speed): - """Adds a bbox. - - Args: - img_id: A unique image identifier. - score: The confidence score. - box: [1 x 7] numpy array. - difficulty: The difficulty of a box. On KITTI this is one of [0, 1, 2, 3]. - distance: The binned distance of a box. - num_points: Number of laser points in box. - rotation: The binned rotation of a box. - height_in_pixels: The height of the 2D bbox of this object in the camera - image. - speed: A [1 x 2] numpy array with speed of object in world frame. - """ - if self._size >= self._capacity: - # resize - if self._capacity: - # Increase the capacity exponentially. - self._capacity += self._capacity // 4 - else: - self._capacity = 100 - self._buf = self._buf.Transform(self._Resize) - self._buf.imgids[self._size] = img_id - self._buf.scores[self._size] = score - self._buf.boxes[self._size] = box - self._buf.difficulties[self._size] = difficulty - self._buf.distances[self._size] = distance - self._buf.num_points[self._size] = num_points - self._buf.rotations[self._size] = rotation - self._buf.heights_in_pixels[self._size] = height_in_pixels - self._buf.speeds[self._size] = speed - self._size += 1 - - def _Resize(self, arr): - n = self._capacity - ret = np.empty([n] + list(arr.shape)[1:], dtype=arr.dtype) - ret[:arr.shape[0]] = arr - return ret - - @property - def imgids(self): - return self._buf.imgids[:self._size] - - @property - def scores(self): - return self._buf.scores[:self._size] - - @property - def boxes(self): - return self._buf.boxes[:self._size] - - @property - def difficulties(self): - return self._buf.difficulties[:self._size] - - @property - def distances(self): - return self._buf.distances[:self._size] - - @property - def num_points(self): - return self._buf.num_points[:self._size] - - @property - def rotations(self): - return self._buf.rotations[:self._size] - - @property - def heights_in_pixels(self): - return self._buf.heights_in_pixels[:self._size] - - @property - def speeds(self): - return self._buf.speeds[:self._size] - - -class APMetrics(BaseMetric): - """Measure an assortment of precision-recall metrics on a dataset.""" - - @classmethod - def Params(cls, metadata): - """Params builder for APMetrics.""" - p = hyperparams.InstantiableParams(cls) - p.Define( - 'metadata', metadata, - 'Instance of class obeying EvaluationMetadata interface consisting of ' - 'parameters specifying the details of the evaluation.') - p.Define( - 'breakdown_metrics', [], - 'List of extra breakdown metrics when computing AP. Valid values ' - 'include: ["num_points", "distance", "rotation"]. See ' - 'breakdown_metric.py:ByName for the full list. ByDifficulty is ' - 'always used.') - p.Define( - 'metric_weights', None, - 'For metrics that have multiple breakdown metrics, ' - 'a user may want the value() function to be a function ' - 'of the various breakdowns. If provided, this specifies ' - 'the weights assigned to each class\'s metric contribution. ' - 'metric_weights should be a dictionary mapping every difficulty ' - 'level to a numpy vector of weights whose values corresponds ' - 'to the order of metadata.EvalClassIndices() weighting.') - p.Define( - 'box_type', '3d', 'Specifies what kind of box evaluation will ' - 'be performed (only supported by waymo metric). ' - 'One of ["2d", "3d"]: 3d means to do 3D AP calculation, and 2d ' - 'means to do a top down Birds-Eye-View AP calculation.') - return p - - def __init__(self, params): - """Initialize the metrics.""" - self.params = params.Copy() - self._is_eval_complete = False - self._groundtruth = {} # keyed by class id. - self._prediction = {} # keyed by class id. - self._str_to_imgid = {} - self._iou_thresholds = self.params.metadata.IoUThresholds() - - self.metadata = self.params.metadata - assert self.params.box_type in ['2d', '3d'] - - # We must always include ByDifficulty. - metrics_params = breakdown_metric.ByDifficulty.Params().Set( - metadata=self.metadata) - self._breakdown_metrics = { - 'difficulty': breakdown_metric.ByDifficulty(metrics_params) - } - - for breakdown_metric_name in self.params.breakdown_metrics: - self._breakdown_metrics[breakdown_metric_name] = ( - breakdown_metric.ByName(breakdown_metric_name)(metrics_params)) - - def _GetImageId(self, str_id): - if str_id in self._str_to_imgid: - return self._str_to_imgid[str_id] - else: - imgid = len(self._str_to_imgid) - self._str_to_imgid[str_id] = imgid - return imgid - - def _AddGroundtruth(self, box): - """Record a ground truth box.""" - (str_imgid, classid, score, box, difficulty, distance, num_points, rot, - speed) = box - imgid = self._GetImageId(str_imgid) - assert classid > 0 and classid < self.metadata.NumClasses(), ( - '{} vs. {}'.format(classid, self.metadata.NumClasses())) - - boxes = self._groundtruth.get(classid) - if boxes is None: - boxes = Boxes3D() - self._groundtruth[classid] = boxes - boxes.Add(imgid, score, box, difficulty, distance, num_points, rot, -1, - speed) - # Invalidate the evaluation. - self._is_eval_complete = False - - def _LoadBoundingBoxes(self, - box_type, - class_id, - distance=None, - num_points=None, - rotation=None): - """Load a specified set of bounding boxes. - - If no boxes are found, return None. - - Note that we do *not* specify 'difficulty' as an option due to how the KITTI - evaluates bounding boxes across difficulty levels. - - Args: - box_type: string. Either 'groundtruth' or 'prediction' - class_id: int32 specifying the class - distance: int32 specifying a binned Euclidean distance of the ground truth - bounding box. If None is specified, all distances are selected. - num_points: int32 specifying a binned number of laser points within the - ground truth bounding box. If None is specified, all boxes are selected. - rotation: int32 specifying a binned rotation within the ground truth - bounding box. If None is specified, all boxes are selected. - - Returns: - Boxes3D containing bounding boxes or None if no bounding boxes available. - """ - assert box_type in ['groundtruth', 'prediction'] - if box_type == 'groundtruth': - boxes_by_class = self._groundtruth - else: - boxes_by_class = self._prediction - - if class_id not in boxes_by_class: - return None - boxes = boxes_by_class[class_id] - - if boxes is not None and distance is not None: - # Filter bounding boxes based a binned (integer) distance. - filtered_boxes = None - for i, s, b, d, dist, n, r, h, v in zip(boxes.imgids, boxes.scores, - boxes.boxes, boxes.difficulties, - boxes.distances, boxes.num_points, - boxes.rotations, - boxes.heights_in_pixels, - boxes.speeds): - if dist == distance: - if filtered_boxes is None: - filtered_boxes = Boxes3D() - filtered_boxes.Add(i, s, b, d, dist, n, r, h, v) - boxes = filtered_boxes - - if boxes is not None and num_points is not None: - # Filter bounding boxes based a binned (integer) number of points. - filtered_boxes = None - for i, s, b, d, dist, n, r, h, v in zip(boxes.imgids, boxes.scores, - boxes.boxes, boxes.difficulties, - boxes.distances, boxes.num_points, - boxes.rotations, - boxes.heights_in_pixels, - boxes.speeds): - if n == num_points: - if filtered_boxes is None: - filtered_boxes = Boxes3D() - filtered_boxes.Add(i, s, b, d, dist, n, r, h, v) - boxes = filtered_boxes - - if boxes is not None and rotation is not None: - # Filter bounding boxes based a binned (integer) rotation. - filtered_boxes = None - for i, s, b, d, dist, n, r, h, v in zip(boxes.imgids, boxes.scores, - boxes.boxes, boxes.difficulties, - boxes.distances, boxes.num_points, - boxes.rotations, - boxes.heights_in_pixels, - boxes.speeds): - if r == rotation: - if filtered_boxes is None: - filtered_boxes = Boxes3D() - filtered_boxes.Add(i, s, b, d, dist, n, r, h, v) - boxes = filtered_boxes - - return boxes - - def _GetData(self, - classid, - difficulty=None, - distance=None, - num_points=None, - rotation=None): - """Returns groundtruth and prediction for the classid in a NestedMap. - - Args: - classid: int32 specifying the class - difficulty: String in [easy, moderate, hard]. If None specified, all - difficulty levels are permitted. - distance: int32 specifying a binned Euclidean distance of the ground truth - bounding box. If None is specified, all distances are selected. - num_points: int32 specifying a binned number of laser points within the - ground truth bounding box. If None is specified, all boxes are selected. - rotation: int32 specifying a binned rotation within the ground truth - bounding box. If None is specified, all boxes are selected. - - Returns: - NestedMap containing iou_threshold, groundtruth and predictions for - specified, classid, difficulty level and binned distance. If no bboxes - are found with these parameters, returns None. - """ - raise NotImplementedError('_GetData must be implemented') - - def _BuildMetric(self, feed_data, classid): - raise NotImplementedError('_BuildMetric must be implemented') - - def _ComputeFinalMetrics(self, - classids=None, - difficulty=None, - distance=None, - num_points=None, - rotation=None): - """Compute precision-recall curves as well as average precision. - - Args: - classids: A list of N int32. - difficulty: String in [easy, moderate, hard]. If None specified, all - difficulty levels are permitted. - distance: int32 specifying a binned Euclidean distance of the ground truth - bounding box. If None is specified, all distances are selected. - num_points: int32 specifying a binned number of laser points within the - ground truth bounding box. If None is specified, all boxes are selected. - rotation: int32 specifying a binned rotation within the ground truth - bounding box. If None is specified, all boxes are selected. - - Returns: - dict. Each entry in the dict is a list of C (number of classes) dicts - containing mapping from metric names to individual results. Individual - entries may be the following items. - - scalars: A list of C (number of classes) dicts mapping metric - names to scalar values. - - curves: A list of C dicts mapping metrics names to np.float32 - arrays of shape [NumberOfPrecisionRecallPoints()+1, 2]. In the last - dimension, 0 indexes precision and 1 indexes recall. - - calibrations: A list of C dicts mapping metrics names to np.float32 - arrays of shape [number of predictions, 2]. The first column is the - predicted probabilty and the second column is 0 or 1 indicating that the - prediction matched a ground truth item. - """ - raise NotImplementedError('_ComputeFinalMetric must be implemented') - - def Update(self, str_id, result): - """Update this metric with a newly evaluated image. - - Args: - str_id: A string. Unique identifier of an image. - result: A NestedMap with these fields: - groundtruth_labels - [N]. Groundtruth boxes' labels. - - groundtruth_bboxes - [N, 7]. Groundtruth boxes coordinates. - - groundtruth_difficulties - [N]. Groundtruth boxes difficulties. - - groundtruth_num_points - [N]. Number of laser points in bounding - boxes. - - groundtruth_speed - [N, 2] Speed in (vx, vy) of the ground truth - object. - - detection_scores - [C, M] - For each class (C classes) we have (up to) - M predicted boxes. - - detection_boxes - [C, M, 7] - [i, j, :] are coordinates for i-th - classes j-th predicted box. - - detection_heights_in_pixels - [C, M]. [i, j] is the height of i-th - classes j-th boxes 2D image coordinate (camera view). - - """ - n = result.groundtruth_labels.shape[0] - assert result.groundtruth_bboxes.shape == (n, 7) - - if 'groundtruth_speed' not in result: - result.groundtruth_speed = np.zeros((n, 2), dtype=np.float32) - - groundtruth_result = py_utils.NestedMap( - bboxes=result.groundtruth_bboxes, - num_points=result.groundtruth_num_points, - difficulties=result.groundtruth_difficulties, - labels=result.groundtruth_labels) - for m in self._breakdown_metrics.values(): - m.AccumulateHistogram(groundtruth_result) - m.AccumulateCumulative(groundtruth_result) - - # TODO(vrv): The Boxes3D structure currently expects a fixed - # number of arguments corresponding to every possible - # breakdown metric, even if not requested or present, requiring - # dummy values in the latter case. We should figure - # out how to avoid requiring these dummy values by making - # the Boxes3D object take a dynamic set of attributes. - num_points = [0] * n - rotations = [0] * n - distances = [0] * n - if 'num_points' in self._breakdown_metrics: - num_points = self._breakdown_metrics['num_points'].Discretize( - result.groundtruth_num_points) - if 'rotation' in self._breakdown_metrics: - rotations = self._breakdown_metrics['rotation'].Discretize( - result.groundtruth_bboxes) - if 'distance' in self._breakdown_metrics: - distances = self._breakdown_metrics['distance'].Discretize( - result.groundtruth_bboxes) - - for label, bbox, difficulty, distance, num_points, rotations, speed in zip( - result.groundtruth_labels, result.groundtruth_bboxes, - result.groundtruth_difficulties, distances, num_points, rotations, - result.groundtruth_speed): - self._AddGroundtruth((str_id, label, 1., bbox, difficulty, distance, - num_points, rotations, speed)) - - c = result.detection_scores.shape[0] - assert c == self.metadata.NumClasses(), '%s vs. %s' % ( - c, self.metadata.NumClasses()) - - str_imgid = self._GetImageId(str_id) - - # Iterate first by class. - for class_id in range(1, c): - assert class_id > 0 and class_id < self.metadata.NumClasses(), ( - '{} vs. {}'.format(class_id, self.metadata.NumClasses())) - - # Get or create the box list for the class. This is done for speed -- we - # do it once per class to avoid dictionary lookups to fetch the field for - # every box. - boxes_for_class = self._prediction.get(class_id) - if boxes_for_class is None: - boxes_for_class = Boxes3D() - self._prediction[class_id] = boxes_for_class - - # Get boxes and scores for this class. - bboxes = result.detection_boxes[class_id, :] - scores = result.detection_scores[class_id, :] - heights_in_pixels = result.detection_heights_in_pixels[class_id, :] - # Select bboxes where scores > 0. - non_zero_bboxes = bboxes[scores > 0] - non_zero_scores = scores[scores > 0] - non_zero_heights_in_pixels = heights_in_pixels[scores > 0] - - rotations = [0] * len(non_zero_bboxes) - distances = [0] * len(non_zero_bboxes) - if 'distance' in self._breakdown_metrics: - # Compute all distances for non-zero-bboxes in one shot. - distances = self._breakdown_metrics['distance'].Discretize( - non_zero_bboxes) - if 'rotation' in self._breakdown_metrics: - rotations = self._breakdown_metrics['rotation'].Discretize( - non_zero_bboxes) - - # Add each box to the list. - # - # NOTE: The length of this loop can be large (e.g., for an early - # checkpoint), so any code inside of this for loop should be - # double-checked for efficiency. - dummy_speed = np.zeros((1, 2), dtype=np.float32) - for box_id in range(non_zero_bboxes.shape[0]): - boxes_for_class.Add( - img_id=str_imgid, - score=non_zero_scores[box_id], - box=non_zero_bboxes[box_id], - difficulty=0, - distance=distances[box_id], - num_points=0, - rotation=rotations[box_id], - height_in_pixels=non_zero_heights_in_pixels[box_id], - speed=dummy_speed) - - def _EvaluateIfNecessary(self): - """Evaluate all precision recall metrics.""" - if self._is_eval_complete: - return - compute_metrics_fn = functools.partial( - self._ComputeFinalMetrics, classids=self.metadata.EvalClassIndices()) - for metric_class in self._breakdown_metrics.values(): - metric_class.ComputeMetrics(compute_metrics_fn) - self._is_eval_complete = True - - @property - def value(self): - if self.params.metric_weights is None: - # Choose a backwards compatible default. The default assumes: - # - # if KITTI, metric_weights is all zeros except for car/moderate. - # - # if not KITTI, metric_weights are all equal. - metric_weights = {} - keys = list(self.metadata.DifficultyLevels().keys()) - for difficulty in keys: - if 'moderate' in keys and len(keys) == 3: - # The KITTI case. - if difficulty == 'moderate': - metric_weights[difficulty] = np.array([1., 0., 0.]) - else: - metric_weights[difficulty] = np.array([0., 0., 0.]) - else: - # Every difficulty and every class is weighted equally. - metric_weights[difficulty] = np.array( - [1.] * len(self.metadata.EvalClassIndices())) - else: - metric_weights = self.params.metric_weights - - # Compute weighted average of AP scores across all classes and difficulties. - num_sum = 0.0 - denom_sum = 0.0 - for difficulty, value in self._AveragePrecisionByDifficulty().items(): - metric_weight = metric_weights[difficulty] - denom = np.sum(metric_weight) - if denom == 0.0: - continue - num_sum += np.sum(np.nan_to_num(value) * metric_weight) - denom_sum += denom - - # Ensure at least some weights are specified - assert denom_sum > 0., 'All AP metric weights were 0!' - return num_sum / denom_sum - - def _AveragePrecisionByDifficulty(self): - """Special case to identify mAP versus difficulty.""" - self._EvaluateIfNecessary() - return self._breakdown_metrics['difficulty']._average_precisions # pylint:disable=protected-access - - def Summary(self, name): - self._EvaluateIfNecessary() - - ret = tf.Summary() - - # Put '.value' first (so it shows up in logs / summaries, etc). - ret.value.add(tag='{}/weighted_mAP'.format(name), simple_value=self.value) - - average_precision_by_difficulty = self._AveragePrecisionByDifficulty() - for i, j in enumerate(self.metadata.EvalClassIndices()): - classname = self.metadata.ClassNames()[j] - for difficulty in self.metadata.DifficultyLevels(): - tag_str = '{}/{}/AP_{}'.format(name, classname, difficulty) - ap_value = average_precision_by_difficulty[difficulty][i] - ret.value.add(tag=tag_str, simple_value=ap_value) - - for metric_class in self._breakdown_metrics.values(): - image_summaries = metric_class.GenerateSummaries(name) - for image_summary in image_summaries: - ret.value.extend(image_summary.value) - return ret diff --git a/lingvo/tasks/car/base_decoder.py b/lingvo/tasks/car/base_decoder.py deleted file mode 100644 index ab0ea8f3..00000000 --- a/lingvo/tasks/car/base_decoder.py +++ /dev/null @@ -1,89 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Base models for point-cloud based detection.""" - -from lingvo import compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils -from lingvo.core.ops import record_pb2 - - -class BaseDecoder(base_layer.BaseLayer): - """A decoder to use for decoding a detector model.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'summarize_boxes_on_image', False, - 'If true, enable the summary metric that projects bounding boxes ' - 'to the camera image to view predictions from camera view.') - - p.Define('ap_metric', None, 'Configuration of AP metric for decoding.') - - p.Define( - 'laser_sampling_rate', 0.05, - 'Rate at which real laser outputs are added to decoder output. ' - 'Because the laser outputs are large, we only want to output ' - 'the lasers on a small number of the batches.') - return p - - def _SampleLaserForVisualization(self, points_xyz, points_padding): - """Samples laser points based on configured laser_sampling_rate. - - Args: - points_xyz: [batch, num_points, 3] float Tensor. - points_padding: [batch, num_points] float Tensor. - - Returns: - .NestedMap: - - - points_xyz: 0.0 or points_xyz float Tensor passthrough. - - points_padding: 0.0 or points_padding float Tensor passthrough. - - points_sampled: scalar bool Tensor if points were sampled. - If false, points_xyz and points_padding are scalar 0s to - reduce the amount of data transferred. - """ - p = self.params - rand = tf.random.uniform(shape=(), minval=0, maxval=1, dtype=tf.float32) - sample_laser = rand < p.laser_sampling_rate - points_xyz, points_padding = tf.cond( - sample_laser, lambda: (points_xyz, points_padding), lambda: (0.0, 0.0)) - return py_utils.NestedMap({ - 'points_xyz': points_xyz, - 'points_padding': points_padding, - 'points_sampled': sample_laser - }) - - def SaveTensors(self, tensor_map): - """Returns a serialized representation of the contents of `tensor_map`. - - Args: - tensor_map: A NestedMap of string keys to numpy arrays. - - Returns: - A serialized record_pb2.Record() of the contents of 'tensor_map'. - """ - - def AddKeyVals(**kwargs): - record = record_pb2.Record() - for k, v in kwargs.items(): - record.fields[k].CopyFrom(tf.make_tensor_proto(v)) - return record - - records = AddKeyVals(**tensor_map) - serialized = records.SerializeToString() - return serialized diff --git a/lingvo/tasks/car/base_extractor.py b/lingvo/tasks/car/base_extractor.py deleted file mode 100644 index ac07f73b..00000000 --- a/lingvo/tasks/car/base_extractor.py +++ /dev/null @@ -1,308 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Base extractor interface.""" - -from lingvo import compat as tf -from lingvo.core import base_input_generator -from lingvo.core import datasource -from lingvo.core import generic_input -from lingvo.core import hyperparams -from lingvo.core import py_utils - -# Items exceeding this value will be dropped and not sent to the trainer. -BUCKET_UPPER_BOUND = 9999 - - -def _ParseSequenceExample(record, feature_map, context_map): - """Parse a SequenceExample, adding the context features to the features.""" - context, features = tf.io.parse_single_sequence_example( - serialized=record, - context_features=context_map, - sequence_features=feature_map) - # Add all keys from context to features. Keys must not overlap. - common_keys = set(context.keys()) & set(features.keys()) - if common_keys: - raise ValueError( - 'Keys {} are present in context and features.'.format(common_keys)) - features.update(context) - return features - - -def _TextInput(record, feature_map): - # record is a Tensor containing a string line. - if feature_map: - raise ValueError('For PlainText datasets, FeatureMap() must be empty.') - return {'line': record} - - -# Supported raw record types and the corresponding parsing functions. -_PARSING_FUNCTIONS = { - 'EXAMPLE': tf.io.parse_single_example, - 'SEQUENCE_EXAMPLE': _ParseSequenceExample, - 'TEXT': _TextInput, -} - - -class _BaseExtractor(base_input_generator.BaseInputGeneratorFromFiles): - """The base extractor for all V06+-derived Minecraft datasets. - - Subclasses should define and pass in a custom dictionary of extractors to - select which fields from V06+ datasets to output from an input - generator. - - Preprocessors are applied to all the extracted outputs jointly, in the - specified sequence. - """ - - @classmethod - def Params(cls, extractors): - """Defaults params. - - Args: - extractors: An hyperparams.Params of extractor names to Extractors. A few - extractor types are *required*: - 'labels': A LabelExtractor.Params(). - - Returns: - A base_layer Params object. - """ - p = super().Params() - p.Define('extractors', extractors, - 'A hyperparams.Params() of FieldsExtractors.') - p.Define('preprocessors', hyperparams.Params(), - 'A Params() of Preprocessors.') - p.Define( - 'preprocessors_order', [], - 'A list corresponding to flattened keys in preprocessors ' - 'Params(). This specifies the execution order of the ' - 'preprocessors.') - p.Define('record_type', 'EXAMPLE', - 'Raw record format, default to tf.Example.') - - p.batch_size = 64 - p.use_per_host_infeed = True - p.file_random_seed = 0 - - p.file_datasource = datasource.SimpleDataSource.Params() - - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - # Instantiate every extractor as a child layer. - self._extractors = py_utils.NestedMap() - for (name, eparam) in p.extractors.IterParams(): - name = name.replace('.', '_') - self.CreateChild(name, eparam) - self._extractors[name] = self.children[name] - - # Instantiate preprocessors based on their ordering. - flattened_processors = dict(p.preprocessors.IterParams()) - - # Validate that all keys in preprocessors_order appear are valid. - if not set(p.preprocessors_order).issubset( - list(flattened_processors.keys())): - raise ValueError( - 'preprocessor_order specifies keys which were not found in ' - 'preprocessors. preprocessors_order={} preprocessors keys={}'.format( - p.preprocessors_order, list(flattened_processors.keys()))) - - preprocessors = [flattened_processors[key] for key in p.preprocessors_order] - self.CreateChildren('preprocessors', preprocessors) - - dtypes = self.DType() - shapes = self.Shape() - if not dtypes.IsCompatible(shapes): - raise ValueError('{} vs. {}'.format(dtypes.DebugString(), - shapes.DebugString())) - dtypes.Pack(list(zip(dtypes.Flatten(), - shapes.Flatten()))).VLog(0, 'InpGen: ') - - def FeatureMap(self): - """Get a mapping from feature names to feature tensors.""" - feature_map = {} - self._extractors.Transform(lambda e: feature_map.update(e.FeatureMap())) - return feature_map - - def ContextMap(self): - """Get a mapping from context names to context tensors.""" - context_map = {} - self._extractors.Transform(lambda e: context_map.update(e.ContextMap())) - return context_map - - def Shape(self): - shapes = self._extractors.Transform(lambda x: x.Shape()) - for preprocessor in self.preprocessors: - shapes = preprocessor.TransformShapes(shapes) - return shapes - - def DType(self): - dtypes = self._extractors.Transform(lambda x: x.DType()) - for preprocessor in self.preprocessors: - dtypes = preprocessor.TransformDTypes(dtypes) - return dtypes - - @property - def class_names(self): - raise NotImplementedError('Return a list of class names strings.') - - def _DataSourceFromFilePattern(self, file_pattern): - - def Proc(record): - """Parses a serialized tf.Example record.""" - bucket, outputs = self.ExtractUsingExtractors(record) - return outputs.Flatten(), bucket - - # Ensure buckets above BUCKET_UPPER_BOUND are dropped. - args = self.CommonInputOpArgs() - args['bucket_upper_bound'] = [BUCKET_UPPER_BOUND - 1] - return generic_input.GenericInput( - processor=Proc, file_pattern=file_pattern, **args) - - def ProcessFeatures(self, features): - """Process extracted features. - - Args: - features: A dict of extracted Tensors from the records. - - Returns: - A tuple of tensors: - - - bucket_id: A scalar int Tensor. - - extracted: a NestedMap of Tensors extracted. - """ - def ExtractAndFilter(e): - with tf.name_scope(e.params.name): - with tf.name_scope('extract'): - extracted = e.Extract(features) - with tf.name_scope('filter'): - bucket = e.Filter(extracted) - return bucket, extracted - - bucket_extracted = self._extractors.Transform(ExtractAndFilter) - buckets = bucket_extracted.Transform(lambda x: x[0]) - extracted = bucket_extracted.Transform(lambda x: x[1]) - - # Return the maximum bucket id so that any extractor can decide whether - # to filter the entire example. - max_bucket = tf.reduce_max(buckets.Flatten()) - - def NullLike(): - """A function to return the same Tensor signature as Preprocess. - - This is necessary for the tf.cond() to avoid executing the preprocessor - for examples that are going to be dropped because it exceeds the bucket - limit; tf.cond() requires that the output of both branches yields the same - structure. - - Returns: - A structure with the same Tensor dtype as the output of - Preprocess. - """ - shapes = self.Shape() - rets = [] - for dtype, shape in zip(self.DType().Flatten(), shapes.Flatten()): - if shape.is_fully_defined(): - rets += [tf.zeros(dtype=dtype, shape=shape)] - else: - rets += [tf.zeros(dtype=dtype, shape=[])] # Our best guess. - return shapes.Pack(rets) - - def Preprocess(extracted): - for key, preprocessor in zip(self.params.preprocessors_order, - self.preprocessors): - with tf.name_scope(key), tf.name_scope(preprocessor.params.name): - extracted = preprocessor.TransformFeatures(extracted) - return extracted - - # If the extractor wants to filter the example, don't run the preprocessor. - # - # Preprocessors can then assume that only examples that pass filtering will - # be executed. - # - # Note that the NullLike branch may return tensors with shapes different - # from self.Shape(). - final_output = tf.cond( - tf.less(max_bucket, BUCKET_UPPER_BOUND), lambda: Preprocess(extracted), - NullLike) - - return max_bucket, final_output - - def ExtractUsingExtractors(self, record): - """Extracts Tensors from a tf.Example record using self.extractors. - - Args: - record: A tf.Example input to pass to tf.io.parse_single_example. - - Returns: - A tuple of tensors: - - - bucket_id: A scalar int Tensor. - - extracted: a NestedMap of Tensors extracted. - """ - if self.params.record_type not in _PARSING_FUNCTIONS: - raise ValueError('Invalid record_type: {}'.format( - self.params.record_type)) - - parsing_fn = _PARSING_FUNCTIONS[self.params.record_type] - if self.params.record_type == 'SEQUENCE_EXAMPLE': - features = parsing_fn(record, self.FeatureMap(), self.ContextMap()) - else: - features = parsing_fn(record, self.FeatureMap()) - - return self.ProcessFeatures(features) - - def _InputBatch(self): - batched_outputs, bucket_keys = self._BuildDataSource() - ret = self._NestedMapFromBatchedOutputs(batched_outputs) - ret.bucket_keys = bucket_keys - return ret - - def _NestedMapFromBatchedOutputs(self, outputs): - """Create a NestedMap from a tuple of outputs from generic_input_op.""" - batch_size = self.InfeedBatchSize() - shapes = self.Shape() - shapes.VLog(0, 'input extractor shape: ') - flatten_shapes = shapes.Flatten() - dtypes = self.DType() - flatten_dtypes = dtypes.FlattenItems() - assert len(flatten_shapes) == len(outputs), '{} vs. {}'.format( - len(flatten_shapes), len(outputs)) - assert len(flatten_dtypes) == len(outputs), '{} vs. {}'.format( - len(flatten_dtypes), len(outputs)) - - rets = [] - for (output, (name, dtype), shape) in zip(outputs, flatten_dtypes, - flatten_shapes): - assert dtype == output.dtype, '{}: {} vs. {}'.format( - name, dtype, output.dtype) - # Pad every output to make shapes fixed according to the corresponding - # declared shape, since the shapes of outputs are lost through - # generic_input_op. - try: - shape.assert_is_fully_defined() - except ValueError as e: - raise ValueError('Invalid shape for %s: %s' % (name, e)) - padded = py_utils.PadOrTrimTo(output, [batch_size] + shape.as_list()) - rets += [padded] - - rets = shapes.Pack(rets) - if py_utils.use_tpu(): - # Drops tf.string tensors, which is not supported on TPUs. - rets = rets.Filter(lambda x: x.dtype != tf.string) - return rets diff --git a/lingvo/tasks/car/breakdown_metric.py b/lingvo/tasks/car/breakdown_metric.py deleted file mode 100644 index 60666b29..00000000 --- a/lingvo/tasks/car/breakdown_metric.py +++ /dev/null @@ -1,778 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Library for calculating precision recall conditioned on a variate. - -ByDistance: Calculate precision recall based on distance in world coordinates. -ByRotation: Calculate precision recall based on rotation in world coordinates. -ByNumPoints: Calculate maximum recall based on number of points in bounding box. -ByDifficulty: Calculate precision recall based on difficulty. -""" - -from lingvo import compat as tf -from lingvo.core import hyperparams -from lingvo.core import plot -from lingvo.tasks.car import calibration_processing as calibration -import numpy as np - - -def _FindRecallAtGivenPrecision(precision_recall, precision_level): - """Find the recall at given precision level. - - Args: - precision_recall: np.array of shape [n, m, 2] where n is the number of - classes, m is then number of values in the curve, 2 indexes between - precision [0] and recall [1]. np.float32. - precision_level: float32. Selected precision level between (0, 1). - Typically, this may be 0.95. - - Returns: - recall: np.array of shape [n] consisting of recall for all classes - where the values are 0.0 if a given precision is never achieved. - """ - # The method for computing precision-recall inserts precision = 0.0 - # when a particular recall value has not been achieved. The maximum - # recall value is therefore the highest recall value when the associated - # precision > 0. - assert len(precision_recall.shape) == 3, 'Invalid precision recall curve.' - assert precision_recall.shape[-1] == 2, 'Invalid precision recall curve.' - assert precision_level > 0.0, 'Precision must be greater then 0.' - assert precision_level < 1.0, 'Precision must be less then 1.' - num_classes = precision_recall.shape[0] - - recall = np.zeros(shape=(num_classes), dtype=np.float32) - for i in range(num_classes): - precisions = precision_recall[i, :, 0] - recalls = precision_recall[i, :, 1] - indices_at_precision_level = np.flatnonzero(precisions >= precision_level) - if indices_at_precision_level.size > 0: - recall[i] = np.max(recalls[indices_at_precision_level]) - return recall - - -def _FindMaximumRecall(precision_recall): - """Find the maximum recall in all precision recall curves. - - Args: - precision_recall: np.array of shape [n, m, 2] where n is the number of - classes, m is then number of values in the curve, 2 indexes between - precision [0] and recall [1]. np.float32. - - Returns: - max_recall: np.array of shape [n] consisting of max recall for all classes - where the values are 0.0 if objects are found. - """ - # The method for computing precision-recall inserts precision = 0.0 - # when a particular recall value has not been achieved. The maximum - # recall value is therefore the highest recall value when the associated - # precision > 0. - assert len(precision_recall.shape) == 3, 'Invalid precision recall curve.' - assert precision_recall.shape[-1] == 2, 'Invalid precision recall curve.' - num_classes = precision_recall.shape[0] - - max_recall = np.zeros(shape=(num_classes), dtype=np.float32) - - valid_precisions = precision_recall[:, :, 0] > 0.0 - for i in range(num_classes): - valid_precisions_indices = valid_precisions[i, :] - if np.any(valid_precisions_indices): - max_recall[i] = np.max(precision_recall[i, valid_precisions_indices, 1]) - return max_recall - - -class BreakdownMetric: - """Base class for calculating precision recall conditioned on a variate.""" - - @classmethod - def Params(cls): - p = hyperparams.Params() - p.Define( - 'metadata', None, - 'Class obeying EvaluationMetadata interface consisting of ' - 'parameters specifying the details of the evaluation.') - return p - - def __init__(self, p): - self.params = p - assert p.metadata is not None - self._histogram = np.zeros( - shape=(self.NumBinsOfHistogram(), p.metadata.NumClasses()), - dtype=np.int32) - self._values = np.zeros( - shape=(self.NumBinsOfHistogram(), 1), dtype=np.float32) - self._cumulative_distribution = {} - for l in range(p.metadata.NumClasses()): - self._cumulative_distribution[l] = [] - self._average_precisions = {} - self._precision_recall = {} - self._calibration = {} - self._classnames = p.metadata.ClassNames() - self._classids = p.metadata.EvalClassIndices() - - # We only want to calculate calibration for the lowest difficulty setting - # which will include *all* data. Including all data is due to the way - # _GetData() works in the kitti_ap_metric. - difficulties = p.metadata.DifficultyLevels() - lowest_difficulty_str = min(difficulties, key=lambda k: difficulties[k]) - self._calibration[lowest_difficulty_str] = ( - calibration.CalibrationCalculator(p.metadata)) - - def NumBinsOfHistogram(self): - """Returns int32 of number of bins in histogram.""" - return NotImplementedError() - - def ComputeMetrics(self, compute_metrics_fn): - """Compute precision-recall analysis conditioned on particular metric. - - Args: - compute_metrics_fn: Function that that calculates precision-recall metrics - and accepts named arguments for conditioning. Typically, this would be - APMetrics._ComputeFinalMetrics(). - - Returns: - nothing - """ - del compute_metrics_fn - return NotImplementedError() - - def GenerateSummaries(self, name): - """Generate list of image summaries plotting precision-recall analysis. - - Args: - name: string providing scope - - Returns: - list of image summaries - """ - del name - return NotImplementedError() - - def Discretize(self, values): - """Discretize statistics into integer values. - - Args: - values: 1-D np.array of variate to be discretized. - - Returns: - 1-D np.array of int32 ranging within [0, cls.NumOfBinsOfHistogram()] - """ - del values - return NotImplementedError() - - def _AccumulateHistogram(self, statistics=None, labels=None): - """Accumulate histogram of binned statistic by label. - - Args: - statistics: int32 np.array of shape [K, 1] of binned statistic - labels: int32 np.array of shape [K, 1] of labels - - Returns: - nothing - """ - assert np.issubdtype(statistics.dtype, np.integer) - if not statistics.size: - return - p = self.params - assert np.max(statistics) < self._histogram.shape[0], ( - 'Histogram shape too small %d vs %d' % - (np.max(statistics), self._histogram.shape[0])) - for l in range(p.metadata.NumClasses()): - indices = np.where(labels == l)[0] - for s in statistics[indices]: - self._histogram[s, l] += 1 - - def _AccumulateCumulative(self, statistics=None, labels=None): - """Accumulate cumulative of real-valued statistic by label. - - Args: - statistics: float32 np.array of shape [K, 1] of statistic - labels: int32 np.array of shape [K, 1] of labels - - Returns: - nothing - """ - p = self.params - for l in range(p.metadata.NumClasses()): - indices = np.where(labels == l)[0] - if indices.size: - self._cumulative_distribution[l].extend(statistics[indices].tolist()) - - def AccumulateCumulative(self, result): - """Accumulate cumulative of real-valued statistic by label. - - Args: - result: A NestedMap with these fields: - .labels: [N]. Groundtruth boxes' labels. - .bboxes: [N, 7]. Groundtruth boxes coordinates. - .difficulties: [N]. Groundtruth boxes difficulties. - .num_points: [N]. Number of laser points in bounding boxes. - - Returns: - nothing - """ - pass - - -def ByName(breakdown_metric_name): - """Return a BreakdownMetric class by name.""" - breakdown_mapping = { - 'distance': ByDistance, - 'num_points': ByNumPoints, - 'rotation': ByRotation, - 'difficulty': ByDifficulty - } - if breakdown_metric_name not in breakdown_mapping: - raise ValueError('Invalid breakdown name: %s, valid names are %s' % - (breakdown_metric_name, list(breakdown_mapping.keys()))) - return breakdown_mapping[breakdown_metric_name] - - -class ByDistance(BreakdownMetric): - """Calculate average precision as function of distance.""" - - def NumBinsOfHistogram(self): - p = self.params - return int( - np.rint(p.metadata.MaximumDistance() / p.metadata.DistanceBinWidth())) - - @classmethod - def _CalculateEuclideanDistanceFromOrigin(cls, bboxes): - """Calculate the Euclidean distance from the origin for each bounding box. - - Note that the LabelsExtractor originally returns groundtruth_bboxes of - shape [N, 7] where N is the number of bounding boxes. The last axis is - ordered [x, y, z, w, h, d, phi]. Hence, the Euclidean distance to the origin - is the L2 norm of the first 3 entries. - - Args: - bboxes: [N, 7] np.float of N bounding boxes. See details above. - - Returns: - np.array [N] of Euclidean distances. - """ - positions = bboxes[:, :3] - # Note that we employ axis=1 to compute the norm over [x, y, z] for each - # of N bounding boxes. - return np.linalg.norm(positions, axis=1, keepdims=False) - - def Discretize(self, bboxes): - p = self.params - distances = self._CalculateEuclideanDistanceFromOrigin(bboxes) - distances_binned = np.digitize( - distances, - np.arange(0.0, p.metadata.MaximumDistance(), - p.metadata.DistanceBinWidth())) - # index == 0 corresponds to distances outside less than 0.0. Since this is - # not possible, we discard this possibility and make the output 0 indexed to - # match the behavior of np.histogram(). - assert np.all(distances_binned > 0.0), 'Euclidean distance is negative.' - distances_binned -= 1 - return distances_binned - - def AccumulateHistogram(self, result): - distances = self.Discretize(result.bboxes) - self._AccumulateHistogram(statistics=distances, labels=result.labels) - - def ComputeMetrics(self, compute_metrics_fn): - tf.logging.info('Calculating by distance: start') - p = self.params - for d in range(self.NumBinsOfHistogram()): - value_at_histogram = ( - d * p.metadata.DistanceBinWidth() + - p.metadata.DistanceBinWidth() / 2.0) - self._values[d] = value_at_histogram - - metrics = compute_metrics_fn(distance=d) - curves = metrics['curves'] - self._precision_recall[d] = np.array([c['pr'] for c in curves]) - assert len(self._values) == len(list(self._precision_recall.keys())) - tf.logging.info('Calculating by distance: finished') - - def GenerateSummaries(self, name): - """Generate an image summary for AP versus distance by class.""" - p = self.params - - legend_names = [] - for j in p.metadata.EvalClassIndices(): - legend_names.append(p.metadata.ClassNames()[j]) - - num_distances = self._values.shape[0] - ys = np.zeros( - shape=(num_distances, len(p.metadata.EvalClassIndices())), - dtype=np.float32) - - for dist in self._precision_recall: - ys[dist, :] = _FindMaximumRecall(self._precision_recall[dist]) - - def _Setter(fig, axes): - """Configure the plot for mAP versus distance.""" - axes.grid(b=True) - fontsize = 14 - for i, j in enumerate(p.metadata.EvalClassIndices()): - for d, x in enumerate(self._values): - h = self._histogram[d][j] - y = min(ys[d][i] + 0.03, 1.0) - if h > 0: - text_label = '{} {}s'.format(h, legend_names[i].lower()[:3]) - axes.text(x, y, text_label, fontdict={'fontsize': fontsize - 2}) - - axes.set_xlabel('distance (world coordinates)', fontsize=fontsize) - axes.set_xticks( - np.arange( - 0.0, - p.metadata.MaximumDistance() + p.metadata.DistanceBinWidth(), - p.metadata.DistanceBinWidth())) - axes.set_ylabel('maximum recall', fontsize=fontsize) - axes.set_ylim([-0.01, 1.05]) - axes.set_yticks(np.arange(0.0, 1.05, 0.1)) - axes.legend([name.lower() for name in legend_names], numpoints=1, loc=3) - fig.tight_layout() - - tag_str = '{}/recall_by_distance'.format(name) - image_summary = plot.Curve( - name=tag_str, - figsize=(10, 8), - xs=self._values, - ys=ys, - setter=_Setter, - marker='.', - markersize=14, - linestyle='-', - linewidth=2, - alpha=0.5) - return [image_summary] - - -class ByNumPoints(BreakdownMetric): - """Calculate average precision as function of the number of points.""" - - def NumBinsOfHistogram(self): - return self.params.metadata.NumberOfPointsBins() - - def _LogSpacedBinEdgesofPoints(self): - p = self.params - return np.logspace( - np.log10(1.0), np.log10(p.metadata.MaximumNumberOfPoints()), - p.metadata.NumberOfPointsBins() + 1) - - def Discretize(self, num_points): - num_points_binned = np.digitize(num_points, - self._LogSpacedBinEdgesofPoints()) - # index == 0 corresponds to boxes with 0 points. Because we plot everything - # logarithmically, this is a pain in the buttocks. For simplicity, we merely - # accumulate the boxes with 0 points into the first bin. - num_points_binned[num_points_binned == 0] = 1 - num_bins = len(self._LogSpacedBinEdgesofPoints()) - # index == len(self._LogSpacedBinEdgesofPoints()) corresponds to - # points with to points outside of the range of the last edge. We map - # these points back to the final bucket for simplicity. - num_points_binned[num_points_binned == num_bins] -= 1 - # There is an inconsistency between how np.digitize() and np.histogram() - # index their bins and this is due to the fact that index == 0 is reserved - # for examples less than the minimum bin edge. - num_points_binned -= 1 - return num_points_binned - - def AccumulateHistogram(self, result): - num_points = self.Discretize(result.num_points) - self._AccumulateHistogram(statistics=num_points, labels=result.labels) - - def AccumulateCumulative(self, result): - self._AccumulateCumulative( - statistics=result.num_points, labels=result.labels) - - def ComputeMetrics(self, compute_metrics_fn): - tf.logging.info('Calculating by number of points: start') - # Note that we skip the last edge as the number of edges is one greater - # then the number of bins. - self._values = self._LogSpacedBinEdgesofPoints()[:-1] - for n, _ in enumerate(self._values): - metrics = compute_metrics_fn(num_points=n) - curves = metrics['curves'] - self._precision_recall[n] = np.array([c['pr'] for c in curves]) - assert len(self._values) == len(list(self._precision_recall.keys())) - tf.logging.info('Calculating by number of points: finished') - - def GenerateSummaries(self, name): - """Generate an image summary for max recall by number of points by class.""" - image_summaries = self._GenerateCumulativeSummaries(name) - p = self.params - - legend_names = [] - for j in p.metadata.EvalClassIndices(): - legend_names.append(p.metadata.ClassNames()[j]) - - num_points_bins = self._values.shape[0] - ys = np.zeros( - shape=(num_points_bins, len(p.metadata.EvalClassIndices())), - dtype=np.float32) - - for num_points in self._precision_recall: - ys[num_points, :] = _FindMaximumRecall(self._precision_recall[num_points]) - - def _Setter(fig, axes): - """Configure the plot for max recall versus number of points.""" - axes.grid(b=True) - fontsize = 14 - for i, j in enumerate(p.metadata.EvalClassIndices()): - for n, x in enumerate(self._values): - h = self._histogram[n][j] - y = min(ys[n][i] + 0.03, 1.0) - if h > 0: - text_label = '{} {}s'.format(h, legend_names[i].lower()[:3]) - axes.text(x, y, text_label, fontdict={'fontsize': fontsize - 2}) - - axes.set_xlabel('number of points', fontsize=fontsize) - axes.set_xticks(self._values) - axes.set_ylabel('maximum recall', fontsize=fontsize) - axes.set_ylim([-0.01, 1.05]) - axes.set_xlim([(1.0 / 1.3) * self._values[0], 1.3 * self._values[-1]]) - axes.set_yticks(np.arange(0.0, 1.05, 0.1)) - axes.set_xscale('log') - axes.legend([name.lower() for name in legend_names], - numpoints=1, - loc='upper left') - fig.tight_layout() - - tag_str = '{}/recall_by_num_points'.format(name) - image_summary = plot.Curve( - name=tag_str, - figsize=(10, 8), - xs=self._values, - ys=ys, - setter=_Setter, - marker='.', - markersize=14, - linestyle='-', - linewidth=2, - alpha=0.5) - image_summaries.append(image_summary) - return image_summaries - - def _GenerateCumulativeSummaries(self, name): - """Generate an image summary for CDF of a variate.""" - xs = [] - ys = [] - num_zeros = [] - legend_names = [] - min_value = 5.0 - p = self.params - - for i, j in enumerate(p.metadata.EvalClassIndices()): - legend_names.append(p.metadata.ClassNames()[j]) - if len(self._cumulative_distribution[j]) > min_value: - self._cumulative_distribution[j].sort() - x = np.array(self._cumulative_distribution[j]) - nonzeros = np.flatnonzero(x) - cdf = np.arange(x.size).astype(np.float) / x.size - xs.append(x) - ys.append(cdf) - num_zeros.append(x.size - nonzeros.size) - else: - xs.append(None) - ys.append(None) - num_zeros.append(None) - - image_summaries = [] - for i, j in enumerate(p.metadata.EvalClassIndices()): - classname = p.metadata.ClassNames()[j] - - def _Setter(fig, axes): - """Configure the plot for CDF of the variate.""" - axes.grid(b=False) - fontsize = 14 - - axes.set_ylim([0, 1.05]) - axes.set_xlim([1.0, 11500]) - axes.set_ylabel('cumulative distribution', fontsize=fontsize) - axes.set_xlabel('number of points', fontsize=fontsize) - axes.set_xscale('log') - legend_text = '{} {}s ({} contain zero points)'.format( - xs[i].size, - p.metadata.ClassNames()[j].lower(), num_zeros[i]) - axes.legend({legend_text}, loc='upper left') - fig.tight_layout() - - if xs[i] is not None: - tag_str = '{}/{}/cdf_of_num_points'.format(name, classname) - image_summary = plot.Curve( - name=tag_str, - figsize=(10, 8), - xs=xs[i], - ys=ys[i], - setter=_Setter, - marker='', - linestyle='-', - linewidth=2, - alpha=0.5) - image_summaries.append(image_summary) - return image_summaries - - -class ByRotation(BreakdownMetric): - """Calculate average precision as function of rotation.""" - - def NumBinsOfHistogram(self): - return self.params.metadata.NumberOfRotationBins() - - def _CalculateRotation(self, bboxes): - """Calculate rotation angle mod between (0, 2 * pi) for each box. - - Args: - bboxes: [N, 7] np.float of N bounding boxes. See details above. - - Returns: - np.array [N] of rotation angles in radians. - """ - if not bboxes.size: - return np.empty_like(bboxes) - p = self.params - # Although groundtruth is constrained to be in [-pi, pi], predictions are - # unbounded. We map all predictions to their equivalent value in [-pi, pi]. - rotations = np.copy(bboxes[:, -1]) - rotations += np.pi - rotations = np.mod(rotations, 2.0 * np.pi) - rotations -= np.pi - # Now we remove ambiguity in 180 degree rotations as measured by our IOU - # calculations by mapping everything to [0, pi] range. - rotations = np.where(rotations > 0.0, rotations, rotations + np.pi) - # Floating numerical issues can surface occasionally particularly within - # subsequent binning. The clipping makes these operations reliable. - epsilon = 1e-5 - rotations = np.clip(rotations, epsilon, - p.metadata.MaximumRotation() - epsilon) - return rotations - - def Discretize(self, bboxes): - rotations = self._CalculateRotation(bboxes) - p = self.params - bin_width = ( - p.metadata.MaximumRotation() / float(self.NumBinsOfHistogram())) - # TODO(shlens): Consider merging the entries with -1 and 0 bin index - # because rotation is circular. - rotations_binned = np.digitize( - rotations, np.arange(0.0, p.metadata.MaximumRotation(), bin_width)) - # index == 0 corresponds to distances outside less than 0.0. Since this is - # not possible, we discard this possibility and make the output 0 indexed to - # match the behavior of np.histogram(). - assert np.all(rotations_binned > 0.0), ('Rotation is negative: %s' % - rotations_binned) - rotations_binned -= 1 - return rotations_binned - - def AccumulateHistogram(self, result): - rotations = self.Discretize(result.bboxes) - self._AccumulateHistogram(statistics=rotations, labels=result.labels) - - def ComputeMetrics(self, compute_metrics_fn): - tf.logging.info('Calculating by rotation: start') - p = self.params - self._values = np.zeros( - shape=(self.NumBinsOfHistogram(), 1), dtype=np.float32) - bin_width = ( - p.metadata.MaximumRotation() / float(self.NumBinsOfHistogram())) - for r in range(self.NumBinsOfHistogram()): - # Calculate the center of the histogram bin. - value_at_histogram = r * bin_width + bin_width / 2.0 - self._values[r] = value_at_histogram - metrics = compute_metrics_fn(rotation=r) - curves = metrics['curves'] - self._precision_recall[r] = np.array([c['pr'] for c in curves]) - assert len(self._values) == len(list(self._precision_recall.keys())) - tf.logging.info('Calculating by rotation: finished') - - def GenerateSummaries(self, name): - """Generate an image summary for AP versus rotation by class.""" - p = self.params - legend_names = [] - for j in p.metadata.EvalClassIndices(): - legend_names.append(p.metadata.ClassNames()[j]) - - num_rotations = self._values.shape[0] - rotation_in_degrees = self._values * 180.0 / np.pi - ys = np.zeros( - shape=(num_rotations, len(p.metadata.EvalClassIndices())), - dtype=np.float32) - - for rotation in self._precision_recall: - ys[rotation, :] = _FindMaximumRecall(self._precision_recall[rotation]) - - def _Setter(fig, axes): - """Configure the plot for max recall versus distance.""" - axes.grid(b=True) - fontsize = 14 - for i, j in enumerate(p.metadata.EvalClassIndices()): - for r, x in enumerate(rotation_in_degrees): - h = self._histogram[r][j] - y = min(ys[r][i] + 0.03, 1.0) - text_label = '{} {}s'.format(h, legend_names[i].lower()[:3]) - axes.text(x, y, text_label, fontdict={'fontsize': fontsize - 2}) - - axes.set_xlabel('rotation (degrees)', fontsize=fontsize) - bin_width = ( - p.metadata.MaximumRotation() / float(self.NumBinsOfHistogram())) - axes.set_xticks( - np.arange(0.0, - p.metadata.MaximumRotation() + bin_width, bin_width) * - 180.0 / np.pi) - axes.set_ylabel('maximum recall', fontsize=fontsize) - axes.set_ylim([-0.01, 1.05]) - axes.set_yticks(np.arange(0.0, 1.05, 0.1)) - axes.set_xlim([0.0, 180.0]) - axes.legend([name.lower() for name in legend_names], - numpoints=1, - loc='upper right') - fig.tight_layout() - - tag_str = '{}/recall_by_rotation'.format(name) - image_summary = plot.Curve( - name=tag_str, - figsize=(10, 8), - xs=rotation_in_degrees, - ys=ys, - setter=_Setter, - marker='.', - markersize=14, - linestyle='-', - linewidth=2, - alpha=0.5) - return [image_summary] - - -class ByDifficulty(BreakdownMetric): - """Calculate average precision as function of difficulty.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('ap_key', 'ap', 'Metrics key for the AP value.') - p.Define('pr_key', 'pr', 'Metrics key for the PR value.') - return p - - def NumBinsOfHistogram(self): - return len(self.params.metadata.DifficultyLevels()) + 1 - - def Discretize(self, difficulties): - return difficulties.astype(np.int32) - - def AccumulateHistogram(self, result): - difficulties = self.Discretize(result.difficulties) - self._AccumulateHistogram(statistics=difficulties, labels=result.labels) - - def ComputeMetrics(self, compute_metrics_fn): - p = self.params - tf.logging.info('Calculating by difficulty: start') - for difficulty in self.params.metadata.DifficultyLevels(): - metrics = compute_metrics_fn(difficulty=difficulty) - scalars = metrics['scalars'] - curves = metrics['curves'] - self._average_precisions[difficulty] = [s[p.ap_key] for s in scalars] - self._precision_recall[difficulty] = np.array( - [c[p.pr_key] for c in curves]) - - # Only KITTI metrics contains calibration data, - # And we only compute calibration for the lowest difficulty level - # to include all data. - if difficulty in self._calibration: - self._calibration[difficulty].Calculate(metrics) - - tf.logging.info('Calculating by difficulty: finished') - - def GenerateSummaries(self, name): - """Generate an image summary for PR by difficulty and for calibration. - - Args: - name: str, name of summary. - - Returns: - list of summaries - """ - - legend = {} - p = self.params - for class_id in p.metadata.EvalClassIndices(): - legend[class_id] = [] - for difficulty, i in p.metadata.DifficultyLevels().items(): - num_objects = self._histogram[i][class_id] - legend[class_id].append('%s (%d)' % (difficulty, num_objects)) - - summaries = [] - for i, j in enumerate(p.metadata.EvalClassIndices()): - - def _PRSetter(fig, axes): - """Configure the plot for precision recall.""" - ticks = np.arange(0, 1.05, 0.1) - axes.grid(b=False) - axes.set_xlabel('Recall') - axes.set_xticks(ticks) - axes.set_ylabel('Precision') - axes.set_yticks(ticks) - # pylint: disable=undefined-loop-variable - axes.legend(legend[j], numpoints=1) # pylint: disable=cell-var-from-loop - # pylint: enable=undefined-loop-variable - fig.tight_layout() - - classname = p.metadata.ClassNames()[j] - # Generate Precision-Recall curves. - rs = [] - ps = [] - for difficulty in p.metadata.DifficultyLevels(): - ps += [self._precision_recall[difficulty][i][:, 0]] - rs += [self._precision_recall[difficulty][i][:, 1]] - tag_str = '{}/{}/PR'.format(name, classname) - image_summary = plot.Curve( - name=tag_str, - figsize=(10, 8), - xs=rs[0], - ys=np.array(ps).T, - setter=_PRSetter, - marker='.', - markersize=14, - linestyle='-', - linewidth=2, - alpha=0.5) - summaries.append(image_summary) - - for difficulty, c in self._calibration.items(): - # Note that we only generate a calibration for a single difficulty level. - calibration_summaries = c.Summary(name) - for calibration_summary in calibration_summaries: - summaries.append(calibration_summary) - - # Generate scalar summaries for the various recalls for each difficulty. - for difficulty in p.metadata.DifficultyLevels(): - max_recall = _FindMaximumRecall(self._precision_recall[difficulty]) - for i, j in enumerate(p.metadata.EvalClassIndices()): - classname = p.metadata.ClassNames()[j] - summary = tf.Summary(value=[ - tf.Summary.Value( - tag='{}/{}/max_recall_{}'.format(name, classname, difficulty), - simple_value=max_recall[i]) - ]) - summaries.append(summary) - - for precision_level in p.metadata.RecallAtPrecision(): - recall_at_precision = _FindRecallAtGivenPrecision( - self._precision_recall[difficulty], precision_level) - for i, j in enumerate(p.metadata.EvalClassIndices()): - classname = p.metadata.ClassNames()[j] - summary = tf.Summary(value=[ - tf.Summary.Value( - tag='{}/{}/recall_{}_{}'.format( - name, classname, int(precision_level * 100), difficulty), - simple_value=recall_at_precision[i]) - ]) - summaries.append(summary) - return summaries diff --git a/lingvo/tasks/car/breakdown_metric_test.py b/lingvo/tasks/car/breakdown_metric_test.py deleted file mode 100644 index 2f95815c..00000000 --- a/lingvo/tasks/car/breakdown_metric_test.py +++ /dev/null @@ -1,418 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for breakdown_metric.""" - -from lingvo import compat as tf -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.tasks.car import breakdown_metric -from lingvo.tasks.car import kitti_ap_metric -from lingvo.tasks.car import kitti_metadata -import numpy as np - -FLAGS = tf.flags.FLAGS - - -class BreakdownMetricTest(test_utils.TestCase): - - def _GenerateRandomBBoxes(self, num_bboxes): - xyz = np.random.uniform(low=-1.0, high=1.0, size=(num_bboxes, 3)) - dimension = np.random.uniform(low=-1, high=1.0, size=(num_bboxes, 3)) - rotation = np.random.uniform(low=-np.pi, high=np.pi, size=(num_bboxes, 1)) - bboxes = np.concatenate([xyz, dimension, rotation], axis=-1) - return bboxes - - def _GenerateBBoxesAtDistanceAndRotation(self, num_boxes, distance, rotation): - bboxes = np.zeros(shape=(num_boxes, 7)) - bboxes[:, -1] = rotation - bboxes[:, 0] = distance - return bboxes - - def _GenerateMetricsWithTestData(self, num_classes): - metadata = kitti_metadata.KITTIMetadata() - num_bins_of_distance = int( - np.rint(metadata.MaximumDistance() / metadata.DistanceBinWidth())) - num_bins_of_rotation = metadata.NumberOfRotationBins() - num_bins_of_points = metadata.NumberOfPointsBins() - - # Generate ground truth bounding boxes with prescribed labels, distances, - # rotations and number of points. - expected_objects_at_distance = np.random.randint( - low=0, high=8, size=(num_classes, num_bins_of_distance), dtype=np.int32) - expected_objects_at_rotation = np.zeros( - shape=(num_classes, num_bins_of_rotation), dtype=np.int32) - # Note that we need preserve the same number of objects for each label. - expected_objects_at_points = np.zeros( - shape=(num_classes, num_bins_of_points), dtype=np.int32) - prob = 1.0 / float(num_bins_of_points) - for c in range(num_classes): - num_objects_for_class = np.sum(expected_objects_at_distance[c, :]) - expected_objects_at_points[c, :] = np.random.multinomial( - num_objects_for_class, pvals=num_bins_of_points * [prob]) - # Zero out the number of boxes in the background class. - expected_objects_at_distance[0, :] = 0 - expected_objects_at_points[0, :] = 0 - expected_objects_at_rotation[0, :] = 0 - - bboxes = [] - labels = [] - num_points = [] - bin_width = ( - metadata.MaximumRotation() / float(metadata.NumberOfRotationBins())) - # Note that we always skip 'Background' class 0. - for label in range(1, num_classes): - for distance_index in range(num_bins_of_distance): - distance = ( - distance_index * metadata.DistanceBinWidth() + - metadata.DistanceBinWidth() / 2.0) - num_box = expected_objects_at_distance[label, distance_index] - - if num_box > 0: - rotation_index = np.random.randint(num_bins_of_rotation) - expected_objects_at_rotation[label, rotation_index] += num_box - rotation = rotation_index * bin_width + bin_width / 2.0 - - bboxes.append( - self._GenerateBBoxesAtDistanceAndRotation(num_box, distance, - rotation)) - labels.append(label * np.ones(shape=[num_box], dtype=np.int32)) - - point_bin_edges = np.logspace( - np.log10(1.0), np.log10(metadata.MaximumNumberOfPoints()), - metadata.NumberOfPointsBins() + 1) - for point_index in range(num_bins_of_points): - num_box = expected_objects_at_points[label, point_index] - for _ in range(num_box): - points = (point_bin_edges[point_index] + - point_bin_edges[point_index + 1]) / 2.0 - num_points.append([points]) - - bboxes = np.concatenate(bboxes) - labels = np.concatenate(labels) - num_points = np.concatenate(num_points) - - # Generate dummy predictions as placeholders for the API. - num_predictions = 9 - prediction_scores = np.random.uniform(size=[num_classes, num_predictions]) - prediction_bboxes = self._GenerateRandomBBoxes( - num_predictions * num_classes).reshape( - (num_classes, num_predictions, 7)) - - # Update the metrics. - metric_names = ['rotation', 'num_points', 'distance'] - ap_params = kitti_ap_metric.KITTIAPMetrics.Params(metadata).Set( - breakdown_metrics=metric_names) - metrics = ap_params.Instantiate() - metrics.Update( - 'dummy_image1', - py_utils.NestedMap( - groundtruth_labels=labels, - groundtruth_bboxes=bboxes, - groundtruth_difficulties=np.ones(shape=(bboxes.shape[0])), - groundtruth_num_points=num_points, - detection_scores=prediction_scores, - detection_boxes=prediction_bboxes, - detection_heights_in_pixels=np.ones( - shape=prediction_bboxes.shape[0:2]) * 100)) - - return py_utils.NestedMap( - metrics=metrics, - expected_objects_at_distance=expected_objects_at_distance, - expected_objects_at_points=expected_objects_at_points, - expected_objects_at_rotation=expected_objects_at_rotation) - - def testLoadBoundingBoxes(self): - # Test if all of the groundtruth data loads correctly for each label - # when no distance is specified. - metadata = kitti_metadata.KITTIMetadata() - num_classes = len(metadata.ClassNames()) - test_data = self._GenerateMetricsWithTestData(num_classes) - - expected_num_objects = np.sum( - test_data.expected_objects_at_distance, axis=1) - - # Note that we always skip 'Background' class 0. - for label in range(1, num_classes): - data = test_data.metrics._LoadBoundingBoxes( - 'groundtruth', label, distance=None) - - if expected_num_objects[label] == 0: - self.assertIsNone(data) - else: - self.assertEqual(expected_num_objects[label], len(data.boxes)) - self.assertEqual(expected_num_objects[label], len(data.imgids)) - self.assertEqual(expected_num_objects[label], len(data.scores)) - self.assertEqual(expected_num_objects[label], len(data.difficulties)) - - self.assertAllEqual( - np.ones(shape=[expected_num_objects[label]]), data.scores) - self.assertAllEqual( - np.zeros(shape=[expected_num_objects[label]]), data.imgids) - - def testLoadBoundingBoxesDifficulty(self): - metadata = kitti_metadata.KITTIMetadata() - num_classes = len(metadata.ClassNames()) - test_data = self._GenerateMetricsWithTestData(num_classes) - - expected_num_objects = np.sum( - test_data.expected_objects_at_distance, axis=1) - - difficulty_metric = test_data.metrics._breakdown_metrics['difficulty'] - - # Test if difficulties are properly accumulated. - for d in metadata.DifficultyLevels().values(): - if d == 1: - self.assertAllEqual(expected_num_objects, - difficulty_metric._histogram[d, :]) - else: - self.assertAllEqual( - np.zeros_like(expected_num_objects), - difficulty_metric._histogram[d, :]) - - def testLoadBoundingBoxesDistance(self): - # Test if all of the groundtruth data loads correctly for each label - # when distance is specified. - metadata = kitti_metadata.KITTIMetadata() - num_classes = len(metadata.ClassNames()) - test_data = self._GenerateMetricsWithTestData(num_classes) - num_bins_of_distance = int( - np.rint(metadata.MaximumDistance() / metadata.DistanceBinWidth())) - - distance_metric = test_data.metrics._breakdown_metrics['distance'] - - # Test if all of the groundtruth data loads correctly for each label - # when no distance is specified. - self.assertAllEqual(test_data.expected_objects_at_distance, - np.transpose(distance_metric._histogram)) - - # Note that we always skip 'Background' class 0. - for label in range(1, num_classes): - for distance in range(num_bins_of_distance): - data = test_data.metrics._LoadBoundingBoxes( - 'groundtruth', label, distance=distance) - - if test_data.expected_objects_at_distance[label, distance] == 0: - self.assertIsNone(data) - else: - self.assertEqual( - test_data.expected_objects_at_distance[label, distance], - len(data.boxes)) - self.assertEqual( - test_data.expected_objects_at_distance[label, distance], - len(data.imgids)) - self.assertEqual( - test_data.expected_objects_at_distance[label, distance], - len(data.scores)) - self.assertEqual( - test_data.expected_objects_at_distance[label, distance], - len(data.difficulties)) - - self.assertAllEqual( - np.ones(shape=[ - test_data.expected_objects_at_distance[label, distance] - ]), data.scores) - self.assertAllEqual( - np.zeros(shape=[ - test_data.expected_objects_at_distance[label, distance] - ]), data.imgids) - - def testLoadBoundingBoxesNumPoints(self): - # Test if all of the groundtruth data loads correctly for each label - # when number of points is specified. - metadata = kitti_metadata.KITTIMetadata() - num_classes = len(metadata.ClassNames()) - test_data = self._GenerateMetricsWithTestData(num_classes) - num_bins_of_points = metadata.NumberOfPointsBins() - - num_points_metric = test_data.metrics._breakdown_metrics['num_points'] - - self.assertAllEqual(test_data.expected_objects_at_points, - np.transpose(num_points_metric._histogram)) - - # Note that we always skip 'Background' class 0. - for label in range(1, num_classes): - for num_points in range(num_bins_of_points): - data = test_data.metrics._LoadBoundingBoxes( - 'groundtruth', label, num_points=num_points) - - if test_data.expected_objects_at_points[label, num_points] == 0: - self.assertIsNone(data) - else: - # Skip the first bin because it is a special case. - if num_points == 0: - continue - self.assertEqual( - test_data.expected_objects_at_points[label, num_points], - len(data.boxes)) - self.assertEqual( - test_data.expected_objects_at_points[label, num_points], - len(data.imgids)) - self.assertEqual( - test_data.expected_objects_at_points[label, num_points], - len(data.scores)) - self.assertEqual( - test_data.expected_objects_at_points[label, num_points], - len(data.difficulties)) - - self.assertAllEqual( - np.ones(shape=[ - test_data.expected_objects_at_points[label, num_points] - ]), data.scores) - self.assertAllEqual( - np.zeros(shape=[ - test_data.expected_objects_at_points[label, num_points] - ]), data.imgids) - - def testLoadBoundingBoxesRotation(self): - # Test if all of the groundtruth data loads correctly for each label - # when rotation is specified. - metadata = kitti_metadata.KITTIMetadata() - num_classes = len(metadata.ClassNames()) - test_data = self._GenerateMetricsWithTestData(num_classes) - num_bins_of_rotation = metadata.NumberOfRotationBins() - - rotation_metric = test_data.metrics._breakdown_metrics['rotation'] - - # Test if all of the groundtruth data loads correctly for each label - # when no distance is specified. - self.assertAllEqual(test_data.expected_objects_at_rotation, - np.transpose(rotation_metric._histogram)) - - # Note that we always skip 'Background' class 0. - for label in range(1, num_classes): - for rotation in range(num_bins_of_rotation): - data = test_data.metrics._LoadBoundingBoxes( - 'groundtruth', label, rotation=rotation) - - if test_data.expected_objects_at_rotation[label, rotation] == 0: - self.assertIsNone(data) - else: - self.assertEqual( - test_data.expected_objects_at_rotation[label, rotation], - len(data.boxes)) - self.assertEqual( - test_data.expected_objects_at_rotation[label, rotation], - len(data.imgids)) - self.assertEqual( - test_data.expected_objects_at_rotation[label, rotation], - len(data.scores)) - self.assertEqual( - test_data.expected_objects_at_rotation[label, rotation], - len(data.difficulties)) - - self.assertAllEqual( - np.ones(shape=[ - test_data.expected_objects_at_rotation[label, rotation] - ]), data.scores) - self.assertAllEqual( - np.zeros(shape=[ - test_data.expected_objects_at_rotation[label, rotation] - ]), data.imgids) - - def testAccumulateHistogram(self): - metadata = kitti_metadata.KITTIMetadata() - num_per_class = np.arange(metadata.NumClasses()) + 1 - statistics = [ - 1 * np.ones(shape=(np.sum(num_per_class)), dtype=np.int32), - 2 * np.ones(shape=(np.sum(2 * num_per_class)), dtype=np.int32) - ] - statistics = np.concatenate(statistics) - - labels = [] - for i, n in enumerate(num_per_class): - labels.extend([i] * n) - for i, n in enumerate(num_per_class): - labels.extend([i] * 2 * n) - labels = np.array(labels) - assert len(statistics) == len(labels) - - metrics_params = breakdown_metric.BreakdownMetric.Params().Set( - metadata=metadata) - test_breakdown_metric = breakdown_metric.ByDifficulty(metrics_params) - test_breakdown_metric._AccumulateHistogram( - statistics=statistics, labels=labels) - - for class_index, n in enumerate(num_per_class): - self.assertEqual(n, test_breakdown_metric._histogram[1, class_index]) - self.assertEqual(2 * n, test_breakdown_metric._histogram[2, class_index]) - - def testByName(self): - metric_class = breakdown_metric.ByName('difficulty') - self.assertEqual(metric_class, breakdown_metric.ByDifficulty) - with self.assertRaises(ValueError): - breakdown_metric.ByName('undefined') - - def testFindMaximumRecall(self): - # The shape of the precision_recall_curves is [n, m, 2] where n is the - # number of classes, m is then number of values in the curve, 2 indexes - # between precision [0] and recall [1]. - car = np.transpose( - np.array( - [[0.9, 0.7, 0.5, 0.1, 0.0, 0.0], [0.0, 0.2, 0.5, 0.9, 1.0, 1.0]], - dtype=np.float32)) - ped = np.transpose( - np.array( - [[0.9, 0.7, 0.5, 0.0, 0.0, 0.0], [0.0, 0.2, 0.5, 0.9, 1.0, 1.0]], - dtype=np.float32)) - cyc = np.transpose( - np.array( - [[0.9, 0.7, 0.0, 0.0, 0.0, 0.0], [0.0, 0.2, 0.5, 0.9, 1.0, 1.0]], - dtype=np.float32)) - foo = np.transpose( - np.array( - [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.2, 0.5, 0.9, 1.0, 1.0]], - dtype=np.float32)) - precision_recall_curves = np.stack([car, ped, cyc, foo]) - max_recall = breakdown_metric._FindMaximumRecall(precision_recall_curves) - self.assertAllEqual([4], max_recall.shape) - self.assertNear(0.9, max_recall[0], 1e-7) - self.assertNear(0.5, max_recall[1], 1e-7) - self.assertNear(0.2, max_recall[2], 1e-7) - self.assertNear(0.0, max_recall[3], 1e-7) - - def testFindRecallAtGivenPrecision(self): - # The shape of the precision_recall_curves is [n, m, 2] where n is the - # number of classes, m is then number of values in the curve, 2 indexes - # between precision [0] and recall [1]. - car = np.transpose( - np.array( - [[0.9, 0.7, 0.5, 0.1, 0.0, 0.0], [0.0, 0.2, 0.5, 0.9, 1.0, 1.0]], - dtype=np.float32)) - ped = np.transpose( - np.array( - [[0.9, 0.7, 0.5, 0.0, 0.0, 0.0], [0.0, 0.2, 0.5, 0.9, 1.0, 1.0]], - dtype=np.float32)) - cyc = np.transpose( - np.array( - [[0.9, 0.7, 0.0, 0.0, 0.0, 0.0], [0.0, 0.2, 0.5, 0.9, 1.0, 1.0]], - dtype=np.float32)) - foo = np.transpose( - np.array( - [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.2, 0.5, 0.9, 1.0, 1.0]], - dtype=np.float32)) - precision_recall_curves = np.stack([car, ped, cyc, foo]) - precision_level = 0.5 - recall = breakdown_metric._FindRecallAtGivenPrecision( - precision_recall_curves, precision_level) - self.assertAllEqual([4], recall.shape) - self.assertNear(0.5, recall[0], 1e-7) - self.assertNear(0.5, recall[1], 1e-7) - self.assertNear(0.2, recall[2], 1e-7) - self.assertNear(0.0, recall[3], 1e-7) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/builder_lib.py b/lingvo/tasks/car/builder_lib.py deleted file mode 100644 index 05b6ce44..00000000 --- a/lingvo/tasks/car/builder_lib.py +++ /dev/null @@ -1,1151 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""A collection of helper functions to build lingvo layer params.""" - -import functools -import math - -from lingvo import compat as tf -from lingvo.core import builder_layers -from lingvo.core import layers -from lingvo.core import py_utils -from lingvo.tasks.car import car_layers - -import numpy as np - -# Keys for NestedMap for points: points, features, and padding. -POINTS_KEY = 'points' -FEATURES_KEY = 'features' -PADDING_KEY = 'padding' - -# Keys for NestedMap for range images: xyz, features, and mask. -XYZ_KEY = 'xyz' -MASK_KEY = 'mask' - - -################################################################################ -# Initializations -################################################################################ -def TruncatedGaussianInit(filter_shape): - factor = 2.0 - trunc_stddev = math.sqrt(1.3 * factor / filter_shape[-1]) - return py_utils.WeightInit.TruncatedGaussian(scale=trunc_stddev) - - -def KaimingUniformFanInRelu(shape): - del shape - return py_utils.WeightInit.KaimingUniformFanInRelu() - - -################################################################################ -# Lingvo Layer Builders -################################################################################ -# pyformat: disable -class ModelBuilderBase: - """Model builder with commonly used layers.""" - - def __init__(self): - self.conv_init_method = TruncatedGaussianInit - self.linear_params_init = None - self.bn_params_init = None - self.activation_fn = tf.nn.relu - self.fc_bn_after_linear = False - - def _Rep(self, name, repeat, *subs): - """Helper to construct a sequential layer repeated several time.""" - return builder_layers.SequentialLayer.Params().Set( - name=name, repeat=repeat, sub=list(subs)) - - def _Seq(self, name, *subs): - return builder_layers.SequentialLayer.Params().Set( - name=name, repeat=1, sub=list(subs)) - - def _Branch(self, name, body, fetches): - return builder_layers.BranchLayer.Params().Set( - name=name, body=body, fetches=fetches) - - def _BN(self, name, dims): - bn_params = layers.BatchNormLayer.Params().Set( - name=name, - dim=dims, - decay=0.99, - # TODO(b/148537111): consider setting this to True. - add_stats_to_moving_average_variables=False) - if self.bn_params_init: - bn_params = bn_params.Set(params_init=self.bn_params_init) - return bn_params - - def _Linear(self, name, idims, odims, params_init=None): - linear_params = builder_layers.LinearLayer.Params().Set( - name=name, input_dims=idims, output_dims=odims) - - params_init = params_init or self.linear_params_init - if params_init is not None: - linear_params = linear_params.Set(params_init=params_init) - - return linear_params - - def _Bias(self, name, dims, params_init=None): - bias_params = builder_layers.BiasLayer.Params().Set(name=name, dims=dims) - if params_init is not None: - bias_params = bias_params.Set(params_init=params_init) - return bias_params - - def _Activation(self, name, activation_fn): - return builder_layers.MapLayer.Params().Set(name=name, fn=activation_fn) - - def _Relu(self, name): - return self._Activation(name, activation_fn=tf.nn.relu) - - def _Sigmoid(self, name): - return self._Activation(name, activation_fn=tf.nn.sigmoid) - - def _FC(self, name, idims, odims, use_bn=True, activation_fn=None): - """Fully connected layer, with optional batch norm.""" - activation_fn = activation_fn or self.activation_fn - if isinstance(idims, (list, tuple)): - idims = idims[0] - fc_layers = [self._Linear('linear', idims, odims)] - if use_bn and self.fc_bn_after_linear: - # Note that bn should use odims, since after linear. - fc_layers = fc_layers + [self._BN('bn', odims)] - elif use_bn and (not self.fc_bn_after_linear): - # Note that bn should use idims, since before linear. - fc_layers = [self._BN('bn', idims)] + fc_layers - else: - # Add bias since no batch norm that folds in bias. - fc_layers = fc_layers + [self._Bias('bias', odims)] - fc_layers += [self._Activation('activation', activation_fn=activation_fn)] - - return self._Seq(name, *fc_layers) - - def _MLP(self, name, dims, use_bn=True, activation_fn=None): - l = [] - for n, (i, o) in enumerate(zip(dims[:-1], dims[1:])): - l += [self._FC('l%03d' % n, i, o, use_bn=use_bn, - activation_fn=activation_fn)] - return self._Seq(name, *l) - - def _Map(self, name, fn, **kwargs): - return builder_layers.MapLayer.Params().Set(name=name, fn=fn, **kwargs) - - def _Max(self, name): - return self._Map(name=name, fn=tf.reduce_max, kwargs={'axis': -2}) - - def _Reshape(self, name, shape): - return builder_layers.MapLayer.Params().Set( - name=name, fn=tf.reshape, kwargs={'shape': shape}) - - def _Matmul(self, name, *subs): - def ParFn(*xs): - result = xs[0] - for v in xs[1:]: - result = tf.matmul(result, v) - return result - return self._Par(name, ParFn, *subs) - - def _GLU(self, name, idims, odims): - - def Gate(x): - u, v = tf.split(x, 2, axis=-1) - return u * tf.sigmoid(v) - - return self._Seq( - name, - self._Linear('linear', idims, odims * 2), - self._Bias('bias', odims * 2), - builder_layers.MapLayer.Params().Set(name='gate', fn=Gate)) - - def _Dropout(self, name, keep_prob): - return layers.DropoutLayer.Params().Set(name=name, keep_prob=keep_prob) - - def _Fetch(self, name): - return layers.FetchLayer.Params().Set(name=name) - - def _FirstN(self, name, n): - """Return the first n args.""" - return builder_layers.FirstNLayer.Params().Set(name=name, n=n) - - def _ArgIdx(self, name, index): - return builder_layers.ArgIndexLayer.Params().Set(name=name, idx=index) - - def _Join(self, name, *subs): - r"""Juxtapose outputs from \*subs in a single output tuple.""" - - def Join(xs): - arg_lists = [list(x) for x in xs] - result = arg_lists[0] - for v in arg_lists[1:]: - result = result + v - return tuple(result) - - return builder_layers.ParallelLayer.Params().Set( - name=name, sub=list(subs), merge=Join) - - def _Par(self, name, fn, *subs): - """Helper to construct a parallel layer which merge branches.""" - - def _Merge(xs): - rets = [] - for ys in zip(*xs): - rets.append(fn(*ys)) - return tuple(rets) - - # TODO(zhifengc): fill merge_meta to compute flops and output shape, etc. - return builder_layers.ParallelLayer.Params().Set( - name=name, sub=list(subs), merge=_Merge) - - def _Concat(self, name, *subs): - r"""Concatenate outputs from \*subs along the last dimensions.""" - return self._Par(name, lambda *xs: tf.concat(xs, axis=-1), *subs) - - def _BroadcastConcat(self, name, *subs): - r"""Concatenate outputs from \*subs, broadcasting to match leading shape.""" - - def _Merge(*xs): - """Broadcast all dimensions except the last, and concat on last dim.""" - - # Stack all shapes and take max on each dimension to get leading shape. - leading_shape = tf.stack([tf.shape(x)[:-1] for x in xs]) - leading_shape = tf.reduce_max(leading_shape, axis=0) - # Broadcast each x. - broadcast_xs = [] - for x in xs: - broadcast_shape = tf.concat([leading_shape, tf.shape(x)[-1:]], axis=0) - broadcast_xs.append(tf.broadcast_to(x, broadcast_shape)) - - # Concat on last dimension. - concat_xs = tf.concat(broadcast_xs, axis=-1) - - return concat_xs - - return self._Par(name, _Merge, *subs) - - def _ApplyFnMulti(self, name, fn, *subs): - """A common use case where every branch produces a single output tensor.""" - return self._Par(name, fn, *subs) - - def _ApplyInParallelAndMerge(self, name, merge_fn, *subs): - """Applies the subs in parallel to the given input and merges their outputs. - - Args: - name: String layer name. - merge_fn: Function to merge the outputs of `subs`, which will be a - flattened list of subs outputs. For example, if there were 3 subs with - outputs (out1,), (out21, out22), (out31, out32), the output will be - [out1, out21, out22, out31, out32]. - *subs: Modules to be applied in parallel to the input - - Returns: - Params for this layer. - """ - - return self._Seq( - name, - self._Join('parallel_subs', *subs), - self._ApplyFn('merge', merge_fn)) - - def _ApplyFn(self, name, fn): - """Apply fn over the input tuple.""" - return builder_layers.ParallelLayer.Params().Set( - name=name, sub=[self._Seq('id')], merge=lambda xs: (fn(*(xs[0])),)) - - def _MakeInputFeatureFromPoints(self, name): - """Transforms points to features using the points with constant padding. - - Layer input/output shapes: [N x P x 3] -> ([N x P x 3], [N x P x 4]) - - Args: - name: String layer name. - - Returns: - Params for this layer. - """ - - def PadOne(inp): - inp = py_utils.HasShape(inp, [-1, -1, 3]) - return tf.pad(inp, [[0, 0], [0, 0], [0, 1]], constant_values=1.0) - - return self._Join(name, self._Seq('id'), self._ApplyFn('pad', fn=PadOne)) - - def _Squeeze(self, name, axis=None): - def _SqueezeFn(x): - return tf.squeeze(x, axis=axis) - return self._ApplyFn(name, fn=_SqueezeFn) - - def _ConvPlain(self, name, filter_shape, filter_stride=(1, 1), - padding='SAME', conv_init_method=None): - conv_init_method = conv_init_method or self.conv_init_method(filter_shape) - return layers.Conv2DLayerNoPadding.Params().Set( - name=name, - filter_shape=filter_shape, - filter_stride=filter_stride, - padding=padding, - params_init=conv_init_method) - - def _DeconvPlain(self, name, filter_shape, filter_stride=(1, 1)): - return layers.DeconvLayer.Params().Set( - name=name, - filter_shape=filter_shape, - filter_stride=filter_stride, - params_init=self.conv_init_method(filter_shape)) - - def _Conv(self, name, filter_shape, stride=(1, 1), padding='SAME', - use_bn=True): - """Helper to construct a conv/normalize/actvation layer.""" - # TODO(zhifengc): Revisit whether BatchNormLayer should apply gamma when the - # following activation is a relu. - if isinstance(stride, tuple): - filter_stride = stride - elif isinstance(stride, int): - filter_stride = (stride, stride) - else: - raise ValueError( - 'Input stride not a tuple or int. Is a {}'.format(type(stride))) - norm = self._BN('bn', filter_shape[3]) if use_bn else self._Identity(name) - return self._Seq( - name, - self._ConvPlain('conv', filter_shape, filter_stride, padding), - norm, - self._Relu('relu')) - - def _Identity(self, name): - """Apply identity transformation.""" - return layers.IdentityLayer.Params().Set(name=name) - - def _Shortcut(self, name, idims, odims, stride): - """Apply ResNet shortcut transformation.""" - if idims != odims or stride != 1: - return self._ConvPlain(name, (1, 1, idims, odims), stride) - else: - return self._Identity(name) - - def _ResidualLayer(self, name, filter_size, stride): - """ResNet basic layer, as in https://arxiv.org/pdf/1512.03385.pdf Fig. 2. - - Args: - name: string layer name. - filter_size: tuple of integers (filter_height, filter_width, idims, odims) - , which represent filter height, filter width, input channel size, - output channel size. - stride: integer of tuple of integers to apply to all dimensions when using - 2D convolution. This will be applied to the shortcut layer and the - first convolution layer. - - Returns: - Params for a residual layer. - """ - filter_height, filter_width, idims, odims = filter_size - repeated_filter_size = (filter_height, filter_width, odims, odims) - return self._Seq( - name, - self._Add( - 'add', - self._Shortcut('shortcut', idims, odims, stride), - self._Seq( - 'residual', - self._Conv('conv_a', filter_size, stride), - self._ConvPlain('conv_b', repeated_filter_size), - self._BN('residual_bn', odims))), - self._Relu('relu_add')) - - def _ResidualBlock(self, name, filter_size, stride, repeats): - """ResNet block that downsamples at the beginning. - - Args: - name: string block name. - filter_size: tuple of integers (filter_height, filter_width, idims, odims) - , which represent filter height, filter width, input channel size, - output channel size. - stride: integer of tuple of integers to apply to all dimensions when using - 2D convolution. It is applied only to the first downsampling layer for - efficiency. - repeats: integer number of residual layers to stack. - - Returns: - Params for a residual block. - """ - filter_height, filter_width, _, odims = filter_size - repeated_filter_size = (filter_height, filter_width, odims, odims) - return self._Seq( - name, - self._ResidualLayer('res3x3_0', filter_size, stride), - self._Rep( - 'rep', - repeats, - self._ResidualLayer('res3x3_1', repeated_filter_size, 1)), - self._Fetch('final')) - - ########################################## - # Self-attention stack. - # - # NOTE: layers_with_attention has TransformerLayers. We do not use that - # directly because those layers assumes input tensors are in [time, batch, - # dims] so that it's convenient for left-to-right decoding use case. Point - # cloud modeling is unlikely need that. - # - # We also restrict the attention type to the simplest dot-attention for now. - ########################################## - def _SelfAttenStack(self, name, depth, dims, hdims, heads, keep_prob): - """A self-attentional stack. - - Args: - name: string layer name. - depth: int. The number of transformer layers. - dims: int. The dimension of each transformer layer. - hdims: int. The hidden dimension of each transformer layer. - Typically, hdims >= dims. - heads: int. The number of attention heads. - keep_prob: Dropout keep probability. - - Returns: - Params for this layer. - """ - return self._Rep( - name, depth, self._Atten('atten', dims, hdims, heads, keep_prob)) - - def _Atten(self, name, dims, hdims, heads, keep_prob=1.): - """Transformer self-attention layer.""" - return self._Seq( - name, - self._AttenSelf('self', dims, hdims, heads, keep_prob), - self._AttenFF('ff', dims, hdims, keep_prob)) - - def _LN(self, name, dims): - """Layer norm.""" - return layers.LayerNorm.Params().Set(name=name, input_dim=dims) - - def _Project(self, name, idims, odims): - """Project layer. Simply X*W+b.""" - return self._Seq( - name, - self._Linear('linear', idims, odims), - self._Bias('bias', odims)) - - def _Add(self, name, lhs, rhs): - """Add two branches. out = lhs(in) + rhs(in).""" - return self._ApplyFnMulti(name, lambda x, y: x + y, lhs, rhs) - - def _Multiply(self, name, lhs, rhs): - """Multiply two branches. out = lhs(in) * rhs(in).""" - return self._ApplyFnMulti(name, lambda x, y: x * y, lhs, rhs) - - def _AttenFF(self, name, dims, hdims, keep_prob=1.): - """Transformer feed-forward layer.""" - return self._Seq( - name, - self._Add( - 'residual', - self._Seq('id'), - self._Seq( - 'ff', - self._LN('ln', dims), - self._FC('fc', dims, hdims, use_bn=False), - self._Project('proj', hdims, dims), - self._Dropout('dropout', keep_prob)))) - - def _AttenSelf(self, name, dims, hdims, heads, keep_prob=1.): - """Dot-attention, multiple heads, self-attention.""" - assert hdims % heads == 0, 'hdims={} heads={}'.format(hdims, heads) - - def _Atten(query, key, val): - """Returns weighted val based on dot-attention between query and key.""" - b, n, _ = py_utils.GetShape(query) - - # Query. - query = py_utils.HasShape(query, [b, n, hdims]) - query = tf.reshape(query, [b, n, heads, hdims // heads]) - query = tf.transpose(query, [0, 2, 1, 3]) - - # Key. - key = py_utils.HasShape(key, [b, n, hdims]) - key = tf.reshape(key, [b, n, heads, hdims // heads]) - key = tf.transpose(key, [0, 2, 1, 3]) - - # query:[b, heads, n, hdims // heads] - # key: [b, heads, n, hdims // heads]^T - dotp = tf.matmul(query, key, transpose_b=True) - probs = tf.nn.softmax(dotp) - probs = py_utils.HasShape(probs, [b, heads, n, n]) - - # value (aka. context) - val = py_utils.HasShape(val, [b, n, hdims]) - val = tf.reshape(val, [b, n, heads, hdims // heads]) - val = tf.transpose(val, [0, 2, 1, 3]) - val = py_utils.HasShape(val, [b, heads, n, hdims // heads]) - - # Weighted average of value (context). [b, heads, n, hdims // heads] - out = tf.matmul(probs, val) - out = tf.transpose(out, [0, 2, 1, 3]) - out = tf.reshape(out, [b, n, hdims]) - return out - - return self._Seq( - name, - self._Add( - 'residual', - self._Seq('id'), - self._Seq( - 'lapd', - self._LN('ln', dims), - self._ApplyFnMulti( - 'atten', - _Atten, - self._Project('query', dims, hdims), - self._Project('key', dims, hdims), - self._Project('value', dims, hdims)), - self._Project('post', hdims, dims), - self._Dropout('dropout', keep_prob)))) - - ########################################## - # Location and Padding Aware Layers and Helpers - # - # These layers expect as input a NestedMap containing the following keys: - # - # points: tensor with shape [..., 3] containing xyz coordinates - # features: tensor with shape [..., C] containing features for each point - # padding: tensor with same leading shape containing 0/1 with - # 1 representing a padded point, and 0 representing a real point. - # - ########################################## - class _Decorators: - """Internal decorators for builder functions.""" - - @classmethod - def ExpectsNestedMapTensor(cls, expected_keys=()): - """Adds a validation layer before the layer produced by builder_fn. - - Args: - expected_keys: A string, or an iterable of strings that are expected to - be in the input NestedMap. - - Returns: - A decorator function that can be applied on builder functions. - """ - - if isinstance(expected_keys, str): - expected_keys = [expected_keys] - expected_keys = set(expected_keys) - - def _ValidateFn(inp): - """Validate that inp contains the expected keys.""" - if not isinstance(inp, py_utils.NestedMap): - raise ValueError('Input not a `NestedMap`. Is a {}'.format(type(inp))) - input_keys = set(inp.keys()) - if not expected_keys.issubset(input_keys): - missing_keys = expected_keys - input_keys - raise ValueError('Input is missing key(s): {}'.format( - ', '.join(missing_keys))) - return inp - - def _Decorator(builder_fn): - def _DecoratedFn(self, *args, **kwargs): - p = builder_fn(self, *args, **kwargs) - # pylint: disable=protected-access - return self._Seq('validated_%s' % p.name, - self._ApplyFn('validate', fn=_ValidateFn), p) - # pylint: enable=protected-access - return _DecoratedFn - - return _Decorator - - @classmethod - def ExpectsNestedMapPointsTensor(cls, builder_fn): - """Adds a validation layer before the layer produced by builder_fn.""" - return cls.ExpectsNestedMapTensor( - expected_keys=(POINTS_KEY, FEATURES_KEY, PADDING_KEY))(builder_fn) - - @classmethod - def ExpectsNestedMapRangeImage(cls, builder_fn): - """Adds a validation layer before the layer produced by builder_fn.""" - return cls.ExpectsNestedMapTensor( - expected_keys=(XYZ_KEY, FEATURES_KEY, MASK_KEY))(builder_fn) - - @_Decorators.ExpectsNestedMapTensor() - def _GetValue(self, name, key, default=None): - """Expects a NestedMap as input and produces the value for the given key.""" - - def GetValueFn(inp): - if not isinstance(inp, py_utils.NestedMap): - raise ValueError('At layer with name={}. Unable to retrieve key {}, ' - 'input not a `NestedMap`. Is a {}'.format( - name, key, type(inp))) - return inp.get(key, default) - - return self._Map(name, fn=GetValueFn) - - @_Decorators.ExpectsNestedMapTensor() - def _ParMap(self, name, key_to_sub): - """Perform parallel layers and create a NestedMap from the outputs. - - Parallel branches on an input `NestedMap`. Each branch should expect the - same `NestedMap` as input; each branch's output will be mapped to the - specified key in key_to_sub. - - Args: - name: String layer name. - key_to_sub: Dictionary mapping keys to sub params. Each sub should expect - a NestedMap input. - - Returns: - Params for this layer. - """ - sorted_keys = sorted(key_to_sub.keys()) - sorted_subs = [key_to_sub[k] for k in sorted_keys] - - def _MakeNestedMap(*vals): - return py_utils.NestedMap(dict(zip(sorted_keys, vals))) - - return self._ApplyFnMulti(name, _MakeNestedMap, *sorted_subs) - - @_Decorators.ExpectsNestedMapTensor() - def _SeqToKey(self, name, key, *subs): - """Apply sequence to only update the value at a specific key. - - Note: This is the more general version of SeqOnKey. SeqOnKey automatically - extracts the key value before running subs, while SeqToKey passes the entire - input through. - - This function expects a NestedMap as an input, and applies the provided - subs on the entire NestedMap, but only updates the value at the specified - key. The resulting value (tensor or NestedMap) from these subs is placed as - the new value for the specified key in the original NestedMap. The rest of - the key, value pairs are returned untouched. - - Args: - name: string layer name. - key: string key to update after running sequence on. - *subs: A list of sub layer Params that should expect the input NestedMap. - The output of the subs will be placed back to the specified key. - Returns: - Params for this layer. - """ - - def _Merge(outputs): - """Merges the outputs from ParallelLayer. - - Args: - outputs: A tuple of two elements: (a) the original input NestedMap and - (b) the outputs from applying the sub layers in subs. - - Returns: - A new NestedMap map with the specified key's value replaced with the - result from subs. - """ - - input_map = outputs[0][0] - seq_result = outputs[1][0] - new_map = input_map.DeepCopy() - new_map[key] = seq_result - - return (new_map,) - - return builder_layers.ParallelLayer.Params().Set( - name=name, sub=[ - self._Seq('id'), - self._Seq('key_seq', - *subs) - ], merge=_Merge) - - def _SeqOnKey(self, name, key, *subs): - """Sequential layers operating on a specific key only.""" - return self._SeqToKey(name, key, self._GetValue('get_{}'.format(key), key), - *subs) - - @_Decorators.ExpectsNestedMapTensor(FEATURES_KEY) - def _SeqOnFeatures(self, name, *subs): - """Sequential layers operating on the features only.""" - return self._SeqOnKey(name, FEATURES_KEY, *subs) - - @_Decorators.ExpectsNestedMapPointsTensor - def _PaddedMax(self, name): - """Padding aware max pooling layer, emits a single tensor.""" - def _PaddedMaxFn(inp): - """Apply padded max using reduce_max with paddings replaced by neginf.""" - # Replace all padded features with -inf. - neginf_padding = tf.where( - inp.padding > 0, -np.inf * inp.padding, inp.padding) - features = inp.features + neginf_padding[..., tf.newaxis] - features = tf.reduce_max(features, axis=-2) - - # Replace features of all padded points by zeros. If a batch of points are - # all padded, then reduce_min over the padding will be 1. We set the - # features to be zero, so that we don't get any downstream issue with - # NaNs. Note that inf * 0 = NaN. - all_padded = tf.cast(tf.reduce_min(inp.padding, axis=-1), tf.bool) - all_padded = tf.broadcast_to(all_padded[..., tf.newaxis], - py_utils.GetShape(features)) - features = tf.where(all_padded, tf.zeros_like(features), features) - return py_utils.CheckNumerics(features) - - return self._Map(name=name, fn=_PaddedMaxFn) - - @_Decorators.ExpectsNestedMapPointsTensor - def _PaddedMean(self, name): - """Padding aware mean pooling layer, emits a single tensor.""" - - def _PaddedMeanFn(inp): - """Apply padded mean using reduce_sum and dividing by # real points.""" - # Replace all padded features with 0 by masking the padded features out. - mask = 1 - inp.padding - features = inp.features * mask[..., tf.newaxis] - features = tf.reduce_sum(features, axis=-2) - num_real_points = tf.reduce_sum(mask, axis=-1, keepdims=True) - # Prevent the divisor of our padded mean from ever being 0, so that - # the gradient flowing back through this op doesn't give us NaNs. - num_real_points = tf.maximum(num_real_points, 1) - features = features / num_real_points - - # Replace features of all padded points by zeros. If a batch of points are - # all padded, then num_real_points will be zero. We set the features to be - # zero, so that we don't get any downstream issue with NaNs. - # Note that inf * 0 = NaN. - all_padded = tf.equal(num_real_points, 0.) - all_padded = tf.broadcast_to(all_padded, py_utils.GetShape(features)) - features = tf.where(all_padded, tf.zeros_like(features), features) - return py_utils.CheckNumerics(features) - - return self._Map(name=name, fn=_PaddedMeanFn) - - @_Decorators.ExpectsNestedMapPointsTensor - def _PaddedSum(self, name): - """Padding aware sum pooling layer, emits a single tensor.""" - - def _PaddedSumFn(inp): - # Replace all padded features with 0 by masking the padded features out. - mask = 1 - inp.padding - features = inp.features * mask[..., tf.newaxis] - features = tf.reduce_sum(features, axis=-2) - return features - - return self._Map(name=name, fn=_PaddedSumFn) - - @_Decorators.ExpectsNestedMapTensor(FEATURES_KEY) - def _FeaturesFC(self, name, idims, odims, use_bn=True, activation_fn=None): - """Applies a FC layer to `features` key, emits a `NestedMap`.""" - activation_fn = activation_fn or self.activation_fn - bn_join_layer = self._Join( - 'join_features_padding', - self._GetValue('get_features', FEATURES_KEY), - self._Seq('expand_padding', - self._GetValue('get_padding', 'padding'), - self._ApplyFn('expand', - fn=lambda t: t[..., tf.newaxis]))) - if use_bn and self.fc_bn_after_linear: - # Note that bn should use odims, since after linear. - seq_p = [ - self._SeqToKey('linear_seq', - FEATURES_KEY, - self._GetValue('get_features', FEATURES_KEY), - self._Linear('linear', idims, odims)), - bn_join_layer, - self._BN('bn', odims), - ] - elif use_bn and (not self.fc_bn_after_linear): - # Note that bn should use idims, since before linear. - seq_p = [ - bn_join_layer, - self._BN('bn', idims), - self._Linear('linear', idims, odims), - ] - else: - # Add bias since no batch norm that folds in bias. - seq_p = [ - self._GetValue('get_features', FEATURES_KEY), - self._Linear('linear', idims, odims), - self._Bias('bias', odims), - ] - seq_p += [self._Activation('activation', activation_fn=activation_fn)] - return self._SeqToKey(name, FEATURES_KEY, *seq_p) - - @_Decorators.ExpectsNestedMapTensor(FEATURES_KEY) - def _FeaturesMLP(self, name, dims, use_bn=True, activation_fn=None): - l = [] - for n, (i, o) in enumerate(zip(dims[:-1], dims[1:])): - l += [self._FeaturesFC('l%03d' % n, i, o, use_bn=use_bn, - activation_fn=activation_fn)] - return self._Seq(name, *l) - - def _CondFC(self, name, idims, adims, odims, use_bn=True, activation_fn=None): - """Conditional FC layer to use with GIN as a combiner. - - ([..., P, idims], [..., 1, adims]) -> [..., P, odims] - - This layer expects an input tuple (features, aggregate), where features - contains per-point features, and aggregate is a global feature (e.g., - computed using max-pooling over all the points). The aggregate tensor - is used to compute a linear transformation that is used in this FC layer. - Each example in the batch has a different linear transformation. This layer - is similar to the T-Net transformation in PointNet. - - Args: - name: String name for this layer. - idims: Dimension for last axis of features tensor. - adims: Dimension for last axis of aggregate tensor. - odims: Number of output dimensions. - use_bn: Whether to enable batch norm. - activation_fn: Optional override for the activation function. If None, - defaults to the activation fn that the builder is initialized with. - - Returns: - Params for a layer. - """ - activation_fn = activation_fn or self.activation_fn - - def _ReshapeTransform(inp): - """Reshape the transformation tensor to [..., idims, odims].""" - base_shape = py_utils.GetShape(inp)[:-1] - out_shape = list(base_shape) + [idims, odims] - return tf.reshape(inp, out_shape) - - transform_net = self._Matmul( - 'cond_transform', - self._Seq( - 'prep_features', - self._ArgIdx('arg0', index=[0]), - self._BN('bn', idims) if use_bn else self._Seq('id')), - self._Seq( - 'compute_linear_transform', - self._ArgIdx('arg1', index=[1]), - self._Squeeze('squeeze', axis=-2), - # TODO(jngiam): Consider other configurations for FC dims here. - self._FC('fc0', adims, adims * 2, use_bn=use_bn, - activation_fn=activation_fn), - # We use identity for the output since we want the transformation - # matrix to have negative values too. - self._FC('fc1', adims * 2, idims * odims, use_bn=use_bn, - activation_fn=tf.identity), - self._ApplyFn('reshape', fn=_ReshapeTransform))) - - return self._Seq( - name, - transform_net, - self._Seq('id') if use_bn else self._Bias('bias', odims), - self._Activation('activation', activation_fn=activation_fn)) - - def _GINCondFC(self, name, lhs, rhs, idims, adims, odims, use_bn=True, - activation_fn=None): - return self._Seq( - name, - self._Join('join', lhs, rhs), - self._CondFC('cond_fc', idims, adims, odims, use_bn=use_bn, - activation_fn=activation_fn)) - - @_Decorators.ExpectsNestedMapPointsTensor - def _GIN(self, - name, - mlp_dims, - aggregate_sub, - readout_sub, - combine_method='add', - eps=0., - use_bn=True): - """Graph Isomorphism Network (GIN) [1]. - - This implements the GIN network. The input goes through multiple - GINIntermediateLayers which performs the update equation (eqn 4.1 in paper). - - The final network output is the concatenation of all the intermediate - layer outputs (eqn 4.2 in paper), with readout aggregation being mean or - max. - - output = concat([readout(f0), readout(f1),..., readout(fN)]), - where f1 = intermediate_gin(g0), etc. - - This is akin to a DenseNet structure, where all the intermediate layers - are short-circuited to the end. - - Using sum corresponds to the proposed method by [1], which enables learning - graphs with repeated nodes. 'mean' and 'max' may confuse graphs with - repeated nodes but could better capture distribution statistics or distinct - elements. - - Note that eps=0 performs well in their results (Section 7). - - [1] How Powerful are Graph Neural Networks? - Keyulu Xu, Weihua Hu, Jure Leskovec, Stefanie Jegelka. ICLR 2019. - https://arxiv.org/abs/1810.00826 - - Args: - name: String layer name. - mlp_dims: List of list of ints representing the intermediate feature MLPs. - The first dim should match the dim of the input features. Each output - dim of the MLP should match the next. - aggregate_sub: Params for a layer that has padding-aware aggregation - (e.g., PaddedMean, PaddedMax, PaddedSum). This is used by the - intermediate layers. - readout_sub: Params for a layer that has padding-aware aggregation - (e.g., PaddedMean, PaddedMax, PaddedSum). This is used to aggregate - the feature vectors for representing the entire graph. - combine_method: Either 'add', 'concat', or 'cond_fc'. This is used by the - intermediate layers. - eps: float scale parameter, see _GINIntermediateLayer for details. - use_bn: Whether to enable batch norm in intermediate FC layers. - - Returns: - Params for this layer. This layer expects a `NestedMap` with points, - features, and padding. - """ - if combine_method not in ['add', 'concat', 'cond_fc']: - raise ValueError('Unexpected combine method: {}'.format(combine_method)) - - # Validate mlp_dims; every adjacent MLP should have a matching number of - # channels. - for idx, (mlp_i, mlp_o) in enumerate(zip(mlp_dims[:-1], mlp_dims[1:])): - prev_layer_dims = mlp_i[-1] - next_layer_dims = mlp_o[0] - # When concatenating, we expect the next layer to have double the input - # channels. - if combine_method == 'concat': - prev_layer_dims *= 2 - if prev_layer_dims != next_layer_dims: - raise ValueError( - 'mlp_dims do not match ({} != {}) at layer {} ' - 'with dims: {} and {}'.format( - idx, prev_layer_dims, next_layer_dims, mlp_i, mlp_o)) - - # Build the network recursively in a way that concats all the - # intermediate outputs. - def _Build(depth): - if depth == len(mlp_dims): - return readout_sub - else: - return self._Concat( - 'gin_concat', - readout_sub, - self._Seq( - 'seq', - self._GINIntermediateLayer( - 'gin_intermediate', - mlp_dims[depth], - aggregate_sub, - combine_method, - eps, - use_bn), - _Build(depth + 1))) - - p = _Build(depth=0) - - return self._Seq(name, p) - - @_Decorators.ExpectsNestedMapPointsTensor - def _GINIntermediateLayer( - self, - name, - dims, - aggregate_sub, - combine_method='add', - eps=0., - use_bn=True): - """Implements an intermediate GIN layer. - - This layer takes in a `NestedMap` and produces a `NestedMap` that contains - points, features, and padding. - - Given features f_i for each point i, compute new features f'_i: - - f'_i = MLP((1 + eps) * f_i + aggregate(f_j)) where j are neighbors of i - - Note: This layer aggregates over the entire point cloud, not just the - neighbors. In practice, aggregation over the neighbors can be done by having - pre-grouping the points before invoking this layer. - - Args: - name: String layer name. - dims: List of int dims for the MLP layer. - aggregate_sub: Params for a layer that has padding-aware aggregation - (e.g., PaddedMean, PaddedMax, PaddedSum). - combine_method: Either 'add', 'concat', or 'cond_fc'. Add follows the - paper suggestion of adding the aggregated feature to each point. Concat - broadcasts the aggregated features and concats it to each point. Cond_FC - computes an example-dependent linear transformation; this is similar to - T-Net. - eps: float scale parameter, see equation above for details. - use_bn: Whether to enable batch norm in FC layers. - - Returns: - Params for this layer. - """ - combine_method_to_fn = { - 'add': self._Add, - 'concat': self._BroadcastConcat, - 'cond_fc': functools.partial( - self._GINCondFC, - idims=dims[0], odims=dims[0], adims=dims[0], - use_bn=use_bn) - } - if combine_method not in combine_method_to_fn: - raise ValueError('Unexpected combine method: {}'.format(combine_method)) - combine_fn = combine_method_to_fn[combine_method] - return self._Seq( - name, - self._SeqToKey( - 'map_features', - FEATURES_KEY, - combine_fn( - 'combine', - self._Seq( - 'left', - self._GetValue('get_features', FEATURES_KEY), - self._ApplyFn('eps_scale', fn=lambda t: (1. + eps) * t)), - self._Seq( - 'right', - aggregate_sub, - self._ApplyFn('expand_dims', - fn=lambda t: tf.expand_dims(t, axis=-2))))), - self._FeaturesMLP('mlp', dims, use_bn=use_bn)) - - @_Decorators.ExpectsNestedMapPointsTensor - def _ConcatPointsToFeatures(self, name): - return self._SeqToKey( - name, - FEATURES_KEY, - self._Concat( - 'concat', - self._GetValue('get_points', 'points'), - self._GetValue('get_features', FEATURES_KEY))) - - @_Decorators.ExpectsNestedMapPointsTensor - def _SetAbstraction(self, - name, - feature_extraction_sub, - num_samples, - group_size, - ball_radius, - sample_neighbors_uniformly=True): - """Set abstraction layer that samples, groups points, and featurizes groups. - - This is based on PointNet++ concept of set abstractions. - - This layer samples `num_samples` points using Farthest Point Sampling - algorithm. For each sampled point, it forms a group of size `group_size` - consisting the points within the distance of `ball_radius` from the sampled - point. - - It then applies the feature_extraction_sub to the extracted groups, note - that the tensors passed to the feature_extraction_sub will have leading - shapes corresponding to [batch_size, num_query_points, num_points_per_group, - ...]. - - A simple feature_extraction_sub that does a MLP followed by max pooling - on the grouped points could be composed using FeaturesMLP and PaddedMax. - - This layer expects a NestedMap of points, features, and padding; it produces - a corresponding NestedMap with updated points, features, and padding. - - Args: - name: name for the layer. - feature_extraction_sub: Params for a layer that takes in a NestedMap of - points, features, and padding. These tensors correspond to grouped - points. Note that the leading shape of these tensors are [batch_size, - num_query_points, num_points_per_group, ...]. This layer should produce - a single tensor representing the features computed by summarizing the - grouped points. - num_samples: Number of points to be sampled. - group_size: Number neighbours for each sampled point. - ball_radius: The distance around each sampled point to obtain neighbours. - sample_neighbors_uniformly: Whether to sample neighbors uniformly within - the ball radius. - - Returns: - Params for Set Abstraction layer based on PointNet++. - """ - return self._Seq( - name, - car_layers.SamplingAndGroupingLayer.Params().Set( - name='sample_group', - num_samples=num_samples, - ball_radius=ball_radius, - group_size=group_size, - sample_neighbors_uniformly=sample_neighbors_uniformly), - # Output of sampling and grouping is a NestedMap of grouped_points and - # query_points. `query_points` is a nestedmap consists of the sampled - # points and padding. `grouped_points` consists of the groups around the - # sampled points, corresponding features and padding. - self._ParMap( - 'pmap', - dict( - points=self._Seq('seq_points', - self._GetValue('q', 'query_points'), - self._GetValue('p', 'points')), - features=self._Seq( - 'seq_features', - self._GetValue('q', 'grouped_points'), - feature_extraction_sub), - padding=self._Seq('seq_padding', - self._GetValue('q', 'query_points'), - self._GetValue('p', 'padding')), - ))) # pyformat: disable - - @_Decorators.ExpectsNestedMapPointsTensor - def _PointConvParametricConv(self, - name, - mlp_dims, - num_in_channels, - num_out_channels): - """Parametric convolution based on PointConv. - - Note that this layer assumes that the features have already been weighted - by density. This layer follows Figure 5. in [1] but without the initial - inverse density scale weighting part. When working with range images, - one can use RIScaleFeaturesByDensity to scale the features beforehand. - - [1] PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019. - Wu, Wenxuan and Qi, Zhongang and Fuxin, Li. - - Args: - name: string name for this layer. - mlp_dims: dims for mlp applied to the points. the first dimension of - mlp_dims must be 3. - num_in_channels: integer number of input channels (features) - num_out_channels: integer number of output channels (features). - - Returns: - Params for a parametric conv layer. - """ - if mlp_dims[0] != 3: - raise ValueError( - 'First dimension of mlp_dims must be 3. mlp_dims={}'.format(mlp_dims)) - - def _CombineLastTwoDims(x): - shape = py_utils.GetShape(x) - return tf.reshape(x, shape[:-2] + [np.prod(shape[-2:])]) - - return self._Seq( - name, - self._Par( - 'transpose_matmul', - lambda x, y: tf.matmul(x, y, transpose_a=True), - # Features [..., points, num_in_channels]. - self._GetValue('get_features', 'features'), - self._Seq( - 'transform_points', - # Map points into features to use FeaturesMLP which is padding - # batchnorm aware. - self._SeqToKey( - 'points_as_features', 'features', - self._GetValue('get_points', 'points')), - self._FeaturesMLP('points_mlp', mlp_dims), - # Output of this should be [..., points, mlp_dims[-1]]. - self._GetValue('get_transformed_points', 'features'))), - # Post transform_matmul, should be [..., num_in_channels, mlp_dims[-1]]. - # Note: The paper's use of conv is equivalent to reshaping so that the - # last two dims are combined together. - self._ApplyFn('reshape', fn=_CombineLastTwoDims), - # TODO(jngiam): Consider handling batch norm carefully here, not all - # center points have valid values. - self._FC('fc', num_in_channels * mlp_dims[-1], num_out_channels)) -# pyformat: enable diff --git a/lingvo/tasks/car/builder_lib_test.py b/lingvo/tasks/car/builder_lib_test.py deleted file mode 100644 index 48fb6f85..00000000 --- a/lingvo/tasks/car/builder_lib_test.py +++ /dev/null @@ -1,466 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for builder_lib.""" - -from lingvo import compat as tf -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.tasks.car import builder_lib -import numpy as np - - -class BuilderLibTest(test_utils.TestCase): - - def _TestFProp(self, p, in_shape, expected_out_shape): - g = tf.Graph() - with g.as_default(): - l = p.Instantiate() - x = tf.random.normal(shape=in_shape) - y = l.FPropDefaultTheta(x) - if isinstance(y, (list, tuple)): - self.assertEqual(len(y), 1) - y = y[0] - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - val = self.evaluate(y) - self.assertEqual(val.shape, expected_out_shape) - - def testLN(self): - self._TestFProp(builder_lib.ModelBuilderBase()._LN('ln', 128), - (4, 100, 128), (4, 100, 128)) - - def testProject(self): - self._TestFProp(builder_lib.ModelBuilderBase()._Project('p', 64, 128), - (4, 100, 64), (4, 100, 128)) - - def testAdd(self): - b = builder_lib.ModelBuilderBase() - p = b._Add('p', b._Seq('l'), b._Seq('r')) - self._TestFProp(p, (4, 100, 64), (4, 100, 64)) - - def testAttenFF(self): - b = builder_lib.ModelBuilderBase() - p = b._AttenFF('p', 128, 256, 0.8) - self._TestFProp(p, (4, 100, 128), (4, 100, 128)) - - def testAttenSelf(self): - b = builder_lib.ModelBuilderBase() - p = b._AttenSelf('p', 128, 256, 4, 0.8) - self._TestFProp(p, (4, 100, 128), (4, 100, 128)) - - def testAtten(self): - b = builder_lib.ModelBuilderBase() - p = b._Atten('p', 128, 256, 4, 0.8) - self._TestFProp(p, (4, 100, 128), (4, 100, 128)) - - def testSelfAttenStack(self): - b = builder_lib.ModelBuilderBase() - p = b._SelfAttenStack('p', 6, 128, 256, 4, 0.8) - self._TestFProp(p, (4, 100, 128), (4, 100, 128)) - - def testGetValue(self): - b = builder_lib.ModelBuilderBase() - p = b._GetValue('p', 'test_key') - l = p.Instantiate() - x = py_utils.NestedMap({ - 'test_key': tf.constant(2.0), - 'ignore_key': tf.constant(1.0) - }) - y = l.FPropDefaultTheta(x) - with self.session(): - actual_y = self.evaluate(y) - self.assertEqual(actual_y, 2.0) - - def testGetValueDefault(self): - b = builder_lib.ModelBuilderBase() - p = b._GetValue('p', 'not_a_key', default=tf.constant(3.0)) - l = p.Instantiate() - x = py_utils.NestedMap({ - 'test_key': tf.constant(2.0), - 'ignore_key': tf.constant(1.0) - }) - y = l.FPropDefaultTheta(x) - with self.session(): - actual_y = self.evaluate(y) - self.assertEqual(actual_y, 3.0) - - def testSeqToKeyFeaturesOnly(self): - b = builder_lib.ModelBuilderBase() - p = b._SeqToKey('p', 'features', b._GetValue('get_features', 'features'), - b._Sigmoid('sigmoid')) - l = p.Instantiate() - x = py_utils.NestedMap({ - 'points': tf.constant(3.0), - 'features': tf.constant(0.0), - 'padding': tf.constant(1.0), - }) - y = l.FPropDefaultTheta(x) - with self.session(): - actual_y = self.evaluate(y) - self.assertDictEqual( - actual_y, { - 'points': np.asarray([3.0]), - 'features': np.asarray([0.5]), - 'padding': np.asarray([1.0]), - }) - - def testSeqToKeyFeaturesRaisesIfNotNestedMap(self): - b = builder_lib.ModelBuilderBase() - p = b._SeqToKey('p', 'features', b._GetValue('get_features', 'features'), - b._Sigmoid('sigmoid')) - l = p.Instantiate() - x = tf.constant(1.0) - with self.assertRaisesRegex(ValueError, - r'Input not a `NestedMap`. Is a .*'): - l.FPropDefaultTheta(x) - - def testSeqToKey(self): - b = builder_lib.ModelBuilderBase() - # Update features with sigmoid(padding). - p = b._SeqToKey('p', 'features', b._GetValue('get_padding', 'padding'), - b._Sigmoid('sigmoid')) - l = p.Instantiate() - x = py_utils.NestedMap({ - 'points': tf.constant(3.0), - 'features': tf.constant(5.0), - 'padding': tf.constant(0.0), - 'extra_key': tf.constant(1.0), - }) - y = l.FPropDefaultTheta(x) - with self.session(): - actual_y = self.evaluate(y) - self.assertDictEqual( - actual_y, { - 'points': np.asarray([3.0]), - 'features': np.asarray([0.5]), - 'padding': np.asarray([0.0]), - 'extra_key': np.asarray([1.0]), - }) - - def _getNestedMapTestData(self): - np_x = py_utils.NestedMap({ - 'points': - np.random.rand(3, 5, 3), - 'features': - np.random.rand(3, 5, 5), - 'padding': - np.asarray([[0, 0, 1, 1, 1], [0, 1, 1, 1, 1], [1, 1, 1, 1, 1]]), - }) - x = py_utils.NestedMap({ - 'points': tf.constant(np_x.points, dtype=tf.float32), - 'features': tf.constant(np_x.features, dtype=tf.float32), - 'padding': tf.constant(np_x.padding, dtype=tf.float32), - }) - return np_x, x - - def testPaddedMax(self): - b = builder_lib.ModelBuilderBase() - p = b._PaddedMax('p') - l = p.Instantiate() - - np_x, x = self._getNestedMapTestData() - y = l.FPropDefaultTheta(x) - - expected_y = np.stack([ - # First example should take max over the first two points. - np.amax(np_x.features[0, :2, :], axis=0), - # Second example should take max over only the first points. - np_x.features[1, 0, :], - # Third example should be all zeros, since all points are padded. - np.zeros_like(np_x.features[2, 0, :]), - ], axis=0) # pyformat: disable - with self.session(): - actual_y = self.evaluate(y) - self.assertAllClose(actual_y, expected_y) - - def testPaddedMean(self): - b = builder_lib.ModelBuilderBase() - p = b._PaddedMean('p') - l = p.Instantiate() - - np_x, x = self._getNestedMapTestData() - y = l.FPropDefaultTheta(x) - - expected_y = np.stack([ - # First example should take mean over the first two points. - np.mean(np_x.features[0, :2, :], axis=0), - # Second example should take mean over only the first points. - np_x.features[1, 0, :], - # Third example should be all zeros, since all points are padded. - np.zeros_like(np_x.features[2, 0, :]), - ], axis=0) # pyformat: disable - with self.session(): - actual_y = self.evaluate(y) - self.assertAllClose(actual_y, expected_y) - - def testPaddedMeanGrad(self): - b = builder_lib.ModelBuilderBase() - p = b._Seq('seq', b._FeaturesFC('fc', 5, 10), b._PaddedMean('p')) - l = p.Instantiate() - - _, x = self._getNestedMapTestData() - y = l.FPropDefaultTheta(x) - loss = tf.reduce_sum(y) - - all_vars = tf.trainable_variables() - grads = tf.gradients(loss, all_vars) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - np_grads = self.evaluate(grads) - for np_grad in np_grads: - self.assertTrue(np.all(np.isfinite(np_grad))) - - def testPaddedSum(self): - b = builder_lib.ModelBuilderBase() - p = b._PaddedSum('p') - l = p.Instantiate() - - np_x, x = self._getNestedMapTestData() - y = l.FPropDefaultTheta(x) - - expected_y = np.stack([ - # First example should take sum over the first two points. - np.sum(np_x.features[0, :2, :], axis=0), - # Second example should take sum over only the first points. - np_x.features[1, 0, :], - # Third example should be all zeros, since all points are padded. - np.zeros_like(np_x.features[2, 0, :]), - ], axis=0) # pyformat: disable - with self.session(): - actual_y = self.evaluate(y) - self.assertAllClose(actual_y, expected_y) - - def testFeaturesFC(self): - b = builder_lib.ModelBuilderBase() - p = b._FeaturesFC('p', idims=5, odims=6, use_bn=True) - l = p.Instantiate() - - np_x, x = self._getNestedMapTestData() - y = l.FPropDefaultTheta(x) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - actual_y = self.evaluate(y) - # Points and padding should be equal, and features should be transformed. - self.assertAllClose(actual_y.points, np_x.points) - self.assertAllEqual(actual_y.padding, np_x.padding) - self.assertAllEqual(actual_y.features.shape, (3, 5, 6)) - self.assertTrue(np.all(np.isfinite(actual_y.features))) - - def testFeaturesFCNoBatchNorm(self): - b = builder_lib.ModelBuilderBase() - p = b._FeaturesFC('p', idims=5, odims=6, use_bn=False) - l = p.Instantiate() - - np_x, x = self._getNestedMapTestData() - y = l.FPropDefaultTheta(x) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - actual_y = self.evaluate(y) - # Points and padding should be equal, and features should be transformed. - self.assertAllClose(actual_y.points, np_x.points) - self.assertAllEqual(actual_y.padding, np_x.padding) - self.assertAllEqual(actual_y.features.shape, (3, 5, 6)) - self.assertTrue(np.all(np.isfinite(actual_y.features))) - - def testFeaturesFCBNAfterLinear(self): - b = builder_lib.ModelBuilderBase() - b.fc_bn_after_linear = True - p = b._FeaturesFC('p', idims=5, odims=6, use_bn=True) - l = p.Instantiate() - - np_x, x = self._getNestedMapTestData() - y = l.FPropDefaultTheta(x) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - actual_y = self.evaluate(y) - # Points and padding should be equal, and features should be transformed. - self.assertAllClose(actual_y.points, np_x.points) - self.assertAllEqual(actual_y.padding, np_x.padding) - self.assertAllEqual(actual_y.features.shape, (3, 5, 6)) - self.assertTrue(np.all(np.isfinite(actual_y.features))) - - def testCondFC(self): - b = builder_lib.ModelBuilderBase() - p = b._CondFC('p', idims=10, adims=8, odims=12) - l = p.Instantiate() - y = l.FPropDefaultTheta( - tf.random.uniform((3, 4, 5, 10)), tf.random.uniform((3, 4, 1, 8))) - with self.session(): - self.evaluate(tf.global_variables_initializer()) - actual_y = self.evaluate(y) - self.assertAllEqual(actual_y.shape, (3, 4, 5, 12)) - self.assertTrue(np.all(np.isfinite(actual_y))) - - def testGINWithCondFC(self): - b = builder_lib.ModelBuilderBase() - p = b._GIN( - 'p', [[5, 2, 6], [6, 8, 7]], - b._PaddedMax('p_max'), - b._PaddedMax('p_max'), - combine_method='cond_fc') - l = p.Instantiate() - - _, x = self._getNestedMapTestData() - y = l.FPropDefaultTheta(x) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - actual_y = self.evaluate(y) - # We expect the output to have all the features concatenated together. - self.assertAllEqual(actual_y.shape, (3, 5 + 6 + 7)) - self.assertTrue(np.all(np.isfinite(actual_y))) - - def testGINIntermediateLayer(self): - b = builder_lib.ModelBuilderBase() - p = b._GINIntermediateLayer('p', [5, 3, 6], b._PaddedSum('sum')) - l = p.Instantiate() - - np_x, x = self._getNestedMapTestData() - y = l.FPropDefaultTheta(x) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - actual_y = self.evaluate(y) - # Points and padding should be equal, and features should be transformed. - self.assertAllClose(actual_y.points, np_x.points) - self.assertAllEqual(actual_y.padding, np_x.padding) - self.assertAllEqual(actual_y.features.shape, (3, 5, 6)) - self.assertTrue(np.all(np.isfinite(actual_y.features))) - - def testGINLayer(self): - b = builder_lib.ModelBuilderBase() - p = b._GIN('p', [[5, 2, 6], [6, 8, 7]], b._PaddedMax('p_max'), - b._PaddedMax('p_max')) - l = p.Instantiate() - - _, x = self._getNestedMapTestData() - y = l.FPropDefaultTheta(x) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - actual_y = self.evaluate(y) - # We expect the output to have all the features concatenated together. - self.assertAllEqual(actual_y.shape, (3, 5 + 6 + 7)) - self.assertTrue(np.all(np.isfinite(actual_y))) - - def testGINLayerRaisesUnexpectedCombiner(self): - b = builder_lib.ModelBuilderBase() - with self.assertRaisesRegex(ValueError, 'Unexpected combine method: .*'): - b._GIN( - 'p', [[5, 2, 6], [6, 8, 7]], - b._PaddedMax('p_max'), - b._PaddedMax('p_max'), - combine_method='unknown') - - def testGINLayerRaisesDimsMismatch(self): - b = builder_lib.ModelBuilderBase() - with self.assertRaisesRegex(ValueError, 'mlp_dims do not match .*'): - b._GIN( - 'p', [[5, 2, 6], [6, 8, 7]], - b._PaddedMax('p_max'), - b._PaddedMax('p_max'), - combine_method='concat') - - def testBroadcastConcat(self): - b = builder_lib.ModelBuilderBase() - p = b._BroadcastConcat('p', b._ArgIdx('arg0', [0]), b._ArgIdx('arg1', [1])) - l = p.Instantiate() - - # 2 x 1 x 3 - x1 = np.asarray([[[1, 2, 3]], [[4, 5, 6]]]) - self.assertEqual(x1.shape, (2, 1, 3)) - # 1 x 4 x 2 - x2 = np.asarray([[[7, 8], [9, 10], [11, 12], [13, 14]]]) - self.assertEqual(x2.shape, (1, 4, 2)) - - y = l.FPropDefaultTheta(tf.constant(x1), tf.constant(x2)) - - expected_y = np.concatenate([ - np.broadcast_to(x1, [2, 4, 3]), - np.broadcast_to(x2, [2, 4, 2]), - ], axis=-1) # pyformat: disable - with self.session(): - actual_y = self.evaluate(y) - # x1 will be broadcasted to [2, 4, 3] - # x2 will be broadcasted to [2, 4, 2] - # Concatenation on axis=-1 should result in a tensor of shape [2, 4, 5] - self.assertEqual(actual_y.shape, (2, 4, 5)) - self.assertAllEqual(actual_y, expected_y) - - def testGINLayerWithConcat(self): - b = builder_lib.ModelBuilderBase() - p = b._GIN( - # The first dim in each MLP should be double the last output as we are - # using concat combiner. - 'p', - [[10, 2, 6], [12, 8, 7]], - b._PaddedMax('p_max'), - b._PaddedMax('p_max'), - combine_method='concat') - l = p.Instantiate() - - _, x = self._getNestedMapTestData() - y = l.FPropDefaultTheta(x) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - actual_y = self.evaluate(y) - # We expect the output to have all the features concatenated together. - self.assertAllEqual(actual_y.shape, (3, 5 + 6 + 7)) - self.assertTrue(np.all(np.isfinite(actual_y))) - - def testResidualBlock(self): - b = builder_lib.ModelBuilderBase() - p_square_stride = b._ResidualBlock('residual_block', (3, 3, 4, 8), (2, 2), - 2) - self._TestFProp(p_square_stride, (4, 100, 100, 4), (4, 50, 50, 8)) - p_nonsquare_stride = b._ResidualBlock('residual_block', (3, 3, 4, 8), - (2, 1), 2) - self._TestFProp(p_nonsquare_stride, (4, 100, 100, 4), (4, 50, 100, 8)) - - def testPointConvParametricConvShapes(self): - batch_size, num_groups, points_per_group, num_in_channels = 4, 5, 6, 7 - num_out_channels = 8 - b = builder_lib.ModelBuilderBase() - p = b._PointConvParametricConv('test', [3, 4, 9], num_in_channels, - num_out_channels) - l = p.Instantiate() - x = py_utils.NestedMap( - points=tf.random.uniform((batch_size, num_groups, points_per_group, 3), - dtype=tf.float32), - features=tf.random.uniform( - (batch_size, num_groups, points_per_group, num_in_channels), - dtype=tf.float32), - padding=tf.cast( - tf.random.uniform((batch_size, num_groups, points_per_group), - minval=0, - maxval=2, - dtype=tf.int32), tf.float32)) - y = l.FPropDefaultTheta(x) - - with self.session(): - self.evaluate(tf.global_variables_initializer()) - actual_y = self.evaluate(y) - self.assertAllEqual(actual_y.shape, - (batch_size, num_groups, num_out_channels)) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/calibration_processing.py b/lingvo/tasks/car/calibration_processing.py deleted file mode 100644 index 5060096c..00000000 --- a/lingvo/tasks/car/calibration_processing.py +++ /dev/null @@ -1,228 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Library for calculating calibration on a prediction.""" - -from lingvo import compat as tf -from lingvo.core import plot -import numpy as np - - -def ExpectedCalibrationError(confidence, - empirical_accuracy, - num_examples, - min_confidence=None): - """Calculate the expected calibration error. - - Args: - confidence: 1-D np.array of float32 binned confidence scores with one number - per bin - empirical_accuracy: 1-D np.array of float32 binned empirical accuracies with - one number per bin - num_examples: 1-D np.array of int for the number of examples within a bin. - min_confidence: float32 of minimum confidence score to use in the - calculation. If None, no filtering is applied. - - Returns: - float32 of expected calibration error - """ - assert confidence.shape[0] == empirical_accuracy.shape[0] - assert empirical_accuracy.shape[0] == num_examples.shape[0] - - ece = np.abs(empirical_accuracy - confidence) * num_examples - - if min_confidence: - bin_indices = np.where(confidence > min_confidence) - ece = ece[bin_indices] - num_examples = num_examples[bin_indices] - - ece = np.sum(ece) - total_num_examples = np.sum(num_examples) - if total_num_examples != 0: - ece /= total_num_examples - else: - ece = 0.0 - return ece - - -def CalibrationCurve(scores, hits, num_bins): - """Compute data for calibration reliability diagrams. - - Args: - scores: 1-D np.array of float32 confidence scores - hits: 1-D np.array of int32 (either 0 or 1) indicating whether predicted - label matches the ground truth label - num_bins: int for the number of calibration bins - - Returns: - A tuple containing: - - mean_predicted_accuracies: np.array of mean predicted accuracy for each - bin - - mean_empirical_accuracies: np.array of mean empirical accuracy for each - bin - - num_examples: np.array of the number of examples in each bin - """ - mean_predicted_accuracies = [] - mean_empirical_accuracies = [] - num_examples = [] - - # Bin the hits and scores based on the scores. - edges = np.linspace(0.0, 1.0, num_bins + 1) - bin_indices = np.digitize(scores, edges, right=True) - # Put examples with score equal to 0 in bin 1 because we will skip bin 0. - bin_indices = np.where(scores == 0.0, 1, bin_indices) - - for j in range(num_bins + 1): - if j == 0: - continue - indices = np.where(bin_indices == j)[0] - # pylint: disable=g-explicit-length-test - if len(indices) > 0: - mean_predicted_accuracy = np.mean(scores[indices]) - mean_empirical_accuracy = np.mean(hits[indices]) - num_example = len(indices) - else: - mean_predicted_accuracy = (edges[j - 1] + edges[j]) / 2.0 - mean_empirical_accuracy = 0.0 - num_example = 0 - # pylint: enable=g-explicit-length-test - - mean_predicted_accuracies.append(mean_predicted_accuracy) - mean_empirical_accuracies.append(mean_empirical_accuracy) - num_examples.append(num_example) - - mean_predicted_accuracies = np.array(mean_predicted_accuracies) - mean_empirical_accuracies = np.array(mean_empirical_accuracies) - num_examples = np.array(num_examples) - return mean_predicted_accuracies, mean_empirical_accuracies, num_examples - - -class CalibrationCalculator: - """Base class for calculating calibration on a prediction.""" - - def __init__(self, metadata): - self._metadata = metadata - self._num_calibration_bins = self._metadata.NumberOfCalibrationBins() - self._calibration_by_class = None - self._classnames = self._metadata.ClassNames() - self._classids = self._metadata.EvalClassIndices() - - def Calculate(self, metrics): - """Calculate metrics for calibration. - - Args: - metrics: A dict. Each entry in the dict is a list of C (number of classes) - dicts containing mapping from metric names to individual results. - Individual entries may be the following items: - - scalars: A list of C (number of classes) dicts mapping metric names to - scalar values. - - curves: A list of C dicts mapping metrics names to np.float32 arrays of - shape [NumberOfPrecisionRecallPoints()+1, 2]. In the last dimension, 0 - indexes precision and 1 indexes recall. - - calibrations: A list of C dicts mapping metrics names to np.float32 - arrays of shape [number of predictions, 2]. The first column is the - predicted probabilty and the second column is 0 or 1 indicating that the - prediction matched a ground truth item. - - Returns: - nothing - """ - if 'calibrations' not in metrics: - tf.logging.info( - 'CalibrationProcessing invoked but no metrics available ' - 'for calculating calibration.') - return - - self._calibration_by_class = {} - for i, c in enumerate(metrics['calibrations']): - classid = self._classids[i] - classname = self._classnames[classid] - - if np.all(np.isnan(c['calibrations'])) or c['calibrations'].size == 0: - tf.logging.info( - 'Skipping %s for calibration calculation because no ' - 'output provided.' % classname) - continue - tf.logging.info('Calculating calibration for %s: %d items.' % - (classname, len(c['calibrations']))) - - # Ensure that all counts are greater then zero and less then or equal - # to 1.0 to guarantee that all scores are counted. - scores_and_hits = np.clip(c['calibrations'], 1e-10, 1.0) - scores = scores_and_hits[:, 0] - hits = scores_and_hits[:, 1] - curve_data = CalibrationCurve(scores, hits, self._num_calibration_bins) - self._calibration_by_class[classname] = np.array(curve_data[0:3]) - tf.logging.info('Finished calculating calibration for %s.' % - classname) - - def Summary(self, name): - """Generate tf summaries for calibration. - - Args: - name: str, name of summary. - - Returns: - list of tf.Summary - """ - summaries = [] - for class_id in self._metadata.EvalClassIndices(): - classname = self._metadata.ClassNames()[class_id] - tag_str = '{}/{}/calibration'.format(name, classname) - - if classname not in self._calibration_by_class: - continue - - # Extract the data. - mean_predicted_accuracy = self._calibration_by_class[classname][0, :] - mean_empirical_accuracy = self._calibration_by_class[classname][1, :] - num_examples_per_bin = self._calibration_by_class[classname][-1, :] - total_examples = np.sum(num_examples_per_bin) - legend = ['%s (%d)' % (classname, total_examples)] - - def _CalibrationSetter(fig, axes): - """Configure the plot for calibration.""" - ticks = np.arange(0, 1.05, 0.1) - axes.grid(b=False) - axes.set_xlabel('Predicted accuracy') - axes.set_xticks(ticks) - axes.set_ylabel('Empirical accuracy') - axes.set_yticks(ticks) - axes.legend(legend, numpoints=1) # pylint: disable=cell-var-from-loop - fig.tight_layout() - - calibration_curve_summary = plot.Curve( - name=tag_str, - figsize=(10, 8), - xs=mean_predicted_accuracy, - ys=mean_empirical_accuracy, - setter=_CalibrationSetter, - marker='.', - markersize=14, - linestyle='-', - linewidth=2, - alpha=0.5) - - ece = ExpectedCalibrationError(mean_predicted_accuracy, - mean_empirical_accuracy, - num_examples_per_bin) - - ece_summary = tf.Summary(value=[ - tf.Summary.Value( - tag='{}/{}/calibration_ece'.format(name, classname), - simple_value=ece) - ]) - summaries.extend([calibration_curve_summary, ece_summary]) - return summaries diff --git a/lingvo/tasks/car/calibration_processing_test.py b/lingvo/tasks/car/calibration_processing_test.py deleted file mode 100644 index cfe8ebce..00000000 --- a/lingvo/tasks/car/calibration_processing_test.py +++ /dev/null @@ -1,138 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for calibration_processing.""" - -from lingvo import compat as tf -from lingvo.core import test_utils -from lingvo.tasks.car import calibration_processing -from lingvo.tasks.car import kitti_metadata - -import numpy as np - - -class CalibrationProcessingTest(test_utils.TestCase): - - def testExpectedCalibrationError(self): - confidence = np.array([0.05, 0.15, 0.2, 0.3]) - accuracies = np.array([0.25, 0.15, 0.2, 0.3]) - num_examples = np.array([1000, 10, 10, 10]) - ece = calibration_processing.ExpectedCalibrationError( - confidence, accuracies, num_examples, min_confidence=0.1) - self.assertNear(ece, 0.0, 1e-4) - ece = calibration_processing.ExpectedCalibrationError( - confidence, accuracies, num_examples) - self.assertNear(ece, 0.1942, 1e-4) - - def testCalibrationCurveEqualScoresAndHits(self): - # Test calibration curve data when scores are equal to hits. - # This is an edge case where scores are either 0 or 1. - num_bins = 2 - scores = np.array([0, 1, 0]) - hits = scores - mean_predicted_accuracies, mean_empirical_accuracies, num_examples = \ - calibration_processing.CalibrationCurve(scores, hits, num_bins) - expected_mean_predicted_accuracies = np.array([0.0, 1.0]) - self.assertAllEqual(expected_mean_predicted_accuracies, - mean_predicted_accuracies) - expected_mean_empirical_accuracies = np.array([0, 1]) - self.assertAllEqual(expected_mean_empirical_accuracies, - mean_empirical_accuracies) - expected_num_examples = np.zeros(shape=num_bins) - expected_num_examples[0] = 2 - expected_num_examples[num_bins - 1] = 1 - self.assertAllEqual(expected_num_examples, num_examples) - ece = calibration_processing.ExpectedCalibrationError( - mean_predicted_accuracies, mean_empirical_accuracies, num_examples) - self.assertEqual(ece, 0.0) - - def testCalibrationCurvePerfectCalibration(self): - # Test calibration curve data when empirical accuracy corresponds to mean - # predicted accuracy. - num_bins = 2 - scores = np.array([0.25, 0.25, 0.25, 0.25, 0.75, 0.75, 0.75, 0.75]) - hits = np.array([1, 0, 0, 0, 1, 1, 1, 0]) - mean_predicted_accuracies, mean_empirical_accuracies, num_examples = \ - calibration_processing.CalibrationCurve(scores, hits, num_bins) - expected_mean_predicted_accuracies = np.array([0.25, 0.75]) - self.assertAllEqual(expected_mean_predicted_accuracies, - mean_predicted_accuracies) - expected_mean_empirical_accuracies = np.array([0.25, 0.75]) - self.assertAllEqual(expected_mean_empirical_accuracies, - mean_empirical_accuracies) - expected_num_examples = np.zeros(shape=num_bins) - expected_num_examples[0] = 4 - expected_num_examples[num_bins - 1] = 4 - self.assertAllEqual(expected_num_examples, num_examples) - ece = calibration_processing.ExpectedCalibrationError( - mean_predicted_accuracies, mean_empirical_accuracies, num_examples) - self.assertEqual(ece, 0.0) - - def testAllDataInOneBin(self): - # Test calibration curve data when all data is in one bin. - num_bins = 2 - scores = np.array([1, 1, 1]) - hits = np.array([0, 0, 0]) - mean_predicted_accuracies, mean_empirical_accuracies, num_examples = \ - calibration_processing.CalibrationCurve(scores, hits, num_bins) - expected_mean_predicted_accuracies = np.array([0.25, 1.0]) - self.assertAllEqual(expected_mean_predicted_accuracies, - mean_predicted_accuracies) - expected_mean_empirical_accuracies = np.array([0, 0]) - self.assertAllEqual(expected_mean_empirical_accuracies, - mean_empirical_accuracies) - expected_num_examples = np.zeros(shape=num_bins) - expected_num_examples[0] = 0 - expected_num_examples[num_bins - 1] = 3 - self.assertAllEqual(expected_num_examples, num_examples) - ece = calibration_processing.ExpectedCalibrationError( - mean_predicted_accuracies, mean_empirical_accuracies, num_examples) - self.assertEqual(ece, 1.0) - - def testEmptyBins(self): - # Test calibration curve data when there are no examples. - num_bins = 2 - scores = np.array([]) - hits = np.array([]) - mean_predicted_accuracies, mean_empirical_accuracies, num_examples = \ - calibration_processing.CalibrationCurve(scores, hits, num_bins) - expected_mean_predicted_accuracies = np.array([0.25, 0.75]) - self.assertAllEqual(expected_mean_predicted_accuracies, - mean_predicted_accuracies) - expected_mean_empirical_accuracies = np.array([0, 0]) - self.assertAllEqual(expected_mean_empirical_accuracies, - mean_empirical_accuracies) - expected_num_examples = np.zeros(shape=num_bins) - self.assertAllEqual(expected_num_examples, num_examples) - ece = calibration_processing.ExpectedCalibrationError( - mean_predicted_accuracies, mean_empirical_accuracies, num_examples) - self.assertEqual(ece, 0.0) - - def testCalibrationCalculator(self): - # End to end test for the calibration calculator. - metadata = kitti_metadata.KITTIMetadata() - calculator = calibration_processing.CalibrationCalculator(metadata) - scores_and_hits = np.array([[0.3, 1], [0.5, 1], [0.7, 1]]) - metrics = {} - metrics['calibrations'] = [{'calibrations': scores_and_hits}] - calculator.Calculate(metrics) - summaries = calculator.Summary('Test') - self.assertEqual(len(summaries), 2) - ece_summary = summaries[1] - self.assertEqual(0.5, ece_summary.value[0].simple_value) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/car_layers.py b/lingvo/tasks/car/car_layers.py deleted file mode 100644 index fa41f624..00000000 --- a/lingvo/tasks/car/car_layers.py +++ /dev/null @@ -1,110 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Common layers for car models. - -These are usually sub-classes of base_layer.BaseLayer used by builder_lib. -""" - -from lingvo import compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils -from lingvo.tasks.car import car_lib - - -class SamplingAndGroupingLayer(base_layer.BaseLayer): - """Sampling and Grouping layer (based on PointNet++).""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'num_samples', 0, - 'Number of points to be sampled. Each sampled point will be ' - 'returned with its corresponding group of neighbors.') - p.Define('ball_radius', 0, - 'The distance around each sampled point to obtain neighbours.') - p.Define('group_size', 0, 'Number neighbours for each sampled point.') - p.Define( - 'sample_neighbors_uniformly', True, - 'Whether to sample neighbors uniformly within the ball radius. ' - 'If False, this will pick the nearest neighbors by distance.') - return p - - def FProp(self, theta, input_data): - """Apply projection to inputs. - - Args: - theta: A NestedMap object containing weights' values of this layer and its - children layers. - input_data: A NestedMap object containing 'points', 'features', 'padding' - Tensors, all of type tf.float32. - 'points': Shape [N, P1, 3] - 'features': Shape [N, P1, F] - 'padding': Shape [N, P1] where 0 indicates real, 1 indicates padded. - - Returns: - A NestedMap consisting of the following two NestedMaps, - grouped_points: consists of the grouped points, features and padding. - query_points: consists of the sampled points and padding. - """ - - p = self.params - features = input_data.features - n, p1, c = py_utils.GetShape(features) - points = py_utils.HasShape(input_data.points, [n, p1, 3]) - padding = py_utils.HasShape(input_data.padding, [n, p1]) - - # Sampling - sampled_idx, _ = car_lib.FarthestPointSampler( - points, padding, num_sampled_points=p.num_samples) - query_points = car_lib.MatmulGather(points, tf.expand_dims(sampled_idx, -1)) - query_points = tf.squeeze(query_points, -2) - - # Grouping - grouped_idx, grouped_padding = car_lib.NeighborhoodIndices( - points, - query_points, - p.group_size, - points_padding=padding, - max_distance=p.ball_radius, - sample_neighbors_uniformly=p.sample_neighbors_uniformly) - grouped_points = car_lib.MatmulGather(points, grouped_idx) - # Normalize the grouped points based on the location of the query point. - grouped_points -= tf.expand_dims(query_points, -2) - grouped_features = car_lib.MatmulGather(features, grouped_idx) - - # Get the padding for the query points. - query_padding = tf.array_ops.batch_gather(padding, sampled_idx) - - # Verify the shapes of output tensors. - query_points = py_utils.HasShape(query_points, [n, p.num_samples, 3]) - query_padding = py_utils.HasShape(query_padding, [n, p.num_samples]) - grouped_features = py_utils.HasShape(grouped_features, - [n, p.num_samples, p.group_size, c]) - grouped_padding = py_utils.HasShape(grouped_padding, - [n, p.num_samples, p.group_size]) - - output_grouped_points = py_utils.NestedMap( - points=grouped_points, - features=grouped_features, - padding=grouped_padding) - output_query = py_utils.NestedMap( - points=query_points, padding=query_padding) - output_map = py_utils.NestedMap({ - 'grouped_points': output_grouped_points, - 'query_points': output_query - }) - return output_map diff --git a/lingvo/tasks/car/car_layers_test.py b/lingvo/tasks/car/car_layers_test.py deleted file mode 100644 index 0a065031..00000000 --- a/lingvo/tasks/car/car_layers_test.py +++ /dev/null @@ -1,82 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for car_layers.""" - -from lingvo import compat as tf -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.tasks.car import car_layers - - -class CarLayersTest(test_utils.TestCase): - - def _testNestedOutShape(self, p, input_shape, expected_shape): - batch_size, num_points, _ = input_shape - g = tf.Graph() - with g.as_default(): - net = p.Instantiate() - input_data = py_utils.NestedMap( - points=tf.random.uniform(input_shape[:-1] + (3,)), - features=tf.random.uniform(input_shape), - padding=tf.zeros((batch_size, num_points), dtype=tf.float32), - label=tf.random.uniform((batch_size,), - minval=0, - maxval=16, - dtype=tf.int32)) - result = net.FPropDefaultTheta(input_data) - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - np_result = self.evaluate(result) - grouped_points_result = np_result.grouped_points - self.assertEqual(grouped_points_result.features.shape, - expected_shape.grouped_points.features) - self.assertEqual(grouped_points_result.points.shape, - expected_shape.grouped_points.points) - self.assertEqual(grouped_points_result.padding.shape, - expected_shape.grouped_points.padding) - - query_points_result = np_result.query_points - self.assertEqual(query_points_result.points.shape, - expected_shape.query_points.points) - self.assertEqual(query_points_result.padding.shape, - expected_shape.query_points.padding) - - def testSamplingAndGrouping(self): - for num_points in [1024, 256]: - for input_dims in [3, 6, 9]: - for group_size in [32, 64]: - p = car_layers.SamplingAndGroupingLayer.Params().Set( - name='SampleGroupTest', - num_samples=256, - ball_radius=0.2, - group_size=group_size, - sample_neighbors_uniformly=True) - grouped_points_shape = py_utils.NestedMap( - features=(8, 256, group_size, input_dims), - points=(8, 256, group_size, 3), - padding=(8, 256, group_size)) - query_points_shape = py_utils.NestedMap( - points=(8, 256, 3), padding=(8, 256)) - expected_shape = py_utils.NestedMap({ - 'grouped_points': grouped_points_shape, - 'query_points': query_points_shape - }) - self._testNestedOutShape(p, (8, num_points, input_dims), - expected_shape) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/car_lib.py b/lingvo/tasks/car/car_lib.py deleted file mode 100644 index 49d289b6..00000000 --- a/lingvo/tasks/car/car_lib.py +++ /dev/null @@ -1,596 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Library of functions on tensors for car layers, builders, and models.""" - - -# pylint:enable=g-direct-tensorflow-import -import lingvo.compat as tf -from lingvo.core import py_utils - - -def SquaredDistanceMatrix(pa, pb, mem_optimized=False): - """Compute pair-wise squared distances. - - Expanded version (faster but potentially numerically unstable): - distance = pa^2 - 2*pa*pb + pb^2 - - Non-expanded version (slow but numerically stable): - distance = (pa - pb)^2 - - Args: - pa: tensor of shape [N, P1, dims] - pb: tensor of shape [N, P2, dims] - mem_optimized: Whether to use the memory-optimized expanded formulation. - Defaults to False. If enabled, the expanded version is used that - may have numerical issues. - - Returns: - tensor of shape [N, P1, P2] - """ - - def _ExpandedSquaredDistanceMatrix(pa, pb): - squared_pa = tf.reduce_sum(tf.square(pa), axis=2, keepdims=True) - squared_pb = tf.transpose( - tf.reduce_sum(tf.square(pb), axis=2, keepdims=True), perm=[0, 2, 1]) - # We have observed that entries may < 0. when using the expanded version. - # The max operation guards that from being possible. - return tf.maximum( - squared_pa - 2 * tf.matmul(pa, pb, transpose_b=True) + squared_pb, 0.0) - - def _NonExpandedSquaredDistanceMatrix(pa, pb): - diff = tf.expand_dims(pa, axis=2) - tf.expand_dims(pb, axis=1) - squared_diff = tf.square(diff) - squared_dis = tf.reduce_sum(squared_diff, axis=3) - return squared_dis - - if mem_optimized: - return _ExpandedSquaredDistanceMatrix(pa, pb) - else: - return _NonExpandedSquaredDistanceMatrix(pa, pb) - - -def NeighborSquaredDistanceMatrix(points, neighbor_points): - """Compute the squared distance matrix between points and their neighbors. - - Args: - points: A float tf.Tensor of shape [N, P1, 3] with point positions. - neighbor_points: A float tf.Tensor fo shape [N, P1, K, 3] with neighbor - positions. - - Returns: - Squared distance matrix between points and their K nearest neighbors - as a float tf.Tensor of shape [N, P1, K]. - """ - points = py_utils.HasShape(points, [-1, -1, 3]) - n, p1 = py_utils.GetShape(points, 2) - neighbor_points = py_utils.HasShape(neighbor_points, [n, p1, -1, 3]) - _, _, k = py_utils.GetShape(neighbor_points, 3) - - sq_diff = tf.square(neighbor_points - tf.reshape(points, [n, p1, 1, 3])) - sq_dist = tf.reduce_sum(sq_diff, axis=3) - return py_utils.HasShape(sq_dist, [n, p1, k]) - - -def KnnIndices(points, query_points, k, valid_num=None, max_distance=None): - """k-nearest neighbors of query_points in points. - - The caller should ensure that points[i, :valid_num[i], :] are the non-padding - points. - - Padding is returned alongside indices. Non-padded points are guaranteed to - be unique (non-repeated) points from original non-padded points. - - Padded points arise due to either a lack of points (k exceeds valid_num) - or points are too far away (exceeds max distance). - - TODO(weihan,jngiam): For backwards compatibility with PointCNN, if there are - fewer than k points to select (possibly because of valid_num), the points - selected will first consist of those in the non-padded points, and - then those from the padded points. This assumes that the padded points are - duplications of the original points. PointCNN should be updated to respect - padding. - - The auxiliary input 'valid_num' marks the number of non-padding points in each - sample. This is needed because we randomly duplicated points to make the input - fix-sized, we want search for k-NN in non-padding points first otherwise the - result may degenerate to be k-duplications of the query point itself. - - Args: - points: tensor of shape [N, P1, dims]. - query_points: tensor of shape [N, P2, dims] - k: Integer. - valid_num: tensor of shape [N,] - max_distance: float representing the maximum distance that each neighbor can - be. If there are no points within the distance, then the closest point is - returned (regardless of distance). If this is set to None, then - max_distance is not used. - - Returns: - A pair of tensors: - - - indices: tensor of shape [N, P2, k]. - - padding: tensor of shape [N, P2 ,k] where 1 represents a padded point, and - 0 represents an unpadded (real) point. - - """ - p1 = tf.shape(points)[1] - padding = None - if valid_num is not None: - padding = tf.greater_equal(tf.range(p1), tf.expand_dims( - valid_num, -1)) # [N, P1], False/True padding - return NeighborhoodIndices(points, query_points, k, padding, max_distance) - - -def NeighborhoodIndices(points, - query_points, - k, - points_padding=None, - max_distance=None, - sample_neighbors_uniformly=False): - """Get indices to k-neighbors of query_points in points. - - Padding is returned along-side indices. Non-padded points are guaranteed to - be unique (non-repeated) points from original non-padded points. - - Padded points arise due to either a lack of points (k exceeds the number - of original non-padded points) or points are too far away (exceeds max - distance). - - Note: Padded point indices may refer to padded points from the original, or - may be duplicates of the closest point. - - TODO(weihan,jngiam): PointCNN implementation makes an assumption that padded - points are repeated points from the original points. This behavior is - maintained here, but we should update PointCNN to respect indices paddings. - - Args: - points: tensor of shape [N, P1, dims]. - query_points: tensor of shape [N, P2, dims] - k: Integer. - points_padding: optional tensor of shape [N, P1] containing True/1.0 iff the - point is a padded point. if None, then all points are considered real - points. - max_distance: float representing the maximum distance that each neighbor can - be. If there are no points within the distance, then the closest point is - returned (regardless of distance). If this is set to None, then no - filtering by distance is performed. - sample_neighbors_uniformly: boolean specifying whether to sample neighbors - uniformly if they are within max distance. - - Returns: - A pair of tensors: - - - indices: tensor of shape [N, P2, k]. - - padding: tensor of shape [N, P2, k] where 1 represents a padded point, and - 0 represents an unpadded (real) point. - - """ - n, p1 = py_utils.GetShape(points, 2) - query_points = py_utils.HasShape(query_points, [n, -1, -1]) - _, p2 = py_utils.GetShape(query_points, 2) - - # Compute pair-wise squared distances. - # Note that dist_mat contains the squared distance (without sqrt). Thus, when - # using max_distance, we will need to square max_distance to make sure it's - # in the same units. - dist_mat = SquaredDistanceMatrix(query_points, points) - dist_mat = py_utils.HasShape(dist_mat, [n, p2, p1]) - - # Add a large scalar to the distances for padded points. - # dist_mat[i, j, k] will be: - # if k < valid_num[i]: distance between points[i, k] and query_points[i, j] - # otherwise: a large scalar added to dist_mat[i, j, k] - if points_padding is not None: - points_padding = tf.cast(tf.expand_dims(points_padding, 1), tf.float32) - points_padding = py_utils.HasShape(points_padding, [n, 1, p1]) - large_scalar = tf.reduce_max(dist_mat) + 1 - dist_mat += points_padding * large_scalar - - # To perform sampling neighbors uniformly efficiently, we set all neighbors - # that are within the distance threshold to have distances be drawn uniformly - # at random. Using top_k with this enables selecting a random set quickly - # without replacement. - if sample_neighbors_uniformly: - if max_distance is not None: - mask_by_distance = tf.less_equal(dist_mat, max_distance**2) - dist_mat = tf.where( - mask_by_distance, - tf.square(max_distance) * tf.random.uniform(tf.shape(dist_mat)), - dist_mat) - else: - raise ValueError('Uniform sampling requires specifying max_distance.') - - top_k_dist, indices = tf.nn.top_k(-dist_mat, k=k, sorted=True) # N x P2 x K - - # Set padding using top_k_dist; padded points will have distance exceeding - # the large_scalar. - if points_padding is not None: - paddings = tf.greater_equal(-top_k_dist, large_scalar) - else: - paddings = tf.zeros_like(top_k_dist, dtype=tf.bool) - - # Filter by max_distances by setting all indices that exceed the max_distance - # to the closest point. - if max_distance is not None: - # Mask is true for points that are further than max_distance. - mask_by_distance = tf.greater(-top_k_dist, tf.square(max_distance)) - closest_idx = tf.tile(indices[:, :, :1], [1, 1, k]) - indices = tf.where(mask_by_distance, closest_idx, indices) - paddings |= mask_by_distance - - indices = tf.reshape(indices, [n, p2, k]) - paddings = tf.cast(paddings, tf.float32) - - return indices, paddings - - -def MatmulGather(source, indices): - """Drop in replacement for tf.gather_nd() optimized for speed on TPU. - - TODO(weihan): tf.gather_nd() is supposed to be implemented in the same way - on TPU. Investigate why it's much slower. - - Args: - source: tensor of shape [N, P1, C] - indices: tensor of shape [N, P2, K] - - Returns: - tensor of shape [N, P2, K, C] - """ - source = py_utils.HasRank(source, 3) - n, p1, c = py_utils.GetShape(source) - indices = py_utils.HasShape(indices, [n, -1, -1]) - _, p2, k = py_utils.GetShape(indices) - - onehot = tf.one_hot(indices, depth=p1) # N x P2 x K x P1 - reshaped = tf.reshape(onehot, [n, -1, p1]) # N x (P2 x K) x P1 - target = tf.matmul(reshaped, source) # N x (P2 x K) x C - return tf.reshape(target, [n, p2, k, c]) - - -def FarthestPointSampler(points, - padding, - num_sampled_points, - precomputed_squared_distance=None, - num_seeded_points=0, - random_seed=None): - """Samples num_sampled_points from points using farthest point sampling. - - Algorithm: - 1. Start by selecting a random point and adding to a selected set. - 2. For all remaining points, find the furthest point from those selected. - 3. Add furthest point to selected. - 4. Repeat 2-3 until num_sampled_points are selected. - - More details at https://en.wikipedia.org/wiki/Farthest-first_traversal - - This output of this function can be used with tf.array_ops.batch_gather to - extract the desired points, for example: - tf.array_ops.batch_gather(points, sampled_idx) - - Args: - points: floating point tf.Tensor of shape [N, P1, dims] - padding: A floating point tf.Tensor of shape [N, P1] with 0 if the point is - real, and 1 otherwise. - num_sampled_points: integer number of points to sample. - precomputed_squared_distance: optional tf.Tensor of shape [N, P1, P1] of - distances between each point. if None, distances will be computed on the - fly. - num_seeded_points: If num_seeded_points > 0, then the first - num_seeded_points in points are considered to be seeded in the FPS - sampling. Note that we assume that these points are *not* padded, and do - not check padding when seeding them. - random_seed: optional integer random seed to use with all the random ops. - - Returns: - A tuple of tf.Tensors (sampled_idx, closest_idx) of types - (tf.int32, tf.int32). - - sampled_idx is of shape [N, num_sampled_points] representing the indices - selected using the sampler. This will have range of [0, P1]. - - closest_idx is of shape [N, P1] representing the indices of the closest - sampled points for each input point. closest_idx is used in PCNN as part of - the pooling operation: each point is assigned to the closest sampled point - and a max is taken over them. This will have a range of [0, P2] with the - index of the closest sampled point that remains. - """ - points = py_utils.HasRank(points, 3) - batch_size, num_points, dims = py_utils.GetShape(points, 3) - - points = py_utils.with_dependencies( - [py_utils.assert_greater_equal(num_points, num_sampled_points)], points) - - # Add a tiny bit of noise to the distance matrix or points so all - # points are unique. This will also ensure true repeated points - # like padded points are only selected after all valid points are selected. - if precomputed_squared_distance is not None: - precomputed_squared_distance = py_utils.HasShape( - precomputed_squared_distance, [batch_size, num_points, num_points]) - precomputed_squared_distance += tf.random.uniform( - (batch_size, num_points, 1), - minval=1e-6, - maxval=1e-5, - dtype=tf.float32, - seed=random_seed) - else: - points += tf.random.uniform((batch_size, num_points, dims), - minval=1e-6, - maxval=1e-5, - dtype=tf.float32, - seed=random_seed) - - # TensorArray to store the sampled indices in the loop. - sampled_idx = tf.TensorArray(tf.int32, num_sampled_points) - - # Initialize distance_to_selected to inf for all points. - distance_to_selected = float('inf') * tf.ones((batch_size, num_points)) - - # For tracking the index to the closest selected point. - closest_idx = tf.zeros((batch_size, num_points), dtype=tf.int32) - - # Current loop index counter. - curr_idx = tf.constant(0, dtype=tf.int32) - - # Get number of valid points (1 is padded, so num_points - num_padded). - num_valid_points = tf.cast( - tf.cast(num_points, dtype=tf.float32) - tf.reduce_sum(padding, axis=1), - dtype=tf.int32) - - def _BodyFn(curr_idx, distance_to_selected, sampled_idx, closest_idx): - """Loop body for farthest point sampler.""" - - def _GetRandomRealPoint(): - """Select the first point. - - For the first point, we want any random real (non padded) point, so we - create a random values per point, and then set all padded ones to - some large value (more than the maxval). We then take the min per batch - element to get the first points. - - Returns: - Tensor containing the index of a random point selected for each example - in the batch. - """ - random_values = tf.random.uniform((batch_size, num_points), - minval=0, - maxval=1, - dtype=tf.float32, - seed=random_seed) - random_values = tf.where( - tf.equal(padding, 0.0), random_values, padding * 10) - return tf.argmin(random_values, axis=1, output_type=tf.int32) - - def _GetFurthestPoint(): - """Get point that is furthest from those already selected. - - We also bias the sampling towards real points by setting the distance - to padded points negative until we are out of real points. - - Returns: - Tensor containing the index of the next farthest point selected for each - example in the batch. - """ - # Set padded points distance to negative so they aren't selected. - padding_masked_distance_to_selected = tf.where( - tf.equal(padding, 0.0), distance_to_selected, -1.0 * tf.ones( - (batch_size, num_points), dtype=tf.float32)) - # But only do this when we still have valid points left. - padding_masked_distance_to_selected = tf.where( - tf.less(curr_idx, num_valid_points), - padding_masked_distance_to_selected, distance_to_selected) - return tf.argmax( - padding_masked_distance_to_selected, axis=-1, output_type=tf.int32) - - def _GetSeededPoint(): - """Select a seeded point. - - Seeded points are assumed to be at the beginning of the original points. - - Returns: - Tensor containing the index of the next seeded point to select for each - example in the batch. - """ - return tf.ones((batch_size,), dtype=tf.int32) * curr_idx - - # Select indices for this loop iteration. - def _Seeded(): - return tf.cond( - tf.less(curr_idx, num_seeded_points), _GetSeededPoint, - _GetFurthestPoint) - - def _Real(): - return tf.cond( - tf.equal(curr_idx, 0), _GetRandomRealPoint, _GetFurthestPoint) - - new_selected = tf.cond(tf.greater(num_seeded_points, 0), _Seeded, _Real) - sampled_idx = sampled_idx.write(curr_idx, new_selected) - - # Extract the distance to the latest point selected to update - # distance_to_selected. - new_selected_gather_idx = tf.stack([tf.range(batch_size), new_selected], - axis=1) - if precomputed_squared_distance is not None: - new_distance = tf.gather_nd(precomputed_squared_distance, - new_selected_gather_idx) - else: - new_points = tf.reshape( - tf.gather_nd(points, new_selected_gather_idx), [batch_size, 1, dims]) - new_distance = tf.reshape( - SquaredDistanceMatrix(points, new_points), [batch_size, num_points]) - - is_newly_closest = tf.less(new_distance, distance_to_selected) - distance_to_selected = tf.minimum(distance_to_selected, new_distance) - - # Track the index to the closest selected point. - new_selected_tiled = tf.tile([[curr_idx]], [batch_size, num_points]) - closest_idx = tf.cond( - tf.equal(curr_idx, 0), - # At the first loop iteration, the init points are the closest. - lambda: new_selected_tiled, - # Otherwise, update with the new points based on the distances. - lambda: tf.where(is_newly_closest, new_selected_tiled, closest_idx)) - return curr_idx + 1, distance_to_selected, sampled_idx, closest_idx - - _, _, sampled_idx, closest_idx = tf.while_loop( - lambda curr_idx, *args: tf.less(curr_idx, num_sampled_points), - _BodyFn, - loop_vars=(curr_idx, distance_to_selected, sampled_idx, closest_idx), - back_prop=False, - maximum_iterations=num_sampled_points) - - sampled_idx = sampled_idx.stack() # num_sampled_points x n - sampled_idx = tf.transpose(sampled_idx, [1, 0]) - - if isinstance(batch_size, int) and isinstance(num_sampled_points, int): - sampled_idx.set_shape((batch_size, num_sampled_points)) - - return sampled_idx, closest_idx - - -# TODO(bencaine): This was moved so that we can make this more generic in the -# future and provide min/avg/max pooling with one function. -def MaxPool3D(points, point_features, pooling_idx, closest_idx): - """Apply max pooling to a point cloud with computed sampling indices. - - sampled_idx and closest_idx are the outputs of a sampler such as - FurthestPointSampler. - - The pooling operation results in a point cloud with fewer points, where the - pooled points are specified by pooling_idx. Each element of pooling_idx - contains an integer in the range [0, P1) containing the index of the point in - points/points_features. - - Max pooling is performed by assigning each point to its closest pooled point, - and then taking a max over the features of points assigned. We assume that - this mapping is provided by closest_idx, where each element should contain - an integer in the range [0, P2) containing the index of the pooled point that - each point is assigned to. - - Note: This logic for pooling assumes that there will be at least - one value > 0 per sampled region for each feature, otherwise it will return 0. - Additionally, it does a reduce over a masked version of the features, so - mean and min would not work without a change in the logic. - - Args: - points: a floating point tf.Tensor with shape [N, P1, 3] - point_features: a floating point tf.Tensor with shape [N, P1, C] - pooling_idx: A tf.int32 tf.Tensor of shape [N, P2] with the index of which - points we want to keep. Each value should be in the range [0, P1]. - closest_idx: A tf.int32 tf.Tensor of shape [N, P1] representing which - sampled point is closest to each original point. Each value should be in - the range of [0, P2]. - - Returns: - A tuple of tf.Tensors (pooled_points, pooled_features). - - pooled_points has shape [N, P2, 3] representing the locations of each - selected point. P2 corresponds to num_pooled_points. - - pooled_features has shape [N, P2, C] representing the pooled features at - each point. - """ - batch_size, num_points = py_utils.GetShape(points, 2) - point_features = py_utils.HasShape(point_features, - [batch_size, num_points, -1]) - pooling_idx = py_utils.HasShape(pooling_idx, [batch_size, -1]) - _, num_output_points = py_utils.GetShape(pooling_idx) - _, _, feature_dims = py_utils.GetShape(point_features, 3) - - # Gather new point locations. - pooled_points = tf.array_ops.batch_gather(points, pooling_idx) - - mask = tf.one_hot(closest_idx, num_output_points) # [N, P1, P2] - mask = tf.transpose(mask, [2, 0, 1]) # [P2, N, P1] - - def _PartialPoolFeaturesFn(partial_mask): - partial_mask = tf.tile( - tf.reshape(partial_mask, [batch_size, num_points, 1]), - [1, 1, feature_dims]) - # Note: This method of pooling assumes there will be a value > 0 - # And will only work with max under this condition. - return tf.reduce_max(partial_mask * point_features, axis=1) - - # Performing a map_fn over the pooled points is more memory efficient. - pooled_point_features = tf.map_fn(_PartialPoolFeaturesFn, mask) # [P2, N, P1] - pooled_point_features = tf.transpose(pooled_point_features, [1, 0, 2]) - - return pooled_points, pooled_point_features - - -def SegmentPool3D(points, - point_features, - pooling_idx, - closest_idx, - pooling_method='max'): - """Performs {min/max/average} pooling over a pointcloud given indices. - - This should be functionally identical when using max to the above - MaxPool3D function, except it turns out to be much more memory efficient - on a TPU, and supports min/max/mean. - - Args: - points: A float tf.Tensor of shape [N, P1, 3] with point locations. - point_features: A float tf.Tensor of shape [N, P1, C] with point features. - pooling_idx: A tf.int32 tf.Tensor of shape [N, P2] with the index of which - points we want to keep. Each value should be in the range [0, P1]. - closest_idx: A tf.int32 tf.Tensor of shape [N, P1] representing which - sampled point is closest to each original point. Each value should be in - the range of [0, P2]. - pooling_method: A string for which pooling function to use. Should be one of - {'min', 'max', 'mean'}. - - Returns: - pooled_points: A float tf.Tensor of shape [N, P2, 3] with the pooled - point locations. - pooled_features: A float tf.Tensor of shape [N, P2, C] with the pooled - features. - Raises: - ValueError: If pooling_method is not one of {min/max/mean}. - """ - segment_pooling_functions = { - 'min': tf.math.unsorted_segment_min, - 'max': tf.math.unsorted_segment_max, - 'mean': tf.math.unsorted_segment_mean - } - - if pooling_method not in segment_pooling_functions: - raise ValueError('`pooling_method` must be one of {}.'.format( - list(segment_pooling_functions.keys()))) - segment_fn = segment_pooling_functions[pooling_method] - - points = py_utils.HasShape(points, [-1, -1, 3]) - n, p1 = py_utils.GetShape(points, 2) - point_features = py_utils.HasShape(point_features, [n, p1, -1]) - _, _, c = py_utils.GetShape(point_features) - pooling_idx = py_utils.HasShape(pooling_idx, [n, -1]) - _, p2 = py_utils.GetShape(pooling_idx) - closest_idx = py_utils.HasShape(closest_idx, [n, p1]) - - # Subselect our output points - pooled_points = tf.array_ops.batch_gather(points, pooling_idx) - - # Loop over batch dimension of our features/indices, as unsorted_segment_X - # does not currently support a batch dimension. - def _LoopFn(args): - example_features, example_closest_idx = args - return segment_fn(example_features, example_closest_idx, num_segments=p2) - - pooled_features = tf.map_fn( - fn=_LoopFn, elems=(point_features, closest_idx), dtype=tf.float32) - - return (py_utils.HasShape(pooled_points, [n, p2, 3]), - py_utils.HasShape(pooled_features, [n, p2, c])) diff --git a/lingvo/tasks/car/car_lib_test.py b/lingvo/tasks/car/car_lib_test.py deleted file mode 100644 index be823c0e..00000000 --- a/lingvo/tasks/car/car_lib_test.py +++ /dev/null @@ -1,466 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for car_lib.""" - -import lingvo.compat as tf -from lingvo.core import test_utils -from lingvo.tasks.car import car_lib -import numpy as np - - -class CarLibTest(test_utils.TestCase): - - def np_sq_dis(self, pa, pb): - sq_dis = np.zeros((pa.shape[0], pa.shape[1], pb.shape[1])) - for n in range(pa.shape[0]): - for i in range(pa.shape[1]): - for j in range(pb.shape[1]): - sq_dis[n, i, j] = np.sum(np.square(pa[n, i] - pb[n, j])) - return sq_dis - - def _np_sq_dis_neighbors(self, points, neighbor_idx): - """Run the np square distance function and gather based on neighbor.""" - n, p1, k = neighbor_idx.shape - sq_dist_all = self.np_sq_dis(points, points) - sq_dist_neighbors = np.zeros((n, p1, k), dtype=np.float32) - for b_i in range(n): - for p_i in range(p1): - for k_i in range(k): - sq_dist_neighbors[b_i, p_i, k_i] = sq_dist_all[ - b_i, p_i, neighbor_idx[b_i, p_i, k_i]] - return sq_dist_neighbors - - def testSquaredDistance(self): - g = tf.Graph() - with g.as_default(): - pa_tensor = tf.placeholder(tf.float32, shape=[None, None, 4]) - pb_tensor = tf.placeholder(tf.float32, shape=[None, None, 4]) - sq_dis_tensor = car_lib.SquaredDistanceMatrix( - pa_tensor, pb_tensor, mem_optimized=False) - sq_dis_tensor_mem_optimized = car_lib.SquaredDistanceMatrix( - pa_tensor, pb_tensor, mem_optimized=True) - - num_trials = 3 - for _ in range(num_trials): - rand_pa = np.random.randn(10, 5, 4) - rand_pb = np.random.randn(10, 6, 4) - np_sq_dis = self.np_sq_dis(rand_pa, rand_pb) - - with self.session(graph=g) as sess: - sq_dis, sq_dis_mem_optimized = sess.run( - [sq_dis_tensor, sq_dis_tensor_mem_optimized], - feed_dict={ - pa_tensor: rand_pa, - pb_tensor: rand_pb, - }) - self.assertAllClose(sq_dis, np_sq_dis) - self.assertAllClose(sq_dis_mem_optimized, np_sq_dis) - - def testNeighborSquaredDistance(self): - n, p1, k = 2, 10, 3 - points = tf.random.uniform((n, p1, 3)) - neighbor_idx = tf.random.uniform((n, p1, k), - minval=0, - maxval=p1, - dtype=tf.int32) - neighbor_points = car_lib.MatmulGather(points, neighbor_idx) - - sq_dist_result = car_lib.NeighborSquaredDistanceMatrix( - points, neighbor_points) - - with self.session(): - [np_points, np_neighbor_idx, np_sq_dist_result - ] = self.evaluate([points, neighbor_idx, sq_dist_result]) - np_sq_dist_expected = self._np_sq_dis_neighbors(np_points, - np_neighbor_idx) - self.assertAllClose(np_sq_dist_result, np_sq_dist_expected) - - def testKnnIndicesMaxDistance(self): - points = tf.constant([[[1, 1, 1], [2, 2, 2], [4, 4, 4], [5, 5, 5]]], - dtype=tf.float32) - query_points = tf.constant([[ - [1, 1, 1], - [4, 4, 4], - ]], dtype=tf.float32) - valid_num = tf.constant([4], dtype=tf.int32) - - # Max distance so that one neighbor can be selected. - max_distance = 2.0 - expected_1nn = np.array([[[0], [2]]], dtype=np.int32) - expected_1nn_padding = np.array([[[0], [0]]], dtype=np.float32) - expected_2nn = np.array([[[0, 1], [2, 3]]], dtype=np.int32) - expected_2nn_padding = np.array([[[0, 0], [0, 0]]], dtype=np.float32) - expected_3nn = np.array([[[0, 1, 0], [2, 3, 2]]], dtype=np.int32) - expected_3nn_padding = np.array([[[0, 0, 1], [0, 0, 1]]], dtype=np.float32) - with self.session(): - output_1nn, output_1nn_padding = self.evaluate( - car_lib.KnnIndices(points, query_points, 1, valid_num, max_distance)) - self.assertAllEqual(output_1nn, expected_1nn) - self.assertAllEqual(output_1nn_padding, expected_1nn_padding) - - output_2nn, output_2nn_padding = self.evaluate( - car_lib.KnnIndices(points, query_points, 2, valid_num, max_distance)) - self.assertAllEqual(output_2nn, expected_2nn) - self.assertAllEqual(output_2nn_padding, expected_2nn_padding) - - output_3nn, output_3nn_padding = self.evaluate( - car_lib.KnnIndices(points, query_points, 3, valid_num, max_distance)) - self.assertAllEqual(output_3nn, expected_3nn) - self.assertAllEqual(output_3nn_padding, expected_3nn_padding) - - # Max distance so that only itself can be selected. - max_distance = 0.1 - expected_1nn = np.array([[[0], [2]]], dtype=np.int32) - expected_1nn_padding = np.array([[[0], [0]]], dtype=np.float32) - expected_2nn = np.array([[[0, 0], [2, 2]]], dtype=np.int32) - expected_2nn_padding = np.array([[[0, 1], [0, 1]]], dtype=np.float32) - expected_3nn = np.array([[[0, 0, 0], [2, 2, 2]]], dtype=np.int32) - expected_3nn_padding = np.array([[[0, 1, 1], [0, 1, 1]]], dtype=np.float32) - with self.session(): - output_1nn, output_1nn_padding = self.evaluate( - car_lib.KnnIndices(points, query_points, 1, valid_num, max_distance)) - self.assertAllEqual(output_1nn, expected_1nn) - self.assertAllEqual(output_1nn_padding, expected_1nn_padding) - - output_2nn, output_2nn_padding = self.evaluate( - car_lib.KnnIndices(points, query_points, 2, valid_num, max_distance)) - self.assertAllEqual(output_2nn, expected_2nn) - self.assertAllEqual(output_2nn_padding, expected_2nn_padding) - - output_3nn, output_3nn_padding = self.evaluate( - car_lib.KnnIndices(points, query_points, 3, valid_num, max_distance)) - self.assertAllEqual(output_3nn, expected_3nn) - self.assertAllEqual(output_3nn_padding, expected_3nn_padding) - - def testKnnIndices(self): - points = tf.constant([[[1, 1], [2, 2], [4, 4], [5, 5]]], dtype=tf.float32) - query_points = tf.constant([[ - [1, 1], - [4, 4], - ]], dtype=tf.float32) - - # Case 1: when all points are valid. - valid_num = tf.constant([4], dtype=tf.int32) - expected_1nn = np.array([[[0], [2]]], dtype=np.int32) - expected_2nn = np.array([[[0, 1], [2, 3]]], dtype=np.int32) - expected_3nn = np.array([[[0, 1, 2], [2, 3, 1]]], dtype=np.int32) - with self.session(): - output_1nn, _ = self.evaluate( - car_lib.KnnIndices(points, query_points, 1, valid_num)) - self.assertAllEqual(output_1nn, expected_1nn) - - output_2nn, _ = self.evaluate( - car_lib.KnnIndices(points, query_points, 2, valid_num)) - self.assertAllEqual(output_2nn, expected_2nn) - - output_3nn, _ = self.evaluate( - car_lib.KnnIndices(points, query_points, 3, valid_num)) - self.assertAllEqual(output_3nn, expected_3nn) - - # Case 2: not all points are valid. - valid_num = tf.constant([2], dtype=tf.int32) - expected_1nn = np.array([[[0], [1]]], dtype=np.int32) - expected_1nn_padding = np.array([[[0], [0]]], dtype=np.float32) - expected_2nn = np.array([[[0, 1], [1, 0]]], dtype=np.int32) - expected_2nn_padding = np.array([[[0, 0], [0, 0]]], dtype=np.float32) - expected_3nn = np.array([[[0, 1, 2], [1, 0, 2]]], dtype=np.int32) - expected_3nn_padding = np.array([[[0, 0, 1], [0, 0, 1]]], dtype=np.float32) - - with self.session(): - output_1nn, output_1nn_padding = self.evaluate( - car_lib.KnnIndices(points, query_points, 1, valid_num)) - self.assertAllEqual(output_1nn, expected_1nn) - self.assertAllEqual(output_1nn_padding, expected_1nn_padding) - - output_2nn, output_2nn_padding = self.evaluate( - car_lib.KnnIndices(points, query_points, 2, valid_num)) - self.assertAllEqual(output_2nn, expected_2nn) - self.assertAllEqual(output_2nn_padding, expected_2nn_padding) - - output_3nn, output_3nn_padding = self.evaluate( - car_lib.KnnIndices(points, query_points, 3, valid_num)) - self.assertAllEqual(output_3nn, expected_3nn) - self.assertAllEqual(output_3nn_padding, expected_3nn_padding) - - # Case 3: explicit padding. - padding = tf.constant([[1, 0, 1, 0]]) - expected_1nn = np.array([[[1], [3]]], dtype=np.int32) - expected_1nn_padding = np.array([[[0], [0]]], dtype=np.float32) - expected_2nn = np.array([[[1, 3], [3, 1]]], dtype=np.int32) - expected_2nn_padding = np.array([[[0, 0], [0, 0]]], dtype=np.float32) - expected_3nn = np.array([[[1, 3, 0], [3, 1, 2]]], dtype=np.int32) - expected_3nn_padding = np.array([[[0, 0, 1], [0, 0, 1]]], dtype=np.float32) - with self.session(): - output_1nn, output_1nn_padding = self.evaluate( - car_lib.NeighborhoodIndices(points, query_points, 1, padding)) - self.assertAllEqual(output_1nn, expected_1nn) - self.assertAllEqual(output_1nn_padding, expected_1nn_padding) - - output_2nn, output_2nn_padding = self.evaluate( - car_lib.NeighborhoodIndices(points, query_points, 2, padding)) - self.assertAllEqual(output_2nn, expected_2nn) - self.assertAllEqual(output_2nn_padding, expected_2nn_padding) - - output_3nn, output_3nn_padding = self.evaluate( - car_lib.NeighborhoodIndices(points, query_points, 3, padding)) - self.assertAllEqual(output_3nn, expected_3nn) - self.assertAllEqual(output_3nn_padding, expected_3nn_padding) - - def testNeighborhoodIndicesWithUniformSampling(self): - points = tf.constant([[[1, 1], [2, 2], [4, 4], [5, 5]]], dtype=tf.float32) - query_points = tf.constant([[ - [2, 2], - [5, 5], - ]], dtype=tf.float32) - - padding = tf.constant([[1, 0, 1, 0]]) - - # With max_distance=1.1, only the nearest point will be returned (and - # repeated). - expected_3nn = np.array([[[1, 1, 1], [3, 3, 3]]], dtype=np.int32) - expected_paddings = np.array([[[0, 1, 1], [0, 1, 1]]], dtype=np.float32) - with self.session(): - output_3nn, paddings = self.evaluate( - car_lib.NeighborhoodIndices( - points, - query_points, - 3, - padding, - max_distance=1.1, - sample_neighbors_uniformly=True)) - self.assertAllEqual(output_3nn, expected_3nn) - self.assertAllEqual(paddings, expected_paddings) - - def testNeighborhoodIndicesWithUniformSamplingRaisesIfNoMaxDistance(self): - points = tf.constant([[[1, 1], [2, 2], [4, 4], [5, 5]]], dtype=tf.float32) - query_points = tf.constant([[ - [1, 1], - [4, 4], - ]], dtype=tf.float32) - padding = tf.constant([[1, 0, 1, 0]]) - with self.assertRaisesRegex( - ValueError, r'.*Uniform sampling requires specifying max_distance.*'): - car_lib.NeighborhoodIndices( - points, query_points, 1, padding, sample_neighbors_uniformly=True) - - def testFarthestPointSamplerOnePoint(self): - points = tf.constant([ - [[1, 1, 1, 1]], - [[2, 2, 2, 2]], - ], dtype=tf.float32) - padding = tf.zeros((2, 1), dtype=tf.float32) - selected_idx, _ = car_lib.FarthestPointSampler(points, padding, 1) - with self.session(): - selected_idx = self.evaluate(selected_idx) - self.assertAllEqual(selected_idx, [[0], [0]]) - - def testFarthestPointSamplerInsufficientPoints(self): - points = tf.constant([ - [[0, 1, 1]], - [[2, 2, 2]], - ], dtype=tf.float32) - padding = tf.zeros((2, 1), dtype=tf.float32) - with self.session(): - with self.assertRaisesRegex(tf.errors.InvalidArgumentError, - r'.*Condition x >= y did not hold.*'): - sampled_idx, closest_idx = car_lib.FarthestPointSampler( - points, padding, 2) - self.evaluate((sampled_idx, closest_idx)) - - def testFarthestPointSamplerSelectMinMax(self): - points = tf.constant([ - [[0, 1, 1], [1, 1, 1], [2, 1, 1], [3, 1, 1], [4, 1, 1], [5, 1, 1]], - [[0, 2, 1], [1, 2, 1], [2, 2, 1], [3, 2, 1], [4, 2, 1], [5, 2, 1]], - [[0, 2, 3], [1, 2, 3], [2, 2, 3], [3, 2, 3], [4, 2, 3], [5, 2, 3]], - [[0, 2, 1], [1, 2, 1], [2, 2, 1], [3, 2, 1], [4, 2, 1], [5, 2, 1]], - ], dtype=tf.float32) # pyformat: disable - padding = tf.zeros((4, 6), dtype=tf.float32) - selected_idx, closest_idx = car_lib.FarthestPointSampler(points, padding, 2) - with self.session(): - selected_idx, closest_idx = self.evaluate([selected_idx, closest_idx]) - for batch_idx in range(4): - self.assertIn( - selected_idx[batch_idx, 1], [0, 5], - msg=('The second selected point must be one of the end ' - 'points corresponding to index 0 or 5.')) - - # Closest indices should either be 0 or 1 since we picked 2 points. - self.assertTrue( - np.all((closest_idx >= 0) & (closest_idx < 2)), - msg='Closest index must be among selected indices.') - - def testFarthestPointSamplerSeeded(self): - points = tf.constant([ - [[0, 1, 1], [1, 1, 1], [2, 1, 1], [3, 1, 1], [4, 1, 1], [5, 1, 1]], - [[0, 2, 1], [1, 2, 1], [2, 2, 1], [3, 2, 1], [4, 2, 1], [5, 2, 1]], - [[0, 2, 3], [1, 2, 3], [2, 2, 3], [3, 2, 3], [4, 2, 3], [5, 2, 3]], - [[0, 2, 1], [1, 2, 1], [2, 2, 1], [3, 2, 1], [4, 2, 1], [5, 2, 1]], - ], dtype=tf.float32) # pyformat: disable - padding = tf.zeros((4, 6), dtype=tf.float32) - selected_idx, closest_idx = car_lib.FarthestPointSampler( - points, padding, 3, num_seeded_points=2) - with self.session(): - selected_idx, closest_idx = self.evaluate([selected_idx, closest_idx]) - # First two selected points are seeded. - self.assertTrue(np.all(selected_idx[:, 0] == 0)) - self.assertTrue(np.all(selected_idx[:, 1] == 1)) - # Third point is the last point since it's farthest. - self.assertTrue(np.all(selected_idx[:, 2] == 5)) - # Closest indices should either be 0, 1 or 2 since we picked 3 points. - self.assertTrue( - np.all((closest_idx >= 0) & (closest_idx < 3)), - msg='Closest index must be among selected indices.') - - def testFarthestPointSamplerAllPoints(self): - points = tf.constant([ - [[0, 1, 1], [1, 1, 1], [2, 1, 1], [3, 1, 1], [4, 1, 1], [5, 1, 1]], - [[0, 2, 1], [1, 2, 1], [2, 2, 1], [3, 2, 1], [4, 2, 1], [5, 2, 1]], - [[0, 2, 3], [1, 2, 3], [2, 2, 3], [3, 2, 3], [4, 2, 3], [5, 2, 3]], - [[0, 2, 1], [1, 2, 1], [2, 2, 1], [3, 2, 1], [4, 2, 1], [5, 2, 1]], - ], dtype=tf.float32) # pyformat: disable - padding = tf.zeros((4, 6), dtype=tf.float32) - sampled_idx, closest_idx = car_lib.FarthestPointSampler(points, padding, 6) - with self.session(): - sampled_idx, closest_idx = self.evaluate([sampled_idx, closest_idx]) - for batch_n in range(4): - self.assertSetEqual( - set(sampled_idx[batch_n, :]), - set(np.arange(6)), - msg='All points should be selected.') - for point_idx in range(6): - # For each selected point in sampled_idx, we verify that the - # closest_idx assigned to that point matches itself. This is done by - # finding the location of the point in sampled_idx (which is shuffled - # during sampling). The value of the point in closest_idx should match - # the index assigned to the point in sampled_idx. - expected_closest_idx = None - # location refers to the index where the point appears in sampled_idx. - # This should be what closest_idx refers to. - for location, sample_idx in enumerate(sampled_idx[batch_n, :]): - if point_idx == sample_idx: - expected_closest_idx = location - break - - self.assertIsNotNone(expected_closest_idx, - 'Point not found in sampled_idx result.') - self.assertEqual( - closest_idx[batch_n][point_idx], - expected_closest_idx, - msg='Closest index should be the point itself.') - - def testFarthestPointSamplerGatherPoints(self): - points = tf.constant([ - [[0, 1, 1], [1, 1, 1], [2, 1, 1], [3, 1, 1], [4, 1, 1], [5, 1, 1]], - [[0, 2, 1], [1, 2, 1], [2, 2, 1], [3, 2, 1], [4, 2, 1], [5, 2, 1]], - [[0, 2, 3], [1, 2, 3], [2, 2, 3], [3, 2, 3], [4, 2, 3], [5, 2, 3]], - [[0, 2, 1], [1, 2, 1], [2, 2, 1], [3, 2, 1], [4, 2, 1], [5, 2, 1]], - ], dtype=tf.float32) # pyformat: disable - padding = tf.zeros((4, 6), dtype=tf.float32) - n = 4 - num_points = 3 - selected_idx, _ = car_lib.FarthestPointSampler(points, padding, num_points) - gather_indices = tf.stack([ - tf.tile(tf.expand_dims(tf.range(n), 1), [1, num_points]), selected_idx - ], - axis=2) - sampled_points = tf.gather_nd(points, gather_indices) - with self.session(): - sampled_points = self.evaluate(sampled_points) - self.assertEqual(sampled_points.shape, (n, num_points, 3)) - - def testFarthestPointSamplerPadding(self): - points = tf.constant([ - [[0, 1, 1], [1, 1, 1], [2, 1, 1], [3, 1, 1], [4, 1, 1], [5, 1, 1]], - [[0, 2, 1], [1, 2, 1], [2, 2, 1], [3, 2, 1], [4, 2, 1], [5, 2, 1]], - [[0, 2, 3], [1, 2, 3], [2, 2, 3], [3, 2, 3], [4, 2, 3], [5, 2, 3]], - [[0, 2, 1], [1, 2, 1], [2, 2, 1], [3, 2, 1], [4, 2, 1], [5, 2, 1]], - ], - dtype=tf.float32) - - padding = tf.constant([[0, 0, 0, 0, 1, 1], [0, 0, 1, 1, 0, 0], - [1, 1, 0, 0, 0, 0], [1, 0, 0, 0, 0, 1]], - dtype=tf.float32) - - np_expected_selected_idx = np.array( - [[0, 1, 2, 3], [0, 1, 4, 5], [2, 3, 4, 5], [1, 2, 3, 4]], - dtype=np.int32) - - num_points = 4 - selected_idx, _ = car_lib.FarthestPointSampler(points, padding, num_points) - - with self.session(): - np_selected_idx = self.evaluate(selected_idx) - np_selected_idx.sort(axis=1) - self.assertAllEqual(np_selected_idx, np_expected_selected_idx) - - def _testPooling3D(self, pooling_fn): - num_points_in = 100 - num_points_out = 10 - batch_size = 8 - num_features = 32 - points = tf.random.uniform( - shape=(batch_size, num_points_in, 3), - minval=-1, - maxval=1, - dtype=tf.float32) - - # Note: This max pooling impl is incorrect if the feature range is negative. - features = tf.random.uniform( - shape=(batch_size, num_points_in, num_features), - minval=0, - maxval=1, - dtype=tf.float32) - padding = tf.zeros((batch_size, num_points_in), dtype=tf.float32) - - pooling_idx, closest_idx = car_lib.FarthestPointSampler( - points, padding, num_points_out) - - pooled_points, pooled_features = pooling_fn( - points=points, - point_features=features, - pooling_idx=pooling_idx, - closest_idx=closest_idx) - - with self.session(): - [ - np_points, np_features, np_pooling_idx, np_closest_idx, - np_pooled_points, np_pooled_features - ] = self.evaluate([ - points, features, pooling_idx, closest_idx, pooled_points, - pooled_features - ]) - - for batch_n in range(batch_size): - # Grab the selected pooling points from our sampler to compare to - # the output of our pooling. - expected_pooled_pts = np_points[batch_n, np_pooling_idx[batch_n, :], :] - self.assertAllClose(expected_pooled_pts, np_pooled_points[batch_n, ...]) - - np_batch_features = np_features[batch_n, ...] - for idx in range(num_points_out): - in_group = np_closest_idx[batch_n, :] == idx - expected_max = np.max(np_batch_features[in_group], axis=0) - self.assertAllClose(expected_max, np_pooled_features[batch_n, idx]) - - def testMaxPool3D(self): - return self._testPooling3D(car_lib.MaxPool3D) - - def testSegmentPool3D(self): - return self._testPooling3D(car_lib.SegmentPool3D) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/detection_3d_lib.py b/lingvo/tasks/car/detection_3d_lib.py deleted file mode 100644 index 1838b698..00000000 --- a/lingvo/tasks/car/detection_3d_lib.py +++ /dev/null @@ -1,863 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Library of useful for functions for working with 3D object detection.""" - -from lingvo import compat as tf -from lingvo.core import py_utils -from lingvo.tasks.car import geometry -from lingvo.tasks.car import ops -import numpy as np - - -class Utils3D: - """Helper routines for 3D detection problems. - - One common method to do 3D anchor box assignment is to match anchors to - ground-truth bboxes. First, we generate proposal anchors at given priors - (center locations, dimension prior, offset prior) to tile the input space. - After tiling the input space, each anchor can be assigned to a ground-truth - bbox by measuring IOU and taking a threshold. Note that a ground-truth bbox - may be assigned to multiple anchors - this is expected, and is managed at - inference time by non-max suppression. - - Note: This implementation is designed to be used at input generation, and - does not support a batch dimension. - - The following functions in this utility class helps with that: - - - CreateDenseCoordinates: Makes it easy to create a dense grid of - coordinates that would usually correspond to center locations. - - - MakeAnchorBoxes: Given a list of center coordinates, dimension priors, and - offset priors, this function creates actual anchor bbox parameters at each - coordinate. More than one box can be at each center. - - - IOUAxisAlignedBoxes: This function computes the IOU between two lists of - boxes. - - - AssignAnchors: This function assigns each anchor a ground-truth bbox. Note - that one ground-truth bbox can be assigned to multiple anchors. The model - is expected to regress the residuals each anchor and it's corresponding - ground-truth bbox parameters. - """ - - def ScaledHuberLoss(self, labels, predictions, weights=1.0, delta=1.0): - r"""Scaled Huber (SmoothL1) Loss. - - This function wraps tf.losses.huber_loss to rescale it by 1 / delta, and - uses Reduction.NONE. - - This scaling results in the following formulation instead:: - - (1/d) * 0.5 * x^2 if \|x\| <= d - \|x\| - 0.5 * d if \|x\| > d - - where x is labels - predictions. - - Hence, delta changes where the quadratic bowl is, but does not change the - overall shape of the loss outside of delta. - - Args: - labels: The ground truth output tensor, same dimensions as 'predictions'. - predictions: The predicted outputs. - weights: Optional Tensor whose rank is either 0, or the same rank as - labels, and must be broadcastable to labels (i.e., all dimensions must - be either 1, or the same as the corresponding losses dimension). - delta: float, the point where the huber loss function changes from a - quadratic to linear. - - Returns: - Weighted loss float Tensor. This has the same shape as labels. - """ - return (1. / delta) * tf.losses.huber_loss( - labels, - predictions, - weights=weights, - delta=delta, - reduction=tf.losses.Reduction.NONE) - - def CornerLoss(self, gt_bboxes, predicted_bboxes, symmetric=True): - """Corner regularization loss. - - This function computes the corner loss, an alternative regression loss - for box residuals. This was used in the Frustum-PointNets paper [1]. - - We compute the predicted bboxes (all 8 corners) and compute a SmoothedL1 - loss between the corners of the predicted boxes and ground truth. Hence, - this loss can help encourage the model to maximize the IoU of the - predictions. - - [1] Frustum PointNets for 3D Object Detection from RGB-D Data - https://arxiv.org/pdf/1711.08488.pdf - - Args: - gt_bboxes: tf.float32 of shape [..., 7] which contains (x, y, z, dx, dy, - dz, phi), corresponding to ground truth bbox parameters. - predicted_bboxes: tf.float32 of same shape as gt_bboxes containing - predicted bbox parameters. - symmetric: boolean. If True, computes the minimum of the corner loss - with respect to both the gt box and the gt box rotated 180 degrees. - - Returns: - tf.float32 Tensor of shape [...] where each entry contains the corner loss - for the corresponding bbox. - """ - bbox_shape = py_utils.GetShape(gt_bboxes) - batch_size = bbox_shape[0] - - gt_bboxes = tf.reshape(gt_bboxes, [batch_size, -1, 7]) - predicted_bboxes = tf.reshape(predicted_bboxes, [batch_size, -1, 7]) - - gt_corners = geometry.BBoxCorners(gt_bboxes) - predicted_corners = geometry.BBoxCorners(predicted_bboxes) - huber_loss = self.ScaledHuberLoss(gt_corners, predicted_corners) - huber_loss = tf.reduce_sum(huber_loss, axis=[-2, -1]) - - if symmetric: - # Compute the loss assuming the ground truth is flipped 180, and - # take the minimum of the two losses. - rot = tf.constant([[[0., 0., 0., 0., 0., 0., np.pi]]], dtype=tf.float32) - rotated_gt_bboxes = gt_bboxes + rot - rotated_gt_corners = geometry.BBoxCorners(rotated_gt_bboxes) - rotated_huber_loss = self.ScaledHuberLoss( - labels=rotated_gt_corners, predictions=predicted_corners) - rotated_huber_loss = tf.reduce_sum(rotated_huber_loss, axis=[-2, -1]) - huber_loss = tf.minimum(huber_loss, rotated_huber_loss) - - huber_loss = tf.reshape(huber_loss, bbox_shape[:-1]) - return huber_loss - - def CreateDenseCoordinates(self, ranges): - """Create a matrix of coordinate locations corresponding to a dense grid. - - Example: To create (x, y) coordinates corresponding over a 10x10 grid with - step sizes 1, call ``CreateDenseCoordinates([(1, 10, 10), (1, 10, 10)])``. - - Args: - ranges: A list of 3-tuples, each tuple is expected to contain (min, max, - num_steps). Each list element corresponds to one dimesion. Each tuple - will be passed into np.linspace to create the values for a single - dimension. - - Returns: - tf.float32 tensor of shape [total_points, len(ranges)], where - total_points = product of all num_steps. - - """ - total_points = int(np.prod([r_steps for _, _, r_steps in ranges])) - cycle_steps = total_points - stack_coordinates = [] - - for r_start, r_stop, r_steps in ranges: - values = tf.linspace( - tf.cast(r_start, tf.float32), tf.cast(r_stop, tf.float32), - tf.cast(r_steps, tf.int32)) - cycle_steps //= r_steps - gather_idx = (tf.range(total_points) // cycle_steps) % r_steps - stack_coordinates.append(tf.gather(values, gather_idx)) - - return tf.stack(stack_coordinates, axis=1) - - def MakeAnchorBoxes(self, - anchor_centers, - anchor_box_dimensions, - anchor_box_offsets, - anchor_box_rotations=None): - """Create anchor boxes from centers, dimensions, offsets. - - Args: - anchor_centers: [A, dims] tensor. Center locations to generate boxes at. - anchor_box_dimensions: [B, dims] tensor corresponding to dimensions of - each box. The inner-most dimension of this tensor must match - anchor_centers. - anchor_box_offsets: [B, dims] tensor corresponding to offsets of each box. - anchor_box_rotations: [B] tensor corresponding to rotation of each box. If - None, rotation will be set to 0. - - Returns: - A [num_anchors_center, num_boxes_per_center, 2 * dims + 1] tensor. Usually - dims=3 for 3D, where ``[..., :dims]`` corresponds to location, - ``[..., dims:2*dims]`` corresponds to dimensions, and ``[..., -1]`` - corresponds to rotation. - """ - num_centers, dims = py_utils.GetShape(anchor_centers) - num_box_per_center = py_utils.GetShape(anchor_box_dimensions, 1)[0] - - # Offset the centers by the box offsets - anchor_centers = tf.reshape(anchor_centers, [num_centers, 1, dims]) - anchor_box_offsets = tf.reshape(anchor_box_offsets, - [1, num_box_per_center, dims]) - anchor_box_centers = anchor_centers + anchor_box_offsets - anchor_box_centers = py_utils.HasShape( - anchor_box_centers, [num_centers, num_box_per_center, dims]) - - # Concat the dimensions and rotation parameters - anchor_box_dimensions = tf.tile( - tf.expand_dims(anchor_box_dimensions, 0), [num_centers, 1, 1]) - if anchor_box_rotations is None: - anchor_box_rotations = tf.zeros([num_centers, num_box_per_center, 1]) - else: - anchor_box_rotations = tf.tile( - tf.reshape(anchor_box_rotations, [1, num_box_per_center, 1]), - [num_centers, 1, 1]) - - anchor_bboxes = tf.concat( - [anchor_box_centers, anchor_box_dimensions, anchor_box_rotations], - axis=-1) - - return anchor_bboxes - - def IOU2DRotatedBoxes(self, bboxes_u, bboxes_v): - """Computes IoU between every pair of bboxes with headings. - - This function ignores the z dimension, which is not usually considered - during anchor assignment. - - Args: - bboxes_u: tf.float32. [U, dims]. [..., :7] are (x, y, z, dx, dy, dz, r). - bboxes_v: tf.float32. [V, dims]. [..., :7] are (x, y, z, dx, dy, dz, r). - - Returns: - tf.float32 tensor with shape [U, V], where [i, j] is IoU between - i-th bbox of bboxes_u and j-th bbox of bboxes_v. - """ - - def _IgnoreZCoordinate(bboxes): - """Set z center to 0, and z dimension to 1.""" - num_bboxes = py_utils.GetShape(bboxes, 1)[0] - return tf.stack([ - bboxes[:, 0], bboxes[:, 1], tf.zeros((num_bboxes,)), - bboxes[:, 3], bboxes[:, 4], tf.ones((num_bboxes,)), - bboxes[:, 6] - ], axis=1) # pyformat: disable - - bboxes_u = _IgnoreZCoordinate(bboxes_u) - bboxes_v = _IgnoreZCoordinate(bboxes_v) - return ops.pairwise_iou3d(bboxes_u, bboxes_v) - - def AssignAnchors(self, - anchor_bboxes, - gt_bboxes, - gt_bboxes_labels, - gt_bboxes_mask, - foreground_assignment_threshold=0.5, - background_assignment_threshold=0.35, - background_class_id=0, - force_match=True, - similarity_fn=None): - """Assigns anchors to bboxes using a similarity function (SSD-based). - - Each anchor box is assigned to the top matching ground truth box. - Ground truth boxes can be assigned to multiple anchor boxes. - - Assignments can result in 3 outcomes: - - - Positive assignment (if score >= foreground_assignment_threshold): - assigned_gt_labels will reflect the assigned box label and - assigned_cls_mask will be set to 1.0 - - Background assignment (if score <= background_assignment_threshold): - assigned_gt_labels will be background_class_id and assigned_cls_mask - will be set to 1.0 - - Ignore assignment (otherwise): - assigned_gt_labels will be background_class_id and assigned_cls_mask - will be set to 0.0 - - The detection loss function would usually: - - - Use assigned_cls_mask for weighting the classification loss. The mask - is set such that the loss applies to foreground and background - assignments only - ignored anchors will be set to 0. - - Use assigned_reg_mask for weighting the regression loss. The mask is set - such that the loss applies to foreground assignments only. - - The thresholds (foreground_assignment_threshold and - background_assignment_threshold) should be tuned per dataset. - - TODO(jngiam): Consider having a separate threshold for regression boxes; a - separate threshold is used in PointRCNN. - - Args: - anchor_bboxes: tf.float32. [A, 7], where [..., :] corresponds to box - parameters (x, y, z, dx, dy, dz, r). - gt_bboxes: tf.float32. [G, 7], where [..., :] corresponds to ground truth - box parameters (x, y, z, dx, dy, dz, r). - gt_bboxes_labels: tensor with shape [G]. Ground truth labels for each - bounding box. - gt_bboxes_mask: tensor with shape [G]. Mask for ground truth boxes, 1 iff - the gt_bbox is a real bbox. - foreground_assignment_threshold: Similarity score threshold for assigning - foreground bounding boxes; scores need to be >= - foreground_assignment_threshold to be assigned to foreground. - background_assignment_threshold: Similarity score threshold for assigning - background bounding boxes; scores need to be <= - background_assignment_threshold to be assigned to background. - background_class_id: class id to be assigned to anchors_gt_class if no - anchor boxes match. - force_match: Boolean specifying if force matching is enabled. If - force matching is enabled, then matched anchors which are also the - highest scoring with a ground-truth box are considered foreground - matches as long as their similarity score > 0. - similarity_fn: Function that computes the a similarity score (e.g., IOU) - between pairs of bounding boxes. This function should take in two - tensors corresponding to anchor and ground-truth bboxes, and return a - matrix [A, G] with the similarity score between each pair of bboxes. The - score must be non-negative, with greater scores representing more - similar. The fore/background_assignment_thresholds will be applied to - this score to determine if the an anchor is foreground, background or - ignored. If set to None, the function will default to IOU2DRotatedBoxes. - - Returns: - NestedMap with the following keys - - - assigned_gt_idx: shape [A] index corresponding to the index of the - assigned ground truth box. Anchors not assigned to a ground truth box - will have the index set to -1. - - assigned_gt_bbox: shape [A, 7] bbox parameters assigned to each anchor. - - assigned_gt_similarity_score: shape [A] (iou) score between the anchor - and the gt bbox. - - assigned_gt_labels: shape [A] label assigned to bbox. - - assigned_cls_mask: shape [A] mask for classification loss per anchor. - This should be 1.0 if the anchor has a foreground or background - assignment; otherwise, it will be assigned to 0.0. - - assigned_reg_mask: shape [A] mask for regression loss per anchor. - This should be 1.0 if the anchor has a foreground assignment; - otherwise, it will be assigned to 0.0. - Note: background anchors do not have regression targets. - """ - if similarity_fn is None: - similarity_fn = self.IOU2DRotatedBoxes - - # Shape validation. - anchor_bboxes = py_utils.HasShape(anchor_bboxes, [-1, 7]) - num_anchor_bboxes, _ = py_utils.GetShape(anchor_bboxes, 2) - gt_bboxes = py_utils.HasShape(gt_bboxes, [-1, 7]) - num_gt_bboxes, _ = py_utils.GetShape(gt_bboxes, 2) - - # Compute similarity score and reduce max by anchors and by ground-truth. - similarity_score = similarity_fn(anchor_bboxes, gt_bboxes) - similarity_score = py_utils.HasShape(similarity_score, - [num_anchor_bboxes, num_gt_bboxes]) - - # Reduce over ground-truth boxes, so we have the max score per anchor. - anchor_max_score = tf.reduce_max(similarity_score, axis=1) - anchor_max_idx = tf.argmax(similarity_score, axis=1) - - if force_match: - # Reduce over anchors, so we have the max score per ground truth box. - gt_max_score = tf.reduce_max(similarity_score, axis=0, keepdims=True) - - # Force matches occur when the top matching gt bbox for an anchor is the - # top matching anchor for the gt bbox. When force matching, we match - # these boxes as long as their similarity score exceeds 0. - force_matches = ( - tf.equal(similarity_score, gt_max_score) - & tf.equal(similarity_score, anchor_max_score[..., tf.newaxis]) - & tf.greater(similarity_score, 0.) - & tf.cast(gt_bboxes_mask[tf.newaxis, ...], tf.bool)) - force_match_indicator = tf.reduce_any(force_matches, axis=1) - force_match_idx = tf.argmax(tf.cast(force_matches, tf.int32), axis=1) - - # In assigning foreground/background anchors later, force_match_indicator - # is used to determine which anchors are force foreground, and the index - # assigned will be taken from anchor_max_idx. - - # Force matchers must also be the max scoring gt bbox per anchor. - # We overwrite anchor_max_idx to ensure that the right match is done. - anchor_max_idx = tf.where(force_match_indicator, force_match_idx, - anchor_max_idx) - - # Ensure that max score boxes are not padded boxes by setting score to 0 - # for boxes that are padded. - gathered_mask = tf.array_ops.batch_gather(gt_bboxes_mask, anchor_max_idx) - anchor_max_score = tf.where( - tf.equal(gathered_mask, 1), anchor_max_score, - tf.zeros_like(anchor_max_score)) - - # Boolean tensors corresponding to whether an anchor is background or - # foreground based on thresholding. - background_anchors = tf.less_equal(anchor_max_score, - background_assignment_threshold) - foreground_anchors = tf.greater_equal(anchor_max_score, - foreground_assignment_threshold) - if force_match: - # Background anchors are below threshold and not force matches. - background_anchors &= ~force_match_indicator - # Foreground anchors are above thresholds or force matches. - foreground_anchors |= force_match_indicator - - # Add dummy background bbox to gt_boxes to facilitate batch gather. - dummy_bbox = tf.constant([[0, 0, 0, 1, 1, 1, 0]], dtype=tf.float32) - - # Since we are concatenating the dummy bbox, the index corresponds to the - # number of boxes. - dummy_bbox_idx = py_utils.GetShape(gt_bboxes, 1)[0] - dummy_bbox_idx = tf.cast(dummy_bbox_idx, tf.int64) - - gt_bboxes = tf.concat([gt_bboxes, dummy_bbox], axis=0) - gt_bboxes_labels = tf.concat([gt_bboxes_labels, [background_class_id]], - axis=0) - - # Gather indices so that all foreground boxes are gathered from gt_bboxes, - # while all background and ignore boxes gather the dummy_bbox. - anchor_gather_idx = tf.where(foreground_anchors, anchor_max_idx, - tf.ones_like(anchor_max_idx) * dummy_bbox_idx) - - # Gather the bboxes and weights. - assigned_gt_bbox = tf.array_ops.batch_gather(gt_bboxes, anchor_gather_idx) - assigned_gt_labels = tf.array_ops.batch_gather(gt_bboxes_labels, - anchor_gather_idx) - - # Set masks for classification and regression losses. - assigned_cls_mask = tf.cast(background_anchors | foreground_anchors, - tf.float32) - assigned_reg_mask = tf.cast(foreground_anchors, tf.float32) - - # Set assigned_gt_idx such that dummy boxes have idx = -1. - assigned_gt_idx = tf.where( - tf.equal(anchor_gather_idx, dummy_bbox_idx), - tf.ones_like(anchor_gather_idx) * -1, anchor_gather_idx) - assigned_gt_idx = tf.cast(assigned_gt_idx, tf.int32) - - return py_utils.NestedMap( - assigned_gt_idx=assigned_gt_idx, - assigned_gt_bbox=assigned_gt_bbox, - assigned_gt_similarity_score=anchor_max_score, - assigned_gt_labels=assigned_gt_labels, - assigned_cls_mask=assigned_cls_mask, - assigned_reg_mask=assigned_reg_mask) - - def LocalizationResiduals(self, anchor_bboxes, assigned_gt_bboxes): - """Computes the anchor residuals for every bbox. - - For a given bbox, compute residuals in the following way: - - Let ``anchor_bbox = (x_a, y_a, z_a, dx_a, dy_a, dz_a, phi_a)`` - and ``assigned_gt_bbox = (x_gt, y_gt, z_gt, dx_gt, dy_gt, dz_gt, phi_gt)`` - - Define ``diagonal_xy = sqrt(dx_a^2 + dy_a^2)`` - - Then the corresponding residuals are given by:: - - x_residual = (x_gt - x_a) / (diagonal_xy) - y_residual = (y_gt - y_a) / (diagonal_xy) - z_residual = (z_gt - z_a) / (dz_a) - - dx_residual = log(dx_gt / dx_a) - dy_residual = log(dy_gt / dy_a) - dz_residual = log(dz_gt / dz_a) - - phi_residual = phi_gt - phi_a - - The normalization for x and y residuals by the diagonal was first - proposed by [1]. Intuitively, this reflects that objects can usually - move freely in the x-y plane, including diagonally. On the other hand, - moving in the z-axis (up and down) can be considered orthogonal to x-y. - - For phi_residual, one way to frame the loss is with - SmoothL1(sine(phi_residual - phi_predicted)). - The use of sine to wrap the phi residual was proposed by [2]. This - stems from the observation that bboxes at phi and phi + pi are the same - bbox, fully overlapping in 3D space, except that the direction is - different. Note that the use of sine makes this residual invariant to - direction when a symmetric loss like SmoothL1 is used. In - ResidualsToBBoxes, we ensure that the phi predicted is between [0, pi). - - The Huber (SmoothL1) loss can then be applied to the delta between these - target residuals and the model predicted residuals. - - [1] VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection - https://arxiv.org/abs/1711.06396 - - [2] SECOND: Sparsely Embedded Convolutional Detection - https://pdfs.semanticscholar.org/5125/a16039cabc6320c908a4764f32596e018ad3.pdf - - Args: - anchor_bboxes: tf.float32. where [..., :7] contains (x, y, z, dx, dy, dz, - phi), corresponding to each anchor bbox parameters. - assigned_gt_bboxes: tf.float32 of the same shape as anchor_bboxes - containing the corresponding assigned ground-truth bboxes. - - Returns: - A tf.float32 tensor of the same shape as anchor_bboxes with target - residuals for every corresponding bbox. - """ - anchor_bboxes_shape = py_utils.GetShape(anchor_bboxes) - anchor_bboxes = py_utils.with_dependencies( - [py_utils.assert_equal(anchor_bboxes_shape[-1], 7)], anchor_bboxes) - assigned_gt_bboxes = py_utils.HasShape(assigned_gt_bboxes, - anchor_bboxes_shape) - - x_a, y_a, z_a, dx_a, dy_a, dz_a, phi_a = tf.unstack( - anchor_bboxes, num=7, axis=-1) - x_gt, y_gt, z_gt, dx_gt, dy_gt, dz_gt, phi_gt = tf.unstack( - assigned_gt_bboxes, num=7, axis=-1) - - diagonal_xy = tf.sqrt(tf.square(dx_a) + tf.square(dy_a)) - - # The anchor dimensions is usually a hard-coded param given to the input - # generator and should not be 0. We use CheckNumerics to ensure that is the - # case. - x_residual = py_utils.CheckNumerics((x_gt - x_a) / diagonal_xy) - y_residual = py_utils.CheckNumerics((y_gt - y_a) / diagonal_xy) - z_residual = py_utils.CheckNumerics((z_gt - z_a) / dz_a) - - dx_residual = py_utils.CheckNumerics(tf.math.log(dx_gt / dx_a)) - dy_residual = py_utils.CheckNumerics(tf.math.log(dy_gt / dy_a)) - dz_residual = py_utils.CheckNumerics(tf.math.log(dz_gt / dz_a)) - - phi_residual = phi_gt - phi_a - - return tf.stack([ - x_residual, y_residual, z_residual, - dx_residual, dy_residual, dz_residual, - phi_residual, - ], axis=-1) # pyformat: disable - - def ResidualsToBBoxes(self, - anchor_bboxes, - residuals, - min_angle_rad=-np.pi, - max_angle_rad=np.pi): - r"""Converts anchor_boxes and residuals to predicted bboxes. - - This converts predicted residuals into bboxes using the following formulae:: - - x_predicted = x_a + x_residual * diagonal_xy - y_predicted = y_a + y_residual * diagonal_xy - z_predicted = z_a + z_residual * dz_a - - dx_predicted = dx_a * exp(dx_residual) - dy_predicted = dy_a * exp(dy_residual) - dz_predicted = dz_a * exp(dz_residual) - - # Adding the residual, and bounding it between - # [min_angle_rad, max_angle_rad] - phi_predicted = NormalizeAngleRad(phi_a + phi_residual, - min_angle_rad, max_angle_rad) - - These equations follow from those in LocalizationResiduals, where we solve - for the \*_gt variables. - - Args: - anchor_bboxes: tf.float32. where [..., :7] contains (x, y, z, dx, dy, dz, - phi), corresponding to each anchor bbox parameters. - residuals: tf.float32 of the same shape as anchor_bboxes containing - predicted residuals at each anchor location. - min_angle_rad: Scalar with the minimum angle allowed (before wrapping) - in radians. - max_angle_rad: Scalar with the maximum angle allowed (before wrapping) - in radians. This value usually should be pi. - - Returns: - A tf.float32 tensor of the same shape as anchor_bboxes with predicted - bboxes. - """ - anchor_bboxes_shape = py_utils.GetShape(anchor_bboxes) - anchor_bboxes = py_utils.with_dependencies( - [py_utils.assert_equal(anchor_bboxes_shape[-1], 7)], anchor_bboxes) - residuals = py_utils.HasShape(residuals, anchor_bboxes_shape) - - x_a, y_a, z_a, dx_a, dy_a, dz_a, phi_a = tf.unstack( - anchor_bboxes, num=7, axis=-1) - (x_residual, y_residual, z_residual, dx_residual, dy_residual, dz_residual, - phi_residual) = tf.unstack( - residuals, num=7, axis=-1) - - diagonal_xy = tf.sqrt(tf.square(dx_a) + tf.square(dy_a)) - - x_predicted = x_a + x_residual * diagonal_xy - y_predicted = y_a + y_residual * diagonal_xy - z_predicted = z_a + z_residual * dz_a - - dx_predicted = dx_a * tf.exp(dx_residual) - dy_predicted = dy_a * tf.exp(dy_residual) - dz_predicted = dz_a * tf.exp(dz_residual) - - # We bound the angle between [min_angle_rad, max_angle_rad], which should - # be passed in depending on the heading handling in the calling model. - # If the model uses a sine(delta_phi) transformation in the loss, then it - # cannot distinguish direction and a [0, np.pi] - # [min_angle_rad, max_angle_rad] should be used. - # If there is a heading encoding that is directional, most likely you - # should use a [-np.pi, np.pi] [min_angle_rad, max_angle_rad]. - phi_predicted = phi_a + phi_residual - phi_predicted = geometry.WrapAngleRad(phi_predicted, min_angle_rad, - max_angle_rad) - - return tf.stack([ - x_predicted, y_predicted, z_predicted, - dx_predicted, dy_predicted, dz_predicted, - phi_predicted, - ], axis=-1) # pyformat: disable - - def NMSIndices(self, - bboxes, - scores, - max_output_size, - nms_iou_threshold=0.3, - score_threshold=0.01): - """Apply NMS to a series of 3d bounding boxes in 7-DOF format. - - Args: - bboxes: A [num_boxes, 7] floating point Tensor of bounding boxes in [x, y, - z, dx, dy, dz, phi] format. - scores: A [num_boxes] floating point Tensor containing box - scores. - max_output_size: Maximum number of boxes to predict per input. - nms_iou_threshold: IoU threshold to use when determining whether two boxes - overlap for purposes of suppression. - score_threshold: The score threshold passed to NMS that allows NMS to - quickly ignore irrelevant boxes. - - Returns: - The NMS indices and the mask of the padded indices. - """ - bboxes = py_utils.HasShape(bboxes, [-1, 7]) - - # Extract x, y, w, h, then convert to extrema. - # - # Note that we drop the rotation angle because we don't have an NMS - # operation that takes rotation into account. - bboxes_2d = tf.stack( - [bboxes[:, 0], bboxes[:, 1], bboxes[:, 3], bboxes[:, 4]], axis=-1) - bboxes_extrema = geometry.XYWHToBBoxes(bboxes_2d) - - # Compute NMS with padding; we use the padded version so this function can - # be used in a map_fn. This function returns the scalar number of boxes - # for each example. - # - # We use an IoU threshold of 0.3 since our anchor boxes have rotations - # that make the default IoU threshold of 0.5 possibly too high. - nms_index_padded, num_valid = tf.image.non_max_suppression_padded( - bboxes_extrema, - scores, - iou_threshold=nms_iou_threshold, - max_output_size=max_output_size, - score_threshold=score_threshold, - pad_to_max_output_size=True) - - # Return the mask of valid indices instead of just a scalar number. - mask = tf.concat( - [tf.ones([num_valid]), - tf.zeros([max_output_size - num_valid])], axis=0) - - nms_index_padded = tf.where(mask > 0, nms_index_padded, - tf.zeros_like(nms_index_padded)) - return nms_index_padded, mask - - def BatchedNMSIndices(self, - bboxes, - scores, - nms_iou_threshold=0.3, - score_threshold=0.01, - max_num_boxes=None): - """Batched version of NMSIndices. - - Args: - bboxes: A [batch_size, num_boxes, 7] floating point Tensor of bounding - boxes in [x, y, z, dx, dy, dz, phi] format. - scores: A [batch_size, num_boxes, num_classes] floating point Tensor - containing box scores. - nms_iou_threshold: IoU threshold to use when determining whether two boxes - overlap for purposes of suppression. - score_threshold: The score threshold passed to NMS that allows NMS to - quickly ignore irrelevant boxes. - max_num_boxes: The maximum number of boxes per example to emit. If None, - this value is set to num_boxes from the shape of bboxes. - - Returns: - The NMS indices and the mask of the padded indices for each example - in the batch. - """ - batch_size, num_boxes = py_utils.GetShape(bboxes, 2) - - if max_num_boxes is not None: - max_output_size = max_num_boxes - else: - max_output_size = num_boxes - - output_shape = [batch_size, max_output_size] - - def NMSBody(args): - bbox, score = args - return self.NMSIndices(bbox, score, max_output_size, nms_iou_threshold, - score_threshold) - - nms_indices, valid_mask = tf.map_fn( - fn=NMSBody, - elems=(bboxes, scores), - dtype=(tf.int32, tf.float32), - back_prop=False) - - nms_indices = py_utils.PadOrTrimTo(nms_indices, output_shape) - return nms_indices, valid_mask - - def BatchedOrientedNMSIndices(self, bboxes, scores, nms_iou_threshold, - score_threshold, max_boxes_per_class): - """Runs batched version of a Per-Class 3D (7-DOF) Non Max Suppression. - - All outputs have shape [batch_size, num_classes, max_boxes_per_class]. - - Args: - bboxes: A [batch_size, num_boxes, 7] floating point Tensor of bounding - boxes in [x, y, z, dx, dy, dz, phi] format. - scores: A [batch_size, num_boxes, num_classes] floating point Tensor - containing box scores. - nms_iou_threshold: Either a float or a list of floats of len num_classes - with the IoU threshold to use when determining whether two boxes overlap - for purposes of suppression. - score_threshold: Either a float or a list of floats of len num_classes - with the score threshold that allows NMS to quickly ignore boxes. - max_boxes_per_class: An integer scalar with the maximum number of boxes - per example to emit per class. - - Returns: - A tuple of 3 tensors: - - - bbox_indices: An int32 Tensor with the indices of the chosen boxes. - Values are in sort order until the class_idx switches. - - bbox_scores: A float32 Tensor with the score for each box. - - valid_mask: A float32 Tensor with 1/0 values indicating the validity of - each box. 1 indicates valid, and 0 invalid. - """ - bboxes = py_utils.HasShape(bboxes, [-1, -1, 7]) - batch_size, num_boxes = py_utils.GetShape(bboxes, 2) - scores = py_utils.HasShape(scores, [batch_size, num_boxes, -1]) - _, _, num_classes = py_utils.GetShape(scores) - - # Force the thresholds to be tensors of len num_classes - nms_iou_threshold = tf.broadcast_to( - tf.convert_to_tensor(nms_iou_threshold), [num_classes]) - score_threshold = tf.broadcast_to( - tf.convert_to_tensor(score_threshold), [num_classes]) - - def NMSBody(args): - per_sample_bboxes, per_sample_scores = args - indices, scores, mask = ops.non_max_suppression_3d( - per_sample_bboxes, - per_sample_scores, - nms_iou_threshold=nms_iou_threshold, - score_threshold=score_threshold, - max_boxes_per_class=max_boxes_per_class) - return indices, scores, mask - - bbox_indices, bbox_scores, valid_mask = tf.map_fn( - fn=NMSBody, - elems=(bboxes, scores), - dtype=(tf.int32, tf.float32, tf.float32), - back_prop=False) - - output_shape = [batch_size, num_classes, max_boxes_per_class] - bbox_indices = py_utils.PadOrTrimTo(bbox_indices, output_shape) - bbox_scores = py_utils.PadOrTrimTo(bbox_scores, output_shape) - valid_mask = py_utils.PadOrTrimTo(valid_mask, output_shape) - return bbox_indices, bbox_scores, valid_mask - - def CornersToImagePlane(self, corners, velo_to_image_plane): - """Project 3d box corners to the image plane. - - Args: - corners: A [batch, num_boxes, 8, 3] floating point tensor containing the 8 - corners points for each 3d bounding box. - velo_to_image_plane: A [batch, 3, 4] batch set of projection matrices from - velo xyz to image plane xy. After multiplication, you need to divide by - last coordinate to recover 2D pixel locations. - - Returns: - A [batch, num_boxes, 8, 2] floating point Tensor containing the 3D - bounding box corners projected to the image plane. - """ - batch_size, num_boxes, _, _ = py_utils.GetShape(corners, 4) - - def CornersToPlaneBody(args): - """Body of function to convert each bounding box to the image plane.""" - (corners, velo_to_image_plane) = args - # corners[i] is [num_boxes, 8, 3]: flatten the points in this batch and do - # the conversion in one call. - bbox_corners = tf.reshape(corners, [-1, 3]) - image_plane_corners = geometry.PointsToImagePlane(bbox_corners, - velo_to_image_plane) - image_plane_corners = tf.reshape(image_plane_corners, [-1, 8, 2]) - return image_plane_corners - - corners_in_image_plane = tf.map_fn( - fn=CornersToPlaneBody, - elems=(corners, velo_to_image_plane), - dtype=tf.float32, - back_prop=False) - - corners_in_image_plane = py_utils.HasShape(corners_in_image_plane, - [batch_size, num_boxes, 8, 2]) - return corners_in_image_plane - - -def RandomPadOrTrimTo(tensor_list, num_points_out, seed=None): - """Pads or Trims a list of Tensors on the major dimension. - - Slices if there are more points, or pads if not enough. - - In this implementation: - Padded points are random duplications of real points. - Sliced points are a random subset of the real points. - - Args: - tensor_list: A list of tf.Tensor objects to pad or trim along first dim. All - tensors are expected to have the same first dimension. - num_points_out: An int for the requested number of points to trim/pad to. - seed: Random seed to use for random generators. - - Returns: - A tuple of output_tensors and a padding indicator. - - - output_tensors: A list of padded or trimmed versions of our tensor_list - input tensors, all with the same first dimension. - - padding: A tf.float32 tf.Tensor of shape [num_points_out] with 0 if the - point is real, 1 if it is padded. - """ - actual_num = tf.shape(tensor_list[0])[0] - point_idx = tf.range(num_points_out, dtype=tf.int32) - padding_tensor = tf.where(point_idx < actual_num, - tf.zeros([num_points_out], dtype=tf.float32), - tf.ones([num_points_out], dtype=tf.float32)) - - def _Slicing(): - # Choose a random set of indices. - indices = tf.range(actual_num) - indices = tf.random.shuffle(indices, seed=seed)[:num_points_out] - return [tf.gather(t, indices, axis=0) for t in tensor_list] - - def _Padding(): - indices = tf.random.uniform([num_points_out - actual_num], - minval=0, - maxval=actual_num, - dtype=tf.int32, - seed=seed) - padded = [] - for t in tensor_list: - padded.append(tf.concat([t, tf.gather(t, indices, axis=0)], axis=0)) - return padded - - def _PadZeros(): - padded = [] - for t in tensor_list: - shape = tf.concat([[num_points_out], tf.shape(t)[1:]], axis=0) - padded.append(tf.zeros(shape=shape, dtype=t.dtype)) - return padded - - data = tf.cond( - actual_num > num_points_out, - _Slicing, lambda: tf.cond(tf.equal(actual_num, 0), _PadZeros, _Padding)) - return (data, padding_tensor) diff --git a/lingvo/tasks/car/detection_3d_lib_test.py b/lingvo/tasks/car/detection_3d_lib_test.py deleted file mode 100644 index b5c6dbe5..00000000 --- a/lingvo/tasks/car/detection_3d_lib_test.py +++ /dev/null @@ -1,528 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for detection_3d_lib.""" - -from lingvo import compat as tf -from lingvo.core import test_utils -from lingvo.tasks.car import detection_3d_lib -import numpy as np - - -class Utils3DTest(test_utils.TestCase): - - def testScaledHuberLoss(self): - utils_3d = detection_3d_lib.Utils3D() - labels = tf.constant([1, 2, 3], dtype=tf.float32) - # Predictions are less than delta, exactly at delta, and more than delta, - # respectively. - predictions = tf.constant([1.4, 1.2, 4.0], dtype=tf.float32) - delta = 0.8 - expected_loss = [ - 1. / delta * 0.5 * (0.4)**2, - 0.5 * delta, - 1.0 - 0.5 * delta, - ] - loss = utils_3d.ScaledHuberLoss(labels, predictions, delta=delta) - with self.session(): - actual_loss = self.evaluate(loss) - self.assertAllClose(actual_loss, expected_loss) - - def testCornerLoss(self): - utils_3d = detection_3d_lib.Utils3D() - gt_bboxes = tf.constant([[[[0., 0., 0., 1., 1., 1., 0.], - [0., 0., 0., 1., 1., 1., 0.], - [0., 0., 0., 1., 1., 1., 0.], - [0., 0., 0., 1., 1., 1., 0.], - [0., 0., 0., 1., 1., 1., 0.]]]]) - predicted_bboxes = tf.constant([[[ - [0., 0., 0., 1., 1., 1., 0.], # Same as GT - [0., 0., 0., 1., 1., 1., np.pi], # Opposite heading - [0., 0., 0., 1., 1., 1., np.pi / 2.], # 90-deg rotation - [1., 1., 1., 1., 1., 1., 0], # Different center - [0., 0., 0., 2., 2., 2., 0], # Different size - ]]]) - loss = utils_3d.CornerLoss(gt_bboxes, predicted_bboxes) - with self.session(): - actual_loss = self.evaluate(loss) - self.assertEqual(actual_loss.shape, (1, 1, 5)) - - def testCornerLossAsym(self): - utils_3d = detection_3d_lib.Utils3D() - gt_bboxes = tf.constant([[[[0., 0., 0., 1., 1., 1., 0.], - [0., 0., 0., 1., 1., 1., 0.]]]]) - predicted_bboxes = tf.constant([[[ - [0., 0., 0., 1., 1., 1., 0.], # Same as GT - [0., 0., 0., 1., 1., 1., np.pi], # Opposite heading - ]]]) - expected_loss = [[[ - 0., - 8, - ]]] - loss = utils_3d.CornerLoss(gt_bboxes, predicted_bboxes, symmetric=False) - with self.session(): - actual_loss = self.evaluate(loss) - print(actual_loss) - self.assertAllClose(actual_loss, expected_loss) - - def testCreateDenseCoordinates(self): - utils_3d = detection_3d_lib.Utils3D() - one_dim = utils_3d.CreateDenseCoordinates([(0.5, 1.5, 3)]) - with self.session(): - actual_one_dim = self.evaluate(one_dim) - self.assertAllEqual(actual_one_dim, [[0.5], [1.0], [1.5]]) - - two_by_two = utils_3d.CreateDenseCoordinates([(0, 1, 2), (1, 2, 2)]) - with self.session(): - actual_two_by_two = self.evaluate(two_by_two) - self.assertAllEqual(actual_two_by_two, [[0, 1], [0, 2], [1, 1], [1, 2]]) - - three_dims = utils_3d.CreateDenseCoordinates([(0, 1, 5), (1, 2, 5), - (0, 10, 5)]) - self.assertAllEqual(three_dims.shape, [5 * 5 * 5, 3]) - - def testMakeAnchorBoxesWithoutRotation(self): - utils_3d = detection_3d_lib.Utils3D() - anchor_bboxes = utils_3d.MakeAnchorBoxes( - anchor_centers=tf.constant([[0, 0, 0], [1, 1, 1]], dtype=tf.float32), - anchor_box_dimensions=tf.constant([[1, 2, 3], [3, 4, 5]], - dtype=tf.float32), - anchor_box_offsets=tf.constant([[0, 0, 0], [1, 1, 1]], - dtype=tf.float32), - anchor_box_rotations=None) - with self.session(): - actual_anchor_bboxes = self.evaluate(anchor_bboxes) - self.assertAllEqual(actual_anchor_bboxes, - [[[0, 0, 0, 1, 2, 3, 0], [1, 1, 1, 3, 4, 5, 0]], - [[1, 1, 1, 1, 2, 3, 0], [2, 2, 2, 3, 4, 5, 0]]]) - - def testMakeAnchorBoxesWithRotation(self): - utils_3d = detection_3d_lib.Utils3D() - anchor_bboxes = utils_3d.MakeAnchorBoxes( - anchor_centers=tf.constant([[0, 0, 0], [1, 1, 1]], dtype=tf.float32), - anchor_box_dimensions=tf.constant([[1, 2, 3], [3, 4, 5]], - dtype=tf.float32), - anchor_box_offsets=tf.constant([[0, 0, 0], [1, 1, 1]], - dtype=tf.float32), - anchor_box_rotations=tf.constant([0, 0.5])) - with self.session(): - actual_anchor_bboxes = self.evaluate(anchor_bboxes) - self.assertAllEqual(actual_anchor_bboxes, - [[[0, 0, 0, 1, 2, 3, 0], [1, 1, 1, 3, 4, 5, 0.5]], - [[1, 1, 1, 1, 2, 3, 0], [2, 2, 2, 3, 4, 5, 0.5]]]) - - def testAssignAnchors(self): - utils_3d = detection_3d_lib.Utils3D() - anchor_bboxes = tf.constant( - [ - [0, 1, 1, 2, 2, 2, 0], # Ignored - [-1, 1, 1, 2, 2, 2, 0], # Background - [0.9, 1, 1, 2, 2, 2, 0], # Foreground - [5, 5, 5, 1, 1, 2, 0], # Force matched to foreground - ], - dtype=tf.float32) - - # Second gt box should be forced match, third one should be ignored. - gt_bboxes = tf.constant([[1, 1, 1, 2, 2, 2, 0], [5, 5, 5, 2, 2, 2, 0], - [10, 10, 10, 2, 2, 2, 0]], - dtype=tf.float32) - gt_bboxes_labels = tf.constant([1, 2, 3]) - gt_bboxes_mask = tf.constant([1, 1, 1]) - - assigned_anchors = utils_3d.AssignAnchors( - anchor_bboxes, - gt_bboxes, - gt_bboxes_labels, - gt_bboxes_mask, - foreground_assignment_threshold=0.5, - background_assignment_threshold=0.25) - with self.session(): - actual_assigned_anchors, gt_bboxes = self.evaluate( - (assigned_anchors, gt_bboxes)) - - self.assertAllEqual(actual_assigned_anchors.assigned_gt_idx, - [-1, -1, 0, 1]) - self.assertAllEqual(actual_assigned_anchors.assigned_gt_labels, - [0, 0, 1, 2]) - self.assertAllEqual(actual_assigned_anchors.assigned_gt_bbox, [ - [0, 0, 0, 1, 1, 1, 0], - [0, 0, 0, 1, 1, 1, 0], - [1, 1, 1, 2, 2, 2, 0], - [5, 5, 5, 2, 2, 2, 0], - ]) - - self.assertAllEqual(actual_assigned_anchors.assigned_cls_mask, - [0, 1, 1, 1]) - self.assertAllEqual(actual_assigned_anchors.assigned_reg_mask, - [0, 0, 1, 1]) - - self.assertAllEqual( - actual_assigned_anchors.assigned_gt_similarity_score.shape, [4]) - - def testAssignAnchorsWithoutForceMatch(self): - utils_3d = detection_3d_lib.Utils3D() - anchor_bboxes = tf.constant( - [ - [0, 1, 1, 2, 2, 2, 0], # Ignored - [-1, 1, 1, 2, 2, 2, 0], # Background - [0.9, 1, 1, 2, 2, 2, 0], # Foreground - [5, 5, 5, 1, 1, 2, 0], # Background, since no force match - ], - dtype=tf.float32) - - # Second gt box should be forced match, third one should be ignored. - gt_bboxes = tf.constant([[1, 1, 1, 2, 2, 2, 0], [5, 5, 5, 2, 2, 2, 0], - [10, 10, 10, 2, 2, 2, 0]], - dtype=tf.float32) - gt_bboxes_labels = tf.constant([1, 2, 3]) - gt_bboxes_mask = tf.constant([1, 1, 1]) - - assigned_anchors = utils_3d.AssignAnchors( - anchor_bboxes, - gt_bboxes, - gt_bboxes_labels, - gt_bboxes_mask, - foreground_assignment_threshold=0.5, - background_assignment_threshold=0.25, - force_match=False) - with self.session(): - actual_assigned_anchors, gt_bboxes = self.evaluate( - (assigned_anchors, gt_bboxes)) - - self.assertAllEqual(actual_assigned_anchors.assigned_gt_idx, - [-1, -1, 0, -1]) - self.assertAllEqual(actual_assigned_anchors.assigned_gt_labels, - [0, 0, 1, 0]) - self.assertAllEqual(actual_assigned_anchors.assigned_gt_bbox, [ - [0, 0, 0, 1, 1, 1, 0], - [0, 0, 0, 1, 1, 1, 0], - [1, 1, 1, 2, 2, 2, 0], - [0, 0, 0, 1, 1, 1, 0], - ]) - - self.assertAllEqual(actual_assigned_anchors.assigned_cls_mask, - [0, 1, 1, 1]) - self.assertAllEqual(actual_assigned_anchors.assigned_reg_mask, - [0, 0, 1, 0]) - - self.assertAllEqual( - actual_assigned_anchors.assigned_gt_similarity_score.shape, [4]) - - def testAssignAnchorsWithPadding(self): - utils_3d = detection_3d_lib.Utils3D() - anchor_bboxes = tf.constant([[0, 0, 0, 1, 2, 3, 0], [1, 1, 1, 3, 4, 5, 0.5], - [1, 1, 1, 1, 2, 3, 0], [2, 2, 2, 3, 4, 5, - 0.5]]) - gt_bboxes = anchor_bboxes + 0.05 - gt_bboxes_labels = tf.constant([1, 2, 3, 4]) - gt_bboxes_mask = tf.constant([1, 1, 0, 0]) - - assigned_anchors = utils_3d.AssignAnchors(anchor_bboxes, gt_bboxes, - gt_bboxes_labels, gt_bboxes_mask) - with self.session(): - actual_assigned_anchors, gt_bboxes = self.evaluate( - (assigned_anchors, gt_bboxes)) - - # Last two boxes are padded, thus not assigned. - self.assertAllEqual(actual_assigned_anchors.assigned_gt_idx, - [0, 1, -1, -1]) - self.assertAllEqual(actual_assigned_anchors.assigned_gt_labels, - [1, 2, 0, 0]) - self.assertAllEqual(actual_assigned_anchors.assigned_gt_bbox[0:2, :], - gt_bboxes[0:2, :]) - - # 2nd and 3rd should match dummy bbox. - self.assertAllEqual(actual_assigned_anchors.assigned_gt_bbox[2, :], - [0, 0, 0, 1, 1, 1, 0]) - self.assertAllEqual(actual_assigned_anchors.assigned_gt_bbox[3, :], - [0, 0, 0, 1, 1, 1, 0]) - - # First two are foreground, last two are background. - self.assertAllEqual(actual_assigned_anchors.assigned_cls_mask, - [1, 1, 1, 1]) - self.assertAllEqual(actual_assigned_anchors.assigned_reg_mask, - [1, 1, 0, 0]) - - self.assertAllEqual( - actual_assigned_anchors.assigned_gt_similarity_score.shape, [4]) - - def testLocalizationResiduals(self): - utils_3d = detection_3d_lib.Utils3D() - - anchor_bboxes = tf.constant([[1, 2, 3, 4, 3, 6, 0]], dtype=tf.float32) - gt_bboxes = tf.constant([[2, 22, 303, 4, 9, 12, 0.5]], dtype=tf.float32) - - # diagonal_xy = 5 [since sqrt(3^2 + 4^2) = 5] - expected_residuals = np.asarray([[ - 1. / 5, - 20. / 5, - 300. / 6, - 0., - np.log(9. / 3.), - np.log(12. / 6.), - 0.5, - ]]) - residuals = utils_3d.LocalizationResiduals(anchor_bboxes, gt_bboxes) - - with self.session(): - actual_residuals = self.evaluate(residuals) - self.assertAllClose(actual_residuals, expected_residuals) - - def testResidualsToBBoxes(self): - utils_3d = detection_3d_lib.Utils3D() - - anchor_bboxes = tf.constant([[1, 2, 3, 4, 3, 6, 0]], dtype=tf.float32) - expected_predicted_bboxes = np.asarray([[2, 22, 303, 4, 9, 12, 0.5]]) - - residuals = tf.constant([[ - 1. / 5, 20. / 5, 300. / 6, 0., - np.log(9. / 3.), - np.log(12. / 6.), - 0.5, - ]], dtype=tf.float32) # pyformat: disable - predicted_bboxes = utils_3d.ResidualsToBBoxes(anchor_bboxes, residuals) - - with self.session(): - actual_predicted_bboxes = self.evaluate(predicted_bboxes) - self.assertAllClose(actual_predicted_bboxes, expected_predicted_bboxes) - - def testResidualsToBBoxesNegPiToPi(self): - utils_3d = detection_3d_lib.Utils3D() - - anchor_bboxes = tf.constant( - [[1, 2, 3, 4, 3, 6, 0.2], [1, 2, 3, 4, 3, 6, -0.2]], dtype=tf.float32) - expected_predicted_bboxes = np.asarray( - [[2, 22, 303, 4, 9, 12, -np.pi + 0.2], - [2, 22, 303, 4, 9, 12, np.pi - 0.2]]) - - residuals = tf.constant([ - [1. / 5, 20. / 5, 300. / 6, 0., - np.log(9. / 3.), np.log(12. / 6.), np.pi], - [1. / 5, 20. / 5, 300. / 6, 0., - np.log(9. / 3.), np.log(12. / 6.), -np.pi] - ], dtype=tf.float32) # pyformat: disable - predicted_bboxes = utils_3d.ResidualsToBBoxes( - anchor_bboxes, residuals, min_angle_rad=-np.pi, max_angle_rad=np.pi) - - with self.session(): - actual_predicted_bboxes = self.evaluate(predicted_bboxes) - self.assertAllClose(actual_predicted_bboxes, expected_predicted_bboxes) - - def testZeroResiduals(self): - utils_3d = detection_3d_lib.Utils3D() - - anchor_bboxes = tf.constant([[1, 2, 3, 4, 3, 6, 0]], dtype=tf.float32) - expected_predicted_bboxes = np.asarray([[1, 2, 3, 4, 3, 6, 0]]) - - residuals = tf.zeros((1, 7)) - predicted_bboxes = utils_3d.ResidualsToBBoxes(anchor_bboxes, residuals) - - with self.session(): - actual_predicted_bboxes = self.evaluate(predicted_bboxes) - self.assertAllClose(actual_predicted_bboxes, expected_predicted_bboxes) - - def testResidualsToBBoxPhiFloorMod(self): - utils_3d = detection_3d_lib.Utils3D() - - anchor_bboxes = tf.constant([[1, 2, 3, 4, 3, 6, np.pi]], dtype=tf.float32) - - # We expected the returned phi value to be floormod w.r.t. pi. - expected_predicted_bboxes = np.asarray([[1, 2, 3, 4, 3, 6, 1.]]) - - residuals = tf.constant([[0, 0, 0, 0, 0, 0, 1.0]], dtype=tf.float32) - predicted_bboxes = utils_3d.ResidualsToBBoxes( - anchor_bboxes, residuals, min_angle_rad=0.0) - - with self.session(): - actual_predicted_bboxes = self.evaluate(predicted_bboxes) - self.assertAllClose(actual_predicted_bboxes, expected_predicted_bboxes) - - def testNMSIndices(self): - utils_3d = detection_3d_lib.Utils3D() - - # Create three anchor boxes, two largely overlapping and one - # not overlapping with either. - # - # Set a batch size of 1 and use the Batched version to test - # both functions. - anchor_bboxes = tf.constant( - [[[1, 2, 3, 4, 3, 6, 0.], [1, 2, 2, 4, 3, 6, 0.], - [10, 20, 30, 4, 3, 6, 0.]]], - dtype=tf.float32) - - # Treat them all as high scores. - scores = tf.constant([[0.7, 0.8, 0.6]]) - - with self.session(): - nms_indices, valid_mask = utils_3d.BatchedNMSIndices( - anchor_bboxes, scores) - indices, mask = self.evaluate([nms_indices, valid_mask]) - # One box is filtered out. - self.assertEqual(2, np.sum(mask)) - # The two boxes that remain are the second one (because of its higher - # score) and the last one (which overlaps with nothing). - self.assertAllEqual([[1, 2, 0]], indices) - - # Flip the scores; expect the first box to be chosen instead. - # Change the last box's threshold to be 0.0, so that the - # default setting for the score threshold filters it out too. - scores_2 = tf.constant([[0.8, 0.7, 0.0]]) - nms_indices, valid_mask = utils_3d.BatchedNMSIndices( - anchor_bboxes, scores_2) - indices, mask = self.evaluate([nms_indices, valid_mask]) - self.assertEqual(1, np.sum(mask)) - self.assertAllEqual([[0, 0, 0]], indices) - - def testOrientedNMSIndices(self): - utils_3d = detection_3d_lib.Utils3D() - - # Assignments and IoU scores calculated offline. - bboxes_data = tf.constant( - [[ - [10.35, 8.429, -1.003, 3.7, 1.64, 1.49, 1.582], - [10.35, 8.429, -1.003, 3.7, 1.64, 1.49, 0.0], # box 0 rotated - [11.5, 8.429, -1.003, 3.7, 1.64, 1.49, 1.0], # Rotated to overlap - [13.01, 8.149, -0.953, 4.02, 1.55, 1.52, 1.592], - [13.51, 8.39, -1.0, 4.02, 1.55, 1.52, 1.592], # Slight translation - [13.51, 8.39, -1.0, 1.0, 1.0, 1.52, 1.592], # Smaller box - [13.51, 8.39, -1.0, 1.0, 1.0, 1.52, 1.9], # Smaller box - ]], - dtype=tf.float32) - - # Notes on the data: - # Lets say we have 3 classes and a thresh of 0.1 - # Keep box [0, 3] for class 0 - # Keep box [6] only for class 1 - # Keep box [2] for class 2 - scores_data = tf.constant([[ - [0.9, 0.1, 0.0], - [0.89, 0.1, 0.01], - [0.5, 0.01, 0.49], - [0.8, 0.1, 0.1], - [0.79, 0.11, 0.2], - [0.2, 0.8, 0.1], - [0.1, 0.9, 0.0], - ]], - dtype=tf.float32) - - with self.session(): - outputs = utils_3d.BatchedOrientedNMSIndices( - bboxes_data, - scores_data, - nms_iou_threshold=0.1, - score_threshold=0.3, - max_boxes_per_class=5) - indices, scores, valid_mask = self.evaluate(outputs) - - class_masks = [ - valid_mask[0, cls_idx, :].astype(np.bool) for cls_idx in range(3) - ] - # Check the correct number of valid results per class - self.assertEqual(class_masks[0].sum(), 2) - self.assertEqual(class_masks[1].sum(), 1) - self.assertEqual(class_masks[2].sum(), 1) - - # Check the results for each class - self.assertAllEqual(indices[0, 0, class_masks[0]], [0, 3]) - self.assertAllClose(scores[0, 0, class_masks[0]], [0.9, 0.8]) - - self.assertAllEqual(indices[0, 1, class_masks[1]], [6]) - self.assertAllClose(scores[0, 1, class_masks[1]], [0.9]) - - self.assertAllEqual(indices[0, 2, class_masks[2]], [2]) - self.assertAllClose(scores[0, 2, class_masks[2]], [0.49]) - - # Use a list of score thresholds instead - outputs = utils_3d.BatchedOrientedNMSIndices( - bboxes_data, - scores_data, - nms_iou_threshold=[0.1, 0.1, 0.1], - score_threshold=[0.899, 0.5, 0.3], - max_boxes_per_class=5) - indices, scores, valid_mask = self.evaluate(outputs) - - class_masks = [ - valid_mask[0, cls_idx, :].astype(np.bool) for cls_idx in range(3) - ] - # Check the correct number of valid results per class - self.assertEqual(class_masks[0].sum(), 1) - self.assertEqual(class_masks[1].sum(), 1) - self.assertEqual(class_masks[2].sum(), 1) - - # Check the results for each class - self.assertAllEqual(indices[0, 0, class_masks[0]], [0]) - self.assertAllClose(scores[0, 0, class_masks[0]], [0.9]) - - self.assertAllEqual(indices[0, 1, class_masks[1]], [6]) - self.assertAllClose(scores[0, 1, class_masks[1]], [0.9]) - - self.assertAllEqual(indices[0, 2, class_masks[2]], [2]) - self.assertAllClose(scores[0, 2, class_masks[2]], [0.49]) - - def testRandomPadOrTrimToTrim(self): - points = tf.constant([[1., 2., 3.], [4., 5., 6.], [7., 8., 9.], - [10., 11., 12.]]) - features = tf.constant([[100.], [200.], [300.], [400.]]) - - points, features = detection_3d_lib.RandomPadOrTrimTo([points, features], - 2, - seed=123)[0] - with self.session(): - points_np, features_np = self.evaluate([points, features]) - # Slicing choose a random 2 points. - self.assertAllClose([[1., 2., 3.], [10., 11., 12.]], points_np) - self.assertAllClose([[100.], [400.]], features_np) - - def testRandomPadOrTrimToPad(self): - points = tf.constant([[1., 2., 3.], [4., 5., 6.], [7., 8., 9.], - [10., 11., 12.]]) - features = tf.constant([[100.], [200.], [300.], [400.]]) - - points, features = detection_3d_lib.RandomPadOrTrimTo([points, features], - 10, - seed=123)[0] - with self.session(): - points_np, features_np = self.evaluate([points, features]) - # Padding repeats a random set of points. - self.assertAllClose([[1., 2., 3.], [1., 2., 3.], [10., 11., 12.], - [7., 8., 9.], [7., 8., 9.], [4., 5., 6.]], - points_np[4:]) - self.assertAllClose([[100.], [100.], [400.], [300.], [300.], [200.]], - features_np[4:]) - - def testRandomPadOrTrimToEmpty(self): - points = tf.constant([[1., 2., 3.]]) - features = tf.constant([[100.]]) - points, features = detection_3d_lib.RandomPadOrTrimTo( - [points[0:0], features[0:0]], 10, seed=123)[0] - with self.session(): - points_np, features_np = self.evaluate([points, features]) - self.assertAllClose(points_np, np.zeros(shape=(10, 3))) - self.assertAllClose(features_np, np.zeros(shape=(10, 1))) - - def testCornersToImagePlane(self): - utils_3d = detection_3d_lib.Utils3D() - batch = 4 - num_boxes = 50 - - corners = tf.random.uniform([batch, num_boxes, 8, 3]) - velo_to_image_plane = tf.random.uniform([batch, 3, 4]) - corners_to_image_plane = utils_3d.CornersToImagePlane( - corners, velo_to_image_plane) - self.assertEqual([batch, num_boxes, 8, 2], corners_to_image_plane.shape) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/detection_3d_metrics.py b/lingvo/tasks/car/detection_3d_metrics.py deleted file mode 100644 index b5b9d9d8..00000000 --- a/lingvo/tasks/car/detection_3d_metrics.py +++ /dev/null @@ -1,546 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Metrics for 3D detection problems.""" - -from lingvo import compat as tf -from lingvo.core import metrics -from lingvo.core import plot -from lingvo.core import py_utils -from lingvo.tasks.car import summary -from lingvo.tasks.car import transform_util -import matplotlib.colors as matplotlib_colors -import matplotlib.patches as matplotlib_patches -import matplotlib.patheffects as path_effects -import numpy as np -import PIL.Image as Image -import PIL.ImageDraw as ImageDraw -import PIL.ImageFont as ImageFont -from tensorboard.plugins.mesh import summary as mesh_summary - - -class TopDownVisualizationMetric(metrics.BaseMetric): - """Top-down detection visualization, expecting 3D laser points and 2D bboxes. - - Updates to this metric is expected to be `.NestedMap` containing: - - visualization_labels: [N, B1] int tensor containing visualization labels. - - predicted_bboxes: [N, B1, 5] float tensor containing predicted 2D bboxes - each with (x, y, dx, dy, phi). - - visualization_weights: [N, B1] float tensor containing weights for each - prediction. predictions with 0 weight will not be drawn. - - points_xyz: [N, P, 3] float tensor containing points (x, y, z) coordinates. - - points_padding: [N, P] tensor containing 1 if the point is a padded point. - - gt_bboxes_2d: [N, B2, 5] float tensor containing ground-truth 2D bboxes. - - gt_bboxes_2d_weights: [N, B2] float tensor containing weights for each - ground-truth. predictions with 0 weight will not be drawn. The - ground-truth mask can be used here. - - labels: [N, B2] int tensor containing ground-truth labels. - - difficulties: [N, B2]: int tensor containing the difficulty levels of - each groundtruth box. - - Default parameters visualize the area around the car, with the car centered in - the image, over a 32m x 48m range. - - Ground-truth boxes will be drawn with color=cyan (see DrawBBoxesOnImages for - details). Predicted boxes will be drawn with a color from the PIL color list, - with a different color per class. - """ - - def __init__(self, - top_down_transform, - class_id_to_name=None, - image_height=1536, - image_width=1024, - figsize=None, - ground_removal_threshold=-1.35, - sampler_num_samples=8): - """Initialize TopDownVisualizationMetric. - - Args: - top_down_transform: transform_util.Transform object that specifies - how to transform a coordinate in the world coordinate to the top - down projection. See documentation for - transform_util.MakeCarToImageTransform for more details on - configuration. - class_id_to_name: Dictionary mapping from class id to name. - image_height: int image height. - image_width: int image width. - figsize: (w, h) float tuple. This is the size of the rendered figure in - inches. A dpi=100 is used in plot.Image; note that the axes and title - will take up space in the final rendering. If None, this will default to - (image_width / 100 * 1.5, image_height / 100 * 1.5). - ground_removal_threshold: Floating point value used to color ground points - differently. Defaults to -1.35 which happens to work well for KITTI. - sampler_num_samples: Number of batches to keep for visualizing. - """ - self._class_id_to_name = class_id_to_name or {} - self._image_width = image_width - self._image_height = image_height - - figsize = figsize or (image_width / 100. * 1.5, image_height / 100. * 1.5) - self._figsize = figsize - - self._ground_removal_threshold = ground_removal_threshold - self._sampler = py_utils.UniformSampler(num_samples=sampler_num_samples) - self._top_down_transform = top_down_transform - self._summary = None - - def Update(self, decoded_outputs): - """Add top down visualization to summaries. - - Args: - decoded_outputs: A `.NestedMap` containing the following fields: - visualization_labels, predicted_bboxes, visualization_weights, - points_xyz, points_padding, gt_bboxes_2d, gt_bboxes_2d_weights, and - labels. - """ - self._sampler.Add(decoded_outputs) - # Invalidate cache. - self._summary = None - - def _XYWHToExtrema(self, bboxes): - """Convert from x, y, dx, dy to extrema ymin, xmin, ymax, xmax.""" - mtrix = np.array([ - # x y dx dy - [0.0, 1.0, 0.0, -.5], # ymin - [1.0, 0.0, -.5, 0.0], # xmin - [0.0, 1.0, 0.0, 0.5], # ymax - [1.0, 0.0, 0.5, 0.0], # xmax - ]).T - bboxes = bboxes.copy() - bboxes[..., :4] = np.matmul(bboxes[..., :4], mtrix) - return bboxes - - def _DrawLasers(self, images, points_xyz, points_padding, transform): - """Draw laser points.""" - for batch_idx in range(images.shape[0]): - for points_idx in range(points_xyz.shape[1]): - if points_padding[batch_idx, points_idx] == 0: - x, y, z = points_xyz[batch_idx, points_idx, :3] - tx, ty, _ = transform_util.TransformPoint(transform, x, y, z) - if tx < 0 or ty < 0 or tx >= images.shape[2] or ty >= images.shape[1]: - continue - - # Drop ground points from visualization. - if z < self._ground_removal_threshold: - # Brown out the color for ground points. - color = (64, 48, 48) - else: - color = (255, 255, 255) - - images[batch_idx, int(ty), int(tx), :] = color - - def Summary(self, name): - self._EvaluateIfNecessary(name) - return self._summary - - def _EvaluateIfNecessary(self, name): - """Create a top down image summary, if not already created.""" - if self._summary is not None: - return - - tf.logging.info('Generating top down summary.') - ret = tf.Summary() - - transform = self._top_down_transform - - for batch_idx, batch_sample in enumerate(self._sampler.samples): - batch_size = batch_sample.labels.shape[0] - visualization_labels = batch_sample.visualization_labels - predicted_bboxes = batch_sample.predicted_bboxes - visualization_weights = batch_sample.visualization_weights - points_xyz = batch_sample.points_xyz - points_padding = batch_sample.points_padding - gt_bboxes_2d = batch_sample.gt_bboxes_2d - gt_bboxes_2d_weights = batch_sample.gt_bboxes_2d_weights - labels = batch_sample.labels - difficulties = batch_sample.difficulties - source_ids = batch_sample.source_ids - - # Create base images for entire batch that we will update. - images = np.zeros([batch_size, self._image_height, self._image_width, 3], - dtype=np.uint8) - - # Draw lasers first, so that bboxes can be on top. - self._DrawLasers(images, points_xyz, points_padding, transform) - - # Draw ground-truth bboxes. - gt_bboxes_2d = np.where( - np.expand_dims(gt_bboxes_2d_weights > 0, -1), gt_bboxes_2d, - np.zeros_like(gt_bboxes_2d)) - transformed_gt_bboxes_2d = summary.TransformBBoxesToTopDown( - gt_bboxes_2d, transform) - - summary.DrawBBoxesOnImages( - images, - transformed_gt_bboxes_2d, - gt_bboxes_2d_weights, - labels, - self._class_id_to_name, - groundtruth=True) - - # Draw predicted bboxes. - predicted_bboxes = np.where( - np.expand_dims(visualization_weights > 0, -1), predicted_bboxes, - np.zeros_like(predicted_bboxes)) - transformed_predicted_bboxes = summary.TransformBBoxesToTopDown( - predicted_bboxes, transform) - - summary.DrawBBoxesOnImages( - images, - transformed_predicted_bboxes, - visualization_weights, - visualization_labels, - self._class_id_to_name, - groundtruth=False) - - # Draw the difficulties on the image. - self.DrawDifficulty(images, transformed_gt_bboxes_2d, - gt_bboxes_2d_weights, difficulties) - - for idx in range(batch_size): - source_id = source_ids[idx] - - def AnnotateImage(fig, axes, source_id=source_id): - """Add source_id to image.""" - del fig - # Draw in top middle of image. - text = axes.text( - 500, - 15, - source_id, - fontsize=16, - color='blue', - fontweight='bold', - horizontalalignment='center') - text.set_path_effects([ - path_effects.Stroke(linewidth=3, foreground='lightblue'), - path_effects.Normal() - ]) - - image_summary = plot.Image( - name='{}/{}/{}'.format(name, batch_idx, idx), - aspect='equal', - figsize=self._figsize, - image=images[idx, ...], - setter=AnnotateImage) - ret.value.extend(image_summary.value) - - tf.logging.info('Done generating top down summary.') - self._summary = ret - - def DrawDifficulty(self, images, gt_bboxes, gt_box_weights, difficulties): - """Draw the difficulty values on each ground truth box.""" - batch_size = np.shape(images)[0] - try: - font = ImageFont.truetype('arial.ttf', size=20) - except IOError: - font = ImageFont.load_default() - - for batch_id in range(batch_size): - image = images[batch_id, :, :, :] - original_image = image - image = Image.fromarray(np.uint8(original_image)).convert('RGB') - draw = ImageDraw.Draw(image) - difficulty_vector = difficulties[batch_id] - box_data = gt_bboxes[batch_id] - - for box_id in range(box_data.shape[0]): - box_weight = gt_box_weights[batch_id, box_id] - if box_weight == 0: - continue - center_x = box_data[box_id, 0] - center_y = box_data[box_id, 1] - difficulty_value = str(difficulty_vector[box_id]) - - # Draw a rectangle background slightly larger than the text. - text_width, text_height = font.getsize(difficulty_value) - draw.rectangle( - [(center_x - text_width / 1.8, center_y - text_height / 1.8), - (center_x + text_width / 1.8, center_y + text_height / 1.8)], - fill='darkcyan') - - # Center the text in the rectangle - draw.text((center_x - text_width / 2, center_y - text_height / 2), - str(difficulty_value), - fill='lightcyan', - font=font) - np.copyto(original_image, np.array(image)) - - -class WorldViewer(metrics.BaseMetric): - """World Viewer for 3d point cloud scenes.""" - # Defines the maximum hue range for point cloud colorization by distance. - _MAX_HUE = 0.65 - - # Distance from car after which we consider all points equally far. - _MAX_DISTANCE_METERS = 40. - - def __init__(self, sampler_num_samples=8): - """Init.""" - self._sampler = py_utils.UniformSampler(num_samples=sampler_num_samples) - self._summary = None - - def Update(self, decoded_outputs): - """Add point cloud mesh data to be summarized. - - Args: - decoded_outputs: A `.NestedMap` containing the following fields: - visualization_labels, predicted_bboxes, visualization_weights, - points_xyz, points_padding, gt_bboxes_2d, gt_bboxes_2d_weights, and - labels. - """ - self._sampler.Add(decoded_outputs) - # Invalidate cache. - self._summary = None - - def Summary(self, name): - self._EvaluateIfNecessary(name) - return self._summary - - def _EvaluateIfNecessary(self, name): - """Create a mesh summary, if not already created.""" - if self._summary is not None: - return - - summ = None - tf.logging.info('Generating mesh summary.') - for i, batch_sample in enumerate(self._sampler.samples): - points_xyz = batch_sample.points_xyz[i:i + 1] - points_padding = batch_sample.points_padding[i:i + 1] - points_mask = (1. - points_padding).astype(bool) - # Apply mask and expand to include a batch dimension. - points_xyz = points_xyz[points_mask][np.newaxis, ...] - - # Compute colors based off distance from car. - distance = np.sqrt(points_xyz[0, :, 0]**2 + points_xyz[0, :, 1]**2 + - points_xyz[0, :, 2]**2) - # Normalize by some max distance beyond which we don't distinguish - # distance. - max_distance = np.ones_like(distance) * WorldViewer._MAX_DISTANCE_METERS - distance = np.minimum(max_distance, distance) - scale = (max_distance - distance) / max_distance - - # Convert to RGB. - hue = np.minimum(WorldViewer._MAX_HUE, scale)[..., np.newaxis] - # Invert hue so red is closer. - hue = WorldViewer._MAX_HUE - hue - s, v = np.ones_like(hue), np.ones_like(hue) - hsv = np.hstack([hue, s, v]) - rgb = matplotlib_colors.hsv_to_rgb(hsv) - colors = np.minimum(255., rgb * 255.).astype(np.uint8) - colors = colors[np.newaxis, ...] - summ = mesh_summary.pb( - '{}/point_cloud/{}'.format(name, i), - vertices=points_xyz, - colors=colors, - faces=None) - # At the moment, only one scene summary is supported; writing - # more makes the TensorBoard mesh visualizer hang. - break - - if summ: - self._summary = summ - - -class CameraVisualization(metrics.BaseMetric): - """Camera detection visualization. - - Visualizes a camera image and predicted bounding boxes on top - of the image. - - Updates to this metric is expected to be `.NestedMap` containing: - - camera_images: [N, W, H, 3] float tensor containing camera image data. - - bbox_corners: [N, B1, 8, 2] float tensor containing bounding box corners. - For each batch (N), for each box B, there are 8 corners, each with - an X and Y value. - - bbox_scores: [N, B1] float tensor containing predicted box scores. - - """ - - def __init__(self, - figsize=(15, 15), - bbox_score_threshold=0.01, - sampler_num_samples=8, - draw_3d_boxes=True): - """Initialize CameraVisualization. - - Args: - figsize: (w, h) float tuple. This is the size of the rendered figure in - inches. A dpi=100 is used in plot.Image; note that the axes and title - will take up space in the final rendering. If None, this will default to - (image_width / 100 * 1.5, image_height / 100 * 1.5). - bbox_score_threshold: The threshold over which bboxes will be drawn on the - image. - sampler_num_samples: Number of batches to keep for visualizing. - draw_3d_boxes: Whether to draw 2d or 3d bounding boxes. 3d bounding - boxes depict the 8 corners of the bounding box, whereas the 2d - bounding boxes depict the extrema x and y dimensions of the boxes - on the image plane. - """ - self._figsize = figsize - self._bbox_score_threshold = bbox_score_threshold, - self._sampler = py_utils.UniformSampler(num_samples=sampler_num_samples) - self._draw_3d_boxes = draw_3d_boxes - self._summary = None - - def Update(self, decoded_outputs): - self._sampler.Add(decoded_outputs) - # Invalidate cache. - self._summary = None - - def Summary(self, name): - self._EvaluateIfNecessary(name) - return self._summary - - def _EvaluateIfNecessary(self, name): - """Create a camera image summary if not already created.""" - if self._summary is not None: - return - - ret = tf.Summary() - - for sample_idx, sample in enumerate(self._sampler.samples): - batch_size = sample.camera_images.shape[0] - - for batch_idx in range(batch_size): - image = sample.camera_images[batch_idx] - - # [num bboxes, 8, 2]. - bbox_corners = sample.bbox_corners[batch_idx] - - # [num_bboxes] - bbox_scores = sample.bbox_scores[batch_idx] - - def Draw3DBoxes(fig, - axes, - bbox_corners=bbox_corners, - bbox_scores=bbox_scores): - """Draw 3d bounding boxes.""" - del fig - for bbox_id in range(bbox_corners.shape[0]): - # Skip visualizing low-scoring boxes. - bbox_score = bbox_scores[bbox_id] - if bbox_score < self._bbox_score_threshold: - continue - bbox_data = bbox_corners[bbox_id] - - # Draw the score of each box. - # - # Turn score into an integer for better display. - center_x = np.mean(bbox_data[:, 0]) - center_y = np.mean(bbox_data[:, 1]) - bbox_score = int(bbox_score * 100) - text = axes.text( - center_x, - center_y, - bbox_score, - fontsize=12, - color='red', - fontweight='bold') - text.set_bbox(dict(facecolor='yellow', alpha=0.4)) - - # The BBoxToCorners function produces the points - # in a deterministic order, which we use to draw - # the faces of the polygon. - # - # The first 4 points are the "top" of the bounding box. - # The second 4 points are the "bottom" of the bounding box. - # - # We then draw the last 4 connecting points by choosing - # two of the connecting faces in the right order. - face_points = [] - face_points += [[ - bbox_data[0, :], bbox_data[1, :], bbox_data[2, :], - bbox_data[3, :] - ]] - face_points += [[ - bbox_data[4, :], bbox_data[5, :], bbox_data[6, :], - bbox_data[7, :] - ]] - face_points += [[ - bbox_data[1, :], bbox_data[2, :], bbox_data[6, :], - bbox_data[5, :] - ]] - face_points += [[ - bbox_data[0, :], bbox_data[3, :], bbox_data[7, :], - bbox_data[4, :] - ]] - for face in face_points: - # Each face is a list of 4 x,y points - face_xy = np.array(face) - axes.add_patch( - matplotlib_patches.Polygon( - face_xy, closed=True, edgecolor='red', facecolor='none')) - - def Draw2DBoxes(fig, - axes, - bbox_corners=bbox_corners, - bbox_scores=bbox_scores): - """Draw 2d boxes on the figure.""" - del fig - # Extract the 2D extrema of each bbox and the max score - for bbox_id in range(bbox_corners.shape[0]): - # Skip visualizing low-scoring boxes. - bbox_score = bbox_scores[bbox_id] - if bbox_score < self._bbox_score_threshold: - continue - bbox_data = bbox_corners[bbox_id] - - ymin = np.min(bbox_data[:, 1]) - xmin = np.min(bbox_data[:, 0]) - ymax = np.max(bbox_data[:, 1]) - xmax = np.max(bbox_data[:, 0]) - height = ymax - ymin - width = xmax - xmin - # Turn score into an integer for better display. - bbox_score = int(bbox_score * 100) - text = axes.text( - xmin, - ymin, - bbox_score, - fontsize=12, - color='red', - fontweight='bold') - text.set_bbox(dict(facecolor='yellow', alpha=0.4)) - axes.add_patch( - matplotlib_patches.Rectangle((xmin, ymin), - width, - height, - edgecolor='red', - facecolor='none')) - - # For each image, draw the boxes on that image. - draw_fn = Draw3DBoxes if self._draw_3d_boxes else Draw2DBoxes - image_summary = plot.Image( - name='{}/{}/{}'.format(name, sample_idx, batch_idx), - aspect='equal', - figsize=self._figsize, - image=image, - setter=draw_fn) - ret.value.extend(image_summary.value) - self._summary = ret diff --git a/lingvo/tasks/car/detection_3d_metrics_test.py b/lingvo/tasks/car/detection_3d_metrics_test.py deleted file mode 100644 index 8a43445f..00000000 --- a/lingvo/tasks/car/detection_3d_metrics_test.py +++ /dev/null @@ -1,109 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for detection_3d_metrics.""" - -from lingvo import compat as tf -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.tasks.car import detection_3d_metrics -from lingvo.tasks.car import transform_util -import numpy as np - - -class Detection3dMetricsTest(test_utils.TestCase): - - def testTopDownVisualizationMetric(self): - top_down_transform = transform_util.MakeCarToImageTransform( - pixels_per_meter=32., - image_ref_x=512., - image_ref_y=1408., - flip_axes=True) - metric = detection_3d_metrics.TopDownVisualizationMetric(top_down_transform) - - batch_size = 4 - num_preds = 10 - num_gt = 12 - num_points = 128 - - visualization_labels = np.random.randint(0, 2, (batch_size, num_preds)) - predicted_bboxes = np.random.rand(batch_size, num_preds, 5) - visualization_weights = np.abs(np.random.rand(batch_size, num_preds)) - - labels = np.random.randint(0, 2, (batch_size, num_gt)) - gt_bboxes_2d = np.random.rand(batch_size, num_gt, 5) - gt_bboxes_2d_weights = np.abs(np.random.rand(batch_size, num_gt)) - difficulties = np.random.randint(0, 3, (batch_size, num_gt)) - - points_xyz = np.random.rand(batch_size, num_points, 3) - points_padding = np.random.randint(0, 2, (batch_size, num_points)) - source_ids = np.full([batch_size], '012346') - - metric.Update( - py_utils.NestedMap({ - 'visualization_labels': visualization_labels, - 'predicted_bboxes': predicted_bboxes, - 'visualization_weights': visualization_weights, - 'labels': labels, - 'gt_bboxes_2d': gt_bboxes_2d, - 'gt_bboxes_2d_weights': gt_bboxes_2d_weights, - 'points_xyz': points_xyz, - 'points_padding': points_padding, - 'difficulties': difficulties, - 'source_ids': source_ids, - })) - - _ = metric.Summary('test') - - def testCameraVisualization(self): - metric = detection_3d_metrics.CameraVisualization() - - batch_size = 4 - num_preds = 10 - - images = np.random.rand(batch_size, 512, 1024, 3) - bbox_corners = np.random.rand(batch_size, num_preds, 8, 2) - bbox_scores = np.random.rand(batch_size, num_preds) - - metric.Update( - py_utils.NestedMap({ - 'camera_images': images, - 'bbox_corners': bbox_corners, - 'bbox_scores': bbox_scores - })) - - # Test that the metric runs. - _ = metric.Summary('test') - - def testMesh(self): - metric = detection_3d_metrics.WorldViewer() - - batch_size = 4 - num_points = 128 - points_xyz = np.random.rand(batch_size, num_points, 3) * 40. - points_padding = np.random.randint(0, 2, (batch_size, num_points)) - - metric.Update( - py_utils.NestedMap({ - 'points_xyz': points_xyz, - 'points_padding': points_padding, - })) - - # Test that the metric runs. - _ = metric.Summary('test') - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/detection_decoder.py b/lingvo/tasks/car/detection_decoder.py deleted file mode 100644 index 41b26d23..00000000 --- a/lingvo/tasks/car/detection_decoder.py +++ /dev/null @@ -1,220 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Functions to help with decoding detector model outputs.""" - -from lingvo import compat as tf -from lingvo.core import py_utils -from lingvo.tasks.car import detection_3d_lib - - -def DecodeWithNMS(predicted_bboxes, - classification_scores, - nms_iou_threshold, - score_threshold, - max_boxes_per_class=None, - use_oriented_per_class_nms=False): - """Perform NMS on predicted bounding boxes / associated logits. - - Args: - predicted_bboxes: [batch_size, num_boxes, 7] float Tensor containing - predicted bounding box coordinates. - classification_scores: [batch_size, num_boxes, num_classes] float Tensor - containing predicted classification scores for each box. - nms_iou_threshold: IoU threshold to use when determining whether two boxes - overlap for purposes of suppression. Either a float or a list of len - num_classes. - score_threshold: The score threshold passed to NMS that allows NMS to - quickly ignore irrelevant boxes. Either a float or a list of len - num_classes. - max_boxes_per_class: The maximum number of boxes per example to emit. - If None, this value is set to num_boxes from the shape of - predicted_bboxes. - use_oriented_per_class_nms: Whether to use the oriented per class NMS - or treat everything as one class and having no orientation. - - Returns: - bbox_indices: Indices of the boxes selected after NMS. Tensor of shape - [batch_size, num_classes, max_boxes_per_class] if per class NMS is used. - If single class NMS, this will be of shape [batch_size, - max_boxes_per_class]. - predicted_bboxes: Filtered bboxes after NMS of shape - [batch_size, num_classes, max_boxes_per_class, 7]. - bbox_scores: A float32 Tensor with the score for each box of shape - [batch_size, num_classes, max_boxes_per_class]. - valid_mask: A float32 Tensor with 1/0 values indicating the validity of - each box. 1 indicates valid, and 0 invalid. Tensor of shape - [batch_size, num_classes, max_boxes_per_class]. - """ - if use_oriented_per_class_nms: - nms_fn = _MultiClassOrientedDecodeWithNMS - else: - nms_fn = _SingleClassDecodeWithNMS - - return nms_fn( - predicted_bboxes=predicted_bboxes, - classification_scores=classification_scores, - nms_iou_threshold=nms_iou_threshold, - score_threshold=score_threshold, - max_boxes_per_class=max_boxes_per_class) - - -def _MultiClassOrientedDecodeWithNMS(predicted_bboxes, - classification_scores, - nms_iou_threshold, - score_threshold, - max_boxes_per_class=None): - """Perform Oriented Per Class NMS on predicted bounding boxes / logits. - - Args: - predicted_bboxes: [batch_size, num_boxes, 7] float Tensor containing - predicted bounding box coordinates. - classification_scores: [batch_size, num_boxes, num_classes] float Tensor - containing predicted classification scores for each box. - nms_iou_threshold: IoU threshold to use when determining whether two boxes - overlap for purposes of suppression. Either a float or a list of len - num_classes. - score_threshold: The score threshold passed to NMS that allows NMS to - quickly ignore irrelevant boxes. Either a float or a list of len - num_classes. It is strongly recommended that the score for non-active - classes (like background) be set to 1 so they are discarded. - max_boxes_per_class: The maximum number of boxes per example to emit. If - None, this value is set to num_boxes from the shape of predicted_bboxes. - - Returns: - bbox_indices: Indices of the boxes selected after NMS. Tensor of shape - [batch_size, num_classes, max_boxes_per_class]. - predicted_bboxes: Filtered bboxes after NMS of shape - [batch_size, num_classes, max_boxes_per_class, 7]. - bbox_scores: A float32 Tensor with the score for each box of shape - [batch_size, num_classes, max_boxes_per_class]. - valid_mask: A float32 Tensor with 1/0 values indicating the validity of - each box. 1 indicates valid, and 0 invalid. Tensor of shape - [batch_size, num_classes, max_boxes_per_class]. - """ - utils_3d = detection_3d_lib.Utils3D() - predicted_bboxes = py_utils.HasShape(predicted_bboxes, [-1, -1, 7]) - batch_size, num_predicted_boxes, _ = py_utils.GetShape(predicted_bboxes) - classification_scores = py_utils.HasShape( - classification_scores, [batch_size, num_predicted_boxes, -1]) - _, _, num_classes = py_utils.GetShape(classification_scores) - - if max_boxes_per_class is None: - max_boxes_per_class = num_predicted_boxes - - # Compute NMS for every sample in the batch. - bbox_indices, bbox_scores, valid_mask = utils_3d.BatchedOrientedNMSIndices( - predicted_bboxes, - classification_scores, - nms_iou_threshold=nms_iou_threshold, - score_threshold=score_threshold, - max_boxes_per_class=max_boxes_per_class) - - # TODO(bencaine): Consider optimizing away the tf.tile or make upstream - # changes to make predicted boxes include a class dimension. - # Get the original box for each index selected by NMS. - predicted_bboxes = tf.tile(predicted_bboxes[:, tf.newaxis, :, :], - [1, num_classes, 1, 1]) - predicted_bboxes = tf.array_ops.batch_gather(predicted_bboxes, bbox_indices) - return bbox_indices, predicted_bboxes, bbox_scores, valid_mask - - -def _SingleClassDecodeWithNMS(predicted_bboxes, - classification_scores, - nms_iou_threshold, - score_threshold, - max_boxes_per_class=None): - """Perform NMS on predicted bounding boxes / associated logits. - - Args: - predicted_bboxes: [batch_size, num_boxes, 7] float Tensor containing - predicted bounding box coordinates. - classification_scores: [batch_size, num_boxes, num_classes] float Tensor - containing predicted classification scores for each box. - nms_iou_threshold: IoU threshold to use when determining whether two boxes - overlap for purposes of suppression. - score_threshold: The score threshold passed to NMS that allows NMS to - quickly ignore irrelevant boxes. - max_boxes_per_class: The maximum number of boxes per example to emit. If - None, this value is set to num_boxes from the shape of predicted_bboxes. - - Returns: - nms_indices: Indices of the boxes selected after NMS. Tensor of shape - [batch_size, num_classes, max_boxes_per_class]. - predicted_bboxes: Filtered bboxes after NMS of shape - [batch_size, num_classes, max_boxes_per_class, 7]. - bbox_scores: A float32 Tensor with the score for each box of shape - [batch_size, num_classes, max_boxes_per_class]. - valid_mask: A float32 Tensor with 1/0 values indicating the validity of - each box. 1 indicates valid, and 0 invalid. Tensor of shape - [batch_size, num_classes, max_boxes_per_class]. - """ - utils_3d = detection_3d_lib.Utils3D() - predicted_bboxes = py_utils.HasShape(predicted_bboxes, [-1, -1, 7]) - batch_size, num_predicted_boxes, _ = py_utils.GetShape(predicted_bboxes) - classification_scores = py_utils.HasShape( - classification_scores, [batch_size, num_predicted_boxes, -1]) - _, _, num_classes = py_utils.GetShape(classification_scores) - - if not isinstance(nms_iou_threshold, float): - raise ValueError('Single class NMS only supports a scalar ' - '`nms_iou_threshold`.') - if not isinstance(score_threshold, float): - raise ValueError('Single class NMS only supports a scalar ' - '`score_threshold`.') - - if max_boxes_per_class is None: - max_boxes_per_class = num_predicted_boxes - - # TODO(jngiam): Change to be per-class bboxes, and hence, per-class NMS, and - # per-class thresholding. - # [batch, num_predicted_boxes] - nms_scores = tf.reduce_max(classification_scores, axis=-1) - - # Compute the most likely label by computing the highest class score from - # the output of the sigmoid. - likely_labels = tf.argmax(classification_scores, axis=-1) - - # When background is the most likely class for the box, mask out the scores - # of that box from NMS scoring so the background boxes don't dominate the - # NMS. - nms_scores *= tf.cast(likely_labels > 0, tf.float32) - - # Compute NMS for every sample in the batch. - nms_indices, valid_mask = utils_3d.BatchedNMSIndices( - predicted_bboxes, - nms_scores, - nms_iou_threshold=nms_iou_threshold, - score_threshold=score_threshold, - max_num_boxes=max_boxes_per_class) - - # Reorder the box data and logits according to NMS scoring. - predicted_bboxes = tf.array_ops.batch_gather(predicted_bboxes, nms_indices) - classification_scores = tf.array_ops.batch_gather(classification_scores, - nms_indices) - - # Now reformat the output of NMS to match the format of the - # MultiClassOrientedDecodeWithNMS, which outputs a per class NMS result. - # This takes the leading shape of - # [batch_size, num_classes, max_boxes_per_class] for all outputs, which - # means since this NMS is not class specific we need to tile the outputs - # num_classes times or reorder the data such that its [batch, num_classes]. - predicted_bboxes = tf.tile(predicted_bboxes[:, tf.newaxis, :, :], - [1, num_classes, 1, 1]) - classification_scores = tf.transpose(classification_scores, (0, 2, 1)) - classification_scores = py_utils.HasShape( - classification_scores, [batch_size, num_classes, max_boxes_per_class]) - valid_mask = tf.tile(valid_mask[:, tf.newaxis, :], [1, num_classes, 1]) - return nms_indices, predicted_bboxes, classification_scores, valid_mask diff --git a/lingvo/tasks/car/detection_decoder_test.py b/lingvo/tasks/car/detection_decoder_test.py deleted file mode 100644 index 9599045c..00000000 --- a/lingvo/tasks/car/detection_decoder_test.py +++ /dev/null @@ -1,120 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for detection_decoder.""" - -from lingvo import compat as tf -from lingvo.core import test_utils -from lingvo.tasks.car import detection_decoder - - -class DetectionDecoderTest(test_utils.TestCase): - """Tests for detection_decoder library.""" - - def testDecoderWithOrientedPerClassNMS(self): - batch_size = 4 - num_preds = 8 - num_classes = 10 - - # An example of setting the score threshold high and IOU threshold low - # for classes we don't care about - score_threshold = [1.0] * num_classes - score_threshold[1] = 0.05 - - nms_iou_threshold = [0.0] * num_classes - nms_iou_threshold[1] = 0.5 - - with tf.Graph().as_default(): - tf.random.set_seed(12345) - predicted_bboxes = tf.random.normal([batch_size, num_preds, 7]) - classification_scores = tf.random.uniform( - [batch_size, num_preds, num_classes], minval=0, maxval=1) - - idxs, bboxes, bbox_scores, valid_mask = detection_decoder.DecodeWithNMS( - predicted_bboxes, - classification_scores, - nms_iou_threshold=nms_iou_threshold, - score_threshold=score_threshold, - use_oriented_per_class_nms=True) - - with self.session(): - outputs = self.evaluate([ - predicted_bboxes, classification_scores, idxs, bboxes, bbox_scores, - valid_mask - ]) - (input_bboxes, input_scores, output_idxs, output_bboxes, output_scores, - mask) = outputs - - self.assertEqual((batch_size, num_preds, 7), input_bboxes.shape) - self.assertEqual((batch_size, num_classes, num_preds), - output_idxs.shape) - self.assertEqual((batch_size, num_classes, num_preds, 7), - output_bboxes.shape) - self.assertEqual((batch_size, num_preds, num_classes), - input_scores.shape) - self.assertEqual((batch_size, num_classes, num_preds), - output_scores.shape) - self.assertEqual((batch_size, num_classes, num_preds), mask.shape) - - # Assert that NMS did some kind of filtering for each class - for cls_idx in range(num_classes): - self.assertEqual( - mask[:, cls_idx, :].sum(), - (input_scores[:, :, cls_idx] > score_threshold[cls_idx]).sum()) - self.assertEqual( - mask[:, cls_idx, :].sum(), - (output_scores[:, cls_idx, :] > score_threshold[cls_idx]).sum()) - - def testDecoderSingleClassNMS(self): - batch_size = 4 - num_preds = 8 - num_classes = 10 - - score_threshold = 0.05 - nms_iou_threshold = 0.5 - with tf.Graph().as_default(): - tf.random.set_seed(12345) - predicted_bboxes = tf.random.normal([batch_size, num_preds, 7]) - classification_scores = tf.random.uniform( - [batch_size, num_preds, num_classes], minval=0, maxval=1) - - idxs, bboxes, bbox_scores, valid_mask = detection_decoder.DecodeWithNMS( - predicted_bboxes, - classification_scores, - nms_iou_threshold=nms_iou_threshold, - score_threshold=score_threshold, - use_oriented_per_class_nms=False) - - with self.session(): - outputs = self.evaluate([ - predicted_bboxes, classification_scores, idxs, bboxes, bbox_scores, - valid_mask - ]) - (input_bboxes, input_scores, output_idxs, output_bboxes, output_scores, - mask) = outputs - - self.assertEqual((batch_size, num_preds, 7), input_bboxes.shape) - self.assertEqual((batch_size, num_preds), output_idxs.shape) - self.assertEqual((batch_size, num_classes, num_preds, 7), - output_bboxes.shape) - self.assertEqual((batch_size, num_preds, num_classes), - input_scores.shape) - self.assertEqual((batch_size, num_classes, num_preds), - output_scores.shape) - self.assertEqual((batch_size, num_classes, num_preds), mask.shape) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/evaluation_metadata.py b/lingvo/tasks/car/evaluation_metadata.py deleted file mode 100644 index e1b09b1b..00000000 --- a/lingvo/tasks/car/evaluation_metadata.py +++ /dev/null @@ -1,102 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Base class specifying metadata used in evaluation.""" - -import numpy as np - - -class EvaluationMetadata: - """Interface for defining metadata of dataset.""" - - def __init__(self, name): - self.name = name - - def ClassNames(self): - """Returns a list of human-interpretable strings.""" - raise NotImplementedError() - - def LabelMap(self): - """Return a label map of int -> str for each class.""" - return dict(zip(range(self.NumClasses()), self.ClassNames())) - - def NumClasses(self): - """Total number of classes for the dataset.""" - return len(self.ClassNames()) - - def DifficultyLevels(self): - """Dictionary of difficulty level strings to int32 indices.""" - raise NotImplementedError() - - def EvalClassIndices(self): - """List of int32 indices for the classes that should be evaled.""" - eval_classes = sorted(self.IoUThresholds().keys()) - return [self.ClassNames().index(name) for name in eval_classes] - - def IoUThresholds(self): - """Dictionary of IoU thresholds for every evaluated class. - - The keys of the dictionary are used to compute EvalClassIndices(). - """ - raise NotImplementedError() - - def IgnoreClassIndices(self): - """List of int32 indices for the classes that should be ignored. - - A detection that matches with a groundtruth bbox of any neighbor class will - not be considered as false positive in eval. - """ - raise NotImplementedError() - - def NumberOfPrecisionRecallPoints(self): - """Number of points on the precision-recall curve.""" - raise NotImplementedError() - - def MaximumDistance(self): - """Maximum empirically observed Euclidean distance in world coordinates.""" - raise NotImplementedError() - - def DistanceBinWidth(self): - """The width of each bin for Euclidean distance in world coordinates.""" - raise NotImplementedError() - - def MaximumNumberOfPoints(self): - """Maximum empirically observed number of points in bounding box.""" - raise NotImplementedError() - - def NumberOfPointsBins(self): - """Number of logarithmically space bins for examining points.""" - raise NotImplementedError() - - def MaximumRotation(self): - """Maximum rotation angle in world coordinates.""" - return np.pi - - def NumberOfRotationBins(self): - """Number of linear spaced rotations to bin.""" - raise NotImplementedError() - - def MinHeight2D(self): - """Minimum height of detections to be evaluated. - - Returns: - A dictionary of difficulty level strings to ints that are height - thresholds for each level. - """ - return NotImplementedError() - - def RecallAtPrecision(self): - """Report the recall at a given precision level..""" - return [0.50, 0.95] diff --git a/lingvo/tasks/car/geometry.py b/lingvo/tasks/car/geometry.py deleted file mode 100644 index 7f88683a..00000000 --- a/lingvo/tasks/car/geometry.py +++ /dev/null @@ -1,645 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Routines related to geometric operations on bboxes. - -Bboxes and coordinates are always in the last dimension of a tensor. - -2D BBoxes are represented by (ymin, xmin, ymax, xmax). - -2D BBoxes can also be represented by (centroid x, centroid y, width, height). - -2D coordinates are represented by (x, y). - -3D coordinates are represented by (x, y, z). -""" - -import lingvo.compat as tf -from lingvo.core import py_utils -import numpy as np - - -def _BroadcastMatmul(x, y): - """Broadcast y and matmul with x. - - Args: - x: A tensor of shape [..., b]. - y: A matrix of shape [b, c]. - - Returns: - Tensor: ``z[..., c]``, where ``z[i..., :] = matmul(x[i..., :], y)`` - """ - y = py_utils.HasRank(y, 2) - x_reshaped = tf.reshape(x, [-1, tf.shape(x)[-1]]) - result = tf.matmul(x_reshaped, y) - return tf.reshape(result, - tf.concat( - [tf.shape(x)[:-1], tf.shape(y)[-1:]], axis=0)) - - -def _MakeRotationMatrix(yaw, roll, pitch): - """Create a 3x3 rotation matrix from yaw, roll, pitch (angles in radians). - - Note: Yaw -> Z, Roll -> X, Pitch -> Y. - - Args: - yaw: float tensor representing a yaw angle in radians. - roll: float tensor representing a roll angle in radians. - pitch: float tensor representing a pitch angle in radians. - - Returns: - A [3, 3] tensor corresponding to a rotation matrix. - """ - - # pyformat: disable - def _UnitX(angle): - return tf.reshape([1., 0., 0., - 0., tf.cos(angle), -tf.sin(angle), - 0., tf.sin(angle), tf.cos(angle)], - shape=[3, 3]) - - def _UnitY(angle): - return tf.reshape([tf.cos(angle), 0., tf.sin(angle), - 0., 1., 0., - -tf.sin(angle), 0., tf.cos(angle)], - shape=[3, 3]) - - def _UnitZ(angle): - return tf.reshape([tf.cos(angle), -tf.sin(angle), 0., - tf.sin(angle), tf.cos(angle), 0., - 0., 0., 1.], - shape=[3, 3]) - # pyformat: enable - return tf.matmul(tf.matmul(_UnitZ(yaw), _UnitX(roll)), _UnitY(pitch)) - - -def CoordinateTransform(points, pose): - """Translate 'points' to coordinates according to 'pose' vector. - - pose should contain 6 floating point values: - translate_x, translate_y, translate_z: The translation to apply. - yaw, roll, pitch: The rotation angles in radians. - - Args: - points: Float shape [..., 3]: Points to transform to new coordinates. - pose: Float shape [6]: [translate_x, translate_y, translate_z, yaw, roll, - pitch]. The pose in the frame that 'points' comes from, and the defintion - of the rotation and translation angles to apply to points. - - Returns: - 'points' transformed to the coordinates defined by 'pose'. - """ - translate_x = pose[0] - translate_y = pose[1] - translate_z = pose[2] - - # Translate the points so the origin is the pose's center. - translation = tf.reshape([translate_x, translate_y, translate_z], shape=[3]) - translated_points = points + translation - - # Compose the rotations along the three axes. - # - # Note: Yaw->Z, Roll->X, Pitch->Y. - yaw, roll, pitch = pose[3], pose[4], pose[5] - rotation_matrix = _MakeRotationMatrix(yaw, roll, pitch) - - # Finally, rotate the points about the pose's origin according to the - # rotation matrix. - rotated_points = _BroadcastMatmul(translated_points, rotation_matrix) - return rotated_points - - -def TransformPoints(points, transforms): - """Apply 4x4 transforms to a set of points. - - Args: - points: A [..., num_points, 3] tensor of xyz point locations. - transforms: A [..., 4, 4] tensor with the same leading shape as points. - - Returns: - A tensor with the same shape as points, transformed respectively. - """ - # Create homogeneous coordinates for points. - points = tf.concat([points, tf.ones_like(points[..., :1])], axis=-1) - - # Apply transformations, and divide by last axis to project back to 3D-space. - # Transpose the transforms since the transformation is usually expected to - # be applied such that new_points = T * current_point. - points = tf.matmul(points, transforms, transpose_b=True) - points = points[..., :3] / points[..., 3:] - - return points - - -def WrapAngleRad(angles_rad, min_val=-np.pi, max_val=np.pi): - """Wrap the value of `angles_rad` to the range [min_val, max_val].""" - max_min_diff = max_val - min_val - return min_val + tf.math.floormod(angles_rad + max_val, max_min_diff) - - -def TransformBBoxes3D(bboxes_3d, transforms): - """Apply 4x4 transforms to 7 DOF bboxes (change center and rotation). - - Args: - bboxes_3d: A [..., num_boxes, 7] tensor representing 3D bboxes. - transforms: A [..., 4, 4] tensor with the same leading shape as bboxes_3d. - These transforms are expected to only affect translation and rotation, - while not scaling the data. This ensures that the bboxes have the same - dimensions post transformation. - - Returns: - A tensor with the same shape as bboxes_3d, with transforms applied to each - bbox3d. - """ - center_xyz = bboxes_3d[..., :3] - dimensions = bboxes_3d[..., 3:6] - rot = bboxes_3d[..., 6:] - - # Transform center and rotation, assuming that dimensions are not changed. - center_xyz = TransformPoints(center_xyz, transforms) - rot += tf.atan2(transforms[..., 1:2, 0:1], transforms[..., 0:1, 0:1]) - rot = WrapAngleRad(rot) - - return tf.concat([ - center_xyz, - dimensions, - rot, - ], axis=-1) # pyformat: disable - - -def XYWHToBBoxes(xywh): - """Converts xywh to bboxes.""" - mtrx = tf.constant( - np.array([ - # x y h w - [0.0, 1.0, 0.0, -.5], # ymin - [1.0, 0.0, -.5, 0.0], # xmin - [0.0, 1.0, 0.0, 0.5], # ymax - [1.0, 0.0, 0.5, 0.0], # xmax - ]).T, - dtype=xywh.dtype) - return _BroadcastMatmul(xywh, mtrx) - - -def PointsToImagePlane(points, velo_to_image_plane): - """Converts 3D points to the image plane. - - Args: - points: A [N, 3] Floating point tensor containing xyz points. Points are - assumed to be in velo coordinates. - velo_to_image_plane: A [3, 4] matrix from velo xyz to image plane xy. After - multiplication, you need to divide by last coordinate to recover 2D pixel - locations. - - Returns: - A [N, 2] Floating point tensor containing points in the image plane. - """ - points = py_utils.HasRank(points, 2) - num_points = tf.shape(points)[0] - points = py_utils.HasShape(points, [num_points, 3]) - velo_to_image_plane = py_utils.HasShape(velo_to_image_plane, [3, 4]) - - # Add homogenous coordinates to points. - points = tf.concat([points, tf.ones((num_points, 1))], axis=-1) - - # Perform projection and divide by last coordinate to recover 2D pixel - # locations. - points_image = tf.matmul(points, velo_to_image_plane, transpose_b=True) - points_image = points_image[:, :2] / points_image[:, 2:3] - - points_image = py_utils.HasShape(points_image, [num_points, 2]) - return points_image - - -def BBoxesToXYWH(bboxes): - """Converts bboxes to xywh.""" - mtrx = tf.constant( - np.array([ - # ymin xmin ymax xmax - [0.0, 0.5, 0.0, 0.5], # x centroid - [0.5, 0.0, 0.5, 0.0], # y centroid - [0.0, -1., 0.0, 1.0], # width - [-1., 0.0, 1.0, 0.0], # height - ]).T, - dtype=bboxes.dtype) - return _BroadcastMatmul(bboxes, mtrx) - - -def BBoxesCentroid(bboxes): - """Returns the centroids of bboxes.""" - mtrx = tf.constant( - np.array([ - # ymin xmin ymax xmax - [0.0, 0.5, 0.0, 0.5], # x centroid - [0.5, 0.0, 0.5, 0.0], # y centroid - ]).T, - dtype=bboxes.dtype) - return _BroadcastMatmul(bboxes, mtrx) - - -def ReorderIndicesByPhi(anchor, bboxes): - """Sort bboxes based their angles relative to the anchor point. - - Args: - anchor: A vector of (x0, y0). - bboxes: A matrix of shape [N, 4]. - - Returns: - A permutation of tf.range(n) which can be used to reshuffle bboxes to the - sorted order. (e.g., tf.gather(bboxes, indices)). - """ - - def _True(inputs): - """True branch when num of inputs.bboxes is non-zero.""" - n = tf.shape(inputs.bboxes)[0] - centroid = BBoxesCentroid(inputs.bboxes) - - # Computed dot products between centroid and the anchor point. - dot = tf.squeeze( - tf.matmul(centroid, tf.expand_dims(inputs.anchor, 1)), axis=1) - - # Normalize dot to get the cosine of the angles. - norm = tf.norm(inputs.anchor) * tf.norm(centroid, axis=1) - cosine = tf.where( - tf.greater(norm, 0), dot / norm, tf.zeros([n], norm.dtype)) - - # Disambiguates the angle anchor--O--point is positive or negative by the - # sign of cross products between angle and points. tf.linalg.cross takes - # 3-vector (x, y, z), so we set z to 0. tf.linalg.cross does not support - # broadcasting, so we tile anchor to shape [n, 3]. - cross = tf.linalg.cross( - tf.tile( - tf.pad(tf.expand_dims(inputs.anchor, 0), [[0, 0], [0, 1]]), [n, 1]), - tf.pad(centroid, [[0, 0], [0, 1]])) - - # If the sign is positive, the points lie on the clockwise side of - # O-->anchor. Hence, -1 - cosine moves the cosine values to [-2, 0]. If the - # sign is negative, the points lie on the counter-clockwise side of - # O-->anchor. 1 + cosine moves the cosine values to [0, 2]. - # - # The car dataset shows that the points are scanned in the counter-clockwise - # fashion. Therefore, top-k orders the points in the same order in which - # bboxes appears in the spin. - score = tf.where(tf.greater(cross, 0)[:, 2], -1 - cosine, 1 + cosine) - - _, indices = tf.nn.top_k(score, n, sorted=True) - return indices - - def _False(_): - return tf.zeros([0], dtype=tf.int32) - - n = tf.shape(bboxes)[0] - return py_utils.If( - tf.greater(n, 0), py_utils.NestedMap(anchor=anchor, bboxes=bboxes), _True, - _False) - - -def _SmoothL1Norm(a): - """Smoothed L1 norm.""" - # F&F paper formula (3). - # http://openaccess.thecvf.com/content_cvpr_2018/papers/Luo_Fast_and_Furious_CVPR_2018_paper.pdf - return tf.where(tf.abs(a) < 1, 0.5 * tf.square(a), tf.abs(a) - 0.5) - - -def DistanceBetweenCentroidsAndBBoxesFastAndFurious(centroids, bboxes, masks): - """Computes the distance between centroids and bboxes. - - The distance/loss is loosely following the 'Fast and Furious' paper by Luo et - al., CVPR'18. This is just one way of calculating the distances. We will - probably develop other ways. - - Args: - centroids: [..., 4]. x/y/w/h for bboxes. - bboxes: [..., 4]. ymin/xmin/ymax/xmax for bboxes. - masks: [...]. masks[i] == 1 means i-th entry (centroids[i] and bboxes[i]) - should be considered in the distance/loss calculation. - - Returns: - A [...] tensor. i-th value is the distance measure of centroids[i] and - bboxes[i]. - """ - x, y, w, h = tf.unstack(centroids, axis=-1, num=4) - # "gt" suffix means 'ground truth'. - x_gt, y_gt, w_gt, h_gt = tf.unstack(BBoxesToXYWH(bboxes), axis=-1, num=4) - - def Pos(x): - return tf.maximum(tf.constant(1e-8, x.dtype), x) - - # The following terms are zeros when masks[i] is 0. - l_x = py_utils.CheckNumerics(masks * (x - x_gt) / Pos(w_gt)) - l_y = py_utils.CheckNumerics(masks * (y - y_gt) / Pos(h_gt)) - s_w = py_utils.CheckNumerics(masks * tf.math.log(Pos(w) / Pos(w_gt))) - s_h = py_utils.CheckNumerics(masks * tf.math.log(Pos(h) / Pos(h_gt))) - return (_SmoothL1Norm(l_x) + _SmoothL1Norm(l_y) + _SmoothL1Norm(s_w) + - _SmoothL1Norm(s_h)) - - -def DistanceBetweenCentroids(u, v, masks): - """Computes the distance between centroids. - - Args: - u: [..., 4]. x/y/w/h for bboxes. - v: [..., 4]. x/y/w/h for bboxes. - masks: [...]. masks[i] == 1 means i-th entry (u[i] and v[i]) should be - considered in the distance/loss calculation. - - Returns: - A [...] tensor. i-th value is the distance measure of u[i] and v[i]. - """ - return masks * tf.reduce_sum(_SmoothL1Norm(u - v), axis=-1) - - -# TODO(zhifengc/drpng): Consider other possible loss formuation: -# E.g., -# (L1(u[x], v[x]) + L1(u[w], v[w]))*(L1(u[y], v[y]) + L1(u[h], v[h])) - - -def _IsOnLeftHandSideOrOn(point, v1, v2): - """Checks if a point lays on a vector direction, or is to the left. - - Args: - point: a tensor of shape [..., 2] of points to check. - v1: a float tensor of shape [..., 2] of vertices. - v2: a tensor of shape and type as v1. The second vertices. - - Returns: - A tensor of booleans indicating whether each point is on the left - of, or exactly on, the direction indicated by the vertices. - """ - v1 = py_utils.HasShape(v1, tf.shape(v2)) - # Prepare for broadcast: All point operations are on the right, - # and all v1/v2 operations are on the left. This is faster than left/right - # under the assumption that we have more points than vertices. - point_x = point[..., tf.newaxis, :, 0] - point_y = point[..., tf.newaxis, :, 1] - v1_x = v1[..., 0, tf.newaxis] - v2_x = v2[..., 0, tf.newaxis] - v1_y = v1[..., 1, tf.newaxis] - v2_y = v2[..., 1, tf.newaxis] - d1 = (point_y - v1_y) * (v2_x - v1_x) - d2 = (point_x - v1_x) * (v2_y - v1_y) - return d1 >= d2 - - -def _IsCounterClockwiseDirection(v1, v2, v3): - """Checks if the path from v1 to v3 via v2 is counter-clockwise. - - When v1 is equal to v2, or v2 equals v3, return true, by fiat. Tis will - work when the v's are padded vectors. - - Args: - v1: a float Tensor of shape [..., 2], indicating the starting point. - v2: a Tensor of same type and shape as v1, indicating the via point. - v3: a Tensor of same type and shape as v1, indicating the ending point. - - Returns: - True for all directions such that v1 to v3 via v2 is a counter clockwise - direction. - """ - # Check if it's on the left hand side, strictly, and without broadcasting. - v1 = py_utils.HasShape(v1, tf.shape(v2)) - v1 = py_utils.HasShape(v1, tf.shape(v3)) - v1_x, v1_y = v1[..., 0], v1[..., 1] - v2_x, v2_y = v2[..., 0], v2[..., 1] - v3_x, v3_y = v3[..., 0], v3[..., 1] - d1 = (v3_y - v1_y) * (v2_x - v1_x) - d2 = (v3_x - v1_x) * (v2_y - v1_y) - return d1 >= d2 - - -def IsWithinBBox(points, bbox): - """Checks if points are within a 2-d bbox. - - The function returns true if points are strictly inside the box. It also - returns true when the points are exactly on the box edges. - - Args: - points: a float Tensor of shape [..., 2] of points to be tested. The last - coordinates are (x, y). - bbox: a float Tensor of shape [..., 4, 2] of bboxes. The last coordinates - are the four corners of the bbox and (x, y). The corners are assumed to be - given in counter-clockwise order. - - Returns: - Tensor: If ``pshape = tf.shape(points)[:-1]`` and - ``bshape = tf.shape(bbox)[:-2]``, returns a boolean tensor of shape - ``tf.concat(pshape, bshape)``, where each element is true if the point is - inside to the corresponding box. If a point falls exactly on an edge of the - bbox, it is also true. - """ - bshape = py_utils.GetShape(bbox)[:-2] - pshape = py_utils.GetShape(points)[:-1] - bbox = py_utils.HasShape(bbox, tf.concat([bshape, [4, 2]], axis=0)) - points = py_utils.HasShape(points, tf.concat([pshape, [2]], axis=0)) - # Enumerate all 4 edges: - v1, v2, v3, v4 = (bbox[..., 0, :], bbox[..., 1, :], bbox[..., 2, :], - bbox[..., 3, :]) - v1v2v3_check = tf.reduce_all(_IsCounterClockwiseDirection(v1, v2, v3)) - v2v3v4_check = tf.reduce_all(_IsCounterClockwiseDirection(v2, v3, v4)) - v4v1v2_check = tf.reduce_all(_IsCounterClockwiseDirection(v4, v1, v2)) - v3v4v1_check = tf.reduce_all(_IsCounterClockwiseDirection(v3, v4, v1)) - with tf.control_dependencies([ - py_utils.Assert(v1v2v3_check, [v1, v2, v3]), - py_utils.Assert(v2v3v4_check, [v3, v3, v4]), - py_utils.Assert(v4v1v2_check, [v4, v1, v2]), - py_utils.Assert(v3v4v1_check, [v3, v4, v1]) - ]): - is_inside = tf.math.logical_and( - tf.math.logical_and( - _IsOnLeftHandSideOrOn(points, v1, v2), - _IsOnLeftHandSideOrOn(points, v2, v3)), - tf.math.logical_and( - _IsOnLeftHandSideOrOn(points, v3, v4), - _IsOnLeftHandSideOrOn(points, v4, v1))) - # Swap the last two dimensions. - is_inside = tf.einsum('...ij->...ji', tf.cast(is_inside, tf.int32)) - return tf.cast(is_inside, tf.bool) - - -def BBoxCorners2D(bboxes): - """Extract the corner points from a 5-DOF bbox representation. - - Args: - bboxes: A [..., 5] floating point bounding box representation ([x, y, dx, - dy, phi]). - - Returns: - A [..., 4, 2] floating point Tensor containing - the corner (x, y) points for every bounding box. - """ - corners = tf.constant([ - [0.5, 0.5], - [-0.5, 0.5], - [-0.5, -0.5], - [0.5, -0.5], - ]) - - leading_shape = py_utils.GetShape(bboxes)[:-1] - - # Extract location, dimension, and rotation. - location = bboxes[..., :2] - dimensions = bboxes[..., 2:4] - phi_world = bboxes[..., 4] - - # Convert rotation_phis into rotation matrices along unit z. - cos = tf.cos(phi_world) - sin = tf.sin(phi_world) - rotations_world = tf.reshape( - tf.stack([cos, -sin, sin, cos], axis=-1), leading_shape + [2, 2]) - - # Create axis-aligned corners from length/width/height. - corners = tf.einsum('...i,ji->...ji', dimensions, corners) - - # Rotate the corners coordinates to the rotated world frame. - corners = tf.einsum('...ij,...kj->...ki', rotations_world, corners) - - # Translate corners to the world location. - corners = corners + tf.reshape(location, leading_shape + [1, 2]) - return corners - - -def BBoxCorners(bboxes): - """Extract the corner points from a 7-DOF bbox representation. - - Args: - bboxes: A [batch, num_boxes, 7] floating point bounding box representation - ([x, y, z, dx, dy, dz, phi]). - - Returns: - A [batch, num_boxes, 8, 3] floating point Tensor containing - the corner (x, y, z) points for every bounding box. - """ - # Code adapted from vale/soapbox codebase. - # - # Corners in normalized box frame (unit cube centered at origin). - # - # Dimensions is [length, width, height]. - corners = tf.constant([ - [0.5, 0.5, 0.5], # top - [-0.5, 0.5, 0.5], # top - [-0.5, -0.5, 0.5], # top - [0.5, -0.5, 0.5], # top - [0.5, 0.5, -0.5], # bottom - [-0.5, 0.5, -0.5], # bottom - [-0.5, -0.5, -0.5], # bottom - [0.5, -0.5, -0.5], # bottom - ]) - - batch, nb, _ = py_utils.GetShape(bboxes, 3) - - # Extract location, dimension, and rotation. - location = bboxes[:, :, :3] - dimensions = bboxes[:, :, 3:6] - phi_world = bboxes[:, :, 6] - - # Convert rotation_phis into rotation matrices along unit z. - cos = tf.cos(phi_world) - sin = tf.sin(phi_world) - zero = tf.zeros_like(cos) - one = tf.ones_like(cos) - rotations_world = tf.reshape( - tf.stack([cos, -sin, zero, sin, cos, zero, zero, zero, one], axis=2), - [batch, nb, 3, 3]) - - # Create axis-aligned corners from length/width/height. - corners = tf.einsum('bni,ji->bnji', dimensions, corners) - - # Rotate the corners coordinates to the rotated world frame. - corners = tf.einsum('bnij,bnkj->bnki', rotations_world, corners) - - # Translate corners to the world location. - corners = corners + tf.reshape(location, (batch, nb, 1, 3)) - return corners - - -def IsWithinBBox3D(points_3d, bboxes_3d): - """Checks if points are within a 3-d bbox. - - Args: - points_3d: [num_points, 3] float32 Tensor specifying points in 3-d space as - [x, y, z] coordinates. - bboxes_3d: [num_bboxes, 7] float32 Tensor specifying a 3-d bboxes specified - as [x, y, z, dx, dy, dz, phi] where x, y and z is the center of the box. - - Returns: - boolean Tensor of shape [num_points, num_bboxes] indicating whether the - points belong within each box. - """ - points_3d = py_utils.HasRank(points_3d, 2) - points_3d = py_utils.HasShape(points_3d, [-1, 3]) - num_points, _ = py_utils.GetShape(points_3d, 2) - - bboxes_3d = py_utils.HasRank(bboxes_3d, 2) - bboxes_3d = py_utils.HasShape(bboxes_3d, [-1, 7]) - num_bboxes, _ = py_utils.GetShape(bboxes_3d, 2) - - # Compute the 3-D corners of the bounding boxes. - bboxes_3d_b = tf.expand_dims(bboxes_3d, 0) - bbox_corners = BBoxCorners(bboxes_3d_b) - bbox_corners = py_utils.HasShape(bbox_corners, [1, -1, 8, 3]) - # First four points are the top of the bounding box. - # Counter-clockwise arrangement of points specifying 2-d Euclidean box. - # (x0, y1) <--- (x1, y1) - # ^ - # | - # | - # (x0, y0) ---> (x1, y0) - bboxes_2d_corners = bbox_corners[0, :, 0:4, 0:2] - bboxes_2d_corners = py_utils.HasShape(bboxes_2d_corners, [-1, 4, 2]) - # Determine if points lie within 2-D (x, y) plane for all bounding boxes. - points_2d = points_3d[:, :2] - is_inside_2d = IsWithinBBox(points_2d, bboxes_2d_corners) - is_inside_2d = py_utils.HasShape(is_inside_2d, [num_points, num_bboxes]) - - # Determine if points lie with the z-dimension for all bounding boxes. - [_, _, z, _, _, dz, _] = tf.split(bboxes_3d, 7, axis=-1) - - def _ComputeLimits(center, width): - left = center - width / 2.0 - right = center + width / 2.0 - return left, right - - z0, z1 = _ComputeLimits(z, dz) - z_points = tf.expand_dims(points_3d[:, 2], -1) - - is_inside_z = tf.math.logical_and( - tf.less_equal(z_points, z1[tf.newaxis, :, 0]), - tf.greater_equal(z_points, z0[tf.newaxis, :, 0])) - is_inside_z = py_utils.HasShape(is_inside_z, [num_points, num_bboxes]) - - return tf.math.logical_and(is_inside_z, is_inside_2d) - - -def SphericalCoordinatesTransform(points_xyz): - """Converts points from xyz coordinates to spherical coordinates. - - https://en.wikipedia.org/wiki/Spherical_coordinate_system#Coordinate_system_conversions - for definitions of the transformations. - - Args: - points_xyz: A floating point tensor with shape [..., 3], where the inner 3 - dimensions correspond to xyz coordinates. - - Returns: - A floating point tensor with the same shape [..., 3], where the inner - dimensions correspond to (dist, theta, phi), where phi corresponds to - azimuth/yaw (rotation around z), and theta corresponds to pitch/inclination - (rotation around y). - """ - dist = tf.sqrt(tf.reduce_sum(tf.square(points_xyz), axis=-1)) - theta = tf.acos(points_xyz[..., 2] / tf.maximum(dist, 1e-7)) - # Note: tf.atan2 takes in (y, x). - phi = tf.atan2(points_xyz[..., 1], points_xyz[..., 0]) - return tf.stack([dist, theta, phi], axis=-1) diff --git a/lingvo/tasks/car/geometry_test.py b/lingvo/tasks/car/geometry_test.py deleted file mode 100644 index e84b8744..00000000 --- a/lingvo/tasks/car/geometry_test.py +++ /dev/null @@ -1,458 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for geometry.""" - -import lingvo.compat as tf -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.tasks.car import geometry -import numpy as np - - -class GeometryTest(test_utils.TestCase): - - def testXYWHBBoxesCentroid(self): - with self.session(): - xywh = np.tile( - np.array([[[10, 20, 8, 6], [0, 0, 0, 0], [-10, -20, 2.4, 3.6]]]), - (3, 2, 1)) - bboxes = np.tile( - np.array([[[17, 6, 23, 14], [0, 0, 0, 0], [-21.8, -11.2, -18.2, - -8.8]]]), (3, 2, 1)) - centroid = np.tile(np.array([[[10, 20], [0, 0], [-10, -20]]]), (3, 2, 1)) - print('shapes = ', xywh.shape, bboxes.shape, centroid.shape) - self.assertAllClose(geometry.XYWHToBBoxes(xywh).eval(), bboxes) - self.assertAllClose(geometry.BBoxesToXYWH(bboxes).eval(), xywh) - self.assertAllClose(geometry.BBoxesCentroid(bboxes).eval(), centroid) - - def testPointsToImage(self): - # From a KITTI example. - velo_to_image_plane = tf.constant( - [[6.09695409e+02, -7.21421597e+02, -1.25125855e+00, -1.23041806e+02], - [1.80384202e+02, 7.64479802e+00, -7.19651474e+02, -1.01016688e+02], - [9.99945389e-01, 1.24365378e-04, 1.04513030e-02, -2.69386912e-01]]) - - points = tf.constant([[1.25120001e+01, -5.09700012e+00, -7.26999998e-01], - [1.27309999e+01, -5.21099997e+00, 1.85000002e-01], - [-100, -20, -30]]) - - with self.session(): - points_image = geometry.PointsToImagePlane(points, velo_to_image_plane) - result = points_image.eval() - # First two points map to the image plane, the last one falls outside. - expected = [[914.15222168, 215.81137085], [914.61260986, 162.28927612], - [463.57763672, -32.7820015]] - self.assertAllClose(expected, result) - - def testReorderIndicesByPhi(self): - # Picks the anchor point somewhere on the x-y plane. - phi0 = 2 * np.pi / 3 - anchor = np.array([np.cos(phi0), np.sin(phi0)]) - - # Generates points clock-wise relative to the anchor. - pnts = [] - n, delta = 16, 0.01 - for i in range(n): - phi = phi0 + 1e-6 + 2 * np.pi / n * i - x, y = (1 + i) * np.cos(phi), (1 + i) * np.sin(phi) - pnts += [[y - delta, x - delta, y + delta, x + delta]] - pnts = np.array(pnts) - labels = np.arange(n) - - # Randomly permutate the points. - perm = np.random.permutation(n) - pnts = pnts[perm] - labels = labels[perm] - - # Uses ReorderIndicesByPhi to figure out how to reshuffle the points to - # recover their original order. - with self.session(): - indices = geometry.ReorderIndicesByPhi(anchor, pnts).eval() - - print('indices = ', indices) - print('labels = ', labels[indices]) - self.assertAllEqual(labels[indices], np.flip(np.arange(n), 0)) - - def testReorderIndicesByPhiEmpty(self): - with self.session(): - indices = geometry.ReorderIndicesByPhi( - tf.constant([1., 0.], tf.float32), - tf.constant(np.zeros([0, 4]), tf.float32)).eval() - - self.assertEqual(indices.shape, (0,)) - - def testDistanceBetweenCentroidsAndBBoxesFastAndFurious(self): - # pyformat: disabled - predicted = np.array([ - [1., 1., 2., 2.], # Perfect. - [2., 2., 2., 2.], # w/h are perfect. - [1., 1., 1., 1.], # centroids are perfect. - [0., 0., 0., 0.], # /0 if not padded properly. - ]) - groundtruth = np.array([ - [0., 0., 2., 2.], - [0., 0., 2., 2.], - [0., 0., 2., 2.], - [0., 0., 0., 0.], - ]) - masks = np.array([ - 1., - 1., - 1., - 0., - ]) - # pyformat: enabled - with self.session(): - distance = geometry.DistanceBetweenCentroidsAndBBoxesFastAndFurious( - predicted, groundtruth, masks) - self.assertAllClose( - distance.eval(), - [0.0, 1.0 / 4, np.square(np.log(1. / 2)), 0.0]) - - def testDistanceBetweenCentroids(self): - # pyformat: disabled - u = np.array([ - [1., 1., 2., 2.], # Perfect. - [2., 2., 2., 2.], # w/h are perfect. - [1., 1., 1., 1.], # centroids are perfect. - [0., 1., 2., 3.], # does not matter. - ]) - v = np.array([ - [1., 1., 2., 2.], # Perfect. - [1., 1., 2., 2.], # w/h are perfect. - [1., 1., 2., 2.], - [1., 2., 3., 4.], - ]) - masks = np.array([ - 1., - 1., - 1., - 0., - ]) - # pyformat: enabled - with self.session(): - distance = geometry.DistanceBetweenCentroids(u, v, masks) - self.assertAllClose(distance.eval(), [0.0, 1.0, 1.0, 0.0]) - - def testCoordinateTransform(self): - # This is a validated test case from a colab on a real scene. - # - # A single point [1, 1, 3]. - point = tf.constant([[[5736.94580078, 1264.85168457, 45.0271225]]], - dtype=tf.float32) - # Replicate the point to test broadcasting behavior. - replicated_points = tf.tile(point, [2, 4, 1]) - - # Pose of the car (x, y, z, yaw, roll, pitch). - # - # We negate the translations so that the coordinates are translated - # such that the car is at the origin. - pose = tf.constant([ - -5728.77148438, -1264.42236328, -45.06399918, -3.10496902, 0.03288471, - 0.00115049 - ], - dtype=tf.float32) - - transformed_points = geometry.CoordinateTransform(replicated_points, pose) - with self.session(): - result = transformed_points.eval() - - # We expect the point to be translated close to the car, and then rotated - # mostly around the x-axis. - expected = np.tile([[[-8.18451203, -0.13086951, -0.04200766]]], [2, 4, 1]) - - self.assertAllClose(expected, result) - - def _MakeTransformTestRotationMatrices(self, batch_size): - # Make a batch of 4x4 transformation matrices that only has rotation around - # the z-axis (world rotation). - rot_matrices = [] - for _ in range(batch_size): - rot_matrix = geometry._MakeRotationMatrix(tf.random.uniform([]), 0., 0.) - # Embed rotation matrix into a 4 x 4 matrix - rot_matrix = tf.pad(rot_matrix, [[0, 1], [0, 1]]) + tf.linalg.tensor_diag( - [0, 0, 0, 1.]) - rot_matrices.append(rot_matrix) - transforms = tf.stack(rot_matrices, axis=0) - return transforms - - def _MakeTransformTestTranslationMatrices(self, batch_size): - # Make a batch of 4x4 transformation matrices that translate in all - # directions. - translation_matrices = [] - for _ in range(batch_size): - translation_matrix = tf.random.uniform([3, 1]) - translation_matrix = tf.pad(translation_matrix, [[0, 1], [3, 0]]) - translation_matrix += tf.linalg.tensor_diag([1., 1., 1., 1.]) - translation_matrices.append(translation_matrix) - transforms = tf.stack(translation_matrices, axis=0) - return transforms - - def testTransformPointsRotation(self): - batch_size, num_points = 10, 8 - points = tf.random.uniform((batch_size, num_points, 3)) - transforms = self._MakeTransformTestRotationMatrices(batch_size) - points_transformed = geometry.TransformPoints(points, transforms) - with self.session(): - actual_points, actual_points_transformed = self.evaluate( - (points, points_transformed)) - # Points are the same on the z-axis (no rotation). - self.assertAllClose(actual_points[:, :, 2], actual_points_transformed[:, :, - 2]) - # Points are transformed, and different. - self.assertNotAllClose(actual_points, actual_points_transformed) - - def testTransformPointsTranslation(self): - batch_size, num_points = 10, 8 - points = tf.random.uniform((batch_size, num_points, 3)) - transforms = self._MakeTransformTestTranslationMatrices(batch_size) - points_transformed = geometry.TransformPoints(points, transforms) - with self.session(): - actual_points, actual_points_transformed, actual_transforms = self.evaluate( - (points, points_transformed, transforms)) - # Points are transformed, and different. - self.assertNotAllClose(actual_points, actual_points_transformed) - # Manually transform points and check that they are as expected. - actual_translation = actual_transforms[:, :3, 3] - self.assertAllClose(actual_points + actual_translation[:, np.newaxis, :], - actual_points_transformed) - - def testWrapAngleRad(self): - angles = tf.random.uniform([100], - minval=-100., - maxval=100., - dtype=tf.float32) - wrapped_angles = geometry.WrapAngleRad(angles) - with self.session(): - actual_angles, actual_wrapped_angles = self.evaluate( - (angles, wrapped_angles)) - - # The sine values of the angles should remain the same after wrapping. - self.assertAllClose( - np.sin(actual_angles), np.sin(actual_wrapped_angles), atol=1e-5) - - # Check ranges match the wrapped expectations. - self.assertTrue(np.all(actual_wrapped_angles >= -np.pi)) - self.assertTrue(np.all(actual_wrapped_angles <= np.pi)) - - def testTransformBBoxes3D(self): - batch_size, num_boxes = 10, 20 - bboxes_3d = tf.random.uniform((batch_size, num_boxes, 7)) - transforms = self._MakeTransformTestTranslationMatrices(batch_size) - bboxes_3d_transformed = geometry.TransformBBoxes3D(bboxes_3d, transforms) - with self.session(): - actual_bboxes_3d, actual_bboxes_3d_transformed = self.evaluate( - (bboxes_3d, bboxes_3d_transformed)) - - self.assertAllEqual(actual_bboxes_3d.shape, - actual_bboxes_3d_transformed.shape) - - # Dimensions (slice 3:6) should remain unchanged. - self.assertAllClose(actual_bboxes_3d[..., 3:6], - actual_bboxes_3d_transformed[..., 3:6]) - - # Rotation should remain unchanged. - self.assertAllClose(actual_bboxes_3d[..., 6], - actual_bboxes_3d_transformed[..., 6]) - - # Center xyz should be different. - self.assertNotAllClose(actual_bboxes_3d[..., :3], - actual_bboxes_3d_transformed[..., :3]) - - def testTransformBBoxes3DConsistentWithPoints(self): - num_boxes, num_points = 20, 100 - points = tf.random.uniform((num_points, 3)) - bboxes_3d = tf.random.uniform((num_boxes, 7)) - in_bboxes = geometry.IsWithinBBox3D(points, bboxes_3d) - transforms = self._MakeTransformTestTranslationMatrices(1)[0] - points_transformed = geometry.TransformPoints(points, transforms) - bboxes_3d_transformed = geometry.TransformBBoxes3D(bboxes_3d, transforms) - in_bboxes_transformed = geometry.IsWithinBBox3D(points_transformed, - bboxes_3d_transformed) - with self.session(): - actual_in_bboxes, actual_in_bboxes_transformed = self.evaluate( - (in_bboxes, in_bboxes_transformed)) - self.assertAllEqual(actual_in_bboxes, actual_in_bboxes_transformed) - - def testIsOnLeftHandSideOrOn(self): - v1 = tf.constant([[0., 0.]], dtype=tf.float32) - v2 = tf.constant([[1., 0.]], dtype=tf.float32) - p = tf.constant([[.5, .5], [-1., -3], [-1., 1.]], dtype=tf.float32) - with self.session(): - actual = self.evaluate(geometry._IsOnLeftHandSideOrOn(p, v1, v2)) - self.assertAllEqual([[True, False, True]], actual) - actual = self.evaluate(geometry._IsOnLeftHandSideOrOn(v1, v1, v2)) - self.assertAllEqual([[True]], actual) - actual = self.evaluate(geometry._IsOnLeftHandSideOrOn(v2, v1, v2)) - self.assertAllEqual([[True]], actual) - - def testIsWithinBBox3D(self): - num_points, num_bboxes = 19, 4 - # rotate the first box by pi / 2 so dim_x and dim_y are swapped. - # The last box is a cube rotated by 45 degrees. - bboxes = tf.constant([[1.0, 2.0, 3.0, 6.0, 0.4, 6.0, np.pi / 2], - [4.0, 5.0, 6.0, 7.0, 0.8, 7.0, 0.0], - [0.4, 0.3, 0.2, 0.1, 0.1, 0.2, 0.0], - [-10., -10., -10., 3., 3., 3., np.pi / 4]], - dtype=tf.float32) - points = tf.constant( - [ - [1.0, 2.0, 3.0], # box 0 (centroid) - [0.8, 2.0, 3.0], # box 0 (below x) - [1.1, 2.0, 3.0], # box 0 (above x) - [1.3, 2.0, 3.0], # box 0 (too far x) - [0.7, 2.0, 3.0], # box 0 (too far x) - [4.0, 5.0, 6.0], # box 1 (centroid) - [4.0, 4.6, 6.0], # box 1 (below y) - [4.0, 5.4, 6.0], # box 1 (above y) - [4.0, 4.5, 6.0], # box 1 (too far y) - [4.0, 5.5, 6.0], # box 1 (too far y) - [0.4, 0.3, 0.2], # box 2 (centroid) - [0.4, 0.3, 0.1], # box 2 (below z) - [0.4, 0.3, 0.3], # box 2 (above z) - [0.4, 0.3, 0.0], # box 2 (too far z) - [0.4, 0.3, 0.4], # box 2 (too far z) - [5.0, 7.0, 8.0], # none - [1.0, 5.0, 3.6], # box0, box1 - [-11.6, -10., -10.], # box3 (rotated corner point). - [-11.4, -11.4, -10.], # not in box3, would be if not rotated. - ], - dtype=tf.float32) - expected_is_inside = np.array([ - [True, False, False, False], - [True, False, False, False], - [True, False, False, False], - [False, False, False, False], - [False, False, False, False], - [False, True, False, False], - [False, True, False, False], - [False, True, False, False], - [False, False, False, False], - [False, False, False, False], - [False, False, True, False], - [False, False, True, False], - [False, False, True, False], - [False, False, False, False], - [False, False, False, False], - [False, False, False, False], - [True, True, False, False], - [False, False, False, True], - [False, False, False, False], - ]) - assert points.shape[0] == num_points - assert bboxes.shape[0] == num_bboxes - assert expected_is_inside.shape[0] == num_points - assert expected_is_inside.shape[1] == num_bboxes - - with self.session(): - is_inside = self.evaluate(geometry.IsWithinBBox3D(points, bboxes)) - self.assertAllEqual([num_points, num_bboxes], is_inside.shape) - self.assertAllEqual(expected_is_inside, is_inside) - - def testIsWithinBBox(self): - bbox = tf.constant([[[0., 0.], [1., 0.], [1., 1.], [0., 1.]]], - dtype=tf.float32) - points = tf.constant( - [[-.5, -.5], [.5, -.5], [1.5, -.5], [1.5, .5], [1.5, 1.5], [.5, 1.5], - [-.5, 1.5], [-.5, .5], [1., 1.], [.5, .5]], - dtype=tf.float32) - with self.session(): - is_inside = self.evaluate(geometry.IsWithinBBox(points, bbox)) - expected = [[False]] * 8 + [[True]] * 2 - self.assertAllEqual(expected, is_inside) - - def testIsWithinRotatedBBox(self): - bbox = tf.constant([[[.2, 0.], [1., .2], [.8, 1.], [0., .8]]], - dtype=tf.float32) - points = tf.constant([[0., 0.], [1., 0], [1., 1.], [0., 1.], [.5, .5]], - dtype=tf.float32) - with self.session(): - is_inside = self.evaluate(geometry.IsWithinBBox(points, bbox)) - expected = [[False]] * 4 + [[True]] - self.assertAllEqual(expected, is_inside) - - def testIsCounterClockwiseDirection(self): - points = tf.constant( - [[[0., 0.], [0., 0.], [0., 1.]], [[0., 0.], [0., 1.], [1., 1.]], - [[0., .8], [.2, 0.], [1., .2]], [[.2, .0], [1., .2], [.0, 1.]], - [[1., 1.], [1., 0], [0., 0.]]], - dtype=tf.float32) - # points ~ [num points, v1/v2/v3, x/y]. - points = py_utils.HasShape(points, [-1, 3, 2]) - expected = [True, False, True, True, False] - with self.session(): - dircheck = self.evaluate( - geometry._IsCounterClockwiseDirection(points[:, 0, :], - points[:, 1, :], points[:, - 2, :])) - self.assertAllEqual(expected, dircheck) - - def testBBoxCorners(self): - # Create four bounding boxes, two identical in each batch. - # - # This tests both that the batching and number of box dimensions are handled - # properly. - bboxes = tf.constant([[[1, 2, 3, 4, 3, 6, 0.], [1, 2, 3, 4, 3, 6, 0.]], - [[1, 2, 3, 4, 3, 6, np.pi / 2.], - [1, 2, 3, 4, 3, 6, np.pi / 2.]]]) - corners = geometry.BBoxCorners(bboxes) - with self.session(): - corners_np = self.evaluate(corners) - self.assertEqual((2, 2, 8, 3), corners_np.shape) - - # Extrema of first two boxes are ([-1, 3], [0.5, 3.5], [0, 6]) - for i in [0, 1]: - self.assertAllClose(-1, np.min(corners_np[0, i, :, 0])) - self.assertAllClose(3, np.max(corners_np[0, i, :, 0])) - self.assertAllClose(0.5, np.min(corners_np[0, i, :, 1])) - self.assertAllClose(3.5, np.max(corners_np[0, i, :, 1])) - self.assertAllClose(0, np.min(corners_np[0, i, :, 2])) - self.assertAllClose(6, np.max(corners_np[0, i, :, 2])) - - # Extrema of second two boxes is ([-0.5, 2.5], [0, 4], [0, 6]) - # because it's the first box rotated by 90 degrees. - for i in [0, 1]: - self.assertAllClose(-0.5, np.min(corners_np[1, i, :, 0])) - self.assertAllClose(2.5, np.max(corners_np[1, i, :, 0])) - self.assertAllClose(0, np.min(corners_np[1, i, :, 1])) - self.assertAllClose(4, np.max(corners_np[1, i, :, 1])) - self.assertAllClose(0, np.min(corners_np[1, i, :, 2])) - self.assertAllClose(6, np.max(corners_np[1, i, :, 2])) - - def testSphericalCoordinatesTransform(self): - np_xyz = np.random.randn(5, 6, 3) - points_xyz = tf.constant(np_xyz, dtype=tf.float32) - spherical_coordinates = geometry.SphericalCoordinatesTransform(points_xyz) - - with self.session(): - actual_spherical_coordinates = self.evaluate(spherical_coordinates) - - # Convert coordinates back to xyz to verify. - dist = actual_spherical_coordinates[..., 0] - theta = actual_spherical_coordinates[..., 1] - phi = actual_spherical_coordinates[..., 2] - - x = dist * np.sin(theta) * np.cos(phi) - y = dist * np.sin(theta) * np.sin(phi) - z = dist * np.cos(theta) - - self.assertAllClose(x, np_xyz[..., 0]) - self.assertAllClose(y, np_xyz[..., 1]) - self.assertAllClose(z, np_xyz[..., 2]) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/input_extractor.py b/lingvo/tasks/car/input_extractor.py deleted file mode 100644 index 20737825..00000000 --- a/lingvo/tasks/car/input_extractor.py +++ /dev/null @@ -1,188 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Input extractors. - -Input extractors are an API for parsing and processing a set of fields from -serialized records. -""" - -from lingvo import compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils - -from lingvo.tasks.car import base_extractor - -BaseExtractor = base_extractor._BaseExtractor # pylint:disable=protected-access -BUCKET_UPPER_BOUND = base_extractor.BUCKET_UPPER_BOUND - - -################################################################################ -# Extractors for car data. -################################################################################ -class FieldsExtractor(base_layer.BaseLayer): - """An API for parsing and processing a set of fields from serialized records. - - Input generators often need to parse several fields from a serialized record. - This involves two stages: specifying the name and type of the fields to - extract from serialized records (tf.Example or tf.SequenceExample), and then - processing the raw output into a form to be consumed by higher-level callers. - - This class attempts to modularize this processing within the Minecraft input - generators, so that users can easily create input generator pipelines that mix - and match the composition of different fields from the same dataset. - - A descendant of this class will implement three functions: - - 1) FeatureMap(): returning a dictionary of field names to field types, e.g., - 'images' to tf.io.VarLenFeature(tf.string). For PlainTextIterator - datasets, FeatureMap() should be empty. - - 2) _Extract(features): Given a 'features' dictionary containing the result - from calling tf.io.parse_example or tf.parse_sequence_example on all - extractors' features, produce a NestedMap of Tensors. - - NOTE: The return of the overall pipeline is a NestedMap of batched - Tensors. However, the names and associations of the fields of each - extractor are lost on the boundary of the map fn. At the moment, one - must implement _Extract() such that the names of the fields returned in - the NestedMap matches self.Shape()'s keys; this is checked during the - parent's Extract() call. - - 3) Shape(): A NestedMap mapping names of outputs to their static shape, - without the batch dimension. In _InputBatch, this shape will be used to - ensure that every output has a statically known shape. - - The caller of Extractors calls each extractor's FeatureMap() to populate the - schema passed to tf.io.parse_example() or tf.parse_sequence_example(). The - resulting dicationary of Tensors is then passed to each extractor's _Extract() - function (via FieldsExtractor.Extract()) to return each extractor's output. - - It is the responsibility of the caller to maintain orders of outputs, since - NestedMaps do not have any inherent ordering during iteration. - """ - - @classmethod - def Params(cls): - """Defaults params.""" - p = super().Params() - p.name = cls.__name__ - return p - - def __init__(self, params): - super().__init__(params) - self.SetVariableFree() - - def FeatureMap(self): - """Return a dictionary from tf.Example feature names to Features.""" - raise NotImplementedError() - - def ContextMap(self): - """Return a dict mapping tf.SequenceExample context names to Features.""" - return {} - - def Extract(self, features): - """Given 'feature' (Sparse)Tensors, output Tensors for consumption. - - NOTE: Implementation provided by subclasses's _Extract() method. - - Args: - features: A dictionary of (Sparse)Tensors which includes tensors from all - extractors. - - Returns: - A NestedMap of output Tensors. - """ - outputs = self._Extract(features) - shapes = self.Shape() - assert outputs.IsCompatible(shapes), '{} vs. {}'.format( - outputs.DebugString(), shapes.DebugString()) - return outputs - - def Filter(self, outputs): - """Return the bucket based on the result of Extract(). - - This function should return 1 if the example should pass through without - being dropped, and BUCKET_UPPER_BOUND if the example should be dropped. - - Args: - outputs: The NestedMap returned by this extractor's _Extract() function. - This is useful to implement filtering based on the values of the - extracted example. - - Returns: - A scalar bucket id. - """ - del outputs - return 1 - - def Shape(self): - """Return a NestedMap of un-batched fully-specified tf.TensorShapes.""" - raise NotImplementedError() - - def DType(self): - """Return a NestedMap mapping names to tf.DType.""" - raise NotImplementedError() - - def _Extract(self, features): - """The subclass-defined implementation of Extract(). - - Args: - features: A dictionary of (Sparse)Tensors which includes tensors from all - extractors. - - Returns: - A NestedMap of output Tensors whose key names match self.Shape()'s keys. - """ - raise NotImplementedError() - - -class LaserExtractor(FieldsExtractor): - """Interface for extracting laser data. - - Must produce: - points_xyz: [max_num_points, 3] - XYZ coordinates of laser points. - - points_feature: [max_num_points, num_features] - Features for each point in - points_xyz. - - points_padding: [max_num_points]: Padding for points. 0 means the - corresponding point is the original, and 1 means there is no point - (xyz or feature) present. Only present if max_num_points is not - None. - - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('max_num_points', None, 'The number of points per spin.') - p.Define('num_features', 1, 'Number of features per laser point.') - return p - - def Shape(self): - p = self.params - ret = py_utils.NestedMap( - points_xyz=tf.TensorShape([p.max_num_points, 3]), - points_feature=tf.TensorShape([p.max_num_points, p.num_features])) - if p.max_num_points is not None: - ret.points_padding = tf.TensorShape([p.max_num_points]) - return ret - - def DType(self): - ret = py_utils.NestedMap(points_xyz=tf.float32, points_feature=tf.float32) - if self.params.max_num_points is not None: - ret.points_padding = tf.float32 - return ret diff --git a/lingvo/tasks/car/input_extractor_test.py b/lingvo/tasks/car/input_extractor_test.py deleted file mode 100644 index 9f5934c8..00000000 --- a/lingvo/tasks/car/input_extractor_test.py +++ /dev/null @@ -1,45 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for input_extractor.""" - -from lingvo import compat as tf -from lingvo.core import hyperparams -from lingvo.core import test_utils -from lingvo.tasks.car import input_extractor -from lingvo.tasks.car import input_preprocessors - - -class InputExtractorTest(test_utils.TestCase): - - def testBaseExtractorRaisesErrorWithMissingPreprocessorKeys(self): - extractors = hyperparams.Params() - preprocessors = hyperparams.Params() - preprocessors.Define( - 'count_points', - input_preprocessors.CountNumberOfPointsInBoxes3D.Params(), '') - preprocessors.Define('viz_copy', - input_preprocessors.CreateDecoderCopy.Params(), '') - p = input_extractor.BaseExtractor.Params(extractors).Set( - preprocessors=preprocessors, - preprocessors_order=['count_points', 'missing_key', 'viz_copy']) - with self.assertRaisesRegex( - ValueError, - r'preprocessor_order specifies keys which were not found .*'): - p.Instantiate() - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/input_preprocessors.py b/lingvo/tasks/car/input_preprocessors.py deleted file mode 100644 index 1faeac57..00000000 --- a/lingvo/tasks/car/input_preprocessors.py +++ /dev/null @@ -1,3450 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Input preprocessors.""" - -from lingvo import compat as tf -from lingvo.core import base_layer -from lingvo.core import py_utils -from lingvo.tasks.car import car_lib -from lingvo.tasks.car import detection_3d_lib -from lingvo.tasks.car import geometry -from lingvo.tasks.car import ops -import numpy as np -# pylint:disable=g-direct-tensorflow-import -from tensorflow.python.ops import inplace_ops -# pylint:enable=g-direct-tensorflow-import - - -def _ConsistentShuffle(tensors, seed): - """Shuffle multiple tensors with the same shuffle order.""" - shuffled_idx = tf.range(tf.shape(tensors[0])[0]) - shuffled_idx = tf.random.shuffle(shuffled_idx, seed=seed) - return tuple([tf.gather(t, shuffled_idx) for t in tensors]) - - -def _GetApplyPointMaskFn(points_mask): - """Returns a function that applies a mask to one of our points tensors.""" - - def _ApplyPointMaskFn(points_tensor): - """Applies a mask to the points tensor.""" - if points_tensor is None: - return points_tensor - return tf.boolean_mask(points_tensor, points_mask) - - return _ApplyPointMaskFn - - -def _Dense(sparse): - return tf.sparse_to_dense( - sparse_indices=sparse.indices, - output_shape=sparse.dense_shape, - sparse_values=sparse.values, - default_value=0) - - -class Preprocessor(base_layer.BaseLayer): - """Base class for input preprocessor. - - Input preprocessors expect the combined output of all extractors and performs - a transformation on them. Input preprocessors can add/edit/remove fields - from the NestedMap of features. - - Note: Features correspond to that for one example (no batch dimension). - - Sub-classes need to implement the following three functions: - - 1) TransformFeatures(features): Given a NestedMap of features representing the - output of all the extractors, apply a transformation on the features. - - 2) TransformShapes(shapes): Given a corresponding NestedMap of shapes, - produce a NestedMap of shapes that corresponds to the transformation of the - features after TransformFeatures. - - 3) TransformDTypes(dtypes): Given a corresponding NestedMap of dtypes, - produce a NestedMap of dtypes that corresponds to the transformation of the - features after TransformFeatures. - - The preprocessor is expected to explicitly pass through untouched fields. - For example, a preprocessor that does data augmentation should modify the - features NestedMap on the fields it cares about augmenting, and then return - the features NestedMap. - """ - - @classmethod - def Params(cls): - """Default params.""" - p = super().Params() - p.name = cls.__name__ - return p - - def FProp(self, theta, features): - """Performs TransformFeatures.""" - del theta # unused - return self.TransformFeatures(features) - - def TransformFeatures(self, features): - """Transforms the features for one example. - - Args: - features: A `NestedMap` of tensors. - - Returns: - A `NestedMap` of tensors corresponding. - """ - raise NotImplementedError() - - def TransformShapes(self, shapes): - """Sets correct shapes corresponding to TransformFeatures. - - Args: - shapes: A `NestedMap` of TensorShapes, corresponding to the - pre-transformed features. - - Returns: - A `NestedMap` of TensorShapes corresponding to the transformed features. - """ - raise NotImplementedError() - - def TransformDTypes(self, dtypes): - """Sets correct dtypes corresponding to TransformFeatures. - - Args: - dtypes: A `NestedMap` of DTypes, corresponding to the pre-transformed - features. - - Returns: - A `NestedMap` of DTypes corresponding to the transformed features. - """ - raise NotImplementedError() - - -class EntryPreprocessor(Preprocessor): - """A Preprocessor that transforms a NestedMap sub-structure. - - Some preprocessors want to apply a function to any NestedMap whose key matches - a specific prefix. An EntryPreprocessor provides an interface for specifying - the function transformation for a NestedMap of inputs, adding, modifying, or - deleting the entries in that NestedMap. - - For example, if an input contains a nested structure such as: - - lasers.front.xyz - .features - - lasers.side.xyz - .features - - and one wants to apply a transform that modifies the .xyz features - on both structures, one can define an EntryPreprocessor that implements: - - UpdateEntry(entry): - UpdateEntryShape(shapes): - UpdateEntryDType(dtypes): - - and set self.params.prefixes = ['lasers.front', 'lasers.side'] - where the prefixes refer to a fully-qualified NestedMap sub-structure. - - The arguments to these functions will contain just the NestedMap structure - whose key prefix can be found in self.params.prefixes. One can then modify - these structures as desired. - - Example: - def UpdateEntry(self, entry): - # entry is a NestedMap. - assert 'xyz' in entry - entry.xyz = self._ApplyFn(entry.xyz) - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('prefixes', ['pseudo_ri'], 'List of keys to apply to.') - return p - - def _ApplyToMatchingStructure(self, nested_map, fn): - """Apply fn to any NestedMap sub-structure whose prefix is in p.prefixes.""" - p = self.params - # Don't mutate the original. - nested_map = nested_map.DeepCopy() - updated_entries = [] - for prefix in p.prefixes: - entry = nested_map.GetItem(prefix) - if not isinstance(entry, py_utils.NestedMap): - raise TypeError('Prefix key {} selected a {}, not a NestedMap!'.format( - prefix, type(entry))) - fn(entry) - updated_entries.append(entry) - return nested_map, updated_entries - - def UpdateEntry(self, entry): - """Update the Tensors in a NestedMap entry. - - Args: - entry: A NestedMap of Tensors. - """ - raise NotImplementedError() - - def UpdateEntryShape(self, shapes): - """Update the shapes in a NestedMap entry. - - Args: - shapes: A NestedMap of TensorShapes. - """ - raise NotImplementedError() - - def UpdateEntryDType(self, dtypes): - """Transform the dtypes in a NestedMap entry. - - Args: - dtypes: A NestedMap of dtypes. - """ - raise NotImplementedError() - - def TransformFeatures(self, features): - features, _ = self._ApplyToMatchingStructure(features, self.UpdateEntry) - return features - - def TransformShapes(self, shapes): - shapes, _ = self._ApplyToMatchingStructure(shapes, self.UpdateEntryShape) - return shapes - - def TransformDTypes(self, dtypes): - dtypes, _ = self._ApplyToMatchingStructure(dtypes, self.UpdateEntryDType) - return dtypes - - -class CreateDecoderCopy(Preprocessor): - """Creates references to current lasers, images, and labels. - - This is useful if the data is further transformed. - - If desired, the keys that are copied can be customized by overriding the - default keys param. - - This preprocessor expects features to optionally contain the following keys: - - lasers - a NestedMap of tensors - - images - a NestedMap of tensors - - labels - a NestedMap of tensors - - Adds the following features (if the features existed): - - decoder_copy.lasers - a copy of the lasers NestedMap - - decoder_copy.images - a copy of the images NestedMap - - decoder_copy.labels - a copy of the labels NestedMap - - The processor also by default pads the laser features; this can be disabled - by setting the pad_lasers param to None. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('keys', ['lasers', 'labels', 'images'], - 'Keys to look for and copy if exists.') - p.Define('parent_key', 'decoder_copy', 'The key to nest the copies under.') - p.Define('pad_lasers', PadLaserFeatures.Params(), - 'Params for a layer that pads the laser features.') - p.name = 'create_decoder_copy' - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.pad_lasers is not None: - self.CreateChild('pad_lasers', p.pad_lasers) - - def _DeepCopyIfExists(self, keys, nested_map, parent_key): - """Deep copy a specific key to a parent key if it exists.""" - for key in keys: - if key in nested_map: - if parent_key not in nested_map: - nested_map[parent_key] = py_utils.NestedMap() - nested_map[parent_key][key] = nested_map[key].DeepCopy() - return nested_map - - def TransformFeatures(self, features): - p = self.params - features = self._DeepCopyIfExists(p.keys, features, p.parent_key) - if p.pad_lasers is not None: - features[p.parent_key] = self.pad_lasers.TransformFeatures( - features[p.parent_key]) - return features - - def TransformShapes(self, shapes): - p = self.params - shapes = self._DeepCopyIfExists(p.keys, shapes, p.parent_key) - if p.pad_lasers is not None: - shapes[p.parent_key] = self.pad_lasers.TransformShapes( - shapes[p.parent_key]) - return shapes - - def TransformDTypes(self, dtypes): - p = self.params - dtypes = self._DeepCopyIfExists(p.keys, dtypes, p.parent_key) - if p.pad_lasers is not None: - dtypes[p.parent_key] = self.pad_lasers.TransformDTypes( - dtypes[p.parent_key]) - return dtypes - - -class FilterByKey(Preprocessor): - """Filters features to keep only specified keys. - - This keeps only feature entries that are specified. This allows us to reduce - the number of fields returned. For example, during training, one may not - need the actual laser points if training with a pillars based model that - has a preprocessor that already maps the points to grid. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'keep_key_prefixes', [''], 'Prefixes of keys to keep. If this ' - 'contains the empty string, then it will keep all the keys.') - return p - - def _FilterFn(self, key, entry): - """Filter a nested map.""" - del entry # unused - p = self.params - for prefix in p.keep_key_prefixes: - if key.startswith(prefix): - return True - return False - - def TransformFeatures(self, features): - return features.FilterKeyVal(self._FilterFn) - - def TransformShapes(self, shapes): - return shapes.FilterKeyVal(self._FilterFn) - - def TransformDTypes(self, dtypes): - return dtypes.FilterKeyVal(self._FilterFn) - - -class FilterGroundTruthByNumPoints(Preprocessor): - """Removes ground truth boxes with less than params.min_num_points points. - - This preprocessor expects features to contain the following keys:: - labels.labels of shape [..., L] - labels.bboxes_3d of shape [..., L, 7] - labels.bboxes_3d_mask of shape [..., L] - labels.unfiltered_bboxes_3d_mask of shape [..., L] - labels.bboxes_3d_num_points of shape [..., L]. - - Modifies the bounding box data to turn off ground truth objects that don't - meet the params.min_num_points point filter: - - labels.labels: Boxes with less than params.min_num_points have their label - set to params.background_id (defaults to 0). - - labels.bboxes_3d_mask: Boxes with less than params.min_num_points are set - to 0. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'min_num_points', 1, 'The minimum number of points allowed before ' - 'the associated ground truth box is turned off. Defaults to 1.') - p.Define( - 'background_id', 0, 'The ID of the background class we set ' - 'filtered boxes to. Defaults to 0.') - return p - - def TransformFeatures(self, features): - p = self.params - bbox_is_valid = tf.greater_equal(features.labels.bboxes_3d_num_points, - p.min_num_points) - features.labels.labels = tf.where( - bbox_is_valid, features.labels.labels, - p.background_id * tf.ones_like(features.labels.labels)) - features.labels.bboxes_3d_mask *= tf.cast(bbox_is_valid, tf.float32) - return features - - def TransformShapes(self, shapes): - return shapes - - def TransformDTypes(self, dtypes): - return dtypes - - -class FilterGroundTruthByDifficulty(Preprocessor): - """Removes groundtruth boxes based on detection difficulty. - - This preprocessor expects features to contain the following keys:: - labels.single_frame_detection_difficulties of shape [..., L] - labels.labels of shape [..., L] - labels.bboxes_3d_mask of shape [..., L] - labels.unfiltered_bboxes_3d_mask of shape [..., L] - - The preprocessor masks out the bboxes_3d_mask / labels based on whether - single_frame_detection_difficulties is greater than p.difficulty_threshold. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'background_id', 0, 'The ID of the background class we set ' - 'filtered boxes to. Defaults to 0.') - p.Define( - 'difficulty_threshold', 1, - 'Filter groundtruth bounding boxes whose detection difficulty is ' - 'greater than `difficulty_threshold`') - return p - - def TransformFeatures(self, features): - p = self.params - bbox_is_valid = tf.less_equal( - features.labels.single_frame_detection_difficulties, - p.difficulty_threshold) - features.labels.labels = tf.where( - bbox_is_valid, features.labels.labels, - p.background_id * tf.ones_like(features.labels.labels)) - features.labels.bboxes_3d_mask *= tf.cast(bbox_is_valid, tf.float32) - return features - - def TransformShapes(self, shapes): - return shapes - - def TransformDTypes(self, dtypes): - return dtypes - - -class CountNumberOfPointsInBoxes3D(Preprocessor): - """Computes bboxes_3d_num_points. - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [P, 3] - - labels.bboxes_3d of shape [L, 7] - - labels.bboxes_3d_mask of shape [L] - - and optionally points_padding of shape [P] corresponding to the padding. - if points_padding is None, then all points are considered valid. - - Adds the following features: - labels.bboxes_3d_num_points: [L] - integer tensor containing the number of - laser points for each corresponding bbox. - """ - - def TransformFeatures(self, features): - points_xyz = features.lasers.points_xyz - if 'points_padding' in features.lasers: - points_mask = 1 - features.lasers.points_padding - points_xyz = tf.boolean_mask(points_xyz, points_mask) - - points_in_bboxes_mask = geometry.IsWithinBBox3D(points_xyz, - features.labels.bboxes_3d) - bboxes_3d_num_points = tf.reduce_sum( - tf.cast(points_in_bboxes_mask, tf.int32), axis=0, keepdims=False) - bboxes_3d_num_points *= tf.cast(features.labels.bboxes_3d_mask, tf.int32) - - features.labels.bboxes_3d_num_points = bboxes_3d_num_points - return features - - def TransformShapes(self, shapes): - num_bboxes = shapes.labels.bboxes_3d[0] - shapes.labels.bboxes_3d_num_points = tf.TensorShape([num_bboxes]) - return shapes - - def TransformDTypes(self, dtypes): - dtypes.labels.bboxes_3d_num_points = tf.int32 - return dtypes - - -class AddPerPointLabels(Preprocessor): - """Computes the class and bbox id of each point. - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [P, 3] - - labels.bboxes_3d of shape [L, 7] - - labels.labels of shape [L] - - This makes an assumption that each point is only in 1 box, which should - almost always true in 3D. In cases where this is not true, the largest - label integer and largest bbox_id will be assigned. - - NOTE: Be very careful that this is performed after any modifications - to the semantic labels of each point in the pointcloud. Examples of this - would be operators like GroundTruthAugmentation, or DropBoxesOutOfRange. - - Adds the following features: - lasers.points_label: [P] - integer tensor containing the class id of each - point. - lasers.points_bbox_id: [P] - integer tensor containing box id of each - point from 0 to num_bboxes, where an id of num_bboxes indicates a - background point. - lasers.points_bbox_3d: [P, 7] - float tensor containing bounding box of - each point. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'per_dimension_adjustment', None, - 'A list of len 3 of floats with the amount (in meters) to add to ' - 'each dimension of the box before using it to select points. ' - 'If enabled, this is designed to protect against overly tight box ' - 'annotations that appear in KITTI.') - return p - - def TransformFeatures(self, features): - p = self.params - points_xyz = features.lasers.points_xyz - bboxes_3d = features.labels.bboxes_3d - num_points, _ = py_utils.GetShape(points_xyz) - num_bboxes, _ = py_utils.GetShape(bboxes_3d) - - if p.per_dimension_adjustment: - if len(p.per_dimension_adjustment) != 3: - raise ValueError( - 'param `per_dimension_adjustment` expected to be len 3.') - dims_adjustment = tf.constant([0, 0, 0] + p.per_dimension_adjustment + - [0]) - bboxes_3d = bboxes_3d + dims_adjustment - - # Find which points are in each box and what class each box is. - points_in_bboxes_mask = geometry.IsWithinBBox3D(points_xyz, bboxes_3d) - points_in_bboxes_mask = tf.cast(points_in_bboxes_mask, tf.int32) - points_in_bboxes_mask = py_utils.HasShape(points_in_bboxes_mask, - [num_points, num_bboxes]) - - # points_in_bboxes_mask is a [num_points, num_bboxes] 0/1 tensor - # indicating whether that point is in a given box. - # Each point should only be in one box, so after broadcasting the label - # across the binary mask, we do a reduce_max to get the max label id - # for each point. Since each point only belongs to one box, it will be - # the only non-zero (background) label in that box. - # Note: We assume background to be class_id == 0 - points_label = tf.reduce_max( - points_in_bboxes_mask * features.labels.labels, axis=1) - points_bbox_id = tf.argmax( - points_in_bboxes_mask, axis=1, output_type=tf.int32) - # If the class is background, make its id == num_bboxes - points_bbox_id = tf.where(points_label > 0, points_bbox_id, - tf.broadcast_to(num_bboxes, [num_points])) - - # For each point, get the bbox_3d data. - dummy_bbox = tf.constant([[0, 0, 0, 0, 0, 0, 0]], dtype=tf.float32) - bboxes_3d = tf.concat([bboxes_3d, dummy_bbox], axis=0) - points_bbox_3d = tf.gather(bboxes_3d, points_bbox_id) - - points_label = tf.reshape(points_label, [num_points]) - points_bbox_id = tf.reshape(points_bbox_id, [num_points]) - features.lasers.points_label = points_label - features.lasers.points_bbox_id = points_bbox_id - features.lasers.points_bbox_3d = points_bbox_3d - return features - - def TransformShapes(self, shapes): - num_points = shapes.lasers.points_xyz[0] - shapes.lasers.points_label = tf.TensorShape([num_points]) - shapes.lasers.points_bbox_id = tf.TensorShape([num_points]) - shapes.lasers.points_bbox_3d = tf.TensorShape([num_points, 7]) - return shapes - - def TransformDTypes(self, dtypes): - dtypes.lasers.points_label = tf.int32 - dtypes.lasers.points_bbox_id = tf.int32 - dtypes.lasers.points_bbox_3d = tf.float32 - return dtypes - - -class PointsToGrid(Preprocessor): - """Bins points to a 3D-grid using custom op: ops.point_to_grid. - - Expects features to have keys: - - lasers.points_xyz of shape [P, 3] - - and optionally points_padding of shape [P] corresponding to the padding. - if points_padding is None, then all points are considered valid. - - If normalizing the labels is enabled, then also expects: - - labels.weights - - labels.bboxes_td - - labels.bboxes_td_mask - - labels.bboxes_3d_mask - - Let: - gx, gy, gz = p.grid_size - F = 3 + num_laser_features - - Adds the following features: - grid_centers: [gx, gy, gz, 3]: For each grid cell, the (x,y,z) - floating point coordinate of its center. - grid_num_points: [gx, gy, gz]: The number of points in each grid - cell (integer). - laser_grid: [gx, gy, gz, num_points_per_cell, F] - A 5D floating - point Tensor containing the laser data placed into a fixed grid. - - Modifies the bboxes in labels to also be within the grid range x/y by default. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_points_per_cell', 100, - 'The maximum number of points per cell.') - p.Define('grid_size', (40, 40, 1), 'Grid size along x,y,z axis.') - - # The max range of x and y is [-80, 80]. - p.Define('grid_range_x', (-80, 80), 'The X-axis Range covered by the grid') - p.Define('grid_range_y', (-80, 80), 'The Y-axis Range covered by the grid') - p.Define('grid_range_z', (-2, 4), 'The Z-axis Range covered by the grid') - - p.Define('normalize_td_labels', True, - 'Whether to clip the labels to the grid limits.') - return p - - def _NormalizeLabels(self, ymin, xmin, ymax, xmax, x_range, y_range): - """Normalizes the bboxes within a given range.""" - assert x_range, 'Must specify x_range if clipping.' - assert y_range, 'Must specify y_range if clipping.' - assert len(x_range) == 2, 'x_range %s must be 2 elements.' % x_range - assert len(y_range) == 2, 'y_range %s must be 2 elements.' % y_range - - x_range_min = x_range[0] - x_range_len = x_range[1] - x_range[0] - y_range_min = y_range[0] - y_range_len = y_range[1] - y_range[0] - - xmin = tf.cast(xmin - x_range_min, tf.float32) / tf.cast( - x_range_len, tf.float32) - xmax = tf.cast(xmax - x_range_min, tf.float32) / tf.cast( - x_range_len, tf.float32) - ymin = tf.cast(ymin - y_range_min, tf.float32) / tf.cast( - y_range_len, tf.float32) - ymax = tf.cast(ymax - y_range_min, tf.float32) / tf.cast( - y_range_len, tf.float32) - - return ymin, xmin, ymax, xmax - - def TransformFeatures(self, features): - p = self.params - - points_xyz = features.lasers.points_xyz - points_feature = features.lasers.points_feature - if ('points_padding' in features.lasers and - features.lasers.points_padding is not None): - points_mask = 1 - features.lasers.points_padding - points_xyz = tf.boolean_mask(points_xyz, points_mask) - points_feature = tf.boolean_mask(points_feature, points_mask) - - points_full = tf.concat([points_xyz, points_feature], axis=-1) - points_grid_full, grid_centers, num_points = ops.point_to_grid( - points_full, p.num_points_per_cell, p.grid_size[0], p.grid_size[1], - p.grid_size[2], p.grid_range_x, p.grid_range_y, p.grid_range_z) - - features.laser_grid = points_grid_full - features.grid_centers = grid_centers - features.grid_num_points = num_points - - if p.normalize_td_labels: - # Normalize bboxes_td w.r.t grid range. - obb = features.labels - x_range = p.grid_range_x - y_range = p.grid_range_y - ymin, xmin, ymax, xmax = tf.unstack(obb.bboxes_td[..., :4], axis=-1) - ymin, xmin, ymax, xmax = self._NormalizeLabels( - ymin, xmin, ymax, xmax, x_range=x_range, y_range=y_range) - obb.bboxes_td = tf.concat( - [tf.stack([ymin, xmin, ymax, xmax], axis=-1), obb.bboxes_td[..., 4:]], - axis=-1) - - return features - - def TransformShapes(self, shapes): - p = self.params - shapes.grid_centers = tf.TensorShape(list(p.grid_size) + [3]) - shapes.grid_num_points = tf.TensorShape(list(p.grid_size)) - shapes.laser_grid = tf.TensorShape( - list(p.grid_size) + - [p.num_points_per_cell, 3 + shapes.lasers.points_feature[-1]]) - return shapes - - def TransformDTypes(self, dtypes): - dtypes.grid_centers = tf.float32 - dtypes.grid_num_points = tf.int32 - dtypes.laser_grid = tf.float32 - return dtypes - - -class _PointPillarGridSettings: - """Settings for PointPillars model defined in paper. - - https://arxiv.org/abs/1812.05784 - """ - # Chooses grid sizes that are a multiple of 16 to support point pillars - # model requirements. These also happen to match the values - # in the PointPillars paper (voxel width of 0.16m in x, y) - GRID_X = 432 - GRID_Y = 496 - GRID_Z = 1 - - # These fields are set in the subclasses. - GRID_X_RANGE = None - GRID_Y_RANGE = None - GRID_Z_RANGE = None - - @classmethod - def UpdateGridParams(cls, grid_params): - """Apply PointPillars settings to grid_params.""" - grid_params.grid_size = (cls.GRID_X, cls.GRID_Y, cls.GRID_Z) - grid_params.grid_range_x = cls.GRID_X_RANGE - grid_params.grid_range_y = cls.GRID_Y_RANGE - grid_params.grid_range_z = cls.GRID_Z_RANGE - - @classmethod - def UpdateAnchorGridParams(cls, anchor_params, output_stride=2): - """Apply PointPillars settings to anchor_params.""" - # Set anchor settings to match grid settings. - # Grid size for anchors is half the resolution. - anchor_params.grid_size = (cls.GRID_X // output_stride, - cls.GRID_Y // output_stride, cls.GRID_Z) - anchor_params.grid_range_x = cls.GRID_X_RANGE - anchor_params.grid_range_y = cls.GRID_Y_RANGE - # Grid along z axis should be pinned to 0. - anchor_params.grid_range_z = (0, 0) - - -def MakeGridSettings(grid_x_range, grid_y_range, grid_z_range, grid_x, grid_y, - grid_z): - """Returns configured class for PointPillar grid settings.""" - - class GridSettings(_PointPillarGridSettings): - GRID_X_RANGE = grid_x_range - GRID_Y_RANGE = grid_y_range - GRID_Z_RANGE = grid_z_range - GRID_X = grid_x - GRID_Y = grid_y - GRID_Z = grid_z - - return GridSettings - - -PointPillarGridCarSettings = MakeGridSettings( - grid_x_range=(0, 69.12), - grid_y_range=(-39.68, 39.68), - grid_z_range=(-3, 1), - grid_x=432, - grid_y=496, - grid_z=1) - -PointPillarGridPedCycSettings = MakeGridSettings( - grid_x_range=(0, 47.36), - grid_y_range=(-19.84, 19.84), - grid_z_range=(-2.5, 0.5), - grid_x=432, - grid_y=496, - grid_z=1) - - -class GridToPillars(Preprocessor): - """Create pillars from a grid of points. - - Expects features to have keys: - grid_centers: [gx, gy, gz, 3] - - grid_num_points: [gx, gy, gz] - - laser_grid: [gx, gy, gz, num_points_per_cell, F] - - Adds the following features: - point_count: [num_pillars]. The number of points in the pillar. - - point_locations: [num_pillars, 3]. The grid location of each pillar. - - pillar_points: [num_pillars, num_points_per_cell, F]. Points of each - pillar. - - Drops the following features by default: - laser_grid - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_points_per_cell', 100, - 'The maximum number of points per cell.') - p.Define('num_pillars', 12000, 'The maximum number of pillars to produce.') - p.Define('drop_laser_grid', True, 'Whether to drop the laser_grid feature.') - # The density based sampler is more expensive. - p.Define('use_density_sampler', False, - 'Use a density based sampler during pillar selection.') - return p - - def _GumbelTransform(self, probs): - """Adds gumbel noise to log probabilities for multinomial sampling. - - This enables fast sampling from a multinomial distribution without - replacement. See https://arxiv.org/abs/1611.01144 for details. - A colab that demonstrates this in practice is here: - http://colab/drive/1iuMt2n_r7dKPQG9T0UVMuK3fkbBayKjd - - Args: - probs: A 1-D float tensor containing probabilities, summing to 1. - - Returns: - A 1-D float tensor of the same size of probs, with gumbel noise added to - log probabilities. Taking the top k elements from this provides a - multinomial sample without replacement. - """ - p = self.params - log_prob = tf.math.log(probs) - probs_shape = tf.shape(probs) - uniform_samples = tf.random.uniform( - shape=probs_shape, - dtype=probs.dtype, - seed=p.random_seed, - name='uniform_samples') - gumbel_noise = -tf.math.log(-tf.math.log(uniform_samples)) - return gumbel_noise + log_prob - - def _DensitySample(self, num_points): - p = self.params - - # Flatten to [nx * ny * nz] for convenience during sampling. - num_grid_points = np.prod(p.grid_size) - flattened_num_points = tf.reshape(num_points, [num_grid_points]) - - # Normalize flattened_num_points to sum to 1. - flattened_num_points = tf.cast(flattened_num_points, tf.float32) - flattened_num_points /= tf.reduce_sum(flattened_num_points) - - # TODO(jngiam): Consider generalizing this to enable other methods of - # sampling: e.g., use largest deviation in z-axis. The gumbel transform - # can still be applied regardless. - - # Add gumbel noise for multinomial sampling. - sampling_logits = self._GumbelTransform(flattened_num_points) - _, locations = tf.nn.top_k( - sampling_logits, k=min(p.num_pillars, num_grid_points)) - - # Unravel coordinates back to grid locations. - locations = tf.unravel_index(locations, p.grid_size) - - # Unravel index will return a 3 x num_locations tensor, this needs to be - # transposed so that we have it as num_locations x 3. - locations = py_utils.HasShape(locations, [3, -1]) - locations = tf.transpose(locations) - - return locations - - def TransformFeatures(self, features): - p = self.params - - num_points = features.grid_num_points - if p.use_density_sampler: - locations = self._DensitySample(num_points) - else: - # Select non-empty cells uniformly at random. - locations = tf.random.shuffle(tf.cast(tf.where(num_points > 0), tf.int32)) - - num_features = py_utils.GetShape(features.laser_grid)[-1] - - # [nx, ny, nz, np, 4] (x, y, z, f) - points = features.laser_grid - # [K, np, 4] (x, y, z, f) - points = tf.gather_nd(points, locations) - # [nx, ny, nz, 1, 3] (cx, cy, cz) - centers = features.grid_centers[..., tf.newaxis, :] - # [K, 1, 3] (cx, cy, cz) - centers = tf.gather_nd(centers, locations) - # NOTE: If there are fewer pillars than p.num_pillars, the following - # padding creates many 'fake' pillars at grid cell (0, 0, 0) with - # an all-zero pillar. Hopefully, the model can learn to ignore these. - # - # pillar_points[i, :, :] is the pillar located at pillar_locations[i, :3], - # and pillar_points[i, :, :] == points_grid_full[pillar_locations[i, :3]]. - # for 0 <= i < pillar_count; - # pillar_locations[i, :3] are zero-ed, for i >= pillar_count. - features.pillar_count = tf.shape(locations)[0] - features.pillar_locations = py_utils.PadOrTrimTo(locations, - [p.num_pillars, 3]) - features.pillar_points = py_utils.PadOrTrimTo( - points, [p.num_pillars, p.num_points_per_cell, num_features]) - features.pillar_centers = py_utils.PadOrTrimTo(centers, - [p.num_pillars, 1, 3]) - - if p.drop_laser_grid: - del features['laser_grid'] - - return features - - def TransformShapes(self, shapes): - p = self.params - num_features = shapes.laser_grid[-1] - shapes.pillar_count = tf.TensorShape([]) - shapes.pillar_locations = tf.TensorShape([p.num_pillars, 3]) - shapes.pillar_points = tf.TensorShape( - [p.num_pillars, p.num_points_per_cell, num_features]) - shapes.pillar_centers = tf.TensorShape([p.num_pillars, 1, 3]) - if p.drop_laser_grid: - del shapes['laser_grid'] - return shapes - - def TransformDTypes(self, dtypes): - p = self.params - dtypes.pillar_count = tf.int32 - dtypes.pillar_locations = tf.int32 - dtypes.pillar_points = tf.float32 - dtypes.pillar_centers = tf.float32 - if p.drop_laser_grid: - del dtypes['laser_grid'] - return dtypes - - -class GridAnchorCenters(Preprocessor): - """Create anchor centers on a grid. - - Anchors are placed in the middle of each grid cell. For example, on a 2D grid - range (0 -> 10, 0 -> 10) with a 10 x 5 grid size, the anchors will be placed - at [(0.5, 1), (0.5, 3), ... , (9.5, 7), (9.5, 9)]. - - Adds the following features: - anchor_centers: [num_locations, 3] - Floating point output containing the - center (x, y, z) locations for tiling anchor boxes. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'grid_size', (20, 20, 1), 'Grid size along x,y,z axis. This will ' - 'be used to generate the anchor center locations. Note that this ' - 'would likely be different from the grid_* parameters in ' - 'LaserGridExtractor: the grid extractor may choose to extract ' - 'points more densely. Instead, this should correspond to the ' - 'model\'s prediction layer: the predicted anchor box residuals ' - 'should match this grid.') - p.Define('grid_range_x', (-25, 25), 'The x-axis range covered by the grid.') - p.Define('grid_range_y', (-25, 25), 'The y-axis range covered by the grid.') - p.Define('grid_range_z', (0, 0), 'The z-axis range covered by the grid.') - return p - - def TransformFeatures(self, features): - p = self.params - utils_3d = detection_3d_lib.Utils3D() - - # Compute the grid cell size and adjust the range sent to dense coordinates - # by half a cell size so as to ensure that the anchors are placed in the - # center of each grid cell. - grid_size_x, grid_size_y, grid_size_z = p.grid_size - grid_cell_sizes = [ - float(p.grid_range_x[1] - p.grid_range_x[0]) / grid_size_x, - float(p.grid_range_y[1] - p.grid_range_y[0]) / grid_size_y, - float(p.grid_range_z[1] - p.grid_range_z[0]) / grid_size_z, - ] - half_size_x, half_size_y, half_size_z = np.asarray(grid_cell_sizes) / 2.0 - - grid_shape = list(p.grid_size) + [3] - anchor_centers = utils_3d.CreateDenseCoordinates([ - [ - p.grid_range_x[0] + half_size_x, - p.grid_range_x[1] - half_size_x, - grid_size_x - ], - [ - p.grid_range_y[0] + half_size_y, - p.grid_range_y[1] - half_size_y, - grid_size_y - ], - [ - p.grid_range_z[0] + half_size_z, - p.grid_range_z[1] - half_size_z, - grid_size_z - ], - ]) # pyformat: disable - features.anchor_centers = tf.reshape(anchor_centers, grid_shape) - - return features - - def TransformShapes(self, shapes): - p = self.params - shapes.anchor_centers = tf.TensorShape(list(p.grid_size) + [3]) - return shapes - - def TransformDTypes(self, dtypes): - dtypes.anchor_centers = tf.float32 - return dtypes - - -class SparseCenterSelector(Preprocessor): - """Select centers for anchors and cells. - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [P, 3] - - and optionally points_padding of shape [P] corresponding to the padding. - if points_padding is None, then all points are considered valid. - - If lasers.num_seeded_points of shape [] is provided, it indicates that the - first num_seeded_points of lasers.points_xyz should be used as seeds for - farthest point sampling (e.g., always chosen). Currently the concept - of seeding is not implemented for anything but farthest point sampling. - - Adds the following features: - anchor_centers: [num_cell_centers, 3] - Floating point output containing the - center (x, y, z) locations for tiling anchor boxes. - cell_center_xyz: [num_cell_centers, 3] - Floating point output containing - the center (x, y, z) locations for each cell to featurize. - """ - - _SAMPLING_METHODS = ['farthest_point', 'random_uniform'] - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_cell_centers', 256, 'Number of centers.') - p.Define( - 'features_preparation_layers', [], - 'A list of Params for layers to run on the features before ' - 'performing farthest point sampling. For example, one may wish to ' - 'drop points out of frustum for KITTI before selecting centers. ' - 'Note that these layers will not mutate the original features, ' - 'instead, a copy will be made.') - p.Define( - 'sampling_method', 'farthest_point', - 'Which sampling method to use. One of {}'.format(cls._SAMPLING_METHODS)) - p.Define( - 'fix_z_to_zero', True, 'Whether to fix z to 0 when retrieving the ' - 'center xyz coordinates.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - if p.sampling_method not in self._SAMPLING_METHODS: - raise ValueError('Param `sampling_method` must be one of {}.'.format( - self._SAMPLING_METHODS)) - if p.features_preparation_layers is not None: - self.CreateChildren('features_preparation_layers', - p.features_preparation_layers) - - def _FarthestPointSampleCenters(self, points_xyz, num_seeded_points): - """Samples centers with Farthest Point Sampling. - - Args: - points_xyz: An unpadded tf.float32 Tensor of shape [P, 3] with per point - (x, y, z) locations. We expect any padded points to be removed before - this function is called. - num_seeded_points: integer indicating how many of the first - num_seeded_points points in points_xyz should be considered - as seeds for FPS (always chosen). - - Returns: - A tf.float32 Tensor of shape [p.num_cell_centers, 3] with selected centers - to use as anchors. - """ - p = self.params - num_points = tf.shape(points_xyz)[0] - points_padding = tf.zeros((num_points,), dtype=tf.float32) - padded_num_points = tf.maximum(num_points, p.num_cell_centers) - - # Pad both the points and padding if for some reason the input pointcloud - # has less points than p.num_cell_centers. - points_xy = py_utils.PadOrTrimTo(points_xyz[:, :2], [padded_num_points, 2]) - points_padding = py_utils.PadOrTrimTo( - points_padding, [padded_num_points], pad_val=1.0) - - sampled_idx, _ = car_lib.FarthestPointSampler( - points_xy[tf.newaxis, ...], - points_padding[tf.newaxis, ...], - p.num_cell_centers, - num_seeded_points=num_seeded_points, - random_seed=p.random_seed) - sampled_idx = sampled_idx[0, :] - - # Gather centers. - if p.fix_z_to_zero: - centers = tf.concat([ - tf.gather(points_xy, sampled_idx), - tf.zeros((p.num_cell_centers, 1)), - ], axis=-1) # pyformat: disable - else: - centers = tf.gather(points_xyz, sampled_idx) - - return centers - - def _RandomUniformSampleCenters(self, points_xyz): - """Samples centers with Random Uniform Sampling. - - Args: - points_xyz: An unpadded tf.float32 Tensor of shape [P, 3] with per point - (x, y, z) locations. We expect any padded points to be removed before - this function is called. - - Returns: - A tf.float32 Tensor of shape [p.num_cell_centers, 3] with selected centers - to use as anchors. - """ - p = self.params - # We want the center Z value to be 0 so just exclude it - centers_xy = tf.random.shuffle(points_xyz[:, :2], seed=p.random_seed) - selected_centers_xy = py_utils.PadOrTrimTo(centers_xy, - [p.num_cell_centers, 2]) - return tf.concat([selected_centers_xy, - tf.zeros((p.num_cell_centers, 1))], - axis=-1) - - def _SampleCenters(self, points_xyz, num_seeded_points): - p = self.params - if p.sampling_method == 'farthest_point': - return self._FarthestPointSampleCenters(points_xyz, num_seeded_points) - elif p.sampling_method == 'random_uniform': - if num_seeded_points > 0: - raise NotImplementedError( - 'Random sampling with seeded points not yet implemented.') - return self._RandomUniformSampleCenters(points_xyz) - else: - raise ValueError('Param `sampling_method` must be one of {}.'.format( - self._SAMPLING_METHODS)) - - def TransformFeatures(self, features): - p = self.params - - prepared_features = features.DeepCopy() - for prep_layer in self.features_preparation_layers: - prepared_features = prep_layer.FPropDefaultTheta(prepared_features) - - num_seeded_points = prepared_features.lasers.get('num_seeded_points', 0) - points_data = prepared_features.lasers - - points_xyz = points_data.points_xyz - if 'points_padding' in points_data: - points_padding = points_data.points_padding - points_mask = 1 - points_padding - points_xyz = tf.boolean_mask(points_xyz, points_mask) - - centers = self._SampleCenters(points_xyz, num_seeded_points) - centers = py_utils.HasShape(centers, [p.num_cell_centers, 3]) - - features.anchor_centers = centers - features.cell_center_xyz = centers - - return features - - def TransformShapes(self, shapes): - p = self.params - shapes.anchor_centers = tf.TensorShape([p.num_cell_centers, 3]) - shapes.cell_center_xyz = tf.TensorShape([p.num_cell_centers, 3]) - return shapes - - def TransformDTypes(self, dtypes): - dtypes.anchor_centers = tf.float32 - dtypes.cell_center_xyz = tf.float32 - return dtypes - - -class SparseCellGatherFeatures(Preprocessor): - """Select local features for each cell. - - This preprocessor expects features to contain: - - lasers.points_xyz of shape [P, 3] - - lasers.points_feature of shape [P, F] - - cell_center_xyz of shape [C, 3] - - and optionally points_padding of shape [P] corresponding to the padding. - if points_padding is None, then all points are considered valid. - - Adds the following features: - cell_points_xyz: [num_centers, num_points_per_cell, 3] - Floating point - output containing the (x, y, z) locations for each point for a given - center. - cell_feature: [num_centers, num_points_per_cell, F] - Floating point output - containing the features for each point for a given center. - cell_points_padding: [num_centers, num_points_per_cell] - 0/1 padding - for the points in each cell. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_points_per_cell', 128, 'The number of points per cell.') - p.Define('max_distance', 3.0, 'Max distance of point to cell center.') - p.Define( - 'sample_neighbors_uniformly', False, - 'Whether to sample the neighbor points for every cell center ' - 'uniformly at random. If False, this will default to selecting by ' - 'distance.') - return p - - def TransformFeatures(self, features): - p = self.params - - num_centers = py_utils.GetShape(features.cell_center_xyz, 1)[0] - num_features = py_utils.GetShape(features.lasers.points_feature)[-1] - - points_xyz = features.lasers.points_xyz - points_feature = features.lasers.points_feature - if 'points_padding' in features.lasers: - points_mask = 1 - features.lasers.points_padding - points_xyz = tf.boolean_mask(points_xyz, points_mask) - points_feature = tf.boolean_mask(points_feature, points_mask) - - # Note: points_xyz and points_feature must be unpadded as we pass - # padding=None to neighborhood indices. Ensuring that it is unpadded - # helps improve performance. - - # Get nearby points using kNN. - sample_indices, sample_indices_padding = car_lib.NeighborhoodIndices( - tf.expand_dims(points_xyz, 0), - tf.expand_dims(features.cell_center_xyz, 0), - p.num_points_per_cell, - points_padding=None, - max_distance=p.max_distance, - sample_neighbors_uniformly=p.sample_neighbors_uniformly) - - # Take first example since NeighboorhoodIndices expects batch dimension. - sample_indices = sample_indices[0, :, :] - sample_indices_padding = sample_indices_padding[0, :, :] - - sample_indices = py_utils.HasShape(sample_indices, - [num_centers, p.num_points_per_cell]) - - cell_points_xyz = tf.gather(points_xyz, sample_indices) - cell_points_xyz = py_utils.HasShape(cell_points_xyz, - [num_centers, p.num_points_per_cell, 3]) - - cell_feature = tf.gather(points_feature, sample_indices) - cell_feature = py_utils.HasShape( - cell_feature, [num_centers, p.num_points_per_cell, num_features]) - - cell_points_padding = py_utils.HasShape( - sample_indices_padding, [num_centers, p.num_points_per_cell]) - - features.update({ - 'cell_points_xyz': cell_points_xyz, - 'cell_feature': cell_feature, - 'cell_points_padding': cell_points_padding, - }) - return features - - def TransformShapes(self, shapes): - p = self.params - num_centers = shapes.cell_center_xyz[0] - base_shape = [num_centers, p.num_points_per_cell] - num_features = shapes.lasers.points_feature[-1] - shapes.cell_points_xyz = tf.TensorShape(base_shape + [3]) - shapes.cell_feature = tf.TensorShape(base_shape + [num_features]) - shapes.cell_points_padding = tf.TensorShape(base_shape) - return shapes - - def TransformDTypes(self, dtypes): - dtypes.cell_points_xyz = tf.float32 - dtypes.cell_feature = tf.float32 - dtypes.cell_points_padding = tf.float32 - return dtypes - - -class SparseCellCentersTopK(Preprocessor): - """Given selected centers and gathered points/features, apply a filter. - - This preprocessor expects features to contain `cell_center_xyz` and all - entries in params.features_to_modify, and that the leading dimension should - all be the same (num_cell_centers from SparseCenterSelector). - - We then modify all values in features that are specified in - params.features_to_modify by sorting them with the specified sort function - (specified by params.sort_by) operating on features.cell_center_xyz, and then - taking the top K (specified by params.num_cell_centers) along the first - dimension. - """ - - _REGISTERED_SORT_FUNCTIONS = ['distance'] - - @classmethod - def Params(cls): - p = super().Params() - p.Define('num_cell_centers', 512, 'The number of centers after filtering.') - p.Define( - 'sort_by', 'distance', 'A string specifying which sort function ' - 'to use. Currently we just support `distance`.') - p.Define('features_to_modify', [ - 'cell_center_xyz', 'anchor_centers', 'cell_points_xyz', 'cell_feature', - 'cell_points_padding' - ], 'A list of keys from the features dict to modify.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.sort_by not in self._REGISTERED_SORT_FUNCTIONS: - raise ValueError('{} not supported. We only support {}.'.format( - p.sort_by, self._REGISTERED_SORT_FUNCTIONS)) - if len(p.features_to_modify) < 1: - raise ValueError('Need to modify at least one feature.') - - def _SortByDistance(self, features): - dist = tf.linalg.norm(features.cell_center_xyz, axis=-1) - return tf.argsort(dist, axis=-1, direction='ASCENDING') - - def _Sort(self, features): - p = self.params - if p.sort_by == 'distance': - return self._SortByDistance(features) - else: - raise ValueError('Unsupported sort function: {}.'.format(p.sort_by)) - - def TransformFeatures(self, features): - p = self.params - sort_indices = self._Sort(features) - sort_indices_top_k = sort_indices[:p.num_cell_centers, ...] - - # Gather each of the relevant items - for key in p.features_to_modify: - shape = py_utils.GetShape(features[key]) - output_shape = [p.num_cell_centers] + shape[1:] - features[key] = py_utils.PadOrTrimTo( - tf.gather(features[key], sort_indices_top_k), output_shape) - return features - - def TransformShapes(self, shapes): - p = self.params - for key in p.features_to_modify: - shapes[key] = tf.TensorShape([p.num_cell_centers] + shapes[key][1:]) - return shapes - - def TransformDTypes(self, dtypes): - return dtypes - - -class TileAnchorBBoxes(Preprocessor): - """Creates anchor_bboxes given anchor_centers. - - This preprocessor expects features to contain the following keys: - - anchor_centers of shape [...base shape..., 3] - - Adds the following features: - anchor_bboxes: base_shape + [7] - Floating point anchor box - output containing the anchor boxes and the 7 floating point - values for each box that define the box (x, y, z, dx, dy, dz, phi). - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('anchor_box_dimensions', [], - 'List of anchor box sizes per center.') - p.Define('anchor_box_offsets', [], 'List of anchor box offsets per center.') - p.Define('anchor_box_rotations', [], - 'List of anchor box rotations per center.') - return p - - def TransformFeatures(self, features): - p = self.params - utils_3d = detection_3d_lib.Utils3D() - - assert p.anchor_box_dimensions - assert p.anchor_box_offsets - assert p.anchor_box_rotations - - base_shape = py_utils.GetShape(features.anchor_centers)[:-1] - num_box_per_center = len(p.anchor_box_dimensions) - - anchor_centers = tf.reshape(features.anchor_centers, [-1, 3]) - anchor_bboxes = utils_3d.MakeAnchorBoxes( - anchor_centers, tf.identity(p.anchor_box_dimensions), - tf.identity(p.anchor_box_offsets), tf.identity(p.anchor_box_rotations)) - features.anchor_bboxes = tf.reshape(anchor_bboxes, - base_shape + [num_box_per_center, 7]) - - return features - - def TransformShapes(self, shapes): - p = self.params - base_shape = shapes.anchor_centers[:-1] - num_box_per_center = len(p.anchor_box_dimensions) - shapes.anchor_bboxes = base_shape.concatenate([num_box_per_center, 7]) - return shapes - - def TransformDTypes(self, dtypes): - dtypes.anchor_bboxes = tf.float32 - return dtypes - - -class _AnchorBoxSettings: - """Helper class to parameterize and update anchor box settings.""" - # Implementations should fill out the following class members. - DIMENSION_PRIORS = [] - ROTATIONS = [] - CENTER_X_OFFSETS = [] - CENTER_Y_OFFSETS = [] - CENTER_Z_OFFSETS = [] - - @classmethod - def NumAnchors(cls): - return np.prod([ - len(cls.DIMENSION_PRIORS), - len(cls.ROTATIONS), - len(cls.CENTER_X_OFFSETS), - len(cls.CENTER_Y_OFFSETS), - len(cls.CENTER_Z_OFFSETS) - ]) - - @classmethod - def GenerateAnchorSettings(cls): - """Generate anchor settings. - - Returns: - A `NestedMap` containing three lists of the same length: - - anchor_box_dimensions - - anchor_box_rotations - - anchor_box_offsets - - These can be used with the TileAnchorBBoxes preprocessor. - """ - anchor_box_dimensions = [] - anchor_box_rotations = [] - anchor_box_offsets = [] - - # The following is equivalent to a formulation of itertools.product, but - # is explicitly listed for readability. - - # *Please note*: The ordering is important for ModelV2, which makes - # assumptions that the offset dimensions come first. - for cx in cls.CENTER_X_OFFSETS: - for cy in cls.CENTER_Y_OFFSETS: - for cz in cls.CENTER_Z_OFFSETS: - for rot in cls.ROTATIONS: - for dims in cls.DIMENSION_PRIORS: - anchor_box_dimensions += [dims] - anchor_box_rotations += [rot] - anchor_box_offsets += [(cx, cy, cz)] - - # Check one of the lists has entries. - assert anchor_box_dimensions - - return py_utils.NestedMap( - anchor_box_dimensions=anchor_box_dimensions, - anchor_box_rotations=anchor_box_rotations, - anchor_box_offsets=anchor_box_offsets) - - @classmethod - def Update(cls, params): - """Updates anchor box settings from input configuration lists. - - Given dimensions priors, rotations, and offsets, computes the cartesian - product of the settings. - - Args: - params: The KITTIAnchorExtractorBase.Params() object to update. - - Returns: - Params updated with the anchor settings. - - In total there are N combinations, where each (anchor_box_dimensions[i], - anchor_box_rotations[i], anchor_box_offsets[i]) for i in range(N) is an - option. - """ - p = params - settings = cls.GenerateAnchorSettings() - p.anchor_box_dimensions = settings.anchor_box_dimensions - p.anchor_box_rotations = settings.anchor_box_rotations - p.anchor_box_offsets = settings.anchor_box_offsets - return p - - -def MakeAnchorBoxSettings(dimension_priors, rotations, center_x_offsets, - center_y_offsets, center_z_offsets): - """Returns a configured class for setting anchor box settings.""" - - class CustomAnchorBoxSettings(_AnchorBoxSettings): - DIMENSION_PRIORS = dimension_priors - ROTATIONS = rotations - CENTER_X_OFFSETS = center_x_offsets - CENTER_Y_OFFSETS = center_y_offsets - CENTER_Z_OFFSETS = center_z_offsets - - return CustomAnchorBoxSettings - - -class SparseCarV1AnchorBoxSettings(_AnchorBoxSettings): - """Anchor box settings for training on Cars for Sparse models.""" - # Borrowed from PointPillar dimension prior for cars. - DIMENSION_PRIORS = [(1.6, 3.9, 1.56)] - - # 4 Rotations with axis aligned and both diagonals. - ROTATIONS = [0, np.pi / 2, np.pi / 4, 3 * np.pi / 4] - - # 25 offsets per anchor box with fixed z offset at -1. - CENTER_X_OFFSETS = np.linspace(-1.5, 1.5, 5) - CENTER_Y_OFFSETS = np.linspace(-1.5, 1.5, 5) - CENTER_Z_OFFSETS = [-1.] - - -class PointPillarAnchorBoxSettingsCar(_AnchorBoxSettings): - DIMENSION_PRIORS = [(1.6, 3.9, 1.56)] - ROTATIONS = [0, np.pi / 2] - # Fixed offset for every anchor box, based on a reading of the paper / code - # 0 offsets for x and y, and -1 for z. - CENTER_X_OFFSETS = [0.] - CENTER_Y_OFFSETS = [0.] - CENTER_Z_OFFSETS = [-1.] - - -class PointPillarAnchorBoxSettingsPed(PointPillarAnchorBoxSettingsCar): - DIMENSION_PRIORS = [(0.6, 0.8, 1.73)] - CENTER_Z_OFFSETS = [-0.6] - - -class PointPillarAnchorBoxSettingsCyc(PointPillarAnchorBoxSettingsCar): - DIMENSION_PRIORS = [(0.6, 1.76, 1.73)] - CENTER_Z_OFFSETS = [-0.6] - - -class PointPillarAnchorBoxSettingsPedCyc(PointPillarAnchorBoxSettingsCar): - DIMENSION_PRIORS = [(0.6, 0.8, 1.7), (0.6, 1.76, 1.73)] - CENTER_Z_OFFSETS = [-0.6] - - -class AnchorAssignment(Preprocessor): - """Perform anchor assignment on the features. - - This preprocessor expects features to contain the following keys: - - anchor_bboxes of shape [...base shape..., 7] - - labels.bboxes_3d - - labels.labels - - labels.bboxes_3d_mask - - Adds the following features: - - anchor_localization_residuals: base_shape + [7] floating point tensor of - residuals. The model is expected to regress against these residuals as - targets. The residuals can be converted back into bboxes using - detection_3d_lib.Utils3D.ResidualsToBBoxes. - assigned_gt_idx: base_shape - The corresponding index of the ground - truth bounding box for each anchor box in anchor_bboxes, anchors not - assigned will have idx be set to -1. - assigned_gt_bbox: base_shape + [7] - The corresponding ground - truth bounding box for each anchor box in anchor_bboxes. - assigned_gt_labels: base_shape - The assigned groundtruth label - for each anchor box. - assigned_gt_similarity_score: base_shape - The similarity score - for each assigned anchor box. - assigned_cls_mask: base_shape mask for classification loss per anchor. - This should be 1.0 if the anchor has a foreground or background - assignment; otherwise, it will be assigned to 0.0. - assigned_reg_mask: base_shape mask for regression loss per anchor. - This should be 1.0 if the anchor has a foreground assignment; - otherwise, it will be assigned to 0.0. - Note: background anchors do not have regression targets. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'foreground_assignment_threshold', 0.5, - 'Score (usually IOU) threshold for assigning a box as foreground.') - p.Define( - 'background_assignment_threshold', 0.35, - 'Score (usually IOU) threshold for assigning a box as background.') - return p - - def TransformFeatures(self, features): - p = self.params - utils_3d = detection_3d_lib.Utils3D() - - # anchor_bboxes will be returned with shape [#centers, #boxes_per_center, 7] - # flatten boxes here for matching. - base_shape = py_utils.GetShape(features.anchor_bboxes)[:-1] - anchor_bboxes = tf.reshape(features.anchor_bboxes, [-1, 7]) - - assigned_anchors = utils_3d.AssignAnchors( - anchor_bboxes, - features.labels.bboxes_3d, - features.labels.labels, - features.labels.bboxes_3d_mask, - foreground_assignment_threshold=p.foreground_assignment_threshold, - background_assignment_threshold=p.background_assignment_threshold) - - # Add new features. - features.assigned_gt_idx = tf.reshape(assigned_anchors.assigned_gt_idx, - base_shape) - features.assigned_gt_bbox = tf.reshape(assigned_anchors.assigned_gt_bbox, - base_shape + [7]) - features.assigned_gt_labels = tf.reshape( - assigned_anchors.assigned_gt_labels, base_shape) - features.assigned_gt_similarity_score = tf.reshape( - assigned_anchors.assigned_gt_similarity_score, base_shape) - features.assigned_cls_mask = tf.reshape(assigned_anchors.assigned_cls_mask, - base_shape) - features.assigned_reg_mask = tf.reshape(assigned_anchors.assigned_reg_mask, - base_shape) - - # Compute residuals. - features.anchor_localization_residuals = utils_3d.LocalizationResiduals( - features.anchor_bboxes, features.assigned_gt_bbox) - - return features - - def TransformShapes(self, shapes): - base_shape = shapes.anchor_bboxes[:-1] - box_shape = base_shape.concatenate([7]) - - shapes.anchor_localization_residuals = box_shape - shapes.assigned_gt_idx = base_shape - shapes.assigned_gt_bbox = box_shape - shapes.assigned_gt_labels = base_shape - shapes.assigned_gt_similarity_score = base_shape - shapes.assigned_cls_mask = base_shape - shapes.assigned_reg_mask = base_shape - return shapes - - def TransformDTypes(self, dtypes): - dtypes.anchor_localization_residuals = tf.float32 - dtypes.assigned_gt_idx = tf.int32 - dtypes.assigned_gt_bbox = tf.float32 - dtypes.assigned_gt_labels = tf.int32 - dtypes.assigned_gt_similarity_score = tf.float32 - dtypes.assigned_cls_mask = tf.float32 - dtypes.assigned_reg_mask = tf.float32 - return dtypes - - -class DropLaserPointsOutOfRange(Preprocessor): - """Drops laser points that are out of pre-defined x/y/z ranges. - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [P, 3] - - lasers.points_feature of shape [P, F] - - and optionally points_padding of shape [P] corresponding to the padding. - if points_padding is None, then all points are considered valid. - - Modifies the following features: - Removes or sets padding to 1 for all points outside a given range. Modifies - all items in the lasers subdictionary like lasers.points_xyz, - lasers.points_feature, lasers.points_padding, and optionally - lasers.points_label, lasers.points_bbox_id. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('keep_x_range', (-np.inf, np.inf), - 'Only points that have x coordinates within this range are kept.') - p.Define('keep_y_range', (-np.inf, np.inf), - 'Only points that have y coordinates within this range are kept.') - p.Define( - 'keep_z_range', (-np.inf, np.inf), - 'Only points that have z coordinates within this range are kept. ' - 'Approximate ground-removal can be performed by specifying a ' - 'lower-bound on the z-range.') - return p - - def TransformFeatures(self, features): - p = self.params - - points_xyz = features.lasers.points_xyz - if 'points_padding' in features.lasers: - points_mask = tf.cast(1 - features.lasers.points_padding, tf.bool) - else: - # All points are real, we keep points unpadded by applying boolean_mask - # on points_mask later. - points_mask = tf.ones_like(points_xyz[:, 0], dtype=tf.bool) - - min_x, max_x = p.keep_x_range - min_y, max_y = p.keep_y_range - min_z, max_z = p.keep_z_range - - # Short-circuit if all ranges are set to -inf, inf. - if (np.all(np.isneginf([min_x, min_y, min_z])) and - np.all(np.isposinf([max_x, max_y, max_z]))): - return features - - if min_x != -np.inf: - points_mask &= points_xyz[:, 0] >= min_x - if min_y != -np.inf: - points_mask &= points_xyz[:, 1] >= min_y - if min_z != -np.inf: - points_mask &= points_xyz[:, 2] >= min_z - - if max_x != np.inf: - points_mask &= points_xyz[:, 0] <= max_x - if max_y != np.inf: - points_mask &= points_xyz[:, 1] <= max_y - if max_z != np.inf: - points_mask &= points_xyz[:, 2] <= max_z - - if 'points_padding' in features.lasers: - # Suffices to just update the padding. - features.lasers.points_padding = 1. - tf.cast(points_mask, tf.float32) - else: - features.lasers = features.lasers.Transform( - _GetApplyPointMaskFn(points_mask)) - - return features - - def TransformShapes(self, shapes): - return shapes - - def TransformDTypes(self, dtypes): - return dtypes - - -class KITTIDropPointsOutOfFrustum(Preprocessor): - """Drops laser points that are outside of the camera frustum. - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [P, 3] - - lasers.points_feature of shape [P, F] - - images.velo_to_image_plane of shape [3, 4] - - images.width of shape [1] - - images.height of shape [1] - - and optionally points_padding of shape [P] corresponding to the padding. - if points_padding is None, then all points are considered valid. - - Modifies the following features: - lasers.points_xyz, lasers.points_feature, lasers.points_padding, and - optionally lasers.points_label, lasers.points_bbox_id so that - points outside the frustum have padding set to 1 or are removed. - """ - - def TransformFeatures(self, features): - # Drop points behind the car (behind x-axis = 0). - images = features.images - front_indices = features.lasers.points_xyz[:, 0] >= 0 - - if 'points_padding' not in features.lasers: - # Keep tensors unpadded and small using boolean_mask. - features.lasers.points_xyz = tf.boolean_mask(features.lasers.points_xyz, - front_indices) - features.lasers.points_feature = tf.boolean_mask( - features.lasers.points_feature, front_indices) - - # Drop those points outside the image plane. - points_image = geometry.PointsToImagePlane(features.lasers.points_xyz, - images.velo_to_image_plane) - in_image_plane = ( - (points_image[:, 0] >= 0) & - (points_image[:, 0] <= tf.cast(images.width, tf.float32)) & - (points_image[:, 1] >= 0) & - (points_image[:, 1] <= tf.cast(images.height, tf.float32))) - - if 'points_padding' in features.lasers: - # Update padding to only include front indices and in image plane. - points_mask = tf.cast(1 - features.lasers.points_padding, tf.bool) - points_mask &= front_indices - points_mask &= in_image_plane - features.lasers.points_padding = 1. - tf.cast(points_mask, tf.float32) - else: - features.lasers = features.lasers.Transform( - _GetApplyPointMaskFn(in_image_plane)) - return features - - def TransformShapes(self, shapes): - return shapes - - def TransformDTypes(self, dtypes): - return dtypes - - -class RandomWorldRotationAboutZAxis(Preprocessor): - """Rotates the world randomly as a form of data augmentation. - - Rotations are performed around the *z-axis*. This assumes that the car is - always level. In general, we'd like to instead rotate the car on the spot, - this would then make sense for cases where the car is on a slope. - - When there are leading dimensions, this will rotate the boxes with the same - transformation across all the frames. This is useful when the input is a - sequence of frames from the same run segment. - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [..., 3] - - labels.bboxes_3d of shape [..., 7] - - Modifies the following features: - lasers.points_xyz, labels.bboxes_3d with the same rotation applied to both. - - Adds the following features: - world_rot_z which contains the rotation applied to the example. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'max_rotation', None, - 'The rotation amount will be randomly picked from ' - '[-max_rotation, max_rotation).') - p.Define( - 'include_world_rot_z', True, - 'Whether to include the applied rotation as an additional tensor. ' - 'It can be helpful to disable this when using the preprocessor in a ' - 'way that expects the structure of the features to be the same ' - '(e.g., as a branch in tf.cond).') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.max_rotation is None: - raise ValueError('max_rotation needs to be specified, instead of None.') - - def TransformFeatures(self, features): - p = self.params - rot = tf.random.uniform((), - minval=-p.max_rotation, - maxval=p.max_rotation, - seed=p.random_seed) - - # Rotating about the z-axis is equal to experiencing yaw. - pose = [0., 0., 0., rot, 0., 0.] - - # Rotate points. - features.lasers.points_xyz = geometry.CoordinateTransform( - features.lasers.points_xyz, pose) - - # Rotate bboxes, note that heading has a special case. - bboxes_xyz = features.labels.bboxes_3d[..., :3] - bboxes_dims = features.labels.bboxes_3d[..., 3:6] - bboxes_rot = features.labels.bboxes_3d[..., 6:] - - bboxes_xyz = geometry.CoordinateTransform(bboxes_xyz, pose) - - # The heading correction should subtract rot from the bboxes rotations. - bboxes_rot = geometry.WrapAngleRad(bboxes_rot - rot) - - features.labels.bboxes_3d = tf.concat([bboxes_xyz, bboxes_dims, bboxes_rot], - axis=-1) - if p.include_world_rot_z: - features.world_rot_z = rot - return features - - def TransformShapes(self, shapes): - if self.params.include_world_rot_z: - shapes.world_rot_z = tf.TensorShape([]) - return shapes - - def TransformDTypes(self, dtypes): - if self.params.include_world_rot_z: - dtypes.world_rot_z = tf.float32 - return dtypes - - -class DropPointsOutOfFrustum(Preprocessor): - """Drops points outside of pre-defined theta / phi ranges. - - Note that the ranges for keep_phi_range can be negative, this is because the - phi values wrap around 2*pi. Thus, a valid range that filters the 90 deg - frontal field of view of the car can be specified as [-pi/4, pi/4]. - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [P, 7] - - lasers.points_feature of shape [P] - - Modifies the following features: - - lasers.points_xyz removing any points out of frustum. - - lasers.points_feature removing any points out of frustum. - - Note: We expect a downstream processor that filters out boxes with few points - to drop the corresponding bboxes. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('keep_theta_range', (0., np.pi), - 'Only points that have theta coordinates within this range.') - p.Define('keep_phi_range', (0., 2. * np.pi), - 'Only points that have phi coordinates within this range.') - return p - - def TransformFeatures(self, features): - p = self.params - - if 'points_padding' in features.lasers: - raise ValueError('DropPointsOutOfFrustum preprocessor does not support ' - 'padded lasers.') - - points_xyz = features.lasers.points_xyz - points_feature = features.lasers.points_feature - - min_theta, max_theta = p.keep_theta_range - if (min_theta < 0. or min_theta > np.pi or max_theta < 0. or - max_theta > np.pi): - raise ValueError('Valid values for theta are between 0 and pi, ' - 'keep_theta_range={}'.format(p.keep_theta_range)) - - if min_theta > max_theta: - raise ValueError('min_theta must be <= max_theta, ' - 'keep_theta_range={}'.format(p.keep_theta_range)) - - min_phi, max_phi = p.keep_phi_range - if (min_phi < -2. * np.pi or min_phi > 2. * np.pi or - max_phi < -2. * np.pi or max_phi > 2. * np.pi): - raise ValueError('Valid values for phi are between -2*pi and 2*pi,' - 'keep_phi_range={}'.format(p.keep_phi_range)) - - if min_phi > max_phi: - raise ValueError('min_phi must be <= max_phi, ' - 'keep_phi_range={}'.format(p.keep_phi_range)) - - _, theta, phi = tf.unstack( - geometry.SphericalCoordinatesTransform(points_xyz), axis=-1) - - # phi is returned in range [-pi, pi], we shift the values which are between - # [-pi, 0] to be [pi, 2pi] instead to make the logic below easier to follow. - # Hence, all phi values after this will be [0, 2pi]. - phi = tf.where(phi >= 0., phi, 2. * np.pi + phi) - - # Theta does not have circular boundary conditions, a simple check suffices. - points_mask = (theta >= min_theta) & (theta <= max_theta) - - if min_phi < 0. and max_phi < 0.: - # Both are less than zero, we just just add 2pi and will use the regular - # check. - min_phi += 2. * np.pi - max_phi += 2. * np.pi - - if min_phi < 0.: - # The minimum threshold is below 0, so we split into checking between - # (0 to min_phi) and (0 to max_phi). Note that min_phi is negative, but - # phi is always positive, so we take 2*pi + min_phi to get the range of - # appropriate values. - points_mask &= (phi >= (2. * np.pi + min_phi)) | (phi <= max_phi) - else: - # Both must be greater than 0 if we get to this condition. - assert min_phi >= 0. - assert max_phi >= 0. - points_mask &= (phi >= min_phi) & (phi <= max_phi) - - features.lasers.points_xyz = tf.boolean_mask(points_xyz, points_mask) - features.lasers.points_feature = tf.boolean_mask(points_feature, - points_mask) - return features - - def TransformShapes(self, shapes): - return shapes - - def TransformDTypes(self, dtypes): - return dtypes - - -class DropBoxesOutOfRange(Preprocessor): - """Drops boxes outside of pre-defined x/y/z ranges (boundaries inclusive). - - This preprocessor expects features to contain the following keys: - - labels.bboxes_3d of shape [N, 7] - - labels.bboxes_3d_mask of shape [N] - - Modifies the following features: - - labels.bboxes_3d_mask to mask out any additional boxes. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('keep_x_range', (-np.inf, np.inf), - 'Only boxes that have x coordinates within this range are kept.') - p.Define('keep_y_range', (-np.inf, np.inf), - 'Only boxes that have y coordinates within this range are kept.') - p.Define('keep_z_range', (-np.inf, np.inf), - 'Only boxes that have z coordinates within this range are kept.') - return p - - def TransformFeatures(self, features): - p = self.params - - min_x, max_x = p.keep_x_range - min_y, max_y = p.keep_y_range - min_z, max_z = p.keep_z_range - - # Short-circuit if all ranges are set to -inf, inf. - if (np.all(np.isneginf([min_x, min_y, min_z])) and - np.all(np.isposinf([max_x, max_y, max_z]))): - return features - - # For each bounding box, compute whether any of its extrema - # fall outside of the range. - bboxes_3d_corners = geometry.BBoxCorners( - features.labels.bboxes_3d[tf.newaxis, ...])[0] - bboxes_3d_corners = py_utils.HasShape(bboxes_3d_corners, [-1, 8, 3]) - - min_bbox_x = tf.reduce_min(bboxes_3d_corners[:, :, 0], axis=-1) - max_bbox_x = tf.reduce_max(bboxes_3d_corners[:, :, 0], axis=-1) - - min_bbox_y = tf.reduce_min(bboxes_3d_corners[:, :, 1], axis=-1) - max_bbox_y = tf.reduce_max(bboxes_3d_corners[:, :, 1], axis=-1) - - min_bbox_z = tf.reduce_min(bboxes_3d_corners[:, :, 2], axis=-1) - max_bbox_z = tf.reduce_max(bboxes_3d_corners[:, :, 2], axis=-1) - - mask = ( - tf.math.logical_and(min_bbox_x >= min_x, max_bbox_x <= max_x) - & tf.math.logical_and(min_bbox_y >= min_y, max_bbox_y <= max_y) - & tf.math.logical_and(min_bbox_z >= min_z, max_bbox_z <= max_z)) - - max_num_boxes = py_utils.GetShape(features.labels.bboxes_3d_mask) - mask = py_utils.HasShape(mask, max_num_boxes) - - features.labels.bboxes_3d_mask *= tf.cast(mask, tf.float32) - return features - - def TransformShapes(self, shapes): - return shapes - - def TransformDTypes(self, dtypes): - return dtypes - - -class PadLaserFeatures(Preprocessor): - """Pads laser features so that the dimensions are fixed. - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [P, 3] - - lasers.points_feature of shape [P, F] - - and optionally points_padding of shape [P] corresponding to the padding. - if points_padding is None, then all points are considered valid. - - Modifies the following features: - lasers.points_xyz and lasers.points_feature to add padding. - Optionally also modifies lasers.points_label and lasers.points_bbox_id - if they exist to add padding. - Modifies/adds the following features: - labels.points_padding of shape [P] representing the padding. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('max_num_points', 128500, - 'Max number of points to pad the points to.') - return p - - def TransformFeatures(self, features): - p = self.params - - if 'points_padding' in features.lasers: - points_mask = 1 - features.lasers.points_padding - points_mask = tf.cast(points_mask, tf.bool) - features.lasers = features.lasers.Transform( - _GetApplyPointMaskFn(points_mask)) - - npoints = tf.shape(features.lasers.points_xyz)[0] - features.lasers.points_padding = tf.ones([npoints]) - - shuffled_idx = tf.range(npoints) - shuffled_idx = tf.random.shuffle(shuffled_idx, seed=p.random_seed) - - def _PadOrTrimFn(points_tensor): - # Shuffle before trimming so we have a random sampling - points_tensor = tf.gather(points_tensor, shuffled_idx) - return py_utils.PadOrTrimTo(points_tensor, [p.max_num_points] + - points_tensor.shape[1:].as_list()) - - features.lasers = features.lasers.Transform(_PadOrTrimFn) - features.lasers.points_padding = 1.0 - features.lasers.points_padding - return features - - def TransformShapes(self, shapes): - p = self.params - - def _TransformShape(points_shape): - return tf.TensorShape([p.max_num_points] + points_shape[1:].as_list()) - - shapes.lasers = shapes.lasers.Transform(_TransformShape) - shapes.lasers.points_padding = tf.TensorShape([p.max_num_points]) - return shapes - - def TransformDTypes(self, dtypes): - dtypes.lasers.points_padding = tf.float32 - return dtypes - - -class WorldScaling(Preprocessor): - """Scale the world randomly as a form of data augmentation. - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [P, 3] - - labels.bboxes_3d of shape [L, 7] - - Modifies the following features: - lasers.points_xyz, labels.bboxes_3d with the same scaling applied to both. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('scaling', None, 'The scaling range.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.scaling is None: - raise ValueError('scaling needs to be specified, instead of None.') - if len(p.scaling) != 2: - raise ValueError('scaling needs to be a list of two elements.') - - def TransformFeatures(self, features): - p = self.params - scaling = tf.random.uniform((), - minval=p.scaling[0], - maxval=p.scaling[1], - seed=p.random_seed, - dtype=features.lasers.points_xyz.dtype) - - # Scale points [num_points, 3]. - features.lasers.points_xyz *= scaling - - # Scaling bboxes (location and dimensions). - bboxes_xyz = features.labels.bboxes_3d[..., :3] * scaling - bboxes_dims = features.labels.bboxes_3d[..., 3:6] * scaling - bboxes_rot = features.labels.bboxes_3d[..., 6:] - - features.labels.bboxes_3d = tf.concat([bboxes_xyz, bboxes_dims, bboxes_rot], - axis=-1) - return features - - def TransformShapes(self, shapes): - return shapes - - def TransformDTypes(self, dtypes): - return dtypes - - -class RandomDropLaserPoints(Preprocessor): - """Randomly dropout laser points and the corresponding features. - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [P, 3] - - lasers.points_feature of shape [P, K] - - - Modifies the following features: - lasers.points_xyz, lasers.points_feature. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('keep_prob', 0.95, 'Probability for keeping points.') - return p - - def TransformFeatures(self, features): - p = self.params - if 'points_padding' in features.lasers: - points_mask = 1 - features.lasers.points_padding - points_xyz = tf.boolean_mask(features.lasers.points_xyz, points_mask) - points_feature = tf.boolean_mask(features.lasers.points_feature, - points_mask) - else: - points_xyz = features.lasers.points_xyz - points_feature = features.lasers.points_feature - - num_points, _ = py_utils.GetShape(features.lasers.points_xyz) - - pts_keep_sample_prob = tf.random.uniform([num_points], - minval=0, - maxval=1, - seed=p.random_seed) - pts_keep_mask = pts_keep_sample_prob < p.keep_prob - - points_xyz = tf.boolean_mask(points_xyz, pts_keep_mask) - points_feature = tf.boolean_mask(points_feature, pts_keep_mask) - - if 'points_padding' in features.lasers: - features.lasers.points_xyz = py_utils.PadOrTrimTo( - points_xyz, tf.shape(features.lasers.points_xyz)) - features.lasers.points_feature = py_utils.PadOrTrimTo( - points_feature, tf.shape(features.lasers.points_feature)) - total_points = tf.shape(points_xyz)[0] - features.lasers.points_padding = 1.0 - py_utils.PadOrTrimTo( - tf.ones([total_points]), tf.shape(features.lasers.points_padding)) - else: - features.lasers.points_xyz = points_xyz - features.lasers.points_feature = points_feature - - return features - - def TransformShapes(self, shapes): - return shapes - - def TransformDTypes(self, dtypes): - return dtypes - - -class RandomFlipY(Preprocessor): - """Flip the world along axis Y as a form of data augmentation. - - When there are leading dimensions, this will flip the boxes with the same - transformation across all the frames. This is useful when the input is a - sequence of frames from the same run segment. - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [..., 3] - - labels.bboxes_3d of shape [..., 7] - - Modifies the following features: - lasers.points_xyz, labels.bboxes_3d with the same flipping applied to both. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('flip_probability', 0.5, 'Probability of flipping.') - return p - - def TransformFeatures(self, features): - p = self.params - threshold = 1. - p.flip_probability - choice = tf.random.uniform( - (), minval=0.0, maxval=1.0, seed=p.random_seed) >= threshold - - # Flip points - points_xyz = features.lasers.points_xyz - points_y = tf.where(choice, -points_xyz[..., 1:2], points_xyz[..., 1:2]) - features.lasers.points_xyz = tf.concat( - [points_xyz[..., 0:1], points_y, points_xyz[..., 2:3]], axis=-1) - - # Flip boxes - bboxes_xyz = features.labels.bboxes_3d[..., :3] - bboxes_y = tf.where(choice, -bboxes_xyz[..., 1:2], bboxes_xyz[..., 1:2]) - bboxes_xyz = tf.concat( - [bboxes_xyz[..., 0:1], bboxes_y, bboxes_xyz[..., 2:3]], axis=-1) - # Compensate rotation. - bboxes_dims = features.labels.bboxes_3d[..., 3:6] - bboxes_rot = features.labels.bboxes_3d[..., 6:] - bboxes_rot = tf.where(choice, geometry.WrapAngleRad(-bboxes_rot), - bboxes_rot) - features.labels.bboxes_3d = tf.concat([bboxes_xyz, bboxes_dims, bboxes_rot], - axis=-1) - return features - - def TransformShapes(self, shapes): - return shapes - - def TransformDTypes(self, dtypes): - return dtypes - - -class GlobalTranslateNoise(Preprocessor): - """Add global translation noise of xyz coordinates to points and boxes. - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [P, 3] - - labels.bboxes_3d of shape [L, 7] - - Modifies the following features: - lasers.points_xyz, labels.bboxes_3d with the same - random translation noise applied to both. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('noise_std', [0.2, 0.2, 0.2], - 'Standard deviation of translation noise per axis.') - return p - - def TransformFeatures(self, features): - p = self.params - # Use three different seeds but the same base seed so - # that the values are different. - base_seed = p.random_seed - x_seed = base_seed - y_seed = None if base_seed is None else base_seed + 1 - z_seed = None if base_seed is None else base_seed + 2 - random_translate_x = tf.random.normal((), - mean=0.0, - stddev=p.noise_std[0], - seed=x_seed) - random_translate_y = tf.random.normal((), - mean=0.0, - stddev=p.noise_std[1], - seed=y_seed) - random_translate_z = tf.random.normal((), - mean=0.0, - stddev=p.noise_std[2], - seed=z_seed) - - pose = tf.stack([ - random_translate_x, random_translate_y, random_translate_z, 0.0, 0.0, - 0.0 - ], - axis=0) - - # Translate points. - points_xyz = features.lasers.points_xyz - features.lasers.points_xyz = geometry.CoordinateTransform(points_xyz, pose) - - # Translate boxes - bboxes_xyz = features.labels.bboxes_3d[..., :3] - bboxes_xyz = geometry.CoordinateTransform(bboxes_xyz, pose) - features.labels.bboxes_3d = tf.concat( - [bboxes_xyz, features.labels.bboxes_3d[..., 3:]], axis=-1) - return features - - def TransformShapes(self, shapes): - return shapes - - def TransformDTypes(self, dtypes): - return dtypes - - -class RandomBBoxTransform(Preprocessor): - """Randomly transform bounding boxes and the points inside them. - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [P, 3] - - lasers.points_feature of shape [P, K] - - lasers.points_padding of shape [P] - - labels.bboxes_3d of shape [L, 7] - - labels.bboxes_3d_mask of shape [L] - - Modifies the following features: - lasers.points_{xyz,feature,padding}, labels.bboxes_3d with the - transformed bounding boxes and points. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'max_rotation', None, - 'The rotation amount will be randomly picked from ' - '[-max_rotation, max_rotation).') - # At the moment we don't use this because it can cause boxes to collide with - # each other. We need to compute box intersections when deciding whether to - # apply the translation jitter. Theoretically we should also do this for - # rotation. - p.Define('noise_std', [0.0, 0.0, 0.0], - 'Standard deviation of translation noise per axis.') - p.Define( - 'max_scaling', None, - 'When max_scaling is not none, delta parameters s_x, s_y, s_z are ' - 'drawn from [-max_scaling[i], max_scaling[i]] where i is in [0, 3].') - p.Define( - 'max_shearing', None, - 'When max_shearing is not none, shearing parameters sh_x^y, sh_x^z, ' - 'sh_y^x, sh_y^z, sh_z^x, sh_z^y are drawn from ' - '[-max_shearing[i], max_shearing[i]], where i is in [0, 5].') - p.Define( - 'max_num_points_per_bbox', 16384, - 'The maximum number of points that fall within a bounding box. ' - 'Bounding boxes with more points than this value will ' - 'have some points droppped.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.max_rotation is None: - raise ValueError('max_rotation needs to be specified, instead of None.') - if p.max_scaling is not None: - if len(p.max_scaling) != 3: - raise ValueError('max_scaling needs to be specified as either None or ' - 'list of 3 floating point numbers, instead of {}.' - ''.format(p.max_scaling)) - if p.max_shearing is not None: - if len(p.max_shearing) != 6: - raise ValueError('max_shearing needs to be specified as either None or ' - 'list of 6 floating point numbers, instead of {}.' - ''.format(p.max_shearing)) - - def _Foreground(self, features, points_xyz, points_feature, real_bboxes_3d, - points_in_bbox_mask, rotation, translate_pose, transform_fn): - """Extract and transform foreground points and features.""" - out_bbox_xyz, out_bbox_feature, out_bbox_mask = self._ForLoopBuffers( - features) - - # Only iterate over the actual number of boxes in the scene. - actual_num_bboxes = tf.reduce_sum( - tf.cast(features.labels.bboxes_3d_mask, tf.int32)) - - ret = py_utils.ForLoop( - body=transform_fn, - start=0, - limit=actual_num_bboxes, - delta=1, - loop_state=py_utils.NestedMap( - points_xyz=points_xyz, - points_feature=points_feature, - bboxes_3d=real_bboxes_3d, - points_in_bbox_mask=points_in_bbox_mask, - rotation=rotation, - translate_pose=translate_pose, - out_bbox_points=out_bbox_xyz, - out_bbox_feature=out_bbox_feature, - out_bbox_mask=out_bbox_mask)) - - # Gather all of the transformed points and features - out_bbox_xyz = tf.reshape(ret.out_bbox_points, [-1, 3]) - num_features = features.lasers.points_feature.shape[-1] - out_bbox_feature = tf.reshape(ret.out_bbox_feature, [-1, num_features]) - out_bbox_mask = tf.cast(tf.reshape(ret.out_bbox_mask, [-1]), tf.bool) - fg_xyz = tf.boolean_mask(out_bbox_xyz, out_bbox_mask) - fg_feature = tf.boolean_mask(out_bbox_feature, out_bbox_mask) - return fg_xyz, fg_feature - - def _Background(self, points_xyz, points_feature, points_in_bbox_mask): - # If a point is in any bounding box, it is a foreground point. - foreground_points_mask = tf.reduce_any(points_in_bbox_mask, axis=-1) - # All others are background. We rotate all of the foreground points to - # final_points_* and keep the background points unchanged - background_points_mask = tf.math.logical_not(foreground_points_mask) - background_points_xyz = tf.boolean_mask(points_xyz, background_points_mask) - background_points_feature = tf.boolean_mask(points_feature, - background_points_mask) - return background_points_xyz, background_points_feature - - def _ForLoopBuffers(self, features): - """Create and return the buffers for the for loop.""" - p = self.params - bboxes_3d = features.labels.bboxes_3d - - # Compute the shapes and create the buffers for the For loop. - max_num_bboxes = tf.shape(bboxes_3d)[0] - per_box_shape = [max_num_bboxes, p.max_num_points_per_bbox, 3] - out_bbox_points = inplace_ops.empty( - per_box_shape, dtype=tf.float32, init=True) - - num_features = features.lasers.points_feature.shape[-1] - bbox_feature_shape = [ - max_num_bboxes, p.max_num_points_per_bbox, num_features - ] - out_bbox_feature = inplace_ops.empty( - bbox_feature_shape, dtype=tf.float32, init=True) - - per_box_mask_shape = [max_num_bboxes, p.max_num_points_per_bbox] - out_bbox_mask = inplace_ops.empty( - per_box_mask_shape, dtype=tf.float32, init=True) - - return out_bbox_points, out_bbox_feature, out_bbox_mask - - def TransformFeatures(self, features): - p = self.params - - num_features = features.lasers.points_feature.shape[-1] - - def Transform(i, state): - """Transform the points in bounding box `i`.""" - state.points_xyz = tf.reshape(state.points_xyz, [-1, 3]) - bbox_mask = tf.reshape(state.points_in_bbox_mask[:, i], [-1]) - - # Fetch only the points in the bounding box. - points_xyz_masked = tf.boolean_mask(state.points_xyz, bbox_mask) - points_feature_masked = tf.boolean_mask(state.points_feature, bbox_mask) - - num_points = tf.shape(points_xyz_masked)[0] - - # TODO(vrv): Fold the following into a single transformation - # matrix. - # - # Translate the box to the origin, then rotate the desired - # rotation angle. - translation_vec = state.bboxes_3d[i, 0:3] - rotation_vec = [state.rotation[i], 0., 0.] - pose = tf.concat([-translation_vec, rotation_vec], axis=0) - points_xyz_adj = geometry.CoordinateTransform(points_xyz_masked, pose) - if p.max_scaling is not None or p.max_shearing is not None: - # Translate the points in the bounding box by moving dz/2 so that the - # bottom of the bounding box is at Z = 0 when any of the two - # (max_scaling or max_shearing) is not None - translation_scale_or_shear = tf.stack( - [0., 0., state.bboxes_3d[i, 5] / 2], axis=0) - pose1 = tf.concat([translation_scale_or_shear, [0., 0., 0.]], axis=0) - points_xyz_adj = geometry.CoordinateTransform(points_xyz_adj, pose1) - else: - translation_scale_or_shear = tf.stack([0., 0., 0.], axis=0) - - if p.max_scaling is not None: - # Perform scaling to the point cloud - # Scaling matrix - # [[s_x+1 0 0] - # [ 0 s_y+1 0] - # [ 0 0 s_z+1]] - sx = tf.random.uniform([], - minval=-p.max_scaling[0], - maxval=p.max_scaling[0], - seed=p.random_seed) - sy = tf.random.uniform([], - minval=-p.max_scaling[1], - maxval=p.max_scaling[1], - seed=p.random_seed) - sz = tf.random.uniform([], - minval=-p.max_scaling[2], - maxval=p.max_scaling[2], - seed=p.random_seed) - scaling_matrix = tf.stack( - [[sx + 1., 0., 0.], [0., sy + 1., 0.], [0., 0., sz + 1.]], axis=0) - - points_xyz_adj = tf.einsum('ij,kj->ki', scaling_matrix, points_xyz_adj) - - if p.max_shearing is not None: - # Perform shearing to the point cloud - # Shearing matrix - # [[1 sh_x^y sh_x^z] - # [sh_y^x 1 sh_y^z] - # [sh_z^x sh_z^y 1 ]] - sxy = tf.random.uniform([], - minval=-p.max_shearing[0], - maxval=p.max_shearing[0], - seed=p.random_seed) - sxz = tf.random.uniform([], - minval=-p.max_shearing[1], - maxval=p.max_shearing[1], - seed=p.random_seed) - syx = tf.random.uniform([], - minval=-p.max_shearing[2], - maxval=p.max_shearing[2], - seed=p.random_seed) - syz = tf.random.uniform([], - minval=-p.max_shearing[3], - maxval=p.max_shearing[3], - seed=p.random_seed) - szx = tf.random.uniform([], - minval=-p.max_shearing[4], - maxval=p.max_shearing[4], - seed=p.random_seed) - szy = tf.random.uniform([], - minval=-p.max_shearing[5], - maxval=p.max_shearing[5], - seed=p.random_seed) - shearing_matrix = tf.stack( - [[1., sxy, sxz], [syx, 1., syz], [szx, szy, 1.]], axis=0) - points_xyz_adj = tf.einsum('ij,kj->ki', shearing_matrix, points_xyz_adj) - - # Translate the points back, adding noise if needed. - translation_with_noise = ( - translation_vec - translation_scale_or_shear + - state.translate_pose[i]) - pose2 = tf.concat([translation_with_noise, [0., 0., 0.]], axis=0) - final_points_xyz = geometry.CoordinateTransform(points_xyz_adj, pose2) - - # final_points_xyz is an [M, 3] Tensor where M is the number of points in - # the box. - points_mask = tf.ones([num_points], dtype=tf.float32) - - final_points_xyz = py_utils.PadOrTrimTo(final_points_xyz, - [p.max_num_points_per_bbox, 3]) - final_points_feature = py_utils.PadOrTrimTo( - points_feature_masked, [p.max_num_points_per_bbox, num_features]) - points_mask = py_utils.PadOrTrimTo(points_mask, - [p.max_num_points_per_bbox]) - state.out_bbox_points = inplace_ops.alias_inplace_update( - state.out_bbox_points, [i], tf.expand_dims(final_points_xyz, 0)) - state.out_bbox_feature = inplace_ops.alias_inplace_update( - state.out_bbox_feature, [i], tf.expand_dims(final_points_feature, 0)) - state.out_bbox_mask = inplace_ops.alias_inplace_update( - state.out_bbox_mask, [i], tf.expand_dims(points_mask, 0)) - - return state - - # Get the points and features that reside in boxes. - if 'points_padding' in features.lasers: - points_mask = 1 - features.lasers.points_padding - points_xyz = tf.boolean_mask(features.lasers.points_xyz, points_mask) - points_feature = tf.boolean_mask(features.lasers.points_feature, - points_mask) - else: - points_xyz = features.lasers.points_xyz - points_feature = features.lasers.points_feature - - # Fetch real bounding boxes and compute point mask. - real_bboxes_3d = tf.boolean_mask(features.labels.bboxes_3d, - features.labels.bboxes_3d_mask) - points_in_bbox_mask = geometry.IsWithinBBox3D(points_xyz, real_bboxes_3d) - - # Choose a random rotation for every real box. - num_boxes = tf.shape(real_bboxes_3d)[0] - rotation = tf.random.uniform([num_boxes], - minval=-p.max_rotation, - maxval=p.max_rotation, - seed=p.random_seed) - - base_seed = p.random_seed - x_seed = base_seed - y_seed = None if base_seed is None else base_seed + 1 - z_seed = None if base_seed is None else base_seed + 2 - random_translate_x = tf.random.normal([num_boxes], - mean=0.0, - stddev=p.noise_std[0], - seed=x_seed) - random_translate_y = tf.random.normal([num_boxes], - mean=0.0, - stddev=p.noise_std[1], - seed=y_seed) - random_translate_z = tf.random.normal([num_boxes], - mean=0.0, - stddev=p.noise_std[2], - seed=z_seed) - - translate_pose = tf.stack( - [random_translate_x, random_translate_y, random_translate_z], axis=1) - - fg_xyz, fg_feature = self._Foreground(features, points_xyz, points_feature, - real_bboxes_3d, points_in_bbox_mask, - rotation, translate_pose, Transform) - - # Concatenate them with the background points and features. - bg_xyz, bg_feature = self._Background(points_xyz, points_feature, - points_in_bbox_mask) - all_points = tf.concat([bg_xyz, fg_xyz], axis=0) - all_features = tf.concat([bg_feature, fg_feature], axis=0) - - # Shuffle the points/features randomly. - all_points, all_features = _ConsistentShuffle((all_points, all_features), - p.random_seed) - - # Padding should technically be unnecessary: the number of points before and - # after should be the same, but in practice we sometimes seem to drop a few - # points, and so we pad to make the shape fixed. - # - # TODO(vrv): Identify the source of this problem and then assert a shape - # matching check. - if 'points_padding' in features.lasers: - features.lasers.points_xyz = py_utils.PadOrTrimTo( - all_points, tf.shape(features.lasers.points_xyz)) - features.lasers.points_feature = py_utils.PadOrTrimTo( - all_features, tf.shape(features.lasers.points_feature)) - total_points = tf.shape(all_points)[0] - features.lasers.points_padding = 1.0 - py_utils.PadOrTrimTo( - tf.ones([total_points]), tf.shape(features.lasers.points_padding)) - else: - features.lasers.points_xyz = all_points - features.lasers.points_feature = all_features - - # Translate noise. - bboxes_xyz = real_bboxes_3d[..., :3] - bboxes_xyz += translate_pose[..., :3] - - bboxes_dim = real_bboxes_3d[..., 3:6] - # Rotate bboxes by their corresponding rotation. - bboxes_rot = real_bboxes_3d[..., 6:] - bboxes_rot -= rotation[:, tf.newaxis] - features.labels.bboxes_3d = py_utils.PadOrTrimTo( - tf.concat([bboxes_xyz, bboxes_dim, bboxes_rot], axis=-1), - tf.shape(features.labels.bboxes_3d)) - features.labels.bboxes_3d_mask = py_utils.PadOrTrimTo( - tf.ones(tf.shape(real_bboxes_3d)[0]), - tf.shape(features.labels.bboxes_3d_mask)) - return features - - def TransformShapes(self, shapes): - return shapes - - def TransformDTypes(self, dtypes): - return dtypes - - -class GroundTruthAugmentor(Preprocessor): - """Augment bounding box labels and points from a database. - - This preprocessor expects features to contain the following keys: - lasers.points_xyz of shape [P, 3] - - lasers.points_feature of shape [P, K] - - lasers.points_padding of shape [P] - - labels.bboxes_3d of shape [L, 7] - - labels.bboxes_3d_mask of shape [L] - - labels.labels of shape [L] - - Modifies the above features so that additional objects from - a groundtruth database are added. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'groundtruth_database', None, - 'If not None, loads groundtruths from this database and adds ' - 'them to the current scene. Groundtruth database is expected ' - 'to be a TFRecord of KITTI or Waymo crops.') - p.Define( - 'num_db_objects', None, - 'Number of objects in the database. Because we use TFRecord ' - 'we cannot easily query the number of objects efficiencly.') - p.Define('max_num_points_per_bbox', 2048, - 'Maximum number of points in each bbox to augment with.') - p.Define( - 'filter_min_points', 0, - 'Minimum number of points each database object must have ' - 'to be included in an example.') - p.Define( - 'filter_max_points', None, - 'Maximum number of points each database object must have ' - 'to be included in an example.') - p.Define( - 'difficulty_sampling_probability', None, - 'Probability for sampling ground truth example whose difficulty ' - 'equals {0, 1, 2, 3, ...}. Example: [1.0, 1.0, 1.0, 1.0] for ' - 'uniform sampling 4 different difficulties. Default value is ' - 'None = uniform sampling for all difficulties.') - p.Define( - 'class_sampling_probability', None, - 'Probability for sampling ground truth example based on its class index' - ' Example: For KITTI classes are [Background, Car, Van, Truck, ' - 'Pedestrian, Person_sitting, Cyclist, Tram, Misc, DontCare], using ' - 'probability vector [0., 1.0, 1.0, 0., 0., 0., 0.,0., 0., 0.], we ' - 'uniformly sampling Car and Van. Default value is None: Uses ' - 'label_filter flag and does not sample based on class.') - p.Define('filter_min_difficulty', 0, - 'Filter ground truth boxes whose difficulty is < this value.') - p.Define('max_augmented_bboxes', 15, - 'Maximum number of augmented bounding boxes per scene.') - p.Define( - 'label_filter', [], - 'A list where if specified, only examples of these label integers will ' - 'be included in an example.') - p.Define( - 'batch_mode', False, 'Bool value to control whether the whole' - 'groundtruth database is loaded or partially loaded to save memory' - 'usage. Setting to False loads the whole ground truth database into ' - 'memory. Otherwise, only a fraction of the data will be loaded into ' - 'the memory.') - return p - - def _ReadDB(self, file_patterns): - """Read the groundtruth database and return as a NestedMap of Tensors.""" - p = self.params - - def Process(record): - """Process a groundtruth record.""" - feature_map = { - 'num_points': tf.io.FixedLenFeature((), tf.int64, 0), - 'points': tf.io.VarLenFeature(dtype=tf.float32), - 'points_feature': tf.io.VarLenFeature(dtype=tf.float32), - 'bbox_3d': tf.io.VarLenFeature(dtype=tf.float32), - 'label': tf.io.FixedLenFeature((), tf.int64, 0), - 'difficulty': tf.io.FixedLenFeature((), tf.int64, 0), - 'text': tf.io.VarLenFeature(dtype=tf.string), - } - - example_data = tf.io.parse_single_example(record, feature_map) - num_points = example_data['num_points'] - - points = tf.reshape(_Dense(example_data['points']), [num_points, 3]) - features = tf.reshape( - _Dense(example_data['points_feature']), [num_points, 1]) - points_mask = tf.ones(num_points, dtype=tf.bool) - - # TODO(vrv): Use random selection instead of first N points. - points = py_utils.PadOrTrimTo(points, [p.max_num_points_per_bbox, 3]) - features = py_utils.PadOrTrimTo(features, [p.max_num_points_per_bbox, 1]) - points_mask = py_utils.PadOrTrimTo(points_mask, - [p.max_num_points_per_bbox]) - - bboxes_3d = tf.reshape(_Dense(example_data['bbox_3d']), [7]) - label = tf.cast(example_data['label'], tf.int32) - difficulty = tf.cast(example_data['difficulty'], tf.int32) - return (points, features, points_mask, bboxes_3d, label, difficulty) - - if p.batch_mode: - # Prepare dataset for ground truth bounding boxes. Randomly shuffle the - # file patterns. - file_count = len(tf.io.gfile.glob(file_patterns)) - dataset = tf.stateless_list_files(file_patterns) - dataset = dataset.apply(tf.stateless_cache_dataset()) - dataset = dataset.apply( - tf.stateless_shuffle_dataset( - buffer_size=file_count, reshuffle_each_iteration=True)) - dataset = dataset.interleave( - tf.data.TFRecordDataset, cycle_length=10, num_parallel_calls=10) - dataset = dataset.repeat() - # Only prefetch a few objects from the database to reduce memory - # consumption. - dataset = dataset.map(Process, num_parallel_calls=10) - # We need more bboxes than max_augmented_bboxes in a batch, because some - # of the boxes are filtered out. - dataset = dataset.batch(p.max_augmented_bboxes * 10) - dataset = dataset.apply(tf.stateless_cache_dataset()).prefetch( - p.max_augmented_bboxes * 30) - else: - # Prepare dataset for ground truth bounding boxes. - dataset = tf.stateless_list_files(file_patterns) - dataset = dataset.interleave( - tf.data.TFRecordDataset, cycle_length=10, num_parallel_calls=10) - # Read the entire dataset into memory. - dataset = dataset.take(p.num_db_objects) - dataset = dataset.map(Process, num_parallel_calls=10) - # We batch the output of the dataset into a very large Tensor, then cache - # it in memory. - dataset = dataset.batch(p.num_db_objects) - dataset = dataset.apply(tf.stateless_cache_dataset()).repeat() - - iterator = dataset.make_one_shot_iterator() - input_batch = iterator.get_next() - - (db_points_xyz, db_points_feature, db_points_mask, db_bboxes, db_labels, - db_difficulties) = input_batch - return py_utils.NestedMap( - points_xyz=db_points_xyz, - points_feature=db_points_feature, - points_mask=db_points_mask, - bboxes_3d=db_bboxes, - labels=db_labels, - difficulties=db_difficulties) - - def _CreateExampleFilter(self, db): - """Construct db example filter. - - Args: - db: NestedMap of the following Tensors: points_mask - [N, P] - The points - mask for every object in the database, where N is the number of objects - and P is the maximum number of points per object. labels - [N] - int32 - Label for each object in the database. difficulties - [N] - int32 - Difficulty for each label in the database. - - Returns: - A [N] boolean Tensor for each object in the database, True if - that corresponding object passes the filter. - """ - p = self.params - db_points_mask = db.points_mask - db_label = db.labels - db_difficulty = db.difficulties - - num_objects_in_database = tf.shape(db_points_mask)[0] - - # Filter number of objects. - points_per_object = tf.reduce_sum(tf.cast(db_points_mask, tf.int32), axis=1) - example_filter = points_per_object >= p.filter_min_points - if p.filter_max_points: - example_filter = tf.math.logical_and( - example_filter, points_per_object <= p.filter_max_points) - - if p.difficulty_sampling_probability is not None: - # Sample db based on difficulity of each example. - sampling_prob = p.difficulty_sampling_probability - db_difficulty_probability = tf.zeros_like(db_difficulty, dtype=tf.float32) - for difficulty_idx, difficulty_prob in enumerate(sampling_prob): - db_difficulty_probability += ( - tf.cast(tf.equal(db_difficulty, difficulty_idx), tf.float32) * - difficulty_prob) - - sampled_filter = tf.random.uniform( - tf.shape(example_filter), - minval=0, - maxval=1, - dtype=tf.float32, - seed=p.random_seed) - sampled_filter = sampled_filter < db_difficulty_probability - example_filter &= sampled_filter - else: - # Filter out db examples below min difficulty - example_filter = tf.math.logical_and( - example_filter, db_difficulty >= p.filter_min_difficulty) - - example_filter = tf.reshape(example_filter, [num_objects_in_database]) - db_label = tf.reshape(db_label, [num_objects_in_database]) - if p.class_sampling_probability is not None: - # Sample example based on its class probability. - sampling_prob = p.class_sampling_probability - db_class_probability = tf.zeros_like(db_label, dtype=tf.float32) - - for class_idx, class_prob in enumerate(sampling_prob): - db_class_probability += ( - tf.cast(tf.equal(db_label, class_idx), tf.float32) * class_prob) - - sampled_filter = tf.random.uniform( - tf.shape(example_filter), - minval=0, - maxval=1, - dtype=tf.float32, - seed=p.random_seed) - sampled_filter = sampled_filter < db_class_probability - example_filter &= sampled_filter - elif p.label_filter: - # Filter based on labels. - # Create a label filter where all is false - valid_labels = tf.constant(p.label_filter) - label_mask = tf.reduce_any( - tf.equal(db_label[..., tf.newaxis], valid_labels), axis=1) - example_filter = tf.math.logical_and(example_filter, label_mask) - return example_filter - - # TODO(vrv): Create an overlap filter that also ensures that boxes don't - # overlap with groundtruth points, so that the scenes are more plausible. - def _FilterIndices(self, gt_bboxes_3d, db_bboxes, db_idx): - """Identify database boxes that don't overlap with other boxes.""" - # We accomplish overlap filtering by first computing the pairwise 3D IoU of - # all boxes (concatenated) as a way of computing pairwise box overlaps. - num_gt_bboxes = tf.shape(gt_bboxes_3d)[0] - filtered_bboxes = tf.gather(db_bboxes, db_idx) - all_bboxes = tf.concat([gt_bboxes_3d, filtered_bboxes], axis=0) - pairwise_overlap = ops.pairwise_iou3d(all_bboxes, all_bboxes) - - # We now have an M x M matrix with 1s on the diagonal and non-zero entries - # whenever a box collides with another. - # - # To increase the number of boxes selected, we filter the upper triangular - # entries so that the boxes are chosen greedily: boxes with smaller indices - # will be selected before later boxes, because earlier boxes will not appear - # to collide with later boxes, but later boxes may collide with earlier - # ones. - pairwise_overlap = tf.linalg.band_part(pairwise_overlap, -1, 0) - - # We compute the sum of the IoU overlaps for all database boxes. - db_overlap_sums = tf.reduce_sum(pairwise_overlap[num_gt_bboxes:], axis=1) - - # Those boxes that don't overlap with any other boxes will only have - # a 1.0 IoU with itself. - non_overlapping_boxes = tf.reshape(db_overlap_sums <= 1., [-1]) - - # Filter to select only those object ids that pass this filter. - db_idx = tf.boolean_mask(db_idx, non_overlapping_boxes) - return db_idx - - def TransformFeatures(self, features): - p = self.params - - tf.logging.info('Loading groundtruth database at %s' % - (p.groundtruth_database)) - db = p.groundtruth_database.Instantiate().BuildDataSource(self._ReadDB).data - - original_features_shape = tf.shape(features.lasers.points_feature) - - # Compute the number of bboxes to augment. - num_bboxes_in_scene = tf.reduce_sum( - tf.cast(features.labels.bboxes_3d_mask, tf.int32)) - max_bboxes = tf.shape(features.labels.bboxes_3d_mask)[0] - num_augmented_bboxes = tf.minimum(max_bboxes - num_bboxes_in_scene, - p.max_augmented_bboxes) - - # Compute an object index over all objects in the database. - num_objects_in_database = tf.shape(db.points_xyz)[0] - db_idx = tf.range(num_objects_in_database) - - # Find those indices whose examples pass the filters, and select only those - # indices. - example_filter = self._CreateExampleFilter(db) - db_idx = tf.boolean_mask(db_idx, example_filter) - - # At this point, we might still have a large number of object candidates, - # from which we only need a sample. - # To reduce the amount of computation, we randomly subsample to slightly - # more than we want to augment. - db_idx = tf.random.shuffle( - db_idx, seed=p.random_seed)[0:num_augmented_bboxes * 5] - - # After filtering, further filter out the db boxes that would occlude with - # other boxes (including other database boxes). - # - # Gather the filtered ground truth bounding boxes according to the mask, so - # we can compute overlaps below. - gt_bboxes_3d_mask = tf.cast(features.labels.bboxes_3d_mask, tf.bool) - gt_bboxes_3d = tf.boolean_mask(features.labels.bboxes_3d, gt_bboxes_3d_mask) - gt_bboxes_3d = py_utils.HasShape(gt_bboxes_3d, [num_bboxes_in_scene, 7]) - db_idx = self._FilterIndices(gt_bboxes_3d, db.bboxes_3d, db_idx) - - # From the filtered object ids, select only as many boxes as we need. - shuffled_idx = db_idx[0:num_augmented_bboxes] - num_augmented_bboxes = tf.shape(shuffled_idx)[0] - - # Gather based off the indices. - sampled_points_xyz = tf.gather(db.points_xyz, shuffled_idx) - sampled_points_feature = tf.gather(db.points_feature, shuffled_idx) - sampled_mask = tf.reshape( - tf.gather(db.points_mask, shuffled_idx), - [num_augmented_bboxes, p.max_num_points_per_bbox]) - sampled_bboxes = tf.gather(db.bboxes_3d, shuffled_idx) - sampled_labels = tf.gather(db.labels, shuffled_idx) - - # Mask points/features. - sampled_points_xyz = tf.boolean_mask(sampled_points_xyz, sampled_mask) - sampled_points_feature = tf.boolean_mask(sampled_points_feature, - sampled_mask) - - # Flatten before concatenation with ground truths. - sampled_points_xyz = tf.reshape(sampled_points_xyz, [-1, 3]) - sampled_points_feature = tf.reshape(sampled_points_feature, - [-1, original_features_shape[-1]]) - sampled_bboxes = tf.reshape(sampled_bboxes, [-1, 7]) - - # Concatenate the samples with the ground truths. - if 'points_padding' in features.lasers: - points_mask = tf.cast(1. - features.lasers.points_padding, tf.bool) - # Densify the original points. - dense_points_xyz = tf.boolean_mask(features.lasers.points_xyz, - points_mask) - dense_points_feature = tf.boolean_mask(features.lasers.points_feature, - points_mask) - - # Concatenate the dense original points with our new sampled oints. - points_xyz = tf.concat([dense_points_xyz, sampled_points_xyz], axis=0) - points_feature = tf.concat([dense_points_feature, sampled_points_feature], - axis=0) - original_points_shape = tf.shape(features.lasers.points_xyz) - features.lasers.points_xyz = py_utils.PadOrTrimTo(points_xyz, - original_points_shape) - features.lasers.points_feature = py_utils.PadOrTrimTo( - points_feature, original_features_shape) - # Compute the modified mask / padding. - final_points_mask = py_utils.PadOrTrimTo( - tf.ones(tf.shape(points_xyz)[0]), - tf.shape(features.lasers.points_padding)) - features.lasers.points_padding = 1. - final_points_mask - else: - points_xyz = tf.concat([features.lasers.points_xyz, sampled_points_xyz], - axis=0) - points_feature = tf.concat( - [features.lasers.points_feature, sampled_points_feature], axis=0) - features.lasers.points_xyz = points_xyz - features.lasers.points_feature = points_feature - - # Reconstruct a new, dense, bboxes_3d vector that includes the filtered - # groundtruth bounding boxes followed by the database augmented boxes. - bboxes_3d = tf.concat([gt_bboxes_3d, sampled_bboxes], axis=0) - bboxes_3d = py_utils.PadOrTrimTo(bboxes_3d, [max_bboxes, 7]) - features.labels.bboxes_3d = bboxes_3d - bboxes_3d_mask = tf.ones( - num_bboxes_in_scene + num_augmented_bboxes, dtype=tf.float32) - features.labels.bboxes_3d_mask = py_utils.PadOrTrimTo( - bboxes_3d_mask, [max_bboxes]) - - gt_labels = tf.boolean_mask(features.labels.labels, gt_bboxes_3d_mask) - gt_labels = py_utils.HasShape(gt_labels, [num_bboxes_in_scene]) - - labels = tf.concat([gt_labels, sampled_labels], axis=0) - features.labels.labels = py_utils.PadOrTrimTo(labels, [max_bboxes]) - - return features - - def TransformShapes(self, shapes): - return shapes - - def TransformDTypes(self, dtypes): - return dtypes - - -class FrustumDropout(Preprocessor): - """Randomly drops out points in a frustum. - - All points are first converted to spherical coordinates, and then a point - is randomly selected. All points in the frustum around that point within - a given phi, theta angle width and distance to the original greater than - a given value are dropped with probability = 1 - keep_prob. - - Here, we can specify whether the dropped frustum is the union or intersection - of the phi and theta angle filters. - - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [P, 3] - - lasers.points_feature of shape [P, K] - - Optionally points_padding of shape [P] corresponding to the padding. - if points_padding is None, then all points are considered valid. - - Modifies the following features: - lasers.points_xyz, lasers.points_feature, lasers.points_padding with points - randomly dropped out. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('theta_width', 0.03, 'Theta angle width for dropping points.') - p.Define('phi_width', 0.0, 'Phi angle width for dropping points.') - p.Define( - 'distance', 0.0, 'Drop points that have larger distance to the' - 'origin than the value given here.') - p.Define( - 'keep_prob', 0.0, 'keep_prob: 1. = drop no points in the Frustum,' - '0 = drop all points, between 0 and 1 = down sample the points.') - p.Define( - 'drop_type', 'union', 'Drop either the union or intersection of ' - 'phi width and theta width.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.phi_width < 0: - raise ValueError('phi_width must be >= 0, phi_width={}'.format( - p.phi_width)) - if p.theta_width < 0: - raise ValueError('theta_width must be >= 0, theta_width={}'.format( - p.theta_width)) - if p.distance < 0: - raise ValueError('distance must be >= 0, distance={}'.format(p.distance)) - if p.keep_prob < 0 or p.keep_prob > 1: - raise ValueError('keep_prob must be >= 0 and <=1, keep_prob={}'.format( - p.keep_prob)) - if p.drop_type not in ['union', 'intersection']: - raise ValueError('drop_type must be union or intersection ,' - 'drop_type={}'.format(p.drop_type)) - - def TransformFeatures(self, features): - p = self.params - points_xyz = features.lasers.points_xyz - points_feature = features.lasers.points_feature - if 'points_padding' in features.lasers: - points_padding = features.lasers.points_padding - else: - points_padding = None - - if points_padding is not None: - points_mask = tf.cast(1 - points_padding, tf.bool) - num_total_points = py_utils.GetShape(points_mask)[0] - real_points_idx = tf.boolean_mask( - tf.range(0, num_total_points, dtype=tf.int32), points_mask) - num_points = py_utils.GetShape(real_points_idx)[0] - else: - points_mask = tf.ones_like(points_xyz[:, 0], dtype=tf.bool) - num_total_points = py_utils.GetShape(points_mask)[0] - num_points = py_utils.GetShape(points_xyz)[0] - - r, theta, phi = tf.unstack( - geometry.SphericalCoordinatesTransform(points_xyz), axis=-1) - - def _PickRandomPoint(): - point_idx = tf.random.uniform((), - minval=0, - maxval=num_points, - dtype=tf.int32) - if points_padding is not None: - point_idx = real_points_idx[point_idx] - return point_idx - - # Pick a point at random and drop all points that are near that point in the - # frustum for distance larger than r; repeat this for both theta and phi. - if p.theta_width > 0: - theta_half_width = p.theta_width / 2. - point_idx = _PickRandomPoint() - # Points within theta width and further than distance will be dropped. - theta_drop_filter = ((theta < (theta[point_idx] + theta_half_width)) & - (theta > (theta[point_idx] - theta_half_width)) & - (r > p.distance)) - else: - theta_drop_filter = tf.zeros_like(points_mask, dtype=tf.bool) - - if p.phi_width > 0: - phi_half_width = p.phi_width / 2. - point_idx = _PickRandomPoint() - # Points within phi width and further than distance will be dropped. - phi_drop_filter = ((phi < (phi[point_idx] + phi_half_width)) & - (phi > - (phi[point_idx] - phi_half_width)) & (r > p.distance)) - else: - phi_drop_filter = tf.zeros_like(points_mask, dtype=tf.bool) - - # Create drop_filter by combining filters. This contains a filter for the - # points to be removed. One can use the intersection method to limit the - # dropped points be within both phi and theta ranges. - if p.drop_type == 'union': - drop_filter = theta_drop_filter | phi_drop_filter - elif p.drop_type == 'intersection': - drop_filter = theta_drop_filter & phi_drop_filter - - if p.keep_prob == 0: - # Drop all points in drop_filter. - down_sampling_filter = drop_filter - else: - # Randomly drop points in drop_filter based on keep_prob. - sampling_drop_filter = tf.random.uniform([num_total_points], - minval=0, - maxval=1, - dtype=tf.float32) - # Points greater than the threshold (keep_prob) will be dropped. - sampling_drop_filter = sampling_drop_filter > p.keep_prob - - # Instead of dropping all points in the frustum, we drop out points - # that are in the selected frustum (drop_filter). - down_sampling_filter = drop_filter & sampling_drop_filter - - points_mask &= ~down_sampling_filter - - if points_padding is not None: - features.lasers.points_padding = 1 - tf.cast(points_mask, tf.float32) - else: - features.lasers.points_xyz = tf.boolean_mask(points_xyz, points_mask) - features.lasers.points_feature = tf.boolean_mask(points_feature, - points_mask) - - return features - - def TransformShapes(self, shapes): - return shapes - - def TransformDTypes(self, dtypes): - return dtypes - - -class RepeatPreprocessor(Preprocessor): - """Repeat a preprocessor multiple times. - - This preprocessor takes a preprocessor as a subprocessor and apply the - subprocessor to features multiple times (repeat_count). - - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('repeat_count', 1, 'Number of times the subprocessor is applied to' - ' features.') - p.Define('subprocessor', None, 'One of the input preprocessors.') - - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.subprocessor is None: - raise ValueError('No subprocessor was specified for RepeatPreprocessor.') - if p.repeat_count < 0 or not isinstance(p.repeat_count, int): - raise ValueError( - 'repeat_count must be >= 0 and int, repeat_count={}'.format( - p.repeat_count)) - - self.CreateChild('subprocessor', p.subprocessor) - - def TransformFeatures(self, features): - p = self.params - for _ in range(p.repeat_count): - features = self.subprocessor.FPropDefaultTheta(features) - - return features - - def TransformShapes(self, shapes): - p = self.params - for _ in range(p.repeat_count): - shapes = self.subprocessor.TransformShapes(shapes) - - return shapes - - def TransformDTypes(self, dtypes): - p = self.params - for _ in range(p.repeat_count): - dtypes = self.subprocessor.TransformDTypes(dtypes) - - return dtypes - - -class RandomApplyPreprocessor(Preprocessor): - """Randomly apply a preprocessor with certain probability. - - This preprocessor takes a preprocessor as a subprocessor and apply the - subprocessor to features with certain probability. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('prob', 1.0, 'The probability the subprocessor being executed.') - p.Define('subprocessor', None, 'Params for an input preprocessor.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.subprocessor is None: - raise ValueError('No subprocessor was specified for RepeatPreprocessor.') - if p.prob < 0 or p.prob > 1 or not isinstance(p.prob, float): - raise ValueError( - 'prob must be >= 0 and <=1 and float type, prob={}'.format(p.prob)) - - self.CreateChild('subprocessor', p.subprocessor) - - def TransformFeatures(self, features): - p = self.params - choice = tf.random.uniform( - (), minval=0.0, maxval=1.0, seed=p.random_seed) <= p.prob - # Features is passed downstream and may be modified, we make deep copies - # here to use with tf.cond to avoid having tf.cond access updated - # versions. Note that we need one copy for each branch in case the branches - # further modify features. - features_0, features_1 = features.DeepCopy(), features.DeepCopy() - features = tf.cond(choice, - lambda: self.subprocessor.TransformFeatures(features_0), - lambda: features_1) - return features - - def TransformShapes(self, shapes): - shapes_transformed = self.subprocessor.TransformShapes(shapes) - - if not shapes.IsCompatible(shapes_transformed): - raise ValueError( - 'NestedMap structures are different between shapes and transformed' - 'shapes. Original shapes: {}. Transformed shapes: {}'.format( - shapes, shapes_transformed)) - - def IsCompatibleWith(a, b): - return a.is_compatible_with(b) - - if not all( - py_utils.Flatten( - py_utils.Transform(IsCompatibleWith, shapes, shapes_transformed))): - raise ValueError( - 'Shapes after transformation - {} are different from original ' - 'shapes - {}.'.format(shapes_transformed, shapes)) - - return shapes - - def TransformDTypes(self, dtypes): - transformed_dtypes = self.subprocessor.TransformDTypes(dtypes) - if transformed_dtypes != dtypes: - raise ValueError( - 'DTypes after transformation of preprocessor - {} should be ' - 'the same as {}, but get {}.'.format(self.params.subprocessor, dtypes, - transformed_dtypes)) - return dtypes - - -class SparseSampler(Preprocessor): - """Fused SparseCenterSelector and SparseCellGatherFeatures. - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [P, 3] - - lasers.points_feature of shape [P, F] - - Adds the following features: - anchor_centers - [num_centers, 3] - Floating point output containing the - center (x, y, z) locations for tiling anchor boxes. - - cell_center_xyz - [num_centers, 3] - Floating point output containing - the center (x, y, z) locations for each cell to featurize. - - cell_center_padding - [num_centers] - 0/1 padding for each center. - - cell_points_xyz - [num_centers, num_neighbors, 3] - Floating point - output containing the (x, y, z) locations for each point for a given - center. - - cell_feature - [num_centers, num_neighbors, F] - Floating point output - containing the features for each point for a given center. - - cell_points_padding - [num_centers, num_neighbors] - 0/1 padding - for the points in each cell. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('center_selector', 'farthest', 'Method to sample centers. ' - 'Valid options - uniform, farthest.') - p.Define('neighbor_sampler', 'uniform', 'Method to select neighbors. ' - 'Valid options - uniform, closest.') - p.Define('num_centers', 16, 'The number of centers to sample.') - p.Define( - 'features_preparation_layers', [], - 'A list of Params for layers to run on the features before ' - 'performing farthest point sampling. For example, one may wish to ' - 'drop points out of frustum for KITTI before selecting centers. ' - 'Note that these layers will not mutate the original features, ' - 'instead, a copy will be made.') - p.Define( - 'keep_z_range', (-np.inf, np.inf), - 'Only points that have z coordinates within this range are kept. ' - 'Approximate ground-removal can be performed by specifying a ' - 'lower-bound on the z-range.') - p.Define('num_neighbors', 64, 'Sample these many points within the ' - 'neighorhood.') - p.Define( - 'max_distance', 1.0, 'Points with L2 distances from a center ' - 'larger than this threshold are not considered to be in the ' - 'neighborhood.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.features_preparation_layers: - self.CreateChildren('features_preparation_layers', - p.features_preparation_layers) - - def TransformFeatures(self, features): - p = self.params - n, m = p.num_centers, p.num_neighbors - - prepared_features = features.DeepCopy() - if p.features_preparation_layers: - for prep_layer in self.features_preparation_layers: - prepared_features = prep_layer.FPropDefaultTheta(prepared_features) - - points_data = prepared_features.lasers - points = py_utils.HasShape(points_data.points_xyz, [-1, 3]) - - if 'points_padding' in points_data: - points_mask = 1 - points_data.points_padding - points = tf.boolean_mask(points, points_mask) - - # If num_points < num_centers, pad points to have at least num_centers - # points. - num_points = tf.shape(points)[0] - required_num_points = tf.maximum(num_points, p.num_centers) - zeros = tf.zeros([required_num_points - num_points, 3]) - points = tf.concat([points, zeros], axis=0) - - num_seeded_points = points_data.get('num_seeded_points', 0) - - neighbor_algorithm = 'auto' - # Based on benchmarks, the hash solution works better when the number of - # centers is >= 16 and there are at least 10k points per point cloud. - if p.num_centers >= 16: - neighbor_algorithm = 'hash' - - centers, center_paddings, indices, indices_paddings = ops.sample_points( - points=tf.expand_dims(points, 0), - points_padding=tf.zeros([1, required_num_points], tf.float32), - num_seeded_points=num_seeded_points, - center_selector=p.center_selector, - neighbor_sampler=p.neighbor_sampler, - neighbor_algorithm=neighbor_algorithm, - num_centers=p.num_centers, - center_z_min=p.keep_z_range[0], - center_z_max=p.keep_z_range[1], - num_neighbors=p.num_neighbors, - max_distance=p.max_distance, - random_seed=p.random_seed if p.random_seed else -1) - centers = py_utils.HasShape(centers, [1, n])[0, :] - center_paddings = py_utils.HasShape(center_paddings, [1, n])[0, :] - indices = py_utils.HasShape(indices, [1, n, m])[0, :] - indices_paddings = py_utils.HasShape(indices_paddings, [1, n, m])[0, :] - features.cell_center_padding = center_paddings - features.cell_center_xyz = py_utils.HasShape( - tf.gather(points, centers), [n, 3]) - features.anchor_centers = features.cell_center_xyz - features.cell_points_xyz = py_utils.HasShape( - tf.gather(points, indices), [n, m, 3]) - features.cell_feature = tf.gather(points_data.points_feature, indices) - features.cell_points_padding = indices_paddings - return features - - def TransformShapes(self, shapes): - p = self.params - n, m, f = p.num_centers, p.num_neighbors, shapes.lasers.points_feature[-1] - shapes.anchor_centers = tf.TensorShape([n, 3]) - shapes.cell_center_padding = tf.TensorShape([n]) - shapes.cell_center_xyz = tf.TensorShape([n, 3]) - shapes.cell_points_xyz = tf.TensorShape([n, m, 3]) - shapes.cell_feature = tf.TensorShape([n, m, f]) - shapes.cell_points_padding = tf.TensorShape([n, m]) - return shapes - - def TransformDTypes(self, dtypes): - dtypes.anchor_centers = tf.float32 - dtypes.cell_center_padding = tf.float32 - dtypes.cell_center_xyz = tf.float32 - dtypes.cell_points_xyz = tf.float32 - dtypes.cell_feature = tf.float32 - dtypes.cell_points_padding = tf.float32 - return dtypes diff --git a/lingvo/tasks/car/kitti_ap_metric.py b/lingvo/tasks/car/kitti_ap_metric.py deleted file mode 100644 index 8df5d500..00000000 --- a/lingvo/tasks/car/kitti_ap_metric.py +++ /dev/null @@ -1,255 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Average Precision metric class for KITTI.""" - -from lingvo import compat as tf -from lingvo.core import py_utils -from lingvo.tasks.car import ap_metric -from lingvo.tasks.car import ops -import numpy as np - - -class KITTIAPMetrics(ap_metric.APMetrics): - """The KITTI implementation of AP metric.""" - - def _GetData(self, - classid, - difficulty=None, - distance=None, - num_points=None, - rotation=None): - """Returns groundtruth and prediction for the classid in a NestedMap. - - Args: - classid: int32 specifying the class - difficulty: String in [easy, moderate, hard]. If None specified, all - difficulty levels are permitted. - distance: int32 specifying a binned Euclidean distance of the ground truth - bounding box. If None is specified, all distances are selected. - num_points: int32 specifying a binned number of laser points within the - ground truth bounding box. If None is specified, all boxes are selected. - rotation: int32 specifying a binned rotation within the ground truth - bounding box. If None is specified, all boxes are selected. - - Returns: - NestedMap containing iou_threshold, groundtruth and predictions for - specified, classid, difficulty level and binned distance. If no bboxes - are found with these parameters, returns None. - """ - assert classid > 0 and classid < self.metadata.NumClasses() - g = self._LoadBoundingBoxes( - 'groundtruth', - classid, - distance=distance, - num_points=num_points, - rotation=rotation) - # Note that we do not specify num_points for predictions because only - # groundtruth boxes contain points. - p = self._LoadBoundingBoxes( - 'prediction', classid, distance, num_points=None, rotation=rotation) - if g is None or p is None: - return None - - # KITTI ignores detections that are too small. - min_height = 0 - if difficulty: - min_height = self.metadata.MinHeight2D()[difficulty] - pd_ignore = p.heights_in_pixels < min_height - # We first fetch the ground truth boxes for the specified class id - # into gt_boxes. - gt_boxes = g.boxes - gt_imgids = g.imgids - # Ignore bboxes that are more difficult than this levels by setting their - # gt_ignore to 1 (IgnoreOneMatch). - if difficulty: - gt_ignore = g.difficulties < self.metadata.DifficultyLevels()[difficulty] - gt_ignore = gt_ignore.astype(np.int32) - else: - gt_ignore = np.zeros_like(g.difficulties, dtype=np.int32) - - # Extract the bounding boxes from the groundtruth for 'similar' - # categories. Add these to the list of ground truth bounding boxes - # with an ignore setting of 1 (IgnoreOneMatch). - if classid in self.metadata.IgnoreClassIndices(): - for class_id_to_ignore in self.metadata.IgnoreClassIndices()[classid]: - g_ignore = self._LoadBoundingBoxes( - 'groundtruth', class_id=class_id_to_ignore) - - if g_ignore is not None: - n_ignore_boxes = g_ignore.boxes.shape[0] - gt_imgids = np.concatenate([gt_imgids, g_ignore.imgids]) - gt_boxes = np.concatenate([gt_boxes, g_ignore.boxes]) - gt_ignore = np.concatenate([gt_ignore, np.ones(n_ignore_boxes)]) - - # Extract the DontCare bounding boxes from the data, and add - # these to the list of bounding boxes to evaluate with an ignore - # setting of 2 (IgnoreAllMatches). Only relevant to KITTI. - if 'DontCare' in self.metadata.ClassNames(): - g_ignore = self._LoadBoundingBoxes( - 'groundtruth', class_id=self.metadata.ClassNames().index('DontCare')) - if g_ignore is not None: - n_ignore_boxes = g_ignore.boxes.shape[0] - gt_imgids = np.concatenate([gt_imgids, g_ignore.imgids]) - gt_boxes = np.concatenate([gt_boxes, g_ignore.boxes]) - gt_ignore = np.concatenate([gt_ignore, 2 * np.ones(n_ignore_boxes)]) - - iou_threshold = self._iou_thresholds[self.metadata.ClassNames()[classid]] - return py_utils.NestedMap( - iou_threshold=iou_threshold, - gt=py_utils.NestedMap(imgid=gt_imgids, bbox=gt_boxes, ignore=gt_ignore), - pd=py_utils.NestedMap( - imgid=p.imgids, bbox=p.boxes, score=p.scores, ignore=pd_ignore)) - - def _BuildMetric(self, feed_data, classid): - """Construct tensors and the feed_dict for KITTI metric op. - - Args: - feed_data: a NestedMap returned by _GetData() - classid: integer. Unused in this implementation. - - Returns: - A tuple of 3 dicts: - - - scalar_metrics: a dict mapping all the metric names to fetch tensors. - - curves: a dict mapping all the curve names to fetch tensors. - - feed_dict: a dict mapping the tensors in feed_tensors to feed values. - """ - if feed_data is None: - dummy_scalar = tf.constant(np.nan) - dummy_calibration = tf.constant(np.nan) - dummy_curve = tf.zeros([self.metadata.NumberOfPrecisionRecallPoints(), 2], - tf.float32) - scalar_metrics = {'ap': dummy_scalar} - curve_metrics = {'pr': dummy_curve} - calibration_metrics = {'calibrations': dummy_calibration} - - return py_utils.NestedMap( - feed_dict={}, - scalar_metrics=scalar_metrics, - curve_metrics=curve_metrics, - calibration_metrics=calibration_metrics) - - feed_dict = {} - - f_iou = tf.placeholder(tf.float32) - feed_dict[f_iou] = feed_data.iou_threshold - - f_gt_bbox = tf.placeholder(tf.float32) - feed_dict[f_gt_bbox] = feed_data.gt.bbox - - f_gt_imgid = tf.placeholder(tf.int32) - feed_dict[f_gt_imgid] = feed_data.gt.imgid - - f_gt_ignore = tf.placeholder(tf.int32) - feed_dict[f_gt_ignore] = feed_data.gt.ignore - - f_pd_bbox = tf.placeholder(tf.float32) - feed_dict[f_pd_bbox] = feed_data.pd.bbox - - f_pd_imgid = tf.placeholder(tf.int32) - feed_dict[f_pd_imgid] = feed_data.pd.imgid - - f_pd_ignore = tf.placeholder(tf.int32) - feed_dict[f_pd_ignore] = feed_data.pd.ignore - - f_pd_score = tf.placeholder(tf.float32) - feed_dict[f_pd_score] = feed_data.pd.score - - # TODO(shlens): The third returned argument contain statistics for measuring - # the calibration error. Use it. - ap, pr, calibration = ops.average_precision3d( - iou_threshold=f_iou, - groundtruth_bbox=f_gt_bbox, - groundtruth_imageid=f_gt_imgid, - groundtruth_ignore=f_gt_ignore, - prediction_bbox=f_pd_bbox, - prediction_imageid=f_pd_imgid, - prediction_ignore=f_pd_ignore, - prediction_score=f_pd_score, - num_recall_points=self.metadata.NumberOfPrecisionRecallPoints()) - - scalar_metrics = {'ap': ap} - curve_metrics = {'pr': pr} - calibration_metrics = {'calibrations': calibration} - return py_utils.NestedMap( - feed_dict=feed_dict, - scalar_metrics=scalar_metrics, - curve_metrics=curve_metrics, - calibration_metrics=calibration_metrics) - - def _ComputeFinalMetrics(self, - classids=None, - difficulty=None, - distance=None, - num_points=None, - rotation=None): - """Compute precision-recall curves as well as average precision. - - Args: - classids: A list of N int32. - difficulty: String in [easy, moderate, hard]. If None specified, all - difficulty levels are permitted. - distance: int32 specifying a binned Euclidean distance of the ground truth - bounding box. If None is specified, all distances are selected. - num_points: int32 specifying a binned number of laser points within the - ground truth bounding box. If None is specified, all boxes are selected. - rotation: int32 specifying a binned rotation within the ground truth - bounding box. If None is specified, all boxes are selected. - - Returns: - dict. Each entry in the dict is a list of C (number of classes) dicts - containing mapping from metric names to individual results. Individual - entries may be the following items. - - scalars: A list of C (number of classes) dicts mapping metric - names to scalar values. - - curves: A list of C dicts mapping metrics names to np.float32 - arrays of shape [NumberOfPrecisionRecallPoints()+1, 2]. In the last - dimension, 0 indexes precision and 1 indexes recall. - - calibrations: A list of C dicts mapping metrics names to np.float32 - arrays of shape [number of predictions, 2]. The first column is the - predicted probabilty and the second column is 0 or 1 indicating that the - prediction matched a ground truth item. - """ - tf.logging.info('Computing final KITTI metrics.') - assert classids is not None, 'classids must be supplied.' - feed_dict = {} - g = tf.Graph() - scalar_fetches = [] - curve_fetches = [] - calibration_fetches = [] - with g.as_default(): - for classid in classids: - data = self._GetData( - classid, - difficulty=difficulty, - distance=distance, - num_points=num_points, - rotation=rotation) - metrics = self._BuildMetric(data, classid) - scalar_fetches += [metrics.scalar_metrics] - curve_fetches += [metrics.curve_metrics] - calibration_fetches += [metrics.calibration_metrics] - feed_dict.update(metrics.feed_dict) - - with tf.Session(graph=g) as sess: - results = sess.run([scalar_fetches, curve_fetches, calibration_fetches], - feed_dict=feed_dict) - tf.logging.info('Finished computing final KITTI metrics.') - return { - 'scalars': results[0], - 'curves': results[1], - 'calibrations': results[2] - } diff --git a/lingvo/tasks/car/kitti_decoder.py b/lingvo/tasks/car/kitti_decoder.py deleted file mode 100644 index 01caf1ae..00000000 --- a/lingvo/tasks/car/kitti_decoder.py +++ /dev/null @@ -1,389 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Base models for point-cloud based detection.""" - -from lingvo import compat as tf -from lingvo.core import metrics -from lingvo.core import py_utils -from lingvo.tasks.car import base_decoder -from lingvo.tasks.car import detection_3d_lib -from lingvo.tasks.car import detection_3d_metrics -from lingvo.tasks.car import geometry -from lingvo.tasks.car import kitti_ap_metric -from lingvo.tasks.car import kitti_metadata -from lingvo.tasks.car import transform_util -import numpy as np - - -class KITTIDecoder(base_decoder.BaseDecoder): - """A decoder to use for decoding a detector model on KITTI. - - This class implements the basic Decoder metrics for KITTI to provide - visualizations and AP calculations. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'filter_predictions_outside_frustum', False, - 'If true, predictions whose bounding box center is outside of the ' - 'image frustum are dropped.') - p.Define( - 'truncation_threshold', 0.0, - 'Specifies how much of a bounding box can be truncated ' - 'by the edge of the image frustum and still be kept. A value of 0.0 ' - 'means that we only drop predictions whose 2d bounding box ' - 'falls entirely outside the image frustum. A value of 1.0 means ' - 'we drop predictions where *any* portion of the bounding box falls ' - 'outside the frustum.') - p.ap_metric = kitti_ap_metric.KITTIAPMetrics.Params( - kitti_metadata.KITTIMetadata()) - return p - - def CreateDecoderMetrics(self): - """Decoder metrics for KITTI.""" - p = self.params - - kitti_metric_p = p.ap_metric.Copy().Set(cls=kitti_ap_metric.KITTIAPMetrics) - apm = kitti_metric_p.Instantiate() - class_names = apm.metadata.ClassNames() - - # Convert the list of class names to a dictionary mapping class_id -> name. - class_id_to_name = dict(enumerate(class_names)) - - top_down_transform = transform_util.MakeCarToImageTransform( - pixels_per_meter=32., - image_ref_x=512., - image_ref_y=1408., - flip_axes=True) - - decoder_metrics = py_utils.NestedMap({ - 'top_down_visualization': - (detection_3d_metrics.TopDownVisualizationMetric( - top_down_transform, - image_height=1536, - image_width=1024, - class_id_to_name=class_id_to_name)), - 'num_samples_in_batch': metrics.AverageMetric(), - 'kitti_AP_v2': apm, - }) - - decoder_metrics.mesh = detection_3d_metrics.WorldViewer() - - if p.summarize_boxes_on_image: - decoder_metrics.camera_visualization = ( - detection_3d_metrics.CameraVisualization( - bbox_score_threshold=p.visualization_classification_threshold)) - - return decoder_metrics - - def _CreateFrustumMask(self, bbox_corners_image, bbox2d_corners_image_clipped, - image_height, image_width): - """Creates a box mask for boxes whose projections fall outside of image.""" - p = self.params - batch_size, num_boxes = py_utils.GetShape(bbox_corners_image, 2) - if not p.filter_predictions_outside_frustum: - return tf.ones(shape=(batch_size, num_boxes), dtype=tf.float32) - - def _MinMax(bbox_corners): - """Computes the min and max over corners.""" - bbox_min = tf.reduce_min(bbox_corners, axis=-1) - bbox_max = tf.reduce_max(bbox_corners, axis=-1) - bbox_min = py_utils.HasShape(bbox_min, [batch_size, num_boxes]) - bbox_max = py_utils.HasShape(bbox_max, [batch_size, num_boxes]) - return bbox_min, bbox_max - - bbox_min_x, bbox_max_x = _MinMax(bbox_corners_image[:, :, :, 0]) - bbox_min_y, bbox_max_y = _MinMax(bbox_corners_image[:, :, :, 1]) - - # Compute the fraction of the clipped 2d image projection and the - # full 2d image projection. We simply need to divide the area - # of each cropped box by the area of the full box to get the - # overlap fraction. - original_area = (bbox_max_x - bbox_min_x) * (bbox_max_y - bbox_min_y) - bbox_clipped_x_min = bbox2d_corners_image_clipped[..., 0] - bbox_clipped_y_min = bbox2d_corners_image_clipped[..., 1] - bbox_clipped_x_max = bbox2d_corners_image_clipped[..., 2] - bbox_clipped_y_max = bbox2d_corners_image_clipped[..., 3] - clipped_area = (bbox_clipped_x_max - bbox_clipped_x_min) * ( - bbox_clipped_y_max - bbox_clipped_y_min) - fraction = clipped_area / original_area - - frustum_mask = (fraction > p.truncation_threshold) - frustum_mask = py_utils.HasShape(frustum_mask, [batch_size, num_boxes]) - frustum_mask = tf.cast(frustum_mask, tf.float32) - return frustum_mask - - def _BBox2DImage(self, bbox_corners_image, input_images): - """Compute [xmin, ymin, xmax, ymax] 2D bounding boxes from corners.""" - # Clip the boundaries of the bounding box to the image width/height. - bci_x = bbox_corners_image[..., 0:1] - image_width = tf.broadcast_to( - input_images.width[..., tf.newaxis, tf.newaxis], tf.shape(bci_x)) - bci_x = tf.clip_by_value(bci_x, 0.0, tf.cast(image_width, tf.float32)) - - bci_y = bbox_corners_image[..., 1:2] - image_height = tf.broadcast_to( - input_images.height[..., tf.newaxis, tf.newaxis], tf.shape(bci_y)) - bci_y = tf.clip_by_value(bci_y, 0.0, tf.cast(image_height, tf.float32)) - - bbox_corners_image_clipped = tf.concat([bci_x, bci_y], axis=-1) - - # Compute the [xmin, ymin, xmax, ymax] bounding boxes from [batch, - # num_boxes, 8, 2] extrema. - min_vals = tf.math.reduce_min(bbox_corners_image_clipped, axis=2) - max_vals = tf.math.reduce_max(bbox_corners_image_clipped, axis=2) - bbox2d_corners_image = tf.concat([min_vals, max_vals], axis=2) - return bbox2d_corners_image - - def ProcessOutputs(self, input_batch, model_outputs): - """Produce additional decoder outputs for KITTI. - - Args: - input_batch: A .NestedMap of the inputs to the model. - model_outputs: A .NestedMap of the outputs of the model, including:: - - per_class_predicted_bboxes: [batch, num_classes, num_boxes, 7] float - Tensor with per class 3D (7 DOF) bounding boxes. - - per_class_predicted_bbox_scores: [batch, num_classes, num_boxes] float - Tensor with per class, per box scores. - - per_class_valid_mask: [batch, num_classes, num_boxes] masking Tensor - indicating which boxes were still kept after NMS for each class. - - Returns: - A NestedMap of additional decoder outputs needed for - PostProcessDecodeOut. - """ - p = self.params - per_class_predicted_bboxes = model_outputs.per_class_predicted_bboxes - batch_size, num_classes, num_boxes, _ = py_utils.GetShape( - per_class_predicted_bboxes) - flattened_num_boxes = num_classes * num_boxes - - input_labels = input_batch.decoder_copy.labels - input_lasers = input_batch.decoder_copy.lasers - input_images = input_batch.decoder_copy.images - - with tf.device('/cpu:0'): - # Convert the predicted bounding box points to their corners - # and then project them to the image plane. - # - # This output can be used to: - # - # A) Visualize bounding boxes (2d or 3d) on the camera image. - # - # B) Compute the height of the predicted boxes to filter 'too small' boxes - # as is done in the KITTI eval. - predicted_bboxes = tf.reshape(per_class_predicted_bboxes, - [batch_size, flattened_num_boxes, 7]) - bbox_corners = geometry.BBoxCorners(predicted_bboxes) - bbox_corners = py_utils.HasShape(bbox_corners, - [batch_size, flattened_num_boxes, 8, 3]) - utils_3d = detection_3d_lib.Utils3D() - bbox_corners_image = utils_3d.CornersToImagePlane( - bbox_corners, input_images.velo_to_image_plane) - bbox_corners_image = py_utils.HasShape( - bbox_corners_image, [batch_size, flattened_num_boxes, 8, 2]) - - # Clip the bounding box corners so they remain within - # the image coordinates. - bbox2d_corners_image_clipped = self._BBox2DImage(bbox_corners_image, - input_images) - bbox2d_corners_image_clipped = py_utils.HasShape( - bbox2d_corners_image_clipped, [batch_size, flattened_num_boxes, 4]) - - # Compute the frustum mask to filter out bounding boxes that - # are 'outside the frustum'. - frustum_mask = self._CreateFrustumMask(bbox_corners_image, - bbox2d_corners_image_clipped, - input_images.height, - input_images.width) - - # Reshape all of these back to [batch_size, num_classes, num_boxes, ...] - bbox_corners_image = tf.reshape( - bbox_corners_image, [batch_size, num_classes, num_boxes, 8, 2]) - - bbox2d_corners_image_clipped = tf.reshape( - bbox2d_corners_image_clipped, [batch_size, num_classes, num_boxes, 4]) - frustum_mask = tf.reshape(frustum_mask, - [batch_size, num_classes, num_boxes]) - - ret = py_utils.NestedMap({ - # For mAP eval - 'source_ids': input_labels.source_id, - 'difficulties': input_labels.difficulties, - 'num_points_in_bboxes': input_batch.labels.bboxes_3d_num_points, - # For exporting. - 'velo_to_image_plane': input_images.velo_to_image_plane, - 'velo_to_camera': input_images.velo_to_camera, - # Predictions. - 'bbox_corners_image': bbox_corners_image, - 'bbox2d_corners_image': bbox2d_corners_image_clipped, - 'frustum_mask': frustum_mask, - # Ground truth. - 'bboxes_3d': input_labels.bboxes_3d, - 'bboxes_3d_mask': input_labels.bboxes_3d_mask, - 'unfiltered_bboxes_3d_mask': input_labels.unfiltered_bboxes_3d_mask, - 'labels': input_labels.labels, - }) - - laser_sample = self._SampleLaserForVisualization( - input_lasers.points_xyz, input_lasers.points_padding) - ret.update(laser_sample) - - if p.summarize_boxes_on_image: - ret.camera_images = input_images.image - return ret - - def PostProcessDecodeOut(self, dec_out_dict, dec_metrics_dict): - """Post-processes the decoder outputs.""" - p = self.params - # Update num_samples_in_batch. - batch_size, num_classes, num_boxes, _ = ( - dec_out_dict.per_class_predicted_bboxes.shape) - dec_metrics_dict.num_samples_in_batch.Update(batch_size) - - # Apply frustum mask to predicted box outputs. - masked_pred_bbox_scores = ( - dec_out_dict.per_class_predicted_bbox_scores * - dec_out_dict.frustum_mask) - visualization_weights = ( - dec_out_dict.visualization_weights * dec_out_dict.frustum_mask) - - # TODO(bencaine): Add class base colors to camera image projection. - if p.summarize_boxes_on_image: - # Update the camera visualization. - flattened_bbox_corners = np.reshape(dec_out_dict.bbox_corners_image, - [batch_size, -1, 8, 2]) - flattened_visualization_weights = np.reshape(visualization_weights, - [batch_size, -1]) - dec_metrics_dict.camera_visualization.Update( - py_utils.NestedMap({ - 'camera_images': dec_out_dict.camera_images, - # TODO(vrv): Use 2D output instead of 3D output. - 'bbox_corners': flattened_bbox_corners, - 'bbox_scores': flattened_visualization_weights, - })) - - # Update decoder output by removing z-coordinate, thus reshaping the bboxes - # to [batch, num_bboxes, 5] to be compatible with - # TopDownVisualizationMetric. - - # Indices corresponding to the 2D bbox parameters (x, y, dx, dy, phi). - bbox_2d_idx = np.asarray([1, 1, 0, 1, 1, 0, 1], dtype=np.bool) - bboxes_2d = dec_out_dict.bboxes_3d[..., bbox_2d_idx] - predicted_bboxes = dec_out_dict.per_class_predicted_bboxes[..., bbox_2d_idx] - - if dec_out_dict.points_sampled: - tf.logging.info('Updating sample for top down visualization') - dec_metrics_dict.mesh.Update( - py_utils.NestedMap({ - 'points_xyz': dec_out_dict.points_xyz, - 'points_padding': dec_out_dict.points_padding, - })) - - # Flatten our predictions/scores to match the API of the visualization - # The last dimension of flattened_bboxes is 5 due to the mask - # above using bbox_2d_idx. - flattened_bboxes = np.reshape(predicted_bboxes, - [batch_size, num_classes * num_boxes, 5]) - flattened_visualization_weights = np.reshape( - visualization_weights, [batch_size, num_classes * num_boxes]) - # Create a label id mask for now to maintain compatibility. - # TODO(bencaine): Refactor visualizations to reflect new structure. - flattened_visualization_labels = np.tile( - np.arange(0, num_classes)[np.newaxis, :, np.newaxis], - [batch_size, 1, num_boxes]) - flattened_visualization_labels = np.reshape( - flattened_visualization_labels, [batch_size, num_classes * num_boxes]) - - dec_metrics_dict.top_down_visualization.Update( - py_utils.NestedMap({ - 'visualization_labels': flattened_visualization_labels, - 'predicted_bboxes': flattened_bboxes, - 'visualization_weights': flattened_visualization_weights, - 'points_xyz': dec_out_dict.points_xyz, - 'points_padding': dec_out_dict.points_padding, - 'gt_bboxes_2d': bboxes_2d, - 'gt_bboxes_2d_weights': dec_out_dict.bboxes_3d_mask, - 'labels': dec_out_dict.labels, - 'difficulties': dec_out_dict.difficulties, - 'source_ids': dec_out_dict.source_ids, - })) - - # Update KITTI AP metrics. - for batch_idx in range(batch_size): - # Use class scores since it's masked - pred_bboxes = dec_out_dict.per_class_predicted_bboxes[batch_idx] - pred_bbox_scores = masked_pred_bbox_scores[batch_idx] - - # Compute height in 2D perspective view, that will be used for filtering - # outputs downstream. - pred_bbox2d_corners_image = dec_out_dict.bbox2d_corners_image[batch_idx] - # We assume y2 >= y1, so pred_heights_image will always be >= 0. - pred_heights_image = ( - pred_bbox2d_corners_image[:, :, 3] - - pred_bbox2d_corners_image[:, :, 1]) - - gt_mask = dec_out_dict.unfiltered_bboxes_3d_mask[batch_idx].astype(bool) - gt_labels = dec_out_dict.labels[batch_idx][gt_mask] - gt_bboxes = dec_out_dict.bboxes_3d[batch_idx][gt_mask] - gt_difficulties = dec_out_dict.difficulties[batch_idx][gt_mask] - gt_num_points = dec_out_dict.num_points_in_bboxes[batch_idx][gt_mask] - - for metric_class in [dec_metrics_dict.kitti_AP_v2]: - metric_class.Update( - dec_out_dict.source_ids[batch_idx], - py_utils.NestedMap( - groundtruth_labels=gt_labels, - groundtruth_bboxes=gt_bboxes, - groundtruth_difficulties=gt_difficulties, - groundtruth_num_points=gt_num_points, - detection_scores=pred_bbox_scores, - detection_boxes=pred_bboxes, - detection_heights_in_pixels=pred_heights_image, - )) - - # Returned values are saved in model_dir/decode. We can offline convert - # them into KITTI's format. - # velo_to_image_plane and velo_to_camera are needed for converting 3D bboxes - # back to the camera coordinate as required by KITTI's format. - # - output_to_save = [] - for i in range(batch_size): - gt_save_mask = dec_out_dict.unfiltered_bboxes_3d_mask[i].astype(bool) - pd_save_mask = dec_out_dict.per_class_valid_mask[i] > 0 - # Create a class id matrix we can then mask out - # since when you use the boolean mask in numpy it collapses the matrix - # from [num_classes, num_bboxes,...] to just [num_valid_bbboxes]. - class_ids = np.tile(np.arange(num_classes)[:, np.newaxis], [1, num_boxes]) - - saved_results = py_utils.NestedMap( - img_id=dec_out_dict.source_ids[i], - bboxes=dec_out_dict.per_class_predicted_bboxes[i][pd_save_mask], - scores=masked_pred_bbox_scores[i][pd_save_mask], - class_ids=class_ids[pd_save_mask], - gt_labels=dec_out_dict.labels[i][gt_save_mask], - gt_bboxes=dec_out_dict.bboxes_3d[i][gt_save_mask], - gt_difficulties=dec_out_dict.difficulties[i][gt_save_mask], - velo_to_image_plane=dec_out_dict.velo_to_image_plane[i], - velo_to_camera=dec_out_dict.velo_to_camera[i], - bboxes_2d=dec_out_dict.bbox2d_corners_image[i][pd_save_mask], - ) - - serialized = self.SaveTensors(saved_results) - output_to_save += [(dec_out_dict.source_ids[i], serialized)] - return output_to_save diff --git a/lingvo/tasks/car/kitti_input_generator.py b/lingvo/tasks/car/kitti_input_generator.py deleted file mode 100644 index ca8ca136..00000000 --- a/lingvo/tasks/car/kitti_input_generator.py +++ /dev/null @@ -1,649 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Input generator for KITTI data.""" - -from lingvo import compat as tf -from lingvo.core import datasource -from lingvo.core import hyperparams -from lingvo.core import ops -from lingvo.core import py_utils -from lingvo.tasks.car import geometry -from lingvo.tasks.car import input_extractor -from lingvo.tasks.car import input_preprocessors -from lingvo.tasks.car import kitti_metadata - - -def _Dense(sparse, default_value=0): - return tf.sparse_to_dense( - sparse_indices=sparse.indices, - output_shape=sparse.dense_shape, - sparse_values=sparse.values, - default_value=default_value) - - -def _NestedMapToParams(nmap): - p = hyperparams.Params() - for k, v in nmap.FlattenItems(): - p.Define(k, v, '') - return p - - -def ComputeKITTIDifficulties(box_image_height, occlusion, truncation): - """Compute difficulties from box height, occlusion, and truncation.""" - # Easy: No occlusion, max truncation 15% - easy_level = tf.cast((box_image_height >= 40.) & (occlusion <= 0.) - & (truncation <= 0.15), tf.int32) * 3 - # Moderate: max occlusion: partly occluded, max truncation 30% - moderate_level = tf.cast((occlusion <= 1.) & (truncation <= 0.3) - & (box_image_height >= 25.), tf.int32) * 2 - # Hard: Difficult to see, max truncation 50% - hard_level = tf.cast((occlusion <= 2.) & (truncation <= 0.5) - & (box_image_height >= 25.), tf.int32) * 1 - - # Occlusion = 3 and higher truncation is "super hard", and - # will map to 0 (ignored). - difficulties = tf.maximum(tf.maximum(hard_level, moderate_level), easy_level) - - return difficulties - - -class KITTILaserExtractor(input_extractor.LaserExtractor): - """Base extractor for the laser points from a KITTI tf.Example.""" - - @classmethod - def Params(cls): - p = super().Params().Set(max_num_points=None, num_features=1) - return p - - def FeatureMap(self): - feature_map = { - 'pointcloud/xyz': tf.io.VarLenFeature(dtype=tf.float32), - 'pointcloud/reflectance': tf.io.VarLenFeature(dtype=tf.float32), - } - return feature_map - - def _Extract(self, features): - p = self.params - points_xyz = tf.reshape(_Dense(features['pointcloud/xyz']), [-1, 3]) - points_feature = tf.reshape( - _Dense(features['pointcloud/reflectance']), [-1, p.num_features]) - - if p.max_num_points is not None: - npoints = tf.shape(points_xyz)[0] - points_xyz = py_utils.PadOrTrimTo(points_xyz, [p.max_num_points, 3]) - points_feature = py_utils.PadOrTrimTo(points_feature, - [p.max_num_points, p.num_features]) - points_padding = 1.0 - py_utils.PadOrTrimTo( - tf.ones([npoints]), [p.max_num_points]) - - ret = py_utils.NestedMap( - points_xyz=points_xyz, points_feature=points_feature) - if p.max_num_points is not None: - ret.points_padding = points_padding - return ret - - -class KITTIImageExtractor(input_extractor.FieldsExtractor): - """Extracts the image information (left camera) from a KITTI tf.Example. - - Produces: - image: [512, 1382, 3] - Floating point Tensor containing image data. Note - that image may not be produced if decode_image is set to False. During - training, we may not want to decode the images. - - width: [1] - integer scalar width of the original image. - - height: [1] - integer scalar width of the original image. - - velo_to_image_plane: [3, 4] - transformation matrix from velo xyz to image - plane xy. After multiplication, you need to divide by last coordinate to - recover 2D pixel locations. - - velo_to_camera: [4, 4] - transformation matrix from velo xyz to camera xyz. - - camera_to_velo: [4, 4] - transformation matrix from camera xyz to velo xyz. - """ - _KITTI_MAX_HEIGHT = 512 - _KITTI_MAX_WIDTH = 1382 - - @classmethod - def Params(cls): - p = super().Params() - p.Define('decode_image', True, 'Whether to decode and produce image.') - return p - - def FeatureMap(self): - p = self.params - feature_map = { - 'image/format': - tf.io.FixedLenFeature((), tf.string, default_value='png'), - 'image/height': - tf.io.FixedLenFeature((), tf.int64, default_value=1), - 'image/width': - tf.io.FixedLenFeature((), tf.int64, default_value=1), - 'image/source_id': - tf.io.FixedLenFeature((), tf.string, default_value=''), - # The camera calibration matrices can be used later with width/height - # to perform out of camera frustum point dropping. - 'transform/velo_to_image_plane': - tf.io.FixedLenFeature(shape=(3, 4), dtype=tf.float32), - 'transform/velo_to_camera': - tf.io.FixedLenFeature(shape=(4, 4), dtype=tf.float32), - 'transform/camera_to_velo': - tf.io.FixedLenFeature(shape=(4, 4), dtype=tf.float32), - } - if p.decode_image: - feature_map['image/encoded'] = tf.io.FixedLenFeature((), - tf.string, - default_value='') - return feature_map - - def _Extract(self, features): - p = self.params - - if p.decode_image: - raw = features['image/encoded'] - image = tf.image.decode_png(raw, channels=3) - image = tf.image.convert_image_dtype(image, tf.float32) - # Padding instead of rescaling to preserve the pixel coordinates. - image = py_utils.PadOrTrimTo( - image, [self._KITTI_MAX_HEIGHT, self._KITTI_MAX_WIDTH, 3]) - - width = tf.reshape(features['image/width'], [1]) - height = tf.reshape(features['image/height'], [1]) - - velo_to_image_plane = features['transform/velo_to_image_plane'] - velo_to_camera = features['transform/velo_to_camera'] - camera_to_velo = features['transform/camera_to_velo'] - - extracted_features = py_utils.NestedMap( - width=width, - height=height, - velo_to_image_plane=velo_to_image_plane, - velo_to_camera=velo_to_camera, - camera_to_velo=camera_to_velo) - - if p.decode_image: - extracted_features.image = image - - return extracted_features - - def Shape(self): - p = self.params - shape = py_utils.NestedMap( - width=tf.TensorShape([1]), - height=tf.TensorShape([1]), - velo_to_image_plane=tf.TensorShape([3, 4]), - velo_to_camera=tf.TensorShape([4, 4]), - camera_to_velo=tf.TensorShape([4, 4])) - if p.decode_image: - shape.image = tf.TensorShape( - [self._KITTI_MAX_HEIGHT, self._KITTI_MAX_WIDTH, 3]) - return shape - - def DType(self): - p = self.params - dtype = py_utils.NestedMap( - width=tf.int64, - height=tf.int64, - velo_to_image_plane=tf.float32, - velo_to_camera=tf.float32, - camera_to_velo=tf.float32) - if p.decode_image: - dtype.image = tf.float32 - return dtype - - -# Various coordinate systems in the outputs: -# - bboxes: 2D "image" coordinate. -# - bboxes_3d[3:6] (locations): 3D "world" coordinate. -# - bboxes_3d[:3] (dimension) -# - points_xyz: 3D "world" coordinate. -# -# To convert from: -# "camera" to "world": use extrinsics/R and extrinsics/t. -# "camera" to "image": use intrinsics/K - - -class KITTILabelExtractor(input_extractor.FieldsExtractor): - """Extracts the object labels from a KITTI tf.Example. - - Emits: - bboxes_count: Scalar number of 2D bounding boxes in the example. - - bboxes: [p.max_num_objects, 4] - 2D bounding box data in [ymin, xmin, ymax, - xmax] format. - - bboxes_padding: [p.max_num_objects] - Padding for bboxes. - - bboxes_3d: [p.max_num_objects, 7] - 3D bounding box data in [x, y, z, dx, - dy, dz, phi] format. x, y, z are the object center; dx, dy, dz are the - dimensions of the box, and phi is the rotation angle around the z-axis. - 3D bboxes are defined in the velodyne coordinate frame. - - bboxes_3d_mask: [p.max_num_objects] - Mask for bboxes (mask is the inversion - of padding). - - bboxes3d_proj_to_image_plane: [p.max_num_objects, 8, 2] - For each - bounding box, the 8 corners of the bounding box in projected image - coordinates (x, y). - - bboxes_td: [p.max_num_objects, 4] - The 3D bounding box data in top down - projected coordinates (ymin, xmin, ymax, xmax). This currently ignores - rotation. - - bboxes_td_mask: [p.max_num_objects]: Mask for bboxes_td. - - bboxes_3d_num_points: [p.max_num_objects]: Number of points in each box. - - labels: [p.max_num_objects] - Integer label for each bounding box object - corresponding to the index in KITTI_CLASS_NAMES. - - texts: [p.max_num_objects] - The class name for each label in labels. - - source_id: Scalar string. The unique identifier for each example. - - See ComputeKITTIDifficulties for more info of the following:: - - box_image_height: [p.max_num_objects] - The height of the box in pixels - of each box in the projected image plane. - - occlusion: [p.max_num_objects] - The occlusion level of each bounding box. - - truncation: [p.max_num_objects] - The truncation level of each bounding box. - - difficulties: [p.max_num_objects] - The computed difficulty based on the - above three factors. - """ - KITTI_CLASS_NAMES = kitti_metadata.KITTIMetadata().ClassNames() - - # Sub-classes for filtering labels when training class specific models. - SUBCLASS_DICT = { - 'human': [4, 5], - 'cyclist': [6], - 'motor': [1, 2, 3, 7], - 'pedestrian': [4], - } - - @classmethod - def Params(cls): - p = super().Params() - p.Define('max_num_objects', 50, 'The number of objects per example.') - p.Define('filter_labels', None, 'If not None, specifies a list of label ' - 'indices to keep.') - return p - - def FeatureMap(self): - return { - 'image/source_id': - tf.io.FixedLenFeature((), tf.string, ''), - 'object/image/bbox/xmin': - tf.io.VarLenFeature(tf.float32), - 'object/image/bbox/xmax': - tf.io.VarLenFeature(tf.float32), - 'object/image/bbox/ymin': - tf.io.VarLenFeature(tf.float32), - 'object/image/bbox/ymax': - tf.io.VarLenFeature(tf.float32), - 'object/label': - tf.io.VarLenFeature(tf.string), - 'object/has_3d_info': - tf.io.VarLenFeature(dtype=tf.int64), - 'object/occlusion': - tf.io.VarLenFeature(dtype=tf.int64), - 'object/truncation': - tf.io.VarLenFeature(dtype=tf.float32), - 'object/velo/bbox/xyz': - tf.io.VarLenFeature(dtype=tf.float32), - 'object/velo/bbox/dim_xyz': - tf.io.VarLenFeature(dtype=tf.float32), - 'object/velo/bbox/phi': - tf.io.VarLenFeature(dtype=tf.float32), - 'transform/velo_to_image_plane': - tf.io.FixedLenFeature(shape=(3, 4), dtype=tf.float32), - } - - def _Extract(self, features): - p = self.params - - source_id = py_utils.HasShape(features['image/source_id'], []) - xmin = _Dense(features['object/image/bbox/xmin']) - xmax = _Dense(features['object/image/bbox/xmax']) - ymin = _Dense(features['object/image/bbox/ymin']) - ymax = _Dense(features['object/image/bbox/ymax']) - - # 2d bounding box in image coordinates. - bboxes = tf.stack([ymin, xmin, ymax, xmax], axis=1) - bboxes_count = tf.shape(bboxes)[0] - bboxes = py_utils.PadOrTrimTo(bboxes, [p.max_num_objects, 4]) - - bboxes_padding = 1.0 - py_utils.PadOrTrimTo( - tf.ones([bboxes_count]), [p.max_num_objects]) - - dim_xyz = tf.reshape(_Dense(features['object/velo/bbox/dim_xyz']), [-1, 3]) - loc_xyz = tf.reshape(_Dense(features['object/velo/bbox/xyz']), [-1, 3]) - phi = tf.reshape(_Dense(features['object/velo/bbox/phi']), [-1, 1]) - # bboxes_3d is in [x, y, z, dx, dy, dz, phi]. - bboxes_3d = tf.concat([loc_xyz, dim_xyz, phi], axis=1) - - cx, cy, _, dx, dy, _, _ = tf.unstack(bboxes_3d, num=7, axis=-1) - bboxes_td = tf.stack([ - cy - dy / 2, - cx - dx / 2, - cy + dy / 2, - cx + dx / 2, - ], axis=-1) # pyformat: disable - bboxes_td = py_utils.PadOrTrimTo(bboxes_td, [p.max_num_objects, 4]) - - has_3d_info = tf.cast(_Dense(features['object/has_3d_info']), tf.float32) - bboxes_3d_mask = py_utils.PadOrTrimTo(has_3d_info, [p.max_num_objects]) - bboxes_td_mask = bboxes_3d_mask - - # Fill in difficulties from bounding box height, truncation and occlusion. - bb_height = ymax - ymin - box_image_height = py_utils.PadOrTrimTo(bb_height, [p.max_num_objects]) - box_image_height *= bboxes_3d_mask - - # 0 to 3 indicating occlusion level. 0 means fully visible, 1 means partly, - occlusion = tf.reshape(_Dense(features['object/occlusion']), [-1]) - occlusion = tf.cast(occlusion, tf.float32) - occlusion = py_utils.PadOrTrimTo(occlusion, [p.max_num_objects]) - occlusion *= bboxes_3d_mask - - # Truncation: 0 -> not truncated, 1.0 -> truncated - truncation = tf.reshape(_Dense(features['object/truncation']), [-1]) - truncation = py_utils.PadOrTrimTo(truncation, [p.max_num_objects]) - truncation *= bboxes_3d_mask - - difficulties = ComputeKITTIDifficulties(box_image_height, occlusion, - truncation) - difficulties = py_utils.PadOrTrimTo(difficulties, [p.max_num_objects]) - - # Make a batch axis to call BBoxCorners, and take the first result back. - bbox3d_corners = geometry.BBoxCorners(bboxes_3d[tf.newaxis, ...])[0] - - # Project the 3D bbox to the image plane. - velo_to_image_plane = features['transform/velo_to_image_plane'] - bboxes3d_proj_to_image_plane = geometry.PointsToImagePlane( - tf.reshape(bbox3d_corners, [-1, 3]), velo_to_image_plane) - - # Output is [num_objects, 8 corners per object, (x, y)]. - bboxes3d_proj_to_image_plane = tf.reshape(bboxes3d_proj_to_image_plane, - [-1, 8, 2]) - bboxes3d_proj_to_image_plane = py_utils.PadOrTrimTo( - bboxes3d_proj_to_image_plane, [p.max_num_objects, 8, 2]) - - texts = features['object/label'].values - labels = ops.static_map_string_int(x=texts, keys=self.KITTI_CLASS_NAMES) - - labels = py_utils.PadOrTrimTo(labels, [p.max_num_objects]) - texts = py_utils.PadOrTrimTo(texts, [p.max_num_objects]) - - # Filter labels by setting bboxes_padding, bboxes_3d_mask, and - # bboxes_td_mask appropriately. - if p.filter_labels is not None: - valid_labels = tf.constant([p.filter_labels]) - bbox_mask = tf.reduce_any( - tf.equal(tf.expand_dims(labels, 1), valid_labels), axis=1) - bbox_mask = tf.cast(bbox_mask, tf.float32) - bboxes_padding = 1 - bbox_mask * (1 - bboxes_padding) - filtered_bboxes_3d_mask = bboxes_3d_mask * bbox_mask - bboxes_td_mask *= bbox_mask - else: - filtered_bboxes_3d_mask = bboxes_3d_mask - - # Placeholder for counting the number of laser points that reside within - # each 3-d bounding box. This must be filled in outside of this function - # based on the loaded 3-d laser points. - bboxes_3d_num_points = tf.zeros([p.max_num_objects], dtype=tf.int32) - bboxes_3d_num_points = py_utils.PadOrTrimTo(bboxes_3d_num_points, - [p.max_num_objects]) - - # Pad bboxes_3d. - bboxes_3d = py_utils.PadOrTrimTo(bboxes_3d, [p.max_num_objects, 7]) - - return py_utils.NestedMap( - source_id=source_id, - bboxes_count=bboxes_count, - bboxes=bboxes, - bboxes_padding=bboxes_padding, - bboxes_3d=bboxes_3d, - bboxes_3d_mask=filtered_bboxes_3d_mask, - unfiltered_bboxes_3d_mask=bboxes_3d_mask, - bboxes3d_proj_to_image_plane=bboxes3d_proj_to_image_plane, - bboxes_td=bboxes_td, - bboxes_td_mask=bboxes_td_mask, - bboxes_3d_num_points=bboxes_3d_num_points, - labels=labels, - texts=texts, - box_image_height=box_image_height, - occlusion=occlusion, - truncation=truncation, - difficulties=difficulties) - - def Shape(self): - p = self.params - return py_utils.NestedMap( - source_id=tf.TensorShape([]), - bboxes_count=tf.TensorShape([]), - bboxes=tf.TensorShape([p.max_num_objects, 4]), - bboxes_padding=tf.TensorShape([p.max_num_objects]), - bboxes_3d=tf.TensorShape([p.max_num_objects, 7]), - bboxes_3d_mask=tf.TensorShape([p.max_num_objects]), - unfiltered_bboxes_3d_mask=tf.TensorShape([p.max_num_objects]), - bboxes3d_proj_to_image_plane=tf.TensorShape([p.max_num_objects, 8, 2]), - bboxes_td=tf.TensorShape([p.max_num_objects, 4]), - bboxes_td_mask=tf.TensorShape([p.max_num_objects]), - bboxes_3d_num_points=tf.TensorShape([p.max_num_objects]), - labels=tf.TensorShape([p.max_num_objects]), - texts=tf.TensorShape([p.max_num_objects]), - box_image_height=tf.TensorShape([p.max_num_objects]), - occlusion=tf.TensorShape([p.max_num_objects]), - truncation=tf.TensorShape([p.max_num_objects]), - difficulties=tf.TensorShape([p.max_num_objects])) - - def DType(self): - return py_utils.NestedMap( - source_id=tf.string, - bboxes_count=tf.int32, - bboxes=tf.float32, - bboxes_padding=tf.float32, - bboxes_3d=tf.float32, - bboxes_3d_mask=tf.float32, - unfiltered_bboxes_3d_mask=tf.float32, - bboxes3d_proj_to_image_plane=tf.float32, - bboxes_td=tf.float32, - bboxes_td_mask=tf.float32, - bboxes_3d_num_points=tf.int32, - labels=tf.int32, - texts=tf.string, - box_image_height=tf.float32, - occlusion=tf.float32, - truncation=tf.float32, - difficulties=tf.int32) - - -class KITTIBase(input_extractor.BaseExtractor): - """KITTI dataset base parameters.""" - - @classmethod - def Params(cls, *args, **kwargs): - p = super().Params(*args, **kwargs) - - # Subclasses should set the following in file_datasource: - # - file_pattern_prefix: path to data directory (may be overridden at - # runtime) - # - file_pattern: file pattern of records relative to the - # data directory - p.file_datasource = datasource.PrefixedDataSource.Params() - p.file_datasource.file_type = 'tfrecord' - - return p - - @property - def class_names(self): - return KITTILabelExtractor.KITTI_CLASS_NAMES - - -class KITTILaser(KITTIBase): - """KITTI object detection dataset. - - This class emits KITTI images, labels, and the raw laser - representation of the data. See KITTIGrid and KITTISparse - for alternative laser representations. - - Input batch contains outputs from: - - KITTIImageExtractor - - KITTILabelExtractor - - KITTILaserExtractor - """ - - @classmethod - def Params(cls): - """Defaults params.""" - extractors = hyperparams.Params() - extractors.Define('labels', KITTILabelExtractor.Params(), '') - extractors.Define('lasers', KITTILaserExtractor.Params(), '') - extractors.Define('images', KITTIImageExtractor.Params(), '') - preprocessors = py_utils.NestedMap( - count_points=input_preprocessors.CountNumberOfPointsInBoxes3D.Params(), - viz_copy=input_preprocessors.CreateDecoderCopy.Params(), - pad_lasers=input_preprocessors.PadLaserFeatures.Params().Set( - max_num_points=128500)) - - p = super().Params(extractors).Set( - preprocessors=_NestedMapToParams(preprocessors), - preprocessors_order=['viz_copy', 'count_points', 'pad_lasers']) - return p - - @property - def class_names(self): - return KITTILabelExtractor.KITTI_CLASS_NAMES - - -class KITTISparseLaser(KITTIBase): - """KITTI object detection dataset for sparse detection models. - - This class emits KITTI images, labels, and the sparse laser - representation of the data. See KITTIGrid and KITTISparse - for alternative laser representations. - - Input batch contains outputs from: - - KITTILabelExtractor - - KITTILaserExtractor - - Transformed with: - - Metadata annotation: - - CountNumberOfPointsInBoxes3D - - Visualization: - - CreateDecoderCopy - - Sparse gather of points for featurization: - - SparseCenterSelector - - SparseCellGatherFeatures - - Anchor creation for classification regression targets: - - TileAnchorBBoxes - - AnchorAssignment - """ - - @classmethod - def Params(cls): - """Defaults params.""" - extractors = hyperparams.Params() - extractors.Define('labels', KITTILabelExtractor.Params(), '') - extractors.Define('lasers', KITTILaserExtractor.Params(), '') - extractors.Define('images', KITTIImageExtractor.Params(), '') - preprocessors = py_utils.NestedMap( - count_points=input_preprocessors.CountNumberOfPointsInBoxes3D.Params(), - viz_copy=input_preprocessors.CreateDecoderCopy.Params(), - keep_xyz_range=input_preprocessors.DropLaserPointsOutOfRange.Params(), - select_centers=input_preprocessors.SparseCenterSelector.Params(), - gather_features=input_preprocessors.SparseCellGatherFeatures.Params(), - tile_anchors=input_preprocessors.TileAnchorBBoxes.Params(), - assign_anchors=input_preprocessors.AnchorAssignment.Params(), - pad_lasers=input_preprocessors.PadLaserFeatures.Params().Set(), - ) - - p = super().Params(extractors).Set( - preprocessors=_NestedMapToParams(preprocessors), - preprocessors_order=[ - 'viz_copy', - 'keep_xyz_range', - 'count_points', - 'select_centers', - 'gather_features', - 'tile_anchors', - 'assign_anchors', - 'pad_lasers', - ], - ) - return p - - -class KITTIGrid(KITTIBase): - """KITTI object detection dataset. - - This class emits KITTI images, labels, and the fixed grid laser - representation of the data. - - Input batch contains outputs from: - - KITTILabelExtractor - - KITTILaserExtractor - - Transformed with: - - Metadata annotation: - - CountNumberOfPointsInBoxes3D - - Visualization: - - CreateDecoderCopy - - Points to Pillars - - PointsToGrid - - GridToPillars - - Anchor creation for classification regression targets: - - GridAnchorCenters - - TileAnchorBBoxes - - AnchorAssignment - """ - - @classmethod - def Params(cls): - """Defaults params.""" - extractors = hyperparams.Params() - extractors.Define('labels', KITTILabelExtractor.Params(), '') - extractors.Define('lasers', KITTILaserExtractor.Params(), '') - preprocessors = py_utils.NestedMap( - count_points=input_preprocessors.CountNumberOfPointsInBoxes3D.Params(), - viz_copy=input_preprocessors.CreateDecoderCopy.Params(), - points_to_grid=input_preprocessors.PointsToGrid.Params(), - grid_to_pillars=input_preprocessors.GridToPillars.Params(), - grid_anchor_centers=input_preprocessors.GridAnchorCenters.Params(), - tile_anchors=input_preprocessors.TileAnchorBBoxes.Params(), - assign_anchors=input_preprocessors.AnchorAssignment.Params(), - pad_lasers=input_preprocessors.PadLaserFeatures.Params().Set( - max_num_points=128500), - ) - p = super().Params(extractors).Set( - preprocessors=_NestedMapToParams(preprocessors), - preprocessors_order=[ - 'viz_copy', - 'count_points', - 'points_to_grid', - 'grid_to_pillars', - 'grid_anchor_centers', - 'tile_anchors', - 'assign_anchors', - 'pad_lasers', - ], - ) - return p diff --git a/lingvo/tasks/car/kitti_metadata.py b/lingvo/tasks/car/kitti_metadata.py deleted file mode 100644 index 9c2d0553..00000000 --- a/lingvo/tasks/car/kitti_metadata.py +++ /dev/null @@ -1,86 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Metadata for KITTI dataset employed in evaluation.""" - -from lingvo.tasks.car import evaluation_metadata - - -class KITTIMetadata(evaluation_metadata.EvaluationMetadata): - """Metadata describing the KITTI dataset used for evaluation.""" - - def __init__(self): - super().__init__('kitti') - - def ClassNames(self): - """Returns a list of human-interpretable strings.""" - return [ - 'Background', 'Car', 'Van', 'Truck', 'Pedestrian', 'Person_sitting', - 'Cyclist', 'Tram', 'Misc', 'DontCare' - ] - - def DifficultyLevels(self): - """Dictionary of difficulty level strings to int32 indices.""" - return {'hard': 1, 'moderate': 2, 'easy': 3} - - def IoUThresholds(self): - return { - 'Car': 0.7, - 'Pedestrian': 0.5, - 'Cyclist': 0.5, - } - - def IgnoreClassIndices(self): - """Dictionary of int32 indices for the classes that should be ignored.""" - # A detection that matches with a groundtruth bbox of any neighbor class - # will not be considered as false positive in eval. - return { - self.ClassNames().index('Car'): [self.ClassNames().index('Van')], - self.ClassNames().index('Pedestrian'): [ - self.ClassNames().index('Person_sitting') - ], - } - - def NumberOfPrecisionRecallPoints(self): - """Number of points on the precision-recall curve.""" - return 41 - - def MaximumDistance(self): - """Maximum empirically observed Euclidean distance in world coordinates.""" - return 80.0 - - def DistanceBinWidth(self): - """The width of each bin for Euclidean distance in world coordinates.""" - return 10.0 - - def MaximumNumberOfPoints(self): - """Maximum empirically observed number of points in bounding box.""" - return 3000.0 - - def NumberOfPointsBins(self): - """Number of logarithmically space bins for examining points.""" - return 20 - - def NumberOfRotationBins(self): - """Number of linear spaced rotations to bin.""" - return 10 - - def NumberOfCalibrationBins(self): - """Number of linear spaced calibration bins.""" - return 15 - - def MinHeight2D(self): - """Minimum height of detections to be evaluated.""" - return {'hard': 25, 'moderate': 25, 'easy': 40} diff --git a/lingvo/tasks/car/lr_util.py b/lingvo/tasks/car/lr_util.py deleted file mode 100644 index 4ca10426..00000000 --- a/lingvo/tasks/car/lr_util.py +++ /dev/null @@ -1,105 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Learning rate schedule utility functions.""" - -import math - -import lingvo.compat as tf -from lingvo.core import cluster_factory -from lingvo.core import py_utils -from lingvo.core import schedule - - -def _GetTrainingStatistics(train_input_p): - """Get training statistics, including total batch size and steps per epoch.""" - cluster = cluster_factory.Current() - # E.g., this is 1 for a single GPU, 8 for a 2x2 TPU, 32 for a 4x4 TPU, - # or 0 if no training job is launched. - total_num_cores = cluster.total_worker_devices - total_batch_size = max(train_input_p.batch_size * total_num_cores, 1) - steps_per_epoch = float(train_input_p.num_samples) / total_batch_size - return py_utils.NestedMap( - total_num_cores=total_num_cores, - total_batch_size=total_batch_size, - steps_per_epoch=steps_per_epoch) - - -def SetExponentialLR(train_p, - train_input_p, - exp_start_epoch, - total_epoch, - warmup_epoch=0, - limit_epoch=None, - multiplier_min=0.01, - warmup_init=0.): - """Sets a linear rampup and exponential decay LR schedule on train_p. - - This is a wrapper around LinearRampupExponentialDecayScaledByNumSplitSchedule - that sets the steps using epochs and the training statistics. - - Args: - train_p: train parameters. - train_input_p: The training set input parameters. - exp_start_epoch: The start epoch of exponential annealing. - total_epoch: Total number of epoch to train. - warmup_epoch: Epoch for the warm up ramp to end at. Note that the learning - rate will be fixed between the end of the warmup phase and the beginning - of the exponential annealing phase. - limit_epoch: Epoch to end exponential annealing. If None, this will be set - to 0.95 * total_epoch, that is, the last 5% of training time will be at - the minimum learning rate. - multiplier_min: The multiplier minimum at the end of exponential decay. - warmup_init: Initial value for the warmup phase. Note that warm up can be - disabled by either setting warmup_init to 1 or setting warmup_epoch to 0. - """ - - # Determine steps based on the training statistics, since the number of steps - # depends on the number of examples per step. - train_stats = _GetTrainingStatistics(train_input_p) - warmup_steps = warmup_epoch * train_stats.steps_per_epoch - exp_start_steps = exp_start_epoch * train_stats.steps_per_epoch - total_steps = total_epoch * train_stats.steps_per_epoch - if limit_epoch is None: - limit_epoch = 0.95 * total_epoch - limit_steps = limit_epoch * train_stats.steps_per_epoch - - tf.logging.info('#cores = %d batch size = %d steps/epoch = %d', - train_stats.total_num_cores, - train_stats.total_batch_size, - train_stats.steps_per_epoch) - tf.logging.info( - 'warmup_steps = %d exp_start_steps = %d total_steps = %d ' - 'limit_steps = %d', warmup_steps, exp_start_steps, total_steps, - limit_steps) - - assert 0. <= warmup_steps <= exp_start_steps <= limit_steps <= total_steps - - # Ensure that warmup is disabled by also setting warmup_init to 1 if - # warmup_epoch is set to 0. - if warmup_epoch == 0.: - warmup_init = 1. - - train_p.max_steps = math.ceil(total_epoch * train_stats.steps_per_epoch) - train_p.lr_schedule = ( - schedule.LinearRampupExponentialDecayScaledByNumSplitSchedule.Params()) - train_p.lr_schedule.Set( - warmup=warmup_steps, - decay_start=exp_start_steps, - decay_end=limit_steps, - min=multiplier_min, - warmup_init=warmup_init, - # Set num_splits to 1 so that no further scaling is done. - num_splits=1) diff --git a/lingvo/tasks/car/lr_util_test.py b/lingvo/tasks/car/lr_util_test.py deleted file mode 100644 index c3299715..00000000 --- a/lingvo/tasks/car/lr_util_test.py +++ /dev/null @@ -1,72 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lr_util.""" - -from lingvo import compat as tf -from lingvo.core import test_utils -from lingvo.tasks.car import kitti_input_generator -from lingvo.tasks.car import lr_util -from lingvo.tasks.car import pillars - - -class LrUtilTest(test_utils.TestCase): - - def _testInput(self): - p = kitti_input_generator.KITTIGrid.Params() - p.batch_size = 8 - p.num_samples = 128 - return p - - def _testParams(self): - p = pillars.ModelV1.Params() - p.name = 'test' - p.input = self._testInput() - return p - - def testExponentialWithLinearRamp(self): - p = self._testParams() - lr_util.SetExponentialLR( - p.train, - p.input, - warmup_epoch=1, - exp_start_epoch=2, - total_epoch=10, - warmup_init=0.) - schedule_layer = p.train.lr_schedule.Instantiate() - with self.session(): - # Linear ramp up. - self.assertLess(self.evaluate(schedule_layer.Value(8)), 1.) - # Peak learning rate. - self.assertEqual(self.evaluate(schedule_layer.Value(16)), 1.) - # Still at peak learning rate. - self.assertEqual(self.evaluate(schedule_layer.Value(24)), 1.) - # Exponential ramp down. - self.assertLess(self.evaluate(schedule_layer.Value(48)), 1.) - - def testExponentialWithoutLinearRamp(self): - p = self._testParams() - lr_util.SetExponentialLR( - p.train, p.input, exp_start_epoch=0, total_epoch=10) - schedule_layer = p.train.lr_schedule.Instantiate() - with self.session(): - # Peak learning rate at 0. - self.assertEqual(self.evaluate(schedule_layer.Value(0)), 1.) - # Exponential ramp down within first epoch. - self.assertLess(self.evaluate(schedule_layer.Value(4)), 1.) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/ops/BUILD b/lingvo/tasks/car/ops/BUILD deleted file mode 100644 index f41500c8..00000000 --- a/lingvo/tasks/car/ops/BUILD +++ /dev/null @@ -1,181 +0,0 @@ -load( - "//lingvo:lingvo.bzl", - "custom_kernel_library", - "gen_op_cclib", - "gen_op_pylib", - "lingvo_cc_library", - "lingvo_cc_test", -) - -package(default_visibility = [ - "//visibility:public", -]) - -licenses(["notice"]) # Apache 2.0 - -lingvo_cc_library( - name = "box_util", - srcs = ["box_util.cc"], - hdrs = ["box_util.h"], -) - -lingvo_cc_library( - name = "image_metrics", - srcs = ["image_metrics.cc"], - hdrs = ["image_metrics.h"], -) - -# Op definitions to be used from python. -gen_op_cclib( - name = "car_ops", - srcs = ["car_ops.cc"], -) - -gen_op_pylib( - name = "py_car_ops", - srcs = ["__init__.py"], - cc_lib_name = "car_ops", - kernel_deps = [ - ":op_kernels", - ], - py_deps = [ - "//lingvo:compat", - ], -) - -py_library( - name = "ops", - srcs = ["__init__.py"], - srcs_version = "PY3", - deps = [ - ":py_car_ops", # buildcleaner: keep - "//lingvo:compat", - ], -) - -# Op kernel deps. -cc_library( - name = "op_kernels", - deps = [ - ":average_precision_3d_op", - ":nms_3d_op", - ":pairwise_iou_op", - ":point_grid_op", - ":sampling_ops", - ], -) - -custom_kernel_library( - name = "sampling_ops", - srcs = ["sampling_ops.cc"], - op_def_lib = [":car_ops"], - deps = [ - ":ps_utils", - ], -) - -py_test( - name = "sampling_ops_test", - srcs = ["sampling_ops_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ops", - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -lingvo_cc_library( - name = "ps_utils", - srcs = ["ps_utils.cc"], - hdrs = ["ps_utils.h"], - deps = [ - # Implicit absl.synchronization dependency. - ], -) - -custom_kernel_library( - name = "pairwise_iou_op", - srcs = ["pairwise_iou_op.cc"], - op_def_lib = [":car_ops"], - deps = [ - ":box_util", - ], -) - -custom_kernel_library( - name = "point_grid_op", - srcs = ["point_grid_op.cc"], - op_def_lib = [":car_ops"], -) - -custom_kernel_library( - name = "average_precision_3d_op", - srcs = ["average_precision_3d_op.cc"], - op_def_lib = [":car_ops"], - deps = [ - ":box_util", - ":image_metrics", - ], -) - -py_test( - name = "average_precision_3d_op_test", - srcs = ["average_precision_3d_op_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ops", - "//lingvo:compat", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_test( - name = "point_grid_op_test", - srcs = ["point_grid_op_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ops", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - # Implicit tensorflow dependency. - ], -) - -custom_kernel_library( - name = "nms_3d_op", - srcs = ["nms_3d_op.cc"], - op_def_lib = [":car_ops"], - deps = [ - ":box_util", - ], -) - -py_test( - name = "nms_3d_op_test", - srcs = ["nms_3d_op_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":ops", - "//lingvo:compat", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -lingvo_cc_test( - name = "ps_utils_test", - srcs = ["ps_utils_test.cc"], - deps = [ - ":ps_utils", - ], -) diff --git a/lingvo/tasks/car/ops/__init__.py b/lingvo/tasks/car/ops/__init__.py deleted file mode 100644 index 5d0650da..00000000 --- a/lingvo/tasks/car/ops/__init__.py +++ /dev/null @@ -1,38 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Car Operations.""" - -from lingvo import compat as tf - -# Try static linking: -try: - from lingvo.tasks.car.ops import gen_car_ops # pylint: disable=g-import-not-at-top -except ImportError: - gen_car_ops = tf.load_op_library( - tf.resource_loader.get_path_to_datafile('car_ops.so')) - -# Set gen_car_ops function module so sphinx generates documentation. -for v in gen_car_ops.__dict__.values(): - try: - v.__module__ = 'lingvo.tasks.car.ops' - except: # pylint: disable=bare-except - pass - -pairwise_iou3d = gen_car_ops.pairwise_iou3d -point_to_grid = gen_car_ops.point_to_grid -non_max_suppression_3d = gen_car_ops.non_max_suppression3d -average_precision3d = gen_car_ops.average_precision3d -sample_points = gen_car_ops.sample_points diff --git a/lingvo/tasks/car/ops/average_precision_3d_op.cc b/lingvo/tasks/car/ops/average_precision_3d_op.cc deleted file mode 100644 index 389ff61b..00000000 --- a/lingvo/tasks/car/ops/average_precision_3d_op.cc +++ /dev/null @@ -1,219 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include - -#include "lingvo/tasks/car/ops/box_util.h" -#include "lingvo/tasks/car/ops/image_metrics.h" -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" -#include "tensorflow/core/lib/core/errors.h" -#include "tensorflow/core/lib/core/status.h" - -namespace tensorflow { -namespace lingvo { -namespace { - -enum class APAlgorithm { - kVOC = 0, - kKITTI = 1, -}; - -class AP3DOp final : public OpKernel { - public: - explicit AP3DOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, ctx->GetAttr("num_recall_points", &num_recall_points_)); - string ap_algorithm_name; - OP_REQUIRES_OK(ctx, ctx->GetAttr("algorithm", &ap_algorithm_name)); - if (ap_algorithm_name == "KITTI") { - ap_algorithm_ = APAlgorithm::kKITTI; - } else if (ap_algorithm_name == "VOC") { - ap_algorithm_ = APAlgorithm::kVOC; - } else { - OP_REQUIRES( - ctx, false, - errors::InvalidArgument("algorithm must be one of \"KITTI\", \"VOC\"," - "but got ", - ap_algorithm_name)); - } - - OP_REQUIRES( - ctx, num_recall_points_ > 0, - errors::InvalidArgument("num_recall_points must be positive but get ", - num_recall_points_)); - } - - void Compute(OpKernelContext* ctx) override { - const Tensor* gt_bbox = nullptr; - OP_REQUIRES_OK(ctx, ctx->input("groundtruth_bbox", >_bbox)); - OP_REQUIRES(ctx, TensorShapeUtils::IsMatrix(gt_bbox->shape()), - errors::InvalidArgument("bbox must be a matrix, but get ", - gt_bbox->shape().DebugString())); - OP_REQUIRES(ctx, gt_bbox->dim_size(1) == 7, - errors::InvalidArgument("bbox must be [:, 7], but get ", - gt_bbox->shape().DebugString())); - const int n = gt_bbox->dim_size(0); - - const Tensor* gt_imageid = nullptr; - OP_REQUIRES_OK(ctx, ctx->input("groundtruth_imageid", >_imageid)); - OP_REQUIRES(ctx, TensorShapeUtils::IsVector(gt_imageid->shape()), - errors::InvalidArgument("imageid must be a vector, but get ", - gt_imageid->shape().DebugString())); - OP_REQUIRES(ctx, gt_imageid->dim_size(0) == n, - errors::InvalidArgument("imageid shape mismatch, get ", - gt_imageid->shape().DebugString())); - const Tensor* gt_ignore = nullptr; - OP_REQUIRES_OK(ctx, ctx->input("groundtruth_ignore", >_ignore)); - OP_REQUIRES(ctx, gt_imageid->shape().IsSameSize(gt_ignore->shape()), - errors::InvalidArgument("gt_ignore shape mismatch: ", - gt_ignore->shape().DebugString())); - - std::vector> groundtruth; - std::vector groundtruth_boxes = - box::ParseBoxesFromTensor(*gt_bbox); - groundtruth.reserve(n); - // Ignore difficulty as it's currently used by VOC only. - for (int i = 0; i < n; ++i) { - image::Detection g; - g.difficult = false; - g.imgid = gt_imageid->flat()(i); - g.score = 1.0; - g.box = groundtruth_boxes[i]; - g.ignore = static_cast(gt_ignore->flat()(i)); - groundtruth.push_back(g); - } - - const Tensor* pd_bbox = nullptr; - OP_REQUIRES_OK(ctx, ctx->input("prediction_bbox", &pd_bbox)); - OP_REQUIRES(ctx, TensorShapeUtils::IsMatrix(pd_bbox->shape()), - errors::InvalidArgument("bbox must be a matrix, but get ", - pd_bbox->shape().DebugString())); - OP_REQUIRES(ctx, pd_bbox->dim_size(1) == 7, - errors::InvalidArgument("bbox must be [:, 7], but get ", - pd_bbox->shape().DebugString())); - const int m = pd_bbox->dim_size(0); - - const Tensor* pd_imageid = nullptr; - OP_REQUIRES_OK(ctx, ctx->input("prediction_imageid", &pd_imageid)); - OP_REQUIRES(ctx, TensorShapeUtils::IsVector(pd_imageid->shape()), - errors::InvalidArgument("imageid must be a vector, but get ", - pd_imageid->shape().DebugString())); - OP_REQUIRES(ctx, pd_imageid->dim_size(0) == m, - errors::InvalidArgument("imageid shape mismatch, get ", - pd_imageid->shape().DebugString())); - - const Tensor* pd_score = nullptr; - OP_REQUIRES_OK(ctx, ctx->input("prediction_score", &pd_score)); - OP_REQUIRES(ctx, TensorShapeUtils::IsVector(pd_score->shape()), - errors::InvalidArgument("score must be a vector, but get ", - pd_score->shape().DebugString())); - OP_REQUIRES(ctx, pd_score->dim_size(0) == m, - errors::InvalidArgument("score shape mismatch, get ", - pd_score->shape().DebugString())); - - const Tensor* pd_ignore = nullptr; - OP_REQUIRES_OK(ctx, ctx->input("prediction_ignore", &pd_ignore)); - OP_REQUIRES(ctx, pd_imageid->shape().IsSameSize(pd_ignore->shape()), - errors::InvalidArgument("pt_ignore shape mismatch: ", - pd_ignore->shape().DebugString())); - - const Tensor* iou_threshold = nullptr; - OP_REQUIRES_OK(ctx, ctx->input("iou_threshold", &iou_threshold)); - OP_REQUIRES( - ctx, TensorShapeUtils::IsScalar(iou_threshold->shape()), - errors::InvalidArgument("iou_threshold must be a scalar, but get ", - iou_threshold->shape().DebugString())); - - std::vector> prediction; - std::vector prediction_boxes = - box::ParseBoxesFromTensor(*pd_bbox); - - prediction.reserve(m); - for (int i = 0; i < m; ++i) { - image::Detection p; - p.difficult = false; - p.imgid = pd_imageid->flat()(i); - p.score = pd_score->flat()(i); - p.box = prediction_boxes[i]; - p.ignore = static_cast(pd_ignore->flat()(i)); - prediction.push_back(p); - } - - // Prediction score and binary indicator whether the prediction is a - // positive hit for each prediction for the specified IOU threshold. - Tensor out_score_and_hit(DT_FLOAT, {m, 2}); - auto t_out_score_and_hit = out_score_and_hit.matrix(); - std::vector is_hit; - is_hit.reserve(m); - std::vector score; - score.reserve(m); - - Tensor out_ap(DT_FLOAT, {}); - Tensor out_pr(DT_FLOAT, {num_recall_points_, 2}); - auto t_out_pr = out_pr.matrix(); - image::AveragePrecision::Options opts; - opts.iou_threshold = iou_threshold->scalar()(); - opts.num_recall_points = num_recall_points_; - std::vector pr; - if (ap_algorithm_ == APAlgorithm::kKITTI) { - out_ap.scalar()() = - image::AveragePrecision(opts).FromBoxesKITTI( - groundtruth, prediction, &pr, &is_hit, &score); - } else { - out_ap.scalar()() = - image::AveragePrecision(opts).FromBoxes( - groundtruth, prediction, &pr); - // TODO(shlens): Potentially implement this statistic for the VOC analysis - // as well. In the mean time, we just insert dummy values. - for (int i = 0; i < prediction.size(); ++i) { - score.push_back(-1.0); - is_hit.push_back(-1.0); - } - } - // Save out the detection score and the is_hit binary indicator. - CHECK_EQ(prediction.size(), is_hit.size()); - for (int i = 0; i < prediction.size(); ++i) { - t_out_score_and_hit(i, 0) = static_cast(score[i]); - t_out_score_and_hit(i, 1) = static_cast(is_hit[i]); - } - - if (pr.size() < num_recall_points_) { - LOG(WARNING) << "PR array size smaller than expected, expect: " - << num_recall_points_ << ", got " << pr.size(); - } - for (int i = 0; i < pr.size() && i < num_recall_points_; ++i) { - t_out_pr(i, 0) = pr[i].p; - t_out_pr(i, 1) = pr[i].r; - } - for (int i = pr.size(); i < num_recall_points_; ++i) { - t_out_pr(i, 0) = 0; - t_out_pr(i, 1) = i / static_cast(num_recall_points_ - 1); - } - ctx->set_output(0, out_ap); - ctx->set_output(1, out_pr); - ctx->set_output(2, out_score_and_hit); - } - - private: - int num_recall_points_ = -1; - APAlgorithm ap_algorithm_; -}; - -REGISTER_KERNEL_BUILDER(Name("AveragePrecision3D").Device(DEVICE_CPU), AP3DOp); - -} // namespace -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/tasks/car/ops/average_precision_3d_op_test.py b/lingvo/tasks/car/ops/average_precision_3d_op_test.py deleted file mode 100644 index 3320536a..00000000 --- a/lingvo/tasks/car/ops/average_precision_3d_op_test.py +++ /dev/null @@ -1,131 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lingvo.tasks.car.ops.car_metrics_ops.""" - -from lingvo import compat as tf -from lingvo.core import test_utils -from lingvo.tasks.car import ops -import numpy as np - - -class ImageMetricsOpsTest(test_utils.TestCase): - - def _GenerateRandomBBoxes(self, num_images, num_bboxes): - xyz = np.random.uniform(low=-1.0, high=1.0, size=(num_bboxes, 3)) - dimension = np.random.uniform(low=0.1, high=1.0, size=(num_bboxes, 3)) - rotation = np.random.uniform(low=-np.pi, high=np.pi, size=(num_bboxes, 1)) - bboxes = np.concatenate([xyz, dimension, rotation], axis=-1) - imageid = np.random.randint(0, num_images, size=[num_bboxes]) - scores = np.random.uniform(size=[num_bboxes]) - return bboxes, imageid, scores - - def _GetAP(self, gt_bbox, gt_imgid, pd_bbox, pd_imgid, pd_score, algorithm): - g = tf.Graph() - with g.as_default(): - iou, pr, score_and_hit = ops.average_precision3d( - iou_threshold=0.5, - groundtruth_bbox=gt_bbox, - groundtruth_imageid=gt_imgid, - groundtruth_ignore=tf.zeros_like(gt_imgid, dtype=tf.int32), - prediction_bbox=pd_bbox, - prediction_imageid=pd_imgid, - prediction_score=pd_score, - prediction_ignore=tf.zeros_like(pd_imgid, dtype=tf.int32), - num_recall_points=41, - algorithm=algorithm) - with self.session(graph=g): - val = self.evaluate([iou, pr, score_and_hit]) - return val - - def testAPKITTI(self): - k, n, m = 10, 100, 20 - gt_bbox, gt_imgid, _ = self._GenerateRandomBBoxes(k, n) - pd_bbox, pd_imgid, pd_score = self._GenerateRandomBBoxes(k, m) - # IoU between two set of random boxes; - iou, _, score_and_hit = self._GetAP( - gt_bbox, gt_imgid, pd_bbox, pd_imgid, pd_score, algorithm='KITTI') - self.assertAllEqual(score_and_hit.shape, (m, 2)) - self.assertTrue(0 <= iou and iou <= 1.0) - - # Make the predictions be a duplicate of the ground truth to emulate - # perfect detection. - iou, _, score_and_hit = self._GetAP( - gt_bbox, gt_imgid, gt_bbox, gt_imgid, np.ones(n), algorithm='KITTI') - self.assertAllEqual(score_and_hit.shape, (n, 2)) - self.assertAllEqual(score_and_hit[:, 1], np.ones(n)) - self.assertEqual(1, iou) - - # Ditto as above but make the detection scores unique so that one can test - # that the scores are correctly returned. - iou, _, score_and_hit = self._GetAP( - gt_bbox, - gt_imgid, - gt_bbox, - gt_imgid, - np.linspace(0, 1, n), - algorithm='KITTI') - self.assertAllEqual(score_and_hit.shape, (n, 2)) - self.assertAllClose(score_and_hit[:, 0], np.linspace(0, 1, n)) - self.assertAllEqual(score_and_hit[:, 1], np.ones(n)) - self.assertEqual(1, iou) - - # IoU of empty detection - iou, _, score_and_hit = self._GetAP( - gt_bbox, gt_imgid, pd_bbox, pd_imgid + n, pd_score, algorithm='KITTI') - self.assertAllEqual(score_and_hit.shape, (m, 2)) - self.assertAllEqual(score_and_hit[:, 1], np.zeros(m)) - self.assertEqual(0, iou) - - def testAPVOC(self): - k, n, m = 10, 100, 20 - gt_bbox, gt_imgid, _ = self._GenerateRandomBBoxes(k, n) - pd_bbox, pd_imgid, pd_score = self._GenerateRandomBBoxes(k, m) - # IoU between two set of random boxes; - iou, _, _ = self._GetAP( - gt_bbox, gt_imgid, pd_bbox, pd_imgid, pd_score, algorithm='VOC') - self.assertTrue(0 <= iou and iou <= 1.0) - # IoU of perfect detection - iou, _, score_and_hit = self._GetAP( - gt_bbox, gt_imgid, gt_bbox, gt_imgid, np.ones(n), algorithm='VOC') - # Just check that dummy values are returned. - self.assertAllEqual(score_and_hit.shape, (n, 2)) - self.assertAllEqual(score_and_hit, -1.0 * np.ones(shape=(n, 2))) - - self.assertEqual(1, iou) - # IoU of empty detection - iou, _, _ = self._GetAP( - gt_bbox, gt_imgid, pd_bbox, pd_imgid + n, pd_score, algorithm='VOC') - self.assertEqual(0, iou) - - def testAllZeroValue(self): - k, n, m = 10, 100, 20 - gt_bbox, gt_imgid, _ = self._GenerateRandomBBoxes(k, n) - pd_bbox, pd_imgid, pd_score = self._GenerateRandomBBoxes(k, m) - # IoU between two set of random boxes; - iou, pr, _ = self._GetAP( - gt_bbox * 0, - gt_imgid * 0, - pd_bbox * 0, - pd_imgid * 0, - pd_score * 0, - algorithm='KITTI') - self.assertEqual(0, iou) - self.assertAllEqual(pr.shape, (41, 2)) - self.assertAllEqual(np.zeros(41), pr[:, 0]) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/ops/box_util.cc b/lingvo/tasks/car/ops/box_util.cc deleted file mode 100644 index ddd6d29b..00000000 --- a/lingvo/tasks/car/ops/box_util.cc +++ /dev/null @@ -1,327 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/tasks/car/ops/box_util.h" - -#include -#include - -namespace tensorflow { -namespace lingvo { -namespace box { - -const double kEPS = 1e-8; - -// Min,max box dimensions (length, width, height). Boxes with dimensions that -// exceed these values will have box intersections of 0. -constexpr double kMinBoxDim = 1e-3; -constexpr double kMaxBoxDim = 1e6; - -// A line with the representation a*x + b*y + c = 0. -struct Line { - double a = 0; - double b = 0; - double c = 0; - - Line(const Vertex& v1, const Vertex& v2) - : a(v2.y - v1.y), b(v1.x - v2.x), c(v2.x * v1.y - v2.y * v1.x) {} - - // Computes the line value for a vertex v as a * v.x + b * v.y + c - double LineValue(const Vertex& v) const { return a * v.x + b * v.y + c; } - - // Computes the intersection point with the other line. - Vertex IntersectionPoint(const Line& other) const { - const double w = a * other.b - b * other.a; - CHECK_GT(std::fabs(w), kEPS) << "No intersection between the two lines."; - return Vertex((b * other.c - c * other.b) / w, - (c * other.a - a * other.c) / w); - } -}; - -// Computes the coordinates of its four vertices given a 2D rotated box, -std::vector ComputeBoxVertices(const double cx, const double cy, - const double w, const double h, - const double heading) { - const double dxcos = (w / 2.) * std::cos(heading); - const double dxsin = (w / 2.) * std::sin(heading); - const double dycos = (h / 2.) * std::cos(heading); - const double dysin = (h / 2.) * std::sin(heading); - return {Vertex(cx - dxcos + dysin, cy - dxsin - dycos), - Vertex(cx + dxcos + dysin, cy + dxsin - dycos), - Vertex(cx + dxcos - dysin, cy + dxsin + dycos), - Vertex(cx - dxcos - dysin, cy - dxsin + dycos)}; -} - -// Computes the intersection points between two rotated boxes, by following: -// -// 1. Initiazlizes the current intersection points with the vertices of one box, -// and the other box is taken as the cutting box; -// -// 2. For each cutting line in the cutting box (four cutting lines in total): -// For each point in the current intersection points: -// If the point is inside of the cutting line: -// Adds it to the new intersection points; -// if current point and its next point are in the opposite side of the -// cutting line: -// Computes the line of current points and its next point as tmp_line; -// Computes the intersection point between the cutting line and -// tmp_line; -// Adds the intersection point to the new intersection points; -// After checking each cutting line, sets current intersection points as -// new intersection points; -// -// 3. Returns the final intersection points. -std::vector ComputeIntersectionPoints( - const std::vector& rbox_1, const std::vector& rbox_2) { - std::vector intersection = rbox_1; - const int vertices_len = rbox_2.size(); - for (int i = 0; i < rbox_2.size(); ++i) { - const int len = intersection.size(); - if (len <= 2) { - break; - } - const Vertex& p = rbox_2[i]; - const Vertex& q = rbox_2[(i + 1) % vertices_len]; - Line cutting_line(p, q); - // Computes line value. - std::vector line_values; - line_values.reserve(len); - for (int j = 0; j < len; ++j) { - line_values.push_back(cutting_line.LineValue(intersection[j])); - } - // Updates current intersection points. - std::vector new_intersection; - for (int j = 0; j < len; ++j) { - const double s_val = line_values[j]; - const Vertex& s = intersection[j]; - // Adds the current vertex. - if (s_val <= 0 || std::fabs(s_val) <= kEPS) { - new_intersection.push_back(s); - } - const double t_val = line_values[(j + 1) % len]; - // Skips the checking of intersection point if the next vertex is on the - // line. - if (std::fabs(t_val) <= kEPS) { - continue; - } - // Adds the intersection point. - if ((s_val > 0 && t_val < 0) || (s_val < 0 && t_val > 0)) { - Line s_t_line(s, intersection[(j + 1) % len]); - new_intersection.push_back(cutting_line.IntersectionPoint(s_t_line)); - } - } - intersection = new_intersection; - } - return intersection; -} - -// Computes the area of a convex polygon, -double ComputePolygonArea(const std::vector& convex_polygon) { - const int len = convex_polygon.size(); - if (len <= 2) { - return 0; - } - double area = 0; - for (int i = 0; i < len; ++i) { - const Vertex& p = convex_polygon[i]; - const Vertex& q = convex_polygon[(i + 1) % len]; - area += p.x * q.y - p.y * q.x; - } - return std::fabs(0.5 * area); -} - -RotatedBox2D::RotatedBox2D(const double cx, const double cy, const double w, - const double h, const double heading) - : cx_(cx), cy_(cy), w_(w), h_(h), heading_(heading) { - // Compute loose bounds on dimensions of box that doesn't require computing - // full intersection. We can do this by trying to compute the largest circle - // swept by rotating the box around its center. The radius of that circle - // is the length of the ray from the center to the box corner. The upper - // bound for this value is the length of the longer dimension divided by two - // and then multiplied by root(2) (worst-case being a square box); we choose - // 1.5 as slightly higher than root(2), and then use these extrema to do - // simple extrema box checks without having to compute the true cos/sin value. - double max_dim = std::max(w_, h_) / 2. * 1.5; - loose_min_x_ = cx_ - max_dim; - loose_max_x_ = cx_ + max_dim; - loose_min_y_ = cy_ - max_dim; - loose_max_y_ = cy_ + max_dim; - - extreme_box_dim_ = (w_ <= kMinBoxDim || h_ <= kMinBoxDim); - extreme_box_dim_ |= (w_ >= kMaxBoxDim || h_ >= kMaxBoxDim); -} - -double RotatedBox2D::Area() const { - if (area_ < 0) { - const double area = ComputePolygonArea(box_vertices()); - area_ = std::fabs(area) <= kEPS ? 0 : area; - } - return area_; -} - -const std::vector& RotatedBox2D::box_vertices() const { - if (box_vertices_.empty()) { - box_vertices_ = ComputeBoxVertices(cx_, cy_, w_, h_, heading_); - } - - return box_vertices_; -} - -bool RotatedBox2D::NonZeroAndValid() const { return !extreme_box_dim_; } - -bool RotatedBox2D::MaybeIntersects(const RotatedBox2D& other) const { - // If the box dimensions of either box are too small / large, - // assume they are not well-formed boxes (otherwise we are - // subject to issues due to catastrophic cancellation). - if (extreme_box_dim_ || other.extreme_box_dim_) { - return false; - } - - // Check whether the loose extrema overlap -- if not, then there is - // no chance that the two boxes overlap even when computing the true, - // more expensive overlap. - if ((loose_min_x_ > other.loose_max_x_) || - (loose_max_x_ < other.loose_min_x_) || - (loose_min_y_ > other.loose_max_y_) || - (loose_max_y_ < other.loose_min_y_)) { - return false; - } - - return true; -} - -double RotatedBox2D::Intersection(const RotatedBox2D& other) const { - // Do a fast intersection check - if the boxes are not near each other - // then we can return early. If they are close enough to maybe overlap, - // we do the full check. - if (!MaybeIntersects(other)) { - return 0.0; - } - - // Computes the intersection polygon. - const std::vector intersection_polygon = - ComputeIntersectionPoints(box_vertices(), other.box_vertices()); - // Computes the intersection area. - const double intersection_area = ComputePolygonArea(intersection_polygon); - - return std::fabs(intersection_area) <= kEPS ? 0 : intersection_area; -} - -double RotatedBox2D::IoU(const RotatedBox2D& other) const { - // Computes the intersection area. - const double intersection_area = Intersection(other); - if (intersection_area == 0) { - return 0; - } - // Computes the union area. - const double union_area = Area() + other.Area() - intersection_area; - if (std::fabs(union_area) <= kEPS) { - return 0; - } - return intersection_area / union_area; -} - -std::vector ParseBoxesFromTensor(const Tensor& boxes_tensor) { - int num_boxes = boxes_tensor.dim_size(0); - - const auto t_boxes_tensor = boxes_tensor.matrix(); - - std::vector bboxes3d; - bboxes3d.reserve(num_boxes); - for (int i = 0; i < num_boxes; ++i) { - const double center_x = t_boxes_tensor(i, 0); - const double center_y = t_boxes_tensor(i, 1); - const double center_z = t_boxes_tensor(i, 2); - const double dimension_x = t_boxes_tensor(i, 3); - const double dimension_y = t_boxes_tensor(i, 4); - const double dimension_z = t_boxes_tensor(i, 5); - const double heading = t_boxes_tensor(i, 6); - const double z_min = center_z - dimension_z / 2; - const double z_max = center_z + dimension_z / 2; - RotatedBox2D box2d(center_x, center_y, dimension_x, dimension_y, heading); - if (dimension_x <= 0 || dimension_y <= 0) { - bboxes3d.emplace_back(RotatedBox2D(), z_min, z_max); - } else { - bboxes3d.emplace_back(box2d, z_min, z_max); - } - } - return bboxes3d; -} - -bool Upright3DBox::NonZeroAndValid() const { - // If min is larger than max, the upright box is invalid. - // - // If the min and max are equal, the height of the box is 0. and thus the box - // is zero. - if (z_min - z_max >= 0.) { - return false; - } - - return rbox.NonZeroAndValid(); -} - -double Upright3DBox::IoU(const Upright3DBox& other) const { - // Check that both boxes are non-zero and valid. Otherwise, - // return 0. - if (!NonZeroAndValid() || !other.NonZeroAndValid()) { - return 0; - } - - // Quickly check whether z's overlap; if they don't, we can return 0. - const double z_inter = - std::max(.0, std::min(z_max, other.z_max) - std::max(z_min, other.z_min)); - if (z_inter == 0) { - return 0; - } - - const double base_inter = rbox.Intersection(other.rbox); - if (base_inter == 0) { - return 0; - } - - const double volume_1 = rbox.Area() * (z_max - z_min); - const double volume_2 = other.rbox.Area() * (other.z_max - other.z_min); - const double volume_inter = base_inter * z_inter; - const double volume_union = volume_1 + volume_2 - volume_inter; - return volume_inter > 0 ? volume_inter / volume_union : 0; -} - -double Upright3DBox::Overlap(const Upright3DBox& other) const { - // Check that both boxes are non-zero and valid. Otherwise, - // return 0. - if (!NonZeroAndValid() || !other.NonZeroAndValid()) { - return 0; - } - - const double z_inter = - std::max(.0, std::min(z_max, other.z_max) - std::max(z_min, other.z_min)); - if (z_inter == 0) { - return 0; - } - - const double base_inter = rbox.Intersection(other.rbox); - if (base_inter == 0) { - return 0; - } - - const double volume_1 = rbox.Area() * (z_max - z_min); - const double volume_inter = base_inter * z_inter; - // Normalizes intersection of volume by the volume of this box. - return volume_inter > 0 ? volume_inter / volume_1 : 0; -} - -} // namespace box -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/tasks/car/ops/box_util.h b/lingvo/tasks/car/ops/box_util.h deleted file mode 100644 index 51273efb..00000000 --- a/lingvo/tasks/car/ops/box_util.h +++ /dev/null @@ -1,143 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#ifndef THIRD_PARTY_PY_LINGVO_TASKS_CAR_OPS_BOX_UTIL_H_ -#define THIRD_PARTY_PY_LINGVO_TASKS_CAR_OPS_BOX_UTIL_H_ - -#include -#include - -#include "tensorflow/core/framework/tensor.h" - -namespace tensorflow { -namespace lingvo { -namespace box { - -struct Vertex; - -// A rotated 2D bounding box represented as (cx, cy, w, h, r). cx, cy are the -// box center coordinates; w, h are the box width and height; heading is the -// rotation angle in radian relative to the 'positive x' direction. -class RotatedBox2D { - public: - // Creates an empty rotated 2D box. - RotatedBox2D() : RotatedBox2D(0, 0, 0, 0, 0) {} - - RotatedBox2D(const double cx, const double cy, const double w, const double h, - const double heading); - - // Returns the area of the box. - double Area() const; - - // Returns the intersection area between this box and the given box. - double Intersection(const RotatedBox2D& other) const; - - // Returns the IoU between this box and the given box. - double IoU(const RotatedBox2D& other) const; - - // Returns true if the box is valid (width and height are not extremely - // large or small). - bool NonZeroAndValid() const; - - private: - // Computes / caches box_vertices_ calculation. - const std::vector& box_vertices() const; - - // Returns true if this box and 'other' might intersect. - // - // If this returns false, the two boxes definitely do not intersect. If this - // returns true, it is still possible that the two boxes do not intersect, and - // the more expensive intersection code will be called. - bool MaybeIntersects(const RotatedBox2D& other) const; - - double cx_ = 0; - double cy_ = 0; - double w_ = 0; - double h_ = 0; - double heading_ = 0; - - // Loose boundaries for fast intersection test. - double loose_min_x_ = -1; - double loose_max_x_ = -1; - double loose_min_y_ = -1; - double loose_max_y_ = -1; - - // True if the dimensions of the box are very small or very large in any - // dimension. - bool extreme_box_dim_ = false; - - // The following fields are computed on demand. They are logically - // const. - - // Cached area. Access via Area() public API. - mutable double area_ = -1; - - // Stores the vertices of the box. Access via box_vertices(). - mutable std::vector box_vertices_; -}; - -// A 3D box of 7-DOFs: only allows rotation around the z-axis. -struct Upright3DBox { - RotatedBox2D rbox = RotatedBox2D(); - double z_min = 0; - double z_max = 0; - - // Creates an empty rotated 3D box. - Upright3DBox() = default; - - // Creates a 3D box from the raw input data with size 7. The data format is - // (center_x, center_y, center_z, dimension_x, dimension_y, dimension_z, - // heading) - Upright3DBox(const std::vector& raw) - : rbox(raw[0], raw[1], raw[3], raw[4], raw[6]), - z_min(raw[2] - raw[5] / 2.0), - z_max(raw[2] + raw[5] / 2.0) {} - - Upright3DBox(const RotatedBox2D& rb, const double z_min, const double z_max) - : rbox(rb), z_min(z_min), z_max(z_max) {} - - // Computes intersection over union (of the volume). - double IoU(const Upright3DBox& other) const; - - // Computes overlap: intersection of this box and the given box normalized - // over the volume of this box. - double Overlap(const Upright3DBox& other) const; - - // Returns true if the box is valid (width and height are not extremely - // large or small, and zmin < zmax). - bool NonZeroAndValid() const; -}; - -// Converts a [N, 7] tensor to a vector of N Upright3DBox objects. -std::vector ParseBoxesFromTensor(const Tensor& boxes_tensor); - -// A vertex with (x, y) coordinate. -// -// This is an internal implementation detail of RotatedBox2D. -struct Vertex { - double x = 0; - double y = 0; - - // Creates an empty Vertex. - Vertex() = default; - - Vertex(const double x, const double y) : x(x), y(y) {} -}; - -} // namespace box -} // namespace lingvo -} // namespace tensorflow - -#endif // THIRD_PARTY_PY_LINGVO_TASKS_CAR_OPS_BOX_UTIL_H_ diff --git a/lingvo/tasks/car/ops/car_ops.cc b/lingvo/tasks/car/ops/car_ops.cc deleted file mode 100644 index e9e04cc0..00000000 --- a/lingvo/tasks/car/ops/car_ops.cc +++ /dev/null @@ -1,258 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "tensorflow/core/framework/op.h" -#include "tensorflow/core/framework/shape_inference.h" - -namespace tensorflow { -namespace { - -REGISTER_OP("PairwiseIou3D") - .Input("boxes_a: float") - .Input("boxes_b: float") - .Output("iou: float") - .SetShapeFn([](tensorflow::shape_inference::InferenceContext* c) { - c->set_output( - 0, c->MakeShape({c->Dim(c->input(0), 0), c->Dim(c->input(1), 0)})); - return tensorflow::Status::OK(); - }) - .Doc(R"doc( -Calculate pairwise IoUs between two set of 3D bboxes. Every bbox is represented -as [center_x, center_y, center_z, dim_x, dim_y, dim_z, heading]. -boxes_a: A tensor of shape [num_boxes_a, 7] -boxes_b: A tensor of shape [num_boxes_b, 7] -)doc"); - -REGISTER_OP("PointToGrid") - .Input("points: float") - .Output("output_points: float") - .Output("grid_centers: float") - .Output("num_points: int32") - .Attr("num_points_per_cell: int") - .Attr("x_intervals: int") - .Attr("y_intervals: int") - .Attr("z_intervals: int") - .Attr("x_range: list(float)") - .Attr("y_range: list(float)") - .Attr("z_range: list(float)") - .SetShapeFn([](shape_inference::InferenceContext* c) { - auto feature_dim = c->Dim(c->input(0), 1); - int gx, gy, gz, n; - TF_RETURN_IF_ERROR(c->GetAttr("x_intervals", &gx)); - TF_RETURN_IF_ERROR(c->GetAttr("y_intervals", &gy)); - TF_RETURN_IF_ERROR(c->GetAttr("z_intervals", &gz)); - TF_RETURN_IF_ERROR(c->GetAttr("num_points_per_cell", &n)); - c->set_output(0, c->MakeShape({gx, gy, gz, n, feature_dim})); - c->set_output(1, c->MakeShape({gx, gy, gz, 3})); - c->set_output(2, c->MakeShape({gx, gy, gz})); - return ::tensorflow::Status::OK(); - }) - .Doc(R"doc( -Re-organize input points into equally spaced grids. Points in each grid cell are -shuffled. When not enough available points, the center of each cell with all 0 -on feature dimensions will be used as padding. - -The number of points in each grid cell. I.e., -output_points[i, j, k, num_points[i, j, k]\:, \:] are padded points. - -points: [n, d]. d >= 3 and the first 3 dimensions are treated as x,y,z. -output_points: - [x_intervals, y_intervals, z_intervals, num_per_grid, d]. -grid_centers: - [x_intervals, y_intervals, z_intervals, 3]. Grid cell centers. -num_points: - [x_intervals, y_intervals, z_intervals]. -num_points_per_cell: int. Number of points to keep in each cell. -x_intervals: int. Number of cells along x-axis. -y_intervals: int. Number of cells along y-axis. -z_intervals: int. Number of cells along z-axis. -x_range: tuple of two scalars\: (xmin, xmax). Spatial span of the grid. -y_range: tuple of two scalars\: (ymin, ymax). Spatial span of the grid. -z_range: tuple of two scalars\: (zmin, zmax). Spatial span of the grid. -)doc"); - -REGISTER_OP("NonMaxSuppression3D") - .Input("bboxes: float") - .Input("scores: float") - .Input("nms_iou_threshold: float") - .Input("score_threshold: float") - .Output("bbox_indices: int32") - .Output("bbox_scores: float") - .Output("valid_mask: float") - .Attr("max_boxes_per_class: int") - .SetShapeFn([](tensorflow::shape_inference::InferenceContext* c) { - shape_inference::ShapeHandle bboxes_shape; - shape_inference::ShapeHandle scores_shape; - shape_inference::ShapeHandle iou_threshold_shape; - shape_inference::ShapeHandle score_threshold_shape; - TF_RETURN_IF_ERROR(c->WithRank(c->input(0), 2, &bboxes_shape)); - TF_RETURN_IF_ERROR(c->WithRank(c->input(1), 2, &scores_shape)); - TF_RETURN_IF_ERROR(c->WithRank(c->input(2), 1, &iou_threshold_shape)); - TF_RETURN_IF_ERROR(c->WithRank(c->input(3), 1, &score_threshold_shape)); - - int max_boxes_per_class; - TF_RETURN_IF_ERROR( - c->GetAttr("max_boxes_per_class", &max_boxes_per_class)); - auto num_classes = c->Dim(c->input(1), 1); - - shape_inference::ShapeHandle output_shape = - c->MakeShape({num_classes, max_boxes_per_class}); - c->set_output(0, output_shape); - c->set_output(1, output_shape); - c->set_output(2, output_shape); - return tensorflow::Status::OK(); - }) - .Doc(R"doc( -Greedily selects the top subset of 3D (7 DOF format) bounding boxes per class. - -This implementation is rotation and class aware, and for each class takes the -best boxes that are above our score_threshold and also don't overlap more than -our nms_iou_threshold with any better scoring boxes. - -bboxes: A tf.float32 Tensor of shape [num_bboxes, 7] where the box is of - format [center_x, center_y, center_z, dim_x, dim_y, dim_z, heading]. -scores: A tf.float32 Tensor of shape [num_bboxes, num_classes] with a score - per box for each class. -nms_iou_threshold: A tf.float32 Tensor of shape [num_classes] specifying the - max overlap between two boxes we allow before saying these boxes overlap, - and suppressing one of them. -score_threshold: A tf.float32 Tensor of shape [num_classes] specifying the - minimum class score (per class) a box can have before it is removed. -max_boxes_per_class: An integer specifying how many (at most) boxes to - return for each class. -bbox_indices: - [num_classes, max_boxes_per_class] with the indices of selected boxes - for each class. -bbox_scores: - [num_classes, max_boxes_per_class] with the score of selected boxes - for each class. -valid_mask: - [num_classes, max_boxes_per_class] with a 1 for a valid box and a 0 - for invalid boxes for each class. -)doc"); - -REGISTER_OP("AveragePrecision3D") - .Input("iou_threshold: float") - .Input("groundtruth_bbox: float") - .Input("groundtruth_imageid: int32") - .Input("groundtruth_ignore: int32") - .Input("prediction_bbox: float") - .Input("prediction_imageid: int32") - .Input("prediction_ignore: int32") - .Input("prediction_score: float") - .Output("average_precision: float") - .Output("precision_recall: float") - .Output("score_and_hit: float") - .Attr("num_recall_points: int >= 1 = 1") - .Attr("algorithm: string = \"KITTI\"") - .Doc(R"doc( -Computes average precision for 3D bounding boxes. - -The output PR is sorted by recall in descending order. When there isn't enough -data for num_recall_points + 1 sample points, this tensor will be zero-padded. - -iou_threshold: IoU threshold. -groundtruth_bbox: [N, 7]. N ground truth bounding boxes. -groundtruth_imageid: [N]. N image ids for ground truth bounding boxes. -groundtruth_ignore: [N]. Valid values are 0 - Don't ignore; 1 - Ignore the - first match; 2 - Ignore all matches. -prediction_bbox: [M, 7]. M predicted bounding boxes. -prediction_imageid: [M]. M image ids for the predicted bounding boxes. -prediction_score: [M]. M scores for each predicted bounding box. -prediction_ignore: [N]. The ignore types for predictions. Currently only used by - the KITTI AP. Valid values are 0 - Don't ignore; 1 - Ignore the first match. -average_precision: - A scalar. The AP metric. -precision_recall: - [num_recall_points, 2]. List of PR points. -score_and_hit: - [M, 2] Prediction score and corresponding binary indication whether a - prediction detected a ground truth item. -algorithm: string. One of ["KITTI", "VOC"]. See this paper "Supervised - learning and evaluation of KITTI's cars detector with DPM", Section III.A for - the differences between KITTI AP and VOC AP. -)doc"); - -// TODO(vrv): Convert some of the attrs to inputs. -REGISTER_OP("SamplePoints") - .Input("points: float") - .Input("points_padding: float") - .Input("num_seeded_points: int32") - .Output("center: int32") - .Output("center_padding: float") - .Output("indices: int32") - .Output("indices_padding: float") - .Attr("center_selector: string") - .Attr("neighbor_sampler: string") - .Attr("neighbor_algorithm: string = \"auto\"") - .Attr("num_centers: int") - .Attr("center_z_min: float") - .Attr("center_z_max: float") - .Attr("num_neighbors: int") - .Attr("max_distance: float") - .Attr("random_seed: int = -1") - .SetShapeFn([](::tensorflow::shape_inference::InferenceContext* c) { - shape_inference::ShapeHandle points; - TF_RETURN_IF_ERROR(c->WithRank(c->input(0), 3, &points)); - shape_inference::DimensionHandle batch_size = c->Dim(points, 0); - - int num_centers, num_neighbors; - TF_RETURN_IF_ERROR(c->GetAttr("num_centers", &num_centers)); - TF_RETURN_IF_ERROR(c->GetAttr("num_neighbors", &num_neighbors)); - c->set_output(0, c->MakeShape({batch_size, num_centers})); - c->set_output(1, c->MakeShape({batch_size, num_centers})); - c->set_output(2, c->MakeShape({batch_size, num_centers, num_neighbors})); - c->set_output(3, c->MakeShape({batch_size, num_centers, num_neighbors})); - return ::tensorflow::Status::OK(); - }) - .Doc(R"doc( -Sample points among 'points'. - -points: [B, N, K]. B is the batch size; N is the number of points; K is the - number of dimensions of each point. -points_padding: [B, N]. 0/1 padding of points. If points_padding[b, i] is 0., - points[b, i, :] are valid point coordinates. Otherwise, point[b, i, :] are - all zeros. -num_seeded_points: If num_seeded_points > 0, then the first - num_seeded_points in points are considered to be seeded in the FPS - sampling. Note that we assume that these points are *not* padded, and do - not check padding when seeding them. -center: - [B, M]. the indices of selected centers. -center_padding: - [B, M]. If center_padding[b, i] is 0., center[b, i], - indices[b, i, :] and indices_padding[b, i, :] are valid sampled center. - Otherwise, center_padding[b, i] is 1.0, and center[b, i] and indices[b, i, :] - are all zeros while indices_padding[b, i, :] are all 1.0. -indices: - [B, M, P]. the indices of selected points. -indices_padding: - [B, M, P]. 0/1 padding of indices. -center_selector: Valid options - 'farthest', 'uniform'. -neighbor_sampler: Valid options - 'uniform', 'closest'. -neighbor_algorithm: Valid options - 'auto', 'hash'. -num_centers: The number of centers to sample for each batch example (M). -center_z_min: Points with z less than center_z_min are not considered for - center selection. -center_z_max: Points with z greater than center_z_max are not considered - for center selection. -num_neighbors: Sample these many points within the neighborhood (P). -max_distance: Points with L2 distances from a center larger than this - threshold are not considered to be in the neighborhood. -random_seed: The random seed. -)doc"); - -} // namespace -} // namespace tensorflow diff --git a/lingvo/tasks/car/ops/image_metrics.cc b/lingvo/tasks/car/ops/image_metrics.cc deleted file mode 100644 index d1ecfd54..00000000 --- a/lingvo/tasks/car/ops/image_metrics.cc +++ /dev/null @@ -1,156 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include "lingvo/tasks/car/ops/image_metrics.h" - -#include -#include - -namespace tensorflow { -namespace lingvo { -namespace image { - -float Box2D::Length(const Box2D::Interval& a) { - return std::max(0.f, a.max - a.min); -} - -float Box2D::Intersection(const Box2D::Interval& a, const Box2D::Interval& b) { - Interval c; - c.min = std::max(a.min, b.min); - c.max = std::min(a.max, b.max); - return Length(c); -} - -float Box2D::Area() const { return Length(x) * Length(y); } - -float Box2D::Intersection(const Box2D& other) const { - return Intersection(x, other.x) * Intersection(y, other.y); -} - -float Box2D::Union(const Box2D& other) const { - return Area() + other.Area() - Intersection(other); -} - -float Box2D::IoU(const Box2D& other) const { - const float total = Union(other); - if (total > 0) { - return Intersection(other) / total; - } else { - return 0.0; - } -} - -float Box2D::Overlap(const Box2D& other) const { - const float intersection = Intersection(other); - return intersection > 0 ? intersection / Area() : 0.0; -} - -namespace KITTI { -bool IsBetterMatch(const MatchResult& match1, const MatchResult& match2, - MatchingCriterion criterion) { - // Always true if match2 is "unmatched". - if (match2.matched_idx == kUnMatched) { - return true; - } - if (criterion == kBestScore) { - return match1.detection_score > match2.detection_score; - } else if (criterion == kBestIoU) { - if (match1.pd_ignore_type == kDontIgnore) { - if (match2.pd_ignore_type != kDontIgnore || - match1.matched_overlap > match2.matched_overlap) { - return true; - } - } - } - return false; -} - -float ComputePrecision( - const std::unordered_map>& pd_assignments, - const float score_threshold) { - int total = 0; - int tp = 0; - int self_ignore = 0, match_ignore = 0, unmatched = 0; - for (const auto& per_image_assignment : pd_assignments) { - for (const auto& match : per_image_assignment.second) { - if (match.detection_score >= score_threshold && - match.gt_ignore_type == kDontIgnore && - match.pd_ignore_type == kDontIgnore) { - total++; - if (match.matched_idx != kUnMatched) { - tp++; - } - } - // Only for debugging. - if (match.pd_ignore_type != kDontIgnore) { - self_ignore++; - } - if (match.gt_ignore_type != kDontIgnore) { - match_ignore++; - } - if (match.matched_idx == kUnMatched) { - unmatched++; - } - } - } - - if (total == 0) { - return 1; - } else { - return static_cast(tp) / total; - } -} - -std::vector FindThresholds( - const std::unordered_map>& gt_assignment, - const int num_recall_points) { - std::vector thresholds; - std::vector all_matched_scores; - int total_gt = 0; - for (const auto& per_img_assignmeht : gt_assignment) { - for (const auto& match : per_img_assignmeht.second) { - if (match.gt_ignore_type == kDontIgnore) { - if (match.matched_idx != kUnMatched && - match.pd_ignore_type == kDontIgnore) { - all_matched_scores.push_back(match.detection_score); - } - total_gt++; - } - } - } - if (total_gt == 0) { - // There's no groundtruth to evaluate. We return an empty vector. - return thresholds; - } - std::sort(all_matched_scores.begin(), all_matched_scores.end(), - std::greater()); - for (int i = 1; i <= all_matched_scores.size(); ++i) { - const float left_recall = static_cast(i) / total_gt; - const float right_recall = static_cast(i + 1) / total_gt; - const float target_recall = - thresholds.size() / static_cast(num_recall_points - 1); - if (right_recall - target_recall >= target_recall - left_recall || - i == all_matched_scores.size()) { - thresholds.push_back(all_matched_scores[i - 1]); - } - } - return thresholds; -} - -} // namespace KITTI - -} // namespace image -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/tasks/car/ops/image_metrics.h b/lingvo/tasks/car/ops/image_metrics.h deleted file mode 100644 index ea12ef6d..00000000 --- a/lingvo/tasks/car/ops/image_metrics.h +++ /dev/null @@ -1,468 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#ifndef THIRD_PARTY_PY_LINGVO_TASKS_CAR_OPS_IMAGE_METRICS_H_ -#define THIRD_PARTY_PY_LINGVO_TASKS_CAR_OPS_IMAGE_METRICS_H_ - -#include -#include -#include - -#include "tensorflow/core/framework/tensor.h" - -namespace tensorflow { -namespace lingvo { -namespace image { - -struct Box2D { - struct Interval { - float min = 0; - float max = 0; - }; - Interval x; - Interval y; - static float Length(const Interval& a); - static float Intersection(const Interval& a, const Interval& b); - float Area() const; - float Intersection(const Box2D& other) const; - float Union(const Box2D& other) const; - // Intersection of this box and the given box normalized over the union of - // this box and the given box. - float IoU(const Box2D& other) const; - // Intersection of this box and the given box normalized over the area of - // this box. - float Overlap(const Box2D& other) const; -}; - -// If the value is: -// - kDontIgnore: The object is included in this evaluation. -// - kIgnoreOneMatch: the first matched prediction bbox will be ignored. This -// is useful when this groundtruth object is not intended to be evaluated. -// - kIgnoreAllMatches: all matched prediction bbox will be ignored. Typically -// it is used to mark an area that has not been labeled. -enum IgnoreType { - kDontIgnore = 0, - kIgnoreOneMatch = 1, - kIgnoreAllMatches = 2, -}; - -template -struct Detection { - public: - bool difficult = false; - int64_t imgid = 0; - float score = 0; - BoxType box; - IgnoreType ignore = IgnoreType::kDontIgnore; -}; - -// Precision and recall. -struct PR { - float p = 0; - float r = 0; - PR(const float p_, const float r_) : p(p_), r(r_) {} -}; - -template -class AveragePrecision { - public: - // iou_threshold: A predicted box matches a ground truth box if and only if - // IoU between these two are larger than this iou_threshold. Default: 0.5. - // num_recall_points: AP is computed as the average of maximum precision at (1 - // + num_recall_points) recall levels. E.g., if num_recall_points is 10, - // recall levels are 0., 0.1, 0.2, ..., 0.9, 1.0. Default: 100 - // normalized_coordinate: If true, bounding box's coordinates are within [0, - // 1), and the interval is left-close and right-open. Hence, the length of an - // interval is simply interval.max - interval.min. Otherwise, coordinates are - // in the image pixel space and the interval is close on both ends. I.e., the - // length of an interval is interval.max - interval.min + 1. - // NOTE: voc_eval.py used by Fast-RCNN sets normalized_coordinate=false. - struct Options { - float iou_threshold = 0.5; - int num_recall_points = 100; - }; - AveragePrecision() : AveragePrecision(Options()) {} - explicit AveragePrecision(const Options& opts) : opts_(opts) {} - - // Given a sequence of precision-recall points ordered by the recall in - // non-increasing order, returns the average of maximum precisions at - // different recall values (0.0, 0.1, 0.2, ..., 0.9, 1.0). - // The p-r pairs at these fixed recall points will be written to pr_out, if - // it is not null_ptr. - float FromPRCurve(const std::vector& pr, - std::vector* pr_out = nullptr); - - // An axis aligned bounding box for an image with id 'imageid'. Score - // indicates its confidence. - // - // 'difficult' is a special bit specific to Pascal VOC dataset and tasks using - // the data. If 'difficult' is true, by convention, the box is often ignored - // during the AP calculation. I.e., if a predicted box matches a 'difficult' - // ground box, this predicted box is ignored as if the model does not make - // such a prediction. - - // Given the set of ground truth boxes and a set of predicted boxes, returns - // the average of the maximum precisions at different recall values. - float FromBoxes(const std::vector>& groundtruth, - const std::vector>& prediction, - std::vector* pr_out = nullptr); - - // The KITTI variant of AP calculation. - // Comparing with VOC/COCO AP, there are 3 main differences. - // - KITTI AP has two matching passes. The 1st pass matches every gt bbox - // with the pd bbox of highest score while having an IoU larger than the - // threshold. The 2nd pass matches every gt with the pd bbox of largest - // IoU. In VOC/COCO AP there's only one pass that does the same as KITTI's - // 1st pass. - // - // - The operating points on PR curve are decided by the result of the 1st - // pass whereas the actuall PR values are computed using 2nd pass. - // - // - Special "ignore" bit can be set on groundtruth or prediction bboxes. - // - On groundtruth, "ignore" means the bbox is of a neighbor class, or - // of a "difficuty" level that is not intended to be evaluated. - // - On prediction, "ignore" means the bbox, when projected to the - // camera image space, has its height smaller than a threshold. - float FromBoxesKITTI(const std::vector>& groundtruth, - const std::vector>& prediction, - std::vector* pr_out = nullptr, - std::vector* is_hit = nullptr, - std::vector* score = nullptr); - - private: - Options opts_; -}; - -template -float AveragePrecision::FromPRCurve(const std::vector& pr, - std::vector* pr_out) { - // Because pr[...] are ordered by recall, iterate backward to compute max - // precision. p(r) = max_{r' >= r} p(r') for r in 0.0, 0.1, 0.2, ..., 0.9, - // 1.0. Then, take the average of (num_recal_points) quantities. - float p = 0; - float sum = 0; - int r_level = opts_.num_recall_points; - for (int i = pr.size() - 1; i >= 0; --i) { - const PR& item = pr[i]; - if (i > 0) { - CHECK_GE(item.r, pr[i - 1].r); // Ordered. - } - // Because r takes values opts_.num_recall_points, opts_.num_recall_points - - // 1, ..., 0, the following condition is checking whether item.r crosses r / - // opts_.num_recall_points. I.e., 1.0, 0.90, ..., 0.01, 0.0. We don't use - // float to represent r because 0.01 is not representable precisely. - while (item.r * opts_.num_recall_points < r_level) { - const float recall = - static_cast(r_level) / opts_.num_recall_points; - CHECK_GE(r_level, 0); - sum += p; - r_level -= 1; - if (pr_out != nullptr) { - pr_out->emplace_back(p, recall); - } - } - p = std::max(p, item.p); - } - for (; r_level >= 0; --r_level) { - const float recall = static_cast(r_level) / opts_.num_recall_points; - sum += p; - if (pr_out != nullptr) { - pr_out->emplace_back(p, recall); - } - } - return sum / (1 + opts_.num_recall_points); -} - -template -float AveragePrecision::FromBoxes( - const std::vector>& groundtruth, - const std::vector>& prediction, - std::vector* pr_out) { - // Index ground truth boxes based on imageid. - std::unordered_map>> gt; - int num_gt = 0; - for (auto& box : groundtruth) { - gt[box.imgid].push_back(box); - if (!box.difficult && box.ignore == kDontIgnore) { - ++num_gt; - } - } - - if (num_gt == 0) { - return NAN; - } - - // Sort all predicted boxes by their scores in a non-ascending order. - std::vector> pd = prediction; - std::sort(pd.begin(), pd.end(), - [](const Detection& a, const Detection& b) { - return a.score > b.score; - }); - - // Computes p-r for every prediction. - std::vector pr; - int correct = 0; - int num_pd = 0; - for (int i = 0; i < pd.size(); ++i) { - const Detection& b = pd[i]; - auto* g = >[b.imgid]; - auto best = g->end(); - float best_iou = -INFINITY; - for (auto it = g->begin(); it != g->end(); ++it) { - const auto iou = b.box.IoU(it->box); - if (iou > best_iou) { - best = it; - best_iou = iou; - } - } - if ((best != g->end()) && (best_iou >= opts_.iou_threshold)) { - if (best->difficult) { - continue; - } - switch (best->ignore) { - case kDontIgnore: { - ++correct; - ++num_pd; - g->erase(best); - pr.push_back({static_cast(correct) / num_pd, - static_cast(correct) / num_gt}); - break; - } - case kIgnoreOneMatch: { - g->erase(best); - break; - } - case kIgnoreAllMatches: { - break; - } - } - } else { - ++num_pd; - pr.push_back({static_cast(correct) / num_pd, - static_cast(correct) / num_gt}); - } - } - return FromPRCurve(pr, pr_out); -} - -namespace KITTI { - -enum MatchingCriterion { - // Match a groundtruth to the prediction of highest score. - kBestScore = 0, - // Match a groundtruth to the prediction of largest IoU. - kBestIoU = 1, -}; - -const int kUnMatched = -1; -struct MatchResult { - int matched_idx = kUnMatched; - // When MatchingMode is kBestIoU and matched_idx is not kUnMatched, - // this field will contain the IoU between the prediction and the groundtruth. - float matched_overlap = 0; - // When MatchingMode is kBestIoU and matched_idx is not kUnMatched, - // this field will contain the confidence score of the prediction. - float detection_score = 0; - IgnoreType gt_ignore_type = IgnoreType::kDontIgnore; - IgnoreType pd_ignore_type = IgnoreType::kDontIgnore; -}; - -// Whether match1 is a better match than match2. -bool IsBetterMatch(const MatchResult& match1, const MatchResult& match2, - MatchingCriterion criterion); - -// Compute precision from detection assignments given a score threshold. -// Input arguments: -// pd_assignments: the 2nd pass matching results on all predictions. -// score_threshold: only the predictions with score above this threshold will -// be evaluated. -float ComputePrecision( - const std::unordered_map>& pd_assignments, - const float score_threshold); - -// Match groundtruth with predictions from one scene. -template -void MatchOneScene(const std::vector>& groundtruth, - const std::vector>& prediction, - const MatchingCriterion criterion, const float iou_threshold, - const float score_threshold, - std::vector* gt_assignment, - std::vector* pd_assignment) { - gt_assignment->clear(); - pd_assignment->clear(); - gt_assignment->resize(groundtruth.size()); - pd_assignment->resize(prediction.size()); - for (int j_pd = 0; j_pd < prediction.size(); ++j_pd) { - // Transfer over detection scores and ignore type from 'prediction' - pd_assignment->at(j_pd).detection_score = prediction[j_pd].score; - pd_assignment->at(j_pd).pd_ignore_type = prediction[j_pd].ignore; - } - for (int i_gt = 0; i_gt < groundtruth.size(); ++i_gt) { - if (groundtruth[i_gt].ignore == IgnoreType::kIgnoreAllMatches) { - continue; - } - MatchResult best_matched_pd; - best_matched_pd.gt_ignore_type = groundtruth[i_gt].ignore; - for (int j_pd = 0; j_pd < prediction.size(); ++j_pd) { - // For this groundtruth, find the best unassigned prediction that matches. - if (pd_assignment->at(j_pd).matched_idx != kUnMatched || - prediction[j_pd].score < score_threshold) { - // Skip predictions that have already been matched, or predictions below - // the score threshold. - continue; - } - const float iou = groundtruth[i_gt].box.IoU(prediction[j_pd].box); - MatchResult curr; - curr.matched_idx = j_pd; - curr.matched_overlap = iou; - curr.detection_score = prediction[j_pd].score; - curr.gt_ignore_type = groundtruth[i_gt].ignore; - curr.pd_ignore_type = prediction[j_pd].ignore; - if (iou > iou_threshold && - IsBetterMatch(curr, best_matched_pd, criterion)) { - best_matched_pd = curr; - } - } - MatchResult best_matched_gt = best_matched_pd; - best_matched_gt.matched_idx = i_gt; - - gt_assignment->at(i_gt) = best_matched_pd; - if (best_matched_pd.matched_idx != kUnMatched) { - pd_assignment->at(best_matched_pd.matched_idx) = best_matched_gt; - } - } -} - -// Input arguments: -// groundtruth: image_id -> list of bboxes mapping. -// prediction: image_id -> list of bboxes mapping. The key set must be the -// same as the one in groundtruth. -// criterion: should be kBestScore in 1st pass matching and kBestIoU in the -// 2nd pass. -// iou_threshold: bboxes can be matched if ther IoU is greater than this. -// score_thresold: predictions with score lower than this are ignored. -// gt_assignments: the matching results for every groundtruth. -// pd_assignments: the matching results for every prediction. -template -void MatchAll( - const std::unordered_map>>& - groundtruth, - const std::unordered_map>>& - prediction, - const MatchingCriterion criterion, const float iou_threshold, - const float score_threshold, - std::unordered_map>* gt_assignments, - std::unordered_map>* pd_assignments) { - CHECK_EQ(groundtruth.size(), prediction.size()); - std::vector all_img_keys; - for (const auto& gt_iter : groundtruth) { - const int64_t img_id = gt_iter.first; - const auto pd_iter = prediction.find(img_id); - CHECK(pd_iter != prediction.end()) - << "Groundtruth and prediction must have the same key set"; - const auto& gt = gt_iter.second; - const auto& pd = pd_iter->second; - MatchOneScene(gt, pd, criterion, iou_threshold, score_threshold, - &((*gt_assignments)[img_id]), &((*pd_assignments)[img_id])); - } -} - -std::vector FindThresholds( - const std::unordered_map>& gt_assignment, - const int num_recall_points); - -} // namespace KITTI - -// TODO(shlens): Consider removing score as a returned value as this is in -// theory already available in prediction. Another option is to combine as a -// std::pair with is_hit. -template -float AveragePrecision::FromBoxesKITTI( - const std::vector>& groundtruth, - const std::vector>& prediction, std::vector* pr_out, - std::vector* is_hit, std::vector* score) { - std::unordered_map>> gt_bins; - std::unordered_map>> pd_bins; - // Ensure that gt_bins and pd_bins have the same key sets. - for (const auto& box : groundtruth) { - gt_bins[box.imgid].push_back(box); - if (pd_bins.find(box.imgid) == pd_bins.end()) { - pd_bins[box.imgid] = std::vector>(); - } - } - std::vector index_to_image_id; - index_to_image_id.reserve(prediction.size()); - std::vector index_to_prediction_index; - index_to_prediction_index.reserve(prediction.size()); - for (int i = 0; i < prediction.size(); ++i) { - auto& box = prediction[i]; - pd_bins[box.imgid].push_back(box); - index_to_image_id.push_back(box.imgid); - index_to_prediction_index.push_back(pd_bins[box.imgid].size() - 1); - if (gt_bins.find(box.imgid) == gt_bins.end()) { - gt_bins[box.imgid] = std::vector>(); - } - } - // Pass 1 matching: find overlapping detection of best score. - std::unordered_map> gt_assignments; - std::unordered_map> pd_assignments; - KITTI::MatchAll(gt_bins, pd_bins, KITTI::kBestScore, opts_.iou_threshold, 0, - >_assignments, &pd_assignments); - for (int i = 0; i < prediction.size(); ++i) { - const int image_id = index_to_image_id[i]; - const int prediction_index = index_to_prediction_index[i]; - const float detection_score = - pd_assignments[image_id][prediction_index].detection_score; - float correct = 0.0; - if (pd_assignments[image_id][prediction_index].matched_idx != - KITTI::kUnMatched) { - correct = 1.0; - } - if (score != nullptr) { - score->push_back(detection_score); - } - if (is_hit != nullptr) { - is_hit->push_back(correct); - } - } - - const auto thresholds = - KITTI::FindThresholds(gt_assignments, opts_.num_recall_points); - pr_out->clear(); - float ap = 0; - for (int i = 0; i < thresholds.size(); ++i) { - // Pass 2 matching: find detection above the score threshold with largest - // overlap. - KITTI::MatchAll(gt_bins, pd_bins, KITTI::kBestIoU, opts_.iou_threshold, - thresholds[i], >_assignments, &pd_assignments); - const float precision = - KITTI::ComputePrecision(pd_assignments, thresholds[i]); - const float recall = static_cast(i) / (opts_.num_recall_points - 1); - - pr_out->push_back({precision, recall}); - ap += precision; - } - - for (int i = thresholds.size() - 2; i >= 0; --i) { - pr_out->at(i).p = std::max(pr_out->at(i + 1).p, pr_out->at(i).p); - } - return ap / opts_.num_recall_points; -} - -} // namespace image -} // namespace lingvo -} // namespace tensorflow - -#endif // THIRD_PARTY_PY_LINGVO_TASKS_CAR_OPS_IMAGE_METRICS_H_ diff --git a/lingvo/tasks/car/ops/nms_3d_op.cc b/lingvo/tasks/car/ops/nms_3d_op.cc deleted file mode 100644 index cc10e636..00000000 --- a/lingvo/tasks/car/ops/nms_3d_op.cc +++ /dev/null @@ -1,196 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include -#include -#include - -#include "lingvo/tasks/car/ops/box_util.h" -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" -#include "tensorflow/core/lib/core/errors.h" -#include "tensorflow/core/platform/logging.h" - -namespace tensorflow { -namespace lingvo { -namespace { - -struct PairHashFn { - size_t operator()(const std::pair& pair) const { - return std::hash{}((static_cast(pair.first) << 32) | - static_cast(pair.second)); - } -}; - -class NonMaxSuppression3DOp : public OpKernel { - public: - explicit NonMaxSuppression3DOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, - ctx->GetAttr("max_boxes_per_class", &max_boxes_per_class_)); - } - - void CheckShapes(OpKernelContext* ctx) { - const Tensor& bboxes_3d = ctx->input(0); - const Tensor& class_scores = ctx->input(1); - const Tensor& nms_iou_threshold = ctx->input(2); - const Tensor& score_threshold = ctx->input(3); - OP_REQUIRES(ctx, TensorShapeUtils::IsMatrix(bboxes_3d.shape()), - errors::InvalidArgument("In[0] must be a matrix, but get ", - bboxes_3d.shape().DebugString())); - OP_REQUIRES(ctx, TensorShapeUtils::IsMatrix(class_scores.shape()), - errors::InvalidArgument("In[1] must be a matrix, but get ", - class_scores.shape().DebugString())); - OP_REQUIRES( - ctx, TensorShapeUtils::IsVector(nms_iou_threshold.shape()), - errors::InvalidArgument("In[2] must be a vector, but get ", - nms_iou_threshold.shape().DebugString())); - OP_REQUIRES(ctx, TensorShapeUtils::IsVector(score_threshold.shape()), - errors::InvalidArgument("In[3] must be a vector, but get ", - score_threshold.shape().DebugString())); - OP_REQUIRES(ctx, bboxes_3d.dim_size(1) == 7, - errors::InvalidArgument("bboxes must be of shape [-1, 7]. Is: ", - bboxes_3d.shape().DebugString())); - const int num_classes = class_scores.dim_size(1); - OP_REQUIRES(ctx, nms_iou_threshold.dim_size(0) == num_classes, - errors::InvalidArgument( - "nms_iou_threshold must be of shape [num_classes]. Is: ", - nms_iou_threshold.shape().DebugString())); - OP_REQUIRES(ctx, score_threshold.dim_size(0) == num_classes, - errors::InvalidArgument( - "score_threshold must be of shape [num_classes]. Is: ", - nms_iou_threshold.shape().DebugString())); - } - - void Compute(OpKernelContext* ctx) override { - CheckShapes(ctx); - const Tensor& bboxes_3d = ctx->input(0); - const Tensor& class_scores = ctx->input(1); - const Tensor& nms_iou_threshold = ctx->input(2); - const Tensor& score_threshold = ctx->input(3); - - const int num_bboxes = bboxes_3d.dim_size(0); - const int num_classes = class_scores.dim_size(1); - - // Parse data into usable forms - std::vector boxes = box::ParseBoxesFromTensor(bboxes_3d); - const auto& t_class_scores = class_scores.matrix(); - const auto& t_nms_iou_threshold = nms_iou_threshold.vec(); - const auto& t_score_threshold = score_threshold.vec(); - - struct Candidate { - int box_idx; - float score; - }; - - auto score_cmp = [](const Candidate& box1, const Candidate& box2) { - return box1.score < box2.score; - }; - - // Allocate outputs - Tensor* bbox_indices = nullptr; - Tensor* bbox_scores = nullptr; - Tensor* valid_mask = nullptr; - - auto output_shape = TensorShape({num_classes, max_boxes_per_class_}); - OP_REQUIRES_OK( - ctx, ctx->allocate_output("bbox_indices", output_shape, &bbox_indices)); - OP_REQUIRES_OK( - ctx, ctx->allocate_output("bbox_scores", output_shape, &bbox_scores)); - OP_REQUIRES_OK( - ctx, ctx->allocate_output("valid_mask", output_shape, &valid_mask)); - auto t_bbox_indices = bbox_indices->matrix(); - auto t_bbox_scores = bbox_scores->matrix(); - auto t_valid_mask = valid_mask->matrix(); - t_bbox_indices.setZero(); - t_bbox_scores.setZero(); - t_valid_mask.setZero(); - - // Create cache for IoU calculations - // Keys are pairs of box ids where the first id is always less than the - // second id due to symmetry in the calculation. - std::unordered_map, float, PairHashFn> iou_cache; - auto get_iou = [&iou_cache, &boxes](int32 idx1, int32 idx2) { - std::pair key{std::min(idx1, idx2), std::max(idx1, idx2)}; - auto it = iou_cache.find(key); - if (it != iou_cache.end()) { - return it->second; - } - - float iou = boxes[idx1].IoU(boxes[idx2]); - iou_cache.emplace(key, iou); - return iou; - }; - - for (int cls_idx = 0; cls_idx < num_classes; ++cls_idx) { - // Use priority queue to sort candidates above the score threshold - std::priority_queue, decltype(score_cmp)> - candidate_priority_queue(score_cmp); - for (int box_idx = 0; box_idx < num_bboxes; ++box_idx) { - if (t_class_scores(box_idx, cls_idx) >= t_score_threshold(cls_idx)) { - candidate_priority_queue.emplace( - Candidate({box_idx, t_class_scores(box_idx, cls_idx)})); - } - } - - std::vector selected; - Candidate next_candidate; - while ((selected.size() < max_boxes_per_class_) && - (!candidate_priority_queue.empty())) { - next_candidate = candidate_priority_queue.top(); - candidate_priority_queue.pop(); - - // Idea taken from tensorflow/core/kernels/non_max_suppression_op.cc - // Overlapping boxes are likely to have similar scores, - // therefore we iterate through the previously selected boxes backwards - // in order to see if `next_candidate` should be suppressed. - bool should_select = true; - for (int selected_idx = static_cast(selected.size()) - 1; - selected_idx >= 0; --selected_idx) { - if (get_iou(next_candidate.box_idx, selected[selected_idx].box_idx) > - t_nms_iou_threshold(cls_idx)) { - should_select = false; - break; - } - } - - if (should_select) { - selected.push_back(next_candidate); - } - } - - // For each class, copy results into output tensors. - // We can just use size since we protect against ever selecting more - // than max_boxes_per_class_ per class. - for (int insert_idx = 0; insert_idx < selected.size(); insert_idx++) { - const auto& to_insert = selected[insert_idx]; - t_bbox_indices(cls_idx, insert_idx) = to_insert.box_idx; - t_bbox_scores(cls_idx, insert_idx) = to_insert.score; - t_valid_mask(cls_idx, insert_idx) = 1.0; - } - } - } - - private: - // These are attributes and logically const. - int max_boxes_per_class_; -}; - -REGISTER_KERNEL_BUILDER(Name("NonMaxSuppression3D").Device(DEVICE_CPU), - NonMaxSuppression3DOp); - -} // namespace -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/tasks/car/ops/nms_3d_op_test.py b/lingvo/tasks/car/ops/nms_3d_op_test.py deleted file mode 100644 index 1b4f07e0..00000000 --- a/lingvo/tasks/car/ops/nms_3d_op_test.py +++ /dev/null @@ -1,204 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -import time -import unittest -from lingvo import compat as tf -from lingvo.core import test_utils -from lingvo.tasks.car import ops -import numpy as np - - -class Nms3dOpTest(test_utils.TestCase): - - def _GetData(self): - # Assignments and IoU scores derived from externally calculated test cases - # created with shapely. - bboxes = tf.constant( - [ - [10.35, 8.429, -1.003, 3.7, 1.64, 1.49, 1.582], - [10.35, 8.429, -1.003, 3.7, 1.64, 1.49, 0.0 - ], # Box 0 rotated ~90 deg - [11.5, 8.429, -1.003, 3.7, 1.64, 1.49, 1.0], # Rotated to overlap - [13.01, 8.149, -0.953, 4.02, 1.55, 1.52, 1.592], - [13.51, 8.39, -1.0, 4.02, 1.55, 1.52, 1.592], # Slight translation - [13.51, 8.39, -1.0, 1.0, 1.0, 1.52, 1.592], # Smaller box - [13.51, 8.39, -1.0, 1.0, 1.0, 1.52, 1.9], # Smaller box - ], - dtype=tf.float32) - - scores = tf.constant([ - [0.9, 0.09, 0.0], - [0.88, 0.109, 0.011], - [0.5, 0.01, 0.49], - [0.8, 0.1, 0.1], - [0.79, 0.12, 0.19], - [0.2, 0.79, 0.11], - [0.1, 0.9, 0.0], - ], - dtype=tf.float32) - return bboxes, scores - - def _TestNMSOp(self, bboxes_3d, class_scores, nms_iou_threshold, - score_threshold, max_boxes_per_class, expected_indices): - with self.session(): - bbox_indices, bbox_scores, valid_mask = ops.non_max_suppression_3d( - bboxes_3d, - class_scores, - nms_iou_threshold=nms_iou_threshold, - score_threshold=score_threshold, - max_boxes_per_class=max_boxes_per_class) - bbox_idx, scores, mask = self.evaluate( - [bbox_indices, bbox_scores, valid_mask]) - - num_classes = len(expected_indices) - expected_shape = (num_classes, max_boxes_per_class) - self.assertEqual(bbox_idx.shape, expected_shape) - self.assertEqual(scores.shape, expected_shape) - self.assertEqual(mask.shape, expected_shape) - - total_expected_valid_boxes = sum([len(exp) for exp in expected_indices]) - self.assertEqual(mask.sum(), total_expected_valid_boxes) - - for cls_idx in range(num_classes): - cls_mask = mask[cls_idx, :].astype(np.bool) - self.assertEqual(cls_mask.sum(), len(expected_indices[cls_idx])) - self.assertAllEqual(bbox_idx[cls_idx, cls_mask], - expected_indices[cls_idx]) - - def testMultiClassNMS(self): - bboxes_3d, class_scores = self._GetData() - expected_indices = [[0, 3], [6], [2]] - self._TestNMSOp( - bboxes_3d, - class_scores, - nms_iou_threshold=[0.1, 0.1, 0.1], - score_threshold=[0.3, 0.3, 0.3], - max_boxes_per_class=5, - expected_indices=expected_indices) - - def testLowerScoreThreshold(self): - bboxes_3d, class_scores = self._GetData() - # Lower threshold means more boxes are included. - expected_indices = [[0, 3], [6, 1], [2, 4]] - self._TestNMSOp( - bboxes_3d, - class_scores, - nms_iou_threshold=[0.1, 0.1, 0.1], - score_threshold=[0.01, 0.01, 0.01], - max_boxes_per_class=5, - expected_indices=expected_indices) - - def testHighIoUThreshold(self): - bboxes_3d, class_scores = self._GetData() - expected_indices = [[0, 1, 3, 4, 2, 5, 6], [6, 5, 4, 1, 3, 0, 2], - [2, 4, 5, 3, 1]] - # Increase IoU Threshold and max number of boxes so - # all non-zero score boxes are returned. - self._TestNMSOp( - bboxes_3d, - class_scores, - nms_iou_threshold=[0.999, 0.999, 0.999], - score_threshold=[0.01, 0.01, 0.01], - max_boxes_per_class=10, - expected_indices=expected_indices) - - def testOneClassVsMultiClass(self): - # Check running on all 3 classes versus each independently. - bboxes_3d, class_scores = self._GetData() - num_classes = 3 - max_boxes_per_class = 5 - with self.session(): - bbox_indices, bbox_scores, valid_mask = ops.non_max_suppression_3d( - bboxes_3d, - class_scores, - nms_iou_threshold=[0.1, 0.1, 0.1], - score_threshold=[0.3, 0.3, 0.3], - max_boxes_per_class=max_boxes_per_class) - multiclass_indices, multiclass_scores, multiclass_valid_mask = self.evaluate( - [bbox_indices, bbox_scores, valid_mask]) - self.assertEqual(multiclass_indices.shape, - (num_classes, max_boxes_per_class)) - self.assertEqual(multiclass_scores.shape, - (num_classes, max_boxes_per_class)) - self.assertEqual(multiclass_valid_mask.shape, - (num_classes, max_boxes_per_class)) - - # For each class, get results for just that class and compare. - for cls_idx in range(num_classes): - bbox_idx, bbox_scores, valid_mask = ops.non_max_suppression_3d( - bboxes_3d, - class_scores[:, cls_idx:cls_idx + 1], - nms_iou_threshold=[0.1], - score_threshold=[0.3], - max_boxes_per_class=max_boxes_per_class) - per_class_indices, per_class_scores, per_class_valid_mask = self.evaluate( - [bbox_idx, bbox_scores, valid_mask]) - - self.assertEqual(per_class_indices.shape, (1, max_boxes_per_class)) - self.assertEqual(per_class_scores.shape, (1, max_boxes_per_class)) - self.assertEqual(per_class_valid_mask.shape, (1, max_boxes_per_class)) - - per_class_mask = per_class_valid_mask[0, :].astype(np.bool) - multiclass_mask = multiclass_valid_mask[cls_idx, :].astype(np.bool) - self.assertAllEqual(per_class_indices[0, per_class_mask], - multiclass_indices[cls_idx, multiclass_mask]) - self.assertAllEqual(per_class_scores[0, per_class_mask], - multiclass_scores[cls_idx, multiclass_mask]) - - @unittest.skip('Speed benchmark') - def testSpeed(self): - num_bboxes_list = [500, 1000, 10000] - num_classes_list = [3, 10, 25] - - for num_bboxes in num_bboxes_list: - for num_classes in num_classes_list: - bboxes_3d = tf.random.uniform((num_bboxes, 7), - minval=0.1, - maxval=2, - dtype=tf.float32) - # Make half zero so we can see behavior with very low values that - # will get filtered out quickly. - class_scores = tf.concat([ - tf.random.uniform((num_bboxes // 2, num_classes), - minval=0, - maxval=1, - dtype=tf.float32), - tf.zeros((num_bboxes // 2, num_classes), dtype=tf.float32) - ], - axis=0) - - with self.session(): - outputs = ops.non_max_suppression_3d( - bboxes_3d, - class_scores, - max_boxes_per_class=1000, - nms_iou_threshold=[0.1] * num_classes, - score_threshold=[0.3] * num_classes) - - timings = [] - for _ in range(10): - start = time.time() - _ = self.evaluate(outputs) - end = time.time() - timings.append(end - start) - avg = sum(timings) / len(timings) - print('[{},{},{},{},{}]'.format(num_bboxes, num_classes, min(timings), - avg, max(timings))) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/ops/pairwise_iou_op.cc b/lingvo/tasks/car/ops/pairwise_iou_op.cc deleted file mode 100644 index b9d781c7..00000000 --- a/lingvo/tasks/car/ops/pairwise_iou_op.cc +++ /dev/null @@ -1,72 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include - -#include "lingvo/tasks/car/ops/box_util.h" -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" -#include "tensorflow/core/lib/core/errors.h" - -namespace tensorflow { -namespace lingvo { -namespace { - -class PairwiseIoUOp : public OpKernel { - public: - explicit PairwiseIoUOp(OpKernelConstruction* ctx) : OpKernel(ctx) {} - - void Compute(OpKernelContext* ctx) override { - const Tensor& a = ctx->input(0); - const Tensor& b = ctx->input(1); - OP_REQUIRES(ctx, TensorShapeUtils::IsMatrix(a.shape()), - errors::InvalidArgument("In[0] must be a matrix, but get ", - a.shape().DebugString())); - OP_REQUIRES(ctx, TensorShapeUtils::IsMatrix(b.shape()), - errors::InvalidArgument("In[0] must be a matrix, but get ", - b.shape().DebugString())); - OP_REQUIRES(ctx, 7 == a.dim_size(1), - errors::InvalidArgument("Matrix size-incompatible: In[0]: ", - a.shape().DebugString())); - OP_REQUIRES(ctx, 7 == b.dim_size(1), - errors::InvalidArgument("Matrix size-incompatible: In[1]: ", - b.shape().DebugString())); - - const int n_a = a.dim_size(0); - const int n_b = b.dim_size(0); - - Tensor* iou_a_b = nullptr; - OP_REQUIRES_OK( - ctx, ctx->allocate_output("iou", TensorShape({n_a, n_b}), &iou_a_b)); - - auto t_iou_a_b = iou_a_b->matrix(); - - std::vector box_a = box::ParseBoxesFromTensor(a); - std::vector box_b = box::ParseBoxesFromTensor(b); - for (int i_a = 0; i_a < n_a; ++i_a) { - for (int i_b = 0; i_b < n_b; ++i_b) { - t_iou_a_b(i_a, i_b) = box_a[i_a].IoU(box_b[i_b]); - } - } - } -}; - -REGISTER_KERNEL_BUILDER(Name("PairwiseIou3D").Device(DEVICE_CPU), - PairwiseIoUOp); - -} // namespace -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/tasks/car/ops/point_grid_op.cc b/lingvo/tasks/car/ops/point_grid_op.cc deleted file mode 100644 index 740eba30..00000000 --- a/lingvo/tasks/car/ops/point_grid_op.cc +++ /dev/null @@ -1,192 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include -#include -#include -#include -#include -#include - -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" -#include "tensorflow/core/lib/core/errors.h" - -namespace tensorflow { -namespace lingvo { -namespace { - -const unsigned int kOpRandomSeed = 16; - -int FindBucket(const float val, const float min_val, - const float interval_size) { - return static_cast(std::floor((val - min_val) / interval_size)); -} - -class PointToGridOp : public OpKernel { - public: - explicit PointToGridOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - OP_REQUIRES_OK(ctx, - ctx->GetAttr("num_points_per_cell", &num_points_per_cell_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("x_intervals", &x_intervals_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("y_intervals", &y_intervals_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("z_intervals", &z_intervals_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("x_range", &x_range_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("y_range", &y_range_)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("z_range", &z_range_)); - - OP_REQUIRES(ctx, x_intervals_ > 0 && y_intervals_ > 0 && z_intervals_ > 0, - errors::InvalidArgument("intervals must be positive.")); - OP_REQUIRES( - ctx, - x_range_.size() == 2 && y_range_.size() == 2 && z_range_.size() == 2, - errors::InvalidArgument("intervals must be tuple or list of two.")); - OP_REQUIRES( - ctx, - x_range_[0] < x_range_[1] && y_range_[0] < y_range_[1] && - z_range_[0] < z_range_[1], - errors::InvalidArgument( - "intervals must have lower bounds smaller than upper bounds.")); - } - - void Compute(OpKernelContext* ctx) override { - const Tensor& input_points = ctx->input(0); - OP_REQUIRES(ctx, TensorShapeUtils::IsMatrix(input_points.shape()), - errors::InvalidArgument("points must be a matrix, but get ", - input_points.shape().DebugString())); - OP_REQUIRES(ctx, input_points.dim_size(1) >= 3, - errors::InvalidArgument( - "points must have at least 3 on 2nd dimension.")); - - const float xmin_ = x_range_[0]; - const float xmax_ = x_range_[1]; - const float ymin_ = y_range_[0]; - const float ymax_ = y_range_[1]; - const float zmin_ = z_range_[0]; - const float zmax_ = z_range_[1]; - const float x_interval_size = (xmax_ - xmin_) / x_intervals_; - const float y_interval_size = (ymax_ - ymin_) / y_intervals_; - const float z_interval_size = (zmax_ - zmin_) / z_intervals_; - - // Create a bucket structure where the first dimension is the bucket id and - // the second vector contains the indices of the points that fall into the - // bucket. - // - // We linearize the indexing so that each bucket coordinate maps to a single - // offset. - std::vector> buckets_vec; - - // The number of buckets is the product of all the intervals. - buckets_vec.resize(x_intervals_ * y_intervals_ * z_intervals_); - - const auto& t_input_points = input_points.matrix(); - const int n_input_points = input_points.dim_size(0); - const int n_input_features = input_points.dim_size(1); - - std::vector visit_order(n_input_points); - std::iota(visit_order.begin(), visit_order.end(), 0); - std::shuffle(visit_order.begin(), visit_order.end(), - std::default_random_engine(kOpRandomSeed)); - - for (const int i : visit_order) { - int bucket_x = FindBucket(t_input_points(i, 0), xmin_, x_interval_size); - int bucket_y = FindBucket(t_input_points(i, 1), ymin_, y_interval_size); - int bucket_z = FindBucket(t_input_points(i, 2), zmin_, z_interval_size); - if (bucket_x >= 0 && bucket_x < x_intervals_ && bucket_y >= 0 && - bucket_y < y_intervals_ && bucket_z >= 0 && bucket_z < z_intervals_) { - // Compute the linearized bucket offset. - auto bucket_id = BucketId(bucket_x, bucket_y, bucket_z); - buckets_vec[bucket_id].push_back(i); - } - } - - Tensor* output_points = nullptr; - OP_REQUIRES_OK(ctx, - ctx->allocate_output( - "output_points", - TensorShape({x_intervals_, y_intervals_, z_intervals_, - num_points_per_cell_, n_input_features}), - &output_points)); - Tensor* grid_centers = nullptr; - OP_REQUIRES_OK( - ctx, ctx->allocate_output( - "grid_centers", - TensorShape({x_intervals_, y_intervals_, z_intervals_, 3}), - &grid_centers)); - Tensor* num_points = nullptr; - OP_REQUIRES_OK(ctx, - ctx->allocate_output( - "num_points", - TensorShape({x_intervals_, y_intervals_, z_intervals_}), - &num_points)); - - auto t_output_points = output_points->tensor(); - auto t_grid_centers = grid_centers->tensor(); - auto t_num_points = num_points->tensor(); - - // Padded points will be set to 0. Users can compute the mean by taking - // the sum and dividing by effective number of points. - t_output_points.setConstant(0); - - for (int bx = 0; bx < x_intervals_; ++bx) { - for (int by = 0; by < y_intervals_; ++by) { - for (int bz = 0; bz < z_intervals_; ++bz) { - const float cell_x_center = xmin_ + (bx + 0.5) * x_interval_size; - const float cell_y_center = ymin_ + (by + 0.5) * y_interval_size; - const float cell_z_center = zmin_ + (bz + 0.5) * z_interval_size; - t_grid_centers(bx, by, bz, 0) = cell_x_center; - t_grid_centers(bx, by, bz, 1) = cell_y_center; - t_grid_centers(bx, by, bz, 2) = cell_z_center; - - // Compute the bucket and the number of points that mapped to that - // bucket. - auto bucket_id = BucketId(bx, by, bz); - const std::vector& bucket = buckets_vec[bucket_id]; - const int effective_num = - std::min(num_points_per_cell_, static_cast(bucket.size())); - t_num_points(bx, by, bz) = effective_num; - - // Add the points in bucket to the output. - for (int i = 0; i < effective_num; ++i) { - t_output_points.chip<0>(bx).chip<0>(by).chip<0>(bz).chip<0>(i) = - t_input_points.chip<0>(bucket[i]); - } - } - } - } - } - - private: - // Linearizes the bucket index. - int BucketId(int bucket_x, int bucket_y, int bucket_z) { - return (bucket_z + (bucket_y * z_intervals_) + - (bucket_x * y_intervals_ * z_intervals_)); - } - - int num_points_per_cell_; - int x_intervals_; - int y_intervals_; - int z_intervals_; - std::vector x_range_; - std::vector y_range_; - std::vector z_range_; -}; - -REGISTER_KERNEL_BUILDER(Name("PointToGrid").Device(DEVICE_CPU), PointToGridOp); - -} // namespace -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/tasks/car/ops/point_grid_op_test.py b/lingvo/tasks/car/ops/point_grid_op_test.py deleted file mode 100644 index a77bb61b..00000000 --- a/lingvo/tasks/car/ops/point_grid_op_test.py +++ /dev/null @@ -1,133 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lingvo.tasks.car.ops.point_grid_op.""" - -from lingvo import compat as tf -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.tasks.car import ops -import numpy as np - - -class PointGridOpTest(test_utils.TestCase): - - def _testGrid(self, points, num_points_per_cell, grid_size, grid_range): - points = np.array(points) # p x (3 + d) - with self.session(use_gpu=False): - out_points, grid_centers, num_points = ops.point_to_grid( - points, - num_points_per_cell=num_points_per_cell, - x_intervals=grid_size[0], - y_intervals=grid_size[1], - z_intervals=grid_size[2], - x_range=grid_range[0], - y_range=grid_range[1], - z_range=grid_range[2]) - # Test shape - out_points = py_utils.HasShape( - out_points, - list(grid_size) + [num_points_per_cell, -1]) - grid_centers = py_utils.HasShape(grid_centers, list(grid_size) + [3]) - num_points = py_utils.HasShape(num_points, grid_size) - out_points, grid_centers, num_points = self.evaluate( - [out_points, grid_centers, num_points]) - - # Test points are in the right cell. - boundaries = [] - for axis in range(3): - boundaries += [ - np.linspace( - start=grid_range[axis][0], - stop=grid_range[axis][1], - num=grid_size[axis] + 1, - ) - ] - boundaries = np.stack(np.meshgrid(*boundaries, indexing='ij'), axis=-1) - - # Cast num_points to the points_mask, indicating whether a point - # within a pillar is a real point or not. - points_index = np.arange(num_points_per_cell, dtype=num_points.dtype) - points_index = np.reshape(points_index, [1, 1, 1, num_points_per_cell]) - real_points_mask = np.less(points_index, num_points[..., np.newaxis]) - real_points_mask = np.tile( - np.expand_dims(real_points_mask, axis=-1), [1, 1, 1, 1, 3]) - - # Check whether real points are within the right boundaries. - valid_bound = np.logical_and( - out_points[..., :3] >= boundaries[:-1, :-1, :-1, np.newaxis, ...], - out_points[..., :3] < boundaries[1:, 1:, 1:, np.newaxis, ...]) - padded_points_mask = 1 - real_points_mask - # Check if points are either in valid bounds or padded. - valid = np.logical_or(valid_bound, padded_points_mask) - self.assertTrue(np.all(valid)) - - # Check that points are either zero (padded) or real. - valid = np.logical_or(out_points[..., :3] == 0, real_points_mask) - self.assertTrue(np.all(valid)) - - counts = np.zeros(grid_size) - - def idx(v, l, u, k): - if v < l or v >= u: - return None - return int((v - l) / (u - l) * k) - - for (x, y, z) in points: - ix = idx(x, grid_range[0][0], grid_range[0][1], grid_size[0]) - iy = idx(y, grid_range[1][0], grid_range[1][1], grid_size[1]) - iz = idx(z, grid_range[2][0], grid_range[2][1], grid_size[2]) - if all(i is not None for i in (ix, iy, iz)): - counts[ix, iy, iz] += 1 - counts = np.minimum(counts, num_points_per_cell) - self.assertAllEqual(counts, num_points) - print(np.min(counts), np.max(counts)) - - def testSimpleGrid(self): - points = [[0, 0, 0], [1, 1, 1], [2, 2, 2]] - self._testGrid(points, 1, [2, 2, 2], [[0, 2.1], [0, 2.1], [0, 2.1]]) - - def testRandomPoints(self): - np.random.seed(7483) - points = np.random.random((100, 3)) - grid_size = (5, 4, 3) - grid_range = ((0, 0.3), (-1, 1.4), (0, 1.0)) - num_points_per_cell = 3 - self._testGrid(points, num_points_per_cell, grid_size, grid_range) - - -class PointGridOpBenchmark(tf.test.Benchmark): - - def benchmarkSimpleGrid(self): - points = np.random.uniform(0.0, 1.0, (200000, 3)) - num_points_per_cell = 100 - grid_size = [200, 200, 1] - grid_range = [[0., 1.], [0., 1], [0, 1]] - sess = tf.Session() - out_points, _, _ = ops.point_to_grid( - points, - num_points_per_cell=num_points_per_cell, - x_intervals=grid_size[0], - y_intervals=grid_size[1], - z_intervals=grid_size[2], - x_range=grid_range[0], - y_range=grid_range[1], - z_range=grid_range[2]) - - print(self.run_op_benchmark(sess, out_points, min_iters=1000)) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/ops/ps_utils.cc b/lingvo/tasks/car/ops/ps_utils.cc deleted file mode 100644 index 26a8e64b..00000000 --- a/lingvo/tasks/car/ops/ps_utils.cc +++ /dev/null @@ -1,501 +0,0 @@ -/* Copyright 2019 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ -#include "lingvo/tasks/car/ops/ps_utils.h" - -#include -#include -#include -#include -#include - -#include "absl/synchronization/mutex.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/framework/tensor_shape.h" -#include "tensorflow/core/lib/strings/stringprintf.h" -#include "tensorflow/core/platform/macros.h" - -namespace tensorflow { -namespace lingvo { -namespace car { - -namespace { - -int BucketId(int bucket_x, int bucket_y, int bucket_z, int y_intervals, - int z_intervals) { - return (bucket_z + (bucket_y * z_intervals) + - (bucket_x * y_intervals * z_intervals)); -} - -int FindBucket(const float val, const float min_val, - const float interval_size) { - return static_cast(std::floor((val - min_val) / interval_size)); -} - -// Given a sequence of identifiers (Add), returns k elements uniformly sampled -// from the sequence. -class UniformSampler { - public: - explicit UniformSampler(int k, uint64 seed) : k_(k), rnd_(seed) { - CHECK_GT(k, 0); - ids_.reserve(k_); - } - - void Add(int id, float unused_score) { - ++num_; - if (ids_.size() < k_) { - ids_.push_back(Item{id}); - } else { - const int64 rnd = rnd_() % num_; - // Replace w/ prob k_ / num_ following the reservoir sampling algorithm R. - if (rnd < k_) { - ids_[rnd].id = id; - } - } - } - - struct Item { - int id; - }; - - const std::vector& Get() const { return ids_; } - - void Reset() { - num_ = 0; - ids_.clear(); - } - - private: - const int32 k_; - std::mt19937 rnd_; - int32 num_ = 0; - std::vector ids_; -}; - -// Given a sequence of identifiers with scores (Add), returns k elements with -// the smallest scores. -class TopKSampler { - public: - explicit TopKSampler(int k, uint64 unused_seed) : k_(k) { CHECK_GT(k, 0); } - - void Add(int id, float score) { - if (selected_ && score > items_[k_ - 1].score) { - return; - } - items_.push_back(Item{id, score}); - if (items_.size() > 2 * k_) { - Shrink(); - } - } - - struct Item { - int id; - float score; - }; - - const std::vector& Get() { - if (items_.size() > k_) { - Shrink(); - } - return items_; - } - - void Reset() { - items_.clear(); - selected_ = false; - } - - private: - const int k_ = -1; - bool selected_ = false; - std::vector items_; - struct Cmp { - bool operator()(const Item& a, const Item& b) const { - return a.score < b.score; - } - }; - Cmp cmp_; - - void Shrink() { - std::nth_element(items_.begin(), items_.begin() + k_ - 1, items_.end(), - cmp_); - items_.resize(k_); - selected_ = true; - } -}; - -// Uniform center selection among n points (w/o replacement). -class UniformSelector { - public: - UniformSelector(const std::vector& candidates, uint64 seed) { - for (int i = 0; i < candidates.size(); ++i) { - if (candidates[i]) { - ids_.push_back(i); - } - } - std::shuffle(ids_.begin(), ids_.end(), std::mt19937(seed)); - } - - int Get() { - if (ids_.empty()) { - return -1; - } - int id = ids_.back(); - ids_.resize(ids_.size() - 1); - return id; - } - - void Update(int j, float score) {} // No op. - - private: - std::vector ids_; -}; - -// Select the point farthest from selected centers so far. -class FarthestSelector { - public: - FarthestSelector(std::vector candidates, uint64 seed) - : num_(candidates.size()), - candidates_(std::move(candidates)), - min_dist_sq_(DT_FLOAT, {num_}), - min_dist_sq_t_(min_dist_sq_.vec()), - farthest_(DT_INT32, {}), - farthest_t_(farthest_.scalar()) { - CHECK_LT(kLargeDistA, kLargeDistB); - // Initialize min_dist_sq_ so that the 1st selected point is random. - std::mt19937 gen(seed); - std::uniform_real_distribution dis(kLargeDistA, kLargeDistB); - for (int i = 0; i < num_; ++i) { - if (candidates_[i]) { - ++num_valid_; - min_dist_sq_t_(i) = dis(gen); - } else { - min_dist_sq_t_(i) = -1.0; - } - } - } - - int Get() { - if (num_sampled_ >= num_valid_) { - return -1; - } - farthest_t_ = min_dist_sq_t_.argmax().template cast().eval(); - ++num_sampled_; - return farthest_t_(); - } - - // Updates j-th distance squared to all chosen centers. - void Update(int j, float dist_sq) { - DCHECK_LT(dist_sq, kLargeDistA); - if (candidates_[j] && (dist_sq < min_dist_sq_t_(j))) { - min_dist_sq_t_(j) = dist_sq; - } - } - - private: - const float kLargeDistA = 1e30; - const float kLargeDistB = kLargeDistA * 1.1; - - const int num_; // # of points. - int num_valid_ = 0; // # of points candidates_[i] is True. - int num_sampled_ = 0; // # of Get() has been called. - std::vector candidates_; // i-th can be selected iff candidates_[i]. - Tensor min_dist_sq_; - decltype(min_dist_sq_.vec()) min_dist_sq_t_; - Tensor farthest_; - decltype(farthest_.scalar()) farthest_t_; -}; - -class RNG { - public: - RNG() : rng_(std::random_device("/dev/urandom")()) {} - - uint64 Get() { - absl::MutexLock l(&mu_); - return rng_(); - } - - private: - absl::Mutex mu_; - std::mt19937_64 rng_; -}; - -} // namespace - -uint64 PSUtils::Seed() const { - if (opts_.random_seed != -1) { - return opts_.random_seed; - } - static RNG* rng = new RNG(); - return rng->Get(); -} - -template -T Square(T x) { - return x * x; -} - -template -PSUtils::Result PSUtils::DoSampling(const Tensor& points, - const Tensor& points_padding, - const int32 num_seeded_points) const { - // Points must be of rank 3, and padding must be a matrix. - DCHECK_EQ(points.dims(), 3); - DCHECK_EQ(points_padding.dims(), 2); - // 3D points. - DCHECK_EQ(points.dim_size(2), 3); - DCHECK_EQ(points.dim_size(0), points_padding.dim_size(0)); - DCHECK_EQ(points.dim_size(1), points_padding.dim_size(1)); - - auto points_t = points.tensor(); - auto points_padding_t = points_padding.matrix(); - const int64 batch_size = points.dim_size(0); - const int64 num_points = points.dim_size(1); - - Result result; - result.center = Tensor(DT_INT32, {batch_size, opts_.num_centers}); - result.center_padding = Tensor(DT_FLOAT, {batch_size, opts_.num_centers}); - result.indices = - Tensor(DT_INT32, {batch_size, opts_.num_centers, opts_.num_neighbors}); - result.indices_padding = - Tensor(DT_FLOAT, {batch_size, opts_.num_centers, opts_.num_neighbors}); - - auto center_t = result.center.matrix(); - center_t.setConstant(0); - auto center_padding_t = result.center_padding.matrix(); - auto indices_t = result.indices.tensor(); - indices_t.setConstant(0); - auto padding_t = result.indices_padding.tensor(); - padding_t.setConstant(1.0); - - // Max distance squared as the threshold. - const float threshold = Square(opts_.max_dist); - - // The idea behind the hash lookup is to only do neighbor / distance checks - // for plausibly close neighbors, rather than looking at all points for each - // center. We do this by gridifying the points, and for each center only - // looking at points in nearby grid cells. This a cheap version of a more - // sophisticated algorithm like using a KDTree or RangeTree. - const bool use_hash_lookup = - (opts_.neighbor_search_algorithm == PSUtils::Options::N_HASH); - - for (int cur_batch = 0; cur_batch < batch_size; ++cur_batch) { - std::vector candidates(num_points); - - float xmin = std::numeric_limits::max(); - float ymin = std::numeric_limits::max(); - float zmin = std::numeric_limits::max(); - float xmax = std::numeric_limits::lowest(); - float ymax = std::numeric_limits::lowest(); - float zmax = std::numeric_limits::lowest(); - - for (int i = 0; i < num_points; ++i) { - // The first num_seeded_points are not candidates of the selector, because - // they are always selected. - candidates[i] = - (i >= num_seeded_points && points_padding_t(cur_batch, i) == 0.0) && - (opts_.center_z_min <= points_t(cur_batch, i, 2)) && - (points_t(cur_batch, i, 2) <= opts_.center_z_max); - - // Find min / max points for computing grid buckets. - if (use_hash_lookup && points_padding_t(cur_batch, i) == 0.0) { - xmin = std::min(points_t(cur_batch, i, 0), xmin); - xmax = std::max(points_t(cur_batch, i, 0), xmax); - ymin = std::min(points_t(cur_batch, i, 1), ymin); - ymax = std::max(points_t(cur_batch, i, 1), ymax); - zmin = std::min(points_t(cur_batch, i, 2), zmin); - zmax = std::max(points_t(cur_batch, i, 2), zmax); - } - } - - // Stores a mapping of bucket_id -> list of point indices. The buckets are - // the voxelized breakdown of the 3D space and points fall into these - // voxels. The length of the cube is the max_distance. - std::vector> buckets_vec; - std::vector> buckets_values; - - int x_intervals = 0; - int y_intervals = 0; - int z_intervals = 0; - - if (use_hash_lookup) { - // Adjust boundaries to avoid edge conditions. We use max_dist as a - // conservative estimate. - xmin -= opts_.max_dist; - ymin -= opts_.max_dist; - zmin -= opts_.max_dist; - xmax += opts_.max_dist; - ymax += opts_.max_dist; - zmax += opts_.max_dist; - - x_intervals = std::ceil((xmax - xmin) / opts_.max_dist); - y_intervals = std::ceil((ymax - ymin) / opts_.max_dist); - z_intervals = std::ceil((zmax - zmin) / opts_.max_dist); - - // The number of buckets is the product of all the intervals. - buckets_vec.resize(x_intervals * y_intervals * z_intervals); - for (int i = 0; i < num_points; ++i) { - // Compute which bucket each valid point falls into. - // - // A valid is a non-padded, non-seeded point. - if (points_padding_t(cur_batch, i) == 0.0 && i >= num_seeded_points) { - int bucket_x = - FindBucket(points_t(cur_batch, i, 0), xmin, opts_.max_dist); - int bucket_y = - FindBucket(points_t(cur_batch, i, 1), ymin, opts_.max_dist); - int bucket_z = - FindBucket(points_t(cur_batch, i, 2), zmin, opts_.max_dist); - if (bucket_x >= 0 && bucket_x < x_intervals && bucket_y >= 0 && - bucket_y < y_intervals && bucket_z >= 0 && - bucket_z < z_intervals) { - // Compute the linearized bucket offset. - auto bucket_id = BucketId(bucket_x, bucket_y, bucket_z, y_intervals, - z_intervals); - buckets_vec[bucket_id].push_back(i); - } - } - } - } - - Selector selector(candidates, Seed()); - Sampler sampler(opts_.num_neighbors, Seed()); - - for (int i = 0; i < opts_.num_centers; ++i) { - // Pick a point as i-th center. - int k; - if (i < num_seeded_points) { - k = i; - } else { - // Pick a point as i-th center. - k = selector.Get(); - } - - if (k < 0) { - center_padding_t(cur_batch, i) = 1.0; - continue; - } - center_padding_t(cur_batch, i) = 0.0; - center_t(cur_batch, i) = k; - - // Goes through all *non-seeded* points. If j-th point is within a radius - // of center, adds it to the sampler. - sampler.Reset(); - - std::vector neighbor_idx; - if (use_hash_lookup) { - // For each center, compute the bucket it is in. - int bucket_x = - FindBucket(points_t(cur_batch, k, 0), xmin, opts_.max_dist); - int bucket_y = - FindBucket(points_t(cur_batch, k, 1), ymin, opts_.max_dist); - int bucket_z = - FindBucket(points_t(cur_batch, k, 2), zmin, opts_.max_dist); - - // Iterate over 3x3x3 buckets centered at [bucket_x, bucket_y, bucket_z] - // - // Extract the neighborhood indices from there. - for (int bx = bucket_x - 1; bx <= bucket_x + 1; ++bx) { - if (bx < 0 || bx >= x_intervals) continue; - for (int by = bucket_y - 1; by <= bucket_y + 1; ++by) { - if (by < 0 || by >= y_intervals) continue; - for (int bz = bucket_z - 1; bz <= bucket_z + 1; ++bz) { - if (bz < 0 || bz >= z_intervals) continue; - auto bucket_id = BucketId(bx, by, bz, y_intervals, z_intervals); - auto bucket_indices = buckets_vec[bucket_id]; - neighbor_idx.insert(neighbor_idx.end(), bucket_indices.begin(), - bucket_indices.end()); - } - } - } - - } else { - neighbor_idx.reserve(num_points); - for (int j = num_seeded_points; j < num_points; ++j) { - if (points_padding_t(cur_batch, j) == 0.0) { - neighbor_idx.push_back(j); - } - } - } - - // Iterate over all neighbor indices. - for (int j : neighbor_idx) { - auto ss_xy = - Square(points_t(cur_batch, k, 0) - points_t(cur_batch, j, 0)) + - Square(points_t(cur_batch, k, 1) - points_t(cur_batch, j, 1)); - auto z = points_t(cur_batch, j, 2); - auto ss_xyz = ss_xy + Square(points_t(cur_batch, k, 2) - z); - if (ss_xyz <= threshold) { - sampler.Add(j, ss_xyz); - } - selector.Update(j, ss_xy); - } - auto ids = sampler.Get(); - CHECK_LE(0, ids.size()); - CHECK_LE(ids.size(), opts_.num_neighbors); - for (int j = 0; j < ids.size(); ++j) { - indices_t(cur_batch, i, j) = ids[j].id; - padding_t(cur_batch, i, j) = 0.0f; - } - } - } - - return result; -} - -string PSUtils::Options::DebugString() const { - // clang-format off - return strings::Printf( - "cmethod/#centers/zmin/zmax/nmethod/#neighbors/maxdist/seed " - "%s/%d/%.3f/%.3f/%s/%d/%.3f/%d", - cmethod == C_UNIFORM ? "uniform" : "farthest", - num_centers, - center_z_min, - center_z_max, - nmethod == N_UNIFORM ? "uniform" : "closest", num_neighbors, - max_dist, - random_seed); - // clang-format on -} - -PSUtils::Result PSUtils::Sample(const Tensor& points, - const Tensor& points_padding, - const int32 num_seeded_points) const { - if (opts_.cmethod == Options::C_UNIFORM && - opts_.nmethod == Options::N_UNIFORM) { - CHECK_EQ(num_seeded_points, 0) - << "Seeding only supported for Farthest Point Sampling"; - return DoSampling(points, points_padding, - num_seeded_points); - } - if (opts_.cmethod == Options::C_UNIFORM && - opts_.nmethod == Options::N_CLOSEST) { - CHECK_EQ(num_seeded_points, 0) - << "Seeding only supported for Farthest Point Sampling"; - return DoSampling(points, points_padding, - num_seeded_points); - } - if (opts_.cmethod == Options::C_FARTHEST && - opts_.nmethod == Options::N_UNIFORM) { - return DoSampling(points, points_padding, - num_seeded_points); - } - CHECK_EQ(opts_.cmethod, Options::C_FARTHEST); - CHECK_EQ(opts_.nmethod, Options::N_CLOSEST); - return DoSampling(points, points_padding, - num_seeded_points); -} - -} // namespace car -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/tasks/car/ops/ps_utils.h b/lingvo/tasks/car/ops/ps_utils.h deleted file mode 100644 index e3ed6a8a..00000000 --- a/lingvo/tasks/car/ops/ps_utils.h +++ /dev/null @@ -1,103 +0,0 @@ -/* Copyright 2019 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ -#ifndef THIRD_PARTY_PY_LINGVO_TASKS_CAR_OPS_PS_UTILS_H_ -#define THIRD_PARTY_PY_LINGVO_TASKS_CAR_OPS_PS_UTILS_H_ - -#include - -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/platform/types.h" - -namespace tensorflow { -namespace lingvo { -namespace car { - -// Point cloud sampling utilities. -class PSUtils { - public: - struct Options { - // Center selection method. - enum CenterSelectionMethod { C_UNIFORM, C_FARTHEST }; - CenterSelectionMethod cmethod = C_FARTHEST; - - // The number of centers to sample. - int num_centers = 128; - - // Points with z outside of the following range are not considered for - // center selection. - float center_z_min = std::numeric_limits::lowest(); - float center_z_max = std::numeric_limits::max(); - - // Center selection method. - enum NeighborSelectionMethod { N_UNIFORM, N_CLOSEST }; - NeighborSelectionMethod nmethod = N_UNIFORM; - - // For each center, sample this many points within the neighorhood. - int num_neighbors = 1024; - - // Points with L2 distances in 3D from a center larger than this threshold - // is not considered to be in the neighborhood. - float max_dist = std::numeric_limits::max(); - - // The random seed. - int random_seed = -1; - - enum NeighborSearchAlgorithm { N_AUTO, N_HASH }; - NeighborSearchAlgorithm neighbor_search_algorithm = N_AUTO; - - string DebugString() const; - }; - - explicit PSUtils(const Options& opts) : opts_(opts) {} - - // Samples centers within 'points' (3D) and creates neighborhood for each - // center. - // - // If num_seeded_points is > 0, then the first num_seeded_points are used as - // seeds for the center selection component of sampling, but are ignored for - // neighborhood selection. - struct Result { - // [num_centers]. Indices of the center points. - Tensor center; - - // [num_centers]. 0/1 paddings indicating that 0 means i-th center is a real - // sampled center point, while 1 means otherwise. - Tensor center_padding; - - // [num_centers, num_neighbors]. Indices of neighbors. - Tensor indices; - - // indices[i, j] is a real point if and only if indices_padding[i, j] is - // 0. Otherwise, indices_padding[i, j] is 1.0. - Tensor indices_padding; - }; - Result Sample(const Tensor& points, const Tensor& points_padding, - const int32 num_seeded_points) const; - - private: - const Options opts_; - - uint64 Seed() const; - - template - Result DoSampling(const Tensor& points, const Tensor& points_padding, - const int32 num_seeded_points) const; -}; - -} // namespace car -} // namespace lingvo -} // namespace tensorflow - -#endif // THIRD_PARTY_PY_LINGVO_TASKS_CAR_OPS_PS_UTILS_H_ diff --git a/lingvo/tasks/car/ops/ps_utils_test.cc b/lingvo/tasks/car/ops/ps_utils_test.cc deleted file mode 100644 index c05f1509..00000000 --- a/lingvo/tasks/car/ops/ps_utils_test.cc +++ /dev/null @@ -1,371 +0,0 @@ -/* Copyright 2019 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ -#include "lingvo/tasks/car/ops/ps_utils.h" - -#include -#include - -#include -#include "tensorflow/core/lib/strings/stringprintf.h" -#include "tensorflow/core/platform/test_benchmark.h" - -namespace tensorflow { -namespace lingvo { -namespace car { - -struct Point { - float x; - float y; -}; - -// Generates batch_size sets of points, the k-th example in the batch has n-k -// centers (i-th center is on (i, i)) and m points near each center. -void GeneratePoints(int batch_size, int n, int m, Tensor* points, - Tensor* points_padding) { - std::mt19937 rng(39183); - *points = Tensor(DT_FLOAT, {batch_size, n * m, 3}); - *points_padding = Tensor(DT_FLOAT, {batch_size, n * m}); - auto points_t = points->tensor(); - auto points_padding_t = points_padding->matrix(); - points_padding_t.setConstant(0.0); - - for (int cur_batch = 0; cur_batch < batch_size; ++cur_batch) { - std::vector points; - for (int i = 0; i < n - cur_batch; ++i) { - for (int j = 0; j < m; ++j) { - float v = i + j / 1000.0; - points.push_back(Point{v, v}); - } - } - std::shuffle(points.begin(), points.end(), rng); - for (int i = 0; i < (n - cur_batch) * m; ++i) { - points_t(cur_batch, i, 0) = points[i].x; - points_t(cur_batch, i, 1) = points[i].y; - points_t(cur_batch, i, 2) = 0; - } - for (int i = (n - cur_batch) * m; i < n * m; ++i) { - points_padding_t(cur_batch, i) = 1.0; - } - } -} - -void Log(const Tensor& points, const PSUtils::Result& result) { - const int batch_size = result.center.dim_size(0); - const int n = result.center.dim_size(1); - CHECK_EQ(result.indices.dim_size(0), batch_size); - CHECK_EQ(result.indices.dim_size(1), n); - - const int m = result.indices.dim_size(2); - CHECK_EQ(result.indices_padding.dim_size(1), n); - CHECK_EQ(result.indices_padding.dim_size(2), m); - - auto points_t = points.tensor(); - auto center_t = result.center.matrix(); - auto center_padding_t = result.center_padding.matrix(); - auto indices_t = result.indices.tensor(); - auto indices_padding_t = result.indices_padding.tensor(); - for (int cur_batch = 0; cur_batch < batch_size; ++cur_batch) { - fprintf(stdout, "batch id %d\n", cur_batch); - for (int i = 0; i < n; ++i) { - CHECK_EQ(0.0, center_padding_t(cur_batch, i)); - fprintf(stdout, - "(%5.3f %5.3f): ", points_t(cur_batch, center_t(cur_batch, i), 0), - points_t(cur_batch, center_t(cur_batch, i), 1)); - for (int j = 0; j < m; ++j) { - fprintf(stdout, "(%5.3f %5.3f)/%1.0f, ", - points_t(cur_batch, indices_t(cur_batch, i, j), 0), - points_t(cur_batch, indices_t(cur_batch, i, j), 1), - indices_padding_t(cur_batch, i, j)); - } - fprintf(stdout, "\n"); - } - } -} - -std::vector GetCenters(const Tensor& points, - const PSUtils::Result& result) { - const int batch_size = result.center.dim_size(0); - const int n = result.center.dim_size(1); - CHECK_EQ(result.indices.dim_size(0), batch_size); - CHECK_EQ(result.indices.dim_size(1), n); - - const int m = result.indices.dim_size(2); - CHECK_EQ(result.indices_padding.dim_size(1), n); - CHECK_EQ(result.indices_padding.dim_size(2), m); - - auto points_t = points.tensor(); - auto center_t = result.center.matrix(); - auto center_padding_t = result.center_padding.matrix(); - auto indices_t = result.indices.tensor(); - std::vector centers; - for (int cur_batch = 0; cur_batch < batch_size; ++cur_batch) { - for (int i = 0; i < n; ++i) { - CHECK_EQ(0.0, center_padding_t(cur_batch, i)); - const int center = - static_cast(points_t(cur_batch, center_t(cur_batch, i), 0)); - for (int j = 0; j < m; ++j) { - CHECK_EQ(points_t(cur_batch, indices_t(cur_batch, i, j), 0), - points_t(cur_batch, indices_t(cur_batch, i, j), 1)); - CHECK_EQ(center, static_cast(points_t( - cur_batch, indices_t(cur_batch, i, j), 0))); - } - centers.push_back(center); - } - } - return centers; -} - -#if defined(PLATFORM_GOOGLE) -TEST(PSUtilsTest, Uniform_Uniform) { - PSUtils::Options opts; - opts.cmethod = PSUtils::Options::C_UNIFORM; - opts.nmethod = PSUtils::Options::N_UNIFORM; - opts.num_centers = 8; - opts.num_neighbors = 16; - opts.max_dist = 1.0; - opts.random_seed = 12345; - PSUtils fu(opts); - Tensor points; - Tensor points_padding; - GeneratePoints(3, 8, 100, &points, &points_padding); - auto ret = fu.Sample(points, points_padding, 0); - Log(points, ret); - // Generated points will correspond to clusters like: - // (0, 0), (0, 0.001), (0, 0.002), ... - // (1, 1), (1, 1.001), (1, 1.002), ... - // ... - // (7, 7), (7, 7.001), (7, 7.002), ... - // - // GetCenters returns the first coordinate of each point, and hence, we - // expect them to repeat. - // Some clusters are sampled more than once. - EXPECT_EQ(GetCenters(points, ret), - std::vector({1, 0, 2, 4, 0, 3, 1, 3, // 1st example. - 1, 2, 5, 3, 5, 2, 0, 0, // 2nd example. - 1, 2, 5, 2, 5, 4, 5, 4})); // 3rd example. -} - -TEST(PSUtilsTest, Uniform_Closest) { - PSUtils::Options opts; - opts.cmethod = PSUtils::Options::C_UNIFORM; - opts.nmethod = PSUtils::Options::N_CLOSEST; - opts.num_centers = 8; - opts.num_neighbors = 16; - opts.max_dist = 1.0; - opts.random_seed = 12345; - PSUtils fu(opts); - Tensor points; - Tensor points_padding; - GeneratePoints(3, 8, 100, &points, &points_padding); - auto ret = fu.Sample(points, points_padding, 0); - Log(points, ret); - // Some clusters are sampled more than once. - EXPECT_EQ(GetCenters(points, ret), - std::vector({1, 0, 2, 4, 0, 3, 1, 3, // 1st example. - 1, 2, 5, 3, 5, 2, 0, 0, // 2nd example. - 1, 2, 5, 2, 5, 4, 5, 4})); // 3rd example. -} - -TEST(PSUtilsTest, Farthest_Uniform) { - PSUtils::Options opts; - opts.cmethod = PSUtils::Options::C_FARTHEST; - opts.nmethod = PSUtils::Options::N_UNIFORM; - opts.num_centers = 8; - opts.num_neighbors = 16; - opts.max_dist = 1.0; - opts.random_seed = 12345; - PSUtils fu(opts); - Tensor points; - Tensor points_padding; - GeneratePoints(3, 8, 100, &points, &points_padding); - auto ret = fu.Sample(points, points_padding, 0); - Log(points, ret); - // Generated points will correspond to clusters like: - // (0, 0), (0, 0.001), (0, 0.002), ... - // (1, 1), (1, 1.001), (1, 1.002), ... - // ... - // (7, 7), (7, 7.001), (7, 7.002), ... - // - // GetCenters returns the first coordinate of each point. With farthest point - // sampling, for an example without any paddings, we expect the samples to - // have all different first coordinates, and all 8 clusters should be covered. - EXPECT_EQ( - GetCenters(points, ret), - std::vector( - {3, 7, 0, 5, 1, 6, 4, 2, // 1st example. - 6, 0, 3, 1, 4, 5, 2, 0, // 2nd example, last one is a duplicate. - 3, 0, 5, 1, 4, 2, 0, 1})); // 3rd example, last two are duplicates. -} - -TEST(PSUtilsTest, Farthest_Uniform_Hash) { - PSUtils::Options opts; - opts.cmethod = PSUtils::Options::C_FARTHEST; - opts.nmethod = PSUtils::Options::N_UNIFORM; - opts.neighbor_search_algorithm = PSUtils::Options::N_HASH; - opts.num_centers = 8; - opts.num_neighbors = 16; - opts.max_dist = 1.0; - opts.random_seed = 12345; - PSUtils fu(opts); - Tensor points; - Tensor points_padding; - GeneratePoints(3, 8, 100, &points, &points_padding); - auto ret = fu.Sample(points, points_padding, 0); - Log(points, ret); - // Like above, but notice that the order is different since the algorithm is - // changed and uniform provides no ordering guarantees. - EXPECT_EQ( - GetCenters(points, ret), - std::vector( - {3, 6, 1, 4, 7, 0, 2, 5, // 1st example. - 6, 4, 0, 2, 3, 1, 5, 4, // 2nd example, last one is a duplicate. - 3, 5, 1, 0, 4, 2, 0, 3})); // 3rd example, last two are duplicates. -} - -TEST(PSUtilsTest, Farthest_Closest) { - PSUtils::Options opts; - opts.cmethod = PSUtils::Options::C_FARTHEST; - opts.nmethod = PSUtils::Options::N_CLOSEST; - opts.num_centers = 8; - opts.num_neighbors = 16; - opts.max_dist = 10.0; - opts.random_seed = 12345; - PSUtils fu(opts); - Tensor points; - Tensor points_padding; - GeneratePoints(3, 8, 100, &points, &points_padding); - auto ret = fu.Sample(points, points_padding, 0); - Log(points, ret); - // All 8 clusters are covered. - EXPECT_EQ( - GetCenters(points, ret), - std::vector( - {3, 7, 0, 5, 1, 6, 4, 2, // 1st example. - 6, 0, 3, 1, 4, 5, 2, 0, // 2nd example, last one is a duplicate. - 3, 0, 5, 1, 4, 2, 0, 1})); // 3rd example, last two are duplicates. -} - -TEST(PSUtilsTest, Farthest_Closest_Hash) { - PSUtils::Options opts; - opts.cmethod = PSUtils::Options::C_FARTHEST; - opts.nmethod = PSUtils::Options::N_CLOSEST; - opts.neighbor_search_algorithm = PSUtils::Options::N_HASH; - opts.num_centers = 8; - opts.num_neighbors = 16; - opts.max_dist = 10.0; - opts.random_seed = 12345; - PSUtils fu(opts); - Tensor points; - Tensor points_padding; - GeneratePoints(3, 8, 100, &points, &points_padding); - auto ret = fu.Sample(points, points_padding, 0); - Log(points, ret); - // All 8 clusters are covered. - EXPECT_EQ( - GetCenters(points, ret), - std::vector( - {3, 7, 0, 5, 1, 6, 4, 2, // 1st example. - 6, 0, 3, 1, 4, 5, 2, 0, // 2nd example, last one is a duplicate. - 3, 0, 5, 1, 4, 2, 0, 1})); // 3rd example, last two are duplicates. -} - -TEST(PSUtilsTest, TestSeeded) { - PSUtils::Options opts; - opts.cmethod = PSUtils::Options::C_FARTHEST; - opts.nmethod = PSUtils::Options::N_CLOSEST; - opts.num_centers = 4; - opts.num_neighbors = 2; - opts.max_dist = 10.0; - opts.random_seed = 12345; - PSUtils fu(opts); - - // Six points along a line: (0, 1, 2, 3, 4, 5). - Tensor points(DT_FLOAT, {1, 6, 3}); - Tensor points_padding(DT_FLOAT, {1, 6}); - auto points_t = points.tensor(); - auto points_padding_t = points_padding.matrix(); - points_padding_t.setConstant(0.0); - for (int i = 0; i < 6; ++i) { - points_t(0, i, 0) = i; - points_t(0, i, 1) = 0; - points_t(0, i, 2) = 0; - } - - // Choose the first two points as the seed (0, and 1). - auto ret = fu.Sample(points, points_padding, 2); - Log(points, ret); - - // The first two are always chosen, and then the next one is 5, since it is - // the farthest from 0 and 1. The remaining is the middle point between 1 and - // 5. - auto center_t = ret.center.matrix(); - EXPECT_EQ(points_t(0, center_t(0, 0), 0), 0.); - EXPECT_EQ(points_t(0, center_t(0, 1), 0), 1.); - EXPECT_EQ(points_t(0, center_t(0, 2), 0), 5.); - EXPECT_EQ(points_t(0, center_t(0, 3), 0), 3.); - - // Seeded points are not neighbors; the closest neighbor of the seeded - // point is 2., not 1. - auto indices_t = ret.indices.tensor(); - EXPECT_EQ(points_t(0, indices_t(0, 0, 0), 0), 2.); -} - -void BenchmarkFarthestPoint(int iters, int num_centers, int num_neighbors, - PSUtils::Options opts) { - testing::StopTiming(); - testing::SetLabel(strings::Printf("#Centers=%4d #Neighbors=%4d", num_centers, - num_neighbors)); - PSUtils fu(opts); - Tensor points; - Tensor points_padding; - GeneratePoints(1, 1000, 100, &points, &points_padding); - testing::StartTiming(); - for (int i = 0; i < iters; ++i) { - auto ret = fu.Sample(points, points_padding, 0); - } -} - -void BM_Farthest(int iters, int num_centers, int num_neighbors) { - PSUtils::Options opts; - opts.cmethod = PSUtils::Options::C_FARTHEST; - opts.nmethod = PSUtils::Options::N_UNIFORM; - opts.num_centers = num_centers; - opts.num_neighbors = num_neighbors; - opts.max_dist = 1.0; - opts.random_seed = -1; - BenchmarkFarthestPoint(iters, num_centers, num_neighbors, opts); -} - -BENCHMARK(BM_Farthest)->RangePair(1, 1024, 1, 1024); - -void BM_FarthestHash(int iters, int num_centers, int num_neighbors) { - PSUtils::Options opts; - opts.cmethod = PSUtils::Options::C_FARTHEST; - opts.nmethod = PSUtils::Options::N_UNIFORM; - opts.neighbor_search_algorithm = PSUtils::Options::N_HASH; - opts.num_centers = num_centers; - opts.num_neighbors = num_neighbors; - opts.max_dist = 1.0; - opts.random_seed = -1; - BenchmarkFarthestPoint(iters, num_centers, num_neighbors, opts); -} - -BENCHMARK(BM_FarthestHash)->RangePair(1, 1024, 1, 1024); - -#endif - -} // namespace car -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/tasks/car/ops/sampling_ops.cc b/lingvo/tasks/car/ops/sampling_ops.cc deleted file mode 100644 index d53a032a..00000000 --- a/lingvo/tasks/car/ops/sampling_ops.cc +++ /dev/null @@ -1,83 +0,0 @@ -/* Copyright 2019 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ -#include "lingvo/tasks/car/ops/ps_utils.h" -#include "tensorflow/core/framework/op_kernel.h" -#include "tensorflow/core/framework/tensor.h" -#include "tensorflow/core/lib/core/errors.h" - -namespace tensorflow { -namespace lingvo { -namespace car { - -class SamplePointsOp : public OpKernel { - public: - explicit SamplePointsOp(OpKernelConstruction* ctx) : OpKernel(ctx) { - string method; - OP_REQUIRES_OK(ctx, ctx->GetAttr("center_selector", &method)); - OP_REQUIRES(ctx, method == "uniform" || method == "farthest", - errors::InvalidArgument(method, " is not valid.")); - if (method == "uniform") { - opts_.cmethod = PSUtils::Options::C_UNIFORM; - } else { - CHECK_EQ(method, "farthest"); - opts_.cmethod = PSUtils::Options::C_FARTHEST; - } - OP_REQUIRES_OK(ctx, ctx->GetAttr("neighbor_sampler", &method)); - OP_REQUIRES(ctx, method == "uniform" || method == "closest", - errors::InvalidArgument(method, " is not valid.")); - if (method == "uniform") { - opts_.nmethod = PSUtils::Options::N_UNIFORM; - } else { - CHECK_EQ(method, "closest"); - opts_.nmethod = PSUtils::Options::N_CLOSEST; - } - - OP_REQUIRES_OK(ctx, ctx->GetAttr("neighbor_algorithm", &method)); - OP_REQUIRES( - ctx, method == "auto" || method == "hash", - errors::InvalidArgument(method, " is not a valid neighbor algorithm.")); - if (method == "hash") { - opts_.neighbor_search_algorithm = PSUtils::Options::N_HASH; - } - - LOG(INFO) << "Sampling options: " << opts_.DebugString(); - OP_REQUIRES_OK(ctx, ctx->GetAttr("num_centers", &opts_.num_centers)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("center_z_min", &opts_.center_z_min)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("center_z_max", &opts_.center_z_max)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("num_neighbors", &opts_.num_neighbors)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("max_distance", &opts_.max_dist)); - OP_REQUIRES_OK(ctx, ctx->GetAttr("random_seed", &opts_.random_seed)); - } - - void Compute(OpKernelContext* ctx) override { - PSUtils ps(opts_); - auto ret = ps.Sample(ctx->input(0), ctx->input(1), - ctx->input(2).scalar()()); - ctx->set_output(0, ret.center); - ctx->set_output(1, ret.center_padding); - ctx->set_output(2, ret.indices); - ctx->set_output(3, ret.indices_padding); - } - - private: - PSUtils::Options opts_; -}; - -REGISTER_KERNEL_BUILDER(Name("SamplePoints").Device(DEVICE_CPU), - SamplePointsOp); - -} // namespace car -} // namespace lingvo -} // namespace tensorflow diff --git a/lingvo/tasks/car/ops/sampling_ops_test.py b/lingvo/tasks/car/ops/sampling_ops_test.py deleted file mode 100644 index e66161f4..00000000 --- a/lingvo/tasks/car/ops/sampling_ops_test.py +++ /dev/null @@ -1,160 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lingvo.tasks.car.ops.sampling_ops.""" - -from absl.testing import parameterized -from lingvo import compat as tf -from lingvo.core import test_utils -from lingvo.tasks.car import ops - -import numpy as np - - -class SamplingOpsTest(parameterized.TestCase, test_utils.TestCase): - - @parameterized.named_parameters([ - ('uniform_uniform', 'uniform', 'uniform', 'auto'), - ('uniform_closest', 'uniform', 'closest', 'auto'), - ('farthest_uniform', 'farthest', 'uniform', 'auto'), - ('farthest_closest', 'farthest', 'closest', 'auto'), - ('farthest_closest_hash', 'farthest', 'closest', 'hash'), - ]) - def testBasic(self, cmethod, nmethod, nalgo): - b, n, m, k = 3, 10000, 128, 128 - g = tf.Graph() - with g.as_default(): - points = tf.random.uniform(shape=(b, n, 3)) - points_padding = tf.zeros(shape=(b, n)) - center, center_padding, indices, indices_padding = ops.sample_points( - points=points, - points_padding=points_padding, - num_seeded_points=0, - center_selector=cmethod, - neighbor_sampler=nmethod, - neighbor_algorithm=nalgo, - num_centers=m, - center_z_min=-np.inf, - center_z_max=np.inf, - num_neighbors=k, - max_distance=0.25) - - # Ensure shapes are known at graph construction. - self.assertListEqual(center.shape.as_list(), [b, m]) - self.assertListEqual(center_padding.shape.as_list(), [b, m]) - self.assertListEqual(indices.shape.as_list(), [b, m, k]) - self.assertListEqual(indices_padding.shape.as_list(), [b, m, k]) - - with self.session(graph=g): - cp, c, i, p = self.evaluate( - [center_padding, center, indices, indices_padding]) - - # Very basic validity checking. - self.assertEqual(cp.shape, (b, m)) - self.assertTrue(np.all(cp == 0.0)) - self.assertEqual(c.shape, (b, m)) - self.assertTrue(np.all(np.logical_and(0 <= c, c < n))) - self.assertEqual(i.shape, (b, m, k)) - self.assertTrue(np.all(np.logical_and(0 <= i, i < n))) - self.assertEqual(p.shape, (b, m, k)) - self.assertTrue(np.all(np.logical_or(0. == p, 1. == p))) - - @parameterized.named_parameters([ - ('uniform_uniform', 'uniform', 'uniform', 'auto'), - ('uniform_closest', 'uniform', 'closest', 'auto'), - ('farthest_uniform', 'farthest', 'uniform', 'auto'), - ('farthest_closest', 'farthest', 'closest', 'auto'), - ('farthest_closest_hash', 'farthest', 'closest', 'hash'), - ]) - def testZFilter(self, cmethod, nmethod, nalgo): - b, n, m, k = 1, 10000, 128, 128 - g = tf.Graph() - with g.as_default(): - points = tf.random.uniform(shape=(b, n, 3)) - points_padding = tf.zeros(shape=(b, n)) - center, center_padding, indices, indices_padding = ops.sample_points( - points=points, - points_padding=points_padding, - num_seeded_points=0, - center_selector=cmethod, - neighbor_sampler=nmethod, - num_centers=m, - center_z_min=0.25, - center_z_max=0.75, - num_neighbors=k, - max_distance=0.25) - - # Ensure shapes are known at graph construction. - self.assertListEqual(center.shape.as_list(), [b, m]) - self.assertListEqual(center_padding.shape.as_list(), [b, m]) - self.assertListEqual(indices.shape.as_list(), [b, m, k]) - self.assertListEqual(indices_padding.shape.as_list(), [b, m, k]) - - with self.session(graph=g): - c1, p1 = self.evaluate([center, points]) - c2, p2 = self.evaluate([center, points]) - - # With extremely high probability, sampling centers twice should be - # different. - self.assertGreater(np.setdiff1d(c1, c2).size, 0) - - # Centers should be filtered by z range. - self.assertTrue((0.25 <= p1[0, c1[0], 2]).all()) - self.assertTrue((p1[0, c1[0], 2] <= 0.75).all()) - self.assertTrue((0.25 <= p2[0, c2[0], 2]).all()) - self.assertTrue((p2[0, c2[0], 2] <= 0.75).all()) - - @parameterized.named_parameters([ - ('uniform_uniform', 'uniform', 'uniform'), - ('uniform_closest', 'uniform', 'closest'), - ('farthest_uniform', 'farthest', 'uniform'), - ('farthest_closest', 'farthest', 'closest'), - ]) - def testSampleFewerCentersThanPoints(self, cmethod, nmethod): - b, n, m, k = 1, 100, 128, 8 - g = tf.Graph() - with g.as_default(): - points = tf.random.uniform(shape=(b, n, 3)) - points_padding = tf.zeros(shape=(b, n)) - center, center_padding, indices, indices_padding = ops.sample_points( - points=points, - points_padding=points_padding, - num_seeded_points=0, - center_selector=cmethod, - neighbor_sampler=nmethod, - num_centers=m, - center_z_min=-np.inf, - center_z_max=np.inf, - num_neighbors=k, - max_distance=0.25) - - # Ensure shapes are known at graph construction. - self.assertListEqual(center.shape.as_list(), [b, m]) - self.assertListEqual(center_padding.shape.as_list(), [b, m]) - self.assertListEqual(indices.shape.as_list(), [b, m, k]) - self.assertListEqual(indices_padding.shape.as_list(), [b, m, k]) - - with self.session(graph=g): - p, c = self.evaluate([center_padding, center]) - - self.assertAllEqual(p[0, :n], np.zeros([n])) - self.assertAllEqual(p[0, n:], np.ones([m - n])) - tf.logging.info('c[:n]=%s', c[0, :n]) - self.assertAllEqual(np.sort(c[0, :n]), np.arange(n)) - self.assertAllEqual(c[0, n:], np.zeros([m - n])) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/params/BUILD b/lingvo/tasks/car/params/BUILD deleted file mode 100644 index c62c60c0..00000000 --- a/lingvo/tasks/car/params/BUILD +++ /dev/null @@ -1,57 +0,0 @@ -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -py_library( - name = "kitti", - srcs = ["kitti.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo:model_registry", - "//lingvo/core:base_model_params", - "//lingvo/core:cluster_factory", - "//lingvo/core:datasource", - "//lingvo/core:optimizer", - "//lingvo/core:py_utils", - "//lingvo/tasks/car:input_preprocessors", - "//lingvo/tasks/car:kitti_input_generator", - "//lingvo/tasks/car:lr_util", - "//lingvo/tasks/car:starnet", - # Implicit numpy dependency. - ], -) - -py_library( - name = "waymo", - srcs = ["waymo.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:model_registry", - "//lingvo/core:base_model_params", - "//lingvo/core:cluster_factory", - "//lingvo/core:hyperparams", - "//lingvo/core:optimizer", - "//lingvo/core:py_utils", - "//lingvo/tasks/car:input_preprocessors", - "//lingvo/tasks/car:lr_util", - "//lingvo/tasks/car:starnet", - "//lingvo/tasks/car/waymo:waymo_decoder", - "//lingvo/tasks/car/waymo:waymo_metadata", - "//lingvo/tasks/car/waymo:waymo_open_input_generator", - # Implicit numpy dependency. - ], -) - -py_library( - name = "params", - srcs = ["params.py"], - srcs_version = "PY3", - tags = ["keep_dep"], - deps = [ - ":kitti", - ":waymo", - ], -) diff --git a/lingvo/tasks/car/params/kitti.py b/lingvo/tasks/car/params/kitti.py deleted file mode 100644 index 0d229760..00000000 --- a/lingvo/tasks/car/params/kitti.py +++ /dev/null @@ -1,628 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Train models on KITTI data.""" - -import os - -from lingvo import compat as tf -from lingvo import model_registry -from lingvo.core import base_model_params -from lingvo.core import cluster_factory -from lingvo.core import datasource -from lingvo.core import optimizer -from lingvo.core import py_utils -from lingvo.tasks.car import input_preprocessors -from lingvo.tasks.car import kitti_input_generator -from lingvo.tasks.car import lr_util -from lingvo.tasks.car import starnet -import numpy as np - - -# Set $KITTI_DIR to the base path of where all the KITTI files can be found. -# -# E.g., 'gs://your-bucket/kitti/3d' -_KITTI_BASE = os.environ.get('KITTI_DIR', 'FILL-ME-IN') - - -# Specifications for the different dataset splits. -def KITTITrainSpec(params): - p = params.Copy() - p.file_datasource.file_pattern = ( - 'kitti_object_3dop_train.tfrecord-*-of-00100') - p.num_samples = 3712 - return p - - -def KITTIValSpec(params): - p = params.Copy() - p.file_datasource.file_pattern = ('kitti_object_3dop_val.tfrecord-*-of-00100') - p.num_samples = 3769 - return p - - -def KITTITestSpec(params): - p = params.Copy() - p.file_datasource.file_pattern = ('kitti_object_test.tfrecord-*-of-00100') - p.num_samples = 7518 - return p - - -class KITTITrain(kitti_input_generator.KITTILaser): - """KITTI train set with raw laser data.""" - - @classmethod - def Params(cls): - """Defaults params.""" - p = super().Params() - return KITTITrainSpec(p) - - -class KITTIValidation(kitti_input_generator.KITTILaser): - """KITTI validation set with raw laser data.""" - - @classmethod - def Params(cls): - """Defaults params.""" - p = super().Params() - return KITTIValSpec(p) - - -class KITTITest(kitti_input_generator.KITTILaser): - """KITTI test set with raw laser data.""" - - @classmethod - def Params(cls): - p = super().Params() - return KITTITestSpec(p) - - -class KITTIGridTrain(kitti_input_generator.KITTIGrid): - """KITTI train set with grid laser data.""" - - @classmethod - def Params(cls): - p = super().Params() - return KITTITrainSpec(p) - - -class KITTIGridValidation(kitti_input_generator.KITTIGrid): - """KITTI validation set with grid laser data.""" - - @classmethod - def Params(cls): - p = super().Params() - return KITTIValSpec(p) - - -class KITTIGridTest(kitti_input_generator.KITTIGrid): - """KITTI validation set with grid laser data.""" - - @classmethod - def Params(cls): - p = super().Params() - return KITTITestSpec(p) - - -class KITTISparseLaserTrain(kitti_input_generator.KITTISparseLaser): - """KITTI train set with sparse laser data.""" - - @classmethod - def Params(cls): - p = super().Params() - return KITTITrainSpec(p) - - -class KITTISparseLaserValidation(kitti_input_generator.KITTISparseLaser): - """KITTI validation set with sparse laser data.""" - - @classmethod - def Params(cls): - p = super().Params() - return KITTIValSpec(p) - - -class KITTISparseLaserTest(kitti_input_generator.KITTISparseLaser): - """KITTI test set with sparse laser data.""" - - @classmethod - def Params(cls): - p = super().Params() - return KITTITestSpec(p) - - -def _MaybeRemove(values, key): - """Remove the entry 'key' from 'values' if present.""" - if key in values: - values.remove(key) - - -def AddLaserAndCamera(params): - """Adds laser and camera extractors.""" - cluster = cluster_factory.Current() - job = cluster.job - if job != 'decoder': - return params - - extractor_params = list(dict(params.extractors.IterParams()).values()) - extractor_classes = [p.cls for p in extractor_params] - - # Add images if not present. - if kitti_input_generator.KITTIImageExtractor not in extractor_classes: - params.extractors.Define('images', - kitti_input_generator.KITTIImageExtractor.Params(), - '') - - # Add raw lasers if not present. - if kitti_input_generator.KITTILaserExtractor not in extractor_classes: - labels = None - for p in extractor_params: - if p.cls == kitti_input_generator.KITTILabelExtractor: - labels = p - if labels is None: - labels = kitti_input_generator.KITTILabelExtractor.Params() - params.extractors.Define( - 'lasers', kitti_input_generator.KITTILaserExtractor.Params(labels), '') - - return params - - -################################################################################ -# StarNet -################################################################################ -@model_registry.RegisterSingleTaskModel -class StarNetCarsBase(base_model_params.SingleTaskModelParams): - """StarNet model for cars.""" - RUN_LOCALLY = False - NUM_ANCHOR_BBOX_OFFSETS = 25 - NUM_ANCHOR_BBOX_ROTATIONS = 4 - NUM_ANCHOR_BBOX_DIMENSIONS = 1 - FOREGROUND_ASSIGNMENT_THRESHOLD = 0.6 - BACKGROUND_ASSIGNMENT_THRESHOLD = 0.45 - INCLUDED_CLASSES = ['Car'] - - class AnchorBoxSettings(input_preprocessors.SparseCarV1AnchorBoxSettings): - ROTATIONS = [0, np.pi / 2, 3. * np.pi / 4, np.pi / 4] - - def _configure_input(self, p): - """Base function managing the delegation of job specific input configs.""" - self._configure_generic_input(p) - cluster = cluster_factory.Current() - job = cluster.job - if job.startswith('trainer'): - self._configure_trainer_input(p) - elif job.startswith('decoder'): - self._configure_decoder_input(p) - elif job.startswith('evaler'): - self._configure_evaler_input(p) - else: - tf.logging.info('There are no input configuration changes to for ' - 'job {}.'.format(job)) - if self.RUN_LOCALLY: - p.num_batcher_threads = 1 - p.file_buffer_size = 1 - p.file_parallelism = 1 - - def _configure_generic_input(self, p): - """Update input_config `p` for all jobs.""" - p.file_datasource.file_pattern_prefix = _KITTI_BASE - - # Perform frustum dropping before ground removal (keep_xyz_range). - p.preprocessors.Define( - 'remove_out_of_frustum', - (input_preprocessors.KITTIDropPointsOutOfFrustum.Params()), '') - p.preprocessors_order.insert( - p.preprocessors_order.index('keep_xyz_range'), 'remove_out_of_frustum') - - # Approximate ground removal. - p.preprocessors.keep_xyz_range.keep_z_range = (-1.35, np.inf) - - # Max num points can be smaller since we have dropped points out of frustum. - p.preprocessors.pad_lasers.max_num_points = 32768 - - # TODO(jngiam): Analyze if these settings are optimal. - p.preprocessors.select_centers.num_cell_centers = 256 - p.preprocessors.gather_features.num_points_per_cell = 512 - p.preprocessors.gather_features.sample_neighbors_uniformly = True - p.preprocessors.gather_features.max_distance = 3.0 - - p.preprocessors.assign_anchors.foreground_assignment_threshold = ( - self.FOREGROUND_ASSIGNMENT_THRESHOLD) - p.preprocessors.assign_anchors.background_assignment_threshold = ( - self.BACKGROUND_ASSIGNMENT_THRESHOLD) - - # Apply car anchor box settings. - tile_anchors_p = p.preprocessors.tile_anchors - self.AnchorBoxSettings.Update(p.preprocessors.tile_anchors) - num_anchor_configs = ( - self.NUM_ANCHOR_BBOX_OFFSETS * self.NUM_ANCHOR_BBOX_ROTATIONS * - self.NUM_ANCHOR_BBOX_DIMENSIONS) - - assert len(tile_anchors_p.anchor_box_dimensions) == num_anchor_configs - assert len(tile_anchors_p.anchor_box_rotations) == num_anchor_configs - assert len(tile_anchors_p.anchor_box_offsets) == num_anchor_configs - - # Filter label extractor for anchors and visualization. - if 'labels' in p.extractors: - filtered_labels = [ - kitti_input_generator.KITTILabelExtractor.KITTI_CLASS_NAMES.index( - class_name) for class_name in self.INCLUDED_CLASSES - ] - p.extractors.labels.filter_labels = filtered_labels - p = AddLaserAndCamera(p) - - def _configure_trainer_input(self, p): - """Update input_config `p` for jobs running training.""" - # TODO(bencaine): Change the default in input_generator to be False - # and only set this true in _configure_decoder_input - p.extractors.images.decode_image = False - _MaybeRemove(p.preprocessors_order, 'count_points') - _MaybeRemove(p.preprocessors_order, 'viz_copy') - p.preprocessors.Define( - 'rot_box', (input_preprocessors.RandomBBoxTransform.Params().Set( - max_rotation=np.pi / 20.)), '') - p.preprocessors.Define('random_flip', - input_preprocessors.RandomFlipY.Params(), '') - p.preprocessors.Define( - 'global_rot', - (input_preprocessors.RandomWorldRotationAboutZAxis.Params().Set( - max_rotation=np.pi / 4.)), '') - p.preprocessors.Define( - 'world_scaling', - (input_preprocessors.WorldScaling.Params().Set(scaling=[0.95, 1.05])), - '') - - # Do per object transforms, then random flip, then global rotation, then - # global scaling. - preprocessor_order = [ - 'rot_box', 'random_flip', 'global_rot', 'world_scaling' - ] - insert_index = p.preprocessors_order.index('select_centers') - p.preprocessors_order = ( - p.preprocessors_order[:insert_index] + preprocessor_order + - p.preprocessors_order[insert_index:]) - - # Add ground truth augmenter to before all preprocessors. - allowed_label_ids = [ - kitti_input_generator.KITTILabelExtractor.KITTI_CLASS_NAMES.index( - class_name) for class_name in self.INCLUDED_CLASSES - ] - groundtruth_db = datasource.PrefixedDataSource.Params() - groundtruth_db.file_pattern_prefix = _KITTI_BASE - groundtruth_db.file_pattern = ('kitti_train_object_cls.tfrecord-*-of-00100') - - p.preprocessors.Define( - 'bbox_aug', (input_preprocessors.GroundTruthAugmentor.Params().Set( - groundtruth_database=groundtruth_db, - num_db_objects=19700, - filter_min_points=5, - max_augmented_bboxes=15, - label_filter=allowed_label_ids, - )), '') - p.preprocessors_order = ['bbox_aug'] + p.preprocessors_order - - p.preprocessors.Define('frustum_dropout', - (input_preprocessors.FrustumDropout.Params().Set( - theta_width=0.03, phi_width=0.0)), '') - p.preprocessors_order.insert( - p.preprocessors_order.index('gather_features'), 'frustum_dropout') - - p.batch_size = 2 - p.file_parallelism = 64 - p.num_batcher_threads = 64 - - def _configure_decoder_input(self, p): - """Update input_config `p` for jobs running decoding.""" - p.batch_size = 4 - p.file_parallelism = 8 - p.num_batcher_threads = 8 - p.file_buffer_size = 500 - - def _configure_evaler_input(self, p): - """Update input_config `p` for jobs running evaluation.""" - # TODO(bencaine): Change the default in input_generator to be False - # and only set this true in _configure_decoder_input - p.extractors.images.decode_image = False - _MaybeRemove(p.preprocessors_order, 'count_points') - _MaybeRemove(p.preprocessors_order, 'viz_copy') - p.batch_size = 4 - p.file_parallelism = 8 - p.num_batcher_threads = 8 - p.file_buffer_size = 500 - - def Train(self): - p = KITTISparseLaserTrain.Params() - self._configure_input(p) - return p - - def Test(self): - p = KITTISparseLaserTest.Params() - self._configure_input(p) - return p - - def Dev(self): - p = KITTISparseLaserValidation.Params() - self._configure_input(p) - return p - - def Task(self): - num_classes = len( - kitti_input_generator.KITTILabelExtractor.KITTI_CLASS_NAMES) - p = starnet.ModelV2.Params( - num_classes, - num_anchor_bboxes_offsets=self.NUM_ANCHOR_BBOX_OFFSETS, - num_anchor_bboxes_rotations=self.NUM_ANCHOR_BBOX_ROTATIONS, - num_anchor_bboxes_dimensions=self.NUM_ANCHOR_BBOX_DIMENSIONS) - - p.name = 'sparse_detector' - - tp = p.train - tp.optimizer = optimizer.Adam.Params() - tp.clip_gradient_norm_to_value = 5 - - ep = p.eval - # Evaluate the whole dataset. - ep.samples_per_summary = 0 - - # To be tuned. - p.train.l2_regularizer_weight = 1e-4 - - # Adapted from V1 tuning. - tp.ema_decay = 0.99 - # TODO(b/148537111): consider setting this to True. - tp.ema_decay_moving_vars = False - tp.learning_rate = 0.001 - lr_util.SetExponentialLR( - train_p=tp, - train_input_p=self.Train(), - exp_start_epoch=150, - total_epoch=650) - - p.dimension_loss_weight = .3 - p.location_loss_weight = 3. - p.loss_weight_classification = 1. - p.loss_weight_localization = 3. - p.rotation_loss_weight = 0.3 - - return p - - -@model_registry.RegisterSingleTaskModel -class StarNetCarModel0701(StarNetCarsBase): - """StarNet Car model trained on KITTI.""" - - class AnchorBoxSettings(input_preprocessors.SparseCarV1AnchorBoxSettings): - CENTER_X_OFFSETS = np.linspace(-1.294, 1.294, 5) - CENTER_Y_OFFSETS = np.linspace(-1.294, 1.294, 5) - - def _configure_generic_input(self, p): - super()._configure_generic_input(p) - # For selecting centers, drop points out of frustum and do approximate - # ground removal. - p.preprocessors.select_centers.features_preparation_layers = [ - input_preprocessors.KITTIDropPointsOutOfFrustum.Params(), - input_preprocessors.DropLaserPointsOutOfRange.Params().Set( - keep_z_range=(-1., np.inf)), - ] - - # Remove frustum dropping from original preprocessors. - p.preprocessors_order.remove('remove_out_of_frustum') - - # Keep all points in front of the car for featurizing, do not remove ground. - p.preprocessors.keep_xyz_range.keep_x_range = (0., np.inf) - p.preprocessors.keep_xyz_range.keep_y_range = (-40., 40.) - p.preprocessors.keep_xyz_range.keep_z_range = (-np.inf, np.inf) - p.preprocessors.pad_lasers.max_num_points = 72000 - - p.preprocessors.select_centers.sampling_method = 'farthest_point' - - p.preprocessors.select_centers.num_cell_centers = 768 - - p.preprocessors.gather_features.max_distance = 3.75 - - p.preprocessors.assign_anchors.foreground_assignment_threshold = 0.567087 - # Disable ignore class, by setting background threshold > foreground. - p.preprocessors.assign_anchors.background_assignment_threshold = 1.0 - - p.preprocessors.select_centers.features_preparation_layers = [ - input_preprocessors.KITTIDropPointsOutOfFrustum.Params(), - input_preprocessors.DropLaserPointsOutOfRange.Params().Set( - keep_z_range=(-1.4, np.inf)), - ] - - def _configure_trainer_input(self, p): - super()._configure_trainer_input(p) - - p.preprocessors.Define( - 'global_loc_noise', - (input_preprocessors.GlobalTranslateNoise.Params().Set( - noise_std=[0., 0., 0.35])), '') - p.preprocessors_order.insert( - p.preprocessors_order.index('world_scaling') + 1, 'global_loc_noise') - - def Task(self): - p = super().Task() - - # Builder configuration. - builder = starnet.Builder() - builder.linear_params_init = py_utils.WeightInit.KaimingUniformFanInRelu() - gin_layer_sizes = [32, 256, 512, 256, 256, 128] - num_laser_features = 1 - gin_layers = [ - # Each layer should expect as input - 2 * dims of the last layer's - # output. We assume a middle layer that's the size of 2 * dim_out. - [dim_in * 2, dim_out * 2, dim_out] - for (dim_in, dim_out) in zip(gin_layer_sizes[:-1], gin_layer_sizes[1:]) - ] - p.cell_feature_dims = sum(gin_layer_sizes) - p.cell_featurizer = builder.GINFeaturizerV2( - name='feat', - fc_dims=gin_layer_sizes[0], - mlp_dims=gin_layers, - num_laser_features=num_laser_features, - fc_use_bn=False) - p.anchor_projected_feature_dims = 512 - - # Loss and training params - p.train.learning_rate = 0.001 / 2. # Divide by batch size. - p.focal_loss_alpha = 0.2 - p.focal_loss_gamma = 3.0 - class_name_to_idx = kitti_input_generator.KITTILabelExtractor.KITTI_CLASS_NAMES - num_classes = len(class_name_to_idx) - p.per_class_loss_weight = [0.] * num_classes - p.per_class_loss_weight[class_name_to_idx.index('Car')] = 1. - - # Decoding / NMS params. - p.use_oriented_per_class_nms = True - p.max_nms_boxes = 512 - p.nms_iou_threshold = [0.0] * num_classes - p.nms_iou_threshold[class_name_to_idx.index('Car')] = 0.0831011 - p.nms_score_threshold = [1.0] * num_classes - p.nms_score_threshold[class_name_to_idx.index('Car')] = 0.321310 - p.output_decoder.truncation_threshold = 0.65 - p.output_decoder.filter_predictions_outside_frustum = True - return p - - -@model_registry.RegisterSingleTaskModel -class StarNetPedCycModel0704(StarNetCarsBase): - """StarNet Ped/Cyc model trained on KITTI.""" - - INCLUDED_CLASSES = ['Pedestrian', 'Cyclist'] - - FOREGROUND_ASSIGNMENT_THRESHOLD = 0.48 - # Any value > FOREGROUND is equivalent. - BACKGROUND_ASSIGNMENT_THRESHOLD = 0.80 - - NUM_ANCHOR_BBOX_OFFSETS = 9 - NUM_ANCHOR_BBOX_ROTATIONS = 4 - NUM_ANCHOR_BBOX_DIMENSIONS = 3 - - class AnchorBoxSettings(input_preprocessors.SparseCarV1AnchorBoxSettings): - # PointPillars priors for pedestrian/cyclists. - DIMENSION_PRIORS = [(0.6, 0.8, 1.7), (0.6, 0.6, 1.2), (0.6, 1.76, 1.73)] - ROTATIONS = [0, np.pi / 2, 3. * np.pi / 4, np.pi / 4] - CENTER_X_OFFSETS = np.linspace(-0.31, 0.31, 3) - CENTER_Y_OFFSETS = np.linspace(-0.31, 0.31, 3) - CENTER_Z_OFFSETS = [-0.6] - - def _configure_generic_input(self, p): - super()._configure_generic_input(p) - # For selecting centers, drop points out of frustum and do approximate - # ground removal. - p.preprocessors.select_centers.features_preparation_layers = [ - input_preprocessors.KITTIDropPointsOutOfFrustum.Params(), - input_preprocessors.DropLaserPointsOutOfRange.Params().Set( - keep_z_range=(-1., np.inf)), - ] - - # Remove frustum dropping from original preprocessors. - p.preprocessors_order.remove('remove_out_of_frustum') - - # Keep all points in front of the car for featurizing, do not remove ground. - p.preprocessors.keep_xyz_range.keep_x_range = (0., 48.0) - p.preprocessors.keep_xyz_range.keep_y_range = (-20., 20.) - p.preprocessors.keep_xyz_range.keep_z_range = (-np.inf, np.inf) - p.preprocessors.pad_lasers.max_num_points = 72000 - p.preprocessors.select_centers.sampling_method = 'farthest_point' - p.preprocessors.select_centers.num_cell_centers = 512 - p.preprocessors.select_centers.features_preparation_layers = [ - input_preprocessors.KITTIDropPointsOutOfFrustum.Params(), - input_preprocessors.DropLaserPointsOutOfRange.Params().Set( - keep_z_range=(-1.4, np.inf)), - ] - - p.preprocessors.gather_features.max_distance = 2.55 - - def _configure_trainer_input(self, p): - super()._configure_trainer_input(p) - - allowed_label_ids = [ - kitti_input_generator.KITTILabelExtractor.KITTI_CLASS_NAMES.index( - class_name) for class_name in self.INCLUDED_CLASSES - ] - p.preprocessors.bbox_aug.Set( - num_db_objects=19700, - filter_min_difficulty=2, - filter_min_points=7, - max_augmented_bboxes=2, - max_num_points_per_bbox=1558, - label_filter=allowed_label_ids, - ) - p.batch_size = 2 - - def _configure_decoder_input(self, p): - """Update input_config `p` for jobs running decoding.""" - super()._configure_decoder_input(p) - p.batch_size = 4 - - def _configure_evaler_input(self, p): - """Update input_config `p` for jobs running evaluation.""" - super()._configure_evaler_input(p) - p.batch_size = 4 - - def Task(self): - p = super().Task() - p.train.learning_rate = 7e-4 - - builder = starnet.Builder() - builder.linear_params_init = py_utils.WeightInit.KaimingUniformFanInRelu() - gin_layer_sizes = [32, 256, 512, 256, 256, 128] - num_laser_features = 1 - gin_layers = [ - # Each layer should expect as input - 2 * dims of the last layer's - # output. We assume a middle layer that's the size of 2 * dim_out. - [dim_in * 2, dim_out * 2, dim_out] - for (dim_in, dim_out) in zip(gin_layer_sizes[:-1], gin_layer_sizes[1:]) - ] - p.cell_feature_dims = sum(gin_layer_sizes) - # Disable BN on first layer - p.cell_featurizer = builder.GINFeaturizerV2( - 'feat', - gin_layer_sizes[0], - gin_layers, - num_laser_features, - fc_use_bn=False) - p.anchor_projected_feature_dims = 512 - - class_name_to_idx = kitti_input_generator.KITTILabelExtractor.KITTI_CLASS_NAMES - num_classes = len(class_name_to_idx) - p.per_class_loss_weight = [0.] * num_classes - p.per_class_loss_weight[class_name_to_idx.index('Pedestrian')] = 3.5 - p.per_class_loss_weight[class_name_to_idx.index('Cyclist')] = 3.25 - - p.focal_loss_alpha = 0.9 - p.focal_loss_gamma = 1.25 - - p.use_oriented_per_class_nms = True - p.max_nms_boxes = 1024 - p.nms_iou_threshold = [0.0] * num_classes - p.nms_iou_threshold[class_name_to_idx.index('Cyclist')] = 0.49 - p.nms_iou_threshold[class_name_to_idx.index('Pedestrian')] = 0.32 - - p.nms_score_threshold = [1.0] * num_classes - p.nms_score_threshold[class_name_to_idx.index('Cyclist')] = 0.11 - p.nms_score_threshold[class_name_to_idx.index('Pedestrian')] = 0.23 - - p.output_decoder.filter_predictions_outside_frustum = True - p.output_decoder.truncation_threshold = 0.65 - # Equally weight pedestrian and cyclist moderate classes. - p.output_decoder.ap_metric.metric_weights = { - 'easy': np.array([0.0, 0.0, 0.0]), - 'moderate': np.array([0.0, 1.0, 1.0]), - 'hard': np.array([0.0, 0.0, 0.0]) - } - - return p diff --git a/lingvo/tasks/car/params/params.py b/lingvo/tasks/car/params/params.py deleted file mode 100644 index 661d802a..00000000 --- a/lingvo/tasks/car/params/params.py +++ /dev/null @@ -1,22 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Car models hyper-parameters.""" - -# Import ModelParams to ensure that they are added to the global registry. -# pylint: disable=unused-import -import lingvo.tasks.car.params.kitti -import lingvo.tasks.car.params.waymo -# pylint: enable=unused-import diff --git a/lingvo/tasks/car/params/waymo.py b/lingvo/tasks/car/params/waymo.py deleted file mode 100644 index d14143f2..00000000 --- a/lingvo/tasks/car/params/waymo.py +++ /dev/null @@ -1,473 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Train models on the waymo open dataset.""" - -import os - -from lingvo import model_registry -from lingvo.core import base_model_params -from lingvo.core import cluster_factory -from lingvo.core import hyperparams -from lingvo.core import optimizer -from lingvo.core import py_utils -from lingvo.tasks.car import input_preprocessors -from lingvo.tasks.car import lr_util -from lingvo.tasks.car import starnet -from lingvo.tasks.car.waymo import waymo_decoder -from lingvo.tasks.car.waymo import waymo_metadata -from lingvo.tasks.car.waymo import waymo_open_input_generator -import numpy as np - -# Set $WAYMO_DIR to the base path of where all the WAYMO files can be found. -_WAYMO_BASE = os.environ.get('WAYMO_DIR', 'FILL-ME-IN') - - -def WaymoTrainSpec(params): - """Training set.""" - p = params.Copy() - p.file_datasource.file_pattern = 'train.tfr-*-of-01000' - p.num_samples = 158361 - return p - - -def WaymoMiniTrainSpec(params): - """Training shards used for decoding.""" - p = params.Copy() - p.file_datasource.file_pattern = 'train.tfr-000[0-2]?-of-01000' - p.num_samples = 4773 - return p - - -def WaymoValSpec(params): - """Validation set. - - Validation contains no overlapping run segments with training. - - Args: - params: A Waymo open dataset input params. - - Returns: - An updated Waymo open dataset params with the validation spec. - """ - p = params.Copy() - p.file_datasource.file_pattern = 'valid.tfr-*-of-01000' - p.num_samples = 40077 - return p - - -def WaymoMinivalSpec(params): - """Miniature validation set (10% of full validation set).""" - p = params.Copy() - p.file_datasource.file_pattern = 'valid.tfr-000??-of-01000' - p.num_samples = 4109 - return p - - -def WaymoTestSpec(params): - # For now, using validation set, since test set is not available. - return WaymoValSpec(params) - - -class WaymoSparseLaserTrain(waymo_open_input_generator.WaymoSparseLaser): - """Waymo train set with sparse laser data.""" - - @classmethod - def Params(cls): - p = super().Params() - return WaymoTrainSpec(p) - - -class WaymoSparseLaserValidation(waymo_open_input_generator.WaymoSparseLaser): - """Waymo validation set with sparse laser data.""" - - @classmethod - def Params(cls): - p = super().Params() - return WaymoValSpec(p) - - -class WaymoSparseLaserMinival(waymo_open_input_generator.WaymoSparseLaser): - """Waymo mini validation set with sparse laser data.""" - - @classmethod - def Params(cls): - p = super().Params() - return WaymoMinivalSpec(p) - - -class WaymoSparseLaserTest(waymo_open_input_generator.WaymoSparseLaser): - """Waymo test set with sparse laser data. Has no groundtruth labels.""" - - @classmethod - def Params(cls): - p = super().Params() - return WaymoTestSpec(p) - - -def _FilterKeepLabels(params, label_names): - """Keep only label names in 'label_names' from input.""" - metadata = waymo_metadata.WaymoMetadata() - filtered_labels = [ - metadata.ClassNames().index(label_name) for label_name in label_names - ] - params.extractors.labels.filter_labels = filtered_labels - - -@model_registry.RegisterSingleTaskModel -class StarNetBase(base_model_params.SingleTaskModelParams): - """StarNet model for running on Waymo. - - This the base model, please refer to specialized vehicles, pedestrians, etc. - models below to get an appropriate model to train. - """ - RUN_LOCALLY = False - NUM_ANCHOR_BBOX_OFFSETS = 25 - NUM_ANCHOR_BBOX_ROTATIONS = 4 - NUM_ANCHOR_BBOX_DIMENSIONS = 1 - - # Ground truth filtering - GT_MIN_NUM_POINTS = 5 - - # Architecture Params - NUM_GIN_LAYERS = 5 - GIN_HIDDEN_DIMS = 64 - NUM_POINTS_PER_CELL = 512 - - class AnchorBoxSettings(input_preprocessors.SparseCarV1AnchorBoxSettings): - ROTATIONS = [0, np.pi / 2, 3. * np.pi / 4, np.pi / 4] - - def _configure_input(self, p, split): - p.file_datasource.file_pattern_prefix = _WAYMO_BASE - - job_type = cluster_factory.Current().job - - max_num_points = int(64 * 2650 * 1.5) - p.preprocessors = hyperparams.Params() - p.preprocessors.Define('filter_nlz_points', - waymo_open_input_generator.FilterNLZPoints.Params(), - '') - # TODO(bencaine): Change this to filter based on difficulty instead - p.preprocessors.Define( - 'filter_groundtruth', - input_preprocessors.FilterGroundTruthByNumPoints.Params(), '') - p.preprocessors.Define('viz_copy', - input_preprocessors.CreateDecoderCopy.Params(), '') - p.preprocessors.Define('select_centers', - input_preprocessors.SparseCenterSelector.Params(), - '') - p.preprocessors.Define( - 'gather_features', - input_preprocessors.SparseCellGatherFeatures.Params(), '') - p.preprocessors.Define('tile_anchors', - input_preprocessors.TileAnchorBBoxes.Params(), '') - p.preprocessors.Define('assign_anchors', - input_preprocessors.AnchorAssignment.Params(), '') - p.preprocessors.Define( - 'pad_lasers', - input_preprocessors.PadLaserFeatures.Params().Set( - max_num_points=max_num_points), '') - - p.preprocessors.viz_copy.pad_lasers.max_num_points = max_num_points - p.preprocessors.filter_groundtruth.min_num_points = self.GT_MIN_NUM_POINTS - - p.preprocessors.select_centers.num_cell_centers = 1024 - p.preprocessors.gather_features.num_points_per_cell = self.NUM_POINTS_PER_CELL - p.preprocessors.gather_features.sample_neighbors_uniformly = True - p.preprocessors.gather_features.max_distance = 2.75 - - p.preprocessors.assign_anchors.foreground_assignment_threshold = 0.6 - p.preprocessors.assign_anchors.background_assignment_threshold = 0.45 - - p.preprocessors_order = [ - 'filter_nlz_points', - 'filter_groundtruth', - 'viz_copy', - 'select_centers', - 'gather_features', - 'tile_anchors', - 'assign_anchors', - 'pad_lasers', - ] - - # Apply car anchor box settings. - tile_anchors_p = p.preprocessors.tile_anchors - self.AnchorBoxSettings.Update(p.preprocessors.tile_anchors) - num_anchor_configs = self.AnchorBoxSettings.NumAnchors() - - assert len(tile_anchors_p.anchor_box_dimensions) == num_anchor_configs - assert len(tile_anchors_p.anchor_box_rotations) == num_anchor_configs - assert len(tile_anchors_p.anchor_box_offsets) == num_anchor_configs - - # If this is not the decoder job (e.g., this is trainer), turn off - # image decoding, do not count points, and do not make visualization copies. - if job_type != 'decoder': - p.preprocessors_order.remove('viz_copy') - # Do not need laser points during training for current V2 model. This - # reduces amount of data sent over during training. - p.preprocessors.pad_lasers.max_num_points = 0 - - p.file_buffer_size = 32 - p.file_parallelism = 8 - p.num_batcher_threads = 8 - if self.RUN_LOCALLY: - p.num_batcher_threads = 1 - p.file_buffer_size = 1 - p.file_parallelism = 1 - - if job_type.startswith('trainer'): - p.batch_size = 2 - else: - p.batch_size = 4 - p.file_buffer_size = 64 - p.file_parallelism = 16 - p.num_batcher_threads = 16 - return p - - def Train(self): - p = waymo_open_input_generator.WaymoSparseLaser.Params() - p = WaymoTrainSpec(p) - p = self._configure_input(p, 'Train') - return p - - def Minitrain(self): - p = self.Train() - p = WaymoMiniTrainSpec(p) - return p - - def Test(self): - p = waymo_open_input_generator.WaymoSparseLaser.Params() - p = WaymoTestSpec(p) - p = self._configure_input(p, 'Test') - return p - - def Dev(self): - p = waymo_open_input_generator.WaymoSparseLaser.Params() - p = WaymoValSpec(p) - p = self._configure_input(p, 'Dev') - return p - - def Minidev(self): - p = self.Dev() - p = WaymoMinivalSpec(p) - return p - - def Task(self): - metadata = waymo_metadata.WaymoMetadata() - num_classes = len(metadata.ClassNames()) - p = starnet.ModelV2.Params( - num_classes, - num_anchor_bboxes_offsets=self.NUM_ANCHOR_BBOX_OFFSETS, - num_anchor_bboxes_rotations=self.NUM_ANCHOR_BBOX_ROTATIONS, - num_anchor_bboxes_dimensions=self.NUM_ANCHOR_BBOX_DIMENSIONS, - num_laser_features=3) - - # Update the Point Cloud Featurizer architecture - starnet_builder = starnet.Builder() - starnet_builder.linear_params_init = ( - py_utils.WeightInit.KaimingUniformFanInRelu()) - - gin_layers = [ - [self.GIN_HIDDEN_DIMS*2, self.GIN_HIDDEN_DIMS*4, self.GIN_HIDDEN_DIMS] - ] * self.NUM_GIN_LAYERS # pyformat: disable - - p.cell_featurizer = starnet_builder.GINFeaturizerV2( - 'feat', - num_laser_features=3, - fc_dims=self.GIN_HIDDEN_DIMS, - mlp_dims=gin_layers, - fc_use_bn=False) - p.cell_feature_dims = self.GIN_HIDDEN_DIMS * (self.NUM_GIN_LAYERS + 1) - - p.output_decoder = waymo_decoder.WaymoOpenDatasetDecoder.Params() - p.max_nms_boxes = 512 - p.use_oriented_per_class_nms = True - - # Note: Sub-classes need to set nms_iou_threshold and nms_score_threshold - # appropriately. - p.nms_iou_threshold = [0.0] * num_classes - - # TODO(jngiam): 1.1 for untrained classes is needed to avoid an issue - # with boxutils error. - p.nms_score_threshold = [1.1] * num_classes - - p.name = 'starnet' - tp = p.train - tp.optimizer = optimizer.Adam.Params() - tp.clip_gradient_norm_to_value = 5 - - ep = p.eval - - # Train set uses a smaller decoding set, so we can - # safely eval over the entire input. - ep.samples_per_summary = 0 - - # To be tuned. - p.train.l2_regularizer_weight = 1e-8 - - cluster = cluster_factory.Current() - train_cluster_p = cluster.params.Copy() - train_cluster_p.job = 'trainer_client' - train_cluster_p.mode = 'sync' - - # When running a decoding only job, there are no trainer workers, so we set - # worker replicas to 1 as a dummy value. - if train_cluster_p.worker.replicas <= 0: - train_cluster_p.worker.replicas = 1 - - # Set learning rate and schedule. - with cluster_factory.Cluster(train_cluster_p): - train_input_p = self.Train() - - # Adapted from V1 tuning. - tp.ema_decay = 0.99 - # TODO(b/148537111): consider setting this to True. - tp.ema_decay_moving_vars = False - tp.learning_rate = 0.001 - lr_util.SetExponentialLR( - train_p=tp, - train_input_p=train_input_p, - exp_start_epoch=5, - total_epoch=75) - - p.dimension_loss_weight = .3 - p.location_loss_weight = 3. - p.loss_weight_classification = 1. - p.loss_weight_localization = 3. - p.rotation_loss_weight = 0.3 - - return p - - -@model_registry.RegisterSingleTaskModel -class StarNetVehicle(StarNetBase): - """StarNet model for cars, running on Waymo.""" - GT_MIN_NUM_POINTS = 5 - - class AnchorBoxSettings(input_preprocessors.SparseCarV1AnchorBoxSettings): - DIMENSION_PRIORS = [(4.725808, 2.079292, 1.768998)] - ROTATIONS = [0, np.pi / 2, 3. * np.pi / 4, np.pi / 4] - CENTER_X_OFFSETS = np.linspace(-1.294, 1.294, 5) - CENTER_Y_OFFSETS = np.linspace(-1.294, 1.294, 5) - CENTER_Z_OFFSETS = [0.819622] - - def _configure_input(self, p, split): - p = super()._configure_input(p, split) - - # Select points in approx z range, set using 10 and 90 percentile from - # train data. - p.preprocessors.select_centers.features_preparation_layers = [ - input_preprocessors.DropLaserPointsOutOfRange.Params().Set( - keep_z_range=(0.197431, 2.046459)), - ] - - _FilterKeepLabels(p, ['Vehicle']) - return p - - def Task(self): - p = super().Task() - p.train.l2_regularizer_weight = 3e-5 - - class_names = waymo_metadata.WaymoMetadata().ClassNames() - num_classes = len(class_names) - - p.per_class_loss_weight = [0.] * num_classes - p.per_class_loss_weight[class_names.index('Vehicle')] = 1.0 - - # See WaymoMetadata.EvalClassIndices for correspondences for metric_weights. - p.output_decoder.ap_metric.metric_weights = { - 'default': np.array([1.0, 0.0, 0.0]), - } - p.nms_iou_threshold[class_names.index('Vehicle')] = 0.03 - p.nms_score_threshold[class_names.index('Vehicle')] = 0.31 - return p - - -@model_registry.RegisterSingleTaskModel -class StarNetPed(StarNetBase): - """StarNet model for pedestrians, running on Waymo.""" - NUM_ANCHOR_BBOX_ROTATIONS = 2 - GT_MIN_NUM_POINTS = 5 - - class AnchorBoxSettings(input_preprocessors.SparseCarV1AnchorBoxSettings): - DIMENSION_PRIORS = [(0.901340, 0.857218, 1.712443)] - ROTATIONS = [0, np.pi / 2] - CENTER_X_OFFSETS = np.linspace(-0.6, 0.6, 5) - CENTER_Y_OFFSETS = np.linspace(-0.6, 0.6, 5) - CENTER_Z_OFFSETS = [0.819622] - - def _configure_input(self, p, split): - p = super()._configure_input(p, split) - - # Select points in approx z range, set using 10 and 90 percentile from - # train data. - p.preprocessors.select_centers.features_preparation_layers = [ - input_preprocessors.DropLaserPointsOutOfRange.Params().Set( - keep_z_range=(0.09522381, 1.720825)), - ] - - _FilterKeepLabels(p, ['Pedestrian']) - return p - - def Task(self): - p = super().Task() - p.train.l2_regularizer_weight = 3e-6 - - class_names = waymo_metadata.WaymoMetadata().ClassNames() - num_classes = len(class_names) - - p.per_class_loss_weight = [0.] * num_classes - p.per_class_loss_weight[class_names.index('Pedestrian')] = 1.0 - - # See WaymoMetadata.EvalClassIndices for correspondences for metric_weights. - p.output_decoder.ap_metric.metric_weights = { - 'default': np.array([0.0, 1.0, 0.0]), - } - p.nms_iou_threshold[class_names.index('Pedestrian')] = 0.46 - p.nms_score_threshold[class_names.index('Pedestrian')] = 0.01 - - return p - - -@model_registry.RegisterSingleTaskModel -class StarNetPedFused(StarNetPed): - """StarNet Pedestrian model with SparseSampler fused input preprocessor.""" - - def _configure_input(self, p, split): - p = super()._configure_input(p, split) - - job_type = cluster_factory.Current().job - - if job_type.startswith('trainer') and not self.RUN_LOCALLY: - p.file_buffer_size = 48 - p.file_parallelism = 48 - p.num_batcher_threads = 48 - - # Fuses select_centers and gather_features into one sampler. - p.preprocessors.Define( - 'sampler', - input_preprocessors.SparseSampler.Params().Set( - center_selector='farthest', - neighbor_sampler='uniform', - num_centers=p.preprocessors.select_centers.num_cell_centers, - keep_z_range=(0.09522381, 1.720825), - num_neighbors=p.preprocessors.gather_features.num_points_per_cell, - max_distance=2.75), '') - p.preprocessors.Delete('select_centers') - p.preprocessors.Delete('gather_features') - p.preprocessors_order.remove('select_centers') - p.preprocessors_order[p.preprocessors_order.index( - 'gather_features')] = 'sampler' - return p diff --git a/lingvo/tasks/car/pillars.py b/lingvo/tasks/car/pillars.py deleted file mode 100644 index e737f8e2..00000000 --- a/lingvo/tasks/car/pillars.py +++ /dev/null @@ -1,665 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -r"""PointPillars implementation. - -[1] PointPillars. https://arxiv.org/abs/1812.05784 -""" - -import enum -import functools -from lingvo import compat as tf -from lingvo.core import base_layer -from lingvo.core import layers -from lingvo.core import optimizer -from lingvo.core import py_utils -from lingvo.tasks.car import builder_lib -from lingvo.tasks.car import detection_3d_lib -from lingvo.tasks.car import geometry -from lingvo.tasks.car import point_detector -import numpy as np - - -def SparseToDense(grid_shape, locations, feats): - """Converts a sparse representation back to the dense grid. - - Args: - grid_shape: (nx, ny, nz). The shape of the grid. - locations: [b, p, 3]. Locations of the pillars. - feats: [b, p, fdims]. Extracted features for pillars. - - Returns: - grid_feats of shape [b, nx, ny, nz * fdims]. - """ - nx, ny, nz = grid_shape - b, p, _ = py_utils.GetShape(locations, 3) - feats = py_utils.HasShape(feats, [b, p, -1]) - _, _, fdims = py_utils.GetShape(feats, 3) - indices = tf.concat( - [tf.tile(tf.range(b)[:, tf.newaxis, tf.newaxis], [1, p, 1]), locations], - axis=2) - grid = tf.scatter_nd(indices, feats, [b, nx, ny, nz, fdims]) - return tf.reshape(grid, [b, nx, ny, nz * fdims]) - - -class PointsToGridFeaturizer(base_layer.BaseLayer): - """Layer for processing points to grid outputs.""" - - @classmethod - def Params(cls, num_laser_features, num_output_features=64): - p = super().Params() - p.Define('num_laser_features', num_laser_features, - 'The number of (non-xyz) laser features of the input.') - - builder = Builder() - total_num_laser_features = 9 + num_laser_features - p.Define( - 'featurizer', - builder.Featurizer('feat', - [total_num_laser_features, num_output_features]), - 'Point cloud feature extractor.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self.CreateChild('featurizer', p.featurizer) - - def FProp(self, theta, input_batch): - # pyformat: disable - """Compute features for the pillars and convert them back to a dense grid. - - Args: - theta: A `.NestedMap` object containing variable values of this task. - input_batch: A `.NestedMap` object containing input tensors. Following - keys are required: - - - grid_num_points: Integer tensor with shape [batch size, nx, ny, nz], - where nx, ny, nz corresponds to the grid sizes (i.e., number of voxels - in each axis dimension). - - pillar_points: Float tensor with shape [batch size, num_pillars, - num_points_per_pillar, 3 + num_laser_features] - - pillar_centers: Float tensor with shape [batch size, num_pillars, - num_points_per_pillar, 3] - - pillar_locations: Float tensor with shape [batch size, num_pillars, 3] - - Returns: - The dense features with shape [b, nx, ny, nz * fdims]. - """ - # pyformat: enable - p = self.params - bs, nx, ny, nz = py_utils.GetShape(input_batch.grid_num_points, 4) - # Process points to concatenate a set of fixed features (e.g., - # add means, centers, normalize points to means). - num_features = 3 + p.num_laser_features - pillar_points = py_utils.HasShape(input_batch.pillar_points, - [bs, -1, -1, num_features]) - _, npillars, npoints, _ = py_utils.GetShape(pillar_points, 4) - pillar_xyz = pillar_points[..., :3] - - # Compute number of points per pillar and prepare for broadcasting. - pillar_num_points = tf.gather_nd( - input_batch.grid_num_points, input_batch.pillar_locations, batch_dims=1) - pillar_num_points = pillar_num_points[..., tf.newaxis, tf.newaxis] - - # Compute mean by computing sum and dividing by number of points. Clip the - # denominator by 1.0 to gracefully handle empty pillars. - pillar_sum = tf.reduce_sum(pillar_xyz, axis=2, keepdims=True) - pillar_means = pillar_sum / tf.maximum( - tf.cast(pillar_num_points, tf.float32), 1.0) - - pillar_feats = pillar_points[..., 3:] - pillar_centers = py_utils.HasShape(input_batch.pillar_centers, - [bs, -1, 1, 3]) - pillar_concat = tf.concat( - axis=3, - values=[ - pillar_xyz - pillar_means, pillar_feats, - tf.tile(pillar_means, [1, 1, npoints, 1]), - tf.tile(pillar_centers, [1, 1, npoints, 1]) - ]) - # Featurize pillars. - pillar_features = self.featurizer.FProp(theta.featurizer, pillar_concat) - - # Convert back to the dense grid. - pillar_locations = py_utils.HasShape(input_batch.pillar_locations, - [bs, npillars, 3]) - dense_features = SparseToDense( - grid_shape=(nx, ny, nz), - locations=pillar_locations, - feats=pillar_features) - return dense_features - - -# pyformat: disable -class Builder(builder_lib.ModelBuilderBase): - """Builder for the Pillars model.""" - - def __init__(self): - super().__init__() - self.conv_init_method = builder_lib.KaimingUniformFanInRelu - self.linear_params_init = py_utils.WeightInit.KaimingUniformFanInRelu() - self.bn_params_init = py_utils.WeightInit.UniformPositive() - - def Featurizer(self, name, dims): - return self._Seq( - name, - self._MLP('mlp', dims), - self._Max('max')) - - def _Deconv(self, name, filter_shape, stride): - return layers.DeconvLayer.Params().Set( - name=name, - filter_shape=filter_shape, - filter_stride=(stride, stride)) - - def _Block(self, name, stride, repeats, idims, odims): - """[1]. Sec 2.2.""" - return self._Seq( - name, - self._Conv('c3x3', (3, 3, idims, odims), stride), - self._Rep( - 'rep', - repeats, - self._Conv('c3x3', (3, 3, odims, odims))), - self._Fetch('final')) - - def _TopDown(self, name, strides=(2, 2, 2), channel_multiplier=1): - """[1]. Sec 2.2.""" - if len(strides) != 3: - raise ValueError('`strides` expected to be list/tuple of len 3.') - - return self._Seq( - name, - self._Block('b0', strides[0], 3, channel_multiplier * 64, - channel_multiplier * 64), - self._Block('b1', strides[1], 5, channel_multiplier * 64, - channel_multiplier * 128), - self._Block('b2', strides[2], 5, channel_multiplier * 128, - channel_multiplier * 256)) - - def _Upsample(self, name, stride, idims, odims): - """[1]. Sec 2.2.""" - # Match the kernel size to the stride in order to ensure that the output - # activation map has no holes and to minimize any checkerboard artifacts. - # TODO(shlens): Consider replacing this in the future with a bilinear - # interpolation followed by a 3x3 convolution. - kernel = stride - return self._Seq( - name, - self._Deconv('deconv', (kernel, kernel, odims, idims), stride), - self._BN('bn', odims), - self._Relu('relu')) - - def Contract(self, down_strides=(2, 2, 2), channel_multiplier=1): - """Contracting part of [1] Sec 2.2.""" - return self._Branch( - 'branch', - self._TopDown('topdown', strides=down_strides, - channel_multiplier=channel_multiplier), - ['b1.final', 'b0.final']) - - def Expand(self, odims, channel_multiplier=1): - """Expanding part of [1] Sec 2.2.""" - # Note that the resulting output will be 3*odims - return self._Concat( - 'concat', - self._Seq( - 'b2', - self._ArgIdx('idx', [0]), - self._Upsample('ups', 4, channel_multiplier * 256, odims)), - self._Seq( - 'b1', - self._ArgIdx('idx', [1]), - self._Upsample('ups', 2, channel_multiplier * 128, odims)), - self._Seq( - 'b0', - self._ArgIdx('idx', [2]), - self._Upsample('ups', 1, channel_multiplier * 64, odims))) - - def Backbone(self, odims, down_strides=(2, 2, 2), channel_multiplier=1): - """[1]. Sec 2.2.""" - # We assume (H, W) are multiple of 8. So that we can concat - # multiple-scale feature maps together after upsample. - return self._Seq( - 'backbone', - self.Contract(down_strides, channel_multiplier=channel_multiplier), - self.Expand(odims, channel_multiplier=channel_multiplier)) - - def Detector(self, name, idims, odims, conv_init_method=None, - bias_params_init=None): - # Implemented according to VoxelNet - # https://arxiv.org/pdf/1711.06396.pdf - # May add more Conv2D layers before predictor for better performance. - return self._Seq( - name, - self._ConvPlain('predict', (3, 3, idims, odims), - conv_init_method=conv_init_method), - self._Bias('predict_bias', odims, bias_params_init)) - -# pyformat: enable - - -class LossNormType(enum.Enum): - NO_NORM = 0 - NORM_BY_NUM_POSITIVES = 1 - - -class ModelV1(point_detector.PointDetectorBase): - """PointPillars model. - - Base class implements common Decoder functions, though they can be - overridden if desired. - """ - - NUM_OUTPUT_CHANNELS = 128 - - @classmethod - def Params(cls, - grid_size_z=1, - num_anchors=2, - num_classes=1, - num_laser_features=1): - p = super().Params(num_classes=num_classes) - p.Define('grid_size_z', grid_size_z, 'The grid size along the z-axis.') - p.Define('num_anchors', num_anchors, 'The number of anchor boxes.') - p.Define('num_laser_features', num_laser_features, - 'The number of (non-xyz) laser features of the input.') - p.Define('input_featurizer', - PointsToGridFeaturizer.Params(num_laser_features), - 'Point cloud feature extractor.') - - builder = Builder() - p.Define('backbone', builder.Backbone(cls.NUM_OUTPUT_CHANNELS), - 'Dense features pyramid.') - # Backbone() concatenates 3 different scales of features. - idims = 3 * cls.NUM_OUTPUT_CHANNELS - # 7: predicted (dx, dy, dz, dw, dl, dh, dt). - class_odims = grid_size_z * num_anchors * num_classes - reg_odims = grid_size_z * num_anchors * 7 - rot_odims = grid_size_z * num_anchors * 2 - # Although theoretically a single conv layer can generate both the - # regression and classification logits, we try to implement the paper - # faithfully, which uses two different layers. - p.Define('class_detector', builder.Detector('class', idims, class_odims), - 'Dense features to class logits.') - p.Define('regression_detector', builder.Detector('reg', idims, reg_odims), - 'Dense features to regression logits.') - p.Define('direction_classifier', builder.Detector('dir', idims, rot_odims), - 'Dense features to rotation direction classifier.') - # We disable the direction classifier by default since it has - # weird discontinous optimization objectives around the threshold - # and it doesn't improve mAP. - p.Define( - 'direction_classifier_weight', 0.0, - 'If > 0, adds a direction classifier to the model and adds ' - 'to the total loss with this weight.') - p.Define( - 'direction_aware_rot_loss', False, 'If True, changes the heading loss ' - 'from sin(theta_delta) to WrapAngleRad(theta_delta), which makes the ' - 'model produce headings between [-pi to pi].') - - p.Define( - 'squash_rotation_predictions', False, - 'Apply tanh squashing to rotation predictions to ensure outputs ' - 'are between (-pi, pi).') - p.Define('focal_loss_alpha', 0.25, 'The alpha parameter in focal loss ' - '(see paper eq. 4).') - p.Define('focal_loss_gamma', 2.0, 'The gamma parameter in focal loss ' - '(see paper eq. 4).') - p.Define( - 'localization_loss_weight', 2.0, - 'Localization loss weight factor between localization and ' - 'class loss contributions.') - p.Define( - 'classification_loss_weight', 1.0, - 'Classification loss weight factor between localization and ' - 'class loss contributions.') - p.Define( - 'location_loss_weight', 1.0, - 'Weight multiplier for contribution of location loss ' - 'to full localization/regression loss') - p.Define( - 'dimension_loss_weight', 1.0, - 'Weight multiplier for contribution of dimension loss ' - 'to full localization/regression loss') - p.Define( - 'rotation_loss_weight', 1.0, - 'Weight multiplier for contribution of rotation loss ' - 'to full localization/regression loss') - - p.Define('loss_norm_type', LossNormType.NORM_BY_NUM_POSITIVES, - 'Normalization function for class and regularization weights.') - - p.Define('oracle_location', False, - 'If true, the model predicts the ground truth for location.') - p.Define('oracle_dimension', False, - 'If true, the model predicts the ground truth for dimension.') - p.Define('oracle_rotation', False, - 'If true, the model predicts the ground truth for rotation.') - - tp = p.train - tp.learning_rate = 0.001 - tp.optimizer = optimizer.Momentum.Params().Set(alpha=0.9) - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self._utils = detection_3d_lib.Utils3D() - - self.CreateChild('input_featurizer', p.input_featurizer) - self.CreateChild('backbone', p.backbone) - self.CreateChild('class_detector', p.class_detector) - self.CreateChild('regression_detector', p.regression_detector) - if p.direction_classifier_weight > 0.0: - self.CreateChild('direction_classifier', p.direction_classifier) - - def ComputePredictions(self, theta, input_batch): - """Computes predictions for `input_batch`. - - Args: - theta: A `.NestedMap` object containing variable values of this task. - input_batch: A `.NestedMap` object containing input tensors to this tower. - - Returns: - A `.NestedMap` contains - logits - [b, nx, ny, nz, na, 7 + num_classes] - """ - p = self.params - input_batch.Transform(lambda x: (x.shape, x.shape.num_elements())).VLog( - 0, 'input_batch shapes: ') - - # Make pillars representation from input_batch. - dense_features = self.input_featurizer.FProp(theta.input_featurizer, - input_batch) - - # Backbone - tf.logging.vlog(1, 'dense_features.shape = %s', dense_features.shape) - act = self.backbone.FProp(theta.backbone, dense_features) - tf.logging.vlog(1, 'act.shape = %s', act.shape) - - # Convert the output of the backbone into class logits and regression - # residuals using two different layers. - class_detection = self.class_detector.FProp(theta.class_detector, act) - reg_detection = self.regression_detector.FProp(theta.regression_detector, - act) - bs, nx, ny, _ = py_utils.GetShape(class_detection, 4) - predicted_classification_logits = tf.reshape( - class_detection, - [bs, nx, ny, p.grid_size_z, p.num_anchors, p.num_classes]) - predicted_residuals = tf.reshape( - reg_detection, [bs, nx, ny, p.grid_size_z, p.num_anchors, 7]) - - if p.squash_rotation_predictions: - predicted_rotations = predicted_residuals[..., 6:] - predicted_rotations = np.pi * tf.tanh(predicted_rotations) - predicted_residuals = tf.concat( - [predicted_residuals[..., :6], predicted_rotations], axis=-1) - - if p.oracle_location or p.oracle_dimension or p.oracle_rotation: - gt_residuals = py_utils.HasShape( - input_batch.anchor_localization_residuals, - [bs, nx, ny, p.grid_size_z, p.num_anchors, 7]) - - # Replace the predicted components with the ground truth if needed. - if p.oracle_location: - location = gt_residuals[..., 0:3] - else: - location = predicted_residuals[..., 0:3] - - if p.oracle_dimension: - dimension = gt_residuals[..., 3:6] - else: - dimension = predicted_residuals[..., 3:6] - - if p.oracle_rotation: - rotation = gt_residuals[..., 6:] - else: - rotation = predicted_residuals[..., 6:] - predicted_residuals = tf.concat([location, dimension, rotation], axis=-1) - - ret = py_utils.NestedMap({ - 'residuals': predicted_residuals, - 'classification_logits': predicted_classification_logits, - }) - - if p.direction_classifier_weight > 0.0: - predicted_dir = self.direction_classifier.FProp( - theta.direction_classifier, act) - predicted_dir = tf.reshape(predicted_dir, - [bs, nx, ny, p.grid_size_z, p.num_anchors, 2]) - ret.predicted_dir = predicted_dir - - return ret - - def _ComputeClassificationLoss(self, predictions, input_batch, class_weights): - """Compute classification loss for the given predictions. - - Args: - predictions: The output of `ComputePredictions`, contains: logits - [b, - nx, ny, nz, na, 7 + num_classes]. na is the number of anchor - boxes per cell. [..., :7] are (dx, dy, dz, dw, dl, dh, dt). - input_batch: The input batch from which we accesses the groundtruth. - class_weights: Per-class weights to use in loss computation. - - Returns: - Classification loss. - - """ - p = self.params - predicted_class_logits = py_utils.HasShape( - predictions.classification_logits, - [-1, -1, -1, -1, p.num_anchors, p.num_classes]) - bs, nx, ny, nz, na, _ = py_utils.GetShape(predicted_class_logits, 6) - assigned_gt_labels = py_utils.HasShape(input_batch.assigned_gt_labels, - [bs, nx, ny, nz, na]) - class_loss = py_utils.SigmoidCrossEntropyFocalLoss( - logits=predicted_class_logits, - labels=tf.one_hot(assigned_gt_labels, p.num_classes), - alpha=p.focal_loss_alpha, - gamma=p.focal_loss_gamma) - class_loss *= class_weights[..., tf.newaxis] - class_loss_sum = tf.reduce_sum(class_loss) - return class_loss_sum - - def ComputeLoss(self, theta, predictions, input_batch): - """Computes loss and other metrics for the given predictions. - - Args: - theta: A `.NestedMap` object containing variable values of this task. - predictions: The output of `ComputePredictions`, contains: logits - [b, - nx, ny, nz, na, 7 + num_classes]. na is the number of anchor - boxes per cell. [..., :7] are (dx, dy, dz, dw, dl, dh, dt). - input_batch: The input batch from which we accesses the groundtruth. - - Returns: - Two dicts defined as BaseTask.ComputeLoss. - """ - p = self.params - predicted_residuals = py_utils.HasShape(predictions.residuals, - [-1, -1, -1, -1, p.num_anchors, 7]) - predicted_class_logits = py_utils.HasShape( - predictions.classification_logits, - [-1, -1, -1, -1, p.num_anchors, p.num_classes]) - bs, nx, ny, nz, na, _ = py_utils.GetShape(predicted_class_logits, 6) - - # Compute class and regression weights. - class_weights = input_batch.assigned_cls_mask - class_weights = py_utils.HasShape(class_weights, [bs, nx, ny, nz, na]) - reg_weights = input_batch.assigned_reg_mask - reg_weights = py_utils.HasShape(reg_weights, [bs, nx, ny, nz, na]) - reg_weights = tf.expand_dims(reg_weights, -1) - - if p.loss_norm_type == LossNormType.NORM_BY_NUM_POSITIVES: - # Compute number of positive anchors per example. - foreground_mask = py_utils.HasShape(input_batch.assigned_reg_mask, - [bs, nx, ny, nz, na]) - # Sum to get the number of foreground anchors for each example. - loss_normalization = tf.reduce_sum(foreground_mask, axis=[1, 2, 3, 4]) - loss_normalization = tf.maximum(loss_normalization, - tf.ones_like(loss_normalization)) - # Reshape for broadcasting. - loss_normalization = tf.reshape(loss_normalization, [bs, 1, 1, 1, 1, 1]) - - class_weights /= loss_normalization - reg_weights /= loss_normalization - - # Classification loss. - class_loss_sum = self._ComputeClassificationLoss(predictions, input_batch, - class_weights) - - # Regression loss. - anchor_localization_residuals = py_utils.HasShape( - input_batch.anchor_localization_residuals, [bs, nx, ny, nz, na, 7]) - - # Location and dimensions loss. - reg_loc_and_dims_loss = self._utils.ScaledHuberLoss( - predictions=py_utils.HasShape(predicted_residuals[..., :6], - [bs, nx, ny, nz, na, 6]), - labels=anchor_localization_residuals[..., :6], - delta=1 / (3.**2)) - - # Rotation loss is computed on a transform on rot_delta. For a direction - # aware loss, we simply wrap the angles to -pi to pi; for a loss that is - # symmetric to direction (i.e., rotating by pi), we use a sin transform. - rot_delta_transform = tf.sin - if p.direction_aware_rot_loss: - rot_delta_transform = functools.partial( - geometry.WrapAngleRad, min_val=-np.pi, max_val=np.pi) - - rot_delta = ( - predicted_residuals[..., 6:] - anchor_localization_residuals[..., 6:]) - reg_rot_loss = self._utils.ScaledHuberLoss( - predictions=rot_delta_transform(rot_delta), - labels=tf.zeros_like(rot_delta), - delta=1 / (3.**2)) - - # Direction loss - if p.direction_classifier_weight > 0.0: - # The target rotations are in the assigned_gt_bbox tensor, - # which already has assigned a gt bounding box to every anchor. - rot_target = input_batch.assigned_gt_bbox[..., 6] - # If rotation is > 0, the class is 1, else it is 0. - rot_dir = tf.cast(rot_target > 0., tf.int32) - - # Compute one-hot labels as a target. - rot_dir_onehot = tf.one_hot(rot_dir, 2) - - # Manually handle loss reduction. - dir_loss = tf.losses.softmax_cross_entropy( - onehot_labels=rot_dir_onehot, - logits=predictions.predicted_dir, - weights=tf.squeeze(reg_weights, axis=-1), - reduction=tf.losses.Reduction.NONE) - # Reduce across all dimensions (we'll divide by the batch size below). - dir_loss_sum = tf.reduce_sum(dir_loss) - else: - dir_loss_sum = 0.0 - - # Compute loss contribution from location and dimension separately. - reg_loc_loss = reg_loc_and_dims_loss[..., :3] * reg_weights - reg_loc_loss_sum = tf.reduce_sum(reg_loc_loss) - - reg_dim_loss = reg_loc_and_dims_loss[..., 3:6] * reg_weights - reg_dim_loss_sum = tf.reduce_sum(reg_dim_loss) - - # Compute rotation loss contribution. - reg_rot_loss *= reg_weights - reg_rot_loss_sum = tf.reduce_sum(reg_rot_loss) - - # Num. predictions. - # TODO(zhifengc): Consider other normalization factors. E.g., # of bboxes. - preds = tf.cast(bs, class_loss_sum.dtype) - - # Normalize all of the components by batch size. - reg_loc_loss = reg_loc_loss_sum / preds - reg_dim_loss = reg_dim_loss_sum / preds - reg_rot_loss = reg_rot_loss_sum / preds - class_loss = class_loss_sum / preds - dir_loss = dir_loss_sum / preds - - # Compute total localization regression loss. - reg_loss = ( - p.location_loss_weight * reg_loc_loss + - p.dimension_loss_weight * reg_dim_loss + - p.rotation_loss_weight * reg_rot_loss) - - # Apply weights to normalized class losses. - loss = ( - class_loss * p.classification_loss_weight + - reg_loss * p.localization_loss_weight + - dir_loss * p.direction_classifier_weight) - - metrics_dict = { - 'loss': (loss, preds), - 'loss/class': (class_loss, preds), - 'loss/reg': (reg_loss, preds), - 'loss/reg/rot': (reg_rot_loss, preds), - 'loss/reg/loc': (reg_loc_loss, preds), - 'loss/reg/dim': (reg_dim_loss, preds), - 'loss/dir': (dir_loss, preds), - } - - # Calculate dimension errors - min_angle_rad = -np.pi if p.direction_aware_rot_loss else 0 - gt_bboxes = self._utils_3d.ResidualsToBBoxes( - input_batch.anchor_bboxes, - anchor_localization_residuals, - min_angle_rad=min_angle_rad, - max_angle_rad=np.pi) - predicted_bboxes = self._utils_3d.ResidualsToBBoxes( - input_batch.anchor_bboxes, - predicted_residuals, - min_angle_rad=min_angle_rad, - max_angle_rad=np.pi) - dimension_errors_dict = self._BBoxDimensionErrors(gt_bboxes, - predicted_bboxes, - reg_weights) - metrics_dict.update(dimension_errors_dict) - - per_example_dict = { - 'residuals': predicted_residuals, - 'classification_logits': predicted_class_logits, - } - - return metrics_dict, per_example_dict - - def _BBoxesAndLogits(self, input_batch, predictions): - """Decode an input batch, computing predicted bboxes from residuals.""" - p = self.params - - # Decode residuals. - min_angle_rad = -np.pi if p.direction_aware_rot_loss else 0 - predicted_bboxes = self._utils.ResidualsToBBoxes( - input_batch.anchor_bboxes, - predictions.residuals, - min_angle_rad=min_angle_rad, - max_angle_rad=np.pi) - - # predicted_bboxes is a [batch, nx, ny, nz, na, 7] Tensor. - batch_size, nx, ny, nz, na, _ = py_utils.GetShape(predicted_bboxes, 6) - num_boxes = nx * ny * nz * na - - # Reshape to [batch_size, num_boxes, 7] - predicted_bboxes = tf.reshape(predicted_bboxes, [batch_size, num_boxes, 7]) - - classification_logits = tf.reshape(predictions.classification_logits, - [batch_size, num_boxes, -1]) - - return py_utils.NestedMap({ - 'predicted_bboxes': predicted_bboxes, - 'classification_logits': classification_logits - }) diff --git a/lingvo/tasks/car/point_detector.py b/lingvo/tasks/car/point_detector.py deleted file mode 100644 index dbfbf6e3..00000000 --- a/lingvo/tasks/car/point_detector.py +++ /dev/null @@ -1,343 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Base models for point-cloud based detection.""" - -from lingvo import compat as tf -from lingvo.core import base_model -from lingvo.core import py_utils -from lingvo.tasks.car import detection_3d_lib -from lingvo.tasks.car import detection_decoder - -from lingvo.tasks.car import kitti_decoder -import numpy as np - - -class PointDetectorBase(base_model.BaseTask): - """Base class for implementing point-based detectors. - - Subclasses should implement _BBoxesAndLogits() to compute the bounding box and - scores given an input batch, and specify an appropriate decoder - implementation. - """ - - @classmethod - def Params(cls, num_classes): - p = super().Params() - # We choose a high number of boxes per example by default to bound overall - # runtime but not so low that we end up missing real boxes from complicated - # scenes. - p.Define('num_classes', num_classes, - 'The number of classes, including the background class.') - p.Define( - 'max_nms_boxes', 1024, - 'Maximum number of boxes per example to emit from non-max-suppression.') - p.Define( - 'nms_iou_threshold', 0.3, - 'NMS IoU threshold for suppressing overlapping boxes. ' - 'Can either be a float or a list of len num_classes.') - p.Define( - 'nms_score_threshold', 0.01, 'NMS threshold for scores. ' - 'Can either be a float or a list of len num_classes. ' - 'It is recommended that this be 1 for all non-active classes ' - 'like background.') - p.Define( - 'visualization_classification_threshold', 0.25, - 'Classification score threshold for determining if a prediction ' - 'is positive for the purposes of visualizations.') - p.Define('output_decoder', kitti_decoder.KITTIDecoder.Params(), - 'Implementation of decoder.') - p.Define( - 'use_oriented_per_class_nms', False, - 'Whether to use oriented per class nms or single class non-oriented.') - p.Define( - 'inference_batch_size', None, - 'If specified, hardcodes the inference batch size to this value. ' - 'Useful mostly for computing the FLOPS of a model so that the shape is ' - 'fully defined.') - p.Define( - 'decode_include_residuals', False, - 'If True, includes the residuals and ground truth anchors in the ' - 'decoder output dictionary. This can be helpful for downstream ' - 'analysis.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self._utils_3d = detection_3d_lib.Utils3D() - self.CreateChild('output_decoder', p.output_decoder) - - def CreateDecoderMetrics(self): - """Create decoder metrics.""" - return self.output_decoder.CreateDecoderMetrics() - - def _BBoxesAndLogits(self, input_batch, predictions): - """Fetch and return the bounding boxes and logits from an input. - - Args: - input_batch: The input batch from which to produce boxes and logits. - predictions: The output dictionary of ComputePredictions. - - Returns: - A .NestedMap containing - - - predicted_bboxes: A [batch_size, num_boxes, 7] floating point Tensor. - - classification_logits: A [batch_size, num_boxes, num_classes] floating - point Tensor. - """ - raise NotImplementedError('_BoxesAndLogits method not implemented.') - - def _Placeholders(self): - """Return a NestedMap of placeholders to fill in for inference. - - Runs the configured input pipeline to generate the expected shapes and types - of the inputs. - - Returns: - A NestedMap of placeholders matching the input structure of - the inference model. - """ - p = self.params - with tf.Graph().as_default(): - inputs = self.params.input.Instantiate() - - # Turn those inputs into placeholders. - placeholders = [] - for input_shape, dtype in zip(inputs.Shape().Flatten(), - inputs.DType().Flatten()): - batched_input_shape = [p.inference_batch_size] + input_shape.as_list() - placeholders.append(tf.placeholder(dtype, batched_input_shape)) - - result = inputs.DType().Pack(placeholders) - return result - - def _BBoxDimensionErrors(self, - gt_bboxes, - pred_bboxes, - regression_weights, - epsilon=1e-6): - """Calculates the errors per bounding box dimension for assigned anchors. - - Args: - gt_bboxes: float Tensor of shape [..., 7] with the ground truth bounding - box for each anchor. - pred_bboxes: float Tensor of shape [..., 7] with the predicted bounding - box for each anchor. - regression_weights: float Tensor with 0/1 weights indicating whether the - anchor had a positive assignment with same base shape as `gt_bboxes` and - `pred_bboxes` excluding the last dimension. - epsilon: A float epsilon for the denominiator of our MaskedAverage. - - Returns: - A metrics dict with mean bounding box errors for all positive assigned - anchor locations. - """ - if py_utils.GetShape(gt_bboxes)[-1] != 7: - raise ValueError('`gt_bboxes` last dimension should be 7.') - if py_utils.GetShape(pred_bboxes)[-1] != 7: - raise ValueError('`pred_bboxes` last dimension should be 7.') - batch_size = py_utils.GetShape(pred_bboxes)[0] - # Get the leading dims for later (the -1 is to exclude the last dim). - leading_dims = len(py_utils.GetShape(pred_bboxes)) - 1 - - sum_regression_weights = tf.reduce_sum(regression_weights) + epsilon - - def _MaskedAverage(value, axis=None): - return (tf.reduce_sum(value * regression_weights, axis=axis) / - sum_regression_weights) - - center_error = tf.linalg.norm( - gt_bboxes[..., :3] - pred_bboxes[..., :3], axis=-1, keepdims=True) - mean_center_error = _MaskedAverage(center_error) - - # Dimension error as shape [3] so we can get separate height, width, length - mean_dimension_error = _MaskedAverage( - gt_bboxes[..., 3:6] - pred_bboxes[..., 3:6], - axis=list(range(leading_dims))) - - # Angular error in degrees - mean_angular_error_rad = _MaskedAverage(gt_bboxes[..., 6:] - - pred_bboxes[..., 6:]) - mean_angular_error_deg = mean_angular_error_rad * (180 / np.pi) - - return py_utils.NestedMap({ - 'error/center_distance': (mean_center_error, batch_size), - 'error/length': (mean_dimension_error[0], batch_size), - 'error/width': (mean_dimension_error[1], batch_size), - 'error/height': (mean_dimension_error[2], batch_size), - 'error/rotation_deg': (mean_angular_error_deg, batch_size), - }) - - def Inference(self): - """Builds the inference graph. - - Default subgraph should return: - - predicted_bboxes: A [batch_size, num_boxes, 7] float Tensor. - - classification_scores: A [batch_size, num_boxes, num_classes] float - Tensor. - - Returns: - A dictionary whose values are a tuple of fetches and feeds. - """ - p = self.params - subgraphs = {} - with tf.name_scope('inference'): - input_placeholders = self._Placeholders() - predictions = self.ComputePredictions(self.theta, input_placeholders) - bboxes_and_logits = self._BBoxesAndLogits(input_placeholders, predictions) - predicted_bboxes = bboxes_and_logits.predicted_bboxes - classification_logits = bboxes_and_logits.classification_logits - classification_scores = tf.sigmoid(classification_logits) - - _, per_cls_bboxes, per_cls_bbox_scores, per_cls_valid_mask = ( - detection_decoder.DecodeWithNMS( - predicted_bboxes, - classification_scores, - nms_iou_threshold=p.nms_iou_threshold, - score_threshold=p.nms_score_threshold, - max_boxes_per_class=p.max_nms_boxes, - use_oriented_per_class_nms=p.use_oriented_per_class_nms)) - per_cls_bbox_scores *= per_cls_valid_mask - - # TODO(vrv): Fix the inference graph for KITTI, since we need - # to apply frustum clipping. This requires customizing the - # inference placeholders for each model. - fetches = { - 'per_class_predicted_bboxes': per_cls_bboxes, - 'per_class_predicted_bbox_scores': per_cls_bbox_scores, - 'per_class_valid_mask': per_cls_valid_mask - } - subgraphs['default'] = fetches, dict(input_placeholders.FlattenItems()) - return subgraphs - - # TODO(bencaine): Reduce code duplication between Inference/Decode. - def Decode(self, input_batch): - """Decode an input batch, computing predicted bboxes from residuals.""" - p = self.params - - predictions = self.ComputePredictions(self.theta, input_batch) - bboxes_and_logits = self._BBoxesAndLogits(input_batch, predictions) - predicted_bboxes = bboxes_and_logits.predicted_bboxes - batch_size, num_bboxes, _ = py_utils.GetShape(predicted_bboxes, 3) - classification_logits = bboxes_and_logits.classification_logits - classification_logits = py_utils.HasShape( - classification_logits, [batch_size, num_bboxes, p.num_classes]) - - classification_scores = tf.sigmoid(classification_logits) - - _, per_example_dict = self.ComputeLoss(self.theta, predictions, input_batch) - if 'score_scaler' in per_example_dict: - classification_scores *= per_example_dict['score_scaler'] - - with tf.device('/cpu:0'): - # Decode the predicted bboxes, performing NMS. - per_cls_idxs, per_cls_bboxes, per_cls_bbox_scores, per_cls_valid_mask = ( - detection_decoder.DecodeWithNMS( - predicted_bboxes, - classification_scores, - nms_iou_threshold=p.nms_iou_threshold, - score_threshold=p.nms_score_threshold, - max_boxes_per_class=p.max_nms_boxes, - use_oriented_per_class_nms=p.use_oriented_per_class_nms)) - - # per_cls_valid_mask is [batch, num_classes, num_boxes] Tensor that - # indicates which boxes were selected by NMS. Each example will have a - # different number of chosen bboxes, so the mask is present to allow us - # to keep the boxes as a batched dense Tensor. - # - # We mask the scores by the per_cls_valid_mask so that none of these boxes - # will be interpreted as valid. - per_cls_bbox_scores *= per_cls_valid_mask - visualization_weights = py_utils.HasShape( - per_cls_bbox_scores, [batch_size, p.num_classes, p.max_nms_boxes]) - - # For top down visualization, filter boxes whose scores are not above the - # visualization threshold. - visualization_weights = tf.where( - tf.greater_equal(visualization_weights, - p.visualization_classification_threshold), - visualization_weights, tf.zeros_like(visualization_weights)) - - model_outputs = py_utils.NestedMap() - model_outputs.per_class_predicted_bboxes = per_cls_bboxes - model_outputs.per_class_predicted_bbox_scores = per_cls_bbox_scores - model_outputs.per_class_valid_mask = per_cls_valid_mask - - decoder_outputs = py_utils.NestedMap({ - 'per_class_predicted_bboxes': per_cls_bboxes, - 'per_class_predicted_bbox_scores': per_cls_bbox_scores, - 'per_class_valid_mask': per_cls_valid_mask, - 'visualization_weights': visualization_weights, - }) - - if p.decode_include_residuals: - # Including the residuals in the decoder output makes it possible to save - # the outputs for further analysis. Note that we ensure that the outputs - # match the per-class NMS output format of [batch, num_classes, ...]. - def _ReshapeGather(tensor): - """Reshapes tensor and then gathers using the nms indices.""" - tensor = tf.gather( - tf.reshape(tensor, [batch_size, num_bboxes, -1]), - per_cls_idxs, - batch_dims=1) - if not p.use_oriented_per_class_nms: - # Tile so that the data fits the expected per class shape of - # [batch_size, num_classes, ...]. When *not* using oriented NMS, the - # num_classes dimension will be missing since the indices will not - # have it. - tensor = tf.tile(tensor[:, tf.newaxis, :, :], - [1, p.num_classes, 1, 1]) - return tensor - - decoder_outputs.update({ - 'per_class_gt_residuals': - _ReshapeGather(input_batch.anchor_localization_residuals), - 'per_class_gt_labels': - _ReshapeGather(input_batch.assigned_gt_labels), - 'per_class_residuals': - _ReshapeGather(predictions.residuals), - 'per_class_logits': - _ReshapeGather(predictions.classification_logits), - 'per_class_anchor_boxes': - _ReshapeGather(input_batch.anchor_bboxes), - }) - - decoder_outputs.update( - self.output_decoder.ProcessOutputs(input_batch, model_outputs)) - - # Produce global step as an output (which is the step - # of the checkpoint being decoded.) - decoder_outputs.global_step = py_utils.GetGlobalStep() - - return decoder_outputs - - def PostProcessDecodeOut(self, dec_out_dict, dec_metrics_dict): - return self.output_decoder.PostProcessDecodeOut(dec_out_dict, - dec_metrics_dict) - - def DecodeFinalize(self, decode_finalize_args): - decode_out_path = decode_finalize_args.decode_out_path - decode_out = decode_finalize_args.decode_out - - if not decode_out: - return - - # Write out a tf record file for all values in decode_out. - with tf.io.TFRecordWriter(decode_out_path) as f: - for _, v in decode_out: - f.write(v) diff --git a/lingvo/tasks/car/pointnet.py b/lingvo/tasks/car/pointnet.py deleted file mode 100644 index fe505b74..00000000 --- a/lingvo/tasks/car/pointnet.py +++ /dev/null @@ -1,236 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""PointNet architecture. - -PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation - https://arxiv.org/abs/1612.00593 -""" - -from lingvo import compat as tf -from lingvo.core import builder_layers -from lingvo.core import py_utils -from lingvo.tasks.car import builder_lib -import numpy as np - - -class PointNet(builder_lib.ModelBuilderBase): - """Builder for PointNet layers.""" - - def _ConcatWithBranch(self, name, tile, *subs): - """Concat x with subs(x).""" - - def Merge(xs): - act_self = xs[0][0] - act_branch = xs[1][0] - if tile: - num_pts = tf.shape(act_self)[1] - act_branch = tf.tile(tf.expand_dims(act_branch, 1), [1, num_pts, 1]) - return (tf.concat([act_self, act_branch], axis=-1),) - - return builder_layers.ParallelLayer.Params().Set( - name=name, - sub=[self._Seq('id'), self._Seq('branch', *subs)], - merge=Merge) - - def _ConcatWithOnehot(self, name): - """Make an onehot from shape category id and concat with point features. - - See original Pointnet paper for justification. - - Args: - name: Name of the layer. - - Returns: - Params for a layer that updates the NestedMap features value with - the input features + a one hot encoding of the category id. - """ - - def _ConcatOnehotFn(input_data): - """Concat the input features with a onehot version of the label ids.""" - features = input_data.features - label = input_data.label - num_pts = tf.shape(features)[1] - label_one_hot = tf.one_hot(tf.cast(label, tf.int32), depth=16) - label_one_hot = tf.tile(tf.expand_dims(label_one_hot, 1), [1, num_pts, 1]) - input_data.features = tf.concat([features, label_one_hot], axis=-1) - return input_data - - return self._ApplyFn(name, fn=_ConcatOnehotFn) - - def _TNet(self, name, idims): - return self._Matmul( - name, - self._Seq('iden'), - self._Seq( - 'tran', - self._MLP('mlp0', [idims, 64, 128, 1024]), - self._Max('max'), - self._MLP('mlp1', [1024, 512, 256]), - # This layer is initialized as an identity transformation. - self._Linear('linear', 256, idims * idims) - .Set(params_init=py_utils.WeightInit.Constant(scale=0)), - self._Bias('bias', idims * idims).Set( - params_init=py_utils.WeightInit.Constant( - scale=np.eye(idims).reshape([-1]).tolist())), - self._Reshape('reshape', [-1, idims, idims]), - self._Fetch('transform_matrix') - )) - - def Classifier(self, - name='pointnet', - keep_prob=0.7, - input_dims=3, - feature_dims=256): - """PointNet architecture for classification.""" - # pyformat: disable - p = self._Seq( - name, - self._SeqOnFeatures( - 'point_features', - self._TNet('inp_trans', input_dims), - self._Fetch('inp_transformed'), - self._MLP('feat', [input_dims, 64, 64]), - self._TNet('feat_trans', 64), - self._MLP('mlp1', [64, 64, 128, 1024]), - self._Fetch('pre_maxpool')), - self._PaddedMax('max'), - self._Seq( - 'cls_features', - self._Fetch('feature'), - self._FC('fc0', 1024, 512), - self._Dropout('dropout', keep_prob), - self._FC('fc1', 512, feature_dims))) - # pyformat: enable - - p.Define('output_dim', feature_dims, 'Final output dimension.') - return p - - def Segmentation(self, name='pointnet_segmentation'): - """PointNet archetecture for segmentation.""" - # dropout is NOT used in the vanilla setting from the paper. - # pyformat: disable - main_tower = self._Seq( - 'main_tower', - self._GetValue('get_features', builder_lib.FEATURES_KEY), - self._TNet('inp_trans', 3), - self._MLP('feat', [3, 64, 64]), - self._TNet('feat_trans', 64), - self._ConcatWithBranch( - 'concat1', - True, - self._MLP('mlp1', [64, 64, 128, 1024]), - self._Max('max'), - self._Fetch('feature'))) - segmentation_tower = self._Seq( - name, - main_tower, - self._Fetch('point_feature'), - self._MLP('final_mlp', [1088, 512, 256, 128])) - # pyformat: enable - p = segmentation_tower - p.Define('output_dim', 128, 'Final output dimension.') - return p - - def SegmentationShapeNet(self, name='pointnet_shapenet', keep_prob=0.8): - """The modified network for part segmentation in ShapeNet.""" - main_tower = self._SeqOnFeatures( - 'main_tower_seq', self._TNet('inp_trans', 3), self._FC('fc1', 3, 64), - self._ConcatWithBranch( - 'concat1', False, self._FC('fc2', 64, 128), - self._ConcatWithBranch( - 'concat2', False, self._FC('fc3', 128, 128), - self._ConcatWithBranch( - 'concat3', False, self._TNet('fea_trans', 128), - self._ConcatWithBranch( - 'concat4', False, self._FC('fc4', 128, 512), - self._ConcatWithBranch('concat5', True, - self._FC('fc5', 512, 2048), - self._Max('max'))))))) - p = self._Seq(name, main_tower, self._ConcatWithOnehot('concat_onehot'), - self._GetValue('get_features', builder_lib.FEATURES_KEY), - self._FC('final_fc1', 3024, 256), - self._Dropout('dropout1', keep_prob), - self._FC('final_fc2', 256, 256), - self._Dropout('dropout2', keep_prob), - self._FC('final_fc3', 256, 128)) - # pyformat: enabled - p.Define('output_dim', 128, 'Final output dimension.') - return p - - -class PointNetPP(builder_lib.ModelBuilderBase): - """Builder for PointNet++ Model.""" - - def _SetAbstractionWithMLPMax(self, - name, - mlp_dims, - num_samples, - group_size, - ball_radius, - sample_neighbors_uniformly=True): - feature_extraction_sub = self._Seq( - 'mlpmax', - self._ConcatPointsToFeatures('concat_feat_points'), - self._FeaturesMLP('sa_MLP', mlp_dims), - self._PaddedMax('sa_max_pool')) # pyformat: disable - return self._SetAbstraction( - name=name, - feature_extraction_sub=feature_extraction_sub, - num_samples=num_samples, - group_size=group_size, - ball_radius=ball_radius, - sample_neighbors_uniformly=sample_neighbors_uniformly) - - def _ModelNet40Featurizer(self, input_dims): - return self._Seq( - 'sa_featurizer', - self._SetAbstractionWithMLPMax( - name='sa0', - num_samples=512, - ball_radius=0.2, - group_size=32, - sample_neighbors_uniformly=True, - mlp_dims=[input_dims + 3, 64, 64, 128]), - self._SetAbstractionWithMLPMax( - name='sa1', - num_samples=128, - ball_radius=0.4, - group_size=32, - sample_neighbors_uniformly=True, - mlp_dims=[128 + 3, 128, 128, 256]), - self._ConcatPointsToFeatures('concat_feat_points'), - self._FeaturesMLP('sa_MLP', [256 + 3, 256, 512, 1024]), - self._PaddedMax('sa_max_pool')) - - def Classifier(self, - name='pointnetpp', - input_dims=3, - feature_dims=256, - keep_prob=0.8, - num_points=1024): - """PointNet++ architecture for modelnet40 classification.""" - - featurizer_sub = self._ModelNet40Featurizer(input_dims) - p = self._Seq( - name, - featurizer_sub, - self._FC('fc0', 1024, 512), - self._Dropout('dropout0', keep_prob), - self._FC('fc1', 512, feature_dims), - self._Dropout('dropout1', keep_prob)) # pyformat: disable - p.Define('output_dim', feature_dims, 'Final output dimension.') - - return p diff --git a/lingvo/tasks/car/pointnet_test.py b/lingvo/tasks/car/pointnet_test.py deleted file mode 100644 index 82ce01e1..00000000 --- a/lingvo/tasks/car/pointnet_test.py +++ /dev/null @@ -1,75 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for pointnet.""" - -from absl.testing import parameterized -from lingvo import compat as tf -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.tasks.car import pointnet - - -class PointNetTest(test_utils.TestCase, parameterized.TestCase): - - def _testOutShape(self, p, input_shape, expected_shape): - batch_size, num_points, _ = input_shape - g = tf.Graph() - with g.as_default(): - net = p.Instantiate() - input_data = py_utils.NestedMap( - points=tf.random.uniform((batch_size, num_points, 3)), - features=tf.random.uniform(input_shape), - padding=tf.zeros((batch_size, num_points), dtype=tf.float32), - label=tf.random.uniform((batch_size,), - minval=0, - maxval=16, - dtype=tf.int32)) - result = net.FPropDefaultTheta(input_data) - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - np_result = self.evaluate(result) - self.assertEqual(np_result.shape, expected_shape) - - @parameterized.parameters((128, 3), (128, 9), (256, 3)) - def testPointNetClassifier(self, feature_dims, input_dims): - p = pointnet.PointNet().Classifier( - input_dims=input_dims, feature_dims=feature_dims) - # Network should produce a global feature of feature_dims. - self.assertEqual(p.output_dim, feature_dims) - self._testOutShape(p, (8, 128, input_dims), (8, feature_dims)) - - def testPointNetSegmentation(self): - p = pointnet.PointNet().Segmentation() - # Network takes batch_size=8 input and produce 128-dim pointwise feature. - self.assertEqual(p.output_dim, 128) - self._testOutShape(p, (8, 100, 3), (8, 100, 128)) - - def testPointNetSegmentationShapeNet(self): - p = pointnet.PointNet().SegmentationShapeNet() - self.assertEqual(p.output_dim, 128) - self._testOutShape(p, (8, 2000, 3), (8, 2000, 128)) - - @parameterized.parameters((128, 3), (128, 9), (256, 3)) - def testPointNetPPClassifier(self, feature_dims, input_dims): - p = pointnet.PointNetPP().Classifier( - input_dims=input_dims, feature_dims=feature_dims) - # Network should produce a global feature of feature_dims. - self.assertEqual(p.output_dim, feature_dims) - self._testOutShape(p, (8, 1024, input_dims), (8, feature_dims)) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/starnet.py b/lingvo/tasks/car/starnet.py deleted file mode 100644 index 062090ad..00000000 --- a/lingvo/tasks/car/starnet.py +++ /dev/null @@ -1,909 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""StarNet: A sparse, targeted detection network. - -Point clouds and anchors are sampled from the raw points, at centers determined -by a sampler. Each center can have multiple anchors bboxes. Each center's -point cloud is featurized and then used to regress and classify the associated -anchors bboxes. - -The featurizer could be range from a simple MLPMax network, to more complex -point-based models such as PointNet++, PointCNN, PCNN, etc. - -In V1, anchor_bboxes shared the same featurization around each cell -(featurized_cell), but they had different regression/classification networks -for each offset/dimension/rotation that produced predictions. - -In V2, we bring the featurization closer to each anchor_bbox, by featurizing -at each anchor_bbox (offset) location instead. Note that anchor_bboxes at the -same offset, but with different rotation/dimension priors will have the same -featurization. -""" - -import enum -import functools -from lingvo import compat as tf -from lingvo.core import py_utils -from lingvo.tasks.car import builder_lib -from lingvo.tasks.car import detection_3d_lib -from lingvo.tasks.car import geometry -from lingvo.tasks.car import point_detector -import numpy as np - - -class Builder(builder_lib.ModelBuilderBase): - """Builder for StarNet Model.""" - - def MLPMaxFeaturizer(self, dims): - """MLP followed by Max Featurizer.""" - return self._Seq('feat', self._GetValue('get_value', 'features'), - self._MLP('mlp', dims), self._Max('max')) - - def PaddedMLPMaxFeaturizer(self, idims, dims, use_bn=True): - """MLP followed by Max on padded points.""" - - # TODO(bencaine): Add simple builder test - # Avoid batch norm in the first layer since our FC layers do BN-FC-Relu - # So the first op on the raw data would be BN, which is weird for absolute - # xyz values. - return self._Seq('feat', - self._FeaturesFC('input_fc', idims, dims[0], use_bn=False), - self._FeaturesMLP('mlp', dims, use_bn=use_bn), - self._PaddedMax('max')) - - def FC(self, name, idims, odims, use_bn=True, activation_fn=tf.nn.relu): - """Fully-connected layer.""" - return self._FC(name, idims, odims, use_bn, activation_fn) - - def Linear(self, name, idims, odims, params_init=None): - """Linear layer for predicting residuals and classification logits.""" - return self._Linear(name, idims, odims, params_init) - - def Bias(self, name, dims, params_init=None): - """Bias layer with optional initialization.""" - return self._Bias(name, dims, params_init) - - def LinearWithBias(self, - name, - idims, - odims, - linear_params_init=None, - bias_params_init=None): - """Linear with bias layer with optional initialization.""" - return self._Seq(name, - self.Linear('linear', idims, odims, linear_params_init), - self.Bias('bias', odims, bias_params_init)) - - def Atten(self, - name, - depth, - dims, - hdims, - heads, - odims, - keep_prob=1.0, - linear_params_init=None, - bias_params_init=None): - """Stacked self-attention followed with a projection.""" - return self._Seq( - name, - self._SelfAttenStack('attens', depth, dims, hdims, heads, keep_prob), - self.LinearWithBias('proj', dims, odims, linear_params_init, - bias_params_init)) - - def GINFeaturizer(self, name, fc_dims, mlp_dims, num_laser_features=1): - """GIN-based Featurizer.""" - total_input_features = 3 + num_laser_features - return self._Seq( - name, - # Dropping the cell center xyz prevents the model from over-fitting - # to the absolute coordinates. - self._SeqToKey( - 'drop_cell_center_xyz', 'features', - self._GetValue('get_features', 'features'), - self._ApplyFn('drop_cell_center_xyz', fn=lambda t: t[..., 3:])), - self._FeaturesFC('fc0', total_input_features, fc_dims), - self._GIN( - 'gin', - mlp_dims, - aggregate_sub=self._PaddedMax('p_max'), - readout_sub=self._PaddedMean('p_mean'), - combine_method='concat')) # pyformat: disable - - def GINFeaturizerV2(self, - name, - fc_dims, - mlp_dims, - num_laser_features=1, - fc_use_bn=True): - """GIN-based Featurizer for Model V2.""" - total_input_features = 3 + num_laser_features - return self._Seq( - name, - self._FeaturesFC( - 'fc0', total_input_features, fc_dims, use_bn=fc_use_bn), - self._GIN( - 'gin', - mlp_dims, - aggregate_sub=self._PaddedMax('p_max'), - readout_sub=self._PaddedMean('p_mean'), - combine_method='concat')) # pyformat: disable - - def ZerosCellFeaturizer(self, name, dims): - """Produces features with 0 values, used for disabling cell featurizer.""" - - def _ZerosFeature(nested_data): - # Points has shape [batch_size, num_anchors, points_per_anchor, 3]. - # The featurizer should produce features for each anchor. - shape = py_utils.GetShape(nested_data.points) - return tf.zeros(shape[:-2] + [dims]) - - return self._ApplyFn(name, fn=_ZerosFeature) - - -class LossNormType(enum.Enum): - NO_NORM = 0 - NORM_BY_NUM_POS_PER_CENTER = 1 - - -class ModelBase(point_detector.PointDetectorBase): - """StarNet Detection Model. - - This model expects that input_batch contains: - cell_center_xyz: [N, C, 3] - cell_points_xyz: [N, C, P, 3] - cell_feature: [N, C, P, 1] - anchor_bboxes: [N, C, B, 7] - anchor_localization_residuals: [N, C, B, 7] - assigned_gt_labels: [N, C, B] - assigned_cls_mask: [N, C, B] - - where: - N - batch size - C - num centers - P - num points per center - B - num anchor bboxes per center - - The centers for the anchor_bboxes should match those of cell_center_xyz. - Specifically, num_anchor_bboxes_per_center should match that of the - corresponding input generator. - - Base class implements common Decoder functions, though they can be - overridden if desired. - - Sub-classes are expected to implement ComputePredictions. - """ - - @classmethod - def Params(cls, - num_classes, - num_anchor_bboxes_per_center, - num_laser_features=1): - p = super().Params(num_classes=num_classes) - p.Define( - 'num_anchor_bboxes_per_center', num_anchor_bboxes_per_center, - 'The number of anchor bboxes per center. This should match that ' - 'of the corresponding input generator.') - - p.Define('focal_loss_alpha', 0.25, - 'Alpha parameter for focal loss for classification.') - p.Define('focal_loss_gamma', 2.0, - 'Gamma parameter for focal loss for classification.') - - p.Define('huber_loss_delta', 1. / (3.**2), - 'delta threshold for scaled huber loss.') - p.Define('loss_weight_localization', 2.0, - 'Weighting factor for localization loss.') - p.Define('loss_weight_classification', 1.0, - 'Weighting factor for classification loss.') - - p.Define('loss_norm_type', None, - 'Normalization type for class and regularization weights.') - - p.Define( - 'squash_rotation_predictions', False, - 'Apply tanh squashing to rotation predictions to ensure outputs ' - 'are between (-pi, pi).') - p.Define( - 'direction_aware_rot_loss', False, 'If True, changes the heading loss ' - 'from sin(theta_delta) to WrapAngleRad(theta_delta), which makes the ' - 'model produce headings between [-pi to pi].') - - p.Define( - 'location_loss_weight', 1.0, - 'Weight multiplier for contribution of location loss ' - 'to full localization/regression loss') - p.Define( - 'dimension_loss_weight', 1.0, - 'Weight multiplier for contribution of dimension loss ' - 'to full localization/regression loss') - p.Define( - 'rotation_loss_weight', 1.0, - 'Weight multiplier for contribution of rotation loss ' - 'to full localization/regression loss') - p.Define( - 'corner_loss_weight', 0.0, - 'Weight multiplier for contribution of corner loss ' - 'to full localization/regression loss') - p.Define( - 'per_class_loss_weight', [0.] + [1.] * (num_classes - 1), - 'A list with a float value per class with a multiple to multiply ' - 'the classification losses of that classes anchors by. Note that ' - 'the background class is always class 0, and should be assigned a ' - 'weight of 0.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self._utils = detection_3d_lib.Utils3D() - - if len(p.per_class_loss_weight) != p.num_classes: - raise ValueError('`Need `per_class_loss_weight` to be of len equal ' - 'to the number of classes.') - if p.per_class_loss_weight[0] != 0.0: - raise ValueError('Background class should be assigned 0 weight. ' - 'per_class_loss_weight={}'.format( - str(p.per_class_loss_weight))) - - def ComputeLoss(self, theta, predictions, input_batch): - """Compute loss for the sparse detector model v1. - - Args: - theta: A `.NestedMap` object containing variable values of this task. - predictions: A `.NestedMap` object containing residuals and - classification_logits. - input_batch: A `.NestedMap` expected to contain cell_center_xyz, - cell_points_xyz, cell_feature, anchor_bboxes, - anchor_localization_residuals, assigned_gt_labels, and - assigned_cls_mask. See class doc string for details. - - Returns: - Two dicts: - - - A dict containing str keys and (metric, weight) pairs as values, where - one of the keys is expected to be 'loss'. - - A dict containing arbitrary tensors describing something about each - training example, where the first dimension of each tensor is the batch - index. - """ - p = self.params - - batch_size, num_centers = py_utils.GetShape(input_batch.cell_center_xyz, 2) - - # Assert shapes of inputs. - anchor_bboxes = py_utils.HasShape( - input_batch.anchor_bboxes, - [batch_size, num_centers, p.num_anchor_bboxes_per_center, 7]) - anchor_localization_residuals = py_utils.HasShape( - input_batch.anchor_localization_residuals, - [batch_size, num_centers, p.num_anchor_bboxes_per_center, 7]) - predicted_residuals = py_utils.HasShape( - predictions.residuals, - [batch_size, num_centers, p.num_anchor_bboxes_per_center, 7]) - - assigned_gt_labels = py_utils.HasShape( - input_batch.assigned_gt_labels, - [batch_size, num_centers, p.num_anchor_bboxes_per_center]) - predicted_classification_logits = py_utils.HasShape( - predictions.classification_logits, [ - batch_size, num_centers, p.num_anchor_bboxes_per_center, - p.num_classes - ]) - - # assigned_cls_mask is for weighting the classification loss. - # Ignored targets will have their mask = 0; this happens when their IOU is - # not high enough to be a foreground object and not low enough to be - # background. - class_weights = py_utils.HasShape( - input_batch.assigned_cls_mask, - [batch_size, num_centers, p.num_anchor_bboxes_per_center]) - class_weights = tf.reshape( - class_weights, - [batch_size, num_centers, p.num_anchor_bboxes_per_center, 1]) - - # Broadcast per class loss weights. For each anchor, there are num_classes - # prediction heads, we weight the outputs of these heads by the per class - # loss weights. - per_class_loss_weight = tf.constant([[[p.per_class_loss_weight]]], - dtype=tf.float32) - per_class_loss_weight = py_utils.HasShape(per_class_loss_weight, - [1, 1, 1, p.num_classes]) - class_weights *= per_class_loss_weight - class_weights = py_utils.HasShape(class_weights, [ - batch_size, num_centers, p.num_anchor_bboxes_per_center, p.num_classes - ]) - - # We use assigned_reg_mask for masking the regression loss. - # Only foreground objects will have assigned_reg_mask = 1. - reg_weights = py_utils.HasShape( - input_batch.assigned_reg_mask, - [batch_size, num_centers, p.num_anchor_bboxes_per_center]) - reg_weights = tf.reshape( - reg_weights, - [batch_size, num_centers, p.num_anchor_bboxes_per_center, 1]) - - if p.loss_norm_type == LossNormType.NORM_BY_NUM_POS_PER_CENTER: - # Compute number of positive anchors per example. - foreground_mask = py_utils.HasShape( - input_batch.assigned_reg_mask, - [batch_size, num_centers, p.num_anchor_bboxes_per_center]) - - # Sum to get the number of foreground anchors for each example. - loss_normalization = tf.reduce_sum(foreground_mask, axis=2) - loss_normalization = tf.maximum(loss_normalization, - tf.ones_like(loss_normalization)) - - # Reshape for broadcasting. - loss_normalization = tf.reshape(loss_normalization, - [batch_size, num_centers, 1, 1]) - - # Normalize so that the loss is independent of # centers. - loss_normalization *= num_centers - class_weights /= loss_normalization - reg_weights /= loss_normalization - - classification_loss = py_utils.SigmoidCrossEntropyFocalLoss( - logits=predicted_classification_logits, - labels=tf.one_hot(assigned_gt_labels, p.num_classes), - alpha=p.focal_loss_alpha, - gamma=p.focal_loss_gamma) - - # Apply mask. - classification_loss *= class_weights - - # TODO(jngiam): Consider normalizing by num_foreground_anchors for each - # example instead. This would match the 1/N_positive normalization in - # point pillars. - - # Reduce sum over centers, boxes and classes. - classification_loss = tf.reduce_sum(classification_loss, axis=[1, 2, 3]) - - # Reduce mean over batch. - classification_loss = tf.reduce_mean(classification_loss) - - # Localization regression loss with Huber loss (SmoothL1). - regression_loc_and_dims_loss = self._utils_3d.ScaledHuberLoss( - labels=anchor_localization_residuals[..., :6], - predictions=predicted_residuals[..., :6], - delta=p.huber_loss_delta) - - # Rotation loss is computed on a transform on rotation_delta. For a - # direction aware loss, we simply wrap the angles to -pi to pi; for a loss - # that is symmetric to direction (i.e., rotating by pi), we use a sin - # transform. - rotation_delta_transform = tf.sin - if p.direction_aware_rot_loss: - rotation_delta_transform = functools.partial( - geometry.WrapAngleRad, min_val=-np.pi, max_val=np.pi) - rotation_delta = ( - predicted_residuals[..., 6:] - anchor_localization_residuals[..., 6:]) - regression_rotation_loss = self._utils_3d.ScaledHuberLoss( - labels=tf.zeros_like(rotation_delta), - predictions=rotation_delta_transform(rotation_delta), - delta=p.huber_loss_delta) - - reg_loc_loss = regression_loc_and_dims_loss[..., :3] - reg_dim_loss = regression_loc_and_dims_loss[..., 3:6] - - gt_bboxes = self._utils_3d.ResidualsToBBoxes( - anchor_bboxes, - anchor_localization_residuals, - min_angle_rad=-np.pi, - max_angle_rad=np.pi) - predicted_bboxes = self._utils_3d.ResidualsToBBoxes( - anchor_bboxes, - predicted_residuals, - min_angle_rad=-np.pi, - max_angle_rad=np.pi) - - # Apply mask to individual losses. - # - # And then reduce sum over centers, boxes, residuals, and batch - # and divide by the batch_size. - regression_rotation_loss *= reg_weights - reg_rot_loss = tf.reduce_sum(regression_rotation_loss) / batch_size - - reg_loc_loss *= reg_weights - reg_loc_loss = tf.reduce_sum(reg_loc_loss) / batch_size - - reg_dim_loss *= reg_weights - reg_dim_loss = tf.reduce_sum(reg_dim_loss) / batch_size - - # Do not create corner loss graph if weight is 0.0 - # TODO(bcyang): Remove condition after fixing corner loss NaN issue - if p.corner_loss_weight != 0.0: - reg_corner_loss = self._utils_3d.CornerLoss( - gt_bboxes=gt_bboxes, predicted_bboxes=predicted_bboxes) - reg_corner_loss = tf.expand_dims(reg_corner_loss, axis=-1) - - reg_corner_loss *= reg_weights - reg_corner_loss = tf.reduce_sum(reg_corner_loss) / batch_size - else: - reg_corner_loss = 0.0 - - # Sum components of regression loss. - regression_loss = ( - p.location_loss_weight * reg_loc_loss + - p.dimension_loss_weight * reg_dim_loss + - p.rotation_loss_weight * reg_rot_loss + - p.corner_loss_weight * reg_corner_loss) - - # Compute total loss. - total_loss = ( - p.loss_weight_localization * regression_loss + - p.loss_weight_classification * classification_loss) - - metrics_dict = py_utils.NestedMap({ - 'loss': (total_loss, batch_size), - 'loss/regression': (regression_loss, batch_size), - 'loss/regression/loc': (reg_loc_loss, batch_size), - 'loss/regression/dim': (reg_dim_loss, batch_size), - 'loss/regression/rot': (reg_rot_loss, batch_size), - 'loss/regression/corner': (reg_corner_loss, batch_size), - 'loss/classification': (classification_loss, batch_size), - }) - - # Calculate dimension errors - dimension_errors_dict = self._BBoxDimensionErrors(gt_bboxes, - predicted_bboxes, - reg_weights) - metrics_dict.update(dimension_errors_dict) - - per_example_dict = py_utils.NestedMap({ - 'residuals': predicted_residuals, - 'classification_logits': predicted_classification_logits, - 'predicted_bboxes': predicted_bboxes, - 'gt_bboxes': gt_bboxes, - 'reg_weights': reg_weights, - }) - - return metrics_dict, per_example_dict - - def ComputePredictions(self, theta, input_batch): - """Computes predictions for `input_batch`. - - Args: - theta: A `.NestedMap` object containing variable values of this task. - input_batch: A `.NestedMap` expected to contain cell_center_xyz, - cell_points_xyz, cell_feature, anchor_bboxes, - anchor_localization_residuals, assigned_gt_labels, and - assigned_cls_mask. See class doc string for details. - - Returns: - A `.NestedMap` object containing residuals and classification_logits. - """ - raise NotImplementedError('Abstract method: %s' % type(self)) - - def _BBoxesAndLogits(self, input_batch, predictions): - """Decode an input batch, computing predicted bboxes from residuals.""" - batch_size, num_centers, num_predictions_per_center = py_utils.GetShape( - predictions.residuals, 3) - - # Decode residuals. - predicted_bboxes = self._utils_3d.ResidualsToBBoxes( - input_batch.anchor_bboxes, - predictions.residuals, - min_angle_rad=-np.pi, - max_angle_rad=np.pi) - - # Reshape to [batch_size, num_centers * num_predictions_per_center, ...] - num_predicted_boxes = num_centers * num_predictions_per_center - predicted_bboxes = tf.reshape(predicted_bboxes, - [batch_size, num_predicted_boxes, -1]) - - classification_logits = tf.reshape(predictions.classification_logits, - [batch_size, num_predicted_boxes, -1]) - - return py_utils.NestedMap({ - 'predicted_bboxes': predicted_bboxes, - 'classification_logits': classification_logits, - }) - - -class ModelV1(ModelBase): - """StarNet Model V1. - - In this model, each center is first featurized into a single feature vector, - which is then used to predict the residuals for all the bboxes at that center. - Concretely, each featurized center needs to make B * (7 + num_classes) - predictions. - - Effectively, anchor_bboxes shared the same featurization around each cell - (featurized_cell), but have different regression/classification networks - for each offset/dimension/rotation that produced predictions. - """ - - @classmethod - def Params(cls, - num_classes, - num_anchor_bboxes_per_center, - num_laser_features=1): - - p = super().Params( - num_classes=num_classes, - num_anchor_bboxes_per_center=num_anchor_bboxes_per_center, - num_laser_features=num_laser_features) - - builder = Builder() - final_feature_dims = 1024 - # The first 7 here corresponds to the concatenation of the following - # features: - # x, y, z of the center. - # x, y, z of each point relative to the center. - # features of the point. - num_features = 6 + num_laser_features - p.Define( - 'cell_featurizer', - builder.MLPMaxFeaturizer([num_features, 32, 256, final_feature_dims]), - 'Point cloud feature extractor.') - - p.Define( - 'localization_regressor', - builder.Linear('localization', final_feature_dims, - num_anchor_bboxes_per_center * 7), - 'Localization residual regressor.') - p.Define( - 'classifier', - builder.Linear('classification', final_feature_dims, - num_anchor_bboxes_per_center * num_classes), - 'Classification layer (producing logits).') - - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - self.CreateChild('cell_featurizer', p.cell_featurizer) - self.CreateChild('localization_regressor', p.localization_regressor) - self.CreateChild('classifier', p.classifier) - - def ComputePredictions(self, theta, input_batch): - """Computes predictions for `input_batch`. - - Args: - theta: A `.NestedMap` object containing variable values of this task. - input_batch: A `.NestedMap` expected to contain cell_center_xyz, - cell_points_xyz, cell_feature, anchor_bboxes, - anchor_localization_residuals, assigned_gt_labels, and - assigned_cls_mask. See class doc string for details. - - Returns: - A `.NestedMap` object containing residuals and classification_logits. - """ - p = self.params - input_batch.Transform(lambda x: (x.shape, x.shape.num_elements())).VLog( - 1, 'input_batch shapes: ') - - cell_feature = py_utils.HasRank(input_batch.cell_feature, 4) - batch_size, num_centers, num_points_per_cell = py_utils.GetShape( - cell_feature, 3) - - cell_points_xyz = py_utils.HasShape( - input_batch.cell_points_xyz, - [batch_size, num_centers, num_points_per_cell, 3]) - cell_center_xyz = py_utils.HasShape(input_batch.cell_center_xyz, - [batch_size, num_centers, 3]) - - cell_points_padding = py_utils.HasShape( - input_batch.cell_points_padding, - [batch_size, num_centers, num_points_per_cell]) - - # TODO(jngiam): Make concat_feature computation a layer or configureable. - cell_center_xyz = tf.reshape(cell_center_xyz, - [batch_size, num_centers, 1, 3]) - centered_cell_points_xyz = cell_points_xyz - cell_center_xyz - concat_feature = tf.concat([ - tf.tile(cell_center_xyz, [1, 1, num_points_per_cell, 1]), - centered_cell_points_xyz, cell_feature - ], axis=-1) # pyformat: disable - - # Featurize point clouds at each center. - point_input = py_utils.NestedMap({ - 'points': centered_cell_points_xyz, - 'features': concat_feature, - 'padding': cell_points_padding, - }) - featurized_cell = self.cell_featurizer.FProp(theta.cell_featurizer, - point_input) - featurized_cell = py_utils.HasShape(featurized_cell, - [batch_size, num_centers, -1]) - - # Predict localization residuals. - predicted_residuals = self.localization_regressor.FProp( - theta.localization_regressor, featurized_cell) - predicted_residuals = tf.reshape( - predicted_residuals, - [batch_size, num_centers, p.num_anchor_bboxes_per_center, 7]) - - if p.squash_rotation_predictions: - predicted_rotations = predicted_residuals[..., 6:] - predicted_rotations = np.pi * tf.tanh(predicted_rotations) - predicted_residuals = tf.concat( - [predicted_residuals[..., :6], predicted_rotations], axis=-1) - - # Predict object classification at each bbox. - predicted_classification_logits = self.classifier.FProp( - theta.classifier, featurized_cell) - predicted_classification_logits = tf.reshape( - predicted_classification_logits, [ - batch_size, num_centers, p.num_anchor_bboxes_per_center, - p.num_classes - ]) - - return py_utils.NestedMap({ - 'residuals': predicted_residuals, - 'classification_logits': predicted_classification_logits, - }) - - -class ModelV2(ModelBase): - """StarNet Model V2. - - This model is similar to V1 except that featurizations are computed at the - location of each anchor_bbox instead of each cell center. - - In V2, we don't share the featurization among anchor_bboxes in the same cell. - Instead, the model featurizes at each anchor_bbox (offset) location. - Note that anchor_bboxes at the same offset, but with different - rotation/dimension priors will have the same featurization. - - Note: This model makes assumptions about the ordering of anchor_bboxes, and - assumes that offsets correspond to the 'outer dimensions'; see - input_generator._AnchorBoxSettings.GenerateAnchorSettings for details. - - Note that the different rotation/dimension settings will have their own - classification/regression heads - and these are now shared across all offsets. - - In summary, we want the featurizer to be location (xyz coordinate) specific, - and at each location, we want to have rotation/dimension specific regressors. - This makes it more natural to leverage featurizers that produce features - at these anchor-offset specific locations. For example, one could apply a - PCNN/PointConv model on the entire point cloud (instead of each cell) and - produce features at given anchor locations. - """ - - @classmethod - def Params(cls, - num_classes, - num_anchor_bboxes_offsets, - num_anchor_bboxes_rotations, - num_anchor_bboxes_dimensions, - num_laser_features=1): - num_anchor_bboxes_per_center = ( - num_anchor_bboxes_offsets * num_anchor_bboxes_rotations * - num_anchor_bboxes_dimensions) - p = super().Params(num_classes, num_anchor_bboxes_per_center, - num_laser_features) - - # Good defaults from V1 tuning. - p.loss_norm_type = LossNormType.NORM_BY_NUM_POS_PER_CENTER - - # V2 specifics below. - p.Define( - 'num_anchor_bboxes_offsets', num_anchor_bboxes_offsets, - 'The number of anchor bboxes offsets per center. ' - 'This should match that of the corresponding input generator.') - p.Define( - 'num_anchor_bboxes_rotations', num_anchor_bboxes_rotations, - 'The number of anchor bboxes rotations per center. ' - 'This should match that of the corresponding input generator.') - p.Define( - 'num_anchor_bboxes_dimensions', num_anchor_bboxes_dimensions, - 'The number of anchor bboxes dimensions per center. ' - 'This should match that of the corresponding input generator.') - - builder = Builder() - builder.linear_params_init = py_utils.WeightInit.KaimingUniformFanInRelu() - - # In Model V2, the inputs to the cell featurizer only include the relative - # coordinates of each point concatenated with the point's features. - # GINFeaturizerV2 takes that into account and expects input_dims=4. - num_gin_layers = 5 - gin_hidden_dims = 128 - gin_layers = [ - [gin_hidden_dims * 2, gin_hidden_dims * 4, gin_hidden_dims] - ] * num_gin_layers # pyformat: disable - - # Models can specify different cell featurizers as long as cell_feature_dims - # is also set correctly. - p.Define( - 'cell_feature_dims', gin_hidden_dims * (num_gin_layers + 1), - 'Dimensions of the features produced by the cell featurizer. ' - 'This should match the output of the cell_featurizer.') - p.Define( - 'cell_featurizer', - builder.GINFeaturizerV2( - 'feat', - gin_hidden_dims, - gin_layers, - num_laser_features=num_laser_features), - 'Point cloud local cell feature extractor.') - - # The cell_feature_projector layer projects the featurized cells to each - # offset location. This is mainly to keep compatibility with V1. Note that - # this may be removed once we find a better global featurizer. - # A FC layer will automatically be created at init that projects the - # cell features to each anchor location with this dims. - p.Define( - 'anchor_projected_feature_dims', 128, 'Dimensions of projected ' - 'features from cell featurizer. A FC layer will be used to ' - 'project the cell features.') - - p.Define('oracle_location', False, - 'If true, the model predicts the ground truth for location.') - p.Define('oracle_dimension', False, - 'If true, the model predicts the ground truth for dimension.') - p.Define('oracle_rotation', False, - 'If true, the model predicts the ground truth for rotation.') - p.Define( - 'oracle_classification', False, - 'If true, the model predicts the ground truth for classification.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - builder = Builder() - builder.linear_params_init = py_utils.WeightInit.KaimingUniformFanInRelu() - - num_anchors_per_offset = ( - p.num_anchor_bboxes_rotations * p.num_anchor_bboxes_dimensions) - - # Create params for fixed sub layers. - cell_feature_projector_p = builder.FC( - 'cell_feature_projector', p.cell_feature_dims, - p.num_anchor_bboxes_offsets * p.anchor_projected_feature_dims) - - localization_regressor_p = builder.Linear('localization_regressor', - p.anchor_projected_feature_dims, - num_anchors_per_offset * 7, - py_utils.WeightInit.Constant(0.)) - - classifier_p = builder.LinearWithBias( - 'classifier', - p.anchor_projected_feature_dims, - num_anchors_per_offset * p.num_classes, - bias_params_init=py_utils.WeightInit.Constant(-4.595)) - - self.CreateChild('cell_featurizer', p.cell_featurizer) - self.CreateChild('cell_feature_projector', cell_feature_projector_p) - self.CreateChild('localization_regressor', localization_regressor_p) - self.CreateChild('classifier', classifier_p) - - def _CellFeaturizer(self, theta, input_batch): - """Featurizes each center location.""" - # Validate Shapes - cell_feature = py_utils.HasRank(input_batch.cell_feature, 4) - batch_size, num_centers, num_points_per_cell = py_utils.GetShape( - cell_feature, 3) - - cell_points_xyz = py_utils.HasShape( - input_batch.cell_points_xyz, - [batch_size, num_centers, num_points_per_cell, 3]) - cell_center_xyz = py_utils.HasShape(input_batch.cell_center_xyz, - [batch_size, num_centers, 3]) - - cell_points_padding = py_utils.HasShape( - input_batch.cell_points_padding, - [batch_size, num_centers, num_points_per_cell]) - - # Center each cell - cell_center_xyz = tf.reshape(cell_center_xyz, - [batch_size, num_centers, 1, 3]) - centered_cell_points_xyz = cell_points_xyz - cell_center_xyz - concat_feature = tf.concat([ - centered_cell_points_xyz, cell_feature - ], axis=-1) # pyformat: disable - - # Featurize point clouds at each center. - point_input = py_utils.NestedMap({ - 'points': centered_cell_points_xyz, - 'features': concat_feature, - 'padding': cell_points_padding, - }) - featurized_cell = self.cell_featurizer.FProp(theta.cell_featurizer, - point_input) - featurized_cell = py_utils.HasShape(featurized_cell, - [batch_size, num_centers, -1]) - return featurized_cell - - def ComputePredictions(self, theta, input_batch): - """Computes predictions for `input_batch`. - - Args: - theta: A `.NestedMap` object containing variable values of this task. - input_batch: A `.NestedMap` expected to contain lasers.points_xyz, - lasers.points_feature, lasers.points_padding, cell_center_xyz, - cell_points_xyz, cell_feature, anchor_bboxes, - anchor_localization_residuals, assigned_gt_labels, and - assigned_cls_mask. See class doc string for details. - - Returns: - A `.NestedMap` object containing residuals and classification_logits. - """ - p = self.params - input_batch.Transform(lambda x: (x.shape, x.shape.num_elements())).VLog( - 1, 'input_batch shapes: ') - cell_feature = py_utils.HasRank(input_batch.cell_feature, 4) - batch_size, num_centers = py_utils.GetShape(cell_feature, 2) - - featurized_cell = self._CellFeaturizer(theta, input_batch) - - # Project each featurized_cell features to each bbox per center. - featurized_anchors = self.cell_feature_projector.FProp( - theta.cell_feature_projector, featurized_cell) - - # Reshape output so that we have features per offset. - featurized_anchors = tf.reshape( - featurized_anchors, - [batch_size, num_centers, p.num_anchor_bboxes_offsets, -1]) - - # Predict localization residuals. - predicted_residuals = self.localization_regressor.FProp( - theta.localization_regressor, featurized_anchors) - predicted_residuals = tf.reshape( - predicted_residuals, - [batch_size, num_centers, p.num_anchor_bboxes_per_center, 7]) - - if any([p.oracle_location, p.oracle_dimension, p.oracle_rotation]): - gt_residuals = py_utils.HasShape( - input_batch.anchor_localization_residuals, - [batch_size, num_centers, p.num_anchor_bboxes_per_center, 7]) - residuals = [] - if p.oracle_location: - residuals.append(gt_residuals[..., 0:3]) - else: - residuals.append(predicted_residuals[..., 0:3]) - - if p.oracle_dimension: - residuals.append(gt_residuals[..., 3:6]) - else: - residuals.append(predicted_residuals[..., 3:6]) - - if p.oracle_rotation: - residuals.append(gt_residuals[..., 6:]) - else: - residuals.append(predicted_residuals[..., 6:]) - predicted_residuals = tf.concat(residuals, axis=-1) - - if p.squash_rotation_predictions: - predicted_rotations = predicted_residuals[..., 6:] - predicted_rotations = np.pi * tf.tanh(predicted_rotations) - predicted_residuals = tf.concat( - [predicted_residuals[..., :6], predicted_rotations], axis=-1) - - # Predict object classification at each bbox. - predicted_classification_logits = self.classifier.FProp( - theta.classifier, featurized_anchors) - predicted_classification_logits = tf.reshape( - predicted_classification_logits, [ - batch_size, num_centers, p.num_anchor_bboxes_per_center, - p.num_classes - ]) - - if p.oracle_classification: - assigned_gt_labels = py_utils.HasShape( - input_batch.assigned_gt_labels, - [batch_size, num_centers, p.num_anchor_bboxes_per_center]) - predicted_classification_logits = tf.one_hot(assigned_gt_labels, - p.num_classes) - - return py_utils.NestedMap({ - 'residuals': predicted_residuals, - 'classification_logits': predicted_classification_logits, - }) diff --git a/lingvo/tasks/car/summary.py b/lingvo/tasks/car/summary.py deleted file mode 100644 index 9ce333cb..00000000 --- a/lingvo/tasks/car/summary.py +++ /dev/null @@ -1,594 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Functions to help with plotting summaries. - -These functions try to take care in ensuring that transformations on points and -headings are consistent. For example, when points are transformed from the -standard x/y plane (+x going right, +y going up) to an alternate formulation -such as 'flipped axes', the heading must also be updated to account for the -change in axis directions. -""" - -import collections -import math - -from lingvo import compat as tf -from lingvo.core import plot -from lingvo.tasks.car import transform_util -import matplotlib.patheffects as path_effects -import numpy as np - -import PIL.Image as Image -import PIL.ImageColor as ImageColor -import PIL.ImageDraw as ImageDraw -import PIL.ImageFont as ImageFont - - -# Source code for keys can be found at: -# https://github.com/python-pillow/Pillow/blob/080bfd3ee1412b401d520fe26c51e2f5515e3a65/src/PIL/ImageColor.py#L163 -def _PILColorList(): - pil_color_list = sorted(ImageColor.colormap.keys()) - pil_color_list.remove('black') - return pil_color_list - - -PIL_COLOR_LIST = _PILColorList() - - -def ExtractRunIds(run_segments): - """Extract the RunIds from the run_segments feature field. - - Args: - run_segments: a string Tensor of shape [batch, 1] containing a text proto. - - See `SummaryTest.testExtractRunIds` for an example. - - Returns: - A string Tensor of shape [batch], containing the extracted run id. - """ - run_segments = tf.convert_to_tensor(run_segments)[:, 0] - return tf.strings.regex_replace(run_segments, - r'[^:]+: "(.+)"\n[^:]+: (\d+)(.|\n)*', - r'\1_\2') - - -def CameraImageSummary(frontal_images, run_segment_strings, figsize=(6, 4)): - """Write frontal_images as tf.Summaries. - - Args: - frontal_images: Float tensor of frontal camera images: Shape: [batch, - height, width, depth]. Expected aspect ratio of 3:2 for visualization. - run_segment_strings: Tensor of strings: Shape: [batch, 1]. The associated - RunSegment proto for the batch. - figsize: Tuple indicating size of camera image. Default is (6, 4) - indicating a 3:2 aspect ratio for visualization. - """ - # Parse the run segment strings to extract the run segment info. - run_segment_ids = ExtractRunIds(run_segment_strings) - - def DrawCameraImage(fig, axes, frontal_image, run_segment_id): - """Draw camera image for image summary.""" - plot.AddImage( - fig=fig, - axes=axes, - data=frontal_image / 256., - show_colorbar=False, - suppress_xticks=True, - suppress_yticks=True) - txt = axes.text( - x=0.5, - y=0.01, - s=run_segment_id, - color='blue', - fontsize=14, - transform=axes.transAxes, - horizontalalignment='center') - txt.set_path_effects([ - path_effects.Stroke(linewidth=3, foreground='lightblue'), - path_effects.Normal() - ]) - - with plot.MatplotlibFigureSummary( - 'examples', figsize=figsize, max_outputs=10) as fig: - # Plot raw frontal image samples for each example. - fig.AddSubplot([frontal_images, run_segment_ids], DrawCameraImage) - - -def _CarToImageTransform(): - return transform_util.MakeCarToImageTransform( - pixels_per_meter=10., image_ref_x=250, image_ref_y=750, flip_axes=True) - - -def DrawTopDown(lasers): - """Draw the laser points in the top down car view.""" - # For every laser point, convert the point to a top down image. - - # Lasers is a [r, b, s*6] tensor where b is the batch size. - # Transpose back to batch major - lasers = np.transpose(lasers, [1, 0, 2]) - # Reshape data to get all points in the spin. - lasers = np.reshape(lasers, [np.shape(lasers)[0], -1]) - - car_to_image_transform = _CarToImageTransform() - - # Create an empty image of the appropriate size. - batch_size = min(8, np.shape(lasers)[0]) - images = np.zeros(shape=(batch_size, 1000, 500, 3), dtype=np.uint8) - - # TODO(vrv): Slice the lasers into [b, x, y, z] matrix - # and then do a batch_matmul on all points at once. - max_npoints = np.shape(lasers)[1] // 6 - for b in range(batch_size): - for i in range(max_npoints): - index = i * 6 - x = lasers[b][index] - y = lasers[b][index + 1] - z = lasers[b][index + 2] - - # TODO(vrv): Use lasers_padding to filter out invalid points. - # For now, just assume that all zeros means that the point - # shouldn't be drawn. - if x == 0 and y == 0 and z == 0: - continue - - tx, ty, _ = transform_util.TransformPoint(car_to_image_transform, x, y, z) - # Point outside image. - if tx < 0 or ty < 0 or tx >= images.shape[2] or ty >= images.shape[1]: - continue - - # Fill in that point in the image - images[b, int(ty), int(tx), :] = (255, 255, 255) - - return images - - -def MakeRectangle(l, w, theta, offset=(0, 0)): - """Make rotated rectangle.""" - c, s = math.cos(theta), math.sin(theta) - coordinates = [(l / 2.0, w / 2.0), (l / 2.0, -w / 2.0), (-l / 2.0, -w / 2.0), - (-l / 2.0, w / 2.0)] - return [(c * x - s * y + offset[0], s * x + c * y + offset[1]) - for (x, y) in coordinates] - - -def DrawHeadingTriangle(draw, x, y, heading, color, scale=25): - """Draw a triangle indicating `heading` at `x`, `y` with `color`.""" - ch = scale * math.cos(heading) - sh = scale * math.sin(heading) - lead_point = (x - ch, y - sh) - anchor_1 = (x - sh / 3, y + ch / 3) - anchor_2 = (x + sh / 3, y - ch / 3) - draw.line([anchor_1, lead_point, anchor_2], fill=color, width=4) - - -def DrawCircle(draw, x, y, fill, outline, circle_size=5): - """Draw a circle at `x`, `y`.""" - draw.ellipse( - (x - circle_size, y - circle_size, x + circle_size, y + circle_size), - fill=fill, - outline=outline) - - -# TODO(vrv): Support multiple display strings. -def DrawBoundingBoxOnImage(image, - box, - display_str, - color='red', - thickness=4, - text_loc='BOTTOM'): - """Draw bounding box on the input image.""" - original_image = image - image = Image.fromarray(np.uint8(original_image)).convert('RGB') - draw = ImageDraw.Draw(image) - - center_x, center_y, width, height, heading = box - box2d = transform_util.Box2D(center_x, center_y, width, height, heading) - corners = list(box2d.corners.reshape(-1, 2)) - points = [tuple(c) for c in corners] - points += [points[0]] - draw.line(points, fill=color, width=thickness) - - # Draw heading. - max_dim = max(width, height) / 2. - end_heading_point = (center_x + max_dim * math.cos(heading), - center_y + max_dim * math.sin(heading)) - start_heading_point = ((end_heading_point[0] - center_x) / 2 + center_x, - (end_heading_point[1] - center_y) / 2 + center_y) - - heading_points = [start_heading_point, end_heading_point] - draw.line(heading_points, fill=color, width=thickness) - - # Compute extremes so we can anchor the labels to them. - xs = [x[0] for x in points] - ys = [x[1] for x in points] - left = np.min(xs) - bottom = np.min(ys) - top = np.max(ys) - - try: - font = ImageFont.truetype('arial.ttf', 24) - except IOError: - font = ImageFont.load_default() - - text_width, text_height = font.getsize(display_str) - margin = np.ceil(0.05 * text_height) - if text_loc == 'TOP': - text_bottom = top - else: - text_bottom = bottom + text_height - - draw.rectangle([(left, text_bottom - text_height - 2 * margin), - (left + text_width, text_bottom)], - fill=color) - draw.text((left + margin, text_bottom - text_height - margin), - display_str, - fill='black', - font=font) - np.copyto(original_image, np.array(image)) - - -def VisualizeBoxes(image, - boxes, - classes, - scores, - class_id_to_name, - min_score_thresh=.25, - line_thickness=4, - groundtruth_box_visualization_color='black', - skip_scores=False, - skip_labels=False, - text_loc='TOP'): - """Visualize boxes on top down image.""" - box_to_display_str_map = collections.defaultdict(str) - box_to_color_map = collections.defaultdict(str) - num_boxes = boxes.shape[0] - for i in range(num_boxes): - if scores is not None and scores[i] < min_score_thresh: - continue - box = tuple(boxes[i].tolist()) - display_str = '' - if not skip_labels: - if classes[i] in class_id_to_name: - class_name = class_id_to_name[classes[i]] - display_str = str(class_name) - else: - display_str = 'N/A' - if not skip_scores: - if not display_str: - display_str = '{}%'.format(int(100 * scores[i])) - else: - display_str = '{}: {}%'.format(display_str, int(100 * scores[i])) - box_to_display_str_map[box] = display_str - if scores is None: - box_to_color_map[box] = groundtruth_box_visualization_color - else: - box_to_color_map[box] = PIL_COLOR_LIST[classes[i] % len(PIL_COLOR_LIST)] - - # Draw all boxes onto image. - for box, color in box_to_color_map.items(): - DrawBoundingBoxOnImage( - image, - box, - color=color, - thickness=line_thickness, - display_str=box_to_display_str_map[box], - text_loc=text_loc) - return image - - -def TransformBBoxesToTopDown(bboxes, car_to_image_transform=None): - """Convert bounding boxes from car coordinates to top down pixel coordinates. - - Args: - bboxes: A (batch, nbboxes, 4 or 5) np.float32 tensor containing bounding box - xywhh in car coordinates (smooth adjusted by car pose). - car_to_image_transform: An optional Transform object. If None, this will be - created using _CarToImageTransform. - - Returns: - np.array of shape (batch, nbboxes) containing the bounding boxes in top down - image space. - """ - if car_to_image_transform is None: - car_to_image_transform = _CarToImageTransform() - - batch_size = np.shape(bboxes)[0] - nbboxes = np.shape(bboxes)[1] - transformed_boxes = np.zeros_like(bboxes) - - for batch_id in range(batch_size): - for box_id in range(nbboxes): - # TODO(vrv): When we predict heading, we should assert - # that the length of bbox_data is 5. - bbox_data = np.squeeze(bboxes[batch_id, box_id, :]) - - x, y, width, height = (bbox_data[0], bbox_data[1], bbox_data[2], - bbox_data[3]) - - if len(bbox_data) == 5: - heading = bbox_data[4] - else: - heading = 0.0 - - # Skip boxes that cannot be visualized. - if width <= 0 or height <= 0: - continue - if any([np.isnan(c) or np.isinf(c) for c in bbox_data]): - continue - - # Bounding boxes are in car coordinates (smooth adjusted by car pose). - # Transform from car coordinates to new coordinates. - bbox_car = transform_util.Box2D(x, y, width, height, heading) - bbox_transformed = bbox_car.Apply(car_to_image_transform) - bbox_values = bbox_transformed.AsNumpy() - if len(bbox_data) == 4: - bbox_values = bbox_values[:-1] - transformed_boxes[batch_id, box_id, :] = bbox_values - - return transformed_boxes - - -def DrawBBoxesOnImages(images, bboxes, box_weights, labels, class_id_to_name, - groundtruth): - """Draw ground truth boxes on top down image. - - Args: - images: A 4D uint8 array (batch, height, width, depth) of images to draw on - top of. - bboxes: A (batch, nbboxes, 4 or 5) np.float32 tensor containing bounding box - xywhh to draw specified in top down pixel values. - box_weights: A (batch, nbboxes) float matrix indicating the predicted score - of the box. If the score is 0.0, no box is drawn. - labels: A (batch, nbboxes) integer matrix indicating the true or predicted - label indices. - class_id_to_name: Dictionary mapping from class id to name. - groundtruth: Boolean indicating whether bounding boxes are ground truth. - - Returns: - 'images' with the bboxes drawn on top. - """ - # Assert 4d shape. - assert len(np.shape(images)) == 4 - - if np.shape(images)[3] == 1: - # Convert from grayscale to RGB. - images = np.tile(images, (1, 1, 1, 3)) - - # Assert channel dimension is 3 dimensional. - assert np.shape(images)[3] == 3 - - batch_size = np.shape(images)[0] - nbboxes = np.shape(bboxes)[1] - - for batch_id in range(batch_size): - image = images[batch_id, :, :, :] - # Draw a box for each box and label if weights is 1. - - transformed_boxes = [] - box_scores = [] - label_ids = [] - - for box_id in range(nbboxes): - box_weight = box_weights[batch_id, box_id] - # If there is no box to draw, continue. - if box_weight == 0.0: - continue - - # TODO(vrv): When we predict heading, we should assert - # that the length of bbox_data is 5. - bbox_data = np.squeeze(bboxes[batch_id, box_id, :]) - if len(bbox_data) == 5: - x, y, width, length, heading = bbox_data - else: - x, y, width, length = bbox_data - heading = 0.0 - - # Check whether we can draw the box. - bbox = transform_util.Box2D(x, y, width, length, heading) - ymin, xmin, ymax, xmax = bbox.Extrema() - - if ymin == 0 and xmin == 0 and ymax == 0 and xmax == 0: - continue - - # TODO(vrv): Support drawing boxes on the edge of the - # image. - if (xmin < 0 or ymin < 0 or xmax >= image.shape[1] or - ymax >= image.shape[0]): - continue - - # We can draw a box on the image, so fill in the score, the boxes, - # and the label. - transformed_boxes.append([x, y, width, length, heading]) - box_scores.append(box_weight) - label_ids.append(labels[batch_id, box_id]) - - if transformed_boxes: - transformed_boxes = np.stack(transformed_boxes, axis=0) - scores = None if groundtruth else np.array(box_scores) - text_loc = 'TOP' if groundtruth else 'BOTTOM' - VisualizeBoxes( - image=image, - boxes=transformed_boxes, - classes=label_ids, - scores=scores, - class_id_to_name=class_id_to_name, - groundtruth_box_visualization_color='cyan', - skip_scores=groundtruth, - skip_labels=False, - text_loc=text_loc) - return images - - -def DrawTrajectory(image, bboxes, masks, labels, is_groundtruth): - """Draw the trajectory of bounding boxes on 'image'. - - Args: - image: The uint8 image array to draw on. Assumes [1000, 500, 3] input with - RGB value ranges. - bboxes: A [num_steps, num_objects, 5] float array containing the bounding - box information over a sequence of steps. bboxes are expected to be in - car coordinates. - masks: A [num_steps, num_objects] integer array indicating whether the - corresponding bbox entry in bboxes is present (1 = present). - labels: A [num_steps, num_objects] integer label indicating which class is - being predicted. Used for colorizing based on labels. - is_groundtruth: True if the scene is the groundtruth vs. the predicted. - - Returns: - The updated image array. - """ - image = Image.fromarray(np.uint8(image)).convert('RGB') - draw = ImageDraw.Draw(image) - - try: - font = ImageFont.truetype('arial.ttf', 20) - except IOError: - font = ImageFont.load_default() - - pixels_per_meter = 10. - image_ref_x = 250. - image_ref_y = 750. - car_to_image_transform = transform_util.MakeCarToImageTransform( - pixels_per_meter=pixels_per_meter, - image_ref_x=image_ref_x, - image_ref_y=image_ref_y, - flip_axes=True) - - # Iterate over each object and produce the series of visualized trajectories - # over time. - for object_idx in range(bboxes.shape[1]): - # Annotate the box with the class type - label = labels[0, object_idx] - - # Choose a label_consistent color. - color = PIL_COLOR_LIST[label % len(PIL_COLOR_LIST)] - # Make predictions white. - if not is_groundtruth: - color = 'white' - - # Convert string color name to RGB so we can manipulate it. - color_rgb = ImageColor.getrgb(color) - - # For each sample, extract the data, transform to image coordinates, and - # store in centroids. - centroids = [] - for time in range(bboxes.shape[0]): - if masks[time, object_idx] == 0: - continue - - center_x, center_y, width, height, heading = bboxes[time, object_idx, :] - - # Compute the new heading. - heading = transform_util.TransformHeading(car_to_image_transform, heading) - - # Transform from car to image coords. - x, y, _ = transform_util.TransformPoint(car_to_image_transform, center_x, - center_y, 0.0) - - # Hack to scale from meters to pixels. - width *= pixels_per_meter - height *= pixels_per_meter - - # Collect the centroids of all of the points. - centroids.append((x, y, heading)) - - # Draw the groundtruth bounding box at the first timestep. - if is_groundtruth and time == 0: - # Draw a rectangle - rect = MakeRectangle(height, width, heading, offset=(x, y)) - rect += [rect[0]] - draw.line(rect, fill=color_rgb, width=4) - - delta = 20 - - # Annotate the box with the object index - draw.text((x + delta, y + delta), - str(object_idx), - fill='white', - font=font) - - # Draw a callout - draw.line([(x, y), (x + delta, y + delta)], fill='white', width=1) - - # Extract the point pairs from centroids and draw a line through them. - point_pairs = [] - for (x, y, heading) in centroids: - point_pairs.append((x, y)) - if point_pairs: - draw.line(point_pairs, width=4, fill=color_rgb) - - # Draw the centroids. - triangle_color_rgb = color_rgb - for i, (x, y, heading) in enumerate(centroids): - if i == 0: - # Draw the heading for the first timestep. - scale = 25 if is_groundtruth else 15 - DrawHeadingTriangle(draw, x, y, heading, triangle_color_rgb, scale) - else: - # Draw a circle for the centroids of other timesteps. - outline_color = color_rgb - circle_size = 5 if is_groundtruth else 3 - DrawCircle( - draw, - x, - y, - fill=triangle_color_rgb, - outline=outline_color, - circle_size=circle_size) - - # Desaturate the color with every timestep. - increment = 45 # Allow this to be modified? - triangle_color_rgb = (triangle_color_rgb[0] - increment, - triangle_color_rgb[1] - increment, - triangle_color_rgb[2] - increment) - - return np.array(image) - - -def GetTrajectoryComparison(gt_bboxes, gt_masks, gt_labels, pred_bboxes, - pred_masks, pred_labels): - """Draw a trajectory comparison of groundtruth and predicted. - - Args: - gt_bboxes: A [batch_size, num_steps, num_objects, 5] float array containing - the bounding box information over a sequence of steps. - gt_masks: A [batch_size, num_steps, num_objects] integer array indicating - whether the corresponding bbox entry in bboxes is present (1 = present). - gt_labels: A [batch_size, num_steps, num_objects] integer label indicating - which class is being predicted. Used for colorizing based on labels. - pred_bboxes: A [batch_size, num_steps, num_objects, 5] float array - containing the bounding box information over a sequence of steps. - pred_masks: A [batch_size, num_steps, num_objects] integer array indicating - whether the corresponding bbox entry in bboxes is present (1 = present). - pred_labels: A [batch_size, num_steps, num_objects] integer label indicating - which class is being predicted. Used for colorizing based on labels. - - Returns: - images: A np.uint8 images array that can be displayed of size - [batch_size, 1000, 500, 3]. - """ - batch_size = gt_bboxes.shape[0] - images = np.zeros([batch_size, 1000, 500, 3], dtype=np.uint8) - for b in range(batch_size): - images[b] = DrawTrajectory( - images[b], gt_bboxes[b], gt_masks[b], gt_labels[b], is_groundtruth=True) - images[b] = DrawTrajectory( - images[b], - pred_bboxes[b], - pred_masks[b], - pred_labels[b], - is_groundtruth=False) - return images diff --git a/lingvo/tasks/car/summary_test.py b/lingvo/tasks/car/summary_test.py deleted file mode 100644 index 98a5021e..00000000 --- a/lingvo/tasks/car/summary_test.py +++ /dev/null @@ -1,112 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for summary.""" - -from lingvo import compat as tf -from lingvo.core import test_utils -from lingvo.tasks.car import summary -import numpy as np - - -class SummaryTest(test_utils.TestCase): - """Tests for Summary utilities.""" - - def testDrawBBoxes(self): - bs = 5 - nbboxes = 2 - class_id_to_name = {0: 'foo', 1: 'bar'} - - loc_weights = np.ones(shape=(bs, nbboxes)) - labels = np.ones(shape=(bs, nbboxes)) - cbboxes = np.zeros(shape=(bs, nbboxes, 4)) - # Ensures ymin/xmin/ymax/ymin. - cbboxes[:, :, :2] = 100. * np.random.normal(size=(bs, nbboxes, 2)) - cbboxes[:, :, 2:4] = cbboxes[:, :, :2] + 10. * np.random.uniform( - size=(bs, nbboxes, 2)) - - images = np.zeros(shape=(bs, 100, 100, 1), dtype=np.uint8) - summary.DrawBBoxesOnImages(images, cbboxes, loc_weights, labels, - class_id_to_name, True) - - def testTransformBBoxesToTopDown(self): - bs = 5 - nbboxes = 2 - cbboxes = np.zeros(shape=(bs, nbboxes, 4)) - # Ensures ymin/xmin/ymax/ymin. - cbboxes[:, :, :2] = 100. * np.random.normal(size=(bs, nbboxes, 2)) - cbboxes[:, :, 2:4] = cbboxes[:, :, :2] + 10. * np.random.uniform( - size=(bs, nbboxes, 2)) - - image_bboxes = summary.TransformBBoxesToTopDown(cbboxes) - self.assertAllEqual(cbboxes.shape, image_bboxes.shape) - - def testExtractRunIds(self): - - def PythonExtractRunIds(run_segments): - """Extract the RunIds from the run_segments feature field.""" - num_segments = run_segments.shape[0] - run_ids = [] - for i in range(num_segments): - # Run segment is a serialized RunSegmentProto. We don't have access to - # it, but it's pretty simple so just manually parse the string for now - # to get the run id. - # - # One can add a duplicate proto to make parsing more explicit. - run_segment = run_segments[i][0] - run_id = run_segment.split('\n')[0].split('"')[1] - start_time = str(int(float(run_segment.split('\n')[1].split(' ')[1]))) - run_ids.append(run_id + '_' + start_time) - run_ids = [tf.compat.as_bytes(r) for r in run_ids] - return np.stack(run_ids) - - example_run_segments = [ - [ - 'run: "20170903_161642_C00844"\nstart_offset:' - ' 6876.0795650482178\nend_offset: 6896.0795650482178\nstart_ts:' - ' 1504462278.6018586\nend_ts: 1504462298.6018586\n' - ], - [ - 'run: "20001029_0123458_C00844"\nstart_offset:' - ' 66666.0795650482178\nend_offset: 77777.0795650482178\nstart_ts:' - ' 1504462278.6018586\nend_ts: 1504462298.6018586\n' - ], - ] - expected_ids = PythonExtractRunIds(np.array(example_run_segments)) - with self.session(): - run_ids = self.evaluate( - summary.ExtractRunIds(tf.constant(example_run_segments))) - self.assertAllEqual(expected_ids, run_ids) - - def testTrajectory(self): - # Test that the trajectory code can execute. A test that validates the - # output is challenging - bs = 2 - steps = 5 - nbboxes = 8 - gt_bboxes = np.zeros(shape=(bs, steps, nbboxes, 5)) - gt_masks = np.ones(shape=(bs, steps, nbboxes)) - - pred_bboxes = np.zeros(shape=(bs, steps, nbboxes, 5)) - pred_masks = np.ones(shape=(bs, steps, nbboxes)) - labels = np.ones(shape=(bs, steps, nbboxes)).astype(np.int32) - - image = summary.GetTrajectoryComparison(gt_bboxes, gt_masks, labels, - pred_bboxes, pred_masks, labels) - self.assertEqual((bs, 1000, 500, 3), image.shape) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/testdata/BUILD b/lingvo/tasks/car/testdata/BUILD deleted file mode 100644 index 613f4152..00000000 --- a/lingvo/tasks/car/testdata/BUILD +++ /dev/null @@ -1,24 +0,0 @@ -package(default_visibility = ["//visibility:public"]) - -licenses(["notice"]) # Apache 2.0 - -filegroup( - name = "kitti_raw", - data = glob([ - "kitti_raw_*", - ]), -) - -filegroup( - name = "waymo_tutorial_frames", - data = glob([ - "waymo_*", - ]), -) - -filegroup( - name = "waymo_ap_testdata", - data = glob([ - "waymo_decode_sparse_dev*", - ]), -) diff --git a/lingvo/tasks/car/testdata/kitti_raw_calib_testdata.txt b/lingvo/tasks/car/testdata/kitti_raw_calib_testdata.txt deleted file mode 100644 index 2b8496d5..00000000 --- a/lingvo/tasks/car/testdata/kitti_raw_calib_testdata.txt +++ /dev/null @@ -1,7 +0,0 @@ -P0: 7.215377000000e+02 0.000000000000e+00 6.095593000000e+02 0.000000000000e+00 0.000000000000e+00 7.215377000000e+02 1.728540000000e+02 0.000000000000e+00 0.000000000000e+00 0.000000000000e+00 1.000000000000e+00 0.000000000000e+00 -P1: 7.215377000000e+02 0.000000000000e+00 6.095593000000e+02 -3.875744000000e+02 0.000000000000e+00 7.215377000000e+02 1.728540000000e+02 0.000000000000e+00 0.000000000000e+00 0.000000000000e+00 1.000000000000e+00 0.000000000000e+00 -P2: 7.215377000000e+02 0.000000000000e+00 6.095593000000e+02 4.485728000000e+01 0.000000000000e+00 7.215377000000e+02 1.728540000000e+02 2.163791000000e-01 0.000000000000e+00 0.000000000000e+00 1.000000000000e+00 2.745884000000e-03 -P3: 7.215377000000e+02 0.000000000000e+00 6.095593000000e+02 -3.395242000000e+02 0.000000000000e+00 7.215377000000e+02 1.728540000000e+02 2.199936000000e+00 0.000000000000e+00 0.000000000000e+00 1.000000000000e+00 2.729905000000e-03 -R0_rect: 9.999239000000e-01 9.837760000000e-03 -7.445048000000e-03 -9.869795000000e-03 9.999421000000e-01 -4.278459000000e-03 7.402527000000e-03 4.351614000000e-03 9.999631000000e-01 -Tr_velo_to_cam: 7.533745000000e-03 -9.999714000000e-01 -6.166020000000e-04 -4.069766000000e-03 1.480249000000e-02 7.280733000000e-04 -9.998902000000e-01 -7.631618000000e-02 9.998621000000e-01 7.523790000000e-03 1.480755000000e-02 -2.717806000000e-01 -Tr_imu_to_velo: 9.999976000000e-01 7.553071000000e-04 -2.035826000000e-03 -8.086759000000e-01 -7.854027000000e-04 9.998898000000e-01 -1.482298000000e-02 3.195559000000e-01 2.024406000000e-03 1.482454000000e-02 9.998881000000e-01 -7.997231000000e-01 diff --git a/lingvo/tasks/car/testdata/kitti_raw_label_testdata.txt b/lingvo/tasks/car/testdata/kitti_raw_label_testdata.txt deleted file mode 100644 index 766f76ec..00000000 --- a/lingvo/tasks/car/testdata/kitti_raw_label_testdata.txt +++ /dev/null @@ -1,7 +0,0 @@ -Truck 0.00 0 -1.57 599.41 156.40 629.75 189.25 2.85 2.63 12.34 0.47 1.49 69.44 -1.56 -Car 0.00 0 1.85 387.63 181.54 423.81 203.12 1.67 1.87 3.69 -16.53 2.39 58.49 1.57 -Cyclist 0.00 3 -1.65 676.60 163.95 688.98 193.93 1.86 0.60 2.02 4.59 1.32 45.84 -1.55 -DontCare -1 -1 -10 503.89 169.71 590.61 190.13 -1 -1 -1 -1000 -1000 -1000 -10 -DontCare -1 -1 -10 511.35 174.96 527.81 187.45 -1 -1 -1 -1000 -1000 -1000 -10 -DontCare -1 -1 -10 532.37 176.35 542.68 185.27 -1 -1 -1 -1000 -1000 -1000 -10 -DontCare -1 -1 -10 559.62 175.83 575.40 183.15 -1 -1 -1 -1000 -1000 -1000 -10 diff --git a/lingvo/tasks/car/tools/BUILD b/lingvo/tasks/car/tools/BUILD deleted file mode 100644 index 16d33d62..00000000 --- a/lingvo/tasks/car/tools/BUILD +++ /dev/null @@ -1,109 +0,0 @@ -package(default_visibility = [ - "//visibility:public", -]) - -licenses(["notice"]) # Apache 2.0 - -py_binary( - name = "kitti_exporter", - srcs = ["kitti_exporter.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":kitti_data", - # Implicit network file system dependency. - # Implicit PIL dependency. - # Implicit absl.app dependency. - # Implicit absl.flags dependency. - # Implicit absl.logging dependency. - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_binary( - name = "create_kitti_crop_dataset", - srcs = [ - "create_kitti_crop_dataset.py", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - # Implicit absl.app dependency. - # Implicit absl.flags dependency. - # Implicit apache_beam dependency. - "//lingvo:compat", - "//lingvo:model_registry", - "//lingvo/core:cluster_factory", - "//lingvo/core:py_utils", - "//lingvo/tasks/car:geometry", - "//lingvo/tasks/car:input_extractor", - "//lingvo/tasks/car/params", - "//lingvo/tools:beam_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "kitti_data", - srcs = ["kitti_data.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_test( - name = "kitti_data_test", - srcs = ["kitti_data_test.py"], - data = [ - "//lingvo/tasks/car/testdata:kitti_raw", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":kitti_data", - "//lingvo:compat", - "//lingvo/core:test_helper", - "//lingvo/core:test_utils", - "//lingvo/tasks/car:geometry", - # Implicit numpy dependency. - ], -) - -py_library( - name = "export_kitti_detection_lib", - srcs = ["export_kitti_detection.py"], - srcs_version = "PY3", - deps = [ - ":kitti_data", - # Implicit absl.flags dependency. - "//lingvo:compat", - "//lingvo/core/ops:record_py_pb2", - "//lingvo/tasks/car:kitti_metadata", - # Implicit numpy dependency. - ], -) - -py_binary( - name = "export_kitti_detection", - srcs = ["export_kitti_detection.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [":export_kitti_detection_lib"], -) - -py_binary( - name = "compare_params", - srcs = ["compare_params_main.py"], - main = "compare_params_main.py", - python_version = "PY3", - srcs_version = "PY3", - deps = [ - # Implicit absl.flags dependency. - "//lingvo:compat", - "//lingvo/tasks/car/params:kitti", - "//lingvo/tools:compare_params_lib", - ], -) diff --git a/lingvo/tasks/car/tools/__init__.py b/lingvo/tasks/car/tools/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/lingvo/tasks/car/tools/compare_params_main.py b/lingvo/tasks/car/tools/compare_params_main.py deleted file mode 100644 index f61e3a9f..00000000 --- a/lingvo/tasks/car/tools/compare_params_main.py +++ /dev/null @@ -1,47 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tool for comparing two models / hyperparams.""" - -from absl import flags -from lingvo import compat as tf -from lingvo.tasks.car.params import kitti # pylint:disable=unused-import -from lingvo.tools import compare_params - -FLAGS = flags.FLAGS -flags.DEFINE_string("model1", None, - "Registered name or path to params.txt of model 1.") -flags.DEFINE_string("model2", None, - "Registered name or path to params.txt of model 2.") - - -def main(argv): - if len(argv) > 1: - raise tf.app.UsageError("Too many command-line arguments.") - - cfg1_text = compare_params.get_model_params_as_text(FLAGS.model1) - cfg2_text = compare_params.get_model_params_as_text(FLAGS.model2) - - cfg1_not_cfg2, cfg2_not_cfg1, cfg1_and_cfg2_diff = ( - compare_params.hyperparams_text_diff(cfg1_text, cfg2_text)) - - compare_params.print_hyperparams_text_diff(FLAGS.model1, FLAGS.model2, - cfg1_not_cfg2, cfg2_not_cfg1, - cfg1_and_cfg2_diff) - return 0 - - -if __name__ == "__main__": - tf.app.run(main) diff --git a/lingvo/tasks/car/tools/create_kitti_crop_dataset.py b/lingvo/tasks/car/tools/create_kitti_crop_dataset.py deleted file mode 100644 index 8f8e1d6c..00000000 --- a/lingvo/tasks/car/tools/create_kitti_crop_dataset.py +++ /dev/null @@ -1,272 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -r"""Creates KITTI Classification dataset out of the KITTI Object detection data. - -Produces TFRecords with the following format: --------------------------------------------------------------------------------- - -Values: - num_points: Int64 scalar with the number of points in that box. - - points: Float list of 3D locations (X, Y, Z) per point. - - points_feature: Float list of per-point feature (reflectance scalar). - - bbox_3d: Float list of the 7 box coordinates. bboxes_3d[0:3] is the absolute - center location in meters. bboxes_3d[3:6] is the bbox dimensions in meters, - and bboxes_3d[6] is the rotation in radians. - - label: An Int64 scalar with the class label id. - - text: A bytelist with the string name of the class. - - difficulty: An Int64 scalar with the difficulty level of the object. - - occlusion: An Int64 scalar with the occlusion level of the object. - - scene_id: An Int64 scalar with the original scene id the object is from. - - bbox_id: An Int64 scalar with which number bounding box the object is from. - - -Note: All scalars are wrapped in a list of len 1. --------------------------------------------------------------------------------- - -IMPORTANT: Depending on your model, you may or may not want to enable -preprocessors to export for data augmentation. If unsure, do *not* run with -preprocessors as that might add additional data augmentation steps. - -To run: - -bazel run -c opt \ - //lingvo/tasks/car/tools:create_kitti_crop_dataset \ - --model_name=car.kitti.PillarsModelV1 \ - --norun_preprocessors \ - --input_file_pattern=/path/to/kitti/train_pattern \ - --output_file_pattern=/path/to/output/gt_objects@100 -""" - -from absl import app -from absl import flags - -import apache_beam as beam - -from lingvo import compat as tf -from lingvo import model_registry -from lingvo.core import cluster_factory -from lingvo.core import py_utils -from lingvo.tasks.car import geometry -from lingvo.tasks.car import input_extractor -from lingvo.tasks.car.params import kitti # pylint: disable=unused-import -from lingvo.tools import beam_utils - -import numpy as np - - -flags.DEFINE_string('input_file_pattern', None, 'Where to get the data.') -flags.DEFINE_string('output_file_pattern', None, - 'Output directory + prefix where to write files.') -flags.DEFINE_string( - 'model_name', 'car.kitti.PillarsModelV1', - 'Name of registered model whose input pipeline ' - 'will be used to mine groundtruth objects from.') -flags.DEFINE_enum('split', 'Train', ['Train', 'Dev'], - 'Split of data to generate GT objects from') -flags.DEFINE_bool( - 'run_preprocessors', False, - 'Whether to run the preprocessors when extracting data. ' - 'Leave this as False if you are generating data for the ' - 'purposes of analysis. If you are generating data for ground ' - 'truth bbox augmentation, enable this.') - -FLAGS = flags.FLAGS - - -def _GetFilteredBoundingBoxData(kitti_data): - """Given a single batch element of data, process it for writing. - - Args: - kitti_data: A NestedMap of KITTI input generator returned data with a batch - size of 1. - - Returns: - A NestedMap of all the output data we need to write per bounding box - cropped pointclouds. - """ - points = kitti_data.lasers.points_xyz - points_feature = kitti_data.lasers.points_feature - bboxes_3d = kitti_data.labels.bboxes_3d - bboxes_3d_mask = kitti_data.labels.bboxes_3d_mask - bboxes_3d = tf.boolean_mask(bboxes_3d, bboxes_3d_mask) - - if 'points_padding' in kitti_data.lasers: - points_validity_mask = tf.cast(kitti_data.lasers.points_padding - 1, - tf.bool) - points = tf.boolean_mask(points, points_validity_mask) - points_feature = tf.boolean_mask(points_feature, points_validity_mask) - - points_in_bboxes_mask = geometry.IsWithinBBox3D(points, bboxes_3d) - - output_map = py_utils.NestedMap() - # Points and features contain the whole pointcloud, which we will use - # per box boolean masks later in _ToTFExampleProto to subselect data per box. - output_map.points = points - output_map.points_feature = points_feature - output_map.points_in_bboxes_mask = points_in_bboxes_mask - - output_map.source_id = kitti_data.labels.source_id - - # Add additional data - output_keys = [ - 'bboxes_3d', - 'labels', - 'texts', - 'occlusion', - 'difficulties', - 'truncation', - ] - for key in output_keys: - output_map[key] = tf.boolean_mask(kitti_data.labels[key], - kitti_data.labels.bboxes_3d_mask) - return output_map - - -class _ProcessShard(beam.DoFn): - """Process a given shard.""" - - def __init__(self, model_name, split, run_preprocessors): - self._model_name = model_name - self._split = split - self._run_preprocessors = run_preprocessors - self._sess = None - - # Create a cluster configuration assuming evaluation; the input pipelines - # need to know the cluster job type to set up the outputs correctly. - cluster = cluster_factory.Current() - cluster.params.job = 'evaler' - cluster.params.mode = 'sync' - cluster.params.task = 0 - cluster.params.evaler.replicas = 1 - self._cluster = cluster_factory.Cluster(cluster.params) - - def _create_graph(self): - if self._sess is not None: - return - - with self._cluster: - cfg = model_registry.GetParams(self._model_name, self._split) - cfg.input.batch_size = 1 - # Turn off label filtering so the database contains - # all objects. - cfg.input.extractors.labels.filter_labels = None - - # Disable preprocessors if they are not required. - if not self._run_preprocessors: - cfg.input.preprocessors_order = [] - - graph = tf.Graph() - with graph.as_default(): - inp = cfg.input.Instantiate() - self._elem = tf.placeholder(tf.string) - bucket, batch = inp.ExtractUsingExtractors(self._elem) - self._filtered_data = _GetFilteredBoundingBoxData(batch) - self._bucket = bucket - self._sess = tf.Session(graph=graph) - - def _ToTFExampleProto(self, filtered_data, bbox_idx): - num_boxes = filtered_data.bboxes_3d.shape[0] - if bbox_idx >= num_boxes: - raise ValueError('`bbox_id` should be < num_boxes') - bbox_mask = filtered_data.points_in_bboxes_mask[:, bbox_idx] - - num_points, _ = py_utils.GetShape(filtered_data.points[bbox_mask], 2) - - example = tf.train.Example() - feature = example.features.feature - feature['num_points'].int64_list.value[:] = [num_points] - feature['points'].float_list.value[:] = ( - filtered_data.points[bbox_mask].ravel().tolist()) - feature['points_feature'].float_list.value[:] = ( - filtered_data.points_feature[bbox_mask].ravel().tolist()) - # Note: bbox_3d, label, text, difficulty are singular, which is inconsistent - # with the original KITTI data, but that is because each example - # has only one bounding box, one label, one text, one difficulty. - feature['bbox_3d'].float_list.value[:] = ( - filtered_data.bboxes_3d[bbox_idx, :].ravel().tolist()) - feature['label'].int64_list.value[:] = [ - filtered_data.labels[bbox_idx].astype(np.int64) - ] - feature['text'].bytes_list.value[:] = [filtered_data.texts[bbox_idx]] - feature['difficulty'].int64_list.value[:] = [ - filtered_data.difficulties[bbox_idx].astype(np.int64) - ] - feature['occlusion'].int64_list.value[:] = [ - filtered_data.occlusion[bbox_idx].astype(np.int64) - ] - feature['scene_id'].int64_list.value[:] = [int(filtered_data.source_id)] - feature['bbox_id'].int64_list.value[:] = [int(bbox_idx)] - return example - - def process(self, value): - self._create_graph() - elem_str = value.SerializeToString() - - b, bucket = self._sess.run([self._filtered_data, self._bucket], - feed_dict={self._elem: elem_str}) - if bucket > input_extractor.BUCKET_UPPER_BOUND: - return - b = py_utils.NestedMap(b) - - # Flatten the batch. - flatten = b.FlattenItems() - if not flatten: - return - - num_boxes = b.bboxes_3d.shape[0] - - # For each box, get the pointcloud and write it as an example. - for bbox_id in range(num_boxes): - tf_example = self._ToTFExampleProto(b, bbox_id) - yield tf_example - - -def main(_): - beam_utils.BeamInit() - - if not FLAGS.output_file_pattern: - raise ValueError('Must provide an output_file_pattern') - - reader = beam.io.ReadFromTFRecord( - FLAGS.input_file_pattern, coder=beam.coders.ProtoCoder(tf.train.Example)) - - model_name = FLAGS.model_name - split = FLAGS.split - run_preprocessors = FLAGS.run_preprocessors - - with beam_utils.GetPipelineRoot() as root: - _ = ( - root - | 'Read' >> reader - | 'ToTFExample' >> beam.ParDo( - _ProcessShard(model_name, split, run_preprocessors)) - | 'Reshuffle' >> beam.Reshuffle() - | 'Write' >> beam.io.WriteToTFRecord( - FLAGS.output_file_pattern, - coder=beam.coders.ProtoCoder(tf.train.Example))) - - -if __name__ == '__main__': - app.run(main) diff --git a/lingvo/tasks/car/tools/export_kitti_detection.py b/lingvo/tasks/car/tools/export_kitti_detection.py deleted file mode 100644 index 84607497..00000000 --- a/lingvo/tasks/car/tools/export_kitti_detection.py +++ /dev/null @@ -1,232 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -r"""Read saved Decoder's outputs and convert to KITTI text format. - -First, obtain a KITTI camera calibration file. - -To export all detections from a single model: - -python export_kitti_detection.py \ ---decoder_path=/path/to/decoder_out_000103000 \ ---calib_file=/tmp/kitti_test_calibs.npz \ ---output_dir=/tmp/my-kitti-export-directory \ ---logtostderr - ---- OR --- - -Export combined detections selected from multiple models: - -python export_kitti_detection.py \ ---car_decoder_path=/path/to/car_decoder_out \ ---ped_decoder_path=/path/to/ped_decoder_out \ ---cyc_decoder_path=/path/to/cyc_decoder_out \ ---calib_file=/tmp/kitti_test_calibs.npz \ ---output_dir=/tmp/my-kitti-export-directory \ ---logtostderr -""" - -from absl import flags -from lingvo import compat as tf -from lingvo.core.ops import record_pb2 -from lingvo.tasks.car import kitti_metadata -from lingvo.tasks.car.tools import kitti_data -import numpy as np - - -FLAGS = flags.FLAGS -flags.DEFINE_string( - "decoder_path", None, "Paths to decoder file containing output " - "of decoder for everything. Either supply this argument or individual " - "decoder paths for cars, pedestrians and cyclists.") -flags.DEFINE_string( - "car_decoder_path", None, - "Paths to decoder file containing output of decoder for cars." - "Either supply plus cyclists and pedestrians or supply one " - "decoder for all labels.") -flags.DEFINE_string( - "ped_decoder_path", None, - "Paths to decoder file containing output of decoder for " - "pedestrians. Either supply plus cyclists and cars or " - "supply one decoder for all labels.") -flags.DEFINE_string( - "cyc_decoder_path", None, - "Paths to decoder file containing output of decoder for cyclist. " - "Either supply plus cars and pedestrians or supply one " - "decoder for all labels.") -flags.DEFINE_string( - "calib_file", None, - "Path to a npz file that contains all calibration matrices.") -flags.DEFINE_string("output_dir", None, "Place to write detections.") -flags.DEFINE_float("score_threshold", 0, "Ignore detections with lower score.") - - -def LoadCalibData(fname): - """Load and parse calibration data from NPZ file.""" - # If this throws an error, make sure the npz file was generated from - # the same version of python as this binary. - npz = np.load(fname) - scene_to_calib = {} - for idx, scene_id in enumerate(npz["scene_id"]): - tf.logging.info("Processing %s", scene_id) - raw_calib = {} - raw_calib["P0"] = npz["P0"][idx] - raw_calib["P1"] = npz["P1"][idx] - raw_calib["P2"] = npz["P2"][idx] - raw_calib["P3"] = npz["P3"][idx] - raw_calib["R0_rect"] = npz["R0_rect"][idx] - raw_calib["Tr_velo_to_cam"] = npz["Tr_velo_to_cam"][idx] - raw_calib["Tr_imu_to_velo"] = npz["Tr_imu_to_velo"][idx] - - calib = kitti_data.ParseCalibrationDict(raw_calib) - scene_to_calib[scene_id] = calib - return scene_to_calib - - -def ExtractNpContent(np_dict, calib): - """Parse saved np arrays and convert 3D bboxes to camera0 coordinates. - - Args: - np_dict: a dict of numpy arrays. - calib: a parsed calibration dictionary. - - Returns: - A tuple of 6 ndarrays: - - - location_camera: [N, 3]. [x, y, z] in camera0 coordinate. - - dimension_camera: [N, 3]. The [height, width, length] of objects. - - phi_camera: [N]. Rotation around y-axis in camera0 coodinate. - - bboxes_2d: [N, 4]. The corresponding 2D bboxes in the image coordinate. - - scores: [N]. Confidence scores for each box for the assigned class. - - class_ids: [N]. The class id assigned to each box. - """ - bboxes = np_dict["bboxes"] - scores = np_dict["scores"] - class_ids = np_dict["class_ids"] - bboxes_2d = np_dict["bboxes_2d"] - - # Transform from velodyne coordinates to camera coordinates. - velo_to_cam_transform = kitti_data.VeloToCameraTransformation(calib) - location_cam = np.zeros((len(bboxes), 3)) - dimension_cam = np.zeros((len(bboxes), 3)) - rotation_cam = np.zeros((len(bboxes), 1)) - for idx, bbox in enumerate(bboxes): - location_cam[idx, :], dimension_cam[idx, :], rotation_cam[idx, :] = ( - kitti_data.BBox3DToKITTIObject(bbox, velo_to_cam_transform)) - - return location_cam, dimension_cam, rotation_cam, bboxes_2d, scores, class_ids - - -_INCLUDED_KITTI_CLASS_NAMES = ["Car", "Pedestrian", "Cyclist"] - - -def ExportKITTIDetection(out_dir, source_id, location_cam, dimension_cam, - rotation_cam, bboxes_2d, scores, class_name, is_first): - """Write detections to a text file in KITTI format.""" - tf.logging.info("Exporting %s for %s" % (class_name, source_id)) - fname = out_dir + "/" + source_id + ".txt" - with tf.io.gfile.GFile(fname, "a") as fid: - # Ensure we always create a file even when there's no detection. - # TODO(shlens): Test whether this is actually necessary on the KITTI - # eval server. - if is_first: - fid.write("") - for location, dimension, ry, bbox_2d, score in zip( - location_cam, dimension_cam, rotation_cam, bboxes_2d, scores): - if score < FLAGS.score_threshold: - continue - # class_name, truncated(ignore), alpha(ignore), bbox2D x 4 - part1 = [class_name, -1, -1, -10] + list(bbox_2d) - # dimesion x 3, location x 3, rotation_y x 1, score x 1 - fill = tuple(part1 + list(dimension) + list(location) + [ry] + [score]) - kitti_format_string = ("%s %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf " - "%lf %lf %lf %lf") - kitti_line = kitti_format_string % fill - fid.write(kitti_line + "\n") - - -def main(argv): - if len(argv) > 1: - raise tf.app.UsageError("Too many command-line arguments.") - - if FLAGS.decoder_path: - assert not FLAGS.car_decoder_path and not FLAGS.ped_decoder_path \ - and not FLAGS.cyc_decoder_path, ("Either provide decoder_path or " - "individual decoders but not both.") - else: - assert FLAGS.car_decoder_path and FLAGS.ped_decoder_path and \ - FLAGS.cyc_decoder_path, ("No decoder_path specified. Please supply all " - "individual decoder_paths for labels.") - is_single_decoder_file = FLAGS.decoder_path is not None - - if is_single_decoder_file: - list_of_decoder_paths = [FLAGS.decoder_path] - else: - # Note the correspondence between _INCLUDED_KITTI_CLASS_NAMES ordering and - # this list. - list_of_decoder_paths = [ - FLAGS.car_decoder_path, FLAGS.ped_decoder_path, FLAGS.cyc_decoder_path - ] - - # A list of dictionaries mapping img ids to a dictionary of numpy tensors. - table_data = [] - - img_ids = [] - for table_path in list_of_decoder_paths: - img_id_dict = {} - for serialized in tf.io.tf_record_iterator(table_path): - record = record_pb2.Record() - record.ParseFromString(serialized) - img_id = str(tf.make_ndarray(record.fields["img_id"])) - img_ids.append(img_id) - np_dict = {k: tf.make_ndarray(v) for k, v in record.fields.items()} - img_id_dict[img_id] = np_dict - table_data.append(img_id_dict) - img_ids = list(set(img_ids)) - - if not tf.io.gfile.exists(FLAGS.output_dir): - tf.io.gfile.mkdir(FLAGS.output_dir) - - all_kitti_class_names = kitti_metadata.KITTIMetadata().ClassNames() - calib_data = LoadCalibData(tf.io.gfile.GFile(FLAGS.calib_file, "rb")) - count = 0 - for img_id in img_ids: - # Ignore padded samples where the img_ids are empty. - if not img_id: - continue - for table_index, img_id_dict in enumerate(table_data): - if img_id in img_id_dict: - np_dict = img_id_dict[img_id] - - (location_cam, dimension_cam, rotation_cam, bboxes_2d, scores, - class_ids) = ExtractNpContent(np_dict, calib_data[img_id + ".txt"]) - if is_single_decoder_file: - valid_labels = _INCLUDED_KITTI_CLASS_NAMES - else: - valid_labels = [_INCLUDED_KITTI_CLASS_NAMES[table_index]] - is_first = table_index == 0 - for class_name in valid_labels: - class_mask = (class_ids == all_kitti_class_names.index(class_name)) - ExportKITTIDetection(FLAGS.output_dir, img_id, - location_cam[class_mask], - dimension_cam[class_mask], - rotation_cam[class_mask], bboxes_2d[class_mask], - scores[class_mask], class_name, is_first) - count += 1 - tf.logging.info("Total example exported: %d", count) - - -if __name__ == "__main__": - tf.app.run(main) diff --git a/lingvo/tasks/car/tools/kitti_data.py b/lingvo/tasks/car/tools/kitti_data.py deleted file mode 100644 index 40c2d444..00000000 --- a/lingvo/tasks/car/tools/kitti_data.py +++ /dev/null @@ -1,364 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Library for parsing KITTI raw data.""" - -from lingvo import compat as tf -import numpy as np - - -def LoadVeloBinFile(filepath): - """Reads and parse raw KITTI velodyne binary file. - - Args: - filepath: Path to a raw KITTI velodyne binary file. - - Returns: - A dictionary with keys xyz and reflectance containing numpy arrays. - """ - with tf.io.gfile.GFile(filepath, 'rb') as f: - scan = np.frombuffer(f.read(), dtype=np.float32).reshape((-1, 4)) - xyz = scan[:, :3] - reflectance = scan[:, 3:] - return { - 'xyz': xyz, - 'reflectance': reflectance, - } - - -def LoadLabelFile(filepath): - """Reads and parse raw KITTI label file. - - The ordering of the arrays for bbox, dimensions, and location follows the - order in the table below. We refer to the length (dx), width (dy), height (dz) - for clarity. - - Each line in the label contains (per KITTI documentation): - - +--------+------------+------------------------------------------------------+ - | Values | Name | Description | - +========+============+======================================================+ - | 1 | type | Describes the type of object: 'Car', 'Van', 'Truck', | - | | | 'Pedestrian', 'Person_sitting', 'Cyclist', 'Tram', | - | | | 'Misc' or 'DontCare' | - +--------+------------+------------------------------------------------------+ - | 1 | truncated | Float from 0 (non-truncated) to 1 (truncated), where | - | | | truncated refers to the object leaving image | - | | | boundaries. | - +--------+------------+------------------------------------------------------+ - | 1 | occluded | Integer (0,1,2,3) indicating occlusion state: | - | | | 0 = fully visible, 1 = partly occluded | - | | | 2 = largely occluded, 3 = unknown | - +--------+------------+------------------------------------------------------+ - | 1 | alpha | Observation angle of object, ranging [-pi..pi] | - +--------+------------+------------------------------------------------------+ - | 4 | bbox | 2D bounding box of object in the image (0-based | - | | | index): left, top, right, bottom pixel coordinates. | - +--------+------------+------------------------------------------------------+ - | 3 | dimensions | 3D object dimensions: height, width, length (meters) | - +--------+------------+------------------------------------------------------+ - | 3 | location | 3D object location x,y,z in camera coordinates | - | | | (in meters) | - +--------+------------+------------------------------------------------------+ - | 1 | rotation_y | Rotation ry around Y-axis in camera coordinates | - | | | [-pi..pi] | - +--------+------------+------------------------------------------------------+ - | 1 | score | Only for results: Float, indicating confidence in | - | | | detection, needed for p/r curves, higher is better. | - +--------+------------+------------------------------------------------------+ - - Args: - filepath: Path to a raw KITTI label file. - - Returns: - A list of dictionary with keys corresponding to the name column above. type, - truncated, occluded, alpha, bbox, dimensions, location, rotation_y, score. - Note that the order of the floats in bbox, dimensions, and location - correspond to that in the doc-string above. - """ - objects = [] - with tf.io.gfile.GFile(filepath, 'r') as f: - for line in f: - line = line.strip() - if not line: # Skip empty lines - continue - line_splits = line.split(' ') - if len(line_splits) not in [15, 16]: - raise ValueError( - 'Found {} tokens in Line: "{}". Expects only 15/16 token'.format( - len(line_splits), line)) - - # If score does not exist, we append a -1 to indicate so. - if len(line_splits) == 15: - line_splits.append(-1.) - - (obj_type, truncated, occluded, alpha, - bbox_left, bbox_top, bbox_right, bbox_bottom, - height, width, length, - cam_x, cam_y, cam_z, - rotation_y, score) = line_splits # pyformat: disable - - obj = { - 'type': obj_type, - 'truncated': float(truncated), - 'occluded': int(occluded), - 'alpha': float(alpha), - 'bbox': [ - float(x) for x in [bbox_left, bbox_top, bbox_right, bbox_bottom] - ], - 'dimensions': [float(x) for x in [height, width, length]], - 'location': [float(x) for x in [cam_x, cam_y, cam_z]], - 'rotation_y': float(rotation_y), - 'score': float(score), - } - _ValidateLabeledObject(obj) - objects.append(obj) - - return objects - - -def _ValidateLabeledObject(obj): - """Validate that obj has expected values.""" - - if obj['type'] not in [ - 'Car', 'Van', 'Truck', 'Pedestrian', 'Person_sitting', 'Cyclist', 'Tram', - 'Misc', 'DontCare' - ]: - raise ValueError('Invalid type: %s' % obj['type']) - - if not ((obj['truncated'] == -1.0) or - (obj['truncated'] >= 0.0 and obj['truncated'] <= 1.0)): - raise ValueError('Invalid truncated value: %s' % obj['truncated']) - - if obj['occluded'] not in [-1, 0, 1, 2, 3]: - raise ValueError('Invalid occluded value: %s' % obj['occluded']) - - if not (obj['alpha'] == -10. or - (obj['alpha'] >= -np.pi and obj['alpha'] <= np.pi)): - raise ValueError('Invalid alpha value: %s' % obj['alpha']) - - if not (obj['rotation_y'] == -10. or - (obj['rotation_y'] >= -np.pi and obj['rotation_y'] <= np.pi)): - raise ValueError('Invalid rotation_y value: %s' % obj['rotation_y']) - - return obj - - -def ParseCalibrationDict(raw_calib): - """Parse transformation matrices in a raw KITTI calibration dictionary. - - Per the KITTI documentation: - - All matrices are stored row-major, i.e., the first values correspond - to the first row. R0_rect contains a 3x3 matrix which you need to extend to - a 4x4 matrix by adding a 1 as the bottom-right element and 0's elsewhere. - Tr_xxx is a 3x4 matrix (R|t), which you need to extend to a 4x4 matrix - in the same way. - - IMPORTANT: The coordinates in the camera coordinate system can be projected in - the image by using the 3x4 projection matrix in the calib folder, where for - the left color camera for which the images are provided, P2 must be used. - - Args: - raw_calib: A dictionary of raw KITTI calibration values with keys P0, P1, - P2, P3, R0_rect, Tr_imu_to_velo, and Tr_velo_to_cam containing flattened - matrices of appropriate size. - - Returns: - A dictionary with keys P0, P1, P2, P3, R0_rect, Tr_imu_to_velo, - and Tr_velo_to_cam containing reshaped and extended matrices. - """ - # The projection matrices are 3x4 matrices. - calib = {} - calib['P0'] = raw_calib['P0'].reshape([3, 4]) - calib['P1'] = raw_calib['P1'].reshape([3, 4]) - calib['P2'] = raw_calib['P2'].reshape([3, 4]) - calib['P3'] = raw_calib['P3'].reshape([3, 4]) - - # R0_rect contains a 3x3 matrix which you need to extend to a 4x4 matrix by - # adding a 1 as the bottom-right element and 0's elsewhere. - extended_r0_rect = np.eye(4) - extended_r0_rect[:3, :3] = raw_calib['R0_rect'].reshape([3, 3]) - calib['R0_rect'] = extended_r0_rect - - # Tr_xxx is a 3x4 matrix (R|t), which you need to extend to a 4x4 matrix - # in the same way! - extended_tr_imu_to_velo = np.eye(4) - extended_tr_imu_to_velo[:3, :4] = raw_calib['Tr_imu_to_velo'].reshape([3, 4]) - calib['Tr_imu_to_velo'] = extended_tr_imu_to_velo - - extended_tr_velo_to_cam = np.eye(4) - extended_tr_velo_to_cam[:3, :4] = raw_calib['Tr_velo_to_cam'].reshape([3, 4]) - calib['Tr_velo_to_cam'] = extended_tr_velo_to_cam - - return calib - - -def LoadCalibrationFile(filepath): - """Read and parse a raw KITTI calibration file. - - Args: - filepath: Path to a raw KITTI calibration file. - - Returns: - A dictionary with keys P0, P1, P2, P3, R0_rect, Tr_imu_to_velo, - and Tr_velo_to_cam containing reshaped and extended transformation - matrices. - """ - raw_calib = {} - with tf.io.gfile.GFile(filepath, 'r') as f: - for line in f: - line = line.strip() - if not line: # Skip empty lines - continue - key, value = line.split(':', 1) - raw_calib[key] = np.array([float(x) for x in value.split()]) - return ParseCalibrationDict(raw_calib) - - -def VeloToImagePlaneTransformation(calib): - """Compute the transformation matrix from velo xyz to image plane xy. - - Per the KITTI documentation, to project a point from Velodyne coordinates into - the left color image, you can use this formula: - - x = P2 * R0_rect * Tr_velo_to_cam * y - - After applying the transformation, you will need to divide the by the last - coordinate to recover the 2D pixel locations. - - Args: - calib: A calibration dictionary returned by LoadCalibrationFile. - - Returns: - A numpy 3x4 transformation matrix. - """ - return np.dot(calib['P2'], np.dot(calib['R0_rect'], calib['Tr_velo_to_cam'])) - - -def VeloToCameraTransformation(calib): - """Compute the transformation matrix from velo xyz to camera xyz. - - Per the KITTI documentation, to project a point from Velodyne coordinates into - the left color image, you can use this formula: - - x = P2 * R0_rect * Tr_velo_to_cam * y - - NOTE: The above formula further projects the xyz point to the image plane - using P2, which we do not apply in this function since we are working with - xyz (3D coordinates). - - Args: - calib: A calibration dictionary returned by LoadCalibrationFile. - - Returns: - A numpy 4x4 transformation matrix. - """ - return np.dot(calib['R0_rect'], calib['Tr_velo_to_cam']) - - -def CameraToVeloTransformation(calib): - """Compute the transformation matrix from camera to velo. - - This is the inverse transformation of CameraToVeloTransformation. - - Args: - calib: A calibartion dictionary returned by LoadCalibrationFile. - - Returns: - A numpy 4x4 transformation matrix. - """ - return np.linalg.pinv(VeloToCameraTransformation(calib)) - - -def AnnotateKITTIObjectsWithBBox3D(objects, calib): - """Add our canonical bboxes 3d format to KITTI objects. - - The annotated bboxes 3d are in the velodyne coordinate frame. - - Args: - objects: A list of KITTI objects returned by LoadLabelFile. - calib: A calibartion dictionary returned by LoadCalibrationFile. - - Returns: - The original list of KITTI objects, where each object has new keys - 'has_3d_info' indicating if the object has valid 3D bounding box data, and - 'bbox3d' which corresponds to our canonical bboxes 3d format. - """ - - # All objects will share the same transformation matrix, which we compute - # once here. - transformation_matrix = CameraToVeloTransformation(calib) - for obj in objects: - obj['bbox3d'] = _KITTIObjectToBBox3D(obj, transformation_matrix) - obj['has_3d_info'] = _KITTIObjectHas3DInfo(obj) - return objects - - -def _KITTIObjectHas3DInfo(obj): - """Check whether KITTI object has valid 3D bounding box information.""" - height, width, length = obj['dimensions'] - # KITTI raw data has -1 for all 3 dimensions when no 3D box info is present. - return not (width == -1 or length == -1 or height == -1) - - -def _KITTIObjectToBBox3D(obj, cam_to_velo_transform): - """Convert one object given the transformation matrix.""" - height, width, length = obj['dimensions'] - - # Avoid transforming objects with invalid boxes. - if not _KITTIObjectHas3DInfo(obj): - return [-1000, -1000, -1000, -1, -1, -1, -10] - - velo_xyz = np.dot(cam_to_velo_transform, np.asarray(obj['location'] + [1.])) - x, y, z = velo_xyz.tolist()[:3] - - # In the raw data, the z coordinate is at the bottom of the object, we need - # to reposition z so that it's at the center of the object. - z += height / 2. - - # Our velodyne bbox rotation goes the other direction and is rotated by - # -np.pi/2. See http://www.cvlibs.net/datasets/kitti/setup.php. - rot = -obj['rotation_y'] - rot -= np.pi / 2. - bbox3d = [x, y, z] + [length, width, height] + [rot] - - return bbox3d - - -def BBox3DToKITTIObject(bbox3d, velo_to_cam_transform): - """Convert one bbox3d into KITTI's location, dimension, and rotation_y.""" - x, y, z, length, width, height, rot = bbox3d - - # Avoid transforming objects with invalid boxes. See _KITTIObjectHas3DInfo. - if width == -1 or length == -1 or height == -1: - return [-1000, -1000, -1000], [-1, -1, -1], -10 - - # Convert our velodyne bbox rotation back to camera. Reverse the direction and - # rotate by np.pi/2. See http://www.cvlibs.net/datasets/kitti/setup.php. - rotation_y = rot + np.pi / 2. - rotation_y = -rotation_y - rotation_y = np.mod(rotation_y, 2 * np.pi) - rotation_y = np.where(rotation_y >= np.pi, rotation_y - 2 * np.pi, rotation_y) - - # Reposition z so that it is at the bottom of the object. - if height > 0: - z -= height / 2. - - camera_xyz = np.dot(velo_to_cam_transform, np.asarray([x, y, z, 1.])) - location = camera_xyz.tolist()[:3] - dimensions = height, width, length - - return location, dimensions, rotation_y diff --git a/lingvo/tasks/car/tools/kitti_data_test.py b/lingvo/tasks/car/tools/kitti_data_test.py deleted file mode 100644 index c5f4ff86..00000000 --- a/lingvo/tasks/car/tools/kitti_data_test.py +++ /dev/null @@ -1,139 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for kitti_data.""" - -from lingvo import compat as tf -from lingvo.core import test_helper -from lingvo.core import test_utils -from lingvo.tasks.car import geometry -from lingvo.tasks.car.tools import kitti_data -import numpy as np - - -class KittiDataTest(test_utils.TestCase): - - def setUp(self): - super().setUp() - self._label_file = test_helper.test_src_dir_path( - 'tasks/car/testdata/kitti_raw_label_testdata.txt') - self._calib_file = test_helper.test_src_dir_path( - 'tasks/car/testdata/kitti_raw_calib_testdata.txt') - - def testLoadLabelFile(self): - objects = kitti_data.LoadLabelFile(self._label_file) - self.assertEqual(len(objects), 7) - self.assertEqual(objects[1]['type'], 'Car') - self.assertEqual(objects[1]['truncated'], 0.0) - self.assertEqual(objects[1]['occluded'], 0) - self.assertEqual(objects[1]['alpha'], 1.85) - self.assertAllClose(objects[1]['bbox'], [387.63, 181.54, 423.81, 203.12]) - self.assertAllClose(objects[1]['dimensions'], [1.67, 1.87, 3.69]) - self.assertAllClose(objects[1]['location'], [-16.53, 2.39, 58.49]) - self.assertEqual(objects[1]['rotation_y'], 1.57) - self.assertEqual(objects[1]['score'], -1.) - - def testLoadCalibrationFile(self): - calib = kitti_data.LoadCalibrationFile(self._calib_file) - - self.assertAllEqual(calib['P0'].shape, [3, 4]) - self.assertAllEqual(calib['P1'].shape, [3, 4]) - self.assertAllEqual(calib['P2'].shape, [3, 4]) - self.assertAllEqual(calib['P3'].shape, [3, 4]) - - self.assertAllEqual(calib['R0_rect'].shape, [4, 4]) - self.assertAllEqual(calib['R0_rect'][3, :], [0., 0., 0., 1.]) - self.assertAllEqual(calib['R0_rect'][:, 3], [0., 0., 0., 1.]) - - self.assertAllEqual(calib['Tr_imu_to_velo'].shape, [4, 4]) - self.assertAllEqual(calib['Tr_imu_to_velo'][3, :], [0., 0., 0., 1.]) - self.assertAllEqual(calib['Tr_velo_to_cam'].shape, [4, 4]) - self.assertAllEqual(calib['Tr_velo_to_cam'][3, :], [0., 0., 0., 1.]) - - def testVeloToCamAndCamToVeloAreInverses(self): - calib = kitti_data.LoadCalibrationFile(self._calib_file) - velo_to_cam = kitti_data.VeloToCameraTransformation(calib) - cam_to_velo = kitti_data.CameraToVeloTransformation(calib) - self.assertAllClose(cam_to_velo.dot(velo_to_cam), np.eye(4)) - - def testAnnotateKITTIObjectsWithBBox3D(self): - objects = kitti_data.LoadLabelFile(self._label_file) - calib = kitti_data.LoadCalibrationFile(self._calib_file) - objects = kitti_data.AnnotateKITTIObjectsWithBBox3D(objects, calib) - for obj in objects: - self.assertEqual(len(obj['bbox3d']), 7) - - # atol=0.01 corresponds to a 1cm tolerance. - self.assertAllClose( - objects[0]['bbox3d'][:3], [69.72, -0.45, 0.58], atol=0.01) - self.assertAllClose(objects[0]['bbox3d'][3:], - [12.34, 2.63, 2.85, -0.01079633]) - self.assertAllEqual(objects[0]['has_3d_info'], True) - - # no 3D data - self.assertAllEqual(objects[3]['has_3d_info'], False) - - def testKITTIObjToBBoxAndInverse(self): - objects = kitti_data.LoadLabelFile(self._label_file) - calib = kitti_data.LoadCalibrationFile(self._calib_file) - for obj in objects: - bbox3d = kitti_data._KITTIObjectToBBox3D( - obj, kitti_data.CameraToVeloTransformation(calib)) - location, dimensions, rotation_y = kitti_data.BBox3DToKITTIObject( - bbox3d, kitti_data.VeloToCameraTransformation(calib)) - self.assertAllClose(obj['location'], location) - self.assertAllClose(obj['dimensions'], dimensions) - self.assertAllClose(obj['rotation_y'], rotation_y) - - def testVeloToImagePlaneTransformation(self): - objects = kitti_data.LoadLabelFile(self._label_file) - calib = kitti_data.LoadCalibrationFile(self._calib_file) - - # Only apply to object 0. - obj = objects[0] - bbox3d = kitti_data._KITTIObjectToBBox3D( - obj, kitti_data.CameraToVeloTransformation(calib)) - - # Convert to corners in our canonical space. - corners = geometry.BBoxCorners(tf.constant([[bbox3d]], dtype=tf.float32)) - with self.session(): - corners_np = self.evaluate(corners) - corners_np = corners_np.reshape([8, 3]) - - # Add homogenous coordinates. - corners_np = np.concatenate([corners_np, np.ones((8, 1))], axis=-1) - - # Apply the velo to image plane transformation. - velo_to_img = kitti_data.VeloToImagePlaneTransformation(calib) - corners_np = np.dot(corners_np, velo_to_img.T) - - # Divide by the last coordinate to recover pixel locations. - corners_np[:, 0] /= corners_np[:, 2] - corners_np[:, 1] /= corners_np[:, 2] - - # Obtain 2D bbox. - min_x = np.min(corners_np[:, 0]) - max_x = np.max(corners_np[:, 0]) - min_y = np.min(corners_np[:, 1]) - max_y = np.max(corners_np[:, 1]) - bbox = [min_x, min_y, max_x, max_y] # left, top, right, bottom. - - # This should correspond to the GT bbox in obj['bbox']. - # We use atol=0.1 here since they should close to the nearest pixel. - self.assertAllClose(bbox, obj['bbox'], atol=0.1) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/tools/kitti_exporter.py b/lingvo/tasks/car/tools/kitti_exporter.py deleted file mode 100644 index 62c0249b..00000000 --- a/lingvo/tasks/car/tools/kitti_exporter.py +++ /dev/null @@ -1,265 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Create TFRecords files from KITTI raw data. - -Parses KITTI raw data with different splits, indicated with split files. -A split file is a text file that specifies frame names to be included in the -split, with one name per line. Splits with 'test' in the filename use testing -data, while other splits use training data. This program expects KITTI raw data -in the following directory structure: - - kitti_object/ - training/ # Contains KITTI raw train data - label2/ - velodyne/ - calib/ - image_2/ - testing/ # Contains KITTI raw test data - velodyne/ - calib/ - image_2/ - splits/ # Contains split files identifying frame names in the split. - split_name.txt - -Outputs examples in TFRecords files correspond to KITTI frames with the -following format: - - # frame information - image/source_id: unique frame name e.g '000000', '000010' - - # 2D image data - image/encoded: PNG encoded string - image/height: image height - image/width: image width - image/format: 'PNG' - - # 3D velodyne pointcloud data (variable P points per frame) - pointcloud/xyz = point positions (P x 3 tensor). - pointcloud/reflectance: point reflectances (P x 1 tensor). - - # Object level data (variable N objects per frame) - object/image/bbox/xmin: min X pixel location in raw image (N x 1 tensor). - - object/image/bbox/xmax: max X pixel location in raw image (N x 1 tensor). - - object/image/bbox/ymin: min Y pixel location in raw image (N x 1 tensor). - - object/image/bbox/ymax: max Y pixel location in raw image (N x 1 tensor). - - object/label: one of {'Car', 'Pedestrian', 'Cyclist'} identifying object - class (N x 1 tensor). - - object/has_3d_info: 1 if object has valid 3D info else 0 (N x 1 tensor). - - object/occlusion: int in {0, 1, 2, 3} of occlusion state (N x 1 tensor). - - object/truncation: float in 0 (non-truncated) to 1 (truncated) (N x 1 tensor). - - object/velo/bbox/xyz: 3D bbox locations in velo frame (N x 3 tensor). - - object/velo/bbox/dim_xyz: length (dx), width (dy), height (dz) indicating - object dimensions (N x 3 tensor). - - object/velo/bbox/phi: bbox rotation in velo frame (N x 1 tensor). - - # Transformation matrices - transform/velo_to_image_plane: 3x4 matrix from velo xyz to image plane xy. - After multiplication, you need to divide by last coordinate to recover 2D - pixel locations. - - transform/velo_to_camera: 4x4 matrix from velo xyz to camera xyz. - - transform/camera_to_velo 4x4 matrix from camera xyz to velo xyz. -""" - -import contextlib -import io -import os - -from absl import app -from absl import flags -from absl import logging - -from lingvo import compat as tf -from lingvo.tasks.car.tools import kitti_data -import numpy as np -from PIL import Image - -FLAGS = flags.FLAGS - -flags.DEFINE_string('kitti_object_dir', None, - 'Path to a kitti object directory.') -flags.DEFINE_string('split', None, - 'Name of the split file to be used for parsing.') -flags.DEFINE_string('tfrecord_path', None, 'Output tfrecord path.') -flags.DEFINE_integer( - 'num_shards', 1, 'Number of output shards (between 1 and 99999). Files' - 'named {tfrecord_path}-{shard_num}-of-{total_shards}.') - - -def _ReadObjectDataset(root_dir, frame_names): - """Reads and parses KITTI dataset files into a list of TFExample protos.""" - examples = [] - - total_frames = len(frame_names) - for frame_index, frame_name in enumerate(frame_names): - image_file_path = os.path.join(root_dir, 'image_2', frame_name + '.png') - calib_file_path = os.path.join(root_dir, 'calib', frame_name + '.txt') - velo_file_path = os.path.join(root_dir, 'velodyne', frame_name + '.bin') - label_file_path = os.path.join(root_dir, 'label_2', frame_name + '.txt') - - example = tf.train.Example() - feature = example.features.feature - - # frame information - feature['image/source_id'].bytes_list.value[:] = [frame_name] - - # 2D image data - encoded_image = tf.io.gfile.GFile(image_file_path).read() - feature['image/encoded'].bytes_list.value[:] = [encoded_image] - image = np.array(Image.open(io.BytesIO(encoded_image))) - assert image.ndim == 3 - assert image.shape[2] == 3 - image_width = image.shape[1] - image_height = image.shape[0] - feature['image/width'].int64_list.value[:] = [image_width] - feature['image/height'].int64_list.value[:] = [image_height] - feature['image/format'].bytes_list.value[:] = ['PNG'] - - # 3D velodyne point data - velo_dict = kitti_data.LoadVeloBinFile(velo_file_path) - point_list = velo_dict['xyz'].ravel().tolist() - feature['pointcloud/xyz'].float_list.value[:] = point_list - reflectance_list = velo_dict['reflectance'].ravel().tolist() - feature['pointcloud/reflectance'].float_list.value[:] = reflectance_list - - # Object data - calib_dict = kitti_data.LoadCalibrationFile(calib_file_path) - if tf.io.gfile.exists(label_file_path): - # Load object labels for training data - object_dicts = kitti_data.LoadLabelFile(label_file_path) - object_dicts = kitti_data.AnnotateKITTIObjectsWithBBox3D( - object_dicts, calib_dict) - else: - # No object labels for test data - object_dicts = {} - - num_objects = len(object_dicts) - xmins = [None] * num_objects - xmaxs = [None] * num_objects - ymins = [None] * num_objects - ymaxs = [None] * num_objects - labels = [None] * num_objects - has_3d_infos = [None] * num_objects - - # 3D info - occlusions = [None] * num_objects - truncations = [None] * num_objects - xyzs = [None] * num_objects - dim_xyzs = [None] * num_objects - phis = [None] * num_objects - - for object_index, object_dict in enumerate(object_dicts): - xmins[object_index] = object_dict['bbox'][0] - xmaxs[object_index] = object_dict['bbox'][2] - ymins[object_index] = object_dict['bbox'][1] - ymaxs[object_index] = object_dict['bbox'][3] - labels[object_index] = object_dict['type'] - has_3d_infos[object_index] = 1 if object_dict['has_3d_info'] else 0 - occlusions[object_index] = object_dict['occluded'] - truncations[object_index] = object_dict['truncated'] - xyzs[object_index] = object_dict['bbox3d'][:3] - dim_xyzs[object_index] = object_dict['bbox3d'][3:6] - phis[object_index] = object_dict['bbox3d'][6] - - feature['object/image/bbox/xmin'].float_list.value[:] = xmins - feature['object/image/bbox/xmax'].float_list.value[:] = xmaxs - feature['object/image/bbox/ymin'].float_list.value[:] = ymins - feature['object/image/bbox/ymax'].float_list.value[:] = ymaxs - feature['object/label'].bytes_list.value[:] = labels - feature['object/has_3d_info'].int64_list.value[:] = has_3d_infos - feature['object/occlusion'].int64_list.value[:] = occlusions - feature['object/truncation'].float_list.value[:] = truncations - xyzs = np.array(xyzs).ravel().tolist() - feature['object/velo/bbox/xyz'].float_list.value[:] = xyzs - dim_xyzs = np.array(dim_xyzs).ravel().tolist() - feature['object/velo/bbox/dim_xyz'].float_list.value[:] = dim_xyzs - feature['object/velo/bbox/phi'].float_list.value[:] = phis - - # Transformation matrices - velo_to_image_plane = kitti_data.VeloToImagePlaneTransformation(calib_dict) - feature['transform/velo_to_image_plane'].float_list.value[:] = ( - velo_to_image_plane.ravel().tolist()) - velo_to_camera = kitti_data.VeloToCameraTransformation(calib_dict) - feature['transform/velo_to_camera'].float_list.value[:] = ( - velo_to_camera.ravel().tolist()) - cam_to_velo = kitti_data.CameraToVeloTransformation(calib_dict) - feature['transform/camera_to_velo'].float_list.value[:] = ( - cam_to_velo.ravel().tolist()) - - examples.append(example) - if frame_index % 100 == 0: - logging.info('Processed frame %d of %d.', frame_index, total_frames) - - return examples - - -def _ExportObjectDatasetToTFRecord(root_dir, split_file, tfrecord_path, - num_shards): - """Exports KITTI dataset files to TFRecord files.""" - if num_shards <= 0: - raise ValueError('TFRecord dataset must have at least one shard.') - - logging.info('Reading frame names from split_file %s.', split_file) - frame_names = [line.rstrip('\n') for line in tf.io.gfile.GFile(split_file)] - logging.info('Reading object dataset with %d frames.', len(frame_names)) - dataset = _ReadObjectDataset(root_dir, frame_names) - logging.info('Saving object dataset at %s with %d shards.', tfrecord_path, - num_shards) - - tf_record_output_filenames = [ - '{}-{:05d}-of-{:05d}'.format(tfrecord_path, index, num_shards) - for index in range(num_shards) - ] - - with contextlib.ExitStack() as exit_stack: - tf_record_writers = [ - exit_stack.enter_context(tf.io.TFRecordWriter(filename)) - for filename in tf_record_output_filenames - ] - total_examples = len(dataset) - for example_index, example in enumerate(dataset): - output_shard_index = example_index % num_shards - serialized_example = example.SerializeToString() - tf_record_writers[output_shard_index].write(serialized_example) - if example_index % 100 == 0: - logging.info('Wrote frame %d of %d.', example_index, total_examples) - - -def main(unused_argv): - split_type = 'testing' if 'test' in FLAGS.split else 'training' - root_dir = os.path.join(FLAGS.kitti_object_dir, split_type) - split_file = os.path.join(FLAGS.kitti_object_dir, 'splits', - '{}.txt'.format(FLAGS.split)) - _ExportObjectDatasetToTFRecord(root_dir, split_file, FLAGS.tfrecord_path, - FLAGS.num_shards) - - -if __name__ == '__main__': - flags.mark_flag_as_required('kitti_object_dir') - flags.mark_flag_as_required('split') - flags.mark_flag_as_required('tfrecord_path') - app.run(main) diff --git a/lingvo/tasks/car/transform_util.py b/lingvo/tasks/car/transform_util.py deleted file mode 100644 index 8d45abe3..00000000 --- a/lingvo/tasks/car/transform_util.py +++ /dev/null @@ -1,188 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Utilities for performing 3D transformations on points.""" - -import copy - -import numpy as np - - -class Box2D: - """A representation of a 2D rotated bounding box. - - Box2D is based on conventions for 3D coordinate systems where y-x are - switched. - """ - - def __init__(self, x, y, width, length, angle): - """Initialize a Box2D. - - Args: - x: center x coordinate of bounding box. - y: center y coordinate of bounding box. - width: Width of box in the x-dimension. - length: Length of box in the y-dimension. - angle: Angle in radians wrt to the direction of the longer axis. - """ - self._center = np.array([x, y]) - self._width = width - self._length = length - self._angle = angle - self._angle_v = np.array([np.cos(angle), np.sin(angle)]) - - offset = self._angle_v * (length / 2.) - self._start = self._center - offset - self._end = self._center + offset - # Compute the four corners of the rotated box. - self._corners = self._ComputeCorners() - - @property - def corners(self): - """Returns a [4, 2] numpy matrix containing the four corner points.""" - return self._corners - - def _ComputeCorners(self): - """Compute the four corners of the bounding box.""" - if self._length > 0: - perp_unit = np.array([-self._angle_v[1], self._angle_v[0]]) - else: - perp_unit = np.array([0., 0.]) - w2 = perp_unit * (self._width / 2.) - - corner_1 = np.array([self._start[0] + w2[0], self._start[1] + w2[1]]) - corner_2 = np.array([self._end[0] + w2[0], self._end[1] + w2[1]]) - corner_3 = np.array([self._end[0] - w2[0], self._end[1] - w2[1]]) - corner_4 = np.array([self._start[0] - w2[0], self._start[1] - w2[1]]) - return np.array([corner_1, corner_2, corner_3, corner_4]) - - def Extrema(self): - """Returns the extrema of the bounding box.""" - ymin = np.min(self._corners[:, 1]) - xmin = np.min(self._corners[:, 0]) - ymax = np.max(self._corners[:, 1]) - xmax = np.max(self._corners[:, 0]) - return ymin, xmin, ymax, xmax - - def Apply(self, transform): - """Apply `transform` to the current box and return a new box.""" - # Transform corner points using the transform matrix. - new_corners = [] - - # TODO(vrv): vectorize - for corner in self._corners: - # Extend to z, w. - corner = np.concatenate([corner, np.array([0., 1.])], axis=0) - # Apply the transform. - corner_adjusted = np.matmul(transform, corner) - new_corners.append(corner_adjusted[0:2]) - - new_corners = np.stack(new_corners, axis=0) - - # Compute the new center. - ymin = np.min(new_corners[:, 1]) - xmin = np.min(new_corners[:, 0]) - ymax = np.max(new_corners[:, 1]) - xmax = np.max(new_corners[:, 0]) - center_x = xmin + (xmax - xmin) / 2. - center_y = ymin + (ymax - ymin) / 2. - - # Compute the new width and length. - scale_transform = CopyTransform(transform)[0:2, 0:2] - w_l = np.array([self._width, self._length]) - new_wl = np.abs(np.matmul(scale_transform, w_l)) - new_width = new_wl[0] - new_length = new_wl[1] - - # Compute the transformed heading. - transformed_heading = TransformHeading(transform, self._angle) - return Box2D(center_x, center_y, new_width, new_length, transformed_heading) - - def AsNumpy(self): - """Return the 5DOF (xywhh) representation as a numpy array.""" - return np.array([ - self._center[0], self._center[1], self._width, self._length, self._angle - ]) - - -def TransformHeading(transform, heading): - """Compute 'heading' given transform. - - The heading provided as input is assumed to be in the original coordinate - space. When the coordinate space undergoes a transformation (e.g., with - CarToImageTransform), the heading in the new coordinate space must be - recomputed. - - We compute this by deriving the formula for the angle of transformed unit - vector defined by 'heading'. - - Args: - transform: 4x4 numpy matrix used to convert from car to image coordinates. - heading: Floating point scalar heading. - - Returns: - Heading in the transformed coordinate system. - """ - x1, y1 = np.cos(heading), np.sin(heading) - - # Transform the unit ray. - unit_ray = np.array([x1, y1, 0.0, 1.0]) - transform_no_shift = CopyTransform(transform) - transform_no_shift[0, 3] = 0 - transform_no_shift[1, 3] = 0 - transformed_ray = np.matmul(transform_no_shift, unit_ray) - x2, y2 = transformed_ray[0:2] - - # Use arctan2 to compute the new rotation angle; note that arctan2 takes 'y' - # and then 'x'. - new_heading = np.arctan2(y2, x2) - return new_heading - - -def TransformPoint(transform, x, y, z): - """Transform an x, y, z point given the 4x4 `transform`.""" - result = np.matmul(transform, np.array([x, y, z, 1.])) - return result[0], result[1], result[2] - - -def CopyTransform(transform): - """Return a copy of `transform`.""" - return copy.copy(transform) - - -def MakeCarToImageTransform(pixels_per_meter, image_ref_x, image_ref_y, - flip_axes): - """Creates a 4x4 numpy matrix for car to top down image coordinates. - - Args: - pixels_per_meter: Number of pixels that represent a meter in top down view. - image_ref_x: Number of pixels to shift the car in the x direction. - image_ref_y: Number of pixels to shift the car in the y direction. - flip_axes: Boolean indicating whether the x/y axes should be flipped[ during - the transform. - - Returns: - A 4x4 matrix transform. - """ - ppm1 = 0. if flip_axes else pixels_per_meter - ppm2 = -pixels_per_meter if flip_axes else 0. - # pyformat: disable - car_to_image_transform = np.array([ - [ppm1, ppm2, 0., image_ref_x], - [ppm2, ppm1, 0., image_ref_y], - [0., 0., 1., 0.], - [0., 0., 0., 1.]]) - # pyformat: enable - return car_to_image_transform diff --git a/lingvo/tasks/car/transform_util_test.py b/lingvo/tasks/car/transform_util_test.py deleted file mode 100644 index f2053719..00000000 --- a/lingvo/tasks/car/transform_util_test.py +++ /dev/null @@ -1,135 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for transform_util.""" - -import lingvo.compat as tf -from lingvo.tasks.car import transform_util -import numpy as np - - -class TransformUtilTest(tf.test.TestCase): - """Tests for methods in transform_util.""" - - def testMakeCarToImageTransformFlipAxesTrue(self): - transform = transform_util.MakeCarToImageTransform( - pixels_per_meter=10.0, image_ref_x=250, image_ref_y=750, flip_axes=True) - # pyformat: disable - self.assertAllClose( - np.matrix([[0., -10., 0., 250.,], - [-10., 0., 0., 750.,], - [0., 0., 1., 0.,], - [0., 0., 0., 1.,]]), - transform) - # pyformat: enable - - def testMakeCarToImageTransformFlipAxesFalse(self): - transform = transform_util.MakeCarToImageTransform( - pixels_per_meter=10.0, - image_ref_x=250, - image_ref_y=750, - flip_axes=False) - # pyformat: disable - self.assertAllClose( - np.matrix([[10., 0., 0., 250.,], - [0., 10., 0., 750.,], - [0., 0., 1., 0.,], - [0., 0., 0., 1.,]]), - transform) - # pyformat: enable - - def testTransformPoint(self): - transform = transform_util.MakeCarToImageTransform( - pixels_per_meter=10.0, - image_ref_x=250, - image_ref_y=750, - flip_axes=False) - tx, ty, tz = transform_util.TransformPoint(transform, 0.0, 1.0, 0.0) - # X gets translated. - self.assertEqual(250., tx) - # Y gets translated and scaled by pixels_per_meter. - self.assertEqual(760., ty) - self.assertEqual(0., tz) - - def testCopyTransform(self): - # Same transform as above. - transform = transform_util.MakeCarToImageTransform( - pixels_per_meter=10.0, - image_ref_x=250, - image_ref_y=750, - flip_axes=False) - # Test that copying the transform yields the same result. - copy_transform = transform_util.CopyTransform(transform) - tx, ty, tz = transform_util.TransformPoint(copy_transform, 0.0, 1.0, 0.0) - self.assertEqual(250., tx) - self.assertEqual(760., ty) - self.assertEqual(0., tz) - - def testBox2DCornerAxisAligned(self): - # Center at 1., 1., width of 1. and length of 2. - # - # The heading of 0. indicates the heading of the 'long' - # side of the box, so it is longer in the positive y direction. - box = transform_util.Box2D(1.0, 1.0, 2., 1., 0.) - self.assertAllClose(box.corners, - [[0.5, 2.], [1.5, 2.], [1.5, 0.], [0.5, 0.]]) - - def testBox2DCornerRotated(self): - # Like above but rotated 90 degrees; the order is important. - box = transform_util.Box2D(1.0, 1.0, 2., 1., np.pi / 2.) - self.assertAllClose(box.corners, - [[0., 0.5], [0., 1.5], [2., 1.5], [2., 0.5]]) - - def testBox2DTransform(self): - # Take the box from above and apply a car-image transform. - box = transform_util.Box2D(1.0, 1.0, 2., 1., 0.) - transform = transform_util.MakeCarToImageTransform( - pixels_per_meter=10.0, image_ref_x=250, image_ref_y=750, flip_axes=True) - new_box = box.Apply(transform) - - # The center flips across the x=y axis to -1, -1. After the scaling and - # translation, the box should be centered (240, 740). Because the box flips - # across the axis, the width and length get flipped from 2, 1 to [10, 20]. - # - # The flip axes should cause the heading to go from 0. to -pi/2. - self.assertAllClose([240., 740., 10., 20., -np.pi / 2.], new_box.AsNumpy()) - - # Check ymin/xmin/ymax/xmax: the rectangle is now longer in the y-dimension - # than the x-dimension. - self.assertAllClose((730., 235., 750., 245.), new_box.Extrema()) - - def testTransformHeading(self): - transform = transform_util.MakeCarToImageTransform( - pixels_per_meter=1.0, image_ref_x=123, image_ref_y=455, flip_axes=True) - - # Ray (0, 1): 90 degrees becomes -180 degrees. - self.assertAllClose(-np.pi, - transform_util.TransformHeading(transform, np.pi / 2.)) - - # Ray (1, 0): 0 degrees becomes -90 - self.assertAllClose(-np.pi / 2., - transform_util.TransformHeading(transform, 0.)) - - # (-1, 0) becomes (0, 1) or np.pi / 2. - self.assertAllClose(np.pi / 2., - transform_util.TransformHeading(transform, np.pi)) - - # (0, -1) becomes (1, 0) or 0 - self.assertAllClose(0., - transform_util.TransformHeading(transform, 1.5 * np.pi)) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/waymo/BUILD b/lingvo/tasks/car/waymo/BUILD deleted file mode 100644 index 8ce1d690..00000000 --- a/lingvo/tasks/car/waymo/BUILD +++ /dev/null @@ -1,121 +0,0 @@ -# Car related tasks. - -package(default_visibility = [ - "//visibility:public", -]) - -licenses(["notice"]) # Apache 2.0 - -py_test( - name = "waymo_ap_metric_test", - srcs = ["waymo_ap_metric_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":waymo_ap_metric", - ":waymo_metadata", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - # Implicit Waymo Open Dataset label dependency. - # Implicit Waymo Open Dataset metrics_proto dependency. - ], -) - -py_test( - name = "waymo_open_input_generator_test", - srcs = ["waymo_open_input_generator_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":waymo_open_input_generator", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "waymo_ap_metric", - srcs = [ - "waymo_ap_metric.py", - ], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/tasks/car:ap_metric", - # Implicit numpy dependency. - # Implicit Waymo Open Dataset label dependency. - # Implicit Waymo Open Dataset detection metrics dependency. - # Implicit Waymo Open Dataset py_metrics dependency. - # Implicit Waymo Open Dataset config_util_py dependency. - # Implicit Waymo Open Dataset breakdown_proto dependency. - # Implicit Waymo Open Dataset metrics_proto dependency. - ], -) - -py_library( - name = "waymo_decoder", - srcs = ["waymo_decoder.py"], - srcs_version = "PY3", - deps = [ - ":waymo_ap_metric", - ":waymo_metadata", - "//lingvo:compat", - "//lingvo/core:metrics", - "//lingvo/core:py_utils", - "//lingvo/tasks/car:base_decoder", - "//lingvo/tasks/car:detection_3d_metrics", - "//lingvo/tasks/car:transform_util", - # Implicit numpy dependency. - ], -) - -py_library( - name = "waymo_metadata", - srcs = [ - "waymo_metadata.py", - ], - srcs_version = "PY3", - deps = [ - "//lingvo/tasks/car:evaluation_metadata", - ], -) - -py_library( - name = "waymo_open_input_generator", - srcs = ["waymo_open_input_generator.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:datasource", - "//lingvo/core:hyperparams", - "//lingvo/core:py_utils", - "//lingvo/tasks/car:input_extractor", - "//lingvo/tasks/car:input_preprocessors", - # Implicit numpy dependency. - ], -) - -py_library( - name = "export_to_submission_format_lib", - srcs = ["export_to_submission_format.py"], - srcs_version = "PY3", - deps = [ - # Implicit absl.flags dependency. - "//lingvo:compat", - "//lingvo/core/ops:record_py_pb2", - # Implicit Waymo Open Dataset metrics_proto dependency. - ], -) - -py_binary( - name = "export_to_submission_format", - srcs = ["export_to_submission_format.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [":export_to_submission_format_lib"], -) diff --git a/lingvo/tasks/car/waymo/__init__.py b/lingvo/tasks/car/waymo/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/lingvo/tasks/car/waymo/export_to_submission_format.py b/lingvo/tasks/car/waymo/export_to_submission_format.py deleted file mode 100644 index 77069572..00000000 --- a/lingvo/tasks/car/waymo/export_to_submission_format.py +++ /dev/null @@ -1,138 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -r"""Read Waymo Decoder's outputs and convert to submission format. - -Example: - -python export_to_submission_format.py \ - --decoder_path=/tmp/decoder_out_000060600 \ - --output_dir=/tmp/test_decoder_output - -preds.bin and gts.bin will be found in /tmp/test_decoder_output/ dir. -""" - -import os - -from absl import flags -from lingvo import compat as tf -from lingvo.core.ops import record_pb2 -from waymo_open_dataset.protos import metrics_pb2 - -FLAGS = flags.FLAGS -flags.DEFINE_string("decoder_path", None, - "Path to the decoder output tf.Record file.") -flags.DEFINE_string("output_dir", None, "Place to write detections.") -flags.DEFINE_float("score_threshold", 0, "Ignore detections with lower score.") - - -def convert_detections(table_path): - """Convert detections in `table_path` to metric format. - - Args: - table_path: Path to TFRecord file of decoder outputs. - - Returns: - (preds, gts): metric_pb2.Objects() of predictions and groundtruths. - """ - img_ids = [] - img_id_dict = {} - for serialized in tf.io.tf_record_iterator(table_path): - record = record_pb2.Record() - record.ParseFromString(serialized) - img_id = str(tf.make_ndarray(record.fields["frame_id"])) - img_ids.append(img_id) - np_dict = {k: tf.make_ndarray(v) for k, v in record.fields.items()} - img_id_dict[img_id] = np_dict - - preds = metrics_pb2.Objects() - gts = metrics_pb2.Objects() - for img_id in img_ids: - # Extract the underlying context string and timestamp - # from the image id. - # - # TODO(vrv): Consider embedding these values into the decoder output - # individually. - context_name = img_id[2:img_id.rindex("_")] - timestamp = int(img_id[img_id.rindex("_") + 1:-1]) - - np_dict = img_id_dict[img_id] - pred_bboxes = np_dict["bboxes"] # [max boxes, 7] - pred_scores = np_dict["scores"] # [max_boxes] - gt_bboxes = np_dict["gt_bboxes"] # [num_gt_boxes, 7] - gt_labels = np_dict["gt_labels"] # [num_gt_boxes] - class_ids = np_dict["class_ids"] # [max_boxes] - - def _add_box(label, box_vec): - label.box.center_x = box_vec[0] - label.box.center_y = box_vec[1] - label.box.center_z = box_vec[2] - label.box.length = box_vec[3] - label.box.width = box_vec[4] - label.box.height = box_vec[5] - label.box.heading = box_vec[6] - - num_gts = gt_bboxes.shape[0] - for gt_idx in range(num_gts): - gt_object = metrics_pb2.Object() - gt_object.context_name = context_name - gt_object.frame_timestamp_micros = timestamp - label = gt_object.object - _add_box(label, gt_bboxes[gt_idx]) - label.type = gt_labels[gt_idx] - # We should fill in the difficulty level once we want to measure the - # breakdown by LEVEL. - label.detection_difficulty_level = 0 - gts.objects.append(gt_object) - - num_pds = pred_bboxes.shape[0] - for pd_idx in range(num_pds): - score = pred_scores[pd_idx] - if score < FLAGS.score_threshold: - continue - pd_object = metrics_pb2.Object() - pd_object.context_name = context_name - pd_object.frame_timestamp_micros = timestamp - pd_object.score = score - label = pd_object.object - _add_box(label, pred_bboxes[pd_idx]) - label.type = class_ids[pd_idx] - preds.objects.append(pd_object) - - return preds, gts - - -def main(argv): - if len(argv) > 1: - raise tf.app.UsageError("Too many command-line arguments.") - - preds, gts = convert_detections(FLAGS.decoder_path) - - if not tf.io.gfile.exists(FLAGS.output_dir): - tf.io.gfile.mkdir(FLAGS.output_dir) - - # Write the predictions and gts into individual files. - # - # The outputs can then be passed to the official metrics implementations and - # server. - with tf.io.gfile.GFile(os.path.join(FLAGS.output_dir, "preds.bin"), "w") as f: - f.write(preds.SerializeToString()) - - with tf.io.gfile.GFile(os.path.join(FLAGS.output_dir, "gts.bin"), "w") as f: - f.write(gts.SerializeToString()) - - -if __name__ == "__main__": - tf.app.run(main) diff --git a/lingvo/tasks/car/waymo/tools/BUILD b/lingvo/tasks/car/waymo/tools/BUILD deleted file mode 100644 index 47011c43..00000000 --- a/lingvo/tasks/car/waymo/tools/BUILD +++ /dev/null @@ -1,40 +0,0 @@ -package(default_visibility = [ - "//visibility:public", -]) - -licenses(["notice"]) # Apache 2.0 - -py_library( - name = "waymo_proto_to_tfe", - srcs = [ - "waymo_proto_to_tfe.py", - ], - srcs_version = "PY3", - deps = [ - # Implicit apache_beam dependency. - "//lingvo:compat", - "//lingvo/tasks/car:geometry", - # Implicit numpy dependency. - # Implicit Waymo Open Dataset proto dependency. - # Implicit Waymo Open Dataset range_image_utils dependency. - # Implicit Waymo Open Dataset transform_utils dependency. - ], -) - -py_binary( - name = "generate_waymo_tf", - srcs = [ - "generate_waymo_tf.py", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":waymo_proto_to_tfe", - # Implicit absl.app dependency. - # Implicit absl.flags dependency. - # Implicit apache_beam dependency. - "//lingvo:compat", - "//lingvo/tools:beam_utils", - # Implicit Waymo Open Dataset proto dependency. - ], -) diff --git a/lingvo/tasks/car/waymo/tools/__init__.py b/lingvo/tasks/car/waymo/tools/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/lingvo/tasks/car/waymo/tools/generate_waymo_tf.py b/lingvo/tasks/car/waymo/tools/generate_waymo_tf.py deleted file mode 100644 index fdc31261..00000000 --- a/lingvo/tasks/car/waymo/tools/generate_waymo_tf.py +++ /dev/null @@ -1,77 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -r"""Tool to convert Waymo Open Dataset to tf.Examples. - -An example of running on Cloud DataFlow: (not verified working yet). - -export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json - -path/to/generate_waymo_tf.py \ - --input_file_pattern=gs://path/to/waymo/inputs \ - --output_filebase=gs://$BUCKET/waymo/training/output@1000 -- \ - --project=$PROJECT \ - --temp_location=$TEMP_DIR \ - --runner=DataflowRunner -""" - -from absl import app -from absl import flags - -import apache_beam as beam -from lingvo import compat as tf -from lingvo.tasks.car.waymo.tools import waymo_proto_to_tfe -from lingvo.tools import beam_utils -from waymo_open_dataset import dataset_pb2 - -tf.enable_eager_execution() - -flags.DEFINE_string('input_file_pattern', None, 'Path to read input') -flags.DEFINE_string('output_filebase', None, 'Path to write output') - -FLAGS = flags.FLAGS - - -def main(argv): - beam_utils.BeamInit() - - assert FLAGS.input_file_pattern - assert FLAGS.output_filebase - - # Construct pipeline options from argv. - options = beam.options.pipeline_options.PipelineOptions(argv[1:]) - - reader = beam_utils.GetReader( - 'tfrecord', - FLAGS.input_file_pattern, - value_coder=beam.coders.ProtoCoder(dataset_pb2.Frame)) - - writer = beam_utils.GetWriter( - 'tfrecord', - file_pattern=FLAGS.output_filebase, - value_coder=beam.coders.ProtoCoder(tf.train.Example)) - - emitter_fn = beam_utils.GetEmitterFn('tfrecord') - with beam_utils.GetPipelineRoot(options=options) as root: - _ = ( - root - | 'Read' >> reader - | 'ConvertToTFExample' >> beam.ParDo( - waymo_proto_to_tfe.WaymoOpenDatasetConverter(emitter_fn)) - | 'Write' >> writer) - - -if __name__ == '__main__': - app.run(main) diff --git a/lingvo/tasks/car/waymo/tools/waymo_proto_to_tfe.py b/lingvo/tasks/car/waymo/tools/waymo_proto_to_tfe.py deleted file mode 100644 index d7f51d84..00000000 --- a/lingvo/tasks/car/waymo/tools/waymo_proto_to_tfe.py +++ /dev/null @@ -1,592 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -r"""Library to convert Waymo Open Dataset to tf.Examples. - -Generates a tf.Example proto for every dataset_pb2.Frame containing -the following keys and their values: - -* Frame-level metadata - -run_segment: string - The identifier of the driving sequence in the dataset. - -run_start_offset: int64 - The start offset within the run_segment sequence. - -time_of_day: string - Categorical description of time of day, e.g., "Day". - -location: string - Categorical description of geographical location, e.g., -"location_sf". - -weather: string - Categorical description of weather of scene, e.g., "sunny". - -pose: float: 4x4 transformation matrix for converting from "world" coordinates -to SDC coordinates. - -* Lasers - -There are 5 LIDAR sensors: "TOP", "SIDE_LEFT", "SIDE_RIGHT", "FRONT", "REAR". -Each LIDAR currently provides two returns, "ri1" and "ri2" for the first and -second returns of each shot. - -For every $LASER and $RI, we embed the raw range image: - -$LASER_$RI: float - flattened range image data of shape [H, W, C] from the -original proto. - -$LASER_$RI_shape: int64 - shape of the range image. - -For every lidar $LASER, we extract the calibrations: - -$LASER_beam_inclinations: float - List of beam angle inclinations for TOP -LIDAR (non-uniform). - -$LASER_beam_inclination_min: float - Minimum beam inclination for uniform -LIDARs. - -$LASER_beam_inclination_max: float - Maximum beam inclination for uniform -LIDARs. - -$LASER_extrinsics: float - 4x4 transformation matrix for converting from -SDC coordinates to LIDAR coordinates. - -The TOP LIDAR currently has a per-pixel range image pose to accommodate for -rolling shutter effects when projecting to 3D cartesian coordinates. We -embed this range image pose as TOP_pose. - -To allow for easier use, we also project all $LASERs to a stacked 3D cartesian -coordinate point cloud as: - -laser_$LASER_$RI - float: An [N, 6] matrix where there are N total points, -the first three dimensions are the x, y, z caresian coordinates, and the last -three dimensions are the intensity, elongation, and "is_in_no_label_zone" -bit for each point. - -* Camera images - -There are 5 cameras in the dataset: "FRONT", "FRONT_LEFT", "FRONT_RIGHT", -"SIDE_LEFT", and "SIDE_RIGHT". - -For each $CAM, we store: - -image_$CAM: string - Scalar Png format camera image. - -image_$CAM_shape: int64 - [3] - Vector containing the shape of the camera -image as [height, width, channels]. - -image_$CAM_pose: float - [4, 4] Matrix transformation for converting from -world coordinates to camera center. - -image_$CAM_pose: float - Scalar timestamp offset of when image was taken. - -image_$CAM_shutter: float - Scalar shutter value. - -image_$CAM_velocity: float - [6] Vector describing velocity of camera for -rolling shutter adjustment. See original proto for details. - -image_%CAM_camera_trigger_time: Scalar float for when camera was triggered. - -image_$CAM_camera_readout_done_time: Scalar float for when camera image finished -reading out data. - -camera_$CAM_extrinsics: float - 4x4 pose transformation for converting from -camera center coordinates to 2d projected view. - -camera_$CAM_intrinsics: float - [9] intrinsics transformation for converting -from camera center coordinates to 2d projected view. - -camera_$CAM_width: int64 - Scalar width of image. - -camera_$CAM_height: int64 - Scalar height of image. - -camera_$CAM_rolling_shutter_direction: int64 - Scalar value indicating the -direction of the rolling shutter adjustment. - -* Labels - -For each frame, we store the following label information for the M bounding -boxes in the frame. - -labels: int64 - [M] - The integer label class for every 3D bounding box -corresponding to the enumeration defined in the proto. - -label_ids: string - [M] - The unique label string identifying each labeled -object. This can be used for associating the same object across frames of the -same run segment. - -bboxes_3d: float - A flattened [M, 7] matrix where there are M boxes in the -frame, and each box is defined by a 7-DOF format - [center_x ,center_y, -center_z, length, width, height, heading]. - -label_metadata: floating point - A flattened [M, 4] matrix where there are -M boxes in the frame, and each md entry is the [speed_x, speed_y, accel_x, -accel_y] of the object. - -bboxes_3d_num_points: int64 - [M] - The number of points that fall into each -3D bounding box: can be used for computing the difficulty of each bounding -box. - -detection_difficulties: int64 - DO NOT USE FOR EVALUATION. Indicates whether the -labelers have determined that the object is of LEVEL_2 difficulty. -Should be used jointly with num_points above to set the difficulty level, -which we save in `single_frame_detection_difficulties`. Because it does not -include information about the number of points in its calculation, -it is an incomplete definition of difficulty and will not correspond to the -leaderboard if used to calculate metrics. - -single_frame_detection_difficulties: int64 - Indicates the difficulty level as -either LEVEL_1 (1), or LEVEL_2 (2), or IGNORE (999). We first ignore all 3D -labels without any LiDAR points. Next, we assign LEVEL_2 to examples where -either the labeler annotates as hard or if the example has <= 5 LiDAR points. -Finally, the rest of the examples are assigned to LEVEL_1. - -tracking_difficulties: int64 - Indicates whether the labelers have determined -that the tracked object is of LEVEL_2 difficulty. - -nlz_proto_strs: string - Vector of NoLabelZone polygon protos. Currently -unused. -""" - -import zlib - -import apache_beam as beam -from lingvo import compat as tf -import numpy as np -from waymo_open_dataset import dataset_pb2 -from waymo_open_dataset.utils import range_image_utils -from waymo_open_dataset.utils import transform_utils - - -class WaymoOpenDatasetConverter(beam.DoFn): - """Converts WaymoOpenDataset into tf.Examples. See file docstring.""" - - def __init__(self, emitter_fn): - self._emitter_fn = emitter_fn - - def process(self, item): - """Convert 'item' into tf.Example format.""" - output = tf.train.Example() - feature = output.features.feature - - # Convert run segment - run_segment = item.context.name - run_start_offset = item.timestamp_micros - key = run_segment + '_' + str(run_start_offset) - feature['run_segment'].bytes_list.value[:] = [ - tf.compat.as_bytes(run_segment) - ] - feature['run_start_offset'].int64_list.value[:] = [run_start_offset] - - # Extract metadata about frame. - feature['time_of_day'].bytes_list.value[:] = [ - tf.compat.as_bytes(item.context.stats.time_of_day) - ] - feature['location'].bytes_list.value[:] = [ - tf.compat.as_bytes(item.context.stats.location) - ] - feature['weather'].bytes_list.value[:] = [ - tf.compat.as_bytes(item.context.stats.weather) - ] - - # Convert pose: a 4x4 transformation matrix. - feature['pose'].float_list.value[:] = list(item.pose.transform) - self.frame_pose = tf.convert_to_tensor( - np.reshape(np.array(item.pose.transform), [4, 4]), dtype=tf.float32) - - # Extract laser names. - laser_names = [] - for laser in item.lasers: - laser_name = laser.name - real_name = dataset_pb2.LaserName.Name.Name(laser_name) - laser_names += [real_name] - - # Extract laser data (range images) and the calibrations. - self.extract_lasers(feature, item.lasers) - - self.extract_laser_calibrations(feature, item.context.laser_calibrations) - - range_image_pose = self._get_range_image_pose(item.lasers) - feature['TOP_pose'].float_list.value[:] = range_image_pose.numpy().reshape( - [-1]) - - # From the range images, also turn them into 3D point clouds. - self.add_point_cloud(feature, laser_names, range_image_pose) - merged_pointcloud_xyz = self._get_merged_pointcloud(feature, laser_names) - - self.add_labels(feature, item.laser_labels, merged_pointcloud_xyz) - self.add_no_label_zones(feature, item.no_label_zones) - - camera_calibrations_dict = ({ - camera_calibration.name: camera_calibration - for camera_calibration in item.context.camera_calibrations - }) - # Extract camera image data and the calibrations. - self.extract_camera_images(feature, item.images, camera_calibrations_dict) - self.extract_camera_calibrations(feature, - list(camera_calibrations_dict.values())) - - return self._emitter_fn(key, output) - - def _get_range_image_pose(self, lasers): - """Fetches the per-pixel pose information for the range image.""" - range_image_gbr_pose = None - for laser in lasers: - if laser.name != dataset_pb2.LaserName.TOP: - continue - pose_str = zlib.decompress(laser.ri_return1.range_image_pose_compressed) - # Deserialize from MatrixFloat serialization. - range_image_gbr_pose = dataset_pb2.MatrixFloat() - range_image_gbr_pose.ParseFromString(pose_str) - - assert range_image_gbr_pose is not None - shape = list(range_image_gbr_pose.shape.dims) - range_image_gbr_pose_tensor = np.array( - range_image_gbr_pose.data).reshape(shape) - range_image_gbr_pose_tensor_rotation = transform_utils.get_rotation_matrix( - range_image_gbr_pose_tensor[..., 0], - range_image_gbr_pose_tensor[..., 1], - range_image_gbr_pose_tensor[..., 2]) - range_image_gbr_pose_tensor_translation = range_image_gbr_pose_tensor[..., - 3:] - range_image_gbr_pose_tensor = transform_utils.get_transform( - range_image_gbr_pose_tensor_rotation, - range_image_gbr_pose_tensor_translation) - - assert range_image_gbr_pose_tensor.shape == (64, 2650, 4, 4) - return range_image_gbr_pose_tensor - - def _parse_range_image(self, range_image): - """Parse range_image proto and convert to MatrixFloat form.""" - if range_image.range_image_compressed: - ri_str = zlib.decompress(range_image.range_image_compressed) - # Deserialize from MatrixFloat serialization. - ri = dataset_pb2.MatrixFloat() - ri.ParseFromString(ri_str) - else: - ri = range_image.range_image - return ri - - def extract_camera_images(self, feature, camera_images, - camera_calibrations_dict): - """Extract the images into the tf.Example feature map. - - Args: - feature: A tf.Example feature map. - camera_images: A repeated car.open_dataset.CameraImage proto. - camera_calibrations_dict: A dictionary maps camera name to - car.open_dataset.CameraCalibration proto. - """ - for camera_image in camera_images: - camera_name = camera_image.name - camera_calibration = camera_calibrations_dict[camera_name] - real_name = dataset_pb2.CameraName.Name.Name(camera_name) - feature['image_%s' % real_name].bytes_list.value[:] = [ - tf.compat.as_bytes(camera_image.image) - ] - feature['image_%s_shape' % real_name].int64_list.value[:] = ([ - camera_calibration.height, camera_calibration.width, 3 - ]) - feature['image_%s_pose' % real_name].float_list.value[:] = ( - list(camera_image.pose.transform)) - velocity = camera_image.velocity - feature['image_%s_velocity' % real_name].float_list.value[:] = ([ - velocity.v_x, velocity.v_y, velocity.v_z, velocity.w_x, velocity.w_y, - velocity.w_z - ]) - feature['image_%s_pose_timestamp' % - real_name].float_list.value[:] = ([camera_image.pose_timestamp]) - feature['image_%s_shutter' % - real_name].float_list.value[:] = ([camera_image.shutter]) - feature['image_%s_camera_trigger_time' % - real_name].float_list.value[:] = ([ - camera_image.camera_trigger_time - ]) - feature['image_%s_camera_readout_done_time' % - real_name].float_list.value[:] = ([ - camera_image.camera_readout_done_time - ]) - - def extract_camera_calibrations(self, feature, camera_calibrations): - """Extract the camera calibrations into the tf.Example feature map. - - Args: - feature: A tf.Example feature map. - camera_calibrations: A CameraCalibration proto from the Waymo Dataset. - """ - for camera_calibration in camera_calibrations: - camera_name = camera_calibration.name - real_name = dataset_pb2.CameraName.Name.Name(camera_name) - - feature['camera_%s_extrinsics' % real_name].float_list.value[:] = list( - camera_calibration.extrinsic.transform) - feature['camera_%s_intrinsics' % real_name].float_list.value[:] = list( - camera_calibration.intrinsic) - feature['camera_%s_width' % - real_name].int64_list.value[:] = [camera_calibration.width] - feature['camera_%s_height' % - real_name].int64_list.value[:] = [camera_calibration.height] - feature['camera_%s_rolling_shutter_direction' % - real_name].int64_list.value[:] = [ - camera_calibration.rolling_shutter_direction - ] - - def extract_lasers(self, feature, lasers): - """Extract the lasers from range_images into the tf.Example feature map. - - Args: - feature: A tf.Example feature map. - lasers: A repeated car.open_dataset.Laser proto. - """ - for laser in lasers: - ri1 = self._parse_range_image(laser.ri_return1) - ri2 = self._parse_range_image(laser.ri_return2) - - # Add the range image data (flattened) and their original shape - # to the output feature map. - laser_name = laser.name - real_name = dataset_pb2.LaserName.Name.Name(laser_name) - feature['%s_ri1' % real_name].float_list.value[:] = ri1.data - feature['%s_ri1_shape' % real_name].int64_list.value[:] = ri1.shape.dims - feature['%s_ri2' % real_name].float_list.value[:] = ri2.data - feature['%s_ri2_shape' % real_name].int64_list.value[:] = ri2.shape.dims - - def extract_laser_calibrations(self, feature, laser_calibrations): - """Extract the laser calibrations into the tf.Example feature map. - - Args: - feature: A tf.Example feature map. - laser_calibrations: A LaserCalibrations proto from the Waymo Dataset. - """ - for laser_calibration in laser_calibrations: - laser_name = laser_calibration.name - real_name = dataset_pb2.LaserName.Name.Name(laser_name) - feature['%s_beam_inclinations' % real_name].float_list.value[:] = ( - laser_calibration.beam_inclinations) - feature['%s_beam_inclination_min' % real_name].float_list.value[:] = ([ - laser_calibration.beam_inclination_min - ]) - feature['%s_beam_inclination_max' % real_name].float_list.value[:] = ([ - laser_calibration.beam_inclination_max - ]) - feature['%s_extrinsics' % real_name].float_list.value[:] = list( - laser_calibration.extrinsic.transform) - - def add_point_cloud(self, feature, laser_names, range_image_pose): - """Convert the range images in `feature` to 3D point clouds. - - Adds the point cloud data to the tf.Example feature map. - - Args: - feature: A tf.Example feature map. - laser_names: A list of laser names (e.g., 'TOP', 'REAR', 'SIDE_LEFT'). - range_image_pose: A range image pose Tensor for the GBR. - """ - for laser_name in laser_names: - beam_inclinations = np.array(feature['%s_beam_inclinations' % - laser_name].float_list.value[:]) - # beam_inclinations will be populated if there is a non-uniform - # beam configuration (e.g., for the TOP lasers). Others that have - # uniform beam inclinations are only parameterized by the min and max. - # We use these min and max if the beam_inclinations are not present, - # and turn them into a uniform inclinations array. - if beam_inclinations.size == 0: - beam_inclination_min = feature['%s_beam_inclination_min' % - laser_name].float_list.value[:] - beam_inclination_max = feature['%s_beam_inclination_max' % - laser_name].float_list.value[:] - - laser_ri_name = '%s_ri1' % laser_name - range_image_shape = feature[laser_ri_name + - '_shape'].int64_list.value[:] - height = tf.cast(range_image_shape[0], tf.float32) - - beam_inclinations = tf.constant( - [beam_inclination_min[0], beam_inclination_max[0]]) - beam_inclinations = range_image_utils.compute_inclination( - beam_inclinations, height) - - beam_extrinsics = np.array( - feature['%s_extrinsics' % laser_name].float_list.value[:]).reshape( - 4, 4) - - for ri_type in ['ri1', 'ri2']: - laser_ri_name = '%s_%s' % (laser_name, ri_type) - # For each of the 4 features of the lasers: - range_image = np.array(feature[laser_ri_name].float_list.value[:]) - range_image_shape = feature[laser_ri_name + - '_shape'].int64_list.value[:] - range_image = range_image.reshape(range_image_shape) - # Compute mask. At the moment, invalid values in the range image - # representation are indicated via a -1. entry. Callers are expected - # to create this mask when passing into the conversion function below. - range_image_mask = range_image[..., 0] >= 0 - - # Get the 'range' feature from the range images. - range_image_range = range_image[..., 0] - - # Call utility to convert point cloud to cartesian coordinates. - # - # API expects a batch dimension for all inputs. - batched_pixel_pose = None - batched_frame_pose = None - # At the moment, only the GBR has per-pixel pose. - if laser_name == 'TOP': - batched_pixel_pose = range_image_pose[tf.newaxis, ...] - batched_frame_pose = self.frame_pose[tf.newaxis, ...] - - batched_range_image_range = tf.convert_to_tensor( - range_image_range[np.newaxis, ...], dtype=tf.float32) - batched_extrinsics = tf.convert_to_tensor( - beam_extrinsics[np.newaxis, ...], dtype=tf.float32) - batched_inclinations = tf.convert_to_tensor( - beam_inclinations[np.newaxis, ...], dtype=tf.float32) - - batched_inclinations = tf.reverse(batched_inclinations, axis=[-1]) - - range_image_cartesian = ( - range_image_utils.extract_point_cloud_from_range_image( - batched_range_image_range, - batched_extrinsics, - batched_inclinations, - pixel_pose=batched_pixel_pose, - frame_pose=batched_frame_pose)) - - points_xyz = tf.gather_nd(range_image_cartesian[0], - tf.where(range_image_mask)) - - # Fetch the features corresponding to each xyz coordinate and - # concatentate them together. - points_features = tf.cast( - tf.gather_nd(range_image[..., 1:], tf.where(range_image_mask)), - tf.float32) - points_data = tf.concat([points_xyz, points_features], axis=-1) - - # Add laser feature to output. - # - # Skip embedding shape since we assume that all points have six features - # and so we can reconstruct the number of points. - points_list = list(points_data.numpy().reshape([-1])) - feature['laser_%s' % laser_ri_name].float_list.value[:] = points_list - - def _get_merged_pointcloud(self, feature, laser_names): - """From the feature list, get a merged pointcloud of all laser sources.""" - points_list = [] - for laser_name in laser_names: - for ri_type in ['ri1', 'ri2']: - laser_ri_name = '%s_%s' % (laser_name, ri_type) - points_xyz = np.array(feature['laser_%s' % - laser_ri_name].float_list.value[:]) - points_list.append(np.reshape(points_xyz, (-1, 3))) - return np.concatenate(points_list, axis=0) - - def _single_frame_detection_difficulty(self, human_difficulty, num_points): - """Create the `single_frame_detection_difficulty` field. - - When labeling, humans have the option to label a particular frame's bbox - as difficult, which overrides the normal number of points based - definition. Additionally, boxes with 0 points are ignored by the metric - code. - - Args: - human_difficulty: What the human raters labeled the difficulty as. This is - from the detection_difficulty_level field, and will be either 0 (default - value, which is UKNOWN in the proto enum) or 2 (LEVEL_2 difficulty). - num_points: The number of points in the bbox. - - Returns: - single_frame_detection_difficulty: The single frame detection difficulty - per the Waymo Open Dataset paper's definition. - """ - if num_points <= 0: - return 999 - - if human_difficulty: - return human_difficulty - - if num_points <= 5: - return 2 - else: - return 1 - - def add_labels(self, feature, labels, points_xyz): - """Add 3d bounding box labels into the output feature map. - - Args: - feature: A tf.Example feature map. - labels: A repeated car.open_dataset.Label proto. - points_xyz: A numpy array of shape [-1, 3] with the pointcloud. This is - used to calculate the number of points in each 3D bounding box. - """ - label_classes = [] - label_ids = [] - detection_difficulty_levels = [] - tracking_difficulty_levels = [] - bboxes_3d_num_points = [] - single_frame_detection_difficulty_levels = [] - bboxes = [] - label_md = [] - - for label in labels: - box = label.box - bbox_3d = [ - box.center_x, box.center_y, box.center_z, box.length, box.width, - box.height, box.heading - ] - md = [ - label.metadata.speed_x, label.metadata.speed_y, - label.metadata.accel_x, label.metadata.accel_y - ] - label_md += md - bboxes += bbox_3d - label_classes += [label.type] - label_ids += [tf.compat.as_bytes(label.id)] - detection_difficulty_levels += [label.detection_difficulty_level] - tracking_difficulty_levels += [label.tracking_difficulty_level] - bboxes_3d_num_points += [label.num_lidar_points_in_box] - - # Compute the single frame difficulty level per object. - human_labeler_difficulty = label.detection_difficulty_level - num_points = bboxes_3d_num_points[-1] - single_frame_detection_difficulty = ( - self._single_frame_detection_difficulty(human_labeler_difficulty, - num_points)) - single_frame_detection_difficulty_levels += [ - single_frame_detection_difficulty - ] - - bboxes = np.array(bboxes).reshape(-1) - label_md = np.array(label_md).reshape(-1) - feature['labels'].int64_list.value[:] = label_classes - feature['label_ids'].bytes_list.value[:] = label_ids - feature['detection_difficulties'].int64_list.value[:] = ( - detection_difficulty_levels) - feature['single_frame_detection_difficulties'].int64_list.value[:] = ( - single_frame_detection_difficulty_levels) - feature['tracking_difficulties'].int64_list.value[:] = ( - tracking_difficulty_levels) - feature['bboxes_3d'].float_list.value[:] = list(bboxes) - feature['label_metadata'].float_list.value[:] = list(label_md) - feature['bboxes_3d_num_points'].int64_list.value[:] = (bboxes_3d_num_points) - - def add_no_label_zones(self, feature, no_label_zones): - """Add no label zones into the output feature map. - - Args: - feature: A tf.Example feature map. - no_label_zones: A repeated car.open_dataset.Polygon2dProto proto. - """ - nlz_proto_strs = [] - for nlz in no_label_zones: - nlz_proto_strs += [tf.compat.as_bytes(nlz.SerializeToString())] - feature['no_label_zones'].bytes_list.value[:] = nlz_proto_strs diff --git a/lingvo/tasks/car/waymo/waymo_ap_metric.py b/lingvo/tasks/car/waymo/waymo_ap_metric.py deleted file mode 100644 index 2ac804d3..00000000 --- a/lingvo/tasks/car/waymo/waymo_ap_metric.py +++ /dev/null @@ -1,456 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Average Precision metric class for Waymo open dataset. - -The Waymo library provides a metrics breakdown API for a set of breakdowns -implemented in their library. This wrapper uses our basic abstraction for -building AP metrics but only allows breakdowns that are supported in the -Waymo breakdown API. Should you want other breakdowns, consider using the -standard AP metrics implementation with our custom breakdowns. -""" - -from lingvo import compat as tf -from lingvo.core import plot -from lingvo.core import py_utils -from lingvo.tasks.car import ap_metric -from lingvo.tasks.car import breakdown_metric - -import numpy as np -from waymo_open_dataset import label_pb2 -from waymo_open_dataset.metrics.ops import py_metrics_ops -from waymo_open_dataset.metrics.python import config_util_py as config_util -from waymo_open_dataset.protos import breakdown_pb2 -from waymo_open_dataset.protos import metrics_pb2 - - -def _BuildWaymoMetricConfig(metadata, box_type, waymo_breakdown_metrics): - """Build the Config proto for Waymo's metric op.""" - config = metrics_pb2.Config() - # config.num_desired_score_cutoffs = metadata.NumberOfPrecisionRecallPoints() - num_pr_points = metadata.NumberOfPrecisionRecallPoints() - config.score_cutoffs.extend( - [i * 1.0 / (num_pr_points - 1) for i in range(num_pr_points)]) - config.matcher_type = metrics_pb2.MatcherProto.Type.TYPE_HUNGARIAN - if box_type == '2d': - config.box_type = label_pb2.Label.Box.Type.TYPE_2D - else: - config.box_type = label_pb2.Label.Box.Type.TYPE_3D - # Default values - config.iou_thresholds[:] = [0.7, 0.7, 0.7, 0.7, 0.7] - for class_name, threshold in metadata.IoUThresholds().items(): - cls_idx = metadata.ClassNames().index(class_name) - config.iou_thresholds[cls_idx] = threshold - # Run on all the data for 2 difficulty levels - config.breakdown_generator_ids.append(breakdown_pb2.Breakdown.ONE_SHARD) - difficulty = metrics_pb2.Difficulty() - difficulty.levels.append(label_pb2.Label.DifficultyLevel.Value('LEVEL_1')) - difficulty.levels.append(label_pb2.Label.DifficultyLevel.Value('LEVEL_2')) - config.difficulties.append(difficulty) - - # Add extra breakdown metrics. - for breakdown_value in waymo_breakdown_metrics: - breakdown_id = breakdown_pb2.Breakdown.GeneratorId.Value(breakdown_value) - config.breakdown_generator_ids.append(breakdown_id) - difficulty = metrics_pb2.Difficulty() - difficulty.levels.append(label_pb2.Label.DifficultyLevel.Value('LEVEL_1')) - difficulty.levels.append(label_pb2.Label.DifficultyLevel.Value('LEVEL_2')) - config.difficulties.append(difficulty) - return config - - -class WaymoAPMetrics(ap_metric.APMetrics): - """The Waymo Open Dataset implementation of AP metric.""" - - @classmethod - def Params(cls, metadata): - """Params builder for APMetrics.""" - p = super().Params(metadata) - p.Define( - 'waymo_breakdown_metrics', [], - 'List of extra waymo breakdown metrics when computing AP. These ' - 'should match the names of the proto entries in metrics.proto, such ' - 'as `RANGE` or `OBJECT_TYPE`.') - return p - - def __init__(self, params): - super().__init__(params) - self._waymo_metric_config = _BuildWaymoMetricConfig( - self.metadata, self.params.box_type, - self.params.waymo_breakdown_metrics) - # Compute only waymo breakdown metrics. - breakdown_names = config_util.get_breakdown_names_from_config( - self._waymo_metric_config) - waymo_params = WaymoBreakdownMetric.Params().Set( - metadata=self.metadata, breakdown_list=breakdown_names) - self._breakdown_metrics['waymo'] = WaymoBreakdownMetric(waymo_params) - - # Remove the base metric. - del self._breakdown_metrics['difficulty'] - - def _GetData(self, - classid, - difficulty=None, - distance=None, - num_points=None, - rotation=None): - """Returns groundtruth and prediction for the classid in a NestedMap. - - Args: - classid: int32 specifying the class - difficulty: Not used. - distance: int32 specifying a binned Euclidean distance of the ground truth - bounding box. If None is specified, all distances are selected. - num_points: int32 specifying a binned number of laser points within the - ground truth bounding box. If None is specified, all boxes are selected. - rotation: int32 specifying a binned rotation within the ground truth - bounding box. If None is specified, all boxes are selected. - - Returns: - NestedMap containing iou_threshold, groundtruth and predictions for - specified, classid, difficulty level and binned distance. If no bboxes - are found with these parameters, returns None. - """ - del difficulty - assert classid > 0 and classid < self.metadata.NumClasses() - - g = self._LoadBoundingBoxes( - 'groundtruth', - classid, - distance=distance, - num_points=num_points, - rotation=rotation) - # Note that we do not specify num_points for predictions because only - # groundtruth boxes contain points. - p = self._LoadBoundingBoxes( - 'prediction', classid, distance, num_points=None, rotation=rotation) - if g is None or p is None: - return None - - gt_boxes = g.boxes - gt_imgids = g.imgids - gt_speeds = g.speeds - iou_threshold = self._iou_thresholds[self.metadata.ClassNames()[classid]] - - return py_utils.NestedMap( - iou_threshold=iou_threshold, - gt=py_utils.NestedMap( - imgid=gt_imgids, - bbox=gt_boxes, - speed=gt_speeds, - difficulty=g.difficulties), - pd=py_utils.NestedMap(imgid=p.imgids, bbox=p.boxes, score=p.scores)) - - def _BuildMetric(self, feed_data, classid): - """Construct tensors and the feed_dict for Waymo metric op. - - Args: - feed_data: a NestedMap returned by _GetData(). - classid: integer. - - Returns: - A tuple of 3 dicts: - - - scalar_metrics: a dict mapping all the metric names to fetch tensors. - - curves: a dict mapping all the curve names to fetch tensors. - - feed_dict: a dict mapping the tensors in feed_tensors to feed values. - """ - breakdown_names = config_util.get_breakdown_names_from_config( - self._waymo_metric_config) - if feed_data is None: - dummy_scalar = tf.constant(np.nan) - dummy_curve = tf.zeros([self.metadata.NumberOfPrecisionRecallPoints(), 2], - tf.float32) - scalar_metrics = {'ap': dummy_scalar, 'ap_ha_weighted': dummy_scalar} - curve_metrics = {'pr': dummy_curve, 'pr_ha_weighted': dummy_curve} - - for i, metric in enumerate(breakdown_names): - scalar_metrics['ap_%s' % metric] = dummy_scalar - scalar_metrics['ap_ha_weighted_%s' % metric] = dummy_scalar - curve_metrics['pr_%s' % metric] = dummy_curve - curve_metrics['pr_ha_weighted_%s' % metric] = dummy_curve - - return py_utils.NestedMap( - feed_dict={}, - scalar_metrics=scalar_metrics, - curve_metrics=curve_metrics) - - feed_dict = {} - - f_gt_bbox = tf.placeholder(tf.float32) - feed_dict[f_gt_bbox] = feed_data.gt.bbox - - f_gt_imgid = tf.placeholder(tf.int32) - feed_dict[f_gt_imgid] = feed_data.gt.imgid - - f_gt_speed = tf.placeholder(tf.float32) - feed_dict[f_gt_speed] = feed_data.gt.speed - - f_gt_difficulty = tf.placeholder(tf.uint8) - feed_dict[f_gt_difficulty] = feed_data.gt.difficulty - - f_pd_bbox = tf.placeholder(tf.float32) - feed_dict[f_pd_bbox] = feed_data.pd.bbox - - f_pd_imgid = tf.placeholder(tf.int32) - feed_dict[f_pd_imgid] = feed_data.pd.imgid - - f_pd_score = tf.placeholder(tf.float32) - feed_dict[f_pd_score] = feed_data.pd.score - - num_gt_bboxes = feed_data.gt.imgid.shape[0] - num_pd_bboxes = feed_data.pd.imgid.shape[0] - gt_class_ids = tf.constant(classid, dtype=tf.uint8, shape=[num_gt_bboxes]) - pd_class_ids = tf.constant(classid, dtype=tf.uint8, shape=[num_pd_bboxes]) - ap, ap_ha, pr, pr_ha, _ = py_metrics_ops.detection_metrics( - prediction_bbox=f_pd_bbox, - prediction_type=pd_class_ids, - prediction_score=f_pd_score, - prediction_frame_id=tf.cast(f_pd_imgid, tf.int64), - prediction_overlap_nlz=tf.zeros_like(f_pd_imgid, dtype=tf.bool), - ground_truth_bbox=f_gt_bbox, - ground_truth_type=gt_class_ids, - ground_truth_frame_id=tf.cast(f_gt_imgid, tf.int64), - ground_truth_difficulty=f_gt_difficulty, - ground_truth_speed=f_gt_speed, - config=self._waymo_metric_config.SerializeToString()) - - # All tensors returned by Waymo's metric op have a leading dimension - # B=number of breakdowns. At this moment we always use B=1 to make - # it compatible to the python code. - scalar_metrics = {'ap': ap[0], 'ap_ha_weighted': ap_ha[0]} - curve_metrics = {'pr': pr[0], 'pr_ha_weighted': pr_ha[0]} - - for i, metric in enumerate(breakdown_names): - # There is a scalar / curve for every breakdown. - scalar_metrics['ap_%s' % metric] = ap[i] - scalar_metrics['ap_ha_weighted_%s' % metric] = ap_ha[i] - curve_metrics['pr_%s' % metric] = pr[i] - curve_metrics['pr_ha_weighted_%s' % metric] = pr_ha[i] - return py_utils.NestedMap( - feed_dict=feed_dict, - scalar_metrics=scalar_metrics, - curve_metrics=curve_metrics) - - def _ComputeFinalMetrics(self, - classids=None, - difficulty=None, - distance=None, - num_points=None, - rotation=None): - """Compute precision-recall curves as well as average precision. - - Args: - classids: A list of N int32. - difficulty: Not used. - distance: int32 specifying a binned Euclidean distance of the ground truth - bounding box. If None is specified, all distances are selected. - num_points: int32 specifying a binned number of laser points within the - ground truth bounding box. If None is specified, all boxes are selected. - rotation: int32 specifying a binned rotation within the ground truth - bounding box. If None is specified, all boxes are selected. - - Returns: - dict. Each entry in the dict is a list of C (number of classes) dicts - containing mapping from metric names to individual results. Individual - entries may be the following items. - - scalars: A list of C (number of classes) dicts mapping metric - names to scalar values. - - curves: A list of C dicts mapping metrics names to np.float32 - arrays of shape [NumberOfPrecisionRecallPoints()+1, 2]. In the last - dimension, 0 indexes precision and 1 indexes recall. - """ - del difficulty - tf.logging.info('Computing final Waymo metrics.') - assert classids is not None, 'classids must be supplied.' - feed_dict = {} - g = tf.Graph() - scalar_fetches = [] - curve_fetches = [] - with g.as_default(): - for classid in classids: - data = self._GetData( - classid, - distance=distance, - num_points=num_points, - rotation=rotation) - metrics = self._BuildMetric(data, classid) - scalar_fetches += [metrics.scalar_metrics] - curve_fetches += [metrics.curve_metrics] - feed_dict.update(metrics.feed_dict) - - with tf.Session(graph=g) as sess: - results = sess.run([scalar_fetches, curve_fetches], feed_dict=feed_dict) - tf.logging.info('Finished computing final Waymo metrics.') - return {'scalars': results[0], 'curves': results[1]} - - @property - def value(self): - """Returns weighted mAP over all eval classes.""" - self._EvaluateIfNecessary() - ap = self._breakdown_metrics['waymo']._average_precisions # pylint:disable=protected-access - breakdown_names = config_util.get_breakdown_names_from_config( - self._waymo_metric_config) - - num_sum = 0.0 - denom_sum = 0.0 - # Compute the average AP over all eval classes. The first breakdown - # is the overall mAP. - for class_index in range(len(self.metadata.EvalClassIndices())): - num_sum += np.nan_to_num(ap[breakdown_names[0]][class_index]) - denom_sum += 1. - return num_sum / denom_sum - - def Summary(self, name): - """Implements custom Summary for Waymo metrics.""" - self._EvaluateIfNecessary() - - ret = tf.Summary() - # Put '.value' first (so it shows up in logs / summaries, etc). - ret.value.add(tag='{}/weighted_mAP'.format(name), simple_value=self.value) - - ap = self._breakdown_metrics['waymo']._average_precisions # pylint:disable=protected-access - aph = self._breakdown_metrics['waymo']._average_precision_headings # pylint:disable=protected-access - breakdown_names = config_util.get_breakdown_names_from_config( - self._waymo_metric_config) - - for i, class_index in enumerate(self.metadata.EvalClassIndices()): - classname = self.metadata.ClassNames()[class_index] - for breakdown_name in breakdown_names: - # 'ONE_SHARD' breakdowns are the overall metrics (not sliced up) - # So we should make that the defualt metric. - if 'ONE_SHARD' in breakdown_name: - # For the overall mAP, include the class name - # and set the breakdown_str which will have the level - prefix = '{}/{}'.format(name, classname) - postfix = breakdown_name.replace('ONE_SHARD_', '') - breakdown_str = postfix if postfix else 'UNKNOWN' - # Otherwise check that the class we are looking at is in the breakdown. - elif classname.lower() in breakdown_name.lower(): - prefix = '{}_extra'.format(name) - breakdown_str = breakdown_name - else: - continue - - tag_str = '{}/AP_{}'.format(prefix, breakdown_str) - ap_value = ap[breakdown_name][i] - ret.value.add(tag=tag_str, simple_value=ap_value) - tag_str = '{}/APH_{}'.format(prefix, breakdown_str) - aph_value = aph[breakdown_name][i] - ret.value.add(tag=tag_str, simple_value=aph_value) - - image_summaries = self._breakdown_metrics['waymo'].GenerateSummaries(name) - for image_summary in image_summaries: - ret.value.extend(image_summary.value) - - return ret - - -class WaymoBreakdownMetric(breakdown_metric.BreakdownMetric): - """Calculate average precision as function of difficulty.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'breakdown_list', [], - 'A list of breakdown names corresponding to the breakdown ' - 'metrics computed from the Waymo breakdown generator config.') - return p - - def __init__(self, p): - super().__init__(p) - self._average_precision_headings = {} - self._precision_recall_headings = {} - - def ComputeMetrics(self, compute_metrics_fn): - p = self.params - tf.logging.info('Calculating waymo AP breakdowns: start') - metrics = compute_metrics_fn() - scalars = metrics['scalars'] - curves = metrics['curves'] - - for breakdown_str in p.breakdown_list: - self._average_precisions[breakdown_str] = [ - s['ap_%s' % breakdown_str] for s in scalars - ] - self._average_precision_headings[breakdown_str] = [ - s['ap_ha_weighted_%s' % breakdown_str] for s in scalars - ] - self._precision_recall[breakdown_str] = np.array( - [c['pr_%s' % breakdown_str] for c in curves]) - self._precision_recall_headings[breakdown_str] = np.array( - [c['pr_ha_weighted_%s' % breakdown_str] for c in curves]) - tf.logging.info('Calculating waymo AP breakdowns: finished') - - def GenerateSummaries(self, name): - """Generate an image summary for precision recall by difficulty.""" - p = self.params - - image_summaries = [] - for i, class_index in enumerate(p.metadata.EvalClassIndices()): - - def _Setter(fig, axes): - """Configure the plot for precision recall.""" - ticks = np.arange(0, 1.05, 0.1) - axes.grid(b=False) - axes.set_xlabel('Recall') - axes.set_xticks(ticks) - axes.set_ylabel('Precision') - axes.set_yticks(ticks) - # TODO(vrv): Add legend indicating number of objects in breakdown. - fig.tight_layout() - - classname = p.metadata.ClassNames()[class_index] - for breakdown_name in p.breakdown_list: - # 'ONE_SHARD' breakdowns are the overall metrics (not sliced up) - # So we should never skip this. - if 'ONE_SHARD' in breakdown_name: - breakdown_str = breakdown_name.replace('ONE_SHARD_', '') - tag_str = '{}/{}/{}/PR'.format(name, classname, breakdown_str) - # Otherwise check that the class we are looking at is in the breakdown. - elif classname.lower() in breakdown_name.lower(): - tag_str = '{}/{}/{}/PR'.format(name, classname, breakdown_name) - else: - continue - - ps = [self._precision_recall[breakdown_name][i][:, 0]] - rs = [self._precision_recall[breakdown_name][i][:, 1]] - image_summary = plot.Curve( - name=tag_str, - figsize=(10, 8), - xs=rs[0], - ys=np.array(ps).T, - setter=_Setter, - marker='.', - markersize=14, - linestyle='-', - linewidth=2, - alpha=0.5) - image_summaries.append(image_summary) - return image_summaries - - # Fill in dummy implementations which are largely - # unused. The current implementation does not provide breakdown - # image summaries that do bucketing; we assume that the waymo breakdown - # implementations will break things down as necessary. - def AccumulateHistogram(self, result): - pass - - def AccumulateCumulative(self, result): - pass - - def NumBinsOfHistogram(self): - return 1 diff --git a/lingvo/tasks/car/waymo/waymo_ap_metric_test.py b/lingvo/tasks/car/waymo/waymo_ap_metric_test.py deleted file mode 100644 index 2f6db482..00000000 --- a/lingvo/tasks/car/waymo/waymo_ap_metric_test.py +++ /dev/null @@ -1,130 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for waymo_ap_metric.""" - -from lingvo import compat as tf -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.tasks.car.waymo import waymo_ap_metric -from lingvo.tasks.car.waymo import waymo_metadata -import numpy as np -from waymo_open_dataset import label_pb2 - -FLAGS = tf.flags.FLAGS - - -class APTest(test_utils.TestCase): - - def testWaymoAPConfig(self): - metadata = waymo_metadata.WaymoMetadata() - # Use 2D metric. - config = waymo_ap_metric._BuildWaymoMetricConfig(metadata, '2d', []) - vehicle_idx = label_pb2.Label.Type.Value('TYPE_VEHICLE') - ped_idx = label_pb2.Label.Type.Value('TYPE_PEDESTRIAN') - cyc_idx = label_pb2.Label.Type.Value('TYPE_CYCLIST') - - thresholds_meta = metadata.IoUThresholds() - self.assertNear(config.iou_thresholds[vehicle_idx], - thresholds_meta['Vehicle'], 1e-6) - self.assertNear(config.iou_thresholds[ped_idx], - thresholds_meta['Pedestrian'], 1e-6) - self.assertNear(config.iou_thresholds[cyc_idx], thresholds_meta['Cyclist'], - 1e-6) - - def testPerfectBox(self): - metadata = waymo_metadata.WaymoMetadata() - params = waymo_ap_metric.WaymoAPMetrics.Params(metadata) - m = params.Instantiate() - # Make one update with a perfect box. - update_dict = py_utils.NestedMap( - groundtruth_labels=np.array([1]), - groundtruth_bboxes=np.ones(shape=(1, 7)), - groundtruth_difficulties=np.zeros(shape=(1)), - groundtruth_num_points=None, - detection_scores=np.ones(shape=(5, 1)), - detection_boxes=np.ones(shape=(5, 1, 7)), - detection_heights_in_pixels=np.ones(shape=(5, 1))) - - m.Update('1234', update_dict) - - waymo_ap = m.value - self.assertAllClose(waymo_ap, 1. / 3.) - - # Write a summary. - summary = m.Summary('foo') - # Check that both AP and APH are in the tags. - tags = [v.tag for v in summary.value] - self.assertIn('foo/Pedestrian/AP_LEVEL_1', tags) - self.assertIn('foo/Pedestrian/APH_LEVEL_1', tags) - self.assertIn('foo/Pedestrian/AP_LEVEL_2', tags) - self.assertIn('foo/Pedestrian/APH_LEVEL_2', tags) - - def testWaymoBreakdowns(self): - metadata = waymo_metadata.WaymoMetadata() - params = waymo_ap_metric.WaymoAPMetrics.Params(metadata) - params.waymo_breakdown_metrics = ['RANGE', 'VELOCITY'] - - m = params.Instantiate() - # Make one update with a perfect box. - update_dict = py_utils.NestedMap( - groundtruth_labels=np.array([1]), - groundtruth_bboxes=np.ones(shape=(1, 7)), - groundtruth_difficulties=np.zeros(shape=(1)), - groundtruth_num_points=None, - groundtruth_speed=np.zeros(shape=(1, 2)), - detection_scores=np.ones(shape=(5, 1)), - detection_boxes=np.ones(shape=(5, 1, 7)), - detection_heights_in_pixels=np.ones(shape=(5, 1))) - - m.Update('1234', update_dict) - - # Write a summary. - summary = m.Summary('foo') - # Check that the summary value for default ap and - # a waymo breakdown version by range is the same. - for v in summary.value: - if v.tag == 'foo/Vehicle/AP_LEVEL_1': - default_val = v.simple_value - elif v.tag == 'foo/Vehicle/APH_LEVEL_1': - aph_default_val = v.simple_value - elif v.tag == 'foo_extra/AP_RANGE_TYPE_VEHICLE_[0, 30)_LEVEL_1': - ap_bd_val_l1 = v.simple_value - elif v.tag == 'foo_extra/AP_RANGE_TYPE_VEHICLE_[0, 30)_LEVEL_2': - ap_bd_val_l2 = v.simple_value - elif v.tag == 'foo_extra/APH_RANGE_TYPE_VEHICLE_[0, 30)_LEVEL_1': - aph_bd_val_l1 = v.simple_value - elif v.tag == 'foo_extra/APH_RANGE_TYPE_VEHICLE_[0, 30)_LEVEL_2': - aph_bd_val_l2 = v.simple_value - elif v.tag == 'foo_extra/AP_VELOCITY_TYPE_VEHICLE_STATIONARY_LEVEL_1': - vbd_val_l1 = v.simple_value - elif v.tag == 'foo_extra/AP_VELOCITY_TYPE_VEHICLE_STATIONARY_LEVEL_2': - vbd_val_l2 = v.simple_value - - self.assertEqual(ap_bd_val_l1, default_val) - self.assertEqual(ap_bd_val_l2, default_val) - self.assertEqual(aph_bd_val_l1, aph_default_val) - self.assertEqual(aph_bd_val_l2, aph_default_val) - self.assertEqual(vbd_val_l1, default_val) - self.assertEqual(vbd_val_l2, default_val) - - # Check that eval classes not evaluated are not present. - tags = [v.tag for v in summary.value] - self.assertNotIn('foo_extra/APH_RANGE_TYPE_SIGN_[0, 30)_LEVEL_1', tags) - self.assertNotIn('foo_extra/APH_RANGE_TYPE_SIGN_[0, 30)_LEVEL_2', tags) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/car/waymo/waymo_decoder.py b/lingvo/tasks/car/waymo/waymo_decoder.py deleted file mode 100644 index b80496a7..00000000 --- a/lingvo/tasks/car/waymo/waymo_decoder.py +++ /dev/null @@ -1,294 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Base models for point-cloud based detection.""" - -from lingvo import compat as tf -from lingvo.core import metrics -from lingvo.core import py_utils -from lingvo.tasks.car import base_decoder -from lingvo.tasks.car import detection_3d_metrics -from lingvo.tasks.car import transform_util -from lingvo.tasks.car.waymo import waymo_ap_metric -from lingvo.tasks.car.waymo import waymo_metadata -import numpy as np - - -class WaymoOpenDatasetDecoder(base_decoder.BaseDecoder): - """A decoder to use for decoding a detector model on Waymo.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'draw_visualizations', False, 'Boolean for whether to draw ' - 'visualizations. This is independent of laser_sampling_rate.') - p.ap_metric = waymo_ap_metric.WaymoAPMetrics.Params( - waymo_metadata.WaymoMetadata()) - p.Define( - 'extra_ap_metrics', {}, - 'Dictionary of extra AP metrics to run in the decoder. The key' - 'is the name of the metric and the value is a sub-class of ' - 'APMetric') - p.Define( - 'save_residuals', False, - 'If True, this expects the residuals and ground-truth to be available ' - 'in the decoder output dictionary, and it will save it to the decoder ' - 'output file. See decode_include_residuals in PointDetectorBase ' - 'for details.') - return p - - def CreateDecoderMetrics(self): - """Decoder metrics for WaymoOpenDataset.""" - p = self.params - - waymo_metric_p = p.ap_metric.Copy().Set(cls=waymo_ap_metric.WaymoAPMetrics) - waymo_metrics = waymo_metric_p.Instantiate() - class_names = waymo_metrics.metadata.ClassNames() - - # TODO(bencaine,vrv): There's some code smell with this ap_metrics params - # usage. We create local copies of the params to then instantiate them. - # Failing to do this risks users editing the params after construction of - # the object, making each object method call have the potential for side - # effects. - # Create a new dictionary with copies of the params converted to objects - # so we can then add these to the decoder metrics. - extra_ap_metrics = {} - for k, metric_p in p.extra_ap_metrics.items(): - extra_ap_metrics[k] = metric_p.Instantiate() - - waymo_metric_bev_p = waymo_metric_p.Copy() - waymo_metric_bev_p.box_type = '2d' - waymo_metrics_bev = waymo_metric_bev_p.Instantiate() - # Convert the list of class names to a dictionary mapping class_id -> name. - class_id_to_name = dict(enumerate(class_names)) - - # TODO(vrv): This uses the same top down transform as for KITTI; - # re-visit these settings since detections can happen all around - # the car. - top_down_transform = transform_util.MakeCarToImageTransform( - pixels_per_meter=32., - image_ref_x=512., - image_ref_y=1408., - flip_axes=True) - decoder_metrics = py_utils.NestedMap({ - 'top_down_visualization': - (detection_3d_metrics.TopDownVisualizationMetric( - top_down_transform, - image_height=1536, - image_width=1024, - class_id_to_name=class_id_to_name)), - 'num_samples_in_batch': metrics.AverageMetric(), - 'waymo_metrics': waymo_metrics, - 'waymo_metrics_bev': waymo_metrics_bev, - }) - self._update_metrics_class_keys = ['waymo_metrics_bev', 'waymo_metrics'] - for k, metric in extra_ap_metrics.items(): - decoder_metrics[k] = metric - self._update_metrics_class_keys.append(k) - - decoder_metrics.mesh = detection_3d_metrics.WorldViewer() - return decoder_metrics - - def ProcessOutputs(self, input_batch, model_outputs): - """Produce additional decoder outputs for WaymoOpenDataset. - - Args: - input_batch: A .NestedMap of the inputs to the model. - model_outputs: A .NestedMap of the outputs of the model, including:: - - per_class_predicted_bboxes: [batch, num_classes, num_boxes, 7] float - Tensor with per class 3D (7 DOF) bounding boxes. - - per_class_predicted_bbox_scores: [batch, num_classes, num_boxes] float - Tensor with per class, per box scores. - - per_class_valid_mask: [batch, num_classes, num_boxes] masking Tensor - indicating which boxes were still kept after NMS for each class. - - Returns: - A NestedMap of additional decoder outputs needed for - PostProcessDecodeOut. - """ - del model_outputs - p = self.params - input_labels = input_batch.labels - input_metadata = input_batch.metadata - source_ids = tf.strings.join([ - input_metadata.run_segment, - tf.as_string(input_metadata.run_start_offset) - ], - separator='_') - ret = py_utils.NestedMap({ - 'num_points_in_bboxes': input_batch.labels.bboxes_3d_num_points, - # Ground truth. - 'bboxes_3d': input_labels.bboxes_3d, - 'bboxes_3d_mask': input_labels.bboxes_3d_mask, - 'labels': input_labels.labels, - 'label_ids': input_labels.label_ids, - 'speed': input_labels.speed, - 'acceleration': input_labels.acceleration, - # Fill the following in. - 'source_ids': source_ids, - 'difficulties': input_labels.single_frame_detection_difficulties, - 'unfiltered_bboxes_3d_mask': input_labels.unfiltered_bboxes_3d_mask, - 'run_segment': input_metadata.run_segment, - 'run_start_offset': input_metadata.run_start_offset, - 'pose': input_metadata.pose, - }) - if p.draw_visualizations: - laser_sample = self._SampleLaserForVisualization( - input_batch.lasers.points_xyz, input_batch.lasers.points_padding) - ret.update(laser_sample) - return ret - - def PostProcessDecodeOut(self, dec_out_dict, dec_metrics_dict): - """Post-processes the decoder outputs.""" - p = self.params - # Update num_samples_in_batch. - batch_size, num_classes, num_boxes, _ = ( - dec_out_dict.per_class_predicted_bboxes.shape) - dec_metrics_dict.num_samples_in_batch.Update(batch_size) - - # Update decoder output by removing z-coordinate, thus reshaping the bboxes - # to [batch, num_bboxes, 5] to be compatible with - # TopDownVisualizationMetric. - - # Indices corresponding to the 2D bbox parameters (x, y, dx, dy, phi). - bbox_2d_idx = np.asarray([1, 1, 0, 1, 1, 0, 1], dtype=np.bool) - bboxes_2d = dec_out_dict.bboxes_3d[..., bbox_2d_idx] - predicted_bboxes = dec_out_dict.per_class_predicted_bboxes[..., bbox_2d_idx] - - if p.draw_visualizations and dec_out_dict.points_sampled: - tf.logging.info('Updating sample for top down visualization') - dec_metrics_dict.mesh.Update( - py_utils.NestedMap({ - 'points_xyz': dec_out_dict.points_xyz, - 'points_padding': dec_out_dict.points_padding, - })) - - # Flatten our predictions/scores to match the API of the visualization - # The last dimension of flattened_bboxes is 5 due to the mask - # above using bbox_2d_idx. - flattened_bboxes = np.reshape(predicted_bboxes, - [batch_size, num_classes * num_boxes, 5]) - flattened_visualization_weights = np.reshape( - dec_out_dict.visualization_weights, - [batch_size, num_classes * num_boxes]) - - # Create a label id mask for now to maintain compatibility. - # TODO(bencaine): Refactor visualizations to reflect new structure. - flattened_visualization_labels = np.tile( - np.arange(0, num_classes)[np.newaxis, :, np.newaxis], - [batch_size, 1, num_boxes]) - flattened_visualization_labels = np.reshape( - flattened_visualization_labels, [batch_size, num_classes * num_boxes]) - dec_metrics_dict.top_down_visualization.Update( - py_utils.NestedMap({ - 'visualization_labels': flattened_visualization_labels, - 'predicted_bboxes': flattened_bboxes, - 'visualization_weights': flattened_visualization_weights, - 'points_xyz': dec_out_dict.points_xyz, - 'points_padding': dec_out_dict.points_padding, - 'gt_bboxes_2d': bboxes_2d, - 'gt_bboxes_2d_weights': dec_out_dict.bboxes_3d_mask, - 'labels': dec_out_dict.labels, - 'difficulties': dec_out_dict.difficulties, - 'source_ids': dec_out_dict.source_ids, - })) - - # Update AP metrics. - - # Skip zeroth step decoding. - if dec_out_dict.global_step == 0: - return None - - # TODO(bencaine/vrv): Refactor to unify Waymo code and KITTI - - # Returned values are saved in model_dir/decode_* directories. - output_to_save = [] - - for batch_idx in range(batch_size): - pred_bboxes = dec_out_dict.per_class_predicted_bboxes[batch_idx] - pred_bbox_scores = dec_out_dict.per_class_predicted_bbox_scores[batch_idx] - - # The current API expects a 'height' matrix to be passed for filtering - # detections based on height. This is a KITTI-ism that we need to remove, - # but for now we just give a height of 1. The MinHeight metadata function - # for non-KITTI datasets should have a threshold lower than this value. - heights = np.ones((num_classes, num_boxes)).astype(np.float32) - - gt_mask = dec_out_dict.bboxes_3d_mask[batch_idx].astype(bool) - gt_labels = dec_out_dict.labels[batch_idx][gt_mask] - gt_bboxes = dec_out_dict.bboxes_3d[batch_idx][gt_mask] - gt_difficulties = dec_out_dict.difficulties[batch_idx][gt_mask] - gt_num_points = dec_out_dict.num_points_in_bboxes[batch_idx][gt_mask] - # Note that this is not used in the KITTI evaluation. - gt_speed = dec_out_dict.speed[batch_idx][gt_mask] - - # TODO(shlens): Update me - for metric_key in self._update_metrics_class_keys: - metric_cls = dec_metrics_dict[metric_key] - metric_cls.Update( - dec_out_dict.source_ids[batch_idx], - py_utils.NestedMap( - groundtruth_labels=gt_labels, - groundtruth_bboxes=gt_bboxes, - groundtruth_difficulties=gt_difficulties, - groundtruth_num_points=gt_num_points, - groundtruth_speed=gt_speed, - detection_scores=pred_bbox_scores, - detection_boxes=pred_bboxes, - detection_heights_in_pixels=heights, - )) - - # We still want to save all ground truth (even if it was filtered - # in some way) so we use the unfiltered_bboxes_3d_mask here. - gt_save_mask = dec_out_dict.unfiltered_bboxes_3d_mask[batch_idx].astype( - bool) - pd_save_mask = dec_out_dict.per_class_valid_mask[batch_idx] > 0 - class_ids = np.tile(np.arange(num_classes)[:, np.newaxis], [1, num_boxes]) - - saved_results = py_utils.NestedMap( - pose=dec_out_dict.pose[batch_idx], - frame_id=dec_out_dict.source_ids[batch_idx], - bboxes=pred_bboxes[pd_save_mask], - scores=pred_bbox_scores[pd_save_mask], - gt_labels=dec_out_dict.labels[batch_idx][gt_save_mask], - gt_label_ids=dec_out_dict.label_ids[batch_idx][gt_save_mask], - gt_speed=dec_out_dict.speed[batch_idx][gt_save_mask], - gt_acceleration=dec_out_dict.acceleration[batch_idx][gt_save_mask], - class_ids=class_ids[pd_save_mask], - gt_bboxes=dec_out_dict.bboxes_3d[batch_idx][gt_save_mask], - gt_difficulties=dec_out_dict.difficulties[batch_idx][gt_save_mask], - ) - - if p.save_residuals: - # The leading shapes of these tensors should match bboxes and scores. - # These are the underlying tensors that can are used to compute score - # and bboxes. - saved_results.update({ - 'bboxes_gt_residuals': - dec_out_dict.per_class_gt_residuals[batch_idx][pd_save_mask], - 'bboxes_gt_labels': - dec_out_dict.per_class_gt_labels[batch_idx][pd_save_mask], - 'bboxes_residuals': - dec_out_dict.per_class_residuals[batch_idx][pd_save_mask], - 'bboxes_logits': - dec_out_dict.per_class_logits[batch_idx][pd_save_mask], - 'bboxes_anchor_boxes': - dec_out_dict.per_class_anchor_boxes[batch_idx][pd_save_mask], - }) - - serialized = self.SaveTensors(saved_results) - output_to_save += [(dec_out_dict.source_ids[batch_idx], serialized)] - return output_to_save diff --git a/lingvo/tasks/car/waymo/waymo_metadata.py b/lingvo/tasks/car/waymo/waymo_metadata.py deleted file mode 100644 index 3c626df8..00000000 --- a/lingvo/tasks/car/waymo/waymo_metadata.py +++ /dev/null @@ -1,92 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Metadata for Waymo dataset employed in evaluation.""" - -from lingvo.tasks.car import evaluation_metadata - - -class WaymoMetadata(evaluation_metadata.EvaluationMetadata): - """Metadata describing the Waymo dataset used for evaluation. - - Some of these entries need to be revisited and set correctly for - the final version of the Waymo dataset (e.g., MaximumDistance) - based off an analysis of the dataset. - """ - - def __init__(self): - super().__init__('waymo') - - def ClassNames(self): - """Returns a list of human-interpretable strings.""" - return ['Unknown', 'Vehicle', 'Pedestrian', 'Sign', 'Cyclist'] - - def DifficultyLevels(self): - """Dictionary of difficulty level strings to int32 indices.""" - return {'UNKNOWN': 0, 'LEVEL_1': 1, 'LEVEL_2': 2} - - def IoUThresholds(self): - return { - 'Vehicle': 0.7, - 'Pedestrian': 0.5, - 'Cyclist': 0.5, - } - - def EvalClassIndices(self): - """List of int32 indices for the classes that should be evaled.""" - return [ - self.ClassNames().index(name) - for name in ['Vehicle', 'Pedestrian', 'Cyclist'] - ] - - def IgnoreClassIndices(self): - """Dictionary of int32 indices for the classes that should be ignored.""" - # A detection that matches with a groundtruth bbox of any neighbor class - # will not be considered as false positive in eval. - return {} - - def NumberOfPrecisionRecallPoints(self): - """Number of points on the precision-recall curve.""" - return 101 - - def MaximumDistance(self): - """Maximum empirically observed Euclidean distance in world coordinates.""" - # TODO(vrv): Compute the actual value. - return 80.0 - - def DistanceBinWidth(self): - """The width of each bin for Euclidean distance in world coordinates.""" - return 5.0 - - def MaximumNumberOfPoints(self): - """Maximum empirically observed number of points in bounding box.""" - # TODO(vrv): Compute the actual value. - return 30000.0 - - def NumberOfPointsBins(self): - """Number of logarithmically space bins for examining points.""" - return 20 - - def NumberOfRotationBins(self): - """Number of linear spaced rotations to bin.""" - return 10 - - def NumberOfCalibrationBins(self): - """Number of linear spaced calibration bins.""" - return 15 - - def MinHeight2D(self): - """Minimum height of detections to be evaluated.""" - return {level: 0 for level in self.DifficultyLevels()} diff --git a/lingvo/tasks/car/waymo/waymo_open_input_generator.py b/lingvo/tasks/car/waymo/waymo_open_input_generator.py deleted file mode 100644 index 8bfe9405..00000000 --- a/lingvo/tasks/car/waymo/waymo_open_input_generator.py +++ /dev/null @@ -1,1000 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Input generator for waymo open dataset (WaymoOD).""" - -from lingvo import compat as tf -from lingvo.core import datasource -from lingvo.core import hyperparams -from lingvo.core import py_utils -from lingvo.tasks.car import input_extractor -from lingvo.tasks.car import input_preprocessors - -import numpy as np - - -def _Dense(sparse, default_value=0): - return tf.sparse_to_dense( - sparse_indices=sparse.indices, - output_shape=sparse.dense_shape, - sparse_values=sparse.values, - default_value=default_value) - - -def _NestedMapToParams(nmap): - p = hyperparams.Params() - for k, v in nmap.FlattenItems(): - p.Define(k, v, '') - return p - - -class WaymoFrameMetadataExtractor(input_extractor.FieldsExtractor): - """Extracts per frame metadata from a WaymoOD tf.Example. - - Emits: - pose: [4, 4] - A float Tensor with the 4x4 transformation matrix for - converting from "world" coordinates SDC coordinates. - - run_segment: string scalar - The run segment identifier. - - run_start_offset: int64 scalar - Offset of this scene from the start of the - run segment (in microseconds). - - time_of_day: string scalar - Categorical description of time of day, - e.g., "Day". - - location: string scalar - Categorical description of geographical location, - e.g., "location_sf". - - weather: string scalar - Categorical description of weather of scene, - e.g., "sunny". - """ - - # Valid options for metadata that we can use for validation - # Filters that aren't in this list will still be allowed, but these will - # be checked for extra safety. - VALIDATED_FILTER_OPTIONS = py_utils.NestedMap( - time_of_day=['Day', 'Dawn/Dusk', 'Night'], - # Generated test data uses 'rain', so we keep the word 'rain' for - # backwards compatibility. Eventually we should converge on using 'rainy'. - weather=['rain', 'rainy', 'sunny', 'unknown'], - location=[ - 'location_sf', 'location_phx', 'location_kir', 'location_other' - ]) - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'equality_filters', None, 'A list of tuples(str, list) ' - 'where each first value is a metadata key (e.g. `weather`) ' - 'and the second value is a list of valid values to filter for. ' - 'Each filter will check whether the value of a given example ' - 'for that metadata key matches one of the allowed filter values. ' - 'Then the result of each filter (each tuple) will be AND-ed ' - 'together. Example usage would be: ' - '[("location", ["location_sf"]), ("weather", ["sunny"])] ' - 'Which would only allow through examples that are in SF ' - 'AND have sunny weather. ') - return p - - def _ValidateFilterValues(self): - """Check the filter against several blessed values.""" - p = self.params - for filter_key, filter_values in p.equality_filters: - # Type check - if (not isinstance(filter_key, str) or - not isinstance(filter_values, list)): - raise ValueError('Each element in `equality_filters` must be a ' - 'tuple of (str, list).') - # If it's not one of the "blessed" validated options, just let it through - if filter_key not in self.VALIDATED_FILTER_OPTIONS: - continue - - # If we do know its valid options, check each value against this list - valid_options = self.VALIDATED_FILTER_OPTIONS[filter_key] - for filter_value in filter_values: - if filter_value not in valid_options: - raise ValueError( - 'Filter {} value: {} not in valid options: {}'.format( - filter_key, filter_value, valid_options)) - - def __init__(self, params): - super().__init__(params) - p = self.params - - if p.equality_filters: - if not isinstance(p.equality_filters, list): - raise ValueError('`equality_filters` param must be a list.') - if not all([isinstance(val, tuple) for val in p.equality_filters]): - raise ValueError('Every item in `equality_filters` must be a tuple.') - self._ValidateFilterValues() - - def FeatureMap(self): - """Return a dictionary from tf.Example feature names to Features.""" - feature_map = {} - feature_map['pose'] = tf.io.VarLenFeature(dtype=tf.float32) - feature_map['run_segment'] = tf.io.FixedLenFeature((), tf.string, '') - feature_map['run_start_offset'] = tf.io.FixedLenFeature((), tf.int64, 0) - feature_map['time_of_day'] = tf.io.FixedLenFeature((), tf.string, '') - feature_map['location'] = tf.io.FixedLenFeature((), tf.string, '') - feature_map['weather'] = tf.io.FixedLenFeature((), tf.string, '') - return feature_map - - def _Extract(self, features): - """Extract data into Tensor format.""" - vehicle_pose = tf.reshape(_Dense(features['pose']), [4, 4]) - run_segment = features['run_segment'] - run_start_offset = features['run_start_offset'] - time_of_day = features['time_of_day'] - location = features['location'] - weather = features['weather'] - return py_utils.NestedMap( - pose=vehicle_pose, - run_segment=run_segment, - run_start_offset=run_start_offset, - time_of_day=time_of_day, - location=location, - weather=weather) - - def Shape(self): - """The expected shape of each field.""" - return py_utils.NestedMap( - pose=tf.TensorShape([4, 4]), - run_segment=tf.TensorShape([]), - run_start_offset=tf.TensorShape([]), - time_of_day=tf.TensorShape([]), - location=tf.TensorShape([]), - weather=tf.TensorShape([])) - - def DType(self): - """The Dtype of each field.""" - return py_utils.NestedMap( - pose=tf.float32, - run_segment=tf.string, - run_start_offset=tf.int64, - time_of_day=tf.string, - location=tf.string, - weather=tf.string) - - def Filter(self, outputs): - """Optionally filters the data based on context info.""" - p = self.params - if p.equality_filters is None: - return 1 - - allowed_example = tf.convert_to_tensor(True) - for filter_key, filter_values in p.equality_filters: - if filter_key not in outputs: - raise ValueError( - 'Filter key `{}` not found in extracted data.'.format(filter_key)) - has_allowed_data = tf.reduce_any( - tf.equal(outputs[filter_key], filter_values)) - allowed_example = tf.math.logical_and(allowed_example, has_allowed_data) - - not_allowed_example = 1 - tf.cast(allowed_example, tf.int32) - return 1 + (not_allowed_example * input_extractor.BUCKET_UPPER_BOUND) - - -class WaymoImageExtractor(input_extractor.FieldsExtractor): - """Extracts the camera image data from a WaymoOD tf.Example. - - The cameras are [FRONT, FRONT_LEFT, FRONT_RIGHT, SIDE_LEFT, SIDE_RIGHT]. - - Emits dictionary, where each camera is a key (camera name) and the value is - a NestedMap containing: - - image: [height, width, 3] - Images from the corresponding cameras. - - intrinsics: [9] - Instrinsics of the camera. - - extrinsics: [4, 4] - Extrinsics of the camera - - pose: [4, 4] - Pose of the camera when the corresponding image is taken. - - velocity: [6] - Velocity of the camera when the corresponding image is - taken. The first three numbers (vx, vy, vz) are velocities in world frame, - in m/s. The last three numbers (roll, pitch, yaw) are the rotation rates - in vehicle frame, in rad/s. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('camera_names', - ['FRONT', 'FRONT_LEFT', 'FRONT_RIGHT', 'SIDE_LEFT', 'SIDE_RIGHT'], - 'The names of the cameras from which images will be extracted.') - p.Define('image_shape', [1280, 1920, 3], - 'The shape that images are cropped to.') - return p - - def FeatureMap(self): - """Return a dictionary from tf.Example feature names to Features.""" - p = self.params - features = {} - features['pose'] = tf.io.VarLenFeature(dtype=tf.float32) - - for camera_name in p.camera_names: - features['image_%s' % camera_name] = tf.io.VarLenFeature(dtype=tf.string) - features['image_%s_shape' % camera_name] = ( - tf.io.VarLenFeature(dtype=tf.int64)) - features['camera_%s_intrinsics' % - camera_name] = tf.io.VarLenFeature(dtype=tf.float32) - features['camera_%s_extrinsics' % - camera_name] = tf.io.VarLenFeature(dtype=tf.float32) - - features['camera_%s_rolling_shutter_direction' % - camera_name] = tf.io.FixedLenFeature( - dtype=tf.int64, shape=()) - features['image_%s_pose' % - camera_name] = tf.io.VarLenFeature(dtype=tf.float32) - features['image_%s_velocity' % - camera_name] = tf.io.VarLenFeature(dtype=tf.float32) - - for feat in [ - 'pose_timestamp', 'shutter', 'camera_trigger_time', - 'camera_readout_done_time' - ]: - features['image_%s_%s' % (camera_name, feat)] = tf.io.FixedLenFeature( - dtype=tf.float32, shape=()) - return features - - def _Extract(self, features): - """Returns the image Tensor.""" - outputs = py_utils.NestedMap() - p = self.params - for camera_name in p.camera_names: - image_shape = tf.reshape( - _Dense(features['image_%s_shape' % camera_name]), [-1]) - image = tf.io.decode_png( - tf.strings.reduce_join( - _Dense(features['image_%s' % camera_name], default_value=''))) - image = tf.reshape(image, image_shape) - image = py_utils.PadOrTrimTo(image, p.image_shape) - intrinsics = tf.reshape( - _Dense(features['camera_%s_intrinsics' % camera_name]), [9]) - extrinsics = tf.reshape( - _Dense(features['camera_%s_extrinsics' % camera_name]), [4, 4]) - pose = tf.reshape(_Dense(features['image_%s_pose' % camera_name]), [4, 4]) - velocity = tf.reshape( - _Dense(features['image_%s_velocity' % camera_name]), [6]) - - outputs[camera_name] = py_utils.NestedMap() - outputs[camera_name]['image'] = image - outputs[camera_name]['intrinsics'] = intrinsics - outputs[camera_name]['extrinsics'] = extrinsics - outputs[camera_name]['pose'] = pose - outputs[camera_name]['velocity'] = velocity - outputs[camera_name]['rolling_shutter_direction'] = features[ - 'camera_%s_rolling_shutter_direction' % camera_name] - - for feat in [ - 'shutter', 'camera_trigger_time', 'camera_readout_done_time', - 'pose_timestamp' - ]: - outputs[camera_name][feat] = features['image_%s_%s' % - (camera_name, feat)] - - return outputs - - def Shape(self): - """Shape of images.""" - p = self.params - shapes = py_utils.NestedMap() - for camera_name in p.camera_names: - shapes[camera_name] = py_utils.NestedMap() - shapes[camera_name]['image'] = tf.TensorShape(p.image_shape) - # 1d Array of [f_u, f_v, c_u, c_v, k{1, 2}, p{1, 2}, k{3}]. - # Note that this intrinsic corresponds to the images after scaling. - # Camera model: pinhole camera. - # Lens distortion: - # Radial distortion coefficients: k1, k2, k3. - # Tangential distortion coefficients: p1, p2. - # k_{1, 2, 3}, p_{1, 2} follows the same definition as OpenCV. - shapes[camera_name]['intrinsics'] = tf.TensorShape([9]) - shapes[camera_name]['extrinsics'] = tf.TensorShape([4, 4]) - shapes[camera_name]['pose'] = tf.TensorShape([4, 4]) - shapes[camera_name]['velocity'] = tf.TensorShape([6]) - for feat in [ - 'pose_timestamp', 'shutter', 'camera_trigger_time', - 'camera_readout_done_time' - ]: - shapes[camera_name][feat] = tf.TensorShape([]) - shapes[camera_name]['rolling_shutter_direction'] = tf.TensorShape([]) - - return shapes - - def DType(self): - """Dtypes of images.""" - p = self.params - dtypes = py_utils.NestedMap() - for camera_name in p.camera_names: - dtypes[camera_name] = py_utils.NestedMap() - dtypes[camera_name]['image'] = tf.uint8 - dtypes[camera_name]['intrinsics'] = tf.float32 - dtypes[camera_name]['extrinsics'] = tf.float32 - dtypes[camera_name]['pose'] = tf.float32 - dtypes[camera_name]['velocity'] = tf.float32 - for feat in [ - 'pose_timestamp', 'shutter', 'camera_trigger_time', - 'camera_readout_done_time' - ]: - dtypes[camera_name][feat] = tf.float32 - dtypes[camera_name]['rolling_shutter_direction'] = tf.int64 - return dtypes - - -class WaymoLaserExtractor(input_extractor.LaserExtractor): - """Extracts the raw laser data from a WaymoOD tf.Example.""" - - @classmethod - def Params(cls): - p = super().Params().Set(max_num_points=None, num_features=3) - p.Define('lidar_names', ['TOP', 'SIDE_LEFT', 'SIDE_RIGHT', 'FRONT', 'REAR'], - 'The names of the lidars from which lasers will be extracted.') - p.Define( - 'lidar_returns', ['ri1', 'ri2'], 'Which return from the LiDAR to ' - 'extract when we merge the point cloud.') - return p - - def FeatureMap(self): - """Return a dictionary from tf.Example feature names to Features.""" - p = self.params - features = {} - for lidar in p.lidar_names: - for ri in p.lidar_returns: - features['laser_%s_%s' % - (lidar, ri)] = tf.io.VarLenFeature(dtype=tf.float32) - return features - - def _Extract(self, features): - """Returns the laser Tensor.""" - p = self.params - all_xyzs = [] - all_laser_features = [] - - for lidar in p.lidar_names: - for ri in p.lidar_returns: - feature_name = 'laser_%s_%s' % (lidar, ri) - laser_data = tf.reshape( - _Dense(features[feature_name]), [-1, 3 + p.num_features]) - points_xyz = laser_data[..., 0:3] - points_feature = laser_data[..., 3:] - - all_xyzs += [points_xyz] - all_laser_features += [points_feature] - - # Stack all of the points along the major dimension - points_xyz = tf.concat(all_xyzs, axis=0) - points_feature = tf.concat(all_laser_features, axis=0) - - if p.max_num_points is not None: - npoints = tf.shape(points_xyz)[0] - points_xyz = py_utils.PadOrTrimTo(points_xyz, [p.max_num_points, 3]) - points_feature = py_utils.PadOrTrimTo(points_feature, - [p.max_num_points, p.num_features]) - points_padding = 1.0 - py_utils.PadOrTrimTo( - tf.ones([npoints]), [p.max_num_points]) - - ret = py_utils.NestedMap( - points_xyz=points_xyz, points_feature=points_feature) - if p.max_num_points is not None: - ret.points_padding = points_padding - return ret - - -class WaymoLabelExtractor(input_extractor.FieldsExtractor): - """Extracts the bounding box and label info from a WaymoOD tf.Example. - - Emits: - labels: [p.max_num_objects] - Integer label for each bounding box object - corresponding to the index in car.open_dataset.Label.Type (shifted by 1 to - have 0 represent the background class). - - label_ids: [p.max_num_objects] - String unique identifier for each labeled - object on a per run_segment basis. This can be used for associating - objects across frames (over time). - - detection_difficulties: [p.max_num_objects] - DO NOT USE FOR EVALUATION. - The per-box difficulty level for detection task as defined in - car.open_dataset.Label.DifficultyLevel. This is the human raters - difficulty level, which does NOT include information about the number - of points per box. Therefore, it is an incomplete definition of difficulty - and will not correspond to the leaderboard if used to calculate metrics. - - single_frame_detection_difficulties: [p.max_num_objects] - The per-box - difficulty level derived via both detection_difficulties (labeler defined) - and metric defined (number of points in box). - - tracking_difficulties: [p.max_num_objects] - The per-box difficulty level - for tracking task as defined in car.open_dataset.Label.DifficultyLevel. - - bboxes_3d: [p.max_num_objects, 7] - 3D bounding box data in [x, y, z, l, w, - h, heading] format. x, y, z are the object center in world coordinates; - l, w, h are the dimensions of the box, and heading is the rotation angle - around the z-axis. See car.open_dataset.Label.Box for definitions. - - bboxes_3d_mask: [p.max_num_objects] - Mask for all the above tensors (mask - is the inversion of padding). - - bboxes_3d_num_points: [p.max_num_objects] - Integer for each box indicating - how many points are in that ground truth box. - - unfiltered_bboxes_3d_mask: [p.max_num_objects] - The mask before filtering - out bboxes whose labels are not in p.filter_labels. - - speed: [p.max_num_objects, 2] - The object speed in x, y. - - acceleration: [p.max_num_objects, 2] - The object acceleration in x, y. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('max_num_objects', 512, - 'Each frame may contain up to these many bbox.') - p.Define( - 'filter_labels', [], 'Specifies a list of label ' - 'indices to keep. If empty, no filtering is done.') - return p - - def FeatureMap(self): - """Return a dictionary from tf.Example feature names to Features.""" - feature_map = {} - feature_map['labels'] = tf.io.VarLenFeature(dtype=tf.int64) - feature_map['label_ids'] = tf.io.VarLenFeature(dtype=tf.string) - feature_map['detection_difficulties'] = tf.io.VarLenFeature(dtype=tf.int64) - feature_map['single_frame_detection_difficulties'] = tf.io.VarLenFeature( - dtype=tf.int64) - feature_map['tracking_difficulties'] = tf.io.VarLenFeature(dtype=tf.int64) - feature_map['bboxes_3d'] = tf.io.VarLenFeature(dtype=tf.float32) - feature_map['bboxes_3d_num_points'] = tf.io.VarLenFeature(dtype=tf.int64) - feature_map['label_metadata'] = tf.io.VarLenFeature(dtype=tf.float32) - return feature_map - - def _Extract(self, features): - p = self.params - # Label values match the proto enum car.open_dataset.Label.Type. The value - # range is [1..4] for non-background labels. - labels = tf.cast(_Dense(features['labels']), tf.int32) - labels = py_utils.PadOrTrimTo(labels, [p.max_num_objects]) - label_ids = tf.reshape(_Dense(features['label_ids'], ''), [-1]) - label_ids = py_utils.PadOrTrimTo(label_ids, [p.max_num_objects], '') - bboxes_3d = tf.reshape(_Dense(features['bboxes_3d']), [-1, 7]) - bboxes_3d_mask = tf.ones([tf.shape(bboxes_3d)[0]]) - bboxes_3d_num_points = tf.cast( - _Dense(features['bboxes_3d_num_points']), tf.int32) - bboxes_3d = py_utils.PadOrTrimTo(bboxes_3d, [p.max_num_objects, 7]) - bboxes_3d_mask = py_utils.PadOrTrimTo(bboxes_3d_mask, [p.max_num_objects]) - bboxes_3d_num_points = py_utils.PadOrTrimTo(bboxes_3d_num_points, - [p.max_num_objects]) - label_metadata = tf.reshape(_Dense(features['label_metadata']), [-1, 4]) - label_metadata = py_utils.PadOrTrimTo(label_metadata, - [p.max_num_objects, 4]) - - detection_difficulties = py_utils.PadOrTrimTo( - tf.cast(_Dense(features['detection_difficulties']), tf.int32), - [p.max_num_objects]) - single_frame_detection_difficulties = py_utils.PadOrTrimTo( - tf.cast( - _Dense(features['single_frame_detection_difficulties']), tf.int32), - [p.max_num_objects]) - tracking_difficulties = py_utils.PadOrTrimTo( - tf.cast(_Dense(features['tracking_difficulties']), tf.int32), - [p.max_num_objects]) - unfiltered_bboxes_3d_mask = bboxes_3d_mask - - if p.filter_labels: - valid_labels = tf.constant([p.filter_labels]) - bbox_mask = tf.reduce_any( - tf.equal(tf.expand_dims(labels, 1), valid_labels), axis=1) - bboxes_3d_mask *= tf.cast(bbox_mask, tf.float32) - - outputs = { - 'labels': - labels, - 'label_ids': - label_ids, - 'detection_difficulties': - detection_difficulties, - 'single_frame_detection_difficulties': - single_frame_detection_difficulties, - 'tracking_difficulties': - tracking_difficulties, - 'bboxes_3d': - bboxes_3d, - 'bboxes_3d_mask': - bboxes_3d_mask, - 'bboxes_3d_num_points': - bboxes_3d_num_points, - 'unfiltered_bboxes_3d_mask': - unfiltered_bboxes_3d_mask, - 'speed': - label_metadata[:, :2], - 'acceleration': - label_metadata[:, 2:], - } - - return py_utils.NestedMap(outputs) - - def Shape(self): - """Shape of BBoxes.""" - p = self.params - shapes = { - 'labels': - tf.TensorShape([p.max_num_objects]), - 'label_ids': - tf.TensorShape([p.max_num_objects]), - 'detection_difficulties': - tf.TensorShape([p.max_num_objects]), - 'single_frame_detection_difficulties': - tf.TensorShape([p.max_num_objects]), - 'tracking_difficulties': - tf.TensorShape([p.max_num_objects]), - 'bboxes_3d': - tf.TensorShape([p.max_num_objects, 7]), - 'bboxes_3d_mask': - tf.TensorShape([p.max_num_objects]), - 'bboxes_3d_num_points': - tf.TensorShape([p.max_num_objects]), - 'unfiltered_bboxes_3d_mask': - tf.TensorShape([p.max_num_objects]), - 'speed': - tf.TensorShape([p.max_num_objects, 2]), - 'acceleration': - tf.TensorShape([p.max_num_objects, 2]) - } - return py_utils.NestedMap(shapes) - - def DType(self): - """Dtypes of BBoxes.""" - dtypes = py_utils.NestedMap() - dtypes.labels = tf.int32 - dtypes.label_ids = tf.string - dtypes.detection_difficulties = tf.int32 - dtypes.single_frame_detection_difficulties = tf.int32 - dtypes.tracking_difficulties = tf.int32 - dtypes.bboxes_3d = tf.float32 - dtypes.bboxes_3d_mask = tf.float32 - dtypes.bboxes_3d_num_points = tf.int32 - dtypes.unfiltered_bboxes_3d_mask = tf.float32 - dtypes.speed = tf.float32 - dtypes.acceleration = tf.float32 - return dtypes - - -class RangeImageExtractor(input_extractor.FieldsExtractor): - """Extracts the range images from a Waymo OD tf.Example. - - The outputs contain the following: - - Let ri_shape = [H, W] of the corresponding range image. - - - For every short range laser (params.cbr_laser_names): - For every return (params.returns): - $LASERNAME_RETURN: - .xyz - tf.float32 of ri_shape + [3] - - .features - tf.float32 of ri_shape + [4] - - .mask - tf.float32 of ri_shape indicating whether the laser - xyz and feature at each coordinate is real or padded. A coordinate - has a real point iff the mask is set to 1. - - $LASERNAME_beam_inclinations: tf.float32 [2] listing the - min and max beam inclinations. - - $LASERNAME_extrinsics: tf.float32 [4, 4] extrinsics matrix. - - - For every longer range laser (params.gbr_laser_names): - For every return (params.returns): - .xyz: tf.float32 of ri_shape + [3] - .features: tf.float32 of ri_shape + [4] - .mask: tf.float32 of ri_shape - - $LASERNAME_beam_inclinations: tf.float32 [64] listing the - non-uniform beam inclinations for the longer range laser. - - $LASERNAME_extrinsics: tf.float32 [4, 4] extrinsics matrix - - $LASERNAME_pose: tf.float32 of ri_shape + [4, 4], which is the - per-pixel pose. - - On laser returns: - ri1 and ri2 are the first and second returns of the sensors. - - On laser sensors: - If there are 5 total sensors, there will be 5 * len(returns) - outputs. - - The last dimension of range image is 4, indicating the following - features: - - range: (if entry is -1, it means there is no laser value there). - - intensity - - elongation - - 1. if laser point entry in 'no label zone', 0. otherwise. - - The xyz range image output is a [H, W, 3] Tensor indicating the cartesian - coordinates corresponding to each range image pixel in the range image. One - should use the mask computed from the 'range' channel of the range image to - select only the points that exist. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'cbr_laser_names', ['SIDE_LEFT', 'FRONT', 'REAR', 'SIDE_RIGHT'], - 'The names of the CBR sensors from which range images ' - 'will be extracted.') - p.Define('cbr_ri_shape', [200, 600, 4], 'Shape of each CBR range image.') - p.Define( - 'gbr_laser_names', ['TOP'], - 'The names of the GBR sensors from which range images ' - 'will be extracted.') - p.Define( - 'returns', ['ri1', 'ri2'], - 'The names of the laser returns to export. E.g., ri1 is ' - 'the first return, ri2 is the second return.') - p.Define('gbr_ri_shape', [64, 2650, 4], 'Shape of each GBR range image.') - return p - - def FeatureMap(self): - """Return a dictionary from tf.Example feature names to Features.""" - p = self.params - feature_map = {} - for laser in p.cbr_laser_names + p.gbr_laser_names: - feature_map['%s_beam_inclinations' % laser] = ( - tf.io.VarLenFeature(dtype=tf.float32)) - feature_map['%s_beam_inclination_min' % laser] = ( - tf.io.VarLenFeature(dtype=tf.float32)) - feature_map['%s_beam_inclination_max' % laser] = ( - tf.io.VarLenFeature(dtype=tf.float32)) - feature_map['%s_extrinsics' % - laser] = tf.io.VarLenFeature(dtype=tf.float32) - if laser in p.gbr_laser_names: - feature_map['%s_pose' % laser] = tf.io.VarLenFeature(dtype=tf.float32) - - for returns in p.returns: - feature_map['%s_%s' % - (laser, returns)] = tf.io.VarLenFeature(dtype=tf.float32) - feature_map['%s_%s_shape' % - (laser, returns)] = tf.io.VarLenFeature(dtype=tf.int64) - feature_map['pose'] = tf.io.VarLenFeature(dtype=tf.float32) - return feature_map - - def _Extract(self, features): - p = self.params - ri_outputs = {} - outputs = {} - frame_pose = tf.reshape(_Dense(features['pose']), [4, 4]) - for laser in p.cbr_laser_names + p.gbr_laser_names: - # Extract range images. - for returns in p.returns: - ri_shape = tf.reshape( - _Dense(features['%s_%s_shape' % (laser, returns)]), [-1]) - range_image = tf.reshape( - _Dense(features['%s_%s' % (laser, returns)]), ri_shape) - - shape_to_check = ( - p.cbr_ri_shape if laser in p.cbr_laser_names else p.gbr_ri_shape) - range_image = py_utils.HasShape(range_image, shape_to_check) - - ri_outputs['%s_%s' % (laser, returns)] = range_image - - # Extract beam inclinations and extrinsics - outputs['%s_extrinsics' % laser] = tf.reshape( - _Dense(features['%s_extrinsics' % laser]), [4, 4]) - - # CBRs have uniform inclination - for laser in p.cbr_laser_names: - beam_inclination_min = tf.reshape( - _Dense(features['%s_beam_inclination_min' % laser]), []) - beam_inclination_max = tf.reshape( - _Dense(features['%s_beam_inclination_max' % laser]), []) - outputs['%s_beam_inclinations' % laser] = tf.stack( - [beam_inclination_min, beam_inclination_max], axis=0) - - # GBRs have non-uniform inclinations defined by 64 floats. - for laser in p.gbr_laser_names: - outputs['%s_beam_inclinations' % laser] = tf.reshape( - _Dense(features['%s_beam_inclinations' % laser]), [64]) - - # Embed xyz onto each range image pixel. - for laser in p.cbr_laser_names + p.gbr_laser_names: - extrinsics = outputs['%s_extrinsics' % laser] - inclinations = outputs['%s_beam_inclinations' % laser] - if laser in p.cbr_laser_names: - ri_shape = p.cbr_ri_shape - - # Convert from 2-tuple range inclination to the full range - # via linear interpolation. - # - # CBR lasers currently are always uniform inclinations specified by a - # length 2 vector. - height = ri_shape[0] - min_inclination = inclinations[0] - max_inclination = inclinations[1] - diff = max_inclination - min_inclination - ratio = (.5 + tf.cast(tf.range(0, height), tf.float32)) / tf.cast( - height, tf.float32) - # interpolate from min to max inclination. - inclinations = (ratio * diff) + min_inclination - else: - ri_shape = p.gbr_ri_shape - - pixel_pose = None - if laser in p.gbr_laser_names: - pixel_pose = tf.reshape( - _Dense(features['%s_pose' % laser]), - shape=p.gbr_ri_shape[0:2] + [4, 4]) - outputs['%s_pose' % laser] = pixel_pose - - for returns in p.returns: - range_image = ri_outputs['%s_%s' % (laser, returns)] - range_image = tf.reshape(range_image, ri_shape) - range_image_mask = range_image[..., 0] >= 0 - ri_xyz = tf.cast( - self._XYZFromRangeImage(range_image, range_image_mask, extrinsics, - inclinations, pixel_pose, frame_pose), - tf.float32) - - # Produce the NestedMap of xyz, features, mask. - ri_result = py_utils.NestedMap({ - 'xyz': ri_xyz, - 'features': range_image, - 'mask': tf.cast(range_image_mask, tf.float32), - }) - - outputs['%s_%s' % (laser, returns)] = ri_result - - return py_utils.NestedMap(outputs) - - def _XYZFromRangeImage(self, - lidar_image, - lidar_image_mask, - extrinsics, - inclinations, - pixel_pose=None, - frame_pose=None): - """Extract the cartesian coordinates from the range image. - - Args: - lidar_image: [H, W, C] range image Tensor. - lidar_image_mask: [H, W] boolean indicating which 2d coordinates in the - lidar image are present. - extrinsics: [4, 4] float matrix representing transformation matrix to - world coordinates. - inclinations: [V] beam inclinations vector. - pixel_pose: [64, 2650, 4, 4] tensor representing per pixel pose of GBR. - frame_pose: [4, 4] matrix representing vehicle to world transformation. - - Returns: - [H, W, 3] range image cartesian coordinates. - """ - height, width, channels = py_utils.GetShape(lidar_image, 3) - - conversion_dtype = tf.float32 - lidar_image = tf.cast(lidar_image, conversion_dtype) - extrinsics = tf.cast(extrinsics, conversion_dtype) - inclinations = tf.cast(inclinations, conversion_dtype) - inclinations = tf.reverse(inclinations, axis=[-1]) - - az_correction = py_utils.HasShape( - tf.atan2(extrinsics[1, 0], extrinsics[0, 0]), []) - ratios = (tf.cast(tf.range(width, 0, -1), dtype=conversion_dtype) - - .5) / tf.cast(width, conversion_dtype) - ratios = py_utils.HasShape(ratios, [width]) - - azimuth = (ratios * 2. - 1.) * np.pi - az_correction[..., tf.newaxis] - azimuth = py_utils.HasShape(azimuth, [width]) - - lidar_image_mask = lidar_image_mask[..., tf.newaxis] - lidar_image_mask = tf.tile(lidar_image_mask, [1, 1, channels]) - lidar_image = tf.where(lidar_image_mask, lidar_image, - tf.zeros_like(lidar_image)) - lidar_image_range = lidar_image[..., 0] - - azimuth = py_utils.HasShape(azimuth[tf.newaxis, ...], [1, width]) - inclinations = py_utils.HasShape(inclinations[..., tf.newaxis], [height, 1]) - - cos_azimuth = tf.cos(azimuth) - sin_azimuth = tf.sin(azimuth) - cos_incl = tf.cos(inclinations) - sin_incl = tf.sin(inclinations) - - x = cos_azimuth * cos_incl * lidar_image_range - y = sin_azimuth * cos_incl * lidar_image_range - z = sin_incl * lidar_image_range - - lidar_image_points = tf.stack([x, y, z], -1) - lidar_image_points = py_utils.HasShape(lidar_image_points, - [height, width, 3]) - rotation = extrinsics[0:3, 0:3] - translation = extrinsics[0:3, 3][tf.newaxis, ...] - - # Transform the image points in cartesian coordinates to - # the world coordinate system using the extrinsics matrix. - # - # We first flatten the points, apply rotation, then - # reshape to restore the original input and then apply - # translation. - lidar_image_points = tf.matmul( - tf.reshape(lidar_image_points, [-1, 3]), rotation, transpose_b=True) - lidar_image_points = tf.reshape(lidar_image_points, [height, width, 3]) - lidar_image_points += translation - - lidar_image_points = py_utils.HasShape(lidar_image_points, - [height, width, 3]) - # GBR uses per pixel pose. - if pixel_pose is not None: - pixel_pose_rotation = pixel_pose[..., 0:3, 0:3] - pixel_pose_translation = pixel_pose[..., 0:3, 3] - lidar_image_points = tf.einsum( - 'hwij,hwj->hwi', pixel_pose_rotation, - lidar_image_points) + pixel_pose_translation - if frame_pose is None: - raise ValueError('frame_pose must be set when pixel_pose is set.') - # To vehicle frame corresponding to the given frame_pose - # [4, 4] - world_to_vehicle = tf.linalg.inv(frame_pose) - world_to_vehicle_rotation = world_to_vehicle[0:3, 0:3] - world_to_vehicle_translation = world_to_vehicle[0:3, 3] - # [H, W, 3] - lidar_image_points = tf.einsum( - 'ij,hwj->hwi', world_to_vehicle_rotation, - lidar_image_points) + world_to_vehicle_translation[tf.newaxis, - tf.newaxis, :] - - return lidar_image_points - - def Shape(self): - """Shape of BBoxes.""" - p = self.params - shapes = {} - for laser in p.cbr_laser_names: - cbr_shape = p.cbr_ri_shape[:-1] - for returns in p.returns: - shape_dict = py_utils.NestedMap({ - 'xyz': tf.TensorShape(cbr_shape + [3]), - 'features': tf.TensorShape(cbr_shape + [4]), - 'mask': tf.TensorShape(cbr_shape), - }) - shapes['%s_%s' % (laser, returns)] = shape_dict - - shapes['%s_extrinsics' % laser] = tf.TensorShape([4, 4]) - shapes['%s_beam_inclinations' % laser] = tf.TensorShape([2]) - for laser in p.gbr_laser_names: - gbr_shape = p.gbr_ri_shape[:-1] - for returns in p.returns: - shape_dict = py_utils.NestedMap({ - 'xyz': tf.TensorShape(gbr_shape + [3]), - 'features': tf.TensorShape(gbr_shape + [4]), - 'mask': tf.TensorShape(gbr_shape), - }) - shapes['%s_%s' % (laser, returns)] = shape_dict - shapes['%s_extrinsics' % laser] = tf.TensorShape([4, 4]) - shapes['%s_beam_inclinations' % laser] = tf.TensorShape([64]) - shapes['%s_pose' % laser] = tf.TensorShape(gbr_shape + [4, 4]) - - return py_utils.NestedMap(shapes) - - def DType(self): - """Dtypes of BBoxes.""" - p = self.params - dtypes = {} - for laser in p.cbr_laser_names + p.gbr_laser_names: - for returns in p.returns: - dtype_dict = py_utils.NestedMap({ - 'xyz': tf.float32, - 'features': tf.float32, - 'mask': tf.float32, - }) - dtypes['%s_%s' % (laser, returns)] = dtype_dict - dtypes['%s_extrinsics' % laser] = tf.float32 - dtypes['%s_beam_inclinations' % laser] = tf.float32 - for laser in p.gbr_laser_names: - dtypes['%s_pose' % laser] = tf.float32 - return py_utils.NestedMap(dtypes) - - -class FilterNLZPoints(input_preprocessors.Preprocessor): - """Filters points that are in no-label-zones. - - This preprocessor expects features to contain the following keys: - - lasers.points_xyz of shape [P, 3] - - lasers.points_feature of shape [P, F] - - Modifies the following features: - - lasers.points_xyz of shape [P2, 3] - - lasers.points_feature of shape [P2, F] - - where P - P2 are the number of points dropped because the corresponding - point was in a no-label-zone. - """ - - @classmethod - def Params(cls): - p = super().Params() - return p - - def TransformFeatures(self, features): - # We assume that the lasers are not padded, and all points are real. - if ('points_padding' in features.lasers and - features.lasers.points_padding is not None): - raise ValueError('FilterNLZPoints preprocessor does not support ' - 'padded lasers.') - - # The 3rd feature in the laser is 1.0 for points in a no-label-zone - # and -1. for normal points. - is_not_nlz = tf.not_equal(features.lasers.points_feature[:, 2], 1.0) - features.lasers.points_xyz = tf.boolean_mask(features.lasers.points_xyz, - is_not_nlz) - features.lasers.points_feature = tf.boolean_mask( - features.lasers.points_feature, is_not_nlz) - return features - - def TransformShapes(self, shapes): - return shapes - - def TransformDTypes(self, dtypes): - return dtypes - - -class WaymoSparseLaser(input_extractor.BaseExtractor): - """Sparse laser input extractor for Waymo dataset.""" - - @classmethod - def Params(cls): - """Defaults params.""" - extractors = hyperparams.Params() - extractors.Define('lasers', WaymoLaserExtractor.Params(), '') - extractors.Define('labels', WaymoLabelExtractor.Params(), '') - extractors.Define('metadata', WaymoFrameMetadataExtractor.Params(), '') - - preprocessors = py_utils.NestedMap( - count_points=input_preprocessors.CountNumberOfPointsInBoxes3D.Params(), - viz_copy=input_preprocessors.CreateDecoderCopy.Params(), - keep_xyz_range=input_preprocessors.DropLaserPointsOutOfRange.Params(), - filter_nlz_points=FilterNLZPoints.Params(), - select_centers=input_preprocessors.SparseCenterSelector.Params(), - gather_features=input_preprocessors.SparseCellGatherFeatures.Params(), - tile_anchors=input_preprocessors.TileAnchorBBoxes.Params(), - assign_anchors=input_preprocessors.AnchorAssignment.Params(), - pad_lasers=input_preprocessors.PadLaserFeatures.Params().Set(), - ) - p = super().Params(extractors).Set( - preprocessors=_NestedMapToParams(preprocessors), - preprocessors_order=[ - 'viz_copy', - 'keep_xyz_range', - 'filter_nlz_points', - 'count_points', - 'select_centers', - 'gather_features', - 'tile_anchors', - 'assign_anchors', - 'pad_lasers', - ], - ) - - p.file_datasource = datasource.PrefixedDataSource.Params() - p.file_datasource.file_type = 'tfrecord' - - return p diff --git a/lingvo/tasks/car/waymo/waymo_open_input_generator_test.py b/lingvo/tasks/car/waymo/waymo_open_input_generator_test.py deleted file mode 100644 index 64101dc0..00000000 --- a/lingvo/tasks/car/waymo/waymo_open_input_generator_test.py +++ /dev/null @@ -1,53 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for WaymoOpenInputGenerator.""" - -from lingvo import compat as tf -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.tasks.car.waymo import waymo_open_input_generator -import numpy as np - -FLAGS = tf.flags.FLAGS - - -class WaymoOpenInputGeneratorTest(test_utils.TestCase): - - def testFilterNLZ(self): - num_points = 32 - num_features = 3 - points_feature = np.random.rand(num_points, num_features).astype(np.float32) - # Mark one point as being in nlz. - points_feature[-1, 2] = 1. - lasers_np = py_utils.NestedMap( - points_xyz=np.random.rand(num_points, 3).astype(np.float32), - points_feature=points_feature) - features_np = py_utils.NestedMap(lasers=lasers_np) - features = features_np.Transform(tf.constant) - - preprocessor_p = waymo_open_input_generator.FilterNLZPoints.Params() - processor = preprocessor_p.Instantiate() - - features = processor.TransformFeatures(features) - with self.session(): - actual_features = self.evaluate(features) - # one point dropped because it was in nlz - self.assertEqual((num_points - 1, 3), - actual_features.lasers.points_xyz.shape) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/image/BUILD b/lingvo/tasks/image/BUILD deleted file mode 100644 index 684d52d3..00000000 --- a/lingvo/tasks/image/BUILD +++ /dev/null @@ -1,64 +0,0 @@ -# Image related tasks. -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -py_library( - name = "classifier", - srcs = ["classifier.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_model", - "//lingvo/core:layers", - "//lingvo/core:metrics", - "//lingvo/core:plot", - "//lingvo/core:py_utils", - "//lingvo/core:schedule", - # Implicit numpy dependency. - ], -) - -py_test( - name = "classifier_test", - srcs = ["classifier_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":classifier", - ":input_generator", - "//lingvo:compat", - "//lingvo:model_registry", - "//lingvo/core:cluster_factory", - "//lingvo/core:layers", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - "//lingvo/tasks/image/params:mnist", - # Implicit numpy dependency. - ], -) - -py_library( - name = "input_generator", - srcs = ["input_generator.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_input_generator", - ], -) - -py_test( - name = "input_generator_test", - srcs = ["input_generator_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":input_generator", - "//lingvo:compat", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) diff --git a/lingvo/tasks/image/__init__.py b/lingvo/tasks/image/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/lingvo/tasks/image/classifier.py b/lingvo/tasks/image/classifier.py deleted file mode 100644 index c8e82176..00000000 --- a/lingvo/tasks/image/classifier.py +++ /dev/null @@ -1,335 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Image classification models.""" - -import lingvo.compat as tf -from lingvo.core import base_model -from lingvo.core import layers -from lingvo.core import metrics -from lingvo.core import plot -from lingvo.core import py_utils -from lingvo.core import schedule -import numpy as np - - -def TopKAccuracy(k, logits, labels, weights): - """Compute top-k accuracy. - - Args: - k: An int scalar. Top-k. - logits: A [N, C] float tensor. - labels: A [N] int vector. - weights: A [N] float vector. - - Returns: - A float scalar. The accuracy at precision k. - """ - logits = py_utils.HasRank(logits, 2) - n, _ = tf.unstack(tf.shape(logits), 2) - labels = py_utils.HasShape(labels, [n]) - weights = py_utils.HasShape(weights, [n]) - correct = tf.nn.in_top_k(targets=labels, predictions=logits, k=k) - return tf.reduce_sum(tf.cast(correct, weights.dtype) * weights) / tf.maximum( - 1e-8, tf.reduce_sum(weights)) - - -class BaseClassifier(base_model.BaseTask): - """Base class for image classifier.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('softmax', layers.SimpleFullSoftmax.Params(), 'Softmax layer.') - p.Define('add_image_summary', True, 'If True, adds image summary in ' - 'evaler/decoder jobs.') - return p - - def _AddSummary(self, batch, prediction): - """Adds image summaries for the batch.""" - p = self.params - if not self.do_eval or not p.add_image_summary: - # Image summaries only works in evaler/decoder. - return - - def Draw(fig, axes, img, label, pred): - plot.AddImage( - fig=fig, - axes=axes, - data=img[:, :, 0] / 256., - show_colorbar=False, - suppress_xticks=True, - suppress_yticks=True) - axes.text( - x=0.5, - y=0, - s=u'%d vs. %d' % (label, pred), - transform=axes.transAxes, - horizontalalignment='center') - - with plot.MatplotlibFigureSummary( - 'examples', figsize=(1, 1), max_outputs=10) as fig: - fig.AddSubplot([batch.raw, batch.label, prediction], Draw) - - def _Accuracy(self, k, logits, labels, weights): - """Compute top-k accuracy. - - Args: - k: An int scalar. Top-k. - logits: A [N, C] float tensor. - labels: A [N] int vector. - weights: A [N] float vector. - - Returns: - A float scalar. The accuracy at precision k. - """ - return TopKAccuracy(k, logits, labels, weights) - - -class ModelV1(BaseClassifier): - """CNNs with maxpooling followed by a softmax.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'filter_shapes', [(0, 0, 0, 0)], - 'Conv filter shapes. Must be a list of sequences of 4. ' - 'Elements are in order of height, width, in_channel, out_channel') - p.Define( - 'window_shapes', [(0, 0)], - 'Max pooling window shapes. Must be a list of sequences of 2. ' - 'Elements are in order of height, width.') - p.Define('batch_norm', False, 'Apply BN or not after the conv.') - p.Define('dropout_prob', 0.0, - 'Probability of the dropout applied after pooling.') - - tp = p.train - tp.learning_rate = 1e-4 # Adam base LR. - tp.lr_schedule = ( - schedule.LinearRampupExponentialDecayScaledByNumSplitSchedule.Params() - .Set(warmup=100, decay_start=100000, decay_end=1000000, min=0.1)) - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - - assert len(p.filter_shapes) == len(p.window_shapes) - - # A few conv + max pooling layers. - shape = [None] + list(p.input.data_shape) - conv_params = [] - pooling_params = [] - for i, (kernel, window) in enumerate(zip(p.filter_shapes, p.window_shapes)): - conv_params.append(layers.ConvLayer.Params().Set( - name='conv%d' % i, - filter_shape=kernel, - filter_stride=(1, 1), - batch_norm=p.batch_norm)) - pooling_params.append(layers.PoolingLayer.Params().Set( - name='pool%d' % i, window_shape=window, window_stride=window)) - self.CreateChildren('conv', conv_params) - self.CreateChildren('pool', pooling_params) - - # Logs expected activation shapes. - for i in range(len(self.conv)): - tf.logging.info('shape %d %s', i, shape) - shape = self.conv[i].OutShape(shape) - tf.logging.info('shape %d %s', i, shape) - shape = self.pool[i].OutShape(shape) - tf.logging.info('shape %s', shape) - - # FC layer to project down to p.softmax.input_dim. - self.CreateChild( - 'fc', - layers.FCLayer.Params().Set( - name='fc', - input_dim=np.prod(shape[1:]), - output_dim=p.softmax.input_dim)) - self.CreateChild('softmax', p.softmax) - - def FPropTower(self, theta, input_batch): - p = self.params - batch = tf.shape(input_batch.data)[0] - height, width, depth = p.input.data_shape - act = tf.reshape(input_batch.data, [batch, height, width, depth]) - for i in range(len(self.conv)): - # Conv, BN (optional) - act, _ = self.conv[i].FProp(theta.conv[i], act) - # MaxPool - act, _ = self.pool[i].FProp(theta.pool[i], act) - # Dropout (optional) - if p.dropout_prob > 0.0 and not self.do_eval: - act = tf.nn.dropout(act, rate=p.dropout_prob, seed=p.random_seed) - # FC - act = self.fc.FProp(theta.fc, tf.reshape(act, [batch, -1])) - - # Softmax - labels = tf.cast(input_batch.label, tf.int64) - xent = self.softmax.FProp( - theta=theta.softmax, - inputs=act, - class_weights=input_batch.weight, - class_ids=labels) - - self._AddSummary(input_batch, xent.per_example_argmax) - - rets = { - 'loss': (xent.avg_xent, batch), - 'log_pplx': (xent.avg_xent, batch), - 'num_preds': (batch, 1), - } - if self.do_eval: - acc1 = self._Accuracy(1, xent.logits, labels, input_batch.weight) - acc5 = self._Accuracy(5, xent.logits, labels, input_batch.weight) - rets.update( - accuracy=(acc1, batch), - acc5=(acc5, batch), - error=(1. - acc1, batch), - error5=(1. - acc5, batch)) - return rets, {'loss': xent.per_example_xent} - - def Decode(self, input_batch): - with tf.name_scope('decode'): - return self.FPropDefaultTheta(input_batch)[0] - - def CreateDecoderMetrics(self): - return { - 'num_samples_in_batch': metrics.AverageMetric(), - } - - def PostProcessDecodeOut(self, dec_out_dict, dec_metrics_dict): - dec_metrics_dict['num_samples_in_batch'].Update( - dec_out_dict['num_samples_in_batch'][0]) - - -class ModelV2(BaseClassifier): - """CNNs followed by a softmax.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('extract', None, 'Param for the layer to extract image features.') - p.Define('label_smoothing', 0., 'Smooth the labels towards 1/num_classes.') - p.Define('compute_accuracy_for_training', False, - 'Whether to compute accuracy for training.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.name - - self.CreateChild('extract', p.extract) - self.CreateChild('softmax', p.softmax) - - def ComputePredictions(self, theta, input_batch): - # Forward through layers. - act = self.extract.FProp(theta.extract, input_batch.data) - # Avg pool - act = tf.reduce_mean(act, axis=[1, 2]) - logits = self.softmax.Logits(theta.softmax, act) - return py_utils.NestedMap(act=act, logits=logits) - - def ComputeLoss(self, theta, predictions, input_batch): - p = self.params - batch = tf.shape(input_batch.data)[0] - act = predictions.act - with tf.ops.colocate_with(act): - tf.logging.info("{}'s device: {}".format(act, act.device)) - # Softmax - labels = tf.cast(input_batch.label, tf.int64) - onehot_labels = tf.one_hot(labels, p.softmax.num_classes) - if p.label_smoothing > 0: - smooth_positives = 1.0 - p.label_smoothing - smooth_negatives = p.label_smoothing / p.softmax.num_classes - onehot_labels = onehot_labels * smooth_positives + smooth_negatives - - xent = self.softmax.FProp( - theta=theta.softmax, - inputs=act, - class_weights=input_batch.weight, - class_probabilities=onehot_labels) - - self._AddSummary(input_batch, xent.per_example_argmax) - - rets = { - 'loss': (xent.avg_xent, batch), - 'log_pplx': (xent.avg_xent, batch), - 'num_preds': (batch, 1), - } - if self.do_eval or p.compute_accuracy_for_training: - acc1 = self._Accuracy(1, xent.logits, labels, input_batch.weight) - acc5 = self._Accuracy(5, xent.logits, labels, input_batch.weight) - rets.update( - accuracy=(acc1, batch), - acc5=(acc5, batch), - error=(1. - acc1, batch), - error5=(1. - acc5, batch)) - return rets, {'loss': xent.per_example_xent} - - def Inference(self): - """Constructs inference subgraphs. - - Returns: - dict: A dictionary of the form ``{'subgraph_name': (fetches, feeds)}``. - Each of fetches and feeds is itself a dictionary which maps a string name - (which describes the tensor) to a corresponding tensor in the inference - graph which should be fed/fetched from. - """ - subgraphs = {} - with tf.name_scope('inference'): - subgraphs['default'] = self._InferenceSubgraph_Default() - return subgraphs - - def _InferenceSubgraph_Default(self): - """Constructs graph for single-image inference. - - Returns: - (fetches, feeds) where both fetches and feeds are dictionaries. Each - dictionary consists of keys corresponding to tensor names, and values - corresponding to a tensor in the graph which should be input/read from. - """ - p = self.params - with tf.name_scope('default'): - normalized_image = tf.placeholder( - dtype=p.dtype, shape=p.input.data_shape, name='normalized_image') - inputs = py_utils.NestedMap(data=normalized_image[tf.newaxis, ...]) - logits = tf.reshape( - self.ComputePredictions(self.theta, inputs).logits, - [p.softmax.num_classes], - name='logits') - feeds = { - 'normalized_image': normalized_image, - } - fetches = { - 'logits': logits, - 'probs': tf.nn.softmax(logits, name='probs'), - 'prediction': tf.argmax(logits, name='prediction'), - } - return fetches, feeds - - def DecodeWithTheta(self, theta, input_batch): - """Constructs the decode graph for decoding with theta.""" - predictions = self.ComputePredictions(theta, input_batch) - labels = tf.cast(input_batch.label, tf.int64) - ret = py_utils.NestedMap() - ret.correct_top1 = tf.nn.in_top_k( - targets=labels, predictions=predictions.logits, k=1) - ret.correct_top5 = tf.nn.in_top_k( - targets=labels, predictions=predictions.logits, k=5) - return ret diff --git a/lingvo/tasks/image/classifier_test.py b/lingvo/tasks/image/classifier_test.py deleted file mode 100644 index b857bd6b..00000000 --- a/lingvo/tasks/image/classifier_test.py +++ /dev/null @@ -1,126 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for classifier.""" - -from lingvo import model_registry -import lingvo.compat as tf -from lingvo.core import cluster_factory -from lingvo.core import layers -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.core.test_utils import CompareToGoldenSingleFloat -from lingvo.tasks.image import classifier -from lingvo.tasks.image import input_generator -from lingvo.tasks.image.params import mnist -import numpy as np - - -@model_registry.RegisterSingleTaskModel -class MnistV2(mnist.Base): - """A test MNIST model for classifier.ModelV2.""" - - @classmethod - def Task(cls): - p = classifier.ModelV2.Params() - p.name = 'testv2' - p.extract = layers.Conv2DLayerNoPadding.Params().Set( - filter_shape=(5, 5, 1, 50), filter_stride=(2, 2)) - p.label_smoothing = 0.1 - p.softmax.input_dim = 50 - p.softmax.num_classes = 10 - p.train.learning_rate = 0.1 - return p - - -class ClassifierTest(test_utils.TestCase): - - def setUp(self): - self.data_path = input_generator.FakeMnistData( - self.get_temp_dir(), train_size=0) - - def _runOneStep(self, model): - f_loss = self.evaluate(model.GetTask().loss) - self.evaluate(model.GetTask().train_op) - return f_loss - - def testMnistLeNet5(self): - g = tf.Graph() - with g.as_default(): - tf.random.set_seed(1618) - p = model_registry.GetParams('image.mnist.LeNet5', 'Test') - p.random_seed = 73234288 - p.input.ckpt = self.data_path - p.task.params_init = py_utils.WeightInit.Uniform(0.1, seed=73234288) - with cluster_factory.ForTestingWorker(mode='sync', job='trainer_client'): - model = p.Instantiate() - model.ConstructFPropBPropGraph() - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - CompareToGoldenSingleFloat(self, 2.302583, self._runOneStep(model)) - CompareToGoldenSingleFloat(self, 2.302405, self._runOneStep(model)) - - def testMnistV2(self): - g = tf.Graph() - with g.as_default(): - tf.random.set_seed(1618) - p = model_registry.GetParams('test.MnistV2', 'Test') - p.random_seed = 73234288 - p.input.ckpt = self.data_path - p.task.params_init = py_utils.WeightInit.Uniform(0.1, seed=73234288) - with cluster_factory.ForTestingWorker(mode='sync', job='trainer_client'): - model = p.Instantiate() - model.ConstructFPropBPropGraph() - with self.session(graph=g): - self.evaluate(tf.global_variables_initializer()) - CompareToGoldenSingleFloat(self, 2.302583, self._runOneStep(model)) - CompareToGoldenSingleFloat(self, 2.142516, self._runOneStep(model)) - - def testInference(self): - with self.session() as sess: - tf.random.set_seed(1618) - p = model_registry.GetParams('test.MnistV2', 'Test') - p.random_seed = 73234288 - p.input.ckpt = self.data_path - p.task.params_init = py_utils.WeightInit.Uniform(0.1, seed=73234288) - model = p.Instantiate() - subgraphs = model.GetTask().Inference() - self.assertCountEqual(['default'], list(subgraphs.keys())) - fetches, feeds = subgraphs['default'] - self.assertCountEqual(['normalized_image'], list(feeds.keys())) - self.assertCountEqual(['logits', 'probs', 'prediction'], - list(fetches.keys())) - self.evaluate(tf.global_variables_initializer()) - fetch_results = sess.run( - fetches, {feeds['normalized_image']: np.zeros(p.input.data_shape)}) - self.assertAllEqual([p.task.softmax.num_classes], - fetch_results['logits'].shape) - self.assertAllEqual([p.task.softmax.num_classes], - fetch_results['probs'].shape) - self.assertAllEqual([], fetch_results['prediction'].shape) - - def testDecodeRuns(self): - g = tf.Graph() - with g.as_default(): - p = model_registry.GetParams('test.MnistV2', 'Test') - model = p.Instantiate() - task = model.GetTask() - input_batch = task.GetInputBatch()[0] - result = task.Decode(input_batch) - self.assertIn('correct_top1', result) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/image/input_generator.py b/lingvo/tasks/image/input_generator.py deleted file mode 100644 index 21601fd9..00000000 --- a/lingvo/tasks/image/input_generator.py +++ /dev/null @@ -1,87 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Input generator for image data.""" - -import os -import lingvo.compat as tf -from lingvo.core import base_input_generator - -from tensorflow.python.ops import io_ops - - -class _MnistInputBase(base_input_generator.BaseTinyDatasetInput): - """Base input params for MNIST.""" - - @classmethod - def Params(cls): - """Defaults params.""" - p = super().Params() - p.data_dtype = tf.uint8 - p.data_shape = (28, 28, 1) - p.label_dtype = tf.uint8 - return p - - def _Preprocess(self, raw): - data = tf.stack([ - tf.image.per_image_standardization(img) for img in tf.unstack(raw) - ]) - data.set_shape(raw.shape) - return data - - -class MnistTrainInput(_MnistInputBase): - """MNist training set.""" - - @classmethod - def Params(cls): - """Defaults params.""" - p = super().Params() - p.data = 'x_train' - p.label = 'y_train' - p.num_samples = 60000 - p.batch_size = 256 - p.repeat = True - return p - - -class MnistTestInput(_MnistInputBase): - """MNist test set.""" - - @classmethod - def Params(cls): - """Defaults params.""" - p = super().Params() - p.data = 'x_test' - p.label = 'y_test' - p.num_samples = 10000 - p.batch_size = 256 - p.repeat = False - return p - - -def FakeMnistData(tmpdir, train_size=60000, test_size=10000): - """Fake Mnist data for unit tests.""" - data_path = os.path.join(tmpdir, 'ckpt') - with tf.Graph().as_default(): - with tf.Session() as sess: - x_train = tf.ones((train_size, 28, 28, 1), dtype=tf.uint8) - y_train = tf.ones((train_size), dtype=tf.uint8) - x_test = tf.ones((test_size, 28, 28, 1), dtype=tf.uint8) - y_test = tf.ones((test_size), dtype=tf.uint8) - sess.run( - io_ops.save_v2(data_path, ['x_train', 'y_train', 'x_test', 'y_test'], - [''] * 4, [x_train, y_train, x_test, y_test])) - return data_path diff --git a/lingvo/tasks/image/input_generator_test.py b/lingvo/tasks/image/input_generator_test.py deleted file mode 100644 index 5979d4a8..00000000 --- a/lingvo/tasks/image/input_generator_test.py +++ /dev/null @@ -1,99 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for input_generator.""" - -import lingvo.compat as tf -from lingvo.core import test_utils -from lingvo.tasks.image import input_generator -import numpy as np - - -class InputGeneratorTest(test_utils.TestCase): - - def setUp(self): - self.data_path = input_generator.FakeMnistData(self.get_temp_dir()) - - def _trainInput(self): - p = input_generator.MnistTrainInput.Params() - p.ckpt = self.data_path - p.batch_size = 100 - return p - - def _testInput(self): - p = input_generator.MnistTestInput.Params() - p.ckpt = self.data_path - return p - - def testMnistTrain(self): - p = self._trainInput() - with self.session(): - inp = p.Instantiate() - inp_batch = inp.GetPreprocessedInputBatch() - for _ in range(10): - batch = self.evaluate(inp_batch) - self.assertEqual(batch.data.shape, (100, 28, 28, 1)) - self.assertEqual(batch.data.dtype, np.float32) - self.assertEqual(batch.label.shape, (100,)) - self.assertEqual(batch.label.dtype, np.float32) - - def testMnistTest(self): - p = self._testInput() - with self.session(): - inp = p.Instantiate() - inp_batch = inp.GetPreprocessedInputBatch() - ids = [] - for _ in range(39): - batch = self.evaluate(inp_batch) - self.assertEqual(batch.data.shape, (256, 28, 28, 1)) - self.assertEqual(batch.data.dtype, np.float32) - self.assertEqual(batch.label.shape, (256,)) - self.assertEqual(batch.label.dtype, np.float32) - ids += batch.sample_ids.tolist() - batch = self.evaluate(inp_batch) - self.assertEqual(batch.data.shape, (256, 28, 28, 1)) - self.assertEqual(batch.data.dtype, np.float32) - self.assertEqual(batch.label.shape, (256,)) - self.assertEqual(batch.label.dtype, np.float32) - ids += batch.sample_ids.tolist() - self.assertEqual(list(range(p.num_samples)), - sorted(ids)) # Exactly 1 epoch. - - # repeat=False. We should see OutOfRange error. - with self.assertRaises(tf.errors.OutOfRangeError): - _ = self.evaluate(inp_batch) - - def _GetIds(self, p, sample_ids): - """Goes through one epoch of inp and returns the sample ids.""" - iters = int((p.num_samples + p.batch_size - 1) / p.batch_size) - ids = [] - for _ in range(iters): - ids += self.evaluate(sample_ids).tolist() - return ids - - def testMnistTrainRandomness(self): - p = self._trainInput() - with self.session(): - inp = p.Instantiate() - batch = inp.GetPreprocessedInputBatch() - epoch0 = self._GetIds(p, batch.sample_ids) - epoch1 = self._GetIds(p, batch.sample_ids) - self.assertEqual(list(range(p.num_samples)), sorted(epoch0)) - self.assertEqual(list(range(p.num_samples)), sorted(epoch1)) - self.assertNotEqual(epoch0, epoch1) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/image/params/BUILD b/lingvo/tasks/image/params/BUILD deleted file mode 100644 index 807b56c5..00000000 --- a/lingvo/tasks/image/params/BUILD +++ /dev/null @@ -1,28 +0,0 @@ -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -py_library( - name = "mnist", - srcs = ["mnist.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:model_registry", - "//lingvo/core:base_model_params", - "//lingvo/core:program_lib", - "//lingvo/tasks/image:classifier", - "//lingvo/tasks/image:input_generator", - ], -) - -py_library( - name = "params", - srcs = ["params.py"], - srcs_version = "PY3", - tags = ["keep_dep"], - deps = [ - ":mnist", - ], -) diff --git a/lingvo/tasks/image/params/mnist.py b/lingvo/tasks/image/params/mnist.py deleted file mode 100644 index 273d821f..00000000 --- a/lingvo/tasks/image/params/mnist.py +++ /dev/null @@ -1,66 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Train models on MNIST data.""" - -from lingvo import model_registry -from lingvo.core import base_model_params -from lingvo.tasks.image import classifier -from lingvo.tasks.image import input_generator - - -class Base(base_model_params.SingleTaskModelParams): - """Input params for MNIST.""" - - @property - def path(self): - # Generated using lingvo/tools:keras2ckpt. - return '/tmp/mnist/mnist' - - def Train(self): - p = input_generator.MnistTrainInput.Params() - p.ckpt = self.path - return p - - def Test(self): - p = input_generator.MnistTestInput.Params() - p.ckpt = self.path - return p - - def Dev(self): - return self.Test() - - -@model_registry.RegisterSingleTaskModel -class LeNet5(Base): - """LeNet params for MNIST classification.""" - - BN = False - DROP = 0.2 - - def Task(self): - p = classifier.ModelV1.Params() - p.name = 'lenet5' - # Overall architecture: - # conv, maxpool, conv, maxpool, fc, softmax. - p.filter_shapes = [(5, 5, 1, 20), (5, 5, 20, 50)] - p.window_shapes = [(2, 2), (2, 2)] - p.batch_norm = self.BN - p.dropout_prob = self.DROP - p.softmax.input_dim = 300 - p.softmax.num_classes = 10 - p.train.save_interval_seconds = 10 # More frequent checkpoints. - p.eval.samples_per_summary = 0 # Eval the whole set. - return p diff --git a/lingvo/tasks/image/params/params.py b/lingvo/tasks/image/params/params.py deleted file mode 100644 index 907e9ff9..00000000 --- a/lingvo/tasks/image/params/params.py +++ /dev/null @@ -1,21 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Image models hyper-parameters.""" - -# Import ModelParams to ensure that they are added to the global registry. -# pylint: disable=unused-import -import lingvo.tasks.image.params.mnist -# pylint: enable=unused-import diff --git a/lingvo/tasks/lm/BUILD b/lingvo/tasks/lm/BUILD deleted file mode 100644 index 6572e026..00000000 --- a/lingvo/tasks/lm/BUILD +++ /dev/null @@ -1,104 +0,0 @@ -load( - "//lingvo:lingvo.bzl", - "lingvo_cuda_py_test", -) - -# Language modelling related tasks. -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -py_library( - name = "input_generator", - srcs = ["input_generator.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_input_generator", - "//lingvo/core:generic_input", - "//lingvo/core:py_utils", - "//lingvo/core:tokenizers", - ], -) - -py_test( - name = "input_generator_test", - srcs = ["input_generator_test.py"], - data = [ - "//lingvo/tasks/lm/testdata:lm1b_100", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":input_generator", - "//lingvo:compat", - "//lingvo/core:test_helper", - "//lingvo/core:test_utils", - ], -) - -py_library( - name = "layers", - srcs = ["layers.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:batch_major_attention", - "//lingvo/core:layers", - "//lingvo/core:layers_with_attention", - "//lingvo/core:layers_with_gpipe", - "//lingvo/core:py_utils", - "//lingvo/core:rnn_cell", - "//lingvo/core:rnn_layers", - ], -) - -lingvo_cuda_py_test( - name = "layers_test", - srcs = ["layers_test.py"], - python_version = "PY3", - shard_count = 10, - tags = ["noasan"], - deps = [ - ":layers", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "model", - srcs = ["model.py"], - srcs_version = "PY3", - deps = [ - ":layers", - "//lingvo:compat", - "//lingvo/core:base_model", - "//lingvo/core:py_utils", - "//lingvo/core:schedule", - ], -) - -py_test( - name = "model_test", - srcs = ["model_test.py"], - data = [ - "//lingvo/tasks/lm/testdata:lm1b_100", - "//lingvo/tasks/lm/testdata:small_word_vocab", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":input_generator", - ":model", - "//lingvo:compat", - "//lingvo/core:test_helper", - "//lingvo/core:test_utils", - "//lingvo/core:tokenizers", - ], -) diff --git a/lingvo/tasks/lm/__init__.py b/lingvo/tasks/lm/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/lingvo/tasks/lm/input_generator.py b/lingvo/tasks/lm/input_generator.py deleted file mode 100644 index 1adf1cc2..00000000 --- a/lingvo/tasks/lm/input_generator.py +++ /dev/null @@ -1,92 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Language model input generator.""" - -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import generic_input -from lingvo.core import py_utils -from lingvo.core import tokenizers - - -class LmInput(base_input_generator.BaseSequenceInputGenerator): - """Reads tokenized plain text input such as from lm1b.""" - - @classmethod - def Params(cls): - """Defaults params for `LmInput`.""" - p = super().Params() - p.Define('fixed_input_shape', False, 'Fixed input shape or not.') - p.tokenizer = tokenizers.AsciiTokenizer.Params() - return p - - def __init__(self, params): - params.pad_to_max_seq_length = True - super().__init__(params) - p = self.params - p.fixed_input_shape = p.fixed_input_shape or py_utils.use_tpu() - - (text, self._word_count), self._bucket_keys = self._BuildDataSource() - self._ids, self._labels, self._paddings = self.StringsToIds(text) - tf.summary.histogram('examples/sequence_length', - tf.reduce_sum(1.0 - self._paddings, axis=1)) - self._weights = 1.0 - self._paddings - if p.fixed_input_shape: - if py_utils.use_tpu(): - # When flush_every_n is on, at end of each epoch, our input - # generator can generate a batch smaller than - # bucket_batch_limit - assert not p.flush_every_n, 'flush_every_n is not allowed on TPU.' - assert min(self.infeed_bucket_batch_limit) == max( - self.infeed_bucket_batch_limit) - bs = min(self.infeed_bucket_batch_limit) - else: - bs = max(self.infeed_bucket_batch_limit) - - def SetShape(x): - x.set_shape([bs, p.target_max_length]) - - SetShape(self._ids) - SetShape(self._labels) - SetShape(self._paddings) - SetShape(self._weights) - self._word_count.set_shape([bs]) - - def InfeedBatchSize(self): - """Override BaseSequenceInputGenerator.""" - return tf.shape(self._ids)[0] - - def _DataSourceFromFilePattern(self, file_pattern): - - def ReadInput(line): - word_count = tf.size(tf.strings.split([line])) - strlen = tf.size(tf.strings.split([line], '')) - return [line, word_count], strlen - - return generic_input.GenericInput( - file_pattern=file_pattern, - processor=ReadInput, - **self.CommonInputOpArgs()) - - def _InputBatch(self): - ret = py_utils.NestedMap() - ret.bucket_key = self._bucket_keys - ret.ids = self._ids - ret.labels = self._labels - ret.paddings = self._paddings - ret.weights = self._weights - ret.word_count = self._word_count - return ret diff --git a/lingvo/tasks/lm/input_generator_test.py b/lingvo/tasks/lm/input_generator_test.py deleted file mode 100644 index 1bc722d4..00000000 --- a/lingvo/tasks/lm/input_generator_test.py +++ /dev/null @@ -1,58 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lm.input_generator.""" - -import lingvo.compat as tf -from lingvo.core import test_helper -from lingvo.core import test_utils -from lingvo.tasks.lm import input_generator - - -class InputGeneratorTest(test_utils.TestCase): - - def _InputParams(self): - p = input_generator.LmInput.Params() - p.file_pattern = "text:" + test_helper.test_src_dir_path( - "tasks/lm/testdata/lm1b_100.txt") - p.file_random_seed = 31415 - p.file_parallelism = 1 - p.file_buffer_size = 1 - p.bucket_upper_bound = [20] - p.bucket_batch_limit = [2] - p.target_max_length = 20 - return p - - def testLmInputGen(self): - p = self._InputParams() - - with self.session(use_gpu=False): - inp = p.Instantiate() - inp_batch = self.evaluate(inp.GetPreprocessedInputBatch()) - print(inp_batch) - # pyformat: disable - # pylint: disable=line-too-long - expected_ids = [ - [1, 13, 24, 3, 23, 5, 13, 8, 3, 24, 12, 5, 24, 3, 24, 27, 19, 3, 19, 10], - [1, 24, 12, 9, 3, 17, 5, 14, 19, 22, 13, 24, 29, 3, 27, 13, 16, 16, 3, 6], - ] - # pylint: enable=line-too-long - # pyformat: enable - self.assertEqual(expected_ids, inp_batch.ids.tolist()) - self.assertEqual([[1.0] * 20, [1.0] * 20], inp_batch.weights.tolist()) - - -if __name__ == "__main__": - tf.test.main() diff --git a/lingvo/tasks/lm/layers.py b/lingvo/tasks/lm/layers.py deleted file mode 100644 index 6cb4204e..00000000 --- a/lingvo/tasks/lm/layers.py +++ /dev/null @@ -1,1322 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Common layers for language models.""" - -import math -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import batch_major_attention -from lingvo.core import layers -from lingvo.core import layers_with_attention -from lingvo.core import layers_with_gpipe -from lingvo.core import py_utils -from lingvo.core import rnn_cell -from lingvo.core import rnn_layers - - -class BaseLanguageModel(base_layer.BaseLayer): - """Abstract base class for a language model layer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('vocab_size', 0, 'Number of vocabulary tokens.') - return p - - @classmethod - def UpdateTargetVocabSize(cls, p, vocab_size, wpm_model=None): - """Sets the params with the given vocab size and wpm model. - - Args: - p: model params. - vocab_size: size of the vocabulary. - wpm_model: file name prefix pointing to a wordpiece model. - - Returns: - Model params updated with the vocab size and wpm model. - """ - p.vocab_size = vocab_size - return p - - def zero_state(self, theta, batch_size): - raise NotImplementedError('Abstract method') - - def FProp(self, theta, inputs, paddings, state0, *args, **kwargs): - """Computes xent loss given the language model inputs. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - inputs: a tensor of shape [time, batch] or [time, batch, dims]. - paddings: a 0/1 tensor of shape [time, batch]. - state0: A `.NestedMap` containing the initial recurrent state. - *args: optional extra arguments. - **kwargs: optional extra keyword arguments. - - Returns: - (xent_output, state1). `xent_output` is a `.NestedMap` as defined by - `SoftmaxLayer`'s return value and `state1` is the next recurrent state. - """ - raise NotImplementedError('Abstract method') - - def Logits(self, theta, inputs, paddings, *args, **kwargs): - """FProp and returns the logits for the whole sequence.""" - xent_output, _ = self.FProp( - theta, - inputs, - paddings, - state0=self.zero_state(theta, - tf.shape(inputs)[1]), - *args, - **kwargs) - return xent_output.logits - - @classmethod - def StepOutputDimension(cls, params): - """Returns dimensions of `Step()`'s output dimension. - - Args: - params: Params for this layer. - - Returns: - A `.NestedMap` with fields - logits: a python int. - The vocab size. - last_hidden: a python int. - The last hidden layer's dimension. - """ - raise NotImplementedError('Abstract method') - - def Step(self, theta, inputs, paddings, state0, *args, **kwargs): - """FProp one step. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - inputs: a tensor of shape [batch] or [batch, dims]. - paddings: a 0/1 tensor of shape [batch]. - state0: A `.NestedMap` containing the initial recurrent state. - *args: optional extra arguments. - **kwargs: optional extra keyword arguments. - - Returns: - A tuple (output, state1). - output: A `.NestedMap` with fields. - logits: - [batch, vocab_size]. - log_probs: - [batch, vocab_size]. - last_hidden: - [batch, dims]. - state1: - The new recurrent state. - """ - - def ExpandTime(x): - return tf.expand_dims(x, axis=0) - - xent_output, state1 = self.FProp( - theta=theta, - inputs=ExpandTime(inputs), - paddings=ExpandTime(paddings), - state0=state0, - *args, - **kwargs) - - output = py_utils.NestedMap() - output.log_probs = tf.squeeze(xent_output.log_probs, axis=0) - output.probs = tf.squeeze(xent_output.probs, axis=0) - output.last_hidden = tf.squeeze(xent_output.last_hidden, axis=0) - if 'logits' in xent_output: - # FstLm doesn't return logits. - output.logits = tf.squeeze(xent_output.logits, axis=0) - return output, state1 - - def GetFeedDict(self): - """Returns an optional feed dict with str keys and Tensor values.""" - return {} - - def CombineStates(self, state0, state1, switch_cond): - """Combines states based on a switch conditional. - - Args: - state0: a NestedMap of states to use for batch elements where switch_cond - is true. - state1: a NestedMap of states to use for batch elements where switch_cond - is false. - switch_cond: bool tensor of shape [batch] on which to switch. - - Returns: - a NestedMap of states. - """ - raise NotImplementedError('Abstract method') - - -class NullLm(BaseLanguageModel): - """A trivial language model does nothing really.""" - - def zero_state(self, theta, batch_size): - return py_utils.NestedMap( - m=tf.zeros([batch_size, 0], dtype=self.params.dtype)) - - def FProp(self, theta, inputs, paddings, state0, *args, **kwargs): - p = self.params - time = tf.shape(inputs)[0] - batch = tf.shape(inputs)[1] - logits = tf.zeros([time, batch, p.vocab_size], dtype=p.dtype) - return py_utils.NestedMap( - logits=logits, - probs=tf.nn.softmax(logits), - log_probs=tf.nn.log_softmax(logits), - last_hidden=tf.zeros([time, batch, 0], dtype=p.dtype)), state0 - - def Logits(self, theta, inputs, paddings, *args, **kwargs): - """FProp and returns the logits for the whole sequence.""" - p = self.params - del theta, paddings - time, batch = tf.unstack(tf.shape(inputs)[:2]) - return tf.zeros([time, batch, p.vocab_size], dtype=p.dtype) - - @classmethod - def StepOutputDimension(cls, params): - """Returns dimensions of `Step()`'s output dimension.""" - return py_utils.NestedMap(logits=params.vocab_size, last_hidden=0) - - def Step(self, theta, inputs, paddings, state0, *args, **kwargs): - """FProp one step.""" - p = self.params - batch = tf.shape(inputs)[0] - logits = tf.zeros([batch, p.vocab_size], dtype=p.dtype) - return py_utils.NestedMap( - logits=logits, - log_probs=tf.nn.log_softmax(logits), - probs=tf.nn.softmax(logits), - last_hidden=tf.zeros([batch, 0], dtype=p.dtype)), state0 - - def CombineStates(self, state0, state1, switch_cond): - """Combines states based on a switch conditional. - - Args: - state0: a NestedMap of states to use for batch elements where switch_cond - is true. - state1: a NestedMap of states to use for batch elements where switch_cond - is false. - switch_cond: bool tensor of shape [batch] on which to switch. - - Returns: - a NestedMap of states. - """ - return state0 - - -def _RnnOutputSize(rnns): - cell = rnns.cell_tpl[-1] - return cell.num_output_nodes - - -def ComputeXentOutput(softmax_layer, - softmax_theta, - activations, - labels, - num_samples=1): - """Compute Softmax CrossEntropy output.""" - seqlen, batch, _ = tf.unstack(tf.shape(activations), num=3) - if labels is None: - # We can only compute the logits here. - logits = softmax_layer.Logits( - theta=softmax_theta, - inputs=tf.reshape(activations, [seqlen * batch * num_samples, -1])) - xent_output = py_utils.NestedMap( - logits=tf.reshape(logits, [seqlen, batch, -1])) - elif 'class_ids' in labels: - # labels.class_ids: [len, batch] - if num_samples > 1: - class_ids = tf.tile(labels.class_ids, [1, num_samples]) - class_weights = tf.tile(labels.class_weights, [1, num_samples]) - else: - class_ids = labels.class_ids - class_weights = labels.class_weights - xent_output = softmax_layer.FProp( - theta=softmax_theta, - inputs=activations, - class_weights=class_weights, - class_ids=class_ids) - else: - assert 'class_probabilities' in labels - if num_samples > 1: - class_probabilities = tf.tile(labels.class_probabilities, - [1, num_samples]) - class_weights = tf.tile(labels.class_weights, [1, num_samples]) - else: - class_probabilities = labels.class_probabilities - class_weights = labels.class_weights - xent_output = softmax_layer.FProp( - theta=softmax_theta, - inputs=activations, - class_weights=class_weights, - class_probabilities=class_probabilities) - return xent_output - - -class RnnLmNoEmbedding(BaseLanguageModel): - """Stacked RNN based language model layer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('rnns', rnn_layers.StackedFRNNLayerByLayer.Params(), - 'The stacked-RNNs layer params.') - p.Define('softmax', layers.SimpleFullSoftmax.Params(), - 'The softmax layer params.') - p.Define( - 'direct_features_dim', 0, - 'If > 0, then the number of dimensions of direct features ' - 'that bypass the RNN and are provided directly to the softmax ' - 'input.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if not isinstance(p.rnns.cell_tpl, (list, tuple)): - p.rnns.cell_tpl = [p.rnns.cell_tpl] - p.rnns.allow_implicit_capture = p.allow_implicit_capture - - cell_output_size = _RnnOutputSize(p.rnns) - output_layer_size = cell_output_size + p.direct_features_dim - - if output_layer_size != p.softmax.input_dim: - raise ValueError( - 'Output layer size %d does not match softmax input size %d! ' - 'cell_output_size: %d direct_features_dim: %d ' % - (output_layer_size, p.softmax.input_dim, cell_output_size, - p.direct_features_dim)) - if p.softmax.num_classes != p.vocab_size: - raise ValueError( - 'softmax num of classess %d does not match vocabulary size %d!' % - (p.softmax.num_classes, p.vocab_size)) - - self.CreateChild('rnns', p.rnns) - self.CreateChild('softmax', p.softmax) - - def zero_state(self, theta, batch_size): - return self.rnns.zero_state(theta.rnns, batch_size) - - @classmethod - def StepOutputDimension(cls, params): - return py_utils.NestedMap( - logits=params.vocab_size, last_hidden=params.softmax.input_dim) - - def Step(self, - theta, - inputs, - paddings, - state0, - direct_features=None, - *args, - **kwargs): - """FProp one step. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - inputs: a tensor of shape [batch] or [batch, dims]. - paddings: a 0/1 tensor of shape [batch]. - state0: A `.NestedMap` containing the initial recurrent state. - direct_features: If not None, a tensor of [batch, direct_feature_dims] - that is concatenated to the output of the last RNN layer. - *args: optional extra arguments. - **kwargs: optional extra keyword arguments. - - Returns: - A tuple (output, state1). - output: A `.NestedMap` with fields. - logits: - [batch, vocab_size]. - last_hidden: - [batch, dims]. - state1: - The new recurrent state. - """ - - def ExpandTime(x): - return tf.expand_dims(x, axis=0) - - if direct_features is not None: - direct_features = py_utils.HasRank(direct_features, 2) - direct_features = ExpandTime(direct_features) - - xent_output, state1 = self.FProp( - theta=theta, - inputs=ExpandTime(inputs), - paddings=ExpandTime(paddings), - state0=state0, - direct_features=direct_features, - *args, - **kwargs) - - output = py_utils.NestedMap() - output.logits = tf.squeeze(xent_output.logits, axis=0) - output.probs = tf.squeeze(xent_output.probs, axis=0) - output.log_probs = tf.squeeze(xent_output.log_probs, axis=0) - output.last_hidden = tf.squeeze(xent_output.last_hidden, axis=0) - return output, state1 - - def FProp(self, - theta, - inputs, - paddings, - state0, - labels=None, - direct_features=None): - """Computes xent loss given the language model input activations. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - inputs: input activation. A tensor of shape [time, batch, dims]. - paddings: a 0/1 tensor of shape [time, batch]. - state0: A `.NestedMap` containing the initial recurrent state. - labels: If not None, a `.NestedMap` containing the following fields. - - - class_weights, a tensor with shape [time, batch] containing the - weights for each target word. - - class_ids, a tensor with shape [time, batch] of int32 dtype containing - the target class labels. - - class_probabilities, a tensor with shape [time, batch, vocab_size] of - float values indicating class-membership probabilities. - direct_features: - If not None, a tensor of [time, batch, direct_feature_dims] that is - concatenated to the output of the last RNN layer. - - Returns: - If `labels` is not None, returns (xent_output, state1), where - `xent_output` is a `.NestedMap` as defined by `SoftmaxLayer`'s return - value and `state1` is the next recurrent state. Otherwise, - `xent_output` contains the softmax logits, probabilities (.probs) and - log-probabilities (.log_probs). - """ - inputs = py_utils.HasRank(inputs, 3) - seqlen, batch, _ = tf.unstack(tf.shape(inputs), num=3) - paddings = py_utils.HasShape(paddings, [seqlen, batch]) - assert state0 is not None - activation, state1 = self.rnns.FProp(theta.rnns, inputs, - tf.expand_dims(paddings, 2), state0) - - if direct_features is not None: - direct_features = py_utils.HasRank(direct_features, 3) - activation = tf.concat([activation, direct_features], axis=2) - - if labels is None: - # We can only compute the logits here. - logits = self.softmax.Logits( - theta=theta.softmax, - inputs=tf.reshape(activation, [seqlen * batch, -1])) - xent_output = py_utils.NestedMap( - logits=tf.reshape(logits, [seqlen, batch, -1])) - xent_output.probs = tf.nn.softmax(xent_output.logits) - xent_output.log_probs = tf.nn.log_softmax(xent_output.logits) - elif 'class_ids' in labels: - xent_output = self.softmax.FProp( - theta=theta.softmax, - inputs=activation, - class_weights=labels.class_weights, - class_ids=labels.class_ids) - else: - assert 'class_probabilities' in labels - xent_output = self.softmax.FProp( - theta=theta.softmax, - inputs=activation, - class_weights=labels.class_weights, - class_probabilities=labels.class_probabilities) - xent_output.last_hidden = activation - return xent_output, state1 - - def CombineStates(self, state0, state1, switch_cond): - """Combines states based on a switch conditional. - - Args: - state0: a NestedMap of states to use for batch elements where switch_cond - is true. - state1: a NestedMap of states to use for batch elements where switch_cond - is false. - switch_cond: bool tensor of shape [batch] on which to switch. - - Returns: - a NestedMap of states. - """ - updated_rnn_states = [] - for i in range(self.params.rnns.num_layers): - updated_rnn_states.append( - py_utils.NestedMap({ - 'c': tf.where(switch_cond, state0.rnn[i].c, state1.rnn[i].c), - 'm': tf.where(switch_cond, state0.rnn[i].m, state1.rnn[i].m) - })) - combined_state = py_utils.NestedMap({'rnn': updated_rnn_states}) - return combined_state - - -class RnnLm(RnnLmNoEmbedding): - """Stacked RNN based language model layer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('emb', layers.EmbeddingLayer.Params(), - 'The embedding layer params.') - p.Define('embedding_dropout_keep_prob', 1.0, 'Embedding dropout keep prob.') - p.Define('embedding_dropout_seed', None, 'Embedding dropout seed.') - p.emb.max_num_shards = 1 - return p - - # TODO(zhifengc): Consider merge Params() and CommonParams(). - @classmethod - def CommonParams(cls, - vocab_size, - emb_dim=1024, - num_layers=2, - rnn_dims=2048, - rnn_hidden_dims=0, - residual_start=1, - layer_norm=False, - softmax_max_alloc=None): - """A LM model parameterized by vocab size, etc. - - Args: - vocab_size: Vocab size. - emb_dim: Embedding dimension. - num_layers: The number of rnn layers. - rnn_dims: Each RNN layer has this many output nodes. - rnn_hidden_dims: If > 0, each RNN layer has this many hidden nodes. - residual_start: index of the first layer with a residual connection; - higher index layers also have residuals. - layer_norm: If true, use LayerNormalizedLSTMCellSimple otherwise use - LSTMCellSimple. - softmax_max_alloc: If set to a positive integer the soft-max - computation is chunked into allocations of at most - `softmax_max_alloc`; when left to its default value of None no - chunking is done. - - Returns: - A `RnnLm` parameter object. - """ - p = cls.Params() - p.vocab_size = vocab_size - - init_scale = 1.0 / math.sqrt(rnn_dims) - - # Embedding. - p.emb.vocab_size = vocab_size - p.emb.embedding_dim = emb_dim - p.emb.scale_sqrt_depth = True - p.emb.params_init = py_utils.WeightInit.Uniform(init_scale) - - # RNNs - p.rnns.num_layers = num_layers - # Which layer starts to have the residual connection. - p.rnns.skip_start = residual_start - lstm_cell_tpl = rnn_cell.LayerNormalizedLSTMCellSimple.Params( - ) if layer_norm else rnn_cell.LSTMCellSimple.Params() - lstm_cell_tpl.Set( - num_input_nodes=rnn_dims, - num_output_nodes=rnn_dims, - num_hidden_nodes=rnn_hidden_dims) - if num_layers > 1: - p.rnns.cell_tpl = [lstm_cell_tpl.Copy(), lstm_cell_tpl.Copy()] - else: - p.rnns.cell_tpl = [lstm_cell_tpl.Copy()] - p.rnns.cell_tpl[0].num_input_nodes = emb_dim - - # Softmax - p.softmax.input_dim = rnn_dims - p.softmax.num_classes = vocab_size - p.softmax.params_init = py_utils.WeightInit.Uniform(init_scale) - if softmax_max_alloc: - # If the vocab is very large, computes the softmax chunk-by-chunk. - p.softmax.chunk_size = max(1, int(softmax_max_alloc / vocab_size)) - - return p - - def __init__(self, params, verify_sizes=True): - super().__init__(params) - p = self.params - - if verify_sizes: - assert p.emb.vocab_size == p.vocab_size, ('{} vs. {}'.format( - p.emb.vocab_size, p.vocab_size)) - assert p.emb.embedding_dim == p.rnns.cell_tpl[0].num_input_nodes, ( - '{} vs. {}'.format(p.emb.embedding_dim, - p.rnns.cell_tpl[0].num_input_nodes)) - - self.CreateChild('emb', p.emb) - - def FProp(self, - theta, - inputs, - paddings, - state0, - labels=None, - direct_features=None): - """Computes xent loss given the language model input activations. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - inputs: input ids. An int32 tensor of shape [time, batch]. - paddings: a 0/1 tensor of shape [time, batch]. - state0: A `.NestedMap` containing the initial recurrent state. - labels: If not None, a `.NestedMap` containing the following fields: - - - class_weights, a tensor with shape [time, batch] containing the - weights for each target word. - - class_ids, a tensor with shape [time, batch] of int32 dtype containing - the target class labels. - - class_probabilities, a tensor with shape [time, batch, vocab_size] of - float values indicating class-membership probabilities. - direct_features: - If not None, a tensor of [time, batch, direct_feature_dims] that is - concatenated to the output of the last RNN layer. - - Returns: - If `labels` is not None, returns (xent_output, state1), where - `xent_output` is a `.NestedMap` as defined by `SoftmaxLayer`'s return - value and `state1` is the next recurrent state. Otherwise, - `xent_output` only contains the softmax logits. - """ - ids = py_utils.HasRank(inputs, 2) - paddings = py_utils.HasShape(paddings, tf.shape(ids)) - assert state0 - activation = self.emb.EmbLookup(theta.emb, ids) - # Dropout on embeddings is only applied in training. - p = self.params - if p.embedding_dropout_keep_prob < 1.0 and not self.do_eval: - activation = tf.nn.dropout( - activation, - rate=1 - p.embedding_dropout_keep_prob, - seed=p.embedding_dropout_seed) - return super().FProp(theta, activation, paddings, state0, labels, - direct_features) - - -class ConditionalRnnLm(RnnLmNoEmbedding): - """RnnLm where looked up embedding is concatenated with a condition vector.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('condition_dim', 128, 'The size of the condition vector.') - p.Define('emb', layers.EmbeddingLayer.Params(), - 'The embedding layer params.') - p.Define( - 'embedding_dropout_keep_prob', 1.0, 'Embedding dropout keep prob.' - 'Dropout is applied after concatenating with condition vector.') - p.Define('embedding_dropout_seed', None, 'Embedding dropout seed.') - p.emb.max_num_shards = 1 - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - assert p.emb.vocab_size == p.vocab_size, ('{} vs. {}'.format( - p.emb.vocab_size, p.vocab_size)) - assert (p.emb.embedding_dim + p.condition_dim == - p.rnns.cell_tpl[0].num_input_nodes), ('{} vs. {}'.format( - p.emb.embedding_dim, p.rnns.cell_tpl[0].num_input_nodes)) - - self.CreateChild('emb', p.emb) - - def FProp(self, - theta, - inputs, - paddings, - state0, - condition, - labels=None, - direct_features=None): - """Computes xent loss given the language model input and condition. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: input ids. An int32 tensor of shape [time, batch]. - paddings: a 0/1 tensor of shape [time, batch]. - state0: A `.NestedMap` containing the initial recurrent state. - condition: input condition. A tensor of shape [batch, condition_dim]. - labels: If not None, a `.NestedMap` containing the following fields: - - class_weights, a tensor with shape [time, batch] containing the weights - for each target word. - class_ids, a tensor with shape [time, batch] of - int32 dtype containing the target class labels. - class_probabilities, a - tensor with shape [time, batch, vocab_size] of float values indicating - class-membership probabilities. - direct_features: If not None, a tensor of [time, batch, - direct_feature_dims] that is concatenated to the output of the last RNN - layer. - - Returns: - If `labels` is not None, returns (xent_output, state1), where - `xent_output` is a `.NestedMap` as defined by `SoftmaxLayer`'s return - value and `state1` is the next recurrent state. Otherwise, - `xent_output` only contains the softmax logits. - """ - p = self.params - # `condition` should have shape (batch_size, dim) - condition = py_utils.HasShape(condition, - [tf.shape(paddings)[1], p.condition_dim]) - # Expand the time dimension -> (time, batch_size, dim) - condition = tf.tile( - tf.expand_dims(condition, 0), [tf.shape(inputs)[0], 1, 1]) - - ids = py_utils.HasRank(inputs, 2) - paddings = py_utils.HasShape(paddings, tf.shape(ids)) - activation = self.emb.EmbLookup(theta.emb, ids) - activation = tf.concat([activation, tf.cast(condition, p.dtype)], -1) - # Dropout on embeddings is only applied in training. - if p.embedding_dropout_keep_prob < 1.0 and not self.do_eval: - activation = tf.nn.dropout( - activation, - rate=1 - p.embedding_dropout_keep_prob, - seed=p.embedding_dropout_seed) - return super().FProp(theta, activation, paddings, state0, labels, - direct_features) - - -class MoeLm(BaseLanguageModel): - """Mixture of experts language modeling class.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define( - 'emb', - layers.EmbeddingLayer.Params().Set(max_num_shards=1), - 'The embedding layer params.') - p.Define('shared_emb', True, 'If true, uses a single embedding') - p.Define( - 'add_postgating_rnn', True, 'If true, add an RNNLM post gating. ' - 'If false, add only a softmax on top.') - p.Define('rnns', rnn_layers.StackedFRNNLayerByLayer.Params(), - 'The stacked-RNNs layer params.') - p.Define('number_of_experts', 7, 'Number of experts.') - p.Define('merge', RnnLmNoEmbedding.Params(), - 'The LM to use for the merged LM') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if not isinstance(p.rnns.cell_tpl, (list, tuple)): - p.rnns.cell_tpl = [p.rnns.cell_tpl] - - assert p.emb.vocab_size == p.vocab_size, ('{} vs. {}'.format( - p.emb.vocab_size, p.vocab_size)) - assert p.emb.embedding_dim == p.rnns.cell_tpl[0].num_input_nodes, ( - '{} vs. {}'.format(p.emb.embedding_dim, - p.rnns.cell_tpl[0].num_input_nodes)) - if p.add_postgating_rnn: - assert p.merge.vocab_size == p.vocab_size, ('{} vs. {}'.format( - p.merge.vocab_size, p.vocab_size)) - - # Embeddings - if p.shared_emb: - self.CreateChild('emb', p.emb) - else: - # 0-th embedding is for the domain predictor. - self.CreateChildren('emb', [ - p.emb.Copy().Set(name='emb_%d' % i) - for i in range(1 + p.number_of_experts) - ]) - - # Rnns - # 0-th rnns is for the domain predictor. - self.CreateChildren('rnns', - [p.rnns.Copy() for i in range(1 + p.number_of_experts)]) - - # Softmax - rnn_output_size = _RnnOutputSize(p.rnns) - sm_params = layers.SimpleFullSoftmax.Params() - sm_params.name = 'domain_predictor_softmax' - sm_params.input_dim = rnn_output_size - sm_params.num_classes = p.number_of_experts - self.CreateChild('domain_predictor_softmax', sm_params) - - # Merge - if p.add_postgating_rnn: - self.CreateChild('merge', p.merge) - else: - output_sm_params = layers.SimpleFullSoftmax.Params() - output_sm_params.name = 'output_softmax' - output_sm_params.input_dim = rnn_output_size - output_sm_params.num_classes = p.vocab_size - self.CreateChild('output_softmax', output_sm_params) - - def zero_state(self, theta, batch_size): - p = self.params - state0 = py_utils.NestedMap(rnns=[ - x.zero_state(x_theta, batch_size) - for x, x_theta in zip(self.rnns, theta.rnns) - ]) - if p.add_postgating_rnn: - state0.merge = self.merge.zero_state(theta.merge, batch_size) - return state0 - - def FProp(self, theta, inputs, paddings, state0, labels=None): - """Forward compute.""" - p = self.params - - ids = py_utils.HasRank(inputs, 2) - paddings = py_utils.HasShape(paddings, tf.shape(ids)) - seqlen, batch = tf.unstack(tf.shape(inputs), num=2) - assert state0 - - paddings_3d = tf.expand_dims(paddings, axis=2) - - # RNNs - if p.shared_emb: - emb_act = [self.emb.EmbLookup(theta.emb, inputs) - ] * (1 + p.number_of_experts) - else: - emb_act = [ - self.emb[i].EmbLookup(theta.emb[i], inputs) - for i in range(1 + p.number_of_experts) - ] - state1 = py_utils.NestedMap(rnns=[]) - rnns_act = [] - for i, act in enumerate(emb_act): - act, state = self.rnns[i].FProp(theta.rnns[i], act, paddings_3d, - state0.rnns[i]) - act = py_utils.HasRank(act, 3) - rnns_act += [act] - state1.rnns += [state] - - # [time, batch, experts, dims]. - expert_stacked = tf.stack(rnns_act[1:], axis=2) - - # Compute gating softmax. The 0-th rnns is used as the expert - # predictor. Because SoftmaxLayer.Logits takes a matrix as input, - # we reshape rnns_act[0], the domain predictor activation, to a - # matrix here. - act = tf.reshape(rnns_act[0], [seqlen * batch, -1]) - logits = self.domain_predictor_softmax.Logits( - theta.domain_predictor_softmax, act) - # [time, batch, experts] - gating = tf.reshape(tf.nn.softmax(logits), [seqlen, batch, -1]) - - # Mix the experts. - # [time, batch, dims] - combined = tf.squeeze( - tf.matmul( - # [time, batch, 1, experts] - tf.expand_dims(gating, axis=2), - # [time, batch, experts, dims] - expert_stacked), - axis=2) - - if p.add_postgating_rnn: - # Note that this layer includes 1 or more RNN layers followed - # by a softmax. - xent_loss, state1.merge = self.merge.FProp(theta.merge, combined, - paddings, state0.merge, labels) - else: - xent_loss = self.output_softmax.FProp( - theta=theta.output_softmax, - inputs=combined, - class_weights=labels.class_weights, - class_ids=labels.class_ids) - - # return xent_loss, state1 - return xent_loss, state1 - - -class TransformerLmNoEmbedding(BaseLanguageModel): - """Transformer language model.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('position_emb', layers.PositionalEmbeddingLayer.Params(), - 'Position embedding layer params.') - p.Define( - 'model_dim', 512, 'Model dimension that applies to embedding ' - 'layers and all Transformer layers.') - p.Define('num_trans_layers', 6, 'Number of Transformer layers.') - p.Define('trans_tpl', layers_with_attention.TransformerLayer.Params(), - 'Transformer Layer params.') - p.Define('input_dropout_prob', 0.0, 'Prob at which we do input dropout.') - p.Define( - 'residual_dropout_prob', 0.0, 'Dropout prob to the output of ' - 'each sub-layer before it is added to the sub-layer input.') - p.Define( - 'atten_dropout_prob', 0.0, 'Dropout prob to the attention ' - 'weights in each Transformer attention sub-layer.') - p.Define( - 'relu_dropout_prob', 0.0, 'Dropout prob to the inner layer ' - 'output (ReLU activation) in each Transformer feed-forward ' - 'sub-layer.') - p.Define('softmax', layers.SimpleFullSoftmax.Params(), - 'The softmax layer params.') - - # Default config for the transformer layers. - p.trans_tpl.has_aux_atten = False - p.trans_tpl.mask_self_atten = True - p.trans_tpl.tr_atten_tpl.num_attention_heads = 8 - p.trans_tpl.tr_atten_tpl.atten_tpl.enable_ctx_pre_proj = True - p.trans_tpl.tr_atten_tpl.atten_tpl.enable_ctx_post_proj = True - p.trans_tpl.tr_fflayer_tpl.hidden_dim = 2048 - - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - p.trans_tpl.tr_atten_tpl.residual_dropout_prob = p.residual_dropout_prob - p.trans_tpl.tr_atten_tpl.atten_dropout_prob = p.atten_dropout_prob - p.trans_tpl.tr_fflayer_tpl.residual_dropout_prob = p.residual_dropout_prob - p.trans_tpl.tr_fflayer_tpl.relu_dropout_prob = p.relu_dropout_prob - - p.position_emb.embedding_dim = p.model_dim - self.CreateChild('position_emb', p.position_emb) - - dropout_tpl = layers.DropoutLayer.Params().Set( - keep_prob=(1.0 - p.input_dropout_prob)) - self.CreateChild('input_dropout', dropout_tpl) - - params_trans_layers = [] - for i in range(p.num_trans_layers): - params = p.trans_tpl.Copy() - params.source_dim = p.model_dim - params.name = 'layer_%d' % i - params_trans_layers.append(params) - self.CreateChildren('trans', params_trans_layers) - - p.softmax.input_dim = p.model_dim - p.softmax.num_classes = p.vocab_size - self.CreateChild('softmax', p.softmax) - - def zero_state(self, theta, batch_size): - p = self.params - state0 = py_utils.NestedMap() - for layer in range(p.num_trans_layers): - state0['layer_%d' % layer] = py_utils.NestedMap({ - 'key': tf.zeros([0, batch_size, p.model_dim]), - 'value': tf.zeros([0, batch_size, p.model_dim]), - }) - return state0 - - @classmethod - def StepOutputDimension(cls, params): - return py_utils.NestedMap( - logits=params.vocab_size, last_hidden=params.softmax.input_dim) - - def Step(self, theta, inputs, paddings, state0, *args, **kwargs): - """FProp one step. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - inputs: a tensor of shape [batch, model_dim]. - paddings: a 0/1 tensor of shape [batch]. Unused here. - state0: A `.NestedMap` containing the prefix states up to step t-1. - *args: optional extra arguments. - **kwargs: optional extra keyword arguments. - - Returns: - A tuple (output, state1). - output: A `.NestedMap` with fields. - logits: - [batch, vocab_size]. - last_hidden: - [batch, model_dims]. - state1: - The updated prefix states including step t. - """ - - prefix_len, _ = py_utils.GetShape(state0['layer_0'].key, 2) - # [1, model_dim] - posit_embs = self.position_emb.FProp(theta.position_emb, - prefix_len + 1)[-1:, :] - # [batch, model_dim] - input_embs = inputs + posit_embs - input_embs = self.input_dropout.FProp(theta.input_dropout, input_embs) - - # Make a copy of the input. - state1 = state0.Pack(state0.Flatten()) - - layer_in = input_embs - for i, (layer, layer_theta) in enumerate(zip(self.trans, theta.trans)): - layer_prefix_states = state0['layer_%i' % i] - # [batch, model_dim] - layer_out, _, updated_prefix_states = layer.ExtendStep( - layer_theta, layer_in, layer_prefix_states) - state1['layer_%i' % i] = updated_prefix_states - layer_in = layer_out - - # [batch, vocab_size] - logits = self.softmax.Logits(theta=theta.softmax, inputs=layer_out) - - output = py_utils.NestedMap(logits=logits, last_hidden=layer_out) - return output, state1 - - def FProp(self, theta, inputs, paddings, state0=None, labels=None): - """Computes xent loss given the language model input activations. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - inputs: Input activation. A tensor of shape [time, batch, model_dim]. - paddings: A 0/1 tensor of shape [time, batch]. - state0: Not used for Transformer. - labels: If not None, a `.NestedMap` containing the following fields: - - - class_weights, a tensor with shape [time, batch] containing the - weights for each target word. - - class_ids, a tensor with shape [time, batch] of int32 dtype containing - the target class labels. - - class_probabilities, a tensor with shape [time, batch, vocab_size] of - float values indicating class-membership probabilities. - - Returns: - If `labels` is not None, returns (xent_output, None), where - `xent_output` is a `.NestedMap` as defined by `SoftmaxLayer`'s return - value. Otherwise, `xent_output` only contains the softmax logits. - """ - p = self.params - inputs = py_utils.HasRank(inputs, 3) - seqlen, batch, _ = tf.unstack(tf.shape(inputs), num=3) - inputs = py_utils.HasShape(inputs, [seqlen, batch, p.model_dim]) - paddings = py_utils.HasShape(paddings, [seqlen, batch]) - - # [time, 1, model_dim] - posit_embs = tf.expand_dims( - self.position_emb.FProp(theta.position_emb, seqlen), 1) - # [time, batch, model_dim] - input_embs = inputs + posit_embs - input_embs = self.input_dropout.FProp(theta.input_dropout, input_embs) - - layer_in = input_embs - for layer, layer_theta in zip(self.trans, theta.trans): - # [time, batch, model_dim] - layer_out, _ = layer.FProp(layer_theta, layer_in, paddings) - layer_in = layer_out - - if labels is None: - # We can only compute the logits here. - logits = self.softmax.Logits( - theta=theta.softmax, - inputs=tf.reshape(layer_out, [seqlen * batch, -1])) - xent_output = py_utils.NestedMap( - logits=tf.reshape(logits, [seqlen, batch, -1])) - elif 'class_ids' in labels: - xent_output = self.softmax.FProp( - theta=theta.softmax, - inputs=layer_out, - class_weights=labels.class_weights, - class_ids=labels.class_ids) - else: - assert 'class_probabilities' in labels - xent_output = self.softmax.FProp( - theta=theta.softmax, - inputs=layer_out, - class_weights=labels.class_weights, - class_probabilities=labels.class_probabilities) - xent_output.last_hidden = layer_out - return xent_output, None - - -class TransformerLm(TransformerLmNoEmbedding): - """Stacked RNN based language model layer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('emb', layers.EmbeddingLayer.Params(), - 'The embedding layer params.') - p.emb.max_num_shards = 1 - return p - - @classmethod - def CommonParams(cls, - model_dim, - hidden_dim, - num_heads, - num_layers, - learning_rate, - warmup_steps, - vocab_size, - input_dropout_prob=0.0, - residual_dropout_prob=0.1, - atten_dropout_prob=0.0, - relu_dropout_prob=0.0, - softmax_max_alloc=None): - """Common setup for Transformer language models. - - Args: - model_dim: model dimension. - hidden_dim: hidden dimension of feed-forward inner layer. - num_heads: number of attention heads. - num_layers: number of layers in the transformer LM. - learning_rate: learning rate. - warmup_steps: warmup steps for TransformerSchedule. - vocab_size: vocab size. - input_dropout_prob: dropout prob to the sums of the token embeddings and - the position embeddings. - residual_dropout_prob: dropout prob to the output of each sub-layer before - it is added to the sub-layer input. - atten_dropout_prob: dropout prob to the attention weights in each - Transformer attention sub-layer. - relu_dropout_prob: dropout prob to the inner layer output (ReLU - activation) in each Transformer feed-forward sub-layer. - softmax_max_alloc: If set to a positive integer the soft-max - computation is chunked into allocations of at most - softmax_max_alloc; when left to its default value of None no - chunking is done. - - Returns: - A Params object containing the parameters that set up a Transformer LM. - """ - p = cls.Params() - p.name = 'transformerlm' - - p.model_dim = model_dim - p.vocab_size = vocab_size - p.num_trans_layers = num_layers - p.input_dropout_prob = input_dropout_prob - p.residual_dropout_prob = residual_dropout_prob - p.atten_dropout_prob = atten_dropout_prob - p.relu_dropout_prob = relu_dropout_prob - - default_params_init = py_utils.WeightInit.Xavier(1.0) - emb_params_init = py_utils.WeightInit.Gaussian(1.0 / math.sqrt(p.model_dim)) - p.emb.Set( - vocab_size=vocab_size, - embedding_dim=p.model_dim, - max_num_shards=16, - params_init=emb_params_init, - scale_sqrt_depth=True) - - p.position_emb.Set(embedding_dim=p.model_dim, trainable_scaling=False) - - p.trans_tpl.has_aux_atten = False - p.trans_tpl.mask_self_atten = True - - p.trans_tpl.tr_atten_tpl.Set( - num_attention_heads=num_heads, params_init=default_params_init) - - p.trans_tpl.tr_atten_tpl.atten_tpl.Set( - enable_ctx_pre_proj=True, enable_ctx_post_proj=True) - - p.trans_tpl.tr_fflayer_tpl.Set( - hidden_dim=hidden_dim, params_init=default_params_init) - - p.softmax.Set( - num_classes=vocab_size, num_shards=16, params_init=emb_params_init) - - if softmax_max_alloc: - # If the vocab is very large, computes the softmax chunk-by-chunk. - p.softmax.chunk_size = max(1, int(softmax_max_alloc / vocab_size)) - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - assert p.emb.vocab_size == p.vocab_size, ('{} vs. {}'.format( - p.emb.vocab_size, p.vocab_size)) - assert p.emb.embedding_dim == p.position_emb.embedding_dim, ( - '{} vs. {}'.format(p.emb.embedding_dim, p.position_emb.embedding_dim)) - assert p.emb.embedding_dim == p.model_dim, ('{} vs. {}'.format( - p.emb.embedding_dim, p.model_dim)) - - self.CreateChild('emb', p.emb) - - def FProp(self, theta, inputs, paddings, state0=None, labels=None): - """Computes xent loss given the language model input activations. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - inputs: Input ids. An int32 tensor of shape [time, batch]. - paddings: A 0/1 tensor of shape [time, batch]. - state0: Not used for Transformer. - labels: If not None, a `.NestedMap` containing the following fields: - - - class_weights, a tensor with shape [time, batch] containing the - weights for each target word. - - class_ids, a tensor with shape [time, batch] of int32 dtype containing - the target class labels. - - class_probabilities, a tensor with shape [time, batch, vocab_size] of - float values indicating class-membership probabilities. - - Returns: - If `labels` is not None, returns (xent_output, state1), where - `xent_output` is a `.NestedMap` as defined by `SoftmaxLayer`'s return - value and `state1` is the next recurrent state. Otherwise, - `xent_output` only contains the softmax logits. - """ - ids = py_utils.HasRank(inputs, 2) - paddings = py_utils.HasShape(paddings, tf.shape(ids)) - activation = self.emb.EmbLookup(theta.emb, ids) - return super().FProp(theta, activation, paddings, labels=labels) - - -class GPipeTransformerLm(BaseLanguageModel): - """GPipe Transformer based language model layer.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('stack', layers_with_gpipe.GPipeTransformerStack.Params(), - 'GPipeTransformerStack Layer params.') - return p - - @classmethod - def CommonParams(cls, - vocab_size, - model_dim, - attention_hidden_dim=None, - hidden_dim=1024, - num_heads=8, - num_layers=6, - splits=1, - num_micro_batches=1, - micro_batch_size=None, - num_shards=16, - input_dropout_prob=0.0, - residual_dropout_prob=0.1, - atten_dropout_prob=0.0, - relu_dropout_prob=0.0, - softmax_max_alloc=None): - """Common setup for Transformer language models. - - Args: - vocab_size: vocab size. - model_dim: model dimension. - attention_hidden_dim: hidden dim for the attention layer. - hidden_dim: hidden dimension of feed-forward inner layer. - num_heads: number of attention heads. - num_layers: number of layers in the transformer LM. - splits: list or number of partitions for GPipe. - num_micro_batches: number of micro batches for GPipe. - micro_batch_size: size of a micro batch. - num_shards: num_shards for softmax. Assert vocab_size % num_shards == 0 - input_dropout_prob: dropout prob to the sums of the token embeddings and - the position embeddings. - residual_dropout_prob: dropout prob to the output of each sub-layer before - it is added to the sub-layer input. - atten_dropout_prob: dropout prob to the attention weights in each - Transformer attention sub-layer. - relu_dropout_prob: dropout prob to the inner layer output (ReLU - activation) in each Transformer feed-forward sub-layer. - softmax_max_alloc: If set to a positive integer the soft-max computation - is chunked into allocations of at most softmax_max_alloc; when left to - its default value of None no chunking is done. - - Returns: - A Params object containing the parameters that set up a Transformer LM. - """ - p = cls.Params() - p.name = 'transformerlm' - p.vocab_size = vocab_size - p.stack.splits = splits - p.stack.model_dim = model_dim - p.stack.num_micro_batches = num_micro_batches - p.stack.micro_batch_size = micro_batch_size - p.stack.num_encoder_layers = num_layers - p.stack.batch_dim = 0 - emb_params_init = py_utils.WeightInit.Gaussian(1.0 / math.sqrt(model_dim)) - p.stack.emb_tpl.ret_task_ids = True - p.stack.emb_tpl.token_emb.Set( - use_matmul=False, - use_3d_weight_tensor=False, - vocab_size=vocab_size, - embedding_dim=model_dim, - params_init=emb_params_init) - p.stack.emb_tpl.position_emb.Set( - embedding_dim=model_dim, trainable_scaling=False) - p.stack.emb_tpl.input_dropout_prob = input_dropout_prob - - trans_tpl = batch_major_attention.GPipeTransformerLayer.Params() - trans_tpl.has_aux_atten = False - trans_tpl.mask_self_atten = True - trans_tpl.input_dim = model_dim - trans_tpl.output_dim = model_dim - trans_tpl.tr_atten_tpl.input_dim = model_dim - trans_tpl.tr_atten_tpl.hidden_dim = attention_hidden_dim or model_dim - trans_tpl.tr_atten_tpl.is_masked = True - trans_tpl.tr_atten_tpl.num_heads = num_heads - trans_tpl.tr_atten_tpl.atten_tpl.use_bias = False - trans_tpl.tr_fflayer_tpl.hidden_dim = hidden_dim - p.stack.encoder_tpl = trans_tpl - - p.stack.softmax_tpl.Set( - num_classes=vocab_size, input_dim=model_dim, num_shards=num_shards) - if softmax_max_alloc: - # If the vocab is very large, computes the softmax chunk-by-chunk. - p.stack.softmax_tpl.chunk_size = max(1, - int(softmax_max_alloc / vocab_size)) - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - p.stack.name = p.name - p.stack.state_dtype = p.dtype - if p.fprop_dtype: - p.stack.state_dtype = p.fprop_dtype - self.CreateChild('stack', p.stack) - - def zero_state(self, theta, batch_size): - return py_utils.NestedMap() - - def FProp(self, theta, inputs, paddings, state0=None, labels=None): - """Computes xent loss given the language model input activations. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - inputs: Input ids. An int32 tensor of shape [batch, time]. - paddings: A 0/1 tensor of shape [batch, time]. - state0: Not used for Transformer. - labels: If not None, a `.NestedMap` containing the following fields: - - class_weights, a tensor with shape [batch, time] containing the weights - for each target word. - class_ids, a tensor with shape [batch, time] of - int32 dtype containing the target class labels. - class_probabilities, a - tensor with shape [batch, time, vocab_size] of float values indicating - class-membership probabilities. - - Returns: - If `labels` is not None, returns (xent_output, state1), where - `xent_output` is a `.NestedMap` as defined by `SoftmaxLayer`'s return - value and `state1` is the next recurrent state. Otherwise, - `xent_output` only contains the softmax logits. - """ - ids = py_utils.HasRank(inputs, 2) - paddings = py_utils.HasShape(paddings, tf.shape(ids)) - per_example_xent, logits = self.stack.FProp(theta.stack, ids, paddings, - None, None, None, None, - labels.class_ids, - labels.class_weights) - per_example_argmax = py_utils.ArgMax(logits) - total_xent = tf.reduce_sum(per_example_xent * labels.class_weights) - total_weights = tf.reduce_sum(labels.class_weights) - xent_output = py_utils.NestedMap( - total_weight=total_weights, - per_example_xent=per_example_xent, - logits=logits, - per_example_argmax=per_example_argmax, - avg_xent=total_xent / total_weights, - total_xent=total_xent) - return xent_output, {} diff --git a/lingvo/tasks/lm/layers_test.py b/lingvo/tasks/lm/layers_test.py deleted file mode 100644 index 64f784e5..00000000 --- a/lingvo/tasks/lm/layers_test.py +++ /dev/null @@ -1,972 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for layers.""" - -import lingvo.compat as tf -from lingvo.core import batch_major_attention -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.tasks.lm import layers as lm_layers -import numpy as np - -FLAGS = tf.flags.FLAGS - - -class RnnLmNoEmbeddingTest(test_utils.TestCase): - - def _testParams(self, dims, vocab): - p = lm_layers.RnnLmNoEmbedding.Params() - p.name = 'rnnlm' - p.vocab_size = vocab - p.rnns.cell_tpl.num_output_nodes = dims - p.rnns.cell_tpl.num_input_nodes = dims - p.softmax.input_dim = dims - p.softmax.num_classes = vocab - return p - - def testBasic(self): - time, batch, dims, vocab = 5, 3, 6, 8 - p = self._testParams(dims, vocab) - - with self.session(use_gpu=True): - lm = p.Instantiate() - np.random.seed(12345) - inputs = np.random.normal(size=[time, batch, dims]) - inputs = tf.constant(inputs, tf.float32) - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float32) - targets = tf.constant( - np.random.randint(vocab, size=(time, batch)), tf.int32) - self.evaluate(tf.global_variables_initializer()) - xent_output, state1 = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - state0=lm.zero_state(lm.theta, batch), - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - - xent_output_val, state1_val = self.evaluate([xent_output, state1]) - - print('xent_output_val', xent_output_val) - print('state1', state1_val) - test_utils.CompareToGoldenSingleFloat(self, 2.1042602, xent_output_val.avg_xent) # pyformat: disable pylint: disable=line-too-long - self.assertAllEqual(xent_output_val.per_example_argmax, - np.argmax(xent_output_val.logits, axis=-1)) - - def testProjection(self): - time, batch, dims, proj, vocab = 5, 3, 6, 4, 8 - p = self._testParams(dims, vocab) - p.rnns.cell_tpl.num_output_nodes = proj - p.rnns.cell_tpl.num_hidden_nodes = dims - p.softmax.input_dim = proj - - with self.session(use_gpu=True): - lm = p.Instantiate() - np.random.seed(12345) - inputs = np.random.normal(size=[time, batch, dims]) - inputs = tf.constant(inputs, tf.float32) - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float32) - targets = tf.constant( - np.random.randint(vocab, size=(time, batch)), tf.int32) - self.evaluate(tf.global_variables_initializer()) - xent_output, state1 = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - state0=lm.zero_state(lm.theta, batch), - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - - xent_output_val, state1_val = self.evaluate([xent_output, state1]) - - print('xent_output_val', xent_output_val) - print('state1', state1_val) - test_utils.CompareToGoldenSingleFloat(self, 2.1322777, xent_output_val.avg_xent) # pyformat: disable pylint: disable=line-too-long - self.assertAllEqual(xent_output_val.per_example_argmax, - np.argmax(xent_output_val.logits, axis=-1)) - - def testBasicGrad(self): - time, batch, dims, vocab = 5, 3, 6, 8 - p = self._testParams(dims, vocab) - p.dtype = tf.float64 - - with self.session(use_gpu=False, graph=tf.Graph()) as sess: - lm = p.Instantiate() - np.random.seed(12345) - inputs = np.random.normal(size=[time, batch, dims]) - inputs = tf.constant(inputs, tf.float64) - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float64) - targets = tf.constant( - np.random.randint(vocab, size=(time, batch)), tf.int32) - - xent_output, _ = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - state0=lm.zero_state(lm.theta, batch), - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - - lm_vars = lm.vars.Flatten() - # Now add the backward graph. - grads = tf.gradients(xent_output.avg_xent, lm_vars) - - self.evaluate(tf.global_variables_initializer()) - self.assertEqual(len(lm_vars), len(grads)) - for x, grad_x in zip(lm_vars, grads): - grad_symbolic = self.evaluate(grad_x) - grad_numeric = test_utils.ComputeNumericGradient( - sess, xent_output.avg_xent, x, delta=1e-6) - self.assertAllClose(grad_symbolic, grad_numeric, atol=0.005) - - def testDirectFeatures(self,): - time, batch, dims, vocab = 5, 3, 6, 8 - p = self._testParams(dims, vocab) - direct_features_dim = 4 - p.direct_features_dim = direct_features_dim - p.softmax.input_dim = dims + direct_features_dim - - with self.session(use_gpu=True): - lm = p.Instantiate() - np.random.seed(12345) - inputs = np.random.normal(size=[time, batch, dims]) - inputs = tf.constant(inputs, tf.float32) - - direct_features = np.random.normal( - size=[time, batch, direct_features_dim]) - direct_features = tf.constant(direct_features, tf.float32) - - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float32) - targets = tf.constant( - np.random.randint(vocab, size=(time, batch)), tf.int32) - self.evaluate(tf.global_variables_initializer()) - xent_output, state1 = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - state0=lm.zero_state(lm.theta, batch), - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets), - direct_features=direct_features) - - xent_output_val, state1_val = self.evaluate([xent_output, state1]) - - print('xent_output_val', xent_output_val) - print('state1', state1_val) - test_utils.CompareToGoldenSingleFloat(self, 2.3522419929504395, xent_output_val.avg_xent) # pyformat: disable pylint: disable=line-too-long - self.assertAllEqual(xent_output_val.per_example_argmax, - np.argmax(xent_output_val.logits, axis=-1)) - - def testCombineStates(self): - time, batch, dims, vocab = 5, 3, 6, 8 - p = self._testParams(dims, vocab) - - with self.session(use_gpu=True): - lm = p.Instantiate() - np.random.seed(12345) - inputs = np.random.normal(size=[time, batch, dims]) - inputs = tf.constant(inputs, tf.float32) - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float32) - targets = tf.constant( - np.random.randint(vocab, size=(time, batch)), tf.int32) - self.evaluate(tf.global_variables_initializer()) - - state0 = lm.zero_state(lm.theta, batch) - _, state1 = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - state0=state0, - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - switch = tf.constant([True, True, False], dtype=tf.bool) - combined_state = lm.CombineStates(state0, state1, switch) - state0_val, state1_val, combined_state_val = self.evaluate( - [state0, state1, combined_state]) - - print('state1_val', state1_val) - print('combined_state_val', combined_state_val) - combined_m_state = combined_state_val.rnn[0].m - combined_c_state = combined_state_val.rnn[0].c - self.assertAllEqual(combined_m_state[0], state0_val.rnn[0].m[0]) - self.assertAllEqual(combined_c_state[0], state0_val.rnn[0].c[0]) - self.assertAllEqual(combined_m_state[1], state0_val.rnn[0].m[1]) - self.assertAllEqual(combined_c_state[1], state0_val.rnn[0].c[1]) - self.assertAllEqual(combined_m_state[2], state1_val.rnn[0].m[2]) - self.assertAllEqual(combined_c_state[2], state1_val.rnn[0].c[2]) - - -class RnnLmTest(test_utils.TestCase): - - def testBasic(self): - time, batch, dims, vocab = 5, 3, 6, 8 - - p = lm_layers.RnnLm.Params() - p.name = 'rnnlm' - p.vocab_size = vocab - p.emb.vocab_size = vocab - p.emb.embedding_dim = dims - p.rnns.cell_tpl.num_output_nodes = dims - p.rnns.cell_tpl.num_input_nodes = dims - p.softmax.input_dim = dims - p.softmax.num_classes = vocab - - with self.session(use_gpu=True): - lm = p.Instantiate() - np.random.seed(12345) - inputs = np.random.randint(vocab, size=[time, batch]) - targets = np.zeros([time, batch]) - targets[:-1] = inputs[1:] - inputs = tf.constant(inputs, tf.int32) - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float32) - targets = tf.constant(targets, tf.int32) - self.evaluate(tf.global_variables_initializer()) - xent_output, state1 = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - state0=lm.zero_state(lm.theta, batch), - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - - xent_output_val, state1_val = self.evaluate([xent_output, state1]) - - print('xent_output_val', xent_output_val) - print('state1', state1_val) - test_utils.CompareToGoldenSingleFloat(self, 2.0853612, xent_output_val.avg_xent) # pyformat: disable pylint: disable=line-too-long - self.assertAllEqual(xent_output_val.per_example_argmax, - np.argmax(xent_output_val.logits, axis=-1)) - - def testDropout(self): - seed = 12345 - tf.random.set_seed(seed) - np.random.seed(seed) - - time, batch, dims, vocab = 5, 3, 6, 8 - - p = lm_layers.RnnLm.Params() - p.name = 'rnnlm' - p.vocab_size = vocab - p.emb.vocab_size = vocab - p.emb.embedding_dim = dims - p.embedding_dropout_keep_prob = 0.8 - p.embedding_dropout_seed = seed - p.rnns.cell_tpl.num_output_nodes = dims - p.rnns.cell_tpl.num_input_nodes = dims - p.softmax.input_dim = dims - p.softmax.num_classes = vocab - - with self.session(use_gpu=True): - lm = p.Instantiate() - inputs = np.random.randint(vocab, size=[time, batch]) - targets = np.zeros([time, batch]) - targets[:-1] = inputs[1:] - inputs = tf.constant(inputs, tf.int32) - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float32) - targets = tf.constant(targets, tf.int32) - self.evaluate(tf.global_variables_initializer()) - xent_output, state1 = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - state0=lm.zero_state(lm.theta, batch), - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - - xent_output_val, state1_val = self.evaluate([xent_output, state1]) - - print('xent_output_val', xent_output_val) - print('state1', state1_val) - test_utils.CompareToGoldenSingleFloat(self, 2.084798, xent_output_val.avg_xent) # pyformat: disable pylint: disable=line-too-long - self.assertAllEqual(xent_output_val.per_example_argmax, - np.argmax(xent_output_val.logits, axis=-1)) - - def testBasicGrad(self): - time, batch, dims, vocab = 5, 3, 6, 8 - - p = lm_layers.RnnLm.Params() - p.dtype = tf.float64 - p.name = 'rnnlm' - p.vocab_size = vocab - p.emb.vocab_size = vocab - p.emb.embedding_dim = dims - p.rnns.cell_tpl.num_output_nodes = dims - p.rnns.cell_tpl.num_input_nodes = dims - p.softmax.input_dim = dims - p.softmax.num_classes = vocab - - with self.session(use_gpu=False, graph=tf.Graph()) as sess: - lm = p.Instantiate() - np.random.seed(12345) - inputs = np.random.randint(vocab, size=[time, batch]) - targets = np.zeros([time, batch]) - targets[:-1] = inputs[1:] - inputs = tf.constant(inputs, tf.int32) - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float64) - targets = tf.constant(targets, tf.int32) - self.evaluate(tf.global_variables_initializer()) - xent_output, _ = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - state0=lm.zero_state(lm.theta, batch), - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - - lm_vars = lm.vars.Flatten() - # Now add the backward graph. - grads = tf.gradients(xent_output.avg_xent, lm_vars) - - for i, x in enumerate(grads): - if isinstance(x, tf.IndexedSlices): - grads[i] = tf.math.unsorted_segment_sum(x.values, x.indices, - x.dense_shape[0]) - - self.evaluate(tf.global_variables_initializer()) - self.assertEqual(len(lm_vars), len(grads)) - for x, grad_x in zip(lm_vars, grads): - grad_symbolic = self.evaluate(grad_x) - grad_numeric = test_utils.ComputeNumericGradient( - sess, xent_output.avg_xent, x, delta=1e-6) - self.assertAllClose(grad_symbolic, grad_numeric, atol=0.005) - - -class ConditionalRnnLmTest(test_utils.TestCase): - - def testBasic(self): - time, batch, dims, vocab, condition_dim = 5, 3, 6, 8, 7 - - p = lm_layers.ConditionalRnnLm.Params() - p.name = 'conditionalrnnlm' - p.vocab_size = vocab - p.emb.vocab_size = vocab - p.emb.embedding_dim = dims - model_dim = dims + condition_dim - p.rnns.cell_tpl.num_output_nodes = model_dim - p.rnns.cell_tpl.num_input_nodes = model_dim - p.softmax.input_dim = model_dim - p.softmax.num_classes = vocab - p.condition_dim = condition_dim - - with self.session(use_gpu=True): - lm = p.Instantiate() - np.random.seed(12345) - inputs = np.random.randint(vocab, size=[time, batch]) - targets = np.zeros([time, batch]) - targets[:-1] = inputs[1:] - inputs = tf.constant(inputs, tf.int32) - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float32) - targets = tf.constant(targets, tf.int32) - condition = tf.constant(np.ones([batch, condition_dim]), tf.float64) - self.evaluate(tf.global_variables_initializer()) - xent_output, state1 = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - state0=lm.zero_state(lm.theta, batch), - condition=condition, - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - - xent_output_val, state1_val = self.evaluate([xent_output, state1]) - - print('xent_output_val', xent_output_val) - print('state1', state1_val) - test_utils.CompareToGoldenSingleFloat(self, 2.10713076, xent_output_val.avg_xent) # pyformat: disable pylint: disable=line-too-long - self.assertAllEqual(xent_output_val.per_example_argmax, - np.argmax(xent_output_val.logits, axis=-1)) - - def testDropout(self): - seed = 12345 - tf.random.set_seed(seed) - np.random.seed(seed) - - time, batch, dims, vocab, condition_dim = 5, 3, 6, 8, 7 - - p = lm_layers.ConditionalRnnLm.Params() - p.name = 'conditionalrnnlm' - p.vocab_size = vocab - p.emb.vocab_size = vocab - p.emb.embedding_dim = dims - p.embedding_dropout_keep_prob = 0.8 - p.embedding_dropout_seed = seed - model_dim = dims + condition_dim - p.rnns.cell_tpl.num_output_nodes = model_dim - p.rnns.cell_tpl.num_input_nodes = model_dim - p.softmax.input_dim = model_dim - p.softmax.num_classes = vocab - p.condition_dim = condition_dim - - with self.session(use_gpu=True): - lm = p.Instantiate() - inputs = np.random.randint(vocab, size=[time, batch]) - targets = np.zeros([time, batch]) - targets[:-1] = inputs[1:] - inputs = tf.constant(inputs, tf.int32) - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float32) - targets = tf.constant(targets, tf.int32) - condition = tf.constant(np.ones([batch, condition_dim]), tf.float64) - self.evaluate(tf.global_variables_initializer()) - xent_output, state1 = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - state0=lm.zero_state(lm.theta, batch), - condition=condition, - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - - xent_output_val, state1_val = self.evaluate([xent_output, state1]) - - print('xent_output_val', xent_output_val) - print('state1', state1_val) - test_utils.CompareToGoldenSingleFloat(self, 2.17278885, xent_output_val.avg_xent) # pyformat: disable pylint: disable=line-too-long - self.assertAllEqual(xent_output_val.per_example_argmax, - np.argmax(xent_output_val.logits, axis=-1)) - - def testBasicGrad(self): - time, batch, dims, vocab, condition_dim = 5, 3, 6, 8, 7 - - p = lm_layers.ConditionalRnnLm.Params() - p.name = 'conditionalrnnlm' - p.dtype = tf.float64 - p.vocab_size = vocab - p.emb.vocab_size = vocab - p.emb.embedding_dim = dims - model_dim = dims + condition_dim - p.rnns.cell_tpl.num_output_nodes = model_dim - p.rnns.cell_tpl.num_input_nodes = model_dim - p.softmax.input_dim = model_dim - p.softmax.num_classes = vocab - p.condition_dim = condition_dim - - with self.session(use_gpu=False, graph=tf.Graph()) as sess: - lm = p.Instantiate() - np.random.seed(12345) - inputs = np.random.randint(vocab, size=[time, batch]) - targets = np.zeros([time, batch]) - targets[:-1] = inputs[1:] - inputs = tf.constant(inputs, tf.int32) - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float64) - targets = tf.constant(targets, tf.int32) - condition = tf.constant(np.ones([batch, condition_dim]), tf.float64) - self.evaluate(tf.global_variables_initializer()) - xent_output, _ = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - state0=lm.zero_state(lm.theta, batch), - condition=condition, - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - - lm_vars = lm.vars.Flatten() - # Now add the backward graph. - grads = tf.gradients(xent_output.avg_xent, lm_vars) - - for i, x in enumerate(grads): - if isinstance(x, tf.IndexedSlices): - grads[i] = tf.math.unsorted_segment_sum(x.values, x.indices, - x.dense_shape[0]) - - self.evaluate(tf.global_variables_initializer()) - self.assertEqual(len(lm_vars), len(grads)) - for x, grad_x in zip(lm_vars, grads): - grad_symbolic = self.evaluate(grad_x) - grad_numeric = test_utils.ComputeNumericGradient( - sess, xent_output.avg_xent, x, delta=1e-6) - self.assertAllClose(grad_symbolic, grad_numeric, atol=0.005) - - -class MoeLmTest(test_utils.TestCase): - - def _MoeLmParams(self, vocab, shared_emb, add_postgating_rnn=True): - p = lm_layers.MoeLm.Params() - p.name = 'moelm' - p.vocab_size = vocab - p.emb.vocab_size = vocab - p.emb.embedding_dim = 17 - p.shared_emb = shared_emb - p.add_postgating_rnn = add_postgating_rnn - p.rnns.cell_tpl.num_input_nodes = 17 - p.rnns.cell_tpl.num_output_nodes = 23 - p.number_of_experts = 4 - p.merge.vocab_size = vocab - p.merge.rnns.cell_tpl.num_input_nodes = 23 - p.merge.rnns.cell_tpl.num_output_nodes = 32 - p.merge.softmax.input_dim = 32 - p.merge.softmax.num_classes = vocab - return p - - def _GetData(self, vocab, time, batch): - inputs = np.random.randint(vocab, size=[time, batch]) - targets = np.zeros([time, batch]) - targets[:-1] = inputs[1:] - inputs = tf.constant(inputs, tf.int32) - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float32) - targets = tf.constant(targets, tf.int32) - labels = py_utils.NestedMap(class_weights=1 - paddings, class_ids=targets) - return inputs, paddings, labels - - def testFPropNoSharedEmb(self): - vocab, time, batch = 7, 13, 3 - p = self._MoeLmParams(vocab, False) - - with self.session(graph=tf.Graph()): - np.random.seed(54321) - tf.random.set_seed(123456) - lm = p.Instantiate() - inputs, paddings, labels = self._GetData(vocab, time, batch) - self.evaluate(tf.global_variables_initializer()) - xent_output, state1 = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - state0=lm.zero_state(lm.theta, batch), - labels=labels) - - xent_output_val, state1_val = self.evaluate([xent_output, state1]) - - print('xent_output_val', xent_output_val) - print('state1', state1_val) - test_utils.CompareToGoldenSingleFloat(self, 1.9460623, xent_output_val.avg_xent) # pyformat: disable pylint: disable=line-too-long - self.assertAllEqual(xent_output_val.per_example_argmax, - np.argmax(xent_output_val.logits, axis=-1)) - - def testFPropSharedEmb(self): - vocab, time, batch = 7, 13, 3 - p = self._MoeLmParams(vocab, False) - - with self.session(graph=tf.Graph()): - np.random.seed(54321) - tf.random.set_seed(123456) - lm = p.Instantiate() - inputs, paddings, labels = self._GetData(vocab, time, batch) - self.evaluate(tf.global_variables_initializer()) - xent_output, state1 = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - state0=lm.zero_state(lm.theta, batch), - labels=labels) - - xent_output_val, state1_val = self.evaluate([xent_output, state1]) - - print('xent_output_val', xent_output_val) - print('state1', state1_val) - test_utils.CompareToGoldenSingleFloat(self, 1.9460623, xent_output_val.avg_xent) # pyformat: disable pylint: disable=line-too-long - self.assertAllEqual(xent_output_val.per_example_argmax, - np.argmax(xent_output_val.logits, axis=-1)) - - def testFPropNoPostGatingRNN(self): - vocab, time, batch = 7, 13, 3 - p = self._MoeLmParams(vocab, False, False) - - with self.session(graph=tf.Graph()): - np.random.seed(54321) - tf.random.set_seed(123456) - lm = p.Instantiate() - inputs, paddings, labels = self._GetData(vocab, time, batch) - self.evaluate(tf.global_variables_initializer()) - xent_output, state1 = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - state0=lm.zero_state(lm.theta, batch), - labels=labels) - - xent_output_val, state1_val = self.evaluate([xent_output, state1]) - - print('xent_output_val', xent_output_val) - print('state1', state1_val) - test_utils.CompareToGoldenSingleFloat(self, 1.9443978, xent_output_val.avg_xent) # pyformat: disable pylint: disable=line-too-long - self.assertAllEqual(xent_output_val.per_example_argmax, - np.argmax(xent_output_val.logits, axis=-1)) - - def testBProp(self): - vocab, time, batch = 7, 4, 3 - p = self._MoeLmParams(vocab, True) - p.dtype = tf.float64 - - with self.session(graph=tf.Graph()) as sess: - np.random.seed(54321) - tf.random.set_seed(123456) - lm = p.Instantiate() - inputs, paddings, labels = self._GetData(vocab, time, batch) - self.evaluate(tf.global_variables_initializer()) - xent_output, _ = lm.FPropDefaultTheta( - inputs=inputs, - paddings=tf.cast(paddings, p.dtype), - state0=lm.zero_state(lm.theta, batch), - labels=labels) - - lm_vars = lm.vars.Flatten() - # Now add the backward graph. - grads = tf.gradients(xent_output.avg_xent, lm_vars) - - for i, x in enumerate(grads): - if isinstance(x, tf.IndexedSlices): - grads[i] = tf.math.unsorted_segment_sum(x.values, x.indices, - x.dense_shape[0]) - - self.evaluate(tf.global_variables_initializer()) - self.assertEqual(len(lm_vars), len(grads)) - step = 11 # Speed up the test. - for x, grad_x in zip(lm_vars, grads): - grad_symbolic = self.evaluate(grad_x) - grad_numeric = test_utils.ComputeNumericGradient( - sess, xent_output.avg_xent, x, step=step, delta=1e-6) - self.assertAllClose( - grad_symbolic.reshape([-1])[::step], - grad_numeric.reshape([-1])[::step]) - - -class TransformerLmNoEmbeddingTest(test_utils.TestCase): - - def _testParams(self, dtype=tf.float32): - model_dim, hidden_dim, vocab_size = 4, 6, 8 - p = lm_layers.TransformerLmNoEmbedding.Params() - p.name = 'xformerlm' - p.random_seed = 93820986 - p.dtype = dtype - p.vocab_size = vocab_size - p.model_dim = model_dim - p.num_trans_layers = 3 - p.position_emb.embedding_dim = model_dim - p.trans_tpl.source_dim = model_dim - p.trans_tpl.tr_atten_tpl.num_attention_heads = 2 - p.trans_tpl.tr_fflayer_tpl.hidden_dim = hidden_dim - p.softmax.input_dim = model_dim - p.softmax.num_classes = vocab_size - return p - - def _testInputs(self, dtype=tf.float32, last_padding=1.0): - time, batch, model_dim, vocab_size = 5, 3, 4, 8 - np.random.seed(12345) - inputs = np.random.normal(size=[time, batch, model_dim]) - inputs = tf.constant(inputs, dtype) - paddings = np.zeros([time, batch]) - paddings[-1] = last_padding - paddings = tf.constant(paddings, dtype) - targets = tf.constant( - np.random.randint(vocab_size, size=(time, batch)), tf.int32) - return inputs, paddings, targets - - def testBasic(self): - p = self._testParams(dtype=tf.float32) - with self.session(use_gpu=True): - lm = p.Instantiate() - inputs, paddings, targets = self._testInputs(dtype=tf.float32) - self.evaluate(tf.global_variables_initializer()) - xent_output, _ = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - xent_output_val = self.evaluate(xent_output) - print('xformer xent_output_val.avg_xent', xent_output_val.avg_xent) - test_utils.CompareToGoldenSingleFloat(self, 3.018613, xent_output_val.avg_xent) # pyformat: disable pylint: disable=line-too-long - self.assertAllEqual(xent_output_val.per_example_argmax, - np.argmax(xent_output_val.logits, axis=-1)) - - def testBasicGrad(self): - p = self._testParams(dtype=tf.float64) - with self.session(use_gpu=False, graph=tf.Graph()) as sess: - lm = p.Instantiate() - inputs, paddings, targets = self._testInputs(dtype=tf.float64) - xent_output, _ = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - - lm_vars = lm.vars.Flatten() - # Now add the backward graph. - grads = tf.gradients(xent_output.avg_xent, lm_vars) - - self.evaluate(tf.global_variables_initializer()) - self.assertEqual(len(lm_vars), len(grads)) - for x, grad_x in zip(lm_vars, grads): - grad_symbolic = self.evaluate(grad_x) - grad_numeric = test_utils.ComputeNumericGradient( - sess, xent_output.avg_xent, x, delta=1e-6) - self.assertAllClose(grad_symbolic, grad_numeric, atol=0.005) - - def testStep(self): - p = self._testParams(dtype=tf.float32) - with self.session(use_gpu=True): - lm = p.Instantiate() - inputs, paddings, _ = self._testInputs(dtype=tf.float32, last_padding=0.0) - - self.evaluate(tf.global_variables_initializer()) - xent_output, _ = lm.FPropDefaultTheta(inputs=inputs, paddings=paddings) - logits1 = xent_output.logits - - time, batch = 5, 3 - prefix_states = lm.zero_state(lm.theta, batch) - logits2 = [] - for i in range(time): - l_i_out, prefix_states = lm.Step(lm.theta, inputs[i, :, :], - paddings[i, :], prefix_states) - logits2.append(l_i_out.logits) - logits2 = tf.stack(logits2) - - self.evaluate(tf.global_variables_initializer()) - logits1_v, logits2_v = self.evaluate([logits1, logits2]) - print('xformer logits1_v', logits1_v) - print('xformer logits2_v', logits2_v) - self.assertAllClose(logits1_v, logits2_v) - - -class TransformerLmTest(test_utils.TestCase): - - def testBasic(self): - time, batch, dims, hidden_dim, vocab = 5, 3, 6, 4, 8 - - p = lm_layers.TransformerLm.Params() - p.name = 'transformerlm' - p.vocab_size = vocab - p.emb.vocab_size = vocab - p.emb.embedding_dim = dims - p.model_dim = dims - p.num_trans_layers = 3 - p.position_emb.embedding_dim = dims - p.trans_tpl.source_dim = dims - p.trans_tpl.tr_atten_tpl.num_attention_heads = 2 - p.trans_tpl.tr_fflayer_tpl.hidden_dim = hidden_dim - p.softmax.input_dim = dims - p.softmax.num_classes = vocab - - with self.session(use_gpu=True): - lm = p.Instantiate() - np.random.seed(12345) - inputs = np.random.randint(vocab, size=[time, batch]) - targets = np.zeros([time, batch]) - targets[:-1] = inputs[1:] - inputs = tf.constant(inputs, tf.int32) - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float32) - targets = tf.constant(targets, tf.int32) - self.evaluate(tf.global_variables_initializer()) - xent_output, _ = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - - xent_output_val = self.evaluate(xent_output) - - print('xent_output_val', xent_output_val) - test_utils.CompareToGoldenSingleFloat(self, 3.0489848, xent_output_val.avg_xent) # pyformat: disable pylint: disable=line-too-long - self.assertAllEqual(xent_output_val.per_example_argmax, - np.argmax(xent_output_val.logits, axis=-1)) - - def testDropout(self): - seed = 12345 - tf.random.set_seed(seed) - np.random.seed(seed) - - time, batch, dims, hidden_dim, vocab = 5, 3, 6, 4, 8 - - p = lm_layers.TransformerLm.Params() - p.name = 'transformerlm' - p.vocab_size = vocab - p.emb.vocab_size = vocab - p.emb.embedding_dim = dims - p.model_dim = dims - p.num_trans_layers = 3 - p.trans_tpl.source_dim = dims - p.trans_tpl.tr_atten_tpl.num_attention_heads = 2 - p.trans_tpl.tr_fflayer_tpl.hidden_dim = hidden_dim - p.softmax.input_dim = dims - p.softmax.num_classes = vocab - - with self.session(use_gpu=True): - lm = p.Instantiate() - inputs = np.random.randint(vocab, size=[time, batch]) - targets = np.zeros([time, batch]) - targets[:-1] = inputs[1:] - inputs = tf.constant(inputs, tf.int32) - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float32) - targets = tf.constant(targets, tf.int32) - self.evaluate(tf.global_variables_initializer()) - xent_output, _ = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - - xent_output_val = self.evaluate(xent_output) - - print('xent_output_val', xent_output_val) - test_utils.CompareToGoldenSingleFloat(self, 3.038596, xent_output_val.avg_xent) # pyformat: disable pylint: disable=line-too-long - self.assertAllEqual(xent_output_val.per_example_argmax, - np.argmax(xent_output_val.logits, axis=-1)) - - def testBasicGrad(self): - time, batch, dims, hidden_dim, vocab = 5, 3, 6, 4, 8 - - p = lm_layers.TransformerLm.Params() - p.dtype = tf.float64 - p.name = 'transformerlm' - p.vocab_size = vocab - p.emb.vocab_size = vocab - p.emb.embedding_dim = dims - p.model_dim = dims - p.num_trans_layers = 1 - p.trans_tpl.source_dim = dims - p.trans_tpl.tr_atten_tpl.num_attention_heads = 2 - p.trans_tpl.tr_fflayer_tpl.hidden_dim = hidden_dim - p.softmax.input_dim = dims - p.softmax.num_classes = vocab - - with self.session(use_gpu=False, graph=tf.Graph()) as sess: - lm = p.Instantiate() - np.random.seed(12345) - inputs = np.random.randint(vocab, size=[time, batch]) - targets = np.zeros([time, batch]) - targets[:-1] = inputs[1:] - inputs = tf.constant(inputs, tf.int32) - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float64) - targets = tf.constant(targets, tf.int32) - xent_output, _ = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - - lm_vars = lm.vars.Flatten() - grads = tf.gradients(xent_output.avg_xent, lm_vars) - - for i, x in enumerate(grads): - if isinstance(x, tf.IndexedSlices): - grads[i] = tf.math.unsorted_segment_sum(x.values, x.indices, - x.dense_shape[0]) - - self.evaluate(tf.global_variables_initializer()) - self.assertEqual(len(lm_vars), len(grads)) - for x, grad_x in zip(lm_vars, grads): - grad_symbolic = self.evaluate(grad_x) - grad_numeric = test_utils.ComputeNumericGradient( - sess, xent_output.avg_xent, x, delta=1e-6) - self.assertAllClose(grad_symbolic, grad_numeric, atol=0.005) - - -class GPipeTransformerLmTest(test_utils.TestCase): - - def _testParams(self, batch, dims, hidden_dim, vocab): - p = lm_layers.GPipeTransformerLm.Params() - p.name = 'transformerlm' - p.vocab_size = vocab - p.stack.num_encoder_layers = 2 - p.stack.emb_tpl.ret_task_ids = True - p.stack.emb_tpl.token_emb.vocab_size = vocab - p.stack.emb_tpl.token_emb.embedding_dim = dims - p.stack.emb_tpl.position_emb.embedding_dim = dims - p.stack.model_dim = dims - p.stack.splits = 2 - p.stack.softmax_tpl.input_dim = dims - p.stack.softmax_tpl.num_classes = vocab - p.stack.batch_dim = 0 - trans_tpl = batch_major_attention.GPipeTransformerLayer.Params() - trans_tpl.has_aux_atten = False - trans_tpl.mask_self_atten = True - trans_tpl.input_dim = dims - trans_tpl.output_dim = dims - trans_tpl.tr_atten_tpl.input_dim = dims - trans_tpl.tr_atten_tpl.hidden_dim = dims - trans_tpl.tr_atten_tpl.is_masked = True - trans_tpl.tr_atten_tpl.num_heads = 2 - trans_tpl.tr_fflayer_tpl.hidden_dim = hidden_dim - p.stack.encoder_tpl = trans_tpl - p.random_seed = 12345 - return p - - def _SetupGraph(self, p, time, batch, vocab, return_grad=False): - lm = p.Instantiate() - np.random.seed(12345) - inputs = np.random.randint(vocab, size=[time, batch]) - targets = np.zeros([time, batch]) - targets[:-1] = inputs[1:] - inputs = tf.constant(inputs, tf.int32) - paddings = np.zeros([time, batch]) - paddings[-1] = 1.0 - paddings = tf.constant(paddings, tf.float64 if return_grad else tf.float32) - targets = tf.constant(targets, tf.int32) - xent_output, _ = lm.FPropDefaultTheta( - inputs=inputs, - paddings=paddings, - labels=py_utils.NestedMap( - class_weights=1 - paddings, class_ids=targets)) - if not return_grad: - return xent_output - - lm_vars = lm.vars.Flatten() - grads = tf.gradients(xent_output.avg_xent, lm_vars) - for i, x in enumerate(grads): - if isinstance(x, tf.IndexedSlices): - grads[i] = tf.math.unsorted_segment_sum(x.values, x.indices, - x.dense_shape[0]) - self.assertEqual(len(lm_vars), len(grads)) - return xent_output, lm_vars, grads - - def testBasic(self): - time, batch, dims, hidden_dim, vocab = 5, 3, 6, 4, 8 - p = self._testParams(batch, dims, hidden_dim, vocab) - xent_output = self._SetupGraph(p, time, batch, vocab) - assert p.stack.encoder_tpl.tr_atten_tpl.is_masked - with self.session(): - self.evaluate(tf.global_variables_initializer()) - xent_output_val = self.evaluate(xent_output) - - print('xent_output_val', xent_output_val) - test_utils.CompareToGoldenSingleFloat(self, 5.12745, - xent_output_val.avg_xent) - self.assertAllEqual(xent_output_val.per_example_argmax, - np.argmax(xent_output_val.logits, axis=-1)) - - def testBasicGrad(self): - time, batch, dims, hidden_dim, vocab = 5, 3, 6, 4, 8 - p = self._testParams(batch, dims, hidden_dim, vocab) - p.dtype = tf.float64 - xent_output, lm_vars, grads = self._SetupGraph( - p, time, batch, vocab, return_grad=True) - with self.session() as sess: - self.evaluate(tf.global_variables_initializer()) - for x, grad_x in zip(lm_vars, grads): - grad_symbolic = self.evaluate(grad_x) - grad_numeric = test_utils.ComputeNumericGradient( - sess, xent_output.avg_xent, x, delta=1e-6) - self.assertAllClose(grad_symbolic, grad_numeric, atol=0.05) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/lm/model.py b/lingvo/tasks/lm/model.py deleted file mode 100644 index d0d86217..00000000 --- a/lingvo/tasks/lm/model.py +++ /dev/null @@ -1,288 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""LM models.""" - -import lingvo.compat as tf -from lingvo.core import base_model -from lingvo.core import py_utils -from lingvo.core import schedule -from lingvo.tasks.lm import layers - - -class LanguageModel(base_model.BaseTask): - """Language model.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('lm', layers.RnnLm.Params(), 'LM layer.') - - tp = p.train - tp.Define( - 'max_lstm_gradient_norm', 0.0, - 'Clip gradient for vars in lstm layers by setting this value to ' - 'something > 0.') - tp.Define( - 'sum_loss_across_tokens_in_batch', False, - 'Sum the logP across predicted tokens in batch when set to True; ' - 'average across predicted tokens in batch o/w (default).') - - tp.lr_schedule = schedule.PiecewiseConstantSchedule.Params().Set( - boundaries=[350000, 500000, 600000], values=[1.0, 0.1, 0.01, 0.001]) - tp.vn_start_step = 20000 - tp.vn_std = 0.0 - tp.learning_rate = 0.001 - tp.l2_regularizer_weight = 1e-6 - tp.clip_gradient_norm_to_value = 1.0 - tp.grad_norm_to_clip_to_zero = 100.0 - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - assert p.lm.vocab_size == p.input.tokenizer.vocab_size, ( - 'lm.vocab_size does not match input.tokenizer.vocab_size: %d vs %d' % - (p.lm.vocab_size, p.input.tokenizer.vocab_size)) - - # Construct the model. - self.CreateChild('lm', p.lm) - - @classmethod - def UpdateTargetVocabSize(cls, p, vocab_size, wpm_model=None): - """Updates the params with the input vocab_size and WPM model. - - Args: - p: model params. - vocab_size: size of the vocabulary. - wpm_model: file name prefix pointing to a wordpiece model. - - Returns: - Model params updated with the vocab size and wpm model. - """ - p.lm = p.lm.cls.UpdateTargetVocabSize(p.lm, vocab_size, wpm_model) - return p - - def _TrimIfPossibleThenTranspose(self, ids, paddings, labels, weights): - data = (ids, paddings, labels, weights) - if not py_utils.use_tpu(): - max_seq_len = tf.cast( - tf.reduce_max(tf.reduce_sum(1.0 - paddings, 1)), tf.int32) - data = (x[:, :max_seq_len] for x in data) - return (tf.transpose(x) for x in data) - - def FPropTower(self, theta, input_batch): - p = self.params - ids, paddings, labels_ids, weights = self._TrimIfPossibleThenTranspose( - input_batch.ids, input_batch.paddings, input_batch.labels, - input_batch.weights) - - batch_size = tf.shape(ids)[1] - state0 = self.lm.zero_state(theta.lm, batch_size) - labels = py_utils.NestedMap(class_ids=labels_ids, class_weights=weights) - xent_output, _ = self.lm.FProp(theta.lm, ids, paddings, state0, labels) - - # +1 to account for the end of sequence symbol. - num_words = tf.cast( - tf.reduce_sum(input_batch.word_count + tf.constant(1, dtype=tf.int32)), - tf.float32) - predicted_labels = tf.cast(xent_output.per_example_argmax, labels_ids.dtype) - - num_preds = xent_output.total_weight - mean_acc = tf.reduce_sum( - tf.cast(tf.equal(labels_ids, predicted_labels), tf.float32) * - weights) / ( - num_preds + 1e-4) - - loss = xent_output.avg_xent - if p.train.sum_loss_across_tokens_in_batch: - loss = xent_output.total_xent - return { - 'loss': (loss, num_preds), - 'fraction_of_correct_next_step_preds': (mean_acc, num_preds), - 'log_pplx': (xent_output.avg_xent, num_preds), - 'log_pplx_per_word': (xent_output.total_xent / num_words, num_words), - 'num_predictions': (num_preds, 1), - 'num_words': (num_words, 1) - }, {} - - def AdjustGradients(self, var_grad): - """Clip LSTM gradients. - - Args: - var_grad: a `.NestedMap` of (variable, gradient). You can view - `var_grad` as an ordered list of (key, (var, grad)) tuples. Every - key of `var_grad` exists in `vmap`. Every variable in `vmap` that - contributes to loss must exist in `var_grad`. Every var of `var_grad` - must exist in `vmap`. `grad` is the corresponding gradient computed - for `var`. `grad` is guaranteed to be not None. - - Returns: - adjusted version of `var_grad` that has clipped the LSTM gradients - if `self.params.max_lstm_gradient_norm` is set. - """ - - p = self.params - if p.train.max_lstm_gradient_norm: - lstm_var_grad = var_grad.lm.rnns - lstm_vars = lstm_var_grad.Transform(lambda x: x[0]).Flatten() - lstm_grads = lstm_var_grad.Transform(lambda x: x[1]).Flatten() - clipped_lstm_grads, _ = tf.clip_by_global_norm( - lstm_grads, p.train.max_lstm_gradient_norm) - var_grad.lm.rnns = var_grad.lm.rnns.Pack( - list(zip(lstm_vars, clipped_lstm_grads))) - - return var_grad - - def Inference(self): - """Constructs the inference subgraphs. - - Returns: - dict: ``{'subgraph_name': (fetches, feeds)}`` - """ - subgraphs = {} - with tf.name_scope('inference'): - subgraphs['default'] = self._InferenceSubgraph_Default() - return subgraphs - - def _InferenceSubgraph_Default(self): - """Default inference subgraph. - - Returns: - (fetches, feeds): - - - fetches: A dictionary of fetches, containing: - - - log_pplx_per_token: A matrix of shape [batch, time]. [i, j] - is i-th input text's j-th token's log prob. - - paddings: A matrix of shape [batch, time]. The padding mask. - - log_pplx_per_sample: A vector of shape [batch]. [i] - is i-th input text's log prob. - - num_oovs_per_sample: A vector of shape [batch] counting the total - number of out-of-vocabulary tokens in each input. - - tokens_from_labels: A vector of shape [batch] returning the predicted - tokens as a sequence after mapping them back to strings from ids using - the vocabulary. - - ids: A matrix of shape [batch, time]. [i, j] - is i-th input text's j-th token's id. - - - feeds: A dictionary of feeds, containing: - - - text: A placeholder for a vector of strings. - """ - text = tf.placeholder(tf.string, shape=[None]) - # [batch, time] - ids, labels, paddings = self.input_generator.StringsToIds(text) - lengths = tf.reduce_sum(tf.cast(1 - paddings, tf.int32), axis=1) - tokens_from_labels = self.input_generator.IdsToStrings(labels, lengths) - oovs = tf.equal(labels, self.input_generator.tokenizer.unk_id) - num_oovs_per_sample = tf.cast( - tf.round( - tf.reduce_sum(tf.cast(oovs, tf.float32) * (1 - paddings), axis=1)), - tf.int32) - # [time, batch] - ids, paddings, labels, weights = self._TrimIfPossibleThenTranspose( - ids, paddings, labels, 1.0 - paddings) - batch_size = tf.shape(ids)[1] - xent_output, _ = self.lm.FPropDefaultTheta( - inputs=ids, - paddings=paddings, - state0=self.lm.zero_state(self.theta.lm, batch_size), - labels=py_utils.NestedMap(class_ids=labels, class_weights=weights)) - - per_example_xent = py_utils.HasShape(xent_output.per_example_xent, - tf.shape(ids)) - log_pplx_per_sample = tf.reduce_sum( - per_example_xent * (1 - paddings), axis=0) - fetches = { - 'log_pplx_per_token': # [batch, time] - tf.transpose(per_example_xent), - 'paddings': # [batch, time] - tf.transpose(paddings), - 'lengths': # [batch] - lengths, - 'log_pplx_per_sample': # [batch] - log_pplx_per_sample, - 'num_oovs_per_sample': # [batch], int32 - num_oovs_per_sample, - 'tokens_from_labels': # [batch], string - tokens_from_labels, - 'ids': # [batch, time], int32 - ids - } - feeds = {'text': text} - return fetches, feeds - - -class FixedShapeInputLanguageModel(LanguageModel): - - def _TrimIfPossibleThenTranspose(self, ids, paddings, labels, weights): - data = (ids, paddings, labels, weights) - if not py_utils.use_tpu() and self.do_eval: - max_seq_len = tf.cast( - tf.reduce_max(tf.reduce_sum(1.0 - paddings, 1)), tf.int32) - data = (x[:, :max_seq_len] for x in data) - return (tf.transpose(x) for x in data) - - -class BatchMajorLanguageModel(LanguageModel): - """Batch major implementation of the language model.""" - - def _TrimIfPossible(self, ids, paddings, labels, weights): - data = (ids, paddings, labels, weights) - if not py_utils.use_tpu(): - max_seq_len = tf.cast( - tf.reduce_max(tf.reduce_sum(1.0 - paddings, 1)), tf.int32) - data = (x[:, :max_seq_len] for x in data) - return data - - def FPropTower(self, theta, input_batch): - p = self.params - tf.logging.info('input_batch=%r', input_batch) - ids, paddings, labels_ids, weights = self._TrimIfPossible( - input_batch.ids, input_batch.paddings, input_batch.labels, - input_batch.weights) - fprop_dtype = py_utils.FPropDtype(p) - paddings = tf.cast(paddings, fprop_dtype) - weights = tf.cast(weights, fprop_dtype) - tf.logging.info('inputs={}'.format((ids, paddings, labels_ids, weights))) - - batch_size = tf.shape(ids)[0] - state0 = self.lm.zero_state(theta.lm, batch_size) - labels = py_utils.NestedMap(class_ids=labels_ids, class_weights=weights) - xent_output, _ = self.lm.FProp(theta.lm, ids, paddings, state0, labels) - - # +1 to account for the end of sequence symbol. - num_words = tf.cast( - tf.reduce_sum(input_batch.word_count + tf.constant(1, dtype=tf.int32)), - fprop_dtype) - predicted_labels = tf.cast(xent_output.per_example_argmax, labels_ids.dtype) - - num_preds = xent_output.total_weight - mean_acc = tf.reduce_sum( - tf.cast(tf.equal(labels_ids, predicted_labels), fprop_dtype) * - weights) / tf.math.maximum(num_preds, 1) - loss = xent_output.avg_xent - if p.train.sum_loss_across_tokens_in_batch: - loss = xent_output.total_xent - return { - 'loss': (loss, num_preds), - 'fraction_of_correct_next_step_preds': (mean_acc, num_preds), - 'log_pplx': (xent_output.avg_xent, num_preds), - 'log_pplx_per_word': (xent_output.total_xent / num_words, num_words), - 'num_predictions': (num_preds, 1), - 'num_words': (num_words, 1) - }, {} diff --git a/lingvo/tasks/lm/model_test.py b/lingvo/tasks/lm/model_test.py deleted file mode 100644 index f4c6cd76..00000000 --- a/lingvo/tasks/lm/model_test.py +++ /dev/null @@ -1,170 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for lm.model.""" - -import lingvo.compat as tf -from lingvo.core import test_helper -from lingvo.core import test_utils -from lingvo.core import tokenizers -from lingvo.tasks.lm import input_generator -from lingvo.tasks.lm import model - - -class ModelTest(test_utils.TestCase): - - def _InputParams(self, for_training): - p = input_generator.LmInput.Params() - p.file_pattern = 'text:' + test_helper.test_src_dir_path( - 'tasks/lm/testdata/lm1b_100.txt') - if for_training: - p.file_buffer_size = 10000 - p.bucket_upper_bound = [50] - p.bucket_batch_limit = [64] - p.target_max_length = 1000 - else: - p.file_random_seed = 31415 - p.file_parallelism = 1 - p.file_buffer_size = 1 - p.bucket_upper_bound = [20] - p.bucket_batch_limit = [2] - p.target_max_length = 20 - p.tokenizer.vocab_size = 64 - return p - - def _Params(self): - p = model.LanguageModel.Params() - p.name = 'lm_test' - vocab, dims = 64, 64 - p.lm.vocab_size = vocab - p.lm.emb.vocab_size = vocab - p.lm.emb.embedding_dim = dims - p.lm.rnns.num_layers = 4 - p.lm.rnns.cell_tpl.num_output_nodes = dims - p.lm.rnns.cell_tpl.num_input_nodes = dims - p.lm.softmax.input_dim = dims - p.lm.softmax.num_classes = vocab - return p - - def testLmFprop(self): - tf.random.set_seed(93820986) - p = self._Params() - p.input = self._InputParams(for_training=False) - - with self.session(use_gpu=False): - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - loss = mdl.eval_metrics['loss'][0] - logp = mdl.eval_metrics['log_pplx'][0] - logp_per_word = mdl.eval_metrics['log_pplx_per_word'][0] - accuracy = mdl.eval_metrics['fraction_of_correct_next_step_preds'][0] - self.evaluate(tf.global_variables_initializer()) - - loss, logp, logp_per_word, accuracy = self.evaluate( - [loss, logp, logp_per_word, accuracy]) - test_utils.CompareToGoldenSingleFloat(self, 4.160992, loss) - test_utils.CompareToGoldenSingleFloat(self, 4.160992, logp) - test_utils.CompareToGoldenSingleFloat(self, 5.944274, logp_per_word) - test_utils.CompareToGoldenSingleFloat(self, 0.000000, accuracy) - - def testLmTrain(self): - p = self._Params() - p.input = self._InputParams(for_training=True) - tp = p.train - tp.learning_rate = 3e-3 - - with self.session(): - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - mdl.BProp() - loss = mdl.eval_metrics['loss'][0] - self.evaluate(tf.global_variables_initializer()) - - # Run some steps and we expect the loss goes down. - loss_val, _ = self.evaluate([loss, mdl.train_op]) - self.assertGreater(loss_val, 4.0) - for i in range(10): - loss_val, _ = self.evaluate([loss, mdl.train_op]) - tf.logging.info('%d loss = %f', i, loss_val) - self.assertLess(loss_val, 3.8) - - def testLmInference(self): - tf.random.set_seed(93820986) - p = self._Params() - p.input = self._InputParams(for_training=False) - tf.logging.info('Params: %s', p.ToText()) - - with self.session(use_gpu=False) as sess: - mdl = p.Instantiate() - subgraphs = mdl.Inference() - self.assertIn('default', subgraphs) - fetches, feeds = subgraphs['default'] - self.evaluate(tf.global_variables_initializer()) - vals = sess.run( - fetches=fetches, - feed_dict={feeds['text']: ['pray for world peace', 'happy birthday']}) - print('actual vals = ', vals) - self.assertEqual(vals['log_pplx_per_sample'].shape, (2,)) - self.assertEqual(vals['log_pplx_per_token'].shape, (2, 20)) - self.assertEqual(vals['paddings'].shape, (2, 20)) - - def testLmInferenceWordLevel(self): - tf.random.set_seed(93820986) - p = self._Params() - p.input = self._InputParams(for_training=False) - p.input.tokenizer = tokenizers.VocabFileTokenizer.Params() - p.input.tokenizer.vocab_size = 64 - p.input.target_max_length = 5 - p.input.pad_to_max_seq_length = True - # target_{sos,eos,unk}_id must be consistent with token_vocab_filepath. - p.input.tokenizer.target_sos_id = 1 - p.input.tokenizer.target_eos_id = 2 - p.input.tokenizer.target_unk_id = 3 - p.input.tokenizer.token_vocab_filepath = test_helper.test_src_dir_path( - 'tasks/lm/testdata/small_word_vocab.txt') - tf.logging.info('Params: %s', p.ToText()) - - with self.session(use_gpu=False) as sess: - mdl = p.Instantiate() - subgraphs = mdl.Inference() - self.assertIn('default', subgraphs) - fetches, feeds = subgraphs['default'] - self.evaluate(tf.global_variables_initializer()) - vals = sess.run( - fetches=fetches, - feed_dict={ - feeds['text']: [ - 'pray for more peace', 'happy about', 'one flambergastic will' - ] - }) - print('actual vals = ', vals) - self.assertEqual(vals['log_pplx_per_sample'].shape, (3,)) - self.assertEqual(vals['log_pplx_per_token'].shape, (3, 5)) - self.assertEqual(vals['paddings'].shape, (3, 5)) - expected_tokens_from_labels = [ - b' for more ', b' about ', - b'one will ' - ] - self.assertListEqual(vals['tokens_from_labels'].tolist(), - expected_tokens_from_labels) - expected_num_oovs_per_sample = [2, 1, 1] - self.assertListEqual(vals['num_oovs_per_sample'].tolist(), - expected_num_oovs_per_sample) - expected_ids = [[1, 1, 1], [3, 3, 41], [21, 8, 3], [35, 2, 61], [3, 2, 2]] - self.assertListEqual(vals['ids'].tolist(), expected_ids) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/lm/params/BUILD b/lingvo/tasks/lm/params/BUILD deleted file mode 100644 index b3d44db3..00000000 --- a/lingvo/tasks/lm/params/BUILD +++ /dev/null @@ -1,33 +0,0 @@ -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -py_library( - name = "one_billion_wds", - srcs = ["one_billion_wds.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:model_registry", - "//lingvo/core:base_model_params", - "//lingvo/core:layers", - "//lingvo/core:optimizer", - "//lingvo/core:py_utils", - "//lingvo/core:schedule", - "//lingvo/core:tokenizers", - "//lingvo/tasks/lm:input_generator", - "//lingvo/tasks/lm:layers", - "//lingvo/tasks/lm:model", - ], -) - -py_library( - name = "params", - srcs = ["params.py"], - srcs_version = "PY3", - tags = ["keep_dep"], - deps = [ - ":one_billion_wds", - ], -) diff --git a/lingvo/tasks/lm/params/one_billion_wds.py b/lingvo/tasks/lm/params/one_billion_wds.py deleted file mode 100644 index 28928011..00000000 --- a/lingvo/tasks/lm/params/one_billion_wds.py +++ /dev/null @@ -1,259 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Train word-level LMs on 1 Billion Words benchmark data.""" - -import os - -from lingvo import model_registry -from lingvo.core import base_model_params -from lingvo.core import layers -from lingvo.core import optimizer -from lingvo.core import py_utils -from lingvo.core import schedule -from lingvo.core import tokenizers -from lingvo.tasks.lm import input_generator as lm_inp -from lingvo.tasks.lm import layers as lm_layers -from lingvo.tasks.lm import model - - -class WordLevelOneBwdsBase(base_model_params.SingleTaskModelParams): - """Params for training a word-level LM on One Billion Wds text corpus. - - Tries to match https://github.com/rafaljozefowicz/lm. - """ - - # Generated using lingvo/tasks/lm/tools:download_lm1b. - CORPUS_DIR = '/tmp/lm1b/1-billion-word-language-modeling-benchmark-r13output/' - - # BIG-LSTM model size: embedding/projection dim = 1024; LSTM state dim = 8192 - EMBEDDING_DIM = 1024 - MAX_TOKENS = 1024 - NUM_EMBEDDING_SHARDS = 8 - NUM_SAMPLED = 8192 - NUM_SOFTMAX_SHARDS = 8 - RNN_STATE_DIM = 8192 - VOCAB_SIZE = 793472 # includes - WORD_VOCAB = os.path.join(CORPUS_DIR, 'vocab.txt') - - def Train(self): - p = lm_inp.LmInput.Params() - p.bucket_upper_bound = [10, 20, 30, 40, 50, 100, 256, 512, 1024] - p.bucket_batch_limit = [1024, 512, 256, 256, 128, 128, 64, 32, 16] - p.file_buffer_size = 10000000 - p.file_parallelism = 10 - p.file_pattern = 'text:' + os.path.join( - self.CORPUS_DIR, 'training-monolingual.tokenized.shuffled', 'news.en*') - p.name = '1bwds_train_set' - p.tokenizer = tokenizers.VocabFileTokenizer.Params() - p.num_batcher_threads = 16 - p.target_max_length = self.MAX_TOKENS - p.tokenizer.target_sos_id = 1 - p.tokenizer.target_eos_id = 2 - p.tokenizer.target_unk_id = 3 - p.tokenizer.token_vocab_filepath = self.WORD_VOCAB - p.tokenizer.vocab_size = self.VOCAB_SIZE - return p - - def Dev(self): - p = self.Train() - # Use small batches for eval. - p.bucket_upper_bound = [10, 20, 30, 40, 50, 100, 256, 512, 1024] - p.bucket_batch_limit = [128, 64, 32, 32, 16, 16, 4, 2, 1] - p.file_buffer_size = 1 - p.file_parallelism = 1 - p.file_pattern = 'text:' + os.path.join( - self.CORPUS_DIR, 'heldout-monolingual.tokenized.shuffled', - 'news.en.heldout-00001*') - p.name = '1bwds_dev_set' - p.num_batcher_threads = 1 - p.num_samples = 6206 # Number of sentences to evaluate on. - return p - - def Test(self): - p = self.Dev() - p.file_pattern = 'text:' + os.path.join( - self.CORPUS_DIR, 'heldout-monolingual.tokenized.shuffled', - 'news.en.heldout-00000*') - p.name = '1bwds_test_set' - p.num_samples = 6075 # Number of sentences to evaluate on. - return p - - def Task(self): - p = model.LanguageModel.Params() - p.name = '1bwds_word_level_lm' - p.eval.samples_per_summary = 10000 - - p.lm = lm_layers.RnnLm.CommonParams( - vocab_size=self.VOCAB_SIZE, - emb_dim=self.EMBEDDING_DIM, - num_layers=2, - residual_start=3, # disable residuals - rnn_dims=self.EMBEDDING_DIM, - rnn_hidden_dims=self.RNN_STATE_DIM) - - # Input embedding needs to be sharded. - p.lm.emb.max_num_shards = self.NUM_EMBEDDING_SHARDS - p.lm.embedding_dropout_keep_prob = 0.75 - # Match the initialization in github code. - p.lm.emb.params_init = py_utils.WeightInit.UniformUnitScaling( - 1.0 * self.NUM_EMBEDDING_SHARDS) - - # We also want dropout after each of the RNN layers. - p.lm.rnns.dropout.keep_prob = 0.75 - - # Adjusts training params. - tp = p.train - tp.sum_loss_across_tokens_in_batch = True - # Disable any so called "clipping" (gradient scaling really). - tp.clip_gradient_norm_to_value = 0.0 - tp.grad_norm_to_clip_to_zero = 0.0 - # Do clip the LSTM gradients. - tp.max_lstm_gradient_norm = 16 - # Straight Adagrad; very sensitive to initial accumulator value, the default - # 0.1 value is far from adequate. - # TODO(ciprianchelba): tune accumulator value, learning rate, clipping - # threshold. - tp.learning_rate = 0.2 - tp.lr_schedule = ( - schedule.PiecewiseConstantSchedule.Params().Set( - boundaries=[], values=[1.0])) - tp.l2_regularizer_weight = None # No regularization. - tp.optimizer = optimizer.Adagrad.Params() - return p - - -@model_registry.RegisterSingleTaskModel -class WordLevelOneBwdsSimpleSampledSoftmax(WordLevelOneBwdsBase): - """Use sampled soft-max in training.""" - - def Task(self): - p = super().Task() - num_input_dim = p.lm.softmax.input_dim - p.lm.softmax = layers.SimpleFullSoftmax.Params() - p.lm.softmax.input_dim = num_input_dim - p.lm.softmax.num_classes = self.VOCAB_SIZE - p.lm.softmax.num_sampled = self.NUM_SAMPLED - p.lm.softmax.num_shards = self.NUM_SOFTMAX_SHARDS - # Match the initialization in github code. - p.lm.softmax.params_init = py_utils.WeightInit.UniformUnitScaling( - 1.0 * self.NUM_SOFTMAX_SHARDS) - assert p.lm.softmax.num_classes % p.lm.softmax.num_shards == 0 - return p - - -@model_registry.RegisterSingleTaskModel -class WordLevelOneBwdsSimpleSampledSoftmaxTiny( - WordLevelOneBwdsSimpleSampledSoftmax): - """Tiny model size for local, debugging runs of the above.""" - - EMBEDDING_DIM = 7 - MAX_TOKENS = 1024 - NUM_EMBEDDING_SHARDS = 1 - NUM_SAMPLED = 8 - NUM_SOFTMAX_SHARDS = 8 - RNN_STATE_DIM = 32 - - -# Example large transformer model using GPIPE. -# Instructions: -# trainer --run_locally=gpu --mode=sync \ -# --model=lm.one_billion_wds.OneBWdsGPipeTransformerWPM \ -# --logdir=/tmp/lm/log --logtostderr --worker_split_size=4 --worker_gpus=4 -# Relative throughput on multiple V100s, each with 16GB ram. -# GPUs throughput -# 1 1 -# 2 0.93 -# 4 0.85 -# 8 0.775 -@model_registry.RegisterSingleTaskModel -class OneBWdsGPipeTransformerWPM(WordLevelOneBwdsBase): - """LM using gpipe transformer.""" - VOCAB_SIZE = 32000 - EMBEDDING_DIM = 2048 - BATCH_SIZE = 32 - MAX_TOKENS = 1024 # The max sequence length in one example. - - # GPIPE related params. - GPUS = 4 - # A list of ending index for each split/partition in ascending order. - # For example SPLITS = [8, 16, 24, 32] defined a 32 layer model with 4 splits, - # each of which contains 8 layers. - # The number belows runs on 16GB-V100s. Your mileage may vary. - SPLITS = [8 * (i + 1) for i in range(GPUS)] - LAYERS = SPLITS[-1] - # Set NUM_MICRO_BATCHES >= len(SPLITS) * 4 to minimize gpipe bubble. - NUM_MICRO_BATCHES = 32 - - def Train(self): - p = super().Train() - # Replace it with your own wordpiece tokenizer. - p.tokenizer = tokenizers.AsciiTokenizer.Params() - p.target_max_length = self.MAX_TOKENS - p.tokenizer.target_sos_id = 1 - p.tokenizer.target_eos_id = 2 - p.tokenizer.target_unk_id = 0 - p.tokenizer.vocab_size = self.VOCAB_SIZE - p.bucket_upper_bound = [self.MAX_TOKENS] - p.bucket_batch_limit = [self.BATCH_SIZE] - p.fixed_input_shape = True - return p - - def Dev(self): - p = self.Train() - p.file_pattern = 'text:' + os.path.join( - self.CORPUS_DIR, 'heldout-monolingual.tokenized.shuffled', - 'news.en.heldout-00001*') - p.name = '1bwds_dev_set' - p.num_batcher_threads = 1 - p.num_samples = 6206 # Number of sentences to evaluate on. - return p - - def Test(self): - p = self.Dev() - p.file_pattern = 'text:' + os.path.join( - self.CORPUS_DIR, 'heldout-monolingual.tokenized.shuffled', - 'news.en.heldout-00000*') - p.name = '1bwds_test_set' - p.num_samples = 6075 # Number of sentences to evaluate on. - return p - - def Task(self): - """Language model on 1bw dataset using gpipe transformer.""" - p = model.BatchMajorLanguageModel.Params() - p.eval.samples_per_summary = 0 - p.name = '1bwds_wpm_level_lm' - p.lm = lm_layers.GPipeTransformerLm.CommonParams( - model_dim=self.EMBEDDING_DIM, - vocab_size=self.VOCAB_SIZE, - hidden_dim=self.EMBEDDING_DIM * 4, - num_layers=self.LAYERS, - splits=self.SPLITS, - num_micro_batches=self.NUM_MICRO_BATCHES, - micro_batch_size=self.BATCH_SIZE // self.NUM_MICRO_BATCHES, - num_heads=16, - softmax_max_alloc=128 * (2**20), - atten_dropout_prob=0.1, - residual_dropout_prob=0.1) - - p.train.Set( - learning_rate=0.5, - optimizer=optimizer.Adam.ParamsA(), - clip_gradient_norm_to_value=0.0, - grad_norm_to_clip_to_zero=0.0, - lr_schedule=schedule.TransformerSchedule.Params().Set( - warmup_steps=40000, worker_replicas=1, - model_dim=self.EMBEDDING_DIM)) - return p diff --git a/lingvo/tasks/lm/params/params.py b/lingvo/tasks/lm/params/params.py deleted file mode 100644 index 46ffb9ff..00000000 --- a/lingvo/tasks/lm/params/params.py +++ /dev/null @@ -1,21 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Language models hyper-parameters.""" - -# Import ModelParams to ensure that they are added to the global registry. -# pylint: disable=unused-import -import lingvo.tasks.lm.params.one_billion_wds -# pylint: enable=unused-import diff --git a/lingvo/tasks/lm/testdata/BUILD b/lingvo/tasks/lm/testdata/BUILD deleted file mode 100644 index 5bcb9ff3..00000000 --- a/lingvo/tasks/lm/testdata/BUILD +++ /dev/null @@ -1,16 +0,0 @@ -package(default_visibility = ["//visibility:public"]) - -licenses(["notice"]) # Apache 2.0 - -# First 100 training sentences from the 1 Billion Word Language Model Benchmark. -# http://www.statmt.org/lm-benchmark -filegroup( - name = "lm1b_100", - data = ["lm1b_100.txt"], -) - -# Small word-level vocab file (taken from a list of stopwords). -filegroup( - name = "small_word_vocab", - data = ["small_word_vocab.txt"], -) diff --git a/lingvo/tasks/lm/testdata/lm1b_100.txt b/lingvo/tasks/lm/testdata/lm1b_100.txt deleted file mode 100644 index 0197314b..00000000 --- a/lingvo/tasks/lm/testdata/lm1b_100.txt +++ /dev/null @@ -1,100 +0,0 @@ -The U.S. Centers for Disease Control and Prevention initially advised school systems to close if outbreaks occurred , then reversed itself , saying the apparent mildness of the virus meant most schools and day care centers should stay open , even if they had confirmed cases of swine flu . -When Ms. Winfrey invited Suzanne Somers to share her controversial views about bio-identical hormone treatment on her syndicated show in 2009 , it won Ms. Winfrey a rare dollop of unflattering press , including a Newsweek cover story titled " Crazy Talk : Oprah , Wacky Cures & You . " -Elk calling -- a skill that hunters perfected long ago to lure game with the promise of a little romance -- is now its own sport . -Don 't ! -Fish , ranked 98th in the world , fired 22 aces en route to a 6-3 , 6-7 ( 5 / 7 ) , 7-6 ( 7 / 4 ) win over seventh-seeded Argentinian David Nalbandian . -Why does everything have to become such a big issue ? -AMMAN ( Reuters ) - King Abdullah of Jordan will meet U.S. President Barack Obama in Washington on April 21 to lobby on behalf of Arab states for a stronger U.S. role in Middle East peacemaking , palace officials said on Sunday . -To help keep traffic flowing the Congestion Charge will remain in operation through-out the strike and TfL will be suspending road works on major London roads wherever possible . -If no candidate wins an absolute majority , there will be a runoff between the top two contenders , most likely in mid-October . -Authorities previously served search warrants at Murray 's Las Vegas home and his businesses in Las Vegas and Houston . -Brent North Sea crude for November delivery rose 84 cents to 68.88 dollars a barrel . -That seems to have been their model up til now . -Gordon will join Luol Deng on the GB team ; their respective NBA teams , the Detroit Pistons and the Chicago Bulls , play tonight . -Nikam maintains the attacks were masterminded by the Muslim militant group Lashkar-e-Taiba . -Last year , Williams was unseeded , ranked 81st and coming off one of her worst losses on tour -- in a Tier 4 event at Hobart -- yet she beat six seeded players en route to the title at Melbourne Park . -It said that two officers involved in the case had been disciplined . -" There is more intelligence now being gathered , " the official said , adding that such efforts would continue for some time . -The majority will be of the standard 6X6 configuration for carrying personnel . -" Consequently , necessary actions may not be taken to reduce the risks to children of sexual exploitation and drug or alcohol misuse , " the report said . • Almost two-thirds of inspected schools were good or outstanding , but the number of underperforming secondaries remained " stubborn and persistent . " -What a World Cup . -But , there have also been many cases of individuals and small groups of people protesting , as in the case of Rongye Adak , a nomad who called for the return of the Dalai Lama and for the freedom of Tibet during the Lithang Horse Racing Festival , in eastern Tibet . -James Duncan , head of transportation at Bournemouth Borough Council , said : " Our legal team is reviewing the entitlement of taxis to drop and pick up passengers at bus stops , only for as long as is absolutely necessary to fulfil that function and for no other reason . -To Mo concerning the food log you kept -- Dr. Buchholz recommends the same thing . -The CBO estimates that only 23 percent of that would be spent in 2009 and 2010 . -Even so , Democrats slammed Bush as out of touch . -An information campaign will be launched later to raise awareness of employment rights and how to enforce them . -At the gallery the concept is less vague , as Ms. Piper cites specific instances of racial violence , political assassinations and the devastation of Hurricane Katrina . -There have been some exceptions -- such as Medicare in 1965 . -The government guidance will be reviewed early next year after a period of public comment . -It wasn 't the most seaworthy of prizes . -LOUIS - A former Anheuser-Busch executive who was the company 's top-ranking woman has sued the brewer claiming it encourages a " frat party " atmosphere and pays women less in salary and bonuses than men . -What does it say about Pelosi that she lost 95 dem votes when she has all the power . -Skea , a 3-month-old Taipei store that has nothing to do with Ikea , sells custom-made , small-scale buildings as well as cars , boats , baseball mitts or just about anything else that can be made from paper , cardboard and light wood . -A dozen athletes with connections to Lake Placid competed in Vancouver . -They owe $ 10,500 on their car . -" We need them to move on and accept change , because their problems are a distraction from our goal of building a more integrated Asean , " said Ong Keng Yong , the association 's secretary general . -The internet is sort-of-40 this year . -His approach was inquisitive , a meeting of artful hesitation with fluid technique . -Katrin 's father , Dr Edmund Radmacher , inherited a chemistry company that he transformed into a flourishing concern . -On that basis Mr Perry is not a man fit for higher office . -Grand ladies in exquisitely tailored houndstooth check may have had husbands who bought them their rarefied Dior clothing , but it is only too clear who was wearing the trousers . -The two men reportedly lived alone in the shelter , a two-bedroom flat that is run by a private social care company , financed partly by the city authorities . -A lot rode on the opening lead . -Why you support the FAKE war on terror I understand , you are just following orders . -The motive for the attack on the pope remains unclear but it has not been linked to Islamic issues . -A post-mortem examination is due to be carried out in the next few days , a spokesman said . -HBOS is the lead bank in a syndicate that includes Lloyds TSB , Royal Bank of Scotland , HSBC and National Australia Bank and granted New Star the loan last year . -You do , however , still have to choose between those technologies on most larger screens . -There are currently some 5 million homeowners that are 90 days or more past due on their mortgages , according to Fannie Mae chief economist Doug Duncan . -There was an extended paralysis of racing in the major horse racing states of New South Wales and Queensland , including the loss of popular traditional racing carnivals . -They wanted things to be monumental , so they could be seen from a distance , " explains Laurence Mouillefarine , associate curator of the show , picking out some big , showy brooches to illustrate the point . -More than 45,000 women are diagnosed with breast cancer each year in the UK , and more than 12,000 die from the disease . -That agency 's grant panels do not use culturally specific criteria when awarding money . -In the five months since he started the company , he has proven his hunch : People are willing to pay for New York City tap water . -They also maintain that because Google 's system for identifying and displaying ads is more lucrative than Yahoo 's approach , the deal will generate additional revenue for Yahoo that will it make it a more formidable competitor to both Google and Microsoft . -Forecasts called for plunging temperatures and afternoon rain storms in the early Southern Hemisphere autumn . -Kloppers said BHP would continue talking to its customers about the merger . -Sailing is a passion . -I would argue that the media caters just to that , if not simple minded , to those who are not disciplined enough to see out the truth , THEIR OWN truth . -Android is being developed , " assures Mobiclip 's vice-president of marketing Denis Pagnac . -LONDON ( Reuters ) - Want to know your chance of having a baby through in-vitro fertilization ? -NEW YORK ( Reuters ) - Toll Brothers ( TOL.N : Quote , Profile , Research ) said on Thursday that it expected to report a 36 percent drop in quarterly home-building revenue , while net orders for new homes fell more steeply than in prior quarters , reflecting last month 's deepening decline in the U.S. housing market . -The six , who call themselves " Ztohoven , " claim that the aim of the project - called Media Reality - was to show how reality can be manipulated by the media . -Stupak had pledged to oppose the health care package unless given greater assurance that it would not allow federal funding of elective abortions . -Sheik Salman al-Feraiji , al-Sadr 's chief representative in Sadr City , issued a statement with demands to quell the discontent , including the release of Sadrist detainees , an end to military operations against them and al-Maliki 's resignation . -Cowles declined to comment to ABC News for this story . -In fairness , the Scots finished the half more adeptly than they 'd started and McFadden was unlucky not to win a penalty when he was obstructed in the box . -Results from dozens of clinical studies will be released at the gathering opening Friday and running through June 2 . -There , 1 per cent of the population owns 49 per cent of the land ; here , 0.3 per cent owns 69 per cent . -" One wall is enough , " he said . -" Next question , " said Fisher with a smile . -He seems genuinely affected and I almost suggest we meet again , informally as friends , but retrieve the last scraps of my dignity . -Prosecutors said Plotkin ran the schemes , enlisting David Pajcin , a former Goldman Sachs Group Inc. analyst who has pleaded guilty to charges in the case and is cooperating with the government . -In anticipation of prolonged outages , Rockford city officials coordinated assistance to provide access to warming centers for residents . -From TPP-- that 's how i see it. it 's a resource . -In the nearby village of Tepalcatepec , concrete homes built with dollars sent home by migrants stand next to tin-roof shacks . -They do not listen , or even know how to listen , they are operating on " eternal truths " that are not open to debate . -Most vacationers visit America 's national parks during the summer months--when the roads are crowded , restaurants are packed and prices are at their peak . -Good , there you go ! -" In World War II , we knew who the enemy was and we fought with them . -To some this may seem puzzling , given that dishonour is spread fairly evenly between both of the main parties . -3 ( UPI ) -- An Arizona elementary school principal placed on leave for writing a letter jokingly calling children " stupid " has apologized to parents , officials said . -So embrace the idea of big . -Reclusive North Korean leader Kim Jong Il isn 't always easy to explain . -Items were stolen during the incident . -Santiago tried to stop the fight after the 11th round , but Cotto went back out to take even more punishment before a final flurry along the ropes prompted referee Kenny Bayless to end it . -Knecht and Benner presented the fossil last month in Krakow , Poland , during the Second International Congress on Ichnology . -John McMahon , manager of the Torquay Tigers Football Club , said the pictures of Klan members were posted on the Internet by a volunteer , a retired member of the club , without the knowledge of officials , The Age reported Thursday . -The sub-Billy Liar / Elliot one , where the gruff father is disgusted by his son 's airy-fairy wholemeal ambitions , is particularly good . -Meanwhile , many Algerians say they have not felt the effects of the country 's new wealth . -A secret review of Fletcher 's death concluded two years ago that there was enough evidence to charge two Libyans , The Daily Telegraph reported earlier this month . -Earlier , state television said the blast happened in central Iran , without giving details . -Coach K and North Carolina 's Roy Williams are busy arguing over injury reports , so we know they didn 't hear Capel on Tobacco Road . -Lance , Inc. has scheduled a conference call and presentation with investors at 9 : 00 am eastern time on Friday , July 24 , 2009 to discuss financial results . To participate in the conference call , the dial-in number is ( 800 ) 789-3681 for U.S. callers or ( 702 ) 696-4943 for international callers . A continuous telephone replay of the call will be available beginning at 12 : 00 pm on July 24th and running through July 31st at midnight . The replay telephone number is ( 800 ) 642-1687 for U.S. callers or ( 706 ) 645-9291 for international callers . The replay access code is 17972828 . Investors may also access a web-based replay of the conference call at Lance 's website http : / / ir.lance.com. -We all have an interest in the success of these institutions . -Translated , that means debt for each household grew by $ 582 in the last six months , compared with $ 418 in the previous six . -It turns out that woman have sex for all of these reasons and more , and that their choices are not arbitrary ; there may be evolutionary explanations at work . -For months , the panel had been investigating the Democratic senator 's appointment and seating . -Because most blacks today enjoy improved education and quality of life , " dealing with the African-American community is the same as dealing with any other community , " said Dick Harpootlian , a white Obama supporter and former chairman of the South Carolina Democratic Party . -Ditto her bum-grazing cut-offs . diff --git a/lingvo/tasks/lm/testdata/small_word_vocab.txt b/lingvo/tasks/lm/testdata/small_word_vocab.txt deleted file mode 100644 index bfef33f6..00000000 --- a/lingvo/tasks/lm/testdata/small_word_vocab.txt +++ /dev/null @@ -1,64 +0,0 @@ - 0 - 1 - 2 - 3 -'s 4 -'t 5 -, 6 -. 7 -about 8 -am 9 -an 10 -and 11 -are 12 -as 13 -at 14 -be 15 -been 16 -but 17 -by 18 -did 19 -do 20 -for 21 -had 22 -has 23 -have 24 -he 25 -her 26 -him 27 -how 28 -if 29 -in 30 -is 31 -it 32 -less 33 -me 34 -more 35 -my 36 -no 37 -not 38 -of 39 -on 40 -one 41 -or 42 -our 43 -out 44 -she 45 -so 46 -that 47 -the 48 -then 49 -they 50 -this 51 -to 52 -too 53 -was 54 -we 55 -were 56 -what 57 -when 58 -who 59 -why 60 -will 61 -with 62 -you 63 diff --git a/lingvo/tasks/lm/tools/BUILD b/lingvo/tasks/lm/tools/BUILD deleted file mode 100644 index c6396780..00000000 --- a/lingvo/tasks/lm/tools/BUILD +++ /dev/null @@ -1,15 +0,0 @@ -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -py_binary( - name = "download_lm1b", - srcs = ["download_lm1b.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - ], -) diff --git a/lingvo/tasks/lm/tools/__init__.py b/lingvo/tasks/lm/tools/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/lingvo/tasks/lm/tools/download_lm1b.py b/lingvo/tasks/lm/tools/download_lm1b.py deleted file mode 100644 index 47f2c812..00000000 --- a/lingvo/tasks/lm/tools/download_lm1b.py +++ /dev/null @@ -1,60 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Downloads and processes lm1b dataset (http://www.statmt.org/lm-benchmark).""" - -import glob -import os -import lingvo.compat as tf - -tf.flags.DEFINE_string("outdir", "/tmp/lm1b", "The output directory.") -tf.flags.DEFINE_integer( - "count_cutoff", 3, "Ignore tokens that appear fewer than " - "this amount of times when creating the vocab file.") - -FLAGS = tf.flags.FLAGS - - -def main(_): - basename = "1-billion-word-language-modeling-benchmark-r13output" - fname = basename + ".tar.gz" - url = "http://www.statmt.org/lm-benchmark/" + fname - sha256hash = "01ba60381110baf7f189dfd2b8374de371e8c9a340835793f190bdae9e90a34e" - - tf.keras.utils.get_file( - fname, url, file_hash=sha256hash, cache_subdir=FLAGS.outdir, extract=True) - - tf.logging.info("Generating vocab file. This may take a few minutes.") - vocab = {} - for fname in glob.glob( - os.path.join(FLAGS.outdir, basename, - "training-monolingual.tokenized.shuffled", "news.en*")): - with open(fname) as f: - for line in f: - for w in line.split(): - vocab[w] = vocab.get(w, 0) + 1 - - with open(os.path.join(FLAGS.outdir, basename, "vocab.txt"), "w") as f: - f.write("\t0\n\t1\n\t2\n\t3\n") - id = 4 - for k, v in sorted(list(vocab.items()), key=lambda kv: (-kv[1], kv[0])): - if v < FLAGS.count_cutoff: - break - f.write("%s\t%d\n" % (k, id)) - id += 1 - - -if __name__ == "__main__": - tf.app.run(main) diff --git a/lingvo/tasks/mt/BUILD b/lingvo/tasks/mt/BUILD deleted file mode 100644 index 46ebd35c..00000000 --- a/lingvo/tasks/mt/BUILD +++ /dev/null @@ -1,284 +0,0 @@ -load( - "//lingvo:lingvo.bzl", - "lingvo_proto_cc", - "lingvo_proto_py", -) - -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -py_library( - name = "layers", - srcs = ["layers.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:layers", - "//lingvo/core:layers_with_attention", - ], -) - -py_test( - name = "layers_test", - srcs = ["layers_test.py"], - python_version = "PY3", - shard_count = 2, - srcs_version = "PY3", - deps = [":layers_test_lib"], -) - -py_library( - name = "layers_test_lib", - testonly = 1, - srcs = ["layers_test.py"], - srcs_version = "PY3", - deps = [ - ":layers", - "//lingvo:compat", - "//lingvo/core:layers_with_attention", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "encoder", - srcs = ["encoder.py"], - srcs_version = "PY3", - deps = [ - ":layers", - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:batch_major_attention", - "//lingvo/core:layers", - "//lingvo/core:model_helper", - "//lingvo/core:py_utils", - "//lingvo/core:rnn_cell", - "//lingvo/core:summary_utils", - ], -) - -py_test( - name = "encoder_test", - srcs = ["encoder_test.py"], - python_version = "PY3", - shard_count = 12, - srcs_version = "PY3", - deps = [ - ":encoder", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:self_attention_layer", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "decoder", - srcs = ["decoder.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:attention", - "//lingvo/core:base_decoder", - "//lingvo/core:batch_major_attention", - "//lingvo/core:layers", - "//lingvo/core:layers_with_attention", - "//lingvo/core:model_helper", - "//lingvo/core:plot", - "//lingvo/core:py_utils", - "//lingvo/core:quant_utils", - "//lingvo/core:rnn_cell", - "//lingvo/core:rnn_layers", - "//lingvo/core:summary_utils", - ], -) - -py_test( - name = "decoder_test", - srcs = ["decoder_test.py"], - python_version = "PY3", - shard_count = 20, - srcs_version = "PY3", - deps = [":decoder_test_lib"], -) - -py_library( - name = "decoder_test_lib", - testonly = 1, - srcs = ["decoder_test.py"], - srcs_version = "PY3", - deps = [ - ":decoder", - # Implicit absl.testing.parameterized dependency. - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:input_generator_helper", - "//lingvo/core:layers", - "//lingvo/core:layers_with_attention", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - "//lingvo/core/ops:hyps_py_pb2", - # Implicit numpy dependency. - ], -) - -py_library( - name = "data_augmenter", - srcs = ["data_augmenter.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:py_utils", - "//lingvo/core/ops", - # Implicit numpy dependency. - ], -) - -py_test( - name = "data_augmenter_test", - srcs = ["data_augmenter_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":data_augmenter", - "//lingvo:compat", - "//lingvo/core:base_layer", - "//lingvo/core:py_utils", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -py_library( - name = "input_generator", - srcs = ["input_generator.py"], - srcs_version = "PY3", - deps = [ - ":text_input_py_pb2", - "//lingvo:compat", - "//lingvo/core:base_input_generator", - "//lingvo/core:generic_input", - "//lingvo/core:py_utils", - "//lingvo/core:summary_utils", - "//lingvo/core:tokenizers", - "//lingvo/core/ops", - ], -) - -py_test( - name = "input_generator_test", - srcs = ["input_generator_test.py"], - data = [ - ":wpm_ende", - "//lingvo/tasks/mt/testdata:input_test_data", - "//lingvo/tasks/mt/testdata:mlperf_tfexample", - "//lingvo/tasks/mt/testdata:wmt14_ende_tfexample", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":data_augmenter", - ":input_generator", - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/core:test_helper", - "//lingvo/core:test_utils", - "//lingvo/core:tokenizers", - # Implicit numpy dependency. - ], -) - -py_library( - name = "model", - srcs = ["model.py"], - srcs_version = "PY3", - deps = [ - ":decoder", - ":encoder", - "//lingvo:compat", - "//lingvo/core:base_model", - "//lingvo/core:insertion", - "//lingvo/core:metrics", - "//lingvo/core:py_utils", - ], -) - -py_test( - name = "model_test", - size = "large", - srcs = ["model_test.py"], - python_version = "PY3", - shard_count = 20, - srcs_version = "PY3", - deps = [":model_test_lib"], -) - -py_library( - name = "model_test_lib", - testonly = 1, - srcs = ["model_test.py"], - data = [ - "//lingvo/tasks/mt/testdata:wmt14_ende_tfexample", - ], - srcs_version = "PY3", - deps = [ - ":decoder", - ":encoder", - ":input_generator", - ":model", - "//lingvo:compat", - "//lingvo/core:base_input_generator", - "//lingvo/core:base_layer", - "//lingvo/core:cluster_factory", - "//lingvo/core:optimizer", - "//lingvo/core:py_utils", - "//lingvo/core:schedule", - "//lingvo/core:test_helper", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) - -filegroup( - name = "wpm_ende", - srcs = glob(include = [ - "wpm-ende*.voc", - ]), -) - -py_library( - name = "base_config", - srcs = ["base_config.py"], - srcs_version = "PY3", - deps = [ - ":decoder", - ":encoder", - ":input_generator", - "//lingvo/core:attention", - "//lingvo/core:layers", - "//lingvo/core:optimizer", - "//lingvo/core:py_utils", - "//lingvo/core:rnn_cell", - "//lingvo/core:rnn_layers", - "//lingvo/core:schedule", - ], -) - -lingvo_proto_cc( - name = "text_input_proto", - src = "text_input.proto", -) - -lingvo_proto_py( - name = "text_input_py_pb2", - src = "text_input.proto", - deps = [":text_input_proto"], -) diff --git a/lingvo/tasks/mt/README.md b/lingvo/tasks/mt/README.md deleted file mode 100644 index 0619d678..00000000 --- a/lingvo/tasks/mt/README.md +++ /dev/null @@ -1,322 +0,0 @@ -# Local machine translation quick start - -To build a local translation model, we provide the tools to run a WMT16 -Multimodal EN-DE experiment. This dataset consists of translated image captions -(we are ignoring the associated images). The vocabulary is limited and sentences -are short, meaning you can build a working system quickly, using only CPUs. - -## Download and prepare the data - -We provide scripts to download the data. These scripts require the repository to -be git cloned and assumes the lingvo pip package is installed. - -```shell -cd lingvo/tasks/mt/tools -./wmtm16_get_data.sh -``` - -The master script above calls scripts for individual steps. If any step fails, -it can be re-run by hand. Similar steps are described in individual detail under -the Full Machine translation task below. The entire process should complete in -fewer than 10 minutes. - -In total, `250MB` of free space are required. The location is configured in -`lingvo/tasks/mt/tools/wmtm16_lib.sh`, defaulting to -`/tmp/wmtm16`. The final output is in -`/tmp/wmtm16/wpm`, stored in `tfrecord` format, which is described -in detail under the Full Machine translation task. - -To install aria2c, run: - -```shell -sudo apt-get install aria2 -``` - -## Training a toy system - -MT training is most naturally run on a cluster with many parallel processes, but -for this small task, we can run locally, taking advantage of multiple CPUs on a -reasonable development machine. - -```shell -mkdir /tmp/wmtm16/log - -python3 -m lingvo.trainer \ - --run_locally=cpu --mode=sync --saver_max_to_keep=3 \ - --logdir=/tmp/wmtm16/log \ - --job=controller,trainer_client \ - --model=mt.wmtm16_en_de.WmtCaptionEnDeTransformer \ - --logtostderr >& /tmp/wmtm16/log/train.log -``` - -This will begin training a model, providing statistics on training perplexity -(log_pplx) as training runs. Typically, you will also want to monitor your -progress on a held-out set. Background the above process or open a new terminal -to start two other processes to monitor our progress, one to track the -perplexity assigned to reference translations: - -```shell -python3 -m lingvo.trainer \ - --run_locally=cpu --mode=sync --saver_max_to_keep=3 \ - --logdir=/tmp/wmtm16/log \ - --job=evaler_Dev \ - --model=mt.wmtm16_en_de.WmtCaptionEnDeTransformer \ - --logtostderr >& /tmp/wmtm16/log/eval_dev.log -``` - -And one to report the accuracy of the decoder's output when compared to those -reference translations: - -```shell -python3 -m lingvo.trainer \ - --run_locally=cpu --mode=sync --saver_max_to_keep=3 \ - --logdir=/tmp/wmtm16/log \ - --job=decoder_Dev \ - --model=mt.wmtm16_en_de.WmtCaptionEnDeTransformer \ - --logtostderr >& /tmp/wmtm16/log/decode_dev.log -``` - -Note that we give the same log directory for both of these processes. The -monitors use the trainer's checkpoints, saved in the log directory, to load -models and compute scores. - -Example translation outputs appear in -`/tmp/wmtm16/log/decode_dev.log`. To track development BLEU and -perplexity, run tensorboard on `/tmp/wmtm16/log`. Training should -achieve well over 30 BLEU in fewer than 10,000 steps. - -# Full Machine translation task - -We provide the tools to run the WMT14 EN-DE setup. The system is described in -[The Best of Both Worlds: Combining Recent Advances in Neural Machine -Translation](http://aclweb.org/anthology/P18-1008) - -## Downloading the data - -We provide scripts to download the data. These scripts require the repository to -be git cloned and assumes the lingvo pip package is installed. - -```shell -cd lingvo/tasks/mt/tools -./wmt14_get_data.sh -``` - -The master script above calls scripts for individual steps. If any step fails, -it can be re-run by hand. - -In total, `13GB` of free space are required. The location is configured in -`lingvo/tasks/mt/tools/wmt14_lib.sh`, defaulting to -`/tmp/wmt14`. - -The final output is in `/tmp/wmt14/wpm`. It comes in the form of -`tf.Example` protos in `tfrecord` format. The examples contain the following -features: - -* `source_id`: An `int64` tensor of length `S`, with wordpiece IDs, always - terminated with `2`, the ID for ``. -* `source_padding`: A `float` tensor of zeros, of length `S`. -* `source_word`: A `string` tensor of wordpieces, always ending in ``, but - not starting with ``. -* `target_id`: An `int64` tensor of length `T` with wordpiece IDs, always - starting with `1`, the ID for ``, but not terminated with ``. -* `target_padding`: A `float` tensor of zeros, of length `T`. -* `target_word`: A `string` tensor of wordpiece, always starting with `` - but not ending in ``. -* `target_label`: An `int64` tensor of length `T`, with wordpiece IDs, like - `target_id`, but, instead of starting with ``, is terminated with ``. -* `target_weight`: A `float` tensor of ones, of length `T`. -* `natural_order`: An `int64` which is always `1`. - -Above, `S` is the length of the source (EN), in word pieces, including the -trailing ``. Similarly, `T` is the length of the target (DE), in word -pieces, including either the beginning ``, or the trailing ``. - -Both source and target share the same wordpiece inventory. - -### Downloading the Moses scripts - -The first step, `wmt14.01.download_moses_scripts.sh`, obtains a copy of the -moses decoder from github. In particular, the tokenizer, cleaning, and SGM -conversion scripts will be used. - -### Downloading the training data - -On a typical 100Mbit connection, the second step, `wmt14.02.download_train.sh`, -will take roughly half an hour to complete. It will download the following into -the `${ROOT}/raw` directory: - -* `training-parallel-europarl-v7.tgz`: `628M` -* `training-parallel-commoncrawl.tgz`: `876M` -* `training-parallel-nc-v9.tgz`: `77M` - -### Downloading the dev set - -The step `wmt14.03.download_devtest.sh` will result in `${ROOT}/raw`: - -* `dev.tgz`: `17M` -* `test-filtered.tgz`: `3.2M` - -### Unpacking the data - -The steps `wmt14.04.unpack_train.sh` and `wmt14.05.unpack_devtest.sh` unpack the -train data into `${ROOT}/unpacked`. Note that we will add SGM-converted test -files into the `${ROOT}/unpacked` directory later. - -### Tokenizing the data - -The steps `wmt14.06.tokenize_train.sh` and `wmt14.07.tokenize_devtest.sh` -tokenize the training data, into `${ROOT}/tokenized/{train,dev,test}`. - -For test, an additional step of converting the SGM is done prior to -tokenization. For train data, a cleaning step is done after tokenization. The -following files are going to be used as input to the next stage: - -* dev/newstest2013.de: `391K`, 3000 lines -* dev/newstest2013.en: `342K`, 3000 lines -* test/newstest2014.de: `369K`, 2737 lines -* test/newstest2014.en: `337K`, 2737 lines -* train/commoncrawl.clean.de: `329M`, `2.4M` lines -* train/commoncrawl.clean.en: `305M`, `2.4M` lines -* train/europarl-v7.clean.de: `315M`, `1.9M` lines -* train/europarl-v7.clean.en: `276M`, `1.9M` lines -* train/news-commentary-v9.clean.de: `33M`, `2.4M` lines -* train/news-commentary-v9.clean.en: `28M`, `2.4M` lines - -### Wordpiece model encoding - -The next steps, `wmt14.08.wpm_encode_train.sh` and -`wmt14.09.wpm_encode_devtest.sh`, encode the traiing data with the wordpiece -model, as well as transcoding to the final format, to wit `tf.Example` in -`tfrecord` format. - -Conversion of the training data will take about one hour on a relatively recent -machine. - -## Setting up your cluster - -**NOTE: This section is outdated and may not work as documented.** - -We provide a script running a docker fleet as an debugging example -(`lingvo/docker/run_distributed.py`). Please take a look at the -script to see the cluster configuration. - -### Shared filesystem - -You will need a cluster with a distributed filesystem, such as `HDFS`, `NFS`, -`sshfs`, or Google [filestore](https://cloud.google.com/filestore/). The -filesystem will be used to store the checkpoints and the trainer binary. - -The shared filesystem must mount the data and the log directory, possibly in two -different directories. - -### Cluster configuration - -For each cluster node, you will need to designate a role and a port number. It -is assumed that nodes in the cluster may occasionally crash or be replaced at -any time. However, when they come back, they must remain under the same -hostname, answering to the same port. - -We will train in synchronous mode. For training, you will need the following -nodes: - -* `trainer_client`: (singleton) controls the distributed workers. -* `controller`: (singleton) saves the checkpoints. -* `worker`: (8 replicas) runs the actual training. - -To continuous evaluation, you will need to add: - -* `decoder_`: runs the beam search on the dev or eval set. -* `evaler_`: runs the BLEU scoring. - -#### Cluster spec - -For training, processes need to be aware of each other. This is done by -specifying a cluster spec string. Let us assume the following cluster spec: - -```python -{ - "worker": [ - "wk0.example.com:43222", - "wk1.example.com:43222", - "wk2.example.com:43222", - ], - "controller": ["cont.example.com:43214",], - "trainer_client": ["tc.example.com:24601"], -} -``` - -The `trainer_client` is running on a machine under hostname `tc.example.com`, -with an available port `24601`, and each worker is on a machine with hostname -`wk0.example.com`, `wk1.example.com`, and `wk2.example.com` respectively. There -is no naming convention required for the hostnames. The cluster spec is -specified as follows: - -``` -cluster_spec := ( '@', )* -role_spec := (, '=', , (',' )* ) -machine_spec := (, ':', ) -``` - -In the example above, the role spec for the `worker` role would be: - -``` -worker=wk0.example.com:43222,wk1.example.com:43222,wk2.example.com:43222 -``` - -and the final cluster spec would be: - -``` -worker=wk0.example.com:43222,wk1.example.com:43222,wk2.example.com:43222@\ -controler=cont.example.com:43214@\ -trainer_client=tc.example.com:24601 -``` - -For decoding, each process has its own cluster spec, for instance, -`decoder_test=dec.example.com:3984`. - -### Uploading the binary - -**NOTE: This section is outdated and needs to be updated to use the pip package -instead.** - -There is a single binary which can be obtained as follows: - -```shell -bazel build -c opt lingvo:trainer.par -``` - -You may then copy the binary to the shared filesystem. Henceforth, the trainer -will be invoked using `trainer.par --...` from the unpacked directory. The -`trainer.par` binary is a self-contained executable in a zip format. - -All cluster nodes must have tensorflow installed, with the same version as the -one that was used to build lingvo. - -### Uploading the data - -You will need to upload the data files to the shared filesystem. The location -must match the one specified in -`lingvo/tasks/mt/params/wmt14_en_de.py` as `DATADIR`. - -## Running the model - -You are now ready to run the model. - -For each of the roles (`trainer_client`, `controller`, `decoder_dev`, etc), you -must run the trainer with the proper cluster spec: - -```shell -trainer.par --cluster_spec= \ - --model=mt.wmt14_en_de.WmtEnDeTransformerSmall \ - --job= --task= --mode=sync --logtostderr \ - --logdir= -``` - -as described above. - -## Examining results - -To examine the results, it is best to run a tensorboard process on the log -directory. - - diff --git a/lingvo/tasks/mt/__init__.py b/lingvo/tasks/mt/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/lingvo/tasks/mt/base_config.py b/lingvo/tasks/mt/base_config.py deleted file mode 100644 index 4cbcdeaf..00000000 --- a/lingvo/tasks/mt/base_config.py +++ /dev/null @@ -1,509 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Several functions to initialize typical values of dataset parameters.""" - -import math - -from lingvo.core import attention -from lingvo.core import layers -from lingvo.core import optimizer -from lingvo.core import py_utils -from lingvo.core import rnn_cell -from lingvo.core import rnn_layers -from lingvo.core import schedule -from lingvo.tasks.mt import decoder -from lingvo.tasks.mt import encoder -from lingvo.tasks.mt import input_generator - - -def InitTrainDatasetParams(vocab_size=None, params=None): - """Initializes typical values for train datasets. - - Args: - vocab_size: the number of tokens in your vocabulary. The default is None - because this parameter is often not used. - params: initial Params value, e.g. `NmtInput.Params()`. - - Returns: - a `Params` object. - """ - if params is None: - params = input_generator.NmtInput.Params() - params.is_nmt_example = True - - params.file_random_seed = 0 - - # How many threads to run in parallel. - params.file_parallelism = 16 - - # Note, for training, we prefer to use big file_buffer_size (as long as all - # fits in RAM), to more thoroughly randomize the training examples. when the - # file_buffer_size too small, we run the risk of sequentially going over the - # example as they are stored in the sstable which may not be random (e.g. - # maybe alphabetically ordered). - params.file_buffer_size = 10000000 - - if vocab_size is not None: - params.tokenizer.vocab_size = vocab_size - - # The bucket upper bound is determined based on an exponentially growing - # scheme, with _GenerateBuckets(10, 100) resulting buckets starting from - # minimum bucket size of 10 to maximum bucket size of 137. - # For word and sub-word level NMT, we train on sequences up to maximum - # bucket size and discard the examples that are longer than 137. - params.bucket_upper_bound = [10, 14, 19, 26, 36, 50, 70, 98, 137] - - # The bucket batch limit determines how many examples are there in each - # batch during training. We reduce the batch size for the buckets that - # have higher upper bound (batches that consist of longer sequences eg. - # 98, 137) in order to prevent out of memory issues. - # Note that this hyperparameter varies widely based on the model and language. - # larger models may warrant smaller batches in order to fit in memory, for - # example; and ideographical languages like Chinese may benefit from more - # buckets. - params.bucket_batch_limit = [128] * 8 + [64] - return params - - -def InitTestDatasetParams(vocab_size=None, params=None): - """Initializes typical values for test and dev datasets. - - Args: - vocab_size: the number of tokens in your vocabulary. - params: initial Params value, e.g. `NmtInput.Params()`. - - Returns: - a `Params` object. - """ - - if params is None: - params = input_generator.NmtInput.Params() - - params.file_random_seed = 27182818 - - # How many threads to run in parallel. - params.file_parallelism = 1 - - # In order to make exactly one pass over the dev/test sets, we set buffer - # size to 1. Greater numbers may cause inaccurate dev/test scores. - params.file_buffer_size = 1 - - if vocab_size is not None: - params.tokenizer.vocab_size = vocab_size - - # The largest bucket upper bound must be larger than the longest sequence - # length in dev/test set. Since we discard sequences longer than the - # max(bucket_upper_bound) we may end up having scores based on only shorter - # sequences only if we mistakenly set this to be too small. - params.bucket_upper_bound = [10, 14, 19, 26, 36, 50, 70, 98, 137, 200] - params.bucket_batch_limit = [128] * 8 + [64] + [32] - return params - - -def InitTransformerTestBuckets(params): - params.bucket_upper_bound = [10, 14, 19, 26, 36, 50, 70, 98, 137, 200] - params.bucket_batch_limit = [16] * 10 - return params - - -def InitTransformerTrainBuckets(params): - params.bucket_upper_bound = [8, 12, 16, 24, 32, 48, 64, 96] - params.bucket_batch_limit = [512, 341, 256, 170, 128, 85, 64, 42] - return params - - -def SetupTransformerParams(p, - name, - vocab_size, - model_dim, - hidden_dim, - num_heads, - num_layers, - learning_rate, - warmup_steps, - residual_dropout_prob=0.1, - input_dropout_prob=0.0, - atten_dropout_prob=0.0, - relu_dropout_prob=0.0, - label_smoothing_uncertainty=0.1, - is_transparent=False, - activation='RELU', - add_unnormalized_residuals=False, - atten_hidden_dim=0, - num_encoder_layers=None, - num_decoder_layers=None): - """Common model setup for different transformer models. - - Args: - p: The initial params object to modify. - name: An identifier for an instance of a transformer model. - vocab_size: an integer representing the size of the vocabulary, probably - 16000 or 32000. - model_dim: dimension of the transformer block (column) - hidden_dim: dimension of Feed-Forward neural network in each layer - num_heads: number of attention heads to use for the transformer - num_layers: number of layers in the transformer - learning_rate: learning rate for Adam. For the base model, we use 1.0; for - the big model, 3.0 - warmup_steps: warmup steps for `TransformerSchedule`. For the - base model, we use 4000; for the big model, 40000 - residual_dropout_prob: dropout prob to the output of each sub-layer before - it is added to the sub-layer input - input_dropout_prob: dropout prob to the sums of the token embeddings and the - position embeddings - atten_dropout_prob: dropout prob to the attention weights in each - Transformer attention sub-layer - relu_dropout_prob: dropout prob to the inner layer output (ReLU activation) - in each Transformer feed-forward sub-layer - label_smoothing_uncertainty: if this value is 0, no label smoothing will be - applied - is_transparent: If set, decoder layers attend to weighted combinations of - encoder layers. - activation: Non-linearity for feed-forward layers. - add_unnormalized_residuals: If set, uses un-normalized residuals in - TransformerAttentionLayer - atten_hidden_dim: Explicitly set attention hidden dim. - num_encoder_layers: to set a different number of layers for the encoder. - num_decoder_layers: to set a different number of layers for the decoder. - - Returns: - A Params object containing the parameters that specify a transformer model - (Vaswani 2017) - - """ - p.name = name - - # Transformer encoder and decoder setup - num_encoder_layers = num_encoder_layers or num_layers - num_decoder_layers = num_decoder_layers or num_layers - p.encoder = SetupTransformerEncoder( - model_dim, vocab_size, num_encoder_layers, num_heads, hidden_dim, - residual_dropout_prob, input_dropout_prob, atten_dropout_prob, - relu_dropout_prob, is_transparent, activation, add_unnormalized_residuals, - atten_hidden_dim) - p.decoder = SetupTransformerDecoder( - model_dim, vocab_size, num_decoder_layers, num_heads, hidden_dim, - residual_dropout_prob, input_dropout_prob, atten_dropout_prob, - relu_dropout_prob, label_smoothing_uncertainty, is_transparent, - activation, add_unnormalized_residuals, atten_hidden_dim) - - p.train.Set( - learning_rate=learning_rate, - optimizer=optimizer.Adam.ParamsB(), - clip_gradient_norm_to_value=0.0, - grad_norm_to_clip_to_zero=0.0, - lr_schedule=schedule.TransformerSchedule.Params().Set( - warmup_steps=warmup_steps, worker_replicas=1, model_dim=model_dim)) - - p.eval.samples_per_summary = 12000 - return p - - -def SetupTransformerDecoder(model_dim, - vocab_size, - num_layers, - num_heads, - hidden_dim, - residual_dropout_prob=0.1, - input_dropout_prob=0.0, - atten_dropout_prob=0.0, - relu_dropout_prob=0.0, - label_smoothing_uncertainty=0.1, - is_transparent=False, - activation='RELU', - add_unnormalized_residuals=False, - atten_hidden_dim=0): - """Common setup for transformer model decoder.""" - disable_vn = py_utils.VariationalNoiseParams(1.0, False, False) - default_params_init = py_utils.WeightInit.Xavier(1.0) - emb_params_init = py_utils.WeightInit.Gaussian(1.0 / math.sqrt(model_dim)) - - # Decoder - decoder_params = decoder.TransformerDecoder.Params() - decoder_params.source_dim = model_dim - decoder_params.model_dim = model_dim - decoder_params.num_trans_layers = num_layers - decoder_params.input_dropout_prob = input_dropout_prob - - decoder_params.token_emb.Set( - vocab_size=vocab_size, - embedding_dim=model_dim, - max_num_shards=16, - params_init=emb_params_init, - vn=disable_vn, - scale_sqrt_depth=True) - - decoder_params.position_emb.Set( - embedding_dim=model_dim, trainable_scaling=False, vn=disable_vn) - - decoder_params.trans_tpl.source_dim = model_dim - decoder_params.trans_tpl.tr_atten_tpl.Set( - source_dim=model_dim, - num_attention_heads=num_heads, - residual_dropout_prob=residual_dropout_prob, - atten_dropout_prob=atten_dropout_prob, - params_init=default_params_init, - add_unnormalized_input=add_unnormalized_residuals, - atten_hidden_dim=atten_hidden_dim, - vn=disable_vn) - - decoder_params.trans_tpl.tr_atten_tpl.atten_tpl.Set( - enable_ctx_pre_proj=True, - enable_ctx_post_proj=True, - context_dim=model_dim, - vn=disable_vn) - - decoder_params.trans_tpl.tr_fflayer_tpl.Set( - input_dim=model_dim, - hidden_dim=hidden_dim, - residual_dropout_prob=residual_dropout_prob, - relu_dropout_prob=relu_dropout_prob, - params_init=default_params_init, - vn=disable_vn, - activation=activation) - - decoder_params.softmax.Set( - num_classes=vocab_size, - vn=disable_vn, - params_init=emb_params_init, - num_shards=16) - - decoder_params.per_word_avg_loss = True - decoder_params.label_smoothing = layers.UniformLabelSmoother.Params() - decoder_params.label_smoothing.num_classes = vocab_size - decoder_params.label_smoothing.uncertainty = label_smoothing_uncertainty - - if is_transparent: - decoder_params.is_transparent = True - - return decoder_params - - -def SetupTransformerEncoder(model_dim, - vocab_size, - num_layers, - num_heads, - hidden_dim, - residual_dropout_prob=0.1, - input_dropout_prob=0.0, - atten_dropout_prob=0.0, - relu_dropout_prob=0.0, - is_transparent=False, - activation='RELU', - add_unnormalized_residuals=False, - atten_hidden_dim=0): - """Common setup for transformer model encoder. - - Args: - model_dim: specifies dimension of transformer layers, token embeddings, - and positional embeddings as well context vectors (attention values). - vocab_size: for token embeddings. - num_layers: number of transformer layers. - num_heads: number of attention heads. - hidden_dim: in transformer feedforward layer. - residual_dropout_prob: used in transformer feedforward and attention layer. - input_dropout_prob: input dropout. - atten_dropout_prob: used in attention layer. - relu_dropout_prob: used in transformer feedforward layer. - is_transparent: if set, outputs a merger of embeddings and layer outputs. - activation: Non-linearity for feed-forward layers. - add_unnormalized_residuals: If set, uses un-normalized residuals in - TransformerAttentionLayer - atten_hidden_dim: Explicitly set attention hidden dim. - - Returns: - Encoder params. - """ - disable_vn = py_utils.VariationalNoiseParams(1.0, False, False) - default_params_init = py_utils.WeightInit.Xavier(1.0) - emb_params_init = py_utils.WeightInit.Gaussian(1.0 / math.sqrt(model_dim)) - - # Encoder - encoder_params = encoder.TransformerEncoder.Params() - - encoder_params.token_emb.Set( - embedding_dim=model_dim, - max_num_shards=16, - params_init=emb_params_init, - vocab_size=vocab_size, - vn=disable_vn, - scale_sqrt_depth=True) - - encoder_params.position_emb.Set( - embedding_dim=model_dim, trainable_scaling=False, vn=disable_vn) - - # Encoder TransformerStack params - encoder_params.model_dim = model_dim - encoder_params.transformer_stack.model_dim = model_dim - encoder_params.transformer_stack.num_transformer_layers = num_layers - encoder_params.input_dropout_prob = input_dropout_prob - - encoder_params.transformer_stack.transformer_tpl.tr_atten_tpl.Set( - num_attention_heads=num_heads, - residual_dropout_prob=residual_dropout_prob, - atten_dropout_prob=atten_dropout_prob, - params_init=default_params_init, - add_unnormalized_input=add_unnormalized_residuals, - atten_hidden_dim=atten_hidden_dim, - vn=disable_vn) - - encoder_params.transformer_stack.transformer_tpl.tr_atten_tpl.atten_tpl.Set( - num_attention_heads=num_heads, - enable_ctx_pre_proj=True, - enable_ctx_post_proj=True, - context_dim=model_dim, - vn=disable_vn) - - encoder_params.transformer_stack.transformer_tpl.tr_fflayer_tpl.Set( - hidden_dim=hidden_dim, - residual_dropout_prob=residual_dropout_prob, - relu_dropout_prob=relu_dropout_prob, - params_init=default_params_init, - vn=disable_vn, - activation=activation) - - if is_transparent: - encoder_params.transformer_stack.is_transparent = True - - return encoder_params - - -def SetupRNMTParams(p, - name, - vocab_size, - embedding_dim, - hidden_dim, - num_heads, - num_encoder_layers, - num_decoder_layers, - learning_rate, - lr_warmup_steps, - lr_decay_start, - lr_decay_end, - lr_min, - atten_dropout_prob, - residual_dropout_prob, - ls_uncertainty, - l2_regularizer_weight, - is_transparent=False, - num_hyps_per_beam=16, - adam_beta1=0.9, - adam_beta2=0.999, - adam_epsilon=8e-07): - """Creates RNMT+ params common to all datasets. - - Args: - p: The initial params object to modify. - name: A descriptive name for your model. - vocab_size: size of the vocabulary. Probably 32000 or 16000. - embedding_dim: Dimension of token embeddings. - hidden_dim: LSTM cell size. - num_heads: number of attention heads. - num_encoder_layers: Number of layers in the encoder. - num_decoder_layers: Number of layers in the decoder. - learning_rate: Optimizer learning rate. - lr_warmup_steps: Warm-up steps for the optimizer. - lr_decay_start: Learning rate exponential decay starting step. - lr_decay_end: Learning rate exponential decay end step. - lr_min: Minimum learning rate (ratio with initial learning rate). - atten_dropout_prob: Dropout for the attention. - residual_dropout_prob: Dropout for residual layers. - ls_uncertainty: Label smoothing uncertainty. - l2_regularizer_weight: Weight for l2 regularization on parameters. - is_transparent: If set, decoder attends to weighted combination of encoder - layers. - num_hyps_per_beam: Number of hyps to keep per source sequence. - adam_beta1: Beta-1 parameter of Adam optimizer. - adam_beta2: Beta-2 parameter of Adam optimizer. - adam_epsilon: Epsilon parameter of Adam optimizer. - - Returns: - a Params() object specifying the RNMT+ Parameters. - """ - - # TODO(orhanf): add transparent connections. - del is_transparent - - p.name = name - - default_params_init = py_utils.WeightInit.Uniform(0.04) - rnn_cell_tpl = rnn_cell.LayerNormalizedLSTMCellSimple.Params().Set( - num_output_nodes=hidden_dim, - output_nonlinearity=False, - params_init=default_params_init) - - # RNMT+ encoder setup. - p.encoder = encoder.MTEncoderBiRNN.Params().Set( - num_lstm_layers=num_encoder_layers, - lstm_cell_size=hidden_dim, - encoder_out_dim=hidden_dim, - lstm_tpl=rnn_cell_tpl.Copy(), - dropout_prob=residual_dropout_prob) - p.encoder.emb.embedding_dim = embedding_dim - p.encoder.emb.vocab_size = vocab_size - - # RNMT+ decoder setup. - p.decoder = decoder.MTDecoderV1.Params().Set( - rnn_layers=num_decoder_layers, - rnn_cell_tpl=rnn_cell_tpl.Copy(), - atten_rnn_cell_tpl=rnn_cell_tpl.Copy(), - dropout_prob=residual_dropout_prob, - attention=attention.MultiHeadedAttention.Params().Set( - source_dim=hidden_dim, - hidden_dim=hidden_dim, - query_dim=hidden_dim, - context_dim=hidden_dim, - num_attention_heads=num_heads, - inner_atten_params=attention.AdditiveAttention.Params(), - use_source_vec_as_attention_value=True, - enable_ctx_pre_proj=False, - enable_query_proj=True, - atten_dropout_prob=atten_dropout_prob, - atten_dropout_deterministic=True), - atten_rnn_cls=rnn_layers.FRNNWithAttention, - feed_attention_context_vec_to_softmax=True, - label_smoothing=layers.UniformLabelSmoother.Params().Set( - num_classes=vocab_size, uncertainty=ls_uncertainty)) - p.decoder.emb.vocab_size = vocab_size - p.decoder.emb.embedding_dim = embedding_dim - p.decoder.softmax.num_classes = vocab_size - p.decoder.source_dim = hidden_dim - - # Inference related. - p.decoder.beam_search.num_hyps_per_beam = num_hyps_per_beam - - # Optimization setup. - learning_rate_schedule = ( - schedule.LinearRampupExponentialDecayScaledByNumSplitSchedule.Params() - .Set( - warmup=lr_warmup_steps, - decay_start=lr_decay_start, - decay_end=lr_decay_end, - min=lr_min)) - p.train.Set( - l2_regularizer_weight=l2_regularizer_weight, - grad_norm_tracker=layers.GradNormTracker.Params().Set( - name='gradient_norm_tracker'), - learning_rate=learning_rate, - lr_schedule=learning_rate_schedule, - grad_norm_to_clip_to_zero=100000.0, - optimizer=optimizer.Adam.Params().Set( - beta1=adam_beta1, beta2=adam_beta2, epsilon=adam_epsilon), - ) - - # Evaluation related - p.eval.samples_per_summary = 12000 - return p diff --git a/lingvo/tasks/mt/data_augmenter.py b/lingvo/tasks/mt/data_augmenter.py deleted file mode 100644 index 29a69d69..00000000 --- a/lingvo/tasks/mt/data_augmenter.py +++ /dev/null @@ -1,106 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Classes which modifies inputs for pretraining. - -Currently support: MASS -""" - -from lingvo.core import base_layer -from lingvo.core import ops -from lingvo.core import py_utils -import numpy as np - - -class MASS(base_layer.BaseLayer): - """Prepare data for MASS pretraining. - - This class is a wrapper for the MASS op. - """ - - @classmethod - def Params(cls): - # Params for the MASS op api. - p = super().Params() - p.Define('mask_id', 3, 'Id for the mask token.') - p.Define('mask_ratio', 0.5, 'Mask fraction.') - p.Define('mask_minlen', 0, 'Minimum number of tokens to mask.') - p.Define( - 'span_len', 100000, - 'Split total mask_len into segments of this size and randomly' - ' distribute them across the src sentence for segmented masking.') - p.Define( - 'random_start_prob', 0.6, - 'Probability that placement of masked segments will be entirely ' - 'random. Remaining cases are split evenly between masking at the ' - 'beginning and at the end.') - # keep_prob/rand_prob/mask_prob must sum to 1. - p.Define( - 'keep_prob', 0.1, - 'Probability that a token designated for masking will be ' - 'unchanged.') - p.Define( - 'rand_prob', 0.1, - 'Probability that a token designated for masking will be replaced ' - 'with a random token.') - p.Define( - 'mask_prob', 0.8, - 'Probability that a token designated for masking will be replaced ' - 'with mask_id.') - p.Define( - 'mask_target', True, - 'If true, the target is masked with the inverse of the source ' - 'mask.') - p.Define('vocab_size', 0, - 'Used when selecting a random token to replaced a masked token.') - p.Define( - 'first_unreserved_id', 4, - 'Tokens greater than or equal to this may be selected at random ' - 'to replace a masked token.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if not np.isclose([p.keep_prob + p.rand_prob + p.mask_prob], [1]): - raise ValueError('keep_prob, rand_prob, mask_prob must sum to 1') - if p.vocab_size == 0: - raise ValueError('vocab_size parameter must be set explicitly') - - def Mask(self, seq_ids, weights, actual_seq_len): - p = self.params - (src_ids, tgt_ids, tgt_labels, tgt_weights) = ops.mass( - seq_ids, - weights, - actual_seq_len, - mask_id=p.mask_id, - mask_ratio=p.mask_ratio, - mask_minlen=p.mask_minlen, - span_len=p.span_len, - random_start_prob=p.random_start_prob, - keep_prob=p.keep_prob, - rand_prob=p.rand_prob, - mask_prob=p.mask_prob, - mask_target=p.mask_target, - vocab_size=p.vocab_size, - first_unreserved_id=p.first_unreserved_id) - - mass_out = py_utils.NestedMap() - mass_out.src = py_utils.NestedMap() - mass_out.src.ids = src_ids - mass_out.tgt = py_utils.NestedMap() - mass_out.tgt.ids = tgt_ids - mass_out.tgt.labels = tgt_labels - mass_out.tgt.weights = tgt_weights - return mass_out diff --git a/lingvo/tasks/mt/data_augmenter_test.py b/lingvo/tasks/mt/data_augmenter_test.py deleted file mode 100644 index 8a1ce80b..00000000 --- a/lingvo/tasks/mt/data_augmenter_test.py +++ /dev/null @@ -1,65 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for data_augmenter.""" - -from lingvo import compat as tf -from lingvo.core import test_utils -from lingvo.tasks.mt import data_augmenter -import numpy as np - -FLAGS = tf.flags.FLAGS - - -class MassLayerTest(test_utils.TestCase): - - def _MassParams(self): - p = data_augmenter.MASS.Params() - p.mask_id = 3 - p.mask_ratio = 0.5 - p.mask_minlen = 0 - p.span_len = 3 - p.random_start_prob = 0.8 - p.keep_prob = 0 - p.rand_prob = 0 - p.mask_prob = 1 - p.mask_target = True - p.vocab_size = 64 - p.first_unreserved_id = 4 - p.name = "mass_layer" - return p - - def testMassLayer(self): - with self.session(use_gpu=False) as sess: - batch_size = 3 - seq_len = 10 - p = self._MassParams() - mass_layer = data_augmenter.MASS(p) - seq_ids = tf.fill([batch_size, seq_len], 4) - weights = tf.ones([batch_size, seq_len]) - actual_seq_len = tf.fill([batch_size], 10) - mass_out = mass_layer.Mask(seq_ids, weights, actual_seq_len) - (src_ids, tgt_ids, tgt_labels, tgt_weights) = sess.run([ - mass_out.src.ids, mass_out.tgt.ids, mass_out.tgt.labels, - mass_out.tgt.weights - ]) - self.assertAllEqual(np.sum(src_ids == 3, axis=1), [5, 5, 5]) - self.assertAllEqual(np.sum(tgt_ids == 3, axis=1), [5, 5, 5]) - self.assertAllEqual(tgt_labels, - 4 * np.ones([batch_size, seq_len], dtype=np.int32)) - self.assertAllEqual(np.sum(tgt_weights, axis=1), [5., 5., 5.]) - - -if __name__ == "__main__": - tf.test.main() diff --git a/lingvo/tasks/mt/decoder.py b/lingvo/tasks/mt/decoder.py deleted file mode 100644 index dbb2c1e7..00000000 --- a/lingvo/tasks/mt/decoder.py +++ /dev/null @@ -1,2708 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Machine translation decoder. -""" - -import math -import lingvo.compat as tf -from lingvo.core import attention -from lingvo.core import base_decoder -from lingvo.core import batch_major_attention -from lingvo.core import layers -from lingvo.core import layers_with_attention -from lingvo.core import model_helper -from lingvo.core import plot -from lingvo.core import py_utils -from lingvo.core import quant_utils -from lingvo.core import rnn_cell -from lingvo.core import rnn_layers -from lingvo.core import summary_utils - - -class MTBaseDecoder(base_decoder.BaseBeamSearchDecoder): - """Base class for Lingvo MT decoders.""" - - @classmethod - def Params(cls): - p = super().Params() - p.Define('label_smoothing', None, 'Label smoothing class.') - p.Define('softmax', layers.SimpleFullSoftmax.Params(), 'Softmax params.') - p.Define( - 'per_word_avg_loss', False, 'Compute loss averaged per word. If False ' - 'loss is computed averaged per sequence.') - p.Define('unidi_rnn_type', 'func', 'Options: func, native_cudnn. ' - 'func: FRNN, native_cudnn: CuDNNLSTM.') - p.Define('feed_attention_context_vec_to_softmax', False, - 'Whether to concatenate attention context vector to rnn output' - ' before softmax.') - p.Define('per_example_tensors', False, 'Return per example tensors') - - # Default config for the softmax part. - p.softmax.num_classes = 32000 # 32k - p.softmax.num_shards = 8 - - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.label_smoothing is not None: - p.label_smoothing.name = 'smoother' - p.label_smoothing.num_classes = p.softmax.num_classes - self.CreateChild('smoother', p.label_smoothing) - - @classmethod - def UpdateTargetVocabSize(cls, p, vocab_size, wpm_model=None): - """Sets the params with the given vocab size and wpm model. - - Args: - p: model params. - vocab_size: size of the vocabulary. - wpm_model: file name prefix pointing to a wordpiece model. - - Returns: - Model params updated with the vocab size and wpm model. - """ - p.softmax.num_classes = vocab_size - return p - - def _ComputeXentLoss(self, - theta, - softmax_input, - target_labels, - target_weights, - target_paddings, - target_segment_ids=None, - time_axis=0): - """Computes cross-entropy loss given the softmax input, labels and weights. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - softmax_input: A tensor of shape [time, batch, p.softmax.input_dim]. - target_labels: A matrix of tf.int32. [time, batch]. - target_weights: A matrix of params.dtype. [time, batch]. - target_paddings: A matrix of params.dtype. [time, batch]. - target_segment_ids: A matrix of params.dtype. [time, batch]. - time_axis: If 0, the inputs are time-major: [time, batch, ...]; if 1, the - inputs are batch-major: [batch, time, ...]. - - Returns: - The cross entropy loss. - """ - p = self.params - softmax_input = tf.reshape(softmax_input, [-1, p.softmax.input_dim]) - - if p.label_smoothing is None: - xent_loss = self.softmax.FProp( - theta.softmax, [softmax_input], - class_weights=tf.reshape(target_weights, [-1, 1]), - class_ids=tf.reshape(target_labels, [-1, 1])) - else: - # [time, batch, num_classes] - if time_axis == 0: - target_probs = tf.transpose( - self.smoother.FProp( - theta.smoother, - tf.transpose(target_paddings), - tf.transpose(target_labels), - target_ids=None), [1, 0, 2]) - else: - target_probs = self.smoother.FProp( - theta.smoother, target_paddings, target_labels, target_ids=None) - xent_loss = self.softmax.FProp( - theta.softmax, [softmax_input], - class_weights=tf.reshape(target_weights, [-1, 1]), - class_probabilities=tf.reshape(target_probs, - [-1, p.softmax.num_classes])) - return xent_loss - - def _ComputeSoftmaxMetrics(self, - xent_loss, - target_labels, - target_weights, - target_segment_ids=None, - time_axis=0): - """Computes cross-entropy metrics given the cross-entropy loss. - - Args: - xent_loss: The output of `_ComputeXentLoss`. - target_labels: A matrix of tf.int32. [time, batch]. - target_weights: A matrix of params.dtype. [time, batch]. - target_segment_ids: A matrix of params.dtype. [time, batch]. - time_axis: If 0, the inputs are time-major: [time, batch, ...]; if 1, the - inputs are batch-major: [batch, time, ...]. - - Returns: - A tuple (metrics, per_example_tensors). - metrics: - A dictionary containing metrics for the xent loss and prediction - accuracy. - per_example_tensors: - A dictionary of per-example tensors. - """ - p = self.params - if p.per_word_avg_loss: - final_loss = tf.identity(xent_loss.avg_xent, name='loss') - loss_weight = tf.identity(xent_loss.total_weight, name='num_predictions') - else: - # NOTE: Per-sequence loss is the sum of each example's loss. The - # final loss for a training batch is the mean loss of sequences in - # the batch. - # [time, batch] - per_example_loss = tf.reshape(xent_loss.per_example_xent, - py_utils.GetShape(target_weights)) - per_sequence_loss = tf.reduce_sum( - per_example_loss * target_weights, axis=time_axis) - if p.packed_input: - assert target_segment_ids is not None, ( - 'Need target segment ids for ' - 'normalizing loss when training with packed inputs.') - num_samples = tf.cast( - tf.reduce_sum( - tf.reduce_max(target_segment_ids, 0) - - tf.reduce_min(target_segment_ids, 0) + 1), - dtype=per_sequence_loss.dtype) - final_loss = tf.reduce_sum(per_sequence_loss) / num_samples - else: - final_loss = tf.reduce_mean(per_sequence_loss) - loss_weight = py_utils.GetShape(per_sequence_loss)[0] - - metrics = { - 'loss': (final_loss, loss_weight), - 'log_pplx': (xent_loss.avg_xent, xent_loss.total_weight), - } - - per_example_tensors = {} - if p.per_example_tensors: - per_example_tensors['per_example_loss'] = tf.reshape( - xent_loss.per_example_xent, py_utils.GetShape(target_weights)) - per_example_tensors['per_sequence_loss'] = tf.reduce_sum( - per_example_tensors['per_example_loss'] * target_weights, - axis=time_axis) - per_example_tensors['loss'] = per_example_tensors['per_sequence_loss'] - per_example_tensors['logits'] = tf.reshape( - xent_loss.logits, - tf.concat([py_utils.GetShape(target_weights), [-1]], 0)) - per_example_tensors['log_probs'] = tf.reshape( - xent_loss.log_probs, - tf.concat([py_utils.GetShape(target_weights), [-1]], 0)) - - # NOTE: tf.argmax is not implemented for the JF backend, see b/36093673 - # Skip the fraction_of_correct_next_step_preds during training. - if self.do_eval: - logits = xent_loss.logits - correct_preds = tf.cast( - tf.equal( - tf.cast(tf.reshape(tf.argmax(logits, 1), [-1]), tf.int32), - tf.reshape(target_labels, [-1])), p.dtype) - correct_next_preds = tf.reduce_sum( - correct_preds * tf.reshape(tf.cast(target_weights, p.dtype), [-1])) - num_preds = tf.reduce_sum(tf.cast(target_weights, p.dtype)) - accuracy = tf.identity( - correct_next_preds / num_preds, - name='fraction_of_correct_next_step_preds') - metrics['fraction_of_correct_next_step_preds'] = (accuracy, num_preds) - return metrics, per_example_tensors - - def _FPropSoftmax(self, - theta, - softmax_input, - target_labels, - target_weights, - target_paddings, - target_segment_ids=None, - time_axis=0): - """Computes cross-entropy loss given the softmax input, labels and weights. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - softmax_input: A tensor of shape [time, batch, p.softmax.input_dim]. - target_labels: A matrix of tf.int32. [time, batch]. - target_weights: A matrix of params.dtype. [time, batch]. - target_paddings: A matrix of params.dtype. [time, batch]. - target_segment_ids: A matrix of params.dtype. [time, batch]. - time_axis: If 0, the inputs are time-major: [time, batch, ...]; if 1, the - inputs are batch-major: [batch, time, ...]. - - Returns: - A tuple (metrics, per_example_tensors). - metrics: - A dictionary containing metrics for the xent loss and prediction - accuracy. - per_example_tensors: - A dictionary of per-example tensors. - """ - xent_loss = self._ComputeXentLoss(theta, softmax_input, target_labels, - target_weights, target_paddings) - return self._ComputeSoftmaxMetrics(xent_loss, target_labels, target_weights, - target_segment_ids, time_axis=time_axis) - - def ComputeLoss(self, theta, predictions, targets): - """Populates a metrics dictionary based on the output of ComputePredictions. - - Args: - theta: Nested map describing decoder model parameters. - predictions: NestedMap describing the decoding process, requiring: - .softmax_input: Tensor of shape [time, batch, params.softmax.input_dim]. - targets: NestedMap describing the target sequences. - - Returns: - Two dicts. - - - A map from metric name (a python string) to a tuple (value, weight). - Both value and weight are scalar Tensors. - - A map from name to arbitrary tensors, where the first dimension must - be the batch index. - """ - segment_id = None - if self.params.packed_input: - segment_id = tf.transpose(targets.segment_ids) - if isinstance(predictions, py_utils.NestedMap): - predictions = predictions.softmax_input - return self._FPropSoftmax(theta, predictions, tf.transpose(targets.labels), - tf.transpose(targets.weights), - tf.transpose(targets.paddings), segment_id) - - def _TruncateTargetSequence(self, targets): - """Truncate padded time steps from all sequences.""" - # The following tensors are all in the [batch, time] shape. - # Let's make a copy of targets. - targets = targets.Pack(targets.Flatten()) - target_ids = targets.ids - target_labels = targets.labels - target_weights = targets.weights - target_paddings = targets.paddings - max_seq_length = tf.cast( - tf.round(tf.reduce_max(tf.reduce_sum(1.0 - target_paddings, 1))), - tf.int32) - summary_utils.scalar('max_seq_length', max_seq_length) - # Assert to make sure after max_seq_length, all are padded steps for all - # sequences. - target_paddings = py_utils.with_dependencies([ - py_utils.assert_equal( - tf.constant(True, tf.bool), - tf.reduce_all(target_paddings[:, max_seq_length:] > 0.5)) - ], target_paddings) - target_ids = py_utils.with_dependencies([ - py_utils.AssertIdShape( - py_utils.GetShape(target_ids), py_utils.GetShape(target_labels), - py_utils.GetShape(target_paddings), - py_utils.GetShape(target_weights)) - ], target_ids) - targets.ids = target_ids[:, :max_seq_length] - targets.labels = target_labels[:, :max_seq_length] - targets.weights = target_weights[:, :max_seq_length] - targets.paddings = target_paddings[:, :max_seq_length] - return targets - - def _AddAttenProbsSummary(self, source_paddings, targets, atten_probs): - """Add summary of attention probs. - - Args: - source_paddings: source padding, of shape [src_len, src_batch]. - targets: A dict of string to tensors representing the targets one try to - predict. Each tensor in targets is of shape [tgt_batch, tgt_len]. - atten_probs: a list of attention probs, each element is of shape [tgt_len, - tgt_batch, src_len]. - """ - if not self.cluster.add_summary: - return - - self._AddAttenProbsImageSummary(source_paddings, targets, atten_probs) - self._AddAttenProbsHistogramSummary(atten_probs) - - def _AddAttenProbsHistogramSummary(self, atten_probs): - """Add histogram summary of attention probs. - - Args: - atten_probs: a list of attention probs, each element is of shape [tgt_len, - tgt_batch, src_len]. - """ - for i, probs in enumerate(atten_probs): - # a prefix from the context will be used, which looks like - # fprop/wmt14_en_de_transformer/tower_0_0/dec/ - summary_utils.histogram('atten{}'.format(i + 1), probs) - - def _AddAttenProbsImageSummary(self, source_paddings, targets, atten_probs): - """Add image summary of attention probs. - - Args: - source_paddings: source padding, of shape [src_len, src_batch]. - targets: A dict of string to tensors representing the targets one try to - predict. Each tensor in targets is of shape [tgt_batch, tgt_len]. - atten_probs: a list of attention probs, each element is of shape [tgt_len, - tgt_batch, src_len]. - """ - def PlotAttention(fig, axes, cur_atten_probs, title, set_x_label): - plot.AddImage(fig, axes, cur_atten_probs, title=title) - axes.set_ylabel(plot.ToUnicode('Output sequence index'), wrap=True) - if set_x_label: - axes.set_xlabel(plot.ToUnicode('Input sequence index'), wrap=True) - - index = 0 - srclen = tf.cast( - tf.round(tf.reduce_sum(1 - source_paddings[:, index])), tf.int32) - tgtlen = tf.cast( - tf.round(tf.reduce_sum(1 - targets.paddings[index, :])), tf.int32) - - num_rows = len(atten_probs) - with plot.MatplotlibFigureSummary( - 'decoder_example', - figsize=(6, 3 * num_rows), - max_outputs=1, - subplot_grid_shape=(num_rows, 1)) as fig: - for i, probs in enumerate(atten_probs): - # Extract first entry in batch of attention prob matrices - # [tgt_len, src_len] - probs = probs[:, index, :] - probs = tf.expand_dims(probs[:tgtlen, :srclen], 0) - fig.AddSubplot([probs], - PlotAttention, - title='atten_probs_%d' % i, - set_x_label=(i == len(atten_probs) - 1)) - - def _ExpandToNumHyps(self, source_enc_len, num_hyps_per_beam): - """Repeat each value according to num hyps. - - Args: - source_enc_len: source encoder length; int [batch]. - num_hyps_per_beam: number of hypotheses - - Returns: - New version of source_enc_len; int [batch * num_hyps_per_beam]. - Target_batch is (num_hyps_per_beam * batch). - Example: src_enc_len = [3, 2, 1] and num_hyps_per_beam = 2 - --> [3, 2, 1, 3, 2, 1] - """ - x = tf.tile(input=source_enc_len, multiples=[num_hyps_per_beam]) - return x - - -class MTDecoderV1(MTBaseDecoder, quant_utils.QuantizableLayer): - """MT decoder v1.""" - - @classmethod - def Params(cls): - p = super().Params() - # Shared embedding. - p.Define('emb', layers.EmbeddingLayer.Params(), 'Embedding layer params.') - p.Define('source_dim', 1024, 'Dimension of the source encoding.') - p.Define('attention', attention.AdditiveAttention.Params(), - 'Additive attention params.') - p.Define('atten_rnn_cell_tpl', rnn_cell.LSTMCellSimple.Params(), - 'Attention RNNCell params template.') - p.Define('rnn_cell_tpl', rnn_cell.LSTMCellSimple.Params(), - 'RNNCell params template.') - p.Define('rnn_cell_dim', 1024, 'size of the rnn cells.') - p.Define('rnn_layers', 8, 'Number of rnn layers.') - p.Define('residual_start', 2, 'Start residual connections from this layer.') - p.Define('atten_rnn_cls', rnn_layers.FRNNWithAttention, - 'Which atten rnn cls to use.') - p.Define('use_prev_atten_ctx', False, - 'If True, all decoder layers use previous attention context as ' - 'input. Otherwise, only first decoder layer uses previous ' - 'attention context and the rest of the layers use current ' - 'attention context.') - p.Define('dropout_prob', 0.0, 'Prob at which we do dropout.') - # Default value was mildly tuned. Could be further tuned in the future. - p.Define('qlogsoftmax_range_min', -10.0, 'Quantization of the output of ' - 'log softmax.') - p.Define( - 'use_zero_atten_state', False, 'To use zero attention state ' - 'instead of computing attention with zero query vector.') - - p.Define('cc_schedule', None, 'Clipping cap schedule.') - p.Define( - 'init_step_ids', False, - 'Initializes beam search with first target id instead of .' - 'Use this when decoding starts with target_lang id intead of ' - 'token at time step 0. Make sure the training data has ' - 'target_lang id as the first token in target sequence.') - - disable_vn = py_utils.VariationalNoiseParams(1.0, False, False) - default_params_init = py_utils.WeightInit.Uniform(0.04) - - # Default config for the embedding. - p.emb.vn = disable_vn - p.emb.vocab_size = 32000 - p.emb.embedding_dim = 1024 - p.emb.max_num_shards = 16 - p.emb.params_init = default_params_init - - # Default config for the attention model. - p.attention.vn = disable_vn - p.attention.hidden_dim = 1024 - p.attention.params_init = None # Filled in after dims are known. - # Default config for the attention rnn cell. - p.atten_rnn_cell_tpl.vn = disable_vn - p.atten_rnn_cell_tpl.params_init = default_params_init - # Default config for the rnn cell. - p.rnn_cell_tpl.vn = disable_vn - p.rnn_cell_tpl.params_init = default_params_init - # Default config for the softmax part. - p.softmax.vn = disable_vn - p.softmax.num_classes = 32000 # 32k - p.softmax.num_shards = 16 - p.softmax.params_init = default_params_init - - # Default config for beam search. - p.target_seq_len = 300 - p.beam_search.length_normalization = 0.2 - p.beam_search.coverage_penalty = 0.2 - - return p - - @classmethod - def UpdateTargetVocabSize(cls, p, vocab_size, wpm_model=None): - """Updates the params with the input vocab_size and WPM model. - - Args: - p: model params. - vocab_size: size of the vocabulary. - wpm_model: file name prefix pointing to a wordpiece model. - - Returns: - Model params updated with the vocab size and wpm model. - """ - p = super().UpdateTargetVocabSize(p, vocab_size) - p.emb.vocab_size = vocab_size - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.emb.vocab_size == p.softmax.num_classes - - if p.cc_schedule is None: - self.cc_schedule = None - else: - self.CreateChild('cc_schedule', p.cc_schedule) - - self.CreateChild('emb', p.emb) - - p.attention.dtype = p.dtype - p.attention.source_dim = p.source_dim - p.attention.query_dim = p.rnn_cell_dim - p.attention.packed_input = p.packed_input - if p.attention.params_init is None: - p.attention.params_init = py_utils.WeightInit.Gaussian( - 1. / math.sqrt(p.attention.source_dim + p.attention.query_dim), - seed=p.random_seed) - atten_params = p.attention.Copy() - - params = p.atten_rnn_cell_tpl.Copy() - params.name = 'atten_rnn' - params.dtype = p.dtype - params.reset_cell_state = p.packed_input - params.num_input_nodes = p.emb.embedding_dim + p.attention.source_dim - params.num_output_nodes = p.rnn_cell_dim - atten_rnn_cell = params.Copy() - - params = p.atten_rnn_cls.Params() - params.name = 'frnn_with_atten' - params.dtype = p.dtype - params.cell = atten_rnn_cell - params.attention = atten_params - params.output_prev_atten_ctx = p.use_prev_atten_ctx - params.packed_input = p.packed_input - params.use_zero_atten_state = p.use_zero_atten_state - params.atten_context_dim = p.attention.source_dim - self.CreateChild('frnn_with_atten', params) - - # TODO(zhifengc): Avoid this? - self._atten = self.frnn_with_atten.attention - - rnn_layers_params = [] - for i in range(1, p.rnn_layers): - params = p.rnn_cell_tpl.Copy() - params.name = 'rnn%d' % i - params.dtype = p.dtype - params.num_input_nodes = p.rnn_cell_dim + p.attention.source_dim - params.num_output_nodes = p.rnn_cell_dim - params.reset_cell_state = p.packed_input - rnn_cell_p = params - - params = model_helper.CreateUnidirectionalRNNParams( - self.params, rnn_cell_p) - params.name = 'frnn%d' % i - params.packed_input = p.packed_input - rnn_layers_params.append(params) - - self.CreateChildren('frnn', rnn_layers_params) - - p.softmax.dtype = p.dtype - if p.feed_attention_context_vec_to_softmax: - p.softmax.input_dim = p.rnn_cell_dim + p.attention.source_dim - else: - p.softmax.input_dim = p.rnn_cell_dim - self.CreateChild('softmax', p.softmax) - - def _CreateChildrenVariables(self): - with tf.variable_scope(self.params.name): - if py_utils.use_tpu(): - emb_device = self.cluster.WorkerDeviceInModelSplit(0) - else: - emb_device = '' - with tf.device(emb_device): - self.emb.InstantiateVariables() - self.frnn_with_atten.InstantiateVariables() - for frnn in self.frnn: - frnn.InstantiateVariables() - super()._CreateChildrenVariables() - - def ApplyDropout(self, x_in): - p = self.params - assert 0 <= p.dropout_prob and p.dropout_prob < 1.0 - if self.do_eval or p.dropout_prob == 0.0: - return x_in - else: - return tf.nn.dropout(x_in, rate=p.dropout_prob) - - def ApplyClipping(self, theta, x): - if self.cc_schedule: - return self.cc_schedule.ApplyClipping(theta.cc_schedule, x) - else: - return x - - @py_utils.NameScopeDecorator('MTDecoderV1/ComputePredictions') - def ComputePredictions(self, theta, encoder_outputs, targets): - """Decodes `targets` given encoded source. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - encoder_outputs: a NestedMap computed by encoder. Expected to contain: - encoded - source encoding, of shape [time, batch, depth]. - padding - source encoding's padding, of shape [time, batch]. - segment_id - (optional) source segment id, of shape [time, batch]. - targets: A dict of string to tensors representing the targets one try to - predict. Each tensor in targets is of shape [batch, time]. - - Returns: - A `.NestedMap` containing information about the decoding process. At a - minimum, this should contain: - softmax_input: Tensor of shape [time, batch, params.softmax.input_dim]. - attention: `.NestedMap` of attention distributions of shape [batch, - time, source_len]. - source_enc_len: Lengths of source sentences. Tensor of shape [batch]. - """ - p = self.params - source_paddings = encoder_outputs.padding - time, batch = py_utils.GetShape(source_paddings, 2) - source_encs = py_utils.HasShape(encoder_outputs.encoded, - [time, batch, p.source_dim]) - with tf.name_scope(p.name): - target_ids = tf.transpose(targets.ids) - target_paddings = py_utils.HasRank(targets.paddings, 2) - target_paddings = tf.expand_dims(tf.transpose(target_paddings), 2) - if p.packed_input: - target_segment_id = tf.expand_dims(tf.transpose(targets.segment_ids), 2) - else: - target_segment_id = tf.zeros_like(target_paddings) - - if py_utils.use_tpu(): - emb_device = self.cluster.WorkerDeviceInModelSplit(0) - else: - emb_device = '' - with tf.device(emb_device): - inputs = self.emb.EmbLookup(theta.emb, target_ids) - inputs = self.ApplyClipping(theta, inputs) - summary_utils.histogram('input_emb', inputs) - inputs = self.ApplyDropout(inputs) - self._emb_out = inputs - - # Layer 0 intertwines with attention. - (accumulated_states, _, - side_info) = self.frnn_with_atten.AccumulateStates( - theta.frnn_with_atten, - source_encs, - source_paddings, - inputs, - target_paddings, - src_segment_id=getattr(encoder_outputs, 'segment_id', None), - segment_id=target_segment_id) - - (atten_ctxs, xs, atten_probs) = self.frnn_with_atten.PostProcessStates( - accumulated_states, side_info) - - self._AddAttenProbsSummary(source_paddings, targets, [atten_probs]) - - atten_ctxs = self.ApplyClipping(theta, atten_ctxs) - summary_utils.histogram('atten_ctxs', atten_ctxs) - - for i, (layer, layer_theta) in enumerate(zip(self.frnn, theta.frnn)): - # Forward through Layer-(i + 1) because Layer-0 handled before. - ys, _ = layer.FProp( - layer_theta, - tf.concat([xs, atten_ctxs], 2), - target_paddings, - segment_id=target_segment_id) - ys = self.ApplyDropout(ys) - if 1 + i >= p.residual_start: - xs += ys # Residual skip - xs = self.ApplyClipping(theta, xs) - else: - xs = ys - summary_utils.histogram('layer_out_%s' % i, xs) - - if p.feed_attention_context_vec_to_softmax: - xs = tf.concat([xs, atten_ctxs], 2) - - # Get intermediate attention information - atten_states = accumulated_states.atten_state - if isinstance(atten_states, py_utils.NestedMap): - additional_atten_probs = sorted( - [(name, tensor) - for name, tensor in atten_states.FlattenItems() - if name.endswith('probs')]) - else: - additional_atten_probs = [] - attention_map = py_utils.NestedMap(probs=accumulated_states.atten_probs) - attention_map.update(additional_atten_probs) - - # Transpose attention probs from [target_length, batch, source_length] - # to [batch, target_length, source_length] - def _TransposeAttentions(x): - return tf.transpose(x, [1, 0, 2]) - - attention_map = attention_map.Transform(_TransposeAttentions) - if isinstance(source_paddings, tf.Tensor): - source_enc_len = tf.reduce_sum(1 - source_paddings, axis=0) - - return py_utils.NestedMap( - softmax_input=xs, - attention=attention_map, - source_enc_len=source_enc_len) - - def AddExtraDecodingInfo(self, encoder_outputs, targets): - """Adds extra decoding information to encoded_outputs. - - Args: - encoder_outputs: a NestedMap computed by encoder. - targets: a NestedMap containing target input fields. - - Returns: - encoder_ouputs with extra information used for decoding. - """ - p = self.params - if p.init_step_ids: - encoder_outputs['init_step_ids'] = targets.ids[:, 0] - return encoder_outputs - - @py_utils.NameScopeDecorator('MTDecoderV1/InitDecoder') - def _InitDecoder(self, theta, encoder_outputs, num_hyps): - """Returns initial decoder states. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - encoder_outputs: a NestedMap computed by encoder. - num_hyps: Scalar Tensor of type int, Number of hypothesis maintained in - beam search, equal to beam_size * num_hyps_per_beam. - - Returns: - Tuple of initial model states. Also inserts 'packed_src' to - 'encoder_outputs'. - """ - p = self.params - source_paddings = encoder_outputs.padding - time, batch = py_utils.GetShape(source_paddings, 2) - source_encs = py_utils.HasShape(encoder_outputs.encoded, - [time, batch, p.source_dim]) - rnn_states = [ - self.frnn_with_atten.cell.zero_state(theta.frnn_with_atten.cell, - num_hyps) - ] - for layer, layer_theta in zip(self.frnn, theta.frnn): - rnn_states.append(layer.rnn_cell.zero_state(layer_theta, num_hyps)) - - if p.use_zero_atten_state: - encoder_outputs.packed_src = self._atten.InitForSourcePacked( - theta.frnn_with_atten.atten, source_encs, source_encs, - source_paddings) - s_seq_len = tf.shape(source_encs)[0] - context_dim = tf.shape(source_encs)[2] - atten_context = tf.zeros([num_hyps, context_dim], dtype=source_encs.dtype) - atten_states = self._atten.ZeroAttentionState(s_seq_len, num_hyps) - atten_probs = tf.zeros([num_hyps, s_seq_len], dtype=source_encs.dtype) - else: - encoder_outputs.packed_src = self._atten.InitForSourcePacked( - theta.frnn_with_atten.atten, source_encs, source_encs, - source_paddings) - - src_seq_len = tf.shape(source_encs)[0] - zero_atten_state = self._atten.ZeroAttentionState(src_seq_len, num_hyps) - - (atten_context, atten_probs, - atten_states) = self._atten.ComputeContextVectorWithSource( - theta.frnn_with_atten.atten, - encoder_outputs.packed_src, - tf.zeros([num_hyps, p.rnn_cell_dim], dtype=py_utils.FPropDtype(p)), - attention_state=zero_atten_state) - - assert atten_states is not None - return rnn_states, atten_context, atten_probs, atten_states - - @py_utils.NameScopeDecorator('MTDecoderV1/DecodeStep') - def _DecodeStep(self, theta, encoder_outputs, embs, step_paddings, - prev_atten_context, rnn_states, prev_atten_states): - """Decode one step.""" - p = self.params - new_rnn_states = [] - new_rnn_states_0, _ = self.frnn_with_atten.cell.FProp( - theta.frnn_with_atten.cell, rnn_states[0], - py_utils.NestedMap( - act=[tf.concat([embs, prev_atten_context], 1)], - padding=step_paddings, - reset_mask=tf.ones_like(step_paddings))) - new_rnn_states.append(new_rnn_states_0) - rnn_out = self.frnn_with_atten.cell.GetOutput(new_rnn_states_0) - cur_atten_context, atten_probs, atten_states = ( - self._atten.ComputeContextVectorWithSource( - theta.frnn_with_atten.atten, - encoder_outputs.packed_src, - rnn_out, - attention_state=prev_atten_states)) - assert atten_states is not None - - if p.use_prev_atten_ctx: - atten_context = prev_atten_context - else: - atten_context = cur_atten_context - - for i, (layer, layer_theta) in enumerate(zip(self.frnn, theta.frnn)): - new_rnn_states_i, _ = layer.rnn_cell.FProp( - layer_theta.cell, rnn_states[1 + i], - py_utils.NestedMap( - act=[tf.concat([rnn_out, atten_context], 1)], - padding=step_paddings, - reset_mask=tf.ones_like(step_paddings))) - new_rnn_states.append(new_rnn_states_i) - new_rnn_out = layer.rnn_cell.GetOutput(new_rnn_states_i) - if 1 + i >= p.residual_start: - rnn_out += new_rnn_out - rnn_out = self.ApplyClipping(theta, rnn_out) - else: - rnn_out = new_rnn_out - # Concatenating atten_context vec to rnn output before softmax might help - if p.feed_attention_context_vec_to_softmax: - step_out = tf.concat([rnn_out, atten_context], 1) - else: - step_out = rnn_out - return (cur_atten_context, atten_probs, new_rnn_states, step_out, - atten_states) - - def _GetAttentionInitState(self): - """Gets the attention initialization state. - - It is valid to call this after `_DecoderInit()`. Inference subclasses use - this to split computation across subgraph boundaries. - - Returns: - `.NestedMap` of attention source states. - """ - return self._atten.GetInitializationSourceState() - - def _SetAttentionInitState(self, new_init_state): - """Sets the attention initialization state. - - Args: - new_init_state: `.NestedMap` compatible with that returned from - `_GetAttentionSourceState`. - """ - self._atten.SetInitializationSourceState(new_init_state) - - def _InitBeamSearchStateCallback(self, theta, encoder_outputs, - num_hyps_per_beam): - """Returns initial beams search states. - - Args: - theta: a NestedMap of parameters. - encoder_outputs: a NestedMap computed by encoder. - num_hyps_per_beam: An int, number hyps to keep for source sentence. - - Returns: - A tuple (initial_results, states). - initial_results: a `.NestedMap` of initial results. - atten_probs: - The initial attention probs, of shape [tgt_batch, src_len]. - states: a `.NestedMap` of initial model states. - rnn_states: - Initial state of the RNN. - atten_context: - Initial attention context vector. - atten_states: - Initial attention state. - """ - p = self.params - num_beams = py_utils.GetShape(encoder_outputs.padding)[1] - num_hyps = num_beams * num_hyps_per_beam - rnn_states, init_atten_context, atten_probs, atten_states = ( - self._InitDecoder(theta, encoder_outputs, num_hyps)) - - initial_results = py_utils.NestedMap( - log_probs=tf.zeros([num_hyps, p.softmax.num_classes], - dtype=py_utils.FPropDtype(p)), - atten_probs=atten_probs) - - if p.init_step_ids and hasattr(encoder_outputs, 'init_step_ids'): - initial_results['step_ids'] = tf.expand_dims( - self._ExpandToNumHyps(encoder_outputs.init_step_ids, - num_hyps_per_beam), 1) - - return initial_results, py_utils.NestedMap({ - 'time_step': tf.constant(0), - 'rnn_states': rnn_states, - 'atten_context': init_atten_context, - 'atten_probs': atten_probs, - 'atten_states': atten_states, - }) - - @py_utils.NameScopeDecorator('MTDecoderV1/PreBeamSearchStepCallback') - def _PreBeamSearchStepCallback(self, theta, encoder_outputs, step_ids, states, - num_hyps_per_beam): - """Returns logits for sampling ids and the next model states. - - Args: - theta: a NestedMap of parameters. - encoder_outputs: a NestedMap computed by encoder. - step_ids: A tensor of shape [tgt_batch, 1]. - states: A `.NestedMap` of tensors representing states that the clients - would like to keep track of for each of the active hyps. - num_hyps_per_beam: Beam size. - Returns: - A tuple (results, out_states). - results: A `.NestedMap` of beam search results. - atten_probs: - The updated attention probs, of shape [tgt_batch, src_len]. - log_probs: - Log prob for each of the tokens in the target vocab. This is of shape - [tgt_batch, vocab_size]. - out_states: A `.NestedMap`. The updated states. - rnn_states: - Last state of the RNN. - atten_context: - Updated attention context vector. - atten_states: - Updates attention states. - """ - p = self.params - - prev_rnn_states = states['rnn_states'] - prev_atten_context = states['atten_context'] - prev_atten_probs = states['atten_probs'] - prev_atten_states = states['atten_states'] - step_paddings = tf.zeros(py_utils.GetShape(step_ids), dtype=p.dtype) - embs = self.emb.EmbLookup(theta.emb, tf.reshape(step_ids, [-1])) - embs = self.ApplyClipping(theta, embs) - atten_context, atten_probs, rnn_states, step_out, atten_states = ( - self._DecodeStep(theta, encoder_outputs, embs, step_paddings, - prev_atten_context, prev_rnn_states, - prev_atten_states)) - atten_probs = tf.reshape(atten_probs, tf.shape(prev_atten_probs)) - - logits = self.softmax.Logits(theta.softmax, [step_out]) - log_probs = self.fns.qlogsoftmax( - logits, qmin=p.qlogsoftmax_range_min, qmax=0.0) - - if p.use_prev_atten_ctx: - cur_atten_probs = prev_atten_probs - else: - cur_atten_probs = atten_probs - - bs_results = py_utils.NestedMap({ - 'atten_probs': cur_atten_probs, # the probs exposed to beam search - 'log_probs': log_probs, - }) - new_states = py_utils.NestedMap({ - 'time_step': states.time_step + 1, - 'rnn_states': rnn_states, - 'atten_context': atten_context, - 'atten_probs': atten_probs, # the updated attention probs - 'atten_states': atten_states, - }) - - return bs_results, new_states - - def _PostBeamSearchStepCallback(self, theta, encoder_outputs, new_step_ids, - states): - # There is nothing to do here. - return states - - -class TransformerDecoder(MTBaseDecoder): - """Transformer decoder. - - Implements the decoder of Transformer model: - https://arxiv.org/abs/1706.03762. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('token_emb', layers.EmbeddingLayer.Params(), - 'Token embedding layer params.') - p.Define('position_emb', layers.PositionalEmbeddingLayer.Params(), - 'Position embedding layer params.') - p.Define('source_dim', 1024, 'Dimension of encoder outputs.') - p.Define('model_dim', 1024, 'Model dimension that applies to embedding ' - 'layers and all Transformer layers.') - p.Define('num_trans_layers', 6, 'Number of Transformer layers.') - p.Define( - 'trans_tpl', layers_with_attention.TransformerLayer.Params(), - 'Transformer layer params. ' - ' Can be a list. num_trans_layers should be divisible by ' - 'len(trans_tpl).') - p.Define('input_dropout_prob', 0.0, 'Prob at which we do input dropout.') - p.Define( - 'is_transparent', False, 'If set, expects a tensor of shape ' - '[time, batch, source_dim, num_trans_layers] as source encodings.') - p.Define( - 'add_multiheaded_attention_scalar_summary', False, - 'If set, will include scalar summaries for multi-headed attention' - ' to visualize the sparsity statistics of attention weights.') - p.Define('ln_tpl', layers.LayerNorm.Params(), 'Layer norm default params') - p.Define( - 'ln_output', False, 'If True, layer normalization is applied to the ' - 'final output of the decoder.') - - # TODO(miachen): Extend this to more general logic of adding multiple - # embedding fields. - p.Define('task_emb', None, 'Task embedding layer params.') - p.Define( - 'init_step_ids', False, - 'Initializes beam search with first target id instead of .' - 'Use this when decoder has target language token intead of ' - 'token at time step 0.' - 'Make sure the training is done in similar manner.') - # MASS pretraining related (https://github.com/microsoft/MASS) - p.Define( - 'use_lang_dependent_atten', False, 'If True, attention between ' - 'encoder and decoder is language dependent.') - - p.Define('zero_token_embs_first_time_step', False, - 'If True, the first time step uses zeros as the post-emb lookup.') - - # Default config for the token embedding. - p.token_emb.vocab_size = 32000 - p.token_emb.embedding_dim = p.model_dim - p.token_emb.max_num_shards = 16 - p.token_emb.params_init = py_utils.WeightInit.Gaussian( - 1.0 / math.sqrt(p.token_emb.embedding_dim)) - p.token_emb.scale_sqrt_depth = True - - # Default config for the position embedding. - p.position_emb.embedding_dim = p.model_dim - - # Default config for the transformer layers. - p.trans_tpl.source_dim = p.model_dim - p.trans_tpl.tr_atten_tpl.source_dim = p.model_dim - p.trans_tpl.tr_atten_tpl.num_attention_heads = 8 - p.trans_tpl.tr_fflayer_tpl.input_dim = p.model_dim - p.trans_tpl.tr_fflayer_tpl.hidden_dim = 2048 - - # Default config for beam search. - p.target_seq_len = 300 - p.beam_search.length_normalization = 0.5 - p.beam_search.coverage_penalty = 0.0 - p.beam_search.batch_major_state = False - - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - if p.softmax.cls == layers.SharedSoftmaxLayer: - self._token_emb_vocab_size = p.softmax.num_classes - self._token_emb_dim = p.model_dim - self._share_sm_emb = True - else: - self._token_emb_vocab_size = p.token_emb.vocab_size - self._token_emb_dim = p.token_emb.embedding_dim - self._share_sm_emb = False - - assert self._token_emb_vocab_size == p.softmax.num_classes - assert self._token_emb_dim == p.position_emb.embedding_dim - if p.model_dim != self._token_emb_dim: - tf.logging.warning( - 'token_emb.embedding_dim != model_dim (%s vs. %s), ' - 'creating a projection!') - proj_p = layers.ProjectionLayer.Params().Copy() - proj_p.name = 'emb_proj' - proj_p.input_dim = p.token_emb.embedding_dim - proj_p.output_dim = p.model_dim - self.CreateChild('emb_proj', proj_p) - - if p.use_lang_dependent_atten and p.task_emb: - p.trans_tpl.num_aux_atten_post_proj = p.task_emb.vocab_size - - if not self._share_sm_emb: - self.CreateChild('token_emb', p.token_emb) - self.CreateChild('position_emb', p.position_emb) - if p.task_emb: - assert p.task_emb.embedding_dim == self._token_emb_dim - self.CreateChild('task_emb', p.task_emb) - - dropout_tpl = layers.DropoutLayer.Params() - dropout_tpl.keep_prob = (1.0 - p.input_dropout_prob) - self.CreateChild('input_dropout', dropout_tpl) - - params_trans_layers = [] - denom = 1 - if isinstance(p.trans_tpl, list): - denom = len(p.trans_tpl) - assert p.num_trans_layers % denom == 0 - for i in range(p.num_trans_layers // denom): - if isinstance(p.trans_tpl, list): - for q in p.trans_tpl: - params = q.Copy() - params_trans_layers.append(params) - else: - params = p.trans_tpl.Copy() - params_trans_layers.append(params) - - for i, params in enumerate(params_trans_layers): - params.name = 'trans_layer_%d' % i - params.packed_input = p.packed_input - params.has_aux_atten = True - params.mask_self_atten = True - - # Initialize decoder output layer norm - if p.ln_output: - params = p.ln_tpl.Copy() - params.name = 'dec_out_ln' - params.input_dim = p.model_dim - self.CreateChild('layer_norm_out', params) - - self.CreateChildren('trans', params_trans_layers) - - p.softmax.input_dim = p.model_dim - self.CreateChild('softmax', p.softmax) - - def _CreateChildrenVariables(self): - if self._share_sm_emb: - # Taking shared emb/softmax layer out of the decoder variable scope so - # that it can also be shared by encoder if needed. - with tf.variable_scope('shared_emb', reuse=tf.AUTO_REUSE): - self.softmax.InstantiateVariables() - super()._CreateChildrenVariables() - - def _RemoveEOSProbs(self, p, probs, source_enc_len): - """Remove the attention probs on EOS symbol and renormalize. - - Args: - p: decoder params. - probs: attention probs matrix; float [batch, target_len, source_len]. - source_enc_len: source encoder length; int [batch]. - - Returns: - probs with value on last actual token (EOS token) replaced by 0 and - renormalized so that final dim (src_len) sums to 1 again; float - [batch, target_len, source_len]. - """ - batch = py_utils.GetShape(probs)[0] - source_enc_len = py_utils.HasShape(source_enc_len, [batch]) - - # Set -1 values - target_len = py_utils.GetShape(probs)[1] - replacements = tf.ones([py_utils.GetShape(probs)[0], target_len], - dtype=py_utils.FPropDtype(p)) * (-1) - - index_0 = tf.reshape(tf.range(batch), shape=[batch, 1, 1]) - index_0 *= tf.ones(shape=[batch, target_len, 1], dtype=tf.int32) - - index_1 = tf.ones(shape=[batch, 1], dtype=tf.int32) - index_1 *= tf.expand_dims(tf.range(target_len), 0) - index_1 = tf.expand_dims(index_1, -1) - - index_2 = tf.reshape(source_enc_len, shape=[batch, 1, 1]) - 1 # Note the -1 - index_2 = tf.cast(index_2, tf.int32) - index_2 *= tf.ones(shape=[batch, target_len, 1], dtype=tf.int32) - - index = tf.concat([index_0, index_1, index_2], axis=2) - - # Original update matrix contained -1 values. Change all to 1 except for - # those positions coming from scatter which will be 0. - updates = tf.scatter_nd( - index, updates=replacements, shape=py_utils.GetShape(probs)) - updates += 1 - res = probs * updates - - # Normalize to that probs sum to 1. - # Add eps to sum to deal with case where all probs except last one are 0. - # In this case then, attention probs will not sum to 1 but this seems still - # better then evenly distributing attention probs in this case. - s = tf.reduce_sum(res, axis=2, keepdims=True) - epsilon = tf.constant(value=1e-6, dtype=py_utils.FPropDtype(p)) - s += epsilon - res /= s - return res - - def _ZeroOutFirstTimeStep(self, token_embs, batch, target_time): - """Zeroes out the first time step. - - Args: - token_embs: [batch, time, model_dim] embeding lookups - batch: Batch size scalar - target_time: Target sequence length scalar. - - Returns: - modified token_embs with the first time step zeroed out. - """ - p = self.params - - zero_out_index = tf.expand_dims(tf.constant([0]), axis=1) - # [[[0]]] - zero_out_index = tf.expand_dims(zero_out_index, axis=1) - - # [[0]...[target_time-1]] - time_steps = tf.expand_dims(tf.range(target_time), axis=1) - condition = tf.equal(zero_out_index, time_steps) - mask = tf.logical_not(tf.tile(condition, [batch, 1, p.model_dim])) - mask = tf.cast(mask, dtype=tf.float32) - return token_embs * mask - - def _FProp(self, theta, encoder_outputs, targets): - """Decodes `targets` given encoded source. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - encoder_outputs: a NestedMap computed by encoder. Expected to contain: - encoded - source encoding. When `p.is_transparent` is False, it is a - tensor of shape [time, batch, depth]. When `p.is_transparent` is True, - it is a tensor of shape [time, batch, depth, num_trans_layers] if - `self.do_eval` is True, and a list of `num_trans_layers` tensors of - shape [time, batch, depth] if `self.do_eval` is False. padding - source - encoding's padding, of shape [time, batch]. segment_id - source segment - id, of shape [time, batch]. - targets: A dict of string to tensors representing the targets one try to - predict. Each tensor in targets is of shape [batch, time]. - - Returns: - A `.NestedMap` containing output of last decoder layer and attention probs - - - softmax_input: Tensor of shape [time, batch, params.softmax.input_dim]. - - attention: `.NestedMap` of attention distributions of shape - [batch, target_length, source_length]. - """ - p = self.params - source_encs = encoder_outputs.encoded - source_paddings = encoder_outputs.padding - src_segment_id = getattr(encoder_outputs, 'segment_id', None) - time, batch = py_utils.GetShape(source_paddings, 2) - if p.is_transparent: - if self.do_eval: - source_encs = py_utils.HasShape( - source_encs, [time, batch, p.source_dim, p.num_trans_layers]) - source_encs = tf.unstack(source_encs, axis=3) - else: - assert isinstance(source_encs, list) - assert len(source_encs) == p.num_trans_layers - for i in range(p.num_trans_layers): - source_encs[i] = py_utils.HasShape(source_encs[i], - [time, batch, p.source_dim]) - else: - source_encs = py_utils.HasShape(source_encs, [time, batch, p.source_dim]) - source_encs = [source_encs] * p.num_trans_layers - with tf.name_scope(p.name): - # [batch, time] - target_ids = targets.ids - # [time, batch] - target_paddings = tf.transpose(targets.paddings) - target_segment_pos = None - target_segment_id = None - if p.packed_input: - target_segment_id = tf.transpose(targets.segment_ids) - target_segment_pos = targets.segment_pos - assert src_segment_id is not None, ('Need to provide src_segment_id ' - 'for packed input.') - - # Embedding layer - # [batch, time, model_dim] - if not self._share_sm_emb: - token_embs = self.token_emb.EmbLookup(theta.token_emb, target_ids) - else: - token_embs = self.softmax.EmbLookup(theta.softmax, target_ids) - - target_batch = py_utils.GetShape(target_ids)[0] - target_time = py_utils.GetShape(target_ids)[1] - - if p.zero_token_embs_first_time_step: - # For models that do not use an explicit start-of-sequence token - # with associated embedding, but instead use zeros. - token_embs = self._ZeroOutFirstTimeStep(token_embs, target_batch, - target_time) - - # [1, time, model_dim] - if p.packed_input: - posit_embs = self.position_emb.FPropWithPosition( - theta.position_emb, target_segment_pos) - else: - posit_embs = tf.expand_dims( - self.position_emb.FProp(theta.position_emb, target_time), 0) - - # [time, batch, model_dim] - input_embs = token_embs + posit_embs - - atten_idx = None - if p.task_emb: - if p.use_lang_dependent_atten: - atten_idx = targets.task_ids - # Works for both packed and unpacked inputs. - atten_idx = tf.reshape(tf.transpose(atten_idx), [-1]) - input_embs += self.task_emb.EmbLookup(theta.task_emb, targets.task_ids) - - if p.model_dim != self._token_emb_dim: - input_embs = self.emb_proj.FProp(theta.emb_proj, input_embs) - - input_embs = tf.transpose(input_embs, [1, 0, 2]) - input_embs = self.input_dropout.FProp(theta.input_dropout, input_embs) - - if not p.packed_input: - src_enc_len = tf.reduce_sum(1 - source_paddings, axis=0) - num_hyps_per_beam = tf.div( - py_utils.GetShape(target_paddings)[1], - py_utils.GetShape(source_paddings)[1]) - src_enc_len = self._ExpandToNumHyps(src_enc_len, num_hyps_per_beam) - - layer_in = input_embs - per_layer_attn_probs = [] - for i, (layer, layer_theta) in enumerate(zip(self.trans, theta.trans)): - extra_kwargs = dict() - if isinstance(layer, layers_with_attention.TransformerWithContextLayer): - # If the encoder contains encodings for the context and the - # transformer layer in the decoder is able to attend to it, we pass - # them to the transformer layer. - extra_kwargs['tertiary_vecs'] = encoder_outputs.context_encoded - extra_kwargs['tertiary_paddings'] = encoder_outputs.context_padding - # [time, batch, model_dim] - layer_out, probs = layer.FProp( - layer_theta, - layer_in, - target_paddings, - source_encs[i], - source_paddings, - source_segment_id=target_segment_id, - aux_segment_id=src_segment_id, - atten_idx=atten_idx, - **extra_kwargs) - layer_in = layer_out - pl_probs = tf.transpose(probs, [1, 0, 2]) - if p.packed_input: - # For packed inputs we are currently not removing the EOS token. - per_layer_attn_probs.append(pl_probs) - else: - # Remove attention weight on last (EOS) token and re-normalize - # so that last dimension sums to 1. See b/129097156. - # Original probs shape: [trg time, batch, src time] - norma_atten_probs_3d = self._RemoveEOSProbs(p, pl_probs, src_enc_len) - per_layer_attn_probs.append(norma_atten_probs_3d) - - if p.ln_output: - layer_out = self.layer_norm_out.FProp(theta.layer_norm_out, layer_out) - - # per_layer_attn_probs shape: [batch, trg time, src time] - self._AddAttenProbsSummary(source_paddings, targets, per_layer_attn_probs) - - # Aggregate per-layer attention probs. - aggregated_atten_probs = ( - tf.math.add_n(per_layer_attn_probs) / len(per_layer_attn_probs)) - - attention_map = py_utils.NestedMap(probs=aggregated_atten_probs) - return py_utils.NestedMap( - softmax_input=layer_out, attention=attention_map) - - def AddExtraDecodingInfo(self, encoder_outputs, targets): - """Adds extra decoding information to encoded_outputs. - - Args: - encoder_outputs: a NestedMap computed by encoder. - targets: a NestedMap containing target input fields. - - Returns: - encoder_ouputs with extra information used for decoding. - """ - p = self.params - if p.task_emb: - encoder_outputs['target_task_ids'] = targets.task_ids[:, 0] - if p.init_step_ids: - encoder_outputs['init_step_ids'] = targets.ids[:, 0] - return encoder_outputs - - def ExtendStep(self, theta, encoder_outputs, new_ids, t, prefix_states): - """Extend prefix as represented by `prefix_states` by one more step. - - This function is expected to be called during fast decoding of Transformer - models. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - encoder_outputs: a NestedMap computed by encoder, containing: - - - encoded: source encoding, of shape [time, batch, depth]. Can be [time, - bs, depth, num_trans_layers] if is_transparent is set. - - padding: source encoding's padding, of shape [time, batch]. - new_ids: new input ids, of shape [batch]. - t: a scalar, the current time step, 0-based. - prefix_states: a `.NestedMap` representing the prefix that has already - been decoded. - - Returns: - A tuple (last_decoder_out, prefix_states, atten_probs), where - last_decoder_out is the output of the last decoder layer of - shape [batch, model_dim], `prefix_states` is the update prefix states, - and atten_probs contains attention in shape [batch, src_len] for the - given target position. - """ - p = self.params - source_paddings = encoder_outputs.padding - time, batch = py_utils.GetShape(source_paddings, 2) - if p.is_transparent: - source_encs = py_utils.HasShape( - encoder_outputs.encoded, - [time, batch, p.source_dim, p.num_trans_layers]) - source_encs = tf.unstack(source_encs, axis=3) - else: - source_encs = py_utils.HasShape(encoder_outputs.encoded, - [time, batch, p.source_dim]) - source_encs = [source_encs] * p.num_trans_layers - with tf.name_scope(p.name): - # Embedding layer - # [batch, time, model_dim] - if not self._share_sm_emb: - token_embs = self.token_emb.EmbLookup(theta.token_emb, new_ids) - else: - token_embs = self.softmax.EmbLookup(theta.softmax, new_ids) - - if p.zero_token_embs_first_time_step: - # For models that do not use an explicit start-of-sequence token - # with associated embedding, but instead use zeros. - zeros = tf.zeros_like(token_embs) - token_embs = tf.cond(tf.equal(t, 0), lambda: zeros, lambda: token_embs) - - # [time, model_dim] - posit_embs = tf.slice( - self.position_emb.FProp(theta.position_emb, p.target_seq_len), [t, 0], - [1, p.model_dim]) - input_embs = token_embs + posit_embs - - # Infer num_hyps_per_beam: new_ids has orig_batch_size * num_hyps_per_beam - # source_paddings has orig_batch_size. - num_hyps_per_beam = tf.div( - py_utils.GetShape(new_ids)[0], - py_utils.GetShape(source_paddings)[1]) - - atten_idx = None - if p.task_emb: - task_ids = self._ExpandToNumHyps(encoder_outputs.target_task_ids, - num_hyps_per_beam) - if p.use_lang_dependent_atten: - atten_idx = task_ids - input_embs += self.task_emb.EmbLookup(theta.task_emb, task_ids) - - if p.model_dim != self._token_emb_dim: - input_embs = self.emb_proj.FProp(theta.emb_proj, input_embs) - - input_embs = self.input_dropout.FProp(theta.input_dropout, input_embs) - # Make a copy of the input. - out_prefix_states = prefix_states.Pack(prefix_states.Flatten()) - - layer_in = input_embs - - # Infer true source encoder length from the padding. - src_enc_len = tf.reduce_sum(1 - source_paddings, axis=0) - - # Need to expand src_enc_len to reflect multiple hypotheses. - src_enc_len = self._ExpandToNumHyps(src_enc_len, num_hyps_per_beam) - - atten_probs = [] - for i, (layer, layer_theta) in enumerate(zip(self.trans, theta.trans)): - extra_kwargs = dict() - if isinstance(layer, layers_with_attention.TransformerWithContextLayer): - # If the encoder contains encodings for the context and the - # transformer layer in the decoder is able to attend to it, we pass - # them to the transformer layer. - extra_kwargs['tertiary_vecs'] = encoder_outputs.context_encoded - extra_kwargs['tertiary_paddings'] = encoder_outputs.context_padding - # [time, batch, model_dim] - layer_prefix_states = prefix_states['layer_%i' % i] - layer_out, probs, updated_prefix_states = layer.ExtendStep( - layer_theta, - layer_in, - layer_prefix_states, - source_encs[i], - source_paddings, - t=t if p.beam_search.name == 'tpu_beam_search' else None, - atten_idx=atten_idx, - **extra_kwargs) - out_prefix_states['layer_%i' % i] = updated_prefix_states - layer_in = layer_out - # Enforce shape: [batch, src_len] - probs = tf.squeeze(probs, [0]) - # Remove attention weight on last (EOS) token and re-normalize - # so that last dimension sums to 1. See b/129097156. - probs_3d = tf.expand_dims(probs, axis=1) - probs_3d = self._RemoveEOSProbs(p, probs_3d, src_enc_len) - probs = tf.squeeze(probs_3d, axis=1) - - atten_probs.append(probs) - - if p.ln_output: - layer_out = self.layer_norm_out.FProp(theta.layer_norm_out, layer_out) - - # Aggregate per-layer attention probs. - aggregated_atten_probs = tf.math.add_n(atten_probs) / len(atten_probs) - return layer_out, out_prefix_states, aggregated_atten_probs - - def ComputePredictions(self, theta, encoder_outputs, targets): - """Decodes `targets` given encoded source. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - encoder_outputs: a NestedMap computed by encoder. Expected to contain: - - encoded - source encoding, of shape [time, batch, depth]. Can be [time, - batch, depth, num_layers] if is_transparent is set. - - padding - source encoding's padding, of shape [time, batch]. - segment_id - source segment id, of shape [time, batch]. - targets: A dict of string to tensors representing the targets one try to - predict. Each tensor in targets is of shape [batch, time]. - - Returns: - A `.NestedMap` containing output of last decoder layer and attention probs - - - softmax_input: Tensor of shape [time, batch, params.softmax.input_dim]. - - attention: `.NestedMap` of attention distributions of shape - [batch, time, source_len]. - """ - return self._FProp(theta, encoder_outputs, targets) - - def SampleSequenceDecode(self, encoder_outputs): - """Decode via sampling from softmax at each step. - - Args: - encoder_outputs: the outputs of the encoder. - - Returns: - BeamSearchDecodeOutput, same as what BeamSearchDecode returns. - """ - p = self.params - non_tpu = p.beam_search.name != 'tpu_beam_search' - - def InitCallback(theta, encoder_outputs, num_hyps_per_beam=1): - """Wrapper for _InitBeamSearchStateCallback for sequence sampler. - - The main change is to ensure state tensors have fixed shapes. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer - and its children layers. - encoder_outputs: a NestedMap computed by encoder. - num_hyps_per_beam: An int, number hyps to keep for source sentence. - - Returns: - A NestedMap of - - - initial_results: a `.NestedMap` of initial results. - - states: a `.NestedMap` of initial model states. - """ - init_results, states = self._InitBeamSearchStateCallback( - theta, encoder_outputs, num_hyps_per_beam) - if non_tpu: - prefix_states = states['prefix_states'] - for layer in range(p.num_trans_layers): - key = prefix_states['layer_%d' % layer]['key'] - value = prefix_states['layer_%d' % layer]['value'] - key_shapes = py_utils.GetShape(key) - bs = key_shapes[1] - atten_dim = key_shapes[2] - zeros = tf.zeros([p.target_seq_len, bs, atten_dim], - dtype=py_utils.FPropDtype(p)) - prefix_states['layer_%d' % layer]['key'] = tf.concat([key, zeros], 0) - prefix_states['layer_%d' % layer]['value'] = tf.concat([value, zeros], - 0) - return init_results, states - - def PreBeamSearchCallback(theta, - encoder_outputs, - step_ids, - states, - num_hyps_per_beam=1): - """Wrapper for _PreBeamSearchStepCallback for sequence sampler. - - The main change is to ensure state tensors have fixed shapes. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer - and its children layers. - encoder_outputs: a NestedMap computed by encoder. - step_ids: A tensor of shape [tgt_batch, 1]. - states: A `.NestedMap` of tensors representing states that the clients - would like to keep track of for each of the active hyps. - num_hyps_per_beam: Beam size. - - Returns: - A NestedMap of - - - results: A `.NestedMap` of beam search results. - - out_states: A `.NestedMap`. The updated states. - """ - - if non_tpu: - # Strip off paddings. - prefix_states = states['prefix_states'] - target_time = states.time_step - for layer in range(p.num_trans_layers): - key = prefix_states['layer_%d' % layer]['key'] - val = prefix_states['layer_%d' % layer]['value'] - prefix_states['layer_%d' % layer]['key'] = tf.slice( - key, [0, 0, 0], [target_time, -1, -1]) - prefix_states['layer_%d' % layer]['value'] = tf.slice( - val, [0, 0, 0], [target_time, -1, -1]) - - bs_results, new_states = self._PreBeamSearchStepCallback( - theta, encoder_outputs, step_ids, states, num_hyps_per_beam) - - if non_tpu: - # Add back paddings (to maintain paddings shape). - bs = tf.shape(new_states.prefix_states['layer_0']['key'])[1] - dim = tf.shape(new_states.prefix_states['layer_0']['key'])[2] - pad = tf.zeros([p.target_seq_len - new_states.time_step, bs, dim], - dtype=py_utils.FPropDtype(p)) - for layer in range(p.num_trans_layers): - key = new_states.prefix_states['layer_%d' % layer]['key'] - val = new_states.prefix_states['layer_%d' % layer]['value'] - new_states.prefix_states['layer_%d' % layer]['key'] = tf.concat( - [key, pad], axis=0) - new_states.prefix_states['layer_%d' % layer]['value'] = tf.concat( - [val, pad], axis=0) - - return bs_results, new_states - - random_seed = tf.random.uniform( - shape=[], maxval=(2**31 - 1), dtype=tf.int32, seed=p.random_seed) - sample = self.target_sequence_sampler.Sample( - self.theta, encoder_outputs, random_seed, InitCallback, - PreBeamSearchCallback, self._PostBeamSearchStepCallback) - bs = tf.shape(sample.ids)[0] - # Only need to make sure topk_hyps has the right shape - # [bs, num_hyps_per_beam], where num_hyps_per_beam=1 for sampling. - # TODO(yuancao): Support sampling multiple sequences and remove - # num_hyps_per_beam constraint. - assert self.params.beam_search.num_hyps_per_beam == 1 - sample.topk_hyps = tf.zeros([bs, 1], dtype=tf.string) - sample.topk_ids = sample.ids - weights = 1 - sample.paddings - sample.topk_lens = tf.cast(tf.reduce_sum(weights, axis=1), dtype=tf.int32) - sample.topk_scores = tf.reduce_sum( - tf.math.log(tf.reduce_max(tf.nn.softmax(sample.logits), axis=2)) * - weights, - axis=1) - return sample - - def _InitBeamSearchStateCallback(self, theta, encoder_outputs, - num_hyps_per_beam): - """Returns initial beams search states. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - encoder_outputs: a NestedMap computed by encoder. - num_hyps_per_beam: An int, number hyps to keep for source sentence. - Returns: - A tuple (initial_results, states). - initial_results: a `.NestedMap` of initial results. - atten_probs: - The initial attention probs, of shape [tgt_batch, src_len]. - states: a `.NestedMap` of initial model states. - source_encs: - A tensor of shape [src_batch, src_len, source_dim]. - source_paddings: - A tensor of shape [src_batch, src_len]. - target_ids: - Initial empty list of decoded ids. [num_hyps, 0]. - - """ - p = self.params - - source_encs = encoder_outputs.encoded - num_hyps = py_utils.GetShape(source_encs)[1] * num_hyps_per_beam - source_len = py_utils.GetShape(source_encs)[0] - - # Dummy attention probs - atten_probs = tf.ones([num_hyps, source_len]) / tf.cast( - source_len, tf.float32) - initial_results = py_utils.NestedMap( - log_probs=tf.zeros([num_hyps, p.softmax.num_classes], - dtype=py_utils.FPropDtype(p)), - atten_probs=atten_probs) - - if p.init_step_ids: - initial_results['step_ids'] = tf.expand_dims( - self._ExpandToNumHyps(encoder_outputs.init_step_ids, - num_hyps_per_beam), 1) - - batch_size = num_hyps - if isinstance(p.trans_tpl, list): - atten_hidden_dim = p.trans_tpl[0].tr_atten_tpl.atten_hidden_dim - assert [tpl.tr_atten_tpl.atten_hidden_dim for tpl in p.trans_tpl - ].count(atten_hidden_dim) == len( - p.trans_tpl), 'atten_hidden_dim must match' - else: - atten_hidden_dim = p.trans_tpl.tr_atten_tpl.atten_hidden_dim - - if not atten_hidden_dim: - atten_hidden_dim = p.model_dim - - if p.beam_search.name == 'tpu_beam_search': - seq_len = p.target_seq_len - else: - seq_len = 0 - - prefix_states = py_utils.NestedMap() - for layer in range(p.num_trans_layers): - prefix_states['layer_%d' % layer] = py_utils.NestedMap({ - 'key': - tf.zeros([seq_len, batch_size, atten_hidden_dim], - dtype=py_utils.FPropDtype(p)), - 'value': - tf.zeros([seq_len, batch_size, atten_hidden_dim], - dtype=py_utils.FPropDtype(p)), - }) - - return initial_results, py_utils.NestedMap({ - 'prefix_states': prefix_states, - 'time_step': tf.constant(0) - }) - - def _PreBeamSearchStepCallback(self, theta, encoder_outputs, step_ids, states, - num_hyps_per_beam): - """Returns logits for sampling ids and the next model states. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - encoder_outputs: a NestedMap computed by encoder. - step_ids: A tensor of shape [tgt_batch, 1]. - states: A `.NestedMap` of tensors representing states that the clients - would like to keep track of for each of the active hyps. - num_hyps_per_beam: Beam size. - Returns: - A tuple (results, out_states). - results: A `.NestedMap` of beam search results. - atten_probs: - The updated attention probs, of shape [tgt_batch, src_len]. - log_probs: - Log prob for each of the tokens in the target vocab. This is of - shape [tgt_batch, vocab_size]. - out_states: A `.NestedMap`. The updated states. - source_encs: - A tensor of shape [src_batch, src_len, source_dim]. - source_paddings: - A tensor of shape [src_batch, src_len]. - target_ids: - Updated list of decoded ids. [num_hyps, Num of decoded ids]. - - """ - p = self.params - - target_time = states.time_step - prefix_states = states.prefix_states - - new_states = states.Pack(states.Flatten()) - - layer_out, updated_prefix_states, atten_probs = self.ExtendStep( - theta, encoder_outputs, tf.squeeze(step_ids, 1), target_time, - prefix_states) - - new_states.prefix_states = updated_prefix_states - new_states.time_step = target_time + 1 - - softmax_input = tf.reshape(layer_out, [-1, p.softmax.input_dim]) - logits = self.softmax.Logits(theta.softmax, [softmax_input]) - - num_hyps = py_utils.GetShape(step_ids)[0] - # [time * batch, num_classes] -> [time, batch, num_classes] - logits = tf.reshape(logits, (-1, num_hyps, p.softmax.num_classes)) - # [time, batch, num_classes] -> [batch, time, num_classes] - logits = tf.transpose(logits, (1, 0, 2)) - - # Only return logits for the last ids - log_probs = tf.nn.log_softmax(tf.squeeze(logits, axis=1)) - - bs_results = py_utils.NestedMap({ - 'atten_probs': atten_probs, - 'log_probs': log_probs, - }) - - return bs_results, new_states - - def _PostBeamSearchStepCallback(self, theta, encoder_outputs, new_step_ids, - states): - # There is nothing to do here. - return states - - def _AddAttenProbsScalarSummary(self, source_paddings, targets, atten_probs): - """Add scalar summary of multi-headed transformer attention probs. - - This summary is primarily used to show statistics of the multi-headed - attention that reveals potential sparsity related properties. The - multi-headed attention probability tensors are exposed by - `MultiHeadedAttention.ComputeContextVectorWithSource` with the name - `multi_headed_atten_prob`. The following statistics are summarized: - - - 1_v_2: margin of the largest value vs. the 2nd largest - - 1_v_3: similar, but vs the 3rd largest - - mean: mean of the attention probs. NOTE: the sequences in a mini-batch - are not always of the same length. The attention probability for the - padded time index in target sequences are removed. However, the padding - for the source sequences are left unchanged. As a result, the atten - probs vectors will have some extra zero entries, so the mean calculated - here will be smaller than the true mean. - - source_padding_ratio: as explained above, the source paddings are not - handled when computing the mean. This summary show the average ratio - of time-steps that are padded values in the source sequences, to give - a reference of roughly how much the mean summarized above should be - adjusted. - - 1_v_mean: margin of the largest value vs the mean value. - - sum: the sum of the attention prob vectors. Should always be 1, for sanity - check only. - - The quantity above are computed for each sequence in the mini-batch, each - valid (target) sequence index, and each attention head, and then the - average value is reported to the tensorboard as a scalar summary. - - Args: - source_paddings: source padding, of shape [src_len, src_batch]. - targets: A dict of string to tensors representing the targets one try to - predict. Each tensor in targets is of shape [tgt_batch, tgt_len]. - atten_probs: a list of attention probs, each element is of shape [tgt_len, - tgt_batch, src_len]. - """ - default_graph = tf.get_default_graph() - # looks like fprop/wmt14_en_de_transformer/tower_0_0/dec - name_scope = default_graph.get_name_scope() - # NOTE: shapes - # source_paddings: [src_len, src_batch] - # targets.paddings: [tgt_batch, tgt_len]. - source_time = tf.shape(source_paddings)[0] - source_batch = tf.shape(source_paddings)[1] - target_time = tf.shape(targets.paddings)[1] - target_batch = tf.shape(targets.paddings)[0] - num_heads = self.trans[0].self_atten.params.num_attention_heads - with tf.control_dependencies([tf.assert_equal(source_batch, target_batch)]): - target_batch = tf.identity(target_batch) - - source_padding_ratio = tf.cast( - tf.reduce_sum(source_paddings, axis=0), tf.float32) - source_padding_ratio /= tf.cast(tf.shape(source_paddings)[0], tf.float32) - summary_utils.scalar('source_padding_ratio', - tf.reduce_mean(source_padding_ratio)) - - for i in range(len(atten_probs)): - suffix = '_{}'.format(i) if i > 0 else '' - # Tensor exported from MultiHeadedAttention.ComputeContextVectorWithSource - # shape [target_time * batch_size, num_heads, source_time] - try: - mha_probs = default_graph.get_tensor_by_name( - name_scope + ('/aux_atten{}/MultiHeadedAttention/' - 'ComputeContextVectorWithSource/' - 'multi_headed_atten_prob:0').format(suffix)) - except KeyError: - # no such tensor found, stop here - return - - mha_probs = tf.reshape( - mha_probs, (target_time, target_batch, num_heads, source_time)) - - # remove time padding from target_time - # (tgt_t, batch, n_heads, src_t) => (n_valid, n_heads, src_t) - # explicit reshape is used here to give masks static ndims, otherwise - # tf.boolean_mask will fail - masks = tf.reshape( - tf.equal(targets.paddings, 0), (target_time, target_batch)) - mha_probs = tf.boolean_mask(mha_probs, masks) - - # note we did not remove invalid entries according to source_paddings, - # because the result will no longer be a rectangular tensor, just - # remember when interpreting some statistics like mean, there are some - # padded zero entries due to non-uniform sequence lengths - - # (n_valid, n_heads, src_t) => (n_valid*n_heads, src_t) - mha_probs = tf.reshape(mha_probs, (-1, tf.shape(mha_probs)[-1])) - - probs_top3, _ = tf.math.top_k(mha_probs, k=3) - probs_mean = tf.math.reduce_mean(mha_probs, axis=1) - probs_sum = tf.math.reduce_sum(mha_probs, axis=1) # sanity check - - margins_12 = tf.reduce_mean(probs_top3[:, 0] - probs_top3[:, 1]) - margins_13 = tf.reduce_mean(probs_top3[:, 0] - probs_top3[:, 2]) - margins_1m = tf.reduce_mean(probs_top3[:, 0] - probs_mean) - summary_utils.scalar('1_v_2/atten{}'.format(i), margins_12) - summary_utils.scalar('1_v_3/atten{}'.format(i), margins_13) - summary_utils.scalar('1_v_mean/atten{}'.format(i), margins_1m) - summary_utils.scalar('mean/atten{}'.format(i), tf.reduce_mean(probs_mean)) - summary_utils.scalar('sum/atten{}'.format(i), tf.reduce_mean(probs_sum)) - - def _AddAttenProbsSummary(self, source_paddings, targets, atten_probs): - """Add summary of attention probs. - - Args: - source_paddings: source padding, of shape [src_len, src_batch]. - targets: A dict of string to tensors representing the targets one try to - predict. Each tensor in targets is of shape [tgt_batch, tgt_len]. - atten_probs: a list of attention probs, each element is of shape [tgt_len, - tgt_batch, src_len]. - """ - super()._AddAttenProbsSummary(source_paddings, targets, atten_probs) - if self.cluster.add_summary and self.params.add_multiheaded_attention_scalar_summary: - self._AddAttenProbsScalarSummary(source_paddings, targets, atten_probs) - - -class InsertionDecoder(base_decoder.BaseBeamSearchDecoder): - """Basic Insertion decoder for MT (or any symbol based sequence). - - References: - KERMIT: https://arxiv.org/pdf/1906.01604.pdf - Insertion Transformer: https://arxiv.org/pdf/1902.03249.pdf - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('token_emb', layers.EmbeddingLayer.Params(), - 'Token embedding layer params.') - p.Define('position_emb', layers.PositionalEmbeddingLayer.Params(), - 'Position embedding layer params.') - p.Define( - 'model_dim', 1024, 'Model dimension that applies to embedding ' - 'layers and all Transformer layers.') - p.Define('num_trans_layers', 6, 'Number of Transformer layers.') - p.Define('trans_tpl', layers_with_attention.TransformerLayer.Params(), - 'Transformer layer params.') - p.Define('softmax', layers.SimpleFullSoftmax.Params(), 'Softmax params.') - p.Define('input_dropout_prob', 0.0, 'Prob at which we do input dropout.') - - # Default config for the token embeddings. - p.token_emb.vocab_size = 32000 * 2 - p.token_emb.embedding_dim = p.model_dim - p.token_emb.max_num_shards = 16 - p.token_emb.params_init = py_utils.WeightInit.Gaussian( - 1.0 / math.sqrt(p.token_emb.embedding_dim)) - p.token_emb.scale_sqrt_depth = True - - # Default config for the position embeddings. - p.position_emb.embedding_dim = p.model_dim - - # Default config for the transformer layers. - p.trans_tpl.source_dim = p.model_dim - p.trans_tpl.tr_atten_tpl.source_dim = p.model_dim - p.trans_tpl.tr_atten_tpl.num_attention_heads = 8 - p.trans_tpl.tr_fflayer_tpl.input_dim = p.model_dim - p.trans_tpl.tr_fflayer_tpl.hidden_dim = 4096 - - # Default config for the softmax. - p.softmax.num_classes = 32000 - p.softmax.num_shards = 8 - - p.target_seq_len = 300 - - return p - - @classmethod - def UpdateTargetVocabSize(cls, p, vocab_size, wpm_model=None): - """Sets the vocab size in the params. - - Args: - p: model params. - vocab_size: size of the vocabulary. - wpm_model: file name prefix pointing to a wordpiece model. - - Returns: - Model params updated with the vocab size and wpm model. - """ - p.softmax.num_classes = vocab_size - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.token_emb.vocab_size % p.softmax.num_classes == 0 - assert p.token_emb.embedding_dim == p.position_emb.embedding_dim - assert p.token_emb.embedding_dim == p.model_dim - - self.CreateChild('token_emb', p.token_emb) - self.CreateChild('position_emb', p.position_emb) - - dropout_tpl = layers.DropoutLayer.Params() - dropout_tpl.keep_prob = (1.0 - p.input_dropout_prob) - self.CreateChild('input_dropout', dropout_tpl) - - params_trans_layers = [] - for i in range(p.num_trans_layers): - params = p.trans_tpl.Copy() - params.name = 'trans_layer_%d' % i - params.packed_input = p.packed_input - params.has_aux_atten = False - params.mask_self_atten = True - params_trans_layers.append(params) - self.CreateChildren('trans', params_trans_layers) - - p.softmax.input_dim = p.model_dim - self.CreateChild('softmax', p.softmax) - - def ComputePredictions(self, theta, encoder_outputs, targets): - """Compute 1-step of the insertion iteration. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - encoder_outputs: This should be None. - targets: A `.NestedMap`. - - ids: The target ids of shape [batch_size, time_dim]. - - paddings: The target paddings of shape [batch_size, time_dim]. - - Returns: - A `.NestedMap`. - - outputs: The contextualized output vectors of shape - [batch_size, time_dim, model_dim]. - """ - p = self.params - - # TODO(williamchan): Enable cross-attention. - assert encoder_outputs is None - - with tf.name_scope(p.name): - # [batch, time] - target_ids = targets.ids - # [time, batch] - target_paddings = tf.transpose(targets.paddings) - - # Embedding layer - # [batch, time, model_dim] - token_embs = self.token_emb.EmbLookup(theta.token_emb, target_ids) - target_time = py_utils.GetShape(target_ids)[1] - - # [1, time, model_dim] - posit_embs = tf.expand_dims( - self.position_emb.FProp(theta.position_emb, target_time), 0) - - # [time, batch, model_dim] - input_embs = token_embs + posit_embs - - input_embs = tf.transpose(input_embs, [1, 0, 2]) - input_embs = self.input_dropout.FProp(theta.input_dropout, input_embs) - - layer_in = input_embs - for layer, layer_theta in zip(self.trans, theta.trans): - # [time, batch, model_dim] - layer_out, _ = layer.FProp(layer_theta, layer_in, target_paddings) - layer_in = layer_out - - return py_utils.NestedMap(outputs=layer_out) - - def ComputeLoss(self, theta, predictions, targets): - # pyformat: disable - """Returns the insertion loss. - - Args: - theta: A `.NestedMap` object capturing decoder model parameters. - predictions: A `.NestedMap` describing the decoding process, requiring - .outputs: Tensor of shape [time, batch, params.softmax.input_dim]. - targets: A `.NestedMap`. - - - target_indices: A Tensor capturing the relevant insertion tokens to - tf.gather_nd the log-probs. - - - target_weights: A Tensor capturing the relevant insertion tokens' - weights. - - Returns: - Two dicts. - - A map from metric name (a python string) to a tuple (value, weight). - Both value and weight are scalar Tensors. - - A map from name to arbitrary tensors, where the first dimension must - be the batch index. - """ - # pyformat: enable - p = self.params - - batch_size = py_utils.GetShape(predictions.outputs)[0] - - state = tf.reshape(predictions.outputs, [-1, p.softmax.input_dim]) - logits = self.softmax.Logits(theta.softmax, state) - logits = tf.reshape( - logits, - tf.concat([ - py_utils.GetShape(predictions.outputs)[:-1], - [p.softmax.num_classes] - ], 0)) - log_probs = tf.nn.log_softmax(logits) - - # `target_indices` are in the form [batch, time, vocab], where as `logits` - # are in the form [time, batch, vocab]. We need to swap the columns. - target_indices = tf.concat([ - predictions.tgt.target_indices[:, 1:2], - predictions.tgt.target_indices[:, 0:1], - predictions.tgt.target_indices[:, 2:3], - ], 1) - - loss = tf.reduce_sum( - tf.gather_nd(log_probs, target_indices) * - predictions.tgt.target_weights) - loss_weight = tf.cast(batch_size, tf.float32) - - return ({ - 'loss': (loss, loss_weight) - }, { - 'log_probs': log_probs, - 'logits': logits - }) - - -class TransformerBatchMajorDecoder(MTBaseDecoder): - """Transformer decoder with batch major implementation. - - Implements the decoder of Transformer model: - https://arxiv.org/abs/1706.03762. - """ - - @classmethod - def Params(cls): - p = super().Params() - p.Define('token_emb', layers.EmbeddingLayer.Params(), - 'Token embedding layer params.') - p.Define('shared_emb', None, 'Embedding shared with softmax.') - p.Define('position_emb', layers.PositionalEmbeddingLayer.Params(), - 'Position embedding layer params.') - p.Define('source_dim', 1024, 'Dimension of encoder outputs.') - p.Define( - 'model_dim', 1024, 'Model dimension that applies to embedding ' - 'layers and all Transformer layers.') - p.Define('num_trans_layers', 6, 'Number of Transformer layers.') - p.Define('trans_decoder_tpl', - batch_major_attention.TransformerDecoderLayer.Params(), - 'Transformer layer params.') - p.Define('input_dropout_prob', 0.0, 'Prob at which we do input dropout.') - p.Define('input_dropout_tpl', layers.DropoutLayer.Params(), - 'Input dropout layer params.') - p.Define('final_layer_norm', False, - 'Whether or not to apply layer norm after transformer stack.') - p.Define('use_fused_layernorm', False, 'Whether to use fused layernorm.') - p.Define('use_fast_softmax', False, - 'Whether or not to use a faster softmax with label smoothing.') - p.Define( - 'input_data_format', 'TBC', 'The data format of input features: ' - 'TBC for [time, batch, feature_dim], ' - 'BTC for [batch, time, feature_dim].') - p.Define( - 'prediction_data_format', 'TBC', - 'The data format of predictions and per-example losses: ' - 'TBC for [time, batch, ...], ' - 'BTC for [batch, time, ...].') - - # Default config for the token embedding. - p.token_emb.vocab_size = 32000 - p.token_emb.embedding_dim = p.model_dim - p.token_emb.max_num_shards = 16 - p.token_emb.params_init = py_utils.WeightInit.Gaussian( - 1.0 / math.sqrt(p.token_emb.embedding_dim)) - p.token_emb.scale_sqrt_depth = True - - # Default config for the position embedding. - p.position_emb.embedding_dim = p.model_dim - - # Default config for the transformer decoder layers. - p.trans_decoder_tpl.input_dim = p.model_dim - p.trans_decoder_tpl.tr_atten_tpl.input_dim = p.model_dim - p.trans_decoder_tpl.tr_atten_tpl.num_heads = 8 - p.trans_decoder_tpl.tr_fflayer_tpl.input_dim = p.model_dim - p.trans_decoder_tpl.tr_fflayer_tpl.hidden_dim = 2048 - - # Default config for beam search. - p.target_seq_len = 300 - p.beam_search.length_normalization = 0.5 - p.beam_search.coverage_penalty = 0.0 - p.beam_search.batch_major_state = False - p.beam_search.batch_major_compute = True - p.beam_search.short_seq_limit = 40 - - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - if p.shared_emb: - self.CreateChild('softmax', p.shared_emb) - - if not p.shared_emb: - self.CreateChild('token_emb', p.token_emb) - self.CreateChild('position_emb', p.position_emb) - - dropout_tpl = p.input_dropout_tpl.Copy() - dropout_tpl.keep_prob = (1.0 - p.input_dropout_prob) - self.CreateChild('input_dropout', dropout_tpl) - - params_trans_layers = [] - for i in range(p.num_trans_layers): - params = p.trans_decoder_tpl.Copy() - params.name = 'decoder_trans_layer_%d' % i - params_trans_layers.append(params) - self.CreateChildren('decoder_trans', params_trans_layers) - - p.softmax.input_dim = p.model_dim - if not p.shared_emb: - self.CreateChild('softmax', p.softmax) - - if p.final_layer_norm: - layer_norm_p = layers.LayerNorm.Params().Set( - name='final_ln', - input_dim=p.model_dim, - use_fused_layernorm=p.use_fused_layernorm, - fprop_dtype=p.input_dropout_tpl.fprop_dtype) - self.CreateChild('final_ln', layer_norm_p) - - def _CreateChildrenVariables(self): - if self.params.shared_emb: - with tf.variable_scope('shared_emb', reuse=tf.AUTO_REUSE): - self.softmax.InstantiateVariables() - super()._CreateChildrenVariables() - - def _MaybeTransposeEncoderOutputs(self, encoder_outputs, target_data_format): - p = self.params - if p.input_data_format == target_data_format: - return encoder_outputs - transposed = py_utils.NestedMap( - encoded=tf.transpose(encoder_outputs.encoded, [1, 0, 2]), - padding=tf.transpose(encoder_outputs.padding)) - if getattr(encoder_outputs, 'segment_id', None) is None: - transposed.segment_id = None - else: - transposed.segment_id = tf.transpose(encoder_outputs.segment_id) - return transposed - - def _MaybeTransposeTargets(self, targets): - p = self.params - if p.prediction_data_format == 'BTC': - return targets - transposed = py_utils.NestedMap() - for k, v in targets.items(): - if v is not None: - with tf.name_scope('transpose_%s' % k): - v = tf.transpose(py_utils.HasShape(v, [-1, -1])) - transposed[k] = v - return transposed - - def _FProp(self, theta, encoder_outputs, targets): - """Decodes `targets` given encoded source. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - encoder_outputs: A '.NestedMap' object computed by encoder. * encoded - - Source encoding of shape [source_time, source_batch, dim] or - [source_batch, source_time, dim], depending on p.input_data_format. * - paddings - Source encoding's padding of shape [source_time, - source_batch] or [source_batch, source_time]. - targets: A dict of string to tensors representing the targets one try to - predict. Each tensor in targets is of shape [batch, target_time]. - - Returns: - softmax_input: Tensor of shape [target_time, batch, dim]. - """ - p = self.params - # [batch, source_time, dim] - encoder_out_bm = self._MaybeTransposeEncoderOutputs(encoder_outputs, 'BTC') - aux_vec = encoder_out_bm.encoded - aux_paddings = encoder_out_bm.padding - aux_segment_id = getattr(encoder_out_bm, 'segment_id', None) - - with tf.name_scope(p.name): - # [batch, target_time] - target_ids = targets.ids - target_paddings = targets.paddings - target_time = py_utils.GetShape(target_ids)[1] - target_segment_pos = None - target_segment_id = None - if p.packed_input: - target_segment_id = targets.segment_ids - target_segment_pos = targets.segment_pos - assert aux_segment_id is not None, ('Need to provide aux_segment_id ' - 'for packed input.') - - # Embedding layer - # [batch, target_time, dim] - if not p.shared_emb: - token_embs = self.token_emb.EmbLookup(theta.token_emb, target_ids) - else: - token_embs = self.softmax.EmbLookup(theta.softmax, target_ids) - # [1, target_time, dim] - if p.packed_input: - posit_embs = self.position_emb.FPropWithPosition( - theta.position_emb, target_segment_pos) - else: - posit_embs = tf.expand_dims( - self.position_emb.FProp(theta.position_emb, target_time), 0) - # [batch, target_time, dim] - input_embs = token_embs + posit_embs - - if p.input_dropout_tpl.fprop_dtype: - input_embs = tf.cast(input_embs, p.input_dropout_tpl.fprop_dtype) - target_paddings = tf.cast(target_paddings, - p.input_dropout_tpl.fprop_dtype) - - input_embs = self.input_dropout.FProp(theta.input_dropout, input_embs) - layer_in = input_embs - # Explicitly set the input shape of Transformer layers, to avoid - # unknown shape error occurred to tf.einsum on nonTPU devices. - batch, _, dim = py_utils.GetShape(aux_vec, 3) - layer_in = tf.reshape(layer_in, [batch, target_time, dim]) - if p.packed_input: - segment_mask = batch_major_attention.SegmentMask( - target_segment_id, target_segment_id, dtype=layer_in.dtype) - causal_padding = tf.expand_dims( - tf.tile( - tf.expand_dims( - batch_major_attention.CausalPadding( - target_time, dtype=layer_in.dtype), 0), [batch, 1, 1]), - 1) - causal_mask = causal_padding * segment_mask.dtype.max * tf.constant( - -0.7, dtype=segment_mask.dtype) - segment_mask += causal_mask - aux_segment_mask = batch_major_attention.SegmentMask( - target_segment_id, aux_segment_id, dtype=layer_in.dtype) - for layer, layer_theta in zip(self.decoder_trans, theta.decoder_trans): - # [batch, target_time, dim] - layer_out, _ = layer.FProp( - layer_theta, - layer_in, - target_paddings, - aux_vec, - aux_paddings, - segment_mask=segment_mask if p.packed_input else None, - aux_segment_mask=aux_segment_mask if p.packed_input else None) - layer_in = layer_out - - if p.final_layer_norm: - layer_out = self.final_ln.FProp(theta.final_ln, layer_out) - if p.prediction_data_format == 'TBC': - # Transpose the softmax_input to match the input requirement of - # ComputePredictions. - layer_out = tf.transpose(layer_out, [1, 0, 2]) - return layer_out - - def ExtendStep(self, - theta, - encoder_outputs, - new_ids, - time_step, - prefix_states, - use_short_seq_opt=False): - """Extend prefix as represented by `prefix_states` by one more step. - - This function is expected to be called during fast decoding of Transformer - models. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - encoder_outputs: A '.NestedMap' object computed by encoder. * encoded - - Source encoding of shape [source_time, source_batch, dim] or - [source_batch, source_time, dim], depending on p.input_data_format. * - paddings - Source encoding's padding of shape [source_time, - source_batch] or [source_batch, source_time]. - new_ids: New input ids, of shape [target_batch, 1]. - time_step: A scalar, the current decode step, 0-based. - prefix_states: A `.NestedMap` representing the previous decoded states. - key - [target_time, target_batch, num_heads, dim_per_head]. value - - [target_time, target_batch, num_heads, dim_per_head]. - use_short_seq_opt: A bool, whether using short sequence optimization. - - Returns: - last_decoder_out: The last decoder layer of shape [target_batch, dim]. - updated_prefix_states: A `.NestedMap` representing the updated states. - key - [target_time, target_batch, num_heads, dim_per_head]. - value - [target_time, target_batch, num_heads, dim_per_head]. - """ - p = self.params - encoder_out_bm = self._MaybeTransposeEncoderOutputs(encoder_outputs, 'BTC') - # [source_batch, source_time, dim] - aux_vec = encoder_out_bm.encoded - # [source_batch, source_time] - aux_paddings = encoder_out_bm.padding - - with tf.name_scope(p.name): - # Embedding layer - # [target_batch, 1, dim] - if not p.shared_emb: - token_embs = self.token_emb.EmbLookup(theta.token_emb, new_ids) - else: - token_embs = self.softmax.EmbLookup(theta.softmax, new_ids) - # [1, 1, dim] - if isinstance(time_step, tf.Tensor): - time_step_t = tf.reshape(time_step, [1, 1]) - elif isinstance(time_step, int): - time_step_t = tf.constant([[time_step]], dtype=tf.int32) - else: - raise ValueError('Unexpected input type `%s` for `time_step`.' % - type(time_step)) - posit_embs = self.position_emb.FPropWithPosition(theta.position_emb, - time_step_t) - # [target_batch, 1, dim] - input_embs = token_embs + posit_embs - - if p.input_dropout_tpl.fprop_dtype: - input_embs = tf.cast(input_embs, p.input_dropout_tpl.fprop_dtype) - - # Make a copy of the input. - updated_prefix_states = prefix_states.DeepCopy() - - input_embs = self.input_dropout.FProp(theta.input_dropout, input_embs) - layer_in = input_embs - for i, (layer, layer_theta) in enumerate( - zip(self.decoder_trans, theta.decoder_trans)): - # [target_batch, 1, dim] - layer_out, updated_states = layer.ExtendStep( - layer_theta, layer_in, aux_vec, aux_paddings, - prefix_states['layer_%i' % i], time_step, use_short_seq_opt) - updated_prefix_states['layer_%i' % i] = updated_states - layer_in = layer_out - - # [target_batch, dim] - last_decoder_out = tf.squeeze(layer_out, 1) - if p.final_layer_norm: - last_decoder_out = self.final_ln.FProp(theta.final_ln, last_decoder_out) - return last_decoder_out, updated_prefix_states - - def ComputePredictions(self, theta, encoder_outputs, targets): - """Decodes `targets` given encoded source. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - encoder_outputs: A '.NestedMap' object computed by encoder. * encoded - - Source encoding of shape [source_time, source_batch, dim] or - [source_batch, source_time, dim], depending on p.input_data_format. * - paddings - Source encoding's padding of shape [source_time, - source_batch] or [source_batch, source_time]. - targets: A dict of string to tensors representing the targets one try to - predict. Each tensor in targets is of shape [batch, target_time]. - - Returns: - Output of the last decoder layer, of shape [target_time, batch, dim]. - """ - return self._FProp(theta, encoder_outputs, targets) - - def _FPropFastSoftmax(self, - theta, - softmax_input, - target_labels, - target_weights, - time_axis=0): - """Computes cross-entropy loss with label smoothing. - - As compared to the _FPropSoftmax, this version is faster by removing the - data formatting overheads and bias of the linear projection. A normalizing - factor is also added to the xentropy result be better model quality. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - softmax_input: A tensor of shape [time, batch, p.softmax.input_dim]. - target_labels: A matrix of tf.int32. [time, batch]. - target_weights: A matrix of params.dtype. [time, batch]. - time_axis: If 0, the inputs are time-major: [time, batch, ...]; if 1, the - inputs are batch-major: [batch, time, ...]. - - Returns: - A tuple (metrics, per_example_tensors). - metrics: - A dictionary containing metrics for the xent loss and prediction - accuracy. - per_example_tensors: - A dictionary of per-example tensors. - """ - p = self.params - assert p.label_smoothing is not None - assert p.per_word_avg_loss - - softmax_input = tf.reshape(softmax_input, [-1, p.softmax.input_dim]) - - logits = self.softmax.SimpleLogits(theta.softmax, softmax_input) - logits = tf.cast(logits, tf.float32) - - high_confidence = 1.0 - p.label_smoothing.uncertainty - low_confidence = p.label_smoothing.uncertainty / tf.cast( - p.label_smoothing.num_classes - 1, tf.float32) - normalizing = -( - high_confidence * tf.math.log(high_confidence) + - tf.cast(p.softmax.num_classes - 1, tf.float32) * low_confidence * - tf.math.log(low_confidence + 1e-20)) - - target_labels = tf.reshape(target_labels, [-1]) - soft_targets = tf.one_hot( - tf.cast(target_labels, tf.int32), - depth=p.softmax.num_classes, - on_value=high_confidence, - off_value=low_confidence) - - xentropy = tf.nn.softmax_cross_entropy_with_logits( - logits=logits, labels=soft_targets) - xent = xentropy - normalizing - - target_weights_shape = py_utils.GetShape(target_weights) - orig_target_weights = target_weights - target_weights = tf.cast(tf.reshape(target_weights, [-1]), xent.dtype) - total_xent = tf.reduce_sum(xent * target_weights) - total_weights = tf.reduce_sum(target_weights) - - final_loss = total_xent / total_weights - loss_weight = total_weights - - metrics = { - 'loss': (final_loss, loss_weight), - 'log_pplx': (final_loss, loss_weight), - } - - per_example_tensors = {} - if p.per_example_tensors: - per_example_tensors['per_example_loss'] = tf.reshape( - xent, target_weights_shape) - per_example_tensors['per_sequence_loss'] = tf.reduce_sum( - per_example_tensors['per_example_loss'] * orig_target_weights, - axis=time_axis) - per_example_tensors['loss'] = per_example_tensors['per_sequence_loss'] - per_example_tensors['logits'] = tf.reshape( - logits, tf.concat([target_weights_shape, [-1]], 0)) - per_example_tensors['log_probs'] = tf.reshape( - tf.nn.log_softmax(logits), tf.concat([target_weights_shape, [-1]], 0)) - - # NOTE: tf.argmax is not implemented for the JF backend, see b/36093673 - # Skip the fraction_of_correct_next_step_preds during training. - if self.do_eval: - correct_preds = tf.cast( - tf.equal( - tf.cast(tf.reshape(tf.argmax(logits, 1), [-1]), tf.int32), - tf.reshape(target_labels, [-1])), p.dtype) - correct_next_preds = tf.reduce_sum( - correct_preds * tf.reshape(tf.cast(target_weights, p.dtype), [-1])) - num_preds = tf.reduce_sum(tf.cast(target_weights, p.dtype)) - accuracy = tf.identity( - correct_next_preds / num_preds, - name='fraction_of_correct_next_step_preds') - metrics['fraction_of_correct_next_step_preds'] = (accuracy, num_preds) - return metrics, per_example_tensors - - def ComputeLoss(self, theta, predictions, targets): - """Populates a metrics dictionary based on the output of ComputePredictions. - - Args: - theta: Nested map describing decoder model parameters. - predictions: NestedMap describing the decoding process, requiring: - .softmax_input: Tensor of shape [time, batch, params.softmax.input_dim]. - targets: NestedMap describing the target sequences. - - Returns: - Two dicts. - - - A map from metric name (a python string) to a tuple (value, weight). - Both value and weight are scalar Tensors. - - A map from name to arbitrary tensors, where the first dimension must - be the batch index. - """ - p = self.params - targets = self._MaybeTransposeTargets(targets) - if isinstance(predictions, py_utils.NestedMap): - predictions = predictions.softmax_input - time_axis = {'TBC': 0, 'BTC': 1}.get(p.prediction_data_format) - if p.use_fast_softmax: - return self._FPropFastSoftmax( - theta, - predictions, - targets.labels, - targets.weights, - time_axis=time_axis) - else: - return self._FPropSoftmax( - theta, - predictions, - targets.labels, - targets.weights, - targets.paddings, - targets.get('segment_ids', None), - time_axis=time_axis) - - def _InitBeamSearchStateCallback(self, theta, encoder_outputs, - num_hyps_per_beam): - """Returns initial beams search states. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - encoder_outputs: A '.NestedMap' object computed by encoder. * encoded - - Source encoding of shape [source_time, source_batch, dim] or - [source_batch, source_time, dim], depending on p.input_data_format. * - paddings - Source encoding's padding of shape [source_time, - source_batch] or [source_batch, source_time]. - num_hyps_per_beam: An int, number hyps to keep for source sentence. - - Returns: - initial_results: A `.NestedMap` of initial beam search results. - log_probs - Log prob for each of the tokens in the target vocab, - of shape [target_batch, vocab_size]. - atten_probs - The updated attention probs, of shape - [target_batch, source_time]. - states: A `.NestedMap` of initial model states. - prefix_states - A `.NestedMap` representing the empty decoded states. - key - [target_time, target_batch, num_heads, dim_per_head]. - value - [target_time, target_batch, num_heads, dim_per_head]. - time_step - A scalar, the initial decode step (0). - """ - p = self.params - - # [source_batch, source_time, dim] - encoder_out_bm = self._MaybeTransposeEncoderOutputs(encoder_outputs, 'BTC') - aux_vec = encoder_out_bm.encoded - target_batch = py_utils.GetShape(aux_vec)[0] * num_hyps_per_beam - source_time = py_utils.GetShape(aux_vec)[1] - target_time = p.target_seq_len - - log_probs = tf.zeros([target_batch, p.softmax.num_classes], - dtype=py_utils.FPropDtype(p)) - # Dummy attention probs - atten_probs = ( - tf.ones([target_batch, source_time], dtype=py_utils.FPropDtype(p)) / - tf.cast(source_time, py_utils.FPropDtype(p))) - initial_results = py_utils.NestedMap( - log_probs=log_probs, atten_probs=atten_probs) - - prefix_states = py_utils.NestedMap() - for layer in range(p.num_trans_layers): - prefix_states['layer_%d' % layer] = self.decoder_trans[layer].InitStates( - theta.decoder_trans[layer], target_batch, target_time) - - return initial_results, py_utils.NestedMap({ - 'prefix_states': prefix_states, - 'time_step': tf.constant(0) - }) - - def _PreBeamSearchStepCallback(self, - theta, - encoder_outputs, - new_ids, - states, - num_hyps_per_beam, - use_short_seq_opt=False): - """Returns logits for sampling ids and the next model states. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - encoder_outputs: A '.NestedMap' object computed by encoder. * encoded - - Source encoding of shape [source_time, source_batch, dim] or - [source_batch, source_time, dim], depending on p.input_data_format. * - paddings - Source encoding's padding of shape [source_time, - source_batch] or [source_batch, source_time]. - new_ids: A tensor of shape [target_batch, 1]. - states: A `.NestedMap` of tensors representing states that the clients - would like to keep track of for each of the active hyps. prefix_states - - A `.NestedMap` representing the previous decoded states. key - - [target_time, target_batch, num_heads, dim_per_head]. value - - [target_time, target_batch, num_heads, dim_per_head]. time_step - A - scalar, the current decode step, 0-based. - num_hyps_per_beam: A scalar, beam size. - use_short_seq_opt: A bool, whether using short sequence optimization. - - Returns: - bs_results: A `.NestedMap` of beam search results. - log_probs - Log prob for each of the tokens in the target vocab, - of shape [target_batch, vocab_size]. - atten_probs - The updated attention probs, of shape - [target_batch, source_time]. - new_states: A `.NestedMap` object. The updated states. - prefix_states - A `.NestedMap` representing the updated decoded states. - key - [target_time, target_batch, num_heads, dim_per_head]. - value - [target_time, target_batch, num_heads, dim_per_head]. - time_step - A scalar, the current decode step, 0-based. - """ - p = self.params - # [source_batch, source_time, dim] - encoder_out_bm = self._MaybeTransposeEncoderOutputs(encoder_outputs, 'BTC') - - target_batch = py_utils.GetShape(new_ids)[0] - source_batch = target_batch // num_hyps_per_beam - - new_states = states.Pack(states.Flatten()) - time_step = states.time_step - prefix_states = states.prefix_states - - # The inputs are ordered as num_hyps_per_beam by num_beams, - # which needs to be transposed for the layer computation. - # [num_hyps_per_beam, source_batch, 1] - new_ids = tf.reshape(new_ids, [num_hyps_per_beam, source_batch, 1]) - # [source_batch, num_hyps_per_beam, 1] - new_ids = tf.transpose(new_ids, [1, 0, 2]) - # [source_batch * num_hyps_per_beam, 1] - new_ids = tf.reshape(new_ids, [-1, 1]) - - softmax_input, updated_prefix_states = self.ExtendStep( - theta, encoder_outputs, new_ids, time_step, prefix_states, - use_short_seq_opt) - - # Transpose the outputs as num_beams by num_hyps_per_beam to match the - # beam search requirement. - # [source_batch, num_hyps_per_beam, dim] - softmax_input = tf.reshape(softmax_input, - [source_batch, num_hyps_per_beam, -1]) - # [num_hyps_per_beam, source_batch, dim] - softmax_input = tf.transpose(softmax_input, [1, 0, 2]) - # [num_hyps_per_beam * source_batch, dim] - softmax_input = tf.reshape(softmax_input, [target_batch, -1]) - - # [target_batch, vocab_size] - logits = self.softmax.Logits(theta.softmax, [softmax_input]) - - # Only return logits for the last ids - log_probs = tf.nn.log_softmax(logits) - - # Dummy attention probs - source_time = py_utils.GetShape(encoder_out_bm.padding)[1] - atten_probs = ( - tf.ones([target_batch, source_time], dtype=py_utils.FPropDtype(p)) / - tf.cast(source_time, py_utils.FPropDtype(p))) - - bs_results = py_utils.NestedMap({ - 'log_probs': log_probs, - 'atten_probs': atten_probs, - }) - - new_states.prefix_states = updated_prefix_states - new_states.time_step = time_step + 1 - - return bs_results, new_states - - def _PostBeamSearchStepCallback(self, theta, encoder_outputs, new_step_ids, - states): - # There is nothing to do here. - return states diff --git a/lingvo/tasks/mt/decoder_test.py b/lingvo/tasks/mt/decoder_test.py deleted file mode 100644 index 3b239df6..00000000 --- a/lingvo/tasks/mt/decoder_test.py +++ /dev/null @@ -1,1374 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for mt.decoder.""" - -from absl.testing import parameterized -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import input_generator_helper as ig_helper -from lingvo.core import layers -from lingvo.core import layers_with_attention -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.core.ops.hyps_pb2 import Hypothesis -from lingvo.core.test_utils import CompareToGoldenSingleFloat -from lingvo.tasks.mt import decoder -import numpy as np - -FLAGS = tf.flags.FLAGS - -_NUMPY_RANDOM_SEED = 9885784 -_TF_RANDOM_SEED = 8372749040 - - -class DecoderTestCaseBase(test_utils.TestCase): - - def _Inputs(self, dtype=tf.float32, init_step_ids=False): - np.random.seed(_NUMPY_RANDOM_SEED) - src_seq_len = 5 - # batch = 2 - src_enc = tf.constant( - np.random.normal(size=[src_seq_len, 2, 4]), dtype=dtype) - src_enc_padding = tf.constant( - [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 1.0], [1.0, 1.0]], - dtype=dtype) - # batch = 4, time = 3. - target_ids = tf.transpose( - tf.constant([[0, 1, 2, 3], [0, 5, 6, 7], [0, 10, 11, 12]], - dtype=tf.int32)) - target_labels = tf.transpose( - tf.constant([[1, 2, 3, 4], [5, 6, 7, 8], [10, 11, 12, 13]], - dtype=tf.int32)) - target_paddings = tf.transpose( - tf.constant([[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 1, 0]], dtype=dtype)) - target_weights = 1.0 - target_paddings - targets = py_utils.NestedMap({ - 'ids': target_ids, - 'labels': target_labels, - 'weights': target_weights, - 'paddings': target_paddings - }) - encoder_outputs = py_utils.NestedMap( - encoded=src_enc, padding=src_enc_padding, segment_id=None) - if init_step_ids: - tgt_prefix = tf.constant(np.random.randint(4, size=[2]), dtype=tf.int32) - encoder_outputs['init_step_ids'] = tgt_prefix - return encoder_outputs, targets - - def _DecoderParams( - self, - per_word_avg_loss=False, - dtype=tf.float32, - fprop_dtype=None, - decoder_cls=decoder.MTDecoderV1, - ): - p = decoder_cls.Params() - p.name = 'decoder' - p.source_dim = 4 - p.emb.vocab_size = 16 - p.emb.embedding_dim = 4 - p.emb.max_num_shards = 1 - p.rnn_cell_dim = 4 - p.rnn_layers = 3 - p.attention.hidden_dim = 2 - p.softmax.num_classes = 16 - p.softmax.num_shards = 1 - p.per_word_avg_loss = per_word_avg_loss - p.dtype = dtype - p.target_seq_len = 5 - p.random_seed = 12345 - p.emb.params_init = py_utils.WeightInit.Uniform(0.04, 12345) - p.atten_rnn_cell_tpl.params_init = py_utils.WeightInit.Uniform(0.04, 12345) - p.rnn_cell_tpl.params_init = py_utils.WeightInit.Uniform(0.04, 12345) - p.softmax.params_init = py_utils.WeightInit.Uniform(0.04, 123) - - for lp in base_layer.RecursiveFindLayerParams(p): - lp.dtype = dtype - - if fprop_dtype: - py_utils.UpdateFpropDtype(p, fprop_dtype) - - return p - - def _DecoderFPropHelper(self, - decoder_cls, - dtype, - fprop_dtype, - feed_att_context_to_softmax, - expected_loss, - per_example_tensors=False): - with self.session(use_gpu=True): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams( - dtype=dtype, fprop_dtype=fprop_dtype, decoder_cls=decoder_cls) - p.per_example_tensors = per_example_tensors - - p.feed_attention_context_vec_to_softmax = feed_att_context_to_softmax - dec = p.Instantiate() - encoder_outputs, targets = self._Inputs(dtype=fprop_dtype) - fprop_out = dec.FPropDefaultTheta(encoder_outputs, targets) - loss = fprop_out.metrics['loss'][0] - - self.evaluate(tf.global_variables_initializer()) - actual_loss = loss.eval() - print('actual loss = ', actual_loss) - CompareToGoldenSingleFloat(self, expected_loss, actual_loss) - if per_example_tensors: - per_example = fprop_out.per_sequence - self.assertIn('loss', per_example) - self.assertAllEqual(per_example['loss'].shape.as_list(), [4]) - - def _DecoderGradientCheckerHelper(self, - decoder_cls, - feed_att_context_to_softmax=False): - with self.session(use_gpu=True, graph=tf.Graph()) as sess: - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(dtype=tf.float64, decoder_cls=decoder_cls) - p.feed_attention_context_vec_to_softmax = feed_att_context_to_softmax - dec = p.Instantiate() - encoder_outputs, targets = self._Inputs(dtype=tf.float64) - loss, _ = dec.FPropDefaultTheta(encoder_outputs, targets).metrics['loss'] - all_vars = tf.trainable_variables() - grads = tf.gradients(loss, all_vars) - print('num of vars ', len(all_vars)) - - def DenseGrad(var, grad): - if isinstance(grad, tf.Tensor): - return grad - elif isinstance(grad, tf.IndexedSlices): - return tf.math.unsorted_segment_sum(grad.values, grad.indices, - tf.shape(var)[0]) - - grads = [DenseGrad(x, y) for x, y in zip(all_vars, grads)] - - self.evaluate(tf.global_variables_initializer()) - symbolic_grads = [gd.eval() for gd in grads] - numerical_grads = [] - for v in all_vars: - numerical_grads.append( - test_utils.ComputeNumericGradient(sess, loss, v, delta=1e-5)) - - rets = {} - for v, x, y in zip(all_vars, symbolic_grads, numerical_grads): - print('symbolic_grads, numerical_grads :', v.name) - print(x) - print(y) - self.assertAllClose(x, y) - rets[v.name] = x - - return rets - - def _DecoderPerWordAvgLossFPropHelper(self, - decoder_cls, - feed_att_context_to_softmax=False): - with self.session(use_gpu=True): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(True, decoder_cls=decoder_cls) - p.feed_attention_context_vec_to_softmax = feed_att_context_to_softmax - dec = p.Instantiate() - encoder_outputs, targets = self._Inputs() - loss, _ = dec.FPropDefaultTheta(encoder_outputs, targets).metrics['loss'] - self.evaluate(tf.global_variables_initializer()) - actual_loss = loss.eval() - print('actual loss = ', actual_loss) - if p.feed_attention_context_vec_to_softmax: - CompareToGoldenSingleFloat(self, 2.768977, actual_loss) - else: - CompareToGoldenSingleFloat(self, 2.772613, actual_loss) - - -class DecoderTest(DecoderTestCaseBase, parameterized.TestCase): - - def testDecoderConstruction(self): - p = self._DecoderParams() - _ = decoder.MTDecoderV1(p) - - def testDecoderFPropFunctional(self): - self._DecoderFPropHelper(decoder.MTDecoderV1, tf.float64, tf.float64, False, - 7.624605) - - def testDecoderFPropFunctionalFloat64Dtype(self): - self._DecoderFPropHelper(decoder.MTDecoderV1, tf.float32, tf.float64, False, - 7.624684) - - def testDecoderFPropFunctionalFloat64FpropDtype(self): - self._DecoderFPropHelper(decoder.MTDecoderV1, tf.float64, tf.float32, False, - 7.624604) - - def testDecoderFPropFunctionalFeedingAttContext(self): - self._DecoderFPropHelper(decoder.MTDecoderV1, tf.float64, tf.float64, True, - 7.640674) - - def testDecoderFPropPerExampleTensors(self): - self._DecoderFPropHelper( - decoder.MTDecoderV1, - tf.float64, - tf.float64, - False, - 7.624605, - per_example_tensors=True) - - def testDecoderBPropFunctional(self): - self._DecoderGradientCheckerHelper(decoder.MTDecoderV1) - - def testDecoderBPropFunctionalFeedingAttContext(self): - self._DecoderGradientCheckerHelper( - decoder.MTDecoderV1, feed_att_context_to_softmax=True) - - def testDecoderPerWordAvgLossFPropFunctional(self): - self._DecoderPerWordAvgLossFPropHelper(decoder.MTDecoderV1) - - def testDecoderPerWordAvgLossFPropFunctionalFeedingAttContext(self): - self._DecoderPerWordAvgLossFPropHelper( - decoder.MTDecoderV1, feed_att_context_to_softmax=True) - - def testBeamSearchDecode(self, dtype=tf.float32): - with self.session(use_gpu=True), self.SetEval(True): - tf.random.set_seed(_TF_RANDOM_SEED) - src_batch = 2 - p = self._DecoderParams(dtype=dtype) - src_time = p.target_seq_len - p.beam_search.num_hyps_per_beam = 2 - p.rnn_cell_dim = 32 - dec = decoder.MTDecoderV1(p) - encoder_outputs, _ = self._Inputs(dtype=dtype) - decode = dec.BeamSearchDecode(encoder_outputs) - # topk_decoded is None in MT decoder, set it to a fake tensor to pass - # self.evaluate(decode). - decode = decode._replace(topk_decoded=tf.constant(0, tf.float32)) - - self.evaluate(tf.global_variables_initializer()) - actual_decode = self.evaluate(decode) - - self.assertTupleEqual( - (src_time, src_batch * p.beam_search.num_hyps_per_beam), - actual_decode.done_hyps.shape) - self.assertTupleEqual( - (src_batch, p.beam_search.num_hyps_per_beam), - actual_decode.topk_hyps.shape) - self.assertTupleEqual( - (src_batch * p.beam_search.num_hyps_per_beam, src_time), - actual_decode.topk_ids.shape) - self.assertTupleEqual( - (src_batch * p.beam_search.num_hyps_per_beam,), - actual_decode.topk_lens.shape) - self.assertTupleEqual( - (src_batch, p.beam_search.num_hyps_per_beam), - actual_decode.topk_scores.shape) - - expected_topk_ids = [[2, 0, 0, 0, 0], [13, 2, 0, 0, 0], [0, 0, 0, 0, 0], - [0, 0, 0, 0, 0]] - - expected_topk_lens = [1, 2, 0, 0] - expected_topk_scores = [[-3.783162, -5.767723], [0., 0.]] - - self.assertAllEqual(expected_topk_ids, actual_decode.topk_ids) - self.assertAllEqual(expected_topk_lens, actual_decode.topk_lens) - self.assertAllClose(expected_topk_scores, actual_decode.topk_scores) - - def testBeamSearchDecodeTgtPrefix(self, dtype=tf.float32): - with self.session(use_gpu=True), self.SetEval(True): - tf.random.set_seed(_TF_RANDOM_SEED) - src_batch = 2 - p = self._DecoderParams(dtype=dtype) - p.init_step_ids = True # initializes beam search with predefined ids. - p.beam_search.num_hyps_per_beam = 2 - p.rnn_cell_dim = 32 - dec = decoder.MTDecoderV1(p) - encoder_outputs, _ = self._Inputs(dtype=dtype, init_step_ids=True) - decode = dec.BeamSearchDecode(encoder_outputs) - # topk_decoded is None in MT decoder, set it to a fake tensor to pass - # self.evaluate(decode). - decode = decode._replace(topk_decoded=tf.constant(0, tf.float32)) - self.evaluate(tf.global_variables_initializer()) - actual_decode = self.evaluate(decode) - - num_hyps = src_batch * p.beam_search.num_hyps_per_beam - self.assertTupleEqual((p.target_seq_len, num_hyps), - actual_decode.done_hyps.shape) - self.assertTupleEqual((src_batch, p.beam_search.num_hyps_per_beam), - actual_decode.topk_hyps.shape) - self.assertTupleEqual((num_hyps, p.target_seq_len), - actual_decode.topk_ids.shape) - self.assertTupleEqual((num_hyps,), actual_decode.topk_lens.shape) - self.assertTupleEqual((src_batch, p.beam_search.num_hyps_per_beam), - actual_decode.topk_scores.shape) - - expected_topk_ids = [[2, 0, 0, 0, 0], [13, 2, 0, 0, 0], [0, 0, 0, 0, 0], - [0, 0, 0, 0, 0]] - expected_topk_lens = [1, 2, 0, 0] - expected_topk_scores = [[-3.783162, -5.767723], [0., 0.]] - self.assertAllEqual(expected_topk_ids, actual_decode.topk_ids) - self.assertAllEqual(expected_topk_lens, actual_decode.topk_lens) - self.assertAllClose(expected_topk_scores, actual_decode.topk_scores) - - @parameterized.named_parameters( - ('Bias0ConsistentFalse', 0., False), - ('Bias0ConsistentTrue', 0., True), - ('Bias1ConsistentFalse', 1., False), - ('Bias1ConsistentTrue', 1., True), - ) - def testBeamSearchDecodeBiased(self, bias, bias_only_if_consistent): - dtype = tf.float32 - with self.session(use_gpu=True), self.SetEval(True): - tf.random.set_seed(_TF_RANDOM_SEED) - src_batch = 2 - p = self._DecoderParams(dtype=dtype) - p.bias_only_if_consistent = bias_only_if_consistent - p.target_seq_len = 6 - p.beam_search.num_hyps_per_beam = 2 - p.rnn_cell_dim = 32 - dec = p.Instantiate() - encoder_outputs, _ = self._Inputs(dtype=dtype) - encoder_outputs['targets'] = py_utils.NestedMap( - labels=tf.constant([[1, 3, 0, 0], [3, 4, 5, 2]]), - paddings=tf.constant([[0, 0, 1, 1], [0, 0, 0, 0]], dtype=dtype)) - encoder_outputs['targets']['weights'] = tf.fill( - tf.shape(encoder_outputs.targets.labels), bias) - decode = dec.BeamSearchDecodeBiased(encoder_outputs) - - # topk_decoded is None in MT decoder, set it to a fake tensor to pass - # self.evaluate(decode). - decode = decode._replace(topk_decoded=tf.constant(0, tf.float32)) - - self.evaluate(tf.global_variables_initializer()) - actual_decode = self.evaluate(decode) - - num_hyps = src_batch * p.beam_search.num_hyps_per_beam - self.assertTupleEqual((p.target_seq_len, num_hyps), - actual_decode.done_hyps.shape) - self.assertTupleEqual((src_batch, p.beam_search.num_hyps_per_beam), - actual_decode.topk_hyps.shape) - self.assertTupleEqual((num_hyps, p.target_seq_len), - actual_decode.topk_ids.shape) - self.assertTupleEqual((num_hyps,), actual_decode.topk_lens.shape) - self.assertTupleEqual((src_batch, p.beam_search.num_hyps_per_beam), - actual_decode.topk_scores.shape) - - if bias == 0: - expected_topk_ids = [[2, 0, 0, 0, 0, 0], [13, 2, 0, 0, 0, 0], - [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]] - expected_topk_lens = [1, 2, 0, 0] - expected_topk_scores = [[-3.783162, -5.767723], [0., 0.]] - elif bias == 1 and bias_only_if_consistent: - expected_topk_ids = [[1, 3, 2, 0, 0, 0], [1, 3, 13, 2, 0, 0], - [3, 4, 5, 2, 0, 0], [0, 0, 0, 0, 0, 0]] - expected_topk_lens = [3, 4, 4, 0] - expected_topk_scores = [[-3.073836, -5.474799], [-0.415888, 0.]] - elif bias == 1 and (not bias_only_if_consistent): - expected_topk_ids = [[1, 3, 2, 0, 0, 0], [1, 3, 13, 2, 0, 0], - [3, 4, 5, 2, 0, 0], [3, 4, 0, 2, 0, 0]] - expected_topk_lens = [3, 4, 4, 4] - expected_topk_scores = [[-3.073836, -5.474799], [-0.415888, -23.295631]] - - self.assertAllEqual(expected_topk_ids, actual_decode.topk_ids) - self.assertAllEqual(expected_topk_lens, actual_decode.topk_lens) - self.assertAllClose(expected_topk_scores, actual_decode.topk_scores) - - def testBeamSearchDecodeUseZeroAttenState(self, dtype=tf.float32): - with self.session(use_gpu=True), self.SetEval(True): - tf.random.set_seed(_TF_RANDOM_SEED) - src_batch = 2 - p = self._DecoderParams(dtype=dtype) - src_time = p.target_seq_len - p.beam_search.num_hyps_per_beam = 2 - p.use_zero_atten_state = True - p.rnn_cell_dim = 32 - dec = decoder.MTDecoderV1(p) - encoder_outputs, _ = self._Inputs(dtype=dtype) - decode = dec.BeamSearchDecode(encoder_outputs) - # topk_decoded is None in MT decoder, set it to a fake tensor to pass - # self.evaluate(decode). - decode = decode._replace(topk_decoded=tf.constant(0, tf.float32)) - - self.evaluate(tf.global_variables_initializer()) - actual_decode = self.evaluate(decode) - - self.assertTupleEqual( - (src_time, src_batch * p.beam_search.num_hyps_per_beam), - actual_decode.done_hyps.shape) - self.assertTupleEqual((src_batch, p.beam_search.num_hyps_per_beam), - actual_decode.topk_hyps.shape) - self.assertTupleEqual( - (src_batch * p.beam_search.num_hyps_per_beam, src_time), - actual_decode.topk_ids.shape) - self.assertTupleEqual((src_batch * p.beam_search.num_hyps_per_beam,), - actual_decode.topk_lens.shape) - self.assertTupleEqual((src_batch, p.beam_search.num_hyps_per_beam), - actual_decode.topk_scores.shape) - - expected_topk_ids = [[2, 0, 0, 0, 0], [13, 2, 0, 0, 0], [0, 0, 0, 0, 0], - [0, 0, 0, 0, 0]] - - expected_topk_lens = [1, 2, 0, 0] - expected_topk_scores = [[-3.783176, -5.767704], [0., 0.]] - - self.assertAllEqual(expected_topk_ids, actual_decode.topk_ids) - self.assertAllEqual(expected_topk_lens, actual_decode.topk_lens) - self.assertAllClose(expected_topk_scores, actual_decode.topk_scores) - - def testBeamSearchDecodeFeedingAttContext(self, dtype=tf.float32): - with self.session(use_gpu=True), self.SetEval(True): - tf.random.set_seed(_TF_RANDOM_SEED) - src_batch = 2 - p = self._DecoderParams(dtype=dtype) - src_time = p.target_seq_len - p.beam_search.num_hyps_per_beam = 2 - p.rnn_cell_dim = 32 - p.feed_attention_context_vec_to_softmax = True - dec = decoder.MTDecoderV1(p) - encoder_outputs, _ = self._Inputs(dtype=dtype) - decode = dec.BeamSearchDecode(encoder_outputs) - # topk_decoded is None in MT decoder, set it to a fake tensor to pass - # self.evaluate(decode). - decode = decode._replace(topk_decoded=tf.constant(0, tf.float32)) - self.evaluate(tf.global_variables_initializer()) - actual_decode_feeding_att_context = self.evaluate(decode) - - self.assertTupleEqual( - (src_time, src_batch * p.beam_search.num_hyps_per_beam), - actual_decode_feeding_att_context.done_hyps.shape) - self.assertTupleEqual( - (src_batch, p.beam_search.num_hyps_per_beam), - actual_decode_feeding_att_context.topk_hyps.shape) - self.assertTupleEqual( - (src_batch * p.beam_search.num_hyps_per_beam, src_time), - actual_decode_feeding_att_context.topk_ids.shape) - self.assertTupleEqual( - (src_batch * p.beam_search.num_hyps_per_beam,), - actual_decode_feeding_att_context.topk_lens.shape) - self.assertTupleEqual( - (src_batch, p.beam_search.num_hyps_per_beam), - actual_decode_feeding_att_context.topk_scores.shape) - - expected_topk_ids = [[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [2, 0, 0, 0, 0], - [8, 2, 0, 0, 0]] - - expected_topk_lens = [0, 0, 1, 2] - expected_topk_scores = [[0., 0.], [-3.292501, -5.533068]] - - self.assertAllEqual(expected_topk_ids, - actual_decode_feeding_att_context.topk_ids) - self.assertAllEqual(expected_topk_lens, - actual_decode_feeding_att_context.topk_lens) - self.assertAllClose(expected_topk_scores, - actual_decode_feeding_att_context.topk_scores) - - def testSampleTargetSequences(self, dtype=tf.float32): - with self.session(use_gpu=True), self.SetEval(True): - tf.random.set_seed(_TF_RANDOM_SEED) - src_batch = 2 - p = self._DecoderParams(dtype=dtype) - if p.cls != decoder.MTDecoderV1: - tf.logging.info('Skipping testSampleTargetSequences for %s', p.cls) - return - p.rnn_cell_dim = 32 - dec = p.Instantiate() - encoder_outputs, _ = self._Inputs(dtype=dtype) - sample = dec.SampleTargetSequences( - dec.theta, - encoder_outputs, - random_seed=tf.constant(1, dtype=tf.int32)) - self.evaluate(tf.global_variables_initializer()) - actual_sample = self.evaluate(sample) - - self.assertTupleEqual((src_batch, p.target_seq_len), - actual_sample.ids.shape) - self.assertTupleEqual((src_batch, p.target_seq_len), - actual_sample.paddings.shape) - - expected_ids = [[0, 12, 12, 13, 5], [12, 10, 15, 1, 2]] - self.assertAllEqual(expected_ids, actual_sample.ids) - - -class TransformerDecoderTestCaseBase(test_utils.TestCase): - - def _DecoderParams(self, - per_word_avg_loss=False, - is_transparent=False, - dtype=tf.float32, - fprop_dtype=None, - use_task_emb=False, - init_step_ids=False): - p = decoder.TransformerDecoder.Params() - p.name = 'decoder' - p.source_dim = 4 - p.model_dim = 4 - p.num_trans_layers = 6 - disable_vn = py_utils.VariationalNoiseParams(1.0, False, False) - p.token_emb.vn = disable_vn - p.token_emb.vocab_size = 20 - p.token_emb.embedding_dim = 4 - p.token_emb.max_num_shards = 1 - p.token_emb.params_init = py_utils.WeightInit.GaussianSqrtDim(seed=12345) - p.position_emb.embedding_dim = 4 - if use_task_emb: - p.task_emb = p.token_emb.Copy() - p.task_emb.vocab_size = 4 - p.trans_tpl.vn = disable_vn - p.init_step_ids = init_step_ids - p.trans_tpl.source_dim = 4 - p.trans_tpl.tr_atten_tpl.source_dim = 4 - p.trans_tpl.tr_atten_tpl.num_attention_heads = 2 - p.trans_tpl.tr_fflayer_tpl.input_dim = 4 - p.trans_tpl.tr_fflayer_tpl.hidden_dim = 8 - p.label_smoothing = layers.LocalizedLabelSmoother.Params() - p.label_smoothing.offsets = [-2, -1, 1, 2] - p.label_smoothing.weights = [0.015, 0.035, 0.035, 0.015] - p.softmax.vn = disable_vn - p.softmax.num_classes = 20 - p.softmax.num_shards = 1 - p.per_word_avg_loss = per_word_avg_loss - p.random_seed = 1234 - p.dtype = dtype - p.target_seq_len = 5 - p.is_transparent = is_transparent - - for lp in base_layer.RecursiveFindLayerParams(p): - lp.dtype = dtype - - py_utils.UpdateFpropDtype(p, fprop_dtype) - - return p - - def _Inputs(self, dtype=tf.float32, has_task_ids=False, init_step_ids=False): - np.random.seed(_NUMPY_RANDOM_SEED) - src_time = 5 - src_batch = 4 - num_hyps = 2 - emb_dims = 4 - src_enc = tf.constant( - np.random.normal(size=[src_time, src_batch, emb_dims]), dtype=dtype) - src_paddings = tf.zeros([src_time, src_batch], dtype=dtype) - tgt_time = 5 - tgt_batch = src_batch * num_hyps - self.tgt_batch = tgt_batch - - tgt_ids = tf.constant( - np.random.randint(20, size=[tgt_batch, tgt_time]), dtype=tf.int32) - tgt_labels = tf.constant( - np.random.randint(20, size=[tgt_batch, tgt_time]), dtype=tf.int32) - tgt_paddings = tf.zeros([tgt_batch, tgt_time], dtype=dtype) - tgt_weights = 1.0 - tgt_paddings - tgts = py_utils.NestedMap({ - 'ids': tgt_ids, - 'labels': tgt_labels, - 'weights': tgt_weights, - 'paddings': tgt_paddings - }) - encoder_outputs = py_utils.NestedMap( - encoded=src_enc, padding=src_paddings, segment_id=None) - - if has_task_ids: - task_ids = tf.constant( - np.random.randint(4, size=[src_batch]), dtype=tf.int32) - tgts['task_ids'] = tf.tile( - tf.expand_dims(tf.tile(task_ids, [num_hyps]), 1), [1, tgt_time]) - encoder_outputs['target_task_ids'] = task_ids - if init_step_ids: - tgt_prefix = tf.constant( - np.random.randint(4, size=[src_batch]), dtype=tf.int32) - encoder_outputs['init_step_ids'] = tgt_prefix - return (encoder_outputs, tgts, num_hyps) - - def _InputsForAttentionTest(self, dtype=tf.float32, has_task_ids=False): - np.random.seed(_NUMPY_RANDOM_SEED) - src_time = 5 - src_batch = 2 - num_hyps = 2 - emb_dims = 4 - src_enc = tf.constant( - np.random.normal(size=[src_time, src_batch, emb_dims]), dtype=dtype) - src_paddings = tf.constant( - [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 1.0], [0.0, 1.0]], - dtype=dtype) - tgt_time = 5 - tgt_batch = src_batch * num_hyps - self.tgt_batch = tgt_batch - - tgt_ids = tf.constant( - np.random.randint(20, size=[tgt_batch, tgt_time]), dtype=tf.int32) - tgt_labels = tf.constant( - np.random.randint(20, size=[tgt_batch, tgt_time]), dtype=tf.int32) - tgt_paddings = tf.zeros([tgt_batch, tgt_time], dtype=dtype) - tgt_weights = 1.0 - tgt_paddings - tgts = py_utils.NestedMap({ - 'ids': tgt_ids, - 'labels': tgt_labels, - 'weights': tgt_weights, - 'paddings': tgt_paddings - }) - encoder_outputs = py_utils.NestedMap( - encoded=src_enc, padding=src_paddings, segment_id=None) - - if has_task_ids: - task_ids = tf.constant( - np.random.randint(4, size=[src_batch]), dtype=tf.int32) - tgts['task_ids'] = tf.tile( - tf.expand_dims(tf.tile(task_ids, [num_hyps]), 1), [1, tgt_time]) - encoder_outputs['target_task_ids'] = task_ids - - return (encoder_outputs, tgts, num_hyps) - - -class TransformerDecoderTest(TransformerDecoderTestCaseBase): - - def testDecoderConstruction(self): - p = self._DecoderParams() - dec = decoder.TransformerDecoder(p) - self.assertIsInstance(dec, p.cls) - - def testDecoderWithNgramMaskConstruction(self): - p = self._DecoderParams() - # Turn on N-gram masking in the TransformerLayer. - # Before doing so though copy the self-attention params to avoid - # the auxilliary attention being masked as well. - p.trans_tpl.tr_aux_atten_tpl = p.trans_tpl.tr_atten_tpl.Copy() - p.trans_tpl.tr_atten_tpl.is_masked = True - p.trans_tpl.tr_atten_tpl.mask_ngram_order = 3 - p.trans_tpl.tr_atten_tpl.mask_type = 'ngram' - dec = decoder.TransformerDecoder(p) - self.assertIsInstance(dec, p.cls) - - def testDecoderConstructionWithTplList(self): - p = self._DecoderParams() - p.trans_tpl = [p.trans_tpl.Copy(), p.trans_tpl.Copy()] - dec = decoder.TransformerDecoder(p) - self.assertIsInstance(dec, p.cls) - - def testTransparentDecoderConstruction(self): - p = self._DecoderParams(is_transparent=True) - dec = decoder.TransformerDecoder(p) - self.assertIsInstance(dec, p.cls) - - def _testPackedInputs(self, dtype=tf.float32): - p = self._DecoderParams() - np.random.seed(_NUMPY_RANDOM_SEED) - src_time = 5 - batch = 2 - emb_dims = 4 - tgt_time = 5 - src_enc = tf.constant( - np.random.normal(size=[src_time, batch, p.source_dim]), dtype=dtype) - paddings = tf.zeros([src_time, batch], dtype=dtype) - tgt_ids = tf.constant( - np.random.randint(20, size=[batch, tgt_time]), dtype=tf.int32) - tgt_labels = tf.constant( - np.random.randint(20, size=[batch, tgt_time]), dtype=tf.int32) - tgt_paddings = tf.zeros([batch, tgt_time], dtype=dtype) - tgt_weights = 1.0 - tgt_paddings - tgts = py_utils.NestedMap({ - 'ids': tgt_ids, - 'labels': tgt_labels, - 'weights': tgt_weights, - 'paddings': tgt_paddings - }) - - src_enc_packed = tf.transpose(src_enc, [1, 0, 2]) - src_enc_packed = tf.reshape(src_enc_packed, [-1, 1, emb_dims]) - src_enc_padding_packed = tf.reshape(paddings, [-1, 1]) - target_packed = py_utils.NestedMap({ - 'ids': tf.reshape(tgts.ids, [1, -1]), - 'labels': tf.reshape(tgts.labels, [1, -1]), - 'weights': tf.reshape(tgts.weights, [1, -1]), - 'paddings': tf.reshape(tgts.paddings, [1, -1]) - }) - src_segment_id = tf.transpose( - tf.constant( - np.asarray([[0, 0, 0, 0, 0, 1, 1, 1, 1, 1]]), dtype=tf.float32)) - target_packed.segment_ids = tf.constant( - np.asarray([[0, 0, 0, 0, 0, 1, 1, 1, 1, 1]]), dtype=tf.float32) - target_packed.segment_pos = tf.constant( - np.asarray([[0, 1, 2, 3, 4, 0, 1, 2, 3, 4]])) - return (src_enc, paddings, tgts, src_enc_packed, src_enc_padding_packed, - src_segment_id, target_packed) - - def _testTransparentInputs(self, - num_layers=6, - dtype=tf.float32, - is_eval_mode=False): - src_time = 5 - src_batch = 4 - emb_dims = 4 - encoder_outputs, tgts, num_hyps = self._Inputs(dtype) - src_enc = tf.constant( - np.random.normal(size=[src_time, src_batch, emb_dims, num_layers]), - dtype=dtype) - if not is_eval_mode: - src_enc = tf.unstack(src_enc, axis=3) - encoder_outputs.encoded = src_enc - return (encoder_outputs, tgts, num_hyps) - - def testDecoderFPropWithPacking(self, dtype=tf.float32): - with self.session(use_gpu=True): - with tf.variable_scope('transformer_test', reuse=tf.AUTO_REUSE): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(per_word_avg_loss=True, dtype=dtype) - # Localized label smoother messes up the loss with packing. - p.label_smoothing = None - dec = decoder.TransformerDecoder(p) - p_packed = p.Copy() - p_packed.packed_input = True - dec_packed = decoder.TransformerDecoder(p_packed) - - (src_enc, paddings, tgts, src_enc_packed, src_enc_padding_packed, - src_segment_id, target_packed) = self._testPackedInputs() - encoder_outputs = py_utils.NestedMap( - encoded=src_enc, padding=paddings, segment_id=None) - loss, _ = dec.FProp(dec.theta, encoder_outputs, tgts).metrics['loss'] - encoder_outputs_packed = py_utils.NestedMap( - encoded=src_enc_packed, - padding=src_enc_padding_packed, - segment_id=src_segment_id) - loss_packed, _ = dec_packed.FProp(dec_packed.theta, - encoder_outputs_packed, - target_packed).metrics['loss'] - self.evaluate(tf.global_variables_initializer()) - actual_loss, packed_loss = self.evaluate([loss, loss_packed]) - self.assertAlmostEqual( - np.float32(packed_loss), np.float32(actual_loss), delta=1e-4) - - def testTransparentDecoderFProp(self, dtype=tf.float32): - with self.session(use_gpu=True): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(is_transparent=True, dtype=dtype) - dec = decoder.TransformerDecoder(p) - encoder_outputs, targets, _ = self._testTransparentInputs(dtype=dtype) - loss, _ = dec.FPropDefaultTheta(encoder_outputs, targets).metrics['loss'] - self.evaluate(tf.global_variables_initializer()) - actual_loss = loss.eval() - print('actual loss = ', actual_loss) - CompareToGoldenSingleFloat(self, 19.725393, actual_loss) - - def test_ExpandToNumHyps(self, dtype=tf.float32): - with self.session(use_gpu=True): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(is_transparent=True, dtype=dtype) - dec = decoder.TransformerDecoder(p) - - src_enc_len = tf.constant([3, 2, 1]) - num_hyps = 2 - expected = tf.constant([3, 2, 1, 3, 2, 1]) - expanded = dec._ExpandToNumHyps(src_enc_len, num_hyps) - expanded_v, expected_v = self.evaluate([expanded, expected]) - self.assertAllEqual(expanded_v, expected_v) - - def test_RemoveEOSProbs(self, dtype=tf.float32): - with self.session(use_gpu=True): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(is_transparent=True, dtype=dtype) - dec = decoder.TransformerDecoder(p) - - src_enc_len = tf.constant([5, 3, 5]) - - # [batch, target_len, source_len] - probs = tf.constant([[[0.2, 0.2, 0.2, 0.2, 0.2]], - [[0.2, 0.3, 0.5, 0.0, 0.0]], - [[0.0, 0.0, 0.0, 0.0, 1.0]]]) - new_probs = dec._RemoveEOSProbs(p, probs, src_enc_len) - new_probs_v = self.evaluate([new_probs]) - - expected_probs = tf.constant([[[0.25, 0.25, 0.25, 0.25, 0.0]], - [[0.4, 0.6, 0.0, 0.0, 0.0]], - [[0.0, 0.0, 0.0, 0.0, 0.0]]]) - - new_probs_v, expected_probs_v = self.evaluate([new_probs, expected_probs]) - - self.assertAllClose(expected_probs_v, new_probs_v) - - def testDecoderFPropWithTaskEmb(self, dtype=tf.float32): - with self.session(use_gpu=True): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(dtype=dtype, use_task_emb=True) - dec = decoder.TransformerDecoder(p) - encoder_outputs, targets, _ = self._Inputs(dtype=dtype, has_task_ids=True) - loss, _ = dec.FPropDefaultTheta(encoder_outputs, targets).metrics['loss'] - self.evaluate(tf.global_variables_initializer()) - actual_loss = loss.eval() - CompareToGoldenSingleFloat(self, 18.374338, actual_loss) - - def testDecoderFPropWithLangDepAtten(self, dtype=tf.float32): - with self.session(use_gpu=True): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(dtype=dtype, use_task_emb=True) - # 4 tasks, 2 languages. - p.use_lang_dependent_atten = True - dec = decoder.TransformerDecoder(p) - encoder_outputs, targets, _ = self._Inputs(dtype=dtype, has_task_ids=True) - loss, _ = dec.FPropDefaultTheta(encoder_outputs, targets).metrics['loss'] - self.evaluate(tf.global_variables_initializer()) - actual_loss = loss.eval() - CompareToGoldenSingleFloat(self, 16.200066, actual_loss) - - def testDecoderFPropWithContext(self, dtype=tf.float32): - with self.session(use_gpu=True): - with tf.variable_scope('transformer_test', reuse=tf.AUTO_REUSE): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(per_word_avg_loss=True, dtype=dtype) - p.trans_tpl = layers_with_attention.TransformerWithContextLayer.Params() - p.trans_tpl.source_dim = 4 - p.trans_tpl.tr_atten_tpl.source_dim = 4 - p.trans_tpl.tr_atten_tpl.num_attention_heads = 2 - p.trans_tpl.tr_fflayer_tpl.input_dim = 4 - p.trans_tpl.tr_fflayer_tpl.hidden_dim = 8 - dec = p.Instantiate() - - enc_outputs, targets, _ = self._Inputs() - encoder_outputs = py_utils.NestedMap( - encoded=enc_outputs.encoded, - padding=enc_outputs.padding, - context_encoded=enc_outputs.encoded, - context_padding=enc_outputs.padding) - loss, _ = dec.FProp(dec.theta, encoder_outputs, targets).metrics['loss'] - self.evaluate(tf.global_variables_initializer()) - CompareToGoldenSingleFloat(self, 3.816574, loss.eval()) - - def testDecoderFPropWithZeroFirstStep(self, dtype=tf.float32): - with self.session(use_gpu=True): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(dtype=dtype) - p.zero_token_embs_first_time_step = True - dec = decoder.TransformerDecoder(p) - encoder_outputs, targets, _ = self._Inputs(dtype=dtype, has_task_ids=True) - loss, _ = dec.FPropDefaultTheta(encoder_outputs, targets).metrics['loss'] - self.evaluate(tf.global_variables_initializer()) - actual_loss = loss.eval() - CompareToGoldenSingleFloat(self, 21.425932, actual_loss) - - def _testExtendStep(self, sess, dec, encoder_outputs, tgts, num_hyps): - p = self._DecoderParams() - - # Infer true source encoder length from the padding. - src_enc_len = tf.reduce_sum(1 - encoder_outputs.padding, axis=0) - src_enc_len = dec._ExpandToNumHyps(src_enc_len, num_hyps) - - # Run Fprop - fprop_out = dec._FProp(dec.theta, encoder_outputs, tgts) - l_out1 = fprop_out.softmax_input - attention_map_fprop = fprop_out.attention - - # run ExtendStep - prefix_states = py_utils.NestedMap() - for i in range(6): - layer_i_states = py_utils.NestedMap() - # The first dim is for the decode step (sequence length). - # Here's 0 as placeholder - layer_i_states.key = tf.zeros([0, self.tgt_batch, p.model_dim]) - layer_i_states.value = tf.zeros([0, self.tgt_batch, p.model_dim]) - prefix_states['layer_%i' % i] = layer_i_states - - l_out2 = [] - per_step_atten_probs = [] - for i in range(5): - l_i_out, prefix_states, atten_probs = dec.ExtendStep( - dec.theta, encoder_outputs, tgts.ids[:, i], i, prefix_states) - l_out2.append(l_i_out) - per_step_atten_probs.append(atten_probs) - l_out2 = tf.stack(l_out2) - bs_atten_probs = tf.stack(per_step_atten_probs) - - attention_map_bs = py_utils.NestedMap(probs=bs_atten_probs) - - def _TransposeAttentions(x): - return tf.transpose(x, [1, 0, 2]) - - attention_map_bs = attention_map_bs.Transform(_TransposeAttentions) - - self.evaluate(tf.global_variables_initializer()) - - l_out1_v, l_out2_v, attention_map_fprop_v, attention_map_bs_v, src_enc_len_v = self.evaluate( - [l_out1, l_out2, attention_map_fprop, attention_map_bs, src_enc_len]) - - # Ensure that FProp and BeamSearch output are the same. - self.assertAllClose(l_out1_v, l_out2_v, rtol=1e-05, atol=1e-05) - - # Ensure that FProp and BeamSearch attention matrix is the same. - self.assertAllClose(attention_map_fprop_v.probs, attention_map_bs_v.probs) - - print('attention map', attention_map_fprop_v.probs) - - # End-to-end test attention probs -- ensure EOS symbol and positions - # behind EOS have 0 probability. - for i in range(0, len(src_enc_len_v)): - pos = int(src_enc_len_v[i]) - 1 - self.assertEqual( - np.count_nonzero(attention_map_fprop_v.probs[i][:, pos:]), 0) - - def testDecoderExtendStep(self, dtype=tf.float32): - with self.session(use_gpu=True) as sess: - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(dtype=dtype) - dec = decoder.TransformerDecoder(p) - encoder_outputs, targets, num_hyps = ( - self._InputsForAttentionTest(dtype=dtype)) - - self._testExtendStep(sess, dec, encoder_outputs, targets, num_hyps) - - def testDecoderWithNgramMaskExtendStep(self, dtype=tf.float32): - with self.session(use_gpu=True) as sess: - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(dtype=dtype) - # Turn on N-gram masking in the TransformerLayer. - # Before doing so though copy the self-attention params to avoid - # the auxilliary attention being masked as well. - p.trans_tpl.tr_aux_atten_tpl = p.trans_tpl.tr_atten_tpl.Copy() - p.trans_tpl.tr_atten_tpl.is_masked = True - p.trans_tpl.tr_atten_tpl.mask_ngram_order = 3 - p.trans_tpl.tr_atten_tpl.mask_type = 'ngram' - dec = decoder.TransformerDecoder(p) - encoder_outputs, targets, num_hyps = ( - self._InputsForAttentionTest(dtype=dtype)) - - self._testExtendStep(sess, dec, encoder_outputs, targets, num_hyps) - - def testDecoderExtendStepWithTaskEmb(self, dtype=tf.float32): - with self.session(use_gpu=True) as sess: - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(dtype=dtype, use_task_emb=True) - dec = decoder.TransformerDecoder(p) - encoder_outputs, targets, num_hyps = ( - self._InputsForAttentionTest(dtype=dtype, has_task_ids=True)) - - self._testExtendStep(sess, dec, encoder_outputs, targets, num_hyps) - - def testDecoderExtendStepZeroFirstTimeStep(self, dtype=tf.float32): - with self.session(use_gpu=True) as sess: - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(dtype=dtype) - p.zero_token_embs_first_time_step = True - dec = decoder.TransformerDecoder(p) - encoder_outputs, targets, num_hyps = ( - self._InputsForAttentionTest(dtype=dtype)) - self._testExtendStep(sess, dec, encoder_outputs, targets, num_hyps) - - def testTransparentDecoderExtendStep(self, dtype=tf.float32): - with self.session(use_gpu=True) as sess, self.SetEval(True): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(is_transparent=True, dtype=dtype) - dec = decoder.TransformerDecoder(p) - encoder_outputs, targets, num_hyps = self._testTransparentInputs( - dtype=dtype, is_eval_mode=True) - self._testExtendStep(sess, dec, encoder_outputs, targets, num_hyps) - - def testDecoderExtendStepWithContext(self, dtype=tf.float32): - with self.session(use_gpu=True) as sess: - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(dtype=dtype) - p.trans_tpl = layers_with_attention.TransformerWithContextLayer.Params() - p.trans_tpl.source_dim = 4 - p.trans_tpl.tr_atten_tpl.source_dim = 4 - p.trans_tpl.tr_atten_tpl.num_attention_heads = 2 - p.trans_tpl.tr_fflayer_tpl.input_dim = 4 - p.trans_tpl.tr_fflayer_tpl.hidden_dim = 8 - dec = p.Instantiate() - enc_outputs, targets, num_hyps = ( - self._InputsForAttentionTest(dtype=dtype, has_task_ids=True)) - encoder_outputs = py_utils.NestedMap( - encoded=enc_outputs.encoded, - padding=enc_outputs.padding, - context_encoded=enc_outputs.encoded, - context_padding=enc_outputs.padding) - self._testExtendStep(sess, dec, encoder_outputs, targets, num_hyps) - - def testDecoderFPropSplitBatch(self, dtype=tf.float32): - with self.session(use_gpu=True): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._DecoderParams(dtype=dtype) - dec = decoder.TransformerDecoder(p) - - encoder_outputs, targets, _ = self._Inputs(dtype=dtype) - src_enc1, src_enc2 = tf.split(encoder_outputs.encoded, 2, 1) - src_paddings1, src_paddings2 = tf.split(encoder_outputs.padding, 2, 1) - - # source idx <-> target idx: - # 0 <-> (0, 4), 1 <-> (1, 5), 2 <-> (2, 6), 3 <-> (3, 7) - tgts = ig_helper.SplitDictOfTensors(targets, 4) - targets1 = py_utils.NestedMap({ - 'ids': tf.concat([tgts[0]['ids'], tgts[2]['ids']], 0), - 'labels': tf.concat([tgts[0]['labels'], tgts[2]['labels']], 0), - 'weights': tf.concat([tgts[0]['weights'], tgts[2]['weights']], 0), - 'paddings': tf.concat([tgts[0]['paddings'], tgts[2]['paddings']], 0) - }) - targets2 = py_utils.NestedMap({ - 'ids': tf.concat([tgts[1]['ids'], tgts[3]['ids']], 0), - 'labels': tf.concat([tgts[1]['labels'], tgts[3]['labels']], 0), - 'weights': tf.concat([tgts[1]['weights'], tgts[3]['weights']], 0), - 'paddings': tf.concat([tgts[1]['paddings'], tgts[3]['paddings']], 0) - }) - - loss, _ = dec.FPropDefaultTheta(encoder_outputs, targets).metrics['loss'] - encoder_outputs1 = py_utils.NestedMap( - encoded=src_enc1, padding=src_paddings1, segment_id=None) - loss1, _ = dec.FPropDefaultTheta(encoder_outputs1, - targets1).metrics['loss'] - encoder_outputs2 = py_utils.NestedMap( - encoded=src_enc2, padding=src_paddings2, segment_id=None) - loss2, _ = dec.FPropDefaultTheta(encoder_outputs2, - targets2).metrics['loss'] - - self.evaluate(tf.global_variables_initializer()) - actual_loss, actual_loss1, actual_loss2 = self.evaluate( - [loss, loss1, loss2]) - print('actual loss = ', actual_loss) - print('actual loss1 = ', actual_loss1) - print('actual loss2 = ', actual_loss2) - self.assertAlmostEqual( - actual_loss, np.mean([actual_loss1, actual_loss2]), delta=0.0001) - - def _testBeamSearch(self, - expected_values, - dtype=tf.float32, - init_step_ids=False, - has_task_ids=False): - tf.random.set_seed(_TF_RANDOM_SEED) - src_batch = 4 - src_time = 5 - p = self._DecoderParams(dtype=dtype, init_step_ids=init_step_ids) - p.beam_search.num_hyps_per_beam = 2 - p.beam_search.coverage_penalty = 0.0 - p.beam_search.length_normalization = 0 - dec = decoder.TransformerDecoder(p) - encoder_outputs, _, _ = self._Inputs( - dtype=dtype, has_task_ids=has_task_ids, init_step_ids=init_step_ids) - decode = dec.BeamSearchDecode(encoder_outputs) - # topk_decoded is None in MT decoder, set it to a fake tensor to pass - # self.evaluate(decode). - decode = decode._replace(topk_decoded=tf.constant(0, tf.float32)) - - with self.session(use_gpu=True): - self.evaluate(tf.global_variables_initializer()) - actual_decode = self.evaluate(decode) - - self.assertTupleEqual( - (src_time, src_batch * p.beam_search.num_hyps_per_beam), - actual_decode.done_hyps.shape) - self.assertTupleEqual( - (src_batch, p.beam_search.num_hyps_per_beam), - actual_decode.topk_hyps.shape) - self.assertTupleEqual( - (src_batch * p.beam_search.num_hyps_per_beam, src_time), - actual_decode.topk_ids.shape) - self.assertTupleEqual( - (src_batch * p.beam_search.num_hyps_per_beam,), - actual_decode.topk_lens.shape) - self.assertTupleEqual( - (src_batch, p.beam_search.num_hyps_per_beam), - actual_decode.topk_scores.shape) - - # Assert expected IDs etc - self.assertAllEqual(expected_values['topk_ids'], actual_decode.topk_ids) - self.assertAllEqual(expected_values['topk_lens'], actual_decode.topk_lens) - self.assertAllClose(expected_values['topk_scores'], - actual_decode.topk_scores) - - # Assert expected attention probs. - hypstr = actual_decode.topk_hyps.flatten()[1] - hyp = Hypothesis() - hyp.ParseFromString(hypstr) - print('HYP:', hyp) - - atten_vec_0 = list(np.expand_dims(np.array(hyp.atten_vecs[0].prob), 0)[0]) - atten_vec_1 = list(np.expand_dims(np.array(hyp.atten_vecs[1].prob), 0)[0]) - - self.assertAllClose(atten_vec_0, expected_values['atten_vec_0']) - self.assertAllClose(atten_vec_1, expected_values['atten_vec_1']) - - # Test normalized scores of hypotheses. - CompareToGoldenSingleFloat(self, expected_values['normalized_score'], - hyp.normalized_score) - - def testBeamSearchDecode(self, dtype=tf.float32): - expected_values = {} - expected_values['topk_ids'] = [[5, 2, 0, 0, 0], [17, 2, 0, 0, 0], - [5, 2, 0, 0, 0], [17, 3, 2, 0, 0], - [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], - [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]] - expected_values['topk_lens'] = [2, 2, 2, 3, 0, 0, 0, 0] - expected_values['topk_scores'] = [[-3.821746, -3.980103], - [-3.817123, -4.522634], [0., 0.], - [0., 0.]] - - expected_values['atten_vec_0'] = [ - 0.532658, 0.140424, 0.122954, 0.203961, 0. - ] - expected_values['atten_vec_1'] = [ - 0.067983, 0.532731, 0.284223, 0.115062, 0. - ] - expected_values['normalized_score'] = -3.980103 - - self._testBeamSearch( - expected_values=expected_values, - dtype=dtype, - init_step_ids=False, - has_task_ids=False) - - def testBeamSearchDecodeTgtPrefix(self, dtype=tf.float32): - expected_values = {} - expected_values['topk_ids'] = [[5, 2, 0, 0, 0], [1, 2, 0, 0, 0], - [5, 2, 0, 0, 0], [5, 3, 2, 0, 0], - [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], - [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]] - expected_values['topk_lens'] = [2, 2, 2, 3, 0, 0, 0, 0] - expected_values['topk_scores'] = [[-3.837574, -4.1681], - [-3.821537, -5.069403], [0., 0.], - [0., 0.]] - - expected_values['atten_vec_0'] = [0.58009, 0.129324, 0.109171, 0.181414, 0.] - expected_values['atten_vec_1'] = [ - 0.072272, 0.474015, 0.330433, 0.123279, 0. - ] - expected_values['normalized_score'] = -4.1681 - - self._testBeamSearch( - expected_values=expected_values, - dtype=dtype, - init_step_ids=True, - has_task_ids=False) - - def _testSampleSequence(self, - expected_values, - dtype=tf.float32, - init_step_ids=False, - has_task_ids=False): - tf.random.set_seed(_TF_RANDOM_SEED) - src_batch = 4 - src_time = 5 - p = self._DecoderParams(dtype=dtype, init_step_ids=init_step_ids) - p.beam_search.num_hyps_per_beam = 1 - p.beam_search.coverage_penalty = 0.0 - p.beam_search.length_normalization = 0 - dec = p.Instantiate() - encoder_outputs, _, _ = self._Inputs( - dtype=dtype, has_task_ids=has_task_ids, init_step_ids=init_step_ids) - decode = dec.SampleSequenceDecode(encoder_outputs) - - with self.session(use_gpu=True): - self.evaluate(tf.global_variables_initializer()) - actual_decode = self.evaluate(decode) - - self.assertTupleEqual((src_batch, p.beam_search.num_hyps_per_beam), - actual_decode.topk_hyps.shape) - self.assertTupleEqual( - (src_batch * p.beam_search.num_hyps_per_beam, src_time), - actual_decode.topk_ids.shape) - self.assertTupleEqual((src_batch * p.beam_search.num_hyps_per_beam,), - actual_decode.topk_lens.shape) - - self.assertAllEqual(expected_values['topk_ids'], actual_decode.topk_ids) - self.assertAllEqual(expected_values['topk_lens'], actual_decode.topk_lens) - self.assertAllClose(expected_values['topk_scores'], - actual_decode.topk_scores) - - def testSampleSequenceDecode(self, dtype=tf.float32): - expected_values = {} - expected_values['topk_ids'] = [[4, 3, 12, 7, 11], [17, 7, 2, 2, 2], - [17, 2, 2, 2, 2], [17, 1, 1, 16, 4]] - expected_values['topk_lens'] = [5, 3, 2, 5] - expected_values['topk_scores'] = [ - -6.985453, -3.714368, -2.899912, -8.281981 - ] - self._testSampleSequence( - expected_values=expected_values, - dtype=dtype, - init_step_ids=True, - has_task_ids=False) - - -class InsertionDecoderTest(TransformerDecoderTestCaseBase): - - def testDecoderConstruction(self): - p = decoder.InsertionDecoder.Params() - p.name = 'insertion_decoder' - dec = p.Instantiate() - self.assertIsInstance(dec, decoder.InsertionDecoder) - - -class TransformerBatchMajorDecoderTest(test_utils.TestCase, - parameterized.TestCase): - """Test Transformer decoder.""" - - def _ConstructTransformerBatchMajorDecoder(self, - dtype=tf.float32, - packed_input=False, - **kwargs): - p = decoder.TransformerBatchMajorDecoder.Params() - p.name = 'decoder' - p.packed_input = packed_input - p.source_dim = 4 - p.model_dim = 4 - p.num_trans_layers = 6 - disable_vn = py_utils.VariationalNoiseParams(1.0, False, False) - p.token_emb.vn = disable_vn - p.token_emb.vocab_size = 20 - p.token_emb.embedding_dim = 4 - p.token_emb.max_num_shards = 1 - p.token_emb.params_init = py_utils.WeightInit.GaussianSqrtDim() - p.position_emb.embedding_dim = 4 - p.trans_decoder_tpl.vn = disable_vn - p.trans_decoder_tpl.input_dim = 4 - p.trans_decoder_tpl.tr_atten_tpl.input_dim = 4 - p.trans_decoder_tpl.tr_atten_tpl.num_heads = 2 - p.trans_decoder_tpl.tr_fflayer_tpl.input_dim = 4 - p.trans_decoder_tpl.tr_fflayer_tpl.hidden_dim = 8 - p.trans_decoder_tpl.packed_input = packed_input - p.softmax.vn = disable_vn - p.softmax.num_classes = 20 - p.softmax.num_shards = 1 - p.per_word_avg_loss = False - p.random_seed = 12345 - p.target_seq_len = 5 - p.beam_search.num_hyps_per_beam = 2 - p.beam_search.coverage_penalty = 0.0 - p.beam_search.length_normalization = 0 - p.dtype = dtype - for k, v in kwargs.items(): - setattr(p, k, v) - for lp in base_layer.RecursiveFindLayerParams(p): - lp.dtype = dtype - return decoder.TransformerBatchMajorDecoder(p) - - def _Inputs(self, dtype=tf.float32, packed_input=False): - np.random.seed(9885784) - source_time = 5 - batch = 4 - dim = 4 - encoded = tf.constant( - np.random.normal(size=[source_time, batch, dim]), dtype=dtype) - padding = tf.zeros([source_time, batch], dtype=dtype) - target_time = 5 - target_ids = tf.constant( - np.random.randint(20, size=[batch, target_time]), dtype=tf.int32) - target_labels = tf.constant( - np.random.randint(20, size=[batch, target_time]), dtype=tf.int32) - target_paddings = tf.zeros([batch, target_time], dtype=dtype) - target_weights = 1.0 - target_paddings - targets = py_utils.NestedMap({ - 'ids': target_ids, - 'labels': target_labels, - 'weights': target_weights, - 'paddings': target_paddings - }) - encoder_outputs = py_utils.NestedMap(encoded=encoded, padding=padding) - - if packed_input: - encoder_outputs.segment_id = tf.tile( - tf.constant(np.asarray([[1, 1, 2, 2, 2]]), dtype=tf.float32), - [batch, 1]) - encoder_outputs.segment_id = tf.transpose(encoder_outputs.segment_id) - targets.segment_pos = tf.tile( - tf.constant(np.asarray([[0, 1, 0, 1, 2]]), dtype=tf.float32), - [batch, 1]) - targets.segment_ids = tf.tile( - tf.constant(np.asarray([[1, 1, 2, 2, 2]]), dtype=tf.float32), - [batch, 1]) - return encoder_outputs, targets - - def testDecoderConstruction(self): - _ = self._ConstructTransformerBatchMajorDecoder() - - def testDecoderConstructionPackedInput(self): - self._ConstructTransformerBatchMajorDecoder(packed_input=True) - - @parameterized.named_parameters(('TBC', 'TBC'), ('BTC', 'BTC')) - def testDecoderFProp(self, prediction_data_format): - with self.session(use_gpu=True) as sess: - dec = self._ConstructTransformerBatchMajorDecoder( - prediction_data_format=prediction_data_format, - per_example_tensors=True) - encoder_outputs, targets = self._Inputs() - dec_out = dec.FPropDefaultTheta(encoder_outputs, targets) - dec_out = tf.nest.map_structure(tf.convert_to_tensor, dec_out) - tf.global_variables_initializer().run() - actual_dec_out = sess.run(dec_out) - print('actual decoder output =', actual_dec_out) - self.assertAllClose(27.047781, actual_dec_out.metrics['loss'][0]) - - def testDecoderFPropPackedInput(self): - with self.session(use_gpu=True) as sess: - dec = self._ConstructTransformerBatchMajorDecoder(packed_input=True) - encoder_outputs, targets = self._Inputs(packed_input=True) - loss, _ = dec.FPropDefaultTheta(encoder_outputs, targets).metrics['loss'] - tf.global_variables_initializer().run() - actual_loss = sess.run(loss) - print('actual loss = ', actual_loss) - self.assertAllClose(15.041874, actual_loss) - - def testDecoderExtendStep(self): - with self.session(use_gpu=True) as sess: - dec = self._ConstructTransformerBatchMajorDecoder() - encoder_outputs, targets = self._Inputs() - - layer_out1 = dec._FProp(dec.theta, encoder_outputs, targets) - - prefix_states = py_utils.NestedMap() - for i in range(6): - layer_i_states = py_utils.NestedMap() - layer_i_states.key = tf.zeros([5, 4, 2, 2]) - layer_i_states.value = tf.zeros([5, 4, 2, 2]) - prefix_states['layer_%i' % i] = layer_i_states - - layer_out2 = [] - for i in range(5): - layer_i_out, prefix_states = dec.ExtendStep( - dec.theta, encoder_outputs, tf.expand_dims(targets.ids[:, i], 1), i, - prefix_states) - layer_out2.append(layer_i_out) - layer_out2 = tf.stack(layer_out2) - - tf.global_variables_initializer().run() - actual_layer_out1, actual_layer_out2 = sess.run([layer_out1, layer_out2]) - self.assertAllClose(actual_layer_out1, actual_layer_out2) - - def testBeamSearchDecode(self): - with self.session(use_gpu=True) as sess: - dec = self._ConstructTransformerBatchMajorDecoder() - encoder_outputs, _ = self._Inputs() - decode = dec.BeamSearchDecode(encoder_outputs) - # topk_decoded is None in MT decoder, set it to a fake tensor to pass - # sess.run(decode). - decode = decode._replace(topk_decoded=tf.constant(0, tf.float32)) - - tf.global_variables_initializer().run() - actual_decode = sess.run(decode) - - source_batch = 4 - source_time = 5 - num_hyps_per_beam = 2 - self.assertTupleEqual((source_time, source_batch * num_hyps_per_beam), - actual_decode.done_hyps.shape) - self.assertTupleEqual((source_batch, num_hyps_per_beam), - actual_decode.topk_hyps.shape) - self.assertTupleEqual((source_batch * num_hyps_per_beam, source_time), - actual_decode.topk_ids.shape) - self.assertTupleEqual((source_batch * num_hyps_per_beam,), - actual_decode.topk_lens.shape) - self.assertTupleEqual((source_batch, num_hyps_per_beam), - actual_decode.topk_scores.shape) - - expected_topk_ids = [[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [8, 16, 2, 0, 0], - [8, 11, 7, 2, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], - [8, 8, 8, 2, 0], [8, 11, 7, 2, 0]] - expected_topk_lens = [0, 0, 3, 4, 0, 0, 4, 4] - expected_topk_scores = [[0., 0.], [-3.154168, -3.2543223], [0., 0.], - [-4.8691816, -4.9210157]] - tf.logging.info(['testBeamSearchDecode actual_decode', actual_decode]) - - self.assertAllEqual(expected_topk_ids, actual_decode.topk_ids) - self.assertAllEqual(expected_topk_lens, actual_decode.topk_lens) - self.assertAllClose(expected_topk_scores, actual_decode.topk_scores) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/mt/encoder.py b/lingvo/tasks/mt/encoder.py deleted file mode 100644 index 1730164c..00000000 --- a/lingvo/tasks/mt/encoder.py +++ /dev/null @@ -1,924 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Encoders for the machine translation model. -""" - -import math -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import batch_major_attention -from lingvo.core import layers -from lingvo.core import model_helper -from lingvo.core import py_utils -from lingvo.core import rnn_cell -from lingvo.core import summary_utils -from lingvo.tasks.mt import layers as mt_layers - -tf.flags.DEFINE_bool('transformer_encoder_truncates_inputs', False, - 'Whether TransformerEncoder truncates inputs to max len.') - - -class MTEncoderV1(base_layer.BaseLayer): - """Machine translation encoder version 1.""" - - @classmethod - def Params(cls): - """Configs for `MTEncoderV1`.""" - p = super().Params() - p.Define('emb', layers.EmbeddingLayer.Params(), 'Embedding layer params.') - p.Define('lstm_tpl', - rnn_cell.LSTMCellSimple.Params(), - 'Configs template for the RNN layer.') - p.Define('lstm_tpl_uni', None, - 'Override configs template for the unidirectional RNN layers.') - p.Define('lstm_tpl_bidi', None, - 'Override configs template for the bidirectional RNN layer.') - p.Define('lstm_cell_size', 1024, 'LSTM cell size for the RNN layer.') - p.Define('num_lstm_layers', 8, 'Number of rnn layers to create') - p.Define('dropout_prob', 0.0, 'Prob at which we do dropout.') - p.Define('unidi_rnn_type', 'func', 'Options: func. ' 'func: FRNN.') - p.Define('bidi_rnn_type', 'func', 'Options: func. ' - 'func: BidirectionalFRNN. ') - p.Define('cc_schedule', None, 'Clipping cap schedule.') - p.Define( - 'packed_input', False, 'If True, encoder and all layers support ' - 'multiple examples in a single sequence.') - - disable_vn = py_utils.VariationalNoiseParams(1.0, False, False) - default_params_init = py_utils.WeightInit.Uniform(0.04) - - # Default config for the embedding. - p.emb.vn = disable_vn - p.emb.vocab_size = 32000 - p.emb.embedding_dim = 1024 - p.emb.max_num_shards = 16 - p.emb.params_init = default_params_init - - for tpl in [p.lstm_tpl, p.lstm_tpl_uni, p.lstm_tpl_bidi]: - if tpl is not None: - tpl.vn = disable_vn - tpl.params_init = default_params_init - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert not p.packed_input, ('Packed inputs are not yet supported for ' - 'MTEncoderV1.') - - if p.cc_schedule is not None: - self.CreateChild('cc_schedule', p.cc_schedule) - - self.CreateChild('emb', p.emb) - - rnn_layers_params = [] - - # L0 is a bi-directional lstm. - - # L0's forward lstm cell - if p.lstm_tpl_bidi is None: - params = p.lstm_tpl.Copy() - else: - params = p.lstm_tpl_bidi.Copy() - params.name = 'L0_rnn_fwd' - params.num_input_nodes = p.emb.embedding_dim - params.num_output_nodes = p.lstm_cell_size - forward_lstm = params - - # L0's backward lstm cell - params = params.Copy() - params.name = 'L0_rnn_bak' - backward_lstm = params - - # L0 layer. - params = model_helper.CreateBidirectionalRNNParams(self.params, - forward_lstm, - backward_lstm) - params.name = 'L0' - rnn_layers_params.append(params) - - # The latter layers are all uni-directional lstm. - input_size = 2 * p.lstm_cell_size - for i in range(1, p.num_lstm_layers): - # Forward lstm cell. - if p.lstm_tpl_uni is None: - cell = p.lstm_tpl.Copy() - else: - cell = p.lstm_tpl_uni.Copy() - cell.name = 'L%d_rnn' % i - cell.num_input_nodes = input_size - cell.num_output_nodes = p.lstm_cell_size - # Forward lstm layer. - params = model_helper.CreateUnidirectionalRNNParams(self.params, cell) - params.name = 'L%d' % i - rnn_layers_params.append(params) - input_size = p.lstm_cell_size - - self.CreateChildren('rnn', rnn_layers_params) - - dropout_p = layers.DropoutLayer.Params().Set( - name='dropout_layer', - keep_prob=1.0 - p.dropout_prob, - random_seed=p.random_seed + 84828474 if p.random_seed else None) - self.CreateChild('dropout', dropout_p) - - def ApplyClipping(self, theta, x): - p = self.params - if not p.cc_schedule: - return x - cap = tf.cast(self.cc_schedule.GetState(theta.cc_schedule), x.dtype) - return tf.clip_by_value(x, -cap, cap) - - def FProp(self, theta, input_batch): - """Encodes source as represented by `inputs` and `paddings`. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - input_batch: A `.NestedMap` with fields: - - ids: The inputs tensor. It is expected to be of shape [batch, time]. - - paddings: The paddings tensor. Expected shape [batch, time]. - - Returns: - A NestedMap containing: - - - encoded: The encoded features, a tensor of shape [time, batch, depth] - - padding: of shape [time, batch] - - segment_id: [time, batch] if packed inputs are supported by the model - (and all layers), or None otherwise. - """ - p = self.params - src_segment_id = None - with tf.name_scope(p.name): - # Now the rnn layers. - inputs = py_utils.with_dependencies([ - py_utils.assert_shape_match(tf.shape(input_batch.ids), [-1, -1]), - py_utils.assert_shape_match( - tf.shape(input_batch.ids), tf.shape(input_batch.paddings)) - ], tf.transpose(input_batch.ids)) - paddings = tf.expand_dims(tf.transpose(input_batch.paddings), 2) - xs = self.emb.EmbLookup(theta.emb, inputs) - xs = self.ApplyClipping(theta, xs) - self._emb_out = xs - ps = paddings - # When cc_schedule is specified, make sure lstm_tpl is QuantizedLSTMCell - # with the same cc_schedule so that the RNN layer output is within - # clipping range. - xs = self.rnn[0].FProp(theta.rnn[0], xs, ps) - xs = self.dropout.FProp(theta.dropout, xs) - for i in range(1, p.num_lstm_layers): - layer = self.rnn[i] - ys, _ = layer.FProp(theta.rnn[i], xs, ps) - ys = self.dropout.FProp(theta.dropout, ys) - if hasattr(layer.params, 'cell'): - layer_params = layer.params.cell - else: - layer_params = layer.params - if layer_params.num_input_nodes == layer_params.num_output_nodes: - xs += ys # Residual skip - xs = self.ApplyClipping(theta, xs) - else: - # When cc_schedule is specified, make sure lstm_tpl is - # QuantizedLSTMCell with the same cc_schedule so that the RNN layer - # output is within clipping range. - xs = ys - return py_utils.NestedMap( - encoded=xs, padding=tf.squeeze(ps, [2]), segment_id=src_segment_id) - - -class MTEncoderUniRNN(base_layer.BaseLayer): - """MT encoder that consists of a stack of uni-directional RNN layers.""" - - @classmethod - def Params(cls): - """Configs for `MTEncoderUniRNN`.""" - p = super().Params() - p.Define('emb', layers.EmbeddingLayer.Params(), 'Embedding layer params.') - p.Define('lstm_tpl', rnn_cell.LSTMCellSimple.Params(), - 'Configs template for the RNN layer.') - p.Define('lstm_cell_size', 512, 'LSTM cell size for the RNN layer.') - p.Define('num_lstm_layers', 8, 'Number of rnn layers to create') - p.Define('dropout_prob', 0.0, 'Prob at which we do dropout.') - p.Define('residual_start', 2, - 'Layer at which we start residual connections.') - p.Define('unidi_rnn_type', 'func', 'Options: func. ' 'func: FRNN.') - p.Define('cc_schedule', None, 'Clipping cap schedule.') - - p.Define('is_transparent', False, - 'If set, outputs a merger of layer outputs.') - p.Define( - 'transparent_merger_tpl', - layers.WeightedSumLayer.Params().Set(add_weight_summaries=True), - 'Merger op for layer outputs.') - p.Define( - 'packed_input', False, 'If True, encoder and all layers support ' - 'multiple examples in a single sequence.') - - disable_vn = py_utils.VariationalNoiseParams(1.0, False, False) - default_params_init = py_utils.WeightInit.Uniform(0.04) - - # Default config for the embedding. - p.emb.vn = disable_vn - p.emb.vocab_size = 32000 - p.emb.embedding_dim = 1024 - p.emb.max_num_shards = 16 - p.emb.params_init = default_params_init - - p.lstm_tpl.vn = disable_vn - p.lstm_tpl.params_init = default_params_init - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert not p.packed_input, ('Packed inputs are not yet supported for ' - 'MTEncoderUniRNN.') - - if p.cc_schedule is None: - self.cc_schedule = None - else: - self.CreateChild('cc_schedule', p.cc_schedule) - - self.CreateChild('emb', p.emb) - - rnn_layers_params = [] - - num_input_nodes = p.emb.embedding_dim - for i in range(p.num_lstm_layers): - cell = p.lstm_tpl.Copy() - cell.name = 'L%d_rnn' % i - cell.num_input_nodes = num_input_nodes - cell.num_output_nodes = p.lstm_cell_size - params = model_helper.CreateUnidirectionalRNNParams(self.params, cell) - params.name = 'L%d' % i - rnn_layers_params.append(params) - num_input_nodes = cell.num_output_nodes - - self.CreateChildren('rnn', rnn_layers_params) - - dropout_p = layers.DropoutLayer.Params().Set( - name='dropout_layer', - keep_prob=1.0 - p.dropout_prob, - random_seed=p.random_seed + 827366448 if p.random_seed else None) - self.CreateChild('dropout', dropout_p) - - if p.is_transparent: - transparent_params = p.transparent_merger_tpl.Copy() - transparent_params.name = 'transparent' - transparent_params.num_sources = p.num_lstm_layers - self.CreateChild('transparent_merger', transparent_params) - - def ApplyClipping(self, theta, x): - if self.cc_schedule: - return self.cc_schedule.ApplyClipping(theta.cc_schedule, x) - else: - return x - - def zero_state(self, theta, batch_size): - return py_utils.NestedMap(rnn=[ - self.rnn[i].zero_state(theta.rnn[i], batch_size) - for i in range(len(self.rnn)) - ]) - - def FProp(self, theta, input_batch, state0=None): - p = self.params - src_segment_id = None - with tf.name_scope(p.name): - # Reshape to [t, b] - inputs = py_utils.with_dependencies([ - py_utils.assert_shape_match(tf.shape(input_batch.ids), [-1, -1]), - py_utils.assert_shape_match( - tf.shape(input_batch.ids), tf.shape(input_batch.paddings)) - ], tf.transpose(input_batch.ids)) - paddings = tf.expand_dims(tf.transpose(input_batch.paddings), 2) - - # Setup streaming states. - if not state0: - state0 = self.zero_state(theta, tf.shape(inputs)[1]) - state1 = py_utils.NestedMap(rnn=[None] * p.num_lstm_layers) - - xs = self.emb.EmbLookup(theta.emb, inputs) - xs = self.ApplyClipping(theta, xs) - summary_utils.histogram('input_emb', xs) - xs = self.dropout.FProp(theta.dropout, xs) - ps = paddings - # Now the rnn layers. - outputs_list = [] - for i in range(0, p.num_lstm_layers): - layer = self.rnn[i] - ys, state1.rnn[i] = layer.FProp( - theta.rnn[i], xs, ps, state0=state0.rnn[i]) - ys = self.dropout.FProp(theta.dropout, ys) - if i >= p.residual_start: - xs += ys # Residual skip - xs = self.ApplyClipping(theta, xs) - else: - xs = ys - outputs_list.append(xs) - summary_utils.histogram('layer_out_%s' % i, xs) - - if p.is_transparent: - xs = self.transparent_merger.FProp(theta.transparent_merger, - outputs_list) - - return py_utils.NestedMap( - encoded=xs, - padding=tf.squeeze(ps, [2]), - segment_id=src_segment_id, - state=state1) - - -class MTEncoderBiRNN(base_layer.BaseLayer): - """MT encoder that consists of a stack of bi-directional RNN layers.""" - - @classmethod - def Params(cls): - """Configs for `MTEncoderBiRNN`.""" - p = super().Params() - p.Define('emb', layers.EmbeddingLayer.Params(), 'Embedding layer params.') - p.Define('lstm_tpl', - rnn_cell.LSTMCellSimple.Params(), - 'Configs template for the RNN layer.') - p.Define('proj_tpl', layers.ProjectionLayer.Params(), - 'Configs template for the projection layer.') - p.Define('lstm_cell_size', 512, 'LSTM cell size for the RNN layer.') - p.Define('num_lstm_layers', 8, 'Number of rnn layers to create') - p.Define('dropout_prob', 0.0, 'Prob at which we do dropout.') - p.Define('residual_start', 2, - 'Layer at which we start residual connections.') - p.Define('encoder_out_dim', 1024, 'Depth of the encoder output.') - p.Define('bidi_rnn_type', 'func', 'Options: func. ' - 'func: BidirectionalFRNN. ') - p.Define('cc_schedule', None, 'Clipping cap schedule.') - - p.Define('is_transparent', False, - 'If set, outputs a merger of layer outputs.') - p.Define( - 'transparent_merger_tpl', - layers.WeightedSumLayer.Params().Set(add_weight_summaries=True), - 'Merger op for layer outputs.') - p.Define( - 'packed_input', False, 'If True, encoder and all layers support ' - 'multiple examples in a single sequence.') - - disable_vn = py_utils.VariationalNoiseParams(1.0, False, False) - default_params_init = py_utils.WeightInit.Uniform(0.04) - - # Default config for the embedding. - p.emb.vn = disable_vn - p.emb.vocab_size = 32000 - p.emb.embedding_dim = 1024 - p.emb.max_num_shards = 16 - p.emb.params_init = default_params_init - - p.lstm_tpl.vn = disable_vn - p.lstm_tpl.params_init = default_params_init - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - if p.cc_schedule is None: - self.cc_schedule = None - else: - self.CreateChild('cc_schedule', p.cc_schedule) - - self.CreateChild('emb', p.emb) - - rnn_layers_params = [] - - for i in range(p.num_lstm_layers): - params = p.lstm_tpl.Copy() - params.name = 'L%d_rnn_fwd' % i - if i == 0: - params.num_input_nodes = p.emb.embedding_dim - else: - params.num_input_nodes = 2 * p.lstm_cell_size - params.num_output_nodes = p.lstm_cell_size - params.reset_cell_state = p.packed_input - forward_lstm = params - - params = params.Copy() - params.name = 'L%d_rnn_bak' % i - params.reset_cell_state = p.packed_input - backward_lstm = params - - params = model_helper.CreateBidirectionalRNNParams( - self.params, forward_lstm, backward_lstm) - params.packed_input = p.packed_input - params.name = 'L%d' % i - rnn_layers_params.append(params) - - self.CreateChildren('rnn', rnn_layers_params) - - if p.lstm_cell_size * 2 != p.encoder_out_dim: - # Project the encoder output to the desired dim. - proj_p = p.proj_tpl.Copy().Set( - name='proj', - batch_norm=False, - input_dim=p.lstm_cell_size * 2, - output_dim=p.encoder_out_dim) - if p.cc_schedule is not None: - proj_p.has_bias = False - proj_p.activation = 'TANH' - else: - proj_p.has_bias = True - proj_p.activation = 'NONE' - self.CreateChild('final_proj', proj_p) - - dropout_p = layers.DropoutLayer.Params().Set( - name='dropout_layer', - keep_prob=1.0 - p.dropout_prob, - random_seed=p.random_seed + 827366448 if p.random_seed else None) - self.CreateChild('dropout', dropout_p) - - if p.is_transparent: - transparent_params = p.transparent_merger_tpl.Copy() - transparent_params.name = 'transparent' - transparent_params.num_sources = p.num_lstm_layers - self.CreateChild('transparent_merger', transparent_params) - - def ApplyClipping(self, theta, x): - if self.cc_schedule: - return self.cc_schedule.ApplyClipping(theta.cc_schedule, x) - else: - return x - - def FProp(self, theta, input_batch): - p = self.params - with tf.name_scope(p.name): - inputs = py_utils.with_dependencies([ - py_utils.assert_shape_match(tf.shape(input_batch.ids), [-1, -1]), - py_utils.assert_shape_match( - tf.shape(input_batch.ids), tf.shape(input_batch.paddings)) - ], tf.transpose(input_batch.ids)) - paddings = tf.expand_dims(tf.transpose(input_batch.paddings), 2) - if p.packed_input: - src_segment_id = tf.expand_dims( - tf.transpose(input_batch.segment_ids), 2) - else: - src_segment_id = None - xs = self.emb.EmbLookup(theta.emb, inputs) - xs = self.ApplyClipping(theta, xs) - summary_utils.histogram('input_emb', xs) - xs = self.dropout.FProp(theta.dropout, xs) - ps = paddings - # Now the rnn layers. - outputs_list = [] - for i in range(0, p.num_lstm_layers): - layer = self.rnn[i] - ys = layer.FProp(theta.rnn[i], xs, ps, segment_id=src_segment_id) - ys = self.dropout.FProp(theta.dropout, ys) - if i >= p.residual_start: - xs += ys # Residual skip - xs = self.ApplyClipping(theta, xs) - else: - xs = ys - outputs_list.append(xs) - summary_utils.histogram('layer_out_%s' % i, xs) - - if p.is_transparent: - xs = self.transparent_merger.FProp(theta.transparent_merger, - outputs_list) - - if p.lstm_cell_size * 2 != p.encoder_out_dim: - # Project to the right depth. - xs = self.final_proj.FProp(theta.final_proj, xs, ps) - summary_utils.histogram('final_proj_out', xs) - - if src_segment_id is not None: - src_segment_id = tf.squeeze(src_segment_id, [2]) - - return py_utils.NestedMap( - encoded=xs, padding=tf.squeeze(ps, [2]), segment_id=src_segment_id) - - -class TransformerEncoder(base_layer.BaseLayer): - """Transformer stack with sinusoidal positional embeddings and attention. - - Implements the encoder of 'Attention is All You Need': - https://arxiv.org/abs/1706.03762. - """ - - @classmethod - def Params(cls): - """Configs for `TransformerEncoder`.""" - p = super().Params() - - # Embedding related - p.Define('token_emb', - layers.EmbeddingLayer.Params().Set( - vocab_size=32000, - embedding_dim=1024, - max_num_shards=16, - params_init=py_utils.WeightInit.Gaussian( - 1.0 / math.sqrt(1024)), - scale_sqrt_depth=True), 'Embedding layer params.') - - p.Define('shared_emb', None, 'Embedding shared with Decoder.') - - # Positional embedding related - p.Define( - 'position_emb', - layers.PositionalEmbeddingLayer.Params().Set(embedding_dim=1024), - 'Positional Embedding layer params.') - - # TODO(miachen): Extend this to more general logic of adding multiple - # embedding fields. - # Task embedding related - p.Define('task_emb', None, 'Task embedding layer params.') - - p.Define('model_dim', 1024, 'Characteristic depth (dimension).') - p.Define('input_dropout_prob', 0.0, 'Prob at which we do input dropout.') - - p.Define('transformer_stack', mt_layers.TransformerStack.Params(), - 'TransformerStack layer params.') - p.Define( - 'packed_input', False, 'If True, encoder and all layers support ' - 'multiple examples in a single sequence.') - - # MASS pretraining related (https://github.com/microsoft/MASS). - p.Define( - 'apply_source_mask', False, 'If True, apply source mask ' - '(corresponding to those masked words) to encoder states.') - p.Define( - 'source_mask_id', 0, 'Id for masked words in source inputs. ' - 'Only needed when p.apply_source_mask is True.') - - p.transformer_stack.num_transformer_layers = 6 - p.transformer_stack.transformer_tpl.tr_atten_tpl.num_attention_heads = 8 - p.transformer_stack.transformer_tpl.tr_fflayer_tpl.hidden_dim = 8192 - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - if p.shared_emb: - # Naming this 'softmax' to match the name of the same component in the - # decoder. Variable names need to be the same in order to be reused. - self.CreateChild('softmax', p.shared_emb) - - assert p.token_emb.embedding_dim == p.position_emb.embedding_dim - p.transformer_stack.Set(model_dim=p.model_dim, packed_input=p.packed_input) - if p.model_dim != p.token_emb.embedding_dim: - tf.logging.warning('token_emb.embedding_dim != model_dim (%s vs. %s), ' - 'creating a projection!') - proj_p = layers.ProjectionLayer.Params().Copy() - proj_p.name = 'emb_proj' - proj_p.input_dim = p.token_emb.embedding_dim - proj_p.output_dim = p.model_dim - proj_p.batch_norm = True - self.CreateChild('emb_proj', proj_p) - - # Token embeddings - if not p.shared_emb: - p.token_emb.dtype = p.dtype - self.CreateChild('token_emb', p.token_emb) - - # Positional embeddings - self.CreateChild('position_emb', p.position_emb) - - # Task embeddings. - if p.task_emb: - assert p.task_emb.embedding_dim == p.token_emb.embedding_dim - self.CreateChild('task_emb', p.task_emb) - - dropout_tpl = layers.DropoutLayer.Params() - dropout_tpl.keep_prob = (1.0 - p.input_dropout_prob) - self.CreateChild('input_dropout', dropout_tpl) - - p.transformer_stack.name = p.name - self.CreateChild('transformer_stack', p.transformer_stack) - - def _CreateChildrenVariables(self): - if self.params.shared_emb: - with tf.variable_scope('shared_emb', reuse=tf.AUTO_REUSE): - self.softmax.InstantiateVariables() - self.transformer_stack.InstantiateVariables() - super()._CreateChildrenVariables() - - def FProp(self, theta, input_batch): - """Embeds source ids and transforms with TransformerStack. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - input_batch: A `.NestedMap` with fields: - - - ids: The inputs tensor. It is expected to be of shape [batch, time]. - - paddings: The paddings tensor. Expected shape [batch, time]. - - task_ids: If p.task_emb is provided, must contain per-token task - ids of shape [batch, time]. - - Returns: - A NestedMap containing - - - encoded: The encoded features, either a tensor of shape - [time, batch, depth], or a list of tensors if is_transparent is set in - transformer_stack. - - padding: of shape [time, batch] - - segment_id: [time, batch] if packed inputs are supported by the model - (and all layers), or None otherwise. - - embedded_inputs: [time, batch, depth] embedded inputs tokens without - positional encodings. - """ - - p = self.params - with tf.name_scope(p.name): - src_segment_id = None - src_segment_pos = None - input_ids = py_utils.with_dependencies([ - py_utils.assert_shape_match( - tf.shape(input_batch.ids), tf.shape(input_batch.paddings)), - py_utils.assert_equal(tf.rank(input_batch.ids), 2) - ], input_batch.ids) - - if (not py_utils.use_tpu() and - tf.flags.FLAGS.transformer_encoder_truncates_inputs): - max_seq_length = tf.cast( - tf.reduce_max(tf.reduce_sum(1.0 - input_batch.paddings, 1)), - tf.int32) - paddings = py_utils.with_dependencies([ - py_utils.assert_equal( - tf.constant(True, tf.bool), - tf.reduce_all(input_batch.paddings[:, max_seq_length:] > 0.5)) - ], input_batch.paddings) - input_ids = input_ids[:, :max_seq_length] - paddings = paddings[:, :max_seq_length] - if p.packed_input: - src_segment_id = input_batch.segment_ids[:, :max_seq_length] - src_segment_pos = input_batch.segment_pos[:, :max_seq_length] - else: - paddings = input_batch.paddings - if p.packed_input: - src_segment_id = input_batch.segment_ids - src_segment_pos = input_batch.segment_pos - - max_time = tf.shape(input_ids)[1] - - # Input token embeddings + positional embeddings - if not p.shared_emb: - input_embs = self.token_emb.EmbLookup(theta.token_emb, - tf.reshape(input_ids, [-1])) - else: - input_embs = self.softmax.EmbLookup(theta.softmax, - tf.reshape(input_ids, [-1])) - - input_embs = tf.reshape(input_embs, - [-1, max_time, p.token_emb.embedding_dim]) - # [time, batch, dim] - orig_input_embs = tf.transpose(input_embs, [1, 0, 2]) - - if p.packed_input: - position_embs = self.position_emb.FPropWithPosition( - theta.position_emb, src_segment_pos) - else: - position_embs = self.position_emb.FProp(theta.position_emb, max_time) - position_embs = tf.reshape(position_embs, - [1, max_time, p.token_emb.embedding_dim]) - input_embs += position_embs - if p.task_emb: - input_embs += self.task_emb.EmbLookup(theta.task_emb, - input_batch.task_ids) - - if p.model_dim != p.token_emb.embedding_dim: - input_embs = self.emb_proj.FProp(theta.emb_proj, input_embs) - - paddings = tf.cast(tf.transpose(paddings), py_utils.FPropDtype(p)) - if p.packed_input: - src_segment_id = tf.transpose(src_segment_id) - input_embs = self.input_dropout.FProp(theta.input_dropout, input_embs) - - # [time, batch, dim] - transformer_input = tf.transpose(input_embs, [1, 0, 2]) - - if not self.do_eval and p.apply_source_mask: - # Augment padding for masked source word positions. - dtype = paddings.dtype - source_mask = tf.where( - tf.equal(input_ids, p.source_mask_id), - tf.ones_like(input_ids, dtype=dtype), - tf.zeros_like(input_ids, dtype=dtype)) - # Make sure padding is between 0 and 1. - paddings = tf.clip_by_value(paddings + tf.transpose(source_mask), 0.0, - 1.0) - - encoded, padding, segment_id = self.transformer_stack.FProp( - theta.transformer_stack, transformer_input, paddings, src_segment_id) - return py_utils.NestedMap( - encoded=encoded, - padding=padding, - segment_id=segment_id, - embedded_inputs=orig_input_embs) - - def FPropFullSequence(self, theta, ids, paddings): - return self.FProp(theta, py_utils.NestedMap(ids=ids, - paddings=paddings))['encoded'] - - -class TransformerBatchMajorEncoder(base_layer.BaseLayer): - """Transformer encoder with batch major implementation. - - This encoder first applies dropout to the input embeddings, - then returns encoded output produced by p.transformer_stack using - self_attention_layer builder. - - Example definition for a stack of 6 transformer layers: - - builder_params = self_attention_layer.Builder.Params().Set( - model_dim=model_dim, - ff_hidden_dim=ff_hidden_dim, - num_heads=num_heads, - selfatten_add_unnormalized_input=False, - selfatten_enable_value_proj=True) - - p.transformer_stack = builder_params.Instantiate().TransformerStack( - 'transformer_stack', 6) - - Implements the encoder of 'Attention is All You Need': - https://arxiv.org/abs/1706.03762. - """ - - @classmethod - def Params(cls): - p = super().Params() - - # Default config for the token embedding. - p.Define( - 'token_emb', - layers.EmbeddingLayer.Params().Set( - vocab_size=32000, - embedding_dim=1024, - max_num_shards=16, - params_init=py_utils.WeightInit.Gaussian(1.0 / math.sqrt(1024)), - scale_sqrt_depth=True), 'Embedding layer params.') - - p.Define('shared_emb', None, 'Embedding shared with Decoder.') - - # Default config for the position embedding. - p.Define('position_emb', - layers.PositionalEmbeddingLayer.Params().Set(embedding_dim=1024), - 'Positional Embedding layer params.') - - p.Define('model_dim', 1024, 'Characteristic depth (dimension).') - p.Define('input_dropout_prob', 0.0, 'Prob at which we do input dropout.') - p.Define('input_dropout_tpl', layers.DropoutLayer.Params(), - 'Input dropout layer params.') - p.Define('transformer_stack', None, 'TransformerStack layer params.') - p.Define( - 'packed_input', False, 'If True, encoder and all layers support ' - 'multiple examples in a single sequence.') - p.Define('final_layer_norm', False, - 'Whether or not to apply the final layer normalization.') - p.Define('use_fused_layernorm', False, 'Whether to use fused layernorm.') - p.Define( - 'output_data_format', 'TBC', 'The data format of output features: ' - 'TBC for [time, batch, feature_dim], ' - 'BTC for [batch, time, feature_dim].') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - assert p.output_data_format in ('TBC', 'BTC') - - if p.shared_emb: - self.CreateChild('softmax', p.shared_emb) - - p.token_emb.dtype = p.dtype - if not p.shared_emb: - self.CreateChild('token_emb', p.token_emb) - self.CreateChild('position_emb', p.position_emb) - - dropout_tpl = p.input_dropout_tpl.Copy() - dropout_tpl.keep_prob = (1.0 - p.input_dropout_prob) - self.CreateChild('input_dropout', dropout_tpl) - - if p.transformer_stack: - self.CreateChild('transformer_stack', p.transformer_stack) - - if p.final_layer_norm: - layer_norm_p = layers.LayerNorm.Params().Set( - name='final_ln', - input_dim=p.model_dim, - use_fused_layernorm=p.use_fused_layernorm, - fprop_dtype=p.input_dropout_tpl.fprop_dtype) - self.CreateChild('final_ln', layer_norm_p) - - def _CreateChildrenVariables(self): - if self.params.shared_emb: - with tf.variable_scope('shared_emb', reuse=tf.AUTO_REUSE): - self.softmax.InstantiateVariables() - super()._CreateChildrenVariables() - - def FProp(self, theta, input_batch): - """Embeds source ids and transforms with TransformerStack. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - input_batch: A `.NestedMap` object containing: ids - The inputs tensor of - shape [batch, time]. paddings - The ids' paddings of shape [batch, - time]. - - Returns: - A '.NestedMap' object containing: - encoded - The encoded features of shape [time, batch, dim] or [batch, - time, dim], depending p.output_data_format. - padding - The encoded features' padding of shape [time, batch] or - [batch, time]. - segment_id - The segmentation of packed inputs of shape [time, batch] or - [batch, time] if it is supported by the model, or None otherwise. - embedded_inputs - The embedded inputs tokens without positional - encodings of shape [time, batch, dim] or [batch, time, dim]. - """ - - p = self.params - with tf.name_scope(p.name): - # [batch, time] - input_ids = input_batch.ids - # [batch, time] - paddings = input_batch.paddings - - # [batch, time] - segment_ids = input_batch.segment_ids if p.packed_input else None - - batch = py_utils.GetShape(input_ids)[0] - time = py_utils.GetShape(input_ids)[1] - - # Embedding layer. - # [batch, time, dim] - if not p.shared_emb: - input_embs = self.token_emb.EmbLookup(theta.token_emb, input_ids) - else: - input_embs = self.softmax.EmbLookup(theta.softmax, input_ids) - orig_input_embs = input_embs - - # [1, time, dim] - if p.packed_input: - positions = input_batch.segment_pos - position_embs = tf.expand_dims( - self.position_emb.FPropWithPosition(theta.position_emb, positions), - 0) - else: - position_embs = tf.expand_dims( - self.position_emb.FProp(theta.position_emb, time), 0) - - # [batch, time, dim] - input_embs += position_embs - - if p.input_dropout_tpl.fprop_dtype: - input_embs = tf.cast(input_embs, p.input_dropout_tpl.fprop_dtype) - paddings = tf.cast(paddings, p.input_dropout_tpl.fprop_dtype) - - input_embs = self.input_dropout.FProp(theta.input_dropout, input_embs) - # [batch, time, dim] - transformer_input = input_embs - # Explicitly set the input shape of Transformer layers, to avoid - # unknown shape error occurred to tf.einsum on nonTPU devices. - transformer_input = tf.reshape(transformer_input, - [batch, time, p.model_dim]) - - # Compute self-attention segment mask once. - if p.packed_input: - segment_mask = batch_major_attention.SegmentMask( - segment_ids, segment_ids, dtype=transformer_input.dtype) - else: - segment_mask = tf.zeros([batch, 1, time, time]) - - encoded, padding = self.transformer_stack.FProp(theta.transformer_stack, - transformer_input, - paddings, segment_mask) - - if p.final_layer_norm: - encoded = self.final_ln.FProp(theta.final_ln, encoded) - - seq_lengths = tf.cast(tf.reduce_sum(1. - padding, axis=1), tf.int32) - - if p.output_data_format == 'TBC': - encoded = tf.transpose(encoded, [1, 0, 2]) # [time, batch, dim] - padding = tf.transpose(padding) # [time, batch] - segment_ids = tf.transpose(segment_ids) if p.packed_input else None - orig_input_embs = tf.transpose(orig_input_embs, [1, 0, 2]) - - return py_utils.NestedMap( - encoded=encoded, - padding=padding, - seq_lengths=seq_lengths, # used by beam_search_helper. - segment_id=segment_ids, - embedded_inputs=orig_input_embs) diff --git a/lingvo/tasks/mt/encoder_test.py b/lingvo/tasks/mt/encoder_test.py deleted file mode 100644 index 4081d017..00000000 --- a/lingvo/tasks/mt/encoder_test.py +++ /dev/null @@ -1,570 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Tests for mt.encoder.""" - -import lingvo.compat as tf -from lingvo.core import py_utils -from lingvo.core import self_attention_layer -from lingvo.core import test_utils -from lingvo.tasks.mt import encoder -import numpy as np - - -class EncoderTest(test_utils.TestCase): - - def _EncoderParams(self): - p = encoder.MTEncoderV1.Params() - p.name = 'encoder' - p.emb.vocab_size = 16 - p.emb.embedding_dim = 4 - p.emb.max_num_shards = 1 - p.lstm_cell_size = 4 - p.num_lstm_layers = 3 - p.random_seed = 837464 - return p - - def testEncoderConstruction(self): - p = self._EncoderParams() - _ = encoder.MTEncoderV1(p) - - def testForwardPass(self): - with self.session(use_gpu=False): - tf.random.set_seed(8372749040) - p = self._EncoderParams() - mt_enc = encoder.MTEncoderV1(p) - batch = py_utils.NestedMap() - batch.ids = tf.transpose(tf.reshape(tf.range(0, 8, 1), [4, 2])) - batch.paddings = tf.zeros([2, 4]) - enc_out = mt_enc.FPropDefaultTheta(batch).encoded - - self.evaluate(tf.global_variables_initializer()) - actual_enc_out = enc_out.eval() - tf.logging.info('testForwardPass actual_enc_out %r' % actual_enc_out) - expected_enc_out = [ - [[-2.5584161e-06, -5.6742726e-07, -8.1548797e-06, 2.6712776e-06], - [1.1781749e-06, -4.7786052e-08, 4.2439538e-06, -3.3840388e-06]], - [[-2.6852279e-06, 2.0878532e-07, -1.0491179e-05, 5.9619756e-06], - [2.0423495e-06, 3.1651740e-07, 5.7234793e-06, -3.8120934e-06]], - [[3.0904158e-07, -1.2983286e-06, -1.2469604e-05, 6.6027828e-06], - [-3.8620223e-07, 3.8890593e-07, 1.9976458e-06, 1.0078909e-06]], - [[1.0130438e-07, -1.1145677e-06, -1.2745468e-05, 8.0924037e-06], - [-1.3496270e-06, -3.2355717e-06, -3.0266469e-06, -3.9747570e-06]] - ] - self.assertAllClose(expected_enc_out, actual_enc_out) - - def _UniEncoderParams(self): - p = encoder.MTEncoderUniRNN.Params() - p.name = 'encoder' - p.emb.vocab_size = 16 - p.emb.embedding_dim = 2 - p.emb.max_num_shards = 1 - p.lstm_cell_size = 2 - p.num_lstm_layers = 3 - p.random_seed = 837464 - return p - - def _BiEncoderParams(self): - p = encoder.MTEncoderBiRNN.Params() - p.name = 'encoder' - p.emb.vocab_size = 16 - p.emb.embedding_dim = 4 - p.emb.max_num_shards = 1 - p.lstm_cell_size = 4 - p.num_lstm_layers = 3 - p.encoder_out_dim = 2 - p.random_seed = 837464 - return p - - def testBiEncoderForwardPassWithInputPacking(self): - with self.session(use_gpu=False): - with tf.variable_scope('bienc_test', reuse=tf.AUTO_REUSE): - bs = 3 - sl = 3 - tf.random.set_seed(8372749040) - p = self._BiEncoderParams() - mt_enc = encoder.MTEncoderBiRNN(p) - packed_params = p.Copy() - packed_params.packed_input = True - mt_enc_packed = encoder.MTEncoderBiRNN(packed_params) - - batch = py_utils.NestedMap() - batch.ids = tf.constant( - np.random.randint(low=0, high=15, size=[bs, sl], dtype=np.int32)) - batch.paddings = tf.zeros([bs, sl]) - - packed_batch = py_utils.NestedMap() - packed_batch.ids = tf.reshape(batch.ids, [1, -1]) - packed_batch.paddings = tf.reshape(batch.paddings, [1, -1]) - packed_batch.segment_ids = tf.constant( - [[0, 0, 0, 1, 1, 1, 2, 2, 2]], dtype=tf.float32) - packed_batch.segment_pos = tf.constant( - [[0, 1, 2, 0, 1, 2, 0, 1, 2]], dtype=tf.int32) - enc_out = mt_enc.FPropDefaultTheta(batch).encoded - enc_out = tf.transpose(enc_out, [1, 0, 2]) - - packed_enc_out = mt_enc_packed.FPropDefaultTheta(packed_batch) - packed_enc_out = tf.reshape(packed_enc_out.encoded, tf.shape(enc_out)) - - self.evaluate(tf.global_variables_initializer()) - actual_enc_out, actual_packed_enc_out = self.evaluate( - [enc_out, packed_enc_out]) - self.assertAllClose(actual_packed_enc_out, actual_enc_out) - - def testTransparentEncoderConstruction(self): - p = self._BiEncoderParams() - p.is_transparent = True - _ = encoder.MTEncoderBiRNN(p) - - def testUniEncoderForwardPass(self): - with self.session(use_gpu=False): - tf.random.set_seed(8372749040) - p = self._UniEncoderParams() - mt_enc = encoder.MTEncoderUniRNN(p) - batch = py_utils.NestedMap() - batch.ids = tf.transpose(tf.reshape(tf.range(0, 8, 1), [4, 2])) - batch.paddings = tf.zeros([2, 4]) - enc_out = mt_enc.FPropDefaultTheta(batch).encoded - - self.evaluate(tf.global_variables_initializer()) - actual_enc_out = enc_out.eval() - tf.logging.info('testUniEncoderForwardPass actual_enc_out %r' % - actual_enc_out) - expected_enc_out = [[[-4.3304257e-07, 5.4100457e-07], - [-4.0170832e-07, -2.6441572e-07]], - [[-1.7024040e-07, -1.8555815e-07], - [-6.4563977e-07, -3.7835261e-07]], - [[-2.4001852e-07, 5.1114228e-07], - [-3.4349023e-07, -1.0049351e-06]], - [[1.8068013e-07, -6.8982729e-08], - [3.3005003e-07, -8.8834116e-07]]] - self.assertAllClose(expected_enc_out, actual_enc_out) - - def testBiEncoderForwardPass(self): - with self.session(use_gpu=False): - tf.random.set_seed(8372749040) - p = self._BiEncoderParams() - mt_enc = encoder.MTEncoderBiRNN(p) - batch = py_utils.NestedMap() - batch.ids = tf.transpose(tf.reshape(tf.range(0, 8, 1), [4, 2])) - batch.paddings = tf.zeros([2, 4]) - enc_out = mt_enc.FPropDefaultTheta(batch).encoded - - self.evaluate(tf.global_variables_initializer()) - actual_enc_out = enc_out.eval() - tf.logging.info('testBiEncoderForwardPass actual_enc_out %r' % - actual_enc_out) - expected_enc_out = [[[-2.47998378e-06, 7.36457878e-06], - [7.89248020e-07, -2.67464316e-06]], - [[-2.98803275e-06, 8.20233890e-06], - [1.00139073e-06, -2.24554151e-06]], - [[-5.06675951e-06, 1.15983785e-05], - [-4.58391014e-07, -2.99553108e-07]], - [[-4.34937465e-06, 8.58816838e-06], - [-1.74859031e-06, 3.99598093e-06]]] - self.assertAllClose(expected_enc_out, actual_enc_out) - - def testBiEncoderForwardPassWithDropout(self): - with self.session(use_gpu=False): - tf.random.set_seed(8372749040) - p = self._BiEncoderParams() - p.dropout_prob = 0.5 - mt_enc = encoder.MTEncoderBiRNN(p) - batch = py_utils.NestedMap() - batch.ids = tf.transpose(tf.reshape(tf.range(0, 8, 1), [4, 2])) - batch.paddings = tf.zeros([2, 4]) - enc_out = mt_enc.FPropDefaultTheta(batch).encoded - - self.evaluate(tf.global_variables_initializer()) - actual_enc_out = enc_out.eval() - print('bi_enc_actual_enc_out_with_dropout', np.array_repr(actual_enc_out)) - expected_enc_out = [[[-1.8358192e-05, 1.2103478e-05], - [2.9347059e-06, -3.0652325e-06]], - [[-8.1282624e-06, 4.5443494e-06], - [3.0826509e-06, -5.2950490e-06]], - [[-4.6669629e-07, 2.4246765e-05], - [-1.5221613e-06, -1.9654153e-06]], - [[-1.1511075e-05, 1.9061190e-05], - [-5.7250163e-06, 9.2785704e-06]]] - self.assertAllClose(expected_enc_out, actual_enc_out) - - def testBiEncoderForwardPassWithTransparent(self): - with self.session(use_gpu=False): - tf.random.set_seed(8372749040) - p = self._BiEncoderParams() - p.is_transparent = True - mt_enc = encoder.MTEncoderBiRNN(p) - batch = py_utils.NestedMap() - batch.ids = tf.transpose(tf.reshape(tf.range(0, 8, 1), [4, 2])) - batch.paddings = tf.zeros([2, 4]) - enc_out = mt_enc.FPropDefaultTheta(batch).encoded - - self.evaluate(tf.global_variables_initializer()) - actual_enc_out = enc_out.eval() - tf.logging.info( - 'testBiEncoderForwardPassWithTransparent actual_enc_out %r' % - actual_enc_out) - expected_enc_out = [[[1.53976856e-04, -1.66475205e-04], - [-1.02031634e-04, 1.39693424e-04]], - [[1.62726530e-04, -2.22654475e-04], - [-4.89080339e-05, 1.10912690e-04]], - [[1.28586107e-04, -1.62333992e-04], - [7.22907062e-05, -9.17545694e-05]], - [[9.02724860e-05, -1.71898617e-04], - [-9.77059244e-06, 7.55862275e-05]]] - self.assertAllClose(expected_enc_out, actual_enc_out) - - -class TransformerEncoderTest(test_utils.TestCase): - - def _EncoderParams(self): - p = encoder.TransformerEncoder.Params() - p.name = 'transformer_encoder' - p.token_emb.params_init = py_utils.WeightInit.GaussianSqrtDim() - p.token_emb.vocab_size = 64 - p.token_emb.embedding_dim = 16 - p.token_emb.max_num_shards = 1 - p.token_emb.vn = py_utils.VariationalNoiseParams(1.0, False, False) - p.model_dim = 16 - p.position_emb.embedding_dim = 16 - ts = p.transformer_stack - ts.num_transformer_layers = 6 - ts.transformer_tpl.tr_atten_tpl.num_attention_heads = 2 - ts.transformer_tpl.tr_fflayer_tpl.hidden_dim = 5 - return p - - def testEncoderConstruction(self): - p = self._EncoderParams() - _ = encoder.TransformerEncoder(p) - - def testTransparentEncoderConstruction(self): - p = self._EncoderParams() - p.transformer_stack.is_transparent = True - p.transformer_stack.num_transparent_outputs = 2 - _ = encoder.TransformerEncoder(p) - - def testForwardPass(self): - with self.session(use_gpu=False): - bs = 2 - sl = 21 - tf.random.set_seed(8372749040) - p = self._EncoderParams() - mt_enc = encoder.TransformerEncoder(p) - batch = py_utils.NestedMap() - batch.ids = tf.constant( - np.random.randint(low=0, high=63, size=[bs, sl], dtype=np.int32)) - batch.paddings = tf.zeros([bs, sl]) - out = mt_enc.FPropDefaultTheta(batch) - enc_out_sum = tf.reduce_sum(out.encoded, 0) - emb_out_sum = tf.reduce_sum(out.embedded_inputs, 0) - enc_padding = out.padding - - self.evaluate(tf.global_variables_initializer()) - actual_enc_out, actual_enc_out_sum, actual_emb_out_sum, \ - actual_padding = self.evaluate( - [out.encoded, enc_out_sum, emb_out_sum, enc_padding]) - - # pyformat: disable - # pylint: disable=bad-whitespace - expected_enc_out = [ - [ 49.45291519, -31.5743885 , 39.43684387, -47.67513275, - 35.39754105, 14.41970444, 29.58752823, -43.06747055, - 24.09403419, -7.62717247, 18.48112106, 20.42408371, - 5.1519866 , -19.66542244, 29.81095314, 56.90407944], - [ 55.26333618, -30.39743614, 29.68314743, -37.61392975, - 43.02292252, 13.88345146, 15.73033905, -24.68696213, - 24.70776558, -29.18026161, 15.41469955, 27.77672577, - -5.36326742, -22.78984642, 22.15843391, 22.7237072 ]] - expected_emb_out_sum = [ - [ 3.11785889, 1.33086884, -1.96904886, -4.81911993, 1.25389254, - 1.52582073, 0.79906291, 4.07078457, -1.20546532, -2.97308111, - 0.22460097, 2.99702668, -2.29453254, 6.06631422, 1.68836212, - 5.35728741], - [ 1.41723049, -1.39409399, -1.49569404, -0.24654561, 1.09658146, - 4.51638842, 2.72023368, -0.45651400, 3.46091199, -0.43925080, - 1.02091551, 3.89704037, 1.87841535, -0.27947778, -0.91630745, - 1.34230828]] - # pylint: enable=bad-whitespace - # pyformat: enable - self.assertAllEqual(actual_enc_out.shape, [sl, bs, p.model_dim]) - self.assertAllEqual(actual_padding.shape, [sl, bs]) - self.assertAllClose( - expected_enc_out, actual_enc_out_sum, rtol=1e-05, atol=1e-05) - self.assertAllClose( - expected_emb_out_sum, actual_emb_out_sum, rtol=1e-05, atol=1e-05) - - def testForwardPassWithTaskEmb(self): - with self.session(use_gpu=False): - bs = 2 - sl = 21 - tf.random.set_seed(8372749040) - p = self._EncoderParams() - p.task_emb = p.token_emb.Copy() - p.task_emb.vocab_size = 4 - mt_enc = encoder.TransformerEncoder(p) - batch = py_utils.NestedMap() - batch.ids = tf.constant( - np.random.randint(low=0, high=63, size=[bs, sl], dtype=np.int32)) - batch.task_ids = tf.constant( - np.random.randint(low=0, high=3, size=[bs, sl], dtype=np.int32)) - batch.paddings = tf.zeros([bs, sl]) - - enc_out = mt_enc.FPropDefaultTheta(batch) - enc_out_sum = tf.reduce_sum(enc_out.encoded, 0) - - self.evaluate(tf.global_variables_initializer()) - actual_enc_out = enc_out_sum.eval() - - # pyformat: disable - # pylint: disable=bad-whitespace - expected_enc_out = [ - [ 1.2796677, -31.786997, -0.4054339, -32.61311 , - 42.41403, 11.020338, 54.115948, -61.322887, - 39.593548, 15.315696, -20.373957, 1.8548622, - -17.743631, 3.140956, 30.730812, 41.4348], - [ -1.0373995, -31.306532, -2.6323462, -32.078648, - 45.80049, 16.409424, 55.00114, -63.102333, - 40.4261, 14.198621, -23.027012, 1.0839912, - -20.739473, 0.7242553, 32.49956, 41.592197]] - # pylint: enable=bad-whitespace - # pyformat: enable - self.assertAllClose( - expected_enc_out, actual_enc_out, rtol=1e-05, atol=1e-05) - - def testForwardPassWithSourceMask(self): - with self.session(use_gpu=False): - bs = 2 - sl = 21 - tf.random.set_seed(8372749040) - p = self._EncoderParams() - p.task_emb = p.token_emb.Copy() - p.task_emb.vocab_size = 4 - # 4 tasks, 2 languages. - p.apply_source_mask = True - mt_enc = encoder.TransformerEncoder(p) - batch = py_utils.NestedMap() - batch.ids = tf.constant( - np.random.randint(low=0, high=63, size=[bs, sl], dtype=np.int32)) - batch.task_ids = tf.constant( - np.random.randint(low=0, high=3, size=[bs, sl], dtype=np.int32)) - batch.paddings = tf.zeros([bs, sl]) - - enc_out = mt_enc.FPropDefaultTheta(batch) - enc_out_sum = tf.reduce_sum(enc_out.encoded, 0) - - self.evaluate(tf.global_variables_initializer()) - actual_enc_out = enc_out_sum.eval() - # pyformat: disable - # pylint: disable=bad-whitespace - print(actual_enc_out) - - expected_enc_out = [ - [1.2796695, -31.786999, -0.4054371, -32.61311, 42.414032, 11.020337, - 54.11595, -61.322884, 39.59355, 15.315693, -20.373957, 1.8548615, - -17.743631, 3.1409538, 30.730812, 41.4348], - [-1.0374013, -31.306532, -2.6323478, -32.078648, 45.800484, 16.40942, - 55.001144, -63.10233, 40.4261, 14.19862, -23.027012, 1.0839913, - -20.739471, 0.7242559, 32.499565, 41.592197]] - # pylint: enable=bad-whitespace - # pyformat: enable - self.assertAllClose( - expected_enc_out, actual_enc_out, rtol=1e-05, atol=1e-05) - - def testForwardPassWithInputPacking(self): - with self.session(use_gpu=False): - with tf.variable_scope('transformer_test', reuse=tf.AUTO_REUSE): - bs = 3 - sl = 3 - tf.random.set_seed(8372749040) - p = self._EncoderParams() - mt_enc = encoder.TransformerEncoder(p) - packed_params = p.Copy() - packed_params.packed_input = True - mt_enc_packed = encoder.TransformerEncoder(packed_params) - - batch = py_utils.NestedMap() - batch.ids = tf.constant( - np.random.randint(low=0, high=63, size=[bs, sl], dtype=np.int32)) - batch.paddings = tf.zeros([bs, sl]) - - packed_batch = py_utils.NestedMap() - packed_batch.ids = tf.reshape(batch.ids, [1, -1]) - packed_batch.paddings = tf.reshape(batch.paddings, [1, -1]) - packed_batch.segment_ids = tf.constant( - [[0, 0, 0, 1, 1, 1, 2, 2, 2]], dtype=tf.float32) - packed_batch.segment_pos = tf.constant( - [[0, 1, 2, 0, 1, 2, 0, 1, 2]], dtype=tf.int32) - enc_out = mt_enc.FPropDefaultTheta(batch).encoded - enc_out = tf.transpose(enc_out, [1, 0, 2]) - - packed_enc_out = mt_enc_packed.FPropDefaultTheta(packed_batch) - packed_enc_out = tf.reshape(packed_enc_out.encoded, tf.shape(enc_out)) - - enc_out = tf.reduce_sum(enc_out, axis=0) - packed_enc_out = tf.reduce_sum(packed_enc_out, axis=0) - - self.evaluate(tf.global_variables_initializer()) - actual_enc_out, actual_packed_enc_out = self.evaluate( - [enc_out, packed_enc_out]) - - self.assertAllClose(actual_packed_enc_out, actual_enc_out) - - def testForwardPassSplitBatch(self): - with self.session(use_gpu=False): - bs = 8 - sl = 20 - tf.random.set_seed(8372749040) - p = self._EncoderParams() - p.random_seed = 1234 - mt_enc = encoder.TransformerEncoder(p) - - batch = py_utils.NestedMap() - batch.ids = tf.constant( - np.random.randint(low=0, high=63, size=[bs, sl], dtype=np.int32)) - batch.paddings = tf.zeros([bs, sl]) - out = mt_enc.FPropDefaultTheta(batch) - enc_out = out.encoded - emb_out = out.embedded_inputs - - inputs1, inputs2 = tf.split(batch.ids, 2, 0) - paddings1, paddings2 = tf.split(batch.paddings, 2, 0) - - batch.ids = inputs1 - batch.paddings = paddings1 - out1 = mt_enc.FPropDefaultTheta(batch) - enc_out1 = out1.encoded - emb_out1 = out1.embedded_inputs - - batch.ids = inputs2 - batch.paddings = paddings2 - out2 = mt_enc.FPropDefaultTheta(batch) - enc_out2 = out2.encoded - emb_out2 = out2.embedded_inputs - - self.evaluate(tf.global_variables_initializer()) - actual_enc_out, actual_enc_out1, actual_enc_out2, \ - actual_emb_out, actual_emb_out1, actual_emb_out2 = self.evaluate( - [enc_out, enc_out1, enc_out2, emb_out, emb_out1, emb_out2]) - self.assertAllClose(actual_enc_out, - np.concatenate([actual_enc_out1, actual_enc_out2], 1)) - self.assertAllClose(actual_emb_out, - np.concatenate([actual_emb_out1, actual_emb_out2], 1)) - - def testEncoderVars(self): - p = self._EncoderParams() - mt_enc = encoder.TransformerEncoder(p) - enc_vars = mt_enc.vars - flatten_vars = enc_vars.Flatten() - self.assertEqual(len(flatten_vars), 91) - - -class TransformerBatchMajorEncoderTest(test_utils.TestCase): - - def _TestBuilder(self, model_dim, ff_hidden_dim, num_heads, packed_input): - return self_attention_layer.Builder.Params().Set( - model_dim=model_dim, - ff_hidden_dim=ff_hidden_dim, - num_heads=num_heads, - selfatten_add_unnormalized_input=False, - selfatten_enable_value_proj=False, - packed_input=packed_input).Instantiate() - - def _EncoderParams(self, packed_input=False): - p = encoder.TransformerBatchMajorEncoder.Params() - p.name = 'transformer_encoder' - p.packed_input = packed_input - p.token_emb.params_init = py_utils.WeightInit.GaussianSqrtDim() - p.token_emb.vocab_size = 64 - p.token_emb.embedding_dim = 16 - p.token_emb.max_num_shards = 1 - p.token_emb.vn = py_utils.VariationalNoiseParams(1.0, False, False) - p.model_dim = 16 - p.position_emb.embedding_dim = 16 - stack = self._TestBuilder(16, 5, 2, packed_input).TransformerStack( - 'transformer_stack', 6) - p.transformer_stack = ( - self_attention_layer.StackedTransformerEncoderLayers.Cast(stack)) - p.params_init = py_utils.WeightInit.Xavier(scale=1.0, seed=0) - return p - - def testEncoderConstruction(self): - p = self._EncoderParams() - p.Instantiate() - - def testForwardPass(self): - with self.session(use_gpu=False) as sess: - bs = 2 - sl = 21 - d = 16 - tf.random.set_seed(8372749040) - p = self._EncoderParams() - mt_enc = p.Instantiate() - batch = py_utils.NestedMap() - batch.ids = tf.constant( - np.random.randint(low=0, high=63, size=[bs, sl], dtype=np.int32)) - batch.paddings = tf.zeros([bs, sl]) - out = mt_enc.FPropDefaultTheta(batch) - enc_out_sum = tf.reduce_sum(out.encoded) - - tf.global_variables_initializer().run() - actual_enc_out, actual_enc_out_sum = sess.run([out.encoded, enc_out_sum]) - - self.assertAllEqual([sl, bs, d], actual_enc_out.shape) - self.assertAllClose(306.010132, actual_enc_out_sum) - - def testForwardPassPackedInput(self): - with self.session(use_gpu=False) as sess: - bs = 2 - sl = 21 - d = 16 - tf.random.set_seed(8372749040) - p = self._EncoderParams(packed_input=True) - - mt_enc = p.Instantiate() - batch = py_utils.NestedMap() - batch.ids = tf.constant( - np.random.randint(low=0, high=63, size=[bs, sl], dtype=np.int32)) - - # Pack these into a single batch - packed_bs = 1 - packed_sl = 2 * sl - batch.ids = tf.reshape(batch.ids, [packed_bs, packed_sl]) - - batch.paddings = tf.zeros([packed_bs, packed_sl]) - batch.segment_pos = [ - list(range(sl)) + list(range(sl)), - ] - batch.segment_ids = [ - [0 for i in range(sl)] + [1 for i in range(sl)], - ] - - out = mt_enc.FPropDefaultTheta(batch) - enc_out_sum = tf.reduce_sum(out.encoded) - - tf.global_variables_initializer().run() - actual_enc_out, actual_enc_out_sum = sess.run([out.encoded, enc_out_sum]) - - self.assertAllEqual([packed_sl, packed_bs, d], actual_enc_out.shape) - self.assertAllClose(306.010132, actual_enc_out_sum) - - def testEncoderVars(self): - p = self._EncoderParams() - mt_enc = p.Instantiate() - enc_vars = mt_enc.vars - flatten_vars = enc_vars.Flatten() - self.assertLen(flatten_vars, 91) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/mt/input_generator.py b/lingvo/tasks/mt/input_generator.py deleted file mode 100644 index dfcf8915..00000000 --- a/lingvo/tasks/mt/input_generator.py +++ /dev/null @@ -1,1056 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Machine translation input generator.""" - -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import generic_input -from lingvo.core import hyperparams -from lingvo.core import ops -from lingvo.core import py_utils -from lingvo.core import summary_utils -from lingvo.core import tokenizers -from lingvo.tasks.mt import text_input_pb2 - -from google.protobuf import descriptor_pb2 - - -class NmtInput(base_input_generator.BaseSequenceInputGenerator): - """Generator for NMT.""" - - @classmethod - def Params(cls): - """Defaults params for `NmtInput`.""" - p = super().Params() - p.Define( - 'natural_order_model', True, - 'Whether the model consuming the input is a natural order model. Input ' - 'is generated in natural order if True. Input is generated in reversed ' - 'order if False. The value should be consistent with the underlying ' - 'model. Set to True if training or using a natural order model, ' - 'otherwise set to False.') - p.tokenizer = tokenizers.VocabFileTokenizer.Params() - p.source_max_length = 300 - return p - - def _DataSourceFromFilePattern(self, file_pattern): - - def Proc(record): - """Parses a serialized tf.Example record.""" - outputs = [ - ('source_id', tf.io.VarLenFeature(tf.int64)), - ('source_padding', tf.io.VarLenFeature(tf.float32)), - ('target_id', tf.io.VarLenFeature(tf.int64)), - ('target_padding', tf.io.VarLenFeature(tf.float32)), - ('target_label', tf.io.VarLenFeature(tf.int64)), - ('target_weight', tf.io.VarLenFeature(tf.float32)), - ] - features = tf.io.parse_single_example(record, dict(outputs)) - for k, v in features.items(): - features[k] = v.values - bucket_key = tf.cast( - tf.maximum( - tf.reduce_sum(1.0 - features['source_padding']), - tf.reduce_sum(1.0 - features['target_padding'])), tf.int32) - return [features[k] for k, _ in outputs], bucket_key - - return generic_input.GenericInput( - file_pattern=file_pattern, - processor=Proc, - dynamic_padding_dimensions=[0] * 6, - dynamic_padding_constants=[0, 1, 0, 1, 0, 0], - **self.CommonInputOpArgs()) - - def __init__(self, params): - super().__init__(params) - p = self.params - - self.natural_order_model = p.natural_order_model - - (self._src_ids, self._src_paddings, self._tgt_ids, self._tgt_paddings, - self._tgt_labels, - self._tgt_weights), self._bucket_keys = self._BuildDataSource() - - if p.pad_to_max_seq_length: - self._PadSequences() - - def _PadSequences(self): - p = self.params - assert p.source_max_length - - if min(self.infeed_bucket_batch_limit) == max( - self.infeed_bucket_batch_limit): - source_shape = [min(self.infeed_bucket_batch_limit), p.source_max_length] - target_shape = [min(self.infeed_bucket_batch_limit), p.target_max_length] - else: - source_shape = None - target_shape = None - self._src_ids = py_utils.PadSequenceDimension(self._src_ids, - p.source_max_length, 0, - source_shape) - self._src_paddings = py_utils.PadSequenceDimension(self._src_paddings, - p.source_max_length, 1, - source_shape) - self._tgt_ids = py_utils.PadSequenceDimension(self._tgt_ids, - p.target_max_length, 0, - target_shape) - self._tgt_paddings = py_utils.PadSequenceDimension(self._tgt_paddings, - p.target_max_length, 1, - target_shape) - self._tgt_labels = py_utils.PadSequenceDimension(self._tgt_labels, - p.target_max_length, 0, - target_shape) - self._tgt_weights = py_utils.PadSequenceDimension(self._tgt_weights, - p.target_max_length, 0, - target_shape) - - def InfeedBatchSize(self): - """Override BaseSequenceInputGenerator.""" - return tf.shape(self._src_ids)[0] - - def _InputBatch(self): - ret = py_utils.NestedMap() - - ret.bucket_keys = self._bucket_keys - - ret.src = py_utils.NestedMap() - ret.src.ids = tf.cast(self._src_ids, dtype=tf.int32) - ret.src.paddings = self._src_paddings - - ret.tgt = py_utils.NestedMap() - ret.tgt.ids = self._tgt_ids - ret.tgt.labels = tf.cast(self._tgt_labels, dtype=tf.int32) - ret.tgt.weights = self._tgt_weights - ret.tgt.paddings = self._tgt_paddings - - if (self.params.fprop_dtype is None or - self.params.dtype == self.params.fprop_dtype): - return ret - - def _Cast(v): - if not v.dtype.is_floating: - return v - return tf.cast(v, self.params.fprop_dtype) - - return ret.Transform(_Cast) - - -class MlPerfInput(base_input_generator.BaseSequenceInputGenerator): - """Generator for MLPerf TFRecords.""" - - @classmethod - def Params(cls): - """Default params for `MlPerfInput`.""" - p = super().Params() - - p.Define('natural_order_model', True, '') - p.Define( - 'sos_id', 0, 'Start of sentence id' - 'Note in the MLPerf encoding, this is actually , however we can ' - 'make use of it since we never actually use .') - - p.Define( - 'packed_input', False, - 'If True, then we also consume {inputs,targets}_{position,segementation}' - ) - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - self.natural_order_model = p.natural_order_model - - ( - self._src_ids, - self._src_paddings, - self._tgt_ids, - self._tgt_paddings, - self._tgt_labels, - self._tgt_weights, - self._src_seg_pos, - self._src_seg_ids, - self._tgt_seg_pos, - self._tgt_seg_ids, - ), self._bucket_keys = self._BuildDataSource() - - if p.pad_to_max_seq_length: - assert p.source_max_length - - if min(self.infeed_bucket_batch_limit) == max( - self.infeed_bucket_batch_limit): - source_shape = [ - min(self.infeed_bucket_batch_limit), p.source_max_length - ] - target_shape = [ - min(self.infeed_bucket_batch_limit), p.target_max_length - ] - else: - source_shape = None - target_shape = None - self._src_ids = py_utils.PadSequenceDimension(self._src_ids, - p.source_max_length, 0, - source_shape) - self._src_paddings = py_utils.PadSequenceDimension( - self._src_paddings, p.source_max_length, 1, source_shape) - self._tgt_ids = py_utils.PadSequenceDimension(self._tgt_ids, - p.target_max_length, 0, - target_shape) - self._tgt_paddings = py_utils.PadSequenceDimension( - self._tgt_paddings, p.target_max_length, 1, target_shape) - self._tgt_labels = py_utils.PadSequenceDimension(self._tgt_labels, - p.target_max_length, 0, - target_shape) - self._tgt_weights = py_utils.PadSequenceDimension(self._tgt_weights, - p.target_max_length, 0, - target_shape) - - self._src_seg_ids = py_utils.PadSequenceDimension(self._src_seg_ids, - p.source_max_length, 0, - source_shape) - self._src_seg_pos = py_utils.PadSequenceDimension(self._src_seg_pos, - p.source_max_length, 0, - source_shape) - self._tgt_seg_ids = py_utils.PadSequenceDimension(self._tgt_seg_ids, - p.target_max_length, 0, - target_shape) - self._tgt_seg_pos = py_utils.PadSequenceDimension(self._tgt_seg_pos, - p.target_max_length, 0, - target_shape) - - def InfeedBatchSize(self): - """Override BaseSequenceInputGenerator.""" - return tf.shape(self._src_ids)[0] - - def _DataSourceFromFilePattern(self, file_pattern): - p = self._params - - def _DerivePaddingsAndIds(src_ids, tgt_labels): - """tgt_ids is tgt_labels shifted right by one, with a SOS ID prepended.""" - tgt_ids = tf.concat([[p.sos_id], tgt_labels[:-1]], axis=0) - src_paddings = tf.zeros(tf.shape(src_ids), dtype=tf.float32) - tgt_paddings = tf.zeros(tf.shape(tgt_ids), dtype=tf.float32) - tgt_weights = tf.ones(tf.shape(tgt_ids), dtype=tf.float32) - - bucket_key = tf.cast( - tf.maximum( - tf.reduce_sum(1.0 - src_paddings), - tf.reduce_sum(1.0 - tgt_paddings)), tf.int32) - - return src_paddings, tgt_ids, tgt_paddings, tgt_weights, bucket_key - - def _ProcPacked(record): - """TFExample -> Tensors for PackedInput.""" - outputs = [ - ('inputs', tf.io.VarLenFeature(tf.int64)), - ('targets', tf.io.VarLenFeature(tf.int64)), - ('inputs_segmentation', tf.io.VarLenFeature(tf.int64)), - ('inputs_position', tf.io.VarLenFeature(tf.int64)), - ('targets_segmentation', tf.io.VarLenFeature(tf.int64)), - ('targets_position', tf.io.VarLenFeature(tf.int64)), - ] - - features = tf.io.parse_single_example(record, dict(outputs)) - for k, v in features.items(): - features[k] = v.values - - src_ids = features['inputs'] - tgt_labels = features['targets'] - - src_pos = features['inputs_position'] - src_seg = features['inputs_segmentation'] - - tgt_pos = features['targets_position'] - tgt_seg = features['targets_segmentation'] - - src_paddings, tgt_ids, tgt_paddings, tgt_weights, bucket_key = _DerivePaddingsAndIds( - src_ids, tgt_labels) - return [ - src_ids, - src_paddings, - tgt_ids, - tgt_paddings, - tgt_labels, - tgt_weights, - src_pos, - src_seg, - tgt_pos, - tgt_seg, - ], bucket_key - - def _Proc(record): - """Parses a serialized tf.Example record.""" - outputs = [ - ('inputs', tf.io.VarLenFeature(tf.int64)), - ('targets', tf.io.VarLenFeature(tf.int64)), - ] - features = tf.io.parse_single_example(record, dict(outputs)) - for k, v in features.items(): - features[k] = v.values - - src_ids = features['inputs'] - tgt_labels = features['targets'] - - # Derive trivial segmentation for unpacked input. - src_paddings, tgt_ids, tgt_paddings, tgt_weights, bucket_key = _DerivePaddingsAndIds( - src_ids, tgt_labels) - - src_len = tf.shape(src_ids)[0] - tgt_len = tf.shape(tgt_ids)[0] - src_pos = tf.range(src_len, dtype=tf.int32) - src_seg = tf.zeros_like(src_paddings) - tgt_pos = tf.range(tgt_len, dtype=tf.int32) - tgt_seg = tf.zeros_like(tgt_paddings) - - return [ - src_ids, - src_paddings, - tgt_ids, - tgt_paddings, - tgt_labels, - tgt_weights, - src_pos, - src_seg, - tgt_pos, - tgt_seg, - ], bucket_key - - if not p.packed_input: - processor_fn = _Proc - else: - processor_fn = _ProcPacked - - return generic_input.GenericInput( - file_pattern=file_pattern, - processor=processor_fn, - dynamic_padding_dimensions=[0] * 10, - dynamic_padding_constants=[0, 1, 0, 1, 0, 0, 0, 0, 0, 0], - **self.CommonInputOpArgs()) - - def _InputBatch(self): - ret = py_utils.NestedMap() - ret.bucket_keys = self._bucket_keys - - ret.src = py_utils.NestedMap() - ret.src.ids = tf.cast(self._src_ids, dtype=tf.int32) - ret.src.paddings = self._src_paddings - - ret.tgt = py_utils.NestedMap() - ret.tgt.ids = self._tgt_ids - ret.tgt.labels = tf.cast(self._tgt_labels, dtype=tf.int32) - ret.tgt.weights = self._tgt_weights - ret.tgt.paddings = self._tgt_paddings - - ret.src.segment_pos = self._src_seg_pos - ret.src.segment_ids = self._src_seg_ids - - ret.tgt.segment_pos = self._tgt_seg_pos - ret.tgt.segment_ids = self._tgt_seg_ids - - if (self.params.fprop_dtype is None or - self.params.dtype == self.params.fprop_dtype): - return ret - - def _Cast(v): - if not v.dtype.is_floating: - return v - return tf.cast(v, self.params.fprop_dtype) - - return ret.Transform(_Cast) - - -def _GetSegmentPos(weights): - """Returns a segment_pos tensor from the given weights tensor.""" - maxlen = tf.shape(weights)[1] - ret = tf.cast(tf.range(maxlen), dtype=tf.float32) - return tf.cast(weights * ret, dtype=tf.int32) - - -def _GetDescriptorSetForTextInput(): - """Returns a string for tf.io.decode_proto's descriptor_source.""" - file_descriptor_set = descriptor_pb2.FileDescriptorSet() - text_input_pb2.DESCRIPTOR.CopyToProto(file_descriptor_set.file.add()) - return b'bytes://' + file_descriptor_set.SerializeToString() - - -class TextPackedInput(base_input_generator.BaseSequenceInputGenerator): - """Generator for packed text input.""" - - @classmethod - def Params(cls): - r"""Defaults params for TextInput. - - Returns: - A Params object for TextPackedInput. - - Notes about usage: - - * Input files contain UTF8 encoded texts. p.input_file_type controls - what format to extract these texts from. The default is 'tsv', in which - case p.file_pattern should be prefixed by file type 'text:', and - every line in the input file should have text columns separated by - a tab '\t'. Otherwise p.input_file_type can be Sentence or SentencePair - protos. - - For tsv input files, in the default case, the file should contain 2 - columns, for the source and the target sentence. Special cases: - - - When quality scores are present (see p.quality_score_filter_fn below), - it should contain 3 columns, the last being a quality score. - - When MASS is enabled, it should contain a single column. - - * p.tokenizer or p.tokenizer_dict is used to perform string to id - conversions. If key `src` or `tgt` is present in p.tokenizer_dict, - it will be used for generating the ids for src or tgt, respectively. - Otherwise the default tokenizer will be used. - - * p.packing_factor depends on the training data and max lengths used. - - If this value is too small, we generate packed batches that contain - too many padding that could have been used to pack more examples. - If this value is too large, we use more memory and randomly discard - examples that could not fit. - - One can look at the num_samples_in_batch graph to determine if its - value is too small. For example, with an effective scaled batch size of - 1024, suppose we set p.packing_factor=3.0, and observe that - num_samples_in_batch is saturated at 3072(=1024x3), this means 3.0 is - likely too small. If we instead observe that num_samples_in_batch - fluctuates around 2500, this means 3.0 is larger than needed. - - We believe that there can be a slight bias against longer sequences - when packing is enabled. The remedy is either use larger effective - batch size, or use a larger-than-optimal packing factor when effective - batch size is smaller. For example, p.packing_factor = 8 seems to work - reasonably well in practice. - - * p.source_max_length and p.target_max_length control both the shape of - the generated input batch (how long each row is) and the filtering - (max allowed lengths for source and targt, respectively). - - p.bucket_upper_bound also conrols the filtering of examples. Inputs - with either source or target sequence lengths exceeding it will be - filtered out. - - It's not meaningful to set p.bucket_upper_bound higher than both - p.source_max_length and p.target_max_length. - - When packing is enabled, however, a smaller p.bucket_upper_bound means - that individual sequences have a smaller max length, but the packed - batch may have a larger total length. - - * p.file_pattern_task_ids, p.task_to_{src,tgt}_lang_map are all used - to manipulate batch.{src,tgt}.task_ids. - - For each eaxmple, its task is obtained from the source id, which is - the index of the example's origin file in p.file_pattern. The task - id populated in the input batch is determined by: - p.task_to_{src,tgt}_lang_map[ p.file_pattern_task_ids[souce_id] ], - for src and tgt, respectively, where if a list is empty it falls - back to an identity map. - - In the future we may define a separate lang_ids field to the input - batch to disambiguate. - - * p.quality_score_filter_fn can be used when a column of quality score - is present in the input .tsv file. The quality score must be the last - column. This filter function returns True to filter, e.g. use - p.quality_score_filter_fn = lambda x: x <= 0.3 for scores where higher - means better. - - p.quality_score_filter_fn typically should only contain a simple - comparison (<, >, <=, or >=), as it relies on tf.Tensor's overloading - of __le__() etc. to work. For example: lambda x: ( 0.3 < x and x < 0.9) - won't work. But tf.math.logical_and(0.3 < x, x < 0.9) is okay. - - Also note that 'p.quality_score_filter_fn = lambda _: False' is - equivalent with 'p.quality_score_filter_fn = None', in which case - no quality score column is needed (or evaluated). - - * Consider enabling multithreading for the trainer job (in the Train() - method). For example: p.num_batcher_threads = 128. - """ - p = super().Params() - - p.Define('file_pattern_task_ids', [], - 'task_id corresponding to list of file_patterns.') - p.Define('task_to_src_lang_map', [], 'Map of task id to src language id.') - p.Define('task_to_tgt_lang_map', [], 'Map of task id to tgt language id.') - - p.Define( - 'packing_factor', None, - 'A multiplicative factor for packing. This is the ratio between ' - 'pre-packing batch size and after-packing batch size. If None, ' - 'packing is disabled; otherwise the packing factor should be a ' - 'float with a value greater than 1.') - - p.Define( - 'quality_score_filter_fn', None, - 'A user defined boolean function on a float (quality score). ' - 'When present, the input .tsv file has an additional column ' - 'of floats representing a quality score, and each line is ' - 'filtered out when this function returns True on that score.') - - p.Define( - 'input_file_type', 'tsv', 'The type of input file contents.' - ' Must be one of ["tsv", "sentence_proto"], for tab-separated' - ' values, or Sentence/SentencePair protos, respectively.') - p.Define( - 'single_column_input', False, 'Indicates input is single-column rather' - ' than double-column. When input_file_type is sentence_proto, this' - ' means Sentence proto rather than SentencePair proto.') - - p.Define('natural_order_model', True, 'Only True is supported now.') - p.Define('target_language', '', 'Language on target side.') - p.Define('mass_layer', None, 'If not None, use the specified layer to do ' - 'MASS masking.') - p.Define( - 'mass_task_ids', None, 'List of task IDs for MASS. If None and ' - 'single_column_input=True, apply MASS to all tasks, otherwise ' - 'only apply to the specified tasks.') - # Back translation - p.Define('bt_task_ids', [], 'List of task ids for back-translation.') - # Denoising (https://arxiv.org/pdf/1711.00043) - p.Define('denoise', hyperparams.Params(), 'Params for denosing tasks.') - p.denoise.Define('task_ids', [], 'List of task IDs for denoising.') - p.denoise.Define('noise_sent_prob', 1, - 'Probability of noising an input sentence.') - p.denoise.Define( - 'shuffle_tok_range', 3, 'Range of noise for shuffling tokens, following' - ' https://arxiv.org/pdf/1711.00043. Note that shuffle_tok_range of 3 ' - 'implies tokens may be permuted at most 3 position.') - p.denoise.Define('drop_tok_prob', 0.1, 'Probability of dropping tokens.') - p.denoise.Define('blank_tok_prob', 0.1, 'Probability of blanking tokens.') - p.denoise.Define('blank_id', 3, 'ID of blank token.') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - if not p.natural_order_model: - raise ValueError('Only p.natural_order_model=True is supported now.') - self.natural_order_model = p.natural_order_model - - if p.packing_factor: - # Packing is enabled. We override p.bucket_batch_limit with the - # pre-packing batch size. - if p.packing_factor <= 1.0: - raise ValueError('p.packing_factor must be > 1.0: ', p.packing_factor) - if len(p.bucket_upper_bound) != 1 or len(p.bucket_batch_limit) != 1: - raise ValueError( - 'when packing is enabled, p.bucket_upper_bound ' - 'and p.bucket_batch_limits must be arrays of length ' - '1:', p.bucket_upper_bound, p.bucket_batch_limit) - self._packed_batch_size = p.bucket_batch_limit[0] - p.bucket_batch_limit[0] = int(p.bucket_batch_limit[0] * p.packing_factor) - else: - self._packed_batch_size = None - - # Ensure that the max lengths are not None, as tokenizer.StringsToIds() - # might use it to pad the encoded ids tensor. - if p.target_max_length is None: - p.target_max_length = p.bucket_upper_bound[-1] - if p.source_max_length is None: - p.source_max_length = p.target_max_length - - if p.input_file_type not in ['tsv', 'sentence_proto']: - raise ValueError('p.input_file_type must be one of ["tsv",' - ' "sentence_proto"], got {}'.format( - params.input_file_type)) - if p.quality_score_filter_fn: - if not isinstance(p.quality_score_filter_fn(0.0), bool) and not ( - isinstance(p.quality_score_filter_fn(0.0), tf.Tensor) and - p.quality_score_filter_fn(0.0).dtype == tf.bool): - raise ValueError( - 'p.quality_score_filter_fn must return a bool on a float input, ' - 'e.g. p.quality_score_filter_fn = lambda x: x <= 0.3.') - if p.input_file_type != 'tsv': - raise ValueError( - 'p.quality_score_filter_fn requires p.input_file_type == "tsv".') - - self._src_tokenizer_key = ('src' if 'src' in self.tokenizer_dict else - base_input_generator.DEFAULT_TOKENIZER_KEY) - self._src_tokenizer = self.tokenizer_dict[self._src_tokenizer_key] - self._tgt_tokenizer_key = ('tgt' if 'tgt' in self.tokenizer_dict else - base_input_generator.DEFAULT_TOKENIZER_KEY) - self._tgt_tokenizer = self.tokenizer_dict[self._tgt_tokenizer_key] - - # TODO(alisonlui): Support single-column Sentence proto input. - if p.single_column_input and p.input_file_type == 'sentence_proto': - raise NotImplementedError( - 'Single column Sentence proto input not yet supported.') - if p.mass_layer is not None and not p.single_column_input: - raise ValueError('Must be single_column_input if mass layer is provided.') - if p.mass_layer is not None: - # Creat the MASS layer (wrapper for the MASS op). - self.CreateChild('mass_layer', p.mass_layer) - - # A `.NestedMap` of input tensors for the current input batch. - # We memoize it here to avoid accidentally calling self._BuildDataSource() - # more than once. - self._batch = self._DataSourceToInputBatch() - - def _GetBucketKey(self, features, filtered): - """Returns a the bucket key for a given input.""" - # The token ids are not truncated if and only if it ends with padding - # or the last id is EOS. - src_fits = tf.math.logical_or( - tf.math.equal(features.src.ids_indicator[-1], 0), - tf.math.equal(features.src.ids[-1], self._src_tokenizer.eos_id)) - tgt_fits = tf.math.logical_or( - tf.math.equal(features.tgt.ids_indicator[-1], 0), - tf.math.equal(features.tgt.labels[-1], self._tgt_tokenizer.eos_id)) - - # We return the max of sourcec or target sequence length if and only if both - # src and tgt fit. Otherwise we return a key of -1 to filter out this input. - def _MaxLen(): - src_len = tf.cast( - tf.math.reduce_sum(features.src.ids_indicator), dtype=tf.int32) - tgt_len = tf.cast( - tf.math.reduce_sum(features.tgt.ids_indicator), dtype=tf.int32) - return tf.math.maximum(src_len, tgt_len) - - filtered = tf.math.logical_or( - filtered, tf.math.logical_not(tf.math.logical_and(src_fits, tgt_fits))) - return tf.cond(filtered, lambda: -1, _MaxLen) - - def _GetTaskIds(self, source_id): - """Look up the correct task_id from the source_id tensor.""" - if self.params.file_pattern_task_ids: - file_task_ids = tf.constant( - self.params.file_pattern_task_ids, dtype=tf.int32) - return tf.gather(file_task_ids, source_id) - return source_id - - def _GetLangIds(self, source_id): - """Look up the correct lang_id from the source_id tensor.""" - task_id = self._GetTaskIds(source_id) - src_lang_id = task_id - tgt_lang_id = task_id - if self.params.task_to_src_lang_map: - src_langs = tf.constant(self.params.task_to_src_lang_map, dtype=tf.int32) - src_lang_id = tf.gather(src_langs, task_id) - if self.params.task_to_tgt_lang_map: - tgt_langs = tf.constant(self.params.task_to_tgt_lang_map, dtype=tf.int32) - tgt_lang_id = tf.gather(tgt_langs, task_id) - return src_lang_id, tgt_lang_id - - def _ProcessSingleInput(self, source_id, src, tgt): - """Performs strings-to-ids on the given input pair via p.tokenizer_dict.""" - _, src_labels, src_paddings = self.StringsToIds( - tf.reshape(src, [1]), is_source=True, key=self._src_tokenizer_key) - tgt_ids, tgt_labels, tgt_paddings = self.StringsToIds( - tf.reshape(tgt, [1]), is_source=False, key=self._tgt_tokenizer_key) - # Mask positions to 0 where padding is 1 for consistency. We do this because - # tokenizer implementation may use EOS token to pad. - src_labels = py_utils.ApplyPadding(src_paddings, src_labels) - tgt_ids = py_utils.ApplyPadding(tgt_paddings, tgt_ids) - tgt_labels = py_utils.ApplyPadding(tgt_paddings, tgt_labels) - - features = py_utils.NestedMap() - features.src = py_utils.NestedMap() - features.src.ids = src_labels - # ids_indicator is 1 if and only if the output from tokenizer has a - # non-padded id. Unlike weights, it will not mutate and can be used for - # determining actual sequence length, for example. - features.src.ids_indicator = 1 - src_paddings - features.src.weights = 1 - src_paddings - features.src.paddings = src_paddings - features.tgt = py_utils.NestedMap() - features.tgt.ids = tgt_ids - features.tgt.labels = tgt_labels - features.tgt.ids_indicator = 1 - tgt_paddings - features.tgt.weights = 1 - tgt_paddings - features.tgt.paddings = tgt_paddings - - src_task_id, tgt_task_id = self._GetLangIds(source_id) - # task_ids are padded with zeros. - features.src.task_ids = tf.cast( - features.src.ids_indicator, dtype=tf.int32) * src_task_id - features.src.source_ids = tf.cast( - features.src.ids_indicator, dtype=tf.int32) * source_id - features.tgt.task_ids = tf.cast( - features.tgt.ids_indicator, dtype=tf.int32) * tgt_task_id - features.tgt.source_ids = tf.cast( - features.tgt.ids_indicator, dtype=tf.int32) * source_id - - if not py_utils.use_tpu(): - features.src.strs = src - features.tgt.strs = tgt - return features.Transform(tf.squeeze) - - def _ProcessMASSInput(self, source_id, src): - """Perform MASS input processing.""" - if self.do_eval or self.mass_layer is None: - # At eval time, we copy src to tgt - return self._ProcessSingleInput(source_id, src, src) - - _, labels, paddings = self.StringsToIds( - tf.reshape(src, [1]), is_source=True, key=self._src_tokenizer_key) - weights = 1 - paddings - actual_seq_len = tf.cast(tf.reduce_sum(weights, 1), tf.int32) - src_lang_ids, tgt_lang_ids = self._GetLangIds(source_id) - - mass_out = self.mass_layer.Mask(labels, weights, actual_seq_len) - - features = py_utils.NestedMap() - features.src = py_utils.NestedMap() - features.src.ids = mass_out.src.ids - features.src.paddings = paddings - features.src.weights = weights - features.src.task_ids = tf.cast( - features.src.weights, dtype=tf.int32) * src_lang_ids - features.src.source_ids = tf.cast( - features.src.weights, dtype=tf.int32) * source_id - features.src.ids_indicator = weights - features.tgt = py_utils.NestedMap() - features.tgt.ids = mass_out.tgt.ids - features.tgt.labels = mass_out.tgt.labels - features.tgt.paddings = paddings - features.tgt.weights = mass_out.tgt.weights - features.tgt.task_ids = tf.cast(weights, dtype=tf.int32) * tgt_lang_ids - features.tgt.source_ids = tf.cast(weights, dtype=tf.int32) * source_id - features.tgt.ids_indicator = weights - - if not py_utils.use_tpu(): - features.src.strs = src - features.tgt.strs = src - return features.Transform(tf.squeeze) - - def _ReadRecordTsv(self, record): - """Reads a single input record from a tab-separated values file.""" - # Assuming UTF8 text input separated by tabs. - sentences = tf.strings.split([record], sep='\t', result_type='RaggedTensor') - # If the row_lengths are not enough (e.g. row has only 1 column), - # record_batcher throws away this record but it does not crash the program - # per lingvo/core/ops/record_batcher.cc. - # This means that it's okay if the file contains more columns than needed. - src = sentences[0, 0] - tgt = sentences[0, 1] - # We manually filter the record if either source or target sentence is an - # empty string. - filtered = tf.math.logical_or( - tf.math.equal(tf.strings.length(src), 0), - tf.math.equal(tf.strings.length(tgt), 0)) - if not self.params.quality_score_filter_fn: - return src, tgt, filtered - filtered = tf.math.logical_or( - filtered, - self.params.quality_score_filter_fn( - tf.strings.to_number(sentences[0, 2]))) - return src, tgt, filtered - - def _ReadRecordTsvSingleColumn(self, record): - """Reads an input record, taking first column of one or more TSV columns.""" - # Assuming UTF8 text input which may have 1 or more tab-separated columns - sentences = tf.strings.split([record], sep='\t', result_type='RaggedTensor') - src = sentences[0, 0] - filtered = tf.math.equal(tf.strings.length(src), 0) - return src, filtered - - def _ReadRecordSentencePairProto(self, record): - """Reads the input record as a binary SentencePair proto.""" - # We defer handling the `lang` field in the proto until TextPackedInput - # figures out how to handle lang_ids. For now `lang` fields are ignored. - _, sentence_protos = tf.io.decode_proto( - bytes=record, - message_type='tensorflow.lingvo.SentencePair', - field_names=['src_sentence', 'tgt_sentence'], - output_types=[tf.string, tf.string], - descriptor_source=_GetDescriptorSetForTextInput()) - sentence_protos = tf.squeeze(sentence_protos) - _, sentences = tf.io.decode_proto( - bytes=sentence_protos, - message_type='tensorflow.lingvo.Sentence', - field_names=['sentence'], - output_types=[tf.string], - descriptor_source=_GetDescriptorSetForTextInput()) - sentences = tf.squeeze(sentences) - return sentences[0], sentences[1] - - def _DataSourceFromFilePattern(self, file_pattern, input_source_weights=None): - - def Processor(source_id, record): - """Parses a record, which is a line of text.""" - - task_id = self._GetTaskIds(source_id) - - if self.params.input_file_type == 'tsv': - - def _ApplyMass(task_id): - mass_task_ids = tf.constant(self.params.mass_task_ids, dtype=tf.int32) - return tf.reduce_any(tf.equal(task_id, mass_task_ids)) - - def _MASSInput(): - src, filtered = self._ReadRecordTsvSingleColumn(record) - return self._ProcessMASSInput(source_id, src), filtered - - def _SingleInput(): - src, tgt, filtered = self._ReadRecordTsv(record) - return self._ProcessSingleInput(source_id, src, tgt), filtered - - if self.params.single_column_input: - # For monolingual input, MASS is applied by default. - # If mass_task_ids is specified, only apply MASS to specified tasks. - if self.params.mass_task_ids is not None: - cond = _ApplyMass(task_id) - features, filtered = tf.cond(cond, _MASSInput, _SingleInput) - else: - features, filtered = _MASSInput() - else: - features, filtered = _SingleInput() - - else: - src, tgt = self._ReadRecordSentencePairProto(record) - filtered = tf.constant(False, dtype=tf.bool) - features = self._ProcessSingleInput(source_id, src, tgt) - - return features, self._GetBucketKey(features, filtered) - - return generic_input.GenericInput( - processor=Processor, - file_pattern=file_pattern, - input_source_weights=input_source_weights, - **self.CommonInputOpArgs()) - - def _Pack(self, batch): - """Packs a given batch. - - Note that this may change the batch size. - - This function packs the input batch and adds .segment_ids and .segment_pos - fields to its `src` and `tgt` fields. - - Args: - batch: a `.NestedMap` of input tensors to be packed. It is modified in - place. - """ - src_actual_seq_len = tf.math.reduce_sum( - tf.cast(batch.src.ids_indicator, tf.int32), axis=1) - tgt_actual_seq_len = tf.math.reduce_sum( - tf.cast(batch.tgt.ids_indicator, tf.int32), axis=1) - summary_utils.histogram('source_seq_lengths', src_actual_seq_len) - summary_utils.histogram('target_seq_lengths', tgt_actual_seq_len) - - if not self.params.packing_factor: - # Supply segment_ids and segment_pos with no packing. - batch.src.segment_ids = batch.src.ids_indicator - batch.src.segment_pos = _GetSegmentPos(batch.src.ids_indicator) - batch.tgt.segment_ids = batch.tgt.ids_indicator - batch.tgt.segment_pos = _GetSegmentPos(batch.tgt.ids_indicator) - return - - (src_segment_ids, src_segment_pos, src_indices_in_input, tgt_segment_ids, - tgt_segment_pos, tgt_indices_in_input) = ops.pack_sequences( - src_actual_seq_len, tgt_actual_seq_len, self._ScaledBatchSize(), - self.params.source_max_length, self.params.target_max_length) - - uniq_src_indices_in_input = tf.unique( - tf.reshape(src_indices_in_input, [-1])).y - uniq_tgt_indices_in_input = tf.unique( - tf.reshape(tgt_indices_in_input, [-1])).y - summary_utils.histogram( - 'packed_source_seq_lengths', - tf.gather(src_actual_seq_len, uniq_src_indices_in_input, axis=0)) - summary_utils.histogram( - 'packed_target_seq_lengths', - tf.gather(tgt_actual_seq_len, uniq_tgt_indices_in_input, axis=0)) - - # We deferred adding .paddings and use its complement .ids_indicator - # exclusively so that we can apply the packing with padding set to 0 for all - # fields. - def ApplyPackingToSource(x): - if x.dtype == tf.string: - return ops.apply_packing(x, '\t', src_segment_ids, src_indices_in_input) - return ops.apply_packing(x, 0, src_segment_ids, src_indices_in_input) - - src_paddings = ops.apply_packing(batch.src.paddings, 1, src_segment_ids, - src_indices_in_input) - batch.src = batch.src.Transform(ApplyPackingToSource) - batch.src.paddings = src_paddings - batch.src.segment_ids = tf.cast(src_segment_ids, tf.float32) - batch.src.segment_pos = src_segment_pos - - def ApplyPackingToTarget(x): - if x.dtype == tf.string: - return ops.apply_packing(x, '\t', tgt_segment_ids, tgt_indices_in_input) - return ops.apply_packing(x, 0, tgt_segment_ids, tgt_indices_in_input) - - tgt_paddings = ops.apply_packing(batch.tgt.paddings, 1, tgt_segment_ids, - tgt_indices_in_input) - batch.tgt = batch.tgt.Transform(ApplyPackingToTarget) - batch.tgt.paddings = tgt_paddings - batch.tgt.segment_ids = tf.cast(tgt_segment_ids, tf.float32) - batch.tgt.segment_pos = tgt_segment_pos - - def _ScaledBatchSize(self): - # Adjust (post-packing) batch size according to the cluster spec. - # See the impl of BaseSequenceInputGenerator.infeed_bucket_batch_limit() - cluster = self.cluster - batch_size = (self._packed_batch_size or self.params.bucket_batch_limit[0]) - scaled_batch_size = batch_size * cluster.num_splits_per_client - if self.params.use_per_host_infeed and cluster.num_tpu_hosts > 0: - scaled_batch_size = scaled_batch_size // cluster.num_tpu_hosts - return scaled_batch_size - - # TODO(yujieq): create a separate wrapper layer for the noise-adding process - def _AddNoise(self, batch): - """Adding noise the src (see https://arxiv.org/pdf/1711.00043). - - This function implement 3 types of noise (hyparams defined in - self.params.denoise): - 1) slightly shuffle the sentence following p.shuffle_tok_range - 2) randomly drop tokens with probability p.drop_tok_prob - 3) randomly mask tokens with probability p.blank_tok_prob - The noises are added to the input with probability p.noise_sent_prob. - - Args: - batch: a `.NestedMap` of the input batch. - """ - - def IsSpecialExample(task_ids, special_task_ids): - """A utility function indicates whether inputs belong to specific tasks. - - Args: - task_ids: Task ids for the input batch. Tensor of shape [batch]. - special_task_ids: A list of specified task ids. - - Returns: - A tensor indicating whether each sample in the batch belong to the - specified task. Return a tensor of size [batch]. - """ - batch_size = py_utils.GetShape(task_ids)[0] - return tf.reduce_any( - tf.equal( - tf.expand_dims(task_ids, -1), - tf.cast( - tf.broadcast_to( - special_task_ids, - [batch_size, len(special_task_ids)]), tf.int32)), -1) - - p = self.params.denoise - batch_size = tf.shape(batch.src.ids)[0] - source_max_len = tf.shape(batch.src.ids)[1] - - # Shuffle tokens according to p.shuffle_tok_range - noise = tf.random.uniform([batch_size, source_max_len], 0, - p.shuffle_tok_range + 1) - - # Don't shuffle eos or padding - shuffle_tok_range = tf.fill([batch_size, source_max_len], - float(p.shuffle_tok_range)) - shifted_paddings = tf.pad( - batch.src.paddings[:, 1:], [[0, 0], [0, 1]], constant_values=1) - noise = tf.where(tf.equal(shifted_paddings, 0), noise, shuffle_tok_range) - indices = tf.broadcast_to( - tf.range(source_max_len, dtype=tf.int32), [batch_size, source_max_len]) - noisy_indices = tf.cast(indices, dtype=tf.float32) + noise - permutations = tf.argsort(noisy_indices) - stacked = tf.stack([batch.src.ids, permutations], axis=1) - denoise_src_ids = tf.stack( - tf.map_fn(lambda x: tf.gather(x[0], x[1]), stacked), axis=0) - - # Select tokens to drop with probability=p.drop_tok_prob - random_drop_tok = tf.random.uniform([batch_size, source_max_len]) - # Don't drop eos token - is_keep_tok = tf.math.logical_or( - tf.greater(random_drop_tok, p.drop_tok_prob), - tf.equal(denoise_src_ids, self._src_tokenizer.eos_id)) - denoise_src_ids = tf.ragged.boolean_mask(denoise_src_ids, - is_keep_tok).to_tensor( - default_value=0, - shape=tf.shape(batch.src.ids)) - denoise_src_paddings = tf.ragged.boolean_mask( - batch.src.paddings, is_keep_tok).to_tensor( - default_value=1, shape=tf.shape(batch.src.ids)) - - # Select tokens to blank with probability=p.blank_tok_prob - # Don't blank eos token - random_blank_tok = tf.random.uniform([batch_size, source_max_len]) - shifted_paddings = tf.pad( - denoise_src_paddings[:, 1:], [[0, 0], [0, 1]], constant_values=1) - is_blank_tok = tf.math.logical_and( - tf.less(random_blank_tok, p.blank_tok_prob), - tf.equal(shifted_paddings, 0)) - blank_id = tf.fill([batch_size, source_max_len], p.blank_id) - denoise_src_ids = tf.where(is_blank_tok, blank_id, denoise_src_ids) - - # Select denoising task examples with probability=p.denoise_sent_prob - random_uniform_sent = tf.random.uniform([batch_size]) - is_denoise_sent = tf.math.logical_and( - tf.less(random_uniform_sent, p.noise_sent_prob), - IsSpecialExample( - self._GetTaskIds(batch.src.source_ids[:, 0]), p.task_ids)) - batch.src.ids = tf.where(is_denoise_sent, denoise_src_ids, batch.src.ids) - batch.src.paddings = tf.where(is_denoise_sent, denoise_src_paddings, - batch.src.paddings) - batch.src.ids_indicator = 1 - batch.src.paddings - batch.src.weights = batch.src.ids_indicator - - def _DataSourceToInputBatch(self): - """The current input batch as a `.NestedMap` of input tensors.""" - ret, _ = self._BuildDataSource() - - p = self.params - if p.denoise.task_ids and p.denoise.noise_sent_prob > 0: - self._AddNoise(ret) - - self._Pack(ret) - if 'weights' not in ret.src or 'weights' not in ret.tgt: - ret.src.weights = ret.src.ids_indicator - ret.tgt.weights = ret.tgt.ids_indicator - if 'paddings' not in ret.src or 'paddings' not in ret.tgt: - ret.src.paddings = 1 - ret.src.weights - ret.tgt.paddings = 1 - ret.tgt.weights - del ret.src.ids_indicator - del ret.tgt.ids_indicator - - if self.params.pad_to_max_seq_length: - assert self.params.source_max_length - - def _EnsureSrcShape(x): - if x.dtype == tf.string: - return tf.ensure_shape(x, [self._ScaledBatchSize()]) - return tf.ensure_shape( - x, [self._ScaledBatchSize(), self.params.source_max_length]) - - def _EnsureTgtShape(x): - if x.dtype == tf.string: - return tf.ensure_shape(x, [self._ScaledBatchSize()]) - return tf.ensure_shape( - x, [self._ScaledBatchSize(), self.params.target_max_length]) - - ret.src = ret.src.Transform(_EnsureSrcShape) - ret.tgt = ret.tgt.Transform(_EnsureTgtShape) - - summary_utils.histogram('source_token_ids', ret.src.ids) - summary_utils.histogram('target_token_ids', ret.tgt.ids) - - # Casts floating point tensors to fprop_dtype before returning. - return ret.Transform(self.Cast) - - def _InputBatch(self): - """The current input batch. - - Returns: - A `.NestedMap` of input tensors. - """ - return self._batch - - def GlobalBatchSize(self): - """Returns the total number of examples in the current batch.""" - # The number of examples is indicated by the segment_ids of the target. - num_segments = tf.math.reduce_max(self._batch.tgt.segment_ids, axis=1) - return tf.reduce_sum(tf.cast(num_segments, dtype=tf.int32)) diff --git a/lingvo/tasks/mt/input_generator_test.py b/lingvo/tasks/mt/input_generator_test.py deleted file mode 100644 index a354fc64..00000000 --- a/lingvo/tasks/mt/input_generator_test.py +++ /dev/null @@ -1,452 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for input generator.""" - -import lingvo.compat as tf -from lingvo.core import py_utils -from lingvo.core import test_helper -from lingvo.core import test_utils -from lingvo.core import tokenizers -from lingvo.tasks.mt import input_generator -import numpy as np - - -class InputTest(test_utils.TestCase): - - def _CreateMlPerfInputParams(self): - p = input_generator.MlPerfInput.Params() - input_file = test_helper.test_src_dir_path( - 'tasks/mt/testdata/translate_ende_wmt32k-train-00511-of-00512') - p.file_pattern = 'tfrecord:' + input_file - p.file_random_seed = 31415 - p.file_parallelism = 1 - p.bucket_upper_bound = [20, 40] - p.bucket_batch_limit = [4, 8] - return p - - def _CreateMlPerfPackedInputParams(self): - p = input_generator.MlPerfInput.Params() - input_file = test_helper.test_src_dir_path( - 'tasks/mt/testdata/translate_ende_mlperf.packed.tfrecord') - p.file_pattern = 'tfrecord:' + input_file - p.packed_input = True - p.file_random_seed = 31415 - p.file_parallelism = 1 - p.bucket_upper_bound = [20, 240] - p.bucket_batch_limit = [4, 4] - return p - - def _CreateNmtInputParams(self): - p = input_generator.NmtInput.Params() - input_file = test_helper.test_src_dir_path( - 'tasks/mt/testdata/wmt14_ende_wpm_32k_test.tfrecord') - p.file_pattern = 'tfrecord:' + input_file - p.file_random_seed = 31415 - p.file_parallelism = 1 - p.bucket_upper_bound = [20, 40] - p.bucket_batch_limit = [4, 8] - return p - - def testBasic(self): - p = self._CreateNmtInputParams() - with self.session(use_gpu=False): - inp = input_generator.NmtInput(p) - # Runs a few steps. - for _ in range(10): - self.evaluate(inp.GetPreprocessedInputBatch()) - - def testMlPerfPackedInput(self): - p = self._CreateMlPerfPackedInputParams() - with self.session(use_gpu=False): - inp = input_generator.MlPerfInput(p) - for _ in range(1): - fetched = py_utils.NestedMap( - self.evaluate(inp.GetPreprocessedInputBatch())) - tf.logging.info(fetched.src.ids.shape) - tf.logging.info(fetched.src.segment_ids.shape) - tf.logging.info(fetched.src.segment_pos.shape) - tf.logging.info(fetched.tgt.segment_ids.shape) - tf.logging.info(fetched.tgt.segment_pos.shape) - - def checkPadShape(self, x, pad, batch_size, actual_max, pad_length): - # Check the shape: (batch, maxlen) - self.assertEqual(x.shape, (batch_size, pad_length)) - # Check the padding. - self.assertAllEqual(x[:, actual_max:], - np.full((batch_size, (pad_length - actual_max)), pad)) - - def testMlPerfPackedInputPadToMax(self): - p = self._CreateMlPerfPackedInputParams() - p.source_max_length = 300 - p.target_max_length = 300 - p.pad_to_max_seq_length = True - with self.session(use_gpu=False): - inp = input_generator.MlPerfInput(p) - for _ in range(1): - fetched = py_utils.NestedMap( - self.evaluate(inp.GetPreprocessedInputBatch())) - - self.checkPadShape( - fetched.src.ids, pad=0, batch_size=4, actual_max=240, pad_length=300) - - self.checkPadShape( - fetched.tgt.ids, pad=0, batch_size=4, actual_max=240, pad_length=300) - - self.checkPadShape( - fetched.tgt.segment_ids, - pad=0, - batch_size=4, - actual_max=240, - pad_length=300) - - self.checkPadShape( - fetched.tgt.segment_pos, - pad=0, - batch_size=4, - actual_max=240, - pad_length=300) - - def testMlPerf(self): - p = self._CreateMlPerfInputParams() - with self.session(use_gpu=False): - inp = input_generator.MlPerfInput(p) - # Runs a few steps. - for _ in range(10): - fetched = py_utils.NestedMap( - self.evaluate(inp.GetPreprocessedInputBatch())) - tf.logging.info(fetched) - - def testMlPerfPadToMax(self): - p = self._CreateMlPerfInputParams() - p.bucket_upper_bound = [20] - p.bucket_batch_limit = [4] - p.source_max_length = 30 - p.target_max_length = 30 - p.pad_to_max_seq_length = True - - with self.session(use_gpu=False): - inp = input_generator.MlPerfInput(p) - # Runs a few steps. - for _ in range(10): - fetched = py_utils.NestedMap( - self.evaluate(inp.GetPreprocessedInputBatch())) - - def Check(x, pad): - # Check the shape: (batch, maxlen) - self.assertEqual(x.shape, (4, 30)) - # Check the padding. - self.assertAllEqual(x[:, 20:], np.full((4, 10), pad)) - Check(fetched.src.ids, 0) - Check(fetched.src.paddings, 1) - Check(fetched.tgt.ids, 0) - Check(fetched.tgt.labels, 0) - Check(fetched.tgt.weights, 0) - Check(fetched.tgt.paddings, 1) - - def testPadToMax(self): - p = self._CreateNmtInputParams() - p.bucket_upper_bound = [20] - p.bucket_batch_limit = [4] - p.source_max_length = 30 - p.target_max_length = 30 - p.pad_to_max_seq_length = True - with self.session(use_gpu=False): - inp = input_generator.NmtInput(p) - fetched = py_utils.NestedMap( - self.evaluate(inp.GetPreprocessedInputBatch())) - - def Check(x, pad): - # Check the shape: (batch, maxlen) - self.assertEqual(x.shape, (4, 30)) - # Check the padding. - self.assertAllEqual(x[:, 20:], np.full((4, 10), pad)) - - Check(fetched.src.ids, 0) - Check(fetched.src.paddings, 1) - Check(fetched.tgt.ids, 0) - Check(fetched.tgt.labels, 0) - Check(fetched.tgt.weights, 0) - Check(fetched.tgt.paddings, 1) - - def testSplitSources(self): - p = self._CreateNmtInputParams() - num_splits = 2 - expected_ids_split_1 = [ - [ - 228, 58, 854, 11, 392, 45, 77, 67, 1346, 30, 25, 10, 2283, 933, 14, - 3, 872, 4677, 5, 2 - ], - [ - 328, 22, 463, 571, 134, 10, 3815, 6311, 8, 2203, 3, 654, 2724, 1064, - 5, 2, 0, 0, 0, 0 - ], - ] - - expected_ids_split_2 = [ - [ - 16, 599, 11, 8, 113, 3, 145, 558, 489, 4373, 36, 55, 8988, 5, 2, 0, - 0, 0, 0, 0 - ], - [ - 16, 343, 95, 296, 4550, 4786, 1798, 23019, 8, 10296, 3, 107, 6428, - 1812, 5, 2, 0, 0, 0, 0 - ], - ] - - with self.session(use_gpu=False): - inp = input_generator.NmtInput(p) - splits = inp.SplitInputBatch(num_splits) - split_ids = self.evaluate([splits[0].src.ids, splits[1].src.ids]) - self.assertAllEqual(expected_ids_split_1, split_ids[0]) - self.assertAllEqual(expected_ids_split_2, split_ids[1]) - - def testSplitTargets(self): - p = self._CreateNmtInputParams() - num_splits = 2 - - with self.session(use_gpu=False): - inp = input_generator.NmtInput(p) - fetched = self.evaluate(inp.SplitInputBatch(num_splits)) - - expected_ids_split_1 = [ - [ - 1, 400, 5548, 12, 583, 43, 61, 179, 1265, 22, 27, 7193, 16, 5, 782, - 14077, 6734, 4, 0 - ], - [ - 1, 1639, 32, 1522, 93, 38, 6812, 2624, 9, 2440, 3, 39, 11, 2364, - 24238, 9, 317, 4, 0 - ], - ] - - expected_ids_split_2 = [ - [ - 1, 53, 17787, 12, 3, 5, 1554, 871, 9, 1398, 3, 2784, 18, 25579, 942, - 29828, 5998, 77, 4 - ], - [ - 1, 67, 4141, 11483, 2008, 6, 483, 46, 23, 14852, 3, 39, 5, 9732, - 495, 3176, 21523, 4, 0 - ], - ] - - self.assertAllEqual(expected_ids_split_1, fetched[0].tgt.ids) - self.assertAllEqual(expected_ids_split_2, fetched[1].tgt.ids) - - def testTextPackedInputProto(self): - p = input_generator.TextPackedInput.Params() - p.flush_every_n = 0 - p.require_sequential_order = True - p.repeat_count = 1 - p.file_pattern = 'tfrecord:' + test_helper.test_src_dir_path( - 'tasks/mt/testdata/en_fr.tfrecord') - p.pad_to_max_seq_length = True - p.tokenizer = tokenizers.AsciiTokenizer.Params() - p.input_file_type = 'sentence_proto' - p.source_max_length = 22 - p.target_max_length = 24 - p.bucket_batch_limit = [2] - with self.session() as sess: - inp = p.Instantiate() - batch_tensor = inp.GetPreprocessedInputBatch() - for k, x in batch_tensor.FlattenItems(): - self.assertTrue(x.shape.is_fully_defined(), k) - batch, num_examples = sess.run([batch_tensor, inp.GlobalBatchSize()]) - self.assertEqual(num_examples, 2) - self.assertEqual(len(batch.src), 8) - self.assertAllEqual(batch.src.strs, - [b'I love paragliding!', b'vol biv paragliding']) - self.assertAllEqual(batch.tgt.strs, - [b"J'adore le parapente!", b'vol biv parapente']) - self.assertAllEqual( - batch.src.ids, - np.array([ - [ - 13, 3, 16, 19, 26, 9, 3, 20, 5, 22, 5, 11, 16, 13, 8, 13, 18, - 11, 35, 2, 0, 0 - ], - [ - 26, 19, 16, 3, 6, 13, 26, 3, 20, 5, 22, 5, 11, 16, 13, 8, 13, - 18, 11, 2, 0, 0 - ], - ])) - self.assertAllEqual( - batch.tgt.ids, - np.array([ - [ - 1, 14, 32, 5, 8, 19, 22, 9, 3, 16, 9, 3, 20, 5, 22, 5, 20, 9, - 18, 24, 9, 35, 0, 0 - ], - [ - 1, 26, 19, 16, 3, 6, 13, 26, 3, 20, 5, 22, 5, 20, 9, 18, 24, 9, - 0, 0, 0, 0, 0, 0 - ], - ])) - self.assertAllEqual( - batch.tgt.labels, - np.array([ - [ - 14, 32, 5, 8, 19, 22, 9, 3, 16, 9, 3, 20, 5, 22, 5, 20, 9, 18, - 24, 9, 35, 2, 0, 0 - ], - [ - 26, 19, 16, 3, 6, 13, 26, 3, 20, 5, 22, 5, 20, 9, 18, 24, 9, 2, - 0, 0, 0, 0, 0, 0 - ], - ])) - - def testTextPackedInputNoPerHostInfeed(self): - # We need to move the call to _DataSourceToInputBatch() to in _InputBatch() - # to place it per host. This in turn requires us to stop override - # GlobalBatchSize() but add a metric instead. - # For now, use_per_host_infeed is not supported. - p = input_generator.TextPackedInput.Params() - p.file_pattern = 'text:' + test_helper.test_src_dir_path( - 'tasks/mt/testdata/en_de.text') - p.use_per_host_infeed = True - p.file_random_seed = 0 - self.assertRaisesRegex( - ValueError, - 'This input generator does not support p.use_per_host_infeed', - p.Instantiate) - - def testTextPackedInputTextWpm(self): - p = input_generator.TextPackedInput.Params() - p.flush_every_n = 0 - p.require_sequential_order = True - p.repeat_count = 1 - p.file_pattern = 'text:' + test_helper.test_src_dir_path( - 'tasks/mt/testdata/en_de.text') - p.tokenizer = tokenizers.WpmTokenizer.Params().Set( - vocab_filepath=test_helper.test_src_dir_path( - 'tasks/mt/wpm-ende-2k.voc'), - vocab_size=2000) - p.source_max_length = 12 - p.target_max_length = 15 - p.bucket_batch_limit = [2] - with self.session() as sess: - inp = p.Instantiate() - batch_tensor = inp.GetPreprocessedInputBatch() - batch, num_examples = sess.run([batch_tensor, inp.GlobalBatchSize()]) - self.assertEqual(num_examples, 2) - print(batch) - self.assertAllEqual( - batch.src.ids, - np.array([[109, 251, 98, 595, 1009, 245, 326, 129, 4, 2, 0, 0], - [115, 276, 18, 66, 2, 0, 0, 0, 0, 0, 0, 0]])) - self.assertAllEqual( - batch.tgt.ids, - np.array([[ - 1, 197, 446, 458, 419, 284, 323, 1411, 571, 456, 409, 13, 4, 0, 0 - ], [1, 115, 281, 18, 66, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])) - self.assertAllEqual( - batch.tgt.labels, - np.array([[ - 197, 446, 458, 419, 284, 323, 1411, 571, 456, 409, 13, 4, 2, 0, 0 - ], [115, 281, 18, 66, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])) - - def testTextPackedInputTextPacking(self): - p = input_generator.TextPackedInput.Params() - p.flush_every_n = 0 - p.require_sequential_order = True - p.file_pattern = 'text:' + test_helper.test_src_dir_path( - 'tasks/mt/testdata/en_de.text') - p.tokenizer = tokenizers.WpmTokenizer.Params().Set( - vocab_filepath=test_helper.test_src_dir_path( - 'tasks/mt/wpm-ende-2k.voc'), - vocab_size=2000) - # We repeat the 2-line file twice for a batch of 2, each packing both lines. - p.repeat_count = 2 - p.source_max_length = 16 - p.target_max_length = 20 - p.bucket_batch_limit = [2] - p.packing_factor = 2 - with self.session() as sess: - inp = p.Instantiate() - batch_tensor = inp.GetPreprocessedInputBatch() - batch, num_examples = sess.run([batch_tensor, inp.GlobalBatchSize()]) - self.assertEqual(num_examples, 4) - self.assertAllEqual( - batch.src.ids, - np.array([ - [ - 109, 251, 98, 595, 1009, 245, 326, 129, 4, 2, 115, 276, 18, 66, - 2, 0 - ], - [ - 115, 276, 18, 66, 2, 109, 251, 98, 595, 1009, 245, 326, 129, 4, - 2, 0 - ], - ])) - self.assertAllEqual( - batch.src.segment_ids, - np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 0], - [1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0]], - dtype=np.float32)) - self.assertAllEqual( - batch.src.segment_pos, - np.array([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 0], - [0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0]])) - self.assertAllEqual( - batch.src.strs, - np.array([ - b'Too much has changed.\tHello!', b'Hello!\tToo much has changed.' - ])) - - self.assertAllEqual( - batch.tgt.ids, - np.array([ - [ - 1, 197, 446, 458, 419, 284, 323, 1411, 571, 456, 409, 13, 4, 1, - 115, 281, 18, 66, 0, 0 - ], - [ - 1, 115, 281, 18, 66, 1, 197, 446, 458, 419, 284, 323, 1411, 571, - 456, 409, 13, 4, 0, 0 - ], - ])) - self.assertAllEqual( - batch.tgt.labels, - np.array([ - [ - 197, 446, 458, 419, 284, 323, 1411, 571, 456, 409, 13, 4, 2, - 115, 281, 18, 66, 2, 0, 0 - ], - [ - 115, 281, 18, 66, 2, 197, 446, 458, 419, 284, 323, 1411, 571, - 456, 409, 13, 4, 2, 0, 0 - ], - ])) - self.assertAllEqual( - batch.tgt.segment_ids, - np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 0, 0], - [1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0]], - dtype=np.float32)) - self.assertAllEqual( - batch.tgt.segment_pos, - np.array( - [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 0, 1, 2, 3, 4, 0, 0], - [0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 0, 0]])) - self.assertAllEqual( - batch.tgt.strs, - np.array([ - b'Daf\xc3\xbcr hat sich zu viel ver\xc3\xa4ndert.\tHallo!', - b'Hallo!\tDaf\xc3\xbcr hat sich zu viel ver\xc3\xa4ndert.' - ])) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/mt/layers.py b/lingvo/tasks/mt/layers.py deleted file mode 100644 index 2b25fcb1..00000000 --- a/lingvo/tasks/mt/layers.py +++ /dev/null @@ -1,190 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Lingvo MT layers. -""" - -import lingvo.compat as tf -from lingvo.core import base_layer -from lingvo.core import layers -from lingvo.core import layers_with_attention - - -class TransformerStack(base_layer.BaseLayer): - """Stacked self- multi-head attention and fully connected layers. - - With optional layer normalization applied to the final output. - - See 'Attention Is All You Need' https://arxiv.org/abs/1706.03762 - for details. - """ - - @classmethod - def Params(cls): - """Configs for TransformerStack.""" - p = super().Params() - - # Transformer related - p.Define('model_dim', 1024, 'Characteristic depth (dimension).') - p.Define('num_transformer_layers', 6, 'Number of transformer layers.') - p.Define( - 'transformer_tpl', layers_with_attention.TransformerLayer.Params(), - 'TransformerLayer params tpl. Can be a list of params. ' - 'num_transformer_layers should be divisible by len(transformer_tpl).') - - p.Define('ln_tpl', layers.LayerNorm.Params(), 'Layer norm default params') - p.Define('ln_output', False, - 'If set, layer normalization is applied to the final output' - ' of the encoder transformer stack.') - - p.Define('is_transparent', False, - 'If set, outputs a merger of embeddings and layer outputs.') - p.Define('num_transparent_outputs', 6, 'Number of transparent outputs.') - p.Define( - 'transparent_merger_tpl', - layers.WeightedSumLayer.Params().Set(add_weight_summaries=True), - 'Merger op for layer outputs.') - p.Define('packed_input', False, - 'If True, assumes multiple training samples per input.') - p.Define('has_aux_attention', False, - 'Allows encoder layers to attend auxiliary inputs.') - p.Define('mask_self_atten', False, 'If True, use masked self-attention.') - p.transformer_tpl.tr_atten_tpl.num_attention_heads = 8 - p.transformer_tpl.tr_fflayer_tpl.hidden_dim = 8192 - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - # Add transformer layers. - transformer_layer_params = [] - denom = 1 - if isinstance(p.transformer_tpl, list): - denom = len(p.transformer_tpl) - assert p.num_transformer_layers % len(p.transformer_tpl) == 0 - for i in range(p.num_transformer_layers // denom): - if isinstance(p.transformer_tpl, list): - for q in p.transformer_tpl: - params = q.Copy() - transformer_layer_params.append(params) - else: - params = p.transformer_tpl.Copy() - transformer_layer_params.append(params) - - for i, params in enumerate(transformer_layer_params): - params.name = 'trans_%d' % i - params.source_dim = p.model_dim - params.packed_input = p.packed_input - params.has_aux_atten = p.has_aux_attention - params.mask_self_atten = p.mask_self_atten - - self.CreateChildren('trans', transformer_layer_params) - - # Initialize TransformerStack output layer norm - if p.ln_output: - params = p.ln_tpl.Copy() - # Keeping historic 'enc_out_ln' name for checkpoint compatibility. - params.name = 'enc_out_ln' - params.input_dim = p.model_dim - self.CreateChild('layer_norm_out', params) - - if p.is_transparent: - transparent_params = [] - if not p.num_transparent_outputs: - raise ValueError('num_transparent_outputs should be greater than 0.') - for i in range(p.num_transparent_outputs): - transparent_param = p.transparent_merger_tpl.Copy() - transparent_param.name = 'transparent_%d' % i - transparent_param.num_sources = 1 + len(transformer_layer_params) - transparent_params.append(transparent_param) - self.CreateChildren('transparent_merger', transparent_params) - - def FProp(self, - theta, - transformer_input, - paddings, - src_segment_id=None, - aux_vecs=None, - aux_paddings=None, - aux_segment_id=None): - """Transforms source sequence of Tensors with Transformers layers. - - Args: - theta: A `.NestedMap` object containing weights' values of this - layer and its children layers. - transformer_input: A sequence of input Tensors of [time, batch, dim] - shape. - paddings: A sequence of 0s and 1s indicating input paddings of - [time, batch] shape. - src_segment_id: A sequence of ints indicating segment ids of - [time, batch] shape. - aux_vecs: A sequence of input Tensors of [aux_time, batch, dim] shape, as - context for the cross-attention layer. - aux_paddings: A sequence of 0s and 1s indicating input paddings of - [aux_time, batch] shape. - aux_segment_id: A sequence of ints indicating segment ids of - [aux_time, batch] shape. - - Returns: - (outputs, out_paddings, segment_ids) tuple. `outputs` is of the shape - [time, batch, depth], and `out_paddings` has shape [time, batch]. If - is_transparent is True, can return a list of num_transformer_layers - tensors of shape [time, batch, depth] if `self.do_eval` is False, and a - [time, batch, depth, num_transparent_outputs] tensor if `self.do_eval` is - True. If packed_input is True, also returns segment_id, otherwise returns - None. - """ - p = self.params - if p.packed_input: - assert src_segment_id is not None, ('Need to specify src_segment_id if ' - 'packed input is supported.') - outputs_list = [transformer_input] - with tf.name_scope(p.name): - for i, transformer_l in enumerate(self.trans): - - # For encoder, keys, values and queries are the same - transformer_output, _ = transformer_l.FProp( - theta.trans[i], - transformer_input, - paddings, - aux_vecs=aux_vecs, - aux_paddings=aux_paddings, - source_segment_id=src_segment_id, - aux_segment_id=aux_segment_id) - transformer_input = transformer_output - outputs_list.append(transformer_output) - - if p.ln_output: - transformer_output = self.layer_norm_out.FProp(theta.layer_norm_out, - transformer_output) - - # When is_transparent is set, it outputs a list of tensors during - # training and the stacked tensors otherwise. This dual behavior is meant - # to avoid excessive memory usage during training (which was prohibiting - # training on TPUs), and simplify the beam search interface. - if p.is_transparent: - if p.num_transparent_outputs == 1: - transformer_output = self.transparent_merger[0].FProp( - theta.transparent_merger[0], outputs_list) - else: - transformer_output = [] - for i in range(p.num_transparent_outputs): - merged_outputs = self.transparent_merger[i].FProp( - theta.transparent_merger[i], outputs_list) - transformer_output.append(merged_outputs) - if self.do_eval: - transformer_output = tf.stack(transformer_output, 3) - - return transformer_output, paddings, src_segment_id diff --git a/lingvo/tasks/mt/layers_test.py b/lingvo/tasks/mt/layers_test.py deleted file mode 100644 index dc14a28a..00000000 --- a/lingvo/tasks/mt/layers_test.py +++ /dev/null @@ -1,363 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -"""Tests for mt.layers.""" - -import lingvo.compat as tf -from lingvo.core import layers_with_attention -from lingvo.core import py_utils -from lingvo.core import test_utils -from lingvo.tasks.mt import layers as mt_layers -import numpy as np - - -NUMPY_RANDOM_SEED = 505837249 - - -class LayersTest(test_utils.TestCase): - - def _TransformerParams(self, layer=mt_layers.TransformerStack): - model_dim = 2 - params = layer.Params() - params.name = 'transformer' - params.model_dim = model_dim - params.num_transformer_layers = 1 - # Note: hidden_dim % num_attention_heads == 0 - params.transformer_tpl.tr_atten_tpl.num_attention_heads = 1 - params.transformer_tpl.tr_fflayer_tpl.hidden_dim = model_dim - params.random_seed = 0 - return params - - def _ContextualTransformerParams(self, - layer=mt_layers.TransformerStack): - params = self._TransformerParams(layer) - params.has_aux_attention = True - return params - - def _MaskedTransformerParams(self, - layer=mt_layers.TransformerStack): - params = self._TransformerParams(layer) - params.mask_self_atten = True - return params - - def _TransformerStackFProp(self, dtype, fprop_dtype, layer): - # time = 2, - batch = 3 - tf.flags.FLAGS.tpu_compatible = True - with self.session(use_gpu=False): - params = self._TransformerParams(layer=layer) - params.dtype = dtype - params.fprop_dtype = fprop_dtype - xformer = layer(params) - - input_arr = np.array( - [ - [[0, 1]] * batch, - [[1, -1]] * batch, - ], dtype=int) - paddings_arr = np.array([[0] * batch, [0] * batch], dtype=int) - inputs = tf.constant(input_arr.tolist(), dtype=fprop_dtype) - paddings = tf.constant(paddings_arr.tolist(), dtype=fprop_dtype) - - output, _, _ = xformer.FProp(xformer.theta, inputs, paddings) - - self.evaluate(tf.global_variables_initializer()) - output = self.evaluate(output) - - self.assertAllCloseAccordingToType( - [[[2.761079, -3.756719]] * batch, [[-2.623049, 3.3679538]] * batch], - output) - - def testTransformerStackFPropFp32Fp32(self): - self._TransformerStackFProp(tf.float32, tf.float32, - mt_layers.TransformerStack) - - def testTransformerStackAlternateLayers(self): - batch = 3 - tf.flags.FLAGS.tpu_compatible = True - with self.session(use_gpu=False): - model_dim = 2 - num_transformer_layers = 2 - transformer_tpl = layers_with_attention.TransformerLayer.Params() - transformer_tpl.tr_atten_tpl.num_attention_heads = 1 - transformer_tpl.tr_fflayer_tpl.hidden_dim = 2 - - params = mt_layers.TransformerStack.Params().Set( - name='transformer', - model_dim=model_dim, - num_transformer_layers=num_transformer_layers, - transformer_tpl=[ - transformer_tpl.Copy() for _ in range(num_transformer_layers) - ], - random_seed=123456) - - xformer = mt_layers.TransformerStack(params) - input_arr = np.array([ - [[0, 1]] * batch, - [[1, -1]] * batch, - ], dtype=int) - paddings_arr = np.array([[0] * batch, [0] * batch], dtype=int) - inputs = tf.constant( - input_arr.tolist(), dtype=py_utils.FPropDtype(params)) - paddings = tf.constant( - paddings_arr.tolist(), dtype=py_utils.FPropDtype(params)) - output, _, _ = xformer.FProp(xformer.theta, inputs, paddings) - - self.evaluate(tf.global_variables_initializer()) - output = self.evaluate(output) - print(repr(output)) - self.assertAllCloseAccordingToType( - np.array([[[-0.940543, 1.479253]] * batch, - [[-0.413938, -2.550903]] * batch]), output) - - def testTransformerStackFPropWithPackedInputs(self): - # batch = 2. time = 2, depth = 2 - with self.session(use_gpu=True): - with tf.variable_scope('packing_test', reuse=tf.AUTO_REUSE): - params = self._TransformerParams() - xformer = mt_layers.TransformerStack(params) - packed_params = params.Copy() - packed_params.packed_input = True - xformer_packed = mt_layers.TransformerStack(packed_params) - - input_arr = np.array([[[0, 1], [1, -1]], [[1, 2], [-2, -1]]], dtype=int) - paddings_arr = np.array([[0, 0], [0, 0]], dtype=int) - seg_id_arr = np.array([[0, 1, 0, 1]], dtype=int) - - inputs = tf.constant(input_arr.tolist(), dtype=tf.float32) - paddings = tf.constant(paddings_arr.tolist(), dtype=tf.float32) - inputs_packed = tf.reshape(inputs, [-1, 1, 2]) - paddings_packed = tf.reshape(paddings, [-1, 1]) - seg_id = tf.transpose( - tf.constant(seg_id_arr.tolist(), dtype=tf.float32)) - - output, _, _ = xformer.FProp(xformer.theta, inputs, paddings, seg_id) - - output_packed, _, _ = xformer_packed.FProp( - xformer_packed.theta, inputs_packed, paddings_packed, seg_id) - output_packed = tf.reshape(output_packed, tf.shape(output)) - - self.evaluate(tf.global_variables_initializer()) - output, output_packed = self.evaluate([output, output_packed]) - - self.assertAllClose(output_packed, output) - - def testTransparentTransformerStackTrainFProp(self): - # time = 2, batch = 1 - with self.session(use_gpu=True): - params = self._TransformerParams() - params.is_transparent = True - params.num_transparent_outputs = 2 - - xformer = mt_layers.TransformerStack(params) - - input_arr = np.array([[[0, 1]], [[1, -1]]], dtype=int) - paddings_arr = np.array([[0], [0]], dtype=int) - - inputs = tf.constant(input_arr.tolist(), dtype=tf.float32) - paddings = tf.constant(paddings_arr.tolist(), dtype=tf.float32) - - self.evaluate(tf.global_variables_initializer()) - outputs, _, _ = xformer.FPropDefaultTheta(inputs, paddings) - out_1, out_2 = self.evaluate(outputs) - self.assertAllClose([[[1.38054, -1.37836]], [[-0.811525, 1.183977]]], - out_1) - self.assertAllClose([[[1.38054, -1.37836]], [[-0.811525, 1.183977]]], - out_2) - - def testTransparentTransformerStackEvalFProp(self): - # time = 2, batch = 1 - with self.session(use_gpu=True), self.SetEval(True): - params = self._TransformerParams() - params.is_transparent = True - params.num_transparent_outputs = 2 - - xformer = mt_layers.TransformerStack(params) - - input_arr = np.array([[[0, 1]], [[1, -1]]], dtype=int) - paddings_arr = np.array([[0], [0]], dtype=int) - - inputs = tf.constant(input_arr.tolist(), dtype=tf.float32) - paddings = tf.constant(paddings_arr.tolist(), dtype=tf.float32) - - self.evaluate(tf.global_variables_initializer()) - outputs, _, _ = xformer.FPropDefaultTheta(inputs, paddings) - out = self.evaluate(outputs) - self.assertAllClose([[[1.38054, -1.37836]], [[-0.811525, 1.183977]]], - out[:, :, :, 0]) - self.assertAllClose([[[1.38054, -1.37836]], [[-0.811525, 1.183977]]], - out[:, :, :, 1]) - - def _TransformerSingleSourceInputs(self, depth=3, dtype=tf.float32): - np.random.seed(NUMPY_RANDOM_SEED) - source_vecs = tf.stack( - [tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(5)]) - source_padding = tf.transpose( - tf.constant([[0, 0, 1, 1, 0], [1, 0, 0, 0, 1]], dtype=dtype)) - aux_source_vecs = tf.stack( - [tf.constant(np.random.rand(2, depth), dtype=dtype) for _ in range(7)]) - aux_source_paddings = tf.transpose( - tf.constant( - [[0, 1, 0, 1, 0, 1, 0], [1, 0, 1, 0, 1, 0, 1]], dtype=dtype)) - return source_vecs, source_padding, aux_source_vecs, aux_source_paddings - - def _TransformerMultiSourceInputs(self, depth=3, dtype=tf.float32): - np.random.seed(NUMPY_RANDOM_SEED) - src_names = ['en1', 'en2', 'de'] - slens = [11, 10, 9] - sbatch = 3 - tlen = 5 - source_vecs = tf.constant( - np.random.uniform(size=(tlen, sbatch*2, depth)), dtype) - source_padding = tf.constant(np.zeros([tlen, sbatch*2, 1]), dtype) - aux_source_vecs = py_utils.NestedMap() - aux_source_paddings = py_utils.NestedMap() - for slen, sname in zip(slens, src_names): - aux_source_vecs[sname] = tf.constant( - np.random.uniform(size=[slen, sbatch, depth]), dtype) - aux_source_paddings[sname] = tf.constant(np.zeros([slen, sbatch]), dtype) - return (source_vecs, source_padding, aux_source_vecs, aux_source_paddings) - - def _ExpectedSingleSourceResults(self): - # pylint: disable=bad-whitespace - # pyformat: disable - expected_ctx = [ - [[-1.2441386 , 0.79554689, 0.92997617, -0.11617106], - [-0.35435563, -1.3054322 , 1.22436929, 0.78819978]], - [[ 0.25911736, 1.59861779, -0.79328805, -0.69870573], - [-0.07478544, -0.97613871, -0.28578228, 1.68864977]], - [[ 1.70767629, 0.47948313, -1.04493523, -0.77614671], - [-0.7304405 , -0.97698098, 1.06615043, 0.99392742]], - [[-0.8379035 , -0.4325709 , 1.64835262, -0.01260686], - [ 1.9789927 , -0.0406196 , -0.70901859, -0.87701303]], - [[-0.31843656, -0.88714617, -0.10433094, 1.6746937 ], - [-0.75752175, 0.063187 , -0.50349152, 1.54967213]]] - expected_probs = [ - [[ 0.2506268 , 0., 0.2491456 , 0., 0.24775073, 0., 0.25247693], - [ 0., 0.33213311, 0., 0.3369872 , 0., 0.33087969, 0. ]], - [[ 0.2478824 , 0., 0.24961777, 0., 0.24944726, 0., 0.25305259], - [ 0., 0.32697931, 0., 0.33510387, 0., 0.33791685, 0. ]], - [[ 0.24868786, 0., 0.2501137 , 0., 0.25135297, 0., 0.24984553], - [ 0., 0.33173615, 0. , 0.3365739, 0., 0.33168989, 0. ]], - [[ 0.25286067, 0., 0.24919504, 0., 0.24831679, 0., 0.24962756], - [ 0., 0.33384994, 0., 0.33144677, 0., 0.33470333, 0. ]], - [[ 0.24830991, 0., 0.25117707, 0., 0.25028574, 0., 0.25022724], - [ 0., 0.32832602, 0., 0.33358034, 0., 0.33809367, 0. ]]] - # pyformat: enable - # pylint: enable=bad-whitespace - return expected_ctx, expected_probs - - def testTransformerAttentionLayerReference(self): - depth = 4 - p = layers_with_attention.TransformerAttentionLayer.Params() - p.name = 'transformer_atten' - p.source_dim = depth - p.is_masked = False - p.num_attention_heads = 2 - p.atten_tpl.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - transformer_atten_ref = layers_with_attention.TransformerAttentionLayer(p) - - (query_vec, _, aux_vecs, - aux_paddings) = self._TransformerSingleSourceInputs(depth) - - ctx_ref, probs_ref = transformer_atten_ref.FPropDefaultTheta( - query_vec, aux_paddings, aux_vecs) - - expected_ctx, expected_probs = self._ExpectedSingleSourceResults() - with self.session(use_gpu=True): - self.evaluate(tf.global_variables_initializer()) - actual_ctx_ref, actual_probs_ref = self.evaluate([ctx_ref, probs_ref]) - tf.logging.info(np.array_repr(actual_ctx_ref)) - tf.logging.info(np.array_repr(actual_probs_ref)) - self.assertAllClose(expected_ctx, actual_ctx_ref) - self.assertAllClose(expected_probs, actual_probs_ref) - - def testContextualTransformerStackFProp(self): - # time = 2, - batch = 3 - dtype = tf.float32 - fprop_dtype = tf.float32 - layer = mt_layers.TransformerStack - tf.flags.FLAGS.tpu_compatible = True - with self.session(use_gpu=False): - params = self._ContextualTransformerParams(layer=layer) - params.dtype = dtype - params.fprop_dtype = fprop_dtype - xformer = layer(params) - - input_arr = np.array([ - [[0, 1]] * batch, - [[1, -1]] * batch, - ], dtype=int) - - context_arr = np.array([ - [[0, 1]] * batch, - [[1, -1]] * batch, - [[-1, 1]] * batch, - ], - dtype=int) - paddings_arr = np.array([[0] * batch, [0] * batch], dtype=int) - context_paddings_arr = np.array([[0] * batch, [0] * batch, [0] * batch], - dtype=int) - inputs = tf.constant(input_arr.tolist(), dtype=fprop_dtype) - paddings = tf.constant(paddings_arr.tolist(), dtype=fprop_dtype) - context = tf.constant(context_arr.tolist(), dtype=fprop_dtype) - context_paddings = tf.constant( - context_paddings_arr.tolist(), dtype=fprop_dtype) - - output, _, _ = xformer.FProp( - xformer.theta, - inputs, - paddings, - aux_vecs=context, - aux_paddings=context_paddings) - - self.evaluate(tf.global_variables_initializer()) - output = self.evaluate(output) - self.assertAllCloseAccordingToType( - [[[-2.622666, 3.367474]] * batch, [[2.719156, -3.707336]] * batch], - output) - - def testMaskedTransformerStackFProp(self): - batch = 3 - dtype = tf.float32 - fprop_dtype = tf.float32 - layer = mt_layers.TransformerStack - tf.flags.FLAGS.tpu_compatible = True - with self.session(use_gpu=False): - params = self._MaskedTransformerParams(layer=layer) - params.dtype = dtype - params.fprop_dtype = fprop_dtype - xformer = layer(params) - - input_arr = np.array([ - [[0, 1]] * batch, - [[1, -1]] * batch, - ], dtype=int) - - paddings_arr = np.array([[0] * batch, [0] * batch], dtype=int) - - inputs = tf.constant(input_arr.tolist(), dtype=fprop_dtype) - paddings = tf.constant(paddings_arr.tolist(), dtype=fprop_dtype) - - output, _, _ = xformer.FProp(xformer.theta, inputs, paddings) - - self.evaluate(tf.global_variables_initializer()) - output = self.evaluate(output) - self.assertAllClose( - [[[2.761103, -3.756748]] * batch, [[-2.623049, 3.367954]] * batch], - output) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/mt/model.py b/lingvo/tasks/mt/model.py deleted file mode 100644 index 45fb1237..00000000 --- a/lingvo/tasks/mt/model.py +++ /dev/null @@ -1,386 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""MT models.""" - -import lingvo.compat as tf -from lingvo.core import base_model -from lingvo.core import insertion -from lingvo.core import metrics -from lingvo.core import py_utils -from lingvo.tasks.mt import decoder -from lingvo.tasks.mt import encoder - - -class MTBaseModel(base_model.BaseTask): - """Base Class for NMT models.""" - - def _EncoderDevice(self): - """Returns the device to run the encoder computation.""" - if py_utils.use_tpu(): - return tf.device(self.cluster.WorkerDeviceInModelSplit(0)) - else: - return tf.device('') - - def _DecoderDevice(self): - """Returns the device to run the decoder computation.""" - if py_utils.use_tpu(): - return tf.device(self.cluster.WorkerDeviceInModelSplit(1)) - else: - return tf.device('') - - def __init__(self, params): - super().__init__(params) - p = self.params - if p.encoder: - self.CreateChild('enc', p.encoder) - self.CreateChild('dec', p.decoder) - - def _CreateChildrenVariables(self): - p = self.params - with tf.variable_scope(p.name): - with self._EncoderDevice(): - if p.encoder: - self.enc.InstantiateVariables() - with self._DecoderDevice(): - self.dec.InstantiateVariables() - super()._CreateChildrenVariables() - - def ComputePredictions(self, theta, batch): - p = self.params - - with self._EncoderDevice(): - encoder_outputs = ( - self.enc.FProp(theta.enc, batch.src) if p.encoder else None) - with self._DecoderDevice(): - predictions = self.dec.ComputePredictions(theta.dec, encoder_outputs, - batch.tgt) - if isinstance(predictions, py_utils.NestedMap): - # Pass through encoder output as well for possible use as a FProp output - # for various meta-MT modeling approaches, such as MT quality estimation - # classification. - predictions['encoder_outputs'] = encoder_outputs - return predictions - - def ComputeLoss(self, theta, predictions, input_batch): - with self._DecoderDevice(): - return self.dec.ComputeLoss(theta.dec, predictions, input_batch.tgt) - - def _GetTokenizerKeyToUse(self, key): - """Returns a tokenizer key to use for the provided `key`.""" - if key in self.input_generator.tokenizer_dict: - return key - return None - - def _BeamSearchDecode(self, input_batch): - p = self.params - with tf.name_scope('fprop'), tf.name_scope(p.name): - encoder_outputs = self.enc.FPropDefaultTheta(input_batch.src) - encoder_outputs = self.dec.AddExtraDecodingInfo(encoder_outputs, - input_batch.tgt) - decoder_outs = self.dec.BeamSearchDecode(encoder_outputs) - - topk_hyps = decoder_outs.topk_hyps - topk_ids = decoder_outs.topk_ids - topk_lens = decoder_outs.topk_lens - topk_scores = decoder_outs.topk_scores - - slen = tf.cast( - tf.round(tf.reduce_sum(1 - input_batch.src.paddings, 1) - 1), - tf.int32) - srcs = self.input_generator.IdsToStrings( - input_batch.src.ids, slen, self._GetTokenizerKeyToUse('src')) - topk_decoded = self.input_generator.IdsToStrings( - topk_ids, topk_lens - 1, self._GetTokenizerKeyToUse('tgt')) - topk_decoded = tf.reshape(topk_decoded, tf.shape(topk_hyps)) - topk_scores = tf.reshape(topk_scores, tf.shape(topk_hyps)) - - refs = self.input_generator.IdsToStrings( - input_batch.tgt.labels, - tf.cast( - tf.round(tf.reduce_sum(1.0 - input_batch.tgt.paddings, 1) - 1.0), - tf.int32), self._GetTokenizerKeyToUse('tgt')) - - ret_dict = { - 'target_ids': input_batch.tgt.ids, - 'target_labels': input_batch.tgt.labels, - 'target_weights': input_batch.tgt.weights, - 'target_paddings': input_batch.tgt.paddings, - 'sources': srcs, - 'targets': refs, - 'topk_decoded': topk_decoded, - 'topk_lens': topk_lens, - 'topk_scores': topk_scores, - } - return ret_dict - - def _PostProcessBeamSearchDecodeOut(self, dec_out_dict, dec_metrics_dict): - """Post processes the output from `_BeamSearchDecode`.""" - p = self.params - topk_scores = dec_out_dict['topk_scores'] - topk_decoded = dec_out_dict['topk_decoded'] - targets = dec_out_dict['targets'] - sources = dec_out_dict['sources'] - unsegment = dec_metrics_dict['corpus_bleu'].unsegmenter - - num_samples = len(targets) - assert num_samples == len(topk_decoded), ( - '%s vs %s' % (num_samples, len(topk_decoded))) - assert num_samples == len(sources) - dec_metrics_dict['num_samples_in_batch'].Update(num_samples) - - key_value_pairs = [] - for i in range(num_samples): - src, tgt = sources[i], targets[i] - src_unseg, tgt_unseg = unsegment(src), unsegment(tgt) - tf.logging.info('source: %s', src_unseg) - tf.logging.info('target: %s', tgt_unseg) - hyps = topk_decoded[i] - assert p.decoder.beam_search.num_hyps_per_beam == len(hyps) - info_str = u'src: {} tgt: {} '.format(src_unseg, tgt_unseg) - for n, (score, hyp_str) in enumerate(zip(topk_scores[i], hyps)): - hyp_str_unseg = unsegment(hyp_str) - tf.logging.info(' %f: %s', score, hyp_str_unseg) - info_str += u' hyp{n}: {hyp} score{n}: {score}'.format( - n=n, hyp=hyp_str_unseg, score=score) - # Only aggregate scores of the top hypothesis. - if n == 0: - dec_metrics_dict['corpus_bleu'].Update(tgt, hyp_str) - key_value_pairs.append((src_unseg, info_str)) - return key_value_pairs - - def CreateDecoderMetrics(self): - decoder_metrics = { - 'num_samples_in_batch': metrics.AverageMetric(), - 'corpus_bleu': metrics.CorpusBleuMetric(separator_type='wpm'), - } - return decoder_metrics - - def Decode(self, input_batch): - """Constructs the decoding graph.""" - return self._BeamSearchDecode(input_batch) - - def PostProcessDecodeOut(self, dec_out, dec_metrics): - return self._PostProcessBeamSearchDecodeOut(dec_out, dec_metrics) - - -class TransformerModel(MTBaseModel): - """Transformer Model. - - Implements Attention is All You Need: - https://arxiv.org/abs/1706.03762 - """ - - @classmethod - def Params(cls): - p = super().Params() - p.encoder = encoder.TransformerEncoder.Params() - p.decoder = decoder.TransformerDecoder.Params() - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - assert p.encoder.model_dim == p.decoder.source_dim - - -class RNMTModel(MTBaseModel): - """RNMT+ Model. - - Implements RNMT Variants in The Best of Both Worlds paper: - https://aclweb.org/anthology/P18-1008 - """ - - @classmethod - def Params(cls): - p = super().Params() - p.encoder = encoder.MTEncoderBiRNN.Params() - p.decoder = decoder.MTDecoderV1.Params() - return p - - -class InsertionModel(MTBaseModel): - """Insertion-based model. - - References: - KERMIT: https://arxiv.org/pdf/1906.01604.pdf - Insertion Transformer: https://arxiv.org/pdf/1902.03249.pdf - """ - - @classmethod - def Params(cls): - p = super().Params() - p.decoder = decoder.InsertionDecoder.Params() - p.Define('insertion', insertion.SymbolInsertionLayer.Params(), - 'Insertion specifications (i.e., rollin and oracle policy).') - return p - - def __init__(self, params): - super().__init__(params) - p = self.params - - self.CreateChild('insertion', p.insertion) - - def _SampleCanvasAndTargets(self, x, x_paddings): - """Sample a canvas and its corresponding targets. - - Args: - x: A Tensor representing the canvas. - x_paddings: A Tensor representing the canvas paddings. - - Returns: - A `NestedMap` capturing the new sampled canvas and its targets. - """ - p = self.params - - # TODO(williamchan): Consider grabbing `eos_id` from `x` instead of `p`. - eos_id = p.decoder.target_eos_id - - # Sample a canvas (and it's corresponding targets). - return self.insertion.FProp(None, x, x_paddings, eos_id, True) - - def _CreateCanvasAndTargets(self, batch): - # pyformat: disable - """Create the canvas and targets. - - Args: - batch: A `.NestedMap`. - - - src: A `.NestedMap`. - - ids: The source ids, ends in . - - paddings: The source paddings. - - - tgt: A `.NestedMap`. - - ids: The target ids, ends in . - - paddings: The target paddings. - - Returns: - A `NestedMap`. - - canvas: The canvas (based off of the `rollin_policy`) of shape - [batch_size, c_dim]. - - canvas_paddings: The paddings of `canvas_indices`. - - target_indices: The target indices (i.e., use these indices to - tf.gather_nd the log-probs). Optional, only during training. - - target_weights: The target weights. Optional, only during training. - """ - # pyformat: enable - p = self.params - - if not self.do_eval: - # Sample our src and tgt canvas. - src_descriptor = self._SampleCanvasAndTargets(batch.src.ids, - batch.src.paddings) - tgt_descriptor = self._SampleCanvasAndTargets(batch.tgt.ids, - batch.tgt.paddings) - - # Offset the src ids (to unshare embeddings between src/tgt). Note, we - # only offset the canvas ids, but we do not offset the vocab ids. This - # will result in unshared embeddings, but shared softmax. This is due to - # GPU/TPU memory limitations, empirically it is known that unsharing - # everything results in better performance. - vocab_size = p.decoder.softmax.num_classes - src_descriptor.canvas = tf.where( - tf.equal(src_descriptor.canvas_paddings, 0), - src_descriptor.canvas + vocab_size, src_descriptor.canvas) - - # Offset the tgt indices (need shift according to src length). - batch_size = py_utils.GetShape(batch.src.ids)[0] - # `target_batch` is a [num_targets, batch_size] tensor where each row - # identifies which batch the target belongs to. Note the observation that, - # tf.reduce_sum(target_batch, 1) == 1 \forall rows. - target_batch = tf.cast( - tf.equal( - tf.expand_dims(tf.range(batch_size), 0), - tf.expand_dims(tgt_descriptor.target_indices[:, 0], 1)), tf.int32) - src_lens = tf.cast( - tf.reduce_sum(1 - src_descriptor.canvas_paddings, 1), tf.int32) - # `tgt_offset` is shape [num_targets] where each entry corresponds to the - # offset needed for that target (due to the source length). - tgt_offset = tf.matmul(target_batch, tf.expand_dims(src_lens, 1)) - # We shift the tgt slot without touching the batch or vocab. - tgt_descriptor.target_indices += tf.concat( - [tf.zeros_like(tgt_offset), tgt_offset, - tf.zeros_like(tgt_offset)], 1) - - # The canvas is simply the sequence-level concat of the src and tgt. - canvas, canvas_paddings = insertion.SequenceConcat( - src_descriptor.canvas, src_descriptor.canvas_paddings, - tgt_descriptor.canvas, tgt_descriptor.canvas_paddings) - target_indices = tf.concat( - [src_descriptor.target_indices, tgt_descriptor.target_indices], 0) - target_weights = tf.concat( - [src_descriptor.target_weights, tgt_descriptor.target_weights], 0) - - return py_utils.NestedMap( - canvas=canvas, - canvas_paddings=canvas_paddings, - target_indices=target_indices, - target_weights=target_weights) - - def ComputePredictions(self, theta, batch): - # pyformat: disable - """Compute the model predictions. - - Args: - theta: A `.NestedMap` object containing weights' values of this layer and - its children layers. - batch: A `.NestedMap`. - - - src: A `.NestedMap`. - - ids: The source ids, ends in . - - paddings: The source paddings. - - - tgt: A `.NestedMap`. - - ids: The target ids, ends in . - - paddings: The target paddings. - - Returns: - A `.NestedMap`. - - outputs: The contextualized output vectors of shape - [batch_size, time_dim, model_dim]. - - tgt: A `.NestedMap` (optional, only during training). - - ids: The canvas ids. - - paddings: The canvas paddings. - - target_indices: The target indices. - - target_weights: The target weights. - """ - # pyformat: enable - p = self.params - - # TODO(williamchan): Currently, we only support KERMIT mode (i.e., no - # encoder, unified architecture). - assert not p.encoder - - # Sometimes src and tgt have different types. We reconcile here and use - # int32. - batch.src.ids = tf.cast(batch.src.ids, tf.int32) - batch.tgt.ids = tf.cast(batch.tgt.ids, tf.int32) - - canvas_and_targets = self._CreateCanvasAndTargets(batch) - batch = py_utils.NestedMap( - tgt=py_utils.NestedMap( - ids=canvas_and_targets.canvas, - paddings=canvas_and_targets.canvas_paddings)) - - predictions = super().ComputePredictions(theta, batch) - - if not self.do_eval: - predictions.tgt = py_utils.NestedMap( - ids=canvas_and_targets.canvas, - paddings=canvas_and_targets.canvas_paddings, - target_indices=canvas_and_targets.target_indices, - target_weights=canvas_and_targets.target_weights) - - return predictions diff --git a/lingvo/tasks/mt/model_test.py b/lingvo/tasks/mt/model_test.py deleted file mode 100644 index e19146c4..00000000 --- a/lingvo/tasks/mt/model_test.py +++ /dev/null @@ -1,704 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for MT Models.""" - -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import base_layer -from lingvo.core import cluster_factory -from lingvo.core import optimizer -from lingvo.core import py_utils -from lingvo.core import schedule -from lingvo.core import test_helper -from lingvo.core import test_utils -from lingvo.tasks.mt import decoder -from lingvo.tasks.mt import encoder -from lingvo.tasks.mt import input_generator -from lingvo.tasks.mt import model -import numpy as np - - -FLAGS = tf.flags.FLAGS - -_TF_RANDOM_SEED = 93820986 - - -class TestInputGenerator(base_input_generator.BaseSequenceInputGenerator): - - @classmethod - def Params(cls): - p = super().Params() - p.Define('split', True, '') - return p - - def __init__(self, params): - super().__init__(params) - self._step = 0 - - def InfeedBatchSize(self): - if self.params.split: - return 10 / 2 - - return 10 - - def _InputBatch(self): - np.random.seed(1) - bs, sl = 10, 7 - src_ids = tf.constant( - np.random.randint(low=0, high=8192 - 1, size=[bs, sl], dtype=np.int32)) - tgt_ids = tf.constant( - np.random.randint(low=0, high=8192 - 1, size=[bs, sl], dtype=np.int32)) - tgt_labels = tf.constant( - np.random.randint(low=0, high=8192 - 1, size=[bs, sl], dtype=np.int32)) - tgt_weights = tf.constant(np.ones(shape=[bs, sl], dtype=np.float32)) - - src_paddings = tf.zeros([bs, sl]) - tgt_paddings = tf.zeros([bs, sl]) - - ret = py_utils.NestedMap() - ret.src = py_utils.NestedMap() - ret.tgt = py_utils.NestedMap() - - if self.params.split: - src_ids = tf.split(src_ids, 2, 0) - src_paddings = tf.split(src_paddings, 2, 0) - tgt_ids = tf.split(tgt_ids, 2, 0) - tgt_labels = tf.split(tgt_labels, 2, 0) - tgt_paddings = tf.split(tgt_paddings, 2, 0) - tgt_weights = tf.split(tgt_weights, 2, 0) - - ret.src.ids = tf.cond( - tf.equal(tf.math.floormod(py_utils.GetGlobalStep(), 2), 0), - lambda: src_ids[0], lambda: src_ids[1]) - ret.src.paddings = tf.cond( - tf.equal(tf.math.floormod(py_utils.GetGlobalStep(), 2), 0), - lambda: src_paddings[0], lambda: src_paddings[1]) - ret.tgt.ids = tf.cond( - tf.equal(tf.math.floormod(py_utils.GetGlobalStep(), 2), 0), - lambda: tgt_ids[0], lambda: tgt_ids[1]) - ret.tgt.labels = tf.cond( - tf.equal(tf.math.floormod(py_utils.GetGlobalStep(), 2), 0), - lambda: tgt_labels[0], lambda: tgt_labels[1]) - ret.tgt.paddings = tf.cond( - tf.equal(tf.math.floormod(py_utils.GetGlobalStep(), 2), 0), - lambda: tgt_paddings[0], lambda: tgt_paddings[1]) - ret.tgt.weights = tf.cond( - tf.equal(tf.math.floormod(py_utils.GetGlobalStep(), 2), 0), - lambda: tgt_weights[0], lambda: tgt_weights[1]) - else: - ret.src.ids = src_ids - ret.src.paddings = src_paddings - ret.tgt.ids = tgt_ids - ret.tgt.labels = tgt_labels - ret.tgt.paddings = tgt_paddings - ret.tgt.weights = tgt_weights - - return ret - - -class TransformerModelTest(test_utils.TestCase): - - def _InputParams(self): - p = input_generator.NmtInput.Params() - input_file = test_helper.test_src_dir_path( - 'tasks/mt/testdata/wmt14_ende_wpm_32k_test.tfrecord') - vocab_file = test_helper.test_src_dir_path( - 'tasks/mt/testdata/wmt14_ende_wpm_32k_test.vocab') - p.file_pattern = 'tfrecord:' + input_file - p.file_random_seed = 31415 - p.file_parallelism = 1 - p.bucket_upper_bound = [40] - p.bucket_batch_limit = [8] - p.source_max_length = 200 - p.target_max_length = 200 - - p.tokenizer.token_vocab_filepath = vocab_file - p.tokenizer.vocab_size = 32000 - return p - - def _EncoderParams(self): - p = encoder.TransformerEncoder.Params() - p.name = 'encoder' - p.random_seed = 1234 - p.model_dim = 4 - p.token_emb.embedding_dim = 4 - p.token_emb.max_num_shards = 1 - p.token_emb.params_init = py_utils.WeightInit.GaussianSqrtDim( - seed=p.random_seed) - p.position_emb.embedding_dim = 4 - p.transformer_stack.transformer_tpl.tr_atten_tpl.num_attention_heads = 2 - p.transformer_stack.transformer_tpl.tr_fflayer_tpl.hidden_dim = 5 - return p - - def _DecoderParams(self): - p = decoder.TransformerDecoder.Params() - p.name = 'decoder' - p.random_seed = 1234 - p.source_dim = 4 - p.model_dim = 4 - p.token_emb.embedding_dim = 4 - p.token_emb.max_num_shards = 1 - p.token_emb.params_init = py_utils.WeightInit.GaussianSqrtDim( - seed=p.random_seed) - p.position_emb.embedding_dim = 4 - p.trans_tpl.source_dim = 4 - p.trans_tpl.tr_atten_tpl.source_dim = 4 - p.trans_tpl.tr_atten_tpl.num_attention_heads = 2 - p.trans_tpl.tr_fflayer_tpl.input_dim = 4 - p.trans_tpl.tr_fflayer_tpl.hidden_dim = 8 - p.softmax.num_shards = 1 - p.target_seq_len = 5 - return p - - def _testParams(self): - p = model.TransformerModel.Params() - p.name = 'test_mdl' - p.input = self._InputParams() - p.encoder = self._EncoderParams() - p.decoder = self._DecoderParams() - p.train.learning_rate = 2e-4 - return p - - def testConstruction(self): - with self.session(): - p = self._testParams() - mdl = p.Instantiate() - print('vars = ', mdl.vars) - flatten_vars = mdl.vars.Flatten() - print('vars flattened = ', flatten_vars) - self.assertEqual(len(flatten_vars), 238) - - # Should match tf.trainable_variables(). - self.assertEqual(len(tf.trainable_variables()), len(flatten_vars)) - - def testFProp(self, dtype=tf.float32, fprop_dtype=tf.float32): - with self.session(): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._testParams() - p.dtype = dtype - if fprop_dtype: - p.fprop_dtype = fprop_dtype - p.input.dtype = fprop_dtype - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - loss = mdl.loss - logp = mdl.eval_metrics['log_pplx'][0] - self.evaluate(tf.global_variables_initializer()) - vals = [] - for _ in range(5): - vals += [self.evaluate((loss, logp))] - - print('actual vals = %s' % np.array_repr(np.array(vals))) - self.assertAllClose(vals, [[233.57518, 10.381119], [236.10052, 10.378047], - [217.99896, 10.380901], [217.94647, 10.378406], - [159.5997, 10.380468]]) - - def testFPropEvalMode(self): - with self.session(), self.SetEval(True): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._testParams() - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - loss = mdl.loss - logp = mdl.eval_metrics['log_pplx'][0] - self.evaluate(tf.global_variables_initializer()) - vals = [] - for _ in range(5): - vals += [self.evaluate((loss, logp))] - print('actual vals = ', vals) - self.assertAllClose(vals, [ - [233.57518, 10.381119], - [236.10052, 10.378047], - [217.99896, 10.380901], - [217.94647, 10.378406], - [159.5997, 10.380468], - ]) - - def testBProp(self): - with self.session(): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._testParams() - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - mdl.BProp() - loss = mdl.loss - logp = mdl.eval_metrics['log_pplx'][0] - - self.evaluate(tf.global_variables_initializer()) - vals = [] - for _ in range(5): - vals += [self.evaluate((loss, logp, mdl.train_op))[:2]] - print('BProp actual vals = ', vals) - expected_vals = [ - [233.57518, 10.381119], - [236.05138, 10.375884], - [217.9087, 10.376605], - [217.77725, 10.370345], - [159.43497, 10.369753], - ] - self.assertAllClose(vals, expected_vals) - - def testBPropWithAccumComparison(self): - - def _SetDefaults(p): - p.random_seed = 12345 - p.decoder.input_dropout_prob = 0.0 - mp = p.encoder.transformer_stack.transparent_merger_tpl - mp.weighted_merger_dropout_prob = 0.0 - disable_vn = py_utils.VariationalNoiseParams(1.0, False, False) - for lp in base_layer.RecursiveFindLayerParams(p): - # TODO(lepikhin): lp.dtype = dtype - lp.params_init = py_utils.WeightInit.Gaussian(0.1, 12345) - lp.vn = disable_vn - - tp = p.train - assert tp.l2_regularizer_weight is None - tp.clip_gradient_norm_to_value = False - tp.grad_norm_to_clip_to_zero = False - tp.optimizer = optimizer.SGD.Params() - tp.learning_rate = 1e-2 - tp.lr_schedule = schedule.ContinuousSchedule.Params() - for l in p.ToText().split('\n'): - print(l) - return p - - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._testParams() - p.input = TestInputGenerator.Params() - p.input.split = True - p = _SetDefaults(p) - p.train.optimizer = optimizer.Accumulator.Params().Set( - accum_steps=2, optimizer_tpl=p.train.optimizer) - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - mdl.BProp() - - self.evaluate(tf.global_variables_initializer()) - - for _ in range(2): - self.evaluate(mdl.train_op) - - expected = self.evaluate(mdl.dec.softmax.vars['weight_0']) - - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._testParams() - p.input = TestInputGenerator.Params() - p.input.split = False - p = _SetDefaults(p) - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - mdl.BProp() - - self.evaluate(tf.global_variables_initializer()) - - self.evaluate(mdl.train_op) - - actual = self.evaluate(mdl.dec.softmax.vars['weight_0']) - - self.assertAllClose(expected, actual, rtol=1e-2, atol=1e-2) - - def testBatchSplit(self): - - def Run(num_splits): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(93820981) - p = self._testParams() - p.input.bucket_batch_limit = [ - b * 2 / num_splits for b in p.input.bucket_batch_limit - ] - with cluster_factory.ForTestingWorker(gpus=num_splits): - mdl = p.Instantiate() - metrics = mdl.FPropDefaultTheta()[0] - self.evaluate(tf.global_variables_initializer()) - return self.evaluate(metrics['loss']) - - res1, res2 = Run(1), Run(2) - self.assertAllClose(res1[0], res2[0]) - self.assertAllEqual(res1[1], res2[1]) - - def testBatchSizeInInputGenerator(self): - with self.session(): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._testParams() - with cluster_factory.ForTestingWorker( - mode='sync', job='trainer_client', gpus=5): - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - loss = mdl.loss - self.evaluate(tf.global_variables_initializer()) - _ = self.evaluate(loss) - self.assertEqual(mdl.input_generator.infeed_bucket_batch_limit, [40]) - - def testDecode(self): - with self.session(use_gpu=False): - tf.random.set_seed(93820985) - p = self._testParams() - mdl = p.Instantiate() - input_batch = mdl.input_generator.GetPreprocessedInputBatch() - dec_out_dict = mdl.Decode(input_batch) - self.evaluate(tf.global_variables_initializer()) - dec_out = self.evaluate(dec_out_dict) - metrics_dict = mdl.CreateDecoderMetrics() - key_value_pairs = mdl.PostProcessDecodeOut(dec_out, metrics_dict) - self.assertNear(0.0, metrics_dict['corpus_bleu'].value, 1.0e-5) - self.assertLen(key_value_pairs, 8) - for k, v in key_value_pairs: - self.assertIn(k, v) - - -class RNMTModelTest(test_utils.TestCase): - - def _InputParams(self): - p = input_generator.NmtInput.Params() - input_file = test_helper.test_src_dir_path( - 'tasks/mt/testdata/wmt14_ende_wpm_32k_test.tfrecord') - vocab_file = test_helper.test_src_dir_path( - 'tasks/mt/testdata/wmt14_ende_wpm_32k_test.vocab') - p.file_pattern = 'tfrecord:' + input_file - p.file_random_seed = 31415 - p.file_parallelism = 1 - p.bucket_upper_bound = [40] - p.bucket_batch_limit = [8] - p.source_max_length = 200 - p.target_max_length = 200 - - p.tokenizer.token_vocab_filepath = vocab_file - p.tokenizer.vocab_size = 32000 - return p - - def _EncoderParams(self): - p = encoder.MTEncoderBiRNN.Params() - p.name = 'encoder' - p.emb.vocab_size = 32000 - p.emb.embedding_dim = 4 - p.emb.max_num_shards = 1 - p.lstm_cell_size = 4 - p.num_lstm_layers = 3 - p.encoder_out_dim = 4 - return p - - def _DecoderParams(self): - p = decoder.MTDecoderV1.Params() - p.name = 'decoder' - p.source_dim = 4 - p.emb.vocab_size = 32000 - p.emb.embedding_dim = 4 - p.emb.max_num_shards = 1 - p.rnn_cell_dim = 4 - p.rnn_layers = 3 - p.attention.hidden_dim = 2 - p.softmax.num_classes = 32000 - p.softmax.num_shards = 1 - return p - - def _testParams(self): - p = model.RNMTModel.Params() - p.name = 'test_mdl' - p.input = self._InputParams() - p.encoder = self._EncoderParams() - p.decoder = self._DecoderParams() - p.train.learning_rate = 1.0 - return p - - def testConstruction(self): - with self.session(): - p = self._testParams() - mdl = p.Instantiate() - flatten_vars = mdl.vars.Flatten() - # encoder/embedding: 1 - # encoder/lstms: 2 * (3 (forward) + 3 (backward)) - # encoder/proj: 2 - # decoder/embedding: 1 - # decoder/atten: 3 - # decoder/lstms: 2 * 3 - # decoder/softmax: 2 - self.assertEqual(len(flatten_vars), 1 + 12 + 2 + 1 + 3 + 6 + 2) - - # Should match tf.trainable_variables(). - self.assertEqual(len(tf.trainable_variables()), len(flatten_vars)) - - def testFProp(self): - with self.session(): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._testParams() - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - loss = mdl.loss - logp = mdl.eval_metrics['log_pplx'][0] - self.evaluate(tf.global_variables_initializer()) - vals = [] - for _ in range(5): - vals += [self.evaluate((loss, logp))] - self.assertAllClose(vals, [ - [233.403564, 10.373495], - [235.996948, 10.373494], - [217.843338, 10.373493], - [217.843338, 10.373491], - [159.492432, 10.373494], - ]) - - def testFPropEvalMode(self): - with self.session(), self.SetEval(True): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._testParams() - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - loss = mdl.loss - logp = mdl.eval_metrics['log_pplx'][0] - self.evaluate(tf.global_variables_initializer()) - vals = [] - for _ in range(5): - vals += [self.evaluate((loss, logp))] - self.assertAllClose(vals, [ - [233.403564, 10.373495], - [235.996948, 10.373494], - [217.843338, 10.373493], - [217.843338, 10.373491], - [159.492432, 10.373494], - ]) - - def testBProp(self): - with self.session(): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._testParams() - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - mdl.BProp() - loss = mdl.loss - logp = mdl.eval_metrics['log_pplx'][0] - - self.evaluate(tf.global_variables_initializer()) - vals = [] - for _ in range(5): - vals += [self.evaluate((loss, logp, mdl.train_op))[:2]] - expected_vals = [ - [233.403564, 10.373495], - [219.442184, 9.645809], - [181.665314, 8.650729], - [185.266647, 8.822222], - [157.343857, 10.233747], - ] - self.assertAllClose(vals, expected_vals, atol=1e-3) - - def testDecode(self): - with self.session(use_gpu=False), self.SetEval(True): - tf.random.set_seed(93820985) - p = self._testParams() - mdl = p.Instantiate() - input_batch = mdl.input_generator.GetPreprocessedInputBatch() - dec_out_dict = mdl.Decode(input_batch) - self.evaluate(tf.global_variables_initializer()) - dec_out = self.evaluate(dec_out_dict) - metrics_dict = mdl.CreateDecoderMetrics() - key_value_pairs = mdl.PostProcessDecodeOut(dec_out, metrics_dict) - self.assertNear(0.0, metrics_dict['corpus_bleu'].value, 1.0e-5) - self.assertLen(key_value_pairs, 8) - for k, v in key_value_pairs: - self.assertIn(k, v) - - def testBatchSplit(self): - - def Run(num_splits): - with self.session(use_gpu=False, graph=tf.Graph()): - tf.random.set_seed(93820981) - p = self._testParams() - p.input.bucket_batch_limit = [ - b * 2 / num_splits for b in p.input.bucket_batch_limit - ] - with cluster_factory.ForTestingWorker(gpus=num_splits): - mdl = p.Instantiate() - metrics = mdl.FPropDefaultTheta()[0] - self.evaluate(tf.global_variables_initializer()) - return self.evaluate(metrics['loss']) - - res1, res2 = Run(1), Run(2) - self.assertAllClose(res1[0], res2[0]) - self.assertAllEqual(res1[1], res2[1]) - - def testBatchSizeInInputGenerator(self): - with self.session(): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._testParams() - cluster_params = cluster_factory.Cluster.Params() - cluster_params.mode = 'sync' - cluster_params.job = 'trainer_client' - cluster_params.worker.name = '/job:localhost' - cluster_params.worker.gpus_per_replica = 5 - cluster_params.input.name = '/job:localhost' - cluster_params.input.replicas = 1 - cluster_params.input.gpus_per_replica = 0 - with cluster_params.Instantiate(): - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - loss = mdl.loss - self.evaluate(tf.global_variables_initializer()) - _ = self.evaluate(loss) - self.assertEqual(mdl.input_generator.infeed_bucket_batch_limit, [40]) - - -class InsertionModelTest(test_utils.TestCase): - - def _InputParams(self): - p = input_generator.NmtInput.Params() - input_file = test_helper.test_src_dir_path( - 'tasks/mt/testdata/wmt14_ende_wpm_32k_test.tfrecord') - vocab_file = test_helper.test_src_dir_path( - 'tasks/mt/testdata/wmt14_ende_wpm_32k_test.vocab') - p.file_pattern = 'tfrecord:' + input_file - p.file_random_seed = 31415 - p.file_parallelism = 1 - p.bucket_upper_bound = [40] - p.bucket_batch_limit = [8] - p.source_max_length = 200 - p.target_max_length = 200 - - p.tokenizer.token_vocab_filepath = vocab_file - p.tokenizer.vocab_size = 32000 - return p - - def _DecoderParams(self): - p = decoder.InsertionDecoder.Params() - p.name = 'decoder' - return p - - def _testParams(self): - p = model.InsertionModel.Params() - p.name = 'insertion' - p.input = self._InputParams() - p.decoder = self._DecoderParams() - p.random_seed = 12345 - return p - - def testSampleCanvasAndTargets(self): - with self.session(): - tf.random.set_seed(_TF_RANDOM_SEED) - - x = np.asarray([[10, 11, 12, 13, 14, 15, 2], [10, 11, 12, 13, 14, 15, 2], - [2, 0, 0, 0, 0, 0, 0], [10, 11, 12, 13, 14, 2, 0]], - np.int32) - x_paddings = np.asarray([[0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], - [0, 1, 1, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 1]], - np.float32) - - p = self._testParams() - mdl = p.Instantiate() - - descriptor = mdl._SampleCanvasAndTargets( - tf.convert_to_tensor(x), tf.convert_to_tensor(x_paddings)) - - canvas, canvas_paddings, target_indices, target_weights = self.evaluate([ - descriptor.canvas, descriptor.canvas_paddings, - descriptor.target_indices, descriptor.target_weights - ]) - - canvas_gold = np.asarray([[13, 15, 2, 0, 0], [10, 11, 14, 2, 0], - [2, 0, 0, 0, 0], [10, 11, 13, 14, 2]], np.int32) - canvas_paddings_gold = np.asarray( - [[0., 0., 0., 1., 1.], [0., 0., 0., 0., 1.], [0., 1., 1., 1., 1.], - [0., 0., 0., 0., 0.]], np.float32) - target_indices_gold = np.asarray( - [[0, 0, 10], [0, 0, 11], [0, 0, 12], [0, 0, 2], [0, 1, 14], [0, 1, 2], - [0, 2, 2], [1, 0, 2], [1, 1, 2], [1, 2, 12], [1, 2, 13], [1, 2, 2], - [1, 3, 15], [1, 3, 2], [2, 0, 2], [3, 0, 2], [3, 1, 2], [3, 2, 12], - [3, 2, 2], [3, 3, 2], [3, 4, 2]], np.int32) - target_weights_gold = np.asarray([1, 1, 1, 0, 1, 0, 1] + - [1, 1, 1, 1, 0, 1, 0] + [1] + - [1, 1, 1, 0, 1, 1], np.float32) - target_weights_gold = np.reshape(target_weights_gold, - [target_weights_gold.shape[0], 1]) - - self.assertAllEqual(canvas, canvas_gold) - self.assertAllEqual(canvas_paddings, canvas_paddings_gold) - self.assertAllEqual(target_indices, target_indices_gold) - self.assertAllEqual(target_weights, target_weights_gold) - - def testCreateCanvasAndTargets(self): - with self.session(): - tf.random.set_seed(_TF_RANDOM_SEED) - batch = py_utils.NestedMap( - src=py_utils.NestedMap( - ids=tf.convert_to_tensor( - np.asarray([ - [10, 11, 12, 14, 2, 0], - [20, 21, 22, 24, 25, 2], - ], np.int32)), - paddings=tf.convert_to_tensor( - np.asarray([[0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 0]], - np.float32))), - tgt=py_utils.NestedMap( - ids=tf.convert_to_tensor( - np.asarray([[100, 101, 102, 104, 2, 0], - [200, 201, 202, 204, 205, 2]], np.int32)), - paddings=tf.convert_to_tensor( - np.asarray([[0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 0]], - np.float32)))) - - p = self._testParams() - mdl = p.Instantiate() - - descriptor = mdl._CreateCanvasAndTargets(batch) - - canvas, canvas_paddings, target_indices, target_weights = self.evaluate([ - descriptor.canvas, descriptor.canvas_paddings, - descriptor.target_indices, descriptor.target_weights - ]) - - canvas_gold = np.asarray([ - [32014, 32002, 104, 2, 0, 0, 0, 0], - [32020, 32021, 32022, 32002, 200, 201, 202, 2], - ], np.int32) - canvas_paddings_gold = np.asarray( - [[0., 0., 0., 0., 1., 1., 1., 1.], [0., 0., 0., 0., 0., 0., 0., 0.]], - np.float32) - target_indices_gold = np.asarray( - [[0, 0, 10], [0, 0, 11], [0, 0, 12], [0, 0, 2], [0, 1, 2], [1, 0, 2], - [1, 1, 2], [1, 2, 2], [1, 3, 24], [1, 3, 25], [1, 3, 2], [0, 2, 100], - [0, 2, 101], [0, 2, 102], [0, 2, 2], [0, 3, 2], [1, 4, 2], [1, 5, 2], - [1, 6, 2], [1, 7, 204], [1, 7, 205], [1, 7, 2]], np.int32) - target_weights_gold = np.asarray([1, 1, 1, 0, 1] + [1, 1, 1, 1, 1, 0] + - [1, 1, 1, 0, 1] + [1, 1, 1, 1, 1, 0], - np.float32) - target_weights_gold = np.reshape(target_weights_gold, - [target_weights_gold.shape[0], 1]) - - self.assertAllEqual(canvas, canvas_gold) - self.assertAllEqual(canvas_paddings, canvas_paddings_gold) - self.assertAllEqual(target_indices, target_indices_gold) - self.assertAllEqual(target_weights, target_weights_gold) - - def testConstruction(self): - with self.session(): - p = self._testParams() - mdl = p.Instantiate() - flatten_vars = mdl.vars.Flatten() - self.assertEqual(len(flatten_vars), 122) - self.assertEqual(len(tf.trainable_variables()), len(flatten_vars)) - - def testFPropGraph(self): - """Test the construction of the fprop graph, then fprop the graph.""" - with self.session(): - p = self._testParams() - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - - self.evaluate(tf.global_variables_initializer()) - self.evaluate(mdl.loss) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/mt/params/BUILD b/lingvo/tasks/mt/params/BUILD deleted file mode 100644 index cc88e3bd..00000000 --- a/lingvo/tasks/mt/params/BUILD +++ /dev/null @@ -1,30 +0,0 @@ -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -filegroup( - name = "param_files", - srcs = glob( - include = [ - "*.py", - ], - ), -) - -py_library( - name = "params", - srcs = [ - ":param_files", - ], - srcs_version = "PY3", - tags = ["keep_dep"], - deps = [ - "//lingvo:model_registry", - "//lingvo/core:base_model_params", - "//lingvo/tasks/mt:base_config", - "//lingvo/tasks/mt:input_generator", - "//lingvo/tasks/mt:model", - ], -) diff --git a/lingvo/tasks/mt/params/params.py b/lingvo/tasks/mt/params/params.py deleted file mode 100644 index 1d0b522f..00000000 --- a/lingvo/tasks/mt/params/params.py +++ /dev/null @@ -1,22 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Machine translation model hyper-parameters.""" - -# Import ModelParams to ensure that they are added to the global registry. -# pylint: disable=unused-import -import lingvo.tasks.mt.params.wmt14_en_de -import lingvo.tasks.mt.params.wmtm16_en_de -# pylint: enable=unused-import diff --git a/lingvo/tasks/mt/params/wmt14_en_de.py b/lingvo/tasks/mt/params/wmt14_en_de.py deleted file mode 100644 index 10dfa5e4..00000000 --- a/lingvo/tasks/mt/params/wmt14_en_de.py +++ /dev/null @@ -1,202 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Train NMT Models on WMT'14 English-German machine translation task.""" - -import os - -from lingvo import model_registry -from lingvo.core import base_model_params -from lingvo.tasks.mt import base_config -from lingvo.tasks.mt import input_generator -from lingvo.tasks.mt import model - - -@model_registry.RegisterSingleTaskModel -class WmtEnDeTransformerBase(base_model_params.SingleTaskModelParams): - """Params for WMT'14 En->De.""" - - DATADIR = '/tmp/wmt14/wpm/' - VOCAB_SIZE = 32000 - - def _CommonInputParams(self, is_eval): - """Input generator params for WMT'14 En->De.""" - p = input_generator.NmtInput.Params() - - p.tokenizer.vocab_size = self.VOCAB_SIZE - - if is_eval: - p.file_random_seed = 27182818 - p.file_parallelism = 1 - p.file_buffer_size = 1 - p.bucket_upper_bound = [10, 14, 19, 26, 36, 50, 70, 98, 137, 200] - p.bucket_batch_limit = [16] * 8 + [4] * 2 - else: - p.file_random_seed = 0 - p.file_parallelism = 16 - p.file_buffer_size = 10000000 - p.bucket_upper_bound = ([8, 10, 12, 14, 16, 20, 24, 28] + - [32, 40, 48, 56, 64, 80, 96]) - p.bucket_batch_limit = ([512, 409, 341, 292, 256, 204, 170, 146] + - [128, 102, 85, 73, 64, 51, 42]) - - return p - - def Train(self): - p = self._CommonInputParams(is_eval=False) - - p.file_pattern = 'tfrecord:' + os.path.join(self.DATADIR, - 'train.tfrecords-*') - p.tokenizer.token_vocab_filepath = os.path.join(self.DATADIR, - 'wpm-ende.voc') - p.num_samples = 4492447 - return p - - def Dev(self): - p = self._CommonInputParams(is_eval=True) - p.file_pattern = 'tfrecord:' + os.path.join(self.DATADIR, 'dev.tfrecords') - p.tokenizer.token_vocab_filepath = os.path.join(self.DATADIR, - 'wpm-ende.voc') - p.num_samples = 3000 - return p - - def Test(self): - p = self._CommonInputParams(is_eval=True) - p.file_pattern = 'tfrecord:' + os.path.join(self.DATADIR, 'test.tfrecords') - p.tokenizer.token_vocab_filepath = os.path.join(self.DATADIR, - 'wpm-ende.voc') - p.num_samples = 2737 - return p - - def Task(self): - p = base_config.SetupTransformerParams( - model.TransformerModel.Params(), - name='wmt14_en_de_transformer_base', - vocab_size=self.VOCAB_SIZE, - model_dim=512, - hidden_dim=2048, - num_heads=8, - num_layers=6, - residual_dropout_prob=0.1, - input_dropout_prob=0.1, - learning_rate=3.0, - warmup_steps=40000) - p.eval.samples_per_summary = 7500 - return p - - -@model_registry.RegisterSingleTaskModel -class WmtEnDeTransformerSmall(WmtEnDeTransformerBase): - """Small Transformer Params for WMT'14 En->De.""" - - def Task(self): - p = base_config.SetupTransformerParams( - model.TransformerModel.Params(), - name='wmt14_en_de_transformer_small', - vocab_size=self.VOCAB_SIZE, - model_dim=64, - hidden_dim=128, - num_heads=2, - num_layers=2, - residual_dropout_prob=0.1, - input_dropout_prob=0.1, - learning_rate=3.0, - warmup_steps=40000) - p.eval.samples_per_summary = 7500 - return p - - -@model_registry.RegisterSingleTaskModel -class WmtEnDeTransformerSmallCloudTpu(WmtEnDeTransformerSmall): - """Small Transformer Params for WMT'14 En->De on TPU.""" - - def _CommonInputParams(self, is_eval): - p = super()._CommonInputParams(is_eval) - - p.pad_to_max_seq_length = True - p.source_max_length = p.bucket_upper_bound[-1] - p.bucket_batch_limit = [64] * len(p.bucket_upper_bound) - return p - - def Task(self): - p = super().Task() - - p.decoder.token_emb.max_num_shards = 1 - p.encoder.token_emb.max_num_shards = 1 - return p - - -@model_registry.RegisterSingleTaskModel -class WmtEnDeRNMT(WmtEnDeTransformerBase): - """Params for WMT'14 En->De in sync training.""" - - def _CommonInputParams(self, is_eval): - p = super()._CommonInputParams(is_eval) - - if is_eval: - p.bucket_upper_bound = [10, 14, 19, 26, 36, 50, 70, 98, 200] - p.bucket_batch_limit = [128] * 8 + [32] - else: - p.bucket_upper_bound = [10, 14, 19, 26, 36, 50, 70, 98] - p.bucket_batch_limit = [128] * 7 + [64] - - return p - - def Task(self): - p = base_config.SetupRNMTParams( - model.RNMTModel.Params(), - name='wmt14_en_de_rnmtplus_base', - vocab_size=self.VOCAB_SIZE, - embedding_dim=1024, - hidden_dim=1024, - num_heads=4, - num_encoder_layers=6, - num_decoder_layers=8, - learning_rate=1e-4, - l2_regularizer_weight=1e-5, - lr_warmup_steps=500, - lr_decay_start=400000, - lr_decay_end=1200000, - lr_min=0.5, - ls_uncertainty=0.1, - atten_dropout_prob=0.3, - residual_dropout_prob=0.3, - adam_beta2=0.98, - adam_epsilon=1e-6, - ) - p.eval.samples_per_summary = 7500 - return p - - -@model_registry.RegisterSingleTaskModel -class WmtEnDeRNMTCloudTpu(WmtEnDeRNMT): - """Params for WMT'14 En->De in sync training on TPU.""" - - def _CommonInputParams(self, is_eval): - p = super()._CommonInputParams(is_eval) - - p.pad_to_max_seq_length = True - p.source_max_length = p.bucket_upper_bound[-1] - p.bucket_batch_limit = [16] * len(p.bucket_upper_bound) - - return p - - def Task(self): - p = super().Task() - - p.encoder.emb.max_num_shards = 1 - p.decoder.emb.max_num_shards = 1 - - return p diff --git a/lingvo/tasks/mt/params/wmtm16_en_de.py b/lingvo/tasks/mt/params/wmtm16_en_de.py deleted file mode 100644 index 28fe3aa1..00000000 --- a/lingvo/tasks/mt/params/wmtm16_en_de.py +++ /dev/null @@ -1,114 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Train NMT Models on WMT'16 MMT English-German machine translation task.""" - -import os - -from lingvo import model_registry -from lingvo.core import base_model_params -from lingvo.tasks.mt import base_config -from lingvo.tasks.mt import input_generator -from lingvo.tasks.mt import model - - -@model_registry.RegisterSingleTaskModel -class WmtCaptionEnDeTransformer(base_model_params.SingleTaskModelParams): - """Params for WMT'16 En->De Captions (ignoring the images).""" - - # Generated using scripts in lingvo/mt/tools. - DATADIR = '/tmp/wmtm16/wpm/' - VOCAB_SIZE = 2000 - VOCAB_FILE = 'wpm-ende-2k.voc' - - def _CommonInputParams(self, is_eval): - """Input generator params for WMT'16 En->De.""" - p = input_generator.NmtInput.Params() - if is_eval: - p.file_random_seed = 27182818 - p.file_parallelism = 1 - p.file_buffer_size = 1 - p.bucket_upper_bound = [10, 14, 19, 26, 36, 50, 70, 98, 137, 200] - p.bucket_batch_limit = [16] * 8 + [4] * 2 - else: - p.file_random_seed = 0 - p.file_parallelism = 1 - p.file_buffer_size = 29000 - p.bucket_upper_bound = [14, 17, 20, 24, 29, 35, 45, 75] - p.bucket_batch_limit = [292, 240, 204, 170, 141, 117, 91, 54] - - p.tokenizer.vocab_size = self.VOCAB_SIZE - p.tokenizer.token_vocab_filepath = os.path.join(self.DATADIR, - self.VOCAB_FILE) - - return p - - def Train(self): - p = self._CommonInputParams(is_eval=False) - p.file_pattern = 'tfrecord:' + os.path.join(self.DATADIR, 'train.tfrecords') - p.num_samples = 29000 - return p - - def Dev(self): - p = input_generator.NmtInput.Params() - p.file_pattern = 'tfrecord:' + os.path.join(self.DATADIR, 'val.tfrecords') - p.num_samples = 1014 - return p - - def Test(self): - p = input_generator.NmtInput.Params() - p.file_pattern = 'tfrecord:' + os.path.join(self.DATADIR, 'test.tfrecords') - p.num_samples = 1000 - return p - - def Task(self): - p = base_config.SetupTransformerParams( - model.TransformerModel.Params(), - name='wmt14_en_de_transformer_base', - vocab_size=self.VOCAB_SIZE, - model_dim=256, - hidden_dim=512, - num_heads=2, - num_layers=2, - residual_dropout_prob=0.2, - input_dropout_prob=0.2, - learning_rate=1.0, - warmup_steps=1000) - p.eval.samples_per_summary = 7500 - p.train.save_interval_seconds = 60 - p.train.max_steps = 12000 - return p - - -@model_registry.RegisterSingleTaskModel -class WmtCaptionEnDeTransformerCloudTpu(WmtCaptionEnDeTransformer): - """Params for WMT'16 En->De Captions (ignoring the images) on TPU.""" - - def _CommonInputParams(self, is_eval): - p = super()._CommonInputParams(is_eval) - - p.pad_to_max_seq_length = True - p.source_max_length = p.bucket_upper_bound[-1] - p.bucket_batch_limit = [16] * len(p.bucket_batch_limit) - - return p - - def Task(self): - p = super().Task() - - p.encoder.token_emb.max_num_shards = 1 - p.decoder.token_emb.max_num_shards = 1 - - return p diff --git a/lingvo/tasks/mt/testdata/BUILD b/lingvo/tasks/mt/testdata/BUILD deleted file mode 100644 index 0fc88ccb..00000000 --- a/lingvo/tasks/mt/testdata/BUILD +++ /dev/null @@ -1,29 +0,0 @@ -package(default_visibility = ["//visibility:public"]) - -licenses(["notice"]) # Apache 2.0 - -# First 100 sentences of the news-test 2013 from WMT'14 development set. -# http://www.statmt.org/wmt14/dev.tgz -filegroup( - name = "wmt14_ende_tfexample", - data = [ - "wmt14_ende_wpm_32k_test.tfrecord", - "wmt14_ende_wpm_32k_test.vocab", - ], -) - -filegroup( - name = "mlperf_tfexample", - data = [ - "translate_ende_mlperf.packed.tfrecord", - "translate_ende_wmt32k-train-00511-of-00512", - ], -) - -filegroup( - name = "input_test_data", - data = [ - "en_de.text", - "en_fr.tfrecord", - ], -) diff --git a/lingvo/tasks/mt/testdata/en_de.text b/lingvo/tasks/mt/testdata/en_de.text deleted file mode 100644 index 1eecbd63..00000000 --- a/lingvo/tasks/mt/testdata/en_de.text +++ /dev/null @@ -1,2 +0,0 @@ -Too much has changed. Dafür hat sich zu viel verändert. -Hello! Hallo! diff --git a/lingvo/tasks/mt/testdata/en_fr.tfrecord b/lingvo/tasks/mt/testdata/en_fr.tfrecord deleted file mode 100644 index 3422000a..00000000 Binary files a/lingvo/tasks/mt/testdata/en_fr.tfrecord and /dev/null differ diff --git a/lingvo/tasks/mt/testdata/translate_ende_mlperf.packed.tfrecord b/lingvo/tasks/mt/testdata/translate_ende_mlperf.packed.tfrecord deleted file mode 100644 index 761111f9..00000000 Binary files a/lingvo/tasks/mt/testdata/translate_ende_mlperf.packed.tfrecord and /dev/null differ diff --git a/lingvo/tasks/mt/testdata/translate_ende_wmt32k-train-00511-of-00512 b/lingvo/tasks/mt/testdata/translate_ende_wmt32k-train-00511-of-00512 deleted file mode 100644 index 301038bb..00000000 Binary files a/lingvo/tasks/mt/testdata/translate_ende_wmt32k-train-00511-of-00512 and /dev/null differ diff --git a/lingvo/tasks/mt/testdata/wmt14_ende_wpm_32k_test.tfrecord b/lingvo/tasks/mt/testdata/wmt14_ende_wpm_32k_test.tfrecord deleted file mode 100644 index 4e72fd12..00000000 Binary files a/lingvo/tasks/mt/testdata/wmt14_ende_wpm_32k_test.tfrecord and /dev/null differ diff --git a/lingvo/tasks/mt/testdata/wmt14_ende_wpm_32k_test.vocab b/lingvo/tasks/mt/testdata/wmt14_ende_wpm_32k_test.vocab deleted file mode 100644 index d96d5870..00000000 --- a/lingvo/tasks/mt/testdata/wmt14_ende_wpm_32k_test.vocab +++ /dev/null @@ -1,32000 +0,0 @@ - 0.00010073416224276751 - 0 - 0.034041494367619232 -▁, 0.044370583669798951 -▁. 0.034193368449234512 -▁the 0.024996024997377464 -▁in 0.013820257295280546 -▁of 0.013096835231150048 -▁and 0.011855031910074663 -▁die 0.011740110104447195 -▁der 0.011334449579483175 -▁to 0.011119680102548058 -▁und 0.010981523899890322 -▁a 0.0072767267451170738 -▁is 0.0058704945092683571 -▁that 0.0044993657990812239 -▁zu 0.0044366939207065685 -▁for 0.0043121192137855853 -▁von 0.0040675083304576049 -▁den 0.00404126592572441 -▁ist 0.0034980023078314011 -▁für 0.003348984697007059 -▁on 0.0032371163601561119 -▁mit 0.0030027758910379895 -▁an 0.0029572542026773617 -▁( 0.0029510714966683284 -▁with 0.0029246434592179508 -▁) 0.0029235523934516504 -▁auf 0.0029215483177906345 -▁das 0.0028834633033857247 -▁The 0.0028719464980747804 -▁be 0.0028172227307337958 -▁are 0.0027216521887669549 -▁des 0.0026750621651768251 -▁I 0.0025450510715383408 -▁eine 0.0024546555151158259 -▁as 0.0023835430307435437 -▁this 0.002364900202146452 -▁: 0.0022545396575697667 -▁Sie 0.0022526795419751376 -▁im 0.0022435039819543785 -▁nicht 0.0021727665514392615 -▁" 0.002111117547220512 -▁we 0.0021092271242434853 -▁it 0.0021047454200714957 -▁- 0.0020964071014894603 -▁werden 0.0020879437649550356 -▁ein 0.0020739114469051217 -▁by 0.0020056478564781785 -▁have 0.0019806063817724737 -▁dass 0.0019659262434238191 -▁dem 0.00195816755353013 -▁not 0.0019549511825732247 -▁sich 0.0019108387874940678 -▁Die 0.0018981665132292293 -▁you 0.0018804480597952532 -▁wir 0.0018604792832182834 -▁es 0.001792825628862084 -▁will 0.0017900865991779349 -▁which 0.0017094765388469229 -▁auch 0.0016807148329519592 -▁was 0.0016336133722838762 -▁from 0.0016308061509893333 -s 0.0016028778981102931 -▁sind 0.0015479344022469286 -▁at 0.0015133082178581023 -▁or 0.0014936879761787004 -▁wird 0.0014589557159514837 -▁has 0.0014394491269560715 -▁In 0.0013971248674383508 -es 0.0013324034523290826 -▁über 0.001292912933065502 -▁als 0.0012920643263583797 -▁um 0.0012841237921703075 -▁can 0.0012529867451799613 -▁ich 0.0012514372802549033 -▁all 0.001250524270360187 -▁oder 0.0012454780911910495 -▁also 0.0011776880536140625 -▁our 0.0011752520977538858 -▁' 0.0011725736828345312 -▁haben 0.0011563819637888152 -▁European 0.0011486535812775236 -ing 0.0011432967514388146 -▁/ 0.0011248622860957041 -▁so 0.0011227597114418968 -ed 0.0011172134604632053 -▁aus 0.0011037645595244378 -- 0.001093854045480546 -▁hat 0.0010883911398034468 -▁einer 0.0010788973522675169 -▁! 0.0010655924113951362 -▁sie 0.0010651567427731763 -▁? 0.0010530944045790817 -▁wie 0.0010497833230521854 -en 0.0010272308421258522 -▁zur 0.0010010528405802516 -er 0.00098061429957612464 -▁einen 0.0009742270187359101 -▁s 0.00096388083607005842 -▁Das 0.0009567586012067112 -▁bei 0.00095570541966840772 -▁Union 0.000946503340688051 -▁your 0.00092571247636355665 -▁'s 0.00091399109622143085 -▁We 0.00090434198335071522 -▁Ich 0.00088891931413332878 -▁können 0.00087744418147318046 -▁but 0.00085214130559429959 -▁its 0.00084567446787529226 -▁nach 0.00082966838154675923 -▁– 0.00082007609501803835 -▁am 0.00081728402741469426 -▁would 0.00081509053061369532 -▁more 0.00080059602498222374 -▁their 0.00079561046057788072 -e 0.0007949436981651418 -▁einem 0.00079203797787780813 -▁uns 0.00077510372796336024 -▁been 0.00077380808736587892 -▁’ 0.0007671025790104935 -▁durch 0.00076420064714595947 -▁Wir 0.00076088956561906292 -▁vor 0.00075604417285830708 -▁zum 0.00075543044836476325 -▁This 0.000752498209117832 -▁It 0.00074820971450862512 -i 0.00074789527541625387 -o 0.00074594802609723241 -▁one 0.00074219369910277647 -▁diese 0.000739950952805382 -▁nur 0.0007375718232872 -a 0.00072052013226596293 -▁should 0.00070793499172551639 -▁kann 0.00070666587008763268 -▁Mr 0.00070654842898084345 -▁; 0.00069951332784188821 -▁Der 0.00069749788691247287 -▁dieser 0.000661568485080566 -▁Commission 0.0006507904222155538 -▁EU 0.00065002894923282351 -▁very 0.00064103902192924752 -▁Es 0.00063728090651199193 -▁must 0.00063708769694920966 -▁... 0.00063106031627496208 -▁A 0.00062999955789105931 -▁daß 0.0006262300772054048 -▁er 0.00062021406179955615 -▁Kommission 0.00061681963497106727 -▁there 0.00061680826970266834 -▁they 0.00061316001854660269 -▁noch 0.00061047781520444844 -y 0.000608780601790204 -▁new 0.00060785243820428913 -▁do 0.000599506542776654 -▁wurde 0.00059842305385595328 -▁other 0.00059783205989920752 -▁Hotel 0.00059718423960046692 -▁“ 0.00059128187687860784 -▁only 0.00058804656380770435 -▁aber 0.00057848079623858107 -▁like 0.00057809816553581615 -▁Herr 0.00057794662862383 -▁about 0.00057130552345603875 -▁time 0.00056780123236635993 -▁B 0.0005651569132522022 -▁wenn 0.00056440680553787093 -▁sehr 0.00054936676702324941 -▁S 0.0005473172302886373 -▁sein 0.000545858687510771 -▁müssen 0.00054376747812536267 -▁President 0.00054175961404154661 -▁de 0.00053396682834266082 -▁man 0.00053339098807711363 -▁P 0.00052664759549373168 -▁war 0.00051676360040943761 -▁these 0.00051601349269510635 -▁D 0.00051332750093015256 -in 0.00050956559709009733 -▁who 0.00050898217997895074 -▁up 0.00050648939777677923 -▁us 0.00050185236827000423 -▁L 0.000491991103722508 -▁diesem 0.00049076365473542057 -k 0.00049029767873106321 -▁if 0.00048509238580434031 -▁no 0.00048442562339160144 -▁T 0.00048395964738724413 -▁mehr 0.00048258823833376982 -▁C 0.00047809895731618129 -t 0.0004762426301443514 -▁out 0.00047573119306639828 -el 0.0004725943789882858 -▁alle 0.00046638136559685523 -▁Europäischen 0.00046511224395897152 -▁any 0.0004648356890945969 -▁M 0.00046275963340038713 -an 0.00046086163357776113 -▁% 0.00045811123862521321 -▁Europe 0.0004545463327707399 -▁K 0.00045376591767401145 -▁H 0.00045362195760762462 -▁Parliament 0.000451523171376617 -▁people 0.00044890537122205693 -▁into 0.00044856441317008819 -em 0.00044719300411661387 -▁were 0.00044709071670102327 -▁R 0.00044435926386247361 -on 0.00044034353569484162 -▁bis 0.000439150182512951 -ers 0.00043886605080297706 -▁unter 0.00043607398319963297 -▁möchte 0.00043362666207105726 -al 0.00043361529680265828 -▁F 0.00043222115721238606 -▁such 0.00043083080604491348 -et 0.0004271257285468531 -te 0.00042528076664342219 -▁what 0.00042159084283656037 -▁G 0.00042023458747428468 -▁States 0.00041885181315241138 -ic 0.00041759784520572634 -u 0.00041715838816096658 -ung 0.00041364273180288883 -▁report 0.00041314265999333464 -▁eines 0.00041152879188068259 -▁Member 0.0004082139219309864 -is 0.0004068879739511079 -ar 0.00040560748704482525 -▁my 0.00040498997412848184 -▁need 0.00040465659292211243 -▁countries 0.00040439140332613674 -▁than 0.00040345945131742213 -ischen 0.00040323214594944297 -▁vom 0.000401311415590019 -n 0.00039935658942539815 -or 0.00039776924027234365 -▁when 0.00039770483708474958 -b 0.00039470440622742454 -▁his 0.00039465515673102908 -▁well 0.0003938671647887013 -▁ihre 0.00039180247436289051 -ten 0.00038935894165711445 -▁Präsident 0.00038934757638871552 -▁work 0.00038863156447958115 -▁use 0.00038858610340598528 -▁most 0.00038749882606248497 -ig 0.00038653656667137317 -▁some 0.00038640018345058563 -▁them 0.00038526744503348945 -▁Council 0.00038520304184589538 -▁first 0.00038483177641152939 -▁[ 0.00038426351299158148 -▁] 0.000383395964170461 -▁sowie 0.00038317244722528148 -z 0.00037908095060165648 -▁now 0.00037827022812253079 -▁dieses 0.00037799746168095576 -▁make 0.00037741025614700959 -l 0.00037339831640217729 -▁gibt 0.00037294749408901861 -▁many 0.0003703713665852547 -▁hier 0.00037015921490847416 -▁important 0.0003695151830325332 -st 0.00036950002934133457 -▁2 0.00036651854059800785 -▁hotel 0.00036645413741041372 -. 0.00036620410150563665 -▁between 0.00036582147080287174 -r 0.00036152539934806549 -h 0.00036074119582853734 -▁W 0.00035615341581815785 -▁because 0.00035514569535345018 -▁zwischen 0.00035507371532025677 -m 0.00035470244988589084 -at 0.00035463804669829671 -▁N 0.00035365684519318664 -▁he 0.00035290673747885538 -▁take 0.0003528196037544634 -▁ 0.00035048593530987728 -▁un 0.00034889479773402308 -as 0.00034812953632849325 -de 0.00034769765612933283 -▁over 0.00034760294555934152 -▁ab 0.000347019528448195 -ische 0.00034593603952749426 -▁muss 0.00034415548081165748 -▁Parlament 0.00034415169238885779 -▁V 0.00034309851085055431 -d 0.00034082545717076268 -▁Bericht 0.00034081030347956405 -▁Ein 0.00034033296220680782 -▁p 0.0003383895013105859 -▁unsere 0.00033720751339709427 -▁1 0.00033614675501319146 -▁keine 0.00033532845568846648 -f 0.00033465790485292791 -c 0.00033442681106214911 -▁l 0.00033382824025980396 -ie 0.00033376004864941024 -▁„ 0.00033306676727707377 -▁just 0.00033155897500281197 -▁m 0.00033095661577766717 -▁d 0.00033049063977330987 -▁Ihnen 0.00032975568575017723 -os 0.00032974810890457794 -▁two 0.00032824410505311578 -▁b 0.00032818349028832135 -▁t 0.00032799406914833871 -▁information 0.00032786147435035087 -p 0.00032475117923183588 -▁support 0.00032380028510912306 -▁made 0.00032242129921004946 -▁years 0.00032187197790409981 -ation 0.000321553750388929 -▁had 0.00032003459284626819 -▁If 0.00031970121163989873 -ia 0.000318988988153564 -▁anderen 0.00031761000225449039 -ungs 0.00031675760712456853 -▁way 0.00031406025009121576 -▁Europa 0.00031304116435810917 -▁Zeit 0.00031275324422533552 -it 0.00031272672526573797 -▁Im 0.00031232515244897479 -▁those 0.00031115452980388209 -▁An 0.0003107605338327182 -▁Mitgliedstaaten 0.00031019227041277023 -▁world 0.00030959369961042514 -ungen 0.00030657432663910183 -ly 0.00030557797144279314 -▁where 0.000304259600308514 -▁3 0.00030364208739217061 -▁Wenn 0.00030276317330265115 -▁policy 0.0003001605268392897 -us 0.00029900505788539562 -igen 0.00029797839530668968 -▁immer 0.00029762607198632196 -▁dann 0.0002961485870944574 -▁system 0.00029494386864416781 -▁” 0.00029362549750988864 -▁being 0.00029199268728323827 -ern 0.00029102663946932683 -ieren 0.00029088646782573965 -iert 0.00029058718242456713 -den 0.00028888996901032263 -▁Er 0.00028796938227000705 -▁may 0.00028673814486011986 -▁diesen 0.00028641234049934974 -▁damit 0.00028520383362626051 -ige 0.00028451055225392404 -▁good 0.0002834384286016223 -w 0.00028319975796524415 -▁gegen 0.00028274135880648619 -▁wurden 0.00028190790579056259 -▁da 0.00028182077206617055 -▁even 0.00028167681199978378 -▁dies 0.000280888820057456 -g 0.00027835057678168862 -▁5 0.000277877023931732 -▁E 0.00027772169859694628 -▁Frau 0.00027716480044539729 -▁right 0.00027523649324037405 -▁Diese 0.00027388023787809835 -ate 0.00027381204626770458 -▁As 0.00027187237379428235 -ent 0.00027110332396595283 -▁area 0.00027040246574801708 -▁10 0.00026824306475221497 -▁part 0.00026770132029186466 -▁under 0.0002668224062023452 -▁For 0.00026663298506236256 -▁unserer 0.00026662161979396358 -▁order 0.00026452662198575561 -▁Ihre 0.00026443948826136357 -▁r 0.000264053069135799 -▁year 0.00026114734884846534 -▁Menschen 0.00026094656244008373 -▁habe 0.00026071168022650523 -le 0.00026065485388451044 -▁f 0.00026026843475894588 -ur 0.00025865456664629378 -▁sondern 0.00025842347285551497 -▁here 0.00025795749685115767 -▁4 0.00025772261463757922 -▁You 0.00025768473040958269 -▁seine 0.00025733998393481425 -▁jedoch 0.00025683233527966083 -▁market 0.00025633605189290628 -▁Europäische 0.00025486993226944064 -▁through 0.00025458201213666705 -▁place 0.00025317650727799584 -▁c 0.00025292647137321878 -um 0.00025271431969643823 -lich 0.00025267643546844171 -ist 0.00025222182473248338 -▁possible 0.00025163461919853721 -v 0.00025060416819703163 -▁same 0.00025013819219267433 -▁after 0.00024962296669192157 -▁& 0.00024953583296752953 -▁Welt 0.00024771360160089653 -▁neue 0.00024737264354892779 -▁Mit 0.00024724762559653928 -▁see 0.00024627021251422886 -▁heute 0.00024577392912747431 -▁own 0.00024378879558045627 -▁h 0.00024374333450686043 -▁There 0.000243409953300491 -▁Jahr 0.00024340237645489169 -▁political 0.0002428719972629403 -▁could 0.00024123918703628994 -▁want 0.0002393563409048625 -▁then 0.0002392123808384757 -▁within 0.00023903053654409238 -▁development 0.00023898507547049654 -liche 0.00023884111540410974 -▁Rat 0.000237496225310233 -▁Z 0.00023705676826547327 -ens 0.00023604904780076563 -sch 0.00023532545904603194 -lichen 0.0002352572674356382 -▁bereits 0.00023518528740244478 -▁rights 0.00023449579445290798 -▁international 0.00023401087633455244 -▁economic 0.00023374189831577708 -ter 0.00023311302013103471 -▁much 0.00023236291241670348 -▁both 0.00023191966694914411 -▁against 0.00023182874480195243 -▁mich 0.00023179464899675554 -▁Ab 0.00023129457718720138 -re 0.00023113925185241563 -▁used 0.00023113546342961596 -est 0.000230851331719642 -▁find 0.00023070358323045557 -ge 0.00023062781477446251 -il 0.00023023003038049895 -▁therefore 0.00022975647753054236 -▁how 0.00022897606243381388 -C 0.00022876012233423367 -▁Auf 0.00022872602652903681 -▁sollte 0.00022809335992149478 -▁ihrer 0.00022790393878151214 -▁public 0.00022761601864873854 -▁J 0.00022750236596474896 -▁Aus 0.0002274190206631566 -▁neuen 0.00022719171529517743 -der 0.00022707048576558854 -▁ohne 0.00022706290891998923 -▁europäischen 0.00022614611060247327 -▁Jahren 0.00022454360775822014 -▁n 0.0002244337434970302 -ik 0.00022375182739309269 -end 0.00022365332840030173 -▁does 0.00022338435038152637 -S 0.00022296383545076492 -▁andere 0.00022216447824003818 -ra 0.0002219220191808604 -ol 0.00022161136851128887 -im 0.00022131587153291596 -▁country 0.00022111508512453435 -▁zwei 0.00022075518495856734 -id 0.00022022480576661596 -▁ver 0.00021982323294985275 -▁ganz 0.00021969063815186491 -ier 0.00021882308933074442 -isch 0.00021805782792521457 -ien 0.00021801994369721804 -▁without 0.0002179517520868243 -▁still 0.00021741000762647394 -▁Entwicklung 0.00021617498179378713 -▁best 0.00021603102172740033 -▁number 0.00021578098582262324 -▁Sch 0.00021507633918188782 -▁great 0.00021483009169991039 -A 0.00021459899790913158 -ies 0.000214485345225142 -▁sollten 0.00021447776837954269 -▁finden 0.0002138110059668038 -▁me 0.00021370871855121317 -▁denen 0.00021362916167242047 -▁Frage 0.00021358748902162427 -▁different 0.00021328441519765206 -▁fact 0.00021275782442850032 -▁services 0.0002124547506045281 -▁wieder 0.00021221986839094963 -▁point 0.0002114280880258222 -▁social 0.00021109470681945273 -able 0.00021083709406907635 -j 0.00021065146135189336 -▁last 0.00021062873081509546 -▁free 0.00020841629190009823 -▁available 0.00020823065918291523 -ad 0.00020818898653211906 -▁st 0.00020784424005735065 -ant 0.00020618491087110274 -ir 0.00020607883503271247 -ch 0.00020606368134151384 -▁bietet 0.00020584774124193363 -▁view 0.00020555982110916004 -▁Vor 0.00020396110668770656 -ine 0.00020374516658812635 -ot 0.0002031958452821767 -▁say 0.00020283594511620969 -▁room 0.00020262379343942912 -▁So 0.00020247604495024266 -▁w 0.00020241164176264858 -T 0.00020219570166306837 -▁per 0.00020141149814354022 -▁Land 0.00020133572968754717 -▁selbst 0.00020131678757354891 -ian 0.00020101750217237633 -▁dazu 0.00020101371374957669 -▁Ver 0.00020079398522719684 -▁20 0.00020068790938880654 -▁la 0.00020055910301361836 -▁k 0.00020009312700926106 -▁liegt 0.00020006660804966351 -ab 0.00019964988154170168 -▁able 0.00019940363405972426 -▁get 0.00019939605721412494 -▁end 0.00019866110319099231 -▁St 0.00019848304731940862 -▁machen 0.00019843758624581281 -▁allen 0.00019835045252142078 -▁know 0.0001980284365834503 -▁viele 0.0001977405164506767 -▁already 0.00019761928692108781 -▁her 0.00019750563423709822 -▁future 0.00019730105940591697 -▁believe 0.00019680477601916244 -▁weil 0.00019664566226157702 -▁Unternehmen 0.0001966077780335805 -▁before 0.00019638804951120064 -am 0.00019536896377809405 -▁That 0.00019410741898580967 -▁O 0.00019338004180827635 -M 0.00019304666060190688 -▁today 0.00019291406580391904 -▁All 0.00019265645305354266 -▁course 0.00019254658879235272 -▁denn 0.00019229276446477598 -▁human 0.0001922586686595791 -▁too 0.00019191771060761035 -▁jetzt 0.00019174344315882634 -2 0.00019141763879805619 -ion 0.00019115623762488014 -ut 0.00019111456497408397 -▁process 0.00019085695222370759 -▁products 0.00019037961095095133 -▁service 0.00018986817387299821 -ere 0.00018911427773586731 -▁seiner 0.00018817853730435306 -▁darauf 0.0001877655992191909 -▁Arbeit 0.00018727310425523604 -▁think 0.00018717081683964541 -▁national 0.00018704958731005652 -▁cannot 0.00018685258932447458 -▁Lage 0.00018639040174291694 -▁data 0.00018575773513537493 -▁geht 0.00018545844973420235 -▁ob 0.00018531070124501591 -B 0.00018504551164904022 -▁schon 0.00018475380309346696 -▁But 0.00018459847775868118 -▁Maßnahmen 0.00018402263749313396 -▁Informationen 0.00018392792692314264 -▁set 0.00018392035007754333 -P 0.000183378605617193 -▁offers 0.000182548941024069 -▁case 0.00018220040612650096 -om 0.00018203371552331625 -ul 0.00018191248599372736 -▁dafür 0.00018085930445542388 -é 0.00018037817475986798 -▁einige 0.00018035165580027041 -▁Bereich 0.00017975687342072492 -▁2009 0.00017958639439474055 -un 0.00017946137644235202 -▁question 0.00017935530060396172 -▁6 0.0001790522267799895 -▁Commissioner 0.00017886659406280651 -▁Um 0.00017859382762123151 -▁city 0.00017841956017244749 -▁high 0.00017840819490404854 -▁15 0.000178370310676052 -▁Internet 0.00017790054624889506 -▁Nach 0.00017780962410170339 -ated 0.00017713528484336521 -▁Dies 0.0001763169855186402 -D 0.00017616544860665409 -▁en 0.00017590783585627769 -▁g 0.00017581312528628637 -▁Herrn 0.00017575629894429158 -▁level 0.00017542291773792214 -▁small 0.00017521076606114158 -▁during 0.00017500240280716068 -▁30 0.00017499103753876172 -1 0.00017485844274077388 -ro 0.0001748091932443784 -▁ihr 0.00017450990784320582 -▁bin 0.00017418410348243567 -▁wo 0.00017391133704086067 -ations 0.00017383935700766728 -▁ihren 0.00017319911355452596 -▁mir 0.00017317259459492839 -▁allem 0.00017309682613893534 -▁viel 0.00017277102177816518 -▁v 0.00017245279426299435 -ver 0.00017211562463382527 -▁e 0.00017195272245344018 -▁said 0.00017178224342745581 -▁financial 0.00017155872648227629 -▁debate 0.00017131247900029886 -▁Jahre 0.00017130490215469958 -▁real 0.00017083513772754263 -▁issue 0.00017081619561354436 -▁means 0.00017047144913877595 -▁every 0.00017030097011279158 -I 0.00017026308588479505 -▁At 0.0001702517206163961 -▁day 0.00017016837531480373 -▁Für 0.0001700433573624152 -▁example 0.00016994106994682457 -▁quality 0.00016984635937683325 -▁| 0.00016972512984724436 -▁proposal 0.00016948645921086622 -▁Stadt 0.00016946751709686795 -▁meine 0.00016914550115889747 -▁business 0.000169096251662502 -▁clear 0.00016908109797130337 -▁access 0.00016869467884573879 -▁including 0.00016861133354414645 -▁long 0.0001684938924373572 -▁rooms 0.00016765665099863393 -▁Eine 0.00016756194042864261 -▁On 0.00016756194042864261 -▁Zimmer 0.00016745207616745267 -F 0.00016736494244306066 -▁since 0.00016733084663786381 -iv 0.00016721340553107455 -▁Our 0.00016653148942713706 -▁However 0.00016641404832034781 -▁help 0.0001661753776839697 -▁unseren 0.00016585715016879886 -E 0.0001657169785252117 -▁situation 0.00016567530587441553 -3 0.000165637421646419 -▁wollen 0.00016540253943284053 -▁Teil 0.0001652358488296558 -▁offer 0.00016505779295807212 -x 0.00016399703457416936 -▁Art 0.00016385307450778253 -op 0.00016363334598540268 -▁each 0.00016304235202865684 -▁würde 0.00016294006461306621 -iger 0.00016187551780636378 -ive 0.00016165957770678357 -ischer 0.00016153834817719467 -▁particular 0.00016149288710359886 -▁back 0.00016143984918440371 -▁gut 0.00016105721848163877 -▁Daten 0.00015956457989857559 -▁Wie 0.00015910239231701795 -and 0.0001590493543978228 -▁Rahmen 0.00015883720272104226 -▁Und 0.00015852655205147073 -▁Parlaments 0.000158446995172678 -▁etwas 0.00015837501513948462 -▁während 0.00015837501513948462 -▁Committee 0.00015816286346270406 -▁Ziel 0.00015804163393311516 -▁möglich 0.00015780296329673705 -▁nun 0.00015754535054636067 -ag 0.00015741654417117246 -▁three 0.00015739002521157489 -▁common 0.00015692404920721761 -▁next 0.000156810396523228 -▁form 0.00015680660810042836 -▁sagen 0.00015679524283202941 -▁problem 0.00015663234065164434 -▁far 0.0001563709394784683 -ings 0.00015615499937888809 -▁areas 0.00015613226884209015 -ors 0.00015590875189691066 -▁doch 0.00015581025290411967 -▁He 0.00015557158226774156 -▁measures 0.00015515106733698009 -▁seit 0.00015498437673379538 -▁8 0.00015479495559381272 -▁12 0.00015473812925181793 -▁few 0.00015460174603103044 -▁give 0.00015449945861543981 -▁7 0.000153809965665903 -ell 0.00015357887187512418 -▁And 0.00015343112338593774 -▁18 0.00015329852858794989 -▁energy 0.0001531128958707669 -ap 0.00015310153060236795 -ierte 0.0001527757262415978 -▁provide 0.00015266965040320753 -age 0.00015247644084042522 -▁� 0.00015224534704964642 -▁Länder 0.000151999099567669 -▁insbesondere 0.00015183998581008357 -ale 0.00015176042893129085 -ions 0.00015150660460371411 -▁citizens 0.00015149145091251551 -▁taken 0.00015135885611452767 -▁To 0.00015127172239013563 -▁erhalten 0.00015115049286054674 -G 0.00015079816954017904 -ure 0.00015070345897018773 -▁agreement 0.00015051782625300473 -ip 0.00015043069252861272 -O 0.00015020338716063356 -▁weiter 0.00014998365863825371 -▁China 0.00014968437323708113 -L 0.00014961239320388772 -▁Al 0.00014941539521830577 -▁put 0.00014924870461512107 -▁again 0.00014854026955158598 -▁waren 0.00014838115579400056 -▁large 0.00014806671670162939 -ity 0.00014778637341445508 -▁special 0.00014724084053130507 -gen 0.00014714612996131375 -▁ensure 0.00014713097627011516 --S 0.00014710824573331723 -▁further 0.00014670667291655405 -4 0.00014668394237975612 -▁Un 0.00014647936754857489 -▁beim 0.00014606264104061306 -▁Bürger 0.0001459641420478221 -▁go 0.00014593762308822453 -▁Grund 0.00014584291251823321 -ug 0.00014583912409543357 -▁come 0.0001454830123522662 -ow 0.00014539587862787416 -▁re 0.00014533905228587937 -▁soll 0.00014473290463793494 -▁du 0.00014455484876635125 -▁Fall 0.00014454727192075194 -▁ersten 0.00014430860128437383 -▁problems 0.00014397900850080404 -▁company 0.00014388050950801308 -▁modern 0.00014382747158881793 -▁Kunden 0.00014369108836803041 -▁einfach 0.00014366456940843287 -▁ins 0.00014360395464363841 -▁Zusammenarbeit 0.00014334255347046236 -eln 0.00014333497662486307 -ete 0.00014331982293366445 -▁tun 0.00014289551958010334 -▁shall 0.00014286142377490646 -▁lassen 0.00014283111639250924 -▁necessary 0.00014281975112411029 -▁position 0.00014268715632612244 -▁Unterstützung 0.00014266442578932451 -ise 0.00014234998669695334 -▁change 0.00014231967931455612 -R 0.00014206585498697938 -▁Group 0.00014197872126258738 -▁security 0.00014197872126258738 -bar 0.0001418650685785978 -7 0.00014182718435060127 -▁given 0.00014141045784263944 -▁geben 0.00014135363150064465 -iz 0.00014089523234188668 -▁pro 0.00014084977126829084 -▁why 0.00014078157965789707 -▁Recht 0.00014067171539670716 -▁however 0.00014051260163912174 -▁Vorschlag 0.00014042925633752938 -od 0.00014026256573434464 -▁down 0.00013943290114122069 -ierten 0.00013937228637642626 -▁einmal 0.00013920938419604119 -▁vote 0.00013910709678045056 -▁besteht 0.00013899723251926062 -▁located 0.00013898586725086167 -▁while 0.00013893282933166652 -▁Problem 0.00013878508084248008 -▁hand 0.000138709312386487 -▁cooperation 0.00013861460181649568 -▁2010 0.000138535044937703 -▁2008 0.00013839487329411583 -ak 0.00013837214275731793 -▁stehen 0.00013825849007332834 -▁sch 0.00013817893319453562 -▁stay 0.00013817893319453562 -5 0.00013780766776016966 -▁Bei 0.00013779630249177071 -▁group 0.00013770538034457903 -▁main 0.00013761066977458771 -ang 0.00013719773168942555 -▁wichtig 0.00013717878957542729 -▁Sicherheit 0.00013706892531423735 -▁With 0.00013694390736184882 -▁drei 0.00013678100518146376 -sten 0.00013674690937626687 -▁always 0.00013648171978029118 -▁System 0.00013640973974709777 -▁US 0.00013628472179470923 -▁needs 0.00013625062598951235 -ierung 0.00013619758807031723 -▁Ländern 0.00013607635854072833 -▁steht 0.0001357657078711568 -▁life 0.00013575813102555749 -▁They 0.00013572024679756097 -▁Namen 0.00013560280569077174 -▁sehen 0.00013556492146277522 -▁law 0.00013536034663159396 -▁hope 0.00013523911710200506 -▁Fragen 0.0001350269654252245 -▁seinen 0.00013486027482203979 -▁Euro 0.0001346102389172627 -▁better 0.00013353811526496095 -▁result 0.00013296985184501305 -▁wäre 0.00013271223909463667 -▁legal 0.00013255691375985089 -▁United 0.00013251902953185436 -▁Ch 0.00013246220318985957 -▁50 0.00013232203154627241 -▁dabei 0.00013218564832548493 -▁protection 0.00013215155252028804 -ste 0.00013206063037309637 -▁Was 0.0001319545545347061 -▁full 0.00013195076611190645 -▁House 0.00013189393976991166 -8 0.00013183711342791687 -▁present 0.00013140523322875645 -▁action 0.00013137492584635923 -enden 0.00013127642685356825 -▁Kollegen 0.00013116656259237833 -▁zusammen 0.00013090516141920229 -▁using 0.00013088243088240435 -▁letzten 0.00012994290202809048 -▁What 0.00012987092199489707 -▁open 0.00012966255874091617 -ue 0.00012960952082172102 -▁100 0.00012951102182893006 -▁USA 0.00012944661864133596 -▁worden 0.00012929887015214949 -▁did 0.00012923446696455539 -▁Am 0.0001290336805561738 -tr 0.00012889729733538632 -▁ge 0.00012886320153018943 -N 0.0001288253173021929 -ert 0.00012849572451862311 -▁around 0.00012842753290822937 -▁Community 0.00012841237921703075 -▁based 0.00012839343710303248 -ck 0.00012825326545944533 -ates 0.00012764332938870122 -ac 0.00012751452301351305 --B 0.00012734783241032831 -ism 0.00012709021965995193 -▁große 0.0001270447585863561 -▁Verfügung 0.00012695383643916442 -▁großen 0.00012695383643916442 -▁Ihr 0.0001269007985199693 -io 0.00012669622368878804 -▁welche 0.00012652953308560331 -▁working 0.00012647270674360852 -▁current 0.00012637041932801789 -▁deren 0.00012610144130924256 -▁Sp 0.00012580215590806998 -oc 0.00012574911798887483 -▁power 0.00012565819584168318 -▁Seite 0.00012564304215048456 -▁Bedeutung 0.00012560894634528768 -6 0.00012533239148091303 -▁terms 0.00012529071883011686 -▁production 0.00012502552923414115 -▁muß 0.00012486641547655575 -els 0.00012463153326297728 -licher 0.00012461259114897902 -▁aller 0.00012460880272617935 -▁become 0.00012455576480698423 -▁These 0.00012453682269298596 -▁centre 0.00012452545742458698 -▁whether 0.00012450651531058872 -▁Mrs 0.00012436255524420192 -▁particularly 0.0001243322478618047 -▁Aber 0.00012392309819944219 -▁Markt 0.00012386248343464776 -▁Möglichkeit 0.00012386248343464776 -▁zurück 0.00012360487068427138 -▁together 0.00012337377689349254 -W 0.00012335483477949428 -H 0.00012333968108829566 -ating 0.00012315783679391233 -▁* 0.00012301008830472587 -ke 0.00012286612823833907 -▁When 0.00012283582085594185 -▁another 0.00012283582085594185 -▁local 0.00012267670709835643 -▁Auch 0.00012254790072316823 -▁Als 0.00012254032387756895 -▁km 0.00012206298260481269 -▁trade 0.0001218129467000356 -th 0.00012169171717044671 -ess 0.00012162731398285261 -▁water 0.00012153260341286129 -▁second 0.00012124089485728803 -K 0.0001211310305960981 -ets 0.00012083174519492552 -▁children 0.00012080901465812762 -ill 0.00012040365341856476 -▁Politik 0.000120248328083779 -eten 0.00011994146583700712 -▁Mittel 0.00011974825627422483 -U 0.00011974446785142517 -▁issues 0.00011966491097263248 -ts 0.00011958914251663941 -▁bieten 0.000119430028759054 -▁2000 0.00011937320241705921 -▁Unter 0.00011934289503466198 -▁German 0.00011924060761907135 -▁davon 0.00011920651181387448 -▁Thema 0.0001191648391630783 -▁2007 0.00011849807675033942 -ard 0.0001184109430259474 -▁Kommissar 0.00011831623245595608 -▁conditions 0.00011814954185277136 -▁darüber 0.00011814196500717205 -ud 0.0001179942165179856 -▁daher 0.00011797906282678699 -ay 0.00011782373749200122 -▁alles 0.00011751687524522934 -▁stellen 0.00011747520259443317 -▁kein 0.00011731608883684775 -▁Weg 0.00011727820460885122 -▁certain 0.00011717591719326059 -▁gegenüber 0.00011702816870407414 -he 0.00011694103497968212 -▁Über 0.00011686905494648872 -▁Probleme 0.00011686147810088941 -▁systems 0.00011685011283249046 -ber 0.00011667205696090677 -▁Kinder 0.0001166455380013092 -▁Frauen 0.0001165735579681158 -▁eigenen 0.00011635382944573593 -▁experience 0.00011624396518454601 -▁page 0.00011617198515135261 -▁Beispiel 0.00011612652407775677 -▁various 0.0001160280250849658 -▁global 0.00011598256401136997 -▁Weise 0.00011589543028697795 -▁role 0.00011582723867658421 -▁16 0.0001157552586433908 -▁health 0.00011574389337499184 -▁sowohl 0.00011572495126099358 -▁11 0.00011563402911380192 -▁women 0.00011543703112821997 -▁befindet 0.00011541051216862239 -▁whole 0.00011541051216862239 -▁25 0.00011539914690022344 -▁weitere 0.00011534232055822865 --M 0.0001152665521022356 -▁Regierung 0.00011524003314263802 -▁companies 0.00011520214891464149 -▁kommen 0.00011519457206904218 -▁industry 0.00011511501519024947 -ok 0.00011498620881506128 -▁Man 0.0001149369593186658 -▁Le 0.00011476648029268142 -▁respect 0.00011468313499108907 --T 0.00011458463599829809 -▁Bar 0.00011458463599829809 -▁sector 0.00011451265596510468 -▁politischen 0.00011450886754230504 -▁Ihrer 0.00011447477173710815 -▁basis 0.00011447477173710815 -▁Zukunft 0.00011434975378471962 -▁Schutz 0.00011434217693912031 -▁New 0.0001143270232479217 -▁9 0.00011430429271112378 -▁könnte 0.00011425125479192865 -▁erreichen 0.00011422094740953142 -▁project 0.00011411487157114114 -▁sp 0.00011409592945714288 -▁unser 0.00011403531469234843 -▁wissen 0.00011402394942394948 -▁Tr 0.00011400879573275086 -▁state 0.00011399364204155225 -▁Ende 0.00011396333465915504 -▁especially 0.00011395196939075607 -▁Hier 0.00011389135462596162 -▁car 0.00011380422090156962 -▁really 0.00011376254825077343 -▁account 0.00011368677979478038 -▁Service 0.00011364131872118454 -▁natürlich 0.00011359206922478906 -▁No 0.00011358070395639009 -▁control 0.00011356933868799114 -her 0.00011350493550039704 -be 0.00011347462811799982 -▁meiner 0.00011341780177600503 -▁research 0.00011338370597080816 -▁wirklich 0.00011325489959561996 -ne 0.00011323974590442135 -▁matter 0.00011319807325362517 -▁besonders 0.00011306547845563732 -▁budget 0.00011298213315404496 -▁hatte 0.00011297834473124531 -▁unterstützen 0.00011285711520165642 -9 0.00011281544255086024 -▁close 0.00011277755832286371 -▁staff 0.0001124896381900901 -▁+ 0.00011186454842814739 -ä 0.00011175847258975712 -ov 0.0001114705524569835 -▁EUR 0.00011142887980618733 -▁done 0.00011133795765899565 -▁government 0.00011123945866620468 -▁ihnen 0.00011113717125061406 -▁hin 0.0001111258059822151 -▁verschiedenen 0.00011099321118422725 -▁entfernt 0.0001108681932318387 -▁Staaten 0.00011083030900384217 -▁Tag 0.00011074317527945017 -▁z 0.00011030750665749009 -▁Haus 0.00011008777813511024 -aus 0.00011000443283351787 -ical 0.00010999306756511892 -V 0.000109970337028321 -▁private 0.00010995139491432274 -▁create 0.00010992866437752482 -▁Seiten 0.00010981501169353524 -▁innerhalb 0.00010980364642513627 -▁La 0.00010978849273393766 -▁location 0.00010972030112354392 -▁look 0.00010963695582195155 -▁_ 0.00010958012947995676 -▁transport 0.0001093604009575769 -▁Mar 0.00010932251672958037 -▁vielen 0.00010920128719999148 -▁region 0.00010911036505279981 -▁dort 0.00010891715549001752 -og 0.00010864438904844252 -ics 0.00010861029324324565 -▁site 0.00010851179425045468 -▁direkt 0.0001084625447540592 -eg 0.00010816325935288662 -▁Programm 0.00010813674039328905 -▁Germany 0.00010803445297769843 -▁major 0.00010801172244090051 -▁Re 0.00010797762663570364 -▁X 0.00010792837713930815 -▁Rolle 0.00010768970650293002 -ment 0.00010766697596613211 -art 0.00010761393804693696 -▁pr 0.00010736390214215988 -ein 0.00010729571053176614 -se 0.00010727297999496822 --K 0.00010712144308298211 -▁single 0.000106969906170996 -▁Umwelt 0.00010687898402380432 -▁U 0.00010660242915942968 -▁zwar 0.00010656833335423279 -ben 0.00010653044912623627 -▁old 0.00010646983436144182 -▁specific 0.0001064395269790446 -ans 0.000106424373287846 -▁million 0.00010641679644224669 -▁internationalen 0.00010641300801944703 -▁Br 0.00010636375852305155 -all 0.00010624252899346266 -▁möchten 0.00010606826154467863 -▁something 0.00010600764677988418 -ib 0.00010584095617669947 -land 0.00010572351506991023 -▁itself 0.00010553409392992759 -▁several 0.00010544317178273592 -▁start 0.00010519692430075849 -▁personal 0.00010516661691836127 -▁off 0.0001050681179255703 -▁rapporteur 0.00010494688839598141 -▁Ihren 0.0001049393115503821 -ins 0.00010472715987360154 -▁general 0.00010470821775960328 -▁Alle 0.00010467033353160675 -▁Uhr 0.00010442787447242898 -▁less 0.00010414374276245501 -▁subject 0.00010407933957486092 -▁continue 0.00010393916793127377 --s 0.00010393537950847411 -up 0.00010392780266287481 -ator 0.00010386339947528071 -▁making 0.00010374595836849148 -▁2006 0.00010354896038290953 -▁Richtlinie 0.00010348834561811508 -▁programme 0.00010343151927612029 -ah 0.0001033746929341255 -▁excellent 0.0001033746929341255 -▁resolution 0.00010329513605533279 -▁third 0.00010325346340453661 -▁wish 0.00010298448538576126 -▁gentlemen 0.00010268141156178904 -▁often 0.00010243516407981162 -▁least 0.00010240106827461473 -▁Ar 0.00010231014612742307 -elle 0.00010230256928182376 -heit 0.00010221543555743175 -ass 0.00010217755132943522 -▁away 0.00010198434176665293 -▁le 0.0001019540343842557 -▁name 0.00010175703639867377 -▁14 0.00010170021005667898 -▁going 0.00010151457733949598 -▁Hotels 0.00010138198254150814 -▁Zu 0.0001013365214679123 -▁following 0.00010127211828031821 -▁management 0.00010120392666992446 -▁Form 0.00010116225401912827 -▁ch 0.00010093873707394877 -▁erste 0.00010093494865114911 -▁regard 0.00010084023808115779 -▁europäische 0.0001007909885847623 -▁Software 0.00010068491274637203 -ial 0.00010064324009557585 -▁individual 0.00010054852952558453 -▁directive 0.00010053716425718557 -▁Wirtschaft 0.00010045381895559321 -ina 0.00010041972315039634 -▁17 0.00010038941576799911 -▁rules 0.00010027197466120987 -ea 0.00010018484093681786 -ner 0.00010013559144042237 -▁crisis 0.00010011664932642411 -▁World 0.00010005224613883002 -▁short 9.9984054528436261e-05 -if 9.9961323991638342e-05 -.com 9.9904497649643551e-05 -▁Artikel 9.9749172314857787e-05 -ER 9.9722653355260226e-05 -▁software 9.95483859064762e-05 -▁Y 9.94915595644814e-05 --F 9.9461252182084177e-05 -▁environment 9.9430944799686961e-05 -ons 9.929456157889946e-05 -▁ask 9.9245312082503966e-05 -ce 9.92263699685057e-05 -▁hinaus 9.92112162773071e-05 -▁online 9.9169543626510912e-05 -▁food 9.9146813089712993e-05 -▁money 9.9124082552915074e-05 -▁führen 9.910514043891681e-05 -▁found 9.9071044633719938e-05 -▁Meinung 9.9040737251322722e-05 -▁Da 9.89536035269307e-05 -▁forward 9.8930872990132782e-05 -▁welcome 9.8923296144533485e-05 -als 9.8915719298934174e-05 -▁once 9.8885411916536958e-05 -▁price 9.8707356044953274e-05 -▁product 9.8620222320561254e-05 -▁towards 9.86088570521623e-05 -▁Fl 9.8593703360963693e-05 -ile 9.8563395978566463e-05 -ach 9.8548242287367855e-05 -iges 9.8442166448977584e-05 -sp 9.8252745308994934e-05 -▁13 9.8245168463395637e-05 -▁little 9.8230014772197029e-05 -▁weit 9.8104996819808481e-05 -ende 9.8101208397008825e-05 -▁range 9.79989209814182e-05 -▁period 9.7775404036238688e-05 -▁thank 9.7438234407069585e-05 -▁things 9.7370042796675842e-05 -▁until 9.7123795314698415e-05 -la 9.7085911086701888e-05 -au 9.704802685870536e-05 -▁family 9.7013931053508475e-05 -▁Leben 9.6930585751916123e-05 -▁ent 9.6858605718722711e-05 -▁institutions 9.6858605718722711e-05 -▁Zusammenhang 9.6775260417130359e-05 -▁Situation 9.6767683571531048e-05 -ub 9.6763895148731393e-05 -▁o 9.6676761424339386e-05 -▁unserem 9.6646454041942156e-05 -▁home 9.6612358236745284e-05 -▁beiden 9.645703290195952e-05 -ous 9.6430513942361945e-05 -▁Madam 9.64267255195623e-05 -ort 9.6271400184776539e-05 -▁Be 9.6188054883184174e-05 -▁customers 9.61615359235866e-05 -▁kommt 9.6085767467593545e-05 -Y 9.5782693643621327e-05 -▁days 9.5733444147225833e-05 -▁Personen 9.5661464114032434e-05 -▁air 9.5468254551250143e-05 -ite 9.54000629408564e-05 -ze 9.5051528043288343e-05 -▁sicher 9.5028797506490424e-05 -ler 9.4918933245300484e-05 -qu 9.4865895326105349e-05 -▁aid 9.484316478930743e-05 -▁design 9.4839376366507788e-05 -▁Treaty 9.46688973405234e-05 -▁fast 9.4631013112526875e-05 -▁progress 9.4456745663742847e-05 -ger 9.44453803953439e-05 -▁increase 9.4426438281345631e-05 -▁him 9.4335516134153968e-05 -▁approach 9.4301420328957083e-05 -▁Dieses 9.4278689792159164e-05 -ost 9.4244593986962292e-05 -pl 9.4225651872964028e-05 -▁opportunity 9.4206709758965765e-05 -ement 9.4191556067767156e-05 -▁24 9.41006339205755e-05 -hr 9.4036230732981392e-05 -ob 9.3941520162990074e-05 -á 9.3850598015798411e-05 -ary 9.383544432459979e-05 -▁call 9.3702849526611944e-05 -▁growth 9.3630869493418545e-05 -ast 9.3623292647819235e-05 -X 9.3592985265422018e-05 -one 9.3558889460225146e-05 -▁'t 9.3532370500627572e-05 -▁Rechts 9.34982746954307e-05 -är 9.34338715078366e-05 -▁® 9.34338715078366e-05 -▁rather 9.3426294662237288e-05 -▁00 9.3395987279840071e-05 -▁Hilfe 9.33618914746432e-05 -▁International 9.333158409224597e-05 -▁y 9.3214142985456732e-05 -▁quite 9.31762587574602e-05 -▁activities 9.3100490301467151e-05 -▁San 9.30398755366727e-05 -▁agree 9.2998202885876535e-05 -ide 9.296410708067965e-05 -▁stellt 9.296410708067965e-05 -▁Mal 9.2945164966681386e-05 -▁play 9.2930011275482778e-05 -uch 9.2926222852683123e-05 -ance 9.2748166981099439e-05 -▁Members 9.2717859598702223e-05 -▁Region 9.2672398525106385e-05 -▁ja 9.2445093157127222e-05 -▁Lösung 9.2399632083531384e-05 -▁etwa 9.2361747855534856e-05 -▁positive 9.2327652050337985e-05 -▁Präsidentin 9.2304921513540066e-05 -▁bringen 9.2229153057547011e-05 -▁list 9.2024578226365767e-05 -▁2005 9.1914713965175827e-05 -are 9.1895771851177563e-05 -▁jeder 9.1725292825193191e-05 --P 9.1710139133994583e-05 -▁provided 9.1679831751597366e-05 -▁weniger 9.1649524369200137e-05 -ick 9.1630582255201873e-05 -▁bzw. 9.1300989471632081e-05 -▁resources 9.1229009438438682e-05 -▁allow 9.1210067324440418e-05 -▁fully 9.1179759942043188e-05 -chen 9.1145664136846316e-05 -▁40 9.1054741989654654e-05 -▁might 9.1016857761658126e-05 -▁daran 9.1013069338858471e-05 -▁line 9.0994127224860207e-05 -▁gilt 9.095624299686368e-05 -▁Deshalb 9.0941089305665072e-05 -▁i 9.0899416654868889e-05 -ec 9.0853955581273051e-05 -▁Bank 9.0744091320083125e-05 -▁Durch 9.0664534441290415e-05 -▁bitte 9.05698238712991e-05 -▁del 9.0478901724107421e-05 -▁j 9.0331153234920968e-05 -▁indem 9.0255384778927913e-05 -fl 9.0228865819330339e-05 -▁value 9.0001560451351175e-05 -ied 8.9880330921762283e-05 -▁gesagt 8.9880330921762283e-05 -▁favour 8.986896565336333e-05 -▁El 8.9857600384964377e-05 -▁decision 8.9796985620169931e-05 -zen 8.9641660285384167e-05 -▁yet 8.9592410788988673e-05 -▁Berlin 8.95583149837918e-05 -ys 8.9482546527798747e-05 -▁Kosten 8.9478758104999091e-05 -▁sea 8.9372682266608821e-05 -▁include 8.93347980386123e-05 -▁September 8.9262818005418881e-05 -▁After 8.9247664314220273e-05 -▁town 8.9190837972225482e-05 -▁interest 8.9092338979434508e-05 -▁technology 8.9016570523441454e-05 -▁State 8.8982474718244582e-05 -▁nämlich 8.891807153065048e-05 -▁left 8.8857456765856034e-05 -▁ladies 8.8838514651857771e-05 -▁sogar 8.8724861967868189e-05 -▁fest 8.8721073545068534e-05 -its 8.862257455227756e-05 -00 8.8588478747080688e-05 -▁se 8.8580901901481377e-05 -▁field 8.8478614485890762e-05 -▁having 8.8425576566695613e-05 -▁reason 8.8421788143895971e-05 -▁contact 8.8402846029897708e-05 -▁Produkte 8.8383903915899444e-05 -▁ideal 8.836496180190118e-05 -aten 8.8327077573904653e-05 -▁Millionen 8.8171752239118889e-05 -▁Europas 8.80580995551293e-05 -▁proposals 8.8020215327132766e-05 -▁text 8.7963388985137976e-05 -▁amendments 8.7808063650352211e-05 -igkeit 8.7607277241970622e-05 -▁solution 8.7588335127972359e-05 -▁al 8.7482259289582074e-05 -▁times 8.74557403299845e-05 -▁nehmen 8.7429221370386939e-05 -▁among 8.7425432947587284e-05 -▁Gemeinschaft 8.7398913987989709e-05 -▁Bild 8.7364818182792837e-05 -▁members 8.735724133719354e-05 -▁di 8.7315568686397357e-05 -▁easy 8.7281472881200472e-05 -igung 8.7273896035601175e-05 -ised 8.7152666506012282e-05 -▁Punkt 8.7141301237613329e-05 -▁2004 8.7137512814813674e-05 -▁minutes 8.7095840164017491e-05 -▁et 8.703143697642339e-05 -EN 8.7004918016825829e-05 -▁safety 8.69746106344286e-05 -▁framework 8.6883688487236936e-05 -▁show 8.6815496876843193e-05 -igt 8.6690478924454645e-05 -▁Rates 8.6671536810456381e-05 -▁costs 8.6648806273658462e-05 -ve 8.6573037817665408e-05 -our 8.6516211475670617e-05 -▁economy 8.6345732449686244e-05 -▁above 8.6197983960499777e-05 -men 8.6144946041304642e-05 -ö 8.6133580772905689e-05 -▁ebenfalls 8.5936582787323742e-05 -▁Web 8.5807776412135552e-05 -▁politische 8.5701700573745268e-05 -▁ihn 8.5694123728145957e-05 -▁adopted 8.56107784265536e-05 -▁Unsere 8.560699000375395e-05 -▁content 8.5535009970560551e-05 -▁Dieser 8.552743312496124e-05 -▁taking 8.5512279433762632e-05 -▁One 8.5413780440971658e-05 -▁Deutschland 8.5391049904173739e-05 -br 8.5281185642983813e-05 -▁enjoy 8.5197840341391448e-05 -▁gemacht 8.5197840341391448e-05 -iss 8.5194051918591792e-05 -▁makes 8.5175109804593529e-05 -▁schnell 8.5053880275004636e-05 -▁addition 8.4807632793027209e-05 -▁called 8.4788690679028946e-05 -▁seinem 8.4750806451032418e-05 -isches 8.4686403263438317e-05 -▁ihm 8.4572750579448735e-05 -▁UN 8.4557596888250127e-05 -zeit 8.4515924237453944e-05 -▁behalf 8.4478040009457417e-05 -▁Preis 8.4398483130664707e-05 -ett 8.43833294394661e-05 -▁darf 8.4315137829072355e-05 -▁website 8.4307560983473044e-05 -▁gehen 8.4296195715074092e-05 -▁opinion 8.4220427259081024e-05 -▁Gr 8.4209061990682071e-05 --L 8.4163600917086233e-05 -▁Aussprache 8.4102986152291787e-05 -▁Geld 8.4102986152291787e-05 -▁Bezug 8.4046159810297e-05 -▁dessen 8.4008275582300469e-05 -▁Ort 8.3932507126307414e-05 -▁Sh 8.39173534351088e-05 -▁others 8.3818854442317832e-05 -▁unterstützt 8.3799912328319569e-05 -ore 8.3792335482720272e-05 -▁fl 8.3754451254723731e-05 -▁legislation 8.3652163839133115e-05 -▁provides 8.3405916357155675e-05 -▁genau 8.3402127934356033e-05 -▁difficult 8.33566668607602e-05 -▁she 8.3318782632763668e-05 -▁klar 8.3314994209964012e-05 -▁bar 8.32998405187654e-05 -▁gute 8.3250591022369911e-05 -▁application 8.3243014176770613e-05 -▁past 8.3133149915580687e-05 -▁results 8.3129361492781032e-05 -▁key 8.3110419378782768e-05 -▁building 8.2992978271993531e-05 -▁Fraktion 8.2943728775598037e-05 -▁standard 8.2795980286411584e-05 -elt 8.2792191863611929e-05 -▁points 8.2758096058415057e-05 -▁projects 8.2735365521617138e-05 -ium 8.2731577098817482e-05 -▁darin 8.2485329616840055e-05 -▁2003 8.2428503274845264e-05 -▁attention 8.2409561160847e-05 -▁beautiful 8.2360311664451507e-05 -▁Haupt 8.2288331631258108e-05 -▁due 8.2269389517259845e-05 -▁Rechte 8.224287055766227e-05 -▁erst 8.2167102101669216e-05 -liches 8.1954950424888661e-05 -▁Kultur 8.18791819688956e-05 -ich 8.1826144049700471e-05 -▁concerned 8.1814778781301518e-05 -▁gerade 8.1795836667303254e-05 -ismus 8.1731433479709153e-05 -▁kind 8.1704914520111578e-05 -▁glaube 8.1602627104520949e-05 -▁dar 8.1587473413322341e-05 -▁responsibility 8.1553377608125469e-05 -▁person 8.1488974420531368e-05 --D 8.1477609152132415e-05 -▁sh 8.14700323065331e-05 -▁policies 8.14473017697352e-05 -▁simply 8.1435936501336232e-05 -▁con 8.1432148078536577e-05 -▁Qualität 8.139426385054005e-05 -▁dürfen 8.1337437508545259e-05 -me 8.1250303784153252e-05 -▁öffentlichen 8.1238938515754285e-05 -▁nichts 8.1019209993374433e-05 -▁Website 8.0969960496978939e-05 -▁einigen 8.0939653114581723e-05 -▁TV 8.09244994233831e-05 -▁questions 8.09017688865852e-05 -▁risk 8.0651732981808113e-05 -▁groups 8.0466100264625119e-05 -▁heart 8.0458523419025822e-05 -▁importance 8.0409273922630328e-05 -▁Restaurant 8.0333505466637274e-05 -▁jeden 8.0197122245849773e-05 -▁schaffen 8.0121353789856719e-05 -▁brauchen 8.0094834830259144e-05 -▁light 8.001906637426609e-05 -▁Spiel 8.0011489528666793e-05 -▁effective 8.0000124260267826e-05 -▁known 7.9984970569069218e-05 -▁step 7.9916778958675461e-05 -kt 7.9867529462279981e-05 -ain 7.985995261668067e-05 -▁Abstimmung 7.9856164193881028e-05 -▁Hand 7.9852375771081373e-05 -▁Finanz 7.9806914697485535e-05 -▁sollen 7.9776607315088318e-05 -▁Dank 7.9549301947109141e-05 -▁ihrem 7.9530359833110878e-05 -▁society 7.9473533491116087e-05 -▁keinen 7.943564926311956e-05 -▁natural 7.9424283994720607e-05 -▁nationalen 7.9212132317940052e-05 -ica 7.9162882821544558e-05 -▁complete 7.9117421747948734e-05 -▁environmental 7.9075749097152551e-05 -▁version 7.9056806983154288e-05 -▁Park 7.9041653291955679e-05 -▁meisten 7.9026499600757058e-05 -▁greater 7.90075574867588e-05 -▁besser 7.8981038527161233e-05 -ü 7.896209641316297e-05 -▁top 7.893936587636505e-05 -▁handelt 7.88711742659713e-05 -▁strategy 7.8848443729173388e-05 -ust 7.8840866883574077e-05 --C 7.8795405809978239e-05 -pr 7.872342577678484e-05 -ellen 7.8602196247195947e-05 -▁2001 7.8363525610817831e-05 -▁standards 7.8337006651220257e-05 -ster 7.8242296081228939e-05 -av 7.8223353967230675e-05 -▁Wasser 7.79884717536522e-05 -▁weiterhin 7.7943010680056363e-05 -▁Gesellschaft 7.77876853452706e-05 --A 7.7742224271674774e-05 -▁along 7.7689186352479625e-05 -▁event 7.766645581568172e-05 -▁principle 7.763614843328449e-05 -ade 7.7590687359688652e-05 -▁central 7.7530072594894219e-05 -▁betrifft 7.7465669407300117e-05 -▁Wahl 7.742778517930359e-05 -les 7.7333074609312272e-05 -▁deal 7.7314132495314008e-05 -▁tr 7.72989788041154e-05 -▁National 7.7261094576118873e-05 -ges 7.7143653469329636e-05 -fen 7.7136076623730325e-05 -▁required 7.708303870453519e-05 -▁Ebene 7.7075461858935879e-05 -▁words 7.706030816773727e-05 -▁19 7.6931501792549081e-05 -▁views 7.6848156490956715e-05 -▁user 7.68405796453574e-05 -▁four 7.6700408001770248e-05 -▁bekannt 7.668525431057164e-05 -▁serious 7.6651158505374768e-05 -▁Produkt 7.65602363581831e-05 -▁Se 7.64314299829949e-05 -man 7.6401122600597686e-05 -▁bedeutet 7.6223066729014e-05 -▁breakfast 7.6105625622224766e-05 -▁Windows 7.6041222434630664e-05 -▁macht 7.6037434011831022e-05 -▁training 7.5976819247036576e-05 -▁€ 7.5976819247036576e-05 -▁total 7.5935146596240393e-05 -▁deutlich 7.589347394544421e-05 -▁deshalb 7.5870743408646291e-05 -▁Energie 7.5866954985846636e-05 -▁besten 7.583664760344942e-05 -▁clearly 7.5825282335050467e-05 -▁meeting 7.578739810705394e-05 -▁0 7.575709072465671e-05 -▁Projekt 7.5722994919459838e-05 -▁sei 7.5673745423064344e-05 -▁pay 7.5556304316275107e-05 -▁solche 7.551842008827858e-05 -ri 7.5499477974280316e-05 -▁000 7.5480535860282053e-05 -▁Jahres 7.5450228477884836e-05 --R 7.5442651632285526e-05 -▁authorities 7.538961371309039e-05 -▁via 7.5351729485093863e-05 -▁inter 7.52683841835015e-05 -▁recent 7.5245653646703579e-05 -▁near 7.5082751466318517e-05 -▁voll 7.5078963043518862e-05 -▁essential 7.50600209295206e-05 -▁peace 7.4942579822731361e-05 -▁Vertrag 7.4900907171935178e-05 -▁cultural 7.4881965057936915e-05 -▁November 7.4832715561541435e-05 -▁aufgrund 7.4760735528348036e-05 -az 7.47228513003515e-05 -▁technical 7.4684967072354968e-05 -▁Zugang 7.4624352307560535e-05 -und 7.459783334796296e-05 -▁würden 7.459025650236365e-05 -▁young 7.454100700596817e-05 -▁achieve 7.4533430160368859e-05 -▁pages 7.45296417375692e-05 -ness 7.45220648919699e-05 -auf 7.4351585865985533e-05 -▁bed 7.43061247923897e-05 -▁wichtige 7.42341447591963e-05 -▁Arbeits 7.4226567913596986e-05 -▁Von 7.4215202645198033e-05 -▁IT 7.4203837376799066e-05 -ene 7.4147011034804276e-05 -eh 7.4090184692809485e-05 -ou 7.4067454156011566e-05 -▁address 7.4010627814016775e-05 -▁feel 7.4003050968417478e-05 -▁bring 7.3881821438828585e-05 -kl 7.3790899291636922e-05 -ph 7.3771957177638659e-05 -▁internal 7.3734072949642118e-05 -▁findet 7.3677246607647327e-05 -▁directly 7.366966976204803e-05 -▁side 7.3662092916448719e-05 -▁facilities 7.3635573956851158e-05 -▁visit 7.3525709695661218e-05 -▁longer 7.3514344427262266e-05 -▁Menschenrechte 7.3408268588871981e-05 -▁Natur 7.3400691743272684e-05 -▁fundamental 7.3385538052074062e-05 -▁solutions 7.335144224687719e-05 -í 7.3336288555678582e-05 -▁success 7.3328711710079271e-05 -▁enough 7.32946159048824e-05 -▁statt 7.32870390592831e-05 -▁months 7.3211270603290034e-05 -▁Richtung 7.3199905334891082e-05 -▁nächsten 7.3199905334891082e-05 -▁Ansicht 7.2923350470516425e-05 -▁el 7.2866524128521634e-05 -▁proposed 7.2855158860122681e-05 -▁nature 7.2851370437323026e-05 -▁Gebiet 7.2805909363727188e-05 -▁2002 7.2802120940927546e-05 -▁responsible 7.2786967249728924e-05 -ität 7.2771813558530316e-05 -ice 7.2737717753333444e-05 -▁employment 7.26922566797376e-05 -▁April 7.2684679834138309e-05 -▁guests 7.2650584028941423e-05 -▁currently 7.2620276646544207e-05 -isierung 7.2597546109746288e-05 -▁nutzen 7.2574815572948369e-05 -▁add 7.245358604335949e-05 -▁history 7.2438432352160868e-05 -of 7.242327866096226e-05 -▁Umsetzung 7.2366452318967469e-05 -▁Nähe 7.2358875473368158e-05 -▁requirements 7.2226280675380313e-05 -▁art 7.2101262722991779e-05 -▁Herren 7.1968667925003933e-05 -▁interests 7.1942148965406359e-05 -▁‘ 7.1904264737409832e-05 -▁reform 7.1824707858617122e-05 -ss 7.1779246785021284e-05 -▁allerdings 7.1733785711425459e-05 -▁events 7.1669382523831357e-05 -▁web 7.16655941010317e-05 -ose 7.1597402490637958e-05 -▁Please 7.1548152994242465e-05 -ures 7.1510268766245938e-05 --c 7.144586557865185e-05 -isten 7.14420771558522e-05 -▁Demokratie 7.1381462391057748e-05 -▁hoch 7.1279174975467119e-05 -▁cost 7.1192041251075112e-05 -ft 7.1116272795082057e-05 -▁regions 7.11124843722824e-05 -ative 7.10480811846883e-05 -▁involved 7.1044292761888645e-05 -▁let 7.1044292761888645e-05 -▁From 7.1021562225090726e-05 -▁Verantwortung 7.0972312728695246e-05 -▁themselves 7.09116979639008e-05 -ically 7.0851083199106353e-05 -▁Text 7.08397179307074e-05 -ful 7.0820775816709137e-05 -▁network 7.0805622125510529e-05 -▁live 7.0798045279911218e-05 -▁22 7.075258420631538e-05 -▁beispielsweise 7.0680604173121981e-05 -▁needed 7.0661662059123717e-05 -▁parties 7.06389315223258e-05 -▁Haushalts 7.0616200985527879e-05 -Z 7.0604835717128926e-05 -▁direct 7.0582105180331007e-05 -▁hoffe 7.0555586220733433e-05 -▁never 7.0494971455939e-05 -▁extremely 7.0449510382343162e-05 -▁sure 7.03585882351515e-05 -▁competition 7.0316915584355317e-05 -▁voted 7.0275242933559134e-05 -▁please 7.0263877665160181e-05 -ders 7.0210839745965032e-05 -▁Bedingungen 7.0210839745965032e-05 -▁station 7.0180532363567816e-05 -▁gleich 7.0142648135571289e-05 -▁Con 7.0082033370776842e-05 -▁democracy 7.0066879679578234e-05 -▁music 6.9949438572789e-05 -▁night 6.9949438572789e-05 -▁gemeinsamen 6.9926708035991078e-05 -▁notwendig 6.991155434479247e-05 -▁language 6.9896400653593862e-05 -▁efforts 6.9854728002797679e-05 -▁vier 6.9854728002797679e-05 -▁force 6.9835785888799416e-05 -▁freedom 6.9706979513611212e-05 -▁implementation 6.96918258224126e-05 -uss 6.9608480520820239e-05 -▁23 6.96046920980206e-05 -ish 6.9574384715623367e-05 -▁gehört 6.9498616259630312e-05 -▁etc 6.945694360883413e-05 -▁Ziele 6.9449366763234832e-05 -▁ago 6.9426636226436913e-05 -▁wohl 6.9381175152841075e-05 -▁kleinen 6.9373598307241764e-05 -▁Vorschläge 6.9339502502044893e-05 -▁upon 6.9320560388046629e-05 -▁century 6.9290253005649413e-05 -▁60 6.92826761600501e-05 -▁Interesse 6.92675224688515e-05 -▁GmbH 6.9222061395255656e-05 -▁Grundlage 6.9218272972456e-05 -ew 6.92106961268567e-05 -ized 6.9199330858457737e-05 -▁vielleicht 6.9176600321659831e-05 -▁read 6.9153869784861912e-05 -▁changes 6.9112197134065729e-05 -ken 6.9093255020067466e-05 -▁verwendet 6.9070524483269547e-05 -▁plan 6.896066022207962e-05 -ind 6.8930352839682391e-05 -fer 6.8926564416882749e-05 -▁Anwendung 6.8907622302884485e-05 -▁Ihrem 6.8828065424091775e-05 -▁$ 6.8816700155692809e-05 -ats 6.8805334887293856e-05 -▁1. 6.8790181196095248e-05 -schaft 6.8672740089306011e-05 -▁team 6.8642432706908781e-05 -bereich 6.8608336901711909e-05 -▁developed 6.8562875828116071e-05 -▁run 6.8540145291318152e-05 -▁beach 6.853635686851851e-05 -▁Russia 6.84567999897258e-05 -▁comfortable 6.8411338916129962e-05 -▁keep 6.8358300996934827e-05 -▁regional 6.83204167689383e-05 -▁sprechen 6.8316628346138644e-05 -▁sozialen 6.8282532540941772e-05 -▁Daher 6.8237071467345934e-05 -▁lot 6.81840335481508e-05 -▁Inter 6.8130995628955663e-05 -▁= 6.8100688246558434e-05 -▁Verfahren 6.80476503273633e-05 -ionen 6.7945362911772669e-05 -▁regulation 6.7941574488973014e-05 -▁De 6.7892324992577534e-05 -▁erreicht 6.7877171301378926e-05 -ally 6.7850652341781351e-05 -▁Geschichte 6.7820344959384135e-05 -▁Lebens 6.779382599978656e-05 -▁By 6.7767307040188986e-05 -▁bl 6.7755941771790033e-05 -▁situated 6.7676384892997323e-05 -▁Reihe 6.76536543561994e-05 -▁restaurant 6.76536543561994e-05 -▁.. 6.7642289087800451e-05 -▁My 6.7623346973802188e-05 -▁though 6.760819328260358e-05 -▁strong 6.7585462745805661e-05 -▁Pr 6.7562732209007742e-05 -ino 6.7555155363408431e-05 -▁either 6.7505905867012951e-05 -▁Minister 6.7471810061816079e-05 -▁Kunst 6.7456656370617457e-05 -gr 6.744907952501816e-05 -▁Ex 6.7433925833819552e-05 -▁comes 6.7407406874221977e-05 -sk 6.7358157377826484e-05 -▁werde 6.7327849995429267e-05 -▁cases 6.7255869962235868e-05 -▁negotiations 6.7252081539436213e-05 -▁halten 6.724071627103726e-05 -▁sc 6.7217985734239341e-05 -ean 6.7146005701045929e-05 -▁Möglichkeiten 6.7146005701045929e-05 -▁effect 6.7142217278246287e-05 -▁France 6.70853909362515e-05 -▁weltweit 6.70853909362515e-05 -▁markets 6.7009622480258428e-05 -▁Sport 6.68353550314744e-05 -▁AG 6.6831566608674758e-05 -▁bleiben 6.6831566608674758e-05 -▁West 6.6823989763075447e-05 -ix 6.6774740266679967e-05 -▁May 6.6752009729882048e-05 -▁everything 6.6717913924685176e-05 -▁bereit 6.6687606542287947e-05 -▁Information 6.6589107549496973e-05 -▁Bl 6.6573953858298365e-05 -▁foreign 6.6517127516303574e-05 -▁developing 6.6448935905909817e-05 -▁dis 6.6448935905909817e-05 -bl 6.6433782214711209e-05 -isation 6.6429993791911553e-05 -▁land 6.6414840100712945e-05 -▁Debatte 6.6403474832313992e-05 -lichkeit 6.6301187416723363e-05 -per 6.6286033725524755e-05 -ana 6.6259514765927181e-05 -▁final 6.6225418960730309e-05 -▁request 6.6198900001132735e-05 -▁true 6.6126919967939335e-05 -▁Damen 6.6111766276740727e-05 -▁unique 6.6054939934745936e-05 -ling 6.6047363089146626e-05 -▁arbeiten 6.5986748324352179e-05 -▁created 6.5960229364754618e-05 -ON 6.5952652519155308e-05 -politik 6.58882493315612e-05 -▁How 6.5876884063162253e-05 -▁additional 6.5804904029968854e-05 -mer 6.5759442956373016e-05 -▁hard 6.5702616614378225e-05 -▁rest 6.5660943963582043e-05 -▁house 6.5604117621587252e-05 -▁consider 6.55927523531883e-05 -▁verschiedene 6.55435028567928e-05 -▁Paris 6.5516983897195231e-05 -▁fr 6.5516983897195231e-05 -▁later 6.5475311246399062e-05 -▁prices 6.54639459780001e-05 -▁improve 6.5426061750003568e-05 -▁könnten 6.5418484904404271e-05 -▁states 6.5312409066013987e-05 -▁almost 6.5308620643214331e-05 -▁French 6.52707364152178e-05 -▁according 6.5266947992418149e-05 -▁Berichterstatter 6.5259371146818851e-05 -▁capital 6.5259371146818851e-05 -▁En 6.525179430121954e-05 -▁{ 6.5221486918822324e-05 -▁Presidency 6.5217698496022669e-05 -▁meet 6.5187391113625439e-05 -▁Blick 6.51836026908258e-05 -▁living 6.5122987926031351e-05 -▁education 6.51191995032317e-05 -co 6.5104045812033087e-05 -▁post 6.5028277356040033e-05 -▁neu 6.4929778363249059e-05 -▁road 6.4876740444053924e-05 -▁oft 6.4838856216057383e-05 -▁share 6.4797183565261214e-05 -▁verfügt 6.4728991954867457e-05 -▁file 6.4694896149670585e-05 --G 6.4691107726870929e-05 -berg 6.4596397156879611e-05 -▁Hinblick 6.4577455042881348e-05 -▁included 6.45660897744824e-05 -ters 6.4539570814884821e-05 -▁Einsatz 6.45168402780869e-05 -▁gemeinsame 6.4505475009687949e-05 -▁climate 6.448274447289003e-05 -▁democratic 6.4467590781691422e-05 -▁turn 6.4467590781691422e-05 -alen 6.4395610748498009e-05 -▁Minuten 6.4369091788900448e-05 -▁understand 6.4369091788900448e-05 -▁actually 6.4365303366100793e-05 -▁thus 6.4338784406503218e-05 -▁1999 6.4334995983703563e-05 -▁Du 6.4331207560903921e-05 -▁ermöglicht 6.4270592796109475e-05 -att 6.4255439104910866e-05 -▁21 6.4247862259311556e-05 -▁co 6.4191035917316765e-05 -▁Rück 6.4156940112119893e-05 -▁Platz 6.4028133736931689e-05 -▁idea 6.4016768468532737e-05 -▁www. 6.3997826354534473e-05 -▁Doch 6.3994037931734818e-05 -▁wenig 6.39637305493376e-05 -▁lange 6.3922057898541419e-05 -itz 6.3895538938943844e-05 -ij 6.3891750516144189e-05 -▁accept 6.3884173670544891e-05 -ising 6.3869019979346283e-05 -▁Design 6.3853866288147662e-05 -▁Institutionen 6.3815982060151134e-05 -▁stand 6.3815982060151134e-05 -▁soon 6.3789463100553573e-05 -▁program 6.3781886254954263e-05 -▁Auswirkungen 6.377052098655531e-05 -▁tax 6.374021360415808e-05 -▁Wert 6.3687175684962944e-05 -ize 6.3675810416563978e-05 -▁seen 6.3645503034166762e-05 -ES 6.3637926188567451e-05 -▁Druck 6.3637926188567451e-05 -▁receive 6.3573523000973363e-05 -ren 6.3471235585382734e-05 -▁reach 6.345229347138447e-05 -hen 6.3429562934586551e-05 -▁zeigt 6.3338640787394889e-05 -▁Strategie 6.3278026022600442e-05 -▁hatten 6.3247718640203226e-05 -ha 6.323635337180426e-05 -▁Musik 6.323635337180426e-05 -▁Th 6.323635337180426e-05 -▁care 6.3202257566607388e-05 -ization 6.3122700687814678e-05 -▁Israel 6.3118912265015023e-05 -▁führt 6.2978740621427866e-05 -▁type 6.2971163775828569e-05 -▁» 6.2956010084629961e-05 -▁Dienstleistungen 6.2918125856633434e-05 -▁appropriate 6.2846145823440021e-05 -▁erforderlich 6.281962686384246e-05 -▁sub 6.2793107904244886e-05 -fe 6.2785531058645575e-05 -▁Ad 6.2702185757053223e-05 -▁Tatsache 6.2702185757053223e-05 -▁funds 6.2668089951856338e-05 -ka 6.2637782569459121e-05 -we 6.2630205723859811e-05 -▁que 6.2630205723859811e-05 -▁original 6.2626417301060169e-05 -▁held 6.2611263609861547e-05 -anz 6.26074751870619e-05 -▁outside 6.25998983414626e-05 -▁clean 6.2592321495863283e-05 -▁par 6.257337938186502e-05 -▁wirtschaftlichen 6.2516553039870229e-05 -▁Ergebnis 6.2490034080272668e-05 -▁lead 6.239532351028135e-05 -ann 6.23915350874817e-05 -▁gesamten 6.2274093980692457e-05 -▁Ausdruck 6.223620975269593e-05 -▁Verbraucher 6.223620975269593e-05 -▁designed 6.2228632907096619e-05 -▁procedure 6.2183171833500781e-05 -▁moment 6.2164229719502518e-05 -▁thing 6.212634549150599e-05 -▁walk 6.2084672840709821e-05 -▁Schritt 6.2001327539117456e-05 --f 6.198996227071849e-05 -▁Regionen 6.1986173847918848e-05 --W 6.1963443311120929e-05 -▁American 6.1880098009528564e-05 -▁rate 6.1880098009528564e-05 -▁ebenso 6.1846002204331692e-05 -erm 6.1777810593937935e-05 -▁einzelnen 6.1747503211540718e-05 -▁features 6.1736137943141765e-05 -▁Landes 6.170204213794488e-05 -▁knowledge 6.170204213794488e-05 -▁Partner 6.1675523178347319e-05 -▁danken 6.1618696836352528e-05 -▁War 6.1614908413552873e-05 -▁entwickelt 6.1565658917157379e-05 -▁darum 6.1535351534760163e-05 -▁famous 6.1527774689160852e-05 -rat 6.152398626636121e-05 -▁term 6.1482313615565028e-05 -▁week 6.1478525192765372e-05 -▁Kolleginnen 6.1463371501566764e-05 -▁Entscheidung 6.1455794655967453e-05 -▁big 6.14444293875685e-05 -▁links 6.1376237777174743e-05 -ks 6.1364872508775791e-05 -▁workers 6.1353507240376824e-05 -▁} 6.1326988280779263e-05 -▁majority 6.1266373515984817e-05 -▁200 6.12587966703855e-05 -▁Is 6.1224700865188634e-05 -▁Ma 6.1186816637192107e-05 -▁weiß 6.1186816637192107e-05 -▁impact 6.11792397915928e-05 -▁Version 6.1092106067200789e-05 -tern 6.1084529221601478e-05 -▁Sk 6.1080740798801823e-05 -▁early 6.1073163953202526e-05 -▁professional 6.1039068148005654e-05 -▁Seit 6.0986030228810512e-05 -▁looking 6.0986030228810512e-05 -ign 6.09784533832112e-05 -▁Erfolg 6.0967088114812241e-05 -▁customer 6.0963299692012593e-05 -werk 6.0959511269212937e-05 -▁five 6.091026177281745e-05 -▁Außerdem 6.0861012276421964e-05 -▁Kon 6.0807974357226828e-05 -▁communication 6.0781455397629261e-05 -▁Zum 6.0777666974829605e-05 -▁denke 6.0747359592432382e-05 -▁… 6.0747359592432382e-05 -▁throughout 6.0743571169632733e-05 -to 6.0735994324033422e-05 -ists 6.0717052210035159e-05 -▁amount 6.07056869416362e-05 -▁Antwort 6.0641283754042104e-05 -▁space 6.0629918485643145e-05 -▁Mitglieder 6.0550361606850435e-05 -▁Wort 6.0504900533254604e-05 -▁fair 6.0421555231662245e-05 -▁guarantee 6.0410189963263286e-05 -▁August 6.0398824694864326e-05 -▁hours 6.0379882580866063e-05 -▁takes 6.03609404668678e-05 -IS 6.0353363621268495e-05 -eren 6.0323056238871272e-05 -▁South 6.0292748856474049e-05 -▁across 6.0288960433674393e-05 -▁internationale 6.0258653051277177e-05 -▁wide 6.0251076205677866e-05 -▁Änderungsanträge 6.02321340916796e-05 -▁doing 6.020940355488169e-05 -▁Nicht 6.0171519326885163e-05 -▁Abgeordneten 6.0133635098888629e-05 -▁jedem 6.0046501374496615e-05 -▁works 6.0035136106097662e-05 -▁certainly 5.9997251878101135e-05 -▁date 5.997073291850356e-05 -▁Verhandlungen 5.9944213958905993e-05 -▁parts 5.990254130810981e-05 -IN 5.9898752885310161e-05 -ink 5.9868445502912938e-05 -▁Ge 5.9860868657313634e-05 -▁context 5.9845714966115019e-05 -IT 5.9822984429317107e-05 -▁Bereichen 5.9788888624120228e-05 -▁gemeinsam 5.9785100201320573e-05 -▁Industrie 5.9747215973324046e-05 -▁users 5.97434275505244e-05 -▁traditional 5.9697966476928559e-05 -▁konnte 5.9690389631329255e-05 -ages 5.9660082248932032e-05 -▁Fr 5.962598644373516e-05 -▁Änderungsantrag 5.9588102215738633e-05 -icht 5.9584313792938977e-05 -▁befinden 5.9584313792938977e-05 -▁send 5.9565371678940714e-05 -AN 5.9561583256141065e-05 -ran 5.9561583256141065e-05 -▁hätte 5.9523699028144531e-05 -▁gest 5.94782379545487e-05 -▁guest 5.946687268614974e-05 -▁Not 5.9357008424959814e-05 -▁Zentrum 5.933806631096155e-05 -▁perfect 5.9303970505764672e-05 -▁gleichzeitig 5.9285028391766408e-05 -AL 5.9273663123367449e-05 -▁Unser 5.9266086277768145e-05 -▁sl 5.9213048358573e-05 -▁check 5.9209259935773354e-05 -▁investment 5.9197894667374394e-05 -▁setzen 5.9186529398975435e-05 -▁ever 5.9088030406184461e-05 -▁book 5.9034992486989326e-05 -dern 5.9012261950191407e-05 -▁weiteren 5.89971082589928e-05 -▁Vereinigten 5.8985742990593839e-05 -▁Red 5.8906186111801129e-05 -▁although 5.8902397689001481e-05 -▁Kraft 5.889482084340217e-05 -▁act 5.889482084340217e-05 -uf 5.8891032420602521e-05 -▁relations 5.8879667152203562e-05 -▁took 5.8879667152203562e-05 -▁z.B. 5.8837994501407379e-05 -▁concerns 5.8815263964609467e-05 -▁27 5.8811475541809811e-05 -▁pool 5.8807687119010156e-05 -▁Beziehungen 5.8690246012220919e-05 -▁existing 5.86599386298237e-05 -▁ar 5.861826597902752e-05 -▁Turkey 5.8508401717837587e-05 -▁Liste 5.8493248026638978e-05 -▁neben 5.8485671181039674e-05 -form 5.8451575375842796e-05 -▁Raum 5.8417479570645924e-05 -▁Bevölkerung 5.837959534264939e-05 -▁Ent 5.837959534264939e-05 -stand 5.8375806919849741e-05 -▁Gesundheit 5.8375806919849741e-05 -▁Or 5.8356864805851478e-05 -▁Online 5.831898057785495e-05 -▁> 5.8311403732255646e-05 -▁connection 5.826215423586016e-05 -▁remain 5.8247000544661545e-05 -▁Japan 5.8231846853462937e-05 -▁Programme 5.8231846853462937e-05 -▁heißt 5.8110617323874044e-05 -▁ha 5.8084098364276476e-05 -▁Abkommen 5.8076521518677172e-05 -och 5.8050002559079605e-05 -е 5.8046214136279949e-05 -▁programmes 5.8012118331083071e-05 -▁Familien 5.8008329908283422e-05 -ants 5.7993176217084807e-05 -▁English 5.7989387794285158e-05 -▁try 5.7890888801494185e-05 -▁jede 5.7845427727898353e-05 -▁significant 5.7818908768300779e-05 -▁low 5.7796178231502867e-05 -▁je 5.7788601385903556e-05 -ots 5.7773447694704948e-05 -▁mentioned 5.7769659271905292e-05 -▁card 5.77128329299105e-05 -▁America 5.7701467661511542e-05 -▁helfen 5.7674948701913974e-05 -▁apply 5.7614333937119528e-05 -▁Interessen 5.7610545514319879e-05 -gl 5.759918024592092e-05 -▁model 5.7591603400321616e-05 -▁Let 5.7493104407530642e-05 -▁rund 5.7485527561931338e-05 -▁leading 5.7474162293532379e-05 -▁kleine 5.7466585447933074e-05 -▁writing 5.7409759105938284e-05 -▁tra 5.7398393837539324e-05 -▁Aufgabe 5.7383240146340709e-05 -▁Beitrag 5.73605096095428e-05 -▁concerning 5.7349144341143837e-05 -▁Government 5.7330202227145574e-05 -▁Court 5.7315048535946959e-05 -▁completely 5.7273375885150783e-05 -▁Sicherheits 5.7269587462351127e-05 -▁don 5.7269587462351127e-05 -▁friendly 5.7250645348352864e-05 -▁verbunden 5.724306850275356e-05 -▁reasons 5.7197607429157722e-05 -▁Person 5.7159723201161194e-05 -▁nothing 5.7118050550365019e-05 -▁related 5.7091531590767444e-05 -▁half 5.7068801053969532e-05 -▁Handels 5.7053647362770917e-05 -▁aware 5.6985455752377167e-05 -▁Türkei 5.6974090483978214e-05 -▁community 5.6974090483978214e-05 -▁Selbst 5.6970302061178558e-05 --m 5.693620625598168e-05 -▁decisions 5.6913475719183768e-05 -▁includes 5.6902110450784808e-05 -bare 5.68869567595862e-05 -▁Best 5.687559149118724e-05 -über 5.6856649377188977e-05 -▁fünf 5.6852860954389321e-05 -▁Do 5.6841495685990362e-05 -.000 5.6826341994791754e-05 -▁Forschung 5.6818765149192449e-05 -10 5.6814976726392794e-05 -▁Themen 5.6799823035194186e-05 -▁ad 5.679603461239453e-05 -▁gestellt 5.6754361961598348e-05 -▁internet 5.6716477733601821e-05 -▁member 5.6652074546007726e-05 -▁London 5.6636920854809111e-05 -▁wichtigen 5.6636920854809111e-05 -▁enthalten 5.6633132432009462e-05 -▁game 5.6617978740810847e-05 -▁basic 5.6606613472411888e-05 -▁front 5.658009451281432e-05 -▁meinen 5.658009451281432e-05 -▁Kontrolle 5.6545998707617448e-05 -vers 5.6538421862018137e-05 -▁heraus 5.6523268170819529e-05 -▁material 5.6447499714826475e-05 -▁Förderung 5.6405827064030292e-05 -▁unseres 5.638309652723238e-05 -▁committee 5.6314904916838629e-05 -▁population 5.6242924883645224e-05 -▁military 5.6220194346847304e-05 -▁holiday 5.6216405924047656e-05 -ely 5.6170944850451818e-05 -▁Geschäfts 5.6170944850451818e-05 --in 5.6167156427652169e-05 -▁Par 5.6152002736453554e-05 -▁80 5.6102753240058067e-05 -▁allows 5.6061080589261892e-05 -▁media 5.6038350052463972e-05 -▁culture 5.6000465824467445e-05 -▁practice 5.599667740166779e-05 -▁develop 5.5981523710469182e-05 -AR 5.5962581596470918e-05 -enn 5.5939851059673e-05 -▁Ausschuss 5.5909543677275776e-05 -▁extra 5.5864082603679945e-05 -▁test 5.5852717335280985e-05 -▁Since 5.5784525724887235e-05 -▁liegen 5.5754218342490012e-05 --H 5.5742853074091059e-05 -▁gab 5.5742853074091059e-05 -amm 5.57390646512914e-05 -▁mean 5.57314878056921e-05 -▁Verbindung 5.5678449886496957e-05 -AT 5.5651930926899389e-05 -ual 5.5651930926899389e-05 -▁agreements 5.5648142504099734e-05 -▁gestimmt 5.5621623544502166e-05 -▁mind 5.5610258276103207e-05 -▁jedes 5.5606469853303558e-05 -baren 5.5591316162104943e-05 -▁gegeben 5.5564797202507375e-05 -▁travel 5.5534489820110152e-05 -oll 5.5500394014913281e-05 -▁choice 5.5420837136120571e-05 -▁Kr 5.5405683444921956e-05 -▁Reform 5.5371587639725084e-05 -▁bleibt 5.5356433948526469e-05 -▁bekommen 5.528445391533307e-05 -agen 5.5231415996137928e-05 -oss 5.5197320190941056e-05 -alt 5.5182166499742441e-05 -innen 5.5170801231343488e-05 -▁body 5.5144282271745914e-05 -eller 5.5072302238552515e-05 -gel 5.50571485473539e-05 -▁objectives 5.4958649554562933e-05 -eter 5.4954861131763278e-05 -▁received 5.4954861131763278e-05 -ars 5.4905611635367791e-05 -ep 5.4867727407371264e-05 -▁jobs 5.4829843179374736e-05 -ó 5.4826054756575081e-05 -▁values 5.4826054756575081e-05 -▁wegen 5.4822266333775432e-05 -▁ex 5.4761651568980986e-05 -tra 5.4697248381386884e-05 -▁partners 5.468967153578758e-05 -▁Tra 5.4666940998989668e-05 -▁package 5.4651787307791053e-05 -▁simple 5.4640422039392094e-05 -▁Entschließung 5.461011465699487e-05 -AS 5.4606326234195222e-05 -▁Lisbon 5.4579807274597654e-05 -▁established 5.4443424053810153e-05 -▁Strand 5.44396356310105e-05 -▁files 5.4386597711815363e-05 -▁center 5.4379020866216052e-05 -▁richtig 5.4363867175017444e-05 -▁performance 5.4341136638219524e-05 -ationen 5.4329771369820565e-05 -▁euro 5.4284310296224734e-05 -▁civil 5.4261579759426815e-05 -▁Damit 5.4235060799829247e-05 -iven 5.42312723770296e-05 -lin 5.4227483954229943e-05 -ines 5.4204753417432024e-05 -▁equipment 5.418581130343376e-05 -▁applications 5.4140350229837929e-05 -▁computer 5.4132773384238625e-05 -▁Bad 5.4113831270240361e-05 -▁entire 5.4091100733442442e-05 -ond 5.4060793351045219e-05 -▁Verordnung 5.4015332277449388e-05 -▁gerne 5.4003967009050428e-05 -30 5.3992601740651469e-05 -▁Prozess 5.397744804945286e-05 -▁pl 5.39660827810539e-05 -ente 5.3947140667055637e-05 -ong 5.3939563821456333e-05 -▁aim 5.3916833284658414e-05 -▁Nutzung 5.3901679593459806e-05 -urs 5.3878949056661887e-05 -▁regarding 5.3867583788262927e-05 -▁Behörden 5.3852430097064319e-05 -▁red 5.3784238486670569e-05 -▁move 5.3769084795471954e-05 -▁Meer 5.3757719527072994e-05 -che 5.3708470030677514e-05 -ette 5.3704681607877859e-05 -▁concern 5.3700893185078204e-05 -▁Bitte 5.368195107107994e-05 -▁wichtigsten 5.3663008957081676e-05 -▁Here 5.3655432111482372e-05 -▁Mitarbeiter 5.3628913151884805e-05 -▁High 5.36213363062855e-05 -▁Ausschusses 5.3579663655489318e-05 -ese 5.35645099642907e-05 -▁Iran 5.35569331186914e-05 -▁Port 5.3522837313494527e-05 -▁stop 5.3515260467895216e-05 -▁agricultural 5.3496318353896953e-05 -▁sound 5.3492529931097304e-05 -▁quickly 5.3469799394299385e-05 -▁keyword 5.3450857280301121e-05 -▁mehrere 5.3435703589102513e-05 -▁Krise 5.34205498979039e-05 -▁below 5.3416761475104249e-05 -▁objective 5.3375088824308067e-05 -▁Car 5.3359935133109452e-05 -▁Kl 5.33561467103098e-05 -▁Auswahl 5.3322050905112925e-05 -▁answer 5.32841666771164e-05 -▁image 5.3280378254316749e-05 -▁demand 5.3242494026320222e-05 -▁prevent 5.3223551912321958e-05 -▁Str 5.3189456107125079e-05 -▁reports 5.317809083872612e-05 -recht 5.316672557032716e-05 -▁fight 5.316672557032716e-05 -▁Den 5.31477834563289e-05 -▁gr 5.3094745537133761e-05 -▁Wirtschafts 5.3072015000335842e-05 -▁Standard 5.3018977081140707e-05 -▁Post 5.3003823389942092e-05 -fr 5.2958362316346261e-05 -▁hinsichtlich 5.2954573893546605e-05 -▁Gruppe 5.2931843356748693e-05 -▁pass 5.2928054933949037e-05 -los 5.2924266511149382e-05 -▁built 5.2916689665550078e-05 -port 5.2844709632356679e-05 -▁ganze 5.2825767518358415e-05 -▁met 5.2795460135961192e-05 -ute 5.2772729599163273e-05 -▁goods 5.2757575907964665e-05 -▁ganzen 5.2723480102767786e-05 -▁City 5.2689384297570915e-05 -▁face 5.2670442183572651e-05 -▁Ne 5.266286533797334e-05 -▁Em 5.2617404264377509e-05 -▁potential 5.26022505731789e-05 -▁enthält 5.2598462150379245e-05 -▁wobei 5.2583308459180637e-05 -▁500 5.2541635808384454e-05 -▁East 5.2515116848786887e-05 -platz 5.2511328425987231e-05 -▁lack 5.2499963157588272e-05 --d 5.2477232620790359e-05 -oy 5.2473444197990704e-05 -▁besch 5.2431771547194521e-05 -▁benefit 5.2420406278795562e-05 -▁Erweiterung 5.2363579936800771e-05 -istischen 5.2333272554403548e-05 -▁industrial 5.2333272554403548e-05 -ives 5.23294841316039e-05 -uer 5.2310542017605636e-05 -▁return 5.2302965172006332e-05 -▁contains 5.2299176749206676e-05 -▁ermöglichen 5.2291599903607372e-05 -▁Verbesserung 5.2257504098410493e-05 -▁double 5.223856198441223e-05 -▁function 5.2204466179215358e-05 -ech 5.215900510561952e-05 -▁Gäste 5.2121120877622993e-05 -oren 5.2113544032023689e-05 -▁« 5.2105967186424385e-05 -▁search 5.2094601918025425e-05 -▁achieved 5.2075659804027161e-05 -▁derzeit 5.2071871381227506e-05 -▁sense 5.2068082958427857e-05 -▁Anfang 5.2037775576030634e-05 -▁Macht 5.2022621884832026e-05 -▁seems 5.1996102925234452e-05 -▁Kampf 5.1977160811236188e-05 --E 5.1969583965636884e-05 -▁Handel 5.1965795542837228e-05 -ante 5.1943065006039316e-05 -▁party 5.1912757623642093e-05 -uc 5.18559312816473e-05 -▁men 5.18559312816473e-05 -▁Gewalt 5.1840777590448687e-05 -ür 5.1821835476450424e-05 -▁indeed 5.1806681785251815e-05 -▁Barcelona 5.1783951248453896e-05 -▁complex 5.1776374402854592e-05 -▁focus 5.1761220711655977e-05 -▁recently 5.1738490174858065e-05 -▁zeigen 5.173470175205841e-05 -▁Personal 5.17195480608598e-05 -▁Affairs 5.1700605946861538e-05 -▁II 5.1685452255662923e-05 -▁initiative 5.1685452255662923e-05 -▁treatment 5.1681663832863274e-05 -▁hinter 5.1666510141664659e-05 -IC 5.1617260645269173e-05 -▁See 5.1609683799669868e-05 -▁nice 5.1507396384079246e-05 -▁Gelegenheit 5.1499819538479935e-05 -▁Insel 5.1473300578882368e-05 -▁job 5.1461935310483408e-05 -▁Team 5.1424051082486881e-05 -fall 5.1420262659687232e-05 -ole 5.1416474236887577e-05 -▁Finally 5.1405108968488617e-05 -50 5.1382378431690705e-05 -▁gehören 5.1329340512495563e-05 -▁Angebot 5.1325552089695908e-05 --1 5.1302821552897995e-05 -▁highly 5.1280091016100076e-05 -▁Bo 5.1272514170500772e-05 -▁June 5.11740151777098e-05 -▁beginning 5.1166438332110495e-05 -▁successful 5.1086881453317785e-05 -amp 5.1064150916519866e-05 -▁Schaffung 5.1048997225321257e-05 -▁Central 5.1041420379721953e-05 -▁note 5.1033843534122649e-05 -▁floor 5.1007324574525075e-05 -▁völlig 5.0988382460526811e-05 -▁Now 5.0984594037727163e-05 -▁Management 5.094292138693098e-05 -▁equipped 5.0848210816939662e-05 -▁Zahl 5.0840633971340351e-05 -▁Gen 5.0833057125741047e-05 -▁spielen 5.08292687029414e-05 -▁Darüber 5.0817903434542439e-05 -▁dialogue 5.078380762934556e-05 -ano 5.0780019206545911e-05 -▁levels 5.0742134978549384e-05 -▁lang 5.0738346555749729e-05 -▁presented 5.0734558132950073e-05 -▁br 5.0708039173352506e-05 -▁Ergebnisse 5.0704250750552857e-05 -▁gar 5.0704250750552857e-05 -▁gl 5.0647424408558059e-05 -▁everyone 5.0643635985758411e-05 -▁Some 5.0628482294559796e-05 -▁huge 5.0601963334962228e-05 -▁Verwendung 5.0492099073772295e-05 -ele 5.0480733805373342e-05 -▁verbessern 5.0431484308977856e-05 -▁governments 5.0420119040578896e-05 -▁fort 5.0393600080981328e-05 -ums 5.0367081121383754e-05 -▁integration 5.035950427578445e-05 -▁bestehen 5.0332985316186882e-05 -▁Business 5.0317831624988267e-05 -▁increased 5.0272370551392436e-05 -▁recht 5.0257216860193828e-05 -▁lässt 5.0253428437394172e-05 -out 5.0223121054996949e-05 -ord 5.02193326321973e-05 -▁Kontakt 5.0185236827000422e-05 -▁Wein 5.0166294713002158e-05 -▁active 5.0143564176204239e-05 -iens 5.0120833639406327e-05 -que 5.0109468371007367e-05 -▁bus 5.0109468371007367e-05 -▁später 5.0086737834209448e-05 -▁guten 5.0045065183413273e-05 -▁Q 5.0029911492214658e-05 -▁consumers 4.9965508304620563e-05 -▁comp 4.9961719881820907e-05 -,000 4.9920047231024725e-05 -ulation 4.9920047231024725e-05 -▁Zeitpunkt 4.991247038542542e-05 -▁higher 4.991247038542542e-05 -▁carried 4.9867009311829589e-05 -▁Her 4.9840490352232021e-05 -▁verwenden 4.9832913506632711e-05 -▁com 4.98101829698348e-05 -▁Woche 4.9806394547035143e-05 -▁stage 4.9795029278636183e-05 -weise 4.9749568205040352e-05 -▁Russland 4.971547239984348e-05 -▁min 4.9700318708644865e-05 -▁eigene 4.9696530285845217e-05 -▁reviews 4.9685165017446257e-05 -▁General 4.96813765946466e-05 -▁non 4.9677588171846953e-05 -cken 4.9643492366650075e-05 -▁series 4.9613184984252851e-05 -▁Wer 4.9582877601855628e-05 -▁Tage 4.9575300756256324e-05 -▁agreed 4.956772391065702e-05 -▁verfügen 4.9552570219458405e-05 -▁whose 4.95449933738591e-05 --p 4.9522262837061189e-05 -▁großer 4.9518474414261533e-05 -▁Notwendigkeit 4.948816703186431e-05 -▁mein 4.9423763844270215e-05 -▁amendment 4.941997542147056e-05 -▁office 4.9401033307472296e-05 -▁press 4.9397244884672641e-05 -▁cl 4.9382091193474033e-05 -bau 4.932905327427889e-05 -▁Dialog 4.9313899583080282e-05 -▁soziale 4.9310111160280627e-05 -▁direction 4.9306322737480978e-05 -▁principles 4.9306322737480978e-05 -▁choose 4.9302534314681323e-05 -▁leicht 4.9245707972686532e-05 -▁asked 4.9238131127087228e-05 -▁Linux 4.9222977435888613e-05 -▁mal 4.9219189013088964e-05 -ax 4.913205528869695e-05 -err 4.9113113174698687e-05 -uen 4.9113113174698687e-05 -▁First 4.9097959483500072e-05 -▁similar 4.9097959483500072e-05 -▁Lösungen 4.9079017369501808e-05 -▁Transport 4.9044921564304936e-05 -▁PC 4.9010825759108058e-05 -▁— 4.9007037336308409e-05 -ition 4.8919903611916395e-05 -vor 4.8882019383919868e-05 -▁meinem 4.8882019383919868e-05 -▁perhaps 4.8874442538320557e-05 -iere 4.8855500424322293e-05 -▁commitment 4.8851712001522645e-05 -▁Therefore 4.883655831032403e-05 -▁Reise 4.87986740823275e-05 -▁shows 4.8749424585932016e-05 -▁zunächst 4.870775193513584e-05 -ase 4.8681232975538265e-05 -ents 4.866229086154e-05 -▁gesamte 4.8658502438740353e-05 -▁rates 4.8635771901942434e-05 -▁Tat 4.8620618210743826e-05 -▁shown 4.8620618210743826e-05 -ight 4.8597887673945907e-05 -а 4.8578945559947643e-05 -▁dr 4.8567580291548684e-05 -▁fördern 4.8567580291548684e-05 -ments 4.8476658144357014e-05 -▁Änderungen 4.8476658144357014e-05 -▁added 4.8472869721557366e-05 -▁hotels 4.8461504453158406e-05 -▁Nord 4.8442562339160143e-05 -▁Debian 4.8434985493560839e-05 -▁matters 4.8404678111163615e-05 -▁thanks 4.840088968836396e-05 -▁Sozial 4.8385735997165352e-05 -▁angenommen 4.8381947574365696e-05 -▁innovative 4.8336486500769865e-05 -▁nuclear 4.8313755963971946e-05 -▁Juni 4.8302390695572986e-05 -ct 4.8275871735975419e-05 -ious 4.8245564353578196e-05 -sen 4.8234199085179236e-05 -▁Mitglied 4.8234199085179236e-05 -▁gleichen 4.8188738011583405e-05 -ker 4.8158430629186182e-05 -▁tatsächlich 4.8131911669588614e-05 -▁Frankreich 4.8116757978390006e-05 -▁aspects 4.8097815864391742e-05 -ank 4.8075085327593823e-05 -▁Of 4.8075085327593823e-05 -ät 4.8067508481994519e-05 -▁Mai 4.8067508481994519e-05 -essen 4.803341267679764e-05 -▁protect 4.8025835831198336e-05 -▁sorgen 4.8014470562799377e-05 -▁latest 4.8003105294400417e-05 -▁Article 4.7991740026001458e-05 -▁considered 4.7991740026001458e-05 -min 4.796522106640389e-05 -▁Berg 4.7923548415607714e-05 -▁joint 4.7919759992808059e-05 -▁70 4.7915971570008403e-05 -▁EN 4.7915971570008403e-05 -▁Buch 4.7893241033210491e-05 -▁Während 4.7870510496412572e-05 -▁promote 4.7847779959614653e-05 -str 4.78364146912157e-05 -▁vers 4.7825049422816741e-05 -yn 4.7802318886018822e-05 -▁Herzen 4.7775799926421254e-05 -▁came 4.7753069389623335e-05 -▁Initiative 4.774928096682368e-05 -▁à 4.7726550430025767e-05 -▁enable 4.7707608316027504e-05 -▁previous 4.7707608316027504e-05 -▁Qu 4.7703819893227848e-05 -alten 4.7669724088030976e-05 -▁effects 4.7658358819632017e-05 -itt 4.7624263014435138e-05 -▁schriftlich 4.7624263014435138e-05 -▁Ant 4.760910932323653e-05 -way 4.7590167209238267e-05 -▁opportunities 4.7582590363638962e-05 -ek 4.75636482496407e-05 -▁behind 4.7559859826841043e-05 -ven 4.7556071404041388e-05 -▁geführt 4.7521975598844516e-05 -▁300 4.7487879793647638e-05 -▁Regierungen 4.7487879793647638e-05 -▁Erfahrung 4.7480302948048334e-05 -▁popular 4.746136083405007e-05 -ark 4.7423476606053543e-05 -▁stark 4.7419688183253894e-05 -▁possibility 4.7404534492055279e-05 -▁Spain 4.7396957646455975e-05 -ht 4.7362861841259096e-05 -▁Pl 4.7359073418459448e-05 -▁Fach 4.7347708150060488e-05 -ame 4.7340131304461184e-05 -▁commercial 4.7324977613262569e-05 -▁speak 4.7321189190462921e-05 -▁summer 4.7298458653665e-05 -▁entwickeln 4.72908818080657e-05 -▁external 4.7271939694067434e-05 -▁Dabei 4.726436284846813e-05 -▁accommodation 4.726436284846813e-05 -▁easily 4.7241632311670211e-05 -▁ne 4.7234055466070907e-05 -▁Sc 4.7230267043271251e-05 -▁Open 4.7199959660874028e-05 -▁äußerst 4.71620754328775e-05 -assen 4.71544985872782e-05 -▁setting 4.7150710164478541e-05 -.1 4.7143133318879237e-05 -▁atmosphere 4.7139344896079582e-05 -▁materials 4.7120402782081318e-05 -▁bisher 4.7112825936482014e-05 -enen 4.7074941708485487e-05 -raum 4.7059788017286879e-05 -▁oil 4.7044634326088264e-05 -kon 4.7040845903288615e-05 -ock 4.7029480634889656e-05 -▁follow 4.7029480634889656e-05 -и 4.7010538520891392e-05 -▁reached 4.7006750098091737e-05 -▁offen 4.7002961675292088e-05 -▁providing 4.695750060169625e-05 -▁verstehen 4.695750060169625e-05 -▁Einführung 4.6949923756096946e-05 -▁ten 4.6915827950900074e-05 -▁Organisation 4.6908251105300763e-05 -▁exchange 4.6874155300103891e-05 -▁kurz 4.6874155300103891e-05 -istische 4.6859001608905276e-05 -▁video 4.6859001608905276e-05 -lagen 4.6851424763305972e-05 -▁response 4.6840059494907013e-05 -▁York 4.6824905803708405e-05 -▁prepared 4.68173289581091e-05 -▁Trans 4.6787021575711877e-05 -▁Italy 4.6768079461713614e-05 -kommen 4.6764291038913958e-05 -▁reduce 4.674913734771535e-05 -▁Nur 4.6737772079316391e-05 -iver 4.6730195233717086e-05 -▁Flug 4.6730195233717086e-05 -ear 4.6703676274119512e-05 -▁sustainable 4.6684734160121248e-05 -▁Amsterdam 4.6643061509325072e-05 -▁relevant 4.6624119395326809e-05 -zug 4.6593812012929586e-05 -▁creation 4.6586235167330282e-05 -▁head 4.6578658321730971e-05 -▁hohen 4.6548350939333748e-05 -▁neuer 4.6529408825335484e-05 -▁Standpunkt 4.6514255134136876e-05 -▁lediglich 4.6499101442938268e-05 -▁Schw 4.6483947751739653e-05 -▁Position 4.6468794060541045e-05 -▁tool 4.6468794060541045e-05 -▁North 4.6465005637741389e-05 -eth 4.645364036934243e-05 -▁Anzahl 4.6446063523743126e-05 -▁somit 4.6446063523743126e-05 -▁pressure 4.644227510094347e-05 -▁Republic 4.6430909832544517e-05 -▁Umgebung 4.6430909832544517e-05 -▁bevor 4.6400602450147294e-05 -nen 4.6377871913349375e-05 -pt 4.6377871913349375e-05 -▁decided 4.6366506644950416e-05 -.00 4.6298315034556665e-05 -geb 4.6286949766157706e-05 -▁gewährleisten 4.62717960749591e-05 -▁Film 4.6252853960960834e-05 -ja 4.6199816041765692e-05 -▁em 4.6196027618966043e-05 -▁digital 4.6192239196166388e-05 -▁Bau 4.6184662350567084e-05 -▁Schl 4.6158143390969516e-05 -▁Familie 4.6127836008572293e-05 -OR 4.6116470740173333e-05 -▁port 4.6101317048974725e-05 -▁size 4.6078586512176806e-05 -▁submitted 4.60710096665775e-05 -▁coming 4.6044490706979927e-05 -▁capacity 4.6033125438580975e-05 -▁gebracht 4.60066064789834e-05 -▁closed 4.5983875942185488e-05 -▁St. 4.5976299096586177e-05 -▁non- 4.5961145405387569e-05 -act 4.5949780136988609e-05 -▁style 4.5930838022990346e-05 -▁gives 4.5900530640593123e-05 -▁Bus 4.5889165372194163e-05 -▁wirtschaftliche 4.587780010379521e-05 --b 4.58702232581959e-05 -▁Ro 4.5862646412596595e-05 -▁details 4.5862646412596595e-05 -▁Pro 4.5832339030199372e-05 -▁places 4.5824762184600068e-05 -▁Sprache 4.5820973761800413e-05 -▁British 4.580960849340146e-05 -▁Fehler 4.5764147419805622e-05 -igten 4.5756570574206318e-05 -▁26 4.574141688300771e-05 -akt 4.5737628460208054e-05 -▁increasing 4.57338400374084e-05 -▁Convention 4.573005161460875e-05 --t 4.5688378963812568e-05 -▁res 4.5677013695413608e-05 -▁remains 4.5673225272613959e-05 -▁FR 4.5661860004215e-05 -▁WTO 4.5658071581415344e-05 -▁Neben 4.5654283158615696e-05 -▁leben 4.5639129467417081e-05 -▁ausgestattet 4.5635341044617432e-05 -uren 4.5631552621817777e-05 -▁concept 4.5608822085019858e-05 -▁x 4.5586091548221945e-05 -▁sites 4.558230312542229e-05 -asse 4.5567149434223682e-05 -▁actions 4.5544418897425763e-05 -▁rec 4.5544418897425763e-05 -▁obwohl 4.55254767834275e-05 -▁supply 4.5510323092228891e-05 -ities 4.5498957823829931e-05 -ierungs 4.5487592555430972e-05 -ire 4.5472438864232357e-05 -mann 4.5464862018633053e-05 -haus 4.5430766213436181e-05 -▁train 4.5423189367836877e-05 -▁Also 4.5381516717040694e-05 -▁March 4.5351209334643471e-05 -dr 4.5343632489044167e-05 -▁Africa 4.5343632489044167e-05 -▁Cl 4.5336055643444863e-05 -▁December 4.5324690375045904e-05 -▁anderer 4.5324690375045904e-05 -▁property 4.5317113529446593e-05 -▁enlargement 4.5309536683847289e-05 -▁priority 4.5298171415448329e-05 -▁est 4.5264075610251457e-05 -▁Kommissarin 4.5245133496253194e-05 -▁island 4.5214826113855971e-05 -▁bestimmte 4.5207249268256666e-05 -▁cause 4.5184518731458747e-05 -▁document 4.5184518731458747e-05 -▁credit 4.5165576617460484e-05 -▁Integration 4.514663450346222e-05 -ces 4.5142846080662565e-05 -▁inform 4.5142846080662565e-05 -▁quiet 4.5142846080662565e-05 -arm 4.5131480812263612e-05 -▁table 4.5101173429866389e-05 -OS 4.5089808161467429e-05 -▁Erachtens 4.5089808161467429e-05 -mittel 4.5086019738667774e-05 -▁partner 4.5086019738667774e-05 -ues 4.5082231315868125e-05 -ux 4.5082231315868125e-05 -▁Ukraine 4.5082231315868125e-05 -▁balance 4.507844289306847e-05 -▁leave 4.5055712356270551e-05 -▁Bekämpfung 4.5014039705474375e-05 -▁Kosovo 4.5014039705474375e-05 -▁Dinge 4.4991309168676456e-05 -▁Freiheit 4.4983732323077152e-05 -▁meines 4.4972367054678192e-05 -▁billion 4.4945848095080625e-05 -▁Vertrags 4.4942059672280969e-05 -ucht 4.4934482826681665e-05 -▁Produktion 4.49155407126834e-05 -▁zahlreiche 4.49155407126834e-05 -ible 4.4889021753085834e-05 -▁consumer 4.4881444907486523e-05 -▁six 4.4877656484686874e-05 -▁zweite 4.4877656484686874e-05 -▁Museum 4.4854925947888955e-05 --N 4.4805676451493468e-05 -▁35 4.4794311183094509e-05 -▁CD 4.4767792223496941e-05 -lage 4.4764003800697293e-05 -iel 4.4760215377897637e-05 -burg 4.472233114990111e-05 -▁ange 4.4714754304301806e-05 -▁Dr 4.4699600613103191e-05 -▁limited 4.4692023767503887e-05 -stellen 4.4680658499104927e-05 -▁funding 4.4661716385106664e-05 -ma 4.4657927962307015e-05 -▁1998 4.4654139539507359e-05 -▁Landwirtschaft 4.4650351116707704e-05 -ios 4.4646562693908055e-05 -▁Lissabon 4.4646562693908055e-05 -▁former 4.4638985848308745e-05 -con 4.4627620579909792e-05 -▁bestimmten 4.4623832157110136e-05 -▁einschließlich 4.4582159506313954e-05 -ome 4.4555640546716386e-05 -ash 4.4551852123916731e-05 -▁Western 4.4548063701117082e-05 -▁90 4.4510179473120555e-05 -▁2. 4.449502578192194e-05 -▁assistance 4.4487448936322636e-05 -▁age 4.4457141553925413e-05 -inn 4.4423045748728541e-05 -▁sign 4.4407892057529926e-05 -▁sk 4.4400315211930622e-05 -▁Wachstum 4.4392738366331318e-05 -▁Aktivitäten 4.4381373097932358e-05 -▁Fortschritte 4.4373796252333054e-05 -red 4.4332123601536871e-05 -erg 4.4316969910338263e-05 -schen 4.429802779634e-05 -over 4.4294239373540344e-05 -▁supported 4.4290450950740689e-05 -▁movement 4.4263931991143121e-05 -ack 4.4248778299944513e-05 -▁ground 4.4241201454345209e-05 -▁tragen 4.4237413031545553e-05 -igte 4.42336246087459e-05 -▁scheint 4.4195740380749371e-05 -▁independent 4.41692214211518e-05 -ements 4.4135125615954924e-05 -ry 4.4104818233557708e-05 -na 4.4089664542359093e-05 -rad 4.4089664542359093e-05 -▁Club 4.4055568737162221e-05 -▁Zur 4.4055568737162221e-05 -▁forms 4.4036626623163958e-05 -bs 4.4029049777564647e-05 -iden 4.4029049777564647e-05 -▁Co 4.4025261354765e-05 -mal 4.4013896086366039e-05 -▁nie 4.3987377126768471e-05 -ca 4.3972223435569856e-05 -▁Dezember 4.3960858167170896e-05 -▁bet 4.3949492898771944e-05 -wert 4.3934339207573329e-05 -▁Mon 4.393055078477368e-05 -▁charge 4.3904031825176106e-05 -ient 4.38964549795768e-05 -▁außerdem 4.38964549795768e-05 -▁erster 4.3892666556777146e-05 -▁brought 4.3866147597179578e-05 -ves 4.386235917437993e-05 -sche 4.3854782328780619e-05 -▁restaurants 4.3828263369183051e-05 -ieb 4.3801744409585483e-05 -▁sagte 4.3794167563986179e-05 -▁versuchen 4.3794167563986179e-05 -▁regulations 4.3786590718386869e-05 -▁allein 4.3744918067590693e-05 -▁Preise 4.3733552799191733e-05 -▁speed 4.3725975953592429e-05 -EL 4.370324541679451e-05 -▁Hoch 4.3695668571195206e-05 -▁pleased 4.3684303302796246e-05 -▁Funktion 4.3665361188797983e-05 -▁Italian 4.3653995920399023e-05 -▁Gruppen 4.3650207497599368e-05 -CH 4.3642630652000064e-05 -▁mobile 4.3612323269602841e-05 -▁Mehrheit 4.3600958001203888e-05 -▁January 4.3585804310005273e-05 -▁variety 4.355549692760805e-05 -▁creating 4.3536554813609786e-05 -▁Dem 4.3517612699611523e-05 -▁solchen 4.3506247431212563e-05 -▁Medien 4.3502459008412914e-05 -▁Jahrhundert 4.3498670585613259e-05 -▁airport 4.34873053172143e-05 -ows 4.3472151626015691e-05 -▁gew 4.3472151626015691e-05 -▁Konferenz 4.3464574780416387e-05 -▁dadurch 4.3464574780416387e-05 -▁Stelle 4.3460786357616732e-05 -▁Gefahr 4.3456997934817076e-05 -▁rule 4.3456997934817076e-05 -▁sur 4.3456997934817076e-05 -ont 4.3434267398019164e-05 -▁zweiten 4.3419113706820549e-05 -▁insgesamt 4.3403960015621941e-05 -kr 4.3396383170022637e-05 -▁gemäß 4.3392594747222981e-05 -▁Stunden 4.3377441056024373e-05 -▁hohe 4.3377441056024373e-05 -▁Gu 4.3366075787625414e-05 -▁sofort 4.3343345250827495e-05 -▁Bestimmungen 4.3324403136829231e-05 -▁Benutzer 4.3316826291229927e-05 -▁construction 4.3316826291229927e-05 -▁nor 4.3286518908832704e-05 -▁Party 4.3252423103635832e-05 -▁Bel 4.3244846258036521e-05 -▁apartment 4.3233480989637568e-05 -▁28 4.3210750452839649e-05 -unden 4.3176654647642771e-05 -▁safe 4.3172866224843122e-05 -▁contract 4.3169077802043467e-05 -▁link 4.31501356880452e-05 -▁Erklärung 4.3138770419646244e-05 -▁discussion 4.3134981996846595e-05 -▁qu 4.310846303724902e-05 -▁alten 4.3104674614449372e-05 -▁log 4.3097097768850068e-05 -▁purpose 4.3089520923250757e-05 -▁October 4.3085732500451108e-05 -ingen 4.3081944077651453e-05 -programm 4.3070578809252493e-05 -▁message 4.3070578809252493e-05 -▁einzige 4.3025117735656662e-05 -▁largest 4.30137524672577e-05 -▁instead 4.3002387198858743e-05 -▁Star 4.2979656662060831e-05 -▁DE 4.2945560856863952e-05 -és 4.2937984011264648e-05 -enz 4.2915253474466729e-05 -▁aufgenommen 4.2907676628867425e-05 -▁Frühstück 4.2900099783268121e-05 -amb 4.2896311360468465e-05 -führung 4.2892522937668817e-05 -ET 4.28773692464702e-05 -ory 4.2847061864072979e-05 -enten 4.284327344127333e-05 -▁geb 4.2839485018473674e-05 -lo 4.2828119750074715e-05 -▁erneut 4.2816754481675762e-05 -▁ca. 4.2797812367677492e-05 -▁produced 4.2797812367677492e-05 -▁code 4.2782658676478884e-05 -▁Aufmerksamkeit 4.277508183087958e-05 -▁became 4.2767504985280275e-05 -▁PL 4.2759928139680965e-05 -ited 4.275235129408166e-05 -▁vergangenen 4.2748562871282012e-05 -▁B. 4.2737197602883052e-05 -▁conference 4.2718255488884789e-05 -▁avoid 4.26690059924893e-05 -▁stability 4.2574295422497984e-05 -▁Booking.com 4.2570506999698328e-05 -ini 4.2555353308499713e-05 -▁interesting 4.2543988040100761e-05 -▁Basis 4.2521257503302842e-05 -▁school 4.2521257503302842e-05 -▁study 4.2506103812104234e-05 -▁Staats 4.2502315389304578e-05 -unternehmen 4.2464431161308051e-05 -▁Server 4.2449277470109436e-05 -▁Ste 4.2445489047309787e-05 -orn 4.2441700624510132e-05 -▁express 4.2411393242112909e-05 -▁Entscheidungen 4.2362143745717422e-05 -▁ways 4.2354566900118118e-05 -▁click 4.2350778477318469e-05 -▁Ha 4.233941320891951e-05 -▁Flughafen 4.2335624786119854e-05 -▁steps 4.2331836363320206e-05 -▁Steuer 4.2316682672121591e-05 -▁expect 4.2312894249321942e-05 -▁film 4.2309105826522287e-05 -о 4.2301528980922982e-05 --Z 4.2271221598525759e-05 -▁item 4.2267433175726104e-05 -eit 4.2263644752926455e-05 -▁gesch 4.22598563301268e-05 -esten 4.224849106172784e-05 -▁März 4.224849106172784e-05 -▁erfolgreich 4.2233337370529232e-05 -▁sent 4.2221972102130273e-05 -▁Today 4.2214395256530968e-05 -US 4.2203029988132009e-05 -▁effort 4.2203029988132009e-05 -▁genießen 4.2203029988132009e-05 -ote 4.2191664719733049e-05 -▁Reg 4.2191664719733049e-05 -▁started 4.2187876296933394e-05 -arbeit 4.2176511028534441e-05 -▁Entwicklungs 4.216893418293513e-05 -▁rich 4.2153780491736522e-05 -▁ändern 4.21234731093393e-05 -▁Punkte 4.2096954149741731e-05 -▁schwer 4.2081800458543116e-05 -▁Golf 4.2078012035743468e-05 -▁Oktober 4.2078012035743468e-05 -gest 4.2074223612943812e-05 -▁los 4.2059069921745204e-05 -▁allowed 4.2036339384947285e-05 -▁regards 4.1990878311351454e-05 -▁origin 4.1941628814955967e-05 -▁task 4.1934051969356663e-05 -▁Wettbewerb 4.1862071936163257e-05 -▁Urlaub 4.18582835133636e-05 -▁produce 4.1854495090563953e-05 -▁source 4.1854495090563953e-05 -▁fishing 4.1839341399365338e-05 -▁Anforderungen 4.182418770816673e-05 -AM 4.181282243976777e-05 -▁vertreten 4.181282243976777e-05 -▁activity 4.1793880325769507e-05 -▁procedures 4.17787266345709e-05 -schutz 4.1774938211771243e-05 -.2 4.1767361366171939e-05 -▁Küche 4.1763572943372284e-05 -ald 4.1748419252173676e-05 -▁Real 4.17219002925761e-05 -▁Unternehmens 4.1699169755778189e-05 -▁ges 4.1699169755778189e-05 -ff 4.1695381332978533e-05 -▁tell 4.1691592910178885e-05 -ged 4.1687804487379229e-05 -▁Gästebewertungen 4.166507395058131e-05 -▁status 4.1646131836583047e-05 -▁download 4.16423434137834e-05 -▁Services 4.1630978145384438e-05 -▁compromise 4.1627189722584783e-05 -ators 4.1623401299785134e-05 -▁operation 4.1615824454185823e-05 -▁Wochen 4.1600670762987215e-05 -aut 4.1566574957790344e-05 -▁helpful 4.1558998112191033e-05 -isiert 4.1551421266591729e-05 -▁Sicht 4.152869072979381e-05 -▁Berichts 4.15135370385952e-05 -▁Italien 4.1509748615795546e-05 -▁correct 4.15059601929959e-05 -▁Air 4.1498383347396593e-05 -▁review 4.1487018078997634e-05 -▁legislative 4.1483229656197978e-05 -isieren 4.147944123339833e-05 -▁Rights 4.1464287542199715e-05 -▁Mac 4.1449133851001106e-05 -▁Juli 4.1426403314203187e-05 -▁contribution 4.1415038045804228e-05 -▁Villa 4.1411249623004579e-05 -▁Bilder 4.1392307509006316e-05 -▁Öffentlichkeit 4.1380942240607356e-05 -▁geschlossen 4.13771538178077e-05 -▁Beschäftigung 4.1369576972208397e-05 -▁schließlich 4.1362000126609092e-05 -▁word 4.1316539053013254e-05 -▁größten 4.1312750630213606e-05 -af 4.1297596939014991e-05 -pe 4.1297596939014991e-05 -verfahren 4.1297596939014991e-05 -▁Vorschriften 4.1293808516215342e-05 -▁Spanish 4.1263501133818119e-05 -▁park 4.122940532862124e-05 -▁Ben 4.1221828483021936e-05 -▁leider 4.1210463214622977e-05 -▁Staat 4.1191521100624713e-05 -▁erwarten 4.1142271604229226e-05 -▁gen 4.1142271604229226e-05 -ively 4.1134694758629922e-05 -▁damage 4.1130906335830267e-05 -▁village 4.1119541067431314e-05 -▁Bereiche 4.1111964221832e-05 -▁written 4.11043873762327e-05 -▁exp 4.1096810530633395e-05 -▁Luft 4.109302210783374e-05 -isen 4.1077868416635132e-05 -▁Transparenz 4.1077868416635132e-05 -▁Grenzen 4.1066503148236172e-05 -▁eigentlich 4.1062714725436523e-05 -▁distance 4.1058926302636868e-05 -led 4.1043772611438253e-05 -▁class 4.1039984188638604e-05 -▁Fund 4.1028618920239645e-05 -stein 4.1009676806241381e-05 -ime 4.1002099960642077e-05 -ane 4.0990734692243118e-05 -▁provisions 4.0971792578244854e-05 -▁eingesetzt 4.096421573264555e-05 -▁konnten 4.0949062041446935e-05 -▁instruments 4.0945273618647286e-05 -▁During 4.0933908350248327e-05 -par 4.0930119927448671e-05 -▁Development 4.0926331504649023e-05 -son 4.0918754659049712e-05 -ären 4.0892235699452144e-05 -▁cr 4.0839197780257009e-05 -ör 4.0835409357457353e-05 -ari 4.0808890397859786e-05 -▁relation 4.0793736706661171e-05 -▁Höhe 4.0786159861061867e-05 -▁sun 4.0774794592662907e-05 -▁giving 4.0755852478664643e-05 -▁ES 4.0729333519067076e-05 -▁option 4.0729333519067076e-05 -▁Ber 4.072554509626742e-05 -▁leisten 4.072554509626742e-05 -▁plant 4.0714179827868461e-05 -▁got 4.0710391405068812e-05 -▁historical 4.0699026136669853e-05 -▁Site 4.06952377138702e-05 -▁Anfrage 4.0680084022671589e-05 -▁labour 4.0680084022671589e-05 -▁statement 4.0668718754272629e-05 -▁wenige 4.065735348587367e-05 -bo 4.0634622949075758e-05 -▁documents 4.06308345262761e-05 -▁Ansatz 4.0627046103476447e-05 -▁July 4.0627046103476447e-05 -▁Portugal 4.0611892412277839e-05 -▁poor 4.0611892412277839e-05 -▁ourselves 4.0604315566678534e-05 -▁Arbeitnehmer 4.0600527143878879e-05 -▁Folgen 4.0600527143878879e-05 -▁criteria 4.0592950298279575e-05 -▁Min 4.0547489224683744e-05 -▁draw 4.0479297614289993e-05 -▁Präsidenten 4.046035550029173e-05 -▁build 4.0418682849495547e-05 -erb 4.0411106003896243e-05 -▁Rad 4.0373221775899716e-05 -▁base 4.036943335310006e-05 -▁geworden 4.0346702816302141e-05 -barkeit 4.0335337547903189e-05 -ational 4.0323972279504229e-05 -▁Bildung 4.0323972279504229e-05 -uk 4.0312607011105269e-05 -▁genommen 4.0312607011105269e-05 -12 4.030124174270631e-05 -▁conflict 4.030124174270631e-05 -▁efficient 4.0293664897107006e-05 -▁Groß 4.02860880515077e-05 -▁durchgeführt 4.02860880515077e-05 -▁Diskussion 4.0278511205908391e-05 -▁Sa 4.0278511205908391e-05 -▁Such 4.0278511205908391e-05 -▁Computer 4.0244415400711519e-05 -▁Many 4.0240626977911864e-05 -▁Pre 4.0229261709512911e-05 -▁violence 4.0229261709512911e-05 -▁Meine 4.0210319595514641e-05 -▁benefits 4.0210319595514641e-05 -▁types 4.0195165904316032e-05 -▁Sommer 4.0180012213117424e-05 -▁draft 4.0180012213117424e-05 -▁structure 4.0172435367518113e-05 -▁news 4.0157281676319505e-05 -▁benötigen 4.015349325351985e-05 -▁wichtiger 4.015349325351985e-05 -▁Systems 4.0115609025523322e-05 -▁fun 4.0043628992329923e-05 -anten 4.002468687833166e-05 -▁ind 4.002468687833166e-05 -▁Investitionen 4.0017110032732349e-05 -▁require 4.0009533187133045e-05 -▁Januar 3.9990591073134781e-05 -▁suchen 3.995649526793791e-05 -▁wine 3.99489184223386e-05 -▁Nationen 3.994512999953895e-05 -▁ant 3.9937553153939646e-05 -kosten 3.9903457348742767e-05 -▁aktuellen 3.98769383891452e-05 -▁1997 3.9861784697946585e-05 -ex 3.9846631006747977e-05 -▁highest 3.9827688892749713e-05 -ging 3.9816323624350753e-05 -▁Präsidentschaft 3.9808746778751449e-05 -▁immediately 3.9793593087552834e-05 -▁Firma 3.9778439396354226e-05 -▁useful 3.9767074127955267e-05 --2 3.9759497282355963e-05 --l 3.97405551683577e-05 -▁kannst 3.9721613054359435e-05 -▁star 3.971782463155978e-05 -▁Although 3.9714036208760131e-05 -▁gel 3.9710247785960476e-05 -▁lower 3.9698882517561516e-05 -geben 3.9691305671962212e-05 -▁Fre 3.9676151980763604e-05 -▁Your 3.9668575135164293e-05 -▁distribution 3.9619325638768806e-05 -▁gas 3.9615537215969158e-05 -▁nahe 3.9596595101970894e-05 -J 3.9581441410772279e-05 -eil 3.9554922451174711e-05 -▁University 3.9528403491577144e-05 -ierter 3.9524615068777488e-05 -▁europäischer 3.952082664597784e-05 -ult 3.950946137757888e-05 -▁Sinne 3.9490519263580617e-05 -▁kennen 3.9479153995181657e-05 -▁elements 3.9464000303983042e-05 -▁transfer 3.9464000303983042e-05 -▁transparency 3.9464000303983042e-05 -ias 3.9445058189984778e-05 -ai 3.942990449878617e-05 -▁integrated 3.9426116075986515e-05 -▁friends 3.9422327653186866e-05 -▁finally 3.9407173961988251e-05 -▁ihres 3.9407173961988251e-05 -▁method 3.9407173961988251e-05 -▁Gründen 3.94033855391886e-05 -ada 3.9399597116388947e-05 -▁verb 3.9399597116388947e-05 -▁alternative 3.9392020270789643e-05 -etz 3.9354136042793116e-05 -inen 3.935034761999346e-05 -▁gefunden 3.935034761999346e-05 -ites 3.9342770774394156e-05 -industrie 3.9327617083195548e-05 -▁Sal 3.9320040237596237e-05 -▁normal 3.9301098123597974e-05 -▁section 3.9278367586800061e-05 -ology 3.92670023184011e-05 -▁employees 3.92670023184011e-05 -▁weder 3.9263213895601446e-05 -erk 3.9236694936003879e-05 -▁parking 3.9225329667604919e-05 -▁Can 3.9191233862408047e-05 -▁late 3.9187445439608392e-05 -ability 3.9179868594009088e-05 -▁Plan 3.9179868594009088e-05 -▁Aufenthalt 3.9168503325610128e-05 -▁rel 3.9168503325610128e-05 -ath 3.9157138057211169e-05 -▁agenda 3.9157138057211169e-05 -▁angesichts 3.9141984366012561e-05 -▁child 3.9138195943212905e-05 -hof 3.9111676983615338e-05 -▁mountain 3.9111676983615338e-05 -▁Stand 3.9104100138016033e-05 -▁Projekte 3.9100311715216378e-05 -▁Centre 3.9096523292416723e-05 -▁equal 3.9070004332819155e-05 -OM 3.901696641362402e-05 -▁round 3.9013177990824364e-05 -acht 3.9009389568024709e-05 -▁drive 3.9001812722425405e-05 -▁numerous 3.8982870608427141e-05 -▁bezüglich 3.8979082185627492e-05 -▁Auffassung 3.8975293762827837e-05 -▁Furthermore 3.8960140071629222e-05 -iers 3.893740953483131e-05 -▁challenges 3.892604426643235e-05 -▁Sache 3.8922255843632695e-05 -▁applied 3.8914678998033391e-05 -▁halte 3.8914678998033391e-05 -è 3.8884371615636167e-05 -▁reading 3.88578526560386e-05 -▁29 3.8854064233238944e-05 -halt 3.8827545273641377e-05 -ella 3.8823756850841728e-05 -▁daily 3.8812391582442768e-05 -▁Gra 3.8801026314043809e-05 -▁Heute 3.8801026314043809e-05 -hip 3.8789661045644849e-05 -▁Fahr 3.8789661045644849e-05 -ians 3.8785872622845194e-05 -▁tools 3.877829577724589e-05 -itions 3.8763142086047282e-05 -OL 3.8747988394848667e-05 -▁Sol 3.8740411549249363e-05 -amen 3.8736623126449714e-05 -▁reality 3.8713892589651795e-05 -▁bathroom 3.8676008361655268e-05 -▁elections 3.8672219938855612e-05 -▁poverty 3.8660854670456653e-05 -▁EC 3.86494894020577e-05 -▁begrüße 3.86494894020577e-05 -▁payment 3.86494894020577e-05 -va 3.8638124133658741e-05 -▁Fuß 3.8619182019660477e-05 -▁Cr 3.8611605174061166e-05 -ara 3.8607816751261517e-05 -▁processes 3.8596451482862558e-05 -▁Beginn 3.8566144100465335e-05 -▁apartments 3.8566144100465335e-05 -zer 3.8562355677665686e-05 -▁offering 3.8562355677665686e-05 -▁# 3.8558567254866031e-05 -ular 3.8547201986467071e-05 -▁wider 3.8547201986467071e-05 -▁Test 3.8543413563667416e-05 -ön 3.8528259872468808e-05 -möglichkeiten 3.8516894604069848e-05 -▁Free 3.8516894604069848e-05 -hin 3.85131061812702e-05 -▁sieht 3.8509317758470544e-05 -▁eher 3.8505529335670888e-05 -▁death 3.8463856684874713e-05 -▁illegal 3.84562798392754e-05 --An 3.8437337725277138e-05 -▁solidarity 3.8429760879677834e-05 -▁consequences 3.842218403407853e-05 -▁­ 3.841081876567957e-05 -▁led 3.8403241920080266e-05 -▁treffen 3.8403241920080266e-05 -▁bringt 3.8380511383282347e-05 -iker 3.835399242368478e-05 -▁Regeln 3.834262715528582e-05 --In 3.8331261886886861e-05 -▁enter 3.8319896618487908e-05 -▁saying 3.83123197728886e-05 -teil 3.8304742927289293e-05 -▁Forschungs 3.8300954504489644e-05 -▁beste 3.8289589236090685e-05 -▁Linie 3.827443554489207e-05 -▁talk 3.8266858699292766e-05 -▁bes 3.8259281853693462e-05 -▁gra 3.8255493430893806e-05 -▁analysis 3.82479165852945e-05 -▁Ba 3.8236551316895542e-05 -▁diejenigen 3.8232762894095894e-05 -ba 3.819487866609936e-05 -▁Per 3.819487866609936e-05 -▁ziehen 3.8168359706501792e-05 -▁Gesch 3.8156994438102833e-05 -▁Schlusselwortern 3.8156994438102833e-05 -▁Tagesordnung 3.8156994438102833e-05 -▁gelten 3.814562916970388e-05 -▁beyond 3.8141840746904224e-05 -▁green 3.8138052324104569e-05 -▁implement 3.8092591250508738e-05 -bt 3.8088802827709082e-05 -▁entsprechenden 3.8085014404909434e-05 -▁öffentliche 3.8085014404909434e-05 -▁beitragen 3.806607229091117e-05 -▁instrument 3.8058495445311866e-05 -▁paid 3.8039553331313596e-05 -▁End 3.8035764908513947e-05 -ott 3.8009245948916379e-05 -▁official 3.8001669103317068e-05 -istic 3.799788068051742e-05 -▁entsprechend 3.799788068051742e-05 -▁colleagues 3.7982726989318805e-05 -▁happy 3.7967573298120197e-05 -▁offered 3.7967573298120197e-05 -bel 3.7952419606921582e-05 -▁board 3.7948631184121933e-05 -ierungen 3.7925900647324014e-05 -▁goes 3.7914535378925054e-05 -▁Kar 3.790695853332575e-05 -press 3.7899381687726446e-05 -▁practical 3.7876651150928527e-05 -▁Aufgaben 3.7865285882529568e-05 -weg 3.7846343768531304e-05 -esch 3.7834978500132351e-05 -▁Besuch 3.7819824808933736e-05 -ably 3.7816036386134088e-05 -▁Human 3.7804671117735128e-05 -▁exist 3.7793305849336169e-05 -▁thought 3.7793305849336169e-05 -▁Bemühungen 3.777057531253825e-05 -▁persons 3.777057531253825e-05 -▁@ 3.77667868897386e-05 -▁requires 3.7751633198539986e-05 -▁Fest 3.7725114238942418e-05 -zu 3.7717537393343114e-05 -▁Verkehr 3.7717537393343114e-05 -ika 3.7706172124944155e-05 -▁vergessen 3.7698595279344851e-05 -res 3.7668287896947627e-05 -▁Fällen 3.7664499474147972e-05 -▁31 3.7660711051348323e-05 -▁Nr. 3.7634192091750749e-05 -iet 3.76304036689511e-05 -▁ideas 3.76304036689511e-05 -▁Mediterranean 3.7626615246151445e-05 -▁bit 3.7611461554952837e-05 -▁Dr. 3.7596307863754222e-05 -▁Monaten 3.7596307863754222e-05 -▁carry 3.7588731018154918e-05 -▁minimum 3.7554635212958046e-05 -▁warm 3.7543269944559086e-05 -▁Restaurants 3.7535693098959775e-05 -▁van 3.7535693098959775e-05 -▁tour 3.7512962562161863e-05 -▁Auto 3.74940204481636e-05 -▁cut 3.7490232025363944e-05 -▁Änderung 3.748265517976464e-05 -ying 3.7475078334165336e-05 -ungs- 3.7452347797367417e-05 -less 3.7444770951768113e-05 -▁Microsoft 3.7425828837769849e-05 -▁Even 3.7418251992170545e-05 -int 3.7410675146571234e-05 -▁zehn 3.7410675146571234e-05 -▁Wohn 3.7372790918574707e-05 -▁Public 3.7353848804576443e-05 -▁verhindern 3.7353848804576443e-05 -vent 3.7331118267778531e-05 -ider 3.732354142217922e-05 -ün 3.732354142217922e-05 -lauf 3.7297022462581652e-05 -nisse 3.7274291925783733e-05 -cl 3.7262926657384781e-05 -EC 3.7259138234585125e-05 -▁Bürgern 3.7251561388985821e-05 --up 3.7243984543386517e-05 -▁getan 3.72288308521879e-05 -▁forces 3.7217465583788943e-05 -▁Ressourcen 3.7209888738189639e-05 -▁historic 3.7209888738189639e-05 -▁shopping 3.720610031538999e-05 -▁behandelt 3.7198523469790679e-05 -▁erfolgt 3.7190946624191375e-05 -▁advantage 3.7134120282196584e-05 -▁congratulate 3.7103812899799361e-05 -▁refer 3.7096236054200057e-05 -igungen 3.70848707858011e-05 -▁Instrument 3.7081082363001442e-05 -▁modified 3.7077293940201793e-05 -▁Austria 3.7065928671802834e-05 -arten 3.7062140249003178e-05 -▁Bundes 3.7062140249003178e-05 -▁Wunsch 3.7062140249003178e-05 -▁tomorrow 3.7062140249003178e-05 -▁32 3.705835182620353e-05 -▁organisations 3.7043198135004915e-05 -▁Grand 3.7028044443806307e-05 -▁Produktions 3.7028044443806307e-05 -▁Zusammen 3.7028044443806307e-05 -▁hätten 3.7005313907008387e-05 -▁clients 3.6986371793010124e-05 -▁waste 3.6982583370210475e-05 -ius 3.697879494741082e-05 -▁Security 3.6971218101811516e-05 -▁Tagen 3.6971218101811516e-05 -▁doubt 3.6971218101811516e-05 -hn 3.6963641256212212e-05 -▁sales 3.6963641256212212e-05 -▁weeks 3.6963641256212212e-05 -▁genannten 3.6937122296614637e-05 -▁crucial 3.6914391759816725e-05 -▁Des 3.6895449645818461e-05 --St 3.6891661223018806e-05 -▁starting 3.688787280021915e-05 -ail 3.68765075318202e-05 -▁implemented 3.6861353840621583e-05 -▁nimmt 3.6849988572222623e-05 -▁45 3.6834834881024015e-05 -▁Gl 3.6834834881024015e-05 -▁Video 3.6804527498626792e-05 -▁server 3.6785585384628528e-05 -▁morgen 3.6766643270630265e-05 -gesch 3.6736335888233041e-05 -▁Start 3.6728759042633737e-05 -▁z. 3.6709816928635474e-05 -ko 3.6706028505835818e-05 -▁anderem 3.6698451660236514e-05 -plan 3.6694663237436859e-05 -yl 3.6675721123438595e-05 -▁Republik 3.6664355855039636e-05 -▁reference 3.6656779009440332e-05 -ect 3.6652990586640683e-05 -▁published 3.6652990586640683e-05 -ischem 3.6603741090245196e-05 -▁paar 3.6603741090245196e-05 -len 3.6588587399046581e-05 -▁operating 3.6584798976246933e-05 -IL 3.6562068439449014e-05 -weisen 3.6554491593849709e-05 -▁maximum 3.6546914748250405e-05 -▁fre 3.653176105705179e-05 -▁besondere 3.6520395788652831e-05 -▁häufig 3.6516607365853182e-05 -▁Augen 3.6505242097454223e-05 -▁hold 3.6505242097454223e-05 -▁lines 3.6505242097454223e-05 -▁str 3.6501453674654567e-05 -▁comprehensive 3.6482511560656304e-05 -▁sometimes 3.6478723137856655e-05 -▁Enterprise 3.6459781023858391e-05 -▁growing 3.6444627332659776e-05 -▁learn 3.6440838909860121e-05 -▁morning 3.6437050487060472e-05 -▁Inhalt 3.6421896795861857e-05 -▁ready 3.6421896795861857e-05 -▁wer 3.6421896795861857e-05 -▁Natürlich 3.6418108373062209e-05 -▁Atmosphäre 3.639537783626429e-05 -üt 3.6391589413464641e-05 -▁versucht 3.638401256786533e-05 -▁monitoring 3.6380224145065682e-05 -▁Behandlung 3.63461283398688e-05 -▁E-Mail 3.6330974648670195e-05 -▁machine 3.6327186225870539e-05 -▁Amendment 3.6323397803070891e-05 -▁Room 3.6312032534671931e-05 -▁efficiency 3.6312032534671931e-05 -▁Bürgerinnen 3.6300667266272972e-05 -▁lives 3.6289301997874012e-05 -▁erklärt 3.6251417769877485e-05 -etten 3.6247629347077829e-05 -ps 3.6243840924278181e-05 -wer 3.6232475655879221e-05 -▁Werte 3.6228687233079566e-05 -▁Gegen 3.6183226159483735e-05 -▁Green 3.6183226159483735e-05 -▁options 3.6171860891084775e-05 -▁Erfahrungen 3.6156707199886167e-05 -▁sechs 3.6156707199886167e-05 -▁reduction 3.6141553508687552e-05 -▁street 3.6141553508687552e-05 -▁mon 3.61377650858879e-05 -gend 3.61301882402886e-05 -▁Garten 3.61301882402886e-05 -▁Directive 3.6126399817488944e-05 -▁Werk 3.611882297188964e-05 -get 3.6111246126290329e-05 -▁Folge 3.610745770349068e-05 -▁select 3.6103669280691025e-05 -▁join 3.6092304012292065e-05 -▁DVD 3.6080938743893112e-05 -▁caused 3.6077150321093457e-05 -maßnahmen 3.60733618982938e-05 -▁fordern 3.6020323979098666e-05 -▁While 3.6016535556299011e-05 -▁sports 3.6008958710699707e-05 -▁hervor 3.60013818651004e-05 -▁Struktur 3.5990016596701443e-05 -uten 3.5986228173901788e-05 -▁relating 3.5982439751102139e-05 -▁Pol 3.5971074482703179e-05 -▁Tour 3.5971074482703179e-05 -▁überhaupt 3.595970921430422e-05 -▁ber 3.5952132368704916e-05 -▁Me 3.594834394590526e-05 -▁anything 3.5940767100305956e-05 -20 3.59369786775063e-05 -▁Mehr 3.5910459717908733e-05 -▁email 3.5910459717908733e-05 -EM 3.5887729181110821e-05 -▁She 3.5883940758311165e-05 -▁warum 3.5868787067112557e-05 -▁begin 3.5815749147917415e-05 -▁probably 3.5800595456718807e-05 -pf 3.5796807033919151e-05 -▁garden 3.5774076497121232e-05 -▁prior 3.5758922805922624e-05 -▁Lassen 3.5747557537523665e-05 -use 3.5717250155126442e-05 --V 3.5702096463927833e-05 -▁establish 3.5702096463927833e-05 -▁swimming 3.5679365927129914e-05 -▁Center 3.5675577504330259e-05 -▁Neu 3.5675577504330259e-05 -▁Spanien 3.567178908153061e-05 -▁players 3.567178908153061e-05 -▁zahlreichen 3.5652846967532347e-05 --based 3.5630116430734428e-05 -▁languages 3.5626328007934779e-05 -▁1996 3.5622539585135124e-05 -▁accordance 3.5618751162335468e-05 -▁vital 3.5614962739535819e-05 -▁border 3.5611174316736164e-05 -ional 3.5599809048337205e-05 -▁berücksichtigt 3.5588443779938245e-05 -▁Airport 3.5584655357138596e-05 -▁technologies 3.5580866934338941e-05 -issen 3.5546771129142069e-05 -▁worldwide 3.5542982706342414e-05 -▁Denn 3.553540586074311e-05 -▁richtige 3.5497521632746582e-05 -▁gewesen 3.5478579518748319e-05 -▁farmers 3.5417964753953873e-05 -ild 3.5414176331154217e-05 -▁target 3.5395234217155953e-05 -erv 3.5391445794356305e-05 -▁Gebäude 3.5391445794356305e-05 -▁tourist 3.5383868948757e-05 -▁welches 3.5380080525957345e-05 -▁größte 3.5372503680358041e-05 -▁stimmen 3.5372503680358041e-05 -▁terrorism 3.5364926834758737e-05 -▁lo 3.5353561566359778e-05 -▁stress 3.5349773143560122e-05 -▁Google 3.5338407875161163e-05 -▁Praxis 3.5334619452361514e-05 -▁Vergangenheit 3.5330831029561859e-05 -▁erfüllen 3.5311888915563595e-05 -▁Gemeinschafts 3.5304312069964291e-05 -▁Terrorismus 3.5296735224364987e-05 -▁comfort 3.5281581533166372e-05 -▁Conference 3.5266427841967764e-05 -▁mention 3.5258850996368453e-05 -▁Thank 3.5232332036770885e-05 -erst 3.5209601499972973e-05 -▁geschaffen 3.5205813077173317e-05 -▁mass 3.5202024654373662e-05 -▁´ 3.5202024654373662e-05 -▁Einzel 3.51906593859747e-05 -▁interested 3.51830825403754e-05 -endes 3.517929411757575e-05 -▁challenge 3.516792884917679e-05 -ards 3.5141409889579222e-05 -▁extent 3.5114890929981648e-05 -▁jeweiligen 3.5111102507182e-05 -ta 3.5103525661582688e-05 -ata 3.509973723878304e-05 -▁methods 3.5095948815983384e-05 -▁Comp 3.5084583547584425e-05 -▁Aspekte 3.5077006701985121e-05 -sh 3.5069429856385817e-05 -▁ones 3.5065641433586161e-05 -▁Swiss 3.50466993195879e-05 -▁south 3.50466993195879e-05 -▁Viele 3.5042910896788249e-05 -▁elegant 3.5039122473988594e-05 -▁voting 3.502018035999033e-05 -▁UK 3.5012603514391026e-05 -▁secure 3.500881509159137e-05 -▁Gesamt 3.4963354017995539e-05 -▁wählen 3.4963354017995539e-05 -▁stets 3.4948200326796931e-05 -▁zumindest 3.4948200326796931e-05 -▁historischen 3.494062348119762e-05 -▁schützen 3.494062348119762e-05 -▁Zweitens 3.4925469789999012e-05 -oth 3.4906527676000748e-05 -voll 3.4906527676000748e-05 -▁pre 3.4902739253201093e-05 -▁verpflichtet 3.4902739253201093e-05 -halten 3.4891373984802133e-05 -▁ger 3.4887585562002485e-05 -▁Finanzierung 3.4883797139202829e-05 -▁Kern 3.4880008716403181e-05 -▁scientific 3.4880008716403181e-05 -▁neues 3.4876220293603525e-05 -▁° 3.4868643448004221e-05 -▁games 3.486106660240491e-05 -▁adopt 3.4826970797208038e-05 -▁Vergleich 3.4823182374408383e-05 -▁bald 3.4815605528809079e-05 -▁hear 3.481181710600943e-05 -feld 3.4808028683209775e-05 -80 3.4762567609613943e-05 -▁release 3.4762567609613943e-05 -▁Technologie 3.4754990764014633e-05 -▁Chinese 3.4751202341214984e-05 -▁chance 3.474362549561568e-05 -▁außerhalb 3.4739837072816024e-05 -ands 3.4724683381617416e-05 -▁Winter 3.4724683381617416e-05 -▁wanted 3.4713318113218457e-05 -▁Ph 3.4705741267619153e-05 -▁hot 3.47019528448195e-05 -▁Schritte 3.4679222308021578e-05 -▁acc 3.4667857039622619e-05 -▁seines 3.4667857039622619e-05 -▁selection 3.4667857039622619e-05 -▁les 3.466406861682297e-05 -▁modernen 3.466406861682297e-05 -▁sectors 3.4656491771223666e-05 -▁walking 3.4656491771223666e-05 -iner 3.465270334842401e-05 -▁John 3.465270334842401e-05 -ID 3.4633761234425747e-05 -▁Süd 3.4629972811626091e-05 -net 3.4622395966026787e-05 -ico 3.4614819120427483e-05 -▁fine 3.458451173803026e-05 -▁Firmen 3.4580723315230605e-05 -▁maintain 3.4580723315230605e-05 -▁Opfer 3.4569358046831652e-05 -▁images 3.4546627510033733e-05 -ierende 3.4531473818835125e-05 -ierenden 3.4531473818835125e-05 -nahme 3.4512531704836854e-05 -▁ma 3.4508743282037206e-05 -ender 3.4497378013638246e-05 -AC 3.4470859054040678e-05 -ima 3.4444340094443111e-05 -š 3.4444340094443111e-05 -▁businesses 3.4444340094443111e-05 -▁disc 3.4432974826044151e-05 -▁Vertrauen 3.4395090598047624e-05 --w 3.4387513752448313e-05 -▁verantwortlich 3.4387513752448313e-05 -▁despite 3.4383725329648664e-05 -▁sport 3.4360994792850745e-05 -▁vollständig 3.4349629524451786e-05 -▁Verb 3.4334475833253178e-05 -▁Ob 3.4330687410453522e-05 -atz 3.4323110564854218e-05 -▁Mo 3.4323110564854218e-05 -za 3.4319322142054563e-05 -▁Spieler 3.4311745296455259e-05 -▁visitors 3.430795687365561e-05 -▁Berichterstatterin 3.4304168450855955e-05 -sicht 3.4292803182456995e-05 -▁Dann 3.4281437914058036e-05 -▁150 3.4273861068458731e-05 -▁lies 3.4270072645659076e-05 -▁rep 3.4270072645659076e-05 -▁long-term 3.4247342108861164e-05 -▁lost 3.4247342108861164e-05 -▁Rechtsvorschriften 3.4232188417662549e-05 -▁kam 3.4224611572063245e-05 -▁Angelegenheit 3.4198092612465677e-05 -▁genug 3.4186727344066717e-05 -▁bank 3.4182938921267062e-05 -ola 3.4179150498467413e-05 -▁Krieg 3.4179150498467413e-05 -eal 3.414884311607019e-05 -▁Frieden 3.4145054693270535e-05 -▁Binnenmarkt 3.4133689424871582e-05 -▁influence 3.4126112579272271e-05 -▁powers 3.4126112579272271e-05 -nis 3.4110958888073663e-05 -▁Ireland 3.4110958888073663e-05 -▁Mess 3.4107170465274008e-05 -▁Nun 3.4099593619674704e-05 -▁Dazu 3.40920167740754e-05 -▁heutigen 3.4084439928476095e-05 -▁Mor 3.408065150567644e-05 -▁separate 3.4061709391678176e-05 -▁persönlichen 3.4057920968878521e-05 -▁Dokument 3.4046555700479568e-05 -▁Gold 3.4016248318082345e-05 -40 3.4004883049683385e-05 -▁absolutely 3.3993517781284426e-05 -▁raised 3.3993517781284426e-05 -▁imp 3.3989729358484771e-05 -▁trying 3.3959421976087554e-05 -▁unten 3.3951845130488243e-05 -ms 3.393290301648998e-05 -▁Ober 3.393290301648998e-05 -ado 3.3929114593690331e-05 -▁Herausforderungen 3.3925326170890676e-05 -▁Schwierigkeiten 3.3917749325291371e-05 -▁accepted 3.3902595634092757e-05 -▁Allerdings 3.3895018788493452e-05 -▁technischen 3.3895018788493452e-05 -▁bel 3.3891230365693804e-05 -▁Just 3.3876076674495189e-05 -▁trotz 3.387228825169554e-05 -▁einge 3.3868499828895885e-05 -▁folgt 3.3868499828895885e-05 -▁Parteien 3.3860922983296581e-05 -▁erlaubt 3.3838192446498662e-05 -äre 3.3830615600899358e-05 -wirtschaft 3.3811673486901094e-05 -stellung 3.3807885064101438e-05 -▁ausschließlich 3.379273137290283e-05 -je 3.3747270299306992e-05 -▁Sonder 3.3720751339709425e-05 -▁stärker 3.371317449411012e-05 -▁deutschen 3.3701809225711161e-05 -▁Regel 3.36828671117129e-05 -▁bessere 3.36828671117129e-05 -▁erwartet 3.3679078688913242e-05 -▁Falls 3.365634815211533e-05 -atory 3.3652559729315674e-05 -ichte 3.3652559729315674e-05 -▁Rom 3.364498288371637e-05 -▁Vertreter 3.3629829192517762e-05 -▁usw. 3.3629829192517762e-05 -rea 3.36184639241188e-05 -▁e-mail 3.3614675501319147e-05 -▁frei 3.36108870785195e-05 -▁Moreover 3.3595733387320883e-05 -▁™ 3.3595733387320883e-05 -▁More 3.3591944964521228e-05 -▁Einige 3.3588156541721579e-05 -▁Hinsicht 3.3584368118921924e-05 -ift 3.3580579696122275e-05 -▁demands 3.3573002850522964e-05 -▁² 3.3569214427723316e-05 -▁strategic 3.3561637582124012e-05 -omm 3.3557849159324356e-05 -▁erinnern 3.3557849159324356e-05 -yr 3.3550272313725052e-05 -▁leaders 3.3508599662928869e-05 -▁Secondly 3.349723439452991e-05 -▁Prozent 3.348586912613095e-05 -fin 3.34820807033313e-05 -▁fall 3.3478292280531646e-05 -▁gelegen 3.3463138589333038e-05 -▁Angelegenheiten 3.3459350166533383e-05 -▁fields 3.3447984898134423e-05 -.3 3.3425254361336511e-05 -unt 3.3410100670137896e-05 -▁Res 3.3402523824538592e-05 -▁changed 3.3402523824538592e-05 -▁Hel 3.3394946978939288e-05 -RE 3.3376004864941024e-05 -▁Ihres 3.3376004864941024e-05 -▁borders 3.3376004864941024e-05 -▁Rede 3.3353274328143105e-05 -▁Mag 3.33456974825438e-05 -▁Licht 3.3341909059744146e-05 -▁Only 3.3341909059744146e-05 -▁overall 3.3334332214144841e-05 -▁Nacht 3.3322966945745882e-05 -▁Modell 3.3319178522946233e-05 -▁innovation 3.3315390100146578e-05 -▁path 3.3307813254547274e-05 -ru 3.330023640894797e-05 -▁namely 3.3296447986148314e-05 -cy 3.3292659563348659e-05 -▁Annahme 3.328887114054901e-05 -▁Russian 3.3281294294949706e-05 -▁IN 3.3266140603751091e-05 -ED 3.3239621644153523e-05 -▁Last 3.3232044798554219e-05 -ination 3.3224467952954908e-05 -ified 3.3201737416156996e-05 -▁eben 3.3201737416156996e-05 -ziehen 3.3197948993357341e-05 -▁Einrichtung 3.3197948993357341e-05 -▁wrong 3.3190372147758037e-05 -ales 3.3179006879359077e-05 -▁groß 3.3179006879359077e-05 -▁annual 3.3175218456559422e-05 -▁IP 3.3137334228562894e-05 -ST 3.312975738296359e-05 -ogen 3.3118392114564631e-05 -м 3.3107026846165678e-05 -▁represent 3.3107026846165678e-05 -▁übernehmen 3.3088084732167408e-05 -haft 3.3084296309367759e-05 -▁folgenden 3.3084296309367759e-05 -▁Ist 3.3065354195369495e-05 -▁erfahren 3.3065354195369495e-05 -0 3.306156577256984e-05 -AD 3.3046412081371232e-05 -▁sort 3.3046412081371232e-05 -▁falls 3.3016104698974009e-05 -markt 3.3004739430575049e-05 -▁calls 3.3000951007775394e-05 -verkehr 3.2989585739376441e-05 -▁Gas 3.2989585739376441e-05 -▁privaten 3.2989585739376441e-05 -▁Besch 3.2985797316576786e-05 -ato 3.298200889377713e-05 -UR 3.2974432048177826e-05 -▁negative 3.2974432048177826e-05 -▁cover 3.2966855202578522e-05 -▁yourself 3.2963066779778867e-05 -▁representatives 3.2955489934179562e-05 -gew 3.29441246657806e-05 -iegen 3.29365478201813e-05 -▁Med 3.2928970974581995e-05 -▁Mont 3.2894875169385116e-05 -▁allgemeinen 3.2868356209787548e-05 -▁player 3.28645677869879e-05 -▁President-in-Office 3.2856990941388589e-05 -▁Solidarität 3.285320251858894e-05 -▁etc. 3.2826683558991366e-05 -▁bef 3.2815318290592413e-05 -▁gesehen 3.2815318290592413e-05 -▁worked 3.2811529867792758e-05 -zeiten 3.28077414449931e-05 -▁sw 3.2788799330994839e-05 -▁photo 3.2781222485395535e-05 -▁Arbeitsplätze 3.277364563979623e-05 -▁menu 3.2766068794196926e-05 -▁bitten 3.2750915102998311e-05 -▁begrüßen 3.27357614117997e-05 -▁beginnen 3.2724396143400744e-05 -anc 3.2720607720601088e-05 -fahren 3.2720607720601088e-05 -▁initiatives 3.2720607720601088e-05 -▁Verkehrs 3.2709242452202136e-05 -▁automatisch 3.2701665606602825e-05 -▁Its 3.2694088761003521e-05 -ton 3.2690300338203872e-05 -▁evening 3.2682723492604561e-05 -unter 3.26637813786063e-05 -▁Afghanistan 3.2648627687407689e-05 -imm 3.2644839264608034e-05 -gehen 3.2641050841808385e-05 -▁processing 3.2641050841808385e-05 -▁expressed 3.262589715060977e-05 -▁setzt 3.262589715060977e-05 -▁Obwohl 3.2606955036611507e-05 -▁zudem 3.2606955036611507e-05 -▁client 3.2584224499813588e-05 -än 3.2580436077013939e-05 -▁entscheiden 3.2580436077013939e-05 -▁Sea 3.2576647654214283e-05 -▁intended 3.2569070808614979e-05 -▁Volk 3.2565282385815324e-05 -▁fresh 3.2553917117416371e-05 -▁fund 3.2553917117416371e-05 -iß 3.2550128694616716e-05 -▁His 3.2550128694616716e-05 -▁beide 3.2531186580618452e-05 -▁NATO 3.2508456043820533e-05 -▁banks 3.2504667621020884e-05 -▁mainly 3.2493302352621925e-05 -▁relationship 3.2493302352621925e-05 -▁flexible 3.2481937084222965e-05 -ware 3.247814866142331e-05 -▁Milliarden 3.247814866142331e-05 -▁bad 3.247814866142331e-05 -▁heard 3.247814866142331e-05 -▁limit 3.247814866142331e-05 -▁darstellt 3.24554181246254e-05 -▁th 3.2432687587827479e-05 -ises 3.2425110742228175e-05 -▁accession 3.241753389662887e-05 -▁endlich 3.2406168628229911e-05 -▁plans 3.2406168628229911e-05 -ence 3.2398591782630607e-05 -▁deine 3.2391014937031296e-05 -▁ability 3.2383438091431992e-05 -▁traffic 3.2383438091431992e-05 -▁talking 3.2379649668632343e-05 -▁committed 3.2375861245832688e-05 -▁ban 3.2372072823033032e-05 -▁cities 3.2368284400233384e-05 -▁Iraq 3.2364495977433728e-05 -cr 3.2341765440635809e-05 -▁month 3.2337977017836161e-05 -▁Armut 3.2330400172236857e-05 -▁vorgeschlagen 3.23266117494372e-05 -▁Wann 3.2322823326637546e-05 -▁letzte 3.2315246481038242e-05 -reich 3.2300092789839633e-05 -▁verl 3.2288727521440674e-05 -▁proper 3.228115067584137e-05 -schaften 3.2265996984642755e-05 -▁goal 3.2265996984642755e-05 -igungs 3.2262208561843106e-05 -▁Gast 3.2220535911046924e-05 -▁eindeutig 3.21902285286497e-05 -▁IS 3.2182651683050396e-05 -aden 3.2167497991851781e-05 -▁running 3.2159921146252477e-05 -▁Hause 3.2152344300653173e-05 -ification 3.2129613763855254e-05 -AP 3.2125825341055605e-05 -▁character 3.2125825341055605e-05 -▁People 3.21182484954563e-05 -▁placed 3.2114460072656646e-05 -▁Erstens 3.2110671649856991e-05 -▁element 3.2106883227057342e-05 -ystem 3.2095517958658382e-05 -▁love 3.2091729535858727e-05 -▁terrace 3.2087941113059078e-05 -▁alone 3.2068998999060815e-05 -▁brand 3.2068998999060815e-05 -▁appear 3.2046268462262896e-05 -▁Einfluss 3.2019749502665328e-05 -▁suggest 3.2015961079865672e-05 -fragen 3.2012172657066017e-05 -▁dringend 3.1985653697468449e-05 -▁versch 3.1985653697468449e-05 -▁Sil 3.1970500006269841e-05 -▁entgegen 3.1947769469471922e-05 -▁discuss 3.1943981046672273e-05 -▁Leute 3.1936404201072963e-05 -▁Pass 3.1921250509874354e-05 -.0 3.191367366427505e-05 -▁Christian 3.1909885241475395e-05 -▁Verfassung 3.1887154704677476e-05 -▁Roma 3.1883366281877827e-05 -▁families 3.1883366281877827e-05 -▁trust 3.1872001013478868e-05 -pro 3.1860635745079908e-05 -achen 3.1856847322280259e-05 -▁improved 3.1856847322280259e-05 -▁planning 3.1837905208281989e-05 -▁folgen 3.1818963094283725e-05 -ule 3.1792444134686158e-05 -▁kaum 3.1735617792691367e-05 -▁Pal 3.1724252524292407e-05 -▁benutzt 3.1720464101492759e-05 -▁opening 3.1720464101492759e-05 -▁Falle 3.1705310410294144e-05 -iten 3.1701521987494495e-05 -▁foot 3.168636829629588e-05 -▁students 3.1682579873496232e-05 -ones 3.1678791450696576e-05 -▁black 3.1675003027896921e-05 -hal 3.1659849336698313e-05 -ira 3.1656060913898657e-05 -▁comments 3.1656060913898657e-05 -▁außer 3.1652272491099008e-05 -▁Anstrengungen 3.1648484068299353e-05 -▁Ja 3.1648484068299353e-05 -▁Ausschuß 3.1637118799900393e-05 -▁buy 3.1633330377100745e-05 -▁cohesion 3.1633330377100745e-05 -▁God 3.1629541954301089e-05 -▁approved 3.1614388263102481e-05 -ona 3.160681141750317e-05 -▁author 3.1603022994703522e-05 -▁Val 3.1599234571903866e-05 -▁structural 3.1591657726304562e-05 -▁genannt 3.1584080880705258e-05 -▁größere 3.15802924579056e-05 -▁reforms 3.154998507550838e-05 -atic 3.1542408229909075e-05 -go 3.1542408229909075e-05 -▁Waren 3.1534831384309771e-05 -ds 3.1527254538710467e-05 -▁currency 3.1527254538710467e-05 -▁automatically 3.1515889270311508e-05 -▁affected 3.1504524001912548e-05 -▁suitable 3.1493158733513589e-05 -gebiet 3.147800504231498e-05 -▁Costa 3.147800504231498e-05 -▁administrative 3.147800504231498e-05 -▁Tätigkeit 3.1474216619515325e-05 -▁ausge 3.1474216619515325e-05 -▁increasingly 3.147042819671567e-05 -▁med 3.147042819671567e-05 -ake 3.1466639773916021e-05 -▁justice 3.1466639773916021e-05 -▁feature 3.1462851351116366e-05 -▁Verpflichtungen 3.1459062928316717e-05 -▁Sitzung 3.1451486082717406e-05 -▁genutzt 3.1451486082717406e-05 -▁decide 3.1440120814318453e-05 -▁zusätzliche 3.1421178700320183e-05 -▁Zustimmung 3.1406025009121575e-05 -▁unbedingt 3.1406025009121575e-05 -ama 3.1402236586321919e-05 -▁Netz 3.1394659740722615e-05 -▁mod 3.1387082895123311e-05 -angen 3.1383294472323656e-05 -itäts 3.1383294472323656e-05 -▁bo 3.1383294472323656e-05 -▁Kauf 3.1375717626724352e-05 -▁oben 3.1368140781125047e-05 -itäten 3.1352987089926439e-05 -▁moderne 3.1349198667126784e-05 -▁export 3.1337833398727824e-05 -zentrum 3.1326468130328865e-05 -▁Qualitäts 3.1326468130328865e-05 -▁bilden 3.1303737593530953e-05 -▁conclusion 3.1303737593530953e-05 -▁emissions 3.1292372325131993e-05 -▁Organisationen 3.1277218633933378e-05 -▁transparent 3.1273430211133729e-05 -bild 3.1269641788334074e-05 -▁entspricht 3.1254488097135466e-05 -▁regime 3.1246911251536155e-05 -▁eingehen 3.1239334405936851e-05 -▁circumstances 3.1224180714738243e-05 -▁usually 3.1216603869138939e-05 -▁reduced 3.1212815446339283e-05 -▁collection 3.1209027023539628e-05 -▁host 3.1186296486741715e-05 -▁Bed 3.11711427955431e-05 -с 3.1159777527144141e-05 -▁delivery 3.1159777527144141e-05 -▁trans 3.1159777527144141e-05 -▁phone 3.1137046990346229e-05 -▁überzeugt 3.1137046990346229e-05 -bed 3.111431645354831e-05 -▁kon 3.10991627623497e-05 -for 3.1076432225551782e-05 -ley 3.1072643802752127e-05 -old 3.1061278534353174e-05 -punkt 3.1061278534353174e-05 -▁Zug 3.1053701688753863e-05 -▁Idee 3.1027182729156296e-05 -▁Verwaltung 3.1027182729156296e-05 -▁began 3.1023394306356647e-05 -▁supports 3.1019605883556992e-05 -▁confidence 3.1015817460757336e-05 -▁vorgelegt 3.1012029037957688e-05 -IA 3.1004452192358377e-05 -▁Tor 3.0996875346759073e-05 -▁Nations 3.0989298501159768e-05 -▁fish 3.0985510078360113e-05 -▁gesetzt 3.0985510078360113e-05 -▁laws 3.0981721655560464e-05 -▁Kompromiss 3.0966567964361849e-05 -▁Tier 3.0966567964361849e-05 -▁administration 3.0966567964361849e-05 -▁extensive 3.0943837427563937e-05 -ree 3.0932472159164978e-05 -▁erkennen 3.0924895313565674e-05 -▁Öl 3.0924895313565674e-05 --h 3.0921106890766018e-05 -▁selbstverständlich 3.0917318467966363e-05 -▁whom 3.0917318467966363e-05 -▁priorities 3.0902164776767755e-05 -▁Holz 3.089458793116845e-05 -▁contribute 3.0890799508368795e-05 -ache 3.0875645817170187e-05 -▁Anerkennung 3.0868068971570883e-05 -hol 3.0864280548771227e-05 -omp 3.0856703703171923e-05 -ides 3.0849126857572619e-05 -▁Social 3.0849126857572619e-05 -▁plus 3.0849126857572619e-05 -▁Abschluss 3.0841550011973308e-05 -heiten 3.083776158917366e-05 -▁aktiv 3.0818819475175396e-05 -▁powerful 3.0811242629576085e-05 -▁unemployment 3.0807454206776436e-05 -isk 3.0803665783976781e-05 -▁mag 3.0792300515577822e-05 -▁Wahlen 3.0784723669978517e-05 -system 3.0780935247178869e-05 -ekt 3.0765781555980254e-05 -▁Boden 3.074683944198199e-05 -uß 3.0739262596382686e-05 -▁says 3.0739262596382686e-05 -▁Cor 3.0724108905184071e-05 -▁mindestens 3.0724108905184071e-05 --O 3.0705166791185808e-05 -▁cross 3.0701378368386159e-05 -▁infrastructure 3.067864783158824e-05 -▁north 3.0640763603591713e-05 -air 3.0629398335192753e-05 -▁wonderful 3.0625609912393104e-05 -schl 3.0614244643994145e-05 -▁rise 3.059909095279553e-05 -▁Spa 3.0591514107196226e-05 -▁remember 3.0591514107196226e-05 -▁unterschiedlichen 3.0591514107196226e-05 -▁white 3.058772568439657e-05 -ays 3.0583937261596922e-05 -prozess 3.0580148838797266e-05 -▁ensuring 3.0557418301999354e-05 -▁wesentlich 3.0557418301999354e-05 -▁Don 3.05536298791997e-05 -▁Vers 3.0549841456400043e-05 -▁else 3.0549841456400043e-05 -▁Indeed 3.053847618800109e-05 -▁berücksichtigen 3.053847618800109e-05 -▁rein 3.0534687765201435e-05 -▁comment 3.053089934240178e-05 -▁electronic 3.0515745651203171e-05 -▁Com 3.0511957228403519e-05 -▁Agenda 3.0508168805603864e-05 -▁Mann 3.0504380382804212e-05 -▁assist 3.0496803537204908e-05 -▁encourage 3.04892266916056e-05 -une 3.0485438268805948e-05 -▁scope 3.0485438268805948e-05 -▁verfügbar 3.0481649846006296e-05 -▁inv 3.0477861423206644e-05 -kraft 3.0455130886408725e-05 -fern 3.0451342463609073e-05 -▁Czech 3.0447554040809421e-05 -▁Körper 3.043240034961081e-05 -▁Kriterien 3.0413458235612546e-05 -▁getroffen 3.0402092967213586e-05 -▁seem 3.039072769881463e-05 -▁victims 3.0386939276014975e-05 -daten 3.0383150853215323e-05 -▁Vereinbarung 3.0383150853215323e-05 -▁Most 3.0367997162016711e-05 -▁gezeigt 3.0367997162016711e-05 -izing 3.0337689779619491e-05 -▁Osten 3.0337689779619491e-05 -▁kl 3.0333901356819836e-05 -▁danke 3.0326324511220532e-05 -▁Gar 3.032253608842088e-05 -▁broad 3.0299805551622961e-05 -▁Each 3.0288440283224005e-05 -▁mehreren 3.0269498169225741e-05 -icher 3.0258132900826782e-05 -▁bewusst 3.0258132900826782e-05 -▁ged 3.025434447802713e-05 -▁participation 3.0224037095629906e-05 -▁played 3.0212671827230947e-05 -▁fellow 3.0205094981631643e-05 -▁crime 3.0197518136032339e-05 -▁specifically 3.0193729713232683e-05 -ym 3.0178576022034072e-05 -▁manner 3.0178576022034072e-05 -▁Frankfurt 3.0159633908035808e-05 -▁agriculture 3.0152057062436504e-05 -▁larger 3.0148268639636852e-05 -▁1995 3.0140691794037545e-05 -Q 3.0121749680039281e-05 -▁Hintergrund 3.0095230720441713e-05 -▁tabled 3.0091442297642058e-05 -ordnung 3.0083865452042754e-05 -▁introduced 3.0083865452042754e-05 -▁Durchführung 3.0072500183643794e-05 -▁Lebensmittel 3.0072500183643794e-05 -▁entsprechende 3.0072500183643794e-05 -▁Anspruch 3.0068711760844142e-05 -▁getting 3.0061134915244838e-05 -▁Inn 3.0053558069645531e-05 -à 3.0049769646845879e-05 -▁aktuelle 3.0042192801246574e-05 -▁ass 3.0042192801246574e-05 -ined 3.0038404378446919e-05 -▁sources 3.0038404378446919e-05 -▁extrem 3.0034615955647267e-05 -▁reg 3.0030827532847615e-05 -vel 3.0027039110047963e-05 -erei 3.0011885418849351e-05 -▁demokratischen 3.0011885418849351e-05 -▁detailed 3.0004308573250044e-05 -▁Vorteile 3.0000520150450392e-05 -▁nachdem 3.0000520150450392e-05 -▁Besucher 2.999673172765074e-05 -▁Download 2.9989154882051436e-05 -▁effectively 2.9977789613652476e-05 -▁hinweisen 2.9966424345253517e-05 -▁Stabilität 2.995505907685456e-05 -000 2.9951270654054905e-05 -ga 2.9947482231255253e-05 -▁animals 2.9947482231255253e-05 -▁bi 2.9947482231255253e-05 -▁import 2.99436938084556e-05 -▁competitive 2.9928540117256989e-05 -▁functions 2.9928540117256989e-05 -▁nationale 2.9924751694457337e-05 -▁mar 2.9920963271657685e-05 -own 2.991717484885803e-05 -iste 2.9913386426058378e-05 -▁served 2.9894444312060114e-05 -ips 2.9875502198061851e-05 -▁record 2.9867925352462546e-05 -▁mm 2.9848983238464279e-05 -▁Mitteilung 2.9845194815664627e-05 -▁glauben 2.9845194815664627e-05 -▁India 2.9833829547265671e-05 -▁pleasant 2.9830041124466016e-05 -▁geeignet 2.9826252701666364e-05 -▁installation 2.9822464278866712e-05 -▁globalen 2.9811099010467752e-05 -q 2.98073105876681e-05 -▁purposes 2.9799733742068796e-05 -▁ship 2.9799733742068796e-05 -▁referred 2.9769426359671573e-05 -▁Sprachen 2.9758061091272613e-05 -▁Part 2.9754272668472961e-05 -cker 2.9750484245673309e-05 -▁rural 2.9750484245673309e-05 -▁aimed 2.973911897727435e-05 -▁erwähnt 2.9735330554474698e-05 -▁gesprochen 2.9735330554474698e-05 -▁rechts 2.9712600017676782e-05 -▁Inhalte 2.9701234749277823e-05 -▁400 2.9693657903678519e-05 -▁Asia 2.9693657903678519e-05 -▁critical 2.9693657903678519e-05 -▁sitting 2.9678504212479907e-05 -▁exactly 2.9663350521281295e-05 -zt 2.9655773675681988e-05 -uck 2.9640619984483376e-05 -▁reception 2.9640619984483376e-05 -ife 2.9636831561683724e-05 -▁serve 2.962925471608442e-05 -▁partnership 2.9625466293284768e-05 -▁eingerichtet 2.9621677870485113e-05 -▁protected 2.9621677870485113e-05 -▁motion 2.9595158910887545e-05 -▁regular 2.9595158910887545e-05 -▁Eur 2.9587582065288238e-05 -▁kitchen 2.9583793642488586e-05 -▁Homepage 2.9580005219688934e-05 -ologie 2.9576216796889281e-05 -▁http 2.9568639951289974e-05 -▁benutzen 2.9557274682891018e-05 -▁valid 2.9557274682891018e-05 -IM 2.9553486260091362e-05 -▁Abend 2.9545909414492058e-05 -▁besonderen 2.9545909414492058e-05 -▁euch 2.95345441460931e-05 -cc 2.9530755723293447e-05 -ances 2.9519390454894487e-05 -▁wahrscheinlich 2.9515602032094835e-05 -▁Österreich 2.9511813609295183e-05 -▁Telefon 2.9504236763695879e-05 -▁Ger 2.9500448340896224e-05 -▁Zeichen 2.9496659918096572e-05 -▁camera 2.9496659918096572e-05 -▁Ag 2.949287149529692e-05 -▁Komfort 2.9485294649697615e-05 -ique 2.9473929381298656e-05 -▁Reformen 2.9466352535699348e-05 -board 2.9458775690100044e-05 -▁risks 2.9447410421701085e-05 -▁Dienst 2.9443621998901433e-05 -▁Ausgaben 2.9439833576101781e-05 -▁becoming 2.9432256730502473e-05 -▁Tradition 2.9428468307702821e-05 -▁akzeptieren 2.9424679884903169e-05 -▁Ost 2.9409526193704558e-05 -▁Ind 2.9398160925305598e-05 -▁2020 2.9394372502505946e-05 -▁institutional 2.9386795656906642e-05 -▁sicherzustellen 2.938300723410699e-05 -▁Irak 2.9379218811307334e-05 -rechte 2.9375430388507682e-05 -▁Why 2.9375430388507682e-05 -▁display 2.9375430388507682e-05 -ichten 2.9367853542908378e-05 -▁Power 2.9367853542908378e-05 -uz 2.9360276697309071e-05 -▁Karte 2.9360276697309071e-05 -▁Informations 2.9341334583310807e-05 -ike 2.9333757737711503e-05 -▁discussed 2.9322392469312544e-05 -▁wants 2.9322392469312544e-05 -▁Partei 2.931481562371324e-05 -▁represents 2.9311027200913584e-05 -bes 2.930345035531428e-05 -▁spacious 2.9295873509714976e-05 -▁precisely 2.9276931395716712e-05 -ished 2.9273142972917057e-05 -▁February 2.9273142972917057e-05 -▁police 2.9273142972917057e-05 -▁Waffen 2.92617777045181e-05 --Ver 2.9257989281718445e-05 -RA 2.9257989281718445e-05 -▁Schließlich 2.9250412436119141e-05 -▁siehe 2.9246624013319489e-05 -olle 2.9231470322120878e-05 -▁Kn 2.9231470322120878e-05 -▁developments 2.9231470322120878e-05 -▁Sitz 2.9208739785322962e-05 -▁Then 2.9208739785322962e-05 -▁erzielt 2.9208739785322962e-05 -utz 2.919358609412435e-05 -▁Middle 2.919358609412435e-05 -▁stated 2.919358609412435e-05 -▁Zunächst 2.9189797671324695e-05 -▁discussions 2.9186009248525043e-05 -▁Code 2.9182220825725391e-05 -▁unver 2.9182220825725391e-05 -▁Agreement 2.9167067134526779e-05 -▁Datei 2.9159490288927475e-05 -▁Zudem 2.9155701866127823e-05 -▁Haushalt 2.9151913443328168e-05 -▁bestehenden 2.9148125020528516e-05 -▁likely 2.9148125020528516e-05 -▁Board 2.9144336597728864e-05 -IP 2.9140548174929212e-05 -▁spend 2.9110240792531989e-05 -stadt 2.9106452369732336e-05 -verb 2.9102663946932681e-05 -▁Cal 2.9087510255734073e-05 -▁freuen 2.9087510255734073e-05 -▁managed 2.9087510255734073e-05 -TA 2.9076144987335113e-05 -.5 2.9049626027737542e-05 -▁Friedens 2.9049626027737542e-05 -▁Protokoll 2.904583760493789e-05 -▁Rec 2.904583760493789e-05 -▁programs 2.904583760493789e-05 -▁möglicherweise 2.9030683913739279e-05 -ork 2.9023107068139975e-05 -reg 2.9023107068139975e-05 -▁otherwise 2.9015530222540667e-05 -▁Sektor 2.9007953376941363e-05 -gruppe 2.9004164954141711e-05 -▁Verwaltungs 2.9000376531342059e-05 -▁3. 2.8996588108542403e-05 -▁budgetary 2.8996588108542403e-05 -▁Stil 2.8981434417343792e-05 -ING 2.8970069148944836e-05 -▁mark 2.8966280726145184e-05 -▁thousands 2.8966280726145184e-05 -stoff 2.8962492303345528e-05 -▁metres 2.8958703880545876e-05 -▁Schlüssel 2.8951127034946572e-05 -ring 2.8928396498148653e-05 -▁Wissen 2.8924608075349e-05 -▁inside 2.8920819652549349e-05 -eau 2.8909454384150389e-05 -▁won 2.8898089115751433e-05 -▁selected 2.8890512270152126e-05 -ours 2.8882935424552822e-05 -rit 2.8882935424552822e-05 -▁loss 2.887914700175317e-05 -val 2.8875358578953514e-05 -▁eng 2.886778173335421e-05 -▁Switzerland 2.88526280421556e-05 -▁geändert 2.88526280421556e-05 -▁Chance 2.8848839619355947e-05 -▁Nutzen 2.8841262773756639e-05 -hand 2.8833685928157335e-05 -▁shower 2.8833685928157335e-05 -este 2.8829897505357683e-05 -▁Policy 2.8814743814159071e-05 -▁festgelegt 2.8814743814159071e-05 -▁hands 2.8803378545760112e-05 -▁Mark 2.8795801700160808e-05 -▁Wettbewerbsfähigkeit 2.8795801700160808e-05 -▁bestimmt 2.8792013277361156e-05 -▁expected 2.8792013277361156e-05 -ron 2.87882248545615e-05 -▁covered 2.87882248545615e-05 -▁measure 2.87882248545615e-05 -▁tradition 2.8780648008962196e-05 -▁Unterkategorien 2.8776859586162544e-05 -▁winter 2.8776859586162544e-05 -▁season 2.8773071163362892e-05 -▁Research 2.8769282740563237e-05 -yst 2.8765494317763585e-05 -genden 2.8750340626564973e-05 -▁Rest 2.8750340626564973e-05 -arbeiten 2.8738975358166017e-05 -zimmer 2.8735186935366362e-05 -▁hoffen 2.873139851256671e-05 -▁entry 2.8727610089767057e-05 -▁Haltung 2.8716244821368098e-05 -▁color 2.8712456398568446e-05 -▁prüfen 2.8712456398568446e-05 -▁actual 2.8708667975768794e-05 -▁anders 2.8708667975768794e-05 -▁umgesetzt 2.8708667975768794e-05 -▁zuletzt 2.8704879552969142e-05 -werte 2.8701091130169486e-05 -▁medical 2.8701091130169486e-05 -▁sub-categories 2.8701091130169486e-05 -▁Fort 2.8697302707369834e-05 -▁assessment 2.8682149016171223e-05 -▁attack 2.8682149016171223e-05 -▁Straßen 2.8674572170571919e-05 -▁harmon 2.8674572170571919e-05 -60 2.8670783747772267e-05 -aum 2.8670783747772267e-05 -▁Antrag 2.8655630056573655e-05 -▁notwendigen 2.8636687942575391e-05 -aren 2.8625322674176432e-05 -▁closely 2.8617745828577128e-05 -▁Rooms 2.8606380560178168e-05 -▁Straße 2.8602592137378516e-05 -▁debt 2.8602592137378516e-05 -▁gerecht 2.8602592137378516e-05 -▁erfüllt 2.8595015291779209e-05 -igkeiten 2.8591226868979557e-05 -▁schw 2.8587438446179905e-05 -▁listed 2.8583650023380253e-05 -Ã 2.8579861600580597e-05 -▁earlier 2.8579861600580597e-05 -▁anderes 2.8572284754981293e-05 -▁freien 2.855334264098303e-05 -▁followed 2.8549554218183377e-05 -▁happen 2.8549554218183377e-05 -▁paper 2.8549554218183377e-05 -▁vorher 2.8545765795383722e-05 -▁dienen 2.8534400526984766e-05 -▁votes 2.8519246835786154e-05 -ipp 2.8511669990186847e-05 -ump 2.8507881567387195e-05 -▁guidelines 2.8507881567387195e-05 -▁Funktionen 2.8500304721787891e-05 -▁introduction 2.8492727876188583e-05 -▁finanziellen 2.8481362607789627e-05 -eut 2.847378576219032e-05 -ief 2.8466208916591016e-05 -▁worth 2.8454843648192056e-05 -▁zufrieden 2.84434783797931e-05 -▁Click 2.8439689956993444e-05 -▁jederzeit 2.843211311139414e-05 -▁Kenntnis 2.8424536265794833e-05 -▁Col 2.8401805728996917e-05 -▁ergreifen 2.8386652037798306e-05 -▁01 2.8382863614998654e-05 -▁Monate 2.8371498346599694e-05 -▁Partnerschaft 2.8348767809801778e-05 -▁Erst 2.8344979387002126e-05 -▁Technologien 2.8344979387002126e-05 -▁jene 2.8341190964202474e-05 -ting 2.8333614118603167e-05 -▁Privat 2.8329825695803515e-05 -▁Kommunikation 2.8326037273003863e-05 -▁Schweiz 2.8318460427404555e-05 -▁beziehen 2.8318460427404555e-05 -▁Dateien 2.8314672004604903e-05 -▁enterprises 2.8314672004604903e-05 -ains 2.83070951590056e-05 -▁Suche 2.8299518313406292e-05 -ices 2.829572989060664e-05 -▁Gew 2.829572989060664e-05 -hren 2.825405723981046e-05 -▁Ausbildung 2.8250268817010805e-05 -▁fahren 2.8250268817010805e-05 -buch 2.82426919714115e-05 -▁lösen 2.82426919714115e-05 -▁strengthen 2.8238903548611849e-05 -▁kostenlos 2.8235115125812197e-05 -▁wünschen 2.8231326703012541e-05 -▁1, 2.8219961434613585e-05 -▁braucht 2.8208596166214626e-05 -▁Gesetz 2.8201019320615322e-05 -▁opened 2.8201019320615322e-05 -▁save 2.8201019320615322e-05 -enk 2.818586562941671e-05 -urt 2.818586562941671e-05 -▁Produkten 2.8182077206617055e-05 -▁buildings 2.8178288783817403e-05 -▁vermeiden 2.8174500361017751e-05 -arr 2.8159346669819139e-05 -▁ISO 2.8151769824219835e-05 -▁verbessert 2.8151769824219835e-05 -▁discover 2.8136616133021223e-05 -gruppen 2.8132827710221571e-05 -▁Office 2.8129039287421916e-05 -▁gleiche 2.8106308750624e-05 -▁Act 2.8102520327824348e-05 -▁Aufnahme 2.8098731905024696e-05 -▁Außen 2.8094943482225041e-05 -▁improving 2.8087366636625737e-05 -iken 2.8083578213826085e-05 -▁Klima 2.8083578213826085e-05 -hl 2.8076001368226777e-05 -▁Konzept 2.8076001368226777e-05 -rag 2.8072212945427125e-05 -ser 2.8057059254228513e-05 -▁firm 2.8057059254228513e-05 -▁op 2.8049482408629209e-05 -▁schwierig 2.8045693985829557e-05 -▁-- 2.8034328717430598e-05 -▁spirit 2.8034328717430598e-05 -▁Apartments 2.8030540294630946e-05 -▁Vielfalt 2.8019175026231986e-05 -▁Kindern 2.8004021335033375e-05 -▁networks 2.8000232912233723e-05 -▁ed 2.7992656066634419e-05 -▁speech 2.7985079221035111e-05 -ager 2.7981290798235459e-05 -ests 2.7981290798235459e-05 -▁trip 2.7981290798235459e-05 -ony 2.7977502375435807e-05 -▁Zweck 2.7977502375435807e-05 -▁super 2.79699255298365e-05 -▁Arten 2.7958560261437543e-05 -▁linked 2.793961814743928e-05 -▁track 2.7932041301839972e-05 -▁beds 2.7924464456240668e-05 -gang 2.7920676033441013e-05 -▁darunter 2.7920676033441013e-05 -▁sa 2.7920676033441013e-05 -▁Einrichtungen 2.7916887610641361e-05 -▁Meter 2.7913099187841709e-05 -▁exclusive 2.7913099187841709e-05 -▁map 2.7909310765042057e-05 -▁sufficient 2.7909310765042057e-05 -ication 2.7905522342242405e-05 -▁lesen 2.7905522342242405e-05 -▁operations 2.7901733919442749e-05 -▁supporting 2.7901733919442749e-05 -.de 2.7897945496643097e-05 -ki 2.7897945496643097e-05 -▁telephone 2.7886580228244138e-05 -▁uses 2.7882791805444486e-05 -▁understanding 2.7879003382644833e-05 -atur 2.7875214959845181e-05 -▁Kapital 2.7875214959845181e-05 -▁Greece 2.786006126864657e-05 -▁Wettbewerbs 2.784869600024761e-05 -▁considerable 2.7833542309049e-05 -▁immigration 2.7833542309049e-05 -▁Bush 2.7822177040650043e-05 -▁Innovation 2.7822177040650043e-05 -▁Off 2.7818388617850391e-05 -▁degree 2.7807023349451431e-05 -▁so-called 2.7803234926651779e-05 -aff 2.779186965825282e-05 -▁marketing 2.779186965825282e-05 -▁Standards 2.7769139121454904e-05 -orm 2.7765350698655248e-05 -yp 2.7761562275855596e-05 -anlagen 2.775019700745664e-05 -▁Top 2.775019700745664e-05 -▁fallen 2.775019700745664e-05 -ita 2.7735043316258029e-05 -welt 2.7727466470658721e-05 -▁jeweils 2.7723678047859069e-05 -▁Economic 2.7716101202259765e-05 -▁touch 2.7716101202259765e-05 -▁sal 2.7708524356660458e-05 -▁sicherlich 2.7708524356660458e-05 -▁Moment 2.7704735933860806e-05 -▁Betrieb 2.7700947511061154e-05 -▁Mus 2.76971590882615e-05 -▁criminal 2.7693370665461846e-05 -▁länger 2.7689582242662194e-05 -▁Sinn 2.767064012866393e-05 -▁entschieden 2.767064012866393e-05 -ahn 2.7663063283064626e-05 -IG 2.7621390632268444e-05 -▁mis 2.7621390632268444e-05 -▁verw 2.7621390632268444e-05 -▁ständig 2.7606236941069832e-05 -▁provision 2.760244851827018e-05 -▁went 2.7598660095470528e-05 -eat 2.7579717981472264e-05 -ights 2.7560775867474e-05 -č 2.7560775867474e-05 -asch 2.7553199021874693e-05 -т 2.7549410599075041e-05 -▁angezeigt 2.7545622176275389e-05 -▁entweder 2.7530468485076778e-05 -kte 2.7522891639477474e-05 -▁components 2.7522891639477474e-05 -▁cuisine 2.7507737948278862e-05 -▁gern 2.7503949525479207e-05 -▁Go 2.7496372679879902e-05 -▁Kur 2.74887958342806e-05 -▁alte 2.74887958342806e-05 -▁founded 2.7485007411480943e-05 -▁Tages 2.7469853720282331e-05 -▁detail 2.7469853720282331e-05 -▁relax 2.7462276874683027e-05 -▁e.g. 2.7458488451883375e-05 -▁Wege 2.7454700029083723e-05 -▁komplett 2.743575791508546e-05 -▁Stellen 2.7431969492285804e-05 -▁door 2.7428181069486152e-05 -▁gelangen 2.7420604223886848e-05 -ison 2.7416815801087193e-05 -▁properly 2.7413027378287541e-05 -▁speziell 2.7413027378287541e-05 -▁'ll 2.7409238955487888e-05 -▁Menge 2.7397873687088929e-05 -▁Every 2.7394085264289277e-05 -▁playing 2.7386508418689973e-05 -▁presence 2.7378931573090665e-05 -▁übernachten 2.7378931573090665e-05 -▁purchase 2.7371354727491361e-05 -▁erzielen 2.7363777881892054e-05 -▁Mass 2.7359989459092402e-05 -▁che 2.7359989459092402e-05 -▁tourism 2.7359989459092402e-05 -▁Mir 2.735620103629275e-05 -▁stock 2.7352412613493098e-05 -▁experts 2.7348624190693442e-05 -▁gave 2.7341047345094138e-05 -▁animal 2.7337258922294486e-05 -tw 2.7329682076695179e-05 -▁36 2.7318316808296222e-05 -▁Grundsatz 2.7310739962696915e-05 -▁Stimme 2.7310739962696915e-05 -ös 2.7303163117097611e-05 -▁items 2.7295586271498303e-05 -▁Februar 2.7291797848698651e-05 -▁teilweise 2.7291797848698651e-05 -▁pres 2.7265278889101084e-05 -allen 2.7250125197902472e-05 -▁Belarus 2.724633677510282e-05 -▁ging 2.7242548352303165e-05 -mar 2.7238759929503513e-05 -▁zeit 2.7238759929503513e-05 -cks 2.7231183083904209e-05 -▁typical 2.7227394661104556e-05 -ith 2.72236062383049e-05 -satz 2.72236062383049e-05 -▁vorgesehen 2.72236062383049e-05 -▁Street 2.7219817815505249e-05 -▁Abschließend 2.7216029392705597e-05 -bew 2.7204664124306637e-05 -schw 2.7204664124306637e-05 -▁su 2.7200875701506985e-05 -▁wahr 2.7197087278707333e-05 -uit 2.7193298855907681e-05 -▁entstehen 2.7181933587508722e-05 -▁Mod 2.7159203050710806e-05 -line 2.7144049359512195e-05 -▁duty 2.7140260936712539e-05 -▁erhöhen 2.7140260936712539e-05 -▁Guest 2.7136472513912887e-05 -tens 2.7132684091113235e-05 -▁remind 2.7125107245513931e-05 -▁Brussels 2.7113741977114971e-05 -▁S. 2.7113741977114971e-05 -▁entirely 2.7113741977114971e-05 -▁fear 2.7109953554315319e-05 -gende 2.7102376708716012e-05 -ox 2.7102376708716012e-05 -▁dank 2.709858828591636e-05 -▁generally 2.7094799863116708e-05 -▁Forderung 2.7091011440317056e-05 -▁Station 2.7079646171918096e-05 -▁pol 2.7072069326318792e-05 -▁coffee 2.7053127212320525e-05 -▁resort 2.7053127212320525e-05 -▁Status 2.7045550366721221e-05 -70 2.7037973521121917e-05 -hm 2.7037973521121917e-05 -▁ausdrücklich 2.7026608252722957e-05 -▁coordination 2.7022819829923305e-05 -▁fisheries 2.7022819829923305e-05 -▁Arab 2.7007666138724694e-05 -▁besuchen 2.7007666138724694e-05 -▁competitiveness 2.7007666138724694e-05 -ction 2.6992512447526082e-05 -▁na 2.698872402472643e-05 -ull 2.6984935601926778e-05 -aw 2.6981147179127123e-05 -,00 2.6977358756327471e-05 -▁plants 2.6977358756327471e-05 -▁presidency 2.6977358756327471e-05 -▁Menschenrechts 2.6958416642329207e-05 -▁Health 2.6950839796729903e-05 -▁technische 2.6947051373930248e-05 -▁Vereinten 2.6943262951130596e-05 -▁Because 2.6939474528330943e-05 -▁une 2.6935686105531291e-05 -ev 2.6928109259931984e-05 -▁gekommen 2.6928109259931984e-05 -▁welcher 2.6928109259931984e-05 -▁bas 2.6924320837132332e-05 -▁Jetzt 2.6912955568733372e-05 -leg 2.6901590300334416e-05 -▁Arbeiten 2.6901590300334416e-05 -▁po 2.6901590300334416e-05 -▁vorhanden 2.6901590300334416e-05 -▁Wo 2.6897801877534764e-05 -▁PHP 2.6894013454735109e-05 -▁u 2.6894013454735109e-05 -▁peaceful 2.6890225031935457e-05 -д 2.6875071340736845e-05 -▁erstellt 2.6875071340736845e-05 -handel 2.6871282917937193e-05 -ester 2.6867494495137541e-05 -▁Poland 2.6867494495137541e-05 -frage 2.6859917649538234e-05 -ektor 2.6856129226738582e-05 -▁stimme 2.6856129226738582e-05 -äch 2.6844763958339622e-05 -ände 2.6837187112740318e-05 -▁Material 2.6837187112740318e-05 -▁Pakistan 2.6837187112740318e-05 -▁difficulties 2.6837187112740318e-05 -▁permanent 2.6837187112740318e-05 -oph 2.6833398689940666e-05 -▁authority 2.6833398689940666e-05 -▁sagt 2.6833398689940666e-05 -▁wären 2.6833398689940666e-05 -▁dealing 2.6829610267141014e-05 -▁bear 2.6818244998742054e-05 -▁kommenden 2.6818244998742054e-05 -▁Black 2.6803091307543443e-05 -▁install 2.6799302884743791e-05 -▁relate 2.6791726039144483e-05 -▁Mitte 2.6780360770745527e-05 -▁funktioniert 2.6780360770745527e-05 -▁schl 2.6780360770745527e-05 -▁recommend 2.6765207079546916e-05 -▁Name 2.6750053388348304e-05 -ahr 2.6746264965548652e-05 -best 2.6746264965548652e-05 -▁wollte 2.6742476542749e-05 -▁Gott 2.6738688119949344e-05 -▁Kurz 2.6734899697149692e-05 -ß 2.6727322851550388e-05 -▁shops 2.6723534428750736e-05 -▁Leistungen 2.6715957583151429e-05 -ften 2.6708380737552125e-05 -▁Entwurf 2.6700803891952817e-05 -▁betonen 2.6700803891952817e-05 -▁summit 2.6700803891952817e-05 -ange 2.6693227046353513e-05 -▁beschlossen 2.6693227046353513e-05 -ille 2.6681861777954554e-05 -▁organisation 2.666291966395629e-05 -▁ph 2.6628823858759415e-05 -▁erhält 2.6625035435959763e-05 -▁Banken 2.6621247013160111e-05 -▁facing 2.6613670167560803e-05 -stoffe 2.6609881744761151e-05 -▁Unfortunately 2.6609881744761151e-05 -▁format 2.6602304899161847e-05 -▁informieren 2.6602304899161847e-05 -▁Berichte 2.6598516476362192e-05 -omb 2.659472805356254e-05 -ush 2.659472805356254e-05 -orr 2.6590939630762888e-05 -▁son 2.6587151207963236e-05 -ativen 2.658336278516358e-05 -heim 2.6579574362363928e-05 -▁Leider 2.6575785939564276e-05 -▁combat 2.6571997516764624e-05 -▁hour 2.6556843825566012e-05 -▁Veränderungen 2.6549266979966705e-05 -▁Great 2.65416901343674e-05 -▁district 2.6537901711567749e-05 -▁handeln 2.6534113288768097e-05 -90 2.6522748020369137e-05 -zeichen 2.6515171174769833e-05 -▁2011 2.6511382751970178e-05 -uche 2.6507594329170526e-05 -▁Instrumente 2.6500017483571222e-05 -▁lernen 2.6488652215172262e-05 -etr 2.6477286946773303e-05 -▁vehicles 2.6469710101174e-05 -▁machines 2.6465921678374346e-05 -onen 2.6439402718776775e-05 -▁Teile 2.6439402718776775e-05 -▁average 2.6435614295977123e-05 -▁Warum 2.6431825873177471e-05 -▁Abgeordnete 2.64053069135799e-05 -▁campaign 2.64053069135799e-05 -▁speaking 2.64053069135799e-05 -▁Tel 2.6401518490780248e-05 -▁meetings 2.6390153222381289e-05 -legen 2.6374999531182677e-05 -▁las 2.6367422685583373e-05 -▁Webseite 2.6363634262783721e-05 -▁applies 2.6363634262783721e-05 -▁fragen 2.6363634262783721e-05 -▁fixed 2.6359845839984069e-05 -▁forget 2.6356057417184413e-05 -itor 2.6352268994384761e-05 -ode 2.6348480571585109e-05 -ining 2.633711530318615e-05 -ty 2.6314384766388234e-05 -▁Initiativen 2.6314384766388234e-05 -▁court 2.6314384766388234e-05 -▁thinking 2.6299231075189623e-05 -▁EADS 2.6276500538391707e-05 -▁Engagement 2.6276500538391707e-05 -▁Vielen 2.6272712115592055e-05 -▁darstellen 2.6272712115592055e-05 -▁dagegen 2.6249981578794136e-05 -▁sm 2.6246193155994484e-05 -iums 2.623861631039518e-05 -no 2.623861631039518e-05 -▁institution 2.6219674196396913e-05 -▁Maschinen 2.6212097350797609e-05 -▁Zeitraum 2.6208308927997957e-05 -iar 2.6204520505198305e-05 -▁Systeme 2.6200732082398649e-05 -▁Geschäft 2.6185578391200041e-05 -▁Miss 2.6170424700001429e-05 -▁consideration 2.6170424700001429e-05 -schlag 2.6166636277201774e-05 -▁Casino 2.6166636277201774e-05 -▁kn 2.6162847854402122e-05 -▁Herz 2.6155271008802818e-05 -▁Bern 2.6143905740403858e-05 -▁becomes 2.6143905740403858e-05 -▁income 2.6136328894804554e-05 -ior 2.6128752049205247e-05 -uring 2.6124963626405595e-05 -▁numbers 2.6124963626405595e-05 -▁stabil 2.6121175203605943e-05 -▁Installation 2.6113598358006635e-05 -▁Zeiten 2.6109809935206983e-05 -▁.... 2.6102233089607679e-05 -▁aspect 2.6090867821208719e-05 -▁durchaus 2.6083290975609415e-05 -▁wenigen 2.6083290975609415e-05 -▁Once 2.607950255280976e-05 -▁einz 2.6075714130010108e-05 -▁gefordert 2.6075714130010108e-05 -dem 2.6071925707210456e-05 -▁born 2.6064348861611152e-05 -▁object 2.6060560438811496e-05 -▁unternehmen 2.6045406747612885e-05 -▁Gipfel 2.6034041479213929e-05 -▁deep 2.6022676210814969e-05 -▁erleben 2.6022676210814969e-05 -▁Ra 2.6011310942416013e-05 -▁opposition 2.6007522519616358e-05 -ags 2.5984791982818442e-05 -▁Währung 2.5984791982818442e-05 -▁dev 2.5984791982818442e-05 -▁def 2.5973426714419482e-05 -▁picture 2.5973426714419482e-05 -▁veröffentlicht 2.5958273023220871e-05 -ced 2.5950696177621567e-05 -▁Madrid 2.5950696177621567e-05 -▁stärken 2.5950696177621567e-05 -▁Angaben 2.5939330909222607e-05 -▁Cas 2.5939330909222607e-05 -▁konkrete 2.5935542486422955e-05 -▁absolut 2.5920388795224344e-05 -▁flexibility 2.5920388795224344e-05 -▁Absatz 2.591281194962504e-05 -▁told 2.5897658258426428e-05 -lex 2.5893869835626776e-05 -RO 2.5890081412827124e-05 -▁Fonds 2.5878716144428164e-05 -▁gener 2.5874927721628512e-05 -▁Har 2.5871139298828857e-05 -▁mer 2.5871139298828857e-05 -▁Kritik 2.5863562453229553e-05 -▁Sub 2.5863562453229553e-05 -▁practices 2.5863562453229553e-05 -▁screen 2.5863562453229553e-05 -▁advanced 2.5852197184830593e-05 -IR 2.5848408762030941e-05 -▁erklären 2.5848408762030941e-05 -▁database 2.583325507083233e-05 -▁dec 2.5814312956834066e-05 -▁Breakfast 2.5780217151637191e-05 -▁Bra 2.5765063460438579e-05 -▁Mio. 2.5757486614839275e-05 -▁Mitgliedstaat 2.5757486614839275e-05 -▁Netzwerk 2.5753698192039623e-05 -▁gewährleistet 2.5753698192039623e-05 -bedingungen 2.5749909769239968e-05 -▁domestic 2.5749909769239968e-05 -▁aims 2.5746121346440316e-05 -stra 2.5738544500841012e-05 -▁Mult 2.5738544500841012e-05 -▁attempt 2.573475607804136e-05 -rechts 2.5727179232442052e-05 -▁signal 2.5727179232442052e-05 -▁Hauses 2.5719602386842748e-05 -▁PT 2.5719602386842748e-05 -▁According 2.5715813964043093e-05 -▁Glück 2.5715813964043093e-05 -▁ST 2.5715813964043093e-05 -tes 2.5700660272844484e-05 -▁benötigt 2.5700660272844484e-05 -▁freue 2.5693083427245177e-05 -▁condition 2.5685506581645873e-05 -▁rev 2.5685506581645873e-05 -▁middle 2.5681718158846221e-05 -▁klare 2.5677929736046565e-05 -igem 2.5670352890447261e-05 -▁trägt 2.5666564467647609e-05 -spr 2.5662776044847954e-05 -▁Anteil 2.5662776044847954e-05 -▁Global 2.5658987622048302e-05 -▁PDF 2.5658987622048302e-05 -iting 2.565519919924865e-05 -▁bodies 2.565519919924865e-05 -▁defined 2.5647622353649346e-05 -▁myself 2.5647622353649346e-05 -▁easier 2.5640045508050038e-05 -▁Private 2.5632468662450734e-05 -ister 2.5621103394051774e-05 -▁Klicken 2.5617314971252122e-05 -▁liebe 2.5602161280053511e-05 -▁hält 2.5598372857253859e-05 -.4 2.5594584434454207e-05 -▁politics 2.5594584434454207e-05 -▁Herausforderung 2.55870075888549e-05 -▁lokalen 2.55870075888549e-05 -▁propose 2.5571853897656288e-05 -ily 2.5549123360858372e-05 -▁mid 2.5549123360858372e-05 -▁Worte 2.5530181246860108e-05 -▁Santa 2.5526392824060456e-05 -▁himself 2.55226044012608e-05 -▁signed 2.5518815978461149e-05 -▁respond 2.5507450710062189e-05 -ichen 2.5503662287262537e-05 -aler 2.5492297018863581e-05 -see 2.5488508596063926e-05 --st 2.5484720173264274e-05 -▁Sim 2.5469566482065662e-05 -▁klicken 2.5469566482065662e-05 -▁Prioritäten 2.546577805926601e-05 -▁strength 2.546577805926601e-05 -▁Brüssel 2.5461989636466358e-05 -▁ancient 2.5454412790867051e-05 -▁angesprochen 2.5454412790867051e-05 -▁Gebieten 2.54506243680674e-05 -▁Hafen 2.54506243680674e-05 -▁erfolgen 2.5446835945267747e-05 -isierten 2.5443047522468095e-05 -▁parliamentary 2.5443047522468095e-05 -▁erscheint 2.5439259099668442e-05 -fre 2.5431682254069135e-05 -▁initial 2.5431682254069135e-05 -▁Schul 2.5420316985670175e-05 -▁Wieder 2.5420316985670175e-05 -▁financing 2.5420316985670175e-05 -▁experienced 2.5416528562870523e-05 -▁informed 2.5416528562870523e-05 -▁teil 2.5416528562870523e-05 -aster 2.5412740140070871e-05 -ünd 2.5412740140070871e-05 -▁translation 2.5412740140070871e-05 -▁3D 2.539758644887226e-05 -▁Versuch 2.5393798026072608e-05 -▁stable 2.5390009603272956e-05 -spiel 2.5378644334873996e-05 -▁Regelung 2.5371067489274692e-05 -▁seriously 2.5371067489274692e-05 -▁Zwischen 2.5367279066475037e-05 -я 2.5352125375276429e-05 -▁Herstellung 2.5348336952476773e-05 -▁Any 2.5329394838478509e-05 -▁Association 2.5325606415678857e-05 -▁establishment 2.5318029570079553e-05 -▁failure 2.5318029570079553e-05 -ological 2.5306664301680594e-05 -cher 2.5295299033281634e-05 -▁möglichst 2.5295299033281634e-05 -▁inf 2.5291510610481982e-05 -dienst 2.5276356919283371e-05 -▁Dienste 2.5272568496483719e-05 -▁Paul 2.5272568496483719e-05 -▁optim 2.5272568496483719e-05 -eder 2.5257414805285107e-05 -▁promoting 2.5253626382485455e-05 -▁estate 2.5242261114086496e-05 -▁Forum 2.5238472691286843e-05 -▁dangerous 2.5238472691286843e-05 -▁Sand 2.5234684268487191e-05 -▁dennoch 2.5227107422887884e-05 -▁konzentrieren 2.5227107422887884e-05 -▁voice 2.5227107422887884e-05 -▁Titel 2.5215742154488928e-05 -▁Note 2.5211953731689272e-05 -▁anyone 2.5211953731689272e-05 -▁Ö 2.5200588463290316e-05 -amt 2.5189223194891357e-05 -pen 2.5185434772091705e-05 -▁Stellungnahme 2.5181646349292053e-05 -▁Camp 2.5174069503692745e-05 -▁Zweifel 2.5170281080893093e-05 -▁finanzielle 2.5170281080893093e-05 -▁erstellen 2.5158915812494134e-05 -▁nächste 2.5151338966894829e-05 -▁Dennoch 2.5143762121295525e-05 -ée 2.5136185275696218e-05 -▁Voll 2.5136185275696218e-05 -fähigkeit 2.5128608430096914e-05 -▁Klein 2.5124820007297258e-05 -lose 2.5121031584497606e-05 -▁Jahrhunderts 2.5113454738898302e-05 -▁dir 2.510966631609865e-05 -▁Rot 2.5102089470499343e-05 -▁Regional 2.5098301047699691e-05 -ection 2.5094512624900039e-05 -▁Nachfolgend 2.5090724202100387e-05 -annt 2.507178208810212e-05 -▁Home 2.507178208810212e-05 -▁Völker 2.507178208810212e-05 -▁Achtung 2.5064205242502816e-05 -▁connected 2.5064205242502816e-05 -▁lie 2.5064205242502816e-05 -▁differences 2.5060416819703163e-05 -▁threat 2.5060416819703163e-05 -ativ 2.5056628396903511e-05 -▁Einhaltung 2.5056628396903511e-05 -▁positions 2.5056628396903511e-05 -▁bequem 2.5052839974103856e-05 -▁models 2.5052839974103856e-05 -isse 2.5045263128504552e-05 -▁Forderungen 2.50414747057049e-05 -▁Aufbau 2.5037686282905244e-05 -urn 2.503010943730594e-05 -gt 2.5018744168906981e-05 -▁studies 2.5014955746107329e-05 -у 2.5011167323307677e-05 -▁Ed 2.5007378900508025e-05 -falls 2.5003590477708369e-05 -▁difference 2.5003590477708369e-05 -oten 2.4999802054908717e-05 -▁Hall 2.4996013632109065e-05 -▁break 2.4996013632109065e-05 -▁combination 2.4996013632109065e-05 -ved 2.4988436786509761e-05 -▁Einigung 2.4988436786509761e-05 -▁Entschließungsantrag 2.4988436786509761e-05 -▁unterschiedliche 2.497328309531115e-05 -▁gegründet 2.4969494672511497e-05 -▁defence 2.496191782691219e-05 -abel 2.4954340981312886e-05 -▁Terrasse 2.4954340981312886e-05 -▁Will 2.4942975712913926e-05 -▁täglich 2.4939187290114274e-05 -▁Bet 2.4935398867314622e-05 -▁persönlich 2.4935398867314622e-05 -▁beaches 2.4927822021715315e-05 -▁box 2.4920245176116011e-05 -▁Mel 2.4916456753316355e-05 -▁appears 2.4916456753316355e-05 -▁booking 2.4901303062117747e-05 -▁Kingdom 2.4897514639318092e-05 -▁verfolgen 2.4897514639318092e-05 -▁Schengen 2.4889937793718788e-05 -▁structures 2.4889937793718788e-05 -▁associated 2.4882360948119484e-05 -▁Saint 2.4878572525319828e-05 -▁Aspekt 2.4867207256920872e-05 -▁Unterkunft 2.4867207256920872e-05 -▁humanitarian 2.4867207256920872e-05 -▁übertragen 2.4867207256920872e-05 -▁carefully 2.4863418834121217e-05 -▁Formen 2.4855841988521912e-05 -▁verd 2.4855841988521912e-05 -▁Kirche 2.485205356572226e-05 -▁amerikanischen 2.4844476720122953e-05 -▁happened 2.4844476720122953e-05 -▁reservation 2.48406882973233e-05 -▁33 2.4829323028924341e-05 -▁Hamburg 2.4829323028924341e-05 -▁rem 2.4829323028924341e-05 -▁Analyse 2.4825534606124689e-05 -▁ski 2.4821746183325037e-05 -▁article 2.4814169337725733e-05 -rech 2.4799015646527122e-05 -▁ref 2.4791438800927814e-05 -▁implementing 2.4787650378128162e-05 -▁Jugend 2.478386195532851e-05 -▁courses 2.47687082641299e-05 -▁Vis 2.4764919841330246e-05 -▁Irish 2.4761131418530594e-05 -▁Over 2.4761131418530594e-05 -▁exhibition 2.4761131418530594e-05 -▁houses 2.4757342995730939e-05 -▁Paket 2.4753554572931287e-05 -▁legen 2.4749766150131635e-05 -▁Gesundheits 2.4745977727331983e-05 -▁geleistet 2.4738400881732675e-05 -▁spielt 2.4738400881732675e-05 -ifying 2.4734612458933023e-05 -urg 2.4734612458933023e-05 -▁instance 2.4734612458933023e-05 -▁Firstly 2.4730824036133371e-05 -▁betrachtet 2.4730824036133371e-05 -kten 2.4727035613333719e-05 -▁umfasst 2.4727035613333719e-05 -▁nennen 2.4711881922135108e-05 -▁Texte 2.4700516653736148e-05 -▁verlassen 2.4696728230936496e-05 -▁Großbritannien 2.4689151385337189e-05 -▁prefer 2.4689151385337189e-05 -▁u. 2.4689151385337189e-05 -11 2.4685362962537537e-05 -inter 2.4685362962537537e-05 -▁75 2.4685362962537537e-05 -▁Fischerei 2.4685362962537537e-05 -vollen 2.4681574539737885e-05 -▁statements 2.467399769413858e-05 -▁unabhängig 2.467399769413858e-05 -▁una 2.4662632425739621e-05 -▁1994 2.4655055580140313e-05 -▁Aussicht 2.4655055580140313e-05 -▁attractive 2.4655055580140313e-05 -▁impossible 2.4651267157340661e-05 -▁names 2.463611346614205e-05 -▁Main 2.4628536620542746e-05 -▁diskutieren 2.4624748197743094e-05 -▁schönen 2.4624748197743094e-05 -▁shared 2.4624748197743094e-05 -▁Sehenswürdigkeiten 2.460580608374483e-05 -▁concrete 2.4594440815345871e-05 -▁agencies 2.4586863969746563e-05 -richt 2.4583075546946911e-05 -▁denken 2.4571710278547955e-05 -▁leader 2.4571710278547955e-05 -▁spent 2.4571710278547955e-05 -▁immediate 2.4556556587349343e-05 -anda 2.4545191318950384e-05 -▁Gut 2.4545191318950384e-05 -▁architecture 2.4545191318950384e-05 -▁reducing 2.4545191318950384e-05 -▁Resort 2.453761447335108e-05 -▁pictures 2.453761447335108e-05 -▁Wellness 2.4533826050551424e-05 -box 2.4530037627751772e-05 -▁48 2.452624920495212e-05 -länder 2.4522460782152468e-05 -▁Due 2.4514883936553161e-05 -▁Trade 2.4503518668154205e-05 -▁producers 2.4499730245354549e-05 -▁Gemeinsamen 2.4492153399755245e-05 -▁announced 2.4492153399755245e-05 -▁alt 2.4476999708556633e-05 -▁coast 2.4469422862957329e-05 -▁arrival 2.4465634440157677e-05 -▁Dorf 2.445805759455837e-05 -▁rent 2.445805759455837e-05 -▁Ideen 2.444669232615941e-05 -▁reasonable 2.444669232615941e-05 -▁Lesung 2.4439115480560106e-05 -▁Ser 2.4439115480560106e-05 -▁watch 2.4439115480560106e-05 -ehen 2.4423961789361495e-05 -▁adoption 2.4416384943762191e-05 -zy 2.4412596520962535e-05 -▁Mindest 2.4412596520962535e-05 -▁positiv 2.4408808098162883e-05 -▁garantiert 2.4405019675363231e-05 -▁Digital 2.4386077561364967e-05 -▁Links 2.4378500715765663e-05 -▁net 2.4378500715765663e-05 -▁ben 2.4374712292966008e-05 -▁contain 2.4370923870166356e-05 -▁truly 2.4370923870166356e-05 -▁counter 2.4367135447366704e-05 -▁hören 2.4367135447366704e-05 -▁Are 2.4363347024567052e-05 -▁opt 2.4363347024567052e-05 -▁Support 2.4355770178967744e-05 -▁evidence 2.4351981756168092e-05 -uff 2.434819333336844e-05 -▁Hersteller 2.434819333336844e-05 -▁television 2.4344404910568788e-05 -ci 2.4340616487769133e-05 -ding 2.4340616487769133e-05 -▁Ruhe 2.4340616487769133e-05 -ih 2.4336828064969481e-05 -ead 2.4333039642169829e-05 -▁Anwendungen 2.4333039642169829e-05 -▁guarantees 2.4333039642169829e-05 -▁background 2.4329251219370177e-05 -▁calling 2.4325462796570521e-05 -▁feststellen 2.4325462796570521e-05 -iff 2.4321674373770869e-05 -▁Umfang 2.4321674373770869e-05 -▁unto 2.4314097528171565e-05 -app 2.4306520682572258e-05 -▁Richtlinien 2.4306520682572258e-05 -▁availability 2.4306520682572258e-05 -▁arbeitet 2.4302732259772606e-05 -▁charged 2.4302732259772606e-05 -▁released 2.4302732259772606e-05 -▁erforderlichen 2.4298943836972953e-05 -▁advance 2.4287578568573994e-05 -▁described 2.4287578568573994e-05 -▁ev 2.428000172297469e-05 -▁gef 2.428000172297469e-05 -mäßig 2.4264848031776078e-05 -▁Sehr 2.4261059608976426e-05 -▁Gal 2.4253482763377119e-05 -▁Sorge 2.4242117494978163e-05 -▁genuine 2.4242117494978163e-05 -▁Gleichzeitig 2.4238329072178507e-05 -▁ahead 2.4230752226579203e-05 -▁Ak 2.42231753809799e-05 -▁imm 2.4215598535380592e-05 -erne 2.421181011258094e-05 -▁Charakter 2.421181011258094e-05 -▁recognition 2.4208021689781287e-05 -▁Vielzahl 2.4204233266981632e-05 -▁Volks 2.4204233266981632e-05 -ament 2.420044484418198e-05 -▁Stra 2.420044484418198e-05 -▁England 2.4196656421382328e-05 -▁überprüfen 2.4177714307384064e-05 -▁Far 2.4166349038985105e-05 -▁Wald 2.4162560616185453e-05 -elling 2.4154983770586149e-05 -▁sol 2.4154983770586149e-05 -bahn 2.4151195347786493e-05 -▁cards 2.4151195347786493e-05 -▁Justice 2.4147406924986841e-05 -▁Ass 2.4143618502187189e-05 -▁obtain 2.4139830079387537e-05 -▁improvement 2.4136041656587885e-05 -▁formal 2.4128464810988578e-05 -▁relaxing 2.4128464810988578e-05 -verl 2.4120887965389273e-05 -▁EG 2.4120887965389273e-05 -ving 2.4117099542589618e-05 -▁fin 2.4117099542589618e-05 -▁wishes 2.4113311119789966e-05 -▁ausreichend 2.4109522696990314e-05 -▁possibilities 2.4105734274190662e-05 -eas 2.410194585139101e-05 -▁Polen 2.4098157428591354e-05 -▁stattfinden 2.40867921601924e-05 -▁Other 2.4079215314593091e-05 -▁urban 2.4079215314593091e-05 -▁Prime 2.4075426891793439e-05 -▁membership 2.4075426891793439e-05 -▁richtigen 2.4060273200594827e-05 -▁Stärkung 2.4056484777795175e-05 -▁nearby 2.4056484777795175e-05 -▁Rechnung 2.4041331086596564e-05 -▁glass 2.4041331086596564e-05 -▁monetary 2.4037542663796912e-05 -▁einiger 2.4026177395397952e-05 -▁discrimination 2.4018600549798648e-05 -▁miss 2.4007235281399688e-05 -▁unacceptable 2.4007235281399688e-05 -▁Roman 2.3999658435800384e-05 -▁Patienten 2.3980716321802121e-05 -▁Ferien 2.3973139476202813e-05 -ently 2.3969351053403161e-05 -▁territory 2.3969351053403161e-05 -▁Arbeitslosigkeit 2.3965562630603509e-05 -▁kulturellen 2.3950408939404898e-05 -zahl 2.3946620516605246e-05 -▁Seine 2.3946620516605246e-05 -▁Live 2.3939043671005938e-05 -▁ret 2.3939043671005938e-05 -preis 2.3935255248206286e-05 -▁Landschaft 2.3935255248206286e-05 -▁Tourismus 2.3935255248206286e-05 -ool 2.3931466825406634e-05 -▁hingewiesen 2.3931466825406634e-05 -▁tasks 2.3931466825406634e-05 -▁consumption 2.3920101557007674e-05 -▁continued 2.3916313134208022e-05 -▁unless 2.3908736288608715e-05 -▁Entwicklungen 2.3901159443009411e-05 -▁Ins 2.3897371020209759e-05 -▁tätig 2.3897371020209759e-05 --Sch 2.3893582597410107e-05 -▁zuvor 2.3889794174610451e-05 -▁übrigen 2.38860057518108e-05 -▁Fisch 2.3882217329011147e-05 -▁Pack 2.387464048341184e-05 -by 2.3870852060612188e-05 -ishing 2.3870852060612188e-05 -▁cur 2.3870852060612188e-05 -alk 2.3863275215012884e-05 -▁CO2 2.3863275215012884e-05 -▁USB 2.3863275215012884e-05 -▁skills 2.3863275215012884e-05 -▁seek 2.3855698369413576e-05 -▁wood 2.3855698369413576e-05 -AG 2.3851909946613924e-05 -▁Bedarf 2.3851909946613924e-05 -▁miteinander 2.3851909946613924e-05 -hilfe 2.384433310101462e-05 -▁Künstler 2.384433310101462e-05 -lands 2.3840544678214968e-05 -▁Poker 2.3840544678214968e-05 -▁cars 2.3832967832615661e-05 -▁Barroso 2.3829179409816008e-05 -sta 2.3825390987016356e-05 -▁trading 2.38216025642167e-05 -▁Risiko 2.3817814141417049e-05 -off 2.3814025718617397e-05 -▁Afrika 2.3814025718617397e-05 -▁continues 2.3810237295817745e-05 -eld 2.3798872027418785e-05 -▁Stock 2.3798872027418785e-05 --con 2.3795083604619133e-05 -▁Link 2.3795083604619133e-05 -▁figures 2.3795083604619133e-05 -LE 2.3791295181819481e-05 -▁Foundation 2.3787506759019826e-05 -▁det 2.3787506759019826e-05 -ügen 2.3779929913420522e-05 -▁packages 2.377614149062087e-05 -▁unfortunately 2.377614149062087e-05 -▁erw 2.3772353067821218e-05 -▁bezeichnet 2.3768564645021562e-05 -▁sup 2.3768564645021562e-05 -▁African 2.3760987799422258e-05 -▁combined 2.3757199376622606e-05 -▁devices 2.3753410953822954e-05 -▁hom 2.37496225310233e-05 -▁participate 2.37496225310233e-05 -acy 2.3742045685423995e-05 -vo 2.3738257262624342e-05 -▁introduce 2.3738257262624342e-05 -werke 2.3734468839824687e-05 -▁greatest 2.3726891994225383e-05 -▁Bas 2.3723103571425731e-05 -ugen 2.3715526725826423e-05 -▁profitieren 2.3715526725826423e-05 -▁Beschäftigungs 2.3707949880227119e-05 -▁completed 2.3704161457427467e-05 -▁generation 2.3704161457427467e-05 -▁Michael 2.3700373034627812e-05 -▁strongly 2.3700373034627812e-05 -ora 2.3685219343429204e-05 -▁impl 2.3681430920629548e-05 -▁Leistung 2.3673854075030244e-05 -schein 2.3670065652230592e-05 -▁desire 2.3666277229430937e-05 -▁Fol 2.3662488806631285e-05 -▁Ap 2.3647335115432673e-05 -▁Dollar 2.3647335115432673e-05 -ately 2.3635969847033717e-05 -▁beauty 2.3635969847033717e-05 -▁19. 2.3632181424234065e-05 -▁aufge 2.3628393001434409e-05 -,5 2.3620816155835105e-05 -▁raise 2.3620816155835105e-05 -hotel 2.3617027733035453e-05 -▁surrounding 2.3609450887436146e-05 -▁allowing 2.3605662464636494e-05 -▁• 2.3605662464636494e-05 -▁Back 2.3601874041836842e-05 -▁Festival 2.359808561903719e-05 -igh 2.3590508773437882e-05 -▁uner 2.3590508773437882e-05 -▁weiterer 2.3590508773437882e-05 -▁enables 2.3582931927838578e-05 -ble 2.3579143505038923e-05 -▁zahlen 2.3575355082239271e-05 -▁Deutsche 2.3571566659439619e-05 -sel 2.3567778236639967e-05 -▁explain 2.3563989813840315e-05 -▁perform 2.3541259277042395e-05 -▁Media 2.3537470854242743e-05 -▁eingeführt 2.3537470854242743e-05 -▁stat 2.3529894008643439e-05 -▁command 2.351852874024448e-05 -▁verloren 2.351852874024448e-05 -achten 2.3510951894645176e-05 -▁Irland 2.350716347184552e-05 -▁aufmerksam 2.350716347184552e-05 -▁Pf 2.3499586626246216e-05 -▁Europeans 2.3495798203446564e-05 -▁Bedürfnisse 2.3488221357847257e-05 -▁SP 2.3488221357847257e-05 -▁conduct 2.3488221357847257e-05 -▁Teilnehmer 2.3480644512247953e-05 -▁attractions 2.3480644512247953e-05 -▁gewinnen 2.3480644512247953e-05 -▁symbol 2.34768560894483e-05 -▁wor 2.34768560894483e-05 -▁religious 2.3461702398249689e-05 -▁window 2.3461702398249689e-05 -ense 2.3457913975450037e-05 -▁shop 2.3450337129850729e-05 -▁Konflikt 2.3446548707051077e-05 -▁wh 2.3446548707051077e-05 -ländern 2.343897186145177e-05 -▁Lo 2.343897186145177e-05 -▁Turkish 2.343897186145177e-05 -▁Very 2.3435183438652118e-05 -▁mission 2.3412452901854202e-05 -▁registered 2.3412452901854202e-05 -▁river 2.3412452901854202e-05 -esh 2.340866447905455e-05 -▁spread 2.340866447905455e-05 -▁Altstadt 2.3397299210655591e-05 -▁relativ 2.3389722365056287e-05 --J 2.3382145519456979e-05 -▁absolute 2.3374568673857675e-05 -▁verm 2.3374568673857675e-05 -ores 2.3366991828258368e-05 -▁Vienna 2.3366991828258368e-05 -▁Wissenschaft 2.3366991828258368e-05 -▁economies 2.3366991828258368e-05 -å 2.3363203405458716e-05 -▁Ol 2.3363203405458716e-05 -▁Innen 2.3359414982659063e-05 -▁Justiz 2.3359414982659063e-05 -met 2.3351838137059756e-05 -▁Prot 2.3332896023061492e-05 -▁schön 2.3332896023061492e-05 -▁Fe 2.332910760026184e-05 -▁Problemen 2.332910760026184e-05 -abe 2.3313953909063229e-05 -▁Beschreibung 2.3306377063463925e-05 -▁beginnt 2.3306377063463925e-05 -▁Überwachung 2.3306377063463925e-05 -▁Fotos 2.3302588640664273e-05 -hnen 2.3298800217864617e-05 -gleich 2.3295011795064965e-05 -▁überall 2.3295011795064965e-05 -▁insurance 2.3291223372265313e-05 -▁meters 2.3291223372265313e-05 -▁zentrale 2.3291223372265313e-05 -▁Di 2.3287434949465661e-05 -▁Foto 2.3283646526666006e-05 -pflicht 2.327228125826705e-05 -▁Democrats 2.327228125826705e-05 -▁entitled 2.327228125826705e-05 -▁160 2.3268492835467397e-05 -▁closer 2.3268492835467397e-05 -▁gem 2.3268492835467397e-05 -atten 2.326091598986809e-05 -▁surface 2.326091598986809e-05 -▁geschehen 2.3257127567068438e-05 -▁Tod 2.3253339144268786e-05 -omen 2.3245762298669478e-05 -uct 2.3241973875869826e-05 -stelle 2.3238185453070174e-05 -▁Mer 2.3238185453070174e-05 -▁button 2.3238185453070174e-05 -▁danach 2.3234397030270522e-05 -ches 2.3226820184671215e-05 -▁Angesichts 2.3226820184671215e-05 -▁Geschäftsordnung 2.3226820184671215e-05 -▁einsetzen 2.3226820184671215e-05 -▁Technik 2.3223031761871563e-05 -ables 2.3215454916272259e-05 -▁Beach 2.3211666493472603e-05 -▁dedicated 2.3207878070672951e-05 -▁unterstütze 2.3207878070672951e-05 -ulated 2.3200301225073647e-05 -▁Drogen 2.3177570688275728e-05 -ida 2.3173782265476076e-05 -▁34 2.3169993842676424e-05 -▁Data 2.3169993842676424e-05 -▁geschützt 2.3169993842676424e-05 -▁Beitritt 2.3162416997077116e-05 -ala 2.3154840151477812e-05 -▁regionalen 2.315105172867816e-05 -▁san 2.3147263305878508e-05 -erung 2.31396864602792e-05 -▁il 2.31396864602792e-05 -▁manage 2.3124532769080589e-05 -▁betroffen 2.3113167500681633e-05 -▁pour 2.3113167500681633e-05 -iles 2.3109379077881978e-05 -mel 2.3109379077881978e-05 -в 2.3105590655082326e-05 -▁fruit 2.3105590655082326e-05 -▁sämtliche 2.3079071695484758e-05 -▁sonst 2.3075283272685103e-05 -▁keinerlei 2.307149484988545e-05 -▁vorliegenden 2.307149484988545e-05 -▁Griechenland 2.3060129581486494e-05 -▁Rome 2.3056341158686839e-05 -▁bestätigt 2.3056341158686839e-05 -▁commun 2.3052552735887187e-05 -▁Jeder 2.3048764313087535e-05 -▁Bal 2.3044975890287883e-05 -▁Lake 2.3041187467488231e-05 -▁sale 2.3041187467488231e-05 -▁37 2.3037399044688575e-05 -▁besitzt 2.3037399044688575e-05 -▁ernst 2.3033610621888923e-05 -▁Tal 2.3026033776289619e-05 -▁engine 2.3026033776289619e-05 -▁gegenwärtig 2.3026033776289619e-05 -ätze 2.3022245353489964e-05 -fol 2.3018456930690312e-05 -▁Geb 2.3010880085091008e-05 -zogen 2.30033032394917e-05 -▁Meiner 2.30033032394917e-05 -▁square 2.30033032394917e-05 -▁Eigen 2.2988149548293089e-05 -▁Fortschritt 2.2984361125493437e-05 -▁Priorität 2.2984361125493437e-05 -▁Dis 2.2980572702693784e-05 -▁Kal 2.2980572702693784e-05 -utes 2.2976784279894132e-05 -volle 2.297299585709448e-05 -▁Federal 2.2969207434294825e-05 -▁union 2.2969207434294825e-05 -▁Europäer 2.2961630588695521e-05 -▁Kind 2.2961630588695521e-05 -▁enormous 2.2957842165895869e-05 -▁näher 2.2957842165895869e-05 -▁Anlagen 2.2954053743096213e-05 -karte 2.2946476897496909e-05 -osen 2.2946476897496909e-05 -▁learning 2.2946476897496909e-05 -▁exercise 2.2931323206298298e-05 -▁Wind 2.2927534783498646e-05 -▁Another 2.2923746360698994e-05 -▁vollkommen 2.2916169515099686e-05 -▁großes 2.2912381092300034e-05 -▁spezielle 2.2908592669500382e-05 -▁compared 2.290480424670073e-05 -▁Elemente 2.2901015823901075e-05 -▁einfache 2.2901015823901075e-05 -▁verfolgt 2.2901015823901075e-05 -▁politischer 2.2893438978301771e-05 -▁individuals 2.2885862132702466e-05 -▁perspective 2.2885862132702466e-05 -▁railway 2.2885862132702466e-05 -▁installed 2.2878285287103159e-05 -▁session 2.2878285287103159e-05 -▁reliable 2.2874496864303507e-05 -▁totally 2.2874496864303507e-05 -▁Bis 2.2870708441503855e-05 -▁Dom 2.28669200187042e-05 -stellungen 2.2863131595904547e-05 -▁Prinzip 2.2859343173104895e-05 -▁wind 2.2859343173104895e-05 -ids 2.2855554750305243e-05 -▁asylum 2.2855554750305243e-05 -▁bath 2.2855554750305243e-05 -▁buffet 2.2851766327505591e-05 -▁beteiligt 2.2840401059106632e-05 -▁französischen 2.2832824213507328e-05 -▁luxury 2.2829035790707672e-05 -▁vision 2.282524736790802e-05 -▁bill 2.2821458945108368e-05 -▁Verpflichtung 2.2817670522308716e-05 -▁gestalten 2.2813882099509061e-05 -▁diss 2.2802516831110105e-05 -▁gross 2.2802516831110105e-05 -▁Bewertung 2.2798728408310452e-05 -▁Book 2.2798728408310452e-05 -▁offensichtlich 2.2791151562711145e-05 -▁cons 2.2787363139911493e-05 -iness 2.2775997871512533e-05 -scha 2.2775997871512533e-05 -stell 2.2764632603113577e-05 -▁advertising 2.2764632603113577e-05 -fert 2.2760844180313922e-05 -itter 2.2760844180313922e-05 -wagen 2.2760844180313922e-05 -▁größeren 2.2760844180313922e-05 -▁Cap 2.275705575751427e-05 -▁empfehlen 2.2741902066315658e-05 -▁angeboten 2.2734325220716354e-05 -▁kept 2.2730536797916702e-05 -▁meist 2.2722959952317395e-05 -cia 2.2719171529517743e-05 -▁holidays 2.2715383106718091e-05 -▁storage 2.2715383106718091e-05 -iving 2.2707806261118783e-05 -án 2.2707806261118783e-05 -▁Il 2.2704017838319131e-05 -▁extended 2.2700229415519479e-05 -▁ag 2.2696440992719827e-05 -▁Austrian 2.2692652569920171e-05 -▁gutes 2.2688864147120519e-05 -▁Nevertheless 2.2685075724320867e-05 -▁forced 2.2677498878721563e-05 -aging 2.2673710455921908e-05 -▁1990 2.2669922033122256e-05 -▁Good 2.2669922033122256e-05 -▁wonder 2.2666133610322604e-05 -▁stands 2.2658556764723296e-05 -▁Ir 2.264719149632434e-05 -▁früher 2.2635826227925381e-05 -▁platform 2.2635826227925381e-05 -▁spending 2.2635826227925381e-05 -▁claims 2.2628249382326077e-05 -äm 2.2624460959526425e-05 -▁corporate 2.2620672536726769e-05 -cing 2.2616884113927117e-05 -ater 2.2613095691127465e-05 -▁zusätzlich 2.2613095691127465e-05 -▁Schaden 2.2609307268327813e-05 -▁gu 2.2609307268327813e-05 -▁biggest 2.2605518845528158e-05 -▁Indien 2.2601730422728505e-05 -▁sides 2.2601730422728505e-05 -▁Globalisierung 2.25941535771292e-05 -▁Av 2.2590365154329549e-05 -▁Tisch 2.2586576731529894e-05 -▁device 2.2582788308730242e-05 -▁convinced 2.2575211463130938e-05 -▁AR 2.2571423040331282e-05 -▁Auftrag 2.2571423040331282e-05 -▁Regelungen 2.2571423040331282e-05 -▁warten 2.256763461753163e-05 -any 2.2563846194731978e-05 -▁Foreign 2.2563846194731978e-05 -▁golf 2.2563846194731978e-05 -▁Liefer 2.2560057771932326e-05 -▁story 2.2560057771932326e-05 -ello 2.2548692503533367e-05 -▁Mein 2.2544904080733715e-05 -▁gain 2.2544904080733715e-05 -▁urgent 2.2541115657934063e-05 -▁science 2.2537327235134411e-05 -▁memory 2.2533538812334755e-05 -▁Christ 2.2529750389535103e-05 -▁Worten 2.2525961966735451e-05 -▁failed 2.25221735439358e-05 -▁voraus 2.25221735439358e-05 -▁Botschaft 2.2514596698336492e-05 -▁Grenz 2.2510808275536839e-05 -▁garantieren 2.2510808275536839e-05 -▁AC 2.2488077738738924e-05 -▁Infrastruktur 2.2484289315939268e-05 -▁Les 2.2484289315939268e-05 -▁highlight 2.2480500893139616e-05 -▁Market 2.2476712470339964e-05 -▁ergeben 2.2472924047540312e-05 -▁mountains 2.2472924047540312e-05 -▁Netherlands 2.2465347201941005e-05 -▁Kle 2.2461558779141353e-05 -▁granted 2.2461558779141353e-05 -ieß 2.24577703563417e-05 -▁Day 2.2450193510742393e-05 -öl 2.2446405087942741e-05 -▁umfassende 2.2442616665143089e-05 -▁gegenwärtigen 2.2438828242343437e-05 -▁quick 2.2438828242343437e-05 -ua 2.2435039819543785e-05 -▁Islam 2.2435039819543785e-05 -▁Pap 2.2435039819543785e-05 -▁Karten 2.243125139674413e-05 -eam 2.2427462973944478e-05 -▁Maße 2.2427462973944478e-05 -i- 2.2408520859946214e-05 -▁self 2.2404732437146562e-05 -▁Universität 2.240094401434691e-05 -▁folgende 2.2397155591547254e-05 -▁para 2.2397155591547254e-05 -▁phys 2.2397155591547254e-05 -▁core 2.238957874594795e-05 -lichem 2.2385790323148298e-05 -üsse 2.2385790323148298e-05 -▁38 2.2385790323148298e-05 -▁app 2.2385790323148298e-05 -▁internationaler 2.2378213477548991e-05 -▁passed 2.2378213477548991e-05 -▁Direkt 2.2366848209150035e-05 -▁Eastern 2.2366848209150035e-05 -▁Eindruck 2.2366848209150035e-05 -▁Jo 2.2363059786350379e-05 -ops 2.2344117672352116e-05 -▁moving 2.2340329249552464e-05 -ura 2.233275240395316e-05 -▁beinhaltet 2.233275240395316e-05 -▁advice 2.23213871355542e-05 -▁bra 2.23213871355542e-05 -▁verbundenen 2.23213871355542e-05 -▁conclusions 2.2313810289954896e-05 -▁Begriff 2.231002186715524e-05 -▁Front 2.231002186715524e-05 -▁Alt 2.2306233444355588e-05 -▁controls 2.2306233444355588e-05 -▁Van 2.2302445021555936e-05 -com 2.2291079753156977e-05 -fälle 2.2291079753156977e-05 -oms 2.2291079753156977e-05 -▁Sonnen 2.2283502907557673e-05 -▁Vol 2.2283502907557673e-05 -▁Company 2.2279714484758021e-05 -▁Latin 2.2275926061958365e-05 -isierte 2.2272137639158713e-05 -s- 2.2272137639158713e-05 -▁Meeres 2.2272137639158713e-05 -km 2.2268349216359061e-05 -▁Größe 2.2264560793559409e-05 -▁establishing 2.2264560793559409e-05 -▁weight 2.2264560793559409e-05 -ored 2.2260772370759757e-05 -▁solcher 2.2260772370759757e-05 -▁unit 2.2260772370759757e-05 -▁tief 2.225319552516045e-05 -▁Gebrauch 2.2245618679561146e-05 -▁finance 2.2245618679561146e-05 -Innen 2.2238041833961838e-05 -org 2.2238041833961838e-05 -owing 2.2234253411162186e-05 -istr 2.2215311297163922e-05 -wir 2.2215311297163922e-05 -otel 2.221152287436427e-05 -▁Monat 2.2207734451564618e-05 -▁abgeschlossen 2.2207734451564618e-05 -▁acht 2.2200157605965311e-05 -▁west 2.2200157605965311e-05 -▁Mil 2.2196369183165659e-05 -▁fiscal 2.2196369183165659e-05 -▁saw 2.2196369183165659e-05 -, 2.2192580760366007e-05 -▁deliver 2.2192580760366007e-05 -▁mobil 2.2188792337566351e-05 -▁unable 2.2188792337566351e-05 -nt 2.2181215491967047e-05 -▁guide 2.2173638646367743e-05 -▁re- 2.2173638646367743e-05 -atives 2.2166061800768436e-05 -wasser 2.2162273377968784e-05 -▁disease 2.2162273377968784e-05 -▁Control 2.2158484955169132e-05 -▁Nor 2.2154696532369476e-05 -alter 2.2150908109569824e-05 -▁4. 2.2147119686770172e-05 -▁Nutzer 2.2147119686770172e-05 -▁fire 2.2147119686770172e-05 -▁EL 2.214333126397052e-05 -▁Greek 2.214333126397052e-05 -▁perfectly 2.214333126397052e-05 -lem 2.2139542841170868e-05 -▁Ur 2.2139542841170868e-05 -▁contrary 2.2139542841170868e-05 -aps 2.2135754418371213e-05 -▁church 2.213196599557156e-05 -▁scale 2.2124389149972256e-05 -ham 2.2116812304372949e-05 -▁Bur 2.2113023881573297e-05 -▁Regulation 2.2105447035973993e-05 -ils 2.2097870190374685e-05 -uchten 2.2097870190374685e-05 -▁Märkte 2.2097870190374685e-05 -▁Studio 2.2097870190374685e-05 -▁factors 2.2097870190374685e-05 -ieg 2.2090293344775381e-05 -ude 2.2090293344775381e-05 -▁ro 2.2086504921975729e-05 -▁tried 2.2086504921975729e-05 -▁Beteiligung 2.207513965357677e-05 -▁Gründe 2.207513965357677e-05 -oo 2.2067562807977462e-05 -▁Lizenz 2.2059985962378158e-05 -▁wesentlichen 2.2041043848379894e-05 -▁'re 2.2037255425580242e-05 -▁drawn 2.2037255425580242e-05 -ific 2.2033467002780587e-05 -▁bew 2.2029678579980935e-05 -н 2.2018313311581979e-05 -▁chosen 2.2010736465982671e-05 -▁interface 2.2010736465982671e-05 -▁Constitution 2.2006948043183019e-05 -▁Erklärungen 2.2006948043183019e-05 -▁artists 2.2006948043183019e-05 -hem 2.2003159620383367e-05 -atoren 2.1999371197583715e-05 -▁Palace 2.199558277478406e-05 -ured 2.1991794351984408e-05 -gabe 2.1988005929184756e-05 -▁entsprechen 2.1988005929184756e-05 -▁aufzu 2.1984217506385104e-05 -▁maintenance 2.1984217506385104e-05 -▁Special 2.1980429083585448e-05 --System 2.1972852237986144e-05 -▁Sw 2.1969063815186492e-05 -▁definition 2.196527539238684e-05 -▁98 2.1961486969587185e-05 -ify 2.1957698546787533e-05 -▁te 2.1953910123987881e-05 -▁weapons 2.1953910123987881e-05 -▁compl 2.1938756432789269e-05 -▁fe 2.1938756432789269e-05 -▁laid 2.1938756432789269e-05 -▁mo 2.1938756432789269e-05 -▁rightly 2.1934968009989617e-05 -▁route 2.1934968009989617e-05 -ount 2.1931179587189965e-05 -▁250 2.1927391164390309e-05 -▁Grunde 2.1919814318791005e-05 -fahrt 2.1916025895991353e-05 -▁Kurs 2.1916025895991353e-05 -▁Martin 2.1916025895991353e-05 -▁neither 2.19122374731917e-05 -▁win 2.19122374731917e-05 -▁Thus 2.1908449050392046e-05 -▁erhöht 2.1908449050392046e-05 -▁collect 2.1904660627592394e-05 -▁Bahnhof 2.1900872204792742e-05 -▁blog 2.189708378199309e-05 -ula 2.1893295359193434e-05 -▁honourable 2.1893295359193434e-05 -▁Institute 2.188571851359413e-05 -▁aut 2.188571851359413e-05 -ams 2.1874353245195171e-05 -▁Unabhängigkeit 2.1870564822395519e-05 -▁expression 2.1870564822395519e-05 -▁geöffnet 2.1866776399595867e-05 -▁Yet 2.1862987976796215e-05 -staaten 2.1859199553996559e-05 -eable 2.1855411131196907e-05 -teile 2.1851622708397255e-05 -▁Militär 2.1847834285597603e-05 -▁commitments 2.1844045862797951e-05 -▁Städte 2.1840257439998295e-05 -▁psych 2.1840257439998295e-05 -▁spa 2.1836469017198643e-05 -änder 2.1832680594398991e-05 -▁Maß 2.1832680594398991e-05 -▁photos 2.1821315326000032e-05 -iqu 2.181752690320038e-05 -TS 2.1809950057601076e-05 -ros 2.180616163480142e-05 -ç 2.1798584789202116e-05 -▁Po 2.1798584789202116e-05 -▁candidate 2.1798584789202116e-05 -▁outstanding 2.1798584789202116e-05 -▁Fern 2.1783431098003505e-05 -auer 2.1779642675203853e-05 -▁Marken 2.1779642675203853e-05 -▁update 2.1779642675203853e-05 -bst 2.17758542524042e-05 -▁Erde 2.17758542524042e-05 -▁outcome 2.17758542524042e-05 -▁rock 2.1772065829604545e-05 -▁betrachten 2.1768277406804893e-05 -▁agency 2.1756912138405937e-05 -▁intervention 2.1745546870006977e-05 -▁Verl 2.1741758447207325e-05 -▁Ru 2.173797002440767e-05 -▁speziellen 2.173797002440767e-05 -▁Gegensatz 2.1734181601608018e-05 -▁meaning 2.1726604756008714e-05 -isierungs 2.1722816333209062e-05 -▁Band 2.1715239487609754e-05 -▁Tür 2.1715239487609754e-05 -▁Beim 2.170766264201045e-05 -▁ambitious 2.170766264201045e-05 -▁register 2.170766264201045e-05 -▁dient 2.1703874219210798e-05 -▁high- 2.1703874219210798e-05 -izes 2.1688720528012187e-05 -▁Ko 2.1688720528012187e-05 -▁claim 2.1688720528012187e-05 -▁achieving 2.1684932105212531e-05 -▁ap 2.1684932105212531e-05 -▁patent 2.1684932105212531e-05 --r 2.1681143682412879e-05 -▁Net 2.1681143682412879e-05 -▁flight 2.1673566836813575e-05 -▁sensitive 2.1673566836813575e-05 -ee 2.1669778414013923e-05 -▁Mas 2.1662201568414615e-05 -▁obviously 2.1654624722815311e-05 -unk 2.1650836300015656e-05 -SP 2.1631894186017392e-05 -▁stra 2.1631894186017392e-05 -▁constantly 2.1616740494818784e-05 -▁ECU 2.1612952072019129e-05 -▁Kopf 2.1601586803620173e-05 -▁Studien 2.1601586803620173e-05 -äger 2.1597798380820517e-05 -▁Flexibilität 2.1597798380820517e-05 -▁langen 2.1594009958020865e-05 -▁Strom 2.1590221535221213e-05 -▁approximately 2.1586433112421561e-05 -▁Pool 2.1567490998423297e-05 -▁zentral 2.1563702575623642e-05 -ports 2.155991415282399e-05 -▁aircraft 2.1556125730024338e-05 -▁block 2.1552337307224686e-05 -▁gelungen 2.1552337307224686e-05 -illa 2.1548548884425034e-05 -▁Verständnis 2.1548548884425034e-05 -▁sell 2.1544760461625378e-05 -▁Sam 2.1537183616026074e-05 -▁Hinweis 2.1529606770426767e-05 -▁Kontroll 2.1529606770426767e-05 -▁rail 2.1529606770426767e-05 -▁5. 2.1522029924827463e-05 -▁colleague 2.1522029924827463e-05 -▁promotion 2.1522029924827463e-05 -▁starke 2.1518241502027811e-05 -ected 2.1514453079228159e-05 -ival 2.1514453079228159e-05 -▁vehicle 2.1510664656428503e-05 -▁Datenschutz 2.1506876233628851e-05 -▁schöne 2.1506876233628851e-05 -▁Ereignisse 2.1499299388029547e-05 -gu 2.1495510965229895e-05 -▁obligations 2.1495510965229895e-05 -▁Betriebs 2.149172254243024e-05 -▁surrounded 2.149172254243024e-05 -setzung 2.1484145696830936e-05 -▁Aktionen 2.1453838314433712e-05 -▁paragraph 2.1453838314433712e-05 -▁category 2.145004989163406e-05 -ball 2.1446261468834408e-05 -▁d. 2.1446261468834408e-05 -cht 2.1442473046034753e-05 -ering 2.1442473046034753e-05 -▁britischen 2.1442473046034753e-05 -▁sauber 2.14386846232351e-05 -▁42 2.1434896200435449e-05 -anlage 2.1431107777635797e-05 -OT 2.1427319354836145e-05 -▁informiert 2.1427319354836145e-05 -▁Weitere 2.1423530932036489e-05 -▁limits 2.1423530932036489e-05 -▁someone 2.1419742509236837e-05 -▁election 2.1415954086437185e-05 -leben 2.1408377240837881e-05 -▁continuing 2.1404588818038226e-05 -bildung 2.1400800395238574e-05 -zeug 2.1400800395238574e-05 -▁Bay 2.1400800395238574e-05 -▁daraus 2.1400800395238574e-05 -▁Nachdem 2.1397011972438922e-05 -▁Peter 2.1397011972438922e-05 -▁Geist 2.139322354963927e-05 -▁Jean 2.1389435126839614e-05 -▁organised 2.1389435126839614e-05 -▁yesterday 2.1385646704039962e-05 -haltung 2.1374281435641006e-05 -▁Verein 2.13667045900417e-05 -▁Fin 2.1359127744442394e-05 -▁Immobilien 2.1359127744442394e-05 -▁Gefühl 2.1355339321642739e-05 -▁Gleich 2.1355339321642739e-05 -▁Verkauf 2.1351550898843087e-05 -▁payments 2.1351550898843087e-05 -iation 2.1347762476043435e-05 -▁allgemein 2.1343974053243783e-05 -▁delay 2.1343974053243783e-05 -▁Farben 2.1340185630444131e-05 -▁Süden 2.1340185630444131e-05 -▁Zahlen 2.1336397207644475e-05 -▁successfully 2.1336397207644475e-05 -▁accessible 2.1328820362045171e-05 -▁situations 2.1328820362045171e-05 -▁1992 2.1325031939245519e-05 -▁Under 2.1325031939245519e-05 -▁elected 2.1325031939245519e-05 -▁rapid 2.1321243516445864e-05 -▁guter 2.1317455093646212e-05 -▁é 2.1317455093646212e-05 -▁Lang 2.131366667084656e-05 -▁Wander 2.131366667084656e-05 -ér 2.1309878248046908e-05 -▁contents 2.13023014024476e-05 -gesellschaft 2.1294724556848296e-05 -ipe 2.1294724556848296e-05 -ufen 2.1294724556848296e-05 -▁manchmal 2.1294724556848296e-05 -ulate 2.1290936134048644e-05 -▁Sl 2.1287147711248992e-05 -▁title 2.1287147711248992e-05 -▁Mad 2.1279570865649684e-05 -▁Bill 2.1275782442850032e-05 -▁Feld 2.1275782442850032e-05 -▁custom 2.127199402005038e-05 -▁Schiff 2.1264417174451073e-05 -▁club 2.1264417174451073e-05 -▁equality 2.1264417174451073e-05 -▁Asyl 2.1256840328851769e-05 -▁minute 2.1256840328851769e-05 -▁THE 2.1253051906052117e-05 -.org 2.1249263483252461e-05 -▁radio 2.1249263483252461e-05 -▁goals 2.1245475060452809e-05 -▁20. 2.1237898214853505e-05 -▁192 2.123410979205385e-05 -▁hauptsächlich 2.123410979205385e-05 -▁Empfehlungen 2.1230321369254198e-05 -▁reject 2.1230321369254198e-05 -▁satellite 2.1230321369254198e-05 -▁Te 2.1226532946454546e-05 -▁Wall 2.1226532946454546e-05 -▁Realität 2.1222744523654894e-05 -▁Located 2.1218956100855242e-05 -▁race 2.1218956100855242e-05 -ace 2.1215167678055586e-05 -ades 2.1215167678055586e-05 -▁solid 2.1215167678055586e-05 -▁Ä 2.1200013986856978e-05 -losen 2.1196225564057323e-05 -▁Get 2.119243714125767e-05 -▁Sur 2.119243714125767e-05 -▁rot 2.119243714125767e-05 -▁physical 2.1184860295658366e-05 -oor 2.1181071872858711e-05 -▁Ausland 2.1181071872858711e-05 -▁organization 2.1177283450059059e-05 -era 2.1173495027259407e-05 -▁Niveau 2.1173495027259407e-05 -▁wealth 2.1173495027259407e-05 --3 2.1169706604459755e-05 -▁Anlage 2.1169706604459755e-05 -▁asking 2.1169706604459755e-05 -programme 2.1158341336060795e-05 -▁Mensch 2.1158341336060795e-05 -▁Generation 2.1154552913261143e-05 -▁Haut 2.1154552913261143e-05 -▁gro 2.1154552913261143e-05 -▁Blog 2.1146976067661836e-05 -▁Sun 2.1146976067661836e-05 -ails 2.1143187644862184e-05 -▁reflect 2.1143187644862184e-05 -▁Wirkung 2.113561079926288e-05 -▁EP 2.1131822376463228e-05 -▁Hoffnung 2.1131822376463228e-05 -▁balanced 2.1131822376463228e-05 -▁Agency 2.1128033953663572e-05 -▁Marketing 2.1128033953663572e-05 -▁Agentur 2.112424553086392e-05 -oin 2.1116668685264616e-05 -▁massive 2.1116668685264616e-05 -▁store 2.1116668685264616e-05 -druck 2.1112880262464961e-05 -▁expenditure 2.1093938148466697e-05 -ela 2.1090149725667045e-05 -▁beachten 2.1090149725667045e-05 -▁Mitteln 2.1086361302867393e-05 -▁monitor 2.1086361302867393e-05 -▁Fraktionen 2.1082572880067741e-05 -▁sens 2.1078784457268089e-05 -entr 2.1074996034468433e-05 -▁Vorteil 2.1074996034468433e-05 -▁independence 2.1074996034468433e-05 -▁recommendations 2.1074996034468433e-05 -hafte 2.1071207611668781e-05 -▁Los 2.1071207611668781e-05 -▁ideale 2.1071207611668781e-05 -▁Beschluss 2.105605392047017e-05 -inder 2.1052265497670518e-05 -▁length 2.1052265497670518e-05 -äl 2.1044688652071214e-05 -▁Gedanken 2.1044688652071214e-05 -▁por 2.1044688652071214e-05 -▁allgemeine 2.1040900229271558e-05 -▁tut 2.1040900229271558e-05 -▁laut 2.1037111806471906e-05 -▁herzlich 2.1033323383672254e-05 -fully 2.1029534960872602e-05 -▁rasch 2.1025746538072947e-05 -elles 2.1021958115273295e-05 -▁lake 2.1018169692473643e-05 -▁medium-sized 2.1014381269673991e-05 -▁Lord 2.1010592846874338e-05 -▁Sprach 2.1010592846874338e-05 -▁furnished 2.0999227578475379e-05 -▁ill 2.0980285464477115e-05 -▁gehalten 2.0976497041677463e-05 -schrift 2.0965131773278504e-05 -bringen 2.0961343350478852e-05 -▁mittels 2.09575549276792e-05 -▁seven 2.09575549276792e-05 -▁write 2.09575549276792e-05 -▁geschieht 2.0953766504879544e-05 -angebot 2.0949978082079892e-05 -▁einverstanden 2.0949978082079892e-05 -▁Dimension 2.094618965928024e-05 -▁motor 2.094618965928024e-05 -▁Grad 2.0938612813680933e-05 -▁extend 2.0934824390881281e-05 -24 2.0919670699682669e-05 -▁changing 2.0919670699682669e-05 -▁i.e. 2.0919670699682669e-05 -▁treten 2.0919670699682669e-05 -▁festgestellt 2.0915882276883017e-05 -▁dé 2.0908305431283713e-05 -▁profit 2.0904517008484058e-05 -▁sobald 2.0896940162884753e-05 -abkommen 2.08931517400851e-05 -▁consists 2.08931517400851e-05 -haften 2.0889363317285449e-05 -▁targets 2.0889363317285449e-05 -▁Zoll 2.0885574894485794e-05 -▁merely 2.0881786471686142e-05 -▁Environment 2.087799804888649e-05 -▁Vom 2.087799804888649e-05 -▁Beihilfen 2.0874209626086838e-05 -▁Ke 2.0874209626086838e-05 -▁niemand 2.0870421203287186e-05 -▁Pa 2.086663278048753e-05 -▁Treffen 2.0862844357687878e-05 -ova 2.0859055934888226e-05 -▁Gran 2.0859055934888226e-05 -▁Voraussetzungen 2.0859055934888226e-05 -▁medium 2.0859055934888226e-05 -atch 2.0851479089288919e-05 -ollen 2.0847690666489267e-05 -▁Zugriff 2.0847690666489267e-05 -▁Old 2.0843902243689615e-05 -▁Programms 2.0843902243689615e-05 -▁landscape 2.0843902243689615e-05 -akte 2.0840113820889963e-05 -▁kaufen 2.0828748552491003e-05 -▁Report 2.0824960129691351e-05 -▁Hauptstadt 2.0817383284092044e-05 -▁Those 2.0817383284092044e-05 -▁möglichen 2.0817383284092044e-05 -▁species 2.0813594861292392e-05 -▁participants 2.0806018015693087e-05 -▁schneller 2.0806018015693087e-05 -▁damals 2.0802229592893435e-05 -▁acts 2.079844117009378e-05 -▁einzigen 2.0794652747294128e-05 -▁jüngsten 2.0794652747294128e-05 -▁Las 2.0790864324494476e-05 -▁Risiken 2.0790864324494476e-05 -▁bars 2.0787075901694824e-05 -▁contemporary 2.0783287478895172e-05 -▁smaller 2.0783287478895172e-05 -ossen 2.0779499056095516e-05 -▁Manager 2.0779499056095516e-05 -▁Square 2.0779499056095516e-05 -▁Summit 2.0779499056095516e-05 -▁Gegenteil 2.0775710633295864e-05 -▁obvious 2.0775710633295864e-05 -▁unternommen 2.0775710633295864e-05 -▁waiting 2.0775710633295864e-05 -fte 2.0771922210496212e-05 -▁communities 2.0771922210496212e-05 -▁spricht 2.0771922210496212e-05 -▁Objekt 2.0760556942097253e-05 -▁slow 2.07567685192976e-05 -▁Motor 2.0752980096497949e-05 -▁Enjoy 2.0749191673698297e-05 -▁wall 2.0749191673698297e-05 -ickt 2.0745403250898641e-05 -▁Palestinian 2.0745403250898641e-05 -▁print 2.0745403250898641e-05 -▁putting 2.0741614828098989e-05 -▁erfordert 2.0737826405299337e-05 -▁dimension 2.073024955970003e-05 -▁mail 2.073024955970003e-05 -▁jungen 2.0726461136900378e-05 -▁selten 2.0718884291301073e-05 -licherweise 2.0715095868501421e-05 -▁Ep 2.0715095868501421e-05 -▁mechanism 2.0707519022902114e-05 -▁colour 2.0692365331703502e-05 -▁prevention 2.0692365331703502e-05 -▁vorgeschlagenen 2.0692365331703502e-05 -▁Teilen 2.068857690890385e-05 -ends 2.0684788486104198e-05 -vert 2.0681000063304546e-05 -▁Gebiete 2.0677211640504891e-05 -▁Both 2.0673423217705239e-05 -▁Spezial 2.0662057949306283e-05 -15 2.0654481103706975e-05 -▁CS 2.0646904258107671e-05 -▁Ratspräsident 2.0646904258107671e-05 -▁abge 2.0646904258107671e-05 -▁sold 2.0646904258107671e-05 -elf 2.0639327412508364e-05 -ania 2.0635538989708712e-05 -straße 2.0635538989708712e-05 -▁Ze 2.0635538989708712e-05 -grund 2.06203852985101e-05 -ills 2.06203852985101e-05 -▁Arbeitsmarkt 2.0616596875710448e-05 -▁Royal 2.0616596875710448e-05 -▁schlecht 2.0616596875710448e-05 -iertes 2.0612808452910796e-05 -▁disposal 2.0612808452910796e-05 -▁regelmäßig 2.0612808452910796e-05 -▁contained 2.060902003011114e-05 -▁leads 2.0605231607311488e-05 -▁Bi 2.0601443184511836e-05 -▁guaranteed 2.0601443184511836e-05 -agn 2.0597654761712184e-05 -lie 2.0597654761712184e-05 -▁Phase 2.0597654761712184e-05 -▁persönliche 2.0590077916112877e-05 -▁Energy 2.0586289493313225e-05 -▁Männer 2.0574924224914269e-05 -▁sozial 2.0567347379314961e-05 -▁Via 2.0563558956515309e-05 -▁fraud 2.0559770533715657e-05 -▁Grundrechte 2.055219368811635e-05 -▁Sup 2.055219368811635e-05 -▁nationaler 2.055219368811635e-05 -▁übernommen 2.055219368811635e-05 -▁factor 2.0548405265316698e-05 -▁sage 2.0544616842517046e-05 -▁Rules 2.0540828419717393e-05 -eng 2.0533251574118086e-05 -agt 2.0529463151318434e-05 -▁Kat 2.0529463151318434e-05 -▁bringing 2.0529463151318434e-05 -zone 2.0525674728518782e-05 -▁Regierungs 2.0525674728518782e-05 -TE 2.0521886305719126e-05 -▁Politiker 2.0521886305719126e-05 -▁pointed 2.0518097882919474e-05 -▁regul 2.0518097882919474e-05 -▁Island 2.0514309460119822e-05 -▁Su 2.0514309460119822e-05 -▁grounds 2.0514309460119822e-05 -▁planned 2.0514309460119822e-05 -▁liberal 2.051052103732017e-05 -▁Mitgliedern 2.0499155768921211e-05 -▁proceed 2.0499155768921211e-05 -▁blue 2.0495367346121559e-05 -pp 2.0487790500522255e-05 -▁Bestandteil 2.0480213654922947e-05 -▁zus 2.0480213654922947e-05 -▁verstärkt 2.0468848386523988e-05 -hor 2.0465059963724336e-05 -▁Thanks 2.0465059963724336e-05 -▁convenient 2.0465059963724336e-05 -▁teilen 2.0465059963724336e-05 -ione 2.0461271540924684e-05 -fonds 2.0457483118125032e-05 -▁JavaScript 2.0457483118125032e-05 -▁registration 2.044232942692642e-05 -▁acceptable 2.0434752581327113e-05 -hö 2.043096415852746e-05 -▁substantial 2.0427175735727808e-05 -▁800 2.0423387312928156e-05 -▁nachhaltige 2.0423387312928156e-05 -▁password 2.0423387312928156e-05 -▁nearly 2.0415810467328849e-05 -▁dynamic 2.0412022044529197e-05 -oon 2.0408233621729545e-05 -▁inc 2.0408233621729545e-05 -▁prop 2.0408233621729545e-05 -▁Sweden 2.0404445198929893e-05 -▁Typ 2.0400656776130237e-05 -▁Option 2.0396868353330585e-05 -▁ID 2.0389291507731281e-05 -▁band 2.0385503084931629e-05 -ery 2.0377926239332322e-05 -▁resulting 2.037413781653267e-05 -▁Schluss 2.0366560970933366e-05 -▁anbieten 2.036277254813371e-05 -▁operate 2.036277254813371e-05 -▁III 2.0358984125334058e-05 -▁causes 2.0355195702534406e-05 -lungs 2.0351407279734754e-05 -uchen 2.0351407279734754e-05 -▁RO 2.0351407279734754e-05 -▁farm 2.0351407279734754e-05 -bank 2.0343830434135447e-05 -lav 2.0343830434135447e-05 -▁Welche 2.0343830434135447e-05 -management 2.0340042011335794e-05 -▁examples 2.0340042011335794e-05 -änden 2.033246516573649e-05 -▁Klimaanlage 2.033246516573649e-05 -▁brings 2.033246516573649e-05 -iem 2.0328676742936835e-05 -▁manufacturing 2.0324888320137183e-05 -▁zugleich 2.0321099897337531e-05 -inger 2.0317311474537879e-05 -▁Card 2.0317311474537879e-05 -▁Alter 2.0313523051738223e-05 -▁Del 2.0313523051738223e-05 -▁wodurch 2.0313523051738223e-05 -gn 2.0309734628938571e-05 -▁Königreich 2.0305946206138919e-05 -▁pleasure 2.0305946206138919e-05 -▁Pflicht 2.0298369360539615e-05 -alls 2.0290792514940308e-05 -pass 2.0290792514940308e-05 -▁seeking 2.0290792514940308e-05 -führer 2.0279427246541352e-05 -▁responsibilities 2.0279427246541352e-05 --de 2.0275638823741696e-05 -▁Beg 2.0275638823741696e-05 -▁fuel 2.0275638823741696e-05 -gere 2.0271850400942044e-05 -äng 2.0271850400942044e-05 -▁Touristen 2.0271850400942044e-05 -▁conclude 2.0271850400942044e-05 -▁Dort 2.0268061978142392e-05 -▁cheap 2.026427355534274e-05 -▁Teilnahme 2.0256696709743433e-05 -▁Sant 2.025290828694378e-05 -▁letter 2.025290828694378e-05 -ION 2.0249119864144128e-05 -▁betroffenen 2.0249119864144128e-05 -abl 2.0241543018544821e-05 -▁höher 2.0241543018544821e-05 -▁kleiner 2.0241543018544821e-05 -▁praktisch 2.0233966172945517e-05 -▁klein 2.0226389327346209e-05 -eite 2.0222600904546557e-05 -iler 2.0222600904546557e-05 -▁vorschlagen 2.0222600904546557e-05 -▁testing 2.0215024058947253e-05 -▁multiple 2.02112356361476e-05 -▁strengthening 2.0199870367748642e-05 -ausschuss 2.019608194494899e-05 -▁capable 2.0192293522149334e-05 -ope 2.0188505099349682e-05 -р 2.0188505099349682e-05 -▁Experten 2.0188505099349682e-05 -▁Effizienz 2.0180928253750378e-05 -▁identity 2.0180928253750378e-05 -▁Nov 2.0173351408151071e-05 -▁Chamber 2.0169562985351419e-05 -▁Lin 2.0169562985351419e-05 -▁argument 2.0169562985351419e-05 -▁Pat 2.0158197716952462e-05 -▁Prüfung 2.0158197716952462e-05 -▁Maria 2.0154409294152807e-05 -produktion 2.0150620871353155e-05 -arch 2.0146832448553503e-05 -▁beschäftigt 2.0143044025753851e-05 -▁zählen 2.0143044025753851e-05 -64 2.0139255602954195e-05 -fs 2.0139255602954195e-05 -▁Ebenso 2.0135467180154543e-05 -▁Korea 2.0135467180154543e-05 -▁recovery 2.0135467180154543e-05 -▁taste 2.0135467180154543e-05 -▁Alliance 2.0131678757354891e-05 -▁Liberalisierung 2.0131678757354891e-05 -▁zone 2.0124101911755587e-05 -▁Leit 2.011652506615628e-05 -▁gewählt 2.011652506615628e-05 -▁spaces 2.011652506615628e-05 -zul 2.0108948220556976e-05 -▁association 2.010515979775732e-05 -▁traditionellen 2.010515979775732e-05 -▁Where 2.0101371374957668e-05 -ky 2.0097582952158016e-05 -▁governance 2.0097582952158016e-05 -▁Fil 2.0090006106558712e-05 -▁decades 2.0090006106558712e-05 -▁Apartment 2.0086217683759057e-05 -▁Details 2.0082429260959405e-05 -▁Dutch 2.0082429260959405e-05 -▁Messe 2.0078640838159753e-05 -▁zunehmend 2.0078640838159753e-05 -▁secret 2.00748524153601e-05 -▁Schwerpunkt 2.0071063992560448e-05 -▁optimal 2.0067275569760793e-05 -▁steel 2.0067275569760793e-05 -▁Hälfte 2.0063487146961141e-05 -▁Heil 2.0059698724161489e-05 -ho 2.0052121878562181e-05 -▁auszu 2.0052121878562181e-05 -▁weg 2.0052121878562181e-05 -▁Internationalen 2.0044545032962877e-05 -▁River 2.0036968187363573e-05 -▁h. 2.0036968187363573e-05 -▁Bew 2.0025602918964614e-05 -▁sichern 2.0025602918964614e-05 -reichen 2.0021814496164962e-05 -▁21. 2.0021814496164962e-05 -▁School 2.0021814496164962e-05 -▁deutsche 2.0018026073365306e-05 -▁120 2.0014237650565654e-05 -▁Führung 2.0014237650565654e-05 -staat 2.0002872382166698e-05 -▁Tro 1.9995295536567391e-05 -▁forest 1.9995295536567391e-05 -▁southern 1.9995295536567391e-05 -▁adequate 1.9991507113767739e-05 -▁copy 1.9991507113767739e-05 -▁Erhöhung 1.9987718690968087e-05 -▁Original 1.9987718690968087e-05 -▁int 1.9987718690968087e-05 -▁sets 1.9987718690968087e-05 -po 1.9983930268168435e-05 -hrt 1.9976353422569127e-05 -▁AB 1.9976353422569127e-05 -▁master 1.9976353422569127e-05 -mon 1.9964988154170168e-05 -▁Liebe 1.9964988154170168e-05 -ahl 1.9957411308570863e-05 -▁besseren 1.9957411308570863e-05 -▁hergestellt 1.9957411308570863e-05 -▁Morgen 1.9942257617372252e-05 -▁Charter 1.99384691945726e-05 -▁primary 1.99384691945726e-05 -zuf 1.9934680771772948e-05 -▁driving 1.9930892348973292e-05 -▁previously 1.9930892348973292e-05 -ded 1.9923315503373988e-05 -▁Rotary 1.9923315503373988e-05 -▁notice 1.9923315503373988e-05 -▁Two 1.9911950234975029e-05 -▁Inc 1.9908161812175377e-05 -▁books 1.9908161812175377e-05 -▁wichtigste 1.9900584966576073e-05 -▁Trotz 1.9893008120976765e-05 -▁consensus 1.9893008120976765e-05 -▁patients 1.9893008120976765e-05 -▁diskutiert 1.9885431275377461e-05 -▁versions 1.9885431275377461e-05 -▁andererseits 1.9881642852577809e-05 --4 1.9877854429778154e-05 -▁Zimmern 1.9877854429778154e-05 -iment 1.9866489161379197e-05 -▁Java 1.9866489161379197e-05 -▁Norden 1.9866489161379197e-05 -king 1.9862700738579545e-05 -annten 1.9851335470180586e-05 -aufen 1.9843758624581278e-05 -ury 1.9843758624581278e-05 -▁truth 1.9839970201781626e-05 -▁vis 1.9839970201781626e-05 -itten 1.9836181778981974e-05 -▁discussing 1.9836181778981974e-05 -▁primarily 1.9836181778981974e-05 -gar 1.9821028087783363e-05 -park 1.9817239664983711e-05 -▁600 1.9817239664983711e-05 -lock 1.9809662819384403e-05 -▁Nice 1.9809662819384403e-05 -▁sieben 1.9809662819384403e-05 -▁Überprüfung 1.9809662819384403e-05 -▁Koordinierung 1.98020859737851e-05 -▁requested 1.98020859737851e-05 -▁Stein 1.9798297550985447e-05 -▁White 1.9794509128185795e-05 -▁industries 1.9794509128185795e-05 -▁addressed 1.979072070538614e-05 -▁av 1.9783143859786836e-05 -▁perfekt 1.9783143859786836e-05 -icks 1.9779355436987183e-05 -▁Romania 1.9779355436987183e-05 -▁Aufgrund 1.9775567014187531e-05 -▁remove 1.9775567014187531e-05 -▁190 1.9767990168588224e-05 -inal 1.9764201745788572e-05 -▁Eltern 1.9764201745788572e-05 -▁serv 1.9764201745788572e-05 -▁weiteres 1.9764201745788572e-05 -ktion 1.976041332298892e-05 -▁Absicht 1.976041332298892e-05 -▁Financial 1.9756624900189264e-05 -▁fällt 1.9756624900189264e-05 -▁Flash 1.9752836477389612e-05 -▁Laufe 1.974904805458996e-05 -tel 1.9737682786191e-05 -▁expensive 1.9737682786191e-05 -▁invest 1.9737682786191e-05 -▁Methode 1.9726317517792045e-05 -wand 1.9722529094992389e-05 -▁Ausnahme 1.9718740672192737e-05 -▁affect 1.9714952249393085e-05 -▁kosten 1.9714952249393085e-05 -▁cap 1.9711163826593433e-05 -esse 1.9707375403793781e-05 -▁Stadtzentrum 1.9707375403793781e-05 -▁1993 1.9703586980994126e-05 -▁Ton 1.9703586980994126e-05 -▁begann 1.9703586980994126e-05 -▁attacks 1.9692221712595169e-05 -▁Veranstaltungen 1.9688433289795514e-05 -▁beg 1.9688433289795514e-05 -▁liability 1.9684644866995862e-05 -▁comm 1.968085644419621e-05 -▁kürzlich 1.9677068021396558e-05 -iche 1.9673279598596906e-05 -▁kinds 1.9673279598596906e-05 -▁talks 1.9673279598596906e-05 -▁thereby 1.96657027529976e-05 -▁parallel 1.9661914330197946e-05 -elten 1.9658125907398294e-05 -▁schools 1.9658125907398294e-05 -▁Einkaufs 1.9654337484598642e-05 -▁nations 1.9654337484598642e-05 -aire 1.9650549061798987e-05 -▁U.S. 1.9650549061798987e-05 -▁Je 1.9646760638999335e-05 -▁Gästen 1.9642972216199683e-05 -anden 1.9639183793400031e-05 -ural 1.9635395370600375e-05 -▁Inseln 1.9635395370600375e-05 -▁Obama 1.9635395370600375e-05 -▁audio 1.9635395370600375e-05 -▁cont 1.9635395370600375e-05 -▁representative 1.9635395370600375e-05 -bericht 1.9631606947800723e-05 -iti 1.9631606947800723e-05 -mm 1.9631606947800723e-05 -▁Ess 1.9631606947800723e-05 -▁willkommen 1.9631606947800723e-05 -▁eight 1.9624030102201419e-05 -▁seitens 1.9624030102201419e-05 -entwicklung 1.9620241679401767e-05 -▁appeal 1.9620241679401767e-05 -▁NI 1.9616453256602112e-05 -▁phase 1.9616453256602112e-05 -▁Methoden 1.961266483380246e-05 -▁globale 1.961266483380246e-05 -let 1.96012995654035e-05 -▁finde 1.96012995654035e-05 -▁Pflanzen 1.9593722719804196e-05 -▁Law 1.9589934297004544e-05 -ower 1.9582357451405236e-05 -▁lag 1.9578569028605584e-05 -▁USS 1.9574780605805932e-05 -▁Diskriminierung 1.957099218300628e-05 -legt 1.9567203760206628e-05 -▁jemand 1.9567203760206628e-05 -▁HD 1.9559626914607321e-05 -▁orders 1.9555838491807669e-05 -▁einzelne 1.9544473223408709e-05 -▁junge 1.9533107955009753e-05 -▁apart 1.9521742686610794e-05 -▁reagieren 1.9521742686610794e-05 -▁Tar 1.9510377418211834e-05 -▁demokratische 1.9510377418211834e-05 -icken 1.9506588995412182e-05 -▁55 1.9499012149812878e-05 -geht 1.949143530421357e-05 -ound 1.949143530421357e-05 -▁Formular 1.9487646881413918e-05 -▁Mutter 1.9487646881413918e-05 -▁Wel 1.9487646881413918e-05 -▁akzeptiert 1.9487646881413918e-05 -ł 1.9483858458614266e-05 -▁Find 1.9483858458614266e-05 -▁turned 1.9483858458614266e-05 -▁default 1.9480070035814611e-05 -▁Patent 1.9476281613014959e-05 -▁Revolution 1.9461127921816347e-05 -▁opposite 1.9457339499016695e-05 -▁Angriff 1.9453551076217043e-05 -л 1.9449762653417391e-05 -ube 1.9442185807818084e-05 -▁evaluation 1.9442185807818084e-05 -ames 1.943460896221878e-05 -▁Bon 1.9430820539419128e-05 -▁Express 1.9430820539419128e-05 -▁nation 1.9430820539419128e-05 -▁sinnvoll 1.9427032116619472e-05 -ya 1.942324369381982e-05 -▁Anliegen 1.9419455271020168e-05 -▁extreme 1.9419455271020168e-05 -▁moral 1.9419455271020168e-05 -▁Arch 1.9411878425420864e-05 -ervice 1.9408090002621209e-05 -utions 1.9404301579821557e-05 -▁alter 1.9404301579821557e-05 -verkehrs 1.9396724734222252e-05 -▁Institution 1.9396724734222252e-05 -▁Gel 1.9385359465823293e-05 -▁named 1.9373994197424333e-05 -iest 1.9370205774624681e-05 -öt 1.9370205774624681e-05 -▁prepare 1.9366417351825029e-05 -ilt 1.9362628929025377e-05 -▁delivered 1.9362628929025377e-05 -cept 1.9358840506225725e-05 -uld 1.9358840506225725e-05 -▁Ta 1.9358840506225725e-05 -▁Vorbereitung 1.9358840506225725e-05 --free 1.935505208342607e-05 -▁11. 1.9347475237826766e-05 -▁Willen 1.9347475237826766e-05 -▁corner 1.9347475237826766e-05 -part 1.9343686815027114e-05 -ential 1.9339898392227458e-05 -▁Wegen 1.9339898392227458e-05 -▁anerkannt 1.9336109969427806e-05 -▁Tiere 1.9332321546628154e-05 -▁determined 1.9332321546628154e-05 -▁ländlichen 1.9332321546628154e-05 -under 1.932474470102885e-05 -▁Spe 1.932474470102885e-05 -▁reply 1.932474470102885e-05 -▁Fenster 1.9320956278229195e-05 -▁dealt 1.9320956278229195e-05 -ächt 1.9313379432629891e-05 -▁Versch 1.9309591009830238e-05 -▁echte 1.9305802587030583e-05 -▁Doppel 1.9298225741431279e-05 -setzen 1.9294437318631627e-05 -▁Wien 1.9294437318631627e-05 -▁Gesetzgebung 1.9286860473032319e-05 -▁Theater 1.9283072050232667e-05 -▁behandeln 1.9279283627433015e-05 -▁rad 1.9279283627433015e-05 -üssen 1.9275495204633363e-05 -▁angeht 1.9275495204633363e-05 -▁leg 1.9271706781833708e-05 -▁mutual 1.9271706781833708e-05 -▁Trek 1.9267918359034056e-05 -▁waters 1.9267918359034056e-05 -▁freie 1.9264129936234404e-05 -bens 1.9260341513434752e-05 -▁Zustand 1.9260341513434752e-05 -esen 1.92565530906351e-05 -▁Fischer 1.92565530906351e-05 -▁Mos 1.92565530906351e-05 -▁Prodi 1.92565530906351e-05 -▁betont 1.92565530906351e-05 -zi 1.924518782223614e-05 -▁dritten 1.9241399399436488e-05 -▁noise 1.9241399399436488e-05 -▁Anpassung 1.9233822553837181e-05 -▁Ordnung 1.9233822553837181e-05 -▁nochmals 1.9233822553837181e-05 -▁PR 1.9226245708237877e-05 -▁except 1.9226245708237877e-05 -▁snow 1.9226245708237877e-05 -EP 1.9222457285438224e-05 -ayer 1.9222457285438224e-05 -ía 1.9222457285438224e-05 -▁Freizeit 1.9222457285438224e-05 -▁grand 1.9222457285438224e-05 -▁Fähigkeit 1.9218668862638569e-05 -▁ren 1.9218668862638569e-05 -bert 1.9214880439838917e-05 -ways 1.9207303594239613e-05 -▁positiven 1.9207303594239613e-05 -▁manufacturers 1.9203515171439961e-05 -▁Despite 1.9192149903041e-05 -▁amongst 1.9192149903041e-05 -▁diversity 1.9188361480241349e-05 -▁roll 1.9188361480241349e-05 -bon 1.9180784634642042e-05 -▁berühmten 1.9180784634642042e-05 -▁Jan 1.9173207789042738e-05 -▁WC 1.9173207789042738e-05 -▁hall 1.9173207789042738e-05 -▁Stunde 1.916563094344343e-05 -▁umgeben 1.916563094344343e-05 -amer 1.9158054097844126e-05 -nissen 1.9158054097844126e-05 -25 1.9154265675044474e-05 -▁Church 1.9154265675044474e-05 -▁News 1.9154265675044474e-05 -▁onto 1.9154265675044474e-05 -▁Minutes 1.9150477252244822e-05 -▁sicherstellen 1.9150477252244822e-05 -▁well- 1.9150477252244822e-05 -▁Architektur 1.9146688829445167e-05 -▁qualified 1.9146688829445167e-05 -SE 1.9142900406645515e-05 -▁bewegen 1.9139111983845863e-05 -▁führenden 1.9135323561046211e-05 -▁Verhältnis 1.9131535138246555e-05 -▁Ca 1.9127746715446903e-05 -▁anti 1.9127746715446903e-05 -▁begonnen 1.9127746715446903e-05 -eration 1.9123958292647251e-05 -▁User 1.9116381447047947e-05 -▁skin 1.909743933304968e-05 --Kon 1.9086074064650724e-05 -▁Out 1.9086074064650724e-05 -▁Klimawandel 1.9082285641851072e-05 -▁Export 1.9078497219051416e-05 -▁Sein 1.9074708796251764e-05 -qualität 1.9070920373452112e-05 -▁diverse 1.9070920373452112e-05 -▁Dar 1.906713195065246e-05 -▁Wohl 1.906713195065246e-05 -▁urge 1.906713195065246e-05 -▁accounts 1.9063343527852808e-05 -▁leadership 1.9059555105053153e-05 -▁charming 1.9051978259453849e-05 -▁193 1.9048189836654197e-05 -▁Reaktion 1.9040612991054889e-05 -▁investments 1.9040612991054889e-05 -ibility 1.9036824568255237e-05 -▁Unterschiede 1.9036824568255237e-05 -ubl 1.9033036145455585e-05 -▁Freunde 1.9029247722655933e-05 -▁feeling 1.9029247722655933e-05 -bereiche 1.9025459299856278e-05 -raft 1.9025459299856278e-05 -▁father 1.9025459299856278e-05 -station 1.9017882454256973e-05 -▁Umständen 1.9017882454256973e-05 -▁Fälle 1.9010305608657666e-05 -▁Minderheiten 1.9010305608657666e-05 -▁Englisch 1.9006517185858014e-05 -▁Wachstums 1.9002728763058362e-05 -▁Pet 1.899894034025871e-05 -scher 1.8995151917459058e-05 -uts 1.8991363494659402e-05 -▁39 1.8991363494659402e-05 -▁AD 1.8991363494659402e-05 -house 1.8979998226260446e-05 -99 1.8976209803460791e-05 -▁Verfügbarkeit 1.8968632957861487e-05 -PS 1.8957267689462527e-05 -▁verlieren 1.8957267689462527e-05 -eb 1.8953479266662875e-05 -strategie 1.8938325575464264e-05 -▁\ 1.8938325575464264e-05 -▁Einklang 1.8934537152664612e-05 -▁Munich 1.8934537152664612e-05 -▁Time 1.8934537152664612e-05 -barer 1.8930748729864959e-05 -▁technological 1.8930748729864959e-05 -▁Master 1.8926960307065307e-05 -▁fand 1.8926960307065307e-05 -▁confirm 1.8923171884265652e-05 -▁Well 1.8919383461466e-05 -▁consultation 1.8919383461466e-05 -▁leiden 1.8919383461466e-05 -▁marked 1.8919383461466e-05 -▁moved 1.8919383461466e-05 -▁mechanisms 1.8915595038666348e-05 -iell 1.8911806615866696e-05 -▁Before 1.8911806615866696e-05 -ando 1.8892864501868432e-05 -▁Stück 1.8892864501868432e-05 -▁Super 1.8885287656269125e-05 -▁Take 1.8881499233469473e-05 -▁cm 1.8881499233469473e-05 -▁effizient 1.8881499233469473e-05 -▁1000 1.8877710810669821e-05 -chten 1.8873922387870169e-05 -halts 1.8873922387870169e-05 -▁burden 1.8873922387870169e-05 -▁sex 1.8873922387870169e-05 -▁pension 1.8870133965070513e-05 -▁Adresse 1.8858768696671557e-05 -▁runs 1.885119185107225e-05 -iant 1.8847403428272598e-05 -▁recommended 1.8847403428272598e-05 -▁Sound 1.8843615005472946e-05 -▁fordert 1.8843615005472946e-05 -▁interpret 1.8839826582673293e-05 -▁vielmehr 1.8839826582673293e-05 -▁heavy 1.8836038159873638e-05 -▁starken 1.8836038159873638e-05 -▁überzeugen 1.8836038159873638e-05 -zus 1.8832249737073986e-05 -igsten 1.8828461314274334e-05 -ú 1.8828461314274334e-05 -▁König 1.8828461314274334e-05 -ê 1.8817096045875374e-05 -fest 1.8813307623075722e-05 -▁Dadurch 1.880951920027607e-05 -▁bedarf 1.880951920027607e-05 -▁Schloss 1.8801942354676763e-05 -▁Freude 1.8798153931877111e-05 -▁Log 1.8790577086277807e-05 -▁ehemaligen 1.8790577086277807e-05 -▁zustimmen 1.8790577086277807e-05 -arian 1.8786788663478155e-05 -▁München 1.87830002406785e-05 -▁Zentral 1.87830002406785e-05 -▁wait 1.87830002406785e-05 -▁COM 1.8779211817878847e-05 -reis 1.8775423395079195e-05 -▁Kredit 1.8775423395079195e-05 -▁illustr 1.8775423395079195e-05 -▁Diskussionen 1.8771634972279543e-05 -▁mode 1.8771634972279543e-05 -▁Grenze 1.8767846549479888e-05 -▁knows 1.8767846549479888e-05 -▁improvements 1.8764058126680236e-05 -▁Hans 1.8760269703880584e-05 -▁attitude 1.8760269703880584e-05 -▁expl 1.8756481281080932e-05 -▁fail 1.8756481281080932e-05 -wort 1.8752692858281279e-05 -reiten 1.8748904435481624e-05 -▁Kollege 1.8748904435481624e-05 -▁Fra 1.874132758988232e-05 -▁switch 1.874132758988232e-05 -▁helping 1.8737539167082668e-05 -▁significantly 1.8737539167082668e-05 -aker 1.8733750744283016e-05 -▁Kombination 1.8733750744283016e-05 -EA 1.872996232148336e-05 -▁derzeitigen 1.872996232148336e-05 -▁privacy 1.872996232148336e-05 -ody 1.8726173898683708e-05 -ident 1.8722385475884056e-05 -▁Bedenken 1.8722385475884056e-05 -rei 1.8718597053084404e-05 -▁experiences 1.8718597053084404e-05 -▁friend 1.8718597053084404e-05 -▁Handeln 1.8714808630284749e-05 -anischen 1.8707231784685445e-05 -lt 1.8707231784685445e-05 -▁Play 1.8707231784685445e-05 -▁thr 1.8703443361885793e-05 -▁vorgenommen 1.8699654939086141e-05 -▁ergriffen 1.8695866516286485e-05 -OP 1.8692078093486833e-05 -▁Unterschied 1.8692078093486833e-05 -▁error 1.8692078093486833e-05 -▁nötig 1.8692078093486833e-05 -▁Kunde 1.8688289670687181e-05 -▁react 1.8688289670687181e-05 -iziert 1.8676924402288222e-05 -▁Verhalten 1.8676924402288222e-05 -▁43 1.867313597948857e-05 -iding 1.8665559133889266e-05 -▁befassen 1.8654193865490306e-05 -wa 1.8646617019891002e-05 -▁Angebote 1.8646617019891002e-05 -▁Berücksichtigung 1.8642828597091346e-05 -▁blood 1.8642828597091346e-05 -▁output 1.8642828597091346e-05 -▁politisch 1.8642828597091346e-05 -▁wiederum 1.8642828597091346e-05 -▁requests 1.8635251751492042e-05 -▁compliance 1.863146332869239e-05 -▁expertise 1.863146332869239e-05 -▁prot 1.8623886483093083e-05 -▁bildet 1.8620098060293431e-05 -▁gewisse 1.8620098060293431e-05 -▁cold 1.8616309637493779e-05 -▁shape 1.8616309637493779e-05 -▁competence 1.8612521214694127e-05 -▁regret 1.8612521214694127e-05 -▁Set 1.8608732791894471e-05 -▁bedrooms 1.8604944369094819e-05 -▁seien 1.8604944369094819e-05 -14 1.8597367523495515e-05 -▁presentation 1.8597367523495515e-05 -heits 1.859357910069586e-05 -ict 1.859357910069586e-05 -▁Gesetze 1.8586002255096556e-05 -▁keiner 1.8586002255096556e-05 -▁Menü 1.8578425409497252e-05 -▁exc 1.8578425409497252e-05 -▁feed 1.8578425409497252e-05 -▁streets 1.8578425409497252e-05 -▁läuft 1.8567060141098292e-05 -▁existence 1.856327171829864e-05 -▁höchste 1.856327171829864e-05 -▁speakers 1.856327171829864e-05 -▁Zahlungs 1.8559483295498985e-05 -▁Pan 1.855190644989968e-05 -▁treated 1.855190644989968e-05 -ital 1.8548118027100028e-05 -▁Eigenschaften 1.8540541181500721e-05 -▁Use 1.8540541181500721e-05 -▁lock 1.8540541181500721e-05 -▁95 1.8536752758701069e-05 -▁Vielleicht 1.8536752758701069e-05 -▁Prof 1.8532964335901417e-05 -▁Stat 1.8532964335901417e-05 -▁connect 1.8532964335901417e-05 -▁flat 1.8532964335901417e-05 -stoffen 1.8525387490302113e-05 -▁binding 1.8521599067502457e-05 -▁schließen 1.8517810644702805e-05 -▁Skype 1.8514022221903153e-05 -▁welchen 1.8506445376303846e-05 -ße 1.8502656953504194e-05 -ст 1.8502656953504194e-05 -angs 1.849508010790489e-05 -gegen 1.849508010790489e-05 -▁David 1.849508010790489e-05 -▁Antworten 1.8491291685105238e-05 -▁piece 1.8491291685105238e-05 -▁NL 1.8487503262305582e-05 -▁destination 1.8487503262305582e-05 -pression 1.848371483950593e-05 -▁websites 1.848371483950593e-05 -▁cal 1.8472349571106971e-05 -▁furniture 1.8468561148307319e-05 -LA 1.8464772725507667e-05 -▁Hy 1.8464772725507667e-05 -▁grow 1.8464772725507667e-05 -▁Op 1.8460984302708014e-05 -▁expectations 1.8460984302708014e-05 -▁konkreten 1.8457195879908362e-05 -jahr 1.8449619034309055e-05 -▁Fahrzeug 1.8449619034309055e-05 -▁Voraussetzung 1.8449619034309055e-05 -▁consistent 1.8449619034309055e-05 -▁launched 1.8449619034309055e-05 -▁Amerika 1.8445830611509403e-05 -▁drugs 1.8445830611509403e-05 -▁abhängig 1.8434465343110443e-05 -ward 1.8426888497511139e-05 -▁felt 1.8426888497511139e-05 -ienst 1.8423100074711487e-05 -▁sanctions 1.8419311651911832e-05 -▁Cont 1.841552322911218e-05 -▁Comm 1.8411734806312528e-05 -▁wild 1.8404157960713224e-05 -zial 1.8400369537913568e-05 -▁Radio 1.8400369537913568e-05 -▁eye 1.8400369537913568e-05 -▁betreffen 1.8396581115113916e-05 -▁dishes 1.8385215846714957e-05 -▁equ 1.8381427423915305e-05 -▁sand 1.8381427423915305e-05 -hy 1.8377639001115653e-05 -▁units 1.8377639001115653e-05 -una 1.8373850578316e-05 -book 1.8370062155516348e-05 -▁lose 1.8370062155516348e-05 -▁recognise 1.8370062155516348e-05 -▁Sel 1.8366273732716693e-05 -▁Airbus 1.8362485309917041e-05 -arkt 1.8358696887117389e-05 -einander 1.8343543195918777e-05 -▁browser 1.8335966350319473e-05 -▁Dokumente 1.8328389504720166e-05 -▁p.m. 1.8328389504720166e-05 -entwurf 1.8324601081920514e-05 -▁basiert 1.8324601081920514e-05 -▁compensation 1.8324601081920514e-05 -▁occur 1.8324601081920514e-05 -▁Allgemeinen 1.8320812659120862e-05 -▁Gewinn 1.8320812659120862e-05 -▁adapt 1.8320812659120862e-05 -▁obligation 1.8320812659120862e-05 -▁ser 1.8320812659120862e-05 -altung 1.831702423632121e-05 -▁conf 1.831702423632121e-05 -▁intention 1.831702423632121e-05 -isc 1.8313235813521554e-05 -UN 1.8309447390721902e-05 -▁Untersuchung 1.8309447390721902e-05 -▁genauso 1.8309447390721902e-05 -▁angemessen 1.830565896792225e-05 -preise 1.8301870545122598e-05 -▁Schäden 1.8301870545122598e-05 -karten 1.8294293699523291e-05 -▁Hinter 1.8286716853923987e-05 -▁Wirtschafts- 1.8286716853923987e-05 -oe 1.8282928431124334e-05 -▁entlang 1.8282928431124334e-05 -annte 1.8275351585525027e-05 -ati 1.8275351585525027e-05 -front 1.8275351585525027e-05 -▁Technology 1.8275351585525027e-05 -▁Lauf 1.8271563162725375e-05 -äd 1.8267774739925723e-05 -▁Plaza 1.8267774739925723e-05 -▁18. 1.8248832625927459e-05 -▁ersch 1.8248832625927459e-05 -▁fantastic 1.8248832625927459e-05 -werten 1.8245044203127804e-05 -▁relatively 1.8245044203127804e-05 -▁Kan 1.8241255780328152e-05 -▁rejected 1.8241255780328152e-05 -▁soft 1.8241255780328152e-05 -▁fit 1.82374673575285e-05 -▁represented 1.82374673575285e-05 -öst 1.8233678934728848e-05 -▁wirksam 1.8229890511929196e-05 -▁bedroom 1.822610208912954e-05 -▁constant 1.8218525243530236e-05 -▁extension 1.8218525243530236e-05 -▁Grundsätze 1.8214736820730584e-05 -▁Strukturfonds 1.8214736820730584e-05 -▁emphasis 1.8214736820730584e-05 -reiben 1.8210948397930929e-05 -ücken 1.8207159975131277e-05 -▁older 1.8207159975131277e-05 -organisation 1.8203371552331625e-05 -▁deiner 1.8203371552331625e-05 -▁fo 1.8203371552331625e-05 -▁gestern 1.8203371552331625e-05 -▁Leistungs 1.8199583129531973e-05 -▁vert 1.8199583129531973e-05 -▁6. 1.8192006283932665e-05 -▁Westen 1.8192006283932665e-05 -ungsverfahren 1.8184429438333361e-05 -▁exists 1.8184429438333361e-05 --out 1.8180641015533709e-05 -ssystem 1.8180641015533709e-05 -▁Datenbank 1.8180641015533709e-05 -▁Kooperation 1.8180641015533709e-05 -▁Kor 1.8176852592734054e-05 -▁Welcome 1.8176852592734054e-05 -▁Signal 1.81730641699344e-05 -▁Zusatz 1.81730641699344e-05 -▁helps 1.81730641699344e-05 -ausschusses 1.8169275747134749e-05 -iana 1.8169275747134749e-05 --Pl 1.8165487324335097e-05 -▁pack 1.8165487324335097e-05 -▁trotzdem 1.815791047873579e-05 -UM 1.8154122055936138e-05 -▁verz 1.8154122055936138e-05 -ray 1.8150333633136486e-05 -▁Dec 1.8146545210336834e-05 -▁proud 1.8146545210336834e-05 -▁Hin 1.8142756787537182e-05 -▁volle 1.8142756787537182e-05 -äumen 1.8138968364737526e-05 -elled 1.8131391519138222e-05 -▁65 1.8131391519138222e-05 -▁appreciate 1.8131391519138222e-05 -▁trees 1.812760309633857e-05 -▁articles 1.8123814673538915e-05 -13 1.8120026250739263e-05 -änd 1.8116237827939611e-05 -▁stations 1.8116237827939611e-05 -atische 1.8108660982340307e-05 -▁Element 1.8108660982340307e-05 -▁bezieht 1.8104872559540651e-05 -▁comply 1.8104872559540651e-05 -▁reicht 1.8101084136741e-05 -raf 1.8097295713941347e-05 -▁Dublin 1.8097295713941347e-05 -wn 1.8093507291141695e-05 -▁Acc 1.808971886834204e-05 -▁unmittelbar 1.808971886834204e-05 -▁Definition 1.8085930445542388e-05 -▁keeping 1.8085930445542388e-05 -▁wichtiges 1.8085930445542388e-05 -lichkeiten 1.8082142022742735e-05 -▁resolve 1.8082142022742735e-05 -▁senden 1.8082142022742735e-05 -▁Paper 1.8078353599943083e-05 -▁Tom 1.8078353599943083e-05 -▁King 1.8074565177143431e-05 -▁MySQL 1.8074565177143431e-05 -▁substances 1.8074565177143431e-05 -▁97 1.8070776754343776e-05 -▁Action 1.8063199908744472e-05 -▁Schulden 1.8063199908744472e-05 -▁sprach 1.8063199908744472e-05 -▁arms 1.805941148594482e-05 -mut 1.8044257794746208e-05 -zz 1.8044257794746208e-05 -▁objects 1.8044257794746208e-05 -lag 1.8040469371946556e-05 -oft 1.8040469371946556e-05 -▁44 1.8040469371946556e-05 -▁verlangen 1.8040469371946556e-05 -▁operators 1.8029104103547597e-05 -▁ziemlich 1.8029104103547597e-05 -▁Bes 1.8017738835148637e-05 -▁dinner 1.8017738835148637e-05 -ograph 1.8013950412348985e-05 -▁Aut 1.8013950412348985e-05 -▁minim 1.8006373566749681e-05 -▁hoher 1.8002585143950026e-05 -▁Ges 1.7991219875551069e-05 -▁engineering 1.7991219875551069e-05 -▁entered 1.7991219875551069e-05 -▁umfassenden 1.7987431452751417e-05 -immer 1.7983643029951762e-05 -▁Bef 1.7983643029951762e-05 -▁liefern 1.7983643029951762e-05 -oto 1.7976066184352458e-05 -▁Beifall 1.7976066184352458e-05 -▁award 1.7972277761552806e-05 -▁taxes 1.7972277761552806e-05 -▁emphasise 1.79647009159535e-05 -produkt 1.7960912493153846e-05 -▁vast 1.7960912493153846e-05 -▁Spiele 1.7957124070354194e-05 -▁erhältlich 1.7953335647554542e-05 -▁Austausch 1.7949547224754887e-05 -▁Player 1.7949547224754887e-05 -▁Sanktionen 1.7949547224754887e-05 -▁tests 1.7949547224754887e-05 -▁Est 1.7945758801955235e-05 -▁advantages 1.7945758801955235e-05 -▁wenden 1.7945758801955235e-05 -▁Dach 1.7938181956355931e-05 -cer 1.7934393533556279e-05 -▁Mand 1.7934393533556279e-05 -▁regionale 1.7926816687956971e-05 -etzen 1.7923028265157319e-05 -▁führte 1.7919239842357667e-05 -▁Weiter 1.791166299675836e-05 -▁temperature 1.791166299675836e-05 -▁Leitlinien 1.7907874573958708e-05 -▁whatever 1.7907874573958708e-05 -▁refugees 1.7904086151159056e-05 -▁reichen 1.7900297728359403e-05 -▁Küste 1.7892720882760096e-05 -▁Verbrechen 1.7892720882760096e-05 -▁verändert 1.7888932459960444e-05 -▁arm 1.7885144037160792e-05 -▁Bewegung 1.7881355614361136e-05 -flug 1.7877567191561484e-05 -▁approval 1.7877567191561484e-05 -▁fill 1.7873778768761832e-05 -▁Tur 1.786999034596218e-05 -▁Wahrheit 1.7854836654763569e-05 -▁bekannten 1.7851048231963917e-05 -▁Aktion 1.7847259809164261e-05 -▁centres 1.7847259809164261e-05 -leistungen 1.7843471386364609e-05 -▁Safety 1.7843471386364609e-05 -▁Work 1.7835894540765305e-05 -▁Krisen 1.7832106117965653e-05 -▁willing 1.7832106117965653e-05 -olf 1.7828317695165998e-05 -▁Ministers 1.7828317695165998e-05 -▁Rule 1.7824529272366346e-05 -▁sofern 1.7820740849566694e-05 -ais 1.7816952426767042e-05 -▁USD 1.7813164003967389e-05 -▁erweitert 1.7813164003967389e-05 -▁whilst 1.7813164003967389e-05 -▁geschrieben 1.7809375581167734e-05 -erein 1.7805587158368082e-05 -▁delegation 1.7805587158368082e-05 -▁har 1.7805587158368082e-05 --Fraktion 1.780179873556843e-05 -▁historische 1.780179873556843e-05 -▁Maßnahme 1.7798010312768778e-05 -inden 1.7794221889969122e-05 -▁applicable 1.7794221889969122e-05 -▁affairs 1.779043346716947e-05 -▁latter 1.7786645044369818e-05 -▁texts 1.7786645044369818e-05 -▁höhere 1.7782856621570166e-05 -▁Organ 1.7779068198770514e-05 -▁Gener 1.7775279775970859e-05 -▁cross-border 1.7767702930371555e-05 -▁happens 1.7767702930371555e-05 -19 1.7763914507571903e-05 -▁falsch 1.7763914507571903e-05 -▁GNU 1.7760126084772247e-05 -▁Landwirte 1.7760126084772247e-05 -▁Sen 1.7760126084772247e-05 -trag 1.7756337661972595e-05 -uh 1.7752549239172943e-05 -▁entertainment 1.7748760816373291e-05 -uses 1.7744972393573639e-05 -linien 1.7741183970773984e-05 -▁Monte 1.7741183970773984e-05 -▁arrangements 1.7741183970773984e-05 -▁helped 1.7741183970773984e-05 -▁naturally 1.7741183970773984e-05 -cent 1.7737395547974332e-05 -ancy 1.7726030279575376e-05 -▁Umweltschutz 1.7726030279575376e-05 -▁match 1.7726030279575376e-05 -▁personally 1.7726030279575376e-05 -75 1.772224185677572e-05 -inos 1.772224185677572e-05 -▁involving 1.772224185677572e-05 -▁italienischen 1.7718453433976068e-05 -▁util 1.7718453433976068e-05 -stehen 1.7707088165577109e-05 -▁politicians 1.7707088165577109e-05 -jo 1.7703299742777456e-05 -opp 1.7703299742777456e-05 -ously 1.7703299742777456e-05 -▁SO 1.7703299742777456e-05 -politischen 1.7699511319977804e-05 -▁defend 1.7695722897178152e-05 -▁respective 1.7695722897178152e-05 -▁finding 1.7688146051578845e-05 -▁Strategien 1.7684357628779193e-05 -stellt 1.7680569205979541e-05 -▁spring 1.7680569205979541e-05 -▁Akt 1.7676780783179889e-05 -▁Lag 1.7676780783179889e-05 -▁beträgt 1.7676780783179889e-05 -▁hair 1.7676780783179889e-05 -uter 1.7672992360380233e-05 -▁disaster 1.7672992360380233e-05 -▁dining 1.7665415514780929e-05 -▁eröffnet 1.7665415514780929e-05 -▁Show 1.7661627091981277e-05 -▁ern 1.7661627091981277e-05 -▁si 1.7657838669181625e-05 -▁valuable 1.7657838669181625e-05 -▁Verträge 1.765405024638197e-05 -▁danger 1.765405024638197e-05 -▁tours 1.765405024638197e-05 -itation 1.7650261823582318e-05 -▁Beruf 1.7650261823582318e-05 -▁ans 1.7650261823582318e-05 -▁Essen 1.7642684977983014e-05 -▁outdoor 1.7642684977983014e-05 -arte 1.7638896555183358e-05 -uelle 1.7638896555183358e-05 -▁verk 1.7638896555183358e-05 -▁Ferner 1.7635108132383706e-05 -▁Schule 1.7635108132383706e-05 -▁ann 1.7635108132383706e-05 -▁pat 1.7635108132383706e-05 -▁intend 1.7631319709584054e-05 -▁Dat 1.7627531286784402e-05 -▁exclusively 1.7627531286784402e-05 -▁peoples 1.7627531286784402e-05 -▁AN 1.762374286398475e-05 -utt 1.7619954441185095e-05 -▁Mut 1.7619954441185095e-05 -▁Strategy 1.7619954441185095e-05 -▁covers 1.7619954441185095e-05 -▁Besitz 1.7616166018385443e-05 -▁recognised 1.7608589172786138e-05 -▁museum 1.7604800749986486e-05 -onds 1.7601012327186831e-05 -▁49 1.7601012327186831e-05 -▁Agrar 1.7601012327186831e-05 -▁pain 1.7601012327186831e-05 -▁suffering 1.7601012327186831e-05 -da 1.7593435481587527e-05 -▁teilnehmen 1.7593435481587527e-05 -tend 1.7585858635988219e-05 -▁1991 1.7585858635988219e-05 -▁stone 1.7585858635988219e-05 -oun 1.7582070213188567e-05 -▁od 1.7582070213188567e-05 -punkte 1.7574493367589263e-05 -▁hostels 1.7570704944789611e-05 -▁journey 1.7566916521989956e-05 -istisch 1.7559339676390652e-05 -▁zusätzlichen 1.7559339676390652e-05 -agon 1.7555551253591e-05 -▁Swedish 1.7555551253591e-05 -▁Empfehlung 1.7551762830791344e-05 -olo 1.754418598519204e-05 -▁spot 1.754418598519204e-05 -▁1989 1.7540397562392388e-05 -▁diseases 1.7540397562392388e-05 -▁leisure 1.7540397562392388e-05 -▁self- 1.7540397562392388e-05 -▁Studenten 1.7532820716793081e-05 -▁protest 1.7532820716793081e-05 -▁charm 1.7525243871193777e-05 -▁shift 1.7525243871193777e-05 -behörden 1.7521455448394124e-05 -ook 1.7521455448394124e-05 -▁Führungs 1.7521455448394124e-05 -▁Suite 1.7521455448394124e-05 -▁entdecken 1.7521455448394124e-05 -usion 1.7517667025594472e-05 -▁weltweiten 1.7517667025594472e-05 -position 1.7513878602794817e-05 -▁künftig 1.7513878602794817e-05 -wechsel 1.7510090179995165e-05 -▁AL 1.7510090179995165e-05 -unde 1.7506301757195513e-05 -aste 1.7498724911596205e-05 -▁exceptional 1.7498724911596205e-05 -fällen 1.7494936488796553e-05 -▁creative 1.7487359643197249e-05 -oli 1.7483571220397597e-05 -▁Kenn 1.7483571220397597e-05 -▁solve 1.7483571220397597e-05 -▁erscheinen 1.747599437479829e-05 -▁mobility 1.747599437479829e-05 -▁Freund 1.7472205951998638e-05 -▁Materialien 1.746462910639933e-05 -▁Update 1.746462910639933e-05 -▁gefördert 1.7460840683599678e-05 -▁99 1.7453263838000374e-05 -▁Schweden 1.7449475415200722e-05 -▁opposed 1.7449475415200722e-05 --year 1.7438110146801763e-05 -urm 1.7438110146801763e-05 -▁Abs 1.7438110146801763e-05 -▁Badezimmer 1.7438110146801763e-05 -▁Delegation 1.7438110146801763e-05 -▁weather 1.7438110146801763e-05 -eta 1.7430533301202455e-05 -▁OF 1.7430533301202455e-05 -18 1.7415379610003847e-05 -▁kurzem 1.7415379610003847e-05 -▁gering 1.7411591187204191e-05 -▁var 1.7411591187204191e-05 -▁anti- 1.7407802764404539e-05 -äs 1.7400225918805235e-05 -▁fulfil 1.7400225918805235e-05 -eless 1.7396437496005583e-05 -reifen 1.7396437496005583e-05 -▁zugänglich 1.7396437496005583e-05 -▁confirmed 1.7392649073205928e-05 -▁intellectual 1.7385072227606624e-05 -▁Music 1.7381283804806972e-05 -▁Ambiente 1.7377495382007316e-05 -ón 1.7373706959207664e-05 -▁Az 1.7369918536408012e-05 -▁Erwartungen 1.7369918536408012e-05 -▁cas 1.7369918536408012e-05 -plätze 1.736613011360836e-05 -reise 1.7362341690808708e-05 -weit 1.7362341690808708e-05 -▁familiar 1.7362341690808708e-05 -▁Britain 1.7358553268009053e-05 -▁Kann 1.7358553268009053e-05 -▁renewable 1.73547648452094e-05 -netz 1.7350976422409749e-05 -î 1.7350976422409749e-05 -▁schönsten 1.7350976422409749e-05 -▁Access 1.7347187999610097e-05 -▁interessiert 1.7347187999610097e-05 -▁grundlegenden 1.7343399576810441e-05 -▁discovered 1.7339611154010789e-05 -▁looks 1.7339611154010789e-05 -▁Mittelpunkt 1.7335822731211137e-05 -▁voller 1.7335822731211137e-05 -ema 1.7332034308411485e-05 -ito 1.7332034308411485e-05 -▁Ohne 1.7332034308411485e-05 -aur 1.7320669040012525e-05 -äss 1.7316880617212873e-05 -▁Anfragen 1.7316880617212873e-05 -▁schwierigen 1.7316880617212873e-05 -olog 1.7313092194413221e-05 -▁Umgang 1.7313092194413221e-05 -▁Nachfrage 1.7309303771613569e-05 -▁besitzen 1.7309303771613569e-05 -▁öffnen 1.7309303771613569e-05 -▁laufen 1.7305515348813914e-05 -▁sn 1.7305515348813914e-05 -▁Eisenbahn 1.7301726926014262e-05 -▁hilft 1.7301726926014262e-05 -ieben 1.729793850321461e-05 -▁Lehr 1.729793850321461e-05 -▁eing 1.7294150080414958e-05 -▁jährlich 1.7294150080414958e-05 -▁wesentliche 1.7294150080414958e-05 -▁eigenes 1.7290361657615302e-05 -▁Studie 1.7282784812015998e-05 -esp 1.7275207966416694e-05 -▁dans 1.7275207966416694e-05 -▁obtained 1.7267631120817387e-05 -▁Kategorie 1.7256265852418427e-05 -▁Spr 1.7244900584019471e-05 -▁offices 1.7244900584019471e-05 -▁sum 1.7244900584019471e-05 -▁entstanden 1.7241112161219819e-05 -ric 1.7237323738420164e-05 -ž 1.7237323738420164e-05 -▁repeat 1.7237323738420164e-05 -▁Robert 1.7233535315620511e-05 --20 1.7229746892820859e-05 -ysteme 1.7225958470021207e-05 -▁amb 1.7225958470021207e-05 -▁faced 1.7225958470021207e-05 -grad 1.7222170047221555e-05 -▁combating 1.7222170047221555e-05 -▁Ausgangspunkt 1.72183816244219e-05 -▁Beweis 1.72183816244219e-05 -▁bedeuten 1.7210804778822596e-05 -▁listen 1.7210804778822596e-05 -▁Rock 1.7207016356022944e-05 -▁GDP 1.7203227933223288e-05 -▁directives 1.7203227933223288e-05 -▁passengers 1.7203227933223288e-05 -▁Director 1.7199439510423636e-05 -▁hervorragende 1.7199439510423636e-05 -▁Papier 1.7195651087623984e-05 -▁außen 1.7191862664824332e-05 -▁zuerst 1.7191862664824332e-05 -In 1.718807424202468e-05 -▁zweifellos 1.7184285819225025e-05 -▁Wirklichkeit 1.7180497396425373e-05 -▁largely 1.7180497396425373e-05 -gem 1.7176708973625721e-05 -▁residence 1.7172920550826069e-05 -mod 1.7169132128026413e-05 -▁7. 1.7169132128026413e-05 -▁automatic 1.7169132128026413e-05 --5 1.7161555282427109e-05 --n 1.7161555282427109e-05 -▁determine 1.7161555282427109e-05 -▁Mart 1.7157766859627457e-05 -▁prec 1.7157766859627457e-05 -▁IM 1.715019001402815e-05 -▁niemals 1.715019001402815e-05 -▁millions 1.7146401591228498e-05 -▁verf 1.7146401591228498e-05 -ani 1.7142613168428845e-05 -itive 1.7142613168428845e-05 -esser 1.7138824745629193e-05 -▁AT 1.7138824745629193e-05 -▁ausgezeichnete 1.7138824745629193e-05 -▁constructive 1.7138824745629193e-05 -▁Blut 1.7131247900029886e-05 -▁temporary 1.7131247900029886e-05 -▁08 1.7127459477230234e-05 -▁Oper 1.7127459477230234e-05 -▁Rand 1.7127459477230234e-05 -▁launch 1.7127459477230234e-05 -▁serves 1.7127459477230234e-05 -ohn 1.7123671054430582e-05 -▁fix 1.7123671054430582e-05 -▁impressive 1.7123671054430582e-05 -▁Nachrichten 1.711988263163093e-05 -▁inzwischen 1.711988263163093e-05 -▁weshalb 1.7116094208831274e-05 -к 1.7112305786031622e-05 -▁figure 1.710851736323197e-05 -urch 1.7100940517632666e-05 -▁Beratung 1.7097152094833011e-05 -▁protecting 1.7097152094833011e-05 -▁flow 1.7093363672033359e-05 -▁involvement 1.7093363672033359e-05 -▁interessant 1.7089575249233707e-05 -▁natürlichen 1.7089575249233707e-05 -MS 1.7085786826434055e-05 -gung 1.7085786826434055e-05 -▁occasion 1.70819984036344e-05 -▁Son 1.7074421558035095e-05 -erin 1.7070633135235443e-05 -▁Garden 1.7070633135235443e-05 -▁Agrarpolitik 1.7066844712435791e-05 -▁neuesten 1.7059267866836484e-05 -ually 1.7055479444036832e-05 --0 1.7051691021237179e-05 -gan 1.7051691021237179e-05 -▁Burg 1.7051691021237179e-05 -▁Kandidaten 1.7044114175637872e-05 -▁au 1.7044114175637872e-05 -▁drink 1.7021383638839956e-05 -▁OS 1.7017595216040304e-05 -▁lovely 1.7017595216040304e-05 -▁schreiben 1.7017595216040304e-05 -▁Entwicklungsländern 1.7013806793240652e-05 -▁removed 1.7010018370440997e-05 -▁couple 1.7006229947641345e-05 -▁eyes 1.7006229947641345e-05 -▁increases 1.7006229947641345e-05 -▁2.0 1.7002441524841693e-05 -▁installiert 1.7002441524841693e-05 -vis 1.6998653102042041e-05 -▁Reich 1.6998653102042041e-05 -▁в 1.6998653102042041e-05 -▁96 1.6994864679242385e-05 -▁George 1.6994864679242385e-05 -▁Second 1.6994864679242385e-05 -▁Hol 1.6991076256442733e-05 -▁leaving 1.6991076256442733e-05 -frist 1.6987287833643081e-05 -▁chain 1.6987287833643081e-05 -reisen 1.6983499410843429e-05 -▁Luxembourg 1.6979710988043777e-05 -▁healthy 1.6979710988043777e-05 -▁hinzu 1.6979710988043777e-05 -▁Instead 1.6975922565244122e-05 -▁Ladies 1.6975922565244122e-05 -▁treaty 1.6975922565244122e-05 -▁07 1.697213414244447e-05 -▁amounts 1.697213414244447e-05 -▁Beispiele 1.6968345719644818e-05 -▁newly 1.6968345719644818e-05 -gestellt 1.6964557296845166e-05 -▁Sauna 1.6964557296845166e-05 -xt 1.696076887404551e-05 -▁Presse 1.696076887404551e-05 -▁equally 1.696076887404551e-05 -▁officials 1.696076887404551e-05 -hi 1.6956980451245858e-05 -▁SV 1.6956980451245858e-05 -▁Sign 1.6956980451245858e-05 -▁ruhigen 1.6956980451245858e-05 -▁Alternative 1.6953192028446206e-05 -▁Standort 1.6953192028446206e-05 -▁Verbesserungen 1.6953192028446206e-05 -asen 1.6949403605646554e-05 -kultur 1.6949403605646554e-05 -▁Farb 1.6949403605646554e-05 -▁Vorsitz 1.6949403605646554e-05 -▁happening 1.6949403605646554e-05 -▁reden 1.6949403605646554e-05 -▁Eis 1.6934249914447942e-05 -änge 1.693046149164829e-05 -▁Men 1.6922884646048983e-05 -▁ausgezeichneten 1.6922884646048983e-05 -▁bemüht 1.6919096223249331e-05 -▁Bauern 1.6915307800449679e-05 -▁Stahl 1.6911519377650027e-05 -▁vieler 1.6907730954850371e-05 -iber 1.6900154109251067e-05 -▁seat 1.6896365686451415e-05 -▁Vision 1.6892577263651763e-05 -▁offenen 1.6888788840852108e-05 -▁reserve 1.6888788840852108e-05 -▁Bahn 1.6885000418052456e-05 -▁firstly 1.6885000418052456e-05 -besch 1.6881211995252804e-05 -▁17. 1.6881211995252804e-05 -▁Box 1.6881211995252804e-05 -▁president 1.6881211995252804e-05 -▁laden 1.6877423572453152e-05 -▁strategies 1.6877423572453152e-05 -▁hit 1.6869846726853844e-05 -▁mult 1.6866058304054192e-05 -▁gold 1.6854693035655233e-05 -▁09 1.685090461285558e-05 -legung 1.6847116190055928e-05 -▁einfachen 1.6843327767256276e-05 -▁intelligent 1.6843327767256276e-05 -▁impression 1.6839539344456621e-05 -▁insist 1.6835750921656969e-05 -▁ball 1.6828174076057665e-05 -▁criticism 1.6828174076057665e-05 -weisung 1.6824385653258013e-05 -▁Glas 1.6820597230458357e-05 -▁organizations 1.6816808807658705e-05 -iter 1.6813020384859053e-05 -pack 1.6813020384859053e-05 -▁Ka 1.6813020384859053e-05 -▁communications 1.6813020384859053e-05 -▁entscheidend 1.68092319620594e-05 -▁contributions 1.6805443539259749e-05 -▁handling 1.679407827086079e-05 -▁AS 1.6786501425261482e-05 -▁catch 1.6786501425261482e-05 -▁mandate 1.6786501425261482e-05 -▁ursprünglichen 1.6786501425261482e-05 -ungssystem 1.678271300246183e-05 -▁fühlen 1.6778924579662178e-05 -▁kurze 1.6778924579662178e-05 -wesen 1.6771347734062874e-05 -▁15. 1.6771347734062874e-05 -▁boat 1.6767559311263219e-05 -▁Croatia 1.6763770888463566e-05 -▁preparation 1.6763770888463566e-05 -cle 1.6759982465663914e-05 -▁Town 1.6759982465663914e-05 -ido 1.6756194042864262e-05 -▁universal 1.6756194042864262e-05 -▁balcony 1.6752405620064607e-05 -▁classic 1.6752405620064607e-05 -▁a. 1.6748617197264955e-05 -▁angegeben 1.6748617197264955e-05 -▁glad 1.6748617197264955e-05 -▁Normen 1.6744828774465303e-05 -▁Einstellung 1.6737251928866e-05 -▁Nein 1.6737251928866e-05 -▁discharge 1.6729675083266691e-05 -▁verändern 1.6729675083266691e-05 -▁remaining 1.6725886660467039e-05 -▁root 1.6725886660467039e-05 -16 1.6722098237667387e-05 -▁einerseits 1.671452139206808e-05 -▁gelegt 1.671452139206808e-05 -▁organisiert 1.671452139206808e-05 -▁volume 1.671452139206808e-05 -▁awareness 1.6710732969268428e-05 -▁license 1.6710732969268428e-05 -▁Nieder 1.6706944546468776e-05 -▁Überzeugung 1.6706944546468776e-05 -rab 1.6703156123669124e-05 -▁Dauer 1.6699367700869468e-05 -▁ac 1.6699367700869468e-05 -▁cell 1.6699367700869468e-05 -▁grundlegende 1.6691790855270164e-05 -▁menschlichen 1.6691790855270164e-05 -▁sorgt 1.6691790855270164e-05 -▁spanischen 1.6691790855270164e-05 -й 1.6688002432470512e-05 -▁12. 1.6688002432470512e-05 -▁requirement 1.6688002432470512e-05 -orte 1.668421400967086e-05 -▁charges 1.6680425586871205e-05 -etter 1.6676637164071553e-05 -ny 1.6676637164071553e-05 -ernen 1.66728487412719e-05 -▁Their 1.6669060318472248e-05 -▁früh 1.6665271895672593e-05 -ims 1.6661483472872941e-05 -schluss 1.6661483472872941e-05 -▁Ratsvorsitz 1.6661483472872941e-05 -ENT 1.6653906627273637e-05 -▁Jedes 1.6653906627273637e-05 -▁Kommunikations 1.6653906627273637e-05 -32 1.6650118204473985e-05 -laufen 1.6650118204473985e-05 -▁collective 1.6650118204473985e-05 -▁betreffenden 1.6646329781674329e-05 -▁prov 1.6646329781674329e-05 -▁präsentiert 1.6646329781674329e-05 -▁restrictions 1.6646329781674329e-05 -▁doesn 1.6642541358874677e-05 -▁Berufs 1.6638752936075025e-05 -▁Luc 1.6638752936075025e-05 -wards 1.6631176090475718e-05 -▁fighting 1.6631176090475718e-05 -sicherheit 1.6627387667676066e-05 -schuld 1.6623599244876414e-05 -▁abroad 1.6619810822076762e-05 -▁Brand 1.6612233976477454e-05 -▁earth 1.6612233976477454e-05 -▁Stellung 1.660465713087815e-05 -▁kulturelle 1.660465713087815e-05 -▁liberalisation 1.660465713087815e-05 -▁funktionieren 1.6597080285278846e-05 -hes 1.6593291862479191e-05 -▁camp 1.6593291862479191e-05 -▁musste 1.6593291862479191e-05 -▁KMU 1.6589503439679539e-05 -▁surroundings 1.6585715016879887e-05 -▁10. 1.6581926594080234e-05 -▁Japanese 1.6578138171280579e-05 -▁Press 1.6578138171280579e-05 -ifizierung 1.6574349748480927e-05 -▁issued 1.6570561325681275e-05 -▁stimmt 1.6570561325681275e-05 -▁suite 1.6570561325681275e-05 -▁Bucht 1.6566772902881623e-05 -▁Hof 1.6566772902881623e-05 -▁creates 1.6566772902881623e-05 -▁debates 1.6566772902881623e-05 -▁depends 1.6566772902881623e-05 -▁Rem 1.6562984480081971e-05 -▁Hostelsclub 1.6559196057282315e-05 -▁dark 1.6559196057282315e-05 -▁richten 1.6559196057282315e-05 -▁Angst 1.6555407634482663e-05 -▁ausgesprochen 1.6551619211683011e-05 -▁prove 1.6551619211683011e-05 -▁vous 1.6551619211683011e-05 -▁höchsten 1.6547830788883359e-05 -▁rapporteurs 1.6547830788883359e-05 -▁Mc 1.6544042366083704e-05 -▁operational 1.6544042366083704e-05 -▁starten 1.6544042366083704e-05 -▁achten 1.65364655204844e-05 -▁UNO 1.6532677097684748e-05 -▁Mobil 1.6528888674885096e-05 -▁Veranstaltung 1.6528888674885096e-05 -▁Et 1.652510025208544e-05 -▁Küsten 1.652510025208544e-05 -▁abgelehnt 1.652510025208544e-05 -▁breit 1.652510025208544e-05 -mitteln 1.6521311829285788e-05 -oid 1.6521311829285788e-05 -▁Termin 1.6521311829285788e-05 -▁abzu 1.6521311829285788e-05 -▁cru 1.6521311829285788e-05 -▁lift 1.6517523406486136e-05 -▁Canada 1.6513734983686484e-05 -▁trifft 1.6513734983686484e-05 -▁u.a. 1.6513734983686484e-05 -▁Bulgaria 1.6509946560886829e-05 -inge 1.6506158138087177e-05 -▁percent 1.6502369715287525e-05 -iser 1.6498581292487873e-05 -▁aufzunehmen 1.6498581292487873e-05 -▁beraten 1.6498581292487873e-05 -▁characteristics 1.6494792869688221e-05 -▁domin 1.6494792869688221e-05 -atter 1.6491004446888565e-05 -▁opinions 1.6491004446888565e-05 -▁Dusche 1.6487216024088913e-05 -▁renovated 1.6479639178489609e-05 -rate 1.64720623328903e-05 -▁album 1.64720623328903e-05 -▁lea 1.64720623328903e-05 -ystems 1.6468273910090649e-05 -▁Heimat 1.6460697064491345e-05 -▁returned 1.6460697064491345e-05 -oni 1.645690864169169e-05 -▁Executive 1.645690864169169e-05 -▁wa 1.645690864169169e-05 -itting 1.6453120218892038e-05 -uel 1.6453120218892038e-05 -▁gezogen 1.6453120218892038e-05 -oot 1.6449331796092386e-05 -▁Nahen 1.6449331796092386e-05 -▁Ausstattung 1.6445543373292734e-05 -▁dich 1.6441754950493082e-05 -▁resolutions 1.6441754950493082e-05 -atte 1.6437966527693426e-05 -▁Steuern 1.6437966527693426e-05 -▁View 1.6437966527693426e-05 -▁contracts 1.6437966527693426e-05 -▁sauna 1.6437966527693426e-05 -▁soweit 1.6437966527693426e-05 -▁Wild 1.6434178104893774e-05 -▁maintained 1.6434178104893774e-05 -▁Gentoo 1.6430389682094122e-05 -▁Geschmack 1.6430389682094122e-05 -én 1.642660125929447e-05 -▁decorated 1.642660125929447e-05 -▁größer 1.6422812836494815e-05 -iced 1.6419024413695163e-05 -minister 1.6419024413695163e-05 -▁Armen 1.6419024413695163e-05 -▁übernimmt 1.6419024413695163e-05 -aal 1.6415235990895511e-05 -▁gratulieren 1.6415235990895511e-05 -▁Marina 1.6411447568095859e-05 -▁riesigen 1.6411447568095859e-05 -▁vol 1.6411447568095859e-05 -ired 1.6407659145296207e-05 -▁identify 1.6407659145296207e-05 -ero 1.64000822996969e-05 -▁dritte 1.64000822996969e-05 -▁messages 1.64000822996969e-05 -▁Budget 1.6392505454097595e-05 -▁sexual 1.6392505454097595e-05 -▁dream 1.6388717031297943e-05 -▁strict 1.6388717031297943e-05 -▁ungefähr 1.6384928608498288e-05 -ued 1.6381140185698635e-05 -▁faith 1.6381140185698635e-05 -▁Lie 1.6377351762898983e-05 -▁walls 1.6373563340099331e-05 -▁Kap 1.6369774917299676e-05 -▁aktuell 1.6365986494500024e-05 -▁gets 1.6365986494500024e-05 -ander 1.6362198071700372e-05 -▁Strukturen 1.6362198071700372e-05 -▁brief 1.6362198071700372e-05 -▁Year 1.635840964890072e-05 -▁empf 1.635840964890072e-05 -▁umzusetzen 1.635840964890072e-05 -▁Verfahrens 1.6354621226101068e-05 -▁Wechsel 1.6354621226101068e-05 -▁fee 1.6354621226101068e-05 -▁Ul 1.634704438050176e-05 -▁Washington 1.634704438050176e-05 -iegel 1.6339467534902456e-05 -▁AM 1.63356791121028e-05 -▁youth 1.63356791121028e-05 -▁Harmonisierung 1.6331890689303149e-05 -▁effektiv 1.6331890689303149e-05 -▁hardware 1.6320525420904193e-05 -▁Hohen 1.6316736998104537e-05 -▁erlauben 1.6309160152505233e-05 -▁revolution 1.6309160152505233e-05 -▁verschiedener 1.6309160152505233e-05 -▁WLAN 1.6305371729705581e-05 -ielle 1.6301583306905929e-05 -▁Tem 1.6301583306905929e-05 -▁precise 1.6297794884106274e-05 -can 1.6294006461306621e-05 -▁traditions 1.6294006461306621e-05 -▁Tagung 1.6290218038506969e-05 -▁artist 1.6290218038506969e-05 -▁künftigen 1.6290218038506969e-05 -▁BSE 1.6286429615707317e-05 -▁cast 1.6286429615707317e-05 -amin 1.6282641192907662e-05 -▁gewonnen 1.6282641192907662e-05 -freiheit 1.627885277010801e-05 -führ 1.6275064347308358e-05 -mark 1.6275064347308358e-05 -▁cable 1.6275064347308358e-05 -▁energ 1.6271275924508706e-05 -▁verursacht 1.6267487501709054e-05 -asc 1.62636990789094e-05 -ove 1.6259910656109746e-05 -ras 1.6256122233310094e-05 -▁Thomas 1.6256122233310094e-05 -▁Bestellung 1.6252333810510442e-05 -▁Feuer 1.6252333810510442e-05 -▁durchzuführen 1.6252333810510442e-05 -▁zählt 1.6252333810510442e-05 -▁fashion 1.6248545387710787e-05 -eiten 1.6244756964911135e-05 -▁gender 1.6244756964911135e-05 -▁owner 1.6244756964911135e-05 -▁anstatt 1.6240968542111483e-05 -▁bekämpfen 1.6240968542111483e-05 -▁buchen 1.6240968542111483e-05 -reit 1.6229603273712523e-05 -▁Punkten 1.6222026428113219e-05 -ood 1.6218238005313567e-05 -uellen 1.621066115971426e-05 -▁A3 1.621066115971426e-05 -▁komp 1.621066115971426e-05 -▁occup 1.621066115971426e-05 -▁MEPs 1.6206872736914608e-05 -▁iron 1.6206872736914608e-05 -UT 1.6199295891315303e-05 -rom 1.6199295891315303e-05 -▁reich 1.6199295891315303e-05 -rahmen 1.6195507468515648e-05 -▁Übereinstimmung 1.6195507468515648e-05 -▁Zw 1.6191719045715996e-05 -ruf 1.6187930622916344e-05 -▁Kol 1.6187930622916344e-05 -erien 1.6184142200116692e-05 -liste 1.6184142200116692e-05 -▁Farbe 1.618035377731704e-05 -EF 1.616898850891808e-05 -▁involve 1.616898850891808e-05 -iesen 1.6165200086118428e-05 -▁reserves 1.6165200086118428e-05 -äh 1.6161411663318773e-05 -▁fordere 1.6161411663318773e-05 -▁verlangt 1.6161411663318773e-05 -▁zwe 1.6161411663318773e-05 -seite 1.6157623240519121e-05 -▁gelöst 1.6157623240519121e-05 -▁copyright 1.6153834817719469e-05 -▁meat 1.6146257972120165e-05 -▁parliament 1.6146257972120165e-05 -▁Life 1.6142469549320509e-05 -▁Vater 1.6142469549320509e-05 -iction 1.6138681126520857e-05 -▁MS 1.6138681126520857e-05 -▁Rumänien 1.6138681126520857e-05 -▁Professional 1.6134892703721205e-05 -▁beschränkt 1.6134892703721205e-05 -▁coll 1.6134892703721205e-05 -▁conflicts 1.6134892703721205e-05 -▁mix 1.6134892703721205e-05 -▁regularly 1.6134892703721205e-05 -kor 1.6131104280921553e-05 -▁Abschnitt 1.6127315858121898e-05 -▁hum 1.6127315858121898e-05 -▁Balkan 1.6123527435322246e-05 -▁schafft 1.6123527435322246e-05 -▁Polish 1.6119739012522594e-05 -▁east 1.6119739012522594e-05 -▁Procedure 1.6115950589722942e-05 -▁reported 1.6115950589722942e-05 -▁Host 1.6112162166923289e-05 -▁Reserv 1.6108373744123634e-05 -SS 1.6104585321323982e-05 -abr 1.6104585321323982e-05 -▁transportation 1.6104585321323982e-05 -▁64 1.610079689852433e-05 -▁Ros 1.610079689852433e-05 -▁Geräte 1.6097008475724678e-05 -▁Noch 1.6097008475724678e-05 -▁lautet 1.6097008475724678e-05 -▁visa 1.6097008475724678e-05 -▁Boot 1.6085643207325718e-05 -▁metal 1.6085643207325718e-05 -ental 1.6081854784526066e-05 -▁¤ 1.6081854784526066e-05 -▁Krankheiten 1.6078066361726414e-05 -▁Mittelmeer 1.6078066361726414e-05 -▁Änderungsanträgen 1.6070489516127107e-05 -▁mus 1.6066701093327455e-05 -▁mut 1.6066701093327455e-05 -ñ 1.6062912670527803e-05 -▁Gerichtshof 1.6062912670527803e-05 -▁Maastricht 1.6062912670527803e-05 -▁Verhandlungs 1.6059124247728151e-05 -▁battle 1.6059124247728151e-05 -▁facilitate 1.6059124247728151e-05 -▁mögliche 1.6059124247728151e-05 -▁ur 1.6059124247728151e-05 -IB 1.6055335824928495e-05 -ney 1.6055335824928495e-05 -sc 1.6055335824928495e-05 -▁Gegend 1.6055335824928495e-05 -ziele 1.6051547402128843e-05 -las 1.6047758979329191e-05 -▁Ausführungen 1.6047758979329191e-05 -▁gone 1.6047758979329191e-05 -▁auftreten 1.6043970556529539e-05 -▁schnelle 1.6043970556529539e-05 -äg 1.6040182133729884e-05 -▁Legal 1.6040182133729884e-05 -▁Übersetzung 1.6040182133729884e-05 -▁recommendation 1.6036393710930232e-05 -▁Award 1.603260528813058e-05 -▁Zuständigkeit 1.603260528813058e-05 -▁angesehen 1.603260528813058e-05 -▁drawing 1.6025028442531276e-05 -▁parents 1.6025028442531276e-05 -▁IC 1.602124001973162e-05 -▁cor 1.602124001973162e-05 -aries 1.6017451596931968e-05 -oring 1.6017451596931968e-05 -▁annehmen 1.6017451596931968e-05 -code 1.6009874751332664e-05 -har 1.6009874751332664e-05 -▁Win 1.6009874751332664e-05 -▁individuell 1.6009874751332664e-05 -▁properties 1.6009874751332664e-05 -arischen 1.6002297905733356e-05 -leit 1.6002297905733356e-05 -▁Tatsächlich 1.5998509482933704e-05 -itze 1.5994721060134052e-05 -▁depend 1.5994721060134052e-05 -▁intensive 1.5994721060134052e-05 -unds 1.59909326373344e-05 -▁Deutsch 1.59909326373344e-05 -▁northern 1.59909326373344e-05 -▁sem 1.5987144214534745e-05 -▁hundreds 1.5983355791735093e-05 -▁customs 1.5979567368935441e-05 -▁submit 1.5979567368935441e-05 -▁Cons 1.5975778946135789e-05 -▁topic 1.5971990523336137e-05 -ared 1.5968202100536481e-05 -arf 1.5968202100536481e-05 -▁breite 1.5968202100536481e-05 -▁conducted 1.5968202100536481e-05 -bad 1.5964413677736829e-05 -▁Festlegung 1.5964413677736829e-05 -ritt 1.5960625254937177e-05 -86 1.5956836832137525e-05 -odes 1.5956836832137525e-05 -spar 1.5956836832137525e-05 -▁Gestaltung 1.5956836832137525e-05 -▁dates 1.5956836832137525e-05 -▁wissenschaftlichen 1.5956836832137525e-05 -▁Anti 1.595304840933787e-05 -▁avec 1.595304840933787e-05 -▁pred 1.595304840933787e-05 --Reg 1.5949259986538218e-05 -uber 1.5949259986538218e-05 -▁Vorschlägen 1.5949259986538218e-05 -▁Congress 1.5945471563738566e-05 -▁Garantie 1.5945471563738566e-05 -▁Problems 1.5945471563738566e-05 -▁Regen 1.5945471563738566e-05 -▁zentralen 1.5945471563738566e-05 -▁sehe 1.5941683140938914e-05 -▁somewhat 1.5941683140938914e-05 -▁Earth 1.5937894718139262e-05 -side 1.5934106295339606e-05 -vorschriften 1.5934106295339606e-05 -▁Lu 1.5930317872539954e-05 -▁PRO 1.5930317872539954e-05 -▁erheblich 1.5930317872539954e-05 -▁mitten 1.5926529449740302e-05 -standards 1.592274102694065e-05 -ungsver 1.5915164181341343e-05 -▁Block 1.5915164181341343e-05 -▁aufgefordert 1.5915164181341343e-05 -igend 1.591137575854169e-05 -rats 1.591137575854169e-05 -▁ausgeschlossen 1.591137575854169e-05 -▁ausgezeichnet 1.591137575854169e-05 -▁fac 1.5903798912942386e-05 -▁bezug 1.5900010490142731e-05 -▁worse 1.5900010490142731e-05 -lass 1.5896222067343079e-05 -▁Aktien 1.5892433644543427e-05 -version 1.5888645221743775e-05 -▁Beitritts 1.5888645221743775e-05 -ucher 1.5884856798944123e-05 -▁Fa 1.5884856798944123e-05 -▁Kilometer 1.5884856798944123e-05 -▁einig 1.5884856798944123e-05 -▁Drittländern 1.5877279953344815e-05 -▁ste 1.5877279953344815e-05 -▁Fax 1.5869703107745511e-05 -▁Reis 1.5869703107745511e-05 -▁Stud 1.5869703107745511e-05 -▁terrorist 1.5869703107745511e-05 -▁vacation 1.5869703107745511e-05 -oma 1.5865914684945856e-05 -▁displayed 1.5865914684945856e-05 -▁hervorheben 1.5865914684945856e-05 -▁kurzen 1.5862126262146204e-05 -▁Ideal 1.5858337839346552e-05 -▁DA 1.58545494165469e-05 -▁Miet 1.58545494165469e-05 --Programm 1.5846972570947592e-05 -fach 1.5846972570947592e-05 -üss 1.5846972570947592e-05 -▁beglückwünschen 1.5846972570947592e-05 -▁pros 1.5846972570947592e-05 -produkte 1.584318414814794e-05 -▁depending 1.584318414814794e-05 -räume 1.5835607302548636e-05 -▁Einheit 1.5835607302548636e-05 -▁define 1.5835607302548636e-05 -▁Disk 1.5831818879748981e-05 -▁faster 1.5831818879748981e-05 -▁parliaments 1.5831818879748981e-05 -back 1.5828030456949329e-05 -▁performed 1.5828030456949329e-05 -▁career 1.5824242034149676e-05 -▁interior 1.5824242034149676e-05 -▁holding 1.5820453611350024e-05 -▁regulatory 1.5820453611350024e-05 -ban 1.5816665188550372e-05 -lei 1.5812876765750717e-05 -▁Sach 1.5809088342951065e-05 -▁zugunsten 1.5809088342951065e-05 -▁updated 1.5805299920151413e-05 -▁ir 1.5801511497351761e-05 -▁joined 1.5797723074552105e-05 -pool 1.5793934651752453e-05 -▁decision-making 1.5793934651752453e-05 -ätzen 1.57901462289528e-05 -▁Food 1.57901462289528e-05 -uell 1.5786357806153149e-05 -zeugen 1.5782569383353497e-05 -▁expert 1.5782569383353497e-05 -▁Cyprus 1.5778780960553842e-05 -erc 1.577499253775419e-05 -ivity 1.577499253775419e-05 -ragen 1.577499253775419e-05 -stück 1.577499253775419e-05 -▁Ganz 1.577499253775419e-05 -St 1.5771204114954538e-05 -▁Fähigkeiten 1.5771204114954538e-05 -▁nachhaltigen 1.5771204114954538e-05 -▁Ref 1.5767415692154886e-05 -▁vo 1.5767415692154886e-05 -▁towns 1.5763627269355234e-05 -▁frequently 1.5759838846555578e-05 -▁dead 1.5756050423755926e-05 -▁seeing 1.5756050423755926e-05 -▁zufolge 1.5756050423755926e-05 -IV 1.5752262000956274e-05 -▁Check 1.5752262000956274e-05 -▁Entwicklungsländer 1.5752262000956274e-05 -▁Wird 1.5752262000956274e-05 -▁connections 1.5752262000956274e-05 -▁Sonne 1.5748473578156622e-05 --200 1.5740896732557315e-05 --Ar 1.5740896732557315e-05 -lungen 1.5740896732557315e-05 -▁Umfeld 1.5740896732557315e-05 -▁settings 1.5740896732557315e-05 -▁Hostels 1.573331988695801e-05 -▁answers 1.573331988695801e-05 -▁deals 1.573331988695801e-05 -▁durchführen 1.573331988695801e-05 -▁stylish 1.573331988695801e-05 -ingly 1.5729531464158358e-05 -▁Fair 1.5729531464158358e-05 -▁Hilfs 1.5729531464158358e-05 -▁driver 1.5729531464158358e-05 -▁früheren 1.5729531464158358e-05 -▁verbringen 1.5729531464158358e-05 -chte 1.5725743041358703e-05 -▁Def 1.5725743041358703e-05 -▁Max 1.5725743041358703e-05 -▁Regime 1.5725743041358703e-05 -▁Modern 1.5721954618559051e-05 -▁sodass 1.57181661957594e-05 -▁minority 1.5714377772959747e-05 -ress 1.5710589350160091e-05 -▁emergency 1.5706800927360439e-05 -nehmer 1.5703012504560787e-05 -▁16. 1.5703012504560787e-05 -▁SMEs 1.5703012504560787e-05 -▁zuständigen 1.5703012504560787e-05 -nungs 1.5699224081761135e-05 -▁Detail 1.5699224081761135e-05 -rach 1.5695435658961483e-05 -▁Bro 1.5691647236161828e-05 -order 1.5687858813362176e-05 -▁Alles 1.5687858813362176e-05 -▁Ratspräsidentschaft 1.5687858813362176e-05 -▁islands 1.5687858813362176e-05 -▁mostly 1.5687858813362176e-05 -atischen 1.5684070390562524e-05 -▁Ausstellung 1.5684070390562524e-05 -fra 1.5680281967762872e-05 -upp 1.5680281967762872e-05 -uted 1.567649354496322e-05 -▁ought 1.567649354496322e-05 -finden 1.5672705122163564e-05 -teilen 1.5672705122163564e-05 -Г 1.5672705122163564e-05 -▁02 1.5672705122163564e-05 -▁Road 1.5672705122163564e-05 -▁Gleichgewicht 1.5661339853764608e-05 -з 1.5657551430964953e-05 -▁verabschiedet 1.5657551430964953e-05 -▁Kontext 1.5649974585365649e-05 -asting 1.5646186162565997e-05 -▁Protection 1.5646186162565997e-05 -▁Taiwan 1.5646186162565997e-05 -▁mother 1.5646186162565997e-05 -OW 1.5642397739766344e-05 -▁follows 1.5642397739766344e-05 -▁bestätigen 1.5634820894167037e-05 -▁cash 1.5634820894167037e-05 -▁ultimately 1.5634820894167037e-05 -ektion 1.5631032471367385e-05 -▁Mail 1.5631032471367385e-05 -▁continent 1.5631032471367385e-05 -▁noted 1.5631032471367385e-05 -▁techniques 1.5631032471367385e-05 -▁Ident 1.5627244048567733e-05 -▁Zentralbank 1.5627244048567733e-05 -▁label 1.5604513511769814e-05 -▁proof 1.5604513511769814e-05 -▁Websites 1.5600725088970162e-05 -▁windows 1.5600725088970162e-05 -▁bottom 1.559693666617051e-05 -▁demanding 1.559693666617051e-05 -▁div 1.5593148243370858e-05 -▁Speicher 1.5589359820571202e-05 -geber 1.558557139777155e-05 -▁Monetary 1.558557139777155e-05 -▁d.h. 1.558557139777155e-05 -nahmen 1.5581782974971898e-05 -ena 1.5574206129372594e-05 -umb 1.5574206129372594e-05 -▁Arbeitsplätzen 1.5574206129372594e-05 -▁Erk 1.5574206129372594e-05 -▁Pin 1.5574206129372594e-05 -▁ra 1.5574206129372594e-05 -▁Rund 1.5570417706572939e-05 -17 1.5566629283773287e-05 -▁Nachricht 1.5566629283773287e-05 -▁hostel 1.5566629283773287e-05 -▁ethnic 1.5562840860973635e-05 -jährigen 1.5559052438173983e-05 -▁notwendige 1.5559052438173983e-05 -▁organized 1.5559052438173983e-05 -▁Ziffer 1.5555264015374331e-05 -▁visual 1.5555264015374331e-05 -olit 1.5547687169775023e-05 -▁drug 1.5543898746975371e-05 -▁grateful 1.5543898746975371e-05 -▁46 1.5540110324175719e-05 -▁Read 1.5540110324175719e-05 -▁Übereinkommen 1.5540110324175719e-05 -▁ll 1.5532533478576411e-05 -acher 1.5528745055776759e-05 -▁8. 1.5528745055776759e-05 -▁Wass 1.5528745055776759e-05 -▁electricity 1.5528745055776759e-05 -▁meant 1.5528745055776759e-05 -itzen 1.5524956632977107e-05 -▁inner 1.5524956632977107e-05 -aged 1.5521168210177455e-05 -▁Rel 1.5521168210177455e-05 -▁Drittens 1.55173797873778e-05 -▁Gewicht 1.5513591364578148e-05 -▁Ausführung 1.5509802941778496e-05 -▁Plenum 1.5509802941778496e-05 -▁Arbeitsplatz 1.5506014518978844e-05 -▁MB 1.5506014518978844e-05 -▁interessante 1.5498437673379536e-05 -▁invite 1.5498437673379536e-05 -▁Hot 1.5494649250579884e-05 -▁Sony 1.5494649250579884e-05 -haben 1.548707240498058e-05 -hne 1.548707240498058e-05 -▁Prä 1.548707240498058e-05 -▁castle 1.548707240498058e-05 -▁Ersatz 1.5483283982180925e-05 -▁öffentlich 1.5483283982180925e-05 -▁Deutschen 1.5479495559381273e-05 -amil 1.5475707136581621e-05 -▁Schlafzimmer 1.5475707136581621e-05 -▁divided 1.5475707136581621e-05 -▁Tool 1.5471918713781969e-05 -iano 1.5468130290982317e-05 -▁bezahlen 1.5468130290982317e-05 -▁einzigartige 1.5464341868182661e-05 -▁Redner 1.5460553445383009e-05 -▁produziert 1.5460553445383009e-05 -▁quarter 1.5460553445383009e-05 -inner 1.5445399754184398e-05 -uction 1.5445399754184398e-05 -▁delighted 1.5445399754184398e-05 -▁hängt 1.5445399754184398e-05 -▁library 1.5445399754184398e-05 -▁suites 1.5445399754184398e-05 -▁derartige 1.5441611331384745e-05 -▁mat 1.5441611331384745e-05 -weis 1.5437822908585093e-05 -▁Förder 1.5437822908585093e-05 -aking 1.5426457640186134e-05 -▁intensiv 1.541888079458683e-05 -▁PS 1.5415092371787174e-05 -▁Training 1.540751552618787e-05 -▁relaxation 1.540751552618787e-05 -ý 1.5399938680588566e-05 -▁Eing 1.5399938680588566e-05 -▁James 1.5399938680588566e-05 -▁serviert 1.5399938680588566e-05 -â 1.5396150257788911e-05 -▁statistics 1.5396150257788911e-05 -▁trend 1.5396150257788911e-05 -ever 1.5392361834989259e-05 -▁Anbieter 1.5388573412189607e-05 -▁Buchung 1.5388573412189607e-05 -▁gewissen 1.5388573412189607e-05 -▁Mat 1.5373419720990995e-05 -▁arrive 1.5373419720990995e-05 -▁contrast 1.5369631298191343e-05 -▁Whether 1.5365842875391691e-05 -▁contacts 1.5362054452592036e-05 -▁visible 1.5362054452592036e-05 -▁Verbot 1.5358266029792384e-05 -▁wines 1.5358266029792384e-05 -▁Erholung 1.5354477606992731e-05 -ature 1.5350689184193079e-05 -▁EM 1.5346900761393427e-05 -▁producing 1.5346900761393427e-05 -▁180 1.5343112338593772e-05 -▁Regierungskonferenz 1.5343112338593772e-05 -▁verhindert 1.5343112338593772e-05 -▁expansion 1.533932391579412e-05 -▁succeed 1.533932391579412e-05 -▁ticket 1.533932391579412e-05 -bit 1.5335535492994468e-05 -▁Mexico 1.5335535492994468e-05 -▁dre 1.5331747070194816e-05 -▁widely 1.5331747070194816e-05 -nach 1.5324170224595508e-05 -▁Budapest 1.5324170224595508e-05 -▁Schuld 1.5324170224595508e-05 -▁poll 1.5324170224595508e-05 -ality 1.5320381801795856e-05 -maschine 1.5320381801795856e-05 -▁SC 1.5320381801795856e-05 -▁Partnern 1.5316593378996204e-05 -▁Techn 1.5316593378996204e-05 -▁attend 1.5316593378996204e-05 -▁geprüft 1.5316593378996204e-05 -▁SA 1.5312804956196552e-05 -▁courage 1.5312804956196552e-05 -emb 1.5309016533396897e-05 -gut 1.5309016533396897e-05 -▁Prov 1.5309016533396897e-05 -▁channels 1.5309016533396897e-05 -▁entwickelten 1.5309016533396897e-05 -▁investors 1.5309016533396897e-05 -▁heritage 1.5305228110597245e-05 -▁locations 1.5305228110597245e-05 -fahr 1.5301439687797593e-05 -▁migration 1.5301439687797593e-05 -▁welfare 1.5301439687797593e-05 -▁Videos 1.5297651264997941e-05 -▁tourists 1.5297651264997941e-05 -▁verleihen 1.5293862842198285e-05 -schafts 1.5290074419398633e-05 -▁bedeut 1.5290074419398633e-05 -▁verstanden 1.5290074419398633e-05 -blick 1.5286285996598981e-05 -▁Prinzipien 1.5286285996598981e-05 -▁Yes 1.5282497573799329e-05 -▁entrance 1.5282497573799329e-05 -qual 1.5278709150999677e-05 -▁Europol 1.5278709150999677e-05 -▁stars 1.5278709150999677e-05 -mo 1.5274920728200022e-05 -uls 1.5274920728200022e-05 -west 1.5274920728200022e-05 -▁Planung 1.5274920728200022e-05 -▁Qual 1.5274920728200022e-05 -▁Verringerung 1.527113230540037e-05 -▁Zwei 1.5263555459801065e-05 -▁valley 1.5263555459801065e-05 -▁Kreuz 1.5259767037001413e-05 -▁erreichbar 1.5259767037001413e-05 -▁ausgesetzt 1.525597861420176e-05 -▁divers 1.5248401768602454e-05 -▁starts 1.5248401768602454e-05 -▁konkret 1.52446133458028e-05 -▁pollution 1.52446133458028e-05 -▁Adobe 1.5237036500203494e-05 -▁Ausgabe 1.5233248077403842e-05 -▁aufnehmen 1.522945965460419e-05 -▁weak 1.522945965460419e-05 -▁Prague 1.5225671231804537e-05 -▁pieces 1.5225671231804537e-05 -ación 1.5221882809004885e-05 -▁heat 1.5221882809004885e-05 -▁Pop 1.5218094386205231e-05 -▁index 1.5218094386205231e-05 -▁planet 1.5218094386205231e-05 -key 1.5206729117806273e-05 -wald 1.5206729117806273e-05 -▁exciting 1.5206729117806273e-05 -▁involves 1.5206729117806273e-05 -▁tea 1.5206729117806273e-05 -wohn 1.5202940695006621e-05 -▁Besonders 1.5202940695006621e-05 -▁inhabitants 1.5202940695006621e-05 -▁unmöglich 1.5202940695006621e-05 -▁Ankunft 1.5199152272206967e-05 -▁stored 1.5199152272206967e-05 -ni 1.5195363849407315e-05 -▁Funds 1.5195363849407315e-05 -▁Geschäfte 1.5195363849407315e-05 -▁abgestimmt 1.5195363849407315e-05 -▁careful 1.5195363849407315e-05 -zwe 1.5191575426607661e-05 -▁direkten 1.5191575426607661e-05 -▁'ve 1.5187787003808009e-05 -▁cool 1.5187787003808009e-05 -ario 1.5180210158208704e-05 -ikan 1.5180210158208704e-05 -▁declaration 1.5180210158208704e-05 -▁Lab 1.517642173540905e-05 -▁Nahrungsmittel 1.517642173540905e-05 -iro 1.5172633312609398e-05 -▁bestimmen 1.5172633312609398e-05 -▁entsteht 1.5172633312609398e-05 -▁flights 1.5172633312609398e-05 -▁organ 1.5172633312609398e-05 -▁passiert 1.5168844889809746e-05 -▁presents 1.5168844889809746e-05 -▁afternoon 1.5165056467010092e-05 -▁subsidies 1.5165056467010092e-05 -▁visiting 1.5153691198611134e-05 -▁welchem 1.5153691198611134e-05 -▁Verteidigung 1.5146114353011828e-05 -▁cosy 1.5146114353011828e-05 -▁pers 1.5146114353011828e-05 -▁explore 1.5142325930212176e-05 -gef 1.5138537507412523e-05 -▁IV 1.5138537507412523e-05 -▁drivers 1.5138537507412523e-05 -▁suffer 1.5134749084612871e-05 -altet 1.5130960661813217e-05 -▁Portuguese 1.5127172239013565e-05 -▁Shop 1.5127172239013565e-05 -▁input 1.5127172239013565e-05 -▁remote 1.5127172239013565e-05 -▁showing 1.5127172239013565e-05 -▁Char 1.5123383816213911e-05 -▁Framework 1.5119595393414259e-05 -lösung 1.5115806970614605e-05 -▁Faktoren 1.5115806970614605e-05 -▁Kamera 1.5115806970614605e-05 -▁gardens 1.5112018547814953e-05 -▁oral 1.5112018547814953e-05 -▁Führer 1.5108230125015301e-05 -▁Haftung 1.5108230125015301e-05 -▁Bemerkungen 1.5104441702215647e-05 -▁Bildungs 1.5100653279415995e-05 -▁definiert 1.5100653279415995e-05 -▁virtual 1.5100653279415995e-05 -▁micro 1.5096864856616342e-05 -▁03 1.509307643381669e-05 -▁Audio 1.5089288011017036e-05 -▁Beiträge 1.5089288011017036e-05 -▁Up 1.5089288011017036e-05 -▁bilateral 1.5089288011017036e-05 -▁proved 1.5089288011017036e-05 -izer 1.5085499588217384e-05 -▁indicated 1.5085499588217384e-05 -gesetz 1.5081711165417732e-05 -▁bound 1.5081711165417732e-05 -▁Brazil 1.5077922742618078e-05 -▁Gerichte 1.5077922742618078e-05 -▁Straf 1.5074134319818426e-05 -▁kämpfen 1.5074134319818426e-05 -▁legt 1.5074134319818426e-05 -▁wollten 1.5074134319818426e-05 -fähig 1.5070345897018772e-05 -▁Copenhagen 1.5070345897018772e-05 -▁Wünsche 1.5070345897018772e-05 -▁cutting 1.5062769051419466e-05 -▁visited 1.5062769051419466e-05 -ican 1.5058980628619814e-05 -▁sorgfältig 1.5058980628619814e-05 -▁understood 1.5058980628619814e-05 -▁Format 1.5055192205820161e-05 -▁Schweizer 1.5055192205820161e-05 -▁collaboration 1.5055192205820161e-05 -▁mil 1.5055192205820161e-05 -▁vollständige 1.5055192205820161e-05 -ückt 1.5051403783020509e-05 -▁Blue 1.5051403783020509e-05 -▁focused 1.5051403783020509e-05 -iellen 1.5047615360220857e-05 -▁Freiheiten 1.5047615360220857e-05 -▁maintaining 1.5047615360220857e-05 -▁Zwe 1.5043826937421203e-05 -▁Beziehung 1.5040038514621551e-05 -▁wann 1.5040038514621551e-05 -▁westlichen 1.5040038514621551e-05 -▁Baltic 1.5036250091821897e-05 -▁preis 1.5032461669022245e-05 -ems 1.5028673246222591e-05 -▁Vietnam 1.5028673246222591e-05 -▁echten 1.5028673246222591e-05 -ч 1.5024884823422939e-05 -▁Einbeziehung 1.5024884823422939e-05 -▁Neue 1.5024884823422939e-05 -▁Fleisch 1.5021096400623287e-05 -▁demonstrated 1.5021096400623287e-05 -▁prim 1.5021096400623287e-05 -▁Deswegen 1.5017307977823633e-05 -▁Konsens 1.5017307977823633e-05 -▁ep 1.5017307977823633e-05 -▁fitness 1.5017307977823633e-05 -▁geboren 1.5013519555023981e-05 -▁Friday 1.5009731132224328e-05 -▁accident 1.5009731132224328e-05 -▁dur 1.5005942709424676e-05 -▁provider 1.5005942709424676e-05 -lern 1.5002154286625022e-05 -▁Verw 1.5002154286625022e-05 -▁demonstrate 1.5002154286625022e-05 -▁owners 1.5002154286625022e-05 -▁ausgeführt 1.499836586382537e-05 -▁staatlichen 1.499836586382537e-05 -ey 1.4994577441025718e-05 -systeme 1.4994577441025718e-05 -▁heating 1.4994577441025718e-05 -▁Fernseh 1.4990789018226064e-05 -▁Jede 1.4990789018226064e-05 -▁begrüßt 1.4990789018226064e-05 -▁Ball 1.4987000595426412e-05 -▁Verbindungen 1.4987000595426412e-05 -häuser 1.4983212172626758e-05 -▁Pact 1.4983212172626758e-05 -▁erleichtern 1.4983212172626758e-05 -▁inflation 1.4983212172626758e-05 -ande 1.4979423749827106e-05 -▁Hostel 1.4979423749827106e-05 -▁Nos 1.4975635327027453e-05 -ähr 1.49718469042278e-05 -▁Games 1.4968058481428147e-05 -▁Question 1.4968058481428147e-05 -▁Vorschlags 1.4968058481428147e-05 -▁signs 1.4968058481428147e-05 -abil 1.4964270058628495e-05 -▁carbon 1.4964270058628495e-05 -▁kostet 1.4964270058628495e-05 -▁geltenden 1.4960481635828843e-05 -▁hinzufügen 1.4960481635828843e-05 -▁transition 1.4960481635828843e-05 -▁lounge 1.4956693213029189e-05 -▁beschrieben 1.4952904790229537e-05 -mas 1.4949116367429883e-05 -▁Kompetenz 1.4945327944630231e-05 -halb 1.4941539521830577e-05 -wo 1.4941539521830577e-05 -▁13. 1.4941539521830577e-05 -▁Asien 1.4941539521830577e-05 -▁höchst 1.4941539521830577e-05 -▁nah 1.4941539521830577e-05 -bek 1.4937751099030925e-05 -ösen 1.4937751099030925e-05 -▁Werbung 1.4937751099030925e-05 -▁department 1.4937751099030925e-05 -atis 1.4933962676231273e-05 -linie 1.4933962676231273e-05 -pa 1.4933962676231273e-05 -werks 1.4922597407832314e-05 -▁chair 1.4922597407832314e-05 -▁spoken 1.4922597407832314e-05 -▁höheren 1.4915020562233008e-05 -▁Polizei 1.4911232139433356e-05 -▁Golden 1.4907443716633702e-05 -▁unw 1.4907443716633702e-05 -▁Orte 1.490365529383405e-05 -▁Fun 1.4899866871034398e-05 -▁gezwungen 1.4899866871034398e-05 -▁finished 1.4896078448234744e-05 -▁Zypern 1.4892290025435092e-05 -▁Sorgen 1.4888501602635439e-05 -kop 1.4880924757036133e-05 -▁straight 1.4880924757036133e-05 -alle 1.4877136334236481e-05 -anner 1.4877136334236481e-05 -ava 1.4877136334236481e-05 -▁Ungarn 1.4877136334236481e-05 -▁gespeichert 1.4877136334236481e-05 -▁Gemeinde 1.4873347911436829e-05 -▁Balkon 1.4869559488637175e-05 -fallen 1.4865771065837523e-05 -▁Projekten 1.4865771065837523e-05 -▁privat 1.4865771065837523e-05 -ologischen 1.4861982643037869e-05 -ätz 1.4861982643037869e-05 -▁Gründung 1.4861982643037869e-05 -▁Ware 1.4861982643037869e-05 -▁accompanied 1.4861982643037869e-05 -▁raw 1.4858194220238217e-05 -▁Canon 1.4854405797438563e-05 -▁Che 1.4854405797438563e-05 -▁Dass 1.4854405797438563e-05 -▁pet 1.4854405797438563e-05 -▁solange 1.4854405797438563e-05 -▁theme 1.4854405797438563e-05 -▁circ 1.4850617374638911e-05 -▁2, 1.4846828951839259e-05 -▁Automat 1.4843040529039606e-05 -▁Image 1.4843040529039606e-05 -▁printing 1.4843040529039606e-05 -asser 1.4839252106239953e-05 -▁Putin 1.4839252106239953e-05 -ikt 1.48354636834403e-05 -▁facility 1.48354636834403e-05 -▁fra 1.48354636834403e-05 -ringen 1.4831675260640648e-05 -▁Edition 1.4827886837840994e-05 -▁bele 1.4827886837840994e-05 -▁spät 1.4827886837840994e-05 -ector 1.4820309992241688e-05 -ierend 1.4820309992241688e-05 -▁BIP 1.4820309992241688e-05 -▁gebaut 1.4820309992241688e-05 -ancing 1.4816521569442036e-05 -▁bur 1.4816521569442036e-05 -▁chinesischen 1.4816521569442036e-05 -▁ak 1.4812733146642384e-05 -▁high-quality 1.4812733146642384e-05 -▁nahezu 1.4812733146642384e-05 -osten 1.480894472384273e-05 -▁categories 1.480894472384273e-05 -▁verehrte 1.480894472384273e-05 -med 1.4801367878243425e-05 -elte 1.4797579455443773e-05 -uft 1.4797579455443773e-05 -ausschuß 1.4793791032644119e-05 -set 1.4793791032644119e-05 -▁Vereinbarungen 1.4793791032644119e-05 -▁sad 1.4793791032644119e-05 -ießen 1.4790002609844467e-05 -▁gek 1.4790002609844467e-05 -kehr 1.4786214187044815e-05 -▁char 1.4786214187044815e-05 -▁Betracht 1.4778637341445509e-05 -▁spectacular 1.4778637341445509e-05 -▁Symbol 1.4774848918645855e-05 -put 1.4771060495846203e-05 -▁Project 1.4771060495846203e-05 -▁Uns 1.4771060495846203e-05 -▁settlement 1.4771060495846203e-05 -▁verringern 1.4771060495846203e-05 -▁Rein 1.4767272073046549e-05 -▁2012 1.4763483650246897e-05 -▁Kyoto 1.4763483650246897e-05 -keit 1.4759695227447244e-05 -▁Webseiten 1.4759695227447244e-05 -▁Energ 1.4755906804647592e-05 -▁controlled 1.4755906804647592e-05 -▁9. 1.475211838184794e-05 -▁CO 1.475211838184794e-05 -▁Karl 1.475211838184794e-05 -▁exceed 1.475211838184794e-05 -ave 1.4748329959048286e-05 -▁angemessene 1.4748329959048286e-05 -▁Univers 1.4744541536248634e-05 -▁ferner 1.4744541536248634e-05 -▁ongoing 1.4744541536248634e-05 -▁plenty 1.4744541536248634e-05 -▁rational 1.4744541536248634e-05 -▁Dir 1.474075311344898e-05 -onic 1.4736964690649328e-05 -▁Perhaps 1.4736964690649328e-05 -▁conc 1.4736964690649328e-05 -▁fertig 1.4736964690649328e-05 -▁vorlegen 1.4736964690649328e-05 -ois 1.4733176267849674e-05 -▁Autor 1.4733176267849674e-05 -▁Sar 1.4733176267849674e-05 -▁bemühen 1.4733176267849674e-05 -▁documentation 1.4733176267849674e-05 -▁Anw 1.472559942225037e-05 -▁spr 1.472559942225037e-05 -▁Einwohner 1.4721810999450716e-05 -▁Interess 1.4721810999450716e-05 -▁heutige 1.4721810999450716e-05 -▁labor 1.4721810999450716e-05 -▁exception 1.4718022576651064e-05 -iol 1.4714234153851411e-05 -▁era 1.4714234153851411e-05 -wel 1.4710445731051759e-05 -▁Lö 1.4710445731051759e-05 -ford 1.4702868885452453e-05 -▁enc 1.4702868885452453e-05 -▁safeguard 1.4702868885452453e-05 -▁Fluss 1.4699080462652799e-05 -▁boot 1.4695292039853147e-05 -▁erinnert 1.4695292039853147e-05 -▁personnel 1.4695292039853147e-05 -▁effectiveness 1.4691503617053495e-05 -▁Mobilität 1.4687715194253841e-05 -yer 1.4683926771454189e-05 -halte 1.4680138348654535e-05 -▁Gesellschaften 1.4680138348654535e-05 -▁recon 1.4680138348654535e-05 -▁Fel 1.4676349925854883e-05 -vention 1.467256150305523e-05 -▁definitely 1.467256150305523e-05 -spartner 1.4668773080255578e-05 -otten 1.4661196234656272e-05 -▁Bars 1.465740781185662e-05 -▁ke 1.465740781185662e-05 --und 1.4653619389056966e-05 -▁Network 1.4653619389056966e-05 -▁concluded 1.4653619389056966e-05 -▁integriert 1.4653619389056966e-05 -▁ve 1.4653619389056966e-05 -PE 1.4649830966257314e-05 -▁citizen 1.4649830966257314e-05 -▁committees 1.4638465697858356e-05 -tritt 1.4634677275058702e-05 -▁m2 1.4634677275058702e-05 -▁tickets 1.4634677275058702e-05 -▁Jun 1.463088885225905e-05 -▁syn 1.463088885225905e-05 -▁Argument 1.4627100429459397e-05 -▁Hö 1.4627100429459397e-05 -▁47 1.4623312006659745e-05 -▁Page 1.4623312006659745e-05 -setz 1.4619523583860091e-05 -▁Aktiv 1.4615735161060439e-05 -▁Außenpolitik 1.4615735161060439e-05 -▁Provinz 1.4615735161060439e-05 -▁awarded 1.4615735161060439e-05 -▁educational 1.4615735161060439e-05 -ibe 1.4611946738260785e-05 -▁Veröffentlichung 1.4611946738260785e-05 -▁belong 1.4611946738260785e-05 -▁Viertel 1.4608158315461133e-05 -▁replaced 1.4608158315461133e-05 -▁resistance 1.4604369892661481e-05 -▁Michel 1.4600581469861827e-05 -▁Abst 1.4593004624262521e-05 -▁Ac 1.4589216201462869e-05 -▁Anschluss 1.4589216201462869e-05 -▁umwelt 1.4589216201462869e-05 -▁übrigens 1.4589216201462869e-05 -ending 1.4585427778663216e-05 -▁Roll 1.4585427778663216e-05 -▁Verlust 1.4585427778663216e-05 -aria 1.4581639355863564e-05 -ll 1.4581639355863564e-05 -▁Mill 1.4581639355863564e-05 -▁foundation 1.4581639355863564e-05 -▁Gerät 1.4577850933063912e-05 -▁Gespräche 1.4574062510264258e-05 -▁firms 1.4574062510264258e-05 -▁jegliche 1.4574062510264258e-05 -▁spezifischen 1.4574062510264258e-05 --Pr 1.4570274087464606e-05 -ilit 1.4570274087464606e-05 -▁Amendments 1.4570274087464606e-05 -▁Identität 1.4570274087464606e-05 -▁passenger 1.4570274087464606e-05 -▁possibly 1.4570274087464606e-05 -▁Israeli 1.45626972418653e-05 -▁Chancen 1.4558908819065646e-05 -▁Sollten 1.4558908819065646e-05 -▁behaviour 1.4558908819065646e-05 -▁organis 1.4558908819065646e-05 -▁termin 1.4551331973466341e-05 -▁minorities 1.4547543550666688e-05 -▁bestehende 1.4543755127867036e-05 -▁06 1.4539966705067383e-05 -▁count 1.4532389859468077e-05 -▁occasions 1.4532389859468077e-05 -▁phrase 1.4528601436668425e-05 -▁sugar 1.4528601436668425e-05 -▁Browser 1.4524813013868771e-05 -osa 1.4521024591069119e-05 -▁Pla 1.4521024591069119e-05 -▁Vorgehen 1.4521024591069119e-05 -▁verdient 1.4521024591069119e-05 -ography 1.4517236168269467e-05 -▁700 1.4517236168269467e-05 -▁Northern 1.4513447745469813e-05 -▁corruption 1.4513447745469813e-05 -▁recorded 1.4513447745469813e-05 -will 1.4509659322670161e-05 -▁anzu 1.4509659322670161e-05 -▁geograph 1.4509659322670161e-05 -anne 1.4505870899870508e-05 -asten 1.4505870899870508e-05 --8 1.4502082477070855e-05 -▁crimes 1.4502082477070855e-05 -▁boost 1.4498294054271202e-05 -▁popul 1.4498294054271202e-05 -▁speaker 1.4498294054271202e-05 -▁forum 1.449450563147155e-05 -▁profile 1.449450563147155e-05 -▁rapidly 1.449450563147155e-05 -▁Applause 1.4490717208671896e-05 -▁Women 1.4486928785872244e-05 -▁prem 1.4483140363072592e-05 -▁configuration 1.4479351940272938e-05 -▁Description 1.4475563517473286e-05 -▁usual 1.4471775094673632e-05 -▁verg 1.4471775094673632e-05 -▁arch 1.446798667187398e-05 -▁load 1.446798667187398e-05 -▁sil 1.446798667187398e-05 -▁Belgium 1.4464198249074327e-05 -▁trafficking 1.4464198249074327e-05 -instrument 1.4460409826274675e-05 -▁Ther 1.4460409826274675e-05 -▁Viel 1.4460409826274675e-05 -wege 1.4456621403475022e-05 -▁Alexander 1.4456621403475022e-05 -▁manufacturer 1.4456621403475022e-05 -sb 1.4452832980675369e-05 -istan 1.4449044557875717e-05 -▁Cast 1.4449044557875717e-05 -▁Tibet 1.4445256135076063e-05 -▁letztlich 1.4445256135076063e-05 -▁tackle 1.4445256135076063e-05 -▁85 1.4441467712276411e-05 -▁cycle 1.4433890866677105e-05 -▁grundsätzlich 1.4433890866677105e-05 -etic 1.44263140210778e-05 -▁compatible 1.44263140210778e-05 -▁Indian 1.4422525598278147e-05 -▁behalten 1.4422525598278147e-05 -licht 1.4418737175478494e-05 -▁einfacher 1.4414948752678842e-05 -▁ident 1.4414948752678842e-05 -▁pre- 1.4414948752678842e-05 -▁gallery 1.4411160329879188e-05 -▁liber 1.4411160329879188e-05 -▁territorial 1.4411160329879188e-05 -▁verurteilt 1.4411160329879188e-05 -▁1980 1.4407371907079536e-05 -▁Cam 1.4407371907079536e-05 -▁Sollte 1.4407371907079536e-05 -▁competent 1.4403583484279882e-05 -▁exit 1.4403583484279882e-05 -▁neun 1.439979506148023e-05 -▁woman 1.439979506148023e-05 -kräfte 1.4396006638680578e-05 -▁yes 1.4396006638680578e-05 -▁Soviet 1.4392218215880924e-05 -95 1.4388429793081272e-05 -▁assume 1.4388429793081272e-05 -▁Mach 1.4384641370281618e-05 -▁Email 1.4380852947481966e-05 -▁Insbesondere 1.4380852947481966e-05 -▁Pers 1.4380852947481966e-05 -▁berechtigt 1.4380852947481966e-05 -▁herr 1.4380852947481966e-05 -▁proven 1.4380852947481966e-05 -▁immigrants 1.4377064524682313e-05 -▁Desk 1.4373276101882661e-05 -▁Rezeption 1.4373276101882661e-05 -▁desired 1.4373276101882661e-05 -▁assets 1.4365699256283355e-05 --Projekt 1.4361910833483703e-05 -▁erf 1.4361910833483703e-05 -urlaub 1.4354333987884397e-05 -▁Selbstverständlich 1.4354333987884397e-05 -akes 1.4350545565084743e-05 -▁Besorgnis 1.4350545565084743e-05 -▁honour 1.4350545565084743e-05 -beziehungen 1.4346757142285091e-05 -▁Bord 1.4346757142285091e-05 -▁Dienstleistungs 1.4346757142285091e-05 -▁Oder 1.4346757142285091e-05 -▁Wi-Fi 1.4346757142285091e-05 -stands 1.4342968719485437e-05 -▁Respekt 1.4342968719485437e-05 -▁Werkzeug 1.4342968719485437e-05 -▁overcome 1.4342968719485437e-05 -▁respected 1.4342968719485437e-05 -TO 1.4339180296685785e-05 -last 1.4339180296685785e-05 -üb 1.4339180296685785e-05 -▁HTML 1.4335391873886133e-05 -▁gewährt 1.4335391873886133e-05 -▁routes 1.433160345108648e-05 -▁0, 1.4327815028286828e-05 -▁Büro 1.4327815028286828e-05 -▁Ann 1.4324026605487174e-05 -▁Arm 1.4324026605487174e-05 -▁constitution 1.4324026605487174e-05 -▁Mangel 1.4320238182687522e-05 -▁containing 1.4320238182687522e-05 -▁nights 1.4320238182687522e-05 -▁verbinden 1.4320238182687522e-05 -fläche 1.4316449759887868e-05 -▁Autobahn 1.4316449759887868e-05 -▁nunmehr 1.4316449759887868e-05 -▁Us 1.4312661337088216e-05 -ы 1.4308872914288564e-05 -▁Optionen 1.4308872914288564e-05 -▁Untersuchungen 1.4308872914288564e-05 -▁scheme 1.4308872914288564e-05 -▁Gefahren 1.430508449148891e-05 -▁Phil 1.430508449148891e-05 -▁TO 1.430508449148891e-05 -▁Versand 1.430508449148891e-05 -▁letzter 1.430508449148891e-05 -wahl 1.4301296068689258e-05 -▁52 1.4301296068689258e-05 -▁Apart 1.4301296068689258e-05 -▁Einwanderung 1.4301296068689258e-05 -ckt 1.4297507645889604e-05 -▁einheitlichen 1.4297507645889604e-05 -▁films 1.4297507645889604e-05 -ständ 1.4293719223089952e-05 -▁ersetzt 1.4293719223089952e-05 -▁Preisen 1.4289930800290299e-05 -▁Verbreitung 1.4286142377490647e-05 -▁organisieren 1.4286142377490647e-05 -▁pla 1.4286142377490647e-05 -unch 1.4282353954690995e-05 -▁BMW 1.4282353954690995e-05 -▁Salzburg 1.4282353954690995e-05 -▁Single 1.4282353954690995e-05 -▁rechtlichen 1.4282353954690995e-05 -achs 1.4278565531891341e-05 -organisationen 1.4278565531891341e-05 -▁Given 1.4278565531891341e-05 -porte 1.4274777109091689e-05 -scht 1.4274777109091689e-05 -▁Marktes 1.4270988686292035e-05 -▁alcohol 1.4270988686292035e-05 -▁correctly 1.4270988686292035e-05 -▁Rob 1.4267200263492383e-05 -▁beschäftigen 1.4267200263492383e-05 -▁installieren 1.4267200263492383e-05 -uv 1.4263411840692729e-05 -▁replace 1.4263411840692729e-05 -▁Lager 1.4259623417893077e-05 -▁credibility 1.4259623417893077e-05 -▁entschlossen 1.4259623417893077e-05 -iot 1.4255834995093423e-05 -▁Komponenten 1.4255834995093423e-05 -▁enjoyed 1.4255834995093423e-05 -▁partly 1.4255834995093423e-05 -eds 1.4252046572293771e-05 -rochen 1.4252046572293771e-05 -▁Pen 1.4252046572293771e-05 -▁Tit 1.4252046572293771e-05 -▁Var 1.4248258149494119e-05 -ju 1.4244469726694466e-05 -▁Kapitel 1.4244469726694466e-05 -▁facts 1.4244469726694466e-05 -▁sit 1.4244469726694466e-05 -ı 1.4240681303894814e-05 -▁Energien 1.4240681303894814e-05 -▁concert 1.4240681303894814e-05 -▁elect 1.4240681303894814e-05 -▁Fehl 1.423689288109516e-05 -▁Zielen 1.423689288109516e-05 -▁imposed 1.423689288109516e-05 -▁pract 1.423689288109516e-05 -▁gespielt 1.4233104458295508e-05 -▁individuelle 1.4233104458295508e-05 -▁sending 1.4233104458295508e-05 -aktion 1.4229316035495854e-05 -▁constitutional 1.4229316035495854e-05 -genommen 1.4225527612696202e-05 -irs 1.4225527612696202e-05 -▁direkte 1.4225527612696202e-05 -zentren 1.422173918989655e-05 -▁Nachbar 1.422173918989655e-05 -▁att 1.422173918989655e-05 -▁centuries 1.422173918989655e-05 -▁gestaltet 1.422173918989655e-05 -▁luxurious 1.422173918989655e-05 -ück 1.4217950767096896e-05 -▁corresponding 1.4217950767096896e-05 -bekämpfung 1.4214162344297244e-05 -▁Zwecke 1.4214162344297244e-05 -▁erfolgreiche 1.4214162344297244e-05 -▁manager 1.4214162344297244e-05 -eto 1.421037392149759e-05 -problem 1.421037392149759e-05 -▁transactions 1.421037392149759e-05 -▁vorhandenen 1.421037392149759e-05 -▁vorstellen 1.4206585498697938e-05 -▁Malta 1.4202797075898285e-05 -▁arrived 1.4202797075898285e-05 -vertrag 1.4199008653098633e-05 -▁Bu 1.4199008653098633e-05 -▁subsidiarity 1.4199008653098633e-05 -▁appointed 1.4195220230298979e-05 -hips 1.4191431807499327e-05 -▁lobby 1.4191431807499327e-05 -▁pop 1.4191431807499327e-05 -ín 1.4187643384699675e-05 -▁§ 1.4187643384699675e-05 -▁Hungary 1.4183854961900021e-05 -▁SAP 1.4183854961900021e-05 -▁conditioning 1.4183854961900021e-05 -▁ice 1.4183854961900021e-05 -▁weekend 1.4183854961900021e-05 -▁Tre 1.4180066539100369e-05 -▁professionals 1.4172489693501063e-05 -▁Puerto 1.4168701270701409e-05 -▁stronger 1.4168701270701409e-05 -▁14. 1.4164912847901757e-05 -▁Alpen 1.4164912847901757e-05 -▁SD 1.4164912847901757e-05 -▁emerging 1.4164912847901757e-05 -но 1.4161124425102105e-05 -dert 1.4157336002302452e-05 -▁CF 1.4157336002302452e-05 -▁Egypt 1.4157336002302452e-05 -.D 1.41535475795028e-05 -▁Syn 1.41535475795028e-05 -▁marquis 1.41535475795028e-05 -zeichnet 1.4149759156703146e-05 -▁Vel 1.414218231110384e-05 -▁entscheidende 1.414218231110384e-05 -▁individuellen 1.414218231110384e-05 -▁unfair 1.414218231110384e-05 -anzen 1.4138393888304188e-05 -▁Ski 1.4138393888304188e-05 -▁Sche 1.4134605465504534e-05 -▁VAT 1.4134605465504534e-05 -▁Within 1.4134605465504534e-05 -▁aud 1.4134605465504534e-05 -▁maritime 1.4134605465504534e-05 -▁minister 1.4134605465504534e-05 -▁adults 1.4130817042704882e-05 -▁uniform 1.4130817042704882e-05 -play 1.4123240197105576e-05 -▁distributed 1.4123240197105576e-05 -▁inklusive 1.4123240197105576e-05 -▁Gaza 1.4119451774305924e-05 -▁Politiken 1.4119451774305924e-05 -UNG 1.4115663351506271e-05 -ility 1.4115663351506271e-05 -▁Fitness 1.4115663351506271e-05 -gegangen 1.4111874928706619e-05 -▁Kräfte 1.4111874928706619e-05 -▁Lim 1.4111874928706619e-05 -▁jenen 1.4111874928706619e-05 -itors 1.4108086505906965e-05 -▁Ban 1.4108086505906965e-05 -▁Following 1.4108086505906965e-05 -▁functional 1.4108086505906965e-05 -asst 1.4104298083107313e-05 -▁2013 1.4104298083107313e-05 -▁beruht 1.4104298083107313e-05 -▁lots 1.4104298083107313e-05 -▁Kirk 1.4100509660307661e-05 -▁Met 1.4100509660307661e-05 -▁Tip 1.4100509660307661e-05 -▁außerordentlich 1.4100509660307661e-05 -▁satisfied 1.4100509660307661e-05 -▁fuer 1.4096721237508007e-05 -▁verst 1.4092932814708355e-05 -▁Event 1.4089144391908701e-05 -stre 1.4085355969109049e-05 -▁Serie 1.4085355969109049e-05 -eral 1.4077779123509743e-05 -▁Vorstellung 1.4077779123509743e-05 -▁entsp 1.4077779123509743e-05 -▁Beh 1.4073990700710091e-05 -▁dürfte 1.4073990700710091e-05 -▁fir 1.4073990700710091e-05 -▁ze 1.4073990700710091e-05 -▁declared 1.4070202277910438e-05 -▁functioning 1.4070202277910438e-05 -ни 1.4066413855110786e-05 -▁ursprünglich 1.4066413855110786e-05 -▁britische 1.4062625432311132e-05 -▁destruction 1.4062625432311132e-05 -anst 1.405883700951148e-05 -▁adopting 1.405883700951148e-05 -▁Staff 1.4055048586711826e-05 -frei 1.4051260163912174e-05 -ommen 1.4051260163912174e-05 -▁Bemerkung 1.4051260163912174e-05 -▁unterschiedlich 1.4051260163912174e-05 -▁Freunden 1.404747174111252e-05 -▁Client 1.4043683318312868e-05 -beh 1.4039894895513216e-05 -▁Spaß 1.4039894895513216e-05 -▁Ho 1.4036106472713563e-05 -▁judicial 1.4036106472713563e-05 -▁unverzüglich 1.403231804991391e-05 -imal 1.4028529627114257e-05 -▁Kab 1.4028529627114257e-05 -▁Structural 1.4028529627114257e-05 -▁195 1.4024741204314605e-05 -▁Institut 1.4024741204314605e-05 -▁integr 1.4020952781514951e-05 -▁magnificent 1.4017164358715299e-05 -uro 1.4013375935915647e-05 -▁slightly 1.4009587513115993e-05 -▁Pra 1.4005799090316341e-05 -▁Wil 1.4005799090316341e-05 -▁col 1.4005799090316341e-05 -pos 1.4002010667516687e-05 -▁Water 1.4002010667516687e-05 -▁aussprechen 1.4002010667516687e-05 -▁died 1.3994433821917382e-05 -▁globalisation 1.3994433821917382e-05 -▁rental 1.399064539911773e-05 -▁· 1.399064539911773e-05 -etting 1.3986856976318076e-05 -▁Städten 1.3986856976318076e-05 -▁reduzieren 1.3986856976318076e-05 -▁ruhig 1.3986856976318076e-05 -▁shares 1.3986856976318076e-05 -▁simpl 1.3979280130718772e-05 -zusch 1.3975491707919118e-05 -▁Kom 1.3975491707919118e-05 -▁attract 1.3975491707919118e-05 -▁nu 1.3975491707919118e-05 -zig 1.3971703285119466e-05 -▁stehenden 1.3971703285119466e-05 -▁bright 1.3967914862319812e-05 -▁prakt 1.3967914862319812e-05 -▁fehlt 1.396412643952016e-05 -ziel 1.3960338016720506e-05 -▁Werke 1.3960338016720506e-05 -▁zusammenarbeiten 1.3960338016720506e-05 -echnik 1.3956549593920854e-05 -▁Dre 1.3956549593920854e-05 -▁Kons 1.3956549593920854e-05 -▁courts 1.3952761171121202e-05 -▁inst 1.3952761171121202e-05 -▁schlägt 1.3948972748321549e-05 -▁histor 1.3945184325521897e-05 -einrichtungen 1.3941395902722243e-05 -emen 1.3941395902722243e-05 -▁verstärken 1.3941395902722243e-05 -▁suppliers 1.3937607479922591e-05 -▁Ausmaß 1.3933819057122937e-05 -▁Sav 1.3933819057122937e-05 -▁grant 1.3933819057122937e-05 -▁angegebenen 1.3930030634323285e-05 -▁Hal 1.3926242211523633e-05 -▁Mitarbeitern 1.3926242211523633e-05 -▁eastern 1.3926242211523633e-05 -▁Betrag 1.3922453788723979e-05 -ield 1.3918665365924327e-05 -versorgung 1.3918665365924327e-05 -mus 1.3914876943124673e-05 -rans 1.3914876943124673e-05 -▁Auge 1.3914876943124673e-05 -▁Schönheit 1.3914876943124673e-05 -▁chemical 1.3914876943124673e-05 -▁extraordinary 1.3914876943124673e-05 -▁ließ 1.3914876943124673e-05 -▁milk 1.3914876943124673e-05 -▁eingereicht 1.3911088520325021e-05 -▁miles 1.3911088520325021e-05 -▁verkaufen 1.3911088520325021e-05 -▁Erzeugnisse 1.3907300097525368e-05 -▁kamen 1.3903511674725716e-05 -▁klaren 1.3892146406326758e-05 -ae 1.3888357983527104e-05 -ь 1.3888357983527104e-05 -▁Verlauf 1.3888357983527104e-05 -▁Kir 1.3884569560727452e-05 -▁drop 1.3884569560727452e-05 -leistung 1.3876992715128146e-05 -▁sing 1.3876992715128146e-05 -sam 1.3873204292328492e-05 -▁Kopenhagen 1.3873204292328492e-05 -vi 1.386941586952884e-05 -▁Who 1.3861839023929535e-05 -35 1.3858050601129883e-05 -▁addresses 1.3858050601129883e-05 -▁lists 1.3858050601129883e-05 -▁considering 1.3854262178330229e-05 -▁disp 1.3854262178330229e-05 -▁taxation 1.3854262178330229e-05 -▁Hamas 1.3850473755530577e-05 -▁plays 1.3850473755530577e-05 -olution 1.3846685332730923e-05 -▁Sunday 1.3842896909931271e-05 -▁Wohlstand 1.3842896909931271e-05 -▁domain 1.3842896909931271e-05 -DA 1.3839108487131617e-05 -▁treat 1.3839108487131617e-05 -65 1.3835320064331965e-05 -▁Aktions 1.3835320064331965e-05 -▁consent 1.3835320064331965e-05 -▁identified 1.3835320064331965e-05 -▁Mobile 1.3827743218732659e-05 -▁TH 1.3827743218732659e-05 -▁encouraging 1.3827743218732659e-05 -▁proposes 1.3827743218732659e-05 -eilen 1.3823954795933007e-05 -▁Abkommens 1.3816377950333702e-05 -▁cent 1.3816377950333702e-05 -rieben 1.3812589527534048e-05 -▁Gesetzes 1.3812589527534048e-05 -▁preparing 1.3812589527534048e-05 -prozesses 1.3808801104734396e-05 -▁Balkans 1.3808801104734396e-05 -▁formed 1.3805012681934744e-05 -▁russischen 1.3805012681934744e-05 -▁Gericht 1.3797435836335438e-05 -▁hundred 1.3797435836335438e-05 -▁wiederholen 1.3797435836335438e-05 -light 1.3793647413535784e-05 -▁Stimmen 1.3793647413535784e-05 -bach 1.3789858990736132e-05 -▁Bea 1.3786070567936478e-05 -▁Sektoren 1.3786070567936478e-05 -▁honest 1.3786070567936478e-05 -▁plenary 1.3786070567936478e-05 -▁pal 1.3782282145136826e-05 --to 1.3778493722337173e-05 -▁substance 1.3778493722337173e-05 -osion 1.3774705299537521e-05 -▁providers 1.3774705299537521e-05 -geräte 1.3767128453938215e-05 -▁Aqu 1.3763340031138563e-05 -▁Wetter 1.3763340031138563e-05 -lau 1.3759551608338909e-05 -standing 1.3759551608338909e-05 -regeln 1.3755763185539257e-05 -▁Newsletter 1.3755763185539257e-05 -▁Charles 1.3751974762739603e-05 -▁mini 1.3751974762739603e-05 -▁abs 1.3748186339939951e-05 -▁ergänzen 1.3748186339939951e-05 -▁resolved 1.3748186339939951e-05 -▁suit 1.3748186339939951e-05 -▁ensures 1.37443979171403e-05 -schließen 1.3736821071540993e-05 -wendung 1.3736821071540993e-05 -▁Parkplatz 1.3736821071540993e-05 -▁didn 1.3736821071540993e-05 -▁erhielt 1.3736821071540993e-05 -ede 1.373303264874134e-05 -▁Without 1.373303264874134e-05 -▁sensible 1.373303264874134e-05 -fort 1.3729244225941688e-05 -ially 1.3729244225941688e-05 -▁Ga 1.3729244225941688e-05 -▁jener 1.3729244225941688e-05 -▁vern 1.3729244225941688e-05 -▁äußern 1.3725455803142034e-05 -immt 1.371787895754273e-05 -▁Verd 1.3714090534743076e-05 -▁erfolgreichen 1.3710302111943424e-05 -▁Thirdly 1.3702725266344118e-05 -kurs 1.3698936843544464e-05 -paket 1.3698936843544464e-05 -▁Männern 1.3695148420744812e-05 -▁everyday 1.3695148420744812e-05 -▁vorbereitet 1.3695148420744812e-05 -▁Investment 1.3691359997945159e-05 -▁Programmen 1.3691359997945159e-05 -▁hol 1.3687571575145507e-05 -▁pur 1.3687571575145507e-05 -▁roads 1.3687571575145507e-05 -ikanische 1.3683783152345855e-05 -▁Charta 1.3683783152345855e-05 -▁Exchange 1.3683783152345855e-05 -▁Tochter 1.3683783152345855e-05 -▁erhebliche 1.3683783152345855e-05 -▁Kontrollen 1.3679994729546201e-05 -▁vermitteln 1.3679994729546201e-05 -▁army 1.3676206306746549e-05 -informationen 1.3672417883946895e-05 -▁Massen 1.3672417883946895e-05 -▁artistic 1.3672417883946895e-05 -yt 1.3668629461147243e-05 -▁Schulen 1.3668629461147243e-05 -▁U-Bahn 1.3668629461147243e-05 -▁drinks 1.3668629461147243e-05 -atik 1.3664841038347589e-05 -▁Bereitstellung 1.3664841038347589e-05 -▁actively 1.3664841038347589e-05 -▁western 1.3664841038347589e-05 -ört 1.3661052615547937e-05 -IX 1.3657264192748285e-05 -▁Güter 1.3657264192748285e-05 -▁Schlussfolgerungen 1.3653475769948631e-05 -▁component 1.3653475769948631e-05 -▁Schn 1.3645898924349326e-05 -▁bestimmter 1.3645898924349326e-05 -icherung 1.3642110501549674e-05 -▁Lieferung 1.3642110501549674e-05 -▁Milch 1.3642110501549674e-05 -▁Satz 1.3642110501549674e-05 -▁Zusätzlich 1.3642110501549674e-05 -▁Fundamental 1.363832207875002e-05 -▁vieles 1.363832207875002e-05 -kes 1.3634533655950368e-05 -nd 1.3634533655950368e-05 -▁pe 1.3634533655950368e-05 -▁updates 1.3634533655950368e-05 -mit 1.3626956810351062e-05 -▁Bek 1.3626956810351062e-05 -hre 1.362316838755141e-05 -itionen 1.362316838755141e-05 -ires 1.3619379964751756e-05 -hood 1.3615591541952104e-05 -▁с 1.3615591541952104e-05 -PC 1.3611803119152451e-05 -bene 1.3611803119152451e-05 -icted 1.3611803119152451e-05 -ра 1.3611803119152451e-05 -▁Traum 1.3611803119152451e-05 -▁Valley 1.3611803119152451e-05 -▁eventually 1.3611803119152451e-05 -▁thousand 1.3611803119152451e-05 -▁Gegenstand 1.3608014696352798e-05 -▁empfangen 1.3608014696352798e-05 -▁inclusion 1.3608014696352798e-05 -▁nahm 1.3608014696352798e-05 -▁Cup 1.3604226273553145e-05 -▁assure 1.3604226273553145e-05 -▁deposit 1.3600437850753493e-05 -führt 1.3596649427953841e-05 -▁Assembly 1.3596649427953841e-05 -ivität 1.3592861005154187e-05 -▁adjust 1.3592861005154187e-05 -▁enhance 1.3592861005154187e-05 -dienste 1.3589072582354535e-05 -▁birth 1.3589072582354535e-05 -▁division 1.3589072582354535e-05 -▁strategischen 1.3589072582354535e-05 -▁Add 1.3585284159554881e-05 -gesellschaften 1.3577707313955575e-05 -▁Apple 1.3577707313955575e-05 -▁Chinas 1.3577707313955575e-05 -▁Einstellungen 1.3577707313955575e-05 -▁beigetragen 1.3577707313955575e-05 -eak 1.357013046835627e-05 -other 1.357013046835627e-05 -▁voran 1.3566342045556618e-05 -▁Krankheit 1.3558765199957312e-05 -▁topics 1.3558765199957312e-05 -sprache 1.355497677715766e-05 -ßen 1.355497677715766e-05 -▁proportion 1.3547399931558354e-05 -▁traditionelle 1.3547399931558354e-05 -▁Verbraucherschutz 1.3536034663159396e-05 -▁Genehmigung 1.3532246240359742e-05 -ologische 1.352845781756009e-05 -▁Ram 1.352845781756009e-05 -▁concepts 1.352845781756009e-05 -formen 1.3524669394760437e-05 -ès 1.3524669394760437e-05 -▁Sudan 1.3524669394760437e-05 -▁upper 1.3524669394760437e-05 -law 1.3520880971960785e-05 -werken 1.3517092549161131e-05 -▁Content 1.3517092549161131e-05 -▁musical 1.3517092549161131e-05 -betrieb 1.3513304126361479e-05 -nern 1.3513304126361479e-05 -▁Sens 1.3513304126361479e-05 -▁arr 1.3513304126361479e-05 -▁euros 1.3513304126361479e-05 -stud 1.3509515703561827e-05 -maschinen 1.3505727280762173e-05 -▁eingehalten 1.3505727280762173e-05 -▁decisive 1.3501938857962521e-05 -▁elsewhere 1.3501938857962521e-05 -▁Kabel 1.3498150435162867e-05 -▁Vide 1.3498150435162867e-05 -▁amerikanische 1.3498150435162867e-05 -▁EF 1.3494362012363215e-05 -▁bek 1.3494362012363215e-05 -▁university 1.3494362012363215e-05 -igent 1.3490573589563561e-05 -ques 1.3490573589563561e-05 -▁Next 1.3490573589563561e-05 -▁duties 1.3490573589563561e-05 -kn 1.3486785166763909e-05 -α 1.3486785166763909e-05 -▁CH 1.3486785166763909e-05 -▁Import 1.3486785166763909e-05 -▁großem 1.3486785166763909e-05 -▁toler 1.3486785166763909e-05 -▁TR 1.3482996743964256e-05 -▁kleines 1.3482996743964256e-05 -▁Thursday 1.3479208321164604e-05 -▁Vill 1.3479208321164604e-05 -▁supplies 1.3479208321164604e-05 -▁Rechtsgrundlage 1.3475419898364952e-05 -▁auseinander 1.3475419898364952e-05 -▁bea 1.3475419898364952e-05 -▁spoke 1.3475419898364952e-05 -▁wunderschönen 1.3475419898364952e-05 -deut 1.3467843052765646e-05 -urz 1.3467843052765646e-05 -б 1.3467843052765646e-05 -п 1.3467843052765646e-05 -▁langsam 1.3467843052765646e-05 -▁lokale 1.3467843052765646e-05 -▁ACP 1.346026620716634e-05 -rank 1.3456477784366686e-05 -ifies 1.3452689361567034e-05 -▁Kong 1.3452689361567034e-05 -▁anwesend 1.3452689361567034e-05 --minute 1.3448900938767382e-05 -each 1.3448900938767382e-05 -enter 1.3448900938767382e-05 -ugh 1.3448900938767382e-05 -▁Bedrohung 1.3445112515967728e-05 -▁Professor 1.3445112515967728e-05 -▁Schne 1.3445112515967728e-05 -▁beteiligen 1.3445112515967728e-05 -▁Gest 1.3441324093168076e-05 -▁Inf 1.3441324093168076e-05 -▁deficit 1.3441324093168076e-05 -▁gewiss 1.3437535670368423e-05 -▁pick 1.3437535670368423e-05 -▁Mini 1.3433747247568771e-05 -▁om 1.3433747247568771e-05 -▁vulnerable 1.3433747247568771e-05 -▁Kongress 1.3429958824769117e-05 -▁hart 1.3429958824769117e-05 -▁Klim 1.3422381979169811e-05 -▁erzeugt 1.3422381979169811e-05 -▁proposing 1.3422381979169811e-05 -▁subjects 1.3418593556370159e-05 -endo 1.3411016710770853e-05 -▁Gem 1.3411016710770853e-05 -▁Zus 1.3411016710770853e-05 -▁applying 1.3411016710770853e-05 -oche 1.3407228287971201e-05 -▁Atom 1.3407228287971201e-05 -gemeinschaft 1.3403439865171547e-05 -lung 1.3403439865171547e-05 -▁Pay 1.3403439865171547e-05 -▁absence 1.3403439865171547e-05 -▁aktivieren 1.3403439865171547e-05 -▁handle 1.3403439865171547e-05 -▁Among 1.3399651442371895e-05 -▁Weiß 1.3399651442371895e-05 -MA 1.3395863019572242e-05 -omin 1.3395863019572242e-05 -ories 1.3395863019572242e-05 -▁lessons 1.3395863019572242e-05 -▁solely 1.3395863019572242e-05 -▁stolz 1.3395863019572242e-05 -▁einzigartigen 1.339207459677259e-05 -airs 1.3388286173972938e-05 -reiche 1.3388286173972938e-05 -▁bislang 1.3388286173972938e-05 -apt 1.3384497751173284e-05 -▁Extrem 1.3384497751173284e-05 -▁packaging 1.3384497751173284e-05 -▁theory 1.3384497751173284e-05 -▁Flugzeug 1.3380709328373632e-05 -▁beobachten 1.3376920905573978e-05 -▁countryside 1.3376920905573978e-05 -▁nevertheless 1.3376920905573978e-05 -acc 1.3373132482774326e-05 -▁Fußball 1.3373132482774326e-05 -▁generated 1.3373132482774326e-05 -▁lit 1.3373132482774326e-05 -▁shock 1.3373132482774326e-05 -▁voluntary 1.3373132482774326e-05 -ski 1.336555563717502e-05 -.net 1.3361767214375368e-05 -affen 1.3361767214375368e-05 -ennung 1.3361767214375368e-05 -▁Federation 1.3361767214375368e-05 -▁elektronischen 1.3361767214375368e-05 -▁Amt 1.3354190368776062e-05 -▁fourth 1.3354190368776062e-05 -gebiete 1.3350401945976409e-05 -▁carrying 1.3350401945976409e-05 -▁Erstellung 1.3346613523176757e-05 -▁Pil 1.3346613523176757e-05 -▁Ren 1.3346613523176757e-05 -▁Like 1.3342825100377103e-05 -▁alarm 1.3342825100377103e-05 -▁Gerichts 1.3339036677577451e-05 -iern 1.3335248254777797e-05 -▁shipping 1.3335248254777797e-05 -▁tobacco 1.3335248254777797e-05 -aya 1.3331459831978145e-05 -elli 1.3331459831978145e-05 -▁suggestions 1.3331459831978145e-05 -▁appeared 1.3327671409178493e-05 -▁info 1.3327671409178493e-05 -astic 1.3323882986378839e-05 -astr 1.3323882986378839e-05 -▁Kaffee 1.3323882986378839e-05 -▁disk 1.3320094563579187e-05 -geschäft 1.3316306140779533e-05 -▁Sicherung 1.3316306140779533e-05 -▁legitimate 1.3316306140779533e-05 -▁Baum 1.3312517717979881e-05 -▁Castle 1.3312517717979881e-05 -▁attach 1.3312517717979881e-05 -▁entscheidender 1.3312517717979881e-05 -▁Holiday 1.3308729295180228e-05 -▁pure 1.3308729295180228e-05 -▁villages 1.3308729295180228e-05 -oz 1.3304940872380576e-05 -▁Glaubwürdigkeit 1.3304940872380576e-05 -▁OR 1.3304940872380576e-05 -▁komme 1.3304940872380576e-05 -▁Os 1.3301152449580924e-05 -▁description 1.3301152449580924e-05 -gegeben 1.329736402678127e-05 -▁betreffend 1.329736402678127e-05 -▁departure 1.329736402678127e-05 -▁excessive 1.329736402678127e-05 -▁cop 1.3293575603981618e-05 -▁54 1.3289787181181964e-05 -▁Monday 1.3289787181181964e-05 -▁Sind 1.3289787181181964e-05 -▁Freitag 1.3285998758382312e-05 -▁Travel 1.3285998758382312e-05 -▁liefert 1.3282210335582658e-05 -IF 1.3278421912783006e-05 -dorf 1.3278421912783006e-05 -racht 1.3278421912783006e-05 -▁ausgerichtet 1.3278421912783006e-05 -▁impose 1.3278421912783006e-05 -▁wrote 1.3274633489983352e-05 -▁Halb 1.32708450671837e-05 -▁verkauft 1.32708450671837e-05 -▁Besides 1.3267056644384048e-05 -▁greatly 1.3267056644384048e-05 -▁Hat 1.3263268221584395e-05 -▁wert 1.3263268221584395e-05 -▁Turn 1.3259479798784743e-05 -▁channel 1.3259479798784743e-05 -ě 1.3255691375985089e-05 -▁gearbeitet 1.3251902953185437e-05 -▁liked 1.3251902953185437e-05 -garten 1.3248114530385783e-05 -▁Bulgarien 1.3248114530385783e-05 -▁director 1.3248114530385783e-05 -ueller 1.3244326107586131e-05 -▁della 1.3244326107586131e-05 -▁französische 1.3244326107586131e-05 -▁co- 1.3240537684786479e-05 -▁verbindet 1.3240537684786479e-05 -▁Mission 1.3236749261986825e-05 -▁Wohnung 1.3236749261986825e-05 -▁genannte 1.3236749261986825e-05 -▁XP 1.3232960839187173e-05 -▁fa 1.3232960839187173e-05 -▁Sohn 1.3229172416387519e-05 -▁finish 1.3229172416387519e-05 --related 1.3225383993587867e-05 -▁Mr. 1.3225383993587867e-05 -▁philosophy 1.3225383993587867e-05 -▁tro 1.3225383993587867e-05 -▁Geschwindigkeit 1.3221595570788214e-05 -▁MP3 1.3221595570788214e-05 -▁Schon 1.3221595570788214e-05 -▁abuse 1.3221595570788214e-05 -▁unterb 1.3221595570788214e-05 -prof 1.3217807147988562e-05 -▁Hungarian 1.3214018725188908e-05 -▁deutsch 1.3214018725188908e-05 -▁ships 1.3214018725188908e-05 -▁Darum 1.3210230302389256e-05 -▁offiziellen 1.3210230302389256e-05 -▁repair 1.3210230302389256e-05 -quis 1.3206441879589604e-05 -▁Ansichten 1.3206441879589604e-05 -▁Common 1.3206441879589604e-05 -▁Lam 1.320265345678995e-05 -▁Montag 1.320265345678995e-05 -▁desk 1.320265345678995e-05 -▁schicken 1.320265345678995e-05 -rog 1.3198865033990298e-05 -▁Staates 1.3198865033990298e-05 -▁Konsequenzen 1.3195076611190644e-05 -▁tw 1.3195076611190644e-05 -idad 1.3191288188390992e-05 -▁solches 1.3191288188390992e-05 -▁gained 1.3183711342791686e-05 -▁Geh 1.3176134497192381e-05 -▁Tri 1.3172346074392729e-05 -▁anbelangt 1.3172346074392729e-05 -äft 1.3168557651593075e-05 -▁Gang 1.3168557651593075e-05 -▁Kroatien 1.3168557651593075e-05 -han 1.3164769228793423e-05 -oke 1.3164769228793423e-05 -▁Date 1.3164769228793423e-05 -▁bereitgestellt 1.3164769228793423e-05 -▁Quality 1.3160980805993769e-05 -▁supplier 1.3160980805993769e-05 -▁teams 1.3160980805993769e-05 -cess 1.3153403960394465e-05 -▁exports 1.3153403960394465e-05 -▁zuständig 1.3153403960394465e-05 -▁decor 1.3149615537594811e-05 -▁ou 1.3149615537594811e-05 -▁song 1.3149615537594811e-05 -▁Hom 1.3145827114795159e-05 -▁Jul 1.3145827114795159e-05 -ird 1.3142038691995506e-05 -▁Fahrt 1.3142038691995506e-05 -▁Coll 1.3138250269195853e-05 -▁Anlass 1.31344618463962e-05 -▁Nutz 1.31344618463962e-05 -▁housing 1.31344618463962e-05 --I 1.3130673423596548e-05 -▁Übergangs 1.3130673423596548e-05 -▁wiederholt 1.3126885000796894e-05 -omat 1.3123096577997242e-05 -▁AIDS 1.3123096577997242e-05 -▁Guests 1.3123096577997242e-05 -▁Sterne 1.3123096577997242e-05 -▁erl 1.3123096577997242e-05 -icate 1.311930815519759e-05 -▁delicious 1.311930815519759e-05 -▁iPhone 1.311930815519759e-05 -schule 1.3115519732397936e-05 -▁interpretation 1.3115519732397936e-05 -▁sozialer 1.3115519732397936e-05 -▁wirtschaft 1.3115519732397936e-05 -▁Überlegungen 1.3115519732397936e-05 -▁gere 1.310794288679863e-05 -▁reservations 1.310794288679863e-05 -▁Agriculture 1.3104154463998978e-05 -▁Having 1.3104154463998978e-05 -▁trial 1.3104154463998978e-05 -▁Mur 1.3100366041199325e-05 -▁Head 1.3096577618399673e-05 -▁sounds 1.3096577618399673e-05 -▁banking 1.309278919560002e-05 -▁beantworten 1.309278919560002e-05 -▁comparison 1.3089000772800367e-05 -▁difficulty 1.3089000772800367e-05 -ning 1.3085212350000715e-05 -▁Gewährleistung 1.3085212350000715e-05 -▁Still 1.3081423927201061e-05 -▁einst 1.3081423927201061e-05 -AV 1.3077635504401409e-05 -Ü 1.3077635504401409e-05 -▁receiving 1.3077635504401409e-05 -▁roof 1.3077635504401409e-05 -▁weist 1.3073847081601755e-05 -▁Beschlüsse 1.3070058658802103e-05 -cies 1.3066270236002449e-05 -▁stammen 1.3066270236002449e-05 -▁Anstieg 1.3062481813202797e-05 -▁Opt 1.3062481813202797e-05 -▁Zivilgesellschaft 1.3062481813202797e-05 -▁doors 1.3062481813202797e-05 -▁sleep 1.3062481813202797e-05 -ume 1.3058693390403145e-05 -▁regarded 1.3058693390403145e-05 -▁ähnlich 1.3058693390403145e-05 -richtung 1.3054904967603492e-05 -jährige 1.305111654480384e-05 -ängen 1.305111654480384e-05 -fluss 1.3047328122004186e-05 -▁Chr 1.3047328122004186e-05 -▁Lux 1.3047328122004186e-05 -▁genügend 1.3047328122004186e-05 -hör 1.3043539699204534e-05 -▁innen 1.3043539699204534e-05 -▁Herbst 1.303975127640488e-05 -▁Kollegin 1.303975127640488e-05 -▁specified 1.303975127640488e-05 -▁Rio 1.3035962853605228e-05 -geh 1.3032174430805576e-05 -nummer 1.3032174430805576e-05 -umm 1.3032174430805576e-05 -▁module 1.3032174430805576e-05 -ista 1.302459758520627e-05 -aming 1.3020809162406616e-05 -ifik 1.3020809162406616e-05 -▁Mitgliedschaft 1.3020809162406616e-05 -▁invited 1.3020809162406616e-05 -maß 1.3017020739606964e-05 -sektor 1.3017020739606964e-05 -üh 1.3017020739606964e-05 -▁Eigentum 1.3017020739606964e-05 -▁Schüler 1.3017020739606964e-05 -▁harm 1.3017020739606964e-05 -li 1.3013232316807311e-05 -▁MA 1.3013232316807311e-05 -▁Mrd. 1.3013232316807311e-05 -▁welt 1.3013232316807311e-05 -orf 1.3009443894007659e-05 -▁Jung 1.3009443894007659e-05 -▁Strasbourg 1.3009443894007659e-05 -▁pursue 1.3009443894007659e-05 -▁satisfaction 1.3009443894007659e-05 -▁Direct 1.3005655471208007e-05 -▁inmitten 1.3005655471208007e-05 -▁Hong 1.3001867048408353e-05 -▁rain 1.3001867048408353e-05 -▁corr 1.29980786256087e-05 -▁crew 1.29980786256087e-05 -▁disabled 1.29980786256087e-05 -▁überprüft 1.29980786256087e-05 -▁Potenzial 1.2994290202809047e-05 -▁periods 1.2994290202809047e-05 -▁suggested 1.2994290202809047e-05 -room 1.2990501780009395e-05 -▁abgesch 1.2990501780009395e-05 -▁entspannen 1.2990501780009395e-05 -▁fol 1.2990501780009395e-05 -ui 1.2986713357209741e-05 -▁bu 1.2986713357209741e-05 -▁filter 1.2982924934410089e-05 -▁mittleren 1.2982924934410089e-05 -pol 1.2979136511610435e-05 -▁Budgets 1.2979136511610435e-05 -▁Mov 1.2979136511610435e-05 -▁Positionen 1.2979136511610435e-05 -▁Wissens 1.2979136511610435e-05 -▁colours 1.2975348088810783e-05 -▁grav 1.2975348088810783e-05 -andt 1.2971559666011131e-05 -▁herrscht 1.2971559666011131e-05 -▁manual 1.2971559666011131e-05 -▁Quellen 1.2967771243211478e-05 -▁begrenzt 1.2967771243211478e-05 -▁ernsthaft 1.2967771243211478e-05 -▁letztendlich 1.2967771243211478e-05 -▁wireless 1.2967771243211478e-05 -▁attempts 1.2960194397612172e-05 -▁val 1.295640597481252e-05 -▁wohnen 1.295640597481252e-05 -enkt 1.2952617552012866e-05 -struktur 1.2952617552012866e-05 -▁Finanzierungs 1.2952617552012866e-05 -▁cy 1.2952617552012866e-05 -▁standing 1.2952617552012866e-05 -ô 1.2945040706413562e-05 -▁Eng 1.2945040706413562e-05 -▁edge 1.2945040706413562e-05 -▁forests 1.2945040706413562e-05 -▁overview 1.2945040706413562e-05 -▁Übrigen 1.2945040706413562e-05 -log 1.2941252283613908e-05 -kom 1.2937463860814256e-05 -▁Tennis 1.2937463860814256e-05 -▁mussten 1.2937463860814256e-05 -▁revision 1.2933675438014602e-05 -▁weitgehend 1.2933675438014602e-05 -▁well-known 1.2933675438014602e-05 -wend 1.292988701521495e-05 -▁hinweg 1.2926098592415297e-05 -▁stream 1.2926098592415297e-05 --time 1.2922310169615645e-05 -asts 1.2922310169615645e-05 -erry 1.2922310169615645e-05 -isl 1.2922310169615645e-05 -▁savings 1.2922310169615645e-05 -▁Laut 1.2914733324016339e-05 -▁eingesch 1.2914733324016339e-05 -▁Atlantic 1.2910944901216687e-05 -▁Wählen 1.2910944901216687e-05 -do 1.2907156478417033e-05 -reib 1.2907156478417033e-05 -▁resource 1.2907156478417033e-05 -▁verteidigen 1.2907156478417033e-05 -auschen 1.2903368055617381e-05 -material 1.2903368055617381e-05 -Ä 1.2903368055617381e-05 -▁machte 1.2903368055617381e-05 -▁terror 1.2903368055617381e-05 -▁oper 1.2899579632817727e-05 -▁tennis 1.2899579632817727e-05 -▁Gerechtigkeit 1.2895791210018075e-05 -▁erstmals 1.2895791210018075e-05 -▁script 1.2895791210018075e-05 -ücht 1.2892002787218421e-05 -▁entwickelte 1.2892002787218421e-05 -uting 1.2888214364418769e-05 -▁Hardware 1.2888214364418769e-05 -▁optional 1.2884425941619117e-05 -gänge 1.2880637518819464e-05 -ultur 1.2880637518819464e-05 -▁41 1.2880637518819464e-05 -▁A5-0 1.2880637518819464e-05 -▁Playa 1.2880637518819464e-05 -▁Unternehmer 1.2880637518819464e-05 -▁Erfüllung 1.2873060673220158e-05 -▁Klaus 1.2873060673220158e-05 -▁erlebt 1.2873060673220158e-05 --U 1.2869272250420506e-05 -wahlen 1.2869272250420506e-05 -▁realistic 1.2869272250420506e-05 -▁scene 1.2869272250420506e-05 -▁breiten 1.28616954048212e-05 -▁patient 1.28616954048212e-05 -niveau 1.2857906982021546e-05 -▁fan 1.2854118559221894e-05 -▁gehe 1.2854118559221894e-05 -gebracht 1.2850330136422242e-05 -▁Kulturen 1.2850330136422242e-05 -▁welcoming 1.2850330136422242e-05 -immen 1.2846541713622588e-05 -▁Währungs 1.2846541713622588e-05 -rt 1.2842753290822936e-05 -▁Luxus 1.2842753290822936e-05 -▁betreiben 1.2842753290822936e-05 -▁bridge 1.2842753290822936e-05 -▁passion 1.2842753290822936e-05 -▁Stoffe 1.2838964868023283e-05 -uster 1.2835176445223631e-05 -▁Mak 1.2835176445223631e-05 -▁aktive 1.2835176445223631e-05 -▁Übergang 1.2835176445223631e-05 -irt 1.2831388022423977e-05 -▁Risik 1.2831388022423977e-05 -▁Meister 1.2827599599624325e-05 -▁Pres 1.2827599599624325e-05 -eland 1.2823811176824673e-05 -ience 1.2823811176824673e-05 -▁Eingang 1.2823811176824673e-05 -▁acting 1.2823811176824673e-05 -▁reaction 1.2823811176824673e-05 -▁korrekt 1.2816234331225367e-05 -▁reduziert 1.2816234331225367e-05 -▁Gerade 1.2812445908425713e-05 -▁Situated 1.2808657485626061e-05 -▁Space 1.2808657485626061e-05 -▁arguments 1.2808657485626061e-05 -▁examine 1.2808657485626061e-05 -▁Brasilien 1.2804869062826407e-05 -▁Captain 1.2804869062826407e-05 -▁permitted 1.2804869062826407e-05 -▁Class 1.2801080640026755e-05 -▁federal 1.2801080640026755e-05 -▁rising 1.2801080640026755e-05 -erver 1.279350379442745e-05 -▁allzu 1.279350379442745e-05 -▁confirmation 1.279350379442745e-05 -▁ports 1.279350379442745e-05 -schnitt 1.2785926948828144e-05 -▁Wider 1.2785926948828144e-05 -▁enabling 1.2785926948828144e-05 -steuer 1.2782138526028492e-05 -▁Plus 1.2782138526028492e-05 -▁destroyed 1.2782138526028492e-05 -▁Gehminuten 1.2778350103228838e-05 -▁Passwort 1.2778350103228838e-05 -▁investieren 1.2778350103228838e-05 -etzt 1.2770773257629532e-05 -▁football 1.2770773257629532e-05 -▁wirtschaftlicher 1.2770773257629532e-05 -time 1.276698483482988e-05 -cel 1.2763196412030228e-05 -▁Verletzung 1.2763196412030228e-05 -▁retain 1.2763196412030228e-05 -▁IF 1.2759407989230574e-05 -geschichte 1.2755619566430922e-05 -▁electric 1.2751831143631269e-05 -leitung 1.2748042720831617e-05 -▁Klasse 1.2748042720831617e-05 -behörde 1.2744254298031963e-05 -freie 1.2744254298031963e-05 -▁festival 1.2744254298031963e-05 -▁Reduzierung 1.2740465875232311e-05 -▁betrieben 1.2740465875232311e-05 -▁representation 1.2740465875232311e-05 -▁Aussch 1.2732889029633005e-05 -▁leaves 1.2732889029633005e-05 -▁narrow 1.2732889029633005e-05 -▁religion 1.2732889029633005e-05 -▁© 1.2732889029633005e-05 -piel 1.2729100606833353e-05 -▁1.5 1.2729100606833353e-05 -äum 1.27253121840337e-05 -▁Rea 1.27253121840337e-05 -▁angepasst 1.27253121840337e-05 -▁unforgettable 1.27253121840337e-05 -▁construct 1.2721523761234047e-05 -200 1.2717735338434394e-05 -bedarf 1.2717735338434394e-05 -▁stellte 1.2717735338434394e-05 -inken 1.2713946915634741e-05 -ubs 1.2713946915634741e-05 -uge 1.2713946915634741e-05 -▁könnt 1.2713946915634741e-05 -▁optimale 1.2713946915634741e-05 -eile 1.2710158492835088e-05 -tt 1.2710158492835088e-05 --Er 1.2706370070035436e-05 -▁Korruption 1.2706370070035436e-05 -▁cooperate 1.2706370070035436e-05 -AB 1.2702581647235784e-05 -▁constitute 1.2702581647235784e-05 -▁farming 1.2702581647235784e-05 -▁hierbei 1.2702581647235784e-05 -▁tritt 1.2702581647235784e-05 -▁refers 1.269879322443613e-05 -▁tend 1.269879322443613e-05 -▁Georg 1.2695004801636478e-05 -▁horse 1.2695004801636478e-05 -68 1.2691216378836824e-05 -▁Based 1.2691216378836824e-05 -▁Belgien 1.2691216378836824e-05 -▁bekommt 1.2691216378836824e-05 -car 1.2687427956037172e-05 -▁Family 1.2683639533237518e-05 -▁Liber 1.2683639533237518e-05 -▁Lobby 1.2683639533237518e-05 -▁Vorhaben 1.2683639533237518e-05 -▁obliged 1.2683639533237518e-05 -▁politically 1.2683639533237518e-05 -kontrolle 1.2679851110437866e-05 -▁Lounge 1.2679851110437866e-05 -▁Erd 1.2676062687638214e-05 -▁Investitions 1.2676062687638214e-05 -▁Entlastung 1.2672274264838561e-05 -▁Kontakte 1.2672274264838561e-05 -▁unterscheiden 1.2672274264838561e-05 -▁п 1.2672274264838561e-05 -▁directory 1.2668485842038908e-05 -▁rely 1.2668485842038908e-05 -▁Place 1.2664697419239255e-05 -▁Dist 1.2660908996439603e-05 -▁legitim 1.2660908996439603e-05 -▁Einkommen 1.2657120573639949e-05 -▁gelingt 1.2657120573639949e-05 -▁Islands 1.2653332150840297e-05 -▁führende 1.2653332150840297e-05 -▁Full 1.2649543728040645e-05 -arn 1.2645755305240991e-05 -east 1.2645755305240991e-05 -▁Donnerstag 1.2645755305240991e-05 -▁erwähnen 1.2645755305240991e-05 -▁verschl 1.2645755305240991e-05 -ält 1.2641966882441339e-05 -▁Saison 1.2641966882441339e-05 -▁Stockholm 1.2641966882441339e-05 -üg 1.2638178459641685e-05 -▁Katastrophe 1.2638178459641685e-05 -▁blind 1.2638178459641685e-05 -▁dynam 1.2638178459641685e-05 -▁publication 1.2638178459641685e-05 -uste 1.2634390036842033e-05 -▁Metro 1.2634390036842033e-05 -▁Trotzdem 1.2634390036842033e-05 -▁freundlich 1.2634390036842033e-05 -▁instructions 1.2634390036842033e-05 -▁paying 1.2634390036842033e-05 -ences 1.263060161404238e-05 -▁Kin 1.263060161404238e-05 -▁MP 1.263060161404238e-05 -▁eth 1.263060161404238e-05 -▁versichern 1.263060161404238e-05 -▁Finland 1.2626813191242728e-05 -▁prosperity 1.2626813191242728e-05 -▁ye 1.2623024768443074e-05 -▁zukünftigen 1.2623024768443074e-05 -▁skiing 1.2619236345643422e-05 -▁dry 1.2611659500044116e-05 -▁prof 1.2611659500044116e-05 -anning 1.260408265444481e-05 -▁Grünen 1.260408265444481e-05 -▁Metall 1.260408265444481e-05 -▁retail 1.260408265444481e-05 -▁welcomed 1.260408265444481e-05 -orten 1.2600294231645158e-05 -▁Praktiken 1.2600294231645158e-05 -▁Dafür 1.2596505808845504e-05 -▁Tourism 1.2596505808845504e-05 -▁continuous 1.25889289632462e-05 -▁Konto 1.2585140540446547e-05 -▁renowned 1.2585140540446547e-05 -▁Hor 1.2581352117646895e-05 -▁Liberal 1.2581352117646895e-05 -▁ersetzen 1.2581352117646895e-05 -▁einzusetzen 1.2577563694847241e-05 -por 1.2573775272047589e-05 -▁cleaning 1.2573775272047589e-05 -▁Mö 1.2569986849247935e-05 -▁SE 1.2569986849247935e-05 -▁imports 1.2569986849247935e-05 -▁rare 1.2569986849247935e-05 -▁significance 1.2569986849247935e-05 -pläne 1.2566198426448283e-05 -▁Schiffe 1.2566198426448283e-05 -▁entdeckt 1.2566198426448283e-05 -▁sought 1.2566198426448283e-05 --on 1.2562410003648629e-05 -ESS 1.2562410003648629e-05 -achte 1.2562410003648629e-05 -▁53 1.2562410003648629e-05 -▁killed 1.2562410003648629e-05 -▁Mot 1.2554833158049325e-05 -▁characters 1.2554833158049325e-05 -▁ded 1.2551044735249671e-05 -▁Marke 1.2547256312450019e-05 -▁Prag 1.2547256312450019e-05 -▁einbezogen 1.2547256312450019e-05 -▁trips 1.2547256312450019e-05 -jähr 1.2543467889650366e-05 -▁Refer 1.2539679466850714e-05 -▁Facilities 1.253589104405106e-05 -▁gestattet 1.253589104405106e-05 -▁lunch 1.253589104405106e-05 -anger 1.2532102621251408e-05 -emon 1.2532102621251408e-05 -▁Umsatz 1.2532102621251408e-05 -▁erk 1.2532102621251408e-05 -▁kontaktieren 1.2532102621251408e-05 -▁rank 1.2532102621251408e-05 -▁verteilt 1.2532102621251408e-05 -▁Through 1.2528314198451756e-05 -▁id 1.2528314198451756e-05 -▁societies 1.2528314198451756e-05 -▁umfassend 1.2528314198451756e-05 -ided 1.2524525775652102e-05 -uben 1.2524525775652102e-05 -▁Erlebnis 1.2524525775652102e-05 -▁harmonisation 1.2524525775652102e-05 -▁herunter 1.2524525775652102e-05 -▁legally 1.2524525775652102e-05 -▁manche 1.2524525775652102e-05 -cles 1.252073735285245e-05 -ibly 1.252073735285245e-05 -▁LT 1.252073735285245e-05 -▁beendet 1.252073735285245e-05 -▁dependent 1.252073735285245e-05 -▁freedoms 1.252073735285245e-05 -▁institutionellen 1.252073735285245e-05 -aßt 1.2516948930052796e-05 -estr 1.2516948930052796e-05 -▁CAP 1.2516948930052796e-05 -▁Wand 1.2516948930052796e-05 -▁verfügbaren 1.2516948930052796e-05 -eckt 1.2513160507253144e-05 -▁Schle 1.2513160507253144e-05 -rates 1.250937208445349e-05 -▁radical 1.250937208445349e-05 -uppen 1.2505583661653838e-05 -▁04 1.2505583661653838e-05 -▁Vo 1.2505583661653838e-05 -▁anschließend 1.2505583661653838e-05 -▁loc 1.2505583661653838e-05 -▁Pir 1.2501795238854185e-05 -▁Bio 1.2498006816054533e-05 -▁Erm 1.2498006816054533e-05 -▁Schrift 1.2498006816054533e-05 -▁behaupten 1.2498006816054533e-05 -▁justified 1.2494218393254881e-05 -▁deswegen 1.2490429970455227e-05 -▁getragen 1.2490429970455227e-05 -▁withdraw 1.2490429970455227e-05 -emin 1.2482853124855921e-05 -endung 1.2482853124855921e-05 -▁Game 1.2482853124855921e-05 -▁videos 1.2482853124855921e-05 -GB 1.2479064702056269e-05 -techn 1.2475276279256615e-05 -▁Prozesse 1.2475276279256615e-05 -▁bedeutende 1.2475276279256615e-05 -arge 1.2471487856456963e-05 -▁referendum 1.2471487856456963e-05 -▁specialist 1.2471487856456963e-05 -▁Berliner 1.2467699433657311e-05 -▁ansprechen 1.2467699433657311e-05 -▁none 1.2467699433657311e-05 -uy 1.2463911010857657e-05 -▁illegalen 1.2463911010857657e-05 -▁plug 1.2460122588058005e-05 -ianischen 1.2456334165258352e-05 -▁Eurozone 1.2456334165258352e-05 -▁ord 1.2456334165258352e-05 -ко 1.24525457424587e-05 -▁Verarbeitung 1.24525457424587e-05 -▁tables 1.24525457424587e-05 -▁unterstreichen 1.24525457424587e-05 -▁EZB 1.2448757319659046e-05 -▁Frist 1.2448757319659046e-05 -▁Plattform 1.2448757319659046e-05 -▁vorzulegen 1.2441180474059742e-05 -▁SS 1.2437392051260088e-05 -▁wissenschaftliche 1.2437392051260088e-05 -▁interview 1.2433603628460436e-05 -▁offene 1.2433603628460436e-05 -▁Tätigkeiten 1.2429815205660782e-05 -▁größeres 1.2429815205660782e-05 -▁negotiating 1.2429815205660782e-05 -▁fans 1.242602678286113e-05 -izen 1.2422238360061476e-05 -▁Venice 1.2422238360061476e-05 -▁Genießen 1.2418449937261824e-05 -▁emp 1.2418449937261824e-05 -▁welcomes 1.2418449937261824e-05 -programms 1.2414661514462171e-05 -▁mitteilen 1.2414661514462171e-05 -▁supervision 1.2414661514462171e-05 -ро 1.2410873091662519e-05 -▁konzentriert 1.2410873091662519e-05 -▁langfristige 1.2410873091662519e-05 -▁relative 1.2410873091662519e-05 -inz 1.2407084668862867e-05 -istischer 1.2407084668862867e-05 -nes 1.2407084668862867e-05 -▁bere 1.2407084668862867e-05 -▁homes 1.2407084668862867e-05 -▁looked 1.2407084668862867e-05 -schau 1.2403296246063213e-05 -technik 1.2403296246063213e-05 -▁clubs 1.2403296246063213e-05 -▁univers 1.2403296246063213e-05 -▁UNESCO 1.2399507823263561e-05 -▁Zürich 1.2395719400463907e-05 -▁steigen 1.2395719400463907e-05 -▁viol 1.2395719400463907e-05 -▁vorgesehenen 1.2395719400463907e-05 -kur 1.2391930977664255e-05 -ulations 1.2391930977664255e-05 -▁Sachen 1.2391930977664255e-05 -anter 1.2388142554864601e-05 -▁Education 1.2388142554864601e-05 -▁Deb 1.2384354132064949e-05 -▁surely 1.2384354132064949e-05 -▁Auditors 1.2380565709265297e-05 -▁Ebenen 1.2380565709265297e-05 -▁Stuttgart 1.2376777286465643e-05 -▁Flex 1.2372988863665991e-05 -▁complicated 1.2372988863665991e-05 -▁contributed 1.2372988863665991e-05 -▁hast 1.2372988863665991e-05 -▁herum 1.2372988863665991e-05 -TER 1.2369200440866338e-05 -iance 1.2369200440866338e-05 -▁hospitality 1.2369200440866338e-05 -oper 1.2365412018066686e-05 -▁forthcoming 1.2365412018066686e-05 -▁wellness 1.2365412018066686e-05 -xy 1.2361623595267032e-05 -▁Leg 1.2361623595267032e-05 -▁enterprise 1.2361623595267032e-05 -▁suffered 1.2361623595267032e-05 -▁Seg 1.235783517246738e-05 -ilen 1.2354046749667726e-05 -jekt 1.2354046749667726e-05 -▁Ausflüge 1.2354046749667726e-05 -▁Existenz 1.2354046749667726e-05 -▁appearance 1.2354046749667726e-05 -▁konfrontiert 1.2354046749667726e-05 -▁künftige 1.2354046749667726e-05 -rechen 1.2350258326868074e-05 -▁Rückkehr 1.2350258326868074e-05 -▁cra 1.2350258326868074e-05 -▁kr 1.2350258326868074e-05 -▁sharing 1.2350258326868074e-05 -bus 1.2346469904068422e-05 -modell 1.2346469904068422e-05 -▁intern 1.2346469904068422e-05 -▁spezifische 1.2346469904068422e-05 -äst 1.2342681481268768e-05 -▁Abfall 1.2342681481268768e-05 -▁Entscheidungs 1.2342681481268768e-05 -▁eat 1.2342681481268768e-05 -▁Betten 1.2338893058469116e-05 -▁Gottes 1.2338893058469116e-05 -▁supposed 1.2338893058469116e-05 -trägen 1.2335104635669462e-05 -▁194 1.2335104635669462e-05 -▁dog 1.2335104635669462e-05 -▁everybody 1.2335104635669462e-05 -▁teile 1.2335104635669462e-05 -ugt 1.233131621286981e-05 -▁Situationen 1.233131621286981e-05 -▁preventing 1.233131621286981e-05 -erson 1.2327527790070157e-05 -richtlinie 1.2327527790070157e-05 -ärm 1.2327527790070157e-05 -▁1st 1.2327527790070157e-05 -▁manip 1.2327527790070157e-05 -▁remained 1.2327527790070157e-05 -▁techn 1.2327527790070157e-05 -ochen 1.2323739367270505e-05 -▁Tabak 1.2323739367270505e-05 -▁allocated 1.2323739367270505e-05 -▁Willkommen 1.2319950944470853e-05 -▁altern 1.2319950944470853e-05 -▁expand 1.2319950944470853e-05 -▁geboten 1.2319950944470853e-05 -▁studio 1.2319950944470853e-05 -▁Freizügigkeit 1.2316162521671199e-05 -▁Ruf 1.2316162521671199e-05 -TR 1.2308585676071893e-05 -▁Corporate 1.2308585676071893e-05 -▁amenities 1.2308585676071893e-05 -anische 1.2304797253272241e-05 -▁Weiteren 1.2304797253272241e-05 -▁überein 1.2304797253272241e-05 -mitt 1.2301008830472587e-05 -oman 1.2301008830472587e-05 -▁bisherigen 1.2301008830472587e-05 -▁intr 1.2297220407672935e-05 -▁abstimmen 1.2293431984873282e-05 -▁modules 1.2293431984873282e-05 -▁geraten 1.2289643562073629e-05 -▁originally 1.2285855139273977e-05 -▁parameters 1.2285855139273977e-05 -hören 1.2282066716474324e-05 -▁Kranken 1.2282066716474324e-05 -▁enabled 1.2282066716474324e-05 -▁Übernachtung 1.2282066716474324e-05 -▁è 1.2282066716474324e-05 -bereichen 1.2278278293674672e-05 -▁hardly 1.2278278293674672e-05 -▁A. 1.2274489870875018e-05 -▁Poly 1.2274489870875018e-05 -▁emb 1.2274489870875018e-05 -laden 1.2270701448075366e-05 -▁ordinary 1.2270701448075366e-05 -▁pa 1.2270701448075366e-05 -▁proceedings 1.2266913025275712e-05 -▁Fahrzeuge 1.226312460247606e-05 -ausch 1.2259336179676408e-05 -ø 1.2259336179676408e-05 -▁Ablehnung 1.2259336179676408e-05 -▁unterzeichnet 1.2259336179676408e-05 -mat 1.2255547756876754e-05 -▁Thailand 1.2255547756876754e-05 -uropa 1.2251759334077102e-05 -▁Sky 1.2251759334077102e-05 -▁dom 1.2251759334077102e-05 -▁museums 1.2251759334077102e-05 -▁spe 1.2251759334077102e-05 -used 1.2247970911277449e-05 -▁NGOs 1.2247970911277449e-05 -▁typ 1.2247970911277449e-05 -stant 1.2244182488477796e-05 -▁Does 1.2244182488477796e-05 -trans 1.2240394065678143e-05 -▁Richt 1.2240394065678143e-05 -▁Socialist 1.2240394065678143e-05 -▁romantic 1.2236605642878491e-05 -nehmen 1.2232817220078839e-05 -▁Frühstücksbuffet 1.2232817220078839e-05 -▁calm 1.2232817220078839e-05 -▁directed 1.2232817220078839e-05 -▁geprägt 1.2232817220078839e-05 -▁herunterladen 1.2232817220078839e-05 -▁superior 1.2229028797279185e-05 -ells 1.2225240374479533e-05 -▁beings 1.2225240374479533e-05 -▁compact 1.2225240374479533e-05 -▁hervorgehoben 1.2225240374479533e-05 -▁lay 1.2225240374479533e-05 -▁leistungs 1.2225240374479533e-05 -bruch 1.2221451951679879e-05 -▁Widerspruch 1.2221451951679879e-05 -hält 1.2217663528880227e-05 -▁SK 1.2217663528880227e-05 -▁lieber 1.2217663528880227e-05 -▁realise 1.2213875106080573e-05 -iefer 1.2210086683280921e-05 -▁Perspektive 1.2210086683280921e-05 -▁firmly 1.2210086683280921e-05 -reibung 1.2206298260481268e-05 -rix 1.2206298260481268e-05 -▁AND 1.2206298260481268e-05 -▁cells 1.2206298260481268e-05 -▁cuts 1.2206298260481268e-05 -▁Australia 1.2202509837681616e-05 -▁Det 1.2202509837681616e-05 -▁Medizin 1.2202509837681616e-05 -halle 1.2198721414881963e-05 -▁Stability 1.2198721414881963e-05 -▁multi- 1.2198721414881963e-05 -partner 1.219493299208231e-05 -х 1.219493299208231e-05 -▁Widerstand 1.219493299208231e-05 -▁movie 1.219493299208231e-05 -▁unterliegen 1.219493299208231e-05 -züge 1.2191144569282658e-05 -▁habit 1.2191144569282658e-05 -▁Opposition 1.2187356146483004e-05 -▁vorgeschlagene 1.2187356146483004e-05 -▁AV 1.2183567723683352e-05 -▁EX 1.2183567723683352e-05 -▁Customer 1.2179779300883698e-05 -▁Vista 1.2179779300883698e-05 -spro 1.2175990878084046e-05 -▁programming 1.2175990878084046e-05 -▁indicate 1.2172202455284394e-05 -folge 1.2164625609685088e-05 -▁Joseph 1.2164625609685088e-05 -▁Kommun 1.2164625609685088e-05 -▁Saturday 1.2164625609685088e-05 -▁Piazza 1.2160837186885435e-05 -▁Reisen 1.2160837186885435e-05 -▁Should 1.2157048764085783e-05 -▁Small 1.2157048764085783e-05 -▁errors 1.2157048764085783e-05 -IK 1.2153260341286129e-05 -zess 1.2153260341286129e-05 -▁Americans 1.2153260341286129e-05 -▁Vin 1.2153260341286129e-05 -▁Wirtschaftswachstum 1.2153260341286129e-05 -▁Tim 1.2149471918486477e-05 -▁entstand 1.2145683495686823e-05 -▁rat 1.2145683495686823e-05 --sh 1.2141895072887171e-05 -▁Planeten 1.2141895072887171e-05 -▁motiv 1.2138106650087519e-05 -▁movements 1.2138106650087519e-05 -ruck 1.2134318227287865e-05 -äus 1.2134318227287865e-05 -▁Bett 1.2134318227287865e-05 -▁Mallorca 1.2134318227287865e-05 -▁Urteil 1.2134318227287865e-05 -EG 1.2130529804488213e-05 -ef 1.2130529804488213e-05 -▁Haushaltsplan 1.2130529804488213e-05 -▁barriers 1.2130529804488213e-05 -▁nord 1.2130529804488213e-05 -▁Big 1.2126741381688559e-05 -▁Anwender 1.2122952958888907e-05 -▁Kas 1.2122952958888907e-05 -▁menschliche 1.2122952958888907e-05 -▁ergibt 1.2119164536089254e-05 -▁Suiten 1.2115376113289602e-05 -▁hochwertige 1.2115376113289602e-05 -▁missing 1.2115376113289602e-05 -▁transform 1.2115376113289602e-05 -leiter 1.211158769048995e-05 -zent 1.211158769048995e-05 -ühren 1.211158769048995e-05 -▁deb 1.211158769048995e-05 -▁Serbia 1.2107799267690296e-05 -▁fees 1.2107799267690296e-05 -▁spons 1.2107799267690296e-05 -▁gefallen 1.2104010844890644e-05 -▁recognized 1.2104010844890644e-05 -lösungen 1.210022242209099e-05 -▁behoben 1.210022242209099e-05 -CE 1.2096433999291338e-05 -orial 1.2096433999291338e-05 -richten 1.2096433999291338e-05 -water 1.2096433999291338e-05 -▁rap 1.2096433999291338e-05 -▁anschließen 1.2092645576491684e-05 -▁tree 1.2092645576491684e-05 -stände 1.2088857153692032e-05 -▁Asian 1.2088857153692032e-05 -▁Hard 1.2088857153692032e-05 -▁Brenn 1.208506873089238e-05 -▁angenehm 1.208506873089238e-05 -handels 1.2081280308092726e-05 -lers 1.2081280308092726e-05 -▁68 1.2081280308092726e-05 -▁Pension 1.2081280308092726e-05 -▁explained 1.2081280308092726e-05 -verwaltung 1.2073703462493421e-05 -▁productivity 1.2073703462493421e-05 -▁sight 1.2073703462493421e-05 -▁anhand 1.2069915039693769e-05 -▁anzuzeigen 1.2069915039693769e-05 -read 1.2066126616894115e-05 -▁classes 1.2066126616894115e-05 -▁dist 1.2066126616894115e-05 -▁finanziert 1.2066126616894115e-05 -▁introducing 1.2066126616894115e-05 -CC 1.2062338194094463e-05 -ichts 1.2062338194094463e-05 -▁dialog 1.2062338194094463e-05 -▁internen 1.2062338194094463e-05 -▁remarks 1.2062338194094463e-05 -macht 1.2058549771294809e-05 -▁Christmas 1.2058549771294809e-05 -▁Wor 1.2058549771294809e-05 -▁zukünftige 1.2058549771294809e-05 -▁Hauptbahnhof 1.2054761348495157e-05 -▁ausgehen 1.2054761348495157e-05 -▁Geneva 1.2050972925695505e-05 -▁rechten 1.2050972925695505e-05 -▁saved 1.2050972925695505e-05 -utzen 1.2047184502895851e-05 -▁Barr 1.2047184502895851e-05 -▁Konvent 1.2047184502895851e-05 -▁amazing 1.2047184502895851e-05 -▁whereas 1.2047184502895851e-05 -▁Pe 1.20433960800962e-05 -▁Miß 1.2039607657296545e-05 -▁Nachhaltigkeit 1.2039607657296545e-05 -▁eingegangen 1.2039607657296545e-05 -▁hearing 1.2039607657296545e-05 -▁sustainability 1.2039607657296545e-05 -zahlung 1.2035819234496893e-05 -▁Abendessen 1.2035819234496893e-05 -▁showed 1.2035819234496893e-05 -angel 1.203203081169724e-05 -imp 1.203203081169724e-05 -▁Culture 1.203203081169724e-05 -▁programm 1.203203081169724e-05 -schläge 1.2028242388897588e-05 -ypen 1.2028242388897588e-05 -zahlen 1.2028242388897588e-05 -▁enhanced 1.2028242388897588e-05 -▁functionality 1.2028242388897588e-05 -dition 1.2024453966097936e-05 -▁irre 1.2024453966097936e-05 -▁Resolution 1.2020665543298282e-05 -▁langfristig 1.2020665543298282e-05 -II 1.201687712049863e-05 -ò 1.201687712049863e-05 -▁cam 1.201687712049863e-05 -ahnen 1.2013088697698976e-05 -▁Kre 1.2013088697698976e-05 -▁vereinbart 1.2013088697698976e-05 -▁combine 1.2009300274899324e-05 -▁erg 1.2009300274899324e-05 -▁umfangreiche 1.2009300274899324e-05 -a. 1.200551185209967e-05 -▁sichere 1.200551185209967e-05 -bett 1.2001723429300018e-05 -▁Abschaffung 1.2001723429300018e-05 -▁Euch 1.2001723429300018e-05 -▁schlagen 1.2001723429300018e-05 -mitglieder 1.1997935006500364e-05 -▁Kreis 1.1997935006500364e-05 -▁sections 1.1997935006500364e-05 -▁Veränderung 1.1994146583700712e-05 -▁female 1.1994146583700712e-05 --of 1.199035816090106e-05 -nahm 1.199035816090106e-05 -pre 1.199035816090106e-05 -▁30. 1.199035816090106e-05 -▁Civil 1.199035816090106e-05 -▁obstacles 1.1986569738101407e-05 -geführt 1.1982781315301755e-05 -▁Nachbarn 1.1982781315301755e-05 -▁Rent 1.1982781315301755e-05 --wide 1.1978992892502101e-05 -▁Abhängigkeit 1.1978992892502101e-05 -OC 1.1975204469702449e-05 -anze 1.1975204469702449e-05 -▁Kanada 1.1975204469702449e-05 -▁imagine 1.1975204469702449e-05 -▁militärischen 1.1975204469702449e-05 -▁praktische 1.1975204469702449e-05 -aint 1.1971416046902795e-05 -▁05 1.1971416046902795e-05 -▁transmission 1.1971416046902795e-05 --pr 1.1967627624103143e-05 -▁Allianz 1.1967627624103143e-05 -▁Consumer 1.1967627624103143e-05 -▁Dep 1.1967627624103143e-05 -▁Vorlage 1.1967627624103143e-05 -▁Wähler 1.1967627624103143e-05 -▁vessels 1.1967627624103143e-05 -bestimmungen 1.1963839201303491e-05 -eals 1.1960050778503837e-05 -hers 1.1960050778503837e-05 -hle 1.1956262355704185e-05 -▁Imm 1.1956262355704185e-05 -▁Pläne 1.1956262355704185e-05 -▁Volkes 1.1956262355704185e-05 -▁endorse 1.1956262355704185e-05 -▁metro 1.1956262355704185e-05 -▁präsentieren 1.1956262355704185e-05 -aschen 1.1952473932904531e-05 -oral 1.1948685510104879e-05 -▁Rod 1.1948685510104879e-05 -▁gebeten 1.1948685510104879e-05 -ago 1.1944897087305226e-05 -fassung 1.1944897087305226e-05 -▁sogenannten 1.1944897087305226e-05 -▁Kommissionspräsident 1.1941108664505574e-05 -▁grosse 1.1941108664505574e-05 -▁investigation 1.1941108664505574e-05 -▁tot 1.193732024170592e-05 --6 1.1933531818906268e-05 -▁1960 1.1933531818906268e-05 -▁accurate 1.1933531818906268e-05 -▁dankbar 1.1933531818906268e-05 -▁internationally 1.1929743396106616e-05 -▁songs 1.1925954973306962e-05 -▁regardless 1.192216655050731e-05 -iat 1.1918378127707656e-05 -work 1.1918378127707656e-05 -ühle 1.1914589704908004e-05 -ifen 1.191080128210835e-05 -äumt 1.191080128210835e-05 -▁collected 1.191080128210835e-05 -▁enthaltenen 1.191080128210835e-05 -▁läßt 1.191080128210835e-05 -▁stocks 1.191080128210835e-05 -▁turning 1.191080128210835e-05 -point 1.1907012859308698e-05 -▁authent 1.1907012859308698e-05 -▁organic 1.1907012859308698e-05 -▁reflected 1.1903224436509046e-05 -▁gerichtet 1.1899436013709393e-05 -▁tab 1.1899436013709393e-05 -▁Internationale 1.1895647590909741e-05 -▁exact 1.1895647590909741e-05 -▁Product 1.1891859168110087e-05 -▁computers 1.1888070745310435e-05 -ibr 1.1884282322510781e-05 -▁besonderer 1.1884282322510781e-05 -▁vorliegende 1.1884282322510781e-05 -▁и 1.1884282322510781e-05 -baus 1.1880493899711129e-05 -phase 1.1880493899711129e-05 -▁ba 1.1880493899711129e-05 -▁gift 1.1880493899711129e-05 -▁hervorragenden 1.1880493899711129e-05 -pekt 1.1876705476911477e-05 -▁Erinnerung 1.1876705476911477e-05 -▁Großteil 1.1876705476911477e-05 -▁Islamic 1.1876705476911477e-05 -▁Regierungschefs 1.1876705476911477e-05 -▁consult 1.1876705476911477e-05 -▁exclusion 1.1876705476911477e-05 -▁süd 1.1876705476911477e-05 -▁united 1.1876705476911477e-05 -▁verboten 1.1872917054111823e-05 -▁Dritten 1.1869128631312171e-05 -▁militärische 1.1869128631312171e-05 -▁prison 1.1869128631312171e-05 -ause 1.1865340208512517e-05 -opol 1.1865340208512517e-05 -▁dargestellt 1.1865340208512517e-05 -▁generate 1.1865340208512517e-05 -nimmt 1.1861551785712865e-05 -ssysteme 1.1861551785712865e-05 -▁Meeting 1.1861551785712865e-05 -▁Schlag 1.1861551785712865e-05 -▁cancer 1.1861551785712865e-05 -▁prime 1.1861551785712865e-05 -▁Meines 1.1857763362913212e-05 -▁universities 1.1857763362913212e-05 -▁verdienen 1.1857763362913212e-05 -▁Entspannung 1.185397494011356e-05 -▁Na 1.185397494011356e-05 -▁intens 1.185397494011356e-05 -load 1.1850186517313906e-05 -▁Armee 1.1850186517313906e-05 -text 1.1846398094514254e-05 -▁Aufsichts 1.1846398094514254e-05 -▁Kontinent 1.1846398094514254e-05 -▁pun 1.1846398094514254e-05 -▁repl 1.1846398094514254e-05 -▁Volksgesundheit 1.1842609671714602e-05 -▁hingegen 1.1842609671714602e-05 -▁Konzert 1.1838821248914948e-05 -▁ausser 1.1838821248914948e-05 -▁Börse 1.1835032826115296e-05 -▁Versionen 1.1835032826115296e-05 -▁aer 1.1835032826115296e-05 -▁gap 1.1835032826115296e-05 -▁ride 1.1835032826115296e-05 -▁ca 1.182745598051599e-05 -▁schäd 1.182745598051599e-05 -▁staatliche 1.182745598051599e-05 -▁Gleichstellung 1.1823667557716337e-05 -▁kg 1.1823667557716337e-05 -▁student 1.1823667557716337e-05 -▁travelling 1.1823667557716337e-05 -▁CHF 1.1819879134916684e-05 -chs 1.1816090712117032e-05 -führen 1.1816090712117032e-05 -▁considerably 1.1816090712117032e-05 -▁geplant 1.1816090712117032e-05 -▁vorbei 1.1816090712117032e-05 -sson 1.1812302289317379e-05 -▁ECB 1.1812302289317379e-05 -▁generations 1.1812302289317379e-05 -▁notes 1.1812302289317379e-05 -▁Katastrophen 1.1808513866517727e-05 -▁Zeitung 1.1808513866517727e-05 -▁Fahrrad 1.1804725443718073e-05 -lines 1.1800937020918421e-05 -folgen 1.1797148598118767e-05 -▁covering 1.1797148598118767e-05 -▁DC 1.1793360175319115e-05 -▁Kirchen 1.1793360175319115e-05 -▁Leitung 1.1793360175319115e-05 -▁sights 1.1793360175319115e-05 -▁Modelle 1.1789571752519461e-05 -▁Urlaubs 1.1789571752519461e-05 -▁employ 1.1789571752519461e-05 -▁indirekt 1.1789571752519461e-05 -▁Augenblick 1.1785783329719809e-05 -▁disasters 1.1785783329719809e-05 -▁perm 1.1785783329719809e-05 -▁vari 1.1785783329719809e-05 -▁Befugnisse 1.1781994906920157e-05 -▁Word 1.1781994906920157e-05 -▁adapted 1.1781994906920157e-05 -▁trends 1.1781994906920157e-05 -▁fleet 1.1778206484120504e-05 -▁reaching 1.1778206484120504e-05 -DI 1.1774418061320851e-05 -▁Andere 1.1770629638521198e-05 -▁Betreiber 1.1770629638521198e-05 -▁Fein 1.1770629638521198e-05 -▁Spitzen 1.1770629638521198e-05 -▁layer 1.1770629638521198e-05 -▁Alb 1.1766841215721546e-05 -itel 1.1763052792921892e-05 -▁Bor 1.1763052792921892e-05 -▁Nav 1.1763052792921892e-05 -▁frame 1.175926437012224e-05 -▁ministers 1.175926437012224e-05 -▁Sammlung 1.1755475947322588e-05 -▁jetzigen 1.1755475947322588e-05 -▁pretty 1.1755475947322588e-05 -ila 1.1751687524522934e-05 -▁ens 1.1751687524522934e-05 -▁erfreut 1.1751687524522934e-05 -▁Dokumenten 1.1747899101723282e-05 -▁Grundlagen 1.1747899101723282e-05 -▁Luxemburg 1.1747899101723282e-05 -etch 1.1744110678923628e-05 -▁assess 1.1744110678923628e-05 -▁ju 1.1744110678923628e-05 -▁pan 1.1744110678923628e-05 -CO 1.1740322256123976e-05 -bal 1.1740322256123976e-05 -erbe 1.1740322256123976e-05 -reffen 1.1740322256123976e-05 -▁bike 1.1740322256123976e-05 -▁dignity 1.1740322256123976e-05 -ibil 1.1736533833324323e-05 -oben 1.1736533833324323e-05 -▁HU 1.1736533833324323e-05 -▁Vorausschau 1.1736533833324323e-05 -▁natürliche 1.1736533833324323e-05 -▁strategische 1.1736533833324323e-05 -zustellen 1.1732745410524671e-05 -▁Gespräch 1.1732745410524671e-05 -▁Transfer 1.1732745410524671e-05 -▁Voraus 1.1732745410524671e-05 -▁gewünschten 1.1732745410524671e-05 -▁underground 1.1732745410524671e-05 -▁zieht 1.1732745410524671e-05 -unkt 1.1728956987725018e-05 -▁Commissioners 1.1728956987725018e-05 -▁checks 1.1728956987725018e-05 -▁dra 1.1728956987725018e-05 -▁eignet 1.1728956987725018e-05 -▁till 1.1728956987725018e-05 -aces 1.1721380142125713e-05 -gi 1.1721380142125713e-05 -▁Ang 1.1721380142125713e-05 -▁Nummer 1.1721380142125713e-05 -▁labelling 1.1721380142125713e-05 -▁rück 1.1721380142125713e-05 -band 1.1717591719326059e-05 -▁ER 1.1717591719326059e-05 -▁undoubtedly 1.1717591719326059e-05 --TV 1.1713803296526407e-05 -▁Ku 1.1713803296526407e-05 -▁unabhängigen 1.1713803296526407e-05 -▁C. 1.1710014873726753e-05 -▁penalty 1.1710014873726753e-05 -stitution 1.1706226450927101e-05 -▁Dor 1.1706226450927101e-05 -durch 1.1702438028127447e-05 -▁normalerweise 1.1698649605327795e-05 -▁Beteiligten 1.1694861182528143e-05 -▁nachdrücklich 1.1694861182528143e-05 -▁58 1.169107275972849e-05 --Mod 1.1687284336928838e-05 -obil 1.1687284336928838e-05 -▁'A 1.1687284336928838e-05 -▁recip 1.1687284336928838e-05 --7 1.1683495914129184e-05 -▁Gesicht 1.1683495914129184e-05 -▁normalen 1.1683495914129184e-05 -▁Geheim 1.1679707491329532e-05 -▁hohem 1.1679707491329532e-05 -▁tested 1.1679707491329532e-05 -▁gewähren 1.1675919068529878e-05 -bung 1.1672130645730226e-05 -▁acceptance 1.1672130645730226e-05 -▁consulting 1.1672130645730226e-05 -GE 1.1668342222930574e-05 -▁Effekt 1.1668342222930574e-05 -▁Gall 1.1668342222930574e-05 -▁Märkten 1.1668342222930574e-05 -▁heim 1.1668342222930574e-05 -forschung 1.166455380013092e-05 -▁Heizung 1.166455380013092e-05 -▁Opera 1.166455380013092e-05 -▁worst 1.166455380013092e-05 -▁belief 1.1660765377331268e-05 -▁excluded 1.1660765377331268e-05 -▁gemeinschaftlichen 1.1660765377331268e-05 -onn 1.1656976954531614e-05 -▁discount 1.1656976954531614e-05 -▁zugrunde 1.1656976954531614e-05 -▁AP 1.1653188531731962e-05 -▁kontrollieren 1.1653188531731962e-05 -▁Innenstadt 1.1649400108932309e-05 -▁Kein 1.1649400108932309e-05 -▁filled 1.1649400108932309e-05 -▁leichter 1.1649400108932309e-05 -▁Binnenmarktes 1.1645611686132657e-05 -▁managing 1.1641823263333003e-05 -▁verein 1.1641823263333003e-05 -▁violations 1.1641823263333003e-05 -ktiv 1.1638034840533351e-05 -pot 1.1638034840533351e-05 -änen 1.1638034840533351e-05 -▁legisl 1.1638034840533351e-05 -▁exploitation 1.1634246417733699e-05 -▁pot 1.1634246417733699e-05 -auern 1.1626669572134393e-05 -ji 1.1626669572134393e-05 -▁Wirtschaftspolitik 1.1626669572134393e-05 -SA 1.1622881149334739e-05 -arc 1.1622881149334739e-05 -zub 1.1622881149334739e-05 -▁Muslim 1.1622881149334739e-05 -▁festzulegen 1.1622881149334739e-05 -▁fil 1.1622881149334739e-05 -▁plastic 1.1622881149334739e-05 -▁sichtbar 1.1622881149334739e-05 -▁wirtschaftlich 1.1622881149334739e-05 -▁derart 1.1619092726535087e-05 -▁edition 1.1619092726535087e-05 -▁golden 1.1619092726535087e-05 -▁Industry 1.1615304303735433e-05 -▁Mich 1.1611515880935781e-05 -▁believes 1.1611515880935781e-05 -▁enger 1.1611515880935781e-05 -▁protocol 1.1611515880935781e-05 -schn 1.1607727458136129e-05 -EX 1.1603939035336476e-05 -stit 1.1603939035336476e-05 -▁recover 1.1603939035336476e-05 -22 1.1600150612536824e-05 -▁speichern 1.1600150612536824e-05 -▁19th 1.159636218973717e-05 -▁Casa 1.159636218973717e-05 -і 1.1592573766937518e-05 -▁Reserve 1.1592573766937518e-05 -▁conver 1.1592573766937518e-05 -▁restored 1.1592573766937518e-05 -eting 1.1588785344137864e-05 -ares 1.1584996921338212e-05 -▁mittel 1.1584996921338212e-05 -▁profession 1.1584996921338212e-05 -▁reflects 1.1584996921338212e-05 -▁ähnliche 1.1584996921338212e-05 -▁constit 1.1581208498538558e-05 -egel 1.1577420075738906e-05 -ener 1.1577420075738906e-05 -▁31. 1.1577420075738906e-05 -▁Bür 1.1577420075738906e-05 -▁Prüf 1.1577420075738906e-05 -▁Häuser 1.1573631652939254e-05 -▁externen 1.1573631652939254e-05 -▁fax 1.1573631652939254e-05 --Hotel 1.15698432301396e-05 -▁Aufenthalts 1.15698432301396e-05 -▁Ursachen 1.15698432301396e-05 -▁aff 1.15698432301396e-05 -▁bedanken 1.15698432301396e-05 -▁trag 1.15698432301396e-05 -▁Esp 1.1566054807339948e-05 -▁Lohn 1.1558477961740643e-05 -▁Nero 1.1558477961740643e-05 -orient 1.1554689538940989e-05 -▁Flüchtlinge 1.1554689538940989e-05 -▁Schnell 1.1554689538940989e-05 --Aus 1.1550901116141337e-05 -▁erhoben 1.1550901116141337e-05 -▁Album 1.1547112693341685e-05 -▁Alters 1.1547112693341685e-05 -▁beschränken 1.1547112693341685e-05 -▁nine 1.1547112693341685e-05 -▁satisfy 1.1547112693341685e-05 -▁scheinen 1.1547112693341685e-05 -▁überaus 1.1547112693341685e-05 -▁transferred 1.1543324270542031e-05 -prod 1.1539535847742379e-05 -▁DS 1.1539535847742379e-05 -wärts 1.1535747424942725e-05 -jahres 1.1531959002143073e-05 -sa 1.1531959002143073e-05 -▁Emissions 1.1531959002143073e-05 -▁Reservierung 1.1531959002143073e-05 -ules 1.1528170579343419e-05 -▁Generationen 1.1524382156543767e-05 -▁equivalent 1.1520593733744115e-05 -▁komplette 1.1520593733744115e-05 -ijk 1.1516805310944462e-05 -▁Autos 1.1516805310944462e-05 -▁Non 1.1516805310944462e-05 -▁shadow 1.1516805310944462e-05 -▁Todes 1.151301688814481e-05 -▁poly 1.151301688814481e-05 -▁Gi 1.1509228465345156e-05 -▁Handlungs 1.1509228465345156e-05 -▁clause 1.1509228465345156e-05 -▁stages 1.1509228465345156e-05 -▁subsequent 1.1509228465345156e-05 -▁ut 1.1509228465345156e-05 -gers 1.1505440042545504e-05 -hard 1.1505440042545504e-05 -▁lived 1.1505440042545504e-05 -isions 1.150165161974585e-05 -▁Berichten 1.150165161974585e-05 -▁Coun 1.150165161974585e-05 -85 1.1497863196946198e-05 -▁Cross 1.1497863196946198e-05 -krise 1.1494074774146544e-05 -urr 1.1494074774146544e-05 -▁selling 1.1494074774146544e-05 -▁stuff 1.1494074774146544e-05 -TV 1.1490286351346892e-05 -holt 1.1490286351346892e-05 -▁GM 1.1490286351346892e-05 -▁Konflikte 1.1482709505747586e-05 -▁dig 1.1482709505747586e-05 -▁Rep 1.1478921082947934e-05 -viertel 1.1475132660148281e-05 -▁Gemeinsame 1.1475132660148281e-05 -▁Speisen 1.1475132660148281e-05 -▁anywhere 1.1475132660148281e-05 -▁concentrate 1.1475132660148281e-05 -ahren 1.1471344237348629e-05 -▁Account 1.1471344237348629e-05 -▁Dav 1.1471344237348629e-05 -94 1.1467555814548975e-05 -▁Frank 1.1467555814548975e-05 -▁experiment 1.1467555814548975e-05 -icated 1.1463767391749323e-05 -usch 1.1463767391749323e-05 -▁Verk 1.1463767391749323e-05 -▁erweitern 1.1463767391749323e-05 -▁hoffentlich 1.1463767391749323e-05 -▁Museen 1.1459978968949671e-05 -▁Organe 1.1459978968949671e-05 -▁existiert 1.1459978968949671e-05 -▁reliability 1.1459978968949671e-05 -pal 1.1456190546150017e-05 -▁Protocol 1.1456190546150017e-05 -▁freely 1.1456190546150017e-05 -▁supplied 1.1456190546150017e-05 -orts 1.1452402123350365e-05 -▁fördert 1.1452402123350365e-05 -▁multilateral 1.1452402123350365e-05 -▁untersucht 1.1452402123350365e-05 -bereit 1.1448613700550711e-05 -wind 1.1448613700550711e-05 -▁Textil 1.1448613700550711e-05 -ehm 1.1444825277751059e-05 -reck 1.1444825277751059e-05 -▁Aktionsplan 1.1444825277751059e-05 -▁designs 1.1444825277751059e-05 -▁festgelegten 1.1444825277751059e-05 -▁geringer 1.1444825277751059e-05 -▁losses 1.1444825277751059e-05 -agr 1.1441036854951405e-05 -iam 1.1441036854951405e-05 -▁floors 1.1441036854951405e-05 -anal 1.14334600093521e-05 -▁140 1.14334600093521e-05 -▁einheitliche 1.14334600093521e-05 -bergen 1.1429671586552448e-05 -▁Fran 1.1429671586552448e-05 -▁Georgia 1.1429671586552448e-05 -▁Verteilung 1.1429671586552448e-05 -▁beh 1.1429671586552448e-05 -▁lebens 1.1429671586552448e-05 -▁varied 1.1429671586552448e-05 -osis 1.1425883163752796e-05 -▁Alkohol 1.1425883163752796e-05 -führte 1.1422094740953142e-05 -▁Tausende 1.1422094740953142e-05 -▁Stärke 1.141830631815349e-05 -uj 1.1414517895353836e-05 -▁cart 1.1414517895353836e-05 -▁priv 1.1414517895353836e-05 -▁Fal 1.1410729472554184e-05 -▁undertaken 1.1410729472554184e-05 -▁Cookies 1.1403152626954878e-05 -▁a.m. 1.1403152626954878e-05 -eath 1.1399364204155226e-05 --Ab 1.1395575781355572e-05 -appen 1.1395575781355572e-05 -igr 1.1395575781355572e-05 -isch- 1.1395575781355572e-05 -▁Wochenende 1.1395575781355572e-05 -▁registr 1.1395575781355572e-05 -ted 1.139178735855592e-05 -г 1.139178735855592e-05 -▁Fischere 1.139178735855592e-05 -▁Location 1.139178735855592e-05 -▁Louis 1.139178735855592e-05 -▁umfassen 1.139178735855592e-05 -▁Bonus 1.1387998935756267e-05 -▁Kreditkarte 1.1387998935756267e-05 -▁severe 1.1387998935756267e-05 -projekt 1.1384210512956615e-05 -▁learned 1.1384210512956615e-05 -▁perfekte 1.1384210512956615e-05 -▁Fuss 1.1380422090156961e-05 -▁sovereignty 1.1380422090156961e-05 -▁fert 1.1376633667357309e-05 -organ 1.1372845244557655e-05 -▁Academy 1.1372845244557655e-05 -▁Filter 1.1372845244557655e-05 -bers 1.1369056821758003e-05 -ffekt 1.1369056821758003e-05 -▁Kriminalität 1.1369056821758003e-05 -▁umsetzen 1.1369056821758003e-05 -pers 1.1365268398958351e-05 -spe 1.1365268398958351e-05 -▁Would 1.1365268398958351e-05 -▁wirksame 1.1365268398958351e-05 -reat 1.1361479976158697e-05 -ehn 1.1353903130559392e-05 -ächtigen 1.1353903130559392e-05 -▁Rechten 1.1353903130559392e-05 -▁reconstruction 1.1353903130559392e-05 -▁humanitäre 1.1350114707759739e-05 -▁overlooking 1.1350114707759739e-05 -▁25. 1.1346326284960086e-05 -▁Beide 1.1346326284960086e-05 -▁push 1.1346326284960086e-05 -▁Dra 1.1342537862160434e-05 -▁cultures 1.1342537862160434e-05 -▁nützlich 1.1342537862160434e-05 -▁picturesque 1.1342537862160434e-05 -ichert 1.1338749439360782e-05 -iew 1.1338749439360782e-05 -▁Dritte 1.1338749439360782e-05 -▁non-smoking 1.1338749439360782e-05 -▁zurückzuführen 1.1338749439360782e-05 -▁1970 1.1334961016561128e-05 -▁fishermen 1.1334961016561128e-05 -▁komfortable 1.1334961016561128e-05 -▁sicheren 1.1334961016561128e-05 -▁Angel 1.1331172593761476e-05 -▁Kriegs 1.1331172593761476e-05 -EST 1.1327384170961822e-05 -▁Iron 1.1327384170961822e-05 -▁aufgebaut 1.1327384170961822e-05 -▁restore 1.1327384170961822e-05 -ennt 1.132359574816217e-05 -▁Florenz 1.132359574816217e-05 -▁Himmel 1.132359574816217e-05 -▁Tanz 1.132359574816217e-05 -▁minimal 1.1319807325362516e-05 -▁wonach 1.1316018902562864e-05 -ori 1.1312230479763212e-05 -▁Satelliten 1.1312230479763212e-05 -▁hierfür 1.1312230479763212e-05 -earing 1.1308442056963559e-05 -rep 1.1308442056963559e-05 -▁Finanzkrise 1.1308442056963559e-05 -▁Friedrich 1.1308442056963559e-05 -▁Science 1.1308442056963559e-05 -▁Tests 1.1308442056963559e-05 -▁ré 1.1308442056963559e-05 -▁Allah 1.1304653634163906e-05 -▁Sm 1.1304653634163906e-05 -▁restructuring 1.1304653634163906e-05 -▁strengthened 1.1304653634163906e-05 -▁uncon 1.1304653634163906e-05 -▁villa 1.1304653634163906e-05 -sätze 1.1300865211364253e-05 -▁CON 1.12970767885646e-05 -▁Friendly 1.12970767885646e-05 -▁Sieg 1.12970767885646e-05 -▁Würde 1.12970767885646e-05 -▁Zahlungen 1.12970767885646e-05 -▁qual 1.12970767885646e-05 -▁Rechtsstaatlichkeit 1.1293288365764947e-05 -▁Richard 1.1293288365764947e-05 -▁pas 1.1293288365764947e-05 -▁Disc 1.1289499942965295e-05 -▁Gesam 1.1289499942965295e-05 -▁arise 1.1289499942965295e-05 -esign 1.1285711520165641e-05 -▁begleitet 1.1285711520165641e-05 -▁Öffnung 1.1285711520165641e-05 -▁gefragt 1.1281923097365989e-05 -▁laufenden 1.1281923097365989e-05 -politische 1.1278134674566337e-05 -▁Bour 1.1278134674566337e-05 -▁subs 1.1278134674566337e-05 -▁Notes 1.1274346251766683e-05 -▁unabhängige 1.1274346251766683e-05 -▁56 1.1270557828967031e-05 -atal 1.1266769406167378e-05 -▁hill 1.1266769406167378e-05 -▁solar 1.1266769406167378e-05 -▁spir 1.1266769406167378e-05 -▁Sat 1.1262980983367726e-05 -▁erfordern 1.1259192560568072e-05 -▁everywhere 1.1259192560568072e-05 -rap 1.125540413776842e-05 -▁Easy 1.125540413776842e-05 -▁Verbrauch 1.125540413776842e-05 -▁kompet 1.125540413776842e-05 -▁stattfindet 1.125540413776842e-05 -ained 1.1251615714968768e-05 -▁thermal 1.1251615714968768e-05 -▁wahren 1.1251615714968768e-05 -.2010 1.1247827292169114e-05 -▁Hohe 1.1247827292169114e-05 -▁massage 1.1247827292169114e-05 -lang 1.1244038869369462e-05 -▁angewendet 1.1244038869369462e-05 -▁coordinated 1.1244038869369462e-05 -▁panoramic 1.1244038869369462e-05 -▁wünsche 1.1244038869369462e-05 -komm 1.1240250446569808e-05 -zw 1.1240250446569808e-05 -▁Prim 1.1236462023770156e-05 -▁Schwarz 1.1236462023770156e-05 -▁cook 1.1236462023770156e-05 -▁zugelassen 1.1236462023770156e-05 -▁antworten 1.1232673600970502e-05 -▁grossen 1.1232673600970502e-05 -▁stunning 1.122888517817085e-05 -ties 1.1225096755371197e-05 -ř 1.1225096755371197e-05 -▁Appartement 1.1225096755371197e-05 -▁heißen 1.1225096755371197e-05 -▁vornehmen 1.1225096755371197e-05 -sehen 1.1221308332571545e-05 -▁Quelle 1.1221308332571545e-05 -▁aufs 1.1221308332571545e-05 -34 1.1217519909771893e-05 -▁Haup 1.1217519909771893e-05 -itch 1.1213731486972239e-05 -stag 1.1213731486972239e-05 -systems 1.1213731486972239e-05 -▁editor 1.1213731486972239e-05 -▁rufen 1.1213731486972239e-05 -▁Überblick 1.1213731486972239e-05 -▁Darstellung 1.1209943064172587e-05 -▁Schön 1.1209943064172587e-05 -kreis 1.1206154641372933e-05 -▁Sir 1.1206154641372933e-05 -▁treatments 1.1206154641372933e-05 -▁Marc 1.1202366218573281e-05 -▁infl 1.1202366218573281e-05 -▁Brief 1.1198577795773627e-05 -▁Pakete 1.1194789372973975e-05 -▁Pic 1.1194789372973975e-05 -▁einzuführen 1.1191000950174323e-05 -kette 1.1187212527374669e-05 -▁formats 1.1187212527374669e-05 -▁multinational 1.1187212527374669e-05 -▁ol 1.1187212527374669e-05 -▁Basel 1.1183424104575017e-05 -▁Bereitschaft 1.1183424104575017e-05 -▁Gay 1.1183424104575017e-05 -▁individually 1.1183424104575017e-05 -▁decade 1.1179635681775364e-05 -▁klassischen 1.1175847258975712e-05 -▁neighbouring 1.1175847258975712e-05 -▁province 1.1175847258975712e-05 -▁schnellen 1.1175847258975712e-05 -▁assembly 1.1172058836176058e-05 -▁condemn 1.1172058836176058e-05 -▁incomp 1.1172058836176058e-05 -▁qualitativ 1.1172058836176058e-05 -offen 1.1168270413376406e-05 -▁Sicher 1.1168270413376406e-05 -▁deeply 1.1168270413376406e-05 -▁mehrfach 1.1168270413376406e-05 -▁mittlerweile 1.1168270413376406e-05 -list 1.1164481990576754e-05 -▁Haben 1.1164481990576754e-05 -▁listings 1.1164481990576754e-05 -▁superb 1.1164481990576754e-05 -ili 1.11606935677771e-05 -ünst 1.11606935677771e-05 -▁Weltkrieg 1.11606935677771e-05 -▁Mehrwert 1.1156905144977448e-05 -▁Que 1.1156905144977448e-05 -▁belie 1.1153116722177794e-05 -▁ordered 1.1153116722177794e-05 -uzz 1.1149328299378142e-05 -▁Doha 1.1149328299378142e-05 -▁Light 1.1149328299378142e-05 -open 1.1145539876578488e-05 -ple 1.1145539876578488e-05 -umping 1.1145539876578488e-05 -▁Amts 1.1145539876578488e-05 -▁Bil 1.1145539876578488e-05 -▁H-0 1.1145539876578488e-05 -▁bean 1.1145539876578488e-05 -▁enorm 1.1145539876578488e-05 -▁anniversary 1.1141751453778836e-05 -▁Denmark 1.1137963030979183e-05 -▁geäußert 1.1137963030979183e-05 -▁shuttle 1.1137963030979183e-05 -▁Somit 1.1134174608179531e-05 -▁afford 1.1134174608179531e-05 -▁holds 1.1134174608179531e-05 -▁Mängel 1.1130386185379879e-05 -▁bewältigen 1.1130386185379879e-05 -▁frühen 1.1126597762580225e-05 -ächen 1.1122809339780573e-05 -▁Terror 1.1122809339780573e-05 -▁Thr 1.1122809339780573e-05 -▁briefly 1.1122809339780573e-05 -▁necessarily 1.1122809339780573e-05 -edia 1.1119020916980919e-05 -▁Gre 1.1119020916980919e-05 -▁Quark 1.1119020916980919e-05 -▁acknowledge 1.1119020916980919e-05 -▁eigen 1.1119020916980919e-05 -ägen 1.1115232494181267e-05 -▁approve 1.1115232494181267e-05 -▁bestellen 1.1115232494181267e-05 -▁gestärkt 1.1115232494181267e-05 -▁unge 1.1115232494181267e-05 -▁executive 1.1111444071381613e-05 -▁saving 1.1111444071381613e-05 -aterial 1.1107655648581961e-05 -▁tem 1.1107655648581961e-05 -▁warn 1.1107655648581961e-05 -fähigen 1.1103867225782309e-05 -äte 1.1103867225782309e-05 -▁bathrooms 1.1103867225782309e-05 -echn 1.1100078802982655e-05 -kul 1.1100078802982655e-05 -▁tracks 1.1100078802982655e-05 -öse 1.1096290380183003e-05 -▁Warcraft 1.109250195738335e-05 -▁king 1.109250195738335e-05 --friendly 1.1088713534583698e-05 -▁Straßburg 1.1088713534583698e-05 -▁Wichtig 1.1088713534583698e-05 -▁eingestellt 1.1088713534583698e-05 -▁keinem 1.1088713534583698e-05 -▁rig 1.1088713534583698e-05 -98 1.1084925111784044e-05 -▁Faktor 1.1084925111784044e-05 -osed 1.1081136688984392e-05 -ych 1.1081136688984392e-05 -▁faszin 1.1081136688984392e-05 -▁hiking 1.1081136688984392e-05 -▁ziel 1.1081136688984392e-05 -▁Att 1.1077348266184738e-05 -▁Bearbeitung 1.1077348266184738e-05 -▁Dol 1.1077348266184738e-05 -▁ambition 1.1077348266184738e-05 -▁angebracht 1.1077348266184738e-05 -▁empfohlen 1.1077348266184738e-05 -werfen 1.1073559843385086e-05 -▁Debatten 1.1073559843385086e-05 -▁Niederlanden 1.1073559843385086e-05 -ité 1.1069771420585434e-05 -▁GPS 1.1069771420585434e-05 -▁Russ 1.1069771420585434e-05 -▁Steuerung 1.1069771420585434e-05 -enswert 1.106598299778578e-05 -▁Arbeitgeber 1.106598299778578e-05 -▁URL 1.106598299778578e-05 -▁fairly 1.106598299778578e-05 -▁targeted 1.1062194574986128e-05 -ikanischen 1.1058406152186474e-05 -▁guard 1.1058406152186474e-05 -ustr 1.1054617729386822e-05 -▁Process 1.1054617729386822e-05 -▁Vent 1.1054617729386822e-05 -▁magazine 1.1054617729386822e-05 -▁normally 1.1054617729386822e-05 -▁zerstört 1.1054617729386822e-05 -▁Minute 1.1050829306587169e-05 -▁bauen 1.1050829306587169e-05 -▁schweren 1.1050829306587169e-05 -▁adds 1.1047040883787517e-05 -▁het 1.1047040883787517e-05 -▁manufacture 1.1047040883787517e-05 -67 1.1043252460987865e-05 -▁Saddam 1.1043252460987865e-05 -▁gesetzlichen 1.1043252460987865e-05 -▁selber 1.1043252460987865e-05 -änderungen 1.1039464038188211e-05 -▁Jahrzehnten 1.1039464038188211e-05 -▁Vier 1.1039464038188211e-05 -äß 1.1035675615388559e-05 -▁0. 1.1035675615388559e-05 -▁Petitions 1.1035675615388559e-05 -▁damages 1.1035675615388559e-05 -ritten 1.1031887192588905e-05 -ées 1.1031887192588905e-05 -▁Vereinfachung 1.1031887192588905e-05 -▁teaching 1.1031887192588905e-05 -▁witness 1.1031887192588905e-05 -▁hy 1.1028098769789253e-05 -erte 1.10243103469896e-05 -gesetzt 1.10243103469896e-05 -▁Ticket 1.10243103469896e-05 -▁egal 1.10243103469896e-05 -▁reprod 1.10243103469896e-05 -▁verwende 1.10243103469896e-05 -annen 1.1020521924189947e-05 -▁Parkplätze 1.1020521924189947e-05 -▁sichergestellt 1.1016733501390294e-05 -▁Histor 1.1012945078590641e-05 -▁portfolio 1.1012945078590641e-05 -zeichnungen 1.1009156655790989e-05 -▁Streit 1.1009156655790989e-05 -▁Wandel 1.1009156655790989e-05 -▁excess 1.1009156655790989e-05 -▁Anal 1.1005368232991336e-05 -▁Labour 1.1005368232991336e-05 -▁meal 1.1005368232991336e-05 -▁traum 1.1005368232991336e-05 -▁Kategorien 1.1001579810191684e-05 -▁cond 1.1001579810191684e-05 -richtungen 1.099779138739203e-05 -▁processed 1.099779138739203e-05 -▁Borg 1.0994002964592378e-05 -▁factory 1.0994002964592378e-05 -▁ta 1.0994002964592378e-05 -▁Consequently 1.0990214541792724e-05 -▁Hilf 1.0990214541792724e-05 -▁Konzern 1.0990214541792724e-05 -▁überlassen 1.0990214541792724e-05 -▁überwinden 1.0990214541792724e-05 -▁HIV 1.0986426118993072e-05 -▁reputation 1.098263769619342e-05 -▁selben 1.098263769619342e-05 -DF 1.0978849273393766e-05 -esis 1.0978849273393766e-05 -▁besorgt 1.0978849273393766e-05 -▁decline 1.0978849273393766e-05 -▁digitalen 1.0978849273393766e-05 -bef 1.0975060850594114e-05 -schriften 1.0975060850594114e-05 -▁Bele 1.0975060850594114e-05 -▁SQL 1.0975060850594114e-05 -▁College 1.097127242779446e-05 -▁aw 1.097127242779446e-05 -▁mechan 1.097127242779446e-05 -ettes 1.0967484004994808e-05 -▁twice 1.0967484004994808e-05 -▁wear 1.0967484004994808e-05 -▁Energieeffizienz 1.0963695582195155e-05 -▁olive 1.0963695582195155e-05 -▁Glauben 1.0959907159395503e-05 -▁Tax 1.0959907159395503e-05 -▁smooth 1.0959907159395503e-05 -▁Erhaltung 1.0956118736595851e-05 -▁Search 1.0956118736595851e-05 -▁casino 1.0956118736595851e-05 -▁ergänzt 1.0952330313796197e-05 -▁usage 1.0952330313796197e-05 -gewicht 1.0948541890996545e-05 -▁130 1.0948541890996545e-05 -▁Branche 1.0948541890996545e-05 -▁Gh 1.0948541890996545e-05 -▁betr 1.0948541890996545e-05 -▁separ 1.0944753468196891e-05 -weiten 1.0940965045397239e-05 -▁Belgian 1.0940965045397239e-05 -▁determination 1.0940965045397239e-05 -▁reserved 1.0940965045397239e-05 -▁Zahlung 1.0937176622597585e-05 -▁highlights 1.0937176622597585e-05 -▁türkischen 1.0937176622597585e-05 -▁Zone 1.0933388199797933e-05 -▁engaged 1.0933388199797933e-05 -ordin 1.092959977699828e-05 -▁Nu 1.092959977699828e-05 -▁bonus 1.092959977699828e-05 -▁rechtzeitig 1.092959977699828e-05 -chter 1.0925811354198627e-05 -zahlungen 1.0925811354198627e-05 -▁funktion 1.0925811354198627e-05 -▁presenting 1.0925811354198627e-05 -holen 1.0922022931398975e-05 -rachten 1.0922022931398975e-05 -▁Kommissions 1.0922022931398975e-05 -▁SW 1.0922022931398975e-05 -▁Bestimmung 1.0918234508599322e-05 -▁Koch 1.0918234508599322e-05 -▁Lea 1.0918234508599322e-05 -▁Temperatur 1.0918234508599322e-05 -gaben 1.091444608579967e-05 -region 1.091444608579967e-05 -▁engines 1.091444608579967e-05 -itals 1.0910657663000016e-05 -▁Wirtschaftskrise 1.0910657663000016e-05 -▁unmittelbarer 1.0910657663000016e-05 -▁Daniel 1.0906869240200364e-05 -▁Make 1.0906869240200364e-05 -▁struktur 1.0906869240200364e-05 -gemäß 1.090308081740071e-05 -mitglied 1.090308081740071e-05 -▁Berichterstatters 1.090308081740071e-05 -▁Immer 1.090308081740071e-05 -▁Wal 1.090308081740071e-05 -arts 1.0899292394601058e-05 -ergie 1.0899292394601058e-05 -▁Ferienwohnung 1.0899292394601058e-05 -▁Greens 1.0899292394601058e-05 -▁SL 1.0899292394601058e-05 -▁Si 1.0899292394601058e-05 -▁layout 1.0899292394601058e-05 -▁threats 1.0899292394601058e-05 -üd 1.0895503971801406e-05 -▁Mah 1.0895503971801406e-05 -▁ended 1.0895503971801406e-05 -▁gesetz 1.0895503971801406e-05 -vern 1.0891715549001752e-05 -▁Währungsunion 1.0891715549001752e-05 -unes 1.08879271262021e-05 -▁Investoren 1.08879271262021e-05 -ANT 1.0884138703402447e-05 -heben 1.0884138703402447e-05 -▁350 1.0884138703402447e-05 -▁Deck 1.0884138703402447e-05 -▁Schm 1.0884138703402447e-05 -▁panel 1.0884138703402447e-05 -▁struggle 1.0884138703402447e-05 -▁untersch 1.0884138703402447e-05 -methoden 1.0880350280602794e-05 -▁spezialisiert 1.0880350280602794e-05 -▁Registrierung 1.0876561857803141e-05 -▁110 1.0872773435003489e-05 -▁Porto 1.0872773435003489e-05 -onia 1.0868985012203835e-05 -▁begins 1.0868985012203835e-05 -ATION 1.0865196589404183e-05 -itable 1.0865196589404183e-05 -▁57 1.0865196589404183e-05 -fähige 1.0861408166604531e-05 -▁Arbeiter 1.0861408166604531e-05 -▁Fernsehen 1.0857619743804877e-05 -▁kernel 1.0857619743804877e-05 -▁printed 1.0857619743804877e-05 -ão 1.0853831321005225e-05 -▁herrlichen 1.0853831321005225e-05 -▁Einz 1.0850042898205571e-05 -▁Key 1.0850042898205571e-05 -▁Weil 1.0850042898205571e-05 -▁enorme 1.0850042898205571e-05 -▁Pad 1.0846254475405919e-05 -▁frage 1.0846254475405919e-05 -▁synchron 1.0846254475405919e-05 -▁toll 1.0846254475405919e-05 -▁Zugangs 1.0842466052606266e-05 -onder 1.0838677629806614e-05 -▁RA 1.0838677629806614e-05 -▁geeignete 1.0838677629806614e-05 -ped 1.0834889207006961e-05 -▁Four 1.0834889207006961e-05 -▁Visa 1.0834889207006961e-05 -▁erzeugen 1.0834889207006961e-05 -▁pilot 1.0834889207006961e-05 -▁Parad 1.0831100784207308e-05 -▁besucht 1.0831100784207308e-05 -▁susp 1.0831100784207308e-05 -▁Andererseits 1.0827312361407656e-05 -▁Bö 1.0827312361407656e-05 -▁Zauber 1.0827312361407656e-05 -▁appropriations 1.0827312361407656e-05 -▁beenden 1.0827312361407656e-05 -▁dei 1.0827312361407656e-05 -▁nbsp 1.0827312361407656e-05 -gs 1.0823523938608002e-05 -▁Tickets 1.0823523938608002e-05 -▁diesbezüglich 1.0823523938608002e-05 -▁profits 1.0823523938608002e-05 -strasse 1.081973551580835e-05 -▁lin 1.081973551580835e-05 -▁51 1.0815947093008696e-05 -▁Area 1.0815947093008696e-05 -▁Haar 1.0815947093008696e-05 -▁meets 1.0815947093008696e-05 -▁servers 1.0815947093008696e-05 -arta 1.0812158670209044e-05 -▁Lern 1.0808370247409392e-05 -▁Szen 1.0808370247409392e-05 -▁compar 1.0808370247409392e-05 -▁entering 1.0808370247409392e-05 -▁sim 1.0808370247409392e-05 -▁Gelder 1.0804581824609738e-05 -▁Instruments 1.0804581824609738e-05 -▁Nikon 1.0804581824609738e-05 -erre 1.0800793401810086e-05 -▁Wesentlichen 1.0800793401810086e-05 -▁Wh 1.0800793401810086e-05 -▁eins 1.0800793401810086e-05 -▁griechischen 1.0800793401810086e-05 -▁promised 1.0800793401810086e-05 -▁unr 1.0800793401810086e-05 -▁Society 1.0797004979010433e-05 -atively 1.0793216556210781e-05 -zust 1.0793216556210781e-05 -▁Bug 1.0793216556210781e-05 -▁lasting 1.0793216556210781e-05 -rew 1.0785639710611475e-05 -▁Dresden 1.0785639710611475e-05 -▁Parlamente 1.0785639710611475e-05 -▁assign 1.0785639710611475e-05 -▁combines 1.0785639710611475e-05 -▁intends 1.0785639710611475e-05 -▁Beseitigung 1.0781851287811821e-05 -▁Länge 1.0781851287811821e-05 -▁integrierten 1.0781851287811821e-05 -▁recording 1.0781851287811821e-05 -▁wirken 1.0781851287811821e-05 -arl 1.0778062865012169e-05 -hop 1.0778062865012169e-05 -ikal 1.0778062865012169e-05 -▁Residence 1.0778062865012169e-05 -▁promise 1.0778062865012169e-05 -UP 1.0774274442212517e-05 -round 1.0774274442212517e-05 -▁Sonntag 1.0774274442212517e-05 -▁referring 1.0774274442212517e-05 -▁Ombudsman 1.0770486019412863e-05 -▁rechtliche 1.0770486019412863e-05 -▁opens 1.0766697596613211e-05 -▁prospects 1.0766697596613211e-05 -▁dial 1.0762909173813557e-05 -▁konsequent 1.0762909173813557e-05 -▁produces 1.0762909173813557e-05 -▁toward 1.0762909173813557e-05 -▁Laden 1.0759120751013905e-05 -▁Maschine 1.0759120751013905e-05 -▁Tools 1.0755332328214252e-05 -▁budgets 1.0755332328214252e-05 -▁gleichermaßen 1.0755332328214252e-05 -gebieten 1.07515439054146e-05 -iling 1.07515439054146e-05 -illed 1.07515439054146e-05 -▁Denk 1.0747755482614948e-05 -▁Tonnen 1.0747755482614948e-05 -▁bekannte 1.0747755482614948e-05 -▁soc 1.0747755482614948e-05 -▁urgently 1.0747755482614948e-05 -▁zulassen 1.0747755482614948e-05 -orgen 1.0743967059815294e-05 -äter 1.0743967059815294e-05 -▁efficiently 1.0743967059815294e-05 -prinzip 1.0740178637015642e-05 -▁iPod 1.0740178637015642e-05 -onom 1.0736390214215988e-05 -▁Eigentums 1.0736390214215988e-05 -▁Dam 1.0732601791416336e-05 -▁anzubieten 1.0732601791416336e-05 -▁bestand 1.0732601791416336e-05 -DS 1.0728813368616682e-05 -▁24. 1.0728813368616682e-05 -▁Dingen 1.0728813368616682e-05 -▁execution 1.0728813368616682e-05 -hafter 1.072502494581703e-05 -▁kilometres 1.072502494581703e-05 -linge 1.0721236523017376e-05 -▁ecological 1.0721236523017376e-05 -▁phenomenon 1.0721236523017376e-05 -▁verweisen 1.0721236523017376e-05 -iend 1.0717448100217724e-05 -▁Wiederaufbau 1.0717448100217724e-05 -▁Vertrieb 1.0713659677418072e-05 -ami 1.0709871254618419e-05 -▁Using 1.0709871254618419e-05 -▁mor 1.0709871254618419e-05 -▁Appl 1.0706082831818767e-05 -▁emotional 1.0706082831818767e-05 -▁sole 1.0706082831818767e-05 -mot 1.0702294409019113e-05 -▁arising 1.0702294409019113e-05 -▁gegenwärtige 1.0698505986219461e-05 -▁ländliche 1.0698505986219461e-05 -▁mistake 1.0698505986219461e-05 -▁Bewusstsein 1.0694717563419807e-05 -▁Drei 1.0694717563419807e-05 -▁Zucker 1.0694717563419807e-05 -▁cab 1.0694717563419807e-05 -▁feet 1.0694717563419807e-05 -▁relates 1.0694717563419807e-05 -bend 1.0690929140620155e-05 -eda 1.0690929140620155e-05 -edge 1.0690929140620155e-05 -spolitik 1.0690929140620155e-05 -öss 1.0690929140620155e-05 -▁EU-Mitgliedstaaten 1.0690929140620155e-05 -▁stake 1.0690929140620155e-05 -plans 1.0687140717820503e-05 -using 1.0687140717820503e-05 -▁Polit 1.0687140717820503e-05 -▁errichtet 1.0687140717820503e-05 -▁surprise 1.0687140717820503e-05 -igl 1.0683352295020849e-05 -ül 1.0683352295020849e-05 -iled 1.0679563872221197e-05 -▁Jack 1.0679563872221197e-05 -▁EU-R 1.0675775449421543e-05 -▁Renten 1.0675775449421543e-05 -▁geographical 1.0675775449421543e-05 -▁spreche 1.0675775449421543e-05 -freund 1.0671987026621891e-05 -jan 1.0671987026621891e-05 -miss 1.0671987026621891e-05 -▁Helsinki 1.0671987026621891e-05 -▁Parameter 1.0671987026621891e-05 -ners 1.0668198603822238e-05 -zeuge 1.0668198603822238e-05 -▁KDE 1.0668198603822238e-05 -▁Tauch 1.0668198603822238e-05 -▁camps 1.0668198603822238e-05 -schung 1.0664410181022586e-05 -▁calc 1.0664410181022586e-05 -UB 1.0660621758222932e-05 -▁AF 1.0660621758222932e-05 -▁Lastly 1.065683333542328e-05 -▁Wa 1.065683333542328e-05 -▁Energies 1.0653044912623628e-05 -▁Linien 1.0653044912623628e-05 -▁Verteidigungs 1.0653044912623628e-05 -▁auffordern 1.0653044912623628e-05 -▁essen 1.0653044912623628e-05 -bares 1.0649256489823974e-05 -▁Euros 1.0649256489823974e-05 -▁oldest 1.0649256489823974e-05 -▁stammt 1.0649256489823974e-05 -▁Alm 1.0645468067024322e-05 -▁native 1.0645468067024322e-05 -reibungen 1.0641679644224668e-05 -▁Katalog 1.0641679644224668e-05 -▁Maur 1.0641679644224668e-05 -▁VIP 1.0641679644224668e-05 -▁knapp 1.0641679644224668e-05 -▁lebt 1.0641679644224668e-05 -▁nom 1.0641679644224668e-05 -▁beruhen 1.0637891221425016e-05 -▁froh 1.0637891221425016e-05 -▁occurred 1.0637891221425016e-05 -▁scientists 1.0637891221425016e-05 -97 1.0634102798625362e-05 -▁Journal 1.0634102798625362e-05 -▁Zusammenhalt 1.0634102798625362e-05 -▁wheel 1.0634102798625362e-05 -▁Jugendlichen 1.063031437582571e-05 -▁medicine 1.063031437582571e-05 -▁Have 1.0626525953026058e-05 -▁Höchst 1.0626525953026058e-05 -▁Phot 1.0626525953026058e-05 -▁convert 1.0626525953026058e-05 -oco 1.0622737530226405e-05 -▁Benutzung 1.0622737530226405e-05 -▁Stad 1.0622737530226405e-05 -▁sky 1.0622737530226405e-05 -▁stärkere 1.0622737530226405e-05 -▁acquired 1.0618949107426753e-05 -▁candidates 1.0618949107426753e-05 -▁strictly 1.0618949107426753e-05 -▁untersuchen 1.0618949107426753e-05 -osc 1.0615160684627099e-05 -projekte 1.0615160684627099e-05 -ral 1.0615160684627099e-05 -▁ausgewählt 1.0615160684627099e-05 -▁mouse 1.0615160684627099e-05 -▁records 1.0615160684627099e-05 -▁Sports 1.0611372261827447e-05 -▁hills 1.0611372261827447e-05 -▁tel 1.0611372261827447e-05 -hlen 1.0607583839027793e-05 -weite 1.0607583839027793e-05 -▁highlighted 1.0607583839027793e-05 -▁Fac 1.0603795416228141e-05 -▁depth 1.0603795416228141e-05 -▁seats 1.0603795416228141e-05 -ici 1.0600006993428489e-05 -▁gekennzeichnet 1.0600006993428489e-05 -▁mental 1.0600006993428489e-05 --and 1.0596218570628835e-05 -▁Käufer 1.0596218570628835e-05 -▁negativen 1.0596218570628835e-05 -▁succeeded 1.0596218570628835e-05 -etta 1.0592430147829183e-05 -▁Strecke 1.0592430147829183e-05 -▁Ross 1.0588641725029529e-05 -▁Smart 1.0588641725029529e-05 -▁ide 1.0588641725029529e-05 -▁wel 1.0588641725029529e-05 -gern 1.0584853302229877e-05 -▁meals 1.0584853302229877e-05 -▁schätzen 1.0584853302229877e-05 --off 1.0581064879430224e-05 -▁Equ 1.0581064879430224e-05 -▁liable 1.0581064879430224e-05 -icity 1.0577276456630572e-05 -s. 1.0577276456630572e-05 -▁Religion 1.0577276456630572e-05 -▁unentgeltlich 1.0577276456630572e-05 -eams 1.0573488033830918e-05 -▁Branchen 1.0573488033830918e-05 -▁resulted 1.0573488033830918e-05 -wenden 1.0569699611031266e-05 -▁sah 1.0569699611031266e-05 -▁sophisticated 1.0569699611031266e-05 -uous 1.0565911188231614e-05 -upt 1.0565911188231614e-05 -▁Samstag 1.0565911188231614e-05 -▁beer 1.0565911188231614e-05 -▁essentially 1.0565911188231614e-05 -▁specialists 1.0565911188231614e-05 -▁Industr 1.056212276543196e-05 -▁Vermittlungs 1.056212276543196e-05 -▁tim 1.056212276543196e-05 -A. 1.0558334342632308e-05 -ented 1.0558334342632308e-05 -▁Sn 1.0558334342632308e-05 -▁Call 1.0554545919832654e-05 -▁taxi 1.0554545919832654e-05 -▁Frühjahr 1.0550757497033002e-05 -▁Further 1.0550757497033002e-05 -▁consequence 1.0550757497033002e-05 -▁Š 1.0550757497033002e-05 -▁о 1.0550757497033002e-05 -idence 1.0546969074233349e-05 -zunehmen 1.0546969074233349e-05 -▁Bedürfnissen 1.0546969074233349e-05 -ibel 1.0543180651433696e-05 -▁Rechnungs 1.0543180651433696e-05 -▁evident 1.0543180651433696e-05 -33 1.0539392228634044e-05 -KT 1.0539392228634044e-05 -▁Performance 1.0539392228634044e-05 -▁eventuell 1.0539392228634044e-05 -▁micr 1.0539392228634044e-05 -▁operator 1.0539392228634044e-05 -▁Jesus 1.0535603805834391e-05 -▁Schreiben 1.0535603805834391e-05 -▁hence 1.0535603805834391e-05 -izz 1.0528026960235085e-05 -rain 1.0528026960235085e-05 -▁Sierra 1.0528026960235085e-05 -▁Schwe 1.0524238537435433e-05 -▁complement 1.0524238537435433e-05 -▁mel 1.0524238537435433e-05 -▁Baby 1.0520450114635779e-05 -▁commission 1.0520450114635779e-05 -▁conventional 1.0520450114635779e-05 -▁denjenigen 1.0520450114635779e-05 -▁nieder 1.0520450114635779e-05 -adt 1.0516661691836127e-05 -üm 1.0516661691836127e-05 -▁Versorgung 1.0516661691836127e-05 -▁beinhalten 1.0516661691836127e-05 -ault 1.0512873269036473e-05 -vol 1.0512873269036473e-05 -▁Joint 1.0512873269036473e-05 -▁Journalisten 1.0512873269036473e-05 -▁affects 1.0512873269036473e-05 -▁vorgestellt 1.0512873269036473e-05 -96 1.0509084846236821e-05 -▁Lehrer 1.0509084846236821e-05 -▁Todesstrafe 1.0509084846236821e-05 -▁derzeitige 1.0509084846236821e-05 -▁heading 1.0509084846236821e-05 -▁incorporated 1.0509084846236821e-05 -▁streng 1.0509084846236821e-05 -▁virtually 1.0509084846236821e-05 -▁Suites 1.0505296423437169e-05 -arily 1.0501508000637516e-05 -etzung 1.0501508000637516e-05 -▁airports 1.0497719577837863e-05 -aka 1.049393115503821e-05 -wissenschaft 1.049393115503821e-05 -üren 1.049393115503821e-05 -▁Europ 1.049393115503821e-05 -▁Sicherheitspolitik 1.049393115503821e-05 -▁aviation 1.049393115503821e-05 -▁hierzu 1.049393115503821e-05 -▁Spielen 1.0490142732238558e-05 -▁upgrade 1.0486354309438904e-05 --a 1.0482565886639252e-05 -▁Ablauf 1.0482565886639252e-05 -▁Prin 1.04787774638396e-05 -▁gesp 1.04787774638396e-05 -planung 1.0474989041039946e-05 -▁Bevor 1.0474989041039946e-05 -▁Ham 1.0474989041039946e-05 -▁fly 1.0474989041039946e-05 -gas 1.0471200618240294e-05 -ject 1.0471200618240294e-05 -ober 1.0471200618240294e-05 -uce 1.0471200618240294e-05 -ń 1.0471200618240294e-05 -▁4-star 1.0471200618240294e-05 -▁72 1.0471200618240294e-05 -▁Sex 1.0471200618240294e-05 -▁choices 1.0471200618240294e-05 -▁kontrolliert 1.0471200618240294e-05 -oler 1.046741219544064e-05 -ę 1.046741219544064e-05 -▁Klass 1.046741219544064e-05 -▁considers 1.046741219544064e-05 -▁Entw 1.0463623772640988e-05 -▁gradually 1.0463623772640988e-05 -onym 1.0459835349841335e-05 -▁Berge 1.0459835349841335e-05 -▁offizielle 1.0459835349841335e-05 -inent 1.0456046927041682e-05 -▁App 1.0456046927041682e-05 -▁diplomatic 1.0456046927041682e-05 -▁entscheidenden 1.0456046927041682e-05 -cz 1.0452258504242029e-05 -läss 1.0452258504242029e-05 -▁Mexiko 1.0452258504242029e-05 -▁Vertrages 1.0452258504242029e-05 -▁eigener 1.0452258504242029e-05 -▁gedacht 1.0452258504242029e-05 -▁hidden 1.0452258504242029e-05 -▁rid 1.0452258504242029e-05 -▁senken 1.0452258504242029e-05 --Ein 1.0448470081442377e-05 -emble 1.0448470081442377e-05 -stellungs 1.0448470081442377e-05 -▁stick 1.0444681658642725e-05 -▁Bereits 1.0440893235843071e-05 -▁legend 1.0440893235843071e-05 -▁Ju 1.0437104813043419e-05 -▁verschieden 1.0437104813043419e-05 --Al 1.0433316390243765e-05 -pm 1.0433316390243765e-05 -uth 1.0433316390243765e-05 -▁Maus 1.0433316390243765e-05 -TP 1.0429527967444113e-05 -schte 1.0429527967444113e-05 -▁Dänemark 1.0429527967444113e-05 -▁flu 1.0429527967444113e-05 -ählt 1.0425739544644459e-05 -▁environmentally 1.0425739544644459e-05 -▁voc 1.0425739544644459e-05 -echt 1.0421951121844807e-05 -eins 1.0421951121844807e-05 -▁Kooperations 1.0421951121844807e-05 -▁associations 1.0421951121844807e-05 -▁critic 1.0421951121844807e-05 -▁halb 1.0421951121844807e-05 -▁Verkaufs 1.0418162699045155e-05 -▁ec 1.0418162699045155e-05 -▁Cla 1.0414374276245502e-05 -▁Trib 1.0414374276245502e-05 -▁Umweltfragen 1.0414374276245502e-05 -▁absol 1.0414374276245502e-05 -▁causing 1.0414374276245502e-05 -▁mögen 1.0414374276245502e-05 -▁warning 1.0414374276245502e-05 -▁smart 1.0410585853445849e-05 -▁¼ 1.0410585853445849e-05 -▁Beachtung 1.0406797430646196e-05 -▁Rechnungshof 1.0406797430646196e-05 -▁renoviert 1.0406797430646196e-05 -aß 1.0403009007846544e-05 -ehl 1.0403009007846544e-05 -gent 1.0403009007846544e-05 -▁Auftrags 1.0403009007846544e-05 -▁Cooperation 1.0403009007846544e-05 -▁Kaiser 1.0403009007846544e-05 -▁bedeutenden 1.0403009007846544e-05 -kle 1.039922058504689e-05 -▁Dialogs 1.039922058504689e-05 -▁Gross 1.039922058504689e-05 -▁Sobald 1.039922058504689e-05 -▁abgel 1.039922058504689e-05 -▁praktischen 1.039922058504689e-05 -kund 1.0395432162247238e-05 -▁Zuverlässigkeit 1.0395432162247238e-05 -▁realer 1.0395432162247238e-05 -▁Snow 1.0391643739447586e-05 -▁blocks 1.0391643739447586e-05 -▁producer 1.0391643739447586e-05 -▁Metr 1.0387855316647932e-05 -▁Systemen 1.0387855316647932e-05 -▁befasst 1.0387855316647932e-05 -▁festzustellen 1.0387855316647932e-05 -▁kümmern 1.0387855316647932e-05 -▁verletzt 1.0387855316647932e-05 -flächen 1.038406689384828e-05 -▁Department 1.038406689384828e-05 -▁Ster 1.038406689384828e-05 -▁Wür 1.038406689384828e-05 -▁sphere 1.038406689384828e-05 -▁teachers 1.038406689384828e-05 -▁Mechanismen 1.0380278471048626e-05 -▁Urheber 1.0380278471048626e-05 -▁Venedig 1.0380278471048626e-05 -▁reisen 1.0380278471048626e-05 -lassen 1.0376490048248974e-05 -▁Mode 1.0376490048248974e-05 -▁Vice-President 1.0376490048248974e-05 -▁explanation 1.0376490048248974e-05 -▁isol 1.0376490048248974e-05 -▁vergleich 1.0376490048248974e-05 -▁CEO 1.0372701625449321e-05 -▁Level 1.0372701625449321e-05 -▁Verfassungs 1.0372701625449321e-05 -▁steuer 1.0372701625449321e-05 -zed 1.0368913202649669e-05 -▁Constitutional 1.0368913202649669e-05 -▁ET 1.0368913202649669e-05 -▁Lesen 1.0368913202649669e-05 -▁Ü 1.0368913202649669e-05 -boden 1.0365124779850015e-05 -oven 1.0365124779850015e-05 -▁ON 1.0365124779850015e-05 -▁ordnungsgemäß 1.0365124779850015e-05 -▁Wied 1.0361336357050363e-05 -▁Wissenschaftler 1.0361336357050363e-05 -▁anzunehmen 1.0361336357050363e-05 -recher 1.0357547934250711e-05 -▁Explorer 1.0357547934250711e-05 -▁Register 1.0357547934250711e-05 -▁mot 1.0357547934250711e-05 -▁troops 1.0357547934250711e-05 -▁Dreh 1.0353759511451057e-05 -▁Hindernisse 1.0353759511451057e-05 -▁Jon 1.0353759511451057e-05 -▁widespread 1.0353759511451057e-05 --g 1.0349971088651405e-05 -des 1.0349971088651405e-05 -▁Danish 1.0349971088651405e-05 -▁Labor 1.0349971088651405e-05 -ativer 1.0346182665851751e-05 -ention 1.0346182665851751e-05 -onial 1.0346182665851751e-05 -▁GAP 1.0346182665851751e-05 -▁Three 1.0346182665851751e-05 -▁attached 1.0346182665851751e-05 -▁polit 1.0346182665851751e-05 -▁Ägypten 1.0346182665851751e-05 -▁Fläche 1.0342394243052099e-05 -▁Nokia 1.0342394243052099e-05 -▁minor 1.0342394243052099e-05 -▁offenbar 1.0342394243052099e-05 -iene 1.0338605820252445e-05 -▁encouraged 1.0338605820252445e-05 -▁revenue 1.0338605820252445e-05 -IE 1.0334817397452793e-05 -edi 1.0334817397452793e-05 -ehmen 1.0334817397452793e-05 -▁müsste 1.0334817397452793e-05 -ART 1.0331028974653141e-05 -eich 1.0331028974653141e-05 -ilder 1.0331028974653141e-05 -iva 1.0331028974653141e-05 -▁Franz 1.0331028974653141e-05 -▁bezahlt 1.0331028974653141e-05 -▁vertra 1.0331028974653141e-05 -gerät 1.0327240551853488e-05 -Ö 1.0327240551853488e-05 -▁ends 1.0327240551853488e-05 -▁Hinweise 1.0323452129053836e-05 -igo 1.0319663706254182e-05 -▁Children 1.0319663706254182e-05 -▁berechnet 1.0319663706254182e-05 -▁dance 1.0319663706254182e-05 -▁Internal 1.031587528345453e-05 -▁Sem 1.031587528345453e-05 -▁restricted 1.031587528345453e-05 -▁surprised 1.031587528345453e-05 -▁Mün 1.0312086860654876e-05 -▁Nik 1.0312086860654876e-05 -▁bür 1.0312086860654876e-05 -DP 1.0308298437855224e-05 -down 1.0308298437855224e-05 -entwick 1.0308298437855224e-05 -▁prisoners 1.0308298437855224e-05 -▁soil 1.0308298437855224e-05 -fers 1.030451001505557e-05 -sprach 1.030451001505557e-05 -ukt 1.030451001505557e-05 -ре 1.030451001505557e-05 -▁Ter 1.030451001505557e-05 -▁brachte 1.030451001505557e-05 -▁produzieren 1.030451001505557e-05 -▁wettbewerbs 1.030451001505557e-05 -▁adding 1.0300721592255918e-05 -▁participating 1.0300721592255918e-05 -▁reporting 1.0300721592255918e-05 -grenzen 1.0296933169456266e-05 -regelung 1.0296933169456266e-05 -▁Ausgang 1.0296933169456266e-05 -▁Senkung 1.0296933169456266e-05 -▁colors 1.0296933169456266e-05 -▁typischen 1.0296933169456266e-05 -CD 1.0293144746656612e-05 -▁Vertretern 1.0293144746656612e-05 -▁Wirksamkeit 1.0293144746656612e-05 -▁accidents 1.0293144746656612e-05 -▁seeks 1.0293144746656612e-05 -cil 1.028935632385696e-05 -konferenz 1.028935632385696e-05 -kons 1.028935632385696e-05 -uite 1.028935632385696e-05 -▁contextual 1.028935632385696e-05 -kel 1.0285567901057307e-05 -▁Roth 1.0285567901057307e-05 -▁angemessenen 1.0285567901057307e-05 -▁bomb 1.0285567901057307e-05 -▁puts 1.0285567901057307e-05 -ikel 1.0281779478257655e-05 -▁Appartements 1.0281779478257655e-05 -▁GP 1.0281779478257655e-05 -▁engage 1.0281779478257655e-05 -gra 1.0277991055458e-05 -▁Kohäsionspolitik 1.0277991055458e-05 -▁appl 1.0277991055458e-05 -gehende 1.0274202632658349e-05 -▁Grün 1.0274202632658349e-05 -▁Premierminister 1.0274202632658349e-05 -▁Schlacht 1.0274202632658349e-05 -▁Bü 1.0270414209858697e-05 -▁Bücher 1.0270414209858697e-05 -▁wechseln 1.0270414209858697e-05 -aken 1.0266625787059043e-05 -flow 1.0266625787059043e-05 -▁Aud 1.0266625787059043e-05 -▁Bildschirm 1.0266625787059043e-05 -▁IMF 1.0266625787059043e-05 -▁Obst 1.0266625787059043e-05 -▁geregelt 1.0266625787059043e-05 -ilation 1.0262837364259391e-05 -▁hal 1.0262837364259391e-05 -setzt 1.0259048941459737e-05 -▁Ergebnissen 1.0259048941459737e-05 -▁blo 1.0259048941459737e-05 -▁wooden 1.0259048941459737e-05 -rd 1.0255260518660085e-05 -ulen 1.0255260518660085e-05 -▁niedrig 1.0255260518660085e-05 --Re 1.0251472095860431e-05 -ourse 1.0251472095860431e-05 -ply 1.0251472095860431e-05 -▁22. 1.0251472095860431e-05 -ichtete 1.0247683673060779e-05 -▁alongside 1.0247683673060779e-05 -▁stelle 1.0247683673060779e-05 -▁Ansprüche 1.0243895250261127e-05 -▁Terms 1.0243895250261127e-05 -▁clarify 1.0243895250261127e-05 -▁erklärte 1.0243895250261127e-05 -▁fach 1.0243895250261127e-05 -raten 1.0240106827461474e-05 -▁20th 1.0240106827461474e-05 -▁Anwendungs 1.0240106827461474e-05 -▁Besuchen 1.0240106827461474e-05 -▁Conc 1.0240106827461474e-05 -▁Events 1.0240106827461474e-05 -▁incl 1.0240106827461474e-05 -▁utmost 1.0240106827461474e-05 -ivers 1.0236318404661822e-05 -undes 1.0236318404661822e-05 -▁Authority 1.0236318404661822e-05 -▁richtet 1.0236318404661822e-05 -▁stories 1.0236318404661822e-05 -wegen 1.0232529981862168e-05 -▁Schlaf 1.0232529981862168e-05 -▁Spezialitäten 1.0232529981862168e-05 -▁coun 1.0232529981862168e-05 -▁inspir 1.0232529981862168e-05 -örder 1.0228741559062516e-05 -▁Wohnzimmer 1.0228741559062516e-05 -▁marks 1.0228741559062516e-05 -ictions 1.0224953136262862e-05 -änger 1.0224953136262862e-05 -▁agents 1.0224953136262862e-05 -▁angewandt 1.0224953136262862e-05 -▁stil 1.0224953136262862e-05 -rechten 1.022116471346321e-05 -▁bug 1.022116471346321e-05 -ässt 1.0217376290663556e-05 -то 1.0217376290663556e-05 -▁Finnland 1.0217376290663556e-05 -▁Hu 1.0217376290663556e-05 -▁Verwirklichung 1.0217376290663556e-05 -▁infolge 1.0217376290663556e-05 -▁prize 1.0217376290663556e-05 -versammlung 1.0213587867863904e-05 -ünden 1.0213587867863904e-05 -▁graphics 1.0213587867863904e-05 -27 1.0209799445064252e-05 -ensive 1.0209799445064252e-05 -▁Fahrer 1.0209799445064252e-05 -▁Komp 1.0209799445064252e-05 -▁clock 1.0209799445064252e-05 -▁irischen 1.0209799445064252e-05 -▁trib 1.0209799445064252e-05 -fter 1.0206011022264598e-05 -▁grown 1.0206011022264598e-05 -anken 1.0202222599464946e-05 -▁Aufzug 1.0202222599464946e-05 -▁Fang 1.0202222599464946e-05 -▁vorn 1.0202222599464946e-05 -atorium 1.0198434176665293e-05 -gendes 1.0198434176665293e-05 -▁Int 1.0198434176665293e-05 -▁RSS 1.0198434176665293e-05 -▁älteren 1.0198434176665293e-05 -▁Long 1.0194645753865641e-05 -▁aktiven 1.0194645753865641e-05 -▁serving 1.0194645753865641e-05 -anforderungen 1.0190857331065987e-05 -plant 1.0190857331065987e-05 -▁Florence 1.0190857331065987e-05 -▁achievement 1.0190857331065987e-05 -▁harmful 1.0190857331065987e-05 -▁langfristigen 1.0190857331065987e-05 -▁observed 1.0190857331065987e-05 -▁wachsenden 1.0190857331065987e-05 -▁wesentlicher 1.0190857331065987e-05 -▁BR 1.0187068908266335e-05 -▁Datum 1.0187068908266335e-05 -▁Treaties 1.0187068908266335e-05 -▁gefährdet 1.0187068908266335e-05 -▁preserve 1.0187068908266335e-05 -▁quote 1.0187068908266335e-05 -▁soeben 1.0183280485466683e-05 -▁Leipzig 1.0179492062667029e-05 -▁ausgegeben 1.0179492062667029e-05 -▁indoor 1.0179492062667029e-05 -▁jedenfalls 1.0179492062667029e-05 -aser 1.0175703639867377e-05 -rich 1.0175703639867377e-05 -▁Drittel 1.0175703639867377e-05 -▁Jerusalem 1.0175703639867377e-05 -▁XML 1.0175703639867377e-05 -▁spite 1.0175703639867377e-05 -▁stayed 1.0175703639867377e-05 -FT 1.0171915217067723e-05 -teilung 1.0171915217067723e-05 -ariat 1.0168126794268071e-05 -▁Firefox 1.0168126794268071e-05 -▁restrict 1.0168126794268071e-05 -▁specialized 1.0168126794268071e-05 -probleme 1.0164338371468417e-05 -▁Operation 1.0160549948668765e-05 -▁Ring 1.0160549948668765e-05 -▁effektive 1.0160549948668765e-05 -▁müssten 1.0160549948668765e-05 -▁pillar 1.0160549948668765e-05 -▁politisches 1.0160549948668765e-05 -reff 1.0156761525869112e-05 -▁Anmeldung 1.0156761525869112e-05 -▁Install 1.0156761525869112e-05 -▁permission 1.0156761525869112e-05 -▁Übertragung 1.0156761525869112e-05 -fühl 1.015297310306946e-05 -itis 1.015297310306946e-05 -▁Werb 1.015297310306946e-05 -▁capabilities 1.015297310306946e-05 -▁favor 1.015297310306946e-05 -▁permit 1.015297310306946e-05 -▁quotas 1.015297310306946e-05 -usst 1.0149184680269808e-05 -▁Safe 1.0149184680269808e-05 -▁recognize 1.0149184680269808e-05 -▁Aussage 1.0145396257470154e-05 -▁Automobil 1.0145396257470154e-05 -▁Edinburgh 1.0145396257470154e-05 -▁Norm 1.0145396257470154e-05 -▁Oberfläche 1.0145396257470154e-05 -▁illegale 1.0145396257470154e-05 -▁Entfernung 1.0141607834670502e-05 -▁know-how 1.0141607834670502e-05 -▁OK 1.0137819411870848e-05 -▁een 1.0137819411870848e-05 -▁23. 1.0130242566271542e-05 -▁anzuwenden 1.0130242566271542e-05 -▁wirkt 1.0130242566271542e-05 -▁conv 1.012645414347189e-05 -wick 1.0122665720672238e-05 -▁'m 1.0122665720672238e-05 -▁convention 1.0122665720672238e-05 -▁convergence 1.0122665720672238e-05 -▁dauert 1.0122665720672238e-05 -wirken 1.0118877297872584e-05 -▁Airlines 1.0115088875072932e-05 -▁Amerikaner 1.0115088875072932e-05 -▁soul 1.0115088875072932e-05 -▁maximal 1.0111300452273279e-05 -▁purely 1.0111300452273279e-05 --ch 1.0107512029473627e-05 -▁passing 1.0107512029473627e-05 -endet 1.0103723606673973e-05 -oom 1.0103723606673973e-05 -oses 1.0103723606673973e-05 -▁SM 1.0103723606673973e-05 -▁Einen 1.0099935183874321e-05 -▁Präsenz 1.0099935183874321e-05 -▁Ts 1.0099935183874321e-05 -arbeiter 1.0096146761074667e-05 -irm 1.0096146761074667e-05 -▁Tyrol 1.0096146761074667e-05 -tragen 1.0092358338275015e-05 -▁armed 1.0092358338275015e-05 -▁ausgewogen 1.0092358338275015e-05 -▁steigern 1.0092358338275015e-05 -▁dispute 1.0088569915475363e-05 -schuss 1.0084781492675709e-05 -var 1.0084781492675709e-05 -▁Vorrang 1.0084781492675709e-05 -▁conferences 1.0084781492675709e-05 -▁supplement 1.0084781492675709e-05 -ж 1.0080993069876057e-05 -▁Stre 1.0080993069876057e-05 -▁dat 1.0080993069876057e-05 -▁effiziente 1.0080993069876057e-05 -▁seconds 1.0077204647076404e-05 -▁weisen 1.0077204647076404e-05 -achtung 1.0073416224276751e-05 -▁Handy 1.0073416224276751e-05 -▁Haupts 1.0073416224276751e-05 -ipped 1.0069627801477098e-05 -ackt 1.0065839378677446e-05 -ader 1.0065839378677446e-05 -steigen 1.0065839378677446e-05 -▁Errichtung 1.0065839378677446e-05 -▁Id 1.0065839378677446e-05 -▁Mandat 1.0065839378677446e-05 -▁Premium 1.0065839378677446e-05 -etern 1.0062050955877794e-05 -ipl 1.0062050955877794e-05 -▁Ast 1.0062050955877794e-05 -▁Chile 1.0062050955877794e-05 -▁Minderheit 1.0062050955877794e-05 -▁Prozesses 1.0062050955877794e-05 -▁Staats- 1.0062050955877794e-05 -▁Zy 1.0062050955877794e-05 -▁beeinflussen 1.0062050955877794e-05 -▁inspired 1.0062050955877794e-05 -▁Strände 1.005826253307814e-05 -oß 1.0054474110278488e-05 -ups 1.0054474110278488e-05 -äufe 1.0054474110278488e-05 -▁Modernisierung 1.0054474110278488e-05 -▁freut 1.0054474110278488e-05 -fahrer 1.0050685687478834e-05 -ouses 1.0050685687478834e-05 -▁estimated 1.0050685687478834e-05 -▁orient 1.0050685687478834e-05 -aktivitäten 1.0046897264679182e-05 -▁Siemens 1.0046897264679182e-05 -▁communicate 1.0046897264679182e-05 -▁manifest 1.0046897264679182e-05 -vorschlag 1.0039320419079876e-05 -▁Kop 1.0039320419079876e-05 -▁ausüben 1.0039320419079876e-05 -▁composition 1.0035531996280224e-05 -▁hervorragend 1.0035531996280224e-05 -active 1.0031743573480571e-05 -▁Regulierung 1.0031743573480571e-05 -▁flows 1.0031743573480571e-05 -▁humanitären 1.0027955150680918e-05 -▁Benutzern 1.0024166727881265e-05 -▁Cuba 1.0024166727881265e-05 -▁berühmt 1.0024166727881265e-05 -▁schwierige 1.0024166727881265e-05 -view 1.0020378305081613e-05 -▁Truppen 1.0020378305081613e-05 -▁festlegen 1.0020378305081613e-05 -uppe 1.0016589882281959e-05 -▁Burma 1.0016589882281959e-05 -▁oftmals 1.0016589882281959e-05 -center 1.0012801459482307e-05 -▁Tab 1.0012801459482307e-05 -ühr 1.0009013036682653e-05 -▁Extra 1.0009013036682653e-05 -▁Vorstellungen 1.0009013036682653e-05 -▁Betreuung 1.0005224613883001e-05 -▁Innovationen 1.0005224613883001e-05 -▁Rub 1.0005224613883001e-05 -▁coherent 1.0005224613883001e-05 -▁terr 1.0005224613883001e-05 -gestaltung 9.9976477682836953e-06 -lig 9.9976477682836953e-06 -▁Finde 9.9976477682836953e-06 -DR 9.9862824998847375e-06 -o- 9.9862824998847375e-06 -▁Fan 9.9862824998847375e-06 -▁Source 9.9862824998847375e-06 -▁Tele 9.9862824998847375e-06 -▁dollar 9.9862824998847375e-06 -▁europäisches 9.9862824998847375e-06 -▁teach 9.9862824998847375e-06 -▁wirkliche 9.9862824998847375e-06 -für 9.9824940770850838e-06 -▁1,5 9.9824940770850838e-06 -▁integrate 9.9824940770850838e-06 --day 9.9787056542854317e-06 -essed 9.9787056542854317e-06 -▁airlines 9.9787056542854317e-06 -regierung 9.97491723148578e-06 -tain 9.97491723148578e-06 -▁Contact 9.97491723148578e-06 -▁Muster 9.97491723148578e-06 -▁developers 9.97491723148578e-06 -▁Übersicht 9.97491723148578e-06 -anen 9.9711288086861259e-06 -▁Subject 9.9711288086861259e-06 -▁busy 9.9711288086861259e-06 -▁flag 9.9711288086861259e-06 -▁maxim 9.9711288086861259e-06 -BA 9.9673403858864739e-06 -▁LAN 9.9673403858864739e-06 -▁initially 9.9673403858864739e-06 -▁vorliegen 9.9673403858864739e-06 -▁zeichnet 9.9673403858864739e-06 -▁Innovations 9.96355196308682e-06 -▁interess 9.96355196308682e-06 -▁schm 9.96355196308682e-06 -amente 9.9597635402871681e-06 -iberal 9.9597635402871681e-06 -▁Außen- 9.9597635402871681e-06 -▁adult 9.9597635402871681e-06 -▁Bell 9.9559751174875144e-06 -▁Teams 9.9559751174875144e-06 -▁victory 9.9559751174875144e-06 -▁Jazz 9.9521866946878623e-06 -leb 9.9483982718882086e-06 -▁Major 9.9483982718882086e-06 -▁Solar 9.9483982718882086e-06 -▁Versammlung 9.9483982718882086e-06 -▁percentage 9.9483982718882086e-06 -alled 9.9446098490885565e-06 -uern 9.9446098490885565e-06 -▁abgegeben 9.9446098490885565e-06 -▁ansehen 9.9446098490885565e-06 -▁automat 9.9446098490885565e-06 -▁zweitens 9.9446098490885565e-06 --Software 9.9408214262889045e-06 -artige 9.9408214262889045e-06 -icke 9.9408214262889045e-06 -ktionen 9.9408214262889045e-06 -▁Engineering 9.9408214262889045e-06 -▁Mond 9.9408214262889045e-06 -▁raising 9.9408214262889045e-06 -▁thorough 9.9408214262889045e-06 -SC 9.93703300348925e-06 -▁stores 9.93703300348925e-06 -ishes 9.9332445806895987e-06 -▁Ratschlag 9.9332445806895987e-06 -▁diff 9.9332445806895987e-06 -▁string 9.9332445806895987e-06 -rating 9.929456157889945e-06 -▁Eind 9.929456157889945e-06 -▁Weiterentwicklung 9.929456157889945e-06 -▁Wood 9.929456157889945e-06 -▁geschäft 9.929456157889945e-06 -▁Mädchen 9.9256677350902929e-06 -▁consum 9.9256677350902929e-06 -▁vorzunehmen 9.9256677350902929e-06 -ications 9.92187931229064e-06 -▁Camping 9.92187931229064e-06 -▁italienische 9.92187931229064e-06 -ourn 9.9180908894909871e-06 -▁Jedoch 9.9180908894909871e-06 -▁clim 9.9180908894909871e-06 -▁momentan 9.9180908894909871e-06 -BS 9.9143024666913351e-06 -▁Genau 9.9143024666913351e-06 -▁mill 9.9143024666913351e-06 -▁zurückzu 9.9105140438916814e-06 -UL 9.90672562109203e-06 -▁Riviera 9.90672562109203e-06 -▁medieval 9.90672562109203e-06 -issions 9.9029371982923756e-06 -▁Arc 9.9029371982923756e-06 -▁Ausbau 9.9029371982923756e-06 -▁behält 9.9029371982923756e-06 -▁lebend 9.9029371982923756e-06 -▁wachsen 9.9029371982923756e-06 -▁Ausarbeitung 9.8991487754927235e-06 -▁Fast 9.8991487754927235e-06 -▁s. 9.8991487754927235e-06 -iale 9.89536035269307e-06 -kan 9.89536035269307e-06 -▁brea 9.89536035269307e-06 -▁kennt 9.89536035269307e-06 -▁maybe 9.89536035269307e-06 -▁multi 9.89536035269307e-06 -▁sufficiently 9.89536035269307e-06 -ived 9.8915719298934178e-06 -▁Pak 9.8915719298934178e-06 -▁Räume 9.8915719298934178e-06 -▁accounting 9.8915719298934178e-06 -▁operates 9.8915719298934178e-06 --star 9.8877835070937657e-06 -inet 9.8877835070937657e-06 -▁Wirk 9.8877835070937657e-06 -▁abst 9.8877835070937657e-06 -▁hopes 9.8877835070937657e-06 -▁Betriebe 9.883995084294112e-06 -▁trat 9.883995084294112e-06 -▁Got 9.88020666149446e-06 -▁Palma 9.88020666149446e-06 -imin 9.8764182386948062e-06 -▁Stiftung 9.8764182386948062e-06 -▁Doppelzimmer 9.8726298158951541e-06 -▁Normal 9.8726298158951541e-06 -▁Trend 9.8726298158951541e-06 -▁B5-0 9.8688413930955e-06 -▁Fot 9.8688413930955e-06 --Service 9.8650529702958484e-06 -▁angenehmen 9.8650529702958484e-06 -griffen 9.8612645474961946e-06 -losigkeit 9.8612645474961946e-06 -▁fehlen 9.8612645474961946e-06 -oria 9.8574761246965426e-06 -rü 9.8574761246965426e-06 -▁Multi 9.8574761246965426e-06 -▁promises 9.8574761246965426e-06 -▁prospect 9.8574761246965426e-06 -▁sec 9.8574761246965426e-06 -▁1988 9.85368770189689e-06 -▁absurd 9.85368770189689e-06 -▁ear 9.85368770189689e-06 -▁Gebühren 9.8498992790972368e-06 -▁erkannt 9.8498992790972368e-06 -▁expense 9.8498992790972368e-06 -▁Flughäfen 9.8461108562975847e-06 -▁professionelle 9.8461108562975847e-06 --Bericht 9.842322433497931e-06 --Pro 9.842322433497931e-06 -iga 9.842322433497931e-06 -uchs 9.842322433497931e-06 -ützt 9.842322433497931e-06 -▁12.00 9.842322433497931e-06 -▁Dritt 9.842322433497931e-06 -▁GR 9.842322433497931e-06 -▁Psych 9.842322433497931e-06 -fil 9.838534010698279e-06 -agg 9.8347455878986252e-06 -▁Desktop 9.8347455878986252e-06 -▁Sof 9.8347455878986252e-06 -▁Sä 9.8347455878986252e-06 -▁guided 9.8347455878986252e-06 -klasse 9.8309571650989732e-06 -▁edited 9.8309571650989732e-06 -▁enjoys 9.8309571650989732e-06 -▁sammeln 9.8309571650989732e-06 -dauer 9.8271687422993211e-06 -ember 9.8271687422993211e-06 -ο 9.8271687422993211e-06 -▁Optim 9.8271687422993211e-06 -▁Traditionen 9.8271687422993211e-06 -▁gewählten 9.8271687422993211e-06 -▁offiziell 9.8271687422993211e-06 -quellen 9.8233803194996674e-06 -▁Bibliothek 9.8233803194996674e-06 -▁survey 9.8233803194996674e-06 --10 9.8195918967000154e-06 -` 9.8195918967000154e-06 -eater 9.8195918967000154e-06 -wertung 9.8195918967000154e-06 -far 9.81201505110071e-06 -ängt 9.81201505110071e-06 -▁AKP 9.81201505110071e-06 -▁shortly 9.81201505110071e-06 -▁Dan 9.8082266283010558e-06 -▁recall 9.8082266283010558e-06 -ping 9.8044382055014038e-06 -▁AU 9.8044382055014038e-06 -▁NICHT 9.8044382055014038e-06 -▁Valencia 9.8044382055014038e-06 -ë 9.80064978270175e-06 -▁seemed 9.80064978270175e-06 -▁Index 9.796861359902098e-06 -▁kostenfrei 9.796861359902098e-06 -▁kill 9.793072937102446e-06 -▁sou 9.793072937102446e-06 -▁studied 9.793072937102446e-06 -INE 9.7892845143027922e-06 -gericht 9.7892845143027922e-06 -▁Copyright 9.7892845143027922e-06 -▁Zuge 9.7892845143027922e-06 -▁acceler 9.7892845143027922e-06 -▁adventure 9.7892845143027922e-06 -▁versteht 9.7892845143027922e-06 -▁wunder 9.7892845143027922e-06 -egt 9.78549609150314e-06 -union 9.78549609150314e-06 -▁Alp 9.78549609150314e-06 -▁Spitze 9.78549609150314e-06 -▁Werden 9.78549609150314e-06 -▁approaches 9.78549609150314e-06 -▁entfernen 9.78549609150314e-06 -▁implications 9.78549609150314e-06 -ausgaben 9.7817076687034864e-06 -von 9.7817076687034864e-06 -▁Ven 9.7817076687034864e-06 -▁audience 9.7817076687034864e-06 -so 9.7779192459038344e-06 -▁2.5 9.7779192459038344e-06 -▁broken 9.7779192459038344e-06 -▁crises 9.7779192459038344e-06 -▁forth 9.7779192459038344e-06 -achst 9.77413082310418e-06 -nel 9.77413082310418e-06 -roc 9.77413082310418e-06 -▁Samsung 9.77413082310418e-06 --to- 9.7703424003045286e-06 -▁Domin 9.7703424003045286e-06 -▁Orten 9.7703424003045286e-06 -▁Times 9.7703424003045286e-06 -▁classical 9.7703424003045286e-06 -▁derer 9.7703424003045286e-06 -bet 9.7665539775048766e-06 -ägt 9.7665539775048766e-06 -▁Tieren 9.7665539775048766e-06 -▁erteilt 9.7665539775048766e-06 -ld 9.7627655547052228e-06 -vertr 9.7627655547052228e-06 -▁Exper 9.7627655547052228e-06 -▁Lieferanten 9.7627655547052228e-06 -ulator 9.75897713190557e-06 -▁branch 9.75897713190557e-06 -▁extending 9.75897713190557e-06 -▁pod 9.75897713190557e-06 -▁render 9.75897713190557e-06 -▁Nah 9.755188709105917e-06 -▁Pot 9.755188709105917e-06 -▁relationships 9.755188709105917e-06 -▁beabsichtigt 9.751400286306265e-06 -▁staying 9.751400286306265e-06 -altungen 9.7476118635066113e-06 -▁1950 9.7476118635066113e-06 -▁Arbeitnehmern 9.7476118635066113e-06 -▁Bier 9.7476118635066113e-06 -▁Emissionen 9.7476118635066113e-06 -ovo 9.74382344070696e-06 -▁Gro 9.74382344070696e-06 -▁assigned 9.74382344070696e-06 -▁exhaust 9.74382344070696e-06 -▁gradu 9.74382344070696e-06 -▁meaningfully 9.74382344070696e-06 -▁täglichen 9.74382344070696e-06 --00 9.7400350179073055e-06 -▁Ausblick 9.7400350179073055e-06 -▁Mul 9.7400350179073055e-06 -ituation 9.7362465951076534e-06 -wood 9.7362465951076534e-06 -▁prompt 9.7324581723080014e-06 -▁schemes 9.7324581723080014e-06 -▁vollständigen 9.7324581723080014e-06 -66 9.7286697495083476e-06 -▁Compl 9.7286697495083476e-06 -ulating 9.7248813267086956e-06 -▁Blair 9.7210929039090419e-06 -▁baby 9.7210929039090419e-06 -▁principal 9.7210929039090419e-06 -▁weltweite 9.7210929039090419e-06 -hersteller 9.71730448110939e-06 -▁Café 9.71730448110939e-06 -▁Koordin 9.71730448110939e-06 -▁Zwar 9.71730448110939e-06 -▁booked 9.71730448110939e-06 -▁zuges 9.71730448110939e-06 -▁believed 9.7135160583097361e-06 -etzten 9.709727635510084e-06 -▁Gemeinschaften 9.709727635510084e-06 -▁Philosophie 9.709727635510084e-06 -▁Vordergrund 9.709727635510084e-06 -▁anwenden 9.709727635510084e-06 -ión 9.705939212710432e-06 -▁Pflege 9.7021507899107783e-06 -▁Rev 9.6983623671111262e-06 -▁describe 9.6983623671111262e-06 -iebs 9.69078552151182e-06 -▁Meg 9.69078552151182e-06 -▁ausger 9.69078552151182e-06 -▁bought 9.69078552151182e-06 -outh 9.6869970987121667e-06 -eits 9.6832086759125146e-06 -gangen 9.6832086759125146e-06 -ohl 9.6832086759125146e-06 -ract 9.6832086759125146e-06 -▁exhib 9.6832086759125146e-06 -eters 9.6794202531128626e-06 -▁Listings 9.6794202531128626e-06 -▁negotiation 9.6794202531128626e-06 -irk 9.6756318303132089e-06 -ype 9.6756318303132089e-06 -▁Wunder 9.6756318303132089e-06 -▁ehemalige 9.6756318303132089e-06 -▁eliminate 9.6756318303132089e-06 -▁enthalt 9.6756318303132089e-06 -▁kostenlose 9.6756318303132089e-06 -▁vorgelegten 9.6756318303132089e-06 -▁umgehen 9.6718434075135568e-06 -▁unf 9.6718434075135568e-06 -▁gefährlich 9.6680549847139031e-06 -▁Argumente 9.664266561914251e-06 -▁Serbien 9.664266561914251e-06 -▁Unterkünfte 9.664266561914251e-06 -▁genaue 9.664266561914251e-06 -▁geniessen 9.664266561914251e-06 -▁zweit 9.664266561914251e-06 -▁IWF 9.6604781391145973e-06 -▁LCD 9.6604781391145973e-06 -▁bereitet 9.6604781391145973e-06 -▁satisfactory 9.6604781391145973e-06 -kontroll 9.6566897163149453e-06 -▁Industrial 9.6566897163149453e-06 -▁Lot 9.6566897163149453e-06 -▁aktiviert 9.6566897163149453e-06 -▁günstig 9.6566897163149453e-06 -indungen 9.6529012935152915e-06 -ugs 9.6529012935152915e-06 -▁Kh 9.6529012935152915e-06 -▁constitutes 9.6529012935152915e-06 -▁ingredients 9.6529012935152915e-06 -▁repeated 9.6529012935152915e-06 -▁verringert 9.6529012935152915e-06 -▁Nigeria 9.64911287071564e-06 -▁Revision 9.64911287071564e-06 -▁earn 9.64911287071564e-06 -▁geblieben 9.64911287071564e-06 -▁linken 9.64911287071564e-06 -▁Außenminister 9.6453244479159874e-06 -▁Danach 9.6453244479159874e-06 -▁Sekunden 9.6453244479159874e-06 -45 9.6415360251163337e-06 -gress 9.6415360251163337e-06 -icient 9.6415360251163337e-06 -äck 9.6415360251163337e-06 -▁3, 9.6415360251163337e-06 -▁Konkurrenz 9.6415360251163337e-06 -▁nomin 9.6415360251163337e-06 -▁owned 9.6415360251163337e-06 -EE 9.6377476023166816e-06 -ummer 9.6377476023166816e-06 -zur 9.6377476023166816e-06 -▁Kampagne 9.6377476023166816e-06 -▁LE 9.6377476023166816e-06 -▁Weit 9.6377476023166816e-06 -▁bewerten 9.6377476023166816e-06 -▁neighbours 9.6377476023166816e-06 -▁scheduled 9.6377476023166816e-06 -ischt 9.6339591795170279e-06 -▁adv 9.6339591795170279e-06 -▁amended 9.6339591795170279e-06 -▁beseitigen 9.6339591795170279e-06 -▁entdeckst 9.6339591795170279e-06 -▁kontextuellen 9.6339591795170279e-06 -waffen 9.6301707567173759e-06 -▁Annehmlichkeiten 9.6301707567173759e-06 -▁Beschluß 9.6301707567173759e-06 -▁heads 9.6301707567173759e-06 -▁OLAF 9.6263823339177221e-06 -▁glaub 9.6263823339177221e-06 -▁hundert 9.6263823339177221e-06 -▁finanzieren 9.62259391111807e-06 -ippe 9.618805488318418e-06 -▁Order 9.618805488318418e-06 -▁bal 9.618805488318418e-06 -▁profess 9.618805488318418e-06 -▁splendid 9.618805488318418e-06 -berger 9.6150170655187643e-06 -ision 9.6150170655187643e-06 -▁Employment 9.6150170655187643e-06 -▁organise 9.6150170655187643e-06 -▁surprising 9.6150170655187643e-06 -92 9.6112286427191122e-06 -ighter 9.6112286427191122e-06 -▁poorest 9.6112286427191122e-06 -▁überwachen 9.6112286427191122e-06 -▁medicines 9.6074402199194585e-06 -iebe 9.6036517971198065e-06 -inem 9.6036517971198065e-06 -uden 9.6036517971198065e-06 -▁Bla 9.6036517971198065e-06 -▁Düsseldorf 9.6036517971198065e-06 -▁Bildern 9.5998633743201527e-06 -▁campaigns 9.5998633743201527e-06 -▁chapter 9.5998633743201527e-06 -▁collabor 9.5998633743201527e-06 -gebaut 9.5922865287208469e-06 -▁addressing 9.5922865287208469e-06 -funktion 9.5884981059211949e-06 -ician 9.5884981059211949e-06 -▁Vermögens 9.5884981059211949e-06 -▁grammatically 9.5884981059211949e-06 -▁südlichen 9.5884981059211949e-06 --Bereich 9.5847096831215429e-06 -asche 9.5847096831215429e-06 -nte 9.5847096831215429e-06 -▁Korr 9.5847096831215429e-06 -▁checked 9.5809212603218891e-06 -▁commit 9.5809212603218891e-06 -▁contextually 9.5809212603218891e-06 -▁eingebracht 9.5809212603218891e-06 -▁kontextuell 9.5809212603218891e-06 -▁manchen 9.5809212603218891e-06 -well 9.5771328375222371e-06 -▁Ausbildungs 9.5771328375222371e-06 -▁Empire 9.5771328375222371e-06 -▁concentration 9.5771328375222371e-06 -▁hinzufugen 9.5771328375222371e-06 -▁milit 9.5771328375222371e-06 -▁poker 9.5771328375222371e-06 -▁remarkable 9.5771328375222371e-06 -▁whereby 9.5771328375222371e-06 -instrumente 9.5733444147225833e-06 -kommt 9.5733444147225833e-06 -▁Ach 9.5733444147225833e-06 -▁Chairman 9.5733444147225833e-06 -▁Einer 9.5733444147225833e-06 -▁inn 9.5733444147225833e-06 -▁Cash 9.5695559919229313e-06 -▁grammatisch 9.5695559919229313e-06 -▁grünen 9.5695559919229313e-06 -▁quant 9.5695559919229313e-06 -.ch 9.5657675691232775e-06 -▁verständ 9.5657675691232775e-06 -▁Spaziergang 9.5619791463236255e-06 -▁governing 9.5619791463236255e-06 -mäßige 9.5581907235239735e-06 -▁Ford 9.5581907235239735e-06 -▁HP 9.5581907235239735e-06 -▁Herkunft 9.5581907235239735e-06 -▁drag 9.5581907235239735e-06 -ausend 9.55440230072432e-06 -flanz 9.55440230072432e-06 -ingt 9.55440230072432e-06 -▁Nizza 9.55440230072432e-06 -▁joining 9.55440230072432e-06 -erten 9.5506138779246677e-06 -▁59 9.5506138779246677e-06 -▁Bol 9.5506138779246677e-06 -▁anges 9.5506138779246677e-06 -▁eingeleitet 9.5506138779246677e-06 -▁enforcement 9.5506138779246677e-06 -▁weaken 9.5506138779246677e-06 -indung 9.5468254551250139e-06 -▁activ 9.5468254551250139e-06 -▁uneingeschränkt 9.5468254551250139e-06 -andal 9.5430370323253619e-06 -cs 9.5430370323253619e-06 -ellung 9.5430370323253619e-06 -reicht 9.5430370323253619e-06 -stil 9.5430370323253619e-06 -▁Stern 9.5430370323253619e-06 -▁verwendeten 9.5430370323253619e-06 -iffen 9.5392486095257082e-06 -kunden 9.5392486095257082e-06 -▁Gegner 9.5392486095257082e-06 -▁ausführ 9.5392486095257082e-06 -▁ultimate 9.5392486095257082e-06 -ibl 9.5354601867260561e-06 -▁Schlusselphrase 9.5354601867260561e-06 -▁Schlusselwort 9.5354601867260561e-06 -inary 9.5316717639264041e-06 -küste 9.5316717639264041e-06 -▁J. 9.5316717639264041e-06 -▁Jewish 9.5316717639264041e-06 -▁Mechanismus 9.5316717639264041e-06 -▁erwiesen 9.5316717639264041e-06 -▁technique 9.5316717639264041e-06 -ective 9.52788334112675e-06 -▁aufgezahlt 9.52788334112675e-06 -▁fell 9.52788334112675e-06 -▁schließt 9.52788334112675e-06 -fällig 9.5240949183270983e-06 -▁Ehre 9.5240949183270983e-06 -▁GT 9.5240949183270983e-06 -▁translated 9.5240949183270983e-06 -amps 9.5203064955274445e-06 -lar 9.5203064955274445e-06 -rein 9.5203064955274445e-06 -▁LED 9.5203064955274445e-06 -▁deut 9.5203064955274445e-06 -▁digitale 9.5203064955274445e-06 -▁gemeinsames 9.5203064955274445e-06 -▁urgency 9.5203064955274445e-06 -▁violation 9.5203064955274445e-06 -▁Dokumentation 9.5165180727277925e-06 -▁Intergovernmental 9.5165180727277925e-06 -▁beantwortet 9.5165180727277925e-06 -▁dual 9.5165180727277925e-06 -▁mach 9.5165180727277925e-06 -▁mixed 9.5165180727277925e-06 --oriented 9.5127296499281388e-06 -esa 9.5127296499281388e-06 -fel 9.5127296499281388e-06 -▁Universal 9.5127296499281388e-06 -▁Ut 9.5127296499281388e-06 -OV 9.5089412271284867e-06 -▁FOR 9.5089412271284867e-06 -▁sorry 9.5089412271284867e-06 -▁verbreitet 9.5089412271284867e-06 -märkte 9.505152804328833e-06 -▁Galileo 9.505152804328833e-06 -▁anim 9.505152804328833e-06 -▁spec 9.505152804328833e-06 --style 9.5013643815291809e-06 -erman 9.5013643815291809e-06 -те 9.5013643815291809e-06 -▁enjoying 9.5013643815291809e-06 -▁gewidmet 9.5013643815291809e-06 -▁innovativen 9.5013643815291809e-06 -▁prohib 9.5013643815291809e-06 -▁Slovakia 9.4975759587295289e-06 -▁Spring 9.4975759587295289e-06 -▁empty 9.4975759587295289e-06 -▁hilfreich 9.4975759587295289e-06 -esc 9.4937875359298751e-06 -▁Chem 9.4937875359298751e-06 -▁Nation 9.4937875359298751e-06 -▁Plat 9.4937875359298751e-06 -▁incentives 9.4937875359298751e-06 -gy 9.4899991131302231e-06 -▁Antr 9.4899991131302231e-06 -▁Moscow 9.4899991131302231e-06 -▁ease 9.4899991131302231e-06 -▁nächster 9.4899991131302231e-06 -▁Är 9.4899991131302231e-06 -ibt 9.48621069033057e-06 -▁Gibt 9.48621069033057e-06 -88 9.4824222675309173e-06 -block 9.4824222675309173e-06 -▁Fisheries 9.4824222675309173e-06 -▁Km 9.4824222675309173e-06 -▁Können 9.4824222675309173e-06 -▁chat 9.4824222675309173e-06 -▁variable 9.4824222675309173e-06 -▁esp 9.4786338447312636e-06 -▁max 9.4786338447312636e-06 -▁vollen 9.4786338447312636e-06 --Q 9.4748454219316115e-06 -ermaßen 9.4748454219316115e-06 -struct 9.4748454219316115e-06 -▁Anbetracht 9.4748454219316115e-06 -▁Nehmen 9.4748454219316115e-06 -▁Plug 9.4748454219316115e-06 -▁Wohnungen 9.4748454219316115e-06 -▁verstehe 9.4748454219316115e-06 -IER 9.47105699913196e-06 -▁Priv 9.47105699913196e-06 -▁boasts 9.47105699913196e-06 -▁wife 9.47105699913196e-06 -place 9.4672685763323058e-06 -▁Chat 9.4672685763323058e-06 -▁gegebenenfalls 9.4672685763323058e-06 -▁incident 9.4672685763323058e-06 -▁Publikum 9.4634801535326537e-06 -▁aufzubauen 9.4634801535326537e-06 -▁elev 9.4634801535326537e-06 -▁boats 9.459691730733e-06 -▁compulsory 9.459691730733e-06 --Prozess 9.4559033079333479e-06 -▁overlook 9.4559033079333479e-06 -▁ratification 9.4559033079333479e-06 -▁Arena 9.4521148851336942e-06 -▁Denken 9.4521148851336942e-06 -▁Lernen 9.4521148851336942e-06 -▁Saal 9.4521148851336942e-06 -▁mere 9.4521148851336942e-06 -▁cer 9.4483264623340421e-06 -▁fascinating 9.4483264623340421e-06 -▁hochwertigen 9.4407496167347364e-06 -▁reagiert 9.4407496167347364e-06 -itu 9.4369611939350843e-06 -▁Akteure 9.4369611939350843e-06 -▁Niederlande 9.4369611939350843e-06 -▁employed 9.4369611939350843e-06 -▁preferred 9.4369611939350843e-06 -glich 9.43317277113543e-06 -seh 9.43317277113543e-06 -win 9.43317277113543e-06 -▁aufbauen 9.43317277113543e-06 -▁electrical 9.43317277113543e-06 -▁girls 9.43317277113543e-06 -▁hinzugefügt 9.43317277113543e-06 -▁moments 9.43317277113543e-06 -▁ruhige 9.43317277113543e-06 --Server 9.4293843483357785e-06 -nung 9.4293843483357785e-06 -verbindung 9.4293843483357785e-06 -▁Invest 9.4293843483357785e-06 -▁chairman 9.4293843483357785e-06 -▁residential 9.4293843483357785e-06 -▁Wednesday 9.4255959255361248e-06 -▁aside 9.4255959255361248e-06 -▁halt 9.4255959255361248e-06 -▁imperial 9.4255959255361248e-06 -▁veröffentlichen 9.4255959255361248e-06 -▁-S 9.4218075027364728e-06 -▁Agr 9.4218075027364728e-06 -▁lädt 9.4218075027364728e-06 -▁steam 9.4218075027364728e-06 -hung 9.418019079936819e-06 -▁Sektors 9.418019079936819e-06 -▁marine 9.418019079936819e-06 -▁winning 9.418019079936819e-06 -icherheit 9.414230657137167e-06 -ilo 9.414230657137167e-06 -ï 9.414230657137167e-06 -▁Einkauf 9.414230657137167e-06 -▁definieren 9.414230657137167e-06 -▁keineswegs 9.414230657137167e-06 -orks 9.4104422343375149e-06 -▁Bezeichnung 9.4104422343375149e-06 -▁Hab 9.4104422343375149e-06 -▁League 9.4104422343375149e-06 -▁appointment 9.4104422343375149e-06 -▁robust 9.4104422343375149e-06 -▁schwere 9.4104422343375149e-06 -▁Ausweitung 9.4066538115378612e-06 -▁Fed 9.4066538115378612e-06 -▁completion 9.4066538115378612e-06 -▁dauern 9.4066538115378612e-06 -▁navigation 9.4066538115378612e-06 -ects 9.4028653887382091e-06 -▁confront 9.4028653887382091e-06 -▁foundations 9.4028653887382091e-06 -▁Erwachsene 9.3990769659385554e-06 -▁Pod 9.3990769659385554e-06 -▁William 9.3990769659385554e-06 -ource 9.3952885431389034e-06 -stätten 9.3952885431389034e-06 -urf 9.3952885431389034e-06 -ühl 9.3952885431389034e-06 -▁fewer 9.3952885431389034e-06 -▁reass 9.3952885431389034e-06 -OD 9.39150012033925e-06 -▁Pacific 9.39150012033925e-06 -▁Sowohl 9.39150012033925e-06 -▁deserves 9.39150012033925e-06 -▁portal 9.39150012033925e-06 -▁references 9.39150012033925e-06 -verständ 9.3877116975395976e-06 -öff 9.3877116975395976e-06 -zeichnung 9.3839232747399438e-06 -▁Cost 9.3839232747399438e-06 -▁anz 9.3839232747399438e-06 -▁charakter 9.3839232747399438e-06 -ühlen 9.3801348519402918e-06 -▁Bestell 9.37634642914064e-06 -▁array 9.37634642914064e-06 -▁landwirtschaftlichen 9.37634642914064e-06 -▁legitimacy 9.37634642914064e-06 -reh 9.372558006340986e-06 -▁Agenturen 9.372558006340986e-06 -▁Verzeichnis 9.372558006340986e-06 -▁conciliation 9.372558006340986e-06 -▁height 9.372558006340986e-06 -▁cancellation 9.368769583541334e-06 -hängen 9.36498116074168e-06 -ð 9.36498116074168e-06 -▁Jur 9.36498116074168e-06 -▁pools 9.36498116074168e-06 -▁äußer 9.36498116074168e-06 -OG 9.3611927379420282e-06 -formular 9.3611927379420282e-06 -▁Anders 9.3611927379420282e-06 -▁Souveränität 9.3611927379420282e-06 -▁chief 9.3611927379420282e-06 -zimmern 9.3574043151423744e-06 --term 9.3536158923427224e-06 -37 9.3536158923427224e-06 -ados 9.3536158923427224e-06 -alts 9.3536158923427224e-06 -antes 9.3536158923427224e-06 -▁Bestätigung 9.3536158923427224e-06 -▁Million 9.3536158923427224e-06 -umin 9.34982746954307e-06 -▁Together 9.34982746954307e-06 -▁Verz 9.34982746954307e-06 -▁Youth 9.34982746954307e-06 -▁besonderes 9.34982746954307e-06 -▁cancel 9.34982746954307e-06 -bec 9.3460390467434166e-06 -empf 9.3460390467434166e-06 -▁Belg 9.3460390467434166e-06 -▁Einfach 9.3460390467434166e-06 -▁Portal 9.3460390467434166e-06 --Fl 9.3422506239437646e-06 -▁Eff 9.3422506239437646e-06 -▁extract 9.3422506239437646e-06 -▁incorpor 9.3422506239437646e-06 -▁newsletter 9.3422506239437646e-06 -itude 9.33846220114411e-06 -▁1986 9.33846220114411e-06 -▁Eisen 9.33846220114411e-06 -▁stattgefunden 9.33846220114411e-06 -▁zul 9.33846220114411e-06 -üche 9.3346737783444588e-06 -▁Dein 9.3346737783444588e-06 -▁Fer 9.3346737783444588e-06 -▁faces 9.3346737783444588e-06 -asso 9.330885355544805e-06 -essel 9.330885355544805e-06 -▁timetable 9.330885355544805e-06 -stat 9.327096932745153e-06 -▁Select 9.327096932745153e-06 -▁Vern 9.327096932745153e-06 -eme 9.323308509945501e-06 -service 9.323308509945501e-06 -▁Bre 9.323308509945501e-06 -▁External 9.323308509945501e-06 -▁Verbrauchern 9.323308509945501e-06 -▁autumn 9.323308509945501e-06 -▁coach 9.323308509945501e-06 -unst 9.3195200871458472e-06 -▁Erf 9.3195200871458472e-06 -▁Fassung 9.3195200871458472e-06 -▁Heads 9.3195200871458472e-06 -▁authors 9.3195200871458472e-06 -▁observe 9.3195200871458472e-06 -▁thumbnail 9.3195200871458472e-06 -▁Dur 9.3157316643461952e-06 -▁herzustellen 9.3157316643461952e-06 -▁sitzen 9.3157316643461952e-06 -▁vit 9.3157316643461952e-06 -ieu 9.3119432415465414e-06 -lös 9.3119432415465414e-06 -▁Vorsitzenden 9.3119432415465414e-06 -▁ultra 9.3119432415465414e-06 -▁Print 9.30815481874689e-06 -▁bread 9.30815481874689e-06 -ission 9.3043663959472357e-06 -▁Luftfahrt 9.3043663959472357e-06 -▁erkunden 9.3043663959472357e-06 -▁theatre 9.3043663959472357e-06 -▁considerations 9.3005779731475836e-06 -▁moreover 9.3005779731475836e-06 -hang 9.29678955034793e-06 -▁Fle 9.29678955034793e-06 -▁Ministry 9.29678955034793e-06 -▁desto 9.29678955034793e-06 -neiden 9.2930011275482778e-06 -▁ständigen 9.2930011275482778e-06 -▁Anonymous 9.2892127047486258e-06 -▁erheben 9.2892127047486258e-06 -▁quest 9.2892127047486258e-06 -▁repeatedly 9.2892127047486258e-06 -▁Demokraten 9.285424281948972e-06 -▁Deutschlands 9.285424281948972e-06 -▁wil 9.285424281948972e-06 -▁är 9.285424281948972e-06 -icking 9.28163585914932e-06 -▁Naz 9.28163585914932e-06 -▁Panorama 9.28163585914932e-06 -▁Prof. 9.28163585914932e-06 -▁inadequate 9.28163585914932e-06 -ippen 9.2778474363496663e-06 -▁neighbourhood 9.2778474363496663e-06 --e 9.2740590135500142e-06 -oint 9.2740590135500142e-06 -▁Renn 9.2740590135500142e-06 -▁debating 9.2740590135500142e-06 -▁edit 9.2740590135500142e-06 -▁sheet 9.2740590135500142e-06 -▁travellers 9.2740590135500142e-06 -▁Sep 9.27027059075036e-06 -▁professionellen 9.27027059075036e-06 -▁rout 9.27027059075036e-06 --European 9.2626937451510564e-06 -icious 9.2626937451510564e-06 -indet 9.2626937451510564e-06 -statt 9.2626937451510564e-06 -▁Prozeß 9.2626937451510564e-06 -DE 9.2589053223514027e-06 -führten 9.2589053223514027e-06 -iev 9.2589053223514027e-06 -illing 9.2589053223514027e-06 -▁Ehr 9.2589053223514027e-06 -▁Komm 9.2589053223514027e-06 -▁begun 9.2589053223514027e-06 -▁berichtet 9.2589053223514027e-06 -▁zustande 9.2589053223514027e-06 -ologisch 9.25511689955175e-06 -▁Chief 9.25511689955175e-06 -▁Objekte 9.25511689955175e-06 -▁Werten 9.25511689955175e-06 -▁evol 9.25511689955175e-06 -▁fassen 9.25511689955175e-06 -▁favourable 9.25511689955175e-06 -▁generell 9.25511689955175e-06 -beln 9.2513284767520969e-06 -ähl 9.2513284767520969e-06 -▁coastal 9.2513284767520969e-06 -▁escape 9.2513284767520969e-06 -▁false 9.2513284767520969e-06 -▁journalists 9.2513284767520969e-06 -▁unnecessary 9.2513284767520969e-06 --made 9.2475400539524448e-06 -laub 9.2475400539524448e-06 -▁Phys 9.2475400539524448e-06 -▁Reparatur 9.2475400539524448e-06 -▁exposed 9.2475400539524448e-06 -iders 9.2437516311527911e-06 -reen 9.2437516311527911e-06 -▁Binnen 9.2437516311527911e-06 -▁WE 9.2437516311527911e-06 -▁bearbeiten 9.2437516311527911e-06 -▁festen 9.2437516311527911e-06 -▁secondly 9.2437516311527911e-06 -ador 9.239963208353139e-06 -icit 9.239963208353139e-06 -istent 9.239963208353139e-06 -▁Marco 9.239963208353139e-06 -▁Projekts 9.239963208353139e-06 -▁elegante 9.239963208353139e-06 -▁enge 9.239963208353139e-06 -▁qui 9.239963208353139e-06 -▁secondary 9.239963208353139e-06 -ession 9.2361747855534853e-06 -▁Eines 9.2361747855534853e-06 -▁genuinely 9.2361747855534853e-06 -▁Bosnia 9.2323863627538333e-06 -▁Dynamik 9.2323863627538333e-06 -adel 9.2285979399541812e-06 -▁Versicherungs 9.2285979399541812e-06 -ads 9.2248095171545275e-06 -▁sizes 9.2248095171545275e-06 -anced 9.2210210943548754e-06 -enth 9.2210210943548754e-06 -▁Vereinigte 9.2210210943548754e-06 -▁benef 9.2210210943548754e-06 -erie 9.2172326715552217e-06 -standard 9.2172326715552217e-06 -▁Entschlossenheit 9.2172326715552217e-06 -▁Hill 9.2172326715552217e-06 -▁logic 9.2172326715552217e-06 -▁ökologischen 9.2172326715552217e-06 -ellten 9.21344424875557e-06 -▁Serv 9.21344424875557e-06 -▁einheit 9.21344424875557e-06 -▁Erreichung 9.2096558259559159e-06 -▁codecision 9.2096558259559159e-06 -▁eingel 9.2096558259559159e-06 -▁folder 9.2096558259559159e-06 -▁chinesische 9.2058674031562639e-06 -kret 9.2020789803566118e-06 -reas 9.2020789803566118e-06 -stes 9.2020789803566118e-06 -▁Relax 9.2020789803566118e-06 -▁Schatten 9.2020789803566118e-06 -▁stip 9.2020789803566118e-06 -emp 9.1982905575569581e-06 -▁Ents 9.1982905575569581e-06 -▁Universitäten 9.1982905575569581e-06 -▁palästinensischen 9.1982905575569581e-06 -wander 9.194502134757306e-06 -▁A4 9.194502134757306e-06 -▁Auflösung 9.194502134757306e-06 -▁Ehe 9.194502134757306e-06 -▁analog 9.194502134757306e-06 -▁eingeschränkt 9.194502134757306e-06 -▁flood 9.194502134757306e-06 -▁nutzt 9.194502134757306e-06 -lan 9.1907137119576523e-06 -▁Aspekten 9.1907137119576523e-06 -▁bearing 9.1907137119576523e-06 -▁threshold 9.1907137119576523e-06 -ailing 9.186925289158e-06 -ocation 9.186925289158e-06 -regionen 9.186925289158e-06 -▁Anmerkungen 9.186925289158e-06 -▁Milan 9.186925289158e-06 -▁genauer 9.186925289158e-06 -ienten 9.1831368663583465e-06 -periode 9.1831368663583465e-06 -amed 9.1793484435586945e-06 -▁Erfolge 9.1793484435586945e-06 -▁Nordic 9.1793484435586945e-06 -▁disappear 9.1793484435586945e-06 -▁irr 9.1793484435586945e-06 -ML 9.17556002075904e-06 -▁Entwicklungshilfe 9.17556002075904e-06 -▁underlying 9.17556002075904e-06 -weilen 9.1717715979593887e-06 -▁compete 9.1717715979593887e-06 -▁cookies 9.1717715979593887e-06 -▁lively 9.1717715979593887e-06 -▁murder 9.1717715979593887e-06 -fisch 9.1679831751597366e-06 -▁Beste 9.1679831751597366e-06 -▁Getränke 9.1679831751597366e-06 -▁Nichtraucherzimmer 9.1679831751597366e-06 -▁Patten 9.1679831751597366e-06 -▁Series 9.1679831751597366e-06 -▁Volkspartei 9.1679831751597366e-06 -▁hel 9.1679831751597366e-06 -▁langer 9.1679831751597366e-06 -träge 9.1641947523600829e-06 -▁Ratifizierung 9.1641947523600829e-06 -▁Unsicherheit 9.1641947523600829e-06 -▁cod 9.1641947523600829e-06 -becken 9.1604063295604309e-06 -▁62 9.1604063295604309e-06 -▁Bade 9.1604063295604309e-06 -▁Insgesamt 9.1604063295604309e-06 -▁asset 9.1604063295604309e-06 -▁characteristic 9.1604063295604309e-06 -▁foremost 9.1604063295604309e-06 -▁geliefert 9.1604063295604309e-06 -▁gestatten 9.1604063295604309e-06 -▁speeches 9.1604063295604309e-06 -raums 9.1566179067607771e-06 -▁Elektr 9.1566179067607771e-06 -▁Storage 9.1566179067607771e-06 -ichtet 9.1528294839611251e-06 -ifizieren 9.1528294839611251e-06 -▁CAN 9.1528294839611251e-06 -▁economically 9.1528294839611251e-06 -▁listening 9.1528294839611251e-06 -▁ring 9.1528294839611251e-06 -▁sample 9.1528294839611251e-06 --Version 9.1490410611614713e-06 -ça 9.1490410611614713e-06 -▁dramatic 9.1490410611614713e-06 -▁guaranteeing 9.1490410611614713e-06 -prom 9.14525263836182e-06 -▁gelesen 9.14525263836182e-06 -▁representing 9.14525263836182e-06 -kräften 9.1414642155621672e-06 -▁Rats 9.1414642155621672e-06 -cke 9.1376757927625135e-06 -ellte 9.1376757927625135e-06 -mäßigen 9.1376757927625135e-06 -▁29. 9.1376757927625135e-06 -▁featuring 9.1376757927625135e-06 -▁Kompetenzen 9.1338873699628615e-06 -▁teuer 9.1338873699628615e-06 -▁900 9.1300989471632077e-06 -▁Ereignis 9.1300989471632077e-06 -▁Nachmittag 9.1300989471632077e-06 -▁gear 9.1300989471632077e-06 -zia 9.1263105243635557e-06 -τ 9.1263105243635557e-06 -▁managers 9.1263105243635557e-06 -▁operated 9.1263105243635557e-06 -stechnik 9.1225221015639019e-06 -zugs 9.1225221015639019e-06 -▁Fremdenverkehr 9.1225221015639019e-06 -▁RE 9.1225221015639019e-06 -▁bigger 9.1225221015639019e-06 -▁schauen 9.1225221015639019e-06 -▁vent 9.1225221015639019e-06 -arische 9.11873367876425e-06 -ients 9.11873367876425e-06 -▁Leiter 9.11873367876425e-06 -▁Runde 9.11873367876425e-06 -cal 9.1149452559645979e-06 -▁Ill 9.1149452559645979e-06 -▁Verf 9.1149452559645979e-06 -▁sucht 9.1149452559645979e-06 -add 9.1111568331649441e-06 -▁Smith 9.1111568331649441e-06 -▁erstes 9.1111568331649441e-06 -▁respects 9.1111568331649441e-06 -ι 9.1073684103652921e-06 -▁expanded 9.1073684103652921e-06 -▁subsequently 9.1073684103652921e-06 -▁Several 9.1035799875656383e-06 -▁informal 9.1035799875656383e-06 -▁neutral 9.1035799875656383e-06 -▁zweif 9.1035799875656383e-06 -aton 9.0997915647659863e-06 -▁Massage 9.0997915647659863e-06 -▁frequent 9.0997915647659863e-06 -▁instantly 9.0997915647659863e-06 -▁textile 9.0997915647659863e-06 -▁Soldaten 9.0960031419663325e-06 -▁destroy 9.0960031419663325e-06 -▁grenzüberschreitenden 9.0960031419663325e-06 -▁ho 9.0960031419663325e-06 -▁schwedischen 9.0960031419663325e-06 -brauch 9.09221471916668e-06 -▁sin 9.09221471916668e-06 -▁Designer 9.0884262963670268e-06 -▁Section 9.0884262963670268e-06 -▁gesellschaftlichen 9.0884262963670268e-06 -▁govern 9.0846378735673747e-06 -fäll 9.0808494507677227e-06 -ye 9.0808494507677227e-06 -▁McC 9.0808494507677227e-06 -▁Program 9.0808494507677227e-06 -▁anerkennen 9.0808494507677227e-06 -▁diving 9.0808494507677227e-06 -▁suggests 9.0808494507677227e-06 -icing 9.0770610279680689e-06 -▁Schwellen 9.0770610279680689e-06 -▁slopes 9.0770610279680689e-06 -▁arranged 9.0732726051684169e-06 -▁sovereign 9.0732726051684169e-06 --Dateien 9.0694841823687632e-06 -rogen 9.0694841823687632e-06 -▁D. 9.0694841823687632e-06 -▁History 9.0694841823687632e-06 -▁Schulz 9.0694841823687632e-06 -▁erwerben 9.0694841823687632e-06 -▁intentions 9.0694841823687632e-06 -▁Alps 9.0656957595691111e-06 -▁Job 9.0656957595691111e-06 -▁Mem 9.0656957595691111e-06 -▁Solana 9.0656957595691111e-06 -▁auswählen 9.0656957595691111e-06 -▁frag 9.0656957595691111e-06 -▁gemütliche 9.0656957595691111e-06 -fried 9.0619073367694574e-06 -krieg 9.0619073367694574e-06 -quelle 9.0619073367694574e-06 -▁examination 9.0619073367694574e-06 -aj 9.0581189139698053e-06 -▁Kommentare 9.0581189139698053e-06 -▁beibehalten 9.0581189139698053e-06 -▁openness 9.0581189139698053e-06 -vy 9.0543304911701533e-06 -▁Ashton 9.0543304911701533e-06 -▁enormen 9.0543304911701533e-06 -ouren 9.0505420683705e-06 -utter 9.0505420683705e-06 -▁actors 9.0505420683705e-06 --Sh 9.0467536455708475e-06 -geld 9.0467536455708475e-06 -reserv 9.0467536455708475e-06 -▁bewahren 9.0467536455708475e-06 -▁expenses 9.0429652227711938e-06 -▁terrible 9.0429652227711938e-06 -▁M. 9.0391767999715417e-06 -▁OECD 9.0391767999715417e-06 -▁Slowakei 9.0391767999715417e-06 -▁importantly 9.0391767999715417e-06 -PA 9.035388377171888e-06 -arians 9.035388377171888e-06 -onda 9.035388377171888e-06 -▁Ay 9.035388377171888e-06 -anging 9.0315999543722359e-06 -▁Betriebssystem 9.0315999543722359e-06 -▁Rhein 9.0315999543722359e-06 -▁Stamm 9.0315999543722359e-06 -▁bay 9.0315999543722359e-06 -▁fat 9.0315999543722359e-06 -▁ownership 9.0315999543722359e-06 -▁resist 9.0315999543722359e-06 -ionale 9.0278115315725822e-06 -▁rubber 9.0278115315725822e-06 -ère 9.02402310877293e-06 -▁Mikr 9.02402310877293e-06 -▁geh 9.02402310877293e-06 -▁widmen 9.02402310877293e-06 -CP 9.0202346859732781e-06 -zusetzen 9.0202346859732781e-06 -ña 9.0202346859732781e-06 -▁Haiti 9.0202346859732781e-06 -▁hell 9.0202346859732781e-06 -branche 9.0164462631736244e-06 -füll 9.0164462631736244e-06 -▁Inflation 9.0164462631736244e-06 -▁Zweiten 9.0164462631736244e-06 -▁complaints 9.0164462631736244e-06 -▁willingness 9.0164462631736244e-06 -ierer 9.0126578403739723e-06 -produkten 9.0126578403739723e-06 -▁Seele 9.0126578403739723e-06 -▁ausländischen 9.0126578403739723e-06 -▁breach 9.0126578403739723e-06 -▁einladen 9.0126578403739723e-06 -▁gegangen 9.0126578403739723e-06 -▁hohes 9.0126578403739723e-06 -▁kostenlosen 9.0126578403739723e-06 -prozeß 9.0088694175743186e-06 -▁Jud 9.0088694175743186e-06 -▁Wolf 9.0088694175743186e-06 -▁devoted 9.0088694175743186e-06 -▁schriftliche 9.0088694175743186e-06 -▁Migration 9.0050809947746665e-06 -▁Modul 9.0050809947746665e-06 -zweif 9.0012925719750128e-06 -▁patch 9.0012925719750128e-06 -▁Corporation 8.99750414917536e-06 -▁Köln 8.99750414917536e-06 -▁Zust 8.99750414917536e-06 -ieller 8.9937157263757087e-06 -ota 8.9937157263757087e-06 -▁Administration 8.9937157263757087e-06 -▁arbeit 8.9937157263757087e-06 -▁clicking 8.9937157263757087e-06 -▁dil 8.9937157263757087e-06 -▁verurteilen 8.9937157263757087e-06 -etzte 8.989927303576055e-06 -▁Durchsetzung 8.989927303576055e-06 -▁Garantien 8.989927303576055e-06 -▁Morocco 8.989927303576055e-06 -▁Sta 8.989927303576055e-06 -▁dep 8.989927303576055e-06 -▁insufficient 8.989927303576055e-06 -▁reserv 8.989927303576055e-06 -▁zweiter 8.989927303576055e-06 -▁Lor 8.9861388807764029e-06 -▁certified 8.9861388807764029e-06 -▁Archiv 8.98235045797675e-06 -▁dahin 8.98235045797675e-06 -▁defin 8.98235045797675e-06 -▁excursions 8.98235045797675e-06 -▁famili 8.98235045797675e-06 -▁vocational 8.98235045797675e-06 -märkten 8.9747736123774434e-06 -äden 8.9747736123774434e-06 -▁Bonn 8.9747736123774434e-06 -▁Montenegro 8.9747736123774434e-06 -▁Rechner 8.9747736123774434e-06 -▁jointly 8.9747736123774434e-06 -field 8.9709851895777914e-06 -uis 8.9709851895777914e-06 -▁Unterricht 8.9709851895777914e-06 -▁accessories 8.9709851895777914e-06 -▁bedroht 8.9709851895777914e-06 -▁elektronische 8.9709851895777914e-06 -heb 8.96719676677814e-06 -▁Klarheit 8.96719676677814e-06 -▁behavior 8.96719676677814e-06 --Sp 8.9634083439784856e-06 -▁abandon 8.9634083439784856e-06 -▁berühmte 8.9634083439784856e-06 -füllt 8.9596199211788335e-06 -▁187 8.9596199211788335e-06 -▁Neg 8.9596199211788335e-06 -namen 8.95583149837918e-06 -stream 8.95583149837918e-06 -▁Ret 8.95583149837918e-06 -▁aktualisiert 8.95583149837918e-06 -▁destinations 8.95583149837918e-06 -▁erlassen 8.95583149837918e-06 -▁purchased 8.95583149837918e-06 -▁swe 8.95583149837918e-06 -gro 8.9520430755795278e-06 -idet 8.9520430755795278e-06 -▁Rab 8.9520430755795278e-06 -▁Stimm 8.9520430755795278e-06 -▁apparent 8.9520430755795278e-06 -▁bereiten 8.9520430755795278e-06 -▁reb 8.9520430755795278e-06 -.com. 8.948254652779874e-06 -voller 8.948254652779874e-06 -wass 8.948254652779874e-06 -▁sunny 8.948254652779874e-06 -▁tragic 8.948254652779874e-06 -▁trouble 8.948254652779874e-06 -ikationen 8.944466229980222e-06 -itte 8.944466229980222e-06 -▁kleinere 8.944466229980222e-06 -geschwindigkeit 8.9406778071805682e-06 -quote 8.9406778071805682e-06 -▁1987 8.9406778071805682e-06 -üst 8.9368893843809162e-06 -▁Mol 8.9368893843809162e-06 -▁convenience 8.9368893843809162e-06 -▁jurisdiction 8.9368893843809162e-06 -▁Mang 8.9331009615812641e-06 -▁acquis 8.9331009615812641e-06 -▁calendar 8.9331009615812641e-06 -▁kurzer 8.9331009615812641e-06 -▁reward 8.9331009615812641e-06 -▁transl 8.9331009615812641e-06 -▁upload 8.9331009615812641e-06 -ographie 8.92931253878161e-06 -▁Guide 8.92931253878161e-06 -▁dest 8.92931253878161e-06 -▁existieren 8.92931253878161e-06 -▁twin 8.92931253878161e-06 -lein 8.9255241159819584e-06 -▁1985 8.9255241159819584e-06 -▁26. 8.9255241159819584e-06 -▁Erzeuger 8.9255241159819584e-06 -▁alternatives 8.9255241159819584e-06 -▁letters 8.9255241159819584e-06 -▁simultaneously 8.9217356931823046e-06 -wan 8.9179472703826526e-06 -▁ideally 8.9179472703826526e-06 -▁Chef 8.9141588475829988e-06 -▁Down 8.9141588475829988e-06 -▁VER 8.9141588475829988e-06 -▁Ev 8.9103704247833468e-06 -▁Interview 8.9103704247833468e-06 -▁Polize 8.9103704247833468e-06 -▁fle 8.9103704247833468e-06 -▁vacc 8.9103704247833468e-06 -gelegt 8.9065820019836947e-06 -▁Filme 8.9065820019836947e-06 -▁Gemeinden 8.9065820019836947e-06 -▁Weine 8.9065820019836947e-06 -▁beliebt 8.9065820019836947e-06 -bestand 8.902793579184041e-06 -kern 8.902793579184041e-06 -okrat 8.902793579184041e-06 -▁Empfang 8.902793579184041e-06 -▁solch 8.902793579184041e-06 -eating 8.899005156384389e-06 -oul 8.899005156384389e-06 -▁integrierte 8.899005156384389e-06 -igert 8.8952167335847352e-06 -irts 8.8952167335847352e-06 -▁EIB 8.8952167335847352e-06 -▁Krieges 8.8952167335847352e-06 -▁erteilen 8.8952167335847352e-06 -anch 8.8914283107850832e-06 -osp 8.8914283107850832e-06 -verhalten 8.8914283107850832e-06 -▁bezeichnen 8.8914283107850832e-06 -▁horr 8.8914283107850832e-06 -▁retr 8.8914283107850832e-06 -▁unterschiedlicher 8.8914283107850832e-06 -sburg 8.88763988798543e-06 -weil 8.88763988798543e-06 -zulegen 8.88763988798543e-06 -ändern 8.88763988798543e-06 -▁Amer 8.88763988798543e-06 -▁messen 8.88763988798543e-06 -▁Bergen 8.8838514651857774e-06 -▁fault 8.8838514651857774e-06 -decken 8.8800630423861237e-06 -▁accordingly 8.8800630423861237e-06 -▁clarity 8.8800630423861237e-06 -bour 8.8762746195864716e-06 -▁CR 8.8762746195864716e-06 -▁arts 8.8762746195864716e-06 -▁erarbeitet 8.8762746195864716e-06 -▁posts 8.8762746195864716e-06 -▁therm 8.8762746195864716e-06 -▁umgehend 8.8762746195864716e-06 -othe 8.87248619678682e-06 -▁ED 8.87248619678682e-06 -▁Krebs 8.87248619678682e-06 -▁Ursache 8.87248619678682e-06 -го 8.8686977739871658e-06 -▁armen 8.8686977739871658e-06 -isters 8.8649093511875138e-06 -pose 8.8649093511875138e-06 -ries 8.8649093511875138e-06 -vest 8.8649093511875138e-06 -▁Domain 8.8649093511875138e-06 -▁Missbrauch 8.8649093511875138e-06 -▁Reit 8.8649093511875138e-06 -▁getrennt 8.8649093511875138e-06 -▁vermittelt 8.86112092838786e-06 -▁Francisco 8.857332505588208e-06 -▁desktop 8.857332505588208e-06 -uert 8.8535440827885543e-06 -▁Grünbuch 8.8535440827885543e-06 -▁Sec 8.8535440827885543e-06 -▁measurement 8.8535440827885543e-06 -▁motorway 8.8535440827885543e-06 -▁sparen 8.8535440827885543e-06 -unen 8.8497556599889022e-06 -▁Entwickler 8.8497556599889022e-06 -▁Vizepräsident 8.8497556599889022e-06 -▁vary 8.8497556599889022e-06 -36 8.84596723718925e-06 -lying 8.84596723718925e-06 -▁Ausrüstung 8.84596723718925e-06 -▁Mittwoch 8.84596723718925e-06 -▁flash 8.84596723718925e-06 -▁orch 8.84596723718925e-06 -▁produkt 8.84596723718925e-06 -▁spanische 8.84596723718925e-06 -▁transfers 8.84596723718925e-06 -▁Australien 8.8421788143895964e-06 -▁Near 8.8421788143895964e-06 -▁autor 8.8421788143895964e-06 -▁mine 8.8421788143895964e-06 -▁vertritt 8.8421788143895964e-06 -▁CDs 8.8383903915899444e-06 -▁Sternen 8.8383903915899444e-06 -▁Zurich 8.8383903915899444e-06 -▁freundliche 8.8383903915899444e-06 -▁inspection 8.8383903915899444e-06 -orth 8.83460196879029e-06 -▁Beamten 8.83460196879029e-06 -▁Ursprungs 8.83460196879029e-06 -▁discl 8.83460196879029e-06 -▁falsche 8.83460196879029e-06 -▁gym 8.83460196879029e-06 -ürz 8.8308135459906386e-06 -▁IBM 8.8308135459906386e-06 -▁Reinigung 8.8308135459906386e-06 -▁pul 8.8308135459906386e-06 -▁sinn 8.8308135459906386e-06 -igkeits 8.8270251231909849e-06 -▁A4-0 8.8270251231909849e-06 -▁Ec 8.8270251231909849e-06 -▁Menschheit 8.8270251231909849e-06 -▁kombiniert 8.8270251231909849e-06 -▁transaction 8.8270251231909849e-06 -oning 8.8232367003913328e-06 -▁Nobel 8.8232367003913328e-06 -▁Wichtigkeit 8.8232367003913328e-06 -▁Wireless 8.8232367003913328e-06 -▁postal 8.8232367003913328e-06 -▁spectrum 8.8232367003913328e-06 -pes 8.8194482775916791e-06 -schicht 8.8194482775916791e-06 -▁Logistik 8.8194482775916791e-06 -▁fifth 8.8194482775916791e-06 -▁rece 8.8194482775916791e-06 -.200 8.815659854792027e-06 -OK 8.815659854792027e-06 -kräft 8.815659854792027e-06 -▁CL 8.815659854792027e-06 -▁FI 8.815659854792027e-06 -▁Fortsetzung 8.815659854792027e-06 -▁Line 8.815659854792027e-06 -▁SDL 8.815659854792027e-06 -▁plain 8.815659854792027e-06 -▁refused 8.815659854792027e-06 -opping 8.811871431992375e-06 -▁C5-0 8.811871431992375e-06 -▁buses 8.811871431992375e-06 -▁specially 8.811871431992375e-06 -▁Hände 8.8080830091927213e-06 -▁Pilot 8.8080830091927213e-06 -▁Schicksal 8.8080830091927213e-06 -▁Village 8.8080830091927213e-06 -▁continuously 8.8080830091927213e-06 -▁Vertrauens 8.80429458639307e-06 -▁erkennt 8.80429458639307e-06 -▁strike 8.80429458639307e-06 -▁typisch 8.80429458639307e-06 -▁composed 8.8005061635934155e-06 -PG 8.7967177407937634e-06 -SI 8.7967177407937634e-06 -zehn 8.7967177407937634e-06 -▁Referenz 8.7967177407937634e-06 -▁Weltwirtschaft 8.7967177407937634e-06 -▁anzupassen 8.7967177407937634e-06 -▁convey 8.7967177407937634e-06 -▁evolution 8.7967177407937634e-06 -▁fur 8.7967177407937634e-06 -gebäude 8.79292931799411e-06 -lement 8.79292931799411e-06 -▁Behörde 8.79292931799411e-06 -▁Meilen 8.79292931799411e-06 -▁Point 8.79292931799411e-06 -▁rebu 8.79292931799411e-06 -▁tower 8.79292931799411e-06 -▁wage 8.7891408951944577e-06 -▁beweisen 8.7853524723948056e-06 -▁cu 8.7853524723948056e-06 -▁forg 8.7853524723948056e-06 -▁occupation 8.7853524723948056e-06 -▁tip 8.7853524723948056e-06 -▁twenty 8.7853524723948056e-06 -ckte 8.7815640495951519e-06 -immte 8.7815640495951519e-06 -▁Häfen 8.7815640495951519e-06 -▁Mix 8.7815640495951519e-06 -▁Palestinians 8.7815640495951519e-06 -▁wirklichen 8.7815640495951519e-06 -▁überrascht 8.7815640495951519e-06 -iod 8.7777756267955e-06 -▁Arts 8.7777756267955e-06 -▁Aussagen 8.7777756267955e-06 -▁Lebensmittelsicherheit 8.7777756267955e-06 -▁Visual 8.7777756267955e-06 -▁Wasch 8.7777756267955e-06 -▁certification 8.7777756267955e-06 -▁einher 8.7777756267955e-06 -▁facilit 8.7777756267955e-06 -lager 8.7739872039958461e-06 -▁Arr 8.7739872039958461e-06 -▁Southern 8.7739872039958461e-06 -▁collapse 8.7739872039958461e-06 -▁effizienter 8.7739872039958461e-06 -ausende 8.770198781196194e-06 -iang 8.770198781196194e-06 -schlagen 8.770198781196194e-06 -spielen 8.770198781196194e-06 -iones 8.76641035839654e-06 -ändig 8.76641035839654e-06 -ützen 8.76641035839654e-06 -▁behauptet 8.76641035839654e-06 -▁grass 8.76641035839654e-06 -▁tragedy 8.76641035839654e-06 -illon 8.7626219355968883e-06 -isst 8.7626219355968883e-06 -zeichnen 8.7626219355968883e-06 -▁Bestellungen 8.7626219355968883e-06 -▁Ferienwohnungen 8.7626219355968883e-06 -▁Growth 8.7626219355968883e-06 -▁Lis 8.7626219355968883e-06 -▁coup 8.7626219355968883e-06 -▁subsid 8.7626219355968883e-06 -road 8.7588335127972362e-06 -▁Ausrichtung 8.7588335127972362e-06 -▁innere 8.7588335127972362e-06 -normen 8.7550450899975825e-06 -ufe 8.7550450899975825e-06 -▁Cab 8.7550450899975825e-06 -▁Wen 8.7550450899975825e-06 -▁Zivil 8.7550450899975825e-06 -▁carte 8.7550450899975825e-06 -▁eingetragen 8.7550450899975825e-06 -▁forgotten 8.7550450899975825e-06 -antrag 8.75125666719793e-06 -mals 8.75125666719793e-06 -▁Zeitplan 8.75125666719793e-06 -▁befürworten 8.75125666719793e-06 -▁leb 8.75125666719793e-06 -▁logo 8.75125666719793e-06 -▁verantwortungs 8.75125666719793e-06 -ichteten 8.7474682443982767e-06 -ва 8.7474682443982767e-06 -▁Govern 8.7474682443982767e-06 -▁dupl 8.7474682443982767e-06 -reibt 8.7436798215986246e-06 -▁DJ 8.7436798215986246e-06 -▁revised 8.7398913987989709e-06 -▁¶ 8.7398913987989709e-06 --th 8.7361029759993189e-06 -enne 8.7361029759993189e-06 -schätzung 8.7361029759993189e-06 -▁Befehl 8.7361029759993189e-06 -▁brauch 8.7361029759993189e-06 -▁fried 8.7361029759993189e-06 -mail 8.7323145531996651e-06 -▁Fans 8.7323145531996651e-06 -▁driven 8.7323145531996651e-06 -▁unity 8.7323145531996651e-06 -▁Assist 8.7285261304000131e-06 -▁Nachdruck 8.7285261304000131e-06 -▁finds 8.7285261304000131e-06 -▁modify 8.7285261304000131e-06 -▁umfangreichen 8.7285261304000131e-06 -▁verursachen 8.7285261304000131e-06 -urd 8.724737707600361e-06 -▁Bart 8.724737707600361e-06 -▁Finnish 8.724737707600361e-06 -▁advis 8.724737707600361e-06 -▁riding 8.724737707600361e-06 -▁vergeben 8.724737707600361e-06 -▁versehen 8.724737707600361e-06 -biet 8.7209492848007073e-06 -etes 8.7209492848007073e-06 -▁oppose 8.7209492848007073e-06 -angebote 8.7171608620010553e-06 -gebot 8.7171608620010553e-06 -▁Wahrung 8.7171608620010553e-06 -▁bon 8.7171608620010553e-06 -rak 8.7133724392014015e-06 -▁frequency 8.7133724392014015e-06 -▁innov 8.7133724392014015e-06 -▁Felder 8.70958401640175e-06 -▁geringen 8.70958401640175e-06 -▁interne 8.70958401640175e-06 -▁parks 8.70958401640175e-06 -izieren 8.7057955936020957e-06 -▁Lateinamerika 8.7057955936020957e-06 -▁afraid 8.7057955936020957e-06 -▁assert 8.7057955936020957e-06 -freien 8.7020071708024437e-06 -gehenden 8.7020071708024437e-06 -stätigkeit 8.7020071708024437e-06 -▁28. 8.7020071708024437e-06 -▁nobody 8.7020071708024437e-06 -▁photograph 8.7020071708024437e-06 -▁publish 8.7020071708024437e-06 -▁zw 8.7020071708024437e-06 -felder 8.6982187480027916e-06 -▁Country 8.6982187480027916e-06 -▁Verheugen 8.6982187480027916e-06 -▁deadline 8.6982187480027916e-06 -▁mont 8.6982187480027916e-06 -▁‚ 8.6982187480027916e-06 -zü 8.6944303252031379e-06 -▁Han 8.6944303252031379e-06 -▁Kürze 8.6944303252031379e-06 -▁Otherwise 8.6944303252031379e-06 -▁Wünschen 8.6944303252031379e-06 -▁falling 8.6944303252031379e-06 -▁manufactured 8.6944303252031379e-06 -▁unterbrochen 8.6944303252031379e-06 -bi 8.6868534796038321e-06 -▁doubts 8.6868534796038321e-06 -▁erm 8.6868534796038321e-06 -▁promoted 8.6868534796038321e-06 -▁öffentlicher 8.6868534796038321e-06 -programmen 8.68306505680418e-06 -umer 8.68306505680418e-06 -ees 8.6792766340045263e-06 -entrum 8.6792766340045263e-06 -ε 8.6792766340045263e-06 -▁Put 8.6792766340045263e-06 -▁auswirken 8.6792766340045263e-06 -▁capt 8.6792766340045263e-06 -▁sculpt 8.6792766340045263e-06 -pressed 8.6754882112048743e-06 -uns 8.6754882112048743e-06 -▁Diesen 8.6754882112048743e-06 -▁Vorgehensweise 8.6754882112048743e-06 -▁Zahn 8.6754882112048743e-06 -▁eurozone 8.6754882112048743e-06 -▁knew 8.6754882112048743e-06 -▁lenken 8.6754882112048743e-06 -▁Übernahme 8.6754882112048743e-06 -▁Weiterhin 8.67169978840522e-06 -▁mater 8.67169978840522e-06 -▁whatsoever 8.67169978840522e-06 -▁Abbau 8.6679113656055685e-06 -▁günstigen 8.6679113656055685e-06 -▁Einzelheiten 8.6641229428059165e-06 -▁basically 8.6641229428059165e-06 -▁relaxed 8.6641229428059165e-06 -▁vorliegt 8.6641229428059165e-06 -stehende 8.6603345200062627e-06 -ville 8.6603345200062627e-06 -▁Double 8.6603345200062627e-06 -▁` 8.6603345200062627e-06 -▁einges 8.6603345200062627e-06 -▁einzu 8.6603345200062627e-06 -▁geehrte 8.6603345200062627e-06 -▁zero 8.6603345200062627e-06 -nehm 8.65654609720661e-06 -▁respektieren 8.65654609720661e-06 -▁Erl 8.6527576744069569e-06 -▁comb 8.6527576744069569e-06 -▁paths 8.6527576744069569e-06 -oles 8.6489692516073049e-06 --Spiel 8.6451808288076512e-06 -istence 8.6451808288076512e-06 -kurse 8.6451808288076512e-06 -otic 8.6451808288076512e-06 -▁Pros 8.6451808288076512e-06 -▁Tun 8.6451808288076512e-06 -▁Volkswirtschaften 8.6451808288076512e-06 -▁dimensions 8.6451808288076512e-06 -krit 8.6413924060079991e-06 -schiff 8.6413924060079991e-06 -▁Language 8.6413924060079991e-06 -▁Library 8.6413924060079991e-06 -▁Vertr 8.6413924060079991e-06 -▁ablehnen 8.6413924060079991e-06 -▁stopped 8.6413924060079991e-06 -hebung 8.6376039832083471e-06 -kunde 8.6376039832083471e-06 -rennen 8.6376039832083471e-06 -rig 8.6376039832083471e-06 -▁Zimmerbeschreibung 8.6376039832083471e-06 -▁healthcare 8.6376039832083471e-06 -▁wave 8.6376039832083471e-06 --class 8.6338155604086933e-06 -fertigung 8.6338155604086933e-06 -ührt 8.6338155604086933e-06 -▁Carl 8.6338155604086933e-06 -▁Expl 8.6338155604086933e-06 -▁affecting 8.6338155604086933e-06 -▁theoret 8.6338155604086933e-06 -cial 8.6300271376090413e-06 -jet 8.6300271376090413e-06 -word 8.6300271376090413e-06 -ц 8.6300271376090413e-06 -▁California 8.6300271376090413e-06 -▁Zurück 8.6300271376090413e-06 -▁belongs 8.6300271376090413e-06 -▁körper 8.6300271376090413e-06 -card 8.6262387148093876e-06 -▁Alternativen 8.6262387148093876e-06 -▁Baden 8.6262387148093876e-06 -▁Ten 8.6262387148093876e-06 -▁Zeich 8.6262387148093876e-06 -▁verantwort 8.6262387148093876e-06 -hnt 8.6224502920097355e-06 -reift 8.6224502920097355e-06 -ét 8.6224502920097355e-06 -▁NEW 8.6224502920097355e-06 -▁beef 8.6224502920097355e-06 -plätzen 8.6186618692100818e-06 -▁Leid 8.6186618692100818e-06 -▁dring 8.6186618692100818e-06 -▁pattern 8.6186618692100818e-06 -▁pharmaceutical 8.6186618692100818e-06 -▁quantity 8.6186618692100818e-06 -▁removal 8.6186618692100818e-06 -.C 8.61487344641043e-06 -imo 8.61487344641043e-06 -▁Clubs 8.61487344641043e-06 -▁Cologne 8.61487344641043e-06 -▁Festplatte 8.61487344641043e-06 -▁expressly 8.61487344641043e-06 -▁gelang 8.61487344641043e-06 -▁losing 8.61487344641043e-06 -▁monuments 8.61487344641043e-06 -▁residents 8.61487344641043e-06 -iki 8.6110850236107777e-06 -ris 8.6110850236107777e-06 -▁Russlands 8.6110850236107777e-06 -▁1945 8.6072966008111239e-06 -▁E-Mail-Adresse 8.6072966008111239e-06 -▁aqu 8.6072966008111239e-06 -▁planen 8.6072966008111239e-06 -▁transitional 8.6072966008111239e-06 --Ges 8.6035081780114719e-06 -▁Bekannt 8.6035081780114719e-06 -▁Offenheit 8.6035081780114719e-06 -▁Plätze 8.6035081780114719e-06 -▁bewegt 8.6035081780114719e-06 -▁erleichtert 8.6035081780114719e-06 -▁examined 8.6035081780114719e-06 -▁focuses 8.6035081780114719e-06 -▁gemütlichen 8.6035081780114719e-06 -▁shr 8.6035081780114719e-06 -▁Olympic 8.5997197552118182e-06 -▁arrangement 8.5997197552118182e-06 -▁lib 8.5997197552118182e-06 -▁technisch 8.5997197552118182e-06 -imate 8.5959313324121661e-06 -▁progressive 8.5959313324121661e-06 -.The 8.5921429096125124e-06 -▁Conserv 8.5921429096125124e-06 -▁Statistiken 8.5921429096125124e-06 -IO 8.58835448681286e-06 -rachte 8.58835448681286e-06 -▁Anton 8.58835448681286e-06 -▁Cond 8.58835448681286e-06 -▁Know-how 8.58835448681286e-06 -▁Richter 8.58835448681286e-06 -▁affordable 8.58835448681286e-06 -▁loans 8.58835448681286e-06 -▁plural 8.58835448681286e-06 -▁Peace 8.5845660640132066e-06 -▁begegnen 8.5845660640132066e-06 -col 8.5807776412135545e-06 -prüfung 8.5807776412135545e-06 -▁Circ 8.5807776412135545e-06 -▁Keine 8.5807776412135545e-06 -▁Nichts 8.5807776412135545e-06 -▁Saudi 8.5807776412135545e-06 -▁Taten 8.5807776412135545e-06 -▁ausgestattete 8.5807776412135545e-06 -▁installations 8.5807776412135545e-06 -▁jun 8.5807776412135545e-06 -ziger 8.5769892184139025e-06 -▁Heim 8.5769892184139025e-06 -▁SH 8.5769892184139025e-06 -▁defeat 8.5769892184139025e-06 -▁derselben 8.5769892184139025e-06 -▁engen 8.5769892184139025e-06 -▁günstige 8.5769892184139025e-06 -▁institutionelle 8.5769892184139025e-06 -▁instruct 8.5769892184139025e-06 -▁kürz 8.5769892184139025e-06 -▁productive 8.5769892184139025e-06 -▁visits 8.5769892184139025e-06 -▁endgültige 8.5732007956142488e-06 -▁integrity 8.5732007956142488e-06 -elung 8.5694123728145967e-06 -▁Ocean 8.5694123728145967e-06 -▁Tampere 8.5694123728145967e-06 -▁zwölf 8.5694123728145967e-06 -cript 8.565623950014943e-06 -ocal 8.565623950014943e-06 -▁Aufhebung 8.565623950014943e-06 -▁Cat 8.565623950014943e-06 -▁Thermal 8.565623950014943e-06 -▁buying 8.565623950014943e-06 -▁up-to-date 8.565623950014943e-06 -chnet 8.5618355272152909e-06 -▁Profil 8.5618355272152909e-06 -▁Third 8.5618355272152909e-06 -meister 8.5580471044156372e-06 -öpfen 8.5580471044156372e-06 -▁Nied 8.5580471044156372e-06 -▁könne 8.5580471044156372e-06 -▁relief 8.5580471044156372e-06 -antly 8.5542586816159852e-06 -ases 8.5542586816159852e-06 -▁beruh 8.5542586816159852e-06 -▁Chancengleichheit 8.5504702588163331e-06 -▁abges 8.5504702588163331e-06 -▁delays 8.5504702588163331e-06 -▁fortsetzen 8.5504702588163331e-06 -▁profound 8.5504702588163331e-06 -▁segment 8.5504702588163331e-06 -▁Dev 8.54668183601668e-06 -▁Kernel 8.54668183601668e-06 -▁Mid 8.54668183601668e-06 -▁Mischung 8.54668183601668e-06 -▁Tief 8.54668183601668e-06 -▁Zielsetzungen 8.54668183601668e-06 -▁Zusch 8.54668183601668e-06 -▁davor 8.54668183601668e-06 -▁declare 8.54668183601668e-06 -▁richt 8.54668183601668e-06 -dens 8.5428934132170273e-06 -▁Geschlechter 8.5428934132170273e-06 -▁accuracy 8.5428934132170273e-06 -▁renewed 8.5428934132170273e-06 -bern 8.5391049904173736e-06 -▁66 8.5391049904173736e-06 -▁broader 8.5391049904173736e-06 -▁flexibel 8.5391049904173736e-06 -oes 8.5353165676177215e-06 -▁SMS 8.5353165676177215e-06 -▁ceiling 8.5353165676177215e-06 -▁rob 8.5353165676177215e-06 -▁wunderschöne 8.5353165676177215e-06 -zeichn 8.5315281448180678e-06 -▁Erarbeitung 8.5315281448180678e-06 -▁Steigerung 8.5315281448180678e-06 -▁angekündigt 8.5315281448180678e-06 -▁tatsächlichen 8.5315281448180678e-06 -afe 8.5277397220184158e-06 -▁Eröffnung 8.5277397220184158e-06 -▁pace 8.5277397220184158e-06 -▁suited 8.5277397220184158e-06 -▁Ministerpräsident 8.523951299218762e-06 -▁Renaissance 8.523951299218762e-06 -▁fein 8.523951299218762e-06 -▁refuse 8.523951299218762e-06 -▁turns 8.523951299218762e-06 -▁С 8.523951299218762e-06 -bewegung 8.52016287641911e-06 -eding 8.52016287641911e-06 -▁Nationalpark 8.52016287641911e-06 -▁W. 8.52016287641911e-06 -▁distort 8.52016287641911e-06 -▁electoral 8.52016287641911e-06 -▁ethical 8.52016287641911e-06 -▁gestartet 8.52016287641911e-06 -▁übermitteln 8.52016287641911e-06 -▁Syria 8.5163744536194579e-06 -▁realisiert 8.5163744536194579e-06 -▁tap 8.5163744536194579e-06 -auen 8.5125860308198042e-06 -til 8.5125860308198042e-06 -▁Einnahmen 8.5125860308198042e-06 -▁Halle 8.5125860308198042e-06 -▁Representative 8.5125860308198042e-06 -▁sogenannte 8.5125860308198042e-06 -TC 8.5087976080201521e-06 -▁Intel 8.5087976080201521e-06 -▁advise 8.5087976080201521e-06 -▁sake 8.5087976080201521e-06 -größ 8.5050091852204984e-06 -leicht 8.5050091852204984e-06 -▁92 8.5050091852204984e-06 -▁ok 8.5050091852204984e-06 -bas 8.5012207624208464e-06 -▁Aug 8.5012207624208464e-06 -▁Deep 8.5012207624208464e-06 -▁Fischler 8.5012207624208464e-06 -▁Module 8.5012207624208464e-06 -▁Palette 8.5012207624208464e-06 -▁Pent 8.5012207624208464e-06 -▁Philipp 8.5012207624208464e-06 -▁Vertriebs 8.5012207624208464e-06 -▁automotive 8.5012207624208464e-06 -▁cult 8.5012207624208464e-06 -▁erforderliche 8.5012207624208464e-06 -asien 8.4974323396211926e-06 -rachen 8.4974323396211926e-06 -uced 8.4974323396211926e-06 -▁Betrifft 8.4974323396211926e-06 -▁Gefängnis 8.4974323396211926e-06 -▁arabischen 8.4974323396211926e-06 -▁eleganten 8.4974323396211926e-06 -▁Seen 8.49364391682154e-06 -▁exhibitions 8.49364391682154e-06 -▁spiel 8.49364391682154e-06 -▁vorsieht 8.49364391682154e-06 -indliche 8.4898554940218885e-06 -oga 8.4898554940218885e-06 -▁Send 8.4898554940218885e-06 -▁advent 8.4898554940218885e-06 -▁keys 8.4898554940218885e-06 -▁literature 8.4898554940218885e-06 -stone 8.4860670712222348e-06 -tent 8.4860670712222348e-06 -▁Derzeit 8.4860670712222348e-06 -▁Laser 8.4860670712222348e-06 -▁Millennium 8.4860670712222348e-06 -ktions 8.4822786484225828e-06 -▁Left 8.4822786484225828e-06 -▁präsent 8.4822786484225828e-06 -anke 8.478490225622929e-06 -▁Bosnien 8.478490225622929e-06 -▁Look 8.478490225622929e-06 -▁zukommen 8.478490225622929e-06 -möglichkeit 8.474701802823277e-06 -▁Baut 8.474701802823277e-06 -▁Einkommens 8.474701802823277e-06 -▁Ernst 8.474701802823277e-06 -▁Kenntnisse 8.474701802823277e-06 -▁Liberalen 8.474701802823277e-06 -▁logical 8.474701802823277e-06 -▁2015 8.4709133800236232e-06 -▁Georgien 8.4709133800236232e-06 -▁Händen 8.4709133800236232e-06 -▁Partnership 8.4709133800236232e-06 -▁Wikicars 8.4709133800236232e-06 -▁abgesehen 8.4709133800236232e-06 -▁empfängt 8.4709133800236232e-06 -▁observation 8.4709133800236232e-06 -▁stressed 8.4709133800236232e-06 -ached 8.4671249572239712e-06 -▁Ausnahmen 8.4671249572239712e-06 -▁Governance 8.4671249572239712e-06 -▁architectural 8.4671249572239712e-06 -ikanischer 8.4633365344243174e-06 -stücke 8.4633365344243174e-06 -▁Schluß 8.4633365344243174e-06 -▁initi 8.4633365344243174e-06 -▁irgende 8.4633365344243174e-06 -ask 8.4595481116246654e-06 -kap 8.4595481116246654e-06 -obby 8.4595481116246654e-06 -▁Tee 8.4595481116246654e-06 -▁granting 8.4595481116246654e-06 -Гј 8.4557596888250134e-06 -▁coal 8.4557596888250134e-06 -▁erstreckt 8.4557596888250134e-06 -▁herrschen 8.4557596888250134e-06 -▁zahl 8.4557596888250134e-06 -overs 8.45197126602536e-06 -▁Bureau 8.45197126602536e-06 -▁Defence 8.45197126602536e-06 -▁Gute 8.45197126602536e-06 -▁Lad 8.45197126602536e-06 -▁27. 8.4481828432257076e-06 -▁Corn 8.4481828432257076e-06 -▁Protest 8.4481828432257076e-06 --tra 8.4443944204260538e-06 -ventions 8.4443944204260538e-06 -▁Beschränkungen 8.4443944204260538e-06 -▁Loc 8.4443944204260538e-06 -▁Ufer 8.4443944204260538e-06 -▁Vit 8.4443944204260538e-06 -▁cameras 8.4443944204260538e-06 -▁deserve 8.4443944204260538e-06 -▁ere 8.4443944204260538e-06 -▁gal 8.4443944204260538e-06 -▁parlamentarischen 8.4443944204260538e-06 -▁wars 8.4443944204260538e-06 -ifications 8.4406059976264018e-06 -▁Arzneimittel 8.4406059976264018e-06 -▁Len 8.4406059976264018e-06 -▁mechanical 8.4406059976264018e-06 -entlich 8.4368175748267481e-06 -ş 8.4368175748267481e-06 -▁Cert 8.4368175748267481e-06 -▁Geschenk 8.4368175748267481e-06 -▁Mund 8.4368175748267481e-06 -▁angenommenen 8.4368175748267481e-06 -▁deleg 8.4368175748267481e-06 -▁novel 8.4368175748267481e-06 -▁therapy 8.4368175748267481e-06 -47 8.433029152027096e-06 -irat 8.433029152027096e-06 -▁Mengen 8.433029152027096e-06 -▁Soc 8.433029152027096e-06 -▁catalogue 8.433029152027096e-06 -▁mouth 8.433029152027096e-06 -▁sailing 8.433029152027096e-06 -icy 8.429240729227444e-06 -▁Distribution 8.429240729227444e-06 -▁Zerstörung 8.429240729227444e-06 -▁painting 8.429240729227444e-06 -▁researchers 8.429240729227444e-06 -grenze 8.42545230642779e-06 -lad 8.42545230642779e-06 -nitten 8.42545230642779e-06 -otte 8.42545230642779e-06 -urer 8.42545230642779e-06 -▁Photo 8.42545230642779e-06 -▁whenever 8.42545230642779e-06 -ammer 8.4216638836281382e-06 -ommission 8.4216638836281382e-06 -zing 8.4216638836281382e-06 -▁chose 8.4216638836281382e-06 -▁deutliche 8.4216638836281382e-06 -▁newspaper 8.4216638836281382e-06 -▁pensions 8.4216638836281382e-06 -▁zugestimmt 8.4216638836281382e-06 -apping 8.4178754608284844e-06 -PM 8.4140870380288324e-06 -▁Kompl 8.4140870380288324e-06 -▁Olymp 8.4140870380288324e-06 -▁emphas 8.4140870380288324e-06 -▁erheblichen 8.4140870380288324e-06 -▁innovations 8.4140870380288324e-06 -▁v. 8.4140870380288324e-06 -äge 8.4102986152291787e-06 -▁distinction 8.4102986152291787e-06 -▁trigger 8.4102986152291787e-06 -eans 8.4065101924295266e-06 -▁18th 8.4065101924295266e-06 -▁brands 8.4065101924295266e-06 -▁fiel 8.4065101924295266e-06 -▁publishing 8.4065101924295266e-06 -aufnahme 8.4027217696298746e-06 -chtet 8.4027217696298746e-06 -▁Brok 8.4027217696298746e-06 -▁Empfänger 8.4027217696298746e-06 -▁amending 8.4027217696298746e-06 -▁berichten 8.4027217696298746e-06 -▁confident 8.4027217696298746e-06 -▁duration 8.4027217696298746e-06 -▁erstens 8.4027217696298746e-06 -schlägen 8.39893334683022e-06 -▁verzichten 8.39893334683022e-06 -ichern 8.3951449240305688e-06 -verbindungen 8.3951449240305688e-06 -▁Klang 8.3951449240305688e-06 -▁achievements 8.3951449240305688e-06 -▁obligator 8.3951449240305688e-06 -▁170 8.3913565012309151e-06 -▁soldiers 8.3913565012309151e-06 -▁Rural 8.387568078431263e-06 -▁ähn 8.387568078431263e-06 -iveness 8.38377965563161e-06 -ska 8.38377965563161e-06 -▁Ell 8.38377965563161e-06 -▁Kost 8.38377965563161e-06 -▁Werkzeuge 8.38377965563161e-06 -▁compos 8.38377965563161e-06 -▁rac 8.38377965563161e-06 -ccess 8.3799912328319572e-06 -▁Hyp 8.3799912328319572e-06 -▁bewußt 8.3799912328319572e-06 -▁constructed 8.3799912328319572e-06 -▁erwähnten 8.3799912328319572e-06 -ather 8.3762028100323035e-06 -▁Händler 8.3762028100323035e-06 -▁ocean 8.3762028100323035e-06 -▁prestigious 8.3762028100323035e-06 -▁rom 8.3762028100323035e-06 -▁trains 8.3762028100323035e-06 -aine 8.3724143872326514e-06 -edy 8.3724143872326514e-06 -▁langem 8.3724143872326514e-06 -ults 8.368625964433e-06 -urance 8.368625964433e-06 -▁Größen 8.368625964433e-06 -▁einsch 8.368625964433e-06 -▁mathem 8.368625964433e-06 -▁pressing 8.368625964433e-06 -III 8.3648375416333457e-06 -onte 8.3648375416333457e-06 -schließt 8.3648375416333457e-06 -▁Alltag 8.3648375416333457e-06 -▁Südafrika 8.3648375416333457e-06 -▁Tarifa 8.3648375416333457e-06 -▁gate 8.3648375416333457e-06 -▁tough 8.3648375416333457e-06 -▁unt 8.3648375416333457e-06 -▁üblichen 8.3648375416333457e-06 -93 8.3610491188336936e-06 -reu 8.3610491188336936e-06 -verein 8.3610491188336936e-06 -▁EU- 8.3610491188336936e-06 -▁KOM 8.3610491188336936e-06 -▁anf 8.3610491188336936e-06 -▁bro 8.3610491188336936e-06 -▁officially 8.3610491188336936e-06 -▁regulated 8.3610491188336936e-06 -ässig 8.35726069603404e-06 -▁Beijing 8.35726069603404e-06 -▁Netze 8.35726069603404e-06 -▁Privatsphäre 8.35726069603404e-06 -▁abgest 8.35726069603404e-06 -▁cancelled 8.35726069603404e-06 -▁certificate 8.35726069603404e-06 -▁partic 8.35726069603404e-06 -funktionen 8.3534722732343878e-06 -▁Erfolgs 8.3534722732343878e-06 -▁Rennen 8.3534722732343878e-06 -▁leiten 8.3534722732343878e-06 -▁prin 8.3534722732343878e-06 -abhäng 8.3496838504347341e-06 -▁conversion 8.3496838504347341e-06 -▁freight 8.3496838504347341e-06 -▁größter 8.3496838504347341e-06 -▁schedule 8.3496838504347341e-06 -fortable 8.345895427635082e-06 -▁Erhalt 8.345895427635082e-06 -▁Prom 8.345895427635082e-06 -▁Rückgang 8.345895427635082e-06 -▁Versprechen 8.345895427635082e-06 -▁chemische 8.345895427635082e-06 -▁eröffnen 8.345895427635082e-06 -▁ursprüngliche 8.345895427635082e-06 -herr 8.34210700483543e-06 -klus 8.34210700483543e-06 -▁Charme 8.34210700483543e-06 -▁bot 8.34210700483543e-06 -▁explicitly 8.34210700483543e-06 -▁problemlos 8.34210700483543e-06 -ereien 8.3383185820357763e-06 -rufen 8.3383185820357763e-06 -▁Prize 8.3383185820357763e-06 -▁Wett 8.3383185820357763e-06 -▁evalu 8.3383185820357763e-06 -▁zun 8.3383185820357763e-06 -▁Arzt 8.3345301592361242e-06 -▁Gemeinsam 8.3345301592361242e-06 -eht 8.33074173643647e-06 -heil 8.33074173643647e-06 -▁Andr 8.33074173643647e-06 -▁Bach 8.33074173643647e-06 -▁Come 8.33074173643647e-06 -▁Inhalten 8.33074173643647e-06 -▁Tower 8.33074173643647e-06 -▁beneficial 8.33074173643647e-06 --Ex 8.3269533136368184e-06 -ARD 8.3269533136368184e-06 -abor 8.3269533136368184e-06 -gliche 8.3269533136368184e-06 -stechn 8.3269533136368184e-06 -▁Gesichtspunkt 8.3269533136368184e-06 -▁Grafik 8.3269533136368184e-06 -▁einzusch 8.3269533136368184e-06 -▁lakes 8.3269533136368184e-06 -▁preparations 8.3269533136368184e-06 -▁stops 8.3269533136368184e-06 -▁63 8.3231648908371647e-06 -▁Andalus 8.3231648908371647e-06 -▁consist 8.3231648908371647e-06 -▁lab 8.3231648908371647e-06 -▁niedrigen 8.3231648908371647e-06 -▁noon 8.3231648908371647e-06 -▁unusual 8.3231648908371647e-06 -▁Anzeige 8.3193764680375127e-06 -▁beteiligten 8.3193764680375127e-06 -▁doctor 8.3155880452378589e-06 -▁genießt 8.3155880452378589e-06 -▁interessanten 8.3155880452378589e-06 -▁komfortablen 8.3155880452378589e-06 -betr 8.3117996224382069e-06 -glied 8.3117996224382069e-06 -hört 8.3117996224382069e-06 -▁sandy 8.3117996224382069e-06 -▁vereint 8.3117996224382069e-06 -anca 8.3080111996385548e-06 -isp 8.3080111996385548e-06 -▁anpassen 8.3080111996385548e-06 -▁stärksten 8.3080111996385548e-06 -▁surveillance 8.3080111996385548e-06 -anged 8.3042227768389011e-06 -prozesse 8.3042227768389011e-06 -▁NRO 8.3042227768389011e-06 -▁observ 8.3042227768389011e-06 -▁steigt 8.3042227768389011e-06 -▁zurzeit 8.3042227768389011e-06 -AND 8.300434354039249e-06 -teilungen 8.300434354039249e-06 -▁concerts 8.300434354039249e-06 -▁gemeinsamer 8.300434354039249e-06 -▁populations 8.300434354039249e-06 -▁proximity 8.300434354039249e-06 -▁territories 8.300434354039249e-06 -inf 8.2966459312395953e-06 -uctive 8.2966459312395953e-06 -war 8.2966459312395953e-06 -reiber 8.2928575084399433e-06 -▁Montage 8.2928575084399433e-06 -▁PPE-DE 8.2928575084399433e-06 -aktionen 8.28906908564029e-06 -▁machinery 8.28906908564029e-06 -▁vermutlich 8.28906908564029e-06 -▁delete 8.2852806628406375e-06 -▁eindeutige 8.2852806628406375e-06 -▁einzig 8.2852806628406375e-06 -▁gehabt 8.2852806628406375e-06 -▁stretch 8.2852806628406375e-06 -▁toilet 8.2852806628406375e-06 -ahre 8.2814922400409854e-06 -blatt 8.2814922400409854e-06 -häng 8.2814922400409854e-06 -tin 8.2814922400409854e-06 -▁Buddh 8.2814922400409854e-06 -▁Großen 8.2814922400409854e-06 -▁Sri 8.2814922400409854e-06 -▁premium 8.2814922400409854e-06 -▁split 8.2814922400409854e-06 -▁wächst 8.2814922400409854e-06 -derung 8.2777038172413317e-06 -onie 8.2777038172413317e-06 -▁KG 8.2777038172413317e-06 -▁Konvergenz 8.2777038172413317e-06 -▁Kön 8.2777038172413317e-06 -▁erweisen 8.2777038172413317e-06 -FS 8.27391539444168e-06 -otes 8.27391539444168e-06 -▁Bürokratie 8.27391539444168e-06 -▁Deposit 8.27391539444168e-06 -▁Terroristen 8.27391539444168e-06 -▁focusing 8.27391539444168e-06 -▁gebunden 8.27391539444168e-06 -örung 8.2701269716420259e-06 -▁Ltd 8.2701269716420259e-06 -▁Natural 8.2701269716420259e-06 -▁short-term 8.2701269716420259e-06 -▁verabschieden 8.2701269716420259e-06 -ichtung 8.2663385488423739e-06 -▁Prop 8.2663385488423739e-06 -▁leistet 8.2663385488423739e-06 -▁smoking 8.2663385488423739e-06 -▁undertake 8.2663385488423739e-06 -▁Klassen 8.26255012604272e-06 -▁hab 8.26255012604272e-06 -opf 8.2587617032430681e-06 -▁Minibar 8.2587617032430681e-06 -▁Repräsent 8.2587617032430681e-06 -▁greenhouse 8.2587617032430681e-06 -▁ignore 8.2587617032430681e-06 -▁д 8.2587617032430681e-06 --Br 8.2549732804434143e-06 -▁Som 8.2549732804434143e-06 -▁einiges 8.2549732804434143e-06 -▁entrepreneurs 8.2549732804434143e-06 -▁identification 8.2549732804434143e-06 -▁movies 8.2549732804434143e-06 -▁pays 8.2549732804434143e-06 -▁zentraler 8.2549732804434143e-06 -itar 8.2511848576437623e-06 -änderung 8.2511848576437623e-06 -äuf 8.2511848576437623e-06 -▁Begründung 8.2511848576437623e-06 -▁burn 8.2511848576437623e-06 -▁dieselben 8.2511848576437623e-06 -▁differ 8.2511848576437623e-06 -▁eigentliche 8.2511848576437623e-06 -▁employers 8.2511848576437623e-06 -ixel 8.24739643484411e-06 -zut 8.24739643484411e-06 -▁partition 8.24739643484411e-06 -▁publications 8.24739643484411e-06 -▁typically 8.24739643484411e-06 -▁Profit 8.2436080120444565e-06 -▁Wahr 8.2436080120444565e-06 -▁arrange 8.2436080120444565e-06 -▁dieselbe 8.2436080120444565e-06 -▁solo 8.2436080120444565e-06 -▁umso 8.2436080120444565e-06 -kunst 8.2398195892448045e-06 -▁Ausgrenzung 8.2398195892448045e-06 -▁Wiederherstellung 8.2398195892448045e-06 -▁erarbeiten 8.2398195892448045e-06 -▁harbour 8.2398195892448045e-06 -▁zugute 8.2398195892448045e-06 -beg 8.23603116644515e-06 -ießt 8.23603116644515e-06 -▁centers 8.23603116644515e-06 -idem 8.2322427436454987e-06 -kapital 8.2322427436454987e-06 -▁Alban 8.2322427436454987e-06 -▁Brücke 8.2322427436454987e-06 -▁komplexen 8.2322427436454987e-06 -▁prevented 8.2322427436454987e-06 -formation 8.228454320845845e-06 -polit 8.228454320845845e-06 -äume 8.228454320845845e-06 -▁Updates 8.228454320845845e-06 -▁afrikanischen 8.228454320845845e-06 -▁angeb 8.228454320845845e-06 -▁bege 8.228454320845845e-06 -▁drives 8.228454320845845e-06 -▁necessity 8.228454320845845e-06 -▁precious 8.228454320845845e-06 -▁rechnen 8.228454320845845e-06 -21 8.2246658980461929e-06 -▁lets 8.2246658980461929e-06 -▁organiz 8.2208774752465409e-06 -aft 8.2170890524468871e-06 -gre 8.2170890524468871e-06 -kar 8.2170890524468871e-06 -▁Capital 8.2170890524468871e-06 -▁Kreat 8.2170890524468871e-06 -▁gelangt 8.2170890524468871e-06 -▁Planet 8.2133006296472351e-06 -▁roots 8.2133006296472351e-06 -▁suspended 8.2133006296472351e-06 -▁trav 8.2133006296472351e-06 -reform 8.2095122068475813e-06 -▁Everything 8.2095122068475813e-06 -▁catalog 8.2095122068475813e-06 -▁os 8.2095122068475813e-06 -▁releases 8.2095122068475813e-06 -sprachen 8.20572378404793e-06 -æ 8.20572378404793e-06 -▁Lasten 8.20572378404793e-06 -▁List 8.20572378404793e-06 -▁Umstände 8.20572378404793e-06 -▁civilian 8.20572378404793e-06 -rän 8.2019353612482756e-06 -▁Formulierung 8.1981469384486235e-06 -▁verliehen 8.1981469384486235e-06 -egung 8.1943585156489715e-06 -etti 8.1943585156489715e-06 -usten 8.1943585156489715e-06 -ündet 8.1943585156489715e-06 -▁Bewohner 8.1943585156489715e-06 -▁eingeladen 8.1943585156489715e-06 -▁trained 8.1943585156489715e-06 -di 8.1905700928493177e-06 -▁Einsch 8.1905700928493177e-06 -▁Nic 8.1905700928493177e-06 -▁eintreten 8.1905700928493177e-06 -▁formuliert 8.1905700928493177e-06 -▁kan 8.1905700928493177e-06 -▁pen 8.1905700928493177e-06 -▁verwiesen 8.1905700928493177e-06 -▁2nd 8.1867816700496657e-06 -▁akzept 8.1867816700496657e-06 -▁conform 8.1867816700496657e-06 -▁schwe 8.1867816700496657e-06 -▁vermieden 8.1867816700496657e-06 -entlichen 8.1829932472500119e-06 -schreit 8.1829932472500119e-06 -spunkt 8.1829932472500119e-06 -▁Art. 8.1829932472500119e-06 -▁preserved 8.1829932472500119e-06 -▁österreichischen 8.1829932472500119e-06 -ested 8.17920482445036e-06 -▁125 8.17920482445036e-06 -▁Nichtraucherzonen 8.17920482445036e-06 -▁debated 8.17920482445036e-06 -▁signals 8.17920482445036e-06 -ali 8.1754164016507062e-06 -head 8.1754164016507062e-06 -iking 8.1754164016507062e-06 -▁ausführlich 8.1754164016507062e-06 -▁entr 8.1754164016507062e-06 -▁uncertainty 8.1754164016507062e-06 --Gruppe 8.1716279788510541e-06 --Mitglied 8.1716279788510541e-06 -dere 8.1716279788510541e-06 -ikes 8.1716279788510541e-06 -▁Dest 8.1716279788510541e-06 -▁Flor 8.1716279788510541e-06 -▁arrest 8.1716279788510541e-06 -▁befürwortet 8.1716279788510541e-06 -▁gewünschte 8.1716279788510541e-06 -▁moder 8.1716279788510541e-06 -▁transnational 8.1716279788510541e-06 -▁adjustment 8.1678395560514e-06 -▁favourite 8.1678395560514e-06 -▁interactive 8.1678395560514e-06 -▁invent 8.1678395560514e-06 -▁unions 8.1678395560514e-06 -obe 8.1640511332517483e-06 -▁Problematik 8.1640511332517483e-06 -▁Solutions 8.1640511332517483e-06 -▁Taliban 8.1640511332517483e-06 -▁logistics 8.1640511332517483e-06 -▁revealed 8.1640511332517483e-06 -nah 8.1602627104520963e-06 -orb 8.1602627104520963e-06 -▁admit 8.1602627104520963e-06 -esk 8.1564742876524426e-06 -▁Farm 8.1564742876524426e-06 -▁jüngste 8.1564742876524426e-06 -▁regionaler 8.1564742876524426e-06 -▁1930 8.15268586485279e-06 -▁Fakten 8.15268586485279e-06 -▁Kim 8.15268586485279e-06 -▁ment 8.15268586485279e-06 -▁senior 8.15268586485279e-06 -isons 8.1488974420531368e-06 -kriterien 8.1488974420531368e-06 -lesen 8.1488974420531368e-06 -▁GB 8.1488974420531368e-06 -▁Leon 8.1488974420531368e-06 -▁unknown 8.1488974420531368e-06 -▁musikal 8.1451090192534847e-06 -▁organize 8.1451090192534847e-06 -verteilung 8.141320596453831e-06 -▁Orient 8.141320596453831e-06 -▁dasselbe 8.141320596453831e-06 -▁insert 8.141320596453831e-06 -▁moderner 8.141320596453831e-06 -▁telecommunications 8.141320596453831e-06 -zeichnete 8.1375321736541789e-06 -▁Elektro 8.1375321736541789e-06 -▁Istanbul 8.1375321736541789e-06 -▁Night 8.1375321736541789e-06 -▁ents 8.1375321736541789e-06 -▁ermutigen 8.1375321736541789e-06 -verarbeitung 8.1337437508545269e-06 -▁DIN 8.1337437508545269e-06 -▁Logik 8.1337437508545269e-06 -▁Premier 8.1337437508545269e-06 -▁gegenseitigen 8.1337437508545269e-06 -annter 8.1299553280548732e-06 -aters 8.1299553280548732e-06 -wehr 8.1299553280548732e-06 -▁Superm 8.1299553280548732e-06 -▁ang 8.1299553280548732e-06 -andem 8.1261669052552211e-06 -rec 8.1261669052552211e-06 -▁GUE 8.1261669052552211e-06 -▁Menschenrechtsverletzungen 8.1261669052552211e-06 -▁frustr 8.1261669052552211e-06 -▁vertraut 8.1261669052552211e-06 -gekommen 8.1223784824555674e-06 -zentr 8.1223784824555674e-06 -▁Abenteuer 8.1223784824555674e-06 -▁Jugendliche 8.1223784824555674e-06 -▁PPE 8.1223784824555674e-06 -▁Zunahme 8.1223784824555674e-06 -▁winner 8.1223784824555674e-06 -möglichen 8.1185900596559153e-06 -▁Monti 8.1185900596559153e-06 -▁comprises 8.1185900596559153e-06 -▁hospital 8.1185900596559153e-06 -▁taxpayers 8.1185900596559153e-06 -bor 8.1148016368562616e-06 -stellte 8.1148016368562616e-06 -▁Aluminium 8.1148016368562616e-06 -▁judge 8.1148016368562616e-06 -umen 8.11101321405661e-06 -▁67 8.11101321405661e-06 -▁Experience 8.11101321405661e-06 -▁Rap 8.11101321405661e-06 -▁negotiated 8.11101321405661e-06 -auber 8.1072247912569558e-06 -izenz 8.1072247912569558e-06 -klar 8.1072247912569558e-06 -▁185 8.1072247912569558e-06 -▁photographs 8.1072247912569558e-06 -▁transit 8.1072247912569558e-06 -empt 8.1034363684573038e-06 -name 8.1034363684573038e-06 -▁Föderation 8.1034363684573038e-06 -▁Ker 8.1034363684573038e-06 -▁Spiegel 8.1034363684573038e-06 -▁attended 8.1034363684573038e-06 -▁kleineren 8.1034363684573038e-06 -▁verschaffen 8.1034363684573038e-06 -istor 8.0996479456576517e-06 -▁Verabschiedung 8.0996479456576517e-06 -▁aufgestellt 8.0996479456576517e-06 -▁sach 8.0996479456576517e-06 -▁schwieriger 8.0996479456576517e-06 -▁treas 8.0996479456576517e-06 -elemente 8.095859522857998e-06 -inc 8.095859522857998e-06 -osition 8.095859522857998e-06 -steil 8.095859522857998e-06 -▁abgeben 8.095859522857998e-06 -▁aussehen 8.095859522857998e-06 -▁bedauerlich 8.095859522857998e-06 -▁feedback 8.095859522857998e-06 -▁resp 8.095859522857998e-06 -▁spiritual 8.095859522857998e-06 -erweise 8.0920711000583459e-06 -hrte 8.0920711000583459e-06 -iehen 8.0920711000583459e-06 -situation 8.0920711000583459e-06 -ás 8.0920711000583459e-06 -ähe 8.0920711000583459e-06 -▁Ausbeutung 8.0920711000583459e-06 -▁ruling 8.0920711000583459e-06 -denken 8.0882826772586922e-06 -ennen 8.0882826772586922e-06 -oden 8.0882826772586922e-06 -wirkt 8.0882826772586922e-06 -▁Astr 8.0882826772586922e-06 -▁José 8.0882826772586922e-06 -▁leichte 8.0882826772586922e-06 -▁presidential 8.0882826772586922e-06 -▁reflection 8.0882826772586922e-06 -▁sel 8.0882826772586922e-06 -▁spare 8.0882826772586922e-06 -▁thanking 8.0882826772586922e-06 -▁vergangen 8.0882826772586922e-06 -▁на 8.0882826772586922e-06 -vin 8.08449425445904e-06 -▁Möbel 8.08449425445904e-06 -▁Which 8.08449425445904e-06 -▁indicators 8.08449425445904e-06 -▁integrieren 8.08449425445904e-06 -▁quot 8.08449425445904e-06 -▁Bürgerbeauftragten 8.0807058316593864e-06 -▁conservation 8.0807058316593864e-06 -▁geeigneten 8.0807058316593864e-06 -▁realize 8.0807058316593864e-06 -▁sterben 8.0807058316593864e-06 -▁Hussein 8.0769174088597344e-06 -▁kom 8.0769174088597344e-06 -▁löschen 8.0769174088597344e-06 -▁synth 8.0769174088597344e-06 -alte 8.0731289860600823e-06 -mechanismen 8.0731289860600823e-06 -uba 8.0731289860600823e-06 -▁78 8.0731289860600823e-06 -▁Empf 8.0731289860600823e-06 -▁Funktionieren 8.0731289860600823e-06 -▁Lissabon-Strategie 8.0731289860600823e-06 -▁treiben 8.0731289860600823e-06 -ouv 8.0693405632604286e-06 -▁einstellen 8.0693405632604286e-06 -▁illness 8.0693405632604286e-06 -▁medizinische 8.0693405632604286e-06 -▁mess 8.0693405632604286e-06 -▁threatened 8.0693405632604286e-06 -▁unbek 8.0693405632604286e-06 -omas 8.0655521404607765e-06 -pan 8.0655521404607765e-06 -▁Pflichten 8.0655521404607765e-06 -oys 8.0617637176611228e-06 -▁Until 8.0617637176611228e-06 -▁fails 8.0617637176611228e-06 -▁gewöhn 8.0617637176611228e-06 -▁pos 8.0617637176611228e-06 -▁Haft 8.05797529486147e-06 -▁Wärme 8.05797529486147e-06 -▁hesitate 8.05797529486147e-06 -bringung 8.054186872061817e-06 -gast 8.054186872061817e-06 -ль 8.054186872061817e-06 -▁Bow 8.054186872061817e-06 -▁CE 8.054186872061817e-06 -▁bist 8.054186872061817e-06 --Datei 8.050398449262165e-06 -ли 8.050398449262165e-06 -▁Sym 8.050398449262165e-06 -▁Wälder 8.050398449262165e-06 -▁ausländische 8.050398449262165e-06 -▁heutzutage 8.050398449262165e-06 -gebühren 8.0466100264625129e-06 -rob 8.0466100264625129e-06 -▁Arbeitszeit 8.0466100264625129e-06 -▁Inhalts 8.0466100264625129e-06 -▁Wesen 8.0466100264625129e-06 -gement 8.04282160366286e-06 -▁88 8.04282160366286e-06 -▁Finanzen 8.04282160366286e-06 -▁Holland 8.04282160366286e-06 -▁Kuba 8.04282160366286e-06 -▁Meinungen 8.04282160366286e-06 -▁Non-Smoking 8.04282160366286e-06 -▁Theorie 8.04282160366286e-06 -▁academic 8.04282160366286e-06 -▁wahre 8.04282160366286e-06 -▁Bevölkerungs 8.0390331808632071e-06 -▁Französisch 8.0390331808632071e-06 -▁Katal 8.0390331808632071e-06 -▁Produktivität 8.0390331808632071e-06 -▁Span 8.0390331808632071e-06 -▁acquisition 8.0390331808632071e-06 -▁bat 8.0390331808632071e-06 -komp 8.0352447580635534e-06 -pective 8.0352447580635534e-06 -strukturen 8.0352447580635534e-06 -▁Individual 8.0352447580635534e-06 -▁ratified 8.0352447580635534e-06 -▁sonstige 8.0352447580635534e-06 -olen 8.0314563352639014e-06 -orrow 8.0314563352639014e-06 -▁Handlungen 8.0314563352639014e-06 -▁Inhaber 8.0314563352639014e-06 -▁Mis 8.0314563352639014e-06 -▁durchzusetzen 8.0314563352639014e-06 -▁vereinbar 8.0314563352639014e-06 -.S 8.0276679124642476e-06 -▁purchasing 8.0276679124642476e-06 -▁späteren 8.0276679124642476e-06 -▁teilt 8.0276679124642476e-06 -ieve 8.0238794896645956e-06 -▁RF 8.0238794896645956e-06 -▁auss 8.0238794896645956e-06 -rant 8.0200910668649418e-06 -▁Comb 8.0200910668649418e-06 -▁Kapazität 8.0200910668649418e-06 -▁Kommissionsvorschlag 8.0200910668649418e-06 -▁Mountain 8.0200910668649418e-06 -▁gerechtfertigt 8.0200910668649418e-06 -▁heiß 8.0200910668649418e-06 -.jpg 8.01630264406529e-06 -CA 8.01630264406529e-06 -cor 8.01630264406529e-06 -▁Kennzeichnung 8.01630264406529e-06 -▁bezogen 8.01630264406529e-06 -▁gratis 8.01630264406529e-06 -▁nachhaltig 8.01630264406529e-06 -▁präzise 8.01630264406529e-06 -achtet 8.0125142212656378e-06 -ipel 8.0125142212656378e-06 -▁Genuss 8.0125142212656378e-06 -▁ranging 8.0125142212656378e-06 -▁replacement 8.0125142212656378e-06 -▁supplementary 8.0125142212656378e-06 -ieder 8.008725798465984e-06 -▁Angabe 8.008725798465984e-06 -▁Cra 8.008725798465984e-06 -▁Intern 8.008725798465984e-06 -▁Quoten 8.008725798465984e-06 -▁gebe 8.008725798465984e-06 -▁mistakes 8.008725798465984e-06 -▁silver 8.008725798465984e-06 -amar 8.004937375666332e-06 -▁Shopping 8.004937375666332e-06 -▁certainty 8.004937375666332e-06 -ен 8.0011489528666782e-06 -▁Abgeordneter 8.0011489528666782e-06 -▁Fir 8.0011489528666782e-06 -▁Stufe 8.0011489528666782e-06 -▁endgültigen 8.0011489528666782e-06 -wart 7.9973605300670262e-06 -▁Tw 7.9973605300670262e-06 -tract 7.9935721072673725e-06 -▁privileged 7.9935721072673725e-06 -▁rating 7.9935721072673725e-06 -iny 7.98978368446772e-06 -▁bisschen 7.98978368446772e-06 -▁chron 7.98978368446772e-06 -▁geringe 7.98978368446772e-06 -▁jeglicher 7.98978368446772e-06 -▁lim 7.98978368446772e-06 -museum 7.9859952616680684e-06 -▁Türen 7.9859952616680684e-06 -▁Verordnungen 7.9859952616680684e-06 -▁kommun 7.9859952616680684e-06 -▁Gate 7.9822068388684146e-06 -▁Verfolgung 7.9822068388684146e-06 -▁fehl 7.9822068388684146e-06 -▁hinein 7.9822068388684146e-06 -48 7.9784184160687626e-06 -estra 7.9784184160687626e-06 -▁Bilanz 7.9784184160687626e-06 -▁Drittstaaten 7.9784184160687626e-06 -▁Technologies 7.9784184160687626e-06 -ulf 7.9746299932691088e-06 -vey 7.9746299932691088e-06 -ütt 7.9746299932691088e-06 -▁Nat 7.9746299932691088e-06 -führungs 7.9708415704694568e-06 -gio 7.9708415704694568e-06 -rolle 7.9708415704694568e-06 -▁Kolle 7.9708415704694568e-06 -▁downloaded 7.9708415704694568e-06 -kungen 7.9670531476698031e-06 -▁360 7.9670531476698031e-06 -▁Anträge 7.9670531476698031e-06 -▁Argentina 7.9670531476698031e-06 -▁höchster 7.9670531476698031e-06 -▁retten 7.9670531476698031e-06 -▁ungewöhn 7.9670531476698031e-06 -▁verarbeitet 7.9670531476698031e-06 -gestellten 7.963264724870151e-06 -▁Li 7.963264724870151e-06 -▁bewertet 7.963264724870151e-06 -▁gesichert 7.963264724870151e-06 -▁imported 7.963264724870151e-06 -▁Konjunktur 7.9594763020704973e-06 -▁künst 7.9594763020704973e-06 -edo 7.9556878792708452e-06 -zeichneten 7.9556878792708452e-06 -▁Document 7.9556878792708452e-06 -▁Young 7.9556878792708452e-06 -▁basieren 7.9556878792708452e-06 -▁ess 7.9556878792708452e-06 -▁monument 7.9556878792708452e-06 --Über 7.9518994564711932e-06 -cast 7.9518994564711932e-06 -lossenen 7.9518994564711932e-06 -mente 7.9518994564711932e-06 -suchen 7.9518994564711932e-06 -▁Jordan 7.9518994564711932e-06 -▁Soft 7.9518994564711932e-06 -▁Watch 7.9518994564711932e-06 -▁wid 7.9518994564711932e-06 -iko 7.94811103367154e-06 -pel 7.94811103367154e-06 -▁237 7.94811103367154e-06 -▁Beit 7.94811103367154e-06 -▁demonstrates 7.94811103367154e-06 -▁farms 7.94811103367154e-06 -▁justify 7.94811103367154e-06 -▁negotiate 7.94811103367154e-06 -hausen 7.9443226108718874e-06 -ometer 7.9443226108718874e-06 -schluß 7.9443226108718874e-06 -steh 7.9443226108718874e-06 -teen 7.9443226108718874e-06 -▁1979 7.9443226108718874e-06 -▁Bewerber 7.9443226108718874e-06 -▁alike 7.9443226108718874e-06 -▁komplexe 7.9443226108718874e-06 -osh 7.9405341880722337e-06 -▁Mitglieds 7.9405341880722337e-06 -▁intent 7.9405341880722337e-06 -▁patterns 7.9405341880722337e-06 -mechanismus 7.9367457652725816e-06 -nungen 7.9367457652725816e-06 -▁deaths 7.9367457652725816e-06 -prop 7.9329573424729279e-06 -▁Bühne 7.9329573424729279e-06 -▁Kernkraft 7.9329573424729279e-06 -▁authentic 7.9329573424729279e-06 -▁citizenship 7.9329573424729279e-06 -▁generous 7.9329573424729279e-06 -▁mittlere 7.9329573424729279e-06 -▁religiösen 7.9329573424729279e-06 -▁wachsende 7.9329573424729279e-06 -▁Adriatic 7.9291689196732758e-06 -▁Summe 7.9291689196732758e-06 -▁announce 7.9291689196732758e-06 -▁moves 7.9291689196732758e-06 -100 7.9253804968736238e-06 -▁9001 7.9253804968736238e-06 -▁Catal 7.9253804968736238e-06 -▁Kart 7.9253804968736238e-06 -▁Num 7.9253804968736238e-06 -,7 7.92159207407397e-06 -nisses 7.92159207407397e-06 -▁Abstimmungs 7.92159207407397e-06 -▁Fischereipolitik 7.92159207407397e-06 -▁Picard 7.92159207407397e-06 -▁Zusammensetzung 7.92159207407397e-06 -▁boy 7.92159207407397e-06 -▁flying 7.92159207407397e-06 -▁tal 7.92159207407397e-06 --Str 7.917803651274318e-06 -ypt 7.917803651274318e-06 -▁Marokko 7.917803651274318e-06 -▁downtown 7.917803651274318e-06 -▁obs 7.917803651274318e-06 -▁patents 7.917803651274318e-06 -▁unterscheidet 7.917803651274318e-06 --Ch 7.9140152284746643e-06 -ouse 7.9140152284746643e-06 -ürzt 7.9140152284746643e-06 -ка 7.9140152284746643e-06 -▁enrich 7.9140152284746643e-06 -▁signing 7.9140152284746643e-06 -real 7.9102268056750122e-06 -▁ratio 7.9102268056750122e-06 -ienen 7.9064383828753585e-06 -▁Autom 7.9064383828753585e-06 -▁Bat 7.9064383828753585e-06 -▁Literatur 7.9064383828753585e-06 -▁NOT 7.9064383828753585e-06 -▁Roc 7.9064383828753585e-06 -▁Versuche 7.9064383828753585e-06 -▁angebotenen 7.9064383828753585e-06 -▁aufeinander 7.9064383828753585e-06 -▁consistently 7.9064383828753585e-06 -▁gestiegen 7.9064383828753585e-06 -▁heavily 7.9064383828753585e-06 -▁horizontal 7.9064383828753585e-06 -▁nationals 7.9064383828753585e-06 -lov 7.9026499600757064e-06 -ą 7.9026499600757064e-06 -▁accum 7.9026499600757064e-06 -▁cho 7.9026499600757064e-06 -▁depl 7.9026499600757064e-06 -▁occurs 7.9026499600757064e-06 -▁Conditions 7.8988615372760527e-06 -▁FA 7.8988615372760527e-06 -▁Heraus 7.8988615372760527e-06 -▁blieb 7.8988615372760527e-06 -▁empfiehlt 7.8988615372760527e-06 -▁icon 7.8988615372760527e-06 -▁notably 7.8988615372760527e-06 -▁passieren 7.8988615372760527e-06 -austausch 7.8950731144764e-06 -iente 7.8950731144764e-06 -▁188 7.8950731144764e-06 -▁Fremd 7.8950731144764e-06 -▁Hon 7.8950731144764e-06 -▁Museums 7.8950731144764e-06 -erba 7.8912846916767486e-06 -▁Anreize 7.8912846916767486e-06 -▁Dun 7.8912846916767486e-06 -▁liquid 7.8912846916767486e-06 -▁maximale 7.8912846916767486e-06 -▁wertvolle 7.8912846916767486e-06 -artet 7.8874962688770949e-06 -▁claimed 7.8874962688770949e-06 -▁emerged 7.8874962688770949e-06 -▁fäl 7.8874962688770949e-06 -icon 7.8837078460774428e-06 -iese 7.8837078460774428e-06 -▁Funktions 7.8837078460774428e-06 -▁Klimawandels 7.8837078460774428e-06 -▁Santer 7.8837078460774428e-06 -▁argue 7.8837078460774428e-06 -▁chemicals 7.8837078460774428e-06 -rück 7.8799194232777891e-06 -▁Spektrum 7.8799194232777891e-06 -▁XV 7.8799194232777891e-06 -▁beinahe 7.8799194232777891e-06 -▁clothing 7.8799194232777891e-06 -▁deficits 7.8799194232777891e-06 -▁favorite 7.8799194232777891e-06 -▁indicates 7.8799194232777891e-06 -▁recession 7.8799194232777891e-06 -▁worrying 7.8799194232777891e-06 -.. 7.876131000478137e-06 -aga 7.876131000478137e-06 -inischen 7.876131000478137e-06 -äder 7.876131000478137e-06 -▁FL 7.876131000478137e-06 -▁Scotland 7.876131000478137e-06 -▁bild 7.876131000478137e-06 -▁leather 7.876131000478137e-06 -▁mandatory 7.876131000478137e-06 -▁phones 7.876131000478137e-06 -AF 7.8723425776784833e-06 -uropäischen 7.8723425776784833e-06 -▁FAQ 7.8685541548788313e-06 -▁Gebühr 7.8685541548788313e-06 -▁Herkunfts 7.8685541548788313e-06 -▁Verlängerung 7.8685541548788313e-06 -▁sessions 7.8685541548788313e-06 -▁tram 7.8685541548788313e-06 -anto 7.86476573207918e-06 -ivate 7.86476573207918e-06 -▁Cy 7.86476573207918e-06 -▁beschreibt 7.86476573207918e-06 -▁bureaucracy 7.86476573207918e-06 -▁como 7.86476573207918e-06 -▁extens 7.86476573207918e-06 -▁ruhiger 7.86476573207918e-06 -illo 7.8609773092795255e-06 -rauch 7.8609773092795255e-06 -wendet 7.8609773092795255e-06 -ś 7.8609773092795255e-06 -▁airline 7.8609773092795255e-06 -▁dein 7.8609773092795255e-06 -▁orientation 7.8609773092795255e-06 -▁Fass 7.8571888864798734e-06 -▁apparently 7.8571888864798734e-06 -chem 7.85340046368022e-06 -ference 7.85340046368022e-06 -gener 7.85340046368022e-06 -owers 7.85340046368022e-06 -▁Aer 7.85340046368022e-06 -▁District 7.85340046368022e-06 -▁fanden 7.85340046368022e-06 -▁myth 7.85340046368022e-06 --cl 7.8496120408805677e-06 -aßnahmen 7.8496120408805677e-06 -fung 7.8496120408805677e-06 -igation 7.8496120408805677e-06 -leitungen 7.8496120408805677e-06 -▁closing 7.8496120408805677e-06 -▁creature 7.8496120408805677e-06 -▁füllen 7.8496120408805677e-06 -ego 7.8458236180809139e-06 -inth 7.8458236180809139e-06 -prä 7.8458236180809139e-06 -▁klass 7.8458236180809139e-06 -anas 7.8420351952812619e-06 -space 7.8420351952812619e-06 -ußen 7.8420351952812619e-06 -▁77 7.8420351952812619e-06 -▁Madeira 7.8420351952812619e-06 -▁Sul 7.8420351952812619e-06 -▁einstimmig 7.8420351952812619e-06 -▁returns 7.8420351952812619e-06 -▁Autoren 7.83824677248161e-06 -▁File 7.83824677248161e-06 -▁Hi 7.83824677248161e-06 -▁cooper 7.83824677248161e-06 -▁tre 7.83824677248161e-06 -▁notified 7.8344583496819561e-06 -▁Einwanderer 7.830669926882304e-06 -▁Palästinenser 7.830669926882304e-06 -▁applicant 7.830669926882304e-06 -▁predict 7.830669926882304e-06 -ardo 7.82688150408265e-06 -onts 7.82688150408265e-06 -▁Betrug 7.82688150408265e-06 -▁Excellent 7.82688150408265e-06 -▁Osteuropa 7.82688150408265e-06 -▁advertis 7.82688150408265e-06 -▁aggress 7.82688150408265e-06 -▁precision 7.82688150408265e-06 -▁vereinbarten 7.82688150408265e-06 -idel 7.8230930812829983e-06 -lassungen 7.8230930812829983e-06 -▁Cav 7.8230930812829983e-06 -▁Terminal 7.8230930812829983e-06 -▁architect 7.8230930812829983e-06 -▁aufgeführt 7.8230930812829983e-06 -▁cooking 7.8230930812829983e-06 -▁magic 7.8230930812829983e-06 -▁specify 7.8230930812829983e-06 -haushalt 7.8193046584833445e-06 -▁Darfur 7.8193046584833445e-06 -▁contributing 7.8193046584833445e-06 -ei 7.8155162356836925e-06 -prü 7.8155162356836925e-06 -▁affili 7.8155162356836925e-06 -▁intervene 7.8155162356836925e-06 -▁spektakul 7.8155162356836925e-06 -idd 7.8117278128840387e-06 -igu 7.8117278128840387e-06 -▁Tob 7.8117278128840387e-06 -▁displays 7.8117278128840387e-06 -iessen 7.8079393900843867e-06 -istenz 7.8079393900843867e-06 -methode 7.8079393900843867e-06 -▁License 7.8079393900843867e-06 -▁Wiener 7.8079393900843867e-06 -händler 7.8041509672847347e-06 -▁Bridge 7.8041509672847347e-06 -▁conventions 7.8041509672847347e-06 -▁dargelegt 7.8041509672847347e-06 -▁drinking 7.8041509672847347e-06 -▁technischer 7.8041509672847347e-06 -ews 7.8003625444850809e-06 -▁Beurteilung 7.8003625444850809e-06 -▁Erzeugung 7.8003625444850809e-06 -▁beschleunigen 7.8003625444850809e-06 -▁reveal 7.8003625444850809e-06 -▁vac 7.8003625444850809e-06 -ASS 7.7965741216854289e-06 -▁archive 7.7965741216854289e-06 -▁organisierten 7.7965741216854289e-06 -EO 7.7927856988857751e-06 -ierbar 7.7927856988857751e-06 -ugnis 7.7927856988857751e-06 -▁Brun 7.7927856988857751e-06 -▁anal 7.7927856988857751e-06 -▁angelegt 7.7927856988857751e-06 -▁tag 7.7927856988857751e-06 -▁übergeben 7.7927856988857751e-06 -altungs 7.7889972760861231e-06 -▁Olympus 7.7889972760861231e-06 -▁geschickt 7.7889972760861231e-06 -▁refund 7.7889972760861231e-06 -▁wherever 7.7889972760861231e-06 -apter 7.78520885328647e-06 -eing 7.78520885328647e-06 -kehrt 7.78520885328647e-06 -▁Menschenrechten 7.78520885328647e-06 -▁Referendum 7.78520885328647e-06 -▁englischen 7.78520885328647e-06 -▁geplanten 7.78520885328647e-06 -▁marginal 7.78520885328647e-06 -agements 7.7814204304868173e-06 -ologischer 7.7814204304868173e-06 -▁Tank 7.7814204304868173e-06 -▁Verfasser 7.7814204304868173e-06 -▁Visit 7.7814204304868173e-06 -▁kit 7.7814204304868173e-06 -▁prominent 7.7814204304868173e-06 -▁repräsent 7.7814204304868173e-06 -einen 7.7776320076871653e-06 -menü 7.7776320076871653e-06 -rez 7.7776320076871653e-06 -ungssysteme 7.7776320076871653e-06 -ährt 7.7776320076871653e-06 -▁Absch 7.7776320076871653e-06 -▁Strasse 7.7776320076871653e-06 -▁£ 7.7776320076871653e-06 -▁überwacht 7.7776320076871653e-06 -ismo 7.7738435848875115e-06 -▁Anwesenheit 7.7738435848875115e-06 -▁ausreichende 7.7738435848875115e-06 -▁mächt 7.7738435848875115e-06 -fahrten 7.77005516208786e-06 -verage 7.77005516208786e-06 -▁Albert 7.77005516208786e-06 -▁Bud 7.77005516208786e-06 -▁Erwägung 7.77005516208786e-06 -▁Klar 7.77005516208786e-06 -▁withdrawal 7.77005516208786e-06 -bauen 7.7662667392882057e-06 -büro 7.7662667392882057e-06 -ease 7.7662667392882057e-06 -pat 7.7662667392882057e-06 -▁Schla 7.7662667392882057e-06 -▁ham 7.7662667392882057e-06 -▁nov 7.7662667392882057e-06 -▁restaur 7.7662667392882057e-06 -udd 7.7624783164885537e-06 -▁Sab 7.7624783164885537e-06 -▁gelassen 7.7624783164885537e-06 -▁passt 7.7624783164885537e-06 -sters 7.7586898936889e-06 -▁Mary 7.7586898936889e-06 -▁eigenständ 7.7586898936889e-06 -▁evaluate 7.7586898936889e-06 -tag 7.7549014708892479e-06 -▁Belastung 7.7549014708892479e-06 -▁Feststellung 7.7549014708892479e-06 -▁Gil 7.7549014708892479e-06 -▁council 7.7549014708892479e-06 -▁judgment 7.7549014708892479e-06 -▁tonnes 7.7549014708892479e-06 -▁unanimously 7.7549014708892479e-06 -▁vielseit 7.7549014708892479e-06 -▁Konvention 7.7511130480895942e-06 -▁churches 7.7511130480895942e-06 -▁coordinate 7.7511130480895942e-06 -▁etabl 7.7511130480895942e-06 -fahrts 7.7473246252899421e-06 -▁converted 7.7473246252899421e-06 -erzeug 7.74353620249029e-06 -van 7.74353620249029e-06 -▁Citizens 7.74353620249029e-06 -▁Gallery 7.74353620249029e-06 -▁Präsentation 7.74353620249029e-06 -▁intelligence 7.74353620249029e-06 -emperatur 7.7397477796906363e-06 -▁Beratungen 7.7397477796906363e-06 -▁Exp 7.7397477796906363e-06 -▁Hügel 7.7397477796906363e-06 -▁bedienen 7.7397477796906363e-06 -▁eas 7.7397477796906363e-06 -▁formation 7.7397477796906363e-06 -menge 7.7359593568909843e-06 -unity 7.7359593568909843e-06 -älte 7.7359593568909843e-06 -ć 7.7359593568909843e-06 -▁Pict 7.7359593568909843e-06 -▁liegenden 7.7359593568909843e-06 -▁possess 7.7359593568909843e-06 -▁violent 7.7359593568909843e-06 -▁worry 7.7359593568909843e-06 -wür 7.73217093409133e-06 -äglich 7.73217093409133e-06 -▁BM 7.73217093409133e-06 -▁Heiligen 7.73217093409133e-06 -▁Schutzes 7.73217093409133e-06 -▁credible 7.73217093409133e-06 -▁experiencing 7.73217093409133e-06 -▁girl 7.73217093409133e-06 -▁inneren 7.73217093409133e-06 -▁measuring 7.73217093409133e-06 -BC 7.7283825112916785e-06 -▁Kos 7.7283825112916785e-06 -▁Legisl 7.7283825112916785e-06 -▁directions 7.7283825112916785e-06 -▁gewinnt 7.7283825112916785e-06 -▁ru 7.7283825112916785e-06 -GP 7.7245940884920248e-06 -ialog 7.7245940884920248e-06 -▁Gepäck 7.7245940884920248e-06 -▁Lithuania 7.7245940884920248e-06 -▁Marx 7.7245940884920248e-06 -▁einführen 7.7245940884920248e-06 -▁nehme 7.7245940884920248e-06 -▁qualifications 7.7245940884920248e-06 -▁ändert 7.7245940884920248e-06 -itzung 7.7208056656923727e-06 -▁Datens 7.7208056656923727e-06 -▁Intervention 7.7208056656923727e-06 -▁creat 7.7208056656923727e-06 -▁reibungs 7.7208056656923727e-06 -fällt 7.71701724289272e-06 -konzept 7.71701724289272e-06 -quer 7.71701724289272e-06 -üre 7.71701724289272e-06 -▁Bin 7.71701724289272e-06 -▁Steuerzahler 7.71701724289272e-06 -▁Versorgungs 7.71701724289272e-06 -▁funkt 7.71701724289272e-06 -bindung 7.7132288200930669e-06 -pend 7.7132288200930669e-06 -ritts 7.7132288200930669e-06 -wechs 7.7132288200930669e-06 -▁Lat 7.7132288200930669e-06 -▁Slovenia 7.7132288200930669e-06 -▁Song 7.7132288200930669e-06 -▁Verts 7.7132288200930669e-06 -▁slowly 7.7132288200930669e-06 -▁verstärkte 7.7132288200930669e-06 -▁Geburt 7.7094403972934149e-06 -▁Gewinner 7.7094403972934149e-06 -▁financed 7.7094403972934149e-06 -▁grün 7.7094403972934149e-06 -▁präz 7.7094403972934149e-06 -▁Ausübung 7.7056519744937612e-06 -▁Energiequellen 7.7056519744937612e-06 -▁Ergänzung 7.7056519744937612e-06 -▁Od 7.7056519744937612e-06 -▁Visum 7.7056519744937612e-06 -▁eigentlichen 7.7056519744937612e-06 -▁potentially 7.7056519744937612e-06 -▁sq 7.7056519744937612e-06 -HS 7.7018635516941091e-06 -▁elderly 7.7018635516941091e-06 -▁flo 7.7018635516941091e-06 -▁ineff 7.7018635516941091e-06 -▁placing 7.7018635516941091e-06 -▁systematic 7.7018635516941091e-06 -.- 7.6980751288944554e-06 -zubringen 7.6980751288944554e-06 -▁Gebäuden 7.6980751288944554e-06 -▁Presidents 7.6980751288944554e-06 -▁caught 7.6980751288944554e-06 -▁competences 7.6980751288944554e-06 -▁vorbehalten 7.6980751288944554e-06 -künfte 7.6942867060948033e-06 -▁unc 7.6942867060948033e-06 -objekt 7.6904982832951513e-06 -urb 7.6904982832951513e-06 -▁Democratic 7.6904982832951513e-06 -▁gelingen 7.6904982832951513e-06 -▁victim 7.6904982832951513e-06 -versicherung 7.6867098604954976e-06 -▁3.0 7.6867098604954976e-06 -▁Acht 7.6867098604954976e-06 -▁Cathedral 7.6867098604954976e-06 -▁einnehmen 7.6867098604954976e-06 -▁haus 7.6867098604954976e-06 -▁zit 7.6867098604954976e-06 -igi 7.6829214376958455e-06 -steller 7.6829214376958455e-06 -▁3rd 7.6829214376958455e-06 -▁Technical 7.6829214376958455e-06 -▁answered 7.6829214376958455e-06 -▁column 7.6829214376958455e-06 -▁komfortabel 7.6829214376958455e-06 -▁organisierte 7.6829214376958455e-06 -▁premises 7.6829214376958455e-06 -▁tropical 7.6829214376958455e-06 -apel 7.6791330148961918e-06 -fenster 7.6791330148961918e-06 -kampf 7.6791330148961918e-06 -zie 7.6791330148961918e-06 -▁450 7.6791330148961918e-06 -▁Europar 7.6791330148961918e-06 -▁Küchen 7.6791330148961918e-06 -▁Rang 7.6791330148961918e-06 -▁situ 7.6791330148961918e-06 -very 7.67534459209654e-06 -waren 7.67534459209654e-06 -ю 7.67534459209654e-06 -▁Beschwerden 7.67534459209654e-06 -▁Choose 7.67534459209654e-06 -▁Right 7.67534459209654e-06 -▁approx 7.67534459209654e-06 -▁diet 7.67534459209654e-06 -▁sagten 7.67534459209654e-06 -▁spielte 7.67534459209654e-06 -erseits 7.671556169296886e-06 -okratischen 7.671556169296886e-06 -ulin 7.671556169296886e-06 -▁Installations 7.671556169296886e-06 -▁Isol 7.671556169296886e-06 -▁brain 7.671556169296886e-06 -▁visitor 7.671556169296886e-06 -oup 7.6677677464972339e-06 -▁Anna 7.6677677464972339e-06 -▁Salz 7.6677677464972339e-06 -▁male 7.6677677464972339e-06 -holz 7.66397932369758e-06 -merk 7.66397932369758e-06 -▁Cala 7.66397932369758e-06 -▁analy 7.66397932369758e-06 -▁ehrlich 7.66397932369758e-06 -▁flav 7.66397932369758e-06 -▁geleitet 7.66397932369758e-06 -▁kultur 7.66397932369758e-06 -▁schlechte 7.66397932369758e-06 -PP 7.6601909008979282e-06 -▁Bedien 7.6601909008979282e-06 -▁Flächen 7.6601909008979282e-06 -▁Rahmenbedingungen 7.6601909008979282e-06 -▁accommodate 7.6601909008979282e-06 -▁allocation 7.6601909008979282e-06 -▁jung 7.6601909008979282e-06 -iller 7.6564024780982761e-06 -zeich 7.6564024780982761e-06 -▁BE 7.6564024780982761e-06 -▁Heimatland 7.6564024780982761e-06 -▁humanity 7.6564024780982761e-06 -▁initiated 7.6564024780982761e-06 -▁spätestens 7.6564024780982761e-06 -▁Voyager 7.6526140552986224e-06 -▁timely 7.6526140552986224e-06 -vertreter 7.64882563249897e-06 -▁1984 7.64882563249897e-06 -▁Gewinne 7.64882563249897e-06 -▁bid 7.64882563249897e-06 -▁garage 7.64882563249897e-06 -▁Ungleich 7.6450372096993166e-06 -▁sle 7.6450372096993166e-06 -▁unterwegs 7.6450372096993166e-06 -▁verzeichnen 7.6450372096993166e-06 --the 7.6412487868996645e-06 -ealth 7.6412487868996645e-06 -ibilität 7.6412487868996645e-06 -vorschläge 7.6412487868996645e-06 -▁Andreas 7.6412487868996645e-06 -▁Creat 7.6412487868996645e-06 -▁Hir 7.6412487868996645e-06 -▁creativity 7.6412487868996645e-06 -▁integral 7.6412487868996645e-06 -▁keen 7.6412487868996645e-06 -▁überlegen 7.6412487868996645e-06 -attet 7.63746036410001e-06 -▁Einschränkungen 7.63746036410001e-06 -▁Verletzungen 7.63746036410001e-06 -▁gather 7.63746036410001e-06 -▁gelegenen 7.63746036410001e-06 -▁gewalt 7.63746036410001e-06 -▁keeps 7.63746036410001e-06 -▁silence 7.63746036410001e-06 -▁trop 7.63746036410001e-06 -älter 7.6336719413003588e-06 -▁Blumen 7.6336719413003588e-06 -▁Letzt 7.6336719413003588e-06 -▁Phänomen 7.6336719413003588e-06 -▁Tsch 7.6336719413003588e-06 -▁pm 7.6336719413003588e-06 -atus 7.6298835185007067e-06 -gramm 7.6298835185007067e-06 -▁copies 7.6298835185007067e-06 -▁teilzunehmen 7.6298835185007067e-06 -uchte 7.626095095701053e-06 -▁Cent 7.626095095701053e-06 -▁Laur 7.626095095701053e-06 -▁Paz 7.626095095701053e-06 -eag 7.6223066729014e-06 -hold 7.6223066729014e-06 -▁Ibiza 7.6223066729014e-06 -▁Israelis 7.6223066729014e-06 -▁degrees 7.6223066729014e-06 -▁jährlichen 7.6223066729014e-06 -▁polnischen 7.6223066729014e-06 -▁unb 7.6223066729014e-06 -hab 7.6185182501017472e-06 -igned 7.6185182501017472e-06 -wachsen 7.6185182501017472e-06 -▁imposing 7.6185182501017472e-06 -▁metr 7.6185182501017472e-06 -▁nam 7.6185182501017472e-06 -▁veran 7.6185182501017472e-06 -▁vorrang 7.6185182501017472e-06 -print 7.6147298273020952e-06 -▁Kam 7.6147298273020952e-06 -▁charter 7.6147298273020952e-06 -▁kohär 7.6147298273020952e-06 -mission 7.6109414045024423e-06 -sprogramm 7.6109414045024423e-06 -urger 7.6109414045024423e-06 -▁Ens 7.6109414045024423e-06 -▁Integr 7.6109414045024423e-06 -▁Lar 7.6109414045024423e-06 -▁Tabelle 7.6109414045024423e-06 -▁erschien 7.6109414045024423e-06 -▁irgend 7.6109414045024423e-06 -▁ständige 7.6109414045024423e-06 -assung 7.6071529817027894e-06 -inding 7.6071529817027894e-06 -▁FC 7.6071529817027894e-06 -▁designer 7.6071529817027894e-06 -▁heated 7.6071529817027894e-06 -▁regeln 7.6071529817027894e-06 -▁virus 7.6071529817027894e-06 -stern 7.6033645589031365e-06 -▁eingeräumt 7.6033645589031365e-06 -zugehen 7.5995761361034836e-06 -▁Vort 7.5995761361034836e-06 -▁lighting 7.5995761361034836e-06 -ORT 7.5957877133038307e-06 -det 7.5957877133038307e-06 -▁Alpha 7.5957877133038307e-06 -▁Geschäften 7.5957877133038307e-06 -▁Unterstütz 7.5957877133038307e-06 -▁intense 7.5957877133038307e-06 -▁procurement 7.5957877133038307e-06 -schlüsse 7.5919992905041778e-06 -▁Om 7.5919992905041778e-06 -▁Platten 7.5919992905041778e-06 -▁respectively 7.5919992905041778e-06 -▁sonstigen 7.5919992905041778e-06 -bü 7.5882108677045249e-06 -odox 7.5882108677045249e-06 -volles 7.5882108677045249e-06 -▁Arbeitslosen 7.5882108677045249e-06 -▁medizinischen 7.5882108677045249e-06 -ns 7.5844224449048729e-06 -▁Gegenwart 7.5844224449048729e-06 -▁Route 7.5844224449048729e-06 -▁headquarters 7.5844224449048729e-06 -▁insofern 7.5844224449048729e-06 -ords 7.58063402210522e-06 -▁1.0 7.58063402210522e-06 -▁Oliven 7.58063402210522e-06 -▁dating 7.58063402210522e-06 -▁municipal 7.58063402210522e-06 -▁partial 7.58063402210522e-06 -▁state-of-the-art 7.58063402210522e-06 -▁ähnlichen 7.58063402210522e-06 -NA 7.5768455993055671e-06 -▁Secretary 7.5768455993055671e-06 -▁assurance 7.5768455993055671e-06 -▁eigenem 7.5768455993055671e-06 -▁Unterhaltung 7.5730571765059142e-06 -▁biodiversity 7.5730571765059142e-06 -▁gesundheit 7.5730571765059142e-06 -▁industriellen 7.5730571765059142e-06 -▁stattdessen 7.5730571765059142e-06 -eres 7.5692687537062613e-06 -▁EU-B 7.5692687537062613e-06 -▁Santiago 7.5692687537062613e-06 -▁gent 7.5692687537062613e-06 -▁inakzeptabel 7.5692687537062613e-06 -▁objekt 7.5692687537062613e-06 -▁wake 7.5692687537062613e-06 -▁warmly 7.5692687537062613e-06 -baum 7.5654803309066084e-06 -▁Gärten 7.5654803309066084e-06 -▁demokratischer 7.5654803309066084e-06 -▁derjenigen 7.5654803309066084e-06 -▁workshops 7.5654803309066084e-06 -▁82 7.5616919081069555e-06 -▁externe 7.5616919081069555e-06 -▁vorgehen 7.5616919081069555e-06 -brechen 7.5579034853073026e-06 -ieten 7.5579034853073026e-06 -▁Karriere 7.5579034853073026e-06 -▁Leuten 7.5579034853073026e-06 -▁entries 7.5579034853073026e-06 -▁exposure 7.5579034853073026e-06 -▁stance 7.5579034853073026e-06 -31 7.5541150625076506e-06 -▁Geben 7.5541150625076506e-06 -▁Inkrafttreten 7.5541150625076506e-06 -▁Six 7.5541150625076506e-06 -▁passende 7.5541150625076506e-06 -führungen 7.5503266397079977e-06 -zieh 7.5503266397079977e-06 -▁Adam 7.5503266397079977e-06 -▁Junior 7.5503266397079977e-06 -▁Rassismus 7.5503266397079977e-06 -▁Spannungen 7.5503266397079977e-06 -▁arbitrary 7.5503266397079977e-06 -▁fragment 7.5503266397079977e-06 -ausstattung 7.5465382169083448e-06 -▁fitted 7.5465382169083448e-06 -▁sentence 7.5465382169083448e-06 -▁stays 7.5465382169083448e-06 -▁verwirklichen 7.5465382169083448e-06 -iate 7.5427497941086919e-06 -ifiziert 7.5427497941086919e-06 -opole 7.5427497941086919e-06 -äufer 7.5427497941086919e-06 -▁isn 7.5427497941086919e-06 -▁laser 7.5427497941086919e-06 -▁pil 7.5427497941086919e-06 -▁unstable 7.5427497941086919e-06 -arz 7.538961371309039e-06 -ündete 7.538961371309039e-06 -▁beeinflusst 7.538961371309039e-06 -▁beruflichen 7.538961371309039e-06 -▁extr 7.538961371309039e-06 -▁hyp 7.538961371309039e-06 -▁Afr 7.5351729485093861e-06 -▁Anre 7.5351729485093861e-06 -▁Apache 7.5351729485093861e-06 -▁Rh 7.5351729485093861e-06 -▁contra 7.5351729485093861e-06 -▁endeavour 7.5351729485093861e-06 -▁mas 7.5351729485093861e-06 -▁nat 7.5351729485093861e-06 -▁oberen 7.5351729485093861e-06 -▁zones 7.5351729485093861e-06 -ctional 7.5313845257097332e-06 -ctions 7.5313845257097332e-06 -▁af 7.5313845257097332e-06 -▁prob 7.5313845257097332e-06 -▁terminal 7.5313845257097332e-06 -▁zeitgenöss 7.5313845257097332e-06 --Or 7.52759610291008e-06 -outs 7.52759610291008e-06 -ror 7.52759610291008e-06 -▁Einschätzung 7.52759610291008e-06 -▁Low 7.52759610291008e-06 -▁Yugoslavia 7.52759610291008e-06 -▁disputes 7.52759610291008e-06 -▁gefährden 7.52759610291008e-06 -▁posted 7.52759610291008e-06 -musik 7.5200192573107754e-06 -▁Abschluß 7.5200192573107754e-06 -▁chem 7.5200192573107754e-06 -▁lokaler 7.5200192573107754e-06 -berecht 7.5162308345111225e-06 -iciency 7.5162308345111225e-06 -▁Hitler 7.5162308345111225e-06 -▁UV 7.5162308345111225e-06 -▁agrees 7.5162308345111225e-06 -▁geistigen 7.5162308345111225e-06 -▁größt 7.5162308345111225e-06 -▁pursued 7.5162308345111225e-06 -▁rear 7.5162308345111225e-06 -▁talent 7.5162308345111225e-06 -abilität 7.51244241171147e-06 -quar 7.51244241171147e-06 -üll 7.51244241171147e-06 -▁Anbindung 7.51244241171147e-06 -▁Neues 7.51244241171147e-06 -▁cat 7.51244241171147e-06 -▁findings 7.51244241171147e-06 -▁handled 7.51244241171147e-06 -,4 7.5086539889118167e-06 -ilos 7.5086539889118167e-06 -▁Erbe 7.5086539889118167e-06 -▁appreciated 7.5086539889118167e-06 -▁respektiert 7.5086539889118167e-06 -▁shot 7.5086539889118167e-06 -dienstleistungen 7.5048655661121638e-06 -▁Swimmingpool 7.5048655661121638e-06 -▁equip 7.5048655661121638e-06 -▁merc 7.5048655661121638e-06 -liegen 7.501077143312511e-06 -ütz 7.501077143312511e-06 -▁Hub 7.501077143312511e-06 -▁beschließen 7.501077143312511e-06 -▁reinfor 7.501077143312511e-06 -oge 7.4972887205128589e-06 -ücke 7.4972887205128589e-06 -▁Shanghai 7.4972887205128589e-06 -▁beliebigen 7.4972887205128589e-06 -▁terrorists 7.4972887205128589e-06 -fordern 7.493500297713206e-06 -▁Rußland 7.493500297713206e-06 -▁chamber 7.493500297713206e-06 -▁tar 7.493500297713206e-06 -▁Batter 7.4897118749135531e-06 -▁Nag 7.4897118749135531e-06 -▁Prakt 7.4897118749135531e-06 -▁erfasst 7.4897118749135531e-06 -▁rea 7.4897118749135531e-06 -apped 7.4859234521139e-06 -here 7.4859234521139e-06 -landschaft 7.4859234521139e-06 -räumen 7.4859234521139e-06 -▁21st 7.4859234521139e-06 -▁Develop 7.4859234521139e-06 -▁Rout 7.4859234521139e-06 -▁Stich 7.4859234521139e-06 -▁Trainings 7.4859234521139e-06 -▁angeschlossen 7.4859234521139e-06 -▁ausgesch 7.4859234521139e-06 -▁bem 7.4859234521139e-06 -▁essence 7.4859234521139e-06 -▁pursuant 7.4859234521139e-06 -▁well-being 7.4859234521139e-06 -aki 7.4821350293142473e-06 -betrag 7.4821350293142473e-06 -star 7.4821350293142473e-06 -▁contractual 7.4821350293142473e-06 -▁perspectives 7.4821350293142473e-06 -▁surv 7.4821350293142473e-06 -▁Grundsätzen 7.4783466065145944e-06 -▁Info 7.4783466065145944e-06 -▁lowest 7.4783466065145944e-06 -▁maps 7.4783466065145944e-06 -▁unemployed 7.4783466065145944e-06 -bereitung 7.4745581837149416e-06 -rock 7.4745581837149416e-06 -äche 7.4745581837149416e-06 -▁Force 7.4745581837149416e-06 -▁exempt 7.4745581837149416e-06 -▁tips 7.4745581837149416e-06 -▁Konstruktion 7.4707697609152887e-06 -▁Mehrwertsteuer 7.4707697609152887e-06 -▁Palacio 7.4707697609152887e-06 -▁Prod 7.4707697609152887e-06 -▁artificial 7.4707697609152887e-06 -▁erneuerbaren 7.4707697609152887e-06 -▁prä 7.4707697609152887e-06 -▁suggestion 7.4707697609152887e-06 -▁trail 7.4707697609152887e-06 -▁ärmsten 7.4707697609152887e-06 -▁öffnet 7.4707697609152887e-06 -▁Jacques 7.4669813381156366e-06 -▁Turm 7.4669813381156366e-06 -▁proc 7.4669813381156366e-06 -▁specialities 7.4669813381156366e-06 -onien 7.4631929153159837e-06 -otyp 7.4631929153159837e-06 -ürzen 7.4631929153159837e-06 -▁Herrschaft 7.4631929153159837e-06 -▁Hum 7.4631929153159837e-06 -▁Konzentration 7.4631929153159837e-06 -▁Lib 7.4631929153159837e-06 -▁bestens 7.4631929153159837e-06 -▁lieben 7.4631929153159837e-06 -▁paradise 7.4631929153159837e-06 -▁Drag 7.4594044925163308e-06 -DEN 7.4556160697166779e-06 -gibt 7.4556160697166779e-06 -schaffen 7.4556160697166779e-06 -▁Fünf 7.4556160697166779e-06 -▁Produzenten 7.4556160697166779e-06 -▁Rücken 7.4556160697166779e-06 -▁crash 7.4556160697166779e-06 -generation 7.4518276469170251e-06 -onto 7.4518276469170251e-06 -steht 7.4518276469170251e-06 -▁DER 7.4518276469170251e-06 -▁Freuen 7.4518276469170251e-06 -▁Rubrik 7.4518276469170251e-06 -▁Unterlagen 7.4518276469170251e-06 -▁craft 7.4518276469170251e-06 -▁facto 7.4518276469170251e-06 -ampen 7.4480392241173722e-06 -ecke 7.4480392241173722e-06 -▁AKVIS 7.4480392241173722e-06 -▁Div 7.4480392241173722e-06 -▁Einwander 7.4480392241173722e-06 -▁Exc 7.4480392241173722e-06 -▁Konsum 7.4480392241173722e-06 -▁ernsthafte 7.4480392241173722e-06 -▁stroll 7.4480392241173722e-06 -▁Elevator 7.4442508013177193e-06 -▁Kod 7.4442508013177193e-06 -▁papers 7.4442508013177193e-06 -▁scr 7.4442508013177193e-06 -▁verpflichten 7.4442508013177193e-06 -▁Antonio 7.4404623785180664e-06 -▁Change 7.4404623785180664e-06 -▁Elektronik 7.4404623785180664e-06 -▁Forest 7.4404623785180664e-06 -▁convince 7.4404623785180664e-06 -▁folglich 7.4404623785180664e-06 -▁investiert 7.4404623785180664e-06 -▁kontinuierlich 7.4404623785180664e-06 -▁unl 7.4404623785180664e-06 -▁Amerikas 7.4366739557184143e-06 -▁Angeles 7.4366739557184143e-06 -▁grad 7.4366739557184143e-06 -6-0 7.4328855329187614e-06 -lor 7.4328855329187614e-06 -schreiben 7.4328855329187614e-06 -ucc 7.4328855329187614e-06 -ßt 7.4328855329187614e-06 -▁Classic 7.4328855329187614e-06 -▁alleine 7.4328855329187614e-06 -▁kontroll 7.4328855329187614e-06 -▁practically 7.4328855329187614e-06 -▁temperatures 7.4328855329187614e-06 --EU 7.4290971101191086e-06 -ehmer 7.4290971101191086e-06 -etts 7.4290971101191086e-06 -kung 7.4290971101191086e-06 -▁Besitzer 7.4290971101191086e-06 -▁coverage 7.4290971101191086e-06 -▁erweiterten 7.4290971101191086e-06 -▁platz 7.4290971101191086e-06 -ägern 7.4253086873194557e-06 -▁Bund 7.4253086873194557e-06 -▁Coast 7.4253086873194557e-06 -▁Juan 7.4253086873194557e-06 -▁Ord 7.4253086873194557e-06 -▁Vertretung 7.4253086873194557e-06 -▁workshop 7.4253086873194557e-06 -▁zer 7.4253086873194557e-06 -,8 7.4215202645198028e-06 -ablen 7.4215202645198028e-06 -ietet 7.4215202645198028e-06 -▁Consulting 7.4215202645198028e-06 -▁Gebir 7.4215202645198028e-06 -▁Inform 7.4215202645198028e-06 -▁ausgewählten 7.4215202645198028e-06 -▁kollekt 7.4215202645198028e-06 -ercial 7.41773184172015e-06 -sm 7.41773184172015e-06 -ungsmaßnahmen 7.41773184172015e-06 -▁Innerhalb 7.41773184172015e-06 -▁Samm 7.41773184172015e-06 -▁Anhang 7.413943418920497e-06 -▁Tuesday 7.413943418920497e-06 -▁blame 7.413943418920497e-06 -▁Ordner 7.4101549961208441e-06 -▁Relations 7.4101549961208441e-06 -▁amend 7.4101549961208441e-06 -▁automatische 7.4101549961208441e-06 -▁modular 7.4101549961208441e-06 -▁platforms 7.4101549961208441e-06 -quest 7.4063665733211921e-06 -▁bio 7.4063665733211921e-06 -▁decent 7.4063665733211921e-06 -▁mild 7.4063665733211921e-06 -▁russische 7.4063665733211921e-06 -▁specifications 7.4063665733211921e-06 -aves 7.4025781505215392e-06 -▁Track 7.4025781505215392e-06 -▁dollars 7.4025781505215392e-06 -▁reiz 7.4025781505215392e-06 -▁salt 7.4025781505215392e-06 -velop 7.3987897277218863e-06 -▁fortgesetzt 7.3987897277218863e-06 -▁jemals 7.3987897277218863e-06 -▁solved 7.3987897277218863e-06 -betriebe 7.3950013049222334e-06 -dom 7.3950013049222334e-06 -ranten 7.3950013049222334e-06 -▁Beschäftigten 7.3950013049222334e-06 -▁Studios 7.3950013049222334e-06 -▁desirable 7.3950013049222334e-06 -▁erbaut 7.3950013049222334e-06 -▁rust 7.3950013049222334e-06 -fordert 7.3912128821225805e-06 -▁Regionalpolitik 7.3912128821225805e-06 -▁effektiver 7.3912128821225805e-06 -▁schenken 7.3912128821225805e-06 -atics 7.3874244593229276e-06 -zif 7.3874244593229276e-06 -▁Discover 7.3874244593229276e-06 -▁Listen 7.3874244593229276e-06 -▁Verteidigungspolitik 7.3874244593229276e-06 -▁expanding 7.3874244593229276e-06 -▁kids 7.3874244593229276e-06 -▁viable 7.3874244593229276e-06 -actory 7.3836360365232747e-06 -akers 7.3836360365232747e-06 -anteil 7.3836360365232747e-06 -▁CM 7.3836360365232747e-06 -▁Obviously 7.3836360365232747e-06 -▁Proph 7.3836360365232747e-06 -▁kostengünst 7.3836360365232747e-06 -▁ober 7.3836360365232747e-06 -▁vollem 7.3836360365232747e-06 -entliche 7.3798476137236218e-06 -▁DR 7.3798476137236218e-06 -▁Hannover 7.3798476137236218e-06 -▁Malaysia 7.3798476137236218e-06 -▁breathtaking 7.3798476137236218e-06 -▁crops 7.3798476137236218e-06 -▁eindrucks 7.3798476137236218e-06 -▁kritischen 7.3798476137236218e-06 -▁occupied 7.3798476137236218e-06 -ferenz 7.37605919092397e-06 -hel 7.37605919092397e-06 -▁Nap 7.37605919092397e-06 -▁Progress 7.37605919092397e-06 -try 7.3722707681243169e-06 -▁Mineral 7.3722707681243169e-06 -▁Mitt 7.3722707681243169e-06 -▁Raums 7.3722707681243169e-06 -▁Sard 7.3722707681243169e-06 -▁demokratisch 7.3722707681243169e-06 -▁negativ 7.3722707681243169e-06 -▁remark 7.3722707681243169e-06 -▁м 7.3722707681243169e-06 -▁Kreativität 7.368482345324664e-06 -▁Solche 7.368482345324664e-06 -▁assured 7.368482345324664e-06 -▁bedauere 7.368482345324664e-06 -▁gewünscht 7.368482345324664e-06 -▁treaties 7.368482345324664e-06 -bahnen 7.3646939225250111e-06 -emeinschaft 7.3646939225250111e-06 -gründen 7.3646939225250111e-06 -icians 7.3646939225250111e-06 -oster 7.3646939225250111e-06 -same 7.3646939225250111e-06 -ри 7.3646939225250111e-06 -▁Abteilung 7.3646939225250111e-06 -▁Hen 7.3646939225250111e-06 -▁Lob 7.3646939225250111e-06 -▁Similarly 7.3646939225250111e-06 -▁erörtert 7.3646939225250111e-06 -eaus 7.3609054997253582e-06 -▁24-Stunden-Rezeption 7.3609054997253582e-06 -▁Unterzeichnung 7.3609054997253582e-06 -▁erad 7.3609054997253582e-06 -▁unterbreiten 7.3609054997253582e-06 -,6 7.3571170769257053e-06 -igs 7.3571170769257053e-06 -▁Investition 7.3571170769257053e-06 -▁Merkmale 7.3571170769257053e-06 -▁Release 7.3571170769257053e-06 -▁score 7.3571170769257053e-06 -▁zut 7.3571170769257053e-06 -icket 7.3533286541260524e-06 -▁Gn 7.3533286541260524e-06 -▁Iranian 7.3533286541260524e-06 -▁Schwimmbad 7.3533286541260524e-06 -▁angenehme 7.3533286541260524e-06 -▁describes 7.3533286541260524e-06 -▁disappointed 7.3533286541260524e-06 -element 7.3495402313263995e-06 -ento 7.3495402313263995e-06 -rechtlichen 7.3495402313263995e-06 -▁bewirken 7.3495402313263995e-06 -▁transformation 7.3495402313263995e-06 -▁weitaus 7.3495402313263995e-06 -38 7.3457518085267475e-06 -eria 7.3457518085267475e-06 -▁1978 7.3457518085267475e-06 -▁Au 7.3457518085267475e-06 -▁Direktor 7.3457518085267475e-06 -▁Has 7.3457518085267475e-06 -▁Luz 7.3457518085267475e-06 -▁Past 7.3457518085267475e-06 -▁beizutragen 7.3457518085267475e-06 -▁gültig 7.3457518085267475e-06 -▁plötzlich 7.3457518085267475e-06 -▁tight 7.3457518085267475e-06 -▁Fly 7.3419633857270946e-06 -▁bag 7.3419633857270946e-06 -▁bewiesen 7.3419633857270946e-06 -▁exam 7.3419633857270946e-06 -▁throw 7.3419633857270946e-06 -▁tor 7.3419633857270946e-06 -bür 7.3381749629274417e-06 -inks 7.3381749629274417e-06 -zo 7.3381749629274417e-06 -▁Syrien 7.3381749629274417e-06 -▁chambres 7.3381749629274417e-06 -▁lovers 7.3381749629274417e-06 -▁viewed 7.3381749629274417e-06 -▁2007-2013 7.3343865401277888e-06 -▁Venezuela 7.3343865401277888e-06 -▁choosing 7.3343865401277888e-06 -▁explains 7.3343865401277888e-06 -▁receives 7.3343865401277888e-06 --General 7.3305981173281359e-06 -agement 7.3305981173281359e-06 -yan 7.3305981173281359e-06 -▁detect 7.3305981173281359e-06 -analyse 7.326809694528483e-06 -eli 7.326809694528483e-06 -liness 7.326809694528483e-06 -▁Bit 7.326809694528483e-06 -▁G20 7.326809694528483e-06 -▁Lind 7.326809694528483e-06 -▁Panasonic 7.326809694528483e-06 -▁abschließend 7.326809694528483e-06 -▁decl 7.326809694528483e-06 -▁undermine 7.326809694528483e-06 -aro 7.32302127172883e-06 -ija 7.32302127172883e-06 -ilm 7.32302127172883e-06 -▁Cubase 7.32302127172883e-06 -▁Erschein 7.32302127172883e-06 -▁comparable 7.32302127172883e-06 -▁rect 7.32302127172883e-06 -fü 7.3192328489291781e-06 -éd 7.3192328489291781e-06 -▁Canc 7.3192328489291781e-06 -▁emerge 7.3192328489291781e-06 -▁sweet 7.3192328489291781e-06 -format 7.3154444261295252e-06 -schwe 7.3154444261295252e-06 -▁Cold 7.3154444261295252e-06 -▁Sarkozy 7.3154444261295252e-06 -▁graphic 7.3154444261295252e-06 -▁incon 7.3154444261295252e-06 -▁werfen 7.3154444261295252e-06 -▁Müller 7.3116560033298723e-06 -aturen 7.3078675805302194e-06 -gleichen 7.3078675805302194e-06 -ны 7.3078675805302194e-06 -▁1967 7.3078675805302194e-06 -▁Anhörung 7.3078675805302194e-06 -▁Ans 7.3078675805302194e-06 -▁Enter 7.3078675805302194e-06 -▁Gerichten 7.3078675805302194e-06 -▁Vergnügen 7.3078675805302194e-06 -▁performances 7.3078675805302194e-06 --Un 7.3040791577305665e-06 -.2009 7.3040791577305665e-06 -▁Bab 7.3040791577305665e-06 -▁neg 7.3040791577305665e-06 -oration 7.3002907349309136e-06 -pons 7.3002907349309136e-06 -usste 7.3002907349309136e-06 -▁Büros 7.3002907349309136e-06 -▁Touren 7.3002907349309136e-06 -▁Wellen 7.3002907349309136e-06 -▁biological 7.3002907349309136e-06 -▁geheim 7.3002907349309136e-06 -▁harmony 7.3002907349309136e-06 -▁thoroughly 7.3002907349309136e-06 -itzer 7.2965023121312607e-06 -nische 7.2965023121312607e-06 -rum 7.2965023121312607e-06 -ächte 7.2965023121312607e-06 -▁Kräften 7.2965023121312607e-06 -▁Pas 7.2965023121312607e-06 -▁Verhaltens 7.2965023121312607e-06 -▁desert 7.2965023121312607e-06 -▁ko 7.2965023121312607e-06 -▁verbreit 7.2965023121312607e-06 -▁Hyper 7.2927138893316078e-06 -▁frank 7.2927138893316078e-06 -OB 7.2889254665319558e-06 -istik 7.2889254665319558e-06 -▁Abu 7.2851370437323029e-06 -▁Finger 7.2851370437323029e-06 -▁Somm 7.2851370437323029e-06 -▁entworfen 7.2851370437323029e-06 -▁paradox 7.2851370437323029e-06 -▁teacher 7.2851370437323029e-06 -▁voters 7.2851370437323029e-06 -NE 7.28134862093265e-06 -ergebnisse 7.28134862093265e-06 -freundliche 7.28134862093265e-06 -okratische 7.28134862093265e-06 -ranken 7.28134862093265e-06 -riend 7.28134862093265e-06 -▁Communication 7.28134862093265e-06 -▁Medikamente 7.28134862093265e-06 -▁Wik 7.28134862093265e-06 -▁performing 7.28134862093265e-06 -ols 7.2775601981329971e-06 -steiger 7.2775601981329971e-06 -▁fairen 7.2775601981329971e-06 -▁fern 7.2775601981329971e-06 -▁scenery 7.2775601981329971e-06 -▁turb 7.2775601981329971e-06 --hand 7.2737717753333442e-06 -gewalt 7.2737717753333442e-06 -▁Vermittlung 7.2737717753333442e-06 -boot 7.2699833525336913e-06 -endem 7.2699833525336913e-06 -▁Buffet 7.2699833525336913e-06 -▁SCH 7.2699833525336913e-06 -▁enlarged 7.2699833525336913e-06 -▁gestellten 7.2699833525336913e-06 -▁strange 7.2699833525336913e-06 -▁Dim 7.2661949297340385e-06 -▁Hektar 7.2661949297340385e-06 -▁angel 7.2661949297340385e-06 -▁commend 7.2661949297340385e-06 -▁zweier 7.2661949297340385e-06 -▁ök 7.2661949297340385e-06 -oret 7.2624065069343856e-06 -uktion 7.2624065069343856e-06 -▁Arbeitsgruppe 7.2624065069343856e-06 -▁Hochzeit 7.2624065069343856e-06 -▁honor 7.2624065069343856e-06 -ELL 7.2586180841347335e-06 -▁Mik 7.2586180841347335e-06 -▁develops 7.2586180841347335e-06 -▁niedriger 7.2586180841347335e-06 -▁talked 7.2586180841347335e-06 -initiative 7.2548296613350806e-06 -ution 7.2548296613350806e-06 -▁Funktionalität 7.2548296613350806e-06 -▁Mauer 7.2548296613350806e-06 -▁begründet 7.2548296613350806e-06 -▁meantime 7.2548296613350806e-06 -▁Überleben 7.2548296613350806e-06 -ECT 7.2510412385354277e-06 -going 7.2510412385354277e-06 -▁Kühl 7.2510412385354277e-06 -▁Organization 7.2510412385354277e-06 -▁Unfall 7.2510412385354277e-06 -▁god 7.2510412385354277e-06 -▁preliminary 7.2510412385354277e-06 -abkommens 7.2472528157357748e-06 -zert 7.2472528157357748e-06 -zieht 7.2472528157357748e-06 -▁Ausschüsse 7.2472528157357748e-06 -▁Berater 7.2472528157357748e-06 -▁cheese 7.2472528157357748e-06 -▁defining 7.2472528157357748e-06 -▁einander 7.2472528157357748e-06 -▁vertr 7.2472528157357748e-06 -▁voneinander 7.2472528157357748e-06 -▁übermittelt 7.2472528157357748e-06 --Produkt 7.2434643929361219e-06 -fass 7.2434643929361219e-06 -iegt 7.2434643929361219e-06 -unken 7.2434643929361219e-06 -▁Fire 7.2434643929361219e-06 -▁deliber 7.2434643929361219e-06 -▁mineral 7.2434643929361219e-06 -▁substantially 7.2434643929361219e-06 -▁swift 7.2434643929361219e-06 -o. 7.2396759701364691e-06 -▁Galerie 7.2396759701364691e-06 -▁Terr 7.2396759701364691e-06 -▁avoided 7.2396759701364691e-06 -▁beschreiben 7.2396759701364691e-06 -▁superv 7.2396759701364691e-06 -itory 7.2358875473368162e-06 -kont 7.2358875473368162e-06 -▁Gulf 7.2358875473368162e-06 -▁Mount 7.2358875473368162e-06 -▁französ 7.2358875473368162e-06 -▁heft 7.2358875473368162e-06 -▁inbegriffen 7.2358875473368162e-06 -▁sees 7.2358875473368162e-06 -▁по 7.2358875473368162e-06 --Politik 7.2320991245371633e-06 -ACK 7.2320991245371633e-06 -opolitan 7.2320991245371633e-06 -▁Additionally 7.2320991245371633e-06 -▁Eigentümer 7.2320991245371633e-06 -▁Konferenzen 7.2320991245371633e-06 -▁erworben 7.2320991245371633e-06 -bu 7.2283107017375112e-06 -häusern 7.2283107017375112e-06 -ifizierungs 7.2283107017375112e-06 -iques 7.2283107017375112e-06 -reichung 7.2283107017375112e-06 -▁Ci 7.2283107017375112e-06 -▁Excel 7.2283107017375112e-06 -▁Gemäß 7.2283107017375112e-06 -▁agent 7.2283107017375112e-06 -▁partnerships 7.2283107017375112e-06 -▁schlechter 7.2283107017375112e-06 -▁sensor 7.2283107017375112e-06 -▁settled 7.2283107017375112e-06 -▁Äußerungen 7.2283107017375112e-06 -▁Active 7.2245222789378583e-06 -▁Palm 7.2245222789378583e-06 -▁Patch 7.2245222789378583e-06 -▁fährt 7.2245222789378583e-06 -▁incredible 7.2245222789378583e-06 -▁screens 7.2245222789378583e-06 -▁transp 7.2245222789378583e-06 -ateur 7.2207338561382054e-06 -ieur 7.2207338561382054e-06 -▁Mikro 7.2207338561382054e-06 -▁Ni 7.2207338561382054e-06 -▁Vorfeld 7.2207338561382054e-06 -▁cease 7.2207338561382054e-06 -▁incorrect 7.2207338561382054e-06 -▁spezial 7.2207338561382054e-06 -▁verwalten 7.2207338561382054e-06 -▁Amb 7.2169454333385526e-06 -▁Verluste 7.2169454333385526e-06 -▁emphasised 7.2169454333385526e-06 -▁photography 7.2169454333385526e-06 -icas 7.2131570105389e-06 -pect 7.2131570105389e-06 -▁Review 7.2131570105389e-06 -▁engag 7.2131570105389e-06 -▁normale 7.2131570105389e-06 -▁unbest 7.2131570105389e-06 --19 7.2093685877392468e-06 -wissen 7.2093685877392468e-06 -▁Alc 7.2093685877392468e-06 -▁Communist 7.2093685877392468e-06 -▁Hôtel 7.2093685877392468e-06 -▁Sales 7.2093685877392468e-06 -▁bearbeitet 7.2093685877392468e-06 -▁entschied 7.2093685877392468e-06 -▁gebildet 7.2093685877392468e-06 -▁kompliziert 7.2093685877392468e-06 -▁melden 7.2093685877392468e-06 -▁potent 7.2093685877392468e-06 -lebens 7.2055801649395939e-06 -oberfläche 7.2055801649395939e-06 -ogn 7.2055801649395939e-06 -rooms 7.2055801649395939e-06 -▁Motto 7.2055801649395939e-06 -▁Stoff 7.2055801649395939e-06 --Format 7.201791742139941e-06 -▁Bewältigung 7.201791742139941e-06 -▁Kunststoff 7.201791742139941e-06 -▁funded 7.201791742139941e-06 -▁gehandelt 7.201791742139941e-06 -▁gelernt 7.201791742139941e-06 -▁getötet 7.201791742139941e-06 -▁passen 7.201791742139941e-06 -elev 7.1980033193402889e-06 -eute 7.1980033193402889e-06 -förderung 7.1980033193402889e-06 -läuf 7.1980033193402889e-06 -▁centrally 7.1980033193402889e-06 -▁finest 7.1980033193402889e-06 -▁ohnehin 7.1980033193402889e-06 -▁piano 7.1980033193402889e-06 -▁signature 7.1980033193402889e-06 -▁span 7.1980033193402889e-06 -▁Zimbabwe 7.1942148965406361e-06 -▁aufgerufen 7.1942148965406361e-06 -▁israelischen 7.1942148965406361e-06 -▁joy 7.1942148965406361e-06 -aill 7.1904264737409832e-06 -imes 7.1904264737409832e-06 -▁76 7.1904264737409832e-06 -▁Roh 7.1904264737409832e-06 -▁correspond 7.1904264737409832e-06 -▁inclusive 7.1904264737409832e-06 -▁konstruktive 7.1904264737409832e-06 -▁sam 7.1904264737409832e-06 -▁winds 7.1904264737409832e-06 -aba 7.18663805094133e-06 -▁Besteuerung 7.18663805094133e-06 -▁guitar 7.18663805094133e-06 -▁neglect 7.18663805094133e-06 -r. 7.1828496281416774e-06 -verfahrens 7.1828496281416774e-06 -▁Broad 7.1828496281416774e-06 -▁Lebanon 7.1828496281416774e-06 -▁Rezession 7.1828496281416774e-06 -▁aktualisieren 7.1828496281416774e-06 -▁erwähnte 7.1828496281416774e-06 -▁geschafft 7.1828496281416774e-06 -▁schlicht 7.1828496281416774e-06 -oker 7.1790612053420245e-06 -▁Fat 7.1790612053420245e-06 -▁Nature 7.1790612053420245e-06 -▁Orts 7.1790612053420245e-06 -▁unbe 7.1790612053420245e-06 -kapazität 7.1752727825423716e-06 -viert 7.1752727825423716e-06 -▁Aufruf 7.1752727825423716e-06 -▁Parliamentary 7.1752727825423716e-06 -▁RI 7.1752727825423716e-06 -▁Spy 7.1752727825423716e-06 -▁endgültig 7.1752727825423716e-06 -imensional 7.1714843597427187e-06 -▁Schadens 7.1714843597427187e-06 -▁rivers 7.1714843597427187e-06 -▁zeigte 7.1714843597427187e-06 -alu 7.1676959369430667e-06 -indlich 7.1676959369430667e-06 -kämp 7.1676959369430667e-06 -möglich 7.1676959369430667e-06 -▁Internetzugang 7.1676959369430667e-06 -▁Short 7.1676959369430667e-06 -▁Weißbuch 7.1676959369430667e-06 -▁nachdenken 7.1676959369430667e-06 -▁Gol 7.1639075141434138e-06 -▁angewiesen 7.1639075141434138e-06 -▁birds 7.1639075141434138e-06 -▁consequently 7.1639075141434138e-06 -aper 7.1601190913437609e-06 -▁Iraqi 7.1601190913437609e-06 -▁Sitzungen 7.1601190913437609e-06 -▁gebraucht 7.1601190913437609e-06 -asste 7.156330668544108e-06 -▁Dienstleistung 7.156330668544108e-06 -▁Dritter 7.156330668544108e-06 -▁Eigenschaft 7.156330668544108e-06 -fahrzeuge 7.1525422457444551e-06 -vens 7.1525422457444551e-06 -▁Gaz 7.1525422457444551e-06 -▁Höhen 7.1525422457444551e-06 -▁Kanal 7.1525422457444551e-06 -▁interinstitutional 7.1525422457444551e-06 -▁markt 7.1525422457444551e-06 -berichte 7.1487538229448022e-06 -dokument 7.1487538229448022e-06 -▁Atlantik 7.1487538229448022e-06 -▁Everyone 7.1487538229448022e-06 -▁IA 7.1487538229448022e-06 -▁Johann 7.1487538229448022e-06 -▁erläutern 7.1487538229448022e-06 -▁tribute 7.1487538229448022e-06 -▁warrant 7.1487538229448022e-06 -▁worthy 7.1487538229448022e-06 -anes 7.1449654001451493e-06 -ersatz 7.1449654001451493e-06 -greifende 7.1449654001451493e-06 -kov 7.1449654001451493e-06 -sichtlich 7.1449654001451493e-06 -▁Along 7.1449654001451493e-06 -▁Socialists 7.1449654001451493e-06 -▁automation 7.1449654001451493e-06 -▁enth 7.1449654001451493e-06 -▁sail 7.1449654001451493e-06 -UC 7.1411769773454973e-06 -oping 7.1411769773454973e-06 -▁Eric 7.1411769773454973e-06 -▁Sher 7.1411769773454973e-06 -▁battery 7.1411769773454973e-06 -▁kre 7.1411769773454973e-06 -▁angeblich 7.1373885545458444e-06 -▁celebrate 7.1373885545458444e-06 -▁loved 7.1373885545458444e-06 -▁pin 7.1373885545458444e-06 -▁ven 7.1373885545458444e-06 -,2 7.1336001317461915e-06 -acken 7.1336001317461915e-06 -hot 7.1336001317461915e-06 -laut 7.1336001317461915e-06 -▁Ara 7.1336001317461915e-06 -▁COD 7.1336001317461915e-06 -▁arbeits 7.1336001317461915e-06 -▁exchanges 7.1336001317461915e-06 -▁measured 7.1336001317461915e-06 -▁nonetheless 7.1336001317461915e-06 -ahrung 7.1298117089465386e-06 -kauf 7.1298117089465386e-06 -mp 7.1298117089465386e-06 -uits 7.1298117089465386e-06 -verw 7.1298117089465386e-06 -▁Außer 7.1298117089465386e-06 -▁disturb 7.1298117089465386e-06 -▁einzugehen 7.1298117089465386e-06 -▁köst 7.1298117089465386e-06 -ordnungs 7.1260232861468857e-06 -wich 7.1260232861468857e-06 -▁bilateralen 7.1260232861468857e-06 -▁controversial 7.1260232861468857e-06 -▁expressing 7.1260232861468857e-06 -aktiv 7.1222348633472328e-06 -▁Bezahlung 7.1222348633472328e-06 -▁Carr 7.1222348633472328e-06 -▁erörtern 7.1222348633472328e-06 -▁unterbreitet 7.1222348633472328e-06 -▁ältere 7.1222348633472328e-06 -ception 7.11844644054758e-06 -opt 7.11844644054758e-06 -personal 7.11844644054758e-06 -vision 7.11844644054758e-06 -▁Attraktionen 7.11844644054758e-06 -▁acquire 7.11844644054758e-06 -▁crystal 7.11844644054758e-06 -▁finanziell 7.11844644054758e-06 -▁paintings 7.11844644054758e-06 -▁rechtfertigen 7.11844644054758e-06 -▁squ 7.11844644054758e-06 -▁sym 7.11844644054758e-06 -asis 7.114658017747927e-06 -▁Por 7.114658017747927e-06 -▁Studium 7.114658017747927e-06 -▁capture 7.114658017747927e-06 -▁konkur 7.114658017747927e-06 -▁schrittweise 7.114658017747927e-06 -amo 7.110869594948275e-06 -▁Bedienung 7.110869594948275e-06 -.6 7.1070811721486221e-06 -▁Engl 7.1070811721486221e-06 -▁agr 7.1070811721486221e-06 -▁controlling 7.1070811721486221e-06 -▁forese 7.1070811721486221e-06 -▁peak 7.1070811721486221e-06 -▁raises 7.1070811721486221e-06 -▁sporting 7.1070811721486221e-06 --Cl 7.1032927493489692e-06 -sitz 7.1032927493489692e-06 -uan 7.1032927493489692e-06 -▁3000 7.1032927493489692e-06 -▁Gift 7.1032927493489692e-06 -▁Marine 7.1032927493489692e-06 -▁WM 7.1032927493489692e-06 -▁brut 7.1032927493489692e-06 -▁discrimin 7.1032927493489692e-06 -eign 7.0995043265493163e-06 -etze 7.0995043265493163e-06 -ocratic 7.0995043265493163e-06 -uner 7.0995043265493163e-06 -▁bott 7.0995043265493163e-06 -▁declarations 7.0995043265493163e-06 -▁konzipiert 7.0995043265493163e-06 -▁verkauf 7.0995043265493163e-06 -abet 7.0957159037496634e-06 -einkommen 7.0957159037496634e-06 -inu 7.0957159037496634e-06 -venth 7.0957159037496634e-06 -▁Hoff 7.0957159037496634e-06 -▁Quart 7.0957159037496634e-06 -▁confusion 7.0957159037496634e-06 -▁eingebunden 7.0957159037496634e-06 -▁erfolgte 7.0957159037496634e-06 -▁gelegene 7.0957159037496634e-06 -▁zweck 7.0957159037496634e-06 -alteten 7.0919274809500105e-06 -gangenen 7.0919274809500105e-06 -▁admitted 7.0919274809500105e-06 -▁detaillierte 7.0919274809500105e-06 -▁geist 7.0919274809500105e-06 -▁stützen 7.0919274809500105e-06 -eab 7.0881390581503576e-06 -leistungs 7.0881390581503576e-06 -▁Aff 7.0881390581503576e-06 -▁Energiepolitik 7.0881390581503576e-06 -▁Heating 7.0881390581503576e-06 -▁Konzepte 7.0881390581503576e-06 -▁Lanzarote 7.0881390581503576e-06 -▁emission 7.0881390581503576e-06 -▁realen 7.0881390581503576e-06 -▁statistical 7.0881390581503576e-06 --Staaten 7.0805622125510527e-06 -▁Editor 7.0805622125510527e-06 -▁Gent 7.0805622125510527e-06 -▁Kit 7.0805622125510527e-06 -▁Map 7.0805622125510527e-06 -▁Sau 7.0805622125510527e-06 -▁Wille 7.0805622125510527e-06 -▁dust 7.0805622125510527e-06 -▁isolated 7.0805622125510527e-06 -▁tur 7.0805622125510527e-06 -elen 7.0767737897514e-06 -▁Division 7.0767737897514e-06 -▁Eth 7.0767737897514e-06 -▁Scheitern 7.0767737897514e-06 -▁beiträgt 7.0767737897514e-06 -▁kenn 7.0767737897514e-06 -▁licence 7.0767737897514e-06 -▁parameter 7.0767737897514e-06 -▁Ähn 7.0767737897514e-06 -PR 7.0729853669517469e-06 -utr 7.0729853669517469e-06 -▁Beratungs 7.0729853669517469e-06 -▁Führ 7.0729853669517469e-06 -▁Gelände 7.0729853669517469e-06 -▁Intr 7.0729853669517469e-06 -▁Vorgaben 7.0729853669517469e-06 -▁conveniently 7.0729853669517469e-06 -▁nötigen 7.0729853669517469e-06 -▁tempt 7.0729853669517469e-06 -TEN 7.069196944152094e-06 -ifi 7.069196944152094e-06 -▁About 7.069196944152094e-06 -▁Betr 7.069196944152094e-06 -▁Fluggesellschaften 7.069196944152094e-06 -▁Subsidiarität 7.069196944152094e-06 -▁acknowled 7.069196944152094e-06 -▁executed 7.069196944152094e-06 -▁exerc 7.069196944152094e-06 -▁technologische 7.069196944152094e-06 -ussen 7.0654085213524411e-06 -▁Hence 7.0654085213524411e-06 -▁Viol 7.0654085213524411e-06 -▁ce 7.0654085213524411e-06 -▁erlangen 7.0654085213524411e-06 -▁gutem 7.0654085213524411e-06 -▁modernes 7.0654085213524411e-06 -▁wages 7.0654085213524411e-06 -geräten 7.0616200985527882e-06 -usa 7.0616200985527882e-06 -▁Griff 7.0616200985527882e-06 -▁bien 7.0616200985527882e-06 -▁kommerziellen 7.0616200985527882e-06 -▁magn 7.0616200985527882e-06 -aca 7.0578316757531353e-06 -agentur 7.0578316757531353e-06 -▁Help 7.0578316757531353e-06 -▁Individ 7.0578316757531353e-06 -▁SY 7.0578316757531353e-06 -▁Serien 7.0578316757531353e-06 -▁Speed 7.0578316757531353e-06 -▁Steuers 7.0578316757531353e-06 -▁compare 7.0578316757531353e-06 -▁einhalten 7.0578316757531353e-06 -▁fears 7.0578316757531353e-06 -▁frucht 7.0578316757531353e-06 -▁lesson 7.0578316757531353e-06 -▁radik 7.0578316757531353e-06 -▁vernünftige 7.0578316757531353e-06 --like 7.0540432529534825e-06 -greifen 7.0540432529534825e-06 -ssicherheit 7.0540432529534825e-06 -▁CAM 7.0540432529534825e-06 -▁Fab 7.0540432529534825e-06 -▁Gef 7.0540432529534825e-06 -▁Leicht 7.0540432529534825e-06 -▁Luggage 7.0540432529534825e-06 -▁Stadtteil 7.0540432529534825e-06 -▁Trends 7.0540432529534825e-06 -▁finances 7.0540432529534825e-06 -▁formula 7.0540432529534825e-06 -▁grenzüberschreitende 7.0540432529534825e-06 -▁themes 7.0540432529534825e-06 -ektionen 7.05025483015383e-06 -verse 7.05025483015383e-06 -▁DI 7.05025483015383e-06 -▁Merk 7.05025483015383e-06 -▁Round 7.05025483015383e-06 -▁Total 7.05025483015383e-06 -▁begleiten 7.05025483015383e-06 -▁multimedia 7.05025483015383e-06 -▁realisieren 7.05025483015383e-06 -▁watching 7.05025483015383e-06 -inning 7.0464664073541775e-06 -▁24-Hour 7.0464664073541775e-06 -▁Aktivität 7.0464664073541775e-06 -▁Auftraggeber 7.0464664073541775e-06 -▁Spec 7.0464664073541775e-06 -▁acknowledged 7.0464664073541775e-06 -▁belonging 7.0464664073541775e-06 -▁klimatis 7.0464664073541775e-06 -▁undertakings 7.0464664073541775e-06 -ellos 7.0426779845545246e-06 -▁Cur 7.0426779845545246e-06 -▁Zubehör 7.0426779845545246e-06 -▁begangen 7.0426779845545246e-06 -▁cred 7.0426779845545246e-06 -▁drafting 7.0426779845545246e-06 -▁pact 7.0426779845545246e-06 --by 7.0388895617548717e-06 -akten 7.0388895617548717e-06 -atin 7.0388895617548717e-06 -▁Advanced 7.0388895617548717e-06 -▁Göt 7.0388895617548717e-06 -▁I. 7.0388895617548717e-06 -▁eingerichtete 7.0388895617548717e-06 -▁erinnere 7.0388895617548717e-06 -▁genetic 7.0388895617548717e-06 -▁regulate 7.0388895617548717e-06 -▁romant 7.0388895617548717e-06 -akis 7.0351011389552188e-06 -oro 7.0351011389552188e-06 -▁Kapazitäten 7.0351011389552188e-06 -▁Siehe 7.0351011389552188e-06 -▁ausz 7.0351011389552188e-06 -▁row 7.0351011389552188e-06 -bele 7.031312716155566e-06 -verband 7.031312716155566e-06 -▁Abstimmungen 7.031312716155566e-06 -▁Beobachter 7.031312716155566e-06 -▁Diam 7.031312716155566e-06 -▁Mittelmeerraum 7.031312716155566e-06 -▁registriert 7.031312716155566e-06 -▁shame 7.031312716155566e-06 -ichtungs 7.0275242933559131e-06 -ienste 7.0275242933559131e-06 -▁Bege 7.0275242933559131e-06 -▁CV 7.0275242933559131e-06 -▁Darin 7.0275242933559131e-06 -▁Integrations 7.0275242933559131e-06 -▁Wolfgang 7.0275242933559131e-06 -▁damaged 7.0275242933559131e-06 -aks 7.02373587055626e-06 -istry 7.02373587055626e-06 -▁Parks 7.02373587055626e-06 -▁Zuständigkeiten 7.02373587055626e-06 -▁BL 7.0199474477566081e-06 -▁Haushalte 7.0199474477566081e-06 -▁beurteilen 7.0199474477566081e-06 -▁equity 7.0199474477566081e-06 -▁typische 7.0199474477566081e-06 -▁örtlichen 7.0199474477566081e-06 -ehnten 7.0161590249569552e-06 -ilia 7.0161590249569552e-06 -verbrauch 7.0161590249569552e-06 -▁ignored 7.0161590249569552e-06 -lossene 7.0123706021573023e-06 -ried 7.0123706021573023e-06 -riffe 7.0123706021573023e-06 -▁Arbeitsbedingungen 7.0123706021573023e-06 -▁GS 7.0123706021573023e-06 -▁exklusiven 7.0123706021573023e-06 -▁übersch 7.0123706021573023e-06 -alty 7.0085821793576494e-06 -iable 7.0085821793576494e-06 -isses 7.0085821793576494e-06 -▁MD 7.0085821793576494e-06 -▁Macedonia 7.0085821793576494e-06 -▁Ok 7.0085821793576494e-06 -▁chairs 7.0085821793576494e-06 -▁eingeb 7.0085821793576494e-06 -▁jeopard 7.0085821793576494e-06 -▁1.2 7.0047937565579966e-06 -▁1975 7.0047937565579966e-06 -▁Benutz 7.0047937565579966e-06 -▁Betroffenen 7.0047937565579966e-06 -▁Cultural 7.0047937565579966e-06 -▁mensch 7.0047937565579966e-06 -▁womit 7.0047937565579966e-06 -▁zielt 7.0047937565579966e-06 -▁alloc 7.0010053337583437e-06 -▁spots 7.0010053337583437e-06 -▁tells 7.0010053337583437e-06 -44 6.9972169109586908e-06 -HR 6.9972169109586908e-06 -datei 6.9972169109586908e-06 -nik 6.9972169109586908e-06 -▁Bull 6.9972169109586908e-06 -▁Gemüse 6.9972169109586908e-06 -▁Queen 6.9972169109586908e-06 -▁coordin 6.9972169109586908e-06 -▁divid 6.9972169109586908e-06 -▁engineers 6.9972169109586908e-06 -▁fate 6.9972169109586908e-06 -▁hoped 6.9972169109586908e-06 -▁kritisiert 6.9972169109586908e-06 --Länder 6.9934284881590379e-06 -verschmutzung 6.9934284881590379e-06 -▁Kohle 6.9934284881590379e-06 -▁volunte 6.9934284881590379e-06 -kredit 6.9896400653593858e-06 -▁Superior 6.9896400653593858e-06 -▁aux 6.9896400653593858e-06 -▁depart 6.9896400653593858e-06 -▁tank 6.9896400653593858e-06 -lebnis 6.9858516425597329e-06 -▁71 6.9858516425597329e-06 -▁Colle 6.9858516425597329e-06 -▁Hart 6.9858516425597329e-06 -▁Inspekt 6.9858516425597329e-06 -▁Jackson 6.9858516425597329e-06 -▁Werbe 6.9858516425597329e-06 -▁Zertifik 6.9858516425597329e-06 -▁characterized 6.9858516425597329e-06 -▁interessieren 6.9858516425597329e-06 -▁mirror 6.9858516425597329e-06 -▁mithilfe 6.9858516425597329e-06 -▁tranqu 6.9858516425597329e-06 -▁Erwachsenen 6.98206321976008e-06 -▁Plenars 6.98206321976008e-06 -▁challenging 6.98206321976008e-06 -▁ehrgeiz 6.98206321976008e-06 -▁grundlegender 6.98206321976008e-06 -▁incentive 6.98206321976008e-06 -enna 6.9782747969604272e-06 -enntnis 6.9782747969604272e-06 -helfen 6.9782747969604272e-06 -▁Hung 6.9782747969604272e-06 -▁Soweit 6.9782747969604272e-06 -▁abund 6.9782747969604272e-06 -▁encounter 6.9782747969604272e-06 -▁exert 6.9782747969604272e-06 -▁schwarz 6.9782747969604272e-06 -ancies 6.9744863741607743e-06 -▁Prävention 6.9744863741607743e-06 -▁Wales 6.9744863741607743e-06 -▁congress 6.9744863741607743e-06 -▁gewisser 6.9744863741607743e-06 -▁modest 6.9744863741607743e-06 -▁prohibited 6.9744863741607743e-06 -▁vernünftig 6.9744863741607743e-06 -anna 6.9706979513611214e-06 -▁Deut 6.9706979513611214e-06 -▁Milosevic 6.9706979513611214e-06 -▁Trag 6.9706979513611214e-06 -▁faire 6.9706979513611214e-06 -▁falschen 6.9706979513611214e-06 -▁freund 6.9706979513611214e-06 -▁fulfilled 6.9706979513611214e-06 -▁pharm 6.9706979513611214e-06 --level 6.9669095285614685e-06 -bling 6.9669095285614685e-06 -uchtung 6.9669095285614685e-06 -▁Ferr 6.9669095285614685e-06 -▁NH 6.9669095285614685e-06 -▁Victoria 6.9669095285614685e-06 -▁bedauern 6.9669095285614685e-06 -▁sy 6.9669095285614685e-06 -▁verbesserte 6.9669095285614685e-06 -staats 6.9631211057618164e-06 -▁Entspannen 6.9631211057618164e-06 -▁Ib 6.9631211057618164e-06 -▁Palestine 6.9631211057618164e-06 -▁branches 6.9631211057618164e-06 -▁scandal 6.9631211057618164e-06 --gr 6.9593326829621636e-06 -ständen 6.9593326829621636e-06 -▁downloading 6.9593326829621636e-06 -▁environments 6.9593326829621636e-06 -▁Übersetzungs 6.9593326829621636e-06 -▁überd 6.9593326829621636e-06 -eling 6.9555442601625107e-06 -▁Tempo 6.9555442601625107e-06 -▁anticip 6.9555442601625107e-06 -DER 6.9517558373628578e-06 -▁Anreise 6.9517558373628578e-06 -▁Lehre 6.9517558373628578e-06 -▁Minder 6.9517558373628578e-06 -▁Stick 6.9517558373628578e-06 -▁UC 6.9517558373628578e-06 -▁Zufriedenheit 6.9517558373628578e-06 -iebt 6.9479674145632049e-06 -ommend 6.9479674145632049e-06 -rechnung 6.9479674145632049e-06 -üten 6.9479674145632049e-06 -▁Aires 6.9479674145632049e-06 -▁Lok 6.9479674145632049e-06 -▁abzusch 6.9479674145632049e-06 -▁fehler 6.9479674145632049e-06 -▁frisch 6.9479674145632049e-06 -▁login 6.9479674145632049e-06 -übt 6.944178991763552e-06 -▁1982 6.944178991763552e-06 -▁Afghan 6.944178991763552e-06 -▁Layout 6.944178991763552e-06 -▁ausgest 6.944178991763552e-06 -▁flowers 6.944178991763552e-06 -▁interakt 6.944178991763552e-06 -▁schlechten 6.944178991763552e-06 -getragen 6.9403905689638991e-06 -irtschaft 6.9403905689638991e-06 -leute 6.9403905689638991e-06 -▁Folter 6.9403905689638991e-06 -▁Schnee 6.9403905689638991e-06 -▁Sevilla 6.9403905689638991e-06 -▁Stell 6.9403905689638991e-06 -▁Taxi 6.9403905689638991e-06 -▁Vorredner 6.9403905689638991e-06 -▁ausführen 6.9403905689638991e-06 -▁ausgel 6.9403905689638991e-06 -atisf 6.9366021461642462e-06 -▁Entdecken 6.9366021461642462e-06 -▁Metern 6.9366021461642462e-06 -▁Vari 6.9366021461642462e-06 -▁brows 6.9366021461642462e-06 -▁Features 6.9328137233645942e-06 -▁INTER 6.9328137233645942e-06 -▁Mercedes 6.9328137233645942e-06 -▁Realisierung 6.9328137233645942e-06 -▁befried 6.9328137233645942e-06 -▁regrettable 6.9328137233645942e-06 --dr 6.9290253005649413e-06 -ärt 6.9290253005649413e-06 -▁herrliche 6.9290253005649413e-06 -▁publicly 6.9290253005649413e-06 -ency 6.9252368777652884e-06 -▁warming 6.9252368777652884e-06 -TH 6.9214484549656355e-06 -gebung 6.9214484549656355e-06 -▁DIE 6.9214484549656355e-06 -▁Erste 6.9214484549656355e-06 -▁Kreditkarten 6.9214484549656355e-06 -▁Ministerrat 6.9214484549656355e-06 -▁Oslo 6.9214484549656355e-06 -▁oblig 6.9214484549656355e-06 -▁zunehmende 6.9214484549656355e-06 -cket 6.9176600321659826e-06 -gebühr 6.9176600321659826e-06 -▁Engine 6.9176600321659826e-06 -▁Lehren 6.9176600321659826e-06 -▁fought 6.9176600321659826e-06 -▁iz 6.9176600321659826e-06 -▁ratifiziert 6.9176600321659826e-06 -▁tätigen 6.9176600321659826e-06 --Be 6.91387160936633e-06 -ased 6.91387160936633e-06 -schäd 6.91387160936633e-06 -▁Besuchern 6.91387160936633e-06 -▁Buenos 6.91387160936633e-06 -▁Summer 6.91387160936633e-06 -▁indirect 6.91387160936633e-06 -▁sheets 6.91387160936633e-06 -▁suff 6.91387160936633e-06 -ections 6.9100831865666768e-06 -▁Dikt 6.9100831865666768e-06 -▁Gipfeltreffen 6.9100831865666768e-06 -▁Regulierungs 6.9100831865666768e-06 -▁ansonsten 6.9100831865666768e-06 -▁demographic 6.9100831865666768e-06 -▁größerer 6.9100831865666768e-06 -▁technologischen 6.9100831865666768e-06 -▁unlike 6.9100831865666768e-06 -legte 6.9062947637670239e-06 -reformen 6.9062947637670239e-06 -▁Fas 6.9062947637670239e-06 -▁Lösungs 6.9062947637670239e-06 -▁Servic 6.9062947637670239e-06 -▁blank 6.9062947637670239e-06 -▁bureaucratic 6.9062947637670239e-06 -▁circuit 6.9062947637670239e-06 -▁einzuhalten 6.9062947637670239e-06 -▁fung 6.9062947637670239e-06 -▁thoughts 6.9062947637670239e-06 -▁thread 6.9062947637670239e-06 -asi 6.9025063409673719e-06 -greg 6.9025063409673719e-06 -iferation 6.9025063409673719e-06 -mö 6.9025063409673719e-06 -ätigen 6.9025063409673719e-06 -üste 6.9025063409673719e-06 -▁Close 6.9025063409673719e-06 -▁Instrumenten 6.9025063409673719e-06 -▁Kant 6.9025063409673719e-06 -▁este 6.9025063409673719e-06 -▁globaler 6.9025063409673719e-06 -▁schreibt 6.9025063409673719e-06 -▁shortcomings 6.9025063409673719e-06 -▁unzureichend 6.9025063409673719e-06 -▁vorhin 6.9025063409673719e-06 -arbeits 6.898717918167719e-06 -eph 6.898717918167719e-06 -igter 6.898717918167719e-06 -▁ausreichen 6.898717918167719e-06 -▁grös 6.898717918167719e-06 -iffs 6.8949294953680661e-06 -owski 6.8949294953680661e-06 -▁Bewegungen 6.8949294953680661e-06 -▁Guinea 6.8949294953680661e-06 -▁angefangen 6.8949294953680661e-06 -▁condemned 6.8949294953680661e-06 -kal 6.8911410725684132e-06 -▁Abstand 6.8911410725684132e-06 -▁Vil 6.8911410725684132e-06 -▁complexity 6.8911410725684132e-06 -▁guidance 6.8911410725684132e-06 -▁komplex 6.8911410725684132e-06 -▁survival 6.8911410725684132e-06 -▁Überarbeitung 6.8911410725684132e-06 --Technologie 6.88735264976876e-06 -behandlung 6.88735264976876e-06 -laim 6.88735264976876e-06 -▁Freiwill 6.88735264976876e-06 -▁amer 6.88735264976876e-06 -▁foods 6.88735264976876e-06 -▁translations 6.88735264976876e-06 -AY 6.8835642269691074e-06 -QU 6.8835642269691074e-06 -beit 6.8835642269691074e-06 -ivid 6.8835642269691074e-06 -ämp 6.8835642269691074e-06 -▁Allgemein 6.8835642269691074e-06 -▁Beauty 6.8835642269691074e-06 -▁Elect 6.8835642269691074e-06 -▁Finance 6.8835642269691074e-06 -▁Kompromiß 6.8835642269691074e-06 -▁Marktwirtschaft 6.8835642269691074e-06 -▁brother 6.8835642269691074e-06 -▁enttäuscht 6.8835642269691074e-06 -▁erreichte 6.8835642269691074e-06 -▁notion 6.8835642269691074e-06 -agl 6.8797758041694545e-06 -ocken 6.8797758041694545e-06 -unkel 6.8797758041694545e-06 -▁NE 6.8797758041694545e-06 -▁Techniken 6.8797758041694545e-06 -▁VI 6.8797758041694545e-06 -▁stem 6.8797758041694545e-06 -▁umfassender 6.8797758041694545e-06 -та 6.8759873813698016e-06 -▁Budgetary 6.8759873813698016e-06 -▁Gepäckraum 6.8759873813698016e-06 -▁Products 6.8759873813698016e-06 -▁Quell 6.8759873813698016e-06 -▁REACH 6.8759873813698016e-06 -▁alleged 6.8759873813698016e-06 -▁außerord 6.8759873813698016e-06 -▁founding 6.8759873813698016e-06 -▁gesonder 6.8759873813698016e-06 -▁kurzfristig 6.8759873813698016e-06 -▁shut 6.8759873813698016e-06 -▁unerlässlich 6.8759873813698016e-06 -▁vielfältigen 6.8759873813698016e-06 -adas 6.87219895857015e-06 -▁Credit 6.87219895857015e-06 -▁Hör 6.87219895857015e-06 -▁distinct 6.87219895857015e-06 -▁monitored 6.87219895857015e-06 -wen 6.8684105357704967e-06 -▁Früh 6.8684105357704967e-06 -▁Gestatten 6.8684105357704967e-06 -▁Norway 6.8684105357704967e-06 -▁advoc 6.8684105357704967e-06 -▁excellence 6.8684105357704967e-06 -▁militär 6.8684105357704967e-06 -▁structured 6.8684105357704967e-06 -agten 6.8646221129708438e-06 -akter 6.8646221129708438e-06 -▁Lido 6.8646221129708438e-06 -▁outset 6.8646221129708438e-06 -▁sequ 6.8646221129708438e-06 -alp 6.8608336901711909e-06 -icial 6.8608336901711909e-06 -right 6.8608336901711909e-06 -versicher 6.8608336901711909e-06 -äle 6.8608336901711909e-06 -▁Angriffe 6.8608336901711909e-06 -▁Bos 6.8608336901711909e-06 -▁brutal 6.8608336901711909e-06 -▁kulin 6.8608336901711909e-06 -▁namens 6.8608336901711909e-06 -▁subsidiary 6.8608336901711909e-06 -bro 6.857045267371538e-06 -öpf 6.857045267371538e-06 -▁Flag 6.857045267371538e-06 -▁derartigen 6.857045267371538e-06 -▁traditionell 6.857045267371538e-06 -▁verwaltet 6.857045267371538e-06 -▁weib 6.857045267371538e-06 --prod 6.8532568445718851e-06 -▁Verkäufer 6.8532568445718851e-06 -▁beschäft 6.8532568445718851e-06 -▁dress 6.8532568445718851e-06 -▁ped 6.8532568445718851e-06 -schutzes 6.8494684217722322e-06 -▁Luk 6.8494684217722322e-06 -▁Whirlpool 6.8494684217722322e-06 -▁accepts 6.8494684217722322e-06 -▁dauerhaft 6.8494684217722322e-06 -▁embra 6.8494684217722322e-06 -▁returning 6.8494684217722322e-06 -▁reverse 6.8494684217722322e-06 -▁sharp 6.8494684217722322e-06 -▁zwingen 6.8494684217722322e-06 -ORE 6.8456799989725793e-06 -ausenden 6.8456799989725793e-06 -▁Aust 6.8456799989725793e-06 -▁Holy 6.8456799989725793e-06 -▁Thai 6.8456799989725793e-06 -▁aufgel 6.8456799989725793e-06 -▁Spezialisten 6.8418915761729273e-06 -▁Vogel 6.8418915761729273e-06 -▁conjunction 6.8418915761729273e-06 -▁styles 6.8418915761729273e-06 -▁transatlantic 6.8418915761729273e-06 -▁yellow 6.8418915761729273e-06 -ipping 6.8381031533732744e-06 -▁autonomy 6.8381031533732744e-06 -▁bekräftigt 6.8381031533732744e-06 -iously 6.8343147305736215e-06 -▁Army 6.8343147305736215e-06 -▁Monats 6.8343147305736215e-06 -▁Vorsitzender 6.8343147305736215e-06 -▁bevorstehenden 6.8343147305736215e-06 -▁embry 6.8343147305736215e-06 -▁optical 6.8343147305736215e-06 -ears 6.8305263077739686e-06 -ätzt 6.8305263077739686e-06 -▁Declaration 6.8305263077739686e-06 -▁Lebensmitteln 6.8305263077739686e-06 -▁Nonetheless 6.8305263077739686e-06 -▁Privacy 6.8305263077739686e-06 -▁Provence 6.8305263077739686e-06 -▁Ron 6.8305263077739686e-06 -▁erläutert 6.8305263077739686e-06 -▁irische 6.8305263077739686e-06 -▁long- 6.8305263077739686e-06 -▁monat 6.8305263077739686e-06 -▁realized 6.8305263077739686e-06 -▁unterhalten 6.8305263077739686e-06 -▁verwöhnen 6.8305263077739686e-06 -▁zuf 6.8305263077739686e-06 -▁Gewähr 6.8267378849743157e-06 -▁Indonesia 6.8267378849743157e-06 -▁erzielten 6.8267378849743157e-06 -▁follow-up 6.8267378849743157e-06 -▁gran 6.8267378849743157e-06 -▁lacking 6.8267378849743157e-06 -▁meistens 6.8267378849743157e-06 -▁südlich 6.8267378849743157e-06 -anders 6.8229494621746628e-06 -rade 6.8229494621746628e-06 -на 6.8229494621746628e-06 -▁Echtzeit 6.8229494621746628e-06 -▁Ltd. 6.8229494621746628e-06 -▁Ortschaft 6.8229494621746628e-06 -▁inquiry 6.8229494621746628e-06 -kin 6.81916103937501e-06 -▁Tarif 6.81916103937501e-06 -▁gesammelt 6.81916103937501e-06 -▁rejection 6.81916103937501e-06 -gelegte 6.8153726165753571e-06 -höhe 6.8153726165753571e-06 -▁1968 6.8153726165753571e-06 -▁Kinnock 6.8153726165753571e-06 -▁bla 6.8153726165753571e-06 -WE 6.811584193775705e-06 -atalog 6.811584193775705e-06 -betten 6.811584193775705e-06 -ektors 6.811584193775705e-06 -ütter 6.811584193775705e-06 -▁happ 6.811584193775705e-06 -▁incorporate 6.811584193775705e-06 -▁invitation 6.811584193775705e-06 -▁pione 6.811584193775705e-06 -ographic 6.8077957709760521e-06 -▁Quick 6.8077957709760521e-06 -▁Spir 6.8077957709760521e-06 -▁Tap 6.8077957709760521e-06 -▁Weiterbildung 6.8077957709760521e-06 -▁begeistert 6.8077957709760521e-06 -▁finanz 6.8077957709760521e-06 -▁nutz 6.8077957709760521e-06 -▁tact 6.8077957709760521e-06 -chi 6.8040073481763992e-06 -clusive 6.8040073481763992e-06 -orld 6.8040073481763992e-06 -▁Alpine 6.8040073481763992e-06 -▁Legislativ 6.8040073481763992e-06 -▁Palazzo 6.8040073481763992e-06 -▁Stabilitätspakt 6.8040073481763992e-06 -▁extern 6.8040073481763992e-06 -▁mündlichen 6.8040073481763992e-06 --Seite 6.8002189253767463e-06 -handlungen 6.8002189253767463e-06 -p. 6.8002189253767463e-06 -▁Clearly 6.8002189253767463e-06 -▁Climate 6.8002189253767463e-06 -▁Dich 6.8002189253767463e-06 -▁checking 6.8002189253767463e-06 -▁departments 6.8002189253767463e-06 -▁sonn 6.8002189253767463e-06 -▁testen 6.8002189253767463e-06 -▁zivil 6.8002189253767463e-06 -aving 6.7964305025770935e-06 -fa 6.7964305025770935e-06 -▁Null 6.7964305025770935e-06 -▁PSE 6.7964305025770935e-06 -▁klären 6.7964305025770935e-06 -▁torture 6.7964305025770935e-06 -achsen 6.7926420797774406e-06 -design 6.7926420797774406e-06 -immungs 6.7926420797774406e-06 -ples 6.7926420797774406e-06 -▁unfortunate 6.7926420797774406e-06 --Team 6.7888536569777877e-06 -AST 6.7888536569777877e-06 -emma 6.7888536569777877e-06 -reitung 6.7888536569777877e-06 -utze 6.7888536569777877e-06 -▁Anregungen 6.7888536569777877e-06 -▁Aufforderung 6.7888536569777877e-06 -▁IR 6.7888536569777877e-06 -▁pine 6.7888536569777877e-06 -▁separately 6.7888536569777877e-06 -▁sul 6.7888536569777877e-06 -▁survive 6.7888536569777877e-06 --- 6.7850652341781348e-06 -uhr 6.7850652341781348e-06 -▁Israels 6.7850652341781348e-06 -▁Optimierung 6.7850652341781348e-06 -▁angehören 6.7850652341781348e-06 -▁anx 6.7850652341781348e-06 -▁corn 6.7850652341781348e-06 -▁searching 6.7850652341781348e-06 -▁tailored 6.7850652341781348e-06 -▁vorteil 6.7850652341781348e-06 -28 6.7812768113784827e-06 -basis 6.7812768113784827e-06 -▁Aktualisierung 6.7812768113784827e-06 -▁Sac 6.7812768113784827e-06 -▁guides 6.7812768113784827e-06 --Stand 6.77748838857883e-06 -stehenden 6.77748838857883e-06 -sweise 6.77748838857883e-06 -▁1976 6.77748838857883e-06 -▁Aufnahmen 6.77748838857883e-06 -▁Clinton 6.77748838857883e-06 -▁Rande 6.77748838857883e-06 -▁Unterdrückung 6.77748838857883e-06 -▁governed 6.77748838857883e-06 -▁ul 6.77748838857883e-06 -aner 6.773699965779177e-06 -dateien 6.773699965779177e-06 -eden 6.773699965779177e-06 -▁Feind 6.773699965779177e-06 -▁Liquid 6.773699965779177e-06 -▁Sech 6.773699965779177e-06 -▁denk 6.773699965779177e-06 -▁registrieren 6.773699965779177e-06 -▁teilgenommen 6.773699965779177e-06 -assten 6.7699115429795241e-06 -geschrieben 6.7699115429795241e-06 -▁Stadium 6.7699115429795241e-06 -▁Tschechischen 6.7699115429795241e-06 -▁Verantwortlichen 6.7699115429795241e-06 -▁delicate 6.7699115429795241e-06 -▁phot 6.7699115429795241e-06 -rage 6.7661231201798712e-06 -renz 6.7661231201798712e-06 -▁Hotelsafe 6.7661231201798712e-06 -▁Matt 6.7661231201798712e-06 -▁Moskau 6.7661231201798712e-06 -▁TNG 6.7661231201798712e-06 -▁kauf 6.7661231201798712e-06 -cul 6.7623346973802183e-06 -öten 6.7623346973802183e-06 -▁Kult 6.7623346973802183e-06 -▁abandoned 6.7623346973802183e-06 -▁diagnost 6.7623346973802183e-06 -,3 6.7585462745805654e-06 --Vor 6.7585462745805654e-06 -abeth 6.7585462745805654e-06 -adium 6.7585462745805654e-06 -▁Display 6.7585462745805654e-06 -▁Juden 6.7585462745805654e-06 -▁Sometimes 6.7585462745805654e-06 -▁entscheidet 6.7585462745805654e-06 -▁freundlichen 6.7585462745805654e-06 -▁seminars 6.7585462745805654e-06 -▁vegetables 6.7585462745805654e-06 -ogan 6.7547578517809133e-06 -rägen 6.7547578517809133e-06 -ätter 6.7547578517809133e-06 -▁Darauf 6.7547578517809133e-06 -▁Korean 6.7547578517809133e-06 -▁analys 6.7547578517809133e-06 -▁angeben 6.7547578517809133e-06 -▁characterised 6.7547578517809133e-06 -▁codes 6.7547578517809133e-06 -▁exceptions 6.7547578517809133e-06 -▁keinesfalls 6.7547578517809133e-06 -▁lights 6.7547578517809133e-06 -▁solide 6.7547578517809133e-06 -▁strenge 6.7547578517809133e-06 -▁tolerance 6.7547578517809133e-06 -▁upcoming 6.7547578517809133e-06 -▁Albania 6.7509694289812604e-06 -▁Jak 6.7509694289812604e-06 -▁vend 6.7509694289812604e-06 -platten 6.7471810061816076e-06 -wendungen 6.7471810061816076e-06 -▁Danke 6.7471810061816076e-06 -▁hielt 6.7471810061816076e-06 -aman 6.7433925833819547e-06 -aufgaben 6.7433925833819547e-06 -kontrollen 6.7433925833819547e-06 -laus 6.7433925833819547e-06 -ng 6.7433925833819547e-06 -▁Chicago 6.7433925833819547e-06 -▁Kohlen 6.7433925833819547e-06 -▁Lebensqualität 6.7433925833819547e-06 -▁Partners 6.7433925833819547e-06 -▁Reaktionen 6.7433925833819547e-06 -▁anzeigen 6.7433925833819547e-06 -▁defects 6.7433925833819547e-06 -▁discipline 6.7433925833819547e-06 -▁hosts 6.7433925833819547e-06 -▁loud 6.7433925833819547e-06 -▁reduces 6.7433925833819547e-06 -▁striking 6.7433925833819547e-06 -▁zust 6.7433925833819547e-06 --Auf 6.7396041605823018e-06 -ifer 6.7396041605823018e-06 -igur 6.7396041605823018e-06 -wachs 6.7396041605823018e-06 -▁149 6.7396041605823018e-06 -▁Color 6.7396041605823018e-06 -▁Intell 6.7396041605823018e-06 -▁fut 6.7396041605823018e-06 -▁herstellen 6.7396041605823018e-06 -▁tub 6.7396041605823018e-06 --Vertrag 6.7358157377826489e-06 -▁Cro 6.7358157377826489e-06 -▁LD 6.7358157377826489e-06 -▁derived 6.7358157377826489e-06 -▁drücken 6.7358157377826489e-06 -▁feels 6.7358157377826489e-06 -▁setup 6.7358157377826489e-06 -elson 6.732027314982996e-06 -eted 6.732027314982996e-06 -idge 6.732027314982996e-06 -künd 6.732027314982996e-06 -mb 6.732027314982996e-06 -oftware 6.732027314982996e-06 -▁Jeden 6.732027314982996e-06 -▁LateRooms.com 6.732027314982996e-06 -▁Wartung 6.732027314982996e-06 -▁appoint 6.732027314982996e-06 -▁diskrimin 6.732027314982996e-06 -▁immense 6.732027314982996e-06 -▁inde 6.732027314982996e-06 -▁rede 6.732027314982996e-06 -▁sofa 6.732027314982996e-06 -▁spiegelt 6.732027314982996e-06 -altete 6.7282388921833431e-06 -eler 6.7282388921833431e-06 -igerweise 6.7282388921833431e-06 -▁Einzelnen 6.7282388921833431e-06 -▁Rav 6.7282388921833431e-06 -▁Subventionen 6.7282388921833431e-06 -▁chocolate 6.7282388921833431e-06 -▁mitgeteilt 6.7282388921833431e-06 -▁recycling 6.7282388921833431e-06 -▁simplify 6.7282388921833431e-06 -▁virt 6.7282388921833431e-06 -BER 6.7244504693836911e-06 -üllen 6.7244504693836911e-06 -▁Avenue 6.7244504693836911e-06 -▁Ehren 6.7244504693836911e-06 -▁Vul 6.7244504693836911e-06 -▁vorbe 6.7244504693836911e-06 -▁zurückge 6.7244504693836911e-06 -einheit 6.7206620465840382e-06 -▁Litauen 6.7206620465840382e-06 -▁dish 6.7206620465840382e-06 -▁vereinfachen 6.7206620465840382e-06 --specific 6.7168736237843853e-06 -messe 6.7168736237843853e-06 -▁Emp 6.7168736237843853e-06 -▁Gewerkschaften 6.7168736237843853e-06 -▁Liter 6.7168736237843853e-06 -▁Working 6.7168736237843853e-06 -▁gele 6.7168736237843853e-06 -▁graf 6.7168736237843853e-06 -▁hr 6.7168736237843853e-06 -▁slight 6.7168736237843853e-06 -▁standen 6.7168736237843853e-06 -▁86 6.7130852009847324e-06 -▁Migrations 6.7130852009847324e-06 -▁exploring 6.7130852009847324e-06 -▁hiermit 6.7130852009847324e-06 -26 6.7092967781850795e-06 -ishment 6.7092967781850795e-06 -▁boom 6.7092967781850795e-06 -▁celebrated 6.7092967781850795e-06 -▁grants 6.7092967781850795e-06 -automat 6.7055083553854266e-06 -omaten 6.7055083553854266e-06 -quet 6.7055083553854266e-06 -▁Ath 6.7055083553854266e-06 -▁Beton 6.7055083553854266e-06 -▁Palästinens 6.7055083553854266e-06 -▁Vereinigung 6.7055083553854266e-06 -▁gegenseitige 6.7055083553854266e-06 -▁penalties 6.7055083553854266e-06 -▁philosoph 6.7055083553854266e-06 -▁receipt 6.7055083553854266e-06 -▁taught 6.7055083553854266e-06 -▁trademarks 6.7055083553854266e-06 -▁unnöt 6.7055083553854266e-06 --Daten 6.7017199325857737e-06 -aben 6.7017199325857737e-06 -dos 6.7017199325857737e-06 -eare 6.7017199325857737e-06 -ш 6.7017199325857737e-06 -▁3.5 6.7017199325857737e-06 -▁Entertainment 6.7017199325857737e-06 -▁Mazedonien 6.7017199325857737e-06 -▁Prestige 6.7017199325857737e-06 -greif 6.6979315097861208e-06 -▁automated 6.6979315097861208e-06 -▁inkl 6.6979315097861208e-06 -▁zwanzig 6.6979315097861208e-06 -rane 6.6941430869864688e-06 -▁absor 6.6941430869864688e-06 -▁aggressive 6.6941430869864688e-06 -▁ausgelöst 6.6941430869864688e-06 -▁regimes 6.6941430869864688e-06 -▁schließe 6.6941430869864688e-06 -sinn 6.6903546641868159e-06 -▁Vorsicht 6.6903546641868159e-06 -▁idealen 6.6903546641868159e-06 -▁indication 6.6903546641868159e-06 -eart 6.686566241387163e-06 -ourismus 6.686566241387163e-06 -▁Amtszeit 6.686566241387163e-06 -▁Maxim 6.686566241387163e-06 -▁Ri 6.686566241387163e-06 -▁angehen 6.686566241387163e-06 -▁bedenken 6.686566241387163e-06 -▁beiz 6.686566241387163e-06 -▁erh 6.686566241387163e-06 -▁shar 6.686566241387163e-06 -▁wording 6.686566241387163e-06 -SD 6.68277781858751e-06 -▁2,5 6.68277781858751e-06 -▁Currently 6.68277781858751e-06 -▁Erweiterungs 6.68277781858751e-06 -▁Horn 6.68277781858751e-06 -▁Medical 6.68277781858751e-06 -▁Meinungs 6.68277781858751e-06 -▁begr 6.68277781858751e-06 -▁irgendwelche 6.68277781858751e-06 -▁jeweilige 6.68277781858751e-06 -▁ster 6.68277781858751e-06 -▁veget 6.68277781858751e-06 -prochen 6.6789893957878572e-06 -▁Gumm 6.6789893957878572e-06 -▁Above 6.6752009729882043e-06 -▁Anwendungsbereich 6.6752009729882043e-06 -▁Spar 6.6752009729882043e-06 -▁accepting 6.6752009729882043e-06 -▁circle 6.6752009729882043e-06 -▁coinc 6.6752009729882043e-06 -▁finger 6.6752009729882043e-06 -▁perf 6.6752009729882043e-06 -▁poss 6.6752009729882043e-06 -▁spüren 6.6752009729882043e-06 -bten 6.6714125501885514e-06 -rey 6.6714125501885514e-06 -such 6.6714125501885514e-06 -▁Weltbank 6.6714125501885514e-06 -▁analysieren 6.6714125501885514e-06 -▁badly 6.6714125501885514e-06 -▁deutscher 6.6714125501885514e-06 -▁formul 6.6714125501885514e-06 -▁koordin 6.6714125501885514e-06 -▁relevanten 6.6714125501885514e-06 -▁superm 6.6714125501885514e-06 -▁vielfält 6.6714125501885514e-06 -uds 6.6676241273888985e-06 -▁CA 6.6676241273888985e-06 -▁Einschränkung 6.6676241273888985e-06 -▁Gerichtshofs 6.6676241273888985e-06 -▁editing 6.6676241273888985e-06 -▁sech 6.6676241273888985e-06 -.T 6.6638357045892465e-06 -deck 6.6638357045892465e-06 -▁Kra 6.6638357045892465e-06 -▁SIM 6.6638357045892465e-06 -▁Schiffs 6.6638357045892465e-06 -▁fünft 6.6638357045892465e-06 -▁kost 6.6638357045892465e-06 -▁nutzbar 6.6638357045892465e-06 -▁quota 6.6638357045892465e-06 -▁racism 6.6638357045892465e-06 -▁Age 6.6600472817895936e-06 -▁Dus 6.6600472817895936e-06 -▁Klin 6.6600472817895936e-06 -▁Spuren 6.6600472817895936e-06 -▁Zimmerservice 6.6600472817895936e-06 -▁gastronom 6.6600472817895936e-06 -▁schützt 6.6600472817895936e-06 -aciones 6.6562588589899407e-06 -emberg 6.6562588589899407e-06 -icts 6.6562588589899407e-06 -iefen 6.6562588589899407e-06 -▁Verträgen 6.6562588589899407e-06 -▁officer 6.6562588589899407e-06 -▁telling 6.6562588589899407e-06 -SL 6.6524704361902878e-06 -fig 6.6524704361902878e-06 -het 6.6524704361902878e-06 -▁Protokolls 6.6524704361902878e-06 -▁anc 6.6524704361902878e-06 -▁diplomat 6.6524704361902878e-06 -▁ignoriert 6.6524704361902878e-06 -▁reichhalt 6.6524704361902878e-06 -▁renov 6.6524704361902878e-06 -▁width 6.6524704361902878e-06 -▁DV 6.6486820133906349e-06 -▁ign 6.6486820133906349e-06 -▁interaction 6.6486820133906349e-06 -▁BC 6.644893590590982e-06 -▁Kaz 6.644893590590982e-06 -▁Kont 6.644893590590982e-06 -bey 6.6411051677913291e-06 -gefühl 6.6411051677913291e-06 -hafen 6.6411051677913291e-06 -ière 6.6411051677913291e-06 -ommiss 6.6411051677913291e-06 -▁GBP 6.6411051677913291e-06 -▁Wohlbefinden 6.6411051677913291e-06 -▁geltend 6.6411051677913291e-06 -▁kritisch 6.6411051677913291e-06 -▁Computers 6.6373167449916762e-06 -▁Kel 6.6373167449916762e-06 -▁Lav 6.6373167449916762e-06 -▁Naturally 6.6373167449916762e-06 -▁Ursprung 6.6373167449916762e-06 -▁Zweit 6.6373167449916762e-06 -▁continually 6.6373167449916762e-06 -▁erwart 6.6373167449916762e-06 -▁landscapes 6.6373167449916762e-06 -▁technolog 6.6373167449916762e-06 -▁übrig 6.6373167449916762e-06 -dez 6.6335283221920242e-06 -ifikation 6.6335283221920242e-06 -▁eure 6.6335283221920242e-06 -▁trails 6.6335283221920242e-06 -▁wenigstens 6.6335283221920242e-06 -leiten 6.6297398993923713e-06 -orate 6.6297398993923713e-06 -zeigen 6.6297398993923713e-06 -▁Erkenntnisse 6.6297398993923713e-06 -▁modes 6.6297398993923713e-06 -▁multipl 6.6297398993923713e-06 -▁muy 6.6297398993923713e-06 -▁preferences 6.6297398993923713e-06 -oque 6.6259514765927184e-06 -osing 6.6259514765927184e-06 -▁'d 6.6259514765927184e-06 -▁UND 6.6259514765927184e-06 -▁contest 6.6259514765927184e-06 -▁undertaking 6.6259514765927184e-06 -▁Ein- 6.6221630537930655e-06 -▁Kindes 6.6221630537930655e-06 -▁Silber 6.6221630537930655e-06 -▁associ 6.6221630537930655e-06 -▁configure 6.6221630537930655e-06 -▁denied 6.6221630537930655e-06 -▁gesunden 6.6221630537930655e-06 -▁interim 6.6221630537930655e-06 -▁shareholders 6.6221630537930655e-06 -▁Clean 6.6183746309934126e-06 -▁Eingabe 6.6183746309934126e-06 -▁Eurostat 6.6183746309934126e-06 -▁Guard 6.6183746309934126e-06 -▁angesch 6.6183746309934126e-06 -▁auswärtige 6.6183746309934126e-06 -▁begab 6.6183746309934126e-06 -▁cheaper 6.6183746309934126e-06 -▁deinen 6.6183746309934126e-06 -▁erneuerbare 6.6183746309934126e-06 -▁navig 6.6183746309934126e-06 -▁schrieb 6.6183746309934126e-06 -▁setzte 6.6183746309934126e-06 -keeping 6.61458620819376e-06 -otti 6.61458620819376e-06 -▁CC 6.61458620819376e-06 -▁Ecke 6.61458620819376e-06 -▁einzelner 6.61458620819376e-06 -▁responses 6.61458620819376e-06 -iments 6.6107977853941068e-06 -ittel 6.6107977853941068e-06 -lets 6.6107977853941068e-06 -page 6.6107977853941068e-06 -verz 6.6107977853941068e-06 -▁Conf 6.6107977853941068e-06 -▁Elekt 6.6107977853941068e-06 -▁Kohärenz 6.6107977853941068e-06 -▁drafted 6.6107977853941068e-06 -▁ensured 6.6107977853941068e-06 -aded 6.607009362594454e-06 -itären 6.607009362594454e-06 -mäß 6.607009362594454e-06 -▁arab 6.607009362594454e-06 -▁assumed 6.607009362594454e-06 -▁beider 6.607009362594454e-06 -▁doubl 6.607009362594454e-06 -▁gefährliche 6.607009362594454e-06 -▁gingen 6.607009362594454e-06 -▁pron 6.607009362594454e-06 -▁verp 6.607009362594454e-06 -asa 6.6032209397948019e-06 -streit 6.6032209397948019e-06 -▁DM 6.6032209397948019e-06 -▁Danube 6.6032209397948019e-06 -▁Plast 6.6032209397948019e-06 -▁Rollen 6.6032209397948019e-06 -▁Spanisch 6.6032209397948019e-06 -▁Touch 6.6032209397948019e-06 -▁kompletten 6.6032209397948019e-06 -▁reductions 6.6032209397948019e-06 -▁walks 6.6032209397948019e-06 -▁Übersetzungen 6.6032209397948019e-06 -stunden 6.599432516995149e-06 -unktion 6.599432516995149e-06 -ù 6.599432516995149e-06 -▁Love 6.599432516995149e-06 -▁MT 6.599432516995149e-06 -▁gefährlichen 6.599432516995149e-06 -▁infringement 6.599432516995149e-06 -▁reinforce 6.599432516995149e-06 -▁sustained 6.599432516995149e-06 -▁unlimited 6.599432516995149e-06 -zugreifen 6.5956440941954961e-06 -züglich 6.5956440941954961e-06 -▁Fernseher 6.5956440941954961e-06 -▁Siena 6.5956440941954961e-06 -▁T. 6.5956440941954961e-06 -▁Tourist 6.5956440941954961e-06 -▁absch 6.5956440941954961e-06 -▁combining 6.5956440941954961e-06 -▁doctors 6.5956440941954961e-06 -▁genehmigt 6.5956440941954961e-06 -▁Üb 6.5956440941954961e-06 -▁eingerichteten 6.5918556713958432e-06 -▁imprison 6.5918556713958432e-06 -▁random 6.5918556713958432e-06 -▁reiterate 6.5918556713958432e-06 -schm 6.58806724859619e-06 -▁Aufsicht 6.58806724859619e-06 -▁Konsultation 6.58806724859619e-06 -▁circulation 6.58806724859619e-06 -▁evil 6.58806724859619e-06 -▁gren 6.58806724859619e-06 -▁minds 6.58806724859619e-06 -▁missions 6.58806724859619e-06 -▁strafrecht 6.58806724859619e-06 -▁tat 6.58806724859619e-06 -▁wirkungs 6.58806724859619e-06 -▁Bast 6.5842788257965375e-06 -▁Bewertungen 6.5842788257965375e-06 -▁Lokal 6.5842788257965375e-06 -▁Weihnachts 6.5842788257965375e-06 -▁hüb 6.5842788257965375e-06 -▁verschiedensten 6.5842788257965375e-06 -▁wartet 6.5842788257965375e-06 -bot 6.5804904029968846e-06 -▁Ausgleich 6.5804904029968846e-06 -▁Peru 6.5804904029968846e-06 -▁embedded 6.5804904029968846e-06 -▁quantities 6.5804904029968846e-06 -▁stood 6.5804904029968846e-06 -atisch 6.5767019801972325e-06 -ündung 6.5767019801972325e-06 -▁Ausstellungen 6.5767019801972325e-06 -▁exklusive 6.5767019801972325e-06 -▁glück 6.5767019801972325e-06 -▁internationales 6.5767019801972325e-06 -asters 6.57291355739758e-06 -jor 6.57291355739758e-06 -▁Einwohnern 6.57291355739758e-06 -▁Wikitravel 6.57291355739758e-06 -▁tension 6.57291355739758e-06 --en 6.5691251345979267e-06 -ifiable 6.5691251345979267e-06 -onischen 6.5691251345979267e-06 -vertra 6.5691251345979267e-06 -▁Bisher 6.5691251345979267e-06 -▁Building 6.5691251345979267e-06 -▁G8 6.5691251345979267e-06 -▁Seminar 6.5691251345979267e-06 -▁Wiederauf 6.5691251345979267e-06 -▁m. 6.5691251345979267e-06 -▁thin 6.5691251345979267e-06 -▁verhalten 6.5691251345979267e-06 -Pro 6.5653367117982738e-06 -cribed 6.5653367117982738e-06 -lech 6.5653367117982738e-06 -rem 6.5653367117982738e-06 -▁220 6.5653367117982738e-06 -▁Tragödie 6.5653367117982738e-06 -▁Veranstaltungs 6.5653367117982738e-06 -▁engagiert 6.5653367117982738e-06 -▁impress 6.5653367117982738e-06 -▁participated 6.5653367117982738e-06 -ignano 6.561548288998621e-06 -▁Heritage 6.561548288998621e-06 -▁Mitarbeit 6.561548288998621e-06 -▁anwend 6.561548288998621e-06 -▁dos 6.561548288998621e-06 -▁endet 6.561548288998621e-06 -▁flex 6.561548288998621e-06 --12 6.5577598661989681e-06 -amera 6.5577598661989681e-06 -wing 6.5577598661989681e-06 -▁16th 6.5577598661989681e-06 -▁Mars 6.5577598661989681e-06 -▁fuels 6.5577598661989681e-06 -▁hire 6.5577598661989681e-06 -ighten 6.5539714433993152e-06 -▁Harbour 6.5539714433993152e-06 -▁ausges 6.5539714433993152e-06 -▁trad 6.5539714433993152e-06 -▁ungeachtet 6.5539714433993152e-06 -▁zurückkehren 6.5539714433993152e-06 -agert 6.5501830205996623e-06 -earbeit 6.5501830205996623e-06 -schirm 6.5501830205996623e-06 -wäl 6.5501830205996623e-06 -▁Grande 6.5501830205996623e-06 -▁Hinzu 6.5501830205996623e-06 -▁li 6.5501830205996623e-06 -▁nennt 6.5501830205996623e-06 -▁publ 6.5501830205996623e-06 -agog 6.54639459780001e-06 -kw 6.54639459780001e-06 -wirkungen 6.54639459780001e-06 -zentrums 6.54639459780001e-06 -▁Could 6.54639459780001e-06 -▁assembl 6.54639459780001e-06 -▁deleted 6.54639459780001e-06 -▁isolation 6.54639459780001e-06 -▁vig 6.54639459780001e-06 -zuführen 6.5426061750003573e-06 -änglich 6.5426061750003573e-06 -▁Christen 6.5426061750003573e-06 -▁Gesprächen 6.5426061750003573e-06 -▁Sy 6.5426061750003573e-06 -▁boards 6.5426061750003573e-06 -▁riesige 6.5426061750003573e-06 -▁votre 6.5426061750003573e-06 -▁äußerster 6.5426061750003573e-06 -asma 6.5388177522007045e-06 -owed 6.5388177522007045e-06 -▁Mi 6.5388177522007045e-06 -▁defending 6.5388177522007045e-06 -▁enh 6.5388177522007045e-06 -▁ov 6.5388177522007045e-06 -▁requiring 6.5388177522007045e-06 -▁tiny 6.5388177522007045e-06 -enzen 6.5350293294010516e-06 -opolis 6.5350293294010516e-06 -path 6.5350293294010516e-06 -▁Stay 6.5350293294010516e-06 -▁myst 6.5350293294010516e-06 -▁umweltfreund 6.5350293294010516e-06 -▁ökologische 6.5350293294010516e-06 -chn 6.5312409066013987e-06 -ilding 6.5312409066013987e-06 -▁Benjamin 6.5312409066013987e-06 -▁Moderne 6.5312409066013987e-06 -▁Vorbild 6.5312409066013987e-06 -▁arbeitete 6.5312409066013987e-06 -▁moderate 6.5312409066013987e-06 -▁plat 6.5312409066013987e-06 -▁transformed 6.5312409066013987e-06 -ianische 6.5274524838017458e-06 -ifft 6.5274524838017458e-06 -rikt 6.5274524838017458e-06 -▁BA 6.5274524838017458e-06 -▁Betrugs 6.5274524838017458e-06 -▁Fig 6.5274524838017458e-06 -▁HOTEL 6.5274524838017458e-06 -▁Steu 6.5274524838017458e-06 -▁consistency 6.5274524838017458e-06 -▁cul 6.5274524838017458e-06 -▁handic 6.5274524838017458e-06 -gespr 6.5236640610020929e-06 -▁Anst 6.5236640610020929e-06 -▁Geräten 6.5236640610020929e-06 -▁LA 6.5236640610020929e-06 -▁abwechslungs 6.5236640610020929e-06 -▁derog 6.5236640610020929e-06 -▁simplification 6.5236640610020929e-06 -kürz 6.51987563820244e-06 -tein 6.51987563820244e-06 -▁Charakt 6.51987563820244e-06 -▁Ernährungs 6.51987563820244e-06 -▁Fine 6.51987563820244e-06 -▁PM 6.51987563820244e-06 -▁Verstärkung 6.51987563820244e-06 -▁Zer 6.51987563820244e-06 -▁calculated 6.51987563820244e-06 -▁daughter 6.51987563820244e-06 -▁längst 6.51987563820244e-06 -▁maj 6.51987563820244e-06 -▁175 6.5160872154027879e-06 -▁Entwicklungszusammenarbeit 6.5160872154027879e-06 -▁gefällt 6.5160872154027879e-06 -▁trennen 6.5160872154027879e-06 -▁underline 6.5160872154027879e-06 -▁übersetzt 6.5160872154027879e-06 -bringt 6.5122987926031351e-06 -▁Herberge 6.5122987926031351e-06 -▁Oberflächen 6.5122987926031351e-06 -▁Statistik 6.5122987926031351e-06 -▁abgehalten 6.5122987926031351e-06 --century 6.5085103698034822e-06 -IVE 6.5085103698034822e-06 -zusprechen 6.5085103698034822e-06 -▁Manche 6.5085103698034822e-06 -▁boxes 6.5085103698034822e-06 -▁enemy 6.5085103698034822e-06 -▁sightseeing 6.5085103698034822e-06 -▁struggl 6.5085103698034822e-06 --bu 6.5047219470038293e-06 -estone 6.5047219470038293e-06 -leiben 6.5047219470038293e-06 -ockt 6.5047219470038293e-06 -uman 6.5047219470038293e-06 -▁61 6.5047219470038293e-06 -▁Kü 6.5047219470038293e-06 -▁Libanon 6.5047219470038293e-06 -▁Neub 6.5047219470038293e-06 -▁Ratsvorsitzes 6.5047219470038293e-06 -▁consisting 6.5047219470038293e-06 -▁einziges 6.5047219470038293e-06 -▁listened 6.5047219470038293e-06 -▁motions 6.5047219470038293e-06 -▁newspapers 6.5047219470038293e-06 -▁Bot 6.5009335242041764e-06 -▁DS9 6.5009335242041764e-06 -▁Flüchtlings 6.5009335242041764e-06 -▁fabric 6.5009335242041764e-06 -stätte 6.4971451014045235e-06 -▁93 6.4971451014045235e-06 -▁Karel 6.4971451014045235e-06 -▁Virtual 6.4971451014045235e-06 -▁kooper 6.4971451014045235e-06 -▁kritische 6.4971451014045235e-06 -▁resorts 6.4971451014045235e-06 -▁zunehmenden 6.4971451014045235e-06 --Distribution 6.4933566786048706e-06 -linder 6.4933566786048706e-06 -▁Dienstag 6.4933566786048706e-06 -▁Future 6.4933566786048706e-06 -▁Limited 6.4933566786048706e-06 -▁Origin 6.4933566786048706e-06 -▁Rahmenprogramm 6.4933566786048706e-06 -▁Sozialdemokraten 6.4933566786048706e-06 -▁alliance 6.4933566786048706e-06 -▁continental 6.4933566786048706e-06 -fäh 6.4895682558052177e-06 -ā 6.4895682558052177e-06 -▁announcement 6.4895682558052177e-06 -▁aufweisen 6.4895682558052177e-06 -▁bases 6.4895682558052177e-06 -▁clothes 6.4895682558052177e-06 -▁culinary 6.4895682558052177e-06 -▁gegr 6.4895682558052177e-06 -▁gemessen 6.4895682558052177e-06 -▁plane 6.4895682558052177e-06 --Ge 6.4857798330055657e-06 -rio 6.4857798330055657e-06 -▁Auslegung 6.4857798330055657e-06 -▁Bluetooth 6.4857798330055657e-06 -▁Spons 6.4857798330055657e-06 -▁endless 6.4857798330055657e-06 -▁ethnischen 6.4857798330055657e-06 -▁islam 6.4857798330055657e-06 -asp 6.4819914102059128e-06 -▁Anhänger 6.4819914102059128e-06 -▁commonly 6.4819914102059128e-06 -▁explos 6.4819914102059128e-06 -▁klassische 6.4819914102059128e-06 -mengen 6.47820298740626e-06 -▁Devisen 6.47820298740626e-06 -▁Finanzmittel 6.47820298740626e-06 -▁concentrated 6.47820298740626e-06 -▁vorherige 6.47820298740626e-06 -gewinn 6.474414564606607e-06 -map 6.474414564606607e-06 -▁Kalender 6.474414564606607e-06 -▁Ostsee 6.474414564606607e-06 -▁identical 6.474414564606607e-06 -▁investigate 6.474414564606607e-06 -▁traur 6.474414564606607e-06 --En 6.4706261418069541e-06 -iewer 6.4706261418069541e-06 -utzer 6.4706261418069541e-06 -▁Indem 6.4706261418069541e-06 -▁Shuttle 6.4706261418069541e-06 -▁large-scale 6.4706261418069541e-06 -▁presentations 6.4706261418069541e-06 -▁vicinity 6.4706261418069541e-06 -▁vorherigen 6.4706261418069541e-06 -▁älter 6.4706261418069541e-06 -ahrt 6.4668377190073012e-06 -aktions 6.4668377190073012e-06 -izität 6.4668377190073012e-06 -▁API 6.4668377190073012e-06 -▁angere 6.4668377190073012e-06 -▁beträchtliche 6.4668377190073012e-06 -▁crossing 6.4668377190073012e-06 -▁drast 6.4668377190073012e-06 -▁lohnt 6.4668377190073012e-06 -cape 6.4630492962076483e-06 -ermann 6.4630492962076483e-06 -kirche 6.4630492962076483e-06 -ów 6.4630492962076483e-06 -▁cargo 6.4630492962076483e-06 -▁homepage 6.4630492962076483e-06 -▁respecting 6.4630492962076483e-06 -aterie 6.4592608734079954e-06 -hundert 6.4592608734079954e-06 -▁128 6.4592608734079954e-06 -▁Lärm 6.4592608734079954e-06 -▁Potential 6.4592608734079954e-06 -▁camping 6.4592608734079954e-06 -▁durchgeführten 6.4592608734079954e-06 -▁prev 6.4592608734079954e-06 -▁seal 6.4592608734079954e-06 -▁sont 6.4592608734079954e-06 -meldung 6.4554724506083434e-06 -▁Libya 6.4554724506083434e-06 -▁Magic 6.4554724506083434e-06 -▁Parallel 6.4554724506083434e-06 -▁officers 6.4554724506083434e-06 -würdig 6.4516840278086905e-06 -▁Aufwand 6.4516840278086905e-06 -▁Rechenschaft 6.4516840278086905e-06 -▁cens 6.4516840278086905e-06 -▁gegenseitig 6.4516840278086905e-06 -▁inher 6.4516840278086905e-06 -▁weapon 6.4516840278086905e-06 -▁üblich 6.4516840278086905e-06 -imme 6.4478956050090376e-06 -ż 6.4478956050090376e-06 -▁Initiativ 6.4478956050090376e-06 -▁Scha 6.4478956050090376e-06 -▁Spam 6.4478956050090376e-06 -▁angep 6.4478956050090376e-06 -▁colonial 6.4478956050090376e-06 -▁permanently 6.4478956050090376e-06 -▁weekly 6.4478956050090376e-06 -▁zuverlässig 6.4478956050090376e-06 --room 6.4441071822093847e-06 -üssel 6.4441071822093847e-06 -▁Cook 6.4441071822093847e-06 -▁Eintrag 6.4441071822093847e-06 -▁Manhattan 6.4441071822093847e-06 -▁Marbella 6.4441071822093847e-06 -▁Sak 6.4441071822093847e-06 -▁Schre 6.4441071822093847e-06 -▁Week 6.4441071822093847e-06 -▁announ 6.4441071822093847e-06 -▁bond 6.4441071822093847e-06 -cribe 6.4403187594097318e-06 -ields 6.4403187594097318e-06 -▁Fertigung 6.4403187594097318e-06 -▁dell 6.4403187594097318e-06 -▁starkes 6.4403187594097318e-06 -▁ton 6.4403187594097318e-06 -gäste 6.4365303366100789e-06 -▁Leidenschaft 6.4365303366100789e-06 -▁Tol 6.4365303366100789e-06 -▁ending 6.4365303366100789e-06 -▁entertain 6.4365303366100789e-06 -ки 6.432741913810426e-06 -▁Crist 6.432741913810426e-06 -▁Schauspiel 6.432741913810426e-06 -▁Vorstand 6.432741913810426e-06 -▁humans 6.432741913810426e-06 -▁penal 6.432741913810426e-06 -▁stre 6.432741913810426e-06 -▁yo 6.432741913810426e-06 -OX 6.4289534910107731e-06 -eries 6.4289534910107731e-06 -▁Konfiguration 6.4289534910107731e-06 -▁ernste 6.4289534910107731e-06 -▁spont 6.4289534910107731e-06 --15 6.4251650682111211e-06 -bring 6.4251650682111211e-06 -во 6.4251650682111211e-06 -▁Ausfahrt 6.4251650682111211e-06 -▁Deluxe 6.4251650682111211e-06 -▁ME 6.4251650682111211e-06 -▁MPEG 6.4251650682111211e-06 -▁Slowenien 6.4251650682111211e-06 -▁connecting 6.4251650682111211e-06 -▁districts 6.4251650682111211e-06 -▁enthusiasm 6.4251650682111211e-06 -▁fu 6.4251650682111211e-06 -▁hea 6.4251650682111211e-06 -▁niederländischen 6.4251650682111211e-06 -▁publisher 6.4251650682111211e-06 -ertain 6.4213766454114682e-06 -oßen 6.4213766454114682e-06 -▁Entstehung 6.4213766454114682e-06 -▁Stefan 6.4213766454114682e-06 -▁Vot 6.4213766454114682e-06 -▁denselben 6.4213766454114682e-06 -▁refined 6.4213766454114682e-06 -▁sus 6.4213766454114682e-06 -disziplin 6.4175882226118153e-06 -rückt 6.4175882226118153e-06 -▁provin 6.4175882226118153e-06 -▁restoration 6.4175882226118153e-06 -▁rul 6.4175882226118153e-06 -▁verehrten 6.4175882226118153e-06 -dest 6.4137997998121624e-06 -indlichen 6.4137997998121624e-06 -spruch 6.4137997998121624e-06 -ulse 6.4137997998121624e-06 -▁Heiz 6.4137997998121624e-06 -▁Palästina 6.4137997998121624e-06 -▁banned 6.4137997998121624e-06 -▁dominant 6.4137997998121624e-06 -▁optimum 6.4137997998121624e-06 -▁prejud 6.4137997998121624e-06 -▁sämtlichen 6.4137997998121624e-06 -anges 6.4100113770125095e-06 -▁Bali 6.4100113770125095e-06 -▁Flugzeuge 6.4100113770125095e-06 -▁Wörter 6.4100113770125095e-06 -▁bekanntesten 6.4100113770125095e-06 -▁ethn 6.4100113770125095e-06 -▁suddenly 6.4100113770125095e-06 -▁zeichnen 6.4100113770125095e-06 -.11 6.4062229542128566e-06 -▁Lodge 6.4062229542128566e-06 -▁adher 6.4062229542128566e-06 -▁analyses 6.4062229542128566e-06 -▁aufgehoben 6.4062229542128566e-06 -▁discretion 6.4062229542128566e-06 -▁grew 6.4062229542128566e-06 -▁pursuing 6.4062229542128566e-06 --Ländern 6.4024345314132037e-06 -▁Absichten 6.4024345314132037e-06 -▁Buchungs 6.4024345314132037e-06 -▁EMU 6.4024345314132037e-06 -▁Local 6.4024345314132037e-06 -▁Mittler 6.4024345314132037e-06 -▁SSL 6.4024345314132037e-06 -▁amtierender 6.4024345314132037e-06 -▁bez 6.4024345314132037e-06 -ioni 6.3986461086135517e-06 -▁Geschäftsbedingungen 6.3986461086135517e-06 -▁Grill 6.3986461086135517e-06 -▁Portug 6.3986461086135517e-06 -▁clos 6.3986461086135517e-06 -▁e.V. 6.3986461086135517e-06 -▁exploit 6.3986461086135517e-06 -▁observers 6.3986461086135517e-06 -▁wirksamen 6.3986461086135517e-06 -amous 6.3948576858138988e-06 -oked 6.3948576858138988e-06 -▁Architekten 6.3948576858138988e-06 -▁Piet 6.3948576858138988e-06 -▁Prem 6.3948576858138988e-06 -▁Sz 6.3948576858138988e-06 -▁booth 6.3948576858138988e-06 -▁discovery 6.3948576858138988e-06 -▁exch 6.3948576858138988e-06 -▁unprecedented 6.3948576858138988e-06 -▁verhandeln 6.3948576858138988e-06 -gin 6.3910692630142459e-06 -▁Einfuhr 6.3910692630142459e-06 -▁Hass 6.3910692630142459e-06 -▁Quer 6.3910692630142459e-06 -▁continuation 6.3910692630142459e-06 -▁passes 6.3910692630142459e-06 -▁specialised 6.3910692630142459e-06 -▁Anruf 6.387280840214593e-06 -▁Bruder 6.387280840214593e-06 -▁abend 6.387280840214593e-06 -▁adaptation 6.387280840214593e-06 -▁ausgeb 6.387280840214593e-06 -▁complimentary 6.387280840214593e-06 -▁perpet 6.387280840214593e-06 -▁pull 6.387280840214593e-06 -ühen 6.38349241741494e-06 -▁Lust 6.38349241741494e-06 -▁Minimum 6.38349241741494e-06 -▁Prob 6.38349241741494e-06 -▁accord 6.38349241741494e-06 -▁bevorzug 6.38349241741494e-06 -▁nos 6.38349241741494e-06 -▁separated 6.38349241741494e-06 -▁unteren 6.38349241741494e-06 -▁verständlich 6.38349241741494e-06 -▁zahlreicher 6.38349241741494e-06 --bas 6.3797039946152872e-06 -ILL 6.3797039946152872e-06 -emissionen 6.3797039946152872e-06 -fuhren 6.3797039946152872e-06 -omed 6.3797039946152872e-06 -äne 6.3797039946152872e-06 -▁Transaktionen 6.3797039946152872e-06 -ancen 6.3759155718156344e-06 -onnen 6.3759155718156344e-06 -▁horizont 6.3759155718156344e-06 -▁konserv 6.3759155718156344e-06 -weisungen 6.3721271490159815e-06 -▁XL 6.3721271490159815e-06 -▁ambience 6.3721271490159815e-06 -▁linguistic 6.3721271490159815e-06 -▁part-session 6.3721271490159815e-06 --Netz 6.3683387262163294e-06 -ferenzen 6.3683387262163294e-06 -utzt 6.3683387262163294e-06 -äten 6.3683387262163294e-06 -▁Micro 6.3683387262163294e-06 -▁accused 6.3683387262163294e-06 -▁knowing 6.3683387262163294e-06 -▁lov 6.3683387262163294e-06 -ERS 6.3645503034166765e-06 -▁Eurex 6.3645503034166765e-06 -▁beeinträchtigt 6.3645503034166765e-06 -▁mainstream 6.3645503034166765e-06 -▁ältesten 6.3645503034166765e-06 -iated 6.3607618806170236e-06 -rang 6.3607618806170236e-06 -status 6.3607618806170236e-06 -▁DH 6.3607618806170236e-06 -▁Lul 6.3607618806170236e-06 -▁Verhältnisse 6.3607618806170236e-06 -▁hyper 6.3607618806170236e-06 -▁vergleichen 6.3607618806170236e-06 -club 6.3569734578173707e-06 -raph 6.3569734578173707e-06 -▁Bir 6.3569734578173707e-06 -▁Niemand 6.3569734578173707e-06 -▁Perspektiven 6.3569734578173707e-06 -▁akzeptabel 6.3569734578173707e-06 -▁tackling 6.3569734578173707e-06 -hops 6.3531850350177178e-06 -iess 6.3531850350177178e-06 -▁Algarve 6.3531850350177178e-06 -▁Austr 6.3531850350177178e-06 -▁Ernährung 6.3531850350177178e-06 -▁Navigation 6.3531850350177178e-06 -▁annex 6.3531850350177178e-06 -▁binden 6.3531850350177178e-06 -▁gains 6.3531850350177178e-06 -▁gefangen 6.3531850350177178e-06 -▁jump 6.3531850350177178e-06 -▁preced 6.3531850350177178e-06 -stützung 6.349396612218065e-06 -▁filling 6.349396612218065e-06 -GA 6.3456081894184121e-06 -abschluss 6.3456081894184121e-06 -day 6.3456081894184121e-06 -weight 6.3456081894184121e-06 -▁17th 6.3456081894184121e-06 -▁Abonn 6.3456081894184121e-06 -▁Brit 6.3456081894184121e-06 -▁Eintritt 6.3456081894184121e-06 -▁glaubt 6.3456081894184121e-06 -▁implies 6.3456081894184121e-06 -▁mining 6.3456081894184121e-06 -ammlung 6.3418197666187592e-06 -bilder 6.3418197666187592e-06 -gon 6.3418197666187592e-06 -schten 6.3418197666187592e-06 -uma 6.3418197666187592e-06 -▁94 6.3418197666187592e-06 -▁Cos 6.3418197666187592e-06 -▁Verzögerung 6.3418197666187592e-06 -▁alive 6.3418197666187592e-06 -▁idealer 6.3418197666187592e-06 -▁luggage 6.3418197666187592e-06 -hrop 6.3380313438191071e-06 -▁Erwer 6.3380313438191071e-06 -▁Textes 6.3380313438191071e-06 -adem 6.3342429210194542e-06 -urop 6.3342429210194542e-06 -weichen 6.3342429210194542e-06 -ν 6.3342429210194542e-06 -▁Kathedrale 6.3342429210194542e-06 -▁Maison 6.3342429210194542e-06 -▁Nan 6.3342429210194542e-06 -▁Vall 6.3342429210194542e-06 -▁ausgelegt 6.3342429210194542e-06 -▁jahrel 6.3342429210194542e-06 -▁jüng 6.3342429210194542e-06 -▁twelve 6.3342429210194542e-06 -Net 6.3304544982198013e-06 -ulates 6.3304544982198013e-06 -▁Anweisungen 6.3304544982198013e-06 -▁Bedingung 6.3304544982198013e-06 -▁Outlook 6.3304544982198013e-06 -▁Rosen 6.3304544982198013e-06 -▁einzubeziehen 6.3304544982198013e-06 -▁insp 6.3304544982198013e-06 -▁vernachläss 6.3304544982198013e-06 -KS 6.3266660754201485e-06 -gerecht 6.3266660754201485e-06 -▁Haust 6.3266660754201485e-06 -▁Koordination 6.3266660754201485e-06 -▁Meanwhile 6.3266660754201485e-06 -▁compreh 6.3266660754201485e-06 -▁heil 6.3266660754201485e-06 -▁längere 6.3266660754201485e-06 -▁portugiesischen 6.3266660754201485e-06 -▁Riv 6.3228776526204956e-06 -▁Temperaturen 6.3228776526204956e-06 -▁migrants 6.3228776526204956e-06 -▁spezif 6.3228776526204956e-06 -▁wofür 6.3228776526204956e-06 -▁überwiegend 6.3228776526204956e-06 --Mar 6.3190892298208427e-06 -osph 6.3190892298208427e-06 -▁Winds 6.3190892298208427e-06 -▁adjusted 6.3190892298208427e-06 -▁hide 6.3190892298208427e-06 -▁völl 6.3190892298208427e-06 -iesslich 6.31530080702119e-06 -spflicht 6.31530080702119e-06 -trol 6.31530080702119e-06 -▁bewundern 6.31530080702119e-06 -▁failing 6.31530080702119e-06 -▁lebenden 6.31530080702119e-06 -▁securities 6.31530080702119e-06 -igende 6.3115123842215369e-06 -ärer 6.3115123842215369e-06 -ünde 6.3115123842215369e-06 -▁Demokrat 6.3115123842215369e-06 -▁satisf 6.3115123842215369e-06 -▁schönes 6.3115123842215369e-06 -▁warranty 6.3115123842215369e-06 -afr 6.3077239614218848e-06 -iefe 6.3077239614218848e-06 -mir 6.3077239614218848e-06 -ultural 6.3077239614218848e-06 -öhn 6.3077239614218848e-06 -üsten 6.3077239614218848e-06 -▁Child 6.3077239614218848e-06 -▁Hell 6.3077239614218848e-06 -▁Lied 6.3077239614218848e-06 -▁Norwegen 6.3077239614218848e-06 -▁glücklich 6.3077239614218848e-06 -▁tape 6.3077239614218848e-06 --Ad 6.303935538622232e-06 -dex 6.303935538622232e-06 -ifel 6.303935538622232e-06 -▁Doppelbett 6.303935538622232e-06 -▁friedlichen 6.303935538622232e-06 -▁sicherer 6.303935538622232e-06 -emann 6.3001471158225791e-06 -isi 6.3001471158225791e-06 -kf 6.3001471158225791e-06 -umpen 6.3001471158225791e-06 -▁1500 6.3001471158225791e-06 -▁backing 6.3001471158225791e-06 -▁eingereichten 6.3001471158225791e-06 -igence 6.2963586930229262e-06 -upl 6.2963586930229262e-06 -▁CPU 6.2963586930229262e-06 -▁Impro 6.2963586930229262e-06 -▁air-conditioned 6.2963586930229262e-06 -▁auszuüben 6.2963586930229262e-06 -▁lifestyle 6.2963586930229262e-06 -figuration 6.2925702702232733e-06 -ässigen 6.2925702702232733e-06 -▁befürworte 6.2925702702232733e-06 -▁congratulations 6.2925702702232733e-06 -▁explicit 6.2925702702232733e-06 -▁roles 6.2925702702232733e-06 -▁verankert 6.2925702702232733e-06 -assage 6.28878184742362e-06 -ernobyl 6.28878184742362e-06 -euer 6.28878184742362e-06 -gerufen 6.28878184742362e-06 -größe 6.28878184742362e-06 -ierender 6.28878184742362e-06 -mt 6.28878184742362e-06 -orge 6.28878184742362e-06 -uum 6.28878184742362e-06 -▁1981 6.28878184742362e-06 -▁Sowjetunion 6.28878184742362e-06 -▁Untersuchungs 6.28878184742362e-06 -▁fifteen 6.28878184742362e-06 -▁instit 6.28878184742362e-06 -▁krimin 6.28878184742362e-06 -ción 6.2849934246239675e-06 -mond 6.2849934246239675e-06 -▁Berichterstattern 6.2849934246239675e-06 -▁Caribbean 6.2849934246239675e-06 -▁Divers 6.2849934246239675e-06 -▁Felsen 6.2849934246239675e-06 -▁Haushaltsplans 6.2849934246239675e-06 -▁Henry 6.2849934246239675e-06 -▁abolition 6.2849934246239675e-06 -▁beitreten 6.2849934246239675e-06 -▁beseitigt 6.2849934246239675e-06 -▁consolidation 6.2849934246239675e-06 -▁einzelstaatlichen 6.2849934246239675e-06 -▁schlimmer 6.2849934246239675e-06 -griff 6.2812050018243146e-06 -▁Allein 6.2812050018243146e-06 -▁Blo 6.2812050018243146e-06 -▁Gefangenen 6.2812050018243146e-06 -▁Sus 6.2812050018243146e-06 -▁demonstration 6.2812050018243146e-06 -▁einzigartig 6.2812050018243146e-06 -▁strive 6.2812050018243146e-06 -fit 6.2774165790246626e-06 -üch 6.2774165790246626e-06 -▁Fragestunde 6.2774165790246626e-06 -▁Granada 6.2774165790246626e-06 -▁Kurse 6.2774165790246626e-06 -▁Singapore 6.2774165790246626e-06 -▁Strecken 6.2774165790246626e-06 -▁contributes 6.2774165790246626e-06 -▁greifen 6.2774165790246626e-06 -▁skill 6.2774165790246626e-06 -PL 6.27362815622501e-06 -def 6.27362815622501e-06 -horn 6.27362815622501e-06 -strom 6.27362815622501e-06 -term 6.27362815622501e-06 -▁87 6.27362815622501e-06 -▁Gründer 6.27362815622501e-06 -▁Rechtsrahmen 6.27362815622501e-06 -▁clearer 6.27362815622501e-06 -▁noticed 6.27362815622501e-06 -▁railways 6.27362815622501e-06 -▁tangible 6.27362815622501e-06 -bles 6.2698397334253568e-06 -idity 6.2698397334253568e-06 -isa 6.2698397334253568e-06 -▁Ereignissen 6.2698397334253568e-06 -▁berufliche 6.2698397334253568e-06 -▁decreas 6.2698397334253568e-06 -▁pleas 6.2698397334253568e-06 -greifenden 6.2660513106257039e-06 -orben 6.2660513106257039e-06 -▁arrested 6.2660513106257039e-06 -▁desper 6.2660513106257039e-06 -▁economics 6.2660513106257039e-06 -▁flags 6.2660513106257039e-06 -▁obtaining 6.2660513106257039e-06 -loser 6.262262887826051e-06 -otype 6.262262887826051e-06 -▁Besonderheiten 6.262262887826051e-06 -▁Komplettpreise 6.262262887826051e-06 -▁carrier 6.262262887826051e-06 -▁cinema 6.262262887826051e-06 -▁competitors 6.262262887826051e-06 -▁demanded 6.262262887826051e-06 -▁gepr 6.262262887826051e-06 -▁Annual 6.2584744650263981e-06 -▁Monitoring 6.2584744650263981e-06 -▁Pakt 6.2584744650263981e-06 -▁Way 6.2584744650263981e-06 -▁durchsetzen 6.2584744650263981e-06 -▁refurbished 6.2584744650263981e-06 -listen 6.2546860422267452e-06 -warten 6.2546860422267452e-06 -▁Analysen 6.2546860422267452e-06 -▁Directors 6.2546860422267452e-06 -▁Fehlen 6.2546860422267452e-06 -▁Lande 6.2546860422267452e-06 -▁NT 6.2546860422267452e-06 -▁Reader 6.2546860422267452e-06 -▁S.A. 6.2508976194270923e-06 -▁SAT 6.2508976194270923e-06 -▁Schienen 6.2508976194270923e-06 -▁Zit 6.2508976194270923e-06 -▁benchmark 6.2508976194270923e-06 -▁eating 6.2508976194270923e-06 -▁influenced 6.2508976194270923e-06 -▁roughly 6.2508976194270923e-06 -ernt 6.24710919662744e-06 -gab 6.24710919662744e-06 -inas 6.24710919662744e-06 -ssektor 6.24710919662744e-06 -▁Elite 6.24710919662744e-06 -▁Randlage 6.24710919662744e-06 -▁Senior 6.24710919662744e-06 -▁Spek 6.24710919662744e-06 -▁Zusammenbruch 6.24710919662744e-06 -▁ausb 6.24710919662744e-06 -▁erfre 6.24710919662744e-06 -▁estimates 6.24710919662744e-06 -▁interven 6.24710919662744e-06 -▁rose 6.24710919662744e-06 -oh 6.2433207738277874e-06 -▁Imperial 6.2433207738277874e-06 -▁Internetseite 6.2433207738277874e-06 -▁abzielen 6.2433207738277874e-06 -▁bescheiden 6.2433207738277874e-06 -▁geg 6.2433207738277874e-06 -▁independently 6.2433207738277874e-06 -▁notification 6.2433207738277874e-06 -▁zerstören 6.2433207738277874e-06 -GH 6.2395323510281345e-06 -▁EU-Ebene 6.2395323510281345e-06 -▁doppelt 6.2395323510281345e-06 -▁fing 6.2395323510281345e-06 -▁merg 6.2395323510281345e-06 -▁personenbezogenen 6.2395323510281345e-06 -▁rum 6.2395323510281345e-06 -▁unterliegt 6.2395323510281345e-06 -▁venture 6.2395323510281345e-06 -▁verfüg 6.2395323510281345e-06 -69 6.2357439282284816e-06 -Con 6.2357439282284816e-06 -holung 6.2357439282284816e-06 -▁Hunger 6.2357439282284816e-06 -▁Pul 6.2357439282284816e-06 -▁Zwangs 6.2357439282284816e-06 -▁instant 6.2357439282284816e-06 -▁monthly 6.2357439282284816e-06 -▁schlage 6.2357439282284816e-06 -▁unlikely 6.2357439282284816e-06 -▁EVP 6.2319555054288287e-06 -▁Einfluß 6.2319555054288287e-06 -▁Pant 6.2319555054288287e-06 -▁Vi 6.2319555054288287e-06 -▁gep 6.2319555054288287e-06 -▁unreal 6.2319555054288287e-06 -eff 6.2281670826291758e-06 -fertigt 6.2281670826291758e-06 -gemein 6.2281670826291758e-06 -grenz 6.2281670826291758e-06 -iances 6.2281670826291758e-06 -rada 6.2281670826291758e-06 -schieden 6.2281670826291758e-06 -▁Komplex 6.2281670826291758e-06 -▁bor 6.2281670826291758e-06 -▁elabor 6.2281670826291758e-06 -▁trademark 6.2281670826291758e-06 --Verb 6.2243786598295229e-06 -▁bull 6.2243786598295229e-06 -▁diesmal 6.2243786598295229e-06 -▁za 6.2243786598295229e-06 -▁Bass 6.2205902370298709e-06 -▁EURO 6.2205902370298709e-06 -▁Freien 6.2205902370298709e-06 -▁Pur 6.2205902370298709e-06 -▁Reb 6.2205902370298709e-06 -▁Tatsachen 6.2205902370298709e-06 -▁distinguished 6.2205902370298709e-06 -▁distrib 6.2205902370298709e-06 -▁gleicher 6.2205902370298709e-06 -▁verstoßen 6.2205902370298709e-06 -ensystem 6.216801814230218e-06 -ützte 6.216801814230218e-06 -▁Krem 6.216801814230218e-06 -▁Much 6.216801814230218e-06 -▁Opfern 6.216801814230218e-06 -▁gekauft 6.216801814230218e-06 -▁irgendeiner 6.216801814230218e-06 -▁missed 6.216801814230218e-06 -▁permits 6.216801814230218e-06 -frau 6.2130133914305651e-06 -ometr 6.2130133914305651e-06 -▁1973 6.2130133914305651e-06 -▁Auseinandersetzung 6.2130133914305651e-06 -▁Merc 6.2130133914305651e-06 -▁Peking 6.2130133914305651e-06 -▁ambitions 6.2130133914305651e-06 -▁appart 6.2130133914305651e-06 -▁fires 6.2130133914305651e-06 -▁geopolit 6.2130133914305651e-06 -▁storm 6.2130133914305651e-06 -▁summar 6.2130133914305651e-06 -enses 6.2092249686309122e-06 -▁Bremen 6.2092249686309122e-06 -▁Hierbei 6.2092249686309122e-06 -▁Ot 6.2092249686309122e-06 -▁RS 6.2092249686309122e-06 -▁SOL 6.2092249686309122e-06 -▁Sü 6.2092249686309122e-06 -▁Theatre 6.2092249686309122e-06 -▁Transit 6.2092249686309122e-06 -▁gathered 6.2092249686309122e-06 -▁intoler 6.2092249686309122e-06 -▁reine 6.2092249686309122e-06 -▁unilateral 6.2092249686309122e-06 -▁wen 6.2092249686309122e-06 -atories 6.2054365458312593e-06 -bestimmung 6.2054365458312593e-06 -▁Did 6.2054365458312593e-06 -▁IB 6.2054365458312593e-06 -▁Split 6.2054365458312593e-06 -▁Tendenz 6.2054365458312593e-06 -▁malerischen 6.2054365458312593e-06 -▁sink 6.2054365458312593e-06 -▁vorange 6.2054365458312593e-06 -hängt 6.2016481230316064e-06 -igration 6.2016481230316064e-06 -mor 6.2016481230316064e-06 -oso 6.2016481230316064e-06 -▁bonds 6.2016481230316064e-06 -adi 6.1978597002319535e-06 -anners 6.1978597002319535e-06 -lined 6.1978597002319535e-06 -orientierte 6.1978597002319535e-06 -▁Namens 6.1978597002319535e-06 -▁forums 6.1978597002319535e-06 -▁mailing 6.1978597002319535e-06 -▁renew 6.1978597002319535e-06 -▁revol 6.1978597002319535e-06 -▁stehende 6.1978597002319535e-06 -▁venue 6.1978597002319535e-06 -▁zumal 6.1978597002319535e-06 -▁zusammenzuarbeiten 6.1978597002319535e-06 -bte 6.1940712774323006e-06 -▁1920 6.1940712774323006e-06 -▁Dekor 6.1940712774323006e-06 -▁Königs 6.1940712774323006e-06 -▁Rental 6.1940712774323006e-06 -▁Sud 6.1940712774323006e-06 -▁beabsicht 6.1940712774323006e-06 -▁besetzt 6.1940712774323006e-06 -▁capability 6.1940712774323006e-06 -▁databases 6.1940712774323006e-06 -▁errichten 6.1940712774323006e-06 -lacier 6.1902828546326486e-06 -ulationen 6.1902828546326486e-06 -winkel 6.1902828546326486e-06 -▁CAD 6.1902828546326486e-06 -▁Gegenstände 6.1902828546326486e-06 -▁Verfass 6.1902828546326486e-06 -▁ausgewogenen 6.1902828546326486e-06 -▁ax 6.1902828546326486e-06 -▁einzurichten 6.1902828546326486e-06 -▁verleiht 6.1902828546326486e-06 -ega 6.1864944318329957e-06 -html 6.1864944318329957e-06 -▁Finanzmärkte 6.1864944318329957e-06 -▁Treiber 6.1864944318329957e-06 -▁bow 6.1864944318329957e-06 -▁energies 6.1864944318329957e-06 -▁hierarch 6.1864944318329957e-06 -▁profiles 6.1864944318329957e-06 -ammen 6.1827060090333428e-06 -zustand 6.1827060090333428e-06 -▁Fundament 6.1827060090333428e-06 -▁Saf 6.1827060090333428e-06 -▁Wol 6.1827060090333428e-06 -▁ausgew 6.1827060090333428e-06 -▁authorisation 6.1827060090333428e-06 -▁employee 6.1827060090333428e-06 -▁erschienen 6.1827060090333428e-06 -▁laboratory 6.1827060090333428e-06 -▁sat 6.1827060090333428e-06 -arb 6.17891758623369e-06 -mess 6.17891758623369e-06 -ministerium 6.17891758623369e-06 -runde 6.17891758623369e-06 -▁1983 6.17891758623369e-06 -▁F. 6.17891758623369e-06 -▁Fen 6.17891758623369e-06 -▁Gelegenheiten 6.17891758623369e-06 -▁belegt 6.17891758623369e-06 -▁cope 6.17891758623369e-06 -▁inspiration 6.17891758623369e-06 -▁intim 6.17891758623369e-06 -▁nördlichen 6.17891758623369e-06 -▁produz 6.17891758623369e-06 -▁writ 6.17891758623369e-06 -bilanz 6.175129163434037e-06 -flüge 6.175129163434037e-06 -sprachigen 6.175129163434037e-06 -ünder 6.175129163434037e-06 -▁Anreiz 6.175129163434037e-06 -▁Ky 6.175129163434037e-06 -▁Tagungs 6.175129163434037e-06 -▁geladen 6.175129163434037e-06 -▁nowadays 6.175129163434037e-06 -abilities 6.1713407406343841e-06 -spiele 6.1713407406343841e-06 -▁Bah 6.1713407406343841e-06 -▁Beendigung 6.1713407406343841e-06 -▁Weltmarkt 6.1713407406343841e-06 -▁interinstitution 6.1713407406343841e-06 -▁weißen 6.1713407406343841e-06 -ettet 6.1675523178347312e-06 -fähiger 6.1675523178347312e-06 -uest 6.1675523178347312e-06 -ă 6.1675523178347312e-06 -▁89 6.1675523178347312e-06 -▁Entschließungen 6.1675523178347312e-06 -▁Ll 6.1675523178347312e-06 -▁Rich 6.1675523178347312e-06 -▁Segel 6.1675523178347312e-06 -▁investigations 6.1675523178347312e-06 -▁natur 6.1675523178347312e-06 -amerikanischen 6.1637638950350784e-06 -oba 6.1637638950350784e-06 -▁Partnerschaften 6.1637638950350784e-06 -▁Virgin 6.1637638950350784e-06 -▁deny 6.1637638950350784e-06 -▁differently 6.1637638950350784e-06 -▁palace 6.1637638950350784e-06 -▁rasche 6.1637638950350784e-06 -▁rescue 6.1637638950350784e-06 -▁stärkeren 6.1637638950350784e-06 -aining 6.1599754722354263e-06 -▁Formel 6.1599754722354263e-06 -▁Jet 6.1599754722354263e-06 -▁Kommissare 6.1599754722354263e-06 -▁Nar 6.1599754722354263e-06 -▁Stattdessen 6.1599754722354263e-06 -▁Verona 6.1599754722354263e-06 -▁aufrechterhalten 6.1599754722354263e-06 -▁jud 6.1599754722354263e-06 -▁musicians 6.1599754722354263e-06 -▁unverb 6.1599754722354263e-06 -▁vielfältige 6.1599754722354263e-06 -azione 6.1561870494357734e-06 -estern 6.1561870494357734e-06 -ißen 6.1561870494357734e-06 -▁Kloster 6.1561870494357734e-06 -▁furthermore 6.1561870494357734e-06 -▁grundlegend 6.1561870494357734e-06 -▁labels 6.1561870494357734e-06 -▁más 6.1561870494357734e-06 -▁sustain 6.1561870494357734e-06 -▁sympath 6.1561870494357734e-06 -▁vorsichtig 6.1561870494357734e-06 -osse 6.1523986266361205e-06 -▁69 6.1523986266361205e-06 -▁Erika 6.1523986266361205e-06 -▁Zahlreiche 6.1523986266361205e-06 -▁dun 6.1523986266361205e-06 -▁startet 6.1523986266361205e-06 -ls 6.1486102038364676e-06 -troll 6.1486102038364676e-06 -▁Deine 6.1486102038364676e-06 -▁Torr 6.1486102038364676e-06 -▁Verzögerungen 6.1486102038364676e-06 -▁acqu 6.1486102038364676e-06 -▁deemed 6.1486102038364676e-06 -▁japanischen 6.1486102038364676e-06 -▁stakeholders 6.1486102038364676e-06 -▁uncertain 6.1486102038364676e-06 --comp 6.1448217810368147e-06 -gelegten 6.1448217810368147e-06 -illion 6.1448217810368147e-06 -▁1974 6.1448217810368147e-06 -▁Base 6.1448217810368147e-06 -▁Estonia 6.1448217810368147e-06 -▁Industriel 6.1448217810368147e-06 -▁Mittag 6.1448217810368147e-06 -▁hectares 6.1448217810368147e-06 -▁klarer 6.1448217810368147e-06 -▁translate 6.1448217810368147e-06 -äuter 6.1410333582371619e-06 -▁3-star 6.1410333582371619e-06 -▁comforts 6.1410333582371619e-06 -▁dangers 6.1410333582371619e-06 -▁removing 6.1410333582371619e-06 -brä 6.137244935437509e-06 -lings 6.137244935437509e-06 -▁4-Sterne-Hotel 6.137244935437509e-06 -▁Ankündigung 6.137244935437509e-06 -▁Vir 6.137244935437509e-06 -▁besticht 6.137244935437509e-06 -▁gäbe 6.137244935437509e-06 -▁vierten 6.137244935437509e-06 -omer 6.1334565126378561e-06 -würdige 6.1334565126378561e-06 -▁Genf 6.1334565126378561e-06 -▁Later 6.1334565126378561e-06 -▁Rechn 6.1334565126378561e-06 -▁Sorgfalt 6.1334565126378561e-06 -▁liv 6.1334565126378561e-06 -▁preventive 6.1334565126378561e-06 -▁Ärzte 6.1334565126378561e-06 -49 6.129668089838204e-06 -uffer 6.129668089838204e-06 -▁Muss 6.129668089838204e-06 -▁steep 6.129668089838204e-06 -▁Graph 6.1258796670385511e-06 -▁Inneres 6.1258796670385511e-06 -▁Jugoslawien 6.1258796670385511e-06 -▁Princ 6.1258796670385511e-06 -▁execut 6.1258796670385511e-06 -▁tempor 6.1258796670385511e-06 -▁wünschenswert 6.1258796670385511e-06 --Ger 6.1220912442388982e-06 -icating 6.1220912442388982e-06 -▁Grab 6.1220912442388982e-06 -▁bereitzustellen 6.1220912442388982e-06 -▁mount 6.1220912442388982e-06 -▁transmitted 6.1220912442388982e-06 -.09 6.1183028214392453e-06 -ipper 6.1183028214392453e-06 -jähriger 6.1183028214392453e-06 -uly 6.1183028214392453e-06 -▁Cop 6.1183028214392453e-06 -▁authorised 6.1183028214392453e-06 -▁fruits 6.1183028214392453e-06 -▁substant 6.1183028214392453e-06 -auten 6.1145143986395925e-06 -ii 6.1145143986395925e-06 -kommens 6.1145143986395925e-06 -uktur 6.1145143986395925e-06 -▁Anzeigen 6.1145143986395925e-06 -▁Econom 6.1145143986395925e-06 -▁Freie 6.1145143986395925e-06 -▁anbietet 6.1145143986395925e-06 -▁subscribe 6.1145143986395925e-06 -infrastruktur 6.1107259758399396e-06 -lecht 6.1107259758399396e-06 -passen 6.1107259758399396e-06 -tensive 6.1107259758399396e-06 -▁Absol 6.1107259758399396e-06 -▁Arbeitsweise 6.1107259758399396e-06 -▁Delta 6.1107259758399396e-06 -▁Vorbereitungen 6.1107259758399396e-06 -▁apolog 6.1107259758399396e-06 -▁einmalige 6.1107259758399396e-06 -▁observations 6.1107259758399396e-06 -▁stringent 6.1107259758399396e-06 -▁untergraben 6.1107259758399396e-06 --Spe 6.1069375530402867e-06 -edition 6.1069375530402867e-06 -herz 6.1069375530402867e-06 -▁15th 6.1069375530402867e-06 -▁Auslands 6.1069375530402867e-06 -▁Begriffe 6.1069375530402867e-06 -▁cabin 6.1069375530402867e-06 -▁fed 6.1069375530402867e-06 -▁partially 6.1069375530402867e-06 -▁Frattini 6.1031491302406338e-06 -▁Wassers 6.1031491302406338e-06 -▁gebilligt 6.1031491302406338e-06 -▁intergovernmental 6.1031491302406338e-06 -▁kilometers 6.1031491302406338e-06 -▁kunden 6.1031491302406338e-06 -▁married 6.1031491302406338e-06 -hilfen 6.0993607074409817e-06 -iera 6.0993607074409817e-06 -mögliche 6.0993607074409817e-06 -ogo 6.0993607074409817e-06 -ption 6.0993607074409817e-06 -ström 6.0993607074409817e-06 -ussten 6.0993607074409817e-06 -weich 6.0993607074409817e-06 -zza 6.0993607074409817e-06 -▁Basil 6.0993607074409817e-06 -▁Ele 6.0993607074409817e-06 -▁Etikett 6.0993607074409817e-06 -▁Garda 6.0993607074409817e-06 -▁Mugabe 6.0993607074409817e-06 -▁Them 6.0993607074409817e-06 -▁schnelles 6.0993607074409817e-06 -merzen 6.0955722846413288e-06 -oil 6.0955722846413288e-06 -▁Abd 6.0955722846413288e-06 -▁awards 6.0955722846413288e-06 -▁kons 6.0955722846413288e-06 -▁neueste 6.0955722846413288e-06 -▁wedding 6.0955722846413288e-06 -aked 6.091783861841676e-06 -sprechend 6.091783861841676e-06 -ulierung 6.091783861841676e-06 -▁Multimedia 6.091783861841676e-06 -▁SIS 6.091783861841676e-06 -▁assessments 6.091783861841676e-06 -▁cycling 6.091783861841676e-06 -▁eur 6.091783861841676e-06 -▁freier 6.091783861841676e-06 -▁lens 6.091783861841676e-06 -▁prevail 6.091783861841676e-06 -▁sorgfält 6.091783861841676e-06 -▁vereinfacht 6.091783861841676e-06 -▁verschieben 6.091783861841676e-06 -▁vertrauen 6.091783861841676e-06 -▁winners 6.091783861841676e-06 -▁з 6.091783861841676e-06 -abstimmung 6.0879954390420231e-06 -char 6.0879954390420231e-06 -rov 6.0879954390420231e-06 -tech 6.0879954390420231e-06 -vielfalt 6.0879954390420231e-06 -▁EGF 6.0879954390420231e-06 -▁Mountains 6.0879954390420231e-06 -▁geholfen 6.0879954390420231e-06 -▁pregn 6.0879954390420231e-06 -▁speaks 6.0879954390420231e-06 -▁towels 6.0879954390420231e-06 -wirkung 6.08420701624237e-06 -▁Kommentar 6.08420701624237e-06 -▁SPA 6.08420701624237e-06 -▁Veget 6.08420701624237e-06 -▁designers 6.08420701624237e-06 -▁eg 6.08420701624237e-06 -▁gesellschaftliche 6.08420701624237e-06 -▁nutrition 6.08420701624237e-06 -▁ueber 6.08420701624237e-06 -▁zukünftig 6.08420701624237e-06 -erstoff 6.0804185934427173e-06 -usted 6.0804185934427173e-06 -▁Hollywood 6.0804185934427173e-06 -▁Tunisia 6.0804185934427173e-06 -▁ceram 6.0804185934427173e-06 -▁exquisite 6.0804185934427173e-06 -▁gefasst 6.0804185934427173e-06 -▁konstruktiven 6.0804185934427173e-06 -▁overt 6.0804185934427173e-06 -▁ris 6.0804185934427173e-06 -▁verschoben 6.0804185934427173e-06 -denk 6.0766301706430644e-06 -estro 6.0766301706430644e-06 -eur 6.0766301706430644e-06 -verbot 6.0766301706430644e-06 -ächer 6.0766301706430644e-06 -ängst 6.0766301706430644e-06 -äts 6.0766301706430644e-06 -▁Reihenfolge 6.0766301706430644e-06 -▁Return 6.0766301706430644e-06 -▁Stabilitäts- 6.0766301706430644e-06 -▁besides 6.0766301706430644e-06 -39 6.0728417478434115e-06 -offe 6.0728417478434115e-06 -rüst 6.0728417478434115e-06 -undet 6.0728417478434115e-06 -▁Implementierung 6.0728417478434115e-06 -▁Pierre 6.0728417478434115e-06 -▁Regimes 6.0728417478434115e-06 -▁capacities 6.0728417478434115e-06 -▁dominated 6.0728417478434115e-06 -▁ot 6.0728417478434115e-06 -▁utility 6.0728417478434115e-06 -artigen 6.0690533250437595e-06 -isations 6.0690533250437595e-06 -rence 6.0690533250437595e-06 -▁champions 6.0690533250437595e-06 -iences 6.0652649022441066e-06 -kehren 6.0652649022441066e-06 -vorteil 6.0652649022441066e-06 -▁Christdemokraten 6.0652649022441066e-06 -▁Straßenverkehr 6.0652649022441066e-06 -▁Wagen 6.0652649022441066e-06 -▁appreciation 6.0652649022441066e-06 -▁beliebten 6.0652649022441066e-06 -▁nützliche 6.0652649022441066e-06 -▁sick 6.0652649022441066e-06 --Tr 6.0614764794444537e-06 -writ 6.0614764794444537e-06 -▁Flüge 6.0614764794444537e-06 -▁Fotograf 6.0614764794444537e-06 -▁Konstrukt 6.0614764794444537e-06 -▁Tokyo 6.0614764794444537e-06 -▁boundaries 6.0614764794444537e-06 -▁drängen 6.0614764794444537e-06 -▁führten 6.0614764794444537e-06 -▁verschwinden 6.0614764794444537e-06 -meldungen 6.0576880566448008e-06 -oding 6.0576880566448008e-06 -▁Fix 6.0576880566448008e-06 -▁Tener 6.0576880566448008e-06 -▁Vorsitzende 6.0576880566448008e-06 -▁begreifen 6.0576880566448008e-06 -▁breites 6.0576880566448008e-06 -▁habt 6.0576880566448008e-06 -▁strengen 6.0576880566448008e-06 -▁tone 6.0576880566448008e-06 -▁vorgeschrieben 6.0576880566448008e-06 -aze 6.0538996338451479e-06 -kat 6.0538996338451479e-06 -stabilität 6.0538996338451479e-06 -zeitraum 6.0538996338451479e-06 -▁Halbinsel 6.0538996338451479e-06 -▁Innsbruck 6.0538996338451479e-06 -▁Moldova 6.0538996338451479e-06 -▁benannt 6.0538996338451479e-06 -▁catering 6.0538996338451479e-06 -▁foster 6.0538996338451479e-06 -▁frem 6.0538996338451479e-06 -▁gesunde 6.0538996338451479e-06 -▁porn 6.0538996338451479e-06 -errt 6.050111211045495e-06 -ipes 6.050111211045495e-06 -onze 6.050111211045495e-06 -quenz 6.050111211045495e-06 -▁Dial 6.050111211045495e-06 -▁ECHO 6.050111211045495e-06 -▁acted 6.050111211045495e-06 -▁harte 6.050111211045495e-06 -▁numer 6.050111211045495e-06 -▁younger 6.050111211045495e-06 -boards 6.0463227882458421e-06 -ucing 6.0463227882458421e-06 -was 6.0463227882458421e-06 -▁CN 6.0463227882458421e-06 -▁Defizit 6.0463227882458421e-06 -▁Lac 6.0463227882458421e-06 -▁RAM 6.0463227882458421e-06 -▁Rather 6.0463227882458421e-06 -▁countless 6.0463227882458421e-06 -▁junger 6.0463227882458421e-06 -▁pride 6.0463227882458421e-06 -▁spect 6.0463227882458421e-06 -▁summary 6.0463227882458421e-06 -▁zugeschnitten 6.0463227882458421e-06 -case 6.04253436544619e-06 -erous 6.04253436544619e-06 -istics 6.04253436544619e-06 -operation 6.04253436544619e-06 -partei 6.04253436544619e-06 -▁Jagd 6.04253436544619e-06 -▁Motors 6.04253436544619e-06 -▁Reichtum 6.04253436544619e-06 -▁loading 6.04253436544619e-06 -▁prol 6.04253436544619e-06 -▁supervisory 6.04253436544619e-06 -neys 6.0387459426465372e-06 -▁Bewerb 6.0387459426465372e-06 -▁Vorwand 6.0387459426465372e-06 -▁notwend 6.0387459426465372e-06 -▁promotes 6.0387459426465372e-06 -▁werte 6.0387459426465372e-06 -vet 6.0349575198468843e-06 -wohl 6.0349575198468843e-06 -xis 6.0349575198468843e-06 -öpfe 6.0349575198468843e-06 -▁Allow 6.0349575198468843e-06 -▁Between 6.0349575198468843e-06 -▁Facebook 6.0349575198468843e-06 -▁Impulse 6.0349575198468843e-06 -▁Quadrat 6.0349575198468843e-06 -▁ermitteln 6.0349575198468843e-06 -▁obl 6.0349575198468843e-06 -▁unanimous 6.0349575198468843e-06 -wesens 6.0311690970472314e-06 -▁Schwer 6.0311690970472314e-06 -▁biologischen 6.0311690970472314e-06 -▁indigenous 6.0311690970472314e-06 -ourcing 6.0273806742475785e-06 -ле 6.0273806742475785e-06 -▁Abgesehen 6.0273806742475785e-06 -▁Forst 6.0273806742475785e-06 -▁Gastfreundschaft 6.0273806742475785e-06 -▁Integrität 6.0273806742475785e-06 -▁Seeverkehr 6.0273806742475785e-06 -▁afterwards 6.0273806742475785e-06 -▁analysiert 6.0273806742475785e-06 -▁beat 6.0273806742475785e-06 -▁globalization 6.0273806742475785e-06 -▁plugin 6.0273806742475785e-06 -▁türkische 6.0273806742475785e-06 -isierungen 6.0235922514479256e-06 -rache 6.0235922514479256e-06 -▁Erfordern 6.0235922514479256e-06 -▁Folglich 6.0235922514479256e-06 -▁bedeutend 6.0235922514479256e-06 -▁belgischen 6.0235922514479256e-06 -▁feelings 6.0235922514479256e-06 -▁vineyards 6.0235922514479256e-06 -▁worauf 6.0235922514479256e-06 -KE 6.0198038286482727e-06 -arbeitungs 6.0198038286482727e-06 -days 6.0198038286482727e-06 -parks 6.0198038286482727e-06 -▁Beachten 6.0198038286482727e-06 -▁Durchschnitt 6.0198038286482727e-06 -▁Laeken 6.0198038286482727e-06 -▁Metal 6.0198038286482727e-06 -▁collections 6.0198038286482727e-06 -▁extends 6.0198038286482727e-06 -▁struck 6.0198038286482727e-06 -▁tournament 6.0198038286482727e-06 -▁verliert 6.0198038286482727e-06 -üge 6.01601540584862e-06 -▁Schalt 6.01601540584862e-06 -▁accomod 6.01601540584862e-06 -▁lect 6.01601540584862e-06 -▁margin 6.01601540584862e-06 -▁mehrerer 6.01601540584862e-06 -▁safely 6.01601540584862e-06 -amped 6.0122269830489678e-06 -artig 6.0122269830489678e-06 -bezogene 6.0122269830489678e-06 -▁Expert 6.0122269830489678e-06 -▁Golfplatz 6.0122269830489678e-06 -▁Johannes 6.0122269830489678e-06 -▁Kof 6.0122269830489678e-06 -▁Masse 6.0122269830489678e-06 -▁Mic 6.0122269830489678e-06 -▁Pel 6.0122269830489678e-06 -▁Rimini 6.0122269830489678e-06 -▁getr 6.0122269830489678e-06 -▁luxuriöse 6.0122269830489678e-06 -▁seaside 6.0122269830489678e-06 -chie 6.0084385602493149e-06 -cm 6.0084385602493149e-06 -fang 6.0084385602493149e-06 -gerechte 6.0084385602493149e-06 -ship 6.0084385602493149e-06 -untern 6.0084385602493149e-06 -yard 6.0084385602493149e-06 -▁Sing 6.0084385602493149e-06 -▁ausged 6.0084385602493149e-06 -▁consulted 6.0084385602493149e-06 -▁müsse 6.0084385602493149e-06 -▁q 6.0084385602493149e-06 -▁tiefer 6.0084385602493149e-06 -ICH 6.004650137449662e-06 -aux 6.004650137449662e-06 -bra 6.004650137449662e-06 -still 6.004650137449662e-06 -tegration 6.004650137449662e-06 -▁Appart 6.004650137449662e-06 -▁Thun 6.004650137449662e-06 -▁deck 6.004650137449662e-06 -▁deepen 6.004650137449662e-06 -▁pose 6.004650137449662e-06 -▁revers 6.004650137449662e-06 -▁sicht 6.004650137449662e-06 -▁unmittelbaren 6.004650137449662e-06 -ALL 6.0008617146500091e-06 -ahrer 6.0008617146500091e-06 -interessen 6.0008617146500091e-06 -stall 6.0008617146500091e-06 -▁Laman 6.0008617146500091e-06 -▁Mailand 6.0008617146500091e-06 -▁som 6.0008617146500091e-06 -erber 5.9970732918503562e-06 -itted 5.9970732918503562e-06 -ools 5.9970732918503562e-06 -▁Ax 5.9970732918503562e-06 -▁Jahrzehnte 5.9970732918503562e-06 -▁fotograf 5.9970732918503562e-06 -▁on-line 5.9970732918503562e-06 -onna 5.9932848690507033e-06 -▁Anim 5.9932848690507033e-06 -▁DNA 5.9932848690507033e-06 -▁Dal 5.9932848690507033e-06 -▁Hierzu 5.9932848690507033e-06 -▁Jens 5.9932848690507033e-06 -▁Leonardo 5.9932848690507033e-06 -▁Trennung 5.9932848690507033e-06 -▁Vert 5.9932848690507033e-06 -▁Webs 5.9932848690507033e-06 -▁revenues 5.9932848690507033e-06 -▁versprochen 5.9932848690507033e-06 -entral 5.9894964462510504e-06 -person 5.9894964462510504e-06 -öpfung 5.9894964462510504e-06 -▁Frei 5.9894964462510504e-06 -▁Herzeg 5.9894964462510504e-06 -▁bears 5.9894964462510504e-06 -▁dismant 5.9894964462510504e-06 -▁reaches 5.9894964462510504e-06 -▁vorne 5.9894964462510504e-06 -asses 5.9857080234513975e-06 -oca 5.9857080234513975e-06 -▁Bananen 5.9857080234513975e-06 -▁Beweise 5.9857080234513975e-06 -▁Ged 5.9857080234513975e-06 -▁Krankenhaus 5.9857080234513975e-06 -▁Memory 5.9857080234513975e-06 -▁Zwecken 5.9857080234513975e-06 -▁tausend 5.9857080234513975e-06 -iben 5.9819196006517455e-06 -ipts 5.9819196006517455e-06 -ň 5.9819196006517455e-06 -▁Care 5.9819196006517455e-06 -▁Menschenhandel 5.9819196006517455e-06 -▁lokal 5.9819196006517455e-06 -▁verknüpft 5.9819196006517455e-06 -▁vorgegeben 5.9819196006517455e-06 -adr 5.9781311778520926e-06 -gängen 5.9781311778520926e-06 -▁Co. 5.9781311778520926e-06 -▁Erleben 5.9781311778520926e-06 -▁Palast 5.9781311778520926e-06 -▁Petr 5.9781311778520926e-06 -▁Verst 5.9781311778520926e-06 -▁Zut 5.9781311778520926e-06 -▁albeit 5.9781311778520926e-06 -▁audit 5.9781311778520926e-06 -▁ausgebaut 5.9781311778520926e-06 -▁auto 5.9781311778520926e-06 -▁invested 5.9781311778520926e-06 -▁müßte 5.9781311778520926e-06 -▁ums 5.9781311778520926e-06 -▁ungarischen 5.9781311778520926e-06 -ameras 5.97434275505244e-06 -fahrung 5.97434275505244e-06 -ändigen 5.97434275505244e-06 -▁Große 5.97434275505244e-06 -▁Hochschul 5.97434275505244e-06 -ampagne 5.9705543322527868e-06 -bahnhof 5.9705543322527868e-06 -cas 5.9705543322527868e-06 -ransfer 5.9705543322527868e-06 -uent 5.9705543322527868e-06 -▁Awards 5.9705543322527868e-06 -▁Distrib 5.9705543322527868e-06 -▁Lama 5.9705543322527868e-06 -▁Ultra 5.9705543322527868e-06 -▁conviction 5.9705543322527868e-06 -zern 5.9667659094531339e-06 -▁Aufstieg 5.9667659094531339e-06 -▁Grafiken 5.9667659094531339e-06 -▁Shar 5.9667659094531339e-06 -▁Silver 5.9667659094531339e-06 -▁Südtirol 5.9667659094531339e-06 -▁defense 5.9667659094531339e-06 -▁Kompon 5.962977486653481e-06 -▁Präsidium 5.962977486653481e-06 -▁Versicher 5.962977486653481e-06 -▁weitergegeben 5.962977486653481e-06 -▁wiederher 5.962977486653481e-06 --Gipfel 5.9591890638538281e-06 -cam 5.9591890638538281e-06 -fäl 5.9591890638538281e-06 -kommission 5.9591890638538281e-06 -▁Acr 5.9591890638538281e-06 -▁Albanien 5.9591890638538281e-06 -▁Kommando 5.9591890638538281e-06 -▁Oxford 5.9591890638538281e-06 -▁abol 5.9591890638538281e-06 -▁feiern 5.9591890638538281e-06 -▁spell 5.9591890638538281e-06 -▁stir 5.9591890638538281e-06 -irn 5.9554006410541752e-06 -▁Campingplatz 5.9554006410541752e-06 -▁cup 5.9554006410541752e-06 -▁insight 5.9554006410541752e-06 -▁unterstreicht 5.9554006410541752e-06 -▁unvergesslichen 5.9554006410541752e-06 -ieger 5.9516122182545232e-06 -mung 5.9516122182545232e-06 -▁Armuts 5.9516122182545232e-06 -▁Jam 5.9516122182545232e-06 -▁Loch 5.9516122182545232e-06 -▁Sad 5.9516122182545232e-06 -▁Unions 5.9516122182545232e-06 -▁hole 5.9516122182545232e-06 --v 5.94782379545487e-06 -PO 5.94782379545487e-06 -install 5.94782379545487e-06 -▁Exhibition 5.94782379545487e-06 -▁Pract 5.94782379545487e-06 -▁climbing 5.94782379545487e-06 -▁familien 5.94782379545487e-06 -▁mehrmals 5.94782379545487e-06 -▁versetzt 5.94782379545487e-06 -otheken 5.9440353726552174e-06 -▁Benz 5.9440353726552174e-06 -▁Supervisory 5.9440353726552174e-06 -▁Toil 5.9440353726552174e-06 -▁ages 5.9440353726552174e-06 -CR 5.9402469498555645e-06 -änglichkeit 5.9402469498555645e-06 -▁ALL 5.9402469498555645e-06 -▁Burn 5.9402469498555645e-06 -▁DVDs 5.9402469498555645e-06 -▁dauerhafte 5.9402469498555645e-06 -▁ignorieren 5.9402469498555645e-06 -▁luck 5.9402469498555645e-06 -▁ties 5.9402469498555645e-06 -▁warmen 5.9402469498555645e-06 -▁weigh 5.9402469498555645e-06 -beitrag 5.9364585270559116e-06 -cement 5.9364585270559116e-06 -unn 5.9364585270559116e-06 -▁Zweif 5.9364585270559116e-06 -▁entspann 5.9364585270559116e-06 -▁motivated 5.9364585270559116e-06 -▁nachvoll 5.9364585270559116e-06 -▁query 5.9364585270559116e-06 -▁secured 5.9364585270559116e-06 -iele 5.9326701042562587e-06 -sspiel 5.9326701042562587e-06 -utreten 5.9326701042562587e-06 -öcke 5.9326701042562587e-06 -▁Letz 5.9326701042562587e-06 -▁Rindfleisch 5.9326701042562587e-06 -▁adidas 5.9326701042562587e-06 -▁anstelle 5.9326701042562587e-06 -▁dive 5.9326701042562587e-06 -▁effektiven 5.9326701042562587e-06 -▁sozusagen 5.9326701042562587e-06 -staatlichen 5.9288816814566059e-06 -ädern 5.9288816814566059e-06 -▁Regarding 5.9288816814566059e-06 -▁Simply 5.9288816814566059e-06 -▁Standpunkte 5.9288816814566059e-06 -▁dictatorship 5.9288816814566059e-06 -▁frischen 5.9288816814566059e-06 -▁reconciliation 5.9288816814566059e-06 -▁respons 5.9288816814566059e-06 -▁scenario 5.9288816814566059e-06 -▁sincerely 5.9288816814566059e-06 -▁verheer 5.9288816814566059e-06 -▁wenigsten 5.9288816814566059e-06 -oder 5.925093258656953e-06 -uda 5.925093258656953e-06 -β 5.925093258656953e-06 -▁Ped 5.925093258656953e-06 -▁geklärt 5.925093258656953e-06 -▁genetically 5.925093258656953e-06 -▁haven 5.925093258656953e-06 -▁investing 5.925093258656953e-06 -▁mould 5.925093258656953e-06 -▁spann 5.925093258656953e-06 -hou 5.9213048358573009e-06 -▁Berlusconi 5.9213048358573009e-06 -▁Exporte 5.9213048358573009e-06 -▁Run 5.9213048358573009e-06 -▁Zugleich 5.9213048358573009e-06 -▁constraints 5.9213048358573009e-06 -▁systematisch 5.9213048358573009e-06 -▁untereinander 5.9213048358573009e-06 -idal 5.917516413057648e-06 -▁Allgemeine 5.917516413057648e-06 -▁Carlos 5.917516413057648e-06 -▁bereitstellen 5.917516413057648e-06 -▁clarification 5.917516413057648e-06 -▁dreams 5.917516413057648e-06 -▁gewann 5.917516413057648e-06 -▁gezielt 5.917516413057648e-06 -fined 5.9137279902579951e-06 -ölle 5.9137279902579951e-06 -▁Diesel 5.9137279902579951e-06 -▁Effekte 5.9137279902579951e-06 -▁adj 5.9137279902579951e-06 -▁cooling 5.9137279902579951e-06 -▁orange 5.9137279902579951e-06 -▁villas 5.9137279902579951e-06 -▁wash 5.9137279902579951e-06 -agers 5.9099395674583422e-06 -▁Ausb 5.9099395674583422e-06 -▁Ludwig 5.9099395674583422e-06 -▁Vermeidung 5.9099395674583422e-06 -▁WHO 5.9099395674583422e-06 -▁Watson 5.9099395674583422e-06 -▁atom 5.9099395674583422e-06 -▁colon 5.9099395674583422e-06 -▁zutiefst 5.9099395674583422e-06 -▁ü 5.9099395674583422e-06 -emit 5.9061511446586894e-06 -erve 5.9061511446586894e-06 -füllen 5.9061511446586894e-06 -pher 5.9061511446586894e-06 -preises 5.9061511446586894e-06 -zin 5.9061511446586894e-06 -zusehen 5.9061511446586894e-06 -▁Dry 5.9061511446586894e-06 -▁Fokus 5.9061511446586894e-06 -▁Grundfreiheiten 5.9061511446586894e-06 -▁Pharma 5.9061511446586894e-06 -▁ausgearbeitet 5.9061511446586894e-06 -▁besagt 5.9061511446586894e-06 -▁in-depth 5.9061511446586894e-06 -▁seminar 5.9061511446586894e-06 -▁zunehmen 5.9061511446586894e-06 -sky 5.9023627218590365e-06 -▁Frontex 5.9023627218590365e-06 -▁Legitimität 5.9023627218590365e-06 -▁Manchmal 5.9023627218590365e-06 -▁Materials 5.9023627218590365e-06 -▁Schwier 5.9023627218590365e-06 -▁Unterhaltungs 5.9023627218590365e-06 -▁abgeb 5.9023627218590365e-06 -▁bedeutendsten 5.9023627218590365e-06 -▁friedliche 5.9023627218590365e-06 -▁konsult 5.9023627218590365e-06 -▁neutr 5.9023627218590365e-06 -▁outs 5.9023627218590365e-06 -▁radiation 5.9023627218590365e-06 -sum 5.8985742990593836e-06 -▁Forscher 5.8985742990593836e-06 -▁Mozilla 5.8985742990593836e-06 -▁Prix 5.8985742990593836e-06 -▁V. 5.8985742990593836e-06 -▁außergewöhnliche 5.8985742990593836e-06 -▁gepf 5.8985742990593836e-06 -▁landwirtschaftliche 5.8985742990593836e-06 -▁mündliche 5.8985742990593836e-06 -▁Conn 5.8947858762597307e-06 -▁Converter 5.8947858762597307e-06 -▁athl 5.8947858762597307e-06 -▁dicht 5.8947858762597307e-06 -▁discipl 5.8947858762597307e-06 -▁eligible 5.8947858762597307e-06 -▁ont 5.8947858762597307e-06 -▁scenes 5.8947858762597307e-06 -▁schwerer 5.8947858762597307e-06 -▁uphold 5.8947858762597307e-06 -▁wozu 5.8947858762597307e-06 -anwalt 5.8909974534600786e-06 -▁Advent 5.8909974534600786e-06 -▁Defin 5.8909974534600786e-06 -▁Effektivität 5.8909974534600786e-06 -▁Kamin 5.8909974534600786e-06 -▁Kluft 5.8909974534600786e-06 -▁besseres 5.8909974534600786e-06 -▁korrigieren 5.8909974534600786e-06 -▁panels 5.8909974534600786e-06 -▁separation 5.8909974534600786e-06 -▁umgekehrt 5.8909974534600786e-06 -▁vice 5.8909974534600786e-06 -phan 5.8872090306604257e-06 -rin 5.8872090306604257e-06 -stärke 5.8872090306604257e-06 -ло 5.8872090306604257e-06 -▁Chi 5.8872090306604257e-06 -▁Dringlichkeit 5.8872090306604257e-06 -▁Erreichen 5.8872090306604257e-06 -▁Leistungsfähigkeit 5.8872090306604257e-06 -▁aufre 5.8872090306604257e-06 -▁bre 5.8872090306604257e-06 -▁graph 5.8872090306604257e-06 -▁lod 5.8872090306604257e-06 --pro 5.8834206078607728e-06 -Mail 5.8834206078607728e-06 -mund 5.8834206078607728e-06 -▁Arbeitskräfte 5.8834206078607728e-06 -▁Erzeug 5.8834206078607728e-06 -▁divide 5.8834206078607728e-06 -▁flaw 5.8834206078607728e-06 -▁orth 5.8834206078607728e-06 -▁pedestrian 5.8834206078607728e-06 -▁qualities 5.8834206078607728e-06 -▁Bert 5.87963218506112e-06 -▁Dolmet 5.87963218506112e-06 -▁Hinsichtlich 5.87963218506112e-06 -▁Informationsgesellschaft 5.87963218506112e-06 -▁Kok 5.87963218506112e-06 -▁decentral 5.87963218506112e-06 -▁elegance 5.87963218506112e-06 -▁helicop 5.87963218506112e-06 -▁jahr 5.87963218506112e-06 -▁reproduction 5.87963218506112e-06 -▁stimulate 5.87963218506112e-06 -▁wunderbare 5.87963218506112e-06 --Richtlinie 5.8758437622614671e-06 -▁Cannes 5.8758437622614671e-06 -▁E-Mails 5.8758437622614671e-06 -▁Galax 5.8758437622614671e-06 -▁Spielraum 5.8758437622614671e-06 -▁destabil 5.8758437622614671e-06 -▁genügt 5.8758437622614671e-06 -▁gründlich 5.8758437622614671e-06 -▁häufiger 5.8758437622614671e-06 -▁noisy 5.8758437622614671e-06 -▁realised 5.8758437622614671e-06 --bit 5.8720553394618142e-06 -mittag 5.8720553394618142e-06 -▁Lieblings 5.8720553394618142e-06 -▁Luftverkehrs 5.8720553394618142e-06 -▁Tir 5.8720553394618142e-06 -▁Vorgang 5.8720553394618142e-06 -▁Wahrscheinlichkeit 5.8720553394618142e-06 -▁bitter 5.8720553394618142e-06 -▁explanations 5.8720553394618142e-06 -▁grüne 5.8720553394618142e-06 -▁sac 5.8720553394618142e-06 -▁sta 5.8720553394618142e-06 -MB 5.8682669166621613e-06 -defin 5.8682669166621613e-06 -spolitischen 5.8682669166621613e-06 -systemen 5.8682669166621613e-06 -▁ASEAN 5.8682669166621613e-06 -▁Drive 5.8682669166621613e-06 -▁E. 5.8682669166621613e-06 -▁Krak 5.8682669166621613e-06 -▁currencies 5.8682669166621613e-06 -▁delayed 5.8682669166621613e-06 -▁gewählte 5.8682669166621613e-06 -▁suspension 5.8682669166621613e-06 -▁umf 5.8682669166621613e-06 -zens 5.8644784938625092e-06 -▁Energieversorgung 5.8644784938625092e-06 -▁Ganzes 5.8644784938625092e-06 -▁Lange 5.8644784938625092e-06 -▁damaging 5.8644784938625092e-06 -▁finishing 5.8644784938625092e-06 -▁justification 5.8644784938625092e-06 -▁somewhere 5.8644784938625092e-06 -77 5.8606900710628563e-06 -acity 5.8606900710628563e-06 -institutionen 5.8606900710628563e-06 -oppelt 5.8606900710628563e-06 -präsident 5.8606900710628563e-06 -rost 5.8606900710628563e-06 -▁Monopol 5.8606900710628563e-06 -▁Paradies 5.8606900710628563e-06 -▁durchgesetzt 5.8606900710628563e-06 -▁undermin 5.8606900710628563e-06 -▁wea 5.8606900710628563e-06 -▁Everest 5.8569016482632035e-06 -▁Label 5.8569016482632035e-06 -▁Zulassung 5.8569016482632035e-06 -▁coalition 5.8569016482632035e-06 -▁dogs 5.8569016482632035e-06 -▁leit 5.8569016482632035e-06 -▁washing 5.8569016482632035e-06 -mittlung 5.8531132254635506e-06 -ungsbereich 5.8531132254635506e-06 -▁Appro 5.8531132254635506e-06 -▁Impf 5.8531132254635506e-06 -▁Kopie 5.8531132254635506e-06 -▁Mitwirkung 5.8531132254635506e-06 -▁Nachbarschaft 5.8531132254635506e-06 -▁Plant 5.8531132254635506e-06 -▁US-Dollar 5.8531132254635506e-06 -▁effizienten 5.8531132254635506e-06 -▁vielfach 5.8531132254635506e-06 -ku 5.8493248026638977e-06 -quipped 5.8493248026638977e-06 -ustainable 5.8493248026638977e-06 -▁Alex 5.8493248026638977e-06 -▁Bolkestein 5.8493248026638977e-06 -▁Geschichten 5.8493248026638977e-06 -▁Jews 5.8493248026638977e-06 -▁R. 5.8493248026638977e-06 -▁indispensable 5.8493248026638977e-06 -▁lawyers 5.8493248026638977e-06 -bietern 5.8455363798642448e-06 -▁5th 5.8455363798642448e-06 -▁Besonder 5.8455363798642448e-06 -▁Besteller 5.8455363798642448e-06 -▁Ce 5.8455363798642448e-06 -▁Songs 5.8455363798642448e-06 -▁accountability 5.8455363798642448e-06 -▁deutlicher 5.8455363798642448e-06 -▁finanzieller 5.8455363798642448e-06 -atible 5.8417479570645919e-06 -▁196 5.8417479570645919e-06 -▁Flotte 5.8417479570645919e-06 -▁Ry 5.8417479570645919e-06 -▁Slovak 5.8417479570645919e-06 -▁enjoyable 5.8417479570645919e-06 -▁hairdryer 5.8417479570645919e-06 -▁hinzuweisen 5.8417479570645919e-06 -▁vessel 5.8417479570645919e-06 -arket 5.837959534264939e-06 -fal 5.837959534264939e-06 -igrant 5.837959534264939e-06 -mos 5.837959534264939e-06 -träglich 5.837959534264939e-06 -zellen 5.837959534264939e-06 -▁Gleichbehandlung 5.837959534264939e-06 -▁Oracle 5.837959534264939e-06 -▁marriage 5.837959534264939e-06 -▁strikt 5.837959534264939e-06 -▁symbolic 5.837959534264939e-06 -▁vorangegangenen 5.837959534264939e-06 --Strategie 5.834171111465287e-06 -dis 5.834171111465287e-06 -obile 5.834171111465287e-06 -ohlen 5.834171111465287e-06 -▁Being 5.834171111465287e-06 -▁Benach 5.834171111465287e-06 -▁Brust 5.834171111465287e-06 -▁Kul 5.834171111465287e-06 -▁Pont 5.834171111465287e-06 -▁ausgeglichen 5.834171111465287e-06 -▁stabile 5.834171111465287e-06 -ample 5.8303826886656341e-06 -berto 5.8303826886656341e-06 -etto 5.8303826886656341e-06 -master 5.8303826886656341e-06 -ommer 5.8303826886656341e-06 -▁Abfälle 5.8303826886656341e-06 -▁Binnenmarkts 5.8303826886656341e-06 -▁LateRooms 5.8303826886656341e-06 -▁ausgestatteten 5.8303826886656341e-06 -▁berecht 5.8303826886656341e-06 -▁co-operation 5.8303826886656341e-06 -▁gaming 5.8303826886656341e-06 -bre 5.8265942658659812e-06 -einheiten 5.8265942658659812e-06 -folg 5.8265942658659812e-06 -hlt 5.8265942658659812e-06 -klen 5.8265942658659812e-06 -▁Core 5.8265942658659812e-06 -▁DB 5.8265942658659812e-06 -▁EU-A 5.8265942658659812e-06 -▁Nahrungs 5.8265942658659812e-06 -▁Vorgänger 5.8265942658659812e-06 -▁arriving 5.8265942658659812e-06 -▁pi 5.8265942658659812e-06 -▁qualifizierte 5.8265942658659812e-06 -▁weiterzu 5.8265942658659812e-06 -eckten 5.8228058430663283e-06 -hrer 5.8228058430663283e-06 -▁Ausstellungs 5.8228058430663283e-06 -▁Konsumenten 5.8228058430663283e-06 -▁Need 5.8228058430663283e-06 -▁Schuh 5.8228058430663283e-06 -▁Surf 5.8228058430663283e-06 -▁Tipps 5.8228058430663283e-06 -▁größtenteils 5.8228058430663283e-06 -▁infrastructures 5.8228058430663283e-06 -▁infring 5.8228058430663283e-06 -▁mort 5.8228058430663283e-06 -▁socially 5.8228058430663283e-06 -▁unglaub 5.8228058430663283e-06 -▁zahlt 5.8228058430663283e-06 -▁Dynam 5.8190174202666754e-06 -▁Tut 5.8190174202666754e-06 -▁Virus 5.8190174202666754e-06 -▁attempted 5.8190174202666754e-06 -▁bemerken 5.8190174202666754e-06 -▁kür 5.8190174202666754e-06 -▁limitations 5.8190174202666754e-06 -▁shoes 5.8190174202666754e-06 -▁wettbewerbsfähig 5.8190174202666754e-06 -KA 5.8152289974670225e-06 -bei 5.8152289974670225e-06 -iors 5.8152289974670225e-06 -▁Bob 5.8152289974670225e-06 -▁Vermögen 5.8152289974670225e-06 -▁meter 5.8152289974670225e-06 -▁vorhandene 5.8152289974670225e-06 -forder 5.81144057466737e-06 -gestellte 5.81144057466737e-06 -stuhl 5.81144057466737e-06 -▁1.1 5.81144057466737e-06 -▁Argentinien 5.81144057466737e-06 -▁Diensten 5.81144057466737e-06 -▁Differenz 5.81144057466737e-06 -▁Sett 5.81144057466737e-06 -▁commen 5.81144057466737e-06 -▁droht 5.81144057466737e-06 -▁flour 5.81144057466737e-06 -▁gefolgt 5.81144057466737e-06 -▁metall 5.81144057466737e-06 -▁zweimal 5.81144057466737e-06 -asing 5.8076521518677167e-06 -▁Ank 5.8076521518677167e-06 -▁Jahresbericht 5.8076521518677167e-06 -▁Tony 5.8076521518677167e-06 -▁Trop 5.8076521518677167e-06 -▁designated 5.8076521518677167e-06 -▁dram 5.8076521518677167e-06 -▁earned 5.8076521518677167e-06 -▁fehlende 5.8076521518677167e-06 -▁gleichberecht 5.8076521518677167e-06 -▁journal 5.8076521518677167e-06 -▁optimalen 5.8076521518677167e-06 -▁tit 5.8076521518677167e-06 -kolle 5.8038637290680647e-06 -ough 5.8038637290680647e-06 -ubar 5.8038637290680647e-06 -▁Elementen 5.8038637290680647e-06 -▁Harry 5.8038637290680647e-06 -▁Hind 5.8038637290680647e-06 -▁Neuigkeiten 5.8038637290680647e-06 -▁Sit 5.8038637290680647e-06 -▁Statt 5.8038637290680647e-06 -▁hypoc 5.8038637290680647e-06 -▁kommerzielle 5.8038637290680647e-06 -äut 5.8000753062684118e-06 -▁Interface 5.8000753062684118e-06 -▁Wurzeln 5.8000753062684118e-06 -▁analyt 5.8000753062684118e-06 -▁delivering 5.8000753062684118e-06 -▁flower 5.8000753062684118e-06 -▁inevitable 5.8000753062684118e-06 -▁opponents 5.8000753062684118e-06 -▁printer 5.8000753062684118e-06 -arium 5.7962868834687589e-06 -reet 5.7962868834687589e-06 -▁Aufträge 5.7962868834687589e-06 -▁Persönlichkeit 5.7962868834687589e-06 -▁Prü 5.7962868834687589e-06 -▁Quad 5.7962868834687589e-06 -▁Simbabwe 5.7962868834687589e-06 -▁Stift 5.7962868834687589e-06 -▁await 5.7962868834687589e-06 -▁curt 5.7962868834687589e-06 -▁tier 5.7962868834687589e-06 -▁veröffentlichten 5.7962868834687589e-06 -bur 5.792498460669106e-06 -gelt 5.792498460669106e-06 -idents 5.792498460669106e-06 -▁Christians 5.792498460669106e-06 -▁Informations- 5.792498460669106e-06 -▁Unionsbürger 5.792498460669106e-06 -▁greet 5.792498460669106e-06 -▁hinterlassen 5.792498460669106e-06 -▁rechtlich 5.792498460669106e-06 -roll 5.7887100378694531e-06 -▁Audi 5.7887100378694531e-06 -▁Ernennung 5.7887100378694531e-06 -▁Mitentscheid 5.7887100378694531e-06 -▁Oh 5.7887100378694531e-06 -▁embark 5.7887100378694531e-06 -▁invites 5.7887100378694531e-06 -banken 5.7849216150698e-06 -ijn 5.7849216150698e-06 -▁Cors 5.7849216150698e-06 -▁Industries 5.7849216150698e-06 -▁Zielsetzung 5.7849216150698e-06 -▁actor 5.7849216150698e-06 -▁avoiding 5.7849216150698e-06 -▁featured 5.7849216150698e-06 -▁sed 5.7849216150698e-06 -ansprüche 5.7811331922701473e-06 -establ 5.7811331922701473e-06 -unal 5.7811331922701473e-06 -▁zulässig 5.7811331922701473e-06 -▁Übereinkommens 5.7811331922701473e-06 --Abkommen 5.7773447694704944e-06 -.C. 5.7773447694704944e-06 -action 5.7773447694704944e-06 -▁Graz 5.7773447694704944e-06 -▁Kaffe 5.7773447694704944e-06 -▁Möchten 5.7773447694704944e-06 -▁Unlike 5.7773447694704944e-06 -▁Verg 5.7773447694704944e-06 -▁advocate 5.7773447694704944e-06 -▁belieb 5.7773447694704944e-06 -▁bol 5.7773447694704944e-06 -▁komm 5.7773447694704944e-06 -▁menus 5.7773447694704944e-06 -▁quantitative 5.7773447694704944e-06 -erren 5.7735563466708424e-06 -proof 5.7735563466708424e-06 -veranstaltungen 5.7735563466708424e-06 -▁4th 5.7735563466708424e-06 -▁Aussichten 5.7735563466708424e-06 -▁Brown 5.7735563466708424e-06 -▁Defizite 5.7735563466708424e-06 -▁Leder 5.7735563466708424e-06 -▁Organen 5.7735563466708424e-06 -iegeln 5.7697679238711895e-06 -ić 5.7697679238711895e-06 -ounced 5.7697679238711895e-06 -ynam 5.7697679238711895e-06 -▁2.4 5.7697679238711895e-06 -▁Ratspräsidenten 5.7697679238711895e-06 -▁Texas 5.7697679238711895e-06 -▁Tunesien 5.7697679238711895e-06 -▁epidem 5.7697679238711895e-06 -▁hilfsbereit 5.7697679238711895e-06 -▁kompatibel 5.7697679238711895e-06 -▁qualit 5.7697679238711895e-06 -▁wasn 5.7697679238711895e-06 -nutzung 5.7659795010715366e-06 -uropas 5.7659795010715366e-06 -works 5.7659795010715366e-06 -▁Bangkok 5.7659795010715366e-06 -▁Brother 5.7659795010715366e-06 -▁TU 5.7659795010715366e-06 -▁Titan 5.7659795010715366e-06 -▁dumping 5.7659795010715366e-06 -▁proportional 5.7659795010715366e-06 -▁subscription 5.7659795010715366e-06 -▁wiederhol 5.7659795010715366e-06 -ground 5.7621910782718837e-06 -kamera 5.7621910782718837e-06 -verhältnisse 5.7621910782718837e-06 -▁Alarm 5.7621910782718837e-06 -▁Stellenwert 5.7621910782718837e-06 -▁coherence 5.7621910782718837e-06 -▁einseit 5.7621910782718837e-06 -▁erwe 5.7621910782718837e-06 -▁getestet 5.7621910782718837e-06 -▁household 5.7621910782718837e-06 -▁umstritten 5.7621910782718837e-06 -ARE 5.7584026554722308e-06 -iability 5.7584026554722308e-06 -konvention 5.7584026554722308e-06 -life 5.7584026554722308e-06 -▁Annex 5.7584026554722308e-06 -▁Barnier 5.7584026554722308e-06 -▁anger 5.7584026554722308e-06 -▁beautifully 5.7584026554722308e-06 -▁diam 5.7584026554722308e-06 -▁ehr 5.7584026554722308e-06 -▁umfassendes 5.7584026554722308e-06 -fristig 5.7546142326725779e-06 -laimed 5.7546142326725779e-06 -treten 5.7546142326725779e-06 -▁Anbau 5.7546142326725779e-06 -▁Copy 5.7546142326725779e-06 -▁Festung 5.7546142326725779e-06 -▁Large 5.7546142326725779e-06 -▁Territorium 5.7546142326725779e-06 -▁beeindruck 5.7546142326725779e-06 -▁controller 5.7546142326725779e-06 -▁herein 5.7546142326725779e-06 -▁Boutique 5.750825809872925e-06 -▁Parlamenten 5.750825809872925e-06 -▁Statut 5.750825809872925e-06 -▁Verkehrsmittel 5.750825809872925e-06 -▁outlined 5.750825809872925e-06 -▁promising 5.750825809872925e-06 -▁racing 5.750825809872925e-06 -chens 5.7470373870732721e-06 -ebene 5.7470373870732721e-06 -ez 5.7470373870732721e-06 -link 5.7470373870732721e-06 -▁Ausnahmeregel 5.7470373870732721e-06 -▁Johannesburg 5.7470373870732721e-06 -▁contr 5.7470373870732721e-06 -adet 5.74324896427362e-06 -inson 5.74324896427362e-06 -machen 5.74324896427362e-06 -mont 5.74324896427362e-06 -▁-B 5.74324896427362e-06 -▁1.4 5.74324896427362e-06 -▁1977 5.74324896427362e-06 -▁Aur 5.74324896427362e-06 -▁CD-ROM 5.74324896427362e-06 -▁Forsch 5.74324896427362e-06 -▁Lü 5.74324896427362e-06 -▁Rov 5.74324896427362e-06 -▁Zellen 5.74324896427362e-06 -▁episode 5.74324896427362e-06 -▁gewerb 5.74324896427362e-06 -▁hopefully 5.74324896427362e-06 -▁multilingual 5.74324896427362e-06 -ieso 5.7394605414739672e-06 -strecken 5.7394605414739672e-06 -weiterung 5.7394605414739672e-06 -▁Finanziellen 5.7394605414739672e-06 -▁Pfad 5.7394605414739672e-06 -▁acr 5.7394605414739672e-06 -▁einzustellen 5.7394605414739672e-06 -▁handed 5.7394605414739672e-06 -.05 5.7356721186743143e-06 -▁84 5.7356721186743143e-06 -▁Autonomie 5.7356721186743143e-06 -▁Spezif 5.7356721186743143e-06 -▁Zuerst 5.7356721186743143e-06 -▁anmelden 5.7356721186743143e-06 -▁broadcast 5.7356721186743143e-06 -▁impetus 5.7356721186743143e-06 -▁reun 5.7356721186743143e-06 -▁wunderschön 5.7356721186743143e-06 -IND 5.7318836958746614e-06 -ideo 5.7318836958746614e-06 -prim 5.7318836958746614e-06 -stunde 5.7318836958746614e-06 -▁allies 5.7318836958746614e-06 -▁belegen 5.7318836958746614e-06 -▁bust 5.7318836958746614e-06 -▁getreten 5.7318836958746614e-06 -▁misunder 5.7318836958746614e-06 -▁verwirklicht 5.7318836958746614e-06 -aller 5.7280952730750085e-06 -ussy 5.7280952730750085e-06 -▁Basic 5.7280952730750085e-06 -▁Musiker 5.7280952730750085e-06 -▁Stränden 5.7280952730750085e-06 -▁decoration 5.7280952730750085e-06 -▁laying 5.7280952730750085e-06 --Intern 5.7243068502753556e-06 -aute 5.7243068502753556e-06 -herrschaft 5.7243068502753556e-06 -▁Dolomiten 5.7243068502753556e-06 -▁Inst 5.7243068502753556e-06 -▁Rahmens 5.7243068502753556e-06 -▁eingest 5.7243068502753556e-06 -▁hunger 5.7243068502753556e-06 -▁personenbezogen 5.7243068502753556e-06 -▁realistisch 5.7243068502753556e-06 -▁vorrangig 5.7243068502753556e-06 -▁vorschlägt 5.7243068502753556e-06 -▁wahrnehmen 5.7243068502753556e-06 -auert 5.7205184274757027e-06 -bem 5.7205184274757027e-06 -geordneten 5.7205184274757027e-06 -oi 5.7205184274757027e-06 -▁Eck 5.7205184274757027e-06 -▁Font 5.7205184274757027e-06 -▁Fristen 5.7205184274757027e-06 -▁Libyen 5.7205184274757027e-06 -▁Registr 5.7205184274757027e-06 -▁blocked 5.7205184274757027e-06 -▁brilliant 5.7205184274757027e-06 -▁german 5.7205184274757027e-06 -▁leichten 5.7205184274757027e-06 -▁stimul 5.7205184274757027e-06 -etra 5.71673000467605e-06 -ivo 5.71673000467605e-06 -not 5.71673000467605e-06 -udio 5.71673000467605e-06 -▁G. 5.71673000467605e-06 -▁Grundstück 5.71673000467605e-06 -▁Lady 5.71673000467605e-06 -▁Walter 5.71673000467605e-06 -▁existed 5.71673000467605e-06 -▁legendary 5.71673000467605e-06 -▁zwangsläufig 5.71673000467605e-06 -estic 5.7129415818763978e-06 -ikation 5.7129415818763978e-06 -miet 5.7129415818763978e-06 -▁Künstlern 5.7129415818763978e-06 -▁Timor 5.7129415818763978e-06 -▁entre 5.7129415818763978e-06 -▁grave 5.7129415818763978e-06 -▁heruntergeladen 5.7129415818763978e-06 -▁paint 5.7129415818763978e-06 -▁suspect 5.7129415818763978e-06 -partnern 5.7091531590767449e-06 -schnitte 5.7091531590767449e-06 -sprech 5.7091531590767449e-06 -zonen 5.7091531590767449e-06 -▁Gard 5.7091531590767449e-06 -▁Liv 5.7091531590767449e-06 -▁Markus 5.7091531590767449e-06 -▁Mühe 5.7091531590767449e-06 -▁Save 5.7091531590767449e-06 -▁bestellt 5.7091531590767449e-06 -▁check-in 5.7091531590767449e-06 -▁disagree 5.7091531590767449e-06 -▁libraries 5.7091531590767449e-06 -abteilung 5.705364736277092e-06 -legten 5.705364736277092e-06 -leid 5.705364736277092e-06 -umn 5.705364736277092e-06 -verlust 5.705364736277092e-06 -▁Abweich 5.705364736277092e-06 -▁Application 5.705364736277092e-06 -▁NO 5.705364736277092e-06 -▁Vorsorge 5.705364736277092e-06 -▁Wahrnehmung 5.705364736277092e-06 -▁entity 5.705364736277092e-06 -▁globally 5.705364736277092e-06 --We 5.7015763134774391e-06 -istern 5.7015763134774391e-06 -ivated 5.7015763134774391e-06 -▁Einheiten 5.7015763134774391e-06 -▁Korrektur 5.7015763134774391e-06 -▁Machen 5.7015763134774391e-06 -▁Verpackung 5.7015763134774391e-06 -▁gesorgt 5.7015763134774391e-06 -▁schwarzen 5.7015763134774391e-06 -▁systematically 5.7015763134774391e-06 -▁verbieten 5.7015763134774391e-06 -ealing 5.6977878906777862e-06 -▁74 5.6977878906777862e-06 -▁Days 5.6977878906777862e-06 -▁Mare 5.6977878906777862e-06 -▁compatibility 5.6977878906777862e-06 -▁complain 5.6977878906777862e-06 -▁iP 5.6977878906777862e-06 -▁opera 5.6977878906777862e-06 -▁upgrad 5.6977878906777862e-06 -▁waves 5.6977878906777862e-06 -halter 5.6939994678781334e-06 -ifiz 5.6939994678781334e-06 -mes 5.6939994678781334e-06 -nee 5.6939994678781334e-06 -▁Commander 5.6939994678781334e-06 -▁Fried 5.6939994678781334e-06 -▁Klick 5.6939994678781334e-06 -▁Winters 5.6939994678781334e-06 -▁kid 5.6939994678781334e-06 -▁killing 5.6939994678781334e-06 -▁korrekte 5.6939994678781334e-06 -▁unm 5.6939994678781334e-06 -fix 5.6902110450784805e-06 -komplex 5.6902110450784805e-06 -ondo 5.6902110450784805e-06 -▁91 5.6902110450784805e-06 -▁Baroness 5.6902110450784805e-06 -▁Blizzard 5.6902110450784805e-06 -▁Chemikalien 5.6902110450784805e-06 -▁Corr 5.6902110450784805e-06 -▁Dörfer 5.6902110450784805e-06 -▁angest 5.6902110450784805e-06 -▁benachteil 5.6902110450784805e-06 -▁plate 5.6902110450784805e-06 -▁unterhalb 5.6902110450784805e-06 -▁Dubai 5.6864226222788276e-06 -▁Jac 5.6864226222788276e-06 -▁Welthandelsorganisation 5.6864226222788276e-06 -▁decken 5.6864226222788276e-06 -▁filters 5.6864226222788276e-06 -alität 5.6826341994791755e-06 -eded 5.6826341994791755e-06 -äler 5.6826341994791755e-06 -▁Côte 5.6826341994791755e-06 -▁alpine 5.6826341994791755e-06 -▁ausgedrückt 5.6826341994791755e-06 -▁fundamentally 5.6826341994791755e-06 -▁großartige 5.6826341994791755e-06 -▁loan 5.6826341994791755e-06 -▁robots 5.6826341994791755e-06 -▁veter 5.6826341994791755e-06 -anted 5.6788457766795226e-06 -▁Argent 5.6788457766795226e-06 -▁Funk 5.6788457766795226e-06 -▁Kühlschrank 5.6788457766795226e-06 -▁Mehrere 5.6788457766795226e-06 -▁Sämtliche 5.6788457766795226e-06 -▁Tabellen 5.6788457766795226e-06 -▁Urban 5.6788457766795226e-06 -▁slaughter 5.6788457766795226e-06 -▁ungen 5.6788457766795226e-06 -,0 5.67505735387987e-06 -with 5.67505735387987e-06 -▁Canaria 5.67505735387987e-06 -▁Channel 5.67505735387987e-06 -▁HR 5.67505735387987e-06 -▁gerufen 5.67505735387987e-06 -▁reale 5.67505735387987e-06 -▁serial 5.67505735387987e-06 -▁toil 5.67505735387987e-06 -▁universe 5.67505735387987e-06 -ausbildung 5.6712689310802169e-06 -enny 5.6712689310802169e-06 -kompetenz 5.6712689310802169e-06 -onas 5.6712689310802169e-06 -orientierten 5.6712689310802169e-06 -▁Fertigungs 5.6712689310802169e-06 -▁Fortschritts 5.6712689310802169e-06 -▁Frankreichs 5.6712689310802169e-06 -▁Mong 5.6712689310802169e-06 -▁Rail 5.6712689310802169e-06 -▁architekt 5.6712689310802169e-06 -▁delivers 5.6712689310802169e-06 -▁mad 5.6712689310802169e-06 -achts 5.667480508280564e-06 -ocus 5.667480508280564e-06 -ooking 5.667480508280564e-06 -wid 5.667480508280564e-06 -öd 5.667480508280564e-06 -▁Berechnung 5.667480508280564e-06 -▁Hunder 5.667480508280564e-06 -▁Mittel- 5.667480508280564e-06 -▁Vermarktung 5.667480508280564e-06 -▁authorized 5.667480508280564e-06 -▁beunruh 5.667480508280564e-06 -▁contradict 5.667480508280564e-06 -▁installing 5.667480508280564e-06 -▁regelmäßigen 5.667480508280564e-06 -LAN 5.6636920854809111e-06 -bevölkerung 5.6636920854809111e-06 -eats 5.6636920854809111e-06 -plattform 5.6636920854809111e-06 -schäden 5.6636920854809111e-06 -zügen 5.6636920854809111e-06 -▁Airways 5.6636920854809111e-06 -▁Barack 5.6636920854809111e-06 -▁Einladung 5.6636920854809111e-06 -▁SEO 5.6636920854809111e-06 -▁Torre 5.6636920854809111e-06 -▁clin 5.6636920854809111e-06 -▁cost-eff 5.6636920854809111e-06 -▁inevitably 5.6636920854809111e-06 -▁konvention 5.6636920854809111e-06 -▁langjährige 5.6636920854809111e-06 -▁wet 5.6636920854809111e-06 -▁zuzu 5.6636920854809111e-06 -OST 5.6599036626812582e-06 -landes 5.6599036626812582e-06 -älen 5.6599036626812582e-06 -▁erweiterte 5.6599036626812582e-06 -▁locally 5.6599036626812582e-06 -▁qualitative 5.6599036626812582e-06 -▁skilled 5.6599036626812582e-06 --Men 5.6561152398816061e-06 -artikel 5.6561152398816061e-06 -ifizierte 5.6561152398816061e-06 -▁Ausfuhr 5.6561152398816061e-06 -▁BI 5.6561152398816061e-06 -▁Chaos 5.6561152398816061e-06 -▁Einblick 5.6561152398816061e-06 -▁Gleichheit 5.6561152398816061e-06 -▁Logo 5.6561152398816061e-06 -▁Pos 5.6561152398816061e-06 -▁Spenden 5.6561152398816061e-06 -▁Tok 5.6561152398816061e-06 -▁auction 5.6561152398816061e-06 -▁schöner 5.6561152398816061e-06 -iring 5.6523268170819532e-06 -iron 5.6523268170819532e-06 -ommun 5.6523268170819532e-06 -ählte 5.6523268170819532e-06 -▁...... 5.6523268170819532e-06 -▁Crew 5.6523268170819532e-06 -▁Egal 5.6523268170819532e-06 -▁Getreide 5.6523268170819532e-06 -▁Kub 5.6523268170819532e-06 -▁Little 5.6523268170819532e-06 -▁Unserer 5.6523268170819532e-06 -▁inequality 5.6523268170819532e-06 -▁interfaces 5.6523268170819532e-06 -▁schwedische 5.6523268170819532e-06 -▁surplus 5.6523268170819532e-06 -▁vorkommen 5.6523268170819532e-06 -fahrzeug 5.6485383942823e-06 -irr 5.6485383942823e-06 -▁Landschaften 5.6485383942823e-06 -▁Sich 5.6485383942823e-06 -▁WiFi 5.6485383942823e-06 -▁antiken 5.6485383942823e-06 -▁overnight 5.6485383942823e-06 -▁uncom 5.6485383942823e-06 -▁Generalsekretär 5.6447499714826475e-06 -▁Konflikten 5.6447499714826475e-06 -▁OP 5.6447499714826475e-06 -▁attraction 5.6447499714826475e-06 -▁forming 5.6447499714826475e-06 -▁greif 5.6447499714826475e-06 -▁steckt 5.6447499714826475e-06 -▁tranquil 5.6447499714826475e-06 -dy 5.6409615486829946e-06 -zzo 5.6409615486829946e-06 -ähn 5.6409615486829946e-06 -▁Anmerkung 5.6409615486829946e-06 -▁Atlant 5.6409615486829946e-06 -▁EU-Mitglied 5.6409615486829946e-06 -▁Mozart 5.6409615486829946e-06 -▁Territor 5.6409615486829946e-06 -▁Zell 5.6409615486829946e-06 -▁lands 5.6409615486829946e-06 -aph 5.6371731258833417e-06 -ielten 5.6371731258833417e-06 -tert 5.6371731258833417e-06 -▁Automobilindustrie 5.6371731258833417e-06 -▁Bag 5.6371731258833417e-06 -▁Foren 5.6371731258833417e-06 -▁Frü 5.6371731258833417e-06 -▁Nachfolger 5.6371731258833417e-06 -▁Sicherheits- 5.6371731258833417e-06 -▁Telef 5.6371731258833417e-06 -▁abzielt 5.6371731258833417e-06 -▁gradual 5.6371731258833417e-06 -▁verglichen 5.6371731258833417e-06 -wurf 5.6333847030836888e-06 -zene 5.6333847030836888e-06 -▁Term 5.6333847030836888e-06 -▁bom 5.6333847030836888e-06 -▁calculation 5.6333847030836888e-06 -▁cigar 5.6333847030836888e-06 -▁electronics 5.6333847030836888e-06 -▁mieten 5.6333847030836888e-06 -▁owing 5.6333847030836888e-06 -▁rech 5.6333847030836888e-06 -▁tatsächliche 5.6333847030836888e-06 -anien 5.6295962802840359e-06 -een 5.6295962802840359e-06 -esty 5.6295962802840359e-06 -▁-A 5.6295962802840359e-06 -▁Auskunft 5.6295962802840359e-06 -▁Konflikts 5.6295962802840359e-06 -▁Victor 5.6295962802840359e-06 -▁anlässlich 5.6295962802840359e-06 -▁retirement 5.6295962802840359e-06 -▁subjected 5.6295962802840359e-06 -▁suspend 5.6295962802840359e-06 -▁Ethernet 5.6258078574843838e-06 -▁H. 5.6258078574843838e-06 -▁Kammer 5.6258078574843838e-06 -▁Merkel 5.6258078574843838e-06 -▁Träger 5.6258078574843838e-06 -▁anstreben 5.6258078574843838e-06 -▁carries 5.6258078574843838e-06 -▁high-speed 5.6258078574843838e-06 -▁prosper 5.6258078574843838e-06 -▁workforce 5.6258078574843838e-06 -amkeit 5.622019434684731e-06 -anta 5.622019434684731e-06 -eap 5.622019434684731e-06 -idung 5.622019434684731e-06 -ommerce 5.622019434684731e-06 -▁Abwicklung 5.622019434684731e-06 -▁Behinderungen 5.622019434684731e-06 -▁Beta 5.622019434684731e-06 -▁Sekt 5.622019434684731e-06 -▁deeper 5.622019434684731e-06 -▁gewöhnlich 5.622019434684731e-06 -▁mysq 5.622019434684731e-06 -held 5.6182310118850781e-06 -ilien 5.6182310118850781e-06 -quiries 5.6182310118850781e-06 -▁Canal 5.6182310118850781e-06 -▁Cardass 5.6182310118850781e-06 -▁Schnittstelle 5.6182310118850781e-06 -▁activated 5.6182310118850781e-06 -▁cour 5.6182310118850781e-06 -▁einzeln 5.6182310118850781e-06 -▁geteilt 5.6182310118850781e-06 -▁openly 5.6182310118850781e-06 -▁portra 5.6182310118850781e-06 -▁simplified 5.6182310118850781e-06 -▁wünscht 5.6182310118850781e-06 --Um 5.6144425890854252e-06 -▁Criminal 5.6144425890854252e-06 -▁Nachteil 5.6144425890854252e-06 -▁Solange 5.6144425890854252e-06 -▁Stimmung 5.6144425890854252e-06 -▁admission 5.6144425890854252e-06 -▁bicycle 5.6144425890854252e-06 -▁dereg 5.6144425890854252e-06 -▁herself 5.6144425890854252e-06 -▁tremendous 5.6144425890854252e-06 --Ber 5.6106541662857723e-06 -acks 5.6106541662857723e-06 -ecurity 5.6106541662857723e-06 -power 5.6106541662857723e-06 -▁Flughaf 5.6106541662857723e-06 -▁Girl 5.6106541662857723e-06 -▁batt 5.6106541662857723e-06 -▁bewirkt 5.6106541662857723e-06 -▁recogn 5.6106541662857723e-06 -echnologie 5.6068657434861194e-06 -film 5.6068657434861194e-06 -frü 5.6068657434861194e-06 -rift 5.6068657434861194e-06 -ströme 5.6068657434861194e-06 -▁Aufrechterhaltung 5.6068657434861194e-06 -▁Neuer 5.6068657434861194e-06 -▁Preis-Leist 5.6068657434861194e-06 -▁Romant 5.6068657434861194e-06 -▁Substanzen 5.6068657434861194e-06 -▁Wachstumspakt 5.6068657434861194e-06 -▁Walk 5.6068657434861194e-06 -▁env 5.6068657434861194e-06 -▁sexuellen 5.6068657434861194e-06 -▁symbols 5.6068657434861194e-06 -▁viewing 5.6068657434861194e-06 -▁ä 5.6068657434861194e-06 -▁übertr 5.6068657434861194e-06 --Serie 5.6030773206864665e-06 -bil 5.6030773206864665e-06 -fred 5.6030773206864665e-06 -osure 5.6030773206864665e-06 -ños 5.6030773206864665e-06 -▁Cho 5.6030773206864665e-06 -▁beeinträchtigen 5.6030773206864665e-06 -▁maintains 5.6030773206864665e-06 -▁prevents 5.6030773206864665e-06 -▁prom 5.6030773206864665e-06 --Regierung 5.5992888978868136e-06 -▁Reden 5.5992888978868136e-06 -▁Starfleet 5.5992888978868136e-06 -▁beantragen 5.5992888978868136e-06 -▁capitalism 5.5992888978868136e-06 -▁exceptionally 5.5992888978868136e-06 -▁ideological 5.5992888978868136e-06 -▁judges 5.5992888978868136e-06 -▁nerv 5.5992888978868136e-06 -▁parlamentarische 5.5992888978868136e-06 -▁rub 5.5992888978868136e-06 -▁solving 5.5992888978868136e-06 -▁thirty 5.5992888978868136e-06 -▁wishing 5.5992888978868136e-06 -gesetzten 5.5955004750871616e-06 -ipher 5.5955004750871616e-06 -rators 5.5955004750871616e-06 -▁Annäherung 5.5955004750871616e-06 -▁Buchstaben 5.5955004750871616e-06 -▁Erdbeben 5.5955004750871616e-06 -▁ads 5.5955004750871616e-06 -▁europ 5.5955004750871616e-06 --Systeme 5.5917120522875087e-06 -alg 5.5917120522875087e-06 -hib 5.5917120522875087e-06 -unches 5.5917120522875087e-06 -▁Catholic 5.5917120522875087e-06 -▁Environmental 5.5917120522875087e-06 -▁Leiden 5.5917120522875087e-06 -▁Vere 5.5917120522875087e-06 -▁späten 5.5917120522875087e-06 -atell 5.5879236294878558e-06 -pon 5.5879236294878558e-06 -zweck 5.5879236294878558e-06 -▁Collection 5.5879236294878558e-06 -▁MR 5.5879236294878558e-06 -▁Sofia 5.5879236294878558e-06 -▁regel 5.5879236294878558e-06 -amos 5.5841352066882029e-06 -iftung 5.5841352066882029e-06 -▁Eleganz 5.5841352066882029e-06 -▁Etagen 5.5841352066882029e-06 -▁PPE-DE-Fraktion 5.5841352066882029e-06 -▁Verhandlung 5.5841352066882029e-06 -▁procedural 5.5841352066882029e-06 -▁thumb 5.5841352066882029e-06 -▁wand 5.5841352066882029e-06 -hüt 5.58034678388855e-06 -phone 5.58034678388855e-06 -ár 5.58034678388855e-06 -▁1969 5.58034678388855e-06 -▁Workshops 5.58034678388855e-06 -▁borne 5.58034678388855e-06 -erts 5.5765583610888971e-06 -wissenschaften 5.5765583610888971e-06 -▁Entdeckung 5.5765583610888971e-06 -▁Haushaltsjahr 5.5765583610888971e-06 -▁beantragt 5.5765583610888971e-06 -▁betreibt 5.5765583610888971e-06 -▁christ 5.5765583610888971e-06 -▁complaint 5.5765583610888971e-06 -▁geschätzt 5.5765583610888971e-06 -▁provis 5.5765583610888971e-06 -▁resumed 5.5765583610888971e-06 -▁silent 5.5765583610888971e-06 -ا 5.5727699382892442e-06 -▁FM 5.5727699382892442e-06 -▁Lanka 5.5727699382892442e-06 -▁bold 5.5727699382892442e-06 -▁hôtel 5.5727699382892442e-06 -▁kos 5.5727699382892442e-06 -aspekt 5.5689815154895913e-06 -ref 5.5689815154895913e-06 -usions 5.5689815154895913e-06 -▁Bayer 5.5689815154895913e-06 -▁Cube 5.5689815154895913e-06 -▁Erdöl 5.5689815154895913e-06 -▁MAN 5.5689815154895913e-06 -▁Schwar 5.5689815154895913e-06 -▁Verbindungs 5.5689815154895913e-06 -▁appro 5.5689815154895913e-06 -▁detection 5.5689815154895913e-06 -▁geometr 5.5689815154895913e-06 -▁newest 5.5689815154895913e-06 -eper 5.5651930926899393e-06 -projekten 5.5651930926899393e-06 -▁Konvents 5.5651930926899393e-06 -▁Zap 5.5651930926899393e-06 -▁befreien 5.5651930926899393e-06 -▁begrüß 5.5651930926899393e-06 -▁refriger 5.5651930926899393e-06 -▁tube 5.5651930926899393e-06 -ATE 5.5614046698902864e-06 -antas 5.5614046698902864e-06 -obs 5.5614046698902864e-06 -ulden 5.5614046698902864e-06 -▁Chechnya 5.5614046698902864e-06 -▁Erwerb 5.5614046698902864e-06 -▁Nuklear 5.5614046698902864e-06 -▁PE 5.5614046698902864e-06 -▁Ray 5.5614046698902864e-06 -▁Vulkan 5.5614046698902864e-06 -▁herz 5.5614046698902864e-06 -▁kur 5.5614046698902864e-06 -▁ließen 5.5614046698902864e-06 -▁writer 5.5614046698902864e-06 -▁Diet 5.5576162470906335e-06 -▁GVO 5.5576162470906335e-06 -▁PVC 5.5576162470906335e-06 -▁ausführliche 5.5576162470906335e-06 -▁ausg 5.5576162470906335e-06 -▁backed 5.5576162470906335e-06 -▁horses 5.5576162470906335e-06 -▁jährliche 5.5576162470906335e-06 -▁mö 5.5576162470906335e-06 -▁persönlicher 5.5576162470906335e-06 -▁verwand 5.5576162470906335e-06 --18 5.5538278242909806e-06 -bollah 5.5538278242909806e-06 -ocks 5.5538278242909806e-06 -uliert 5.5538278242909806e-06 -σ 5.5538278242909806e-06 -▁Finn 5.5538278242909806e-06 -▁GF 5.5538278242909806e-06 -▁Sozialpolitik 5.5538278242909806e-06 -▁Volkswagen 5.5538278242909806e-06 -▁closure 5.5538278242909806e-06 -▁ranges 5.5538278242909806e-06 -formationen 5.5500394014913277e-06 -pakete 5.5500394014913277e-06 -▁Ebenfalls 5.5500394014913277e-06 -▁Gültigkeit 5.5500394014913277e-06 -▁Hilfen 5.5500394014913277e-06 -▁Quar 5.5500394014913277e-06 -▁Stabil 5.5500394014913277e-06 -▁Zeitungen 5.5500394014913277e-06 -▁abends 5.5500394014913277e-06 -▁argued 5.5500394014913277e-06 -▁vacuum 5.5500394014913277e-06 -güter 5.5462509786916748e-06 -öne 5.5462509786916748e-06 -▁MIDI 5.5462509786916748e-06 -▁Säule 5.5462509786916748e-06 -▁allmählich 5.5462509786916748e-06 -▁convincing 5.5462509786916748e-06 -▁highway 5.5462509786916748e-06 -▁latein 5.5462509786916748e-06 -▁nost 5.5462509786916748e-06 -▁selecting 5.5462509786916748e-06 -▁spark 5.5462509786916748e-06 -▁utter 5.5462509786916748e-06 -afen 5.5424625558920219e-06 -datum 5.5424625558920219e-06 -icult 5.5424625558920219e-06 -izza 5.5424625558920219e-06 -termin 5.5424625558920219e-06 -ì 5.5424625558920219e-06 -▁1.3 5.5424625558920219e-06 -▁Anzeichen 5.5424625558920219e-06 -▁Disney 5.5424625558920219e-06 -▁Haw 5.5424625558920219e-06 -▁Store 5.5424625558920219e-06 -▁benötigten 5.5424625558920219e-06 -▁donors 5.5424625558920219e-06 -▁hosting 5.5424625558920219e-06 -▁islamischen 5.5424625558920219e-06 -▁mood 5.5424625558920219e-06 -▁reviewed 5.5424625558920219e-06 -perten 5.538674133092369e-06 -zeitige 5.538674133092369e-06 -▁Edel 5.538674133092369e-06 -▁Gerne 5.538674133092369e-06 -▁Rational 5.538674133092369e-06 -▁Romanian 5.538674133092369e-06 -▁beliebtesten 5.538674133092369e-06 -▁heik 5.538674133092369e-06 -▁klares 5.538674133092369e-06 -geordnet 5.534885710292717e-06 -iture 5.534885710292717e-06 -▁Phasen 5.534885710292717e-06 -▁Sortiment 5.534885710292717e-06 -▁abuses 5.534885710292717e-06 -▁anfangen 5.534885710292717e-06 -▁assum 5.534885710292717e-06 -▁damaligen 5.534885710292717e-06 -▁resid 5.534885710292717e-06 -▁thick 5.534885710292717e-06 -▁vertrag 5.534885710292717e-06 -▁zuge 5.534885710292717e-06 -CHT 5.5310972874930641e-06 -erl 5.5310972874930641e-06 -ikon 5.5310972874930641e-06 -▁Einwände 5.5310972874930641e-06 -▁Entschädigung 5.5310972874930641e-06 -▁advances 5.5310972874930641e-06 -▁breast 5.5310972874930641e-06 -▁decrease 5.5310972874930641e-06 -▁eingeben 5.5310972874930641e-06 -▁lat 5.5310972874930641e-06 -▁meint 5.5310972874930641e-06 -▁technically 5.5310972874930641e-06 -appy 5.5273088646934112e-06 -leich 5.5273088646934112e-06 -önen 5.5273088646934112e-06 -▁Consult 5.5273088646934112e-06 -▁Myth 5.5273088646934112e-06 -▁Nothing 5.5273088646934112e-06 -▁PXI 5.5273088646934112e-06 -▁Publik 5.5273088646934112e-06 -▁Terra 5.5273088646934112e-06 -▁contem 5.5273088646934112e-06 -▁distributors 5.5273088646934112e-06 -▁on-site 5.5273088646934112e-06 -▁wise 5.5273088646934112e-06 -udi 5.5235204418937583e-06 -würd 5.5235204418937583e-06 -ño 5.5235204418937583e-06 -▁Cart 5.5235204418937583e-06 -▁Früchte 5.5235204418937583e-06 -▁Fünft 5.5235204418937583e-06 -▁Hay 5.5235204418937583e-06 -▁Leser 5.5235204418937583e-06 -▁Suchen 5.5235204418937583e-06 -▁Tex 5.5235204418937583e-06 -▁Vat 5.5235204418937583e-06 -▁abstained 5.5235204418937583e-06 -▁administer 5.5235204418937583e-06 -▁griechische 5.5235204418937583e-06 -▁künd 5.5235204418937583e-06 --Kl 5.5197320190941054e-06 -büros 5.5197320190941054e-06 -▁Bologna 5.5197320190941054e-06 -▁Florida 5.5197320190941054e-06 -▁Thea 5.5197320190941054e-06 -▁chances 5.5197320190941054e-06 -▁giant 5.5197320190941054e-06 -▁personality 5.5197320190941054e-06 -▁terraces 5.5197320190941054e-06 -▁tiefe 5.5197320190941054e-06 -▁ökonomischen 5.5197320190941054e-06 -91 5.5159435962944525e-06 -BR 5.5159435962944525e-06 -andern 5.5159435962944525e-06 -él 5.5159435962944525e-06 -▁Bestand 5.5159435962944525e-06 -▁Model 5.5159435962944525e-06 -▁Rohstoff 5.5159435962944525e-06 -▁Schiffen 5.5159435962944525e-06 -▁Statute 5.5159435962944525e-06 -▁Swoboda 5.5159435962944525e-06 -▁industrielle 5.5159435962944525e-06 -▁stoppen 5.5159435962944525e-06 -ettbewerb 5.5121551734948e-06 -host 5.5121551734948e-06 -zepte 5.5121551734948e-06 -▁Cotonou 5.5121551734948e-06 -▁Gesamtheit 5.5121551734948e-06 -▁Umstrukturierung 5.5121551734948e-06 -▁bezahl 5.5121551734948e-06 -▁mixture 5.5121551734948e-06 -▁nütz 5.5121551734948e-06 -▁origins 5.5121551734948e-06 -▁reinforced 5.5121551734948e-06 -▁sen 5.5121551734948e-06 -agte 5.5083667506951468e-06 -attung 5.5083667506951468e-06 -lim 5.5083667506951468e-06 -▁Zul 5.5083667506951468e-06 -▁aluminium 5.5083667506951468e-06 -▁ausgewogene 5.5083667506951468e-06 -▁ferry 5.5083667506951468e-06 -▁genauen 5.5083667506951468e-06 -▁hearts 5.5083667506951468e-06 -Tech 5.5045783278954947e-06 -ovina 5.5045783278954947e-06 -stechnologie 5.5045783278954947e-06 -▁Compact 5.5045783278954947e-06 -▁PCs 5.5045783278954947e-06 -▁Teilnehmern 5.5045783278954947e-06 -▁Toleranz 5.5045783278954947e-06 -▁nied 5.5045783278954947e-06 -▁topical 5.5045783278954947e-06 -abot 5.5007899050958418e-06 -erer 5.5007899050958418e-06 -▁Sehen 5.5007899050958418e-06 -▁bekam 5.5007899050958418e-06 -▁gases 5.5007899050958418e-06 -▁haftet 5.5007899050958418e-06 -▁makroökonom 5.5007899050958418e-06 -▁parent 5.5007899050958418e-06 -▁passenden 5.5007899050958418e-06 --Ste 5.4970014822961889e-06 -aufwand 5.4970014822961889e-06 -oa 5.4970014822961889e-06 -▁2.2 5.4970014822961889e-06 -▁Athens 5.4970014822961889e-06 -▁Nutzungs 5.4970014822961889e-06 -▁Organis 5.4970014822961889e-06 -▁acid 5.4970014822961889e-06 -▁ausgeübt 5.4970014822961889e-06 -▁bags 5.4970014822961889e-06 -▁basket 5.4970014822961889e-06 -▁criterion 5.4970014822961889e-06 -▁distr 5.4970014822961889e-06 -▁entities 5.4970014822961889e-06 -▁founder 5.4970014822961889e-06 -▁gewe 5.4970014822961889e-06 -▁nahmen 5.4970014822961889e-06 -▁rarely 5.4970014822961889e-06 -▁verbreiten 5.4970014822961889e-06 -aha 5.493213059496536e-06 -fleisch 5.493213059496536e-06 -oad 5.493213059496536e-06 -▁Chr. 5.493213059496536e-06 -▁EEC 5.493213059496536e-06 -▁Nick 5.493213059496536e-06 -▁Solidaritäts 5.493213059496536e-06 -▁Souven 5.493213059496536e-06 -▁furnishings 5.493213059496536e-06 -▁gelöscht 5.493213059496536e-06 -▁voices 5.493213059496536e-06 -dung 5.4894246366968831e-06 -hauses 5.4894246366968831e-06 -idor 5.4894246366968831e-06 -klausel 5.4894246366968831e-06 -▁Angl 5.4894246366968831e-06 -▁Koll 5.4894246366968831e-06 -▁Möglich 5.4894246366968831e-06 -▁Navig 5.4894246366968831e-06 -▁Tempel 5.4894246366968831e-06 -▁Treib 5.4894246366968831e-06 -▁engagement 5.4894246366968831e-06 -▁repression 5.4894246366968831e-06 -bö 5.48563621389723e-06 -fügt 5.48563621389723e-06 -handlung 5.48563621389723e-06 -igerung 5.48563621389723e-06 -lichsten 5.48563621389723e-06 -spezifische 5.48563621389723e-06 -▁appr 5.48563621389723e-06 -▁beside 5.48563621389723e-06 -▁envisaged 5.48563621389723e-06 -▁rigorous 5.48563621389723e-06 -erklärung 5.4818477910975774e-06 -▁Finanz- 5.4818477910975774e-06 -▁Höhepunkt 5.4818477910975774e-06 -▁Nachbarschaftspolitik 5.4818477910975774e-06 -▁Twitter 5.4818477910975774e-06 -▁affair 5.4818477910975774e-06 -▁bekräftigen 5.4818477910975774e-06 -▁harvest 5.4818477910975774e-06 --Design 5.4780593682979253e-06 -ibus 5.4780593682979253e-06 -oking 5.4780593682979253e-06 -▁Hut 5.4780593682979253e-06 -▁Nazi 5.4780593682979253e-06 -▁Sofern 5.4780593682979253e-06 -▁gründen 5.4780593682979253e-06 -▁koordinieren 5.4780593682979253e-06 -alerie 5.4742709454982724e-06 -coming 5.4742709454982724e-06 -staatliche 5.4742709454982724e-06 -ulichen 5.4742709454982724e-06 -▁A5-00 5.4742709454982724e-06 -▁CNS 5.4742709454982724e-06 -▁EWG 5.4742709454982724e-06 -▁Formul 5.4742709454982724e-06 -▁Kredite 5.4742709454982724e-06 -▁Nob 5.4742709454982724e-06 -▁Prior 5.4742709454982724e-06 -▁Publ 5.4742709454982724e-06 -▁Punta 5.4742709454982724e-06 -▁dänischen 5.4742709454982724e-06 -▁festgehalten 5.4742709454982724e-06 -▁liter 5.4742709454982724e-06 -▁tolle 5.4742709454982724e-06 -ernal 5.4704825226986195e-06 -▁IGC 5.4704825226986195e-06 -▁Indonesien 5.4704825226986195e-06 -▁Think 5.4704825226986195e-06 -▁Tschetschenien 5.4704825226986195e-06 -▁absorb 5.4704825226986195e-06 -▁breath 5.4704825226986195e-06 -▁gesetzlich 5.4704825226986195e-06 -▁gewachsen 5.4704825226986195e-06 -▁zog 5.4704825226986195e-06 -aku 5.4666940998989666e-06 -apital 5.4666940998989666e-06 -ered 5.4666940998989666e-06 -rücken 5.4666940998989666e-06 -▁Brot 5.4666940998989666e-06 -▁Datenbanken 5.4666940998989666e-06 -▁Inanspruchnahme 5.4666940998989666e-06 -▁Netzes 5.4666940998989666e-06 -▁Sax 5.4666940998989666e-06 -▁Ult 5.4666940998989666e-06 -▁Völkern 5.4666940998989666e-06 -▁arises 5.4666940998989666e-06 -▁beweist 5.4666940998989666e-06 -▁ruled 5.4666940998989666e-06 -▁touristischen 5.4666940998989666e-06 -▁vernünftigen 5.4666940998989666e-06 -V. 5.4629056770993137e-06 -adar 5.4629056770993137e-06 -netze 5.4629056770993137e-06 -strategien 5.4629056770993137e-06 -umfang 5.4629056770993137e-06 -▁Infos 5.4629056770993137e-06 -▁Räumen 5.4629056770993137e-06 -▁exclude 5.4629056770993137e-06 -▁outd 5.4629056770993137e-06 -▁Augenmerk 5.4591172542996609e-06 -▁Kommissionsmitglied 5.4591172542996609e-06 -▁Nichtregierungsorganisationen 5.4591172542996609e-06 -▁pump 5.4591172542996609e-06 -▁ran 5.4591172542996609e-06 -▁smoke 5.4591172542996609e-06 -▁stud 5.4591172542996609e-06 -▁troubl 5.4591172542996609e-06 -▁voor 5.4591172542996609e-06 -ti 5.455328831500008e-06 -verhandlungen 5.455328831500008e-06 -▁Lift 5.455328831500008e-06 -▁Viren 5.455328831500008e-06 -▁adverse 5.455328831500008e-06 -▁besichtigen 5.455328831500008e-06 -▁nachzudenken 5.455328831500008e-06 -▁veröffentlichte 5.455328831500008e-06 -▁vorausgesetzt 5.455328831500008e-06 -enken 5.4515404087003551e-06 -portal 5.4515404087003551e-06 -straßen 5.4515404087003551e-06 -ulieren 5.4515404087003551e-06 -▁Bike 5.4515404087003551e-06 -▁Chemie 5.4515404087003551e-06 -▁Muslims 5.4515404087003551e-06 -▁Railway 5.4515404087003551e-06 -▁Tran 5.4515404087003551e-06 -▁ft 5.4515404087003551e-06 -▁globe 5.4515404087003551e-06 -▁verbrauch 5.4515404087003551e-06 -▁RT 5.447751985900703e-06 -▁criminals 5.447751985900703e-06 -▁cycl 5.447751985900703e-06 -▁gültigen 5.447751985900703e-06 -▁predomin 5.447751985900703e-06 -▁vorge 5.447751985900703e-06 -cat 5.44396356310105e-06 -enke 5.44396356310105e-06 -öll 5.44396356310105e-06 -▁Boliv 5.44396356310105e-06 -▁Halt 5.44396356310105e-06 -▁Made 5.44396356310105e-06 -▁Sid 5.44396356310105e-06 -▁immerhin 5.44396356310105e-06 -▁optimieren 5.44396356310105e-06 -▁verstärkten 5.44396356310105e-06 -akh 5.4401751403013972e-06 -peicher 5.4401751403013972e-06 -ulich 5.4401751403013972e-06 -▁Dafürhalten 5.4401751403013972e-06 -▁Ms 5.4401751403013972e-06 -▁Signale 5.4401751403013972e-06 -▁Unix 5.4401751403013972e-06 -▁attempting 5.4401751403013972e-06 -▁steuern 5.4401751403013972e-06 -▁supranational 5.4401751403013972e-06 -▁verhe 5.4401751403013972e-06 --ad 5.4363867175017444e-06 -ruhen 5.4363867175017444e-06 -würdigen 5.4363867175017444e-06 -öhnung 5.4363867175017444e-06 -▁Fels 5.4363867175017444e-06 -▁Solidarity 5.4363867175017444e-06 -▁Users 5.4363867175017444e-06 -▁accommodations 5.4363867175017444e-06 -▁interpreted 5.4363867175017444e-06 -▁lux 5.4363867175017444e-06 -▁profitable 5.4363867175017444e-06 -▁religiöse 5.4363867175017444e-06 -gangs 5.4325982947020915e-06 -üger 5.4325982947020915e-06 -▁Geber 5.4325982947020915e-06 -▁Renov 5.4325982947020915e-06 -▁keyboard 5.4325982947020915e-06 -▁macro 5.4325982947020915e-06 -▁offset 5.4325982947020915e-06 --Gr 5.4288098719024386e-06 -FA 5.4288098719024386e-06 -bücher 5.4288098719024386e-06 -esco 5.4288098719024386e-06 -finder 5.4288098719024386e-06 -▁Infrastrukturen 5.4288098719024386e-06 -▁Turk 5.4288098719024386e-06 -▁legte 5.4288098719024386e-06 -▁schmerz 5.4288098719024386e-06 -begr 5.4250214491027857e-06 -▁Aparthotel 5.4250214491027857e-06 -▁customized 5.4250214491027857e-06 -▁einbringen 5.4250214491027857e-06 -emm 5.4212330263031328e-06 -▁Balance 5.4212330263031328e-06 -▁Konzerte 5.4212330263031328e-06 -▁SU 5.4212330263031328e-06 -▁Vergüt 5.4212330263031328e-06 -▁Vorbeh 5.4212330263031328e-06 -▁adequately 5.4212330263031328e-06 -▁distribute 5.4212330263031328e-06 -▁husband 5.4212330263031328e-06 -▁secr 5.4212330263031328e-06 -▁unber 5.4212330263031328e-06 -▁vertical 5.4212330263031328e-06 -▁vorüber 5.4212330263031328e-06 --Ind 5.4174446035034807e-06 -ibliothek 5.4174446035034807e-06 -▁Beförderung 5.4174446035034807e-06 -▁Busse 5.4174446035034807e-06 -▁Orientierung 5.4174446035034807e-06 -▁carriers 5.4174446035034807e-06 -▁clinical 5.4174446035034807e-06 -▁erreichten 5.4174446035034807e-06 -▁excuse 5.4174446035034807e-06 -▁indirectly 5.4174446035034807e-06 -▁jack 5.4174446035034807e-06 -▁matches 5.4174446035034807e-06 -▁sincere 5.4174446035034807e-06 -▁zukunfts 5.4174446035034807e-06 -▁östlichen 5.4174446035034807e-06 -ounts 5.4136561807038279e-06 -shore 5.4136561807038279e-06 -steuerung 5.4136561807038279e-06 -▁Especially 5.4136561807038279e-06 -▁Kernenergie 5.4136561807038279e-06 -▁Spielern 5.4136561807038279e-06 -▁bands 5.4136561807038279e-06 -▁fünfzehn 5.4136561807038279e-06 -▁qualifizierten 5.4136561807038279e-06 -▁schwach 5.4136561807038279e-06 -▁unzul 5.4136561807038279e-06 -▁vente 5.4136561807038279e-06 -ammelt 5.409867757904175e-06 -hu 5.409867757904175e-06 -nten 5.409867757904175e-06 -▁Jahrhunderte 5.409867757904175e-06 -▁Kompromisse 5.409867757904175e-06 -▁Stalin 5.409867757904175e-06 -▁dimin 5.409867757904175e-06 -▁erfor 5.409867757904175e-06 -▁inhalt 5.409867757904175e-06 -▁nearest 5.409867757904175e-06 -inne 5.4060793351045221e-06 -▁Ansätze 5.4060793351045221e-06 -▁Flach 5.4060793351045221e-06 -▁Gesetzen 5.4060793351045221e-06 -▁Jer 5.4060793351045221e-06 -▁bul 5.4060793351045221e-06 -▁ceremony 5.4060793351045221e-06 -▁coins 5.4060793351045221e-06 -▁gesetzliche 5.4060793351045221e-06 -▁harmonised 5.4060793351045221e-06 -▁stones 5.4060793351045221e-06 --Bar 5.4022909123048692e-06 -61 5.4022909123048692e-06 -investitionen 5.4022909123048692e-06 -▁Autob 5.4022909123048692e-06 -▁Bulgarian 5.4022909123048692e-06 -▁conscious 5.4022909123048692e-06 -▁cotton 5.4022909123048692e-06 -▁dear 5.4022909123048692e-06 -▁frühere 5.4022909123048692e-06 -reens 5.3985024895052163e-06 -verkauf 5.3985024895052163e-06 -volumen 5.3985024895052163e-06 -▁Ly 5.3985024895052163e-06 -▁Mitbürger 5.3985024895052163e-06 -▁Solarium 5.3985024895052163e-06 -▁Studies 5.3985024895052163e-06 -▁Variante 5.3985024895052163e-06 -▁resolving 5.3985024895052163e-06 --year-old 5.3947140667055634e-06 -baut 5.3947140667055634e-06 -edit 5.3947140667055634e-06 -ierendes 5.3947140667055634e-06 -ssp 5.3947140667055634e-06 -▁Dipl 5.3947140667055634e-06 -▁Graf 5.3947140667055634e-06 -▁Spiels 5.3947140667055634e-06 -▁Vielmehr 5.3947140667055634e-06 -▁anderswo 5.3947140667055634e-06 -▁beig 5.3947140667055634e-06 -▁gehörte 5.3947140667055634e-06 -▁italien 5.3947140667055634e-06 -▁pir 5.3947140667055634e-06 -orous 5.3909256439059105e-06 -ziehung 5.3909256439059105e-06 -▁Entfern 5.3909256439059105e-06 -▁Ferienhaus 5.3909256439059105e-06 -▁Messenger 5.3909256439059105e-06 -▁Officer 5.3909256439059105e-06 -▁PSE-Fraktion 5.3909256439059105e-06 -▁Pav 5.3909256439059105e-06 -▁Poll 5.3909256439059105e-06 -▁Telekommunikation 5.3909256439059105e-06 -▁Tu 5.3909256439059105e-06 -▁hazardous 5.3909256439059105e-06 -▁statute 5.3909256439059105e-06 -▁UPS 5.3871372211062585e-06 -▁engagieren 5.3871372211062585e-06 -▁erb 5.3871372211062585e-06 -▁geschmackvoll 5.3871372211062585e-06 -▁investig 5.3871372211062585e-06 -▁recy 5.3871372211062585e-06 -▁transmit 5.3871372211062585e-06 -▁verläss 5.3871372211062585e-06 -▁äuß 5.3871372211062585e-06 -grundlage 5.3833487983066056e-06 -gue 5.3833487983066056e-06 -uggl 5.3833487983066056e-06 -ég 5.3833487983066056e-06 -▁Donau 5.3833487983066056e-06 -▁Ruh 5.3833487983066056e-06 -▁asks 5.3833487983066056e-06 -▁bemerkt 5.3833487983066056e-06 -▁dependence 5.3833487983066056e-06 -▁überw 5.3833487983066056e-06 -ilfe 5.3795603755069527e-06 -izin 5.3795603755069527e-06 -otto 5.3795603755069527e-06 -▁Baus 5.3795603755069527e-06 -▁Entwicklungspolitik 5.3795603755069527e-06 -▁Eurocopter 5.3795603755069527e-06 -▁deriv 5.3795603755069527e-06 -▁elf 5.3795603755069527e-06 -▁englisch 5.3795603755069527e-06 -▁formerly 5.3795603755069527e-06 -▁weitem 5.3795603755069527e-06 -craft 5.3757719527073e-06 -ifizierten 5.3757719527073e-06 -ounded 5.3757719527073e-06 -ährungs 5.3757719527073e-06 -▁Auss 5.3757719527073e-06 -▁Schwimm 5.3757719527073e-06 -▁Upon 5.3757719527073e-06 -▁Valent 5.3757719527073e-06 -▁compr 5.3757719527073e-06 -▁esc 5.3757719527073e-06 -▁fib 5.3757719527073e-06 -▁fortgeschritten 5.3757719527073e-06 -▁instability 5.3757719527073e-06 -ervices 5.3719835299076469e-06 -essourcen 5.3719835299076469e-06 -gebir 5.3719835299076469e-06 -ington 5.3719835299076469e-06 -otr 5.3719835299076469e-06 -▁Comfort 5.3719835299076469e-06 -▁Freedom 5.3719835299076469e-06 -▁adjustments 5.3719835299076469e-06 -▁eignen 5.3719835299076469e-06 -▁petr 5.3719835299076469e-06 -▁reasonably 5.3719835299076469e-06 -▁thir 5.3719835299076469e-06 -▁vorbereiten 5.3719835299076469e-06 -▁überraschen 5.3719835299076469e-06 -LIN 5.368195107107994e-06 -effekt 5.368195107107994e-06 -imb 5.368195107107994e-06 -▁Einbindung 5.368195107107994e-06 -▁Internetseiten 5.368195107107994e-06 -▁Rechtssicherheit 5.368195107107994e-06 -▁barbar 5.368195107107994e-06 -▁hinzuzufügen 5.368195107107994e-06 -▁imperative 5.368195107107994e-06 -▁saubere 5.368195107107994e-06 -▁validity 5.368195107107994e-06 ----- 5.3644066843083411e-06 -fassen 5.3644066843083411e-06 -olved 5.3644066843083411e-06 -ál 5.3644066843083411e-06 -▁Harr 5.3644066843083411e-06 -▁Insofern 5.3644066843083411e-06 -▁Trav 5.3644066843083411e-06 -▁befür 5.3644066843083411e-06 -▁unterschied 5.3644066843083411e-06 -▁Vegas 5.3606182615086882e-06 -▁accessed 5.3606182615086882e-06 -▁breaking 5.3606182615086882e-06 -▁distant 5.3606182615086882e-06 -▁einsetzt 5.3606182615086882e-06 -▁ernannt 5.3606182615086882e-06 -▁literally 5.3606182615086882e-06 -▁stating 5.3606182615086882e-06 -▁vorzubereiten 5.3606182615086882e-06 --Port 5.3568298387090362e-06 -23 5.3568298387090362e-06 -cons 5.3568298387090362e-06 -ergy 5.3568298387090362e-06 -gal 5.3568298387090362e-06 -schul 5.3568298387090362e-06 -▁142 5.3568298387090362e-06 -▁behold 5.3568298387090362e-06 -acies 5.3530414159093833e-06 -üter 5.3530414159093833e-06 -▁Naturkatastrophen 5.3530414159093833e-06 -▁Written 5.3530414159093833e-06 -▁abilities 5.3530414159093833e-06 -▁autonomous 5.3530414159093833e-06 -▁determining 5.3530414159093833e-06 -▁erfassen 5.3530414159093833e-06 -books 5.34925299310973e-06 -deckt 5.34925299310973e-06 -optionen 5.34925299310973e-06 -reams 5.34925299310973e-06 -vermögen 5.34925299310973e-06 -▁Artikels 5.34925299310973e-06 -▁Indikatoren 5.34925299310973e-06 -▁Stream 5.34925299310973e-06 -▁Szene 5.34925299310973e-06 -▁Unless 5.34925299310973e-06 -▁Zealand 5.34925299310973e-06 -▁asp 5.34925299310973e-06 -▁congratulating 5.34925299310973e-06 -▁maßgeschneider 5.34925299310973e-06 -ernity 5.3454645703100775e-06 -ipt 5.3454645703100775e-06 -onian 5.3454645703100775e-06 -ources 5.3454645703100775e-06 -▁Passagiere 5.3454645703100775e-06 -▁Plugin 5.3454645703100775e-06 -▁Ramblas 5.3454645703100775e-06 -▁civilians 5.3454645703100775e-06 -▁theoretical 5.3454645703100775e-06 -aments 5.3416761475104246e-06 -angt 5.3416761475104246e-06 -errasse 5.3416761475104246e-06 -geist 5.3416761475104246e-06 -glichen 5.3416761475104246e-06 -lessly 5.3416761475104246e-06 -weiß 5.3416761475104246e-06 -▁Erneuerung 5.3416761475104246e-06 -▁Latvia 5.3416761475104246e-06 -▁MED 5.3416761475104246e-06 -▁Rose 5.3416761475104246e-06 -▁Terrace 5.3416761475104246e-06 -▁letztes 5.3416761475104246e-06 -▁steigenden 5.3416761475104246e-06 -otos 5.3378877247107717e-06 -▁Photoshop 5.3378877247107717e-06 -▁Production 5.3378877247107717e-06 -▁Telekommunikations 5.3378877247107717e-06 -▁dauerhaften 5.3378877247107717e-06 -▁verlängert 5.3378877247107717e-06 -▁visas 5.3378877247107717e-06 -▁vorzuschlagen 5.3378877247107717e-06 -Card 5.3340993019111188e-06 -▁Frucht 5.3340993019111188e-06 -▁MAR 5.3340993019111188e-06 -▁Steinberg 5.3340993019111188e-06 -▁TE 5.3340993019111188e-06 -▁Verfassungsvertrag 5.3340993019111188e-06 -▁fortzusetzen 5.3340993019111188e-06 -▁günstiger 5.3340993019111188e-06 -▁invoice 5.3340993019111188e-06 --sp 5.3303108791114659e-06 -enta 5.3303108791114659e-06 -erd 5.3303108791114659e-06 -ifier 5.3303108791114659e-06 -imer 5.3303108791114659e-06 -kenntnisse 5.3303108791114659e-06 -l. 5.3303108791114659e-06 -lichung 5.3303108791114659e-06 -onden 5.3303108791114659e-06 -▁Andrew 5.3303108791114659e-06 -▁Cyber 5.3303108791114659e-06 -▁ungerecht 5.3303108791114659e-06 -▁workplace 5.3303108791114659e-06 --CD 5.3265224563118139e-06 -ehens 5.3265224563118139e-06 -günst 5.3265224563118139e-06 -olic 5.3265224563118139e-06 -sprü 5.3265224563118139e-06 -▁Kreatur 5.3265224563118139e-06 -▁Stellungnahmen 5.3265224563118139e-06 -▁englische 5.3265224563118139e-06 -▁fliegen 5.3265224563118139e-06 -▁harten 5.3265224563118139e-06 -▁identifizieren 5.3265224563118139e-06 -▁layers 5.3265224563118139e-06 -▁semi- 5.3265224563118139e-06 -muster 5.322734033512161e-06 -utzten 5.322734033512161e-06 -▁Altern 5.322734033512161e-06 -▁Bundesrepublik 5.322734033512161e-06 -▁EV 5.322734033512161e-06 -▁Garant 5.322734033512161e-06 -▁disagre 5.322734033512161e-06 -▁erzählt 5.322734033512161e-06 -▁vors 5.322734033512161e-06 -ammern 5.3189456107125081e-06 -ape 5.3189456107125081e-06 -kammer 5.3189456107125081e-06 -orious 5.3189456107125081e-06 -zuhalten 5.3189456107125081e-06 -▁Demokratien 5.3189456107125081e-06 -▁Korrekt 5.3189456107125081e-06 -▁Pots 5.3189456107125081e-06 -▁linear 5.3189456107125081e-06 -▁loaded 5.3189456107125081e-06 -▁pillars 5.3189456107125081e-06 -▁restrictive 5.3189456107125081e-06 -▁samples 5.3189456107125081e-06 -alys 5.3151571879128552e-06 -bereitschaft 5.3151571879128552e-06 -eutral 5.3151571879128552e-06 -ictionary 5.3151571879128552e-06 -▁Dark 5.3151571879128552e-06 -▁Landwirtschafts 5.3151571879128552e-06 -▁analyse 5.3151571879128552e-06 -▁beacht 5.3151571879128552e-06 -▁grey 5.3151571879128552e-06 -▁precond 5.3151571879128552e-06 -▁thereof 5.3151571879128552e-06 -Sp 5.3113687651132023e-06 -industr 5.3113687651132023e-06 -rot 5.3113687651132023e-06 -ubb 5.3113687651132023e-06 -wünsche 5.3113687651132023e-06 -▁Ansprechpartner 5.3113687651132023e-06 -▁Creative 5.3113687651132023e-06 -▁Jacuzzi 5.3113687651132023e-06 -▁Mechan 5.3113687651132023e-06 -▁Rücksicht 5.3113687651132023e-06 -▁TW 5.3113687651132023e-06 -▁Zinsen 5.3113687651132023e-06 -▁hängen 5.3113687651132023e-06 -▁renovierte 5.3113687651132023e-06 -▁sensitivity 5.3113687651132023e-06 -burger 5.3075803423135494e-06 -förm 5.3075803423135494e-06 -verfolg 5.3075803423135494e-06 -ет 5.3075803423135494e-06 -▁Gesetzgeber 5.3075803423135494e-06 -▁Intelle 5.3075803423135494e-06 -▁Natura 5.3075803423135494e-06 -▁healing 5.3075803423135494e-06 -▁nachzu 5.3075803423135494e-06 -▁tries 5.3075803423135494e-06 -▁weiten 5.3075803423135494e-06 -▁woods 5.3075803423135494e-06 -undert 5.3037919195138965e-06 -▁Cru 5.3037919195138965e-06 -▁Learn 5.3037919195138965e-06 -▁Termine 5.3037919195138965e-06 -▁angle 5.3037919195138965e-06 -▁beglückwünsche 5.3037919195138965e-06 -▁blend 5.3037919195138965e-06 -▁counts 5.3037919195138965e-06 -▁tighten 5.3037919195138965e-06 -icion 5.3000034967142445e-06 -igste 5.3000034967142445e-06 -ürt 5.3000034967142445e-06 -▁Brüder 5.3000034967142445e-06 -▁Vorbehalte 5.3000034967142445e-06 -▁Whilst 5.3000034967142445e-06 -▁aufgeworfen 5.3000034967142445e-06 -▁classified 5.3000034967142445e-06 -▁counterfeit 5.3000034967142445e-06 -▁cow 5.3000034967142445e-06 -▁definitions 5.3000034967142445e-06 -▁delle 5.3000034967142445e-06 -▁eff 5.3000034967142445e-06 -▁innocent 5.3000034967142445e-06 -▁pointing 5.3000034967142445e-06 -▁scrutiny 5.3000034967142445e-06 -▁tender 5.3000034967142445e-06 -GEL 5.2962150739145916e-06 -endern 5.2962150739145916e-06 -sprodukt 5.2962150739145916e-06 -teilt 5.2962150739145916e-06 -▁Demokratisierung 5.2962150739145916e-06 -▁erkennbar 5.2962150739145916e-06 -▁mangelnde 5.2962150739145916e-06 -beihilfen 5.2924266511149387e-06 -ielt 5.2924266511149387e-06 -konfl 5.2924266511149387e-06 -reif 5.2924266511149387e-06 -ähren 5.2924266511149387e-06 -▁Fragest 5.2924266511149387e-06 -▁Interpretation 5.2924266511149387e-06 -▁Komponente 5.2924266511149387e-06 -▁Spazier 5.2924266511149387e-06 -▁readers 5.2924266511149387e-06 -inations 5.2886382283152858e-06 -ladimir 5.2886382283152858e-06 -reicher 5.2886382283152858e-06 -▁Geschäftsführer 5.2886382283152858e-06 -▁Hongkong 5.2886382283152858e-06 -▁formally 5.2886382283152858e-06 -▁koordiniert 5.2886382283152858e-06 -▁mangeln 5.2886382283152858e-06 -eline 5.2848498055156329e-06 -liefer 5.2848498055156329e-06 -▁Bayern 5.2848498055156329e-06 -▁Drucker 5.2848498055156329e-06 -▁Euroraum 5.2848498055156329e-06 -▁Kleidung 5.2848498055156329e-06 -▁Linken 5.2848498055156329e-06 -▁NOR 5.2848498055156329e-06 -▁Zwischenzeit 5.2848498055156329e-06 -▁fold 5.2848498055156329e-06 -▁fühlt 5.2848498055156329e-06 -▁jazz 5.2848498055156329e-06 -▁passive 5.2848498055156329e-06 -▁plates 5.2848498055156329e-06 -▁preference 5.2848498055156329e-06 -▁vertrieben 5.2848498055156329e-06 -▁wished 5.2848498055156329e-06 -ket 5.28106138271598e-06 -▁Newspapers 5.28106138271598e-06 -▁Rhine 5.28106138271598e-06 -▁Roy 5.28106138271598e-06 -▁abzulehnen 5.28106138271598e-06 -▁advised 5.28106138271598e-06 -▁ageing 5.28106138271598e-06 -▁cathedral 5.28106138271598e-06 -▁examining 5.28106138271598e-06 -▁festhalten 5.28106138271598e-06 -▁hind 5.28106138271598e-06 -▁machten 5.28106138271598e-06 -▁reactions 5.28106138271598e-06 -▁securing 5.28106138271598e-06 -▁tägliche 5.28106138271598e-06 -▁wiederhole 5.28106138271598e-06 -dir 5.2772729599163271e-06 -iety 5.2772729599163271e-06 -ilities 5.2772729599163271e-06 -uftrag 5.2772729599163271e-06 -▁Beat 5.2772729599163271e-06 -▁HTTP 5.2772729599163271e-06 -▁Sin 5.2772729599163271e-06 -▁admir 5.2772729599163271e-06 -▁afr 5.2772729599163271e-06 -▁interrupt 5.2772729599163271e-06 -gipfel 5.2734845371166743e-06 -igel 5.2734845371166743e-06 -stät 5.2734845371166743e-06 -▁73 5.2734845371166743e-06 -▁Eintritts 5.2734845371166743e-06 -▁Strassen 5.2734845371166743e-06 -▁beding 5.2734845371166743e-06 -▁bedürfen 5.2734845371166743e-06 -▁berührt 5.2734845371166743e-06 -▁supporters 5.2734845371166743e-06 -▁surgery 5.2734845371166743e-06 -▁unvermeid 5.2734845371166743e-06 -ITE 5.2696961143170222e-06 -aison 5.2696961143170222e-06 -ancer 5.2696961143170222e-06 -maßen 5.2696961143170222e-06 -öch 5.2696961143170222e-06 -öpft 5.2696961143170222e-06 -▁Adressen 5.2696961143170222e-06 -▁Gipfels 5.2696961143170222e-06 -▁PCI 5.2696961143170222e-06 -▁Photos 5.2696961143170222e-06 -▁Pia 5.2696961143170222e-06 -▁Posit 5.2696961143170222e-06 -▁Stress 5.2696961143170222e-06 -▁audiovis 5.2696961143170222e-06 -▁critique 5.2696961143170222e-06 -▁fulfill 5.2696961143170222e-06 -▁tonight 5.2696961143170222e-06 -erungen 5.2659076915173693e-06 -regelungen 5.2659076915173693e-06 -suche 5.2659076915173693e-06 -▁Ali 5.2659076915173693e-06 -▁Nerv 5.2659076915173693e-06 -▁Porsche 5.2659076915173693e-06 -▁Resultat 5.2659076915173693e-06 -▁certificates 5.2659076915173693e-06 -▁deaktiv 5.2659076915173693e-06 -▁diversen 5.2659076915173693e-06 -▁grat 5.2659076915173693e-06 -▁imag 5.2659076915173693e-06 -▁non-governmental 5.2659076915173693e-06 -d. 5.2621192687177164e-06 -ertal 5.2621192687177164e-06 -icism 5.2621192687177164e-06 -strecke 5.2621192687177164e-06 -▁1972 5.2621192687177164e-06 -▁Kandid 5.2621192687177164e-06 -▁Provider 5.2621192687177164e-06 -▁Technolog 5.2621192687177164e-06 -▁attraktiv 5.2621192687177164e-06 -▁aufweist 5.2621192687177164e-06 -▁distinguish 5.2621192687177164e-06 -▁seitdem 5.2621192687177164e-06 -▁unanimity 5.2621192687177164e-06 -▁zivilen 5.2621192687177164e-06 -beratung 5.2583308459180635e-06 -erzeugung 5.2583308459180635e-06 -estand 5.2583308459180635e-06 -flotte 5.2583308459180635e-06 -gles 5.2583308459180635e-06 -user 5.2583308459180635e-06 -▁Italiens 5.2583308459180635e-06 -▁Therapie 5.2583308459180635e-06 -▁Weis 5.2583308459180635e-06 -▁counterpart 5.2583308459180635e-06 -▁geachtet 5.2583308459180635e-06 -flation 5.2545424231184106e-06 -gg 5.2545424231184106e-06 -öm 5.2545424231184106e-06 -▁Ganze 5.2545424231184106e-06 -▁Gibraltar 5.2545424231184106e-06 -▁Maßstab 5.2545424231184106e-06 -▁Offiz 5.2545424231184106e-06 -▁Rettungs 5.2545424231184106e-06 -▁Trep 5.2545424231184106e-06 -▁audiovisual 5.2545424231184106e-06 -▁cha 5.2545424231184106e-06 -▁couples 5.2545424231184106e-06 -▁dahingehend 5.2545424231184106e-06 -▁shooting 5.2545424231184106e-06 -▁studying 5.2545424231184106e-06 -▁tv 5.2545424231184106e-06 -▁wertvollen 5.2545424231184106e-06 -▁wohin 5.2545424231184106e-06 -▁zuverlässige 5.2545424231184106e-06 --199 5.2507540003187578e-06 -rapp 5.2507540003187578e-06 -step 5.2507540003187578e-06 -ő 5.2507540003187578e-06 -▁Artikeln 5.2507540003187578e-06 -▁Breite 5.2507540003187578e-06 -▁Gramm 5.2507540003187578e-06 -▁Kraftstoff 5.2507540003187578e-06 -▁Pag 5.2507540003187578e-06 -▁Progn 5.2507540003187578e-06 -▁Zutaten 5.2507540003187578e-06 -▁attrib 5.2507540003187578e-06 -▁ausgestellt 5.2507540003187578e-06 -▁breaks 5.2507540003187578e-06 -▁consolidated 5.2507540003187578e-06 -▁geschlossenen 5.2507540003187578e-06 -▁transported 5.2507540003187578e-06 --down 5.2469655775191049e-06 -HT 5.2469655775191049e-06 -finanzierung 5.2469655775191049e-06 -ingo 5.2469655775191049e-06 -opoul 5.2469655775191049e-06 -wig 5.2469655775191049e-06 -▁Erstellen 5.2469655775191049e-06 -▁Kolonial 5.2469655775191049e-06 -▁Verstoß 5.2469655775191049e-06 -▁concludes 5.2469655775191049e-06 -▁interference 5.2469655775191049e-06 -▁unum 5.2469655775191049e-06 -stab 5.243177154719452e-06 -▁Gefähr 5.243177154719452e-06 -▁Hauptziel 5.243177154719452e-06 -▁Jahrzehnt 5.243177154719452e-06 -▁Privileg 5.243177154719452e-06 -▁bedauer 5.243177154719452e-06 -▁compromises 5.243177154719452e-06 -▁disastrous 5.243177154719452e-06 -▁execute 5.243177154719452e-06 -▁sav 5.243177154719452e-06 -▁variables 5.243177154719452e-06 -▁wanting 5.243177154719452e-06 -ivals 5.2393887319198e-06 -körper 5.2393887319198e-06 -nete 5.2393887319198e-06 -struction 5.2393887319198e-06 -veranst 5.2393887319198e-06 -vereinbar 5.2393887319198e-06 -▁Banglad 5.2393887319198e-06 -▁Congo 5.2393887319198e-06 -▁Ganzen 5.2393887319198e-06 -▁Secure 5.2393887319198e-06 -▁gepflegt 5.2393887319198e-06 -▁unterstrichen 5.2393887319198e-06 -amment 5.235600309120147e-06 -▁Lyon 5.235600309120147e-06 -▁P. 5.235600309120147e-06 -▁Subjekt 5.235600309120147e-06 -▁Verband 5.235600309120147e-06 -▁Whit 5.235600309120147e-06 -▁Zuf 5.235600309120147e-06 -▁comfortably 5.235600309120147e-06 -ektoren 5.2318118863204941e-06 -information 5.2318118863204941e-06 -ingu 5.2318118863204941e-06 -iple 5.2318118863204941e-06 -▁Cer 5.2318118863204941e-06 -▁Entschließungsanträge 5.2318118863204941e-06 -▁Hospital 5.2318118863204941e-06 -▁Know 5.2318118863204941e-06 -▁MM 5.2318118863204941e-06 -▁Sahara 5.2318118863204941e-06 -▁arrang 5.2318118863204941e-06 -▁außergewöhnlichen 5.2318118863204941e-06 -▁interventions 5.2318118863204941e-06 -▁preservation 5.2318118863204941e-06 -▁refugee 5.2318118863204941e-06 -▁reiche 5.2318118863204941e-06 -▁ét 5.2318118863204941e-06 -▁Adria 5.2280234635208412e-06 -▁Kapit 5.2280234635208412e-06 -▁amtierende 5.2280234635208412e-06 -▁amtierenden 5.2280234635208412e-06 -▁ausgenommen 5.2280234635208412e-06 -▁galleries 5.2280234635208412e-06 -▁geschätz 5.2280234635208412e-06 -▁moralische 5.2280234635208412e-06 -▁promot 5.2280234635208412e-06 -▁unjust 5.2280234635208412e-06 -eased 5.2242350407211884e-06 -illen 5.2242350407211884e-06 -▁Mario 5.2242350407211884e-06 -▁Verbr 5.2242350407211884e-06 -▁ausmachen 5.2242350407211884e-06 -▁bezaubern 5.2242350407211884e-06 -▁clar 5.2242350407211884e-06 -▁dün 5.2242350407211884e-06 -▁erre 5.2242350407211884e-06 -▁regr 5.2242350407211884e-06 -orship 5.2204466179215355e-06 -▁Disabled 5.2204466179215355e-06 -▁Erdgas 5.2204466179215355e-06 -▁MO 5.2204466179215355e-06 -▁hört 5.2204466179215355e-06 -▁inj 5.2204466179215355e-06 -▁russ 5.2204466179215355e-06 -▁safeguarding 5.2204466179215355e-06 -▁verbindlich 5.2204466179215355e-06 -,9 5.2166581951218826e-06 -appe 5.2166581951218826e-06 -firmen 5.2166581951218826e-06 -iffe 5.2166581951218826e-06 -▁Gegebenheiten 5.2166581951218826e-06 -▁Popular 5.2166581951218826e-06 -▁plot 5.2166581951218826e-06 -▁protests 5.2166581951218826e-06 -HA 5.21286977232223e-06 -appar 5.21286977232223e-06 -igern 5.21286977232223e-06 -▁Bath 5.21286977232223e-06 -▁Cover 5.21286977232223e-06 -▁Crus 5.21286977232223e-06 -▁Looking 5.21286977232223e-06 -▁Wandern 5.21286977232223e-06 -▁autom 5.21286977232223e-06 -▁gesucht 5.21286977232223e-06 -▁pair 5.21286977232223e-06 -▁Alternativ 5.2090813495225776e-06 -▁Auflagen 5.2090813495225776e-06 -▁Finanzrahmen 5.2090813495225776e-06 -▁Ken 5.2090813495225776e-06 -▁Manuel 5.2090813495225776e-06 -▁Prost 5.2090813495225776e-06 -▁Redezeit 5.2090813495225776e-06 -▁Wur 5.2090813495225776e-06 -▁abzuschließen 5.2090813495225776e-06 -▁queries 5.2090813495225776e-06 -▁schien 5.2090813495225776e-06 --Hand 5.2052929267229247e-06 -gehend 5.2052929267229247e-06 -olz 5.2052929267229247e-06 -performance 5.2052929267229247e-06 -▁Besetzung 5.2052929267229247e-06 -▁Cit 5.2052929267229247e-06 -▁Gewährung 5.2052929267229247e-06 -▁Medina 5.2052929267229247e-06 -▁ecos 5.2052929267229247e-06 -▁inviting 5.2052929267229247e-06 -▁sla 5.2052929267229247e-06 -Hz 5.2015045039232719e-06 -ahrscheinlich 5.2015045039232719e-06 -bauten 5.2015045039232719e-06 -chitektur 5.2015045039232719e-06 -▁Hunde 5.2015045039232719e-06 -▁Zen 5.2015045039232719e-06 -▁chemischen 5.2015045039232719e-06 -▁echo 5.2015045039232719e-06 -▁mini- 5.2015045039232719e-06 -▁monopoly 5.2015045039232719e-06 --Unternehmen 5.197716081123619e-06 -easing 5.197716081123619e-06 -leuten 5.197716081123619e-06 -wärtigen 5.197716081123619e-06 -ässer 5.197716081123619e-06 -▁79 5.197716081123619e-06 -▁Adr 5.197716081123619e-06 -▁L. 5.197716081123619e-06 -▁Manchester 5.197716081123619e-06 -▁Naturschutz 5.197716081123619e-06 -▁Später 5.197716081123619e-06 -▁Strateg 5.197716081123619e-06 -▁downloads 5.197716081123619e-06 -▁encomp 5.197716081123619e-06 -▁norm 5.197716081123619e-06 -▁schwerwie 5.197716081123619e-06 -▁transatlantischen 5.197716081123619e-06 -▁verify 5.197716081123619e-06 -▁750 5.1939276583239661e-06 -▁Importe 5.1939276583239661e-06 -▁Moldau 5.1939276583239661e-06 -▁anzugehen 5.1939276583239661e-06 -▁buyer 5.1939276583239661e-06 -▁denkt 5.1939276583239661e-06 -▁disregard 5.1939276583239661e-06 -▁enlar 5.1939276583239661e-06 -▁inspect 5.1939276583239661e-06 -▁integrating 5.1939276583239661e-06 -▁intimate 5.1939276583239661e-06 -▁intuitive 5.1939276583239661e-06 -▁lehnen 5.1939276583239661e-06 -▁letzt 5.1939276583239661e-06 -▁minibar 5.1939276583239661e-06 -▁revolutionary 5.1939276583239661e-06 -▁sympathy 5.1939276583239661e-06 -▁trans-European 5.1939276583239661e-06 -ASE 5.1901392355243132e-06 -affung 5.1901392355243132e-06 -ellers 5.1901392355243132e-06 -ergel 5.1901392355243132e-06 -▁Mercosur 5.1901392355243132e-06 -▁Riesen 5.1901392355243132e-06 -▁anch 5.1901392355243132e-06 -▁annually 5.1901392355243132e-06 -▁desp 5.1901392355243132e-06 -▁wirft 5.1901392355243132e-06 -cking 5.18635081272466e-06 -cription 5.18635081272466e-06 -fügen 5.18635081272466e-06 -ritical 5.18635081272466e-06 -▁Inflations 5.18635081272466e-06 -▁Sydney 5.18635081272466e-06 -▁Vice 5.18635081272466e-06 -▁entlassen 5.18635081272466e-06 -▁multif 5.18635081272466e-06 -▁thinks 5.18635081272466e-06 -ASA 5.1825623899250074e-06 -icked 5.1825623899250074e-06 -ream 5.1825623899250074e-06 -rieren 5.1825623899250074e-06 -ρ 5.1825623899250074e-06 -▁Persönlichkeiten 5.1825623899250074e-06 -▁Spirit 5.1825623899250074e-06 -▁Tiefe 5.1825623899250074e-06 -▁behindert 5.1825623899250074e-06 -▁belt 5.1825623899250074e-06 -▁consultations 5.1825623899250074e-06 -▁erober 5.1825623899250074e-06 -▁fortschritt 5.1825623899250074e-06 -▁mem 5.1825623899250074e-06 -▁modernsten 5.1825623899250074e-06 -▁perception 5.1825623899250074e-06 -auscht 5.1787739671253554e-06 -leist 5.1787739671253554e-06 -leiste 5.1787739671253554e-06 -zahlt 5.1787739671253554e-06 -▁Festivals 5.1787739671253554e-06 -▁Hav 5.1787739671253554e-06 -▁Male 5.1787739671253554e-06 -▁Nich 5.1787739671253554e-06 -▁Rating 5.1787739671253554e-06 -▁Staatsbürger 5.1787739671253554e-06 -▁corporations 5.1787739671253554e-06 -▁europaweit 5.1787739671253554e-06 -▁kar 5.1787739671253554e-06 -▁pra 5.1787739671253554e-06 -▁refusal 5.1787739671253554e-06 -uropäische 5.1749855443257025e-06 -▁Cour 5.1749855443257025e-06 -▁Folgendes 5.1749855443257025e-06 -▁Steel 5.1749855443257025e-06 -▁Stop 5.1749855443257025e-06 -▁US-amer 5.1749855443257025e-06 -▁accompanying 5.1749855443257025e-06 -▁beachtet 5.1749855443257025e-06 -▁elite 5.1749855443257025e-06 -▁erfahrenen 5.1749855443257025e-06 -▁immen 5.1749855443257025e-06 -▁linking 5.1749855443257025e-06 -▁unweit 5.1749855443257025e-06 -metr 5.1711971215260496e-06 -sproblem 5.1711971215260496e-06 -▁Again 5.1711971215260496e-06 -▁Democracy 5.1711971215260496e-06 -▁Raumfahrt 5.1711971215260496e-06 -▁TB 5.1711971215260496e-06 -▁Zivilisation 5.1711971215260496e-06 -▁bekämpft 5.1711971215260496e-06 -▁gebrauch 5.1711971215260496e-06 -▁geschw 5.1711971215260496e-06 -▁incomes 5.1711971215260496e-06 -▁overwhelming 5.1711971215260496e-06 -▁verteilen 5.1711971215260496e-06 -▁wahrgenommen 5.1711971215260496e-06 -ael 5.1674086987263967e-06 -amas 5.1674086987263967e-06 -forest 5.1674086987263967e-06 -inj 5.1674086987263967e-06 -olitik 5.1674086987263967e-06 -ränken 5.1674086987263967e-06 -önt 5.1674086987263967e-06 -▁Bavaria 5.1674086987263967e-06 -▁Gesellschafts 5.1674086987263967e-06 -▁Lands 5.1674086987263967e-06 -▁Segment 5.1674086987263967e-06 -▁breeding 5.1674086987263967e-06 -▁feste 5.1674086987263967e-06 -▁limitation 5.1674086987263967e-06 -▁php 5.1674086987263967e-06 -▁shore 5.1674086987263967e-06 -▁surfaces 5.1674086987263967e-06 --Raum 5.1636202759267438e-06 -▁Laufzeit 5.1636202759267438e-06 -▁Method 5.1636202759267438e-06 -▁Oft 5.1636202759267438e-06 -▁automatischen 5.1636202759267438e-06 -▁katastrophal 5.1636202759267438e-06 -▁ours 5.1636202759267438e-06 -▁panorama 5.1636202759267438e-06 -▁wunderbaren 5.1636202759267438e-06 -abling 5.1598318531270909e-06 -anwendungen 5.1598318531270909e-06 -auswahl 5.1598318531270909e-06 -echan 5.1598318531270909e-06 -enntnissen 5.1598318531270909e-06 -gäng 5.1598318531270909e-06 -▁Begeisterung 5.1598318531270909e-06 -▁Contract 5.1598318531270909e-06 -▁Dog 5.1598318531270909e-06 -▁Guatemala 5.1598318531270909e-06 -▁Vic 5.1598318531270909e-06 -▁Zeile 5.1598318531270909e-06 -▁medicinal 5.1598318531270909e-06 -▁protective 5.1598318531270909e-06 -▁strikes 5.1598318531270909e-06 -erkt 5.156043430327438e-06 -iedlung 5.156043430327438e-06 -oir 5.156043430327438e-06 -pak 5.156043430327438e-06 -ulent 5.156043430327438e-06 -▁81 5.156043430327438e-06 -▁Richtigkeit 5.156043430327438e-06 -▁Werde 5.156043430327438e-06 -▁Zentrums 5.156043430327438e-06 -▁democracies 5.156043430327438e-06 -▁hung 5.156043430327438e-06 -▁zug 5.156043430327438e-06 -▁у 5.156043430327438e-06 -üs 5.1522550075277851e-06 -С 5.1522550075277851e-06 -▁Hoffnungen 5.1522550075277851e-06 -▁Städt 5.1522550075277851e-06 -▁Vorstands 5.1522550075277851e-06 -▁beobachtet 5.1522550075277851e-06 -▁direkter 5.1522550075277851e-06 -▁horm 5.1522550075277851e-06 -▁regelmäßige 5.1522550075277851e-06 -▁schlimmsten 5.1522550075277851e-06 -ICE 5.1484665847281331e-06 -ehlen 5.1484665847281331e-06 -gehörige 5.1484665847281331e-06 -ördern 5.1484665847281331e-06 -▁Frances 5.1484665847281331e-06 -▁Komplexität 5.1484665847281331e-06 -▁Vieh 5.1484665847281331e-06 -▁Yacht 5.1484665847281331e-06 -▁geräumigen 5.1484665847281331e-06 -▁unbesch 5.1484665847281331e-06 -▁verweist 5.1484665847281331e-06 -jar 5.14467816192848e-06 -luft 5.14467816192848e-06 -▁Erkenntnis 5.14467816192848e-06 -▁Herm 5.14467816192848e-06 -▁Nepal 5.14467816192848e-06 -▁Panel 5.14467816192848e-06 -▁Simon 5.14467816192848e-06 -▁Unternehm 5.14467816192848e-06 -▁accommod 5.14467816192848e-06 -▁degrad 5.14467816192848e-06 -▁holes 5.14467816192848e-06 -▁nicht- 5.14467816192848e-06 -▁posed 5.14467816192848e-06 -▁scient 5.14467816192848e-06 -▁wirksamer 5.14467816192848e-06 -quoten 5.1408897391288273e-06 -▁Bald 5.1408897391288273e-06 -▁EU-Haus 5.1408897391288273e-06 -▁Extr 5.1408897391288273e-06 -▁Gegenwärtig 5.1408897391288273e-06 -▁Ozean 5.1408897391288273e-06 -▁Schreib 5.1408897391288273e-06 -▁Seattle 5.1408897391288273e-06 -▁aiming 5.1408897391288273e-06 -▁homogen 5.1408897391288273e-06 -▁nightlife 5.1408897391288273e-06 -▁worker 5.1408897391288273e-06 -ME 5.1371013163291744e-06 -▁2.6 5.1371013163291744e-06 -▁Depending 5.1371013163291744e-06 -▁Imp 5.1371013163291744e-06 -▁Mittelme 5.1371013163291744e-06 -▁Störungen 5.1371013163291744e-06 -▁andauern 5.1371013163291744e-06 -▁anyway 5.1371013163291744e-06 -▁vereinbaren 5.1371013163291744e-06 -sing 5.1333128935295215e-06 -▁Behandlungs 5.1333128935295215e-06 -▁Clar 5.1333128935295215e-06 -▁FREE 5.1333128935295215e-06 -▁benachrichtigt 5.1333128935295215e-06 -▁bereitgestellten 5.1333128935295215e-06 -▁demonstrations 5.1333128935295215e-06 -▁inwieweit 5.1333128935295215e-06 -▁strenger 5.1333128935295215e-06 -▁tschechischen 5.1333128935295215e-06 -▁verlor 5.1333128935295215e-06 -▁† 5.1333128935295215e-06 -amerika 5.1295244707298686e-06 -eman 5.1295244707298686e-06 -hlung 5.1295244707298686e-06 -m2 5.1295244707298686e-06 -seud 5.1295244707298686e-06 -▁'E 5.1295244707298686e-06 -▁Akzeptanz 5.1295244707298686e-06 -▁Diktatur 5.1295244707298686e-06 -▁Gesundheitswesen 5.1295244707298686e-06 -▁Immunität 5.1295244707298686e-06 -▁Kalk 5.1295244707298686e-06 -▁Neither 5.1295244707298686e-06 -▁Unglück 5.1295244707298686e-06 -▁ausgewählte 5.1295244707298686e-06 -▁bloß 5.1295244707298686e-06 -▁compens 5.1295244707298686e-06 -▁developer 5.1295244707298686e-06 -▁ersuchen 5.1295244707298686e-06 -▁memories 5.1295244707298686e-06 -▁tags 5.1295244707298686e-06 -▁worried 5.1295244707298686e-06 -Station 5.1257360479302157e-06 -oded 5.1257360479302157e-06 -ögern 5.1257360479302157e-06 -▁..... 5.1257360479302157e-06 -▁Fußgänger 5.1257360479302157e-06 -▁Monaco 5.1257360479302157e-06 -▁Salon 5.1257360479302157e-06 -▁abgeschafft 5.1257360479302157e-06 -▁deadlines 5.1257360479302157e-06 -▁detention 5.1257360479302157e-06 -▁frische 5.1257360479302157e-06 -▁klin 5.1257360479302157e-06 -▁sums 5.1257360479302157e-06 --fl 5.1219476251305637e-06 -inhaber 5.1219476251305637e-06 -post 5.1219476251305637e-06 -Á 5.1219476251305637e-06 -▁Lee 5.1219476251305637e-06 -▁Lock 5.1219476251305637e-06 -▁cream 5.1219476251305637e-06 -▁portion 5.1219476251305637e-06 -▁pushed 5.1219476251305637e-06 -▁~ 5.1219476251305637e-06 -personen 5.1181592023309108e-06 -schulen 5.1181592023309108e-06 -▁Gestalt 5.1181592023309108e-06 -▁Hilton 5.1181592023309108e-06 -▁Hindernis 5.1181592023309108e-06 -▁Holding 5.1181592023309108e-06 -▁Kapitalismus 5.1181592023309108e-06 -▁SR 5.1181592023309108e-06 -▁amid 5.1181592023309108e-06 -▁anonym 5.1181592023309108e-06 -▁asiatischen 5.1181592023309108e-06 -▁basierend 5.1181592023309108e-06 -▁bestraft 5.1181592023309108e-06 -▁bub 5.1181592023309108e-06 -▁eb 5.1181592023309108e-06 -▁kopieren 5.1181592023309108e-06 -▁mi 5.1181592023309108e-06 -▁noble 5.1181592023309108e-06 -▁perceived 5.1181592023309108e-06 -▁Anyone 5.1143707795312579e-06 -▁Canadian 5.1143707795312579e-06 -▁Canary 5.1143707795312579e-06 -▁Orchestra 5.1143707795312579e-06 -▁Political 5.1143707795312579e-06 -▁Sara 5.1143707795312579e-06 -▁dismiss 5.1143707795312579e-06 -▁exklusiv 5.1143707795312579e-06 -▁fant 5.1143707795312579e-06 -▁genet 5.1143707795312579e-06 -▁kont 5.1143707795312579e-06 -reut 5.110582356731605e-06 -▁Autorität 5.110582356731605e-06 -▁Außengrenzen 5.110582356731605e-06 -▁Chianti 5.110582356731605e-06 -▁Colombia 5.110582356731605e-06 -▁Dienstes 5.110582356731605e-06 -▁Ingenie 5.110582356731605e-06 -▁Maj 5.110582356731605e-06 -▁hochwert 5.110582356731605e-06 -▁modification 5.110582356731605e-06 -institut 5.1067939339319521e-06 -ktoren 5.1067939339319521e-06 -thur 5.1067939339319521e-06 -wal 5.1067939339319521e-06 -zeile 5.1067939339319521e-06 -ätten 5.1067939339319521e-06 -▁83 5.1067939339319521e-06 -▁Begr 5.1067939339319521e-06 -▁Flüss 5.1067939339319521e-06 -▁Südost 5.1067939339319521e-06 -▁Therap 5.1067939339319521e-06 -▁Translation 5.1067939339319521e-06 -▁Veter 5.1067939339319521e-06 -▁bottle 5.1067939339319521e-06 -▁sacrific 5.1067939339319521e-06 -▁stere 5.1067939339319521e-06 -▁theatr 5.1067939339319521e-06 -▁zugreifen 5.1067939339319521e-06 -hat 5.1030055111322992e-06 -pension 5.1030055111322992e-06 -vas 5.1030055111322992e-06 -wickelt 5.1030055111322992e-06 -▁Giovanni 5.1030055111322992e-06 -▁Jahrestag 5.1030055111322992e-06 -▁Schottland 5.1030055111322992e-06 -▁einzuleiten 5.1030055111322992e-06 -▁smallest 5.1030055111322992e-06 -amine 5.0992170883326463e-06 -neten 5.0992170883326463e-06 -oors 5.0992170883326463e-06 -stens 5.0992170883326463e-06 -world 5.0992170883326463e-06 -▁LabVIEW 5.0992170883326463e-06 -▁Landwirten 5.0992170883326463e-06 -▁Olympischen 5.0992170883326463e-06 -▁Teuf 5.0992170883326463e-06 -▁bugs 5.0992170883326463e-06 -▁divisions 5.0992170883326463e-06 -▁exotic 5.0992170883326463e-06 -▁landmark 5.0992170883326463e-06 -▁neighb 5.0992170883326463e-06 -▁proves 5.0992170883326463e-06 -▁safer 5.0992170883326463e-06 -▁willk 5.0992170883326463e-06 -.7 5.0954286655329934e-06 -abetes 5.0954286655329934e-06 -cott 5.0954286655329934e-06 -samen 5.0954286655329934e-06 -ssystems 5.0954286655329934e-06 -äuft 5.0954286655329934e-06 -▁Bash 5.0954286655329934e-06 -▁Bewegungs 5.0954286655329934e-06 -▁Camer 5.0954286655329934e-06 -▁Migranten 5.0954286655329934e-06 -▁Reflex 5.0954286655329934e-06 -▁Rud 5.0954286655329934e-06 -▁XX 5.0954286655329934e-06 -▁adjacent 5.0954286655329934e-06 -▁cattle 5.0954286655329934e-06 -▁deciding 5.0954286655329934e-06 -▁delegations 5.0954286655329934e-06 -▁dynamischen 5.0954286655329934e-06 -▁nnen 5.0954286655329934e-06 -▁ruins 5.0954286655329934e-06 -▁salle 5.0954286655329934e-06 -genda 5.0916402427333414e-06 -▁Ash 5.0916402427333414e-06 -▁Wettbew 5.0916402427333414e-06 -▁confirms 5.0916402427333414e-06 -▁endors 5.0916402427333414e-06 -▁endorsed 5.0916402427333414e-06 -▁irrespective 5.0916402427333414e-06 -▁sequence 5.0916402427333414e-06 -▁stor 5.0916402427333414e-06 -▁verschw 5.0916402427333414e-06 -▁wisdom 5.0916402427333414e-06 -▁wit 5.0916402427333414e-06 --scale 5.0878518199336885e-06 -preisen 5.0878518199336885e-06 -▁105 5.0878518199336885e-06 -▁Andor 5.0878518199336885e-06 -▁Anpassungen 5.0878518199336885e-06 -▁Aufklärung 5.0878518199336885e-06 -▁Pi 5.0878518199336885e-06 -▁korr 5.0878518199336885e-06 -▁punishment 5.0878518199336885e-06 -iform 5.0840633971340356e-06 -ruckt 5.0840633971340356e-06 -zum 5.0840633971340356e-06 -▁Europäisches 5.0840633971340356e-06 -▁Königreichs 5.0840633971340356e-06 -▁abschließen 5.0840633971340356e-06 -▁admire 5.0840633971340356e-06 -▁classification 5.0840633971340356e-06 -▁hero 5.0840633971340356e-06 -▁obstacle 5.0840633971340356e-06 -▁reader 5.0840633971340356e-06 -hältnissen 5.0802749743343827e-06 -iento 5.0802749743343827e-06 -omo 5.0802749743343827e-06 -stellten 5.0802749743343827e-06 -tration 5.0802749743343827e-06 -▁Against 5.0802749743343827e-06 -▁attraktive 5.0802749743343827e-06 -▁beherbergt 5.0802749743343827e-06 -▁boutique 5.0802749743343827e-06 -▁dramatically 5.0802749743343827e-06 -▁guilty 5.0802749743343827e-06 -▁seas 5.0802749743343827e-06 -earning 5.07648655153473e-06 -izard 5.07648655153473e-06 -lave 5.07648655153473e-06 -orum 5.07648655153473e-06 -prises 5.07648655153473e-06 -ynast 5.07648655153473e-06 -▁Beschl 5.07648655153473e-06 -▁Dé 5.07648655153473e-06 -▁Einstieg 5.07648655153473e-06 -▁Gefangen 5.07648655153473e-06 -▁Guantánamo 5.07648655153473e-06 -▁Recently 5.07648655153473e-06 -▁YouTube 5.07648655153473e-06 -▁barrier 5.07648655153473e-06 -▁commerce 5.07648655153473e-06 -▁dro 5.07648655153473e-06 -▁hoc 5.07648655153473e-06 -▁tiefen 5.07648655153473e-06 -.8 5.0726981287350769e-06 -jenige 5.0726981287350769e-06 -▁CSS 5.0726981287350769e-06 -▁Heidelberg 5.0726981287350769e-06 -▁angeführt 5.0726981287350769e-06 -▁berufen 5.0726981287350769e-06 -▁exakt 5.0726981287350769e-06 -▁immunity 5.0726981287350769e-06 -▁kol 5.0726981287350769e-06 -▁sämtlicher 5.0726981287350769e-06 -63 5.068909705935424e-06 -adresse 5.068909705935424e-06 -thaus 5.068909705935424e-06 -ziehungs 5.068909705935424e-06 -▁Valle 5.068909705935424e-06 -▁Währungen 5.068909705935424e-06 -▁amp 5.068909705935424e-06 -▁ausgehend 5.068909705935424e-06 -stoß 5.0651212831357711e-06 -usik 5.0651212831357711e-06 -▁Cant 5.0651212831357711e-06 -▁Hunderte 5.0651212831357711e-06 -▁Quartal 5.0651212831357711e-06 -▁anzusehen 5.0651212831357711e-06 -▁bemerk 5.0651212831357711e-06 -▁bewahrt 5.0651212831357711e-06 -▁praise 5.0651212831357711e-06 -▁tu 5.0651212831357711e-06 -agern 5.0613328603361191e-06 -atlant 5.0613328603361191e-06 -krimin 5.0613328603361191e-06 -▁Börsen 5.0613328603361191e-06 -▁EB 5.0613328603361191e-06 -▁Herzlich 5.0613328603361191e-06 -▁Pferde 5.0613328603361191e-06 -▁festivals 5.0613328603361191e-06 -▁hör 5.0613328603361191e-06 -▁qualifiz 5.0613328603361191e-06 -We 5.0575444375364662e-06 -gehörigen 5.0575444375364662e-06 -▁Modus 5.0575444375364662e-06 -▁Prince 5.0575444375364662e-06 -▁Stan 5.0575444375364662e-06 -▁Strafgerichtshof 5.0575444375364662e-06 -▁Vortrag 5.0575444375364662e-06 -▁fastest 5.0575444375364662e-06 -▁irgendeine 5.0575444375364662e-06 -▁nit 5.0575444375364662e-06 -▁sleeping 5.0575444375364662e-06 -angehörigen 5.0537560147368133e-06 -gasse 5.0537560147368133e-06 -iso 5.0537560147368133e-06 -ittings 5.0537560147368133e-06 -usse 5.0537560147368133e-06 -wear 5.0537560147368133e-06 -ymn 5.0537560147368133e-06 -▁DES 5.0537560147368133e-06 -▁Dicht 5.0537560147368133e-06 -▁Geden 5.0537560147368133e-06 -▁Kast 5.0537560147368133e-06 -▁Luftra 5.0537560147368133e-06 -▁Reifen 5.0537560147368133e-06 -▁dorthin 5.0537560147368133e-06 -▁dürften 5.0537560147368133e-06 -▁english 5.0537560147368133e-06 -▁geführten 5.0537560147368133e-06 -▁römischen 5.0537560147368133e-06 -▁sheep 5.0537560147368133e-06 -▁tens 5.0537560147368133e-06 -▁vibrant 5.0537560147368133e-06 -▁voranzutreiben 5.0537560147368133e-06 --9 5.0499675919371604e-06 --Systems 5.0499675919371604e-06 -blicke 5.0499675919371604e-06 -steuern 5.0499675919371604e-06 -▁Additional 5.0499675919371604e-06 -▁Beitrittsverhandlungen 5.0499675919371604e-06 -▁Germans 5.0499675919371604e-06 -▁Greg 5.0499675919371604e-06 -▁krit 5.0499675919371604e-06 -▁unvor 5.0499675919371604e-06 --Le 5.0461791691375075e-06 --Lösungen 5.0461791691375075e-06 -ahmen 5.0461791691375075e-06 -imental 5.0461791691375075e-06 -night 5.0461791691375075e-06 -ordnet 5.0461791691375075e-06 -▁Castro 5.0461791691375075e-06 -▁Fremden 5.0461791691375075e-06 -▁Lufthansa 5.0461791691375075e-06 -▁SilverFast 5.0461791691375075e-06 -▁Softwar 5.0461791691375075e-06 -▁Sozialisten 5.0461791691375075e-06 -▁Wanderungen 5.0461791691375075e-06 -▁akt 5.0461791691375075e-06 -▁conserv 5.0461791691375075e-06 -▁halben 5.0461791691375075e-06 -▁sett 5.0461791691375075e-06 -▁vergleichbar 5.0461791691375075e-06 -äglichen 5.0423907463378546e-06 -örper 5.0423907463378546e-06 -▁Chris 5.0423907463378546e-06 -▁Freihandels 5.0423907463378546e-06 -▁circum 5.0423907463378546e-06 -▁completing 5.0423907463378546e-06 -▁diesbezüglichen 5.0423907463378546e-06 -▁erle 5.0423907463378546e-06 --market 5.0386023235382018e-06 -ianer 5.0386023235382018e-06 -irus 5.0386023235382018e-06 -osexual 5.0386023235382018e-06 -profit 5.0386023235382018e-06 -rine 5.0386023235382018e-06 -westen 5.0386023235382018e-06 -▁Chip 5.0386023235382018e-06 -▁Cool 5.0386023235382018e-06 -▁PAR 5.0386023235382018e-06 -▁aust 5.0386023235382018e-06 -▁beweg 5.0386023235382018e-06 -▁far-reaching 5.0386023235382018e-06 -▁gemüt 5.0386023235382018e-06 -▁geräum 5.0386023235382018e-06 -▁inland 5.0386023235382018e-06 -▁Übermittlung 5.0386023235382018e-06 -izations 5.0348139007385489e-06 -jev 5.0348139007385489e-06 -▁Fahrzeugen 5.0348139007385489e-06 -▁Herstellern 5.0348139007385489e-06 -▁Montreal 5.0348139007385489e-06 -▁Operationen 5.0348139007385489e-06 -▁Vie 5.0348139007385489e-06 -▁advisory 5.0348139007385489e-06 -▁bestanden 5.0348139007385489e-06 -▁blockiert 5.0348139007385489e-06 -▁führender 5.0348139007385489e-06 -▁gemütlich 5.0348139007385489e-06 -▁prinzip 5.0348139007385489e-06 -▁schreck 5.0348139007385489e-06 -ordnen 5.0310254779388968e-06 -▁Fülle 5.0310254779388968e-06 -▁Schließung 5.0310254779388968e-06 -▁baut 5.0310254779388968e-06 -▁gratuliere 5.0310254779388968e-06 -▁rolling 5.0310254779388968e-06 -▁stabilen 5.0310254779388968e-06 -▁textiles 5.0310254779388968e-06 -▁trials 5.0310254779388968e-06 -enia 5.0272370551392439e-06 -itary 5.0272370551392439e-06 -▁Gedanke 5.0272370551392439e-06 -▁Glaubens 5.0272370551392439e-06 -▁Iceland 5.0272370551392439e-06 -▁Mountainb 5.0272370551392439e-06 -▁Price 5.0272370551392439e-06 -▁ausgepr 5.0272370551392439e-06 -▁ausgerüstet 5.0272370551392439e-06 -▁dict 5.0272370551392439e-06 -▁ermittelt 5.0272370551392439e-06 -▁gambling 5.0272370551392439e-06 -▁б 5.0272370551392439e-06 -eway 5.023448632339591e-06 -gründe 5.023448632339591e-06 -ierbare 5.023448632339591e-06 -iop 5.023448632339591e-06 -wide 5.023448632339591e-06 -ynes 5.023448632339591e-06 -▁115 5.023448632339591e-06 -▁Consider 5.023448632339591e-06 -▁Synth 5.023448632339591e-06 -▁alph 5.023448632339591e-06 -▁baths 5.023448632339591e-06 -▁bedient 5.023448632339591e-06 -▁bekanntlich 5.023448632339591e-06 -▁bemerkens 5.023448632339591e-06 -▁gemeinschaftliche 5.023448632339591e-06 -▁improves 5.023448632339591e-06 -▁modernisation 5.023448632339591e-06 -berry 5.0196602095399381e-06 -itär 5.0196602095399381e-06 -▁Chancellor 5.0196602095399381e-06 -▁Downloads 5.0196602095399381e-06 -▁Police 5.0196602095399381e-06 -▁Upgrade 5.0196602095399381e-06 -▁ausgeliefert 5.0196602095399381e-06 -▁definitive 5.0196602095399381e-06 -▁fore 5.0196602095399381e-06 -▁gaining 5.0196602095399381e-06 -▁markiert 5.0196602095399381e-06 -▁snacks 5.0196602095399381e-06 -▁wholes 5.0196602095399381e-06 -itur 5.0158717867402853e-06 -owser 5.0158717867402853e-06 -tery 5.0158717867402853e-06 -▁Appell 5.0158717867402853e-06 -▁Scott 5.0158717867402853e-06 -▁Warsaw 5.0158717867402853e-06 -▁dynamische 5.0158717867402853e-06 -▁hunting 5.0158717867402853e-06 -▁legislativen 5.0158717867402853e-06 -▁summ 5.0158717867402853e-06 -▁witnessed 5.0158717867402853e-06 -▁österreichische 5.0158717867402853e-06 --making 5.0120833639406324e-06 -lassung 5.0120833639406324e-06 -opy 5.0120833639406324e-06 -schlüssel 5.0120833639406324e-06 -▁BAE 5.0120833639406324e-06 -▁CAR 5.0120833639406324e-06 -▁Rica 5.0120833639406324e-06 -▁buttons 5.0120833639406324e-06 -▁chart 5.0120833639406324e-06 -▁gesellschaft 5.0120833639406324e-06 -▁remedy 5.0120833639406324e-06 -▁terrain 5.0120833639406324e-06 -▁unsicher 5.0120833639406324e-06 -odge 5.0082949411409795e-06 -ränkten 5.0082949411409795e-06 -▁Cruz 5.0082949411409795e-06 -▁Schwerpunkte 5.0082949411409795e-06 -▁consolidate 5.0082949411409795e-06 -▁eco 5.0082949411409795e-06 -▁garant 5.0082949411409795e-06 -TM 5.0045065183413266e-06 -▁DG 5.0045065183413266e-06 -▁Fracht 5.0045065183413266e-06 -▁Industriepolitik 5.0045065183413266e-06 -▁VIS 5.0045065183413266e-06 -▁ung 5.0045065183413266e-06 -▁verfahren 5.0045065183413266e-06 -pet 5.0007180955416745e-06 -terior 5.0007180955416745e-06 -▁Beträge 5.0007180955416745e-06 -▁Hague 5.0007180955416745e-06 -▁Mathem 5.0007180955416745e-06 -▁Tuscany 5.0007180955416745e-06 -▁Zuliefer 5.0007180955416745e-06 -▁Zustellbetten 5.0007180955416745e-06 -▁courtyard 5.0007180955416745e-06 -▁Übertrag 5.0007180955416745e-06 -anbieter 4.9969296727420216e-06 -gesundheit 4.9969296727420216e-06 -öser 4.9969296727420216e-06 -π 4.9969296727420216e-06 -▁Feedback 4.9969296727420216e-06 -▁Projektes 4.9969296727420216e-06 -▁Scandinav 4.9969296727420216e-06 -▁Schnitt 4.9969296727420216e-06 -▁Scr 4.9969296727420216e-06 -▁abzugeben 4.9969296727420216e-06 -▁paragraphs 4.9969296727420216e-06 -▁secular 4.9969296727420216e-06 -ACE 4.9931412499423687e-06 -ITY 4.9931412499423687e-06 -eability 4.9931412499423687e-06 -ktor 4.9931412499423687e-06 -langes 4.9931412499423687e-06 -malen 4.9931412499423687e-06 -ounds 4.9931412499423687e-06 -▁A1 4.9931412499423687e-06 -▁Abt 4.9931412499423687e-06 -▁Branden 4.9931412499423687e-06 -▁multilateralen 4.9931412499423687e-06 -▁positives 4.9931412499423687e-06 -▁stagn 4.9931412499423687e-06 -▁surf 4.9931412499423687e-06 -LICH 4.9893528271427159e-06 -marktes 4.9893528271427159e-06 -trieb 4.9893528271427159e-06 -▁Analys 4.9893528271427159e-06 -▁Diejenigen 4.9893528271427159e-06 -▁Edit 4.9893528271427159e-06 -▁Mall 4.9893528271427159e-06 -▁akadem 4.9893528271427159e-06 -▁eventuelle 4.9893528271427159e-06 -▁fortress 4.9893528271427159e-06 -▁melt 4.9893528271427159e-06 -come 4.985564404343063e-06 -klär 4.985564404343063e-06 -leger 4.985564404343063e-06 -▁PA 4.985564404343063e-06 -▁mache 4.985564404343063e-06 -▁mobilen 4.985564404343063e-06 -▁modifications 4.985564404343063e-06 -▁volumes 4.985564404343063e-06 --West 4.98177598154341e-06 -acion 4.98177598154341e-06 -asm 4.98177598154341e-06 -ivirus 4.98177598154341e-06 -viel 4.98177598154341e-06 -zun 4.98177598154341e-06 -▁Algeria 4.98177598154341e-06 -▁Autonom 4.98177598154341e-06 -▁Bien 4.98177598154341e-06 -▁Bürgerrechte 4.98177598154341e-06 -▁Einreise 4.98177598154341e-06 -▁FTP 4.98177598154341e-06 -▁Harmon 4.98177598154341e-06 -▁Kongo 4.98177598154341e-06 -▁Oz 4.98177598154341e-06 -▁anspruchsvolle 4.98177598154341e-06 -▁femin 4.98177598154341e-06 -▁licensed 4.98177598154341e-06 -▁merk 4.98177598154341e-06 -▁niedrige 4.98177598154341e-06 -▁rumän 4.98177598154341e-06 -▁streben 4.98177598154341e-06 -ischof 4.9779875587437572e-06 -mack 4.9779875587437572e-06 -transport 4.9779875587437572e-06 -urity 4.9779875587437572e-06 -zähl 4.9779875587437572e-06 -▁1940 4.9779875587437572e-06 -▁Cohesion 4.9779875587437572e-06 -▁Dienststellen 4.9779875587437572e-06 -▁NG 4.9779875587437572e-06 -▁Ungleichheiten 4.9779875587437572e-06 -▁Werken 4.9779875587437572e-06 -▁costly 4.9779875587437572e-06 -▁herbe 4.9779875587437572e-06 -▁scholars 4.9779875587437572e-06 -▁tasty 4.9779875587437572e-06 -ipen 4.9741991359441043e-06 -ritte 4.9741991359441043e-06 -μ 4.9741991359441043e-06 -▁Gutes 4.9741991359441043e-06 -▁Mord 4.9741991359441043e-06 -▁Want 4.9741991359441043e-06 -▁Zak 4.9741991359441043e-06 -▁container 4.9741991359441043e-06 -▁cott 4.9741991359441043e-06 -▁disabilities 4.9741991359441043e-06 -▁introduces 4.9741991359441043e-06 -▁kommunizieren 4.9741991359441043e-06 -▁settle 4.9741991359441043e-06 -▁shoulder 4.9741991359441043e-06 -▁tendency 4.9741991359441043e-06 -▁verschärft 4.9741991359441043e-06 -▁Ök 4.9741991359441043e-06 -ammeln 4.9704107131444522e-06 -hardt 4.9704107131444522e-06 -ressourcen 4.9704107131444522e-06 -▁1990s 4.9704107131444522e-06 -▁BER 4.9704107131444522e-06 -▁Carol 4.9704107131444522e-06 -▁Jahreszeit 4.9704107131444522e-06 -▁Motoren 4.9704107131444522e-06 -▁Stere 4.9704107131444522e-06 -▁Though 4.9704107131444522e-06 -▁Verhinderung 4.9704107131444522e-06 -▁Wag 4.9704107131444522e-06 -▁Wohnsitz 4.9704107131444522e-06 -▁aufhalten 4.9704107131444522e-06 -▁erheb 4.9704107131444522e-06 -▁europä 4.9704107131444522e-06 -▁fabulous 4.9704107131444522e-06 -▁gez 4.9704107131444522e-06 -▁zusammengestellt 4.9704107131444522e-06 -agung 4.9666222903447994e-06 -anti 4.9666222903447994e-06 -ialen 4.9666222903447994e-06 -omatic 4.9666222903447994e-06 -ystemen 4.9666222903447994e-06 -▁FS 4.9666222903447994e-06 -▁Linz 4.9666222903447994e-06 -▁beruf 4.9666222903447994e-06 -▁clauses 4.9666222903447994e-06 -▁complementary 4.9666222903447994e-06 -▁konfigur 4.9666222903447994e-06 -▁meaningful 4.9666222903447994e-06 -▁sends 4.9666222903447994e-06 --bed 4.9628338675451465e-06 -echts 4.9628338675451465e-06 -künften 4.9628338675451465e-06 -▁Diskussions 4.9628338675451465e-06 -▁Impuls 4.9628338675451465e-06 -▁Treff 4.9628338675451465e-06 -▁UEN 4.9628338675451465e-06 -▁assessed 4.9628338675451465e-06 -▁dea 4.9628338675451465e-06 -▁roten 4.9628338675451465e-06 -▁verdanken 4.9628338675451465e-06 -VD 4.9590454447454936e-06 -ching 4.9590454447454936e-06 -drom 4.9590454447454936e-06 -tro 4.9590454447454936e-06 -▁CP 4.9590454447454936e-06 -▁Championship 4.9590454447454936e-06 -▁Gleichwohl 4.9590454447454936e-06 -▁Magazine 4.9590454447454936e-06 -▁SN 4.9590454447454936e-06 -▁Sozialpartner 4.9590454447454936e-06 -▁activate 4.9590454447454936e-06 -▁begrenzte 4.9590454447454936e-06 -▁mittel- 4.9590454447454936e-06 -▁tensions 4.9590454447454936e-06 -▁traditionally 4.9590454447454936e-06 -▁très 4.9590454447454936e-06 --vis 4.9552570219458407e-06 -aussch 4.9552570219458407e-06 -axis 4.9552570219458407e-06 -iza 4.9552570219458407e-06 -obb 4.9552570219458407e-06 -▁MEDA 4.9552570219458407e-06 -▁Meinungsäußerung 4.9552570219458407e-06 -▁Schlußfolgerungen 4.9552570219458407e-06 -▁Zeitschrift 4.9552570219458407e-06 -▁err 4.9552570219458407e-06 -▁hous 4.9552570219458407e-06 -▁idyll 4.9552570219458407e-06 -▁jünger 4.9552570219458407e-06 -▁psychological 4.9552570219458407e-06 -▁stead 4.9552570219458407e-06 -jack 4.9514685991461878e-06 -▁Ern 4.9514685991461878e-06 -▁Gemeinschaftsrecht 4.9514685991461878e-06 -▁Objective 4.9514685991461878e-06 -▁Sisko 4.9514685991461878e-06 -▁Verschw 4.9514685991461878e-06 -▁coastline 4.9514685991461878e-06 -▁defen 4.9514685991461878e-06 -▁gemeinschaft 4.9514685991461878e-06 -▁limiting 4.9514685991461878e-06 -▁painted 4.9514685991461878e-06 -▁strateg 4.9514685991461878e-06 -ampagnen 4.9476801763465349e-06 -quenzen 4.9476801763465349e-06 -uliche 4.9476801763465349e-06 -▁-T 4.9476801763465349e-06 -▁Commun 4.9476801763465349e-06 -▁Petersburg 4.9476801763465349e-06 -▁Relig 4.9476801763465349e-06 -▁Skigebiet 4.9476801763465349e-06 -▁selts 4.9476801763465349e-06 -▁unmittelbare 4.9476801763465349e-06 -▁zurückkommen 4.9476801763465349e-06 --Entwicklungs 4.9438917535468829e-06 -benen 4.9438917535468829e-06 -forum 4.9438917535468829e-06 -garant 4.9438917535468829e-06 -▁EU-M 4.9438917535468829e-06 -▁Follow 4.9438917535468829e-06 -▁Fäl 4.9438917535468829e-06 -▁Instit 4.9438917535468829e-06 -▁Storn 4.9438917535468829e-06 -▁aufhören 4.9438917535468829e-06 -▁birthday 4.9438917535468829e-06 -▁distinctive 4.9438917535468829e-06 -▁en-suite 4.9438917535468829e-06 -▁geschenkt 4.9438917535468829e-06 -▁loop 4.9438917535468829e-06 -▁oriented 4.9438917535468829e-06 -▁unexpected 4.9438917535468829e-06 -acle 4.94010333074723e-06 -fels 4.94010333074723e-06 -ologies 4.94010333074723e-06 -rechtlich 4.94010333074723e-06 -▁Gender 4.94010333074723e-06 -▁Hochschulen 4.94010333074723e-06 -▁Pisa 4.94010333074723e-06 -▁Stephen 4.94010333074723e-06 -▁Ziels 4.94010333074723e-06 -▁asc 4.94010333074723e-06 -▁commands 4.94010333074723e-06 -▁cum 4.94010333074723e-06 -▁gericht 4.94010333074723e-06 -▁wr 4.94010333074723e-06 -anza 4.9363149079475771e-06 -bin 4.9363149079475771e-06 -grade 4.9363149079475771e-06 -sph 4.9363149079475771e-06 -zip 4.9363149079475771e-06 -▁Körpers 4.9363149079475771e-06 -▁PayPal 4.9363149079475771e-06 -▁Spann 4.9363149079475771e-06 -▁Train 4.9363149079475771e-06 -▁advert 4.9363149079475771e-06 -▁altitude 4.9363149079475771e-06 -▁bedeutender 4.9363149079475771e-06 -▁cord 4.9363149079475771e-06 -▁perman 4.9363149079475771e-06 -▁tunnel 4.9363149079475771e-06 -▁vigil 4.9363149079475771e-06 -erhö 4.9325264851479242e-06 -uto 4.9325264851479242e-06 -yball 4.9325264851479242e-06 -▁Barcel 4.9325264851479242e-06 -▁Fabrik 4.9325264851479242e-06 -▁Lov 4.9325264851479242e-06 -▁Offen 4.9325264851479242e-06 -▁Organisations 4.9325264851479242e-06 -▁Vergessen 4.9325264851479242e-06 -▁Wechselkurs 4.9325264851479242e-06 -▁ampl 4.9325264851479242e-06 -▁baggage 4.9325264851479242e-06 -▁bankrupt 4.9325264851479242e-06 -▁betreut 4.9325264851479242e-06 -▁contrib 4.9325264851479242e-06 -▁dairy 4.9325264851479242e-06 -▁klarstellen 4.9325264851479242e-06 -OSS 4.9287380623482713e-06 -fehler 4.9287380623482713e-06 -läuft 4.9287380623482713e-06 -weren 4.9287380623482713e-06 -öder 4.9287380623482713e-06 -▁Eurob 4.9287380623482713e-06 -▁gegebenen 4.9287380623482713e-06 -▁geräumige 4.9287380623482713e-06 -▁kalt 4.9287380623482713e-06 -▁veranstaltet 4.9287380623482713e-06 -▁wenngleich 4.9287380623482713e-06 -adies 4.9249496395486184e-06 -essions 4.9249496395486184e-06 -familie 4.9249496395486184e-06 -gebungen 4.9249496395486184e-06 -ologien 4.9249496395486184e-06 -ugg 4.9249496395486184e-06 -▁Author 4.9249496395486184e-06 -▁GO 4.9249496395486184e-06 -▁Posselt 4.9249496395486184e-06 -▁Reihen 4.9249496395486184e-06 -▁identifying 4.9249496395486184e-06 -▁konstitution 4.9249496395486184e-06 -▁künstlerischen 4.9249496395486184e-06 -▁prerequisite 4.9249496395486184e-06 -▁threatening 4.9249496395486184e-06 -ikaner 4.9211612167489655e-06 -ounce 4.9211612167489655e-06 -treffen 4.9211612167489655e-06 -ätigkeit 4.9211612167489655e-06 -▁24-hour 4.9211612167489655e-06 -▁DL 4.9211612167489655e-06 -▁broadcasting 4.9211612167489655e-06 -▁passport 4.9211612167489655e-06 -▁quo 4.9211612167489655e-06 -▁rhythm 4.9211612167489655e-06 -▁städtischen 4.9211612167489655e-06 -restaurant 4.9173727939493126e-06 -º 4.9173727939493126e-06 -▁Langen 4.9173727939493126e-06 -▁Medium 4.9173727939493126e-06 -▁Mär 4.9173727939493126e-06 -▁Zuschauer 4.9173727939493126e-06 -▁attraktiven 4.9173727939493126e-06 -▁bird 4.9173727939493126e-06 -▁landing 4.9173727939493126e-06 -▁unglaublich 4.9173727939493126e-06 -▁yield 4.9173727939493126e-06 -ancement 4.9135843711496606e-06 -aschinen 4.9135843711496606e-06 -ellungen 4.9135843711496606e-06 -ergebnis 4.9135843711496606e-06 -▁10.00 4.9135843711496606e-06 -▁Ansprüchen 4.9135843711496606e-06 -▁Bruch 4.9135843711496606e-06 -▁Hallen 4.9135843711496606e-06 -▁Synerg 4.9135843711496606e-06 -▁Wilhelm 4.9135843711496606e-06 -▁begrenzen 4.9135843711496606e-06 -▁pill 4.9135843711496606e-06 -▁sorts 4.9135843711496606e-06 -cho 4.9097959483500077e-06 -elect 4.9097959483500077e-06 -oda 4.9097959483500077e-06 -onomic 4.9097959483500077e-06 -oppel 4.9097959483500077e-06 -orien 4.9097959483500077e-06 -vr 4.9097959483500077e-06 -▁Afrikas 4.9097959483500077e-06 -▁Gastronomie 4.9097959483500077e-06 -▁Hotelzimmer 4.9097959483500077e-06 -▁Pries 4.9097959483500077e-06 -▁couldn 4.9097959483500077e-06 -▁emphasize 4.9097959483500077e-06 -▁fragile 4.9097959483500077e-06 -▁muslim 4.9097959483500077e-06 -▁protects 4.9097959483500077e-06 -▁qm 4.9097959483500077e-06 -agd 4.9060075255503548e-06 -ension 4.9060075255503548e-06 -figur 4.9060075255503548e-06 -forderungen 4.9060075255503548e-06 -ourge 4.9060075255503548e-06 -wein 4.9060075255503548e-06 -üp 4.9060075255503548e-06 -▁1971 4.9060075255503548e-06 -▁Daraus 4.9060075255503548e-06 -▁Erfassung 4.9060075255503548e-06 -▁Messen 4.9060075255503548e-06 -▁appar 4.9060075255503548e-06 -▁echter 4.9060075255503548e-06 -▁ele 4.9060075255503548e-06 -▁interact 4.9060075255503548e-06 -▁mang 4.9060075255503548e-06 -▁niedr 4.9060075255503548e-06 -▁popularity 4.9060075255503548e-06 -▁vine 4.9060075255503548e-06 -MP 4.9022191027507019e-06 -clus 4.9022191027507019e-06 -ensp 4.9022191027507019e-06 -orders 4.9022191027507019e-06 -yle 4.9022191027507019e-06 -▁Fusion 4.9022191027507019e-06 -▁IND 4.9022191027507019e-06 -▁Müll 4.9022191027507019e-06 -▁Tagungen 4.9022191027507019e-06 -▁Ukrainian 4.9022191027507019e-06 -▁earthquake 4.9022191027507019e-06 -▁hunder 4.9022191027507019e-06 -▁indications 4.9022191027507019e-06 -▁oasis 4.9022191027507019e-06 -▁virtuellen 4.9022191027507019e-06 -drückt 4.898430679951049e-06 -ety 4.898430679951049e-06 -punkten 4.898430679951049e-06 -ügel 4.898430679951049e-06 -▁Beispielsweise 4.898430679951049e-06 -▁Residenz 4.898430679951049e-06 -▁benutzer 4.898430679951049e-06 -▁coat 4.898430679951049e-06 -▁overl 4.898430679951049e-06 -▁starker 4.898430679951049e-06 -diensten 4.8946422571513961e-06 -ereign 4.8946422571513961e-06 -info 4.8946422571513961e-06 -▁Andrea 4.8946422571513961e-06 -▁Ausgangs 4.8946422571513961e-06 -▁Davon 4.8946422571513961e-06 -▁Exekut 4.8946422571513961e-06 -▁Feiert 4.8946422571513961e-06 -▁Grupp 4.8946422571513961e-06 -▁Publikationen 4.8946422571513961e-06 -▁fare 4.8946422571513961e-06 -▁spl 4.8946422571513961e-06 -▁tons 4.8946422571513961e-06 --Produkte 4.8908538343517432e-06 -gien 4.8908538343517432e-06 -otion 4.8908538343517432e-06 -schrauber 4.8908538343517432e-06 -zwungen 4.8908538343517432e-06 -▁Alten 4.8908538343517432e-06 -▁Embry 4.8908538343517432e-06 -▁Endes 4.8908538343517432e-06 -▁Official 4.8908538343517432e-06 -▁Poettering 4.8908538343517432e-06 -▁Provin 4.8908538343517432e-06 -▁irgendwo 4.8908538343517432e-06 -▁massiven 4.8908538343517432e-06 -▁starb 4.8908538343517432e-06 -▁tracking 4.8908538343517432e-06 -▁wirst 4.8908538343517432e-06 -blieben 4.88706541155209e-06 -elefon 4.88706541155209e-06 -itches 4.88706541155209e-06 -xim 4.88706541155209e-06 -▁Besatzung 4.88706541155209e-06 -▁Frequenz 4.88706541155209e-06 -▁GN 4.88706541155209e-06 -▁Gegenden 4.88706541155209e-06 -▁Spock 4.88706541155209e-06 -▁Standorten 4.88706541155209e-06 -▁gebucht 4.88706541155209e-06 -▁mittelalterlichen 4.88706541155209e-06 -▁zuständige 4.88706541155209e-06 -▁übertrieben 4.88706541155209e-06 -hilf 4.8832769887524383e-06 -iane 4.8832769887524383e-06 -plate 4.8832769887524383e-06 -wider 4.8832769887524383e-06 -yram 4.8832769887524383e-06 -▁Kass 4.8832769887524383e-06 -▁Malaga 4.8832769887524383e-06 -▁Verwand 4.8832769887524383e-06 -▁Volumen 4.8832769887524383e-06 -▁algorith 4.8832769887524383e-06 -▁curr 4.8832769887524383e-06 -▁encl 4.8832769887524383e-06 -▁exploited 4.8832769887524383e-06 -▁grid 4.8832769887524383e-06 -▁leitet 4.8832769887524383e-06 -▁painful 4.8832769887524383e-06 -▁pipeline 4.8832769887524383e-06 -▁resol 4.8832769887524383e-06 --Code 4.8794885659527854e-06 --Vers 4.8794885659527854e-06 -low 4.8794885659527854e-06 -stechnologien 4.8794885659527854e-06 -vereinbarung 4.8794885659527854e-06 -zn 4.8794885659527854e-06 -▁Käse 4.8794885659527854e-06 -▁Nachteile 4.8794885659527854e-06 -▁Oliver 4.8794885659527854e-06 -▁einschl 4.8794885659527854e-06 -▁lust 4.8794885659527854e-06 -▁refreshing 4.8794885659527854e-06 -▁sichert 4.8794885659527854e-06 -ehouse 4.8757001431531325e-06 -gir 4.8757001431531325e-06 -▁Funktionsweise 4.8757001431531325e-06 -▁HI 4.8757001431531325e-06 -▁Param 4.8757001431531325e-06 -▁abhängen 4.8757001431531325e-06 -▁affirm 4.8757001431531325e-06 -▁legacy 4.8757001431531325e-06 -▁loy 4.8757001431531325e-06 -▁worlds 4.8757001431531325e-06 -Sh 4.87191172035348e-06 -ennungs 4.87191172035348e-06 -immten 4.87191172035348e-06 -▁Dream 4.87191172035348e-06 -▁Taschen 4.87191172035348e-06 -▁aboard 4.87191172035348e-06 -▁bestät 4.87191172035348e-06 -▁confidentiality 4.87191172035348e-06 -▁kreative 4.87191172035348e-06 -▁shorter 4.87191172035348e-06 -▁verhandelt 4.87191172035348e-06 -ionally 4.8681232975538267e-06 -jer 4.8681232975538267e-06 -ordnungen 4.8681232975538267e-06 -strich 4.8681232975538267e-06 -wes 4.8681232975538267e-06 -▁forestry 4.8681232975538267e-06 -▁leidet 4.8681232975538267e-06 -▁perfekten 4.8681232975538267e-06 -▁spreading 4.8681232975538267e-06 -▁untergebracht 4.8681232975538267e-06 -▁vir 4.8681232975538267e-06 -ivities 4.8643348747541738e-06 -oyd 4.8643348747541738e-06 -uctions 4.8643348747541738e-06 -ugung 4.8643348747541738e-06 -äub 4.8643348747541738e-06 -▁Ausbruch 4.8643348747541738e-06 -▁CO2-Emissionen 4.8643348747541738e-06 -▁Wäscheservice 4.8643348747541738e-06 -▁dekor 4.8643348747541738e-06 -▁erhebt 4.8643348747541738e-06 -▁freundliches 4.8643348747541738e-06 -▁in-house 4.8643348747541738e-06 -▁konsolid 4.8643348747541738e-06 -▁marble 4.8643348747541738e-06 -▁reluct 4.8643348747541738e-06 -▁reservieren 4.8643348747541738e-06 -endums 4.8605464519545209e-06 -erning 4.8605464519545209e-06 -unction 4.8605464519545209e-06 -▁Aar 4.8605464519545209e-06 -▁Feder 4.8605464519545209e-06 -▁Isch 4.8605464519545209e-06 -▁Kohäsion 4.8605464519545209e-06 -▁Kriterium 4.8605464519545209e-06 -▁Meet 4.8605464519545209e-06 -▁Rauch 4.8605464519545209e-06 -▁alert 4.8605464519545209e-06 -▁anre 4.8605464519545209e-06 -▁diplomatischen 4.8605464519545209e-06 -▁toxic 4.8605464519545209e-06 -▁Änderungs 4.8605464519545209e-06 -größte 4.856758029154868e-06 -krank 4.856758029154868e-06 -oj 4.856758029154868e-06 -wär 4.856758029154868e-06 -▁Finanzielle 4.856758029154868e-06 -▁Inzwischen 4.856758029154868e-06 -▁Jim 4.856758029154868e-06 -▁Py 4.856758029154868e-06 -▁Räumlichkeiten 4.856758029154868e-06 -▁Schritten 4.856758029154868e-06 -▁anst 4.856758029154868e-06 -▁kommende 4.856758029154868e-06 -▁persist 4.856758029154868e-06 -▁sof 4.856758029154868e-06 -ceive 4.852969606355216e-06 -edt 4.852969606355216e-06 -lect 4.852969606355216e-06 -machung 4.852969606355216e-06 -▁13th 4.852969606355216e-06 -▁Etage 4.852969606355216e-06 -▁Medic 4.852969606355216e-06 -▁Sitzungsperiode 4.852969606355216e-06 -▁intelligente 4.852969606355216e-06 -▁renewal 4.852969606355216e-06 -▁unified 4.852969606355216e-06 -▁übereinstimmen 4.852969606355216e-06 -▁Aufpreis 4.8491811835555631e-06 -▁Beibehaltung 4.8491811835555631e-06 -▁Feat 4.8491811835555631e-06 -▁Fischereiabkommen 4.8491811835555631e-06 -▁entf 4.8491811835555631e-06 -▁evenings 4.8491811835555631e-06 -▁graphical 4.8491811835555631e-06 -▁lev 4.8491811835555631e-06 -▁mittelständ 4.8491811835555631e-06 -▁pricing 4.8491811835555631e-06 -42 4.84539276075591e-06 -betreiber 4.84539276075591e-06 -umes 4.84539276075591e-06 -▁Century 4.84539276075591e-06 -▁DEM 4.84539276075591e-06 -▁Secretary-General 4.84539276075591e-06 -▁Skif 4.84539276075591e-06 -▁ausgeben 4.84539276075591e-06 -▁gestrichen 4.84539276075591e-06 -▁greift 4.84539276075591e-06 -▁irgendwie 4.84539276075591e-06 -▁ox 4.84539276075591e-06 -▁retained 4.84539276075591e-06 -atar 4.8416043379562573e-06 -▁Anschließend 4.8416043379562573e-06 -▁Genauigkeit 4.8416043379562573e-06 -▁Upper 4.8416043379562573e-06 -▁begrenzten 4.8416043379562573e-06 -▁crow 4.8416043379562573e-06 -▁incidents 4.8416043379562573e-06 -▁serb 4.8416043379562573e-06 -▁umbr 4.8416043379562573e-06 -iba 4.8378159151566044e-06 -otz 4.8378159151566044e-06 -sorge 4.8378159151566044e-06 -test 4.8378159151566044e-06 -▁-D 4.8378159151566044e-06 -▁Anleger 4.8378159151566044e-06 -▁Battle 4.8378159151566044e-06 -▁Beautiful 4.8378159151566044e-06 -▁Cancún 4.8378159151566044e-06 -▁Erz 4.8378159151566044e-06 -▁Final 4.8378159151566044e-06 -▁Franco 4.8378159151566044e-06 -▁Rate 4.8378159151566044e-06 -▁Whatever 4.8378159151566044e-06 -▁resign 4.8378159151566044e-06 -41 4.8340274923569515e-06 -bewegungen 4.8340274923569515e-06 -herapie 4.8340274923569515e-06 -male 4.8340274923569515e-06 -orgung 4.8340274923569515e-06 -stock 4.8340274923569515e-06 -▁Erziehung 4.8340274923569515e-06 -▁Flüchtlingen 4.8340274923569515e-06 -▁Gesetzgebungs 4.8340274923569515e-06 -▁Präsent 4.8340274923569515e-06 -▁erwarte 4.8340274923569515e-06 -▁herange 4.8340274923569515e-06 -▁mediterran 4.8340274923569515e-06 -etry 4.8302390695572986e-06 -freies 4.8302390695572986e-06 -ikers 4.8302390695572986e-06 -▁Dolomites 4.8302390695572986e-06 -▁Grundprinzip 4.8302390695572986e-06 -▁allt 4.8302390695572986e-06 -▁angemessener 4.8302390695572986e-06 -▁antique 4.8302390695572986e-06 -▁bathing 4.8302390695572986e-06 -▁mes 4.8302390695572986e-06 -▁quarters 4.8302390695572986e-06 -▁zoom 4.8302390695572986e-06 -asy 4.8264506467576458e-06 -icker 4.8264506467576458e-06 -kept 4.8264506467576458e-06 -kit 4.8264506467576458e-06 -schub 4.8264506467576458e-06 -ändige 4.8264506467576458e-06 -▁Bestrebungen 4.8264506467576458e-06 -▁Cardiff 4.8264506467576458e-06 -▁Dür 4.8264506467576458e-06 -▁Gewerk 4.8264506467576458e-06 -▁Weste 4.8264506467576458e-06 -▁einsetz 4.8264506467576458e-06 -▁encourages 4.8264506467576458e-06 -▁fridge 4.8264506467576458e-06 -▁versetzen 4.8264506467576458e-06 --res 4.8226622239579937e-06 -62 4.8226622239579937e-06 -jen 4.8226622239579937e-06 -▁Durchführungs 4.8226622239579937e-06 -▁Meas 4.8226622239579937e-06 -▁attacked 4.8226622239579937e-06 -▁confidential 4.8226622239579937e-06 -▁enforce 4.8226622239579937e-06 -▁morgens 4.8226622239579937e-06 -▁optimiert 4.8226622239579937e-06 -▁pflegen 4.8226622239579937e-06 -▁ras 4.8226622239579937e-06 -▁vierte 4.8226622239579937e-06 --Gesch 4.8188738011583408e-06 -bird 4.8188738011583408e-06 -fi 4.8188738011583408e-06 -fun 4.8188738011583408e-06 -füllung 4.8188738011583408e-06 -istall 4.8188738011583408e-06 -mäßigkeit 4.8188738011583408e-06 -rüstung 4.8188738011583408e-06 -stellern 4.8188738011583408e-06 -▁5000 4.8188738011583408e-06 -▁Sender 4.8188738011583408e-06 -▁Verhaltenskodex 4.8188738011583408e-06 -▁automobile 4.8188738011583408e-06 -▁settlements 4.8188738011583408e-06 -▁verwandeln 4.8188738011583408e-06 -▁vorgesehene 4.8188738011583408e-06 -blem 4.8150853783586879e-06 -isierter 4.8150853783586879e-06 -standes 4.8150853783586879e-06 -▁Croatian 4.8150853783586879e-06 -▁Engel 4.8150853783586879e-06 -▁Mannschaft 4.8150853783586879e-06 -▁Zentren 4.8150853783586879e-06 -▁anzuerkennen 4.8150853783586879e-06 -▁convers 4.8150853783586879e-06 -▁donor 4.8150853783586879e-06 -▁organs 4.8150853783586879e-06 -▁regener 4.8150853783586879e-06 -▁vermieten 4.8150853783586879e-06 --Anwendungen 4.811296955559035e-06 -sträger 4.811296955559035e-06 -▁Syr 4.811296955559035e-06 -▁dynamisch 4.811296955559035e-06 -▁hybrid 4.811296955559035e-06 -▁müh 4.811296955559035e-06 -▁refin 4.811296955559035e-06 -bestätigung 4.8075085327593821e-06 -gültig 4.8075085327593821e-06 -oland 4.8075085327593821e-06 -▁10th 4.8075085327593821e-06 -▁Diagn 4.8075085327593821e-06 -▁Erholungs 4.8075085327593821e-06 -▁Five 4.8075085327593821e-06 -▁McCart 4.8075085327593821e-06 -▁Ner 4.8075085327593821e-06 -▁Stabilisierung 4.8075085327593821e-06 -▁Verantwort 4.8075085327593821e-06 -▁nachfolgenden 4.8075085327593821e-06 -▁nick 4.8075085327593821e-06 -▁trick 4.8075085327593821e-06 -eading 4.8037201099597293e-06 -itäre 4.8037201099597293e-06 -▁1900 4.8037201099597293e-06 -▁Iz 4.8037201099597293e-06 -▁Kling 4.8037201099597293e-06 -▁Lyn 4.8037201099597293e-06 -▁Nordirland 4.8037201099597293e-06 -▁encountered 4.8037201099597293e-06 -▁fits 4.8037201099597293e-06 -▁hang 4.8037201099597293e-06 -▁jur 4.8037201099597293e-06 -▁operative 4.8037201099597293e-06 -▁recognises 4.8037201099597293e-06 -▁schriftlichen 4.8037201099597293e-06 -dukt 4.7999316871600764e-06 -reihe 4.7999316871600764e-06 -▁201 4.7999316871600764e-06 -▁Beob 4.7999316871600764e-06 -▁Somalia 4.7999316871600764e-06 -▁Verständ 4.7999316871600764e-06 -▁bedingt 4.7999316871600764e-06 -▁beeindruckende 4.7999316871600764e-06 -▁definitiv 4.7999316871600764e-06 -▁erfr 4.7999316871600764e-06 -▁exzellente 4.7999316871600764e-06 -▁freiwillig 4.7999316871600764e-06 -▁persu 4.7999316871600764e-06 -▁petition 4.7999316871600764e-06 -▁profitiert 4.7999316871600764e-06 -▁versprechen 4.7999316871600764e-06 -hoch 4.7961432643604235e-06 -zio 4.7961432643604235e-06 -▁DO 4.7961432643604235e-06 -▁Nordkorea 4.7961432643604235e-06 -▁Promenade 4.7961432643604235e-06 -▁Taste 4.7961432643604235e-06 -▁Villen 4.7961432643604235e-06 -▁Vorbe 4.7961432643604235e-06 -▁aston 4.7961432643604235e-06 -▁bestehend 4.7961432643604235e-06 -▁lifts 4.7961432643604235e-06 -▁ordering 4.7961432643604235e-06 -gefähr 4.7923548415607714e-06 -igma 4.7923548415607714e-06 -ishop 4.7923548415607714e-06 -▁Anf 4.7923548415607714e-06 -▁Keller 4.7923548415607714e-06 -▁Lit 4.7923548415607714e-06 -▁Magn 4.7923548415607714e-06 -▁Paar 4.7923548415607714e-06 -▁Patient 4.7923548415607714e-06 -▁Pixel 4.7923548415607714e-06 -▁Straß 4.7923548415607714e-06 -▁Toten 4.7923548415607714e-06 -▁aufgetreten 4.7923548415607714e-06 -▁caf 4.7923548415607714e-06 -▁processor 4.7923548415607714e-06 -▁unternehmer 4.7923548415607714e-06 -cco 4.7885664187611185e-06 -etty 4.7885664187611185e-06 -kunft 4.7885664187611185e-06 -modelle 4.7885664187611185e-06 -ätzlich 4.7885664187611185e-06 -▁Begleitung 4.7885664187611185e-06 -▁Fert 4.7885664187611185e-06 -▁Kamp 4.7885664187611185e-06 -▁Marg 4.7885664187611185e-06 -▁Mittagessen 4.7885664187611185e-06 -▁Nachweis 4.7885664187611185e-06 -▁Venet 4.7885664187611185e-06 -▁beam 4.7885664187611185e-06 -▁chips 4.7885664187611185e-06 -▁dioxide 4.7885664187611185e-06 -▁erholen 4.7885664187611185e-06 -▁incons 4.7885664187611185e-06 -arisch 4.7847779959614656e-06 -bildungen 4.7847779959614656e-06 -oints 4.7847779959614656e-06 -ubt 4.7847779959614656e-06 -zyklus 4.7847779959614656e-06 -▁Euratom 4.7847779959614656e-06 -▁Futter 4.7847779959614656e-06 -▁Parking 4.7847779959614656e-06 -▁Wid 4.7847779959614656e-06 -▁kritisieren 4.7847779959614656e-06 -▁Č 4.7847779959614656e-06 -setzungs 4.7809895731618128e-06 -▁Angeboten 4.7809895731618128e-06 -▁Chirac 4.7809895731618128e-06 -▁Famil 4.7809895731618128e-06 -▁Lut 4.7809895731618128e-06 -▁Rohstoffe 4.7809895731618128e-06 -▁Schlüss 4.7809895731618128e-06 -▁alternativen 4.7809895731618128e-06 -▁bass 4.7809895731618128e-06 -▁dub 4.7809895731618128e-06 -▁medizin 4.7809895731618128e-06 -▁safeguards 4.7809895731618128e-06 -▁temporarily 4.7809895731618128e-06 -▁unn 4.7809895731618128e-06 -▁versorgen 4.7809895731618128e-06 --2006 4.77720115036216e-06 -alitäten 4.77720115036216e-06 -uate 4.77720115036216e-06 -useum 4.77720115036216e-06 -▁Deal 4.77720115036216e-06 -▁Estland 4.77720115036216e-06 -▁Sachverständ 4.77720115036216e-06 -▁Streitkräfte 4.77720115036216e-06 -▁Sum 4.77720115036216e-06 -▁aufrufen 4.77720115036216e-06 -▁intensiven 4.77720115036216e-06 -IST 4.773412727562507e-06 -cluded 4.773412727562507e-06 -dar 4.773412727562507e-06 -flügel 4.773412727562507e-06 -itzt 4.773412727562507e-06 -olid 4.773412727562507e-06 -owse 4.773412727562507e-06 -▁Bogen 4.773412727562507e-06 -▁Bru 4.773412727562507e-06 -▁Verpackungs 4.773412727562507e-06 -▁dol 4.773412727562507e-06 -▁erzielte 4.773412727562507e-06 -▁ideology 4.773412727562507e-06 -▁kräft 4.773412727562507e-06 -▁shaped 4.773412727562507e-06 -▁tailor 4.773412727562507e-06 -▁trendy 4.773412727562507e-06 -▁verbind 4.773412727562507e-06 -acter 4.7696243047628541e-06 -antasie 4.7696243047628541e-06 -etre 4.7696243047628541e-06 -iguous 4.7696243047628541e-06 -wogen 4.7696243047628541e-06 -▁Mun 4.7696243047628541e-06 -▁Property 4.7696243047628541e-06 -▁Restaur 4.7696243047628541e-06 -▁beschleunigt 4.7696243047628541e-06 -▁eingenommen 4.7696243047628541e-06 -▁nett 4.7696243047628541e-06 -▁studios 4.7696243047628541e-06 --Qu 4.765835881963202e-06 --line 4.765835881963202e-06 -▁Megap 4.765835881963202e-06 -▁Must 4.765835881963202e-06 -▁Netzwerke 4.765835881963202e-06 -▁Zusammenfassung 4.765835881963202e-06 -▁agieren 4.765835881963202e-06 -▁blow 4.765835881963202e-06 -▁cant 4.765835881963202e-06 -▁elektron 4.765835881963202e-06 -▁liefer 4.765835881963202e-06 -▁unüber 4.765835881963202e-06 -arien 4.7620474591635491e-06 -artial 4.7620474591635491e-06 -ctor 4.7620474591635491e-06 -läufigen 4.7620474591635491e-06 -makers 4.7620474591635491e-06 -▁Kondition 4.7620474591635491e-06 -▁Stadtm 4.7620474591635491e-06 -▁Sänger 4.7620474591635491e-06 -▁ThyssenKrupp 4.7620474591635491e-06 -▁foss 4.7620474591635491e-06 -▁kapital 4.7620474591635491e-06 -▁municipality 4.7620474591635491e-06 -▁neo 4.7620474591635491e-06 -▁scripts 4.7620474591635491e-06 -Point 4.7582590363638962e-06 -insel 4.7582590363638962e-06 -riage 4.7582590363638962e-06 -▁Decision 4.7582590363638962e-06 -▁Häusern 4.7582590363638962e-06 -▁Led 4.7582590363638962e-06 -▁Mön 4.7582590363638962e-06 -▁RES 4.7582590363638962e-06 -▁Scientific 4.7582590363638962e-06 -▁batteries 4.7582590363638962e-06 -▁casual 4.7582590363638962e-06 -▁enhancing 4.7582590363638962e-06 -▁hydro 4.7582590363638962e-06 -▁männ 4.7582590363638962e-06 -▁nan 4.7582590363638962e-06 -▁playground 4.7582590363638962e-06 -▁stri 4.7582590363638962e-06 -▁tyr 4.7582590363638962e-06 -abs 4.7544706135642434e-06 -ankungen 4.7544706135642434e-06 -länd 4.7544706135642434e-06 -twer 4.7544706135642434e-06 -uiten 4.7544706135642434e-06 -örungen 4.7544706135642434e-06 -▁Attra 4.7544706135642434e-06 -▁Birma 4.7544706135642434e-06 -▁Boots 4.7544706135642434e-06 -▁Erhebung 4.7544706135642434e-06 -▁beit 4.7544706135642434e-06 -▁einrichten 4.7544706135642434e-06 -▁flexibler 4.7544706135642434e-06 -▁holder 4.7544706135642434e-06 -▁magazines 4.7544706135642434e-06 -▁merch 4.7544706135642434e-06 -▁underp 4.7544706135642434e-06 -amento 4.7506821907645905e-06 -ellt 4.7506821907645905e-06 -iversität 4.7506821907645905e-06 -kennen 4.7506821907645905e-06 -orking 4.7506821907645905e-06 -unik 4.7506821907645905e-06 -▁10.000 4.7506821907645905e-06 -▁Neuen 4.7506821907645905e-06 -▁economical 4.7506821907645905e-06 -▁impressed 4.7506821907645905e-06 -▁nad 4.7506821907645905e-06 -▁reloc 4.7506821907645905e-06 -▁shell 4.7506821907645905e-06 -▁verbundene 4.7506821907645905e-06 -,1 4.7468937679649376e-06 -imeter 4.7468937679649376e-06 -ologically 4.7468937679649376e-06 -zwecken 4.7468937679649376e-06 -▁Brosch 4.7468937679649376e-06 -▁Eden 4.7468937679649376e-06 -▁Europä 4.7468937679649376e-06 -▁Kug 4.7468937679649376e-06 -▁Romul 4.7468937679649376e-06 -▁floods 4.7468937679649376e-06 -▁responding 4.7468937679649376e-06 -beding 4.7431053451652847e-06 -etrieben 4.7431053451652847e-06 -faktor 4.7431053451652847e-06 -jana 4.7431053451652847e-06 -otted 4.7431053451652847e-06 -tex 4.7431053451652847e-06 -werb 4.7431053451652847e-06 -▁County 4.7431053451652847e-06 -▁Einzelf 4.7431053451652847e-06 -▁Feier 4.7431053451652847e-06 -▁GL 4.7431053451652847e-06 -▁Gebäudes 4.7431053451652847e-06 -▁Klaren 4.7431053451652847e-06 -▁LEG 4.7431053451652847e-06 -▁Sandstrand 4.7431053451652847e-06 -▁Self 4.7431053451652847e-06 -▁categor 4.7431053451652847e-06 -▁evaluated 4.7431053451652847e-06 -▁hervorzuheben 4.7431053451652847e-06 -▁mounted 4.7431053451652847e-06 -▁wachs 4.7431053451652847e-06 -eping 4.7393169223656318e-06 -hire 4.7393169223656318e-06 -schaftlichen 4.7393169223656318e-06 -unterricht 4.7393169223656318e-06 -verhältnis 4.7393169223656318e-06 -zlich 4.7393169223656318e-06 -▁anerkannten 4.7393169223656318e-06 -▁angem 4.7393169223656318e-06 -▁aufgeführten 4.7393169223656318e-06 -▁aufgegeben 4.7393169223656318e-06 -▁betragen 4.7393169223656318e-06 -▁erbringen 4.7393169223656318e-06 -▁gesenkt 4.7393169223656318e-06 -▁konkreter 4.7393169223656318e-06 -▁puls 4.7393169223656318e-06 -▁unzählige 4.7393169223656318e-06 -▁wire 4.7393169223656318e-06 -IEN 4.73552849956598e-06 -exe 4.73552849956598e-06 -hotels 4.73552849956598e-06 -terrasse 4.73552849956598e-06 -▁Astrium 4.73552849956598e-06 -▁Twin 4.73552849956598e-06 -▁Ungleichheit 4.73552849956598e-06 -▁Vertra 4.73552849956598e-06 -▁Wende 4.73552849956598e-06 -▁barg 4.73552849956598e-06 -▁persecution 4.73552849956598e-06 -▁potato 4.73552849956598e-06 -anium 4.7317400767663269e-06 -imon 4.7317400767663269e-06 -ioxid 4.7317400767663269e-06 -undi 4.7317400767663269e-06 -▁Assoc 4.7317400767663269e-06 -▁Bünd 4.7317400767663269e-06 -▁Cort 4.7317400767663269e-06 -▁Franzosen 4.7317400767663269e-06 -▁Konsolidierung 4.7317400767663269e-06 -▁Nutzern 4.7317400767663269e-06 -▁enemies 4.7317400767663269e-06 -▁imagination 4.7317400767663269e-06 -▁ratify 4.7317400767663269e-06 -▁stimulus 4.7317400767663269e-06 -ehand 4.727951653966674e-06 -icable 4.727951653966674e-06 -▁Bandbreite 4.727951653966674e-06 -▁Beobachtungs 4.727951653966674e-06 -▁attitudes 4.727951653966674e-06 -▁befreit 4.727951653966674e-06 -▁separat 4.727951653966674e-06 -ICA 4.7241632311670211e-06 -emos 4.7241632311670211e-06 -ieth 4.7241632311670211e-06 -iè 4.7241632311670211e-06 -▁Champions 4.7241632311670211e-06 -▁Energiem 4.7241632311670211e-06 -▁Liechtenstein 4.7241632311670211e-06 -▁Pfeiler 4.7241632311670211e-06 -▁beauftragt 4.7241632311670211e-06 -▁erhielten 4.7241632311670211e-06 -▁stützt 4.7241632311670211e-06 -▁verantwortungsbew 4.7241632311670211e-06 -heet 4.7203748083673682e-06 -itä 4.7203748083673682e-06 -offt 4.7203748083673682e-06 -spot 4.7203748083673682e-06 -▁Harmonie 4.7203748083673682e-06 -▁Implement 4.7203748083673682e-06 -▁Orth 4.7203748083673682e-06 -▁Partition 4.7203748083673682e-06 -▁Spannung 4.7203748083673682e-06 -▁Tragen 4.7203748083673682e-06 -▁broadly 4.7203748083673682e-06 -▁conducting 4.7203748083673682e-06 -▁correction 4.7203748083673682e-06 -▁fügt 4.7203748083673682e-06 -▁neighborhood 4.7203748083673682e-06 -▁remed 4.7203748083673682e-06 -▁responded 4.7203748083673682e-06 --Im 4.7165863855677153e-06 -athan 4.7165863855677153e-06 -dring 4.7165863855677153e-06 -gelegenen 4.7165863855677153e-06 -opez 4.7165863855677153e-06 -zulassen 4.7165863855677153e-06 -▁Button 4.7165863855677153e-06 -▁Disziplin 4.7165863855677153e-06 -▁Einmal 4.7165863855677153e-06 -▁Try 4.7165863855677153e-06 -▁behindern 4.7165863855677153e-06 -▁democratically 4.7165863855677153e-06 -▁gelegentlich 4.7165863855677153e-06 -▁gewaltige 4.7165863855677153e-06 -▁ggf. 4.7165863855677153e-06 -▁stecken 4.7165863855677153e-06 -.m 4.7127979627680624e-06 -ading 4.7127979627680624e-06 -anol 4.7127979627680624e-06 -übertragung 4.7127979627680624e-06 -▁Beschwerde 4.7127979627680624e-06 -▁Cape 4.7127979627680624e-06 -▁EW 4.7127979627680624e-06 -▁ble 4.7127979627680624e-06 -▁burning 4.7127979627680624e-06 -▁dependen 4.7127979627680624e-06 -▁irgendwann 4.7127979627680624e-06 -▁maßgeblich 4.7127979627680624e-06 -▁pav 4.7127979627680624e-06 -▁privileges 4.7127979627680624e-06 -▁races 4.7127979627680624e-06 -▁routine 4.7127979627680624e-06 -▁vorzugehen 4.7127979627680624e-06 -kopf 4.7090095399684095e-06 -▁ALDE 4.7090095399684095e-06 -▁Bedrohungen 4.7090095399684095e-06 -▁Fett 4.7090095399684095e-06 -▁Match 4.7090095399684095e-06 -▁Monitor 4.7090095399684095e-06 -▁Umstellung 4.7090095399684095e-06 -▁Wicht 4.7090095399684095e-06 -▁Zeilen 4.7090095399684095e-06 -▁assessing 4.7090095399684095e-06 -▁captured 4.7090095399684095e-06 -▁förder 4.7090095399684095e-06 -▁gedruckt 4.7090095399684095e-06 -▁inspections 4.7090095399684095e-06 -▁ses 4.7090095399684095e-06 -endungen 4.7052211171687575e-06 -uri 4.7052211171687575e-06 -▁Algerien 4.7052211171687575e-06 -▁Begrenzung 4.7052211171687575e-06 -▁Däm 4.7052211171687575e-06 -▁Niger 4.7052211171687575e-06 -▁Perfect 4.7052211171687575e-06 -▁buch 4.7052211171687575e-06 -▁clo 4.7052211171687575e-06 -▁fer 4.7052211171687575e-06 -▁übernahm 4.7052211171687575e-06 -access 4.7014326943691046e-06 -amme 4.7014326943691046e-06 -essim 4.7014326943691046e-06 -iquen 4.7014326943691046e-06 -uhl 4.7014326943691046e-06 -▁Father 4.7014326943691046e-06 -▁Zoom 4.7014326943691046e-06 -▁continuity 4.7014326943691046e-06 -▁geschah 4.7014326943691046e-06 -▁preserving 4.7014326943691046e-06 -▁titles 4.7014326943691046e-06 -quir 4.6976442715694517e-06 -roup 4.6976442715694517e-06 -ächten 4.6976442715694517e-06 -▁Akademie 4.6976442715694517e-06 -▁Designs 4.6976442715694517e-06 -▁Subsidiaritätsprinzip 4.6976442715694517e-06 -▁Vollständigkeit 4.6976442715694517e-06 -▁disadvantage 4.6976442715694517e-06 -▁kW 4.6976442715694517e-06 -▁visibility 4.6976442715694517e-06 -▁würd 4.6976442715694517e-06 --Programms 4.6938558487697988e-06 -poses 4.6938558487697988e-06 -uta 4.6938558487697988e-06 -verträge 4.6938558487697988e-06 -ässige 4.6938558487697988e-06 -▁Mes 4.6938558487697988e-06 -▁Portfolio 4.6938558487697988e-06 -▁Thessaloniki 4.6938558487697988e-06 -▁YOU 4.6938558487697988e-06 -▁höherer 4.6938558487697988e-06 -▁sau 4.6938558487697988e-06 -▁stetig 4.6938558487697988e-06 -▁understandable 4.6938558487697988e-06 -AX 4.6900674259701459e-06 -abend 4.6900674259701459e-06 -fect 4.6900674259701459e-06 -κ 4.6900674259701459e-06 -▁Auß 4.6900674259701459e-06 -▁Economy 4.6900674259701459e-06 -▁Ermitt 4.6900674259701459e-06 -▁Texten 4.6900674259701459e-06 -▁Zeitalter 4.6900674259701459e-06 -▁astronom 4.6900674259701459e-06 -▁fisher 4.6900674259701459e-06 -▁folgte 4.6900674259701459e-06 -▁genetisch 4.6900674259701459e-06 -▁imbalances 4.6900674259701459e-06 -▁kontr 4.6900674259701459e-06 -anzierung 4.686279003170493e-06 -ighth 4.686279003170493e-06 -plung 4.686279003170493e-06 -sport 4.686279003170493e-06 -werter 4.686279003170493e-06 -▁Jane 4.686279003170493e-06 -▁Takt 4.686279003170493e-06 -▁WordPress 4.686279003170493e-06 -▁erfolg 4.686279003170493e-06 -▁experiments 4.686279003170493e-06 -▁inappropriate 4.686279003170493e-06 -▁invasion 4.686279003170493e-06 -▁mand 4.686279003170493e-06 -easant 4.68249058037084e-06 -iein 4.68249058037084e-06 -sprechen 4.68249058037084e-06 -stär 4.68249058037084e-06 -thing 4.68249058037084e-06 -town 4.68249058037084e-06 -▁Bestände 4.68249058037084e-06 -▁Schad 4.68249058037084e-06 -▁Schauspieler 4.68249058037084e-06 -▁aufgegriffen 4.68249058037084e-06 -▁institute 4.68249058037084e-06 -▁iranischen 4.68249058037084e-06 -▁radi 4.68249058037084e-06 -▁rocks 4.68249058037084e-06 -▁verfolg 4.68249058037084e-06 -euert 4.6787021575711872e-06 -fo 4.6787021575711872e-06 -gorith 4.6787021575711872e-06 -ither 4.6787021575711872e-06 -rative 4.6787021575711872e-06 -▁1.000 4.6787021575711872e-06 -▁14th 4.6787021575711872e-06 -▁II. 4.6787021575711872e-06 -▁Unabhäng 4.6787021575711872e-06 -▁coin 4.6787021575711872e-06 -▁exacerb 4.6787021575711872e-06 -▁kommunistischen 4.6787021575711872e-06 -▁outermost 4.6787021575711872e-06 -▁pushing 4.6787021575711872e-06 -▁versuchte 4.6787021575711872e-06 --wing 4.6749137347715352e-06 -antwort 4.6749137347715352e-06 -nam 4.6749137347715352e-06 -reuen 4.6749137347715352e-06 -▁Demo 4.6749137347715352e-06 -▁Eins 4.6749137347715352e-06 -▁Enth 4.6749137347715352e-06 -▁Gam 4.6749137347715352e-06 -▁Wollen 4.6749137347715352e-06 -▁erstklassige 4.6749137347715352e-06 -▁estimate 4.6749137347715352e-06 -▁nachfol 4.6749137347715352e-06 -▁nav 4.6749137347715352e-06 -▁quoted 4.6749137347715352e-06 -RS 4.6711253119718823e-06 -UE 4.6711253119718823e-06 -elbe 4.6711253119718823e-06 -igm 4.6711253119718823e-06 -isher 4.6711253119718823e-06 -stitut 4.6711253119718823e-06 -unters 4.6711253119718823e-06 -ärten 4.6711253119718823e-06 -▁Adress 4.6711253119718823e-06 -▁Fertig 4.6711253119718823e-06 -▁Gitar 4.6711253119718823e-06 -▁Palais 4.6711253119718823e-06 -▁Parte 4.6711253119718823e-06 -▁Reichweite 4.6711253119718823e-06 -▁Warm 4.6711253119718823e-06 -▁deployment 4.6711253119718823e-06 -▁entnehmen 4.6711253119718823e-06 -▁fossil 4.6711253119718823e-06 -▁genehm 4.6711253119718823e-06 -▁gründete 4.6711253119718823e-06 -▁measurements 4.6711253119718823e-06 -▁timing 4.6711253119718823e-06 -▁verlängern 4.6711253119718823e-06 -ofen 4.6673368891722294e-06 -tegr 4.6673368891722294e-06 -▁Ausschüssen 4.6673368891722294e-06 -▁Verschlechterung 4.6673368891722294e-06 -▁Wikitravel-Benutzer 4.6673368891722294e-06 -▁anschauen 4.6673368891722294e-06 -▁closest 4.6673368891722294e-06 -▁demnächst 4.6673368891722294e-06 -▁demselben 4.6673368891722294e-06 -▁directors 4.6673368891722294e-06 -▁emer 4.6673368891722294e-06 -▁juristischen 4.6673368891722294e-06 -gebend 4.6635484663725765e-06 -oured 4.6635484663725765e-06 -pflanzen 4.6635484663725765e-06 -▁Euro-Mediterranean 4.6635484663725765e-06 -▁Himm 4.6635484663725765e-06 -▁Never 4.6635484663725765e-06 -▁activists 4.6635484663725765e-06 -▁aven 4.6635484663725765e-06 -▁injury 4.6635484663725765e-06 -▁jemanden 4.6635484663725765e-06 -▁sow 4.6635484663725765e-06 -▁verläuft 4.6635484663725765e-06 --Part 4.6597600435729236e-06 -55 4.6597600435729236e-06 -beck 4.6597600435729236e-06 -ocon 4.6597600435729236e-06 -▁Reisenden 4.6597600435729236e-06 -▁aufrechtzuerhalten 4.6597600435729236e-06 -▁begeben 4.6597600435729236e-06 -▁gemachten 4.6597600435729236e-06 -▁shad 4.6597600435729236e-06 -▁spam 4.6597600435729236e-06 -▁yacht 4.6597600435729236e-06 -IES 4.6559716207732707e-06 -OUR 4.6559716207732707e-06 -ifs 4.6559716207732707e-06 -nov 4.6559716207732707e-06 -stufe 4.6559716207732707e-06 -▁Eye 4.6559716207732707e-06 -▁Fotografie 4.6559716207732707e-06 -▁Money 4.6559716207732707e-06 -▁Seven 4.6559716207732707e-06 -▁differenz 4.6559716207732707e-06 -▁escal 4.6559716207732707e-06 -▁gefertigt 4.6559716207732707e-06 -▁underest 4.6559716207732707e-06 -▁Überwindung 4.6559716207732707e-06 -▁â 4.6559716207732707e-06 -lebt 4.6521831979736178e-06 -▁Hüt 4.6521831979736178e-06 -▁Mercure 4.6521831979736178e-06 -▁cand 4.6521831979736178e-06 -▁holen 4.6521831979736178e-06 -▁nördlich 4.6521831979736178e-06 -▁pretext 4.6521831979736178e-06 -▁propag 4.6521831979736178e-06 -▁unh 4.6521831979736178e-06 -▁zerstör 4.6521831979736178e-06 -finanz 4.6483947751739649e-06 -stan 4.6483947751739649e-06 -tries 4.6483947751739649e-06 -zung 4.6483947751739649e-06 -▁Authent 4.6483947751739649e-06 -▁Body 4.6483947751739649e-06 -▁Geburts 4.6483947751739649e-06 -▁PHARE 4.6483947751739649e-06 -▁QuarkXPress 4.6483947751739649e-06 -▁attentive 4.6483947751739649e-06 -▁credits 4.6483947751739649e-06 -▁folk 4.6483947751739649e-06 -▁ries 4.6483947751739649e-06 -▁varying 4.6483947751739649e-06 -▁Übereinkunft 4.6483947751739649e-06 --Bus 4.6446063523743129e-06 -FOR 4.6446063523743129e-06 -aires 4.6446063523743129e-06 -ât 4.6446063523743129e-06 -▁Angestellten 4.6446063523743129e-06 -▁Informatik 4.6446063523743129e-06 -▁Tav 4.6446063523743129e-06 -▁Workshop 4.6446063523743129e-06 -▁abschließ 4.6446063523743129e-06 -▁druck 4.6446063523743129e-06 -▁geeinigt 4.6446063523743129e-06 -▁gemeint 4.6446063523743129e-06 -▁problematic 4.6446063523743129e-06 -▁scheinbar 4.6446063523743129e-06 -gehalt 4.64081792957466e-06 -▁1.6 4.64081792957466e-06 -▁Anleitung 4.64081792957466e-06 -▁Handhabung 4.64081792957466e-06 -▁Klavier 4.64081792957466e-06 -▁Produkts 4.64081792957466e-06 -▁Unit 4.64081792957466e-06 -▁bedaure 4.64081792957466e-06 -▁einbeziehen 4.64081792957466e-06 -▁particip 4.64081792957466e-06 -▁statutory 4.64081792957466e-06 -▁unbed 4.64081792957466e-06 --Welt 4.6370295067750071e-06 -berge 4.6370295067750071e-06 -illes 4.6370295067750071e-06 -tax 4.6370295067750071e-06 -urses 4.6370295067750071e-06 -▁Angebots 4.6370295067750071e-06 -▁Catalunya 4.6370295067750071e-06 -▁Schall 4.6370295067750071e-06 -▁Sicherstellung 4.6370295067750071e-06 -▁Vergabe 4.6370295067750071e-06 -▁artic 4.6370295067750071e-06 -▁auszur 4.6370295067750071e-06 -▁depr 4.6370295067750071e-06 -▁devote 4.6370295067750071e-06 -▁kostenfreien 4.6370295067750071e-06 -▁metals 4.6370295067750071e-06 -▁viruses 4.6370295067750071e-06 -▁weiterentwickelt 4.6370295067750071e-06 -agne 4.6332410839753542e-06 -chio 4.6332410839753542e-06 -gewiss 4.6332410839753542e-06 -kapazitäten 4.6332410839753542e-06 -quiry 4.6332410839753542e-06 -▁Bod 4.6332410839753542e-06 -▁Rezept 4.6332410839753542e-06 -▁Tauchen 4.6332410839753542e-06 -▁doct 4.6332410839753542e-06 -▁gathering 4.6332410839753542e-06 -▁retention 4.6332410839753542e-06 -▁seekers 4.6332410839753542e-06 -▁vibr 4.6332410839753542e-06 --east 4.6294526611757013e-06 -meer 4.6294526611757013e-06 -▁Dell 4.6294526611757013e-06 -▁HT 4.6294526611757013e-06 -▁Handbuch 4.6294526611757013e-06 -▁Machine 4.6294526611757013e-06 -▁Schätzungen 4.6294526611757013e-06 -▁Spl 4.6294526611757013e-06 -▁competitions 4.6294526611757013e-06 -▁diplomacy 4.6294526611757013e-06 -▁dropped 4.6294526611757013e-06 -▁interconn 4.6294526611757013e-06 -▁launching 4.6294526611757013e-06 -▁recreation 4.6294526611757013e-06 -▁vegetable 4.6294526611757013e-06 -▁widerspiegelt 4.6294526611757013e-06 --Präsident 4.6256642383760484e-06 -glas 4.6256642383760484e-06 -hrend 4.6256642383760484e-06 -ungsverhältnis 4.6256642383760484e-06 -wählen 4.6256642383760484e-06 -▁Angola 4.6256642383760484e-06 -▁Franc 4.6256642383760484e-06 -▁Jobs 4.6256642383760484e-06 -▁Parc 4.6256642383760484e-06 -▁Schein 4.6256642383760484e-06 -▁betrieb 4.6256642383760484e-06 -▁compet 4.6256642383760484e-06 -▁erhöhte 4.6256642383760484e-06 -▁genocide 4.6256642383760484e-06 -▁öffent 4.6256642383760484e-06 -archie 4.6218758155763955e-06 -eutet 4.6218758155763955e-06 -hus 4.6218758155763955e-06 -owina 4.6218758155763955e-06 -ündeten 4.6218758155763955e-06 -▁6th 4.6218758155763955e-06 -▁Clear 4.6218758155763955e-06 -▁Living 4.6218758155763955e-06 -▁Strafe 4.6218758155763955e-06 -▁Ursprün 4.6218758155763955e-06 -▁Wat 4.6218758155763955e-06 -▁conscience 4.6218758155763955e-06 -▁cooperative 4.6218758155763955e-06 -▁industr 4.6218758155763955e-06 -▁initiate 4.6218758155763955e-06 -▁seinerzeit 4.6218758155763955e-06 -▁untersagt 4.6218758155763955e-06 -KK 4.6180873927767427e-06 -deckung 4.6180873927767427e-06 -errungen 4.6180873927767427e-06 -former 4.6180873927767427e-06 -▁3,5 4.6180873927767427e-06 -▁Anfänger 4.6180873927767427e-06 -▁Begleit 4.6180873927767427e-06 -▁Evalu 4.6180873927767427e-06 -▁Fahren 4.6180873927767427e-06 -▁Soll 4.6180873927767427e-06 -▁einheimischen 4.6180873927767427e-06 -▁feeding 4.6180873927767427e-06 -▁hoping 4.6180873927767427e-06 -▁motorcy 4.6180873927767427e-06 -▁verteidigt 4.6180873927767427e-06 -▁Übersetzer 4.6180873927767427e-06 --Partner 4.6142989699770906e-06 -chtung 4.6142989699770906e-06 -fangen 4.6142989699770906e-06 -mans 4.6142989699770906e-06 -optim 4.6142989699770906e-06 -▁Connect 4.6142989699770906e-06 -▁Finanzminister 4.6142989699770906e-06 -▁Rechtsprechung 4.6142989699770906e-06 -▁Turin 4.6142989699770906e-06 -▁eggs 4.6142989699770906e-06 -▁entziehen 4.6142989699770906e-06 -▁fasc 4.6142989699770906e-06 -▁gentle 4.6142989699770906e-06 -▁harder 4.6142989699770906e-06 -▁harsh 4.6142989699770906e-06 -▁hinder 4.6142989699770906e-06 -▁korrigiert 4.6142989699770906e-06 -▁passage 4.6142989699770906e-06 -▁stainless 4.6142989699770906e-06 -ibration 4.6105105471774377e-06 -konto 4.6105105471774377e-06 -ğ 4.6105105471774377e-06 -▁Cleaning 4.6105105471774377e-06 -▁Inc. 4.6105105471774377e-06 -▁Quite 4.6105105471774377e-06 -▁Rotterdam 4.6105105471774377e-06 -▁dan 4.6105105471774377e-06 -▁führ 4.6105105471774377e-06 -▁hosted 4.6105105471774377e-06 -▁kor 4.6105105471774377e-06 -▁laundry 4.6105105471774377e-06 -▁stro 4.6105105471774377e-06 -▁tired 4.6105105471774377e-06 -affe 4.6067221243777848e-06 -kehrungen 4.6067221243777848e-06 -porary 4.6067221243777848e-06 -richtlinien 4.6067221243777848e-06 -twort 4.6067221243777848e-06 -▁Angehörigen 4.6067221243777848e-06 -▁Aussehen 4.6067221243777848e-06 -▁Perl 4.6067221243777848e-06 -▁Tepp 4.6067221243777848e-06 -▁Vital 4.6067221243777848e-06 -▁Wellnessbereich 4.6067221243777848e-06 -▁belastet 4.6067221243777848e-06 -▁privater 4.6067221243777848e-06 -▁unkompl 4.6067221243777848e-06 -▁withdrawn 4.6067221243777848e-06 --30 4.6029337015781319e-06 --Management 4.6029337015781319e-06 -bezogenen 4.6029337015781319e-06 -chern 4.6029337015781319e-06 -unterschied 4.6029337015781319e-06 -ächtige 4.6029337015781319e-06 -▁-K 4.6029337015781319e-06 -▁Arbeitsprogramm 4.6029337015781319e-06 -▁Articles 4.6029337015781319e-06 -▁Chair 4.6029337015781319e-06 -▁Garage 4.6029337015781319e-06 -▁Inneren 4.6029337015781319e-06 -▁Stille 4.6029337015781319e-06 -▁Turnier 4.6029337015781319e-06 -▁backup 4.6029337015781319e-06 -▁forever 4.6029337015781319e-06 -▁heter 4.6029337015781319e-06 -▁schaden 4.6029337015781319e-06 -▁vermehr 4.6029337015781319e-06 -▁öst 4.6029337015781319e-06 -oner 4.599145278778479e-06 -ortium 4.599145278778479e-06 -▁Cafés 4.599145278778479e-06 -▁Propag 4.599145278778479e-06 -▁Symph 4.599145278778479e-06 -▁TEN 4.599145278778479e-06 -▁Verkehrsmitteln 4.599145278778479e-06 -▁Wine 4.599145278778479e-06 -▁escap 4.599145278778479e-06 -▁precautionary 4.599145278778479e-06 -▁prolong 4.599145278778479e-06 -▁rival 4.599145278778479e-06 -▁seed 4.599145278778479e-06 -▁widerspiegeln 4.599145278778479e-06 -alon 4.5953568559788261e-06 -dor 4.5953568559788261e-06 -hness 4.5953568559788261e-06 -orfen 4.5953568559788261e-06 -▁Container 4.5953568559788261e-06 -▁Einleitung 4.5953568559788261e-06 -▁Fü 4.5953568559788261e-06 -▁Rum 4.5953568559788261e-06 -▁Soph 4.5953568559788261e-06 -▁Toskana 4.5953568559788261e-06 -▁embargo 4.5953568559788261e-06 -▁entsprechender 4.5953568559788261e-06 -▁erzählen 4.5953568559788261e-06 -▁konsequ 4.5953568559788261e-06 -▁manually 4.5953568559788261e-06 -▁positively 4.5953568559788261e-06 -▁seg 4.5953568559788261e-06 -▁untr 4.5953568559788261e-06 -▁veranst 4.5953568559788261e-06 -▁Überwachungs 4.5953568559788261e-06 --Plan 4.5915684331791733e-06 --Res 4.5915684331791733e-06 -imar 4.5915684331791733e-06 -materialien 4.5915684331791733e-06 -▁Hit 4.5915684331791733e-06 -▁Konferenz- 4.5915684331791733e-06 -▁Nachtleben 4.5915684331791733e-06 -▁Step 4.5915684331791733e-06 -▁dirty 4.5915684331791733e-06 -▁fundament 4.5915684331791733e-06 -▁hervorr 4.5915684331791733e-06 -▁lending 4.5915684331791733e-06 -▁miß 4.5915684331791733e-06 -▁multilater 4.5915684331791733e-06 -▁versorgt 4.5915684331791733e-06 -▁verwandelt 4.5915684331791733e-06 -▁wheels 4.5915684331791733e-06 -atastrophe 4.58778001037952e-06 -erra 4.58778001037952e-06 -fahrzeugen 4.58778001037952e-06 -▁Below 4.58778001037952e-06 -▁Competition 4.58778001037952e-06 -▁Friends 4.58778001037952e-06 -▁Polo 4.58778001037952e-06 -▁Süß 4.58778001037952e-06 -▁billigen 4.58778001037952e-06 -▁offensicht 4.58778001037952e-06 -▁pursuit 4.58778001037952e-06 -▁schuldig 4.58778001037952e-06 -▁unch 4.58778001037952e-06 -OY 4.5839915875798683e-06 -VER 4.5839915875798683e-06 -atering 4.5839915875798683e-06 -ograf 4.5839915875798683e-06 -stellbar 4.5839915875798683e-06 -▁1980s 4.5839915875798683e-06 -▁Adjust 4.5839915875798683e-06 -▁Napole 4.5839915875798683e-06 -▁Pyren 4.5839915875798683e-06 -▁listing 4.5839915875798683e-06 -▁potenz 4.5839915875798683e-06 -▁unterschiedlichsten 4.5839915875798683e-06 -▁widmet 4.5839915875798683e-06 -M. 4.5802031647802154e-06 -knüp 4.5802031647802154e-06 -oney 4.5802031647802154e-06 -potenzial 4.5802031647802154e-06 -ägliche 4.5802031647802154e-06 -▁1,000 4.5802031647802154e-06 -▁Feucht 4.5802031647802154e-06 -▁Hack 4.5802031647802154e-06 -▁attending 4.5802031647802154e-06 -▁beunruhigt 4.5802031647802154e-06 -▁commodity 4.5802031647802154e-06 -▁compiled 4.5802031647802154e-06 -▁establishments 4.5802031647802154e-06 -▁sexuelle 4.5802031647802154e-06 -▁successes 4.5802031647802154e-06 -ENS 4.5764147419805625e-06 -irmen 4.5764147419805625e-06 -kos 4.5764147419805625e-06 -vorhaben 4.5764147419805625e-06 -ändlern 4.5764147419805625e-06 -▁2014 4.5764147419805625e-06 -▁Crown 4.5764147419805625e-06 -▁E-mail 4.5764147419805625e-06 -▁Haltestelle 4.5764147419805625e-06 -▁Ign 4.5764147419805625e-06 -▁Istr 4.5764147419805625e-06 -▁brit 4.5764147419805625e-06 -▁erstmal 4.5764147419805625e-06 -▁iT 4.5764147419805625e-06 -▁seam 4.5764147419805625e-06 -▁template 4.5764147419805625e-06 -▁unzureich 4.5764147419805625e-06 -▁¿ 4.5764147419805625e-06 -aufsicht 4.57262631918091e-06 -itual 4.57262631918091e-06 -rott 4.57262631918091e-06 -▁112 4.57262631918091e-06 -▁Ersten 4.57262631918091e-06 -▁Gremien 4.57262631918091e-06 -▁Kino 4.57262631918091e-06 -▁Kontinents 4.57262631918091e-06 -▁MEP 4.57262631918091e-06 -▁Seminare 4.57262631918091e-06 -▁Ship 4.57262631918091e-06 -▁lifelong 4.57262631918091e-06 -▁schr 4.57262631918091e-06 -arbeiteten 4.5688378963812568e-06 -enzo 4.5688378963812568e-06 -ombard 4.5688378963812568e-06 -verk 4.5688378963812568e-06 -▁Aufstellung 4.5688378963812568e-06 -▁Blu 4.5688378963812568e-06 -▁Brea 4.5688378963812568e-06 -▁Einerseits 4.5688378963812568e-06 -▁Internets 4.5688378963812568e-06 -▁Präsidentschafts 4.5688378963812568e-06 -▁Rosa 4.5688378963812568e-06 -▁Schriftsteller 4.5688378963812568e-06 -▁burdens 4.5688378963812568e-06 -▁horiz 4.5688378963812568e-06 -▁osteuropäischen 4.5688378963812568e-06 -▁schwäch 4.5688378963812568e-06 -▁swim 4.5688378963812568e-06 -acional 4.5650494735816039e-06 -attern 4.5650494735816039e-06 -orno 4.5650494735816039e-06 -schritt 4.5650494735816039e-06 -umgebung 4.5650494735816039e-06 -zucht 4.5650494735816039e-06 -üne 4.5650494735816039e-06 -▁Agent 4.5650494735816039e-06 -▁Einsparungen 4.5650494735816039e-06 -▁Gedächtnis 4.5650494735816039e-06 -▁beheben 4.5650494735816039e-06 -▁bookings 4.5650494735816039e-06 -▁disappeared 4.5650494735816039e-06 -▁oberste 4.5650494735816039e-06 -▁stresses 4.5650494735816039e-06 -▁vere 4.5650494735816039e-06 -ologen 4.561261050781951e-06 -technologie 4.561261050781951e-06 -trolled 4.561261050781951e-06 -ummern 4.561261050781951e-06 -▁CIA 4.561261050781951e-06 -▁Geschäftsreisende 4.561261050781951e-06 -▁Langzeit 4.561261050781951e-06 -▁Script 4.561261050781951e-06 -▁anymore 4.561261050781951e-06 -▁coordinating 4.561261050781951e-06 -▁deliberately 4.561261050781951e-06 -▁dossier 4.561261050781951e-06 -▁flexiblen 4.561261050781951e-06 -▁großzügig 4.561261050781951e-06 -▁instances 4.561261050781951e-06 -▁mirr 4.561261050781951e-06 -▁solarium 4.561261050781951e-06 -din 4.5574726279822989e-06 -lingen 4.5574726279822989e-06 -▁230 4.5574726279822989e-06 -▁Aufteilung 4.5574726279822989e-06 -▁Fraktions 4.5574726279822989e-06 -▁Kommissars 4.5574726279822989e-06 -▁betreffende 4.5574726279822989e-06 -▁dam 4.5574726279822989e-06 -▁ehren 4.5574726279822989e-06 -▁ganzheit 4.5574726279822989e-06 -▁missbrauch 4.5574726279822989e-06 -▁myster 4.5574726279822989e-06 -▁quasi 4.5574726279822989e-06 -▁rice 4.5574726279822989e-06 -.10 4.553684205182646e-06 -klicken 4.553684205182646e-06 -mietung 4.553684205182646e-06 -sicherung 4.553684205182646e-06 -▁Check-In 4.553684205182646e-06 -▁Chor 4.553684205182646e-06 -▁Exempl 4.553684205182646e-06 -▁GIMP 4.553684205182646e-06 -▁Nest 4.553684205182646e-06 -▁Stoffen 4.553684205182646e-06 -▁Type 4.553684205182646e-06 -▁conven 4.553684205182646e-06 -▁defines 4.553684205182646e-06 -▁entfernten 4.553684205182646e-06 -▁organising 4.553684205182646e-06 -▁packed 4.553684205182646e-06 -▁toys 4.553684205182646e-06 -▁vertiefen 4.553684205182646e-06 -füg 4.5498957823829931e-06 -ulette 4.5498957823829931e-06 -veloped 4.5498957823829931e-06 -widr 4.5498957823829931e-06 -zugang 4.5498957823829931e-06 -▁Analyt 4.5498957823829931e-06 -▁ELDR 4.5498957823829931e-06 -▁Erkrankungen 4.5498957823829931e-06 -▁Inspir 4.5498957823829931e-06 -▁Klingon 4.5498957823829931e-06 -▁Schriften 4.5498957823829931e-06 -▁Temple 4.5498957823829931e-06 -▁dealers 4.5498957823829931e-06 -▁disarm 4.5498957823829931e-06 -▁einschlägigen 4.5498957823829931e-06 -▁fairer 4.5498957823829931e-06 -▁portable 4.5498957823829931e-06 -▁princip 4.5498957823829931e-06 -▁Übersch 4.5498957823829931e-06 --Form 4.54610735958334e-06 --Man 4.54610735958334e-06 -verantwortung 4.54610735958334e-06 -▁Ansehen 4.54610735958334e-06 -▁Hochschule 4.54610735958334e-06 -▁Laundry 4.54610735958334e-06 -▁Ministerpräsidenten 4.54610735958334e-06 -▁aufgeben 4.54610735958334e-06 -▁collecting 4.54610735958334e-06 -▁corrupt 4.54610735958334e-06 -▁getroffenen 4.54610735958334e-06 -▁hinnehmen 4.54610735958334e-06 -▁marking 4.54610735958334e-06 -▁mines 4.54610735958334e-06 -▁mitzuteilen 4.54610735958334e-06 -▁mos 4.54610735958334e-06 -▁unannehm 4.54610735958334e-06 --X 4.5423189367836874e-06 -CAD 4.5423189367836874e-06 -fast 4.5423189367836874e-06 -ilog 4.5423189367836874e-06 -▁Abfahrt 4.5423189367836874e-06 -▁Architekt 4.5423189367836874e-06 -▁Buy 4.5423189367836874e-06 -▁Unregelmäßigkeiten 4.5423189367836874e-06 -▁maler 4.5423189367836874e-06 -▁mortg 4.5423189367836874e-06 -▁unternimmt 4.5423189367836874e-06 -▁verursachten 4.5423189367836874e-06 --Dienst 4.5385305139840345e-06 -82 4.5385305139840345e-06 -komponenten 4.5385305139840345e-06 -zept 4.5385305139840345e-06 -▁Bemühen 4.5385305139840345e-06 -▁Egyptian 4.5385305139840345e-06 -▁Erasmus 4.5385305139840345e-06 -▁Ferrari 4.5385305139840345e-06 -▁Gardens 4.5385305139840345e-06 -▁Gehör 4.5385305139840345e-06 -▁Momente 4.5385305139840345e-06 -▁Scottish 4.5385305139840345e-06 -▁gt 4.5385305139840345e-06 -▁unauf 4.5385305139840345e-06 -▁westliche 4.5385305139840345e-06 --Sicherheits 4.5347420911843816e-06 --Unter 4.5347420911843816e-06 -hänge 4.5347420911843816e-06 -olis 4.5347420911843816e-06 -▁Alexand 4.5347420911843816e-06 -▁Cambod 4.5347420911843816e-06 -▁Kolumbien 4.5347420911843816e-06 -▁Sic 4.5347420911843816e-06 -▁Wi 4.5347420911843816e-06 -▁detected 4.5347420911843816e-06 -▁eliminating 4.5347420911843816e-06 -▁emotions 4.5347420911843816e-06 -▁feasible 4.5347420911843816e-06 -cup 4.5309536683847287e-06 -oyer 4.5309536683847287e-06 -uchtet 4.5309536683847287e-06 -ufer 4.5309536683847287e-06 -▁2.1 4.5309536683847287e-06 -▁7th 4.5309536683847287e-06 -▁BV 4.5309536683847287e-06 -▁C4-0 4.5309536683847287e-06 -▁Instabilität 4.5309536683847287e-06 -▁Mind 4.5309536683847287e-06 -▁OSCE 4.5309536683847287e-06 -▁OSZE 4.5309536683847287e-06 -▁Stufen 4.5309536683847287e-06 -▁accessibility 4.5309536683847287e-06 -▁accompany 4.5309536683847287e-06 -▁buffer 4.5309536683847287e-06 -▁durchschnittlich 4.5309536683847287e-06 -▁educ 4.5309536683847287e-06 -▁foodstuffs 4.5309536683847287e-06 -▁veranlassen 4.5309536683847287e-06 -▁weakness 4.5309536683847287e-06 --end 4.5271652455850766e-06 -esta 4.5271652455850766e-06 -haushalts 4.5271652455850766e-06 -orously 4.5271652455850766e-06 -▁Chinesen 4.5271652455850766e-06 -▁Darlehen 4.5271652455850766e-06 -▁Diagnose 4.5271652455850766e-06 -▁Freilassung 4.5271652455850766e-06 -▁Zusagen 4.5271652455850766e-06 -▁alien 4.5271652455850766e-06 -▁fulfilling 4.5271652455850766e-06 -▁gewahrt 4.5271652455850766e-06 -▁liberty 4.5271652455850766e-06 -▁self-s 4.5271652455850766e-06 -▁uneingeschränkte 4.5271652455850766e-06 -▁verbindliche 4.5271652455850766e-06 -atical 4.5233768227854238e-06 -genehm 4.5233768227854238e-06 -php 4.5233768227854238e-06 -änglichen 4.5233768227854238e-06 -ärkte 4.5233768227854238e-06 -▁Geltungsbereich 4.5233768227854238e-06 -▁Mold 4.5233768227854238e-06 -▁Schwanger 4.5233768227854238e-06 -▁Vö 4.5233768227854238e-06 -▁einfaches 4.5233768227854238e-06 -▁hochrang 4.5233768227854238e-06 -▁informations 4.5233768227854238e-06 -pfung 4.5195883999857709e-06 -vez 4.5195883999857709e-06 -zutreten 4.5195883999857709e-06 -▁Gemein 4.5195883999857709e-06 -▁Industrien 4.5195883999857709e-06 -▁Init 4.5195883999857709e-06 -▁Nit 4.5195883999857709e-06 -▁Nächte 4.5195883999857709e-06 -▁administrat 4.5195883999857709e-06 -▁anscheinend 4.5195883999857709e-06 -▁obgleich 4.5195883999857709e-06 -▁overseas 4.5195883999857709e-06 -▁weaknesses 4.5195883999857709e-06 -▁wholly 4.5195883999857709e-06 -▁wing 4.5195883999857709e-06 -FL 4.515799977186118e-06 -FORM 4.515799977186118e-06 -chant 4.515799977186118e-06 -managements 4.515799977186118e-06 -organis 4.515799977186118e-06 -▁Daneben 4.515799977186118e-06 -▁Erdgeschoss 4.515799977186118e-06 -▁Gemeinschaftsebene 4.515799977186118e-06 -▁Luftverkehr 4.515799977186118e-06 -▁Niel 4.515799977186118e-06 -▁angekommen 4.515799977186118e-06 -▁banana 4.515799977186118e-06 -▁biologische 4.515799977186118e-06 -▁salon 4.515799977186118e-06 -▁selekt 4.515799977186118e-06 -ao 4.5120115543864651e-06 -iw 4.5120115543864651e-06 -izite 4.5120115543864651e-06 -ktiken 4.5120115543864651e-06 -pielen 4.5120115543864651e-06 -ziell 4.5120115543864651e-06 -▁1956 4.5120115543864651e-06 -▁Born 4.5120115543864651e-06 -▁abgedeckt 4.5120115543864651e-06 -▁gehör 4.5120115543864651e-06 -▁gänzlich 4.5120115543864651e-06 -▁hochqual 4.5120115543864651e-06 -L. 4.5082231315868122e-06 -ahm 4.5082231315868122e-06 -arme 4.5082231315868122e-06 -ofin 4.5082231315868122e-06 -▁AVI 4.5082231315868122e-06 -▁EU-Bürger 4.5082231315868122e-06 -▁Lebensbedingungen 4.5082231315868122e-06 -▁Slow 4.5082231315868122e-06 -▁Transl 4.5082231315868122e-06 -▁Warnung 4.5082231315868122e-06 -▁celebration 4.5082231315868122e-06 -▁holl 4.5082231315868122e-06 -▁persuade 4.5082231315868122e-06 -▁reminded 4.5082231315868122e-06 -▁repos 4.5082231315868122e-06 -ANN 4.5044347087871593e-06 -earl 4.5044347087871593e-06 -pielt 4.5044347087871593e-06 -rängt 4.5044347087871593e-06 -steigerung 4.5044347087871593e-06 -▁11.00 4.5044347087871593e-06 -▁Anth 4.5044347087871593e-06 -▁Korn 4.5044347087871593e-06 -▁Lettland 4.5044347087871593e-06 -▁Mari 4.5044347087871593e-06 -▁Questions 4.5044347087871593e-06 -▁Sor 4.5044347087871593e-06 -▁aufrecht 4.5044347087871593e-06 -▁coating 4.5044347087871593e-06 -▁concessions 4.5044347087871593e-06 -▁cure 4.5044347087871593e-06 -▁farb 4.5044347087871593e-06 -▁possession 4.5044347087871593e-06 -▁potenziellen 4.5044347087871593e-06 -▁verwe 4.5044347087871593e-06 -▁verwendete 4.5044347087871593e-06 -Office 4.5006462859875064e-06 -`s 4.5006462859875064e-06 -eint 4.5006462859875064e-06 -embl 4.5006462859875064e-06 -frem 4.5006462859875064e-06 -funk 4.5006462859875064e-06 -ilar 4.5006462859875064e-06 -uing 4.5006462859875064e-06 -▁-F 4.5006462859875064e-06 -▁3.1 4.5006462859875064e-06 -▁BAS 4.5006462859875064e-06 -▁Lack 4.5006462859875064e-06 -▁Stars 4.5006462859875064e-06 -▁Teils 4.5006462859875064e-06 -▁Vene 4.5006462859875064e-06 -▁accomplished 4.5006462859875064e-06 -▁fließen 4.5006462859875064e-06 -▁self-d 4.5006462859875064e-06 -▁socio-economic 4.5006462859875064e-06 -▁teleph 4.5006462859875064e-06 -▁vorgetragen 4.5006462859875064e-06 -WA 4.4968578631878544e-06 -bels 4.4968578631878544e-06 -eckte 4.4968578631878544e-06 -ehle 4.4968578631878544e-06 -gl. 4.4968578631878544e-06 -ihen 4.4968578631878544e-06 -sucht 4.4968578631878544e-06 -würfe 4.4968578631878544e-06 -▁Biokraft 4.4968578631878544e-06 -▁Brücken 4.4968578631878544e-06 -▁Empfangs 4.4968578631878544e-06 -▁Intens 4.4968578631878544e-06 -▁cosm 4.4968578631878544e-06 -▁geste 4.4968578631878544e-06 -▁irregularities 4.4968578631878544e-06 -▁networking 4.4968578631878544e-06 -▁optimized 4.4968578631878544e-06 -▁shelter 4.4968578631878544e-06 -▁unverändert 4.4968578631878544e-06 -bewusstsein 4.4930694403882015e-06 -care 4.4930694403882015e-06 -klärungen 4.4930694403882015e-06 -obi 4.4930694403882015e-06 -▁Atem 4.4930694403882015e-06 -▁Christus 4.4930694403882015e-06 -▁Corbett 4.4930694403882015e-06 -▁Depression 4.4930694403882015e-06 -▁Fuj 4.4930694403882015e-06 -▁Pisten 4.4930694403882015e-06 -▁Recycling 4.4930694403882015e-06 -▁Side 4.4930694403882015e-06 -▁Viertens 4.4930694403882015e-06 -▁aggreg 4.4930694403882015e-06 -▁appelliere 4.4930694403882015e-06 -▁enden 4.4930694403882015e-06 -▁family-run 4.4930694403882015e-06 -▁nein 4.4930694403882015e-06 -▁singer 4.4930694403882015e-06 -▁varieties 4.4930694403882015e-06 --Pak 4.4892810175885486e-06 -IFT 4.4892810175885486e-06 -eaux 4.4892810175885486e-06 -▁Frühling 4.4892810175885486e-06 -▁Liberals 4.4892810175885486e-06 -▁Motorrad 4.4892810175885486e-06 -▁Student 4.4892810175885486e-06 -▁align 4.4892810175885486e-06 -▁fatal 4.4892810175885486e-06 -▁geehrter 4.4892810175885486e-06 -▁jour 4.4892810175885486e-06 -▁mol 4.4892810175885486e-06 -▁outline 4.4892810175885486e-06 -setzungen 4.4854925947888957e-06 -▁Ausflug 4.4854925947888957e-06 -▁Beschäftigungspolitik 4.4854925947888957e-06 -▁ample 4.4854925947888957e-06 -▁anschließ 4.4854925947888957e-06 -▁bilaterale 4.4854925947888957e-06 -▁contradiction 4.4854925947888957e-06 -▁multip 4.4854925947888957e-06 -▁stärkt 4.4854925947888957e-06 -▁writers 4.4854925947888957e-06 -Cs 4.4817041719892428e-06 -amper 4.4817041719892428e-06 -edged 4.4817041719892428e-06 -essing 4.4817041719892428e-06 -mehr 4.4817041719892428e-06 -parteien 4.4817041719892428e-06 -versch 4.4817041719892428e-06 -▁Aven 4.4817041719892428e-06 -▁Fax- 4.4817041719892428e-06 -▁Salvador 4.4817041719892428e-06 -▁Sha 4.4817041719892428e-06 -▁archives 4.4817041719892428e-06 -▁herkömmlichen 4.4817041719892428e-06 -▁influences 4.4817041719892428e-06 -▁multitude 4.4817041719892428e-06 -▁tariff 4.4817041719892428e-06 -▁territorialen 4.4817041719892428e-06 -▁unp 4.4817041719892428e-06 -▁vel 4.4817041719892428e-06 -▁verknüp 4.4817041719892428e-06 -▁É 4.4817041719892428e-06 -▁überarbeitet 4.4817041719892428e-06 --Programme 4.47791574918959e-06 -atorischen 4.47791574918959e-06 -gli 4.47791574918959e-06 -itement 4.47791574918959e-06 -tehen 4.47791574918959e-06 -teous 4.47791574918959e-06 -upport 4.47791574918959e-06 -▁Dalai 4.47791574918959e-06 -▁Erweiterungen 4.47791574918959e-06 -▁Klimaschutz 4.47791574918959e-06 -▁Seas 4.47791574918959e-06 -▁Tang 4.47791574918959e-06 -▁Terrassen 4.47791574918959e-06 -▁Veranstaltungsräume 4.47791574918959e-06 -▁Verwenden 4.47791574918959e-06 -▁experimental 4.47791574918959e-06 -▁gründ 4.47791574918959e-06 -▁incredibly 4.47791574918959e-06 -▁positiver 4.47791574918959e-06 -▁pragm 4.47791574918959e-06 -▁pressures 4.47791574918959e-06 -▁speculation 4.47791574918959e-06 -▁sé 4.47791574918959e-06 -▁³ 4.47791574918959e-06 -▁überdenken 4.47791574918959e-06 -bag 4.474127326389937e-06 -bound 4.474127326389937e-06 -eware 4.474127326389937e-06 -rik 4.474127326389937e-06 -ل 4.474127326389937e-06 -▁Aufstockung 4.474127326389937e-06 -▁Differ 4.474127326389937e-06 -▁Doktor 4.474127326389937e-06 -▁Ecofin 4.474127326389937e-06 -▁Hann 4.474127326389937e-06 -▁Pocket 4.474127326389937e-06 -▁Versicherung 4.474127326389937e-06 -▁draws 4.474127326389937e-06 -▁halls 4.474127326389937e-06 -kir 4.4703389035902841e-06 -stad 4.4703389035902841e-06 -swert 4.4703389035902841e-06 -yse 4.4703389035902841e-06 -▁Beschränkung 4.4703389035902841e-06 -▁Bez 4.4703389035902841e-06 -▁Genfer 4.4703389035902841e-06 -▁Konfigurations 4.4703389035902841e-06 -▁aan 4.4703389035902841e-06 -▁broadband 4.4703389035902841e-06 -▁gedenkt 4.4703389035902841e-06 -▁pf 4.4703389035902841e-06 -▁replacing 4.4703389035902841e-06 -300 4.4665504807906321e-06 -alender 4.4665504807906321e-06 -ifically 4.4665504807906321e-06 -opro 4.4665504807906321e-06 -ulative 4.4665504807906321e-06 -▁GMOs 4.4665504807906321e-06 -▁Geburtstag 4.4665504807906321e-06 -▁Schauen 4.4665504807906321e-06 -▁Wallström 4.4665504807906321e-06 -▁besuchte 4.4665504807906321e-06 -▁concentr 4.4665504807906321e-06 -▁inequalities 4.4665504807906321e-06 -▁preparatory 4.4665504807906321e-06 -▁privileg 4.4665504807906321e-06 -▁rass 4.4665504807906321e-06 -▁thou 4.4665504807906321e-06 -ronen 4.4627620579909792e-06 -ync 4.4627620579909792e-06 -▁Befreiung 4.4627620579909792e-06 -▁Gläub 4.4627620579909792e-06 -▁Gothic 4.4627620579909792e-06 -▁dealer 4.4627620579909792e-06 -▁diesel 4.4627620579909792e-06 -▁donation 4.4627620579909792e-06 -▁generating 4.4627620579909792e-06 -▁kultureller 4.4627620579909792e-06 -▁promenade 4.4627620579909792e-06 -▁protectionism 4.4627620579909792e-06 -▁ranks 4.4627620579909792e-06 -▁reign 4.4627620579909792e-06 -▁reveals 4.4627620579909792e-06 -▁surround 4.4627620579909792e-06 -▁ter 4.4627620579909792e-06 -immungen 4.4589736351913263e-06 -lam 4.4589736351913263e-06 -rupt 4.4589736351913263e-06 -ässe 4.4589736351913263e-06 -ção 4.4589736351913263e-06 -В 4.4589736351913263e-06 -▁Bestandteile 4.4589736351913263e-06 -▁Electronic 4.4589736351913263e-06 -▁Liebhaber 4.4589736351913263e-06 -▁Tunnel 4.4589736351913263e-06 -▁Wes 4.4589736351913263e-06 -▁ausgere 4.4589736351913263e-06 -▁behinderten 4.4589736351913263e-06 -▁debts 4.4589736351913263e-06 -▁diagnosis 4.4589736351913263e-06 -▁galt 4.4589736351913263e-06 -▁hinausgehen 4.4589736351913263e-06 -▁israelische 4.4589736351913263e-06 -▁neighbour 4.4589736351913263e-06 -▁stamp 4.4589736351913263e-06 -gemacht 4.4551852123916734e-06 -ologist 4.4551852123916734e-06 -ospace 4.4551852123916734e-06 -ören 4.4551852123916734e-06 -▁1200 4.4551852123916734e-06 -▁Budd 4.4551852123916734e-06 -▁Electric 4.4551852123916734e-06 -▁Lip 4.4551852123916734e-06 -▁arena 4.4551852123916734e-06 -▁confer 4.4551852123916734e-06 -▁destr 4.4551852123916734e-06 -▁facilitating 4.4551852123916734e-06 -▁shoot 4.4551852123916734e-06 -▁turnover 4.4551852123916734e-06 --Sterne 4.4513967895920205e-06 -arma 4.4513967895920205e-06 -noten 4.4513967895920205e-06 -pling 4.4513967895920205e-06 -prot 4.4513967895920205e-06 -tage 4.4513967895920205e-06 -wachstum 4.4513967895920205e-06 -▁COS 4.4513967895920205e-06 -▁Cin 4.4513967895920205e-06 -▁Gemäl 4.4513967895920205e-06 -▁Kne 4.4513967895920205e-06 -▁Mitgliedsländer 4.4513967895920205e-06 -▁Nass 4.4513967895920205e-06 -▁Sied 4.4513967895920205e-06 -▁Zufall 4.4513967895920205e-06 -▁barr 4.4513967895920205e-06 -▁intensiver 4.4513967895920205e-06 -▁löst 4.4513967895920205e-06 -▁obsc 4.4513967895920205e-06 -▁real-time 4.4513967895920205e-06 -▁trinken 4.4513967895920205e-06 -▁umfangreich 4.4513967895920205e-06 -▁variations 4.4513967895920205e-06 -.03 4.4476083667923676e-06 -aug 4.4476083667923676e-06 -bearbeitung 4.4476083667923676e-06 -dam 4.4476083667923676e-06 -▁Conil 4.4476083667923676e-06 -▁Fortunately 4.4476083667923676e-06 -▁Infineon 4.4476083667923676e-06 -▁Learning 4.4476083667923676e-06 -▁Lieferungen 4.4476083667923676e-06 -▁Radisson 4.4476083667923676e-06 -▁Streben 4.4476083667923676e-06 -▁Tasten 4.4476083667923676e-06 -▁Wartungs 4.4476083667923676e-06 -▁catal 4.4476083667923676e-06 -▁descriptions 4.4476083667923676e-06 -▁durchd 4.4476083667923676e-06 -▁grain 4.4476083667923676e-06 -▁immers 4.4476083667923676e-06 -▁premier 4.4476083667923676e-06 -▁schlimm 4.4476083667923676e-06 -▁sowjet 4.4476083667923676e-06 -▁upgraded 4.4476083667923676e-06 -▁verbesserten 4.4476083667923676e-06 --Seiten 4.4438199439927147e-06 -defizit 4.4438199439927147e-06 -gebnis 4.4438199439927147e-06 -prob 4.4438199439927147e-06 -verständnis 4.4438199439927147e-06 -õ 4.4438199439927147e-06 -üchter 4.4438199439927147e-06 -▁Castell 4.4438199439927147e-06 -▁Equipment 4.4438199439927147e-06 -▁Herzog 4.4438199439927147e-06 -▁Residen 4.4438199439927147e-06 -▁Wassersport 4.4438199439927147e-06 -▁Zagreb 4.4438199439927147e-06 -▁Zukunfts 4.4438199439927147e-06 -▁mangel 4.4438199439927147e-06 -▁non-c 4.4438199439927147e-06 -▁pläd 4.4438199439927147e-06 -▁segments 4.4438199439927147e-06 -▁überleben 4.4438199439927147e-06 --Paket 4.4400315211930618e-06 -hill 4.4400315211930618e-06 -onne 4.4400315211930618e-06 -rel 4.4400315211930618e-06 -spir 4.4400315211930618e-06 -▁EOS 4.4400315211930618e-06 -▁Gastgeber 4.4400315211930618e-06 -▁Hem 4.4400315211930618e-06 -▁OEM 4.4400315211930618e-06 -▁configured 4.4400315211930618e-06 -▁contrad 4.4400315211930618e-06 -▁gigant 4.4400315211930618e-06 -▁macroeconomic 4.4400315211930618e-06 -▁möge 4.4400315211930618e-06 -▁objections 4.4400315211930618e-06 -▁thy 4.4400315211930618e-06 -attform 4.43624309839341e-06 -bauer 4.43624309839341e-06 -itations 4.43624309839341e-06 -leihen 4.43624309839341e-06 -nischen 4.43624309839341e-06 -reitet 4.43624309839341e-06 -schlossen 4.43624309839341e-06 -ätt 4.43624309839341e-06 -überwachung 4.43624309839341e-06 -▁Beantwortung 4.43624309839341e-06 -▁Erlaubnis 4.43624309839341e-06 -▁Gefühle 4.43624309839341e-06 -▁Glob 4.43624309839341e-06 -▁Plätzen 4.43624309839341e-06 -▁Vac 4.43624309839341e-06 -▁chef 4.43624309839341e-06 -▁descend 4.43624309839341e-06 -▁earmark 4.43624309839341e-06 -▁exemption 4.43624309839341e-06 -▁wunderbar 4.43624309839341e-06 -▁zweites 4.43624309839341e-06 -Ex 4.4324546755937569e-06 -ichtigen 4.4324546755937569e-06 -▁Babys 4.4324546755937569e-06 -▁Misch 4.4324546755937569e-06 -▁Weber 4.4324546755937569e-06 -▁disadvantaged 4.4324546755937569e-06 -▁lifting 4.4324546755937569e-06 -▁rigid 4.4324546755937569e-06 -▁shortage 4.4324546755937569e-06 -▁unterrichtet 4.4324546755937569e-06 -▁updating 4.4324546755937569e-06 -▁wiederauf 4.4324546755937569e-06 -platte 4.428666252794104e-06 -rev 4.428666252794104e-06 -▁Aquar 4.428666252794104e-06 -▁EIN 4.428666252794104e-06 -▁Flora 4.428666252794104e-06 -▁Magnet 4.428666252794104e-06 -▁Records 4.428666252794104e-06 -▁Reding 4.428666252794104e-06 -▁Symbole 4.428666252794104e-06 -▁Wiederaufnahme 4.428666252794104e-06 -▁Zinssätze 4.428666252794104e-06 -▁Zusammenhalts 4.428666252794104e-06 -▁built-in 4.428666252794104e-06 -▁electron 4.428666252794104e-06 -▁furn 4.428666252794104e-06 -▁kenne 4.428666252794104e-06 -▁magnet 4.428666252794104e-06 -▁religi 4.428666252794104e-06 -▁transparente 4.428666252794104e-06 -▁verlaufen 4.428666252794104e-06 -▁vorwiegend 4.428666252794104e-06 -▁üb 4.428666252794104e-06 --se 4.4248778299944511e-06 -IZ 4.4248778299944511e-06 -führenden 4.4248778299944511e-06 -gängig 4.4248778299944511e-06 -ofar 4.4248778299944511e-06 -▁Barbar 4.4248778299944511e-06 -▁Festplatten 4.4248778299944511e-06 -▁Kontin 4.4248778299944511e-06 -▁Kurdish 4.4248778299944511e-06 -▁Sozialdemokratischen 4.4248778299944511e-06 -▁Verlag 4.4248778299944511e-06 -▁Vorjahr 4.4248778299944511e-06 -▁Zentralbanken 4.4248778299944511e-06 -▁applaud 4.4248778299944511e-06 -▁austerity 4.4248778299944511e-06 -▁neur 4.4248778299944511e-06 -▁verweigert 4.4248778299944511e-06 -entscheidungen 4.4210894071947982e-06 -idsch 4.4210894071947982e-06 -manager 4.4210894071947982e-06 -opes 4.4210894071947982e-06 -seitigkeit 4.4210894071947982e-06 -щ 4.4210894071947982e-06 -▁Arme 4.4210894071947982e-06 -▁Baumw 4.4210894071947982e-06 -▁CRM 4.4210894071947982e-06 -▁Cambridge 4.4210894071947982e-06 -▁Hop 4.4210894071947982e-06 -▁ICT 4.4210894071947982e-06 -▁Lizenzen 4.4210894071947982e-06 -▁ehrgeizige 4.4210894071947982e-06 -▁geführte 4.4210894071947982e-06 -▁mp 4.4210894071947982e-06 -▁nächstes 4.4210894071947982e-06 -▁poorer 4.4210894071947982e-06 -▁purchases 4.4210894071947982e-06 -▁unterzeichnen 4.4210894071947982e-06 -▁versichert 4.4210894071947982e-06 -▁Äußer 4.4210894071947982e-06 -eav 4.4173009843951453e-06 -gence 4.4173009843951453e-06 -läufe 4.4173009843951453e-06 -posed 4.4173009843951453e-06 -tral 4.4173009843951453e-06 -▁Delegationen 4.4173009843951453e-06 -▁Genehm 4.4173009843951453e-06 -▁Josef 4.4173009843951453e-06 -▁Sciences 4.4173009843951453e-06 -▁Zertifizierung 4.4173009843951453e-06 -▁Zoo 4.4173009843951453e-06 -▁anticipated 4.4173009843951453e-06 -▁beliebige 4.4173009843951453e-06 -▁gefäl 4.4173009843951453e-06 -▁indischen 4.4173009843951453e-06 -▁läng 4.4173009843951453e-06 -▁march 4.4173009843951453e-06 -▁pret 4.4173009843951453e-06 -▁quart 4.4173009843951453e-06 -▁specif 4.4173009843951453e-06 -▁tissue 4.4173009843951453e-06 -▁widen 4.4173009843951453e-06 -▁wiederherzustellen 4.4173009843951453e-06 -▁wissenschaftlich 4.4173009843951453e-06 -folgedessen 4.4135125615954924e-06 -kol 4.4135125615954924e-06 -leidung 4.4135125615954924e-06 -samer 4.4135125615954924e-06 -▁Bere 4.4135125615954924e-06 -▁Course 4.4135125615954924e-06 -▁EA 4.4135125615954924e-06 -▁Hund 4.4135125615954924e-06 -▁Konservativen 4.4135125615954924e-06 -▁aussieht 4.4135125615954924e-06 -▁bulk 4.4135125615954924e-06 -▁consultants 4.4135125615954924e-06 -▁speeds 4.4135125615954924e-06 -▁sü 4.4135125615954924e-06 -aschine 4.4097241387958395e-06 -olar 4.4097241387958395e-06 -orbenen 4.4097241387958395e-06 -vere 4.4097241387958395e-06 -▁1400 4.4097241387958395e-06 -▁1965 4.4097241387958395e-06 -▁Alicante 4.4097241387958395e-06 -▁Ausserdem 4.4097241387958395e-06 -▁Entwick 4.4097241387958395e-06 -▁Found 4.4097241387958395e-06 -▁GE 4.4097241387958395e-06 -▁Keramik 4.4097241387958395e-06 -▁Share 4.4097241387958395e-06 -▁Spaniens 4.4097241387958395e-06 -▁Wave 4.4097241387958395e-06 -▁Welle 4.4097241387958395e-06 -▁birgt 4.4097241387958395e-06 -▁dal 4.4097241387958395e-06 -▁deport 4.4097241387958395e-06 -▁einräumen 4.4097241387958395e-06 -▁nar 4.4097241387958395e-06 -▁obliegt 4.4097241387958395e-06 -▁veränderten 4.4097241387958395e-06 -gewichte 4.4059357159961875e-06 -ibes 4.4059357159961875e-06 -otechn 4.4059357159961875e-06 -voraus 4.4059357159961875e-06 -▁Affiliate 4.4059357159961875e-06 -▁Berechn 4.4059357159961875e-06 -▁Knowledge 4.4059357159961875e-06 -▁Neigh 4.4059357159961875e-06 -▁Pferd 4.4059357159961875e-06 -▁Platform 4.4059357159961875e-06 -▁Verstöße 4.4059357159961875e-06 -▁abzustimmen 4.4059357159961875e-06 -▁atr 4.4059357159961875e-06 -▁bestrebt 4.4059357159961875e-06 -▁cultivation 4.4059357159961875e-06 -▁geplante 4.4059357159961875e-06 -▁rechte 4.4059357159961875e-06 -▁springs 4.4059357159961875e-06 -▁umfangreiches 4.4059357159961875e-06 -▁überschritten 4.4059357159961875e-06 -esar 4.4021472931965346e-06 -mensch 4.4021472931965346e-06 -uries 4.4021472931965346e-06 -zepts 4.4021472931965346e-06 -▁Arafat 4.4021472931965346e-06 -▁BG 4.4021472931965346e-06 -▁Kriege 4.4021472931965346e-06 -▁Monument 4.4021472931965346e-06 -▁Schwächen 4.4021472931965346e-06 -▁Unrecht 4.4021472931965346e-06 -▁confort 4.4021472931965346e-06 -▁herauszufinden 4.4021472931965346e-06 -▁negligence 4.4021472931965346e-06 -▁retreat 4.4021472931965346e-06 -▁unerwün 4.4021472931965346e-06 -agua 4.3983588703968817e-06 -betrieben 4.3983588703968817e-06 -lant 4.3983588703968817e-06 -zufriedenheit 4.3983588703968817e-06 -▁Almost 4.3983588703968817e-06 -▁Dör 4.3983588703968817e-06 -▁Estate 4.3983588703968817e-06 -▁Fische 4.3983588703968817e-06 -▁Kohäsionsfonds 4.3983588703968817e-06 -▁Ohr 4.3983588703968817e-06 -▁Sieger 4.3983588703968817e-06 -▁Trock 4.3983588703968817e-06 -▁dachte 4.3983588703968817e-06 -▁ganzes 4.3983588703968817e-06 -▁outer 4.3983588703968817e-06 -▁preview 4.3983588703968817e-06 -▁sorg 4.3983588703968817e-06 -▁не 4.3983588703968817e-06 --Tra 4.3945704475972288e-06 -46 4.3945704475972288e-06 -istet 4.3945704475972288e-06 -nez 4.3945704475972288e-06 -uras 4.3945704475972288e-06 -öß 4.3945704475972288e-06 -▁Flair 4.3945704475972288e-06 -▁Innenhof 4.3945704475972288e-06 -▁behilflich 4.3945704475972288e-06 -▁hospitals 4.3945704475972288e-06 -▁tutorial 4.3945704475972288e-06 -▁verschwunden 4.3945704475972288e-06 -▁üben 4.3945704475972288e-06 -▁Autovermietung 4.3907820247975759e-06 -▁Bestands 4.3907820247975759e-06 -▁Kanäle 4.3907820247975759e-06 -▁attracted 4.3907820247975759e-06 -▁delight 4.3907820247975759e-06 -▁monastery 4.3907820247975759e-06 -▁rhetoric 4.3907820247975759e-06 -▁truck 4.3907820247975759e-06 -▁verfasst 4.3907820247975759e-06 -▁älteste 4.3907820247975759e-06 -derte 4.386993601997923e-06 -elekt 4.386993601997923e-06 -ersey 4.386993601997923e-06 -iac 4.386993601997923e-06 -ordn 4.386993601997923e-06 -romagnet 4.386993601997923e-06 -▁Lev 4.386993601997923e-06 -▁Röm 4.386993601997923e-06 -▁anhalten 4.386993601997923e-06 -▁befindlichen 4.386993601997923e-06 -▁corresponds 4.386993601997923e-06 -▁erledigen 4.386993601997923e-06 -▁geliefer 4.386993601997923e-06 -▁instruction 4.386993601997923e-06 -▁medi 4.386993601997923e-06 -▁nachkommen 4.386993601997923e-06 -▁outbrea 4.386993601997923e-06 -▁stoßen 4.386993601997923e-06 -▁tackled 4.386993601997923e-06 -▁töd 4.386993601997923e-06 -.01 4.38320517919827e-06 -ISS 4.38320517919827e-06 -meter 4.38320517919827e-06 -onal 4.38320517919827e-06 -städte 4.38320517919827e-06 -ucks 4.38320517919827e-06 -zufordern 4.38320517919827e-06 -▁Acrobat 4.38320517919827e-06 -▁Applikationen 4.38320517919827e-06 -▁Ausschluss 4.38320517919827e-06 -▁Bajor 4.38320517919827e-06 -▁CNC 4.38320517919827e-06 -▁Intensiv 4.38320517919827e-06 -▁Liikanen 4.38320517919827e-06 -▁Reinigungs 4.38320517919827e-06 -▁Sekret 4.38320517919827e-06 -▁VP 4.38320517919827e-06 -▁accus 4.38320517919827e-06 -▁attain 4.38320517919827e-06 -▁chip 4.38320517919827e-06 -▁cos 4.38320517919827e-06 -▁indul 4.38320517919827e-06 -▁injection 4.38320517919827e-06 -▁og 4.38320517919827e-06 -▁redund 4.38320517919827e-06 -▁staatlicher 4.38320517919827e-06 -▁zeitlich 4.38320517919827e-06 -marken 4.3794167563986181e-06 -reb 4.3794167563986181e-06 -regional 4.3794167563986181e-06 -▁Befürchtungen 4.3794167563986181e-06 -▁Cadiz 4.3794167563986181e-06 -▁Gleiche 4.3794167563986181e-06 -▁Rekord 4.3794167563986181e-06 -▁Rumän 4.3794167563986181e-06 -▁Wähl 4.3794167563986181e-06 -▁apt 4.3794167563986181e-06 -▁durchschnittliche 4.3794167563986181e-06 -▁fusion 4.3794167563986181e-06 -▁laptop 4.3794167563986181e-06 -▁recipient 4.3794167563986181e-06 -▁teilnehmenden 4.3794167563986181e-06 -▁temple 4.3794167563986181e-06 -▁wahrhaft 4.3794167563986181e-06 --Entwick 4.3756283335989652e-06 -ICK 4.3756283335989652e-06 -UD 4.3756283335989652e-06 -initiativen 4.3756283335989652e-06 -irds 4.3756283335989652e-06 -ritz 4.3756283335989652e-06 -uevo 4.3756283335989652e-06 -ändigkeit 4.3756283335989652e-06 -▁Jarzembowski 4.3756283335989652e-06 -▁Tierschutz 4.3756283335989652e-06 -▁elimination 4.3756283335989652e-06 -▁geschicht 4.3756283335989652e-06 -▁relevance 4.3756283335989652e-06 -▁schrecklichen 4.3756283335989652e-06 -▁unend 4.3756283335989652e-06 -akult 4.3718399107993123e-06 -fish 4.3718399107993123e-06 -iminal 4.3718399107993123e-06 -vents 4.3718399107993123e-06 -ф 4.3718399107993123e-06 -▁Advis 4.3718399107993123e-06 -▁Ankara 4.3718399107993123e-06 -▁Archer 4.3718399107993123e-06 -▁Auszeichnung 4.3718399107993123e-06 -▁Badewanne 4.3718399107993123e-06 -▁Maybe 4.3718399107993123e-06 -▁Nahrung 4.3718399107993123e-06 -▁Patr 4.3718399107993123e-06 -▁Patri 4.3718399107993123e-06 -▁Zeugen 4.3718399107993123e-06 -▁delightful 4.3718399107993123e-06 -▁gewaltigen 4.3718399107993123e-06 -▁prest 4.3718399107993123e-06 -▁productions 4.3718399107993123e-06 --Komp 4.3680514879996594e-06 --Player 4.3680514879996594e-06 -HO 4.3680514879996594e-06 -ylon 4.3680514879996594e-06 -zwecke 4.3680514879996594e-06 -▁Bedauern 4.3680514879996594e-06 -▁Carmen 4.3680514879996594e-06 -▁Eingangs 4.3680514879996594e-06 -▁Flucht 4.3680514879996594e-06 -▁Fox 4.3680514879996594e-06 -▁GNOME 4.3680514879996594e-06 -▁Mitentscheidung 4.3680514879996594e-06 -▁Motion 4.3680514879996594e-06 -▁Reisende 4.3680514879996594e-06 -▁Schriftliche 4.3680514879996594e-06 -▁Tay 4.3680514879996594e-06 -▁Tul 4.3680514879996594e-06 -▁Ve 4.3680514879996594e-06 -▁Wohlergehen 4.3680514879996594e-06 -▁formulieren 4.3680514879996594e-06 -▁halbe 4.3680514879996594e-06 -▁häufigsten 4.3680514879996594e-06 -▁soviel 4.3680514879996594e-06 -▁vegetation 4.3680514879996594e-06 -roh 4.3642630652000065e-06 -uhe 4.3642630652000065e-06 -unami 4.3642630652000065e-06 -▁1957 4.3642630652000065e-06 -▁IMM 4.3642630652000065e-06 -▁Marie 4.3642630652000065e-06 -▁Ott 4.3642630652000065e-06 -▁Panoram 4.3642630652000065e-06 -▁Rundfunk 4.3642630652000065e-06 -▁altogether 4.3642630652000065e-06 -▁personalized 4.3642630652000065e-06 -▁threaten 4.3642630652000065e-06 -▁virtuelle 4.3642630652000065e-06 --quality 4.3604746424003536e-06 -hew 4.3604746424003536e-06 -sbereich 4.3604746424003536e-06 -λ 4.3604746424003536e-06 -▁Bürgerkrieg 4.3604746424003536e-06 -▁ERP 4.3604746424003536e-06 -▁Füßen 4.3604746424003536e-06 -▁Iber 4.3604746424003536e-06 -▁Immobilienanze 4.3604746424003536e-06 -▁factories 4.3604746424003536e-06 -▁gründet 4.3604746424003536e-06 -▁handelte 4.3604746424003536e-06 -▁schweigen 4.3604746424003536e-06 -beob 4.3566862196007008e-06 -dings 4.3566862196007008e-06 -errung 4.3566862196007008e-06 -fabrik 4.3566862196007008e-06 -fly 4.3566862196007008e-06 -ità 4.3566862196007008e-06 -▁Entschuld 4.3566862196007008e-06 -▁Frankfurter 4.3566862196007008e-06 -▁Geistes 4.3566862196007008e-06 -▁Glückwunsch 4.3566862196007008e-06 -▁Kodex 4.3566862196007008e-06 -▁São 4.3566862196007008e-06 -▁aufgeteilt 4.3566862196007008e-06 -▁conservative 4.3566862196007008e-06 -▁folgendes 4.3566862196007008e-06 -▁ihrerseits 4.3566862196007008e-06 -▁imper 4.3566862196007008e-06 -▁interoper 4.3566862196007008e-06 -▁mig 4.3566862196007008e-06 -▁qualification 4.3566862196007008e-06 -▁sprachen 4.3566862196007008e-06 -▁transeuropäischen 4.3566862196007008e-06 -▁weise 4.3566862196007008e-06 -edict 4.3528977968010479e-06 -gets 4.3528977968010479e-06 -ickets 4.3528977968010479e-06 -ispiel 4.3528977968010479e-06 -presso 4.3528977968010479e-06 -vergabe 4.3528977968010479e-06 -ünen 4.3528977968010479e-06 -▁Biotechnologie 4.3528977968010479e-06 -▁Durchbruch 4.3528977968010479e-06 -▁Protektionismus 4.3528977968010479e-06 -▁Sag 4.3528977968010479e-06 -▁Willens 4.3528977968010479e-06 -▁complied 4.3528977968010479e-06 -▁schle 4.3528977968010479e-06 -kab 4.3491093740013958e-06 -män 4.3491093740013958e-06 -▁Dig 4.3491093740013958e-06 -▁Fifth 4.3491093740013958e-06 -▁Gehirn 4.3491093740013958e-06 -▁IRC 4.3491093740013958e-06 -▁Sympt 4.3491093740013958e-06 -▁daraufhin 4.3491093740013958e-06 -▁deterior 4.3491093740013958e-06 -▁föder 4.3491093740013958e-06 -▁gefährlicher 4.3491093740013958e-06 -▁journalist 4.3491093740013958e-06 -akische 4.3453209512017429e-06 -derten 4.3453209512017429e-06 -rechtliche 4.3453209512017429e-06 -▁Dienstleistungsrichtlinie 4.3453209512017429e-06 -▁Mitgliedsstaaten 4.3453209512017429e-06 -▁Occ 4.3453209512017429e-06 -▁Weihnachten 4.3453209512017429e-06 -▁Wäldern 4.3453209512017429e-06 -▁atemberauben 4.3453209512017429e-06 -▁cave 4.3453209512017429e-06 -▁drawings 4.3453209512017429e-06 -▁problematisch 4.3453209512017429e-06 -▁strongest 4.3453209512017429e-06 -▁teeth 4.3453209512017429e-06 -▁Ökos 4.3453209512017429e-06 --Up 4.34153252840209e-06 --one 4.34153252840209e-06 --way 4.34153252840209e-06 -abler 4.34153252840209e-06 -añ 4.34153252840209e-06 -▁Ble 4.34153252840209e-06 -▁Bulgar 4.34153252840209e-06 -▁Damals 4.34153252840209e-06 -▁Innov 4.34153252840209e-06 -▁Lomé 4.34153252840209e-06 -▁Prüfungen 4.34153252840209e-06 -▁Vormittag 4.34153252840209e-06 -▁besorgniserre 4.34153252840209e-06 -▁compress 4.34153252840209e-06 -▁ruft 4.34153252840209e-06 -▁signatures 4.34153252840209e-06 -▁AKP-Staaten 4.3377441056024371e-06 -▁Brat 4.3377441056024371e-06 -▁Cass 4.3377441056024371e-06 -▁Imper 4.3377441056024371e-06 -▁Sell 4.3377441056024371e-06 -▁Spitz 4.3377441056024371e-06 -▁TACIS 4.3377441056024371e-06 -▁angehört 4.3377441056024371e-06 -▁ausgebild 4.3377441056024371e-06 -▁ausschließ 4.3377441056024371e-06 -▁barely 4.3377441056024371e-06 -▁eliminated 4.3377441056024371e-06 -▁erstre 4.3377441056024371e-06 -▁exercises 4.3377441056024371e-06 -▁flora 4.3377441056024371e-06 -▁imply 4.3377441056024371e-06 -▁incurred 4.3377441056024371e-06 -▁socialist 4.3377441056024371e-06 -▁tanks 4.3377441056024371e-06 -▁tschech 4.3377441056024371e-06 -▁ukra 4.3377441056024371e-06 -43 4.3339556828027843e-06 -icos 4.3339556828027843e-06 -ohnt 4.3339556828027843e-06 -schiffe 4.3339556828027843e-06 -▁Cany 4.3339556828027843e-06 -▁Erfahrungs 4.3339556828027843e-06 -▁Gewässern 4.3339556828027843e-06 -▁Kohl 4.3339556828027843e-06 -▁Konserv 4.3339556828027843e-06 -▁Martini 4.3339556828027843e-06 -▁Rhyth 4.3339556828027843e-06 -▁discon 4.3339556828027843e-06 -▁diskut 4.3339556828027843e-06 -▁erzeug 4.3339556828027843e-06 -▁gemeldet 4.3339556828027843e-06 -▁professionell 4.3339556828027843e-06 -▁renovation 4.3339556828027843e-06 -▁resident 4.3339556828027843e-06 -assy 4.3301672600031314e-06 -erfolg 4.3301672600031314e-06 -klick 4.3301672600031314e-06 -orgungs 4.3301672600031314e-06 -stieg 4.3301672600031314e-06 -verg 4.3301672600031314e-06 -ändiger 4.3301672600031314e-06 -▁1966 4.3301672600031314e-06 -▁Athen 4.3301672600031314e-06 -▁Konsequenz 4.3301672600031314e-06 -▁Mant 4.3301672600031314e-06 -▁Parlamentarier 4.3301672600031314e-06 -▁Rapid 4.3301672600031314e-06 -▁Sachver 4.3301672600031314e-06 -▁bridges 4.3301672600031314e-06 -▁defect 4.3301672600031314e-06 -▁distances 4.3301672600031314e-06 -▁matching 4.3301672600031314e-06 -▁midst 4.3301672600031314e-06 --ray 4.3263788372034785e-06 -anträge 4.3263788372034785e-06 -onet 4.3263788372034785e-06 -▁Norweg 4.3263788372034785e-06 -▁Singapur 4.3263788372034785e-06 -▁Verurteilung 4.3263788372034785e-06 -▁animation 4.3263788372034785e-06 -▁bester 4.3263788372034785e-06 -▁bitt 4.3263788372034785e-06 -▁diameter 4.3263788372034785e-06 -▁fireplace 4.3263788372034785e-06 -▁forbidden 4.3263788372034785e-06 -▁gastr 4.3263788372034785e-06 -▁kombin 4.3263788372034785e-06 -▁mic 4.3263788372034785e-06 -▁repercussions 4.3263788372034785e-06 -▁verarbeiten 4.3263788372034785e-06 -icia 4.3225904144038256e-06 -▁Beteil 4.3225904144038256e-06 -▁Dubrovnik 4.3225904144038256e-06 -▁Einfl 4.3225904144038256e-06 -▁Gewässer 4.3225904144038256e-06 -▁Handlung 4.3225904144038256e-06 -▁Haushaltsausschusses 4.3225904144038256e-06 -▁Jav 4.3225904144038256e-06 -▁Konf 4.3225904144038256e-06 -▁Konzeption 4.3225904144038256e-06 -▁Lago 4.3225904144038256e-06 -▁Pensions 4.3225904144038256e-06 -▁Ryan 4.3225904144038256e-06 -▁Wertpapier 4.3225904144038256e-06 -▁cocktail 4.3225904144038256e-06 -▁ea 4.3225904144038256e-06 -▁zwingend 4.3225904144038256e-06 -azz 4.3188019916041735e-06 -issionen 4.3188019916041735e-06 -onst 4.3188019916041735e-06 -tauchen 4.3188019916041735e-06 -▁Auslieferung 4.3188019916041735e-06 -▁Botschaften 4.3188019916041735e-06 -▁Brows 4.3188019916041735e-06 -▁Instanz 4.3188019916041735e-06 -▁Sort 4.3188019916041735e-06 -▁True 4.3188019916041735e-06 -▁Variablen 4.3188019916041735e-06 -▁enchanting 4.3188019916041735e-06 -▁exot 4.3188019916041735e-06 -▁propaganda 4.3188019916041735e-06 -▁schn 4.3188019916041735e-06 -▁unangem 4.3188019916041735e-06 -▁unvergleich 4.3188019916041735e-06 -pflege 4.3150135688045206e-06 -sätzen 4.3150135688045206e-06 -▁Belastungen 4.3150135688045206e-06 -▁Bene 4.3150135688045206e-06 -▁Erzeugnissen 4.3150135688045206e-06 -▁Meerblick 4.3150135688045206e-06 -▁anmerken 4.3150135688045206e-06 -▁aufw 4.3150135688045206e-06 -▁befürchten 4.3150135688045206e-06 -▁frames 4.3150135688045206e-06 -▁historisch 4.3150135688045206e-06 -▁rehabilitation 4.3150135688045206e-06 -▁sister 4.3150135688045206e-06 -▁zusätzlicher 4.3150135688045206e-06 --Verhältnis 4.3112251460048678e-06 -KW 4.3112251460048678e-06 -bez 4.3112251460048678e-06 -bilden 4.3112251460048678e-06 -udes 4.3112251460048678e-06 -▁Champs 4.3112251460048678e-06 -▁Seil 4.3112251460048678e-06 -▁architects 4.3112251460048678e-06 -▁dauer 4.3112251460048678e-06 -▁documented 4.3112251460048678e-06 -▁graus 4.3112251460048678e-06 -▁hygiene 4.3112251460048678e-06 -▁studierte 4.3112251460048678e-06 -▁triple 4.3112251460048678e-06 --Shop 4.3074367232052149e-06 -debatte 4.3074367232052149e-06 -eno 4.3074367232052149e-06 -oter 4.3074367232052149e-06 -stationen 4.3074367232052149e-06 -▁Auseinandersetzungen 4.3074367232052149e-06 -▁Behandlungen 4.3074367232052149e-06 -▁Certainly 4.3074367232052149e-06 -▁Gor 4.3074367232052149e-06 -▁Shops 4.3074367232052149e-06 -▁boarding 4.3074367232052149e-06 -▁fürchte 4.3074367232052149e-06 -▁refresh 4.3074367232052149e-06 -▁screening 4.3074367232052149e-06 -▁wins 4.3074367232052149e-06 -ambots 4.303648300405562e-06 -ellow 4.303648300405562e-06 -ername 4.303648300405562e-06 -ieden 4.303648300405562e-06 -stage 4.303648300405562e-06 -züg 4.303648300405562e-06 -▁Around 4.303648300405562e-06 -▁Bout 4.303648300405562e-06 -▁Commons 4.303648300405562e-06 -▁Erleichterung 4.303648300405562e-06 -▁Jesolo 4.303648300405562e-06 -▁Privatisierung 4.303648300405562e-06 -▁Stärken 4.303648300405562e-06 -▁deliveries 4.303648300405562e-06 -▁disrupt 4.303648300405562e-06 -▁explo 4.303648300405562e-06 -▁komfort 4.303648300405562e-06 -▁notify 4.303648300405562e-06 -▁promptly 4.303648300405562e-06 -▁vertrauens 4.303648300405562e-06 -.d 4.2998598776059091e-06 -ikalen 4.2998598776059091e-06 -meisterschaft 4.2998598776059091e-06 -▁Annan 4.2998598776059091e-06 -▁Forschungs- 4.2998598776059091e-06 -▁Lieutenant 4.2998598776059091e-06 -▁abhängt 4.2998598776059091e-06 -▁complet 4.2998598776059091e-06 -▁crop 4.2998598776059091e-06 -▁deployed 4.2998598776059091e-06 -▁fav 4.2998598776059091e-06 -▁fro 4.2998598776059091e-06 -▁pap 4.2998598776059091e-06 -▁scrat 4.2998598776059091e-06 -▁skept 4.2998598776059091e-06 -▁steigende 4.2998598776059091e-06 -▁third-country 4.2998598776059091e-06 -▁unauthor 4.2998598776059091e-06 -▁überwältigen 4.2998598776059091e-06 -spieler 4.2960714548062562e-06 -▁Appar 4.2960714548062562e-06 -▁BBC 4.2960714548062562e-06 -▁Diamant 4.2960714548062562e-06 -▁Klärung 4.2960714548062562e-06 -▁Kürzungen 4.2960714548062562e-06 -▁Mittelalter 4.2960714548062562e-06 -▁Options 4.2960714548062562e-06 -▁Pfl 4.2960714548062562e-06 -▁Rin 4.2960714548062562e-06 -▁Stolz 4.2960714548062562e-06 -▁Tan 4.2960714548062562e-06 -▁Trading 4.2960714548062562e-06 -▁beeindruckenden 4.2960714548062562e-06 -▁bürgerliche 4.2960714548062562e-06 -▁fremden 4.2960714548062562e-06 -▁jeglichen 4.2960714548062562e-06 -▁republic 4.2960714548062562e-06 -▁successor 4.2960714548062562e-06 -alm 4.2922830320066033e-06 -inian 4.2922830320066033e-06 -laf 4.2922830320066033e-06 -undung 4.2922830320066033e-06 -▁Bestreben 4.2922830320066033e-06 -▁begannen 4.2922830320066033e-06 -▁billions 4.2922830320066033e-06 -▁crafts 4.2922830320066033e-06 -▁einzus 4.2922830320066033e-06 -▁guiding 4.2922830320066033e-06 -▁protein 4.2922830320066033e-06 -eader 4.2884946092069513e-06 -führende 4.2884946092069513e-06 -haut 4.2884946092069513e-06 -ischerweise 4.2884946092069513e-06 -quart 4.2884946092069513e-06 -seiten 4.2884946092069513e-06 -▁Flusses 4.2884946092069513e-06 -▁GUI 4.2884946092069513e-06 -▁Keep 4.2884946092069513e-06 -▁Rog 4.2884946092069513e-06 -▁Universitäts 4.2884946092069513e-06 -▁Verpack 4.2884946092069513e-06 -▁Walking 4.2884946092069513e-06 -▁deckt 4.2884946092069513e-06 -▁gaps 4.2884946092069513e-06 -▁households 4.2884946092069513e-06 -▁motto 4.2884946092069513e-06 -▁nachteil 4.2884946092069513e-06 -▁powered 4.2884946092069513e-06 -▁scal 4.2884946092069513e-06 -▁shores 4.2884946092069513e-06 -▁Öff 4.2884946092069513e-06 --Zimmer 4.2847061864072984e-06 -gor 4.2847061864072984e-06 -ingang 4.2847061864072984e-06 -ozi 4.2847061864072984e-06 -schaftliche 4.2847061864072984e-06 -▁Centr 4.2847061864072984e-06 -▁Meta 4.2847061864072984e-06 -▁dokument 4.2847061864072984e-06 -▁einzigartiges 4.2847061864072984e-06 -▁tout 4.2847061864072984e-06 -atial 4.2809177636076455e-06 -beauftrag 4.2809177636076455e-06 -institution 4.2809177636076455e-06 -istin 4.2809177636076455e-06 -istisches 4.2809177636076455e-06 -▁137 4.2809177636076455e-06 -▁Asset 4.2809177636076455e-06 -▁Ausdehnung 4.2809177636076455e-06 -▁Schulung 4.2809177636076455e-06 -▁Talent 4.2809177636076455e-06 -▁decides 4.2809177636076455e-06 -▁ehrgeizigen 4.2809177636076455e-06 -▁eingetreten 4.2809177636076455e-06 -▁enttäus 4.2809177636076455e-06 -▁font 4.2809177636076455e-06 -▁gemein 4.2809177636076455e-06 -▁hergest 4.2809177636076455e-06 -▁historischer 4.2809177636076455e-06 -▁jenseits 4.2809177636076455e-06 -▁konstru 4.2809177636076455e-06 -▁organizing 4.2809177636076455e-06 -▁pec 4.2809177636076455e-06 -▁recreational 4.2809177636076455e-06 -arth 4.2771293408079926e-06 -eated 4.2771293408079926e-06 -onomen 4.2771293408079926e-06 -undering 4.2771293408079926e-06 -äb 4.2771293408079926e-06 -▁Lieb 4.2771293408079926e-06 -▁Purch 4.2771293408079926e-06 -▁Screen 4.2771293408079926e-06 -▁Verbündeten 4.2771293408079926e-06 -▁blocking 4.2771293408079926e-06 -▁brown 4.2771293408079926e-06 -▁college 4.2771293408079926e-06 -▁japanische 4.2771293408079926e-06 -▁juristische 4.2771293408079926e-06 -▁linen 4.2771293408079926e-06 -▁momentum 4.2771293408079926e-06 -▁occasionally 4.2771293408079926e-06 -▁param 4.2771293408079926e-06 -▁röm 4.2771293408079926e-06 -▁spür 4.2771293408079926e-06 -egu 4.27334091800834e-06 -otor 4.27334091800834e-06 -uary 4.27334091800834e-06 -venient 4.27334091800834e-06 -wörter 4.27334091800834e-06 -▁2050 4.27334091800834e-06 -▁Ausgewogenheit 4.27334091800834e-06 -▁Gestern 4.27334091800834e-06 -▁Plen 4.27334091800834e-06 -▁Prämien 4.27334091800834e-06 -▁Schmuck 4.27334091800834e-06 -▁anspruchsvollen 4.27334091800834e-06 -▁empfehle 4.27334091800834e-06 -▁gourmet 4.27334091800834e-06 -▁itiner 4.27334091800834e-06 -▁komment 4.27334091800834e-06 -▁offenkundig 4.27334091800834e-06 -▁slide 4.27334091800834e-06 -▁umgewandelt 4.27334091800834e-06 -ampf 4.2695524952086868e-06 -ehme 4.2695524952086868e-06 -esischen 4.2695524952086868e-06 -folgt 4.2695524952086868e-06 -gehörigkeit 4.2695524952086868e-06 -neum 4.2695524952086868e-06 -ochten 4.2695524952086868e-06 -tätigkeit 4.2695524952086868e-06 -ögert 4.2695524952086868e-06 -▁GU 4.2695524952086868e-06 -▁Heinrich 4.2695524952086868e-06 -▁Margar 4.2695524952086868e-06 -▁confronted 4.2695524952086868e-06 -▁fabrics 4.2695524952086868e-06 -▁pragmatic 4.2695524952086868e-06 -▁ästhet 4.2695524952086868e-06 -▁üpp 4.2695524952086868e-06 -NET 4.2657640724090339e-06 -ecution 4.2657640724090339e-06 -opath 4.2657640724090339e-06 -ston 4.2657640724090339e-06 -ugar 4.2657640724090339e-06 -verzeichnis 4.2657640724090339e-06 -▁Buyer 4.2657640724090339e-06 -▁Evaluierung 4.2657640724090339e-06 -▁Haushaltskontrolle 4.2657640724090339e-06 -▁Netto 4.2657640724090339e-06 -▁Pestiz 4.2657640724090339e-06 -▁Richtungen 4.2657640724090339e-06 -▁einzuf 4.2657640724090339e-06 -▁heaven 4.2657640724090339e-06 -▁hub 4.2657640724090339e-06 -▁lush 4.2657640724090339e-06 -▁projekt 4.2657640724090339e-06 -▁versp 4.2657640724090339e-06 -ardin 4.261975649609381e-06 -cements 4.261975649609381e-06 -chtest 4.261975649609381e-06 -eschl 4.261975649609381e-06 -farben 4.261975649609381e-06 -ogh 4.261975649609381e-06 -räder 4.261975649609381e-06 -▁1800 4.261975649609381e-06 -▁Cypr 4.261975649609381e-06 -▁Join 4.261975649609381e-06 -▁broaden 4.261975649609381e-06 -▁commissioned 4.261975649609381e-06 -▁fehlenden 4.261975649609381e-06 -▁livestock 4.261975649609381e-06 -▁längeren 4.261975649609381e-06 -▁massiv 4.261975649609381e-06 -▁positioned 4.261975649609381e-06 -▁undem 4.261975649609381e-06 -▁wearing 4.261975649609381e-06 -▁Öffnen 4.261975649609381e-06 -Star 4.258187226809729e-06 -brücke 4.258187226809729e-06 -enner 4.258187226809729e-06 -gesetzgebung 4.258187226809729e-06 -range 4.258187226809729e-06 -twortet 4.258187226809729e-06 -▁Beck 4.258187226809729e-06 -▁Kommen 4.258187226809729e-06 -▁Milano 4.258187226809729e-06 -▁Willi 4.258187226809729e-06 -▁enshrined 4.258187226809729e-06 -▁strukturellen 4.258187226809729e-06 -END 4.2543988040100761e-06 -empel 4.2543988040100761e-06 -gesetze 4.2543988040100761e-06 -griffe 4.2543988040100761e-06 -lighten 4.2543988040100761e-06 -risiko 4.2543988040100761e-06 -▁1948 4.2543988040100761e-06 -▁Anpass 4.2543988040100761e-06 -▁BB 4.2543988040100761e-06 -▁Controller 4.2543988040100761e-06 -▁Tort 4.2543988040100761e-06 -▁berechnen 4.2543988040100761e-06 -▁massac 4.2543988040100761e-06 -▁restlichen 4.2543988040100761e-06 -nummern 4.2506103812104232e-06 -ocket 4.2506103812104232e-06 -reffend 4.2506103812104232e-06 -ruption 4.2506103812104232e-06 -▁240 4.2506103812104232e-06 -▁Eurojust 4.2506103812104232e-06 -▁Regul 4.2506103812104232e-06 -▁Serviceleistungen 4.2506103812104232e-06 -▁Völkerm 4.2506103812104232e-06 -▁entrepreneur 4.2506103812104232e-06 -▁komplizierten 4.2506103812104232e-06 -▁repar 4.2506103812104232e-06 -▁vorang 4.2506103812104232e-06 -KR 4.24682195841077e-06 -begriff 4.24682195841077e-06 -present 4.24682195841077e-06 -unning 4.24682195841077e-06 -▁Interventionen 4.24682195841077e-06 -▁Jungen 4.24682195841077e-06 -▁Löhne 4.24682195841077e-06 -▁Zun 4.24682195841077e-06 -▁administrator 4.24682195841077e-06 -▁auszubauen 4.24682195841077e-06 -▁deinem 4.24682195841077e-06 -▁ermutigt 4.24682195841077e-06 -▁gi 4.24682195841077e-06 -▁herausgegeben 4.24682195841077e-06 -▁inex 4.24682195841077e-06 -▁junior 4.24682195841077e-06 -▁liquidity 4.24682195841077e-06 -▁sensors 4.24682195841077e-06 -▁trace 4.24682195841077e-06 -▁verification 4.24682195841077e-06 -▁vorbereit 4.24682195841077e-06 --rate 4.2430335356111174e-06 -angaben 4.2430335356111174e-06 -bing 4.2430335356111174e-06 -ichere 4.2430335356111174e-06 -laven 4.2430335356111174e-06 -ranche 4.2430335356111174e-06 -▁Campo 4.2430335356111174e-06 -▁Continental 4.2430335356111174e-06 -▁Lagerung 4.2430335356111174e-06 -▁aspirations 4.2430335356111174e-06 -▁dot 4.2430335356111174e-06 -▁erledigt 4.2430335356111174e-06 -▁makers 4.2430335356111174e-06 -GI 4.2392451128114645e-06 -IAL 4.2392451128114645e-06 -basierten 4.2392451128114645e-06 -hofer 4.2392451128114645e-06 -iffer 4.2392451128114645e-06 -ifikations 4.2392451128114645e-06 -owa 4.2392451128114645e-06 -rid 4.2392451128114645e-06 -urfing 4.2392451128114645e-06 -▁Orange 4.2392451128114645e-06 -▁Trocken 4.2392451128114645e-06 -▁Yesterday 4.2392451128114645e-06 -▁absoluten 4.2392451128114645e-06 -▁asymmetr 4.2392451128114645e-06 -▁beherrschen 4.2392451128114645e-06 -▁lend 4.2392451128114645e-06 -▁seasonal 4.2392451128114645e-06 -▁trek 4.2392451128114645e-06 -Do 4.2354566900118116e-06 -GR 4.2354566900118116e-06 -andid 4.2354566900118116e-06 -ilde 4.2354566900118116e-06 -olls 4.2354566900118116e-06 -▁Gord 4.2354566900118116e-06 -▁Hold 4.2354566900118116e-06 -▁Moder 4.2354566900118116e-06 -▁Starting 4.2354566900118116e-06 -▁Wortlaut 4.2354566900118116e-06 -▁inaug 4.2354566900118116e-06 -▁persönliches 4.2354566900118116e-06 -▁polnische 4.2354566900118116e-06 -▁voranzubringen 4.2354566900118116e-06 --Stra 4.2316682672121587e-06 -BSD 4.2316682672121587e-06 -Cl 4.2316682672121587e-06 -bled 4.2316682672121587e-06 -iculous 4.2316682672121587e-06 -rique 4.2316682672121587e-06 -▁Cham 4.2316682672121587e-06 -▁Otto 4.2316682672121587e-06 -▁Republ 4.2316682672121587e-06 -▁Routen 4.2316682672121587e-06 -▁alleg 4.2316682672121587e-06 -▁deploy 4.2316682672121587e-06 -▁geared 4.2316682672121587e-06 -▁intensity 4.2316682672121587e-06 -▁schnellstmöglich 4.2316682672121587e-06 -▁simpler 4.2316682672121587e-06 -▁unaus 4.2316682672121587e-06 -▁Deg 4.2278798444125067e-06 -▁Naples 4.2278798444125067e-06 -▁Periode 4.2278798444125067e-06 -▁allgemeiner 4.2278798444125067e-06 -▁bekannter 4.2278798444125067e-06 -▁etabliert 4.2278798444125067e-06 -▁maßgeb 4.2278798444125067e-06 -▁reaffirm 4.2278798444125067e-06 -▁sensiblen 4.2278798444125067e-06 -alia 4.2240914216128538e-06 -etion 4.2240914216128538e-06 -zeilen 4.2240914216128538e-06 -▁Anstatt 4.2240914216128538e-06 -▁CMS 4.2240914216128538e-06 -▁Cell 4.2240914216128538e-06 -▁Diplom 4.2240914216128538e-06 -▁Fontaine 4.2240914216128538e-06 -▁Hang 4.2240914216128538e-06 -▁Varianten 4.2240914216128538e-06 -▁communist 4.2240914216128538e-06 -▁echtes 4.2240914216128538e-06 -▁manoeuv 4.2240914216128538e-06 -lässigkeit 4.2203029988132009e-06 -okolle 4.2203029988132009e-06 -opfer 4.2203029988132009e-06 -räge 4.2203029988132009e-06 -ämmen 4.2203029988132009e-06 -▁Berichterstattung 4.2203029988132009e-06 -▁Leop 4.2203029988132009e-06 -▁OM 4.2203029988132009e-06 -▁Oppositions 4.2203029988132009e-06 -▁Perf 4.2203029988132009e-06 -▁Themas 4.2203029988132009e-06 -▁acoust 4.2203029988132009e-06 -▁compensate 4.2203029988132009e-06 -▁forderte 4.2203029988132009e-06 -▁millenn 4.2203029988132009e-06 -▁photographer 4.2203029988132009e-06 -schauen 4.216514576013548e-06 -udia 4.216514576013548e-06 -▁Bearbeitungs 4.216514576013548e-06 -▁Beitrittskandidaten 4.216514576013548e-06 -▁EU-In 4.216514576013548e-06 -▁Komit 4.216514576013548e-06 -▁Serbian 4.216514576013548e-06 -▁Sitzungs 4.216514576013548e-06 -▁Säulen 4.216514576013548e-06 -▁bevorzugen 4.216514576013548e-06 -▁demograf 4.216514576013548e-06 -▁eingegeben 4.216514576013548e-06 -▁licences 4.216514576013548e-06 -▁screw 4.216514576013548e-06 -▁symptoms 4.216514576013548e-06 -bekistan 4.2127261532138951e-06 -esent 4.2127261532138951e-06 -hause 4.2127261532138951e-06 -ränkungen 4.2127261532138951e-06 -wechselt 4.2127261532138951e-06 -▁Buchen 4.2127261532138951e-06 -▁Ethiop 4.2127261532138951e-06 -▁Mobiltelefon 4.2127261532138951e-06 -▁ODER 4.2127261532138951e-06 -▁Oste 4.2127261532138951e-06 -▁Photocopying 4.2127261532138951e-06 -▁Pier 4.2127261532138951e-06 -▁conversation 4.2127261532138951e-06 -▁großartigen 4.2127261532138951e-06 -▁mitt 4.2127261532138951e-06 -▁promotional 4.2127261532138951e-06 -▁zusammengefasst 4.2127261532138951e-06 --speaking 4.2089377304142422e-06 -.08 4.2089377304142422e-06 -alters 4.2089377304142422e-06 -disk 4.2089377304142422e-06 -ehnte 4.2089377304142422e-06 -igenden 4.2089377304142422e-06 -üstet 4.2089377304142422e-06 -▁Abreise 4.2089377304142422e-06 -▁Errungenschaften 4.2089377304142422e-06 -▁Filmen 4.2089377304142422e-06 -▁Outdoor 4.2089377304142422e-06 -▁Rabatt 4.2089377304142422e-06 -▁Race 4.2089377304142422e-06 -▁S-Bahn 4.2089377304142422e-06 -▁Seville 4.2089377304142422e-06 -▁TCP 4.2089377304142422e-06 -▁Unfälle 4.2089377304142422e-06 -▁atmos 4.2089377304142422e-06 -▁beha 4.2089377304142422e-06 -▁beige 4.2089377304142422e-06 -▁juice 4.2089377304142422e-06 -▁kreat 4.2089377304142422e-06 -▁versatile 4.2089377304142422e-06 -▁weitergehen 4.2089377304142422e-06 -izierten 4.2051493076145893e-06 -plement 4.2051493076145893e-06 -▁Brem 4.2051493076145893e-06 -▁EU-K 4.2051493076145893e-06 -▁Kampfes 4.2051493076145893e-06 -▁Mailing 4.2051493076145893e-06 -▁Myanmar 4.2051493076145893e-06 -▁Noten 4.2051493076145893e-06 -▁Roten 4.2051493076145893e-06 -▁eager 4.2051493076145893e-06 -▁motivation 4.2051493076145893e-06 -▁tied 4.2051493076145893e-06 -▁unent 4.2051493076145893e-06 -▁vernünft 4.2051493076145893e-06 -▁vulnerability 4.2051493076145893e-06 -krebs 4.2013608848149373e-06 -utor 4.2013608848149373e-06 -û 4.2013608848149373e-06 -▁1961 4.2013608848149373e-06 -▁Betrieben 4.2013608848149373e-06 -▁Built 4.2013608848149373e-06 -▁Cord 4.2013608848149373e-06 -▁Einsicht 4.2013608848149373e-06 -▁Fotokopiereinrichtungen 4.2013608848149373e-06 -▁Globalisation 4.2013608848149373e-06 -▁beginn 4.2013608848149373e-06 -▁disproportion 4.2013608848149373e-06 -▁reinen 4.2013608848149373e-06 --Adresse 4.1975724620152844e-06 -ilden 4.1975724620152844e-06 -kript 4.1975724620152844e-06 -ursor 4.1975724620152844e-06 -▁Lj 4.1975724620152844e-06 -▁Miles 4.1975724620152844e-06 -▁Olivenöl 4.1975724620152844e-06 -▁Snacks 4.1975724620152844e-06 -▁aged 4.1975724620152844e-06 -▁butter 4.1975724620152844e-06 -▁drohen 4.1975724620152844e-06 -▁exportieren 4.1975724620152844e-06 -▁pic 4.1975724620152844e-06 -▁privilege 4.1975724620152844e-06 -▁recognis 4.1975724620152844e-06 -▁vollständ 4.1975724620152844e-06 -elevant 4.1937840392156315e-06 -idas 4.1937840392156315e-06 -▁Clients 4.1937840392156315e-06 -▁Demokratischen 4.1937840392156315e-06 -▁Hü 4.1937840392156315e-06 -▁Meinungsfreiheit 4.1937840392156315e-06 -▁drittens 4.1937840392156315e-06 -▁dryer 4.1937840392156315e-06 -▁gesendet 4.1937840392156315e-06 -▁irrit 4.1937840392156315e-06 -▁lehnt 4.1937840392156315e-06 -▁luxuriösen 4.1937840392156315e-06 -▁orientieren 4.1937840392156315e-06 -▁pitch 4.1937840392156315e-06 -▁posting 4.1937840392156315e-06 -▁sitzt 4.1937840392156315e-06 -▁verhaftet 4.1937840392156315e-06 -Pr 4.1899956164159786e-06 -ammlungen 4.1899956164159786e-06 -läßlich 4.1899956164159786e-06 -orden 4.1899956164159786e-06 -wäsche 4.1899956164159786e-06 -äuschen 4.1899956164159786e-06 -▁103 4.1899956164159786e-06 -▁Aktualität 4.1899956164159786e-06 -▁Alzheimer 4.1899956164159786e-06 -▁Erwägungen 4.1899956164159786e-06 -▁Informationsschalter 4.1899956164159786e-06 -▁Khan 4.1899956164159786e-06 -▁Lud 4.1899956164159786e-06 -▁Patrick 4.1899956164159786e-06 -▁Posten 4.1899956164159786e-06 -▁Zar 4.1899956164159786e-06 -▁biotechnology 4.1899956164159786e-06 -▁enters 4.1899956164159786e-06 -▁envis 4.1899956164159786e-06 -▁erholsamen 4.1899956164159786e-06 -▁herausragen 4.1899956164159786e-06 -▁seid 4.1899956164159786e-06 -▁substitute 4.1899956164159786e-06 -▁übersehen 4.1899956164159786e-06 -ilater 4.1862071936163257e-06 -oves 4.1862071936163257e-06 -quin 4.1862071936163257e-06 -utzte 4.1862071936163257e-06 -▁135 4.1862071936163257e-06 -▁Batt 4.1862071936163257e-06 -▁Early 4.1862071936163257e-06 -▁Task 4.1862071936163257e-06 -▁attrakt 4.1862071936163257e-06 -▁feier 4.1862071936163257e-06 -▁financially 4.1862071936163257e-06 -▁mask 4.1862071936163257e-06 -aneous 4.1824187708166728e-06 -uble 4.1824187708166728e-06 -▁1964 4.1824187708166728e-06 -▁Bewirtschaftung 4.1824187708166728e-06 -▁Communications 4.1824187708166728e-06 -▁Fischereif 4.1824187708166728e-06 -▁GAT 4.1824187708166728e-06 -▁Nin 4.1824187708166728e-06 -▁Trent 4.1824187708166728e-06 -▁bulgar 4.1824187708166728e-06 -▁chains 4.1824187708166728e-06 -▁colourful 4.1824187708166728e-06 -▁earnings 4.1824187708166728e-06 -▁entdeckte 4.1824187708166728e-06 -▁powder 4.1824187708166728e-06 -▁snack 4.1824187708166728e-06 -▁tuna 4.1824187708166728e-06 -▁verschle 4.1824187708166728e-06 -.9 4.17863034801702e-06 -ameter 4.17863034801702e-06 -andinav 4.17863034801702e-06 -ieh 4.17863034801702e-06 -▁Cir 4.17863034801702e-06 -▁Kitz 4.17863034801702e-06 -▁Lew 4.17863034801702e-06 -▁Peters 4.17863034801702e-06 -▁Schulungen 4.17863034801702e-06 -▁auszuführen 4.17863034801702e-06 -▁density 4.17863034801702e-06 -▁ernsten 4.17863034801702e-06 -▁fakt 4.17863034801702e-06 -▁longest 4.17863034801702e-06 -▁orientiert 4.17863034801702e-06 -▁traf 4.17863034801702e-06 -▁verbleiben 4.17863034801702e-06 --Restaurant 4.174841925217367e-06 -EY 4.174841925217367e-06 -entes 4.174841925217367e-06 -izient 4.174841925217367e-06 -vorgaben 4.174841925217367e-06 -▁1962 4.174841925217367e-06 -▁Arn 4.174841925217367e-06 -▁Bristol 4.174841925217367e-06 -▁Dj 4.174841925217367e-06 -▁Kommunismus 4.174841925217367e-06 -▁Tirol 4.174841925217367e-06 -▁Trinkwasser 4.174841925217367e-06 -▁Zentrale 4.174841925217367e-06 -▁accurately 4.174841925217367e-06 -▁bone 4.174841925217367e-06 -▁cater 4.174841925217367e-06 -▁ci 4.174841925217367e-06 -▁clash 4.174841925217367e-06 -▁einwand 4.174841925217367e-06 -▁fragt 4.174841925217367e-06 -▁fähig 4.174841925217367e-06 -▁glän 4.174841925217367e-06 -▁merit 4.174841925217367e-06 -▁pig 4.174841925217367e-06 -▁placement 4.174841925217367e-06 -▁sooner 4.174841925217367e-06 -▁specification 4.174841925217367e-06 -▁tenden 4.174841925217367e-06 -▁ungehe 4.174841925217367e-06 --ins 4.171053502417715e-06 -78 4.171053502417715e-06 -achtungs 4.171053502417715e-06 -aille 4.171053502417715e-06 -crib 4.171053502417715e-06 -enza 4.171053502417715e-06 -fire 4.171053502417715e-06 -idos 4.171053502417715e-06 -vation 4.171053502417715e-06 -wicht 4.171053502417715e-06 -▁Auswirkung 4.171053502417715e-06 -▁Films 4.171053502417715e-06 -▁Kamer 4.171053502417715e-06 -▁Liga 4.171053502417715e-06 -▁Patente 4.171053502417715e-06 -▁Schweine 4.171053502417715e-06 -▁Writ 4.171053502417715e-06 -▁blicken 4.171053502417715e-06 -▁diesjährigen 4.171053502417715e-06 -▁earliest 4.171053502417715e-06 -▁elektrischen 4.171053502417715e-06 -▁hon 4.171053502417715e-06 -▁imstande 4.171053502417715e-06 -▁langjährigen 4.171053502417715e-06 -▁rush 4.171053502417715e-06 -▁ub 4.171053502417715e-06 -▁vi 4.171053502417715e-06 --y 4.1672650796180621e-06 -Pl 4.1672650796180621e-06 -apsed 4.1672650796180621e-06 -dies 4.1672650796180621e-06 -filter 4.1672650796180621e-06 -ieft 4.1672650796180621e-06 -iful 4.1672650796180621e-06 -irect 4.1672650796180621e-06 -▁Ansonsten 4.1672650796180621e-06 -▁Cocktail 4.1672650796180621e-06 -▁Coup 4.1672650796180621e-06 -▁kilom 4.1672650796180621e-06 -▁sixth 4.1672650796180621e-06 -▁stieg 4.1672650796180621e-06 -▁unberühr 4.1672650796180621e-06 -▁wissenschaftlicher 4.1672650796180621e-06 --Center 4.1634766568184092e-06 --screen 4.1634766568184092e-06 -OU 4.1634766568184092e-06 -aching 4.1634766568184092e-06 -ausstellung 4.1634766568184092e-06 -building 4.1634766568184092e-06 -food 4.1634766568184092e-06 -ieht 4.1634766568184092e-06 -izierte 4.1634766568184092e-06 -ministration 4.1634766568184092e-06 -▁Kath 4.1634766568184092e-06 -▁Stütz 4.1634766568184092e-06 -▁air-conditioning 4.1634766568184092e-06 -▁alarming 4.1634766568184092e-06 -▁eindring 4.1634766568184092e-06 -▁excluding 4.1634766568184092e-06 -▁phenomena 4.1634766568184092e-06 -▁pub 4.1634766568184092e-06 --American 4.1596882340187563e-06 -besitz 4.1596882340187563e-06 -elegenheiten 4.1596882340187563e-06 -hots 4.1596882340187563e-06 -ivial 4.1596882340187563e-06 -staates 4.1596882340187563e-06 -streben 4.1596882340187563e-06 -turn 4.1596882340187563e-06 -▁Attrib 4.1596882340187563e-06 -▁Einstimmigkeit 4.1596882340187563e-06 -▁Energieverbrauch 4.1596882340187563e-06 -▁HA 4.1596882340187563e-06 -▁Pick 4.1596882340187563e-06 -▁Politikern 4.1596882340187563e-06 -▁Uganda 4.1596882340187563e-06 -▁assort 4.1596882340187563e-06 -▁configur 4.1596882340187563e-06 -▁discounts 4.1596882340187563e-06 -▁freigegeben 4.1596882340187563e-06 -▁royal 4.1596882340187563e-06 -▁seating 4.1596882340187563e-06 -▁tariffs 4.1596882340187563e-06 --Pal 4.1558998112191034e-06 -atically 4.1558998112191034e-06 -aufnahmen 4.1558998112191034e-06 -hauptstadt 4.1558998112191034e-06 -teilte 4.1558998112191034e-06 -zesse 4.1558998112191034e-06 -▁Jones 4.1558998112191034e-06 -▁Lamy 4.1558998112191034e-06 -▁Rücktritt 4.1558998112191034e-06 -▁ankommt 4.1558998112191034e-06 -▁displaced 4.1558998112191034e-06 -▁einziger 4.1558998112191034e-06 -▁kontinuierliche 4.1558998112191034e-06 -anned 4.1521113884194505e-06 -eather 4.1521113884194505e-06 -heel 4.1521113884194505e-06 -lava 4.1521113884194505e-06 -▁Dimensionen 4.1521113884194505e-06 -▁Reception 4.1521113884194505e-06 -▁competing 4.1521113884194505e-06 -▁cozy 4.1521113884194505e-06 -▁exceeded 4.1521113884194505e-06 -▁infin 4.1521113884194505e-06 -▁kontinuier 4.1521113884194505e-06 -▁lobb 4.1521113884194505e-06 -▁simulation 4.1521113884194505e-06 -▁vorzu 4.1521113884194505e-06 -▁wertvoll 4.1521113884194505e-06 -▁überschreiten 4.1521113884194505e-06 -einrichtung 4.1483229656197977e-06 -etary 4.1483229656197977e-06 -etches 4.1483229656197977e-06 -pra 4.1483229656197977e-06 -ymph 4.1483229656197977e-06 -▁Alltags 4.1483229656197977e-06 -▁Centro 4.1483229656197977e-06 -▁Figur 4.1483229656197977e-06 -▁Folgenab 4.1483229656197977e-06 -▁Rinder 4.1483229656197977e-06 -▁Umstand 4.1483229656197977e-06 -▁Weitergabe 4.1483229656197977e-06 -▁beurteilt 4.1483229656197977e-06 -▁broke 4.1483229656197977e-06 -▁peaks 4.1483229656197977e-06 -▁saf 4.1483229656197977e-06 --owned 4.1445345428201448e-06 -arious 4.1445345428201448e-06 -icul 4.1445345428201448e-06 -ová 4.1445345428201448e-06 -rafen 4.1445345428201448e-06 -schwer 4.1445345428201448e-06 -ulture 4.1445345428201448e-06 -▁Auftreten 4.1445345428201448e-06 -▁Fitnesscenter 4.1445345428201448e-06 -▁Geheimnis 4.1445345428201448e-06 -▁Satellit 4.1445345428201448e-06 -▁Synchron 4.1445345428201448e-06 -▁Telefonnummer 4.1445345428201448e-06 -▁demokrat 4.1445345428201448e-06 -▁designing 4.1445345428201448e-06 -▁renommierten 4.1445345428201448e-06 -▁shapes 4.1445345428201448e-06 -▁sofortige 4.1445345428201448e-06 -▁voraussichtlich 4.1445345428201448e-06 -▁vorsehen 4.1445345428201448e-06 -▁Australian 4.1407461200204927e-06 -▁Behauptung 4.1407461200204927e-06 -▁Erforschung 4.1407461200204927e-06 -▁Esc 4.1407461200204927e-06 -▁Milliarde 4.1407461200204927e-06 -▁Personals 4.1407461200204927e-06 -▁Rohr 4.1407461200204927e-06 -▁Terrain 4.1407461200204927e-06 -▁Ung 4.1407461200204927e-06 -▁ausgeht 4.1407461200204927e-06 -▁christlichen 4.1407461200204927e-06 -▁gebühr 4.1407461200204927e-06 -▁somehow 4.1407461200204927e-06 --Dokument 4.13695769722084e-06 -agenturen 4.13695769722084e-06 -amour 4.13695769722084e-06 -aring 4.13695769722084e-06 -dena 4.13695769722084e-06 -klagt 4.13695769722084e-06 -verträglich 4.13695769722084e-06 -▁Anderen 4.13695769722084e-06 -▁Bevölkerungsgruppen 4.13695769722084e-06 -▁Lebensdauer 4.13695769722084e-06 -▁Umwelt- 4.13695769722084e-06 -▁aesthetic 4.13695769722084e-06 -▁begehen 4.13695769722084e-06 -▁tail 4.13695769722084e-06 -del 4.1331692744211869e-06 -uung 4.1331692744211869e-06 -▁Akteuren 4.1331692744211869e-06 -▁Belle 4.1331692744211869e-06 -▁Emer 4.1331692744211869e-06 -▁Erwärmung 4.1331692744211869e-06 -▁Jas 4.1331692744211869e-06 -▁Manip 4.1331692744211869e-06 -▁Roboter 4.1331692744211869e-06 -▁Tester 4.1331692744211869e-06 -▁angegriffen 4.1331692744211869e-06 -▁breiter 4.1331692744211869e-06 -▁fahr 4.1331692744211869e-06 -▁fount 4.1331692744211869e-06 -▁reimburs 4.1331692744211869e-06 -▁relig 4.1331692744211869e-06 -▁restriction 4.1331692744211869e-06 -▁strip 4.1331692744211869e-06 -esso 4.129380851621534e-06 -ganis 4.129380851621534e-06 -rance 4.129380851621534e-06 -reichende 4.129380851621534e-06 -roid 4.129380851621534e-06 -sprachige 4.129380851621534e-06 -uens 4.129380851621534e-06 -▁Beihilfe 4.129380851621534e-06 -▁Beobachtung 4.129380851621534e-06 -▁Kron 4.129380851621534e-06 -▁Spass 4.129380851621534e-06 -▁Zend 4.129380851621534e-06 -▁angesprochenen 4.129380851621534e-06 -▁dah 4.129380851621534e-06 -▁einzel 4.129380851621534e-06 -▁magical 4.129380851621534e-06 -▁mittelalter 4.129380851621534e-06 -▁solle 4.129380851621534e-06 -▁thrown 4.129380851621534e-06 -▁widersprechen 4.129380851621534e-06 -▁Änderungsantrags 4.129380851621534e-06 -läufig 4.1255924288218811e-06 -prinzips 4.1255924288218811e-06 -zes 4.1255924288218811e-06 -üff 4.1255924288218811e-06 -▁4000 4.1255924288218811e-06 -▁Atomwaffen 4.1255924288218811e-06 -▁Entscheidungsträger 4.1255924288218811e-06 -▁Gä 4.1255924288218811e-06 -▁Sozialmodell 4.1255924288218811e-06 -▁UR 4.1255924288218811e-06 -▁chronic 4.1255924288218811e-06 -▁derartiger 4.1255924288218811e-06 -▁rough 4.1255924288218811e-06 -▁timber 4.1255924288218811e-06 --discrimination 4.1218040060222283e-06 -aby 4.1218040060222283e-06 -irl 4.1218040060222283e-06 -▁Jahrhunderten 4.1218040060222283e-06 -▁Streitigkeiten 4.1218040060222283e-06 -▁Sturm 4.1218040060222283e-06 -▁abolished 4.1218040060222283e-06 -▁issuing 4.1218040060222283e-06 -▁pict 4.1218040060222283e-06 -▁poet 4.1218040060222283e-06 -bury 4.1180155832225754e-06 -group 4.1180155832225754e-06 -ifest 4.1180155832225754e-06 -kühl 4.1180155832225754e-06 -opolit 4.1180155832225754e-06 -unas 4.1180155832225754e-06 -währung 4.1180155832225754e-06 -▁AA 4.1180155832225754e-06 -▁Geduld 4.1180155832225754e-06 -▁Had 4.1180155832225754e-06 -▁Instand 4.1180155832225754e-06 -▁Italia 4.1180155832225754e-06 -▁Jes 4.1180155832225754e-06 -▁Managing 4.1180155832225754e-06 -▁Profile 4.1180155832225754e-06 -▁Promotion 4.1180155832225754e-06 -▁Römer 4.1180155832225754e-06 -▁Suchmaschinen 4.1180155832225754e-06 -▁caut 4.1180155832225754e-06 -▁erweist 4.1180155832225754e-06 -▁expecting 4.1180155832225754e-06 -▁freiwillige 4.1180155832225754e-06 -▁frozen 4.1180155832225754e-06 -▁sozioökonom 4.1180155832225754e-06 -▁strengere 4.1180155832225754e-06 -▁suits 4.1180155832225754e-06 -▁temper 4.1180155832225754e-06 -▁würdigen 4.1180155832225754e-06 -ammelten 4.1142271604229225e-06 -sein 4.1142271604229225e-06 -upro 4.1142271604229225e-06 -▁MIT 4.1142271604229225e-06 -▁Salzburger 4.1142271604229225e-06 -▁Typen 4.1142271604229225e-06 -▁annehm 4.1142271604229225e-06 -▁boys 4.1142271604229225e-06 -▁emergence 4.1142271604229225e-06 -▁entit 4.1142271604229225e-06 -▁konzentr 4.1142271604229225e-06 -▁trock 4.1142271604229225e-06 -▁worthwhile 4.1142271604229225e-06 --looking 4.1104387376232704e-06 -bedürft 4.1104387376232704e-06 -garde 4.1104387376232704e-06 -hamm 4.1104387376232704e-06 -wei 4.1104387376232704e-06 -▁Columbia 4.1104387376232704e-06 -▁Logos 4.1104387376232704e-06 -▁Skript 4.1104387376232704e-06 -▁assumption 4.1104387376232704e-06 -▁beherrscht 4.1104387376232704e-06 -▁bevorzugt 4.1104387376232704e-06 -▁critics 4.1104387376232704e-06 -▁geschm 4.1104387376232704e-06 -▁lifetime 4.1104387376232704e-06 -▁severely 4.1104387376232704e-06 -▁verified 4.1104387376232704e-06 -anw 4.1066503148236175e-06 -endel 4.1066503148236175e-06 -eton 4.1066503148236175e-06 -face 4.1066503148236175e-06 -isis 4.1066503148236175e-06 -itas 4.1066503148236175e-06 -ogas 4.1066503148236175e-06 -spekt 4.1066503148236175e-06 -stufen 4.1066503148236175e-06 -ägten 4.1066503148236175e-06 -▁E.ON 4.1066503148236175e-06 -▁Evans 4.1066503148236175e-06 -▁Málaga 4.1066503148236175e-06 -▁Vitorino 4.1066503148236175e-06 -▁WWU 4.1066503148236175e-06 -▁ankommen 4.1066503148236175e-06 -▁cluster 4.1066503148236175e-06 -▁criticisms 4.1066503148236175e-06 -▁injured 4.1066503148236175e-06 -▁mothers 4.1066503148236175e-06 -▁prüft 4.1066503148236175e-06 -▁rentals 4.1066503148236175e-06 -▁staat 4.1066503148236175e-06 -▁verschärfen 4.1066503148236175e-06 -▁vorbehalt 4.1066503148236175e-06 -WF 4.1028618920239646e-06 -achter 4.1028618920239646e-06 -oen 4.1028618920239646e-06 -▁Ask 4.1028618920239646e-06 -▁Audit 4.1028618920239646e-06 -▁Available 4.1028618920239646e-06 -▁Eg 4.1028618920239646e-06 -▁Geldpolitik 4.1028618920239646e-06 -▁Nahe 4.1028618920239646e-06 -▁Ponte 4.1028618920239646e-06 -▁Strafen 4.1028618920239646e-06 -▁formulated 4.1028618920239646e-06 -▁höchstem 4.1028618920239646e-06 -▁koennen 4.1028618920239646e-06 -▁magnetic 4.1028618920239646e-06 -▁multinationalen 4.1028618920239646e-06 -▁müßten 4.1028618920239646e-06 -▁realities 4.1028618920239646e-06 -▁stricter 4.1028618920239646e-06 -antast 4.0990734692243118e-06 -etal 4.0990734692243118e-06 -ettern 4.0990734692243118e-06 -g. 4.0990734692243118e-06 -grammen 4.0990734692243118e-06 -oise 4.0990734692243118e-06 -▁Arctic 4.0990734692243118e-06 -▁Kün 4.0990734692243118e-06 -▁Mans 4.0990734692243118e-06 -▁Mitgliedstaates 4.0990734692243118e-06 -▁Raketen 4.0990734692243118e-06 -▁Straftaten 4.0990734692243118e-06 -▁constituency 4.0990734692243118e-06 -▁copper 4.0990734692243118e-06 -▁dated 4.0990734692243118e-06 -▁gezahlt 4.0990734692243118e-06 -▁telefonisch 4.0990734692243118e-06 -72 4.0952850464246589e-06 -NG 4.0952850464246589e-06 -geladen 4.0952850464246589e-06 -keln 4.0952850464246589e-06 -verd 4.0952850464246589e-06 -völ 4.0952850464246589e-06 -▁Doctor 4.0952850464246589e-06 -▁Enttäuschung 4.0952850464246589e-06 -▁LIFE 4.0952850464246589e-06 -▁Lieder 4.0952850464246589e-06 -▁Vog 4.0952850464246589e-06 -▁abb 4.0952850464246589e-06 -▁backdrop 4.0952850464246589e-06 -▁cav 4.0952850464246589e-06 -▁detriment 4.0952850464246589e-06 -▁erleb 4.0952850464246589e-06 -▁holders 4.0952850464246589e-06 -▁html 4.0952850464246589e-06 -▁memorable 4.0952850464246589e-06 -▁raffin 4.0952850464246589e-06 -▁scenic 4.0952850464246589e-06 -▁styl 4.0952850464246589e-06 -▁verkehr 4.0952850464246589e-06 --state 4.091496623625006e-06 -700 4.091496623625006e-06 -▁Autof 4.091496623625006e-06 -▁Jahrt 4.091496623625006e-06 -▁Rathaus 4.091496623625006e-06 -▁Umbr 4.091496623625006e-06 -▁argu 4.091496623625006e-06 -▁bun 4.091496623625006e-06 -▁gerechte 4.091496623625006e-06 -▁parte 4.091496623625006e-06 -ankt 4.0877082008253531e-06 -ologists 4.0877082008253531e-06 -▁ALDE-Fraktion 4.0877082008253531e-06 -▁Entsp 4.0877082008253531e-06 -▁Formate 4.0877082008253531e-06 -▁Fotografen 4.0877082008253531e-06 -▁Fut 4.0877082008253531e-06 -▁Holocaust 4.0877082008253531e-06 -▁Miguel 4.0877082008253531e-06 -▁Muse 4.0877082008253531e-06 -▁SF 4.0877082008253531e-06 -▁Telec 4.0877082008253531e-06 -▁Umweltpolitik 4.0877082008253531e-06 -▁abstract 4.0877082008253531e-06 -▁beziehe 4.0877082008253531e-06 -▁containers 4.0877082008253531e-06 -▁demonstrating 4.0877082008253531e-06 -▁encryption 4.0877082008253531e-06 -▁offener 4.0877082008253531e-06 --minded 4.0839197780257e-06 -ATA 4.0839197780257e-06 -iks 4.0839197780257e-06 -reichend 4.0839197780257e-06 -▁Dez 4.0839197780257e-06 -▁Schengener 4.0839197780257e-06 -▁Supreme 4.0839197780257e-06 -▁Wesent 4.0839197780257e-06 -▁Züge 4.0839197780257e-06 -▁befand 4.0839197780257e-06 -▁begünst 4.0839197780257e-06 -▁tastefully 4.0839197780257e-06 -▁Übernachtungen 4.0839197780257e-06 -gewöhn 4.0801313552260481e-06 -lieferung 4.0801313552260481e-06 -▁BO 4.0801313552260481e-06 -▁Bibliotheken 4.0801313552260481e-06 -▁Edward 4.0801313552260481e-06 -▁Lunch 4.0801313552260481e-06 -▁breakthrough 4.0801313552260481e-06 -▁demnach 4.0801313552260481e-06 -▁einheim 4.0801313552260481e-06 -▁neck 4.0801313552260481e-06 -▁optimization 4.0801313552260481e-06 -▁pity 4.0801313552260481e-06 -attered 4.0763429324263953e-06 -ehnt 4.0763429324263953e-06 -lus 4.0763429324263953e-06 -öffent 4.0763429324263953e-06 -▁Aktual 4.0763429324263953e-06 -▁Brad 4.0763429324263953e-06 -▁Fou 4.0763429324263953e-06 -▁Morgan 4.0763429324263953e-06 -▁Neug 4.0763429324263953e-06 -▁Semin 4.0763429324263953e-06 -▁clarified 4.0763429324263953e-06 -▁declar 4.0763429324263953e-06 -▁devastating 4.0763429324263953e-06 -▁establishes 4.0763429324263953e-06 -▁installer 4.0763429324263953e-06 -▁ros 4.0763429324263953e-06 -▁varies 4.0763429324263953e-06 -▁Äng 4.0763429324263953e-06 -acon 4.0725545096267424e-06 -amba 4.0725545096267424e-06 -anzig 4.0725545096267424e-06 -cap 4.0725545096267424e-06 -forsch 4.0725545096267424e-06 -ugo 4.0725545096267424e-06 -▁Fehlern 4.0725545096267424e-06 -▁Jury 4.0725545096267424e-06 -▁Secret 4.0725545096267424e-06 -▁Substanz 4.0725545096267424e-06 -▁Wart 4.0725545096267424e-06 -▁behindertenfreundliche 4.0725545096267424e-06 -▁defended 4.0725545096267424e-06 -▁herbeizuführen 4.0725545096267424e-06 -▁mankind 4.0725545096267424e-06 -▁purchaser 4.0725545096267424e-06 -▁treating 4.0725545096267424e-06 -▁valleys 4.0725545096267424e-06 -▁wahl 4.0725545096267424e-06 -avan 4.0687660868270895e-06 -haber 4.0687660868270895e-06 -nergies 4.0687660868270895e-06 -obel 4.0687660868270895e-06 -orry 4.0687660868270895e-06 -otherapy 4.0687660868270895e-06 -speed 4.0687660868270895e-06 -▁215 4.0687660868270895e-06 -▁Administrator 4.0687660868270895e-06 -▁Auswertung 4.0687660868270895e-06 -▁Bettwäsche 4.0687660868270895e-06 -▁Bezirk 4.0687660868270895e-06 -▁Ice 4.0687660868270895e-06 -▁Programmier 4.0687660868270895e-06 -▁Vertiefung 4.0687660868270895e-06 -▁Voice 4.0687660868270895e-06 -▁generates 4.0687660868270895e-06 -▁zugewiesen 4.0687660868270895e-06 -BO 4.0649776640274366e-06 -eitet 4.0649776640274366e-06 -itative 4.0649776640274366e-06 -ländischen 4.0649776640274366e-06 -steigt 4.0649776640274366e-06 -ümer 4.0649776640274366e-06 -▁Einmischung 4.0649776640274366e-06 -▁Likewise 4.0649776640274366e-06 -▁Modellen 4.0649776640274366e-06 -▁Passagier 4.0649776640274366e-06 --Website 4.0611892412277837e-06 -atom 4.0611892412277837e-06 -blog 4.0611892412277837e-06 -proofed 4.0611892412277837e-06 -zeigt 4.0611892412277837e-06 -▁Ale 4.0611892412277837e-06 -▁Familienzimmer 4.0611892412277837e-06 -▁Gehäuse 4.0611892412277837e-06 -▁Probe 4.0611892412277837e-06 -▁beneficiaries 4.0611892412277837e-06 -▁gleichnam 4.0611892412277837e-06 -▁hes 4.0611892412277837e-06 -▁literary 4.0611892412277837e-06 -▁polizeil 4.0611892412277837e-06 -▁stattfand 4.0611892412277837e-06 -▁surve 4.0611892412277837e-06 -▁wovon 4.0611892412277837e-06 --Krise 4.0574008184281308e-06 -ERT 4.0574008184281308e-06 -beginn 4.0574008184281308e-06 -ioden 4.0574008184281308e-06 -oment 4.0574008184281308e-06 -▁108 4.0574008184281308e-06 -▁A320 4.0574008184281308e-06 -▁Cox 4.0574008184281308e-06 -▁Ecuador 4.0574008184281308e-06 -▁Maximum 4.0574008184281308e-06 -▁Particular 4.0574008184281308e-06 -▁anget 4.0574008184281308e-06 -▁biometr 4.0574008184281308e-06 -▁geradezu 4.0574008184281308e-06 -▁Überschwemmungen 4.0574008184281308e-06 -▁ört 4.0574008184281308e-06 -hens 4.0536123956284779e-06 -itim 4.0536123956284779e-06 -unge 4.0536123956284779e-06 -▁Bem 4.0536123956284779e-06 -▁Einigkeit 4.0536123956284779e-06 -▁Fabr 4.0536123956284779e-06 -▁Ful 4.0536123956284779e-06 -▁Mayor 4.0536123956284779e-06 -▁entschuldigen 4.0536123956284779e-06 -▁finnischen 4.0536123956284779e-06 -.S. 4.0498239728288259e-06 -ampe 4.0498239728288259e-06 -emed 4.0498239728288259e-06 -endar 4.0498239728288259e-06 -hältnismäß 4.0498239728288259e-06 -rateg 4.0498239728288259e-06 -▁-M 4.0498239728288259e-06 -▁Fu 4.0498239728288259e-06 -▁Gesundheitsversorgung 4.0498239728288259e-06 -▁JPEG 4.0498239728288259e-06 -▁Pressemitteil 4.0498239728288259e-06 -▁incomplete 4.0498239728288259e-06 -▁kitchenette 4.0498239728288259e-06 -▁kombinieren 4.0498239728288259e-06 -▁patio 4.0498239728288259e-06 -▁tongue 4.0498239728288259e-06 --Inter 4.046035550029173e-06 -.06 4.046035550029173e-06 -aminated 4.046035550029173e-06 -handelt 4.046035550029173e-06 -klassen 4.046035550029173e-06 -sprecher 4.046035550029173e-06 -▁Bomben 4.046035550029173e-06 -▁Kleine 4.046035550029173e-06 -▁Milit 4.046035550029173e-06 -▁Sharon 4.046035550029173e-06 -▁TOS 4.046035550029173e-06 -▁Treibhausgas 4.046035550029173e-06 -▁Verantwortungs 4.046035550029173e-06 -▁binary 4.046035550029173e-06 -▁entail 4.046035550029173e-06 -▁entfern 4.046035550029173e-06 -▁herausragende 4.046035550029173e-06 -▁sä 4.046035550029173e-06 -▁verh 4.046035550029173e-06 -ARC 4.04224712722952e-06 -earch 4.04224712722952e-06 -embles 4.04224712722952e-06 -feldern 4.04224712722952e-06 -hut 4.04224712722952e-06 -rücke 4.04224712722952e-06 -äher 4.04224712722952e-06 -öhne 4.04224712722952e-06 -▁Biom 4.04224712722952e-06 -▁Göteborg 4.04224712722952e-06 -▁Schmerz 4.04224712722952e-06 -▁Tiger 4.04224712722952e-06 -▁Treffpunkt 4.04224712722952e-06 -▁angestreb 4.04224712722952e-06 -▁avant 4.04224712722952e-06 -▁commem 4.04224712722952e-06 -▁destroying 4.04224712722952e-06 -▁nachhaltiger 4.04224712722952e-06 -▁renomm 4.04224712722952e-06 -▁transparenter 4.04224712722952e-06 -▁unterstützten 4.04224712722952e-06 -asur 4.0384587044298672e-06 -beam 4.0384587044298672e-06 -isting 4.0384587044298672e-06 -ono 4.0384587044298672e-06 -uzent 4.0384587044298672e-06 -▁Alan 4.0384587044298672e-06 -▁Fachleute 4.0384587044298672e-06 -▁Gig 4.0384587044298672e-06 -▁Kupfer 4.0384587044298672e-06 -▁Leucht 4.0384587044298672e-06 -▁Moral 4.0384587044298672e-06 -▁ausschl 4.0384587044298672e-06 -▁chaos 4.0384587044298672e-06 -▁confused 4.0384587044298672e-06 -▁diente 4.0384587044298672e-06 -▁dramatisch 4.0384587044298672e-06 -▁forecast 4.0384587044298672e-06 -▁licenses 4.0384587044298672e-06 -▁picked 4.0384587044298672e-06 --eff 4.0346702816302143e-06 -auben 4.0346702816302143e-06 -eiter 4.0346702816302143e-06 -ertoire 4.0346702816302143e-06 -prise 4.0346702816302143e-06 -verordnung 4.0346702816302143e-06 -öglichkeiten 4.0346702816302143e-06 -▁Bauer 4.0346702816302143e-06 -▁Cohn-Bendit 4.0346702816302143e-06 -▁Herman 4.0346702816302143e-06 -▁Persön 4.0346702816302143e-06 -▁Terme 4.0346702816302143e-06 -▁Unterscheidung 4.0346702816302143e-06 -▁abd 4.0346702816302143e-06 -▁discr 4.0346702816302143e-06 -▁eBay 4.0346702816302143e-06 -▁inval 4.0346702816302143e-06 -▁lays 4.0346702816302143e-06 -▁mais 4.0346702816302143e-06 -▁postponed 4.0346702816302143e-06 -▁recordings 4.0346702816302143e-06 -▁trac 4.0346702816302143e-06 -▁vereinen 4.0346702816302143e-06 -tal 4.0308818588305614e-06 -▁12th 4.0308818588305614e-06 -▁Chen 4.0308818588305614e-06 -▁IG 4.0308818588305614e-06 -▁Leone 4.0308818588305614e-06 -▁Qualifikationen 4.0308818588305614e-06 -▁Viv 4.0308818588305614e-06 -▁eindeutigen 4.0308818588305614e-06 -▁informative 4.0308818588305614e-06 -▁tent 4.0308818588305614e-06 -▁überwunden 4.0308818588305614e-06 -geschäfte 4.0270934360309085e-06 -iciencies 4.0270934360309085e-06 -positionen 4.0270934360309085e-06 -rium 4.0270934360309085e-06 -wanderung 4.0270934360309085e-06 -xen 4.0270934360309085e-06 -zuladen 4.0270934360309085e-06 -▁Drittländer 4.0270934360309085e-06 -▁Immo 4.0270934360309085e-06 -▁Knochen 4.0270934360309085e-06 -▁Kolleg 4.0270934360309085e-06 -▁Omn 4.0270934360309085e-06 -▁Taking 4.0270934360309085e-06 -▁Vermitt 4.0270934360309085e-06 -▁apr 4.0270934360309085e-06 -▁beträchtlichen 4.0270934360309085e-06 -▁phases 4.0270934360309085e-06 -▁schwarze 4.0270934360309085e-06 -erz 4.0233050132312565e-06 -pendence 4.0233050132312565e-06 -sider 4.0233050132312565e-06 -▁B2 4.0233050132312565e-06 -▁Besatz 4.0233050132312565e-06 -▁Ideologie 4.0233050132312565e-06 -▁Renault 4.0233050132312565e-06 -▁Schaff 4.0233050132312565e-06 -▁Schwarzen 4.0233050132312565e-06 -▁Staub 4.0233050132312565e-06 -▁Verbrauchs 4.0233050132312565e-06 -▁aufgezeigt 4.0233050132312565e-06 -▁documentary 4.0233050132312565e-06 -▁einb 4.0233050132312565e-06 -▁expects 4.0233050132312565e-06 -▁hö 4.0233050132312565e-06 -▁lawyer 4.0233050132312565e-06 -▁participant 4.0233050132312565e-06 -▁seiten 4.0233050132312565e-06 -▁subsidiaries 4.0233050132312565e-06 -▁unwill 4.0233050132312565e-06 -bedded 4.0195165904316036e-06 -rought 4.0195165904316036e-06 -umption 4.0195165904316036e-06 -▁Depart 4.0195165904316036e-06 -▁Griechenlands 4.0195165904316036e-06 -▁McK 4.0195165904316036e-06 -▁Nowadays 4.0195165904316036e-06 -▁Tibetan 4.0195165904316036e-06 -▁draußen 4.0195165904316036e-06 -▁endanger 4.0195165904316036e-06 -▁gast 4.0195165904316036e-06 -▁geograf 4.0195165904316036e-06 -▁icons 4.0195165904316036e-06 -▁oven 4.0195165904316036e-06 -▁storn 4.0195165904316036e-06 -▁sudden 4.0195165904316036e-06 -▁totalitarian 4.0195165904316036e-06 -▁vergröß 4.0195165904316036e-06 -Hotel 4.0157281676319507e-06 -dehnen 4.0157281676319507e-06 -esst 4.0157281676319507e-06 -irgend 4.0157281676319507e-06 -verbände 4.0157281676319507e-06 -zell 4.0157281676319507e-06 -▁Bran 4.0157281676319507e-06 -▁Energietr 4.0157281676319507e-06 -▁Etappe 4.0157281676319507e-06 -▁Reservierungen 4.0157281676319507e-06 -▁Serb 4.0157281676319507e-06 -▁Slideshows 4.0157281676319507e-06 -▁Verknüpfung 4.0157281676319507e-06 -▁Wonder 4.0157281676319507e-06 -▁Za 4.0157281676319507e-06 -▁Zuwanderung 4.0157281676319507e-06 -▁ausschlag 4.0157281676319507e-06 -▁bikes 4.0157281676319507e-06 -▁encrypt 4.0157281676319507e-06 -▁forecasts 4.0157281676319507e-06 -▁ordnungsgemäße 4.0157281676319507e-06 -▁peninsula 4.0157281676319507e-06 -▁üblicherweise 4.0157281676319507e-06 -anche 4.0119397448322978e-06 -bürger 4.0119397448322978e-06 -idation 4.0119397448322978e-06 -opic 4.0119397448322978e-06 -zwingen 4.0119397448322978e-06 -▁Aussteller 4.0119397448322978e-06 -▁ICC 4.0119397448322978e-06 -▁Russians 4.0119397448322978e-06 -▁Traff 4.0119397448322978e-06 -▁Wars 4.0119397448322978e-06 -▁arg 4.0119397448322978e-06 -▁awaiting 4.0119397448322978e-06 -▁beizubehalten 4.0119397448322978e-06 -▁coex 4.0119397448322978e-06 -▁elektr 4.0119397448322978e-06 -▁nachgewiesen 4.0119397448322978e-06 -▁pesticides 4.0119397448322978e-06 -▁unvollständ 4.0119397448322978e-06 -ovolta 4.0081513220326449e-06 -ulsion 4.0081513220326449e-06 -xx 4.0081513220326449e-06 -▁Fischfang 4.0081513220326449e-06 -▁ILO 4.0081513220326449e-06 -▁Kameras 4.0081513220326449e-06 -▁Mike 4.0081513220326449e-06 -▁Tech 4.0081513220326449e-06 -▁Veranstalter 4.0081513220326449e-06 -▁Vollendung 4.0081513220326449e-06 -▁aktiver 4.0081513220326449e-06 -▁discour 4.0081513220326449e-06 -▁indicating 4.0081513220326449e-06 -▁mounting 4.0081513220326449e-06 -▁precedent 4.0081513220326449e-06 -▁schlafen 4.0081513220326449e-06 -anet 4.004362899232992e-06 -ardi 4.004362899232992e-06 -aukasus 4.004362899232992e-06 -flicht 4.004362899232992e-06 -pul 4.004362899232992e-06 -steuers 4.004362899232992e-06 -strahlung 4.004362899232992e-06 -ugoslaw 4.004362899232992e-06 -wachstums 4.004362899232992e-06 -öck 4.004362899232992e-06 -▁EBIT 4.004362899232992e-06 -▁Seitdem 4.004362899232992e-06 -▁Speise 4.004362899232992e-06 -▁erneuerbarer 4.004362899232992e-06 -▁gröss 4.004362899232992e-06 -▁unterworfen 4.004362899232992e-06 -▁verlin 4.004362899232992e-06 --Fra 4.0005744764333391e-06 -assing 4.0005744764333391e-06 -fle 4.0005744764333391e-06 -points 4.0005744764333391e-06 -zusammen 4.0005744764333391e-06 -▁Arte 4.0005744764333391e-06 -▁Bestes 4.0005744764333391e-06 -▁Hope 4.0005744764333391e-06 -▁Jugendherberge 4.0005744764333391e-06 -▁Kundendienst 4.0005744764333391e-06 -▁N. 4.0005744764333391e-06 -▁NGL 4.0005744764333391e-06 -▁Ty 4.0005744764333391e-06 -▁brush 4.0005744764333391e-06 -▁enforced 4.0005744764333391e-06 -▁exported 4.0005744764333391e-06 -▁extremen 4.0005744764333391e-06 -▁persistent 4.0005744764333391e-06 -▁rises 4.0005744764333391e-06 -▁shipment 4.0005744764333391e-06 -5-00 3.9967860536336862e-06 -herstellung 3.9967860536336862e-06 -si 3.9967860536336862e-06 -stellers 3.9967860536336862e-06 -uffet 3.9967860536336862e-06 -urers 3.9967860536336862e-06 -wirtschaftlichen 3.9967860536336862e-06 -▁Aufschwung 3.9967860536336862e-06 -▁Erinnerungen 3.9967860536336862e-06 -▁Independent 3.9967860536336862e-06 -▁Paragraph 3.9967860536336862e-06 -▁Points 3.9967860536336862e-06 -▁Resultate 3.9967860536336862e-06 -▁Verhaft 3.9967860536336862e-06 -▁additives 3.9967860536336862e-06 -▁cache 3.9967860536336862e-06 -▁electr 3.9967860536336862e-06 -▁foreigners 3.9967860536336862e-06 -▁molec 3.9967860536336862e-06 --Em 3.9929976308340342e-06 -aced 3.9929976308340342e-06 -piece 3.9929976308340342e-06 -ransport 3.9929976308340342e-06 -spartnern 3.9929976308340342e-06 -wie 3.9929976308340342e-06 -▁Aussprachen 3.9929976308340342e-06 -▁Beförder 3.9929976308340342e-06 -▁Elis 3.9929976308340342e-06 -▁Prices 3.9929976308340342e-06 -▁Ratspräsidentin 3.9929976308340342e-06 -▁Spur 3.9929976308340342e-06 -▁Zivilisten 3.9929976308340342e-06 -▁café 3.9929976308340342e-06 -▁incoming 3.9929976308340342e-06 -▁inherent 3.9929976308340342e-06 -▁mex 3.9929976308340342e-06 -▁recruitment 3.9929976308340342e-06 -▁translators 3.9929976308340342e-06 --Fin 3.9892092080343813e-06 -tre 3.9892092080343813e-06 -▁650 3.9892092080343813e-06 -▁Currency 3.9892092080343813e-06 -▁Friedensprozess 3.9892092080343813e-06 -▁Haftbefehl 3.9892092080343813e-06 -▁Kro 3.9892092080343813e-06 -▁Kun 3.9892092080343813e-06 -▁Object 3.9892092080343813e-06 -▁Phoen 3.9892092080343813e-06 -▁Sounds 3.9892092080343813e-06 -▁Subs 3.9892092080343813e-06 -▁aren 3.9892092080343813e-06 -▁corrected 3.9892092080343813e-06 -▁illumin 3.9892092080343813e-06 -▁lyr 3.9892092080343813e-06 -▁nationality 3.9892092080343813e-06 -▁strategischer 3.9892092080343813e-06 -steine 3.9854207852347284e-06 -utzung 3.9854207852347284e-06 -▁1944 3.9854207852347284e-06 -▁3.2 3.9854207852347284e-06 -▁Administrat 3.9854207852347284e-06 -▁Evolution 3.9854207852347284e-06 -▁Firm 3.9854207852347284e-06 -▁Haftungs 3.9854207852347284e-06 -▁Hein 3.9854207852347284e-06 -▁Sendung 3.9854207852347284e-06 -▁Vervielfält 3.9854207852347284e-06 -▁Zusage 3.9854207852347284e-06 -▁anybody 3.9854207852347284e-06 -▁behand 3.9854207852347284e-06 -▁bell 3.9854207852347284e-06 -▁failures 3.9854207852347284e-06 -▁likewise 3.9854207852347284e-06 -▁reconstruct 3.9854207852347284e-06 -▁sahen 3.9854207852347284e-06 -▁scan 3.9854207852347284e-06 -▁sciences 3.9854207852347284e-06 -▁warme 3.9854207852347284e-06 -▁überraschend 3.9854207852347284e-06 -DN 3.9816323624350755e-06 -ipse 3.9816323624350755e-06 -kreuz 3.9816323624350755e-06 -orrent 3.9816323624350755e-06 -unities 3.9816323624350755e-06 -▁Automation 3.9816323624350755e-06 -▁GD 3.9816323624350755e-06 -▁ICEcat 3.9816323624350755e-06 -▁Zonen 3.9816323624350755e-06 -▁advocates 3.9816323624350755e-06 -▁appellieren 3.9816323624350755e-06 -▁ausgedehnt 3.9816323624350755e-06 -▁gene 3.9816323624350755e-06 -▁illegally 3.9816323624350755e-06 -▁overr 3.9816323624350755e-06 -▁recommends 3.9816323624350755e-06 -▁reichlich 3.9816323624350755e-06 -▁submission 3.9816323624350755e-06 -▁tox 3.9816323624350755e-06 -▁zurecht 3.9816323624350755e-06 -▁ägypt 3.9816323624350755e-06 -afrika 3.9778439396354226e-06 -onstr 3.9778439396354226e-06 -▁Custom 3.9778439396354226e-06 -▁Gehen 3.9778439396354226e-06 -▁Schwester 3.9778439396354226e-06 -▁Schwäche 3.9778439396354226e-06 -▁imagin 3.9778439396354226e-06 -▁kümmert 3.9778439396354226e-06 -▁liberalization 3.9778439396354226e-06 -▁liegende 3.9778439396354226e-06 -▁manuell 3.9778439396354226e-06 --Modus 3.97405551683577e-06 -aco 3.97405551683577e-06 -parl 3.97405551683577e-06 -zube 3.97405551683577e-06 -▁Caucasus 3.97405551683577e-06 -▁Motiv 3.97405551683577e-06 -▁Oase 3.97405551683577e-06 -▁Prad 3.97405551683577e-06 -▁Schatz 3.97405551683577e-06 -▁Wahlkampf 3.97405551683577e-06 -▁abstra 3.97405551683577e-06 -▁beziehungsweise 3.97405551683577e-06 -▁blieben 3.97405551683577e-06 -▁composer 3.97405551683577e-06 -▁connects 3.97405551683577e-06 -▁definite 3.97405551683577e-06 -▁hoteleigen 3.97405551683577e-06 -▁smell 3.97405551683577e-06 -▁vague 3.97405551683577e-06 -▁wendet 3.97405551683577e-06 -GO 3.9702670940361168e-06 -▁5.1 3.9702670940361168e-06 -▁Brunnen 3.9702670940361168e-06 -▁Kürzung 3.9702670940361168e-06 -▁Particip 3.9702670940361168e-06 -▁criticised 3.9702670940361168e-06 -▁democratisation 3.9702670940361168e-06 -▁honey 3.9702670940361168e-06 -▁konstruktiv 3.9702670940361168e-06 -▁veranlasst 3.9702670940361168e-06 -behand 3.9664786712364639e-06 -ermin 3.9664786712364639e-06 -etik 3.9664786712364639e-06 -inners 3.9664786712364639e-06 -owned 3.9664786712364639e-06 -plänen 3.9664786712364639e-06 -senal 3.9664786712364639e-06 -ubringen 3.9664786712364639e-06 -äme 3.9664786712364639e-06 -▁Dampfbad 3.9664786712364639e-06 -▁Images 3.9664786712364639e-06 -▁Jh 3.9664786712364639e-06 -▁Kohlend 3.9664786712364639e-06 -▁Konsultationen 3.9664786712364639e-06 -▁Oster 3.9664786712364639e-06 -▁archä 3.9664786712364639e-06 -▁console 3.9664786712364639e-06 -▁distanz 3.9664786712364639e-06 -▁grande 3.9664786712364639e-06 -▁gratitude 3.9664786712364639e-06 -▁incap 3.9664786712364639e-06 -▁jedermann 3.9664786712364639e-06 -▁lady 3.9664786712364639e-06 -▁tort 3.9664786712364639e-06 -▁übersetzen 3.9664786712364639e-06 -ERE 3.9626902484368119e-06 -aud 3.9626902484368119e-06 -elter 3.9626902484368119e-06 -▁Freundschaft 3.9626902484368119e-06 -▁Standorte 3.9626902484368119e-06 -▁Verspät 3.9626902484368119e-06 -▁conditioned 3.9626902484368119e-06 -▁debian 3.9626902484368119e-06 -▁gifts 3.9626902484368119e-06 -▁leistungsstark 3.9626902484368119e-06 -▁stack 3.9626902484368119e-06 -I. 3.958901825637159e-06 -exp 3.958901825637159e-06 -gewichts 3.958901825637159e-06 -▁Inde 3.958901825637159e-06 -▁Kampagnen 3.958901825637159e-06 -▁Mahlzeiten 3.958901825637159e-06 -▁UM 3.958901825637159e-06 -▁existierenden 3.958901825637159e-06 -▁fiber 3.958901825637159e-06 -▁friendship 3.958901825637159e-06 -▁inserted 3.958901825637159e-06 -▁likes 3.958901825637159e-06 -▁occurring 3.958901825637159e-06 -▁punish 3.958901825637159e-06 --Datenbank 3.9551134028375061e-06 -BI 3.9551134028375061e-06 -ceptions 3.9551134028375061e-06 -haupt 3.9551134028375061e-06 -spezifischen 3.9551134028375061e-06 -uni 3.9551134028375061e-06 -▁ALE-Fraktion 3.9551134028375061e-06 -▁Abbas 3.9551134028375061e-06 -▁Anlaß 3.9551134028375061e-06 -▁Ausgleichs 3.9551134028375061e-06 -▁Besichtigung 3.9551134028375061e-06 -▁GMO 3.9551134028375061e-06 -▁Gates 3.9551134028375061e-06 -▁Pfund 3.9551134028375061e-06 -▁STO 3.9551134028375061e-06 -▁accountable 3.9551134028375061e-06 -▁billiger 3.9551134028375061e-06 -▁delic 3.9551134028375061e-06 -▁ehe 3.9551134028375061e-06 -▁einm 3.9551134028375061e-06 -▁gezielte 3.9551134028375061e-06 -▁misleading 3.9551134028375061e-06 -▁senten 3.9551134028375061e-06 -▁travelers 3.9551134028375061e-06 -▁versteckt 3.9551134028375061e-06 -▁zügig 3.9551134028375061e-06 -29 3.9513249800378532e-06 -bildungs 3.9513249800378532e-06 -loaden 3.9513249800378532e-06 -▁Architect 3.9513249800378532e-06 -▁Cott 3.9513249800378532e-06 -▁Zigaretten 3.9513249800378532e-06 -▁allerg 3.9513249800378532e-06 -▁approxim 3.9513249800378532e-06 -▁borr 3.9513249800378532e-06 -▁demo 3.9513249800378532e-06 -▁ethnische 3.9513249800378532e-06 -▁handy 3.9513249800378532e-06 -▁hierher 3.9513249800378532e-06 -▁kroat 3.9513249800378532e-06 -▁stellten 3.9513249800378532e-06 -▁subt 3.9513249800378532e-06 -▁veto 3.9513249800378532e-06 -▁worsen 3.9513249800378532e-06 -▁Österreichs 3.9513249800378532e-06 -adores 3.9475365572382e-06 -enteil 3.9475365572382e-06 -gesetzte 3.9475365572382e-06 -itärer 3.9475365572382e-06 -izers 3.9475365572382e-06 -kampagne 3.9475365572382e-06 -loss 3.9475365572382e-06 -mart 3.9475365572382e-06 -vard 3.9475365572382e-06 -▁Behinderung 3.9475365572382e-06 -▁Bildes 3.9475365572382e-06 -▁Customs 3.9475365572382e-06 -▁Gur 3.9475365572382e-06 -▁Holl 3.9475365572382e-06 -▁Ital 3.9475365572382e-06 -▁PH 3.9475365572382e-06 -▁Package 3.9475365572382e-06 -▁belg 3.9475365572382e-06 -▁cafes 3.9475365572382e-06 -▁civilisation 3.9475365572382e-06 -▁o. 3.9475365572382e-06 -▁orn 3.9475365572382e-06 -▁prod 3.9475365572382e-06 -▁tastes 3.9475365572382e-06 -hältnismäßig 3.9437481344385474e-06 -oppeln 3.9437481344385474e-06 -rus 3.9437481344385474e-06 -ventionen 3.9437481344385474e-06 -▁0.5 3.9437481344385474e-06 -▁Breitband 3.9437481344385474e-06 -▁Expo 3.9437481344385474e-06 -▁Freiburg 3.9437481344385474e-06 -▁Planungs 3.9437481344385474e-06 -▁Rechtsetzung 3.9437481344385474e-06 -▁auferlegt 3.9437481344385474e-06 -▁gaben 3.9437481344385474e-06 -▁radioactive 3.9437481344385474e-06 -▁streichen 3.9437481344385474e-06 -▁submitting 3.9437481344385474e-06 -▁türk 3.9437481344385474e-06 -▁unvergess 3.9437481344385474e-06 -▁Ära 3.9437481344385474e-06 -renten 3.9399597116388945e-06 -texts 3.9399597116388945e-06 -▁Admir 3.9399597116388945e-06 -▁Eingliederung 3.9399597116388945e-06 -▁Kontinuität 3.9399597116388945e-06 -▁Schokolade 3.9399597116388945e-06 -▁Straft 3.9399597116388945e-06 -▁Tall 3.9399597116388945e-06 -▁Yahoo 3.9399597116388945e-06 -▁ermögl 3.9399597116388945e-06 -▁irakischen 3.9399597116388945e-06 -▁licensing 3.9399597116388945e-06 -▁quer 3.9399597116388945e-06 -schulden 3.9361712888392417e-06 -▁Ech 3.9361712888392417e-06 -▁Hamm 3.9361712888392417e-06 -▁Mob 3.9361712888392417e-06 -▁Skip 3.9361712888392417e-06 -▁afor 3.9361712888392417e-06 -▁appalling 3.9361712888392417e-06 -▁aufkommen 3.9361712888392417e-06 -▁e. 3.9361712888392417e-06 -▁eröffnete 3.9361712888392417e-06 -▁geleistete 3.9361712888392417e-06 -▁herzlichen 3.9361712888392417e-06 -▁hyd 3.9361712888392417e-06 -▁informellen 3.9361712888392417e-06 -▁unabdingbar 3.9361712888392417e-06 -nergy 3.93238286603959e-06 -▁101 3.93238286603959e-06 -▁ECOFIN 3.93238286603959e-06 -▁Finden 3.93238286603959e-06 -▁RC 3.93238286603959e-06 -▁Spielzeug 3.93238286603959e-06 -▁Tastatur 3.93238286603959e-06 -▁assumes 3.93238286603959e-06 -▁chest 3.93238286603959e-06 -▁gladly 3.93238286603959e-06 -▁unterstützte 3.93238286603959e-06 -▁verweigern 3.93238286603959e-06 --winning 3.9285944432399367e-06 -freundlich 3.9285944432399367e-06 -indest 3.9285944432399367e-06 -schle 3.9285944432399367e-06 -élé 3.9285944432399367e-06 -▁ALE 3.9285944432399367e-06 -▁Mandriva 3.9285944432399367e-06 -▁Ostse 3.9285944432399367e-06 -▁Trainer 3.9285944432399367e-06 -▁drastisch 3.9285944432399367e-06 -▁erfolgreicher 3.9285944432399367e-06 -▁judg 3.9285944432399367e-06 -▁reflekt 3.9285944432399367e-06 -▁twenty-f 3.9285944432399367e-06 -FF 3.9248060204402838e-06 -beteiligung 3.9248060204402838e-06 -heid 3.9248060204402838e-06 -isible 3.9248060204402838e-06 -tres 3.9248060204402838e-06 -vorschrift 3.9248060204402838e-06 -▁Amn 3.9248060204402838e-06 -▁Banquet 3.9248060204402838e-06 -▁Brazilian 3.9248060204402838e-06 -▁Luis 3.9248060204402838e-06 -▁Pinien 3.9248060204402838e-06 -▁Verbrechens 3.9248060204402838e-06 -▁Verwirrung 3.9248060204402838e-06 -▁breed 3.9248060204402838e-06 -▁celeb 3.9248060204402838e-06 -▁grundsätzliche 3.9248060204402838e-06 -▁idyllic 3.9248060204402838e-06 -▁menschen 3.9248060204402838e-06 -▁nette 3.9248060204402838e-06 -▁wander 3.9248060204402838e-06 -afel 3.9210175976406309e-06 -assis 3.9210175976406309e-06 -bum 3.9210175976406309e-06 -medien 3.9210175976406309e-06 -schaff 3.9210175976406309e-06 -ufung 3.9210175976406309e-06 -▁AGB 3.9210175976406309e-06 -▁Einwanderungspolitik 3.9210175976406309e-06 -▁Jos 3.9210175976406309e-06 -▁Mio 3.9210175976406309e-06 -▁Philosoph 3.9210175976406309e-06 -▁Tode 3.9210175976406309e-06 -▁Toy 3.9210175976406309e-06 -▁Weing 3.9210175976406309e-06 -▁caution 3.9210175976406309e-06 -▁high-level 3.9210175976406309e-06 -▁innovativer 3.9210175976406309e-06 -▁instrumental 3.9210175976406309e-06 -▁pued 3.9210175976406309e-06 -▁räum 3.9210175976406309e-06 -▁suppl 3.9210175976406309e-06 -Guide 3.917229174840978e-06 -dra 3.917229174840978e-06 -ellschaft 3.917229174840978e-06 -entum 3.917229174840978e-06 -hrten 3.917229174840978e-06 -idy 3.917229174840978e-06 -ifor 3.917229174840978e-06 -isciplinary 3.917229174840978e-06 -istically 3.917229174840978e-06 -prüf 3.917229174840978e-06 -pton 3.917229174840978e-06 -stanz 3.917229174840978e-06 -üte 3.917229174840978e-06 -▁Applikation 3.917229174840978e-06 -▁Caf 3.917229174840978e-06 -▁Dame 3.917229174840978e-06 -▁Ever 3.917229174840978e-06 -▁Horm 3.917229174840978e-06 -▁Komposition 3.917229174840978e-06 -▁SERV 3.917229174840978e-06 -▁Umstruktur 3.917229174840978e-06 -▁Zuhause 3.917229174840978e-06 -▁assass 3.917229174840978e-06 -▁demonstr 3.917229174840978e-06 -▁disappointment 3.917229174840978e-06 -▁geäußer 3.917229174840978e-06 -▁intermediate 3.917229174840978e-06 -▁mead 3.917229174840978e-06 -▁oils 3.917229174840978e-06 -▁overs 3.917229174840978e-06 -▁prizes 3.917229174840978e-06 -▁subway 3.917229174840978e-06 -ossa 3.9134407520413252e-06 -piele 3.9134407520413252e-06 -▁Anne 3.9134407520413252e-06 -▁Beil 3.9134407520413252e-06 -▁Haag 3.9134407520413252e-06 -▁Kund 3.9134407520413252e-06 -▁Mitteilungen 3.9134407520413252e-06 -▁Turbo 3.9134407520413252e-06 -▁Vizepräsidentin 3.9134407520413252e-06 -▁biofuels 3.9134407520413252e-06 -▁deterioration 3.9134407520413252e-06 -▁drain 3.9134407520413252e-06 -▁fuck 3.9134407520413252e-06 -▁reflecting 3.9134407520413252e-06 -▁umliegenden 3.9134407520413252e-06 -▁zitiere 3.9134407520413252e-06 -▁ökonomische 3.9134407520413252e-06 -ademie 3.9096523292416723e-06 -lac 3.9096523292416723e-06 -via 3.9096523292416723e-06 -▁Apoll 3.9096523292416723e-06 -▁Messung 3.9096523292416723e-06 -▁deposits 3.9096523292416723e-06 -▁ernsthaften 3.9096523292416723e-06 -▁folgender 3.9096523292416723e-06 -▁freiwilligen 3.9096523292416723e-06 -▁ink 3.9096523292416723e-06 -▁jenem 3.9096523292416723e-06 -▁mutually 3.9096523292416723e-06 -▁objection 3.9096523292416723e-06 -▁violated 3.9096523292416723e-06 -arest 3.9058639064420194e-06 -gefügt 3.9058639064420194e-06 -iox 3.9058639064420194e-06 -oons 3.9058639064420194e-06 -plantation 3.9058639064420194e-06 -seit 3.9058639064420194e-06 -teilten 3.9058639064420194e-06 -▁AK 3.9058639064420194e-06 -▁Capt 3.9058639064420194e-06 -▁Dalmat 3.9058639064420194e-06 -▁Stücke 3.9058639064420194e-06 -▁Wolken 3.9058639064420194e-06 -▁accessing 3.9058639064420194e-06 -▁benachbarten 3.9058639064420194e-06 -▁capitalist 3.9058639064420194e-06 -▁deutlichen 3.9058639064420194e-06 -▁emot 3.9058639064420194e-06 -▁fairs 3.9058639064420194e-06 -▁nest 3.9058639064420194e-06 -▁self-employed 3.9058639064420194e-06 --month 3.9020754836423673e-06 -cision 3.9020754836423673e-06 -erverein 3.9020754836423673e-06 -ipeline 3.9020754836423673e-06 -nels 3.9020754836423673e-06 -udget 3.9020754836423673e-06 -▁Accordingly 3.9020754836423673e-06 -▁Friedhof 3.9020754836423673e-06 -▁Kalifornien 3.9020754836423673e-06 -▁Mahl 3.9020754836423673e-06 -▁Quant 3.9020754836423673e-06 -▁Schwach 3.9020754836423673e-06 -▁aussch 3.9020754836423673e-06 -▁einschränken 3.9020754836423673e-06 -▁generic 3.9020754836423673e-06 -▁harmonise 3.9020754836423673e-06 -▁nötige 3.9020754836423673e-06 -▁understands 3.9020754836423673e-06 -▁verletzen 3.9020754836423673e-06 -register 3.8982870608427144e-06 -▁Brig 3.8982870608427144e-06 -▁Castel 3.8982870608427144e-06 -▁Elba 3.8982870608427144e-06 -▁Haushaltsmittel 3.8982870608427144e-06 -▁Various 3.8982870608427144e-06 -▁Waffe 3.8982870608427144e-06 -▁Wäre 3.8982870608427144e-06 -▁Yam 3.8982870608427144e-06 -▁appropriately 3.8982870608427144e-06 -▁fertigen 3.8982870608427144e-06 -▁kingdom 3.8982870608427144e-06 -▁ml 3.8982870608427144e-06 -▁penetr 3.8982870608427144e-06 -▁regulating 3.8982870608427144e-06 -▁urged 3.8982870608427144e-06 -▁wound 3.8982870608427144e-06 -LD 3.8944986380430615e-06 -eatr 3.8944986380430615e-06 -krip 3.8944986380430615e-06 -phas 3.8944986380430615e-06 -ständigen 3.8944986380430615e-06 -ís 3.8944986380430615e-06 -▁Mother 3.8944986380430615e-06 -▁Sammel 3.8944986380430615e-06 -▁Sebastian 3.8944986380430615e-06 -▁angen 3.8944986380430615e-06 -▁detailliert 3.8944986380430615e-06 -▁distortion 3.8944986380430615e-06 -▁entails 3.8944986380430615e-06 -▁erect 3.8944986380430615e-06 -▁moon 3.8944986380430615e-06 -▁robot 3.8944986380430615e-06 -▁slot 3.8944986380430615e-06 -▁surfing 3.8944986380430615e-06 -▁unterzogen 3.8944986380430615e-06 -▁vertrete 3.8944986380430615e-06 -▁ya 3.8944986380430615e-06 --Lösung 3.8907102152434087e-06 -Disk 3.8907102152434087e-06 -ensus 3.8907102152434087e-06 -ikum 3.8907102152434087e-06 -ilateral 3.8907102152434087e-06 -ogl 3.8907102152434087e-06 -schließung 3.8907102152434087e-06 -▁123 3.8907102152434087e-06 -▁Aviation 3.8907102152434087e-06 -▁Fitnessraum 3.8907102152434087e-06 -▁Lehne 3.8907102152434087e-06 -▁Mexican 3.8907102152434087e-06 -▁aufrichtig 3.8907102152434087e-06 -▁außergewöhnlich 3.8907102152434087e-06 -▁erschwert 3.8907102152434087e-06 -▁interc 3.8907102152434087e-06 -▁mitunter 3.8907102152434087e-06 -▁sexy 3.8907102152434087e-06 -▁ähnlicher 3.8907102152434087e-06 --Kr 3.8869217924437558e-06 -conf 3.8869217924437558e-06 -orenen 3.8869217924437558e-06 -werden 3.8869217924437558e-06 -▁Cus 3.8869217924437558e-06 -▁Laus 3.8869217924437558e-06 -▁Verschiebung 3.8869217924437558e-06 -▁Wies 3.8869217924437558e-06 -▁anonymous 3.8869217924437558e-06 -▁beginners 3.8869217924437558e-06 -▁counting 3.8869217924437558e-06 -▁elektronischer 3.8869217924437558e-06 -▁expresses 3.8869217924437558e-06 -▁fitting 3.8869217924437558e-06 -▁ni 3.8869217924437558e-06 -▁offensive 3.8869217924437558e-06 -▁veränderte 3.8869217924437558e-06 -▁zufällig 3.8869217924437558e-06 -600 3.8831333696441029e-06 -bogen 3.8831333696441029e-06 -ehörde 3.8831333696441029e-06 -haltungs 3.8831333696441029e-06 -teenth 3.8831333696441029e-06 -äufers 3.8831333696441029e-06 -▁Bezüglich 3.8831333696441029e-06 -▁Dampf 3.8831333696441029e-06 -▁Fluggesellschaft 3.8831333696441029e-06 -▁Orchester 3.8831333696441029e-06 -▁Throughout 3.8831333696441029e-06 -▁beeindruckt 3.8831333696441029e-06 -▁contamination 3.8831333696441029e-06 -▁fashionable 3.8831333696441029e-06 -▁genügen 3.8831333696441029e-06 -▁geänderten 3.8831333696441029e-06 -▁liberalen 3.8831333696441029e-06 -▁lö 3.8831333696441029e-06 -▁werdet 3.8831333696441029e-06 --Ap 3.87934494684445e-06 --Schn 3.87934494684445e-06 --Vertrags 3.87934494684445e-06 -die 3.87934494684445e-06 -odies 3.87934494684445e-06 -onische 3.87934494684445e-06 -▁Asylbewerber 3.87934494684445e-06 -▁Boy 3.87934494684445e-06 -▁Bungalows 3.87934494684445e-06 -▁Haf 3.87934494684445e-06 -▁KL 3.87934494684445e-06 -▁Kinderbetreuung 3.87934494684445e-06 -▁Latein 3.87934494684445e-06 -▁Quartet 3.87934494684445e-06 -▁Rasse 3.87934494684445e-06 -▁Underground 3.87934494684445e-06 -▁appearing 3.87934494684445e-06 -▁authoritarian 3.87934494684445e-06 -▁centr 3.87934494684445e-06 -▁donations 3.87934494684445e-06 -▁first-class 3.87934494684445e-06 -▁lacks 3.87934494684445e-06 -▁manages 3.87934494684445e-06 -▁physikal 3.87934494684445e-06 -.07 3.8755565240447971e-06 -alo 3.8755565240447971e-06 -bewertung 3.8755565240447971e-06 -cycl 3.8755565240447971e-06 -eilt 3.8755565240447971e-06 -gesehen 3.8755565240447971e-06 -rages 3.8755565240447971e-06 -riff 3.8755565240447971e-06 -spolitische 3.8755565240447971e-06 -▁1958 3.8755565240447971e-06 -▁1963 3.8755565240447971e-06 -▁3-Sterne-Hotel 3.8755565240447971e-06 -▁Airconditioning 3.8755565240447971e-06 -▁Beleg 3.8755565240447971e-06 -▁Hugh 3.8755565240447971e-06 -▁Lan 3.8755565240447971e-06 -▁Pariser 3.8755565240447971e-06 -▁Skulpt 3.8755565240447971e-06 -▁Tuscan 3.8755565240447971e-06 -▁Vertreterin 3.8755565240447971e-06 -▁Zulass 3.8755565240447971e-06 -▁begeistern 3.8755565240447971e-06 -▁consciousness 3.8755565240447971e-06 -▁entzieht 3.8755565240447971e-06 -▁grundsätz 3.8755565240447971e-06 -▁maß 3.8755565240447971e-06 -▁realization 3.8755565240447971e-06 -▁resistant 3.8755565240447971e-06 -▁therapeutic 3.8755565240447971e-06 -▁ungarische 3.8755565240447971e-06 -▁équ 3.8755565240447971e-06 -.e 3.871768101245145e-06 -zepten 3.871768101245145e-06 -▁Adopt 3.871768101245145e-06 -▁Apost 3.871768101245145e-06 -▁Beschleunigung 3.871768101245145e-06 -▁Blatt 3.871768101245145e-06 -▁Botschafter 3.871768101245145e-06 -▁Experiment 3.871768101245145e-06 -▁Sack 3.871768101245145e-06 -▁Sichtweise 3.871768101245145e-06 -▁Ziffern 3.871768101245145e-06 -▁alqu 3.871768101245145e-06 -▁e-c 3.871768101245145e-06 -▁europa 3.871768101245145e-06 -▁hebt 3.871768101245145e-06 -▁laufende 3.871768101245145e-06 -▁niedrigeren 3.871768101245145e-06 -▁portugiesische 3.871768101245145e-06 -▁scept 3.871768101245145e-06 -▁showers 3.871768101245145e-06 -▁sinnvolle 3.871768101245145e-06 -▁wholeheartedly 3.871768101245145e-06 -▁yours 3.871768101245145e-06 -▁zumeist 3.871768101245145e-06 -▁zusammenhän 3.871768101245145e-06 -aar 3.8679796784454921e-06 -belastung 3.8679796784454921e-06 -cin 3.8679796784454921e-06 -itution 3.8679796784454921e-06 -▁Berecht 3.8679796784454921e-06 -▁Dup 3.8679796784454921e-06 -▁Gemeinschaftsrechts 3.8679796784454921e-06 -▁Ing 3.8679796784454921e-06 -▁Ohren 3.8679796784454921e-06 -▁Taj 3.8679796784454921e-06 -▁bil 3.8679796784454921e-06 -▁enthusiasts 3.8679796784454921e-06 -▁ignor 3.8679796784454921e-06 -▁mußte 3.8679796784454921e-06 -▁palm 3.8679796784454921e-06 -▁static 3.8679796784454921e-06 -▁tournaments 3.8679796784454921e-06 -▁traces 3.8679796784454921e-06 --Agent 3.8641912556458393e-06 -achth 3.8641912556458393e-06 -ürgen 3.8641912556458393e-06 -▁Differenzen 3.8641912556458393e-06 -▁Entscheidungsfindung 3.8641912556458393e-06 -▁Exil 3.8641912556458393e-06 -▁Guar 3.8641912556458393e-06 -▁Informationsaustausch 3.8641912556458393e-06 -▁Millenniums 3.8641912556458393e-06 -▁disappointing 3.8641912556458393e-06 -▁emails 3.8641912556458393e-06 -▁ethischen 3.8641912556458393e-06 -▁feinen 3.8641912556458393e-06 -▁mixing 3.8641912556458393e-06 -▁namentlich 3.8641912556458393e-06 -▁scar 3.8641912556458393e-06 -▁zertifiziert 3.8641912556458393e-06 -LY 3.8604028328461864e-06 -fläch 3.8604028328461864e-06 -gency 3.8604028328461864e-06 -getreten 3.8604028328461864e-06 -grams 3.8604028328461864e-06 -käm 3.8604028328461864e-06 -ähre 3.8604028328461864e-06 -▁Barón 3.8604028328461864e-06 -▁Cann 3.8604028328461864e-06 -▁Nationalen 3.8604028328461864e-06 -▁Summ 3.8604028328461864e-06 -▁befaßt 3.8604028328461864e-06 -▁bewährten 3.8604028328461864e-06 -▁headed 3.8604028328461864e-06 -▁notable 3.8604028328461864e-06 -▁optimize 3.8604028328461864e-06 -▁schalten 3.8604028328461864e-06 -▁theater 3.8604028328461864e-06 -▁yea 3.8604028328461864e-06 -hesive 3.8566144100465335e-06 -iani 3.8566144100465335e-06 -lack 3.8566144100465335e-06 -ografie 3.8566144100465335e-06 -▁15.00 3.8566144100465335e-06 -▁Auschecken 3.8566144100465335e-06 -▁Bedauer 3.8566144100465335e-06 -▁Drink 3.8566144100465335e-06 -▁Jeff 3.8566144100465335e-06 -▁Kalten 3.8566144100465335e-06 -▁Konkur 3.8566144100465335e-06 -▁aft 3.8566144100465335e-06 -▁fauna 3.8566144100465335e-06 --Test 3.8528259872468806e-06 -ahlen 3.8528259872468806e-06 -itra 3.8528259872468806e-06 -organe 3.8528259872468806e-06 -reinigung 3.8528259872468806e-06 -trib 3.8528259872468806e-06 -usen 3.8528259872468806e-06 -ÜR 3.8528259872468806e-06 -▁Capp 3.8528259872468806e-06 -▁Disco 3.8528259872468806e-06 -▁Europäern 3.8528259872468806e-06 -▁Evangel 3.8528259872468806e-06 -▁Matthias 3.8528259872468806e-06 -▁Neuf 3.8528259872468806e-06 -▁Orch 3.8528259872468806e-06 -▁Wasserk 3.8528259872468806e-06 -▁Zugeständ 3.8528259872468806e-06 -▁climb 3.8528259872468806e-06 -▁fügen 3.8528259872468806e-06 -▁schott 3.8528259872468806e-06 -athl 3.8490375644472277e-06 -ecuted 3.8490375644472277e-06 -kanal 3.8490375644472277e-06 -kaufen 3.8490375644472277e-06 -modus 3.8490375644472277e-06 -rench 3.8490375644472277e-06 -usive 3.8490375644472277e-06 -▁Arabia 3.8490375644472277e-06 -▁Britt 3.8490375644472277e-06 -▁Louvre 3.8490375644472277e-06 -▁Maßstäbe 3.8490375644472277e-06 -▁Mé 3.8490375644472277e-06 -▁Pfeil 3.8490375644472277e-06 -▁Preises 3.8490375644472277e-06 -▁Rüstungs 3.8490375644472277e-06 -▁Zehn 3.8490375644472277e-06 -▁anspruchsvoll 3.8490375644472277e-06 -▁champion 3.8490375644472277e-06 -▁ermordet 3.8490375644472277e-06 -▁low-c 3.8490375644472277e-06 -▁zugenommen 3.8490375644472277e-06 -▁zuzustimmen 3.8490375644472277e-06 -▁½ 3.8490375644472277e-06 --Konferenz 3.8452491416475756e-06 -UST 3.8452491416475756e-06 -hart 3.8452491416475756e-06 -inski 3.8452491416475756e-06 -▁4.1 3.8452491416475756e-06 -▁4.5 3.8452491416475756e-06 -▁Clin 3.8452491416475756e-06 -▁Köst 3.8452491416475756e-06 -▁Volkswirtschaft 3.8452491416475756e-06 -▁anführen 3.8452491416475756e-06 -▁eingeschlossen 3.8452491416475756e-06 -▁exhibitors 3.8452491416475756e-06 -▁gefl 3.8452491416475756e-06 -▁gekürzt 3.8452491416475756e-06 -▁hindern 3.8452491416475756e-06 -▁judiciary 3.8452491416475756e-06 -▁pdf 3.8452491416475756e-06 -▁tall 3.8452491416475756e-06 -OF 3.8414607188479228e-06 -ieber 3.8414607188479228e-06 -kra 3.8414607188479228e-06 -ophone 3.8414607188479228e-06 -rect 3.8414607188479228e-06 -▁1949 3.8414607188479228e-06 -▁Adv 3.8414607188479228e-06 -▁Besten 3.8414607188479228e-06 -▁Cuban 3.8414607188479228e-06 -▁DivX 3.8414607188479228e-06 -▁Rückzug 3.8414607188479228e-06 -▁Schr 3.8414607188479228e-06 -▁Sogar 3.8414607188479228e-06 -▁Steine 3.8414607188479228e-06 -▁Style 3.8414607188479228e-06 -▁auszuarbeiten 3.8414607188479228e-06 -▁dringende 3.8414607188479228e-06 -▁echt 3.8414607188479228e-06 -▁einsehen 3.8414607188479228e-06 -▁einseitig 3.8414607188479228e-06 -▁geografischen 3.8414607188479228e-06 -▁geringeren 3.8414607188479228e-06 -▁locals 3.8414607188479228e-06 -▁luxuri 3.8414607188479228e-06 -▁ruin 3.8414607188479228e-06 -▁traveling 3.8414607188479228e-06 --Install 3.83767229604827e-06 -auftrag 3.83767229604827e-06 -serklärung 3.83767229604827e-06 -▁1939 3.83767229604827e-06 -▁FIFA 3.83767229604827e-06 -▁Medikamenten 3.83767229604827e-06 -▁Präzision 3.83767229604827e-06 -▁Stehende 3.83767229604827e-06 -▁Vorteilen 3.83767229604827e-06 -▁chapters 3.83767229604827e-06 -▁cocktails 3.83767229604827e-06 -▁dispatch 3.83767229604827e-06 -▁ecommerce 3.83767229604827e-06 -▁freiz 3.83767229604827e-06 -▁gemisch 3.83767229604827e-06 -▁grade 3.83767229604827e-06 -▁palästinensische 3.83767229604827e-06 -▁verank 3.83767229604827e-06 -▁übereinst 3.83767229604827e-06 -abu 3.833883873248617e-06 -atorische 3.833883873248617e-06 -chair 3.833883873248617e-06 -elsohne 3.833883873248617e-06 -ungal 3.833883873248617e-06 -▁Hyde 3.833883873248617e-06 -▁Imag 3.833883873248617e-06 -▁Komplett 3.833883873248617e-06 -▁Mehrzahl 3.833883873248617e-06 -▁SIE 3.833883873248617e-06 -▁Schnittstellen 3.833883873248617e-06 -▁Sv 3.833883873248617e-06 -▁Tram 3.833883873248617e-06 -▁amaz 3.833883873248617e-06 -▁awaits 3.833883873248617e-06 -▁beschriebenen 3.833883873248617e-06 -▁childhood 3.833883873248617e-06 -▁durchge 3.833883873248617e-06 -▁geschützten 3.833883873248617e-06 -▁gesund 3.833883873248617e-06 -▁irresponsible 3.833883873248617e-06 -▁pubs 3.833883873248617e-06 -▁sentences 3.833883873248617e-06 -aufenthalt 3.8300954504489641e-06 -ciliation 3.8300954504489641e-06 -eiteten 3.8300954504489641e-06 -elin 3.8300954504489641e-06 -entwicklungen 3.8300954504489641e-06 -esy 3.8300954504489641e-06 -farbe 3.8300954504489641e-06 -inische 3.8300954504489641e-06 -liegenden 3.8300954504489641e-06 -▁Boston 3.8300954504489641e-06 -▁Entscheid 3.8300954504489641e-06 -▁Honor 3.8300954504489641e-06 -▁Novotel 3.8300954504489641e-06 -▁adapting 3.8300954504489641e-06 -▁conceal 3.8300954504489641e-06 -▁disability 3.8300954504489641e-06 -▁internally 3.8300954504489641e-06 -▁poses 3.8300954504489641e-06 -▁unclear 3.8300954504489641e-06 -beschäftigung 3.8263070276493112e-06 -edite 3.8263070276493112e-06 -kenn 3.8263070276493112e-06 -ograp 3.8263070276493112e-06 -seher 3.8263070276493112e-06 -stext 3.8263070276493112e-06 -▁Ausflugs 3.8263070276493112e-06 -▁Etwa 3.8263070276493112e-06 -▁Ferrero-Waldner 3.8263070276493112e-06 -▁beta 3.8263070276493112e-06 -▁fla 3.8263070276493112e-06 -▁leid 3.8263070276493112e-06 -▁organisms 3.8263070276493112e-06 -▁stimulating 3.8263070276493112e-06 -egr 3.8225186048496583e-06 -praxis 3.8225186048496583e-06 -rekt 3.8225186048496583e-06 -yss 3.8225186048496583e-06 -ähle 3.8225186048496583e-06 -ätzungen 3.8225186048496583e-06 -▁3G 3.8225186048496583e-06 -▁Allen 3.8225186048496583e-06 -▁Auflage 3.8225186048496583e-06 -▁Geg 3.8225186048496583e-06 -▁Mauern 3.8225186048496583e-06 -▁Prozessen 3.8225186048496583e-06 -▁Tabl 3.8225186048496583e-06 -▁Tin 3.8225186048496583e-06 -▁acute 3.8225186048496583e-06 -▁ausländ 3.8225186048496583e-06 -▁beispiel 3.8225186048496583e-06 -▁beschädigt 3.8225186048496583e-06 -▁controvers 3.8225186048496583e-06 -▁identifiziert 3.8225186048496583e-06 -▁nous 3.8225186048496583e-06 -▁straightforward 3.8225186048496583e-06 -epper 3.8187301820500054e-06 -fu 3.8187301820500054e-06 -pler 3.8187301820500054e-06 -rand 3.8187301820500054e-06 -▁Amsterdamer 3.8187301820500054e-06 -▁Beein 3.8187301820500054e-06 -▁Concert 3.8187301820500054e-06 -▁Jeg 3.8187301820500054e-06 -▁Sozial- 3.8187301820500054e-06 -▁Submitted 3.8187301820500054e-06 -▁afghan 3.8187301820500054e-06 -▁dokumentiert 3.8187301820500054e-06 -▁doppelte 3.8187301820500054e-06 -▁empfehlenswert 3.8187301820500054e-06 -▁professionalism 3.8187301820500054e-06 -▁transposition 3.8187301820500054e-06 -▁vertraulich 3.8187301820500054e-06 -IDE 3.8149417592503534e-06 -haftigkeit 3.8149417592503534e-06 -inity 3.8149417592503534e-06 -istance 3.8149417592503534e-06 -kular 3.8149417592503534e-06 -papier 3.8149417592503534e-06 -ém 3.8149417592503534e-06 -▁Address 3.8149417592503534e-06 -▁Arzneimitteln 3.8149417592503534e-06 -▁Asp 3.8149417592503534e-06 -▁Hals 3.8149417592503534e-06 -▁Knoten 3.8149417592503534e-06 -▁Liberties 3.8149417592503534e-06 -▁acquaint 3.8149417592503534e-06 -▁bail 3.8149417592503534e-06 -▁encouragement 3.8149417592503534e-06 -▁influential 3.8149417592503534e-06 -▁morg 3.8149417592503534e-06 -▁offline 3.8149417592503534e-06 -▁weitergeleitet 3.8149417592503534e-06 -alone 3.8111533364507e-06 -atri 3.8111533364507e-06 -faktoren 3.8111533364507e-06 -▁Agrars 3.8111533364507e-06 -▁Communities 3.8111533364507e-06 -▁Gaulle 3.8111533364507e-06 -▁Investitionsbank 3.8111533364507e-06 -▁Lücken 3.8111533364507e-06 -▁Nuclear 3.8111533364507e-06 -▁Schuhe 3.8111533364507e-06 -▁Sicherheitsrat 3.8111533364507e-06 -▁Verschmutzung 3.8111533364507e-06 -▁bab 3.8111533364507e-06 -▁drew 3.8111533364507e-06 -▁eingebettet 3.8111533364507e-06 -▁illegaler 3.8111533364507e-06 -▁matur 3.8111533364507e-06 -▁moralischen 3.8111533364507e-06 -▁pending 3.8111533364507e-06 -▁proprietary 3.8111533364507e-06 -▁va 3.8111533364507e-06 -▁wechs 3.8111533364507e-06 -▁zuverläss 3.8111533364507e-06 -fürst 3.8073649136510476e-06 -ioned 3.8073649136510476e-06 -ollt 3.8073649136510476e-06 -promenade 3.8073649136510476e-06 -rs 3.8073649136510476e-06 -seg 3.8073649136510476e-06 -▁Ausländer 3.8073649136510476e-06 -▁Kais 3.8073649136510476e-06 -▁Malaria 3.8073649136510476e-06 -▁Spekt 3.8073649136510476e-06 -▁Steve 3.8073649136510476e-06 -▁Unic 3.8073649136510476e-06 -▁dictator 3.8073649136510476e-06 -▁drift 3.8073649136510476e-06 -▁judged 3.8073649136510476e-06 -▁peripheral 3.8073649136510476e-06 -▁probieren 3.8073649136510476e-06 -▁sid 3.8073649136510476e-06 -▁suicide 3.8073649136510476e-06 -▁Å 3.8073649136510476e-06 -afi 3.8035764908513947e-06 -ateien 3.8035764908513947e-06 -geordnete 3.8035764908513947e-06 -ommod 3.8035764908513947e-06 -pit 3.8035764908513947e-06 -sektors 3.8035764908513947e-06 -ulichkeit 3.8035764908513947e-06 -▁Biot 3.8035764908513947e-06 -▁Carlo 3.8035764908513947e-06 -▁Carn 3.8035764908513947e-06 -▁Goals 3.8035764908513947e-06 -▁Maustaste 3.8035764908513947e-06 -▁Philharmon 3.8035764908513947e-06 -▁Tell 3.8035764908513947e-06 -▁kreativen 3.8035764908513947e-06 -▁kristall 3.8035764908513947e-06 -▁mangelt 3.8035764908513947e-06 -▁nachzukommen 3.8035764908513947e-06 -▁onwards 3.8035764908513947e-06 -▁sched 3.8035764908513947e-06 -▁succ 3.8035764908513947e-06 -▁trocken 3.8035764908513947e-06 -KO 3.7997880680517418e-06 -ria 3.7997880680517418e-06 -▁Amazon 3.7997880680517418e-06 -▁Moon 3.7997880680517418e-06 -▁Schöne 3.7997880680517418e-06 -▁Trip 3.7997880680517418e-06 -▁applicants 3.7997880680517418e-06 -▁einmaligen 3.7997880680517418e-06 -▁romantischen 3.7997880680517418e-06 -▁ville 3.7997880680517418e-06 -▁vorab 3.7997880680517418e-06 -▁4-Sterne 3.7959996452520889e-06 -▁Delph 3.7959996452520889e-06 -▁Diplomatie 3.7959996452520889e-06 -▁E-Commerce 3.7959996452520889e-06 -▁Gründungs 3.7959996452520889e-06 -▁Mannheim 3.7959996452520889e-06 -▁Termini 3.7959996452520889e-06 -▁VII 3.7959996452520889e-06 -▁Vorgänge 3.7959996452520889e-06 -▁cad 3.7959996452520889e-06 -▁dezentral 3.7959996452520889e-06 -▁elimin 3.7959996452520889e-06 -▁fantast 3.7959996452520889e-06 -▁umzu 3.7959996452520889e-06 -As 3.7922112224524364e-06 -gment 3.7922112224524364e-06 -jour 3.7922112224524364e-06 -regel 3.7922112224524364e-06 -schild 3.7922112224524364e-06 -▁11th 3.7922112224524364e-06 -▁Böge 3.7922112224524364e-06 -▁Darm 3.7922112224524364e-06 -▁Oscar 3.7922112224524364e-06 -▁Pensionen 3.7922112224524364e-06 -▁benachteiligten 3.7922112224524364e-06 -▁geehrten 3.7922112224524364e-06 -▁gescheitert 3.7922112224524364e-06 -▁gewohnt 3.7922112224524364e-06 -▁smoothly 3.7922112224524364e-06 -▁unabhängiger 3.7922112224524364e-06 -MI 3.7884227996527835e-06 -arin 3.7884227996527835e-06 -isan 3.7884227996527835e-06 -äuse 3.7884227996527835e-06 -örm 3.7884227996527835e-06 -▁Erfordernisse 3.7884227996527835e-06 -▁Handels- 3.7884227996527835e-06 -▁Kurt 3.7884227996527835e-06 -▁MSN 3.7884227996527835e-06 -▁Opport 3.7884227996527835e-06 -▁Pok 3.7884227996527835e-06 -▁Surround 3.7884227996527835e-06 -▁achie 3.7884227996527835e-06 -▁begriffen 3.7884227996527835e-06 -▁beliebte 3.7884227996527835e-06 -▁bestimmtes 3.7884227996527835e-06 -▁columns 3.7884227996527835e-06 -▁pag 3.7884227996527835e-06 -▁parish 3.7884227996527835e-06 -▁repräsentiert 3.7884227996527835e-06 -▁slowen 3.7884227996527835e-06 -▁zuversichtlich 3.7884227996527835e-06 -IPS 3.7846343768531307e-06 -LC 3.7846343768531307e-06 -ciation 3.7846343768531307e-06 -cio 3.7846343768531307e-06 -essp 3.7846343768531307e-06 -oppy 3.7846343768531307e-06 -pi 3.7846343768531307e-06 -öpfer 3.7846343768531307e-06 -▁1990er 3.7846343768531307e-06 -▁Amal 3.7846343768531307e-06 -▁Anschein 3.7846343768531307e-06 -▁Create 3.7846343768531307e-06 -▁Ergänz 3.7846343768531307e-06 -▁Finanzdienstleistungen 3.7846343768531307e-06 -▁GA 3.7846343768531307e-06 -▁Gewiss 3.7846343768531307e-06 -▁Koalition 3.7846343768531307e-06 -▁Magazin 3.7846343768531307e-06 -▁NCC-1701 3.7846343768531307e-06 -▁Registrierungs 3.7846343768531307e-06 -▁Tageszeit 3.7846343768531307e-06 -▁Widerstands 3.7846343768531307e-06 -▁annimmt 3.7846343768531307e-06 -▁balconies 3.7846343768531307e-06 -▁celebrations 3.7846343768531307e-06 -▁elekt 3.7846343768531307e-06 -▁erricht 3.7846343768531307e-06 -▁geringfüg 3.7846343768531307e-06 -▁trucks 3.7846343768531307e-06 --Fil 3.7808459540534778e-06 -amma 3.7808459540534778e-06 -chtigkeit 3.7808459540534778e-06 -flut 3.7808459540534778e-06 -grem 3.7808459540534778e-06 -mand 3.7808459540534778e-06 -ässern 3.7808459540534778e-06 -▁Begriffs 3.7808459540534778e-06 -▁Brenner 3.7808459540534778e-06 -▁Commerce 3.7808459540534778e-06 -▁Einverständnis 3.7808459540534778e-06 -▁Erscheinung 3.7808459540534778e-06 -▁Herunterladen 3.7808459540534778e-06 -▁Mobiltelef 3.7808459540534778e-06 -▁Qualifikation 3.7808459540534778e-06 -▁Stable 3.7808459540534778e-06 -▁Swimming 3.7808459540534778e-06 -▁Tours 3.7808459540534778e-06 -▁absent 3.7808459540534778e-06 -▁approaching 3.7808459540534778e-06 -▁enfor 3.7808459540534778e-06 -▁guys 3.7808459540534778e-06 -▁transportiert 3.7808459540534778e-06 -▁unav 3.7808459540534778e-06 -allo 3.7770575312538253e-06 -eca 3.7770575312538253e-06 -gesetzes 3.7770575312538253e-06 -igan 3.7770575312538253e-06 -overy 3.7770575312538253e-06 -problemen 3.7770575312538253e-06 -tress 3.7770575312538253e-06 -zeigten 3.7770575312538253e-06 -zielen 3.7770575312538253e-06 -η 3.7770575312538253e-06 -▁HE 3.7770575312538253e-06 -▁Landhaus 3.7770575312538253e-06 -▁Nürnberg 3.7770575312538253e-06 -▁Ortes 3.7770575312538253e-06 -▁Pause 3.7770575312538253e-06 -▁Russen 3.7770575312538253e-06 -▁Salzkammer 3.7770575312538253e-06 -▁alla 3.7770575312538253e-06 -▁angemeldet 3.7770575312538253e-06 -▁beides 3.7770575312538253e-06 -▁enthusiastic 3.7770575312538253e-06 -▁impro 3.7770575312538253e-06 -▁nachhaltiges 3.7770575312538253e-06 -▁poster 3.7770575312538253e-06 -▁squar 3.7770575312538253e-06 -▁unnötig 3.7770575312538253e-06 -▁volcan 3.7770575312538253e-06 -comm 3.7732691084541724e-06 -echnologien 3.7732691084541724e-06 -gram 3.7732691084541724e-06 -unnen 3.7732691084541724e-06 -zuwirken 3.7732691084541724e-06 -▁Ambassador 3.7732691084541724e-06 -▁Champion 3.7732691084541724e-06 -▁Companies 3.7732691084541724e-06 -▁Malware 3.7732691084541724e-06 -▁Versuchen 3.7732691084541724e-06 -▁adhere 3.7732691084541724e-06 -▁alb 3.7732691084541724e-06 -▁bee 3.7732691084541724e-06 -▁dreißig 3.7732691084541724e-06 -▁emphasises 3.7732691084541724e-06 -▁empire 3.7732691084541724e-06 -▁opposing 3.7732691084541724e-06 -▁pisc 3.7732691084541724e-06 -▁potenzielle 3.7732691084541724e-06 -▁seasons 3.7732691084541724e-06 -▁singing 3.7732691084541724e-06 -▁zugeben 3.7732691084541724e-06 -▁zusammengearbeitet 3.7732691084541724e-06 -alien 3.7694806856545195e-06 -bauern 3.7694806856545195e-06 -υ 3.7694806856545195e-06 -▁10,000 3.7694806856545195e-06 -▁90er 3.7694806856545195e-06 -▁Andernfalls 3.7694806856545195e-06 -▁Crespo 3.7694806856545195e-06 -▁Dos 3.7694806856545195e-06 -▁Nähr 3.7694806856545195e-06 -▁Pedro 3.7694806856545195e-06 -▁WW 3.7694806856545195e-06 -▁fluct 3.7694806856545195e-06 -elier 3.7656922628548666e-06 -frame 3.7656922628548666e-06 -level 3.7656922628548666e-06 -mag 3.7656922628548666e-06 -mobil 3.7656922628548666e-06 -▁Azur 3.7656922628548666e-06 -▁Kurden 3.7656922628548666e-06 -▁Nutzungsbedingungen 3.7656922628548666e-06 -▁Orl 3.7656922628548666e-06 -▁Siedlung 3.7656922628548666e-06 -▁Solid 3.7656922628548666e-06 -▁einzuräumen 3.7656922628548666e-06 -▁gegenseit 3.7656922628548666e-06 -▁impair 3.7656922628548666e-06 -▁komplexer 3.7656922628548666e-06 -▁radioakt 3.7656922628548666e-06 -▁spher 3.7656922628548666e-06 -▁Öffentliche 3.7656922628548666e-06 -atility 3.7619038400552141e-06 -ifffahrt 3.7619038400552141e-06 -lander 3.7619038400552141e-06 -oleum 3.7619038400552141e-06 -strand 3.7619038400552141e-06 -ußt 3.7619038400552141e-06 -zten 3.7619038400552141e-06 -▁Daß 3.7619038400552141e-06 -▁Fourthly 3.7619038400552141e-06 -▁García 3.7619038400552141e-06 -▁Iss 3.7619038400552141e-06 -▁K. 3.7619038400552141e-06 -▁Regisse 3.7619038400552141e-06 -▁Winkel 3.7619038400552141e-06 -▁arc 3.7619038400552141e-06 -▁archaeological 3.7619038400552141e-06 -▁charging 3.7619038400552141e-06 -▁concluding 3.7619038400552141e-06 -▁hem 3.7619038400552141e-06 -▁mediterranen 3.7619038400552141e-06 -▁reichhaltiges 3.7619038400552141e-06 -▁sud 3.7619038400552141e-06 --air 3.7581154172555613e-06 -idi 3.7581154172555613e-06 -modern 3.7581154172555613e-06 -Ž 3.7581154172555613e-06 -▁Bosch 3.7581154172555613e-06 -▁Challenge 3.7581154172555613e-06 -▁Geschäfts- 3.7581154172555613e-06 -▁INI 3.7581154172555613e-06 -▁Kombin 3.7581154172555613e-06 -▁Mitchell 3.7581154172555613e-06 -▁calculations 3.7581154172555613e-06 -▁catastroph 3.7581154172555613e-06 -▁konvert 3.7581154172555613e-06 -▁ritual 3.7581154172555613e-06 -▁teen 3.7581154172555613e-06 -▁verspricht 3.7581154172555613e-06 -900 3.7543269944559084e-06 -alla 3.7543269944559084e-06 -ecutor 3.7543269944559084e-06 -making 3.7543269944559084e-06 -ministers 3.7543269944559084e-06 -stücken 3.7543269944559084e-06 -vorgang 3.7543269944559084e-06 -▁Auffassungen 3.7543269944559084e-06 -▁Customers 3.7543269944559084e-06 -▁Organismen 3.7543269944559084e-06 -▁Poliz 3.7543269944559084e-06 -▁Sites 3.7543269944559084e-06 -▁hinten 3.7543269944559084e-06 -▁jeher 3.7543269944559084e-06 -▁mold 3.7543269944559084e-06 -▁pumps 3.7543269944559084e-06 -▁restoring 3.7543269944559084e-06 -▁seeds 3.7543269944559084e-06 -▁seller 3.7543269944559084e-06 -berater 3.7505385716562555e-06 -eses 3.7505385716562555e-06 -essa 3.7505385716562555e-06 -ilität 3.7505385716562555e-06 -ugend 3.7505385716562555e-06 -▁Fashion 3.7505385716562555e-06 -▁Linke 3.7505385716562555e-06 -▁Lower 3.7505385716562555e-06 -▁Manufact 3.7505385716562555e-06 -▁Mittele 3.7505385716562555e-06 -▁Neph 3.7505385716562555e-06 -▁Remember 3.7505385716562555e-06 -▁Sommers 3.7505385716562555e-06 -▁Verweis 3.7505385716562555e-06 -▁Whenever 3.7505385716562555e-06 -▁ausgegangen 3.7505385716562555e-06 -▁comprising 3.7505385716562555e-06 -▁erbracht 3.7505385716562555e-06 -▁horny 3.7505385716562555e-06 -▁individ 3.7505385716562555e-06 -▁invo 3.7505385716562555e-06 -▁methodology 3.7505385716562555e-06 -▁monopolies 3.7505385716562555e-06 -▁scenarios 3.7505385716562555e-06 -▁verfassungs 3.7505385716562555e-06 -forderung 3.746750148856603e-06 -icle 3.746750148856603e-06 -weiger 3.746750148856603e-06 -▁Bangemann 3.746750148856603e-06 -▁Fi 3.746750148856603e-06 -▁Gesichts 3.746750148856603e-06 -▁Richtlinienvorschlag 3.746750148856603e-06 -▁Sprung 3.746750148856603e-06 -▁Verschiedene 3.746750148856603e-06 -▁angenommene 3.746750148856603e-06 -▁confined 3.746750148856603e-06 -▁economists 3.746750148856603e-06 -▁elektrische 3.746750148856603e-06 -▁entstehenden 3.746750148856603e-06 -▁hop 3.746750148856603e-06 -▁injust 3.746750148856603e-06 -▁lächer 3.746750148856603e-06 -▁null 3.746750148856603e-06 -▁restra 3.746750148856603e-06 -▁seri 3.746750148856603e-06 -▁stilvolle 3.746750148856603e-06 -chner 3.74296172605695e-06 -feuer 3.74296172605695e-06 -izzer 3.74296172605695e-06 -scheiden 3.74296172605695e-06 -verfolgung 3.74296172605695e-06 -▁1933 3.74296172605695e-06 -▁Decken 3.74296172605695e-06 -▁Grunds 3.74296172605695e-06 -▁Prozentsatz 3.74296172605695e-06 -▁Schlussfolgerung 3.74296172605695e-06 -▁Story 3.74296172605695e-06 -▁condemnation 3.74296172605695e-06 -▁getät 3.74296172605695e-06 -▁resume 3.74296172605695e-06 -▁somebody 3.74296172605695e-06 --Benz 3.7391733032572972e-06 -ints 3.7391733032572972e-06 -äufen 3.7391733032572972e-06 -▁Deliver 3.7391733032572972e-06 -▁Hybrid 3.7391733032572972e-06 -▁Laufenden 3.7391733032572972e-06 -▁Mitgefühl 3.7391733032572972e-06 -▁NGL-Fraktion 3.7391733032572972e-06 -▁Namib 3.7391733032572972e-06 -▁Werner 3.7391733032572972e-06 -▁allgemeinem 3.7391733032572972e-06 -▁ausgeweitet 3.7391733032572972e-06 -▁demonstrieren 3.7391733032572972e-06 -▁refuge 3.7391733032572972e-06 -ammt 3.7353848804576443e-06 -arks 3.7353848804576443e-06 -ungsmöglichkeiten 3.7353848804576443e-06 -▁Assozi 3.7353848804576443e-06 -▁Automotive 3.7353848804576443e-06 -▁Calle 3.7353848804576443e-06 -▁Dereg 3.7353848804576443e-06 -▁Nevada 3.7353848804576443e-06 -▁Rentner 3.7353848804576443e-06 -▁eingesetzten 3.7353848804576443e-06 -▁günst 3.7353848804576443e-06 -▁interv 3.7353848804576443e-06 -▁learnt 3.7353848804576443e-06 -▁lieb 3.7353848804576443e-06 -▁nationalism 3.7353848804576443e-06 -▁polar 3.7353848804576443e-06 -▁servants 3.7353848804576443e-06 -▁stattfindenden 3.7353848804576443e-06 -backs 3.7315964576579919e-06 -ckung 3.7315964576579919e-06 -firma 3.7315964576579919e-06 -getr 3.7315964576579919e-06 -itiz 3.7315964576579919e-06 -kä 3.7315964576579919e-06 -lysées 3.7315964576579919e-06 -messungen 3.7315964576579919e-06 -öffnung 3.7315964576579919e-06 -▁Banner 3.7315964576579919e-06 -▁Besondere 3.7315964576579919e-06 -▁Erw 3.7315964576579919e-06 -▁Getränk 3.7315964576579919e-06 -▁Horizont 3.7315964576579919e-06 -▁Münzen 3.7315964576579919e-06 -▁Oil 3.7315964576579919e-06 -▁Racing 3.7315964576579919e-06 -▁Umfrage 3.7315964576579919e-06 -▁Vernichtung 3.7315964576579919e-06 -▁Währungsfonds 3.7315964576579919e-06 -▁ext 3.7315964576579919e-06 -▁reformed 3.7315964576579919e-06 -▁tune 3.7315964576579919e-06 -▁unequiv 3.7315964576579919e-06 -▁wide-ranging 3.7315964576579919e-06 -atischer 3.727808034858339e-06 -ieri 3.727808034858339e-06 -olas 3.727808034858339e-06 -worth 3.727808034858339e-06 -ädagog 3.727808034858339e-06 -▁Assoziierungsabkommen 3.727808034858339e-06 -▁Durchschnitts 3.727808034858339e-06 -▁Kyi 3.727808034858339e-06 -▁LV 3.727808034858339e-06 -▁Province 3.727808034858339e-06 -▁Terroran 3.727808034858339e-06 -▁anerkannte 3.727808034858339e-06 -▁cellar 3.727808034858339e-06 -▁inspiriert 3.727808034858339e-06 -▁maximalen 3.727808034858339e-06 -▁requesting 3.727808034858339e-06 -▁unzähligen 3.727808034858339e-06 -born 3.7240196120586861e-06 -hungen 3.7240196120586861e-06 -konstruktion 3.7240196120586861e-06 -kämpft 3.7240196120586861e-06 -ungsverfahrens 3.7240196120586861e-06 -▁Eier 3.7240196120586861e-06 -▁Interoper 3.7240196120586861e-06 -▁Mehrheits 3.7240196120586861e-06 -▁Nationale 3.7240196120586861e-06 -▁Thematik 3.7240196120586861e-06 -▁Wahlkreis 3.7240196120586861e-06 -▁befest 3.7240196120586861e-06 -▁beverages 3.7240196120586861e-06 -▁bisweilen 3.7240196120586861e-06 -▁ceilings 3.7240196120586861e-06 -▁disparities 3.7240196120586861e-06 -▁erlaub 3.7240196120586861e-06 -▁kostspiel 3.7240196120586861e-06 -▁terra 3.7240196120586861e-06 -▁voranz 3.7240196120586861e-06 -76 3.7202311892590332e-06 -asion 3.7202311892590332e-06 -lair 3.7202311892590332e-06 -ränkte 3.7202311892590332e-06 -đ 3.7202311892590332e-06 -▁280 3.7202311892590332e-06 -▁Animation 3.7202311892590332e-06 -▁Anzahlung 3.7202311892590332e-06 -▁Birmingham 3.7202311892590332e-06 -▁Bürgermeister 3.7202311892590332e-06 -▁Chic 3.7202311892590332e-06 -▁Demonstrationen 3.7202311892590332e-06 -▁Einheim 3.7202311892590332e-06 -▁Emperor 3.7202311892590332e-06 -▁Ries 3.7202311892590332e-06 -▁Session 3.7202311892590332e-06 -▁Urlauber 3.7202311892590332e-06 -▁brasil 3.7202311892590332e-06 -▁footing 3.7202311892590332e-06 -▁mature 3.7202311892590332e-06 -▁propos 3.7202311892590332e-06 -▁stellvertreten 3.7202311892590332e-06 -drücken 3.7164427664593807e-06 -grä 3.7164427664593807e-06 -städten 3.7164427664593807e-06 -weile 3.7164427664593807e-06 - 3.7164427664593807e-06 -▁Balt 3.7164427664593807e-06 -▁Christoph 3.7164427664593807e-06 -▁Clo 3.7164427664593807e-06 -▁Eiffel 3.7164427664593807e-06 -▁GASP 3.7164427664593807e-06 -▁Gefäng 3.7164427664593807e-06 -▁Grüne 3.7164427664593807e-06 -▁Kil 3.7164427664593807e-06 -▁POS 3.7164427664593807e-06 -▁Suu 3.7164427664593807e-06 -▁Treas 3.7164427664593807e-06 -▁ineffective 3.7164427664593807e-06 -▁optimistic 3.7164427664593807e-06 -▁tele 3.7164427664593807e-06 -▁ökonom 3.7164427664593807e-06 --Con 3.7126543436597278e-06 -ROM 3.7126543436597278e-06 -Tra 3.7126543436597278e-06 -ext 3.7126543436597278e-06 -icles 3.7126543436597278e-06 -verantwort 3.7126543436597278e-06 -▁1970s 3.7126543436597278e-06 -▁Brava 3.7126543436597278e-06 -▁DSLR 3.7126543436597278e-06 -▁Oomen-Ruijten 3.7126543436597278e-06 -▁Schäd 3.7126543436597278e-06 -▁Verbraucherpolitik 3.7126543436597278e-06 -▁bestmögliche 3.7126543436597278e-06 -▁empower 3.7126543436597278e-06 -▁esta 3.7126543436597278e-06 -▁grace 3.7126543436597278e-06 -▁hellen 3.7126543436597278e-06 -▁inhuman 3.7126543436597278e-06 -▁jewel 3.7126543436597278e-06 -▁kub 3.7126543436597278e-06 -▁lengthy 3.7126543436597278e-06 -▁synthetic 3.7126543436597278e-06 -▁voltage 3.7126543436597278e-06 -▁̈ 3.7126543436597278e-06 -etrieb 3.7088659208600749e-06 -prozessen 3.7088659208600749e-06 -üllt 3.7088659208600749e-06 -ützten 3.7088659208600749e-06 -▁Democrat 3.7088659208600749e-06 -▁Dominion 3.7088659208600749e-06 -▁Meldung 3.7088659208600749e-06 -▁Plural 3.7088659208600749e-06 -▁angesiedelt 3.7088659208600749e-06 -▁auszuschließen 3.7088659208600749e-06 -▁auszut 3.7088659208600749e-06 -▁cleaned 3.7088659208600749e-06 -▁conquer 3.7088659208600749e-06 -▁erhöhten 3.7088659208600749e-06 -▁erklär 3.7088659208600749e-06 -▁exemplary 3.7088659208600749e-06 -▁forcing 3.7088659208600749e-06 -▁schlug 3.7088659208600749e-06 -▁shaping 3.7088659208600749e-06 -endete 3.705077498060422e-06 -offee 3.705077498060422e-06 -▁-E 3.705077498060422e-06 -▁Cel 3.705077498060422e-06 -▁Einzelpersonen 3.705077498060422e-06 -▁Jar 3.705077498060422e-06 -▁Javascript 3.705077498060422e-06 -▁Kofin 3.705077498060422e-06 -▁Lücke 3.705077498060422e-06 -▁MC 3.705077498060422e-06 -▁Malm 3.705077498060422e-06 -▁Nicolas 3.705077498060422e-06 -▁Schichten 3.705077498060422e-06 -▁Schwed 3.705077498060422e-06 -▁Speichern 3.705077498060422e-06 -▁Tschechische 3.705077498060422e-06 -▁allevi 3.705077498060422e-06 -▁capita 3.705077498060422e-06 -▁gefährd 3.705077498060422e-06 -▁gestrigen 3.705077498060422e-06 -▁gleichzeit 3.705077498060422e-06 -▁inability 3.705077498060422e-06 -▁niederländische 3.705077498060422e-06 -▁unbea 3.705077498060422e-06 -▁verzichtet 3.705077498060422e-06 -▁weekends 3.705077498060422e-06 -decke 3.7012890752607696e-06 -heorie 3.7012890752607696e-06 -▁Aktionsprogramm 3.7012890752607696e-06 -▁Eco 3.7012890752607696e-06 -▁Grä 3.7012890752607696e-06 -▁Handwerk 3.7012890752607696e-06 -▁Meinungsverschieden 3.7012890752607696e-06 -▁Nova 3.7012890752607696e-06 -▁Sonnent 3.7012890752607696e-06 -▁Wertsch 3.7012890752607696e-06 -▁adm 3.7012890752607696e-06 -▁castles 3.7012890752607696e-06 -▁consultancy 3.7012890752607696e-06 -▁cosmetic 3.7012890752607696e-06 -▁creatures 3.7012890752607696e-06 -▁infection 3.7012890752607696e-06 -▁patron 3.7012890752607696e-06 -PI 3.6975006524611167e-06 -bestände 3.6975006524611167e-06 -faden 3.6975006524611167e-06 -fia 3.6975006524611167e-06 -höre 3.6975006524611167e-06 -rett 3.6975006524611167e-06 -ührung 3.6975006524611167e-06 -▁Apr 3.6975006524611167e-06 -▁Luf 3.6975006524611167e-06 -▁engineer 3.6975006524611167e-06 -▁marvell 3.6975006524611167e-06 -▁nin 3.6975006524611167e-06 -▁vorübergehend 3.6975006524611167e-06 -.04 3.6937122296614638e-06 -prüfungen 3.6937122296614638e-06 -rehensible 3.6937122296614638e-06 -▁1959 3.6937122296614638e-06 -▁Blues 3.6937122296614638e-06 -▁Butter 3.6937122296614638e-06 -▁Eigentlich 3.6937122296614638e-06 -▁Monster 3.6937122296614638e-06 -▁Neuse 3.6937122296614638e-06 -▁cann 3.6937122296614638e-06 -▁dramatischen 3.6937122296614638e-06 -▁einleiten 3.6937122296614638e-06 -▁intellekt 3.6937122296614638e-06 -▁murdered 3.6937122296614638e-06 -▁natürlicher 3.6937122296614638e-06 -▁revital 3.6937122296614638e-06 -▁vorherrschen 3.6937122296614638e-06 -comes 3.6899238068618109e-06 -reden 3.6899238068618109e-06 -stru 3.6899238068618109e-06 -teien 3.6899238068618109e-06 -▁Basque 3.6899238068618109e-06 -▁Beamte 3.6899238068618109e-06 -▁Cairo 3.6899238068618109e-06 -▁Focus 3.6899238068618109e-06 -▁Ricc 3.6899238068618109e-06 -▁Schande 3.6899238068618109e-06 -▁Sozialfonds 3.6899238068618109e-06 -▁Stadtr 3.6899238068618109e-06 -▁Stone 3.6899238068618109e-06 -▁Strahl 3.6899238068618109e-06 -▁anerkennt 3.6899238068618109e-06 -▁chambre 3.6899238068618109e-06 -▁ersucht 3.6899238068618109e-06 -▁hect 3.6899238068618109e-06 -▁outlook 3.6899238068618109e-06 --Problem 3.6861353840621584e-06 -IFF 3.6861353840621584e-06 -ameworks 3.6861353840621584e-06 -ementioned 3.6861353840621584e-06 -ongs 3.6861353840621584e-06 -umfeld 3.6861353840621584e-06 -ungsmittel 3.6861353840621584e-06 -unte 3.6861353840621584e-06 -▁320 3.6861353840621584e-06 -▁Easter 3.6861353840621584e-06 -▁Fell 3.6861353840621584e-06 -▁Footprint 3.6861353840621584e-06 -▁KUKA 3.6861353840621584e-06 -▁Kleink 3.6861353840621584e-06 -▁Kyoto-Protokoll 3.6861353840621584e-06 -▁Nordamerika 3.6861353840621584e-06 -▁PAD 3.6861353840621584e-06 -▁disgra 3.6861353840621584e-06 -▁durchschnitt 3.6861353840621584e-06 -▁eingebaut 3.6861353840621584e-06 -▁geschaff 3.6861353840621584e-06 -▁gespeicher 3.6861353840621584e-06 -▁loyal 3.6861353840621584e-06 -▁multiannual 3.6861353840621584e-06 -▁verschlechtert 3.6861353840621584e-06 -▁warmth 3.6861353840621584e-06 -aches 3.6823469612625055e-06 -annung 3.6823469612625055e-06 -ategorie 3.6823469612625055e-06 -beiträge 3.6823469612625055e-06 -eads 3.6823469612625055e-06 -enda 3.6823469612625055e-06 -ermächtigungen 3.6823469612625055e-06 -hindert 3.6823469612625055e-06 -iations 3.6823469612625055e-06 -itsch 3.6823469612625055e-06 -wicklung 3.6823469612625055e-06 -▁Bündnis 3.6823469612625055e-06 -▁Football 3.6823469612625055e-06 -▁Former 3.6823469612625055e-06 -▁Knopf 3.6823469612625055e-06 -▁Table 3.6823469612625055e-06 -▁Verantwortlichkeit 3.6823469612625055e-06 -▁Wettbewerbspolitik 3.6823469612625055e-06 -▁eingehend 3.6823469612625055e-06 -▁erstklassigen 3.6823469612625055e-06 -▁exagger 3.6823469612625055e-06 -▁jam 3.6823469612625055e-06 -▁verpflicht 3.6823469612625055e-06 -▁verstößt 3.6823469612625055e-06 -▁verurs 3.6823469612625055e-06 -▁Überzeug 3.6823469612625055e-06 -marke 3.6785585384628527e-06 -ossiers 3.6785585384628527e-06 -uption 3.6785585384628527e-06 -▁Aids 3.6785585384628527e-06 -▁Favor 3.6785585384628527e-06 -▁Guad 3.6785585384628527e-06 -▁Hoheitsgebiet 3.6785585384628527e-06 -▁Late 3.6785585384628527e-06 -▁Nephi 3.6785585384628527e-06 -▁Notfall 3.6785585384628527e-06 -▁Obergrenze 3.6785585384628527e-06 -▁Paulo 3.6785585384628527e-06 -▁artw 3.6785585384628527e-06 -▁beliebig 3.6785585384628527e-06 -▁cloud 3.6785585384628527e-06 -▁comitology 3.6785585384628527e-06 -▁commodities 3.6785585384628527e-06 -▁continents 3.6785585384628527e-06 -▁desires 3.6785585384628527e-06 -▁drama 3.6785585384628527e-06 -▁inventory 3.6785585384628527e-06 -▁marina 3.6785585384628527e-06 -▁motors 3.6785585384628527e-06 -▁pupils 3.6785585384628527e-06 -▁stell 3.6785585384628527e-06 -▁vergrößern 3.6785585384628527e-06 -,50 3.6747701156631998e-06 -risiken 3.6747701156631998e-06 -sender 3.6747701156631998e-06 -ungspolitik 3.6747701156631998e-06 -wandeln 3.6747701156631998e-06 -ĺ 3.6747701156631998e-06 -▁Angela 3.6747701156631998e-06 -▁Arist 3.6747701156631998e-06 -▁Banking 3.6747701156631998e-06 -▁Dil 3.6747701156631998e-06 -▁Heathrow 3.6747701156631998e-06 -▁Invasion 3.6747701156631998e-06 -▁Krist 3.6747701156631998e-06 -▁Sah 3.6747701156631998e-06 -▁Vorreit 3.6747701156631998e-06 -▁Wikipedia 3.6747701156631998e-06 -▁abortion 3.6747701156631998e-06 -▁administrativen 3.6747701156631998e-06 -▁ausdrückliche 3.6747701156631998e-06 -▁beträchtlich 3.6747701156631998e-06 -▁brachten 3.6747701156631998e-06 -▁own-initiative 3.6747701156631998e-06 -▁relatives 3.6747701156631998e-06 -▁remembered 3.6747701156631998e-06 -▁rings 3.6747701156631998e-06 -▁templates 3.6747701156631998e-06 -▁zuteil 3.6747701156631998e-06 -befugnisse 3.6709816928635473e-06 -isedatum 3.6709816928635473e-06 -itherto 3.6709816928635473e-06 -wuchs 3.6709816928635473e-06 -zukommen 3.6709816928635473e-06 -▁Already 3.6709816928635473e-06 -▁Bein 3.6709816928635473e-06 -▁Entsprechend 3.6709816928635473e-06 -▁Freundlichkeit 3.6709816928635473e-06 -▁Jacob 3.6709816928635473e-06 -▁Leinen 3.6709816928635473e-06 -▁Pharm 3.6709816928635473e-06 -▁Schaltfläche 3.6709816928635473e-06 -▁Schild 3.6709816928635473e-06 -▁Ubuntu 3.6709816928635473e-06 -▁dent 3.6709816928635473e-06 -▁excav 3.6709816928635473e-06 -▁ther 3.6709816928635473e-06 -eaker 3.6671932700638944e-06 -einkommens 3.6671932700638944e-06 -software 3.6671932700638944e-06 -uropä 3.6671932700638944e-06 -ussia 3.6671932700638944e-06 -être 3.6671932700638944e-06 -öhnt 3.6671932700638944e-06 -▁Imaging 3.6671932700638944e-06 -▁Municip 3.6671932700638944e-06 -▁Pausch 3.6671932700638944e-06 -▁Reed 3.6671932700638944e-06 -▁Rehn 3.6671932700638944e-06 -▁atemberaubenden 3.6671932700638944e-06 -▁blauen 3.6671932700638944e-06 -▁conditional 3.6671932700638944e-06 -▁controll 3.6671932700638944e-06 -▁indicator 3.6671932700638944e-06 -▁leer 3.6671932700638944e-06 -▁soziales 3.6671932700638944e-06 -▁touched 3.6671932700638944e-06 -▁verkörp 3.6671932700638944e-06 -▁volcano 3.6671932700638944e-06 -amerikas 3.6634048472642415e-06 -ennetz 3.6634048472642415e-06 -ketten 3.6634048472642415e-06 -tritte 3.6634048472642415e-06 -▁Ander 3.6634048472642415e-06 -▁Auftritt 3.6634048472642415e-06 -▁Bäume 3.6634048472642415e-06 -▁Geister 3.6634048472642415e-06 -▁Karlsruhe 3.6634048472642415e-06 -▁Login 3.6634048472642415e-06 -▁Menu 3.6634048472642415e-06 -▁NV 3.6634048472642415e-06 -▁PER 3.6634048472642415e-06 -▁Rue 3.6634048472642415e-06 -▁Vorgehens 3.6634048472642415e-06 -▁angrenzen 3.6634048472642415e-06 -▁bequemen 3.6634048472642415e-06 -▁einl 3.6634048472642415e-06 -▁inhaft 3.6634048472642415e-06 -▁kathol 3.6634048472642415e-06 -▁minors 3.6634048472642415e-06 -▁redundant 3.6634048472642415e-06 -▁steady 3.6634048472642415e-06 -▁widersprüch 3.6634048472642415e-06 -inationen 3.659616424464589e-06 -▁Brasil 3.659616424464589e-06 -▁Pion 3.659616424464589e-06 -▁accomplish 3.659616424464589e-06 -▁brothers 3.659616424464589e-06 -▁claiming 3.659616424464589e-06 -▁festgeschrieben 3.659616424464589e-06 -▁investigated 3.659616424464589e-06 -▁kurzfristige 3.659616424464589e-06 -▁num 3.659616424464589e-06 -▁slave 3.659616424464589e-06 -▁thickness 3.659616424464589e-06 -▁virtue 3.659616424464589e-06 -▁zurückgreifen 3.659616424464589e-06 -ULT 3.6558280016649362e-06 -nisch 3.6558280016649362e-06 -papiere 3.6558280016649362e-06 -▁Anc 3.6558280016649362e-06 -▁Chapter 3.6558280016649362e-06 -▁Christopher 3.6558280016649362e-06 -▁Geis 3.6558280016649362e-06 -▁Goethe 3.6558280016649362e-06 -▁anged 3.6558280016649362e-06 -▁bisherige 3.6558280016649362e-06 -▁deprived 3.6558280016649362e-06 -▁enjoyment 3.6558280016649362e-06 -▁höchstens 3.6558280016649362e-06 -▁mentioning 3.6558280016649362e-06 -▁oberhalb 3.6558280016649362e-06 -▁predecessor 3.6558280016649362e-06 -▁sacrifice 3.6558280016649362e-06 -▁salaries 3.6558280016649362e-06 -▁spezielles 3.6558280016649362e-06 -▁zögern 3.6558280016649362e-06 -.2008 3.6520395788652833e-06 -geschlossen 3.6520395788652833e-06 -handeln 3.6520395788652833e-06 -krieges 3.6520395788652833e-06 -ouss 3.6520395788652833e-06 -pone 3.6520395788652833e-06 -raucher 3.6520395788652833e-06 -▁Buchten 3.6520395788652833e-06 -▁Gene 3.6520395788652833e-06 -▁Gleichberechtigung 3.6520395788652833e-06 -▁Mitgliedsländern 3.6520395788652833e-06 -▁Verdacht 3.6520395788652833e-06 -▁YORK 3.6520395788652833e-06 -▁cross-country 3.6520395788652833e-06 -▁gemeinnütz 3.6520395788652833e-06 -▁inhaltlich 3.6520395788652833e-06 -▁interfere 3.6520395788652833e-06 -▁sektor 3.6520395788652833e-06 -▁stup 3.6520395788652833e-06 -▁töten 3.6520395788652833e-06 -▁weite 3.6520395788652833e-06 -arbeitete 3.6482511560656304e-06 -being 3.6482511560656304e-06 -orca 3.6482511560656304e-06 -zugleichen 3.6482511560656304e-06 -▁145 3.6482511560656304e-06 -▁Abwesenheit 3.6482511560656304e-06 -▁Belange 3.6482511560656304e-06 -▁Haartrockner 3.6482511560656304e-06 -▁Kev 3.6482511560656304e-06 -▁Minist 3.6482511560656304e-06 -▁Relais 3.6482511560656304e-06 -▁Spin 3.6482511560656304e-06 -▁cofin 3.6482511560656304e-06 -▁entschloss 3.6482511560656304e-06 -▁vorzustellen 3.6482511560656304e-06 -protokoll 3.6444627332659779e-06 -öhnen 3.6444627332659779e-06 -▁Bavarian 3.6444627332659779e-06 -▁Boulevard 3.6444627332659779e-06 -▁Diskrimin 3.6444627332659779e-06 -▁Europa-Mittelmeer 3.6444627332659779e-06 -▁Gemeinschaftspolitik 3.6444627332659779e-06 -▁Mw 3.6444627332659779e-06 -▁Nine 3.6444627332659779e-06 -▁Rechtsanw 3.6444627332659779e-06 -▁Slovenian 3.6444627332659779e-06 -▁administrations 3.6444627332659779e-06 -▁getrennte 3.6444627332659779e-06 -▁infected 3.6444627332659779e-06 -▁influen 3.6444627332659779e-06 -▁oppon 3.6444627332659779e-06 -▁unverzichtbar 3.6444627332659779e-06 -▁vermin 3.6444627332659779e-06 -▁verste 3.6444627332659779e-06 -apie 3.640674310466325e-06 -ausrüstung 3.640674310466325e-06 -codes 3.640674310466325e-06 -inta 3.640674310466325e-06 -lash 3.640674310466325e-06 -lernen 3.640674310466325e-06 -most 3.640674310466325e-06 -rise 3.640674310466325e-06 -äfen 3.640674310466325e-06 -älern 3.640674310466325e-06 -▁Beyond 3.640674310466325e-06 -▁Gesundheits- 3.640674310466325e-06 -▁Gom 3.640674310466325e-06 -▁Klein- 3.640674310466325e-06 -▁Mao 3.640674310466325e-06 -▁Neur 3.640674310466325e-06 -▁Romp 3.640674310466325e-06 -▁Schmidt 3.640674310466325e-06 -▁Tam 3.640674310466325e-06 -▁antibiot 3.640674310466325e-06 -▁geschlagen 3.640674310466325e-06 -▁hochwertiger 3.640674310466325e-06 -▁logos 3.640674310466325e-06 -▁plastics 3.640674310466325e-06 -▁rote 3.640674310466325e-06 -▁third-party 3.640674310466325e-06 -utable 3.6368858876666721e-06 -verlauf 3.6368858876666721e-06 -▁Maritime 3.6368858876666721e-06 -▁Players 3.6368858876666721e-06 -▁Romagna 3.6368858876666721e-06 -▁Sensor 3.6368858876666721e-06 -▁Works 3.6368858876666721e-06 -▁findest 3.6368858876666721e-06 -▁gorge 3.6368858876666721e-06 -▁high-tech 3.6368858876666721e-06 -▁kompl 3.6368858876666721e-06 -▁peripher 3.6368858876666721e-06 -▁supposedly 3.6368858876666721e-06 -IMM 3.6330974648670192e-06 -alling 3.6330974648670192e-06 -etischen 3.6330974648670192e-06 -heizung 3.6330974648670192e-06 -iesst 3.6330974648670192e-06 -inia 3.6330974648670192e-06 -istans 3.6330974648670192e-06 -länge 3.6330974648670192e-06 -olet 3.6330974648670192e-06 -schafft 3.6330974648670192e-06 -ähere 3.6330974648670192e-06 -▁1947 3.6330974648670192e-06 -▁Bestehen 3.6330974648670192e-06 -▁Cafe 3.6330974648670192e-06 -▁Charl 3.6330974648670192e-06 -▁Indones 3.6330974648670192e-06 -▁Ivan 3.6330974648670192e-06 -▁Kenya 3.6330974648670192e-06 -▁Speicherung 3.6330974648670192e-06 -▁Warte 3.6330974648670192e-06 -▁attributes 3.6330974648670192e-06 -▁aufbew 3.6330974648670192e-06 -▁circles 3.6330974648670192e-06 -▁fifty 3.6330974648670192e-06 -▁nicely 3.6330974648670192e-06 -▁norms 3.6330974648670192e-06 -▁offences 3.6330974648670192e-06 -▁stören 3.6330974648670192e-06 -▁zukommt 3.6330974648670192e-06 -CL 3.6293090420673668e-06 -PD 3.6293090420673668e-06 -eitern 3.6293090420673668e-06 -liebhaber 3.6293090420673668e-06 -udad 3.6293090420673668e-06 -▁Blanc 3.6293090420673668e-06 -▁Futtermittel 3.6293090420673668e-06 -▁Geräus 3.6293090420673668e-06 -▁bund 3.6293090420673668e-06 -▁draftsman 3.6293090420673668e-06 -▁repressive 3.6293090420673668e-06 -▁reproductive 3.6293090420673668e-06 -▁vorgelegte 3.6293090420673668e-06 -▁ward 3.6293090420673668e-06 -ahara 3.6255206192677139e-06 -bedien 3.6255206192677139e-06 -geworfen 3.6255206192677139e-06 -ichtlinie 3.6255206192677139e-06 -osity 3.6255206192677139e-06 -willigen 3.6255206192677139e-06 -windigkeiten 3.6255206192677139e-06 -äufig 3.6255206192677139e-06 -ößen 3.6255206192677139e-06 -▁Bibione 3.6255206192677139e-06 -▁Delors 3.6255206192677139e-06 -▁Etwas 3.6255206192677139e-06 -▁Hemm 3.6255206192677139e-06 -▁Italienisch 3.6255206192677139e-06 -▁Toulouse 3.6255206192677139e-06 -▁Urheberrecht 3.6255206192677139e-06 -▁Vorzüge 3.6255206192677139e-06 -▁betrug 3.6255206192677139e-06 -▁comprom 3.6255206192677139e-06 -▁crown 3.6255206192677139e-06 -▁jüdischen 3.6255206192677139e-06 -▁musst 3.6255206192677139e-06 -▁protocols 3.6255206192677139e-06 -▁wöchentlich 3.6255206192677139e-06 -▁zivile 3.6255206192677139e-06 -abgabe 3.621732196468061e-06 -gästen 3.621732196468061e-06 -ildern 3.621732196468061e-06 -▁Bahrain 3.621732196468061e-06 -▁Check-Out 3.621732196468061e-06 -▁Computern 3.621732196468061e-06 -▁Dienstleister 3.621732196468061e-06 -▁Gab 3.621732196468061e-06 -▁Getränken 3.621732196468061e-06 -▁Pressefreiheit 3.621732196468061e-06 -▁Ruinen 3.621732196468061e-06 -▁WIT 3.621732196468061e-06 -▁Wiedergabe 3.621732196468061e-06 -▁Wortes 3.621732196468061e-06 -▁ausl 3.621732196468061e-06 -▁county 3.621732196468061e-06 -▁dancing 3.621732196468061e-06 -▁erneuert 3.621732196468061e-06 -▁expressions 3.621732196468061e-06 -▁filed 3.621732196468061e-06 -▁großzügige 3.621732196468061e-06 -▁heiligen 3.621732196468061e-06 -▁promin 3.621732196468061e-06 -▁vul 3.621732196468061e-06 -▁züg 3.621732196468061e-06 --Sterne-Hotel 3.6179437736684081e-06 -71 3.6179437736684081e-06 -800 3.6179437736684081e-06 -anzlei 3.6179437736684081e-06 -effizient 3.6179437736684081e-06 -etung 3.6179437736684081e-06 -haltestelle 3.6179437736684081e-06 -iktor 3.6179437736684081e-06 -vorsitzenden 3.6179437736684081e-06 -ymp 3.6179437736684081e-06 -▁Ei 3.6179437736684081e-06 -▁Freihandelsabkommen 3.6179437736684081e-06 -▁Installer 3.6179437736684081e-06 -▁NY 3.6179437736684081e-06 -▁Repl 3.6179437736684081e-06 -▁SLR 3.6179437736684081e-06 -▁Saar 3.6179437736684081e-06 -▁Zeitschriften 3.6179437736684081e-06 -▁abgew 3.6179437736684081e-06 -▁benötigte 3.6179437736684081e-06 -▁bets 3.6179437736684081e-06 -▁combinations 3.6179437736684081e-06 -▁dying 3.6179437736684081e-06 -▁exzellenten 3.6179437736684081e-06 -▁jury 3.6179437736684081e-06 -▁klim 3.6179437736684081e-06 -▁parad 3.6179437736684081e-06 -▁remit 3.6179437736684081e-06 -▁rows 3.6179437736684081e-06 -▁xenophobia 3.6179437736684081e-06 -▁öfter 3.6179437736684081e-06 -aufgabe 3.6141553508687556e-06 -folios 3.6141553508687556e-06 -gelegen 3.6141553508687556e-06 -ikanern 3.6141553508687556e-06 -legungen 3.6141553508687556e-06 -odus 3.6141553508687556e-06 -ühm 3.6141553508687556e-06 -▁Abänderungs 3.6141553508687556e-06 -▁CET 3.6141553508687556e-06 -▁Daily 3.6141553508687556e-06 -▁ENT 3.6141553508687556e-06 -▁Figuren 3.6141553508687556e-06 -▁Gare 3.6141553508687556e-06 -▁Panz 3.6141553508687556e-06 -▁Syl 3.6141553508687556e-06 -▁Südkorea 3.6141553508687556e-06 -▁Uruguay 3.6141553508687556e-06 -▁attributed 3.6141553508687556e-06 -▁coupled 3.6141553508687556e-06 -▁crowd 3.6141553508687556e-06 -▁ersta 3.6141553508687556e-06 -▁laufend 3.6141553508687556e-06 -▁mile 3.6141553508687556e-06 -▁towers 3.6141553508687556e-06 -▁transferring 3.6141553508687556e-06 -▁verdeutlicht 3.6141553508687556e-06 -▁übergehen 3.6141553508687556e-06 -erries 3.6103669280691027e-06 -pie 3.6103669280691027e-06 -urcht 3.6103669280691027e-06 -▁Ah 3.6103669280691027e-06 -▁Ahmad 3.6103669280691027e-06 -▁Case 3.6103669280691027e-06 -▁DNS 3.6103669280691027e-06 -▁Geltung 3.6103669280691027e-06 -▁Handelspolitik 3.6103669280691027e-06 -▁Kontr 3.6103669280691027e-06 -▁Kris 3.6103669280691027e-06 -▁Krum 3.6103669280691027e-06 -▁Sorten 3.6103669280691027e-06 -▁appealing 3.6103669280691027e-06 -▁ausdrücken 3.6103669280691027e-06 -▁ausmacht 3.6103669280691027e-06 -▁debut 3.6103669280691027e-06 -▁determines 3.6103669280691027e-06 -▁explizit 3.6103669280691027e-06 -▁gewähr 3.6103669280691027e-06 -▁hydraulic 3.6103669280691027e-06 -▁mayor 3.6103669280691027e-06 -▁pixels 3.6103669280691027e-06 -▁remov 3.6103669280691027e-06 -▁sono 3.6103669280691027e-06 -▁triumph 3.6103669280691027e-06 -▁unerm 3.6103669280691027e-06 -OBIL 3.60657850526945e-06 -clubs 3.60657850526945e-06 -eteness 3.60657850526945e-06 -etermination 3.60657850526945e-06 -▁2D 3.60657850526945e-06 -▁Arrang 3.60657850526945e-06 -▁FE 3.60657850526945e-06 -▁Gletscher 3.60657850526945e-06 -▁Highlights 3.60657850526945e-06 -▁Palermo 3.60657850526945e-06 -▁Soul 3.60657850526945e-06 -▁Verlagerung 3.60657850526945e-06 -▁alkohol 3.60657850526945e-06 -▁angewand 3.60657850526945e-06 -▁gesundheitlichen 3.60657850526945e-06 -▁harmonious 3.60657850526945e-06 -▁lucky 3.60657850526945e-06 -▁readily 3.60657850526945e-06 -hän 3.6027900824697969e-06 -immern 3.6027900824697969e-06 -niveaus 3.6027900824697969e-06 -ußts 3.6027900824697969e-06 -weist 3.6027900824697969e-06 -ymbol 3.6027900824697969e-06 -▁Büchern 3.6027900824697969e-06 -▁Fujifilm 3.6027900824697969e-06 -▁Insp 3.6027900824697969e-06 -▁Observ 3.6027900824697969e-06 -▁Ressource 3.6027900824697969e-06 -▁Sofort 3.6027900824697969e-06 -▁auslösen 3.6027900824697969e-06 -▁bewaffneten 3.6027900824697969e-06 -▁breakdown 3.6027900824697969e-06 -▁fleets 3.6027900824697969e-06 -▁jedw 3.6027900824697969e-06 -▁kraft 3.6027900824697969e-06 -▁neoliber 3.6027900824697969e-06 -▁supplying 3.6027900824697969e-06 -▁unite 3.6027900824697969e-06 -eafter 3.5990016596701445e-06 -elie 3.5990016596701445e-06 -oomla 3.5990016596701445e-06 -▁Fuerteventura 3.5990016596701445e-06 -▁Maler 3.5990016596701445e-06 -▁Rost 3.5990016596701445e-06 -▁aufgeb 3.5990016596701445e-06 -▁beklagen 3.5990016596701445e-06 -▁zurückgezogen 3.5990016596701445e-06 -gression 3.5952132368704916e-06 -idencies 3.5952132368704916e-06 -osex 3.5952132368704916e-06 -öße 3.5952132368704916e-06 -▁155 3.5952132368704916e-06 -▁Military 3.5952132368704916e-06 -▁Motivation 3.5952132368704916e-06 -▁Nationalismus 3.5952132368704916e-06 -▁PD 3.5952132368704916e-06 -▁aufgreifen 3.5952132368704916e-06 -▁clever 3.5952132368704916e-06 -▁conception 3.5952132368704916e-06 -▁employer 3.5952132368704916e-06 -▁gerät 3.5952132368704916e-06 -▁grows 3.5952132368704916e-06 -▁loose 3.5952132368704916e-06 -▁schallisol 3.5952132368704916e-06 -▁sob 3.5952132368704916e-06 -▁therap 3.5952132368704916e-06 -▁traditioneller 3.5952132368704916e-06 -▁undermining 3.5952132368704916e-06 -WB 3.5914248140708387e-06 -esserung 3.5914248140708387e-06 -igar 3.5914248140708387e-06 -э 3.5914248140708387e-06 -▁Hamb 3.5914248140708387e-06 -▁String 3.5914248140708387e-06 -▁educated 3.5914248140708387e-06 -▁hinnehmbar 3.5914248140708387e-06 -▁jenes 3.5914248140708387e-06 -▁kön 3.5914248140708387e-06 -▁lectures 3.5914248140708387e-06 -▁mistaken 3.5914248140708387e-06 -allas 3.5876363912711858e-06 -mord 3.5876363912711858e-06 -schlüsselung 3.5876363912711858e-06 -ümm 3.5876363912711858e-06 -▁80er 3.5876363912711858e-06 -▁Baust 3.5876363912711858e-06 -▁Boeing 3.5876363912711858e-06 -▁Dual 3.5876363912711858e-06 -▁No. 3.5876363912711858e-06 -▁Opern 3.5876363912711858e-06 -▁Plans 3.5876363912711858e-06 -▁Rey 3.5876363912711858e-06 -▁Schau 3.5876363912711858e-06 -▁Seien 3.5876363912711858e-06 -▁Wog 3.5876363912711858e-06 -▁Zufl 3.5876363912711858e-06 -▁blut 3.5876363912711858e-06 -▁day-to-day 3.5876363912711858e-06 -▁differing 3.5876363912711858e-06 -▁schwerwiegende 3.5876363912711858e-06 -▁sunshine 3.5876363912711858e-06 -▁tragischen 3.5876363912711858e-06 -▁vernichtet 3.5876363912711858e-06 -▁weiterver 3.5876363912711858e-06 -▁Änderungsvorschläge 3.5876363912711858e-06 -change 3.5838479684715333e-06 -zählung 3.5838479684715333e-06 -▁Bul 3.5838479684715333e-06 -▁Daimler 3.5838479684715333e-06 -▁Fallout 3.5838479684715333e-06 -▁Fru 3.5838479684715333e-06 -▁Frühstücksraum 3.5838479684715333e-06 -▁Gust 3.5838479684715333e-06 -▁Jose 3.5838479684715333e-06 -▁Kambod 3.5838479684715333e-06 -▁TS 3.5838479684715333e-06 -▁Tenden 3.5838479684715333e-06 -▁Wide 3.5838479684715333e-06 -▁bizarre 3.5838479684715333e-06 -▁boutiques 3.5838479684715333e-06 -▁brill 3.5838479684715333e-06 -▁hereby 3.5838479684715333e-06 -▁letting 3.5838479684715333e-06 -▁liqu 3.5838479684715333e-06 -▁restart 3.5838479684715333e-06 -▁vermarkt 3.5838479684715333e-06 -bedürf 3.5800595456718804e-06 -eine 3.5800595456718804e-06 -entscheidung 3.5800595456718804e-06 -offiz 3.5800595456718804e-06 -rankung 3.5800595456718804e-06 -tet 3.5800595456718804e-06 -ädt 3.5800595456718804e-06 -▁Better 3.5800595456718804e-06 -▁IX 3.5800595456718804e-06 -▁Neapel 3.5800595456718804e-06 -▁None 3.5800595456718804e-06 -▁Produktpalette 3.5800595456718804e-06 -▁Ric 3.5800595456718804e-06 -▁dritt 3.5800595456718804e-06 -▁jacuzzi 3.5800595456718804e-06 -▁mü 3.5800595456718804e-06 -▁patented 3.5800595456718804e-06 -▁reiter 3.5800595456718804e-06 -ateral 3.5762711228722275e-06 -▁BugTraq 3.5762711228722275e-06 -▁Confed 3.5762711228722275e-06 -▁Geschlechts 3.5762711228722275e-06 -▁Gesundheitsschutz 3.5762711228722275e-06 -▁Nicar 3.5762711228722275e-06 -▁Proteste 3.5762711228722275e-06 -▁Rauchen 3.5762711228722275e-06 -▁belasten 3.5762711228722275e-06 -▁brochure 3.5762711228722275e-06 -▁conceived 3.5762711228722275e-06 -▁excurs 3.5762711228722275e-06 -▁flank 3.5762711228722275e-06 -arier 3.5724827000725747e-06 -bart 3.5724827000725747e-06 -drück 3.5724827000725747e-06 -mäßiger 3.5724827000725747e-06 -quality 3.5724827000725747e-06 -spät 3.5724827000725747e-06 -ämmung 3.5724827000725747e-06 -▁Establ 3.5724827000725747e-06 -▁IDE 3.5724827000725747e-06 -▁PDA 3.5724827000725747e-06 -▁Pope 3.5724827000725747e-06 -▁Publishing 3.5724827000725747e-06 -▁geltende 3.5724827000725747e-06 -▁gleichwert 3.5724827000725747e-06 -▁klingt 3.5724827000725747e-06 -▁pak 3.5724827000725747e-06 -▁spürbar 3.5724827000725747e-06 -aunch 3.5686942772729222e-06 -ermeld 3.5686942772729222e-06 -iov 3.5686942772729222e-06 -links 3.5686942772729222e-06 -merks 3.5686942772729222e-06 -präsent 3.5686942772729222e-06 -verleih 3.5686942772729222e-06 -werben 3.5686942772729222e-06 -▁Bequem 3.5686942772729222e-06 -▁Feren 3.5686942772729222e-06 -▁Gep 3.5686942772729222e-06 -▁Oost 3.5686942772729222e-06 -▁Recher 3.5686942772729222e-06 -▁Scient 3.5686942772729222e-06 -▁Sonnenunter 3.5686942772729222e-06 -▁Straßenbahn 3.5686942772729222e-06 -▁detaillierten 3.5686942772729222e-06 -▁grenzüberschreit 3.5686942772729222e-06 -▁langwier 3.5686942772729222e-06 -▁nuklearen 3.5686942772729222e-06 -▁regelm 3.5686942772729222e-06 -▁statistischen 3.5686942772729222e-06 -▁verwir 3.5686942772729222e-06 -echen 3.5649058544732693e-06 -nahms 3.5649058544732693e-06 -spektrum 3.5649058544732693e-06 -tended 3.5649058544732693e-06 -unktionen 3.5649058544732693e-06 -œ 3.5649058544732693e-06 -▁ASPs 3.5649058544732693e-06 -▁Illustr 3.5649058544732693e-06 -▁Nahrungsmitteln 3.5649058544732693e-06 -▁Siedlungen 3.5649058544732693e-06 -▁beheiz 3.5649058544732693e-06 -▁frequ 3.5649058544732693e-06 -▁künstlerische 3.5649058544732693e-06 -▁unglück 3.5649058544732693e-06 --Benutzer 3.5611174316736164e-06 -erstattung 3.5611174316736164e-06 -inien 3.5611174316736164e-06 -iode 3.5611174316736164e-06 -reten 3.5611174316736164e-06 -uzenten 3.5611174316736164e-06 -zeugnisse 3.5611174316736164e-06 -▁Entr 3.5611174316736164e-06 -▁Sauberkeit 3.5611174316736164e-06 -▁Yugoslav 3.5611174316736164e-06 -▁anscha 3.5611174316736164e-06 -▁gewalts 3.5611174316736164e-06 -▁jüd 3.5611174316736164e-06 -▁nachts 3.5611174316736164e-06 -▁nachweis 3.5611174316736164e-06 -▁zubereitet 3.5611174316736164e-06 -ength 3.5573290088739635e-06 -ronik 3.5573290088739635e-06 -▁Demonstration 3.5573290088739635e-06 -▁Lenk 3.5573290088739635e-06 -▁Neuerungen 3.5573290088739635e-06 -▁Piano 3.5573290088739635e-06 -▁Religionsfreiheit 3.5573290088739635e-06 -▁Rettung 3.5573290088739635e-06 -▁Tschechien 3.5573290088739635e-06 -▁ambiance 3.5573290088739635e-06 -▁reduz 3.5573290088739635e-06 -▁therapeut 3.5573290088739635e-06 -▁vollzogen 3.5573290088739635e-06 -▁Übern 3.5573290088739635e-06 -euten 3.553540586074311e-06 -ierbarkeit 3.553540586074311e-06 -▁.NET 3.553540586074311e-06 -▁Diss 3.553540586074311e-06 -▁Senegal 3.553540586074311e-06 -▁andernfalls 3.553540586074311e-06 -▁entir 3.553540586074311e-06 -▁gre 3.553540586074311e-06 -▁gängigen 3.553540586074311e-06 -▁threa 3.553540586074311e-06 -anka 3.5497521632746582e-06 -erwart 3.5497521632746582e-06 -itee 3.5497521632746582e-06 -uhen 3.5497521632746582e-06 -ymb 3.5497521632746582e-06 -▁Check-in 3.5497521632746582e-06 -▁Einsatzes 3.5497521632746582e-06 -▁Gremium 3.5497521632746582e-06 -▁Oran 3.5497521632746582e-06 -▁abhalten 3.5497521632746582e-06 -▁anhaltenden 3.5497521632746582e-06 -▁gründliche 3.5497521632746582e-06 -▁jew 3.5497521632746582e-06 -▁negoti 3.5497521632746582e-06 -▁ordentlich 3.5497521632746582e-06 -▁respekt 3.5497521632746582e-06 -ciano 3.5459637404750053e-06 -hebt 3.5459637404750053e-06 -institute 3.5459637404750053e-06 -pfeiler 3.5459637404750053e-06 -ushing 3.5459637404750053e-06 -▁Apartamentos 3.5459637404750053e-06 -▁Gassen 3.5459637404750053e-06 -▁Liver 3.5459637404750053e-06 -▁Notwend 3.5459637404750053e-06 -▁Pfer 3.5459637404750053e-06 -▁brechen 3.5459637404750053e-06 -▁drehen 3.5459637404750053e-06 -▁eingestuft 3.5459637404750053e-06 -▁todo 3.5459637404750053e-06 -▁verab 3.5459637404750053e-06 --Herzeg 3.5421753176753524e-06 -VE 3.5421753176753524e-06 -ophob 3.5421753176753524e-06 -▁Brut 3.5421753176753524e-06 -▁Cards 3.5421753176753524e-06 -▁Duft 3.5421753176753524e-06 -▁Entdeck 3.5421753176753524e-06 -▁Generaldir 3.5421753176753524e-06 -▁Maz 3.5421753176753524e-06 -▁Notre 3.5421753176753524e-06 -▁Rac 3.5421753176753524e-06 -▁Rechnungshofs 3.5421753176753524e-06 -▁Umlauf 3.5421753176753524e-06 -▁Zeichnungen 3.5421753176753524e-06 -▁erfreulich 3.5421753176753524e-06 -▁kleinste 3.5421753176753524e-06 -▁venez 3.5421753176753524e-06 -gos 3.5383868948757e-06 -ourmet 3.5383868948757e-06 -ument 3.5383868948757e-06 -▁BGB 3.5383868948757e-06 -▁Command 3.5383868948757e-06 -▁Durchmesser 3.5383868948757e-06 -▁Grenzwerte 3.5383868948757e-06 -▁Hond 3.5383868948757e-06 -▁LKW 3.5383868948757e-06 -▁Massenvern 3.5383868948757e-06 -▁Robot 3.5383868948757e-06 -▁Vernunft 3.5383868948757e-06 -▁erstaun 3.5383868948757e-06 -▁runden 3.5383868948757e-06 -▁vielerlei 3.5383868948757e-06 -BF 3.534598472076047e-06 -erben 3.534598472076047e-06 -hein 3.534598472076047e-06 -orie 3.534598472076047e-06 -pret 3.534598472076047e-06 -schland 3.534598472076047e-06 -▁1955 3.534598472076047e-06 -▁Akteur 3.534598472076047e-06 -▁Antibiot 3.534598472076047e-06 -▁Emotionen 3.534598472076047e-06 -▁Frontera 3.534598472076047e-06 -▁Haushaltsausschuss 3.534598472076047e-06 -▁Herstellungs 3.534598472076047e-06 -▁Marse 3.534598472076047e-06 -▁Niederlage 3.534598472076047e-06 -▁Penn 3.534598472076047e-06 -▁erhol 3.534598472076047e-06 -▁frühzeitig 3.534598472076047e-06 -▁gramm 3.534598472076047e-06 -ollars 3.5308100492763941e-06 -tum 3.5308100492763941e-06 -▁Anforderung 3.5308100492763941e-06 -▁Bewert 3.5308100492763941e-06 -▁Firewall 3.5308100492763941e-06 -▁Legitim 3.5308100492763941e-06 -▁SEC 3.5308100492763941e-06 -▁hervorragender 3.5308100492763941e-06 -▁schock 3.5308100492763941e-06 -▁studieren 3.5308100492763941e-06 -▁zauber 3.5308100492763941e-06 -mach 3.5270216264767412e-06 -queren 3.5270216264767412e-06 -▁Besonderheit 3.5270216264767412e-06 -▁Japans 3.5270216264767412e-06 -▁Kosmetik 3.5270216264767412e-06 -▁Leist 3.5270216264767412e-06 -▁Vancouver 3.5270216264767412e-06 -▁engere 3.5270216264767412e-06 -▁kurzfristigen 3.5270216264767412e-06 -▁Česk 3.5270216264767412e-06 -ategor 3.5232332036770888e-06 -▁256 3.5232332036770888e-06 -▁Antrags 3.5232332036770888e-06 -▁Chalet 3.5232332036770888e-06 -▁Electronics 3.5232332036770888e-06 -▁Geschäftsbereich 3.5232332036770888e-06 -▁Gewichts 3.5232332036770888e-06 -▁Math 3.5232332036770888e-06 -▁demzufolge 3.5232332036770888e-06 -▁möbl 3.5232332036770888e-06 -▁necess 3.5232332036770888e-06 -▁telefon 3.5232332036770888e-06 -andschaft 3.5194447808774359e-06 -sparen 3.5194447808774359e-06 -ún 3.5194447808774359e-06 -▁Ambitionen 3.5194447808774359e-06 -▁Byrne 3.5194447808774359e-06 -▁Entschließungs 3.5194447808774359e-06 -▁Fortschritten 3.5194447808774359e-06 -▁exportiert 3.5194447808774359e-06 -▁fühl 3.5194447808774359e-06 -sprüche 3.515656358077783e-06 -▁Angreif 3.515656358077783e-06 -▁Briten 3.515656358077783e-06 -▁Einträge 3.515656358077783e-06 -▁Francis 3.515656358077783e-06 -▁Gul 3.515656358077783e-06 -▁Kontrast 3.515656358077783e-06 -▁Krat 3.515656358077783e-06 -▁Marathon 3.515656358077783e-06 -▁Schach 3.515656358077783e-06 -▁Umwandlung 3.515656358077783e-06 -▁dipl 3.515656358077783e-06 -▁englischer 3.515656358077783e-06 -▁mg 3.515656358077783e-06 -▁neuem 3.515656358077783e-06 -▁obersten 3.515656358077783e-06 -▁raschen 3.515656358077783e-06 -▁verme 3.515656358077783e-06 -lieferungen 3.51186793527813e-06 -vant 3.51186793527813e-06 -▁Boote 3.51186793527813e-06 -▁Bun 3.51186793527813e-06 -▁Erb 3.51186793527813e-06 -▁Glasg 3.51186793527813e-06 -▁PNR 3.51186793527813e-06 -▁explan 3.51186793527813e-06 -▁funktionierenden 3.51186793527813e-06 -▁prep 3.51186793527813e-06 -▁punkt 3.51186793527813e-06 -▁wissens 3.51186793527813e-06 -conn 3.5080795124784776e-06 -egation 3.5080795124784776e-06 -stags 3.5080795124784776e-06 -▁Gior 3.5080795124784776e-06 -▁Hed 3.5080795124784776e-06 -▁Höhlen 3.5080795124784776e-06 -▁Internetanschluss 3.5080795124784776e-06 -▁Menschenleben 3.5080795124784776e-06 -▁Summen 3.5080795124784776e-06 -▁identisch 3.5080795124784776e-06 -▁subst 3.5080795124784776e-06 -▁tagt 3.5080795124784776e-06 -▁widerspricht 3.5080795124784776e-06 -▁zufäl 3.5080795124784776e-06 --Haus 3.5042910896788247e-06 -berühm 3.5042910896788247e-06 -iesta 3.5042910896788247e-06 -kanzler 3.5042910896788247e-06 -▁2-3 3.5042910896788247e-06 -▁Bordeaux 3.5042910896788247e-06 -▁Chávez 3.5042910896788247e-06 -▁Franç 3.5042910896788247e-06 -▁Kinderspiel 3.5042910896788247e-06 -▁MHz 3.5042910896788247e-06 -▁Myst 3.5042910896788247e-06 -▁PV 3.5042910896788247e-06 -▁Roger 3.5042910896788247e-06 -▁US-amerikanischen 3.5042910896788247e-06 -▁aktual 3.5042910896788247e-06 -▁darzustellen 3.5042910896788247e-06 -▁debattieren 3.5042910896788247e-06 -▁inneh 3.5042910896788247e-06 -▁stehe 3.5042910896788247e-06 -itate 3.5005026668791718e-06 -lasten 3.5005026668791718e-06 -ruhe 3.5005026668791718e-06 -▁11.30 3.5005026668791718e-06 -▁Einst 3.5005026668791718e-06 -▁Erwähnung 3.5005026668791718e-06 -▁Ethik 3.5005026668791718e-06 -▁Freed 3.5005026668791718e-06 -▁Gesetzgeb 3.5005026668791718e-06 -▁González 3.5005026668791718e-06 -▁Heutzutage 3.5005026668791718e-06 -▁Ingenieur 3.5005026668791718e-06 -▁Seel 3.5005026668791718e-06 -▁Staatsangehör 3.5005026668791718e-06 -▁Tuber 3.5005026668791718e-06 -▁schwachen 3.5005026668791718e-06 -gnung 3.4967142440795189e-06 -▁1929 3.4967142440795189e-06 -▁Bedürf 3.4967142440795189e-06 -▁Dringlichkeits 3.4967142440795189e-06 -▁Edelstahl 3.4967142440795189e-06 -▁Kriegsverb 3.4967142440795189e-06 -▁colle 3.4967142440795189e-06 -▁gerechten 3.4967142440795189e-06 -▁restruct 3.4967142440795189e-06 -▁sympt 3.4967142440795189e-06 -▁transparenten 3.4967142440795189e-06 -etreten 3.4929258212798665e-06 -ördert 3.4929258212798665e-06 -ů 3.4929258212798665e-06 -▁Fuerte 3.4929258212798665e-06 -▁Gewissen 3.4929258212798665e-06 -▁Gütern 3.4929258212798665e-06 -▁Träume 3.4929258212798665e-06 -▁nationales 3.4929258212798665e-06 -▁polic 3.4929258212798665e-06 -▁sog. 3.4929258212798665e-06 -▁strategisch 3.4929258212798665e-06 -▁unterm 3.4929258212798665e-06 -▁urheber 3.4929258212798665e-06 -▁verdeut 3.4929258212798665e-06 -▁vervollständ 3.4929258212798665e-06 -▁vorantreiben 3.4929258212798665e-06 -erkrankungen 3.4891373984802136e-06 -ς 3.4891373984802136e-06 -▁Aktionäre 3.4891373984802136e-06 -▁Erleb 3.4891373984802136e-06 -▁Stammzellen 3.4891373984802136e-06 -▁blockieren 3.4891373984802136e-06 -▁erfahrene 3.4891373984802136e-06 -▁fürs 3.4891373984802136e-06 -▁verabschiedeten 3.4891373984802136e-06 -▁weiße 3.4891373984802136e-06 -▁Überraschung 3.4891373984802136e-06 -loß 3.4853489756805607e-06 -▁Halbjahr 3.4853489756805607e-06 -▁MyS 3.4853489756805607e-06 -▁Paare 3.4853489756805607e-06 -▁komplizierte 3.4853489756805607e-06 -cksilber 3.4815605528809082e-06 -▁Behinderten 3.4815605528809082e-06 -▁Kompat 3.4815605528809082e-06 -▁Kopien 3.4815605528809082e-06 -▁Tschech 3.4815605528809082e-06 -▁anstehenden 3.4815605528809082e-06 -▁ausgebildet 3.4815605528809082e-06 -▁dispos 3.4815605528809082e-06 -▁unvergessliche 3.4815605528809082e-06 -▁überflüssig 3.4815605528809082e-06 -gült 3.4777721300812553e-06 -republik 3.4777721300812553e-06 -▁Eingriff 3.4777721300812553e-06 -▁Immobilie 3.4777721300812553e-06 -▁Krit 3.4777721300812553e-06 -▁Oct 3.4777721300812553e-06 -▁Offenbar 3.4777721300812553e-06 -▁Plattformen 3.4777721300812553e-06 -▁Schröder 3.4777721300812553e-06 -▁Schwäch 3.4777721300812553e-06 -▁ausfallen 3.4777721300812553e-06 -▁dementsprechend 3.4777721300812553e-06 -▁großzügigen 3.4777721300812553e-06 -anschluss 3.4739837072816024e-06 -nehmern 3.4739837072816024e-06 -regulierung 3.4739837072816024e-06 -▁Kartoff 3.4739837072816024e-06 -▁Kritiker 3.4739837072816024e-06 -▁Malt 3.4739837072816024e-06 -▁Rehab 3.4739837072816024e-06 -▁admin 3.4739837072816024e-06 -▁besetzten 3.4739837072816024e-06 -▁besprechen 3.4739837072816024e-06 -▁verbleibenden 3.4739837072816024e-06 -attungen 3.4701952844819495e-06 -▁Ausgehend 3.4701952844819495e-06 -▁Befürworter 3.4701952844819495e-06 -▁Schreibt 3.4701952844819495e-06 -▁einzul 3.4701952844819495e-06 -▁fantastische 3.4701952844819495e-06 -▁geistige 3.4701952844819495e-06 -▁lebensl 3.4701952844819495e-06 -azin 3.4664068616822971e-06 -▁Kämpfe 3.4664068616822971e-06 -▁Lese 3.4664068616822971e-06 -▁Mandats 3.4664068616822971e-06 -▁entstandenen 3.4664068616822971e-06 -▁erschwing 3.4664068616822971e-06 -▁glaubwürdig 3.4664068616822971e-06 -▁inject 3.4664068616822971e-06 -fet 3.4626184388826442e-06 -ôte 3.4626184388826442e-06 -▁Beitrittsländer 3.4626184388826442e-06 -▁Berufung 3.4626184388826442e-06 -▁Freig 3.4626184388826442e-06 -▁Komment 3.4626184388826442e-06 -▁Phänom 3.4626184388826442e-06 -▁ausreichenden 3.4626184388826442e-06 -▁fantastischen 3.4626184388826442e-06 -▁tibet 3.4626184388826442e-06 -▁vereinfach 3.4626184388826442e-06 -rist 3.4588300160829913e-06 -ücksicht 3.4588300160829913e-06 -▁Bäder 3.4588300160829913e-06 -▁Daran 3.4588300160829913e-06 -▁Fahrplan 3.4588300160829913e-06 -▁Glanz 3.4588300160829913e-06 -▁Kommand 3.4588300160829913e-06 -▁Londoner 3.4588300160829913e-06 -▁Marktführer 3.4588300160829913e-06 -▁Neust 3.4588300160829913e-06 -▁Zivilbevölkerung 3.4588300160829913e-06 -▁bedacht 3.4588300160829913e-06 -▁begünstigt 3.4588300160829913e-06 -▁doppelten 3.4588300160829913e-06 -▁mancher 3.4588300160829913e-06 -▁veh 3.4588300160829913e-06 -▁zwingt 3.4588300160829913e-06 -LAF 3.4550415932833384e-06 -▁Inhaft 3.4550415932833384e-06 -▁einzub 3.4550415932833384e-06 -▁lauten 3.4550415932833384e-06 -▁niem 3.4550415932833384e-06 -▁ultim 3.4550415932833384e-06 -ecut 3.4512531704836859e-06 -ordan 3.4512531704836859e-06 -▁Anleihen 3.4512531704836859e-06 -▁Bügelservice 3.4512531704836859e-06 -▁Fremdenfeindlichkeit 3.4512531704836859e-06 -▁Haushaltslinie 3.4512531704836859e-06 -▁Konzerne 3.4512531704836859e-06 -▁Starten 3.4512531704836859e-06 -▁Stornierungen 3.4512531704836859e-06 -▁impos 3.4512531704836859e-06 -▁vorankommen 3.4512531704836859e-06 -▁weitergeben 3.4512531704836859e-06 --old 3.447464747684033e-06 -vorsorge 3.447464747684033e-06 -zeitig 3.447464747684033e-06 -▁Fassade 3.447464747684033e-06 -▁Föder 3.447464747684033e-06 -▁Hügeln 3.447464747684033e-06 -▁Laptop 3.447464747684033e-06 -▁Stabilitäts 3.447464747684033e-06 -▁Stipend 3.447464747684033e-06 -▁betrachte 3.447464747684033e-06 -▁unangen 3.447464747684033e-06 -▁unbegrenz 3.447464747684033e-06 -▁IP-Adresse 3.44367632488438e-06 -▁Jubiläum 3.44367632488438e-06 -▁Meldungen 3.44367632488438e-06 -▁Präzedenz 3.44367632488438e-06 -▁Unterbrechung 3.44367632488438e-06 -▁Wissenschaftlern 3.44367632488438e-06 -▁erlangt 3.44367632488438e-06 -▁konfiguriert 3.44367632488438e-06 -▁unklar 3.44367632488438e-06 -▁verhält 3.44367632488438e-06 -▁wieviel 3.44367632488438e-06 -alin 3.4398879020847273e-06 -▁Allerg 3.4398879020847273e-06 -▁Furcht 3.4398879020847273e-06 -▁Hygiene 3.4398879020847273e-06 -▁Mitgliedstaats 3.4398879020847273e-06 -▁Vorbereit 3.4398879020847273e-06 -▁cosmet 3.4398879020847273e-06 -▁dür 3.4398879020847273e-06 -▁unterrichten 3.4398879020847273e-06 -just 3.4360994792850748e-06 -örter 3.4360994792850748e-06 -▁Erwart 3.4360994792850748e-06 -▁Expansion 3.4360994792850748e-06 -▁Hervor 3.4360994792850748e-06 -▁Normalerweise 3.4360994792850748e-06 -▁beispiell 3.4360994792850748e-06 -▁besonderem 3.4360994792850748e-06 -▁dreht 3.4360994792850748e-06 -▁eingebrachten 3.4360994792850748e-06 -▁erstaunlich 3.4360994792850748e-06 -▁geringere 3.4360994792850748e-06 -▁Ökonomie 3.4360994792850748e-06 -fangs 3.4323110564854219e-06 -▁Dossier 3.4323110564854219e-06 -▁Flügel 3.4323110564854219e-06 -▁Gebiets 3.4323110564854219e-06 -▁Rahmenrichtlinie 3.4323110564854219e-06 -▁bewährte 3.4323110564854219e-06 -▁objektiv 3.4323110564854219e-06 -▁schar 3.4323110564854219e-06 -▁spannende 3.4323110564854219e-06 -▁unbeg 3.4323110564854219e-06 -falt 3.428522633685769e-06 -▁Fischereisektor 3.428522633685769e-06 -▁Fluggäste 3.428522633685769e-06 -▁Naturpark 3.428522633685769e-06 -▁Transpar 3.428522633685769e-06 -▁anderweit 3.428522633685769e-06 -▁bedroh 3.428522633685769e-06 -▁hor 3.428522633685769e-06 -▁worüber 3.428522633685769e-06 -nutzen 3.4247342108861161e-06 -▁Buchungen 3.4247342108861161e-06 -▁Festland 3.4247342108861161e-06 -▁Karibik 3.4247342108861161e-06 -▁Künd 3.4247342108861161e-06 -▁Veröffentlichungen 3.4247342108861161e-06 -▁bürgerlichen 3.4247342108861161e-06 -▁fragte 3.4247342108861161e-06 -▁herausgestellt 3.4247342108861161e-06 -▁höre 3.4247342108861161e-06 -▁postpon 3.4247342108861161e-06 -▁qualifiziert 3.4247342108861161e-06 -▁resc 3.4247342108861161e-06 -▁sportlichen 3.4247342108861161e-06 -nor 3.4209457880864637e-06 -▁Anstrengung 3.4209457880864637e-06 -▁Gewebe 3.4209457880864637e-06 -▁Panoramabl 3.4209457880864637e-06 -▁Papst 3.4209457880864637e-06 -▁Rahmenbeschluss 3.4209457880864637e-06 -fore 3.4171573652868108e-06 -▁Getr 3.4171573652868108e-06 -▁Schattenberichterstatter 3.4171573652868108e-06 -▁dictators 3.4171573652868108e-06 -▁implementiert 3.4171573652868108e-06 -▁kopiert 3.4171573652868108e-06 -▁lebte 3.4171573652868108e-06 -▁versenden 3.4171573652868108e-06 -bridge 3.4133689424871579e-06 -laß 3.4133689424871579e-06 -▁Herangehensweise 3.4133689424871579e-06 -▁Sat-TV 3.4133689424871579e-06 -▁anhaltende 3.4133689424871579e-06 -▁annähern 3.4133689424871579e-06 -▁ansteigen 3.4133689424871579e-06 -▁broadc 3.4133689424871579e-06 -▁vorgebracht 3.4133689424871579e-06 -▁Senden 3.409580519687505e-06 -▁Tunis 3.409580519687505e-06 -▁Verantwortlichkeiten 3.409580519687505e-06 -▁jetz 3.409580519687505e-06 -▁zuverlässigen 3.409580519687505e-06 -allel 3.4057920968878525e-06 -ecutive 3.4057920968878525e-06 -wertet 3.4057920968878525e-06 -ŕ 3.4057920968878525e-06 -▁Fri 3.4057920968878525e-06 -▁Intelligenz 3.4057920968878525e-06 -▁Verhütung 3.4057920968878525e-06 -▁begründen 3.4057920968878525e-06 -chsel 3.4020036740881996e-06 -irc 3.4020036740881996e-06 -reater 3.4020036740881996e-06 -▁Bruttoinland 3.4020036740881996e-06 -▁Kuchen 3.4020036740881996e-06 -▁geforderten 3.4020036740881996e-06 -▁strukturelle 3.4020036740881996e-06 -poration 3.3982152512885467e-06 -▁Fischbestände 3.3982152512885467e-06 -▁Handtücher 3.3982152512885467e-06 -▁Mütter 3.3982152512885467e-06 -▁Schweigen 3.3982152512885467e-06 -▁gültige 3.3982152512885467e-06 -▁spezieller 3.3982152512885467e-06 -beschreibung 3.3944268284888938e-06 -body 3.3944268284888938e-06 -▁Bean 3.3944268284888938e-06 -▁Befried 3.3944268284888938e-06 -▁Chin 3.3944268284888938e-06 -▁Kroat 3.3944268284888938e-06 -▁Muttersprache 3.3944268284888938e-06 -▁Tot 3.3944268284888938e-06 -▁abzubauen 3.3944268284888938e-06 -▁angestrebt 3.3944268284888938e-06 -▁conn 3.3944268284888938e-06 -▁distribut 3.3944268284888938e-06 -▁erkläre 3.3944268284888938e-06 -▁gefeiert 3.3944268284888938e-06 -▁tät 3.3944268284888938e-06 -▁zurückzukehren 3.3944268284888938e-06 -ZE 3.3906384056892414e-06 -▁bewährt 3.3906384056892414e-06 -▁detr 3.3906384056892414e-06 -▁ehemaliger 3.3906384056892414e-06 -▁Cad 3.3868499828895885e-06 -▁Generaldirektion 3.3868499828895885e-06 -▁marg 3.3868499828895885e-06 -▁trug 3.3868499828895885e-06 -agger 3.3830615600899356e-06 -indungs 3.3830615600899356e-06 -▁Konzer 3.3830615600899356e-06 -▁Satell 3.3830615600899356e-06 -▁Vergn 3.3830615600899356e-06 -▁dän 3.3830615600899356e-06 -▁folg 3.3830615600899356e-06 -▁relev 3.3830615600899356e-06 -▁wesentliches 3.3830615600899356e-06 -obal 3.3792731372902827e-06 -ocktails 3.3792731372902827e-06 -▁Kette 3.3792731372902827e-06 -▁Treat 3.3792731372902827e-06 -▁Treibhaus 3.3792731372902827e-06 -hnung 3.3754847144906302e-06 -ublik 3.3754847144906302e-06 -▁crit 3.3754847144906302e-06 -ensible 3.3716962916909773e-06 -h. 3.3716962916909773e-06 -known 3.3716962916909773e-06 -ployment 3.3716962916909773e-06 -▁kommerzi 3.3716962916909773e-06 -ertung 3.3679078688913244e-06 -klass 3.3679078688913244e-06 -ößt 3.3679078688913244e-06 -▁Virt 3.3679078688913244e-06 -izar 3.3641194460916715e-06 -▁Immun 3.3641194460916715e-06 -▁Deleg 3.3603310232920191e-06 -▁Fakt 3.3603310232920191e-06 -▁diesbezü 3.3603310232920191e-06 -QL 3.3565426004923662e-06 -amilie 3.3565426004923662e-06 -ilom 3.3565426004923662e-06 -Ç 3.3565426004923662e-06 -▁Sov 3.3565426004923662e-06 -▁entwick 3.3565426004923662e-06 -▁satell 3.3565426004923662e-06 -▁spok 3.3565426004923662e-06 -ewall 3.3527541776927133e-06 -ounge 3.3527541776927133e-06 -uers 3.3527541776927133e-06 -▁Amaz 3.3527541776927133e-06 -▁invol 3.3527541776927133e-06 -▁proport 3.3527541776927133e-06 -draul 3.3489657548930604e-06 -bends 3.3451773320934079e-06 -▁Künst 3.3451773320934079e-06 -▁Nam 3.3451773320934079e-06 -andelt 3.341388909293755e-06 -ught 3.341388909293755e-06 -urities 3.341388909293755e-06 -Р 3.3376004864941022e-06 -athon 3.3338120636944493e-06 -ingham 3.3338120636944493e-06 -istribution 3.3338120636944493e-06 -▁Admin 3.3338120636944493e-06 -▁Präm 3.3338120636944493e-06 -ilot 3.3300236408947968e-06 -▁challen 3.3300236408947968e-06 -ensicht 3.3262352180951439e-06 -indlichkeit 3.3262352180951439e-06 -▁Schattenberichterst 3.3262352180951439e-06 -▁Musl 3.322446795295491e-06 -▁angekünd 3.322446795295491e-06 -▁simultane 3.322446795295491e-06 -eases 3.3186583724958381e-06 -här 3.3186583724958381e-06 -▁Konsult 3.3186583724958381e-06 -▁Symb 3.3186583724958381e-06 -lastung 3.3148699496961857e-06 -okument 3.3148699496961857e-06 -verständlich 3.3148699496961857e-06 -▁tom 3.3148699496961857e-06 -itale 3.3110815268965328e-06 -lauben 3.3110815268965328e-06 -üben 3.3110815268965328e-06 -▁Kilom 3.3110815268965328e-06 -▁contin 3.30729310409688e-06 -ainer 3.303504681297227e-06 -bezogen 3.303504681297227e-06 -obility 3.303504681297227e-06 -▁endeav 3.303504681297227e-06 -▁mur 3.303504681297227e-06 -ählten 3.2997162584975745e-06 -▁Gü 3.2997162584975745e-06 -▁Sund 3.2997162584975745e-06 -▁embar 3.2997162584975745e-06 -▁landwirtschaft 3.2997162584975745e-06 -▁Lou 3.2959278356979216e-06 -▁Zert 3.2959278356979216e-06 -▁manufact 3.2959278356979216e-06 -iban 3.2921394128982687e-06 -▁Bearbeit 3.2921394128982687e-06 -▁Chal 3.2921394128982687e-06 -gearbeitet 3.2883509900986163e-06 -insen 3.2883509900986163e-06 -okratie 3.2883509900986163e-06 -▁Anwesen 3.2883509900986163e-06 -▁Fore 3.2845625672989634e-06 -▁Französ 3.2845625672989634e-06 -itteln 3.2807741444993105e-06 -selves 3.2807741444993105e-06 -▁Anh 3.2807741444993105e-06 -▁bz 3.2807741444993105e-06 -→ 3.2769857216996576e-06 -isode 3.2731972989000051e-06 -tle 3.2731972989000051e-06 -▁umge 3.2731972989000051e-06 -▁Überwach 3.2731972989000051e-06 -unting 3.2694088761003522e-06 -rael 3.2618320305010464e-06 -▁Ox 3.2618320305010464e-06 -▁exec 3.2618320305010464e-06 -▁mittler 3.2618320305010464e-06 -▁photograp 3.258043607701394e-06 -▁zerst 3.258043607701394e-06 -isms 3.2542551849017411e-06 -mittels 3.2542551849017411e-06 -▁Freil 3.2542551849017411e-06 -▁ausdrück 3.2504667621020882e-06 -iples 3.2466783393024353e-06 -▁Tyr 3.2466783393024353e-06 -▁Wurz 3.2466783393024353e-06 -▁exclus 3.2466783393024353e-06 -▁recru 3.2466783393024353e-06 -now 3.2428899165027828e-06 -ählen 3.2428899165027828e-06 -▁geziel 3.2428899165027828e-06 -chester 3.23910149370313e-06 -oted 3.23910149370313e-06 -sthet 3.235313070903477e-06 -▁describ 3.235313070903477e-06 -▁langfrist 3.235313070903477e-06 -▁städt 3.235313070903477e-06 -istration 3.2315246481038242e-06 -leister 3.2315246481038242e-06 -reisende 3.2315246481038242e-06 -schwung 3.2315246481038242e-06 -▁memor 3.2315246481038242e-06 -▁replac 3.2315246481038242e-06 -▁Etik 3.2277362253041717e-06 -▁Moz 3.2277362253041717e-06 -▁grenzüber 3.2277362253041717e-06 -cludes 3.2239478025045188e-06 -dress 3.2239478025045188e-06 -enty 3.2239478025045188e-06 -▁begleit 3.2239478025045188e-06 -▁geeign 3.2239478025045188e-06 -▁propriet 3.2239478025045188e-06 -▁ruh 3.2239478025045188e-06 -OME 3.2201593797048659e-06 -arbeitet 3.2201593797048659e-06 -arity 3.2201593797048659e-06 -uxe 3.2201593797048659e-06 -▁begeg 3.2201593797048659e-06 -▁democrat 3.2201593797048659e-06 -▁distingu 3.2201593797048659e-06 -aign 3.2125825341055605e-06 -ilosoph 3.2125825341055605e-06 -▁disadvant 3.2125825341055605e-06 -ething 3.2087941113059077e-06 -istical 3.2087941113059077e-06 -▁depos 3.2087941113059077e-06 -ploy 3.2050056885062548e-06 -▁consequ 3.2050056885062548e-06 -▁häuf 3.2050056885062548e-06 -itts 3.2012172657066019e-06 -standen 3.2012172657066019e-06 -▁drit 3.2012172657066019e-06 -asil 3.1974288429069494e-06 -genz 3.1974288429069494e-06 -▁Zimmers 3.1974288429069494e-06 -▁avail 3.1974288429069494e-06 -▁vict 3.1974288429069494e-06 -▁verur 3.1936404201072965e-06 -ildet 3.1898519973076436e-06 -wegs 3.1898519973076436e-06 -akischen 3.1860635745079907e-06 -reben 3.1860635745079907e-06 -▁Contin 3.1860635745079907e-06 -▁Gren 3.1860635745079907e-06 -dessen 3.1822751517083383e-06 -isol 3.1822751517083383e-06 -issens 3.1822751517083383e-06 -kas 3.1822751517083383e-06 -ulous 3.1822751517083383e-06 -▁Präz 3.1822751517083383e-06 -ن 3.1746983061090325e-06 -▁indigen 3.1746983061090325e-06 -▁opport 3.1746983061090325e-06 -▁Konzentr 3.1709098833093796e-06 -fassungs 3.1671214605097271e-06 -kommunik 3.1671214605097271e-06 -▁bekomm 3.1671214605097271e-06 -▁letz 3.1671214605097271e-06 -▁obser 3.1671214605097271e-06 -onz 3.1633330377100742e-06 -ã 3.1633330377100742e-06 -▁Proc 3.1633330377100742e-06 -messer 3.1595446149104213e-06 -schließ 3.1595446149104213e-06 -tragung 3.1595446149104213e-06 -ί 3.1595446149104213e-06 -▁Bang 3.1595446149104213e-06 -▁Chir 3.1595446149104213e-06 -▁Dut 3.1595446149104213e-06 -▁erstklass 3.1595446149104213e-06 -▁exzellent 3.1595446149104213e-06 -onomie 3.1557561921107684e-06 -spart 3.1557561921107684e-06 -unsten 3.1557561921107684e-06 -ystal 3.1557561921107684e-06 -▁Abhäng 3.1557561921107684e-06 -▁aufer 3.1557561921107684e-06 -▁famil 3.1557561921107684e-06 -zuschließen 3.151967769311116e-06 -▁Dolom 3.151967769311116e-06 -▁Mosk 3.151967769311116e-06 -▁compliment 3.151967769311116e-06 -▁indic 3.151967769311116e-06 -enze 3.1481793465114631e-06 -spezif 3.1481793465114631e-06 -ökonom 3.1481793465114631e-06 -▁Koh 3.1481793465114631e-06 -▁inhab 3.1481793465114631e-06 -folio 3.14439092371181e-06 -oleranz 3.14439092371181e-06 -▁config 3.14439092371181e-06 -▁econom 3.14439092371181e-06 -▁eindeut 3.1406025009121573e-06 -achtungen 3.1368140781125048e-06 -▁Controll 3.1368140781125048e-06 -▁Schü 3.1368140781125048e-06 -iversity 3.1330256553128519e-06 -▁Alph 3.1330256553128519e-06 -▁Gebä 3.1330256553128519e-06 -▁decis 3.1330256553128519e-06 --min 3.1292372325131991e-06 -scheinlich 3.1292372325131991e-06 -▁Rö 3.1292372325131991e-06 -stable 3.1254488097135462e-06 -yll 3.1254488097135462e-06 -▁Ausfl 3.1254488097135462e-06 -▁appropri 3.1254488097135462e-06 -▁pie 3.1254488097135462e-06 -gewiesen 3.1216603869138937e-06 -ilf 3.1216603869138937e-06 -leih 3.1216603869138937e-06 -▁Bürgerbeauftrag 3.1216603869138937e-06 -airo 3.1178719641142408e-06 -▁neol 3.1178719641142408e-06 -fare 3.1140835413145879e-06 -isely 3.1140835413145879e-06 -▁aufricht 3.1140835413145879e-06 -▁volunt 3.1140835413145879e-06 -alian 3.1102951185149354e-06 -▁Pun 3.1102951185149354e-06 -▁devast 3.1102951185149354e-06 -▁weiterentwick 3.1027182729156297e-06 -pap 3.0989298501159768e-06 -brea 3.0951414273163243e-06 -schloss 3.0951414273163243e-06 -▁Represent 3.0951414273163243e-06 -▁anwes 3.0951414273163243e-06 -bieter 3.0913530045166714e-06 -ivation 3.0913530045166714e-06 -ordination 3.0913530045166714e-06 -teid 3.0913530045166714e-06 -ähern 3.0913530045166714e-06 -▁broch 3.0913530045166714e-06 -▁verabschied 3.0913530045166714e-06 -alus 3.0875645817170185e-06 -cend 3.0875645817170185e-06 -vorsitzen 3.0875645817170185e-06 -▁Chines 3.0875645817170185e-06 -▁Zustell 3.0875645817170185e-06 -ince 3.0837761589173656e-06 -zielle 3.0837761589173656e-06 -▁münd 3.0837761589173656e-06 -itarian 3.0799877361177132e-06 -▁Inv 3.0799877361177132e-06 -▁consolid 3.0799877361177132e-06 -racy 3.0761993133180603e-06 -undern 3.0761993133180603e-06 -zte 3.0761993133180603e-06 -▁ö 3.0761993133180603e-06 -eigen 3.0724108905184074e-06 -inzips 3.0724108905184074e-06 -unsch 3.0686224677187545e-06 -ō 3.0686224677187545e-06 -▁Jed 3.0686224677187545e-06 -▁Minim 3.0686224677187545e-06 -▁sever 3.064834044919102e-06 -Script 3.0610456221194491e-06 -sichtig 3.0610456221194491e-06 -trem 3.0610456221194491e-06 -ügt 3.0610456221194491e-06 -▁behin 3.0610456221194491e-06 -▁ermut 3.0610456221194491e-06 -▁künstler 3.0610456221194491e-06 -ücher 3.0572571993197962e-06 -vement 3.0534687765201433e-06 -▁bilater 3.0534687765201433e-06 -▁erz 3.0534687765201433e-06 -▁Übersetz 3.0534687765201433e-06 -− 3.0496803537204909e-06 -▁beher 3.0496803537204909e-06 -▁bewaff 3.0496803537204909e-06 -▁bicy 3.0496803537204909e-06 -ergeb 3.045891930920838e-06 -▁Alger 3.045891930920838e-06 -▁Mig 3.045891930920838e-06 -▁Moroc 3.045891930920838e-06 -▁griech 3.045891930920838e-06 -▁Behinder 3.0421035081211851e-06 -▁Speich 3.0421035081211851e-06 -▁Verbesser 3.0421035081211851e-06 --run 3.0383150853215322e-06 -▁Ira 3.0383150853215322e-06 -▁cert 3.0383150853215322e-06 -▁lad 3.0383150853215322e-06 -▁wed 3.0345266625218797e-06 -abwe 3.0307382397222268e-06 -inein 3.0307382397222268e-06 -▁imbal 3.0307382397222268e-06 -issance 3.0269498169225739e-06 -mögen 3.0231613941229211e-06 -pg 3.0231613941229211e-06 -İ 3.0231613941229211e-06 -klärung 3.0193729713232686e-06 -▁Curr 3.0193729713232686e-06 -▁celebr 3.0193729713232686e-06 -▁enchant 3.0193729713232686e-06 -▁geden 3.0193729713232686e-06 -▁sour 3.0193729713232686e-06 -estaurant 3.0155845485236157e-06 -icans 3.0155845485236157e-06 -▁Agricult 3.0155845485236157e-06 -▁Anhör 3.0155845485236157e-06 -▁Eurost 3.0155845485236157e-06 -▁schuld 3.0155845485236157e-06 -ournal 3.0117961257239628e-06 -rohungen 3.0117961257239628e-06 -µ 3.0117961257239628e-06 -▁Glaub 3.0117961257239628e-06 -wände 3.00800770292431e-06 -öp 3.00800770292431e-06 -▁Puert 3.00800770292431e-06 -▁constr 3.00800770292431e-06 -▁zusätz 3.00800770292431e-06 -XI 3.0042192801246574e-06 -iertel 3.0042192801246574e-06 -insula 3.0042192801246574e-06 -ühungen 3.0042192801246574e-06 -▁hilf 3.0042192801246574e-06 -▁overw 3.0042192801246574e-06 -ooms 3.0004308573250045e-06 -ī 3.0004308573250045e-06 -▁korre 3.0004308573250045e-06 -▁unverzicht 3.0004308573250045e-06 -▁ww 3.0004308573250045e-06 -ategory 2.9966424345253517e-06 -hetor 2.9966424345253517e-06 -idelines 2.9966424345253517e-06 -▁Entschäd 2.9928540117256988e-06 -▁Pest 2.9928540117256988e-06 -▁Zinss 2.9928540117256988e-06 -kauft 2.9890655889260463e-06 -▁Certain 2.9890655889260463e-06 -▁introd 2.9890655889260463e-06 -▁irregular 2.9890655889260463e-06 -Ø 2.9852771661263934e-06 -ämpft 2.9852771661263934e-06 -▁amt 2.9852771661263934e-06 -aue 2.9814887433267405e-06 -▁Balk 2.9814887433267405e-06 -▁kle 2.9814887433267405e-06 -ächst 2.9777003205270876e-06 -▁competen 2.9777003205270876e-06 -▁jähr 2.9777003205270876e-06 -ischung 2.9739118977274352e-06 -▁außergewöhn 2.9739118977274352e-06 -▁kompliz 2.9739118977274352e-06 -▁wöch 2.9739118977274352e-06 -iament 2.9701234749277823e-06 -▁Nichtraucher 2.9701234749277823e-06 -▁vermut 2.9701234749277823e-06 -rohung 2.9663350521281294e-06 -▁Behaupt 2.9663350521281294e-06 -▁Medikament 2.9663350521281294e-06 -▁enthusi 2.9663350521281294e-06 -▁einzigart 2.9625466293284765e-06 -▁lif 2.9625466293284765e-06 -yles 2.958758206528824e-06 -▁Pac 2.958758206528824e-06 -chmark 2.9549697837291711e-06 -mant 2.9549697837291711e-06 -okraten 2.9549697837291711e-06 -▁Kör 2.9549697837291711e-06 -▁damal 2.9549697837291711e-06 -estens 2.9511813609295182e-06 -like 2.9511813609295182e-06 -▁tran 2.9511813609295182e-06 -LR 2.9473929381298653e-06 -gänger 2.9473929381298653e-06 -kre 2.9473929381298653e-06 -П 2.9473929381298653e-06 -▁chines 2.9473929381298653e-06 -▁lingu 2.9473929381298653e-06 -▁nic 2.9473929381298653e-06 -▁Ökonom 2.9473929381298653e-06 -NS 2.9436045153302129e-06 -▁herkömm 2.9436045153302129e-06 -▁wom 2.9436045153302129e-06 -plus 2.93981609253056e-06 -zufügen 2.93981609253056e-06 -rough 2.9360276697309071e-06 -▁ausgezeichn 2.9360276697309071e-06 -itiative 2.9284508241316017e-06 -kind 2.9284508241316017e-06 -spricht 2.9284508241316017e-06 -while 2.9284508241316017e-06 -▁Aufent 2.9284508241316017e-06 -▁Poss 2.9284508241316017e-06 -▁bewähr 2.9284508241316017e-06 -▁cloth 2.9284508241316017e-06 -eshalb 2.9246624013319488e-06 -▁constra 2.9246624013319488e-06 -▁corresp 2.9246624013319488e-06 -▁feat 2.9246624013319488e-06 -ionalität 2.9208739785322959e-06 -iveau 2.9208739785322959e-06 -▁Eure 2.9208739785322959e-06 -auso 2.9170855557326435e-06 -▁aspir 2.9170855557326435e-06 -▁heav 2.9170855557326435e-06 -▁Horiz 2.9132971329329906e-06 -▁Ukra 2.9132971329329906e-06 -beziehung 2.9095087101333377e-06 -kretär 2.9095087101333377e-06 -wise 2.9095087101333377e-06 -▁geschmack 2.9095087101333377e-06 -▁sod 2.9095087101333377e-06 -▁Interpret 2.9057202873336848e-06 -nerg 2.9019318645340323e-06 -ustom 2.9019318645340323e-06 -▁Cred 2.8981434417343794e-06 -▁engl 2.8981434417343794e-06 -strafe 2.8943550189347266e-06 -ährung 2.8943550189347266e-06 -▁Comput 2.8943550189347266e-06 -▁arbitr 2.8943550189347266e-06 -gelassen 2.8905665961350737e-06 -iech 2.8905665961350737e-06 -Gu 2.8867781733354212e-06 -berichterst 2.8867781733354212e-06 -uay 2.8867781733354212e-06 -▁entw 2.8867781733354212e-06 -criptions 2.8829897505357683e-06 -lege 2.8829897505357683e-06 -pendent 2.8829897505357683e-06 -▁appreci 2.8829897505357683e-06 -gencies 2.8792013277361154e-06 -hai 2.8792013277361154e-06 -▁bedro 2.8792013277361154e-06 -▁diagn 2.8792013277361154e-06 -▁extraordin 2.8792013277361154e-06 -istrat 2.87162448213681e-06 -iterran 2.87162448213681e-06 -jud 2.87162448213681e-06 -ml 2.87162448213681e-06 -▁Leh 2.87162448213681e-06 -▁Okt 2.87162448213681e-06 -▁asi 2.87162448213681e-06 -inzip 2.8678360593371572e-06 -katastrop 2.8678360593371572e-06 -ünft 2.8678360593371572e-06 -engen 2.8640476365375043e-06 -minuten 2.8640476365375043e-06 -▁kontakt 2.8640476365375043e-06 -B. 2.8602592137378514e-06 -gebote 2.8564707909381989e-06 -▁Initi 2.8564707909381989e-06 -▁Überrasch 2.8564707909381989e-06 -▁konzip 2.852682368138546e-06 -yo 2.8488939453388931e-06 -öglich 2.8488939453388931e-06 -▁Lic 2.8488939453388931e-06 -geschoss 2.8451055225392402e-06 -ulegen 2.8451055225392402e-06 -▁bir 2.8451055225392402e-06 -gehalten 2.8413170997395878e-06 -▁ander 2.8413170997395878e-06 -▁tang 2.8413170997395878e-06 -clear 2.8375286769399349e-06 -imens 2.8375286769399349e-06 -scheid 2.8375286769399349e-06 -wirk 2.8375286769399349e-06 -▁Zerst 2.8375286769399349e-06 -▁accompl 2.8375286769399349e-06 -ctual 2.833740254140282e-06 -orough 2.833740254140282e-06 -date 2.8299518313406291e-06 -▁spectac 2.8299518313406291e-06 -▁Germ 2.8261634085409766e-06 -▁academ 2.8261634085409766e-06 -▁bene 2.8261634085409766e-06 -lechts 2.8223749857413237e-06 -mert 2.8223749857413237e-06 -schrank 2.8223749857413237e-06 -ر 2.8223749857413237e-06 -▁Religions 2.8223749857413237e-06 -▁Reicht 2.8185865629416708e-06 -▁Thyssen 2.8185865629416708e-06 -▁ehem 2.8185865629416708e-06 -Ђ 2.8147981401420179e-06 -▁bemer 2.8147981401420179e-06 --hour 2.8110097173423655e-06 -ignen 2.8110097173423655e-06 -istiken 2.8110097173423655e-06 -atile 2.8072212945427126e-06 -▁Gastr 2.8034328717430597e-06 -▁jeg 2.8034328717430597e-06 -istrib 2.7996444489434068e-06 -kun 2.7996444489434068e-06 -odex 2.7996444489434068e-06 -zeption 2.7996444489434068e-06 -▁stet 2.7996444489434068e-06 -ayers 2.7958560261437543e-06 -etime 2.7958560261437543e-06 -oyal 2.7958560261437543e-06 -ränkt 2.7958560261437543e-06 -tei 2.7958560261437543e-06 -▁iss 2.7958560261437543e-06 -izit 2.7920676033441014e-06 -äsident 2.7920676033441014e-06 -▁Jugendher 2.7920676033441014e-06 -ensions 2.7882791805444486e-06 -▁condem 2.7882791805444486e-06 -fachen 2.7844907577447957e-06 -▁occas 2.7844907577447957e-06 -ämpfen 2.7807023349451432e-06 -▁Trink 2.7807023349451432e-06 -ahme 2.7769139121454903e-06 -bearbeit 2.7769139121454903e-06 -idden 2.7769139121454903e-06 -▁Agre 2.7769139121454903e-06 -▁Niederl 2.7769139121454903e-06 -▁Heat 2.7693370665461845e-06 -▁Rech 2.7693370665461845e-06 -▁übern 2.7693370665461845e-06 -▁zert 2.7655486437465321e-06 -ksam 2.7617602209468792e-06 -▁gekenn 2.7617602209468792e-06 -▁Columb 2.7541833753475734e-06 -ächlich 2.7503949525479209e-06 -cling 2.746606529748268e-06 -irlpool 2.746606529748268e-06 -öffnete 2.746606529748268e-06 -▁Berat 2.746606529748268e-06 -▁Großbritann 2.746606529748268e-06 -ragt 2.7428181069486151e-06 -rieden 2.7428181069486151e-06 -ulevard 2.7428181069486151e-06 -▁Quest 2.7390296841489627e-06 -▁Wed 2.7390296841489627e-06 -▁demonst 2.7390296841489627e-06 -▁stim 2.7390296841489627e-06 -eachtet 2.7352412613493098e-06 -verän 2.7352412613493098e-06 -zipl 2.7352412613493098e-06 -▁Sonnt 2.7352412613493098e-06 -▁gee 2.7352412613493098e-06 -attformen 2.7314528385496569e-06 -▁legit 2.7314528385496569e-06 -▁schrift 2.7314528385496569e-06 -strukt 2.727664415750004e-06 -▁geleist 2.727664415750004e-06 -eaking 2.7238759929503515e-06 -emark 2.7238759929503515e-06 -▁Hist 2.7238759929503515e-06 -▁Ress 2.7238759929503515e-06 -▁Schmid 2.7238759929503515e-06 -▁incur 2.7238759929503515e-06 -▁Bec 2.7200875701506986e-06 -▁determ 2.7200875701506986e-06 -hältn 2.7162991473510457e-06 -▁Ris 2.7162991473510457e-06 -▁irgendwel 2.7162991473510457e-06 -konstrukt 2.7125107245513928e-06 -stark 2.7125107245513928e-06 -umed 2.7125107245513928e-06 -▁Subsidiaritäts 2.7125107245513928e-06 -anspruch 2.7087223017517404e-06 -mitten 2.7087223017517404e-06 -▁Darf 2.7087223017517404e-06 -▁Gemü 2.7087223017517404e-06 -ether 2.7049338789520875e-06 -▁Echt 2.7049338789520875e-06 -▁negl 2.7049338789520875e-06 -ürlich 2.7011454561524346e-06 -▁astr 2.7011454561524346e-06 -▁debatt 2.7011454561524346e-06 -▁disch 2.7011454561524346e-06 -▁hervorra 2.7011454561524346e-06 -ogle 2.6973570333527817e-06 --Europ 2.6935686105531292e-06 -f. 2.6935686105531292e-06 -iegelt 2.6935686105531292e-06 -▁prepar 2.6935686105531292e-06 -arus 2.6897801877534763e-06 -▁ASP 2.6897801877534763e-06 -ά 2.6859917649538234e-06 -damer 2.6822033421541706e-06 -ieval 2.6822033421541706e-06 -oire 2.6822033421541706e-06 -yards 2.6822033421541706e-06 -▁Bom 2.6822033421541706e-06 -▁freiwill 2.6822033421541706e-06 -strument 2.6784149193545181e-06 -erium 2.6746264965548652e-06 -frastruktur 2.6746264965548652e-06 -▁näch 2.6746264965548652e-06 -▁repress 2.6746264965548652e-06 -öte 2.6708380737552123e-06 -▁Weiterbild 2.6708380737552123e-06 -▁urg 2.6708380737552123e-06 -hing 2.6670496509555594e-06 -overty 2.6670496509555594e-06 -▁Vien 2.6670496509555594e-06 -deb 2.6632612281559069e-06 -orer 2.6632612281559069e-06 -otherap 2.6632612281559069e-06 -ülle 2.6632612281559069e-06 -▁Feed 2.6632612281559069e-06 -elta 2.6594728053562541e-06 -▁ASE 2.6594728053562541e-06 -▁Sozialdem 2.6594728053562541e-06 --sized 2.6556843825566012e-06 -opher 2.6556843825566012e-06 -▁alp 2.6556843825566012e-06 -▁beschäd 2.6556843825566012e-06 -▁wirks 2.6556843825566012e-06 -holders 2.6518959597569483e-06 -inea 2.6518959597569483e-06 -▁Wit 2.6518959597569483e-06 -ingung 2.6481075369572958e-06 -▁feas 2.6481075369572958e-06 -▁valu 2.6481075369572958e-06 -▁canc 2.6443191141576429e-06 -▁hil 2.6443191141576429e-06 -kö 2.64053069135799e-06 -▁Welthandels 2.64053069135799e-06 -▁exklus 2.64053069135799e-06 -agna 2.6367422685583371e-06 -annel 2.6367422685583371e-06 -bul 2.6367422685583371e-06 -gert 2.6367422685583371e-06 -tons 2.6367422685583371e-06 -▁bedau 2.6367422685583371e-06 -▁exempl 2.6367422685583371e-06 -▁reag 2.6367422685583371e-06 -harmon 2.6329538457586847e-06 -▁Kür 2.6329538457586847e-06 -▁deserv 2.6329538457586847e-06 -▁doppel 2.6329538457586847e-06 -▁Welthand 2.6291654229590318e-06 -▁behav 2.6291654229590318e-06 -eterm 2.6253770001593789e-06 -▁beträcht 2.6253770001593789e-06 -erkennung 2.6178001545600735e-06 -wach 2.6178001545600735e-06 -anni 2.6140117317604206e-06 -mosphäre 2.6140117317604206e-06 -ruh 2.6140117317604206e-06 -uage 2.6140117317604206e-06 -▁Kä 2.6140117317604206e-06 -▁democ 2.6140117317604206e-06 -.m. 2.6102233089607677e-06 -empl 2.6102233089607677e-06 -geiz 2.6102233089607677e-06 -▁lauf 2.6102233089607677e-06 -andlung 2.6064348861611148e-06 -▁hydraul 2.6064348861611148e-06 --Mittel 2.6026464633614624e-06 -richtet 2.6026464633614624e-06 -▁Abente 2.6026464633614624e-06 -▁Baron 2.6026464633614624e-06 -▁Eleg 2.6026464633614624e-06 -▁Ausge 2.5988580405618095e-06 -▁Tipp 2.5988580405618095e-06 -arios 2.5950696177621566e-06 -breite 2.5950696177621566e-06 -isure 2.5950696177621566e-06 -▁Akte 2.5950696177621566e-06 -▁Eben 2.5950696177621566e-06 -▁unfor 2.5950696177621566e-06 -erma 2.5912811949625037e-06 -▁preserv 2.5912811949625037e-06 -national 2.5874927721628512e-06 -aubern 2.5837043493631983e-06 -gerichts 2.5837043493631983e-06 -▁E-C 2.5837043493631983e-06 -▁Jew 2.5837043493631983e-06 -▁nob 2.5837043493631983e-06 -arbeitung 2.5799159265635454e-06 -schnitts 2.5799159265635454e-06 -▁Fift 2.5799159265635454e-06 -▁fig 2.5799159265635454e-06 -abled 2.5761275037638926e-06 -bsicht 2.5761275037638926e-06 -sterreich 2.5761275037638926e-06 -ي 2.5761275037638926e-06 -aneben 2.57233908096424e-06 -heiz 2.57233908096424e-06 -ignet 2.57233908096424e-06 -onymous 2.57233908096424e-06 -wärtige 2.57233908096424e-06 -▁Beschäft 2.57233908096424e-06 -elated 2.5685506581645872e-06 -borg 2.5647622353649343e-06 -etzugang 2.5647622353649343e-06 -unately 2.5647622353649343e-06 -wiesen 2.5647622353649343e-06 -eiden 2.5609738125652818e-06 -äßt 2.5609738125652818e-06 -▁gäng 2.5609738125652818e-06 -df 2.5571853897656289e-06 -ilitation 2.5571853897656289e-06 -▁Funkt 2.5571853897656289e-06 -▁Ratsvors 2.5571853897656289e-06 --the- 2.5533969669659761e-06 -▁bran 2.5533969669659761e-06 -ό 2.5496085441663232e-06 -É 2.5458201213666707e-06 -▁erfüll 2.5458201213666707e-06 -bereiten 2.5420316985670178e-06 -▁displ 2.5420316985670178e-06 -ū 2.5382432757673649e-06 -▁chang 2.5382432757673649e-06 -▁unab 2.5382432757673649e-06 -▁vill 2.5382432757673649e-06 -force 2.534454852967712e-06 -idges 2.534454852967712e-06 -ollar 2.534454852967712e-06 -tschaft 2.534454852967712e-06 -▁Erhö 2.5306664301680596e-06 -▁fal 2.5306664301680596e-06 -▁requ 2.5306664301680596e-06 -▁unz 2.5306664301680596e-06 -▁Frühstücks 2.5268780073684067e-06 -▁offiz 2.5268780073684067e-06 -▁socio 2.5268780073684067e-06 -ächtigt 2.5230895845687538e-06 -▁Inner 2.5230895845687538e-06 -essene 2.5193011617691009e-06 -ikas 2.5193011617691009e-06 -▁beaut 2.5193011617691009e-06 -▁expend 2.5193011617691009e-06 -▁grundle 2.5193011617691009e-06 -▁japan 2.5193011617691009e-06 -▁hoff 2.5155127389694484e-06 -hereas 2.5117243161697955e-06 -tenance 2.5117243161697955e-06 -▁Beweg 2.5117243161697955e-06 -▁veränder 2.5117243161697955e-06 -mitting 2.5079358933701426e-06 -otive 2.5079358933701426e-06 -▁fascin 2.5041474705704897e-06 -kategorien 2.5003590477708373e-06 -▁woll 2.4927822021715315e-06 -ideos 2.4889937793718786e-06 -▁Assoziierungs 2.4889937793718786e-06 -orning 2.4852053565722261e-06 -reme 2.4852053565722261e-06 -Ó 2.4852053565722261e-06 -▁Ide 2.4852053565722261e-06 -▁gam 2.4852053565722261e-06 -▁taxp 2.4852053565722261e-06 -▁Booking 2.4814169337725732e-06 -▁Fron 2.4814169337725732e-06 -▁biotechn 2.4814169337725732e-06 -ennbar 2.4776285109729203e-06 -onse 2.4776285109729203e-06 -▁kü 2.4776285109729203e-06 -▁narr 2.4776285109729203e-06 -gefunden 2.4738400881732674e-06 -örderung 2.4738400881732674e-06 -▁Compet 2.4738400881732674e-06 -▁IL 2.4738400881732674e-06 --site 2.470051665373615e-06 -otograf 2.470051665373615e-06 -wohnt 2.470051665373615e-06 -▁Valen 2.470051665373615e-06 -zeugung 2.4662632425739621e-06 -▁conj 2.4662632425739621e-06 -▁crea 2.4662632425739621e-06 -▁definit 2.4662632425739621e-06 -▁israel 2.4662632425739621e-06 -Os 2.4624748197743092e-06 -euerung 2.4624748197743092e-06 -▁SME 2.4624748197743092e-06 --Sta 2.4586863969746563e-06 -amination 2.4586863969746563e-06 -annual 2.4586863969746563e-06 -geteilt 2.4586863969746563e-06 -illigt 2.4586863969746563e-06 -webe 2.4586863969746563e-06 -▁Fourth 2.4586863969746563e-06 -▁dispar 2.4586863969746563e-06 -▁vic 2.4586863969746563e-06 -hone 2.4548979741750038e-06 -▁Lithu 2.4548979741750038e-06 -▁Mex 2.4548979741750038e-06 -beschl 2.4511095513753509e-06 -▁Meth 2.4511095513753509e-06 -▁anspruchs 2.4511095513753509e-06 -iry 2.4473211285756981e-06 -örtern 2.4473211285756981e-06 -▁geein 2.4473211285756981e-06 -▁sect 2.4473211285756981e-06 -▁н 2.4473211285756981e-06 -ulpt 2.4435327057760452e-06 -ilfen 2.4397442829763927e-06 -inema 2.4397442829763927e-06 -ismen 2.4397442829763927e-06 -urement 2.4397442829763927e-06 -م 2.4397442829763927e-06 -▁überwie 2.4397442829763927e-06 -estellt 2.43595586017674e-06 -▁künft 2.43595586017674e-06 -enzial 2.428379014577434e-06 -▁Konz 2.428379014577434e-06 -▁ergän 2.428379014577434e-06 -betreuung 2.4245905917777816e-06 -nier 2.4245905917777816e-06 -▁Veg 2.4245905917777816e-06 -klaus 2.4208021689781287e-06 -ünstler 2.4208021689781287e-06 -▁Feb 2.4208021689781287e-06 -▁Rechtsstaat 2.4208021689781287e-06 -▁behaupt 2.4208021689781287e-06 -▁schein 2.4208021689781287e-06 -ungalows 2.4170137461784758e-06 -▁congratul 2.4170137461784758e-06 -apan 2.4132253233788229e-06 -ohner 2.4132253233788229e-06 -immers 2.4094369005791704e-06 -ivalent 2.4094369005791704e-06 -δ 2.4094369005791704e-06 -mischung 2.4056484777795175e-06 -mosph 2.4056484777795175e-06 -oton 2.4056484777795175e-06 -▁Ahm 2.4056484777795175e-06 -▁Krank 2.4056484777795175e-06 -onenten 2.4018600549798646e-06 -ustain 2.4018600549798646e-06 -▁inakzept 2.4018600549798646e-06 -▁prag 2.4018600549798646e-06 -wärmung 2.3980716321802117e-06 -zeugt 2.3980716321802117e-06 -▁Lom 2.3980716321802117e-06 -▁Rüst 2.3980716321802117e-06 -▁Vors 2.3980716321802117e-06 -▁elig 2.3980716321802117e-06 -hmen 2.3942832093805593e-06 -ventional 2.3942832093805593e-06 -▁Vorder 2.3942832093805593e-06 -▁erschein 2.3942832093805593e-06 -▁Ägyp 2.3942832093805593e-06 -ű 2.3904947865809064e-06 -▁angebot 2.3904947865809064e-06 -▁dif 2.3904947865809064e-06 -ifestyle 2.3867063637812535e-06 -irtschaftung 2.3867063637812535e-06 -äuser 2.3867063637812535e-06 -▁Bevölker 2.3867063637812535e-06 -▁Obs 2.3867063637812535e-06 -▁aesthet 2.3867063637812535e-06 -▁haft 2.3867063637812535e-06 -ocol 2.382917940981601e-06 -▁diplom 2.382917940981601e-06 -icular 2.3791295181819481e-06 -▁plöt 2.3791295181819481e-06 -beteil 2.3715526725826423e-06 -irms 2.3715526725826423e-06 -▁zukünft 2.3715526725826423e-06 -VIS 2.36776424978299e-06 -gebä 2.36776424978299e-06 -rection 2.36776424978299e-06 -örd 2.363975826983337e-06 -ė 2.363975826983337e-06 -▁telec 2.363975826983337e-06 -aufent 2.3601874041836841e-06 -emes 2.3601874041836841e-06 -▁ic 2.3563989813840312e-06 -apers 2.3526105585843787e-06 -▁Gebur 2.3526105585843787e-06 -▁splend 2.3526105585843787e-06 -▁abhäng 2.3488221357847258e-06 -iaries 2.3450337129850729e-06 -▁erreich 2.3450337129850729e-06 -▁impr 2.3450337129850729e-06 -▁occ 2.3450337129850729e-06 -igger 2.34124529018542e-06 -klärt 2.34124529018542e-06 -▁Köl 2.3374568673857676e-06 -▁Micros 2.3374568673857676e-06 -▁monopol 2.3374568673857676e-06 -eldet 2.3298800217864618e-06 -hagen 2.3298800217864618e-06 -imming 2.3298800217864618e-06 -itudes 2.3298800217864618e-06 -ifff 2.3260915989868089e-06 -othing 2.3260915989868089e-06 -› 2.3260915989868089e-06 -▁Bosn 2.3260915989868089e-06 -▁Kriminal 2.3260915989868089e-06 -▁leist 2.3260915989868089e-06 -▁tiss 2.3260915989868089e-06 -▁Feh 2.3223031761871564e-06 -atures 2.3185147533875036e-06 -▁Griechen 2.3185147533875036e-06 -▁erlang 2.3185147533875036e-06 -▁sharehol 2.3185147533875036e-06 -▁timet 2.3185147533875036e-06 -actions 2.3147263305878507e-06 -▁Regard 2.3147263305878507e-06 -▁offenkund 2.3147263305878507e-06 -elegen 2.3109379077881978e-06 -iveaus 2.3109379077881978e-06 -kord 2.3109379077881978e-06 -ätigkeiten 2.3109379077881978e-06 -▁schritt 2.3109379077881978e-06 -▁unan 2.3109379077881978e-06 -bean 2.3071494849885453e-06 -schied 2.3071494849885453e-06 -gänglich 2.3033610621888924e-06 -sparungen 2.3033610621888924e-06 -▁Cres 2.3033610621888924e-06 -▁Prest 2.3033610621888924e-06 -▁anerk 2.3033610621888924e-06 -▁beauf 2.3033610621888924e-06 -▁init 2.3033610621888924e-06 -angehör 2.2995726393892395e-06 -igher 2.2995726393892395e-06 -ughter 2.2995726393892395e-06 -& 2.2957842165895866e-06 -apos; 2.2957842165895866e-06 -Н 2.2957842165895866e-06 -▁irres 2.2957842165895866e-06 -anzler 2.2919957937899342e-06 -ording 2.2919957937899342e-06 -orgt 2.2919957937899342e-06 -pakt 2.2919957937899342e-06 -▁Aller 2.2919957937899342e-06 -▁Erdbe 2.2919957937899342e-06 -▁Karls 2.2919957937899342e-06 --Med 2.2882073709902813e-06 -reien 2.2882073709902813e-06 -ustausch 2.2882073709902813e-06 -▁Indik 2.2882073709902813e-06 -▁mac 2.2882073709902813e-06 -▁einstimm 2.2844189481906284e-06 -ounder 2.2806305253909755e-06 -▁Lond 2.2806305253909755e-06 -▁biet 2.2806305253909755e-06 -▁proxim 2.2806305253909755e-06 --Protokoll 2.276842102591323e-06 -iedelt 2.276842102591323e-06 -± 2.276842102591323e-06 -▁Südt 2.276842102591323e-06 -▁beeinfl 2.276842102591323e-06 -▁haz 2.276842102591323e-06 -▁sag 2.276842102591323e-06 -▁undert 2.276842102591323e-06 -penden 2.27305367979167e-06 -more 2.2692652569920172e-06 -▁intell 2.2692652569920172e-06 --sc 2.2654768341923643e-06 -▁effekt 2.2654768341923643e-06 -klang 2.2616884113927119e-06 -× 2.2616884113927119e-06 -▁freed 2.2616884113927119e-06 -▁webs 2.2616884113927119e-06 -▁getrenn 2.257899988593059e-06 -astet 2.2541115657934061e-06 -eeing 2.2541115657934061e-06 -▁Kup 2.2541115657934061e-06 -▁mak 2.2541115657934061e-06 -habt 2.2503231429937532e-06 -uled 2.2503231429937532e-06 -vors 2.2503231429937532e-06 -▁Frühstücksb 2.2503231429937532e-06 -▁Reh 2.2503231429937532e-06 -▁carr 2.2503231429937532e-06 -▁gewinn 2.2503231429937532e-06 -▁übr 2.2503231429937532e-06 -schalter 2.2465347201941007e-06 -wün 2.2465347201941007e-06 -zufolge 2.2465347201941007e-06 -▁Beif 2.2427462973944478e-06 -▁bureauc 2.2427462973944478e-06 -acio 2.2389578745947949e-06 -brit 2.2389578745947949e-06 -esien 2.2389578745947949e-06 -tendlich 2.2389578745947949e-06 -ziplin 2.2389578745947949e-06 -änderungs 2.2389578745947949e-06 -▁beschleun 2.2389578745947949e-06 -▁großzüg 2.2389578745947949e-06 -assador 2.2351694517951421e-06 -▁spac 2.2351694517951421e-06 -▁vulner 2.2351694517951421e-06 -^ 2.2313810289954896e-06 -▁Cz 2.2313810289954896e-06 -▁autonom 2.2313810289954896e-06 -verletzungen 2.2275926061958367e-06 -berat 2.2238041833961838e-06 -checken 2.2238041833961838e-06 -gedeckt 2.2238041833961838e-06 -izenzen 2.2238041833961838e-06 -▁Kohäsions 2.2238041833961838e-06 -▁roy 2.2238041833961838e-06 -üller 2.2200157605965309e-06 -▁Jahrz 2.2200157605965309e-06 -eople 2.2162273377968784e-06 -▁Rak 2.2162273377968784e-06 -ieße 2.2124389149972256e-06 -▁nörd 2.2124389149972256e-06 -icer 2.2086504921975727e-06 -▁dign 2.2086504921975727e-06 -ingungen 2.2048620693979198e-06 -▁Konfigur 2.2048620693979198e-06 -▁nament 2.2048620693979198e-06 -▁shoul 2.2048620693979198e-06 -ayout 2.2010736465982673e-06 -▁preca 2.2010736465982673e-06 -▁purs 2.2010736465982673e-06 -ellent 2.1972852237986144e-06 -schaffung 2.1972852237986144e-06 -▁Sold 2.1972852237986144e-06 -▁transatlant 2.1972852237986144e-06 -atem 2.1934968009989615e-06 -iscr 2.1934968009989615e-06 -mitted 2.1934968009989615e-06 -▁Vict 2.1934968009989615e-06 -▁unabhäng 2.1934968009989615e-06 -▁wirk 2.1934968009989615e-06 -iary 2.1897083781993091e-06 -γ 2.1897083781993091e-06 -▁Foot 2.1897083781993091e-06 -▁Leban 2.1897083781993091e-06 -▁keines 2.1897083781993091e-06 --orient 2.1859199553996562e-06 -eist 2.1859199553996562e-06 -ingual 2.1859199553996562e-06 -izienz 2.1859199553996562e-06 -▁illeg 2.1859199553996562e-06 -▁länd 2.1859199553996562e-06 --Mit 2.1821315326000033e-06 -invest 2.1821315326000033e-06 -▁Profess 2.1821315326000033e-06 --economic 2.1783431098003504e-06 -irable 2.1783431098003504e-06 -peror 2.1783431098003504e-06 -▁Verpflicht 2.1783431098003504e-06 -▁gescheit 2.1745546870006979e-06 -▁guar 2.1745546870006979e-06 -▁non-s 2.1745546870006979e-06 -▁traff 2.1745546870006979e-06 -▁verbesser 2.1745546870006979e-06 -ereg 2.170766264201045e-06 -fertigen 2.170766264201045e-06 -џ 2.170766264201045e-06 -▁hist 2.170766264201045e-06 -drucks 2.1669778414013921e-06 -gratul 2.1669778414013921e-06 -lez 2.1669778414013921e-06 -▁EZ 2.1669778414013921e-06 -▁ow 2.1669778414013921e-06 -▁österreich 2.1669778414013921e-06 -leun 2.1631894186017392e-06 -oard 2.1631894186017392e-06 -▁candid 2.1631894186017392e-06 -▁sug 2.1631894186017392e-06 -▁Sonn 2.1594009958020868e-06 -hütung 2.1556125730024339e-06 -▁Komprom 2.151824150202781e-06 -▁Schließ 2.151824150202781e-06 -palette 2.1480357274031281e-06 -terne 2.1480357274031281e-06 -▁Fisher 2.1480357274031281e-06 -▁Rechen 2.1480357274031281e-06 -▁incr 2.1480357274031281e-06 -iensten 2.1442473046034756e-06 -▁Sow 2.1442473046034756e-06 -▁scrut 2.1442473046034756e-06 -О 2.1404588818038227e-06 -▁Sept 2.1404588818038227e-06 -iciary 2.13667045900417e-06 -▁Urs 2.13667045900417e-06 -uzierung 2.132882036204517e-06 -К 2.132882036204517e-06 -▁Frem 2.132882036204517e-06 -▁Sowjet 2.1290936134048645e-06 -▁unus 2.1290936134048645e-06 -▁zugesch 2.1290936134048645e-06 -begriffen 2.1253051906052116e-06 -schuldigen 2.1253051906052116e-06 -stütz 2.1253051906052116e-06 -▁Sever 2.1253051906052116e-06 -▁Tai 2.1253051906052116e-06 -▁nutr 2.1253051906052116e-06 -▁zeich 2.1253051906052116e-06 -inery 2.1215167678055587e-06 -▁Flücht 2.1215167678055587e-06 -▁eingere 2.1215167678055587e-06 -▁eleg 2.1215167678055587e-06 -übung 2.1177283450059058e-06 -entrale 2.1139399222062533e-06 -▁fem 2.1139399222062533e-06 -▁jurisd 2.1139399222062533e-06 -reche 2.1101514994066004e-06 -sekretär 2.1063630766069476e-06 -w. 2.1063630766069476e-06 -▁einger 2.1063630766069476e-06 -▁horn 2.1063630766069476e-06 -entially 2.1025746538072947e-06 -othic 2.1025746538072947e-06 -▁Eigent 2.1025746538072947e-06 -frecht 2.0987862310076422e-06 -ienz 2.0987862310076422e-06 -zusagen 2.0987862310076422e-06 -▁Verteid 2.0987862310076422e-06 -▁Völk 2.0987862310076422e-06 -▁cele 2.0987862310076422e-06 -▁forb 2.0987862310076422e-06 -OTEL 2.0949978082079893e-06 -ţ 2.0949978082079893e-06 -▁bearbeit 2.0949978082079893e-06 -▁Handt 2.0912093854083364e-06 -inens 2.0874209626086835e-06 -▁Passag 2.0874209626086835e-06 -▁hosp 2.0798441170093782e-06 -▁ingred 2.0798441170093782e-06 -ublic 2.0760556942097253e-06 -▁Wei 2.0760556942097253e-06 -▁geforder 2.0760556942097253e-06 -yers 2.0722672714100724e-06 -▁Herange 2.0722672714100724e-06 -▁abzul 2.0722672714100724e-06 -▁entscheid 2.0722672714100724e-06 -▁sensib 2.0722672714100724e-06 -plates 2.06847884861042e-06 -uerung 2.06847884861042e-06 -▁Anges 2.06847884861042e-06 -▁Präsid 2.06847884861042e-06 -frastruct 2.064690425810767e-06 -leichen 2.064690425810767e-06 -ounter 2.064690425810767e-06 -▁Situ 2.064690425810767e-06 -▁enor 2.064690425810767e-06 -ague 2.0609020030111141e-06 -estag 2.0609020030111141e-06 -▁lar 2.0609020030111141e-06 -alition 2.0533251574118088e-06 -burgh 2.0533251574118088e-06 -chitekt 2.0533251574118088e-06 -eleist 2.0533251574118088e-06 -▁Colomb 2.0533251574118088e-06 -▁Gree 2.0533251574118088e-06 -▁Grie 2.0533251574118088e-06 -▁Kämp 2.0533251574118088e-06 -▁Migr 2.0533251574118088e-06 -holm 2.0495367346121559e-06 -stüt 2.0495367346121559e-06 -▁Freib 2.0495367346121559e-06 -ellence 2.045748311812503e-06 -sequ 2.045748311812503e-06 -ilton 2.04195988901285e-06 -hende 2.0381714662131976e-06 -ändert 2.0381714662131976e-06 -▁prere 2.0381714662131976e-06 -ürft 2.0305946206138918e-06 -▁Hekt 2.0305946206138918e-06 -▁Sog 2.0305946206138918e-06 -melden 2.026806197814239e-06 -▁betracht 2.026806197814239e-06 -▁geprü 2.026806197814239e-06 -MU 2.0230177750145865e-06 -▁detaill 2.0230177750145865e-06 -eks 2.0192293522149336e-06 -heimer 2.0192293522149336e-06 -icien 2.0192293522149336e-06 -ifikationen 2.0192293522149336e-06 -uant 2.0192293522149336e-06 -ärkt 2.0192293522149336e-06 -▁concess 2.0192293522149336e-06 -▁Voy 2.0154409294152807e-06 -▁mist 2.0154409294152807e-06 -▁parl 2.0154409294152807e-06 -chron 2.0116525066156282e-06 -earbeitet 2.0116525066156282e-06 -imension 2.0116525066156282e-06 -strö 2.0116525066156282e-06 -ungsvertrag 2.0116525066156282e-06 -▁allmäh 2.0116525066156282e-06 -▁cla 2.0116525066156282e-06 -▁traged 2.0116525066156282e-06 -ehler 2.0078640838159753e-06 -ugang 2.0078640838159753e-06 -▁inade 2.0078640838159753e-06 -▁releas 2.0078640838159753e-06 -▁uneingesch 2.0078640838159753e-06 -effizienz 2.0040756610163225e-06 -¡ 2.0040756610163225e-06 -▁Beam 2.0040756610163225e-06 -▁bathr 2.0040756610163225e-06 -utation 2.0002872382166696e-06 -▁Bib 2.0002872382166696e-06 -iefert 1.9964988154170171e-06 -ocument 1.9964988154170171e-06 -ständigkeit 1.9964988154170171e-06 -و 1.9964988154170171e-06 -▁Brü 1.9964988154170171e-06 -▁unabding 1.9964988154170171e-06 -ladung 1.9927103926173642e-06 -▁fab 1.9927103926173642e-06 -▁landscap 1.9927103926173642e-06 --country 1.9889219698177113e-06 -▁24-Stunden 1.9889219698177113e-06 -▁Kaff 1.9889219698177113e-06 -▁ful 1.9889219698177113e-06 -▁größ 1.9889219698177113e-06 -emony 1.9851335470180584e-06 -gehör 1.9851335470180584e-06 -tritts 1.9851335470180584e-06 -▁erneuer 1.9851335470180584e-06 -▁vot 1.9851335470180584e-06 -VO 1.9813451242184059e-06 -self 1.9813451242184059e-06 -uces 1.9813451242184059e-06 -▁eB 1.9813451242184059e-06 --Gen 1.9775567014187531e-06 -▁Bereit 1.9775567014187531e-06 -▁Educ 1.9775567014187531e-06 -▁NAT 1.9775567014187531e-06 -▁Similar 1.9775567014187531e-06 -▁umfang 1.9775567014187531e-06 -fg 1.9737682786191e-06 -gä 1.9737682786191e-06 -ült 1.9737682786191e-06 -▁Guant 1.9737682786191e-06 -▁Hüg 1.9737682786191e-06 -▁forts 1.9737682786191e-06 -▁unterst 1.9737682786191e-06 -Í 1.9699798558194473e-06 -ägigen 1.9699798558194473e-06 -▁Befürwort 1.9699798558194473e-06 -apse 1.9661914330197948e-06 -▁appell 1.9661914330197948e-06 -▁fabr 1.9661914330197948e-06 --Bahn 1.9624030102201419e-06 --party 1.9624030102201419e-06 -eldorf 1.9624030102201419e-06 -meld 1.9624030102201419e-06 -ή 1.9624030102201419e-06 -ت 1.9624030102201419e-06 -▁mov 1.9624030102201419e-06 -▁schol 1.9624030102201419e-06 -letter 1.958614587420489e-06 -▁dens 1.958614587420489e-06 -▁verwend 1.958614587420489e-06 -▁zwing 1.958614587420489e-06 -ademy 1.9548261646208361e-06 -iscal 1.9548261646208361e-06 -▁Landwir 1.9548261646208361e-06 -wunsch 1.9510377418211837e-06 -▁Huss 1.9510377418211837e-06 -junktur 1.9472493190215308e-06 -okol 1.9472493190215308e-06 -osev 1.9472493190215308e-06 -zeichnis 1.9472493190215308e-06 -zufinden 1.9472493190215308e-06 -▁ey 1.9472493190215308e-06 -▁wün 1.9472493190215308e-06 -stitutional 1.9434608962218779e-06 -▁Betroffen 1.9434608962218779e-06 -▁consc 1.9434608962218779e-06 -▁continu 1.9434608962218779e-06 -▁rhetor 1.9434608962218779e-06 -hole 1.939672473422225e-06 -laubnis 1.939672473422225e-06 -▁accomp 1.939672473422225e-06 -▁heut 1.939672473422225e-06 -▁sensit 1.939672473422225e-06 -ifice 1.9358840506225725e-06 -έ 1.9358840506225725e-06 -schra 1.9283072050232667e-06 -iebung 1.9245187822236138e-06 -ї 1.9245187822236138e-06 -▁Krie 1.9245187822236138e-06 -▁Holid 1.9207303594239614e-06 -▁Má 1.9207303594239614e-06 -▁Unterb 1.9207303594239614e-06 -iglich 1.9169419366243085e-06 -lässlich 1.9169419366243085e-06 -party 1.9131535138246556e-06 -▁Kosmet 1.9131535138246556e-06 -▁Stef 1.9131535138246556e-06 -▁gall 1.9131535138246556e-06 -quate 1.9093650910250027e-06 -▁caus 1.9093650910250027e-06 -▁konstrukt 1.9093650910250027e-06 -▁Salv 1.90557666822535e-06 -▁Sebast 1.90557666822535e-06 -ätigung 1.9017882454256973e-06 -▁tak 1.9017882454256973e-06 -▁untern 1.9017882454256973e-06 -amental 1.8979998226260445e-06 -ashion 1.8979998226260445e-06 -fuhr 1.8979998226260445e-06 -äsche 1.8979998226260445e-06 -▁Solidar 1.8979998226260445e-06 -▁chann 1.8979998226260445e-06 -▁transpar 1.8979998226260445e-06 -chst 1.8942113998263918e-06 -ustellen 1.8942113998263918e-06 -▁Gastfreund 1.8942113998263918e-06 -dates 1.8904229770267389e-06 -gument 1.8904229770267389e-06 -undred 1.8904229770267389e-06 -▁dement 1.8904229770267389e-06 -beschluss 1.8866345542270862e-06 -rungen 1.8866345542270862e-06 -▁Hosp 1.8866345542270862e-06 -▁Ministerr 1.8866345542270862e-06 -▁bedür 1.8866345542270862e-06 -örden 1.8828461314274333e-06 -▁TAC 1.8828461314274333e-06 -▁plann 1.8828461314274333e-06 -▁predecess 1.8828461314274333e-06 -▁Dub 1.8790577086277806e-06 -▁Zür 1.8790577086277806e-06 -▁sogen 1.8790577086277806e-06 --Amer 1.8752692858281277e-06 -fält 1.8752692858281277e-06 -ährend 1.8752692858281277e-06 -▁Fok 1.8752692858281277e-06 -usammen 1.8714808630284751e-06 -▁diox 1.8714808630284751e-06 -▁drafts 1.8714808630284751e-06 -▁intermedi 1.8714808630284751e-06 -▁plast 1.8714808630284751e-06 -▁Erne 1.8676924402288222e-06 -liederung 1.8639040174291695e-06 -renzung 1.8639040174291695e-06 -▁Tragöd 1.8639040174291695e-06 -▁einbe 1.8639040174291695e-06 -4-0 1.8601155946295166e-06 -hunder 1.8601155946295166e-06 -▁Steuerzahl 1.8601155946295166e-06 -author 1.8563271718298639e-06 -ceived 1.8563271718298639e-06 -stimmung 1.8563271718298639e-06 -ب 1.8563271718298639e-06 -▁bein 1.8563271718298639e-06 -▁beob 1.8563271718298639e-06 -▁Superv 1.852538749030211e-06 -▁cancell 1.852538749030211e-06 -.20 1.8487503262305583e-06 -astery 1.8487503262305583e-06 -abile 1.8449619034309055e-06 -amber 1.8449619034309055e-06 -findung 1.8449619034309055e-06 -▁adap 1.8449619034309055e-06 -▁oppos 1.8411734806312528e-06 -ifferenz 1.8373850578315999e-06 -▁Malays 1.8373850578315999e-06 -▁measur 1.8373850578315999e-06 -erhalten 1.8335966350319472e-06 -▁Mug 1.8335966350319472e-06 -▁refurb 1.8335966350319472e-06 -fi 1.8335966350319472e-06 -institutional 1.8298082122322945e-06 -äußer 1.8298082122322945e-06 -А 1.8298082122322945e-06 -▁frü 1.8298082122322945e-06 -▁oste 1.8298082122322945e-06 -▁stärk 1.8298082122322945e-06 -▁Besicht 1.8260197894326416e-06 -▁Tos 1.8260197894326416e-06 -▁dez 1.8260197894326416e-06 -▁panoram 1.8260197894326416e-06 -ctic 1.822231366632989e-06 -kompeten 1.822231366632989e-06 -ronic 1.822231366632989e-06 -terr 1.822231366632989e-06 -▁Freit 1.822231366632989e-06 -▁Gefü 1.822231366632989e-06 -▁Wachstumsp 1.822231366632989e-06 -▁tast 1.822231366632989e-06 -itteil 1.8184429438333361e-06 -▁Ungar 1.8184429438333361e-06 -▁kern 1.8184429438333361e-06 -geschritten 1.8146545210336834e-06 -ompl 1.8146545210336834e-06 -▁Sehens 1.8146545210336834e-06 -▁erweit 1.8146545210336834e-06 -fahrens 1.8108660982340305e-06 -älfte 1.8108660982340305e-06 -▁Energiet 1.8108660982340305e-06 -▁NCC 1.8108660982340305e-06 -vec 1.8070776754343778e-06 -▁Struct 1.8070776754343778e-06 -anies 1.8032892526347249e-06 -formance 1.8032892526347249e-06 -М 1.8032892526347249e-06 -▁Breit 1.8032892526347249e-06 -▁Exhib 1.8032892526347249e-06 -▁Verstär 1.8032892526347249e-06 -▁guilt 1.8032892526347249e-06 -unruh 1.7995008298350722e-06 -aupt 1.7957124070354193e-06 -fakt 1.7957124070354193e-06 -▁Ergeb 1.7957124070354193e-06 -▁Vermarkt 1.7957124070354193e-06 -▁Verschm 1.7957124070354193e-06 -▁Wäsch 1.7957124070354193e-06 -▁geleg 1.7957124070354193e-06 -▁verteid 1.7957124070354193e-06 -ansion 1.7919239842357667e-06 -grenzung 1.7919239842357667e-06 -roat 1.7919239842357667e-06 -▁erkl 1.7919239842357667e-06 -stuffs 1.7881355614361138e-06 -▁Schokol 1.7881355614361138e-06 -▁Än 1.7881355614361138e-06 -alking 1.7843471386364611e-06 -fassungen 1.7843471386364611e-06 -itious 1.7843471386364611e-06 -nex 1.7843471386364611e-06 -zwischen 1.7843471386364611e-06 -ľ 1.7843471386364611e-06 -Ł 1.7843471386364611e-06 -▁eigent 1.7843471386364611e-06 -overn 1.7805587158368082e-06 -Ÿ 1.7805587158368082e-06 -▁Schott 1.7805587158368082e-06 -elef 1.7767702930371555e-06 -▁forec 1.7767702930371555e-06 -▁quar 1.7767702930371555e-06 -▁reven 1.7767702930371555e-06 -flanzen 1.7729818702375026e-06 -▁Jugoslaw 1.7729818702375026e-06 -▁überflüss 1.7729818702375026e-06 -ittee 1.76919344743785e-06 -krie 1.76919344743785e-06 -propri 1.76919344743785e-06 -▁Befür 1.76919344743785e-06 -▁Dres 1.76919344743785e-06 -örp 1.7654050246381971e-06 -▁Glet 1.7654050246381971e-06 -▁capac 1.7654050246381971e-06 -brid 1.7616166018385444e-06 -▁Befrei 1.7616166018385444e-06 -▁headquar 1.7616166018385444e-06 -ergabe 1.7578281790388915e-06 -▁Frühl 1.7578281790388915e-06 -▁Kapaz 1.7578281790388915e-06 -▁attent 1.7578281790388915e-06 -ē 1.7540397562392388e-06 -▁Vorred 1.7540397562392388e-06 -▁Zigar 1.7540397562392388e-06 -▁schwier 1.7540397562392388e-06 -ateg 1.7502513334395859e-06 -▁Dubl 1.7502513334395859e-06 -▁telecommun 1.7464629106399332e-06 -▁verfü 1.7464629106399332e-06 -avor 1.7426744878402803e-06 -belast 1.7426744878402803e-06 -imedia 1.7426744878402803e-06 -würdigkeit 1.7426744878402803e-06 -▁Getre 1.7426744878402803e-06 -▁haupts 1.7426744878402803e-06 -▁kompat 1.7426744878402803e-06 -elegenheit 1.7388860650406277e-06 -junkt 1.7388860650406277e-06 -lafen 1.7388860650406277e-06 -▁wünsch 1.7388860650406277e-06 --house 1.7350976422409748e-06 -ipfel 1.7350976422409748e-06 -▁Autover 1.7350976422409748e-06 -▁kolle 1.7350976422409748e-06 -▁Cong 1.7313092194413221e-06 -▁Erweit 1.7313092194413221e-06 -▁Karr 1.7313092194413221e-06 -lights 1.7275207966416692e-06 -spartei 1.7275207966416692e-06 -▁gig 1.7275207966416692e-06 -ampag 1.7237323738420165e-06 -Ś 1.7237323738420165e-06 -‑ 1.7237323738420165e-06 -▁Cub 1.7237323738420165e-06 -▁atmosph 1.7237323738420165e-06 -cohol 1.7199439510423636e-06 -ötet 1.7199439510423636e-06 -▁beherr 1.7199439510423636e-06 -▁unnecess 1.7199439510423636e-06 -kaufs 1.716155528242711e-06 -uters 1.716155528242711e-06 -feit 1.7123671054430581e-06 -▁Verfü 1.7123671054430581e-06 -ergest 1.7085786826434054e-06 -ächtigungen 1.7085786826434054e-06 -▁Hindern 1.7085786826434054e-06 -▁parlament 1.7047902598437525e-06 -▁Satur 1.7010018370440998e-06 -▁berühm 1.7010018370440998e-06 -▁ordnungs 1.7010018370440998e-06 -▁wür 1.7010018370440998e-06 -▁Büg 1.6972134142444469e-06 -▁bekräft 1.6972134142444469e-06 -schefs 1.6934249914447942e-06 -ächtnis 1.6934249914447942e-06 -▁Schicks 1.6934249914447942e-06 -facht 1.6896365686451413e-06 -igital 1.6858481458454887e-06 -Ñ 1.6858481458454887e-06 -▁dim 1.6858481458454887e-06 -ixels 1.6820597230458358e-06 -ought 1.6820597230458358e-06 -verter 1.6820597230458358e-06 -▁jün 1.6820597230458358e-06 -lehnung 1.6782713002461831e-06 -▁tack 1.6782713002461831e-06 -▁Überein 1.6782713002461831e-06 -half 1.6744828774465302e-06 -viously 1.6744828774465302e-06 -Т 1.6744828774465302e-06 -▁Zag 1.6744828774465302e-06 -▁Rückt 1.6706944546468775e-06 -▁Face 1.6669060318472246e-06 -▁pea 1.6669060318472246e-06 -owder 1.663117609047572e-06 -▁imposs 1.663117609047572e-06 -ugnisse 1.6593291862479191e-06 -▁overwhelm 1.6593291862479191e-06 -lys 1.6555407634482664e-06 -ω 1.6555407634482664e-06 -▁Sev 1.6555407634482664e-06 -imination 1.6517523406486135e-06 -lements 1.6517523406486135e-06 -ografen 1.6517523406486135e-06 -rechung 1.6517523406486135e-06 -θ 1.6517523406486135e-06 -isy 1.6479639178489608e-06 -lever 1.6479639178489608e-06 -agnet 1.6441754950493081e-06 -itness 1.6441754950493081e-06 -zessen 1.6441754950493081e-06 -.j 1.6403870722496552e-06 -entren 1.6403870722496552e-06 -▁Result 1.6403870722496552e-06 -▁estim 1.6403870722496552e-06 -ERV 1.6365986494500026e-06 -opped 1.6365986494500026e-06 -uels 1.6365986494500026e-06 -ältigung 1.6365986494500026e-06 -▁Empfeh 1.6365986494500026e-06 -athed 1.6328102266503497e-06 -berechtigung 1.6328102266503497e-06 -rease 1.6328102266503497e-06 -änn 1.6328102266503497e-06 -▁Immobil 1.6328102266503497e-06 -▁verring 1.6328102266503497e-06 --2013 1.629021803850697e-06 -ancial 1.629021803850697e-06 -ittees 1.629021803850697e-06 -ď 1.629021803850697e-06 -▁flexib 1.629021803850697e-06 -▁lik 1.629021803850697e-06 -▁wal 1.629021803850697e-06 --mon 1.6252333810510441e-06 -ansprü 1.6252333810510441e-06 -▁bürger 1.6252333810510441e-06 -atum 1.6214449582513914e-06 -ehol 1.6214449582513914e-06 -▁Venezu 1.6214449582513914e-06 -▁bestimm 1.6214449582513914e-06 -sq 1.6176565354517385e-06 -▁Veröffent 1.6176565354517385e-06 -▁Kurd 1.6138681126520858e-06 -▁artific 1.6138681126520858e-06 -▁befürwort 1.6138681126520858e-06 -▁hors 1.6138681126520858e-06 -ailable 1.610079689852433e-06 -cluding 1.610079689852433e-06 -ärme 1.610079689852433e-06 -▁Ug 1.610079689852433e-06 -ollah 1.6062912670527803e-06 -▁restr 1.6062912670527803e-06 -▁Errungen 1.6025028442531274e-06 -▁Lös 1.6025028442531274e-06 -▁Män 1.6025028442531274e-06 -geordn 1.5987144214534747e-06 -ieties 1.5949259986538218e-06 -mine 1.5949259986538218e-06 -▁US- 1.5949259986538218e-06 -▁chea 1.5949259986538218e-06 -▁stattf 1.5949259986538218e-06 -bereitet 1.5911375758541691e-06 -ceed 1.5911375758541691e-06 -ibet 1.5911375758541691e-06 -▁Erhol 1.5911375758541691e-06 -▁Inkraft 1.5911375758541691e-06 -▁Sauber 1.5911375758541691e-06 -▁doub 1.5911375758541691e-06 -▁geänder 1.5911375758541691e-06 -lossen 1.5835607302548636e-06 -▁Champ 1.5835607302548636e-06 -▁Frid 1.5835607302548636e-06 -▁Slov 1.5835607302548636e-06 -▁trem 1.5835607302548636e-06 -▁Öffent 1.5835607302548636e-06 -azzo 1.5797723074552107e-06 -bewerber 1.5797723074552107e-06 -government 1.5797723074552107e-06 -onger 1.5797723074552107e-06 -spolit 1.5797723074552107e-06 -▁environ 1.5797723074552107e-06 -▁jed 1.5797723074552107e-06 --Leist 1.575983884655558e-06 -bea 1.575983884655558e-06 -flüss 1.575983884655558e-06 -▁Ruß 1.575983884655558e-06 -▁Ub 1.575983884655558e-06 -hrter 1.5721954618559051e-06 -▁Eas 1.5721954618559051e-06 -▁Lith 1.5721954618559051e-06 -▁transe 1.5721954618559051e-06 -zession 1.5684070390562524e-06 -economic 1.5646186162565995e-06 -erner 1.5646186162565995e-06 -graben 1.5646186162565995e-06 -ipment 1.5646186162565995e-06 -zep 1.5646186162565995e-06 -▁Umst 1.5646186162565995e-06 -▁prestig 1.5646186162565995e-06 -ehold 1.5608301934569468e-06 -Ú 1.5608301934569468e-06 -▁CD-R 1.5608301934569468e-06 -▁hazard 1.5608301934569468e-06 -epp 1.557041770657294e-06 -▁Darle 1.557041770657294e-06 -▁Hoheits 1.557041770657294e-06 -▁Stor 1.557041770657294e-06 -▁mun 1.557041770657294e-06 -ipend 1.5532533478576413e-06 -▁Adri 1.5532533478576413e-06 -▁begrenz 1.5532533478576413e-06 -jenigen 1.5494649250579884e-06 -stahl 1.5494649250579884e-06 -▁erforder 1.5494649250579884e-06 -ategie 1.5456765022583357e-06 -quem 1.5456765022583357e-06 -Ε 1.5456765022583357e-06 -▁Buen 1.5456765022583357e-06 -▁Ankünd 1.5418880794586828e-06 -cano 1.5380996566590301e-06 -chit 1.5380996566590301e-06 -folgerung 1.5380996566590301e-06 -ibar 1.5380996566590301e-06 -schlägt 1.5380996566590301e-06 -▁Artik 1.5380996566590301e-06 -▁zusammengef 1.5380996566590301e-06 -▁Gült 1.5343112338593772e-06 -▁anzup 1.5343112338593772e-06 -itage 1.5305228110597246e-06 -medi 1.5305228110597246e-06 -where 1.5305228110597246e-06 -bres 1.522945965460419e-06 -scheidung 1.522945965460419e-06 -▁Anwend 1.522945965460419e-06 -▁Donner 1.522945965460419e-06 -▁Mü 1.522945965460419e-06 -▁herv 1.522945965460419e-06 -▁recruit 1.522945965460419e-06 -affin 1.5191575426607661e-06 --Out 1.5153691198611134e-06 -olidierung 1.5153691198611134e-06 -äußerung 1.5153691198611134e-06 -د 1.5153691198611134e-06 -▁Sorg 1.5153691198611134e-06 -▁kop 1.5153691198611134e-06 -enntnisse 1.5115806970614605e-06 -ould 1.5115806970614605e-06 -venue 1.5115806970614605e-06 -▁Vize 1.5115806970614605e-06 -▁struct 1.5115806970614605e-06 -eleistungen 1.5077922742618078e-06 -▁Erwäg 1.5077922742618078e-06 -▁Niem 1.5077922742618078e-06 -kerung 1.504003851462155e-06 -vielfält 1.504003851462155e-06 -vious 1.504003851462155e-06 -zuarbeiten 1.504003851462155e-06 -▁gesam 1.504003851462155e-06 -itiveness 1.5002154286625023e-06 -positions 1.4964270058628494e-06 -ulle 1.4964270058628494e-06 -▁unterstütz 1.4964270058628494e-06 -spruchs 1.4926385830631967e-06 -ätzung 1.4926385830631967e-06 -▁gült 1.4926385830631967e-06 -wärt 1.4888501602635438e-06 -▁prel 1.4888501602635438e-06 -▁stren 1.4850617374638911e-06 -▁tsch 1.4850617374638911e-06 -hend 1.4812733146642382e-06 -ъ 1.4812733146642382e-06 -▁palästinens 1.4812733146642382e-06 -eatre 1.4774848918645856e-06 -idential 1.4774848918645856e-06 -zil 1.4774848918645856e-06 -▁kat 1.4774848918645856e-06 -orama 1.4736964690649327e-06 -ôtel 1.4736964690649327e-06 -ع 1.4736964690649327e-06 -▁zukunft 1.4736964690649327e-06 -beweg 1.46990804626528e-06 -egal 1.46990804626528e-06 -leisten 1.46990804626528e-06 -▁ICE 1.46990804626528e-06 -▁disapp 1.46990804626528e-06 -befehl 1.4661196234656273e-06 -uisine 1.4661196234656273e-06 -zusammenarbeit 1.4661196234656273e-06 -▁Diplomat 1.4661196234656273e-06 -▁Rind 1.4661196234656273e-06 -▁rehab 1.4661196234656273e-06 -oub 1.4623312006659744e-06 -überwach 1.4623312006659744e-06 -‹ 1.4623312006659744e-06 -▁Scot 1.4623312006659744e-06 -▁innoc 1.4623312006659744e-06 -▁lok 1.4623312006659744e-06 -▁müs 1.4623312006659744e-06 -helm 1.4585427778663217e-06 -▁Cult 1.4585427778663217e-06 -▁archae 1.4585427778663217e-06 -▁schwar 1.4585427778663217e-06 -▁abwechs 1.4547543550666688e-06 -iterranean 1.4509659322670162e-06 -▁unters 1.4509659322670162e-06 -uzzi 1.4471775094673633e-06 -▁überprü 1.4471775094673633e-06 -ratt 1.4433890866677106e-06 -sieht 1.4433890866677106e-06 -▁Sark 1.4433890866677106e-06 -▁Ausser 1.4396006638680577e-06 -eace 1.435812241068405e-06 -▁Maced 1.435812241068405e-06 -▁Nür 1.435812241068405e-06 -▁tow 1.435812241068405e-06 -geschneider 1.4320238182687521e-06 -reate 1.4320238182687521e-06 -ädchen 1.4320238182687521e-06 -▁Demonstr 1.4320238182687521e-06 -▁Pens 1.4320238182687521e-06 -▁vorherr 1.4320238182687521e-06 -gerung 1.4282353954690995e-06 -▁mixt 1.4282353954690995e-06 -etschen 1.4244469726694466e-06 -cuss 1.4206585498697939e-06 -ürk 1.4206585498697939e-06 -ÿ 1.4206585498697939e-06 -▁Ereign 1.4206585498697939e-06 -▁Verfüg 1.4206585498697939e-06 -▁ade 1.4206585498697939e-06 -▁foll 1.4206585498697939e-06 -▁hairdr 1.4206585498697939e-06 -▁ingr 1.4206585498697939e-06 -racies 1.416870127070141e-06 -ufficient 1.416870127070141e-06 -▁luxur 1.416870127070141e-06 -▁Barros 1.4130817042704883e-06 -▁Infl 1.4130817042704883e-06 -▁Slowen 1.4130817042704883e-06 -▁intelle 1.4130817042704883e-06 -chung 1.4092932814708354e-06 -ettbewerbs 1.4092932814708354e-06 -stood 1.4092932814708354e-06 -▁Anze 1.4092932814708354e-06 -▁Komb 1.4092932814708354e-06 -▁MID 1.4092932814708354e-06 -▁destro 1.4092932814708354e-06 -▁earthqu 1.4092932814708354e-06 -▁erörter 1.4092932814708354e-06 -▁grund 1.4092932814708354e-06 --Ben 1.4055048586711827e-06 -etzungen 1.4055048586711827e-06 -▁amen 1.4055048586711827e-06 -▁gentlem 1.4055048586711827e-06 --Ebene 1.4017164358715298e-06 -enende 1.4017164358715298e-06 -omod 1.4017164358715298e-06 -ventura 1.4017164358715298e-06 -▁Belast 1.4017164358715298e-06 -apos 1.3979280130718772e-06 -eichert 1.3979280130718772e-06 -fft 1.3979280130718772e-06 -▁abziel 1.3979280130718772e-06 -limited 1.3941395902722243e-06 -sequently 1.3941395902722243e-06 -Д 1.3941395902722243e-06 -chnen 1.3903511674725716e-06 -hann 1.3903511674725716e-06 -holfen 1.3903511674725716e-06 -ränkung 1.3903511674725716e-06 -▁Gründ 1.3903511674725716e-06 -▁Stär 1.3903511674725716e-06 -▁financ 1.3903511674725716e-06 -▁soph 1.3903511674725716e-06 -ogether 1.3865627446729187e-06 -▁benöt 1.3865627446729187e-06 -etres 1.382774321873266e-06 -hyth 1.382774321873266e-06 -pruch 1.382774321873266e-06 -▁awa 1.382774321873266e-06 -▁gestr 1.382774321873266e-06 -▁schnellst 1.382774321873266e-06 -inians 1.3789858990736131e-06 -ortion 1.3789858990736131e-06 -▁Monteneg 1.3789858990736131e-06 -ambod 1.3751974762739605e-06 -eop 1.3751974762739605e-06 -س 1.3751974762739605e-06 -▁EAD 1.3751974762739605e-06 -▁Nachhalt 1.3751974762739605e-06 -▁Woch 1.3751974762739605e-06 -▁applic 1.3751974762739605e-06 -▁benachricht 1.3751974762739605e-06 -▁rhyth 1.3751974762739605e-06 -abon 1.3714090534743076e-06 -lechter 1.3714090534743076e-06 -osit 1.3676206306746549e-06 -▁fas 1.3676206306746549e-06 -emein 1.363832207875002e-06 -inkel 1.363832207875002e-06 -▁Challen 1.363832207875002e-06 -▁gepl 1.363832207875002e-06 -cow 1.3600437850753493e-06 -eben 1.3600437850753493e-06 -endous 1.3600437850753493e-06 -staben 1.3600437850753493e-06 -▁Foll 1.3600437850753493e-06 -▁tex 1.3600437850753493e-06 -atul 1.3562553622756964e-06 -haltet 1.3562553622756964e-06 -hrase 1.3524669394760437e-06 -▁Castr 1.3524669394760437e-06 -ibliothe 1.3486785166763908e-06 -χ 1.3486785166763908e-06 -▁Montr 1.3486785166763908e-06 -▁agre 1.3486785166763908e-06 -▁vorhand 1.3486785166763908e-06 -hrow 1.3448900938767382e-06 -upten 1.3448900938767382e-06 -yen 1.3448900938767382e-06 -ßig 1.3448900938767382e-06 -äsent 1.3448900938767382e-06 -Б 1.3448900938767382e-06 -ك 1.3448900938767382e-06 -▁fift 1.3448900938767382e-06 -upts 1.3411016710770853e-06 -Ћ 1.3411016710770853e-06 -▁Verknü 1.3411016710770853e-06 -▁Yo 1.3411016710770853e-06 -operations 1.3373132482774326e-06 -verschm 1.3373132482774326e-06 -▁magnific 1.3373132482774326e-06 -prün 1.3335248254777797e-06 -▁Veranst 1.3335248254777797e-06 -▁balcon 1.3335248254777797e-06 -▁inhabit 1.3335248254777797e-06 --discr 1.329736402678127e-06 -ervor 1.329736402678127e-06 -ließen 1.329736402678127e-06 -▁Gemeins 1.329736402678127e-06 -▁Propert 1.329736402678127e-06 -À 1.3259479798784741e-06 -äbe 1.3259479798784741e-06 -▁compat 1.3259479798784741e-06 -afft 1.3221595570788215e-06 -demark 1.3221595570788215e-06 -ibilities 1.3221595570788215e-06 -ivileg 1.3221595570788215e-06 -riendly 1.3221595570788215e-06 -▁besteh 1.3221595570788215e-06 -ponsible 1.3183711342791686e-06 -▁Gonz 1.3183711342791686e-06 -▁Rass 1.3145827114795159e-06 -▁verschär 1.3145827114795159e-06 -ulgar 1.3070058658802103e-06 -▁Softw 1.3070058658802103e-06 -IDS 1.3032174430805574e-06 -▁Gipf 1.3032174430805574e-06 -bourg 1.2994290202809047e-06 -ratic 1.2994290202809047e-06 -▁ECH 1.2994290202809047e-06 -▁Erinner 1.2994290202809047e-06 -▁preg 1.2994290202809047e-06 -ettbew 1.2956405974812518e-06 -overnment 1.2956405974812518e-06 -▁conclus 1.2956405974812518e-06 -▁disappoint 1.2956405974812518e-06 -inely 1.2918521746815992e-06 -▁EU-Bür 1.2918521746815992e-06 -▁champ 1.2918521746815992e-06 -▁wissenschaft 1.2918521746815992e-06 -nütz 1.2880637518819463e-06 -¬ 1.2880637518819463e-06 -▁constituen 1.2880637518819463e-06 -▁recreat 1.2880637518819463e-06 -▁xenophob 1.2880637518819463e-06 -▁ärm 1.2880637518819463e-06 -okoll 1.2842753290822936e-06 -▁Weiterb 1.2842753290822936e-06 -▁beteil 1.2842753290822936e-06 -▁corpor 1.2842753290822936e-06 -gery 1.2804869062826409e-06 -ymmetr 1.2804869062826409e-06 -▁htt 1.2804869062826409e-06 -▁deg 1.276698483482988e-06 -entscheid 1.2729100606833353e-06 -klusive 1.2729100606833353e-06 -prene 1.2729100606833353e-06 -▁Fuer 1.2729100606833353e-06 -▁Galile 1.2729100606833353e-06 -katastrophen 1.2691216378836825e-06 -uffs 1.2691216378836825e-06 -▁Erweiter 1.2653332150840298e-06 -▁Verab 1.2653332150840298e-06 -▁empt 1.2653332150840298e-06 -▁erhöh 1.2653332150840298e-06 -▁neighbor 1.2653332150840298e-06 -vollständ 1.2615447922843769e-06 -äischen 1.2615447922843769e-06 -ämt 1.2615447922843769e-06 -▁gelö 1.2615447922843769e-06 --Sicherheit 1.2577563694847242e-06 -hrough 1.2577563694847242e-06 -meid 1.2577563694847242e-06 -uttle 1.2577563694847242e-06 -▁Affili 1.2577563694847242e-06 -▁pedestr 1.2577563694847242e-06 -erkennen 1.2539679466850713e-06 -irol 1.2539679466850713e-06 -ledge 1.2539679466850713e-06 -▁fäh 1.2539679466850713e-06 -annehm 1.2501795238854186e-06 -▁Verme 1.2501795238854186e-06 -lieder 1.2426026782861131e-06 -÷ 1.2426026782861131e-06 -▁Gipfelt 1.2388142554864602e-06 -sequent 1.2350258326868075e-06 -▁Rei 1.2350258326868075e-06 -▁Vereinfach 1.2350258326868075e-06 -haftet 1.2312374098871546e-06 -ingesch 1.2312374098871546e-06 -▁Biok 1.2312374098871546e-06 -▁atember 1.2312374098871546e-06 -▁confl 1.2312374098871546e-06 -▁enshr 1.2312374098871546e-06 -▁niederländ 1.2312374098871546e-06 -▁verhin 1.2312374098871546e-06 -aceutical 1.2274489870875019e-06 -coni 1.2274489870875019e-06 -istency 1.2274489870875019e-06 -▁sincer 1.2274489870875019e-06 -apest 1.223660564287849e-06 -▁Swed 1.223660564287849e-06 --cent 1.2198721414881963e-06 -ästinens 1.2198721414881963e-06 -▁Fischereise 1.2198721414881963e-06 -▁PN 1.2198721414881963e-06 -▁concr 1.2198721414881963e-06 -▁veröffent 1.2198721414881963e-06 -eregel 1.2160837186885435e-06 -fortun 1.2160837186885435e-06 -indestens 1.2160837186885435e-06 -▁benachbar 1.2160837186885435e-06 -abrik 1.2122952958888908e-06 -▁rar 1.2122952958888908e-06 -zne 1.2085068730892379e-06 -▁Spaz 1.2085068730892379e-06 -akzept 1.2047184502895852e-06 -umenten 1.2047184502895852e-06 -verstanden 1.2047184502895852e-06 -▁wundersch 1.2047184502895852e-06 -gerichtshof 1.2009300274899323e-06 -ibraries 1.2009300274899323e-06 -ölf 1.2009300274899323e-06 -▁Allian 1.2009300274899323e-06 -ORK 1.1971416046902796e-06 -lywood 1.1971416046902796e-06 -ourc 1.1971416046902796e-06 --Bere 1.1933531818906267e-06 -icip 1.1933531818906267e-06 -idesh 1.1933531818906267e-06 -neider 1.1933531818906267e-06 -owth 1.1895647590909741e-06 -schwemm 1.1895647590909741e-06 -ź 1.1895647590909741e-06 -▁Citiz 1.1895647590909741e-06 -▁chap 1.1895647590909741e-06 --Date 1.1857763362913212e-06 -124 1.1857763362913212e-06 -ainless 1.1857763362913212e-06 -ready 1.1857763362913212e-06 -ueber 1.1857763362913212e-06 -drop 1.1819879134916685e-06 -eitigung 1.1819879134916685e-06 -▁Jahrhunder 1.1819879134916685e-06 -▁Lett 1.1819879134916685e-06 -▁Nev 1.1819879134916685e-06 -▁freige 1.1819879134916685e-06 -▁schwed 1.1819879134916685e-06 -achläss 1.1781994906920156e-06 -endum 1.1781994906920156e-06 -ordern 1.1781994906920156e-06 -rengungen 1.1781994906920156e-06 -tags 1.1781994906920156e-06 -zubauen 1.1781994906920156e-06 -äusern 1.1781994906920156e-06 -▁Ausnahm 1.1781994906920156e-06 -icide 1.1744110678923629e-06 -▁kurzfrist 1.1744110678923629e-06 -genöss 1.17062264509271e-06 -icop 1.17062264509271e-06 -kong 1.17062264509271e-06 -▁Heidel 1.17062264509271e-06 -▁Kolumb 1.17062264509271e-06 -▁Liss 1.17062264509271e-06 -▁independ 1.17062264509271e-06 -blas 1.1668342222930573e-06 -mechan 1.1668342222930573e-06 -ochter 1.1668342222930573e-06 -ocop 1.1668342222930573e-06 -sebene 1.1668342222930573e-06 -И 1.1668342222930573e-06 -mittelsicherheit 1.1630457994934045e-06 -zuteilen 1.1630457994934045e-06 -ürf 1.1630457994934045e-06 -▁hinzuf 1.1630457994934045e-06 -▁wur 1.1630457994934045e-06 -ktop 1.1592573766937518e-06 -Fi 1.1554689538940989e-06 -rale 1.1554689538940989e-06 -▁REA 1.1554689538940989e-06 -▁batter 1.1554689538940989e-06 -usal 1.1516805310944462e-06 -▁Bibliothe 1.1516805310944462e-06 -▁Kräft 1.1516805310944462e-06 -▁Opf 1.1516805310944462e-06 -▁accompan 1.1516805310944462e-06 -▁pup 1.1516805310944462e-06 -etus 1.1478921082947933e-06 -äsch 1.1478921082947933e-06 -▁199 1.1478921082947933e-06 --Vert 1.1441036854951406e-06 -▁Bör 1.1441036854951406e-06 -▁Nep 1.1441036854951406e-06 -▁adequ 1.1441036854951406e-06 -▁anzub 1.1441036854951406e-06 -indruck 1.1403152626954877e-06 -tends 1.1403152626954877e-06 -wers 1.1403152626954877e-06 -ongue 1.1365268398958351e-06 -Ѓ 1.1365268398958351e-06 -▁distin 1.1365268398958351e-06 -ileg 1.1327384170961822e-06 -verläss 1.1327384170961822e-06 -ť 1.1327384170961822e-06 -▁Giov 1.1327384170961822e-06 -▁Sou 1.1327384170961822e-06 -ubil 1.1289499942965295e-06 -akei 1.1251615714968766e-06 -esthet 1.1251615714968766e-06 -ocolate 1.1251615714968766e-06 -¥ 1.1251615714968766e-06 -öffnet 1.1251615714968766e-06 -▁Dän 1.1251615714968766e-06 -▁cir 1.1251615714968766e-06 -▁lear 1.1251615714968766e-06 -estelle 1.1213731486972239e-06 -folgung 1.1213731486972239e-06 -ternal 1.1213731486972239e-06 -strichen 1.117584725897571e-06 -▁deter 1.117584725897571e-06 -▁vene 1.117584725897571e-06 -acent 1.1137963030979183e-06 -igrants 1.1137963030979183e-06 -▁Mean 1.1137963030979183e-06 -gerungen 1.1100078802982655e-06 -▁Bav 1.1100078802982655e-06 -▁Blizz 1.1100078802982655e-06 -▁Erleicht 1.1100078802982655e-06 -ereinander 1.1062194574986128e-06 -▁administr 1.1062194574986128e-06 -▁portugies 1.1062194574986128e-06 -Α 1.1024310346989599e-06 -ehnen 1.0986426118993072e-06 -ksamkeit 1.0986426118993072e-06 -▁nuklear 1.0986426118993072e-06 -alties 1.0948541890996545e-06 -isson 1.0948541890996545e-06 -eiler 1.0910657663000016e-06 -utragen 1.0910657663000016e-06 -vre 1.0910657663000016e-06 -Ï 1.0910657663000016e-06 -ooth 1.087277343500349e-06 -ween 1.087277343500349e-06 -▁Erwä 1.087277343500349e-06 -▁Zimb 1.087277343500349e-06 -ernet 1.0834889207006961e-06 -wh 1.0834889207006961e-06 -agent 1.0797004979010434e-06 -leisch 1.0797004979010434e-06 -ronics 1.0797004979010434e-06 -▁Eiff 1.0797004979010434e-06 -▁bout 1.0797004979010434e-06 -▁juris 1.0797004979010434e-06 -armon 1.0759120751013905e-06 -geleg 1.0759120751013905e-06 -riet 1.0759120751013905e-06 -ة 1.0759120751013905e-06 -▁Tus 1.0759120751013905e-06 -asonic 1.0721236523017378e-06 -entrums 1.0721236523017378e-06 -ύ 1.0721236523017378e-06 -embly 1.0683352295020849e-06 -haust 1.0683352295020849e-06 -stitute 1.0683352295020849e-06 -iddle 1.0645468067024322e-06 -iforn 1.0645468067024322e-06 -pts 1.0645468067024322e-06 -¹ 1.0645468067024322e-06 -▁Unterdrück 1.0645468067024322e-06 -▁vorzusch 1.0645468067024322e-06 -beschäft 1.0607583839027793e-06 -essourc 1.0607583839027793e-06 -igebiet 1.0607583839027793e-06 -▁Proced 1.0607583839027793e-06 -▁vorgesch 1.0607583839027793e-06 -ethod 1.0569699611031267e-06 -idency 1.0569699611031267e-06 -reng 1.0569699611031267e-06 -Л 1.0569699611031267e-06 -▁Räum 1.0569699611031267e-06 -▁lic 1.0569699611031267e-06 -▁empfehl 1.0531815383034738e-06 -kei 1.0493931155038211e-06 -▁Cathol 1.0493931155038211e-06 -▁Simb 1.0493931155038211e-06 -▁Stutt 1.0493931155038211e-06 -▁Thail 1.0493931155038211e-06 -irma 1.0456046927041682e-06 -merken 1.0456046927041682e-06 -▁erfahr 1.0456046927041682e-06 -▁hus 1.0456046927041682e-06 -▁schall 1.0456046927041682e-06 -öglichkeit 1.0418162699045155e-06 -Î 1.0380278471048626e-06 -▁ungar 1.0380278471048626e-06 -bitr 1.03423942430521e-06 -ennis 1.03423942430521e-06 -reuung 1.03423942430521e-06 -zeß 1.03423942430521e-06 -▁avo 1.03423942430521e-06 -▁erwä 1.03423942430521e-06 -▁licens 1.03423942430521e-06 -ekut 1.0304510015055571e-06 -gart 1.0304510015055571e-06 -usalem 1.0304510015055571e-06 -▁Ecu 1.0304510015055571e-06 -▁Sehenswürd 1.0304510015055571e-06 -etragen 1.0266625787059044e-06 -feiert 1.0266625787059044e-06 -landen 1.0266625787059044e-06 -▁Bemerk 1.0266625787059044e-06 -▁Entsteh 1.0266625787059044e-06 -▁pestic 1.0266625787059044e-06 -▁ult 1.0266625787059044e-06 -orse 1.0228741559062515e-06 -windigkeit 1.0228741559062515e-06 -▁197 1.0228741559062515e-06 -▁Grem 1.0228741559062515e-06 -▁Recy 1.0228741559062515e-06 -▁compuls 1.0228741559062515e-06 -È 1.0190857331065988e-06 -▁Euror 1.0190857331065988e-06 -▁Leip 1.0190857331065988e-06 -jing 1.0152973103069459e-06 -▁Janu 1.0152973103069459e-06 -merz 1.0115088875072932e-06 -valu 1.0115088875072932e-06 -▁Souverän 1.0115088875072932e-06 -Ê 1.0077204647076404e-06 -▁Verhe 1.0077204647076404e-06 -▁offic 1.0077204647076404e-06 -liegt 1.0039320419079877e-06 -städ 1.0039320419079877e-06 -▁Emot 1.0039320419079877e-06 -▁Ere 1.0039320419079877e-06 -kfast 1.0001436191083348e-06 -Ý 1.0001436191083348e-06 -▁Marok 1.0001436191083348e-06 -▁Wahrnehm 1.0001436191083348e-06 -▁Archit 9.96355196308682e-07 -▁verschied 9.96355196308682e-07 -φ 9.9256677350902921e-07 -▁Ansprech 9.9256677350902921e-07 -▁gefähr 9.9256677350902921e-07 -akul 9.8877835070937653e-07 -arly 9.8877835070937653e-07 -gehens 9.8877835070937653e-07 -▁FIF 9.8877835070937653e-07 -eils 9.8498992790972364e-07 -untu 9.8498992790972364e-07 -▁Beschleun 9.8498992790972364e-07 -▁elder 9.8498992790972364e-07 -ansa 9.81201505110071e-07 -ibeh 9.81201505110071e-07 -▁Pressem 9.81201505110071e-07 -▁simult 9.81201505110071e-07 --in-Office 9.7362465951076539e-07 -allery 9.7362465951076539e-07 -condition 9.7362465951076539e-07 -immung 9.7362465951076539e-07 -▁198 9.7362465951076539e-07 -▁ehemal 9.7362465951076539e-07 -ه 9.6983623671111249e-07 -▁Enttäus 9.6983623671111249e-07 -▁ausgestatt 9.6983623671111249e-07 -▁beru 9.6983623671111249e-07 -י 9.6604781391145981e-07 -▁part-s 9.6604781391145981e-07 --Stunden 9.62259391111807e-07 -iversary 9.62259391111807e-07 -▁misl 9.62259391111807e-07 -oftw 9.5847096831215424e-07 -▁Erfahr 9.5847096831215424e-07 -▁Seat 9.5847096831215424e-07 -▁Tamp 9.5847096831215424e-07 -▁biof 9.5847096831215424e-07 -bsp 9.5468254551250135e-07 -gypt 9.5468254551250135e-07 -inational 9.5468254551250135e-07 -kandidaten 9.5468254551250135e-07 -rankungen 9.5468254551250135e-07 -wieweit 9.5468254551250135e-07 -ändler 9.5468254551250135e-07 -˙ 9.5468254551250135e-07 -▁els 9.5468254551250135e-07 -▁ohn 9.5468254551250135e-07 -▁unentgelt 9.5468254551250135e-07 -empts 9.5089412271284867e-07 -▁Tast 9.5089412271284867e-07 -▁ine 9.5089412271284867e-07 -amentos 9.4710569991319589e-07 -oram 9.4710569991319589e-07 -▁aftern 9.4710569991319589e-07 -azine 9.433172771135431e-07 -cepts 9.433172771135431e-07 -▁Vergangen 9.433172771135431e-07 -▁urs 9.433172771135431e-07 -lauber 9.3952885431389031e-07 -urteilung 9.3952885431389031e-07 -▁Anp 9.3952885431389031e-07 -▁ausreich 9.3952885431389031e-07 -▁berücksicht 9.3952885431389031e-07 -▁princ 9.3952885431389031e-07 -ethe 9.3574043151423753e-07 -isko 9.3574043151423753e-07 -▁Rena 9.3574043151423753e-07 --Dollar 9.3195200871458474e-07 -▁Veränder 9.3195200871458474e-07 -▁precaution 9.3195200871458474e-07 --of-the- 9.28163585914932e-07 -gehoben 9.28163585914932e-07 -ف 9.28163585914932e-07 -▁Hels 9.28163585914932e-07 -▁Omb 9.28163585914932e-07 -▁furnish 9.28163585914932e-07 -▁getest 9.28163585914932e-07 -eständ 9.2437516311527917e-07 -ypern 9.2437516311527917e-07 -Ż 9.2437516311527917e-07 -▁endgült 9.2437516311527917e-07 -andidaten 9.2058674031562639e-07 -hesion 9.2058674031562639e-07 -prechung 9.2058674031562639e-07 -scheinend 9.2058674031562639e-07 -▁Proz 9.2058674031562639e-07 -▁multil 9.2058674031562639e-07 -arsh 9.167983175159736e-07 -є 9.167983175159736e-07 -▁Nether 9.167983175159736e-07 -▁phenomen 9.167983175159736e-07 -▁sozio 9.167983175159736e-07 -giene 9.1300989471632081e-07 -ozy 9.1300989471632081e-07 -vertreten 9.1300989471632081e-07 -¨ 9.1300989471632081e-07 -أ 9.1300989471632081e-07 -▁Dring 9.1300989471632081e-07 -▁Viet 9.1300989471632081e-07 -▁personen 9.1300989471632081e-07 -ambres 9.09221471916668e-07 -pread 9.09221471916668e-07 -verages 9.09221471916668e-07 -У 9.09221471916668e-07 -▁Maßst 9.09221471916668e-07 -▁convin 9.09221471916668e-07 -▁korrig 9.09221471916668e-07 -knü 9.0543304911701524e-07 -rebungen 9.0543304911701524e-07 -vironment 9.0543304911701524e-07 -▁Addition 9.0543304911701524e-07 -▁intu 9.0543304911701524e-07 -amik 9.0164462631736246e-07 -leys 9.0164462631736246e-07 -▁Vierten 9.0164462631736246e-07 -▁gratul 9.0164462631736246e-07 --Pres 8.9785620351770967e-07 --how 8.9785620351770967e-07 -areness 8.9785620351770967e-07 -ermo 8.9785620351770967e-07 -ranspar 8.9785620351770967e-07 -ängerung 8.9785620351770967e-07 -З 8.9785620351770967e-07 -▁behavi 8.9785620351770967e-07 -▁widersp 8.9785620351770967e-07 -enary 8.9406778071805689e-07 -zuheben 8.9406778071805689e-07 -▁Brutto 8.9406778071805689e-07 -▁Ergän 8.9406778071805689e-07 -▁Mitch 8.9406778071805689e-07 -▁orig 8.9406778071805689e-07 -adian 8.902793579184041e-07 -enness 8.902793579184041e-07 -idance 8.902793579184041e-07 -wärtig 8.902793579184041e-07 -▁Ernen 8.902793579184041e-07 -▁obst 8.902793579184041e-07 -▁thumbn 8.902793579184041e-07 -▁Überw 8.902793579184041e-07 -quot 8.8649093511875132e-07 -zicht 8.8649093511875132e-07 -Ş 8.8270251231909853e-07 -▁nachhalt 8.8270251231909853e-07 -inded 8.7891408951944574e-07 -stritten 8.7891408951944574e-07 -trock 8.7891408951944574e-07 -▁Paläst 8.7891408951944574e-07 -▁Seb 8.7891408951944574e-07 -vell 8.75125666719793e-07 -▁Sien 8.75125666719793e-07 --to-d 8.7133724392014017e-07 -▁Guatem 8.7133724392014017e-07 -▁Ora 8.7133724392014017e-07 -▁prü 8.7133724392014017e-07 -Press 8.6754882112048739e-07 -joy 8.6754882112048739e-07 -َ 8.637603983208346e-07 -▁vorank 8.637603983208346e-07 -▁Kommunik 8.5997197552118182e-07 -gleichheit 8.56183552721529e-07 -ibrary 8.56183552721529e-07 -mmern 8.56183552721529e-07 -▁GIM 8.56183552721529e-07 -▁Gelt 8.56183552721529e-07 --condition 8.5239512992187624e-07 -▁Nordir 8.5239512992187624e-07 -eoples 8.4860670712222346e-07 -mbH 8.4860670712222346e-07 -wanne 8.4860670712222346e-07 -▁Wäl 8.4860670712222346e-07 --border 8.4481828432257067e-07 -quiv 8.4481828432257067e-07 -schwing 8.4481828432257067e-07 -tains 8.4481828432257067e-07 -Τ 8.4481828432257067e-07 -ategorien 8.4102986152291789e-07 -fül 8.4102986152291789e-07 -▁blie 8.4102986152291789e-07 -▁hilfs 8.4102986152291789e-07 -utenant 8.372414387232651e-07 -Ќ 8.372414387232651e-07 -▁retir 8.372414387232651e-07 -altar 8.3345301592361232e-07 -appropri 8.3345301592361232e-07 -imir 8.3345301592361232e-07 -lieb 8.3345301592361232e-07 --suite 8.2966459312395953e-07 -▁bess 8.2966459312395953e-07 -▁küm 8.2966459312395953e-07 -▁purch 8.2966459312395953e-07 -kömm 8.2587617032430674e-07 -läß 8.2587617032430674e-07 -▁Kürz 8.2587617032430674e-07 -▁hinnehm 8.2587617032430674e-07 -▁reimb 8.2587617032430674e-07 --Emissionen 8.2208774752465406e-07 -rechend 8.2208774752465406e-07 -wann 8.2208774752465406e-07 -▁Alic 8.2208774752465406e-07 -▁Democ 8.2208774752465406e-07 -ecially 8.1829932472500128e-07 -þ 8.1829932472500128e-07 -▁Tschetschen 8.1829932472500128e-07 -▁glob 8.1829932472500128e-07 -ateful 8.1451090192534849e-07 -everkehr 8.1451090192534849e-07 -Е 8.1072247912569571e-07 -▁tatsäch 8.1072247912569571e-07 -künft 8.0693405632604292e-07 -vements 8.0693405632604292e-07 -▁ermord 8.0693405632604292e-07 -▁gewid 8.0693405632604292e-07 -▁ziem 8.0693405632604292e-07 --Office 8.0314563352639014e-07 -ivil 8.0314563352639014e-07 -▁Akzept 8.0314563352639014e-07 -▁rand 8.0314563352639014e-07 -▁wholehear 8.0314563352639014e-07 -amework 7.9935721072673735e-07 -ikanen 7.9935721072673735e-07 -ilber 7.9935721072673735e-07 -iquid 7.9556878792708457e-07 -▁Lt 7.9556878792708457e-07 -▁unaccept 7.9556878792708457e-07 -hältnisse 7.9178036512743178e-07 -▁Gere 7.9178036512743178e-07 -▁Währ 7.9178036512743178e-07 -door 7.87991942327779e-07 -eauc 7.87991942327779e-07 -ikalien 7.87991942327779e-07 -mär 7.87991942327779e-07 -▁E-M 7.87991942327779e-07 -▁Ernähr 7.87991942327779e-07 -.e. 7.8420351952812621e-07 -itives 7.8420351952812621e-07 -ovol 7.8420351952812621e-07 -▁Sloven 7.8420351952812621e-07 -● 7.8420351952812621e-07 -า 7.8041509672847342e-07 -▁Klär 7.8041509672847342e-07 -▁Subst 7.8041509672847342e-07 -▁air-condition 7.8041509672847342e-07 -bewertungen 7.7662667392882064e-07 -etext 7.7662667392882064e-07 -iovis 7.7662667392882064e-07 --spec 7.7283825112916785e-07 -fety 7.7283825112916785e-07 -udsman 7.7283825112916785e-07 -▁Düss 7.7283825112916785e-07 -stances 7.6904982832951507e-07 -▁Rett 7.6904982832951507e-07 -viron 7.6526140552986228e-07 -▁Bezieh 7.6526140552986228e-07 -aggage 7.6147298273020949e-07 -ascript 7.6147298273020949e-07 -attle 7.6147298273020949e-07 -▁Verhaltensk 7.6147298273020949e-07 -clude 7.5768455993055671e-07 -ewhere 7.5768455993055671e-07 -Ţ 7.5768455993055671e-07 -▁biod 7.5768455993055671e-07 -▁konfront 7.5768455993055671e-07 -▁tas 7.5768455993055671e-07 -iehlt 7.5389613713090392e-07 -▁Entertain 7.5389613713090392e-07 -▁Fortsch 7.5389613713090392e-07 -▁Verhin 7.5389613713090392e-07 --Fi 7.5010771433125114e-07 -ategories 7.5010771433125114e-07 -eget 7.5010771433125114e-07 -yption 7.5010771433125114e-07 -▁Matth 7.5010771433125114e-07 -▁Sier 7.5010771433125114e-07 -▁franz 7.5010771433125114e-07 -heber 7.4631929153159835e-07 -▁Amend 7.4631929153159835e-07 -▁Konver 7.4631929153159835e-07 -Ô 7.4253086873194557e-07 -▁sämt 7.4253086873194557e-07 -oubtedly 7.3874244593229278e-07 -ountry 7.3874244593229278e-07 -▁Slidesh 7.3874244593229278e-07 -▁zwangs 7.3874244593229278e-07 -conom 7.3495402313264e-07 -ifelong 7.3495402313264e-07 -itably 7.3495402313264e-07 -ufact 7.3495402313264e-07 -▁Verbrech 7.3495402313264e-07 -5-0 7.3116560033298721e-07 -quisite 7.3116560033298721e-07 - 7.3116560033298721e-07 -moking 7.2737717753333442e-07 -uggle 7.2737717753333442e-07 -▁Shang 7.2737717753333442e-07 -▁ht 7.2737717753333442e-07 -▁semin 7.2737717753333442e-07 -illance 7.2358875473368164e-07 -sprochen 7.2358875473368164e-07 -äische 7.2358875473368164e-07 -human 7.1980033193402885e-07 -ouver 7.1980033193402885e-07 -▁Auseinander 7.1980033193402885e-07 -▁reaff 7.1980033193402885e-07 -lichungen 7.1601190913437607e-07 -▁Erklär 7.1601190913437607e-07 -▁Gegeben 7.1601190913437607e-07 -▁Orig 7.1601190913437607e-07 -▁allm 7.1601190913437607e-07 -thand 7.1222348633472328e-07 -▁Berücksicht 7.1222348633472328e-07 -▁beglückwün 7.1222348633472328e-07 -▁beibeh 7.1222348633472328e-07 -ahoo 7.084350635350705e-07 -І 7.084350635350705e-07 -▁Nürn 7.084350635350705e-07 -▁bizar 7.084350635350705e-07 -▁Repar 7.0464664073541771e-07 -▁vorant 7.0464664073541771e-07 -britann 7.0085821793576492e-07 -cean 7.0085821793576492e-07 -▁Umwand 7.0085821793576492e-07 -▁fulf 7.0085821793576492e-07 -andon 6.9706979513611214e-07 -enophob 6.9706979513611214e-07 -tera 6.9706979513611214e-07 -▁spekt 6.9706979513611214e-07 -uture 6.9328137233645935e-07 -▁Forder 6.9328137233645935e-07 -▁coher 6.9328137233645935e-07 -▁Äg 6.9328137233645935e-07 -agraph 6.8949294953680657e-07 -▁Arzneim 6.8949294953680657e-07 -▁Toul 6.8949294953680657e-07 -▁exper 6.8949294953680657e-07 -itiativen 6.8570452673715378e-07 -ج 6.8570452673715378e-07 -kest 6.81916103937501e-07 -ª 6.81916103937501e-07 -Ð 6.81916103937501e-07 -ancellor 6.7812768113784821e-07 -fox 6.7812768113784821e-07 -ipedia 6.7812768113784821e-07 -▁upd 6.7812768113784821e-07 -aure 6.7433925833819542e-07 -iscipl 6.7433925833819542e-07 -▁Fotok 6.7433925833819542e-07 -▁medic 6.7433925833819542e-07 -haps 6.7055083553854264e-07 -▁Nichtregierungs 6.7055083553854264e-07 --rea 6.6676241273888985e-07 -▁Frankfur 6.6676241273888985e-07 -ambre 6.6297398993923707e-07 -dresse 6.6297398993923707e-07 -eague 6.6297398993923707e-07 -kompl 6.6297398993923707e-07 -rency 6.6297398993923707e-07 -א 6.6297398993923707e-07 -▁Klav 6.6297398993923707e-07 -▁Nahr 6.6297398993923707e-07 -▁Zin 6.6297398993923707e-07 -▁erleicht 6.6297398993923707e-07 -▁unzure 6.6297398993923707e-07 -lere 6.5918556713958428e-07 -pectives 6.5918556713958428e-07 -phrase 6.553971443399315e-07 -ร 6.553971443399315e-07 -▁Arzne 6.553971443399315e-07 -▁Freiz 6.553971443399315e-07 -▁Vorschl 6.553971443399315e-07 -▁phen 6.553971443399315e-07 -edig 6.5160872154027871e-07 -ו 6.4782029874062592e-07 -▁commod 6.4782029874062592e-07 -▁posit 6.4782029874062592e-07 -hatt 6.4403187594097314e-07 -obei 6.4403187594097314e-07 -ternehmen 6.4403187594097314e-07 -ubiläum 6.4403187594097314e-07 -uese 6.4403187594097314e-07 -▁portug 6.4403187594097314e-07 -stande 6.4024345314132046e-07 -ی 6.4024345314132046e-07 -▁Newsp 6.4024345314132046e-07 -▁twel 6.4024345314132046e-07 -brigen 6.3645503034166767e-07 -Über 6.3645503034166767e-07 -▁Auth 6.3645503034166767e-07 -▁Lear 6.3645503034166767e-07 -▁Thurs 6.3645503034166767e-07 -▁territor 6.3645503034166767e-07 -▁trium 6.3645503034166767e-07 -▁Lanz 6.3266660754201489e-07 -▁Mazed 6.3266660754201489e-07 -▁unzähl 6.3266660754201489e-07 -▁vorbeh 6.3266660754201489e-07 -teneg 6.288781847423621e-07 -′ 6.288781847423621e-07 -▁Secr 6.288781847423621e-07 -▁stär 6.288781847423621e-07 -▁unpre 6.288781847423621e-07 -mittelt 6.2508976194270932e-07 -orteil 6.2508976194270932e-07 -though 6.2508976194270932e-07 -örde 6.2508976194270932e-07 -Ő 6.2508976194270932e-07 -▁Herangehens 6.2508976194270932e-07 -fence 6.2130133914305653e-07 -oments 6.2130133914305653e-07 -▁Washing 6.2130133914305653e-07 -▁infrastruct 6.2130133914305653e-07 -▁langu 6.2130133914305653e-07 -▁sacr 6.2130133914305653e-07 -yside 6.1751291634340375e-07 -prechen 6.13724493543751e-07 -quare 6.13724493543751e-07 -▁beitr 6.13724493543751e-07 -drücklich 6.0993607074409817e-07 -shine 6.0993607074409817e-07 -น 6.0993607074409817e-07 -▁Nachfol 6.0993607074409817e-07 -▁Wohler 6.0993607074409817e-07 -▁gegenwärt 6.0993607074409817e-07 -▁grenz 6.0993607074409817e-07 -▁produc 6.0993607074409817e-07 -roökonom 6.0614764794444539e-07 -Э 6.0614764794444539e-07 -frastr 6.023592251447926e-07 -ärenz 6.023592251447926e-07 -unft 5.9857080234513982e-07 -▁Distr 5.9857080234513982e-07 -▁aufrechter 5.9857080234513982e-07 -glück 5.94782379545487e-07 -ش 5.94782379545487e-07 -▁Kinn 5.94782379545487e-07 -▁Liban 5.94782379545487e-07 -▁unang 5.94782379545487e-07 -bsite 5.9099395674583425e-07 -ployed 5.9099395674583425e-07 -pricht 5.9099395674583425e-07 -▁Subsid 5.9099395674583425e-07 -aukas 5.8720553394618146e-07 -Ф 5.8720553394618146e-07 -▁Birm 5.8720553394618146e-07 -▁Wün 5.8720553394618146e-07 -▁dut 5.8720553394618146e-07 -▁experien 5.8720553394618146e-07 -freiheiten 5.8341711114652867e-07 -kok 5.8341711114652867e-07 -olden 5.8341711114652867e-07 -▁Verring 5.8341711114652867e-07 -▁genu 5.8341711114652867e-07 -▁whe 5.8341711114652867e-07 -konom 5.7962868834687589e-07 -ώ 5.7962868834687589e-07 -▁Meld 5.7962868834687589e-07 -▁benefic 5.7962868834687589e-07 -schließlich 5.758402655472231e-07 -▁Annäher 5.758402655472231e-07 -▁Californ 5.758402655472231e-07 -▁Parl 5.758402655472231e-07 -▁Weihn 5.758402655472231e-07 -▁certific 5.758402655472231e-07 -▁jac 5.758402655472231e-07 -athedrale 5.7205184274757032e-07 -look 5.7205184274757032e-07 -ר 5.7205184274757032e-07 -▁Behand 5.7205184274757032e-07 -▁Croat 5.7205184274757032e-07 -▁Städ 5.7205184274757032e-07 -▁beur 5.7205184274757032e-07 -ecess 5.6826341994791753e-07 -▁attra 5.6826341994791753e-07 -IFE 5.6447499714826475e-07 -conditioning 5.6447499714826475e-07 -ibungs 5.6447499714826475e-07 -เ 5.6447499714826475e-07 -▁Jap 5.6447499714826475e-07 -▁Siem 5.6447499714826475e-07 -olge 5.60686574348612e-07 -▁Prozents 5.60686574348612e-07 -iago 5.5689815154895917e-07 -uminium 5.5689815154895917e-07 -▁KU 5.5689815154895917e-07 -¦ 5.5310972874930639e-07 -ح 5.5310972874930639e-07 -خ 5.5310972874930639e-07 -″ 5.5310972874930639e-07 -fl 5.5310972874930639e-07 -▁self-em 5.493213059496536e-07 -sprüch 5.4553288315000082e-07 -ußert 5.4553288315000082e-07 -Ν 5.4553288315000082e-07 -อ 5.4553288315000082e-07 -▁Tues 5.4553288315000082e-07 -▁Vice-Pres 5.4553288315000082e-07 -▁muse 5.4553288315000082e-07 -anbul 5.41744460350348e-07 -‟ 5.41744460350348e-07 -itime 5.3795603755069525e-07 -Å 5.3795603755069525e-07 -إ 5.3795603755069525e-07 -▁Völ 5.3795603755069525e-07 -itzerland 5.3416761475104246e-07 -taking 5.3416761475104246e-07 -¾ 5.3416761475104246e-07 -Љ 5.3416761475104246e-07 -ِ 5.3416761475104246e-07 -▁Syd 5.3416761475104246e-07 -▁hervorragen 5.3416761475104246e-07 -▁signific 5.3416761475104246e-07 -▁umgew 5.3416761475104246e-07 -auspiel 5.3037919195138967e-07 -istol 5.3037919195138967e-07 -undheit 5.3037919195138967e-07 -vernight 5.3037919195138967e-07 -weitet 5.3037919195138967e-07 -њ 5.2659076915173689e-07 -▁emerg 5.228023463520841e-07 -▁Fischereip 5.1901392355243132e-07 -▁Karib 5.1901392355243132e-07 -▁Untersuch 5.1901392355243132e-07 -▁kommer 5.1901392355243132e-07 -▁sub-c 5.1901392355243132e-07 -eservice 5.1522550075277853e-07 -esonder 5.1522550075277853e-07 -▁Sams 5.1522550075277853e-07 -▁ält 5.1522550075277853e-07 -ancell 5.1143707795312575e-07 -gesamt 5.1143707795312575e-07 -otokoll 5.1143707795312575e-07 -Š 5.1143707795312575e-07 -Ι 5.1143707795312575e-07 -Ο 5.1143707795312575e-07 -№ 5.1143707795312575e-07 -▁Herunter 5.1143707795312575e-07 -▁unem 5.1143707795312575e-07 -▁Beschwer 5.07648655153473e-07 -▁Verhand 5.07648655153473e-07 -▁verwirkl 5.07648655153473e-07 -eutung 5.0386023235382018e-07 -stockung 5.0386023235382018e-07 -völkerung 5.0386023235382018e-07 -ξ 5.0386023235382018e-07 -← 5.0386023235382018e-07 -▁Subsidiar 5.0386023235382018e-07 -▁interf 5.0386023235382018e-07 -▁kontext 5.0386023235382018e-07 --Mittelmeer 5.0007180955416739e-07 -demokraten 5.0007180955416739e-07 -▁Cau 4.962833867545146e-07 -▁Einstimm 4.962833867545146e-07 -▁bec 4.962833867545146e-07 -▁crimin 4.962833867545146e-07 -▁gg 4.962833867545146e-07 -ilege 4.9249496395486182e-07 -※ 4.9249496395486182e-07 -▁Kernen 4.9249496395486182e-07 -， 4.9249496395486182e-07 -Pal 4.88706541155209e-07 -isdom 4.88706541155209e-07 -gess 4.8491811835555625e-07 -zellent 4.8491811835555625e-07 -ämlich 4.8491811835555625e-07 -▁Oly 4.8491811835555625e-07 -▁scen 4.8491811835555625e-07 -的 4.8491811835555625e-07 -Я 4.8112969555590346e-07 -▁Beaut 4.8112969555590346e-07 -▁Fotokop 4.8112969555590346e-07 -▁disreg 4.8112969555590346e-07 -▁hät 4.8112969555590346e-07 -▁unforg 4.8112969555590346e-07 -PEG 4.7734127275625068e-07 -feindlichkeit 4.7734127275625068e-07 -ologne 4.7734127275625068e-07 -¢ 4.7734127275625068e-07 -‡ 4.7734127275625068e-07 -⇒ 4.7734127275625068e-07 -▁Flugh 4.7734127275625068e-07 -ofür 4.7355284995659794e-07 -utory 4.7355284995659794e-07 -▁erziel 4.7355284995659794e-07 -Tube 4.6976442715694516e-07 -inuier 4.6976442715694516e-07 -Đ 4.6976442715694516e-07 -Х 4.6976442715694516e-07 -▁Gespr 4.6976442715694516e-07 -▁Kontinu 4.6976442715694516e-07 -▁entde 4.6976442715694516e-07 -iminary 4.6597600435729237e-07 -ל 4.6597600435729237e-07 -่ 4.6597600435729237e-07 -▁Verzö 4.6597600435729237e-07 -▁Zuer 4.6597600435729237e-07 -▁verschlechter 4.6597600435729237e-07 -dehnung 4.6218758155763959e-07 -ertungen 4.6218758155763959e-07 -▁Elev 4.6218758155763959e-07 -▁irreg 4.6218758155763959e-07 -▁refres 4.6218758155763959e-07 -athem 4.583991587579868e-07 -ม 4.583991587579868e-07 -▁jeweil 4.583991587579868e-07 -▁überras 4.583991587579868e-07 -。 4.583991587579868e-07 -atastrop 4.54610735958334e-07 -ergestellt 4.54610735958334e-07 -ittle 4.54610735958334e-07 -Þ 4.54610735958334e-07 -ņ 4.54610735958334e-07 -▁Benj 4.54610735958334e-07 -▁beeintr 4.54610735958334e-07 -▁wides 4.54610735958334e-07 -中 4.54610735958334e-07 -forward 4.5082231315868123e-07 -isticated 4.5082231315868123e-07 -ocaust 4.5082231315868123e-07 -outique 4.5082231315868123e-07 -pth 4.5082231315868123e-07 -ง 4.5082231315868123e-07 -▁Berlus 4.5082231315868123e-07 -elso 4.4703389035902844e-07 -governmental 4.4703389035902844e-07 -ianti 4.4703389035902844e-07 -uführen 4.4703389035902844e-07 -▁Annehm 4.4703389035902844e-07 -▁hap 4.4703389035902844e-07 -accept 4.4324546755937566e-07 -isory 4.4324546755937566e-07 -ע 4.4324546755937566e-07 -▁Ingen 4.4324546755937566e-07 -▁tourn 4.4324546755937566e-07 -hton 4.3945704475972287e-07 -▁Geträn 4.3566862196007009e-07 -▁Wahrschein 4.3566862196007009e-07 -▁meas 4.3566862196007009e-07 -▁Byr 4.318801991604173e-07 -▁Lanzar 4.318801991604173e-07 -▁datab 4.318801991604173e-07 -IEW 4.2809177636076452e-07 -ocktail 4.2430335356111173e-07 -terity 4.2430335356111173e-07 -▁Anbet 4.2430335356111173e-07 -▁Neap 4.2430335356111173e-07 -binsel 4.2051493076145894e-07 -▁Vitor 4.2051493076145894e-07 -▁bevorzu 4.2051493076145894e-07 -▁suc 4.2051493076145894e-07 -▁Präs 4.1672650796180616e-07 -▁Tosk 4.1672650796180616e-07 -▁gewährleist 4.1672650796180616e-07 -▁municip 4.1672650796180616e-07 -▁Überschwemm 4.1672650796180616e-07 -sych 4.1293808516215337e-07 -▁Angr 4.1293808516215337e-07 -▁dang 4.1293808516215337e-07 -▁encr 4.1293808516215337e-07 -etooth 4.0914966236250064e-07 -klear 4.0914966236250064e-07 -ußer 4.0914966236250064e-07 -trockner 4.0536123956284785e-07 -twoch 4.0536123956284785e-07 -▁Thessalon 4.0536123956284785e-07 -▁betreff 4.0536123956284785e-07 -cedes 4.0157281676319507e-07 -ektionismus 4.0157281676319507e-07 -▁Bost 4.0157281676319507e-07 -▁scener 4.0157281676319507e-07 -hältnis 3.9778439396354228e-07 -ubehör 3.9778439396354228e-07 -▁Bettw 3.9778439396354228e-07 -▁appre 3.9778439396354228e-07 -▁incent 3.9778439396354228e-07 -▁proced 3.9778439396354228e-07 -▁erled 3.939959711638895e-07 -imini 3.9020754836423671e-07 -▁Carib 3.8641912556458393e-07 -erce 3.8263070276493114e-07 -izeil 3.8263070276493114e-07 -nächst 3.8263070276493114e-07 -schlossenheit 3.8263070276493114e-07 -endit 3.7884227996527835e-07 -udem 3.7884227996527835e-07 -▁Araf 3.7884227996527835e-07 --of-the-art 3.7505385716562557e-07 --reaching 3.7505385716562557e-07 -undry 3.7505385716562557e-07 -▁geplan 3.7126543436597278e-07 -▁inappropri 3.7126543436597278e-07 -ivile 3.6747701156632e-07 -meth 3.6747701156632e-07 -sphäre 3.6747701156632e-07 -wier 3.6747701156632e-07 -▁Berech 3.6747701156632e-07 -▁Wednes 3.6747701156632e-07 -▁genoc 3.6747701156632e-07 -kontr 3.6368858876666721e-07 -unya 3.6368858876666721e-07 -▁Gibr 3.6368858876666721e-07 -▁feh 3.6368858876666721e-07 --mail 3.5990016596701443e-07 -ugoslav 3.5990016596701443e-07 -▁genü 3.5990016596701443e-07 --170 3.5611174316736164e-07 -azines 3.5611174316736164e-07 -erbs 3.5611174316736164e-07 -wogenheit 3.5611174316736164e-07 -▁Dail 3.5611174316736164e-07 -▁inev 3.5611174316736164e-07 -ubeh 3.5232332036770885e-07 -öffnen 3.5232332036770885e-07 -▁Caucas 3.5232332036770885e-07 -▁archit 3.5232332036770885e-07 -▁begrü 3.5232332036770885e-07 -▁whol 3.5232332036770885e-07 -▁zunehm 3.5232332036770885e-07 -cany 3.4474647476840328e-07 -ibliot 3.4474647476840328e-07 -osur 3.4474647476840328e-07 -▁resour 3.4474647476840328e-07 --called 3.409580519687505e-07 -ettable 3.409580519687505e-07 -rechter 3.409580519687505e-07 -urecht 3.409580519687505e-07 -wirr 3.409580519687505e-07 -▁Zuverläss 3.409580519687505e-07 -▁shap 3.409580519687505e-07 -▁Berichterst 3.3716962916909771e-07 --ranging 3.3338120636944493e-07 -ossier 3.3338120636944493e-07 -oust 3.3338120636944493e-07 -porate 3.3338120636944493e-07 -Č 3.3338120636944493e-07 -▁Kaliforn 3.3338120636944493e-07 -▁nöt 3.3338120636944493e-07 --Entw 3.2959278356979214e-07 -ehow 3.2959278356979214e-07 -ilst 3.2959278356979214e-07 -▁Lae 3.2959278356979214e-07 -▁Singap 3.2959278356979214e-07 -etimes 3.2580436077013936e-07 -iefung 3.2580436077013936e-07 -kohol 3.2580436077013936e-07 -▁Liebl 3.2580436077013936e-07 -▁Vereinbar 3.2580436077013936e-07 -▁anbiet 3.2580436077013936e-07 -▁bemü 3.2580436077013936e-07 -▁hers 3.2580436077013936e-07 -leet 3.2201593797048657e-07 -omenade 3.2201593797048657e-07 -ziehe 3.2201593797048657e-07 -▁progr 3.2201593797048657e-07 -aurant 3.1822751517083384e-07 -▁Abwesen 3.1822751517083384e-07 -▁refr 3.1822751517083384e-07 -▁Änder 3.1822751517083384e-07 -rasch 3.1443909237118105e-07 -wirklichung 3.1443909237118105e-07 -▁Marb 3.1443909237118105e-07 -erstellen 3.1065066957152827e-07 -wiegend 3.1065066957152827e-07 -▁Begeister 3.1065066957152827e-07 -▁Heut 3.1065066957152827e-07 --Mediterranean 3.0686224677187548e-07 -rengung 3.0686224677187548e-07 -tworten 3.0686224677187548e-07 -ügigkeit 3.0686224677187548e-07 -▁Nok 3.0686224677187548e-07 -▁Unregelmäß 3.0686224677187548e-07 -▁beeinfluss 3.0686224677187548e-07 -▁grundlegen 3.0686224677187548e-07 -▁immedi 3.0686224677187548e-07 -ASP 3.0307382397222269e-07 -aos 3.0307382397222269e-07 -atastroph 3.0307382397222269e-07 -▁Ferrero 3.0307382397222269e-07 -▁Klimasch 3.0307382397222269e-07 -▁Perspekt 2.9928540117256991e-07 -▁assoc 2.9928540117256991e-07 -▁opin 2.9928540117256991e-07 --to-date 2.9549697837291712e-07 -igenz 2.9549697837291712e-07 -▁Manhatt 2.9549697837291712e-07 -▁Tats 2.9549697837291712e-07 -▁Untern 2.9549697837291712e-07 -▁advant 2.9549697837291712e-07 -▁tats 2.9549697837291712e-07 -kthrough 2.9170855557326434e-07 -▁jem 2.9170855557326434e-07 -bry 2.8792013277361155e-07 -ermost 2.8792013277361155e-07 -indruckt 2.8792013277361155e-07 -▁indir 2.8792013277361155e-07 -▁larg 2.8792013277361155e-07 -▁müß 2.8792013277361155e-07 -▁zuver 2.8792013277361155e-07 -ECD 2.8413170997395877e-07 -▁offl 2.8413170997395877e-07 -▁refle 2.8413170997395877e-07 -▁Milosev 2.80343287174306e-07 -▁Urspr 2.80343287174306e-07 -VIEW 2.7655486437465319e-07 -zustimmen 2.7655486437465319e-07 -▁Algar 2.7655486437465319e-07 -▁Eras 2.7655486437465319e-07 -▁Varian 2.7655486437465319e-07 -▁tä 2.7655486437465319e-07 -gestimmt 2.7276644157500041e-07 -▁criter 2.7276644157500041e-07 -expected 2.6897801877534762e-07 -inki 2.6897801877534762e-07 -ј 2.6897801877534762e-07 -▁Gip 2.6897801877534762e-07 -▁Handhab 2.6897801877534762e-07 -▁Infine 2.6897801877534762e-07 -▁Medik 2.6897801877534762e-07 -▁Stras 2.6897801877534762e-07 --Wald 2.6518959597569484e-07 -gän 2.6518959597569484e-07 -strateg 2.6518959597569484e-07 -▁eingen 2.6518959597569484e-07 -▁jen 2.6518959597569484e-07 -▁toget 2.6518959597569484e-07 -▁vorlie 2.6518959597569484e-07 -imensionen 2.6140117317604205e-07 -▁Kompeten 2.6140117317604205e-07 -▁Rotter 2.6140117317604205e-07 -▁gän 2.6140117317604205e-07 -▁incompl 2.6140117317604205e-07 -Kr 2.5761275037638927e-07 -tedly 2.5761275037638927e-07 -▁Bolog 2.5761275037638927e-07 -ckst 2.5382432757673648e-07 -ovon 2.5382432757673648e-07 -ubehalten 2.5382432757673648e-07 -▁Edin 2.5382432757673648e-07 -▁Palest 2.5382432757673648e-07 -besondere 2.5003590477708369e-07 -ugos 2.5003590477708369e-07 --Ru 2.4624748197743091e-07 -abei 2.4624748197743091e-07 -▁auth 2.4624748197743091e-07 -regelmäß 2.4245905917777812e-07 -robat 2.4245905917777812e-07 -▁Declar 2.4245905917777812e-07 -▁inten 2.4245905917777812e-07 -▁surpr 2.4245905917777812e-07 -▁toba 2.4245905917777812e-07 -ESCO 2.3867063637812534e-07 -emost 2.3867063637812534e-07 -rebt 2.3867063637812534e-07 -regungen 2.3867063637812534e-07 -▁Alz 2.3867063637812534e-07 -cknow 2.3488221357847258e-07 -▁Daim 2.3488221357847258e-07 -▁Lufth 2.3488221357847258e-07 -laubens 2.3109379077881979e-07 -merksam 2.3109379077881979e-07 -oboda 2.3109379077881979e-07 -ranational 2.3109379077881979e-07 -täus 2.3109379077881979e-07 -▁Bruss 2.3109379077881979e-07 -▁Verschlechter 2.3109379077881979e-07 -▁gewünsch 2.3109379077881979e-07 -▁unterscheid 2.3109379077881979e-07 --DE 2.27305367979167e-07 -andin 2.27305367979167e-07 -icits 2.27305367979167e-07 -kapaz 2.27305367979167e-07 -laga 2.27305367979167e-07 -odurch 2.27305367979167e-07 -▁einzur 2.27305367979167e-07 -▁indisp 2.2351694517951422e-07 -Rooms 2.1594009958020865e-07 -obald 2.1594009958020865e-07 -▁Verschied 2.1594009958020865e-07 -▁natür 2.1594009958020865e-07 -▁beseit 2.1215167678055586e-07 -▁disprop 2.1215167678055586e-07 -▁incom 2.1215167678055586e-07 -▁erhö 2.0836325398090308e-07 -aceut 2.0457483118125032e-07 -aptop 2.0457483118125032e-07 -ocopying 2.0457483118125032e-07 -▁countr 2.0457483118125032e-07 -▁incred 2.0457483118125032e-07 -▁überrasch 2.0457483118125032e-07 -bergt 2.0078640838159753e-07 -ijten 2.0078640838159753e-07 -▁beein 2.0078640838159753e-07 -▁empfeh 2.0078640838159753e-07 -bezieh 1.9699798558194475e-07 -▁Starf 1.9699798558194475e-07 -usstsein 1.9320956278229196e-07 -▁vess 1.9320956278229196e-07 -▁disastr 1.8942113998263918e-07 -▁includ 1.8942113998263918e-07 -anspruchnahme 1.8563271718298639e-07 -athedral 1.8563271718298639e-07 -ealand 1.8563271718298639e-07 -ehin 1.8563271718298639e-07 -ältigen 1.8563271718298639e-07 -▁Jarz 1.8563271718298639e-07 -▁Oomen 1.8563271718298639e-07 -▁earl 1.8563271718298639e-07 -estock 1.8184429438333361e-07 -utreiben 1.8184429438333361e-07 -weitung 1.8184429438333361e-07 -▁Ferienwohn 1.8184429438333361e-07 --Herz 1.7805587158368082e-07 -Fast 1.7805587158368082e-07 -ützlich 1.7805587158368082e-07 -▁Millenn 1.7805587158368082e-07 -▁benach 1.7805587158368082e-07 --depth 1.7426744878402803e-07 -amew 1.7426744878402803e-07 -anmar 1.7426744878402803e-07 -oeuv 1.7426744878402803e-07 -schließend 1.7426744878402803e-07 -▁Aufmerksam 1.7426744878402803e-07 -▁Einwohn 1.7426744878402803e-07 -▁unterw 1.7426744878402803e-07 -cussions 1.7047902598437525e-07 -grunde 1.7047902598437525e-07 -▁Abgeordn 1.7047902598437525e-07 -▁Göte 1.7047902598437525e-07 -▁besonder 1.7047902598437525e-07 -▁Cohn 1.6669060318472246e-07 -▁indispens 1.6669060318472246e-07 -OFIN 1.6290218038506968e-07 -befinden 1.6290218038506968e-07 -cía 1.6290218038506968e-07 -fghan 1.6290218038506968e-07 -advant 1.5911375758541692e-07 -akings 1.5911375758541692e-07 -peratur 1.5911375758541692e-07 -streicht 1.5911375758541692e-07 -▁herbeiz 1.5911375758541692e-07 -▁secon 1.5911375758541692e-07 -tragt 1.5532533478576413e-07 -▁Urug 1.5532533478576413e-07 -endigung 1.5153691198611135e-07 -ergeleitet 1.5153691198611135e-07 -soever 1.5153691198611135e-07 -▁Mobilt 1.5153691198611135e-07 -ießlich 1.4774848918645856e-07 -▁Drit 1.4774848918645856e-07 -▁Eurocop 1.4774848918645856e-07 -▁Wats 1.4774848918645856e-07 -▁geä 1.4774848918645856e-07 -inland 1.4396006638680578e-07 -yssen 1.4396006638680578e-07 -▁asyl 1.4396006638680578e-07 -▁höch 1.4396006638680578e-07 -ünf 1.40171643587153e-07 -▁recog 1.40171643587153e-07 -▁thres 1.40171643587153e-07 -leiht 1.363832207875002e-07 -stattung 1.363832207875002e-07 -urther 1.363832207875002e-07 -wirklichen 1.363832207875002e-07 -▁Kommiss 1.363832207875002e-07 -▁Thess 1.363832207875002e-07 -▁enem 1.363832207875002e-07 -hme 1.3259479798784742e-07 -▁QuarkX 1.3259479798784742e-07 -▁ermö 1.3259479798784742e-07 -▁requir 1.3259479798784742e-07 -▁verbess 1.3259479798784742e-07 -atastr 1.2880637518819463e-07 -celand 1.2880637518819463e-07 -hältnism 1.2880637518819463e-07 -tenteils 1.2880637518819463e-07 -▁Dubrov 1.2880637518819463e-07 -▁accur 1.2880637518819463e-07 -▁grunds 1.2880637518819463e-07 -▁reas 1.2880637518819463e-07 -▁unfortun 1.2880637518819463e-07 -▁veranl 1.2880637518819463e-07 -chtenstein 1.2501795238854185e-07 -ibiot 1.2501795238854185e-07 -yright 1.2501795238854185e-07 -▁Beibeh 1.2501795238854185e-07 -▁Luxem 1.2501795238854185e-07 -▁persön 1.2501795238854185e-07 -fraid 1.2122952958888906e-07 -ikely 1.2122952958888906e-07 -yond 1.2122952958888906e-07 -▁Sämt 1.2122952958888906e-07 -▁agricult 1.2122952958888906e-07 -incere 1.1744110678923629e-07 -pite 1.1744110678923629e-07 -▁Aufrechter 1.1744110678923629e-07 -▁Febru 1.1744110678923629e-07 -▁Gonzá 1.1744110678923629e-07 -▁Isra 1.1744110678923629e-07 -▁wicht 1.1744110678923629e-07 -icter 1.136526839895835e-07 -itology 1.136526839895835e-07 -▁Maust 1.136526839895835e-07 --Rezeption 1.0986426118993072e-07 -ieweit 1.0986426118993072e-07 -umän 1.0986426118993072e-07 -▁Festpl 1.0986426118993072e-07 -▁Verwirr 1.0986426118993072e-07 -▁erläut 1.0986426118993072e-07 -▁jens 1.0607583839027793e-07 -▁newsp 1.0607583839027793e-07 -astricht 1.0228741559062516e-07 -enhagen 1.0228741559062516e-07 -ettering 1.0228741559062516e-07 -folgerungen 1.0228741559062516e-07 -iffern 1.0228741559062516e-07 -▁Vanc 1.0228741559062516e-07 -▁compon 1.0228741559062516e-07 -▁disag 1.0228741559062516e-07 -▁erwähn 1.0228741559062516e-07 -; 9.8498992790972374e-08 -qualities 9.8498992790972374e-08 -sbruck 9.8498992790972374e-08 -▁& 9.8498992790972374e-08 -▁Mandr 9.8498992790972374e-08 -▁Zuständ 9.8498992790972374e-08 -elservice 9.4710569991319589e-08 -obyl 9.4710569991319589e-08 -▁Fäh 9.4710569991319589e-08 --initiative 9.09221471916668e-08 -emeins 9.09221471916668e-08 -▁Wikitra 9.09221471916668e-08 -▁reper 9.09221471916668e-08 -▁Konsequ 8.7133724392014017e-08 -▁netw 8.7133724392014017e-08 -▁propert 8.7133724392014017e-08 -▁verstär 8.7133724392014017e-08 -▁Phän 8.3345301592361232e-08 -▁Telekommunik 8.3345301592361232e-08 -▁becom 8.3345301592361232e-08 -▁neigh 8.3345301592361232e-08 -wachst 7.9556878792708459e-08 -▁umfass 7.9556878792708459e-08 -▁establ 7.5768455993055674e-08 -▁etw 7.5768455993055674e-08 --Lös 7.1980033193402888e-08 -ispiell 6.81916103937501e-08 -ofern 6.81916103937501e-08 -▁Beantwort 6.81916103937501e-08 -▁Fronte 6.81916103937501e-08 -▁anbel 6.81916103937501e-08 -▁increas 6.81916103937501e-08 -eopard 6.4403187594097317e-08 -iefst 6.4403187594097317e-08 -regierungs 6.4403187594097317e-08 -selben 6.4403187594097317e-08 -▁Jarzemb 6.4403187594097317e-08 -▁Non-S 6.4403187594097317e-08 -▁thous 6.4403187594097317e-08 -benso 6.0614764794444531e-08 -either 6.0614764794444531e-08 -gemeine 6.0614764794444531e-08 -uggage 6.0614764794444531e-08 -▁Anson 6.0614764794444531e-08 -▁Erachten 6.0614764794444531e-08 -▁Erwachs 6.0614764794444531e-08 -▁encour 6.0614764794444531e-08 -▁incor 6.0614764794444531e-08 -▁manufactur 6.0614764794444531e-08 -▁subsidiar 6.0614764794444531e-08 -iereinrichtungen 5.6826341994791752e-08 -▁Coton 5.6826341994791752e-08 -▁Deutschl 5.6826341994791752e-08 -▁Verabschied 5.6826341994791752e-08 -▁somew 5.6826341994791752e-08 --Hour 5.3037919195138966e-08 --to-day 5.3037919195138966e-08 -chtungen 5.3037919195138966e-08 -kämpfung 5.3037919195138966e-08 -namo 5.3037919195138966e-08 -▁Chá 5.3037919195138966e-08 -▁Fratt 5.3037919195138966e-08 -▁beglück 5.3037919195138966e-08 -▁holid 5.3037919195138966e-08 -▁welcom 5.3037919195138966e-08 -comings 4.9249496395486187e-08 -folged 4.9249496395486187e-08 -gricult 4.9249496395486187e-08 -icars 4.9249496395486187e-08 -theless 4.9249496395486187e-08 -uential 4.9249496395486187e-08 -▁Bolkest 4.9249496395486187e-08 -▁Entschließ 4.9249496395486187e-08 -▁NCC-170 4.9249496395486187e-08 -▁Oomen-Ru 4.9249496395486187e-08 -▁behilf 4.9249496395486187e-08 -▁charact 4.9249496395486187e-08 -▁diesbe 4.9249496395486187e-08 -▁ursprün 4.9249496395486187e-08 -▁bereitz 4.54610735958334e-08 -▁negot 4.54610735958334e-08 -▁spezi 4.54610735958334e-08 -▁Änderungsan 4.54610735958334e-08 -enever 4.1672650796180616e-08 -zutage 4.1672650796180616e-08 -▁Amster 4.1672650796180616e-08 -▁Anmer 4.1672650796180616e-08 -▁Chechn 4.1672650796180616e-08 -▁Wachst 4.1672650796180616e-08 -▁beziehungs 4.1672650796180616e-08 -▁enthusias 4.1672650796180616e-08 -▁gemeins 4.1672650796180616e-08 -▁unanim 4.1672650796180616e-08 -▁unmittel 4.1672650796180616e-08 -grees 3.7884227996527837e-08 -korea 3.7884227996527837e-08 -▁curren 3.7884227996527837e-08 -▁katastrop 3.7884227996527837e-08 -Traq 3.4095805196875051e-08 -atever 3.4095805196875051e-08 -ehear 3.4095805196875051e-08 -fortunately 3.4095805196875051e-08 -zedenz 3.4095805196875051e-08 -▁Guantá 3.4095805196875051e-08 -▁provid 3.4095805196875051e-08 -▁spea 3.4095805196875051e-08 -adurch 3.0307382397222265e-08 -chmal 3.0307382397222265e-08 -riedenheit 3.0307382397222265e-08 -▁Katastrop 3.0307382397222265e-08 -▁Vorsitzen 3.0307382397222265e-08 -▁begeist 3.0307382397222265e-08 -▁enthus 3.0307382397222265e-08 -clud 2.6518959597569483e-08 -▁Schlussel 2.6518959597569483e-08 -▁ceil 2.6518959597569483e-08 -▁entreprene 2.6518959597569483e-08 -▁exac 2.6518959597569483e-08 -▁gewün 2.6518959597569483e-08 -ifilm 2.27305367979167e-08 -orgnis 2.27305367979167e-08 -▁Cohn-B 2.27305367979167e-08 -▁Frankf 2.27305367979167e-08 -▁Meinungsver 2.27305367979167e-08 -▁veröffentl 2.27305367979167e-08 -▁wesent 2.27305367979167e-08 -adays 1.8942113998263918e-08 -esterday 1.8942113998263918e-08 -vereign 1.8942113998263918e-08 -▁Ansprü 1.8942113998263918e-08 -▁acknow 1.8942113998263918e-08 -▁guarante 1.8942113998263918e-08 -▁verläng 1.8942113998263918e-08 -inahe 1.5153691198611133e-08 -zuerhalten 1.5153691198611133e-08 -▁Klimawand 1.5153691198611133e-08 -▁besorgnis 1.5153691198611133e-08 -▁obstac 1.5153691198611133e-08 -esondere 1.136526839895835e-08 -etheless 1.136526839895835e-08 -onsten 1.136526839895835e-08 -▁Bevöl 1.136526839895835e-08 -▁Diskuss 1.136526839895835e-08 -▁Ferrero-Wald 1.136526839895835e-08 -▁erinn 1.136526839895835e-08 -▁konkr 1.136526839895835e-08 -▁rapporte 1.136526839895835e-08 -cedented 7.5768455993055664e-09 -ffice 7.5768455993055664e-09 -▁Kohäs 7.5768455993055664e-09 -▁Konfl 7.5768455993055664e-09 -▁Milliar 7.5768455993055664e-09 -▁ThyssenKr 7.5768455993055664e-09 -▁citiz 7.5768455993055664e-09 -▁increa 7.5768455993055664e-09 -▁unterstüt 7.5768455993055664e-09 -▁Etw 3.7884227996527832e-09 -▁colleag 3.7884227996527832e-09 -▁einzube 3.7884227996527832e-09 -▁humanitä 3.7884227996527832e-09 -! 0 -# 0 -$ 0 -% 0 -( 0 -) 0 -* 0 -+ 0 -/ 0 -1; 0 -3; 0 -: 0 -= 0 -? 0 -@ 0 -\ 0 -_ 0 -dernfalls 0 -{ 0 -} 0 -~ 0 -£ 0 -¤ 0 -§ 0 -© 0 -« 0 -­ 0 -® 0 -° 0 -² 0 -³ 0 -´ 0 -¶ 0 -· 0 -» 0 -¼ 0 -½ 0 -¿ 0 -̈ 0 -– 0 -— 0 -‘ 0 -’ 0 -‚ 0 -“ 0 -” 0 -„ 0 -† 0 -• 0 -… 0 -€ 0 -™ 0 -▁&# 0 -▁| 0 -▁ 0 -▁& 0 -▁> 0 -▁" 0 -▁Konfli 0 -▁safegu 0 -▁withdra 0 -� 0 diff --git a/lingvo/tasks/mt/text_input.proto b/lingvo/tasks/mt/text_input.proto deleted file mode 100644 index 14cd5059..00000000 --- a/lingvo/tasks/mt/text_input.proto +++ /dev/null @@ -1,33 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -// Text input data formats for MT tasks. - -syntax = "proto2"; - -package tensorflow.lingvo; - -message Sentence { - optional string sentence = 1; - - // Two-letter ISO 639-1 codes for the language of the sentence: - // https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes - optional string lang = 2; -} - -message SentencePair { - optional Sentence src_sentence = 1; - optional Sentence tgt_sentence = 2; -} diff --git a/lingvo/tasks/mt/tools/__init__.py b/lingvo/tasks/mt/tools/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/lingvo/tasks/mt/tools/wmt14.01.download_moses_scripts.sh b/lingvo/tasks/mt/tools/wmt14.01.download_moses_scripts.sh deleted file mode 100755 index c4bd2868..00000000 --- a/lingvo/tasks/mt/tools/wmt14.01.download_moses_scripts.sh +++ /dev/null @@ -1,22 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. wmt14_lib.sh - -mkdir -p "${ROOT}" -git clone https://github.com/moses-smt/mosesdecoder.git ${ROOT}/mosesdecoder diff --git a/lingvo/tasks/mt/tools/wmt14.02.download_train.sh b/lingvo/tasks/mt/tools/wmt14.02.download_train.sh deleted file mode 100755 index a8e81fe9..00000000 --- a/lingvo/tasks/mt/tools/wmt14.02.download_train.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. wmt14_lib.sh - -mkdir -p "${ROOT}/raw" -# ============================================================================ -# Download WMT data. -# From the WMT14 website (En-De): -echo " -http://www.statmt.org/wmt13/training-parallel-europarl-v7.tgz -http://www.statmt.org/wmt13/training-parallel-commoncrawl.tgz -http://www.statmt.org/wmt14/training-parallel-nc-v9.tgz" \ - | aria2c -x16 -s1 -j1 --dir="${ROOT}/raw" -i - diff --git a/lingvo/tasks/mt/tools/wmt14.03.download_devtest.sh b/lingvo/tasks/mt/tools/wmt14.03.download_devtest.sh deleted file mode 100755 index 0153c428..00000000 --- a/lingvo/tasks/mt/tools/wmt14.03.download_devtest.sh +++ /dev/null @@ -1,25 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. wmt14_lib.sh - -mkdir -p "${ROOT}/raw" -echo " -http://www.statmt.org/wmt14/test-filtered.tgz -http://www.statmt.org/wmt14/dev.tgz" \ - | aria2c -x16 -s1 -j1 --dir="${ROOT}/raw" -i - diff --git a/lingvo/tasks/mt/tools/wmt14.04.unpack_train.sh b/lingvo/tasks/mt/tools/wmt14.04.unpack_train.sh deleted file mode 100755 index 16953e2d..00000000 --- a/lingvo/tasks/mt/tools/wmt14.04.unpack_train.sh +++ /dev/null @@ -1,28 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. wmt14_lib.sh - -mkdir -p "${ROOT}/unpacked" -for file in \ - training-parallel-commoncrawl.tgz \ - training-parallel-europarl-v7.tgz \ - training-parallel-nc-v9.tgz -do - tar xzf "${ROOT}/raw/${file}" -C "${ROOT}/unpacked" -done diff --git a/lingvo/tasks/mt/tools/wmt14.05.unpack_devtest.sh b/lingvo/tasks/mt/tools/wmt14.05.unpack_devtest.sh deleted file mode 100755 index 4bf80c25..00000000 --- a/lingvo/tasks/mt/tools/wmt14.05.unpack_devtest.sh +++ /dev/null @@ -1,24 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. wmt14_lib.sh - -mkdir -p "${ROOT}/unpacked" -for file in dev.tgz test-filtered.tgz; do - tar xzf "${ROOT}/raw/${file}" -C "${ROOT}/unpacked" -done diff --git a/lingvo/tasks/mt/tools/wmt14.06.tokenize_train.sh b/lingvo/tasks/mt/tools/wmt14.06.tokenize_train.sh deleted file mode 100755 index 89a4ebbd..00000000 --- a/lingvo/tasks/mt/tools/wmt14.06.tokenize_train.sh +++ /dev/null @@ -1,44 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. wmt14_lib.sh - -mkdir -p "${ROOT}/tokenized/train" - -function clean_train { - local base=$1 - local output_base="${ROOT}/tokenized/train/${base}" - - # Requires both directions to run and will output both. - "${TRAIN_CLEANER}" "${output_base}" ${SRC} ${TGT} \ - "${output_base}.clean" 1 100 - "${TRAIN_CLEANER}" "${output_base}" ${TGT} ${SRC} \ - "${output_base}.clean" 1 100 -} - -# TODO(drpng): eliminate en/de patterns. -tokenize "commoncrawl.de-en.de" de train commoncrawl -tokenize "commoncrawl.de-en.en" en train commoncrawl -tokenize "training/europarl-v7.de-en.de" de train europarl-v7 -tokenize "training/europarl-v7.de-en.en" en train europarl-v7 -tokenize "training/news-commentary-v9.de-en.de" de train news-commentary-v9 -tokenize "training/news-commentary-v9.de-en.en" en train news-commentary-v9 - -clean_train commoncrawl -clean_train europarl-v7 -clean_train news-commentary-v9 diff --git a/lingvo/tasks/mt/tools/wmt14.07.tokenize_devtest.sh b/lingvo/tasks/mt/tools/wmt14.07.tokenize_devtest.sh deleted file mode 100755 index 09e8d5a0..00000000 --- a/lingvo/tasks/mt/tools/wmt14.07.tokenize_devtest.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. wmt14_lib.sh - -# --- TEST SET. -mkdir -p "${ROOT}/tokenized/test" -function sgm_to_input { - local file=$1 - local output="$2" - - cat ${ROOT}/unpacked/$file | ${INPUT_FROM_SGM} > "${ROOT}/unpacked/test/$2" -} - -# TODO(drpng): eliminate en/de patterns. -sgm_to_input test/newstest2014-deen-src.${SRC}.sgm newstest2014.${SRC} -sgm_to_input test/newstest2014-deen-ref.${TGT}.sgm newstest2014.${TGT} - -tokenize test/newstest2014.${SRC} ${SRC} test newstest2014 -tokenize test/newstest2014.${TGT} ${TGT} test newstest2014 - -# --- DEV SET. -mkdir -p "${ROOT}/tokenized/dev" -# While 2008 onwards are available, we only use 2013 as the dev set. The "input" -# file is already in the tar file, so no need to pull from the SGM. -tokenize dev/newstest2013.${SRC} ${SRC} dev newstest2013 -tokenize dev/newstest2013.${TGT} ${TGT} dev newstest2013 diff --git a/lingvo/tasks/mt/tools/wmt14.08.wpm_encode_train.sh b/lingvo/tasks/mt/tools/wmt14.08.wpm_encode_train.sh deleted file mode 100755 index de5f0554..00000000 --- a/lingvo/tasks/mt/tools/wmt14.08.wpm_encode_train.sh +++ /dev/null @@ -1,40 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. wmt14_lib.sh - -mkdir -p ${ROOT}/wpm - -function make_input_files_list { - local lang=$1 - local indir="${ROOT}/tokenized/train" - echo "${indir}"/{news-commentary-v9,commoncrawl,europarl-v7}.clean.${lang} \ - | tr ' ' ',' -} - - -SRC_FILES=$(make_input_files_list ${SRC}) -TGT_FILES=$(make_input_files_list ${TGT}) - -TRAIN_TFRECORDS="${ROOT}/wpm/train.tfrecords-%5.5d-of-%5.5d" - -# The encoding goes at about 200 sentences per second. -# So, 4.4M sentences will take about 6 hours to run on a single core. -wpm_encode "${SRC_FILES}" "${TGT_FILES}" 200 16 ${TRAIN_TFRECORDS} - -cp -f "${WPM_VOC}" "${ROOT}/wpm" diff --git a/lingvo/tasks/mt/tools/wmt14.09.wpm_encode_devtest.sh b/lingvo/tasks/mt/tools/wmt14.09.wpm_encode_devtest.sh deleted file mode 100755 index 9675904f..00000000 --- a/lingvo/tasks/mt/tools/wmt14.09.wpm_encode_devtest.sh +++ /dev/null @@ -1,33 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. wmt14_lib.sh - -mkdir -p ${ROOT}/wpm - -DEV_SRC_FILES="${ROOT}"/tokenized/dev/newstest2013.${SRC} -TEST_SRC_FILES="${ROOT}"/tokenized/test/newstest2014.${SRC} -DEV_TGT_FILES="${ROOT}"/tokenized/dev/newstest2013.${TGT} -TEST_TGT_FILES="${ROOT}"/tokenized/test/newstest2014.${TGT} -DEV_TFRECORDS="${ROOT}/wpm/dev.tfrecords" -TEST_TFRECORDS="${ROOT}/wpm/test.tfrecords" - -wpm_encode "${DEV_SRC_FILES}" "${DEV_TGT_FILES}" 0 1 ${DEV_TFRECORDS} -wpm_encode "${TEST_SRC_FILES}" "${TEST_TGT_FILES}" 0 1 ${TEST_TFRECORDS} - -cp -f "${WPM_VOC}" "${ROOT}/wpm" diff --git a/lingvo/tasks/mt/tools/wmt14_get_data.sh b/lingvo/tasks/mt/tools/wmt14_get_data.sh deleted file mode 100755 index a27c968a..00000000 --- a/lingvo/tasks/mt/tools/wmt14_get_data.sh +++ /dev/null @@ -1,27 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -./wmt14.01.download_moses_scripts.sh -./wmt14.02.download_train.sh -./wmt14.03.download_devtest.sh -./wmt14.04.unpack_train.sh -./wmt14.05.unpack_devtest.sh -./wmt14.06.tokenize_train.sh -./wmt14.07.tokenize_devtest.sh -./wmt14.08.wpm_encode_train.sh -./wmt14.09.wpm_encode_devtest.sh diff --git a/lingvo/tasks/mt/tools/wmt14_lib.sh b/lingvo/tasks/mt/tools/wmt14_lib.sh deleted file mode 100755 index f850da6c..00000000 --- a/lingvo/tasks/mt/tools/wmt14_lib.sh +++ /dev/null @@ -1,65 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -ROOT=/tmp/wmt14 - -SRC=en -TGT=de - -TOKENIZER="${ROOT}/mosesdecoder/scripts/tokenizer" -TRAIN_CLEANER="${ROOT}/mosesdecoder/scripts/training/clean-corpus-n.perl" -INPUT_FROM_SGM="${ROOT}/mosesdecoder/scripts/ems/support/input-from-sgm.perl" - -function tokenize { - local input_filepath="$1" - local lang="$2" - local set="$3" - local out="$4" - local output_base="${ROOT}/tokenized/${set}/${out}" - - echo "$input_filepath -> ${out}.${lang}" - zcat -f "${ROOT}/unpacked/${input_filepath}" | \ - "${TOKENIZER}"/tokenizer.perl \ - -threads 20 \ - -l ${lang} > \ - "${output_base}.${lang}" -} - -WPM_VOC=../wpm-${SRC}${TGT}.voc - -function wpm_encode { - local source_files="$1" - local target_files="$2" - local max_len="$3" - local num_shards="$4" - local output_template="$5" - - rm -f convert.FAILED - for n in $(seq "${num_shards}"); do - local shard_id=$((n - 1)) - local output_filepath=$(printf ${output_template} ${shard_id} ${num_shards}) - set -x - nice -n 20 python3 -m lingvo.tools.wpm_encode_file --wpm_filepath=${WPM_VOC} --source_filepaths="${source_files}" \ - --target_filepaths="${target_files}" --num_shards="${num_shards}" --shard_id="${shard_id}" \ - --max_len="$max_len" \ - --output_filepath="${output_filepath}" --logtostderr || touch convert.FAILED & - set +x - done - wait - ! [ -f convert.FAILED ] -} - -test -f ${WPM_VOC} || (echo 1>&2 "Could not locate wpm file: ${WPM_VOC}"; exit 2) diff --git a/lingvo/tasks/mt/tools/wmtm16.01.download_moses_scripts.sh b/lingvo/tasks/mt/tools/wmtm16.01.download_moses_scripts.sh deleted file mode 100755 index bd950f50..00000000 --- a/lingvo/tasks/mt/tools/wmtm16.01.download_moses_scripts.sh +++ /dev/null @@ -1,22 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. wmtm16_lib.sh - -mkdir -p "${ROOT}" -git clone https://github.com/moses-smt/mosesdecoder.git ${ROOT}/mosesdecoder diff --git a/lingvo/tasks/mt/tools/wmtm16.02.download_data.sh b/lingvo/tasks/mt/tools/wmtm16.02.download_data.sh deleted file mode 100755 index a50ab80b..00000000 --- a/lingvo/tasks/mt/tools/wmtm16.02.download_data.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. wmtm16_lib.sh - -mkdir -p "${ROOT}/raw" -# ============================================================================ -# Download WMT data. -# From the WMT16 Multimodal website (En-De): -echo " -http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/training.tar.gz -http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/validation.tar.gz -http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/mmt16_task1_test.tar.gz" \ - | aria2c -x16 -s16 --dir="${ROOT}/raw" -i - diff --git a/lingvo/tasks/mt/tools/wmtm16.03.unpack_data.sh b/lingvo/tasks/mt/tools/wmtm16.03.unpack_data.sh deleted file mode 100755 index 286e0c3a..00000000 --- a/lingvo/tasks/mt/tools/wmtm16.03.unpack_data.sh +++ /dev/null @@ -1,28 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. wmtm16_lib.sh - -mkdir -p "${ROOT}/unpacked" -for file in \ - training.tar.gz \ - validation.tar.gz \ - mmt16_task1_test.tar.gz -do - tar xzf "${ROOT}/raw/${file}" -C "${ROOT}/unpacked" -done diff --git a/lingvo/tasks/mt/tools/wmtm16.04.tokenize_data.sh b/lingvo/tasks/mt/tools/wmtm16.04.tokenize_data.sh deleted file mode 100755 index 4591edd1..00000000 --- a/lingvo/tasks/mt/tools/wmtm16.04.tokenize_data.sh +++ /dev/null @@ -1,43 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. wmtm16_lib.sh - -mkdir -p "${ROOT}/tokenized/" - -function clean_train { - local base=$1 - local output_base="${ROOT}/tokenized/${base}" - - # Requires both directions to run and will output both. - "${TRAIN_CLEANER}" "${output_base}" ${SRC} ${TGT} \ - "${output_base}.clean" 1 100 - "${TRAIN_CLEANER}" "${output_base}" ${TGT} ${SRC} \ - "${output_base}.clean" 1 100 -} - -tokenize train ${SRC} -tokenize train ${TGT} -tokenize val ${SRC} -tokenize val ${TGT} -tokenize test ${SRC} -tokenize test ${TGT} - -clean_train train -clean_train val -clean_train test diff --git a/lingvo/tasks/mt/tools/wmtm16.05.wpm_encode_data.sh b/lingvo/tasks/mt/tools/wmtm16.05.wpm_encode_data.sh deleted file mode 100755 index 2a15d01d..00000000 --- a/lingvo/tasks/mt/tools/wmtm16.05.wpm_encode_data.sh +++ /dev/null @@ -1,33 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -. wmtm16_lib.sh - -mkdir -p ${ROOT}/wpm - -# The encoding goes at about 200 sentences per second. -for BASE in "train" "val" "test" -do - echo "Encoding ${BASE}" - TFRECORDS="${ROOT}/wpm/${BASE}.tfrecords" - SRC_FILE="${ROOT}/tokenized/${BASE}.clean.${SRC}" - TGT_FILE="${ROOT}/tokenized/${BASE}.clean.${TGT}" - wpm_encode "${SRC_FILE}" "${TGT_FILE}" 200 1 ${TFRECORDS} -done - -cp -f "${WPM_VOC}" "${ROOT}/wpm" diff --git a/lingvo/tasks/mt/tools/wmtm16_get_data.sh b/lingvo/tasks/mt/tools/wmtm16_get_data.sh deleted file mode 100755 index eb4769b3..00000000 --- a/lingvo/tasks/mt/tools/wmtm16_get_data.sh +++ /dev/null @@ -1,23 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -eu - -./wmtm16.01.download_moses_scripts.sh -./wmtm16.02.download_data.sh -./wmtm16.03.unpack_data.sh -./wmtm16.04.tokenize_data.sh -./wmtm16.05.wpm_encode_data.sh diff --git a/lingvo/tasks/mt/tools/wmtm16_lib.sh b/lingvo/tasks/mt/tools/wmtm16_lib.sh deleted file mode 100755 index 73702f84..00000000 --- a/lingvo/tasks/mt/tools/wmtm16_lib.sh +++ /dev/null @@ -1,64 +0,0 @@ -#!/bin/bash -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -ROOT=/tmp/wmtm16 - -SRC=en -TGT=de - -TOKENIZER="${ROOT}/mosesdecoder/scripts/tokenizer" -TRAIN_CLEANER="${ROOT}/mosesdecoder/scripts/training/clean-corpus-n.perl" -INPUT_FROM_SGM="${ROOT}/mosesdecoder/scripts/ems/support/input-from-sgm.perl" - -function tokenize { - local input_prefix="$1" - local lang="$2" - local out=$input_prefix - local output_base="${ROOT}/tokenized/${out}" - - echo "$input_prefix.$lang -> ${out}.${lang}" - zcat -f "${ROOT}/unpacked/${input_prefix}.$lang" | \ - "${TOKENIZER}"/tokenizer.perl \ - -threads 20 \ - -l ${lang} > \ - "${output_base}.${lang}" -} - -WPM_VOC=../wpm-${SRC}${TGT}-2k.voc - -function wpm_encode { - local source_files="$1" - local target_files="$2" - local max_len="$3" - local num_shards="$4" - local output_template="$5" - - rm -f convert.FAILED - for n in $(seq "${num_shards}"); do - local shard_id=$((n - 1)) - local output_filepath=$(printf ${output_template} ${shard_id} ${num_shards}) - set -x - nice -n 20 python3 -m lingvo.tools.wpm_encode_file --wpm_filepath=${WPM_VOC} --source_filepaths="${source_files}" \ - --target_filepaths="${target_files}" --num_shards="${num_shards}" --shard_id="${shard_id}" \ - --max_len="$max_len" \ - --output_filepath="${output_filepath}" --logtostderr || touch convert.FAILED & - set +x - done - wait - ! [ -f convert.FAILED ] -} - -test -f ${WPM_VOC} || (echo 1>&2 "Could not locate wpm file: ${WPM_VOC}"; exit 2) diff --git a/lingvo/tasks/mt/wpm-ende-2k.voc b/lingvo/tasks/mt/wpm-ende-2k.voc deleted file mode 100644 index 80a37e61..00000000 --- a/lingvo/tasks/mt/wpm-ende-2k.voc +++ /dev/null @@ -1,2000 +0,0 @@ - - - -▁ -. -a -i -n -A -E -e -m -, -t -h -u -d -f -o -M -s -r -w -F -Z -T -g -p -l -v -c -ä -y -b -k -z -H -J -S -ß -G -P -ü -K -D -W -& -q -; -x -L -j -O -B -N -- -V -Y -R -ö -2 -I -C -U -3 -( -! -) -é -4 -5 -: -0 -1 -X -# -? -6 -Ö -8 -9 -Q -Ä -7 -/ -Ü -% -@ -ñ -$ -= -è -_ -à -▁. -▁a -▁A -▁E -▁m -▁, -▁t -▁h -▁d -▁f -▁M -▁s -▁r -▁w -▁F -▁T -▁g -▁p -▁l -▁v -▁b -▁H -▁S -▁K -▁B -in -it -is -ir -il -ea -en -em -et -eh -ed -es -er -el -un -ut -uf -on -om -ot -of -or -ow -ch -ck -▁i -▁n -▁Z -▁c -▁k -▁z -▁G -▁P -▁in -▁is -▁on -▁of -an -at -au -ad -as -ar -al -ay -ack -ei -eb -ein -ou -os -op -ol -▁an -▁ar -▁auf -▁Ein -▁mit -▁th -▁si -▁st -▁sh -▁sp -▁sit -▁wh -▁wit -▁wom -▁Fr -▁gr -▁bl -▁St -ers -und -ung -chen -▁o -▁y -▁J -▁D -▁W -▁L -▁j -▁ein -▁und -am -ag -ach -ie -id -ig -ic -iel -ts -ten -ter -hr -um -us -ur -wo -wei -le -lay -▁man -▁to -▁tr -▁der -▁Mä -▁Man -▁sch -▁bo -▁Sch -ind -ing -ild -ear -▁im -ass -alk -▁and -▁are -▁Eine -▁the -▁woman -▁Frau -▁R -▁ch -im -ik -ier -ue -up -oo -st -ra -re -ge -gt -äl -ähr -ber -ßen -▁at -▁dem -▁die -▁wear -▁Two -▁pe -▁play -▁von -▁vor -ite -irt -irl -eht -ehen -est -ell -own -▁Zwei -▁kl -and -all -ople -old -▁sich -▁spiel -▁sitz -▁with -erson -▁eine -▁einen -▁einem -▁einer -▁Mann -▁O -ap -ab -ahr -ich -ner -eg -eit -ting -her -og -oung -war -ver -king -▁his -▁her -▁dr -▁fr -▁red -▁wat -▁walk -▁girl -▁br -▁Hund -▁Kind -end -ens -ess -▁zu -▁Gr -▁Person -ang -eid -oup -▁steht -▁stand -▁shirt -▁whil -▁white -▁black -▁blue -▁Stra -▁out -▁Jung -▁Mäd -▁boy -▁wearing -▁people -▁klein -▁un -ain -air -ick -nner -ee -ug -ul -ft -ock -ound -sch -rei -ball -über -▁al -▁hold -▁down -▁dog -▁Mens -▁sein -▁rot -▁wei -▁loo -▁bei -▁Sp -itt -isch -ent -emd -ont -▁cl -ard -ark -▁stre -▁sitting -▁group -▁blau -▁young -▁child -upp -ält -ährend -▁playing -▁sitzt -warz -▁walking -▁standing -▁Straß -▁Mädchen -▁& -▁V -▁C -▁or -▁um -▁über -auf -ion -ne -nen -men -te -he -de -den -der -for -gr -leid -ce -ben -qu -▁as -▁aus -▁men -▁hat -▁hält -▁den -▁for -▁während -▁be -▁Hemd -ink -other -▁ihr -▁cr -eben -▁spr -▁jung -ide -▁Männer -▁schwarz -▁front -▁while -▁Junge -▁holding -▁Menschen -▁street -uppe -▁Straße -▁N -▁up -af -ak -aar -iv -ex -een -sp -side -gh -ct -än -bo -ür -▁am -▁An -▁das -▁dur -▁fl -▁sm -▁som -▁run -▁The -▁by -inter -ean -ead -ehr -uft -▁neben -▁car -ange -▁thr -▁gro -▁Wass -▁eines -acht -igh -ump -ingt -▁ander -▁Frauen -▁spielt -▁dress -▁water -▁Kinder -▁Gruppe -▁Personen -form -av -ip -ies -hing -uit -oot -sen -ren -rad -beit -kat -kleid -ze -▁sw -▁wor -▁ge -▁lä -▁back -▁ball -▁Hand -each -ern -ere -one -ong -▁ist -▁one -arben -ough -oll -▁stehen -▁women -▁green -▁Drei -▁jump -imm -iert -raun -ährt -▁spielen -▁sitzen -icht -▁roten -▁weißen -▁looking -ittle -entl -▁blauen -▁orange -▁durch -▁entl -ian -ew -eut -eck -to -hn -hirt -fer -ägt -yc -ying -bl -ks -▁mo -▁macht -▁two -▁fährt -▁sn -▁ro -▁rid -▁Fahr -▁Thr -▁gel -▁geht -▁pos -▁little -▁ver -▁vers -▁bu -▁braun -▁Kleid -ill -enn -▁co -arge -our -▁side -▁grü -▁over -amer -ood -ellow -▁Ober -▁brown -▁outside -▁kleines -▁weiß -▁through -▁Wasser -uß -uch -fen -och -oard -se -lass -br -zen -ück -▁hand -▁ra -▁person -▁large -▁Br -ine -etw -eln -elt -ot; -▁zwei -▁into -ank -ast -art -▁blon -▁yellow -▁Leut -ame -age -agen -igen -▁trägt -▁their -str -▁from -▁watch -▁kleiner -▁unter -▁&qu -teil -▁springt -▁schwarzen -ext -ight -▁entlang -▁snow -▁Three -▁build -amera -▁I -▁U -▁old -▁other -aut -if -ning -ef -ep -ey -hl -uck -ro -gen --S -▁wir -▁Fel -▁gehen -▁pl -▁bik -▁Bl -ischen -esch -unt -ort -owd -▁near -▁next -▁con -▁zus -ant -auen -ask -acket -ebä -eboard -▁grass -amp -älter -ull -▁Spiel -▁children -spiel -▁small -eans -▁dressed -kleidet -▁Fahrrad -▁" -▁building -▁it -▁etw -▁älter -tur -oc -oor -walk -ger -gel -ly -ler -▁Ar -▁hin -▁sc -▁Fuß -▁pink -▁Se -▁Ball -ille -eam -unter -otor -che -▁nach -▁Pe -▁off -ale -acke -oss -▁another -▁around -▁str -▁Str -▁jacket -ield -aby -able -▁seinem -grund -ground -▁crowd -▁some -▁running -ehrere -▁großen -kateboard -▁riding -▁Oberteil -▁Leute -ebäu -▁er -hone -fahr -farben -ool -pp -cer -zt -▁air -▁As -▁talk -▁field -▁Mehrere -▁Fl -▁Tr -▁per -▁lad -▁beh -▁beach -▁Hinter -▁Bau -ity -ist -ett -elm -orts -▁Gebäu -alten -ose -os; -▁sits -▁Frei -▁Der -▁Luft -ige -▁Schn -ieren -ander -▁stands -▁&ap -▁junge -▁smil -zeug -▁gekleidet -▁läuft -▁sidewalk -▁grünen -lasses -▁etwas -▁People -aking -iat -ect -mer -ud -way -weg -les -lat -lich -bei -ün -ör -▁table -▁hinter -▁für -▁sk -▁sur -▁sind -▁pic -▁park -▁ben -▁Bo -itar -unk -omp -oth -eball -ost -opf -▁arbeit -▁Jacke -▁schaut -▁Jungen -board -▁anderen -▁work -▁läch -▁background -▁jumping -ycle -▁Kleidung -▁blond -asket -▁Strand -▁perform -▁behind -▁Hintergrund -▁Freien -▁' -ace -iform -ub -fl -fit -cr -cing -üt -▁top -▁head -▁hoch -▁do -▁des -▁dan -▁rock -▁read -▁wall -▁Tisch -▁pr -▁bic -▁Haar -▁Kopf -eat -etr -▁city -ation -aus -aur -ade -aseball -olor -umen -▁Schl -▁player -▁vorbei -egen -▁girls -▁looks -aufen -▁junger -ake -▁carr -ennis -ucht -▁In -▁bike -untain -spieler -▁pictur -▁'s -▁en -achen -ied -uh -wimm -get -be -zug -üh -ühr -▁mid -▁tan -▁hair -▁T-S -▁bus -▁Hut -▁Son -▁Kl -oto -▁ih -▁camera -▁that -▁sie -▁Stad -amm -iger -▁schl -estaur -veral -▁Gras -angen -▁kleine -▁kleinen -▁dogs -▁beim -menge -▁Nä -ive -▁somet -▁groß -▁andere -uitar -ootball -▁gelben -▁wird -occer -▁Fußball -▁talking -▁Gebäude -asketball -ahn -iz -ping -ling -äft -kt --h -▁ad -▁Es -▁hor -▁food -▁soccer -▁ros -▁gu -▁guitar -▁bar -▁Kamera -etter -ert -▁color -▁Geh -▁Gitar -▁Park -ants -alt -eig -▁gray -▁blick -erst -ammen -iegt -ust -▁toget -▁walks -▁Hunde -▁boys -▁along -ische -▁bef -▁Menschenmenge -▁working -ennt -▁lady -▁Schnee -▁smiling -▁Boden -▁T-Shirt -estaurant -▁Nähe -▁something -▁el -▁op -▁us -ait -ev -eer -uct -dle -door -ob -ok -gn -är --s -▁ab -▁has -▁him -▁sl -▁sing -▁sand -▁rennt -▁get -▁game -▁pol -▁ber -▁Skateboard -▁Ber -▁comp -ath -atch -eich -▁them -▁Die -▁plays -▁Straßen -▁uniform -▁holds -▁Vier -▁ihren -▁schwarzer -beiter -oller -▁road -▁older -farbenen -▁Asian -latz -▁bench -▁guy -▁together -▁Y -▁ea -rop -ve --wei -▁abo -▁Aut -▁dirt -▁dunk -▁Motor -▁suit -▁skateboard -▁Four -▁go -▁gest -▁glasses -▁pur -▁pain -▁pants -▁li -▁Sand -▁Büh -ish -ette -edeck -ore -▁coup -▁kid -▁Pfer -ann -ate -ald -eing -einander -▁stage -▁shorts -uss -▁herum -▁clim -▁asiat -▁ihre -▁Feld -▁zusammen -fahrer -▁carrying -▁picture -▁Sonnen -▁Stadt --hair -▁Gitarre -az -eu -ec -eil -fr -▁Er -▁mus -▁taking -▁hell -▁fem -▁fahr -▁face -▁set -▁sehen -▁wait -▁Foto -▁gl -▁pool -▁lea -▁long -▁baby -▁Bür -▁Baby -ennen -elle -▁Ges -ather -auer -außen -eigt -olle -▁sign -▁Jeans -▁tree -eitet -ogra -▁outdoor -kleidung -▁jumps -▁mountain -▁Brille -▁unterh -▁älterer -▁Several -▁bicycle -▁Gehweg -▁befind -▁Bühne -▁couple -▁climb -▁each -ail -ien -ieht -wagen -bt -üs -▁hav -▁dab -▁dark -▁Mik -▁sun -▁ph -▁bel -▁bag -ire -irts -erk -uth -otogra -owing -▁Gel -aum -osen -olz -▁area -▁shop -achsen -ice -ican -▁boat -▁chair -app -eiten -▁draußen -enden -ockey -▁als -▁seine -▁seinen -▁Ch -achtet -▁coat -▁race -▁watching -▁hinunter -cross -▁doing -▁blickt -▁fahren -▁ex -ac -ichen -tel -haar -dy -pr -per -ves -jo -▁across -▁mic -▁motor -▁male -▁tow -▁team -▁he -▁helm -▁day -▁Mus -▁wind -▁liegt -▁band -▁Hän -▁Bild -▁Bank -▁Baseball -ome -▁Paar -arten -▁ground -▁Wand -▁jeans -▁under -tern -▁train -▁Schwimm -ischer -▁versucht -▁braunen -▁weißer -brille -▁hands -▁Un -▁strip -▁Baum -▁arbeitet -▁lächeln -▁middle -▁dabei -nt -uhl -ff -par -ple -lt -ät -▁machen -▁dar -▁wie -▁wav -▁par -▁pass -▁laufen -▁baseball -▁being -▁Holz -▁Sh -▁Boot -isten -irm -utz -ons -cht -▁kur -asche -als -ebt -▁who -▁ob -▁Dame -amil -▁toy -▁tragen -▁schauen -ingen -ree -▁att -essen -▁mouth -▁room -▁brauner -eschäft -▁lächelt -ührt -▁große -▁Essen -▁eating -▁female -▁helmet -ah -wachsen -geb -gerst -äs -kauf -▁Auf -▁football -▁Men -▁Th -▁gather -▁pre -▁le -▁betr -▁Hü -ition -eten -▁kn -ance -ase -ouch -ope -▁arm -▁sho -▁Wom -ike -ikot -ult -▁all -arkt -▁ihrem -akes -▁swea -▁swing -▁orangefarbenen -▁versch -struct -▁performing -ütze -▁Klein -▁ihm -▁horse -▁using --weiß -▁gestrei -▁purple -▁Pferd -▁music -▁microp -▁striped -aft -od -over -ßt -ül -üst -▁ag -▁halten -▁wood -▁Tennis -▁past -▁Hose -▁Sk -emp -ows -▁zeigt -att -augh -▁sieht -▁store -▁oc -tst -ument -usch -▁Rad -ikan -▁kletter -aper -here -▁look -▁cross -▁spricht -▁junges -▁runs -iese -▁cover -▁grün -▁ältere -▁arbeiten -▁enjo -▁Young -▁Motorrad -▁liest --haired -▁Bürgerst -▁unterhalten -▁Mikr -▁toward -▁prepar -struction -aw -iet -iver -ees -piel -platz -▁app -▁mar -▁tennis -▁fen -▁Meer -▁wer -▁Tag -▁There -▁gegen -▁put -▁lay -▁viel -▁basketball -▁Band -ines -ott -othing -▁catch -▁Gew -ous -▁three -▁star -unglasses -▁Wald -▁schn -▁players -▁drei -ensch -enster -aff -afarben -iving -boy -▁swimm -efon -▁its -▁Trikot -▁reading -▁Haaren --shirt -▁Berg -▁Musik -▁microphone -▁Bürgersteig -mp -round -gest -by -▁mark -▁se -▁sunglasses -▁restaurant -▁part -▁bes -▁bun -▁bear -▁Helm -▁Hosen -▁Kun -ement -ete -eren -eld -els -▁can -▁cap -▁zum -asch -▁Fris -▁Stein -ure -leep -▁schw -stand -repp -▁bright -▁cloth -▁clothing -derly -▁beob -▁during -▁flow -ecken -▁watches -▁gekleidete -ostü -▁rosa -▁kids -▁Sonnenbrille -olley -▁climbing -▁motorc -▁again -▁ocean -ax -aun -ix -ial -eet -ubl -ften -ful -oh -ry -ph -las --B -▁t-shirt -▁hill -▁singt -▁Football -▁Basketball -omm -ofon -ord -owboy -cher -▁Zusch -▁Pol -anz -arm -▁stick -▁grauen -▁outfit -▁seiner -▁blauem -▁umbr -dern -ängt -▁rides -▁gelb -▁weißem -strument -ucks -▁Felsen -▁Arm -▁dancing -▁ihnen -▁elderly -euer -▁Erwachsen -▁Gesicht -olleyball -▁against -top -uchen -des -ser -ging -pf -äuf -zer -öff -▁high -▁dis -▁führt -▁pro -▁bow -emand -ork -▁cell -▁kick -aul -auk -ouse -▁track -▁trink -▁trees -▁Mantel -▁Schild -▁they -ooden -raw -▁One -▁Spr -▁floor -▁posing -▁posiert -ullover -▁Spielzeug -ünf -▁surf -fiti -uhen -▁about -▁Auto -▁having -▁Gelän -▁window -▁darauf -▁Shorts -▁covered -▁Mikrofon -▁umbrell -▁2 -iol -ift -ment -rä -pack -vers -ving -bee -braun -▁tel -▁wooden -▁Fenster -▁bat -▁Bä -itten -iss -uter -ors -cke -▁cost -▁zw -▁zur -▁Pf -asen -igar -ume -▁trying -▁Restaurant -▁blauer -▁schwarzem -eaning -ycl -▁very -▁wirft -▁Blumen -▁construction -ampf -örper -▁workers -▁busy -▁colorful -▁open -▁getting -▁dunkel -▁waiting -▁befindet -▁klettert -▁swimming -▁market -▁ev -ahl -ib -sh -lichen -kind -ject -▁Eis -▁met -▁sehr -▁rennen -▁wr -▁Feuer -▁pull -▁path -▁bal -▁Bad -ene -err -utsch -▁ice -▁new -▁night -▁kit -▁Pullover -atue -arb -arbeiter -▁shirts -▁oder -▁Das -▁Weg -achten -ießt -use -urn -▁trick -▁book -▁Rei -▁Raum -eep -eenag -▁cart -▁versamm -ücke -iff -▁singing -▁This -▁Woman -▁fence -▁es -ium -sel -phone -len -äut -▁af -▁hang -▁Met -▁son -▁sleep -▁Trepp -▁gen -▁pie -▁line -▁light -▁langen -▁Kü -ehnt -off -▁Zeit -▁cut -▁isst -ash -▁stair -unden -ungen -▁Wiese -ush -▁truck -▁board -asse -aben -▁brick -schaft -▁rotem -▁Sport -▁Männern -▁Anzug -▁flag -▁anderer -▁Kindern -▁rail -artet -▁zusch -▁Arbeiter -▁gekleideter -others -▁rosafarben -▁laying -▁essen -▁others -ient -ountain -yp -▁Ab -▁mach -▁making -▁fest -▁Maul -▁ride -▁river -▁wartet -▁paper -▁bed -▁Sn -▁Sur -▁Some -eak -ond -▁cour -▁kar -▁ins -▁inside -▁Wei -ief -▁body -▁Schutz -▁Rück -▁Rucks -▁Ob -▁drink -▁roter -▁Ver -▁hats -▁schwarze -▁backpack -▁posieren -▁weiße -▁Bluse -licher -▁surround -▁gestreiften -▁beard -▁Frisbee -▁clothes -▁Poliz -äufer -▁cellphone -▁kitchen -▁e -▁ent -hal -dler -becken -kl -körper -zeit -ße -▁takes -▁hel -▁full -▁sell -▁skat -▁rep -▁Fünf -▁gold -▁gesch -▁laugh -▁voller -▁big -▁Buch -ots -aden -▁art -▁step -▁grill -▁Stuhl -ersch -iew -tshirt -ooking -raf -este -▁Vor -▁Viel -▁auss -änger -▁Handy -▁grassy -▁Spieler -▁Fluss -liche -boarder -▁beleb -▁towards -▁werden -▁motorcycle -▁umbrella -▁metal -eis -eel -ech -oop -stst -wand -gang -ym -band -▁Afr -▁tank -▁hard -▁sil -▁warten -▁Five -▁ger -▁viol -▁Haus -ini -▁ind -iegel -igt -ummer -▁schi -▁box -ierten -ook -ellen -ogen -▁überqu -▁springen -änk -ipp -immer -▁Oberkörper -▁blonde -strumen -▁Um -▁Seil -▁bereitet -▁compet -▁paint -▁betrachtet -▁Hügel -▁Kleinkind -▁enjoying -▁zwischen -igarette -▁Metall -▁Snow -aid -ny -häut -▁door -▁fis -▁Mütze -▁sub -▁reitet -▁po -▁lake -▁lehnt -▁bea -▁beid -▁Kr -▁Kar -ird -ile -utter -▁zieht -▁Pap -ans -ater -ating -eine -▁she -▁Stock -▁oben -eiter -▁drin -ende -ults -▁alter -ecke -▁Leuten -▁Asiat -▁Bauarbeiter -▁ready -▁tanzen -▁schläft -altung -▁above -▁painting -edeckten -▁photogra -▁Child -▁sweater -▁schneid -▁several -▁piece -▁Viele -iend -ekt -man -dr -fall -oon -ohn -bar -kor -▁ac -▁dav -▁fall -▁Mitt -▁paar -▁lil -▁leg -▁lang -▁vest -ets -esen -▁cor -▁Pl -▁onto -aschen -▁miteinander -▁stone -▁many -▁Schul -ings -▁Renn -▁Rennen -ellt -▁Or -▁herunter -▁brid -▁class -▁umg -immt -ichtung -▁buck -▁snowy -▁Arbeit -▁dance -▁tanzt -▁colored -▁shopping -▁Händen -▁beobachtet -▁Bäumen -▁Rucksack -enen -ones -ane -ender -▁going -▁wave diff --git a/lingvo/tasks/mt/wpm-ende.voc b/lingvo/tasks/mt/wpm-ende.voc deleted file mode 100644 index 6344fe11..00000000 --- a/lingvo/tasks/mt/wpm-ende.voc +++ /dev/null @@ -1,32000 +0,0 @@ - - - -▁ -, -. -t -h -e -i -n -o -f -d -r -a -u -s -z -v -ü -m -( -) -w -T -b -I -S -: -c -& -q -; -- -D -y -l -/ -E -p -! -? -U -k -ö -W -– -’ -M -ß -K -C -“ -H -P -ä -% -# -9 -1 -3 -g -B -„ -A -2 -j -” -F -Z -Y -J -R -0 -5 -4 -L -x -O -� -V -N -8 -G -* -6 -7 -+ -_ -® -Q -▅ -€ -‘ -$ -= -{ -» -Ä -… -} -« -— -à -Ö -­ -@ -° -´ -² -™ -Ü -• -X -\ -¤ -§ -· -© -¼ -è -¶ -é -‚ -£ -ô -á -и -в -† -~ -ú -¿ -³ -ã -н -а -С -½ -ó -̈ -→ -â -í -å -− -ù -à -ç -ê -Č -ý -› -Š -± -× -ñ -^ -с -¡ -‑ -е -Г -ј -‹ -ń -Ђ -д -л -я -ğ -¬ -÷ -Á -В - -ę -¥ -ł -¹ -œ -č -п -о -з -ò -˙ -Ø -Ε -ï -т -¨ -µ -ч -đ -ð -æ -● -ś -ć -κ -α -ι -Å - -Ţ -ă -у -ő -É -ø -к -′ -š -τ -ο -Р -ž -ë -‟ -ř -¦ -″ -¾ -ě -э -î -Ő -№ -← -õ -Ł -ź -Ó -※ -À -， -ş -‡ -Ê -¢ -⇒ -่ -ľ -º -ì -。 -ů -ν -ė -ы -μ -ε -ь -р -Ú -υ -ň -Ћ -г -` -ą -м -ж -й -і -ż -Ç -Ν -Α -Τ -Ο -İ -י -η -û -π -ό -σ -Н -ı -б -中 -λ -θ -ή -Ž -Д -ű -β -的 -م -ن -ţ -Î -γ -ς -ί -х -ц -ر -Ñ -џ -Ќ -ร -Ð -א -Í -Ś -Ѓ -ī -δ -ا -È -Э -φ -ρ -ά -ŕ -خ -ت -ل -د -و -س -ع -ب -А -П -З -О -щ -ª -fi -ي -เ -Ş -Е -ū -Ý -ω -ю -ו -Ÿ -У -أ -М -إ -ج -า -И -К -น -ā -ك -έ -ξ -Љ -ї -ע -ō -Ï -Ô -ÿ -Ф -ť -Þ -ĺ -Đ -ύ -Л -ר -ة -Т -ф -ف -ш -ď -Б -ม -Ι -є -ش -ه -ح -Ż -ی -ง -อ -ē -ώ -χ -þ -І -َ -ъ -ל -њ -Я -fl -Х -ِ -ņ -▁, -▁. -▁t -▁h -▁n -▁f -▁d -▁r -▁a -▁s -▁v -▁m -▁w -▁T -▁b -▁I -▁S -▁D -▁E -▁p -▁M -▁P -▁g -▁B -▁A -et -en -ed -er -ea -es -em -el -it -in -ir -is -il -ig -ot -on -of -or -om -ol -ut -un -ur -us -ch -▁e -▁o -▁z -▁c -▁& -▁l -▁k -▁W -▁K -▁C -▁H -▁F -▁R -▁V -▁G -▁in -▁is -▁of -▁un -ie -iv -im -ic -ier -ies -ion -ich -de -der -at -an -ar -as -am -al -uf -um -ul -ür -ber -ge -▁th -▁to -▁for -▁der -▁an -▁al -▁be -▁pr -ent -▁y -▁U -▁L -▁N -▁on -▁im -he -id -ow -op -ft -re -ra -st -sch -ben -ct -ce -qu -le -ly -kt -ke -▁ha -▁den -▁die -▁das -▁re -▁st -▁se -▁von -▁wir -▁In -ere -ern -est -ine -ing -ist -isch -ort -omm -ung -▁zu -▁und -icht -ation -and -▁the -▁and -▁( -▁) -▁1 -▁2 -▁Z -▁er -▁it -te -tr -ten -ter -hr -her -if -os -den -ab -ay -ap -ach -up -uch -ve -ver -ck -lich -gen -▁ne -▁für -▁de -▁dies -▁as -▁auf -▁are -▁ver -▁mit -▁we -▁wit -▁The -▁Pr -▁ge -ess -iss -urop -▁ein -▁con -▁Ver -▁that -▁Un -ungen -▁: -▁j -▁J -▁O -▁or -eh -eg -eit -ect -ib -ied -ial -for -ro -rea -ag -ug -über -pl -port -än -▁not -▁nicht -▁des -▁dem -▁sich -▁wh -▁St -▁Sie -▁Europ -▁An -end -ers -ell -ind -ill -und -▁ex -▁eine -▁ist -ant -ann -art -▁this -▁als -▁all -▁pro -▁yo -apos -▁with -▁- -▁es -▁über -ein -ik -ien -iel -nen -od -oc -ff -ah -ad -ain -ub -men -ment -pr -gr -00 -▁fr -▁at -▁ab -▁auch -▁sh -▁wer -▁will -▁Th -▁by -▁Sch -▁Die -ear -ity -igh -igen -oll -ust -▁com -▁uns -ions -ass -all -uld -quot -▁have -▁dass -ischen -ommiss -ungs -apos; -▁us -▁ch -▁um -▁ent -ts -tern -hal -iz -ip -fer -mer -we -la -ön -▁has -▁da -▁aus -▁so -▁sp -▁sch -▁sind -▁vor -▁wor -▁was -▁En -▁Er -etz -enn -ens -ean -esch -ement -ure -▁co -▁" -▁' -▁We -ive -ate -ang -ard -ident -scha -▁wird -ische -▁which -▁you -▁from -▁werden -ommission -▁i -▁en -▁ich -ew -ec -ech -ov -ob -ok -og -ous -och -ost -ould -ran -aus -age -ahr -ud -ss -pt -ät -▁hat -▁res -▁am -▁sie -▁man -▁Mit -▁Par -▁Aus -ert -erv -otel -urch -▁oder -▁can -ick -ast -ament -ult -heit -keit -▁haben -▁200 -able -lichen -form -▁European -▁" -schaft -▁/ -▁! -▁? -th -ia -out -our -fü -ue -ze -beit -cc -▁nach -▁ad -▁me -▁ma -▁wie -▁Das -▁Ein -▁pol -▁part -▁Be -▁Ar -▁Al -err -eas -eal -ition -ise -ige -ore -ord -ome -uss -▁our -▁zur -▁cl -▁kön -▁Wir -▁unter -ice -ical -ieren -dern -aten -▁sein -ations -ational -▁einen -▁Parl -▁3 -▁qu -hn -hen -eut -ekt -fen -ren -rit -ac -ail -act -uct -sp -zen -ün -yst -▁do -▁durch -▁ap -▁wur -▁bet -▁but -▁It -▁Ich -▁Sp -▁pl -▁Ber -▁Ab -ember -elt -ign -other -▁&# -▁Fra -iert -ated -ates -ark -▁bei -schl -▁reg -▁einer -▁Union -ellen -▁also -▁your -ight -▁comp -▁können -▁– -tw -tra -iet -ide -ian -oun -ak -uß -ual -se -vel -be -cl -per -pro -är -gl -ger -▁tra -▁mö -▁more -▁would -▁best -▁EU -▁Mr -▁gr -▁gen -ite -ish -chen -▁zum -▁le -▁Commission -▁Herr -▁Vor -▁Gr -ime -imm -are -ary -▁there -ommen -▁their -▁19 -ose -liche -▁einem -ugh -ystem -▁’ -hl -ep -eil -fl -au -ack -ück -üss -bl -ör -äs -äischen -▁.. -▁nur -▁fin -▁dis -▁dar -▁ar -▁ag -▁acc -▁war -▁gro -▁Auf -erm -ely -itt -ong -cht -▁comm -▁li -▁kann -▁Hotel -anz -ank -ance -amm -amp -▁ander -▁been -sten -▁its -oss -reat -ahl -▁This -▁coun -▁'s -▁ihr -▁ -▁; -▁up -▁other -ner -rat -auf -üt -con -ähr -gt -gan -glich -▁noch -▁rel -▁sup -▁soll -▁viel -▁Der -▁Dies -▁Es -▁go -ese -ild -igkeit -olle -ution -che -chte -▁zw -▁Kon -▁Ch -▁For -▁Reg -▁Ge -ierung -ame -geb -▁one -ste -▁new -▁diese -▁dieser -▁jed -▁Jahr -▁should -▁unser -▁daß -▁work -äsident -▁... -▁“ -▁5 -▁4 -to -hre -ient -zu -bar -äl -ges -▁tr -▁fre -▁aber -▁act -▁sa -▁must -▁wenn -▁per -emein -ink -one -ors -unkt -▁ev -▁out -▁ke -▁Kommission -▁Re -▁ind -▁inter -▁intern -▁Um -▁Le -str -schen -ings -▁they -▁need -▁very -▁weit -▁Pres -▁Präsident -ehen -▁prod -echn -formation -▁mehr -▁wurde -üssen -▁countr -▁Y -▁em -fe -uen -ukt -me -wo -ces -ject -▁time -▁ro -▁sow -▁müssen -▁bis -ett -eten -ead -eln -ities -ität -ile -ons -olog -utz -▁ob -▁op -ieß -ater -ammen -alt -▁alle -▁prop -▁year -▁only -reich -▁sehr -▁Pro -▁cont -▁cons -ehmen -ible -▁about -assen -etzt -rans -▁Parlament -▁Bericht -glied -▁like -▁President -▁et -▁el -tschaft -hnen -eop -nah -oy -oin -ook -fahr -ri -ring -wend -▁hin -▁fir -▁dir -▁sol -▁serv -▁Ma -▁Men -▁Member -▁ges -erg -ism -onder -usammen -▁eff -▁la -ionen -atz -ument -ular -▁these -▁any -▁bec -▁import -iden -sta -leich -▁rec -▁diesem -▁gegen -▁prov -ubl -gramm -▁Mitglied -führ -▁polit -▁Arbeit -velop -▁if -ht -ider -ious -iament -av -ake -wer -ced -cess -ld -lin -land -ks -get -xt -▁no -▁such -▁Im -▁Ihr -each -ele -els -ility -igt -iger -olit -unden -ures -▁off -att -ange -▁them -entr -own -stit -▁report -▁20 -▁Zeit -▁develop -▁Unter -ibt -fore -änd -▁who -▁Europäischen -erson -halten -ause -eople -▁% -▁Ü -ne -fol -ru -aff -zeit -ühr -mal -bs -cy -ph -ken -gest -▁he -▁hier -▁sel -▁sub -▁my -▁were -▁well -▁people -▁Pl -ete -esent -itu -its -ond -▁over -▁Rech -▁ins -▁into -▁under -iew -iven -ative -arbeit -ree -rei -stell -sche -schied -ific -ays -acht -▁gew -▁Europe -▁Parliament -▁möchte -▁international -ha -io -fr -ful -dition -sicht -wn -cil --S -lem -lung -äft -▁trans -▁hotel -▁dif -▁real -▁area -▁system -▁vom -▁mark -▁wicht -▁bed -▁bere -▁De -▁Mar -ene -enden -elle -▁europ -▁kn -▁kon -▁Fr -▁Ges -▁inv -ices -ional -icher -ase -▁eines -▁Or -ieder -▁what -▁States -▁use -▁resp -▁groß -▁support -▁product -olitik -▁Über -▁6 -▁Q -ou -akt -vent -mitt -cr -kl -äch -1; -gel -▁her -▁hel -▁dam -▁some -▁most -▁wel -▁bel -etr -esonder -itte -inn -ory -old -ustr -▁cr -▁lie -▁loc -▁Coun -▁Fl -▁incl -ict -ats -▁than -▁Land -stüt -▁str -isten -very -änder -▁when -ments -halt -▁spec -▁Entw -▁ihre -conom -▁countries -▁sowie -▁Menschen -ever -lo -kommen -ös -äische -3; -▁his -▁now -▁aut -▁bas -▁Str -▁gu -▁gibt -▁Bed -▁As -ets -eat -ont -omp -ural -▁leg -▁inc -amen -▁Nach -fter -teil -▁dec -▁dieses -essen -issen -▁just -ages -ights -halb -ween -▁Ent -ards -▁many -▁make -▁polic -▁partic -▁again -▁[ -▁zwischen -▁weiter -oint -▁first -▁important -staaten -▁Council -▁„ -fin -air -uer -wir -▁techn -▁take -▁how -▁high -▁son -▁mo -▁muss -▁Se -▁pla -▁poss -▁Bere -enz -igung -oth -ute -▁econom -▁Comm -ives -anc -any -arm -ale -als -▁US -stand -▁requ -▁Intern -abe -abl -▁ass -▁Prod -▁Verb -ection -rough -▁unsere -▁Art -▁All -▁between -blem -▁] -▁keine -▁room -▁propos -▁because -▁Entwick -▁7 -▁end -hand -nis -ough -dr -dem -ues -weit -bst -yp -öff -▁hand -▁national -▁find -▁rem -▁av -▁soc -▁min -▁mein -▁biet -▁Ihnen -▁pos -▁Me -▁Min -▁Br -ence -ote -▁Ha -▁Her -▁Fin -ating -atter -ans -geben -ows -rauch -▁Prob -▁einz -▁contr -eht -▁Europa -▁world -ements -eute -ount -▁Grund -▁199 -▁good -ublic -▁9 -▁8 -▁Ä -ax -sel -setz -zial -ler -pf -▁two -▁nat -▁fo -▁dr -▁del -▁wo -▁If -▁So -▁Ex -▁Am -eth -ole -urn -uring -▁every -▁Wenn -▁Gesch -▁information -amil -ally -▁immer -rag -cept -▁had -▁see -▁10 -▁Zusammen -achen -gend -▁verb -▁conc -forder -▁where -ternehmen -▁made -ußer -▁anderen -ält -▁even -▁Mitgliedstaaten -▁Problem -ium -ace -uc -kr -king -äm -▁dann -▁sc -▁way -▁want -▁Seit -▁gl -▁gel -▁gest -▁gesch -▁Bür -itz -ited -ital -urt -urs -chten -▁Wh -▁Kr -▁iss -immer -▁those -▁through -▁pre -▁prot -ents -ower -erein -istr -ischer -▁diesen -ellt -▁problem -adt -alls -weise -overn -ordn -▁app -schließ -trag -ganis -▁years -▁direct -▁Produkt -▁” -day -richt -wort -pen -piel -öglich -▁tak -▁fact -▁rest -▁right -▁view -▁may -▁wieder -▁Des -▁At -ina -isen -ision -ork -unity -▁zwe -▁car -▁Welt -▁Con -▁Haus -ivers -uman -▁form -▁being -ster -ister -▁vers -isse -▁On -▁order -▁There -hne -▁wurden -▁Frau -▁Diese -nahmen -ituation -▁differ -icherheit -▁damit -▁belie -▁includ -▁policy -▁Bereich -ior -für -aur -ade -vor -ys -öß -▁num -▁su -▁mon -▁mean -▁Teil -▁bus -▁Ihre -▁par -▁get -▁gut -eration -itä -iness -innen -▁own -▁cent -▁kein -▁Kl -▁Hand -▁Rat -▁beg -rechen -lex -▁habe -▁seine -▁Ziel -ändern -▁exp -▁process -press -▁Schl -ollen -▁compl -zeich -▁disc -▁Geb -▁jedoch -emeins -stitution -▁market -▁know -▁particular -▁sondern -▁0 -tel -ool -fort -ved -ll -äge -gung -gesch -jekt -▁here -▁heute -▁ful -▁fest -▁red -▁sm -▁ser -▁main -▁wür -▁bu -▁ph -itions -▁cur -▁lea -ateg -raft -ragen -bens -▁stand -▁Inter -▁Information -▁Zu -tegr -▁neue -▁deb -round -read -▁Europäische -000 -laub -▁qual -▁quest -▁agre -▁unserer -▁You -▁etw -▁Maß -▁help -stütz -öglichkeit -▁id -hin -ock -fach -zie -min -br -com --M --P -pos -▁human -▁after -▁sim -▁same -▁wen -▁public -▁great -▁gleich -▁Bar -erk -erst -eld -▁eing -▁& -▁kl -▁Kolle -▁Richt -▁Gro -ently -▁Ind -ests -▁15 -upt -▁Unternehmen -▁Sta -▁att -lagen -ouse -osten -▁clear -▁appl -▁Vorschl -▁political -▁bereits -▁place -▁possible -▁requir -▁Än -▁concern -▁est -iß -ix -over -az -satz -bew -ähl -12 -▁fl -▁führ -▁aw -▁sur -▁say -▁mich -▁Tr -▁ber -▁person -▁ger -▁ganz -▁Best -eter -enen -ines -ince -iron -▁ohne -▁lar -▁lang -▁Fran -▁offer -ierte -ction -licher -▁neuen -▁spe -▁could -obal -▁does -▁access -▁ihrer -▁sollte -rift -ruck -▁selbst -▁wichtig -aurant -▁number -▁& -▁Änder -ux -üh -äre -10 -▁rights -▁val -▁much -▁br -▁bring -▁both -▁Is -▁Da -▁point -▁Man -eral -erung -iti -uro -▁eigen -▁Hin -▁How -▁Gen -▁Germ -▁inst -ierten -ata -▁then -▁ange -▁prof -▁present -▁Ne -kehr -▁12 -▁within -lage -ertain -▁andere -▁Jahren -reichen -▁contin -▁effect -▁Arbeits -entral -▁development -▁europäischen -▁increa -▁against -▁economic -▁avail -▁zwei -hip -viron -wick --B -kun -▁hist -▁fund -▁det -▁ac -▁sign -▁vis -▁var -▁med -▁besch -▁Sicherheit -▁Mon -▁gemeins -▁Bl -▁Besch -eam -oftw -utions -urg -▁equ -iver -ivid -ics -icult -ander -ulation -reit -ständ -▁denn -▁denen -▁18 -äng -earch -ustom -▁used -ease -▁play -▁Frage -▁möglich -▁level -immt -▁therefore -▁finden -▁sollten -▁Entwicklung -öffent -eich -net -ohl -ven -ving --A -line -äß -gar -▁som -▁mod -▁System -▁Mus -▁Part -▁Person -▁Ap -▁Akt -ith -ispiel -orr -under -▁opt -▁creat -▁last -▁Kommiss -▁think -▁priv -ention -▁still -lichkeit -▁verschied -agement -ploy -▁exper -▁without -▁ausge -▁cour -zeug -ically -▁group -ährend -▁country -nahme -fahren -▁provid -▁social -▁bietet -▁Bürger -vironment -oftware -▁ens -hol -zahl -ma -les -punkt -▁term -▁fe -▁down -▁meas -▁machen -▁bo -▁Ste -▁Daten -▁gem -▁Aut -ende -isk -▁oper -▁cap -▁long -▁kle -▁He -ule -▁bef -▁impl -owing -ope -rech -▁sec -esse -▁Vers -ected -▁Stadt -ienst -▁able -ize -▁dazu -okrat -▁Wirtschaft -ritt -ünd -▁viele -▁free -▁consider -▁services -esondere -ittee -▁liegt -▁different -hor -nte -ness -fern -ds -ude -zug -ves -▁akt -▁Tra -▁ben -▁back -▁Dem -▁Em -▁govern -uture -▁call -▁letz -▁Cl -▁Fre -atur -▁prin -que -▁start -ists -ationen -▁weil -▁jetzt -ibility -▁Stand -annt -▁allen -▁aller -ahmen -▁char -▁seiner -▁appro -▁financ -▁darauf -▁industr -elegen -stellung -▁special -▁modern -urther -▁business -▁curr -ready -▁available -kunft -hö -nisse -welt -ym -ER -▁hal -▁foll -▁future -▁air -▁voll -▁bes -▁pers -▁Möglichkeit -▁Pre -erb -ination -ised -ilt -omen -▁custom -▁look -▁Rahmen -ator -ante -arten -▁already -▁before -▁Lage -leist -▁rev -ife -▁verl -▁geht -▁einfach -▁progr -▁project -ience -ahre -▁Them -ealth -▁betr -▁größ -rateg -emeinschaft -strument -▁Programm -ismus -scheid -▁believe -▁small -ext -yn -öl -äh -jor -▁set -▁mir -▁bin -▁Med -▁Mein -▁Bes -▁But -eren -urity -▁cann -▁Res -▁Verein -▁Gew -ication -ders -ath -astr -alk -▁too -▁today -▁bek -▁seit -▁erw -ability -▁geb -▁einige -ially -▁design -ellschaft -▁dafür -▁schon -▁Energ -▁environment -udget -▁30 -ossen -▁open -▁service -folg -▁products -abei -▁Finanz -▁Informationen -▁Maßnahmen -▁course -hält -spr -w. --F -led -pe -▁fac -▁data -▁mus -▁Pol -▁Ad -itiz -inen -inal -ire -orn -▁case -▁certain -▁Russ -ica -▁prom -stellen -▁Zimmer -ucht -▁deut -▁gef -▁And -▁allem -▁programm -grund -▁That -allen -ätz -▁Mittel -ultur -▁2009 -ccess -▁adv -ually -ished -▁Herrn -ample -▁ihren -▁individ -aterial -▁offers -▁Rechts -▁respons -▁Internet -▁cannot -▁ed -mö --s -los -politik -▁typ -▁nun -▁besonder -▁Sh -▁Serv -▁Eine -▁put -▁Amer -ether -isierung -ilit -olid -olution -chie -▁zusammen -▁city -▁Kin -▁Für -▁integr -iving -ately -▁impro -opt -operation -tens -iff -ified -uchen -vers -▁Uns -▁Jahre -ahn -ains -▁Schw -▁meine -▁Bei -▁zurück -▁regard -▁Commissioner -▁art -affen -▁invest -▁Qu -▁question -▁Kommissar -▁Ö -nov -ry -ron -aum -ziel -mar -besondere -▁fol -▁day -▁during -▁reas -▁wollen -▁To -▁El -▁Ph -▁Bo -itor -itive -ins -isl -omat -▁citiz -▁Ro -imes -gew -raph -tend -▁neg -▁verfü -▁cond -▁entsp -▁energ -▁Aussch -heiten -▁Als -uction -▁Spiel -flicht -▁unseren -▁activ -▁said -▁law -führt -bsite -setzung -▁global -▁würde -124 -▁implement -▁eas -ohn -uz -ways -bed -lie -lungen -lungs -ply -kon -öt -äuf -▁famil -▁rap -▁achie -▁member -▁Mill -▁Pal -▁Politik -ily -▁Wie -▁Fir -angen -amer -gehen -▁La -lei -ktion -▁stat -▁strateg -ichte -▁example -antwort -present -halts -▁cost -▁major -spar -▁| -immung -▁Form -uktur -▁rooms -▁proposal -▁issue -▁including -▁means -▁debate -▁quality -▁financial -▁each -ove -fass -red -ga -gleich -▁tun -▁hö -▁hear -▁far -▁few -▁doch -▁situation -▁since -▁während -▁bea -▁Deut -▁pass -▁gar -▁Bild -ilities -oren -▁Was -▁Ger -alen -reff -racht -▁stehen -▁sect -ands -▁16 -ache -genommen -▁necess -▁Und -▁Our -▁sicher -▁exc -▁child -▁sozial -▁result -▁meet -▁common -▁interest -▁Parlaments -▁technolog -▁etwas -▁Vorschlag -▁However -▁| -agen -ugen -ward -wert -▁fun -▁ret -▁matter -▁wissen -▁Durch -▁Euro -▁power -▁Bet -ites -ott -org -utes -▁komm -▁Viel -▁Gl -ürlich -▁Ur -▁Länder -▁Lös -opp -rad -rache -▁ref -▁stay -orth -▁erhalten -▁conf -agt -ants -ized -▁amend -zept -▁adopt -▁compet -anken -gangen -▁sagen -▁object -linie -▁Unterstütz -▁insbesondere -mittel -klär -▁Committee -▁Geschäft -▁citizens -▁� -▁ern -▁öffent -fet -falls -come --R -▁richt -▁außer -▁brea -ird -offen -offe -▁cult -▁kommen -▁Kunden -▁Fall -▁Rest -▁Gemeinschaft -ieg -ars -▁three -ential -▁New -▁represent -▁stud -▁11 -▁14 -▁25 -▁next -▁einmal -▁Verfü -indet -illion -▁expl -▁schn -▁schw -tract -▁waren -▁areas -▁bedeut -▁Markt -▁respect -▁Qual -▁author -▁found -ordnung -▁Änderungs -▁histor -▁measures -hör -ek -ipl -uge --T --c -pect -ßen -äd -äum -▁further -▁Techn -▁bl -▁budget -▁Punkt -▁give -▁Beh -esen -ining -igten -uten -utzen -unde -▁lassen -▁thank -▁beim -▁rece -▁wirtschaft -▁Zug -▁ersten -▁organis -▁democ -▁abs -etzen -ünft -▁Ital -▁China -▁employ -▁Projekt -▁direkt -▁provide -▁Zusammenarbeit -▁taken -▁success -itäten -▁discuss -▁build -▁large -▁energy -▁Unterstützung -▁ä -▁X -▁ear -.com -co --D --K -gang -joy -▁du -▁deal -▁drei -▁read -▁vot -▁pay -▁Bel -ilfe -off -orgen -▁link -▁With -▁inde -ieb -ieben -ierungs -asch -ald -▁begr -▁secon -▁wirk -▁erst -cken -▁ask -▁geben -esser -ehm -porte -▁entwick -▁schl -▁addition -▁besteht -igkeiten -▁Umwelt -▁Recht -haft -vention -▁position -▁agreement -▁establ -▁ensure -tain -nung -ogen -zess -mit -len -por -par -kf -ker -▁terr -▁fil -▁did -▁Tag -▁Sw -▁Dienst -▁pop -▁Mad -▁Port -▁ze -▁inform -ather -ash -ums -ules -▁Ländern -▁Namen -▁statt -▁steht -▁sehen -ichts -▁Stra -▁shall -▁come -▁eng -▁Beispiel -▁unterstüt -▁seinen -▁Fragen -▁company -örder -▁opport -altung -▁elect -führung -▁Deb -▁problems -▁Group -▁increas -▁necessary -▁ir -burg -EN -äss -glichen -▁ho -▁hope -▁health -▁dabei -▁sk -▁wäre -▁Dar -▁given -elf -ises -urz -ury -▁lead -▁Wer -▁War -▁Kom -▁Red -▁inner -▁ann -▁bew -▁pract -▁Not -▁imm -▁imp -schein -schrift -▁13 -▁17 -trem -▁why -asse -▁change -▁2008 -eutung -▁tradition -▁key -wendung -▁2010 -▁invol -▁legisl -wirtschaft -▁however -zieh -▁security -▁princ -hof -häng -nten -zü -man -bau --E -lt -pend -AT -AN -▁teil -▁vote -▁muß -▁Beit -▁Bew -ising -isation -igent -▁list -▁Ko -▁Vol -ief -ivil -ively -▁always -▁pres -leg -ters -▁def -▁verw -▁verst -▁je -egt -▁while -▁entfern -▁worden -▁cooperation -ätze -reng -iety -▁needs -ettbew -▁located -▁protection -▁Kollegen -▁accept -▁letzten -gaben -▁second -ift -fang -von -gem -20 -▁nor -▁sens -▁writ -▁Trans -▁Sa -iter -▁life -▁Wor -▁West -▁Hier -▁int -bereich -▁deren -rechte -▁hap -▁refer -ingungen -▁24 -genden -▁They -▁Ob -ialog -eller -undheit -▁allow -ikan -▁short -▁comb -ranspar -▁add -▁better -twort -▁account -▁commun -▁Regierung -▁50 -▁Lebens -▁transport -▁große -▁großen -▁welche -▁Entscheid -▁USA -▁control -▁Verfügung -iend -aut -wards -bo -ged -gre -▁net -▁fiel -▁fav -▁ren -▁men -▁mult -▁wish -▁Ben -▁Aber -▁Auch -▁zahl -▁kom -▁Comp -▁House -▁Fe -▁Fol -▁Famil -▁Rom -▁Lin -leicht -▁100 -abil -▁defin -▁conn -▁United -▁using -▁che -▁cho -ipp -▁Ergeb -▁2007 -sequ -▁action -unktion -▁diff -▁Bedeutung -▁legal -▁Seite -▁What -▁Gebiet -▁full -▁install -▁Kinder -markt -▁* -eign -iod -ique -zus -IN --m -ying -lossen -part -pflicht -▁herv -▁reach -▁around -▁sing -▁mov -▁Sk -▁Mal -▁Bus -eder -iges -▁cor -▁When -▁Gu -immen -atch -reen -▁ermö -text -▁notwend -ellent -▁exist -▁Schutz -▁enjoy -ünst -clus -blick -▁arr -▁zweit -gebot -▁jeder -▁working -▁perf -▁consum -▁Berichterst -▁Community -▁International -dress -▁bieten -▁current -iation -fre -vest -bt -kauf -gabe -▁near -▁sl -▁Beg -▁Air -erc -▁eben -▁crit -▁less -▁km -▁Fort -▁Gä -ared -ames -kte -▁stellen -▁1. -▁Zahl -▁erforder -träge -▁Vertrag -▁Jun -▁whether -▁Stud -▁extrem -▁alles -▁These -▁chang -▁daher -▁davon -obil -▁Mrs -▁darüber -ießen -▁become -entrum -▁production -▁based -▁Haupt -▁centre -▁particularly -▁Zukunft -▁profess -▁terms -▁Thema -hm -hmen -iar -nehmen -ood -fra -rän -uce -by -kungen -xim -▁treat -▁dist -▁Tur -▁brauch -▁San -▁Betr -▁lim -▁Wort -▁Weg -▁Chr -▁Fahr -amt -▁toget -▁another -enten -▁UN -▁step -▁orig -▁esp -ennt -▁Weise -▁research -▁2000 -▁clos -▁Frauen -▁trade -▁weitere -▁sowohl -▁gegenüber -▁subject -▁local -▁Probleme -▁eigenen -▁visit -▁befindet -▁individual -▁rapporte -▁Restaurant -▁difficult -hang -ex -zi -vious --G -▁him -▁non -▁näch -▁dort -▁run -▁mar -▁water -▁Dir -oms -uter -▁care -▁Wahl -anies -▁tour -▁state -▁erreichen -▁organ -▁express -ika -ikel -▁friend -asser -ipt -▁Ende -ecially -▁hatte -▁könnte -▁regional -▁EUR -▁gener -▁politischen -▁Ihrer -▁meiner -▁contrib -▁natürlich -▁issues -▁Haushalts -ategor -▁klar -▁besonders -▁children -▁wirklich -▁unterstützen -▁hervor -▁together -▁old -ev -ners -fäh -aden -zuf --L -yl -▁tax -▁aff -▁bar -▁book -▁bene -▁Sec -▁page -▁gent -orm -▁crea -▁Fest -geordn -▁No -reg -schutz -ocument -▁ihnen -▁address -▁Alle -ording -ritte -ails -▁arbeit -▁gesam -▁systems -▁specific -load -▁guar -▁Mehr -▁Industr -▁Staaten -▁German -▁various -▁verschiedenen -▁experience -▁Energie -▁Service -▁reason -▁conditions -▁innerhalb -▁entfernt -▁Berichterstatter -▁+ -ty -ode -dings -af -bild --C -ES -ße -▁tot -▁role -▁aim -▁women -▁beach -▁Situation -▁pea -▁pack -erien -oment -unter -▁intr -anzen -ances -▁prep -bene -isches -ifft -ifiz -▁whole -adies -▁comfort -thing -ours -▁Abst -▁companies -▁vielen -▁Forsch -▁bereit -▁basis -finden -selves -▁Seiten -▁personal -itiative -▁industry -▁Ausschuss -▁beaut -▁sector -▁benef -▁guarante -▁_ -til -oor -ried -uced -sen -kes -kung -ären -ON -▁hoch -▁di -▁distr -▁met -▁welt -▁Terr -isions -▁occ -▁col -▁Wettbew -▁Kar -▁Rep -▁Rolle -▁innov -ived -ima -imate -aster -▁though -▁top -▁prior -▁staff -▁sever -▁Zwe -uppe -▁Jan -prü -▁spea -▁close -▁möchten -▁grow -▁zwar -oney -▁effort -▁internationalen -▁really -▁restaurant -▁month -▁ideal -▁especially -iber -ober -üd -ms -way -yr -▁turn -▁feat -▁dür -▁done -▁document -▁Tour -▁Mod -▁Bas -eag -ots -▁coll -▁Woch -ican -alten -▁third -▁ans -▁beh -▁Uhr -raum -▁ste -ably -gende -ehmer -indung -▁proced -weisen -▁Engl -▁Spe -▁region -▁bestimm -▁Hotels -▁period -▁hinaus -▁Ihren -▁location -▁food -ething -▁Richtlinie -▁führen -▁private -▁government -▁members -schriften -▁create -iem -zt -▁dra -▁sum -▁site -▁mot -▁Treat -▁Ir -▁Dr -▁pat -▁port -▁glaub -▁Afr -ino -cher -▁Kü -▁instit -ival -ications -icient -amew -▁press -▁Nutz -ides -oph -▁reform -iste -▁erste -ehn -ände -ody -prom -▁enter -ization -iple -echan -▁Park -▁perform -▁event -zeitig -onds -▁gewähr -▁automat -▁Strateg -raucher -▁Germany -▁approach -▁negot -▁Geschäfts -▁favour -ton -iol -druck -wär --b --V -AR -▁million -▁Software -ette -igr -▁Wind -▁Rück -▁offen -imin -atte -ativen -ults -▁town -▁Lie -▁Nov -rael -erne -ifen -achst -▁Prot -▁job -▁Ang -ferenz -▁among -▁2006 -ordin -sprache -▁Berlin -▁general -ishing -▁itself -▁europäische -▁Gesundheit -▁struct -▁decision -▁Artikel -▁langu -▁something -▁Meinung -reffen -fety -▁single -▁rapporteur -▁several -ting -zust -zimmer -In --H -▁hät -▁home -▁fast -▁ra -▁she -▁material -▁Sel -▁post -▁Mark -ences -isis -olf -▁zus -▁lab -▁Komm -▁Fisch -▁Gleich -ierungen -▁anges -▁recht -▁Univers -antrag -▁show -ograph -▁resour -▁resolution -▁making -ians -▁genau -▁198 -ittle -▁Kontr -▁contact -▁etc -ksam -▁Gesellschaft -▁least -▁continue -▁progress -▁programme -▁excellent -kfast -▁netw -▁gentlem -amework -ning -voll -mos -cke --in -less -▁text -▁name -oto -uth -▁z. -▁zun -▁kind -▁often -iere -asst -uft -ufen -▁anal -▁beiden -▁prec -▁Invest -aps -uppen -iken -▁management -▁apart -ußen -prod -▁relations -▁going -▁unserem -▁Leben -▁opin -▁understand -ification -▁Entwicklungs -▁Zusammenhang -▁serious -▁away -▁Partner -▁Personen -▁following -▁family -▁kommt -▁schnell -▁World -▁peace -▁introd -▁gentlemen -haus -gän -▁rules -▁sea -▁sett -▁software -▁mach -▁bz -▁Sc -▁My -inz -iner -▁cas -▁colle -▁let -▁Kultur -▁Car -atives -▁National -▁online -▁streng -ingen -indest -prop -gricult -weis -okument -▁plan -▁pleas -▁Abgeordn -▁regul -▁bestehen -▁finanz -▁Dieses -▁Kont -gestellt -cript -ächst -▁crisis -▁Entschließ -▁particip -▁directive -▁Schluss -▁complet -▁optim -▁entsprechen -▁Deutschl -▁absol -▁weltweit -▁negoti -▁ident -sk -zogen -mäß -wand -ägt -gemein -50 -▁range -▁sof -▁milit -▁money -▁Do -▁Pat -ither -use -▁little -▁Weit -▁Hilfe -▁Fil -icle -uff -bereit -▁towards -▁forward -▁ant -▁price -▁once -▁rep -▁stellt -▁21 -▁Verantwort -▁Jul -ager -ension -though -▁unterstütz -▁doub -▁Span -▁Region -▁Präsidentin -▁Gesetz -▁welcome -itäts -▁etwa -▁feel -▁Demokrat -▁Lösung -▁Technolog -▁Gäste -▁® -▁il -ike -▁dest -▁sw -▁situ -▁mention -▁bal -▁Son -▁pot -▁pur -▁gilt -▁Bank -ches -chit -▁until -▁things -▁Natur -ops -▁stär -ichten -iful -liches -▁neu -▁ja -platz -unds -annten -ference -weil -▁spiel -icken -outh -▁daran -▁recog -▁Organis -armon -▁natural -▁Deshalb -▁weniger -▁bringen -▁Wirtschafts -▁customers -leistungen -▁review -fassung -▁begrü -▁Madam -▁institutions -▁bzw. -▁00 -hing -uellen -IS -ckt -▁test -▁fair -▁days -▁aid -▁bro -▁bitte -edit -otz -▁land -▁Kosten -iesen -recht -▁2. -▁23 -▁Zust -▁einge -präsident -▁umf -▁2005 -füll -▁Grün -ücksicht -▁Char -▁Gegen -▁jeden -baren -▁keep -▁indem -▁empf -reiche -oints -▁gesagt -lands -reiben -gelegt -▁strong -teilung -forderungen -ämlich -ählen -▁increase -▁persön -▁Preis -▁opportunity -▁Treaty -▁ess -ple -▁hard -▁nämlich -▁rather -▁Text -▁big -▁Ser -ems -ined -inter -ison -▁kosten -▁kurz -▁Can -▁Cent -▁Gar -▁Gren -▁Geld -stadt -istic -ommend -ify -ission -▁Vertr -plan -▁Europas -ocus -▁sogar -▁Erfahr -▁'t -▁enc -▁unterschied -▁quite -▁agree -▁40 -wohl -wendet -idence -idency -ächlich -▁Sept -▁positive -▁Sozial -▁Produkte -▁bekannt -▁activities -▁Millionen -▁check -▁growth -na -uh -cel -CH -▁tas -▁sug -▁might -▁Tür -▁Sup -▁Dis -▁Mo -▁Ass -ona -▁lo -▁line -▁Rates -atis -▁Nor -reib -raf -stück -treten -abhäng -app -apan -▁nehmen -▁dev -▁Off -ports -▁Ansicht -fert -▁partners -▁Jahres -▁Members -▁politische -werk -scheiden -▁Fried -▁Ort -▁fully -▁gerade -▁provided -elegenheit -▁program -▁ebenfalls -frage -▁Christ -▁resources -mosph -▁September -▁ü -hunder -ij -völ -mb -wahl -komm -ächt -OR -▁four -▁mag -▁Tod -▁Tats -▁besser -▁Sal -▁perm -▁Per -▁After -ennen -igungs -onger -▁Kap -ana -anst -gegen -▁yet -▁Nicht -rachten -leit -▁erre -▁erfolg -▁State -▁ihn -▁ihm -▁Austr -▁2004 -▁class -▁seinem -▁darf -▁Dieser -▁include -airs -▁request -▁verbunden -▁value -▁signific -▁gemacht -▁Unsere -▁Zugang -▁Deutschland -res -ser -side -system -bon -ling -AS -▁tem -▁dessen -▁wahr -▁wesent -▁Den -▁pa -▁park -▁gute -eless -irm -▁led -▁log -▁ladies -icial -ativ -atten -geh -▁along -▁having -▁Institution -osition -▁gehen -illa -lau -▁Aussprache -amental -▁admin -▁Bezug -spekt -▁Fraktion -programm -▁left -auer -▁solution -ühren -▁proposals -▁educ -▁technology -▁amendments -▁field -▁beautiful -propri -▁unterstützt -ros -sweise -za -mut -beh --f -▁times -▁ak -▁Sol -▁Buch -itzen -▁light -▁Hö -▁Frei -▁unver -▁derzeit -ente -stra -stant -einander -▁framework -▁above -half -schaften -ideo -prochen -▁darin -ratic -▁Regel -▁safety -▁recommend -▁Rechte -fully -lock -▁economy -▁remain -▁minutes -self -setzen -▁appear -benso -▁idea -▁Israel -▁easy -▁öffentlichen -▁Qualität -hört -▁Janu -▁dürfen -▁glaube -fäl -ähe -▁transpar -▁na -▁flex -▁viol -▁Mär -igital -unst -chs -▁Rats -alle -▁tool -▁behalf -racy -▁Instrument -esten -ichtet -▁website -▁week -▁Priv -▁einigen -▁constr -▁Verhand -▁One -▁nichts -▁challen -▁schaffen -▁Website -obe -▁2003 -▁makes -ektor -gebung -▁freed -▁Rei -▁content -▁Früh -▁Flug -▁Meer -ethod -▁prevent -▁taking -▁called -leiben -▁costs -▁adopted -▁brauchen -▁Abstimmung -▁strength -here -nehmer -most -wür --W -pre -OM -▁flo -▁focus -▁agricult -▁bank -▁Sub -▁Dank -▁Disk -irc -ured -▁never -▁Prof -▁Antwort -▁Web -▁results -▁meisten -schluss -▁ihrem -▁others -▁sollen -▁Jahrhunder -utzt -▁Verbraucher -▁handelt -▁nationalen -▁einzel -▁verbess -schließlich -▁keinen -▁beginn -▁standard -▁Kolleg -▁Groß -▁application -▁Beschäft -▁provides -▁charact -▁return -▁building -▁legislation -▁normal -wärt -▁zusätz -▁opinion -word -IT --0 -äus -▁tar -▁team -▁due -▁mass -▁TV -▁Ebene -▁past -▁points -iting -oten -▁Komp -▁Co -▁Far -▁instrument -iers -anced -▁stark -▁Inst -▁22 -achten -▁versch -▁Verl -▁Verkehr -▁Just -eits -reater -aged -earbeit -ampf -▁Leist -ringen -▁heiß -▁policies -▁weiterhin -▁Minister -▁deliver -overnment -▁simply -▁attention -▁concerned -▁follow -▁projects -▁betrifft -▁responsibility -▁perfect -▁Fortsch -uelle -bour -bare -ägen -▁host -▁dom -▁risk -▁macht -▁bor -▁Sitz -▁Bem -edi -isiert -arf -arian -ufact -lect -▁sei -istan -verkehr -▁context -▁Verw -▁Op -▁deshalb -▁young -ustain -▁2001 -▁appropri -▁besten -▁commit -ients -▁solche -▁importance -staat -▁Plan -stellt -▁known -▁Design -zeichnet -▁questions -ziehen -▁wenig -▁effective -▁groups -▁benöt -▁deutlich -▁heart -▁bedeutet -▁Änderungsan -▁total -iellen --1 -▁head -▁holid -▁rad -▁send -▁II -▁Sl -▁Pra -▁greater -▁Apr -▁obl -▁Wasser -▁Von -▁oft -atul -ators -berg -▁beispiel -▁Netz -ceed -qual -▁aufge -▁aufgrund -eite -änger -▁cover -▁reserv -▁Mitarbeit -▁Bezieh -▁clean -▁interess -▁road -frist -▁wichtige -▁heraus -▁society -▁version -▁Klima -▁complete -▁würden -▁standards -▁continu -hören -▁Besuch -▁environmental -ledge -▁strategy -züglich -▁professional -▁€ -▁000 -uit -zlich -weg -käm -äter -▁fail -▁verein -▁moment -▁method -▁bit -▁Dokument -▁EC -▁Ag -erbs -▁central -▁Kunst -▁infl -▁institution -anze -ama -ales -heb -rach -stehen -vert -verb -ada -▁Schul -▁schön -▁words -▁manufact -▁Bevöl -ritten -örden -äsent -gesetzt -frastr -sichtlich -▁Menschenrechte -▁findet -pfung -▁views -▁Richtung -▁required -▁increasing -▁principle -▁nächsten -▁November -.0 -hib -now -fall -ban --O -AD -jen -▁nutzen -▁farm -▁don -▁via -▁mount -▁IT -▁Ed -▁May -▁Pak -eau -uren -▁lear -▁Will -▁Wohn -▁Feh -▁Förder -▁fort -▁Nähe -ided -leben -▁Ins -▁Ju -▁Japan -ellung -▁user -▁vorgesch -▁quick -acy -▁training -▁Herren -▁relax -▁Umsetzung -genda -▁parl -▁clearly -▁allerdings -▁music -▁meeting -▁breakfast -▁Windows -zustellen -▁significant -ai -uell -äche -▁hoffe -▁night -▁rea -▁Such -▁Dec -▁pages -enzen -unch -▁Fern -▁Raum -ims -atal -anten -▁Nord -reet -bend -legen -verständ -▁weiß -aging -▁demand -▁extern -▁entscheiden -ips -▁sour -▁2002 -▁197 -▁works -ießlich -▁recent -▁Struktur -▁technical -▁assist -▁protect -zeichn -▁capital -▁achieve -▁cultural -▁authorities -▁Benutz -▁Selbst -▁Verantwortung -▁Demokratie -▁essential -▁suggest -▁April -▁beispielsweise -▁u -biet -lar -kerung -▁side -▁Ira -▁Date -▁gra -▁Bil -▁Brit -onen -onom -▁cla -▁longer -▁Wert -▁Wachst -heim -key -ernen -ester -▁aspect -▁gegeben -▁gehört -▁Stat -▁enough -ünf -▁according -▁suppl -▁internal -▁solutions -folgt -haltung -▁Bedingungen -wirkungen -▁Grundlage -▁directly -▁Ziele -▁Interesse -▁Vorschläge -▁fundamental -▁gemeinsamen -▁Apart -▁kleinen -▁Empf -▁facilities -▁First -▁jeweil -▁notwendig -▁distrib -▁months -▁‘ -a. -wach -best -OS -▁vier -▁Tages -▁Dam -▁Eur -▁AG -▁Aspekt -inge -isher -ones -▁Cr -▁Real -ari -alking -schn -istischen -▁erziel -verl -▁verwendet -▁demokrat -▁Anwendung -▁extra -gress -▁enthalten -▁Erfolg -▁Einz -üssel -▁vielleicht -▁Kons -ooking -ublik -▁Ihrem -▁guests -▁proposed -▁nature -▁wohl -richtung --Pro -▁history -maß -▁currently -▁responsible -▁employment -▁Direct -▁illeg -mbH -.00 -hem -ding --t -yle -▁sustain -▁vol -▁Druck -▁gere -▁Außer -urf -▁zust -▁civil -▁circ -▁Kön -▁Krit -▁Go -▁GmbH -deut -▁bevor -orteil -ape -gency -▁Verfahren -▁Opt -ienen -▁ausgest -▁sprechen -▁schwer -▁Paris -▁Parte -▁Einsatz -▁events -▁interests -▁subs -▁Please -▁From -▁60 -▁Heraus -▁Geschichte -▁requirements -▁Daher -▁Musik -▁halten -▁sozialen -▁terror -▁ermögl -▁erreicht -▁Reihe -▁character -▁ö -▁$ -▁ur -ta -30 -AP -▁sure -▁bleiben -▁Mot -▁Aug -▁Außen -erous -onstr -▁live -▁Wel -▁foreign -▁anim -rect -▁station -▁erfüll -▁web -▁übern -▁werde -merksam -lauf -▁Erweit -ensive -▁parties -▁please -▁regions -immungen -▁needed -linien -▁themselves -ables -▁apply -▁berücksicht -▁Muse -▁Möglichkeiten -▁promot -▁competition -▁voted -▁Debatte -▁involved -▁happen -▁kompl -▁extremely -▁network -▁recogn -würd -▁= -rieben -zentr -bank -AL -▁fight -▁Sport -▁Soc -▁South -▁Damen -▁camp -ull -▁force -kten -▁separ -▁Integr -ommod -▁27 -ielle -▁Erklär -ätzen -zeigt -▁maxim -itionen -▁könnten -rench -▁ago -▁upon -attet -arbeiten -▁century -▁Sicherheits -▁Aktiv -▁verschiedene -ynam -spruch -▁Öff -▁implementation -▁höch -▁schwier -▁democracy -▁establish -▁Entscheidung -▁changes -▁arbeiten -▁guarantee -▁efforts -▁language -▁freedom -▁vorgeschlagen -val --d -▁wea -▁wind -▁Doch -▁pool -▁By -inner -irt -▁lot -▁low -▁Wissen -▁Hoch -ichen -aren -amb -amous -▁Nat -rab -ingt -andid -▁Zus -iffen -abel -▁Ant -▁soon -▁schrift -obile -aussch -▁regulation -▁oppos -▁contain -▁developed -▁konkr -drück -▁Minuten -istration -▁expect -▁Frank -▁Hinblick -▁gemeinsame -▁Blick -▁Russia -▁Unser -▁verfügt -▁Industrie -▁comfortable -▁situated -▁oblig -▁ermöglicht -hes -dam -gern -▁tats -▁nu -▁nam -▁mess -▁bil -▁Du -▁publ -▁More -igte -orge -unik -▁either -▁caus -▁lange -▁Kir -▁Kraft -▁Cour -▁Rel -atory -ental -ctions -▁star -▁decl -▁contract -erseits -▁comes -▁hatten -▁Auswirkungen -gesellschaft -▁Prozess -▁Platz -▁markets -crib -▁minim -▁previous -▁France -▁vollständ -tended -▁Änderungsantrag -▁zeigt -▁cases -▁negotiations -▁Institutionen -▁akzept -▁Frühstück -▁Diskuss -▁target -▁{ -frei -ware -IC -öll -▁acr -▁bem -▁Bau -erra -usst -▁Kle -▁Hol -▁unique -ano -▁altern -reh -ktionen -adurch -▁attract -wegen -▁schein -▁amb -▁Voraus -▁final -gesehen -▁true -▁bisher -▁developing -▁1999 -▁everything -▁maintain -ählt -▁führt -▁surround -▁created -▁Steuer -rechts -▁function -▁wirtschaftlichen -▁äußer -▁Dienstleistungen -ipped -▁Ergebnis -▁erforderlich -▁gesamten -▁speak -▁Strategie -▁Kontakt -▁Tatsache -jenigen -▁» -.1 -.2 -ügen --In -EL -▁house -▁fünf -▁danken -▁Ide -▁Del -▁pict -▁Aff -eden -▁ebenso -▁cal -▁later -▁living -▁Cas -▁Host -▁Fäl -▁French -▁Rad -▁almost -▁prices -▁states -play -elling -▁allgemein -▁Schritt -▁Webs -▁Ausdruck -▁darum -▁accommod -▁Regionen -barkeit -▁Presidency -wohn -folgen -▁Landes -ymb -▁improve -▁entwickelt -▁additional -▁Familien -▁draw -▁analys -▁education -▁einzelnen -▁Kolleginnen -haps -oid -met -TO -bel -bahn -B. -▁file -▁wa -▁Team -▁Denn -▁Mont -▁getr -oft -change -▁cut -▁komp -atic -ane -arte -▁thus -▁Liber -oper -▁denke -▁Zum -plätze -▁Angebot -prise -▁share -▁Schwe -ipment -▁climate -▁que -schlag -▁leicht -▁196 -▁disk -▁liber -▁actually -phas -mitted -ändert -ception -▁included -▁monitor -▁Staats -▁Version -▁democratic -▁immedi -▁Außerdem -.000 -ländern -EC -go -jähr -▁hold -▁wide -▁ble -▁Binnen -etten -orf -▁lack -▁angenommen -▁behand -▁Lis -▁Lond -▁Luft -idd -▁seen -▁Inform -▁Zur -▁mitt -ibr -änden -▁exam -aint -▁Erst -ense -▁travel -imme -auber -ässt -▁jedem -▁Mitglieder -elef -▁internationale -▁Bereichen -▁Handels -▁General -▁gemeinsam -▁receive -▁Portug -pendent -produkt -inzip -▁Abgeordneten -▁encour -lauben -▁Verhandlungen -▁Änderungsanträge -▁… -▁} -oben -fel -fragen -feld -zun -ült -kra -▁dialog -▁setzen -▁vill -▁bi -▁Iran -▁Süd -▁Dialog -▁Pap -▁Position -ilar -igur -otos -▁euro -▁CD -▁Führ -▁Ris -▁befinden -▁stor -▁Zw -▁Zentrum -artet -▁outside -▁konnte -mitteln -▁held -▁weiteren -abled -▁restr -▁gleichzeitig -▁funds -▁Vereinigten -▁erwä -▁designed -▁type -▁links -▁happ -▁original -fähigkeit -▁hätte -▁appropriate -kämpfung -han -oom -zent -ka -▁heut -▁harmon -▁famous -▁rate -▁mind -▁wider -▁walk -▁IS -▁Mil -ender -isieren -▁eigent -▁Fund -icated -aler -▁thing -▁Leit -▁Liste -erei -▁erhö -apt -verfahren -▁neben -▁Anst -iens -lack -▁später -▁school -▁relev -ologie -▁consult -▁eleg -▁assoc -▁knowledge -▁download -▁American -▁Terror -▁features -▁procedure -▁z.B. -▁umfassen -▁heißt -▁Beziehungen -▁Bevölkerung -▁August -▁London -ikt -ound -riff -zw --st -öhn -gument -▁hon -▁funktion -▁dan -▁PC -etter -▁zuf -▁Währ -▁North -ommer -▁Vertra -▁join -rank -▁amount -▁Abkommen -iance -auen -▁Aufgabe -aufen -gebiet -▁jede -▁workers -▁propert -▁transl -▁helfen -▁Flugh -ösen -teilen -standen -▁posit -▁80 -▁throughout -▁Interessen -▁kleine -▁customer -▁looking -▁majority -▁early -▁impact -▁communication -▁Feb -▁answer -▁absolut -atisf -ba -ble -ID -ca -cies --2 -pose -▁hours -▁five -▁date -▁rund -▁aud -▁völl -▁bat -ilf -▁cop -▁corr -▁> -▁Fuß -▁univers -dert -▁angem -▁beit -kei -eiten -▁exclus -▁users -▁space -▁parts -▁doing -▁emphas -▁hinter -▁gegenwärt -▁201 -mals -▁wichtigen -▁meinen -▁gelten -▁gestellt -▁takes -▁Online -▁equal -▁Themen -▁Programme -▁certainly -▁Beitrag -▁Forschung -▁Kontrolle -▁Schlussel -▁across -iqu -ios -uck -zif --h -kn -ko -künft -öst -▁divers -▁draft -▁mechan -▁Mag -▁Pa -▁Acc -enken -esh -▁lat -▁Werk -▁Kos -▁Kampf -▁Vis -ump -▁although -▁prakt -fte -▁stop -teid -ayer -▁verantwort -issions -▁consequ -▁Verpflicht -▁Stabil -ella -▁proper -▁solid -affung -▁guest -kräft -richtungen -▁spezi -▁determ -▁investment -▁traditional -▁correct -▁colleag -▁Türkei -▁Förderung -▁> -,000 -ee -fung -va -IP -cent -länder -▁spr -▁search -▁Id -▁Met -▁Pass -▁gab -▁ever -▁lux -▁High -▁intens -▁took -▁bekommen -▁erinn -troll -hren -▁verd -esserung -▁Eng -▁resol -▁complex -clear -itted -▁relig -▁jedes -▁unseres -▁record -▁liegen -▁Verbindung -▁gestimmt -▁Handel -fortun -▁Barcel -▁concerns -gangenen -▁connection -▁existing -▁Turkey -▁Wettbewerbs -▁English -igration -▁strengthen -▁bleibt -▁Febru -▁digital -▁wün -▁bess -eath -inden -▁zul -▁card -▁Kop -▁Cor -▁Che -▁Rechn -▁Vorteil -▁inf -iehen -ande -alter -▁Let -rachen -▁wegen -formen -▁demonstr -▁enlar -ourc -zeugen -aking -fluss -üsse -▁try -▁Reform -▁Imm -haupt -▁europä -▁cross -ucing -▁detail -▁model -▁programmes -▁America -bedingungen -amera -▁richtig -▁leading -▁writing -▁friendly -bildung -▁Entschließung -▁mentioned -▁nahe -▁Court -zunehmen -nissen -ox -SE -ci -▁wunder -▁gerne -▁Bitte -ener -empt -int -ini -▁lok -▁Kor -▁Government -ara -▁Up -▁Liss -▁28 -term -abon -▁verhin -▁Prior -▁confl -▁Verordnung -▁joint -▁nothing -ending -▁extens -annte -▁vorlie -▁community -▁related -ological -▁Strand -▁decisions -ämp -▁includes -▁built -▁concerning -▁aware -▁wichtigsten -▁aktiv -▁reasons -äufig -▁Behörden -▁Nutzung -▁completely -▁parliament -▁Damit -▁Frankreich -i- -date -ric -view -ühl -ball -▁herr -▁sag -▁west -▁game -▁Bad -▁Augen -uted -una -▁Wieder -▁Home -▁unabhäng -▁Lux -quis -▁repl -▁26 -osen -▁Ober -▁allows -▁front -▁Auswahl -▁Gruppe -ördern -▁Aufent -▁committee -gesamt -▁internet -▁mehrere -▁hinsichtlich -▁basic -▁assess -▁Krise -▁einges -▁profit -▁half -▁culture -▁population -▁practice -▁Notwend -ikanischen -▁Ausschusses -▁military -▁holiday -▁Mitarbeiter -▁Einzel -face -zub -lang -AM -jahr -▁mer -▁ww -▁Since -▁Dan -▁Eff -▁Mor -▁ganzen -▁ganze -etary -inne -ired -ille -onnen -omb -unities -▁kultur -▁insp -arity -uffer -▁Link -istrat -▁Thank -▁enthält -ennung -ovo -▁nachhalt -▁warm -▁Forder -ava -stellungen -▁Verbesserung -▁agreements -▁einger -▁media -▁Standpunkt -▁advant -▁Urlaub -▁Volks -iterran -▁choice -▁ermöglichen -obei -▁Erweiterung -iddle -▁Lissabon -nern -gliche -▁wobei -▁Term -▁bur -▁body -▁Sant -▁poor -▁Macht -▁Post -ency -erweise -ush -▁Cons -▁Cont -▁Funktion -alien -gef -geführt -▁prim -▁Les -▁Now -▁geeign -▁conclus -▁Anfang -ills -▁south -ervice -▁leben -▁trotz -ology -▁becom -▁official -▁transfer -schließend -brigen -▁simple -▁values -rechend -▁Standard -▁Gewalt -▁garant -▁objectives -▁received -▁zeigen -ercial -▁package -▁jobs -▁collect -▁Lisbon -▁« -sehen -vement -ET -just -▁tel -▁ruh -▁satisf -▁E-M -unt -▁Gelegenheit -ieht -iment -▁pros -▁Ukra -▁Insel -▁Innov -andlung -▁verg -▁gehören -ession -▁Veranst -ream -▁Anal -undes -▁frü -▁Schaffung -assung -▁entire -izient -ourn -▁plann -▁computer -▁500 -▁center -▁applications -▁equipment -▁Personal -▁Club -▁regarding -▁Global -opping -▁established -▁files -▁origin -▁performance -▁Wachstum -ymbol -▁Binnenmarkt -▁völlig -eff -ias -nt -such -ED -pite -gement -AC -▁wonder -▁Su -▁Sim -▁EN -▁purch -▁Mic -▁Pf -erve -itel -▁cru -▁Kern -▁City -▁Here -▁Fach -▁Funkt -▁initi -▁Leg -▁Verwendung -▁überw -gratul -halte -▁sound -▁grund -ützen -kennen -▁below -▁move -▁einzig -sterreich -▁Gebä -▁attack -▁capac -▁objective -▁Darüber -▁keyword -▁Ergebnisse -▁spielen -▁broad -▁agricultural -▁verbessern -▁quickly -▁Barcelona -iot -iable -fonds -zentrum -wander -pekt -▁hous -▁face -▁sun -▁East -▁passen -▁Material -▁lässt -icles -entwick -▁image -▁Zweit -▁erklär -▁jud -▁Anzahl -▁Enter -spiel -▁Chin -▁reports -▁Zeitpunkt --St -▁bedro -▁großer -▁Landwirtschaft -▁See -▁goods -vorschriften -▁guten -▁leaders -▁Management -▁eigene -▁experien -▁Russland -▁gefähr -▁verfügen -iples -▁verwenden -▁benefit -▁Woche -▁Liefer -▁potential -▁Jugend -▁König -elefon -▁www. -son -seit -more -möglichkeit -wide --term -äst -ón -▁nice -▁Tem -▁Tage -igher -▁FR -▁Ferien -▁Ru -▁Gree -iminal -ature -scht -▁seems -istische -▁Prov -▁conv -▁spac -▁code -▁Mitt -ertungen -▁party -renz -spolitik -▁contains -▁recently -steller -klus -▁Amster -fordert -geschlossen -▁industrial -▁verfüg -▁achieved -▁soziale -▁Lösungen -oppel -▁indeed -▁popular -▁sense -▁double -▁Affairs -▁bathr -▁Notwendigkeit -▁ill -zer -lam -▁tw -▁feh -▁fördern -▁veh -▁vict -▁wr -▁Tat -▁Solid -▁Don -▁Eigen -▁Mich -▁los -▁künft -▁CO -▁Central -▁Rot -▁initiative -statt -▁congratul -ags -industr -ahren -ätigkeit -▁click -örper -▁archit -▁frei -▁Reise -ändig -▁self -iveness -▁moder -▁meinem -▁Finally -▁Amend -▁Krieg -▁Änderungen -▁engine -▁Business -▁June -▁treatment -▁gesamte -▁zunächst -▁beginning -verständlich -▁diskut -▁Informations -nehm -fält -rü -aß -ave -RO -▁huge -▁fra -▁rail -▁sorgen -▁block -▁Sil -▁Some -▁Mac -▁gelegen -iglich -▁oil -▁Während -▁Log -▁rein -essourc -▁note -▁vorhand -sprechen -▁Konferenz -▁Mai -▁efficient -arbeitung -▁Dez -▁highly -öffnet -▁Herzen -▁everyone -▁scient -▁carry -▁gleichen -▁Hinter -▁presented -▁Beschl -▁equipped -reiten -▁levels -▁garden -▁successful -▁Transport -▁Juni -▁Second -▁floor -▁tatsächlich -▁dialogue -▁— -oden -afen -mark -buch -lant -ging -ground -▁furn -▁rout -▁mal -▁Det -▁Dabei -▁Pers -▁Agr -iten -isc -utt -▁Wein -▁Kre -▁Ham -atastr -▁former -stances -leisten -▁Initiative -▁veröffent -▁geführt -▁eindeut -▁comprom -▁Bef -▁31 -▁Regierungen -▁active -mehr -▁governments -▁integration -▁entwickeln -▁increased -▁verstär -greifen -▁Linux -▁consumers -▁Mehrheit -▁sofort -▁Erfahrung -▁Social -▁schriftlich -▁Schlusselwort -fortunately -▁x -▁ut -tent -ued --N -gere -▁higher -▁dang -▁rat -▁age -▁sent -▁vern -▁IN -▁DV -▁Män -▁Pan -▁Balk -▁Wahr -▁Col -▁Further -▁Golf -icien -schw -▁stage -▁sex -abor -▁whose -▁Stellung -ube -pret -ausch -▁Einführung -prozess -imming -itting -▁90 -▁carried -▁series -▁agreed -▁amendment -▁ernst -aschinen -▁Debian -▁verstehen -▁Roman -▁reviews -reibung -▁Install -▁äußerst -essourcen -▁à -hab -kraft -▁Bra -ery -ering -onse -oral -organis -ico -atell -ased -▁anzu -▁Lauf -▁Nur -rew -▁29 -▁neuer -agn -▁except -▁schle -▁vorher -okoll -▁Berg -▁Vors -▁supply -▁news -▁Umgebung -fahrt -▁direction -▁office -icherung -▁high- -▁70 -minister -▁similar -ühm -▁somit -zeugt -▁Element -▁confirm -▁asked -▁hohen -▁principles -▁choose -▁innovative -▁lediglich -schnitt -▁subsid -▁Öffent -PE -▁tell -▁dynam -▁rates -▁six -▁Sum -▁DE -▁Eing -▁Aud -▁Ho -▁une -▁Lat -owers -▁Inde -▁verpflicht -anner -▁atmosph -▁shar -▁shown -▁shows -▁300 -▁beteil -▁Sprache -clud -▁traff -▁Konfl -▁perhaps -▁elekt -führen -▁hotels -▁Develop -teilt -▁avoid -▁Therefore -ployment -▁gebracht -▁matters -▁reflect -▁thanks -▁wirtschaftliche -▁Belg -▁added -▁Familie -▁gewährleisten -▁Organisation -▁erfolgreich -▁Höhe -▁Ratspräsident -▁commitment -▁influen -tt -hung -uern -wise -lassen -put -ächen -▁Sicht -itable -unsch -chn -chter -▁zentral -▁Gest -▁Gemeins -arr -entlich -▁Loc -heben -▁erk -▁erschein -iffe -▁depend -▁Of -▁Open -ends -artige -iko -ierenden -▁anderer -▁York -▁obwohl -schieden -▁stress -▁Article -▁foot -▁speed -▁Neben -▁considered -▁Vereinbar -▁Kommunik -▁Film -ufficient -▁Fehler -▁aspects -▁ausgestattet -ausschuss -▁nuclear -▁erwähn -▁latest -ees -▁amer -▁seas -▁Dinge -▁Erachten -▁ged -▁Ba -eng -eria -iles -usion -▁came -▁lik -▁Well -iegen -alu -estellt -▁condition -▁Okt -▁prob -ienste -▁enable -ously -▁Bekämpfung -▁Alt -▁Spain -▁commercial -etting -▁obtain -▁possibility -rought -▁meines -▁Produktion -▁phys -▁effects -▁promote -▁Kommissarin -▁opportunities -▁Komprom -▁zahlreiche -▁Folgen -▁non- -▁Wettbewerb -▁behind -▁bestimmte -▁Freiheit -▁learn -▁Integration -▁accommodation -▁Amsterdam -oo -besch -cing -xis -▁nied -▁fal -▁dring -▁miss -▁Transpar -▁bean -east -eland -ored -uffet -ume -▁stra -▁erhält -osp -▁einschließlich -▁Jed -▁St. -illed -▁exchange -▁entde -▁spir -▁vorgesehen -▁Präsidentschaft -altungs -▁response -setzungen -▁providing -ündet -▁easily -äume -▁Italy -▁Italien -▁ähn -▁reached -▁zweite -▁prepared -▁bestimmten -▁summer -▁materials -▁setting -▁Fortschritte -▁external -▁Aktivitäten -▁E-Mail -▁atmosphere -▁Erachtens -euer -fg -r. -IL -ych -light -lös -▁ten -▁north -▁video -▁woll -▁Dom -▁Ele -▁Pla -emin -orden -orks -▁Cal -▁Camp -▁Rea -▁Gast -stein -stoff -▁erweit -▁erster -cks -▁Oct -ängt -▁exhib -ained -zeuge -▁tragen -eph -ützt -▁decided -▁reduce -▁creation --Fra -▁außerdem -▁Republic -▁pressure -mäßig -kommens -▁sustainable -▁scheint -▁restrict -▁relevant -▁Ukraine -industrie -▁Dezember -,5 -hilfe -uk -uel -US -és -▁terra -▁nie -▁dadurch -▁death -▁size -▁vo -▁pan -▁gives -▁green -▁Booking -oler -▁Com -▁Fed -arily -ulgar -ley -▁reject -▁style -inger -istics -▁Stelle -▁franz -astic -▁bestät -▁allein -▁solchen -▁effizient -▁submitted -▁Überein -▁1998 -▁versuchen -elegenheiten -▁Preise -▁Also -▁Gemeinschafts -▁Vertrags -▁closed -▁Jahrhundert -▁British -▁Museum -▁Kosovo -▁sagte -▁capacity -atastrop -falt -uert -mann -will -IM -licht -önen -▁tro -▁sites -▁virt -▁WTO -▁Ke -▁Gra -icy -▁alten -▁Nieder -idel -▁neigh -▁concept -▁ord -ibilities -▁Stunden -endes -▁coming -ssystem -▁35 -▁regular -ounts -▁train -▁Gruppen -ught -▁Gefahr -▁actions -▁insgesamt -front -▁places -▁remains -richten -▁Convention -▁Bestimmungen -▁details -▁gemäß -▁betroffen -▁hohe -▁zweiten -▁prepar -▁Benutzer -▁Erklärung -▁Schweiz -▁französ -volle -cial --Z -OL -▁table -▁talk -▁suffer -▁Di -▁Down -eting -esp -emb -iture -▁cause -▁Wohl -▁Kur -▁island -geber -▁erheb -▁aufgenommen -▁vergangenen -▁Oper -roat -angt -▁partner -▁quiet -▁register -▁display -▁Aufmerksam -▁obs -▁March -▁credit -▁einzige -pression -▁discover -▁leave -orrow -▁Medien -▁erneut -▁announ -ategorien -▁priority -▁Africa -▁balance -▁December -maßnahmen -▁property -▁enlargement -▁grundle -▁architect -▁ändern -üg -bet -ps -äglich -ja -▁sport -▁billion -▁Mart -▁gas -▁B. -eav -ilom -ilos -▁ca. -▁Cap -▁Rec -imension -ürf -bert -▁Last -quare -▁rech -▁describ -▁Anwend -adem -lav -▁Erw -▁grant -▁genießen -▁effekt -ney -ämt -itzer -bewertungen -▁funding -▁Punkte -▁Western -▁Entscheidungen -▁connect -▁consumer -▁limited -▁Forschungs -rieden -▁Basis -▁März -▁assistance -▁Flughafen -fghan -▁Aufmerksamkeit --r -page -▁niem -▁morgen -▁mobile -▁wid -▁brought -▁Sm -▁Meth -▁Bundes -▁Abend -igungen -unternehmen -▁ko -umb -anden -aper -▁vertreten -▁jün -ugs -▁Star -▁Städ -▁Anforderungen -ollar -▁End -ecut -▁ground -gebracht -▁Berichts -▁konzentr -▁Organ -▁supported -▁forms -▁keiner -▁charge -▁conduct -▁independent -▁renov -▁movement -ategories -▁restaurants -▁Küche -▁regulations -▁Gästebewertungen -▁Server -▁administrat -iterranean -ournal -▁Oktober -▁Booking.com -.5 -sh -mm -▁fig -▁fix -▁rule -▁Sam -▁PL -etimes -▁Kan -ierter -▁Zun -▁Prü -▁Ot -▁Stär -▁export -▁chem -ritts -bei -▁leider -▁Vorschriften -strukt -stitute -▁Exper -▁geschlossen -▁Bereiche -▁Staat -▁variety -▁beschäft -▁Party -▁creating -▁airport -▁größten -▁erwarten -▁Italian -▁schließlich -▁Volk -▁limit -▁apartment -▁pleased -▁Juli -▁Grenzen -▁January -▁Beschäftigung -▁Bilder -▁eigentlich -möglichkeiten -iciency -▁Öffentlichkeit -▁Transparenz -ault -zul --l --up -yse -▁häufig -▁fisher -▁Ist -▁Every -unk -▁lic -▁King -▁Ged -▁Gip -anlage -este -▁Zeich -▁deep -▁Ansatz -▁Anfrage -▁Erzeug -fühl -▁compreh -▁leisten -ört -▁safe -▁produced -altet -▁Arbeitnehmer -▁konnten -▁wenige -▁Unternehmens -▁largest -ühungen -▁instead -▁Neu -lichkeiten -▁discussion -▁friends -▁Wochen -▁Gesundheits -▁purpose -▁Privat -▁construction -▁Verkehrs -▁u. -▁message -▁eingesetzt -▁DVD -▁October -cle -länd -kur -▁dat -▁sus -▁ways -▁Tex -▁Mut -▁Mult -orgt -chsel -▁Villa -▁thous -▁sta -▁stability -isters -▁conference -▁Jo -ibil -iker -prün -ption -spro -▁durchgeführt -▁regel -tragen -▁genommen -▁194 -ütz -▁Präsidenten -▁obvious -▁became -▁geworden -▁wichtiger -▁Meine -▁klein -▁Services -▁Bildung -▁interesting -▁study -porate -porary -▁steps -enntnis -▁Investitionen -▁benötigen -▁Empfeh -▁Diskussion -▁Solidar --Fraktion -▁urs -uis -zone -bür -yond -ki -äte -▁fall -▁rich -▁brit -▁Sign -▁Site -▁Sommer -▁Et -inder -ilfen -▁Wider -▁Kn -▁Cost -▁Gal -▁unmittel -icate -▁Nationen -▁item -▁depart -▁Start -▁regards -▁accord -▁Auftrag -chtung -▁kannst -▁freund -▁suchen -▁Gesamt -akter -▁helpful -▁berühm -▁board -▁aktuellen -▁started -▁Schwier -▁Firma -▁höher -▁film -▁allowed -▁Computer -▁relationship -▁Today -▁Januar -▁associ -dom -vision -mod --p -gra -▁fishing -▁san -▁weder -▁bear -▁Sinne -▁ES -▁Ere -▁path -▁kennen -▁Hel -▁Rights -stre -ifik -▁Veränder -▁exact -prises -oking -▁discr -▁ihres -▁produce -▁Untersuch -▁select -itude -forderung -▁medium --An -▁Systems -▁operation -▁gefunden -▁activity -▁status -▁legislative -▁contribution -▁procedures -▁angesichts -▁Gründen -▁task -gehend -▁source -drücklich -▁Portugal -▁europäischer -▁compromise -▁niedr -▁ursprün -kosten -▁hom -▁dank -▁Sache -▁purs -▁Bur -ony -orgung -utr -▁kit -▁Hung -▁Heute -▁Gott -anischen -gegeben -▁bezüglich -▁Net -legung -▁Innen -▁conven -▁Java -▁Ol -▁Ost -fordern -ellow -▁Schn -▁word -oker -▁arm -▁Auffassung -▁consist -▁provisions -eleist -▁damage -etres -eating -▁1997 -overty -▁Tradition -▁halte -liegen -▁Projekte -▁written -▁distance -▁Spanish -mosphäre -▁liberal -▁village -▁Aufenthalt -▁Development -iell -dienst -ais -EM -pts -▁treffen -▁dut -▁won -▁During -▁Mann -▁got -▁giving -▁Bon -▁Bahn -▁AL -▁Afghan -ero -▁eher -▁Geh -▁überprü -ades -ograf -▁sieht -▁ourselves -▁plant -▁compon -▁relation -geleg -▁Flor -iction -▁sonst -▁prefer -▁option -▁Democ -ronic -▁statement -▁historical -äumen -▁Rede -zusch -zusetzen -▁arrang -▁Beginn -▁documents -▁labour -▁July -▁begrüße -▁übrigen -▁instruments -▁Micros -zy -▁tom -▁bra -▁base -▁Dep -▁Many -▁Pet -▁Bulgar -enk -▁zeit -▁ziehen -▁loss -ivity -stag -▁diejenigen -▁seem -cker -▁Prinzip -ford -indruck -angs -▁Armut -ierende -▁beste -▁Regeln -▁evalu -ateral -▁firm -▁launch -▁minor -zeichnung -▁bringt -▁types -▁entsprechend -▁Beteil -▁öffentliche -▁Koh -▁Linie -freund -▁criteria -zuführen -▁benefits -▁entsprechenden -▁violence -▁Tagesordnung -▁beitragen -▁conflict -▁früher -▁Schlusselwortern -▁Ereign -▁# -▁­ -▁quot -▁urg -vors -EP -▁div -▁mem -▁wine -▁worth -▁Tre -▁Dritt -▁pen -elten -ira -▁kur -▁Fakt -▁Nacht -▁Nr. -idge -▁constant -egation -ängen -▁Anw -ellten -▁Although -ücken -▁Aufgaben -ammlung -▁Your -▁Unterschied -▁useful -▁Bedürf -▁highest -▁require -▁langfrist -▁Monaten -▁Appl -spricht -▁wissenschaft -▁Straßen -scheinlich -ränkt -▁structure -▁University -▁Bemühungen -▁alternative -▁Fällen -▁immediately -▁vergessen -▁ol -rang -zwe -ya -▁mut -▁Ta -▁Test -▁Mos -▁Mess -itness -▁zehn -▁lower -▁krit -▁Ressourcen -iebs -ummer -uls -▁section -▁Zentral -▁erlaub -▁erfolgt -ffice -griffen -▁entw -izin -▁Beruf -▁finally -rats -▁elements -▁hinzu -folge -▁Nachbar -▁Bürgern -▁getan -▁Änderung -▁integrated -▁employees -ünstler -▁Restaurants -▁Spezial -▁kostenlos -▁Centre -▁Vertreter -▁parking -▁transparency -▁Iraq -▁distribution -▁behandelt -▁mechanism -▁verhindern -▁@ -▁ign -eks -aign -ücht -wind -bit -▁hy -▁nimmt -▁d. -▁agenda -▁Tagen -▁bad -▁Sen -▁Econom -▁Bot -urance -▁late -▁Har -▁Free -▁Völ -▁Gel -▁präsent -▁stock -ienz -▁daily -ogle -▁genannten -▁anderem -▁super -▁round -▁laut -▁well- -▁drive -▁numerous -▁applied -▁ausgezeichn -▁Auto -orning -▁reading -▁affect -▁Republik -▁Dr. -▁hätten -▁tools -▁challenges -frastruct -▁mountain -▁bathroom -▁Furthermore -ted -beg --U -park -▁tim -▁teach -▁van -▁besondere -▁Ess -▁Eigent -▁poverty -ola -▁Wunsch -▁unem -iveau -▁Las -▁Inhalt -▁erh -▁Europä -laden -▁coast -▁reality -▁legit -ype -owser -▁drink -▁Kredit -▁Atmosphäre -▁versucht -▁processes -▁offering -▁Bürgerinnen -▁Milliar -▁Behandlung -▁payment -▁elections --mail -▁Betrieb -▁occas -▁apartments -cription -chitekt -▁perman -▁paar -▁illegal -▁gerecht -▁Natürlich -▁consequences -▁solidarity -▁Microsoft -▁ult -dra -setzt -craft -yer -lass -läss -polit -OT -▁sort -▁pal -▁günst -▁e. -▁Human -▁Garten -▁unbed -▁fordern -▁beyond -reifen -▁sechs -ifel -▁verm -▁Vergangen -▁überhaupt -▁unterb -bezogen -▁warum -▁goes -▁offiz -zeiten -▁welcom -▁Aller -▁Herz -▁Produktions -öße -▁saying -▁offered -▁Kommissions -▁Gerichts -▁Werte -▁Folge -▁perfekt -▁Erfahrungen -▁paid -▁construct -frastruktur -▁Wissenschaft -▁analysis -▁colleagues -▁erklärt -UR -▁sufficient -▁mü -▁Tor -▁Est -▁Mass -▁Public -▁Bea -▁ehem -▁cy -▁Hal -▁Ven -▁init -▁Lassen -leb -▁reich -▁ersch -achs -▁Stock -alled -ungs- -▁Ents -▁Altern -▁32 -▁Grand -▁Gef -▁.... -▁45 -▁interpret -▁kont -dresse -▁Minder -▁phot -▁ideas -▁requires -▁modified -ulegen -▁persons -▁größer -▁Mediterranean -▁folgen -▁practical -▁mensch -▁zahlreichen -▁thought -▁Protokoll -▁Weiter -▁Spanien -▁berücksichtigt -▁happy -raten -sych -ücke -mp --w -DF -äger -▁sämt -▁Tab -▁IP -▁SM -▁Even -▁Cop -▁Hy -dere -▁forces -▁stimmen -▁Initi -änge -gruppe -▁shopping -▁Erd -▁Regierungs -gesetz -▁gewesen -▁welches -▁geschaffen -▁curren -▁speech -▁größte -▁richtige -▁historic -▁ski -▁Security -▁Austria -earbeitet -▁Praxis -▁jeweiligen -▁erfüllen -hesion -▁minimum -▁Terrorismus -▁advantage -▁Gebäude -▁Enterprise -▁congratulate -ilosoph -▁jüng -▁staat -▁Vergangenheit -ius -penden -gy -60 -▁syn -▁sales -▁wählen -▁Dat -▁Doppel -▁Mand -▁Bro -▁ey -▁Fäh -▁infrastruct -icit -▁stets -▁Inn -abr -▁neues -▁weeks -▁einheit -▁charm -izienz -▁schützen -▁waste -▁enh -▁clients -▁betreff -▁zumindest -▁frequ -lines -▁simpl -▁attempt -▁starting -▁Finanzierung -gewiesen -▁implemented -▁Viele -▁historischen -▁organisations -▁renew -▁doubt -▁partnership -▁Agent -▁Aspekte -▁getroffen -▁crucial -▁Zweitens -▁verpflichtet -▁tomorrow -fil -sätze -cer -cape -kor -▁morning -▁Tier -▁Tal -▁ban -▁bill -▁bald -▁pie -▁Miss -▁Bud -▁Budget -itors -irl -▁lines -▁Cast -▁Green -ivität -ione -ato -▁bein -▁befür -annen -ocol -▁Schritte -ordination -▁seines -▁growing -fläche -altungen -▁Menschenrechts -▁1996 -krimin -▁server -▁sometimes -▁operating -▁Room -▁Palest -▁außerhalb -▁reference -▁Technologie -▁Center -▁Opfer -▁maximum -▁published -▁modernen -▁comprehensive -.m -vere -üge -▁Dann -▁PH -▁AC -▁lives -▁Val -▁Gem -▁uner -bericht -iding -estic -▁Prem -▁Vergleich -▁Verteid -theless -▁adapt -▁plat -▁Berat -ieten -▁genug -▁efficiency -▁meaning -▁reduction -▁Frankf -▁options -▁Rechtsvorschriften -▁Spieler -▁Firmen -▁ready -▁Airport -▁profession -▁Berichterstatterin -▁Angelegenheit -▁machine -▁repeat -▁Frieden -▁Directive -▁circum -▁Google -▁caused -▁vollständig -▁monitoring -▁honour -▁Vertrauen -▁verantwortlich -▁Amendment -▁Agrar -▁Download -▁° -IA -PS -▁tät -▁har -▁sports -▁She -▁Dav -▁Ak -itation -▁kam -▁kür -▁Video -▁Geg -▁unten -▁Nun -stelle -▁street -▁err -▁erl -upts -▁verab -▁verkauf -ungsver -obby -▁anders -▁relating -▁email -▁anything -▁Play -▁technischen -▁While -▁begin -uption -▁Dazu -▁folgt -pective -hafte -▁persönlichen -▁transparent -▁Parteien -▁Herausforderungen -▁campaign -▁pictures -▁heutigen -ämpfen -▁probably -▁Schwierigkeiten -▁erlaubt -▁Allerdings -▁´ -sb -vis -vollen -lose -karte -gn -▁ded -▁vital -▁Tar -▁Bal -▁Act -igation -▁les -▁Falls -▁Flex -ula -▁anerk -▁prü -opol -quem -▁recon -ischem -egung -▁usw. -▁ausschließlich -angel -▁Einige -▁arch -▁undert -▁Ord -standing -▁worldwide -▁Hinsicht -▁players -leistung -▁erwartet -▁deutschen -losigkeit -▁rapid -▁technologies -ehnt -▁languages -▁Entschließungs -▁swimming -▁stärker -▁border -▁bessere -▁spirit -▁accordance -iant -oot -oon -wün -▁sal -▁symbol -▁TH -▁Telefon -▁Soll -▁Eas -itten -itarian -▁cast -▁Hoff -▁Ra -▁UK -▁Lang -▁NAT -▁Zivil -uche -▁jähr -▁John -▁Annahme -schaff -▁advert -▁Einwander -spunkt -▁produc -▁Prozent -▁Ihres -▁Gespr -▁decor -▁remov -krise -▁Conference -stitutional -erkennung -▁150 -▁Österreich -▁interested -▁Swiss -▁critic -▁tourist -▁Angelegenheiten -▁challenge -▁farmers -▁terrorism -▁elegant -▁angemessen -▁erinnern -g. -80 -▁tal -▁hor -▁ris -▁sil -▁walking -▁Sonder -▁Drit -▁Mc -▁games -ena -▁cert -▁lands -▁Gold -arge -▁angen -▁angebot -▁Licht -▁ones -▁zukünft -▁erfahren -tritt -▁verurs -reas -▁Stellen -▁extent -▁übernehmen -▁Schla -▁chance -▁Einrichtung -▁195 -▁disp -▁saf -▁respond -▁wanted -▁appre -▁spezif -▁valid -▁Islam -▁privaten -▁secure -▁folgenden -äuft -▁voting -▁hervorragen -▁Modell -▁methods -▁Chinese -▁scientific -▁² -▁™ -zulegen -po -äher -gült -▁fine -▁fant -▁setzt -▁candid -▁lies -▁Gran -▁pred -▁prem -▁images -reise -▁zudem -▁italien -abkommen -ielt -izes -▁coup -▁enorm -▁upd -▁releas -▁release -ienten -▁gesehen -▁selection -▁specif -▁natür -▁wiederhol -▁beginnen -posed -▁Arbeitsplätze -hältnis -ätzlich -konom -▁sectors -rengungen -▁Obwohl -▁automatisch -▁begrüßen -▁Friedens -▁Kapital -▁bitten --Office -▁allgemeinen -▁Solidarität -▁ehemal -fs -pons -äden -gent -▁hot -▁falls -▁Ther -▁blo -▁S. -▁Mün -▁lost -▁Winter -ichern -▁beein -reibt -▁deine -▁vertra -▁Versch -ehens -▁despite -indungen -▁exerc -▁beide -▁bestimmt -immern -eilen -▁darstellt -▁counter -ildet -▁Konzept -barer -struct -▁gesetz -▁endlich -▁remember -▁expert -poses -▁Trotz -▁Generation -▁long-term -▁Heim -▁letzte -▁businesses -ördert -board -▁visitors -▁etc. -atalog -▁separate -▁Milliarden -hotel -nel -rich -betr -lern -40 -▁fear -▁powers -▁Moment -▁Bu -inens -▁las -▁lay -▁länd -▁Wann -▁Hause -▁Gas -imination -▁alt -reis -verage -▁deg -▁Prä -▁Antrag -izing -etzung -▁Ausgang -icket -▁Einfluss -fence -action -bezieh -▁genu -▁trying -▁substant -haben -▁incor -handel -▁News -▁accepted -▁motor -▁Ireland -▁raised --in-Office -▁Liberal -▁Erstens -▁absolutely -▁eindeutig -▁influence -▁dringend -hilfen -bin -IV --to -pool -RE -▁sinn -▁Sü -▁Bü -itch -order -▁e-mail -▁länger -▁kaum -▁Krank -▁Falle -▁untern -ulated -▁Leute -ids -ctional -anda -▁Verfassung -▁jem -▁jun -▁demands -▁voraus -▁Erk -▁resort -▁club -▁produz -▁cred -itzt -▁benutzt -▁Freiz -zielle -▁negative -▁strategic -indestens -▁empfeh -▁Modern -fälle -▁manufactur -▁Moreover -▁bemü -▁Hostels -▁Anstrengungen -▁Secondly -▁kitchen -▁Afghanistan -figur -bes -ML -▁fa -▁reag -▁Tätigkeit -▁grad -▁geleist -eater -▁Cy -▁verme -▁gek -ustausch -▁umfang -▁entgegen -zeigen -▁adjust -▁genannt -▁genutzt -▁Grie -auern -▁argument -▁Regional -▁Georg -ergestellt -werte -▁overall -▁Only -zeichen -▁moderne -▁größere -▁Russian -▁Ausschuß -▁irgend -▁annual -▁fields -▁komfort -▁changed -▁innovation --based -▁Zustimmung -▁zusätzliche -▁borders -▁Sitzung -▁claim -▁namely -▁Verpflichtungen -▁wrong -▁unbedingt -▁NATO -.3 -ST -kategorien -äg -QL -▁dipl -▁mis -▁Tsch -▁bir -elte -▁oben -▁calls -▁listen -▁Kauf -▁bezahl -hrte -▁eingehen -▁Verhalten -▁Ja -rogen -▁Stre -▁überzeugt -gruppen -▁entspricht -▁yourself -glück -▁worked -meer -wogen -▁President-in-Office -▁ethn -▁Nachfol -▁highlight -▁Mex -▁Amb -▁belieb -▁player -▁Reserv -▁representatives -▁earl -▁occur --Ver -▁Organisationen -▁flexible -▁bilden -▁Idee -▁speziell -▁season -▁observ -▁Other -back -bringen -cia --con -DE -▁bath -▁Its -▁Sea -▁Mobil -▁Besonder -▁Bade -uses -▁Cz -▁His -▁Voll -▁intended -icip -amin -stoffe -▁erkl -▁erkennen -teile -▁gegr -▁Anerkennung -▁proceed -ado -▁Schwer -▁entstehen -▁vorgelegt -spe -▁Abschluss -▁track -▁gesetzt -ändigen -▁Haft -▁evening -schließen -▁processing -▁award -▁selbstverständlich -▁Autom -▁menu -▁Roma -▁Wahlen -▁expressed -▁Profess -▁Verwaltung -▁initiatives -▁elektron -▁Kompromiss -head -nungs -bum -5-0 -▁tran -▁talking -▁hol -▁mindestens -▁People -isely -▁cell -▁cities -aries -asc -▁bege -ften -ples -endet -▁fresh -▁ability -▁spect -ensicht -▁client -▁plans -see -▁committed -▁goal -▁element -▁1990 -▁mainly -▁photo -▁accession -▁angegeben -▁heard -▁Qualitäts -▁intell -▁slow -▁occup -▁militär -▁unterschiedlichen -▁wesentlich -▁banks -verläss -▁berücksichtigen -▁teleph -▁traffic -▁Elekt -▁datab -▁Costa -.m. -hop -oke -wenden -brit -UN -▁IM -▁Sing -▁Agre -engen -inary -ils -orient -▁love -▁anc -▁alone -▁bee -▁Lab -▁stabil -▁Inf -lichung -▁gezeigt -▁Osten -ibl -äne -▁wert -▁Aussicht -gebaut -▁mehreren -hagen -▁Überwach -▁autom -▁placed -▁concentr -▁reduz -▁Bestand -▁brand -▁Boden -▁bewusst -kaufs -▁running -▁hochwert -▁Christian -▁Bemerk -▁Kriterien -▁Holz -▁danke -▁Immobil -▁verfügbar -▁Vereinbarung -▁terrace -eine -üb -üll -SS -LA -▁tit -▁nöt -▁rom -▁bre -▁Som -onym -ories -▁Körper -▁God -atisch -▁threa -ray -▁orient -eitet -ago -▁Ann -▁Anspruch -▁auszu -▁cohesion -▁nachdem -▁disapp -▁comments -▁trust -▁emer -▁hinweisen -▁Mensch -reiber --System -▁remind -▁aktuelle -▁approved -▁opening -▁improved -▁families -▁Durchführung -▁students -▁Waren -▁entsprechende -▁Garant -▁demokratischen -▁Stabilität -▁planning -ästinens -▁Hintergrund -▁administrative -je -70 -▁htt -▁watch -▁black -▁Dol -▁EG -▁glauben -▁Agenda -▁verz -▁mittel -▁Proc -ibly -undred -law -▁Mitteilung -ertung -▁Einsch -▁Architekt -▁Spa -▁Sprachen -▁plus -gebote -▁gesprochen -▁herunter -▁decide -▁justice -wirken -▁nationale -▁suitable -▁buy -gewöhn -▁globalen -▁Glück -▁Objekt -▁Lebensmittel -▁affected -ifizierung -▁feature -▁reforms -▁automatically -▁structural -▁Besucher -▁increasingly -▁Ident -▁Vorteile -▁geeignet -▁erwähnt -idelines -▁currency -hood -ras -mel -IF -ION -pack -▁Spr -▁Mio -▁Bi -▁Arab -itory -used -▁euch -▁Card -▁Fot -▁touch -quate -▁Inhalte -▁jeg -agraph -artigen -▁Schaden -▁usually -▁court -udem -▁regime -ounge -▁Reformen -▁emissions -▁controll -zeichnen -▁reduced -▁feed -▁box -ündung -▁benutzen -▁besonderen -aschine -haften -afft -▁Fischerei -▁angesprochen -▁collection -▁wahrscheinlich -bourg -▁Homepage -▁eingerichtet -▁conclusion -ecutive -▁circumstances -aw -mus -mack -bus -▁fert -▁mill -▁Tel -▁Sand -▁Av -ores -▁cos -▁cele -▁Kind -atik -▁began -▁Nahr -▁Nations -▁Oly -▁whom -▁Anp -izenz -▁engl -▁siehe -▁Ausgaben -▁compar -ampag -ologischen -entrale -▁kontroll -▁supports -ancen -▁1995 -▁Schließlich -▁phone -▁laws -▁sicherzustellen -▁confidence -▁Komfort -▁Koordin -▁Karte -▁administration -▁delivery -▁Irak -▁Partei -▁akzeptieren -▁Zwischen -▁extensive -cluded -▁Frankfurt -▁Griechen -het -daten -ruf -rid -last -▁fish -▁says -▁Serb -▁Sektor -▁pain -▁Mer -▁Brea -▁Ac -ita -inent -▁Wik -arz -▁regr -▁zug -▁Zudem -▁extend -▁entir -icks -▁meist -▁quant -ideos -▁freuen -struktur -iums -▁Haushalt -zieht -▁applic -▁möglicherweise -▁ausgew -▁powerful -offee -▁contribute -▁priorities -▁bestehenden -▁Salz -▁Datei -▁erzielt -▁wonderful -entwicklung -▁rechts -▁Zunächst -▁Zeichen -ibilität -▁unemployment -▁infrastructure -zut -▁hoffen -▁rise -▁mm -▁Sekt -▁poll -▁Most -▁Milit -▁Pot -une -▁ehr -▁lov -▁Haltung -arus -uly -enty -▁Nutzen -▁stre -▁geändert -▁Verfass -▁Stil -▁Any -▁chann -▁coordin -akes -ongs -▁Chance -▁400 -iente -▁Umst -▁else -riend -▁Unterkategorien -▁scope -▁festgelegt -position -▁Aktions -▁ensuring -▁electronic -▁Assoc -▁encourage -▁luxur -▁Wettbewerbsfähigkeit -▁wundersch -▁victims -▁Indeed -▁Konflikt -pendence -▁Czech -▁Olymp -overy -ded -öm -▁te -▁tabled -▁fellow -▁dienen -▁manner -▁IC -▁Each -▁getting -▁AN -▁bequem -▁Us -▁NI -schlagen -legt -▁Out -▁white -▁Schön -▁umgesetzt -▁ausreich -ickets -iertel -▁Vorbereit -▁comment -▁anderes -▁freien -▁Rev -▁2020 -▁crime -▁appoint -▁Schlüssel -▁larger -▁played -▁muse -▁Öl -▁Straße -▁notwendigen -▁introduced -▁finanziellen -▁participation -▁agriculture -▁erfüllt -▁zuletzt -▁replac -▁Funktionen -▁prüfen -▁specifically -▁http -▁east -lim -▁tre -▁nah -▁SP -▁po -▁Power -▁Pflicht -▁Beach -edia -eap -eme -urb -▁lösen -▁Waffen -▁Kenntnis -▁Code -▁thr -▁beziehen -scher -▁ergreifen -▁jene -allel -eckt -▁Ausbildung -▁3. -▁Konz -führer -▁screen -▁India -▁gering -▁effectively -▁Monate -▁functions -▁competitive -▁Beitritts -▁jederzeit -▁Fischere -▁Partnerschaft -▁pleasant -▁Technologien -äuser -▁Suche -▁sources -▁Dateien -▁animals -▁Klein -▁zufrieden -▁Kommunikation -▁platform -hy -nes -book -pp -▁hundred -▁fahren -▁rural -▁sitting -▁Bec -enhagen -inian -▁ec -▁Gemein -▁formal -▁reception -trägen -ugg -▁ship -▁Schengen -▁church -▁iP -▁unters -schlossen -▁Aufnahme -chtet -▁served -▁Asia -▁protected -▁serve -▁gleiche -▁Richtlinien -▁Kindern -▁Vielfalt -▁referred -▁installation -▁critical -▁braucht -▁aimed -prüfung -▁motion -▁purposes -▁verbessert -▁schwierig -▁Paul -▁According -▁wünschen -▁detailed -▁exactly -▁vermeiden -britann -▁é -.S -ael -▁heav -▁dead -▁mix -▁Tro -▁Meter -▁camera -▁lesen -ickt -aria -amber -▁Infrastruktur -ifying -odies -▁ausger -idents -echt -▁Einrichtungen -▁Arten -iano -▁trav -▁darunter -fahrens -werke -▁auth -▁wants -urses -▁Why -▁Produkten -sprü -losen -▁Board -▁sicherlich -▁represents -▁discussed -mitglied -▁stattfinden -▁Zweck -preis -▁Beschäftigungs -▁institutional -▁February -▁Luxem -▁becoming -▁spacious -▁CO2 -itzerland -▁risks -ham -nam -mission -TA -RA -▁fäl -▁ries -▁süd -▁moral -▁Tun -▁Sinn -▁Middle -icts -ulierung -schreit -▁stated -▁ständig -ingu -iffer -eget -▁Then -▁Schiff -▁entschieden -▁spend -ensions -▁managed -▁police -▁Arch -▁Spec -▁competen -▁developments -▁sub-c -eather -▁angezeigt -▁gemüt -▁Standards -▁facilit -▁discussions -▁precisely -▁temper -▁domin -▁Slov -▁Apartments -▁jeweils -▁Innovation -▁likely -▁Wellness -uisine -▁entweder -▁Agreement -wagen -▁fallen -▁vac -▁mid -▁mand -▁metres -▁Sehens -▁gern -▁Ky -▁Heil -▁Rein -▁Get -▁inn -atively -▁alte -▁1, -osed -▁vert -egu -ellte -▁shower -▁Wege -▁otherwise -▁Lehr -▁sold -▁Menge -werten -▁offensicht -▁selected -▁inside -▁gewün -▁remark -▁gelangen -▁01 -▁Research -▁Policy -▁budgetary -▁Switzerland -ING -▁Turk -▁programs -▁Großbritann -▁radio -▁erzielen -▁übernachten -▁komplett -freiheit -▁mittler -▁convin -▁thousands -ral -pon -kap -gers -gelt -▁mil -▁winter -▁ging -ela -inu -chmal -▁centr -▁Wass -▁Found -ida -rekt -rahmen -▁ergän -▁Stimme -▁entry -idential -▁Abschließend -▁actual -▁provider -▁Frem -▁hands -▁Grundsatz -▁wood -▁expand -▁expected -bruch -▁medical -▁Appart -▁measure -▁Rooms -▁votes -▁teilweise -▁listed -▁Bush -▁closely -▁Verwaltungs -▁covered -▁ISO -▁erhöhen -▁corresp -▁Februar -▁assessment -▁Forderung -gelegten -▁sub-categories -sam -wes -▁rent -▁sagt -▁wären -▁Sow -▁EAD -▁paper -▁Pack -▁Prag -itiveness -umm -▁impress -leisch -▁stimme -▁erf -▁erstellt -trol -▁gekommen -▁Jetzt -▁-- -▁abgel -▁chines -▁ausdrücklich -▁Arbeiten -▁Def -▁welcher -▁guidelines -▁technische -▁dram -▁seek -▁debt -▁eingel -▁Click -▁besuchen -▁Vereinten -▁break -▁kommenden -gemeine -ikanische -▁color -▁introduction -▁stärk -▁Office -▁followed -▁Klimawand -▁Status -▁vorhanden -▁appreci -▁earlier -ül -▁sauber -▁mic -▁Top -▁Mir -▁graph -omin -union -▁Wien -▁Kol -iefer -imb -ania -ala -▁anbiet -▁Name -schuld -▁Angr -▁abst -laus -▁spread -▁Ern -▁Mitte -▁betracht -▁betonen -problem -ishes -▁save -▁Leider -▁Berichte -▁Plen -▁Entwurf -▁Bereit -▁beschlossen -atoren -▁opened -▁improving --called -▁buildings -▁territor -▁informieren -▁electric -▁multil -▁enterprises -▁Leistungen -quality -▁komplex -▁funktioniert -ength -▁wollte -,00 -rum -vements -bad -SP -▁fragen -▁ade -▁breit -▁Banken -eters -itter -▁lernen -▁Kal -▁Kurz -anlagen -techn -▁verur -▁Unabhäng -▁Street -▁abge -▁uses -▁36 -▁leist -▁trip -▁beds -▁Orig -▁supporting -▁handeln -▁Teile -azz -▁operations -▁considerable -▁Belarus -▁linked -▁Warum -▁networks -▁understanding -▁Abgeordnete -leitung -▁Instrumente -▁Webseite -▁Währungs -▁exclusive -▁Check -▁Greece -▁Veränderungen -▁Botschaft -▁Entsp -▁Initiativen -▁telephone -hel -fachen -ünen -möglich -▁tele -▁rob -▁map -▁mobil -▁went -▁Thr -▁bott -▁Eisen -▁Mö -▁Mun -ora -olden -▁ober -▁cuisine -▁Wo -▁Hilf -▁GN -stechn -▁Verständ -▁Station -ield -▁dagegen -▁so-called -▁Wirk -▁Absicht -▁compens -bek -projekt -▁darstellen -oyal -▁provision -▁Zeitraum -achtung -▁criminal -istrib -▁zweif -▁marketing -▁Systeme -gewicht -▁Vielen -▁immigration -▁gewährleist -▁Engagement -▁convenient -▁Economic -▁degree -vo -IE -IG -DP -künd -▁russ -▁Maschinen -▁gain -▁gave -▁Band -▁Bos -iebe -▁Lib -▁Nah -▁Never -▁erleben -gencies -▁verring -▁gefordert -▁Angaben -▁'ll -▁resist -ünden -▁durchaus -ärm -gerungen -▁accomp -ießt -▁Zeiten -öffnen -▁Bruss --Mit -▁wenigen -▁Franz -▁brief -▁acqu -▁playing -▁founded -▁presence -▁Begriff -▁stärken -▁Pakistan -▁konkrete -▁properly -ühlen -▁veröffentlicht -▁Gipfel -▁components -▁e.g. -▁PHP -öp -13 -grad -▁match -▁Tool -▁EL -▁Po -▁Bay -▁Black -ura -▁Harmon -▁Fonds -▁Vill -▁unternehmen -aters -umin -schung -▁items -issionen -grammen -▁Schrift -▁entit -ennis -▁door -▁Absatz -▁193 -▁freue -▁Mitgliedstaat -istence -▁delay -▁glob -▁Hauses -▁klare -▁experts -▁Standort -▁typical -▁Phil -▁explo -▁Guest -▁tourism -▁destro -▁Ratsvors -▁Kritik -▁animal -▁Währung -▁purchase -▁Installation -▁erklären -▁Mio. -▁gewährleistet -house -ooms -set -EA -▁hält -▁vere -▁Te -▁coffee -▁kr -▁kor -▁Health -umän -ulate -▁wirks -▁Anteil -▁alloc -▁entscheid -iziert -▁coordination -aces -▁plants -örde -▁Geist -▁trägt -▁hinzuf -▁liebe -▁strict -▁Klicken -▁eingesch -▁beru -▁competitiveness -▁presidency -▁Worte -▁generally -▁behav -▁Grenz -▁benötigt -▁Herausforderung -▁Casino -▁2011 -▁lokalen -▁fisheries -▁duty -▁permanent -▁Schlaf -▁entirely -▁Because -▁Sehenswürd -▁EADS -▁Brussels -ua -so -mas -wesen -wasser -lad -▁tack -▁hosp -▁sorg -▁Dienste -▁Mail -▁Mount -▁Bern -▁Great -▁unn -▁erfolgen -▁Anges -▁shops -▁willing -ova -okraten -▁mail -hnung -club -flanzen -▁relate -▁indic -führten -▁Brüssel -▁Herstellung -▁klicken -▁möglichst -▁Poland -konferenz -▁refuge -▁authority -▁dealing -▁Gebieten -▁difficulties -ifiziert -▁peaceful -▁Sonnen -▁Regelung -▁Kompon -▁Netzwerk -qualität -▁Prioritäten -▁Königreich -▁erscheint -▁Immobilien -ückt -▁af -▁Titel -▁psych -▁Bran -itze -▁Kenn -▁Fen -ameter -▁Non -ortion -▁erstellen -try -hrase -vey -▁Unfortunately -▁OS -plant -uba -▁früh -▁attra -▁Environment -eals -▁gesp -attle -▁Übergang -▁Allgemein -▁Hafen -▁einzeln -▁format --Pl -▁klass -overs -▁dennoch -leistungs -▁facing -▁organisation -▁Madrid -▁combat -▁nächste -▁district -▁Zweifel -▁summit -▁finanzielle -▁recognis -▁Jahrhunderts -▁Dennoch -▁Stellungnahme -▁konzentrieren -▁Nachfolgend -work --20 -90 -▁täglich -▁hour -▁average -▁sand -▁MP -▁Mechan -▁PT -▁AK -▁Achtung -▁zerst -▁kost -▁Wander -asten -▁forget -reu -▁Int -denken -▁constit -▁jur -ranken -▁Einigung -▁Einhaltung -clude -▁Aufbau -▁percent -▁gegenseit -▁Asyl -▁einzus -▁paragraph -▁applies -▁Freund -▁Gewähr -▁meetings -▁Terrasse -▁speaking -▁machines -▁persönlich -▁unterschiedliche -▁Super -▁calc --star -▁Forderungen -▁vehicles -▁Völker -▁Entschließungsantrag -▁gegründet -▁Schwerpunkt -mär -cement -äten -▁tut -▁suc -▁PDF -oring -oman -▁letzt -aste -asts -umn -▁verfolgen -elled -▁exec -▁übertragen -odes -▁registr -▁legen -▁Formen -▁contem -▁fire -▁Unterkunft -haber -▁Flücht -▁Landwir -▁Atlant -▁numbers -▁thinking -▁Versuch -▁consideration -▁imposs -gemäß -yright -▁spielt -▁umfasst -▁Kirche -▁declar -iques -kräfte -▁Santa -▁becomes -▁Analyse -▁amerikanischen -▁fixed -▁befürwort -▁geleistet -▁Breakfast -▁Großbritannien -▁betrachtet -IR -MU -▁told -▁nennen -▁sym -▁born -▁ST -▁Sach -ema -inc -isierten -▁ep -▁Wicht -▁Fore -▁Vien -enteil -schul -▁denken -inding -▁schönen -peratur -▁Forum -steuer -▁Jahrz -▁income -▁endgült -▁Once -▁Barros -licherweise -▁medic -▁verlassen -äufer -▁functional -zeption -▁practices -tworten --Hotel -▁flexibility -▁Private -verkehrs -▁Paket -▁Motor -▁opposition -▁picture -▁unabhängig -▁diskutieren -▁Gemeinsamen -▁Texte -▁Sehenswürdigkeiten -▁succeed -d. -ress -zesse --y -karten -▁sn -▁bodies -▁Dorf -orfen -▁Wün -▁Katastrop -▁CH -▁Hard -▁Gut -arch -umen -▁imag -lying -▁concr -ibel -iking -▁3D -▁lift -▁superv -▁Umfang -▁myself -▁overl -arbeitet -▁Hersteller -▁feststellen -▁signal -▁Stein -▁advanced -▁hören -▁begrenz -ashion -▁erforderlichen -▁arbeitet -▁aims -▁load -▁domestic -▁Ideen -▁positiv -▁Rechnungs -▁Lesung -▁garantiert -▁Ruhe -▁Anwendungen -▁database -▁Nahrungs -meld -letter -komp -▁sle -▁middle -▁Typ -▁Sof -▁Sorge -▁Mel -▁Alex -▁Wall -▁Hall -arischen -ams -▁Note -reisen -▁stable -▁erleicht -upp -cknow -▁geograph -illing -arta -▁sob -▁Auss -▁Argument -▁propose -▁politics -▁Sehr -▁signed -zeugung -▁approxim -▁Polen -▁easier -▁Vielzahl -▁defined -▁himself -▁Gleichzeitig -▁Charakter -preise -▁Kompeten -esterday -▁Links -▁Hamburg -embly -▁Stärkung -▁überprüfen -▁ancient -▁Association -▁ausreichend -.4 -viel -üter -band -St -loss -▁frag -▁Sur -▁Saint -▁Mindest -▁Werb -atischen -ürz -▁NG -stimmung -▁secr -▁Ze -▁einiger -▁Under -plätzen -ellos -▁clar -▁33 -▁192 -älter -▁eth -▁hingewiesen -▁Seine -▁Entwicklungen -▁75 -▁1994 -krieg -▁Arbeitslosigkeit -▁financing -▁Resort -▁richtigen -▁informed -▁patent -▁seriously -▁Patienten -▁failure -▁establishment -▁translation -▁Rechnung -▁parliamentary -▁kulturellen -▁experienced -stattung -▁initial --Mitglied -.e -tes -no -seh -IB -öder -äck -▁hop -▁acknow -▁Digital -▁Event -▁Main -▁Künstler -▁Finn -▁una -▁threat -▁bezeichnet -quar -▁zuvor -▁erg -▁miteinander -road -▁cooper -▁Ausfl -accept -aken -▁48 -ologische -▁anyone -gestimmt -▁Defin -▁Landschaft -▁Bedarf -▁Verbrauch -kret -▁differences -▁estate -▁models -▁studies -▁connected -▁Afrika -▁promoting -▁Risiko -▁profitieren -▁dangerous -lösung -▁voice -▁tätig -▁Heimat -▁celebr -wan -wurf -▁spl -▁Dollar -eto -▁unz -icians -ichert -strateg -▁Inc -ista -herr -▁defence -▁Over -ehl -▁withdra -schau -▁19. -▁indir -▁weiterer -▁positions -▁difference -▁humanitarian -▁verloren -▁Polize -versorgung -▁combination -▁zahlen -▁organiz -▁booking -▁beaches -▁Tourismus -▁Irish -▁Irland -▁structures -▁eingeführt -▁Support -▁appears -▁Leistung -▁Leistungs -▁reservation -▁happened -jährigen -ibrary -zwischen -▁vernünft -▁neighbour -▁Kingdom -▁associated -▁Bedürfnisse -aill -üst -wel -pak -CO -▁names -▁grenz -igem -omas -▁length -▁HT -▁unaccept -▁forest -regel -▁hair -▁aufmerksam -▁Verst -terne -weist -▁courses -▁trag -▁accur -▁emb -führungen -▁gewinnen -haltet -▁Problemen -▁Teilnehmer --Pr -▁instance -▁Beschreibung -▁Trade -▁article -▁implementing -kontr -▁statements -▁carefully -▁colour -▁England -▁beginnt -▁Firstly --Programm -▁attractive -▁Effekt -▁houses -organisation -▁Altstadt -▁exhibition -▁safegu -▁impossible -ny -fu -gu -▁agencies -▁satell -▁Seg -▁Due -▁Ban -etic -etragen -instrument -▁zentrale -▁Fotos -iche -aby -▁einsetzen -▁condem -▁Os -▁überall -▁shared -ungssystem -▁enth -▁ausführ -▁spent -▁vorbereit -ektion -▁Abs -▁relativ -▁geschehen -▁reducing -▁leader -coming -ohlen -▁unterstütze -▁Beitritt -▁arrival -▁collabor -▁Geschäftsordnung -▁Diskrimin -▁üb -▁Justiz -▁immediate -▁danach -▁announced -▁architecture -▁producers -▁Überwachung -▁Ratsvorsitz -▁Angesichts -▁concrete -ión -uber -uces --3 -11 -▁Poker -▁Are -▁Foto -▁Rand -▁isol -▁unto -iegel -atically -▁Live -tenance -▁einst -▁Jeder -▁Chancen -▁truly -▁evidence -nahm -stitut -▁USB -▁geschützt -ategory -▁calling -▁charged -▁besitzt -▁Meiner -version -▁Deutsche -▁adoption -▁Technik -▁regionalen -▁reasonable -▁guarantees -bieter -andidaten -▁gegenwärtig -known -▁Priorität -▁vorliegenden -▁Michael -▁bestätigt -▁keinerlei -▁sämtliche -▁released -▁Griechenland -▁television -▁Barroso -▁• -where -II -EO -ée -▁näher -▁fle -▁ahead -▁mist -▁Dus -▁Drogen -▁cards -▁Wi -▁Fel -asil -ulations -▁Nav -▁haupts -▁verfolgt -▁Prime -▁excess -präsent -▁chair -▁ausges -eilt -▁politischer -▁großes -▁delight -▁spezielle -▁availability -▁background -▁vollkommen -▁einfache -▁Mittelmeer -▁advance -leihen -▁Festival -▁Zuständ -▁taste -▁Justice -▁Fortschritt -▁relaxing -▁recognition -▁weak -▁Elemente -▁described -▁Europäer -▁genuine -▁Foundation -▁grenzüber -räge -zig -Script --de -99 -▁roll -▁wachs -▁Mik -▁Put -▁Bis -▁Back -ili -onia -▁Custom -▁Gent -ando -rochen -ugos -▁Anlagen -▁entertain -schläge -▁dispos -▁possibilities -▁gestalten -▁monetary -▁owners -▁größeren -▁Media -mögen -▁improvement -▁160 -▁Bewertung -▁wishes -sequent -▁nearby -▁membership -wachs -▁ök -▁urban -▁Verpflichtung -rüst -iscal -▁beteiligt -▁französischen -▁Mutter -▁discrimination -▁Gerichtshof -▁photograph -▁angeboten -▁offensichtlich -öss -AG -LE -▁Syn -▁pour -olz -▁ow -▁kons -▁Wand -▁Colle -▁unf -▁unless -▁prä -▁Lo -▁Lam -steht -ibet -▁Erhö -range -heits -▁trading -örter -▁warten -erma -▁Commun -▁glass -▁Kräft -▁cars -▁gutes -▁Indien -▁continues -▁11. -▁territory -▁skills -box -▁consumption -▁Worten -▁African -▁absolute -▁tasks -▁Regelungen -▁zusätzlich -▁continued -▁garantieren -▁Globalisierung -▁figures -windigkeit -▁empfehlen -.S. -▁unacceptable -lig -▁Tisch -▁Eindruck -enkt -otten -chem -▁Wald -▁unfortunately -▁ergeben -▁einzu -▁Very -roh -▁desire -▁allowing -ador -▁dauer -▁sogen -ensus -▁enables -icking -▁34 -▁37 -▁grunds -▁gratul -▁generation -▁internationaler -▁expend -▁Maße -▁greatest -▁eingere -▁attach -▁folgende -▁Urs -▁explain -▁combined -greif -▁packages -▁raise -▁Universität -▁introduce -▁participate -▁completed -▁strongly -▁devices -▁höchst -▁surrounding -▁umfassende -▁gegenwärtigen -gypt -▁beinhaltet -zz -wissen -bett -brauch -body -can -lement -öse -ßt -äfen -▁dur -▁vul -▁SD -▁Data -▁Mond -ilst -▁liter -▁Korr -▁Karten -▁Ren -▁Nether -ido -reens -verstanden -▁abgeschlossen -▁spät -▁Europeans -▁Größe -▁command -▁solcher -▁transition -▁qualit -▁Monat --sized -▁Direkt -▁beauty -▁verbundenen -▁Märkte -▁window -▁attractions -▁religious -▁burden -▁Vorsitzen -ausschusses -▁registered -▁verabschied -▁romant -riendly -▁Turkish -▁Vienna -ih -wie -▁harm -▁river -▁mission -▁Il -▁Dut -itage -itious -irus -ilot -olge -▁Ca -▁Hist -▁Fac -▁Rome -▁ung -ammer -ürk -▁Lake -achsen -▁shop -allery -▁enor -ource -▁Bek -▁button -▁gramm -▁gross -linge -▁insurance -▁economies -▁appeal -▁Gebrauch -▁surface -▁teilnehmen -▁closer -▁meters -▁steigen -▁Nutzer -▁Gründe -▁Todes -▁wesentlichen -▁Meeres -▁Erklärungen -▁Democrats -▁Beteiligung -▁müs -▁dedicated -▁entitled --J -▁tief -▁dient -▁acht -▁sale -▁square -onic -▁ca -▁Fitness -iech -iest -▁ther -▁beglück -▁Lizenz -▁stim -istisch -istik -▁asyl -▁aufzu -▁vergleich -▁geöffnet -▁Angel -ockt -▁fruit -▁Erde -▁manage -▁betrachten -▁Grunde -▁4. -älfte -▁outd -attform -ancing -stands -▁ausgewogen -partner -enntnisse -▁Gegensatz -hebung -▁learning -▁Relig -▁erhöht -▁speziellen -klusive -▁exceptional -▁Federal -▁Städte -▁Bahnhof -▁enormous -▁Unabhängigkeit -netz -sicherheit -yt -government -ía -▁fern -▁dre -▁session -▁vision -▁buffet -▁Book -▁AR -▁langen -▁Kurs -▁Ganz -▁Good -▁union -asen -ases -leun -ifest -uchten -▁Verz -▁Anre -▁Another -ikal -▁Erhol -atever -▁Beim -▁38 -▁compared -▁reliable -▁kept -mittelt -▁gelungen -vorschlag -▁meant -▁para -▁perspective -▁installed -▁Studio -▁totally -▁Austrian -▁luxury -▁railway -▁Kohäs -▁Flexibilität -▁exercise -▁beglückwün -▁asylum -gal -▁heat -▁Day -▁gemeinschaft -esign -isierte -▁forced -▁behin -▁Ul -▁Lag -ktiv -▁erfre -▁Jac -ello -▁extended -▁spending -▁Einwohn -spart -▁trad -▁daraus --Sch --Sterne -▁Front -▁Nachdem -▁begleit -▁stands -▁humanitä -▁Hinweis -▁Aktionen -▁earth -▁informiert -▁individuals -Innen -▁Market -▁Marken -▁Weitere -▁biggest -▁holidays -▁schöne -▁claims -▁starke -▁storage -▁Kopf -▁britischen -▁Ereignisse -▁advertising -ibliot -▁Militär -▁Verständnis -▁Nevertheless -.de -hold -km -▁diss -▁sides -▁Pool -▁PPE -▁BM -▁lauf -▁Wür -▁Head -▁Rob -▁Van -iced -▁story -▁Vermitt -▁abzu -▁manchmal -▁Gefühl -▁Reich -▁emerg -iode -▁Strom -▁98 -▁science -▁Control -▁guter -▁Aktion -▁Datenschutz -▁250 -▁intelle -▁Company -▁corporate -▁Studien -▁motiv -▁device -▁Farben -▁Slow -▁failed -▁mountains -▁Martin -▁granted -▁Empfehlungen -▁urgent -▁memory -▁Süden -▁diplomat -▁convinced -▁Foreign -▁Netherlands -lag -person -LAN -▁hum -▁nieder -▁divid -▁veget -▁golf -ompl -▁core -▁Fert -▁Niveau -benen -lichem -▁verf -▁eingeb -auso -▁Ausland -▁Wirkung -▁Fraktionen -ittel -▁5. -▁elimin -▁offic -▁20. -klang -▁benach -▁advice -▁Quest -▁Palace -▁passed -▁controls -▁moving -▁beachten -▁Kontroll -▁magn -▁Realität -▁Wiederauf -▁conclusions -▁Mitteln -▁Latin -▁legitim -▁Agentur -▁Eastern -▁Hoffnung -▁intelligent -▁sorgfält --year -▁hauptsächlich -ibliothe -oe -det -bil -▁fiscal -▁rasch -▁DS -elbe -▁Croat -▁Feld -▁unit -▁unable -icious -▁prison -▁Institute -▁weight -▁Press -▁gehalten -▁einverstanden -▁Anlage -▁allgemeine -▁Third -▁Schm -▁enfor -▁canc -▁finance -▁disease -▁Regulation -▁treten -▁saw -▁herzlich -▁guide -▁contrary -▁geschieht -reitet -reitung -▁außerord --Kon -▁establishing -▁definit -▁Zahlen -▁Betriebs -▁ideale -▁perfectly -▁ECU -▁mittels -knü -▁Greek -▁Beschluss -▁Gedanken -▁Special -bas -ließen -▁Treffen -▁Dimension -▁Mas -onial -oram -▁kaufen -▁Kaff -▁Vom -ufe -gehör -▁forth -idung -▁Verkauf -▁jour -▁Journal -▁überein -▁chosen -laufen -▁ausländ -▁'re -active -▁möglichen -▁tried -▁interface --Str -▁damals -▁Haut -▁einzigen -▁scale -▁factors -▁Constitution -▁festgestellt -▁Blog -▁Programms -athed -▁Beihilfen -▁artists -▁schneller -ädt -▁definition -▁Hauptstadt -▁drawn -▁MyS -▁Voraussetzungen -▁Risiken -▁niemand -▁jüngsten -▁sobald -ries -s- -TS -▁türk -▁Peter -▁Bill -▁Kreat -▁Grad -▁unc -anische -rap -vertr -▁neither -▁verk -▁contra -▁jungen -▁Jean -▁Stad -endum -▁abhäng -▁Thus -enswert -▁quar -▁42 -▁Yet -▁laid -▁selten -wirk -▁1992 -▁rightly -▁Teilen -▁maintenance -▁somew -▁blog -▁Gebiete -▁erfordert -▁expression -▁balcon -▁Garden -▁commitments -▁vorgeschlagenen -▁weapons -going -▁spricht -▁route -▁Erwart -▁honourable -▁Gegenteil -▁unternommen -▁destroy -dukt -jet -án -▁agency -▁sage -▁win -erver -olo -▁Rumän -▁Gaz -anal -uftrag -▁Los -▁Inner -▁conver -odurch -uble -größ -▁fried -▁umwelt -▁spa -▁sched -▁Beob -▁update -▁outcome -▁outstanding -▁intervention -▁Mitgliedern -▁Grundrechte -▁photos -▁Arbeitsmarkt -▁vari -▁revolution -▁Phase -▁achieving -▁defend -▁sensitive -ategorie -igrants -▁Marketing -▁persönliche -▁übernommen -▁ambitious -gefähr -▁Männer -▁schlecht -▁chemical -▁regelmäßig -▁candidate -▁condemn -acc -zed -wachst -bea -bring -▁sell -ismen -isierungs -urm -▁Nik -reck -stät -quir -apers -▁Presse -▁Arm -▁Sprach -▁vieler -▁rock -▁gewiss -akte -ventions -▁belong -ancial -▁nationaler -▁appar -ählten -▁flight -▁aircraft -▁investig -▁Politiker --Reg -▁Zugriff -▁teilen -ränken -▁barr -▁zunehm -programme -▁Klimaanlage -▁Welche -▁surrounded -▁colleague -▁nachhaltige -▁verstärkt -▁obviously -▁constantly -▁Bestandteil -▁anbieten -▁approximately -.e. -iov -oul -rant -ügel -bindung -MS -▁wodurch -▁Tele -▁Sun -▁Dort -▁Ep -▁EP -▁Az -eable -▁category -▁Getr -▁unan -alia -▁beob -▁yesterday -▁zugleich -endung -elles -▁vorschlagen -▁Alter -▁Alban -▁compat -▁Ref -▁election -▁Mitglieds -▁someone -▁Teilnahme -▁accessible -▁Manager -▁medizin -▁kleiner -▁organised -sequently -▁limits -▁Nichtraucher -▁massive -▁continuing -▁Option -▁promotion -▁obligations -gesellschaften -▁vehicle -▁betroffenen -▁THE -▁correspond -▁dig -▁Ev -▁Ebenso -▁Bol -▁zählen -▁Hälfte -ancy -cepts -abilities -ipe -▁Wechsel -anger -ausende -▁Beif -▁goals -▁Chem -▁Registr -▁Reis -▁Refer -utzung -▁contents -▁elected -▁gewählt -▁berecht -handels -▁minute -▁Verfüg -▁successfully -▁payments -▁Komb -▁traditionellen -▁distin -▁Karl -▁Touristen -▁situations -wegs -▁equality -▁praktisch -▁Effizienz -▁Located -▁Prüfung -▁beschäftigt -bürger -▁Liberalisierung -▁title -▁gewünsch -▁satellite -ermaßen -governmental -▁zunehmend -räume -▁h. -▁hergestellt -▁race -▁wealth -▁Syr -▁Square -▁por -▁Agency -etern -eshalb -▁Win -▁Kat -▁CS -▁Fax -▁Royal -▁instruct -angebot -▁Lim -▁Lord -▁store -▁asking -▁sichern -annung -▁cool -▁ide -▁Report -▁gesellschaft -ifications -▁Gesicht -▁surv -▁bound -▁deutsche -▁erneuer -▁Internationalen -ränkungen -▁organization -fähig -▁Liebe -▁21. -▁balanced -▁recommendations -▁Prakt -▁Führung -▁besseren -▁Details -▁physical -▁Experten -▁Erhöhung -▁expenditure -TP -▁rig -▁rot -▁Merk -▁Messe -▁Morgen -▁Pop -▁Publik -oret -urr -▁kilom -▁Ros -▁Via -deck -ario -ques -▁Zimmern -apter -uropa -▁Old -▁Streit -prinzip -▁Schne -anging -▁i.e. -▁amt -sprach -▁andererseits -struction -▁lake -▁Überprüfung -istenz -▁800 -▁wichtigste -▁Island -▁Glas -▁independence -▁Straf -▁write -▁changing -▁Norden -straße -▁Ferienwohn -▁self- -▁diskutiert -▁furnished -▁medium-sized -▁JavaScript -▁criticism -▁Koordinierung -hann -nd -oß -vern -24 -▁hil -▁bars -▁Eth -▁pun -▁MEP -era -aschen -▁Those -▁Enjoy -▁Weiß -obility -▁Ausnahme -eases -▁Aufgrund -▁vielfält -▁acts -▁consists -▁weiteres -▁Verbesser -selben -richtlinie -▁begann -▁Akte -▁Autor -▁Stadtzentrum -▁Admin -▁Eltern -▁Urspr -▁irgende -▁Gästen -▁participants -▁III -▁Apartment -▁targets -▁waiting -zwungen -▁merely -▁Veranstaltungen -▁Laufe -▁Summit -▁Methode -▁kürzlich -▁fällt -▁contemporary -▁grenzüberschreit -äut -▁dimension -▁ID -▁Both -▁A3 -▁cer -▁RO -▁Rund -▁unl -▁Nu -▁seven -▁depos -▁webs -▁gee -▁Jose -▁willkommen -ahrt -▁sieben -▁finde -▁Maria -▁kontext -▁respekt -▁Übern -▁species -enzial -ablen -ansion -▁glad -▁prevention -▁aktual -▁smaller -▁seitens -▁Energy -▁putting -▁preserv -▁communities -▁slight -▁labor -gänglich -▁Inseln --200 -▁neighb -▁Methoden -▁Palestinian -▁landscape -▁jemand -▁junge -▁globale -▁einzelne -▁Diskriminierung -hat -iam -üben -TE -15 -ér -▁rac -▁Tit -▁Sä -uting -▁Cult -▁Rem -▁Rules -▁Graf -umer -▁anz -▁re- -▁reprod -▁Zoll -hrt -▁Je -▁Anliegen -wechs -▁Einkaufs -▁count -gendes -▁factor -erstellen -▁bringing --Bahn -▁Autob -▁Class -▁print -▁Formular -▁leads -▁guaranteed -▁optimal -gehoben -▁demokratische -▁contained -▁akzeptiert -produktion -▁inspir -▁prospect -▁fraud -▁sinnvoll -▁reagieren -▁Original -▁Bosn -▁disposal -zil -zip -wers -▁toler -▁nav -▁wall -▁Tiere -▁bür -▁Mat -▁Pflanzen -uters -▁echte -▁Foll -geht -▁stren -▁Zuf -▁Zul -tei -trib -▁Veröffent -▁Wegen -▁Berücksicht -▁registration -▁grounds -owned -▁assign -▁pointed -▁120 -▁capt -▁password -äumt -▁Sweden -▁acceptable --making -▁nearly -▁Begr -▁zunehmen -▁Willen -▁Korea -▁Thanks -▁planned -▁Rotary -organisationen -vereign -▁anerkannt -▁ländlichen -▁substantial -▁Vorbereitung -▁Angriff -▁hospital -▁dritten -▁Ton -▁Dubl -▁pill -▁MS -▁AB -uste -entially -rep -▁verteid -arth -▁School -▁cheap -▁sovereign -etzungen -▁cook -asting -▁betont -bef -▁griech -▁genü -▁190 -▁nochmals -▁freie -▁600 -mittag -▁Alliance -▁seeking -▁valu -▁brings -▁angeht -▁operate -▁responsibilities -▁resulting -▁blue -management -▁Gesetzgebung -▁Zustand -▁Subs -▁manufacturing -▁causes -▁behandeln -▁examples -▁positiven -▁dynamic -grade -▁Fähigkeit -▁Ordnung -▁Anpassung -▁Fenster -pass -CD -▁fuel -▁band -▁Sein -emen -isure -iler -▁lag -▁Kilom -▁River -▁GM -imal -▁bewegen -ctual -▁diesbe -▁conclude -▁Verhältnis -▁sicherstellen -▁Stunde -anne -▁umgeben -▁Chamber -▁trend -▁recovery -▁konstrukt -inning -▁Flash -▁USS -▁assum -▁1993 -▁begonnen -▁führenden -▁surpr -▁capable -▁äuß -▁engag -▁letter -▁pleasure -▁Patent -▁testing -▁strengthening -▁Reaktion -▁berühmten -▁Unterschiede -ampagne -▁Revolution -▁Klimawandel -▁withdraw -▁Dutch -rom -sein -moking -gment -64 -▁fashion -▁mang -▁Sud -▁Pin -erber -otte -orough -unsten -▁zone -▁Fischer -▁Rout -entscheid -▁Nice -ahnen -▁Theater -zeß -▁Einklang -cling -▁emp -▁decades -▁scen -▁Where -▁formul -▁angep -zahlung -▁governance -▁Freunde -▁verlieren -▁Obama -▁multiple -ipping -▁Englisch -▁identity -▁Charter -▁Fälle -▁association --how -▁spaces -lebnis -▁Minderheiten -▁Freizeit -▁Cypr -▁Architektur -▁Umständen -▁secret -vant -▁hilf -▁fand -▁fift -▁dé -▁rul -▁sets -▁Two -▁Dadurch -▁Adresse -etch -itar -igern -iern -ality -▁fordert -ktop -▁stri -▁zuverläss -▁weg -▁notice -▁Stück -▁überzeugen -▁attend -▁Argent -▁leiden -▁vielmehr -▁55 -ettable -utzer -▁consensus -fried -▁Fluss -▁Financial -▁driving -▁versions -▁parallel -▁philosoph -▁books -▁patients -▁starken -▁previously -▁copy -▁primary -▁southern -▁Perspekt -▁authent -▁adequate -▁safeguard -▁Verfügbarkeit -h. --4 -kunden -gas -▁aft -▁Bla -chnet -▁ziem -▁HD -▁beig -▁U.S. -▁Lö -lechter -▁ergriffen -grenzen -fügt -iertes -▁truth -folgerungen -stellungs -▁bedarf -▁Bedenken -▁Prodi -▁remove -▁Express -▁White -▁Handeln -▁expensive -▁Kollege -▁beschrieben -zahlt -▁Freude -▁industries -grundlage -▁Law -▁discussing -▁addressed -▁steel -▁Grenze -fertigt -▁requested -▁derzeitigen -▁zustimmen -▁Diskussionen -▁primarily -▁Romania -▁ehemaligen -▁incorpor -▁vulner -▁Kombination -hops -nach -wart -bu --of -▁nations -▁mother -▁master -▁bildet -iny -oles -▁läuft -▁WC -▁Kunde -▁Hor -▁inhab -ierend -atlant -▁styl -▁seien -▁Angebote -▁vorgenommen -oster -▁thereby -▁liability -▁Champ -aving -▁gewisse -▁Export -pielt -▁wiederum -▁Schloss -▁befassen -orgnis -▁representative -▁explan -▁kinds -▁Gesetze -▁unterschiedlich -▁höchste -.200 -▁schools -▁attacks -nehmern -▁Trek -▁talks -▁München -▁nötig -▁afford -▁Berücksichtigung -.org -mon -ky -▁sem -▁sod -▁scr -▁bon -▁Satell -▁Elect -▁pet -▁Pic -▁eur -▁Find -▁therm -▁alter -▁Präs -issens -▁Ander -▁ausgel -▁doct -▁gründ -▁consolid -▁Menü -bsicht -▁welchen -aire -▁orders -▁phase -▁beruf -▁Amerika -▁Vielleicht -▁earn -▁schließen -▁Dienstleistungs -▁default -bereiche -▁extreme -▁Mehrwert -▁turned -▁Komment -▁landwirtschaft -▁Antworten -▁konkreten -▁Voraussetzung -▁audio -▁Eigenschaften -▁reinfor -▁Johann -▁therap -▁abhängig -▁\ -▁ält -reten -afr -TER -ão -▁nation -▁disch -▁voc -▁Mach -▁Master -▁PR -▁Bearbeit -▁eight -▁zeich -▁cm -▁1000 -▁Prop -▁gepr -▁journal -▁Jes -ikas -iene -▁Schäden -▁umfass -fül -▁39 -▁betreffen -▁genauso -formationen -▁politisch -igerung -▁struktur -▁basiert -▁Expl -▁keines -▁Kompl -▁prepare -▁1980 -▁delivered -▁Dokumente -▁opposite -▁named -jährige -▁diverse -▁Untersuchung -▁evaluation -▁territorial -▁Allgemeinen -strategie -ey -fers -meth -bal -let -kop -öd -25 -Os -▁waters -▁AD -▁Cam -▁Hans -▁Ram -geiz -station -stoffen -▁reply -▁wirksam -verw -verse -▁deiner -▁Untern -enses -rance -▁grand -▁Aufsicht -ährt -▁Church -▁emot -▁noise -▁Grundsätze -▁Sound -▁gestern -▁Despite -▁18. -▁dealt -▁Kooperation -▁Westen -▁corner -▁Wirtschafts- -▁Strukturfonds -▁diversity -▁determined -zubauen -▁trotzdem -▁mutual -▁ultim -▁Premier -▁manufacturers -ho -fallen --00 --cl -▁ton -▁tex -▁hall -▁senden -▁Time -▁bran -▁SS -▁Kranken -▁Gep -▁bezieht -entwurf -▁onto -▁imper -reb -▁reicht -estr -iform -▁entlang -▁accounts -erman -▁43 -folio -ciliation -▁Minutes -▁qualified -▁wichtiges -▁Dafür -▁Datenbank -▁verlangen -▁Gewinn -▁investments -▁hoher -züg -▁Fahrzeug -▁amongst -▁feeling -▁Wachstums -▁urge -▁leadership -▁Brazil -▁unmittelbar -▁charming -▁horiz -▁Definition -▁unanim -▁ziemlich -uy -▁father -▁dah -▁volle -▁wenden -▁Take -▁Kong -▁Know -▁Caf -berecht -▁beiz -▁User -▁Lou -▁NL -stände -▁reichen -▁verändert -ehnten -anges -▁Before -▁6. -▁liefern -▁Product -▁Verbrechen -▁95 -▁führte -▁Bestell -▁spezial -▁Genehm -▁Spiele -▁außergewöhn -▁skin -abilität -▁moved -▁Skype -▁Sanktionen -▁regionale -▁Küste -▁Radio -▁Leitlinien -▁consultation -▁umfassenden -uttle -▁erhältlich -▁Signal -▁repeated -▁David -▁Kyoto -▁Munich -▁Beifall -rupt -mo -OP -ès -▁snow -▁Tibet -▁Set -▁pension -▁Balt -▁Austausch -uts -urement -▁Wil -▁COM -▁Valle -aming -ischung -▁erle -osse -▁geist -place -▁exceed -einrichtungen -▁abgesch -ungsverfahren -▁maß -▁obser -▁marked -▁geschrieben -▁Klim -▁Maßnahme -▁Heiz -▁bekannten -▁höhere -▁technological -▁historische -▁Bewegung -▁Airbus -▁runs -▁sofern -▁Disc -▁recommended -▁Schulden -▁illustr -▁fehl -▁Wahrheit -▁erweitert -▁mechanisms -beziehung -istribution -ae -uj -▁fed -▁Div -▁Dach -▁Ec -▁Egypt -▁Pe -▁gült -igsten -onte -urd -▁lock -▁Hong -atische -heber -rechten -▁eröffnet -▁Verk -▁Verträge -room -▁sprach -aker -alty -▁knows -▁mode -▁Amts -tragung -richtet -▁helping -▁attitude -venue -▁privacy -▁beträgt -▁improvements -ameras -▁Swed -▁Strategien -▁switch -▁significantly -bahnhof -▁wait -▁religi -▁experiences -▁falsch -▁Plaza -▁error -▁italienischen -▁heavy -▁Landwirte -▁essen -zza -kurs -14 -▁sicht -▁Pay -▁Bord -▁Bör -uries -▁Cook -▁inten -▁Use -gence -▁existence -▁shape -▁Schule -▁co- -erry -ieties -▁Konvent -▁Konsequ -▁output -gestellten -▁transform -▁65 -▁Que -▁requests -▁Krisen -▁compliance -▁presentation -▁Besitz -ätzung -▁configur -▁Umweltschutz -▁treated --border -▁Technology -▁zusätzlichen -▁react -▁Ferner -▁speakers -▁bedrooms -▁straight -▁Empfehlung -▁expertise -▁regret -▁streets -▁competence -▁Dublin -füllung -ader -verein -gin -▁felt -▁Tw -▁Tom -▁Tax -▁binding -▁gather -ileg -▁cold -▁lose -▁kü -arkt -▁List -▁erreich -igence -zep -eile -▁44 -eoples -▁Player -▁97 -▁drugs -▁gefördert -▁Schweden -amerika -▁kommun -▁weltweiten -▁destr -▁destination -▁kurzem -▁expectations -▁Materialien -▁künftig -▁furniture -▁erscheinen -▁entdecken -▁sanctions -▁consistent -▁launched -▁Essen -▁piece -▁blood -▁Tschech -▁birth -▁Badezimmer -▁Fremden -▁MySQL -▁websites -eb -fare -med -▁dishes -▁wild -▁Turn -▁p.m. -▁Bang -▁Baum -▁AP -itime -▁Kann -▁units -▁zugänglich -vec -▁geprü -▁convert -▁Ungleich -▁Ohne -▁Anfragen -▁plug -▁Umgang -▁Herkunft -▁parad -▁flat -▁eigenes -▁Gener -▁voller -▁besitzen -▁Mittelpunkt -▁intention -▁Studenten -▁wesentliche -▁interessiert -▁schönsten -▁recognise -▁schwierigen -▁obligation -▁Delegation -▁Führungs -▁bedroom -▁grundlegenden -▁koordin -politischen -▁jährlich -▁fantastic -▁Ambiente -▁compensation -▁Erwartungen -▁hilft -▁öffnen -fest -UM -▁fit -▁sights -▁Tim -▁Tai -▁Action -▁laufen -▁Kategorie -asters -quellen -▁zuerst -▁exists -▁entre -▁entstanden -▁spok -▁advent -▁Beweis -iana -▁relatively -▁Nachfrage -▁USD -ought -▁helps -▁browser -▁Monte -▁Roh -▁represented -▁bedeuten -▁Zahlungs -▁Studie -kommt -behörden --free -fällen -▁Welcome -▁emphasis -künfte -▁extension -▁prosper -▁subsidiar -▁rejected -▁older -▁SME -▁eye -▁hervorragende -▁Ausgangspunkt -▁zweifellos -reuung -uv -wa -▁außen -▁weshalb -▁SO -▁Malt -▁A4 -onn -▁läng -▁Rub -▁inzwischen -iessen -schule -▁stations -essene -▁Jack -▁proud -▁Erinner -cco -▁comply -▁trial -ändige -▁substances -▁1991 -▁Atom -▁Hindern -▁Show -▁articles -▁objects -bauen -▁kombin -▁keeping -▁supplement -▁Paper -▁resolve -▁Update -▁Suite -▁Temperatur -▁Konfli -▁niemals -▁ausgezeichnete -▁Verteidigungs -▁Beratung -▁Agrarpolitik -▁natürlichen -▁trees -▁appreciate -▁GNU -▁Wirklichkeit -▁sleep -fangen -CC -▁tests -▁dog -▁dict -▁dans -▁dinner -▁vess -▁Pia -inos -isp -▁Kandidaten -▁Var -▁Ga -ingesch -ichtete -▁Jew -ugnisse -▁schreiben -uds -▁arms -▁49 -▁sav -▁Nachrichten -▁reden -punkte -▁operators -klärung -▁Beispiele -ettbewerbs -▁favor -▁installiert -▁taxes -▁entered -▁1989 -▁Entwicklungsländern -▁soft -▁neuesten -▁interessant -stehenden -▁Papier -▁emphasise -▁advantages -▁ruhigen -▁engineering -▁ausgezeichneten -▁Eisenbahn -▁refugees -rüstung -▁Verbesserungen -oßen -dessen -pel -path -75 -▁fill -▁si -▁vast -▁Sens -▁Sov -▁Massen -eds -inations -usal -▁zut -▁Rich -▁Rule -üren -▁einfachen -▁whilst -▁whatever -wechsel -flug -inken -▁obst -▁laden -abling -▁7. -handlungen -▁Ministers -▁99 -▁delegation -▁estim -▁verschieden -▁approval -▁stattf -▁Safety -▁Sauna -▁offenen -▁Offen -spruchs -▁Vorsitz -▁Alternative -▁bemüht -▁applicable -▁temperature -▁centres -▁Kohäsions -▁Geträn -▁texts -nummer -uden -▁ta -▁fühlen -▁dining -▁Sou -▁paint -▁Puert -eauc -▁latter -▁Fire -iefe -▁Nein -opolit -▁einerseits -ungsbereich -etzten -formance -▁Einstellung -▁politicians -▁Überzeugung -▁finding -▁Exp -▁protest -▁helped -▁personally -▁Blut -▁Music -▁beträcht -▁Robert --Re -▁respective --Date -▁17. -▁involving -▁affairs -▁Strategy -▁anti -▁naturally -▁Normen -▁Kapaz -▁Zusatz -▁Bauern -▁happens -▁cross-border -▁ursprünglichen -▁arrangements -▁kurze -▁entscheidend -▁entertainment -▁älter -rd -maschinen -cean -▁sorgt -▁mel -▁Sy -▁Sym -▁peoples -▁Mall -▁Burg -▁Access -utter -▁Work -▁Rock -▁Govern -▁intend -▁tours -▁alarm -achte -▁verändern -▁Ungar -agon -plans -prof -▁spanischen -▁covers -▁disaster -gesundheit -ändler -▁gelegt -▁Klar -▁15. -▁kommer -▁organisiert -ächtigen -uiten -▁exclusively -▁spring -▁util -▁danger -▁Jedes -▁suffering -▁grundlegende -▁präsentiert -▁menschlichen -▁betreffenden -▁spectac -▁recognised -▁valuable -material --out -DA -19 -▁Dauer -▁Eis -▁Bucht -erten -urger -▁Kroat -▁KMU -▁Vision -ares -▁forg -▁anwes -▁stimmt -▁starten -uropäischen -▁OF -▁Stahl -antly -izit -▁spot -▁leisure -örung -▁diseases -chtigkeit -▁solve -▁gewalt -▁900 -▁08 -ählte -▁detaill -▁creative -▁musste -liefer -▁richten -legungen -weisung -▁hostels -▁höchsten -areness -▁funktionieren -▁kulturelle -▁museum -▁abgelehnt -▁ausgesprochen -▁outdoor -▁journey -▁Swedish -uens -zin -jo -▁a. -▁achten -▁mobility -▁SV -erin -isory -oli -▁Vir -▁beha -▁Like -idden -▁stone -uchte -▁aufzunehmen -▁gezogen -▁Verd -▁shift -▁Ausstattung -ailing -ysteme -▁George -▁trifft -▁Reisen -▁soweit -▁96 -▁07 -▁fulf -▁fulfil -▁beraten -▁larg -▁Franc -▁12. -▁Steuern -▁looks -▁Angst -▁2.0 -▁Britain -onomic -▁opposed -▁confirmed -▁u.a. -▁Halb -▁Nahen -meldung -kontrolle -▁Dusche -wachstum -täus -nungen -mechan -kund -▁dritte -▁au -▁vous -▁weather -▁Sad -▁pick -▁AT -eta -▁ungefähr -enter -schirm -lege -▁erb -rohung -ikes -▁umzusetzen -▁Erwachs -▁arg -reichung -▁conserv -▁subst -▁Brasil -▁Geschmack -▁10. -▁09 -geschäft -▁privileg -▁aktuell -▁familiar -▁Washing -▁Farb -▁Strukturen -▁Director --the -▁Optim -▁übernimmt -▁Engine -▁Veranstaltung -▁obtained -▁discovered -▁renewable -▁threaten -▁riesigen -▁Harmonisierung -▁gratulieren -▁intellectual --I -pal -18 -▁dich -▁Sund -▁Esp -ires -▁zählt -▁korr -▁Ka -▁Four -▁Glaub -▁GDP -ati -▁thorough -▁anstatt -▁erlauben -▁residence -▁cloth -▁durchzuführen -▁EU-Mitglied -▁trib -▁offices -▁gewonnen -aktiv -▁directives -▁qualifiz -▁Bestellung -▁verschiedener -▁bekämpfen -▁faced -▁Box -▁Tagung -▁combating -▁UNO -▁Secr -▁Küsten -▁automatic -gänge -▁determine -▁Termin -▁passengers -▁künftigen -aßt -▁effektiv -▁Berufs -▁ignor -▁verursacht -▁Proced -▁largely -iability -far -da -üste --5 -▁buchen -▁girl -erte -itts -oria -chron -▁Hof -▁GU -ulich -▁fordere -▁veran -▁vertr -pth -amente -▁Arme -▁Abschnitt -flächen -▁serves -▁Internets -▁gelöst -▁beschränkt -eichert -▁verlangt -▁Punkten -▁involvement -▁Kommun -▁Investitions -▁temporary -▁Farbe -▁protecting -umping -▁Stabilitäts -▁Hohen -▁Kommunikations -▁Übereinstimmung -▁figure -▁occasion -▁constructive -▁millions -▁lovely -inuier -▁impressive -▁enthus -▁Rumänien -▁obstac -▁¤ -vi -EG -A. -▁Szen -▁AS -▁Ath -institution -iefert -▁Ladies -▁Noch -ktor -▁Schü -izz -ipel -▁schafft -ecess -▁amounts -▁Ausführungen -▁Armen -actory -▁mögliche -ilding -▁Gegend -handelt -▁Brand -▁leaving -azine -▁flow -▁acquis --Ab --Adresse -▁beseit --sc -▁increases -▁healthy --Europ -▁treaty -▁Instead -▁Änderungsanträgen -▁bureauc -▁officials -▁uneingesch -▁lautet -▁eyes -▁removed -▁couple -▁Krankheiten -▁Hostelsclub -▁Luxembourg -hi -hle -cap -läuf -▁rank -▁Town -▁AV -ito -uster -▁Wähl -iefen -▁Luc -▁sehe -▁auftreten -grenz -▁chain -▁reserve -▁Spaz -wood -wendungen -▁Marina -▁Übersetzung -eatre -▁firstly -▁Vorschlägen -kehren -▁equally -▁individuell -▁strategies -▁beabsicht -▁Geräte -▁schnelle -▁Behinder -▁annehmen -▁president -▁intent -▁Entscheidungs -▁Feuer -▁Festlegung -▁angesehen -▁Tatsächlich --time -▁happening -▁zentralen -▁Gestaltung -▁ordnungs -▁erheblich -▁wissenschaftlichen -▁breite -▁Gentoo -hard -nischen -krit -▁hit -▁seat -▁od -▁WLAN -▁Vern -▁Vater -ani -ented -▁impression -▁stake -orte -apest -▁verwend -▁verständ -▁mitten -▁einig -▁ausgeschlossen -ätigung -▁Alp -▁newly -echnologie -meister -ergie -▁insist -wirkung -▁handling -▁Executive -▁Problems -▁ausgezeichnet -▁schwed -▁communications -▁contributions -▁hervorheben -▁aufgefordert -▁extraordin -▁preparation -▁Drittländern -▁kurzen -▁Garantie -lastung -▁mandate -▁leistungs -▁organizations -▁Zuständigkeit -▁Croatia -▁balcony -▁Kilometer -rate -aal -seite -mung -kultur -CE -▁münd -▁BSE -eries -ilen -▁catch -▁Wit -▁Himm -▁bezug -▁Late -trans -▁Their -▁Verring -▁Thomas -▁Schatten -▁Einheit -▁Child -▁Imp -▁incent -▁suite -▁requirement -▁awareness -▁doesn -▁boat -▁charges -▁bekomm -▁Boot -▁Deutsch -▁Straß -▁Entwicklungsländer -▁classic -▁remaining -▁vereinbar -▁volume -▁zufolge -▁universal -▁nachhaltigen -▁Balkan -▁license -modell -▁Fähigkeiten -▁zugunsten -▁beglückwünschen -▁discharge -▁Washington -tag -zipl -16 -▁ru -▁avec -▁wettbewerbs -▁Sicher -ofin -▁catalog -▁Riv -▁View -ask -uführen -▁derart -andal -▁abroad -ochen -▁Wird -sprech -▁durchführen -▁Regime -▁Umfeld -▁root -werb -▁Frid -▁decis -ypern -▁verbringen -▁monument -▁Aktien -▁Emp -▁Meinungs -astricht -▁advoc -▁Sonne -▁Canada -▁Japanese -▁zuständigen -▁fighting -▁surroundings -▁collective -▁Ratspräsidentschaft -▁restrictions -▁susp -▁liberalisation -▁früheren --Un -▁kontinuier -▁Professional -▁sodass -ris -32 -▁dark -▁Miet -▁gold -▁Block -▁AM -inational -▁cab -▁Cond -▁unent -decken -▁Life -ibeh -▁prove -▁entstand -▁Ausstellung -▁Kontext -▁Geheim -▁Regen -etty -▁Maastricht -führte -▁Alles -▁issued -▁debates -minuten -▁creates -▁operational -ziele -▁16. -mitten -ENT -▁rapporteurs -▁pattern -▁Gleichgewicht -▁opinions -▁parlament -▁characteristics -▁depends -▁Camping -▁bestätigen -▁renovated -▁Bulgaria -tedly -▁Automobil -▁ehrgeiz -▁präz -▁verabschiedet -▁inhabit -EF -pers -▁fasc -▁ball -▁Merc -orial -▁Word -▁album -▁Na -▁Nap -stood -orten -▁Ziffer -abwe -▁Verfahrens -▁Stift -▁Anton -▁prohib -ubs -▁ausged -▁coher -▁Ausführung -actions -▁sauna -▁64 -▁Nachricht -standards -▁Brenn -ältigen -▁param -▁Arbeitsplätzen -holen -züge -▁notwendige -▁identify -▁returned -▁volunt -▁messages -▁contracts -▁maintained -▁Websites -▁resolutions -▁Übereinkommen -▁Prüf -▁d.h. -▁decorated -▁Drittens -▁disappoint -▁Plenum -▁Through -do -bart -lights -liste -▁hängt -▁nob -▁walls -▁phen -▁push -▁gets -▁Adr -▁Award -eler -igend -oni -▁Fa -▁Rent -asche -▁derartige -▁anspruchs -▁Infl -▁Verg -▁abol -etzte -ateful -▁legend -ystems -▁Year -▁Entfern -▁fee -▁Gewicht -▁Quark -▁Deutschen -▁Redner -▁interessante -▁Ideal -▁intensive -▁einzigartige -▁freiwill -▁Detail -▁sexual -▁Zentralbank -schaffen -▁produziert -▁faith -▁bezahlen -▁Hilfs -▁Schlafzimmer -▁traditions -▁Subsid -banken --min -UT -GB -▁dream -▁ER -▁Mold -iser -orca -▁cable -▁CN -anen -▁Lt -▁Lu -andon -▁ermut -forschung -▁Anbieter -▁youth -▁übersch -udes -▁serviert -▁gewissen -ittees -▁Verbot -▁8. -istry -▁Indones -▁Arbeitsplatz -▁Partnern -ployed -▁artist -▁öffentlich -▁entwickelten -▁Speicher -▁precise -▁hardware -▁Buchung -▁verschw -▁ausgestatt -▁intensiv -▁verhindert -▁Legal -▁Regierungskonferenz -märkte -▁owner -▁Erholung -beauftrag -▁MEPs -▁geprüft -▁iron -uite -macht -äder -▁sin -▁swe -▁Trag -▁DA -▁Earth -▁Polit -▁PRO -▁Agricult -orama -▁lin -▁Wolf -▁FA -▁Viet -ici -iera -▁angeb -▁aufnehmen -▁verstanden -▁verleihen -änen -▁shad -▁Schuld -▁ausgesetzt -▁Ersatz -▁genet -▁gender -▁Max -▁reported -▁Polish -tendlich -▁involve -▁transportation -▁kurzfrist -▁Planung -▁reserves -▁österreich -▁Anti -▁konkret -▁copyright -▁conflicts -▁regularly -▁Prinzipien -▁kontakt -▁aftern -▁Procedure -▁Verringerung -pot -gä -én -▁tä -▁nov -▁mur -▁meat -▁battle -▁gone -▁Fair -▁Vice -▁unmöglich -arium -▁string -▁witness -▁geä -▁jetz -ects -▁Ankunft -▁Ansprü -▁abgestimmt -ighten -▁entsteht -▁Ausgabe -schluß -cles -▁bestimmen -örd -▁recip -▁welchem -▁Är -▁Krie -▁Congress -▁parents -ategie -▁visa -▁Road -▁passiert -▁Geschäfte -▁direkten -code -▁recommendation -usstsein -▁drawing -produkte -▁properties -▁Besonders -▁facilitate -▁erreichbar -▁Taiwan -mat -miss -car -▁dates -▁MB -erkennen -▁kämpfen -▁Kamera -▁Ferr -▁Food -▁Vo -steh -stieg -ichteten -▁Abhäng -▁supposed -▁46 -▁submit -▁legt -fined -weite -▁02 -▁geräum -▁App -▁customs -▁reveal -▁Gerichte -▁Beiträge -▁definiert -▁topic -▁anti- -▁Sony -▁Freiheiten -▁Verhandlungs -▁vollständige -kohol -▁westlichen -▁northern -▁wollten -▁conducted -▁Faktoren -▁Verteidigung -▁Haftung -▁Bemerkungen -onymous -▁hundreds -▁accompan -▁Wünsche -▁sorgfältig -▁somewhat -▁wann -▁Semin -▁gift -▁Bibliothe -▁echten -▁Hot -▁Freit -▁Gene -▁thumb -▁NE -ucher -▁Jer -▁worse -ausend -▁amaz -▁Einbeziehung -▁interior -▁Struct -▁Verbindungen -▁Deswegen -▁Aktual -▁ausgeführt -▁Training -▁geboren -cohol --En -▁define -▁metal -▁performed -▁begrüßt -CHT -▁Beziehung -▁Konsens -▁terrorist -würdigkeit -▁tatsäch -▁Führer -▁parliaments -▁Persön -▁Kreuz -▁depending -▁Jede -▁Schweizer -▁displayed -▁staatlichen -▁updated -▁vacation -▁kostet -▁Nahrungsmittel -▁erleichtern -ooth -las -▁faster -▁Sorgen -▁Dynam -▁PS -▁Videos -▁unfor -berge -▁Nic -schäd -acher -▁gezwungen -▁ju -▁James -▁Anmer -einen -rennen -▁consc -▁Asien -▁seeing -zeichnis -▁Neue -▁180 -▁Format -▁Vorschlags -▁Projekten -▁deals -▁verwand -ikaner -▁career -▁answers -▁towns -▁decision-making -▁settings -▁regulatory -▁holding -▁joined -▁geltenden -▁connections -▁höheren -▁Budapest -▁hinzufügen -▁Werbung -knüp -▁Cyprus -▁stylish -▁Ungarn -▁phenomen -▁Subsidiar -hone -uese -OW -▁dro -▁arab -▁whe -▁Dass -enze -ergeb -oma -▁ought -▁Ware -▁Wesent -▁Fleisch -▁Read -▁islands -ivals -▁Zypern -osis -▁Europol -▁abandon -izza -weile -▁Gründung -▁solange -▁Orte -▁mostly -▁techniques -▁driver -▁pharm -▁follows -▁gebaut -▁Adobe -▁13. -▁multinational -▁Protection -▁nahezu -▁Balkon -▁Vereinbarungen -▁minority -▁frequently -▁Tarif -▁emergency -▁chinesischen -▁verehrte -▁gespeichert -▁Kompetenz -▁Polizei -▁SMEs -▁Spazier -har -afe -mir -maschine -86 -ère -▁tail -▁SA -▁EM -▁BIP -orie -ometr -▁unge -icon -stat -▁noted -▁proof -▁vorlegen -▁worry -▁grateful -▁liqu -ammeln -ventionen -▁Individ -▁Genau -▁continent -▁beschleun -▁Monetary -undert -▁Shop -uzierung -▁Bildungs -▁Betracht --TV -örderung -▁organized -▁label -▁Fernseh -▁demanding -▁windows -▁heutige -▁angemessene -▁Gipf -▁bemühen -▁beeindruck -▁Gemeinde -▁verringern -▁ferner -▁Einwohner -▁ultimately -häuser -ieller -sc -17 -▁bele -▁IV -▁SC -▁Symbol -▁Dipl -erre -itra -itals -▁cash -▁Wild -▁Holid -▁UR -ingly -▁Vermögen -▁exklus -▁ausreichen -ochter -ätigkeiten -▁trop -▁Plug -▁Dev -▁invite -▁Bedien -▁Exc -▁preis -▁suites -▁complement -▁Interess -▁visual -▁integriert -▁Gesellschaften -▁hostel -▁Außenpolitik -ausschuß -▁bilater -▁erinnert -▁Audio -▁castle -▁ethnic -▁Mexico -▁Mobilität -▁electricity -▁bottom -▁delighted -▁divided -▁Veröffentlichung -kreis -gut -▁wines -▁EZ -▁Mem -▁Ball -eachtet -unning -▁ll -▁library -icke -stall -▁refr -ibar -▁überras -earing -▁Vorgehen -▁producing -iox -▁contrast -▁9. -▁drug -▁Whether -▁03 -▁Bars -▁visible -▁Blue -▁Partners -▁statistics -▁arrive -▁contacts -▁bestehende -mäßigen -▁relaxation -▁1970 -▁Hostel -▁verdient -▁Provinz -▁Kriegs -▁übrigens -▁Sollten -▁Gespräche -▁spezifischen -▁Identität -▁jegliche -▁fertig -▁Slovak -▁behavi -▁quarter -▁Gepäck -vol -wald -▁ticket -▁sah -▁migration -▁Terra -▁IB -▁Edition -▁Beam -element -iro -▁Cat -▁Games -▁Golden -arische -stream -▁stars -▁2, -▁Anschluss -▁entrance -schafts -▁nachzu -▁Yes -ruh -▁heritage -▁welfare -▁locations -▁verbind -▁expansion -äßt -▁courage -▁Author -▁Automat -▁starts -▁investors -▁Palästinens -▁Gerät -▁tourists -▁Bevölker -▁Frühstücksb -▁Webseiten -begr -▁palästinens -▁pieces -üllen -▁channels -▁Prague -▁letztlich -spartner -▁magnific -uous -▁vent -▁Equ -▁Funds -▁Ret -▁Viertel -▁GB -▁behalten -ommun -▁neutr -isses -agg -plement -▁'ve -▁Rezeption -werks -▁Image -▁2012 -▁Framework -▁Browser -▁protocol -▁beliebt -▁valley -▁presents --Ber -▁einfacher -▁Besorgnis -▁exciting -▁involves -▁visiting -▁careful -▁Zwei -mäßige -▁Canon -▁Insbesondere -▁Selbstverständlich -▁gegebenen -▁stored -▁widely -lässlich -▁Sollte -▁pollution -▁verurteilt -▁declaration -▁Putin -▁grundsätzlich -▁berechtigt -▁ordnungsgemäß -▁inhabitants -▁Vietnam -aben -zne -passen -gs -▁tea -▁narr -▁Sn -▁Mangel -▁Bö -▁Wart -▁Fun -▁unw -icans -▁Liter -schm -▁Oder -▁oral -▁showing -▁unterst -▁lebens -blieben -▁remote -▁drivers -▁verbinden -▁flights -▁letzter -▁Preisen -▁Medik -▁Respekt --Fi -▁nunmehr -▁explore -▁index -▁Beweg -ifizieren -▁gewährt -▁planet -systeme -▁Verlust -▁Optionen -▁bilateral -▁Portuguese -▁gardens -▁subsidies -▁Untersuchungen -▁Schnitt -▁einheitlichen -▁Einwanderung -▁cosy -▁afternoon -sieht -west -▁sac -▁Sat -iled -othe -▁input -iones -ulf -ulpt -▁ster -▁ersetzt -upl -▁entschlossen -ogether -ignet -hlt -ammt -▁Gefahren -▁indicated -▁gespielt -▁proved -▁understood -aktionen -klass -▁Verbreitung -▁Büro -▁Zielen --Mod -▁visited -▁privat -▁individuelle -▁Marktes -▁organisieren -▁direkte -▁installieren -▁rechtlichen -▁Moz -▁Kapitel -▁focused -▁maintaining -▁Legisl -▁virtual -▁beschäftigen -▁Copenhagen -▁wunderschön -▁Komponenten -▁collaboration -▁Baltic -▁Friday -ties -nitten -▁heating -▁fitness -▁m2 -▁Ib -▁Sar -▁Situ -▁Pact -▁Bre -emm -embl -▁lounge -▁king -▁Roll -imedia -▁Naz -osit -▁neun -izer -▁vorstellen -immungs -▁accident -▁newsp -▁47 -▁Haar -▁gestaltet -▁awa -▁signs --Ar -holm -▁Versicher -▁Email -▁Energien -▁individuellen -▁Phot -▁Zwecke -▁erfolgreiche -▁entscheidende -▁Natural -▁cutting -▁demonstrate -▁demonstrated -▁Michel -▁vorhandenen -schaffung -beziehungen -▁attrakt -▁Question -▁Autobahn -hängen -uing -uers -miet -EE -kan -ören -▁rational -▁mat -organ -▁Vict -▁Vize -▁inklusive -▁Lager -▁Nos -▁jenen -agers -▁proc -▁vorzu -bekämpfung -▁Vorstellung -acks -▁finished -▁Project -▁700 -▁06 -▁Freunden -▁facility -▁Politiken -▁14. -▁popul -▁dürfte -▁raw -▁Serie -▁unverzüglich -▁inflation -▁Fehl -▁Donner -▁britische -▁department -▁ursprünglich -▁Network -▁erläut -▁Bemerkung -▁beruht -▁Alexander -▁Kräfte -▁außerordentlich -▁spoken -▁accompanied --sp -ôtel -▁fuer -▁SL -▁Bug -igar -otive -onien -urlaub -▁categories -▁israel -ativer -ulin -▁bekräft -▁ongoing -ragt -▁era -iforn -▁wechs -ibe -wege -▁ausgeb -▁aussprechen -ación -▁Spaß -▁plenty -▁aggress -▁theme -gelassen -▁85 -▁carbon -▁0, -▁printing -▁zusammenarbeiten -▁stehenden -opped -▁schlägt -▁settlement -▁Perhaps -▁bemer -▁restruct -▁fehlt -▁high-quality -▁Städten -▁reduzieren -▁controlled -▁luxuri -▁AKP -▁dahin -▁spectacular -▁§ -ois -ozi -zehn -DR -pap -kern -▁hoff -▁Meg -▁Page -▁Auge -erie -▁commission -▁ließ -▁kamen -▁Kais -ami -gegangen -▁Niger -lehnung -▁Institut -▁Zert -hrten -▁exception -▁Ausb -▁Ausmaß -▁Alpen -▁transatlant -▁complain -▁smooth -▁personnel -▁effectiveness -▁Betrag -iegelt -▁Werke -▁distingu -▁klaren -▁documentation -▁Mitarbeitern -IDS -▁ruhig -▁verstärken -▁Erzeugnisse -▁verkaufen -▁angegebenen -▁Schönheit -▁Rechnungshof -▁eingereicht -▁BMW -▁definitely -▁в -pa -▁tickets -▁sad -▁Dal -▁gre -enary -imens -rak -▁Zür -upten -▁concluded -▁Verpack -ector -▁Sche -▁Abkommens -periode -▁committees -▁52 -▁firms -ührt -▁großzüg -aktion -▁possibly -▁Indian -▁angewand -▁explicit -▁foundation -ässig -▁marquis -▁educational -▁Israeli -▁Konsult -▁Northern -center -▁Kopenhagen -▁Amendments -▁Wohlstand -▁Depart -▁awarded -▁Salzburg -▁Sektoren -▁replaced -▁russischen -▁resistance -▁HTML -▁contrad -▁horizont -▁Malta -▁Puerto -gess -▁ve -▁phrase -▁gallery -▁Women -▁unt -ulen -▁forum -▁Luf -▁Lad -▁Nob -hereas -▁Verlauf -▁jener -▁Stimmen -▁usual -▁displ -▁Programmen -ruckt -▁crimes -▁Description -▁boot -▁boost -▁Rechtsstaat -▁citizen -▁Gericht -wertung -twortet -▁Ski -▁speaker -▁Parkplatz -▁erfolgreichen -▁Kirk -zentren -▁corruption -▁2013 -▁recorded -▁passenger -▁Männern -▁Belgium -▁minorities -▁Applause -▁erhielt -▁occasions -▁rapidly -▁wiederholen -▁Single -▁ergänzen -▁äußern -▁behaviour -▁· -zur -wählen --8 -▁TO -▁Tochter -▁SAP -▁Pen -▁Kab -▁Gere -▁stil -vertrag -▁vermitteln -▁confer -iences -ustellen -ughter -▁vieles -▁Charta -▁Leon -▁els -▁gewöhn -▁autonom -wirkt -▁Kriminal -▁Desk -▁profile -▁termin -▁competent --DE -▁bestimmter -▁Kontrollen -▁Schlussfolgerungen -▁Gesetzes -▁sugar -▁Schulen -▁Educ -▁Zusätzlich -▁Lieferung -▁trafficking -▁erhebliche -▁manufacturer -▁Bereitstellung -▁tackle -▁vorbereitet -▁Wi-Fi --Strategie -▁Gaza -▁compatible -▁configuration -ni -dauer -ror -zonen -wh -95 -▁nights -▁sla -▁woman -▁Satz -▁yes -▁U-Bahn -▁desired -▁Staff -ellers -▁exit -▁Einstellungen -hlen -▁proven -▁overcome -schefs --Sta -▁respected -▁assets -▁assume -▁verbesser -kehrt -▁Client -▁strategischen -▁Urheber -▁immigrants -▁traditionelle -▁enjoyed -▁arrest -▁Gegenstand -▁empfangen -▁Sozialdem -kommunik -onomie -▁containing -▁Chinas -▁routes -▁cycle -▁Newsletter -▁nahm -▁stimul --Sterne-Hotel -▁beigetragen -▁Soviet --European -waffen -WF -BS -▁rub -▁Dre -▁CF -▁Given -▁Güter -▁Leh -▁imposed -idet -▁theoret -▁constitution -izieren -▁scheme -▁voran -ewhere -▁partly -▁Arzne -▁Chair -geteilt -▁großem -▁Übersetz -demark -▁facts -▁brut -▁kleines -▁Rechtsgrundlage -▁Philosoph -▁films -▁princip -▁arrived -ography -▁eingehalten -▁sending -▁correctly -▁lokale -▁replace -▁amerikanische -▁Krankheit -▁credibility -▁luxurious -▁wunderschönen -▁Ernähr -uminium -fuhr -▁unfair -▁PPE-DE -▁uncertain -▁Genehmigung -oes -▁sit -▁mun -▁Tip -▁Mobile -iga -▁lobby -▁alcohol -▁prest -▁erzeugt -osa -▁constitutional -▁jung -▁Job -ferenzen -▁coal -udi -▁manager -sph -▁Konkur -▁bislang -▁Plat -reift -▁transactions -▁stronger -▁Bedrohung -▁Conc -▁centuries -▁langsam -▁Versand -▁Traum -▁Belast -▁maritime -gänger -▁Sonnt -▁professionals -▁Milch -▁verantwortungs -▁aktivieren -▁einzigart -▁Hamas -▁beteiligen -▁conditioning -▁Hungary -▁McC -▁appointed -▁emerging -▁corresponding -▁subsidiarity -▁anwesend --und -ées -ín -▁afr -▁sud -▁Sav -▁Sicherung -elated -inem -orben -▁lots -▁uniform -aser -idem -▁stolz -▁Thurs -▁vorrang -▁Erstellung -▁adults -gerufen -▁grün -▁Konstrukt -steigen -▁bedauer -▁Nachhalt -▁minister -▁Äg -▁Amt -▁concert -▁Schluß -▁config -▁Works --Gen -regeln -▁diagn -▁unvergess -strafe -▁weekend -▁Verbraucherschutz -▁distributed -▁Werkzeug -▁satisfied -▁einzigartigen -prozesses -▁certific -▁elektronischen -▁acknowled -▁beobachten -▁Following -▁Within --n -▁nutz -▁Sind -▁gearbeitet -▁Aqu -irs -igert -▁Wied -▁Kabel -▁VAT -iva -atures -anw -gefügt -▁Lok -level -ernt -real -▁abziel -ardo -▁Kongress -▁Orient -▁liefert -▁verbindet -hips -athem -▁prompt -▁globalisation -▁komme -▁adopting -▁Charles -▁entscheidender -▁declared -▁functioning -▁broadc -▁judicial -▁französische -▁Bulgarien -▁betreffend -▁verkauft -▁gekenn -▁Dolom -▁ausgerichtet -paket -▁slightly -▁destruction -▁Glaubwürdigkeit -▁Freitag -▁Structural -▁magnificent -▁ice -dienste --Q -▁hart -▁fem -▁died -▁dign -▁rental -▁miles -itim -irtschaft -▁Hen -▁inj -▁incr -anke -▁toil -▁anbel -rain -apie -▁Schon -▁auseinander -▁schicken -▁EU-B -▁genannte -rating -echnik -▁solches -werken -▁Sohn -etheless -yside -▁Staates -▁bright -▁Apple -▁courts -▁Darum -▁bereitgestellt -▁Investment -▁Ansichten -gehens -▁Flugzeug -▁Wohnung -▁suppliers -▁Balkans -▁shares -▁offiziellen -▁Kaffee -▁Konsequenzen -▁Kroatien -▁extraordinary -oub -down -ueller -65 -ás -▁Satur -empts -informationen -▁lists -▁Wäl -▁Water -alte -osion -osing -▁geziel -▁Sterne -▁Anlass -▁plays -geräte -nex -▁Überlegungen -▁Allian -▁Extrem -▁Exchange -▁Geschwindigkeit -▁surf -▁considering -▁zurückzu -▁sozialer -▁didn -▁sensible -▁Fahrt -▁taxation -▁addresses -ivalent -▁identified -▁flood --Projekt -▁zuständig -▁Montag -▁Fußball -▁suspend -▁wiederholt -▁Professor -UNG -▁eastern -▁micro -lossenen -▁Sunday -▁anbelangt -IX -35 -gor -▁Mission -erts -▁Cup -▁Hat -▁Vel -▁Gang -icing -▁stammen -▁erm -▁verste -▁weist -▁Stutt -▁plenary -ologien -▁consent -▁Import -attung -achtet -▁konfront -mittels -▁proposes -▁drop -▁Who -▁carr -▁formed -▁Add -▁Festpl -▁Kollegin -▁Ministerpräsident -▁ökonom -▁encouraging -▁Fundamental -▁Anstieg -▁honest -▁Content -▁Beschlüsse -▁beantworten -▁ähnlich -▁Domin -▁preparing -▁Zivilgesellschaft -▁milk -▁providers -▁Gewährleistung -▁genügend -▁Sudan -ziplin -▁п -.. -rufen -suchen -bach -ökonom -gart -gemeinschaft -▁nörd -▁EF -▁Pil -emark -ofern -urteilung -▁Wetter -▁inmitten -derung -asing -▁angest -stes -verän -ikationen -▁voranz -ounter -ärkt -gerung -▁Mrd. -▁Konfigur -▁Mitgliedschaft -▁suit -▁ensures -▁letztendlich -▁entspannen -▁Without --Vers -▁Normal -▁domain -▁Zusch -▁Positionen -▁resolved -▁herrscht -▁überprüft -▁Eigentum -▁disappear -▁Potenzial -▁mittleren -▁Thirdly -▁Valley -▁substance -▁Schüler -eist -PC -gearbeitet -ju -▁della -▁amen -▁TR -▁ped -empl -oration -▁ou -▁Wählen -▁Rab -▁Vene -▁Nam -▁confront -prene -▁Thail -▁Erfüllung -▁weitgehend -▁Made -▁Übrigen -▁Mini -▁Herbst -▁wohnen -▁everyday -▁pre- -omenade -▁mussten -▁artistic -▁Quellen -▁actively -▁erstmals -▁entwickelte -bildungs -▁eventually -▁machte -weilen -▁Assembly -äusern -▁roads -▁ernsthaft -▁army -▁component -▁drinks -▁updates -betrieb -▁begrenzt -▁verteidigen -▁Gerechtigkeit -NA -▁breiten -▁Pfer -irk -▁innen -▁toba -▁erz -uchs -▁verwe -▁concepts -▁japan -▁Stoffe -▁allzu -merk -enne -oche -auschen -▁betreiben -▁54 -▁inclusion -▁assure -▁deleg -▁glück -▁Next -▁Beschwer -orry -▁aktive -▁Laut -▁Gerade -▁topics -ographic -▁Kulturen -▁potenz -gebiete -▁western -effizienz -▁thousand -▁Gehminuten -▁division -▁enhance -▁reduziert -▁MP3 -▁Verzö -▁deposit -▁euros -▁Unternehmer -▁erlebt -▁Brasilien -▁Holiday -▁Arzneim -sektor -cal -ye -UL -▁vary -▁Tri -▁Treib -etes -eaus -otic -orb -▁kreat -▁CR -▁Hom -derten -▁stellte -▁gehe -▁Still -endo -▁exch -▁spoke -▁upper -▁proportion -folgung -icted -▁Klasse -▁personen -rechter -▁Travel -▁betrieben -▁musical -▁investieren -▁wirtschaftlicher -▁erstre -▁Reduzierung -▁Kühl -wahlen -▁Verletzung -▁supplies -▁university -▁Passwort -▁Eingang -▁duties -▁ACP -uggage -▁korrekt -▁decisive -▁Vizepräsident -▁elsewhere -▁Thursday -.C -MA -▁tritt -▁sen -▁Tyr -▁Tennis -▁blie -ede -eding -eak -itale -▁Rio -reche -stud -andt -▁OR -▁könnt -▁Vorhaben -▁warrant -▁solely -▁hierbei -▁Entlastung -▁proposing -▁handle -▁Meister -▁Among -ordnungen -garten -hältnisse -▁deutsch -▁absence -▁deficit -▁lessons -▁subjects -▁generated -▁hervorra -▁optimale -▁applying -▁Kontakte -▁info -▁Belgien -▁Federation -ademy -▁Castle -LAF -▁Korruption -▁angepasst -▁vulnerable -▁bekommt -▁Ägyp -dorf -ckte -CP -▁fab -▁pe -▁Mur -▁gear -▁AIDS -▁lit -▁Klaus -▁Coll -opes -▁theory -ifies -▁Jung -▁shock -▁shipping -▁Einkommen -▁unterscheiden -▁Mr. -▁Cho -▁countryside -▁hinweg -▁gelingt -▁versichern -▁führende -enende -▁Finanzierungs -▁Schwimm -▁Quality -▁packaging -▁appeared -▁nevertheless -▁suggestions -▁erwähnen -▁carrying -▁liked -ügigkeit -▁zukünftigen -▁Trotzdem -▁Mountain -▁Übergangs -▁Katastrophe -▁Hardware -▁Monday -▁Praktiken -▁voluntary --minute -▁Donnerstag -while --day -▁semin -▁pure -edy -alts -▁Lobby -rein -lech -ktions -▁ersetzen -▁einbezogen -▁description -elli -▁Schiffe -ipfel -ranche -▁manche -▁durchschnitt -▁Grünen -werben -▁Marke -▁Stras -▁Common -▁gestattet -▁greatly -▁Besides -▁XP -▁impose -▁Saison -ski -▁fourth -▁institutionellen -▁grav -▁villages -▁Risik -ommerce -▁Luxus -▁Funktions -▁wrote -▁Latein -▁entdeckt -▁departure -▁legitimate -▁Playa -▁iPhone -helm -▁einzusetzen -▁excessive -▁kontaktieren -▁tobacco -niveau -uits -bü -bre --- -▁MA -▁gained -▁Astr -orer -▁ang -▁anschließend -▁ansprechen -▁beendet -aya -▁verteilt -▁Präsent -▁desk -▁deswegen -iki -iensten -▁abuse -▁ships -▁Bege -▁finish -▁Konto -▁41 -▁Prozesse -▁Orch -▁mini -▁getragen -▁simult -ixel -▁Addition -▁bedeutende -▁electr -▁non-s -weisungen -▁repair -▁teams -▁illegalen -▁Consult -▁verfügbaren -▁freundlich -▁Hungarian -▁channel -▁philosophy -▁umfassend -▁behaupten -▁unforg -▁unterstreichen -aue -günst -▁ven -▁Tätigkeiten -▁Mov -▁Mak -▁Creat -▁Having -▁Lounge -estra -osure -▁verschl -▁mitteilen -▁Verarbeitung -▁exports -ipher -▁vorzulegen -▁Erlebnis -▁Umsatz -▁Plus -▁Plattform -▁Frist -▁Sev -▁Minist -▁delicious -▁director -geschichte -▁Genießen -▁optional -▁Eurozone -▁Guests -freien -ränkung --Gr -▁offene -behandlung -behörde -▁supplier -alerie -▁Volkswirtschaft -▁housing -▁Captain -▁konzentriert -▁langfristige -▁wissenschaftliche -▁interpretation -▁größeres -athedral -▁Agriculture -▁EZB -▁© -.20 -übt -care -cision -AV -Rooms -▁fu -▁song -▁Ill -▁Sachen -▁Ebenen -▁gemein -▁Cub -▁ingr -istent -▁diesel -ective -ieden -▁sounds -▁vorbeh -▁Erm -▁Ergän -ranten -▁Alger -▁doors -▁Spann -▁Space -▁specified -ätzt -▁regarded -▁receiving -▁teile -▁difficulty -▁Situationen -▁Rückkehr -▁Berliner -▁banking -▁reservations -▁künftige -▁vorgesehenen -▁panoram -▁Erwä -▁Budgets -▁spezifische -▁beeintr -▁comparison -▁Ausflüge -▁konfrontiert -▁Strasbourg -▁om -rist -▁batter -▁IF -▁Dist -eted -utation -▁crew -▁Rh -▁wireless -änglich -▁Stamm -▁manual -▁unterzeichnet -▁Ablehnung -▁trademark -▁Vorgehens -▁disabled -▁roof -führungs -aves -▁overview -▁invited -▁Existenz -▁standing -▁attempts -▁geraten -▁module -▁geboten -bedarf -▁Betten -▁Fahrzeuge -▁colours -▁periods -▁Weiteren -▁Willkommen -▁suggested -▁Erweiter -▁bisherigen -▁Metro -▁satisfaction -▁pursue -▁Gottes -▁Budd -▁Freizügigkeit -▁abstimmen -▁Atlantic -▁Übernachtung -afrika -zepte -▁н -zung -▁tennis -▁fas -▁burn -▁A5-0 -orse -▁cul -▁Ker -▁Ruf -▁Vort -▁forests -idad -▁stream -rog -pläne -endar -annter -▁shap -▁Schle -▁hast -▁schwe -▁Wei -▁unterliegen -▁realistic -▁lieber -▁revision -▁Vereinfach -▁gebrauch -▁edge -▁filter -▁multi -▁1.5 -▁hervorgehoben -ESCO -▁Dres -▁vorgeschlagene -▁Metall -▁Place -▁Niederl -▁Widerspruch -▁well-known -▁welcoming -▁Stockholm -▁herunterladen -▁Perspektive -▁geprägt -▁savings -▁LateRooms -▁Frühstücksbuffet -oz -lings -▁hyp -▁rain -▁rising -▁script -▁SE -▁Suiten -▁Pir -enia -chst -▁Full -iess -enth -▁LT -aga -ugt -ierender -▁Abfall -ammelt -▁53 -▁acting -ventional -öffnung -▁scene -▁Geburt -▁bridge -▁Millenn -▁passion -▁breath -▁Haushaltsplan -▁Based -▁patient -▁resource -▁Wirtschaftswachstum -▁permitted -▁Opposition -staatlichen -▁Planeten -▁reaction -▁examine -▁Gefäng -▁menschliche -▁Tabak -▁arguments -▁hochwertige -▁ergibt -▁unzure -▁murder -▁Situated -rates -ui -zellent -68 -ст -▁Bio -▁Catal -▁Game -▁anhand -▁anschließen -hein -▁rechten -essions -▁abzusch -▁enabling -ußt -ärten -▁leb -immers -▁internen -landen -▁Artik -▁04 -▁leaves -▁Islands -▁ausgehen -▁Train -▁Medizin -▁gefallen -▁Quart -▁retain -▁Urteil -▁representation -▁Family -▁behoben -▁ports -▁finanziert -▁eingegangen -▁anzuzeigen -▁confirmation -▁football -▁Abendessen -▁Anwender -▁zukünftige -▁destroyed -▁federal -▁Bevölkerungs -▁Zürich -▁Stuttgart -▁unforgettable -town -aine -AB -▁tend -▁fan -▁rely -▁Dikt -▁Euch -▁Mig -▁Bett -inf -unruh -▁caf -ulse -▁Prom -▁einsch -▁constitute -▁Oste -▁sichere -▁schlagen -▁vorange -sprachen -▁Abschaffung -ianischen -▁Leiden -male -teilungen -▁directory -▁festival -▁refers -▁blind -▁Hauptbahnhof -▁UNESCO -▁vereinbart -▁farming -▁obliged -▁praktische -▁religion -▁Corporate -▁Kanada -▁Widerstand -▁Veränderung -▁langfristig -▁Nachbarn -▁angenehm -▁umfangreiche -▁cooperate -▁Francis -▁narrow -▁municip -time -▁Miß -▁Por -▁läßt -▁Wissens -▁Rap -▁Vide -▁ye -▁Li -herstellung -▁jahr -eitigung -agements -▁Statt -gram -economic -▁grosse -▁Vorlage -▁etabl -▁Pläne -▁Finland -▁discipl -▁publication -▁politically -okratischen -▁gebeten -▁30. -oppelt -▁paying -poration -▁Volkes -bereichen -ESS -▁Tourism -useum -programms -▁enthaltenen -▁twenty -▁dankbar -▁präsentieren -▁welcomed -▁Kommissionspräsident -▁militärischen -▁sogenannten -▁instructions -▁prosperity -▁Abhängigkeit -▁Nachhaltigkeit -▁Allianz -▁personenbezogen -uben -log -kom -▁rare -▁Dritten -▁Mob -▁Pension -▁PSE -▁zivil -▁killed -▁CAP -▁Valen -▁islam -ieval -anter -rachte -iza -laim -▁sought -▁dargestellt -▁trips -▁imports -▁verboten -▁gerichtet -▁Meines -▁besonderer -▁retail -▁bedeutend -▁Regierungschefs -▁renowned -▁Internationale -▁significance -▁Großteil -▁characters -▁cleaning -▁continuous -▁verdienen -▁vorliegende -▁Kreis -beschreibung -sprogramm -▁skiing -▁horse -▁hervorragenden -▁tranqu -▁militärische -▁Facilities -▁Erinnerung -▁Mallorca -▁Thailand -▁ere -200 -▁videos -▁SK -▁Pac -▁A. -▁lö -▁lunch -▁kern -▁Kas -▁anzub -▁dependent -▁geplant -advant -grenze -ogn -auben -▁relative -▁hingegen -▁justified -▁societies -▁dry -▁Attra -ößen -▁angekünd -▁Versionen -▁Resolution -▁Poly -▁excurs -burgh -▁Sky -freie -▁portug -▁Gleichstellung -▁Kontinent -▁freedoms -▁radical -cribe -▁harmonisation -▁Volksgesundheit -rückt -▁staatliche -mitglieder -technik -▁Entspannung -ugoslaw -▁Joseph -▁entreprene -▁Piazza -▁Armee -nete -nimmt -dis -▁tables -▁none -▁fans -▁Sammlung -emon -▁Wed -▁Vista -rob -▁Andere -anning -grunde -▁attrib -halle -izen -▁schäd -▁vorbei -▁manip -▁interview -▁legally -▁prev -▁Schlag -▁Barr -▁specialist -▁looked -▁strategische -▁Betreiber -▁refle -▁referendum -▁Modelle -▁negotiating -▁Leitung -▁Augenblick -gefunden -▁Befugnisse -▁welcomes -▁Venice -▁natürliche -▁clubs -▁begeg -▁supervision -▁indirekt -▁erfreut -▁Secretary -▁Wähler -▁jetzigen -▁Education -oubtedly -▁è -▁eat -dir -betten -▁Big -▁eignet -▁zieht -▁Canc -▁Hit -arn -▁beur -▁Nummer -ichtung -▁1st -▁asp -agent -ubeh -▁nachdrücklich -▁Vorder -▁Zeitung -▁Grundlagen -▁complicated -▁surve -▁surely -ühle -▁hohem -▁contributed -▁enterprise -▁gewähren -▁appearance -▁normalen -▁normalerweise -▁Dokumenten -▁Vorausschau -▁infring -▁unabhängigen -▁Auditors -▁sharing -▁elektr -▁suffered -▁homes -▁Beteiligten -▁Beaut -▁Luxemburg -▁Katastrophen -▁forthcoming -▁gewünschten -▁hospitality -verwaltung -▁Aufsichts -nor -rik -räumen -sitz -sätzen -zusammen -pol -ächte -▁Sied -▁Dub -▁Kein -▁Holl -ulous -ürzt -leiter -▁zugrunde -▁Jos -▁ausser -▁enabled -▁compos -ußert -▁disadvant -▁68 -▁everybody -▁festzulegen -▁modules -▁Wirtschaftspolitik -▁Fahrrad -▁hardly -▁remained -▁Märkten -▁preventing -▁Kirchen -▁leichter -▁originally -▁Binnenmarktes -▁Small -▁Innenstadt -▁Oliven -▁Gespräch -▁kontrollieren -▁AND -▁Konzert -▁allocated -▁NGOs -▁indirect -▁gemeinschaftlichen -▁Heizung -▁parameters -▁amenities -austausch -TR -gro -▁nütz -▁sam -▁EX -▁ceil -▁Fein -▁Feat -▁beings -gener -▁Unions -ugang -▁exzellent -weich -▁enger -▁Befür -▁Spir -▁Spekt -rix -▁wellness -▁bedanken -▁Äußer -▁directed -bry -▁speichern -zahlen -▁studio --Er -▁Does -▁Australia -▁externen -bestimmungen -▁calm -▁Socialist -▁ähnliche -▁ordinary -ssysteme -▁superior -▁proceedings -zutage -▁museums -▁Flüchtlinge -▁Customer -▁romantic --class -▁sichtbar -▁Börse --General -▁simultane -rt -xy -V. -▁nord -▁Dän -▁Hann -▁Häuser -▁Rod -asion -▁immen -stens -▁habit -ehnen -ehme -▁Stability -▁überaus -▁Spitzen -▁Fragest -▁compact -ützlich -▁indicate -▁Berichten -oys -▁realise -▁05 -▁beschränken -▁programming -▁Should -▁Americans -▁Quad -▁wirtschaftlich -▁irre -▁movie --Vert -▁komplette -▁scheinen -▁cuts -▁1960 -▁telec -lösungen -▁firmly -▁erhoben -▁errors -▁Generationen -▁Reservierung -▁cells -kapaz -▁Ursachen -▁unentgelt -▁Saturday -IK -öch -BC -▁fördert -oted -▁zit -▁cra -▁kg -▁Fels -▁Föder -▁Gard -iegt -▁repräsent -istischer -auspiel -▁mak -▁Album -nerg -▁Asian -ypen -▁Brun -iversary -▁festgelegten -▁geringer -▁fees -▁Autos -▁Gewerk -▁erstklass -▁Transfer -▁movements -▁existiert -▁Speisen -▁Christmas -▁Museen -▁recognized -▁Aufenthalts -▁31. -▁Gemeinsame -▁missing -▁erweitern -▁Organe -▁Serbia -▁Aktionsplan -▁untersucht -▁CHF -▁barriers -▁Johannes -▁Konflikte -▁hoffentlich -▁unzureich -.net -eben -nische -bung -baus -lers -▁hes -▁sight -▁whereas -▁Tausende -erbe -▁künstler -▁Civil -▁HU -▁Vul -▁Lit -estaurant -▁Verteilung -▁spons -sson -▁Alters -ords -▁saved -▁recy -▁heim -▁insert -▁productivity -okratie -▁hearing -▁explained -▁showed -▁introducing -▁classes -▁sustainability -operations -▁Passag -▁Moroc -▁Stärke -▁Kreditkarte -▁einheitliche -▁enhanced -hyth -▁remarks -▁juris -▁functionality -▁Culture -▁korrig -▁Geneva -▁amazing -▁Kriminalität -water -yll -li -OC -▁tree -▁nützlich -▁Tok -▁DC -▁Dritte -emes -▁Ku -▁C. -▁anhalten -▁Lohn -▁aufgebaut -▁deserv -▁sicheren -ells -▁umsetzen -thand -▁Beide -▁Alb -bling -reicher -▁hierfür -▁Rechten -▁herum -▁Meeting -▁Handlungs -▁zurückzuführen -▁sections -wertet -▁combine -▁Wochenende -▁Komplex -▁Consumer -riedenheit -▁wonach -▁Nachbarschaft -▁perfekte -▁Gefangen -▁komfortable --condition -▁wirksame -▁imagine -▁humanitäre -▁vessels -▁obstacles -▁liquid -▁female -bsp -junkt -▁soph -▁songs -▁mi -▁Dor -▁EV -▁Bor -▁Bab -▁Kür -▁Hon -bergen -opf -▁Alkohol -▁58 -getragen -phase -▁transmission -▁endorse -▁Emissions -▁Reserve -▁Finanzkrise -▁gefragt -▁regardless -▁investigation -▁25. -▁Zahlungen -▁metro -▁Territor -ksamkeit -▁Textil -▁Öffnung -▁Casa -▁unabhängige -▁collected -▁Schnell -spezif -▁Französ -▁splend -▁accurate -▁begleitet -▁laufenden -▁Würde -▁niederländ -▁Cookies -▁Himmel -▁fascin -▁Rechtsstaatlichkeit --Staaten -▁portugies -viertel --6 -äbe -▁Tests -▁bearbeit -▁Sold -eab -▁Köl -▁Hels -▁Fuss -▁Vin -▁Gall -▁incon -▁united -asis -▁antworten -▁Nero -▁sequ -▁zugelassen -▁erfordern -genz -▁vermut -uggl -▁Sieg -▁exempt -▁Auseinander -▁computers -akers -▁grossen -▁genehm -▁internationally -▁angewendet -holt -ädchen -▁organic -▁generate -▁turning -▁wahren -▁heißen -▁universities -▁exclusion -▁wünsche -▁veröffentl -▁reflected -▁Opera -▁Florenz -▁stocks -▁Islamic -▁stattfindet -▁incred -ümer --wide -DI -point -▁ride -▁rufen -▁gap -▁golden -irol -▁inev -uled -▁bez -▁behaupt -▁prime -istiken -▁verhe -▁Om -▁notes -▁vornehmen -▁cancer -uctive -▁disasters -▁mehrfach -lined -klar -▁Haben -▁einzuführen -▁deliber -eldorf -▁Bürgerbeauftrag -▁considerably -▁Quelle -▁Phän -▁student -▁schnellen -▁democrat -▁Darstellung -▁covering -▁ECB -▁Pakete -▁kompet -quisite -▁Hohe -mmern -▁synth -▁travelling -▁mittlerweile -▁Bereitschaft -▁klassischen -▁distinct -ef -een -yss -chant -▁cam -▁layer -▁Kart -▁Fass -▁pretty -▁Lith -hers -▁aufs -▁enttäus -laut -▁mac -ünde -akzept -ishment -▁Commissioners -▁trends -unktionen -▁Frü -▁invent -▁Überblick -▁ministers -▁gestärkt -▁Weltkrieg -▁belief --Pres -▁Muslim -▁schwar -▁Swimming -▁bewältigen -▁reaching -▁generations -▁Filter -▁encounter -▁Account -▁frühen -gerecht -▁Mängel -▁niedriger -▁Bonus -▁adapted -▁Somit -▁fleet -▁qualitativ -▁Satelliten -▁geäußert -▁dignity -▁ört -fleisch -SA -GE -▁till -▁rück -▁bauen -▁Samm -▁BA -ila -▁Gi -▁angebracht -▁undoubtedly -▁Omb -egel -▁Erschein -▁'A -ordan -spolit -▁bestellen -▁underground -▁Gesam -▁Brief -▁keinem -▁eingestellt -orgungs -▁140 -▁Debatten -▁Industry -wärts -▁labelling -▁tested -▁empfohlen -▁checks -▁schweren -▁Steuerung -▁Niederlanden --related -▁penalty -▁Faktor -▁Anwesen -▁gesetzlichen -▁Appartement -▁zerstört -▁Wichtig -▁Jahrzehnten -▁Bearbeitung -▁Louis -▁Straßburg -▁ingred -iat -durch --7 -ypt -EX -▁filled -▁aer -▁Tanz -▁Bru -▁craft -▁Kategorien -▁Fisher -imp -alus -▁Zerst -▁gepf -issance -iban -▁sichergestellt -▁excluded -▁überlassen -▁frame -▁Schauspiel -▁schr -▁worst -▁managing -euten -▁19th -▁Represent -▁Arbeitgeber -▁selber -▁Allah -▁Minute -vorschläge -▁discount -▁edition -▁verwende -▁acceptance -▁Parkplätze -▁Friedrich -▁violations -▁consulting -▁Analys -▁Diam -▁timet -politische -▁Wikitra -▁exploitation -▁faszin -▁Vereinfachung -yan -22 -▁tab -▁fax -▁ba -▁bike -▁Sier -erer -▁egal -▁Fal -▁Friendly -▁Glauben -▁ineff -ehold -odge -▁Erhaltung -▁plastic -▁recover -▁selben -▁Haup -▁restored -▁believes -▁Bestimmung -▁equivalent -▁experiment -▁besorgt -▁Servic -▁Schwarz -▁Energieeffizienz -▁Berichterstatters -▁rechtzeitig -▁Wirtschaftskrise -▁Reihen -▁Cross -▁digitalen -▁Korrekt -jahres -▁überwinden -▁reflects -▁Location -dienstleistungen -uggle -▁ergänzt -▁Branche -▁Wandel -▁türkischen -iovis -udsman -▁Richard -▁Wikitravel -bul -äle -85 -▁selling -▁Iron -▁Somm -▁Dra -▁Bour -etung -edig -esk -itate -orld -unction -▁lived -▁Vari -▁stages -▁stuff -▁Zut -appen -ections -▁Regul -▁56 -▁Leip -▁gesundheit -werfen -▁Immer -▁subsequent -▁Clear -▁gefäl -spartei -zusehen -▁Zahlung -▁Protocol -▁Investoren -▁Science -▁flexib -▁clause --Produkt -▁transferred -▁satisfy -▁Mittag -▁unmittelbarer -freundliche -▁Academy -▁Währungsunion -▁Registrierung -▁Ferienwohnung -▁spezialisiert -usalem -eing -kette -kampf -94 -gi -▁nine -▁Sir -▁Bir -▁Bele -▁CP -▁Gay -▁erzeugen -achter -outique -▁beenden -▁reliability -▁freely -▁interv -▁objekt -▁anywhere -▁Marc -▁varied -▁besucht -▁Result -▁designs --Form -▁multi- -▁column -▁Basel -▁Doha -systems -▁floors -verbrauch -▁Fernsehen -▁supplied -▁herrlichen -▁Urlaubs -▁geeignete -▁frage -▁losses -bitr -▁Easy -▁Georgia -▁concentrate -▁Konzern -▁multilateral -▁enorme -▁diesbezüglich -▁Andererseits -▁shadow -wig -TV --on -▁tsch -▁a.m. -▁vig -▁Maschine -▁ziel -▁Would -▁CON -▁ungar -▁Länge -leute -▁severe -▁wirken -orts -▁Prim -ahme -▁Weil -▁untersch -uctions -▁Parlamente -▁Arbeiter -angehör -mittlung -owski -▁gleichermaßen -▁Beseitigung -▁bekannte -▁integrierten -ohnt -uzenten -greg -▁rechtliche -zeichnungen -▁Welthand -▁Hochschul -▁minimal -▁zulassen -▁learned -▁Gelder -▁undertaken -▁sympath -▁overlooking -ugoslav -quiry -▁griechischen --scale -▁Wesentlichen -▁Sonntag -hüt -sa -ühen -machen -DER -▁het -▁villa -▁maler -▁Tonnen -▁Ticket -▁Tickets -▁Dingen -▁pas -▁Hunder -▁Gre -▁prag -▁Light -▁Zone -▁veränder -▁verweisen -ffekt -▁enem -▁bestand -gebä -▁Deck -▁konsequent -weiten -▁restore -zahlungen -▁Clean -ronik -▁cultures -▁Notes -EST -▁anticip -▁gegenwärtige -▁fishermen -▁Petitions -▁picturesque -▁reconstruction -▁ländliche -▁Histor -▁unglaub -▁sovereignty -▁Saddam -▁EU-Mitgliedstaaten -▁URL -▁restructuring -▁non-smoking -▁anzubieten -här -pet -pread -kul -▁IR -▁Sever -isa -onst -usch -▁cart -▁Koch -▁GPS -▁inade -iments -attern -arc -quer -▁stunning -▁stammt -▁errichtet -teneg -achts -▁auffordern -▁eins -eitern -agr -▁spreche -▁Parameter -eping -▁arise -änderungen -▁everywhere -▁Zusammenhalt -▁monopol -▁Monitor -▁Blu -▁air-condition -hältn -alender -▁Bewusstsein -▁24. -▁kompliz -▁strengthened -▁Jugendlichen -▁Eigentums -▁begeist -▁Fotograf -▁coordinated -▁Telekommunik -.2010 -uum -uage -van -ji -▁tank -▁hill -▁nbsp -▁froh -▁Borg -▁HIV -▁Rail -▁uncon -ivation -reams -▁Abente -▁traum -▁lebt -▁Vorbeh -▁Konzentr -inki -▁Implement -▁knapp -▁Att -▁formats -▁Zuverläss -obald -▁editor -▁negativen -▁Laden -▁Porto -▁irres -▁130 -ijk -▁Denk -▁massage -▁Benutzung -▁ambition -▁Daniel -▁Silver -▁fantast -▁Strecke -▁Singap -▁ausgewählt -▁untersuchen -▁stärkere -▁beruhen -▁Wiederaufbau -▁cancell -▁gekennzeichnet -▁panoramic -eper -rasch -34 -▁SQL -▁Drei -▁Katalog -▁Käufer -▁H-0 -▁Renten -andern -▁impos -estens -▁ersta -veranst -▁dei -ocop -print -▁shuttle -▁Schreiben -▁schätzen -ograp -▁Parad -▁350 -▁solar -▁province -▁decade -▁assembly -▁ordered -▁Energies -▁individually -▁Warcraft -▁listings -▁Linien -▁treatments -▁marg -▁Denmark -▁Instruments -PEG -▁Samstag -▁Bedürfnissen -▁Labour -▁Process -▁Tools -kretär -▁neighbouring -▁College -▁mangeln -▁thermal --the- -▁unentgeltlich -kre -▁Si -▁SW -ori -▁Cur -▁Gh -asi -bers -▁Lern -quenz -▁denjenigen -▁110 -agd -▁executive -▁champ -▁ausdrück -▁vorgestellt -▁Einfl -▁EU-R -▁warn -▁superb -▁57 -▁saving -inkel -▁hierzu -▁incomp -▁Sicherheitspolitik -rechung -▁Versorgung -▁approve -ohner -uzz -▁necessarily -▁anniversary -▁eventuell -▁Vertrieb -▁derzeitige -▁Frühjahr -▁holds -▁bathrooms -habt -▁Key -▁deeply -▁beinhalten -▁briefly -▁acknowledge -▁Todesstrafe -▁Religion -▁Journalisten -wirklichung -nik -fox -list -prechen -98 -NG -67 -▁hiking -▁myst -▁Spl -▁Search -▁Mah -▁Pad -atum -▁forts -▁Lea -▁verknüp -▁Schicks -ollars -▁Ausgleich -▁adds -▁Bevor -▁Berge -▁Ablauf -▁tracks -▁disag -anzig -▁51 -▁Yugoslav -▁Maxim -▁0. -▁eigener -▁Residence -▁Adv -▁Spielen -region -▁fairly -reibungen -▁magazine -▁normally -▁entscheidenden -▁bevorzug -▁targeted -▁Vertrages -▁offizielle -▁kontrolliert -▁Lehrer -▁unnöt -▁Nikon -▁Dresden -hängt -kö -OB -▁toll -▁senken -▁meal -▁Sobald -ité -▁RA -▁ine -indlichen -▁übertr -▁clin -tragt -being -▁mögen -▁realer -klear -▁damages -▁Bereits -▁Consequently -▁processed --Aus -▁bedeutenden -▁Umweltfragen -▁summ -▁portfolio -▁casino -isticated -▁tempor -▁Höchst -▁Subst -kämp -▁manufacture -▁Dialogs -gebieten -▁praktischen -▁gedacht -▁Galile -gelegte -▁teaching -▁Beachtung -▁synchron -▁Mexiko -▁Branchen -methoden -▁kommerzi -▁Dänemark --Vertrag -▁jurisd -▁¼ -▁и -yard -öhnen -▁wear -▁gastr -▁Area -otechn -usive -▁Greens -ieur -▁anzunehmen -▁Lesen -▁Nok -▁reisen -ague -▁usage -▁Alph -▁accompl -▁Make -hafter -▁Fläche -▁guard -▁highlights -▁appell -▁festzustellen -eldet -▁klim -▁Systemen -▁befasst -▁verletzt -▁engaged -ANT -fähigen -▁reputation -▁verschär -▁Society -▁decline -▁determination -▁Belgian -▁renoviert -utreten -▁Tauch -▁müß -▁reichhalt -▁unnecess -▁Mechanismen -▁kümmern -▁Hindernisse -▁Helsinki -ICH -kir -▁hung -▁Siem -▁Sports -▁Maus -▁Baby -ilm -▁KDE -▁GAP -inein -verhandlungen -▁Anhör -▁adj -ealing -▁Imper -▁ended -▁controvers -▁factory -▁wiederher -▁Klass -▁Hinweise -▁presenting -▁typischen -iffs -▁engines -▁Darf -ATION -▁offenbar -▁Weiterb -▁Performance -▁Zugangs -▁reserved -▁twice -▁brachte -▁olive -▁Senkung -▁Wissenschaftler -▁Venedig -▁beeinfluss -▁produzieren -▁bezahlt -▁transp -▁threats -▁müsste -▁bonus -▁Souverän -▁Vice-Pres -▁Ägypten -▁Weihn -life -UP -▁Clin -def -▁thir -open -stable -schte -ayers -▁Vergn -ielten -adr -▁poly -▁Alm -ektors -seher -inks -strasse -▁Maur -▁gewid -▁berechnet -▁struggle -▁endeav -▁drast -▁begins -geschrieben -▁academ -▁printed -▁Besuchen -▁Photos -iegeln -▁Kooperations -▁Ergebnissen -fähige -▁Vertretern -▁Euros -verletzungen -▁geregelt -▁Veranstaltungs -▁Smart -▁Auftrags -▁angemessenen -▁layout -▁Appartements -▁anzup -▁emotional -▁Premierminister -▁Kohäsionspolitik -▁Sierra -▁Zuverlässigkeit -cow -öten -▁vine -▁wides -▁Dring -▁pilot -▁AF -esis -▁Ross -▁infolge -▁beunruh -▁Ly -▁Lic -▁stelle -▁Verwirklichung -izite -▁meets -usse -▁Wirksamkeit -staats -▁recording -▁US- -▁servers -▁profits -▁lasting -▁aktiven -▁promise -▁promised -▁richtet -▁irischen -▁bewähr -▁combines -▁entering -▁appropriations -▁audiovis -▁Visa -▁erklärte -▁panel -▁langfristigen -▁Spezialitäten -▁Bücher -▁iPod -▁Finnland -▁Jesus -▁intends -▁älteren -▁Ansprüche -▁wechseln -▁Kaiser -▁angewandt -national -uhl -ml -phrase -▁ré -▁Mode -orous -unes -▁toward -▁Anmeldung -arts -ipts -merz -▁soeben -▁traged -▁Aufzug -ermann -▁produces -▁empt -ettes -▁politisches -akei -haust -▁Übertragung -▁Sex -▁ausgegeben -▁zurückge -gehende -▁Bildschirm -▁budgets -▁referring -▁Fahrer -▁jedenfalls -leiten -▁wesentlicher -▁22. -▁illegale -▁Suites --Stunden -▁gefährdet -▁effektive -ifikation -▁urgently -▁Datum -▁Drittel -projekte -▁wachsenden -▁müssten -▁prospects -▁Ombudsman -▁q -hend -metr -DS -UB -▁bla -▁Düss -▁Dreh -▁Metr -▁Messen -eda -▁Call -▁Vent -▁unr -arl -gefühl -▁wirkt -apse -▁Prin -▁Obst -ehnte -▁Stor -▁dauert -▁vorn -▁Mitentscheid -▁arising -▁Chic -▁Level -▁opens -▁sole -▁72 -▁einzur -▁Conn -▁Schlacht -▁Trib -▁Amerikaner -▁steuer -planung -▁Wohnzimmer -▁Oberfläche -▁efficiently -▁Using -▁ecological -▁execution -▁Aussage -▁einstimm -▁kilometres -▁Entfernung -▁phenomenon -▁kernel -look -ped -KT -▁tort -▁tast -▁native -▁dial -▁Tot -▁Ster -▁gran -igger -dex -▁Labor -▁Zauber -▁verank -▁weisen -▁Annehm -▁proxim -oda -oco -▁frem -▁Errichtung -▁Einen -▁durchge -▁relates -▁Register -ström -atering -▁Strände -▁Have -iversity -▁surprise -▁feet -versammlung -bedded -▁oldest -▁steigern -▁23. -▁Edin -▁camps -▁Prozesses -▁widersp -▁anzuwenden -▁effiziente -▁Lastly -▁geographical -▁Events -▁mistake -▁Vermittlungs -▁Präsenz -97 -jan -▁hea -▁hills -▁meals -▁Snow -▁Dör -▁EW -edge -igl -unte -▁Cla -▁CEO -stab -▁Joint -oca -▁ausüben -▁Gross -ütter -▁Regulierung -▁freut -▁festlegen -▁Zucker --Bere -▁Monteneg -ziehung -▁Safe -▁taxp -▁seats --friendly -▁oftmals -▁maximal -▁schwierige -▁Staats- -▁records -▁scientists -▁Minderheit -▁Mandat -abrik -▁candidates -▁Modernisierung -▁occurred -▁acquired -▁strictly -▁medicine -▁humanitären -deckung -▁Explorer -▁Nokia -▁beeinflussen -üm -ket -▁mental -▁ET -▁Pier -iling -using -▁Halt -▁Fang -▁Roth -▁Vog -▁beherr -reut -▁depth -▁Jon -▁Oz -arket -▁Vorstellungen -▁Vorrang -▁liable -▁evident -▁bedau -▁Finde -▁Truppen -recher -▁specialists -▁verläng -▁resulted -▁Betreuung -▁wirkliche -▁taxi -ARE -▁essentially -▁Benutzern -▁bemerk -▁consequence -▁europäisches -▁Innovationen -▁Operation -▁Anwendungs -▁Sams -▁berühmt -▁niedrig -▁highlighted -▁succeeded -▁zeichnet -▁Komplett -grenzung -▁hervorragend -▁sophisticated -vard -lus -äsche -96 -33 -▁hence -▁nom -▁mouse -▁Ter -▁MED -etion -emble -inely -▁Cru -▁inakzept -▁beer -▁Lear -▁Norm -▁ON -ehol -▁Ank -antas -▁abgegeben -ountry -▁upgrade -etta -riage -hai -▁konzip -ionalität -arma -handeln -▁Haupts -▁Handy -▁Ratschlag -▁brother -▁operator -▁Versammlung -boden -▁zweitens -▁heading -▁öst -▁Außen- -▁vorliegen -▁virtually -▁affects -▁Weiterentwicklung -▁incorporated -▁wheel -mittelsicherheit -fakt -dates --pr -▁hidden -▁wom -▁wachsen -▁Sem -▁Mädchen -irn -▁kennt -▁Ring -▁Räume -▁RSS -▁unus -▁behält -▁prol -ceived -quet -▁conventional -rov -▁Anbet -anned -eins -▁vorzunehmen -▁Ausbau -▁Ausarbeitung -▁compuls -ärer -▁4-star -▁belg -▁aviation -▁gestr -▁considers -▁airports --Daten -▁choices -▁Betriebe -▁ansehen -▁Assozi -▁momentan -▁Teams -▁Jedoch -▁Doppelzimmer -▁italienische -▁gradually -▁Intell -▁diplomatic -▁Firefox -▁Department -▁Leipzig -▁с -mot --line -▁fach -▁dea -▁sammeln -▁mor -▁Eind -▁eg -▁een -▁Hu -▁Fig -▁GP -▁inher -anforderungen -ckst -▁gehör -ternal -aka -▁lebend -▁arbitr -▁warning -ettet -▁gewinn -▁gewählten -eams -▁Muster -uzzi -▁Behand -▁sky -▁Airlines -regierung -▁Repar -▁Cooperation -▁causing -▁Flughäfen -▁fehlen -▁blocks -▁erkannt -▁associations -▁Traditionen -▁Mosk -▁teachers -▁professionelle -▁angenehmen -▁Wassers -▁Stiftung -▁Jerusalem -▁sudden -▁Edinburgh -läufig -Press -▁rid -▁Tieren -▁Tenden -▁kop -icity -▁derer -▁Lud -stär -▁revenue -▁stick -▁1,5 -▁Zimmers -▁OK -▁Three -▁sphere -▁Erfordern -▁Spitze -▁Chile -fahrer -gestaltung -▁Orten -▁Übersicht -▁Internal -▁Constitutional -▁Gebühren -▁smart -▁fundament -atorium -▁environmentally -▁versteht -▁entfernen -▁preced -inery -▁kostenfrei -▁encouraged -▁Danish -▁troops -▁offiziell -▁producer -▁NICHT -regelung -▁attached -▁surprised -▁explanation -▁Children -▁Bibliothek -▁Vice-President -▁widespread -pm -kons -▁fly -▁Ts -▁BR -▁cott -▁Huss -▁Fan -▁Varian -▁beibeh -▁Long -rafen -ungal -ichtigen -▁erteilt -osc -▁verwir -oven -▁Ausblick -gerät -▁accidents -▁rough -▁Arbeitnehmern -▁reporting -▁ends -▁seeks -▁Zuge -▁Trend -▁12.00 -▁Emissionen -▁puts -▁stayed -▁Werden -▁adding -ART -▁participating -ariat -▁vollständigen -▁restricted -▁Premium -▁beeinfl -probleme -wissenschaft -▁prisoners -▁enthusias -▁beabsichtigt -gäste -▁täglichen -▁Authority -▁Siemens -▁heil -▁mort -▁bomb -▁Sort -▁Source -▁Ast -inity -ilder -igo -ching -▁liv -▁Fast -▁VIP -▁tourn -▁anwenden -istical -▁Anh -▁vorgelegten -▁plöt -örungen -anzierung -bares -▁20th -stellern -▁wooden --Al -mögliche --sh -ronics -äufe -▁Gemeinschaften -▁engage -▁colors -▁weltweite -▁alongside -▁contextual -▁Again -▁stories -▁Zwar -▁Terms -▁Innovations -▁Lieferanten -▁utmost -▁Florence -▁IMF -übung -▁Verfassungs -▁clarify -▁Philosophie -▁Vordergrund -cas --orient -27 -FT -▁trat -▁dim -▁dance -emann -▁Kampagne -▁LAN -▁Jazz -agog -▁Anze -adays -▁umgehen -▁enthalt -▁soil -etzugang -ackt -ystal -▁grown -▁geniessen -▁genaue -▁agents -▁serving -▁Unterkünfte -▁marks -ächer -ictions -▁Extra -▁geschäft -▁Conf -brid -▁flu -▁linken -▁bereitet -eague -▁kostenlose -würdig -▁aktiviert -▁Danach -▁gefährlich -streben -▁ehemalige -▁Serbien -▁graphics -▁Argumente -vereinbar -▁Listings -junktur -aos -wid -wiesen -HO -▁doppel -▁E. -ilation -ilateral -otokoll -▁Font -▁bewerten -▁prize -▁zuges -▁geblieben -▁spite -ausgaben -▁clock -▁upgrad -▁Revision -▁indoor -▁recreat -▁Sekunden -▁endors -▁capabilities -▁specialized -▁Shar -▁achievement -▁preserve -▁Treaties -▁finanzieren -▁permission -▁Solar -▁Außenminister -valu -▁Pflege -▁überwachen -▁entdeckst -▁quote -▁günstig -▁observed -▁micr -▁verringert -▁harmful -gehörigen -▁kontextuellen -▁Engineering -▁beseitigen -▁thumbn -▁Lithu -▁Annehmlichkeiten -kel -kle -▁s. -▁IWF -▁Psych -▁Bier -▁Bell -inians -▁Wa -▁reiter -▁erfahr -▁einseit -▁eingebracht -▁convention -pruch -▁'m -▁manchen -▁Einer -▁grünen -arbeiter -geladen -weitung -▁Desktop -▁know-how -ölf -▁Palma -▁schlimm -▁1988 -▁purely -▁permit -▁Bildern -▁recognize -▁Steuerzahl -▁erhöh -▁Beschluß -▁Index -▁Burma -▁quotas -▁Gemü -schlossenheit -▁südlichen -▁multilater -▁hinzufugen -▁grammatisch -▁convergence -▁kontextuell -▁pillar -aktivitäten -▁Riviera -▁Konkurrenz -zio -▁hundert -▁bug -▁Ehre -▁AU -eace -oments -▁Cuba -▁Vie -▁pra -▁Nizza -idas -schuss -leger -apital -▁manifest -ourse -▁Arc -örtern -▁Major -bres -▁gemeinsames -▁erwiesen -amerikanischen -▁passing -▁seconds -▁schließt -▁communicate -völkerung -▁aufgezahlt -▁Dokumentation -▁Schlusselphrase -▁inspired -▁beantwortet -euerung -▁armed -▁Copyright -berichterst -▁Gegner -▁dispute -▁eingeleitet -▁Mechanismus -▁digitale -▁Café -▁estimated -▁uneingeschränkt -▁verwendeten -▁Spaziergang -▁conferences -▁Tyrol -▁Š -.D --off -küste -VIS -▁vollen -▁Times -▁Bah -emed -itis -cho -▁Got -▁Gibt -▁Lernen -▁Nehmen -stock -schritt -▁aufzubauen -▁veranst -▁Anim -▁Schalt -▁sou -▁schm -▁nachfol -sprechend -▁composition -gericht -▁verbreitet -▁flows -▁halb -▁Schwar -▁absurd --Ein -▁nächster -▁innovativen -▁Denken -▁Wohnungen -▁Können -▁Contact -▁household -▁benachteil -▁Publikum -▁hilfreich -▁coherent -▁gegebenenfalls -▁verstehe -▁Valencia -▁gewidmet -▁Samsung -▁Anbetracht -SC -BA -▁Tus -▁Ever -▁Bav -▁lädt -▁Cann -▁Cred -▁Rett -▁GR -▁unber -anträge -▁Nation -oses -▁aufbauen -▁Verhe -lac -▁soul -▁Ausweitung -▁Bezeichnung -▁1950 -▁developers -inners -haltungs -aceut -▁busy -▁airlines -▁integrate -▁schwere -▁definieren -▁raising -▁Subject -▁stores -▁ruhige -▁Legitim -▁victory -▁veröffentlichen -▁licens -▁Multi -▁hinzugefügt -▁initially -▁reagiert -▁hochwertigen -▁Sektors -▁percentage -▁Akteure -▁Blair -▁keineswegs -▁prestig -▁Niederlande -s. -var -katastrop -▁dollar -▁Mul -▁Ped -▁B5-0 -▁Ach -itative -▁Wood -▁Lot -reicht -▁Vert -▁OLAF -iale -▁Stern -▁christ -▁dauern -▁schritt -▁schreck -enness -▁adult -▁clim -▁3, -acio -flow -▁Umstruktur -▁maybe -▁flag -▁Bestätigung -▁medieval -▁operates -gleichheit -▁Saal -▁accounting -INE -▁2.5 -▁sufficiently -▁Agenturen -▁irgendwel -▁Sowohl -▁hopes -▁landwirtschaftlichen -▁Erwachsene -▁Nigeria -▁telecommun -▁tschech -▁Souveränität -▁Düsseldorf -▁tournament -ciation -gäng -▁sitzen -▁Dein -▁Echt -undern -▁Km -▁Fassung -▁Vier -alon -▁LE -▁LCD -owth -▁ständigen -ischt -hersteller -ups -essed -▁einschl -▁Verab -▁Anf -▁Antr -ikation -ouses -▁Einfach -▁Einkauf -▁herz -▁herzustellen -▁guided -▁Minim -▁expense --Ben -▁promises -▁edited -▁besonderes -▁stattgefunden -ässer --Em -▁24-Stunden -▁enjoys -drucks -▁desto -fällig -▁Eurost -▁seemed -irlpool -▁Verbrauchern -instrumente -▁Vorsitzenden -▁Dringlichkeit --10 -▁Tamp -irmen -▁Wol -▁Wal -▁Wunder -▁Kh -▁Hekt -▁Ford -▁Virt -umes -▁forec -heiz -▁Zy -▁erheben -ippe -▁Ausbildungs -eiler -ergabe -▁recall -▁overw --Software -klasse -▁festen -▁survey -uxe -▁implications -▁Demokraten -▁Million -▁staying -▁studied --country -▁Werten -▁shortly -▁generell -▁Industrial -iberal -▁approaches -▁Deutschlands -▁broken -▁erkunden -▁Compl -▁Etik -▁professionellen -▁Objekte -üller -▁Champions -begriffen -▁Wednes -aceutical -mess -yles -66 -▁hect -▁nomin -▁fassen -▁röm -▁Cash -▁HP -▁Voy -▁Verkaufs -▁J. -▁Anders -▁atember -▁Erf -▁Erreichung -▁enge -▁Eines -▁climb -twoch -▁acceler -▁59 -▁interf -▁Prozeß -hton -▁crises -▁Entschlossenheit -▁berichtet -▁charakter -▁Vereinigte -▁Projekts -▁XML -▁Transl -▁booked -▁Karr -▁classical -▁zustande -▁elegante -▁audience -▁relationships -▁meaningfully -▁gradu -▁Anpass -▁extending -▁ökologischen -▁assigned -▁adventure -▁Dynamik --Bericht -▁bearbeiten -samen -zimmern -lingen -▁nutzt -▁bought -inea -inien -▁kill -▁Curr -▁Foot -▁GT -berger -estelle -▁exhaust -▁spann -▁schemes -▁Erfolge -▁Arena -flanz -▁Auflösung -▁Chat --Service -▁Order -▁believed -▁Ratifizierung -▁Step -▁Residen -▁confidential -▁render -▁Universal -▁Universitäten -▁genauer -▁Scott -▁Aspekten -ambod -▁Luftfahrt -▁Diagn -▁describe -▁Volkspartei -▁chinesische -▁accompany -▁eingeschränkt -üstet -▁Nichtraucherzimmer -▁Empire --Entw -▁Anmerkungen -▁palästinensischen -▁principal -▁Galileo -▁är -des -burger -cz -▁Ehe -▁Ehr -edite -inet -urities -▁Rena -▁LED -stil -▁Inkraft -ichtlinie -tech -▁aufrecht -▁Verzeichnis -▁-S -▁shoot -▁schauen -▁envis -▁Grem -sichtig -▁gelesen -▁geliefert -▁gestatten -▁eingest -▁Beste -▁langer -▁Neuer -wicklung -▁Unsicherheit -kontroll -▁Portal -▁Nutzungs -▁negotiation -▁Entwicklungshilfe -fertigen -wärtige -verbindung -▁Insgesamt -▁satisfactory -▁constitutes -▁Kompetenzen -▁eliminate -▁bürger -▁branch -▁Getränke -▁ingredients -▁Reparatur -zellen -45 -▁teuer -▁bul -▁Py -▁Pod -▁Kürz -▁Hab -▁Runde -▁Jur -▁chapter -formular -▁sucht -neiden -▁Nachmittag -▁owned -▁Employment -▁amended -▁erstes -kauft -▁organise -▁addressing -▁1986 -▁checked -▁Prof. -▁heads -▁Leiter -▁poorest -▁Panorama -▁29. -▁Ereignis -▁campaigns --to-d -▁Spring -▁medicines -▁unzähl -▁scheduled -stätten -▁gesellschaftlichen -▁grenzüberschreitenden -▁neighbours -▁surprising -▁Fremdenverkehr -▁schwedischen -▁Chairman -gorith -▁splendid -▁Soldaten -▁macro -▁Weiterbild -ping -KA -92 -▁häuf -▁foss -▁vibr -▁baby -▁IG -▁Eure -irable -▁lect -▁luck -▁kolle -▁Hold -▁Fle -▁FOR -▁Ruß -ulating -ulator -▁anerkennen -▁League -andin -▁erwerben -appe -ighter -▁auswählen -▁max -▁qui -bevölkerung -▁whereby -▁compliment -▁governing -▁Adri -▁Schwellen -▁bewahren -▁contextually -▁William -▁weaken -▁joining -speed -▁concentration -▁gemütliche -▁remarkable -▁enormen -▁grammatically -▁Slowakei -▁enforcement -▁Kommentare -▁Jewish -▁beibehalten -beln -IER -äub -▁halt -▁sorry -esa -ints -▁Künst -▁Fer -▁Renn -▁inhalt -▁erled -▁gegangen -▁whol -▁umge -▁vorüber -▁resc -hlung -▁Proz -ergel -▁herbe -▁technique -▁Seele -▁Kling -▁Intergovernmental -▁variable -▁Phys -▁Durchsetzung -▁hohes -▁violation -▁urgency -▁schriftliche -ECD -▁translated -kräften -▁Zweiten -▁widmen -▁Moscow -▁kostenlosen -▁Betriebssystem -▁ultimate -draul -▁Garantien -▁Tunis -▁verurteilen -▁ausländischen -▁imperial -▁Vermögens -▁Slovakia --Bereich -▁ease -.T -service -üche -well -pakt -OV -88 -▁fell -▁wil -▁wife -▁Treff -▁Sep -▁Dur -▁poker -▁Gelt -▁Print -essel -▁einladen -roc -▁Anonymous -undry -asso -ippen -ärme -▁Marco -▁europa -▁boasts -▁enjoying -▁zweiter -regionen -▁bereiten -▁Kontin -▁Sitzungs -▁bedroht -▁kleinere -▁berühmte -▁Massage -▁elektronische -▁Elektr -▁mittelalter -hetor -▁robust -▁Klarheit -▁incentives -▁Inflation -▁Pacific -▁Fisheries -▁empty -▁compulsory -cat -jing -5-00 -▁mere -▁moments -▁winning -▁pf -▁Migration -ieu -imental -ician -▁Ingen -▁erlassen -▁aside -▁Ox -▁residential -proof -osev -▁Konver -▁Chief -▁oppon -▁Rechner -▁undermin -▁Marok -▁incident -▁boats -▁employed -▁Malays -▁existieren -▁Betroffen -▁Modul -▁completion -▁Patten -▁Filme -▁kurzer --Prozess -▁ratification -cluding -▁preferred -zweif -▁Gemeinden -▁mixed -▁overlook -▁chairman -▁aktualisiert -▁maßgeb -▁girls -▁fascinating -▁Köln -▁Wednesday -funktion -uant -bec -37 -OD -ôte -▁drag -▁Hill -▁Fü -▁bezeichnen -▁Ut -ract -städ -▁erteilen -▁erarbeitet -▁vermittelt -▁conciliation -▁chat -▁umgehend -▁Weine -▁array -▁Leban -▁elev -▁solch -▁brill -▁integrierte -ötet -▁famili -▁fewer -▁foundations -ziehungs -▁references -▁marine -▁Select -▁Scot -▁unterschiedlicher -▁Designer -▁Empfang -▁pools -▁26. -▁Erzeuger -▁appointment -▁legitimacy -▁navigation -▁Krieges -▁electrical -▁deserves -▁cancellation --Entwick -andinav -win -cs -pg -▁deter -▁Together -emost -raums -bends -ingung -▁vertritt -▁Verf -▁Stimm -▁chief -▁sparen -▁spanische -rankungen -▁Mittwoch -▁Ausrüstung -▁cancel -becken -amps -▁Reicht -▁Youth -▁Unterricht -▁height -▁62 -▁External -▁Interview -ündeten -▁Besatz -peicher -▁portal -▁Grünbuch -ächtigt -▁Australien --We -▁headquar -▁reass -▁Relax -▁getrennt -▁Similar -▁falsche -▁Smith -▁freundliche -▁Peters -▁Missbrauch -▁Archiv -▁Mikr -▁Ursache -▁Heads --eff -OG -ни -▁dän -ilo -▁lingu -▁Hände -▁Rass -▁thres -kept -▁steam -plate -▁extract -▁Bergen -▁moreover -▁armen -anka -▁indigen -ieße -▁autumn -▁Ashton -▁incom -▁Entwickler -▁Ministry -ältigung -▁considerations -▁faces -▁edit -geworfen -▁Behörde -▁Fortsetzung -▁übr -▁Solana -▁Schulz -▁newsletter -▁Californ -▁umfangreich -▁Menschheit -▁authors -▁Reinigung -▁Wichtigkeit -▁observe -▁kombiniert -▁dictators -▁Beamten -▁sacr -mietung -▁timetable -▁Schicksal -▁thumbnail -zte -FOR -но -▁fühl -▁messen -▁D. -▁M. -▁Meilen -▁Publ -chmark -▁CAN -▁Häfen -▁Rhein -▁beweisen -quelle -▁erörter -▁Prest -▁coach -▁Ausrichtung -prozeß -▁Konzer -sträger -▁Haiti -▁begun -▁würd -▁debating -▁persu -▁typisch -▁wirklichen -▁Lebensmittelsicherheit -füllt -▁Series -▁Edel -▁Nordic -▁schwerwie -▁überwie -▁erkennt -▁false -▁effizienter -▁travellers -märkten -▁Aussagen -▁neighbourhood -▁Ferienwohnungen -republik -▁repeatedly -▁überrascht -▁Ministerr -▁inadequate -▁anzupassen -▁¶ -▁ärm -iop -fect -via -kammer -ögern -él -▁dual -▁sizes -▁vir -▁EB -▁Evalu -erce -▁theatre -▁Intens -ayout -▁einher -▁OECD -▁Sternen -▁escape -adel -▁grat -▁Yo --Server -▁Nachdruck -standard -▁exposed -▁eingetragen -▁secondly -▁secondary -▁favourable -▁sheet -ographie -▁analog -▁Milan -▁Krebs -▁coastal -▁genuinely -▁umfangreichen -▁innere -stechnik -▁befürworten -personen -▁Bosnia -▁journalists -▁Bestellungen -▁conscious -▁behauptet -▁unnecessary -eiden -aupt -▁tenden -▁bread -▁Mang -▁PV -esc -▁RE -▁vergeben -▁conj -schauen -ouren -▁Vorred -ologisch -▁proport -▁Program -▁Zeitplan -geld -▁versehen -▁geringen -▁187 -▁Ök -▁folder -▁öffentlicher -▁1985 -▁1987 -▁logic -ängerung -▁Konserv -▁Corporation -▁Chef -▁codecision -▁Wahrung -▁bearing -▁Bonn -▁verursachen -ampagnen -▁korre --2013 -▁Wünschen -▁Felder -▁History -▁Administration -▁neutral -sphäre -▁Lateinamerika -▁Kürze -▁Storage -▁Montenegro -▁threshold -namen -fisch -dring -zia -ça -▁Pilot -▁Ay -▁lively -▁lenken -▁Fans -▁foremost -▁Lor -▁einheim -antes -▁auswirken -▁Abbau -▁compete -akul -gerichts -immte -block -▁Diesen -▁evol -▁interne -onda -▁underlying -▁Weltwirtschaft -▁irr -▁featuring -voller -vollständ -fäll -▁characteristic -▁guaranteeing -▁28. -▁Einzelheiten -▁Logistik -▁Befehl -▁Domain -▁günstigen -▁unterbrochen -▁Weiterhin -▁speeches -▁listening -▁übereinst -▁geehrte -▁respektieren -▁cookies -▁Venezu -▁Vorgehensweise -▁‚ -jud -ña -▁sample -▁vit -▁Section -▁EIB -▁pod -▁pump -▁Migr -isode -orking -▁Wurz -▁Reit -iehlt -ierer -▁thick -▁Neg -schätzung -▁haz -▁stip -▁Jud -quote -▁managers -aco -iders -▁subsequently -ionale -▁economically -▁operated -▁ablehnen -▁Demonstr -▁Russlands -▁Zimmerbeschreibung -▁representing -▁respects -zession -▁informal -▁Guide -frecht -▁bigger -▁Freihandels -produkten -▁vorliegt -▁Alternativen -▁expanded -▁dramatic -▁Festplatte -▁Volkswirtschaften -▁Several -branche -PA -▁wed -▁Taten -▁Sah -iev -▁derselben -▁Nichts -ston -genöss -▁geh -▁Erl -▁Alps -spir -ologne -▁Plätze -▁asset -▁gelang -▁idyll -▁instantly --Fl -gehenden -▁bewegt --made -▁slopes -▁textile -▁planen -reserv -▁institutionelle -▁suggests -▁Statistiken -▁examination -▁CDs -▁E-Mail-Adresse -oleranz -▁Keine -▁arranged -▁günstige -▁frequent -▁gemütlichen -▁endgültige -▁erleichtert -▁SDL -▁Bekannt -▁Know-how -▁Offenheit -▁ausgestattete -▁Francisco -▁begegnen -normen -neten -funk -sburg -ра -▁tang -▁hell -▁diving -▁Mol -▁Mix -▁Mischung -isst -▁cod -▁Wen -▁FI -aton -amed -▁Lieb -▁seg -verhalten -illigt -▁Erarbeitung -obb -▁könne -▁Aufhebung -▁actors -▁Referenz -▁Leid -▁openness -getreten -▁invo -ionally -▁Übernahme -▁autor -▁importantly -▁Zielsetzungen -▁expenses -▁artific -▁terrible -viously -stehende -▁ALDE -▁kürz -▁Chancengleichheit -▁weakness -▁ownership -▁intentions -▁tatsächlichen -▁IBM -▁complaints -▁fortsetzen -.ch --a -lere -▁ring -▁mater -▁Sark -▁ED -▁Point -eatr -unen -derte -▁Line -stätigkeit -▁convers -illance -▁übermitteln -ads -▁compr -▁davor -▁Eröffnung -▁engen -▁propag -▁Rechen -▁realisiert -▁einzul -▁gestartet -geschwindigkeit -▁Richter -▁mediterran -▁Steigerung -▁breach -▁Object -▁erforderliche -▁Betrifft -ophob -▁devoted -▁flexibel -▁eleganten -▁Clubs -spielen -▁clicking -▁certified -▁willingness -▁sogenannte -ocktail -▁wunderschöne --Version -▁angekündigt -▁Morocco -gebäude -iary -vy -mant --and -emp -empf -ota -otion -▁körper -▁Wars -▁Kenntnisse -▁Citiz -▁Händen -arians -lein -▁zukommen -▁abgesehen -▁vorsieht -ecution -▁Ausnahmen -▁Einnahmen -hnt -ellence -▁Derzeit -▁zwölf -▁insufficient --Sh -▁damal -▁Geschlechter --Part -▁beruh -▁accessories -▁Musl -zugs -▁Palette -zustimmen -▁Nationalpark -▁empfängt -▁Profil -bestand -▁alternatives -▁Liberalen -▁purchased -ummern -▁Village -▁behinderten -▁vocational -▁arabischen -▁Nobel -▁excursions -ziger -wass -yo -▁fat -▁bay -▁DJ -▁patch -institutional -irts -▁WE -▁Ken -▁Gute -schicht -▁reward -ocation -▁Arts -▁upload -achtungs -▁Entschäd -▁Seen -▁Beschränkungen -insen -▁ideally -▁letters -▁destinations -▁Assist -fertigung -▁calendar -▁angenommenen -▁jointly -▁herrschen -▁sunny -▁sharehol -▁trouble -▁convenience -korea -▁Georgien -▁Fischereip -▁behavior -▁tragic -▁apparent -▁Bosnien -▁parlamentarischen -▁Tragöd -▁erstreckt -▁Gefängnis -▁simultaneously -▁pragm -▁jurisdiction -▁Verheugen -mine -wände -lan -OK -ко -▁dil -▁vacc -▁Tipp -▁Bart -▁Baden -utragen -▁CL -▁Cab -▁Halle -asien -▁tox -▁Ufer -▁Near -▁zugestimmt -▁dep -illon -▁Bez -▁clarity -▁arts -▁Mengen -▁technisch -▁carte --Man -▁berichten -▁Freed -▁deutliche -ätzungen -▁erstens -exp -▁Module -▁posts -▁accordingly -▁27. -felder -▁Immun -▁Kernel -▁erwähnten -▁erheblichen -ademie -▁verzichten -öpfen -▁diesbezü -ipeline -▁rubber -▁Zerstörung -field -pes -Hz -36 -OU -▁fur -▁fault -▁Tee -▁Distribution -▁Bünd -▁leiten -▁Rennen -angebote -anbul -alin -▁Nied -hebt -schrank -▁reb -▁Pros -▁constra -▁Versprechen -add -▁Arr -▁discl -▁transfers -▁konfigur -▁Country -▁Geschenk -▁langem -kunde -▁measurement -▁bewußt -fraid -▁Carl -▁Netze -▁distribut -▁obligator -▁Zurich -▁Südafrika -▁inspection -▁twin -▁Privatsphäre -▁1945 -▁motorway -▁üblichen -▁A4-0 --Dateien -▁Arzneimittel --to-date -giene -▁horr -▁vic -▁wage -▁Tief -▁Baut -▁C5-0 -▁Händler -atom -▁Inhalten -▁eröffnen -▁desktop -▁Storn -▁abges -▁spekt -▁Erhalt -▁plain -elta -▁composed -▁genießt -osex -▁commod -strand -▁Alltag -▁problemlos -schließt -▁buses -▁größter -▁refused -▁Rückgang -▁postal -▁Charme -▁interessanten -▁Empfänger -▁Southern -▁Werkzeuge -▁beteiligten -▁Gemeinsam -▁chemische -kurse -▁niedrigen -▁ursprüngliche -▁Thermal -▁spectrum -▁Sowjet -▁Ernst -soever -▁Gesichtspunkt --of-the- -▁Tampere -PG -▁dupl -▁mine -▁wächst -▁Pun -oning -▁KOM -▁Han -▁anpassen -▁Library -▁impr -ctor -▁rebu -▁convey -▁orch -▁Einkommens -▁Arzt -▁zurzeit -▁Größen -▁Growth -▁transaction -▁gemeinsamer -▁Saudi -ikanischer -▁collapse -▁steigt --Haus -▁Double -▁Patr -▁continuously -▁vereint -▁Vertrauens -▁Visual -eeing -▁eindeutige -OTEL -▁Palestinians -▁Language -▁certification --to- -▁komfortablen -▁occupation -▁stärksten -▁Ursprungs -▁fifth -▁strike -▁Governance -▁NEW -▁vermutlich -▁tragedy -iang -SI -lager -▁retr -▁mont -▁Ten -▁Mund -▁gym -▁Circ -▁Futter -▁unity -icars -▁tower -▁NRO -schiff -▁erweisen -▁gehabt -▁einiges -▁adm -▁evolution -▁produkt -▁recru -affe -▁66 -▁finds -▁driven -▁Bürokratie -ocket -▁Großen -▁modify -▁specially -▁revised -▁gebunden -zufügen --Lös -▁Protest -▁Fischler -▁eigentliche -▁Terroristen -▁zentraler -▁verabschieden -▁leistet -▁Finnish -ifestyle -▁forgotten -lieferung -▁Versicherungs -▁dieselben -▁Anzeige -▁California -▁riding -▁afraid -▁v. -▁Mid -▁Buen -▁ASE -elung -irat -▁cu -stücke -▁Intel -▁erarbeiten -tress -▁dieselbe -▁umso -▁Ausgrenzung -programmen -▁grass -eading -▁oppose -▁assert -▁Minibar -▁Brücke -anspruch -▁92 -▁advis -▁doubts -▁Gesetzgeb -▁parks -▁logo -▁herausragen -▁manufactured -▁Erfolgs -▁publish -▁2015 -▁Wiederherstellung -▁rechnen -▁falling -▁SMS -▁Otherwise -▁Wikicars -▁zugute -▁deadline -▁komplexen -▁Philipp -▁einzusch -▁Begründung -▁nobody -▁afrikanischen -▁Renaissance -zähl -yards -▁tip -▁Send -▁Pent -inema -▁Kost -▁unab -anch -verschm -▁eintreten -ibungs -▁allm -ubringen -▁abgest -▁Wireless -schlägen -igkeits -▁Repräsent -▁elder -▁Deep -▁knew -▁whatsoever -▁basically -▁sculpt -▁gelangt -bewegung -▁flash -holders -▁Formulierung -▁Wasch -klärt -▁Bewohner -▁verwiesen -ränkte -▁relaxed -▁promoted -▁Montage -▁Südt -▁eurozone -▁Updates -▁paths -▁Laser -▁Lasten -▁vermieden -▁Umstände -▁eingeladen -▁formuliert -▁österreichischen -▁Konvergenz -▁SH -▁pul -▁Peace -▁Cologne -▁Hyp -▁Fuer -ambres -gehalten -▁anf -▁prel -▁Len -▁stopped -istall -▁geplan -▁Ocean -▁dasselbe -▁plural -▁EU- -aufnahme --Sp -▁belongs -▁Art. -▁Problematik -▁Klang -▁verliehen -▁healthcare -▁renomm -▁expressly -▁regionaler -▁progressive -▁Zurück -▁Conserv -▁losing -▁affecting -▁jüngste -▁Brok -▁frequency -▁removal -▁landscap -▁Olympic -▁befürwortet -▁ausführlich -▁Nichtraucherzonen -▁gewünschte -▁Grafik -▁monuments -▁Gipfelt -▁Welthandels -kömm -▁dimensions -sucht -ville -bean --old -äußer -▁bicy -▁Differ -otograf -chester -▁Corn -asonic -▁Look -▁verschaffen -indliche -▁residents -▁unterscheid -▁up-to-date -staates -▁productive -▁geschmack -▁Klassen -▁Interpret -▁visits -▁moderner -▁170 -▁loans -▁focuses -▁Profit -verständnis -▁examined -ämpft -▁Jugendliche -▁Logik -▁Zunahme -▁arrangement -▁Fakten -▁Menschenrechtsverletzungen -▁vertraut -▁beef -▁quantity -azzo -▁gegenseitigen -▁transitional -▁affordable -▁ermutigen -▁pharmaceutical -▁Tarifa -▁Millennium --oriented -▁Verabschied -hause -TC -IO -like -▁Distr -▁Eben -▁Ell -▁Eleg -ignen -▁ohn -▁W. -▁Kohl -▁stead -▁städt -tex -▁aufgestellt -▁abgeben -▁aussehen -▁Einst -schlüssel -▁leichte -▁delays -▁suited -pressed -▁buying -▁profound -▁Stef -▁musikal -▁integrity -▁integrieren -▁Photo -▁defeat -verbindungen -▁schwieriger -▁declare -▁wave --style -▁Lissabon-Strategie -▁Funktionieren -▁geeigneten -▁kleineren -▁renewed -▁Verstär -▁accuracy -▁Ausbeutung -▁bedauerlich -▁Mobilt -▁Föderation -▁Bürgerbeauftragten -▁Verabschiedung -euert -rig -mail -bi -card -▁treiben -▁fein -▁wahre -▁Sri -▁pace -▁Bach -▁zero -▁KG -▁Cert -▁beher -entlichen -ereien -▁zugesch -▁vergangen -▁einstellen -ados -▁environ -ossier -▁relief -▁Defence -▁63 -▁Meinungen -▁Andalus -▁advise -▁folg -▁refuse -▁electoral -▁installations -▁distort -▁turns -▁broader -▁Superm -zeichnete -▁heutzutage -▁Pflichten -▁ausländische -▁medizinische -▁Syria -▁sterben -▁löschen -▁ceiling -▁Representative -hältnism -▁Abenteuer --Benutzer -tle -gelegen -XI -ún -▁shr -urer -chern -▁Wesen -▁Kuba -▁Hein -▁unbek -bern -rebt -▁segment -teien -ached -▁Theorie -▁Straft -▁Andr -▁ents -▁encr -▁durchzusetzen -▁sake -▁Left -änderung -anca -▁Finanzen -▁Produktivität -▁Handlungen -▁Baron -möglichen --Ges -▁magnet -▁logical -▁Planet -▁späteren -▁exhibitions -▁Bureau -▁sonstige -▁gerechtfertigt -▁Kommissionsvorschlag -▁observation -▁automotive -▁Kennzeichnung -▁ethical -▁literature -▁Kapazität -▁präzise -▁Partnership -▁Wälder -uhe -PM -▁Tower -▁Dest -▁Biot -usten -▁echt -▁cameras -▁Katal -▁bezogen -stone -▁Inhaber -▁gebe -▁Verfolgung -▁Verordnungen -▁deserve -▁Stufe -▁Angabe -▁chose -füg -▁Spiegel -▁wars -▁keys -▁bisschen -stellte -enever -kript -▁geringe -▁Arbeitszeit -lesen -ässern -▁teilt -▁Abgeordneter -▁Türen -▁mechanical -▁nachhaltig -▁Maced -cludes -▁stressed -▁architectural -▁granting -▁jeglicher -▁Möbel -▁endgültigen -bringung -▁Präsid -▁catalogue -▁Buddh -▁Französisch -▁atemberauben -▁ea -col -47 -ро -▁aren -▁mathem -▁bot -▁bist -▁pensions -esien -onie -▁Come -▁Rural -▁bewertet -heil -▁Inhalts -verantwort -▁verarbeitet -▁whenever -▁Anträge -eans -blatt -▁trigger -▁gesichert --Spiel -▁crow -▁basieren -▁brands -▁Genuss -▁18th -▁Shang -▁Beijing -▁Quoten -▁retten -▁amending -zusagen -▁researchers -ARD -▁Scient -▁höchster -▁publishing -▁Drittstaaten -▁ultra -▁painting -▁komplexe -III -▁duration -▁distinction -▁therapy -▁novel -93 -Über -▁tap -▁dokument -▁aqu -▁sailing -▁mol -▁mouth -itual -▁ok -▁ungewöhn -atem -▁Lanz -verteilung -verarbeitung -▁Prize -plus -oga -▁trains -▁gratis -▁Zusammensetzung -▁125 -▁Versuche -▁achievements -▁confident -▁pressing -▁regulated -▁Solutions -▁Bilanz -▁mittlere -wanderung -▁Summe -▁Gastfreund -▁constructed -▁Taliban -▁Bühne -▁Elektro -▁soldiers -▁wachsende -▁religiösen -▁GUE -kriterien -▁Ltd -▁newspaper -zusammenarbeit -▁prestigious -▁Marokko -▁Fischereipolitik -▁unterscheidet -hältnismäß -hens -nier -ozy -aug -mach -▁sach -▁DIN -▁gate -uns -▁lib -▁kan -▁künst -▁Kim -gekommen -▁tough -▁stops -▁asi -▁conversion -▁Jugoslaw -▁abwechs -▁Aluminium -▁1930 -▁overs -▁transnational -▁gestiegen -inahe -▁passieren -▁Bewerber -▁beneficial -▁empfiehlt -▁Verlag -zeichneten -▁maximale -▁Monti -▁Kernkraft -▁Capital -▁Istanbul -▁angebotenen -▁wertvolle -▁Klimawandels -▁Anreize -▁officially -▁cancelled -▁schedule -▁explicitly -▁certificate -▁blieb -▁José -▁proximity -▁unusual -hme -iago -funktionen -rolle -rale -zing --ch -▁fanden -▁füllen -▁Mis -▁Bean -elter -oter -▁ocean -▁lakes -▁Farm -▁beinahe -▁Niem -reform -▁Zin -▁aufeinander -wehr -▁clo -prozesse -▁Konjunktur -▁freight -▁noon -▁67 -▁delete -▁concerts -▁Gebühr -fortable -▁beschreibt -▁185 -▁populations -▁Abstimmungs -▁innovations -▁machinery -apping -▁Verlängerung -▁ruhiger -▁preparations -▁Deposit -▁territories -▁sandy -▁Administrat -▁doctor -▁surveillance -▁Spektrum -▁einstimmig -aj --g -FS -▁fro -▁biod -▁pays -▁Kör -▁Fuj -▁alp -▁Night -▁hab -▁Zimb -dens -indungs -annel -ocal -lair -▁spür -▁Beurteilung -▁dargelegt -▁arrange -▁elig -▁solo -▁Unterdrück -▁stretch -▁avo -▁smoking -▁publications -AND -▁short-term -▁movies -▁Betrug -▁identification -▁Zustell -▁focusing -▁aufgeführt -▁wünsch -▁Everything -▁ignore -▁Erdbe -▁undertake -▁Einwanderer -kapital -▁Slowen -▁vereinbarten -▁toilet -▁Osteuropa -▁entrepreneurs -▁Holland -▁Hussein -situation -gast -то -▁Mahl -▁Bow -unting -▁lets -arier -umed -▁angelegt -steil -▁2nd -▁dein -▁depr -▁geograf -▁Jav -anged -▁partition -▁Regard -neider -▁Menschenrechten -▁technischer -▁78 -▁88 -▁drives -▁centers -▁Autoren -▁typically -▁necessity -▁organisierten -▁employers -▁precious -ska -bereiten -▁prevented -▁greenhouse -▁Erzeugung -▁hyd -lassungen -▁harbour -▁premium -▁englischen -▁grundsätz -▁Referendum -▁beschleunigen -▁ausreichende -▁Palästinenser -▁Literatur -▁deterior -▁geplanten -▁ändert -21 -▁mächt -▁Dampf -▁passt -▁gal -▁CE -▁REA -▁unions -imo -▁haft -andem -▁nehme -▁übergeben -▁frustr -▁chea -▁trained -▁Konvention -▁Chir -▁interactive -▁roots -ouv -amilie -▁gelassen -▁geschickt -▁debated -▁eigenständ -▁signals -▁Shopping -▁inneren -▁Schutzes -▁Feststellung -▁favourite -▁civilian -▁überwacht -▁Beratungen -▁releases -▁Verfasser -▁split -▁preserved -▁Herkunfts -▁Individual -▁uncertainty -▁Belastung -▁suspended -▁Präsentation -▁Erwägung -▁Anwesenheit -▁gewinnt -▁spektakul -tin -facht -aud -ügt -▁nutr -▁Meld -isons -otes -ushing -▁Hügel -▁Fok -▁Gate -▁unknown -reien -▁einführen -▁jen -▁Jordan -ennbar -▁Ausübung -▁admit -▁360 -sekretär -▁vielseit -stru -▁bedienen -etreten -▁liegenden -▁Strasse -▁weiterentwick -▁Brut -kunst -▁organize -▁Energiequellen -▁Technologies -▁logistics -▁1979 -▁eigentlichen -aptop -▁verstärkte -▁Experience -▁adjustment -melden -▁Wiener -▁9001 -mechanismen -▁revealed -elemente -▁Ergänzung -▁Steuerzahler -▁telecommunications -KW -ét -▁wal -▁winner -▁Symb -▁Soft -▁Mug -▁Cra -▁RF -▁reibungs -vez -▁einnehmen -▁conform -▁Ankünd -▁attended -▁abstra -▁comprises -▁chron -▁auss -gebühren -zuarbeiten -▁gelingen -▁Intervention -▁Freil -▁Gewinner -pectives -▁organisierte -▁negotiated -▁spielte -▁237 -▁Museums -▁Deleg -aubern -▁beitr -ucks -▁sagten -▁Gebäuden -▁judge -▁ALE -▁Heiligen -▁Fremd -▁Non-S -▁photographs -rohungen -wirklichen -▁Picard -▁Beschwerden -▁senior -▁vorbehalten -▁taxpayers -tage -name -nisses -riet -kar -gio -ZE -▁ment -etschen -onz -▁Fri -▁Roc -▁Vit -▁VER -icits -▁verzeichnen -▁vergröß -ugnis -▁Europar -▁como -▁spare -▁unterwegs -▁Absch -▁realize -▁verbleiben -▁geleitet -▁Which -▁188 -▁Rahmenbedingungen -▁Gewinne -▁refurb -▁thanking -▁presidential -▁Santer -haushalt -▁Vertriebs -▁Verletzungen -aßnahmen -▁gelegenen -▁Panoram -▁schlechte -▁komfortabel -▁Einschränkungen -▁feedback -▁ehrlich -▁indicators -reisende -▁Heimatland -▁spätestens -▁ruling -▁Č -▁о -oen --e -klaus -OF -▁treas -▁fails -▁Tank -▁Sorg -▁Dun -▁Acht -oton -▁Comb -▁Letzt -▁ständige -▁erhol -▁Until -force -▁überlegen -ainer -▁entr -▁polnischen -etti -▁transit -▁redund -▁Indik --Br -liegenden --Kr -▁teilzunehmen -fällt -zubringen -▁eingeräumt -▁Terminal -▁illness -▁Tabelle -▁Datens -▁spiritual -▁jährlichen -strukturen -▁conservation -▁threatened -▁acquisition -▁FAQ -▁Madeira -▁reflection -▁hesitate -▁Phänomen -▁Non-Smoking -,7 -rungen -museum -CA -äuse -48 -▁rating -▁ranging -▁Sul -▁Document -▁Poss -edo -eaux -inischen -▁Rang -▁Gärten -▁derjenigen -▁regeln -ütt -conn -▁Chines -älte -▁Unterhaltung -▁insofern -▁77 -▁eigenem -▁industriellen -erba -▁certainty -▁Geschäften -▁stattdessen --Ex -hausen -▁Gegenwart -▁purchasing -▁ratified -▁ähnlichen -▁kohär -graben -▁sonstigen -▁erschien -▁Olympus -▁replacement -ülle -▁mistakes -▁medizinischen -▁supplementary -▁Darfur -▁academic -▁inakzeptabel -vin -war -äsch -▁Tob -▁Tues -▁Song -▁Bat -▁Aer -▁züg -▁Watch -▁FL -▁File -▁Rog -▁Gastr -üre -▁Leuten -▁Num -andelt -▁verwirklichen -▁schuld -▁vorgehen -▁Albert -▁Geben -▁imported -▁Flächen -▁begünst --Pak -▁Isol -▁Arbeitslosen -▁Küchen -leitungen -▁externe -▁demokratischer -▁downloaded -▁passende -▁Einschätzung -limited -▁oberen -ladung -▁NGL -▁Argentina -▁beruflichen -mechanismus -▁privileged -▁Spannungen -▁Inkrafttreten -▁Adriatic -▁Rassismus -▁Young -▁allmäh -händler -aft -bor -olen -▁Cav -▁Hi -▁Gil -▁NOT -ested -▁Präm -▁Verts -▁Anbindung -▁Erbe -zugehen -▁justify -▁Bridge -itzung -▁restaur -▁Neues -▁beschließen -▁Blumen -▁downtown -▁citizenship -▁irreg --Ebene -▁Feed -▁moves -▁generous -▁negotiate -▁farms -▁demonstrates -▁lokaler -▁gefährden -▁deaths -▁announce -▁silver -▁beliebigen -▁respektiert -▁geistigen -▁patterns -▁Excellent -▁horizontal --Gruppe -yssen -▁Swimmingpool -▁License -▁beeinflusst -▁Karriere --of-the-art -▁District -▁£ -.j -di -cast -AF -NET -▁telefon -▁flying -▁Mary -▁geden -▁Bin -ometer -▁Wärme -▁Cad -ieve -amar -▁angeschlossen -▁alike -▁stet -▁Zahn -▁notably -anto -menü -spät -▁Abschluß -▁Platten -▁Marx -▁nationals -▁Grundsätzen -bridge -▁signing -ündete -▁größt -▁XV -▁occurs -▁patents -▁ratio -▁Installations -beschl -▁Elev -büro -▁consistently -▁erfasst -vertreter -▁erneuerbaren -▁authentic -▁Conditions -▁Rußland -▁Scotland -▁ärmsten -▁öffnet -▁icon -taking -brechen -yers -▁sessions -▁boy -▁Sab -▁Etw -▁Paz -▁os -▁Face -ingual -▁Anna -anni -▁enrich -▁bestens -▁Herrschaft -▁450 -ologischer -fahrten -▁Unterlagen -▁lieben -▁Produzenten -▁phones -holfen -▁Freuen -versicherung -▁deficits -odex -wachsen -▁kompat -▁ernsthafte -▁verpflichten -▁Cala -▁zeitgenöss -▁claimed -▁orientation -▁mandatory -▁indicates -▁emerged -▁chemicals -▁Route -▁Rubrik -▁favorite --Rezeption -▁argue -papier -▁Konstruktion -▁Konzentration -▁С -zicht --Über -lov -kas -gibt -GP -▁DER -▁Pict -▁Gallery -ifferenz -achläss -▁depl -▁notified -▁Stich -▁Anhang -menge -ourmet -▁3.0 -aced -▁accum -▁alleine -▁recession -▁Entsteh -▁creature -▁airline -▁Besitzer -▁Andreas -▁Amerikas -▁investiert -▁folglich -▁closing -▁automatische -▁1984 -▁Vertretung -▁Israelis -▁returns -▁erweiterten -▁ausgewählten -▁heavily -ungssysteme -▁Innerhalb -▁apparently -▁Mehrwertsteuer -methode -▁clothing -▁bureaucracy -▁kontinuierlich -▁worrying -▁marginal -cor -100 -▁bild -inth -▁leather -▁Kä -▁FC -▁Veg -▁inject -rez -reben -teen -osh -abile -▁conventions -▁Od -illo -ungspolitik -▁schenken -▁Ens -▁cooking -▁embar -▁herkömm -▁specify -▁Exhib -▁vollem -▁prominent -▁contributing -▁Rücken -▁Superv -▁magic -ASS -▁Beschäftigten -▁fortgesetzt -▁Visum -▁effektiver -▁kritischen -▁Ordner -▁advertis -▁predict -▁jemals -▁Regionalpolitik -▁applicant -▁russische -▁Verteidigungspolitik -▁erbaut -▁Ibiza -▁displays -▁intervene -▁kollekt -pan -VO -Гј -▁funkt -▁drit -▁senten -▁Batter -▁Hir -ivate -▁Laur -opole -▁Opf -ehin -mente -prä -▁Thyssen -okol -▁Abteilung -ietet -örm -ratt -▁tram -alties -▁Unterzeichnung -▁myth -othing -▁wherever --site -▁refund -▁Turm -volles -▁kostengünst -▁Solche -▁Raums -▁zufäl -▁Santiago -▁Wachstumsp -▁unterbreiten -ocolate -▁archive -▁gewünscht -▁Kreativität -▁Merkmale --Datei -▁Subsidiaritäts -▁Treibhaus -▁bedauere -▁24-Stunden-Rezeption -▁Voyager -▁erörtert -tons -▁hinein -erz -erium -erzeug -onts -▁WW -▁Kam -iese -anas -▁bewirken -idance -▁haus -▁reven -▁eindrucks -▁abschließend -ausstattung -dernfalls -▁platz -schlüsse -▁Mazed -istency -▁directions -▁Technical -▁bewiesen -▁Junior -▁Direktor -▁affili --Ch -▁precision -▁Investition -▁Konsum -▁Visit -▁judgment -▁evaluate -▁drinking -▁angenehme -▁coordinate -▁churches -▁Syrien -▁withdrawal -▁tonnes -▁unanimously -▁beizutragen -▁gültig -▁converted -▁Schwimmbad -▁Alpha -▁plötzlich -▁Citizens -waren -HS -▁tag -▁BE -emperatur -▁Cathedral -▁Hub -▁Fünf -▁Rind -ürzen -▁Lar -▁imbal -▁reiz -verter -▁abzul -▁werfen -laß -▁3rd -ounder -▁financed -▁council -strö -▁weitaus -uerung -▁82 -▁Büros -▁formation -▁qualifications -▁Apache -▁measuring -▁Adam -▁negativ -▁Gerichten -iminary -▁preca -▁violent -▁demokratisch -iquid -▁experiencing -▁credible -▁intelligence -▁slowly -▁Elektronik -▁Slovenia -▁Kräften -▁caught -▁Anhörung -▁Lithuania -▁Lanzar -▁Zinss -hän -rück -pra -PP -▁ham -▁Bund -invest -▁Right -ali -entliche -stern -sters -sthet -apel -▁chambres -fenster -space -▁Presidents -▁systematic -▁placing -▁possess -▁Brü -▁gestellten -▁zweier -▁Gebir -▁Arbeitsgruppe -▁Paläst -▁Palacio -▁1.0 -▁Studios -▁potentially -▁fairen -OME -▁answered -▁grundlegen -betrag -▁Democratic -▁entworfen -▁premises -▁competences -kompeten -▁Jacques -▁tropical -▁Medikamente -▁Choose -▁Trainings -▁Vergnügen -▁Hektar -▁elderly -▁Shanghai -same -ägern -▁fährt -▁diet -▁Sien -▁Wett -▁Kod -▁Hum -▁Lob -▁Listen -schreiben -▁stell -▁voneinander -isch- -▁Proph -▁einander -▁übermittelt -▁ausgesch -▁Konferenzen -fahrts -landschaft -▁verbreit -▁scener -▁humanity -▁Ausschüsse -▁garage -▁begründet -grees -▁Touren -▁analy -bereitung -▁accommodate --hour -▁Funktionalität -▁initiated -▁Entertain -▁victim -▁timely -ungsvertrag -▁silence -▁Getre -▁allocation -▁AKVIS -▁sympt -▁Angeles -▁Vermittlung -▁Antonio -▁Hitler -▁Äußerungen -▁Hannover -tre -oge -▁flav -▁sq -▁bid -▁bearbeitet -▁pm -▁Mauer -onto -▁Kinn -▁Finger -▁inbegriffen -icer -▁UV -▁Low -▁Nag -▁Neph -▁degrees -▁gehandelt -▁Berater -schlägt -▁Force -▁keen -▁Mineral -▁gelernt -▁Controll -▁getötet -geschoss -▁visitor -▁beschreiben -▁creativity -▁approx -▁erworben -▁integral -▁Bewältigung -▁verwalten -▁keeps -ORT -▁normale -▁Verluste -▁Juan -▁Motto -▁copies -▁schlechter -▁Eigentümer -▁Gemäß -▁heated -▁melden -▁Consulting -▁imposing -,4 -ei --th -31 -▁dating -▁male -▁Six -▁Drag -▁Mü -igu -▁Galerie -▁unstable -ierbar -▁Luz -▁aufgerufen -▁entschied -▁Erne -udd -▁nachdenken -▁Vorfeld -▁Rezession -ismo -▁Überleben -ansprü -▁geschafft -▁Besteuerung -holz -▁gebildet -liegt -konzept -▁respectively -▁state-of-the-art -ärenz -▁lighting -▁kompliziert -▁Release -▁erwähnte -ützten -betriebe -▁Info -▁aktualisieren -▁Classic -▁konstruktive -▁therapeut -▁intense -▁procurement -▁israelischen -ubehör -▁headquarters -▁ohnehin -hatt -oup -bahnen -DEN -pat -▁fitted -▁wake -▁Müller -▁Buffet -iscr -usste -▁stance -▁stays -istor -▁geheim -▁Unfall -indlichkeit -gründen -ungsmaßnahmen -▁entries -ecke -▁partial -▁Aufruf -▁heft -achtungen -▁Internetzugang -▁assurance -▁exposure -▁angewiesen -okratische -▁Track -▁charter -gleichen -▁Dienstleistung -▁zeigte -▁posted -▁Sitzungen -▁1978 -▁workshops -▁Hochzeit -▁warmly -▁Eigenschaft -schwung -▁Dritter -▁endgültig -▁Weißbuch -▁gebraucht -▁arbitrary -▁Malaysia -▁biodiversity -,8 -f. -▁Sard -▁Pest -▁Au -igi -▁zones -▁Gn -▁isn -▁unb -berat -rec -▁erörtern -▁Ans -iller -odox -izar -▁ausl -▁Bezahlung -igned -ampen -▁bedür -▁facto -▁Schließ -geschritten -aturen -▁designer -▁schlicht -▁2007-2013 -uffs -▁sentence -▁brain -▁unterbreitet -▁talent -▁disputes -musik -▁appreciated -▁einzugehen -▁ältere -▁angeblich -ergebnisse -▁bilateralen -▁erläutern -breite -▁Cubase -▁municipal -▁Attraktionen -▁Yugoslavia -▁endeavour -▁schrittweise -▁virus -▁metr -▁ECH -▁Past -▁Patch -▁Bit -isy -▁köst -▁Fly -▁Rö -▁G20 -tritts -abkommens -▁verdeut -▁jüd -alling -ews -objekt -▁Konzepte -▁Change -▁Forest -▁Progress -▁realen -▁handled -▁findings -▁agrees -▁modular -▁terminal -▁Behaupt -▁Studium -▁Energiepolitik -▁rechtfertigen -▁finanziell -▁21st -▁erfolgte --Or -▁terrorists -▁Relations -▁1967 -▁Contin -▁gelegene -▁Panasonic -▁pursued -▁Atlantik -▁fragment -▁detaillierte -▁Bedienung -▁konzipiert -▁Venezuela -▁Sarkozy -▁tips -▁detr -▁maps -initiative -irms -olas -▁Wille -▁Hôtel -▁Gült -▁unbest -▁tow -▁Lind -▁stützen -estag -esthet -▁aufer -▁Verme -▁Vermarkt -▁Stoff -▁Schr -▁Scheitern -▁chamber -▁Abu -akischen -▁Vorgaben -uktion -cedes -▁gutem -▁modernes -▁perspectives -▁technologische --EU -inzips -▁Fluggesellschaften -▁Coast -▁suggestion -▁contractual -▁CD-R -▁pursuant -▁well-being -▁unemployed -▁Hyper -▁nötigen -▁Liban -▁eingebunden -▁essence -▁Subsidiarität -▁kommerziellen -▁beiträgt -▁irregular -,6 -hole -häusern -cend -pie -▁rear -▁merc -▁DR -▁papers -▁Mean -etts -▁CM -▁Ric -▁begangen -▁geein -▁einhalten -▁shot -outs -▁peripher -▁realisieren -▁stroll -▁Stadtteil -▁specialities -▁Abstimmungen -▁postpon -▁radik -▁coverage -▁Aktivität -▁lowest -▁Milosev -▁erinnere -▁eingerichtete -▁convince -▁Kanal -▁Auftraggeber -▁Gelände -▁photograp -▁erlangen -▁Schadens -▁registriert -▁vernünftige -▁Mittelmeerraum -▁Mikro -▁begleiten -▁grenzüberschreitende -▁scrut -▁paradise -▁Kapazitäten -▁artificial -▁Elevator -iate -viert -6-0 -▁dollars -▁mild -▁Dim -▁Pas -▁gam -otyp -unken -▁zielt -anteil -ibiot -▁desirable -ahrung -▁trail -itteln -▁solved -▁Orts -▁decent -▁Ähn -ereinander -▁Haushalte -▁Trends -▁Arbeitsbedingungen -▁equip -▁Steuers -▁Versorgungs -▁typische -gewalt --cent -▁Darin -▁practically -regungen -prüf -▁potent -▁Höhen -aneben -▁platforms -▁temperatures -▁Zuständigkeiten -▁Beobachter -▁Discover -▁exklusiven -▁beurteilen -▁örtlichen -▁klimatis -▁womit -▁Betroffenen -▁Tuesday -förderung -ucc -baum -▁viable -▁pil -emony -▁kids -▁Cold -▁Rac -ioned -atus -asste -▁mithilfe -▁Siehe -urcht -▁Spy -▁Griff -ruhe -▁crash -▁crops -▁Excel -ößt --Politik -▁Palm -▁blame -▁independ -▁Obviously --Leist -▁specifications -oberfläche -rechtlichen -apped -▁interessieren -▁Kunststoff -▁workshop -▁kritisiert -▁Prävention -▁Verhaltens -▁occupied -▁expanding -▁Soweit -▁Plenars -lossene -▁vernünftig -ocktails -▁gewisser -▁paradox -▁verbesserte -▁breathtaking -▁grundlegender -riffe -Tra -38 -NE -▁rust -▁mas -▁SCH -iry -▁lovers -▁Ci -▁Has -▁angel -ento -▁Ni -▁erad -verfahrens -▁aufricht -ego -▁Anreise -▁Eric -▁Parks -▁Alc -▁Review -alteten -▁transformation -▁bedauern -▁Entdecken -▁assured -▁76 -▁score -▁viewed -▁beauf -▁choosing -party -▁treaties -▁Iranian -▁falschen -▁describes -▁Active -▁laser -▁emerge -▁Entspannen -▁Communication -▁Zufriedenheit -▁disappointed -▁Lehre -▁Gemüse -▁Similarly -▁Vorredner -verschmutzung -o- -yen -▁IA -▁Serien -▁Fat -▁Göt -star -lywood -▁Inspekt -▁extr -ollah -▁ausführen -▁enchant -▁part-s -▁3000 -ektionen -aki -coni -steiger -▁Lehren -fahrzeuge -umenten -▁undermine -ilton -▁explains -▁receives -▁performances -▁Sales -▁Besuchern -bestände -▁Realisierung -▁Generaldir -▁Metern -▁herrliche -▁Tempo -▁COD -▁ratifiziert -▁schlechten --run -▁größerer -▁Präz -▁comparable -▁throw -helfen -analyse -▁zunehmende -▁Erwachsenen -▁erstaun -▁Ministerrat -▁Gipfeltreffen -brea -lieder -▁tight -▁rand -▁astr -▁Sher -▁Sau -▁DIE -▁Ara -onsten -omod -▁kenn -▁Certain -▁RI -▁Gol -atics -▁preg -eregel -ersatz -▁schreibt -▁vorhin -▁Erste -▁interakt -▁Leicht -▁konkur -▁Überarbeitung -▁Gebur -azin -▁Isra -▁angefangen -▁detect -▁globaler -▁technologischen -▁Folter -criptions -▁faire -▁biological -▁harmony -▁Organization -▁Kohle -▁memor -▁Broad -▁tätigen -▁einzuhalten -.2009 -▁thoroughly -▁Bewegungen -kapazität -▁unzureichend -▁Fuerte -ns -eres -berichte -▁befried -▁strange -generation -▁überd -ssicherheit -▁Ausnahm -▁Abstand -eils -▁commend -▁gesonder -staben -liness -▁develops -▁Marine -akten -▁Amaz -▁Marktwirtschaft -▁Techniken -▁voters -▁Guant --Länder -▁ansonsten -▁performing -▁Carr -▁erreichte -▁Instrumenten -▁Editor -▁Integrations -ECT -▁umfassender -▁honor -▁enlarged -ACK -▁Gestatten -▁unerlässlich -▁Beratungs -▁graphic -▁hairdr -▁vielfältigen -▁Taxi -opolitan -▁kurzfristig -▁Gepäckraum -▁Features -▁enttäuscht -▁Lanzarote -▁scenery -,2 -sq -medi -▁salt -▁I. -▁DI -▁zer -▁Kit -▁Hosp -▁Gulf -▁unbe -▁derartigen -▁rect -▁Zubehör -▁desert -indlich -einkommen -▁frucht -▁Schott -▁ausz -▁Ausge -ussen -▁grös -ährung -▁zwingen -ordnungs -▁zweck -▁irische -▁verwaltet -▁sensit -▁defining -▁settled -▁bevorstehenden -▁Nature -▁talked -zuschließen -▁Spezialisten -▁teacher -▁substantially -▁Angriffe -▁dauerhaft -greifende -▁meantime -athedrale -▁Schnee -▁Additionally -▁Sevilla -▁incredible -▁preliminary -▁avoided -wich -▁turb -▁bag -▁bien -▁Map -▁cease -▁kulin -▁Gift -aro -▁Layout -reformen -▁sees -▁frisch -▁frank -ausenden -▁unterhalten -▁finest -▁gesammelt -▁Ortschaft -▁inclusive -▁Trink -▁funded -uchtung -▁Short -sparungen -▁verwöhnen -▁Lebensmitteln -▁cheese -▁swift -▁Freiwill -▁erzielten -▁emphasised -▁Simb -▁fehler -▁Kompromiß -▁Kreditkarten -▁partnerships -▁incorrect -▁meistens -▁centrally -▁photography -▁Communist -▁Vorsitzender -▁Division -▁sweet -▁erläutert -▁bekräftigt -agentur -▁Total -▁Zimbabwe -bearbeit -gert -▁feas -▁SY -igs -ofür -▁ce -▁cat -▁kre -▁unbeg -anschluss -quest -▁Intr -ingham -▁weib -▁jew -rock -▁Stell -▁Anregungen -etze -ätigen -akings -▁Aufforderung -▁Parliamentary -everkehr -▁screens -▁monat --Med -▁klären -▁signature -▁sensor -boot -▁Benj -▁Speed -▁testen -▁1975 -▁namens -▁Colomb -▁loved -▁südlich -▁Superior -▁Verkäufer -▁Stabilitätspakt -▁mündlichen -▁Wolfgang -▁begeistert -▁Advanced -▁Weiterbildung -▁Lebanon -▁Israels -zo -vens -▁fung -▁aux -▁Dich -▁piano -▁god -▁CAM -▁Fas -▁Fab -▁Rande -oping -schwe -tends -▁aufgel -▁joy -▁Ok -▁Along -akis -befinden -▁agent -▁Aufnahmen -▁trem -▁interinstitutional -▁71 -▁angehören -▁neglect -▁Lösungs -iebt -▁teilgenommen -legte -▁Hotelsafe -▁Verantwortlichen -▁devast -▁consequently -▁Jugendher -▁Everyone -▁worthy -▁Iraqi -▁birds -▁registrieren -personal -▁Tschechischen -▁Jackson -▁Optimierung -▁tribute -▁Aktualisierung -▁Regulierungs -▁Moskau -▁Unterdrückung -eigen -sm -TEN -bele --tra -lor -UC -öte -▁hielt -▁rivers -▁Bull -▁Wellen -▁GS -alian -▁angeben -▁NH -abet -▁Stick -▁shoul -▁span -geräten -acken -▁solide -▁Flag -▁Whirlpool -▁Monats -rechnung -▁Quell -▁Darauf -▁traditionell -▁Lebensqualität -▁Feind -▁Aires -▁nonetheless -▁expressing -▁1982 -ELL -▁Danke -▁Socialists -▁Reaktionen -▁freundlichen -▁Help -▁entscheidet -▁celebrate -▁strenge -▁keinesfalls -▁sincer -▁Medikament -▁exchanges -▁Milosevic -▁vorteil -▁Tos -▁BL -utze -usa -▁Wales -▁Kant -▁CV -▁Hart -▁Gü -▁forese -▁anzeigen -▁UC -ständen -▁aufrechter -▁mitgeteilt -▁Jeden -rease -▁spiegelt -enna -ateur -astet -▁About -▁herstellen -▁mineral -▁drücken -▁gelö -▁measured -▁arbeits -▁raises -▁Subventionen -▁Juden -▁winds -datei -▁Einzelnen -▁sporting -▁automation -▁controlling -▁Mercedes -▁acquire -▁Queen -▁paintings -▁Schattenberichterst -imensional -▁pedestr -▁aspir -▁Gewerkschaften -▁Litauen -▁controversial -▁Buenos -PR -éd -▁tor -▁hors -▁MD -▁Aust -eli -▁Heating -leister -lebens -agten -mert -ueber -▁admitted -▁hiermit -▁guitar -▁sonn -geschneider -▁standen -▁Vereinigung -▁Round -▁statistical -▁confusion --Dollar -▁schließe -gedeckt -▁Compet -▁1.2 -▁disturb -schutzes -▁peak -▁idealen --Hour -▁vereinfachen -▁Declar -▁discrimin -▁cosmet -▁conveniently -▁gegenseitige -▁declarations -▁Oslo -▁isolated -▁capture -▁ausgelöst -▁Legislativ -▁Victoria -▁battery -▁на --Format -▁Mazedonien -hot -▁sail -▁wages -▁MID -ilia -▁Hence -assten -amo -▁Lido -▁Luggage -rection -ichtungs -▁negl -▁spüren -▁Ernen -▁Speich -▁compare -▁finances -▁diskrimin -▁emission -▁propriet -eopard -▁themes -▁bedenken -▁crystal -kredit -▁formula -▁angehen -ständigkeit -▁deutscher -▁schwarz -▁tempt --19 -▁jeweilige -▁relevanten -▁INTER -▁Anwendungsbereich -▁licence -sprochen -▁fears -▁watching -▁Einschränkung -▁executed -▁Gerichtshofs -▁Cultural -▁Amtszeit -▁Wartung -▁Zertifik -▁irgendwelche -▁Echtzeit -rägen -uan -zert -üten -44 -▁dust -▁Tap -▁Sech -▁pact -itudes -uner -▁Hör -icop -▁Null -▁denk -agger -anna -ocratic -▁shame -▁schützt -▁Spuren -▁genetic -▁zwanzig -venth -▁damaged -▁incur -öffnete -▁geltend -▁parameter -▁getest -▁24-Hour -▁Guinea -▁lesson -▁regulate -▁Verträgen -ija -▁analysieren -▁drafting -▁Protokolls -▁undertakings -etooth -▁Stadium -▁belonging -▁ignoriert -▁Ltd. -▁acknowledged -▁nutzbar -▁Beauty -▁Zimmerservice -▁REACH -feit -dokument -HR -▁fate -▁setzte -▁Ecke -▁gingen -▁Bom -ols -unately -▁eure -▁Chal -▁unpre -anes -▁Luk -▁Liquid -▁zukunft -▁deinen -▁jeopard -▁chairs -▁auswärtige -▁spots -▁Ein- -▁Vorsicht -flüss -▁gesunden -▁este -insula -▁Palazzo -▁bewaff -▁multimedia -▁UND -ifizierungs -▁unverzicht -▁einzelner -▁wenigstens -verband -▁1968 -▁Provence -▁engineers -▁Columb -▁tells -▁Wohlbefinden -▁ignored -▁Erkenntnisse -▁hoped -▁erneuerbare -▁schrieb -▁Macedonia -▁Kinnock -kont -▁row -▁kauf -▁Kohlen -▁Cong -▁inkl -deb -stritten -▁Inv -▁exert -▁abund -ensible -ewall -aca -▁beider -aks -gebühr -altete -arbeits -▁Übersetzungen -▁begab -ocken -▁Blizz -▁equity -fassungs -▁Beton -▁Display -▁challenging -▁Fernseher -▁1976 -▁volunte -▁characterized -▁kompletten -▁eingerichteten -▁gefährlichen -▁gefährliche -▁Summer -▁Kohärenz -vorsitzen -▁kritisch -▁Palestine -▁diplom -▁Kindes -▁Ursprung -▁fulfilled -▁prohibited -▁Konsultation -▁genehmigt -▁Credit -▁Migrations -oard -▁hüb -▁fünft -▁squ -▁wartet -▁wöch -▁Ehren -▁gall -▁gele -▁graf -▁Bisher -▁lif -▁Kult -▁Klin -▁Fift -▁Ron -▁Viol -atile -▁congress -endous -▁exklusive -▁Ausstellungen -igerweise -▁internationales -▁Marb -ancies -demokraten -▁Finance -▁scandal -comings -▁flowers -▁modest -zugreifen -▁Close -▁mirror -ilitation -▁Bewertungen -▁Spanisch -▁downloading -▁Eingabe -▁strafrecht -▁Einwohnern -▁branches -▁verschiedensten -▁incentive -▁Clinton -katastrophen -▁übrig -▁Tragödie -▁environments --hand -QU -▁nennt -▁fought -▁TNG -▁pin -eare -emma -oro -▁prere -▁2,5 -▁Jak -rooms -▁demographic -▁Thai -format -schloss -airo -▁endet -▁Weltbank --Mittel -▁brows -▁Manhatt -▁erwart -▁zurückkehren -▁149 -▁Computers -▁thoughts -▁offenkund -▁sheets -▁login -▁äußerster -ibraries -▁warming -▁Gesprächen -▁regrettable -▁riesige -kontrollen -▁ökologische -▁ungeachtet -▁unlike -▁engagiert -▁Products -▁brutal -▁Erweiterungs -▁LateRooms.com -▁mixt -▁Chicago -▁Prestige -▁hazard -.6 -TH -kind -AY -▁Matt -▁Avenue -▁zert -▁Rollen -▁unfortun -atin -▁Lom -iferation -abeth -▁übersetzt -izenzen -▁schol -▁Spar -ärt -▁embry -isms -elson -▁guidance -▁Herberge -▁publicly -▁Hinzu -höhe -versicher -▁Quick -▁Geräten -▁excellence -▁blank -▁shortcomings -▁Christen -▁arbeitete -▁foods -▁Entwicklungszusammenarbeit -ORE -▁Privacy -▁translations -▁complexity -▁twel -unkel -▁Moderne -▁abgehalten -▁condemned -▁Army -▁survival -▁längst -▁bureaucratic -▁Verstärkung -,3 -WE -▁trennen -▁Sac -eaking -omaten -▁Wäsch -▁Cro -▁Rav -▁VI -▁stem -▁sech -cket -▁neol -▁Ora -ehler -▁notion -▁Auslegung -▁outset -▁alleged -▁socio -▁einziges -▁86 -▁Manche -▁gemessen -▁Rahmenprogramm -▁founding -.com. -▁markt -wärmung -▁structured -▁Charakt -▁Norway -▁circuit -▁Holy -▁monitored -dateien -▁kritische -▁Budgetary -▁ethnischen -▁Ratsvorsitzes -▁benachricht -▁umweltfreund -▁zunehmenden -▁Horiz -▁Sozialdemokraten -irtschaftung -▁gefällt -▁Entertainment -▁Libanon -messe -messer -kov -Gu -▁gren -etus -etime -igter -onenten -▁lad -▁lohnt -▁Gumm -▁thread -▁LD -idges -▁styles -▁Indem -ifi -▁aufweisen -▁einbe -▁shut -ätter -▁Vorbild -aufgaben -▁guilt -▁geleg -▁accepts -▁Climate -ähern -▁Finanzmittel -▁wirkungs -▁Dienstag -zusprechen -▁returning -freiheiten -renzung -▁Color -▁vorherigen -▁vorherige -▁subsidiary -▁klassische -▁embra -▁beträchtliche -▁Alpine -▁Indonesia -▁transatlantic -▁Anhänger -▁conjunction -▁abwechslungs -▁Declaration -right -wen -▁suff -▁muy -elen -▁Horn -▁Ri -▁inquiry -icas -▁angesch -▁reverse -aba -▁verp -▁verhalten -agl -ogan -▁Aufwand -meid -▁realized -▁invitation -▁Seb -▁beschäd -anders -▁optical -athon -▁votre -▁Betrugs --Cl -▁checking -▁starkes -▁follow-up -▁Statistik -▁lacking -▁Korean -▁Silber -▁Wahrnehm -▁sharp -▁rejection -▁kooper -ägigen -▁Nonetheless -▁gegenseitig -▁Seminar -▁autonomy -▁diagnost -▁immense -▁durchgeführten -▁overwhelm -▁Rechenschaft -äußerung -oire -damer -bten -kürz -öpf -Fi -▁tact -▁SIM -▁pione -▁WM -aner -schra -▁denselben -agert -▁demonst -▁Ausfahrt -▁Eintrag -▁Algar -▁3.5 -▁traur -itären -▁Steu -▁Medical -▁Touch -▁ernste --Distribution -▁searching -▁departments -▁Ostsee -▁incorporate -binsel -▁Schiffs -▁verehrten -▁üblich -▁Fertigung -bringt -▁survive -▁hilfs -▁zuverlässig -▁GBP -▁Lokal -▁Konfiguration -▁Leidenschaft -▁niederländischen -▁Clearly -▁gastronom -▁torture -▁Slowenien -▁Entstehung -▁unfortunate -dos -uhr -mischung -borg -kord -▁sy -▁sul -▁codes -▁Kel -▁Kra -▁Kalender -▁CA -▁Vil -alp -▁anx -▁anwend -▁angere -▁aufgehoben -adium -▁upcoming -aterie -rikt -▁guides -▁dress -▁delicate -▁Neub -▁governed -clusive -▁Geschäftsbedingungen -▁bekanntesten -AST -▁Flugzeuge -▁separately -▁sämtlichen -▁Initiativ -▁Sometimes -▁Potential -▁Absichten -▁wirksamen -basis -▁amtierender -▁zeichnen -▁abandoned -▁tailored -▁Devisen -▁aggressive -▁Ernährungs -▁Eurostat -▁Palästina -▁Siena -bro -kong -28 -LR -▁agr -▁T. -▁Age -eling -▁lights -▁derived -entren -rey -rade -rage -stunden -ernet -▁verhandeln -ears -izität -▁vorzusch -▁unteren -▁Spons -emberg -▁Plast -hafen -▁striking -▁rede -▁reduces -▁feels -ivileg -▁defects -▁Transaktionen -▁zahlreicher -▁Comput -▁sofa -▁Peru -▁hosts -▁characterised -▁Oberflächen -▁Mittler -▁exceptions -▁Grande -▁Architekten -▁Flüchtlings -▁vegetables -▁Albania -▁tolerance -▁enthusi -▁verständlich -▁seminars -SD -kw -kest -▁tat -▁tiss -▁pine -▁MT -▁Bast -erhalten -▁G8 -dez -▁yellow -▁Lav -▁Love -▁Lärm -streit -▁220 -vertra -▁Verhältnisse -▁Anruf -einheit -ikalien -ourismus -▁Vorstand -iously --Seite -▁setup -leih -▁exploring -▁Working -automat -▁Guard -▁glaubt -▁loud -▁akzeptabel -▁penalties -▁Harbour -▁simplify -▁innoc -▁celebrated -▁vergleichen -▁Perspektiven -▁ältesten -▁taught -▁Ernährung -▁beeinträchtigt -▁trademarks -▁discipline -▁Bruder -▁Niemand -o. -26 -▁fut -▁rar -▁abend -▁sont -▁wofür -▁DM -isol -onischen -▁Kaz -▁Kathedrale -▁CC -▁HOTEL -aman -▁Lust -terr -▁Anweisungen -▁absch -▁ausgelegt -▁wording -▁Parallel -▁Eintritt -ignano -▁regimes -▁grants -▁indication -▁Bedingung -▁einzube --Be -▁boom -▁binden -▁receipt -▁absor -▁Koordination -▁idealer -▁Tourist -▁Carib -▁Textes -zentrums -▁biotechn -▁automated -▁längere -▁Temperaturen -▁Verzögerung -▁jahrel -▁recycling -▁portugiesischen -▁überwiegend -▁Benjamin -wäl -▁tub -▁telling -▁vend -▁m. -▁E-C -▁BC -emissionen -chi -ifer -platten -▁Schre -▁chocolate -rane -▁Einfuhr -▁Spam -▁Above -bey -strichen -▁Mars -▁61 -▁Haust -▁93 -laubnis -▁gefangen -leiht -▁sicherer -otti -▁175 -▁accepting -zufinden -▁Karel -▁Norwegen -▁schönes -▁circle -▁lebenden -▁Building -▁auszuüben -▁befürworte -▁eingereichten -▁Religions -▁DS9 -▁friedlichen -stätte -▁racism --spec -disziplin -▁Minimum -▁Currently -▁einzubeziehen -ière -sinn -ücher --economic -kal -▁hr -▁dish -▁greifen -▁corn -▁Fine -▁Ress -▁LA -schalter -▁Ansprech -endigung -▁Scha -▁coinc -▁Abonn -▁interaction -▁Rechtsrahmen -▁editing -▁Shuttle -▁Berichterstattern -▁Haushaltsplans --Ländern -▁broch -▁einzelstaatlichen -▁beitreten -▁vernachläss -▁Kurse -▁width -▁preferences -▁Ereignissen -▁quota -▁Doppelbett -▁konserv -▁officer -▁Limited -▁berufliche -▁beseitigt -▁Cathol -▁Naturally -▁configure -▁glücklich -▁Fragestunde -▁Gefangenen -▁verankert -▁Sowjetunion -▁landscapes -▁cheaper -▁д -SL -cul -BER -▁Mi -▁Pakt -▁Hass -▁Lied -▁denied -verz -weichen -▁'d -▁durchsetzen -▁trails -▁Grill -▁zwing -▁zwangs -▁Gefü -▁interim -▁continually -▁modes -▁ensured -▁responses -▁Ökonom -▁schlimmer -▁bewundern -▁sensib -▁Fehlen -▁Euror -▁Deluxe -▁drafted -▁Magic -▁Danube -▁badly -▁undertaking -▁Tschetschen -▁Besonderheiten -▁Origin -▁krimin -fuhren -▁Randlage -▁Navigation -▁navig -▁einzigartig -iesslich -▁Bluetooth -▁Komplettpreise -▁shareholders -▁Marbella -▁Manhattan -design -uay -bot -NS -ре -▁tur -▁dell -orate -▁Wörter -▁Future -▁unlimited -▁Lodge -▁stood -▁Stay -▁schlage -▁Erwer -▁unterliegt -▁EU-Ebene -blas -▁superm -▁embedded -▁Heritage -▁Zusammenbruch -▁reductions -▁128 -▁long- -▁combining -▁Buchungs -▁sustained -▁minds -▁Wahrschein -▁e.V. -▁Bali -▁zerstören -▁assumed -▁doctors -▁reinforce -▁API -▁Internetseite -▁abzielen -▁personenbezogenen -▁infringement -▁Analysen -kirche -▁Vogel -▁verwirkl -▁reiterate -▁imprison -▁Stefan -▁walks -▁iz -oque -fa --by -p. -▁dos -▁diox -▁missions -▁bio -▁MPEG -▁PM -▁oste -▁Heat -racies -stark -keeping -▁reine -abschluss -▁diesmal -mengen -▁vorbe -ianische -▁evil -▁contest -▁gleicher -▁bescheiden -▁Freien -olidierung -▁16th -▁verstoßen -▁Hierbei -▁multipl -▁Allein -imini -▁Gründer -▁Tatsachen -▁Crist -▁circulation -wohnt -opolis -IVE -▁Strecken -▁quantities -▁irgendeiner -lechts -▁Felsen -▁Auseinandersetzung -▁malerischen -▁Einfluß -▁Opfern -dest -reme -wunsch -lets -▁tension -▁tiny -▁hire -▁finger -▁Tol -▁IL -▁Divers -▁ME -▁Could -▁Castr -▁Gelegenheiten -asma -▁Lande -▁Zuer -▁errichten -tein -▁gekauft -▁spont -▁Befrei -▁Spek -▁Berlus -▁zumal -ieweit -aktions -othic -▁deleted -▁debatt -▁requiring -▁Montr -▁boards -▁Finanzmärkte -▁zusammenzuarbeiten -▁stehende -▁participated -▁Gegenstände -betreuung -▁Hunger -▁Völk -▁isolation -▁ausgewogenen -▁fuels -▁EVP -▁Assoziierungs -▁Tendenz -gerichtshof -▁random -warten --12 -kin -го -▁doppelt -▁rasche -▁Softw -▁Piet -asa -▁derog -▁Lul -schten -▁verleiht -änn -obyl -▁Beendigung -spflicht -▁doubl -gery -▁interinstitution -▁consistency -▁underline -▁Frühl -▁belegt -▁assembl -▁Kommissare -▁Lae -▁explos -▁defending -regierungs --Vor -▁Partnerschaften -▁Entschließungen -ächtnis -▁Salv -▁weißen -▁moderate -▁Untersuchungs -▁erschienen -▁simplification -▁stärkeren -▁adequ -▁calculated -▁Schmid -▁SSL -▁Peking -▁transformed -▁nördlichen -▁Stattdessen -▁einzurichten -▁nic -▁Sak -▁DH -▁Rak -▁thin -▁pron -▁Nan -ifiable -▁concess -endem -▁Schienen -▁daughter -▁vorsichtig -ssektor -▁Maison -▁bereitz -▁Deine -▁Quer -▁struggl -▁geladen -▁continental -▁measur -ulevard -▁startet -▁besetzt -▁Elite --Team --Technologie -▁Zahlreiche -▁klarer -▁motions -▁Fundament -▁grundlegend -▁consisting -▁resorts -▁listened -▁Annäher -▁boxes -▁Outlook -schuldigen -▁Anreiz -▁vielfältige -▁newspapers -▁japanischen -▁Verzögerungen -▁enemy -▁Eurex -.jpg -▁aufrechterhalten -▁Algarve -harmon --level -▁nos -▁aesthet -▁vierten -▁Treiber -▁gäbe -▁Rosen -▁Rüst -▁Vot -iewer -▁alliance -aded -eart -▁ausb -▁untergraben -aciones -otherap -▁besticht -▁4-Sterne-Hotel -gespr -rio -▁Üb -▁Überrasch -▁bases -▁endless -▁handic -▁94 -▁Genf -▁presentations -▁commonly -▁large-scale -ässigen -▁Industriel -▁1981 -▁identical -▁Nichtregierungs -cribed -▁announcement -▁concentrated -▁Inneres -▁investigate -▁sightseeing -▁clothes -▁fabric -▁culinary -▁Jugoslawien -▁wünschenswert -▁Ankündigung -▁Sorgfalt -tains -fig -wieweit -kok -Pro -OX -те -▁Sz -▁ov -▁Kup -▁Hell -estone -▁geg -▁Week -▁besteh -▁Dekor -▁Herange -▁Kloster -▁1500 -▁Arbeitsweise -holung -▁Autover -▁gebilligt -gewinn -▁Formel -▁einmalige -▁respecting -▁Werbe -▁homepage -▁schnelles -▁plane -▁neueste -▁unverb -▁weekly -▁crossing -▁Micro --Stand -▁Local -streicht -▁Vorbereitungen -▁anbietet -▁Libya -▁officers -▁Begriffe -status -▁Übersetzungs -ungalows -▁Virtual -▁vicinity -▁bereitzustellen -lements -▁hus -▁seal -▁gäng -▁Nep -▁repress -ernobyl -▁geholfen -igende -▁maj -▁Regimes -▁freier -▁Reb -elev -entrums -mitting -▁ending -▁cargo --Auf -▁Freie -▁gefasst -▁retir -▁corpor -▁districts -▁colonial -▁verschieben -▁verschoben --15 -▁vereinfacht -schnitts -▁publisher -▁furnish -▁Reader -beteil -▁connecting -▁penal -▁Senior -▁permanently -▁zukünftig -▁Granada -▁vertrauen -▁humans -▁gesellschaftliche -▁konstruktiven -▁Stabilitäts- -▁Henry -▁Namens -▁Reihenfolge -▁Implementierung -▁enthusiasm -map -KS -▁nament -▁bond -▁banned -▁Sus -▁SAT -▁EMU -▁grew -▁ASP -▁cens -owed -▁habt -▁refined -roökonom -▁Anzeigen -▁gesunde -cedented -▁Flüge -▁87 -▁drängen -▁prejud -▁restoration -▁begreifen -▁attent -▁führten -▁Heidel -▁Tagungs -▁optimum -▁Christdemokraten -bestimmung -▁analyses -▁Königs -ministerium -▁discretion -▁pursuing -▁Halbinsel -▁S.A. -▁beliebten -▁dominant -▁breites -▁Vorsitzende -▁strengen -▁Kommentar -▁nützliche -▁CPU -▁Imperial -▁suddenly -▁Etikett -▁complimentary -rem -▁harte -▁rhetor -▁Muss -▁Bass -▁Cos -▁Giov -▁forb -▁NT -▁Zer -▁ermitteln -abstimmung -▁geopolit -agna -griff -▁Abgesehen -▁Vorwand -checken -▁perpet -▁provin -▁Berech -▁Bremen -▁discovery -▁booth -▁benannt -▁specialised -▁passes -▁Transit -frau -▁EURO -▁continuation -▁vorgeschrieben -▁1973 -▁Directors -▁camping -▁separated -▁weapon -▁Frühstücks -▁biologischen -▁Straßenverkehr -▁adaptation -▁exploit -▁observers -▁verschwinden -▁junger -▁Monitoring -▁Reichtum -▁zugeschnitten --century -▁unprecedented -üch -äuter -GH -▁müsse -▁TAC -▁Pul -▁Blo -itably -▁luggage -▁unreal -amination -▁exempl -ikely -merzen -▁worauf -▁adher -▁accused -▁mehrerer -▁Impro -▁Krem -▁Weltmarkt -▁knowing -▁mainstream -ulationen -▁verliert -konstrukt -▁Durchschnitt -▁Darle -▁17th -▁Folglich -iaries -▁1983 -▁Integrität -▁ambience -▁analysiert --house -▁Golfplatz -ILL -▁unmittelbaren -▁Beachten -▁tackling -▁Segel -bilder -▁türkische -sprachigen -▁Impulse -▁luxuriöse -▁belgischen -▁Gastfreundschaft -▁part-session -runde -auten -wing -4-0 -▁filling -▁migrants -▁Pant -▁gains -▁F. -▁Fratt -umpen -▁annex -▁alive -opher -hrter -▁versprochen -▁jump -undet -größe -▁werte -▁vorne -▁Erleben -ensystem -oked -▁Beweise -▁1920 -▁interven -▁roy -attformen -▁Menschenhandel -▁Seeverkehr -▁mining -▁89 -▁Trennung -▁implies -▁Hierzu -▁Zwecken -wärtig -.11 -ifikationen -▁Defizit -▁Mailand -▁Jahrzehnte -▁Absol -▁Quadrat -▁Singapore -▁Split -▁verknüpft -▁linguistic -▁Meanwhile -gon -▁tiefer -▁hide -▁fing -▁dun -▁sink -▁más -▁SOL -▁pull -▁Masse -▁Pur -otype -ussten -▁CAD -▁RS -▁Grab -uma -▁Jet -ields -▁ausgebaut -▁vorgegeben -▁Erika -errt -▁unterstreicht -ünder -▁Große -ängst -▁securities -ERS -▁Bewerb -iefst -▁Saf -zustand -▁failing -▁1974 -heimer -würdige -sbruck -▁adjusted -vielfalt -▁Albanien -▁Grafiken -▁Ärzte -▁unvergesslichen -▁warranty -▁ungarischen -▁air-conditioned -▁müßte -▁Präsidium -▁Caribbean -▁ul -bte -69 -▁roles -▁rhyth -▁Torr -▁Dial -▁Base -omed -▁lifestyle -icular -▁anstelle -▁Ll -▁Nar -stuffs -▁Verona -▁Auslands -flüge -▁warmen -▁Aufstieg -▁mehrmals -▁69 -▁strive -▁weitergegeben -▁gescheit -▁versetzt -brä --Mar -beweg -▁feiern -▁backing -▁Palast -▁XL -▁contributes -▁Liebl -▁demonstration -▁congratulations -tenteils -▁hydraul -▁Annual -▁dauerhafte -▁earthqu -bilanz -▁consolidation -▁ignorieren -▁Campingplatz -▁abolition -▁Südtirol -▁refurbished -▁twelve -▁eating -mond -winkel -öpfung -Net -GA -▁tape -▁Seat -▁Way -▁Vall -▁Graph -ulates -stabilität -ifff -▁geklärt -▁Jagd -▁noticed -▁frischen -▁Theatre -▁sozusagen -igebiet -▁Gegeben -▁gewann -▁decreas -▁Exporte -▁clearer -▁flags -▁economics -rada -▁Zugleich -partei -▁Basil -▁Co. -▁wenigsten -▁demanded -arios -jähriger -▁Standpunkte -▁railways -▁obtaining -▁effektiven -▁EGF -greifenden -▁Krankenhaus -▁untereinander -▁arrested -▁verheer -▁tangible -▁unabding -▁Frattini -.- -mor -listen -PL -▁SIS -▁Did -▁Delta -▁Räum -▁Letz -estern -iedelt -▁notification -▁desper -▁ums -▁ausgearbeitet -▁nachvoll -epp -▁Forst -▁indisp -▁systematisch -▁Allgemeine -▁Grundfreiheiten -▁appart -▁Manchmal -▁besagt -▁competitors -▁bereitstellen -▁monthly -▁getrenn -▁influenced -▁Multimedia -▁Akzept -meldungen -▁Effekte -▁friedliche -▁unlikely -▁fifteen -▁landwirtschaftliche -▁außergewöhnliche -▁zutiefst -▁estimates -▁Expert -▁mündliche -▁gezielt -▁bedeutendsten -▁Pierre -▁roughly -▁Legitimität -▁Mugabe -▁Vermeidung -fit -mt --gr -Con -▁tausend -▁merg -▁Tener -▁Much -▁cinema -▁Kernen -imir -idity -herz -▁gep -adas -assage -▁Erreichen -oba -▁Abd -▁Fron -▁wozu -▁Kluft -▁Hinsichtlich -▁benchmark -▁skill --Ge -▁Forscher -astery -▁Kommando -▁besseres -▁schwerer -▁Motors -▁wunderbare -▁Informationsgesellschaft -▁missed -▁Armuts -▁Innsbruck -figuration -▁Leistungsfähigkeit -▁Effektivität -▁Bananen -▁Virgin -▁Fourth -▁venture -▁distinguished -▁carrier -▁korrigieren -▁umgekehrt -▁Fokus -▁independently -iated -df -▁fabr -▁bonds -▁gig -inas -infrastruktur -▁Wats -▁Kof -▁Cres -▁Rental -▁unilateral -lecht -▁Princ -▁Thun -▁durchgesetzt -aufent -altar -▁fires -▁hierarch -▁gewählte -ißen -▁Arbeitskräfte -▁Freib -▁Toil -▁entspann -▁Spielraum -▁intoler -▁Energieversorgung -▁summar -▁permits -▁Hollywood -▁ambitions -ción -▁E-Mails -▁transe -▁databases -▁SPA -▁mailing -▁Ganzes -▁genügt -▁gründlich -▁gathered -▁grüne -▁venue -änglichkeit -▁Garda -lacier -▁häufiger -▁Rindfleisch -▁ic -▁fotograf -▁rose -▁Distrib -edition -isierungen -▁cope -▁Kamin -▁Rimini -asp -▁forums -▁Loch -▁Lac -denk -▁aufre -▁Jens -anners -▁Mitwirkung -▁Paradies -▁vielfach -▁elabor -ergest -▁Flotte -▁nowadays -▁US-Dollar -stützung -▁Besteller -▁profiles --Ad -▁capability -horn -ölle -▁authorisation -▁intim -▁Guatem -idesh -▁Gleichbehandlung -▁finanzieller -▁Kopie -▁effizienten -▁hinzuweisen -öpfe -▁konsult -vernight -▁inspiration -▁Zulassung -▁Propert -▁Facebook -▁ECHO -▁Laeken -aze -vorteil -äts -gab -49 -▁Má -▁Benz -▁Ahm -▁kunden -▁RAM -▁Nür -▁deny -▁storm -▁stabile -▁deemed -weight -▁Ausser -▁3-star -acent -▁Abfälle -iances -bles -zeitraum -ancell -▁Geschichten -atories -▁größtenteils -▁deutlicher -▁energies -▁furthermore -▁employee -▁intu -▁zahlt -▁labels -interessen -▁musicians -▁lokal -▁Binnenmarkts -spiele --Netz -ützte -▁Estonia -wesens -▁Laman -▁investigations -▁qualifizierte -▁ausgestatteten -▁vorangegangenen -▁Tokyo -▁Tipps -▁Songs -▁Diensten -▁za -▁Fix -ioni -▁Later -▁gefolgt -▁exquisite -▁fruits -▁abgeb -▁Errungen -ahrer -▁Vorgang -▁dispar -ansa -▁zweimal -▁differently -▁instant -▁ausgeglichen -▁comforts -▁optimalen -füllen -▁Jahresbericht -hundert -▁preventive -▁translate -▁Metal -▁vorhandene -▁dangers -▁Elementen -▁Neuigkeiten -▁palace -▁Herzeg -▁removing -▁Dolmet -▁Karls -▁laboratory -▁strikt -▁Argentinien -▁fehlende -▁kommerzielle -▁stakeholders -▁wettbewerbsfähig -▁droht -▁Memory -▁bemerken -spolitischen -▁hectares -▁rescue -▁Wahrscheinlichkeit -ädern -▁Sog -▁Sämt -▁Awards -isi -untern -▁Rückt -▁Lama -estro -hrop -▁Jam -annual -adi -▁Aufträge -▁Diesel -▁Unterhaltungs -▁transmitted -▁Zielsetzung -▁gleichberecht -▁Energiet -▁Pharma -▁authorised -ekut -▁Benach --Gipfel -fähiger -▁steep -▁Winds -verbot -.09 -▁Konsumenten -▁subscribe -▁Luftverkehrs -▁Materials -▁Informations- -▁DVDs -parks -▁Mountains -▁Return -▁Leonardo -anwalt -▁EU-Bür -▁Unionsbürger -▁veröffentlichten -▁Oxford -▁Wurzeln -▁towels -liederung -▁Simbabwe -▁Ernennung --room -▁bow -▁DNA -▁Differenz -▁Ax -▁Veget -▁Lange -▁Zag -▁erlang -träglich -▁execut -annehm -aining -ultural -beitrag -hende -▁apolog -linder -▁Impf -▁Fronte -▁15th -▁awards -▁Monopol -▁revol -loser -▁married -zufolge -▁speaks -▁Sonn -▁hinterlassen -▁zulässig -▁assessments -▁Ratspräsidenten -▁Organen -▁Galax -▁Stellenwert -▁observations -▁Aussichten -▁Tunesien -▁Defizite -▁Möchten -▁kilometers -▁umfassendes -institutionen -▁stringent -▁Persönlichkeit -otheken -▁wedding -▁kompatibel --specific -uest -kommission -öcke -39 -▁winners -▁Pel -▁Petr -onze -onna -▁lens -▁Kul -▁Care -icating -▁stir -osph -uropas -▁Obs -▁umstritten -▁adidas -▁partially -rence -anzler -▁intergovernmental -strom -ransfer -▁Parlamenten -änderungs -▁numer -▁besides -▁Besicht -▁designers -▁Finanziellen -▁Festung -▁rechtlich -gängen -▁Carlos -ourcing -▁capacities -▁utility -▁Übereinkommens -▁Kurd -▁dominated -▁prevail -▁Territorium -▁Ludwig -▁Tunisia -▁pregn -▁Ausnahmeregel -▁getestet -▁Berlusconi -üsten -▁Tir -▁belegen -▁Burn -eterm -▁catering -alition -▁anmelden -▁pride -▁Label -▁dens -assador -ipper -▁vorschlägt -▁leichten -▁Aufrechter -▁acted -▁Fristen -itteil -▁Stränden -▁einzustellen -▁Converter -▁Rather -▁Autonomie -▁Rahmens -▁familien -▁Between -▁Zweif -▁wahrnehmen -fristig -▁Statut -▁getreten -▁cycling -▁appreciation -▁Libyen -▁loading --Protokoll --depth -▁Ultra -▁Bolkest -▁boundaries -▁countless -▁cabin -▁Moldova -▁vorrangig -▁summary -▁nutrition -▁verwirklicht -▁Zuerst -▁Watson -was -▁tone -▁foster -▁dicht -▁sick -▁bull -▁Acr -▁leit -▁Krak -▁Vi -iveaus -ieso -ereg -▁erwe -oso -▁gewerb -▁Anbau -▁Schuh -usted -▁bestellt -amped -neys -gestellte -▁realistisch -▁weiterzu -▁Allow -▁afterwards -▁zusammengef -iebung -▁Spezif -▁feelings -preises -▁younger -▁Pract -bietern -▁Musiker -▁promotes -gerechte -▁Verkehrsmittel -eleistungen -▁ALL -▁supervisory -▁unanimous -▁Mozilla -▁champions -▁Cannes -▁Künstlern -▁Mühe -▁zwangsläufig -▁Frontex -eur -▁ax -▁Tut -▁Machen -▁Wagen -▁R. -▁Run -still -▁struck -▁Prix -▁gepl -▁extends -▁Einheiten -▁unterhalb -▁Abweich -▁Chi -▁gesorgt -▁overt -▁großartige -▁Brust -▁verbieten -▁Interface -▁globalization -▁margin -▁Mehrere --Verb -▁collections -▁consulted -▁Pfad -▁seaside -▁Erwäg -▁deepen -▁safely -▁heruntergeladen -zuteilen -▁Kolumb -▁ceram -▁plugin -▁schwarzen -▁vineyards -▁Dörfer -stärke -▁Everest -▁indigenous -▁korrekte -▁Welthandelsorganisation -▁Wahrnehmung -▁indispens -▁Sämtliche -▁Bolkestein -oh -mund -writ -case -PO -▁pose -▁porn -emit -▁Kok -arb -ürft -▁albeit -▁Lett -▁Leder -stuhl -▁decken -▁Verhin -adian -▁Einladung -▁dismant -fahrung --Spe -klen -▁Grundstück -▁Klick -▁Trop -▁gerufen -▁langjährige --Amer -▁Advent -▁Mittel- -author -▁reaches -▁Drive -▁reale -▁Frankreichs -▁erweiterte -▁regelmäßigen -▁bears -▁ausgedrückt -▁Verpackung -▁Dolomiten -▁Kühlschrank -▁Korrektur -▁ASEAN -▁Exhibition -▁revenues -▁Oracle -▁hilfsbereit -▁Berechnung -▁ueber -findung -days -CR -▁Tony -▁Egal -▁Bert -▁Ce -stunde -▁steckt -▁Zit -oding -ipes -webe -eckten -ografen -sspiel -▁Ausstellungs -▁Einblick -aked -▁EU-A -auert -▁accomod -▁Plant -▁Landschaften -isko -▁invested -konvention -geordneten -▁Industries -▁Gleichheit -▁antiken -▁destabil -▁schöner -▁Nordir -ALL -▁Unserer -▁Generalsekretär -▁conviction -▁Biok -▁Kaffe -▁Chemikalien -▁neighbor -rebungen -▁Konflikten -▁schwedische -▁Anmerkung --Emissionen -anspruchnahme -▁Vermarktung -iben -ng -nutzung -ship -kamera -▁ties -▁mieten -▁Boutique -▁Core -▁Ry -ieger -berto -▁anlässlich -▁anstreben -▁revers -verhältnisse -▁deck -▁verglichen -adet -▁Schokol -lafen -▁vorkommen -▁Alarm -▁query -▁Abwicklung -sebene -▁noisy -phan -▁insight -▁konvention -▁secured -▁defense -▁Fortschritts -▁Sicherheits- -▁Simply -▁Harry -▁Nachfolger -herrschaft -palette -▁Behinderungen -▁Automobilindustrie -iciary -▁tatsächliche -▁Schnittstelle -▁gewöhnlich -▁abzielt -▁Supervisory -ansprüche -▁Sebast -▁Lieblings -aute -KE -▁hole -▁Sit -▁DB -▁Dry -▁Eck -▁Pont -▁Bob -▁Bolog -omer -▁Crew -▁bewirkt -agnet -▁Stimmung -iele -▁ages -▁Chaos -▁Foren -leichen --Sicherheit -▁Nachteil -▁geteilt -tegration -▁Barnier -▁investing --Richtlinie -▁Newsp --Tr -legten -▁Solange -▁villas -▁wünscht -▁Logo -▁sexuellen -▁Graz -▁Texas -▁Tabellen -▁reconciliation -▁Konflikts -IEW -▁motivated -▁parlamentarische -▁greif -▁Substanzen -▁climbing -▁Regarding -▁Wachstumspakt -▁sincerely -▁Aufrechterhaltung -ebene -oil -cm --rea --prod -▁hers -▁dreams -▁rum -▁sat -▁TU -▁beat -▁Surf -▁Hind -▁in-depth -amkeit -▁on-line -▁haven -▁weigh -▁Oh -▁atom -▁spell -▁späten -▁Auskunft -peror -zuheben -äler -▁crimin -▁Brown -▁natur -▁Reden --bu -▁Markus -▁Buchstaben -▁bitter -▁Flughaf -▁Audi -▁beantragen -▁clarification -▁cooling -▁Titan -▁scenario -▁Bangkok -▁genetically -▁PPE-DE-Fraktion -▁beeinträchtigen -▁Johannesburg -▁dictatorship -▁constraints -▁Eleganz -▁Annäherung -hou -uent -▁flaw -▁Sich -▁eB -▁ließen -▁Walter -▁V. -▁ungen -▁Ug -▁Lü -leet -▁zuzu -oboda -▁Ausfuhr -erren -▁betreibt -▁Spenden -▁Vorgänger -aux -▁uphold -▁Entdeckung -▁geschätzt -▁qualities -▁Need -▁befreien -boards -partnern -▁1.1 --Ger -▁Haushaltsjahr -▁schnellst -▁Preis-Leist -▁radiation -▁1977 -▁separation -▁elegance -▁audit -▁beantragt -▁panels -▁Etagen -▁Verhaltensk -▁stimulate -▁reproduction -▁scenes -veranstaltungen -▁Weihnachts -▁eligible -▁Erdbeben -▁Getreide -vet -MB -Mail -▁mould -▁Telef -chnen -▁cup -▁Hilfen -▁Reh -▁Rov -ctic -▁orange -überwach -▁Anzeichen -einheiten -▁athl -▁Erwerb -uda -▁finishing -▁5th -▁Zeitungen -▁realised -▁helicop -▁damaging -▁Brother -▁84 -positions -erstoff -▁Träger -▁Appro -▁persönlicher -▁Sozialpolitik -▁benötigten -schnitte -▁divide -▁Copy -▁jährliche -orientierte -▁quantitative -▁ausführliche -▁somewhere -▁Säule -▁explanations -▁Konvents -▁Syd --Abkommen -▁suspension -▁Ebenfalls -▁CD-ROM -▁pedestrian -zens -cuss -ls -pher -▁hôtel -▁dive -▁abends -▁Sett -▁Mare -▁Breit -install -▁WHO -▁Cors -▁Côte -▁G. -▁Gerne -idal -▁eingeben -ubar -▁wash -▁Einwände -▁Being -▁decentral -▁justification -▁Commander -▁meint -▁lawyers -▁accountability -▁enjoyable -▁klares -▁analyt -▁2.4 -schwemm -gesetzten -▁currencies -▁delayed -▁dogs -▁Jews -▁beliebtesten -▁Lady -▁coalition -▁Workshops -▁seminar -▁islamischen -▁damaligen -▁Entschädigung -▁allmählich -▁Gültigkeit -▁hairdryer -▁indispensable -▁м -wann -prechung -kat -kolle -77 -▁tiefe -▁Suchen -▁Aur -ordern -▁lod -▁Kämp -▁Kammer -▁Fünft -defin -alität -▁anfangen -▁-B -▁Schiffen -asses -▁washing -ausbildung -▁outs -▁heik -▁91 -azione -▁industrielle -▁arriving -▁Airways -▁colon -▁Basic --bit -▁begrüß -▁Canaria -ustainable -▁Zwangs -▁stoppen -▁Gesamtheit -▁symbolic -▁griechische -▁co-operation -▁ökonomischen -.C. -▁Dubai -▁Früchte -hütung -▁Sortiment -▁Umstrukturierung -kompetenz -▁Blizzard -fal -rin -zern -▁nahmen -▁vice -▁Sehen -▁Days -▁gaming -unal -▁commen -▁künd -▁Kub -▁GVO -anien -▁Liv -▁Large -▁Zellen -appropri -ainless -▁shoes -▁wasn -ogo -zeitige -frü -▁74 -▁verbreiten -▁Teilnehmern -▁Barack -▁ausgeübt -▁bekam -▁Phasen -inson -zügen -▁1.4 -▁Chr. -▁marriage -geordnet -▁genauen -▁Verhandlung -▁passenden -▁Vorsorge -▁limitations -▁Torre -▁infrastructures -▁attempted -▁ausgewogene -▁vessel -▁proportional -▁Baroness -▁haftet -iors -▁Toleranz -▁SEO -▁Mong -▁greet -onas -unn -chung -▁Funk -▁anger -schäden -▁dez -rost -▁Starf -aller -▁4th -▁invites -▁Überw -weiterung -▁festgehalten -▁designated -▁Finanz- -tensive -fassungen -IND -▁featured -igrant -sky -▁Christians -systemen -▁delivering -▁Affili -▁latein -▁Leser -▁architekt -▁Höhepunkt -▁Application -▁Sofern -▁Fertigungs --Regierung -▁earned -▁koordinieren -▁await -▁bekräftigen -▁makroökonom -▁Nachbarschaftspolitik -▁inevitable -▁opponents -▁dänischen -▁gelöscht -ów -▁tolle -▁meter -▁Möglich -▁curt -▁Räumen -▁beweist -▁NO -▁NCC -▁ont -owder -▁Inanspruchnahme -▁geänder -▁geforder -▁Unlike -▁Annex -ranational -▁gründen -▁Channel -älen -▁actor -landes -▁gewachsen -▁Kredite -ools -▁socially -▁Datenbanken -scheidung --suite -▁menus -▁touristischen -▁Netzes -▁Augenmerk -▁avoiding -strecken -▁Völkern -▁...... -▁Erdöl -▁gesetzlich -▁vernünftigen -▁embark -wissenschaften -▁qualitative -▁Merkel -▁advocate -▁Indonesien -▁hypoc -▁EWG -ströme -▁Nichtregierungsorganisationen -sum -master -▁dumping -▁pi -▁Pos -char -chens -▁zog -▁H. -▁Vere -▁Gibr -▁Girl -istern -hrer -acity -▁outlined -▁intermedi -affin -ondo -▁immerhin -▁Handt -▁flour -▁printer -▁promising -▁Formul -▁Beta -▁hopefully -▁Save -▁Romant -▁criter -ifizierte -▁Myth -▁beautifully -verlust -▁subscription -▁verstärkten -▁Signale -▁Kommissionsmitglied -▁vorausgesetzt -▁multilingual -▁robots -▁Dafürhalten -▁gültigen -▁Timor -▁coherence -▁nachzudenken -▁Mozart -▁Begeister -▁veröffentlichte -▁Tschetschenien -▁ot -▁orth -cam -▁tier -▁BI -▁Bib -▁Bayer -▁Busse -▁Corr -▁Cube -▁HR -▁Rational -▁GF -ulden -▁Little -▁Lanka -quipped -lez -▁rehab -▁einbringen -plates -artig -▁allies -ounced -▁legte -▁handed -▁besichtigen -▁impetus -scheinend -▁steuern -▁optimieren -▁Canal -▁erreichten -.05 -▁1969 -▁Tempel -▁decoration -▁Verschm -▁Infos -▁controller -▁Infrastrukturen -▁epidem -▁episode -▁Sofia -▁Nuklear -▁racing -▁religiöse -▁Befürwort -▁östlichen -▁Konzerte -,0 -dung -ль -▁horn -▁fäh -isations -▁libraries -▁Coton -▁Hüg -ulichen -tert -▁gewe -▁OP -ega -▁Ansätze -▁Thea -▁Erleicht -▁Mitbürger -ruhen -fred -▁Gesetzen -▁Brot -bewerber -▁Spielern -▁Vielmehr -haftet -▁existed -▁check-in -ijn -▁Jahrhunderte -öhnung -▁blocked -▁Beförderung -▁Kompromisse -▁Solidaritäts -▁gesetzliche -▁laying -▁misunder -▁Cardass -▁frühere -vielfält -▁Ethernet -befehl -▁qualifizierten -▁legendary -▁Orientierung -▁broadcast -laimed -▁brilliant -▁Kernenergie -works -bollah -▁wand -▁waves -▁bust -▁Diet -▁Bag -etto -ilien -▁killing -▁Vierten -▁Virus -▁unzul -leid -▁reun -esty -establ -▁erfor -▁-A -prim -▁entity -ussy -▁Geber -▁weitem -▁systematically -▁flower -▁Spiels --bas -▁Entwicklungspolitik -▁loan -staatliche -▁grave -OST -▁Ferienhaus -▁suspect -▁anderswo -▁Collection -zzo -▁compatibility -▁Chemie -▁engagieren -▁Rohstoff -aggage -▁PSE-Fraktion -▁disagree -▁hunger -▁gehörte -▁fundamentally -▁Variante -ancellor -bem -▁MR -▁MAN -irr -▁eignen -▁Koll -▁Ray -▁unm -uliert -abot -▁einsetzt -agte -▁Store -halter -▁ausg -enny -▁Einbindung -▁disreg -aufwand -fahrzeug -▁delivers -▁Rechtssicherheit -▁ernannt -▁schwach -▁authorized -▁filters -▁Swoboda -▁installing -▁Disney -▁Insofern -▁Eurocop -▁fünfzehn -▁PCs -▁universe -▁Recy -▁Florida -zweck -▁hinzuzufügen -▁saubere -komplex -▁vorbereiten -▁Souven -▁Internetseiten -▁Victor -▁vorzubereiten -▁bemerkt -▁slaughter -▁inevitably -▁Telekommunikation -▁alpine -datum -mont -bra -ños -▁herein -▁auction -▁Sid -▁PE -▁kos -▁Hut -▁Lift -▁zuge -abteilung -isson -plattform -artikel -nee -▁mysq -handlung -▁forming -▁surplus -▁verläss -▁skilled -▁diam -▁Naturkatastrophen -ächtigungen -▁Solarium -▁fortgeschritten -▁Kosmet -▁Balance -ianti -▁erfassen -▁globally -▁letztes -▁WiFi -▁dauerhaften -▁steigenden -▁maßgeschneider -▁überraschen -▁contradict -▁Artikels -▁Passagiere -▁inequality -▁interfaces -▁PVC -▁geschmackvoll -▁Indikatoren -▁Erneuerung -▁Bologna -▁Cotonou -ić -fix -rücken -üger -▁harten -▁veter -▁german -▁Baus -eservice -▁overnight -▁kid -▁FM -▁Nick -▁Zigar -osur -▁vertrag -anted -merken -▁adap -ailable -▁plate -▁Unterb -phone -▁Haw -▁carries -▁serial -▁opera -▁cost-eff -▁Urban -▁electronics -▁subjected -▁metall -▁identifizieren -▁fortzusetzen --Intern -▁Stellungnahmen -▁günstiger -▁Verfassungsvertrag -▁attraction -▁Environmental -▁calculation -pakete -▁owing -▁Renov -▁Szene -▁CNS -▁verlängert -.The -▁Punta -▁vorzuschlagen -held -oi -roll -mes -bur -güter -▁bol -erklärung -▁Cart -▁Hay -▁ungerecht -umfang -ulieren -▁deeper -öne -udio -▁admission -örp -▁workforce -▁weiten -▁provis -▁fliegen -▁openly -▁Quar -▁activated -▁1.3 -▁Model -▁Gesetzgeber -▁Demokratien -▁Aparthotel -▁Criminal -▁Donau -▁high-speed -▁Bundesrepublik -▁renovierte -büros -▁simplified -▁gradual -▁Fotok -▁englische -▁exceptionally -▁Kreatur -▁Twitter -▁erzählt -▁tranquil -▁tremendous -▁MIDI -▁Ansprechpartner -▁retirement -▁herself -Tech --dr -link -▁hängen -▁sed -▁Bike -enke -iring -ivated -atible -aspekt -▁dereg -▁Zinsen -▁aufgeworfen -rators -▁freige -▁solving -▁locally -▁viewing -▁wiederhole -▁festhalten -▁maintains -▁beacht -ATE -▁wishing -▁Rücksicht -▁Demokratisierung -▁prevents -▁capitalism -würdigen -▁Chechn -▁Termine -▁judges -▁erkennbar -▁hyper -▁symbols -▁ICE -stechnologie -▁Komponente -▁Navig -▁beglückwünsche -quiries -▁ideological -▁Vorbehalte -▁mangelnde -klausel -investitionen -▁Brüder -▁Catholic -▁Starfleet -▁vente -eded -eats -isdom -▁Ruh -▁uncom -▁geometr -▁Verknü -▁dement -▁schmerz -▁resumed -▁newest -folgerung -▁Flach -▁complaint -▁berührt -▁detection -ERV -▁seitdem -iftung -▁Compact -▁procedural -▁portra -▁Geschäftsführer -▁machten -▁2.2 -▁borne -▁Kleidung -belast -▁diversen -▁Linken -▁Volkswagen -▁englisch -▁Zwischenzeit -projekten -netze -▁Boliv -▁Hongkong -▁koordiniert -▁vertrieben -▁tägliche -▁Gipfels -▁Plugin -bücher -▁fühlt -▁bicycle -▁Interpretation -▁abzulehnen -▁bedürfen -▁Euroraum -Tube -91 -ño -▁nost -▁ranges -▁Tran -▁bom -▁bold -▁PXI -▁geachtet -enophob -unches -▁cigar -▁Kandid -▁Ganze -deckt -bergt -▁Nothing -ifice -▁aufweist -▁Angl -▁Think -▁ausgestellt -▁Erstellen -▁ads -avor -▁wohin -▁verbrauch -▁Maßstab -▁backed -▁Glet -▁Italiens -▁writer -▁closure -▁unvermeid -▁selecting -▁silent -▁wertvollen -▁convincing -▁zuverlässige -▁argued -▁Athens -▁Adressen -▁Breite -▁dahingehend -▁Artikeln -▁Resultat -▁horses -▁zukunfts -▁Chechnya -not -oder -aure -Pal -BR -▁mad -▁wet -▁batt -utreiben -▁Ganzen -▁GIM -▁Unix -▁überflüss -▁abuses -ealand -▁unterstrichen -▁supranational -utzten -▁elf -arbeitungs -▁decrease -▁highway -krie -▁geschlossenen -▁advances -▁Hauptziel -▁reviewed -appy -straßen -▁donors -▁Drucker -▁Landwirtschafts -▁utter -▁Romanian -▁reiche -▁Entschließungsanträge -▁predecess -▁moralische -▁Bayern -▁attraktiv -incere -▁Jahrzehnt -▁amtierenden -▁amtierende -▁außergewöhnlichen -▁refriger -▁zivilen -▁Messenger -▁deaktiv -▁precaution -fassen -▁tube -▁wise -▁Sax -usions -▁Walk -▁RT -alg -▁Ult -▁UPS -▁on-site -verfolg -▁Progn -▁Stalin -adar -▁chances -▁ausmachen -▁vors -▁vorge -▁resid -▁Eintritts -errasse -zene -zuhalten -▁beding -▁Frucht -▁auto -▁technically -▁mood --Men -▁origins -▁affair -hauses -▁Offiz -▁Officer -▁administer -▁hosting -▁Statute -▁Hoheits -▁terraces -▁abstained -öpft -▁Verstoß -berechtigung -▁Diplomat -▁Zutaten -▁thirty -▁distributors -▁recruit -▁Gegebenheiten -dy --party -power -kenntnisse -VIEW -▁hört -▁Tu -▁EEC -▁Pav -▁giant -endern -▁denkt -▁Privileg -anta -▁entities -▁ausgenommen -▁spark -▁schien -ovina -▁Später -▁gesucht -▁Stress -▁geschätz -ocks -ymmetr -▁beside -▁Besetzung -▁Finanzrahmen -▁142 -▁absorb -ifizierten -▁personality -▁topical -▁gases -▁chemischen -▁Redezeit -▁Therapie -▁Intelle -▁Gewährung -▁reinforced -▁Ramblas -▁verbindlich -▁Nazi -▁transatlantischen -▁bezaubern -▁gepflegt -▁mixture -▁furnishings -▁criterion -▁Eurocopter -ez -ii -uels -bö -▁ferry -▁bags -▁SU -▁Posit -erl -▁lehnen -▁Harr -▁Rose -▁Vat -ivo -idor -▁repression -▁wirft -ikon -ecurity -▁Auflagen -ähren -▁Importe --Serie -▁enshr -maßen -▁modernsten -▁hearts -▁breast -arsh -▁engagement -▁Studies -bereitschaft -▁Kapit -▁Subjekt -▁drafts -▁anzugehen -▁voices -▁Comfort -▁Trep -wünsche -▁vacuum -▁rigorous -▁Moldau -▁Persönlichkeiten -▁abzuschließen -▁Telekommunikations -▁Railway -▁envisaged -beschluss -▁rarely -kompl -▁nerv -▁founder -▁Tiefe -▁Poll -▁Pots -▁A5-00 -▁Wandern -▁Klav -▁unum -▁unweit -aph -▁verteilen -portal -▁exclude -aha -▁entlassen -▁env -ahrscheinlich -▁arises -ammern -▁Strassen -▁Stream -▁parent -▁bekämpft -▁Andrew -▁beachtet -▁Folgendes -▁automatischen -wärtigen --18 -▁wunderbaren -suche -bezogene -▁Gefähr -film -▁harvest --Systeme -▁Steinberg -▁behindert -▁ruled -▁Adria -▁Vergüt -volumen -▁Vulkan -▁Photoshop -▁erfahrenen -▁IGC -▁europaweit -▁Vegas -with -wanne -▁delle -▁voor -▁Ms -▁Pens -▁gg -etra -▁katastrop -▁Nerv -▁Zeile -▁erober -▁verweist -▁Oft -▁adverse -▁plast -perten -▁Gramm -flotte -ährungs -finder -▁Verband -▁feste -▁Richtigkeit -▁Auth -▁direkter -▁Werde -▁Naturschutz -▁wahrgenommen -laubens -▁Kraftstoff -▁Staatsbürger -▁profitable -▁Solidarity -▁troubl -beihilfen -▁regelmäßige -▁attempting -▁Hoffnungen -▁Zivilisation -▁Kolonial -strategien -▁Winters -▁geräumigen -▁Komplexität -▁US-amer -▁schlimmsten -▁Muslims -▁criminals -▁Begeisterung -oa -baut --en -gipfel -LIN -61 -▁Büg -erzeugung -▁Hunde -▁inwieweit -ürt -lys -ernal -ington -▁ersuchen -vermögen -▁Prost -▁Störungen -auscht -▁disastr --Ste -▁basket -▁globe -▁customized -▁verlor -ettbewerb -▁Festivals -▁bereitgestellten -▁Natura -▁Gegenwärtig -▁vereinbaren -▁fortschritt -▁distribute -▁accommodations -▁Laufzeit -▁Gesundheitswesen --Um -▁interpreted -▁predomin -▁ausgewählte -▁wirksamer -steuerung -▁beobachtet -▁strenger -▁Creative -▁tschechischen -▁Immunität -▁benachrichtigt -▁husband -▁katastrophal -▁† -tags -kandidaten -▁ran -▁sincere -▁TE -▁TW -▁Male -esco -amment -▁aluminium -▁andauern -termin -ibus -fügen -▁appr -▁bestraft -▁EU-Haus -▁offset -▁basierend -▁excuse -▁nearest -enntnissen -▁1972 --discr -▁harmonised -▁Provider -▁Bulgarian -▁Alternativ -▁Erdgas -▁Erkenntnis -▁Cyber -▁Users -ladimir -▁Segment -▁benachbar -gehörige -▁Unternehm -▁abgeschafft -▁Especially -▁Lufth -▁carriers -▁Diktatur -▁indirectly -▁clinical -▁kopieren -▁Freedom -▁asiatischen -▁adequately -▁Akzeptanz -FA -▁dear -▁bands -▁Sauber -erkt -▁Roy -▁Rating -▁stones -estand -▁zugreifen -hrough -▁Unglück -▁frische -▁coins -▁Ali -akh -▁Jahrestag -▁Production -▁73 -▁smoke --Bar -▁Trav -▁Dienstes -▁keyboard -▁barbar -bereitet -▁Raumfahrt -▁Außengrenzen -▁resolving -▁vorherr -▁Gestalt -▁Städt -sprodukt -▁Kapitalismus -▁Olympischen -▁matches -▁Landwirten -▁Jacuzzi -▁Autorität -▁ceremony -▁Hindernis -▁Viren -▁Inflations -▁Porsche -▁Ozean -▁Rettungs -▁einzuleiten -▁Schottland --reaching -▁Gibraltar -gue -GEL -ва -▁nnen -▁dün -▁Dark -▁MAR -▁Beat -igel -ulle -▁NOR -▁Zentrums -verkauf -▁dependence -▁asks -▁abschließen -▁Thess -▁Weis -▁Alic -▁Aufklärung -condition -etext -▁Mario -▁accessed -▁instability -▁acid -▁Mittelme -konfl -▁statute --Design --Kl -▁reasonably -▁Fußgänger -▁Upon -▁formerly -▁dynamischen -shore -▁Europäisches -▁Valent -▁validity -▁bloß -▁vertical -▁Königreichs -▁constituen -▁Popular -▁Banglad -▁exklusiv -▁Lyon -▁Anpassungen -▁hazardous -rieren -cany -Card -▁ft -▁dimin -▁Written -amik -▁Ub -opoul -leuten -▁stating -erner -▁Zap -▁Unless -▁exakt -gebir -finanzierung -▁berufen -▁Manuel -▁angeführt -▁ausgehend -▁Medina -optionen -▁passive -▁breaking -ikanen -▁distant -▁Secure -▁behold -▁Währungen -▁determining -▁congratulating -ictionary -▁sämtlicher -▁adjustments -▁HTTP -regelmäß -▁literally -▁irgendeine -▁imperative -▁linear -▁Terrace -▁attraktive -▁autonomous -▁Migranten -▁Hunderte -▁cotton -▁Ingenie -▁beherbergt -beratung -nächst -Fast -▁Sin -▁P. -▁Pag -erungen -otto -▁klin -▁Cover -▁anzusehen -▁Zealand -▁geführten -▁jack -iedlung -▁abilities -▁willk -obs -acies -aku -▁Gründ -▁transmit -▁Enttäus -▁Herzlich -▁geschw -▁Klär -▁visas -▁halben -leiste -▁bewahrt -▁dorthin -▁dürften -▁Modus -▁Gesellschafts -▁Sozialisten -jenige -▁civilians -▁Südost -▁PCI -▁Latvia -▁Beitrittsverhandlungen -books -▁Teuf -▁loaded -▁vergleichbar -▁Strafgerichtshof -▁theoretical -▁voranzutreiben -▁Quartal -▁römischen -▁Sahara -oad -ku -ка -▁tiefen -▁Method -orious -olved -▁Fülle -▁Riesen -▁deriv -▁L. -ref -ernity -lichsten -uropäische -ssp -ources -ektoren -▁bestanden -▁thereof -▁workplace -▁Übermittlung -▁Lands -▁79 -▁750 -▁Herstellern -▁Manchester -wickelt -▁Fahrzeugen -▁führender -▁precond -▁Salon -▁Crus -ounded -▁blockiert -▁restrictive -▁Operationen -▁Gedanke -strecke -▁layers -▁stabilen -▁gratuliere -▁gemeinschaftliche -▁pillars -▁Mercosur -▁Learn -▁invoice -▁diesbezüglichen -▁Schließung -▁Sydney -host -aison -zessen -▁tighten -▁healing -▁salle -▁MO -▁Bald -onden -onian -▁Hav -ingo -▁ermittelt -ubil -▁Schreib -aments -▁Einstieg -▁counts -▁tries -stellten -▁markiert -▁bedient -▁Flüss -▁Extr -▁Whilst --Raum -gangs -▁legislativen -▁definitions -▁analyse -▁bekanntlich -▁classified -▁dynamische -ifelong -▁counterfeit -▁ausgerüstet -▁woods -▁Wanderungen -▁Schwerpunkte -▁Pressem -▁ausgeliefert -▁refres -▁Luftra -▁samples -spezifische -▁disagre -▁scrutiny -▁Hospital -▁sensitivity -▁Guatemala -,9 -Sp -23 -▁tender -▁fib -▁wished -▁Dicht -▁Dog -▁MM -▁Bath -ered -▁Cit -geist -enta -▁Lock -▁Nich -▁Stop -▁unsicher -▁cha -ätten -ertal -zeile -▁plates -▁preference -ordnet -ixels -▁Frances -▁pointing -▁securing -▁advised -▁blend -▁Projektes -▁Beträge -ITE -▁readers -▁reactions -▁Industriepolitik -bauten -▁cycl -▁formally -▁Reflex -▁österreichische -▁Appell ----- -▁Rhine -angehörigen -▁multilateralen -▁Zustellbetten -▁Seattle -▁Newspapers -▁~ -ti -westen -läß -ли -▁mache -▁TB -▁petr -▁cathedral -▁Kalk -▁Cer -▁Geden -imer -amas -▁verfahren -▁Diejenigen -thaus -gles -blicke -▁supporters -ancer -▁81 -▁eingen -▁surgery -▁ausgepr -▁Bürgerrechte -▁critique -▁eventuelle -lessly -ASA -▁examining -▁Simon -▁ageing -▁non-governmental -▁niedrige -▁..... -namo -▁mobilen -▁abzugeben --year-old -ibliothek -▁tonight -▁anspruchsvolle -▁Pferde -▁angle -▁bemerkens -▁Tosk -▁innocent -▁Nepal -▁intuitive -▁tags -▁hör -▁baut -▁pir -▁Hilton -▁Gutes -alys -ifier -▁concludes -ehlen -ubb -▁Beibeh -▁queries -▁Vorstands -▁interrupt -▁wanting -▁Monaco -hältnissen -▁zusammengestellt -▁studying -▁Dienststellen -▁Werken -▁transported -▁Jahreszeit --199 -▁positives -▁verdanken -▁Frankfur -▁Downloads -▁couples -▁Therap -▁counterpart -▁streben -▁ét -▁gemütlich -▁execute -regelungen -▁variables -▁consolidated -▁unanimity -▁Kongo -▁Ungleichheiten -▁fulfill -▁distinguish -▁certificates -▁Spirit -▁Behandlungs -▁Holding -▁Guantá -▁Giovanni -firmen -ßig -▁roten -▁Sara -eline -inhaber -▁Glaubens -▁unvor -▁unbesch -▁aufhalten -quoten -ordnen -flation -cons -▁Challen -▁Chianti -▁Reifen -▁interference -▁Lee -▁breaks -utenant -ränkten --CD -▁comfortably -▁Gleichwohl -▁Sozialpartner -▁akadem --Ind -effekt -▁compromises -▁Gemeinschaftsrecht -▁Wohler -sproblem -▁unmittelbare -muster -▁begrenzte -Station -▁kommunizieren -▁Meinungsäußerung -▁grey -▁Schlußfolgerungen -▁verschärft -▁audiovisual -stockung -▁shooting -▁Congo -▁galleries -▁Verhinderung -▁disastrous -▁Thessalon -fft -l. -lassung -pension -äglichen -▁fold -▁rumän -▁Mord -▁Bash -otr -▁cow -▁Herm -▁unjust -▁Looking -▁aufhören -illen -▁Einreise -eutral -▁Abt -▁wenngleich -▁interventions -drop -▁minibar -▁83 -▁geschenkt -▁protests -▁Steel -▁Clar -▁prinzip -▁zurückkommen -▁Skigebiet -▁Disabled -ASE -▁Panel -▁jünger -orientierten -▁LabVIEW -▁refugee -schulen -▁preservation -▁safeguarding -▁Ecu -▁Bewegungs -▁Mountainb -▁Hochschulen -adresse -▁Bavaria -▁Translation -▁Feedback -▁Guantánamo -gasse -▁SR -▁Rud -icism -▁anonym -reif -step -ktoren -▁verify -reate -agern -gründe -thur -▁admir -▁betreut -▁Chip -ergy -▁selts -▁trans-European -▁inviting -▁105 -▁Whit -▁Ziels -▁gericht -▁Andor -▁integrating -▁bedeutender -▁intimate -▁klarstellen -▁Afrikas -▁2.6 -▁Motoren -▁downloads -ssystems -▁Democracy -▁mittel- -▁Börsen -▁Vergessen -▁Ansprüchen -▁geräumige -▁attraktiven -▁künstlerischen -▁sympathy -▁unglaublich -▁veranstaltet -▁städtischen -▁Körpers -▁Softwar -▁Heidelberg -▁disregard -körper -HT -HA -▁pair -▁Mathem -inj -▁echter -▁echo -▁Wur -▁Camer -▁CSS -anwendungen -arly -lever -abetes -önt -▁sozio -▁Erfassung -easing -▁Hotelzimmer -▁perception -▁Maj -▁Zeitschrift -▁konstitution -▁Contract -▁buyer -▁thinks -▁refusal -▁linking -▁begrenzen -▁Tagungen -▁Beispielsweise -▁Daraus -▁annually -▁Chris -▁encomp -▁consultations -künften -langes -▁FREE -▁virtuellen -▁homogen -▁Davon -▁Synth -▁revolutionary -▁Impuls -▁Orchestra -▁monopoly -expected -▁Lufthansa -▁` -gg -▁aust -▁belt -▁Dé -▁DES -▁Pi -information -olic -▁leitet -▁Kast -ierendes -▁wirst -ischof -appar -either -▁Stan -▁esc -▁amp -ongue -▁Gegenden -▁strikes -▁Nachteile -▁Handhab --Port --comp -▁massiven -▁starker -▁zuständige -ICE -▁inspect -▁Endes -▁limitation -▁archae -▁Verschlechter -▁irgendwo -▁ecos -▁Standorten -▁Publikationen -▁Synerg -▁SilverFast -▁accompanying -▁mittelalterlichen -▁Besatzung -▁incomes -▁Gastronomie -▁overwhelming -▁Castro -▁Montreal -▁corporations -läuft -Kr -ár -▁hind -institut -▁Rica -▁Vic -▁Greg -▁anch -rapp -stoß -▁erhebt -▁verhandelt -▁sichert -▁abhängen -▁shore -auswahl -▁ours -▁plot -▁Fracht -ianer -▁leidet -steuern -orship -▁anyway -weren -▁Ash -▁posed -▁Autonom -▁democracies -▁beweg -▁Feiert -▁Canadian -celand -preisen -▁protective -▁perfekten -▁degrad -▁holes -▁breeding -▁CO2-Emissionen -▁medicinal --Unternehmen -▁surfaces -▁kreative -▁Funktionsweise -▁übertrieben -▁qm -▁Cruz -▁Wäscheservice -treffen -nütz -oir -ressourcen -cano -▁sums -▁merk -▁eb -icion -▁Neither -quenzen -▁Inzwischen -▁Unregelmäß -▁jazz -▁Schritten -▁worried -▁'E -▁Ausbruch -▁untergebracht -▁Aufpreis -▁elite -malen -▁Bruch -▁YouTube -▁detention -▁Medium -▁gebucht -▁Finanzielle -▁kommende -▁115 -▁aiming -▁Canary -▁reservieren -▁Wohnsitz -▁nightlife -▁Kriterium -▁starb -▁Diskussions -▁demonstrations -▁Upgrade -▁panorama -▁Depending -▁memories -▁Fischereiabkommen -▁Kohäsion -▁diplomatischen -▁intelligente -▁Political -▁Sitzungsperiode -▁Räumlichkeiten -üs -▁tv -▁hunder -▁Scr -▁PAR -▁greift -▁Bien -▁A1 -▁Cool -▁Hallen -▁Fabrik -▁unang -uliche -▁Pries -▁Verschw -forest -▁Anschließend -lave -▁ausgeben -▁Erziehung -▁amid -ritical -▁EU-M -▁dismiss -▁très -▁perceived -ventura -▁Branden -▁modification -▁muslim -▁norm -▁multif --Out -▁Etage -▁freundliches -ponsible -▁irgendwie -▁Anyone -▁deadlines -▁begrenzten -▁konsolid -▁noble -▁pushed -▁Genauigkeit -▁gestrichen -▁Residenz -▁übereinstimmen -▁Käse -▁Colombia -▁Beibehaltung -ME -▁vierte -utory -unya -user -▁theatr -▁UEN -hrow -▁versetzen -▁Prince -agung -▁desp -▁Stere -icked -eased -▁Grupp -▁worker -▁proves -▁developer -▁deciding -▁Verbr -drückt -▁Grundprinzip -azines -▁Bestrebungen -itiativen -▁Beschleun -▁Embry -▁smallest -▁erwarte -▁confirms -▁Linz -▁cream -▁portion -ettering -▁vorgesehene --like -▁1940 -▁divisions -chitektur -▁angemessener -▁mittelständ -▁Flüchtlingen -▁ruins -▁delegations -▁BAE -▁adjacent -▁pflegen -▁sacrific -▁Verhaltenskodex -m2 -cott -63 -▁wisdom -▁bugs -▁Sisko -▁Müll -itur -itär -▁Cant -▁Fusion -▁FTP -ulent -▁anre -▁Zentren -▁einsetz -▁Anleger -ipen -ipend -▁safer -öser -▁Exekut -ynast --mon -▁Skif -▁Carol -▁optimiert -▁landmark -▁classification -▁schriftlichen -▁Magazine -▁Stabilisierung -▁Provin -▁anzuerkennen -▁Recently -▁morgens -▁vermieten -▁empfehl -▁dramatically -▁bottle -größte -▁punishment -▁obstacle -▁verwandeln -▁Vortrag -▁sequence -▁irrespective -▁Against -▁endorsed -▁admire -sing -sorge -zum -luft -▁Tast -▁Taschen -▁boutique -▁Pisa -▁cattle -▁CAR -▁GO -stande -tration -▁dekor -▁aufgetreten -▁versprechen -▁Price -ugung -▁exotic -▁allt -▁commerce -▁bedingt -▁herange -▁Germans -ymn -▁Police -▁immunity -▁definitive -▁Begleitung -▁Organisations -▁Edit -▁Nordkorea -▁profitiert -▁Mannschaft -▁english -▁Zuliefer -hardt -▁freiwillig -▁beeindruckende -▁по -▁Zuschauer -▁exzellente -▁MEDA -▁nachfolgenden --Entwicklungs -▁Petersburg -osexual -▁guilty -▁bestehend -▁Nordirland -▁bub -▁BER -erhö -usik -▁FS -▁Gonz -▁inappropri -stahl -▁stagn -▁Instit -▁erholen -▁aufrufen -ellungen -▁-T -▁umgew -izard -eway -echts -▁Param -ervices -▁Alten -gebungen -▁hero -▁festivals -▁serb -▁Frequenz -▁Weste -mäßigkeit -▁reader -▁erzielte -▁alternativen -▁Angeboten -▁intensiven -▁dynamisch -▁terrain -▁kritisieren -▁Estland -gültig -▁definitiv -▁Streitkräfte -▁PayPal -▁küm -ergebnis -▁Additional -▁Mittagessen -▁Vieh -▁praise -▁Stephen -▁vibrant -▁Posselt -hu -door -restaurant -vre -ен -▁horm -▁Toten -▁Iceland -▁SN -▁Sender -▁PA -iso -▁Häusern -▁inland -ivirus -anbieter -amine -▁Langen -trock -▁gez -▁Erhebung -ensp -performance -▁agieren -ittings -▁Nachweis -▁10.00 -▁Zusammenfassung -▁versorgen -▁php -▁Maßst -▁Andrea -▁advisory -▁far-reaching -▁fastest -▁textiles -▁completing -▁Netzwerke -▁1971 -▁Eurob -▁Paar -ACE -▁Rauch -▁Ausgangs -▁unternehmer -▁snacks -▁Rohstoffe -▁trials -▁gambling -▁eingenommen -jar -ég -▁müh -▁Sänger -▁DG -▁Dür -▁pestic -▁gän -erd -eman -omo -▁kat -▁Kass -▁Hague -▁Rate -▁erbringen -▁einrichten -oded -▁quo -▁Spock -immten -▁Kondition -▁Chancellor -▁remedy -▁Klaren -▁Nev -▁modernisation -ynes -▁improves -marktes -▁Scandinav -▁Patient -rechtlich -▁verbundene -▁aufgeführten -▁Consider -▁Oliver -▁anerkannten -▁baths -stechnologien -▁gaining -▁Poettering -▁rolling -▁sleeping -▁beschleunigt -▁flexibler -▁Birma -▁Tuscany -machung -schrauber -oors -rine --9 -▁tens -▁Taste -▁DL -▁Pixel -▁kalt -▁Feder -▁erfr -▁wholes -▁entf -▁Engel -▁Erz -▁en-suite -▁Archit -▁betragen -▁buttons -▁gesenkt -▁Meet -▁Gender -▁Stadtm -▁benutzer -▁Durchführungs -▁Malaga -▁hervorzuheben -▁Nutzern -ITY -▁Slidesh -▁aufgegeben -▁Konsolidierung -▁konkreter -▁Gebäudes -▁Rotter -beschäft -▁Brosch -▁courtyard -▁Sandstrand -▁Bandbreite -▁Franzosen -▁Sachverständ -ektionismus -▁barrier -▁consolidate -▁Ungleichheit -läufigen -▁witnessed -▁Championship -▁unzählige -▁Gesetzgebungs -night -üp -jev -▁tu -▁hunting -▁fügt -▁männ -▁Tragen -▁Isch -igste -orien -▁Want -▁Keller -berry -▁stere -▁stützt -▁Promenade -▁Verwand -egal -ialen -▁Spannung -▁Umstellung -▁Bedrohungen -▁Schlüss -▁Neuen -▁modifications -▁befreit -▁Feier -▁stecken -▁sheep --West -▁Cohesion -OSS -▁volumes -▁Pfeiler -▁IND -▁DEM -▁erhielten -▁irgendwann -▁Harmonie -▁paragraphs -wiegend -▁Follow -▁Klimasch -▁gewaltige -▁Algeria -▁Cancún -▁licensed -▁Warsaw -▁beauftragt -vas -▁hoc -▁femin -▁lust -dehnung -▁Lov -eständ -▁Zell -tery -cking -▁Jim -endums -▁übernahm -▁Though -weiß -▁genoc -▁disabilities -oyd -▁10.000 -▁geschah -▁operative -▁secular -▁costly -▁XX -▁Begrenzung -▁introduces -▁settle -▁sends -▁clauses -bestätigung -▁Einzelf -▁assessed -▁gedruckt -▁höherer -▁meaningful -▁1990s -▁kräft -▁behindern -▁Wilhelm -▁complementary -▁vorzugehen -▁fabulous -▁Algerien -▁tasty -antasie -▁maßgeblich -▁tendency -▁Subsidiaritätsprinzip -▁gelegentlich -▁ggf. -.8 -.7 -diensten -wear -world -TM -post -gir -▁tensions -▁Dream -▁Aar -▁HI -▁fortress -▁anst -leys -▁reimb -terior -▁chap -▁chart -ermost -▁5000 -strich -▁Zeitalter -▁liefer -▁Verbindungs -chtenstein -▁versuchte -garant -▁Bogen -▁activate -▁Restaur -▁iranischen -▁limiting -bildungen --Hand -▁Disziplin -▁Texten -▁coastline -▁Vollständigkeit -▁Cardiff -▁Villen -▁psychological -▁painted -▁genetisch -▁verantwortungsbew -▁Beschwerde -▁Objective -▁folgte -▁stetig -▁scholars -ounds -fels -krank -LICH -▁melt -▁baggage -▁Mön -itary -▁Wollen -▁unexpected -▁fore -▁Zen -▁demselben -▁demnächst -▁entnehmen -▁Einmal -▁commands -▁Yacht -▁Bestände -▁couldn -▁Button -▁Energiem -▁Ausschüssen -▁traditionally -▁Designs -▁II. -▁aufgegriffen -▁reaff -▁Akademie -▁altitude -▁oriented --Systems -▁juristischen -▁Erholungs -▁Reichweite -▁Beobachtungs -▁gründete -▁kommunistischen -▁birthday -▁McCart -▁erstklassige -▁distinctive -▁Schauspieler -▁Wikitravel-Benutzer -vereinbarung --Mediterranean -▁shoulder -terrasse -▁Fotokop -▁Verschlechterung -familie -wein -wider -ál -▁Eiff -▁Araf -▁kol -▁Kug -▁FIF -▁VIS -icide -▁anschauen -ereign -▁Zak -trieb -▁defen -▁verfolg -▁-D -earning -▁1900 -ystemen -ankungen -zuerhalten -drom -▁Final -itäre -▁Franco -▁Partition -punkten -▁gemachten -▁verläuft -▁24-hour -▁identifying -▁Official -▁bankrupt -▁container -zulassen -▁sorts -▁jemanden -▁holen -▁Somalia -▁Reisenden -profit -▁threatening -▁ermord -▁verlängern --Lösungen -gelegenen -▁prerequisite -▁Überwindung -hilf -fehler -jack -та -▁nett -▁ras -▁Deal -▁oasis -▁loy -▁Wende -▁yield -▁Lyn -▁Ner -▁gefertigt -▁Anleitung -▁Angestellten -▁dairy -▁Eras -▁Ermitt -anza -iento -▁indications -▁bedaure -standes -▁tunnel -▁landing -▁kostenfreien -ORK -▁protects -▁scheinbar -▁emphasize -▁Ukrainian -▁popularity -▁Übereinkunft -▁Fischereise -▁fragile -▁Sicherstellung -▁QuarkX -▁broadcasting -▁gemeint -eability -▁vigil -▁Fotografie -▁aufrechtzuerhalten -▁Einstimm -▁Chirac -▁ThyssenKr -▁geeinigt -▁rhythm -▁Handhabung -VD -▁fare -▁misl -▁Takt -▁Iz -▁Must -igma -oland -▁Wag -▁Kamp -▁Fett -▁Veter -arisch -asm -amos -alitäten -▁alph -▁begeben -schub -tro -▁abschließ -▁Schätzungen -▁Enth -ourge -acle -iane -ounce -▁Marg -▁Produkts -▁Medic -▁passport -▁Portfolio -▁Strafe -▁1.000 -▁bevorzu -▁nuklear -▁xenophob -▁Tauchen -wogenheit -▁tracking -▁painful -▁Megap -▁nördlich -▁Beautiful -▁widerspiegelt -▁erhöhte -▁weiterentwickelt -▁einbeziehen -▁earthquake -▁ThyssenKrupp -yball -▁tons -▁Däm -▁Eden -▁pipeline -▁Battle -orum -▁in-house -▁algorith -▁Led -owina -▁asc -forum -▁Angehörigen -ubt -▁aboard -▁Schad -▁entziehen -▁coat -▁Aussehen -schaftlichen -▁Eins -▁Arbeitsprogramm -▁legacy -ancement -▁worlds -▁Handbuch -▁seinerzeit -▁Finanzminister -▁Rechtsprechung -omatic -▁Fahren --Le -▁affirm -yram -▁Century -▁loop -▁Euratom -▁separat -betreiber -▁untersagt -▁exploited -▁Wechselkurs --2006 -▁verteidigt -▁korrigiert -▁confidentiality -▁einheimischen -bewegungen -▁refreshing -rot -wal -bird -Sh -MP -42 -▁Meas -edt -offt -▁Hüt -▁Gitar -▁unified -reuen -vertreten -verhältnis -▁Stille -▁auszur -▁encl -▁Einleitung -▁Vorbe -▁reluct -▁konsequ -ächten -▁belastet -gebend -▁versorgt -▁privater -▁Boots -▁13th -▁shorter -▁Liechtenstein -▁entsprechender -▁Magn -▁Konferenz- -▁Nachtleben -▁renewal -▁billigen -▁Wellnessbereich -▁graphical -▁spreading -▁forestry -▁Fertig -▁Inneren -▁Ursprün -▁Cafés -▁erzählen -▁Astrium -▁widerspiegeln -▁Anfänger -▁Toskana -,1 -test -yption -41 -62 -▁förder -▁bird -▁DO -▁Auß -itches -orders -▁cord -▁Five -▁alert -▁pricing -▁Lut -▁LEG -lebt -▁regener -▁vermehr -▁Vergabe -▁Parking -errungen -flügel -▁Rezept -▁interact -▁incidents -axis -▁evenings -▁Whatever -▁differenz -▁persist -gewiss -▁retained -ässige -▁biof -iquen -▁sexuelle -▁widmet -▁Verkehrsmitteln -▁Infine -▁zerstör -▁unterschiedlichsten -transport -▁verwandelt -▁Secretary-General -▁beunruhigt -▁Haltestelle -▁toxic -▁schuldig -abend --vis -jer -jana -▁nit -▁Adress -▁Cort -▁Cau -▁unüber -▁unkompl -ierbare -ceive -erning -plung -undi -▁fridge -izations -▁Ersten -ervor -▁Einerseits -▁Einsparungen -twer -▁Aufstellung -▁County -kehrungen -opez -▁erstmal -▁Romul -▁hervorr -▁marble -▁Geschäftsreisende -▁antique -▁Garage --down -▁Angebots -▁Informatik -▁encourages -▁attacked -▁oberste -▁quarters -▁Twin -▁Seminare -▁Dolomites -▁flexiblen -▁Gremien -▁Kontinents -▁Geburts -▁osteuropäischen -▁Property -▁Thessaloniki -Point -▁ses -▁Tepp -▁Match -▁PN -atar -▁beheben -opy -terity -▁mitzuteilen -übertragung -▁Stoffen -▁Ansehen -▁umbr -▁resign -▁ampl -icker -eutet -▁Aufteilung -▁ele -▁hinnehmen -▁10th -iversität -▁Demo -athan -▁Kommissars -▁Palais -▁entfernten -unterricht -▁Jane -▁settlements -▁Upper -▁Präsidentschafts -zyklus -▁betreffende -▁getroffenen -ungsverhältnis -▁verursachten -▁unternimmt -▁Erkrankungen -▁bathing -▁vertiefen -▁Schriftsteller -▁recognises -▁Croatian -▁enforce -▁großzügig -▁Jobs --Qu -▁einschlägigen -fun -gien -▁ht -▁fits -uto -▁cum -▁kW -▁kapital -▁RES -▁GL -▁unannehm -iein -arien -istol -▁aufgeben -ehouse -iba -▁chips -▁grid -▁kontr -▁Exempl -▁ordering -▁ganzheit -▁schwäch -▁organs -▁Instabilität -▁Bemühen -▁Zusagen -▁westliche -▁Gedächtnis -zwecken -▁Economy -▁Venet -▁Momente -▁automobile -▁Schriften -▁lifts -▁trendy -▁petition -▁veranlassen -▁uneingeschränkte -▁verbindliche -▁Ministerpräsidenten -▁encountered -▁Scientific -▁Freilassung -▁Kolumbien -▁Klavier -▁Authent -▁Unregelmäßigkeiten -▁aston -▁Daneben -insel -archie -▁anscheinend -▁Nächte -owse -▁Inspir -essim -anti -ikers -▁Schein -▁schaden -▁Parc -▁3,5 -▁Abfahrt -ishop -▁trick -▁obgleich -arbeiteten -reihe -stellbar -▁gewahrt -etrieben -▁processor -▁einfaches -▁betrieb -▁Kino -▁Gemeinschaftsebene -▁studios -▁Volumen -IST -lauber -▁donor -▁Hochschule -▁biologische -▁missbrauch -▁safeguards -▁Propag -▁Turnier -▁Mercure -ioxid -▁durchschnittlich -▁temporarily -faktor -▁Geltungsbereich -▁Chinesen -▁dioxide -▁Darlehen -▁QuarkXPress -hoch -hire -IEN -▁Tav -▁buch -▁Self -▁pup -▁Mes -ety -iguous -▁zoom -▁Cape -imeter -▁refin -▁Zeilen -herapie -▁Stufen -endungen -▁abgedeckt -▁Schall -▁Mitch -▁lev -aukas -ologies -▁Orth -▁Warnung -▁Lebensbedingungen -▁Industrien --Code -▁hochrang -▁Decision -ICA --Anwendungen -▁Gehör -▁hybrid -▁Erdgeschoss -▁enhancing -▁PHARE -▁impressed -▁ideology -▁scripts -▁verwendete -▁potenziellen -▁shaped -▁Catalunya -▁incons -▁Beantwort -▁EU-Bürger -▁Lettland -▁gänzlich -uate -vr -RS -▁vene -▁beam -▁Dell -▁categor -▁kar -ivities -anol -verträge -▁Jarz -▁vorgetragen -▁Fraktions -▁reloc -▁underp -klicken -etry -▁economical -▁Teils -▁fließen -▁angekommen -▁playground -▁Emot -▁Schwanger -▁Feucht --market -▁Begleit -▁Turin -▁Angola -▁magazines -▁Perfect -▁unverändert -▁mounted -▁Luftverkehr -▁Container -▁evaluated -▁responding -▁holder -IFE -▁OSZE -▁geehrter -▁tailor -▁Übersetzer -▁floods -▁batteries -▁municipality -▁appelliere -▁Diagnose -▁Viertens -▁¿ -zutreten -low -ägliche -ри -▁E-mail -▁Eye -▁puls -▁Body -▁Gam -▁nein -▁Unit -artial -▁überdenken -▁überarbeitet -▁enden -▁Aufstockung -fahrzeugen -werter -▁Seven -▁geste -▁attitudes -▁Try -▁Genfer -▁Beschränkung -änglichen -▁persecution -freies -▁stärkt -▁Christus -▁merch -untu -▁ratify -▁Süß -▁Beschäftigungspolitik -▁imagination --Präsident -▁Fax- -▁Matth -▁territorialen -▁bilaterale -▁Erweiterungen -robat -▁enemies -▁Veranstaltungsräume -▁herkömmlichen -▁astronom -▁Biokraft -▁neighborhood -▁Rotterdam -fi -sicherung -AX -▁dub -▁races -▁Money -▁Pisten -▁galt -itives -▁ehren -▁Father -gehalt -▁anschließ -ehear -agne -▁Schauen -▁schall -▁selekt -▁Verbrech -▁remed -▁besuchte -folged -▁Polo -▁correction -▁1.6 -▁barg -▁hochqual -▁hinausgehen -▁casual -▁potato -▁Verwenden -▁positiver -▁assessing -▁kultureller -▁inspections -▁routine -▁conducting -▁responded -▁Bestandteile -▁Ausflug -▁Inc. -▁Mitgliedsländer -▁captured -▁privileges -▁israelische -▁stimulus -verantwortung -▁gedenkt -▁Befreiung -▁Frühling -rott -makers -UE -▁hang -▁nad -▁Soph -▁Bod -irma -▁löst -▁Wid -▁Hack -▁Rum -▁dependen -▁überleben -ovol -▁pläd -beck -▁trinken -▁YOU -ändlern -▁Übertrag -▁Brücken -setzungs -▁Bettw -▁blow -rängt -▁understandable -▁administr -▁2014 -▁intensiver -▁broadly -▁Lieferungen -modelle -▁Streben -▁titles -▁ELDR -▁Rosa -▁preserving -▁Symph -▁WordPress -▁verbesserten -▁burning -▁Geburtstag -▁disadvantage -▁democratically --Produkte -▁Connect -▁Pyren -▁Liebhaber --Sicherheits --ad -pflanzen -KK -▁tyr -▁nick -▁bout -▁pushing -ofen -▁ox -▁cir -▁kenne -▁Vö -▁untr -icable -asy -▁Living -▁verlaufen -▁exac -▁shell -▁vorwiegend -▁Erlaubnis -ennungs -▁unterrichtet -▁möge -amme -ilege -▁Plätzen -▁invasion -▁mini- -▁Zusammenhalts -▁visibility -▁experiments -▁Gläub -▁14th -▁zweites -▁Terrassen -▁behilf -▁Doktor -▁Perl -▁transparente -▁continuity -▁Gemäl -▁Machine -▁Napole -▁imbalances -▁Zinssätze -▁Gefühle -▁Nürn -▁inappropriate -▁Beantwortung -dar --bed -▁Istr -▁Ign -▁Script -isierter -ologen -anium -▁Lizenzen -tehen -hrend -▁deployment -▁geführte -▁-K -ahoo -ieth -ärkte -gl. -chtungen -▁Chor -▁outermost -▁indischen -enzo -▁estimate -▁Versicherung -konto -otted -▁11.00 -▁closest -ziell -▁nächstes -unterschied -▁institute -ächtige -▁Crown -▁wunderbar -▁Analyt -▁quoted -▁Zentralbanken -▁beliebige -▁ehrgeizige -▁Sozialdemokratischen -▁verweigert -▁wholehear -▁Symbole -▁Delegationen -▁measurements -▁attentive -▁Klimaschutz -▁exacerb -sport -OUR -ât -▁Tasten -stufe -▁112 -▁einräumen -ehand -ading -▁ausgere -▁aussieht -easant -▁unterzeichnen -▁durchd -▁bestrebt -rative -▁hinnehm -▁Depression -▁aven -▁directors -▁Atem -▁versichert -zugang -uchtet -▁activists -▁Euro-Mediterranean -haushalts -▁Spaniens -▁fairer -▁230 -würfe -▁obliegt -▁herauszufinden -▁Empfangs -ombard -▁Gastgeber -▁timing -▁birgt -▁Verstöße -▁Wiederaufnahme -▁injury -kapazitäten -▁veränderten -▁wiederherzustellen -▁umfangreiches -▁Konservativen -▁Ausserdem -▁geplante -▁GIMP -folgedessen -tax -access -▁dachte -▁miß -▁mangel -▁TEN -▁bass -igm -▁Welle -▁CIA -▁Fische -▁Flair -imon -ulette -▁pretext -▁Niel -heet -▁wire -ortium -▁Unrecht -▁abzustimmen -amento -▁rocks -▁sowjet -▁underest -▁Überschwemm -▁problematic -vorhaben -▁ganzes -▁Schwächen -▁statutory -▁Student -▁Kontinu -▁devote -seitigkeit -▁Kanäle -▁unerwün -▁Langzeit -▁1956 -▁credits --Programms -▁verfasst -▁Kohäsionsfonds -▁überschritten -▁Kriege -▁Ferrari -▁Zertifizierung -▁varying -▁Friends -▁behilflich -hänge --v -kit -55 -OY -ки -▁pav -ilog -chio -▁cant -▁Vitor -ufer -▁stoßen -▁Stars -inded -ocon -▁nachkommen -üne -▁Vorjahr -steigerung -▁6th -▁dropped -richtlinien -▁befindlichen -▁retention -▁competitions -klärungen -▁metals -▁Gleiche -▁Innenhof -▁Flora -▁Applikationen -▁staatlicher -▁seekers -▁Motorrad -▁teilnehmenden -▁Zufall -▁älteste -▁gathering -▁verschwunden -imensionen -▁Workshop -▁durchschnittliche -info -▁Befürchtungen -▁recreation -▁viruses -▁diplomacy -▁Autovermietung -▁Recycling -zucht -IES -▁töd -▁BV -orbenen -▁Hund -▁Nest -▁Zoom -verages -▁jens -▁escal -ighth -▁coin -▁Auszeichnung -rency -aufsicht -▁interconn -▁geliefer -ziehe -▁Quite -parteien -iefung -zufriedenheit -▁gentle -▁üben -▁initiate -▁häufigsten -▁zeitlich -▁launching -hotels -▁Badewanne -▁Nahrung -▁Schriftliche --Plan -▁Alexand -▁hoping -▁cooperative -kopf -▁gewaltigen -▁Reisende -▁Konfigurations -▁Pferd -▁Mitentscheidung -ocaust -▁schrecklichen -gängig -▁fossil -▁erledigen -▁genocide -▁Erasmus -▁³ -CAD -▁tired -▁harsh -▁mes -▁Type -ethe -▁Röm -umgebung -▁Nit -▁stainless -▁aufgeteilt -eint -▁soviel -▁Ausschluss -acion -acter -▁gründet -▁ihrerseits -▁hinder -▁Bedauern -▁Articles -▁handelte -bewusstsein --Bus -▁folgendes -▁passage -▁Glob --Gesch -zufordern -▁2.1 -▁wahrhaft -▁hosted -▁Corbett -▁Temple -▁virtuelle -▁Herzog --wing -▁harder -▁Buy -▁feeding -▁Check-In -▁Gardens -▁formulieren -▁AVI -▁Cleaning -▁eggs -▁Füßen -▁Competition -▁Wohlergehen -iè -ihen -samer --30 -▁Ship -▁EIN -▁gt -elect -▁unend -imar -▁forever -▁Lack -▁rechte -tries -▁weise -▁Sieger -▁Anth -izeil -ermo -feiert -ologically -oyer -▁Entschuld -▁Sekret -▁Berechn -▁Bürgerkrieg -▁halbe -orno -gewichte -▁Durchbruch -▁Below -▁Kodex -▁Protektionismus -▁Damals -▁Glückwunsch -▁Tunnel -technologie -▁Immobilienanze -▁Wallström -▁withdrawn -▁vegetable -▁Mitgliedsstaaten -▁Dienstleistungsrichtlinie -▁conscience -▁Dalai -▁schweigen -▁embargo -▁Salvador -▁transeuropäischen -▁precautionary -din --pro -L. -▁rival -▁ruft -▁Side -▁Mind -▁Pocket -▁BAS -▁Aven -uta -▁lending -▁Korn -▁yacht -idsch -stan -istet -▁1,000 -▁-F -▁manually -amentos -acional -▁compiled -▁laundry -▁problematisch -pielen -▁angehört -▁daraufhin -▁Gewässern -▁Adjust -▁artic --Res -▁listing -▁benefic -▁positively -▁gefährlicher -atastrophe -▁pursuit -▁salon -▁motorcy -▁attending -▁1980s -materialien -läufe -▁Prüfungen -▁wheels -▁AKP-Staaten -▁Wäldern -▁Festplatten -FORM -M. -Office -▁dol -▁dossier -▁Aquar -▁lifelong -▁Wat -▁Kne -ifs -▁verarbeiten -▁Verurteilung -ehle -illes -▁escap -▁sprachen -▁bester -▁Vormittag -▁dirty -▁Mari -▁Übersch -▁Flucht -▁Conil -▁1200 -▁gemeldet -▁besorgnis -▁successes -ässe -gemacht -▁Haushaltsausschusses -▁portable -▁establishments -▁Radisson -ibration -▁professionell -komponenten -▁burdens -▁semi- -▁Konsequenz -▁Baumw --Im -potenzial -▁ausgebild -▁myster -▁Weihnachten -defizit -▁prolong -▁Klingon -▁Intensiv -▁Biotechnologie -▁zwingend -noten -dor -▁Tang -▁Sor -etre -olis -▁Wine -▁unh -ascript -▁anmerken -quiv -▁fürchte -roup -▁Auslieferung -▁3.1 -▁Parlamentarier -▁solarium -▁Handlung -▁Bestands -▁instances -▁backup -▁mirr -▁Gewässer -▁studierte -▁organising -▁bookings -▁Meerblick -▁zusätzlicher -▁Instanz -▁radi -▁stresses -▁Erzeugnissen -▁Botschaften -▁Tierschutz -▁befürchten -▁Konzeption -IFT -▁coordinating -▁Sachver -▁Belastungen -▁disappeared -▁Dubrov -▁deliberately -▁Singapur -widr -▁Scottish -▁Frankfurter -▁Infineon -▁Jarzemb -▁besorgniserre -hus --east -82 -▁Sic -▁EOS -▁Born -▁COS -▁Rhyth -▁toys -▁Nass -▁Init -▁erzeug -genehm -▁Ohr -ahm -▁abhängt -▁ausschließ -▁vorank -eckte -ovon -errung -▁disarm -▁anymore -▁7th -▁Laundry -▁Reding -▁defines -▁packed -▁template -qualities -▁collecting -▁Stärken -▁Privatisierung -▁wissenschaftlich -▁jeglichen -▁Trock -▁Geistes -▁Zeugen -▁steigende -▁Josef -▁Egyptian -▁Sympt -▁Athen -▁beeindruckenden -▁Equipment -▁Keramik -▁Auseinandersetzungen -▁Behandlungen -▁gigant -▁Erleichterung -üchter -▁rass -▁sau -▁bitt -▁Sha -oner -▁Cin -▁Cambod -▁C4-0 -▁Hem -▁unch -former -▁Stolz -pricht -▁spam -spot -akische -glas -gebnis -▁marking -▁deckt -▁mines -▁begannen -▁Bestreben -▁detected -▁dealers -managements -▁foodstuffs -▁1965 -▁überwältigen -iscipl -▁Records -▁Forschungs- -▁Gehirn -▁OSCE -▁eliminating -▁Flusses -▁Electronic -▁Cambridge -▁Ecofin -▁Babys -▁bürgerliche -▁Kürzungen -▁Prozents -▁Ökos -▁commodity -▁feasible -▁Alicante -▁Klärung -▁unangem -oust -vell -mans -▁fer -▁föder -▁seam -▁buffer -▁Seas -unters -▁eco -▁Hop -▁unauf -earl -▁Ausdehnung -▁Alz -hness -akult -prob -gli -▁Rekord -wirr -▁Verbündeten -▁geschicht -▁Barbar -▁accessibility -▁Monument -▁folk -▁1400 -▁historischer -▁swim -▁unvergleich -tauchen -▁liberty -▁corrupt -▁Eingangs -▁Castell -▁empfehle -▁juristische -▁Rundfunk -▁Electric -▁fulfilling -▁ehrgeizigen -▁anspruchsvollen -▁einzigartiges -▁japanische -▁Learning -▁Ausgewogenheit -▁umgewandelt -▁Arafat -▁jenseits -fast -sbereich -meter --fl -WA -FL -VER -▁holl -▁Misch -orously -▁cand -▁cure -▁Lip -▁stro -▁seed -vergabe -▁OEM -▁nachteil -führenden -klick -aires -weitet -▁Brows -▁Herunter -voraus -▁eingetreten -▁Richtungen -atorischen -▁Mittelalter -▁farb -ENS -▁Haushaltskontrolle -▁family-run -▁Schulung -▁Gestern -▁reminded -▁Questions -▁emotions -▁CRM -▁Liikanen -▁mortg -▁wholly -städte -▁persuade -▁Evaluierung -▁weaknesses -▁overseas -▁offenkundig -▁Zagreb -▁Affiliate -swert -во -▁hergest -▁druck -▁rice -▁sé -▁EA -▁BG -▁Bout -▁Kommen -iereinrichtungen -▁repos -teous -▁deinem -▁auszubauen -▁coating -eware -▁ample -zepts -▁outline -elso -ändigkeit -▁berechnen -▁possession -▁restlichen -▁Interventionen -▁varieties -ANN -▁networking -▁optimized -▁Erfahrungs -▁ukra -▁graus -▁Magnet -▁längeren -▁Acrobat -▁Reinigungs -▁celebration -▁strukturellen -▁fehlenden -ubehalten -▁Serviceleistungen -▁komplizierten -▁accomplished -▁fremden -▁irregularities -▁socio-economic -▁concessions -cussions -▁Jarzembowski -oj -ögert -Cs -▁nan -▁fatal -▁wing -▁São -▁Pfl -uri -▁Gothic -▁alien -▁align -leidung -endit -undung -▁shelter -▁polnische -▁Archer -▁Spitz -▁aggreg -chtest -▁Konf --Seiten -▁konstru -ischerweise -lieb -▁writers -▁multitude -▁singer -▁Carmen -▁absoluten -▁Patri -▁persönliches -▁üpp --Welt -▁Öffnen -▁Liberals -▁banana -▁archives -▁1957 -▁Lagerung -▁Löhne -▁Knowledge -▁ermutigt -▁contradiction -▁incredibly -▁Ankara -▁erledigt --Partner -▁Brutto -fo -kos -▁halls -▁Seil -itement -onomen -▁Cany -▁GE -ktiken -▁Zoo -abs -▁confort -▁Jungen -ehow -agua -▁Anson -▁exot -mensch -▁Weber -▁iT -uevo -▁speculation -▁wiederauf -▁experimental -▁Wortlaut -▁pressures -▁Sciences -▁Periode -▁massiv -▁Willens -▁Course -▁allgemeiner -▁draws -▁Anstatt -▁influences -ospace -▁preparatory -▁Estate --Verhältnis -▁IMM -▁replacing -▁Kaliforn -▁GMOs -romagnet -entscheidungen -▁voranzubringen -▁upgraded -▁Cadiz -▁Pestiz -überwachung -300 -▁sü -▁Tul -▁Tay -▁Pensions -initiativen -anmar -▁thou -▁forderte -▁promenade -▁Lago -▁reign -esta -versch -essing -▁Unfälle -▁quasi -▁großartigen -▁debts -▁reveals -▁Themas -▁bekannter -▁Share -▁komment -▁protectionism -▁Berichterstattung -▁generating -▁turnover -▁herausgegeben -▁donation -▁Motion -ambre -▁broadband -schwing -▁Bajor -▁premier -▁ICT -▁facilitating -▁GNOME -laga -▁ranks -▁Platform -▁etabliert -▁inequalities -▁beherrschen -▁echtes -bearbeitung -▁sensiblen -▁schnellstmöglich -▁bevorzugen -ao -wörter -▁Dig -▁Mant -unami -▁Fox -atical -amper -▁reinen -▁ODER -▁descriptions -ersey -ikalen -▁quart -ritz -▁Abreise -▁Lev -ändiger -nez -▁real-time -▁Marie -▁weitergehen -abler -▁einzuf -reffend -▁Fortunately -▁Betrieben -▁Filmen -▁eingegeben -▁tariff -.03 -▁1966 -▁Kampfes -▁Talent -▁Beitrittskandidaten --Player -▁Schmuck -gehörigkeit -▁variations -▁self-em -▁ERP -▁diagnosis -▁Variablen -▁segments --Management -▁hydro -▁Prämien -▁Errungenschaften -▁zusammengefasst -▁Fotokopiereinrichtungen -.10 -hill -▁heter -▁aan -▁sitzt -▁Iber -▁Sag -▁Ble -▁Brat -▁Birm -▁og -▁lehnt -▁Roten -▁tout -estock -▁verhaftet -ibes -▁descend -▁demograf -▁vorang -▁Einsicht -▁Umwand -veloped -▁applaud -▁objections -▁historisch -▁earmark -▁immers -▁Demokratischen -▁self-s -▁Martini -▁exportieren -lässigkeit -▁updating -▁orientieren -▁Sloven -▁Routen -▁Meinungsfreiheit -▁CNC -▁luxuriösen -▁configured -▁disadvantaged -▁exemption -▁IRC -▁macroeconomic -▁arena -▁drittens -▁Lomé -▁TACIS -▁Dubrovnik -▁Sell -▁Posten -▁Buchen -edged -uras -▁Ryan -▁thy -tral -verk -▁deport -▁aufw -▁übersehen -▁vorant -urecht -▁EU-In -▁gesendet -▁Appar -▁Advis -▁Shops -▁Toul -▁ernsten -▁dealer -▁informations -▁Bewirtschaftung -▁shortage -▁Republ -▁Lieutenant -▁Schulungen -ambots -▁built-in -▁purchases -▁Informationsschalter -▁entdeckte -▁obsc -▁vertrauens -▁cosm -▁lifting --Programme -▁Säulen -▁Champs -▁Aktualität -▁drohen -▁Snacks -▁Rabatt -▁herbeiz -▁erholsamen -▁Völkerm -▁Kurdish -▁Erwägungen -conditioning -human --X -46 -ны -▁dal -▁skept -▁widen -erfolg -itations -ofar -▁Wes -▁Rin -▁unp -ulative -▁imstande -▁stamp -▁stieg -▁seid -▁diesjährigen -pling -▁überrasch -▁auszuführen -▁Auswirkung -▁Almost -führende -presso -▁langjährigen -▁Neigh -ronen -▁blicken -▁ästhet -▁multip -▁chef -▁Patente -▁fragt -▁speeds -▁orientiert -▁Annan -wechselt -▁Jesolo -▁Kommunismus -▁elektrischen -▁Olivenöl -▁anticipated -ocopying -▁Mobiltelefon -▁Heinrich -▁fähig -läßlich -▁Ansonsten -räder -seiten -bels -IZ -UD -▁mos -▁Dimensionen -▁BBC -esar -onne -▁Göte -▁instruction -▁ankommt -▁Liga -▁reper -▁Zentrale -platte -indruckt -▁austerity -▁2050 -▁Commons -▁einziger -▁Politikern -▁retreat -▁cultivation -▁Komit -bauer -▁Asset -▁Perf -▁Oppositions -▁grain -▁attracted -▁consultants -▁springs -▁poorer -▁wissenschaftlicher -▁Fischereif -sprüch -▁Mailing -▁rigid -▁kontinuierliche -▁überschreiten -▁Rapid -▁Diamant -▁Controller -▁negligence -▁tissue -▁rhetoric -▁Rücktritt -▁Einstimmigkeit -män -bag -emos -▁Wave -▁Vac -▁begehen -stad -schiffe -ingang -▁einwand -▁Ott -▁vorsehen -▁disprop -▁Auftreten -▁outbrea -▁Umstand -▁preview -▁monastery -▁attain -▁ausgeht -▁Noten -▁Energieverbrauch -▁Universitäts -ISS -▁Moder -▁verschlechter -.01 -▁Maybe -ICK -▁Meta -▁sofortige -▁Folgenab -▁Milliarde -▁Telefonnummer -▁voraussichtlich -▁corresponds -▁tackled -▁hospitals -▁tutorial -▁Fitnesscenter -oeuv -meisterschaft -▁beurteilt -▁unberühr -▁christlichen -▁bulk -▁renommierten -▁Behauptung -debatte -▁freigegeben -▁bizar -▁catal -▁laptop -▁Cass -▁Hang -▁Vital -kthrough -ifically -griffe -▁Erforschung -▁Erwärmung -obi -schaftliche -▁unterw -▁accus -▁outer -▁elimination -gets -▁productions -▁glän -▁angegriffen -▁feier -▁Trading -ync -▁Umwelt- -▁electron -▁137 -▁Lebensdauer -▁march -ylon -▁personalized -▁Weitergabe -▁Norweg -▁Änderungsantrags -▁relevance -▁self-d -marken -▁Campo -▁1948 -zwecke -▁altogether -▁wertvoll -▁angesprochenen -▁delightful -▁Akteuren -▁Anderen -▁ungehe -▁Bevölkerungsgruppen -▁Trinkwasser -▁Fifth -▁â -KR -▁fusion -▁solle -▁Tan -▁bulgar -esischen -ilber -▁Figur -hew -▁undem -▁erweist -▁eindring -▁Occ -▁Spass -beob -▁indul -utzte -▁Margar -klagt -teilte -▁Brem -ämmen -▁factories -▁complied -▁1800 -▁Personals -▁gebühr -▁Eurojust -▁Screen -▁strengere -▁Sozialmodell -▁Jahrhunderten -▁Entscheidungsträger -▁widersprechen -▁Milano -gesetzgebung -▁breiter -▁Beobachtung -▁Streitigkeiten -▁GUI -▁derartiger -▁Geheimnis -▁freiwillige -▁recipient -▁Diplom -▁Wassersport -▁Varianten -▁würdigen -▁injection -uung -wäsche --end --ray -43 -▁fakt -▁Bene -irds -▁KU -▁Cohn -▁Cocktail -▁Gor -▁ankommen -▁imply -stage -▁Orange -assy -laven -ochten -▁Beck -▁traf -▁EU-K -▁interoper -▁Mitgliedstaates -finanz -▁qualification -▁flora -▁True -▁signatures -▁Strafen -▁240 -exe -▁Films -▁strongest -▁Geldpolitik -▁temple -▁Options -▁renovation -▁Otto -▁Geduld -▁Netto -▁exercises -▁höchstem -▁eliminated -▁journalist -▁bevorzugt -▁verschärfen -▁beherrscht -▁Fontaine -▁sozioökonom -zn -▁apt -▁Tirol -▁Bost -▁cave -▁CMS -▁Gord -▁beziehe -opfer -▁versp -▁gezahlt -▁Ung -izierten -ipedia -ardin -okolle -▁truck -▁bridges -reichende -▁Clients -▁Beihilfe --Dienst -manager -▁distances --170 -▁gerechte -▁drawings -▁Schweine -spieler -▁prüft -betrieben -▁teeth -▁Trocken -▁midst -▁matching -▁vegetation -▁Atomwaffen -▁conservative -▁Nahe -▁multinationalen -▁vorbehalt -▁tanks -krebs -▁müßten -▁Straftaten -▁telefonisch -▁incurred -▁Enttäuschung -▁Caucas -▁repercussions -▁з -iw -nummern -fly -añ -äb -▁Task -▁S-Bahn -▁Deg -▁Around -▁Centr -▁Hü -▁beige -stationen -quart -▁repar -▁neo -▁asymmetr -▁demnach -▁Stütz -▁frames -oured -pflege -▁Manip -▁socialist -▁Formate -▁powered -▁Umweltpolitik -▁annehm -▁barely -regional -▁diameter -▁Myanmar -▁1961 -hauptstadt -▁Logos -▁koennen -▁Fotografen -▁Schengener -▁fireplace -▁Typen -Star -▁Übernachtungen -▁Bearbeitungs -▁veranl --Unter -▁ALDE-Fraktion -▁Continental -▁enchanting -▁forbidden -tätigkeit -fabrik -▁mp -▁Tort -▁TCP -▁Skript -▁BB -▁cocktail -alters -istin -▁Züge -▁deployed -▁notify -▁compress -▁chip -ogh -udia -▁resident -▁Auswertung -▁Einmischung -bez -ammelten -▁Cham -▁Leop --Shop -▁Alltags -▁befand -▁documented -▁third-country -▁offener -▁deliveries -▁Suchmaschinen -▁animation -▁1964 -trolled -▁Keep -▁architects -▁hygiene -begriff -bekistan -▁Roboter -▁Satellit -▁Bibliotheken -▁Terrain -▁propaganda -▁behindertenfreundliche -▁Patrick -▁Römer -▁Certainly -▁herbeizuführen -dies -▁atr -▁sister -▁wins -▁wovon -▁Purch -▁gaps -▁Kün -▁Rinder -▁Lieder -▁Instand -lichungen -▁Jap -ugar -prinzips -▁shores -ätt -▁discon -▁Jahrt -▁guiding -▁103 -▁Attrib -vorgaben -▁gleichnam -▁Äng -▁successor -▁defect -▁Modellen -▁geradezu -äuschen -▁Willi -▁Fehlern -▁Familienzimmer -▁1962 -▁trock -▁billions -▁Salzburger -▁screening -▁stattfand -▁refresh -BSD -▁promptly -▁Substanz -▁households -▁Vertiefung -▁Bezirk -▁rehabilitation -▁Bettwäsche -▁Überschwemmungen -▁Alzheimer -währung -Do -на -edict -ilde -▁Walking -▁Kamer -▁Khan -▁Cell -▁Race -▁VP -▁unauthor -▁Lj -ername -▁zugewiesen -▁erleb -▁Join -▁OM -▁Sturm -▁entschuldigen -izierte -eschl -▁Einigkeit -▁polizeil -▁finnischen -▁disrupt -haut -frem -ouver -▁incompl -▁decides -▁Bristol -▁Rathaus -brücke -▁medi -venient -▁Autof -▁organizing -▁motto -▁republic -▁momentum -▁Miles -▁Gesundheitsversorgung -▁boarding -▁occasionally -gesetze -▁Outdoor -▁Synchron -▁kombinieren -risiko -▁diente -▁mig -▁wearing -▁Tester -▁Kitz -▁Kath -icia -arme -▁Lew -▁Nin -▁itiner -tera -▁Jones -zeilen -reichend -▁Unterscheidung -php -▁crop -▁scrat -▁Trent -undering -▁positioned -▁135 -hofer -▁college -▁unterstützten -▁Centro -▁Profile -▁Coup -▁nachhaltiger -▁Fachleute -▁blocking -▁Gehäuse -▁transparenter -▁Buyer -▁Lamy -▁Communications -▁angestreb -▁confronted -▁crafts -▁heaven -▁recreational -▁Treffpunkt -▁pragmatic -▁herausragende --Mittelmeer -▁Herangehens -▁fabrics -▁Uganda -▁Pressemitteil -ruption -GI -▁raffin -▁MIT -▁gourmet -empel -olar -▁linen -▁lush -▁Cir -▁Cord -▁GAT -ieft -ankt -▁pret -verg -▁verification -▁Jury -enner -angaben -aille -steigt -rique -▁Neug -▁Qualifikationen -▁sensors -bound -▁broaden -rechtliche -▁massac -▁aufgezeigt -▁vereinen -▁Kleine -▁Ideologie -▁überwunden -▁eindeutigen --Zimmer -▁Etappe -▁Drittländer -▁junior -▁Reservierungen -▁Griechenlands -▁dramatisch -▁beträchtlichen -▁Ferrero -▁commissioned -▁schwarze -▁livestock -▁Raketen -▁enshrined -fish -förm -IAL -cements -lighten -kab -GR -ле -▁Early -▁Pick -▁Belle -eno -▁lend -▁unvollständ -iculous -opro -▁zuver -antast -▁ausschlag -▁nachgewiesen -▁Arn -▁4000 -▁Reception -▁Yesterday -▁Probe -▁Seitdem -ursor -▁Kron -▁Zuwanderung -▁Rohr -▁Italia -▁draußen -▁verschle -▁intensity -▁Starting -▁simpler -ägten -▁Vollendung -▁üblicherweise -▁erneuerbarer -▁beizubehalten -▁Naples -▁ordnungsgemäße -▁makers -▁Treibhausgas -▁aspirations -▁Bomben -▁Energietr -▁Verknüpfung -▁Slideshows -uary -wicht --way -▁font -▁Batt -irgend -ilden -ilater -onet -▁Kameras -▁HA -▁unaus -atial -alm -▁anget -▁beziehungs -▁Lunch -verzeichnis -▁verlin -▁Verwirr -prot -▁manoeuv -▁Aussprachen -▁trace -▁Aufschwung -ammlungen -▁geschm -▁Bestes -▁aktiver --Tra -▁extremen -▁Ratspräsidentin -▁administrator -verträglich -sprachige -▁Erinnerungen -▁geared -▁Promotion -▁entrepreneur -▁liquidity -▁Resultate -▁Administrator -▁symptoms -▁Jugendherberge -▁Málaga -trockner -▁reaffirm -▁Vitorino -▁Göteborg -▁unterworfen -EY -öhne -▁Sendung -▁Dj -▁Ponte -eader -dehnen -▁Neap -▁deploy -▁Jas -forward -▁Schmerz -▁schlafen -ausstellung -▁Aussteller -iac -ologist -▁alleg -▁protein -ministration -▁brown -▁Programmier -▁strategischer -▁slide --Center -▁Zusage -▁Moral -▁licences --Up -▁reichlich -▁Zivilisten -▁Friedensprozess -▁Built -▁seasonal -▁Haftbefehl -▁Serbian -▁compensate -▁Evans -▁Ethiop -▁juice -▁sahen -▁Photocopying -▁Seville -agenturen -▁Maximum -▁appellieren -OFIN -▁photographer -ubiläum -▁überraschend -hamm --one -▁vel -▁wendet -onal -▁Fabr -▁Gehen -ivile -icter -ichere -▁storn -▁Verhaft -▁Oase -laf -lava -▁manuell -▁aged -▁warme -aufnahmen -▁Esc -▁trium -neum -▁liegende -▁scal -▁versatile -▁acoust -▁privilege -▁Schwäche -▁communist -▁posting -ASP --Wald -▁promotional -▁Globalisation -▁millenn -▁motivation -bilden -▁vulnerability -▁blieben -▁Miguel -▁Passagier -▁Schwarzen -▁Kürzung -▁conversation -▁eager --Krise -▁Columbia -▁kümmert -aukasus -▁beziehungsweise -hots -▁tied -▁nar -▁AA -▁inaug -▁LIFE -▁stellten -dena -▁Anlaß -▁übersetzen -olls -▁chains -▁enters -▁jedermann -▁triple -▁Umbr -▁substitute -▁hoteleigen -enza -▁Staub -END -▁existierenden -▁Speise -▁Fischfang -.08 -▁Edward -▁Holocaust -zwingen -▁Immo -▁Morgan -▁Veranstalter -▁Pfund -▁Audit -shine -zuladen -▁Biom -▁trek -▁Standorte -▁colourful -▁ausgedehnt -▁Konsultationen -▁gezielte -▁E.ON -ergeleitet -▁Mahlzeiten -▁Dampfbad -▁ALE-Fraktion -▁WWU -▁biotechnology -▁veranlasst -bing -cup -Cl -ähn -▁mask -▁wahl -▁Emer -▁Mans -▁gaben -ità -▁Fou -▁streichen -▁Zend -igenden -▁ausschl -▁cozy -▁clash -▁Spur -▁Abwesen -▁gröss -▁Leucht -vents -▁placement -▁Ära -▁dryer -▁Rechtsetzung -▁tuna -▁Bildes -▁Kundendienst -▁ägypt -▁irrit -▁irakischen -▁versteckt -▁specification -▁Australian -▁sooner -ifikations -besitz -▁ethnische -▁earliest -▁Freundschaft -spartnern -▁konstruktiv -▁Behinderung -▁leistungsstark -▁zügig -▁Besichtigung -▁density -▁disproportion -▁Cohn-B -vation -üff -78 -▁hub -▁rush -▁seating -▁Tiger -▁pec -▁Bauer -unft -▁ehe -▁Kampagnen -▁inex -opath -▁eröffnete -▁einb -▁Verschied -▁Verspät -▁Jh -▁excluding -▁Finden -▁Sounds -▁discounts -▁Managing -▁financially -▁Schwester -▁informellen -▁unterstützte -strahlung -▁tariffs -▁Cox -▁Eingliederung -▁Knochen -▁billiger -▁Österreichs -▁geleistete -▁snack -▁accurately -▁A320 -▁alarming -sprecher -▁phenomena --speaking -▁Verpackungs -▁Brunnen -▁air-conditioning -hältnismäßig -▁auferlegt -▁unabdingbar -▁Kontinuität -▁É -rel --quality -We -▁bun -▁SF -▁pitch -▁Brad -▁cater -▁lobb -▁Had -▁Fut -ivial -amour -quin -aching -upro -▁aufkommen -▁Vervielfält -einrichtung -ogas -▁Arte -seud -bled -▁Leone -loaden -▁108 -▁somehow -▁longest -▁Tode -uzent -▁competing --Restaurant -apsed -▁215 -▁broke -aneous -▁biometr -▁merit -rücke -▁erfolgreicher -▁Telec -▁earnings -▁exceeded -▁displaced -▁drastisch -▁herzlichen -▁grundsätzliche -▁Kupfer -▁Cohn-Bendit -bury -kol -keln -▁dot -▁TOS -▁Eg -ettern -etches -elekt -▁Writ -▁Herman -▁Fu -▁Ve -▁Gig -▁offl -schwer -roid -▁umliegenden -▁Mitteilungen -▁Alan -▁butter -▁Maust -garde -▁seiten -▁designing --Komp -▁nette -▁distanz -▁non-c -▁Karib -▁innovativer -▁Komposition -▁Motiv -▁Terme -▁ernsthaften -▁sixth -▁restriction -▁Messung -▁Einwanderungspolitik -▁Überwachungs -▁Evolution -▁geäußer -▁ökonomische -▁verweigern -▁Vizepräsidentin -▁Kohlend -▁royal -▁Schokolade -▁Ecuador -▁Verwirrung -700 -▁boys -▁gi -▁Ale -unas -▁ci -▁GD -▁JPEG -▁Stücke -▁Schatz -▁Schaff -▁mais -steuers -feindlichkeit -▁hierher -ioden -▁Ask -▁assort -▁screw -▁issuing -▁parte -▁Zuhause -▁simulation -▁verletzen -▁deutlichen -▁Haushaltsmittel -▁peaks -▁Doctor -▁AGB -urfing -▁expecting --rate -ländischen -▁Applikation -gesetzte -▁Mandr -▁archä -▁Haftungs -▁nötige -▁thrown -wachstums -▁freiwilligen -▁shapes -▁zitiere -▁Ausgleichs -▁Sharon -▁jenem -▁benachbarten -▁reimburs -▁Kalifornien -turn -fire -Ex -▁fount -▁Ice -▁powder -▁Mike -itärer -utor -▁kroat -▁Wäre -▁Waffe -▁Chen -arious -▁neur -▁verified -esso -ugo -▁projekt -▁-M -▁atmos -▁abb -▁chronic -▁unterzogen -▁suits --Pal -positionen -▁severely -▁Mayor -▁Beförder -▁ähnlicher -▁worthwhile -beginn -▁folgender -▁Verschiebung -▁emergence -▁fertigen -▁rentals -▁Renault -▁Fitnessraum -▁assumption -▁detailliert -▁vertrete -▁abolished -▁einschränken -▁reflekt -▁Omn -▁veränderte -▁aesthetic --American -äme -72 -▁dated -▁bone -▁Warm -▁Kro -rev -verordnung -▁mitunter -▁werdet -▁entzieht -oves -▁Beil -▁Bezüglich -▁Arctic -▁stricter -▁Haag -▁formulated -▁Trainer -▁Schwach -gewichts -▁Spielzeug --Ru -▁Kinderbetreuung -▁Skip -▁critics -ARC -▁Supreme -▁Friedhof -▁magical -▁Fluggesellschaft -▁liberalen -▁Vertreterin -▁erschwert -▁McK -▁elektronischer -▁Available -▁criticisms -▁Pinien -▁mothers -▁außergewöhnlich -▁Secret -▁injured -nergies -▁magnetic -▁frozen -▁zufällig -▁lifetime -▁geänderten -▁Tastatur -food -▁feinen -▁räum -▁pub -▁BO -▁Rasse -asur -ufung -entes -▁jac -▁Oster -ertoire -▁mex -▁3-Sterne-Hotel -▁genügen -stellers --Stra -▁realities -▁strip -eated -vorschrift -▁backdrop -▁Elis -▁Firm -ATA -feldern -▁Sichtweise -▁1944 -▁Ostse -▁timber -▁sinnvolle -▁holders -▁beeindruckt -▁Botschafter -▁Asylbewerber -▁tastefully -▁scenic -building -▁portugiesische -▁niedrigeren -▁ungarische -▁EBIT -▁detriment -▁aufrichtig -▁memorable -▁ICEcat --discrimination -▁constituency -▁Beschleunigung -▁html -▁б -oise -renten -xen -ção -▁Tech -▁Dame -▁B2 -esst -▁zurecht -▁caut -▁Kund -▁infin -aring -▁N. -iffern -▁einm -eiter -endel -wei -▁schalten -obel -▁zumeist -ampe -▁hebt -▁650 -teilten -▁befaßt -▁Traff -▁zusammenhän -▁mußte -▁Zukunfts -▁Nationalen -▁Verantwortungs -▁Pariser -▁containers -zube -▁Informationsaustausch -▁Gemeinschaftsrechts -▁Automation -▁poet -▁ethischen -▁laufende -▁GMO -▁Entscheidungsfindung -▁bewährten -▁Differenzen -▁encryption -▁demonstrating -▁namentlich -.9 -disk -BO -▁ter -▁fügen -▁pity -▁Points -▁Ful -▁anführen -entum -▁zugenommen -▁zuzustimmen -▁Zar -▁Ohren -ehörde -▁mankind -▁Auschecken -acon -▁argu -▁Images -▁Bedauer -▁valleys -▁varies -▁Preises -▁Phoen -äufers -▁breakthrough -▁Zugeständ -▁informative -▁Skulpt -▁treating -▁generates -▁optimization -▁Wertpapier -▁establishes -▁purchaser -▁Europäern -▁ICC -▁eingeschlossen -schulden -▁clarified -▁defended -▁Likewise -embles -▁begeistern -▁abstract -▁devastating -▁zertifiziert -▁Maßstäbe -▁ermordet -tal -beam -▁fahr -▁Ty -▁Sack -▁Prad -▁Byr -▁Bungalows -▁o. -▁Kalten -▁Gates -▁verh -▁einsehen -▁Oomen -▁cluster -▁Abbas -steine -▁Berecht -▁geschützten -▁Besten -▁Particular -▁Stehende -oppeln -▁Beleg -▁installer -▁Mehrzahl -senal -▁patio -▁Rückzug -▁ECOFIN -verbände -amba -▁Vorteilen -▁dringende --ranging -▁gekürzt -▁auszuarbeiten -▁geringeren -▁literary -▁beschriebenen --screen -▁anspruchsvoll -▁palästinensische -▁einseitig -▁geografischen -▁ILO -beteiligung -▁Freiburg -▁Sogar -▁beneficiaries -hut -wier -▁Taking -▁SIE -▁pig -▁Mauern -eton -▁lays -▁Kun -▁Köst -▁Cott -▁Hope -▁Zonen -upport -forsch -▁entail -▁schott -▁3.2 -▁Auflage -▁5.1 -▁hindern -attered -▁Expo -▁Amn -▁12th -▁Blatt -▁Admir -▁confused -wirtschaftlichen -▁identifiziert -.06 -▁Amsterdamer -▁Independent -▁physikal -▁licenses -bezogenen -▁kitchenette -▁destroying -▁picked -geschäfte -▁Orchester -▁Millenniums -▁grande -▁doppelte -▁Prozessen -▁Etwa -▁Präzision -▁beschädigt -elsohne -▁incomplete -▁tongue -▁Tall -▁bikes -▁STO -▁Dail -▁pued -▁Horm -▁Gur -anche -▁Luis -iks -▁Schuhe -ipse -▁Paragraph -▁commem -▁weitergeleitet -▁Haf -▁101 -▁phases -ständigen -▁Steine -▁gemisch --Dokument -▁documentary -▁Drink -▁Sozial- -▁Planungs -▁illegaler -▁Sicherheitsrat -▁expects -▁recordings -▁Pfeil -atastroph -▁Experiment -filter -▁vertraulich -▁moralischen -▁Package -▁eingebettet -▁subsidiaries -▁Investitionsbank -▁Arzneimitteln --Vertrags -▁forecast -▁dokumentiert -▁postponed -▁icons -▁Breitband -▁Maustaste -▁Mandriva -sider -cía --owned -DN -▁vi -▁cache -▁Wonder -▁Heut -▁unwill -ieh -umption -▁UM -▁Lücken -stufen -ová -▁Vorgänge -▁gesund -html -▁endanger -▁Exil -▁Neuf -orrent -ERT -▁binary -▁Russians -▁Quant -▁lawyer -▁einmaligen -ijten -serklärung -▁totalitarian -▁copper -▁Nowadays -▁Agrars -▁probieren -▁Knoten -▁peninsula -▁mangelt -▁Tibetan -▁Wartungs -▁Schnittstellen -▁awaiting -▁kreativen -spolitische -▁forecasts -proofed -▁Medikamenten -▁Verschmutzung -▁empfehlenswert -▁pesticides -kampagne -ähre -GO -▁afghan -anet -arest -▁gewohnt -▁Anne -▁shipment -▁chaos -▁vorab -▁Beein -▁trac -klassen -▁avant -schließung -▁discour -öglichkeiten -▁15.00 -▁Finanzdienstleistungen -▁Elba -▁annimmt -▁Koalition -▁Turbo -▁Wahlkampf -▁participant -▁mounting -▁Handels- -▁unabhängiger -iciencies -▁exported -▁bestimmtes -▁beliebte -aminated -▁1990er -▁indicating -▁romantischen -▁geehrten -medien -▁repräsentiert -▁Currency -▁Erfordernisse -▁precedent -▁benachteiligten -▁encrypt -▁gescheitert -▁Sebastian -▁Diplomatie -▁Pensionen -▁Architect -Pr -Pl -BI -▁rises -▁morg -▁Toy -▁Ech -▁pic -itution -otor -▁oven -▁Hamm -▁Laus -istisches -achth -▁neck -▁Prices -▁Anschein -ikum -▁coex -ulture -▁Einverständnis -▁gene -avan -▁sciences -ymph -▁persistent -▁Gewiss -▁Russen -▁zusammengearbeitet -▁begriffen -▁additives -▁transportiert -kaufen -▁Ortes -▁Tageszeit -▁translators -▁nachhaltiges -▁équ -▁enforced -▁nachzukommen -▁Lehne -▁Erscheinung -▁unnötig -Guide -▁recruitment -beschäftigung -ovolta -▁Herunterladen -▁zuversichtlich -üte -Disk -kühl -ÜR -▁SERV -orenen -▁lyr -ürgen -▁beides -idation -▁zugeben -▁Ziffern -▁Jeg -eiteten -plänen -▁Andernfalls -▁aussch -▁Ausländer -zeigten -▁Yahoo -▁likewise -▁anybody -▁submission -▁inval -▁nationality -▁smell -▁Daß -▁Apoll -▁Versuchen -▁Qualifikation -▁Malaria -▁recommends -▁failures -▁foreigners -▁1963 -▁angemeldet -▁corrected -▁90er -▁1958 -▁Underground -▁Ausflugs -▁Evangel -▁advocates -▁potenzielle -▁inherent -▁Nutzungsbedingungen -▁incoming -▁Mobiltelef -▁Rüstungs -▁eBay -▁½ -zell -▁vague -▁Pause -▁Cus -▁Capt -▁Hugh -▁RC -icul -▁erricht -▁Stable -pton -▁Auffassungen -zuwirken -▁overr -▁Allen -▁Brig -krip -▁scan -piele -▁verspricht -▁123 -▁Particip -ERE -▁Adopt -▁jeher -▁definite -▁Organismen -▁Magazin -▁Öffentliche -▁conditioned -▁connects -▁liberalization -▁Castel -▁reconstruct -▁criticised -▁Schöne -▁geringfüg -▁radioakt -▁komplexer -▁café -▁Siedlung -▁democratisation -▁Sammel -▁reichhaltiges -▁einzuräumen -▁E-Commerce -▁NCC-170 -▁Verbrechens -▁Caucasus -▁ub -▁pap -▁Boy -itas -unge -▁Hals -▁verkehr -essp -promenade -▁abd -▁Weing -ardi -erverein -▁leid -▁Regisse -rium -▁Landhaus -▁0.5 -▁Barón -bewertung -▁ausgegangen -ähle -▁dreißig -haftigkeit -▁Linke -▁Geschäfts- -▁Schlussfolgerung -▁illegally -▁angenommene -▁travelers -▁illumin -▁sexy -▁likes -ografie -▁occurring -▁entstehenden -▁slowen -▁Richtlinienvorschlag -▁Begriffs -plantation -▁Customs -▁Louvre -farben -▁Voice -▁gifts -▁Dalmat -▁gratitude -▁mediterranen -▁Matthias -▁Verschiedene -nels -pul -29 -▁Dup -▁Besondere -etik -▁cav -▁Lan -rages -▁Ing -ifor -▁Jeff -▁worsen -▁Mitgefühl -ossa -elevant -▁submitting -▁Apost -▁ausgeweitet --Fin -▁accountable -▁Disco -▁civilisation -▁allgemeinem -▁demonstrieren -▁eingesetzten -▁Laufenden -▁Kurden -▁1949 -ulsion -▁stattfindenden -▁1939 -basierten -▁punish -zepten -▁erbracht -▁stilvolle -▁Ergänz -▁cafes -▁elektrische -▁composer -▁inserted -élé -▁molec -▁NGL-Fraktion -▁dezentral -belastung -▁misleading -▁Prozentsatz -▁Ferrero-Wald -mart -I. -points -▁nous -▁sä -▁veto -▁ml -▁wander -▁Tabl -▁gast -▁KL -▁Viv -icos -idos -▁stack -▁4-Sterne -▁bisweilen -▁console -▁subt -▁incap -▁Grunds -▁appalling -▁getät -▁Thematik -▁Meinungsver -optim -▁Quartet -geordnete -afel -▁maximalen -▁Mittele -▁friendship -▁anerkannte -▁Münzen -▁Aviation -▁radioactive -▁Währungsfonds -▁Azur -▁Vernichtung -▁Rentner -organe -▁tastes --Datenbank -▁licensing -▁unzähligen -piece -▁Assoziierungsabkommen -otherapy -▁Nürnberg -▁Ferrero-Waldner -fle -`s -▁tent -▁afor -▁Tin -▁Mother -▁Britt -urers -▁lächer -▁Capp -▁ya -heel -tres -▁Style -assis -assing -▁umzu -▁Einheim -▁quer -rengung -▁bestmögliche -nergy -▁Umfrage -▁hinten --Agent -▁Poliz -▁Novotel -▁instrumental -▁Terroran -entwicklungen -▁Anzahlung -▁Taj -▁Hyde -▁imagin -▁Banquet -▁gladly -▁inspiriert -▁assumes -▁Brazilian -▁DivX -▁Getränk -▁Sommers -kreuz -▁Demonstrationen -▁idyllic --Paket -▁stellvertreten -▁lady -▁FIFA -▁Boston -▁̈ -zes -▁ros -▁borr -▁Dereg -▁Mé -isible -▁prizes -▁Lücke -▁Nähr -▁gefl -▁Schichten -▁auszuschließen -ateien -ahlen -stext -▁subway -▁konvert -ceptions -▁seri -ministers -▁Tram -▁brush -▁Bürgermeister -bedürft -▁schlug -▁Kombin -▁angesiedelt -▁Nationale -▁Grüne -▁Verbraucherpolitik -▁Einzelpersonen -▁judg -▁oils -▁high-level -▁reflecting --winning -▁breed -▁verzichtet -▁Sprung -▁Aktionsprogramm -▁kostspiel -▁Tschechische -▁Gaulle -▁disappointment -▁erhöhten -texts -▁Meldung -ifffahrt -▁NCC-1701 -del -sektors -▁mead -etal -▁GASP -isting -▁ersucht -▁einleiten -▁allerg -▁Schande -ollt -▁4.1 -ransport -▁offensive -▁assass -▁handy -▁Interoper -▁accessing -▁Daim -▁Steve -▁befest -▁Schwed -▁objection -▁menschen -▁Malware -▁Wahlkreis -▁generic -▁Carlo -▁violated -▁harmonise -▁Termini -▁trocken -▁Mannheim -▁Eigentlich -▁anerkennt -▁dramatischen -▁caution -▁deposits -▁ruin -▁twenty-f -▁niederländische -▁gestrigen -▁vorübergehend --Website -▁deterioration -▁Speichern -▁biofuels -isciplinary -▁self-employed -▁Meinungsverschieden --ins --res -öck -FF -600 -ет -▁honey -▁ville -▁Banner -irect -▁anonymous -opic -▁Verweis -▁entails -▁3G -▁4.5 -▁freiz -ologists -▁Plural -▁Decken -▁wound -▁geschaff -▁gleichzeit -▁Neuse -beding -▁verstößt -▁understands -▁regulating -▁fairs -▁Sozialfonds -behand -▁Nordamerika -▁Delph -▁gefährd -▁urged -▁natürlicher -▁Mexican -▁Accordingly -▁Kyoto-Protokoll -▁Various -▁Beamte -▁Böge -▁kingdom -▁erstklassigen -▁Wolken -▁Sonnent -▁intermediate -▁vorherrschen -▁Gründungs -▁verschlechtert -▁fiber -▁fitting -▁bell -▁brachten -▁Darm -▁Eier -▁pisc -onische -▁Kurt -▁Kofin -idy -verfolgung -▁eingebaut -▁demo -▁chest -▁Grä -▁counting -▁contamination -▁gespeicher -werden -▁distortion -▁expresses -▁drain -▁eingehend -▁Submitted -▁beginners -▁appropriately --state -▁Obergrenze -▁Summ -▁Wettbewerbspolitik -▁penetr -▁mutually -▁beliebig -▁Philharmon -▁Oscar -▁Registrierungs -▁MSN -▁fashionable -ulichkeit -▁Throughout --Modus -▁surfing -▁erect -▁Honor -kämpft -▁Animation -▁Horizont -▁Zigaretten -▁Hoheitsgebiet -LD -▁scept -▁Tell -▁Dos -▁Bäume -▁Asp -chner -▁Unic -▁alla -weiger -▁auszut -▁manages -▁traces -fläch -ermächtigungen -▁first-class -▁delic -fürst -▁Handwerk -hindert -▁Trip -▁brasil -isedatum -atorische -▁soziales -▁authoritarian -parl -▁Guar -▁Airconditioning -▁showers -▁appearing -▁donations -▁capitalist -▁lacks -▁administrativen -▁Knopf -▁celeb -▁Salzkammer -▁bequemen -▁Entsprechend -öpfer -farbe -▁Brenner -▁ausdrückliche -faktoren -▁Schaltfläche -▁Tuscan -▁Pedro -▁tournaments -▁atemberaubenden -▁у -.d -hungen -ugend -LY -▁hellen -▁Sv -▁Sites -▁Amal -ilität -usen -▁inhaft -▁angrenzen -▁yea -▁notable -▁Schäd -igenz -▁yours -▁Auftritt -▁emails -▁realization -▁Gesichts -▁festgeschrieben -höre -▁zurückgreifen -▁Gleichberechtigung -▁optimize -▁Verantwortlichkeit -▁García -.07 --Inter -▁spezielles -▁resistant -▁Freundlichkeit -▁Verbrauchs -▁verkörp -▁beträchtlich -▁disappointing -▁kurzfristige -reinigung -▁blauen -▁enthusiasts -▁Bündnis -▁vergrößern -▁therapeutic -▁Invasion -▁Crespo -aufenthalt -▁wholeheartedly -▁Oomen-Ru -jour -▁töten -▁nest -▁fuck -▁apr -▁Tours -▁Pok -uni -▁zögern -▁VII -▁GA -▁Za -teenth -abu -ckung -▁orn -▁Story -▁entschloss -▁Imag -▁Orl -▁Monster -amerikas -▁Urlauber -▁Gemeinschaftspolitik -▁Belange -▁Moon -▁Buchten -▁headed -▁bisherige -IDE -▁Mehrheits -▁INI -▁fauna -▁Arabia -▁oberhalb -▁höchstens -▁intellekt -▁Stadtr -▁Mitgliedsländern -▁Bangemann -▁zuteil -▁Verdacht -▁Haartrockner -▁Concert -städten -▁consciousness -▁Amazon -▁Challenge -▁Abwesenheit -▁offline -uential -lander -▁ni -▁pubs -▁cocktails -▁kub -▁K. -▁Calle -athl -▁alqu -owa -▁zukommt -▁verfassungs -▁weite -endete -▁vorzustellen -▁vorgelegte -epper -▁lieb -▁locals -▁Communities -▁findest -zugleichen -▁Address -▁Bosch -▁static -▁slot -▁Speicherung -▁Carn -▁Christoph -▁grade -drücken -▁traveling -▁judiciary -▁exhibitors -▁Gesundheits- -▁hochwertiger -üllt -▁Klein- -▁mixing -▁1933 -▁kristall -▁Champion -▁gemeinnütz -▁Cuban -▁inhaltlich -▁unverzichtbar -▁getrennte -▁Nevada -▁Mitchell -bogen -▁Session -inische -▁zivile -▁Fi -▁Fashion -▁unclear -ieber -stanz -▁stören -adores -ecuted -▁Vorzüge -auftrag -▁Geis -▁Geister -▁conceal -▁geschlagen -▁Bestehen -▁betrug -inski -▁childhood -▁Durchschnitts -▁erneuert -▁Dienstleister -▁Wortes -▁organisms -register -▁Opport -▁widersprüch -IPS -UST -▁dispatch -▁Automotive -▁ecommerce -▁Kyi -▁Pressefreiheit -▁Nuclear -▁straightforward -▁Getränken -▁awaits -▁großzügige -▁irresponsible -▁heiligen -▁Etwas -▁sentences -▁jüdischen -▁wöchentlich -▁adapting -▁chapters -oons -faden -die -kular -▁rote -▁Dil -esy -▁e-c -▁Werner -▁HE -▁Favor -▁übergehen -grä -▁ausmacht -iani -▁disability -▁internally -▁Commerce -▁esta --Benz -▁acute -▁Butter -▁musst -bauern --minded -afi -sender --Herz -wandeln -▁benötigte -▁professionalism -▁influential -▁öfter -hesive -▁Kleink -▁Liberties -▁Südkorea -▁Suu -▁Italienisch --looking -▁stimulating -▁Namib -▁exzellenten -▁champion -▁ausdrücken -▁Freihandelsabkommen -▁Javascript -praxis -▁verdeutlicht -lysées -si -lernen -LC -xx -▁drew -▁pending -▁pdf -▁Pion -elier -unnen -▁ink -▁anged -schild -istans -istically -▁aufbew -▁Schild -ermin -▁poses -▁Rechtsanw -▁Urug -▁explizit -▁Goals -▁low-c -▁encouragement -▁Handelspolitik -▁280 -▁judged -▁Ruinen -▁Brava -▁Hybrid -▁Büchern -▁Ambassador -▁acquaint -▁Surround -▁laufend -▁Nicolas -▁gesundheitlichen -▁transposition -▁Figuren -messungen -▁Geltung -▁auslösen -▁peripheral -▁Racing -▁proprietary -▁bewaffneten -▁Ressource -▁Birmingham -▁Oomen-Ruijten -hart -rand --se -kenn -KO -ло -▁Sorten -▁Bein -▁Aids -igan -▁Create -▁Hemm -▁Ries -decke -atischer -alone -▁LV -▁onwards -▁aufgreifen -▁vermin -▁chambre -izers -schafft -▁Abänderungs -▁jedw -älern -▁interc -arbeitete -▁Überzeug -▁guys -▁Seien -oleum -▁suicide -ähere -zielen -▁11th -▁Malm -▁Nationalismus -▁palm -▁1959 -▁applicants -▁robot --Konferenz -▁Dominion -▁Verlagerung -▁jenes -▁neoliber -▁Computern -▁kathol -▁hinnehmbar -zten -mand -▁Iss -▁Mw -itology -▁unbea -berater -▁Leinen -verd -pler -allo -eca -▁leer -▁Vorreit -zukommen -▁trucks -▁Reed -▁Yam -ührung -▁moon -▁scar -▁Produktpalette -▁parish -▁Manufact -▁Blues -▁Änderungsvorschläge -▁Notfall -▁traditioneller -ikanern -▁Companies -▁Nova -stücken -▁Motivation -öhnt -▁80er -▁Login -▁vernichtet -▁Paulo -▁tragischen -▁Angela -▁balconies -▁smoothly -▁Urheberrecht -▁columns -▁zurückgezogen -▁schwerwiegende -vorsitzenden --month -▁Eiffel -▁KUKA -seg -Hotel -900 -▁tall -▁null -▁drift -▁matur -▁PAD -▁Pausch -elin -▁Kil -▁Kambod -▁Clo -▁Fell -arin -tritte -▁einl -▁-E -izzer -▁adhere -▁320 -▁10,000 -▁klingt -▁beklagen -▁statistischen -▁Geräus -▁absent -▁Maler -▁approaching -▁Proteste -▁opposing -▁emphasises -▁geltende -▁Highlights -▁seasons -ungsverfahrens -▁Rauchen -▁celebrations -▁DSLR -▁empire -▁dictator -▁detaillierten -gästen -▁Footprint -▁Nephi -▁Ricc -▁enthusiastic -▁Heathrow -▁Frühstücksraum -krieges -,50 -ria -cin -pit -MI -▁bail -chair -▁Focus -▁Rehn -▁unerm -▁unav -▁Lower -▁concluding -▁Oil -beiträge -▁Arist -anzlei -amma -ossiers -▁Imaging -cycl -▁belasten -▁Europa-Mittelmeer -▁debian -▁charging -▁Boulevard -äufen -reden -▁Strahl -▁singing -ophone -▁volcan -▁Relais -▁Gesundheitsschutz -▁Sonnenunter -▁Nahrungsmitteln -▁Customers -befugnisse -modern -▁Sensor -▁Siedlungen -▁künstlerische -▁spürbar -▁Religionsfreiheit -▁Fourthly -▁Karlsruhe -▁Ubuntu -▁nuklearen -▁schallisol -▁Neapel -aar -wuchs -▁seller -▁Ei -▁MC -▁POS -ono -▁catastroph -▁Wies -▁Hamb -▁Rue -atility -▁theater -▁andernfalls -▁sektor -▁zubereitet -▁Jar -▁Stone -ennetz -▁Bequem -ermeld -feuer -haltestelle -▁restoring -▁gleichwert -▁fluct -▁langwier --Test -▁impair -▁Banking -▁Christopher -▁Einsatzes -▁Deliver -▁Installer -▁calculations -▁Remember -▁gründliche -▁Tschechien -papiere -▁не -▁anhaltenden -▁Rettung -▁Gremium -▁Neuerungen -▁pumps -▁Futtermittel -▁gängigen -▁Gletscher -▁Sauberkeit -▁archaeological -ROM -être -▁squar -▁mold -▁Eco -▁Kev -▁Gassen -▁injust -alo -▁seeds -▁vermarkt -▁confined -einkommens -grams -▁abhalten -▁resume -▁Arrang -echnologien -▁YORK -▁Menu -elservice -▁gewalts --Schn -▁somebody -ämmung -▁Kris -▁restra -▁gerät -▁vollzogen -oppy -▁Whenever -▁Guad -▁poster -▁Wertsch -▁Goethe -▁Baust -ecutor -▁kleinste -modus -gesetzes -▁Grenzwerte -▁condemnation -▁scenarios -▁Rechnungshofs -▁monopolies -▁eingestuft -▁Demonstration -▁comprising --Lösung -konstruktion -mobil -kanal -▁runden -▁sid -▁Table -▁Treas -▁brechen -▁Cel -▁Romp -▁unglück -iesst -umfeld -▁beheiz -▁Anforderung -▁Schau -▁spher -ogl -▁nachts -ußts -▁servants -▁Plans -▁economists -▁nationalism -▁Geschlechts -▁Krist -▁Durchmesser -▁Romagna -▁Haushaltsausschuss -▁reformed -▁cleaned -▁methodology -▁learnt -▁wide-ranging -▁Province -▁Niederlage -▁ordentlich -▁Zeichnungen -▁1947 -windigkeiten -▁Sofort -▁Cafe -▁drehen -▁vielerlei -▁Massenvern -group -▁Football -prüfungen -▁Schmidt -▁horny -▁Emotionen -rus -rett -making -▁fla -▁raschen -▁mature -▁cad -▁Gom -▁inhuman -▁anscha -▁beverages -▁NV -ifilm -abgabe -▁demzufolge -ungsmöglichkeiten -rankung -▁nachweis -▁beta -▁Spin -▁Umlauf -nahms -▁Emperor -▁studieren -▁145 -ädagog -▁engere -▁Warte --initiative -▁requesting -▁Fortschritten -▁unequiv -▁footing -▁Straßenbahn -▁hervorragender -▁Ambitionen -▁Besonderheit -▁Wasserk -▁frühzeitig -▁erfreulich -▁Jacob -▁Akteur -risiken -▁kurzfristigen -▁ceilings -spezifischen -▁disparities -▁Bibione -▁Toulouse -▁Japans -fia -mord -▁rings -▁bab -▁Ivan -inland -inationen -▁Cairo -▁Gare -ieri -schle -essa -▁Angreif -ahara -grem -▁Einträge -▁Mao -▁Recher -▁Menschenleben -▁String -▁Entdeck -▁sono -pfeiler -▁Neur -▁Pharm -▁Boote -▁tune -▁Geschäftsbereich -▁optimistic -▁identisch -▁neuem -▁Charl -▁Briten -▁Delors -▁funktionierenden -▁exportiert -▁Democrat -backs -▁englischer -▁obersten -▁lengthy -▁empower -▁1970s -▁shaping -▁ineffective -heizung -zählung -Traq -▁Gonzá -▁Umwandlung -cio -länge -76 -ís -▁ritual -▁Duft -▁Panz -▁loyal -▁Gab -▁inability -atri -▁forcing -▁alb -▁Nine -queren -▁conquer -egr -▁abgew -merks -blog -getr -▁Blanc -▁creatures -▁capita -▁Internetanschluss -▁Palermo -▁schwachen -▁stehe -ameworks -▁transparenten -▁weekends -▁grace -▁voltage -▁gerechten -▁widerspricht -▁consultancy -▁Summen -▁castles -▁cellar -IFF -problemen -▁Wiedergabe -▁Erwähnung -▁US-amerikanischen -▁Bahrain -schlüsselung -▁debattieren -▁exemplary -▁Ahmad -▁Heutzutage -firma -rise -born -comm -▁tagt -▁Tam -▁Easter -▁PER -ildern -▁Vanc -▁alkohol -heid -verlauf -▁aufgeb -alla -▁sog. -ementioned -▁möbl -▁darzustellen -aufgabe -▁Former -▁Repl -▁allevi -▁nationales -▁Krum -▁succ -▁Träume -▁Boeing -sparen -▁Fallout -▁Saar -▁marvell -▁Basque -▁Opern -.04 -▁Antibiot --Install -▁Staatsangehör -▁infection -▁Check-in -▁Municip -▁verabschiedeten -▁Aktionäre -ispiell -▁murdered -▁synthetic -▁Fujifilm -▁Fuerteventura -▁Gütern -▁cosmetic -▁Å -gnung -▁va -▁Syl -▁Diss -▁DNS -erwart -▁Case -▁anstehenden -▁LKW -verleih -▁Anc -▁exagger -conf -▁outlook -▁drama -▁Soul -vorgang -erstattung -▁Gewissen -▁promin -▁Kinderspiel -▁marina -▁weiße -rehensible -iktor -▁Paare -▁Kopien -▁ENT -▁blockieren -IMM -▁teen -▁Zulass -▁Halbjahr -▁BugTraq -▁großzügigen -▁unvergessliche -ULT -▁Illustr -▁Stammzellen -▁multiannual -▁komplizierte -OBIL -▁erfahrene -▁jewel -▁Überraschung -▁überflüssig -liebhaber -▁vorantreiben -▁Uruguay -tum -ketten -kanzler -▁PDA -▁Ah -eses -elie -institute -isan -▁CET -▁Gust -▁NY -heorie -▁Zehn -▁desires -▁comitology -▁ausfallen -▁polar -▁Already -▁disgra -▁relatives -▁Chapter -▁Rey -▁Plattformen -worth -▁own-initiative -▁continents -▁besetzten -marke -▁strategisch -▁Kontrast -itherto -▁templates -▁engineer -willigen -▁regelm -▁Players -▁remembered -▁Antrags -▁Immobilie -▁Wikipedia -▁Kriegsverb -▁SLR -▁Publishing -▁commodities -▁verbleibenden -▁repressive -▁dementsprechend -▁pupils -▁Daimler -pi -PI -PD -▁IX -▁pag -▁BGB -inta -▁Wog -▁Rost -ierbarkeit -▁Lenk -▁2D -▁vervollständ -▁abortion -▁schock -▁Ausgehend -▁Beyond -▁Maritime -▁weiterver -▁minors -comes -▁155 -▁flank -▁besprechen -▁Schwäch -bedien -▁Firewall -▁Höhlen -▁Kritiker -oomla -▁warmth -▁conditional -▁Berufung -▁claiming -▁touched -sprüche -▁indicator -▁Zufl -▁geistige -▁Offenbar -▁entstandenen -▁Nicar -▁ausgebildet -▁inventory -▁fantastische -▁redundant -▁ASPs -▁Kämpfe -▁Befürworter -▁Daily -nisch -lash -▁nin -olet -▁zauber -▁Erleb -udad -ussia -▁unterm -▁zwingt -▁Chá -▁lauten -▁Zeitschriften -▁bedacht -▁revital -▁Marktführer -▁Daran -▁Notre -▁enjoyment -▁glaubwürdig -▁mentioning -spektrum -▁Apartamentos -gression -▁Londoner -effizient -▁Eingriff -▁virtue -▁salaries -▁Zivilbevölkerung -▁ausreichenden -▁Beitrittsländer -▁inneh -zedenz -▁investigated -▁Ethik -.2008 -▁Phänom -▁fantastischen -▁accomplish -▁brothers -▁doppelten -▁Dringlichkeits -▁interfere -▁sacrifice -▁deprived -▁begünstigt -▁nicely -▁Kosmetik -▁predecessor -iations -uhen -software -ümm -codes -pone -▁.NET -▁ward -▁Tuber -▁Dual -▁Piano -erben -▁cofin -▁Krat -▁Fru -berühm -▁erschwing -▁Oost -▁Oran -▁Starten -enda -▁Schach -▁Schreibt -allas -ausrüstung -▁betrachte -prozessen -▁artw -▁weitergeben -▁Seel -▁Haushaltslinie -▁Kompat -▁vereinfach -▁volcano -▁Widerstands -freundlich -▁Konzerne -▁Fremdenfeindlichkeit -▁lebensl -▁Fassade -▁Bavarian -▁Kenya -▁Stornierungen -▁Observ -▁Hügeln -▁Bügelservice -▁venez -▁vorankommen -▁Slovenian --Herzeg -▁TS -▁PD -▁Bäder -▁FE -▁Feren -▁Insp -▁Inhaft -andschaft -▁fürs -▁Entr -▁mancher -▁wieviel -▁unterrichten -▁arc -▁Mitgliedstaats -folios -▁Herstellungs -▁besonderem -▁excav -offiz -▁norms -▁Fahrplan -▁third-party -▁motors -▁offences -▁logos -▁infected -▁cross-country -▁Wissenschaftlern -▁IP-Adresse -▁Mandats -▁1955 -▁Military -▁geringere -▁Check-Out -▁dreht -▁eingebrachten -entscheidung -▁Vernunft -▁supposedly -▁Normalerweise -▁attributes -▁thickness -▁konfiguriert -▁Meldungen -▁unbegrenz -▁Frontera -▁erlangt -▁Unterbrechung -▁administrations -niveaus -rs -links -WB -As -800 -▁tibet -▁dent -▁IDE -▁Dossier -▁Bul -eaker -inia -▁Cards -▁Hond -▁Furcht -▁todo -▁beispiell -▁Liver -▁verhält -▁Anleihen -protokoll -ungsmittel -▁worüber -arks -itee -flut -▁Freig -▁Rahmenrichtlinie -▁Gewichts -▁höre -▁expressions -▁No. -▁patron -▁Fluggäste -▁urheber -▁Senegal -▁Präzedenz -▁reproductive -▁protocols -▁bewährte -▁spannende -▁steady -▁erstaunlich -▁Chalet -▁draftsman -▁plastics -▁Fischereisektor -▁circles -▁bizarre -tet -ciano -CL -▁hem -▁dying -▁dancing -▁rows -▁pixels -▁gorge -itsch -utable -▁Winkel -▁unangen -▁unklar -▁towers -idi -istance -▁jam -odus -ouss -▁bets -schland -▁anderweit -▁Marse -loß -▁Rahmenbeschluss --Fil -▁combinations -mäßiger -▁antibiot -▁Buchungen -▁herausgestellt -▁Anstrengung -▁transferring -▁sportlichen -▁Establ -▁coupled --to-day -▁fragte -▁Veröffentlichungen -▁Behinderten -▁attributed -▁Kartoff -▁kopiert -▁bürgerlichen -▁Ingenieur -▁cloud -▁Panoramabl -▁triumph -▁Byrne -kä -71 -▁mile -▁MHz -▁Penn -▁unite -▁ansteigen -stags -uropä -▁Stipend -▁schar -▁vorgebracht -frame -etrieb -▁remit -▁einzub -▁versenden -▁qualifiziert -▁Neust --Ap -zeugnisse -▁implementiert -▁Better -▁Glanz -▁256 -▁filed -▁Bewert -▁Festland --Con -▁Kommand -▁Verantwortlichkeiten -▁Naturpark -▁harmonious -▁determines -▁supplying -▁high-tech -cksilber -▁appealing -▁Robot -▁lebte -▁anhaltende -▁lucky -▁Edelstahl -▁crowd -▁Česk -▁Schattenberichterstatter -▁hydraulic -▁xenophobia -▁Herangehensweise -▁Vancouver -lasten -BF -▁dritt -▁wissens -▁Senden -eafter -▁clever -▁WIT -▁Hervor -▁Laptop -▁stup -▁2-3 -▁Verhütung -▁jury -▁nicht- -echen -▁county -eteness -eads -▁Marathon -▁Allerg -▁mayor -▁Franç -ymp -bedürf -▁readily -▁annähern -regulierung -▁ambiance -▁Papst -▁bedroh -vorsorge -▁breakdown -clubs -▁1929 -▁fleets -▁11.30 -▁Glasg -▁zuverlässigen -▁Electronics -▁Expansion -lieferungen -▁objektiv -▁undermining -▁Roger -▁Ökonomie -▁Rehab -▁boutiques -▁Karibik -▁Chávez -nutzen -gos -ém -▁bund -▁pak -▁Math -▁Bun -etischen -▁letting -▁Wide -▁Hed -▁Gior -▁None -aches -▁conception -▁Jubiläum -▁abzubauen -▁Erb -erries -▁grows -▁Gewebe -idencies -attungen -▁crown -▁Flügel -▁Command -▁differing -▁hereby -▁debut -▁Fischbestände -▁educated --Problem -▁Schweigen -▁spezieller -▁erkläre -▁mistaken -▁Muttersprache -▁strukturelle -▁gültige -▁Bordeaux -▁Sat-TV -▁slave -▁zurückzukehren -▁Intelligenz -▁lectures -etermination -▁geforderten -▁Handtücher -▁PNR -▁González -aunch -mag --air -VE -▁mg -▁SEC -▁punkt -▁Myst -▁Mütter -▁Pope -▁kraft -▁Künd -▁Kuchen -▁Kette -▁Gul -iesta -▁Lese -osity -▁gefeiert -▁conceived -▁ext -▁Befried -▁trug -▁Maz -▁restart -erkrankungen -▁day-to-day -▁blut -▁employer -▁begründen -▁bewährt -fangs -▁Gebiets -▁loose -▁wesentliches -▁sunshine -▁Hygiene -▁ehemaliger -▁patented -▁fifty -▁angestrebt -▁Confed -▁Schröder -▁Generaldirektion -▁brochure -▁Bruttoinland diff --git a/lingvo/tasks/punctuator/BUILD b/lingvo/tasks/punctuator/BUILD deleted file mode 100644 index 1e5106de..00000000 --- a/lingvo/tasks/punctuator/BUILD +++ /dev/null @@ -1,70 +0,0 @@ -# Punctuator tutorial task. -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -py_library( - name = "input_generator", - srcs = ["input_generator.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:base_input_generator", - "//lingvo/core:generic_input", - "//lingvo/core:py_utils", - "//lingvo/core:tokenizers", - ], -) - -py_test( - name = "input_generator_test", - srcs = ["input_generator_test.py"], - data = [ - "//lingvo/tasks/lm/testdata:lm1b_100", - "//lingvo/tasks/punctuator/params:brown_corpus_wpm_data", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":input_generator", - "//lingvo:compat", - "//lingvo/core:base_input_generator", - "//lingvo/core:py_utils", - "//lingvo/core:test_helper", - "//lingvo/core:test_utils", - ], -) - -py_library( - name = "model", - srcs = ["model.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/tasks/mt:model", - ], -) - -py_test( - name = "model_test", - size = "large", - srcs = ["model_test.py"], - data = [ - "//lingvo/tasks/lm/testdata:lm1b_100", - "//lingvo/tasks/punctuator/params:brown_corpus_wpm_data", - ], - python_version = "PY3", - shard_count = 4, - srcs_version = "PY3", - deps = [ - ":input_generator", - ":model", - "//lingvo:compat", - "//lingvo/core:test_helper", - "//lingvo/core:test_utils", - # Implicit numpy dependency. - ], -) diff --git a/lingvo/tasks/punctuator/__init__.py b/lingvo/tasks/punctuator/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/lingvo/tasks/punctuator/input_generator.py b/lingvo/tasks/punctuator/input_generator.py deleted file mode 100644 index 2caf73ac..00000000 --- a/lingvo/tasks/punctuator/input_generator.py +++ /dev/null @@ -1,130 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Punctuator input generator.""" - -import string -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import generic_input -from lingvo.core import py_utils -from lingvo.core import tokenizers - - -class PunctuatorInput(base_input_generator.BaseSequenceInputGenerator): - """Reads text line by line and processes them for the punctuator task.""" - - @classmethod - def Params(cls): - """Defaults params for PunctuatorInput.""" - p = super().Params() - p.tokenizer = tokenizers.WpmTokenizer.Params() - return p - - def _ProcessLine(self, line): - """A single-text-line processor. - - Gets a string tensor representing a line of text that have been read from - the input file, and splits it to graphemes (characters). - We use original characters as the target labels, and the lowercased and - punctuation-removed characters as the source labels. - - Args: - line: a 1D string tensor. - - Returns: - A list of tensors, in the expected order by __init__. - """ - # Tokenize the input into integer ids. - # tgt_ids has the start-of-sentence token prepended, and tgt_labels has the - # end-of-sentence token appended. - tgt_ids, tgt_labels, tgt_paddings = self.StringsToIds( - tf.convert_to_tensor([line])) - - def Normalize(line): - # Lowercase and remove punctuation. - line = line.lower().translate(None, string.punctuation.encode('utf-8')) - # Convert multiple consecutive spaces to a single one. - line = b' '.join(line.split()) - return line - - normalized_line = tf.py_func(Normalize, [line], tf.string, stateful=False) - _, src_labels, src_paddings = self.StringsToIds( - tf.convert_to_tensor([normalized_line]), is_source=True) - # The model expects the source without a start-of-sentence token. - src_ids = src_labels - - # Compute the length for bucketing. - bucket_key = tf.cast( - tf.round( - tf.maximum( - tf.reduce_sum(1.0 - src_paddings), - tf.reduce_sum(1.0 - tgt_paddings))), tf.int32) - tgt_weights = 1.0 - tgt_paddings - - # Return tensors in an order consistent with __init__. - out_tensors = [ - src_ids, src_paddings, tgt_ids, tgt_paddings, tgt_labels, tgt_weights - ] - return [tf.squeeze(t, axis=0) for t in out_tensors], bucket_key - - def _DataSourceFromFilePattern(self, file_pattern): - """Create the input processing op. - - Args: - file_pattern: The file pattern to use as input. - - Returns: - an operation that when executed, calls `_ProcessLine` on a line read - from `file_pattern`. - """ - return generic_input.GenericInput( - file_pattern=file_pattern, - processor=self._ProcessLine, - # Pad dimension 0 to the same length. - dynamic_padding_dimensions=[0] * 6, - # The constant values to use for padding each of the outputs. - dynamic_padding_constants=[0, 1, 0, 1, 0, 0], - **self.CommonInputOpArgs()) - - def __init__(self, params): - super().__init__(params) - - # Build the input processing graph. - (self._src_ids, self._src_paddings, self._tgt_ids, self._tgt_paddings, - self._tgt_labels, - self._tgt_weights), self._bucket_keys = self._BuildDataSource() - - def InfeedBatchSize(self): - """Override BaseSequenceInputGenerator.""" - return tf.shape(self._src_ids)[0] - - def _InputBatch(self): - """Returns a single batch as a `.NestedMap` to be passed to the model.""" - ret = py_utils.NestedMap() - - ret.bucket_keys = self._bucket_keys - - ret.src = py_utils.NestedMap() - ret.src.ids = tf.cast(self._src_ids, dtype=tf.int32) - ret.src.paddings = self._src_paddings - - ret.tgt = py_utils.NestedMap() - ret.tgt.ids = self._tgt_ids - ret.tgt.labels = tf.cast(self._tgt_labels, dtype=tf.int32) - ret.tgt.weights = self._tgt_weights - ret.tgt.paddings = self._tgt_paddings - - return ret diff --git a/lingvo/tasks/punctuator/input_generator_test.py b/lingvo/tasks/punctuator/input_generator_test.py deleted file mode 100644 index ded73b15..00000000 --- a/lingvo/tasks/punctuator/input_generator_test.py +++ /dev/null @@ -1,85 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for input generator.""" - -import string -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import py_utils -from lingvo.core import test_helper -from lingvo.core import test_utils -from lingvo.tasks.punctuator import input_generator - - -class InputGeneratorTest(test_utils.TestCase): - - def _CreatePunctuatorInputParams(self): - p = input_generator.PunctuatorInput.Params() - input_file = 'text:' + test_helper.test_src_dir_path( - 'tasks/lm/testdata/lm1b_100.txt') - p.tokenizer.vocab_filepath = test_helper.test_src_dir_path( - 'tasks/punctuator/params/brown_corpus_wpm.16000.vocab') - p.tokenizer.vocab_size = 16000 - p.file_pattern = input_file - p.file_random_seed = 314 - p.file_parallelism = 1 - p.source_max_length = 200 - p.target_max_length = 200 - p.bucket_upper_bound = [20, 40] - p.bucket_batch_limit = [1, 1] - return p - - def testBasic(self): - p = self._CreatePunctuatorInputParams() - with self.session(use_gpu=False): - inp = input_generator.PunctuatorInput(p) - # Runs a few steps. - for _ in range(10): - self.evaluate(inp.GetPreprocessedInputBatch()) - - def testSourceTargetValues(self): - max_length = 50 - p = self._CreatePunctuatorInputParams() - with self.session(use_gpu=False): - inp = input_generator.PunctuatorInput(p) - tokenizer = inp.tokenizer_dict[base_input_generator.DEFAULT_TOKENIZER_KEY] - - fetched = py_utils.NestedMap( - self.evaluate(inp.GetPreprocessedInputBatch())) - source_ids = fetched.src.ids - tgt_ids = fetched.tgt.ids - tgt_labels = fetched.tgt.labels - - expected_ref = (b'His approach was inquisitive , a meeting of artful ' - b'hesitation with fluid technique .') - - normalized_ref = expected_ref.lower().translate( - None, string.punctuation.encode('utf-8')) - normalized_ref = b' '.join(normalized_ref.split()) - _, expected_src_ids, _ = self.evaluate( - tokenizer.StringsToIds( - tf.convert_to_tensor([normalized_ref]), max_length=max_length)) - expected_tgt_ids, expected_tgt_labels, _ = self.evaluate( - tokenizer.StringsToIds( - tf.convert_to_tensor([expected_ref]), max_length=max_length)) - - self.assertAllEqual(expected_src_ids[0], source_ids[0, :max_length]) - self.assertAllEqual(expected_tgt_ids[0], tgt_ids[0, :max_length]) - self.assertAllEqual(expected_tgt_labels[0], tgt_labels[0, :max_length]) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/punctuator/model.py b/lingvo/tasks/punctuator/model.py deleted file mode 100644 index 3ffaff55..00000000 --- a/lingvo/tasks/punctuator/model.py +++ /dev/null @@ -1,63 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Punctuator model.""" - -import lingvo.compat as tf -from lingvo.core import py_utils -from lingvo.tasks.mt import model as mt_model - - -class RNMTModel(mt_model.RNMTModel): - """The MT model with an inference graph for punctuator.""" - - def Inference(self): - """Constructs the inference subgraphs. - - Returns: - dict: ``{'subgraph_name': (fetches, feeds)}`` - """ - subgraphs = dict() - with tf.name_scope('inference'): - subgraphs['default'] = self._InferenceSubgraph_Default() - return subgraphs - - def _InferenceSubgraph_Default(self): - with tf.name_scope('inference'): - src_strings = tf.placeholder(tf.string, shape=[None]) - _, src_ids, src_paddings = self.input_generator.StringsToIds( - src_strings, is_source=True) - - src_input_map = py_utils.NestedMap(ids=src_ids, paddings=src_paddings) - encoder_outputs = self.enc.FPropDefaultTheta(src_input_map) - decoder_outs = self.dec.BeamSearchDecode(encoder_outputs) - - topk_hyps = decoder_outs.topk_hyps - topk_ids = decoder_outs.topk_ids - topk_lens = decoder_outs.topk_lens - - # topk_lens - 1 to remove the EOS id. - topk_decoded = self.input_generator.IdsToStrings(topk_ids, topk_lens - 1) - topk_decoded = tf.reshape(topk_decoded, tf.shape(topk_hyps)) - - feeds = py_utils.NestedMap({'src_strings': src_strings}) - fetches = py_utils.NestedMap({ - 'src_ids': src_ids, - 'topk_decoded': topk_decoded, - 'topk_scores': decoder_outs.topk_scores, - 'topk_hyps': topk_hyps, - }) - - return fetches, feeds diff --git a/lingvo/tasks/punctuator/model_test.py b/lingvo/tasks/punctuator/model_test.py deleted file mode 100644 index 93bb3aab..00000000 --- a/lingvo/tasks/punctuator/model_test.py +++ /dev/null @@ -1,166 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for Punctuator model.""" - -import lingvo.compat as tf -from lingvo.core import test_helper -from lingvo.core import test_utils -from lingvo.tasks.punctuator import input_generator -from lingvo.tasks.punctuator import model -import numpy as np - -_TF_RANDOM_SEED = 93820986 - - -class PunctuatorModelTest(test_utils.TestCase): - """Tests for the Punctuator model. - - Overriding parameters and inheriting tests from RNMTModelTest. - """ - - def _InputParams(self): - p = input_generator.PunctuatorInput.Params() - input_file = test_helper.test_src_dir_path('tasks/lm/testdata/lm1b_100.txt') - p.tokenizer.vocab_filepath = test_helper.test_src_dir_path( - 'tasks/punctuator/params/brown_corpus_wpm.16000.vocab') - p.tokenizer.vocab_size = 16000 - p.file_pattern = 'text:' + input_file - p.file_random_seed = 31415 - p.file_parallelism = 1 - p.bucket_upper_bound = [40] - p.bucket_batch_limit = [4] - p.source_max_length = 40 - p.target_max_length = 40 - return p - - def _UpdateEncoderParams(self, p): - p.emb.embedding_dim = 4 - p.emb.max_num_shards = 1 - p.lstm_cell_size = 4 - p.num_lstm_layers = 3 - p.encoder_out_dim = 4 - return p - - def _UpdateDecoderParams(self, p): - p.source_dim = 4 - p.emb.embedding_dim = 4 - p.emb.max_num_shards = 1 - p.rnn_cell_dim = 4 - p.rnn_layers = 3 - p.attention.hidden_dim = 2 - p.softmax.num_shards = 1 - return p - - def _testParams(self): - p = model.RNMTModel.Params() - p.name = 'test_mdl' - p.input = self._InputParams() - self._UpdateEncoderParams(p.encoder) - self._UpdateDecoderParams(p.decoder) - p.train.learning_rate = 2e-4 - return p - - def testConstruction(self): - with self.session(): - p = self._testParams() - mdl = p.Instantiate() - print('vars = ', mdl.vars) - flatten_vars = mdl.vars.Flatten() - print('vars flattened = ', flatten_vars) - self.assertEqual(len(flatten_vars), 27) - - # Should match tf.trainable_variables(). - self.assertEqual(len(tf.trainable_variables()), len(flatten_vars)) - - def testFProp(self, dtype=tf.float32): - with self.session(): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._testParams() - p.dtype = dtype - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - loss = mdl.loss - logp = mdl.eval_metrics['log_pplx'][0] - self.evaluate(tf.global_variables_initializer()) - vals = [] - for _ in range(3): - vals += [self.evaluate((loss, logp))] - - print('actual vals = %s' % np.array_repr(np.array(vals))) - expected_vals = [ - [326.765106, 10.373495], - [306.018066, 10.373494], - [280.08429, 10.373492], - ] - self.assertAllClose(vals, expected_vals) - - def testBProp(self): - with self.session(): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._testParams() - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - mdl.BProp() - loss = mdl.loss - logp = mdl.eval_metrics['log_pplx'][0] - - self.evaluate(tf.global_variables_initializer()) - vals = [] - for _ in range(3): - vals += [self.evaluate((loss, logp, mdl.train_op))[:2]] - print('BProp actual vals = ', vals) - expected_vals = [ - [326.765106, 10.373495], - [306.013123, 10.373326], - [280.07666, 10.37321], - ] - self.assertAllClose(vals, expected_vals) - - def testFPropEvalMode(self): - with self.session(), self.SetEval(True): - tf.random.set_seed(_TF_RANDOM_SEED) - p = self._testParams() - mdl = p.Instantiate() - mdl.FPropDefaultTheta() - loss = mdl.loss - logp = mdl.eval_metrics['log_pplx'][0] - self.evaluate(tf.global_variables_initializer()) - vals = [] - for _ in range(3): - vals += [self.evaluate((loss, logp))] - print('actual vals = ', vals) - expected_vals = [ - [326.765106, 10.373495], - [306.018066, 10.373494], - [280.08429, 10.373492], - ] - self.assertAllClose(vals, expected_vals) - - def testInference(self): - with self.session(use_gpu=False) as sess, self.SetEval(True): - tf.random.set_seed(93820985) - p = self._testParams() - mdl = p.Instantiate() - fetches, feeds = mdl.Inference()['default'] - - self.evaluate(tf.global_variables_initializer()) - src_strings = ['the cat sat on the mat', 'the dog sat on the mat'] - dec_out = sess.run(fetches, {feeds['src_strings']: src_strings}) - print('dec_out', dec_out) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tasks/punctuator/params/BUILD b/lingvo/tasks/punctuator/params/BUILD deleted file mode 100644 index 237d822b..00000000 --- a/lingvo/tasks/punctuator/params/BUILD +++ /dev/null @@ -1,36 +0,0 @@ -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -filegroup( - name = "brown_corpus_wpm_data", - srcs = glob(include = ["brown_corpus_wpm.*"]), -) - -py_library( - name = "codelab", - srcs = ["codelab.py"], - data = [":brown_corpus_wpm_data"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo:model_registry", - "//lingvo/core:base_model_params", - "//lingvo/tasks/mt:base_config", - "//lingvo/tasks/mt/params", - "//lingvo/tasks/punctuator:input_generator", - "//lingvo/tasks/punctuator:model", - ], -) - -py_library( - name = "params", - srcs = ["params.py"], - srcs_version = "PY3", - tags = ["keep_dep"], - deps = [ - ":codelab", - ], -) diff --git a/lingvo/tasks/punctuator/params/brown_corpus_wpm.16000.vocab b/lingvo/tasks/punctuator/params/brown_corpus_wpm.16000.vocab deleted file mode 100644 index fe761b3f..00000000 --- a/lingvo/tasks/punctuator/params/brown_corpus_wpm.16000.vocab +++ /dev/null @@ -1,15942 +0,0 @@ - - - - - -▁ -t -h -e -o -f -a -n -d -i -w -s -r -b -y -m -T -v -c -u -l -I -p -H -- -k -g -B -j -A -S -M -. -x -F -1 -W -N -q -2 -U -O -, -J -3 -0 -Y -4 -z -9 -6 -5 -P -E -G -D -8 -" -& -C -7 -K -' -R -L -( -) -V -: -Q -* -; -? -Z -/ -% -$ -X -! -{ -} -[ -] -+ -▅ -~ -` -# -< -= -> -@ -\ -^ -_ -| -th -the -▁the -in -an -on -ing -and -en -▁w -of -▁of -er -ed -or -ou -re -▁b -at -ar -al -▁f -▁and -▁h -ch -▁a -▁in -ion -to -▁to -▁p -▁for -▁c -▁s -is -▁th -▁m -om -▁wh -as -es -▁was -▁that -ent -it -ic -ation -ow -▁d -ly -Th -gh -▁be -ad -▁n -▁wi -▁with -ver -ve -un -im -▁sh -▁con -ro -▁no -ther -▁pro -le -▁re -ct -▁g -▁on -id -ld -ould -qu -▁his -▁had -us -ur -▁he -▁com -ay -am -ex -ment -▁Th -igh -ough -ce -ir -oun -▁this -▁is -▁l -ll -all -rom -▁from -tr -▁it -▁A -ble -our -▁I -if -▁ex -▁by -▁not -ain -ave -▁have -▁as -▁S -ter -ive -ol -il -ul -ut -▁but -▁will -ight -out -▁st -ck -▁su -▁whi -▁which -ction -▁t -▁are -▁would -os -ous -▁an -▁un -▁H -▁The -▁him -ity -ag -▁r -▁C -▁1 -▁M -▁at -ill -own -od -other -▁j -▁de -▁ne -um -ure -▁ch -▁B -▁her -op -over -iv -ound -ak -ist -ig -ack -est -▁tr -ard -ome -▁some -▁19 -end -ence -▁has -ally -▁v -▁e -▁or -▁wor -▁W -▁y -▁you -▁all -▁wer -▁were -▁we -▁P -ers -ess -res -ies -iz -'s -ong -▁se -ate -for -able -▁k -▁le -ud -▁out -pp -▁comp -▁exp -up -▁up -▁sp -▁F -ish -act -ect -▁gr -bout -▁tim -ap -ach -▁ev -el -imp -ost -▁new -▁dis -▁kn -em -▁who -▁when -▁been -ant -ance -▁man -ace -art -▁mor -▁pl -▁al -▁other -▁about -ition -▁app -▁D -ast -et -▁more -ust -st -ot -▁could -ort -ign -▁R -▁N -▁into -▁tw -▁us -▁" -▁comm -ab -old -▁do -ok -▁so -ook -ood -ind -▁G -▁than -ial -ical -▁work -irst -▁their -▁first -▁one -▁only -▁O -▁what -▁two -▁part -ip -▁L -▁E -▁pres -▁pr -▁imp -▁qu -▁over -▁op -▁T -▁J -▁ro -av -▁should -▁she -ious -▁cl -▁str -ang -ob -ik -▁lik -ib -▁even -rough -▁through -▁und -▁ad -▁though -▁such -vel -00 -▁2 -age -ff -ject -▁if -very -▁In -▁time -ose -ame -▁any -▁ag -▁again -ated -▁en -▁- -ult -▁high -tain -velop -▁ob -urn -ub -▁just -▁can -▁ye -▁year -▁ar -▁( -ary -ew -aw -way -ward -ac -oc -ail -▁world -ond -ress -▁ind -iff -▁diff -elf -ful -▁those -▁back -▁bo -▁its -ild -▁look -▁long -▁tak -▁sub -▁develop -▁He -▁now -▁inter -ide -▁like -▁br -fter -▁after -▁mak -reat -▁great -▁how -ire -▁there -▁i -▁our -ie -▁supp -▁most -▁may -▁me -▁must -aid -▁said -aus -are -▁pre -▁your -▁well -ark -ick -▁know -▁hand -ber -▁per -ible -▁befor -▁down -▁dr -▁child -▁they -ove -ian -▁own -▁every -▁good -▁res -▁pos -▁day -▁way -▁po -▁U -ffect -ell -▁under -form -ass -ations -pt -cept -▁coun -▁bec -▁becaus -cess -▁each -▁stud -ens -ine -ge -og -lect -▁state -ities -itt -app -▁against -ittle -▁little -▁It -▁rep -▁cont -▁fr -▁very -▁An -cre -▁cons -ubl -ublic -▁seem -▁before -ade -▁made -ree -one -ice -▁three -ident -▁did -▁des -air -ather -▁But -▁another -▁many -▁bl -cial -▁off -▁turn -▁never -ution -▁ac -▁acc -▁rec -ople -ates -amp -▁K -▁while -ield -▁still -olog -▁3 -ool -chool -▁col -▁pol -▁And -s. -▁both -ath -ific -▁interest -▁school -▁act -uct -irect -oll -ollow -▁wher -▁where -erv -ater -iver -thing -▁because -eople -▁people -▁lif -▁Wh -ative -se -▁these -s, -self -▁himself -▁spe -▁num -▁number -▁import -cl -clud -und -▁commun -▁rel -▁prov -ov -▁proble -ular -▁plan -▁place -▁add -ature -▁hous -▁effect -▁em -▁am -▁rem -▁cr -gr -gram -▁program -▁cour -uch -▁much -ness -overn -overnment -ax -creas -▁old -stem -▁result -ual -▁sto -▁Y -atter -arr -▁around -ces -ord -▁consid -ans -▁trans -ict -▁course -ock -▁go -▁prop -▁prod -oup -▁group -▁resp -▁serv -▁countr -ue -ep -ite -we -ween -tween -▁between -▁follow -▁fact -ystem -ull -uc -▁pers -▁person -▁word -▁without -▁problem -▁public -ational -ility -▁ear -▁play -▁fl -▁te -▁life -▁youn -▁poin -▁point -ink -▁think -ix -▁ph -▁make -▁system -▁sc -▁disc -ph -▁Ch -▁Sh -hip -▁them -▁form -man -▁195 -▁als -▁also -▁show -day -▁quest -▁question -▁years -▁ab -ince -▁direct -▁end -▁sm -▁small -▁This -ually -▁char -▁chang -ied -ily -ia -iet -▁get -ably -▁appro -▁last -▁att -▁home -▁since -▁social -▁St -▁present -▁comple -ile -▁differ -▁gen -▁gener -▁government -▁196 -▁enough -ience -▁call -ory -▁my -▁im -▁consider -▁same -▁become -▁V -▁came -ount -▁cent -▁open -▁alway -gan -▁found -▁bas -▁water -▁fe -term -▁exper -▁wr -▁knew -▁few -ustr -▁industr -uild -▁reg -▁young -▁feel -ember -▁increas -▁class -▁special -▁equ -▁requ -emb -embers -▁ord -▁thought -ertain -ext -az -ugh -▁ent -ient -▁giv -▁Un -▁important -000 -▁prof -ank -▁certain -". -▁right -▁development -ism -▁pass -les -▁dep -▁poss -▁possible -▁18 -▁contin -▁ass -ons -iss -ivid -▁individ -▁individual -▁contr -▁conf -▁cond -▁condition -▁Mr -▁sl -ever -▁however -▁conc -▁might -▁4 -▁art -▁oc -▁occ -▁expect -▁start -▁ref -▁itself -ision -augh -▁ke -ican -erican -▁partic -▁particular -ower -▁states -▁something -▁nothing -▁say -▁read -▁stand -▁def -▁experience -▁head -▁sur -pl -uck -▁next -▁night -▁want -▁report -land -▁build -▁keep -▁house -▁order -ically -▁record -ug -▁eng -uction -▁oper -ense -▁unders -ments -▁lead -atch -urch -▁church -▁charact -▁dec -▁adv -▁On -ning -▁war -▁jo -▁sec -▁second -▁see -▁sever -▁set -ganiz -▁opp -outh -▁general -alk -ission -▁loc -▁law -ise -read -iew -▁New -▁Af -▁position -▁simp -▁subject -▁object -▁necess -▁need -▁nor -ief -▁elect -▁remain -▁inst -▁design -▁sign -▁along -▁among -▁american -▁room -▁national -▁contro -▁bel -▁belie -▁complet -▁examp -▁determ -▁dist -▁process -tom -▁adm -iness -usiness -▁business -less -▁less -▁lef -▁left -▁level -▁tax -ty -▁let -▁members -▁kind -▁addition -▁inv -▁president -▁power -fort -ash -ase -ush -most -▁almost -▁cost -▁cap -▁cor -ors -▁thing -▁real -▁val -▁dem -▁She -▁use -▁take -▁happ -▁find -ating -▁dur -▁during -▁includ -ention -▁5 -▁ide -▁gu -▁continu -ason -▁reason -▁upon -▁light -▁several -▁bro -▁different -▁service -▁concer -ere -▁There -▁above -▁fam -▁family -▁walk -▁became -▁tech -▁techn -▁given -▁accept -▁hum -▁human -▁compan -▁arm -▁Am -▁American -idence -ear -▁appear -arge -▁large -▁white -lish -ified -▁strong -▁gl -▁stre -▁empl -▁mark -▁come -aterial -▁later -cret -ances -▁colle -ions -▁ask -▁support -▁short -pe -ike -▁once -ized -▁used -▁near -▁car -▁co -olution -▁four -▁repres -▁prev -artment -ently -▁leg -▁least -▁known -▁For -▁view -aph -▁close -▁alread -▁var -▁far -▁field -▁term -▁fin -▁went -▁case -▁cle -▁clear -▁care -▁univer -▁job -iqu -ipp -▁yet -▁respons -▁respect -ablish -ents -▁rather -▁example -▁imm -▁name -ibility -▁hard -▁mov -though -▁although -▁organiz -ize -▁offic -▁polit -▁polic -▁political -▁childr -▁children -▁men -enc -▁grow -establish -▁relig -action -anc -▁pract -▁toward -ither -gether -ith -uth -ody -▁mod -▁expl -▁express -▁reach -▁teach -▁$ -ont -▁front -▁ret -▁return -▁rest -onom -conom -▁econom -ton -▁wom -omm -▁either -▁el -led -ited -▁United -▁always -▁away -▁early -ological -▁product -▁rev -pect -ences -▁perform -▁acro -▁mean -ange -ek -▁week -▁sk -ork -▁ele -▁ey -▁eyes -▁ever -igure -▁figure -▁wid -ially -▁talk -▁began -▁rece -▁spec -▁profess -▁put -formation -icult -▁difficult -wh -hn -▁john -▁sw -▁quite -empt -▁approach -▁means -▁cur -▁caus -▁carr -▁took -ately -▁meet -▁street -▁fri -▁friend -ich -▁entire -▁remember -thod -▁method -▁meas -ohn -▁John -oy -▁employ -iety -▁except -▁ext -tim -▁provid -istr -▁distr -ffort -▁effort -▁exist -▁mus -▁whose -▁country -▁min -▁run -▁Mr. -▁Mrs. -▁secret -▁matter -▁move -▁6 -▁anything -emp -▁temp -▁death -▁eas -▁material -cr -▁descr -ccess -▁success -▁feet -▁prin -▁princ -▁princip -▁Comm -▁book -▁proper -▁mon -▁month -▁mind -▁wind -▁speak -▁ground -▁board -▁boy -▁suff -ves -elves -▁measure -istic -▁States -▁amount -aj -▁maj -▁major -▁mr -iction -▁six -▁af -af -▁side -▁too -▁o -▁establish -▁7 -▁prom -▁electr -▁face -▁evidence -▁imag -▁vis -iously -▁invol -▁If -▁necessary -▁provide -▁miss -▁mil -ven -▁attempt -atur -ency -▁college -▁local -gress -side -te -▁here -▁hop -▁prob -▁probably -▁kept -▁unt -▁until -▁united -▁represent -▁plac -▁fac -▁saw -▁market -▁watch -▁got -▁black -truction -▁stock -tern -▁del -▁hel -▁help -▁writ -▁body -▁big -▁then -▁concern -▁morning -▁par -▁give -▁gave -▁activ -▁liv -▁hour -ailable -vailable -▁value -▁valu -▁recog -▁dev -▁rad -▁radi -aving -▁having -▁hold -▁past -iod -▁period -▁thir -▁describ -▁various -▁whe -▁whether -alf -▁half -▁called -educ -▁educ -pr -▁natur -▁taken -▁receiv -▁priv -▁brough -▁vol -▁require -▁sense -wn -▁10 -▁indic -selves -▁themselves -▁const -elt -▁felt -▁free -▁camp -▁cover -▁stood -▁minut -▁letter -▁character -▁surf -▁surface -▁draw -ner -ier -ior -▁step -▁ver -urther -▁pur -▁purp -▁really -▁pop -▁project -▁discus -▁discover -▁subst -▁being -▁When -▁press -▁pressure -orn -ore -▁appl -▁music -ines -gin -▁begin -▁conv -uthor -▁main -▁pain -▁train -▁mill -ability -ten -eng -ength -▁underst -▁understand -▁love -atic -▁economic -▁control -▁specific -▁mother -▁full -inistr -▁administr -▁often -par -▁prepar -▁prec -▁regard -ilities -▁sometim -▁techniqu -▁brought -ead -ake -▁complete -▁community -▁city -▁arr -▁times -▁does -▁door -ches -oph -hind -▁behind -▁gir -▁girl -▁company -ury -▁century -▁production -▁demand -▁land -▁office -▁further -▁fre -▁gre -▁bre -▁break -▁15 -▁usually -▁students -▁already -hap -▁perhap -▁tre -▁spir -▁spirit -▁prim -▁account -▁accord -▁recogn -▁Fr -▁produc -▁road -▁trad -▁tradition -▁experim -ages -▁beginning -▁attention -▁mar -▁contain -itution -▁assoc -▁associ -oub -oice -▁voice -▁ve -▁signific -ished -ved -▁million -rist -ristian -▁list -▁asked -▁reaction -▁sound -icture -erature -▁temperature -▁sup -▁sug -▁sugg -▁suggest -▁sum -▁sure -▁summ -▁test -ibut -ization -uff -body -▁believe -▁lib -abor -ibution -ting -▁relations -▁soc -▁society -▁done -▁institution -▁private -ym -▁sym -irm -▁firm -▁requir -▁They -▁phil -▁conduct -▁picture -▁pay -▁wif -▁religious -▁common -▁assum -ium -▁everything -▁line -▁looked -▁1960 -▁met -▁swe -▁ident -▁ill -▁information -▁best -▁invest -▁committ -▁progress -truct -▁woman -▁opport -▁opportun -▁jud -▁standard -▁ty -▁type -▁typ -▁8 -▁study -▁determin -▁doub -ouble -▁separ -▁frequ -just -▁court -▁Br -▁acros -▁across -▁occur -▁maintain -▁obtain -▁moder -▁modern -ung -unction -▁function -▁det -▁detail -ted -▁deep -aff -▁neg -▁York -▁york -▁tell -▁assign -▁dark -▁remark -▁nature -▁town -▁struct -▁section -▁defense -▁together -▁His -▁neigh -eral -ional -▁outside -▁decision -resident -enn -▁oppos -▁surpr -▁depend -▁department -▁squ -▁purpose -▁shall -▁wall -▁available -▁hist -▁history -tle -▁whole -▁int -outher -▁somewh -▁volum -head -▁instead -▁plant -▁corp -▁sal -▁personal -irection -▁happen -▁lear -▁learn -▁marr -▁12 -▁milit -▁military -▁obv -attle -ling -idd -▁With -▁mom -▁moment -▁becom -▁resear -▁action -▁true -▁pat -▁pe -▁peace -▁involved -▁194 -▁turned -▁leave -ne -▁property -▁seemed -▁cut -▁stopp -▁policy -▁leaders -uture -▁future -▁research -itch -▁obs -▁observ -▁doct -▁inf -lf -▁self -▁infl -▁influ -▁cir -▁circ -▁labor -▁window -▁concept -▁thus -clus -▁author -▁treat -▁third -fully -ouch -tter -▁better -ters -▁told -▁held -▁broad -▁visit -▁ir -▁mag -▁propos -▁clos -▁direction -▁farm -omp -▁brother -ology -▁air -▁join -▁popul -▁cre -reg -▁education -come -▁Some -▁agree -▁decl -▁enter -▁actually -chie -▁why -▁16 -ene -▁We -▁women -▁earth -ources -▁attack -oud -ought -ger -get -tract -fact -▁things -▁square -▁patient -▁popular -raph -ograph -dd -▁sudd -▁sudden -▁ann -▁therefor -joy -▁enjoy -outhern -▁post -▁rock -▁countries -▁slow -▁allow -▁seen -greg -ugg -▁comb -▁hund -▁hundr -▁fund -▁fall -▁space -▁wife -▁responsibility -▁sometimes -ator -▁equip -▁equipment -▁mach -ists -▁President -▁sch -iddle -▁deal -rope -urope -▁alone -▁writt -▁famil -▁ant -▁quick -▁father -▁cool -▁mot -▁mone -▁money -▁else -▁making -▁ser -ives -▁color -avor -▁favor -▁En -▁Eng -▁impress -itud -▁attitud -▁remov -▁going -▁announ -ification -ination -", -osoph -▁philosoph -off -▁intern -▁international -▁today -▁impro -▁histor -▁wond -oot -▁ago -▁discuss -iam -▁district -▁fear -▁strength -▁feature -▁sit -▁situ -▁situation -▁pie -▁hot -▁wonder -▁finally -▁fail -▁norm -▁north -ford -▁engine -▁lo -▁To -▁building -▁coup -▁twent -ev -▁1961 -▁qual -▁dead -chieve -▁collect -▁occas -▁difference -▁influence -▁grad -ital -▁bar -▁appar -▁assist -pec -pecially -▁neighb -▁occup -▁attend -epend -▁independ -▁heav -▁change -eign -▁fire -▁five -▁police -▁abs -▁generally -▁area -▁touch -oth -▁park -▁explain -erman -aut -▁beaut -▁beautif -▁tele -▁pap -▁contribut -▁sat -▁values -▁univers -▁university -▁perhaps -▁phy -▁phys -▁physical -ry -▁getting -▁lines -▁length -▁lim -aim -▁claim -▁sim -▁simil -▁similar -▁wa -▁wait -▁industrial -▁industry -▁English -▁english -let -ley -▁At -▁beautiful -▁earl -▁earlier -▁fil -ington -▁characteristic -▁illustr -▁professional -▁bene -▁text -▁17 -▁wood -▁You -itive -▁effective -▁recomm -▁compet -▁minutes -▁moral -▁wild -▁decid -▁lower -▁desp -rop -▁democ -▁growth -▁experiment -▁div -▁lit -▁literature -▁complex -▁days -▁spr -▁sn -arily -▁neither -▁brok -▁doubt -▁capital -▁idea -▁center -▁hundred -▁scre -▁sol -▁blood -▁orig -▁origin -▁within -▁fell -▁kill -▁apparently -ential -ledge -▁knowledge -erence -▁Al -▁conver -▁congress -hood -▁food -▁detect -▁bill -urd -ivil -▁civil -attern -▁pattern -▁As -▁Ar -▁After -▁investig -▁25 -▁obviously -utom -▁block -▁relationship -▁recommend -▁distribution -▁enc -cent -▁foreign -▁sens -io -oon -▁symb -▁symbol -ices -▁services -▁dom -▁feeling -▁conclus -graph -▁basis -▁official -▁address -son -▁scient -icient -▁sufficient -▁lost -▁running -▁therefore -ute -▁poet -▁wide -▁truth -▁adjust -▁cell -▁science -▁curr -▁Of -▁spok -ival -▁south -▁middle -▁compar -▁eight -▁One -ee -▁committee -▁structure -▁vir -ode -▁god -▁reflect -▁select -▁season -ountain -▁written -▁wrong -▁centr -▁central -idge -▁basic -▁shown -▁doll -▁dollar -▁mass -▁planning -rod -▁introd -ames -▁quiet -ros -ross -▁cross -▁cal -▁es -▁especially -▁trial -▁command -▁slight -▁carried -iate -▁low -▁How -▁below -▁simple -▁simply -ensive -ension -▁areas -aster -ases -▁cases -▁ideas -ister -▁miles -,000 -▁piece -▁clean -▁flo -▁floor -▁brown -▁gone -ump -port -▁activities -like -▁recently -azz -▁liberal -ederal -▁federal -▁satis -▁population -▁natural -▁performance -▁seems -▁activity -encies -isc -▁ter -▁terr -▁throughout -▁relative -umb --f -▁glass -▁makes -▁division -▁What -▁crit -▁mrs -▁hands -▁age -road -▁cold -rem -▁extrem -▁perm -ustom -leas -▁pleas -▁whom -▁aff -▁eff -ground -▁recent -▁pick -▁acqu -▁regular -upp -uce -▁drink -▁extend -▁opportunity -▁facilities -▁thr -ift -aft -▁govern -▁movement -udience -▁ess -eness -▁benef -▁appropr -▁super -▁comfort -▁secretary -▁event -▁stress -▁demons -▁Ind -ird -▁Miss -▁creat -▁threat -▁limited -▁limit -ench -▁numb -▁numbers -urb -pital -ospital -▁pieces -▁magn -br -▁However -▁technical -▁tried -▁chair -▁serious -▁judg -▁heart -▁That -▁treatment -▁z -▁seven -▁election -▁easily -▁disp -▁disapp -▁able -▁formul -ufact -▁manufact -▁thirty -▁dre -▁dream -▁arg -▁club -▁administration -zz -▁encour -▁blue -▁hope -▁forces -▁scientific -▁final -▁mere -aught -▁importance -ify -▁fif -▁fill -▁Then -▁applied -▁emot -▁tend -ians -▁commission -▁win -ape -▁democr -▁increase -gree -▁degree -▁cult -▁corner -▁proc -ec -ashington -▁Washington -▁30 -▁20 -▁force -ides -▁student -▁sent -▁hair -▁God -▁legis -▁legisl -▁All -▁twenty -▁carry -▁unc -estern -ables -▁rom -▁9 -▁ed -▁est -▁estim -▁immed -▁obvious -▁skill -▁100 -▁displ -▁charge -ument -▁demonstr -▁suddenly -▁required -▁pull -▁image -▁table -▁prevent -▁continue -▁11 -▁film -▁symp -ues -▁bu -angu -▁langu -▁sales -ann -ane -▁emph -osph -▁lay -▁circum -▁circumst -▁circumstances -▁following -▁solid -▁hospital -▁eye -▁che -▁check -▁base -▁nations -▁interv -astic -▁somewhat -▁emphas -▁suit -▁west -▁heard -▁possibility -▁possib -▁thick -▁fair -▁affair -▁particip -▁ship -▁nuc -▁nucle -▁member -▁Thes -▁These -tegr -rote -▁wrote -ropp -▁Pro -▁language -▁stage -▁prot -▁contract -▁autom -athe -unk -▁gun -▁lack -equ -▁problems -▁determine -▁trade -▁trave -▁try -▁trouble -ert -erg -▁energ -▁emerg -▁paper -▁custom -▁produce -▁No -▁variety -err -▁los -▁compos -▁frank -▁occurr -▁flow -▁religion -▁provision -▁meaning -▁meeting -▁immedi -▁immediately -▁immediate -town -▁manag -ague -▁exact -▁paid -▁corpor -verage -▁1959 -urg -▁measur -▁thous -▁thousand -▁manner -▁Christian -ires -iol -▁viol -▁vari -▁pool -athol -atholic -▁living -▁range -▁choice -▁percent -▁county -ah -▁manufactur -▁pot -▁potential -▁essential -▁normal -▁exce -▁ball -▁quality -▁coll -▁frequently -▁opin -▁collection -▁worth -rup -ired -▁Or -▁hotel -▁constant -▁substant -▁myself -▁herself -▁ten -▁Q -▁applic -▁stat -eric -▁Americ -▁americ -▁amb -ler -▁sort -ortion -▁teleph -▁brief -▁chief -▁belief -▁party -▁existence -ural -▁model -▁revolution -▁review -▁interview -▁green -ously -▁words -▁series -▁serve -▁consequ -▁construction -▁consist -orm -amm -men -▁wanted -▁rich -▁electron -▁repl -▁sil -▁esc -▁yell -▁fellow -▁concent -▁dropp -▁poem -othe -▁concentr -▁stay -▁staff -▁ste -▁story -▁minim -int -▁domin -ale -▁pret -ories -▁spect -▁spread -▁sun -▁Cong -▁Congress -▁sex -▁saf -▁shar -epartment -▁communic -▁pretty -▁longer -▁carefully -work -wer -swer -▁answer -▁coop -▁statement -▁har -▁laugh -teen -▁colum -▁column -▁192 -▁river -▁scene -▁vict -▁novel -▁christian -▁publ -▁ox -▁oil -ux -▁successful -▁trip -▁leadership -▁differences -es, -hib -▁burn -tra -by -▁convention -▁live -▁foot -▁14 -ompan -▁merely -▁improve -▁prote -▁defin -▁annual -▁actual -eg -▁occasion -▁State -▁rate -▁narr -tal -▁total -▁surv -oint -arch -▁search -unch -▁sharp -▁optim -▁liqu -▁positive -▁England -ection -▁inde -▁belong -▁furn -▁experi -▁cop -▁appropriate -▁radio -▁expected -leep -▁offer -▁neighbor -▁neighborhood -▁clothe -▁contrast -time -owd -▁crowd -▁marri -▁married -▁marriage -▁write -▁latter -▁arch -room -▁organization -▁particularly -ottom -lin -iform -aith -▁faith -avior -▁approx -havior -▁according -ison -ais -▁rais -▁critic -▁summer -▁protect -▁ut -▁vot -▁den -▁surpris -ipt -cious -resh -agement -▁boat -▁box -atory -▁listen -acy -▁purpos -▁Europe -▁europe -itting -▁Department -▁scr -unday -▁east -atives -▁strugg -▁roll -▁unit -▁excell -▁machine -▁restr -▁corres -▁corre -▁opinion -▁extent -▁suffer -▁impos -▁previous -▁volume -▁Ad -▁mention -▁greater -▁approxim -axim -▁maxim -ship -▁possess -▁hall -▁Will -▁William -▁hor -▁horse -▁anyone -▁distance -▁distin -erc -▁exerc -inc -▁exc -▁perf -▁perfect -craft -▁uniform -▁fisc -▁fiscal -▁inside -▁believed -▁famous -ouse -▁couple -▁elements -▁buil -▁built -▁non -osphere -where -▁breath -▁voc -▁size -▁vocational -▁50 -▁sold -▁sleep -▁sof -▁sing -▁single -ators -▁reduc -ony -▁soon -ric -▁1958 --d -▁arrange -▁pref -▁satisf -▁nearly -pre -▁interpre -▁Even -▁price -▁adequ -▁descript -▁stra -▁straight -▁fight -▁democratic -ask -▁correct -▁distinct -▁Government -yond -▁beyond -▁exec -▁significant -▁bridge -▁bring -▁including -lectual -▁neck -add -▁bad -▁enem -▁stop -▁dog -▁politic -▁element -iveness -▁fifty -▁fifteen -▁seek -ants -▁Commun -▁status -▁wheel -▁shot -▁buy -▁jazz -ares -▁mouth -▁song -▁emotional -▁transf -▁jack -obile -lection -▁published -▁corresp -▁correspond -▁display -▁companies -try -arger -▁larger -▁bright -▁secur -▁principle -▁gain -jection -▁sus -▁taking -▁waited -▁From -▁yard -▁england -▁painting -aly -▁analy -▁management -▁House -offee -▁agreement -▁charg -cript -▁maximum -▁followed -▁di -▁dress -▁due -▁contribution -▁received -▁South -▁benefit -▁edit -▁changes -▁index -▁current -▁western -▁wish -▁evident -▁directly -▁recognized -▁developed -▁started -aves -▁nation -gregation -▁adver -▁protest -▁accompan -▁hear -▁leaving -▁entertain -▁23 -▁push -▁aircraft -▁assemb -▁events -ties -▁soft -▁Rep -▁slowly -usion -nection -▁connection -▁dram -▁coming -▁$1 -▁moved -▁encoun -au -▁De -ey -▁easy -▁despite -▁practice -▁game -▁schol -hington -alth -▁health -▁upp -▁proced -▁procedure -▁jun -▁Not -▁Kenn -▁studies -omb -▁bomb -▁Church -▁purch -iation -▁bank -▁Rob -▁murd -▁French -▁reached -▁red -▁exhib -▁extra -▁nuclear -▁exam -▁electronic -▁key -▁hy -▁hyp -▁hon -imin -▁transport -▁191 -▁193 -▁strange -lex -ox -▁Ex -▁Tr -▁additional -▁caught -▁chance -▁objective -▁training -uit -ivity -ailroad -uss -▁Russ -▁russ -▁argument -▁Robert -ique -▁original -▁vac -▁capac -▁capacity -▁trying -▁late -▁achieve -▁certainly -itted -band -usband -▁husband -▁German -▁german -▁compl -mber -bert -wood -rupt -ellectual -▁intellectual -udd -▁approximately -▁contact -anger -▁danger -▁warm -▁trust -ived -▁kenn -▁kitch -▁failure -▁Christ -▁refus -▁conscious -▁finished -▁protection -▁fine -▁advant -▁indicated -▁team -▁higher -orge -eorge -▁devot -▁controll -▁walked -hode -sh -▁fresh -▁comes -tic -▁decided -aturday -▁susp -▁added -▁assistance -▁son -▁samp -▁southern -▁sides -trument -▁behavior -▁emb -▁Nor -▁Neg -▁Negro -▁Now -odd -▁cause -▁request -▁construct -▁consum -aily -▁daily -ata -▁data -▁America -▁opposite -▁honor -▁discussion -theless -▁frame -▁24 -▁brush -orney -anch -▁mech -▁mechan -▁fix -▁musical -▁satisfact -▁North -ception -▁clim -▁average -▁editor -▁former -▁cooper -▁gives -▁requires -▁spot -▁lot -▁Man -▁' -▁Con -▁Cl -▁escape -field -▁conference -▁mission -ource -▁urg -▁bottom -▁appre -▁veh -▁vehic -here -▁however, -▁reve -▁reveal -▁ang -agg -ament -▁christ -▁National -▁educational -▁fundament -▁background -▁usual -▁doctor -▁radiation -▁completely -▁dest -▁destro -▁destroy -▁french -▁advantage -▁extreme -urr -co -oard -▁round -aughter -▁months -hold -▁West -ounc -▁inc -▁incident -▁resident -▁load -▁U. -▁U.S -▁requirements -▁factors -▁master -▁considerable -▁shape -▁studied -cul -▁calcul -▁Gr -▁libr -▁experienc -▁mem -▁memory -▁valley -▁virgin -▁virt -▁source -▁depart -▁resources -▁sources -▁names -▁clearly -chieved -▁subm -▁excess -▁scholar -▁spring -▁reference -▁referr -▁err -celer -▁acceler -urban -▁Although -▁operation -▁odd -partment -part -▁install -▁intens -▁intell -▁intellig -▁heavy -es. -▁Republic -▁republic -▁accur -▁accom -▁accompl -▁deb -ball -▁introduc -oviet -▁plat -▁platform -aint -aul -asons -▁reasons -▁operating -aged -agon -oid -void -ole -ille -ville -▁role -ett -▁imagination -ession -▁union -ashion -▁fashion -▁edge -well -▁aw -▁aware -wise -▁driv -▁prison -▁disturb -▁william -itiz -▁citiz -▁citizens -izing -▁execut -▁sitting -▁facult -▁gas -▁"Th -▁permit -▁cloud -▁active -▁drive -▁receive -▁danc -▁guard -udget -▁ep --c -▁announc -▁properties -▁Since -▁Se -▁Secret -▁Secretary -▁primary -▁primarily -be -▁narrow -▁include -▁promot -▁guest -▁respond -▁conditions -▁types -▁(1 -▁becomes -▁authority -▁cattle -▁battle -▁Col -▁sweet -▁washington -▁ability -▁precise -oul -ouis -ivers -▁parents -▁freed -▁freedom -▁indeed -▁desire -▁resist -▁march -change -▁exchange -▁exclus -▁convinc -▁distinction -▁rul -▁responsible -ourn -▁dramatic -▁ign -aign -▁campaign -▁membership -onscious -▁murder -▁kitchen -▁corporation -▁energy -▁hill -▁hyd -▁none -▁telephone -▁naturally -▁electric -▁quar -ipped -ed. -▁universe -▁expression -▁born -▁bal -▁balance -▁bit -▁bird -▁significance -▁equal -per -▁session -▁declar -▁weak -▁resolution -▁solution -▁justice -▁heavily -▁hours -▁attorney -▁likely -▁liter -▁literary -▁principal -▁principles -▁share -▁yes -▁yield -▁yester -▁yesterday -eneral -▁General -▁proce -akes -itute -▁merc -▁contemp -▁stream -▁divid -▁recall -▁exactly -▁liquid -▁constitution -), -). -▁13 -iar -▁familiar -▁star -▁whate -▁whatever -pir -▁Federal -▁island -chestra -▁attitude -▁knif -▁mult -▁multip -▁& -▁forward -▁cities -▁hom -rel -▁acad -▁academ -▁conce -▁robert -▁enemy -▁Act -▁previously -▁audience -etry -▁poetry -▁poor -▁match -▁agre -▁america -▁related -50 -roll -▁negro -ptember -▁substantial -▁jew -▁smil -▁instance -▁attract -▁intention -▁separate -▁shop -▁adop -▁mis -isk -▁experiences -ime -ultim -▁ultim -▁consult -▁repe -▁empt -ption -▁ideal -▁security -▁association -▁aspect -apid -▁rapid -iment -▁experiments -▁clay -▁cris -▁crisis -▁grand -cient -▁distribut -burg -▁pan -▁panel -▁altern -▁respon -▁response -▁necessarily -▁looking -▁officers -▁officer -year -ych -sych -▁psych -▁entr -▁assess -▁mess -reek -▁lock -▁la -▁Mar -unt -irt -irth -▁birth -▁repeated -▁others -▁elimin -▁understanding -▁writing -oto -▁photo -▁photograph -▁sensitive -ordin -erve -vey -▁survey -▁sport -▁counter -▁Univers -▁captain -▁mix -▁med -▁medical -board -▁arrived -▁Ph -▁Phil -▁switch -▁indicate -▁terms -▁stret -▁stretch -ushch -ushchev -rushchev -hrushchev -▁interpret -▁operations -▁everyone -▁ready -▁reading -▁rates -▁magaz -noon -▁afternoon -▁cook -▁coffee -▁bitter -▁double -mosphere -▁recognize -▁commer -▁commercial -ancial -▁financial -ery -yn -yl -▁upper -pap -▁equival -▁families -▁supply -▁childhood -▁vital -overeign -▁task -▁anim -▁animal -imm -▁snow -▁hung -▁tee -▁teeth -▁"I -▁top -▁host -▁minimum -▁reduce -top -▁beneath -▁blind -▁takes -▁Dr -▁truck -▁ordin -rove -roop -▁troop -▁grant -▁suppl -sue -▁issue -▁stopped -▁Soviet -▁agencies -oke -▁Be -▁Other -aries -▁baseball -itish -▁British -▁Brown -▁british -▁hol -arm -inner -▁dinner -▁22 -▁conviction -▁understo -▁understood -▁surr -▁drew -light -▁weight -▁sight -▁improved -▁budget -▁0 -▁Thus -▁sell -▁policies -▁apprec -▁trail -▁unique -▁twice -▁flat -▁learning -▁concerning -▁Another -▁mainten -▁typical -▁thinking -▁bul -▁bullet -▁ful -▁traff -▁combination -▁provided -▁disappear -▁interf -▁meant -▁somewhere -▁restrict -▁otherwise -▁shoot -▁atmosphere -▁increased -▁increasing -▁costs -▁suspect -▁env -▁analys -▁analysis -▁relief -▁hydro -iforn -ifornia -▁rose -▁spend -itude -▁First -▁fast -▁strik -▁loose -omas -▁moving -▁techniques -ottle -▁bottle -▁Southern -▁extensive -▁bur -ured -▁appe -▁appeal -ender -▁remind -▁him. -▁cand -▁candid -▁hit -▁determined -▁considered -▁pack -▁needed -▁cutting -then -▁railroad -rew -▁growing -▁rif -▁rifle -ney -▁machines -▁manager -ause -▁loss -▁historical -▁ourselves -▁log -▁practical -▁mountain -▁stead -anced -▁concerned -▁winter -ively -▁sexual -▁spent -▁troub -▁Jew -ule -▁culture -▁placed -▁passed -nect -▁connect -▁Char -ubb -▁expert -▁schools -▁ins -▁instrument -▁insist -▁instruction -▁surround -ront -▁confront -▁eg -▁anode -▁plastic -▁Yet -▁delight -▁roof -uous -▁avoid -class -odies -▁feed -▁reserv -▁quickly -▁shel -▁shelter -uzz -▁judge -▁George -athetic -fortun -▁fun -▁poly -▁pock -▁relatively -▁rise -▁trem -▁gold --m -▁realized -een -eenth -▁director -▁bond -▁suppose -▁suppos -▁medium -▁mount -cember -▁maintenance -abb -▁fat -att -▁remove -▁Car -▁minister -▁automobile -▁realize -▁chap -▁rout -▁presence -▁plain -▁plane -▁kid -spap -▁newspap -▁er -▁path -▁excellent -eless -▁defend -▁sky -▁sympath -▁arrang -▁ancient -▁useful -▁fully -uary -▁ordinary -section -▁et -▁moon -▁Pl -▁impl -▁impact -▁rough -▁rid -▁mid -▁integr -▁exercise -▁dust -ugust -▁hardly -▁slightly -▁60 -▁21 -crib -iber -▁lad -▁lov -▁pocket -▁neu -▁net -▁network -▁sett -▁settled -▁neutr -▁youth -▁conclusion -▁gather -▁weather -▁george --p -egre -▁practices -▁promis -▁permitted -▁philosophy -▁interior -▁conserv -icip -▁anticip -ges -▁deliber -▁deliver -▁region -uly -▁conclud -ault -▁described -▁presum -▁preserv -▁formula -iers -▁soldiers -▁created -▁May -ondon -▁envir -▁prior -▁based -▁independence -▁independent -▁soul -▁entirely -▁spiritual -▁Commission -▁happened -▁places -▁domest -▁signal -▁unless -▁giving -ournal -house -look -uel -uge -▁gradu -▁advert -▁gradually -▁assignment -▁Thomas -▁nevertheless -▁provides -ina -inet -▁ninet -▁nineteenth -ins -▁thin -▁machin -▁machinery -▁race -▁regul -▁Pr -orr -orrow -▁"the -▁setting -▁traditional -▁lives -▁lived -roph -urance -▁entrance -▁concret -▁dominant -antic -iant -▁profit -▁atom -▁advanced -▁advance -▁estimated -▁appeared -▁advis -viet -▁soviet -▁rever -▁reb -▁climb -▁shout -▁plann -▁jump -▁treated -▁react -▁traffic -▁intend -▁urban -▁intense -▁artic -▁advertis -▁War -▁knife -▁ran -▁rang -▁expos -ott -▁Jun -▁believes -▁hypoth -▁phase -▁depth -ures -▁cultural -▁negative -adow -▁shadow -how -▁somehow -▁abroad -yro -▁thyro -▁thomas -▁proportion -▁organizations -▁count -▁counc -▁council -▁suc -▁succe -▁served -▁showed -▁societ -▁greatest -▁daughter -▁encounter -▁accident -venue -▁Ed -traction -lor -▁Bl -▁stick -▁rain -▁attach -▁achievement -iest -▁comment -▁More -▁laid -▁Just -▁administrative -▁travel -▁prices -▁suprem -▁score -▁gard -ternal -▁faculty -que -▁spoke -gl -▁obl -▁oblig -▁fruit -▁rule -ipl -▁proud -▁enfor -▁vehicles -amb -▁trees -▁ultimate -▁adequate -▁crack -▁cry -▁army -▁sty -▁style -▁someone -▁ignor -▁waiting -▁scale -▁competition -▁grew -▁superv -▁junior -▁superior -▁message -▁cast -▁evil -culiar -aneously -▁Western -▁domestic -iplom -▁weap -▁speed -▁spee -▁speech -▁stepp -ke -olk -▁folk -▁concert --h -▁opportunities -▁motion -▁obtained -▁efforts -▁journal -▁writers -▁yellow -▁While -▁White -burn -▁gray -iday -▁simult -▁simultaneously -aces -ees -▁employees -▁judgment -▁continued -acr -▁sacr -ron -▁syn -▁grass -▁dropped -▁anti -▁chairman -▁Jack -▁plus -▁tub -▁surprise -▁throw -▁happy -▁smell -▁Here -anover -▁weakness -▁reject -▁library -▁East -dom -claim -▁sav -▁economy -▁thro -▁stain -▁stor -▁organ -▁expend -▁expand -▁eventually -▁uncertain -▁critical -ets -▁learned -▁everybody -▁University -▁reality -pril -▁lip -▁numer -▁communication -▁Frank -▁mail -▁device -▁Catholic -▁catholic -▁prepared -▁margin -irgin -▁Virgin -▁selection -▁pray -▁application -▁honest -lock -▁income -▁Committ -▁vill -▁village -illiant -▁brilliant -▁minor -▁afford -ones -▁tour -▁send -▁practically -▁struggle -lease -▁powerful -thetic -▁earn -▁reported -▁struck -orough -▁thorough -▁friends -akespe -▁contempor -▁contemporary -▁apply -ricult -▁agricult -▁por -▁threw -ictionary -▁seriously -abric -▁fo -▁foam -▁cat -tribut -uesday -▁opened -▁dim -▁dictionary -▁artist -▁billion -▁raised -▁using -bruary -▁included -▁abstract -▁diame -▁possibilities -▁equally -▁emphasis -isto -▁register -evit -▁engaged -▁traged -▁truly -▁nam -etition -▁proceed -▁ahead -▁However, -enry -tery -olute -▁absolute -▁associated -▁purchase -aud -udson -aise -▁results -▁located -▁spl -▁coast -▁methods -▁purposes -▁ce -▁bound -▁Qu -▁fourth -val -▁approval -▁pal -▁pound -▁pitch -▁instant -▁replace -▁catch -arget -▁target -▁bus -▁bodies -▁circl -▁scholarship -▁reduction -obb -▁rob -▁sen -ened -gen -zen -mans -▁expans -ma -▁operator -▁girls -▁port -▁U.S. -▁guid -▁engineer -▁dozen -▁goal -▁moments -▁dry -▁drunk -▁intensity -ern -▁Every -aur -▁restaur -itation -itz -ze -izz -▁tit -▁title -▁eth -▁definite -▁vote -▁affect -▁trend -▁unlike -▁drove -▁blank -▁sovereign -▁communism -▁Tw -▁fabric -oles -ateg -leg -apan -▁vision -▁examples -▁agency -▁compon -▁components -▁boys -oseph -▁stations -▁store -▁card -▁pleasure -▁adult -▁yourself -▁comparison -▁established -▁filled -▁designed -▁warr -arry -▁cow -▁Each -▁By -abilities -ancy -ancisc -onday -ined -▁diplom -▁desk -lict -▁conflict -▁confidence -▁aid -ef -▁evening -▁unf -▁rare -▁anti- -▁drawn -▁corn -agn -▁apparent -▁americans -▁convict -▁registr -▁etc -▁editorial -▁effic -▁governor -▁mechanism -mouth -ago -icago -▁Chicago -▁academic -apt -▁shr -▁nodd -chester -▁apartment -▁rapidly -▁precision -▁peculiar -ewhere -▁prep -▁efficient -▁substantially -▁garden -▁tendency -▁warning -▁flash -▁writer -omial -▁entered -▁enterpr -▁personn -▁personnel -ayburn -▁broken -▁broke -▁So -▁drug -▁assumption -▁mold -▁bes -▁beside -▁besides -aced -▁replaced -▁recover -▁Such -▁horiz -▁proposal -ocr -emocr -▁Democr -empl -▁closed -▁shift -▁shock -▁hydrogen -arg -▁involves -▁expense -▁expensive -▁jury -abit -▁habit -▁cab -▁chem -▁item -▁chemical -iency -▁efficiency -▁tast -▁vast -▁vary -▁enorm -▁fem -▁female -▁portion -▁agricultural -▁Committee -▁interference -erlin -ving -▁situations -▁tension -▁mist -▁mer -▁40 -akespeare -cle -▁muscle -▁primitive -▁huge -▁shook -▁approved -olved -▁victory -▁identification -▁pure -omin -com -▁accomplished -▁involve -▁inform -▁aband -▁abandon -▁relax -▁iron -▁location -▁died -▁settle -▁loud -▁easier -▁ou -▁Z -▁factor -▁neutral -▁Charles -▁beauty -▁access -▁acres -▁hun -▁hang -▁tang -▁hors -▁doing -▁elabor -aire -▁environ -▁environment -▁capable -▁gall -▁Louis -▁louis -0000 -▁nine -▁nav -▁circle -▁enth -▁enthus -▁enthusi -▁needs -nown -known -▁expansion -▁anx -▁anxiety -▁criticism -▁standing -▁lie -▁teachers -▁teacher -▁psychological -rol -▁intelligence -▁working -aking -ishing -egrees -berg -▁executive -▁regional -▁file -▁drop -▁changed -▁remained -▁sea -▁artery -▁romantic -▁conj -▁congregation -▁retain -▁recre -▁"W -▁hypothe -▁entit -▁entitled -lev -▁victim -▁description -ression -▁cheek -▁highly -▁highway --b -▁harm -▁increases -▁throat -▁nearby -▁sand -▁sympathetic -▁lunch -while -▁reform -▁categ -▁strateg -▁identified -▁identity -▁returned -▁arms -century -▁reflection -ope -▁liber -▁produced -▁contrary -▁possibly -▁palm -▁announce -usual -▁unusual -use -▁metaph -▁Communist -▁joy -▁metal -▁pred -▁predict -▁telev -▁television -▁fright -▁dut -▁penn -▁impression -▁risk -▁visual -▁colon -▁reasonable -umbled -▁headqu -▁headquar -▁headquarters -▁kenned -▁kennedy -▁lady -▁station -atically -▁cub -riet -inted -ointed -▁appointed -put -▁colleges -▁aggress -▁Sunday -▁sunday -▁eval -▁evalu -▁persu -▁legal -▁Japan -▁japan -▁financ -▁smile -▁smo -▁smooth -▁fundamental -▁Kenned -▁Kennedy -▁angel -▁fraction -▁suggested -▁fixed -▁Rhode -▁rhode -▁putting -▁flight -erson -ston -▁tremend -▁gent -▁gentle -▁proposed -▁numerous -▁communist -ette -agger -▁rot -▁rod -▁controver -▁capt -▁foc -▁focus -▁cryst -▁technology -▁quant --year -▁sample -▁unknown -▁telegraph -▁letters -▁safe -▁careful -▁technique -▁represents -▁content -▁conven -▁Arm -▁named -▁covered -▁grave -▁utter -▁objection -lfare -▁caused -▁confirm -▁launch -▁safet -etic -▁workers -amber -▁recognition -obby -▁weeks -▁establishment -unicip -▁Park -▁confusion -▁knock -▁tract -▁detective -▁impressive -▁questions -ecause -argest -▁largest -▁orders -▁petition -▁Two -▁convers -▁majority -▁providence -asc -exas -clos -osition -▁definition -▁spons -▁subs -▁fool -▁rules -▁weapons -▁weapon -▁virtually -▁football -▁residential -▁tone -▁stories -▁whisk -▁wit -▁witness -▁expendit -▁expenditures -omen -enomen -▁phenomen -▁electrical -▁illusion -▁guess -▁guil -▁pil -▁pilot -▁prem -▁adoles -▁internal -▁article -▁varied -er, -orld -▁World -▁neglect -▁negot -▁champ -▁date -▁insurance -▁satisfactory -▁automatically -ers, -ed, -▁course, -ades -▁invent -▁interpretation -▁san -▁sem -▁Dur -tober -usett -achusett -▁bath -▁strain -▁strip -▁norther -▁Dr. -▁gar -ovember -▁statements -▁fiction -▁fig -▁adopted -anuary -▁James -▁games -▁james -▁simpl -ervice -right -ylv -ylvan -ylvania -▁spectac -▁ge -▁gro -"? -▁desired -▁doctr -▁conventional -▁wagon -ography -▁merch -inch -▁persist -▁absence -manship -▁scar -▁wine -stand -▁band -▁houses -▁accomplish -▁sched -▁schedul -▁schedule -▁schem -ky -▁preserve -▁equivalent -▁tables -▁appearance -▁dance -▁$2 -▁payment -▁empty -▁presumably -gun -go -▁context -ats -▁swung -▁flu -▁wil -▁inevit -imony -▁testimony -▁confus -▁strike -▁humor -▁fit -▁pale -▁decreas -▁nerv -▁exercis -▁tempor -▁chos -▁chosen -▁chamber -▁driver -▁River -▁bare -▁burd -▁During -umber -▁maybe -▁horses --s -▁ought -▁resistance -▁wave -▁sought -organ -orp -▁Corp -▁blow -▁introduction -iated -▁spite -▁denied -▁cloth -▁31 -▁barr -▁eleven -▁admitted -▁reception -▁hole -ulation -point -▁appoint -▁Americans -▁league -▁lean -▁announced -▁burst -▁strict -▁motor -▁acceptance -▁accepted -▁encourage -▁sentiment -▁restaurant -▁institutions -igr -hower -enhower -isenhower -▁silence -▁involving -▁percentage -annel -▁orchestra -▁sin -oose -ociation -▁extension -fore -ien -▁alien -▁ec -▁wonderful -▁choose -▁particles -ometer -▁qualities -issue -▁wat -▁waters -▁wished -▁missile -▁promise -▁prompt -▁promin -▁root -▁vacation -▁difficulty -▁hur -▁points -▁societies -ensions -▁Prov -▁Providence -▁emotion -▁Cour -▁Court -▁opinions -▁Her -her -▁id -▁charles -▁estate -▁diameter -▁quarter -▁supplies -▁chapter -▁frequencies -▁artists -▁steps -▁millions -▁opposition -▁prove -▁pros -▁prospect -▁persons -▁steel -▁Rich -▁Richard -he -▁save -uto -▁communities -▁thyroid -avid -▁languages -▁occasional -▁pair -▁stair -▁repair -▁doctrine -▁stranger -▁streng -▁1957 -▁bron -▁bronch -▁ly -▁u -▁utop -ued -▁shut -▁closely -▁reven -▁advice -▁faces -▁bab -▁delay -▁cele -▁celebr -▁curve -▁destruction -▁richard -ra -▁irr -30 -▁automatic -▁marked -▁funds -▁thickness -▁quot -▁combined -ulated -▁liberty -sequ -▁subsequ -▁purchas -▁survival -▁stated -▁extraordin -▁coordin -▁edition -▁flex -▁Alex -▁Before -▁ease -▁Democratic -▁Khrushchev -▁khrushchev -▁hearing -oom -▁(A -▁wash -▁measurements -▁dollars -▁leaves -▁waves -▁March -▁violence -▁tall -▁taste -▁assume -▁metrop -▁raise -▁ballet -▁tree -▁favorite -▁site -▁phr -▁phone -▁cock -▁"The -▁breakf -▁breakfast -▁grown -▁Union -▁brain -▁refr -▁refrig -▁imposs -▁impossible -▁bed -▁bedroom -▁Compan -▁swift -▁mir -▁objectives -▁occasionally -▁accuracy -▁soldier -▁tough -▁planned -▁hypothal -▁hypothalam -▁tal -vember -▁older -▁surrender -▁render -▁track -▁Club -▁valuable -▁indicates -▁charm -▁mal -▁causes -▁agreed -▁angle -▁Colle -▁College -lement -▁supplement -entially -▁essentially -▁poll -▁civilization -▁equipped -▁adj -▁Aj -▁egg -▁decade -▁explan -▁reduced -▁consequences -iences -▁phrase -incol -incoln -▁devices -issipp -▁generation -▁chicago -ompson -▁investment -▁offered -▁transform -▁conform -▁assured -▁Sp -▁emergency -iable -▁relation -ama -▁Old -▁unfortun -▁transfer -▁Black -idden -▁visited -▁rush -▁ris -▁Moreover -▁moreover -▁cup -▁it. -▁expens -▁wage -▁archit -▁architect -▁suburban -▁soil -▁allowed -▁picked -▁figures -▁sail -▁prepare -▁congreg -ental -▁severe -igor -▁vigor -▁profound -▁wound -▁absor -▁teaching -▁diagn -▁imagine -▁taught -▁completion -▁metropol -▁1954 -▁peop -▁peoples -▁treas -enetr -▁London -▁london -▁exception -▁programs -aret -▁marsh -▁commerce -rest -aken -▁screen -▁prefer -▁conversation -▁libraries -▁curious -▁grace -▁providing -▁ho -▁senior -terior -▁veter -▁magnific -▁sho -▁shares -▁admission -▁70 -athem -▁mathem -▁mut -▁buck -▁bought -▁shap -cience -▁conscience -oices -▁preparation -▁tissue -▁representative -▁legislative -▁gh -▁* -▁passage -▁slaves -▁fasc -▁considerably -▁inevitably --r --l --st --1 -▁189 -▁(19 -▁plot -▁roy -ny -▁duty -▁oce -▁chain -gain -▁gross -iological -▁implic -olf --w -▁screw -▁preach -▁magnitude -▁smoke -▁minute -▁fought -▁privile -ri -▁wore -▁Re -▁royal -▁flowers -organiz -▁rank -▁concrete -▁competitive -▁candidates -▁transferr -▁porch -▁welfare -▁suburb -▁everywhere -▁dam -▁twist -▁tragedy -▁priest -▁propag -▁representatives -▁scream -▁Virginia -▁accompanied -▁baby -▁skin -▁physic -▁highest -▁effects -▁guide -▁currently -▁authorities -▁util -▁utility -▁sponsor -▁genu -▁genuine -▁marine -▁prevail -▁territ -▁effectiveness -stitution -▁penetr -▁puzz -▁grain -▁pow -▁powers -▁payments -▁separated -▁emotions -uments -▁silent -vent -▁sevent -▁transl -▁enormous -▁eat -▁stages -itial -▁initial -▁exhibit -▁disting -▁distingu -▁distinguished -iving -▁create -▁funeral -▁universal -▁universities -▁municip -tray -▁assumed -▁supposed -▁Russia -▁Russian -▁removed -▁succeed -ibr -ylind -aze -▁chest -▁plans -ongue -▁tight -▁corps -morrow -allas -▁loan -▁reorganiz -▁wear -▁talent -ctor -▁hat -▁bear -▁rear -▁measured -▁difficulties -▁November -▁strengthen -thentic -▁sick -▁myth -▁Stre -▁Street -▁voices -▁churches -▁safety -▁nobody -▁integration -▁begun -▁tear -▁watched -▁watching -▁twel -▁twelve -▁burden -▁unity -▁thrust -▁thrown -▁soph -▁sophistic -▁transition -eff -efferson -▁Off -ieff -ofieff -kofieff -▁features -▁talking -▁mic -▁clothes -▁products -▁groups -▁edges -▁pleasant -▁worr -▁worry -▁worse -▁worship -▁fellowship -▁perspect -irty -▁neighbors -▁coating -▁played -▁playing -▁virginia -▁Har -▁newspaper -▁stim -▁stimul -▁counties -▁calend -▁calendar -▁california -▁milk -▁goes -uries -▁centuries -▁tongue -▁26 -▁27 -▁discour -▁discipl -▁external -▁feather -▁concentration -▁systems -▁horr -▁employment -▁crew -▁Where -▁rh -ham -▁cam -ald -▁Af. -▁explanation -▁liquor -ander -▁brand -▁propagand -tun -▁fortun -▁volun -▁volunte -oston -▁creative -▁stem -▁stone -▁Many -▁adam -back -▁becoming -noc -ners -▁storm -▁laboratory -▁nice -▁erect -▁powd -▁authorized -artin -▁achieved -▁funny -osc -oscow -▁Und -lantic -▁parties -▁vague -▁vivid -▁mile -▁kick -▁example, -▁leader -otten -▁resum -▁valid -▁Indian -▁Count -▁storage -izes -▁rural -▁patrol -▁attribut -▁friendship -▁saying -▁says -▁alex -▁innoc -▁chall -▁channel -▁Merc -umin -▁opening -▁rational -▁forms -▁Most -▁prominent -outhe -▁readily -▁desper -▁sciences -▁darkness -▁affirm -▁affection -▁deny -▁papers -▁saving -oyal -▁loyal -arian -▁tremendous -▁seld -▁fulf -▁fulfill -▁guidance -▁worked -▁selected -▁pulled -▁expressed -▁suspic -▁suspend -▁ocean -▁calm -▁interval -ayne -gry -▁rhy -▁rhyth -▁rhythm -iance -ius -ios -issippi -▁App -▁smith -aker -▁closer -▁smaller -▁recreation -▁legislation -▁transportation -▁cooperation -▁resemb -lem -more -▁shore -gotten -ta -rietta -▁joint -ouncil -▁Council -▁coal -achel -itut -itain -▁drain -pecific -▁September -ancisco -▁hero -ato -▁rat -▁ratio -▁native -▁Co -▁City -▁personally -▁Hanover -▁hanover -ipping -▁Aj. -▁sequ -▁sequence -isdom -▁wisdom -abel -▁wel -▁welcome -▁snake -▁ambig -▁fairly -▁golf -▁matur -▁dispute -▁condem -▁pride -▁200 -▁2: -▁legislature -▁Henry -▁henry -▁arc -▁U.N -▁movements -▁stiff -▁Both -▁household -▁Once -▁vehicle -▁coinc -▁probability -▁remembered -▁alternative -▁conservative -▁lifted -▁unconscious -▁mad -▁properly -▁largely -▁alive -▁alter -▁handsome -oss -▁insect -▁career -▁muscl -▁arrangement -▁arrangements -▁lect -▁compound -▁them. -▁hunting -▁swimm -▁june -▁courage -▁occupied -▁supplied -▁cylind -▁reput -play -▁mentioned -▁vel -▁veloc -▁somebody -▁dangerous -▁classes -▁busy -elph -elphia -adelphia -gia -orgia -▁Ang -▁Angel -▁mel -▁melt -▁radical -▁Bar -▁submar -▁friendly -ieved -▁seldom -lo -▁completed -▁surpl -▁craft -▁crash -▁cred -▁credit -▁clar -ivision -▁Among -▁frag -▁mental -▁poems -▁includes -▁interested -▁discovered -aver -▁attractive -rm -▁Our -▁vers -▁version -▁consists -▁ice -▁notice -place -▁swept -▁Only -▁nur -▁led -▁Adam -apped -▁proved -▁deleg -ibilities -▁responsibilities -▁salt -inct -▁instinct -▁inches -▁therm -▁conclusions -▁morgan -▁phosph -▁compared -red -▁die -▁Em -▁El -urried -▁worried -rupted -thodox -night -mann -▁Never -▁portray -▁instruct -▁Gre -tronom -utonom -down -▁alleg -▁complic -▁complain -▁passion -▁passeng -▁Britain -▁constitute -▁said, -▁yank -ureau -▁repor -▁gift -▁June -▁1962 -▁propriet -aked -artmouth -▁performances -▁instances -▁advantages -▁king -▁perman -erk -▁clerk -▁lawy -kind -▁mankind -▁personality -ulf -bon -emble --S -igaret -▁volunt -▁ambition -▁won -▁unw -▁musicians -▁Rev -▁worst -▁knees -▁improvement -cend -▁chart -▁maturity -ality -expect -▁emission -inese -▁chlor -▁necessity -▁remot -▁France -▁Friday -▁holiday -▁friday -▁visible -avery -▁190 -▁peas -▁decor -▁expenses -▁Sch -▁absorb -▁Company -▁lose -▁fourteen -▁remarkable -▁lat -▁diffusion -▁materials -▁luck -▁subtle -istotle -rozen -oman -▁explor -▁graph -▁anybody -attered -▁sew -▁x -▁ranch -▁barn -▁merit -▁December -▁lord -▁precisely -ged -▁departments -▁Because -▁extremely -▁Berlin -▁surplus -▁territory -▁theory -▁studio -▁structural -▁homes -▁painted -▁identify -▁failed -▁berlin -▁comparable -irable -▁desirable -quir -▁comparative --degrees -ores -▁memor -▁deliberately -▁abrupt -▁manufacturers -▁composition -a, -▁owners -ohnst -ohnstamm -▁steep -▁dispos -▁Service -▁sid -▁sidew -▁sidewalk -erry -erhap -▁Perhap -▁luc -▁indirect -ustain -▁restrain -▁output -▁pul -▁ul -▁ultr -ulate -▁compul -▁outr --C -▁Though -▁Those -▁appears -istrict -▁northern -▁lyric -▁carrying -▁barb -▁advent -▁downtown -▁remains -▁permission -iles -egregation -▁desegregation -▁camer -▁refriger -▁release -▁dyn -▁oxy -▁oxygen -▁scen -▁profession -▁conjug -ites -▁stayed -rook -▁agent -▁agents -▁Pe -▁Peace -▁genius -▁purs -▁pursu -▁Sc -adequ -▁calculated -▁eag -▁specifically -▁colony -▁december -▁continuous -▁seat -▁cours -▁courses -500 -▁300 -▁removal -▁pink -▁pip -▁Scot -igger -▁winning -together -▁magazine -▁routine -▁chlorine -▁squee -▁knee -▁cruel -▁drill -▁descend -▁wire -▁unexpect -abil -▁britain -▁carbon -▁years. -▁occurred -▁october -nomial -▁polynomial -▁dise -▁disease -▁colors -▁errors -▁error -▁link -▁dism -stead -ly, -▁kiss -▁cigaret -▁cas -▁Further -▁Hudson -▁reserve -▁residence -▁satisfied -▁replied -▁satisfaction -▁Do -cell -▁mood -▁scientists -▁flood -▁russian -▁russia -▁anywhere -ude -▁prosper -degrees -▁amend -▁Chinese -naire -▁questionnaire -now -cknow -▁irreg -▁irregular -▁ones -▁patch -▁barrel -▁doctors -▁embarr -▁embarrass -▁magazines -▁observed -▁successfully -▁absurd -west -ced -▁forced -▁Mc -cular -▁institute -▁School -▁buff -▁jewish -▁symph -▁textile -ifications -▁observations -▁casual -▁extraordinary -▁inevitable -▁unable -▁sixteen -▁mineral -40 -▁apart -▁professor -born -▁despair -uces -udge -▁nose -▁introduced -▁jacket -sland -▁Island -▁atomic -fare -▁warfare -▁termin -iny -▁tiny -▁gyro -▁condemn -▁operate -▁planet -▁permanent -▁bord -ret -▁industries -▁Cent -▁surfaces -▁pun -▁widely -▁dismiss -▁oh -▁rac -▁happens -▁driven -▁Roman -▁regardless -"! -▁september -▁Conf -foot -▁rode -▁blond -▁silver -▁municipal -▁records -▁rust -▁preferr -▁interesting -▁rising -▁drawing -anish -▁finish -▁distinguish -▁furnish -▁Jewish -▁sweep -bit -▁arbit -▁Sometim -▁intersection -aust -haust -▁exhaust -▁accus -▁export -▁reporters -▁heat -▁extr -▁books -ening -▁wooden -enrietta -▁multiple -▁fle -▁proclaim -▁eternal -▁salary -▁salv -▁decide -▁veget -▁symptom -▁enterprise -▁terrible -▁driving -▁Par -▁variable -▁variables -athed -▁illustrated -▁sixty -▁manif -▁manifest -ifted -ife -▁hell -▁Therefore -▁investigation -▁scratch -▁page -▁enthusiastic -▁eld -▁convey -▁november -▁dirt -▁refused -▁assert -▁alert -▁magnetic -ifice -▁sacrifice -▁vice -▁dict -▁retired -▁retire -▁tired -▁queen -▁seventeen -▁Ass -▁Association -▁discussed -wave -▁skywave -▁Let -▁La -▁preced -lfred -▁collabor -▁jur -▁juris -▁jurisd -▁jurisdiction -▁destiny -▁angry -▁legend -▁reprod -▁ladies -▁shirt -▁leading -▁boot -▁sket -▁mistake -▁fluid -▁incomp -▁hurt -▁paint -▁height -▁insight -▁exagger -▁earning -▁estimate -▁accurate -iumph -▁triumph -▁escap -▁exclud -▁respectively -ministr -▁Administr -▁relev -▁reasonably -▁July -▁july -otton -▁cotton -orship -▁neat -▁mechanical -▁sam -▁saturday -▁Saturday -▁works -▁passing -▁circuit -▁fish -izable -acific -▁beach -▁units -▁sister -▁coat -▁prose -ansas -▁constitut -▁years, -▁entry -erted -umble -▁monument -raid -▁afraid -▁maid -▁bloom -▁aspects -▁identical -ibly -▁trib -▁label -▁bargain -aware -▁bread -▁grade -▁finding -▁fighting -▁accurately -▁commit -▁ecum -▁quarrel -▁whipp -▁articles -istry -▁outstand -▁sugar -▁route -▁thank -▁quantity -ae -▁note -▁contribute -▁defe -▁County -▁cells -cast -▁broadcast -▁fly -▁fing -▁fingers -▁finger -▁fog -▁slavery -▁consideration -▁crime -▁cru -▁crucial -▁craw -▁geograph -agraph -▁paragraph -▁paral -▁paralle -▁parallel -▁dign -▁dignity -▁der -▁derived -▁cheap -▁doc -▁document -▁Texas -▁substitut -▁organic -▁ownership -▁discussions -▁Great -▁Greek -▁greek -▁diplomatic -▁supreme -▁Rome -▁181 -istol -▁Mary -▁mary -▁offices -▁nons -alfre -▁sewage -▁branch -▁regiment -▁regions -▁regret -▁camera -▁patri -▁Ne -▁Board -▁potato -▁unp -encil -▁pencil -▁bay -▁essay -▁bunk -axon -▁nonspecific -▁Fort -▁District -▁pian -▁please -iably -▁proport -▁sad -▁realistic -▁artistic -▁pup -▁pupil -enced -aneous -▁Lord -▁wilson -▁proof -▁examination -▁examined -▁examine -▁attitudes -▁charged -▁charges -▁mort -▁texas -▁stockhold -▁stir -▁secure -▁conceived -▁helpl -▁helpless -▁flux -▁comfortable -apping -happ -▁unhapp -▁remote -▁eager -ighter -▁ster -▁sustain -▁magic -▁wise -ogue -▁virtue -▁argue -▁Through -▁pour -ourney -▁journey -▁fired -▁spectacular -▁patent -▁component -imination -▁onto -▁experienced -▁delic -▁shooting -▁defeat -▁divided -▁Eisenhower -▁sports -▁explos -▁prohib -mon -stair -▁invar -▁invariably -▁Laos -▁Cub -▁Comp -▁characters -▁confident -▁rail -▁rig -▁rigid -roduct -▁distant -▁massachusett -asp -▁grasp -▁China -▁china -▁auto -▁ren -▁renew -▁reh -▁rehabil -▁rehabilitation -▁Le -▁League -▁continuing -▁hopes -ougl -▁tribut -▁dimension -▁dimensions -▁sufficiently -▁chinese -▁coach -▁sadd -▁steam -▁leather -▁liked -▁anxious -▁ecumen -▁ecumenical -tical -▁affairs -▁brook -▁contest -inomial -▁consciousness -▁observe -▁normally -wich -▁partners -nie -▁(1) -▁Nevertheless -▁glance -▁measurement -▁displace -▁adjustment -▁phon -▁furnit -▁furniture -▁pace -bor -▁harbor -▁harv -▁plaster -thur -orton -▁Philadelphia -▁philadelphia -▁28 -▁188 -▁29 -▁mobile -▁perception -els -▁"S -▁creation -▁destruct -ulous -▁temporary -▁Their -▁formal -areness -▁awareness -▁chron -▁patron -▁chose -▁chrom -boy -▁excite -▁Mike -als -alifornia -▁California -▁logical -▁caf -▁tact --time -▁imitation -▁sweat -▁guilt -▁portra -▁"p -▁"A -▁"f -▁centers -▁bride -▁cav -▁crop -▁dial -▁volumes -employ -▁plent -alian -▁empir -▁prime -▁dynam -assachusett -ics -▁medic -▁subjects -▁April -▁marriages -▁pond -▁follows -andom -▁random -ster -▁myster -▁splend -▁split -▁frustr -▁Soc -▁Society -▁Sen -▁participation -▁frequent -▁seed -▁continually -▁weekend -▁specim -▁publication -▁cabinet -▁enforce -▁retail -▁exclusively -in' -▁troubled -▁troubles -▁Industr -▁India -▁customers -bur -burgh -▁determination -▁fifth -▁losses -▁Manchester -▁Arthur -▁Class -erences -▁conferences -▁ambiguous -▁directed -▁uncle -▁parad -▁arthur -▁advertising -outheast -▁declared -▁newspapers -adequate -▁organized -▁convinced -▁pulmon -▁leap -▁scattered -▁cars -▁Figure -▁latest -▁Prot -▁Protest -▁earnest -▁dependent -▁vent -▁substance -▁St. -▁increasingly -▁rarely -▁August -▁items -▁parts -▁binomial -▁acquired -▁talents -ys -▁metaphys -▁Army -itated -▁irres -▁heaven -olly -▁wholly -▁freight -▁irre -▁sanction -▁starting -▁mo -▁denomin -▁existing -uthentic -▁joined -▁plenty -▁Joseph -▁joseph -▁challeng -▁specified -70 -▁Penn -▁blanket -▁altogether -▁furnished -▁Luc -▁revenue -▁revenues -▁categories -▁informed -▁acquire -inda -▁pistol -▁Ear -▁temple -▁yards -▁estimates -▁roar -▁greatly -▁pushed -▁complicated -▁decades -▁steady -▁steadily -ickey -▁congo -▁paradox -▁subsequent -▁sovereignty -▁turning -▁dragg -wick -▁lew -▁Sw -▁tempt -▁chick -▁striking -loride -quest -▁wages -anta -lanta -▁antis -▁temporarily -▁Paris -load -▁bull -▁comput -▁imagin -▁tail -▁Charl --up -▁changing -▁sentence -▁innocence -▁consequence -▁consequently -▁exceed -▁Wil -▁Wilson -▁watson -▁hence -natur -▁forty -▁forth -▁arbitr -▁frames -▁france -▁contempt -▁contempl -tric -ologist -▁species -▁Under -▁threatening -▁threatened -third -▁Morgan -▁Administration -▁phenomenon -▁thoroughly -▁dirty -▁ultimately -▁mysterious -acious -▁handle -▁Why -angel -▁historian -do -▁outdo -▁argued -ino -inois -▁fallout -▁occurrence -herent -▁ju -▁flexible -▁allot -▁allotment -▁complement -▁entertainment -▁excitement -▁guilty -type -cc -▁obsc -▁frightened -▁bigger -▁lumber -▁exclusive -▁Sam -▁75 -▁attain -oven -▁slipped -▁tur -ese -▁Japanese -▁japanese -oe -▁fault -▁corrupt -▁raw -▁rubb -▁diagon -▁McC -kins -▁wal -▁category -▁Protestant -▁client -not -▁brid -▁scarce -▁skirt -▁senate -▁meanwhile -▁row -▁bronchial -▁congr -▁Three -▁convenient -▁apparat -▁apparatus -▁campus -▁mike -▁challen -▁challenge --class -▁drama -▁printed -ias -▁policem -▁interpreted -▁appreciation -▁advoc -▁combat -▁ble -▁outer -▁asking -ommun -▁slave -▁earliest -under -▁accomm -▁Edward -▁award -▁edward -▁finance -▁antagon -▁holding -▁elaborate -low -pection -▁produces -▁decline -▁Monday -▁monday -▁Me -▁cafe -▁shows -▁uns -▁saddle -▁Shakespeare -▁farther -yearold -▁bother -▁script -▁eisenhower -▁movie -▁tremb -▁fascin -▁balanced -line -▁contained -▁extract -▁laos -▁killed -▁assault -▁senator -▁housing -▁rifles -▁roles -▁bag -▁scot -▁beat -▁bell -▁ways -▁capabilities -▁probabilities -▁inspection -plane -▁counterpart -▁Fin -▁Finally -▁reluct -▁martin -▁Martin -▁releas -▁appreciate -grad -▁duties -▁metropolit -▁aboard -▁frequency -▁requirement -▁perspective -▁belt -▁demonstrated -20 -▁Out -▁trembling -sewhere -▁elsewhere -▁suffering -▁politics -▁excit -▁excited -▁him, -▁survive -▁knows -▁micros -▁inspect -▁developing -▁verb -▁literally -▁pursuant -▁cigarette -▁indian -aulk -aulkner -▁Paul -▁paul -▁divor -▁divorce -▁divine -ceived -iece -▁aside -▁discipline -▁encouraged -▁stom -▁stomach -▁ward -▁till -errill -▁guerrill -▁pill -▁assigned -▁divers -▁drivers -▁don -▁nick -▁sections -opp -▁faced -▁regime -er's -▁code -▁drum -▁accum -umn -▁collective -▁reports -sues -▁issues -▁compuls -▁describe -▁Junior -▁1952 -▁1950 -/2 --to -▁modest -▁february -▁destructive -▁buildings -▁standards -sylvania -▁Pennsylvania -▁pennsylvania -▁vigorous -▁devil -▁lewis -▁prevention -▁revol -▁deter -▁deterg -▁stared -▁Cap -▁scrap -▁eighteenth -▁excuse -antle -▁assistant -▁medicine -▁ethical -▁muse -▁museum -▁rome -▁anger -olecular -ensional -evens -agner -▁38 -keep -▁dish -▁bow -▁ow -▁pictures -▁tomorrow -▁hopeless -▁metropolitan -▁pretend -▁judgments -▁piano -▁frozen -▁crouch -▁processes -▁contrad -shop -▁jones -▁charl -▁abstraction -▁absent -▁innocent -▁punish -▁tun -▁glanced -▁transm -▁ax -▁axis -▁restor -▁forget -▁sinc -▁submitted -▁immort -▁immortality -▁immigr -▁milligr -▁compre -▁compreh -▁comprehensive -▁sympathy -mith -▁Smith -ishop -▁automob -▁automobiles -▁missiles -▁patriot -▁constantly -▁mine -▁Germany -▁enforced -▁experimental -▁High -whel -whelm -▁overwhelm -▁firmly -ingly -ing, -estival -▁flower -power -▁widow -▁David -▁david -▁thread -pread -▁roman -▁Wood -▁individuals -▁diss -▁horn -▁hide -▁hamm -▁strictly -▁crown -▁samples -▁qualified -▁mixt -▁mixture -▁appr -▁Cr -▁du -▁excessive -▁distress -case -▁extended --t -▁Art -▁paus -▁teen -▁georgia -▁outlook -▁desires -▁relieved -▁Francisco -▁exhibition -▁engagement -▁procedures -▁magnificent -▁"M -▁lung -▁speeches -▁carp -pired -▁inspired -▁fiber -▁accommod -▁boil -half -▁complaint -▁assure -▁disaster -▁regulations -▁objects -▁nut -utch -dr -ren -▁Centr -▁charter -▁curios -▁Senate -▁variation -▁legs -▁happiness -▁physiological -▁illumin -▁disch -uccess -▁quarters -▁Dallas -▁Du -▁projects -▁palace -▁sigh -ighth -inson -▁stepped -ruit -▁recruit -▁Furthermore -aution -cence -▁adolescence -▁Civil -oil -▁substitute -stitute -▁morph -▁mistaken -▁greet -▁fur -▁smiled -▁amaz -▁employee -▁troops -▁motors -iors -▁gest -▁gesture -▁surprised -▁discovery -▁faint -▁restraint -▁showing -▁dull -▁statistic -▁admit -ait -illy -▁ceil -▁represented -▁engineering -▁timber -ashed -▁devoted -▁guests -inn -awn -natural -▁publish -▁male -▁monk -▁deserv -▁motiv --range -esus -▁Jesus -▁jesus -▁dallas -▁lips -▁stup -▁stupid -velt -oosevelt -▁contend -▁1956 -▁depression -▁damn -▁damage -arn -▁bridges -▁claims -mer -ccord -▁annivers -icit -▁explicit -▁compens -▁Part -▁iod -▁Robinson -▁offers -▁Thompson -▁seeing -▁clerg -▁clock -▁retreat -▁sept -▁Lew -▁Lewis -itches -▁switches -itarian -▁intervention -▁borrow -▁Alexander -▁alexander -▁wealth -▁monop -▁bench -▁hudson -▁gallery -▁cy -▁cycle -▁harmony -▁symphony -▁favorable -after -irk -▁bask -ki -▁sir -▁orient -▁secular -▁workshop -edd -▁wedd -de -uster -kin -ousin -▁cousin -inister -▁administer -▁conjunction -ugged -▁intended -▁trained -wide -▁grinn -▁referred -ation. -▁hone -▁germany -▁My -▁crimin -ips -ifies -▁spoken -▁signs -book -▁naked -odge -▁forest -esh -▁flesh -▁enact -▁guar -▁guarant -▁Pol -leton -▁gang -▁incorp -▁suggestion -▁circular -agues -▁colleagues -▁imagined -▁Perhaps -▁calls -▁eastern -▁appointment -▁delicate -▁Boston -▁juniors -▁rebell -▁Again -▁lux -▁"H -▁trag -▁tragic -▁comprom -▁pursue -▁Joe -▁January -▁january -▁lawyer -oubted -▁undoubted -▁ton -▁tape -▁Central -▁slid -▁lovely -▁Marsh -▁Bur -▁ideological -▁cos -▁cosm -asm -▁enthusiasm -▁cash -mond -▁fib -▁bureau -▁boston -▁conducted -▁scope -▁operational -ester -▁apportion -▁34 -▁mathematic -▁talked -▁hoped -▁arranged -▁enjoyed -▁explained -▁wondered -▁hurry -▁phases -▁climbed -▁invention -▁washing -▁European -▁buzz -▁satisfy -▁interests -▁presented -▁Yank -▁prescrib -▁wearing -▁desert -▁considerations -▁communications -▁gentleman -ographic -▁symbolic -▁burning -▁formed -▁adapt -▁q -▁alarm -▁reliable -▁Brook -▁recip -▁recons -▁individually -▁supervision -▁concentrated -▁500 -▁spray -ontin -col -▁treasury -▁restrictions -vail -vailability -orth -worth -wers -war -▁flew -▁outdoor -ctober -▁October -▁unpleas -▁cowboy -▁Section -▁Instead -ilton -▁musician -▁Spe -▁mortal -▁upward -▁tonight -utumn -▁stuff -▁gardens -▁radar -▁laws -▁antib -▁confused -▁hyster -▁oxid -▁subsid -pay -▁taxpay -▁assembly -▁pulmonary -▁veteran -▁Like -▁oct -▁european -▁"C -▁officials -▁Moscow -▁moscow -▁thermal -▁copy -▁invited -▁186 -▁images -▁Well -ictor -▁sle -▁slept -▁pursuit -▁relatives -▁envelop -▁suspicion -▁unfortunately -▁astronom -▁scheme -▁transformation -▁loans -▁equation -▁Prokofieff -▁Mississippi -ensity -▁density -swers -▁answers -▁pointed -occ -occup -▁preoccup -▁Sup -▁embr -▁inquir -▁exposure -▁cease -astern -▁stern -▁furthermore -▁refuge -▁weigh -lic -▁2:3 -▁sampling -▁ambass -▁ambassad -▁"c -▁"m -▁sensit -▁sensitivity -▁Constitution -▁continuously -▁india -▁indiff -▁X -▁compil -▁briefly -▁sale -▁saline -▁explo -▁nationalism -omfort -anley -▁detergent -▁shoulder -▁powder -▁assurance -▁partnership -▁confess --inch -▁practic -rica -▁precious -anches -▁branches -▁"B -▁constr -▁keeping -▁obser -▁observer -▁velocity -▁walls -▁haz -▁hazard -▁conception -▁enl -▁movies -▁protestant -▁glob -▁robinson -▁nail -▁vacu -▁vacuum -▁curiosity -▁craz -▁clever -alog -▁Pal -▁Palmer -▁massive --five --foot -▁rib -▁ride -▁hurried -▁stands -tail -▁cocktail -▁racial -▁self- -▁assumpt -▁assumptions -▁basket -eth -abeth -izabeth -eep -▁core -▁dialect -▁engineers -▁commod -▁commodities -▁plug -▁plow -▁glow -▁await -▁francisco -▁controvers -atsu -atsuo -issance -▁discr -▁discrimination -▁scheduled -truments -▁instruments -▁Str -▁luxury -bury -jury -▁voluntary -aissance -alfrey -▁moist -▁suitcase -▁suitable -▁mirac -▁mirr -isters -▁uses -▁institut -▁corporations -▁whis -▁whisper -▁lieut -uthors -▁ruth -▁mirror -▁substances -▁citizen -▁lieuten -▁admir -▁harr -ito -ymn -▁Tuesday -▁tuesday -▁whenever -▁dwell -▁cycl -▁fishing -▁ur -▁urge -rence -▁quietly -agnetic -▁forgotten -▁diseases -▁nervous -▁annoy -▁"L -auc -▁democrats -▁artific -▁prokofieff -▁impul -▁varying -▁Reg -▁refrigerator -▁discharge -▁terror -▁Green -▁Nothing -fred -▁alfred -▁Ann -▁democracy -▁archae -ised -▁granted -▁twentie -▁twentieth -▁eccent -▁eccentric -▁tent -▁resent -▁$3 -▁1953 -osis -▁opposed -▁Educ -▁submarine -▁substr -▁brick -▁oppon -▁discret -▁prol -▁Po -▁spokes -▁wander -ifying -▁loved -▁nomin -▁partner -onn -▁horizon -ored -▁compare -▁rehear -aging -▁tap -▁ple -▁Office -▁allowances -▁repeat -▁Alfred -thlet -icense -▁license -▁od -weight -▁35 -▁2, -▁collap -▁wright -▁arrival -▁mississippi -oster -▁O' -▁Fe -▁February -▁inventory -▁philosophical -andler -▁brood --1/2 -ometric -▁intim -timore -ana -aded -▁persuaded -adjust -ado -mark -astroph -▁catastroph -ounced -ternational -▁comments -▁comed -▁sketch -och -▁Congo -▁nodded -▁handling -▁orbit -▁Little -▁Public -cription -ij -▁characteristics -▁emphasize -▁draft -▁dried -▁drift -▁Tre -asy -▁thyrogl -▁Angl -▁epic -icial -▁appreh -▁Atlantic -▁refund -ider -▁Jess -▁sincere -▁correl -oons -▁effectively -▁hungry -▁glimp -gradu -▁transcend -▁classic -conomic -away -awk -▁walking -▁milligram -▁graduate -▁moves -ulpt -▁sculpt -▁Social -▁documents -▁polar -▁verse -▁prel -▁prelimin -▁preliminary -▁analyz -▁analytic -▁Accord -▁mathemat -▁mathematical -▁phenomena -▁cream -▁revision -▁looks -▁farmers -▁spark -▁assembled -allow -▁swallow -▁traveled -▁Gl -set -▁submarines -▁visitors -ady -▁gear -▁basement -▁(the -▁muscles -▁delightful -▁van -othing -▁begins -▁Massachusett -▁barely -▁Ab -▁dogs -▁fields -▁giant -▁denot -▁curtain -▁spell -25 -▁(195 -level -▁interm -▁reporter -▁contributed -▁squeez -ivery -▁delivery -▁execution -rec -▁shoulders -▁Place -▁ban -antry -▁Republican -▁bat -▁splendid -▁examin -▁tick -▁procure -▁Lincoln -▁lincoln -▁36 -▁Rayburn -▁rayburn -▁details -▁detailed -▁poison -▁eleg -cut -▁continues -▁marginal -▁classical -▁melody -▁dimin -▁sophisticated -iations -▁negotiations -▁advances -▁breast -▁displacement -▁disappoint -▁intelligent -▁spont -▁golden -▁mature -▁literal -▁alternatives -▁Labor -▁blame -▁lieutenant -▁trick -▁retirement -▁man's -▁shelters -▁neares -▁reputation -▁paris -▁37 -▁deput -▁microsc -▁optical -▁reass -▁encouraging -▁continuity -▁Gu -▁sketches -ibald -iative -itiative -▁bi -itative -▁Law -▁Lawrence -▁correspondence -▁accelerometer -▁ru -ournament -▁africa -▁boost -▁respir -▁respiratory -▁whir -▁collected -▁resign -▁resolved -achloride -▁decomp -▁adopt -▁undert -▁undertaken -▁outstanding -▁tied -▁oclock -▁prize -▁observation -▁oxidation -onym -▁Pres -▁Wright -▁Warr -▁Warren -▁Play -▁Plato -▁diver -▁Prince -▁scientist -▁consumption -▁thereby -ichmann -▁smok -▁chicken -▁employed -▁legislators -▁rabb -▁anyway -tock -▁livestock -▁deck -▁flying -▁elev -▁limitations -sued -▁issued -▁helped -▁suffered -aude -▁1, -▁meat -▁birds -▁Jr -▁Jr. -▁Day -▁contour -▁Scotty -▁scotty -chang -▁sang -▁patience -▁eighth -▁pathology -▁Ge -▁Georgia -▁promote -▁promotion -▁convenience -▁mounted -▁rolled -▁narrative -▁tentative -water -▁"w -thodont -▁debate -▁kingdom -▁jess -itual -▁ritual -▁mutual -▁viewpoint -▁clarity -active -▁pen -oly -ico -▁poetic -▁purely -▁deeply -ounce -▁mob -▁depress -atson -▁Watson -▁Earth -▁partially -▁officially -▁Can -▁helpful -▁glad -▁tribute -▁associations -▁Hard -▁trustees -▁eleph -▁Six -▁holy -ruff -▁hypothalamic -▁worn -roke -▁factory -▁violent -▁settlement -▁morphoph -osevelt -▁roosevelt -▁cultiv -▁conceiv -y, -trachloride -▁tetrachloride -▁wrink -▁corresponding -▁controlled -▁thompson -olated -▁reorganization -▁print -▁wet -▁solved -inite -▁infinite -ubble -▁hatred -▁subordin -iter -itor -▁pit -▁discount --year- --year-old -▁supers -uring -▁morality -▁thyroglob -▁amendment -arton -aylor -▁merchants -▁cooperative -▁ghost -▁firms -▁civilian -▁dawn -▁empirical -▁conspir -▁merger -▁outcome -▁overcome -raux -▁abund -esday -nesday -ednesday -essions -▁sessions -▁bless -▁sake -▁reward -▁rein -▁reinfor -path -▁shopp -piece -▁regarded -▁Carl -▁stagger -▁reader -▁technological -▁ratios -▁compell -▁Charlie -▁Neither -▁compromise -▁lake -▁lies -▁enemies -▁candidate -▁beef -▁(wh -▁desperate -▁desperately -ai -awai -stery -▁mystery -▁isolated --J -▁reactivity -▁O'B -▁arise -isp -espread -▁widespread -▁disclos -ambling -▁indifference -▁Conference -▁portrait -▁supernatural -▁particle -▁climate -▁muscular -▁plays -▁midnight -▁disk -▁gradient -erald -▁qualifications -▁probable -▁consumer -▁peer -▁pepp -▁facts -▁emphasized -▁serves -▁Woodruff -▁towns -agr -▁diagr -alski -▁dealing -▁hypothalamus -▁til -▁mantle -▁Mantle -▁lawrence -▁immense -anche -▁containing -▁seeking -▁acting -▁Mercer -vis -▁swimming -▁1955 -▁producing -▁Uncle -mma -ilemma -▁dilemma -▁dig -▁ded -▁deduct -▁dedic -▁deduction -▁ridic -▁ridiculous -▁riding -▁scenes -ependence -▁fever -▁mercer -▁defect -▁infect -▁dear -▁tangent -rap -▁fant -▁fantastic -▁simpler -▁45 -▁400 -import -▁cooling -▁minimal -▁80 -▁flour -▁factories -▁cere -▁upstair -▁planetary -▁wholes -▁ordered -▁classroom -▁pages -▁maintained -▁stall -gre -▁degre -▁enroll -▁assessment -▁prince -odger -▁Jones -▁sacred -▁pron -▁communists -▁flexibility -bridge -▁festival -▁miriam -▁stamp -gregated -▁anticipated -▁disappointed -▁anniversary -▁noise -▁stere -asters -▁batter -▁curves -▁opera -▁operated -▁interaction -▁economical -▁Stud -▁Stanley -▁Tom -▁greg -▁diplomacy -▁1920 -▁volt -▁debt -ubber -▁punishment -▁tube -pire -ards -▁arrow -owalski -▁salvation -▁champion -▁concess -emic -eman -▁worries -▁raising -▁gloom -▁implications -▁variations -▁letting -▁evolution -▁Revolution -▁obey -▁"s -▁Italian --sh -▁Nav -▁"b -▁stores -▁stored -▁saved -cape -▁classification -▁reluctant -▁responses -▁theme -▁gate -▁depos -▁Division -▁mortg -▁Wal -▁parent -▁tapp -cknowledge -▁decay -▁decis -iolet -▁originally -▁chemistry -▁phrases -▁Spanish -▁utterly -▁diet -▁enab -▁recommended -▁marble -▁prolong -▁savage -▁elementary -▁arbitrary -▁mild -▁mc -▁clin -▁clinical -▁Canad -adium -▁surely -▁nest -▁fascinating -▁occasions -▁fluores -ifting -▁aggl -▁ages -▁debut -▁About -▁crystall -▁Peter -▁Leg -▁dancing -▁dancers -▁aj -▁warrant -▁warren -▁specimen -▁deeper -▁Jackson -▁numerical -▁jr -▁jaw -▁partly -▁warmth -▁ol -▁feelings -▁swell -▁songs -▁animals -▁struggling -▁intu -▁intuition -▁proportional -▁exceptional -▁congressional -▁panels -▁jobs -▁indign -▁barriers -overty -▁poverty -▁loyalty -▁financed -▁pump -fall -▁Juniors -▁vess -▁levels -▁Te -voc -▁projection -▁cluster -▁corr -▁Sometimes -▁Res -▁suited -▁suck -▁Pap -▁Papa -about -coming -▁yarn -▁tank -aun -▁Mich -▁mich -▁mud -▁Journal -fulness -▁soften -▁stanley -▁fence -aptist -▁ampl -▁manchester -▁dread -▁agriculture -rical -rican -exican -▁sphere -▁nowhere -▁enclos -clined -clock -'clock -▁o'clock -▁wool -▁Resear -▁nu -▁nutr -▁readers -▁Harr -▁warn -▁flavor -▁guy -lete -▁incomplete -leans -▁Utop -▁spraw -▁surge -▁dissolved -▁constructed -▁supported -▁supper -▁rival -▁smashed -itious -▁ambitious -▁prej -▁prejud -▁hal -iscons -▁1000 -mont -▁pont -▁Pont -ontrol -▁lever -▁peak -agree -▁disagree -reeze -▁boats -▁Supp -yr -▁recommendations -▁productivity -▁unfortunate -▁mechanics -▁colonial -▁scrut -▁"N -▁touched -▁shoes -▁charlie -▁fibers -▁Blue -ermans -▁converted -▁convert -▁foil -▁realization -▁disappro -▁micro -▁microm -rot -▁humble -▁squad -▁Moreover, -▁movable -▁serving -ining -▁dining -inter -torney -▁Attorney -▁shakespeare -▁revealed -▁evaluation -12 -▁Air -iat -▁whereas -▁trace -▁endless -▁shell -▁sheet -cknowled -▁hes -▁hesitated -▁ham -▁managed -▁managers -▁snatch -▁dispatch -▁sink -▁sor -▁sorry -▁Institute -▁monopoly -▁revolutionary -▁improving -▁folkl -▁folklore -lers -▁builders -ging -▁fallen -igible -▁tangible -▁leaned -▁Judge -▁bund -▁exposed -▁long-range -▁purchases -▁discre -▁Actually -▁gradual -▁Seven -▁Youn -▁indisp -▁generous -olina -ologists -rost -ciously -held -▁iv -▁om -rapist -▁unexpected -▁shared -▁confession -▁snakes -▁transmission -▁pronounced -▁Oh -▁Ohio -▁Over -▁shade -▁consistent -▁climax -▁Senator -▁harsh -▁Greg -▁speaker -▁speaking -▁marketing -▁Justice -▁vit -▁ven -▁via -athode -▁stockholders -▁Suprem -▁holes -▁whip -▁arrest -▁secondary -▁insistence --day -▁denial -ouglas -▁lover -▁Hel -▁Pat -▁pet -front -▁manufacture -▁Chief -▁fountain -▁praise -▁analog -▁demands -▁clothing -▁drinking -▁bind -▁consistency -esthetic -ises -▁Assemb -▁marvel -▁committees -▁adjustments -▁stereot -▁boating -astro -▁tests -▁propaganda -▁Govern -▁republican -▁waste -▁burned -▁arres -▁arrested -▁sitter -▁cott -▁"d -▁socially -fur -▁frankfur -▁Bank -▁governments -▁volunteers -▁ure -▁ureth -▁urethane -▁coverage -▁sprink -▁staining -▁illness -▁illinois -llinois -▁infantry -▁wants -▁elections -▁sharply -▁grim -▁grabb -▁ignore -asion -▁helium -▁pottery -▁generations -▁woodruff -iring -▁tractor -▁justify -▁remarks -▁Sec -▁Second -efficient -▁psychiat -▁joe -ghman -▁discip -▁news -▁roads -▁towards -▁mutter -▁Cons -▁Consider -▁undergradu -▁procurement -ughes -▁dancer -▁dictators -▁Hill -▁Have -▁brave -▁"G -▁arous -onymous -▁seized -▁Research -▁aim -lim -imes -▁loom -▁nearest -▁frontier -▁august -▁april -▁Any -▁"D -uman -▁imper -▁imperson -▁resource -▁registration -▁traders -▁trap -▁formulas -▁prayer -▁thyroglobul -▁Anne -▁robb -▁cried -▁financing -ings -▁measuring -▁remaining -▁determining -▁minimize -▁carries -▁survived -inning -unning -▁cabin -▁ohio -▁styles -▁hypocr -▁reaching -▁moderate -▁incorpor -▁provisions -▁negroes -car -▁shear -▁1946 -▁vic -▁vicious -▁indef -▁plants -▁map -▁lap -▁pig -ecast -▁self-c -oir -▁Cur -cer -▁lying -▁mixed -▁ivory -trous -▁bulk -▁demonstrate -▁optimal -olester -olesterol -▁cholesterol -▁90 -▁fragment -▁mont -TO -ATO -▁substrate -ateur -▁amateur -▁interrupted -▁angular -▁brass -▁arrive -▁decisive -▁bone -▁torn -ixon -▁deserted -romwell -▁precaution -▁productive -▁Corps -▁nurs -sis -▁hypothesis -thesis -▁seventeenth -▁wre -▁vector -▁defensive -▁inner -▁Atlanta -▁impat -▁impatient -▁glued -▁announcement -▁Hospital -▁invariant -izzie -▁autonom -▁indefinite -▁resc -▁160 -▁colonel -obbie -apol -apolis -▁time. -▁time, -▁eating -▁jackson -▁resort -▁airport -▁rights -▁compress -ystop -▁(which -▁actions -elangel -▁evangel -elangelo -▁motel -▁motives -▁investigations -▁fleet -▁motive -onies -▁streak -▁fortunate -▁fortune -▁latin -▁composed -▁succeeded -▁taxpayers -ymnastic -chieving -▁ell -▁relevant -fire -▁tire -▁pleased -▁messages -▁bathroom -▁impressions -▁cottage --P -▁"P -▁synthesis -▁absorption -▁Bel -orse -oral -▁naval -▁electricity -intend -▁superintend -ending -▁badly -▁painful -▁doubtful -▁patterns -▁conspiracy -face -▁Def -▁Defense -▁discontin -▁accordance -▁une -▁uneasy -▁scrutiny -iana -▁louisiana -▁anti-S -▁empire -▁decisions -▁windows -▁carpet -▁dil -▁stumbled -throp -owned -▁promised -▁seal --of --term -▁Rachel -▁rachel -▁quack -▁friction -▁thunder -▁lecture -▁dar -▁refresh -▁thresh -▁"O -▁effl -▁efflu -▁effluent -agu -▁185 -▁midst -iazz -▁eliminate --like -▁Mont -neg -▁passages -▁Still -kill -▁sets -▁brothers -seud -▁pseud -▁aud -▁audiences -verse --A -▁overall -▁(S -▁eliminated -▁applause -▁perfectly -imil -▁define -▁ille -▁telephon -annon -anny -▁armed -group -▁privilege -▁superb -▁influenced -▁influential -▁Er -▁surprising -▁meadow -▁congregations -▁taxes -▁thousands -lined -▁180 -▁183 -▁vanished -▁conversion -▁Without -va -ugene -▁inhib -▁Helva -▁parlor -▁parade -▁Gar -▁wart -▁smart -ieve -▁cellar -▁encouragement -10 -60 -,000,000 -▁spending -ept -▁skept -▁mortgage -▁obscure -iph -▁proph -▁intensive -▁implied -▁regarding -▁1951 -▁replacement -▁marks -eele -vely -velyan -▁Trevelyan -▁Treas -▁Treasury -▁assessors -▁Your -gest -▁biggest -▁Bible -▁opium -▁arter -▁filter -▁participate -imensional -five -▁navy -▁predisp -iament -liament -▁parliament -eal -▁zeal -free -▁Time -▁emper -▁Thus, -▁afterward -▁bang -▁submit -ita -itter -▁tea -▁sacrific -▁interrel -ume -▁editors -▁underground -▁captured -▁capture -▁stroke -▁Elect -▁caution -▁dealers -▁thirteen -▁Carleton -▁spectra -▁blonde -develop -▁Develop -▁border -▁Sherman -▁wreck -▁Roosevelt -range -▁longrange -▁Fig -▁insure -▁recovery -▁diagram -nut -▁entries -▁jail -▁transferred -exico -▁obligations -mas -▁discern -▁forever -▁distances -oh -uddh -itchell -▁accustom -▁Linda -▁linda -▁spontaneous -ufficient -▁sober -▁Chap -▁ros -▁possession -▁blast -▁domination -▁reflected -▁omitted -rok -▁buried -▁historic -▁historians -▁whites -▁Cool -▁Castro -▁northwest -▁virtues -▁spanish -▁refer -▁unb -▁delivered -▁considering -ulative -▁Inde -▁Indeed, -▁devotion -▁gently -▁Maybe -▁servant -▁servants -▁younger -▁mete -▁meteor -▁silently -▁presentation -▁connected -▁crossed -▁Found -acing -▁mac -▁mur -urs -ursday -▁birthday -urface -▁seasons -▁disapproval -ieval -▁medieval -▁grants -▁southeast -▁stimulation -▁administrator -▁detach -▁puzzled -▁clutch -▁rescue -▁ing -▁ingred -▁ingredient -▁depends -amin -▁adolescent -▁perp -▁perpet -kansas -▁franc -umbling -ubborn -▁gentlemen -▁Steele -▁Ste -▁surviving -▁joke -clam -▁proclam -▁proto -▁Coolidge -▁essence -▁roleplay -oday -▁Today -aymond -▁juven -▁singing -▁Prof -▁Nations -▁applications -aven -aved -avis -▁rice -▁proposition -▁width -▁fact, -▁disord -▁startling -▁haun -▁consol -▁consolid -olve -▁Curt -▁colt -▁hawk -▁hate -▁improvements -▁frightening -▁Jefferson -▁deprec -▁depreciation -▁thigh -▁Direct -aine -▁suspended -▁hired -quired -▁admired -▁facing -elix -justice -▁uncertainty -▁strongly -▁reducing -▁bol -olores -▁solve -▁foolish -▁artificial -▁repetition -▁reply -▁pause -▁ambigu -iful -▁unemploy -▁devised -▁slender -▁acid -ifier -▁acquis -▁acquisition -▁measures -▁predisposition -imate -itimate -▁legitimate -▁bowl -▁ships -▁lands -▁landscape -▁bundle -▁reservoir -▁Victor -▁crystal -▁gal -▁applies -▁companion -▁associate -▁barbec -▁fert -▁healthy -▁palmer -ered -erem -▁cerem -▁ceremony -▁ceremonies -▁insisted -▁shield -▁381 --four -▁inadequate -▁province -▁(B -▁situated -▁dominated -▁stable -▁imaginary -▁imaginative -▁traveling -ework -▁casework -▁comedy -▁sanct -▁percept -▁signed -▁vegetables -▁guit -▁guitar -▁hesit -▁San -▁confined -▁tactics -▁demonstration -▁*z -▁monitor -▁ion -▁uncomfort -▁jumped -▁ribbon -▁assignments -▁dictatorship -▁meals -▁kitt -▁decomposition -▁shapes -▁honey -▁jerk -▁realism -▁perfection -itary -▁publishing -▁stuck -▁gained -▁sixth -▁chuck -▁exciting -atis -atisf -▁dissatisf -▁stronger -eration -▁exploration -▁prisoners -▁consistently -mother -▁grandmother -lemn -▁solemn -▁rack -▁defined -alled -▁socalled --called -▁receiving -▁sleeping -going -▁explod -▁Good -▁distinctive -▁accompany -▁nineteen -▁dates -▁subsection -inqu -▁delinqu -▁relinqu -quare -▁Af, -izations -▁juvenile -▁initiative -▁arts -▁wives -▁tournament -kirk -ether -▁Whether -▁aristo -▁Blanche -▁streets -▁sounds -▁listening -▁seeds -roof -▁curric -▁curricul -▁sharpe -▁Clay -▁Clark -▁structured -ikk -dikk -dikkat -▁subd -▁discharg -▁enjoyment -▁Field -▁lift -▁embass -▁embod -▁fantasy -▁constitutional -▁infectious -▁trigger -▁pianist -▁references -▁acute -▁handled -▁param -▁paramagnetic -▁seam -▁sees -ream -▁Hardy -▁meaningful -▁peaceful -ocation -▁recommendation -jold -kjold -▁fold -▁bold -▁threshold -▁collar -▁bott -▁commonplace -necess -▁vertical -▁compass -▁chill -▁Bill -▁hills -▁colorful -▁trot -itol -ousse -ousseau -▁unpleasant -itism -emitism -▁anti-Semitism -▁enable -▁laughed -▁International -▁poured -nik -▁Business -▁bible -▁calling -▁contains -▁matters -▁detectives -▁impulse -iments -▁thumb -▁columb -▁non- -airie -▁prairie -▁reproduc -▁eligible -▁wines -▁unm -resent -▁Johnson -▁antisemitism -▁grav -chief -▁destroyed -ewise -▁likewise -▁adequately -ili -▁Port -▁escort -▁Sir -▁Spr -▁Oper -▁acceptable -▁invasion -▁cod -onic -▁filling -▁subdivision -▁misund -▁formation -draw -▁sixties -▁ended -acond -▁locations -▁straw -ainted -▁acquainted -▁judicial -rog -gu -▁1949 -▁basically -▁alb -otic -iced -▁noticed -▁cleaning -▁iodine -▁1927 -▁finds -▁stubborn -ighters -▁alumin -▁aluminum -thletic -▁32 -▁nurse -▁ordinarily -▁arose -▁1948 -▁microorgan -▁settlers -acco -bacco -coat -▁alco -▁alcoh -▁alcohol -▁*zg -ami -unami -▁ceiling -▁protein -iamond -▁suggestions -▁champions -▁chore -liness -▁reck -▁lesson -▁strikes -▁remarkably -▁doors -▁too, -▁42 --century -▁recorded -▁answered -▁asserted -alysis -mes -▁holmes -▁murm -▁amus -usk -▁premier -ouches -▁painter -▁hunter -▁chin -iami -iriam -▁softly -▁unhappy -▁dances -▁chances -▁plung -▁defens -▁segregated -▁wax -▁wednesday -▁Wednesday -uj -ujill -▁relieve -▁spectacle -▁Wagner -▁happily -▁crazy -▁illeg -▁seg -▁Belg -▁Captain -▁absolutely -▁trim -▁passengers -▁slope -▁Franklin -▁spoil -▁circles -▁Norther -▁pounds -▁reverse -alse -▁false -sey -▁oppose -▁dimensional -▁awake -▁invit -▁whit -▁shopping -▁People -▁determines -▁writes -▁retr -▁suite -▁1900 -▁corporate -▁wagner -▁bonds -▁grounds -ows -▁strang -▁stripped -▁crawled -▁backward -isconsin -ienn -ienna -▁Fre -atre -▁wake -itors -itable -▁constit -▁constitu -▁deviation -▁figured -▁prosecut -▁hearst -stream -▁Communists -▁optimistic -▁mutually -▁Represent -▁Stevens -▁dedication -▁dedicated -▁breakdown -▁lawyers -▁jung -▁inquiries -▁irrelev -▁headed -▁approached -▁formerly -▁mostly -▁cert -▁decent -▁clouds -▁curriculum -attan -hattan -▁batt -▁limits -▁composer -▁fluorescence -▁segregation -inal -▁vein -▁rejected -▁boast -▁dynamic -▁couns -▁counsel -▁counted -▁Thursday -▁thursday -▁scarcely -▁Aristotle -▁depending -▁multiplic -uplic -▁rup -▁rupees -▁microscop -▁victor -▁conductor -▁expanding -▁provin -▁Rec -▁depr -▁depict -▁engage -▁cure -aham -▁mah -▁concerns -iffith -▁Contin -▁sheets -ulose -▁migr -▁asleep -▁dystop -▁condens -▁Stein -oop -▁patients -▁ranging -▁"Wh -C. -.C. -▁(C -▁leaf -▁exped -▁expedition -▁alike -jo -▁Franc -80 -ctors -▁Greenwich -▁walnut -▁rendered -▁Avenue -▁spare -▁undoubtedly -▁wink -▁betray -▁Sal -▁vocal -▁TV -▁impart -▁choreograph -▁fuel -▁surviv -▁memorial -ymph -▁1945 -zym -▁enzym -▁honeym -ymen -▁honeymoon -▁declined -umm -▁thyroglobulin -▁salesmen -▁inserted -▁exerted -wr -▁drap -post -▁russians -▁caval -▁cavalry -▁unnecess -▁U.N. -▁statut -▁tor -▁promptly -▁openly -▁similarly -▁Que -▁sne -▁misunders -▁rubber -▁stirred -▁proportions -▁religions -▁foods -▁ens -▁flag -▁precinct -▁panic -▁angeles -▁ech -▁echo -▁stained -▁straightened -rael -▁clergy -▁attraction -▁christmas -▁Christmas -akers -▁cens -▁temper -▁unpre -aska -▁resid -▁vul -▁vulner -▁homog -▁homogene -ogan -▁voltage -▁Democrats -▁bent -▁jungle -ibaldi -ithe -▁Poland -▁crush -▁flush -▁crude -▁knight -▁presidential -orial -▁territorial -▁impressed -▁gathered -▁extending -▁laughter -▁installed -▁mistakes -▁disappointment -▁arriving -▁diagonal -▁diagonalizable -▁Long -▁concluded -▁boards -▁angl -▁engines -▁zer -▁zero -▁slamm -▁reson -▁ug -ommunist -rogen -▁stereoty -▁hidden -▁disl -▁dislik -▁carl -▁restriction -▁os -▁characterized -▁Governor -▁harvest -▁Legis -▁Legisl -▁Legislature -▁geographical -vin -▁Faulkner -▁urgent -▁urged -▁testament -▁stirr -▁damp -atang -▁grat -atom -omination -▁som -▁Com -▁willing -▁shining -▁again, -▁again. -er. -▁builder -▁contents --in -▁accent -cap -oli -▁tool -▁abol -▁streetcar -▁hundreds -yll -▁sull -▁bored -▁unions -▁convictions -▁exceptions -▁creature -▁venture -▁Samuel -▁samuel -▁implement -▁impr -▁minds -▁"a -▁influences -▁kohnstamm -▁Kohnstamm -ecut -▁persuade -build -▁symm -▁self-d -▁amounts -▁discouraged -▁Hot -▁Hotel -▁Henrietta -▁henrietta -▁embrace -▁apprent -▁apprentice -▁apprais -▁circumstance -emperature -▁exempt -▁startled -▁envelope -▁textiles -uther -oos -▁choos -BA -▁heavier -▁Cousin -▁Sever -▁Several -▁mans -▁ends -▁minority -▁diagnosis -▁yankees -▁fist -▁physically -▁receives -▁adap -▁atoms -▁Walter -▁Elizabeth -▁lamp -▁protected -▁"h -▁performed -▁encountered -▁resentment -▁fears -▁surprisingly -reece -▁cree -▁creep -▁bottles -▁discrimin -▁aggressive -▁authentic -▁lunche -▁luncheon -ildren -▁helping -lene -▁enforcement -anston -▁disgu -▁disarm -▁commonly -life -▁wildlife -▁Life -▁world. -▁renaissance -▁demographic -amey -▁franklin -anning -▁cafeter -▁walter -▁horizont -ontroll -▁era -▁curt -▁squeeze -urk -▁1. -oo -oan -▁shame -▁saint -▁auditor -▁spectr -uble -▁noble -▁anacond -▁hay -▁lb -▁lone -▁statue -three -▁Gross -▁ballistic -▁Pike -▁pike -▁pear -▁bubb -▁corrid -▁atlantic -unition -munition -▁ammunition -▁winston -▁strategy -▁stadium -▁stevens -▁astronomy -▁deserve -▁sponsored -ateful -▁grateful -▁controversy -▁foundation -▁sole -▁preceding -▁resulting -▁compete -▁competent -▁achievements -▁provincial -▁contradiction -▁contradict -▁Mark -▁plates -▁life. -state -▁chronic -▁behalf -▁summary -▁impatience -▁soci -▁Del -four -▁mask -▁masters -▁thru -▁lobby -▁haul -▁vert -▁vertex -▁Young -oultry -▁dy -▁dying -▁matching -▁knowing -▁anticipation -▁Square -▁drank -▁Mot -▁Motors -▁Mama -▁mama -▁indians -▁positions -▁poets -▁Who -▁blanche -▁bringing -▁glimpse -nessee -con -▁mainly -▁wildly -▁roughly -▁concerto -▁degrees -arewell -▁prospective -▁seemingly -icism -▁strategic -▁molecular -▁lemon -▁optimism -▁diagnost -▁washed -▁sanctuary -▁receiver -▁slide -▁Chandler -▁gambling -▁induc -▁helva -nic -▁ethnic -▁Harv -▁Harvard -▁annually -▁unre -uk -uter -▁computer -▁compr -▁compat -▁compatible -▁morphophon -▁hatch -▁runs -▁feels -▁massachusetts -▁confronted -▁fram -▁concurr -▁succession -ouri -▁Missouri -▁Four -▁notion -▁explosion -▁stereotype -▁Grand -▁Delaware -▁Carolina -▁Mill -with -alaur -alaurie -▁criminal -astically -▁mountains -▁functions -▁garage -proof -▁committed -▁rector -▁veterans -▁insol -▁insoluble -▁memories -▁adverse -▁femal -▁disposal -▁infr -roadway -▁crossroad -▁humorous -▁jefferson -ondition -▁Palfrey -▁prest -▁prestig -▁prestige -▁shipping -▁Miriam -▁holster -▁infection -▁terribly -▁necessitated -▁synthetic -▁A. -▁congregational -▁Inter -▁peter -▁shiver -▁thrill -▁manifold -▁intimate -▁Hol -▁Holmes -▁Hall -▁implies -▁criter -▁loop -▁bishop -▁fitted -▁$5 -▁recipient -▁prais -▁transom -abama -ading -▁singular -▁experts -▁fee -jured -▁papa -▁locking -▁installations -▁foster -▁insult -▁insert -▁cafeteria -▁resolve -▁tickets -▁titles -▁telling -▁teenag -▁consent -▁indispens -▁subsystem -▁System -▁harvard -▁functional -▁skilled -▁cited -▁cyn -▁caut -tons -▁tons -▁Industrial -▁evidently -▁arguments -▁(F -▁"F -▁Ruth -etical -oretical -▁metaphysical -aunt -limited -▁Step -▁relating -▁scholastic -▁package -▁filed -▁solitary -▁hostile -▁Next -▁educated -aturated -▁investigated -▁investigators -remlin -▁lonely -▁vigorously -▁analyst -writ -▁brightness -▁clamp -▁Independence -▁existed -▁exists -▁existential -▁brings -▁gets -▁(A) --wide -cellulose -▁petitioner -▁sunlight -▁deficit -▁wherever -aeus -▁separately -▁paintings -ways -▁payroll -▁Work -▁interrupt -aney -▁Much -▁elizabeth -▁anatom -▁wip -▁cuba -▁thanks -▁holds -▁Als -▁Also -▁vo -▁voters -▁pilgr -▁disciplined -▁invitation -adden -▁tappet -▁jet -▁closet -▁biography -▁Scott -▁appearances -conscious -▁competence -▁italian -▁strok -▁antibody -▁Africa -▁discrep -▁jews -▁skills -▁daytime -▁candle -▁Occ -▁greenwich -▁sandwich -na -a. -lyn -▁Brooklyn -▁brooklyn -▁seventh -▁expenditure -ureauc -▁departure -▁adventure -▁$4 -▁184 -▁Certain -certain -rim -▁Add -▁them, -ation, -western -owulf -▁cruelty -▁solely -▁rent -inent -tinent -▁pertinent -abri -▁ruled -▁punct -▁persistent -▁carpent -▁capita -▁null -▁gathering -▁unlikely -▁covering -ete -▁micromete -▁85 -▁800 -▁600 -▁serum -▁thre -▁dean -auce -▁sauce -▁benefits -▁Super -▁customer -erto -uerto -▁barbecue -▁expectations -▁O'Ban -▁O'Banion -▁ambassador -▁indication -▁politician -▁Garibaldi -white -▁novelist -▁wrapped -▁Columb -▁instructions -▁long-term -▁verd -▁verdict -▁indict -▁indul -▁undertake -▁Lake -▁seated -▁blunt -▁physician -▁players -▁fees -▁feas -▁feasible -▁boundaries -▁carleton -▁creating -ing. -▁exert -agged --and --and- -▁poland -▁operand -▁operates -▁co- -▁co-op -shire -bl -▁preferred -▁attended -lines -▁implicit -▁qualify -▁clarify -▁tracing -▁selling -▁priority -▁Michelangelo -▁michelangelo -▁biological -▁lots -antly -▁merchant -▁awful -▁provinces -▁demanded -omery -gomery -▁views -iciency -▁receipt -▁spokesmen -gers -tenna -▁agglut -▁endurance -▁gymnastic -▁representing -▁sermon -▁reactions -▁Rock -▁believing -▁belg -▁released -▁overnight -itrogen -▁chew -▁flames -▁orthodont -▁ench -▁enchant -▁Dartmouth -▁dartmouth -▁dot -▁periods -orter -▁centrif -▁ideology -▁reapp -▁revel -tection -▁detection -otte -▁cha -▁chaos -▁glory -▁advisory -▁dug -▁dash -▁disappeared -▁useless -▁scenery -▁opponents -▁opponent -▁65 -▁knocked -▁boss -▁bast -▁ben -▁bush -▁bark -ierce -▁exploit -▁Clayton -▁crises --to- -▁politicians -iac -acent -▁adjacent -acqu -iating -ulating -resco -▁sharing -▁vag -▁Hamm -ean -plan -expl -▁semi -▁neatly -ani -aniel -▁revival -▁revised -▁Morse -▁grades -ourke -▁Louisiana -▁fort -▁bombers -▁bombs -▁listed -aughters -▁polite -▁unlock -▁paused -▁utopian -inth -▁combin -▁combinations -ublin -▁Asia -▁irish -▁stairs -ureaucr -▁Poet -▁gaz -▁planets -▁motif -▁newly -▁secants -▁parame -▁paras -vas -iliar -▁pilgrim -▁voted -▁perfume -▁Something -▁wings -▁J. -▁superintendent -▁rebels -▁rebuild -▁heroic -itially -▁Profess -▁Program -▁sending -▁voting -▁Almost -▁Albert -▁ethics -abasc -thabasc -▁psychology -▁psychoan -iazza -▁phosphor -▁evid -▁evidenced -ifth -▁placing -▁consisting -▁visiting -▁unload -imilar -▁assimil -▁postwar -ometers -▁lovel -▁1910 -▁1912 -▁spontaneously -▁marshall -▁concession -▁enlist -▁forecast -▁delaware --Saxon -hes -▁cohes -▁votes -▁Turn -▁sprang -▁Flor -▁watercol -▁year. -▁continuation -▁systematic -▁chapters -ampshire -▁bills -▁resonance -▁stake -▁sturd -▁mortar -▁Tele -▁attempts -▁crest -▁apprehension -▁dialys -acle -▁abuse -▁abilities -▁tip -▁precip -rad -▁dividend -step -▁sheep -▁unaware -▁etc. -▁abundance -▁diamond -▁duplic -▁dos -vers -▁squat -▁squares -▁applicable -▁painfully -▁cement -▁ceas -▁vibr -▁solo -▁accumulation -▁circulation -▁alienation -▁Economic -▁privacy -▁transformed -▁nerves -▁strengthe -▁strengthening -▁uncon -▁ignored -▁ignorance -ibl -iblical -▁models -oz -cred -credible -▁authors -▁goals -▁rises -▁melod -▁days. -▁brut -▁brutality -▁nursery -▁misery -▁statute -▁orleans -▁Orleans -▁operators -▁informal -▁humanity -osity -▁mos -▁furnishing -▁furnace -▁cheer -▁brisk -▁Marshall -▁tested -▁Middle -▁elder -▁parker -▁fewer -▁propose -▁discretion -▁carri -▁clark -cknowledged -▁worthy -ersey -▁intermed -▁intermediate -pons -▁irrespons -hr -▁coalition -▁Des -▁Desp -▁Despite -▁Dick -▁dick -▁wick -inity -incent -▁sinister -grade -▁onset -▁fatal -▁lion -▁fogg --g -▁serge -▁inventories -eddy -onsciously -▁consciously -▁cort -▁distort -▁cord -▁murmured -▁vern -ogn -nold -▁libert -▁Gorton -▁"R -▁0. -▁radically -▁pm -▁advised -olph -aired -▁mythological -▁eighteen -▁obst -▁presently -▁tends -▁races -▁lucy -▁commend -▁bend -▁reaches -ordination -▁coordination -▁eggs -▁fortunately -▁antique -tructured -▁unstructured -▁Coop -▁mart -▁mcc -▁compulsivity -▁historically -▁allowing -▁enlarg -ymphony -▁Harold -apeut -icnic -▁picnic -▁shrink -ongh -aton -▁locked -▁stretched --B -▁commander -ering -erge -▁emerge -▁processing -eriff -▁sheriff -▁skiff -▁surrounding -itzer -acter -▁bacter -▁Parker -▁plank -▁turk -▁awk -▁awkward -▁ring -▁rip -ising -▁typew -▁typically -▁tyr -▁tyranny -▁Later -▁compartment -▁validity -amily -▁sandburg -▁passes -▁hoping -▁shouting -▁shouted -▁prox -istically -▁belly -▁belov -▁definitely -▁instantly -▁oppress -▁albumin -▁publicity -press -▁unpreced -▁romance -▁meets -▁mothers -▁Vol -▁quantitative -▁avenue -▁approve -▁printing -▁civ -▁civic -▁wedding -▁Illinois -▁limp -▁ling -▁lingu -▁linguistic -▁lid -▁congru -▁According -▁illuminated -▁stol -▁stove -▁stolen -▁combust -iberal -▁Bay -▁array -▁reappear -▁stein -▁vessel -onald -▁disguised -▁finest -▁manufacturer -▁manufacturing -▁featur -▁donald -▁selective -▁voy -▁voyage -▁Roy -avana -▁plate -▁1947 -▁shorter -▁sociology -▁shrugged -▁accused -athology -▁polished -▁compact -▁vocab -▁acquiring --hour -▁missouri -▁battens -▁amp -▁ample -▁patents -▁dang -roit -▁detroit -itr -ris -sunami -▁tsunami -▁shifted -▁reinforce -▁Jr., -▁187 -▁maladjust -▁mons -▁soap -▁ingen -ossom -▁blossom -ers. -ollywood -lywood -▁hollywood -▁Complet -▁combine -▁gotten -19 -▁monot -▁sank -▁associates -▁graduates -▁opens -arah -▁inspir -▁counting -▁adopting -▁claimed -▁ninth -▁wreath -▁antibodies -▁bitterness -▁adapted -▁explosive -▁words, -ez -yard -▁beard -▁ard -hand -▁favorably -▁optimum -▁courts -▁blocks -▁paroch -▁parochial -▁honors -▁scored -▁utopia -▁justified -▁thermometer -▁revolver -apeutic -▁dressed -▁Commer -▁Commerce -▁rabbit -▁coolidge -▁beatnik -▁bunch -▁bid -▁robber -sburgh -ittsburgh -▁affected --W -ecutive -▁atlanta -▁reserved -▁ic -▁technic -▁simplic -▁simplicity -thony -▁resulted -▁controlling -▁alongside -▁precipitated -▁harry -▁Harry -arryow -▁tribun -umulative -▁kansas -▁Kansas -▁Tech -abash -▁involution -▁bounced --run -ujillo -2) -▁(2) -▁retrieved -herited -exper -▁Are -rell -▁conformity -▁unnecessary -▁violation -▁constructive -▁respective -▁respectable -▁antitr -▁antitrust -▁marry -▁liver -▁bronchiol -▁vessels -▁baltimore -▁adjusted -▁Mayor -▁scand -▁merchand -▁Last -sec -▁reflex -▁refers -▁typewr -▁breeze -fl -hl -▁holder -eros -▁Los -ositive -▁Similar -▁cul -▁culm -▁harold -▁ears -habit -▁Edd -▁Jul -▁rect -▁odor -▁cylindrical -▁cylinder -▁"T -▁dair -▁dairy -▁bureaucr -▁autumn -▁marshal -▁unrel -jamin -▁blues -▁daughters -ilization -▁veil -▁bubbles -▁albert -▁Palace -▁attempted -illed -▁corridor -orient -▁drag -▁lag -▁lagoon -ultmann -habitants -▁Ke -▁pipe -▁discrepancy -▁copper -▁copies -▁cope -▁copern -▁stabil -ulously -▁applying -▁conn -▁recording -▁Coast -▁creatures -▁rugged -vocado -▁fraud -▁applaud -▁disposition -▁cosmic -thabascan -▁nude -▁firing -▁analyze -▁monopol -▁exclaim -rapeutic -▁introduce -heim -arse -ara -ourse -▁linear -▁progressive -▁Five -arryowen -▁arteries -▁$7 -aviolet -▁ultraviolet -▁lav -▁accumulated -▁reconstruction -▁chandler -iot -iants -▁giants -week -▁unemployment -▁daylight -▁moonlight -▁denominations -olecul -▁spokesman -▁retention -▁verbal -zold -turn -▁Look -▁1930 -▁1940 -▁250 -▁abolition -sky -insky -▁stems -▁so-called -▁statutory -▁blockade -▁Mus -▁stew -▁Matsuo -▁matsuo -▁merg -▁swing -▁neighboring -▁Negroes -▁Lucy -▁aver -▁averaged -▁urgency -ini -▁persuasion -▁semester -ars -arskjold -▁Hammarskjold -▁hammarskjold -▁summar -▁compensation -▁drugs -▁bronze -▁turns -igan -▁Michigan -▁michigan -▁aesthetic -▁interl -▁bankrupt -▁chiefly -▁lively -lessly -▁shortly -▁lightly -atisfact -▁trait -orious -▁glorious -ector -▁Rector -▁Ramey -▁coc -▁sizable -▁makers -▁earthqu -bre -ibb -▁curb -▁Ob -▁Oct -▁Shar -▁Sharpe -▁coupled -oulev -▁Barn --n -▁Method -89 -▁conceal -▁conceive -▁occupy -▁occupation -▁baker -▁150 -▁emperor -ained -oked -▁longterm -▁carved -▁congruence -▁attracted -▁rooms -▁aristotle -▁Democrat -▁mat -▁mason -▁tooth -othy -▁indispensable -ilitary -▁rebellion -▁combustion -▁Fig. -▁slee -ierre -▁autonomy -ayette -afayette -▁specialized -▁mode -▁farewell -▁fresco -▁fond -▁fury -▁vagina -▁promising -▁Swift -▁educator -▁carriers -▁casually -▁Barco -▁raz -▁razor -▁raid -abriel -▁Rel -▁buying --all -▁over-all -▁cumulative -▁Pacific -▁pacific -▁battery -iding -▁closing -▁hanging -▁creativity -▁engender -▁returning -▁ramey -▁gasp -▁erup -loy -▁facility -▁Above -▁mull -▁(and -eland -▁republicans -▁chores -▁flame -greement -▁Agreement -avement -▁recognizes -lavery --H -▁accounts -▁Table -▁oy -▁distributed -▁SBA -▁sons -nabel -▁cockp -▁cockpit -polog -ologic -pic -▁conspic -▁judges -▁knot -▁knuck -▁duck -arians -▁ensemble -▁shipments -▁castro -▁Navy -▁NATO -▁Hand -▁Otherwise -enz -arried -▁contention -▁fitting -▁instrumental -OD -▁beneficial -▁imprison -▁converse -unc -▁transpar -▁publicly -▁hollow -▁waist -▁tum -▁torr -▁touches -▁controversial -▁coarse -▁melodies -▁fertility -▁kne -▁electoral -▁disrupt -▁hated -unn -oqu -▁eloqu -▁enlight -▁well- -▁indirectly -▁ruined -▁lined -▁Relig -▁subjective -athet -▁excellence -atrical -▁pope -▁pressures -ihood -▁aspir -▁regularly -roach -reach -bach -▁Pre -▁Post -▁Pathology -▁polyether -▁Lat -▁Latin -▁congrat -▁jan -▁participating -assau -auge -ays -▁sturdy -▁Jean -▁Foreign -isan -▁partisan -ennis -▁quantities -▁loves -▁comprise -▁losing -▁preferably -▁eighty -▁mickey -▁Mickey -▁majest -▁dislike -▁African -▁african -▁bacterial -▁bapt -▁bod -▁invitations -▁consisted -▁mg -▁sway -▁soup -▁whiskey -▁whisky -▁independently -▁slack -▁motions -▁acceleration -▁Capitol -▁capitol -▁tab -▁tie -▁turb -▁tensions -▁tale -▁cart -▁correlation -atern -▁jean -▁Volume -bidden -bird -stro -kstro -kstroh -kstrohm -▁corruption -▁psychologist -▁fatig -▁fatigue -▁approaches -▁describes -▁enabling -▁enabled -▁wrinkled -▁flung -▁clung -▁prone -▁prow -▁princess -▁Report -▁Lady -▁sherman -▁registered -▁omin -▁predomin -▁auditorium -▁premium -iscover -apor -▁vapor -▁Bob -▁directors --hand -▁spectators -▁upstairs -▁hen -iffen -ymp -ympathetic -▁intact -▁Turk -▁monstrous -astrous -▁disastrous -▁fed -▁morphophonemic -▁framework -▁spectrum -▁rushed -▁bob -▁strand -▁illustration -▁illustrate -▁Communism -win -▁Within -▁afflict -▁Italy -▁slaughter -▁uncomm -▁standpoint -▁ladd -▁lane -▁belgians -▁Russians -▁jul -uls -ulsion -ulations -▁Supreme -▁tales -▁exercises -▁bridget -▁coordinated -▁emanc -▁emancip -ancing -ipus -edipus -▁slip -▁grip -▁missed -▁zg -rd -▁herd -▁vitality -▁impersonal -▁meal -▁forthcoming -▁proceedings -▁halfway -▁driveway -▁affix -▁affil -clusion -▁strongest -seas -▁overseas -▁micrometeor -▁stack -father -▁grandfather -▁railway -theast -aste -▁emerged -▁Obv -▁Obviously -▁furiously -▁Spenc -▁faded -▁reconstruct -▁reconsider -▁1040 -through -rought -▁conced -▁Red -ared -▁infrared -▁pressed -▁irrelevant -axwell -▁treasur -▁sociological -▁tense -▁calibr -ibrium -▁pion -▁pione -question -illion -▁King -▁33 -ileen -▁phonologic -▁pension -igation --M -▁Laur -▁nitrogen -▁spotted -▁vermont -▁"to -rylic -▁acrylic -▁sector -▁equitable -▁entail -questionably -▁declaration -▁exchang -▁postp -pert -▁turnp -uffled -▁diaph -▁tack -▁penal -▁frost -itical -▁necessities -▁bolt -▁forg -▁forgiveness -▁reun -▁reunion -▁grab -▁scores -ptic -▁popp --degrees-C -▁carolina -▁Cop -▁Copern -▁bearing -▁morse -▁sunset -▁Onset -▁wires -▁pioneer -▁lights -▁findings -▁monkey -▁Walk -▁countless -▁clover -▁nigger -▁championship -▁rum -▁Suppose -▁Eileen -▁spots -▁imagery -▁enterprises -▁46 -▁56 -umid -umidi -▁bru -▁5000 -▁cork -▁Ask -▁realt -▁realtors -▁dealt -tw -▁survivors -▁commerc -▁commercially -▁secant -▁landscap -▁disreg -▁disregard -▁anch -▁anchor -che -▁dialysis -quis -merican --American -american -amiliar -▁unfamiliar -▁unfold -▁unfair -▁slipp -▁microorganism -▁santa -▁Santa -▁womens -acc -dale -▁illum -▁lumin -▁shaft -wind -▁accordingly -▁insufficient -▁shovel -▁cardinal -▁cards -▁pituit -asso -icasso -▁costum -nam -noons -▁afternoons -▁untouch -▁Development -▁tremble -▁indicating -▁turnpik -▁stride -▁doorway -▁kil -▁Patch -▁Patchen -▁turnpike -▁tourist -istence -▁dome -▁parked -▁parking -▁Shayne -▁Cal -loyd -boh -▁waved -▁Fred -▁ridge -gn -▁pregn -▁Up -ffair -▁popularity -▁pupp -▁peril -▁Harbor -▁preparing -▁preparations -▁messeng -▁plastics -▁tilghman -▁Montgomery -▁montgomery -▁crimes -▁thereafter -▁statistics -▁viv -▁domain -▁grinned -▁solicit -▁1944 -▁Twent -▁separation -▁certainty -▁injury -▁whist --free -awaii -▁sunny -▁Rem -▁allocation -11 -▁descent --third -yth -ytholog -▁demytholog -▁pyth -▁python -▁readiness -▁monthly -▁vaguely -▁player -▁coincidence -▁coincide -▁trials -▁delegates -▁parasympathetic -▁laur -▁exhibited -▁studying -▁beam -▁obliged -▁sentimental -▁ultracent -▁females -▁subsequently -restr -▁proprietorship -▁hans -▁fences -▁occurrences -▁occurring -▁Maris -▁colored -▁capability -▁trading -▁Jane -▁sweater -▁Brand -log -▁crawl -▁eliminating -▁wider -▁fans -▁intr -▁intrig -▁lectures -uously -▁tremendously -▁enormously -▁weep -▁champag -▁champagne -▁revers -antern -▁lantern -▁oban -olate -iterate -▁Him -▁raises -▁tubes -▁anglos -▁anglosaxon -▁finite -▁eileen -▁elb -▁chapel -▁maintaining -▁vulnerable -▁texture -▁Education -▁psychologists -▁disposed -▁unlimited -▁rug -▁canvas -▁madison -urrow -▁aristocr -▁(T -▁listened -▁listeners -▁attached -▁gifted -▁gap -▁relationships -utter -ts -▁"E -▁gay -▁gle -▁gleam -ayabash -▁celebrated -▁remaind -▁remainder -▁participated -▁Telegraph -▁refine -▁poised -▁recurr -ang-J -juries -▁keel -▁generator -▁grind -▁susan -anned -tow -▁bestow -ware -▁dare -▁nerve -▁irrad -▁perceived -▁nap -▁Hollywood -ched -chem -chemical -cher -▁Special -▁stimulus -▁tappets -▁denomination -▁plumb -entagon -▁tasks -▁northeast -▁Mon -▁Steph -▁inhibit -phere -▁commuter -ossip -▁gossip -▁tender -▁tended -▁bars -▁intervals -ieties -▁absorbed -roduction -▁Fair -35 -▁embodiment -▁bore -▁distaste -▁preventive -▁Lee -▁spectator -▁welcom -▁freew -▁adventures -▁commitments -▁commitment -▁delayed -▁sensible -▁leads -▁guns -▁gorton -▁metaphysics -rosis -▁proceeded -odore -▁cooking -▁misunderst -▁misunderstand -▁kids -▁houston -▁bangj -pot -▁forgot -▁defeated -▁protective -▁derive -▁specialists -▁regretted -▁middle-class -▁scott -▁scour -hour -▁rejection -▁refrain -▁Holy --dr -rick -▁centrifug -▁coron -▁media -▁epis -▁episode -▁privately --ey -▁ruling -▁ok -▁Nixon -▁nixon -▁trop -▁tropical -▁squeezed -▁obanion -▁argues -▁blink -▁fan -▁middleclass -▁continental -▁foundations -▁explanations -▁fastened -▁Occasion -▁companions -▁conquest -ennial -▁solar -arious -ilborn -ilghman -▁Tilghman -ilibrium -▁equilibrium -▁dramatically -▁gentile -amel -amela -ersted -▁oersted -▁Oersted --mile -▁statist -▁statistical -iston -ieze -▁frieze -▁elected -▁promises -▁penalty -▁binding -▁customary -▁arises -▁gram -▁Bal -▁Baltimore -▁Eddie -▁childrens -▁bangjens -ang-Jens -▁periph -▁epiph -▁ambival -interest -▁travell -osed -imposed -▁hypothetical -▁ticket -▁formally -▁luminous -▁purport -▁purple -▁appet -▁charming -▁ingenious -▁yach -▁typewriter -▁expressions -▁Sept -▁crept -▁watercolor -lav -oslav -ugoslav -aug -aque -▁dialogue -▁alliance -▁faster -master -▁Would -▁Win -▁Winston -▁Women -erm -▁symbols -▁attic -▁eventual -▁quoted -▁comr -▁comrades -▁organism -▁tilted -▁gramm -human -▁solutions -lands -▁Portland -▁territories -▁envision -▁hardship -▁exercised -▁peg -▁flor -▁Century -▁shelf -▁privileged -▁Newport -▁reminded -▁Year -▁underwater -▁*y -▁unst -▁myst -▁(B) -alta -▁roberts -▁thoughts -▁electronics -▁congest -▁Pope -▁Howard -osp -iven -▁abandoned -▁dissatisfaction -▁Ac -athy -▁fabrics -▁hardy -▁frighten -▁deliberate -▁Tim -▁unav -▁unavoid -▁unavailable -▁relent -▁ruin -▁Forest -▁lob -▁lodge -burse -burst -▁diverse -▁insane -▁jane -▁Davis -▁davis -▁fires -▁fate -erosol -▁archaeology -lessness -▁continent -▁contraction -▁inclined -▁incredible -▁ov -▁inherent -▁nato -umont -▁diagnostic -▁recalled -▁lawn -▁reassurance -▁obstacle -▁Completion -sters -▁youngsters -▁spirits -▁occurs -▁proposals -▁spur -ecd -ecdot -▁alec -▁overlook -▁prevented -▁purchased -▁crowded -▁justification -▁offering -▁laughing -▁Austr -inster -minster -▁Bureau -▁sax -▁roast -▁suspicious -▁charco -▁vocabul -▁Ol -▁Fl -▁Follow -▁gratitude -tily -▁hastily -▁unsuccess -▁suggests -▁sprawled -▁decrease -▁whereby -▁gown -▁germans -umbrell -▁smoothe -▁staged -▁stature -comes -▁electrost -▁developments -pton -important -▁smiling -▁devast -▁Hans -▁sounded -▁carryover -▁Manhattan -▁tar -eegan -atanga -▁Orchestra -▁segment -▁bases -▁tumor -aker, -▁rope -▁Beyond -▁considers -▁pile -▁professors -▁cohesive -▁deficiency -▁describing -▁ratt -▁inval -▁invalid -▁elevator -▁illumination -▁"n -▁microph -▁nuclei -▁eyebr -▁Elaine -▁credited -▁resembl -▁resemblance -▁Organiz -▁translate -▁159 -▁collision -▁basketball -▁woods -lets -▁prostitution -uddy -▁shudd -flow -▁Hous -▁Houston -▁vow -▁collage -▁imply -▁Early -▁irony -▁advocate -▁undertaking -▁reactionary -▁pituitary -▁vocabulary -▁ambush -▁draf -▁negotiate -▁faulkner -▁me. -▁deposit -▁epithe -▁tools -▁cereal -▁governmental -alph -▁anchored -▁infancy -ifiable -▁columns -iev -row -▁elbow -▁pin -▁availability -▁drastically -▁cm -▁cough -ms -▁miracle -▁disarmament -▁successor -▁corners -inting -▁ignorant -vant -advant -▁disadvant -▁lev -▁lex -odium -itzerland -▁Maryland -▁polish -rish -▁Irish -▁hut -▁hook -▁codes -▁hart -wart -▁door. -esides -▁Besides -▁classified -▁conqu -tolem -▁oedipus -▁Oedipus -▁oak -akable -▁destined -▁destination -▁tones -urity -augur -▁suppress -▁slim -▁quad -ada -▁canada -▁cad -▁hand, -▁footstep -▁Should -▁cellulose -▁Therefore, -▁palfrey -▁scorn -▁admirable -▁twentyfive -▁twenty-five -▁expressing -▁abdom -▁20th -▁Anglo -▁Anglo-Saxon -▁apartments -▁fusion -tonishing -▁specimens -arc -arst -▁Hearst -OC -▁accelerated -▁originated -▁cavity -▁tunn -▁explore -▁superiority -▁prosperity -▁ninety -yan -▁mercy -▁Mason -▁permanently -▁"l -▁autonomic -▁entering -▁alve -▁uncomfortable -▁modernization -▁swoll -▁swollen -▁tears -▁burg -▁nighttime -▁conjugates -▁Vern -▁hymen -▁eloquent -▁costumes -▁elite -▁elimination -▁poultry -ogg -▁Fogg -▁adher -▁violin -▁amaze -▁breaking -▁johnson -▁johnny -▁day. -▁downward -▁Dec -▁elegant -▁shorts -▁imposed -▁drying -▁testing -▁staring -▁stars -▁falling -▁lists -▁Keith -▁dent -▁full-time -▁Viet -▁viet -▁cheese -▁commute -▁thyrox -▁porter -▁porous -▁179 -ena -itten -chestr -▁resumed -▁indications -▁Eugene -▁eugene -▁Interior -▁deterior -▁kidney -▁dece -▁perce -▁perceive -▁resume -▁resumption -▁ladder -▁ultrason -▁lend -▁loun -ql -iql -▁sisters -▁jersey -▁Jersey -▁abruptly -▁repeatedly -▁intent -▁drifting -▁missionary -▁slump -▁shaking -▁principally -amor -▁translation -▁orientation -▁translated -mosph -mospheric -▁boom -▁conceivably -▁malraux -▁suic -▁suicide -▁Harris -emorial -▁exploited -▁days, -▁tort -▁tortured -▁efficacy -▁Temp -▁teasp -▁raced -▁align -quet -▁Kitt -▁57 -87 -▁aspirations -▁Unt -▁Until -▁evaluate -▁involvement -▁"public -▁umbrell -▁uncover -lord -▁landlord -▁pupils --tw -▁Hence -see -▁(see -▁dispers -▁chip -coats -rock -roc -▁reciproc -▁scandal -▁Maj -▁Mac -▁unmist -▁incorporated -ypt -▁egypt -▁(with -▁dazz -▁speculative -▁specify -▁accommodate -▁visitor -▁surrounded -▁establishing -▁everyday -▁Mull -▁clue -▁horrible -▁horror -▁McK -▁bobby -▁couch -▁cautious -atell -▁satell -▁Mad -▁Madison -▁composite -▁spaces -itiated -▁sentences -▁meaningless -"; -▁E. -▁encroach -▁Im -▁contributions -▁Jews -▁savings -▁Sand -▁Sandburg -▁hind -▁hint -▁hail -boro -▁enlarge -umblebe -▁chromatic -matic -▁electrostatic -▁kilometer -▁Co. -▁deem -▁realities -▁Christianity -▁christianity -▁males -ayana -oft -oulevard -▁steele -▁swoop -ometry -▁electrons -▁W. -▁spinning -▁posit -▁positiv -▁posse -▁ego -▁interd -▁"g -▁childish -▁Stadium -▁inflict -▁visc -▁Force -▁que -▁carriage -▁gum -▁gulf -▁appreciated -▁appropriated -▁identities -▁mines -▁ministry -remier -▁telegram -▁woke -▁input -▁1942 -▁1943 -▁1941 -▁varieties -▁undue -ste -stein -e, -▁organize -rode -▁Symphony -▁handed -▁disturbed -▁halted -▁then, -▁lacking -▁settling -▁ripp -▁villages -▁averages -▁Gold -▁choices -▁allies -▁anonymous -▁specially -▁Harvey -▁grief -▁comfortably -▁hopp -▁freez -▁catalog -▁irradiation -▁"Y -▁mahogan -▁Frankfur -▁unadjust -▁modified -▁durable -▁films -▁pir -▁Burn -oris -▁transferor -▁proclamation -▁wishes -tes -▁grotes -▁grotesque -▁withdraw -▁hiding -▁swear -▁imb -parable -▁limb -▁fighters -▁imperative -▁imperial -▁warned -▁misfortun -▁forgive -▁slop -▁esp -▁dump -▁businessmen -▁Felix -▁felix -▁census -▁39 -▁morris -▁linguist -▁leis -▁leisure -▁planes -▁unhe -▁Corpor -tally -▁Jim -▁anaconda -▁retard -▁Brumidi -▁Brannon -▁leaped -▁intelligible -▁overhead -▁Barton -anim -▁unanim -▁namely -▁tightly -▁insulation -▁goods -▁rating -▁logic -▁maryland -usions -agging -▁Donald -▁prohibition -▁plaint -▁acquaint -▁viscosity -▁masterpiece -▁$6 -▁$8 -▁technicians -▁rubbed -'dikkat -s' -▁manip -▁Puerto -▁puerto -▁galax -▁recognizable -▁hamilton -▁lightning -▁swore -▁elastic -▁antiqu -▁missing -▁feeding -negative -onnaissance -▁(D -▁pickup -ifties -cellaneous -▁72 -▁sera -▁Republicans -umumb -▁unsatisfact -▁averaging -▁162 -irl -▁transit -▁transaction -▁wag -▁wagons -▁Warwick -▁hitting -▁distilled -▁bastard -TV -▁mileage -▁plainly -▁guarante -▁happening -▁spacious -▁barefoot -idel -▁launder -▁snapped -▁parttime -▁fulltime -▁Colon -▁concerts -ythe -▁neon -▁prejudice -vention -▁plentiful -▁ominous -▁prospects -▁cautiously -▁habitat -▁overt -law -▁satisfactor -▁satisfactorily -▁lump -rons -▁patrons -▁Nick -▁penny -▁deputy -retched -▁part-time -▁creek -wild -▁Ham -▁Hamilton -▁wrist -▁girlish --out --type -▁inhibition -▁nutrition -▁Hart -oliage -▁happier -▁Finally, -▁Broadway -▁interchange -▁Revenue -▁florid -▁differential -▁10000 -▁scout -▁academy -▁shortage -▁pas -▁mas -▁mascul -▁mol -▁assuring -▁1917 -▁accred -▁policemen -▁cigarettes -▁marines -▁preoccupied -organic -▁awaken -▁dependence -▁unprecedent -▁architecture -▁conclude -▁Pip -▁returns -▁naz -▁Naz -▁addresses -▁trevelyan -▁spat -▁spun -▁magnet -▁guided -▁stretches -▁dominance -▁cance --haired -▁Milton -▁2. -▁loudly -▁agreements -etery -emetery -▁cemetery -emph -▁Bang-Jens -▁Connect -▁weekly -iour -▁adjo -▁adjoining -▁ugly -▁composers -▁dramat -▁obstruct -▁aggreg -▁dreadful -▁youthful -▁aroused -iqlal -▁appetite -orus -▁chorus -▁Krim -▁krim -▁altered -elp -▁governing -▁melting -ovsky -▁Bord -▁curs -▁structures -▁remembers -isphere -▁disagreement -▁indictment -▁reflects -▁diminished -▁Yes, -▁unimp -▁overwhelming -lean -▁Mean -▁Meanwhile -▁worthwhile -▁eichmann -▁Eichmann -▁subsistence -existence -▁manpower -▁phony -▁adoption -▁indec -▁prede -▁predecess -▁scree -▁See -▁keen -▁keith -▁sul -▁phonemic -▁indicator -▁adhes -▁41 -▁tennessee -▁prediction -▁plaintiff -▁Affair -▁banquet -▁frankfurter -▁Johnny -▁perpetu --19 -▁inquiry -▁elegance -▁convuls -); -▁shrew -▁causing -▁BOD -▁americas -▁conceivable -▁sergeant -▁disab -oyce -▁Joyce -gar -▁S. -▁Simm -▁negl -▁negligible -▁theater -▁parish -▁loos -bb -▁cush -▁cushion -▁chromat -▁chromatography -▁Is -▁Israel -▁specialist -▁comparatively -▁magg -▁magical -▁missionaries -▁conscient -▁conscientious --mind -▁conspicuously -▁consecutive -asteur -asteurization -opasteurization -▁dystopias -▁nond -▁strat -▁democrat -▁formid -▁formidable -▁epid -▁epidemic -▁intercept -▁suscept -▁heads -ependent -▁armies -▁Forces -▁cambridge -▁sel -▁selkirk --bl -▁archaeological -▁Cuba -evelop -▁locate -▁ghet -▁ghetto -▁guerrillas -▁guerrilla -▁$10 -▁Student -▁transparent -▁natives -▁psychiatrist -▁Unfortun -▁checked -▁shaped -▁retained -▁pertain -▁unmistakable -▁dropping -▁feud -▁displeas -atism -▁conservatism -▁harvey -▁totally -ca -▁pans -▁shattered -neapolis -▁anne -▁extrud -▁Premier -▁bleed -▁bleak -▁slum -ifiers -▁gods --school -▁Nob -▁Nobody -▁cubic -itic -▁blindness -▁magnification -▁communicate -▁anticipate -ollar -▁Podger -▁arkansas -▁Arkansas -▁Artic -▁banks -sons -▁orthodox -▁induce -▁rental -ospel -laim -▁fathers -▁scholars -its -itage -▁heritage -▁na -▁naive -▁signals -▁Ben -▁formulation -olation -olas -icolas -evie -brew -▁harris -oat -aters -▁allowance -▁publications -▁emotionally -▁therefore, -▁markets -▁covers -▁collaps -▁assets -▁illiterate -▁airplane -ulton -unct -▁veto -▁gut -▁clayton -▁intuitive -garet -▁incap -▁incapable -▁Taylor -▁Column -▁doubled -▁mare -▁drastic -▁fox -▁severely -▁sectors -▁familiarity -erger -▁riot -▁socialism -elli -▁drainage -▁barton --the -▁pinch -▁hysterical -bel -▁picket -▁"J -▁well-known -▁microscope -▁energies -▁impartial -▁nonetheless -▁candy -▁silk -▁meantime -▁reconnaissance -▁dusty -▁Fromm -▁hem -▁hemisphere -▁restaurants -▁womans -▁Northern -yllables -▁D.C. -▁preservation -▁preserved -▁compelled -▁Foundation -▁representation -▁peasants -▁pressing -▁Rousseau -▁shaken -▁rehears -52 -▁tennis -▁Arnold -▁arnold -▁killpath -illing -▁unwilling -▁tribes -▁ballad -▁simultaneous -▁bust -▁hil -▁hood -▁hymn -etime -▁lifetime -▁frown -▁policeman -▁relevance -▁Outside -▁hughes -▁Hughes -▁obligation -▁gin -inary -▁supervis -▁culmin -▁litigation -▁signature -▁Min -integr -▁disintegr -▁seats -itating -▁correspondent -▁Domin -cles -▁spenc -nected -▁beh -▁behave -▁behaved -▁silh -▁silhou -▁Cook -▁Book -▁Bet -▁Between -▁B'dikkat -▁fierce -▁adams -▁denominational -▁paradoxically -byter -byterian -ges, -▁dough -▁permits -odgers -▁divide -▁Wall -▁Buddh -▁buddh -▁bud -▁tower -▁sewer -▁Mod -▁wholesome -▁recital -▁spong -▁waterfront -▁tells -▁acts -▁actor -▁actors -▁concepts -looking -▁exportimport -ois -▁escaped -▁Throughout -▁Stand -▁edd -▁eddie -▁torqu -icut -▁Connecticut -▁connecticut -▁hoag -▁shayne -▁penetrating -▁dangers -▁katanga -▁designated -▁evaluated -▁radiopasteurization -▁launched -▁elaine -36 -▁succ -ora -endora -▁re- -clusive -▁blew -▁bleach -▁71 -▁perhaps, -▁indulge -▁Professor -▁mayor -▁Mor -orig -▁aborig -orem -▁'em -avinsky -▁Congreg -prof -bably -▁Probably -viol -▁geometric -▁Geometric -quire -▁kinds -▁elephant -▁broadway -▁quaint -▁Gene -▁Geneva -▁femin -undry -▁peered -▁teenagers -▁ramp -▁articulate -▁pleading -▁ontolog -queous -▁antagonism -▁rebel -▁recognizing -archy -▁floating -▁sung -▁pitcher -▁Speaker -▁requiring -▁vegetable -▁oxford -▁locating -▁locally -▁shaw -ity, -▁permitting -▁bones -phem -asphem -▁blasphem -asma -▁plasma -▁plump -▁44 -▁speck -▁beautifully -▁districts -▁scramb -▁scrambled -▁southwest -▁steak -pace -▁sailing -▁graduated -▁lucky -▁bears -restrial -▁jim -▁Does -seven -▁skinny -▁refuse -▁traced -▁tri -▁triang -▁rectang -▁paramount -aress -▁caress -▁Press --I -▁Chapter -▁slug -▁ekstrohm -▁Ekstrohm -▁Ev -erick -▁Frederick -erable -▁miserable -▁idi -issues -▁tissues -▁tile -▁tv -▁tumbled -▁generators -▁Meek -▁journalism -▁Vienna -▁vienna -▁Vincent -▁vincent -▁vine -▁amuse -▁nucleus -▁producers -▁blush -▁Third -▁quarre -▁utilize -▁zoo -▁iodide -▁shrewd -▁meek -▁brannon -▁Russell -▁patronage -▁Family -▁deposited -▁hire -▁admire -▁boundary -▁approaching -▁year, -▁chamb -▁complained -▁advise -▁executives -▁recount -▁Johnnie -▁johnnie -▁Ernie -▁angles -▁employers -violent -▁aimed -▁telephoned -▁fled -▁knitted -▁eminent -▁emerald -▁mand -▁cave -▁decency -▁connot -iencies -▁dealer -▁prodig -▁span -▁peripheral -▁vinegar -▁populated -▁bail -▁kneel -▁supporting -▁1/2 -▁irrational -uted -▁idiom -▁unpaid -imesis -▁Times -▁maris -▁segments -▁criticized -▁critics -▁trimm -▁technically -▁grabbed -ased -▁deceased -las -▁earned -▁leagues -▁personalities -▁nickn -▁nickname -▁jar -▁seventy -▁rigor -▁slightest -▁remed -▁trunk -▁fractions -▁milligrams -▁Pittsburgh -▁pittsburgh -▁evolved -▁Kate -▁broader -▁doz -▁dozens -▁vaginal -▁prosec -▁prosecution -ellow -▁sue -▁spher -▁Representatives -▁110 -▁sunsh -▁addition, -▁flock -▁snap -chloride -▁Bang-Jensen -economic -▁serene -▁mock -armonic -▁unsuccessful -▁instance, -▁cerebr -▁Everything -nings -clination -▁doubtless -▁bon -▁bubble -ubblefield -▁airfield -▁bass -pass -▁postpon -▁unsatisfactory -▁Appar -▁Apparently -▁violently -▁cooperate -▁glaze -▁Ju -▁glasses -▁orchestr -▁zen -▁sullen -▁subscrib -haired -▁tourists -ayabashi -iability -iary -▁ardent -▁gauge -▁Vernon -odynam -▁glamor -▁reckless -incinn -▁genev -▁reign -▁Sm -▁Small -▁tom -▁tomb -m. -.m. -▁p.m. -▁realm -▁pine -eline -▁piazza -▁hostess -▁eve -world -ennings -▁kennings -▁blade -▁greeted -▁ranged -▁podger -▁(P -cy -crecy -▁penetration -▁underdevelop -▁perpetual -ache -▁utilization -▁alph -▁searching -▁Anniston -▁dreams -▁coexistence -ies, -▁pansies -▁profitable -▁arising -▁steph -creage -▁cypress -cribed -▁renewed -▁tsh -entuck -potent -ilpotent -▁nilpotent -▁Haney -▁fred -▁frederick -▁brushed -▁shelley -▁miscellaneous -▁disturbance -▁undergoing -▁racing -▁amazing -lying -▁underlying -▁multiplying -▁Sus -▁Susan -ologically -ologne -▁Start -▁Bobbie -▁Val -▁Valley -▁gig -▁gigantic -▁frantic -▁burial -▁sleeve -▁assass -▁financially -igen -▁antigen -▁Gen -▁splash -ego -▁sud -▁taxi -▁ski -rosh -roshim -iroshim -▁Unfortunately -▁disob -okes -▁dazzling -▁currency -▁230 -▁blot -▁47 -nd -▁amid -etroit -▁Detroit -onghorn -▁Anthony -▁Tit -▁Title -▁(in -▁(M -M. -▁criterion -▁criteria -▁nursing -outherner -▁osc -▁350 -▁Kitti -▁attributed -▁falls -▁intel -▁intellect -elect -print -eu -▁maneu -▁maneuver -eem -▁staircase -▁idol -▁idle -▁15th -▁Fund -▁shifting -▁antenna -▁anecdot -elihood -▁likelihood -▁likes -▁interpretations -▁wail -credibly -▁observers -▁bluff -▁palazz -▁pamp -▁pamph -▁pending -▁torment -OCS -▁Upton -▁subspace -▁comprehension -gricult -break -▁plotted -▁ecl -▁trips -▁transmitted -▁vitamin -▁purchasing -icked -▁cher -▁cherished -ager -▁chatter -▁shatter -aments -amic -amination -SH -ouglass -▁bastards -▁Religion -▁paying -trust -▁anti-trust -romb -rombie -▁bulb -▁suits -▁father's -▁screens -▁thyroxine -▁Utopia -essim -▁pessim -▁islands -16 -▁182 -▁Short -▁adjourn -aggin -▁improves -▁Zen -▁awoke -▁sleeves -▁incest -▁incom -ansom -▁Hom -▁Homeric -ahom -lahom -▁Mah -▁profits -▁devote -▁muffled -▁A.M. -▁tore -▁foliage -▁governors -two --F -▁alaska -▁distortion --shot -▁Ox -▁Oxford -oux -▁producer -gans -▁organs -▁54 -▁horm -▁hormone -▁bulky -enny -▁Philip -▁Father -▁Ell -▁summarized -▁qualification -▁Miami -▁Key -▁alternate -quar --dimensional -▁pursued -▁Use -▁referring -▁celebration -▁rook -▁Westminster -▁westminster -▁Natur -▁Simp -▁stoop -pok -poken -▁Ten -▁Tennessee -▁discomfort -ka -▁marij -▁marital -▁simplified -▁hideous -imeter -▁butter -▁meg -▁me, -gypt -▁Egypt -▁beg -▁begged -org -orgsky -▁georg -▁Director -▁geometry -▁Wisconsin -▁anatomical --made -green -▁subsidiary -▁fuzz -▁muzz -▁fame -▁ma -uing -▁forbidden -ienville -▁waving -▁wisconsin -▁dental -avors -▁Jud -weitzer -▁mariju -▁marijuana -▁filing -▁poign -▁unanimously -▁fragile -atie -▁ago, -▁Phys -▁colleague -▁48 -vol -▁posture -▁Griffith -▁amazement -▁Nevertheless, -▁joyce -▁solidar -▁solidarity -management -▁stockade -▁Giffen -▁airplan -▁cogn -▁Addition -llor -▁chancellor -▁assemble -▁resemble -▁Security -▁accelerometers -lec -▁Alec -▁yankee -▁assuming -theart -▁portions -▁mercen -▁mercenaries -▁mercury -▁compulsive -▁wounded -▁drifted -▁homeric -▁homer -▁precarious -▁acclaim -▁treaty -▁Auto -▁alveol -▁Const -▁ecst -▁tobacco -▁unquestionably -▁stroll -▁Carroll -▁static -▁stating -▁stal -▁stare -yp -priced -▁virus -▁Had -▁purposes. -▁sodium -warf -▁64 -▁memorable -▁peters -▁string -▁strings -▁applicants -▁embro -▁decre -▁sarah -▁Sarah -pirit -▁transpir -▁summit -▁summers -▁day, -▁wired -▁cancer -▁Guard -▁wolf -▁himself. -▁rejo -▁packed -cestry -▁fring -▁fringe -▁simplest -▁batting -▁(or -rov -▁replacing -▁succumb -▁russell -elled -▁(E -▁annoyance -▁bobbie -▁bdikkat -▁clergyman -▁salesman -▁lamb --conscious -▁tune -▁swiftly -▁Royal -▁Dolores -▁dolores -▁Daily -▁hudd -▁outright -▁now, -▁cooperatives -▁expanded -▁Ver -▁Vermont -▁assail -▁shots -▁planners -▁conson -▁consonant -▁shu -▁shake -▁thaw -▁indicators -▁sensual -▁Bern -▁Bird -▁bern -▁Modern -▁mustard -▁Johnston -▁johnston -▁liberalism -▁boge -▁faithful -▁opt -▁Freddy -harmonic -▁analytical -▁thirst -▁qualitative -▁legislator -etric -▁symmetric -▁integral -▁integrated -▁starts -▁nonsense -▁southerners -vet -▁wallpap -emost -anford -▁cake -▁earthquake -▁Furthermore, -▁rousseau -▁martyr -▁accr -▁accruing -▁accuse -cop -roscop -▁said: -▁sloan -▁sincer -alytic -▁Kowalski -▁kowalski -▁Calif -▁gaze -BM -ally, -eller -efeller -44 -ibrary -▁graham -▁condemned -▁griffith -▁bidd -anita -▁pathological -▁bargaining -berry -▁pharm -▁shades -▁strewn -▁victories -▁accreditation -▁proprietor -▁extrapol -view -▁Dominican -▁Eastern -▁cliff -▁fulfillment -▁breathed -▁socio -▁rolling -run -▁Gun -▁Germans -▁haney -▁conclusive -▁contacts -▁States, -wight -▁thereof -▁Cam -▁Cambridge -▁orange -▁charcoal -▁work. -▁away. -▁erected -▁carefree -▁slammed -oriented -▁rustling -▁pavement -▁schoolhouse -▁remedy -▁currents -▁loaded -▁aborigines -termination -▁lee -▁accompanying -▁stirring --six -▁meteoritic -▁stresses -▁gran -▁undergraduate -▁hillside -▁systematically -▁Article -chemistry -▁congressman -▁investors -rei -▁1935 -▁1938 -▁portland -▁superst -▁superstition -▁formulations -▁symmetry -effect -illpath -▁Killpath -▁Tomm -▁void -▁wrath -▁Kore -▁abandonment -▁prevents -▁ballot -ned -▁remarked -▁Science -▁Son -▁Common -▁Command -▁Land -▁tuck -▁refusal -▁Belgians -▁elephants -▁steering -▁muzzle -▁attendant -▁attending -▁cruising -▁shoe -▁Angeles -▁miami -▁say, -▁Beach -▁journalist -▁Colonel -▁utilized -▁utilizing -▁venge -▁vengeance -15 -14 -uart -▁guarantee -▁anxiously -▁realizing -▁nutrient -▁kate -▁Product -▁purified -▁crouched -▁out-of -▁traver -▁interdependence -▁interdependent -▁possessed -rep -▁Test -▁Testament -▁sponsors -▁Dep -▁Dan -▁Deegan -▁engulf -▁roleplaying -▁Baker -▁reverend -▁blend -▁Amb -stal -▁nostal -▁nostalgia -▁curled -▁curse -▁kore --2 -brush -▁Congregational -▁federation -icens -▁licens -▁pamphlets -▁bullsey -▁tablesp -▁nominal --gr --T -▁punch -▁Rather -dick -osdick -osh -▁Ord -▁Orient -▁transd -▁transduc -▁formulated -olored -▁taylor -▁consultant -▁snapping -▁disadvantages -▁Ralph -▁ralph -▁Big -▁(f -▁puzzle -▁puzzling -▁Belgian -▁aggression -▁pastor -▁bulletin -▁2000 -▁manage -stage -adine -▁saloon -▁newport -▁76 -▁mills -▁norms -roth -▁deprived -erent -▁indifferent -neath --story -▁constituents -▁undergraduates -▁purse -robe -▁wardrobe -fect -▁divorced -immy -▁honored -▁Everyone -▁welch -▁Continental -loan -▁Sloan -▁bleeding -▁begging -▁discernible -▁calf -▁Temple -▁adjunct -▁pepper -▁Was -▁graceful -▁Fortun -▁Fortunately -▁Es -▁apportionment -▁infant -itate -▁carrier -▁perplex -▁anniston -▁fertil -▁declarative -▁ernie -▁Sudd -▁Sudden -▁curiously -▁Dickens -▁residents -▁pole -avin -▁Gavin -▁turtle -▁Douglas -▁douglas -▁beloved -▁speakers -▁Queen -▁Washington, -▁Bre -▁Bee -▁exaggerated -▁freddy -▁diaphr -▁diaphrag -▁employes -▁dentist -▁acquies -acies -▁Wel -▁Welch -channel -usually -▁unusually -▁generosity -▁youngster -ierarchy -eld -▁administered -▁Med -▁Medical -▁cynical -ammed -urred -itched -▁dickens -▁warwick -▁stones -▁discont -▁rancher -▁junk -▁noticeable -▁antislavery -▁frustrated -EA -▁Alaska -gean -egean -▁keelson -▁volta -▁volley -▁bodybuild -rophy -▁Which -▁successive -olitical -affirm -▁reaffirm -▁consumm -▁Beowulf -▁encounters -addy -▁enr -▁enrich -▁enh -yd -▁tires -▁fearful -▁Julia -▁julia -sequently -▁Consequently -▁ripe -▁Byron -▁manned -▁frenz -▁monastic -▁buffal -▁buffalo -icus -▁copernicus -▁Copernicus -▁flourish -renaeus -▁preparatory -▁guarantees -lahoma -▁oklahoma -▁morale -▁Brid -▁Bridget -adget -▁gadget -▁toll -▁molecul -▁kern -▁earnings -▁Often -▁dilution -enheim -iless -▁700 -▁abolish -▁Laboratory -▁barrier -▁italy -▁ugliness -ymphoman -▁admiss -▁admissible -▁irresponsible --G -▁earthquakes -▁burns -▁temperatures -▁sticks -▁deplor -▁muddy -elike -▁frances -▁caliber -▁irresist -▁irresistible -▁boredom -▁teaspoon -▁stressed -port-I -▁Export-I -▁Winchester -ymphomaniac -▁encomp -▁marvelous -▁inquired -ermud -urious -▁luxurious -▁disbel -▁fer -▁ferment -▁fidel -▁practition -▁exterior -quisite -berr -▁hydr -▁investigate -▁prohibited -▁Superior -▁slash -▁Well, -▁notes -▁vectors -▁shov -nov -oves -▁successes -▁Members -crimin -▁owner -▁reinforced -▁Yankee -▁stimulated -▁1963 -▁nomination -▁mate -▁Whate -▁Whatever -▁interp -▁Rico -▁rico -▁refrigeration -▁politically -▁mexican -▁Mexican -▁mitchell -▁Mitchell -▁Music -▁reveals -▁deepest -atz -▁Week -▁comprised -▁cats -▁grok -▁groc -▁musk -▁descriptions -▁rocket -▁sterile -▁noted -mates -made -▁Gard -eb -▁dual -arine -arl --war -ologians -▁illustrations -▁couples -▁Engine -ivial -▁trivial -▁fratern -making -uscript -acon -▁screech -▁diversity -ivities -▁bould -▁bumblebe -▁tunnel -▁Billy -▁billy -▁ironic -icans -exicans -▁majestic -▁preference -herence -▁adherence -eeley -▁mamm -erget -▁energet -▁energetic -▁deemed -▁radioactive -▁stability -▁pork -▁pose -▁Alabama -▁alabama -encing -▁gracious -close -▁disclose -urray -▁Also, -▁Richmond -ommitt -▁dun -▁dorm -▁lun -▁lab -▁lurch -▁roots -▁governed -▁Viol -rosine -▁purposes, -▁congen -▁harmonies -▁dynast -▁diminishing -▁Sub -▁Philipp -▁suppression -yroid -ithyroid -▁Florid -▁entities -▁channels -▁insects -▁immigrants -▁linguists -▁negotiated -▁simplify -▁vault -▁interstate -unity -▁immunity -▁testified -▁clevel -▁vases -▁statewide -enched -▁lens -▁tailored -▁capitalism -▁blown -▁unchang -▁farmer -▁meetings -second -▁Author -cloth -▁misc -▁downstair -▁Food -▁grains -li -lies -length -▁wavelength -▁rabbi -▁forthright -▁upright -▁blades -▁giffen -▁carb -▁chord -▁Ford -▁disturbing -▁dismay -▁angu -▁lunar -elkirk -▁garibaldi -▁69 -▁willingness -▁Southerners -▁night. -aited -▁awaited -▁computed -▁accustomed -▁sizes -▁Rad -▁Radio -erate -▁worker -▁passenger -▁isolation -▁Water -▁Indians -▁regression -▁derives -▁fictional --by -irby -▁ambiguity -▁cranston -▁Cranston -▁keys -▁digital -▁dishes -▁Greville -▁litter -▁emergencies -▁appropriations -▁Baptist -▁picturesque -▁loses -herit -▁TSH -▁Town -▁queens -▁folded -▁weird -▁accomplishment -▁terrain -▁formulaic -▁cocon -▁coconut --con -▁painters -▁communion -▁lowest -▁thorn -▁plunged -▁guaranteed -▁formulate -ehead -▁forehead -e. -▁sometime -iane -bean -▁deputies -angford -▁resin -▁wonderfully -▁wondering -▁breathing -▁trends -▁frankfurters --off -▁subt -▁ft -▁fab -▁outset -▁Exper -▁Sk -irling -▁toler -▁secretaries -▁terminal -▁terminate -▁(N -glass -▁Ambass -cern -▁unconcern -▁dimly -▁singled -▁lust -ustin -▁stump -▁epithets -▁hack -attack -▁moisture -.e. -▁i.e. -▁vacant -▁portable -jun -▁amusing -▁fibrosis -▁Glendora -▁hardened -▁hardware -▁bachel -▁significantly -▁intang -▁upset -▁rend -▁rendez -▁Selkirk -▁hasten -▁Fall -▁coats -▁interfere -ancies -▁varies -▁babies -▁icy -▁authoritative -▁dan -▁garb -ba -▁sba -▁pulling -▁cathode -▁iodin -▁skirm -▁pasture -perture -▁grocery -▁pegboard -▁Orlean -▁dive -▁struggled -▁shotgun -▁quit -▁Color -▁Colorado -▁substituted -▁tray -▁spatial -▁swinging -▁dioc -odges -▁scanned -▁sprout -▁Mart -▁supplementary -▁palat -▁palatability -▁allotted -▁concentrate -▁concentrations -▁approximate -▁provoc -▁provocative -▁Ballet -▁skull -▁hull -▁enjo -▁discontent -▁oily -▁restless -▁instinctively -▁noon -▁confrontation -▁Via -▁satellite -▁astonishing -▁Gulf -▁performers -▁performing -▁advisers -ouncing -▁ulcer -▁appall -▁merciless -▁voluntarily -▁stopping -▁nagging -▁niece -▁fashionable -▁haunted -isons -▁subjected -incinnat -umni -▁attacked -pped -▁neurosis -▁chickens -▁generated -▁adults -▁bliss -▁permiss -▁prominently -▁biw -▁neglected -▁swee -▁sweetheart -▁poster -▁posterior -▁discussing -▁crossing -▁tenure -▁wiped -▁dragged -▁dragging -▁economically -ignific -▁cripp -▁philipp -▁crus -▁wheels -▁impuls -▁malaise -agrin -▁Soon -▁proportionately -▁clause -▁Upon -▁wheat -▁flutter -▁athletic -▁unconsciously -▁rage -cott -▁crisp -▁advertise -▁submerg -formed -▁responded -▁respondents -▁defended -olesworth -▁breed -▁breathe -istory -▁revulsion -region -intellectual -▁philip -▁appreciably -▁uniformity -▁rockets -▁physi -▁mimesis -▁encompass -▁Notte -▁exclam -▁perceptions -▁perceptual -▁mathematics -▁sentenced -▁summon -upon -▁allied -▁emptied -▁collapse -▁henri -annah -ahms -opped -▁ministers -▁Its -▁mechanisms -▁embodied -▁feeble -▁thermodynam -▁stray -▁helion -▁grop -▁ragged -▁quint -▁ambiguities --th -▁valued -▁linked -▁cheerful -▁manuscript -▁381( -▁chauff -ancholy -▁melancholy -▁attempting -▁revenge -ael -▁Michael -umumba -:30 -▁pok -▁concord -▁treason -▁trees, -▁gorge -▁blowing -▁lease -▁psychoanalytic -▁repay -aggini -▁superf -▁superficial -▁francis -▁greasy -▁i.e., -▁affects -▁Edythe -▁skimm -▁Hawk -▁Ho -▁Hoag -▁exceedingly -▁overwhelmingly -▁fragr -▁fragments -▁Armed -thlete -▁clubs -▁Quine -▁Quiney -▁dip -▁Rever -▁Reverend -▁deadlock -▁saloons -▁dominican -▁punished -▁yok -▁eyebrows -▁twisted -vest -▁exemption -▁interruption -▁Helion -▁trump -▁bump -▁thump -centive -mount -entucky -▁disabled -▁conversations -endix -▁tolerance -eredith -ithm -ithmetic -▁Meredith -ibles -▁mound -▁Hanover) -▁janitors -▁beck -tricate -▁hesitate -▁cultivated -▁favorites -▁Beck -▁sock -neck -oelect -▁86 -▁room, -▁Yankees -▁singers -▁apple -▁vain -▁Graham -▁interc -▁intercourse -▁Nadine -inute -▁symbolize -▁peel -alraux -▁Malraux -lycer -▁glycer -cherty -ouv -▁happened. -jack -▁facil -▁generously -▁Suddenly -cet -▁Spring -stall -lema -tolema -tolemaic -iciencies -▁deficiencies -▁handic -▁handicapped -▁suspension -ographical -zer -▁flee -eavor -▁endeavor -▁ending -▁wing -▁chant -▁indignant -▁compost -▁injustice -38 -▁1913 -▁avoided -oline -▁gasoline -▁swarm -▁prolonged -▁wretched -▁Garth -▁rotation -▁rotating -▁competitors -▁realistically -▁Point -▁powell -bell -addell -embr -▁pigment -aires -▁concessionaires -▁patchen -▁agglutin -▁Vill -▁Village -illery -▁artillery -▁Very -▁it, -o, -▁Then, -▁Lo -▁case, -▁blankets -▁regulus -▁contemporaries -▁Benjamin -▁benjamin -▁presbyterian -▁Presbyterian -▁bangjensen -▁woven -▁purity --help -▁self-help -▁nation's -ouk -▁roared -owell -elling -ella -▁passionate -▁entre -▁entrepr -▁entreprene -▁persistence -▁elbows -etary -▁monetary -egaton -▁dated -▁(R -▁usefulness -▁adding -▁Police -▁equivalents -▁dignified -▁termination -▁7th -aef -aefer -▁somers -page -▁Memorial -▁Mel -▁Mexico -▁preceded -▁Chamber -▁individualism -valued -▁pillow -rice -▁Mother -▁Mobile -▁seas -▁$500 -▁allev -▁busily -odes -▁alphab -▁teen- --E -▁Prop -▁michael -▁Form -▁brace -▁carto -▁pelt -▁distinguishing -▁simms -▁C. -iento -▁risen -▁throbb -▁spirited -▁killer -▁subordinates -▁correctly -▁engr -▁witnesses -▁transfers -▁respondent -▁gates -▁gn -▁gul -▁strode -erior -▁inferior -▁warrior -▁cuts -lot -▁oddlot -▁depot -▁guys -▁oath -▁senators -▁plantation -▁Player -▁exub -▁exuber -▁Ambassad -▁cunning -▁occupational -▁occupants -▁glist -▁55 -odging -▁Camp -▁slower -▁bullets -▁7070 -▁hierarchy -▁Ro -▁Road -▁mens -▁58 -verb -▁proverb -▁pointing -▁conting -▁opposing -▁discontinued -▁masses -▁attributes -▁unequ -▁protr -▁battles -▁congressmen -itan -▁denounced -▁stren -▁strenuous -▁Shu -▁trucks -unciation -▁Eight -▁dumb -▁precedent -▁unprepar -▁Raymond -▁raymond -▁rebut -ewish -▁frail -▁creates -▁creator -▁meter -ola -tenance -▁countenance -▁flatten -capable -▁Sav -▁Saxon -▁saxon -▁sed -▁suspected -▁zinc -▁zone -▁convertible -▁smelled -▁piled -▁clich -▁calculation -▁reflecting -▁westward -▁ellips -▁Bobby -▁lengths -▁handful -▁Burma -▁God's -▁Truman -▁adaptation --middle -▁Presidential -anna -▁witt -▁incons -▁incon -▁recon -▁slashed -▁hoof -undown -▁ware -▁decorative -giving -▁curved -▁ili -uxili -took -▁metropolis -▁ultracentr -▁ultracentrif -▁winds -▁engaging -gone -▁undergone -▁commencing -▁pronounce -▁Home -enson -▁"th -▁"that -▁spiral -▁russias -▁glitter -▁terrace -▁Mission -▁Mo -▁Moore -▁moore -▁wellknown -▁crystals -▁resembles -▁resembled -▁exempl -▁disinterest -acklin -▁Back -placed -▁misplaced -▁swim -▁guides -▁shifts -▁milton -▁Francis -▁edythe -▁Myra -amble -dam -▁goddam -ica -▁Madden -shot -▁mobility -▁Morr -▁Morris -▁Mil -alia -▁Thirty -▁buffer -▁municipalities -▁jeep -▁coupling -▁Far -▁alveolar -degreesc -cus -cathode -▁photocathode -▁photochemical -▁closest -▁vest -▁dock -▁postpone -roduc -▁Letch -▁bellow -▁reproduction -ilet -ttle -▁competing -▁1924 -ouvanna -▁liberals -▁cents -▁belongs -▁nights -▁etc.. -▁sixteenth -▁scal -▁prayers -▁calc -▁calcium -▁calif -▁induction -▁odd-l -▁odd-lot -▁automot -▁automotive -olitan -olving -▁prescribed -ocolate -▁chocolate -ofore -etofore -▁balloon -▁soloist -▁Having -▁neph -▁nephew -▁man, -▁manhattan -▁swamp -eel -▁Wheel -uxiliary -▁philosopher -▁prosperous -▁Progress -▁alternately -▁procedural -▁chairmen -▁orderly -▁patriotic -▁connois -▁home. -▁Farm -agh -▁pint -▁horizontal -aghty -▁allotments -▁ontological -atical -▁grammatical -beat -▁tomm -▁tomato -▁militia -▁grows -▁homogeneous -▁gripped -▁grill -▁arranging -▁hopeful -32 -▁Andrei -▁authoritarian -▁Trujillo -▁questioned -▁Seld -▁loath -▁pac -exp -expensive -▁vacations -▁favored -▁paradig -▁doubted -▁Center -▁harder -lay -▁differs -▁burma -▁ruins -▁sincerity -EAE -▁shallow -ubert -▁Assembly -▁merchandising -▁fancy -▁fighter -▁plato -▁platoon -▁Light -▁kitti -incinnati -orthy -ivanna -▁schnabel -▁Schnabel -uation -▁conservation -▁immature -▁unimportant -▁evoked -▁deserves -iscuit -bound -ieth -udie -rox -▁baptist -▁enforc -▁ensure -▁dice -▁residue -▁distrust -▁distract -▁acquaintance -▁Take -ugment -▁advancement -▁preoccupation -▁apolog -▁meltzer -▁unbrok -interrupted -▁Shore -▁erotic -ided -▁coincided -.M. -▁devise -▁constitutes -elier -▁prevailing -actical -▁addressed -▁consulted -▁thirties -▁intrins -▁brighter -▁lighter -▁securities -▁inch -▁50% -▁interlock -▁Emper -▁contributing -iscovery -tolemy -▁gland -▁hunch -▁Greece -▁Parliament -▁contours -▁ped -▁plywood -▁micrometeorite -▁sorrow -athedr -▁cathedr -▁contrace -▁discrepancies -▁Imp -▁conspicuous -ptuous -▁Lizzie -▁diaphragm -▁conclusively -▁handicap -▁injured -▁sword -▁satellites -arges -▁discharges -▁privileges -▁cleveland -ifle -▁Serge -▁galley -iances -▁appliances -▁vernon -▁landing -▁99 -▁49 -▁villain -▁identifies -▁invite -▁rendezv -▁graz -▁fortress -▁engross -▁Massachusetts -▁paced -can -▁Near -anda -▁amusement -iodot -▁salaries -▁salami -▁tackle -▁warriors -▁counteract -▁architectural -▁declining -affold -▁scaffold -▁slot -▁Op -pelier -▁fingerprint -▁rescued -▁subdued -▁irregularities -▁blaz -▁cove -▁utmost -rmost -udsons -▁hudsons -▁protrud -▁heroes -▁sites -oric -renic -▁olive -▁gait -▁multiplicity -rome -egan -▁deegan -▁deaf -▁forego -▁nineteenthcentury -▁theatre -▁borders -▁borden -▁Buck -▁Berger -ocket -▁monuments -▁Constant -▁enduring -▁lest -arted -▁1861 -▁bewild -▁Now, --state -▁desolate -▁fascination -▁pinpoint -▁persuas -▁accessories -▁1/ -▁duly -▁bankruptcy -uggage -▁luggage -▁interfaith -▁Decl -▁Declar -rotic -▁neurotic --two -▁himself, -▁entity -▁Borden -andy -▁diversion -▁mahayana -▁Dean -▁clues -▁statues -AM -▁Make -▁intake -▁rebuff -▁slick -marine -ubmarine -▁antisubmarine -▁fifties -▁xr -▁correction -▁somersault -▁Got -▁populations -▁schedules --level -▁pric -▁traveler -negie -▁secrecy -▁indo -DC -▁boxcar -▁novels -product -ps -psam -▁piepsam -▁papas -▁buffet -▁stunn -▁surgeon -▁brig -▁clamber -▁managerial -▁vicinity -▁Mountain -▁Main -▁lurk -▁followers -mile -cession -▁doses -makers -▁narc -▁Ref -▁Rourke -ormal -▁Standard -▁Nassau -▁nassau -▁agony --D -▁scoop -connected -▁Thank -▁Build -▁pamela -▁pa -▁pad -▁geneva -▁biwa -▁beliefs -▁Prairie -▁underneath -▁dictates -▁fuck -▁Chair -▁Chairman -man's -▁endow -trow -astrow -▁heavens -▁1933 -▁housew -▁houses, -bee -▁Room -acob -▁scenic -▁rodd -▁terrestrial -▁newcom -▁cruc -▁librarians -▁sensational -▁aunt -▁titled -▁antithyroid -▁arguing -ellowship -eur -▁comparisons -illac -▁unrec -▁passive -▁sympathize -▁Steinberg -enberg -▁steinberg -61 -▁declare -▁beaten -▁120 -▁boycott -▁humm --Western -▁ske -▁skelet -▁wrinkles -▁manners -▁enjoying -ingham -irmingham -▁Maude -▁antip -▁wartime -▁Means -▁scrutin -▁inflection -ollection -▁recollection -▁fundamentally -▁bitterly -▁displayed -▁displays -▁Boy -▁therapist -▁Planning -oelectric -olese -▁quote -▁initially -000000 -▁subtract -ism, -quarters -colored -▁landmark -▁sustained -▁accompaniment -▁souls -▁interlob -▁interlobular -▁farmhouse -▁lime -essert -▁dessert -ibet -▁anguish -children -▁grandchildren -icken -▁oxen -▁proven -▁drunken -agen -▁Handley -▁tart -▁maude -▁mexico -▁lizzie -▁practicing -▁certific -▁cobb -▁anyhow -▁cady -▁Cady -▁trifle -▁pulp -▁mother's -▁Pete -outheastern -▁manual -▁Tommy -▁tommy -▁reconc -▁designer -▁discon -shore -▁Congolese -▁unto -▁cadillac -esirable -▁undesirable -▁erupted -▁unreason -▁Cor -▁bog -▁greece -▁Ow -unted -▁grunted -▁grap -▁bour -▁feminine -▁throwing -▁bugg -▁braz -▁sprung -▁lessons -▁bug -▁dill -▁dc -▁Conc -acles -▁obstacles -▁improvis -▁"We -▁pollen -▁pollution -▁presents -interpre -▁misinterpre -▁outline -▁differed -▁knit -opular -▁schools, -▁parameters -▁idyll -▁catholics -ilian -▁conceptions -▁totalitarian -▁xray -▁stretching -aring -▁propagation -▁Lalaurie -▁temperament -▁vitally -▁killing -▁posts -▁physics -▁Katang -▁recreational -▁bacteria --stage -▁megat -▁megatons -▁stole -▁ordinance -▁benefact -▁benevol -▁grease -▁Katie -▁Free -▁manipulation -▁manipulate -▁arab -▁pall -▁hesitation -▁sanitation -▁hoop -▁fluids -▁identifiable -▁howard -▁Yalta -▁yalta -▁deterministic -▁dugout -▁witches -▁B.C. -▁convicted -▁Arch -▁propulsion -▁Men -▁Mass -▁harass -assion -▁disclosed -▁anthrop -▁Claude -▁fats -emics -▁morphophonemics -▁flashed -▁lighted -▁trevely -▁treach -▁pulse -▁aged -▁admiration -▁thermometers -▁Minn -▁Ma -▁lin -▁tablespoon -▁histories -ft -fold -▁tactical -▁foams -▁victims -▁sau -▁carroll -74 -▁throttle -▁outraged -▁ranks -▁disciples -▁alban -▁perilous -ousness -▁affiliated -▁knives -▁Power -▁Empire -▁stool -▁psychoanalysis -▁mouse -▁Cuban -iroshima -phosph -osphorus -▁lalaurie -▁Seventh -▁bruised -▁seep -▁Case -▁chase -activity -▁maker -▁others. -▁inconven -idges -▁proves -awaiian -▁Others -▁laz -▁glaz -idency -▁intimid -▁iliad -emia -▁logically -▁parameter -▁maggie -▁Marshal -▁immediacy -▁rendezvous -▁slapped -iature -▁miniature -▁Jon -▁compassion -aggie -▁Maggie -ieves -iepsam -ieving -▁excer -▁excerpt -erated -educated -▁collaborated -▁collaboration -▁sensation -▁lacked -3) -▁(3) -▁pseudo -ebread -▁Alread -▁cromwell -▁Cromwell -▁recourse -▁hip -▁katie -▁expedient -▁provoked -gust -▁disgust -▁Minister -▁kr -▁restore -▁packing -▁mg/ -anthrop -▁philanthrop --surface -▁szold -▁Szold -▁needed. -▁ire -▁aggregate -▁coordinate -▁skeptical -▁burdens -▁delph -▁emerging -▁grove -▁offset -,500 -▁Mahayana -▁vulgar -▁tablespoons -chair -▁groom -upiter -▁Remember -▁squall -▁freely -▁anglican -▁Anglican -▁digest -▁digging -ture -▁sculpture -▁Teach -▁distal -▁shelves -▁garbage -▁congenial -▁chewing -moil -▁turmoil -▁Greene -▁embassy -▁analogy -▁unnatural -▁ventil -▁deliberations -▁collapsed -34 -onnor -ylkill -ylan -▁earnestly -▁74 -▁repud -▁packaging -▁distinctly -▁unmarried -▁words. -▁crafts -▁Joan -▁Picasso -▁(about -▁narrator -▁narrower -▁proteins -▁combining -▁proportionate -bishop -▁miracles -▁England, -ifty -▁negotiation -▁rag -▁thief -▁embarrassment -▁laundry -▁redcoat -▁restoration -▁restorative -▁exaggerate -▁magnificently -ahn -▁ga -▁gaiety -▁nov --looking -▁Sur -▁pushing -▁leaning -▁fake -▁ferv -▁fel -▁pearl -▁uncommon -▁selfd -▁shores -rophores -▁repeal -▁prowestern -▁leav -oque -▁baroque -▁barco -avern -▁Nov -▁hunger -▁uneas -▁uneven -▁Record -▁Recently -▁Determ -▁Determine -▁cyclist -▁enclosure -▁discard -uard -▁(Sept -▁tin -▁rides -▁whipped -▁eclips -▁cath -▁hands. -▁ballroom -▁floors -▁tips -▁grokk -▁alk -▁alkal -▁Stal -▁Stalin -▁shrill -▁Arlene -▁griev -imburse -▁reimburse -▁penetrated -▁cortex -allic -▁metallic -▁unstable -avan -▁brav -▁bravado -▁brist -istent -▁smoothed -▁gospel -▁quirt -▁harness -ules -ullet -▁thirtyfive -▁lately -▁pilgrimage -▁duke -▁"You -▁equations -ations, -▁interrelations -▁taut -▁conjugate -▁unadjusted -▁lowered -▁104 -lington -▁Festival -▁safeg -▁safeguard -▁denver -lain -▁Plain -▁crops -▁2:0 -▁2:30 -90 -▁facade -▁attendance -▁delighted -▁(ex -▁(exclud -uylkill -▁States. -▁1914 -urp -urpose -▁courty -▁Wr -▁gamb -▁semantic -▁indeb -▁indebted -▁indecent -▁bolted -▁emancipation -▁Word -▁knuckles -▁ecstasy -▁Owen -▁masculine -▁accepting --spe -▁chauffeur -▁nineteenth-century -▁synd -▁syndic -▁Stevie -▁Stuart -▁jub -▁bald -den -▁scrub -yrate -▁alumni -▁teamsters -▁photographic -rais -▁Adjust -▁designs -▁weighed -nament -man, -clerk -auclerk -▁Advis -▁deciding -▁decides -▁ideally -▁Pierre -▁pierre -▁instituted -▁enters -▁AI -▁thoughtfully -▁uran -▁uranium -▁corros -rosse -▁radius -▁jokes -▁18th -▁19th -thood -▁garth -▁steal -▁York, -▁relied -entine -▁quine -▁quiney -▁glancing -▁supposedly -▁consequent -▁profile -atharine -son, -▁joining -▁Ambassador -▁impractical -▁Pohl -▁paragraphs -▁applicant -▁resultant -▁exhausted -▁elevated -▁Trans -▁splin -▁cooperating -▁fertiliz -▁arlene -▁ost -▁Horn -▁History -▁one's -lemish -▁reassuring -ourning -▁mourning -▁Love -▁glove -▁glendora -udged -▁surged -▁Timothy -▁sponsorship -▁treasurer -▁rider -▁terrifying -box -▁scent --owned -▁amplitude -▁amplifier -▁analyses -▁impulses -▁compilation -▁induced -▁dared -▁defenses -▁madden -▁embassies -▁magnetism -▁gracefully -▁spherical -▁beckon -▁foremost -▁habits -▁whirling -▁huddled -▁nuts -▁suburbs -▁drinks -▁shadows -▁meanings -▁Express -▁utilities -▁Claim -▁knelt -▁construed -agued -▁wholesale -▁Agricult -▁novelties -▁viewed -lements -▁elong -▁loving -▁incurred -▁slice -▁bailiff -▁1: -ota -▁finishing -eltzer -▁Too -▁pohl -▁institutional -▁broadening -▁terraces -▁terrified -▁beset -real -▁cracked -▁kicked -▁staggered -▁discriminating -aternal -cos -cest -▁ancest -▁ancestor -osium -▁symposium -▁sympathet -▁coronary -▁wealthy -partisan -▁parliamentary -▁1859 -▁expects -▁williams -▁ethic -▁rhythmic -filled -onghold -ounding -▁rim -▁spotlight -▁vex -▁premises -▁coerc -▁coin -otion -▁commotion -▁commenced -▁temptation -▁gigg -▁connoisse -▁rookie -▁psychother -▁proxim -▁Advisory -▁sensory -▁sealed -▁negotiating -▁inconsistent -urities -▁impurities -▁velocities -▁sweeping -▁impose -▁honorable -▁achieving -etically -▁hours. -▁potentially -▁Powell -▁Mos -▁fork -▁ascertain -▁cater --making -▁(as -asper -▁exasper -▁exquisite -▁concentrates -▁padd -▁(H -▁Van -▁imports --cl -▁esthetic -▁solving -▁cape -▁catastrophe -idity -▁humidity -acity -▁humil -▁coil -▁tid -ers' -'' -''" -▁McF -▁proceeding -▁dose -▁germanium -▁susceptible -▁Abel -▁Us -▁Usually -▁notably -▁config -▁configur -▁premise -▁nadine -▁gyp -eramic -▁ceramic -▁uphold -hard -▁Burton -▁rake -▁later, -▁southward -▁Protestants -▁framed -ordon -▁threaten -▁eaten -ponsored -▁promotional -▁presidency -▁disorganiz -▁loot -▁wis -wall -▁"in -ieur -▁satir -▁milling -▁failing -▁adolesc -▁adolescents -cin -uxley -inley -▁McKinley -▁teen-ag -agan -▁strokes -▁flair -▁Northwest -▁Meas -▁peasant -▁tolerant -▁selecting -▁ditch -▁inherited -▁innov -▁H. -▁repuls -▁richmond -▁Could -▁presiding -▁room. -▁halt -▁Bishop -leveland -▁Cleveland -▁Pamela -▁Piepsam --down -▁directions -ressed -▁Di -▁Dill -▁visibly -▁61 -▁retiring -▁retre -▁retreated -▁17th -▁fifteenth -arth -arthy -▁versions -▁diarr -ulletin -▁radish -▁compression -committ -▁Delph -eliness -elle -xel -▁drexel -▁hurr -▁hurrican -▁insights -▁inspector --colored -▁diverg -▁stevie -▁stuart -bright -llustr -▁stunning -▁drawings -▁Allied -▁Design -ensible -▁banner -▁usage -▁trustee -iere -▁jerked -▁compliance -▁evangelism -▁mysteries -s" -▁drought -▁skipped -▁baptized -▁homes. -▁butyrate -▁catcher -▁replaces -▁sunshine -rapy -llan -▁necessitate -▁facilitate -▁monotony -▁pony -▁quart -▁passions -houn -▁pronoun -▁quadric -▁owned -▁jeal -▁jealous -▁sensed -ospitable -▁actively -▁ireland -▁hillsboro -▁feudal -tral -str -▁hemorr -▁Torr -▁Torrio -▁B. -▁triumphantly -▁Hamrick -▁destroying -▁housed -haped -represent -▁booth -▁jewel -▁spontane -fiction -faced --tr -▁unhappily -▁moderately -▁intimately -▁Porter -▁monitoring -keeping -▁discoveries -▁galaxies -▁nickel -▁sediment -▁pirates -▁alg -▁algae -icted -▁underdeveloped -▁praised -▁sins -antically -▁frantically -▁chronological -▁confine -▁confed -▁confederate --filled -▁child's -▁striving -▁oysters -▁oyster -▁"not -▁cel -owski -▁mandatory -iates -▁burr -▁Library -▁Lublin -▁Export-Im -▁Export-Import -▁deploy -▁interviews -▁librarian -▁Been -emen -▁dissem -▁dissolve -▁pier -▁multi -itory -▁bellig -▁belligerent -pirited -▁edited -▁dictator -erator -▁distinctions -▁picasso -▁decad -▁Acad -▁Academ -▁Rh -hole -hous -▁warehous -▁uly -▁unduly -▁artificially -▁Juanita -▁crad -▁Columbia -▁ray --ray -▁trees. -▁authorizations -▁screaming -▁orthodontist -▁Invest -▁Electric -orical -▁explode -▁Code -▁Coun -▁customs -▁bos -▁bosom -omatic -onduct -endell -▁heels -▁F. -▁freezing -▁bathing -▁controls -▁ap -▁prophec -▁rub -▁meteorites -aining -▁implication -▁obtainable -osa -ucosa -▁expended -▁Shaw -▁outward -▁selections -▁Charlotte -ormick -▁None -▁havana -▁embarrassing -▁impaired -gg -▁disbelief -▁superimposed -existent -▁endorse -▁snatched -▁smother -▁proceeds -▁abus -▁flourished -▁Southeast -ubby -rosby -▁frankly -security -▁presentday -▁merge -▁ambul -ulence -▁vowel -▁experimentation -▁aids -▁Freud -ges. -▁nostalg -givanna -cinol -▁guiding -▁humiliation -▁acknowledge -▁pennant -▁Wally -▁Gabriel -▁cuban -▁columbia -▁170 -▁lash -▁sensibility -▁scraw --plane -▁dressing -▁spencer -onder -▁maneuvers -▁abundant -▁junct -rect -▁contemptuous -▁Feather -▁Feb -▁exclusion -▁1890 -▁1896 -▁incub -on, -ona -onious -aac -▁Isaac --step -▁rocks -▁ruthless -▁burgl -▁premiere -▁quill -▁quiver -▁crossroads -▁rows -▁pacif -▁prolif -▁Delphine -▁tanks -▁Lucille -▁sympathies -▁costume -▁yourselves -▁Concord -▁ultra -▁Sixties -▁year's -▁(C) -▁Marx -▁alleged -▁cleared -▁sighed -▁affixed -▁mailed -▁seced -▁juanita -▁genuinely -▁itself. -▁yelled -▁confirmed -ckinley -▁irritation -▁millenn -▁irradiated -▁disconcer -▁collecting -▁roam -▁demythologization -izational -▁civilizational -▁civilized -▁dissent -▁shocked -▁flowing -▁Similarly, -iodoty -iodotyrosine -▁country. -▁redcoats -▁instructed -▁instructor -rise -▁sunrise -anchise -▁sidewise -▁traditionally -▁(AP -▁dese -▁desegregated -▁1925 -▁Twenty -▁evaluating -▁heating -▁investigating -▁(Af -instein -▁compulsion -▁awarded -▁guards -lymouth -▁tumors -▁gib -▁preferences -▁relates -▁examiner -▁mind. -▁lavish -▁ties -▁ration -▁Salem --degrees-F -▁winchester -▁Fulton -▁fulton -▁focal -▁speculate -▁indecision -▁judged -▁judging -grily -▁1832 -▁52 -▁biscuit -itudes -▁budd -▁Agriculture -▁dusk -▁graduating -irov -crack -▁Jessica -▁swallowed -▁easiest -degree -ioles -elley -▁Shelley -▁Show -▁grams -ilt -▁revolt -▁tonal -▁penetrate -▁pessimistic -▁Yok -▁breakthrough -▁Claire -▁repression -▁Feathertop -▁screwed -▁sie -▁sore -▁prud -stiqlal -▁1960, -▁insolence -▁Measure diff --git a/lingvo/tasks/punctuator/params/codelab.py b/lingvo/tasks/punctuator/params/codelab.py deleted file mode 100644 index cbccff13..00000000 --- a/lingvo/tasks/punctuator/params/codelab.py +++ /dev/null @@ -1,125 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Params file for punctuator codelab.""" - -import os -from lingvo import model_registry -import lingvo.compat as tf -from lingvo.core import base_model_params -from lingvo.tasks.mt import base_config -from lingvo.tasks.punctuator import input_generator -from lingvo.tasks.punctuator import model - - -# This base class defines parameters for the input generator for a specific -# dataset. Specific network architectures will be implemented in subclasses. -class BrownCorpusWPM(base_model_params.SingleTaskModelParams): - """Brown Corpus data with a Word-Piece Model tokenizer.""" - - # Generated using - # lingvo/tasks/punctuator/tools:download_brown_corpus. - _DATADIR = '/tmp/punctuator_data' - _VOCAB_FILE = tf.resource_loader.get_path_to_datafile( - 'brown_corpus_wpm.16000.vocab') - # _VOCAB_SIZE needs to be a multiple of 16 because we use a sharded softmax - # with 16 shards. - _VOCAB_SIZE = 16000 - - def Train(self): - p = input_generator.PunctuatorInput.Params() - p.file_pattern = 'text:' + os.path.join(self._DATADIR, 'train.txt') - p.file_random_seed = 0 # Do not use a fixed seed. - p.file_parallelism = 1 # We only have a single input file. - - # The bucket upper bound specifies how to split the input into buckets. We - # train on sequences up to maximum bucket size and discard longer examples. - p.bucket_upper_bound = [10, 20, 30, 60, 120] - - # The bucket batch limit determines how many examples are there in each - # batch during training. We reduce the batch size for the buckets that - # have higher upper bound (batches that consist of longer sequences) - # in order to prevent out of memory issues. - # Note that this hyperparameter varies widely based on the model and - # language. Larger models may warrant smaller batches in order to fit in - # memory, for example; and ideographical languages like Chinese may benefit - # from more buckets. - p.bucket_batch_limit = [512, 256, 160, 80, 40] - - p.tokenizer.vocab_filepath = self._VOCAB_FILE - p.tokenizer.vocab_size = self._VOCAB_SIZE - p.tokenizer.pad_to_max_length = False - - # Set the tokenizer max length slightly longer than the largest bucket to - # discard examples that are longer than we allow. - p.source_max_length = p.bucket_upper_bound[-1] + 2 - p.target_max_length = p.bucket_upper_bound[-1] + 2 - return p - - # There is also a Dev method for dev set params, but we don't have a dev set. - def Test(self): - p = input_generator.PunctuatorInput.Params() - p.file_pattern = 'text:' + os.path.join(self._DATADIR, 'test.txt') - p.file_random_seed = 27182818 # Fix random seed for testing. - # The following two parameters are important if there's more than one input - # file. For this codelab it doesn't actually matter. - p.file_parallelism = 1 # Avoid randomness in testing. - # In order to make exactly one pass over the dev/test sets, we set buffer - # size to 1. Greater numbers may cause inaccurate dev/test scores. - p.file_buffer_size = 1 - - p.bucket_upper_bound = [10, 20, 30, 60, 120, 200] - p.bucket_batch_limit = [16] * 4 + [4] * 2 - - p.tokenizer.vocab_filepath = self._VOCAB_FILE - p.tokenizer.vocab_size = self._VOCAB_SIZE - p.tokenizer.pad_to_max_length = False - - p.source_max_length = p.bucket_upper_bound[-1] + 2 - p.target_max_length = p.bucket_upper_bound[-1] + 2 - return p - - -# This decorator registers the model in the Lingvo model registry. -# This file is lingvo/tasks/punctuator/params/codelab.py, -# so the model will be registered as punctuator.codelab.RNMTModel. -@model_registry.RegisterSingleTaskModel -class RNMTModel(BrownCorpusWPM): - """RNMT+ Model.""" - - def Task(self): - p = base_config.SetupRNMTParams( - model.RNMTModel.Params(), - name='punctuator_rnmt', - vocab_size=self._VOCAB_SIZE, - embedding_dim=1024, - hidden_dim=1024, - num_heads=4, - num_encoder_layers=6, - num_decoder_layers=8, - learning_rate=1e-4, - l2_regularizer_weight=1e-5, - lr_warmup_steps=500, - lr_decay_start=400000, - lr_decay_end=1200000, - lr_min=0.5, - ls_uncertainty=0.1, - atten_dropout_prob=0.3, - residual_dropout_prob=0.3, - adam_beta2=0.98, - adam_epsilon=1e-6, - ) - p.eval.samples_per_summary = 2466 - return p diff --git a/lingvo/tasks/punctuator/params/params.py b/lingvo/tasks/punctuator/params/params.py deleted file mode 100644 index 312e810a..00000000 --- a/lingvo/tasks/punctuator/params/params.py +++ /dev/null @@ -1,21 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Punctuator models hyper-parameters.""" - -# Import ModelParams to ensure that they are added to the global registry. -# pylint: disable=unused-import -import lingvo.tasks.punctuator.params.codelab -# pylint: enable=unused-import diff --git a/lingvo/tasks/punctuator/tools/BUILD b/lingvo/tasks/punctuator/tools/BUILD deleted file mode 100644 index 69e2a824..00000000 --- a/lingvo/tasks/punctuator/tools/BUILD +++ /dev/null @@ -1,15 +0,0 @@ -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -py_binary( - name = "download_brown_corpus", - srcs = ["download_brown_corpus.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - ], -) diff --git a/lingvo/tasks/punctuator/tools/__init__.py b/lingvo/tasks/punctuator/tools/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/lingvo/tasks/punctuator/tools/download_brown_corpus.py b/lingvo/tasks/punctuator/tools/download_brown_corpus.py deleted file mode 100644 index 9c558255..00000000 --- a/lingvo/tasks/punctuator/tools/download_brown_corpus.py +++ /dev/null @@ -1,140 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Downloads and processes the Brown Corpus (http://www.nltk.org/nltk_data).""" - -import os -import random -import string -from xml.etree import ElementTree -import lingvo.compat as tf - -tf.flags.DEFINE_string("outdir", "/tmp/punctuator_data", - "The output directory.") - -FLAGS = tf.flags.FLAGS - - -def main(_): - tf.logging.set_verbosity(tf.logging.INFO) - - basename = "brown_tei" - fname = basename + ".zip" - url = ("https://raw.githubusercontent.com/nltk/nltk_data/" + - "gh-pages/packages/corpora/" + fname) - sha256 = "335bec1ea6362751d5d5c46970137ebb01c80bf7d7d75558787729d275e0a687" - - tf.keras.utils.get_file( - fname, url, file_hash=sha256, cache_subdir=FLAGS.outdir, extract=True) - - tf.logging.info("\nDownload completed. Preprocessing...") - - with open(os.path.join(FLAGS.outdir, basename, "Corpus.xml"), "r") as xml: - root = ElementTree.fromstring(xml.read().replace( - 'xmlns="http://www.tei-c.org/ns/1.0"', "")) - sentences = [] - for sentence in root.findall("./TEI/text/body/p/s"): - # Example input sentence: - # The Fulton - # County Grand - # Jury said - # Friday an investigation - # of Atlanta's recent - # primary election - # produced `` no - # evidence '' that - # any irregularities - # took place . - # Example output text: - # The Fulton County Grand Jury said Friday an investigation of Atlanta's - # recent primary election produced "no evidence" that any irregularities - # took place. - text = "" - prepend_space = False - for child in sentence: - if child.tag == "w": - if prepend_space: - text += " " - text += child.text - prepend_space = True - elif child.tag == "c": - if child.text == "``": - if prepend_space: - text += " " - text += '"' - prepend_space = False - elif child.text == "''": - text += '"' - prepend_space = True - elif child.text == "'": - if prepend_space: - text += " '" - prepend_space = False - else: - text += "'" - prepend_space = True - elif child.text == "(" or child.text == "[": - if prepend_space: - text += " " - text += child.text - prepend_space = False - elif child.text == "-" or child.text == "--": - if prepend_space: - text += " " - text += child.text - prepend_space = True - else: - text += child.text - prepend_space = True - text = text.replace("!!", "!").replace("??", "?").replace("--", "-") - text = text.replace("**", "*").replace(";;", ";").replace("::", ":") - text = text.replace(",,", ",") - - # Filter out bad sentences. - if not set(text) & set(string.ascii_letters): - # No letters. - continue - if text.count('"') % 2 != 0: - # Uneven number of quotes. - continue - if text.count("(") != text.count(")") or text.count("[") != text.count("]"): - # Unbalanced parenthesis. - continue - if (text[0] == '"' and text[-1] == '"' or - text[0] == "(" and text[-1] == ")" or - text[0] == "[" and text[-1] == "]"): - text = text[1:-1] - if text[0] not in string.ascii_letters and text[0] not in string.digits: - # Doesn't start with a letter or number. - continue - text = text[:1].upper() + text[1:] - sentences.append(text) - sentences = sorted(set(sentences)) - random.seed(1234) - random.shuffle(sentences) - - with open(os.path.join(FLAGS.outdir, "train.txt"), "w") as f: - for line in sentences[:int(len(sentences) * 0.95)]: - f.write("%s\n" % line) - - with open(os.path.join(FLAGS.outdir, "test.txt"), "w") as f: - for line in sentences[int(len(sentences) * 0.95):]: - f.write("%s\n" % line) - - tf.logging.info("All done.") - - -if __name__ == "__main__": - tf.app.run(main) diff --git a/lingvo/tools/BUILD b/lingvo/tools/BUILD deleted file mode 100644 index 005f3641..00000000 --- a/lingvo/tools/BUILD +++ /dev/null @@ -1,209 +0,0 @@ -load( - "//lingvo:lingvo.bzl", - "lingvo_cc_binary", -) - -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -py_binary( - name = "gke_launch", - srcs = ["gke_launch.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - # Implicit absl.app dependency. - # Implicit absl.flags dependency. - # Implicit six dependency. - # Implicit yaml dependency. - ], -) - -py_binary( - name = "keras2ckpt", - srcs = ["keras2ckpt.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - ], -) - -py_binary( - name = "bpe_word_tokenizer", - srcs = ["bpe_word_tokenizer.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = ["//lingvo:compat"], -) - -py_binary( - name = "wpm_encode_file", - srcs = ["wpm_encode_file.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":wpm_encode_file_lib", - # Implicit six dependency. - ], -) - -py_library( - name = "wpm_encode_file_lib", - srcs = ["wpm_encode_file.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo/core:wpm_encoder", - # Implicit numpy dependency. - # Implicit six dependency. - ], -) - -py_binary( - name = "print_tf_records", - srcs = ["print_tf_records.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - # Implicit six dependency. - ], -) - -py_binary( - name = "compute_stats", - srcs = ["compute_stats.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - # Implicit numpy dependency. - ], -) - -py_library( - name = "audio_lib", - srcs = ["audio_lib.py"], - srcs_version = "PY3", - deps = [ - # Additional FFT kernels dependency. - "//lingvo:compat", - "//lingvo/core:py_utils", - "//lingvo/tasks/asr:frontend", - ], -) - -py_library( - name = "beam_utils", - srcs = ["beam_utils.py"], - srcs_version = "PY3", -) - -py_test( - name = "beam_utils_test", - srcs = ["beam_utils_test.py"], - data = ["//lingvo/tasks/mt/testdata:wmt14_ende_tfexample"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":beam_utils", - # Implicit apache_beam dependency. - "//lingvo:compat", - "//lingvo/core:test_helper", - "//lingvo/core:test_utils", - ], -) - -py_binary( - name = "count_records", - srcs = ["count_records.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":beam_utils", - # Implicit network file system dependency. - # Implicit apache_beam dependency. - ], -) - -py_test( - name = "audio_lib_test", - srcs = ["audio_lib_test.py"], - data = [ - "//lingvo/tools/testdata:audio_data", - ], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":audio_lib", - "//lingvo:compat", - "//lingvo/core:test_helper", - "//lingvo/core:test_utils", - ], -) - -py_library( - name = "compare_params_lib", - srcs = ["compare_params.py"], - srcs_version = "PY3", - deps = [ - "//lingvo:compat", - "//lingvo:model_registry", - # Implicit six dependency. - ], -) - -py_test( - name = "compare_params_test", - srcs = ["compare_params_test.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":compare_params_lib", - "//lingvo:compat", - "//lingvo/core:hyperparams", - "//lingvo/core:test_utils", - ], -) - -py_binary( - name = "create_asr_features", - srcs = ["create_asr_features.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":audio_lib", - "//lingvo:compat", - ], -) - -py_binary( - name = "create_peoples_speech_asr_features", - srcs = ["create_peoples_speech_asr_features.py"], - python_version = "PY3", - srcs_version = "PY3", - deps = [ - ":audio_lib", - "//lingvo:compat", - ], -) - - -lingvo_cc_binary( - name = "generate_proto_def", - srcs = ["generate_proto_def.cc"], - deps = [ - # Implicit proto dependency. - # Implicit tensorflow C++ proto dependency. - ], -) - -sh_binary( - name = "generate_tf_dot_protos", - srcs = ["generate_tf_dot_protos.sh"], - data = [":generate_proto_def"], -) diff --git a/lingvo/tools/__init__.py b/lingvo/tools/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/lingvo/tools/audio_lib.py b/lingvo/tools/audio_lib.py deleted file mode 100644 index 0f8f7449..00000000 --- a/lingvo/tools/audio_lib.py +++ /dev/null @@ -1,150 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Audio library.""" - -import shlex -import subprocess -from tempfile import NamedTemporaryFile -import lingvo.compat as tf -from lingvo.core import py_utils -from lingvo.tasks.asr import frontend as asr_frontend - -from tensorflow.python.ops import gen_audio_ops as audio_ops # pylint: disable=g-direct-tensorflow-import - - -# There are two ways to decode a wav in tensorflow: -# Through the tensorflow native audio decoder, exported -# via framework, or via tf.contrib.ffmpeg.decode_audio. -# While the latter could technically support FLAC, it does -# not. It also adds an extra dependency on ffmpeg. - -def DecodeToWav(input_bytes, fmt): - with NamedTemporaryFile() as fh: - cmd = f'sox -t {fmt} - -t wav --channels 1 --rate 16000 --encoding signed --bits 16 {fh.name}' - p = subprocess.Popen(shlex.split(cmd), - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - _, err = p.communicate(input=input_bytes) - assert p.returncode == 0, err - out = fh.read() - return out - - -def DecodeFlacToWav(input_bytes): - """Decode a FLAC byte string to WAV.""" - p = subprocess.Popen( - ['sox', '-t', 'flac', '-', '-t', 'wav', '-'], - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - out, err = p.communicate(input=input_bytes) - assert p.returncode == 0, err - return out - - -def DecodeWav(input_bytes): - """Decode a wav file from its contents. - - Args: - input_bytes: a byte array or Tensor with the wav file contents. - - Returns: - A pair of Tensor for sample rate, decoded samples. - """ - result = tf.audio.decode_wav(input_bytes) - return result.sample_rate, result.audio - -def read_wave_via_scipy(string_bytes): - from scipy.io.wavfile import read as wavread - import io - import numpy as np - b = string_bytes.numpy() - sample_rate, wav_file = wavread(io.BytesIO(b)) - wav_file = wav_file[:, np.newaxis] - assert wav_file.dtype == np.int16 - normalized = wav_file.astype(np.float32) * (1.0 / (1 << 15)) - assert normalized.max() <= 1.0 - assert normalized.min() >= -1.0 - return sample_rate, normalized - -""" -This should be bit-for-bit compatible with the output of DecodeWav. However, -there is no unit-test for that at the moment. -""" -def DecodeWavPyFunc(input_bytes): - sample_rate, audio = tf.py_function(read_wave_via_scipy, [input_bytes], [tf.int32, tf.float32]) - sample_rate = tf.ensure_shape(sample_rate, []) - audio = tf.ensure_shape(audio, [None, 1]) - return sample_rate, audio - - -def AudioToMfcc(sample_rate, audio, window_size_ms, window_stride_ms, - num_coefficients): - window_size_samples = sample_rate * window_size_ms // 1000 - window_stride_samples = sample_rate * window_stride_ms // 1000 - spectrogram = audio_ops.audio_spectrogram( - audio, - window_size=window_size_samples, - stride=window_stride_samples, - magnitude_squared=True) - mfcc = audio_ops.mfcc( - spectrogram, sample_rate, dct_coefficient_count=num_coefficients) - return mfcc - - -def ExtractLogMelFeatures(wav_bytes_t): - """Create Log-Mel Filterbank Features from raw bytes. - - Args: - wav_bytes_t: Tensor representing raw wav file as a string of bytes. It is - currently assumed that the wav file is encoded at 16KHz (see DecodeWav, - below). - - Returns: - A Tensor representing three stacked log-Mel filterbank energies, sub-sampled - every three frames. - """ - - # We want to use these parameters exactly. - def _CreateAsrFrontend(): - """Parameters corresponding to default ASR frontend.""" - p = asr_frontend.MelAsrFrontend.Params() - p.sample_rate = 16000. - p.frame_size_ms = 25. - p.frame_step_ms = 10. - p.num_bins = 80 - p.lower_edge_hertz = 125. - p.upper_edge_hertz = 7600. - p.preemph = 0.97 - p.noise_scale = 0. - p.pad_end = False - return p.Instantiate() - - sample_rate, audio = DecodeWav(wav_bytes_t) - audio *= 32768 - # Remove channel dimension, since we have a single channel. - audio = tf.squeeze(audio, axis=1) - # TODO(drpng): make batches. - audio = tf.expand_dims(audio, axis=0) - static_sample_rate = 16000 - mel_frontend = _CreateAsrFrontend() - with tf.control_dependencies( - [tf.assert_equal(sample_rate, static_sample_rate)]): - outputs = mel_frontend.FPropDefaultTheta( - py_utils.NestedMap(src_inputs=audio, paddings=tf.zeros_like(audio))) - log_mel = outputs.src_inputs - return log_mel diff --git a/lingvo/tools/audio_lib_test.py b/lingvo/tools/audio_lib_test.py deleted file mode 100644 index c4165650..00000000 --- a/lingvo/tools/audio_lib_test.py +++ /dev/null @@ -1,90 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for audio_lib.""" - -import lingvo.compat as tf -from lingvo.core import test_helper -from lingvo.core import test_utils -from lingvo.tools import audio_lib - -# The testdata contains: (soxi .../gan_or_vae.wav) -# Channels : 1 -# Sample Rate : 24000 -# Precision : 16-bit -# Duration : 00:00:03.16 = 75900 samples ~ 237.188 CDDA sectors - - -class AudioLibTest(test_utils.TestCase): - - def testDecodeFlacToWav(self): - with open( - test_helper.test_src_dir_path('tools/testdata/gan_or_vae.wav'), - 'rb') as f: - wav = f.read() - with open( - test_helper.test_src_dir_path('tools/testdata/gan_or_vae.flac'), - 'rb') as f: - flac = f.read() - tf.logging.info('flac: %d bytes', len(flac)) - try: - converted = audio_lib.DecodeFlacToWav(flac) - tf.logging.info('wav: %d bytes, converted: %d bytes', len(wav), - len(converted)) - self.assertEqual(wav, converted) - except OSError: - # sox is not installed, ignore this test. - pass - - def testDecodeWav(self): - with open( - test_helper.test_src_dir_path('tools/testdata/gan_or_vae.wav'), - 'rb') as f: - wav = f.read() - with self.session(): - sample_rate, audio = self.evaluate(audio_lib.DecodeWav(wav)) - self.assertEqual(24000, sample_rate) - self.assertEqual(75900, len(audio)) - - def testAudioToMfcc(self): - with open( - test_helper.test_src_dir_path('tools/testdata/gan_or_vae.wav'), - 'rb') as f: - wav = f.read() - sample_rate, audio = audio_lib.DecodeWav(wav) - static_sample_rate = 24000 - mfcc = audio_lib.AudioToMfcc(static_sample_rate, audio, 32, 25, 40) - with self.session(): - audio_sample_rate, mfcc = self.evaluate([sample_rate, mfcc]) - assert audio_sample_rate == static_sample_rate - self.assertAllEqual(mfcc.shape, [1, 126, 40]) - - def testExtractLogMelFeatures(self): - with open( - test_helper.test_src_dir_path('tools/testdata/gan_or_vae.16k.wav'), - 'rb') as f: - wav = f.read() - - wav_bytes_t = tf.constant(wav, dtype=tf.string) - log_mel_t = audio_lib.ExtractLogMelFeatures(wav_bytes_t) - - with self.session(): - log_mel = self.evaluate(log_mel_t) - # Expect 314, 80 dimensional channels. - self.assertAllEqual(log_mel.shape, [1, 314, 80, 1]) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tools/beam_utils.py b/lingvo/tools/beam_utils.py deleted file mode 100644 index 1e746399..00000000 --- a/lingvo/tools/beam_utils.py +++ /dev/null @@ -1,115 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tools for car beam pipelines.""" - -import apache_beam as beam - - -def BeamInit(): - """Initialize the beam program. - - Typically first thing to run in main(). This call is needed before FLAGS - are accessed, for example. - """ - pass - - -def GetPipelineRoot(options=None): - """Return the root of the beam pipeline. - - Typical usage looks like: - - with GetPipelineRoot() as root: - _ = (root | beam.ParDo() | ...) - - In this example, the pipeline is automatically executed when the context is - exited, though one can manually run the pipeline built from the root object as - well. - - Args: - options: A beam.options.pipeline_options.PipelineOptions object. - - Returns: - A beam.Pipeline root object. - """ - return beam.Pipeline(options=options) - - -def GetReader(record_format, file_pattern, value_coder, **kwargs): - """Returns a beam Reader based on record_format and file_pattern. - - Args: - record_format: String record format, e.g., 'tfrecord'. - file_pattern: String path describing files to be read. - value_coder: Coder to use for the values of each record. - **kwargs: arguments to pass to the corresponding Reader object constructor. - - Returns: - A beam reader object. - - Raises: - ValueError: If an unsupported record_format is provided. - """ - if record_format == "tfrecord": - return beam.io.ReadFromTFRecord(file_pattern, coder=value_coder, **kwargs) - - raise ValueError("Unsupported record format: {}".format(record_format)) - - -def GetWriter(record_format, file_pattern, value_coder, **kwargs): - """Returns a beam Writer. - - Args: - record_format: String record format, e.g., 'tfrecord' to write as. - file_pattern: String path describing files to be written to. - value_coder: Coder to use for the values of each written record. - **kwargs: arguments to pass to the corresponding Writer object constructor. - - Returns: - A beam writer object. - - Raises: - ValueError: If an unsupported record_format is provided. - """ - if record_format == "tfrecord": - return beam.io.WriteToTFRecord(file_pattern, coder=value_coder, **kwargs) - raise ValueError("Unsupported record format: {}".format(record_format)) - - -def GetEmitterFn(record_format): - """Returns an Emitter function for the given record_format. - - An Emitter function takes in a key and value as arguments and returns - a structure that is compatible with the Beam Writer associated with - the corresponding record_format. - - Args: - record_format: String record format, e.g., 'tfrecord' to write as. - - Returns: - An emitter function of (key, value) -> Writer's input type. - - Raises: - ValueError: If an unsupported record_format is provided. - """ - - def _ValueEmitter(key, value): - del key - return [value] - - if record_format == "tfrecord": - return _ValueEmitter - raise ValueError("Unsupported record format: {}".format(record_format)) diff --git a/lingvo/tools/beam_utils_test.py b/lingvo/tools/beam_utils_test.py deleted file mode 100644 index f78646da..00000000 --- a/lingvo/tools/beam_utils_test.py +++ /dev/null @@ -1,64 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for beam_utils.""" - -import apache_beam as beam -from lingvo import compat as tf -from lingvo.core import test_helper -from lingvo.core import test_utils -from lingvo.tools import beam_utils - - -class BeamUtilsTest(test_utils.TestCase): - - def testReaders(self): - pattern = test_helper.test_src_dir_path( - 'tasks/mt/testdata/wmt14_ende_wpm_32k_test.tfrecord') - _ = beam_utils.GetReader( - 'tfrecord', - pattern, - value_coder=beam.coders.ProtoCoder(tf.train.Example)) - - with self.assertRaises(ValueError): - _ = beam_utils.GetReader( - 'unknown', - '/tmp/foo', - value_coder=beam.coders.ProtoCoder(tf.train.Example)) - - def testWriters(self): - _ = beam_utils.GetWriter( - 'tfrecord', - '/tmp/foo@1', - value_coder=beam.coders.ProtoCoder(tf.train.Example)) - - with self.assertRaises(ValueError): - _ = beam_utils.GetWriter( - 'unknown', - '/tmp/foo@1', - value_coder=beam.coders.ProtoCoder(tf.train.Example)) - - def testGetPipelineRoot(self): - with beam_utils.GetPipelineRoot() as root: - _ = root | beam.Create([1, 2, 3]) | beam.Map(lambda x: x) - - def testGetEmitterFn(self): - _ = beam_utils.GetEmitterFn('tfrecord') - with self.assertRaises(ValueError): - _ = beam_utils.GetEmitterFn('unknown') - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tools/bpe_word_tokenizer.py b/lingvo/tools/bpe_word_tokenizer.py deleted file mode 100644 index 9e391df9..00000000 --- a/lingvo/tools/bpe_word_tokenizer.py +++ /dev/null @@ -1,102 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Generates the words_to_ids file from a BPE encoded corpus and BPE vocab file. - -Extracts all the words in the corpus with their corresponding list of ids. Each -subword in the vocab file is mapped to their line number as its id. The lines of -the output file are like: -... -TAKE 43,7,50,14 -THAT 16,35 -THE 26 -THEIR 16,4,9,56 -... -Which is compatible with the BPE tokenizer op in core/tokenizer.py. - -Typical workflow: - - subword-nmt learn-bpe train_file code_file - subword-nmt apply-bpe code_file train_file train_bpe_file - subword-nmt get-vocab train_bpe_file vocab_file - - bpe_word_tokenizer train_bpe_file vocab_file words_to_ids_file -""" - -import lingvo.compat as tf - -tf.flags.DEFINE_string('encoded_filepath', '', - 'Path to the BPE encoded corpus file.') -tf.flags.DEFINE_string('vocab_filepath', '', 'Path to the BPE vocab file.') -tf.flags.DEFINE_string('output_filepath', '', - 'The output filepath (word_to_ids).') -FLAGS = tf.flags.FLAGS - - -def _GetVocabulary(vocab_filepath): - """Maps the first word in each line of the given file to its line number.""" - vocab = {} - with open(vocab_filepath, 'r') as vocab_file: - for i, line in enumerate(vocab_file): - word = line.strip('\r\n ').split(' ')[0] - if word: - vocab[word] = i - return vocab - - -def _ExtractTokenization(encoded_filepath, vocab): - """Maps the words in the encoded file to their list of token ids. - - Reads all the subwords in encoded file. Concatenates them while they have @@ - as their last two characters. The last token of a word is the subword without - @@. Maps the full word to the list of corresponding vocab ids of the subwords - from the vocab dictionary. - - Args: - encoded_filepath: String, filepath of the BPE encoded file. - vocab: Dictionary of subwords (string) to token ids (int). - - Returns: - Dictionary of words (string) to list of token ids (list of int). - """ - word_tokenization = {} - with open(encoded_filepath, 'r') as encoded_file: - for line in encoded_file: - full_word = '' - ids = [] - for word in line.strip('\r\n ').split(' '): - ids.append(vocab[word]) - if word[-2:] == '@@': - full_word += word[:-2] - else: - full_word += word - word_tokenization[full_word] = ids - full_word = '' - ids = [] - return word_tokenization - - -def main(_): - vocab = _GetVocabulary(FLAGS.vocab_filepath) - word_tokenization = _ExtractTokenization(FLAGS.encoded_filepath, vocab) - with open(FLAGS.output_filepath, 'w') as output: - for word, ids in word_tokenization.items(): - output.write(word + ' ') - output.write(','.join(map(str, ids))) - output.write('\r\n') - - -if __name__ == '__main__': - tf.app.run(main) diff --git a/lingvo/tools/compare_params.py b/lingvo/tools/compare_params.py deleted file mode 100644 index 17553625..00000000 --- a/lingvo/tools/compare_params.py +++ /dev/null @@ -1,115 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Library for comparing two models / hyperparams.""" - -from lingvo import compat as tf -from lingvo import model_registry -import six - - -def _hyperparams_text_to_dict(cfg_text): - """Converts hyperparams config text to a dictionary of key-value pairs.""" - txt_list = six.ensure_str(cfg_text).split("\n") - pair_list = [] - for v in txt_list: - if not v: - continue - vals = v.split(" : ") - if len(vals) != 2: - raise ValueError(v) - pair_list.append(vals) - return dict(pair_list) - - -def hyperparams_text_diff(cfg1_text, cfg2_text): - """Computes the differences between two hyperparams.Params texts. - - Args: - cfg1_text: A hyperparams.Params().ToText() of the first model config. - cfg2_text: A hyperparams.Params().ToText() of the second model config. - - Returns: - A tuple of 3 elements: - - - cfg1_not_cfg2: A list of keys in cfg1 but not cfg2. - - cfg2_not_cfg1: A list of keys in cfg2 but not cfg1. - - cfg1_and_cfg2_diff: A dict of common keys whose config values differ: each - value is a tuple of the config values from cfg1 and cfg2 respectively. - """ - cfg1_dict = _hyperparams_text_to_dict(cfg1_text) - cfg2_dict = _hyperparams_text_to_dict(cfg2_text) - cfg1_keys = set(cfg1_dict.keys()) - cfg2_keys = set(cfg2_dict.keys()) - - cfg1_not_cfg2 = sorted(list(cfg1_keys - cfg2_keys)) - cfg2_not_cfg1 = sorted(list(cfg2_keys - cfg1_keys)) - - def get_class_name(v): - try: - idx = v.rindex("/") - return v[idx + 1:] - except ValueError: - return v - - cfg1_and_cfg2_diff = {} - for k_intersection in cfg1_keys & cfg2_keys: - c1v = cfg1_dict[k_intersection] - c2v = cfg2_dict[k_intersection] - if k_intersection.endswith(".cls"): - c1v = get_class_name(c1v) - c2v = get_class_name(c2v) - - if c1v != c2v: - cfg1_and_cfg2_diff[k_intersection] = (c1v, c2v) - - return cfg1_not_cfg2, cfg2_not_cfg1, cfg1_and_cfg2_diff - - -def print_hyperparams_text_diff(path1, path2, cfg1_not_cfg2, cfg2_not_cfg1, - cfg1_and_cfg2_diff): - """Prints the differences of the output of hyperparams_text_diff. - - Args: - path1: Name of registered model or path to model 1. - path2: Name of registered model or path to model 2. - cfg1_not_cfg2: A list of keys in cfg1 but not cfg2. - cfg2_not_cfg1: A list of keys in cfg2 but not cfg1. - cfg1_and_cfg2_diff: A dictionary of common keys whose config values differ; - each value is a tuple of the config values from cfg1 and cfg2 - respectively. - """ - if cfg1_not_cfg2: - print("\n\nKeys in %s but not %s: \n%s\n\n" % - (path1, path2, "\n".join(cfg1_not_cfg2))) - if cfg2_not_cfg1: - print("\n\nKeys in %s but not %s: \n%s\n\n" % - (path2, path1, "\n".join(cfg2_not_cfg1))) - - if cfg1_and_cfg2_diff: - print("\n\nKeys with differences and their values: \n\n") - for k, v in sorted(cfg1_and_cfg2_diff.items()): - v1, v2 = v - print("%s: [%s] vs. [%s]" % (k, v1, v2)) - print("\n\n") - - -def get_model_params_as_text(model_path): - try: - cfg = model_registry.GetParams(model_path, "Train") - return cfg.ToText() - except LookupError: - # Try reading as file. - return tf.io.gfile.GFile(model_path).read() diff --git a/lingvo/tools/compare_params_test.py b/lingvo/tools/compare_params_test.py deleted file mode 100644 index 2c06aa57..00000000 --- a/lingvo/tools/compare_params_test.py +++ /dev/null @@ -1,45 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for compare_params.""" - -from lingvo import compat as tf -from lingvo.core import hyperparams -from lingvo.core import test_utils -from lingvo.tools import compare_params - - -class CompareParamsTest(test_utils.TestCase): - - def testBasic(self): - h1 = hyperparams.Params() - h1.Define('a', 1, '') - h1.Define('b', 2, '') - - h2 = hyperparams.Params() - h2.Define('a', 3, '') # different value. - h2.Define('c', 2, '') - - d1, d2, d3 = compare_params.hyperparams_text_diff(h1.ToText(), h2.ToText()) - self.assertEqual(d1, ['b']) - self.assertEqual(d2, ['c']) - self.assertEqual(d3, {'a': ('1', '3')}) - - # Exercise print function - compare_params.print_hyperparams_text_diff('h1', 'h2', d1, d2, d3) - - -if __name__ == '__main__': - tf.test.main() diff --git a/lingvo/tools/compute_stats.py b/lingvo/tools/compute_stats.py deleted file mode 100644 index 3dd710c4..00000000 --- a/lingvo/tools/compute_stats.py +++ /dev/null @@ -1,113 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Compute stats from tfrecords files.""" - -import lingvo.compat as tf -import numpy as np - -tf.flags.DEFINE_string('input_filepattern', '', - 'File pattern of binary tfrecord files.') -tf.flags.DEFINE_integer('frame_size', 1, 'Size of the frame, for reshaping.') -tf.flags.DEFINE_integer('num_buckets', 8, 'Number of buckets for the length.') -tf.flags.DEFINE_string('feature_name', None, 'Name of feature to examine.') - -FLAGS = tf.flags.FLAGS - - -class StatsCollector: - """Collect stats.""" - - def __init__(self,): - self._num_examples = 0 - self._lengths = [] - self._num_frames = 0 - self._mean_acc = np.zeros(FLAGS.frame_size, dtype=np.float64) - self._var_acc = np.zeros(FLAGS.frame_size, dtype=np.float64) - - def _AccumulateMoments(self, float_list): - frames = np.reshape(float_list, [-1, FLAGS.frame_size]) - self._num_frames += frames.shape[0] - self._mean_acc += np.sum(frames, axis=0) - self._var_acc += np.sum(frames * frames, axis=0) - - def _ComputeMeanVar(self): - mu = self._mean_acc / self._num_frames - # The user is in charge of replacing NaNs with a floor value. - v = np.sqrt(self._var_acc / self._num_frames - mu * mu) - return mu, v - - def Accumulate(self, tf_ex): - self._num_examples += 1 - if 0 == self._num_examples % 10000: - tf.logging.info('Processing example %u...', self._num_examples) - v = tf_ex.features.feature[FLAGS.feature_name] - if v.HasField('float_list'): - num_frames = len(v.float_list.value) // FLAGS.frame_size - self._AccumulateMoments(v.float_list.value) - elif v.HasField('int64_list'): - num_frames = len(v.int64_list.value) // FLAGS.frame_size - else: - tf.logging.fatal( - 'Not sure what to do with value. ' - 'Only float/int64 lists are supported: %s', v) - self._lengths.append(num_frames) - - def _PrintLengthBuckets(self): - sorted_lengths = sorted(self._lengths) - num_buckets = FLAGS.num_buckets - n = len(sorted_lengths) - idx = (n * (np.array(list(range(num_buckets - 1))) + 1)) // num_buckets - buckets = [sorted_lengths[i] for i in idx] + [sorted_lengths[-1]] - tf.logging.info('== Buckets.') - tf.logging.info('bucket upper limits: %s', buckets) - tf.logging.info('Other candidates for last bucket:') - tf.logging.info(' 0.1%% loss: %u', sorted_lengths[int(n * .999)]) - tf.logging.info(' 1%% loss: %u', sorted_lengths[int(n * .99)]) - tf.logging.info(' 2%% loss: %u', sorted_lengths[int(n * .98)]) - - def _PrintMeanVar(self): - m, v = self._ComputeMeanVar() - original = np.get_printoptions() - np.set_printoptions(threshold=np.inf) - tf.logging.info('== Mean/variance.') - tf.logging.info('mean = %s', m) - tf.logging.info('var = %s', v) - np.set_printoptions(**original) - - def Print(self): - tf.logging.info('== Total number of examples: %u', self._num_examples) - self._PrintLengthBuckets() - self._PrintMeanVar() - - -def main(_): - tf.logging.set_verbosity(tf.logging.INFO) - if not FLAGS.feature_name: - tf.logging.fatal( - 'Use a --feature_name to specify what to bucketize on. ' - 'For instance, source_id for MT or frames for ASR.') - stats = StatsCollector() - for filepath in tf.io.gfile.glob(FLAGS.input_filepattern): - records = tf.compat.v1.io.tf_record_iterator(filepath) - for serialized in records: - ex = tf.train.Example() - ex.ParseFromString(serialized) - stats.Accumulate(ex) - stats.Print() - - -if __name__ == '__main__': - tf.app.run(main) diff --git a/lingvo/tools/count_records.py b/lingvo/tools/count_records.py deleted file mode 100644 index a6c2e9d4..00000000 --- a/lingvo/tools/count_records.py +++ /dev/null @@ -1,69 +0,0 @@ -# Lint as: python3 -# Copyright 2020 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -r"""Tool to count number of records in a dataset. - -Most other file formats have efficient ways to fetch the number of records in a -dataset. However, some formats such as TFRecord requires you to essentially -scan the files to perform this count. - -# GALVEZ: This could be useful to us - -This is a short little beam script that can leverage many machines to read -all of the files in parallel potentially faster than a single machine script. -It is recommended that for other file formats, simply reading the metadata -available in their formats should work; this file should not really be -extended to any other format that already has efficient ways of counting -records. -""" - -from absl import app -from absl import flags - -import apache_beam as beam -from lingvo.tools import beam_utils - -flags.DEFINE_string('input_file_pattern', None, 'Path to read input') -flags.DEFINE_string('output_count_file', None, 'File to write output to.') -flags.DEFINE_string('record_format', None, - 'Record format of the input, e.g., tfrecord.') - -FLAGS = flags.FLAGS - - -def main(argv): - beam_utils.BeamInit() - - # Construct pipeline options from argv. - options = beam.options.pipeline_options.PipelineOptions(argv[1:]) - - reader = beam_utils.GetReader( - FLAGS.record_format, - FLAGS.input_file_pattern, - value_coder=beam.coders.BytesCoder()) - - with beam_utils.GetPipelineRoot(options=options) as root: - _ = ( - root - | 'Read' >> reader # Read each record. - | 'EmitOne' >> beam.Map(lambda _: 1) # Emit a 1 for each record. - | 'Count' >> beam.CombineGlobally(sum) # Sum counts. - | 'WriteToText' >> beam.io.WriteToText(FLAGS.output_count_file)) - - -if __name__ == '__main__': - flags.mark_flags_as_required( - ['input_file_pattern', 'output_count_file', 'record_format']) - app.run(main) diff --git a/lingvo/tools/create_asr_features.py b/lingvo/tools/create_asr_features.py deleted file mode 100644 index a0e2d5fd..00000000 --- a/lingvo/tools/create_asr_features.py +++ /dev/null @@ -1,218 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Encode the audio tarball contents into tfrecords.""" - -import os -import random -import re -import tarfile -import lingvo.compat as tf -from lingvo.tools import audio_lib - -tf.flags.DEFINE_string('input_tarball', '', 'Input .tar.gz file.') -tf.flags.DEFINE_string('input_text', '', 'Reference text.') -tf.flags.DEFINE_string('output_template', '', 'File of tfrecords.') - -tf.flags.DEFINE_bool('dump_transcripts', False, - 'First pass through the tarball to read the transcripts.') -tf.flags.DEFINE_string('transcripts_filepath', '', - 'Where to put the transcripts.') -tf.flags.DEFINE_bool('generate_tfrecords', False, - 'Second pass generates the tf records') - -tf.flags.DEFINE_integer('shard_id', -1, 'Processor shard.') -tf.flags.DEFINE_integer( - 'num_shards', -1, - 'Number of processor shards. Must divide num_output_shards.') -tf.flags.DEFINE_integer('output_range_begin', -1, 'Begin of output shard IDs.') -tf.flags.DEFINE_integer('output_range_end', -1, 'End of output shard IDs.') -tf.flags.DEFINE_integer('num_output_shards', -1, - 'Total number of output shards.') - -FLAGS = tf.flags.FLAGS - - -def _MakeBytesFeature(unicode_array): - value = [tf.compat.as_bytes(w) for w in unicode_array] - return tf.train.Feature(bytes_list=tf.train.BytesList(value=value)) - - -def _MakeInt64Feature(value): - return tf.train.Feature(int64_list=tf.train.Int64List(value=value)) - - -def _MakeFloatFeature(value): - return tf.train.Feature(float_list=tf.train.FloatList(value=value)) - - -# I ought to use _MakeTfExample directly in my code. -def _MakeTfExample(uttid, frames, text): - flat_frames = frames.flatten() - feature = { - 'uttid': _MakeBytesFeature([uttid]), - 'transcript': _MakeBytesFeature([text.lower()]), - 'frames': _MakeFloatFeature(flat_frames) - } - # Okay, so let's just use tf.train.Example then. Otherwise, things get too confusing. - return tf.train.Example(features=tf.train.Features(feature=feature)) - - -def _ReadTranscriptions(): - """Read all transcription files from the tarball. - - Returns: - A map of utterance id to upper case transcription. - """ - file_obj = tf.io.gfile.GFile(FLAGS.input_tarball, mode='rb') - tar = tarfile.open(fileobj=file_obj, mode='r:gz') - n = 0 - tf.logging.info('First pass: loading text files...') - # TODO(drpng): there's more information in the following files: - # LibriSpeech/LICENSE.TXT - # LibriSpeech/README.TXT - # LibriSpeech/CHAPTERS.TXT - # LibriSpeech/SPEAKERS.TXT - # LibriSpeech/BOOKS.TXT - trans = {} - for tarinfo in tar: - if not tarinfo.isreg(): - continue - n += 1 - if 0 == n % 10000: - tf.logging.info('Scanned %d entries...', n) - if not tarinfo.name.endswith('.trans.txt'): - continue - # The file LibriSpeech/dev-clean/3170/137482/3170-137482.trans.txt - # will contain lines such as: - # 3170-137482-0000 WITH AN EDUCATION WHICH OUGHT TO ... - # 3170-137482-0001 I WAS COMPELLED BY POVERTY ... - key = tarinfo.name.strip('.trans.txt') - f = tar.extractfile(tarinfo) - u = 0 - for l in f.readlines(): - l = l.decode('utf-8') - uttid, txt = l.strip('\n').split(' ', 1) - trans[uttid] = txt - u += 1 - tf.logging.info('[%s] = %d utterances', key, u) - f.close() - tar.close() - file_obj.close() - return trans - - -def _DumpTranscripts(): - trans = _ReadTranscriptions() - with tf.io.gfile.GFile(FLAGS.transcripts_filepath, 'w') as f: - for uttid in sorted(trans): - f.write('%s %s\n' % (uttid, trans[uttid])) - - -def _LoadTranscriptionsFromFile(): - trans = {} - with tf.io.gfile.GFile(FLAGS.transcripts_filepath, 'r') as f: - for line in f.readlines(): - uttid, txt = line.strip('\n').split(' ', 1) - trans[uttid] = txt - return trans - - -def _MakeLogMelFromTensorflowBuiltin(tf_wav_bytes): - sample_rate, audio = audio_lib.DecodeWav(tf_wav_bytes) - static_sample_rate = 16000 - with tf.control_dependencies( - [tf.assert_equal(sample_rate, static_sample_rate)]): - log_mel = audio_lib.AudioToMfcc(static_sample_rate, audio, 25, 25, 40) - return log_mel - - -def _OpenSubShards(): - tf.logging.info('Shards: %d to %d', FLAGS.output_range_begin, - FLAGS.output_range_end) - recordio_writers = [] - for s in range(FLAGS.output_range_begin, FLAGS.output_range_end): - filepath = FLAGS.output_template % (s, FLAGS.num_output_shards) - tf.logging.info('Opening output shard: %s', filepath) - recordio_writers += [tf.python_io.TFRecordWriter(filepath)] - return recordio_writers - - -def _CloseSubShards(files): - for f in files: - f.close() - - -def _SelectRandomShard(files): - subshard = random.randint(0, len(files) - 1) - return files[subshard] - - -def _CreateAsrFeatures(): - # First pass: extract transcription files. - if os.path.exists(FLAGS.transcripts_filepath): - trans = _LoadTranscriptionsFromFile() - else: - tf.logging.info('Running first pass on the fly') - trans = _ReadTranscriptions() - tf.logging.info('Total transcripts: %d', len(trans)) - tf_bytes = tf.placeholder(dtype=tf.string) - # Great! It uses the frontend directly - log_mel = audio_lib.ExtractLogMelFeatures(tf_bytes) - # Second pass: transcode the flac. - file_obj = tf.io.gfile.GFile(FLAGS.input_tarball, mode='rb') - tar = tarfile.open(fileobj=file_obj, mode='r:gz') - n = 0 - recordio_writers = _OpenSubShards() - tfconf = tf.config_pb2.ConfigProto() - tfconf.gpu_options.allow_growth = True - with tf.Session(config=tfconf) as sess: - for tarinfo in tar: - if not tarinfo.name.endswith('.flac'): - continue - n += 1 - if n % FLAGS.num_shards != FLAGS.shard_id: - continue - uttid = re.sub('.*/(.+)\\.flac', '\\1', tarinfo.name) - f = tar.extractfile(tarinfo) - wav_bytes = audio_lib.DecodeFlacToWav(f.read()) - f.close() - frames = sess.run(log_mel, feed_dict={tf_bytes: wav_bytes}) - assert uttid in trans, uttid - num_words = len(trans[uttid]) - tf.logging.info('utt[%d]: %s [%d frames, %d words]', n, uttid, - frames.shape[1], num_words) - ex = _MakeTfExample(uttid, frames, trans[uttid]) - outf = _SelectRandomShard(recordio_writers) - outf.write(ex.SerializeToString()) - tar.close() - file_obj.close() - _CloseSubShards(recordio_writers) - - -def main(_): - tf.logging.set_verbosity(tf.logging.INFO) - if FLAGS.dump_transcripts: - _DumpTranscripts() - elif FLAGS.generate_tfrecords: - _CreateAsrFeatures() - else: - tf.logging.error( - 'Nothing to do! Use --dump_transcripts or --generate_tfrecords') - - -if __name__ == '__main__': - tf.disable_eager_execution() - tf.app.run(main) diff --git a/lingvo/tools/create_peoples_speech_asr_features.py b/lingvo/tools/create_peoples_speech_asr_features.py deleted file mode 100644 index 7bfb9829..00000000 --- a/lingvo/tools/create_peoples_speech_asr_features.py +++ /dev/null @@ -1,189 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Encode the audio tarball contents into tfrecords.""" - -import os -import csv -import random -import tarfile -import lingvo.compat as tf -from lingvo.tools import audio_lib - -tf.flags.DEFINE_string('input_tarball', '', 'Input .tar.gz file.') -tf.flags.DEFINE_string( - 'input_text', '', 'Reference text as csv, filename, transcript, metadata.') -tf.flags.DEFINE_string('output_template', '', 'File of tfrecords.') - -tf.flags.DEFINE_bool('dump_transcripts', False, - 'First pass through the tarball to read the transcripts.') -tf.flags.DEFINE_string('transcripts_filepath', '', - 'Where to put the transcripts.') -tf.flags.DEFINE_bool('generate_tfrecords', False, - 'Second pass generates the tf records') - -tf.flags.DEFINE_integer('shard_id', -1, 'Processor shard.') -tf.flags.DEFINE_integer( - 'num_shards', -1, - 'Number of processor shards. Must divide num_output_shards.') -tf.flags.DEFINE_integer('output_range_begin', -1, 'Begin of output shard IDs.') -tf.flags.DEFINE_integer('output_range_end', -1, 'End of output shard IDs.') -tf.flags.DEFINE_integer('num_output_shards', -1, - 'Total number of output shards.') - -FLAGS = tf.flags.FLAGS - - -def _MakeBytesFeature(unicode_array): - value = [tf.compat.as_bytes(w) for w in unicode_array] - return tf.train.Feature(bytes_list=tf.train.BytesList(value=value)) - - -def _MakeInt64Feature(value): - return tf.train.Feature(int64_list=tf.train.Int64List(value=value)) - - -def _MakeFloatFeature(value): - return tf.train.Feature(float_list=tf.train.FloatList(value=value)) - - -def _MakeTfExample(uttid, frames, text): - flat_frames = frames.flatten() - feature = { - 'uttid': _MakeBytesFeature([uttid]), - 'transcript': _MakeBytesFeature([text.lower()]), - 'frames': _MakeFloatFeature(flat_frames) - } - return tf.train.Example(features=tf.train.Features(feature=feature)) - - -def _ReadTranscriptionsFromCSV(): - trans = {} - with tf.io.gfile.GFile(FLAGS.input_text, 'r') as f: - for row in csv.reader(f): - uttid, txt, metadata = row[:3] - # remove the gs bucket name and prefix - uttid = '/'.join(uttid.split('/')[3:]) - trans[uttid] = txt - return trans - - -def _LoadTranscriptionsFromFile(): - trans = {} - with tf.io.gfile.GFile(FLAGS.transcripts_filepath, 'r') as f: - for line in f.readlines(): - uttid, txt = line.strip('\n').split(' ', 1) - trans[uttid] = txt - return trans - - -def _MakeLogMelFromTensorflowBuiltin(tf_wav_bytes): - sample_rate, audio = audio_lib.DecodeWav(tf_wav_bytes) - static_sample_rate = 16000 - with tf.control_dependencies( - [tf.assert_equal(sample_rate, static_sample_rate)]): - log_mel = audio_lib.AudioToMfcc(static_sample_rate, audio, 25, 25, 40) - return log_mel - - -def _OpenSubShards(): - tf.logging.info('Shards: %d to %d', FLAGS.output_range_begin, - FLAGS.output_range_end) - recordio_writers = [] - for s in range(FLAGS.output_range_begin, FLAGS.output_range_end): - filepath = FLAGS.output_template % (s, FLAGS.num_output_shards) - tf.logging.info('Opening output shard: %s', filepath) - recordio_writers += [tf.python_io.TFRecordWriter(filepath)] - return recordio_writers - - -def _CloseSubShards(files): - for f in files: - f.close() - - -def _SelectRandomShard(files): - subshard = random.randint(0, len(files) - 1) - return files[subshard] - - -def _CreateAsrFeatures(): - # First pass: extract transcription files. - if False: #os.path.exists(FLAGS.transcripts_filepath): - trans = _LoadTranscriptionsFromFile() - else: - tf.logging.info('Running first pass on the fly') - trans = _ReadTranscriptionsFromCSV() - total_utts = len(trans) - tf.logging.info('Total transcripts: %d', len(trans)) - tf_bytes = tf.placeholder(dtype=tf.string) - log_mel = audio_lib.ExtractLogMelFeatures(tf_bytes) - # Second pass: transcode the flac. - file_obj = tf.io.gfile.GFile(FLAGS.input_tarball, mode='rb') - tar = tarfile.open(fileobj=file_obj, mode='r:gz') - n = 0 - recordio_writers = _OpenSubShards() - tfconf = tf.config_pb2.ConfigProto() - tfconf.gpu_options.allow_growth = True - with tf.Session(config=tfconf) as sess: - for tarinfo in tar: - # We can actually decode essentially any audio format, but we - # want to avoid non-audio data. Thus, this condition. - if not (tarinfo.name.endswith('.flac') or tarinfo.name.endswith('.wav') or - tarinfo.name.endswith('.mp3')): - continue - n += 1 - if n % FLAGS.num_shards != FLAGS.shard_id: - continue - f = tar.extractfile(tarinfo) - fmt = tarinfo.name.split('.')[-1] - uttid = tarinfo.name - audio_bytes = f.read() - f.close() - try: - wav_bytes = audio_lib.DecodeToWav(audio_bytes, fmt) - frames = sess.run(log_mel, feed_dict={tf_bytes: wav_bytes}) - except Exception as e: - # raise - trans.pop(uttid) - tf.logging.info(f'{uttid} FAILED featurization') - continue - assert uttid in trans, uttid - num_words = len(trans[uttid]) - tf.logging.info('utt[%d]: %s [%d frames, %d chars]', n, uttid, - frames.shape[1], num_words) - ex = _MakeTfExample(uttid, frames, trans[uttid]) - outf = _SelectRandomShard(recordio_writers) - outf.write(ex.SerializeToString()) - tar.close() - file_obj.close() - _CloseSubShards(recordio_writers) - tf.logging.info(f'Processed {len(trans)} / {total_utts}') - - -def main(_): - tf.logging.set_verbosity(tf.logging.INFO) - if FLAGS.dump_transcripts: - assert False, "dump_transcripts option isn't supported. TODO: Remove." - elif FLAGS.generate_tfrecords: - _CreateAsrFeatures() - else: - tf.logging.error( - 'Nothing to do! Use --dump_transcripts or --generate_tfrecords') - - -if __name__ == '__main__': - tf.disable_eager_execution() - tf.app.run(main) diff --git a/lingvo/tools/generate_proto_def.cc b/lingvo/tools/generate_proto_def.cc deleted file mode 100644 index 8ca2e8d7..00000000 --- a/lingvo/tools/generate_proto_def.cc +++ /dev/null @@ -1,69 +0,0 @@ -/* Copyright 2018 The TensorFlow Authors. All Rights Reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -==============================================================================*/ - -#include -#include -#include -#include - -#include "google/protobuf/descriptor.h" -#include "tensorflow/core/framework/graph.pb.h" -#include "tensorflow/core/framework/types.pb.h" -#include "tensorflow/core/protobuf/meta_graph.pb.h" -#include "tensorflow/core/protobuf/saver.pb.h" - -namespace { -void WriteDotProto(const google::protobuf::FileDescriptor* dot_proto, - const char* output_dirpath) { - std::string output_filepath(output_dirpath); - output_filepath += "/"; - output_filepath += dot_proto->name(); - std::ofstream output_file; - // Assumes the directory tree is already there. - output_file.open(output_filepath); - output_file << dot_proto->DebugString(); - output_file.close(); -} - -void GenerateProtoDef(const google::protobuf::FileDescriptor* dot_proto, - const char* output_dirpath, - std::unordered_set* printed_files) { - if (printed_files->find(dot_proto->name()) != printed_files->end()) { - return; - } - printed_files->insert(dot_proto->name()); - WriteDotProto(dot_proto, output_dirpath); - for (int k = 0; k < dot_proto->dependency_count(); ++k) - GenerateProtoDef(dot_proto->dependency(k), output_dirpath, printed_files); -} - -// Regurgitate the text definitions from binary. -void GenerateProtoDefs(const char* output_dirpath) { - std::unordered_set printed_files; - GenerateProtoDef(tensorflow::GraphDef::descriptor()->file(), output_dirpath, - &printed_files); - GenerateProtoDef(tensorflow::DataType_descriptor()->file(), output_dirpath, - &printed_files); - GenerateProtoDef(tensorflow::SaverDef::descriptor()->file(), output_dirpath, - &printed_files); - GenerateProtoDef(tensorflow::MetaGraphDef::descriptor()->file(), - output_dirpath, &printed_files); -} -} // namespace - -int main(const int argc, const char** argv) { - GenerateProtoDefs(argv[1]); - return 0; -} diff --git a/lingvo/tools/generate_tf_dot_protos.sh b/lingvo/tools/generate_tf_dot_protos.sh deleted file mode 100755 index 333154b2..00000000 --- a/lingvo/tools/generate_tf_dot_protos.sh +++ /dev/null @@ -1,37 +0,0 @@ -#!/bin/bash - -# Copyright 2017 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -# Generates the tensorflow .proto files. This is a thin wrapper around -# generate_proto_def. - -binary=$1 -dest=$2 - -set -e -set -u -set +x - -rm -f ${dest}/.generated - -mkdir -p ${dest}/tensorflow/core/framework -mkdir -p ${dest}/tensorflow/core/protobuf - -${binary} ${dest} - -# genrule requires statically determined outputs, so we package all -# into a single file. -tar -C ${dest} -cf ${dest}/tf_protos.tar tensorflow/core/{framework,protobuf} diff --git a/lingvo/tools/gke_launch.py b/lingvo/tools/gke_launch.py deleted file mode 100644 index 67158651..00000000 --- a/lingvo/tools/gke_launch.py +++ /dev/null @@ -1,498 +0,0 @@ -# Lint as: python3 -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -r"""Launch script for GKE jobs. - -This script generates the GKE deployment configs for TPU training, GPU/CPU -decoding, and TensorBoard jobs. - -It assumes you have: - -a) Copied any input data to GCS. -b) Docker and GKE/GCP tools installed locally. -c) Created the TPU and GPU clusters using `gcloud containers create`. - -This script launches jobs by: - -1) Building a lingvo docker image built from --base_image, copying - the directory pointed to by --build, writing that image to --image - (with an automatically generated date-based tag for versioning). - -2) Identifying the full name of the GKE cluster that each accelerator - job runs in based on --trainer_cell and --decoder_cell - -3) Writing out .yaml configuration files based on --name, --model, --logdir, - that will launch the docker images for each job type. - -Usage looks something like: - - python3 lingvo/tools/gke_launch.py \ - --model=$MODEL \ - --base_image=tensorflow:lingvo_lib_gpu \ - --image=$DOCKER_IMAGE \ - --logdir=$LOGDIR \ - --tpu_type=$TPU_TYPE \ - --trainer_cell=$TPU_CLUSTER_NAME \ - --decoder_cell=$GPU_CLUSTER_NAME \ - --decoder_gpus=1 \ - --gpu_type=$GPU_TYPE \ - --decoder=dev \ - --extra_envs=KITTI_DIR=$GCS_PATH \ - --name=$EXP_NAME \ - --build=$YOUR_CODE_DIR \ - $ACTION $TARGETS - -ACTION specifies whether to start (up), stop (down) or reload the target jobs. -One can also specify "print" to just print out the .yaml configuration files. - -TARGETS specifies whether the action affects all jobs ("all") or just an -individual job ("trainer", "decoder", "tensorboard"). - -See the flags definition below for details on the arguments. -""" - -import datetime -import os -import subprocess -import tempfile - -from absl import app -from absl import flags -import six -import yaml - -FLAGS = flags.FLAGS - -# General flags for all jobs. -flags.DEFINE_string("name", None, "Base name of experiment.") -flags.DEFINE_string("model", None, "Name of registered model.") -flags.DEFINE_string( - "image", None, "Name of docker image to use. If tag is not specified " - "and --build is set, a time-based tag will be used.") -flags.DEFINE_string("logdir", None, "GCS location of base logdir.") -flags.DEFINE_string("gpu_type", "", - "Type of GPU to use, e.g., 'p100' or 'v100'.") - -# Cluster flags -flags.DEFINE_string( - "trainer_cell", "", "Query name for the GKE cluster to use for training. " - "Must uniquely identify among all active clusters.") -flags.DEFINE_string( - "decoder_cell", "", "Query name for the GKE cluster to use for decoding. " - "Must uniquely identify among all active clusters.") - -# TPU training flags. -flags.DEFINE_string("tpu_type", "v3-8", - "Type of TPU to use for training, e.g., 'v3-8' for a 2x2.") - -# Decoder flags. -flags.DEFINE_string("decoder_ram", "24G", - "Amount of CPU memory for the decoder.") -# TODO(vrv): Support launching multiple decoder jobs when this is -# comma-separated. -flags.DEFINE_string("decoder", "dev", "Split(s) of dataset for decoding.") -flags.DEFINE_integer("decoder_gpus", 0, - "Number of GPUs per machine for decoding.") - -# Docker-based flags. -flags.DEFINE_string( - "build", None, "If set, builds the docker image " - "for --image including all code from this directory.") -flags.DEFINE_string( - "base_image", "tensorflow:lingvo_lib", - "Base Lingvo docker image to use. If using GPU, make " - "sure you have built a GPU-enabled image.") -flags.DEFINE_string( - "extra_envs", "", "Extra comma-separated list of key=value environment " - "variables to set when building the docker image.") - - -def _get_or_add(cfg, name): - """Gets cfg[name], or adds 'name' with an empty dict if not present.""" - if name not in cfg: - cfg.update({name: {}}) - return cfg[name] - - -def add_gpu_to_pod(cfg, gpu_type, num_gpus): - """Sets the appropriate GPU fields to cfg. - - Args: - cfg: The YAML-based dictionary to update. - gpu_type: The type of GPU to launch on GKE. - num_gpus: The number of GPUs to launch in the task. - """ - if num_gpus == 0: - return - - if gpu_type == "p100": - gpu_str = "nvidia-tesla-p100" - elif gpu_type == "v100": - gpu_str = "nvidia-tesla-v100" - else: - raise ValueError("Invalid gpu type: ", gpu_type) - cfg["spec"].update( - {"nodeSelector": { - "cloud.google.com/gke-accelerator": gpu_str - }}) - - containers = cfg["spec"]["containers"][0] - resources = _get_or_add(containers, "resources") - resources.update({"limits": {"nvidia.com/gpu": num_gpus}}) - - -def set_pod_cpu_memory(cfg, cpu_memory): - """Sets the amount of CPU memory to request in the container.""" - containers = cfg["spec"]["containers"][0] - resources = _get_or_add(containers, "resources") - resources.update({"requests": {"memory": cpu_memory}}) - - -def decoder_template(job_name, model, image, logdir, decoder_type, - decoder_gpus): - """Constructs the base yaml config for the decoder.""" - name = six.ensure_str(job_name) + ".decoder" - container_name = name.replace(".", "-") - job = "decoder_" + six.ensure_str(decoder_type) - return """ -apiVersion: v1 -kind: Pod -metadata: - name: {job_name} -spec: - restartPolicy: Never - containers: - - name: {container_name} - image: {image} - command: ["/usr/bin/python3"] - args: ["-m", "lingvo.trainer", "--mode=sync", "--alsologtostderr", "--model={model}", "--logdir={logdir}", "--job={job}", "--decoder_gpus={decoder_gpus}", "--decoder_replicas=1", "--cluster_spec={job}=localhost:0"] - """.format( - job_name=name, - container_name=container_name, - image=image, - model=model, - logdir=logdir, - job=job, - decoder_gpus=decoder_gpus) - - -def _tpu_resource(tpu_type): - version, num_chips = six.ensure_str(tpu_type).split("-") - return "cloud-tpus.google.com/{}: {}".format(version, num_chips) - - -def tpu_training_template(job_name, model, image, logdir, tpu_type): - """Constructs the base yaml config for the TPU trainer.""" - name = six.ensure_str(job_name) + ".trainer" - container_name = name.replace(".", "-") - tpu_string = _tpu_resource(tpu_type) - return """apiVersion: batch/v1 -kind: Job -metadata: - name: {name} -spec: - template: - metadata: - annotations: - tf-version.cloud-tpus.google.com: "nightly" - spec: - restartPolicy: Never - containers: - - name: {container_name} - image: {image} - command: ["/usr/bin/python3"] - args: ["-m", "lingvo.trainer", "--mode=sync", "--alsologtostderr", "--model={model}", "--logdir={logdir}", "--tpu=$(KUBE_GOOGLE_CLOUD_TPU_ENDPOINTS)"] - resources: - limits: - {tpu_string} - """.format( - name=name, - container_name=container_name, - image=image, - model=model, - logdir=logdir, - tpu_string=tpu_string) - - -def tensorboard_template(job_name, logdir, port): - """Constructs the tensorboard YAML template.""" - job_name = six.ensure_str(job_name) + ".tensorboard" - container_name = job_name.replace(".", "-") - - print("To poll for tensorboard address, run: $ kubectl get service %s -w" % - (container_name + "-service")) - return """ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: {job_name} -spec: - replicas: 1 - selector: - matchLabels: - name: {job_name} - template: - metadata: - labels: - name: {job_name} - spec: - restartPolicy: Always - containers: - - name: {container_name} - image: gcr.io/tensorflow/tpu-util:r1.11 - command: - - tensorboard - - --logdir=$(MODEL_BUCKET) - env: - - name: MODEL_BUCKET - value: {logdir} - ports: - - containerPort: {port} ---- -apiVersion: v1 -kind: Service -metadata: - name: {container_name}-service -spec: - type: LoadBalancer - selector: - name: {job_name} - ports: - - port: {port} - targetPort: {port} -""".format( - job_name=job_name, container_name=container_name, logdir=logdir, port=port) - - -def build_docker_image(image, base_image, code_directory, extra_envs): - """Build a docker image and push it to the location specified by image. - - Args: - image: String name of tag to use, e.g., 'gcr.io/foo/bar:version' - base_image: String name of base lingvo image to build from. - code_directory: Location of directory whose contents will be copied into the - image. - extra_envs: A comma-separated list of key=value environment variables to be - built into the docker. - """ - preamble = [ - "FROM %s AS lingvo" % base_image, - ] - envs = [ - "ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/kubernetes/bin/nvidia/lib64", - "ENV PATH=${PATH}:/home/kubernetes/bin/nvidia/bin", - ] - for env_pairs in six.ensure_str(extra_envs).split(","): - envs += ["ENV %s" % env_pairs] - - copy_code = ["WORKDIR /tmp/lingvo", "COPY . ."] - - gpu_docker_file = preamble + envs + copy_code - tmp_dockerfile = tempfile.mkstemp(suffix=".dockerfile")[1] - with open(tmp_dockerfile, "w") as f: - f.write("\n".join(gpu_docker_file)) - print("Writing Dockerfile to", tmp_dockerfile) - - os.system("docker build --tag %s --no-cache -f- %s < %s " % - (image, code_directory, tmp_dockerfile)) - os.system("docker push %s" % image) - - -def get_gke_cluster(gke_cluster_spec): - """Get the full name of the GKE cluster given shorthand `gke_cluster_spec`. - - For example, gcloud container cluster list produces: - - NAME LOCATION ... - p100-europe-west4-a-nh16 europe-west4-a ... - test-df-europe europe-west4-a ... - - Then a gke_cluster_spec of 'p100' or 'df' will produce the fully-qualified - cluster. - - A gke_cluster_spec of 'europe' will raise a ValueError because there are two - active clusters that have the string 'europe' in them. - - Args: - gke_cluster_spec: A string specifying a filter on active clusters. - - Returns: - The fully qualified GKE cluster name, or None if not found. - - Raises: - ValueError: If gke_cluster_spec does not uniquely identify an - active cluster. - """ - if not gke_cluster_spec: - return "" - - active_clusters = subprocess.check_output( - ["gcloud", "container", "clusters", "list"]) - cluster_names = [ - c.split(b" ")[0] for c in active_clusters.split(b"\n")[1:] if c - ] - filtered = [ - c for c in cluster_names if six.ensure_binary(gke_cluster_spec) in c - ] - - if len(filtered) > 1: - raise ValueError("Cluster filter was not precise enough.") - - if not filtered: - return None - - result = subprocess.check_output(["kubectl", "config", "view"]) - result = result.split(b"\n") - clusters = [ - r.lstrip() for r in result if " cluster: " in r.decode("utf-8") - ] - cluster = [r for r in clusters if filtered[0] in r] - assert len(cluster) == 1 - cluster = cluster[0].split(b" ")[1] - return cluster - - -# Not yet used, but illustrates one what needs to do. -def create_tpu_cluster(cluster_name, zone): - _ = subprocess.check_output([ - "gcloud", "container", "clusters", "create", cluster_name, - "--cluster_version=1.13", "--scopes=cloud-platform,gke-default", - "--enable-ip-alias", "--enable-tpu", "--async", - "--zone=%s" % zone - ]) - - -def validate_args(argv): - """Validates the input arguments. Raises a UsageError if invalid.""" - valid_commands = ["up", "down", "print", "reload"] - error = "" - if len(argv) < 2: - error = ("Command not provided. " - "Command must be one of %s" % valid_commands) - elif argv[1] not in valid_commands: - error = "Command must be one of %s" % valid_commands - - if len(argv) > 3: - print("Too many arguments.") - - if len(argv) == 3: - valid_targets = ["trainer", "decoder", "tensorboard", "all"] - if argv[2] not in valid_targets: - error = "Target must be one of %s" % valid_targets - - if error: - raise app.UsageError(error) - - -def main(argv): - validate_args(argv) - action = argv[1] - target = argv[2] if len(argv) == 3 else "all" - - if target == "all": - targets = ["trainer", "decoder", "tensorboard"] - else: - targets = [target] - - image = FLAGS.image - - if action == "reload": - actions = ["down", "up"] - else: - actions = [action] - - # Maybe build. - if "up" in actions and FLAGS.build: - # If image does not specify a tag, create a temporary one. - if ":" not in image: - image += ":%s" % datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S") - - build_docker_image(image, FLAGS.base_image, FLAGS.build, FLAGS.extra_envs) - - # Write out the YAML deployment files to a temporary directory - # for post-hoc inspection. - root = tempfile.mkdtemp(prefix=FLAGS.name) - print("Writing out yaml configs to %s" % root) - - action_to_cmd = {"up": "create", "down": "delete"} - - # Start or stop the jobs as requested. - if "tensorboard" in targets: - # Create tensorboard config. Template consists of two yaml configs, so skip - # parsing. - tb_tmpl = tensorboard_template(FLAGS.name, FLAGS.logdir, 6006) - tb_path = os.path.join(root, "tensorboard.yaml") - with open(tb_path, "w") as f: - f.write(tb_tmpl) - if "print" in actions: - print("TB yaml: %s" % tb_tmpl) - else: - for action in actions: - cmd = "kubectl %s -f %s" % (action_to_cmd[action], tb_path) - print("Running: %s" % cmd) - os.system(cmd) - - if "trainer" in targets: - tpu_cluster = get_gke_cluster(FLAGS.trainer_cell) - assert tpu_cluster - tpu_tmpl = tpu_training_template(FLAGS.name, FLAGS.model, image, - FLAGS.logdir, FLAGS.tpu_type) - # Check that it can be loaded. - tpu_cfg = yaml.safe_load(tpu_tmpl) - - trainer_path = os.path.join(root, "tpu_train.yaml") - loaded = yaml.safe_dump(tpu_cfg) - with open(trainer_path, "w") as f: - f.write(six.ensure_str(loaded)) - - if "print" in actions: - print("\n\nTPU yaml: \n%s" % loaded) - else: - for action in actions: - cmd = "kubectl %s -f %s --cluster %s" % (action_to_cmd[action], - trainer_path, tpu_cluster) - print("Running: %s" % cmd) - os.system(cmd) - - if "decoder" in targets: - decoder_cluster = get_gke_cluster(FLAGS.decoder_cell) - decoder_tmpl = decoder_template(FLAGS.name, FLAGS.model, image, - FLAGS.logdir, FLAGS.decoder, - FLAGS.decoder_gpus) - - # Parse into YAML - decoder_cfg = yaml.safe_load(decoder_tmpl) - # Set decoder config values. - add_gpu_to_pod(decoder_cfg, FLAGS.gpu_type, FLAGS.decoder_gpus) - set_pod_cpu_memory(decoder_cfg, FLAGS.decoder_ram) - - assert decoder_cluster - decoder_path = os.path.join(root, "decoder.yaml") - with open(decoder_path, "w") as f: - f.write(six.ensure_str(yaml.safe_dump(decoder_cfg))) - - if "print" in actions: - print("Decoder yaml: \n%s" % yaml.safe_dump(decoder_cfg)) - else: - for action in actions: - cmd = "kubectl %s -f %s --cluster %s" % (action_to_cmd[action], - decoder_path, decoder_cluster) - print("Running: %s" % cmd) - os.system(cmd) - - -if __name__ == "__main__": - flags.mark_flags_as_required(["name", "model", "image", "logdir"]) - app.run(main) diff --git a/lingvo/tools/keras2ckpt.py b/lingvo/tools/keras2ckpt.py deleted file mode 100644 index e52a324c..00000000 --- a/lingvo/tools/keras2ckpt.py +++ /dev/null @@ -1,63 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Converts a keras dataset into a tf checkpoint. - -E.g. - -.. code-block:: bash - - $ bazel run lingvo/tools:keras2ckpt -- --dataset=mnist -""" - -import os -import lingvo.compat as tf -from tensorflow.python.ops import io_ops - -FLAGS = tf.flags.FLAGS - -tf.flags.DEFINE_string("dataset", "", "The dataset name.") -tf.flags.DEFINE_string("out", "", "The output checkpoint path prefix.") - - -def main(argv): - del argv # Unused. - - dataset = getattr(tf.keras.datasets, FLAGS.dataset) - (x_train, y_train), (x_test, y_test) = dataset.load_data() - - def wrap(val): - dtype = tf.as_dtype(val.dtype) - assert dtype != tf.string # tf.string is not supported by py_func. - return tf.py_func(lambda: val, [], dtype) - - out_prefix = FLAGS.out or os.path.join("/tmp", FLAGS.dataset, FLAGS.dataset) - tf.logging.info("Save %s dataset to %s ckpt." % - (FLAGS.dataset, out_prefix)) - - with tf.Session() as sess: - sess.run( - io_ops.save_v2( - prefix=out_prefix, - tensor_names=["x_train", "y_train", "x_test", "y_test"], - shape_and_slices=[""] * 4, - tensors=[wrap(x_train), - wrap(y_train), - wrap(x_test), - wrap(y_test)])) - - -if __name__ == "__main__": - tf.app.run(main) diff --git a/lingvo/tools/print_tf_records.py b/lingvo/tools/print_tf_records.py deleted file mode 100644 index 6f53c215..00000000 --- a/lingvo/tools/print_tf_records.py +++ /dev/null @@ -1,114 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Debug print tf records in text format.""" - -import lingvo.compat as tf -import six - -tf.flags.DEFINE_string('input_filepattern', '', - 'File pattern of binary tfrecord files.') -tf.flags.DEFINE_string('input_format', 'tf.Example', - 'Input format: only "tf.Example" supported for now.') -tf.flags.DEFINE_integer('skip_first_n', 0, 'Skip first records.') -tf.flags.DEFINE_integer('print_only_n', -1, - 'Only print a certain number of records.') -tf.flags.DEFINE_bool('abbreviated', True, 'Print in abbreviated format.') -tf.flags.DEFINE_bool('bytes_as_utf8', True, - 'Print byte strings as UTF-8 strings') -tf.flags.DEFINE_bool('count_only', False, - 'Don\'t print, just count number of entries') - -FLAGS = tf.flags.FLAGS - - -def _ListDebugString(values, to_string=str): - if len(values) <= 8: - return repr(values) - first_values = [to_string(v) for v in values[0:6]] - last_values = [to_string(v) for v in values[-2:]] - return '[' + ' '.join(first_values + ['...'] + last_values) + ']' - - -def _CustomShortDebugString(tf_example): - text = [] - for name, value in sorted(tf_example.features.feature.items()): - if value.HasField('bytes_list'): - if FLAGS.bytes_as_utf8: - utf8_values = [ - six.ensure_text(v, 'utf-8') for v in value.bytes_list.value - ] - value_string = _ListDebugString(utf8_values) - else: - value_string = _ListDebugString(value.bytes_list.value) - elif value.HasField('float_list'): - value_string = _ListDebugString(value.float_list.value) - elif value.HasField('int64_list'): - value_string = _ListDebugString(value.int64_list.value, to_string=repr) - text += ['%s: %s' % (name, value_string)] - return '\n'.join(text) - - -def _PrintHeader(tf_example): - """Prints table of contents.""" - # Typically, tf.Examples have the same features. - tf.logging.info('==== FEATURES ====') - for name, value in sorted(tf_example.features.feature.items()): - type_string = '' - if value.HasField('bytes_list'): - type_string = 'bytes' - elif value.HasField('float_list'): - type_string = 'float' - elif value.HasField('int64_list'): - type_string = 'int64' - tf.logging.info('%s: [%s]', name, type_string) - tf.logging.info('====') - - -def _PrintFiles(): - entry = 0 - for filepath in tf.io.gfile.glob(FLAGS.input_filepattern): - records = tf.compat.v1.io.tf_record_iterator(filepath) - for serialized in records: - if entry < FLAGS.skip_first_n: - entry += 1 - continue - if FLAGS.print_only_n >= 0 and (entry - FLAGS.skip_first_n > - FLAGS.print_only_n): - break - if FLAGS.count_only: - entry += 1 - if (entry % 100000) == 0: - tf.logging.info('Counted %d entries so far...', entry) - continue - assert FLAGS.input_format == 'tf.Example' - ex = tf.train.Example() - ex.ParseFromString(serialized) - if entry == FLAGS.skip_first_n: - _PrintHeader(ex) - text_format = _CustomShortDebugString(ex) if FLAGS.abbreviated else str( - ex) - tf.logging.info('== Record [%d]\n%s', entry, text_format) - entry += 1 - tf.logging.info('== Total entries: %d', entry) - - -def main(_): - tf.logging.set_verbosity(tf.logging.INFO) - _PrintFiles() - - -if __name__ == '__main__': - tf.app.run(main) diff --git a/lingvo/tools/testdata/BUILD b/lingvo/tools/testdata/BUILD deleted file mode 100644 index 3189e12b..00000000 --- a/lingvo/tools/testdata/BUILD +++ /dev/null @@ -1,12 +0,0 @@ -package( - default_visibility = ["//visibility:public"], -) - -licenses(["notice"]) # Apache 2.0 - -filegroup( - name = "audio_data", - srcs = glob(include = [ - "gan_or_vae.*", - ]), -) diff --git a/lingvo/tools/testdata/gan_or_vae.16k.wav b/lingvo/tools/testdata/gan_or_vae.16k.wav deleted file mode 100644 index c960aefd..00000000 Binary files a/lingvo/tools/testdata/gan_or_vae.16k.wav and /dev/null differ diff --git a/lingvo/tools/testdata/gan_or_vae.flac b/lingvo/tools/testdata/gan_or_vae.flac deleted file mode 100644 index b04478b9..00000000 Binary files a/lingvo/tools/testdata/gan_or_vae.flac and /dev/null differ diff --git a/lingvo/tools/testdata/gan_or_vae.wav b/lingvo/tools/testdata/gan_or_vae.wav deleted file mode 100644 index 713ff7be..00000000 Binary files a/lingvo/tools/testdata/gan_or_vae.wav and /dev/null differ diff --git a/lingvo/tools/wpm_encode_file.py b/lingvo/tools/wpm_encode_file.py deleted file mode 100644 index 9833a171..00000000 --- a/lingvo/tools/wpm_encode_file.py +++ /dev/null @@ -1,139 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Encode file using the wpm_encoder.""" - -import lingvo.compat as tf -from lingvo.core import wpm_encoder -import numpy as np -import six -from six import text_type - -tf.flags.DEFINE_string( - 'source_filepaths', '', - 'Comma-separated list of source files. Must align with target') -tf.flags.DEFINE_string('target_filepaths', '', - 'Comma-separated list of target files.') -tf.flags.DEFINE_string('output_filepath', '', 'File of tfrecords.') -tf.flags.DEFINE_string('wpm_filepath', '', 'The wordpiece vocabulary file.') -tf.flags.DEFINE_integer('num_shards', -1, 'Total number of shards.') -tf.flags.DEFINE_integer('shard_id', -1, 'This shard id (0-based).') -tf.flags.DEFINE_integer( - 'max_len', 0, - 'Drop sentence if src/tgt tokens exceed max length, counting and . ' - 'Only use during training. A value of 0 does not filter.') - -FLAGS = tf.flags.FLAGS - - -def _MakeBytesFeature(unicode_array): - value = [tf.compat.as_bytes(w) for w in unicode_array] - return tf.train.Feature(bytes_list=tf.train.BytesList(value=value)) - - -def _MakeInt64Feature(value): - return tf.train.Feature(int64_list=tf.train.Int64List(value=value)) - - -def _MakeFloatFeature(value): - return tf.train.Feature(float_list=tf.train.FloatList(value=value)) - - -def _AssertTextFormat(text): - assert not text.startswith('') - assert not text.endswith('') - assert not text.startswith('') - assert not text.endswith('') - - -def _MakeTfExample(enc, src_i, src_s, tgt_i, tgt_s): - """Creates TfExample from the encoded results.""" - src_i = list(src_i) + [enc.sentence_end_id] - src_s = list(src_s) + [enc.sentence_end_string] - if FLAGS.max_len > 0 and len(src_i) > FLAGS.max_len: - return None - tgt_l = list(tgt_i) + [enc.sentence_end_id] - tgt_i = [enc.sentence_start_id] + list(tgt_i) - tgt_s = [enc.sentence_start_string] + list(tgt_s) - if FLAGS.max_len > 0 and len(tgt_i) > FLAGS.max_len: - return None - feature = { - 'source_id': _MakeInt64Feature(src_i), - 'source_padding': _MakeFloatFeature(np.zeros_like(src_i)), - 'source_word': _MakeBytesFeature(src_s), - 'target_id': _MakeInt64Feature(tgt_i), - 'target_padding': _MakeFloatFeature(np.zeros_like(tgt_i)), - 'target_word': _MakeBytesFeature(tgt_s), - 'target_label': _MakeInt64Feature(tgt_l), - 'target_weight': _MakeFloatFeature(np.ones_like(tgt_l)), - 'natural_order': _MakeInt64Feature([1]), - } - return tf.train.Example(features=tf.train.Features(feature=feature)) - - -def _Preprocess(text): - if not isinstance(text, text_type): - text = six.ensure_text(text, 'utf-8') - return text.strip().replace(' ', '') - - -def _RunEncoding(): - sess = tf.Session() - enc = wpm_encoder.WpmEncoder(FLAGS.wpm_filepath) - src_txt_placeholder = tf.placeholder(tf.string, []) - src_encode_op = enc.Encode(src_txt_placeholder) - tgt_txt_placeholder = tf.placeholder(tf.string, []) - tgt_encode_op = enc.Encode(tgt_txt_placeholder) - pairs = list( - zip(FLAGS.source_filepaths.split(','), FLAGS.target_filepaths.split(','))) - with tf.python_io.TFRecordWriter(FLAGS.output_filepath) as outf: - n = 0 - for p in pairs: - with tf.io.gfile.GFile(p[0], 'r') as sourcef: - with tf.io.gfile.GFile(p[1], 'r') as targetf: - for textp in zip(sourcef.readlines(), targetf.readlines()): - n += 1 - if n % 10000 == 0: - tf.logging.info('Watermark[%d]: %d', FLAGS.shard_id, n) - if n % FLAGS.num_shards != FLAGS.shard_id: - continue - source_text = _Preprocess(textp[0]) - target_text = _Preprocess(textp[1]) - # By convention: - # * source always ends in , never starts with . - # * target never ends in , always starts with . - _AssertTextFormat(source_text) - _AssertTextFormat(target_text) - ((src_i, src_s), (tgt_i, tgt_s)) = sess.run( - [src_encode_op, tgt_encode_op], - feed_dict={ - src_txt_placeholder: source_text, - tgt_txt_placeholder: target_text - }, - ) - ex = _MakeTfExample(enc, src_i, src_s, tgt_i, tgt_s) - if not ex: # Too long. - continue - encoded = ex.SerializeToString() - outf.write(encoded) - - -def main(_): - tf.logging.set_verbosity(tf.logging.INFO) - _RunEncoding() - - -if __name__ == '__main__': - tf.app.run(main) diff --git a/lingvo/trainer.py b/lingvo/trainer.py deleted file mode 100644 index 45ebc41e..00000000 --- a/lingvo/trainer.py +++ /dev/null @@ -1,1882 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -# pylint: disable=line-too-long -"""Trainer. - -To run locally: - -.. code-block:: bash - - $ bazel build -c opt //lingvo:trainer - $ bazel-bin/lingvo/trainer --logtostderr \ - --model=image.mnist.LeNet5 --mode=sync --logdir=/tmp/lenet5 \ - --run_locally=cpu - -To use GPU, add `--config=cuda` to build command and set `--run_locally=gpu`. -""" -# pylint: enable=line-too-long -import os -import re -import sys -import threading -import time - -from lingvo import base_trial -from lingvo import datasets -from lingvo import executor -from lingvo import model_imports -from lingvo import model_registry -import lingvo.compat as tf -from lingvo.core import base_model -from lingvo.core import base_model_params -from lingvo.core import checkpointer -from lingvo.core import cluster_factory -from lingvo.core import inference_graph_exporter -from lingvo.core import metrics -from lingvo.core import py_utils -from lingvo.core import summary_utils -import numpy as np - -from lingvo import base_runner - -# pylint:disable=g-direct-tensorflow-import -from tensorflow.core.protobuf.tpu import compilation_result_pb2 as tpu_compilation_result -from tensorflow.python.tpu import device_assignment as device_assignment_lib -from tensorflow.python.tpu import tpu -from tensorflow.python.tpu import tpu_function -from tensorflow.python.tpu import training_loop as tpu_training_loop -from tensorflow.python.tpu.ops import tpu_ops -# pylint:enable=g-direct-tensorflow-import - -tf.flags.DEFINE_string( - 'model', None, 'Name of the model class to train.' - 'Must be a model defined in the model_registry.') -tf.flags.DEFINE_string( - 'model_task_name', '', 'For multitask models: ' - 'select task to train/evaluate/decode. ' - 'Empty means to sample a task (training only).') -tf.flags.DEFINE_string('logdir', '', 'Log directory.') -tf.flags.DEFINE_bool( - 'interactive', False, - 'If True, enter interactive IPython for the controller job.') - -tf.flags.DEFINE_string( - 'run_locally', '', - 'Can be empty, cpu, or gpu. If not empty, ignores cluster configuration ' - 'flags and runs controller and trainer in a single local process.') - -tf.flags.DEFINE_string('tf_master', '', 'TF runtime.') -tf.flags.DEFINE_string( - 'cluster_spec', '', 'A tf.train.ClusterSpec to override the master. ' - 'The dict is specified as: job=host1:port1,host2:port2,' - 'host3:port3@job2=host3:port4,...') - -tf.flags.DEFINE_string( - 'mode', 'async', 'How this trainer binary is used. ' - 'async: used in an async training setup; ' - 'sync: used in a sync training setup; ' - 'shell: an interactive shell for development; ' - 'inspect_evaler: print evaler dataset names; ' - 'inspect_decoder: print decoder dataset names; ' - 'write_inference_graph: write inference graphs to logdir.') -tf.flags.DEFINE_string('job', '', 'trainer/controller/eval, etc.') -tf.flags.DEFINE_integer('task', 0, 'Task id within the job.') - -tf.flags.DEFINE_string('controller_job', '/job:controller', 'Job name.') -tf.flags.DEFINE_integer('controller_gpus', 0, 'Number of controller GPUs.') - -tf.flags.DEFINE_string('worker_job', '/job:trainer', 'Job name.') -tf.flags.DEFINE_integer('worker_replicas', 1, 'Number of replicas.') -tf.flags.DEFINE_integer('worker_gpus', 0, 'Number of gpus to use per replica.') -tf.flags.DEFINE_integer('worker_tpus', 0, 'Number of tpus to use per replica.') -tf.flags.DEFINE_integer('worker_num_tpu_hosts', 0, 'Number of tpu hosts.') -tf.flags.DEFINE_integer('worker_split_size', 1, - 'Number of devices for one split.') - -tf.flags.DEFINE_string('ps_job', '/job:ps', 'Job name') -tf.flags.DEFINE_integer('ps_replicas', 1, 'Number of replicas.') -tf.flags.DEFINE_integer('ps_gpus', 0, 'Number of gpus to use per replica.') - -tf.flags.DEFINE_string('input_job', '/job:input', 'Job name') -tf.flags.DEFINE_integer('input_replicas', 0, 'Number of replicas.') -tf.flags.DEFINE_string( - 'input_targets', '', 'Target network addresses for the ' - 'input job. E.g., a single ip:port, or a list of ' - 'comma-separated grpc://ip:port, etc.') - -tf.flags.DEFINE_string('evaler_job', '/job:evaler', 'Job name') -tf.flags.DEFINE_integer('evaler_replicas', 0, 'Number of replicas.') -tf.flags.DEFINE_integer('evaler_gpus', 0, 'Number of gpus to use per replica.') - -tf.flags.DEFINE_string('decoder_job', '/job:decoder', 'Job name') -tf.flags.DEFINE_integer('decoder_replicas', 0, 'Number of replicas.') -tf.flags.DEFINE_integer('decoder_gpus', 0, 'Number of gpus to use per replica.') -tf.flags.DEFINE_integer('decoder_tpus', 0, 'Number of tpus to use per replica.') - -tf.flags.DEFINE_integer( - 'inference_graph_random_seed', None, - 'Random seed to fix when exporting inference graph. ' - 'Not fixed when set to None.') -tf.flags.DEFINE_string( - 'inference_graph_filename', None, - 'Output inference graph filename. If unspecified, output two inference ' - 'graphs, one for CPU and one for TPU using the default settings.') -tf.flags.DEFINE_string( - 'inference_graph_device', None, - 'Type of device the output inference graph is for. This flag is applicable ' - 'only when FLAGS.inference_graph_filename is specified.') - -tf.flags.DEFINE_bool( - 'evaler_in_same_address_as_controller', False, - 'Whether or not evaler is in the same address space as ' - 'controller. This flag is meant for unittest only.') - -tf.flags.DEFINE_string( - 'vizier_reporting_job', 'evaler', - 'Job responsible for reporting metrics. This specifies a ' - 'job prefix, evaler will match all evaler jobs, while ' - 'evaler_dev and decoder_dev will only match the corresponding ' - 'jobs that are on the dev set.') - -tf.flags.DEFINE_bool( - 'add_summary', None, - 'Whether we should output summaries. The default value "None", enables ' - 'summaries based on the job type.') - - -@tf.flags.validator('vizier_reporting_job') -def _ValidateVizierReportingJob(value): - if value in ['evaler', 'decoder']: - return True - if value.startswith('evaler_') or value.startswith('decoder_'): - return True - raise tf.flags.ValidationError('Invalid value %s for vizier_reporting_job' % - value) - - -tf.flags.DEFINE_integer( - 'enqueue_max_steps', None, 'Max enqueue steps. -1 meaning no limit.' - ' This flag should be set for unit-test only.') - -tf.flags.DEFINE_integer('saver_max_to_keep', None, - 'Maximum number of recent checkpoints to keep.') - -tf.flags.DEFINE_float('saver_keep_checkpoint_every_n_hours', None, - 'How often to keep a checkpoint.') - -tf.flags.DEFINE_bool( - 'checkpoint_in_trainer_tpu', False, - 'Whether to enable checkpointing in TrainerTpu, allowing for ' - 'operation without a separate Controller task.' - 'This flag also disables checkpointing from the Controller, ' - 'but still allows it to write summaries.') - -tf.flags.DEFINE_string( - 'tpu', None, - 'The Cloud TPU on GCP to use for training. This should be either the name ' - 'used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 ' - 'url. If set, other cluster parameters (such as --cluster_spec) will be ' - 'configured automatically with TPUClusterResolver.') -tf.flags.DEFINE_string( - 'gcp_project', None, - 'Project name for the Cloud TPU-enabled project. If not specified, we ' - 'will attempt to automatically detect the GCE project from metadata.') -tf.flags.DEFINE_string( - 'tpu_zone', None, - 'GCE zone where the Cloud TPU is located in. If not specified, we ' - 'will attempt to automatically detect the GCE project from metadata.') - -# Please consider adding model params instead of adding flags. - -FLAGS = tf.flags.FLAGS - - -# useful for debugging. -def _StartShell(local_ns=None): - # An interactive shell is useful for debugging/development. - import IPython # pylint: disable=g-import-not-at-top - - user_ns = {} - if local_ns: - user_ns.update(local_ns) - user_ns.update(globals()) - IPython.start_ipython(argv=[], user_ns=user_ns) - - -class Controller(base_runner.BaseRunner): - """Controller for a training cluster.""" - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self._job_name = 'controller' - assert not self._model_task_name, 'Controller needs all tasks!' - self._control_dir = os.path.join(self._logdir, 'control') - tf.io.gfile.makedirs(self._control_dir) - self._summary_writer = self._CreateSummaryWriter(self._control_dir) - self._checkpoint_in_controller = True - if FLAGS.checkpoint_in_trainer_tpu: - self._checkpoint_in_controller = False - - with self._graph.as_default(), tf.container(self._container_id): - with self._cluster, tf.device(self._cluster.GetPlacer()): - self._model = self.params.Instantiate() - self._params = self._model.params - self._model.ConstructFPropBPropGraph() - self._summary_op = tf.summary.merge_all() - self._initialize_tables = tf.tables_initializer() - self._initialize_local_vars = tf.local_variables_initializer() - self._initialize_global_vars = tf.global_variables_initializer() - self.enqueue_ops = tf.get_collection(py_utils.ENQUEUE_OPS) - if self._checkpoint_in_controller: - self.checkpointer = self._CreateCheckpointer( - self._train_dir, - self._model, - init_op=self._initialize_global_vars) - - self._ExportMetrics(params=self.params) - self._model_analysis, self._total_num_params = summary_utils.ModelAnalysis( - self._model) - py_utils.LogMultiLines('MODEL ANALYSIS', self._model_analysis) - self._WriteToLog(self._model_analysis, self._control_dir, - 'model_analysis.txt') - self._WriteToLog(self.params.ToText(), self._control_dir, 'params.txt') - tf.io.write_graph(self._graph.as_graph_def(), self._control_dir, - 'train.pbtxt') - - def _CreateCheckpointer(self, train_dir, model, init_op=None): - """Wrapper method for override purposes.""" - return checkpointer.Checkpointer(train_dir, model, init_op) - - def Start(self): - self._RunLoop('controller', self._Loop) - - def StartEnqueueOp(self, op): - self._RunLoop( - 'controller/enqueue_op/%s' % op.name, self._LoopEnqueue, loop_args=[op]) - - def _Loop(self): - self._summary_writer.add_graph(self._graph) - with tf.container(self._container_id), self._GetSession() as sess: - if FLAGS.interactive: - # Into interactive debugging mode. - _StartShell(locals()) - return - - # This initializes local tables - sess.run(self._initialize_tables) - # This initializes local variables. - sess.run(self._initialize_local_vars) - - # TODO(zhifengc): Moves these options into params. - tp = self.params.train - summary_interval_steps = tp.summary_interval_steps - save_interval_seconds = tp.save_interval_seconds - next_summary_step = 1 - - if not self._checkpoint_in_controller: - global_step = self._WaitUntilInit(sess) - - while True: - now = time.time() - next_iteration_seconds = now + min( - 10, save_interval_seconds) # 10 seconds or less - - if self._checkpoint_in_controller: - # Init/restore variable if needed. - self.checkpointer.RestoreIfNeeded(sess) - - global_step = sess.run(self._model.global_step) - if self._trial.ShouldStop() or self._ShouldStop(sess, global_step): - tf.logging.info('Training finished.') - if self._checkpoint_in_controller: - self.checkpointer.Save(sess, global_step) - sess.close() - self._DequeueThreadComplete() - return - - if self._checkpoint_in_controller: - # Checkpoint if it's time. - self.checkpointer.MaybeSave(sess, self._model.global_step) - - # Summary. - if self._summary_op is not None and global_step >= next_summary_step: - tf.logging.info('Write summary @%s', global_step) - summary_str = sess.run(self._summary_op) - if isinstance(summary_str, np.ndarray) and summary_str.size == 0: - tf.logging.info('Skipping summary: %s', summary_str) - else: - self._summary_writer.add_summary(summary_str, global_step) - self._SummarizeValue(global_step, 'total_num_params', - self._total_num_params) - next_summary_step = global_step + summary_interval_steps - tf.logging.info('Write summary done: step %d', global_step) - - now = time.time() - if now < next_iteration_seconds: - time.sleep(next_iteration_seconds - now) - - def _SummarizeValue(self, steps, tag, value): - self._summary_writer.add_summary( - metrics.CreateScalarSummary(tag, value), steps) - - -class Trainer(base_runner.BaseRunner): - """Trainer on non-TPU.""" - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self._job_name = 'trainer' - with self._graph.as_default(), tf.container(self._container_id): - with self._cluster, tf.device(self._cluster.GetPlacer()): - self._model = self.params.Instantiate() - self._params = self._model.params - self._model.ConstructFPropBPropGraph() - self._initialize_tables = tf.tables_initializer() - self._initialize_local_vars = tf.local_variables_initializer() - self.enqueue_ops = tf.get_collection(py_utils.ENQUEUE_OPS) - tf.logging.info('Trainer number of enqueue ops: %d', - len(self.enqueue_ops)) - - try: - self._task_probs_summary_writers = [] - for task in self._model.task_schedule.tasks: - path = os.path.join(os.path.join(self._train_dir, task)) - tf.io.gfile.makedirs(path) - self._task_probs_summary_writers.append(self._CreateSummaryWriter(path)) - except AttributeError: - tf.logging.info('AttributeError. Expected for single task models.') - self._task_probs_summary_writers = [] - - self._step_rate_tracker = summary_utils.StepRateTracker() - - # Saves the graph def. - if self.params.cluster.task > 0: - self._summary_writer = None - else: - self._WriteToLog(self.params.ToText(), self._train_dir, - 'trainer_params.txt') - self._summary_writer = self._CreateSummaryWriter(self._train_dir) - tf.io.write_graph(self._graph.as_graph_def(), self._train_dir, - 'train.pbtxt') - worker_id = self.params.cluster.task - self._start_up_delay_steps = (((worker_id + 1) * worker_id / 2) * - self.params.train.start_up_delay_steps) - - def _SummarizeValue(self, steps, tag, value, writer=None): - if writer: - writer.add_summary(metrics.CreateScalarSummary(tag, value), steps) - elif self._summary_writer: - self._summary_writer.add_summary( - metrics.CreateScalarSummary(tag, value), steps) - - def Start(self): - self._RunLoop('trainer', self._Loop) - - def StartEnqueueOp(self, op): - self._RunLoop( - 'trainer/enqueue_op/%s' % op.name, self._LoopEnqueue, loop_args=[op]) - - def _LoopEnqueue(self, op): - # Evaler/Controller jobs may find that the trial is infeasible and report - # done earlier. This is an important check since the trainer may retry - # indefinitely without it. - if self._trial.ShouldStop(): - tf.logging.info('Training skipped (trial requested to stop).') - return - return super()._LoopEnqueue(op) - - def _Loop(self): - # Evaler/Controller jobs may find that the trial is infeasible and report - # done earlier. This is an important check since the trainer may retry - # indefinitely without it. - if self._trial.ShouldStop(): - tf.logging.info('Training skipped (trial requested to stop).') - return - with tf.container( - self._container_id), self._cluster, self._GetSession() as sess: - # This initializes local tables - sess.run(self._initialize_tables) - # This initializes local variables. - sess.run(self._initialize_local_vars) - global_step = self._WaitUntilInit(sess, self._start_up_delay_steps) - - status_interval_steps = 100 - next_status_step = 1 - eval_metrics = None - while True: - if (self._trial.ShouldStopAndMaybeReport(global_step, eval_metrics) or - self._ShouldStop(sess, global_step)): - tf.logging.info('Training finished.') - if self._early_stop: - time.sleep(300) # controller hangs if it doesn't finish first - self._DequeueThreadComplete() - return - - # If a task is explicitly specified, only train that task. - if self._model_task_name: - task = self._model.GetTask(self._model_task_name) - else: - # Note: This is a slightly stale global_step value from the previous - # sess.run() call. - # For multi-task models, `self._model.task_schedule.cur_probs` will - # be updated. - task = self._model.SampleTask(global_step) - if self._task_probs_summary_writers: - for index, prob in enumerate(self._model.task_schedule.cur_probs): - self._SummarizeValue(global_step, 'task_probability', prob, - self._task_probs_summary_writers[index]) - try: - for index, task in enumerate(self._model.tasks): - self._SummarizeValue(global_step, 'task_weight', - sess.run(task.vars.task_weight), - self._task_probs_summary_writers[index]) - except AttributeError: - pass - - (_, eval_metrics, per_example_tensors) = sess.run([ - task.train_op, - task.eval_metrics, - task.per_example_tensors, - ]) - # Explicitly fetch global_step after running train_op. - # TODO(b/151181934): Investigate this behavior further. - task_global_step = sess.run(task.global_step) - task.ProcessFPropResults(sess, task_global_step, eval_metrics, - per_example_tensors) - - global_step = sess.run(self._model.global_step) - step_rate, example_rate, total_examples = ( - self._step_rate_tracker.ComputeStepRate( - global_step, eval_metrics['num_samples_in_batch'][0])) - self._SummarizeValue(global_step, 'global_step/sec', step_rate) - self._SummarizeValue(global_step, 'examples/sec', example_rate) - self._SummarizeValue(global_step, 'total_samples', total_examples) - - msg = 'step:%6d, steps/sec: %0.2f, examples/sec: %0.2f' % ( - global_step, step_rate, example_rate) - for key, (val, _) in sorted(eval_metrics.items()): - msg += ' %s:%.8g' % (key, val) - self._SummarizeValue(global_step, key, val) - if global_step >= next_status_step: - self._SetStatusMessage(msg) - self._ExportMetrics( - # Metrics expects python int, but global_step is numpy.int64. - global_step=int(global_step), - step_rate=step_rate, - example_rate=example_rate) - next_status_step = global_step + status_interval_steps - else: - tf.logging.info(msg) - self._model.ProcessFPropResults(sess, global_step, eval_metrics, - per_example_tensors) - - -class TrainerTpu(base_runner.BaseRunner): - """Trainer on TPU.""" - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self._job_name = 'trainer_tpu' - - # Multiple TPU trainer tasks not tested/implemented. - assert self._cluster.num_replicas == 1 - data_parallelism = self._cluster.num_splits_per_client - assert data_parallelism - num_devices_per_split = self._cluster.num_devices_per_split - tf.logging.info('data_parallelism: %d, num_devices_per_split: %d', - data_parallelism, num_devices_per_split) - - self._steps_per_loop = min(self.params.train.tpu_steps_per_loop, - self.params.train.max_steps) - self._step_rate_tracker = summary_utils.StepRateTracker() - - self._cluster_def = self._cluster.worker_cluster_def - self._compile_op = None - - self._initialized = threading.Event() - - tf.logging.info( - 'Creating TrainerTpu using data parallelism %s ' - 'and %s steps_per_loop', data_parallelism, self._steps_per_loop) - - @py_utils.RetryOnTransientTfError() - def _WaitUntilInitTpu(): - """Wait until the model is ready.""" - try: - # tpu.initialize_system() is called with None as embedding_config, as - # embedding_config is not available yet. Later in _Loop, it is called - # with the correct embedding_config. Since it cannot be called twice in - # the same graph with different embedding_config, we use a dummy_graph - # here. - dummy_graph = tf.Graph() - with dummy_graph.as_default(): - tpu_initialize_system_op = tf.tpu.initialize_system( - embedding_config=None, job=None) - - with self._GetSession(graph=dummy_graph) as sess: - topology = sess.run(tpu_initialize_system_op) - - device_assignment = device_assignment_lib.device_assignment( - topology, - computation_shape=py_utils.ComputationShape(num_devices_per_split, - topology), - num_replicas=data_parallelism) - py_utils.SetTpuDeviceAssignment(device_assignment) - tf.logging.info('device_assignment.core_assignment: %s', - str(device_assignment.core_assignment)) - tf.logging.info('device_assignment.topology.device_coordinates: %s', - str(device_assignment.topology.device_coordinates)) - except py_utils.transient_tf_errors as e: - tf.logging.info('TPU initialization failed: %s', e) - raise - - _WaitUntilInitTpu() - - with self._graph.as_default(), tf.container(self._container_id): - with self._cluster, tf.device(self._cluster.job_spec.name): - with cluster_factory.SetImmediatelyInstantiateVariables(False): - self._model = self.params.Instantiate() - self._task = self._model.GetTask() - self._task.input.InstantiateVariables() - self._task.input.CreateTpuEnqueueOps() - self._eval_metrics = metrics.TpuEvalMetrics() - # Needed due to the AddExtraTheta() reference to global_step when - # instantiating the InputGenerator. - _ = py_utils.GetOrCreateGlobalStepVar() - - def TpuTrainStep(*args): - """Train a shard of a batch on a single TPU core. - - Args: - *args: metrics values from previous steps. - - Returns: - New summed metrics values and a train_op. - """ - self._model.InstantiateVariables() - self._model.ConstructFPropBPropGraph() - self._load_ops = tf.get_collection(py_utils.TPU_EMBEDDING_LOAD_OPS) - self._retrieve_ops = tf.get_collection( - py_utils.TPU_EMBEDDING_RETRIEVE_OPS) - tpu_embedding_collection = tf.get_collection(py_utils.TPU_EMBEDDING) - self._tpu_embedding = ( - tpu_embedding_collection[0] if tpu_embedding_collection else None) - - per_step_eval_metrics = self._eval_metrics.SetMetrics( - self._task.eval_metrics, args) - outfeed_op = self._OutfeedEnqueue(self._task.per_example_tensors) - summed_metrics = [] - assert len(per_step_eval_metrics) == len(args) - with tf.control_dependencies([outfeed_op]): - for x, y in zip(per_step_eval_metrics, args): - summed_metrics.append(x + y) - return summed_metrics + [self._task.train_op] - - @tpu_function.on_device_training_loop - def TpuTrain(): - loop_result = tpu_training_loop.repeat( - self._steps_per_loop, - TpuTrainStep, - inputs=self._eval_metrics.initial_values, - name='train_loop') - # Final metrics are the avg across self._steps_per_loop steps. - return self._eval_metrics.FinalizeMetrics(loop_result) - - self._compile_op, batch_parallel_res = tpu.split_compile_and_shard( - TpuTrain, - num_shards=data_parallelism, - device_assignment=py_utils.GetTpuDeviceAssignment()) - outfeed_dequeue_op = self._OutfeedDequeueLoop( - self._task.per_example_tensors, self._steps_per_loop, - self._cluster.num_splits_per_client) - self._task.input.CreateTpuEmbeddingEnqueueOps() - - def _ConstructPostTrainingLoop(train_loop_op, outfeed_dequeue_op): - """Returns the op for tpu training with tail cpu computation.""" - # Adds a tail computation that is run after the tpu_training loop - # step finishes. This allows us to run certain computation that - # acts on the variable between tpu_train_loop iterations and - # amortizing the cost of the operations. Alternative of running - # tpu.outside_compilation & using tf.cond is expenseive. - with tf.control_dependencies(train_loop_op): - self._model.ConstructPostTrainingLoop() - with tf.control_dependencies([self._task.post_training_loop_op]): - return ([[tf.identity(o) for o in train_loop_op], - outfeed_dequeue_op]) - - # Get metric result from a single replica; they are all same here. - all_tpu_ops = [t[0] for t in batch_parallel_res] - self._tpu_train_ops = ( - _ConstructPostTrainingLoop(all_tpu_ops, outfeed_dequeue_op)) - - self._initialize_local_vars = tf.local_variables_initializer() - self._initialize_global_vars = tf.global_variables_initializer() - self._initialize_tables = tf.tables_initializer() - - if FLAGS.checkpoint_in_trainer_tpu: - self.checkpointer = checkpointer.Checkpointer( - self._train_dir, self._model, init_op=self._initialize_global_vars) - - self.enqueue_ops = self._task.input.tpu_infeed_op - tf.logging.info('Trainer number of enqueue ops: %d', - len(self.enqueue_ops)) - - self._summary_writer = self._CreateSummaryWriter(self._train_dir) - if FLAGS.checkpoint_in_trainer_tpu: - self._model_analysis, self._total_num_params = ( - summary_utils.ModelAnalysis(self._model)) - py_utils.LogMultiLines('MODEL ANALYSIS', self._model_analysis) - self._WriteToLog(self._model_analysis, self._train_dir, - 'model_analysis.txt') - # Saves the graph def. - tf.io.write_graph(self._graph.as_graph_def(), self._train_dir, - 'train.pbtxt') - - # Saves the trainer params. - self._WriteToLog(self.params.ToText(), self._train_dir, - 'trainer_params.txt') - - def _GetSession(self, **kwargs): - return super()._GetSession(cluster_def=self._cluster_def, **kwargs) - - def _OutfeedEnqueue(self, per_example_tensors): - if not per_example_tensors: - return tf.no_op() - per_example_tensors = py_utils.NestedMap(per_example_tensors) - return tpu_ops.outfeed_enqueue_tuple(per_example_tensors.Flatten()) - - def _OutfeedDequeueLoop(self, per_example_tensors, num_loops, num_devices): - """Process all per-example tensor outfeed data for a TPU sess.run. - - Args: - per_example_tensors: dict of key -> tensor as generated by TpuTrainStep. - num_loops: number of times that TpuTrainStep will be executed by TpuTrain. - num_devices: number of TPU cores assigned to this process. - - Returns: - A dict of per-example tensors from the latest TpuTrainStep. - """ - if not per_example_tensors: - return tf.no_op() - - tensor_shapes = [ - py_utils.GetShape(per_example_tensors[key]) - for key in sorted(per_example_tensors) - ] - tensor_types = [ - tf.as_dtype(per_example_tensors[key].dtype) - for key in sorted(per_example_tensors) - ] - - def LoopBody(i, *input_arrays): - """Process outfeed data for a single TpuTrainStep. - - Args: - i: current loop index. - *input_arrays: One tf.TensorArray per outfeed tensor. - - Returns: - i+1 (new index) plus post-write tf.TensorArray handles. - """ - # Outfeed ops execute on each JF node, so they must be located on the - # nodes. - outfeed_devices = [] - device_assignment = py_utils.GetTpuDeviceAssignment() - assert device_assignment - for replica in range(device_assignment.num_replicas): - for core in range(device_assignment.num_cores_per_replica): - with tf.device(device_assignment.host_device(replica, core)): - outfeed_devices.append( - tpu_ops.outfeed_dequeue_tuple( - tensor_types, - tensor_shapes, - device_ordinal=device_assignment.tpu_ordinal(replica, - core))) - offset = i * num_devices - output_arrays = list(input_arrays) - # Each output_array holds a different per-example tensor. We get results - # for each tensor from each TPU for each TpuTrainStep call. - for j in range(len(output_arrays)): - for k in range(len(outfeed_devices)): - output_arrays[j] = output_arrays[j].write(offset + k, - outfeed_devices[k][j]) - - return tuple([i + 1] + output_arrays) - - def LoopCond(i, *output_arrays): - del output_arrays - return i < num_loops - - output_arrays = [ - tf.TensorArray( - tensor_types[i], - size=num_loops * num_devices, - element_shape=tensor_shapes[i]) for i in range(len(tensor_shapes)) - ] - # Loop once for each time that TpuTrainStep runs. - output_arrays = tf.while_loop( - LoopCond, LoopBody, [0] + output_arrays, parallel_iterations=1)[1:] - concatenated_arrays = [array.concat() for array in output_arrays] - return dict(zip(sorted(per_example_tensors), concatenated_arrays)) - - def Start(self): - # Run training. - self._RunLoop('trainer', self._Loop) - - def _InfeedLoop(self, sess): - tf.logging.info('_InfeedLoop start') - for _ in range(self._steps_per_loop): - sess.run(self.enqueue_ops) - - def StartEnqueueOp(self, op): - # When retrieve ops for TPU embedding is present, we use _InfeedLoop above - # instead to make sure enqueue and retrieve does not happen at the same - # time as required by TPU embedding. - # We can remove this by using a tf.while_loop driven infeed op. - if self._retrieve_ops: - return - self._RunLoop( - 'trainer/enqueue_op/%s' % op.name, self._LoopEnqueue, loop_args=[op]) - - def _SummarizeValue(self, steps, tag, value): - self._summary_writer.add_summary( - metrics.CreateScalarSummary(tag, value), steps) - - def _LoopEnqueue(self, op): - # Evaler/Controller jobs may find that the trial is infeasible and report - # done earlier. This is an important check since the trainer may retry - # indefinitely without it. - if self._trial.ShouldStop(): - tf.logging.info('Training skipped (trial requested to stop).') - return - # Wait for _Loop to initialize variables first before attempting to infeed. - tf.logging.info('_LoopEnqueue waiting for _initialized...') - self._initialized.wait() - tf.logging.info('_LoopEnqueue proceeding.') - - # The global step may not be initialized in this thread if the target server - # uses session state isolation (e.g. Cloud TPUs). - sess = self._GetSession() - if FLAGS.checkpoint_in_trainer_tpu: - self.checkpointer.RestoreGlobalStepIfNeeded(sess) - - return super()._LoopEnqueue(op, sess) - - def _Loop(self): - # Evaler/Controller jobs may find that the trial is infeasible and report - # done earlier. This is an important check since the trainer may retry - # indefinitely without it. - if self._trial.ShouldStop(): - tf.logging.info('Training skipped (trial requested to stop).') - self._DequeueThreadComplete() - return - with tf.container( - self._container_id), self._cluster, self._GetSession() as sess: - config_proto = ( - self._tpu_embedding.config_proto - if self._tpu_embedding is not None else None) - sess.run( - tf.tpu.initialize_system(embedding_config=config_proto, job=None)) - sess.run(self._initialize_tables) - sess.run(self._initialize_local_vars) - - if FLAGS.run_locally == 'tpu': - sess.run(self._initialize_global_vars) - - self._SetStatusMessage('Compiling ...') - compilation_result = sess.run(self._compile_op) - comp_result_proto = tpu_compilation_result.CompilationResultProto() - comp_result_proto.ParseFromString(compilation_result) - if comp_result_proto.status_error_message: - tf.logging.fatal('Compilation failed: {}'.format( - comp_result_proto.status_error_message)) - self._SetStatusMessage('Compiling done.') - - if FLAGS.checkpoint_in_trainer_tpu: - # For b/134415393 -- better to initialize to a known state than - # rely on what's in the session on the trainer/TPU worker. - tf.logging.info('TrainerTpu: Force restore or initialize.') - self.checkpointer.Restore(sess, force_reinitialize=True) - - global_step = sess.run(self._model.global_step) - self._initialized.set() - eval_metrics = None - - sess.run(self._load_ops) - while True: - train_steps_start = time.perf_counter() - if FLAGS.checkpoint_in_trainer_tpu: - # Init/restore variable if needed. - self.checkpointer.RestoreIfNeeded(sess) - if self._trial.ShouldStopAndMaybeReport(global_step, eval_metrics): - # Early terminate gracefully by setting a new max step horizon: three - # more TPU steps to ensure that the enqueue ops can gracefully - # terminate as well. - if self._max_steps is None: - self._max_steps = global_step + 3 * self._steps_per_loop - tf.logging.info('Early stopping at step: %d', self._max_steps) - - if self._ShouldStop(sess, global_step): - tf.logging.info('Training finished.') - if FLAGS.checkpoint_in_trainer_tpu: - self.checkpointer.Save(sess, global_step) - self._DequeueThreadComplete() - return - - if self._retrieve_ops: - infeed_loop_thread = threading.Thread( - target=self._InfeedLoop, args=(sess,)) - infeed_loop_thread.start() - - tpu_train_op_start = time.perf_counter() - values, outfeeds = sess.run(self._tpu_train_ops) - tpu_train_op_secs = time.perf_counter() - tpu_train_op_start - - if self._retrieve_ops: - infeed_loop_thread.join() - tf.logging.info('Retrieve params.') - sess.run(self._retrieve_ops) - tf.logging.info('Retrieve params done.') - - self._eval_metrics.PackMetricsValues(values) - eval_metrics = self._eval_metrics.metrics - - # Note: global_step is incremented by self._steps_per_loop by the - # previous sess.run call. - task_global_step = sess.run(self._task.global_step) - global_step = sess.run(self._model.global_step) - - if not self._task.per_example_tensors: - outfeeds = {} - self._task.ProcessFPropResults(sess, task_global_step, eval_metrics, - outfeeds) - self._model.ProcessFPropResults(sess, global_step, eval_metrics, - outfeeds) - - step_rate, example_rate, total_examples = ( - self._step_rate_tracker.ComputeStepRate( - global_step, - eval_metrics['num_samples_in_batch'][0] * self._steps_per_loop)) - self._SummarizeValue(global_step, 'global_step/sec', step_rate) - self._SummarizeValue(global_step, 'examples/sec', example_rate) - self._SummarizeValue(global_step, 'total_samples', total_examples) - if FLAGS.checkpoint_in_trainer_tpu: - self._SummarizeValue(global_step, 'total_num_params', - self._total_num_params) - msg = 'step:%6d, steps/sec: %0.2f, examples/sec: %0.2f' % ( - global_step, step_rate, example_rate) - for key, (val, _) in sorted(eval_metrics.items()): - msg += ' %s:%.8g' % (key, val) - self._SummarizeValue(global_step, key, val) - - self._SetStatusMessage(msg) - - checkpoint_write_secs = 0.0 - if FLAGS.checkpoint_in_trainer_tpu: - checkpoint_write_start = time.perf_counter() - checkpoint_saved = self.checkpointer.MaybeSave( - sess, self._model.global_step) - if checkpoint_saved: - checkpoint_write_secs = time.perf_counter() - checkpoint_write_start - train_steps_secs = time.perf_counter() - train_steps_start - self._ExportMetrics( - # Metrics expects python int, but global_step is numpy.int64. - global_step=int(global_step), - step_rate=step_rate, - example_rate=example_rate, - tpu_train_op_secs=tpu_train_op_secs, - checkpoint_write_secs=checkpoint_write_secs, - total_train_steps_secs=train_steps_secs) - - -class Evaler(base_runner.BaseRunner): - """Evaler.""" - - def __init__(self, eval_type, *args, **kwargs): - super().__init__(*args, **kwargs) - self._job_name = 'evaler_' + eval_type - self._output_name = 'eval_' + eval_type - self.params.cluster.do_eval = True - self._cluster = cluster_factory.Cluster(self.params.cluster) - self._eval_dir = os.path.join(self._logdir, self._output_name) - if self._model_task_name: - self._eval_dir += '_' + str(self._model_task_name) - tf.io.gfile.makedirs(self._eval_dir) - - self._eval_path = None - # Multitask params doesn't have 'task'. - if 'task' in self.params: - self._eval_path = checkpointer.GetSpecificCheckpoint( - self.params.task.eval.load_checkpoint_from) - - self._summary_writer = self._CreateSummaryWriter(self._eval_dir) - self._should_report_metrics = self._job_name.startswith( - FLAGS.vizier_reporting_job) - - with self._graph.as_default(), tf.container(self._container_id): - with self._cluster, tf.device(self._cluster.GetPlacer()): - self._model = self.params.Instantiate() - self._params = self._model.params - self._model.ConstructFPropGraph() - self._task = self._model.GetTask(self._model_task_name) - self._initialize_tables = tf.tables_initializer() - self._initialize_local_vars = tf.local_variables_initializer() - # No queues are allowed for eval models. - self.enqueue_ops = tf.get_collection(py_utils.ENQUEUE_OPS) - assert not self.enqueue_ops - self.checkpointer = self._CreateCheckpointer(self._train_dir, self._model) - - # Saves the graph def. - self._WriteToLog(self.params.ToText(), self._eval_dir, 'params.txt') - if self.params.cluster.task == 0: - tf.io.write_graph(self._graph.as_graph_def(), self._eval_dir, - '%s.pbtxt' % self._output_name) - - def _CreateCheckpointer(self, train_dir, model): - """Wrapper method for override purposes.""" - return checkpointer.Checkpointer(train_dir, model) - - def Start(self): - self._RunLoop(self._job_name, self._Loop) - - def _Loop(self): - """The main loop.""" - with tf.container( - self._container_id), self._cluster, self._GetSession() as sess: - # This initializes local tables - sess.run(self._initialize_tables) - # This initializes local variables. - sess.run(self._initialize_local_vars) - - if self._eval_path: - self._EvalOnce(self._eval_path, sess) - else: - path = None - while True: - path = self._FindNewCheckpoint(path, sess) - if not path or self._EvalOnce(path, sess): - break - - # Maybe evaluate the last checkpoint if we are not given a specific - # checkpoint to evaluate. - if self._eval_path is None: - self.EvalLatestCheckpoint(path) - - if self._should_report_metrics: - tf.logging.info('Reporting trial done.') - self._trial.ReportDone() - tf.logging.info('Evaluation finished.') - - def EvalLatestCheckpoint(self, last_path=None): - """Runs eval once on the latest checkpoint.""" - with tf.container( - self._container_id), self._cluster, self._GetSession() as sess: - # This initializes local tables - sess.run(self._initialize_tables) - # This initializes local variables. - sess.run(self._initialize_local_vars) - path = tf.train.latest_checkpoint(self._train_dir) - if not path: - tf.logging.info('No checkpoint available.') - return - elif path == last_path: - tf.logging.info('Latest checkpoint was already evaluated.') - return - - self._EvalOnce(path, sess) - - def EvalCheckpoint(self, ckpt_id): - with tf.container(self._container_id), self._GetSession() as sess: - # This initializes local tables - sess.run(self._initialize_tables) - # This initializes local variables. - sess.run(self._initialize_local_vars) - path = '{}/ckpt-{:08d}'.format(self._train_dir, ckpt_id) - self._EvalOnce(path, sess) - - def _EvalOnce(self, path, sess): - """Runs evaluation for a batch of samples. - - Args: - path: checkpoint path. - sess: the tf Session. - - Returns: - should_stop. - """ - - if not FLAGS.evaler_in_same_address_as_controller: - self.checkpointer.RestoreFromPath(sess, path) - - global_step = sess.run(py_utils.GetGlobalStep()) - # Check after how many steps checkpoint got saved. - # And decide whether to run an evaluation. - if global_step < self._task.params.eval.start_eval_after: - return False - - if self._task.params.input.resettable: - tf.logging.info('Resetting input_generator.') - self._task.input_generator.Reset(sess) - - metrics_dict = { - name: metrics.AverageMetric() for name in self._task.eval_metrics - } - num_samples_metric = metrics_dict['num_samples_in_batch'] - while (num_samples_metric.total_value < - self._task.params.eval.samples_per_summary): - # NOTE: We intentionally do not let FProp generate summaries by default, - # because evaler calls FProp multiple times for each checkpoint. Multiple - # summaries at the same step is often confusing. Instead, models should - # update eval_metrics and generate aggregate summaries. - ans = sess.run(self._task.eval_metrics) - for name, (value, weight) in ans.items(): - metrics_dict[name].Update(value, weight) - tf.logging.info('Total examples done: %d/%d', - num_samples_metric.total_value, - self._task.params.eval.samples_per_summary) - - # Replace average values with total values for certain metrics. - if 'num_predictions' in metrics_dict: - metrics_dict['num_predictions'].total_weight = 1.0 - if 'num_words' in metrics_dict: - metrics_dict['num_words'].total_weight = 1.0 - - # When we have evaluated so many samples, generate a summary. - self._WriteSummaries( - self._summary_writer, - os.path.basename(self._eval_dir), - global_step, {k: v.Summary(k) for k, v in metrics_dict.items()}, - text_filename=os.path.join(self._eval_dir, - 'score-{:08d}.txt'.format(global_step))) - - should_stop = global_step >= self.params.train.max_steps - if self._should_report_metrics: - tf.logging.info('Reporting eval measure for step %d.' % global_step) - trial_should_stop = self._trial.ReportEvalMeasure(global_step, - metrics_dict, path) - should_stop = should_stop or trial_should_stop - return should_stop - - -def GetDecoderDir(logdir, decoder_type, model_task_name): - if model_task_name: - decoder_dir = '%s_%s' % (decoder_type, model_task_name) - else: - decoder_dir = decoder_type - return os.path.join(logdir, decoder_dir) - - -def _GetCheckpointIdForDecodeOut(ckpt_id_from_file, global_step): - """Retrieve the checkpoint id for the decoder out file. - - Compares the checkpoint id found in the checkpoint file name to global - step. If they diverge, uses the retrieved id and prints a warning. - - Args: - ckpt_id_from_file: Checkpoint Id from the checkpoint file path. - global_step: int specifying the global step of the model. - - Returns: - Checkpoint id as int. - """ - tf.logging.info('Loaded checkpoint is at global step: %d', global_step) - tf.logging.info('Checkpoint id according to checkpoint path: %d', - ckpt_id_from_file) - if global_step != ckpt_id_from_file: - tf.logging.warning( - 'Checkpoint id %d != global step %d. ' - 'Will use checkpoint id from checkpoint file for ' - 'writing decoder output.', ckpt_id_from_file, global_step) - return ckpt_id_from_file - - -class Decoder(base_runner.BaseRunner): - """Decoder.""" - - def __init__(self, decoder_type, *args, **kwargs): - super().__init__(*args, **kwargs) - self._job_name = 'decoder_' + decoder_type - self.params.cluster.do_eval = True - # Here we go! - self._cluster = cluster_factory.Cluster(self.params.cluster) - self._decoder_dir = GetDecoderDir(self._logdir, self._job_name, - self._model_task_name) - tf.io.gfile.makedirs(self._decoder_dir) - - self._decode_path = None - # Multitask params doesn't have 'task'. - if 'task' in self.params: - self._decode_path = checkpointer.GetSpecificCheckpoint( - self.params.task.eval.load_checkpoint_from) - - self._summary_writer = self._CreateSummaryWriter(self._decoder_dir) - self._should_report_metrics = self._job_name.startswith( - FLAGS.vizier_reporting_job) - - with self._graph.as_default(), tf.container(self._container_id): - with self._cluster, tf.device(self._cluster.GetPlacer()): - self._model = self.params.Instantiate() - self._params = self._model.params - self._task = self._model.GetTask(self._model_task_name) - # Note, different graphs are being constructed for different model - # tasks, which may result in different node names being chosen. - # Obviously, variable names has to be stay the same between train and - # decode. - cluster = self._cluster - with tf.device(cluster.input_device): - input_batch = (self._task.input_generator.GetPreprocessedInputBatch()) - - self._dec_output = self._task.Decode(input_batch) - self._summary_op = tf.summary.merge_all() - self.checkpointer = self._CreateCheckpointer(self._train_dir, - self._model) - self._initialize_tables = tf.tables_initializer() - self._initialize_local_vars = tf.local_variables_initializer() - # No queues are allowed for decoder models. - # Ugh, why??? - self.enqueue_ops = tf.get_collection(py_utils.ENQUEUE_OPS) - assert not self.enqueue_ops - - # Saves the graph def. - self._WriteToLog(self.params.ToText(), self._decoder_dir, 'params.txt') - if self.params.cluster.task == 0: - tf.io.write_graph(self._graph.as_graph_def(), self._decoder_dir, - '%s.pbtxt' % self._job_name) - - def _CreateCheckpointer(self, train_dir, model): - """Wrapper method for override purposes.""" - return checkpointer.Checkpointer(train_dir, model) - - def Start(self): - self._RunLoop(self._job_name, self._Loop) - - def _Loop(self): - with tf.container(self._container_id), self._cluster, self._GetSession( - inline=False) as sess: - # This initializes local tables - sess.run(self._initialize_tables) - # This initializes local variables. - sess.run(self._initialize_local_vars) - - if self._decode_path: - self.DecodeCheckpoint(sess, self._decode_path) - else: - path = None - while True: - path = self._FindNewCheckpoint(path, sess) - if not path or self.DecodeCheckpoint(sess, path): - break - - # Maybe decode the last checkpoint if we are not given a specific - # checkpoint to decode. - if self._decode_path is None: - self.DecodeLatestCheckpoint(path) - - if self._should_report_metrics: - tf.logging.info('Reporting trial done.') - self._trial.ReportDone() - tf.logging.info('Decoding finished.') - - @classmethod - def GetDecodeOutPath(cls, decoder_dir, checkpoint_id): - """Gets the path to decode out file.""" - out_dir = cls._GetTtlDir(decoder_dir, duration='7d') - return os.path.join(out_dir, 'decoder_out_%09d' % checkpoint_id) - - def GetCkptIdFromFile(self, checkpoint_path): - return int(re.sub(r'.*ckpt-', '', checkpoint_path)) - - def DecodeCheckpoint(self, sess, checkpoint_path): - """Decodes `samples_per_summary` examples using `checkpoint_path`.""" - p = self._task.params - ckpt_id_from_file = self.GetCkptIdFromFile(checkpoint_path) - if ckpt_id_from_file < p.eval.start_decoder_after: - return False - samples_per_summary = p.eval.decoder_samples_per_summary - if not samples_per_summary: - samples_per_summary = p.eval.samples_per_summary - self.checkpointer.RestoreFromPath(sess, checkpoint_path) - - global_step = sess.run(py_utils.GetGlobalStep()) - - if self._task.params.input.resettable: - tf.logging.info('Resetting input_generator.') - self._task.input.Reset(sess) - - dec_metrics = self._task.CreateDecoderMetrics() - if not dec_metrics: - tf.logging.info('Empty decoder metrics') - return - buffered_decode_out = [] - num_examples_metric = dec_metrics['num_samples_in_batch'] - start_time = time.time() - while num_examples_metric.total_value < samples_per_summary: - tf.logging.info('Fetching dec_output.') - fetch_start = time.time() - run_options = tf.RunOptions(report_tensor_allocations_upon_oom=False) - if self._summary_op is None: - # No summaries were collected. - dec_out = sess.run(self._dec_output, options=run_options) - else: - dec_out, summary = sess.run([self._dec_output, self._summary_op], - options=run_options) - self._summary_writer.add_summary(summary, global_step) - post_process_start = time.time() - tf.logging.info('Done fetching (%f seconds)' % - (post_process_start - fetch_start)) - decode_out = self._task.PostProcessDecodeOut(dec_out, dec_metrics) - if decode_out: - buffered_decode_out.extend(decode_out) - tf.logging.info( - 'Total examples done: %d/%d ' - '(%f seconds decode postprocess)', num_examples_metric.total_value, - samples_per_summary, - time.time() - post_process_start) - tf.logging.info('Done decoding ckpt: %s', checkpoint_path) - - summaries = {k: v.Summary(k) for k, v in dec_metrics.items()} - elapsed_secs = time.time() - start_time - example_rate = num_examples_metric.total_value / elapsed_secs - summaries['examples/sec'] = metrics.CreateScalarSummary( - 'examples/sec', example_rate) - self._WriteSummaries( - self._summary_writer, - os.path.basename(self._decoder_dir), - global_step, - summaries, - text_filename=os.path.join(self._decoder_dir, - 'score-{:08d}.txt'.format(global_step))) - self._ExportMetrics( - # Metrics expects python int, but global_step is numpy.int64. - decode_checkpoint=int(global_step), - dec_metrics=dec_metrics, - example_rate=example_rate) - # global_step and the checkpoint id from the checkpoint file might be - # different. For consistency of checkpoint filename and decoder_out - # file, use the checkpoint id as derived from the checkpoint filename. - checkpoint_id = _GetCheckpointIdForDecodeOut(ckpt_id_from_file, global_step) - decode_out_path = self.GetDecodeOutPath(self._decoder_dir, checkpoint_id) - - decode_finalize_args = base_model.DecodeFinalizeArgs( - decode_out_path=decode_out_path, decode_out=buffered_decode_out) - self._task.DecodeFinalize(decode_finalize_args) - - should_stop = global_step >= self.params.train.max_steps - if self._should_report_metrics: - tf.logging.info('Reporting eval measure for step %d.' % global_step) - trial_should_stop = self._trial.ReportEvalMeasure(global_step, - dec_metrics, - checkpoint_path) - should_stop = should_stop or trial_should_stop - return should_stop - - def DecodeLatestCheckpoint(self, last_path=None): - """Runs decoder on the latest checkpoint.""" - with tf.container( - self._container_id), self._cluster, self._GetSession() as sess: - # This initializes local tables - sess.run(self._initialize_tables) - # This initializes local variables. - sess.run(self._initialize_local_vars) - path = tf.train.latest_checkpoint(self._train_dir) - if not path: - tf.logging.info('No checkpoint available.') - return - elif path == last_path: - tf.logging.info('Latest checkpoint was already decoded.') - return - self.DecodeCheckpoint(sess, path) - - -def _GetClusterSpecDict(): - """Parses the cluster_spec flag and returns a dict.""" - job_specs = FLAGS.cluster_spec.split('@') - cluster_spec_dict = {} - for job_spec in job_specs: - # ps_host=worker1:1231,worker2:1234 - job_machines = job_spec.split('=') - if len(job_machines) != 2: - raise ValueError('Invalid job specification: %s', job_spec) - cluster_spec_dict[job_machines[0]] = job_machines[1].split(',') - - return cluster_spec_dict - - -class RunnerManager: - """Helper class for managing runners.""" - - # This is a hack so these classes can be overridded with internal - # non-public implementations. - # pylint: disable=invalid-name - inference_graph_exporter = inference_graph_exporter - model_registry = model_registry - Controller = Controller - Trainer = Trainer - TrainerTpu = TrainerTpu - Evaler = Evaler - Decoder = Decoder - ExecutorTpu = executor.ExecutorTpu - TpuDecoder = Decoder # TODO: Don't just fall back to Decoder! - - # pylint: enable=invalid-name - - def __init__(self, model): - self._model_name = model - - def MaybeLaunchTensorFlow(self): - """Starts TF machinary in this process.""" - if FLAGS.run_locally or FLAGS.tpu: - return - - tf.logging.info('Launching tensorflow.') - - target = FLAGS.tf_master - if not target.startswith('localhost'): - # E.g., trainer_client is configured w/ FLAGS.tf_master pointing to - # another job. In that case, start a local server. - cluster_spec_dict = _GetClusterSpecDict() - self._tf_server = tf.distribute.Server( - tf.train.ClusterSpec(cluster_spec_dict), - job_name=FLAGS.job, - task_index=FLAGS.task) - target = self._tf_server.target - if not FLAGS.tf_master: - FLAGS.tf_master = target - with tf.Session(target).as_default(): - value = (tf.constant(1.) + tf.constant(1.)).eval() - assert value == 2.0, 'Something is really wrong.' - tf.logging.info('Launched tensorflow.') - - def GetExecutorParams(self): - """Get the params needed to instantiate the ExecutorTpu. - - Returns: - Tuple (dict, params): - - - ps_params_dict: high_level task_name -> ProgramScheduleParams - - train_cfg: Either a SingleTaskModelParams or MultiTaskModelParams. - """ - cluster = cluster_factory.Current() - self.UpdateClusterParamsFromFlags(cluster.params, 'executor_tpu') - ps_params_dict, train_cfg = executor.GetExecutorParams( - self._model_name, cluster.params, self.model_registry) - - return ps_params_dict, train_cfg - - def GetParamsForDataset(self, job_name, dataset_name): - """Returns params for job `job_name` on the dataset `dataset_name`.""" - # Get the current cluster and update its params from flags. - cluster = cluster_factory.Current() - print("GALVEZ:job_name=", job_name) - self.UpdateClusterParamsFromFlags(cluster.params, job_name) - print("GALVEZ:job_name=", cluster.params.job) - import sys; sys.stdout.flush() - with cluster_factory.Cluster(cluster.params): - try: - cfg = self.model_registry.GetParams(self._model_name, dataset_name) - except base_model_params.DatasetError as e: - dataset_name_retry = dataset_name.title() - tf.logging.warning( - 'Exception configuring dataset %s, retrying as %s: %s', - dataset_name, dataset_name_retry, e) - cfg = self.model_registry.GetParams(self._model_name, - dataset_name_retry) - tf.logging.warning('Succeeded after retrying as %s.' % - dataset_name_retry) - cfg.cluster = cluster.params - - # Updates a few params based on flags. - if FLAGS.enqueue_max_steps is not None: - cfg.train.enqueue_max_steps = FLAGS.enqueue_max_steps - if FLAGS.saver_max_to_keep is not None: - cfg.train.save_max_to_keep = FLAGS.saver_max_to_keep - if FLAGS.saver_keep_checkpoint_every_n_hours is not None: - cfg.train.save_keep_checkpoint_every_n_hours = FLAGS.saver_keep_checkpoint_every_n_hours - return cfg - - def MaybeConfigRunDistributed(self): - """If given a `FLAGS.cluster_spec`, update flags for running distributed.""" - if not FLAGS.cluster_spec: - return - job_specs = FLAGS.cluster_spec.split('@') - cluster_spec_dict = _GetClusterSpecDict() - if FLAGS.job == 'trainer_client': - FLAGS.tf_master = 'grpc://%s' % cluster_spec_dict['worker'][FLAGS.task] - for job in cluster_spec_dict.keys(): - if job.startswith('decoder_'): - assert len(job_specs) == 1, 'Decoder jobs must run on their own' - assert ',' not in job_specs[0], 'Only single machine supported' - FLAGS.decoder_job = '/job:%s' % job - FLAGS.decoder_replicas = 1 - if job.startswith('evaler_'): - assert len(job_specs) == 1, 'Evaler jobs must run on their own' - assert ',' not in job_specs[0], 'Only single machine supported' - FLAGS.evaler_job = '/job:%s' % job - FLAGS.evaler_replicas = 1 - if job.startswith('tpu_decoder_'): - assert len(job_specs) == 1, 'TPU Decoder jobs must run on their own' - assert ',' not in job_specs[0], 'Only single machine supported' - FLAGS.decoder_job = '/job:%s' % job - FLAGS.decoder_replicas = 1 - if FLAGS.mode == 'sync' and FLAGS.job in ('controller', 'trainer_client', - 'worker', 'executor_tpu'): - FLAGS.worker_job = '/job:worker' - FLAGS.worker_replicas = len(cluster_spec_dict['worker']) - FLAGS.ps_job = '/job:worker' - FLAGS.ps_replicas = FLAGS.worker_replicas - if FLAGS.mode == 'async' and FLAGS.job in ('controller', 'trainer', 'ps'): - FLAGS.worker_job = '/job:trainer' - FLAGS.worker_replicas = len(cluster_spec_dict['trainer']) - FLAGS.ps_job = '/job:ps' - FLAGS.ps_replicas = len(cluster_spec_dict['ps']) - - def MaybeConfigCloudTpu(self): - """If given `FLAGS.tpu`, update flags for running on a Cloud TPU.""" - if not FLAGS.tpu: - return - - if not FLAGS.job: - FLAGS.job = 'trainer_client' - - # Is it possible that TPU just simply won't do inference? - if (FLAGS.job not in ('trainer_client', 'executor_tpu') - and not FLAGS.job.startswith('tpu_decoder_')): - raise ValueError('Only trainer_client, executor_tpu, and tpu_decoder jobs are ' - 'supported on TPU.') - - cluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver( - tpu=FLAGS.tpu, - project=FLAGS.gcp_project, - zone=FLAGS.tpu_zone, - job_name=FLAGS.job) - cluster_spec_dict = cluster_resolver.cluster_spec().as_dict() - - FLAGS.mode = 'sync' - FLAGS.tf_master = cluster_resolver.master() - - FLAGS.worker_job = '/job:{}'.format(FLAGS.job) - FLAGS.worker_replicas = 1 - FLAGS.worker_num_tpu_hosts = len(cluster_spec_dict[FLAGS.job]) - FLAGS.worker_tpus = ( - cluster_resolver.num_accelerators()['TPU'] * FLAGS.worker_num_tpu_hosts) - FLAGS.decoder_tpus = ( - cluster_resolver.num_accelerators()['TPU'] * FLAGS.worker_num_tpu_hosts) - FLAGS.decoder_replicas = 1 - FLAGS.ps_job = FLAGS.worker_job - FLAGS.ps_replicas = FLAGS.worker_replicas - - FLAGS.cluster_spec = ('@'.join('{}={}'.format(job, ','.join(hosts)) - for job, hosts in cluster_spec_dict.items())) - - FLAGS.xla_device = 'tpu' - FLAGS.enable_asserts = False - FLAGS.checkpoint_in_trainer_tpu = True - - def UpdateClusterParamsFromFlags(self, cluster, job_name): - """Update `cluster` with a training cluster configuration from flags.""" - cluster.mode = FLAGS.mode - cluster.job = job_name - cluster.task = FLAGS.task - cluster.do_eval = job_name in ['evaler', 'decoder', 'tpu_decoder'] - - cluster.controller.name = FLAGS.controller_job - cluster.controller.gpus_per_replica = FLAGS.controller_gpus - - cluster.worker.name = FLAGS.worker_job - cluster.worker.replicas = FLAGS.worker_replicas - cluster.worker.gpus_per_replica = FLAGS.worker_gpus - cluster.worker.tpus_per_replica = FLAGS.worker_tpus - cluster.worker.num_tpu_hosts = FLAGS.worker_num_tpu_hosts - cluster.worker.devices_per_split = FLAGS.worker_split_size - if FLAGS.tpu: - job_name = cluster.worker.name.replace('/job:', '', 1) - worker_hosts = _GetClusterSpecDict()[job_name] - cluster.worker.targets = ','.join( - 'grpc://{}'.format(host) for host in worker_hosts) - - cluster.ps.name = FLAGS.ps_job - cluster.ps.replicas = FLAGS.ps_replicas - cluster.ps.gpus_per_replica = FLAGS.ps_gpus - - cluster.input.name = FLAGS.input_job - cluster.input.replicas = FLAGS.input_replicas - cluster.input.targets = FLAGS.input_targets - - cluster.evaler.name = FLAGS.evaler_job - cluster.evaler.replicas = FLAGS.evaler_replicas - cluster.evaler.gpus_per_replica = FLAGS.evaler_gpus - - # There we go! - cluster.decoder.name = FLAGS.decoder_job - # When is this set? - cluster.decoder.replicas = FLAGS.decoder_replicas - cluster.decoder.gpus_per_replica = FLAGS.decoder_gpus - # I hope this wowkrs - cluster.decoder.tpus_per_replica = FLAGS.decoder_tpus - - # Consider adding: - cluster.decoder.num_tpu_hosts = FLAGS.worker_num_tpu_hosts - - cluster.add_summary = FLAGS.add_summary - - def _CreateRunner(self, job, model_task_name, logdir, tf_master, trial): - """Create a runner.""" - evaler_job_name_prefix = 'evaler_' - decoder_job_name_prefix = 'decoder_' - tpu_decoder_job_name_prefix = 'tpu_decoder_' - - tf.logging.info('Job %s start', job) - common_args = (model_task_name, logdir, tf_master, trial) - if job == 'controller': - cfg = self.GetParamsForDataset('controller', 'Train') - return self.Controller(cfg, *common_args) - elif job == 'trainer': - cfg = self.GetParamsForDataset('trainer', 'Train') - return self.Trainer(cfg, *common_args) - elif job == 'trainer_client': - cfg = self.GetParamsForDataset('trainer_client', 'Train') - if py_utils.use_tpu(): - return self.TrainerTpu(cfg, *common_args) - else: - return self.Trainer(cfg, *common_args) - elif job.startswith(evaler_job_name_prefix): - dataset_name = job[len(evaler_job_name_prefix):] - cfg = self.GetParamsForDataset('evaler', dataset_name) - return self.Evaler(dataset_name.lower(), cfg, *common_args) - elif job.startswith(decoder_job_name_prefix): - # so decoder_Inference means that dataset is "Inference" - # Currently here - dataset_name = job[len(decoder_job_name_prefix):] - cfg = self.GetParamsForDataset('decoder', dataset_name) - return self.Decoder(dataset_name.lower(), cfg, *common_args) - elif job.startswith(tpu_decoder_job_name_prefix): - dataset_name = job[len(tpu_decoder_job_name_prefix):] - cfg = self.GetParamsForDataset('tpu_decoder', dataset_name) - return self.TpuDecoder(dataset_name.lower(), cfg, *common_args) - elif job in ('ps', 'worker', 'input'): - self._tf_server.join() - elif job == 'executor_tpu': - # Need to base TpuDecoder on ExecutorTpu - ps_cfg_dict, train_cfg = self.GetExecutorParams() - return self.ExecutorTpu(train_cfg, ps_cfg_dict, model_task_name, logdir, - tf_master) - else: - raise ValueError('job %s is not supported' % job) - - def CreateRunners(self, jobs, logdir, trial=base_trial.NoOpTrial()): - """Creates a list of runners based on `FLAGS.mode`. - - Args: - jobs: a list of runner jobs. - logdir: the directory used for logging, usually on CNS. - trial: optional `Trial` object, used for reporting measures and early - stopping. - - Returns: - A list of `.BaseRunner`, one per job in `jobs`. - """ - - runners = [] - for j in jobs: - tf_master = FLAGS.tf_master - # Ensure that decoder or evaler threads do not clobber variables being - # updated by trainer by forcing them to use independent sessions. - if ('trainer' in jobs and - (j.startswith('decoder') or j.startswith('evaler'))): - tf_master = '' - - runner = self._CreateRunner(j, FLAGS.model_task_name, logdir, tf_master, - trial) - runners.append(runner) - return runners - - def StartRunners(self, runners): - """Runs `runners` in parallel threads. - - Returns when all of them finish. - - Args: - runners: a list of `.BaseRunner`. - - Returns: - None. - """ - threads = [] - tf.logging.info('Starting runners') - for runner in runners: - runner_class_name = str(runner) - t = threading.Thread(target=runner.Start, name=runner_class_name) - t.daemon = True - t.start() - threads.append(t) - if runner.enqueue_ops: - tf.logging.info('Total num runner.enqueue_ops: %d', - len(runner.enqueue_ops)) - for i, enqueue_op in enumerate(runner.enqueue_ops): - - def StartEnqueue(runner, op): - tf.logging.info('Starting enqueue op %s', op.name) - return lambda: runner.StartEnqueueOp(op) - - enqueue_name = '%s-enqueue-%d' % (runner_class_name, i) - tq = threading.Thread( - target=StartEnqueue(runner, enqueue_op), name=enqueue_name) - tq.start() - threads.append(tq) - tf.logging.info('Waiting for runners to finish...') - for t in threads: - tf.logging.info('Waiting for thread to finish: %s' % t.name) - while True: - t.join(1) - if not t.isAlive(): - break - tf.logging.info('All runners done.') - - def RunTrial(self, job, logdir, trial): - """A wrapper function for running a trial.""" - # Run each job in separate process/task - # TODO(rpang): add support for running evaler_test and decoder. - self.StartRunners(self.CreateRunners([job], logdir, trial)) - - def MaybeConfigRunLocally(self): - """Update flags if configured to run locally.""" - if not FLAGS.run_locally: - # Do nothing - return - - FLAGS.tf_master = tf.distribute.Server.create_local_server().target - - if not FLAGS.mode: - FLAGS.mode = 'sync' - - if not FLAGS.job: - if FLAGS.run_locally == 'tpu': - FLAGS.job = 'trainer_client' - elif FLAGS.mode == 'async': - FLAGS.job = 'controller,trainer' - else: - FLAGS.job = 'controller,trainer_client' - - FLAGS.task = 0 - local_job = '/job:localhost' - FLAGS.controller_job = local_job - - FLAGS.worker_job = local_job - FLAGS.worker_replicas = 1 - if FLAGS.run_locally == 'gpu': - if not FLAGS.worker_gpus: - FLAGS.worker_gpus = 1 - else: - FLAGS.worker_gpus = 0 - if FLAGS.run_locally == 'tpu': - FLAGS.xla_device = 'tpu' - FLAGS.enable_asserts = False - else: - FLAGS.worker_tpus = 0 - - if not FLAGS.worker_split_size: - FLAGS.worker_split_size = 1 - - FLAGS.ps_job = local_job - FLAGS.ps_replicas = 1 - FLAGS.ps_gpus = 0 - - FLAGS.input_job = local_job - FLAGS.input_replicas = 0 - - FLAGS.evaler_job = local_job - FLAGS.evaler_replicas = 1 - if FLAGS.run_locally == 'gpu': - FLAGS.evaler_gpus = 1 - else: - FLAGS.evaler_gpus = 0 - - FLAGS.decoder_job = local_job - FLAGS.decoder_replicas = 1 - if FLAGS.run_locally == 'gpu': - FLAGS.decoder_gpus = 1 - else: - FLAGS.decoder_gpus = 0 - - def InspectModel(self): - """Prints out model analysis for the model.""" - FLAGS.mode = 'sync' - p = self.GetParamsForDataset('controller', 'Train') - c = cluster_factory.Cluster(p.cluster) - with tf.Graph().as_default(), c, tf.device(c.GetPlacer()): - analysis, _ = summary_utils.ModelAnalysis(p.Instantiate()) - print(analysis) - - def InspectDatasets(self): - """Prints out datasets configured for the model.""" - cls = self.model_registry.GetClass(self._model_name) - print(','.join([dataset.lower() for dataset in datasets.GetDatasets(cls)])) - - def InspectDecoder(self): - """Prints out datasets configured for the decoder.""" - cls = self.model_registry.GetClass(self._model_name) - params = cls() - - has_decoder = False - if issubclass(cls, base_model_params.SingleTaskModelParams): - has_decoder = params.Task( - ).cls.CreateDecoderMetrics != base_model.BaseTask.CreateDecoderMetrics - else: - for _, task_param in params.Model().task_params.IterParams(): - has_decoder |= ( - task_param.cls.CreateDecoderMetrics != - base_model.BaseTask.CreateDecoderMetrics) - if has_decoder: - # We assume that the proper decoder is implemented. - self.InspectDatasets() - else: - print('') - - def SetModelName(self, model_name): - """Sets the model name.""" - self._model_name = model_name - - def WriteInferenceGraph(self): - """Generates the inference graphs for a given model.""" - inference_graph_dir = os.path.join(FLAGS.logdir, 'inference_graphs') - tf.io.gfile.makedirs(inference_graph_dir) - tf.logging.info('Writing inference graphs to dir: %s', inference_graph_dir) - - cfg = self.model_registry.GetParams(self._model_name, 'Test') - task_names = [FLAGS.model_task_name] - if (issubclass(cfg.cls, base_model.MultiTaskModel) and - not FLAGS.model_task_name): - task_names = base_model.MultiTaskModel.TaskNames(cfg) - - if FLAGS.inference_graph_filename: - # Custom inference graph. - for task_name in task_names: - filename_prefix = FLAGS.inference_graph_filename - if task_name: - filename_prefix = '%s_inference' % task_name - filename_prefix = os.path.join(inference_graph_dir, filename_prefix) - - device = '' - var_options = None - if FLAGS.inference_graph_device == 'tpu': - device = 'tpu' - var_options = 'ON_DEVICE' - device_options = inference_graph_exporter.InferenceDeviceOptions( - device=device, - retain_device_placement=False, - var_options=var_options, - gen_init_op=True, - dtype_override=None) - self.inference_graph_exporter.InferenceGraphExporter.Export( - model_cfg=cfg, - model_task_name=task_name, - device_options=device_options, - export_path=filename_prefix + '.pbtxt', - random_seed=FLAGS.inference_graph_random_seed) - else: - for task_name in task_names: - filename_prefix = 'inference' - if task_name: - filename_prefix = '%s_inference' % task_name - filename_prefix = os.path.join(inference_graph_dir, filename_prefix) - - # Standard inference graph. - try: - self.inference_graph_exporter.InferenceGraphExporter.Export( - model_cfg=cfg, - model_task_name=task_name, - export_path=filename_prefix + '.pbtxt', - random_seed=FLAGS.inference_graph_random_seed) - except NotImplementedError as e: - tf.logging.error('Cannot write inference graph: %s', e) - - # TPU inference graph. Not all models support it so fail silently. - try: - device_options = self.inference_graph_exporter.InferenceDeviceOptions( - device='tpu', - retain_device_placement=False, - var_options='ON_DEVICE', - gen_init_op=True, - dtype_override=None) - self.inference_graph_exporter.InferenceGraphExporter.Export( - model_cfg=cfg, - model_task_name=task_name, - device_options=device_options, - export_path=filename_prefix + '_tpu.pbtxt', - random_seed=FLAGS.inference_graph_random_seed) - except Exception as e: # pylint: disable=broad-except - tf.logging.error('Error exporting TPU inference graph: %s' % e) - - def RunEvalerOnce(self): - """Run once evaler.""" - m = re.match(r'evaler_once_([^_@]+)@(\d+)', FLAGS.job) - dataset_name, ckpt_id = m.group(1), int(m.group(2)) - cfg = self.GetParamsForDataset('evaler', dataset_name) - evaler = self.Evaler(dataset_name.lower(), cfg, FLAGS.model_task_name, - FLAGS.logdir, FLAGS.tf_master) - evaler.EvalCheckpoint(ckpt_id) - - def Start(self): - """Start the process.""" - tf.logging.set_verbosity(tf.logging.INFO) - - if FLAGS.mode == 'inspect_model': - self.InspectModel() - return - - if FLAGS.mode == 'inspect_evaler': - self.InspectDatasets() - return - - if FLAGS.mode == 'inspect_decoder': - self.InspectDecoder() - return - - if FLAGS.mode == 'write_inference_graph': - self.WriteInferenceGraph() - return - - if FLAGS.mode == 'shell': - _StartShell(locals()) - return - - assert FLAGS.mode in ['sync', 'async'] - - self.MaybeConfigRunLocally() - self.MaybeConfigRunDistributed() - self.MaybeConfigCloudTpu() - self.MaybeLaunchTensorFlow() - - if FLAGS.job.startswith('evaler_once_'): - # E.g., trainer --model=foo.bar.Model --logdir=... - # --run_locally=cpu --mode=sync --job=evaler_once_test@65200 - self.RunEvalerOnce() - return - - self.StartRunners(self.CreateRunners(FLAGS.job.split(','), FLAGS.logdir)) - - -def main(unused_argv): - RunnerManager(FLAGS.model).Start() - - -if __name__ == '__main__': - tf.disable_eager_execution() - tf.flags.mark_flag_as_required('model') - FLAGS(sys.argv, known_only=True) - model_imports.ImportParams(FLAGS.model) - FLAGS.unparse_flags() - tf.app.run(main) diff --git a/lingvo/trainer_test.py b/lingvo/trainer_test.py deleted file mode 100644 index e9faecc0..00000000 --- a/lingvo/trainer_test.py +++ /dev/null @@ -1,454 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Tests for trainer.""" - -import os -import random -import re - -from absl.testing import flagsaver -from lingvo import base_trial -from lingvo import model_registry -from lingvo import trainer -import lingvo.compat as tf -from lingvo.core import base_input_generator -from lingvo.core import base_model -from lingvo.core import base_model_params -from lingvo.core import hyperparams -from lingvo.core import inference_graph_pb2 -from lingvo.core import test_utils -from lingvo.core import trainer_test_utils -from lingvo.tasks.image.input_generator import FakeMnistData -import numpy as np - -FLAGS = tf.flags.FLAGS - - -class BaseTrainerTest(test_utils.TestCase): - """Base class for the test cases.""" - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self._trial = base_trial.NoOpTrial() - - def setUp(self): - super().setUp() - FLAGS.model_params_override = '' - FLAGS.tf_master = tf.distribute.Server.create_local_server().target - FLAGS.vizier_reporting_job = 'decoder' - - def _CreateController(self, cfg): - return trainer.Controller(cfg, FLAGS.model_task_name, FLAGS.logdir, - FLAGS.tf_master, self._trial) - - def _CreateTrainer(self, cfg): - return trainer.Trainer(cfg, FLAGS.model_task_name, FLAGS.logdir, - FLAGS.tf_master, self._trial) - - def _CreateEvalerDev(self, cfg): - return trainer.Evaler('dev', cfg, FLAGS.model_task_name, FLAGS.logdir, - FLAGS.tf_master, self._trial) - - def _CreateDecoderDev(self, cfg): - return trainer.Decoder('dev', cfg, FLAGS.model_task_name, FLAGS.logdir, - FLAGS.tf_master, self._trial) - - def _HasFile(self, files, substr): - for f in files: - if substr in f: - return True - return False - - def _GetMatchedFileName(self, files, substr): - for f in files: - if substr in f: - return f - return None - - def _HasLine(self, filename, pattern): - """Returns True iff one line in the given file matches the pattern.""" - with tf.io.gfile.GFile(filename, 'r') as f: - lines = f.readlines() - return any(re.search(pattern, _) for _ in lines) - - -class EmptyTask(base_model.BaseTask): - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'empty' - return p - - def Inference(self): - return inference_graph_pb2.InferenceGraph() - - -class EmptyMultiTaskModel(base_model.MultiTaskModel): - - @classmethod - def Params(cls): - p = super().Params() - p.name = 'empty' - p.task_params.Define('a', EmptyTask.Params(), '') - p.task_params.Define('b', EmptyTask.Params(), '') - return p - - -@model_registry.RegisterMultiTaskModel -class EmptyMultiTaskParams(base_model_params.MultiTaskModelParams): - - def Test(self): - p = base_input_generator.BaseInputGenerator.Params() - inputs = hyperparams.Params() - for task_name in ['a', 'b']: - inputs.Define(task_name, p.Copy(), '') - return inputs - - def Model(self): - return EmptyMultiTaskModel.Params() - - -class TrainerTest(BaseTrainerTest): - - def _GetTestConfig(self): - model_name = 'image.mnist.LeNet5' - # So this is how particular parameters are obtained - cfg = model_registry.GetParams(model_name, 'Train') - cfg.cluster.task = 0 - cfg.cluster.mode = 'sync' - cfg.cluster.job = 'trainer_client' - cfg.cluster.worker.name = '/job:localhost' - cfg.cluster.worker.replicas = 1 - cfg.cluster.worker.gpus_per_replica = 0 - cfg.cluster.ps.name = '/job:localhost' - cfg.cluster.ps.replicas = 1 - cfg.cluster.ps.gpus_per_replica = 0 - - # Generate 2 inputs. - cfg.input.ckpt = FakeMnistData( - self.get_temp_dir(), train_size=2, test_size=2) - cfg.input.num_samples = 2 - cfg.input.batch_size = 2 - cfg.train.max_steps = 2 - cfg.train.ema_decay = 0.9999 - return cfg - - @flagsaver.flagsaver - def testController(self): - logdir = os.path.join(tf.test.get_temp_dir(), - 'controller_test' + str(random.random())) - FLAGS.logdir = logdir - cfg = self._GetTestConfig() - - runner_manager = trainer.RunnerManager(cfg.name) - - runner_manager.StartRunners( - [self._CreateController(cfg), - self._CreateTrainer(cfg)]) - - train_files = tf.io.gfile.glob(logdir + '/train/*') - self.assertTrue(self._HasFile(train_files, 'ckpt')) - self.assertTrue(self._HasFile(train_files, 'tfevents')) - control_files = tf.io.gfile.glob(logdir + '/control/*') - self.assertTrue(self._HasFile(control_files, 'params.txt')) - self.assertTrue(self._HasFile(control_files, 'model_analysis.txt')) - self.assertTrue(self._HasFile(control_files, 'train.pbtxt')) - self.assertTrue(self._HasFile(control_files, 'tfevents')) - - # EvalerDev may not run concurrently with Controller in a single process - # because EvalerDev loads checkpoints and overwrites states like global - # steps. - runner_manager.StartRunners([self._CreateEvalerDev(cfg)]) - - dev_files = tf.io.gfile.glob(logdir + '/eval_dev/*') - self.assertTrue(self._HasFile(dev_files, 'params.txt')) - self.assertTrue(self._HasFile(dev_files, 'eval_dev.pbtxt')) - self.assertTrue(self._HasFile(dev_files, 'tfevents')) - self.assertTrue(self._HasFile(dev_files, 'score')) - self.assertTrue( - self._HasLine(self._GetMatchedFileName(dev_files, 'score'), 'log_pplx')) - - @flagsaver.flagsaver - def testDecoder(self): - logdir = os.path.join(tf.test.get_temp_dir(), - 'decoder_test' + str(random.random())) - FLAGS.logdir = logdir - cfg = self._GetTestConfig() - - runner_manager = trainer.RunnerManager(cfg.name) - - runner_manager.StartRunners( - [self._CreateController(cfg), - self._CreateTrainer(cfg)]) - runner_manager.StartRunners([self._CreateDecoderDev(cfg)]) - - dec_files = tf.io.gfile.glob(logdir + '/decoder_dev/*') - self.assertTrue(self._HasFile(dec_files, 'params.txt')) - self.assertTrue(self._HasFile(dec_files, 'decoder_dev.pbtxt')) - self.assertTrue(self._HasFile(dec_files, 'tfevents')) - # Only the score for the 2-step checkpoint should be present. - self.assertTrue( - tf.io.gfile.exists( - os.path.join(logdir, 'decoder_dev/score-00000002.txt'))) - self.assertFalse( - tf.io.gfile.exists( - os.path.join(logdir, 'decoder_dev/score-00000000.txt'))) - self.assertTrue( - self._HasLine( - self._GetMatchedFileName(dec_files, 'score'), 'examples/sec')) - - # Test customization of an eval checkpoint. Create a new logdir / decoder - # but point the eval checkpoint to the 0th checkpoint of the most - # recent experiment. - new_logdir = os.path.join(tf.test.get_temp_dir(), - 'decoder_test' + str(random.random())) - FLAGS.logdir = new_logdir - cfg = self._GetTestConfig() - cfg.task.eval.load_checkpoint_from = os.path.join(logdir, - 'train/ckpt-00000000') - - runner_manager.StartRunners([self._CreateDecoderDev(cfg)]) - # Only the score for the 0th checkpoint should be present. - self.assertTrue( - tf.io.gfile.exists( - os.path.join(new_logdir, 'decoder_dev/score-00000000.txt'))) - self.assertFalse( - tf.io.gfile.exists( - os.path.join(new_logdir, 'decoder_dev/score-00000002.txt'))) - - @flagsaver.flagsaver - def testWriteInferenceGraph(self): - random.seed() - logdir = os.path.join(tf.test.get_temp_dir(), - 'inference_graphs' + str(random.random())) - FLAGS.logdir = logdir - cfg = 'punctuator.codelab.RNMTModel' - trainer.RunnerManager(cfg).WriteInferenceGraph() - inference_files = tf.io.gfile.glob(logdir + '/inference_graphs/*') - self.assertTrue(self._HasFile(inference_files, 'inference.pbtxt')) - self.assertTrue(self._HasFile(inference_files, 'inference_tpu.pbtxt')) - - @flagsaver.flagsaver(model_task_name='a') - def testWriteOneOfMultiTaskInferenceGraph(self): - random.seed() - logdir = os.path.join(tf.test.get_temp_dir(), - 'inference_graphs' + str(random.random())) - FLAGS.logdir = logdir - cfg = 'test.EmptyMultiTaskParams' - trainer.RunnerManager(cfg).WriteInferenceGraph() - inference_files = tf.io.gfile.glob(logdir + '/inference_graphs/*') - self.assertTrue(self._HasFile(inference_files, 'a_inference.pbtxt')) - self.assertTrue(self._HasFile(inference_files, 'a_inference_tpu.pbtxt')) - self.assertFalse(self._HasFile(inference_files, 'b_inference.pbtxt')) - self.assertFalse(self._HasFile(inference_files, 'b_inference_tpu.pbtxt')) - - @flagsaver.flagsaver - def testWriteMultiTaskInferenceGraph(self): - random.seed() - logdir = os.path.join(tf.test.get_temp_dir(), - 'inference_graphs' + str(random.random())) - FLAGS.logdir = logdir - cfg = 'test.EmptyMultiTaskParams' - trainer.RunnerManager(cfg).WriteInferenceGraph() - inference_files = tf.io.gfile.glob(logdir + '/inference_graphs/*') - self.assertTrue(self._HasFile(inference_files, 'a_inference.pbtxt')) - self.assertTrue(self._HasFile(inference_files, 'a_inference_tpu.pbtxt')) - self.assertTrue(self._HasFile(inference_files, 'b_inference.pbtxt')) - self.assertTrue(self._HasFile(inference_files, 'b_inference_tpu.pbtxt')) - - def testRunLocally(self): - logdir = os.path.join(tf.test.get_temp_dir(), - 'run_locally_test' + str(random.random())) - FLAGS.logdir = logdir - FLAGS.run_locally = 'cpu' - FLAGS.mode = 'sync' - FLAGS.model = 'image.mnist.LeNet5' - FLAGS.model_params_override = ( - 'train.max_steps: 2; input.num_samples: 2; input.ckpt: %s' % - FakeMnistData(self.get_temp_dir(), train_size=2, test_size=2)) - trainer.main(None) - - train_files = tf.io.gfile.glob(logdir + '/train/*') - self.assertTrue(self._HasFile(train_files, 'ckpt')) - self.assertTrue(self._HasFile(train_files, 'tfevents')) - control_files = tf.io.gfile.glob(logdir + '/control/*') - self.assertTrue(self._HasFile(control_files, 'params.txt')) - self.assertTrue(self._HasFile(control_files, 'model_analysis.txt')) - self.assertTrue(self._HasFile(control_files, 'train.pbtxt')) - self.assertTrue(self._HasFile(control_files, 'tfevents')) - - -class TrainerWithTrialTest(TrainerTest): - - @flagsaver.flagsaver - def testControllerTrainerEvaler(self): - trial = tf.test.mock.create_autospec(base_trial.Trial, instance=True) - self._trial = trial - - logdir = os.path.join(tf.test.get_temp_dir(), - 'controller_test' + str(random.random())) - FLAGS.logdir = logdir - cfg = self._GetTestConfig() - - trial.Name.return_value = 'trial1' - - def override_model_params(model_params): - model_params.task.softmax.num_classes = 20 - model_params.task.filter_shapes = [(5, 5, 1, 10), (5, 5, 10, 50)] - model_params.task.train.lr_schedule.decay_start = 100 - return model_params - - trial.OverrideModelParams.side_effect = override_model_params - trial.ShouldStop.return_value = False - trial.ShouldStopAndMaybeReport.return_value = False - # Stop trial once ReportEvalMeasure is called. - trial.ReportEvalMeasure.return_value = True - - runners = [self._CreateController(cfg), self._CreateTrainer(cfg)] - # Param override works. - for runner in runners: - self.assertEqual(runner.params.task.softmax.num_classes, 20) - self.assertEqual(runner.params.task.filter_shapes, [(5, 5, 1, 10), - (5, 5, 10, 50)]) - self.assertEqual(runner.params.task.train.lr_schedule.decay_start, 100) - - runner_manager = trainer.RunnerManager(cfg.name) - runner_manager.StartRunners(runners) - # Controller and trainer check whether the trial is stopped. - self.assertGreater(trial.OverrideModelParams.call_count, 0) - self.assertGreater(trial.ShouldStop.call_count, 0) - self.assertGreater(trial.ShouldStopAndMaybeReport.call_count, 0) - # Controller and trainer do not call report_measure, request_trial_stop, or - # report_done. - self.assertEqual(trial.ReportEvalMeasure.call_count, 0) - - train_files = tf.io.gfile.glob(logdir + '/train/*') - self.assertTrue(self._HasFile(train_files, 'params.txt')) - self.assertTrue(self._HasFile(train_files, 'trainer_params.txt')) - self.assertTrue(self._HasFile(train_files, 'ckpt')) - self.assertTrue(self._HasFile(train_files, 'tfevents')) - control_files = tf.io.gfile.glob(logdir + '/control/*') - self.assertTrue(self._HasFile(control_files, 'params.txt')) - self.assertTrue(self._HasFile(control_files, 'model_analysis.txt')) - self.assertTrue(self._HasFile(control_files, 'train.pbtxt')) - self.assertTrue(self._HasFile(control_files, 'tfevents')) - - # EvalerDev may not run concurrently with Controller in a single process - # because EvalerDev loads checkpoints and overwrites states like global - # steps. - self._CreateEvalerDev(cfg).EvalLatestCheckpoint() - # EvalerDev calls report_measure, request_trial_stop, and report_done. - after_eval_count = trial.ReportEvalMeasure.call_count - self.assertEqual(after_eval_count, 0) - - self._CreateDecoderDev(cfg).DecodeLatestCheckpoint() - after_decoder_count = trial.ReportEvalMeasure.call_count - self.assertGreater(after_decoder_count, 0) - - dev_files = tf.io.gfile.glob(logdir + '/eval_dev/*') - self.assertTrue(self._HasFile(dev_files, 'params.txt')) - self.assertTrue(self._HasFile(dev_files, 'eval_dev.pbtxt')) - self.assertTrue(self._HasFile(dev_files, 'tfevents')) - self.assertTrue(self._HasFile(dev_files, 'score')) - self.assertTrue( - self._HasLine(self._GetMatchedFileName(dev_files, 'score'), 'log_pplx')) - - -class ProcessFPropResultsTest(BaseTrainerTest): - - @flagsaver.flagsaver - def testIdentityRegressionModel(self): - logdir = os.path.join(tf.test.get_temp_dir(), - 'identity_regression_test' + str(random.random())) - FLAGS.logdir = logdir - - steps = 100 - cfg = trainer_test_utils.IdentityRegressionModel.Params() - cfg.cluster.task = 0 - cfg.cluster.mode = 'sync' - cfg.cluster.job = 'trainer_client' - cfg.cluster.worker.name = '/job:localhost' - cfg.cluster.worker.replicas = 1 - cfg.cluster.worker.gpus_per_replica = 0 - cfg.cluster.ps.name = '/job:localhost' - cfg.cluster.ps.replicas = 1 - cfg.cluster.ps.gpus_per_replica = 0 - cfg.train.max_steps = steps - cfg.task.train.learning_rate = 0.025 - - runners = [self._CreateController(cfg), self._CreateTrainer(cfg)] - - runner_manager = trainer.RunnerManager(cfg.name) - runner_manager.StartRunners(runners) - train = runners[1] - - # ProcessFPropResults should have been called times on the task - # and times on the model. - - # There are always 2 samples in the batch. - expected_samples_in_batch = [(2, 1.0) for _ in range(steps)] - - self.assertAllEqual( - expected_samples_in_batch, - [m['num_samples_in_batch'] for m in train._model.metrics]) - self.assertAllEqual( - expected_samples_in_batch, - [m['num_samples_in_batch'] for m in train._model._task.metrics]) - - # Global steps should increment by 1 for each batch. - expected_global_steps = [i + 1 for i in range(steps)] - self.assertAllEqual(expected_global_steps, train._model.global_steps) - self.assertAllEqual(expected_global_steps, train._model._task.global_steps) - - # The CountingInputGenerator makes [2,2] inputs that increment for each - # batch, like: - # [[0, 1], [2, 3]], - # [[4, 5], [6, 7]], - # ... - expected_input_tensors = [{ - 'input': np.array([[4 * i, 4 * i + 1], [4 * i + 2, 4 * i + 3]]) - } for i in range(steps)] - - def keep_input_tensors(tensors): - return [{'input': d['input']} for d in tensors] - - self.assertAllClose( - expected_input_tensors, - keep_input_tensors(train._model.result_per_example_tensors)) - self.assertAllClose( - expected_input_tensors, - keep_input_tensors(train._model._task.result_per_example_tensors)) - - # This model is training parameters m and b such that: - # m * (input[0] + input[1]) + b = (input[0] + input[1]) - # So we expect m = 1 and b = 0 after training. - - # m is more stable so that's the one we test with a tight tolerance. - self.assertNear( - 1.0, train._model._task.result_per_example_tensors[-1]['m'][0], 0.1) - self.assertNear(1.0, train._model.result_per_example_tensors[-1]['m'][0], - 0.1) - - # b isn't so stable but shouldn't be too crazy in size. - self.assertNear( - 0.0, train._model._task.result_per_example_tensors[-1]['b'][0], 10.0) - self.assertNear(0.0, train._model.result_per_example_tensors[-1]['b'][0], - 10.0) - - -if __name__ == '__main__': - tf.test.main() diff --git a/missing_files.json b/missing_files.json deleted file mode 100644 index ba9de8aa..00000000 --- a/missing_files.json +++ /dev/null @@ -1 +0,0 @@ -["gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/house.hbs.mars.hrs06RES2154_090401/hrs06RES2154_090401.auto.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/gov.house.hbs.hrs04W_M1100_050303/hrs04W_M1100_050303.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/gov.house.hbs.hrs05APR2359_090424bm2p/hrs05APR2359_090424bm2p.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/house.hbs.mars.hrs05RES1324_100513/hrs05RES1324_100513.auto.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/Musical_Journeys/Musical_Journeys.HD.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/house.hbs.mars.hrs04JUD2141_071213c/hrs04JUD2141_071213c.auto.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/gov.house.hbs.hrs04EI_GW2128_080123/hrs04EI_GW2128_080123.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/Gvtvnews-GVTVNEWS102609NCTV11225/Gvtvnews-GVTVNEWS102609NCTV11225.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/gov.house.hbs.hrs04JUD2141_110210b/hrs04JUD2141_110210b.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/house.hbs.mars.hrs01IR2172_101202/hrs01IR2172_101202.auto.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/gov.house.hbs.hrs05RES1324_100513/hrs05RES1324_100513.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/house.hbs.mars.hrs01Ed_W2175_101207/hrs01Ed_W2175_101207.auto.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/house.hbs.mars.hrs04JUD2141_110210b/hrs04JUD2141_110210b.auto.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/house.hbs.mars.hrs05APR2359_090424bm2p/hrs05APR2359_090424bm2p.auto.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/gov.house.hbs.hrs04JUD2141_071213c/hrs04JUD2141_071213c.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/Gvtvnews-GVTVNEWS102609NCTV11225-3/Gvtvnews-GVTVNEWS102609NCTV11533.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/house.hbs.mars.hrs05SCI2318_090331b/hrs05SCI2318_090331b.auto.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/Holland_Farmers_Market/Holland_Farmers_Market.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/Gvtvnews-GVTVNEWS030409NCTV11973-3/Gvtvnews-GVTVNEWS030409NCTV11973.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/house.hbs.mars.hrs05H_A1310_100511/hrs05H_A1310_100511.auto.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/gov.house.hbs.hrs06E_C2123_070322b/hrs06E_C2123_070322b.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/house.hbs.mars.hrs06E_C2123_070322b/hrs06E_C2123_070322b.auto.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/house.hbs.mars.hrs04TRAN2167_101202/hrs04TRAN2167_101202.auto.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/gov.house.hbs.hrs01IR2172_101202/hrs01IR2172_101202.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/gov.house.hbs.hrs04TRAN2167_101202/hrs04TRAN2167_101202.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/gov.house.hbs.hrs01Ed_W2175_101207/hrs01Ed_W2175_101207.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/cohmiv-Centennial_Park/Centennial_Park.asr.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/house.hbs.mars.hrs04EI_GW2128_080123/hrs04EI_GW2128_080123.auto.srt", "gs://the-peoples-speech-west-europe/archive_org/Mar_7_2021/CC_BY_SA_EXPANDED_LICENSES_FILTERED_ACCESS/house.hbs.mars.hrs04W_M1100_050303/hrs04W_M1100_050303.auto.srt"] \ No newline at end of file diff --git a/pip_package/MANIFEST.in b/pip_package/MANIFEST.in deleted file mode 100644 index b5a1bfdd..00000000 --- a/pip_package/MANIFEST.in +++ /dev/null @@ -1 +0,0 @@ -recursive-include lingvo/ *.so diff --git a/pip_package/README.md b/pip_package/README.md deleted file mode 100644 index 076b6ae7..00000000 --- a/pip_package/README.md +++ /dev/null @@ -1,74 +0,0 @@ -# Guide to releasing a new lingvo pip package - -Update the version number in setup.py and commit it. - -Build the docker image for building the pip package - -``` -docker build --tag tensorflow:lingvo_pip - < lingvo/pip_package/build.Dockerfile -``` - -Now, we assume that the repo is stored at /tmp/lingvo for the rest of these -instructions (either through git clone or copy_to_local). - -Enter the docker image, mapping /tmp/lingvo to /tmp/lingvo: - -``` -docker run --rm -it -v /tmp/lingvo:/tmp/lingvo tensorflow:lingvo_pip bash -``` - -From the /tmp/lingvo directory, run - -``` -rm -rf /tmp/lingvo_pip_package_build -PYTHON_MINOR_VERSION=6 pip_package/build.sh -PYTHON_MINOR_VERSION=7 pip_package/build.sh -PYTHON_MINOR_VERSION=8 pip_package/build.sh -``` - -If everything goes well, this will produce a set of wheels in -/tmp/lingvo_pip_package_build. - -``` -cd /tmp/lingvo_pip_pkg_build -``` - -To upload to the test pypi server: - -``` -python3 -m twine upload --repository-url https://test.pypi.org/legacy/ *manylinux2010*.whl -``` - -To verify that it works as intended: - -``` -python3 -m pip install -i https://test.pypi.org/simple/ --no-deps lingvo -``` - -You can test that the install worked for the common case by running a model -locally like: - -``` -mkdir -p /tmp/lingvo_test/image -cp -r /tmp/lingvo/lingvo/tasks/image/params/*.py /tmp/lingvo_test/image -cd /tmp/lingvo_test -python3 -m lingvo.trainer --model=image.mnist.LeNet5 --run_locally=cpu --logdir=/tmp/lenet5 --mode=sync -``` - -This should try to start training, but will fail if you haven't downloaded the -mnist dataset (see lingvo's base README.md). - -If this works successfully, you can then upload to the production server as -follows. - -``` -python3 -m twine upload *manylinux2010*.whl -``` - -And verify with: - -``` -python3 -m pip install lingvo -``` - -Remember to update the list of releases in the main README. diff --git a/pip_package/build.Dockerfile b/pip_package/build.Dockerfile deleted file mode 100644 index cd4d24ae..00000000 --- a/pip_package/build.Dockerfile +++ /dev/null @@ -1,39 +0,0 @@ -FROM tensorflow/tensorflow:custom-op-ubuntu16 - -ENV GITHUB_BRANCH="master" -ENV PYTHON_VERSION="3" -ENV PYTHON_MINOR_VERSION="8" -ENV PIP_MANYLINUX2010="1" - -# There are some problems with the python3 installation from custom-op-ubuntu16. -# Remove it and install new ones. -RUN apt-get remove --purge -y python3.5 python3.6 -RUN rm -f /etc/apt/sources.list.d/jonathonf-ubuntu-python-3_6-xenial.list -RUN apt-key del F06FC659 - -RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BA6932366A755776 -RUN echo "deb http://ppa.launchpad.net/deadsnakes/ppa/ubuntu xenial main" > /etc/apt/sources.list.d/deadsnakes-ppa-xenial.list -RUN apt-get update && apt-get install -y python3.6 python3.7 python3.8 python3.8-distutils -RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py - -# Download and install bazel. -RUN wget https://github.com/bazelbuild/bazel/releases/download/0.29.0/bazel-0.29.0-installer-linux-x86_64.sh > /dev/null -RUN bash bazel-0.29.0-installer-linux-x86_64.sh - -RUN for python in python3.6 python3.7 python3.8; do \ - $python get-pip.py && \ - $python -m pip install --upgrade pip setuptools auditwheel && \ - $python -m pip install --upgrade \ - dataclasses \ - grpcio>=1.24.3 \ - matplotlib \ - mock \ - model-pruning-google-research \ - numpy>=1.16.0 \ - sympy \ - twine && \ - $python -m pip install tensorflow-gpu \ - ; \ - done - -WORKDIR "/tmp/lingvo" diff --git a/pip_package/build.sh b/pip_package/build.sh deleted file mode 100755 index 611b76c8..00000000 --- a/pip_package/build.sh +++ /dev/null @@ -1,95 +0,0 @@ -#!/bin/bash -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -# This script uses custom-op docker, downloads code, builds and tests and then -# builds a pip package. - -# See README.md for instructions to use this script. - -set -e -x - -# Override the following env variables if necessary. -export PYTHON_VERSION="${PYTHON_VERSION:-3}" -export PYTHON_MINOR_VERSION="${PYTHON_MINOR_VERSION}" -export PIP_MANYLINUX2010="${PIP_MANYLINUX2010:-1}" - -if [[ -z "${PYTHON_MINOR_VERSION}" ]]; then - PYTHON="python${PYTHON_VERSION}" -else - PYTHON="python${PYTHON_VERSION}.${PYTHON_MINOR_VERSION}" -fi - -function write_to_bazelrc() { - echo "$1" >> .bazelrc -} - -function write_action_env_to_bazelrc() { - write_to_bazelrc "build --action_env $1=\"$2\"" -} - -# Remove .bazelrc if it already exist -[ -e .bazelrc ] && rm .bazelrc - -write_to_bazelrc "build -c opt" -write_to_bazelrc 'build --cxxopt="-std=c++11"' -write_to_bazelrc 'build --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0"' -write_to_bazelrc 'build --auto_output_filter=subpackages' -write_to_bazelrc 'build --copt="-Wall" --copt="-Wno-sign-compare"' -write_to_bazelrc 'build --linkopt="-lrt -lm"' -write_to_bazelrc 'build --incompatible_bzl_disallow_load_after_statement=false' -write_to_bazelrc 'query --incompatible_bzl_disallow_load_after_statement=false' - -TF_NEED_CUDA=0 -echo 'Using installed tensorflow' -TF_CFLAGS=( $(${PYTHON} -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))') ) -TF_LFLAGS="$(${PYTHON} -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))')" - -write_action_env_to_bazelrc "TF_HEADER_DIR" ${TF_CFLAGS:2} -SHARED_LIBRARY_DIR=${TF_LFLAGS:2} -SHARED_LIBRARY_NAME=$(echo $TF_LFLAGS | rev | cut -d":" -f1 | rev) -if ! [[ $TF_LFLAGS =~ .*:.* ]]; then - if [[ "$(uname)" == "Darwin" ]]; then - SHARED_LIBRARY_NAME="libtensorflow_framework.dylib" - else - SHARED_LIBRARY_NAME="libtensorflow_framework.so" - fi -fi -write_action_env_to_bazelrc "TF_SHARED_LIBRARY_DIR" ${SHARED_LIBRARY_DIR} -write_action_env_to_bazelrc "TF_SHARED_LIBRARY_NAME" ${SHARED_LIBRARY_NAME} -write_action_env_to_bazelrc "TF_NEED_CUDA" ${TF_NEED_CUDA} -write_to_bazelrc "build:manylinux2010 --crosstool_top=//third_party/toolchains/preconfig/ubuntu16.04/gcc7_manylinux2010-nvcc-cuda10.0:toolchain" - -if [[ "$PIP_MANYLINUX2010" == "1" ]]; then - write_to_bazelrc "build --config=manylinux2010" - write_to_bazelrc "test --config=manylinux2010" -fi - -# It is expected that you have git cloned this repo at the branch you want, -# ideally in our docker. - -bazel clean -bazel build ... -# Just test the core for the purposes of the pip package. -bazel test lingvo/core/... - -DST_DIR="/tmp/lingvo_pip_pkg_build" -./pip_package/build_pip_pkg.sh "$DST_DIR" ${PYTHON_VERSION} -# Comment the following line if you run this outside of the container. -if [[ "${PIP_MANYLINUX2010}" == "1" ]]; then - find "$DST_DIR" -name *.whl | xargs -n1 ./third_party/auditwheel.sh repair --plat manylinux2010_x86_64 -w "$DST_DIR" -fi - -rm .bazelrc diff --git a/pip_package/build_pip_pkg.sh b/pip_package/build_pip_pkg.sh deleted file mode 100755 index 1f8f02fa..00000000 --- a/pip_package/build_pip_pkg.sh +++ /dev/null @@ -1,70 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2019 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -e - -PLATFORM="$(uname -s | tr 'A-Z' 'a-z')" - -PIP_FILE_PREFIX="pip_package/" - -export PYTHON_VERSION="${PYTHON_VERSION:-3}" -export PYTHON_MINOR_VERSION="${PYTHON_MINOR_VERSION}" - -if [[ -z "${PYTHON_MINOR_VERSION}" ]]; then - PYTHON="python${PYTHON_VERSION}" -else - PYTHON="python${PYTHON_VERSION}.${PYTHON_MINOR_VERSION}" -fi - -function main() { - DEST=${1} - if [[ -z "${DEST}" ]]; then - echo "No destination directory provided." - exit 1 - fi - - # Create the directory, then do dirname on a non-existent file inside it to - # give us an absolute paths with tilde characters resolved to the destination - # directory. - mkdir -p "${DEST}" - DEST=$(readlink -f "${DEST}") - echo "=== destination directory: ${DEST}" - - TMPDIR=$(mktemp -d -t tmp.XXXXXXXXXX) - - echo $(date) : "=== Using tmpdir: ${TMPDIR}" - - echo "=== Copy Lingvo files" - - cp ${PIP_FILE_PREFIX}setup.py "${TMPDIR}" - cp ${PIP_FILE_PREFIX}MANIFEST.in "${TMPDIR}" - cp LICENSE "${TMPDIR}" - rsync -avm -L --exclude="*_test.py" lingvo "${TMPDIR}" - rsync -avm -L --include="*.so" --include="*_pb2.py" \ - --exclude="*.runfiles" --exclude="*_obj" --include="*/" --exclude="*" \ - bazel-bin/lingvo "${TMPDIR}" - - pushd ${TMPDIR} - echo $(date) : "=== Building wheel" - - ${PYTHON} setup.py bdist_wheel --project_name lingvo > /dev/null - cp dist/*.whl "${DEST}" - popd - rm -rf ${TMPDIR} - echo $(date) : "=== Output wheel file is in: ${DEST}" -} - -main "$@" diff --git a/pip_package/setup.py b/pip_package/setup.py deleted file mode 100644 index 96dab5d6..00000000 --- a/pip_package/setup.py +++ /dev/null @@ -1,102 +0,0 @@ -# Lint as: python3 -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""Setup script for pip package.""" -import sys - -from setuptools import find_packages -from setuptools import setup -from setuptools.command.install import install -from setuptools.dist import Distribution -import tensorflow as tf # pylint: disable=g-explicit-tensorflow-version-import - -__version__ = '0.7.2' -project_name = 'lingvo' -if '--project_name' in sys.argv: - project_name_idx = sys.argv.index('--project_name') - project_name = sys.argv[project_name_idx + 1] - sys.argv.remove('--project_name') - sys.argv.pop(project_name_idx) - -REQUIRED_PACKAGES = [ - 'dataclasses', - 'ipykernel', - 'jupyter', - 'jupyter_http_over_ws', - 'matplotlib', - 'model-pruning-google-research', - 'Pillow', - 'protobuf>=3.8,<4', - 'sklearn', - 'sympy', - 'tensorflow-gpu==' + tf.__version__, - 'waymo-open-dataset-tf-2-2-0', -] - - -class BinaryDistribution(Distribution): - """This class is needed in order to create OS specific wheels.""" - - def has_ext_modules(self): - return True - - -class InstallCommand(install): - """Override install command. - - Following: - https://github.com/bigartm/bigartm/issues/840. - """ - - def finalize_options(self): - install.finalize_options(self) - if self.distribution.has_ext_modules(): - self.install_lib = self.install_platlib - - -setup( - name=project_name, - version=__version__, - description=('Lingvo libraries.'), - author='Lingvo Authors', - author_email='lingvo-bot@google.com', - packages=find_packages(include=['lingvo*'], exclude=[]), - include_package_data=True, - python_requires='>=3.6,<=3.8', - install_requires=REQUIRED_PACKAGES, - zip_safe=False, - cmdclass={ - 'install': InstallCommand, - }, - distclass=BinaryDistribution, - # PyPI package information. - classifiers=[ - 'Development Status :: 5 - Production/Stable', - 'Intended Audience :: Developers', - 'Intended Audience :: Education', - 'Intended Audience :: Science/Research', - 'License :: OSI Approved :: Apache Software License', - 'Programming Language :: Python :: 3.6', - 'Programming Language :: Python :: 3.7', - 'Programming Language :: Python :: 3.8', - 'Topic :: Scientific/Engineering', - 'Topic :: Scientific/Engineering :: Artificial Intelligence', - 'Topic :: Software Development', - 'Topic :: Software Development :: Libraries', - 'Topic :: Software Development :: Libraries :: Python Modules', - ], - license='Apache 2.0', - keywords='Machine learning framework', -) diff --git a/publications.bib b/publications.bib deleted file mode 100644 index f74fde2f..00000000 --- a/publications.bib +++ /dev/null @@ -1,706 +0,0 @@ -% List of all Lingvo publications, grouped by topic, and listed in -% chronological order with the newest papers on the bottom of each group. - -% Common conferences. -@string{acl = {Proc. Annual Meeting of the Association for Computational Linguistics ({ACL})}} -@string{emnlp = {Proc. Conference on Empirical Methods in Natural Language Processing ({EMNLP})}} -@string{icassp = {Proc. {IEEE} International Conference on Acoustics, Speech, and Signal Processing ({ICASSP})}} -@string{iclr = {Proc. International Conference on Learning Representations ({ICLR})}} -@string{icml = {Proc. International Conference on Machine Learning ({ICML})}} -@string{interspeech = {Proc. Interspeech}} -@string{naacl = {Proc. Conference of the North American Chapter of the Association for Computational Linguistics ({NAACL})}} -@string{nips = {Advances in Neural Information Processing Systems}} -@string{slt = {Proc. {IEEE} Spoken Language Technology Workshop ({SLT})}} - -% Topic annotations, used by update_publications.sh to update the markdown index. -@string{nmt = {Translation}} -@string{tts = {Speech synthesis}} -@string{asr = {Speech recognition}} -@string{st = {Speech translation}} -@string{enhancement = {Speech enhancement}} -@string{nlu = {Language Understanding}} -@string{opt = {Optimization}} - -%%%%%%%%%%%%%%%%%%%% -% Translation papers. - -@techreport{wu2016google, - annote = nmt, - title = {Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation}, - author = {Y. Wu and M. Schuster and Z. Chen and Q. V. Le and M. Norouzi and W. Macherey and M. Krikun and Y. Cao and Q. Gao and K. Macherey and J. Klingner and A. Shah and M. Johnson and X. Liu and \L. Kaiser and S. Gouws and Y. Kato and T. Kudo and H. Kazawa and K. Stevens and G. Kurian and N. Patil and W. Wang and C. Young and J. Smith and J. Riesa and A. Rudnick and O. Vinyals and G. Corrado and M. Hughes and J. Dean}, - year = {2016}, - pdf = {https://arxiv.org/abs/1609.08144}, -} - -@article{johnson-etal-2017-googles, - annote = nmt, - title = "{G}oogle{'}s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation", - author = "Johnson, Melvin and - Schuster, Mike and - Le, Quoc V. and - Krikun, Maxim and - Wu, Yonghui and - Chen, Zhifeng and - Thorat, Nikhil and - Vi{\'e}gas, Fernanda and - Wattenberg, Martin and - Corrado, Greg and - Hughes, Macduff and - Dean, Jeffrey", - journal = "Transactions of the Association for Computational Linguistics", - volume = "5", - year = "2017", - pdf = "https://www.aclweb.org/anthology/Q17-1024", - doi = "10.1162/tacl_a_00065", - pages = "339--351", -} - -@article{eriguchi2018zero, - annote = nmt, - title={Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation}, - author={Eriguchi, Akiko and Johnson, Melvin and Firat, Orhan and Kazawa, Hideto and Macherey, Wolfgang}, - journal={arXiv preprint arXiv:1809.04686}, - year={2018}, - pdf={https://arxiv.org/pdf/1809.04686.pdf}, -} - -@inproceedings{bapna2018training, - annote = nmt, - author = {A. Bapna and M. X. Chen and O. Firat and Y. Cao and Y. Wu}, - title = {Training Deeper Neural Machine Translation Models with Transparent Attention}, - booktitle = emnlp, - year = {2018}, - pdf = {https://arxiv.org/abs/1808.07561}, -} - -@inproceedings{cherry2018revisiting, - annote = nmt, - author = {C. Cherry and G. Foster and A. Bapna and O. Firat and W. Macherey}, - title = {Revisiting Character-Based Neural Machine Translation with Capacity and Compression}, - booktitle = emnlp, - year = {2018}, - pdf = {https://arxiv.org/abs/1808.09943}, -} - -@inproceedings{chen2018best, - annote = nmt, - author = {M. X. Chen and O. Firat and A. Bapna and M. Johnson and W. Macherey and G. Foster and L. Jones and M. Schuster and N. Shazeer and N. Parmar and A. Vaswani and J. Uszkoreit and L. Kaiser and Z. Chen and Y. Wu and M. Hughes}, - title = {{The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation}}, - booktitle = acl, - year = {2018}, - pdf = {https://arxiv.org/abs/1804.09849}, -} - -@article{kuczmarski2018gender, - annote = nmt, - title={Gender-Aware Natural Language Translation}, - author={J. Kuczmarski and M. Johnson}, - year={2018}, - pdf = {https://www.tdcommons.org/dpubs_series/1577/}, -} - -@article{aharoni2019massively, - annote = nmt, - title={Massively Multilingual Neural Machine Translation}, - author={R. Aharoni and M. Johnson and O. Firat}, - booktitle = naacl, - year={2019}, - pdf = {https://arxiv.org/pdf/1903.00089.pdf}, -} - -@article{luo2019neural, - annote = nmt, - title={Neural Decipherment via Minimum-Cost Flow: From Ugaritic to Linear B}, - author={J. Luo and Y. Cao and R. Barzilay}, - booktitle = acl, - year={2019}, - url = {https://www.aclweb.org/anthology/P19-1303/}, -} - - -@inproceedings{milk, - annote = nmt, - title={Monotonic Infinite Lookback Attention for Simultaneous Machine Translation}, - author={Naveen Arivazhagan and Colin Cherry and Wolfgang Macherey and Chung-Cheng Chiu and Semih Yavuz and Ruoming Pang and Wei Li and Colin Raffel}, - booktitle = acl, - year={2019}, - url = {https://arxiv.org/pdf/1906.05218} -} - -@article{freitag2019text, - annote = nmt, - author = {M. Freitag and I. Caswell and S. Roy}, - title = {APE at Scale and Its Implications on MT Evaluation Biases}, - booktitle = {Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)}, - publisher = {Association for Computational Linguistics}, - url = {https://www.aclweb.org/anthology/W19-5204}, - year = {2019}, - pdf = {https://arxiv.org/abs/1904.04790}, -} - -@article{arivazhagan2019, - annote = nmt, - author = {N. Arivazhagan and A. Bapna and O. Firat and D. Lepikhin and M. Johnson and M. Krikun and M. X. Chen and Y. Cao and G. Foster and C. Cherry and W. Macherey and Z. Chen and Y. Wu}, - title = {Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges}, - year = {2019}, - url = {http://arxiv.org/abs/1907.05019}, - archivePrefix = {arXiv}, - eprint = {1907.05019}, -} - -@incollection{huang2019gpipe, - annote = nmt, - author = {Y. Huang and Y. Cheng and A. Bapna and O. Firat and M. X. Chen and D. Chen and H. Lee and J. Ngiam and Q. V. Le and Y. Wu and Z. Chen}, - title = {GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism}, - booktitle = nips, - year = {2019}, - url = {https://arxiv.org/abs/1811.06965} -} - - -%%%%%%%%%%%%%%%%%%%% -% Speech recognition papers. - -@inproceedings{chiu2018state, - annote = asr, - title = {State-of-the-art Speech Recognition With Sequence-to-Sequence Models}, - author = {C.-C.Chiu and T. N. Sainath and Y. Wu and R. Prabhavalkar and P. Nguyen and Z. Chen and A. Kannan and R. J. Weiss and K. Rao and K. Gonina and N. Jaitly and B. Li and J. Chorowski and M. Bacchiani}, - booktitle = icassp, - year = {2018}, - pdf = {https://arxiv.org/abs/1712.01769}, -} - -@inproceedings{toshniwal2018multilingual, - annote = asr, - title = {Multilingual Speech Recognition With A Single End-To-End Model}, - author = {S. Toshniwal and T. N. Sainath and R. J. Weiss and B. Li and P. Moreno and E. Weinstein and K. Rao}, - booktitle = icassp, - year = {2018}, - pdf = {https://arxiv.org/abs/1711.01694}, -} - -@inproceedings{li2018multidialect, - annote = asr, - title = {{Multi-Dialect Speech Recognition With a Single Sequence-to-Sequence Model}}, - author = {B. Li and T. N. Sainath and K. Sim and M. Bacchiani and E. Weinstein and P. Nguyen and Z. Chen and Y. Wu and K. Rao}, - booktitle = icassp, - year = {2018}, - pdf = {https://arxiv.org/abs/1712.01541}, -} - -@inproceedings{sainath2018no, - annote = asr, - title = {{No Need for a Lexicon? Evaluating the Value of the Pronunciation Lexica in End-to-End Models}}, - author = {T. N. Sainath and P. Prabhavalkar and S. Kumar and S. Lee and A. Kannan and D. Rybach and V. Schogol and P. Nguyen and B. Li and Y. Wu and Z. Chen and C. C. Chiu}, - booktitle = icassp, - year = {2018}, - pdf = {https://arxiv.org/abs/1712.01864}, -} - -@inproceedings{lawson2018learning, - annote = asr, - title = {{Learning hard alignments with variational inference}}, - author = {D. Lawson and C. C. Chiu and G. Tucker and C. Raffel and K. Swersky and N. Jaitly}, - booktitle = icassp, - year = {2018}, - pdf = {https://arxiv.org/abs/1705.05524}, -} - -@inproceedings{kannan2018analysis, - annote = asr, - title = {An Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model}, - author = {A. Kannan and Y. Wu and P. Nguyen and T. N. Sainath and Z. Chen and R. Prabhavalkar}, - booktitle = icassp, - year = {2018}, - pdf = {https://arxiv.org/abs/1712.01996}, -} - -@inproceedings{prabhavalkar2018minimum, - annote = asr, - author = {R. Prabhavalkar and T. N. Sainath and Y. Wu and P. Nguyen and Z. Chen and C. C. Chiu and A. Kannan}, - title = {{Minimum Word Error Rate Training for Attention-based Sequence-to-sequence Models}}, - booktitle = icassp, - year = {2018}, - pdf = {https://arxiv.org/abs/1712.01818}, -} - -@inproceedings{sainath2018improving, - annote = asr, - title = {{Improving the Performance of Online Neural Transducer Models}}, - author = {T. N. Sainath and C. C. Chiu and R. Prabhavalkar and A. Kannan and Y. Wu and P. Nguyen and Z. Chen Z}, - booktitle = icassp, - year = {2018}, - pdf = {https://arxiv.org/abs/1712.01807}, -} - -@inproceedings{chiu2018monotonic, - annote = asr, - title = {{Monotonic Chunkwise Attention}}, - author = {C. C. Chiu and C. Raffel}, - booktitle = iclr, - year = {2018}, - pdf = {https://arxiv.org/abs/1712.05382}, -} - -@inproceedings{williams2018contextual, - annote = asr, - title = {{Contextual Speech Recognition in End-to-End Neural Network Systems using Beam Search}}, - author = {I. Williams and A. Kannan and P. Aleksic and D. Rybach and T. N. Sainath TN}, - booktitle = interspeech, - year = {2018}, - pdf = {https://www.isca-speech.org/archive/Interspeech_2018/pdfs/2416.pdf}, -} - -@inproceedings{chui2018speech, - annote = asr, - title = {{Speech recognition for medical conversations}}, - author = {C. C. Chiu and A. Tripathi and K. Chou and C. Co and N. Jaitly and D. Jaunzeikare and A. Kannan and P. Nguyen and H. Sak and A. Sankar and J. Tansuwan and N. Wan and Y. Wu and X. Zhang}, - booktitle = interspeech, - year = {2018}, - pdf = {https://arxiv.org/abs/1711.07274}, -} - -@inproceedings{pang2018compression, - annote = asr, - title = {{Compression of End-to-End Models}}, - author = {R. Pang and T. N. Sainath and R. Prabhavalkar and S. Gupta and Y. Wu and S. Zhang and C. C. Chiu}, - booktitle = interspeech, - year = {2018}, - pdf = {https://www.isca-speech.org/archive/Interspeech_2018/pdfs/1025.pdf}, -} - -@inproceedings{toshniwal2018comparison, - annote = asr, - title = {A comparison of techniques for language model integration in encoder-decoder speech recognition}, - author = {S. Toshniwal and A. Kannan and C. C. Chiu and Y. Wu and T. N. Sainath and K. Livescu}, - booktitle = slt, - year = {2018}, - pdf = {https://arxiv.org/abs/1807.10857}, -} - -@inproceedings{pundak2018deep, - annote = asr, - title = {Deep Context: End-to-End Contextual Speech Recognition}, - author = {G. Pundak and T. N. Sainath and R. Prabhavalkar and A. Kannan and D. Zhao}, - booktitle = slt, - year = {2018}, - pdf = {https://arxiv.org/abs/1808.02480}, -} - -@inproceedings{li2019bytes, - annote = asr, - author = {B. Li and Y. Zhang and T. N. Sainath and Y. Wu and W. Chan}, - title = {Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytes}, - booktitle = icassp, - year = {2019}, - pdf = {https://arxiv.org/abs/1811.09021}, -} - -@inproceedings{guo2019spelling, - annote = asr, - author = {J. Guo and T. N. Sainath and R. J. Weiss}, - title = {A Spelling Correction Model for End-to-End Speech Recognition}, - booktitle = icassp, - year = {2019}, - pdf = {https://arxiv.org/abs/1902.07178}, -} - -@inproceedings{alon2019contextual, - annote = asr, - author = {U. Alon and G. Pundak and T. N. Sainath}, - title = {Contextual Speech Recognition with Difficult Negative Training Examples}, - booktitle = icassp, - year = {2019}, - pdf = {https://arxiv.org/abs/1810.12170}, -} - -@inproceedings{qin2019imperceptible, - annote = asr, - author = {Y. Qin and N. Carlini and I. Goodfellow and G. Cottrell and C. Raffel}, - title = {Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition}, - booktitle = icml, - year = {2019}, - pdf = {https://arxiv.org/abs/1903.10346}, -} - -@inproceedings{park2019specaugment, - annote = asr, - author = {D. S. Park and W. Chan and Y. Zhang and C. Chiu and B. Zoph and E. D. Cubuk and Q. V. Le}, - title = {{SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition}}, - booktitle = {arXiv}, - year = {2019}, - pdf = {https://arxiv.org/pdf/1904.08779.pdf} -} - -@inproceedings{li2019semisupervised, - annote = asr, - title = {Semi-supervised Training for End-to-end Models via Weak Distillation}, - author = {B. Li and T. N. Sainath and R. Pang and Z. Wu}, - booktitle = icassp, - year = {2019}, - pdf = {https://ieeexplore.ieee.org/document/8682172}, -} - -@inproceedings{chang2019joint, - annote = asr, - title = {Joint Endpointing and Decoding with End-to-end Models}, - author = {S.-Y. Chang and R. Prabhavalkar and Y. He and T. N. Sainath and G. Simko}, - booktitle = icassp, - year = {2019}, - pdf = {https://ieeexplore.ieee.org/document/8683109}, -} - -@inproceedings{heymann2019improving, - annote = asr, - title = {Improving CTC Using Stimulated Learning for Sequence Modeling}, - author = {J. Heymann and K. C. Sim and B. Li}, - booktitle = icassp, - year = {2019}, - pdf = {https://ieeexplore.ieee.org/document/8682700}, -} - -@inproceedings{bruguier2019phoebe, - annote = asr, - title = {Phoebe: Pronunciation-aware Contextualization for End-to-end Speech Recognition}, - author = {A. Bruguier and R. Prabhavalkar and G. Pundak and T. N. Sainath}, - booktitle = icassp, - year = {2019}, - pdf = {https://ieeexplore.ieee.org/document/8682441}, -} - -@inproceedings{he2019streaming, - annote = asr, - title = {Streaming End-to-end Speech Recognition for Mobile Devices}, - author = {Y. He and T. N. Sainath and R. Prabhavalkar and I. McGraw and R. Alvarez and D. Zhao and D. Rybach and A. Kannan and Y. Wu and R. Pang and Q. Liang and D. Bhatia and Y. Shangguan and B. Li and G. Pundak and K. C. Sim and T. Bagby and S.-Y. Chang and K. Rao and A. Gruenstein}, - booktitle = icassp, - year = {2019}, - pdf = {https://arxiv.org/abs/1811.06621}, -} - -@inproceedings{irie2019unit, - annote = asr, - author = {K. Irie and R. Prabhavalkar and A. Kannan and A. Bruguier and D. Rybach and P. Nguyen}, - title = {On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition}, - booktitle = interspeech, - year = {2019}, - pdf = {https://arxiv.org/abs/1902.01955}, -} - -@inproceedings{peyser2019numerics, - annote = asr, - title = {{Improving Performance of End-to-End ASR on Numeric Sequences}}, - author = {C. Peyser and H. Zhang and T. N. Sainath and Z. Wu}, - booktitle = interspeech, - year = {2019}, - pdf = {https://arxiv.org/abs/1907.01372}, -} - -@inproceedings{zhao2019shallow, - annote = asr, - author = {D. Zhao and T. N. Sainath and D. Rybach and D. Bhatia and B. Li and R. Pang}, - title = {Shallow-Fusion End-to-End Contextual Biasing}, - booktitle = interspeech, - year = {2019}, - pdf = {https://www.isca-speech.org/archive/Interspeech_2019/pdfs/1209.pdf}, -} - -@inproceedings{TaraRuoming19, - annote = asr, - title={Two-Pass End-to-End Speech Recognition}, - author={T. N. Sainath and R. Pang and D. Rybach and Y. He and R. Prabhavalkar and W. Li and M. Visontai and Q. Liang and T. Strohman and Y. Wu and I. McGraw and C.-C. Chiu}, - booktitle=interspeech, - year={2019}, - pdf = {https://arxiv.org/pdf/1908.10992}, -} - -@inproceedings{Chiu19longform, - annote = asr, - title={A Comparison of End-to-end Models for Long-form Speech Recognition}, - author={C.-C. Chiu and W. Han and Y. Zhang and R. Pang and S. Kishchenko and P. Nguyen and A. Narayanan and H. Liao and S. Zhang and A. Kannan and R. Prabhavalkar and Z. Chen and T. Sainath and Y. Wu}, - booktitle = asru, - year={2019}, - pdf = {https://arxiv.org/pdf/1911.02242} -} - -@inproceedings{narayanan2019longform, - annote = asr, - author={A. Narayanan and R. Prabhavalkar and C.C. Chiu and D. Rybach and T.N. Sainath and T. Strohman}, - title={Recognizing Long-Form Speech Using Streaming End-to-End Models}, - booktitle = asru, - year={2019}, - pdf = {https://arxiv.org/pdf/1910.11455} -} - -@inproceedings{sainath2020, - annote = asr, - title = {An Attention-Based Joint Acoustic and Text On-Device End-to-End Model}, - author = {Tara N. Sainath and Ruoming Pang and Ron Weiss and Yanzhang He and Chung-Cheng Chiu and Trevor Strohman}, - booktitle = icassp, - year = {2020}, -} - -@inproceedings{lu2020, - annote = asr, - title = {Speech Sentiment Analysis via Pre-Trained Features from End-To-End ASR Models}, - author = {Zhiyun Lu and Liangliang Cao and Yu Zhang and Chung-Cheng Chiu and James Fan}, - booktitle = icassp, - year = {2020}, -} - -@inproceedings{park2020, - annote = asr, - title = {SpecAugment on Large Scale Datasets}, - author = {Daniel Park and Yu Zhang and Chung-Cheng Chiu and Youzheng Chen and Bo Li and William Chan and Quoc Le and Yonghui Wu}, - booktitle = icassp, - year = {2020}, - pdf = {https://arxiv.org/abs/1912.05533} -} - -@inproceedings{e2e2020, - annote = asr, - title = {A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency}, - author = {Tara Sainath and Yanzhang He and Bo Li and Arun Narayanan and Ruoming Pang and Antoine Bruguier and Shuo-yiin Chang and Wei Li and Raziel Alvarez and Zhifeng Chen and Chung-cheng Chiu and David Garcia and Alex Gruenstein and Ke Hu and Minho Jin and Anjuli Kannan and Qiao Liang and Ian McGraw and Cal Peyser and Rohit Prabhavalkar and Golan Pundak and David Rybach and Yuan Shangguan and Yash Sheth and Trevor Strohman and Mirko Visontai and Yonghui Wu and Yu Zhang and Ding Zhao}, - booktitle = icassp, - year = {2020}, -} - -@inproceedings{conformer2020, - annote = asr, - title={Conformer: Convolution-augmented Transformer for Speech Recognition}, - author={Anmol Gulati and James Qin and Chung-Cheng Chiu and Niki Parmar and Yu Zhang and Jiahui Yu and Wei Han and Shibo Wang and Zhengdong Zhang and Yonghui Wu and Ruoming Pang}, - booktitle=interspeech, - year={2020}, - pdf = {https://arxiv.org/abs/2005.08100}, -} - -@inproceedings{contextnet2020, - annote = asr, - title={ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context}, - author={Wei Han and Zhengdong Zhang and Yu Zhang and Jiahui Yu and Chung-Cheng Chiu and James Qin and Anmol Gulati and Ruoming Pang and Yonghui Wu}, - booktitle=interspeech, - year={2020}, - pdf = {https://arxiv.org/abs/2005.03191}, -} - -@inproceedings{rescore2020, - annote = asr, - title={Parallel Rescoring with Transformer for Streaming On-Device Speech Recognition}, - author={Wei Li and James Qin and Chung-Cheng Chiu and Ruoming Pang and Yanzhang He}, - booktitle=interspeech, - year={2020} -} - -@inproceedings{noisystudent2020, - annote = asr, - title={Improved Noisy Student Training for Automatic Speech Recognition}, - author={Daniel S. Park and Yu Zhang and Ye Jia and Wei Han and Chung-Cheng Chiu and Bo Li and Yonghui Wu and Quoc V. Le}, - booktitle=interspeech, - year={2020}, - pdf = {https://arxiv.org/abs/2005.09629} -} - -%%%%%%%%%%%%%%%%%%%% -% Language understanding papers. - -@inproceedings{kannan2018semi, - annote = nlu, - author = {A. Kannan and K. Chen and D. Jaunzeikare and A. Rajkomar}, - title = {{Semi-Supervised Learning for Information Extraction from Dialogue}}, - booktitle = interspeech, - year = {2018}, - pdf = {https://www.isca-speech.org/archive/Interspeech_2018/pdfs/1318.pdf}, -} - -@inproceedings{yavuz2018calcs, - annote = nlu, - author = {S. Yavuz and C. C. Chiu and P. Nguyen and Y. Wu}, - title = {{CaLcs: Continuously Approximating Longest Common Subsequence for Sequence Level Optimization}}, - booktitle = emnlp, - year = {2018}, - pdf = {http://aclweb.org/anthology/D18-1406}, -} - -@inproceedings{haghani2018s2p, - annote = nlu, - author = {P. Haghani and A. Narayanan and M. Bacchiani and G. Chuang and N. Gaur and P. Moreno and R. Prabhavalkar and Z. Qu and A. Waters}, - title = {{From Audio to Semantics: Approaches to End-to-End Spoken Language Understanding}}, - booktitle = slt, - year = {2018}, - pdf = {https://arxiv.org/abs/1809.09190}, -} - -@inproceedings{chen2019smartcompose, - annote = nlu, - author = {M. X. Chen and B. N. Lee and G. Bansal and Y. Cao and S. Zhang and J. Lu and J. Tsay and Y. Wang and A. M. Dai and Z. Chen and T. Sohn and Y. Wu}, - title = {Gmail Smart Compose: Real-Time Assisted Writing}, - booktitle = {Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining}, - year = {2019}, - publisher = {Association for Computing Machinery}, - url = {https://dl.acm.org/doi/10.1145/3292500.3330723}, - pdf = {https://arxiv.org/abs/1906.00080}, -} - - -%%%%%%%%%%%%%%%%%%%% -% Speech synthesis papers. - -@inproceedings{shen2018natural, - annote = tts, - title = {Natural {TTS} Synthesis by Conditioning {WaveNet} on Mel Spectrogram Predictions}, - author = {J. Shen and R. Pang and R. J. Weiss and M. Schuster and N. Jaitly and Z. Yang and Z. Chen and Y. Zhang and Y. Wang and R.J. Skerry-Ryan and R. A. Saurous and Y. Agiomyrgiannakis and Y. Wu}, - booktitle = icassp, - year = {2018}, - pdf = {https://arxiv.org/abs/1712.05884}, - sound_examples = {https://google.github.io/tacotron/publications/tacotron2/index.html}, -} - -@inproceedings{chorowski2018styletransfer, - annote = tts, - title = {On Using Backpropagation for Speech Texture Generation and Voice Conversion}, - author = {J. Chorowski and R. J. Weiss and R. A. Saurous and S. Bengio}, - booktitle = icassp, - year = {2018}, - pdf = {https://arxiv.org/abs/1712.08363}, - sound_examples = {https://google.github.io/speech_style_transfer/samples.html}, -} - -@inproceedings{jia2018multispeaker, - annote = tts, - title = {Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis}, - author = {Y. Jia and Y. Zhang and R. J. Weiss and Q. Wang and J. Shen and F. Ren and Z. Chen and P. Nguyen and R. Pang and I. Lopez-Moreno and Y. Wu}, - booktitle = nips, - year = {2018}, - pdf = {https://arxiv.org/abs/1806.04558}, - sound_examples = {https://google.github.io/tacotron/publications/speaker_adaptation/index.html}, -} - -@inproceedings{hsu2019hierarchical, - annote = tts, - title = {Hierarchical Generative Modeling for Controllable Speech Synthesis}, - author = {W. N. Hsu and Y. Zhang and R. J. Weiss and H. Zen and Y. Wu and Y. Wang and Y. Cao and Y. Jia and Z. Chen and J. Shen and P. Nguyen and R. Pang}, - booktitle = iclr, - year = {2019}, - pdf = {https://arxiv.org/abs/1810.07217}, - sound_examples = {https://google.github.io/tacotron/publications/gmvae_controllable_tts/index.html}, -} - -@inproceedings{hsu2018disentangling, - annote = tts, - author = {W. N. Hsu and Y. Zhang and R. J. Weiss and Y. A. Chung and Y. Wang and Y. Wu and J. Glass}, - title = {Disentangling Correlated Speaker and Noise for Speech Synthesis via Data Augmentation and Adversarial Factorization}, - booktitle = {NeurIPS 2018 Workshop on Interpretability and Robustness in Audio, Speech, and Language}, - year = {2018}, - pdf = {https://openreview.net/forum?id=Bkg9ZeBB37}, -} - -@inproceedings{zen2019libritts, - annote = tts, - title = {{LibriTTS}: A Corpus Derived from {LibriSpeech} for Text-to-Speech}, - author = {H. Zen and V. Dang and R. Clark and Y. Zhang and R. J. Weiss and Y. Jia and Z. Chen and Y. Wu}, - booktitle = interspeech, - year = {2019}, - pdf = {https://arxiv.org/abs/1904.02882}, - data = {http://www.openslr.org/60}, -} - -@inproceedings{biadsy2019parrotron, - annote = tts, - author = {F. Biadsy and R. J. Weiss and P. Moreno and D. Kanvesky and Y. Jia}, - title = {Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation}, - booktitle = interspeech, - year = {2019}, - pdf = {https://arxiv.org/abs/1904.04169}, - sound_examples = {https://google.github.io/tacotron/publications/parrotron}, -} - -@inproceedings{zhang2019learning, - annote = tts, - author = {Y. Zhang and R. J. Weiss and H. Zen and Y. Wu and Z. Chen and R. J. Skerry-Ryan and Y. Jia and A. Rosenberg and B. Ramabhadran}, - title = {Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning}, - booktitle = interspeech, - year = {2019}, - pdf = {https://arxiv.org/abs/1907.04448}, - sound_examples = {http://google.github.io/tacotron/publications/multilingual}, -} - -@inproceedings{sun2020finegrained, - annote = tts, - author = {G. Sun and Y. Zhang and R. J. Weiss and Y. Cao and H. Zen and Y. Wu}, - title = {Fully-hierarchical Fine-grained Prosody Modeling for Interpretable Speech Synthesis}, - booktitle = icassp, - year = {2020}, - pdf = {https://arxiv.org/abs/2002.03785}, - sound_examples = {https://google.github.io/tacotron/publications/hierarchical_prosody}, -} - -@inproceedings{sun2020prosodyprior, - annote = tts, - author = {G. Sun and Y. Zhang and R. J. Weiss and Y. Cao and H. Zen and A. Rosenberg and B. Ramabhadran and Y. Wu}, - title = {Generating Diverse and Natural Text-to-Speech Samples Using A Quantized Fine-grained {VAE} and Auto-regressive Prosody Prior}, - booktitle = icassp, - year = {2020}, - pdf = {https://arxiv.org/abs/2002.03788}, - sound_examples = {https://google.github.io/tacotron/publications/prosody_prior}, -} - -%%%%%%%%%%%%%%%%%%%% -% Speech-to-text translation papers. - -@inproceedings{weiss2017sequence, - annote = st, - title = {Sequence-to-Sequence Models Can Directly Translate Foreign Speech}, - author = {R. J. Weiss and J. Chorowski and N. Jaitly and Y. Wu and Z. Chen}, - booktitle = interspeech, - year = {2017}, - pdf = {https://arxiv.org/abs/1703.08581}, -} - -@inproceedings{jia2019leveraging, - annote = st, - author = {Y. Jia and M. Johnson and W. Macherey and R. J. Weiss and Y. Cao and C. C. Chiu and N. Ari and S. Laurenzo and Y. Wu}, - title = {Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation}, - booktitle = icassp, - year = {2019}, - pdf = {https://arxiv.org/abs/1811.02050}, -} - -@inproceedings{jia2019direct, - annote = st, - author = {Y. Jia and R. J. Weiss and F. Biadsy and W. Macherey and M. Johnson and Z. Chen and Y. Wu}, - title = {Direct speech-to-speech translation with a sequence-to-sequence model}, - booktitle = interspeech, - year = {2019}, - pdf = {https://arxiv.org/abs/1904.06037}, - sound_examples = {https://google-research.github.io/lingvo-lab/translatotron}, -} - -%%%%%%%%%%%%%%%%%%%% -% Optimization papers - -@article{anil2020second, - annote = opt, - title={Second Order Optimization Made Practical}, - author={Anil, Rohan and Gupta, Vineet and Koren, Tomer and Regan, Kevin and Singer, Yoram}, - journal={arXiv preprint arXiv:2002.09018}, - pdf = {https://arxiv.org/abs/2002.09018}, - year={2020} -} - -@article{agarwal2020disentangling, - annote = opt, - title={Disentangling Adaptive Gradient Methods from Learning Rates}, - author={Agarwal, Naman and Anil, Rohan and Hazan, Elad and Koren, Tomer and Zhang, Cyril}, - journal={arXiv preprint arXiv:2002.11803}, - pdf = {https://arxiv.org/abs/2002.11803}, - year={2020} -} - -@inproceedings{anil2019memory, - annote = opt, - title={Memory Efficient Adaptive Optimization}, - author={Anil, Rohan and Gupta, Vineet and Koren, Tomer and Singer, Yoram}, - booktitle={Advances in Neural Information Processing Systems}, - pdf = {https://arxiv.org/abs/1901.11150}, - pages={9749--9758}, - year={2019} -} diff --git a/setup.py b/setup.py new file mode 100644 index 00000000..a01b7929 --- /dev/null +++ b/setup.py @@ -0,0 +1,32 @@ +from setuptools import setup, find_packages + +from Cython.Build import cythonize +import numpy + +print("GALVEZ:", find_packages()) + +setup( + name="peoples-speech", + version="1.0", + packages=find_packages(), + install_requires=[ + "pyspark==3.1.2", + "absl-py", + "pandas", + "matching", + "langid", + "tqdm", + "pydub", + "ftfy", + "srt", + "pyarrow", + "textdistance", + "sox", + "Cython", + "webrtcvad" + ], + include_package_data=True, + package_data={'galvasr2': ['*.jar']}, + ext_modules=cythonize("galvasr2/align/smith_waterman.pyx"), + include_dirs=[numpy.get_include()], +) diff --git a/tf_env_collect.sh b/tf_env_collect.sh deleted file mode 100755 index 925426e4..00000000 --- a/tf_env_collect.sh +++ /dev/null @@ -1,127 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2017 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== - -set -u # Check for undefined variables - -die() { - # Print a message and exit with code 1. - # - # Usage: die - # e.g., die "Something bad happened." - - echo $@ - exit 1 -} - -echo "Collecting system information..." - -OUTPUT_FILE=tf_env.txt -python_bin_path=$(which python || which python3 || die "Cannot find Python binary") - -{ - echo - echo "== cat /etc/issue ===============================================" - uname -a - uname=`uname -s` - if [ "$(uname)" == "Darwin" ]; then - echo Mac OS X `sw_vers -productVersion` - elif [ "$(uname)" == "Linux" ]; then - cat /etc/*release | grep VERSION - fi - - echo - echo '== are we in docker =============================================' - num=`cat /proc/1/cgroup | grep docker | wc -l`; - if [ $num -ge 1 ]; then - echo "Yes" - else - echo "No" - fi - - echo - echo '== compiler =====================================================' - c++ --version 2>&1 - - echo - echo '== bazel =====================================================' - bazel version - - echo - echo '== uname -a =====================================================' - uname -a - - echo - echo '== check pips ===================================================' - pip list 2>&1 | grep "proto\|numpy\|tensorflow\|lingvo" - - - echo - echo '== check for virtualenv =========================================' - ${python_bin_path} -c "import sys;print(hasattr(sys, \"real_prefix\"))" - - echo - echo '== tensorflow import ============================================' -} >> ${OUTPUT_FILE} - -cat < /tmp/check_tf.py -import tensorflow as tf; -print("tf.VERSION = %s" % tf.VERSION) -print("tf.GIT_VERSION = %s" % tf.GIT_VERSION) -print("tf.COMPILER_VERSION = %s" % tf.GIT_VERSION) -with tf.Session() as sess: - print("Sanity check: %r" % sess.run(tf.constant([1,2,3])[:1])) -EOF -${python_bin_path} /tmp/check_tf.py 2>&1 >> ${OUTPUT_FILE} - -DEBUG_LD=libs ${python_bin_path} -c "import tensorflow" 2>>${OUTPUT_FILE} > /tmp/loadedlibs - -{ - grep libcudnn.so /tmp/loadedlibs - echo - echo '== env ==========================================================' - if [ -z ${LD_LIBRARY_PATH+x} ]; then - echo "LD_LIBRARY_PATH is unset"; - else - echo LD_LIBRARY_PATH ${LD_LIBRARY_PATH} ; - fi - if [ -z ${DYLD_LIBRARY_PATH+x} ]; then - echo "DYLD_LIBRARY_PATH is unset"; - else - echo DYLD_LIBRARY_PATH ${DYLD_LIBRARY_PATH} ; - fi - - - echo - echo '== nvidia-smi ===================================================' - nvidia-smi 2>&1 - - echo - echo '== cuda libs ===================================================' -} >> ${OUTPUT_FILE} - -find /usr/local -type f -name 'libcudart*' 2>/dev/null | grep cuda | grep -v "\\.cache" >> ${OUTPUT_FILE} -find /usr/local -type f -name 'libudnn*' 2>/dev/null | grep cuda | grep -v "\\.cache" >> ${OUTPUT_FILE} - -# Remove any words with google. -mv $OUTPUT_FILE old-$OUTPUT_FILE -grep -v -i google old-${OUTPUT_FILE} > $OUTPUT_FILE - -echo "Wrote environment to ${OUTPUT_FILE}. You can review the contents of that file." -echo "and use it to populate the fields in the github issue template." -echo -echo "cat ${OUTPUT_FILE}" -echo -